2024-04-02T13:10:00.983 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-02T13:10:00.984 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} --name pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi --owner scheduled_pdonnell@teuthology --verbose -- /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584/orig.config.yaml 2024-04-02T13:10:01.034 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T13:10:01.136 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584 branch: wip-batrick-testing-20240402.004512 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} email: pdonnell@redhat.com first_in_suite: false job_id: '7635584' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240402.004512 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - inode wrongly marked free - bad backtrace on inode - inode table repaired for inode - Scrub error on inode - Scrub error on dir - Metadata damage detected sha1: 191d963ac9de7322da8aff20b567b3fa458e2ef6 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 191d963ac9de7322da8aff20b567b3fa458e2ef6 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240402.004512 sha1: 191d963ac9de7322da8aff20b567b3fa458e2ef6 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 2763 sha1: 191d963ac9de7322da8aff20b567b3fa458e2ef6 sleep_before_teardown: 0 subset: 318/512 suite: fs suite_branch: wip-batrick-testing-20240402.004512 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_191d963ac9de7322da8aff20b567b3fa458e2ef6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 191d963ac9de7322da8aff20b567b3fa458e2ef6 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi044.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOu/jrkQApjbih97vw1u9wYxTxokLSOZ2y1zgfVlNMVM4z366JseNO9ozPlyu/PESU4OadfFOZPGuTdw+IgBQE= smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI/Yrp+Fke5qTtFll39/EG3e2sGtZ0iT5jGlchGRS5la5lv3+spSiTv4T0lCCbUpx2UTqY35RkZ3zN6feEPhpvk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_forward_scrub teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-02_11:52:43 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-02T13:10:01.136 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_191d963ac9de7322da8aff20b567b3fa458e2ef6/qa; will attempt to use it 2024-04-02T13:10:01.137 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_191d963ac9de7322da8aff20b567b3fa458e2ef6/qa/tasks 2024-04-02T13:10:01.137 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-02T13:10:01.139 INFO:teuthology.task.internal:Checking packages... 2024-04-02T13:10:01.164 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '191d963ac9de7322da8aff20b567b3fa458e2ef6' 2024-04-02T13:10:01.164 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-02T13:10:01.164 INFO:teuthology.packaging:ref: None 2024-04-02T13:10:01.164 INFO:teuthology.packaging:tag: None 2024-04-02T13:10:01.164 INFO:teuthology.packaging:branch: wip-batrick-testing-20240402.004512 2024-04-02T13:10:01.164 INFO:teuthology.packaging:sha1: 191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T13:10:01.164 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240402.004512 2024-04-02T13:10:01.314 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2609.g191d963a 2024-04-02T13:10:01.316 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-02T13:10:01.326 INFO:teuthology.task.internal:no buildpackages task found 2024-04-02T13:10:01.326 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-02T13:10:01.350 INFO:teuthology.task.internal:Saving configuration 2024-04-02T13:10:01.367 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-02T13:10:01.466 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T13:10:01.492 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 13:04:58.950234', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOu/jrkQApjbih97vw1u9wYxTxokLSOZ2y1zgfVlNMVM4z366JseNO9ozPlyu/PESU4OadfFOZPGuTdw+IgBQE='} 2024-04-02T13:10:01.519 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 13:04:58.952228', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI/Yrp+Fke5qTtFll39/EG3e2sGtZ0iT5jGlchGRS5la5lv3+spSiTv4T0lCCbUpx2UTqY35RkZ3zN6feEPhpvk='} 2024-04-02T13:10:01.519 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-02T13:10:01.530 INFO:teuthology.task.internal:roles: ubuntu@smithi044.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-02T13:10:01.530 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-02T13:10:01.530 INFO:teuthology.run_tasks:Running task console_log... 2024-04-02T13:10:01.600 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3810eef160>, signals=[15]) 2024-04-02T13:10:01.601 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-02T13:10:01.607 INFO:teuthology.task.internal:Opening connections... 2024-04-02T13:10:01.607 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:10:01.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T13:10:01.711 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:10:01.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T13:10:01.799 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-02T13:10:01.805 DEBUG:teuthology.orchestra.run.smithi044:> uname -m 2024-04-02T13:10:01.823 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64 2024-04-02T13:10:01.823 DEBUG:teuthology.orchestra.run.smithi044:> cat /etc/os-release 2024-04-02T13:10:01.879 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="CentOS Stream" 2024-04-02T13:10:01.880 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="9" 2024-04-02T13:10:01.880 INFO:teuthology.orchestra.run.smithi044.stdout:ID="centos" 2024-04-02T13:10:01.880 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="rhel fedora" 2024-04-02T13:10:01.880 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="9" 2024-04-02T13:10:01.880 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el9" 2024-04-02T13:10:01.880 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-02T13:10:01.880 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31" 2024-04-02T13:10:01.880 INFO:teuthology.orchestra.run.smithi044.stdout:LOGO="fedora-logo-icon" 2024-04-02T13:10:01.880 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-02T13:10:01.880 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://centos.org/" 2024-04-02T13:10:01.880 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-02T13:10:01.880 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-02T13:10:01.880 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-02T13:10:01.881 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2024-04-02T13:10:01.915 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2024-04-02T13:10:01.933 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2024-04-02T13:10:01.933 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2024-04-02T13:10:01.988 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="CentOS Stream" 2024-04-02T13:10:01.988 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="9" 2024-04-02T13:10:01.988 INFO:teuthology.orchestra.run.smithi069.stdout:ID="centos" 2024-04-02T13:10:01.988 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="rhel fedora" 2024-04-02T13:10:01.988 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="9" 2024-04-02T13:10:01.988 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el9" 2024-04-02T13:10:01.988 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-02T13:10:01.988 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2024-04-02T13:10:01.988 INFO:teuthology.orchestra.run.smithi069.stdout:LOGO="fedora-logo-icon" 2024-04-02T13:10:01.988 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-02T13:10:01.988 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://centos.org/" 2024-04-02T13:10:01.988 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-02T13:10:01.989 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-02T13:10:01.989 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-02T13:10:01.989 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-04-02T13:10:02.014 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-02T13:10:02.023 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-02T13:10:02.037 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-02T13:10:02.037 DEBUG:teuthology.orchestra.run.smithi044:> test '!' -e /home/ubuntu/cephtest 2024-04-02T13:10:02.039 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2024-04-02T13:10:02.056 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-02T13:10:02.067 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-02T13:10:02.067 DEBUG:teuthology.orchestra.run.smithi044:> test -z $(ls -A /var/lib/ceph) 2024-04-02T13:10:02.097 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2024-04-02T13:10:02.265 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-02T13:10:02.328 INFO:teuthology.run_tasks:Running task kernel... 2024-04-02T13:10:02.369 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-02T13:10:02.369 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-02T13:10:02.369 DEBUG:teuthology.orchestra.run.smithi044:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T13:10:02.370 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T13:10:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:10:02.391 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-04-02T13:10:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:10:02.393 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-04-02T13:10:02.452 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-361.el9.x86_64 2024-04-02T13:10:02.453 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.14.0-361.el9.x86_64 2024-04-02T13:10:02.453 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum install -y kernel 2024-04-02T13:10:02.458 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-361.el9.x86_64 2024-04-02T13:10:02.458 INFO:teuthology.task.kernel:Running kernel on smithi044: 5.14.0-361.el9.x86_64 2024-04-02T13:10:02.458 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum install -y kernel 2024-04-02T13:10:04.625 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-02T13:10:04.787 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 8.9 MB/s | 6.7 MB 00:00 2024-04-02T13:10:06.092 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-04-02T13:10:06.163 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-04-02T13:10:07.062 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-02T13:10:08.416 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-04-02T13:10:10.429 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 19 MB 00:02 2024-04-02T13:10:13.900 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-04-02T13:10:15.255 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.1 MB 00:01 2024-04-02T13:10:16.772 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 16 kB 00:00 2024-04-02T13:10:18.776 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.1 MB 00:01 2024-04-02T13:10:19.067 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 21 MB 00:02 2024-04-02T13:10:20.402 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-02T13:10:21.270 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 28 MB/s | 21 MB 00:00 2024-04-02T13:10:24.048 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-02T13:10:26.313 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-02T13:10:26.313 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-02T13:10:26.361 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-02T13:10:26.408 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-02T13:10:26.419 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-02T13:10:26.419 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-02T13:10:26.419 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-02T13:10:26.419 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-02T13:10:26.420 INFO:teuthology.orchestra.run.smithi069.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-02T13:10:26.420 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-02T13:10:26.420 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-02T13:10:26.420 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-02T13:10:26.420 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-02T13:10:26.420 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:10:26.420 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-02T13:10:26.420 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-02T13:10:26.420 INFO:teuthology.orchestra.run.smithi069.stdout:Install 4 Packages 2024-04-02T13:10:26.420 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:10:26.421 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 76 M 2024-04-02T13:10:26.421 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 123 M 2024-04-02T13:10:26.421 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-02T13:10:26.566 INFO:teuthology.orchestra.run.smithi069.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-04-02T13:10:27.099 INFO:teuthology.orchestra.run.smithi069.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 22 MB/s | 15 MB 00:00 2024-04-02T13:10:27.758 INFO:teuthology.orchestra.run.smithi069.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 25 MB/s | 34 MB 00:01 2024-04-02T13:10:28.158 INFO:teuthology.orchestra.run.smithi069.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-04-02T13:10:28.161 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-02T13:10:28.161 INFO:teuthology.orchestra.run.smithi069.stdout:Total 44 MB/s | 76 MB 00:01 2024-04-02T13:10:28.162 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-02T13:10:28.512 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-02T13:10:28.513 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-02T13:10:28.752 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-02T13:10:28.752 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-02T13:10:28.852 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-02T13:10:28.860 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-02T13:10:28.860 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-04-02T13:10:28.860 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-02T13:10:28.860 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-02T13:10:28.860 INFO:teuthology.orchestra.run.smithi044.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-02T13:10:28.860 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-04-02T13:10:28.860 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-02T13:10:28.860 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-02T13:10:28.860 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-02T13:10:28.860 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:10:28.860 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-02T13:10:28.860 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-02T13:10:28.860 INFO:teuthology.orchestra.run.smithi044.stdout:Install 4 Packages 2024-04-02T13:10:28.861 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:10:28.861 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 76 M 2024-04-02T13:10:28.861 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 123 M 2024-04-02T13:10:28.861 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-02T13:10:29.005 INFO:teuthology.orchestra.run.smithi044.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-04-02T13:10:29.364 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-02T13:10:29.364 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-02T13:10:29.563 INFO:teuthology.orchestra.run.smithi044.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-04-02T13:10:30.072 INFO:teuthology.orchestra.run.smithi044.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-04-02T13:10:30.472 INFO:teuthology.orchestra.run.smithi044.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-04-02T13:10:30.475 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-02T13:10:30.475 INFO:teuthology.orchestra.run.smithi044.stdout:Total 47 MB/s | 76 MB 00:01 2024-04-02T13:10:30.475 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-02T13:10:30.919 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-02T13:10:30.919 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-02T13:10:31.663 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-02T13:10:31.788 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-02T13:10:31.788 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-02T13:10:32.364 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-02T13:10:32.458 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T13:10:34.209 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-02T13:10:34.827 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-02T13:10:34.914 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T13:10:35.198 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T13:10:35.334 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T13:10:37.785 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T13:10:37.926 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T13:10:40.035 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T13:10:40.158 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T13:10:43.002 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T13:10:43.042 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T13:10:44.904 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T13:10:48.147 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T13:11:14.229 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T13:11:14.229 INFO:teuthology.orchestra.run.smithi069.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-02T13:11:14.230 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:11:14.273 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-02T13:11:14.634 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T13:11:14.635 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-02T13:11:14.635 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T13:11:14.635 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T13:11:15.068 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T13:11:15.069 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:11:15.069 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-02T13:11:15.069 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-02T13:11:15.069 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-02T13:11:15.069 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-02T13:11:15.069 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-02T13:11:15.069 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:11:15.069 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-02T13:11:15.392 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T13:11:15.535 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T13:11:15.536 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T13:11:15.536 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-02T13:11:15.536 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-02T13:11:15.536 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-02T13:11:15.536 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-02T13:11:15.537 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2024-04-02T13:11:15.537 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T13:11:15.615 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T13:11:15.616 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T13:11:15.616 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -qi grub2-tools 2024-04-02T13:11:15.654 INFO:teuthology.orchestra.run.smithi069.stdout:Name : grub2-tools 2024-04-02T13:11:15.654 INFO:teuthology.orchestra.run.smithi069.stdout:Epoch : 1 2024-04-02T13:11:15.654 INFO:teuthology.orchestra.run.smithi069.stdout:Version : 2.06 2024-04-02T13:11:15.654 INFO:teuthology.orchestra.run.smithi069.stdout:Release : 68.el9 2024-04-02T13:11:15.654 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: x86_64 2024-04-02T13:11:15.654 INFO:teuthology.orchestra.run.smithi069.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-02T13:11:15.654 INFO:teuthology.orchestra.run.smithi069.stdout:Group : Unspecified 2024-04-02T13:11:15.654 INFO:teuthology.orchestra.run.smithi069.stdout:Size : 8264254 2024-04-02T13:11:15.654 INFO:teuthology.orchestra.run.smithi069.stdout:License : GPLv3+ 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout:Packager : builder@centos.org 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout:Vendor : CentOS 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout:Summary : Support tools for GRUB. 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout:Description : 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout:hardware devices. 2024-04-02T13:11:15.655 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:11:15.656 INFO:teuthology.orchestra.run.smithi069.stdout:This subpackage provides tools for support of all platforms. 2024-04-02T13:11:15.656 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-02T13:11:15.656 INFO:teuthology.task.kernel:Updating grub on smithi069 to boot 5.14.0-434.el9.x86_64 2024-04-02T13:11:15.657 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-02T13:11:16.258 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2024-04-02T13:11:17.665 INFO:teuthology.orchestra.run.smithi069.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-02T13:11:17.696 INFO:teuthology.orchestra.run.smithi069.stderr:done 2024-04-02T13:11:17.698 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-04-02T13:11:17.713 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.RoMLQkZQGX 2024-04-02T13:11:17.713 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RoMLQkZQGX 2024-04-02T13:11:17.781 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.RoMLQkZQGX 2024-04-02T13:11:17.927 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.RoMLQkZQGX is 6KB 2024-04-02T13:11:17.975 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.RoMLQkZQGX 2024-04-02T13:11:17.990 DEBUG:teuthology.orchestra.run.smithi069:> sudo /bin/ls /boot/loader/entries || true 2024-04-02T13:11:18.060 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-02T13:11:18.061 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-02T13:11:18.061 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-02T13:11:18.061 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-02T13:11:18.062 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-02T13:11:18.216 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2024-04-02T13:11:19.157 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T13:11:19.157 INFO:teuthology.orchestra.run.smithi044.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-02T13:11:19.157 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:11:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-02T13:11:19.582 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T13:11:19.582 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-02T13:11:19.582 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T13:11:19.582 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T13:11:20.023 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T13:11:20.024 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:11:20.024 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-02T13:11:20.024 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-02T13:11:20.024 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-02T13:11:20.024 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-02T13:11:20.024 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-02T13:11:20.024 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:11:20.024 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-02T13:11:20.373 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T13:11:20.522 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T13:11:20.523 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T13:11:20.523 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-02T13:11:20.523 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-02T13:11:20.523 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-02T13:11:20.523 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-02T13:11:20.523 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi044.front.sepia.ceph.com, path=None, version=distro) 2024-04-02T13:11:20.523 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T13:11:20.607 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T13:11:20.607 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T13:11:20.607 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -qi grub2-tools 2024-04-02T13:11:20.654 INFO:teuthology.orchestra.run.smithi044.stdout:Name : grub2-tools 2024-04-02T13:11:20.654 INFO:teuthology.orchestra.run.smithi044.stdout:Epoch : 1 2024-04-02T13:11:20.654 INFO:teuthology.orchestra.run.smithi044.stdout:Version : 2.06 2024-04-02T13:11:20.654 INFO:teuthology.orchestra.run.smithi044.stdout:Release : 68.el9 2024-04-02T13:11:20.654 INFO:teuthology.orchestra.run.smithi044.stdout:Architecture: x86_64 2024-04-02T13:11:20.655 INFO:teuthology.orchestra.run.smithi044.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-02T13:11:20.655 INFO:teuthology.orchestra.run.smithi044.stdout:Group : Unspecified 2024-04-02T13:11:20.655 INFO:teuthology.orchestra.run.smithi044.stdout:Size : 8264254 2024-04-02T13:11:20.655 INFO:teuthology.orchestra.run.smithi044.stdout:License : GPLv3+ 2024-04-02T13:11:20.655 INFO:teuthology.orchestra.run.smithi044.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-02T13:11:20.655 INFO:teuthology.orchestra.run.smithi044.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-02T13:11:20.655 INFO:teuthology.orchestra.run.smithi044.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-02T13:11:20.655 INFO:teuthology.orchestra.run.smithi044.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-02T13:11:20.655 INFO:teuthology.orchestra.run.smithi044.stdout:Packager : builder@centos.org 2024-04-02T13:11:20.655 INFO:teuthology.orchestra.run.smithi044.stdout:Vendor : CentOS 2024-04-02T13:11:20.655 INFO:teuthology.orchestra.run.smithi044.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-02T13:11:20.655 INFO:teuthology.orchestra.run.smithi044.stdout:Summary : Support tools for GRUB. 2024-04-02T13:11:20.656 INFO:teuthology.orchestra.run.smithi044.stdout:Description : 2024-04-02T13:11:20.656 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:11:20.656 INFO:teuthology.orchestra.run.smithi044.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-02T13:11:20.656 INFO:teuthology.orchestra.run.smithi044.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-02T13:11:20.656 INFO:teuthology.orchestra.run.smithi044.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-02T13:11:20.656 INFO:teuthology.orchestra.run.smithi044.stdout:hardware devices. 2024-04-02T13:11:20.656 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:11:20.656 INFO:teuthology.orchestra.run.smithi044.stdout:This subpackage provides tools for support of all platforms. 2024-04-02T13:11:20.657 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-02T13:11:20.657 INFO:teuthology.task.kernel:Updating grub on smithi044 to boot 5.14.0-434.el9.x86_64 2024-04-02T13:11:20.657 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-02T13:11:21.212 INFO:teuthology.orchestra.run.smithi044.stderr:Generating grub configuration file ... 2024-04-02T13:11:22.520 INFO:teuthology.orchestra.run.smithi044.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-02T13:11:22.546 INFO:teuthology.orchestra.run.smithi044.stderr:done 2024-04-02T13:11:22.549 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2024-04-02T13:11:22.564 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.I38z5czndk 2024-04-02T13:11:22.565 DEBUG:teuthology.orchestra.run.smithi044:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.I38z5czndk 2024-04-02T13:11:22.638 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.I38z5czndk 2024-04-02T13:11:22.783 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.I38z5czndk is 6KB 2024-04-02T13:11:22.831 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.I38z5czndk 2024-04-02T13:11:22.846 DEBUG:teuthology.orchestra.run.smithi044:> sudo /bin/ls /boot/loader/entries || true 2024-04-02T13:11:22.913 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-02T13:11:22.914 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-02T13:11:22.914 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-02T13:11:22.914 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-02T13:11:22.915 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-02T13:11:23.076 DEBUG:teuthology.orchestra.run.smithi044:> sudo shutdown -r now 2024-04-02T13:11:48.244 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-02T13:11:48.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-02T13:11:48.246 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T13:11:53.083 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-02T13:11:53.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-04-02T13:11:53.084 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T13:12:48.303 DEBUG:teuthology.orchestra.remote:timed out 2024-04-02T13:12:53.090 DEBUG:teuthology.orchestra.remote:timed out 2024-04-02T13:12:57.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-02T13:12:57.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T13:13:00.733 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-04-02T13:13:01.176 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-02T13:13:01.176 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-04-02T13:13:01.177 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-04-02T13:13:01.192 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-434.el9.x86_64 2024-04-02T13:13:01.192 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-02T13:13:01.192 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-02T13:13:01.192 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-02T13:13:02.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-04-02T13:13:02.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T13:13:02.222 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-02T13:13:02.222 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-02T13:13:02.317 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2024-04-02T13:13:02.360 DEBUG:teuthology.parallel:result is None 2024-04-02T13:13:02.593 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-04-02T13:13:02.997 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-04-02T13:13:02.997 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-04-02T13:13:02.997 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-04-02T13:13:03.011 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-434.el9.x86_64 2024-04-02T13:13:03.011 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-02T13:13:03.011 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-02T13:13:03.011 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-02T13:13:04.012 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-02T13:13:04.013 DEBUG:teuthology.orchestra.run.smithi044:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-02T13:13:04.094 INFO:teuthology.orchestra.run.smithi044.stdout:ttyS1 2024-04-02T13:13:04.139 DEBUG:teuthology.parallel:result is None 2024-04-02T13:13:04.140 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-02T13:13:04.149 INFO:teuthology.task.internal:Creating test directory... 2024-04-02T13:13:04.150 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T13:13:04.152 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T13:13:04.166 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-02T13:13:04.174 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-02T13:13:04.188 INFO:teuthology.task.internal:Creating archive directory... 2024-04-02T13:13:04.188 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T13:13:04.203 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T13:13:04.234 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-02T13:13:04.244 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-02T13:13:04.245 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T13:13:04.268 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T13:13:04.300 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T13:13:04.311 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T13:13:04.313 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T13:13:04.325 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T13:13:04.327 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-02T13:13:04.335 INFO:teuthology.task.internal:Configuring sudo... 2024-04-02T13:13:04.335 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T13:13:04.338 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T13:13:04.380 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-02T13:13:04.391 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-02T13:13:04.392 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T13:13:04.402 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T13:13:04.430 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T13:13:04.493 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T13:13:04.532 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T13:13:04.612 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T13:13:04.650 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:13:04.651 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T13:13:04.715 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T13:13:04.757 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T13:13:04.803 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T13:13:04.882 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T13:13:04.919 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:13:04.920 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T13:13:04.984 DEBUG:teuthology.orchestra.run.smithi044:> sudo service rsyslog restart 2024-04-02T13:13:04.986 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2024-04-02T13:13:05.040 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T13:13:05.102 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T13:13:05.402 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-02T13:13:05.410 INFO:teuthology.task.internal:Starting timer... 2024-04-02T13:13:05.410 INFO:teuthology.run_tasks:Running task pcp... 2024-04-02T13:13:05.435 INFO:teuthology.run_tasks:Running task selinux... 2024-04-02T13:13:05.446 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-02T13:13:05.447 DEBUG:teuthology.orchestra.run.smithi044:> sudo service auditd rotate 2024-04-02T13:13:05.517 INFO:teuthology.orchestra.run.smithi044.stdout:Rotating logs: 2024-04-02T13:13:05.519 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2024-04-02T13:13:05.642 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2024-04-02T13:13:05.644 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-02T13:13:05.644 DEBUG:teuthology.orchestra.run.smithi044:> /usr/sbin/getenforce 2024-04-02T13:13:05.661 INFO:teuthology.orchestra.run.smithi044.stdout:Permissive 2024-04-02T13:13:05.662 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2024-04-02T13:13:05.713 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2024-04-02T13:13:05.713 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi044.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi069.front.sepia.ceph.com': 'permissive'} 2024-04-02T13:13:05.714 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T13:13:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:13:05.742 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T13:13:05.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:13:05.778 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-02T13:13:05.778 DEBUG:teuthology.orchestra.run.smithi044:> sudo /usr/sbin/setenforce permissive 2024-04-02T13:13:05.814 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2024-04-02T13:13:06.108 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-02T13:13:06.118 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-02T13:13:06.124 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-02T13:13:06.270 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-02T13:13:06.287 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-02T13:13:06.288 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi044.front.sepia.ceph.com,smithi069.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-02T13:17:17.379 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi044.front.sepia.ceph.com'), Remote(name='ubuntu@smithi069.front.sepia.ceph.com')] 2024-04-02T13:17:17.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-04-02T13:17:17.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T13:17:17.470 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-04-02T13:17:17.504 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-04-02T13:17:17.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-02T13:17:17.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T13:17:17.592 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-04-02T13:17:17.669 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-02T13:17:17.669 INFO:teuthology.run_tasks:Running task clock... 2024-04-02T13:17:17.683 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-02T13:17:17.683 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T13:17:17.684 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T13:17:17.686 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T13:17:17.687 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T13:17:17.717 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-02T13:17:17.734 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-02T13:17:17.741 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-02T13:17:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-02T13:17:17.763 INFO:teuthology.orchestra.run.smithi044.stderr:sudo: ntpd: command not found 2024-04-02T13:17:17.777 INFO:teuthology.orchestra.run.smithi044.stdout:506 Cannot talk to daemon 2024-04-02T13:17:17.786 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2024-04-02T13:17:17.794 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-02T13:17:17.800 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2024-04-02T13:17:17.810 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-02T13:17:17.816 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-02T13:17:17.833 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-02T13:17:17.862 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-04-02T13:17:17.865 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T13:17:17.865 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-04-02T13:17:17.865 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T13:17:17.865 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T13:17:17.865 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T13:17:17.865 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T13:17:17.892 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-04-02T13:17:17.895 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T13:17:17.895 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-04-02T13:17:17.895 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T13:17:17.895 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T13:17:17.895 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T13:17:17.895 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T13:17:17.897 INFO:teuthology.run_tasks:Running task install... 2024-04-02T13:17:17.906 DEBUG:teuthology.task.install:project ceph 2024-04-02T13:17:17.907 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6'}} 2024-04-02T13:17:17.907 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6'} 2024-04-02T13:17:17.907 INFO:teuthology.task.install:Using flavor: default 2024-04-02T13:17:17.913 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-02T13:17:17.913 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-02T13:17:17.913 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6', 'tag': None, 'wait_for_package': False} 2024-04-02T13:17:17.914 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T13:17:17.916 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6', 'tag': None, 'wait_for_package': False} 2024-04-02T13:17:17.917 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T13:17:18.073 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/ 2024-04-02T13:17:18.073 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2609.g191d963a 2024-04-02T13:17:18.154 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/ 2024-04-02T13:17:18.155 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2609.g191d963a 2024-04-02T13:17:18.261 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-02T13:17:18.261 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:17:18.262 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-02T13:17:18.271 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-02T13:17:18.271 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:17:18.271 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-02T13:17:18.290 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-02T13:17:18.290 DEBUG:teuthology.orchestra.run.smithi069:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/191d963ac9de7322da8aff20b567b3fa458e2ef6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-02T13:17:18.299 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-02T13:17:18.299 DEBUG:teuthology.orchestra.run.smithi044:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/191d963ac9de7322da8aff20b567b3fa458e2ef6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-02T13:17:18.358 DEBUG:teuthology.orchestra.run.smithi069:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-02T13:17:18.367 DEBUG:teuthology.orchestra.run.smithi044:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-02T13:17:18.438 DEBUG:teuthology.orchestra.run.smithi069:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-02T13:17:18.448 DEBUG:teuthology.orchestra.run.smithi044:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-02T13:17:18.504 INFO:teuthology.orchestra.run.smithi069.stdout:check_obsoletes = 1 2024-04-02T13:17:18.506 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum clean all 2024-04-02T13:17:18.514 INFO:teuthology.orchestra.run.smithi044.stdout:check_obsoletes = 1 2024-04-02T13:17:18.515 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum clean all 2024-04-02T13:17:18.750 INFO:teuthology.orchestra.run.smithi069.stdout:74 files removed 2024-04-02T13:17:18.777 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-02T13:17:18.780 INFO:teuthology.orchestra.run.smithi044.stdout:74 files removed 2024-04-02T13:17:18.807 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-02T13:17:19.670 INFO:teuthology.orchestra.run.smithi069.stdout:ceph packages for x86_64 140 kB/s | 83 kB 00:00 2024-04-02T13:17:19.740 INFO:teuthology.orchestra.run.smithi044.stdout:ceph packages for x86_64 133 kB/s | 83 kB 00:00 2024-04-02T13:17:20.174 INFO:teuthology.orchestra.run.smithi044.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-04-02T13:17:20.179 INFO:teuthology.orchestra.run.smithi069.stdout:ceph noarch packages 25 kB/s | 12 kB 00:00 2024-04-02T13:17:20.558 INFO:teuthology.orchestra.run.smithi044.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-04-02T13:17:20.663 INFO:teuthology.orchestra.run.smithi069.stdout:ceph source packages 4.3 kB/s | 1.9 kB 00:00 2024-04-02T13:17:21.144 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-02T13:17:21.640 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 7.1 MB/s | 6.7 MB 00:00 2024-04-02T13:17:22.686 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-04-02T13:17:23.140 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-04-02T13:17:23.894 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.0 MB 00:00 2024-04-02T13:17:25.214 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-04-02T13:17:29.598 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 5.7 MB/s | 19 MB 00:03 2024-04-02T13:17:33.582 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-04-02T13:17:35.450 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-02T13:17:37.462 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-02T13:17:40.860 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-04-02T13:17:42.468 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-02T13:17:44.994 INFO:teuthology.orchestra.run.smithi069.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-02T13:17:44.995 INFO:teuthology.orchestra.run.smithi069.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-02T13:17:45.009 INFO:teuthology.orchestra.run.smithi069.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-02T13:17:45.010 INFO:teuthology.orchestra.run.smithi069.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-02T13:17:45.011 INFO:teuthology.orchestra.run.smithi069.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-02T13:17:45.012 INFO:teuthology.orchestra.run.smithi069.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-02T13:17:45.014 INFO:teuthology.orchestra.run.smithi069.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-02T13:17:45.017 INFO:teuthology.orchestra.run.smithi069.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-02T13:17:45.018 INFO:teuthology.orchestra.run.smithi069.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-02T13:17:45.097 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-02T13:17:45.123 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-04-02T13:17:45.123 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-02T13:17:45.123 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-04-02T13:17:45.123 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: ceph x86_64 2:19.0.0-2609.g191d963a.el9 ceph 6.6 k 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base x86_64 2:19.0.0-2609.g191d963a.el9 ceph 5.4 M 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 890 k 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2609.g191d963a.el9 ceph 145 k 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr x86_64 2:19.0.0-2609.g191d963a.el9 ceph 1.5 M 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 141 k 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 3.5 M 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 7.4 M 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 49 k 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw x86_64 2:19.0.0-2609.g191d963a.el9 ceph 9.8 M 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test x86_64 2:19.0.0-2609.g191d963a.el9 ceph 48 M 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 264 k 2024-04-02T13:17:45.124 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 762 k 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-mirror x86_64 2:19.0.0-2609.g191d963a.el9 ceph 225 k 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-shell x86_64 2:19.0.0-2609.g191d963a.el9 ceph 27 k 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-top noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 18 k 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel x86_64 2:19.0.0-2609.g191d963a.el9 ceph 32 k 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 747 k 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel x86_64 2:19.0.0-2609.g191d963a.el9 ceph 127 k 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-02T13:17:45.125 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs x86_64 2:19.0.0-2609.g191d963a.el9 ceph 161 k 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados x86_64 2:19.0.0-2609.g191d963a.el9 ceph 320 k 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 299 k 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw x86_64 2:19.0.0-2609.g191d963a.el9 ceph 99 k 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 85 k 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.1 M 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 170 k 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrading: 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-04-02T13:17:45.126 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.4 M 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.1 M 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-02T13:17:45.127 INFO:teuthology.orchestra.run.smithi069.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common x86_64 2:19.0.0-2609.g191d963a.el9 ceph 21 M 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 34 k 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds x86_64 2:19.0.0-2609.g191d963a.el9 ceph 2.4 M 2024-04-02T13:17:45.128 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 249 k 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon x86_64 2:19.0.0-2609.g191d963a.el9 ceph 4.7 M 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 17 M 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 16 k 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux x86_64 2:19.0.0-2609.g191d963a.el9 ceph 25 k 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite x86_64 2:19.0.0-2609.g191d963a.el9 ceph 164 k 2024-04-02T13:17:45.129 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 500 k 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 5.1 M 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-02T13:17:45.130 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 45 k 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common x86_64 2:19.0.0-2609.g191d963a.el9 ceph 134 k 2024-04-02T13:17:45.131 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-02T13:17:45.132 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-02T13:17:45.132 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-02T13:17:45.132 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-02T13:17:45.132 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-02T13:17:45.132 INFO:teuthology.orchestra.run.smithi069.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-02T13:17:45.132 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-02T13:17:45.132 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-02T13:17:45.132 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-02T13:17:45.132 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-02T13:17:45.132 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-02T13:17:45.132 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-02T13:17:45.132 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-02T13:17:45.132 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-02T13:17:45.133 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-02T13:17:45.134 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-02T13:17:45.135 INFO:teuthology.orchestra.run.smithi069.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-02T13:17:45.136 INFO:teuthology.orchestra.run.smithi069.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-02T13:17:45.136 INFO:teuthology.orchestra.run.smithi069.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-02T13:17:45.136 INFO:teuthology.orchestra.run.smithi069.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-02T13:17:45.136 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-02T13:17:45.136 INFO:teuthology.orchestra.run.smithi069.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-02T13:17:45.136 INFO:teuthology.orchestra.run.smithi069.stdout:Installing weak dependencies: 2024-04-02T13:17:45.136 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-02T13:17:45.136 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:17:45.136 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-02T13:17:45.136 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-04-02T13:17:45.136 INFO:teuthology.orchestra.run.smithi069.stdout:Install 150 Packages 2024-04-02T13:17:45.136 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrade 24 Packages 2024-04-02T13:17:45.136 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:17:45.137 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 226 M 2024-04-02T13:17:45.137 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-02T13:17:46.127 INFO:teuthology.orchestra.run.smithi069.stdout:(1/174): ceph-19.0.0-2609.g191d963a.el9.x86_64. 27 kB/s | 6.6 kB 00:00 2024-04-02T13:17:46.536 INFO:teuthology.orchestra.run.smithi069.stdout:(2/174): ceph-base-19.0.0-2609.g191d963a.el9.x8 8.2 MB/s | 5.4 MB 00:00 2024-04-02T13:17:46.603 INFO:teuthology.orchestra.run.smithi069.stdout:(3/174): ceph-immutable-object-cache-19.0.0-260 2.1 MB/s | 145 kB 00:00 2024-04-02T13:17:46.853 INFO:teuthology.orchestra.run.smithi069.stdout:(4/174): ceph-mds-19.0.0-2609.g191d963a.el9.x86 9.5 MB/s | 2.4 MB 00:00 2024-04-02T13:17:46.987 INFO:teuthology.orchestra.run.smithi069.stdout:(5/174): ceph-fuse-19.0.0-2609.g191d963a.el9.x8 1.0 MB/s | 890 kB 00:00 2024-04-02T13:17:47.028 INFO:teuthology.orchestra.run.smithi069.stdout:(6/174): ceph-mgr-19.0.0-2609.g191d963a.el9.x86 8.5 MB/s | 1.5 MB 00:00 2024-04-02T13:17:47.604 INFO:teuthology.orchestra.run.smithi069.stdout:(7/174): ceph-common-19.0.0-2609.g191d963a.el9. 12 MB/s | 21 MB 00:01 2024-04-02T13:17:48.287 INFO:teuthology.orchestra.run.smithi069.stdout:(8/174): ceph-osd-19.0.0-2609.g191d963a.el9.x86 13 MB/s | 17 MB 00:01 2024-04-02T13:17:48.396 INFO:teuthology.orchestra.run.smithi069.stdout:(9/174): ceph-radosgw-19.0.0-2609.g191d963a.el9 12 MB/s | 9.8 MB 00:00 2024-04-02T13:17:48.421 INFO:teuthology.orchestra.run.smithi069.stdout:(10/174): ceph-selinux-19.0.0-2609.g191d963a.el 189 kB/s | 25 kB 00:00 2024-04-02T13:17:48.496 INFO:teuthology.orchestra.run.smithi069.stdout:(11/174): cephfs-mirror-19.0.0-2609.g191d963a.e 2.9 MB/s | 225 kB 00:00 2024-04-02T13:17:48.555 INFO:teuthology.orchestra.run.smithi069.stdout:(12/174): cephfs-shell-19.0.0-2609.g191d963a.el 465 kB/s | 27 kB 00:00 2024-04-02T13:17:48.622 INFO:teuthology.orchestra.run.smithi069.stdout:(13/174): libcephfs-devel-19.0.0-2609.g191d963a 475 kB/s | 32 kB 00:00 2024-04-02T13:17:48.722 INFO:teuthology.orchestra.run.smithi069.stdout:(14/174): libcephfs2-19.0.0-2609.g191d963a.el9. 7.3 MB/s | 747 kB 00:00 2024-04-02T13:17:48.797 INFO:teuthology.orchestra.run.smithi069.stdout:(15/174): libcephsqlite-19.0.0-2609.g191d963a.e 2.1 MB/s | 164 kB 00:00 2024-04-02T13:17:48.873 INFO:teuthology.orchestra.run.smithi069.stdout:(16/174): ceph-mon-19.0.0-2609.g191d963a.el9.x8 2.5 MB/s | 4.7 MB 00:01 2024-04-02T13:17:48.898 INFO:teuthology.orchestra.run.smithi069.stdout:(17/174): librados-devel-19.0.0-2609.g191d963a. 1.2 MB/s | 127 kB 00:00 2024-04-02T13:17:49.065 INFO:teuthology.orchestra.run.smithi069.stdout:(18/174): libradosstriper1-19.0.0-2609.g191d963 2.5 MB/s | 500 kB 00:00 2024-04-02T13:17:49.157 INFO:teuthology.orchestra.run.smithi069.stdout:(19/174): python3-ceph-argparse-19.0.0-2609.g19 492 kB/s | 45 kB 00:00 2024-04-02T13:17:49.257 INFO:teuthology.orchestra.run.smithi069.stdout:(20/174): python3-ceph-common-19.0.0-2609.g191d 1.3 MB/s | 134 kB 00:00 2024-04-02T13:17:49.374 INFO:teuthology.orchestra.run.smithi069.stdout:(21/174): librgw2-19.0.0-2609.g191d963a.el9.x86 11 MB/s | 5.1 MB 00:00 2024-04-02T13:17:49.399 INFO:teuthology.orchestra.run.smithi069.stdout:(22/174): python3-cephfs-19.0.0-2609.g191d963a. 1.1 MB/s | 161 kB 00:00 2024-04-02T13:17:49.441 INFO:teuthology.orchestra.run.smithi069.stdout:(23/174): python3-rados-19.0.0-2609.g191d963a.e 4.7 MB/s | 320 kB 00:00 2024-04-02T13:17:49.485 INFO:teuthology.orchestra.run.smithi069.stdout:(24/174): python3-rbd-19.0.0-2609.g191d963a.el9 3.4 MB/s | 299 kB 00:00 2024-04-02T13:17:49.510 INFO:teuthology.orchestra.run.smithi069.stdout:(25/174): python3-rgw-19.0.0-2609.g191d963a.el9 1.4 MB/s | 99 kB 00:00 2024-04-02T13:17:49.561 INFO:teuthology.orchestra.run.smithi069.stdout:(26/174): rbd-fuse-19.0.0-2609.g191d963a.el9.x8 1.1 MB/s | 85 kB 00:00 2024-04-02T13:17:49.661 INFO:teuthology.orchestra.run.smithi069.stdout:(27/174): rbd-nbd-19.0.0-2609.g191d963a.el9.x86 1.7 MB/s | 170 kB 00:00 2024-04-02T13:17:49.761 INFO:teuthology.orchestra.run.smithi069.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2609.g 340 kB/s | 34 kB 00:00 2024-04-02T13:17:49.820 INFO:teuthology.orchestra.run.smithi069.stdout:(29/174): rbd-mirror-19.0.0-2609.g191d963a.el9. 10 MB/s | 3.1 MB 00:00 2024-04-02T13:17:49.837 INFO:teuthology.orchestra.run.smithi069.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2609.g191d963 1.8 MB/s | 141 kB 00:00 2024-04-02T13:17:50.195 INFO:teuthology.orchestra.run.smithi069.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2609.g191d9 9.4 MB/s | 3.5 MB 00:00 2024-04-02T13:17:50.279 INFO:teuthology.orchestra.run.smithi069.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2609.g19 2.9 MB/s | 249 kB 00:00 2024-04-02T13:17:50.346 INFO:teuthology.orchestra.run.smithi069.stdout:(33/174): ceph-mgr-rook-19.0.0-2609.g191d963a.e 742 kB/s | 49 kB 00:00 2024-04-02T13:17:50.413 INFO:teuthology.orchestra.run.smithi069.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2609.g1 246 kB/s | 16 kB 00:00 2024-04-02T13:17:50.513 INFO:teuthology.orchestra.run.smithi069.stdout:(35/174): ceph-volume-19.0.0-2609.g191d963a.el9 2.6 MB/s | 264 kB 00:00 2024-04-02T13:17:50.613 INFO:teuthology.orchestra.run.smithi069.stdout:(36/174): cephadm-19.0.0-2609.g191d963a.el9.noa 7.5 MB/s | 762 kB 00:00 2024-04-02T13:17:50.672 INFO:teuthology.orchestra.run.smithi069.stdout:(37/174): cephfs-top-19.0.0-2609.g191d963a.el9. 313 kB/s | 18 kB 00:00 2024-04-02T13:17:50.814 INFO:teuthology.orchestra.run.smithi069.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-04-02T13:17:50.898 INFO:teuthology.orchestra.run.smithi069.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 7.0 MB/s | 7.4 MB 00:01 2024-04-02T13:17:50.931 INFO:teuthology.orchestra.run.smithi069.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 7.7 MB/s | 920 kB 00:00 2024-04-02T13:17:50.956 INFO:teuthology.orchestra.run.smithi069.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-04-02T13:17:50.981 INFO:teuthology.orchestra.run.smithi069.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-04-02T13:17:51.006 INFO:teuthology.orchestra.run.smithi069.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 635 kB/s | 32 kB 00:00 2024-04-02T13:17:51.074 INFO:teuthology.orchestra.run.smithi069.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-04-02T13:17:51.099 INFO:teuthology.orchestra.run.smithi069.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 191 kB/s | 17 kB 00:00 2024-04-02T13:17:51.124 INFO:teuthology.orchestra.run.smithi069.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-04-02T13:17:51.149 INFO:teuthology.orchestra.run.smithi069.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-04-02T13:17:51.175 INFO:teuthology.orchestra.run.smithi069.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-04-02T13:17:51.209 INFO:teuthology.orchestra.run.smithi069.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-04-02T13:17:51.234 INFO:teuthology.orchestra.run.smithi069.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-04-02T13:17:51.259 INFO:teuthology.orchestra.run.smithi069.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 940 kB/s | 46 kB 00:00 2024-04-02T13:17:51.284 INFO:teuthology.orchestra.run.smithi069.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-04-02T13:17:51.310 INFO:teuthology.orchestra.run.smithi069.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-04-02T13:17:51.335 INFO:teuthology.orchestra.run.smithi069.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-04-02T13:17:51.360 INFO:teuthology.orchestra.run.smithi069.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2024-04-02T13:17:51.786 INFO:teuthology.orchestra.run.smithi069.stdout:(56/174): ceph-test-19.0.0-2609.g191d963a.el9.x 14 MB/s | 48 MB 00:03 2024-04-02T13:17:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 9.2 MB/s | 4.7 MB 00:00 2024-04-02T13:17:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:(58/174): postgresql-13.14-1.el9.x86_64.rpm 2.9 MB/s | 1.5 MB 00:00 2024-04-02T13:17:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:(59/174): postgresql-contrib-13.14-1.el9.x86_64 6.5 MB/s | 817 kB 00:00 2024-04-02T13:17:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:(60/174): postgresql-private-libs-13.14-1.el9.x 1.8 MB/s | 132 kB 00:00 2024-04-02T13:17:51.978 INFO:teuthology.orchestra.run.smithi069.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-04-02T13:17:52.087 INFO:teuthology.orchestra.run.smithi069.stdout:(62/174): postgresql-server-13.14-1.el9.x86_64. 33 MB/s | 5.7 MB 00:00 2024-04-02T13:17:52.112 INFO:teuthology.orchestra.run.smithi069.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 423 kB/s | 78 kB 00:00 2024-04-02T13:17:52.212 INFO:teuthology.orchestra.run.smithi069.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 5.8 MB 00:00 2024-04-02T13:17:52.238 INFO:teuthology.orchestra.run.smithi069.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 147 kB/s | 18 kB 00:00 2024-04-02T13:17:52.263 INFO:teuthology.orchestra.run.smithi069.stdout:(66/174): python3-jsonpatch-1.21-16.el9.noarch. 165 kB/s | 24 kB 00:00 2024-04-02T13:17:52.288 INFO:teuthology.orchestra.run.smithi069.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-04-02T13:17:52.313 INFO:teuthology.orchestra.run.smithi069.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 421 kB/s | 32 kB 00:00 2024-04-02T13:17:52.338 INFO:teuthology.orchestra.run.smithi069.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.3 MB/s | 217 kB 00:00 2024-04-02T13:17:52.414 INFO:teuthology.orchestra.run.smithi069.stdout:(70/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-04-02T13:17:52.439 INFO:teuthology.orchestra.run.smithi069.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2024-04-02T13:17:52.464 INFO:teuthology.orchestra.run.smithi069.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 558 kB/s | 69 kB 00:00 2024-04-02T13:17:52.482 INFO:teuthology.orchestra.run.smithi069.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 620 kB/s | 41 kB 00:00 2024-04-02T13:17:52.506 INFO:teuthology.orchestra.run.smithi069.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-04-02T13:17:52.523 INFO:teuthology.orchestra.run.smithi069.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-04-02T13:17:52.565 INFO:teuthology.orchestra.run.smithi069.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-04-02T13:17:52.582 INFO:teuthology.orchestra.run.smithi069.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 811 kB/s | 47 kB 00:00 2024-04-02T13:17:52.585 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 777 kB/s | 6.1 MB 00:08 2024-04-02T13:17:52.599 INFO:teuthology.orchestra.run.smithi069.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 571 kB/s | 43 kB 00:00 2024-04-02T13:17:52.733 INFO:teuthology.orchestra.run.smithi069.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 296 kB/s | 39 kB 00:00 2024-04-02T13:17:52.883 INFO:teuthology.orchestra.run.smithi069.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-04-02T13:17:52.899 INFO:teuthology.orchestra.run.smithi069.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 138 kB/s | 41 kB 00:00 2024-04-02T13:17:52.925 INFO:teuthology.orchestra.run.smithi069.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.1 MB/s | 220 kB 00:00 2024-04-02T13:17:52.950 INFO:teuthology.orchestra.run.smithi069.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 699 kB/s | 35 kB 00:00 2024-04-02T13:17:52.975 INFO:teuthology.orchestra.run.smithi069.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 618 kB/s | 46 kB 00:00 2024-04-02T13:17:53.017 INFO:teuthology.orchestra.run.smithi069.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.2 MB/s | 300 kB 00:00 2024-04-02T13:17:53.043 INFO:teuthology.orchestra.run.smithi069.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 850 kB/s | 56 kB 00:00 2024-04-02T13:17:53.067 INFO:teuthology.orchestra.run.smithi069.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 686 kB/s | 63 kB 00:00 2024-04-02T13:17:53.093 INFO:teuthology.orchestra.run.smithi069.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-04-02T13:17:53.118 INFO:teuthology.orchestra.run.smithi069.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-04-02T13:17:53.143 INFO:teuthology.orchestra.run.smithi069.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-04-02T13:17:53.177 INFO:teuthology.orchestra.run.smithi069.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-04-02T13:17:53.194 INFO:teuthology.orchestra.run.smithi069.stdout:(92/174): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-04-02T13:17:53.219 INFO:teuthology.orchestra.run.smithi069.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.7 MB/s | 124 kB 00:00 2024-04-02T13:17:53.244 INFO:teuthology.orchestra.run.smithi069.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.7 MB/s | 187 kB 00:00 2024-04-02T13:17:53.745 INFO:teuthology.orchestra.run.smithi069.stdout:(95/174): libquadmath-11.4.1-3.el9.x86_64.rpm 391 kB/s | 195 kB 00:00 2024-04-02T13:17:53.770 INFO:teuthology.orchestra.run.smithi069.stdout:(96/174): jq-1.6-16.el9.x86_64.rpm 340 kB/s | 187 kB 00:00 2024-04-02T13:17:53.854 INFO:teuthology.orchestra.run.smithi069.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-04-02T13:17:53.879 INFO:teuthology.orchestra.run.smithi069.stdout:(98/174): elfutils-libelf-devel-0.190-2.el9.x86 198 kB/s | 21 kB 00:00 2024-04-02T13:17:53.904 INFO:teuthology.orchestra.run.smithi069.stdout:(99/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-04-02T13:17:53.954 INFO:teuthology.orchestra.run.smithi069.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 253 kB/s | 25 kB 00:00 2024-04-02T13:17:54.013 INFO:teuthology.orchestra.run.smithi069.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 2.2 MB/s | 245 kB 00:00 2024-04-02T13:17:54.063 INFO:teuthology.orchestra.run.smithi069.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 2.2 MB/s | 248 kB 00:00 2024-04-02T13:17:54.113 INFO:teuthology.orchestra.run.smithi069.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 447 kB/s | 22 kB 00:00 2024-04-02T13:17:54.130 INFO:teuthology.orchestra.run.smithi069.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 801 kB/s | 94 kB 00:00 2024-04-02T13:17:54.236 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-02T13:17:54.272 INFO:teuthology.orchestra.run.smithi069.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 5.3 MB/s | 863 kB 00:00 2024-04-02T13:17:54.923 INFO:teuthology.orchestra.run.smithi069.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 692 kB/s | 548 kB 00:00 2024-04-02T13:17:54.965 INFO:teuthology.orchestra.run.smithi069.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 464 kB/s | 309 kB 00:00 2024-04-02T13:17:55.031 INFO:teuthology.orchestra.run.smithi069.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-04-02T13:17:55.142 INFO:teuthology.orchestra.run.smithi069.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-02T13:17:55.254 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-04-02T13:17:55.257 INFO:teuthology.orchestra.run.smithi069.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 417 kB/s | 49 kB 00:00 2024-04-02T13:17:55.365 INFO:teuthology.orchestra.run.smithi069.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 623 kB/s | 67 kB 00:00 2024-04-02T13:17:55.457 INFO:teuthology.orchestra.run.smithi069.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.0 MB/s | 4.4 MB 00:00 2024-04-02T13:17:55.483 INFO:teuthology.orchestra.run.smithi069.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-04-02T13:17:55.575 INFO:teuthology.orchestra.run.smithi069.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-04-02T13:17:55.650 INFO:teuthology.orchestra.run.smithi069.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-04-02T13:17:55.683 INFO:teuthology.orchestra.run.smithi069.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-04-02T13:17:55.767 INFO:teuthology.orchestra.run.smithi069.stdout:(117/174): openssl-devel-3.0.7-27.el9.x86_64.rp 2.2 MB/s | 4.1 MB 00:01 2024-04-02T13:17:55.792 INFO:teuthology.orchestra.run.smithi069.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-04-02T13:17:55.818 INFO:teuthology.orchestra.run.smithi069.stdout:(119/174): python3-cachetools-4.2.4-1.el9.noarc 192 kB/s | 32 kB 00:00 2024-04-02T13:17:55.885 INFO:teuthology.orchestra.run.smithi069.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-04-02T13:17:55.952 INFO:teuthology.orchestra.run.smithi069.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-04-02T13:17:56.010 INFO:teuthology.orchestra.run.smithi069.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 359 kB/s | 35 kB 00:00 2024-04-02T13:17:56.094 INFO:teuthology.orchestra.run.smithi069.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.4 MB/s | 203 kB 00:00 2024-04-02T13:17:56.169 INFO:teuthology.orchestra.run.smithi069.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-04-02T13:17:56.211 INFO:teuthology.orchestra.run.smithi069.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-04-02T13:17:56.278 INFO:teuthology.orchestra.run.smithi069.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-02T13:17:56.320 INFO:teuthology.orchestra.run.smithi069.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-04-02T13:17:56.387 INFO:teuthology.orchestra.run.smithi069.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-04-02T13:17:56.429 INFO:teuthology.orchestra.run.smithi069.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-04-02T13:17:56.496 INFO:teuthology.orchestra.run.smithi069.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-04-02T13:17:56.529 INFO:teuthology.orchestra.run.smithi069.stdout:(131/174): python3-cherrypy-18.6.1-2.el9.noarch 487 kB/s | 358 kB 00:00 2024-04-02T13:17:56.605 INFO:teuthology.orchestra.run.smithi069.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-04-02T13:17:56.646 INFO:teuthology.orchestra.run.smithi069.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 4.7 MB/s | 1.0 MB 00:00 2024-04-02T13:17:56.672 INFO:teuthology.orchestra.run.smithi069.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 555 kB/s | 79 kB 00:00 2024-04-02T13:17:56.714 INFO:teuthology.orchestra.run.smithi069.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 530 kB/s | 58 kB 00:00 2024-04-02T13:17:56.764 INFO:teuthology.orchestra.run.smithi069.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-02T13:17:56.789 INFO:teuthology.orchestra.run.smithi069.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 140 kB/s | 16 kB 00:00 2024-04-02T13:17:56.814 INFO:teuthology.orchestra.run.smithi069.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 895 kB/s | 90 kB 00:00 2024-04-02T13:17:56.873 INFO:teuthology.orchestra.run.smithi069.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 259 kB/s | 28 kB 00:00 2024-04-02T13:17:56.898 INFO:teuthology.orchestra.run.smithi069.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 282 kB/s | 31 kB 00:00 2024-04-02T13:17:56.924 INFO:teuthology.orchestra.run.smithi069.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-02T13:17:56.982 INFO:teuthology.orchestra.run.smithi069.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 544 kB/s | 59 kB 00:00 2024-04-02T13:17:57.007 INFO:teuthology.orchestra.run.smithi069.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 329 kB/s | 36 kB 00:00 2024-04-02T13:17:57.033 INFO:teuthology.orchestra.run.smithi069.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 468 kB/s | 51 kB 00:00 2024-04-02T13:17:57.091 INFO:teuthology.orchestra.run.smithi069.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-02T13:17:57.125 INFO:teuthology.orchestra.run.smithi069.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 766 kB/s | 90 kB 00:00 2024-04-02T13:17:57.150 INFO:teuthology.orchestra.run.smithi069.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-02T13:17:57.200 INFO:teuthology.orchestra.run.smithi069.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 183 kB/s | 20 kB 00:00 2024-04-02T13:17:57.267 INFO:teuthology.orchestra.run.smithi069.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-04-02T13:17:57.309 INFO:teuthology.orchestra.run.smithi069.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-02T13:17:57.384 INFO:teuthology.orchestra.run.smithi069.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 31 MB/s | 2.3 MB 00:00 2024-04-02T13:17:57.460 INFO:teuthology.orchestra.run.smithi069.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-04-02T13:17:57.494 INFO:teuthology.orchestra.run.smithi069.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-04-02T13:17:57.536 INFO:teuthology.orchestra.run.smithi069.stdout:(154/174): NetworkManager-tui-1.47.2-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-04-02T13:17:57.577 INFO:teuthology.orchestra.run.smithi069.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-04-02T13:17:57.678 INFO:teuthology.orchestra.run.smithi069.stdout:(156/174): librbd1-19.0.0-2609.g191d963a.el9.x8 7.7 MB/s | 3.1 MB 00:00 2024-04-02T13:17:57.761 INFO:teuthology.orchestra.run.smithi069.stdout:(157/174): librados2-19.0.0-2609.g191d963a.el9. 6.1 MB/s | 3.4 MB 00:00 2024-04-02T13:17:57.845 INFO:teuthology.orchestra.run.smithi069.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 267 kB/s | 71 kB 00:00 2024-04-02T13:17:57.870 INFO:teuthology.orchestra.run.smithi069.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 190 kB/s | 36 kB 00:00 2024-04-02T13:17:57.979 INFO:teuthology.orchestra.run.smithi069.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 212 kB/s | 23 kB 00:00 2024-04-02T13:17:58.037 INFO:teuthology.orchestra.run.smithi069.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.3 MB/s | 256 kB 00:00 2024-04-02T13:17:58.129 INFO:teuthology.orchestra.run.smithi069.stdout:(162/174): elfutils-libelf-0.190-2.el9.x86_64.r 523 kB/s | 192 kB 00:00 2024-04-02T13:17:58.154 INFO:teuthology.orchestra.run.smithi069.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 155 kB 00:00 2024-04-02T13:17:58.205 INFO:teuthology.orchestra.run.smithi069.stdout:(164/174): libblkid-2.37.4-18.el9.x86_64.rpm 480 kB/s | 108 kB 00:00 2024-04-02T13:17:58.255 INFO:teuthology.orchestra.run.smithi069.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-04-02T13:17:58.280 INFO:teuthology.orchestra.run.smithi069.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 509 kB/s | 64 kB 00:00 2024-04-02T13:17:58.322 INFO:teuthology.orchestra.run.smithi069.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 245 kB/s | 29 kB 00:00 2024-04-02T13:17:58.564 INFO:teuthology.orchestra.run.smithi069.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 3.9 MB/s | 1.2 MB 00:00 2024-04-02T13:17:58.689 INFO:teuthology.orchestra.run.smithi069.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.6 MB/s | 465 kB 00:00 2024-04-02T13:17:58.798 INFO:teuthology.orchestra.run.smithi069.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 288 kB/s | 31 kB 00:00 2024-04-02T13:17:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.4 MB/s | 2.3 MB 00:00 2024-04-02T13:17:58.923 INFO:teuthology.orchestra.run.smithi069.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-04-02T13:17:58.957 INFO:teuthology.orchestra.run.smithi069.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 636 kB/s | 74 kB 00:00 2024-04-02T13:17:59.241 INFO:teuthology.orchestra.run.smithi069.stdout:(174/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 2.2 MB/s | 2.2 MB 00:00 2024-04-02T13:17:59.246 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-02T13:17:59.246 INFO:teuthology.orchestra.run.smithi069.stdout:Total 16 MB/s | 226 MB 00:14 2024-04-02T13:17:59.612 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-02T13:17:59.885 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-02T13:17:59.885 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-02T13:18:00.447 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-02T13:18:01.926 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-02T13:18:01.926 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-02T13:18:03.159 INFO:teuthology.orchestra.run.smithi044.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-02T13:18:03.160 INFO:teuthology.orchestra.run.smithi044.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-02T13:18:03.175 INFO:teuthology.orchestra.run.smithi044.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-02T13:18:03.176 INFO:teuthology.orchestra.run.smithi044.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-02T13:18:03.177 INFO:teuthology.orchestra.run.smithi044.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-02T13:18:03.178 INFO:teuthology.orchestra.run.smithi044.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-02T13:18:03.180 INFO:teuthology.orchestra.run.smithi044.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-02T13:18:03.183 INFO:teuthology.orchestra.run.smithi044.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-02T13:18:03.184 INFO:teuthology.orchestra.run.smithi044.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-02T13:18:03.272 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-02T13:18:03.299 INFO:teuthology.orchestra.run.smithi044.stdout:============================================================================================ 2024-04-02T13:18:03.299 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-04-02T13:18:03.299 INFO:teuthology.orchestra.run.smithi044.stdout:============================================================================================ 2024-04-02T13:18:03.300 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-02T13:18:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-02T13:18:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: ceph x86_64 2:19.0.0-2609.g191d963a.el9 ceph 6.6 k 2024-04-02T13:18:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base x86_64 2:19.0.0-2609.g191d963a.el9 ceph 5.4 M 2024-04-02T13:18:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 890 k 2024-04-02T13:18:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2609.g191d963a.el9 ceph 145 k 2024-04-02T13:18:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr x86_64 2:19.0.0-2609.g191d963a.el9 ceph 1.5 M 2024-04-02T13:18:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 141 k 2024-04-02T13:18:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 3.5 M 2024-04-02T13:18:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 7.4 M 2024-04-02T13:18:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 49 k 2024-04-02T13:18:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw x86_64 2:19.0.0-2609.g191d963a.el9 ceph 9.8 M 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test x86_64 2:19.0.0-2609.g191d963a.el9 ceph 48 M 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 264 k 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 762 k 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-mirror x86_64 2:19.0.0-2609.g191d963a.el9 ceph 225 k 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-shell x86_64 2:19.0.0-2609.g191d963a.el9 ceph 27 k 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-top noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 18 k 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel x86_64 2:19.0.0-2609.g191d963a.el9 ceph 32 k 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 747 k 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel x86_64 2:19.0.0-2609.g191d963a.el9 ceph 127 k 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-02T13:18:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-02T13:18:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-02T13:18:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-02T13:18:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-02T13:18:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-02T13:18:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs x86_64 2:19.0.0-2609.g191d963a.el9 ceph 161 k 2024-04-02T13:18:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados x86_64 2:19.0.0-2609.g191d963a.el9 ceph 320 k 2024-04-02T13:18:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 299 k 2024-04-02T13:18:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw x86_64 2:19.0.0-2609.g191d963a.el9 ceph 99 k 2024-04-02T13:18:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 85 k 2024-04-02T13:18:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.1 M 2024-04-02T13:18:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 170 k 2024-04-02T13:18:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-02T13:18:03.303 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrading: 2024-04-02T13:18:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-04-02T13:18:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-04-02T13:18:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-04-02T13:18:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-04-02T13:18:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-02T13:18:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-02T13:18:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-02T13:18:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-02T13:18:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-02T13:18:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-02T13:18:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-02T13:18:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: librados2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.4 M 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.1 M 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-04-02T13:18:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common x86_64 2:19.0.0-2609.g191d963a.el9 ceph 21 M 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 34 k 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds x86_64 2:19.0.0-2609.g191d963a.el9 ceph 2.4 M 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 249 k 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon x86_64 2:19.0.0-2609.g191d963a.el9 ceph 4.7 M 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 17 M 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 16 k 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux x86_64 2:19.0.0-2609.g191d963a.el9 ceph 25 k 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-02T13:18:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-02T13:18:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-02T13:18:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-02T13:18:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-02T13:18:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite x86_64 2:19.0.0-2609.g191d963a.el9 ceph 164 k 2024-04-02T13:18:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-02T13:18:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-02T13:18:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-02T13:18:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-02T13:18:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-02T13:18:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 500 k 2024-04-02T13:18:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-02T13:18:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 5.1 M 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-02T13:18:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 45 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common x86_64 2:19.0.0-2609.g191d963a.el9 ceph 134 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-02T13:18:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-02T13:18:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-02T13:18:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-02T13:18:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-02T13:18:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-02T13:18:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-02T13:18:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-02T13:18:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-02T13:18:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-02T13:18:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-02T13:18:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-02T13:18:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-02T13:18:03.312 INFO:teuthology.orchestra.run.smithi044.stdout:Installing weak dependencies: 2024-04-02T13:18:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-02T13:18:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:18:03.312 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-02T13:18:03.312 INFO:teuthology.orchestra.run.smithi044.stdout:============================================================================================ 2024-04-02T13:18:03.313 INFO:teuthology.orchestra.run.smithi044.stdout:Install 150 Packages 2024-04-02T13:18:03.313 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrade 24 Packages 2024-04-02T13:18:03.313 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:18:03.313 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 226 M 2024-04-02T13:18:03.313 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-02T13:18:03.823 INFO:teuthology.orchestra.run.smithi044.stdout:(1/174): ceph-19.0.0-2609.g191d963a.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-04-02T13:18:04.073 INFO:teuthology.orchestra.run.smithi044.stdout:(2/174): ceph-fuse-19.0.0-2609.g191d963a.el9.x8 3.5 MB/s | 890 kB 00:00 2024-04-02T13:18:04.140 INFO:teuthology.orchestra.run.smithi044.stdout:(3/174): ceph-immutable-object-cache-19.0.0-260 2.1 MB/s | 145 kB 00:00 2024-04-02T13:18:04.465 INFO:teuthology.orchestra.run.smithi044.stdout:(4/174): ceph-mds-19.0.0-2609.g191d963a.el9.x86 7.3 MB/s | 2.4 MB 00:00 2024-04-02T13:18:04.724 INFO:teuthology.orchestra.run.smithi044.stdout:(5/174): ceph-mgr-19.0.0-2609.g191d963a.el9.x86 5.8 MB/s | 1.5 MB 00:00 2024-04-02T13:18:05.208 INFO:teuthology.orchestra.run.smithi044.stdout:(6/174): ceph-common-19.0.0-2609.g191d963a.el9. 13 MB/s | 21 MB 00:01 2024-04-02T13:18:05.300 INFO:teuthology.orchestra.run.smithi044.stdout:(7/174): ceph-mon-19.0.0-2609.g191d963a.el9.x86 8.2 MB/s | 4.7 MB 00:00 2024-04-02T13:18:05.366 INFO:teuthology.orchestra.run.smithi044.stdout:(8/174): ceph-base-19.0.0-2609.g191d963a.el9.x8 3.1 MB/s | 5.4 MB 00:01 2024-04-02T13:18:05.450 INFO:teuthology.orchestra.run.smithi044.stdout:(9/174): ceph-selinux-19.0.0-2609.g191d963a.el9 302 kB/s | 25 kB 00:00 2024-04-02T13:18:06.358 INFO:teuthology.orchestra.run.smithi044.stdout:(10/174): ceph-osd-19.0.0-2609.g191d963a.el9.x8 15 MB/s | 17 MB 00:01 2024-04-02T13:18:06.434 INFO:teuthology.orchestra.run.smithi044.stdout:(11/174): cephfs-mirror-19.0.0-2609.g191d963a.e 2.9 MB/s | 225 kB 00:00 2024-04-02T13:18:06.542 INFO:teuthology.orchestra.run.smithi044.stdout:(12/174): ceph-radosgw-19.0.0-2609.g191d963a.el 7.9 MB/s | 9.8 MB 00:01 2024-04-02T13:18:06.559 INFO:teuthology.orchestra.run.smithi044.stdout:(13/174): cephfs-shell-19.0.0-2609.g191d963a.el 216 kB/s | 27 kB 00:00 2024-04-02T13:18:06.576 INFO:teuthology.orchestra.run.smithi044.stdout:(14/174): libcephfs-devel-19.0.0-2609.g191d963a 950 kB/s | 32 kB 00:00 2024-04-02T13:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:(15/174): libcephfs2-19.0.0-2609.g191d963a.el9. 9.7 MB/s | 747 kB 00:00 2024-04-02T13:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:(16/174): libcephsqlite-19.0.0-2609.g191d963a.e 1.9 MB/s | 164 kB 00:00 2024-04-02T13:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:(17/174): librados-devel-19.0.0-2609.g191d963a. 3.0 MB/s | 127 kB 00:00 2024-04-02T13:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:(18/174): libradosstriper1-19.0.0-2609.g191d963 9.7 MB/s | 500 kB 00:00 2024-04-02T13:18:06.777 INFO:teuthology.orchestra.run.smithi044.stdout:(19/174): python3-ceph-argparse-19.0.0-2609.g19 679 kB/s | 45 kB 00:00 2024-04-02T13:18:06.853 INFO:teuthology.orchestra.run.smithi044.stdout:(20/174): python3-ceph-common-19.0.0-2609.g191d 1.7 MB/s | 134 kB 00:00 2024-04-02T13:18:06.937 INFO:teuthology.orchestra.run.smithi044.stdout:(21/174): python3-cephfs-19.0.0-2609.g191d963a. 1.9 MB/s | 161 kB 00:00 2024-04-02T13:18:07.012 INFO:teuthology.orchestra.run.smithi044.stdout:(22/174): librgw2-19.0.0-2609.g191d963a.el9.x86 15 MB/s | 5.1 MB 00:00 2024-04-02T13:18:07.037 INFO:teuthology.orchestra.run.smithi044.stdout:(23/174): python3-rados-19.0.0-2609.g191d963a.e 3.1 MB/s | 320 kB 00:00 2024-04-02T13:18:07.062 INFO:teuthology.orchestra.run.smithi044.stdout:(24/174): python3-rbd-19.0.0-2609.g191d963a.el9 5.8 MB/s | 299 kB 00:00 2024-04-02T13:18:07.088 INFO:teuthology.orchestra.run.smithi044.stdout:(25/174): python3-rgw-19.0.0-2609.g191d963a.el9 1.9 MB/s | 99 kB 00:00 2024-04-02T13:18:07.113 INFO:teuthology.orchestra.run.smithi044.stdout:(26/174): rbd-fuse-19.0.0-2609.g191d963a.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-02T13:18:07.180 INFO:teuthology.orchestra.run.smithi044.stdout:(27/174): rbd-nbd-19.0.0-2609.g191d963a.el9.x86 2.5 MB/s | 170 kB 00:00 2024-04-02T13:18:07.230 INFO:teuthology.orchestra.run.smithi044.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2609.g 682 kB/s | 34 kB 00:00 2024-04-02T13:18:07.289 INFO:teuthology.orchestra.run.smithi044.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2609.g191d963 2.4 MB/s | 141 kB 00:00 2024-04-02T13:18:07.348 INFO:teuthology.orchestra.run.smithi044.stdout:(30/174): rbd-mirror-19.0.0-2609.g191d963a.el9. 12 MB/s | 3.1 MB 00:00 2024-04-02T13:18:07.515 INFO:teuthology.orchestra.run.smithi044.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2609.g191d9 16 MB/s | 3.5 MB 00:00 2024-04-02T13:18:07.581 INFO:teuthology.orchestra.run.smithi044.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2609.g19 3.7 MB/s | 249 kB 00:00 2024-04-02T13:18:07.640 INFO:teuthology.orchestra.run.smithi044.stdout:(33/174): ceph-mgr-rook-19.0.0-2609.g191d963a.e 849 kB/s | 49 kB 00:00 2024-04-02T13:18:07.690 INFO:teuthology.orchestra.run.smithi044.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2609.g1 329 kB/s | 16 kB 00:00 2024-04-02T13:18:07.749 INFO:teuthology.orchestra.run.smithi044.stdout:(35/174): ceph-volume-19.0.0-2609.g191d963a.el9 4.4 MB/s | 264 kB 00:00 2024-04-02T13:18:07.841 INFO:teuthology.orchestra.run.smithi044.stdout:(36/174): cephadm-19.0.0-2609.g191d963a.el9.noa 8.1 MB/s | 762 kB 00:00 2024-04-02T13:18:07.891 INFO:teuthology.orchestra.run.smithi044.stdout:(37/174): cephfs-top-19.0.0-2609.g191d963a.el9. 366 kB/s | 18 kB 00:00 2024-04-02T13:18:07.975 INFO:teuthology.orchestra.run.smithi044.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-04-02T13:18:08.134 INFO:teuthology.orchestra.run.smithi044.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 662 kB 00:00 2024-04-02T13:18:08.167 INFO:teuthology.orchestra.run.smithi044.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.5 MB/s | 920 kB 00:00 2024-04-02T13:18:08.192 INFO:teuthology.orchestra.run.smithi044.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-04-02T13:18:08.218 INFO:teuthology.orchestra.run.smithi044.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-04-02T13:18:08.243 INFO:teuthology.orchestra.run.smithi044.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 634 kB/s | 32 kB 00:00 2024-04-02T13:18:08.318 INFO:teuthology.orchestra.run.smithi044.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-04-02T13:18:08.343 INFO:teuthology.orchestra.run.smithi044.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 174 kB/s | 17 kB 00:00 2024-04-02T13:18:08.369 INFO:teuthology.orchestra.run.smithi044.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 200 kB/s | 10 kB 00:00 2024-04-02T13:18:08.386 INFO:teuthology.orchestra.run.smithi044.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-04-02T13:18:08.411 INFO:teuthology.orchestra.run.smithi044.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-04-02T13:18:08.445 INFO:teuthology.orchestra.run.smithi044.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-04-02T13:18:08.470 INFO:teuthology.orchestra.run.smithi044.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-04-02T13:18:08.495 INFO:teuthology.orchestra.run.smithi044.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 46 kB 00:00 2024-04-02T13:18:08.521 INFO:teuthology.orchestra.run.smithi044.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-04-02T13:18:08.554 INFO:teuthology.orchestra.run.smithi044.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-04-02T13:18:08.580 INFO:teuthology.orchestra.run.smithi044.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-04-02T13:18:08.605 INFO:teuthology.orchestra.run.smithi044.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 682 kB/s | 34 kB 00:00 2024-04-02T13:18:08.706 INFO:teuthology.orchestra.run.smithi044.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-04-02T13:18:08.747 INFO:teuthology.orchestra.run.smithi044.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-04-02T13:18:08.781 INFO:teuthology.orchestra.run.smithi044.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 817 kB 00:00 2024-04-02T13:18:08.806 INFO:teuthology.orchestra.run.smithi044.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 132 kB 00:00 2024-04-02T13:18:08.915 INFO:teuthology.orchestra.run.smithi044.stdout:(60/174): postgresql-server-13.14-1.el9.x86_64. 43 MB/s | 5.7 MB 00:00 2024-04-02T13:18:08.949 INFO:teuthology.orchestra.run.smithi044.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.2 MB/s | 1.0 MB 00:00 2024-04-02T13:18:08.974 INFO:teuthology.orchestra.run.smithi044.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-04-02T13:18:09.037 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-02T13:18:09.099 INFO:teuthology.orchestra.run.smithi044.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 39 MB/s | 5.8 MB 00:00 2024-04-02T13:18:09.116 INFO:teuthology.orchestra.run.smithi044.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2024-04-02T13:18:09.133 INFO:teuthology.orchestra.run.smithi044.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 543 kB/s | 18 kB 00:00 2024-04-02T13:18:09.141 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-02T13:18:09.158 INFO:teuthology.orchestra.run.smithi044.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-02T13:18:09.175 INFO:teuthology.orchestra.run.smithi044.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 761 kB/s | 32 kB 00:00 2024-04-02T13:18:09.180 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-02T13:18:09.226 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-02T13:18:09.236 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-02T13:18:09.275 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-02T13:18:09.284 INFO:teuthology.orchestra.run.smithi044.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-04-02T13:18:09.301 INFO:teuthology.orchestra.run.smithi044.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-04-02T13:18:09.326 INFO:teuthology.orchestra.run.smithi044.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.1 MB/s | 174 kB 00:00 2024-04-02T13:18:09.329 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-02T13:18:09.352 INFO:teuthology.orchestra.run.smithi044.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-04-02T13:18:09.362 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-02T13:18:09.377 INFO:teuthology.orchestra.run.smithi044.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 819 kB/s | 41 kB 00:00 2024-04-02T13:18:09.388 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-02T13:18:09.402 INFO:teuthology.orchestra.run.smithi044.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-04-02T13:18:09.428 INFO:teuthology.orchestra.run.smithi044.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-04-02T13:18:09.438 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-02T13:18:09.453 INFO:teuthology.orchestra.run.smithi044.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-04-02T13:18:09.478 INFO:teuthology.orchestra.run.smithi044.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 949 kB/s | 47 kB 00:00 2024-04-02T13:18:09.503 INFO:teuthology.orchestra.run.smithi044.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 854 kB/s | 43 kB 00:00 2024-04-02T13:18:09.761 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-02T13:18:09.815 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librados2-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T13:18:09.820 INFO:teuthology.orchestra.run.smithi044.stdout:(78/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-04-02T13:18:09.846 INFO:teuthology.orchestra.run.smithi044.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 113 kB/s | 39 kB 00:00 2024-04-02T13:18:09.873 INFO:teuthology.orchestra.run.smithi044.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 821 kB/s | 41 kB 00:00 2024-04-02T13:18:09.896 INFO:teuthology.orchestra.run.smithi044.stdout:(81/174): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-04-02T13:18:09.922 INFO:teuthology.orchestra.run.smithi044.stdout:(82/174): qatlib-service-24.02.0-1.el9.x86_64.r 693 kB/s | 35 kB 00:00 2024-04-02T13:18:09.937 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T13:18:09.947 INFO:teuthology.orchestra.run.smithi044.stdout:(83/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 918 kB/s | 46 kB 00:00 2024-04-02T13:18:09.966 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T13:18:09.972 INFO:teuthology.orchestra.run.smithi044.stdout:(84/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-04-02T13:18:09.998 INFO:teuthology.orchestra.run.smithi044.stdout:(85/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-04-02T13:18:10.023 INFO:teuthology.orchestra.run.smithi044.stdout:(86/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-04-02T13:18:10.048 INFO:teuthology.orchestra.run.smithi044.stdout:(87/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-04-02T13:18:10.066 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T13:18:10.074 INFO:teuthology.orchestra.run.smithi044.stdout:(88/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-04-02T13:18:10.096 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 12/198 2024-04-02T13:18:10.099 INFO:teuthology.orchestra.run.smithi044.stdout:(89/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-04-02T13:18:10.133 INFO:teuthology.orchestra.run.smithi044.stdout:(90/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-04-02T13:18:10.137 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-02T13:18:10.158 INFO:teuthology.orchestra.run.smithi044.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-04-02T13:18:10.183 INFO:teuthology.orchestra.run.smithi044.stdout:(92/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-04-02T13:18:10.208 INFO:teuthology.orchestra.run.smithi044.stdout:(93/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-04-02T13:18:10.335 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-02T13:18:10.342 INFO:teuthology.orchestra.run.smithi044.stdout:(94/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-02T13:18:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-02T13:18:10.392 INFO:teuthology.orchestra.run.smithi044.stdout:(95/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-04-02T13:18:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T13:18:10.417 INFO:teuthology.orchestra.run.smithi044.stdout:(96/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-04-02T13:18:10.443 INFO:teuthology.orchestra.run.smithi044.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-02T13:18:10.460 INFO:teuthology.orchestra.run.smithi044.stdout:(98/174): elfutils-libelf-devel-0.190-2.el9.x86 512 kB/s | 21 kB 00:00 2024-04-02T13:18:10.477 INFO:teuthology.orchestra.run.smithi044.stdout:(99/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 757 kB/s | 25 kB 00:00 2024-04-02T13:18:10.509 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T13:18:10.527 INFO:teuthology.orchestra.run.smithi044.stdout:(100/174): python3-devel-3.9.17-2.el9.x86_64.rp 4.8 MB/s | 245 kB 00:00 2024-04-02T13:18:10.530 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 17/198 2024-04-02T13:18:10.594 INFO:teuthology.orchestra.run.smithi044.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 3.6 MB/s | 248 kB 00:00 2024-04-02T13:18:10.596 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 17/198 2024-04-02T13:18:10.631 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-02T13:18:10.653 INFO:teuthology.orchestra.run.smithi044.stdout:(102/174): openssl-devel-3.0.7-27.el9.x86_64.rp 21 MB/s | 4.1 MB 00:00 2024-04-02T13:18:10.657 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-02T13:18:10.672 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-02T13:18:10.678 INFO:teuthology.orchestra.run.smithi044.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.1 MB/s | 94 kB 00:00 2024-04-02T13:18:10.858 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-02T13:18:10.880 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-02T13:18:10.912 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-02T13:18:11.102 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-02T13:18:11.245 INFO:teuthology.orchestra.run.smithi044.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-04-02T13:18:11.247 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-02T13:18:11.307 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-02T13:18:11.408 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T13:18:11.458 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T13:18:11.604 INFO:teuthology.orchestra.run.smithi044.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-04-02T13:18:11.662 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T13:18:11.721 INFO:teuthology.orchestra.run.smithi044.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.6 MB/s | 309 kB 00:00 2024-04-02T13:18:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-02T13:18:11.796 INFO:teuthology.orchestra.run.smithi044.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-04-02T13:18:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-02T13:18:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-02T13:18:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-02T13:18:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-02T13:18:12.013 INFO:teuthology.orchestra.run.smithi044.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-04-02T13:18:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-02T13:18:12.080 INFO:teuthology.orchestra.run.smithi044.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-04-02T13:18:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-02T13:18:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-02T13:18:12.155 INFO:teuthology.orchestra.run.smithi044.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-04-02T13:18:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-02T13:18:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-02T13:18:12.255 INFO:teuthology.orchestra.run.smithi044.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-04-02T13:18:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-02T13:18:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-02T13:18:12.339 INFO:teuthology.orchestra.run.smithi044.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-04-02T13:18:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-02T13:18:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-02T13:18:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-common-2:19.0.0-2609.g191d963a.el9. 39/198 2024-04-02T13:18:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-argparse-2:19.0.0-2609.g191d963a.el 40/198 2024-04-02T13:18:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_6 41/198 2024-04-02T13:18:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-02T13:18:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-02T13:18:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-02T13:18:12.831 INFO:teuthology.orchestra.run.smithi044.stdout:(113/174): ceph-test-19.0.0-2609.g191d963a.el9. 6.5 MB/s | 48 MB 00:07 2024-04-02T13:18:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-02T13:18:12.865 INFO:teuthology.orchestra.run.smithi044.stdout:(114/174): protobuf-compiler-3.14.0-13.el9.x86_ 395 kB/s | 863 kB 00:02 2024-04-02T13:18:12.898 INFO:teuthology.orchestra.run.smithi044.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.5 MB/s | 838 kB 00:00 2024-04-02T13:18:12.914 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-02T13:18:12.924 INFO:teuthology.orchestra.run.smithi044.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 6.0 MB/s | 548 kB 00:00 2024-04-02T13:18:12.974 INFO:teuthology.orchestra.run.smithi044.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 588 kB/s | 43 kB 00:00 2024-04-02T13:18:13.032 INFO:teuthology.orchestra.run.smithi044.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 243 kB/s | 32 kB 00:00 2024-04-02T13:18:13.075 INFO:teuthology.orchestra.run.smithi044.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 99 kB/s | 15 kB 00:00 2024-04-02T13:18:13.100 INFO:teuthology.orchestra.run.smithi044.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-04-02T13:18:13.125 INFO:teuthology.orchestra.run.smithi044.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-04-02T13:18:13.167 INFO:teuthology.orchestra.run.smithi044.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-04-02T13:18:13.184 INFO:teuthology.orchestra.run.smithi044.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 424 kB/s | 35 kB 00:00 2024-04-02T13:18:13.268 INFO:teuthology.orchestra.run.smithi044.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.4 MB/s | 203 kB 00:00 2024-04-02T13:18:13.425 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-02T13:18:13.468 INFO:teuthology.orchestra.run.smithi044.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 6.8 MB/s | 2.0 MB 00:00 2024-04-02T13:18:13.493 INFO:teuthology.orchestra.run.smithi044.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 468 kB/s | 144 kB 00:00 2024-04-02T13:18:13.508 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-02T13:18:13.510 INFO:teuthology.orchestra.run.smithi044.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 44 kB/s | 11 kB 00:00 2024-04-02T13:18:13.527 INFO:teuthology.orchestra.run.smithi044.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-04-02T13:18:13.544 INFO:teuthology.orchestra.run.smithi044.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-04-02T13:18:13.561 INFO:teuthology.orchestra.run.smithi044.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 385 kB/s | 19 kB 00:00 2024-04-02T13:18:13.593 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-02T13:18:13.594 INFO:teuthology.orchestra.run.smithi044.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 293 kB/s | 20 kB 00:00 2024-04-02T13:18:13.636 INFO:teuthology.orchestra.run.smithi044.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-04-02T13:18:13.645 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-02T13:18:13.653 INFO:teuthology.orchestra.run.smithi044.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 503 kB/s | 46 kB 00:00 2024-04-02T13:18:13.654 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-02T13:18:13.687 INFO:teuthology.orchestra.run.smithi044.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 873 kB/s | 79 kB 00:00 2024-04-02T13:18:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-02T13:18:13.742 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-02T13:18:13.745 INFO:teuthology.orchestra.run.smithi044.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 531 kB/s | 58 kB 00:00 2024-04-02T13:18:13.779 INFO:teuthology.orchestra.run.smithi044.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-04-02T13:18:13.780 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-2:19.0.0-2609.g191d963a.el9.x86_6 53/198 2024-04-02T13:18:13.804 INFO:teuthology.orchestra.run.smithi044.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 140 kB/s | 16 kB 00:00 2024-04-02T13:18:13.811 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-02T13:18:13.863 INFO:teuthology.orchestra.run.smithi044.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 766 kB/s | 90 kB 00:00 2024-04-02T13:18:13.889 INFO:teuthology.orchestra.run.smithi044.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 258 kB/s | 28 kB 00:00 2024-04-02T13:18:13.914 INFO:teuthology.orchestra.run.smithi044.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 281 kB/s | 31 kB 00:00 2024-04-02T13:18:13.939 INFO:teuthology.orchestra.run.smithi044.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-02T13:18:14.031 INFO:teuthology.orchestra.run.smithi044.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 509 kB/s | 59 kB 00:00 2024-04-02T13:18:14.056 INFO:teuthology.orchestra.run.smithi044.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 253 kB/s | 36 kB 00:00 2024-04-02T13:18:14.081 INFO:teuthology.orchestra.run.smithi044.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 359 kB/s | 51 kB 00:00 2024-04-02T13:18:14.107 INFO:teuthology.orchestra.run.smithi044.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-02T13:18:14.124 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-02T13:18:14.132 INFO:teuthology.orchestra.run.smithi044.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-04-02T13:18:14.166 INFO:teuthology.orchestra.run.smithi044.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-04-02T13:18:14.291 INFO:teuthology.orchestra.run.smithi044.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 109 kB/s | 20 kB 00:00 2024-04-02T13:18:14.316 INFO:teuthology.orchestra.run.smithi044.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-04-02T13:18:14.330 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-02T13:18:14.400 INFO:teuthology.orchestra.run.smithi044.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-04-02T13:18:14.412 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-02T13:18:14.457 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-02T13:18:14.492 INFO:teuthology.orchestra.run.smithi044.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 25 MB/s | 2.3 MB 00:00 2024-04-02T13:18:14.575 INFO:teuthology.orchestra.run.smithi044.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-04-02T13:18:14.743 INFO:teuthology.orchestra.run.smithi044.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 246 kB/s | 41 kB 00:00 2024-04-02T13:18:14.801 INFO:teuthology.orchestra.run.smithi044.stdout:(154/174): librados2-19.0.0-2609.g191d963a.el9. 6.7 MB/s | 3.4 MB 00:00 2024-04-02T13:18:14.826 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-02T13:18:14.826 INFO:teuthology.orchestra.run.smithi044.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 247 kB 00:00 2024-04-02T13:18:14.851 INFO:teuthology.orchestra.run.smithi044.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-04-02T13:18:14.927 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-02T13:18:14.935 INFO:teuthology.orchestra.run.smithi044.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 437 kB/s | 36 kB 00:00 2024-04-02T13:18:14.960 INFO:teuthology.orchestra.run.smithi044.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 536 kB/s | 71 kB 00:00 2024-04-02T13:18:14.984 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-02T13:18:15.019 INFO:teuthology.orchestra.run.smithi044.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.2 MB/s | 192 kB 00:00 2024-04-02T13:18:15.044 INFO:teuthology.orchestra.run.smithi044.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.0 MB/s | 256 kB 00:00 2024-04-02T13:18:15.055 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-02T13:18:15.061 INFO:teuthology.orchestra.run.smithi044.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 550 kB/s | 23 kB 00:00 2024-04-02T13:18:15.085 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-02T13:18:15.112 INFO:teuthology.orchestra.run.smithi044.stdout:(162/174): librbd1-19.0.0-2609.g191d963a.el9.x8 4.0 MB/s | 3.1 MB 00:00 2024-04-02T13:18:15.122 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-02T13:18:15.137 INFO:teuthology.orchestra.run.smithi044.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-04-02T13:18:15.162 INFO:teuthology.orchestra.run.smithi044.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-04-02T13:18:15.169 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-02T13:18:15.179 INFO:teuthology.orchestra.run.smithi044.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-04-02T13:18:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-02T13:18:15.204 INFO:teuthology.orchestra.run.smithi044.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 686 kB/s | 29 kB 00:00 2024-04-02T13:18:15.250 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-02T13:18:15.255 INFO:teuthology.orchestra.run.smithi044.stdout:(167/174): libmount-2.37.4-18.el9.x86_64.rpm 953 kB/s | 136 kB 00:00 2024-04-02T13:18:15.292 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-02T13:18:15.322 INFO:teuthology.orchestra.run.smithi044.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 8.5 MB/s | 1.2 MB 00:00 2024-04-02T13:18:15.364 INFO:teuthology.orchestra.run.smithi044.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 14 MB/s | 2.2 MB 00:00 2024-04-02T13:18:15.389 INFO:teuthology.orchestra.run.smithi044.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 6.8 MB/s | 465 kB 00:00 2024-04-02T13:18:15.406 INFO:teuthology.orchestra.run.smithi044.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 742 kB/s | 31 kB 00:00 2024-04-02T13:18:15.425 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-02T13:18:15.465 INFO:teuthology.orchestra.run.smithi044.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 74 kB 00:00 2024-04-02T13:18:15.472 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-02T13:18:15.502 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-02T13:18:15.523 INFO:teuthology.orchestra.run.smithi044.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.0 MB/s | 1.1 MB 00:00 2024-04-02T13:18:15.543 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-02T13:18:15.624 INFO:teuthology.orchestra.run.smithi044.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 6.2 MB/s | 2.3 MB 00:00 2024-04-02T13:18:15.629 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-02T13:18:15.629 INFO:teuthology.orchestra.run.smithi044.stdout:Total 18 MB/s | 226 MB 00:12 2024-04-02T13:18:15.638 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-02T13:18:15.685 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-02T13:18:15.725 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-02T13:18:15.751 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-02T13:18:15.775 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-02T13:18:15.853 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-02T13:18:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-02T13:18:15.995 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-02T13:18:16.085 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-02T13:18:16.134 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-02T13:18:16.191 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-02T13:18:16.246 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-02T13:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-02T13:18:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-02T13:18:16.433 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-02T13:18:16.480 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-02T13:18:16.618 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-02T13:18:16.664 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-02T13:18:16.794 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-02T13:18:16.975 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-02T13:18:17.086 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-02T13:18:17.220 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-02T13:18:17.259 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-02T13:18:17.299 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-02T13:18:17.329 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-02T13:18:17.407 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-02T13:18:17.439 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-02T13:18:17.472 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-02T13:18:17.514 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-02T13:18:17.549 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-02T13:18:17.638 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-02T13:18:17.727 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-02T13:18:17.770 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-02T13:18:17.802 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-02T13:18:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-02T13:18:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-02T13:18:17.995 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-02T13:18:18.353 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-02T13:18:18.353 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-02T13:18:18.720 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-02T13:18:18.768 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-02T13:18:18.801 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-02T13:18:18.849 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-02T13:18:18.962 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-02T13:18:19.011 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-02T13:18:19.189 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-02T13:18:19.217 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-02T13:18:19.295 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-02T13:18:19.933 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-02T13:18:20.039 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-02T13:18:21.243 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-02T13:18:21.334 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-02T13:18:21.357 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-02T13:18:21.380 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-02T13:18:21.407 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-02T13:18:21.723 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-02T13:18:21.754 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 124/198 2024-04-02T13:18:21.836 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 124/198 2024-04-02T13:18:21.867 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 125/198 2024-04-02T13:18:21.913 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-02T13:18:22.377 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-02T13:18:22.415 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 128/198 2024-04-02T13:18:22.486 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 128/198 2024-04-02T13:18:22.508 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 129/198 2024-04-02T13:18:24.679 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T13:18:24.721 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T13:18:25.256 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T13:18:25.308 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 131/198 2024-04-02T13:18:25.397 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 131/198 2024-04-02T13:18:25.398 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-02T13:18:25.398 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-02T13:18:25.398 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:18:25.434 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 132/198 2024-04-02T13:18:25.536 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-02T13:18:25.620 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-02T13:18:25.646 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-02T13:18:25.676 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-02T13:18:25.685 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-02T13:18:25.737 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-02T13:18:25.791 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-02T13:18:25.832 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-02T13:18:25.866 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-02T13:18:25.919 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-02T13:18:26.262 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-02T13:18:26.317 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librados2-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T13:18:26.442 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T13:18:26.468 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T13:18:26.560 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T13:18:26.589 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 12/198 2024-04-02T13:18:26.628 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-02T13:18:26.826 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-02T13:18:26.871 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-02T13:18:26.898 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T13:18:26.996 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T13:18:27.025 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 17/198 2024-04-02T13:18:27.073 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 17/198 2024-04-02T13:18:27.115 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-02T13:18:27.143 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-02T13:18:27.158 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-02T13:18:27.347 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-02T13:18:27.381 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-02T13:18:27.418 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-02T13:18:27.603 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-02T13:18:27.720 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-02T13:18:27.763 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-02T13:18:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T13:18:27.906 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T13:18:28.128 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T13:18:28.240 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-02T13:18:28.338 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-02T13:18:28.378 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-02T13:18:28.405 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-02T13:18:28.451 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-02T13:18:28.530 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-02T13:18:28.589 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-02T13:18:28.626 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-02T13:18:28.693 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-02T13:18:28.734 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-02T13:18:28.777 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-02T13:18:28.810 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-02T13:18:28.837 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-02T13:18:28.910 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-02T13:18:28.942 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-common-2:19.0.0-2609.g191d963a.el9. 39/198 2024-04-02T13:18:28.986 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-argparse-2:19.0.0-2609.g191d963a.el 40/198 2024-04-02T13:18:29.014 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_6 41/198 2024-04-02T13:18:29.088 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-02T13:18:29.138 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-02T13:18:29.215 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-02T13:18:29.352 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-02T13:18:29.407 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-02T13:18:29.967 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-02T13:18:30.054 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-02T13:18:30.141 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-02T13:18:30.188 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-02T13:18:30.197 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-02T13:18:30.229 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-02T13:18:30.277 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-02T13:18:30.315 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librados-devel-2:19.0.0-2609.g191d963a.el9.x86_6 53/198 2024-04-02T13:18:30.345 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-02T13:18:30.647 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-02T13:18:30.853 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-02T13:18:30.939 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-02T13:18:30.983 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-02T13:18:31.372 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-02T13:18:31.420 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-02T13:18:31.477 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-02T13:18:31.550 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-02T13:18:31.586 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-02T13:18:31.615 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-02T13:18:31.663 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-02T13:18:31.697 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-02T13:18:31.742 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-02T13:18:31.784 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-02T13:18:31.927 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-02T13:18:31.980 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-02T13:18:32.010 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-02T13:18:32.052 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-02T13:18:32.151 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-02T13:18:32.201 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-02T13:18:32.242 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-02T13:18:32.292 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-02T13:18:32.316 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-02T13:18:32.402 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-02T13:18:32.532 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-02T13:18:32.647 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-02T13:18:32.692 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-02T13:18:32.750 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-02T13:18:32.815 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-02T13:18:33.009 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-02T13:18:33.071 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-02T13:18:33.216 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-02T13:18:33.252 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-02T13:18:33.284 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-02T13:18:33.314 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-02T13:18:33.335 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-02T13:18:33.369 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-02T13:18:33.408 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-02T13:18:33.440 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-02T13:18:33.469 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-02T13:18:33.549 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-02T13:18:33.580 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-02T13:18:33.613 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-02T13:18:33.655 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-02T13:18:33.689 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-02T13:18:33.784 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-02T13:18:33.873 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-02T13:18:33.921 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-02T13:18:33.951 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-02T13:18:34.000 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-02T13:18:34.110 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-02T13:18:34.171 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-02T13:18:34.945 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-02T13:18:35.000 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-02T13:18:35.034 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-02T13:18:35.084 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-02T13:18:35.175 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-02T13:18:35.218 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-02T13:18:35.403 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-02T13:18:35.432 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-02T13:18:35.513 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-02T13:18:36.205 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-02T13:18:36.318 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-02T13:18:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 132/198 2024-04-02T13:18:37.235 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /sys 2024-04-02T13:18:37.235 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /proc 2024-04-02T13:18:37.235 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /mnt 2024-04-02T13:18:37.235 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /var/tmp 2024-04-02T13:18:37.235 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /home 2024-04-02T13:18:37.236 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /root 2024-04-02T13:18:37.236 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /tmp 2024-04-02T13:18:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:18:37.390 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 133/198 2024-04-02T13:18:37.405 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 133/198 2024-04-02T13:18:37.405 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:18:37.405 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-02T13:18:37.406 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:18:37.629 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-02T13:18:37.900 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-02T13:18:37.931 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-02T13:18:37.962 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-02T13:18:37.989 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-02T13:18:38.320 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-02T13:18:38.344 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 124/198 2024-04-02T13:18:38.436 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 124/198 2024-04-02T13:18:38.465 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 125/198 2024-04-02T13:18:38.505 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-02T13:18:38.868 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 134/198 2024-04-02T13:18:38.893 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 134/198 2024-04-02T13:18:38.893 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:18:38.894 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-02T13:18:38.894 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T13:18:38.894 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T13:18:38.894 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:18:38.989 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-02T13:18:39.014 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 128/198 2024-04-02T13:18:39.085 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 128/198 2024-04-02T13:18:39.115 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 129/198 2024-04-02T13:18:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 135/198 2024-04-02T13:18:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 135/198 2024-04-02T13:18:39.141 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:18:39.141 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-02T13:18:39.141 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T13:18:39.141 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T13:18:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:18:39.578 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 136/198 2024-04-02T13:18:39.600 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 136/198 2024-04-02T13:18:39.600 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:18:39.600 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-02T13:18:39.600 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T13:18:39.600 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T13:18:39.600 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:18:39.712 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-02T13:18:40.098 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:19.0.0-2609.g191d963a.el9.noarch 138/198 2024-04-02T13:18:40.127 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephadm-2:19.0.0-2609.g191d963a.el9.noarch 138/198 2024-04-02T13:18:40.160 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.e 139/198 2024-04-02T13:18:40.220 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2609.g191d963a. 140/198 2024-04-02T13:18:40.251 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 141/198 2024-04-02T13:18:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 141/198 2024-04-02T13:18:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 142/198 2024-04-02T13:18:41.486 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T13:18:41.520 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T13:18:42.091 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T13:18:42.140 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 131/198 2024-04-02T13:18:42.228 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 131/198 2024-04-02T13:18:42.229 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-02T13:18:42.229 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-02T13:18:42.229 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:18:42.274 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 132/198 2024-04-02T13:18:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 142/198 2024-04-02T13:18:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 143/198 2024-04-02T13:18:42.412 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 143/198 2024-04-02T13:18:42.573 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el 144/198 2024-04-02T13:18:42.613 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 145/198 2024-04-02T13:18:42.636 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 145/198 2024-04-02T13:18:42.637 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:18:42.637 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-02T13:18:42.637 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T13:18:42.637 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T13:18:42.637 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:18:42.682 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 146/198 2024-04-02T13:18:42.692 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 146/198 2024-04-02T13:18:42.745 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-2:19.0.0-2609.g191d963a.el9.x86_64 147/198 2024-04-02T13:18:43.617 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-02T13:18:43.641 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 149/198 2024-04-02T13:18:43.663 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 149/198 2024-04-02T13:18:43.663 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:18:43.664 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-02T13:18:43.664 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T13:18:43.664 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T13:18:43.664 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:18:43.708 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2609.g191d9 150/198 2024-04-02T13:18:43.730 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2609.g191d9 150/198 2024-04-02T13:18:43.730 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:18:43.730 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-02T13:18:43.730 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:18:43.776 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T13:18:43.798 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T13:18:43.798 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:18:43.798 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-02T13:18:43.798 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-02T13:18:43.798 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-02T13:18:43.798 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:18:44.110 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 152/198 2024-04-02T13:18:44.133 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 152/198 2024-04-02T13:18:44.133 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:18:44.133 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-02T13:18:44.133 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T13:18:44.133 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T13:18:44.133 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:18:48.606 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 153/198 2024-04-02T13:18:48.647 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 154/198 2024-04-02T13:18:48.686 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 155/198 2024-04-02T13:18:48.713 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-02T13:18:48.904 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 157/198 2024-04-02T13:18:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-02T13:18:49.077 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 159/198 2024-04-02T13:18:49.132 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-02T13:18:49.186 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-02T13:18:49.229 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-02T13:18:49.253 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_ 163/198 2024-04-02T13:18:49.294 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-02T13:18:49.324 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-04-02T13:18:49.381 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-04-02T13:18:49.446 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-02T13:18:49.489 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-02T13:18:50.087 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T13:18:50.129 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T13:18:50.150 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T13:18:50.975 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 170/198 2024-04-02T13:18:51.049 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-02T13:18:51.090 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-02T13:18:51.181 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-02T13:18:51.211 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-02T13:18:51.236 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-02T13:18:51.261 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-02T13:18:51.296 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-02T13:18:51.296 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-02T13:18:51.376 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-02T13:18:51.405 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-02T13:18:51.406 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-02T13:18:51.498 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-02T13:18:51.525 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-02T13:18:51.558 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-02T13:18:51.583 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-02T13:18:51.608 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-02T13:18:51.641 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-02T13:18:51.681 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-02T13:18:51.713 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-02T13:18:51.738 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-02T13:18:51.763 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-02T13:18:51.797 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-02T13:18:51.822 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-02T13:18:51.829 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T13:18:51.829 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T13:18:52.008 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T13:18:52.009 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-02T13:18:52.091 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-02T13:18:52.123 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-02T13:18:52.150 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-02T13:18:52.175 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-02T13:18:52.200 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-02T13:18:52.274 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-02T13:18:54.818 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 132/198 2024-04-02T13:18:54.818 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /sys 2024-04-02T13:18:54.818 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /proc 2024-04-02T13:18:54.819 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /mnt 2024-04-02T13:18:54.819 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /var/tmp 2024-04-02T13:18:54.819 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /home 2024-04-02T13:18:54.819 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /root 2024-04-02T13:18:54.819 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /tmp 2024-04-02T13:18:54.819 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:18:54.996 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 133/198 2024-04-02T13:18:55.011 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 133/198 2024-04-02T13:18:55.012 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:18:55.012 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-02T13:18:55.012 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:18:56.533 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 134/198 2024-04-02T13:18:56.558 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 134/198 2024-04-02T13:18:56.558 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:18:56.561 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-02T13:18:56.561 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T13:18:56.561 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T13:18:56.561 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:18:56.792 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 135/198 2024-04-02T13:18:56.814 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 135/198 2024-04-02T13:18:56.814 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:18:56.814 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-02T13:18:56.814 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T13:18:56.814 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T13:18:56.814 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:18:57.276 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 136/198 2024-04-02T13:18:57.297 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 136/198 2024-04-02T13:18:57.297 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:18:57.297 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-02T13:18:57.297 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T13:18:57.298 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T13:18:57.298 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:18:57.394 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-02T13:18:57.499 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-02T13:18:57.499 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-2:19.0.0-2609.g191d963a.el9.x86_64 1/198 2024-04-02T13:18:57.499 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 2/198 2024-04-02T13:18:57.499 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 3/198 2024-04-02T13:18:57.499 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 4/198 2024-04-02T13:18:57.500 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2609.g191d9 5/198 2024-04-02T13:18:57.500 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 6/198 2024-04-02T13:18:57.500 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 7/198 2024-04-02T13:18:57.500 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 8/198 2024-04-02T13:18:57.500 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 9/198 2024-04-02T13:18:57.500 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T13:18:57.500 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T13:18:57.500 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 12/198 2024-04-02T13:18:57.500 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 13/198 2024-04-02T13:18:57.500 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 14/198 2024-04-02T13:18:57.500 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_ 15/198 2024-04-02T13:18:57.500 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T13:18:57.500 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 17/198 2024-04-02T13:18:57.501 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-2:19.0.0-2609.g191d963a.el9.x86_6 18/198 2024-04-02T13:18:57.501 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 19/198 2024-04-02T13:18:57.501 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 20/198 2024-04-02T13:18:57.501 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2609.g191d963a.el 21/198 2024-04-02T13:18:57.501 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-common-2:19.0.0-2609.g191d963a.el9. 22/198 2024-04-02T13:18:57.501 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_6 23/198 2024-04-02T13:18:57.501 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 24/198 2024-04-02T13:18:57.501 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 25/198 2024-04-02T13:18:57.501 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 26/198 2024-04-02T13:18:57.501 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 27/198 2024-04-02T13:18:57.501 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 28/198 2024-04-02T13:18:57.501 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 29/198 2024-04-02T13:18:57.517 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2609.g191d963a. 30/198 2024-04-02T13:18:57.517 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 31/198 2024-04-02T13:18:57.517 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 32/198 2024-04-02T13:18:57.517 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 33/198 2024-04-02T13:18:57.517 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el 34/198 2024-04-02T13:18:57.517 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 35/198 2024-04-02T13:18:57.517 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.e 36/198 2024-04-02T13:18:57.517 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 37/198 2024-04-02T13:18:57.517 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephadm-2:19.0.0-2609.g191d963a.el9.noarch 38/198 2024-04-02T13:18:57.518 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 39/198 2024-04-02T13:18:57.518 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-02T13:18:57.518 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-02T13:18:57.518 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-02T13:18:57.518 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-02T13:18:57.518 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-02T13:18:57.518 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-02T13:18:57.518 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-02T13:18:57.518 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-02T13:18:57.518 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-02T13:18:57.518 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-02T13:18:57.518 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-02T13:18:57.518 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-02T13:18:57.519 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-02T13:18:57.520 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-02T13:18:57.521 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-02T13:18:57.522 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-02T13:18:57.522 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-02T13:18:57.522 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-02T13:18:57.522 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-02T13:18:57.522 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-02T13:18:57.522 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-02T13:18:57.522 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-02T13:18:57.522 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-02T13:18:57.522 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-02T13:18:57.522 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-02T13:18:57.522 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-02T13:18:57.522 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-02T13:18:57.522 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-02T13:18:57.523 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-02T13:18:57.524 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-02T13:18:57.525 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-02T13:18:57.525 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-02T13:18:57.525 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-02T13:18:57.525 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-02T13:18:57.525 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-02T13:18:57.525 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-02T13:18:57.525 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-02T13:18:57.525 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-02T13:18:57.525 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-02T13:18:57.525 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-02T13:18:57.525 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T13:18:57.525 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-02T13:18:57.525 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 153/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-02T13:18:57.526 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-02T13:18:57.527 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-02T13:18:57.527 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-02T13:18:57.527 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-02T13:18:57.527 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-02T13:18:57.527 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-02T13:18:57.527 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-02T13:18:57.527 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-02T13:18:57.527 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-02T13:18:57.527 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-02T13:18:57.527 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-02T13:18:57.527 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-02T13:18:57.527 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-02T13:18:57.527 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-02T13:18:57.528 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-02T13:18:57.529 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-02T13:18:57.529 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-02T13:18:57.823 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephadm-2:19.0.0-2609.g191d963a.el9.noarch 138/198 2024-04-02T13:18:57.851 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephadm-2:19.0.0-2609.g191d963a.el9.noarch 138/198 2024-04-02T13:18:57.884 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.e 139/198 2024-04-02T13:18:57.948 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2609.g191d963a. 140/198 2024-04-02T13:18:57.974 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 141/198 2024-04-02T13:18:59.068 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 141/198 2024-04-02T13:18:59.101 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 142/198 2024-04-02T13:19:00.117 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 142/198 2024-04-02T13:19:00.152 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 143/198 2024-04-02T13:19:00.256 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 143/198 2024-04-02T13:19:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el 144/198 2024-04-02T13:19:00.453 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 145/198 2024-04-02T13:19:00.475 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 145/198 2024-04-02T13:19:00.476 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:19:00.476 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-02T13:19:00.476 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T13:19:00.476 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T13:19:00.476 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:00.520 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 146/198 2024-04-02T13:19:00.531 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 146/198 2024-04-02T13:19:00.580 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-02T13:19:00.580 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:00.580 INFO:teuthology.orchestra.run.smithi069.stdout:Upgraded: 2024-04-02T13:19:00.580 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-04-02T13:19:00.580 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-04-02T13:19:00.580 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-04-02T13:19:00.580 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-04-02T13:19:00.580 INFO:teuthology.orchestra.run.smithi069.stdout: acl-2.3.1-4.el9.x86_64 2024-04-02T13:19:00.580 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-02T13:19:00.580 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-02T13:19:00.580 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-02T13:19:00.580 INFO:teuthology.orchestra.run.smithi069.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-02T13:19:00.580 INFO:teuthology.orchestra.run.smithi069.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-02T13:19:00.581 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: automake-1.16.2-8.el9.noarch 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: bison-3.7.4-5.el9.x86_64 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:00.582 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-02T13:19:00.583 INFO:teuthology.orchestra.run.smithi069.stdout: flex-2.6.4-9.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: jq-1.6-16.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-02T13:19:00.584 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: lua-5.4.4-4.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-02T13:19:00.585 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-13.14-1.el9.x86_64 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-02T13:19:00.586 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-02T13:19:00.587 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-02T13:19:00.588 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply-3.11-14.el9.noarch 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-02T13:19:00.589 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-02T13:19:00.590 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: re2-1:20211101-3.el9.x86_64 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-02T13:19:00.591 INFO:teuthology.orchestra.run.smithi069.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-02T13:19:00.592 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-02T13:19:00.592 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-02T13:19:00.592 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-2:19.0.0-2609.g191d963a.el9.x86_64 147/198 2024-04-02T13:19:00.592 INFO:teuthology.orchestra.run.smithi069.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-02T13:19:00.592 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:00.592 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-02T13:19:01.076 DEBUG:teuthology.parallel:result is None 2024-04-02T13:19:01.499 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-02T13:19:01.531 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 149/198 2024-04-02T13:19:01.553 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 149/198 2024-04-02T13:19:01.553 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:19:01.553 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-02T13:19:01.553 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T13:19:01.554 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T13:19:01.554 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:01.599 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2609.g191d9 150/198 2024-04-02T13:19:01.620 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2609.g191d9 150/198 2024-04-02T13:19:01.620 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:19:01.620 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-02T13:19:01.620 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:01.666 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T13:19:01.688 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T13:19:01.688 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:19:01.688 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-02T13:19:01.688 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-02T13:19:01.688 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-02T13:19:01.688 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:02.008 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 152/198 2024-04-02T13:19:02.030 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 152/198 2024-04-02T13:19:02.031 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T13:19:02.031 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-02T13:19:02.031 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T13:19:02.031 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T13:19:02.031 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:06.788 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 153/198 2024-04-02T13:19:06.838 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 154/198 2024-04-02T13:19:06.868 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 155/198 2024-04-02T13:19:06.895 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-02T13:19:07.099 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 157/198 2024-04-02T13:19:07.215 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-02T13:19:07.260 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 159/198 2024-04-02T13:19:07.314 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-02T13:19:07.368 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-02T13:19:07.403 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-02T13:19:07.435 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_ 163/198 2024-04-02T13:19:07.467 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-02T13:19:07.497 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-04-02T13:19:07.567 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-04-02T13:19:07.641 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-02T13:19:07.679 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-02T13:19:08.330 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T13:19:08.369 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T13:19:08.388 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T13:19:09.296 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 170/198 2024-04-02T13:19:09.382 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-02T13:19:09.424 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-02T13:19:09.514 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-02T13:19:09.553 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-02T13:19:09.577 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-02T13:19:09.603 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-02T13:19:09.629 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-02T13:19:09.629 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-02T13:19:09.718 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-02T13:19:09.747 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-02T13:19:09.747 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-02T13:19:09.831 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-02T13:19:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-02T13:19:09.891 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-02T13:19:09.916 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-02T13:19:09.941 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-02T13:19:09.976 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-02T13:19:10.023 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-02T13:19:10.062 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-02T13:19:10.088 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-02T13:19:10.113 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-02T13:19:10.138 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-02T13:19:10.189 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-02T13:19:10.195 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T13:19:10.195 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T13:19:10.375 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T13:19:10.375 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-02T13:19:10.743 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-02T13:19:10.865 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-02T13:19:10.991 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-02T13:19:11.017 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-02T13:19:11.042 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-02T13:19:11.113 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-02T13:19:16.173 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-02T13:19:16.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-2:19.0.0-2609.g191d963a.el9.x86_64 1/198 2024-04-02T13:19:16.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 2/198 2024-04-02T13:19:16.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 3/198 2024-04-02T13:19:16.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 4/198 2024-04-02T13:19:16.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2609.g191d9 5/198 2024-04-02T13:19:16.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 6/198 2024-04-02T13:19:16.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 7/198 2024-04-02T13:19:16.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 8/198 2024-04-02T13:19:16.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 9/198 2024-04-02T13:19:16.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T13:19:16.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T13:19:16.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 12/198 2024-04-02T13:19:16.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 13/198 2024-04-02T13:19:16.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 14/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_ 15/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 17/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados-devel-2:19.0.0-2609.g191d963a.el9.x86_6 18/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 19/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 20/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2609.g191d963a.el 21/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-common-2:19.0.0-2609.g191d963a.el9. 22/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_6 23/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 24/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 25/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 26/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 27/198 2024-04-02T13:19:16.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 28/198 2024-04-02T13:19:16.176 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 29/198 2024-04-02T13:19:16.176 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2609.g191d963a. 30/198 2024-04-02T13:19:16.176 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 31/198 2024-04-02T13:19:16.192 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 32/198 2024-04-02T13:19:16.192 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 33/198 2024-04-02T13:19:16.192 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el 34/198 2024-04-02T13:19:16.192 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 35/198 2024-04-02T13:19:16.192 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.e 36/198 2024-04-02T13:19:16.192 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 37/198 2024-04-02T13:19:16.192 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephadm-2:19.0.0-2609.g191d963a.el9.noarch 38/198 2024-04-02T13:19:16.192 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 39/198 2024-04-02T13:19:16.192 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-02T13:19:16.193 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-02T13:19:16.193 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-02T13:19:16.193 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-02T13:19:16.193 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-02T13:19:16.193 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-02T13:19:16.193 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-02T13:19:16.193 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-02T13:19:16.193 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-02T13:19:16.193 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-02T13:19:16.193 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-02T13:19:16.193 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-02T13:19:16.193 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-02T13:19:16.193 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-02T13:19:16.194 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-02T13:19:16.194 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-02T13:19:16.194 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-02T13:19:16.194 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-02T13:19:16.194 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-02T13:19:16.194 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-02T13:19:16.194 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-02T13:19:16.194 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-02T13:19:16.194 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-02T13:19:16.194 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-02T13:19:16.194 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-02T13:19:16.194 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-02T13:19:16.194 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-02T13:19:16.195 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-02T13:19:16.196 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-02T13:19:16.196 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-02T13:19:16.196 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-02T13:19:16.196 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-02T13:19:16.196 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-02T13:19:16.196 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-02T13:19:16.196 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-02T13:19:16.196 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-02T13:19:16.196 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-02T13:19:16.196 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-02T13:19:16.196 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-02T13:19:16.196 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-02T13:19:16.196 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-02T13:19:16.197 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-02T13:19:16.197 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-02T13:19:16.197 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-02T13:19:16.197 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-02T13:19:16.197 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-02T13:19:16.197 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-02T13:19:16.197 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-02T13:19:16.197 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-02T13:19:16.197 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-02T13:19:16.197 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-02T13:19:16.197 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-02T13:19:16.197 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-02T13:19:16.198 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-02T13:19:16.198 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-02T13:19:16.198 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-02T13:19:16.198 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-02T13:19:16.198 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-02T13:19:16.198 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-02T13:19:16.198 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-02T13:19:16.198 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-02T13:19:16.199 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-02T13:19:16.199 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-02T13:19:16.199 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-02T13:19:16.199 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-02T13:19:16.199 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-02T13:19:16.199 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-02T13:19:16.199 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-02T13:19:16.199 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-02T13:19:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-02T13:19:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-02T13:19:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-02T13:19:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-02T13:19:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-02T13:19:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-02T13:19:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-02T13:19:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-02T13:19:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-02T13:19:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-02T13:19:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-02T13:19:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-02T13:19:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-02T13:19:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-02T13:19:16.202 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-02T13:19:16.202 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-02T13:19:16.202 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T13:19:16.202 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-02T13:19:16.202 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 153/198 2024-04-02T13:19:16.202 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-02T13:19:16.202 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-04-02T13:19:16.202 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-02T13:19:16.202 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-04-02T13:19:16.202 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-02T13:19:16.202 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-04-02T13:19:16.202 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-02T13:19:16.202 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-02T13:19:16.203 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-02T13:19:16.204 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-02T13:19:16.204 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-02T13:19:16.204 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-02T13:19:16.204 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-02T13:19:16.204 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-02T13:19:16.204 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-02T13:19:16.204 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-02T13:19:16.204 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-02T13:19:16.205 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-02T13:19:16.205 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-02T13:19:16.205 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-02T13:19:16.205 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-02T13:19:16.205 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-02T13:19:16.205 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-02T13:19:16.205 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-02T13:19:16.205 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-02T13:19:16.205 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-02T13:19:16.205 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-02T13:19:16.205 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-02T13:19:16.205 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-02T13:19:16.205 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-02T13:19:16.206 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-02T13:19:19.189 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-02T13:19:19.189 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:19.189 INFO:teuthology.orchestra.run.smithi044.stdout:Upgraded: 2024-04-02T13:19:19.189 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-04-02T13:19:19.190 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-04-02T13:19:19.190 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-04-02T13:19:19.190 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-04-02T13:19:19.190 INFO:teuthology.orchestra.run.smithi044.stdout: acl-2.3.1-4.el9.x86_64 2024-04-02T13:19:19.190 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-02T13:19:19.190 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-02T13:19:19.190 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-02T13:19:19.190 INFO:teuthology.orchestra.run.smithi044.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-02T13:19:19.190 INFO:teuthology.orchestra.run.smithi044.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-02T13:19:19.190 INFO:teuthology.orchestra.run.smithi044.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-02T13:19:19.190 INFO:teuthology.orchestra.run.smithi044.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-02T13:19:19.191 INFO:teuthology.orchestra.run.smithi044.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-02T13:19:19.191 INFO:teuthology.orchestra.run.smithi044.stdout: librados2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.191 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.191 INFO:teuthology.orchestra.run.smithi044.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-02T13:19:19.191 INFO:teuthology.orchestra.run.smithi044.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-02T13:19:19.191 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-02T13:19:19.191 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-02T13:19:19.191 INFO:teuthology.orchestra.run.smithi044.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-02T13:19:19.191 INFO:teuthology.orchestra.run.smithi044.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-02T13:19:19.191 INFO:teuthology.orchestra.run.smithi044.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-02T13:19:19.191 INFO:teuthology.orchestra.run.smithi044.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-02T13:19:19.191 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-02T13:19:19.191 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-02T13:19:19.192 INFO:teuthology.orchestra.run.smithi044.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-02T13:19:19.192 INFO:teuthology.orchestra.run.smithi044.stdout: automake-1.16.2-8.el9.noarch 2024-04-02T13:19:19.192 INFO:teuthology.orchestra.run.smithi044.stdout: bison-3.7.4-5.el9.x86_64 2024-04-02T13:19:19.192 INFO:teuthology.orchestra.run.smithi044.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-02T13:19:19.192 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.192 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.192 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.192 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.192 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:19.192 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.192 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.192 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.192 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:19.193 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:19.193 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:19.193 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:19.193 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:19.193 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.193 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.193 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:19.193 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.193 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.193 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.193 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:19.193 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:19.194 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.194 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.194 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T13:19:19.194 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-02T13:19:19.194 INFO:teuthology.orchestra.run.smithi044.stdout: flex-2.6.4-9.el9.x86_64 2024-04-02T13:19:19.194 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-02T13:19:19.194 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-02T13:19:19.194 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-02T13:19:19.194 INFO:teuthology.orchestra.run.smithi044.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-02T13:19:19.194 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-02T13:19:19.194 INFO:teuthology.orchestra.run.smithi044.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-02T13:19:19.194 INFO:teuthology.orchestra.run.smithi044.stdout: jq-1.6-16.el9.x86_64 2024-04-02T13:19:19.194 INFO:teuthology.orchestra.run.smithi044.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-02T13:19:19.195 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-02T13:19:19.195 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-02T13:19:19.195 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.195 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.195 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.195 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-02T13:19:19.195 INFO:teuthology.orchestra.run.smithi044.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-02T13:19:19.195 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-02T13:19:19.195 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-02T13:19:19.195 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-02T13:19:19.195 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.195 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.195 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-02T13:19:19.196 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.196 INFO:teuthology.orchestra.run.smithi044.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-02T13:19:19.196 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-02T13:19:19.196 INFO:teuthology.orchestra.run.smithi044.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-02T13:19:19.196 INFO:teuthology.orchestra.run.smithi044.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-02T13:19:19.196 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-02T13:19:19.196 INFO:teuthology.orchestra.run.smithi044.stdout: lua-5.4.4-4.el9.x86_64 2024-04-02T13:19:19.196 INFO:teuthology.orchestra.run.smithi044.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-02T13:19:19.196 INFO:teuthology.orchestra.run.smithi044.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-02T13:19:19.196 INFO:teuthology.orchestra.run.smithi044.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-02T13:19:19.196 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-02T13:19:19.196 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-02T13:19:19.196 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-02T13:19:19.197 INFO:teuthology.orchestra.run.smithi044.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-02T13:19:19.197 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-13.14-1.el9.x86_64 2024-04-02T13:19:19.197 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-02T13:19:19.197 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-02T13:19:19.197 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-02T13:19:19.197 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-02T13:19:19.197 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-02T13:19:19.197 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-02T13:19:19.197 INFO:teuthology.orchestra.run.smithi044.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-02T13:19:19.197 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-02T13:19:19.197 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-02T13:19:19.197 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-02T13:19:19.197 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.198 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.198 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.198 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-02T13:19:19.198 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-02T13:19:19.198 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-02T13:19:19.198 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-02T13:19:19.198 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-02T13:19:19.198 INFO:teuthology.orchestra.run.smithi044.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-02T13:19:19.198 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-02T13:19:19.198 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-02T13:19:19.198 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-02T13:19:19.198 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-02T13:19:19.199 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-02T13:19:19.199 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-02T13:19:19.199 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-02T13:19:19.199 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-02T13:19:19.199 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-02T13:19:19.199 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-02T13:19:19.199 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-02T13:19:19.199 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-02T13:19:19.199 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-02T13:19:19.199 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-02T13:19:19.199 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-02T13:19:19.199 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-02T13:19:19.199 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-02T13:19:19.200 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-02T13:19:19.200 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-02T13:19:19.200 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-02T13:19:19.200 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-02T13:19:19.200 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-02T13:19:19.200 INFO:teuthology.orchestra.run.smithi044.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-02T13:19:19.200 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-02T13:19:19.200 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply-3.11-14.el9.noarch 2024-04-02T13:19:19.200 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-02T13:19:19.200 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-02T13:19:19.201 INFO:teuthology.orchestra.run.smithi044.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-02T13:19:19.201 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-02T13:19:19.201 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-02T13:19:19.201 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-02T13:19:19.201 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-02T13:19:19.201 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-02T13:19:19.201 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-02T13:19:19.201 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.201 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.201 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-02T13:19:19.201 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-02T13:19:19.201 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-02T13:19:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-02T13:19:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-02T13:19:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-02T13:19:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-02T13:19:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-02T13:19:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-02T13:19:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-02T13:19:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-02T13:19:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-02T13:19:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-02T13:19:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-02T13:19:19.202 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-02T13:19:19.203 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-02T13:19:19.203 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-02T13:19:19.203 INFO:teuthology.orchestra.run.smithi044.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-02T13:19:19.203 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.203 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.203 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T13:19:19.203 INFO:teuthology.orchestra.run.smithi044.stdout: re2-1:20211101-3.el9.x86_64 2024-04-02T13:19:19.203 INFO:teuthology.orchestra.run.smithi044.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-02T13:19:19.203 INFO:teuthology.orchestra.run.smithi044.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-02T13:19:19.203 INFO:teuthology.orchestra.run.smithi044.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-02T13:19:19.203 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-02T13:19:19.203 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-02T13:19:19.204 INFO:teuthology.orchestra.run.smithi044.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-02T13:19:19.204 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:19.204 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-02T13:19:19.640 DEBUG:teuthology.parallel:result is None 2024-04-02T13:19:19.641 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T13:19:19.788 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-02T13:19:19.809 INFO:teuthology.orchestra.run.smithi044.stdout:19.0.0-2609.g191d963a.el9 2024-04-02T13:19:19.810 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2609.g191d963a.el9 2024-04-02T13:19:19.810 INFO:teuthology.task.install:The correct ceph version 19.0.0-2609.g191d963a is installed. 2024-04-02T13:19:19.812 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T13:19:19.951 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-02T13:19:19.977 INFO:teuthology.orchestra.run.smithi069.stdout:19.0.0-2609.g191d963a.el9 2024-04-02T13:19:19.977 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2609.g191d963a.el9 2024-04-02T13:19:19.977 INFO:teuthology.task.install:The correct ceph version 19.0.0-2609.g191d963a is installed. 2024-04-02T13:19:19.979 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-02T13:19:19.980 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:19.980 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-02T13:19:20.010 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:20.010 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-02T13:19:20.052 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-02T13:19:20.052 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:20.052 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/daemon-helper 2024-04-02T13:19:20.081 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-02T13:19:20.149 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:20.149 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2024-04-02T13:19:20.180 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-02T13:19:20.250 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-02T13:19:20.250 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:20.250 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-02T13:19:20.278 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-02T13:19:20.347 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:20.347 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-02T13:19:20.376 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-02T13:19:20.443 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-02T13:19:20.443 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:20.443 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/stdin-killer 2024-04-02T13:19:20.471 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-02T13:19:20.539 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:20.539 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2024-04-02T13:19:20.566 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-02T13:19:20.634 INFO:teuthology.run_tasks:Running task ceph... 2024-04-02T13:19:20.756 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-02T13:19:20.756 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 777 /var/log/ceph 2024-04-02T13:19:20.759 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /var/log/ceph 2024-04-02T13:19:20.786 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-02T13:19:20.786 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-02T13:19:20.827 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-02T13:19:20.854 INFO:tasks.ceph:Creating extra log directories... 2024-04-02T13:19:20.854 DEBUG:teuthology.orchestra.run.smithi044:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-02T13:19:20.894 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-02T13:19:20.928 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-02T13:19:20.929 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-02T13:19:20.929 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-02T13:19:20.929 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:20.929 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-02T13:19:20.975 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-02T13:19:21.097 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-02T13:19:21.124 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:21.124 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-02T13:19:21.139 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-02T13:19:21.259 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-02T13:19:21.291 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-02T13:19:21.291 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-02T13:19:21.291 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584', 'branch': 'wip-batrick-testing-20240402.004512', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/forward-scrub}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7635584', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20240402.004512'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20240402.004512', 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 2763, 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6', 'sleep_before_teardown': 0, 'subset': '318/512', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20240402.004512', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_191d963ac9de7322da8aff20b567b3fa458e2ef6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi044.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOu/jrkQApjbih97vw1u9wYxTxokLSOZ2y1zgfVlNMVM4z366JseNO9ozPlyu/PESU4OadfFOZPGuTdw+IgBQE=', 'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI/Yrp+Fke5qTtFll39/EG3e2sGtZ0iT5jGlchGRS5la5lv3+spSiTv4T0lCCbUpx2UTqY35RkZ3zN6feEPhpvk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_forward_scrub']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-04-02_11:52:43', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-02T13:19:21.292 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-02T13:19:21.294 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-02T13:19:21.340 DEBUG:teuthology.orchestra.run.smithi044:> sudo install -d -m0777 -- /var/run/ceph 2024-04-02T13:19:21.349 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2024-04-02T13:19:21.407 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:21.407 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T13:19:21.430 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T13:19:21.430 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_1 2024-04-02T13:19:21.484 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T13:19:21.484 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T13:19:21.484 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-02T13:19:21.484 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T13:19:21.484 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T13:19:21.484 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-04-02 13:19:11.459989824 +0000 2024-04-02T13:19:21.485 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-04-02 13:16:55.856995268 +0000 2024-04-02T13:19:21.485 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-04-02 13:16:55.856995268 +0000 2024-04-02T13:19:21.485 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-04-02 13:16:55.856995268 +0000 2024-04-02T13:19:21.485 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T13:19:21.549 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-04-02T13:19:21.550 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-04-02T13:19:21.550 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000192579 s, 2.7 MB/s 2024-04-02T13:19:21.551 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T13:19:21.606 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_2 2024-04-02T13:19:21.661 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T13:19:21.661 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T13:19:21.661 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-02T13:19:21.661 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T13:19:21.662 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T13:19:21.662 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-04-02 13:19:11.459989824 +0000 2024-04-02T13:19:21.662 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-04-02 13:16:56.187990320 +0000 2024-04-02T13:19:21.662 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-04-02 13:16:56.187990320 +0000 2024-04-02T13:19:21.662 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-04-02 13:16:56.187990320 +0000 2024-04-02T13:19:21.662 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T13:19:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-04-02T13:19:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-04-02T13:19:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000180515 s, 2.8 MB/s 2024-04-02T13:19:21.728 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T13:19:21.784 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_3 2024-04-02T13:19:21.838 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T13:19:21.838 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T13:19:21.838 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-02T13:19:21.838 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T13:19:21.838 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T13:19:21.838 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-04-02 13:19:11.459989824 +0000 2024-04-02T13:19:21.838 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-04-02 13:16:56.511985477 +0000 2024-04-02T13:19:21.838 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-04-02 13:16:56.511985477 +0000 2024-04-02T13:19:21.838 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-04-02 13:16:56.511985477 +0000 2024-04-02T13:19:21.839 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T13:19:21.902 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-04-02T13:19:21.902 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-04-02T13:19:21.902 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000209369 s, 2.4 MB/s 2024-04-02T13:19:21.903 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T13:19:21.958 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_4 2024-04-02T13:19:22.013 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T13:19:22.013 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T13:19:22.014 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-02T13:19:22.014 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T13:19:22.014 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T13:19:22.014 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-04-02 13:19:11.460989809 +0000 2024-04-02T13:19:22.014 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-04-02 13:16:56.842980528 +0000 2024-04-02T13:19:22.014 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-04-02 13:16:56.842980528 +0000 2024-04-02T13:19:22.014 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-04-02 13:16:56.842980528 +0000 2024-04-02T13:19:22.015 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T13:19:22.079 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-04-02T13:19:22.079 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-04-02T13:19:22.079 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000195102 s, 2.6 MB/s 2024-04-02T13:19:22.080 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T13:19:22.136 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-02T13:19:22.136 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:22.136 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T13:19:22.152 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T13:19:22.152 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2024-04-02T13:19:22.207 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T13:19:22.208 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T13:19:22.208 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-02T13:19:22.208 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T13:19:22.208 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T13:19:22.208 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-02 13:18:52.651265886 +0000 2024-04-02T13:19:22.208 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-02 13:16:54.101690773 +0000 2024-04-02T13:19:22.208 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-02 13:16:54.101690773 +0000 2024-04-02T13:19:22.208 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-02 13:16:54.101690773 +0000 2024-04-02T13:19:22.208 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T13:19:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-02T13:19:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-02T13:19:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000179548 s, 2.9 MB/s 2024-04-02T13:19:22.275 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T13:19:22.331 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2024-04-02T13:19:22.386 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T13:19:22.386 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T13:19:22.386 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-02T13:19:22.386 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T13:19:22.386 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T13:19:22.386 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-02 13:18:52.651265886 +0000 2024-04-02T13:19:22.386 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-02 13:16:54.452682827 +0000 2024-04-02T13:19:22.386 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-02 13:16:54.452682827 +0000 2024-04-02T13:19:22.386 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-02 13:16:54.452682827 +0000 2024-04-02T13:19:22.387 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T13:19:22.451 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-02T13:19:22.452 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-02T13:19:22.452 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.00020758 s, 2.5 MB/s 2024-04-02T13:19:22.453 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T13:19:22.509 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2024-04-02T13:19:22.564 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T13:19:22.564 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T13:19:22.564 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-02T13:19:22.564 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T13:19:22.564 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T13:19:22.565 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-02 13:18:52.652265863 +0000 2024-04-02T13:19:22.565 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-02 13:16:54.786675266 +0000 2024-04-02T13:19:22.565 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-02 13:16:54.786675266 +0000 2024-04-02T13:19:22.565 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-02 13:16:54.786675266 +0000 2024-04-02T13:19:22.566 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T13:19:22.630 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-02T13:19:22.630 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-02T13:19:22.630 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000195962 s, 2.6 MB/s 2024-04-02T13:19:22.631 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T13:19:22.686 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2024-04-02T13:19:22.741 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T13:19:22.741 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T13:19:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-02T13:19:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T13:19:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T13:19:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-02 13:18:52.652265863 +0000 2024-04-02T13:19:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-02 13:16:55.113667862 +0000 2024-04-02T13:19:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-02 13:16:55.113667862 +0000 2024-04-02T13:19:22.741 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-02 13:16:55.113667862 +0000 2024-04-02T13:19:22.742 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T13:19:22.806 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-02T13:19:22.807 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-02T13:19:22.807 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000187586 s, 2.7 MB/s 2024-04-02T13:19:22.808 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T13:19:22.864 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-02T13:19:22.864 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi044.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-02T13:19:22.864 INFO:tasks.ceph:Generating config... 2024-04-02T13:19:22.865 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-02T13:19:22.865 INFO:tasks.ceph:[client] debug client = 20 2024-04-02T13:19:22.865 INFO:tasks.ceph:[client] debug ms = 1 2024-04-02T13:19:22.865 INFO:tasks.ceph:[client] pid file = 2024-04-02T13:19:22.866 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-02T13:19:22.866 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-02T13:19:22.866 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-02T13:19:22.866 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-02T13:19:22.866 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-02T13:19:22.866 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-02T13:19:22.866 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-02T13:19:22.866 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-02T13:19:22.866 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-02T13:19:22.866 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-02T13:19:22.866 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-02T13:19:22.866 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-02T13:19:22.866 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-02T13:19:22.866 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-02T13:19:22.867 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-02T13:19:22.867 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-02T13:19:22.867 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-02T13:19:22.867 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-02T13:19:22.867 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-02T13:19:22.867 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-02T13:19:22.867 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-02T13:19:22.867 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-02T13:19:22.867 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-02T13:19:22.867 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-02T13:19:22.867 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-02T13:19:22.867 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-02T13:19:22.867 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-02T13:19:22.867 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-02T13:19:22.867 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-02T13:19:22.868 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-02T13:19:22.868 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-02T13:19:22.868 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-02T13:19:22.868 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-02T13:19:22.868 INFO:tasks.ceph:Setting up mon.a... 2024-04-02T13:19:22.868 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-02T13:19:22.921 INFO:teuthology.orchestra.run.smithi044.stdout:creating /etc/ceph/ceph.keyring 2024-04-02T13:19:22.924 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-02T13:19:22.970 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-02T13:19:22.996 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.44'), ('mon.b', '172.21.15.69'), ('mon.c', '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]')] 2024-04-02T13:19:22.997 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.44,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-02T13:19:22.998 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:22.998 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-02T13:19:23.052 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.44 --add b 172.21.15.69 --addv c '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:23.136 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:23.137 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool: generated fsid 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:19:23.137 INFO:teuthology.orchestra.run.smithi044.stdout:setting min_mon_release = quincy 2024-04-02T13:19:23.137 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 0 2024-04-02T13:19:23.137 INFO:teuthology.orchestra.run.smithi044.stdout:fsid 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:19:23.137 INFO:teuthology.orchestra.run.smithi044.stdout:last_changed 2024-04-02T13:19:23.136340+0000 2024-04-02T13:19:23.137 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-02T13:19:23.136340+0000 2024-04-02T13:19:23.137 INFO:teuthology.orchestra.run.smithi044.stdout:min_mon_release 17 (quincy) 2024-04-02T13:19:23.137 INFO:teuthology.orchestra.run.smithi044.stdout:election_strategy: 1 2024-04-02T13:19:23.137 INFO:teuthology.orchestra.run.smithi044.stdout:0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-04-02T13:19:23.137 INFO:teuthology.orchestra.run.smithi044.stdout:1: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.b 2024-04-02T13:19:23.137 INFO:teuthology.orchestra.run.smithi044.stdout:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-04-02T13:19:23.137 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-02T13:19:23.139 DEBUG:teuthology.orchestra.run.smithi044:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-02T13:19:23.152 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4101e14a-13fe-452f-8f9a-78614b3ea6ab... 2024-04-02T13:19:23.153 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T13:19:23.195 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T13:19:23.242 INFO:teuthology.orchestra.run.smithi044.stdout:[global] 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: chdir = "" 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: auth supported = cephx 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: filestore xattr use omap = true 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: mon clock drift allowed = 1.000 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: osd crush chooseleaf type = 0 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: auth debug = true 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on old message = true 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on bug = true 2024-04-02T13:19:23.243 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: mon pg warn max object skew = 0 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default size = 2 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow primary affinity = true 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow pg remap = true 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on legacy crush tunables = false 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on crush straw calc version zero = false 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on no sortbitwise = false 2024-04-02T13:19:23.244 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on osd down out interval zero = false 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on too few osds = false 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: mon_allow_pool_size_one = true 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: osd default data pool replay window = 5 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: mon allow pool delete = true 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: mon cluster log file level = debug 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: debug asserts on shutdown = true 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: mon health detail to clog = false 2024-04-02T13:19:23.245 INFO:teuthology.orchestra.run.smithi044.stdout: mon host = "172.21.15.44,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout: fsid = 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout:[osd] 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout: osd journal size = 100 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub load threshold = 5.0 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub max interval = 600 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout: osd mclock profile = high_recovery_ops 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout: osd recover clone overlap = true 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout: osd recovery max chunk = 1048576 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug shutdown = true 2024-04-02T13:19:23.246 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug op order = true 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug verify stray on activate = true 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug trim objects = true 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: osd open classes on start = true 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug pg log writeout = true 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: osd deep scrub update digest min age = 30 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: osd map max advance = 10 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: journal zero on create = true 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: filestore ondisk finisher threads = 3 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: filestore apply finisher threads = 3 2024-04-02T13:19:23.247 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: bdev debug aio = true 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug misdirected ops = true 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore block size = 96636764160 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore fsck on mount = True 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluefs = 20 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluestore = 20 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: debug osd = 20 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: debug rocksdb = 10 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd full ratio = 0.9 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd nearfull ratio = 0.8 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: osd failsafe full ratio = 0.95 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: osd objectstore = bluestore 2024-04-02T13:19:23.248 INFO:teuthology.orchestra.run.smithi044.stdout: osd op complaint time = 180 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout:[mgr] 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout: debug mgr = 20 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout: mgr/telemetry/nag = false 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout: debug client = 20 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout:[mon] 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-04-02T13:19:23.249 INFO:teuthology.orchestra.run.smithi044.stdout: debug paxos = 20 2024-04-02T13:19:23.250 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-04-02T13:19:23.250 INFO:teuthology.orchestra.run.smithi044.stdout: mon data avail warn = 5 2024-04-02T13:19:23.250 INFO:teuthology.orchestra.run.smithi044.stdout: mon mgr mkfs grace = 240 2024-04-02T13:19:23.250 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:19:23.250 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd reporter subtree level = osd 2024-04-02T13:19:23.250 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd prime pg temp = true 2024-04-02T13:19:23.250 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:19:23.250 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.250 INFO:teuthology.orchestra.run.smithi044.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T13:19:23.250 INFO:teuthology.orchestra.run.smithi044.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T13:19:23.250 INFO:teuthology.orchestra.run.smithi044.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T13:19:23.250 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.250 INFO:teuthology.orchestra.run.smithi044.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: # 1m isn't quite enough 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: mon_down_mkfs_grace = 2m 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_filestore_osds = false 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: mon down mkfs grace = 300 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: mon op complaint time = 120 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout:[client] 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: rgw cache enabled = true 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable ops log = true 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable usage log = true 2024-04-02T13:19:23.251 INFO:teuthology.orchestra.run.smithi044.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout: client mount timeout = 600 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout: debug client = 20 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout: pid file = "" 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout: rados mon op timeout = 900 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout: rados osd op timeout = 900 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.a] 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.b] 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.c] 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.a] 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.c] 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.b] 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.d] 2024-04-02T13:19:23.252 INFO:teuthology.orchestra.run.smithi044.stdout:[mds] 2024-04-02T13:19:23.253 INFO:teuthology.orchestra.run.smithi044.stdout: debug mds = 20 2024-04-02T13:19:23.253 INFO:teuthology.orchestra.run.smithi044.stdout: debug mds balancer = 20 2024-04-02T13:19:23.253 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:19:23.253 INFO:teuthology.orchestra.run.smithi044.stdout: mds debug frag = True 2024-04-02T13:19:23.253 INFO:teuthology.orchestra.run.smithi044.stdout: mds debug scatterstat = True 2024-04-02T13:19:23.253 INFO:teuthology.orchestra.run.smithi044.stdout: mds op complaint time = 180 2024-04-02T13:19:23.253 INFO:teuthology.orchestra.run.smithi044.stdout: mds verify scatter = True 2024-04-02T13:19:23.253 INFO:teuthology.orchestra.run.smithi044.stdout: osd op complaint time = 180 2024-04-02T13:19:23.253 INFO:teuthology.orchestra.run.smithi044.stdout: rados mon op timeout = 900 2024-04-02T13:19:23.253 INFO:teuthology.orchestra.run.smithi044.stdout: rados osd op timeout = 900 2024-04-02T13:19:23.254 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-02T13:19:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-04-02T13:19:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T13:19:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-04-02T13:19:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-04-02T13:19:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-04-02T13:19:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-04-02T13:19:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-04-02T13:19:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-04-02T13:19:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-04-02T13:19:23.256 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.44,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-04-02T13:19:23.257 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-04-02T13:19:23.258 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 20 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 20 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 10 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-04-02T13:19:23.259 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.260 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T13:19:23.261 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.262 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T13:19:23.262 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T13:19:23.262 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T13:19:23.262 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.262 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-04-02T13:19:23.262 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-04-02T13:19:23.262 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.262 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-04-02T13:19:23.262 INFO:teuthology.orchestra.run.smithi069.stdout: mon down mkfs grace = 300 2024-04-02T13:19:23.262 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-04-02T13:19:23.262 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:19:23.262 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-04-02T13:19:23.262 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-04-02T13:19:23.263 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-02T13:19:23.264 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-02T13:19:23.266 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-02T13:19:23.266 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-02T13:19:23.361 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-02T13:19:23.361 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:23.361 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-02T13:19:23.375 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:23.375 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-02T13:19:23.430 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:19:23.430 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:23.430 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-02T13:19:23.430 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-02T13:19:23.508 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:23.508 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:23.563 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:19:23.563 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:23.563 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-02T13:19:23.563 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-02T13:19:23.605 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:23.606 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:23.661 INFO:tasks.ceph:Setting up mon nodes... 2024-04-02T13:19:23.661 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-02T13:19:23.661 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-02T13:19:23.716 INFO:teuthology.orchestra.run.smithi044.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-02T13:19:23.719 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-02T13:19:23.789 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-02T13:19:23.791 INFO:tasks.ceph:Setting up mds nodes... 2024-04-02T13:19:23.791 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-02T13:19:23.846 INFO:teuthology.orchestra.run.smithi044.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-02T13:19:23.848 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-02T13:19:23.875 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-02T13:19:23.969 INFO:teuthology.orchestra.run.smithi044.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-02T13:19:23.972 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-02T13:19:23.998 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-02T13:19:24.054 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-02T13:19:24.057 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-02T13:19:24.083 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-02T13:19:24.177 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-02T13:19:24.180 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-02T13:19:24.206 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-02T13:19:24.206 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-02T13:19:24.249 INFO:teuthology.orchestra.run.smithi044.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-02T13:19:24.265 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-02T13:19:24.348 INFO:teuthology.orchestra.run.smithi044.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-02T13:19:24.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-02T13:19:24.406 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-02T13:19:24.421 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-02T13:19:24.504 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-02T13:19:24.520 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-02T13:19:24.520 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi044.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-02T13:19:24.520 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-02T13:19:24.547 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-02T13:19:24.547 INFO:tasks.ceph:role: osd.0 2024-04-02T13:19:24.547 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:19:24.547 DEBUG:teuthology.orchestra.run.smithi044:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-02T13:19:24.622 INFO:teuthology.orchestra.run.smithi044.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T13:19:24.623 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T13:19:24.623 INFO:teuthology.orchestra.run.smithi044.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T13:19:24.623 INFO:teuthology.orchestra.run.smithi044.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T13:19:24.623 INFO:teuthology.orchestra.run.smithi044.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T13:19:24.623 INFO:teuthology.orchestra.run.smithi044.stdout: = sunit=0 swidth=0 blks 2024-04-02T13:19:24.623 INFO:teuthology.orchestra.run.smithi044.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T13:19:24.623 INFO:teuthology.orchestra.run.smithi044.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T13:19:24.623 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T13:19:24.623 INFO:teuthology.orchestra.run.smithi044.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T13:19:25.120 INFO:teuthology.orchestra.run.smithi044.stdout:Discarding blocks...Done. 2024-04-02T13:19:25.122 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi044.front.sepia.ceph.com -o noatime 2024-04-02T13:19:25.122 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-02T13:19:25.257 DEBUG:teuthology.orchestra.run.smithi044:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-02T13:19:25.288 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-02T13:19:25.355 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-02T13:19:25.355 INFO:tasks.ceph:role: osd.1 2024-04-02T13:19:25.355 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:19:25.356 DEBUG:teuthology.orchestra.run.smithi044:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-02T13:19:25.429 INFO:teuthology.orchestra.run.smithi044.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T13:19:25.430 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T13:19:25.430 INFO:teuthology.orchestra.run.smithi044.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T13:19:25.430 INFO:teuthology.orchestra.run.smithi044.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T13:19:25.430 INFO:teuthology.orchestra.run.smithi044.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T13:19:25.430 INFO:teuthology.orchestra.run.smithi044.stdout: = sunit=0 swidth=0 blks 2024-04-02T13:19:25.430 INFO:teuthology.orchestra.run.smithi044.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T13:19:25.430 INFO:teuthology.orchestra.run.smithi044.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T13:19:25.430 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T13:19:25.430 INFO:teuthology.orchestra.run.smithi044.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T13:19:25.880 INFO:teuthology.orchestra.run.smithi044.stdout:Discarding blocks...Done. 2024-04-02T13:19:25.882 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi044.front.sepia.ceph.com -o noatime 2024-04-02T13:19:25.882 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-02T13:19:25.975 DEBUG:teuthology.orchestra.run.smithi044:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-02T13:19:26.007 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-02T13:19:26.074 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-02T13:19:26.074 INFO:tasks.ceph:role: osd.2 2024-04-02T13:19:26.074 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:19:26.074 DEBUG:teuthology.orchestra.run.smithi044:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-02T13:19:26.149 INFO:teuthology.orchestra.run.smithi044.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T13:19:26.149 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T13:19:26.149 INFO:teuthology.orchestra.run.smithi044.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T13:19:26.149 INFO:teuthology.orchestra.run.smithi044.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T13:19:26.149 INFO:teuthology.orchestra.run.smithi044.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T13:19:26.149 INFO:teuthology.orchestra.run.smithi044.stdout: = sunit=0 swidth=0 blks 2024-04-02T13:19:26.149 INFO:teuthology.orchestra.run.smithi044.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T13:19:26.149 INFO:teuthology.orchestra.run.smithi044.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T13:19:26.149 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T13:19:26.149 INFO:teuthology.orchestra.run.smithi044.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T13:19:26.644 INFO:teuthology.orchestra.run.smithi044.stdout:Discarding blocks...Done. 2024-04-02T13:19:26.646 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi044.front.sepia.ceph.com -o noatime 2024-04-02T13:19:26.646 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-02T13:19:26.774 DEBUG:teuthology.orchestra.run.smithi044:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-02T13:19:26.807 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-02T13:19:26.875 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-02T13:19:26.875 INFO:tasks.ceph:role: osd.3 2024-04-02T13:19:26.875 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:19:26.875 DEBUG:teuthology.orchestra.run.smithi044:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-02T13:19:26.949 INFO:teuthology.orchestra.run.smithi044.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T13:19:26.950 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T13:19:26.950 INFO:teuthology.orchestra.run.smithi044.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T13:19:26.950 INFO:teuthology.orchestra.run.smithi044.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T13:19:26.950 INFO:teuthology.orchestra.run.smithi044.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T13:19:26.950 INFO:teuthology.orchestra.run.smithi044.stdout: = sunit=0 swidth=0 blks 2024-04-02T13:19:26.950 INFO:teuthology.orchestra.run.smithi044.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T13:19:26.950 INFO:teuthology.orchestra.run.smithi044.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T13:19:26.950 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T13:19:26.950 INFO:teuthology.orchestra.run.smithi044.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T13:19:27.471 INFO:teuthology.orchestra.run.smithi044.stdout:Discarding blocks...Done. 2024-04-02T13:19:27.473 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi044.front.sepia.ceph.com -o noatime 2024-04-02T13:19:27.473 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-02T13:19:27.599 DEBUG:teuthology.orchestra.run.smithi044:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-02T13:19:27.635 DEBUG:teuthology.orchestra.run.smithi044:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:27.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:27.736+0000 7f5999b06640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-02T13:19:27.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:27.736+0000 7f5999b06640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-02T13:19:27.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:27.736+0000 7f5999b06640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-02T13:19:27.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:27.736+0000 7f5999b06640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-02T13:19:27.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:27.736+0000 7f5999b06640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-02T13:19:27.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:27.737+0000 7f5999b06640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-02T13:19:30.582 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-02T13:19:30.613 DEBUG:teuthology.orchestra.run.smithi044:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:30.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:30.701+0000 7f517552a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-02T13:19:30.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:30.701+0000 7f517552a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-02T13:19:30.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:30.701+0000 7f517552a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-02T13:19:30.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:30.701+0000 7f517552a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-02T13:19:30.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:30.701+0000 7f517552a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-02T13:19:30.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:30.702+0000 7f517552a640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-02T13:19:33.557 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-02T13:19:33.589 DEBUG:teuthology.orchestra.run.smithi044:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:33.678+0000 7f3fe3217640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-02T13:19:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:33.678+0000 7f3fe3217640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-02T13:19:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:33.678+0000 7f3fe3217640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-02T13:19:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:33.678+0000 7f3fe3217640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-02T13:19:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:33.678+0000 7f3fe3217640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-02T13:19:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:33.679+0000 7f3fe3217640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-02T13:19:36.539 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-02T13:19:36.570 DEBUG:teuthology.orchestra.run.smithi044:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:36.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:36.658+0000 7f304e4bd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-02T13:19:36.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:36.658+0000 7f304e4bd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-02T13:19:36.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:36.658+0000 7f304e4bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-02T13:19:36.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:36.658+0000 7f304e4bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-02T13:19:36.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:36.658+0000 7f304e4bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-02T13:19:36.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:36.659+0000 7f304e4bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-02T13:19:39.509 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-02T13:19:39.540 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-02T13:19:39.573 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-02T13:19:39.573 INFO:tasks.ceph:role: osd.4 2024-04-02T13:19:39.574 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:19:39.574 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-02T13:19:39.650 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T13:19:39.651 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T13:19:39.651 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T13:19:39.651 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T13:19:39.651 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T13:19:39.651 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-04-02T13:19:39.651 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T13:19:39.651 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T13:19:39.651 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T13:19:39.651 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T13:19:40.157 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-04-02T13:19:40.159 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-04-02T13:19:40.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-02T13:19:40.289 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-02T13:19:40.326 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-02T13:19:40.395 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-02T13:19:40.395 INFO:tasks.ceph:role: osd.5 2024-04-02T13:19:40.395 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:19:40.395 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-02T13:19:40.471 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T13:19:40.471 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T13:19:40.471 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T13:19:40.471 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T13:19:40.471 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T13:19:40.471 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-04-02T13:19:40.471 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T13:19:40.471 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T13:19:40.471 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T13:19:40.472 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T13:19:40.922 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-04-02T13:19:40.924 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-04-02T13:19:40.924 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-02T13:19:41.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-02T13:19:41.049 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-02T13:19:41.116 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-02T13:19:41.116 INFO:tasks.ceph:role: osd.6 2024-04-02T13:19:41.117 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:19:41.117 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-02T13:19:41.190 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T13:19:41.190 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T13:19:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T13:19:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T13:19:41.191 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T13:19:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-04-02T13:19:41.191 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T13:19:41.191 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T13:19:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T13:19:41.191 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T13:19:41.673 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-04-02T13:19:41.675 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-04-02T13:19:41.675 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-02T13:19:41.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-02T13:19:41.815 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-02T13:19:41.884 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-02T13:19:41.884 INFO:tasks.ceph:role: osd.7 2024-04-02T13:19:41.884 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:19:41.884 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-02T13:19:41.959 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T13:19:41.959 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T13:19:41.959 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T13:19:41.959 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T13:19:41.959 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T13:19:41.959 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-04-02T13:19:41.960 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T13:19:41.960 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T13:19:41.960 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T13:19:41.960 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T13:19:42.483 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-04-02T13:19:42.485 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-04-02T13:19:42.485 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-02T13:19:42.626 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-02T13:19:42.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:42.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:42.771+0000 7fbfbe020640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-02T13:19:42.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:42.772+0000 7fbfbe020640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-02T13:19:42.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:42.772+0000 7fbfbe020640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-02T13:19:42.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:42.772+0000 7fbfbe020640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-02T13:19:42.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:42.772+0000 7fbfbe020640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-02T13:19:42.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:42.772+0000 7fbfbe020640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-02T13:19:45.627 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-02T13:19:45.658 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:45.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:45.749+0000 7f3e15506640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-02T13:19:45.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:45.750+0000 7f3e15506640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-02T13:19:45.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:45.750+0000 7f3e15506640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-02T13:19:45.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:45.750+0000 7f3e15506640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-02T13:19:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:45.750+0000 7f3e15506640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-02T13:19:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:45.750+0000 7f3e15506640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-02T13:19:48.606 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-02T13:19:48.637 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:48.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:48.727+0000 7fcc7e01b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-02T13:19:48.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:48.728+0000 7fcc7e01b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-02T13:19:48.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:48.728+0000 7fcc7e01b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-02T13:19:48.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:48.728+0000 7fcc7e01b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-02T13:19:48.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:48.728+0000 7fcc7e01b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-02T13:19:48.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:48.728+0000 7fcc7e01b640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-02T13:19:51.293 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:19:51.296 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:19:51.586 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-02T13:19:51.617 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:51.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:51.707+0000 7f9f6bcb5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-02T13:19:51.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:51.708+0000 7f9f6bcb5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-02T13:19:51.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:51.708+0000 7f9f6bcb5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-02T13:19:51.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:51.708+0000 7f9f6bcb5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-02T13:19:51.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:51.708+0000 7f9f6bcb5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-02T13:19:51.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:19:51.708+0000 7f9f6bcb5640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-02T13:19:54.566 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-02T13:19:54.597 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-02T13:19:54.597 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:54.597 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-02T13:19:54.629 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:54.630 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-02T13:19:54.701 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:54.701 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-02T13:19:54.769 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:54.770 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-02T13:19:54.838 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:54.839 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-02T13:19:54.907 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:54.907 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-02T13:19:54.975 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:54.975 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-02T13:19:55.042 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:55.042 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-02T13:19:55.075 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:55.075 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-02T13:19:55.144 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:55.144 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-02T13:19:55.213 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:55.213 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-02T13:19:55.282 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:55.282 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-02T13:19:55.350 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:55.351 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-02T13:19:55.418 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:55.418 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-02T13:19:55.486 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:55.486 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-02T13:19:55.503 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:19:55.503 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-02T13:19:55.559 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:55.559 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-02T13:19:55.575 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:19:55.575 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-02T13:19:55.630 INFO:tasks.ceph:Adding keys to all mons... 2024-04-02T13:19:55.630 DEBUG:teuthology.orchestra.run.smithi044:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-02T13:19:55.632 DEBUG:teuthology.orchestra.run.smithi069:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-02T13:19:55.673 INFO:teuthology.orchestra.run.smithi044.stdout:[mgr.y] 2024-04-02T13:19:55.673 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDbBQxmxoquKhAAE5fMnJIuCy5hGJ1aEAx8Vg== 2024-04-02T13:19:55.673 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.a] 2024-04-02T13:19:55.673 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDbBQxmVhZrMhAAklqx5uKCFA5kiqYwyLBKpg== 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.c] 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDbBQxm8BLKORAAjMIrmdQ/bXbPk2sAnYJWTw== 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.0] 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDfBQxm4GP9KxAA/AmOunAfVaczAWRGVXvFVg== 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.1] 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDiBQxmDL/nKRAADF2C30+IGAXL9XwDQDKCUA== 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.2] 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDlBQxm1gOCKBAAuYxN9jNHuRYEqc8WrsYErQ== 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.3] 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDoBQxmLLFQJxAAFMCzphGU7TOt4nC0Nt+y+g== 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout:[mgr.x] 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDbBQxmR5ICLxAAtQeg/gq0OAq6e0mroBZVxQ== 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.b] 2024-04-02T13:19:55.674 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcBQxmalQ+AxAA2SqH9CX/66OIXR7SDID97w== 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.d] 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcBQxmNoaRChAAEqxWmBjBWG79Tkj6dU8FQg== 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.4] 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDuBQxmtk8QLhAAR4gm0irR6C3D2SkhFjvDKQ== 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.5] 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDxBQxm6Ee/LBAAX5h+9ML70ztw2QH/50OzOg== 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.6] 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQD0BQxm35hwKxAAvsgMmiXdLo+c7wlvb9RtVQ== 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.7] 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQD3BQxmYLBOKhAA1vuSOBDcFf2zT78eoTHtHQ== 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A== 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout:[client.2] 2024-04-02T13:19:55.675 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcBQxmKNS5FBAAKvsCNmuG+fI1D/vwu6MC5w== 2024-04-02T13:19:55.676 INFO:teuthology.orchestra.run.smithi044.stdout:[client.1] 2024-04-02T13:19:55.676 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcBQxm3sYsGBAAeRCNbP5qiQ8YnRVsDoaaFQ== 2024-04-02T13:19:55.676 INFO:teuthology.orchestra.run.smithi044.stdout:[client.3] 2024-04-02T13:19:55.676 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcBQxmslsSHhAAPsAfALW7RimJDyXdOa2wbQ== 2024-04-02T13:19:55.695 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.y] 2024-04-02T13:19:55.695 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDbBQxmxoquKhAAE5fMnJIuCy5hGJ1aEAx8Vg== 2024-04-02T13:19:55.695 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDbBQxmVhZrMhAAklqx5uKCFA5kiqYwyLBKpg== 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDbBQxm8BLKORAAjMIrmdQ/bXbPk2sAnYJWTw== 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDfBQxm4GP9KxAA/AmOunAfVaczAWRGVXvFVg== 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDiBQxmDL/nKRAADF2C30+IGAXL9XwDQDKCUA== 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDlBQxm1gOCKBAAuYxN9jNHuRYEqc8WrsYErQ== 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.3] 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDoBQxmLLFQJxAAFMCzphGU7TOt4nC0Nt+y+g== 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.x] 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDbBQxmR5ICLxAAtQeg/gq0OAq6e0mroBZVxQ== 2024-04-02T13:19:55.696 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDcBQxmalQ+AxAA2SqH9CX/66OIXR7SDID97w== 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDcBQxmNoaRChAAEqxWmBjBWG79Tkj6dU8FQg== 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.4] 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDuBQxmtk8QLhAAR4gm0irR6C3D2SkhFjvDKQ== 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.5] 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDxBQxm6Ee/LBAAX5h+9ML70ztw2QH/50OzOg== 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.6] 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQD0BQxm35hwKxAAvsgMmiXdLo+c7wlvb9RtVQ== 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.7] 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQD3BQxmYLBOKhAA1vuSOBDcFf2zT78eoTHtHQ== 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A== 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout:[client.2] 2024-04-02T13:19:55.697 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDcBQxmKNS5FBAAKvsCNmuG+fI1D/vwu6MC5w== 2024-04-02T13:19:55.698 INFO:teuthology.orchestra.run.smithi069.stdout:[client.1] 2024-04-02T13:19:55.698 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDcBQxm3sYsGBAAeRCNbP5qiQ8YnRVsDoaaFQ== 2024-04-02T13:19:55.698 INFO:teuthology.orchestra.run.smithi069.stdout:[client.3] 2024-04-02T13:19:55.698 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDcBQxmslsSHhAAPsAfALW7RimJDyXdOa2wbQ== 2024-04-02T13:19:55.699 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-02T13:19:55.719 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-02T13:19:55.791 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T13:19:55.793 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T13:19:55.849 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T13:19:55.852 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T13:19:55.906 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:55.908 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:55.960 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:55.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:56.014 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:56.016 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:56.067 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:56.069 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:56.121 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-02T13:19:56.123 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-02T13:19:56.175 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T13:19:56.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T13:19:56.233 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T13:19:56.236 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T13:19:56.290 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:56.292 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:56.344 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:56.346 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:56.398 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:56.400 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:56.452 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:56.455 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T13:19:56.506 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T13:19:56.508 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T13:19:56.564 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T13:19:56.566 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T13:19:56.622 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T13:19:56.624 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T13:19:56.679 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T13:19:56.681 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T13:19:56.736 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-02T13:19:56.737 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-02T13:19:56.763 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-02T13:19:56.862 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-02T13:19:56.888 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-02T13:19:56.915 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-02T13:19:57.014 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-02T13:19:57.041 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-02T13:19:57.108 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-02T13:19:57.205 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-02T13:19:57.232 DEBUG:teuthology.orchestra.run.smithi044:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:57.234 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-02T13:19:57.287 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-02T13:19:57.287 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-02T13:19:57.287 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-02T13:19:57.290 INFO:tasks.ceph.mon.a:Started 2024-04-02T13:19:57.290 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-02T13:19:57.290 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-02T13:19:57.329 INFO:tasks.ceph.mon.b:Started 2024-04-02T13:19:57.329 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-02T13:19:57.330 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-02T13:19:57.331 INFO:tasks.ceph.mon.c:Started 2024-04-02T13:19:57.332 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-02T13:19:57.332 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-02T13:19:57.332 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-02T13:19:57.334 INFO:tasks.ceph.mgr.y:Started 2024-04-02T13:19:57.334 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-02T13:19:57.334 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-02T13:19:57.336 INFO:tasks.ceph.mgr.x:Started 2024-04-02T13:19:57.336 INFO:tasks.ceph:Setting crush tunables to default 2024-04-02T13:19:57.337 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --cluster ceph osd crush tunables default 2024-04-02T13:19:57.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.436+0000 7ff97f732640 1 Processor -- start 2024-04-02T13:19:57.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.437+0000 7ff97f732640 1 -- start start 2024-04-02T13:19:57.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.437+0000 7ff97f732640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:19:57.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.437+0000 7ff97f732640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff97815c580 con 0x7ff97815c1a0 2024-04-02T13:19:57.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.437+0000 7ff97f732640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9780d01b0 con 0x7ff9780cfd60 2024-04-02T13:19:57.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.437+0000 7ff97f732640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff978070ad0 con 0x7ff9780d21c0 2024-04-02T13:19:57.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.437+0000 7ff97d4ce640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff97815c1a0 0x7ff9780d1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:34070/0 (socket says 172.21.15.44:34070) 2024-04-02T13:19:57.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.437+0000 7ff97d4ce640 1 -- 172.21.15.44:0/2175603398 learned_addr learned my addr 172.21.15.44:0/2175603398 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:19:57.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.438+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:19:57.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.438+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 msgr2=0x7ff97806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-02T13:19:57.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.438+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 msgr2=0x7ff97806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T13:19:57.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.438+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T13:19:57.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.438+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T13:19:57.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.639+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:19:57.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.639+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 msgr2=0x7ff97806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-02T13:19:57.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.639+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 msgr2=0x7ff97806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T13:19:57.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.639+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T13:19:57.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:57.639+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T13:19:58.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:58.040+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:19:58.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:58.040+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 msgr2=0x7ff97806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-02T13:19:58.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:58.040+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 msgr2=0x7ff97806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T13:19:58.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:58.040+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T13:19:58.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:58.040+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T13:19:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:58.841+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:19:58.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:58.841+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 msgr2=0x7ff97806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-02T13:19:58.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:58.841+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 msgr2=0x7ff97806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T13:19:58.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:58.841+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T13:19:58.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:19:58.841+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T13:20:00.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff976ffd640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 msgr2=0x7ff97806f370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T13:20:00.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff976ffd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.69:3300/0 conn(0x7ff9780d21c0 0x7ff97806f370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:00.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff976ffd640 1 -- 172.21.15.44:0/2175603398 >> v1:172.21.15.69:6789/0 conn(0x7ff9780cfd60 legacy=0x7ff9780cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:00.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff976ffd640 1 -- 172.21.15.44:0/2175603398 >> v1:172.21.15.44:6789/0 conn(0x7ff97815c1a0 legacy=0x7ff9780d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:00.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff976ffd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:00.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff976ffd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:00.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff976ffd640 1 -- 172.21.15.44:0/2175603398 --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff978070ad0 con 0x7ff9640036e0 2024-04-02T13:20:00.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff976ffd640 1 -- 172.21.15.44:0/2175603398 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff964008d20 con 0x7ff964006180 2024-04-02T13:20:00.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:00.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff97d4ce640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 msgr2=0x7ff964005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-02T13:20:00.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff97d4ce640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 msgr2=0x7ff964005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T13:20:00.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T13:20:00.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T13:20:00.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:00.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 msgr2=0x7ff964008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-02T13:20:00.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 msgr2=0x7ff964008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T13:20:00.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T13:20:00.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.438+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T13:20:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.639+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.639+0000 7ff97d4ce640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 msgr2=0x7ff964005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-02T13:20:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.639+0000 7ff97d4ce640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 msgr2=0x7ff964005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T13:20:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.639+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T13:20:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.639+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T13:20:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.639+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.639+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 msgr2=0x7ff964008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-02T13:20:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.639+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 msgr2=0x7ff964008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T13:20:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.639+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T13:20:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:00.639+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T13:20:01.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.040+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:01.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.040+0000 7ff97d4ce640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 msgr2=0x7ff964005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-02T13:20:01.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.040+0000 7ff97d4ce640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 msgr2=0x7ff964005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T13:20:01.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.040+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T13:20:01.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.040+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T13:20:01.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.040+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:01.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.040+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 msgr2=0x7ff964008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-02T13:20:01.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.040+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 msgr2=0x7ff964008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T13:20:01.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.040+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T13:20:01.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.040+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T13:20:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.841+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.841+0000 7ff97d4ce640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 msgr2=0x7ff964005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-02T13:20:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.841+0000 7ff97d4ce640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 msgr2=0x7ff964005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T13:20:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.841+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T13:20:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.841+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T13:20:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.841+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.841+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 msgr2=0x7ff964008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-02T13:20:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.841+0000 7ff97cccd640 1 -- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 msgr2=0x7ff964008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T13:20:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.841+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T13:20:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:01.841+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T13:20:03.069 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:03.068+0000 7f4d38a42040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T13:20:03.086 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:03.085+0000 7fe6c6e32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T13:20:03.122 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:03.121+0000 7f4d38a42040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T13:20:03.141 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:03.140+0000 7fe6c6e32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T13:20:03.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff976ffd640 1 -- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 msgr2=0x7ff964008600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T13:20:03.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff976ffd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff964006180 0x7ff964008600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:03.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff976ffd640 1 -- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 msgr2=0x7ff964005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff976ffd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff976ffd640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff97dccf640 1 -- 172.21.15.44:0/2175603398 reap_dead start 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff976ffd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9640065f0 0x7ff964009180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff976ffd640 1 -- 172.21.15.44:0/2175603398 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff97806fa30 con 0x7ff9640096c0 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff976ffd640 1 -- 172.21.15.44:0/2175603398 --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff978070ad0 con 0x7ff9640036e0 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff976ffd640 1 -- 172.21.15.44:0/2175603398 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff964008d20 con 0x7ff9640065f0 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> v2:172.21.15.44:3300/0 conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9640065f0 0x7ff964009180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff97d4ce640 1 -- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9640065f0 msgr2=0x7ff964009180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9640065f0 0x7ff964009180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff97d4ce640 1 -- 172.21.15.44:0/2175603398 >> v1:172.21.15.69:6789/0 conn(0x7ff9640096c0 legacy=0x7ff96400bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.438+0000 7ff97d4ce640 1 -- 172.21.15.44:0/2175603398 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff96400e4e0 con 0x7ff9640036e0 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.439+0000 7ff97d4ce640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9640036e0 0x7ff964005b40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff970007d20 tx=0x7ff97002c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69070265724255d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:03.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.439+0000 7ff977fff640 1 -- 172.21.15.44:0/2175603398 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97003a070 con 0x7ff9640036e0 2024-04-02T13:20:03.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.439+0000 7ff977fff640 1 -- 172.21.15.44:0/2175603398 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff970035410 con 0x7ff9640036e0 2024-04-02T13:20:03.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.439+0000 7ff977fff640 1 -- 172.21.15.44:0/2175603398 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9700358a0 con 0x7ff9640036e0 2024-04-02T13:20:03.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.440+0000 7ff97f732640 1 -- 172.21.15.44:0/2175603398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9640036e0 msgr2=0x7ff964005b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:03.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.440+0000 7ff97f732640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9640036e0 0x7ff964005b40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff970007d20 tx=0x7ff97002c480 comp rx=0 tx=0).stop 2024-04-02T13:20:03.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.440+0000 7ff97f732640 1 -- 172.21.15.44:0/2175603398 shutdown_connections 2024-04-02T13:20:03.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.440+0000 7ff97f732640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9640065f0 0x7ff964009180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:03.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.440+0000 7ff97f732640 1 --2- 172.21.15.44:0/2175603398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9640036e0 0x7ff964005b40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:03.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.440+0000 7ff97f732640 1 -- 172.21.15.44:0/2175603398 >> 172.21.15.44:0/2175603398 conn(0x7ff97809aa20 msgr2=0x7ff97809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:03.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.440+0000 7ff97f732640 1 -- 172.21.15.44:0/2175603398 shutdown_connections 2024-04-02T13:20:03.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.440+0000 7ff97f732640 1 -- 172.21.15.44:0/2175603398 wait complete. 2024-04-02T13:20:03.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.441+0000 7ff97f732640 1 Processor -- start 2024-04-02T13:20:03.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.441+0000 7ff97f732640 1 -- start start 2024-04-02T13:20:03.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.442+0000 7ff97f732640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9640036e0 0x7ff9781f54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:03.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.442+0000 7ff97f732640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9640065f0 0x7ff9780c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:03.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.442+0000 7ff97f732640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9640096c0 0x7ff9780cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:03.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.442+0000 7ff97f732640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff9780702d0 con 0x7ff9640065f0 2024-04-02T13:20:03.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.442+0000 7ff97f732640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff97806fff0 con 0x7ff9640096c0 2024-04-02T13:20:03.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.442+0000 7ff97f732640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff9780705b0 con 0x7ff9640036e0 2024-04-02T13:20:03.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.442+0000 7ff97cccd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9640065f0 0x7ff9780c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:03.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.442+0000 7ff97cccd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9640065f0 0x7ff9780c9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40060/0 (socket says 172.21.15.44:40060) 2024-04-02T13:20:03.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.442+0000 7ff97d4ce640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9640036e0 0x7ff9781f54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:03.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.442+0000 7ff97cccd640 1 -- 172.21.15.44:0/2009882777 learned_addr learned my addr 172.21.15.44:0/2009882777 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:03.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.442+0000 7ff97dccf640 1 --2- 172.21.15.44:0/2009882777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9640096c0 0x7ff9780cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:03.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.443+0000 7ff97cccd640 1 -- 172.21.15.44:0/2009882777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9640036e0 msgr2=0x7ff9781f54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:03.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.443+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2009882777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9640036e0 0x7ff9781f54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:03.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.443+0000 7ff97cccd640 1 -- 172.21.15.44:0/2009882777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9640096c0 msgr2=0x7ff9780cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:03.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.443+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2009882777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9640096c0 0x7ff9780cb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:03.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.443+0000 7ff97cccd640 1 -- 172.21.15.44:0/2009882777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9780cbd30 con 0x7ff9640065f0 2024-04-02T13:20:03.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.443+0000 7ff97cccd640 1 --2- 172.21.15.44:0/2009882777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9640065f0 0x7ff9780c9140 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff96c018ad0 tx=0x7ff96c018f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:03.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.443+0000 7ff975ffb640 1 -- 172.21.15.44:0/2009882777 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c016da0 con 0x7ff9640065f0 2024-04-02T13:20:03.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.443+0000 7ff975ffb640 1 -- 172.21.15.44:0/2009882777 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff96c028070 con 0x7ff9640065f0 2024-04-02T13:20:03.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.443+0000 7ff97f732640 1 -- 172.21.15.44:0/2009882777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff97806c680 con 0x7ff9640065f0 2024-04-02T13:20:03.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.443+0000 7ff975ffb640 1 -- 172.21.15.44:0/2009882777 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c023620 con 0x7ff9640065f0 2024-04-02T13:20:03.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.443+0000 7ff97f732640 1 -- 172.21.15.44:0/2009882777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff97806cf20 con 0x7ff9640065f0 2024-04-02T13:20:03.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.444+0000 7ff975ffb640 1 -- 172.21.15.44:0/2009882777 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff96c022070 con 0x7ff9640065f0 2024-04-02T13:20:03.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.444+0000 7ff975ffb640 1 -- 172.21.15.44:0/2009882777 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff96c006020 con 0x7ff9640065f0 2024-04-02T13:20:03.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.444+0000 7ff97f732640 1 -- 172.21.15.44:0/2009882777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff978068c00 con 0x7ff9640065f0 2024-04-02T13:20:03.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.446+0000 7ff975ffb640 1 -- 172.21.15.44:0/2009882777 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff96c031020 con 0x7ff9640065f0 2024-04-02T13:20:03.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.513+0000 7ff97f732640 1 -- 172.21.15.44:0/2009882777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7ff9780cd740 con 0x7ff9640065f0 2024-04-02T13:20:03.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.515+0000 7ff975ffb640 1 -- 172.21.15.44:0/2009882777 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7ff96c020040 con 0x7ff9640065f0 2024-04-02T13:20:03.516 INFO:teuthology.orchestra.run.smithi044.stderr:adjusted tunables profile to default 2024-04-02T13:20:03.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.516+0000 7ff97f732640 1 -- 172.21.15.44:0/2009882777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9640065f0 msgr2=0x7ff9780c9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:03.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.516+0000 7ff97f732640 1 --2- 172.21.15.44:0/2009882777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9640065f0 0x7ff9780c9140 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff96c018ad0 tx=0x7ff96c018f90 comp rx=0 tx=0).stop 2024-04-02T13:20:03.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.516+0000 7ff97f732640 1 -- 172.21.15.44:0/2009882777 shutdown_connections 2024-04-02T13:20:03.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.516+0000 7ff97f732640 1 --2- 172.21.15.44:0/2009882777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9640096c0 0x7ff9780cb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:03.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.516+0000 7ff97f732640 1 --2- 172.21.15.44:0/2009882777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9640065f0 0x7ff9780c9140 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:03.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.516+0000 7ff97f732640 1 --2- 172.21.15.44:0/2009882777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9640036e0 0x7ff9781f54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:03.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.516+0000 7ff97f732640 1 -- 172.21.15.44:0/2009882777 >> 172.21.15.44:0/2009882777 conn(0x7ff97809aa20 msgr2=0x7ff978151b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:03.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.516+0000 7ff97f732640 1 -- 172.21.15.44:0/2009882777 shutdown_connections 2024-04-02T13:20:03.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:03.516+0000 7ff97f732640 1 -- 172.21.15.44:0/2009882777 wait complete. 2024-04-02T13:20:03.528 INFO:tasks.ceph:check_enable_crimson: False 2024-04-02T13:20:03.528 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-02T13:20:03.528 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:20:03.528 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-02T13:20:03.555 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:20:03.555 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-02T13:20:03.624 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:20:03.624 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-02T13:20:03.659 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:03.658+0000 7f4d38a42040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T13:20:03.691 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:20:03.691 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-02T13:20:03.698 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:03.698+0000 7fe6c6e32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T13:20:03.713 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:03.712+0000 7f4d38a42040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T13:20:03.724 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:20:03.725 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-02T13:20:03.752 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:20:03.752 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-02T13:20:03.756 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:03.755+0000 7fe6c6e32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T13:20:03.761 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:03.760+0000 7f4d38a42040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T13:20:03.788 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:20:03.788 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-02T13:20:03.802 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:03.802+0000 7fe6c6e32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T13:20:03.818 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:03.817+0000 7f4d38a42040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T13:20:03.844 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:20:03.845 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-02T13:20:03.858 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:03.857+0000 7fe6c6e32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T13:20:03.863 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:03.862+0000 7f4d38a42040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T13:20:03.889 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 30e26323-bca4-4e6a-876d-3760fd190851 0 2024-04-02T13:20:03.902 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:03.901+0000 7fe6c6e32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T13:20:03.986 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:03.984+0000 7f4d38a42040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T13:20:04.023 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:04.022+0000 7fe6c6e32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T13:20:04.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.028+0000 7f767e40d640 1 Processor -- start 2024-04-02T13:20:04.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.028+0000 7f767e40d640 1 -- start start 2024-04-02T13:20:04.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f767e40d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f76780cacd0 0x7f7678165400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f767e40d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76780ceb80 con 0x7f76780af210 2024-04-02T13:20:04.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f767e40d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76780cee60 con 0x7f767815a960 2024-04-02T13:20:04.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f767e40d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f767815ad40 con 0x7f76780cacd0 2024-04-02T13:20:04.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f7677fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f767815a960 0x7f76780ce280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48664/0 (socket says 172.21.15.69:48664) 2024-04-02T13:20:04.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f7677fff640 1 -- 172.21.15.69:0/4123592809 learned_addr learned my addr 172.21.15.69:0/4123592809 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-02T13:20:04.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f767c9aa640 1 --2- 172.21.15.69:0/4123592809 >> v2:172.21.15.44:3300/0 conn(0x7f76780cacd0 0x7f7678165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f7676ffd640 1 -- 172.21.15.69:0/4123592809 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 797275983 0 0) 0x7f76780ceb80 con 0x7f76780af210 2024-04-02T13:20:04.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f7676ffd640 1 -- 172.21.15.69:0/4123592809 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7660003520 con 0x7f76780af210 2024-04-02T13:20:04.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f7676ffd640 1 -- 172.21.15.69:0/4123592809 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2578525873 0 0) 0x7f76780cee60 con 0x7f767815a960 2024-04-02T13:20:04.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f7676ffd640 1 -- 172.21.15.69:0/4123592809 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76780ceb80 con 0x7f767815a960 2024-04-02T13:20:04.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f767c9aa640 1 -- 172.21.15.69:0/4123592809 >> v1:172.21.15.69:6789/0 conn(0x7f767815a960 legacy=0x7f76780ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f767c9aa640 1 -- 172.21.15.69:0/4123592809 >> v1:172.21.15.44:6789/0 conn(0x7f76780af210 legacy=0x7f76780ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.029+0000 7f767c9aa640 1 -- 172.21.15.69:0/4123592809 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76780cf140 con 0x7f76780cacd0 2024-04-02T13:20:04.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.030+0000 7f767c9aa640 1 --2- 172.21.15.69:0/4123592809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76780cacd0 0x7f7678165400 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f767000aea0 tx=0x7f767002d0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a32be3f1a0572cdf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:04.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.030+0000 7f7676ffd640 1 -- 172.21.15.69:0/4123592809 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767002dc70 con 0x7f76780cacd0 2024-04-02T13:20:04.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.030+0000 7f7676ffd640 1 -- 172.21.15.69:0/4123592809 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f767002de10 con 0x7f76780cacd0 2024-04-02T13:20:04.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.030+0000 7f7676ffd640 1 -- 172.21.15.69:0/4123592809 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7670008660 con 0x7f76780cacd0 2024-04-02T13:20:04.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.031+0000 7f767e40d640 1 -- 172.21.15.69:0/4123592809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76780cacd0 msgr2=0x7f7678165400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.031+0000 7f767e40d640 1 --2- 172.21.15.69:0/4123592809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76780cacd0 0x7f7678165400 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f767000aea0 tx=0x7f767002d0d0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.031+0000 7f767e40d640 1 -- 172.21.15.69:0/4123592809 shutdown_connections 2024-04-02T13:20:04.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.031+0000 7f767e40d640 1 --2- 172.21.15.69:0/4123592809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76780cacd0 0x7f7678165400 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.031+0000 7f767e40d640 1 -- 172.21.15.69:0/4123592809 >> 172.21.15.69:0/4123592809 conn(0x7f76780cfca0 msgr2=0x7f76780d0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:04.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.031+0000 7f767e40d640 1 -- 172.21.15.69:0/4123592809 shutdown_connections 2024-04-02T13:20:04.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.031+0000 7f767e40d640 1 -- 172.21.15.69:0/4123592809 wait complete. 2024-04-02T13:20:04.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.032+0000 7f767e40d640 1 Processor -- start 2024-04-02T13:20:04.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.032+0000 7f767e40d640 1 -- start start 2024-04-02T13:20:04.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f767e40d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76780af210 0x7f76780b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f767e40d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76780cacd0 0x7f76780b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f767e40d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f767815a960 0x7f767814d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f767e40d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f76780ae2c0 con 0x7f76780af210 2024-04-02T13:20:04.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f767e40d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f76780ae140 con 0x7f76780cacd0 2024-04-02T13:20:04.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f767e40d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f76780ae440 con 0x7f767815a960 2024-04-02T13:20:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f767c9aa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f767815a960 0x7f767814d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f76777fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76780cacd0 0x7f76780b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f7677fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76780af210 0x7f76780b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f7677fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76780af210 0x7f76780b3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.69:48626/0 (socket says 172.21.15.69:48626) 2024-04-02T13:20:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f767c9aa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f767815a960 0x7f767814d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57820/0 (socket says 172.21.15.69:57820) 2024-04-02T13:20:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f76777fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76780cacd0 0x7f76780b0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49384/0 (socket says 172.21.15.69:49384) 2024-04-02T13:20:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f767c9aa640 1 -- 172.21.15.69:0/4248373871 learned_addr learned my addr 172.21.15.69:0/4248373871 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T13:20:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.033+0000 7f76777fe640 1 -- 172.21.15.69:0/4248373871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f767815a960 msgr2=0x7f767814d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f76777fe640 1 --2- 172.21.15.69:0/4248373871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f767815a960 0x7f767814d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f76777fe640 1 -- 172.21.15.69:0/4248373871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76780af210 msgr2=0x7f76780b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f76777fe640 1 --2- 172.21.15.69:0/4248373871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76780af210 0x7f76780b3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f76777fe640 1 -- 172.21.15.69:0/4248373871 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f767815c4a0 con 0x7f76780cacd0 2024-04-02T13:20:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f767c9aa640 1 --2- 172.21.15.69:0/4248373871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f767815a960 0x7f767814d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:04.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f7677fff640 1 --2- 172.21.15.69:0/4248373871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76780af210 0x7f76780b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:04.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f76777fe640 1 --2- 172.21.15.69:0/4248373871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76780cacd0 0x7f76780b0250 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f766800c390 tx=0x7f766800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:04.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f7674ff9640 1 -- 172.21.15.69:0/4248373871 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7668017070 con 0x7f76780cacd0 2024-04-02T13:20:04.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f7674ff9640 1 -- 172.21.15.69:0/4248373871 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76680052d0 con 0x7f76780cacd0 2024-04-02T13:20:04.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f7674ff9640 1 -- 172.21.15.69:0/4248373871 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7668003810 con 0x7f76780cacd0 2024-04-02T13:20:04.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f767e40d640 1 -- 172.21.15.69:0/4248373871 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76781f7b40 con 0x7f76780cacd0 2024-04-02T13:20:04.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f767e40d640 1 -- 172.21.15.69:0/4248373871 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76781f83e0 con 0x7f76780cacd0 2024-04-02T13:20:04.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f7674ff9640 1 -- 172.21.15.69:0/4248373871 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f766801a070 con 0x7f76780cacd0 2024-04-02T13:20:04.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.034+0000 7f7674ff9640 1 -- 172.21.15.69:0/4248373871 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f766801c440 con 0x7f76780cacd0 2024-04-02T13:20:04.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.035+0000 7f767e40d640 1 -- 172.21.15.69:0/4248373871 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76780684f0 con 0x7f76780cacd0 2024-04-02T13:20:04.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.039+0000 7f7674ff9640 1 -- 172.21.15.69:0/4248373871 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7668021020 con 0x7f76780cacd0 2024-04-02T13:20:04.062 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:04.060+0000 7f4d38a42040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T13:20:04.096 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:04.095+0000 7fe6c6e32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T13:20:04.109 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:04.107+0000 7f4d38a42040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T13:20:04.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.112+0000 7f767e40d640 1 -- 172.21.15.69:0/4248373871 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "30e26323-bca4-4e6a-876d-3760fd190851", "id": 0} v 0) -- 0x7f7678071f40 con 0x7f76780cacd0 2024-04-02T13:20:04.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.115+0000 7f7674ff9640 1 -- 172.21.15.69:0/4248373871 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "30e26323-bca4-4e6a-876d-3760fd190851", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f7668011650 con 0x7f76780cacd0 2024-04-02T13:20:04.117 INFO:teuthology.orchestra.run.smithi069.stdout:0 2024-04-02T13:20:04.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.116+0000 7f767e40d640 1 -- 172.21.15.69:0/4248373871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76780cacd0 msgr2=0x7f76780b0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.116+0000 7f767e40d640 1 --2- 172.21.15.69:0/4248373871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76780cacd0 0x7f76780b0250 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f766800c390 tx=0x7f766800c850 comp rx=0 tx=0).stop 2024-04-02T13:20:04.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.116+0000 7f767e40d640 1 -- 172.21.15.69:0/4248373871 shutdown_connections 2024-04-02T13:20:04.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.116+0000 7f767e40d640 1 --2- 172.21.15.69:0/4248373871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f767815a960 0x7f767814d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.116+0000 7f767e40d640 1 --2- 172.21.15.69:0/4248373871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76780cacd0 0x7f76780b0250 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.116+0000 7f767e40d640 1 --2- 172.21.15.69:0/4248373871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76780af210 0x7f76780b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.116+0000 7f767e40d640 1 -- 172.21.15.69:0/4248373871 >> 172.21.15.69:0/4248373871 conn(0x7f76780cfca0 msgr2=0x7f76780714e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:04.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.116+0000 7f767e40d640 1 -- 172.21.15.69:0/4248373871 shutdown_connections 2024-04-02T13:20:04.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.116+0000 7f767e40d640 1 -- 172.21.15.69:0/4248373871 wait complete. 2024-04-02T13:20:04.129 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 8fad5ee1-c346-4423-a9ab-8844a8c5764c 1 2024-04-02T13:20:04.144 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:04.143+0000 7fe6c6e32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T13:20:04.166 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:04.164+0000 7f4d38a42040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T13:20:04.199 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:04.198+0000 7fe6c6e32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T13:20:04.215 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:04.214+0000 7f4d38a42040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T13:20:04.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.227+0000 7fc0d4f96640 1 Processor -- start 2024-04-02T13:20:04.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.227+0000 7fc0d4f96640 1 -- start start 2024-04-02T13:20:04.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.228+0000 7fc0d4f96640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0d015c0c0 0x7fc0d00cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.228+0000 7fc0d4f96640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00ccf20 0x7fc0d00d1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.228+0000 7fc0d4f96640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0d00d2550 con 0x7fc0d00d20c0 2024-04-02T13:20:04.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.228+0000 7fc0d4f96640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc0d00cd510 con 0x7fc0d015c0c0 2024-04-02T13:20:04.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.228+0000 7fc0d4f96640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc0d00cd7f0 con 0x7fc0d00ccf20 2024-04-02T13:20:04.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.228+0000 7fc0ce59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0d015c0c0 0x7fc0d00cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.228+0000 7fc0c7fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00ccf20 0x7fc0d00d1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.228+0000 7fc0ce59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0d015c0c0 0x7fc0d00cc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49396/0 (socket says 172.21.15.69:49396) 2024-04-02T13:20:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.228+0000 7fc0ce59c640 1 -- 172.21.15.69:0/3298723036 learned_addr learned my addr 172.21.15.69:0/3298723036 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T13:20:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.229+0000 7fc0c7fff640 1 -- 172.21.15.69:0/3298723036 >> v2:172.21.15.69:3300/0 conn(0x7fc0d015c0c0 msgr2=0x7fc0d00cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.229+0000 7fc0c7fff640 1 --2- 172.21.15.69:0/3298723036 >> v2:172.21.15.69:3300/0 conn(0x7fc0d015c0c0 0x7fc0d00cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.229+0000 7fc0c7fff640 1 -- 172.21.15.69:0/3298723036 >> v1:172.21.15.44:6789/0 conn(0x7fc0d00d20c0 legacy=0x7fc0d00ce830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.229+0000 7fc0c7fff640 1 -- 172.21.15.69:0/3298723036 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0d00cdad0 con 0x7fc0d00ccf20 2024-04-02T13:20:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.229+0000 7fc0ce59c640 1 --2- 172.21.15.69:0/3298723036 >> v2:172.21.15.69:3300/0 conn(0x7fc0d015c0c0 0x7fc0d00cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.229+0000 7fc0c7fff640 1 --2- 172.21.15.69:0/3298723036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00ccf20 0x7fc0d00d1b80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80097a0 tx=0x7fc0b802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9bcd8e053945e1b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.229+0000 7fc0cdd9b640 1 -- 172.21.15.69:0/3298723036 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8031780 con 0x7fc0d00ccf20 2024-04-02T13:20:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.229+0000 7fc0cdd9b640 1 -- 172.21.15.69:0/3298723036 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b8037070 con 0x7fc0d00ccf20 2024-04-02T13:20:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.229+0000 7fc0cdd9b640 1 -- 172.21.15.69:0/3298723036 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8031b40 con 0x7fc0d00ccf20 2024-04-02T13:20:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.230+0000 7fc0d4f96640 1 -- 172.21.15.69:0/3298723036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00ccf20 msgr2=0x7fc0d00d1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.230+0000 7fc0d4f96640 1 --2- 172.21.15.69:0/3298723036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00ccf20 0x7fc0d00d1b80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80097a0 tx=0x7fc0b802b820 comp rx=0 tx=0).stop 2024-04-02T13:20:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.230+0000 7fc0d4f96640 1 -- 172.21.15.69:0/3298723036 shutdown_connections 2024-04-02T13:20:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.230+0000 7fc0d4f96640 1 --2- 172.21.15.69:0/3298723036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00ccf20 0x7fc0d00d1b80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.230+0000 7fc0d4f96640 1 --2- 172.21.15.69:0/3298723036 >> v2:172.21.15.69:3300/0 conn(0x7fc0d015c0c0 0x7fc0d00cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.230+0000 7fc0d4f96640 1 -- 172.21.15.69:0/3298723036 >> 172.21.15.69:0/3298723036 conn(0x7fc0d009aa20 msgr2=0x7fc0d009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.230+0000 7fc0d4f96640 1 -- 172.21.15.69:0/3298723036 shutdown_connections 2024-04-02T13:20:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.230+0000 7fc0d4f96640 1 -- 172.21.15.69:0/3298723036 wait complete. 2024-04-02T13:20:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.231+0000 7fc0d4f96640 1 Processor -- start 2024-04-02T13:20:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.231+0000 7fc0d4f96640 1 -- start start 2024-04-02T13:20:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.232+0000 7fc0d4f96640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0d00ccf20 0x7fc0d00c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.232+0000 7fc0d4f96640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d20c0 0x7fc0d00c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.232+0000 7fc0d4f96640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0d015c0c0 0x7fc0d006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.232+0000 7fc0d4f96640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d015c4a0 con 0x7fc0d015c0c0 2024-04-02T13:20:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.232+0000 7fc0d4f96640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d00cfcf0 con 0x7fc0d00ccf20 2024-04-02T13:20:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.232+0000 7fc0d4f96640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc0d00c9380 con 0x7fc0d00d20c0 2024-04-02T13:20:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.232+0000 7fc0ce59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0d00ccf20 0x7fc0d00c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.232+0000 7fc0c7fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d20c0 0x7fc0d00c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.232+0000 7fc0ce59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0d00ccf20 0x7fc0d00c7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49400/0 (socket says 172.21.15.69:49400) 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.232+0000 7fc0ce59c640 1 -- 172.21.15.69:0/342998104 learned_addr learned my addr 172.21.15.69:0/342998104 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.232+0000 7fc0ced9d640 1 --2- 172.21.15.69:0/342998104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0d015c0c0 0x7fc0d006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0c7fff640 1 -- 172.21.15.69:0/342998104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0d00ccf20 msgr2=0x7fc0d00c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0c7fff640 1 --2- 172.21.15.69:0/342998104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0d00ccf20 0x7fc0d00c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0c7fff640 1 -- 172.21.15.69:0/342998104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0d015c0c0 msgr2=0x7fc0d006d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0c7fff640 1 --2- 172.21.15.69:0/342998104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0d015c0c0 0x7fc0d006d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0c7fff640 1 -- 172.21.15.69:0/342998104 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0d006dbc0 con 0x7fc0d00d20c0 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0ce59c640 1 --2- 172.21.15.69:0/342998104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0d00ccf20 0x7fc0d00c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0ced9d640 1 --2- 172.21.15.69:0/342998104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0d015c0c0 0x7fc0d006d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0c7fff640 1 --2- 172.21.15.69:0/342998104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d20c0 0x7fc0d00c4b10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8033000 tx=0x7fc0b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0c6ffd640 1 -- 172.21.15.69:0/342998104 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8032610 con 0x7fc0d00d20c0 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0c6ffd640 1 -- 172.21.15.69:0/342998104 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b80327b0 con 0x7fc0d00d20c0 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0d4f96640 1 -- 172.21.15.69:0/342998104 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0d00c7bd0 con 0x7fc0d00d20c0 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0c6ffd640 1 -- 172.21.15.69:0/342998104 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8032ba0 con 0x7fc0d00d20c0 2024-04-02T13:20:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0d4f96640 1 -- 172.21.15.69:0/342998104 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0d00c8190 con 0x7fc0d00d20c0 2024-04-02T13:20:04.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0c6ffd640 1 -- 172.21.15.69:0/342998104 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc0b804e7d0 con 0x7fc0d00d20c0 2024-04-02T13:20:04.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.233+0000 7fc0c6ffd640 1 -- 172.21.15.69:0/342998104 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fc0b8045b70 con 0x7fc0d00d20c0 2024-04-02T13:20:04.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.234+0000 7fc0d4f96640 1 -- 172.21.15.69:0/342998104 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0d0068c00 con 0x7fc0d00d20c0 2024-04-02T13:20:04.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.236+0000 7fc0c6ffd640 1 -- 172.21.15.69:0/342998104 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc0b8040070 con 0x7fc0d00d20c0 2024-04-02T13:20:04.248 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:04.247+0000 7fe6c6e32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T13:20:04.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.309+0000 7fc0d4f96640 1 -- 172.21.15.69:0/342998104 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8fad5ee1-c346-4423-a9ab-8844a8c5764c", "id": 1} v 0) -- 0x7fc0d00c9500 con 0x7fc0d00d20c0 2024-04-02T13:20:04.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.312+0000 7fc0c6ffd640 1 -- 172.21.15.69:0/342998104 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8fad5ee1-c346-4423-a9ab-8844a8c5764c", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fc0b802f050 con 0x7fc0d00d20c0 2024-04-02T13:20:04.314 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-02T13:20:04.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.313+0000 7fc0d4f96640 1 -- 172.21.15.69:0/342998104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d20c0 msgr2=0x7fc0d00c4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.313+0000 7fc0d4f96640 1 --2- 172.21.15.69:0/342998104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d20c0 0x7fc0d00c4b10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8033000 tx=0x7fc0b80046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.313+0000 7fc0d4f96640 1 -- 172.21.15.69:0/342998104 shutdown_connections 2024-04-02T13:20:04.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.313+0000 7fc0d4f96640 1 --2- 172.21.15.69:0/342998104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0d015c0c0 0x7fc0d006d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.313+0000 7fc0d4f96640 1 --2- 172.21.15.69:0/342998104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d20c0 0x7fc0d00c4b10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.313+0000 7fc0d4f96640 1 --2- 172.21.15.69:0/342998104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0d00ccf20 0x7fc0d00c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.313+0000 7fc0d4f96640 1 -- 172.21.15.69:0/342998104 >> 172.21.15.69:0/342998104 conn(0x7fc0d009aa20 msgr2=0x7fc0d006f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:04.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.313+0000 7fc0d4f96640 1 -- 172.21.15.69:0/342998104 shutdown_connections 2024-04-02T13:20:04.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.313+0000 7fc0d4f96640 1 -- 172.21.15.69:0/342998104 wait complete. 2024-04-02T13:20:04.325 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new d06483ed-67be-45e7-9cbb-3e8c42844491 2 2024-04-02T13:20:04.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.464+0000 7feca633d640 1 Processor -- start 2024-04-02T13:20:04.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.464+0000 7feca633d640 1 -- start start 2024-04-02T13:20:04.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.465+0000 7feca633d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7feca00b0340 0x7feca00cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.465+0000 7feca633d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feca00ce4c0 0x7feca006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.465+0000 7feca633d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feca006f790 con 0x7feca0162010 2024-04-02T13:20:04.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.465+0000 7feca633d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7feca006f540 con 0x7feca00b0340 2024-04-02T13:20:04.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.465+0000 7feca633d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7feca006fa70 con 0x7feca00ce4c0 2024-04-02T13:20:04.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.465+0000 7feca48da640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feca00ce4c0 0x7feca006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.465+0000 7feca48da640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feca00ce4c0 0x7feca006b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57846/0 (socket says 172.21.15.69:57846) 2024-04-02T13:20:04.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.465+0000 7feca48da640 1 -- 172.21.15.69:0/1216042505 learned_addr learned my addr 172.21.15.69:0/1216042505 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T13:20:04.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.466+0000 7fec9ffff640 1 --2- 172.21.15.69:0/1216042505 >> v2:172.21.15.44:3300/0 conn(0x7feca00b0340 0x7feca00cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.466+0000 7feca48da640 1 -- 172.21.15.69:0/1216042505 >> v2:172.21.15.44:3300/0 conn(0x7feca00b0340 msgr2=0x7feca00cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:04.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.466+0000 7feca48da640 1 --2- 172.21.15.69:0/1216042505 >> v2:172.21.15.44:3300/0 conn(0x7feca00b0340 0x7feca00cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.466+0000 7feca48da640 1 -- 172.21.15.69:0/1216042505 >> v1:172.21.15.44:6789/0 conn(0x7feca0162010 legacy=0x7feca00cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.466+0000 7feca48da640 1 -- 172.21.15.69:0/1216042505 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca006fd50 con 0x7feca00ce4c0 2024-04-02T13:20:04.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.466+0000 7feca48da640 1 --2- 172.21.15.69:0/1216042505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feca00ce4c0 0x7feca006b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fec98009e50 tx=0x7fec9802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db8b566ac3832d9e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:04.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.467+0000 7fec9effd640 1 -- 172.21.15.69:0/1216042505 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9802ca70 con 0x7feca00ce4c0 2024-04-02T13:20:04.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.467+0000 7fec9effd640 1 -- 172.21.15.69:0/1216042505 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec9802cc10 con 0x7feca00ce4c0 2024-04-02T13:20:04.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.467+0000 7fec9effd640 1 -- 172.21.15.69:0/1216042505 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec980343f0 con 0x7feca00ce4c0 2024-04-02T13:20:04.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.467+0000 7feca633d640 1 -- 172.21.15.69:0/1216042505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feca00ce4c0 msgr2=0x7feca006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.467+0000 7feca633d640 1 --2- 172.21.15.69:0/1216042505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feca00ce4c0 0x7feca006b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fec98009e50 tx=0x7fec9802bed0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.467+0000 7feca633d640 1 -- 172.21.15.69:0/1216042505 shutdown_connections 2024-04-02T13:20:04.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.467+0000 7feca633d640 1 --2- 172.21.15.69:0/1216042505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feca00ce4c0 0x7feca006b4d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.467+0000 7feca633d640 1 --2- 172.21.15.69:0/1216042505 >> v2:172.21.15.44:3300/0 conn(0x7feca00b0340 0x7feca00cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.467+0000 7feca633d640 1 -- 172.21.15.69:0/1216042505 >> 172.21.15.69:0/1216042505 conn(0x7feca00b2a10 msgr2=0x7feca00b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:04.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.467+0000 7feca633d640 1 -- 172.21.15.69:0/1216042505 shutdown_connections 2024-04-02T13:20:04.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.467+0000 7feca633d640 1 -- 172.21.15.69:0/1216042505 wait complete. 2024-04-02T13:20:04.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.469+0000 7feca633d640 1 Processor -- start 2024-04-02T13:20:04.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.469+0000 7feca633d640 1 -- start start 2024-04-02T13:20:04.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.469+0000 7feca633d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feca00b0340 0x7feca00c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.469+0000 7feca633d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feca00ce4c0 0x7feca0161430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.469+0000 7feca633d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feca0162010 0x7feca015c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.469+0000 7feca633d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feca00cd560 con 0x7feca00b0340 2024-04-02T13:20:04.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.469+0000 7feca633d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7feca0070030 con 0x7feca0162010 2024-04-02T13:20:04.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.469+0000 7feca633d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7feca00730e0 con 0x7feca00ce4c0 2024-04-02T13:20:04.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7feca48da640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feca0162010 0x7feca015c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7feca48da640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feca0162010 0x7feca015c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49416/0 (socket says 172.21.15.69:49416) 2024-04-02T13:20:04.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7feca48da640 1 -- 172.21.15.69:0/4231640409 learned_addr learned my addr 172.21.15.69:0/4231640409 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T13:20:04.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7fec9f7fe640 1 --2- 172.21.15.69:0/4231640409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feca00ce4c0 0x7feca0161430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7fec9ffff640 1 --2- 172.21.15.69:0/4231640409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feca00b0340 0x7feca00c7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7feca48da640 1 -- 172.21.15.69:0/4231640409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feca00ce4c0 msgr2=0x7feca0161430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7feca48da640 1 --2- 172.21.15.69:0/4231640409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feca00ce4c0 0x7feca0161430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7feca48da640 1 -- 172.21.15.69:0/4231640409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feca00b0340 msgr2=0x7feca00c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7feca48da640 1 --2- 172.21.15.69:0/4231640409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feca00b0340 0x7feca00c7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7feca48da640 1 -- 172.21.15.69:0/4231640409 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca00c84f0 con 0x7feca0162010 2024-04-02T13:20:04.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7fec9f7fe640 1 --2- 172.21.15.69:0/4231640409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feca00ce4c0 0x7feca0161430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:04.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7feca48da640 1 --2- 172.21.15.69:0/4231640409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feca0162010 0x7feca015c420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fec98009e50 tx=0x7fec98004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:04.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7fec9cff9640 1 -- 172.21.15.69:0/4231640409 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec98043020 con 0x7feca0162010 2024-04-02T13:20:04.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7fec9cff9640 1 -- 172.21.15.69:0/4231640409 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec9803e450 con 0x7feca0162010 2024-04-02T13:20:04.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7feca633d640 1 -- 172.21.15.69:0/4231640409 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca00c87d0 con 0x7feca0162010 2024-04-02T13:20:04.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7fec9cff9640 1 -- 172.21.15.69:0/4231640409 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec98009510 con 0x7feca0162010 2024-04-02T13:20:04.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.470+0000 7feca633d640 1 -- 172.21.15.69:0/4231640409 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca00c8d90 con 0x7feca0162010 2024-04-02T13:20:04.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.471+0000 7fec9cff9640 1 -- 172.21.15.69:0/4231640409 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fec98046020 con 0x7feca0162010 2024-04-02T13:20:04.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.471+0000 7fec9cff9640 1 -- 172.21.15.69:0/4231640409 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fec98048820 con 0x7feca0162010 2024-04-02T13:20:04.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.472+0000 7feca633d640 1 -- 172.21.15.69:0/4231640409 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feca0068c00 con 0x7feca0162010 2024-04-02T13:20:04.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.475+0000 7fec9cff9640 1 -- 172.21.15.69:0/4231640409 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fec9804d030 con 0x7feca0162010 2024-04-02T13:20:04.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.548+0000 7feca633d640 1 -- 172.21.15.69:0/4231640409 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d06483ed-67be-45e7-9cbb-3e8c42844491", "id": 2} v 0) -- 0x7feca00cb200 con 0x7feca0162010 2024-04-02T13:20:04.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.551+0000 7fec9cff9640 1 -- 172.21.15.69:0/4231640409 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d06483ed-67be-45e7-9cbb-3e8c42844491", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fec9802ca80 con 0x7feca0162010 2024-04-02T13:20:04.554 INFO:teuthology.orchestra.run.smithi069.stdout:2 2024-04-02T13:20:04.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.552+0000 7feca633d640 1 -- 172.21.15.69:0/4231640409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feca0162010 msgr2=0x7feca015c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.552+0000 7feca633d640 1 --2- 172.21.15.69:0/4231640409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feca0162010 0x7feca015c420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fec98009e50 tx=0x7fec98004a40 comp rx=0 tx=0).stop 2024-04-02T13:20:04.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.552+0000 7feca633d640 1 -- 172.21.15.69:0/4231640409 shutdown_connections 2024-04-02T13:20:04.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.552+0000 7feca633d640 1 --2- 172.21.15.69:0/4231640409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feca0162010 0x7feca015c420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.552+0000 7feca633d640 1 --2- 172.21.15.69:0/4231640409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feca00ce4c0 0x7feca0161430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.552+0000 7feca633d640 1 --2- 172.21.15.69:0/4231640409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feca00b0340 0x7feca00c7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.552+0000 7feca633d640 1 -- 172.21.15.69:0/4231640409 >> 172.21.15.69:0/4231640409 conn(0x7feca00b2a10 msgr2=0x7feca00b1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:04.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.552+0000 7feca633d640 1 -- 172.21.15.69:0/4231640409 shutdown_connections 2024-04-02T13:20:04.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.552+0000 7feca633d640 1 -- 172.21.15.69:0/4231640409 wait complete. 2024-04-02T13:20:04.564 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 6334eb96-f083-49fa-8ae2-a307382727d6 3 2024-04-02T13:20:04.616 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:04.614+0000 7f4d38a42040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T13:20:04.644 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:04.643+0000 7fe6c6e32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T13:20:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.663+0000 7fb66bcb7640 1 Processor -- start 2024-04-02T13:20:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.664+0000 7fb66bcb7640 1 -- start start 2024-04-02T13:20:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.664+0000 7fb66bcb7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6640c5cf0 0x7fb664072cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.664+0000 7fb66bcb7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb66406e020 0x7fb6640c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.664+0000 7fb66bcb7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6640731f0 con 0x7fb66406b4d0 2024-04-02T13:20:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.664+0000 7fb66bcb7640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fb66406ff20 con 0x7fb66406e020 2024-04-02T13:20:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.664+0000 7fb66bcb7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb66406f910 con 0x7fb6640c5cf0 2024-04-02T13:20:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.665+0000 7fb669a53640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6640c5cf0 0x7fb664072cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.665+0000 7fb66a254640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb66406b4d0 0x7fb66406b960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48680/0 (socket says 172.21.15.69:48680) 2024-04-02T13:20:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.665+0000 7fb66a254640 1 -- 172.21.15.69:0/3775359399 learned_addr learned my addr 172.21.15.69:0/3775359399 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-02T13:20:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.665+0000 7fb669252640 1 --2- 172.21.15.69:0/3775359399 >> v2:172.21.15.44:3300/0 conn(0x7fb66406e020 0x7fb6640c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.665+0000 7fb669a53640 1 -- 172.21.15.69:0/3775359399 >> v2:172.21.15.44:3300/0 conn(0x7fb66406e020 msgr2=0x7fb6640c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.665+0000 7fb669a53640 1 --2- 172.21.15.69:0/3775359399 >> v2:172.21.15.44:3300/0 conn(0x7fb66406e020 0x7fb6640c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.665+0000 7fb668a51640 1 -- 172.21.15.69:0/3775359399 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1706505593 0 0) 0x7fb660001490 con 0x7fb66406b4d0 2024-04-02T13:20:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.665+0000 7fb669a53640 1 -- 172.21.15.69:0/3775359399 >> v1:172.21.15.69:6789/0 conn(0x7fb66406b4d0 legacy=0x7fb66406b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.665+0000 7fb669a53640 1 -- 172.21.15.69:0/3775359399 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6640c6610 con 0x7fb6640c5cf0 2024-04-02T13:20:04.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.666+0000 7fb669a53640 1 --2- 172.21.15.69:0/3775359399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6640c5cf0 0x7fb664072cb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0097a0 tx=0x7fb65c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fad486e44279befa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:04.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.666+0000 7fb668a51640 1 -- 172.21.15.69:0/3775359399 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0315a0 con 0x7fb6640c5cf0 2024-04-02T13:20:04.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.666+0000 7fb668a51640 1 -- 172.21.15.69:0/3775359399 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb65c037070 con 0x7fb6640c5cf0 2024-04-02T13:20:04.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.666+0000 7fb668a51640 1 -- 172.21.15.69:0/3775359399 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c031960 con 0x7fb6640c5cf0 2024-04-02T13:20:04.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.666+0000 7fb66bcb7640 1 -- 172.21.15.69:0/3775359399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6640c5cf0 msgr2=0x7fb664072cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.666+0000 7fb66bcb7640 1 --2- 172.21.15.69:0/3775359399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6640c5cf0 0x7fb664072cb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0097a0 tx=0x7fb65c02b840 comp rx=0 tx=0).stop 2024-04-02T13:20:04.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.666+0000 7fb66bcb7640 1 -- 172.21.15.69:0/3775359399 shutdown_connections 2024-04-02T13:20:04.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.666+0000 7fb66bcb7640 1 --2- 172.21.15.69:0/3775359399 >> v2:172.21.15.44:3300/0 conn(0x7fb66406e020 0x7fb6640c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.667+0000 7fb66bcb7640 1 --2- 172.21.15.69:0/3775359399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6640c5cf0 0x7fb664072cb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.667+0000 7fb66bcb7640 1 -- 172.21.15.69:0/3775359399 >> 172.21.15.69:0/3775359399 conn(0x7fb6640d0e50 msgr2=0x7fb6640d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:04.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.667+0000 7fb66bcb7640 1 -- 172.21.15.69:0/3775359399 shutdown_connections 2024-04-02T13:20:04.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.667+0000 7fb66bcb7640 1 -- 172.21.15.69:0/3775359399 wait complete. 2024-04-02T13:20:04.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.668+0000 7fb66bcb7640 1 Processor -- start 2024-04-02T13:20:04.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.668+0000 7fb66bcb7640 1 -- start start 2024-04-02T13:20:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.668+0000 7fb66bcb7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb66406b4d0 0x7fb6641e3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.668+0000 7fb66bcb7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb66406e020 0x7fb66414da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.668+0000 7fb66bcb7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6640c5cf0 0x7fb6640ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.668+0000 7fb66bcb7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb66406fbb0 con 0x7fb66406e020 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.668+0000 7fb66bcb7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb6640c7b20 con 0x7fb66406b4d0 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.668+0000 7fb66bcb7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb66415e5f0 con 0x7fb6640c5cf0 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb669a53640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb66406b4d0 0x7fb6641e3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb66a254640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6640c5cf0 0x7fb6640ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb669a53640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb66406b4d0 0x7fb6641e3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49422/0 (socket says 172.21.15.69:49422) 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb669a53640 1 -- 172.21.15.69:0/1180708140 learned_addr learned my addr 172.21.15.69:0/1180708140 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb66a254640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6640c5cf0 0x7fb6640ca280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57874/0 (socket says 172.21.15.69:57874) 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb669252640 1 --2- 172.21.15.69:0/1180708140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb66406e020 0x7fb66414da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb66a254640 1 -- 172.21.15.69:0/1180708140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb66406b4d0 msgr2=0x7fb6641e3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb66a254640 1 --2- 172.21.15.69:0/1180708140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb66406b4d0 0x7fb6641e3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb66a254640 1 -- 172.21.15.69:0/1180708140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb66406e020 msgr2=0x7fb66414da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb66a254640 1 --2- 172.21.15.69:0/1180708140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb66406e020 0x7fb66414da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb66a254640 1 -- 172.21.15.69:0/1180708140 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66415d6e0 con 0x7fb6640c5cf0 2024-04-02T13:20:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb669a53640 1 --2- 172.21.15.69:0/1180708140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb66406b4d0 0x7fb6641e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:04.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb669252640 1 --2- 172.21.15.69:0/1180708140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb66406e020 0x7fb66414da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:04.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb66a254640 1 --2- 172.21.15.69:0/1180708140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6640c5cf0 0x7fb6640ca280 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb66000c6e0 tx=0x7fb66000cba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:04.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb65a7fc640 1 -- 172.21.15.69:0/1180708140 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb660018070 con 0x7fb6640c5cf0 2024-04-02T13:20:04.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb65a7fc640 1 -- 172.21.15.69:0/1180708140 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb660004d60 con 0x7fb6640c5cf0 2024-04-02T13:20:04.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb66bcb7640 1 -- 172.21.15.69:0/1180708140 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6641e3ab0 con 0x7fb6640c5cf0 2024-04-02T13:20:04.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb65a7fc640 1 -- 172.21.15.69:0/1180708140 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66000a880 con 0x7fb6640c5cf0 2024-04-02T13:20:04.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb66bcb7640 1 -- 172.21.15.69:0/1180708140 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6640c94c0 con 0x7fb6640c5cf0 2024-04-02T13:20:04.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb65a7fc640 1 -- 172.21.15.69:0/1180708140 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb66001b070 con 0x7fb6640c5cf0 2024-04-02T13:20:04.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.669+0000 7fb65a7fc640 1 -- 172.21.15.69:0/1180708140 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fb660007800 con 0x7fb6640c5cf0 2024-04-02T13:20:04.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.670+0000 7fb66bcb7640 1 -- 172.21.15.69:0/1180708140 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb664068c00 con 0x7fb6640c5cf0 2024-04-02T13:20:04.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.672+0000 7fb65a7fc640 1 -- 172.21.15.69:0/1180708140 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb6640731f0 con 0x7fb6640c5cf0 2024-04-02T13:20:04.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.745+0000 7fb66bcb7640 1 -- 172.21.15.69:0/1180708140 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6334eb96-f083-49fa-8ae2-a307382727d6", "id": 3} v 0) -- 0x7fb66406e7f0 con 0x7fb6640c5cf0 2024-04-02T13:20:04.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.748+0000 7fb65a7fc640 1 -- 172.21.15.69:0/1180708140 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6334eb96-f083-49fa-8ae2-a307382727d6", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fb660007260 con 0x7fb6640c5cf0 2024-04-02T13:20:04.750 INFO:teuthology.orchestra.run.smithi069.stdout:3 2024-04-02T13:20:04.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.749+0000 7fb66bcb7640 1 -- 172.21.15.69:0/1180708140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6640c5cf0 msgr2=0x7fb6640ca280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.749+0000 7fb66bcb7640 1 --2- 172.21.15.69:0/1180708140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6640c5cf0 0x7fb6640ca280 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb66000c6e0 tx=0x7fb66000cba0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.749+0000 7fb66bcb7640 1 -- 172.21.15.69:0/1180708140 shutdown_connections 2024-04-02T13:20:04.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.749+0000 7fb66bcb7640 1 --2- 172.21.15.69:0/1180708140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6640c5cf0 0x7fb6640ca280 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.749+0000 7fb66bcb7640 1 --2- 172.21.15.69:0/1180708140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb66406e020 0x7fb66414da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.749+0000 7fb66bcb7640 1 --2- 172.21.15.69:0/1180708140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb66406b4d0 0x7fb6641e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.749+0000 7fb66bcb7640 1 -- 172.21.15.69:0/1180708140 >> 172.21.15.69:0/1180708140 conn(0x7fb6640d0e50 msgr2=0x7fb6640aff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:04.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.749+0000 7fb66bcb7640 1 -- 172.21.15.69:0/1180708140 shutdown_connections 2024-04-02T13:20:04.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.749+0000 7fb66bcb7640 1 -- 172.21.15.69:0/1180708140 wait complete. 2024-04-02T13:20:04.756 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:04.755+0000 7f4d38a42040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T13:20:04.761 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new d3f9ff73-9bd8-4177-9931-b3f818a4204c 4 2024-04-02T13:20:04.785 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:04.784+0000 7fe6c6e32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T13:20:04.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.860+0000 7f66cd352640 1 Processor -- start 2024-04-02T13:20:04.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.860+0000 7f66cd352640 1 -- start start 2024-04-02T13:20:04.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.861+0000 7f66cd352640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66c80ce4c0 0x7f66c806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.861+0000 7f66cd352640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c806f790 con 0x7f66c80b0340 2024-04-02T13:20:04.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.861+0000 7f66cd352640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c806fa70 con 0x7f66c8162010 2024-04-02T13:20:04.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.861+0000 7f66cd352640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f66c806f540 con 0x7f66c80ce4c0 2024-04-02T13:20:04.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.861+0000 7f66c77fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66c80ce4c0 0x7f66c806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.861+0000 7f66c77fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66c80ce4c0 0x7f66c806b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49432/0 (socket says 172.21.15.69:49432) 2024-04-02T13:20:04.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.861+0000 7f66c77fe640 1 -- 172.21.15.69:0/1089132050 learned_addr learned my addr 172.21.15.69:0/1089132050 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T13:20:04.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.862+0000 7f66c77fe640 1 -- 172.21.15.69:0/1089132050 >> v1:172.21.15.69:6789/0 conn(0x7f66c8162010 legacy=0x7f66c80cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.862+0000 7f66c5ffb640 1 -- 172.21.15.69:0/1089132050 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1934132933 0 0) 0x7f66c806f790 con 0x7f66c80b0340 2024-04-02T13:20:04.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.862+0000 7f66c77fe640 1 -- 172.21.15.69:0/1089132050 >> v1:172.21.15.44:6789/0 conn(0x7f66c80b0340 legacy=0x7f66c80cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.862+0000 7f66c77fe640 1 -- 172.21.15.69:0/1089132050 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c806fd50 con 0x7f66c80ce4c0 2024-04-02T13:20:04.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.862+0000 7f66c77fe640 1 --2- 172.21.15.69:0/1089132050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66c80ce4c0 0x7f66c806b4d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009e50 tx=0x7f66c002bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c5a0ffc08385aa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:04.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.862+0000 7f66c5ffb640 1 -- 172.21.15.69:0/1089132050 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c002cb20 con 0x7f66c80ce4c0 2024-04-02T13:20:04.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.863+0000 7f66c5ffb640 1 -- 172.21.15.69:0/1089132050 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66c002ccc0 con 0x7f66c80ce4c0 2024-04-02T13:20:04.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.863+0000 7f66c5ffb640 1 -- 172.21.15.69:0/1089132050 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0034610 con 0x7f66c80ce4c0 2024-04-02T13:20:04.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.863+0000 7f66cd352640 1 -- 172.21.15.69:0/1089132050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66c80ce4c0 msgr2=0x7f66c806b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.863+0000 7f66cd352640 1 --2- 172.21.15.69:0/1089132050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66c80ce4c0 0x7f66c806b4d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009e50 tx=0x7f66c002bf80 comp rx=0 tx=0).stop 2024-04-02T13:20:04.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.863+0000 7f66cd352640 1 -- 172.21.15.69:0/1089132050 shutdown_connections 2024-04-02T13:20:04.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.863+0000 7f66cd352640 1 --2- 172.21.15.69:0/1089132050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66c80ce4c0 0x7f66c806b4d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.863+0000 7f66cd352640 1 -- 172.21.15.69:0/1089132050 >> 172.21.15.69:0/1089132050 conn(0x7f66c80b2a10 msgr2=0x7f66c80b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:04.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.863+0000 7f66cd352640 1 -- 172.21.15.69:0/1089132050 shutdown_connections 2024-04-02T13:20:04.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.863+0000 7f66cd352640 1 -- 172.21.15.69:0/1089132050 wait complete. 2024-04-02T13:20:04.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.865+0000 7f66cd352640 1 Processor -- start 2024-04-02T13:20:04.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.866+0000 7f66cd352640 1 -- start start 2024-04-02T13:20:04.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66cd352640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66c80b0340 0x7f66c8157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66cd352640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c80ce4c0 0x7f66c815fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66cd352640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66c8162010 0x7f66c81603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:04.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66cd352640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f66c80730e0 con 0x7f66c80ce4c0 2024-04-02T13:20:04.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66cd352640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f66c8070030 con 0x7f66c8162010 2024-04-02T13:20:04.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66cd352640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f66c8073260 con 0x7f66c80b0340 2024-04-02T13:20:04.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66c6ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66c80b0340 0x7f66c8157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66c77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66c8162010 0x7f66c81603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66c6ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66c80b0340 0x7f66c8157740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57882/0 (socket says 172.21.15.69:57882) 2024-04-02T13:20:04.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66c6ffd640 1 -- 172.21.15.69:0/2666162147 learned_addr learned my addr 172.21.15.69:0/2666162147 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66c67fc640 1 --2- 172.21.15.69:0/2666162147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c80ce4c0 0x7f66c815fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66c6ffd640 1 -- 172.21.15.69:0/2666162147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66c8162010 msgr2=0x7f66c81603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66c6ffd640 1 --2- 172.21.15.69:0/2666162147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66c8162010 0x7f66c81603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66c6ffd640 1 -- 172.21.15.69:0/2666162147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c80ce4c0 msgr2=0x7f66c815fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66c6ffd640 1 --2- 172.21.15.69:0/2666162147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c80ce4c0 0x7f66c815fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66c6ffd640 1 -- 172.21.15.69:0/2666162147 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c81f8270 con 0x7f66c80b0340 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66c77fe640 1 --2- 172.21.15.69:0/2666162147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66c8162010 0x7f66c81603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66c6ffd640 1 --2- 172.21.15.69:0/2666162147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66c80b0340 0x7f66c8157740 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f66bc00c440 tx=0x7f66bc00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.867+0000 7f66af7fe640 1 -- 172.21.15.69:0/2666162147 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc017070 con 0x7f66c80b0340 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.868+0000 7f66af7fe640 1 -- 172.21.15.69:0/2666162147 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66bc005480 con 0x7f66c80b0340 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.868+0000 7f66cd352640 1 -- 172.21.15.69:0/2666162147 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c80c9c90 con 0x7f66c80b0340 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.868+0000 7f66af7fe640 1 -- 172.21.15.69:0/2666162147 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc003830 con 0x7f66c80b0340 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.868+0000 7f66cd352640 1 -- 172.21.15.69:0/2666162147 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c80ca250 con 0x7f66c80b0340 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.868+0000 7f66af7fe640 1 -- 172.21.15.69:0/2666162147 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f66bc01a070 con 0x7f66c80b0340 2024-04-02T13:20:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.868+0000 7f66af7fe640 1 -- 172.21.15.69:0/2666162147 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f66bc012910 con 0x7f66c80b0340 2024-04-02T13:20:04.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.868+0000 7f66cd352640 1 -- 172.21.15.69:0/2666162147 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c8068c00 con 0x7f66c80b0340 2024-04-02T13:20:04.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.870+0000 7f66af7fe640 1 -- 172.21.15.69:0/2666162147 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f66bc021020 con 0x7f66c80b0340 2024-04-02T13:20:04.889 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:04.888+0000 7f4d38a42040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T13:20:04.897 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:04.896+0000 7fe6c6e32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T13:20:04.937 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:04.935+0000 7f4d38a42040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T13:20:04.944 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:04.943+0000 7fe6c6e32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T13:20:04.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.943+0000 7f66cd352640 1 -- 172.21.15.69:0/2666162147 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d3f9ff73-9bd8-4177-9931-b3f818a4204c", "id": 4} v 0) -- 0x7f66c8072860 con 0x7f66c80b0340 2024-04-02T13:20:04.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.949+0000 7f66af7fe640 1 -- 172.21.15.69:0/2666162147 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d3f9ff73-9bd8-4177-9931-b3f818a4204c", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f66bc012c20 con 0x7f66c80b0340 2024-04-02T13:20:04.951 INFO:teuthology.orchestra.run.smithi069.stdout:4 2024-04-02T13:20:04.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.950+0000 7f66cd352640 1 -- 172.21.15.69:0/2666162147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66c80b0340 msgr2=0x7f66c8157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:04.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.950+0000 7f66cd352640 1 --2- 172.21.15.69:0/2666162147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66c80b0340 0x7f66c8157740 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f66bc00c440 tx=0x7f66bc00c900 comp rx=0 tx=0).stop 2024-04-02T13:20:04.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.950+0000 7f66cd352640 1 -- 172.21.15.69:0/2666162147 shutdown_connections 2024-04-02T13:20:04.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.950+0000 7f66cd352640 1 --2- 172.21.15.69:0/2666162147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66c8162010 0x7f66c81603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.950+0000 7f66cd352640 1 --2- 172.21.15.69:0/2666162147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c80ce4c0 0x7f66c815fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.950+0000 7f66cd352640 1 --2- 172.21.15.69:0/2666162147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66c80b0340 0x7f66c8157740 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:04.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.950+0000 7f66cd352640 1 -- 172.21.15.69:0/2666162147 >> 172.21.15.69:0/2666162147 conn(0x7f66c80b2a10 msgr2=0x7f66c806bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:04.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.950+0000 7f66cd352640 1 -- 172.21.15.69:0/2666162147 shutdown_connections 2024-04-02T13:20:04.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:04.950+0000 7f66cd352640 1 -- 172.21.15.69:0/2666162147 wait complete. 2024-04-02T13:20:04.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 37e3b164-c05f-4989-9658-8866f1b512b7 5 2024-04-02T13:20:05.056 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:05.054+0000 7f4d38a42040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T13:20:05.062 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:05.061+0000 7fe6c6e32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T13:20:05.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.062+0000 7f1e70e2a640 1 Processor -- start 2024-04-02T13:20:05.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.062+0000 7f1e70e2a640 1 -- start start 2024-04-02T13:20:05.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e70e2a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1e6c155060 0x7f1e6c155440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e70e2a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e6c0d1030 0x7f1e6c0d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e70e2a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e6c162070 con 0x7f1e6c15eba0 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e70e2a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1e6c162e30 con 0x7f1e6c155060 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e70e2a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1e6c1629a0 con 0x7f1e6c0d1030 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e69d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e6c0d1030 0x7f1e6c0d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e6a59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1e6c155060 0x7f1e6c155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e69d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e6c0d1030 0x7f1e6c0d14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57888/0 (socket says 172.21.15.69:57888) 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e6a59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1e6c155060 0x7f1e6c155440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49450/0 (socket says 172.21.15.69:49450) 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e69d9b640 1 -- 172.21.15.69:0/1460898942 learned_addr learned my addr 172.21.15.69:0/1460898942 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e6ad9d640 1 --1- 172.21.15.69:0/1460898942 >> v1:172.21.15.69:6789/0 conn(0x7f1e6c15eba0 0x7f1e6c15f030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48706/0 (socket says 172.21.15.69:48706) 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e6a59c640 1 -- 172.21.15.69:0/1460898942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e6c0d1030 msgr2=0x7f1e6c0d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e6a59c640 1 --2- 172.21.15.69:0/1460898942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e6c0d1030 0x7f1e6c0d14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e6a59c640 1 -- 172.21.15.69:0/1460898942 >> v1:172.21.15.69:6789/0 conn(0x7f1e6c15eba0 legacy=0x7f1e6c15f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.063+0000 7f1e6a59c640 1 -- 172.21.15.69:0/1460898942 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6c073240 con 0x7f1e6c155060 2024-04-02T13:20:05.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.064+0000 7f1e6a59c640 1 --2- 172.21.15.69:0/1460898942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e6c155060 0x7f1e6c155440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0097a0 tx=0x7f1e5c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b30964568f120957 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:05.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.064+0000 7f1e6959a640 1 -- 172.21.15.69:0/1460898942 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c035020 con 0x7f1e6c155060 2024-04-02T13:20:05.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.064+0000 7f1e6959a640 1 -- 172.21.15.69:0/1460898942 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e5c031830 con 0x7f1e6c155060 2024-04-02T13:20:05.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.064+0000 7f1e6959a640 1 -- 172.21.15.69:0/1460898942 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c031c00 con 0x7f1e6c155060 2024-04-02T13:20:05.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.065+0000 7f1e70e2a640 1 -- 172.21.15.69:0/1460898942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e6c155060 msgr2=0x7f1e6c155440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.065+0000 7f1e70e2a640 1 --2- 172.21.15.69:0/1460898942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e6c155060 0x7f1e6c155440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0097a0 tx=0x7f1e5c02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.065+0000 7f1e70e2a640 1 -- 172.21.15.69:0/1460898942 shutdown_connections 2024-04-02T13:20:05.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.065+0000 7f1e70e2a640 1 --2- 172.21.15.69:0/1460898942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e6c0d1030 0x7f1e6c0d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.065+0000 7f1e70e2a640 1 --2- 172.21.15.69:0/1460898942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e6c155060 0x7f1e6c155440 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.065+0000 7f1e70e2a640 1 -- 172.21.15.69:0/1460898942 >> 172.21.15.69:0/1460898942 conn(0x7f1e6c06c7c0 msgr2=0x7f1e6c06cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:05.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.065+0000 7f1e70e2a640 1 -- 172.21.15.69:0/1460898942 shutdown_connections 2024-04-02T13:20:05.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.065+0000 7f1e70e2a640 1 -- 172.21.15.69:0/1460898942 wait complete. 2024-04-02T13:20:05.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.066+0000 7f1e70e2a640 1 Processor -- start 2024-04-02T13:20:05.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.066+0000 7f1e70e2a640 1 -- start start 2024-04-02T13:20:05.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e70e2a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e6c0d1030 0x7f1e6c1e7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e70e2a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e6c155060 0x7f1e6c157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e70e2a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e6c15eba0 0x7f1e6c1f3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e70e2a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6c162630 con 0x7f1e6c0d1030 2024-04-02T13:20:05.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e70e2a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6c06ebe0 con 0x7f1e6c15eba0 2024-04-02T13:20:05.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e70e2a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1e6c0ccd60 con 0x7f1e6c155060 2024-04-02T13:20:05.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e6ad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e6c15eba0 0x7f1e6c1f3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e69d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e6c155060 0x7f1e6c157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e69d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e6c155060 0x7f1e6c157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57894/0 (socket says 172.21.15.69:57894) 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e6ad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e6c15eba0 0x7f1e6c1f3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49466/0 (socket says 172.21.15.69:49466) 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e69d9b640 1 -- 172.21.15.69:0/4169176232 learned_addr learned my addr 172.21.15.69:0/4169176232 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e6a59c640 1 --2- 172.21.15.69:0/4169176232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e6c0d1030 0x7f1e6c1e7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e69d9b640 1 -- 172.21.15.69:0/4169176232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e6c15eba0 msgr2=0x7f1e6c1f3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e69d9b640 1 --2- 172.21.15.69:0/4169176232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e6c15eba0 0x7f1e6c1f3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e69d9b640 1 -- 172.21.15.69:0/4169176232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e6c0d1030 msgr2=0x7f1e6c1e7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e69d9b640 1 --2- 172.21.15.69:0/4169176232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e6c0d1030 0x7f1e6c1e7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e69d9b640 1 -- 172.21.15.69:0/4169176232 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6c1e8500 con 0x7f1e6c155060 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e6ad9d640 1 --2- 172.21.15.69:0/4169176232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e6c15eba0 0x7f1e6c1f3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e69d9b640 1 --2- 172.21.15.69:0/4169176232 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e6c155060 0x7f1e6c157ce0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000e870 tx=0x7f1e6000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e5affd640 1 -- 172.21.15.69:0/4169176232 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60017070 con 0x7f1e6c155060 2024-04-02T13:20:05.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.067+0000 7f1e5affd640 1 -- 172.21.15.69:0/4169176232 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6000c8e0 con 0x7f1e6c155060 2024-04-02T13:20:05.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.068+0000 7f1e70e2a640 1 -- 172.21.15.69:0/4169176232 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e6c1e87e0 con 0x7f1e6c155060 2024-04-02T13:20:05.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.068+0000 7f1e5affd640 1 -- 172.21.15.69:0/4169176232 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6000ccd0 con 0x7f1e6c155060 2024-04-02T13:20:05.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.068+0000 7f1e70e2a640 1 -- 172.21.15.69:0/4169176232 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e6c1e8da0 con 0x7f1e6c155060 2024-04-02T13:20:05.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.068+0000 7f1e5affd640 1 -- 172.21.15.69:0/4169176232 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1e6001a070 con 0x7f1e6c155060 2024-04-02T13:20:05.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.068+0000 7f1e5affd640 1 -- 172.21.15.69:0/4169176232 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f1e6001c9d0 con 0x7f1e6c155060 2024-04-02T13:20:05.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.068+0000 7f1e70e2a640 1 -- 172.21.15.69:0/4169176232 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e6c068c00 con 0x7f1e6c155060 2024-04-02T13:20:05.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.070+0000 7f1e5affd640 1 -- 172.21.15.69:0/4169176232 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1e6001c450 con 0x7f1e6c155060 2024-04-02T13:20:05.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.143+0000 7f1e70e2a640 1 -- 172.21.15.69:0/4169176232 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "37e3b164-c05f-4989-9658-8866f1b512b7", "id": 5} v 0) -- 0x7f1e6c0cd420 con 0x7f1e6c155060 2024-04-02T13:20:05.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.148+0000 7f1e5affd640 1 -- 172.21.15.69:0/4169176232 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "37e3b164-c05f-4989-9658-8866f1b512b7", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f1e6001c700 con 0x7f1e6c155060 2024-04-02T13:20:05.150 INFO:teuthology.orchestra.run.smithi069.stdout:5 2024-04-02T13:20:05.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.149+0000 7f1e70e2a640 1 -- 172.21.15.69:0/4169176232 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e6c155060 msgr2=0x7f1e6c157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.149+0000 7f1e70e2a640 1 --2- 172.21.15.69:0/4169176232 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e6c155060 0x7f1e6c157ce0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000e870 tx=0x7f1e6000ed30 comp rx=0 tx=0).stop 2024-04-02T13:20:05.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.149+0000 7f1e70e2a640 1 -- 172.21.15.69:0/4169176232 shutdown_connections 2024-04-02T13:20:05.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.149+0000 7f1e70e2a640 1 --2- 172.21.15.69:0/4169176232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e6c15eba0 0x7f1e6c1f3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.149+0000 7f1e70e2a640 1 --2- 172.21.15.69:0/4169176232 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e6c155060 0x7f1e6c157ce0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.149+0000 7f1e70e2a640 1 --2- 172.21.15.69:0/4169176232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e6c0d1030 0x7f1e6c1e7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.149+0000 7f1e70e2a640 1 -- 172.21.15.69:0/4169176232 >> 172.21.15.69:0/4169176232 conn(0x7f1e6c06c7c0 msgr2=0x7f1e6c06cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:05.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.149+0000 7f1e70e2a640 1 -- 172.21.15.69:0/4169176232 shutdown_connections 2024-04-02T13:20:05.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.149+0000 7f1e70e2a640 1 -- 172.21.15.69:0/4169176232 wait complete. 2024-04-02T13:20:05.156 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:05.154+0000 7f4d38a42040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T13:20:05.161 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new a7f2fbf4-1277-4837-a6bb-f7762c4f373f 6 2024-04-02T13:20:05.162 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:05.160+0000 7fe6c6e32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T13:20:05.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.261+0000 7f058b481640 1 Processor -- start 2024-04-02T13:20:05.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.261+0000 7f058b481640 1 -- start start 2024-04-02T13:20:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f058b481640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058415c1c0 0x7f05840cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f058b481640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f05840d0390 0x7f05840d07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f058b481640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f058415c5a0 con 0x7f05840aec60 2024-04-02T13:20:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f058b481640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f05840d0d10 con 0x7f05840d0390 2024-04-02T13:20:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f058b481640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f05840af9d0 con 0x7f058415c1c0 2024-04-02T13:20:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f0589a1e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f05840aec60 0x7f05840af0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48716/0 (socket says 172.21.15.69:48716) 2024-04-02T13:20:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f0589a1e640 1 -- 172.21.15.69:0/2379008415 learned_addr learned my addr 172.21.15.69:0/2379008415 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-02T13:20:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f058921d640 1 --2- 172.21.15.69:0/2379008415 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058415c1c0 0x7f05840cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f0588a1c640 1 --2- 172.21.15.69:0/2379008415 >> v2:172.21.15.44:3300/0 conn(0x7f05840d0390 0x7f05840d07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f057bfff640 1 -- 172.21.15.69:0/2379008415 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3314723481 0 0) 0x7f058415c5a0 con 0x7f05840aec60 2024-04-02T13:20:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f057bfff640 1 -- 172.21.15.69:0/2379008415 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0570003520 con 0x7f05840aec60 2024-04-02T13:20:05.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f058921d640 1 -- 172.21.15.69:0/2379008415 >> v2:172.21.15.44:3300/0 conn(0x7f05840d0390 msgr2=0x7f05840d07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:05.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f058921d640 1 --2- 172.21.15.69:0/2379008415 >> v2:172.21.15.44:3300/0 conn(0x7f05840d0390 0x7f05840d07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f058921d640 1 -- 172.21.15.69:0/2379008415 >> v1:172.21.15.69:6789/0 conn(0x7f05840aec60 legacy=0x7f05840af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.262+0000 7f058921d640 1 -- 172.21.15.69:0/2379008415 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05840afcb0 con 0x7f058415c1c0 2024-04-02T13:20:05.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.263+0000 7f058921d640 1 --2- 172.21.15.69:0/2379008415 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058415c1c0 0x7f05840cc920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f05800097a0 tx=0x7f058002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=931aeb75cc47d48c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:05.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.263+0000 7f057bfff640 1 -- 172.21.15.69:0/2379008415 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580034070 con 0x7f058415c1c0 2024-04-02T13:20:05.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.263+0000 7f057bfff640 1 -- 172.21.15.69:0/2379008415 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0580030610 con 0x7f058415c1c0 2024-04-02T13:20:05.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.263+0000 7f057bfff640 1 -- 172.21.15.69:0/2379008415 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05800309e0 con 0x7f058415c1c0 2024-04-02T13:20:05.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.264+0000 7f058b481640 1 -- 172.21.15.69:0/2379008415 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058415c1c0 msgr2=0x7f05840cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.264+0000 7f058b481640 1 --2- 172.21.15.69:0/2379008415 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058415c1c0 0x7f05840cc920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f05800097a0 tx=0x7f058002b960 comp rx=0 tx=0).stop 2024-04-02T13:20:05.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.264+0000 7f058b481640 1 -- 172.21.15.69:0/2379008415 shutdown_connections 2024-04-02T13:20:05.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.264+0000 7f058b481640 1 --2- 172.21.15.69:0/2379008415 >> v2:172.21.15.44:3300/0 conn(0x7f05840d0390 0x7f05840d07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.264+0000 7f058b481640 1 --2- 172.21.15.69:0/2379008415 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058415c1c0 0x7f05840cc920 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.264+0000 7f058b481640 1 -- 172.21.15.69:0/2379008415 >> 172.21.15.69:0/2379008415 conn(0x7f0584073850 msgr2=0x7f058409aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.264+0000 7f058b481640 1 -- 172.21.15.69:0/2379008415 shutdown_connections 2024-04-02T13:20:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.264+0000 7f058b481640 1 -- 172.21.15.69:0/2379008415 wait complete. 2024-04-02T13:20:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.265+0000 7f058b481640 1 Processor -- start 2024-04-02T13:20:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.265+0000 7f058b481640 1 -- start start 2024-04-02T13:20:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f058b481640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05840aec60 0x7f05841582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f058b481640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05840d0390 0x7f058415c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f058b481640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058415c1c0 0x7f05841f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f058b481640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f05840b0550 con 0x7f05840aec60 2024-04-02T13:20:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f058b481640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f05840b0270 con 0x7f05840d0390 2024-04-02T13:20:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f058b481640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f05840b0a70 con 0x7f058415c1c0 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f0589a1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058415c1c0 0x7f05841f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f0588a1c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05840d0390 0x7f058415c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f0589a1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058415c1c0 0x7f05841f1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57922/0 (socket says 172.21.15.69:57922) 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f0589a1e640 1 -- 172.21.15.69:0/1356206932 learned_addr learned my addr 172.21.15.69:0/1356206932 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f058921d640 1 --2- 172.21.15.69:0/1356206932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05840aec60 0x7f05841582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f0589a1e640 1 -- 172.21.15.69:0/1356206932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05840d0390 msgr2=0x7f058415c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f0589a1e640 1 --2- 172.21.15.69:0/1356206932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05840d0390 0x7f058415c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f0589a1e640 1 -- 172.21.15.69:0/1356206932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05840aec60 msgr2=0x7f05841582f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f0589a1e640 1 --2- 172.21.15.69:0/1356206932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05840aec60 0x7f05841582f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f0589a1e640 1 -- 172.21.15.69:0/1356206932 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058415ee60 con 0x7f058415c1c0 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f058921d640 1 --2- 172.21.15.69:0/1356206932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05840aec60 0x7f05841582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f0589a1e640 1 --2- 172.21.15.69:0/1356206932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058415c1c0 0x7f05841f1af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f057c00cb40 tx=0x7f057c00a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f0579ffb640 1 -- 172.21.15.69:0/1356206932 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c017070 con 0x7f058415c1c0 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f0579ffb640 1 -- 172.21.15.69:0/1356206932 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f057c00aca0 con 0x7f058415c1c0 2024-04-02T13:20:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f0579ffb640 1 -- 172.21.15.69:0/1356206932 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c012620 con 0x7f058415c1c0 2024-04-02T13:20:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.266+0000 7f058b481640 1 -- 172.21.15.69:0/1356206932 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058415f140 con 0x7f058415c1c0 2024-04-02T13:20:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.267+0000 7f0579ffb640 1 -- 172.21.15.69:0/1356206932 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f057c01a070 con 0x7f058415c1c0 2024-04-02T13:20:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.267+0000 7f058b481640 1 -- 172.21.15.69:0/1356206932 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f058415f700 con 0x7f058415c1c0 2024-04-02T13:20:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.267+0000 7f0579ffb640 1 -- 172.21.15.69:0/1356206932 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f057c0127c0 con 0x7f058415c1c0 2024-04-02T13:20:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.268+0000 7f058b481640 1 -- 172.21.15.69:0/1356206932 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0584068c00 con 0x7f058415c1c0 2024-04-02T13:20:05.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.271+0000 7f0579ffb640 1 -- 172.21.15.69:0/1356206932 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f057c022080 con 0x7f058415c1c0 2024-04-02T13:20:05.322 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:05.320+0000 7f4d38a42040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T13:20:05.323 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:05.322+0000 7fe6c6e32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T13:20:05.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.344+0000 7f058b481640 1 -- 172.21.15.69:0/1356206932 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a7f2fbf4-1277-4837-a6bb-f7762c4f373f", "id": 6} v 0) -- 0x7f05840b7dc0 con 0x7f058415c1c0 2024-04-02T13:20:05.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.350+0000 7f0579ffb640 1 -- 172.21.15.69:0/1356206932 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a7f2fbf4-1277-4837-a6bb-f7762c4f373f", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f057c0129c0 con 0x7f058415c1c0 2024-04-02T13:20:05.352 INFO:teuthology.orchestra.run.smithi069.stdout:6 2024-04-02T13:20:05.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.351+0000 7f058b481640 1 -- 172.21.15.69:0/1356206932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058415c1c0 msgr2=0x7f05841f1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.351+0000 7f058b481640 1 --2- 172.21.15.69:0/1356206932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058415c1c0 0x7f05841f1af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f057c00cb40 tx=0x7f057c00a060 comp rx=0 tx=0).stop 2024-04-02T13:20:05.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.351+0000 7f058b481640 1 -- 172.21.15.69:0/1356206932 shutdown_connections 2024-04-02T13:20:05.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.351+0000 7f058b481640 1 --2- 172.21.15.69:0/1356206932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058415c1c0 0x7f05841f1af0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.351+0000 7f058b481640 1 --2- 172.21.15.69:0/1356206932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05840d0390 0x7f058415c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.351+0000 7f058b481640 1 --2- 172.21.15.69:0/1356206932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05840aec60 0x7f05841582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.351+0000 7f058b481640 1 -- 172.21.15.69:0/1356206932 >> 172.21.15.69:0/1356206932 conn(0x7f0584073850 msgr2=0x7f058415b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:05.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.351+0000 7f058b481640 1 -- 172.21.15.69:0/1356206932 shutdown_connections 2024-04-02T13:20:05.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.351+0000 7f058b481640 1 -- 172.21.15.69:0/1356206932 wait complete. 2024-04-02T13:20:05.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4 7 2024-04-02T13:20:05.367 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:05.365+0000 7f4d38a42040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T13:20:05.367 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:05.366+0000 7fe6c6e32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T13:20:05.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.460+0000 7ff0bf2b8640 1 Processor -- start 2024-04-02T13:20:05.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.460+0000 7ff0bf2b8640 1 -- start start 2024-04-02T13:20:05.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.461+0000 7ff0bf2b8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff0b80d2200 0x7ff0b80b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.461+0000 7ff0bf2b8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0b80d13e0 con 0x7ff0b815e550 2024-04-02T13:20:05.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.461+0000 7ff0bf2b8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0b80d0b40 con 0x7ff0b815e170 2024-04-02T13:20:05.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.461+0000 7ff0bf2b8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff0b80d0e20 con 0x7ff0b80d2200 2024-04-02T13:20:05.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.461+0000 7ff0bd855640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff0b80d2200 0x7ff0b80b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.461+0000 7ff0bd855640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff0b80d2200 0x7ff0b80b0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49498/0 (socket says 172.21.15.69:49498) 2024-04-02T13:20:05.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.461+0000 7ff0bd855640 1 -- 172.21.15.69:0/1046128724 learned_addr learned my addr 172.21.15.69:0/1046128724 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T13:20:05.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.462+0000 7ff0abfff640 1 -- 172.21.15.69:0/1046128724 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1831222779 0 0) 0x7ff0b80d13e0 con 0x7ff0b815e550 2024-04-02T13:20:05.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.462+0000 7ff0abfff640 1 -- 172.21.15.69:0/1046128724 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0a4003520 con 0x7ff0b815e550 2024-04-02T13:20:05.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.462+0000 7ff0bd855640 1 -- 172.21.15.69:0/1046128724 >> v1:172.21.15.69:6789/0 conn(0x7ff0b815e170 legacy=0x7ff0b80d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.462+0000 7ff0bd855640 1 -- 172.21.15.69:0/1046128724 >> v1:172.21.15.44:6789/0 conn(0x7ff0b815e550 legacy=0x7ff0b80d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.462+0000 7ff0bd855640 1 -- 172.21.15.69:0/1046128724 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b80d1100 con 0x7ff0b80d2200 2024-04-02T13:20:05.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.462+0000 7ff0bd855640 1 --2- 172.21.15.69:0/1046128724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0b80d2200 0x7ff0b80b0c10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009e80 tx=0x7ff0b002bfe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bff64680836057f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:05.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.462+0000 7ff0abfff640 1 -- 172.21.15.69:0/1046128724 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b002cb80 con 0x7ff0b80d2200 2024-04-02T13:20:05.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.462+0000 7ff0abfff640 1 -- 172.21.15.69:0/1046128724 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b002cd20 con 0x7ff0b80d2200 2024-04-02T13:20:05.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.462+0000 7ff0abfff640 1 -- 172.21.15.69:0/1046128724 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0034610 con 0x7ff0b80d2200 2024-04-02T13:20:05.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.463+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/1046128724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0b80d2200 msgr2=0x7ff0b80b0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.463+0000 7ff0bf2b8640 1 --2- 172.21.15.69:0/1046128724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0b80d2200 0x7ff0b80b0c10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009e80 tx=0x7ff0b002bfe0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.463+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/1046128724 shutdown_connections 2024-04-02T13:20:05.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.463+0000 7ff0bf2b8640 1 --2- 172.21.15.69:0/1046128724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0b80d2200 0x7ff0b80b0c10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.463+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/1046128724 >> 172.21.15.69:0/1046128724 conn(0x7ff0b8073850 msgr2=0x7ff0b809aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:05.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.463+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/1046128724 shutdown_connections 2024-04-02T13:20:05.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.463+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/1046128724 wait complete. 2024-04-02T13:20:05.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.464+0000 7ff0bf2b8640 1 Processor -- start 2024-04-02T13:20:05.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.464+0000 7ff0bf2b8640 1 -- start start 2024-04-02T13:20:05.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.465+0000 7ff0bf2b8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0b80d2200 0x7ff0b80c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.465+0000 7ff0bf2b8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0b815e170 0x7ff0b806fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.465+0000 7ff0bf2b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b815e550 0x7ff0b8070420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.465+0000 7ff0bf2b8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b80cc590 con 0x7ff0b815e550 2024-04-02T13:20:05.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.465+0000 7ff0bf2b8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b80b3250 con 0x7ff0b815e170 2024-04-02T13:20:05.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.465+0000 7ff0bf2b8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0b80d0770 con 0x7ff0b80d2200 2024-04-02T13:20:05.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.465+0000 7ff0bd054640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0b80d2200 0x7ff0b80c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.465+0000 7ff0bd855640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b815e550 0x7ff0b8070420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.465+0000 7ff0bd054640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0b80d2200 0x7ff0b80c9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57924/0 (socket says 172.21.15.69:57924) 2024-04-02T13:20:05.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.465+0000 7ff0bd054640 1 -- 172.21.15.69:0/2730502034 learned_addr learned my addr 172.21.15.69:0/2730502034 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T13:20:05.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.465+0000 7ff0bc853640 1 --2- 172.21.15.69:0/2730502034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0b815e170 0x7ff0b806fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0bd855640 1 -- 172.21.15.69:0/2730502034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0b80d2200 msgr2=0x7ff0b80c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0bd855640 1 --2- 172.21.15.69:0/2730502034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0b80d2200 0x7ff0b80c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0bd855640 1 -- 172.21.15.69:0/2730502034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0b815e170 msgr2=0x7ff0b806fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0bd855640 1 --2- 172.21.15.69:0/2730502034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0b815e170 0x7ff0b806fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0bd855640 1 -- 172.21.15.69:0/2730502034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b81f3f30 con 0x7ff0b815e550 2024-04-02T13:20:05.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0bd054640 1 --2- 172.21.15.69:0/2730502034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0b80d2200 0x7ff0b80c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:05.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0bc853640 1 --2- 172.21.15.69:0/2730502034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0b815e170 0x7ff0b806fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:05.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0bd855640 1 --2- 172.21.15.69:0/2730502034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b815e550 0x7ff0b8070420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0032000 tx=0x7ff0b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0a9ffb640 1 -- 172.21.15.69:0/2730502034 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0043020 con 0x7ff0b815e550 2024-04-02T13:20:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0a9ffb640 1 -- 172.21.15.69:0/2730502034 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b003e450 con 0x7ff0b815e550 2024-04-02T13:20:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/2730502034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b81f4cb0 con 0x7ff0b815e550 2024-04-02T13:20:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0a9ffb640 1 -- 172.21.15.69:0/2730502034 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0009510 con 0x7ff0b815e550 2024-04-02T13:20:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/2730502034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b81f44b0 con 0x7ff0b815e550 2024-04-02T13:20:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0a9ffb640 1 -- 172.21.15.69:0/2730502034 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff0b0046020 con 0x7ff0b815e550 2024-04-02T13:20:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.466+0000 7ff0a9ffb640 1 -- 172.21.15.69:0/2730502034 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7ff0b0048a60 con 0x7ff0b815e550 2024-04-02T13:20:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.467+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/2730502034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0b8068c00 con 0x7ff0b815e550 2024-04-02T13:20:05.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.470+0000 7ff0a9ffb640 1 -- 172.21.15.69:0/2730502034 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff0b004d030 con 0x7ff0b815e550 2024-04-02T13:20:05.530 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:05.529+0000 7fe6c6e32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T13:20:05.550 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:05.549+0000 7f4d38a42040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T13:20:05.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.564+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/2730502034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4", "id": 7} v 0) -- 0x7ff0b80ce7c0 con 0x7ff0b815e550 2024-04-02T13:20:05.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.568+0000 7ff0a9ffb640 1 -- 172.21.15.69:0/2730502034 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7ff0b002cb90 con 0x7ff0b815e550 2024-04-02T13:20:05.570 INFO:teuthology.orchestra.run.smithi069.stdout:7 2024-04-02T13:20:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.570+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/2730502034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b815e550 msgr2=0x7ff0b8070420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.570+0000 7ff0bf2b8640 1 --2- 172.21.15.69:0/2730502034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b815e550 0x7ff0b8070420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0032000 tx=0x7ff0b0004a40 comp rx=0 tx=0).stop 2024-04-02T13:20:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.570+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/2730502034 shutdown_connections 2024-04-02T13:20:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.570+0000 7ff0bf2b8640 1 --2- 172.21.15.69:0/2730502034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b815e550 0x7ff0b8070420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.570+0000 7ff0bf2b8640 1 --2- 172.21.15.69:0/2730502034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0b815e170 0x7ff0b806fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.570+0000 7ff0bf2b8640 1 --2- 172.21.15.69:0/2730502034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0b80d2200 0x7ff0b80c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.570+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/2730502034 >> 172.21.15.69:0/2730502034 conn(0x7ff0b8073850 msgr2=0x7ff0b80cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.570+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/2730502034 shutdown_connections 2024-04-02T13:20:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:05.570+0000 7ff0bf2b8640 1 -- 172.21.15.69:0/2730502034 wait complete. 2024-04-02T13:20:05.582 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-02T13:20:05.582 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-02T13:20:05.584 INFO:tasks.ceph.osd.0:Started 2024-04-02T13:20:05.584 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-02T13:20:05.585 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-02T13:20:05.588 INFO:tasks.ceph.osd.1:Started 2024-04-02T13:20:05.589 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-02T13:20:05.589 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-02T13:20:05.591 INFO:tasks.ceph.osd.2:Started 2024-04-02T13:20:05.591 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-02T13:20:05.591 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-02T13:20:05.593 INFO:tasks.ceph.osd.3:Started 2024-04-02T13:20:05.593 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-02T13:20:05.593 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-02T13:20:05.625 INFO:tasks.ceph.osd.4:Started 2024-04-02T13:20:05.625 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-02T13:20:05.625 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-02T13:20:05.627 INFO:tasks.ceph.osd.5:Started 2024-04-02T13:20:05.627 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-02T13:20:05.627 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-02T13:20:05.629 INFO:tasks.ceph.osd.6:Started 2024-04-02T13:20:05.631 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-02T13:20:05.631 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-02T13:20:05.633 INFO:tasks.ceph.osd.7:Started 2024-04-02T13:20:05.634 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:05.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.747+0000 7faa8f752640 1 Processor -- start 2024-04-02T13:20:05.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.747+0000 7faa8f752640 1 -- start start 2024-04-02T13:20:05.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8f752640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faa88156770 0x7faa88072660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8f752640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8806be20 0x7faa880b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8f752640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8815c6d0 con 0x7faa88071db0 2024-04-02T13:20:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8f752640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7faa88072ba0 con 0x7faa88156770 2024-04-02T13:20:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8f752640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faa88156b50 con 0x7faa8806be20 2024-04-02T13:20:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8d4ee640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faa88156770 0x7faa88072660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8dcef640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8806be20 0x7faa880b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8d4ee640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faa88156770 0x7faa88072660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:50530/0 (socket says 172.21.15.44:50530) 2024-04-02T13:20:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8d4ee640 1 -- 172.21.15.44:0/3746759561 learned_addr learned my addr 172.21.15.44:0/3746759561 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8dcef640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8806be20 0x7faa880b49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:35902/0 (socket says 172.21.15.44:35902) 2024-04-02T13:20:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8dcef640 1 -- 172.21.15.44:0/3746759561 >> v2:172.21.15.69:3300/0 conn(0x7faa88156770 msgr2=0x7faa88072660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8dcef640 1 --2- 172.21.15.44:0/3746759561 >> v2:172.21.15.69:3300/0 conn(0x7faa88156770 0x7faa88072660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8dcef640 1 -- 172.21.15.44:0/3746759561 >> v1:172.21.15.69:6789/0 conn(0x7faa88071db0 legacy=0x7faa8806b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8dcef640 1 -- 172.21.15.44:0/3746759561 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8815c950 con 0x7faa8806be20 2024-04-02T13:20:05.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.748+0000 7faa8d4ee640 1 --2- 172.21.15.44:0/3746759561 >> v2:172.21.15.69:3300/0 conn(0x7faa88156770 0x7faa88072660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:05.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.749+0000 7faa8dcef640 1 --2- 172.21.15.44:0/3746759561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8806be20 0x7faa880b49e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faa840097a0 tx=0x7faa8402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b68048da6675f68 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:05.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.749+0000 7faa7bfff640 1 -- 172.21.15.44:0/3746759561 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa84035020 con 0x7faa8806be20 2024-04-02T13:20:05.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.749+0000 7faa7bfff640 1 -- 172.21.15.44:0/3746759561 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa84031830 con 0x7faa8806be20 2024-04-02T13:20:05.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.749+0000 7faa7bfff640 1 -- 172.21.15.44:0/3746759561 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa84031c00 con 0x7faa8806be20 2024-04-02T13:20:05.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.750+0000 7faa8f752640 1 -- 172.21.15.44:0/3746759561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8806be20 msgr2=0x7faa880b49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.750+0000 7faa8f752640 1 --2- 172.21.15.44:0/3746759561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8806be20 0x7faa880b49e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faa840097a0 tx=0x7faa8402b8d0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.750+0000 7faa8f752640 1 -- 172.21.15.44:0/3746759561 shutdown_connections 2024-04-02T13:20:05.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.750+0000 7faa8f752640 1 --2- 172.21.15.44:0/3746759561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8806be20 0x7faa880b49e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.750+0000 7faa8f752640 1 --2- 172.21.15.44:0/3746759561 >> v2:172.21.15.69:3300/0 conn(0x7faa88156770 0x7faa88072660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.750+0000 7faa8f752640 1 -- 172.21.15.44:0/3746759561 >> 172.21.15.44:0/3746759561 conn(0x7faa880749a0 msgr2=0x7faa88074db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:05.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.750+0000 7faa8f752640 1 -- 172.21.15.44:0/3746759561 shutdown_connections 2024-04-02T13:20:05.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.750+0000 7faa8f752640 1 -- 172.21.15.44:0/3746759561 wait complete. 2024-04-02T13:20:05.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.751+0000 7faa8f752640 1 Processor -- start 2024-04-02T13:20:05.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.751+0000 7faa8f752640 1 -- start start 2024-04-02T13:20:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8f752640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa8806be20 0x7faa8814eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8f752640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa88071db0 0x7faa881519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8f752640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa88156770 0x7faa881547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8f752640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faa8815d1f0 con 0x7faa88071db0 2024-04-02T13:20:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8f752640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faa8815cf10 con 0x7faa8806be20 2024-04-02T13:20:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8f752640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faa8815d4d0 con 0x7faa88156770 2024-04-02T13:20:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8cced640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa88071db0 0x7faa881519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8d4ee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa8806be20 0x7faa8814eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8cced640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa88071db0 0x7faa881519e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56206/0 (socket says 172.21.15.44:56206) 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8cced640 1 -- 172.21.15.44:0/4171801417 learned_addr learned my addr 172.21.15.44:0/4171801417 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8dcef640 1 --2- 172.21.15.44:0/4171801417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa88156770 0x7faa881547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8dcef640 1 -- 172.21.15.44:0/4171801417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa8806be20 msgr2=0x7faa8814eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8dcef640 1 --2- 172.21.15.44:0/4171801417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa8806be20 0x7faa8814eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8dcef640 1 -- 172.21.15.44:0/4171801417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa88071db0 msgr2=0x7faa881519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8dcef640 1 --2- 172.21.15.44:0/4171801417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa88071db0 0x7faa881519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8dcef640 1 -- 172.21.15.44:0/4171801417 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8814f760 con 0x7faa88156770 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.752+0000 7faa8cced640 1 --2- 172.21.15.44:0/4171801417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa88071db0 0x7faa881519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.753+0000 7faa8dcef640 1 --2- 172.21.15.44:0/4171801417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa88156770 0x7faa881547b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faa84033000 tx=0x7faa840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.753+0000 7faa79ffb640 1 -- 172.21.15.44:0/4171801417 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa84035040 con 0x7faa88156770 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.753+0000 7faa79ffb640 1 -- 172.21.15.44:0/4171801417 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa8403b840 con 0x7faa88156770 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.753+0000 7faa79ffb640 1 -- 172.21.15.44:0/4171801417 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8403bc10 con 0x7faa88156770 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.753+0000 7faa8f752640 1 -- 172.21.15.44:0/4171801417 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8814fa40 con 0x7faa88156770 2024-04-02T13:20:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.753+0000 7faa8f752640 1 -- 172.21.15.44:0/4171801417 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faa88156d90 con 0x7faa88156770 2024-04-02T13:20:05.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.753+0000 7faa79ffb640 1 -- 172.21.15.44:0/4171801417 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7faa84038020 con 0x7faa88156770 2024-04-02T13:20:05.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.753+0000 7faa79ffb640 1 -- 172.21.15.44:0/4171801417 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7faa84050ac0 con 0x7faa88156770 2024-04-02T13:20:05.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.754+0000 7faa8f752640 1 -- 172.21.15.44:0/4171801417 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa88068c00 con 0x7faa88156770 2024-04-02T13:20:05.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.756+0000 7faa79ffb640 1 -- 172.21.15.44:0/4171801417 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7faa84050cc0 con 0x7faa88156770 2024-04-02T13:20:05.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.818+0000 7faa8f752640 1 -- 172.21.15.44:0/4171801417 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa880b5c90 con 0x7faa88156770 2024-04-02T13:20:05.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.819+0000 7faa79ffb640 1 -- 172.21.15.44:0/4171801417 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7faa84050cc0 con 0x7faa88156770 2024-04-02T13:20:05.820 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:05.820 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":10,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:05.566313+0000","last_up_change":"0.000000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.820+0000 7faa8f752640 1 -- 172.21.15.44:0/4171801417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa88156770 msgr2=0x7faa881547b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.820+0000 7faa8f752640 1 --2- 172.21.15.44:0/4171801417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa88156770 0x7faa881547b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faa84033000 tx=0x7faa840046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.820+0000 7faa8f752640 1 -- 172.21.15.44:0/4171801417 shutdown_connections 2024-04-02T13:20:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.820+0000 7faa8f752640 1 --2- 172.21.15.44:0/4171801417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa88156770 0x7faa881547b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.820+0000 7faa8f752640 1 --2- 172.21.15.44:0/4171801417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa88071db0 0x7faa881519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.820+0000 7faa8f752640 1 --2- 172.21.15.44:0/4171801417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa8806be20 0x7faa8814eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.820+0000 7faa8f752640 1 -- 172.21.15.44:0/4171801417 >> 172.21.15.44:0/4171801417 conn(0x7faa880749a0 msgr2=0x7faa8814e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.820+0000 7faa8f752640 1 -- 172.21.15.44:0/4171801417 shutdown_connections 2024-04-02T13:20:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:05.820+0000 7faa8f752640 1 -- 172.21.15.44:0/4171801417 wait complete. 2024-04-02T13:20:05.831 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-02T13:20:05.832 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-02T13:20:06.195 INFO:tasks.ceph.osd.0.smithi044.stderr:2024-04-02T13:20:06.194+0000 7f33519bd640 -1 Falling back to public interface 2024-04-02T13:20:06.196 INFO:tasks.ceph.osd.1.smithi044.stderr:2024-04-02T13:20:06.194+0000 7f0e99d38640 -1 Falling back to public interface 2024-04-02T13:20:06.196 INFO:tasks.ceph.osd.0.smithi044.stderr:2024-04-02T13:20:06.194+0000 7f335096c640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-04-02T13:20:06.196 INFO:tasks.ceph.osd.1.smithi044.stderr:2024-04-02T13:20:06.195+0000 7f0e98ce7640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6804/0: (98) Address already in use 2024-04-02T13:20:06.204 INFO:tasks.ceph.osd.2.smithi044.stderr:2024-04-02T13:20:06.203+0000 7f3fad0a0640 -1 Falling back to public interface 2024-04-02T13:20:06.208 INFO:tasks.ceph.osd.3.smithi044.stderr:2024-04-02T13:20:06.207+0000 7f272b4ce640 -1 Falling back to public interface 2024-04-02T13:20:06.230 INFO:tasks.ceph.osd.5.smithi069.stderr:2024-04-02T13:20:06.228+0000 7ff2d349b640 -1 Falling back to public interface 2024-04-02T13:20:06.231 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-04-02T13:20:06.228+0000 7f482296d640 -1 Falling back to public interface 2024-04-02T13:20:06.231 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-04-02T13:20:06.228+0000 7f482191c640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-04-02T13:20:06.231 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-04-02T13:20:06.229+0000 7f482191c640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6810/0: (98) Address already in use 2024-04-02T13:20:06.238 INFO:tasks.ceph.osd.4.smithi069.stderr:2024-04-02T13:20:06.237+0000 7fde40213640 -1 Falling back to public interface 2024-04-02T13:20:06.246 INFO:tasks.ceph.osd.7.smithi069.stderr:2024-04-02T13:20:06.245+0000 7f964ba1e640 -1 Falling back to public interface 2024-04-02T13:20:06.407 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:06.406+0000 7fe6c6e32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T13:20:06.415 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:06.414+0000 7f4d38a42040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T13:20:06.638 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:06.694 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:06.693+0000 7f4d38a42040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T13:20:06.697 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:06.696+0000 7fe6c6e32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T13:20:06.729 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.727+0000 7fa2c2196640 1 Processor -- start 2024-04-02T13:20:06.730 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.727+0000 7fa2c2196640 1 -- start start 2024-04-02T13:20:06.730 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2c2196640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2bc15c210 0x7fa2bc0c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:06.730 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2c2196640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa2bc0d0450 0x7fa2bc0d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:06.730 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2c2196640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2bc15c5f0 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.730 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2c2196640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa2bc0d0dd0 con 0x7fa2bc0d0450 2024-04-02T13:20:06.730 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2c2196640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa2bc0ccb20 con 0x7fa2bc15c210 2024-04-02T13:20:06.730 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2baffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa2bc0d0450 0x7fa2bc0d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:06.730 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2baffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa2bc0d0450 0x7fa2bc0d0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56212/0 (socket says 172.21.15.44:56212) 2024-04-02T13:20:06.730 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2baffd640 1 -- 172.21.15.44:0/1606254094 learned_addr learned my addr 172.21.15.44:0/1606254094 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:06.730 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2bbfff640 1 --1- 172.21.15.44:0/1606254094 >> v1:172.21.15.44:6789/0 conn(0x7fa2bc15fdc0 0x7fa2bc160250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33426/0 (socket says 172.21.15.44:33426) 2024-04-02T13:20:06.730 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2bb7fe640 1 --2- 172.21.15.44:0/1606254094 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2bc15c210 0x7fa2bc0c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:06.731 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2baffd640 1 -- 172.21.15.44:0/1606254094 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2bc15c210 msgr2=0x7fa2bc0c59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:06.731 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2baffd640 1 --2- 172.21.15.44:0/1606254094 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2bc15c210 0x7fa2bc0c59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:06.731 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2baffd640 1 -- 172.21.15.44:0/1606254094 >> v1:172.21.15.44:6789/0 conn(0x7fa2bc15fdc0 legacy=0x7fa2bc160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:06.731 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2baffd640 1 -- 172.21.15.44:0/1606254094 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2bc0cce00 con 0x7fa2bc0d0450 2024-04-02T13:20:06.731 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.728+0000 7fa2ba7fc640 1 -- 172.21.15.44:0/1606254094 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 908701734 0 0) 0x7fa2bc15c5f0 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.732 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.729+0000 7fa2baffd640 1 --2- 172.21.15.44:0/1606254094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc0d0450 0x7fa2bc0d0890 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4009870 tx=0x7fa2a402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d590fe7bee266f66 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:06.732 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.729+0000 7fa2ba7fc640 1 -- 172.21.15.44:0/1606254094 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4034070 con 0x7fa2bc0d0450 2024-04-02T13:20:06.732 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.729+0000 7fa2ba7fc640 1 -- 172.21.15.44:0/1606254094 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2a4030840 con 0x7fa2bc0d0450 2024-04-02T13:20:06.733 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.729+0000 7fa2ba7fc640 1 -- 172.21.15.44:0/1606254094 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4030c10 con 0x7fa2bc0d0450 2024-04-02T13:20:06.733 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.730+0000 7fa2c2196640 1 -- 172.21.15.44:0/1606254094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc0d0450 msgr2=0x7fa2bc0d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:06.733 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.730+0000 7fa2c2196640 1 --2- 172.21.15.44:0/1606254094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc0d0450 0x7fa2bc0d0890 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4009870 tx=0x7fa2a402b980 comp rx=0 tx=0).stop 2024-04-02T13:20:06.733 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.730+0000 7fa2c2196640 1 -- 172.21.15.44:0/1606254094 shutdown_connections 2024-04-02T13:20:06.733 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.730+0000 7fa2c2196640 1 --2- 172.21.15.44:0/1606254094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc0d0450 0x7fa2bc0d0890 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:06.733 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.730+0000 7fa2c2196640 1 --2- 172.21.15.44:0/1606254094 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2bc15c210 0x7fa2bc0c59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:06.733 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.730+0000 7fa2c2196640 1 -- 172.21.15.44:0/1606254094 >> 172.21.15.44:0/1606254094 conn(0x7fa2bc09aa70 msgr2=0x7fa2bc09ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:06.733 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.730+0000 7fa2c2196640 1 -- 172.21.15.44:0/1606254094 shutdown_connections 2024-04-02T13:20:06.733 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.730+0000 7fa2c2196640 1 -- 172.21.15.44:0/1606254094 wait complete. 2024-04-02T13:20:06.733 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.731+0000 7fa2c2196640 1 Processor -- start 2024-04-02T13:20:06.733 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.731+0000 7fa2c2196640 1 -- start start 2024-04-02T13:20:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2c2196640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2bc0d0450 0x7fa2bc1e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2c2196640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2bc15c210 0x7fa2bc0b6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2c2196640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc15fdc0 0x7fa2bc0c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2c2196640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc160b10 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2c2196640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc0cd3c0 con 0x7fa2bc0d0450 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2c2196640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa2bc0cd8e0 con 0x7fa2bc15c210 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2bbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc15fdc0 0x7fa2bc0c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2bbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc15fdc0 0x7fa2bc0c8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56226/0 (socket says 172.21.15.44:56226) 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2bbfff640 1 -- 172.21.15.44:0/4202937373 learned_addr learned my addr 172.21.15.44:0/4202937373 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2baffd640 1 --2- 172.21.15.44:0/4202937373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2bc15c210 0x7fa2bc0b6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2bb7fe640 1 --2- 172.21.15.44:0/4202937373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2bc0d0450 0x7fa2bc1e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2bbfff640 1 -- 172.21.15.44:0/4202937373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2bc15c210 msgr2=0x7fa2bc0b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2bbfff640 1 --2- 172.21.15.44:0/4202937373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2bc15c210 0x7fa2bc0b6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2bbfff640 1 -- 172.21.15.44:0/4202937373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2bc0d0450 msgr2=0x7fa2bc1e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2bbfff640 1 --2- 172.21.15.44:0/4202937373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2bc0d0450 0x7fa2bc1e7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2bbfff640 1 -- 172.21.15.44:0/4202937373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2bc0b7430 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.737 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.732+0000 7fa2baffd640 1 --2- 172.21.15.44:0/4202937373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2bc15c210 0x7fa2bc0b6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:06.737 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.733+0000 7fa2bbfff640 1 --2- 172.21.15.44:0/4202937373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc15fdc0 0x7fa2bc0c8f40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400c700 tx=0x7fa2b400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:06.737 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.733+0000 7fa2b3fff640 1 -- 172.21.15.44:0/4202937373 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4018070 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.737 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.733+0000 7fa2b3fff640 1 -- 172.21.15.44:0/4202937373 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2b400ce10 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.737 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.733+0000 7fa2c2196640 1 -- 172.21.15.44:0/4202937373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2bc0c9820 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.737 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.733+0000 7fa2b3fff640 1 -- 172.21.15.44:0/4202937373 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b400a860 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.737 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.733+0000 7fa2c2196640 1 -- 172.21.15.44:0/4202937373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2bc0c9de0 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.737 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.733+0000 7fa2b3fff640 1 -- 172.21.15.44:0/4202937373 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa2b401b070 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.737 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.733+0000 7fa2b3fff640 1 -- 172.21.15.44:0/4202937373 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fa2b4007880 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.737 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.734+0000 7fa2c2196640 1 -- 172.21.15.44:0/4202937373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2bc0ca5a0 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.737 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.736+0000 7fa2b3fff640 1 -- 172.21.15.44:0/4202937373 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa2bc0ca5a0 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.746 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:06.745+0000 7f4d38a42040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T13:20:06.753 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:06.752+0000 7fe6c6e32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T13:20:06.799 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.798+0000 7fa2c2196640 1 -- 172.21.15.44:0/4202937373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2bc04f8d0 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.799 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.798+0000 7fa2b3fff640 1 -- 172.21.15.44:0/4202937373 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fa2b40070d0 con 0x7fa2bc15fdc0 2024-04-02T13:20:06.799 INFO:teuthology.misc.health.smithi044.stdout: 2024-04-02T13:20:06.799 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:05.566313+0000","last_up_change":"0.000000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:06.800 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.799+0000 7fa2c2196640 1 -- 172.21.15.44:0/4202937373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc15fdc0 msgr2=0x7fa2bc0c8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:06.800 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.799+0000 7fa2c2196640 1 --2- 172.21.15.44:0/4202937373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc15fdc0 0x7fa2bc0c8f40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400c700 tx=0x7fa2b400cbc0 comp rx=0 tx=0).stop 2024-04-02T13:20:06.801 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.799+0000 7fa2c2196640 1 -- 172.21.15.44:0/4202937373 shutdown_connections 2024-04-02T13:20:06.801 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.799+0000 7fa2c2196640 1 --2- 172.21.15.44:0/4202937373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc15fdc0 0x7fa2bc0c8f40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:06.801 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.799+0000 7fa2c2196640 1 --2- 172.21.15.44:0/4202937373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2bc15c210 0x7fa2bc0b6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:06.801 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.799+0000 7fa2c2196640 1 --2- 172.21.15.44:0/4202937373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2bc0d0450 0x7fa2bc1e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:06.801 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.799+0000 7fa2c2196640 1 -- 172.21.15.44:0/4202937373 >> 172.21.15.44:0/4202937373 conn(0x7fa2bc09aa70 msgr2=0x7fa2bc0cdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:06.801 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.799+0000 7fa2c2196640 1 -- 172.21.15.44:0/4202937373 shutdown_connections 2024-04-02T13:20:06.801 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:06.799+0000 7fa2c2196640 1 -- 172.21.15.44:0/4202937373 wait complete. 2024-04-02T13:20:06.809 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-02T13:20:06.864 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:06.863+0000 7f4d38a42040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T13:20:06.876 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-04-02T13:20:06.875+0000 7fe6c6e32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T13:20:07.881 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:07.879+0000 7f4cd95d5640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-02T13:20:07.881 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-04-02T13:20:07.879+0000 7f4cd95d5640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-02T13:20:09.658 INFO:tasks.ceph.osd.1.smithi044.stderr:2024-04-02T13:20:09.657+0000 7f0e99d38640 -1 osd.1 0 log_to_monitors true 2024-04-02T13:20:09.661 INFO:tasks.ceph.osd.0.smithi044.stderr:2024-04-02T13:20:09.660+0000 7f33519bd640 -1 osd.0 0 log_to_monitors true 2024-04-02T13:20:09.677 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-04-02T13:20:09.675+0000 7f482296d640 -1 osd.6 0 log_to_monitors true 2024-04-02T13:20:09.715 INFO:tasks.ceph.osd.3.smithi044.stderr:2024-04-02T13:20:09.714+0000 7f272b4ce640 -1 osd.3 0 log_to_monitors true 2024-04-02T13:20:09.724 INFO:tasks.ceph.osd.2.smithi044.stderr:2024-04-02T13:20:09.723+0000 7f3fad0a0640 -1 osd.2 0 log_to_monitors true 2024-04-02T13:20:09.725 INFO:tasks.ceph.osd.5.smithi069.stderr:2024-04-02T13:20:09.723+0000 7ff2d349b640 -1 osd.5 0 log_to_monitors true 2024-04-02T13:20:09.752 INFO:tasks.ceph.osd.7.smithi069.stderr:2024-04-02T13:20:09.750+0000 7f964ba1e640 -1 osd.7 0 log_to_monitors true 2024-04-02T13:20:09.758 INFO:tasks.ceph.osd.4.smithi069.stderr:2024-04-02T13:20:09.757+0000 7fde40213640 -1 osd.4 0 log_to_monitors true 2024-04-02T13:20:12.831 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-04-02T13:20:12.828+0000 7f481e903640 -1 osd.6 0 waiting for initial osdmap 2024-04-02T13:20:12.845 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-04-02T13:20:12.844+0000 7f4816723640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T13:20:12.894 INFO:tasks.ceph.osd.2.smithi044.stderr:2024-04-02T13:20:12.892+0000 7f3fa9036640 -1 osd.2 0 waiting for initial osdmap 2024-04-02T13:20:12.906 INFO:tasks.ceph.osd.2.smithi044.stderr:2024-04-02T13:20:12.905+0000 7f3fa0e56640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T13:20:12.955 INFO:tasks.ceph.osd.1.smithi044.stderr:2024-04-02T13:20:12.954+0000 7f0e95cce640 -1 osd.1 0 waiting for initial osdmap 2024-04-02T13:20:12.960 INFO:tasks.ceph.osd.7.smithi069.stderr:2024-04-02T13:20:12.957+0000 7f964740c640 -1 osd.7 0 waiting for initial osdmap 2024-04-02T13:20:12.968 INFO:tasks.ceph.osd.7.smithi069.stderr:2024-04-02T13:20:12.967+0000 7f963f23d640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T13:20:12.970 INFO:tasks.ceph.osd.1.smithi044.stderr:2024-04-02T13:20:12.969+0000 7f0e8daee640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T13:20:12.976 INFO:tasks.ceph.osd.4.smithi069.stderr:2024-04-02T13:20:12.975+0000 7fde3bae5640 -1 osd.4 0 waiting for initial osdmap 2024-04-02T13:20:12.988 INFO:tasks.ceph.osd.0.smithi044.stderr:2024-04-02T13:20:12.987+0000 7f334d953640 -1 osd.0 0 waiting for initial osdmap 2024-04-02T13:20:12.989 INFO:tasks.ceph.osd.5.smithi069.stderr:2024-04-02T13:20:12.988+0000 7ff2cf431640 -1 osd.5 0 waiting for initial osdmap 2024-04-02T13:20:12.990 INFO:tasks.ceph.osd.4.smithi069.stderr:2024-04-02T13:20:12.989+0000 7fde33905640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T13:20:12.997 INFO:tasks.ceph.osd.0.smithi044.stderr:2024-04-02T13:20:12.996+0000 7f3345773640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T13:20:13.002 INFO:tasks.ceph.osd.5.smithi069.stderr:2024-04-02T13:20:13.001+0000 7ff2c7251640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T13:20:13.009 INFO:tasks.ceph.osd.3.smithi044.stderr:2024-04-02T13:20:13.008+0000 7f2727464640 -1 osd.3 0 waiting for initial osdmap 2024-04-02T13:20:13.023 INFO:tasks.ceph.osd.3.smithi044.stderr:2024-04-02T13:20:13.022+0000 7f271f284640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T13:20:13.616 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:13.716 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.715+0000 7f735558d640 1 Processor -- start 2024-04-02T13:20:13.717 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.715+0000 7f735558d640 1 -- start start 2024-04-02T13:20:13.718 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f735558d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73500d2dd0 0x7f73500d3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:13.718 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f735558d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7350161f40 0x7f7350162390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:13.718 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f735558d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73500d3770 con 0x7f735015e3c0 2024-04-02T13:20:13.718 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f735558d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7350162ea0 con 0x7f7350161f40 2024-04-02T13:20:13.718 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f735558d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f73500d2330 con 0x7f73500d2dd0 2024-04-02T13:20:13.718 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f734f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7350161f40 0x7f7350162390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:13.719 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f734f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7350161f40 0x7f7350162390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56264/0 (socket says 172.21.15.44:56264) 2024-04-02T13:20:13.719 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f734f7fe640 1 -- 172.21.15.44:0/3392990084 learned_addr learned my addr 172.21.15.44:0/3392990084 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:13.719 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f734e7fc640 1 --2- 172.21.15.44:0/3392990084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73500d2dd0 0x7f73500d3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:13.719 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f734f7fe640 1 -- 172.21.15.44:0/3392990084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73500d2dd0 msgr2=0x7f73500d3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:13.719 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f734f7fe640 1 --2- 172.21.15.44:0/3392990084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73500d2dd0 0x7f73500d3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:13.719 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f734f7fe640 1 -- 172.21.15.44:0/3392990084 >> v1:172.21.15.69:6789/0 conn(0x7f735015e3c0 legacy=0x7f73500b74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:13.719 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f734f7fe640 1 -- 172.21.15.44:0/3392990084 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7350162bc0 con 0x7f7350161f40 2024-04-02T13:20:13.719 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.716+0000 7f734dffb640 1 -- 172.21.15.44:0/3392990084 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2015296267 0 0) 0x7f73500d3770 con 0x7f735015e3c0 2024-04-02T13:20:13.719 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.717+0000 7f734f7fe640 1 --2- 172.21.15.44:0/3392990084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7350161f40 0x7f7350162390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7340009e50 tx=0x7f734002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814dbca1280139b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:13.720 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.717+0000 7f734dffb640 1 -- 172.21.15.44:0/3392990084 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734002ca70 con 0x7f7350161f40 2024-04-02T13:20:13.721 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.717+0000 7f734dffb640 1 -- 172.21.15.44:0/3392990084 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f734002cc10 con 0x7f7350161f40 2024-04-02T13:20:13.721 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.717+0000 7f734dffb640 1 -- 172.21.15.44:0/3392990084 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73400343f0 con 0x7f7350161f40 2024-04-02T13:20:13.721 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.718+0000 7f735558d640 1 -- 172.21.15.44:0/3392990084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7350161f40 msgr2=0x7f7350162390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:13.721 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.718+0000 7f735558d640 1 --2- 172.21.15.44:0/3392990084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7350161f40 0x7f7350162390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7340009e50 tx=0x7f734002bed0 comp rx=0 tx=0).stop 2024-04-02T13:20:13.721 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.718+0000 7f735558d640 1 -- 172.21.15.44:0/3392990084 shutdown_connections 2024-04-02T13:20:13.721 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.718+0000 7f735558d640 1 --2- 172.21.15.44:0/3392990084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7350161f40 0x7f7350162390 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:13.721 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.718+0000 7f735558d640 1 --2- 172.21.15.44:0/3392990084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73500d2dd0 0x7f73500d3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:13.721 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.718+0000 7f735558d640 1 -- 172.21.15.44:0/3392990084 >> 172.21.15.44:0/3392990084 conn(0x7f7350072140 msgr2=0x7f735006b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:13.721 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.718+0000 7f735558d640 1 -- 172.21.15.44:0/3392990084 shutdown_connections 2024-04-02T13:20:13.721 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.718+0000 7f735558d640 1 -- 172.21.15.44:0/3392990084 wait complete. 2024-04-02T13:20:13.721 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.719+0000 7f735558d640 1 Processor -- start 2024-04-02T13:20:13.721 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.719+0000 7f735558d640 1 -- start start 2024-04-02T13:20:13.723 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.720+0000 7f735558d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73500d2dd0 0x7f735014cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:13.723 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.720+0000 7f735558d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f735015e3c0 0x7f735015f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:13.724 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.720+0000 7f735558d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7350161f40 0x7f73500b8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:13.724 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.720+0000 7f735558d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f73501628d0 con 0x7f73500d2dd0 2024-04-02T13:20:13.724 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.720+0000 7f735558d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f73500d28b0 con 0x7f7350161f40 2024-04-02T13:20:13.724 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.720+0000 7f735558d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f735015e7a0 con 0x7f735015e3c0 2024-04-02T13:20:13.724 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.720+0000 7f734effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73500d2dd0 0x7f735014cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:13.724 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.720+0000 7f734effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73500d2dd0 0x7f735014cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56276/0 (socket says 172.21.15.44:56276) 2024-04-02T13:20:13.724 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.720+0000 7f734effd640 1 -- 172.21.15.44:0/3346013587 learned_addr learned my addr 172.21.15.44:0/3346013587 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:13.724 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.720+0000 7f734e7fc640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f735015e3c0 0x7f735015f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:13.724 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.720+0000 7f734f7fe640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7350161f40 0x7f73500b8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:13.724 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.721+0000 7f734effd640 1 -- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f735015e3c0 msgr2=0x7f735015f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:13.725 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.721+0000 7f734effd640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f735015e3c0 0x7f735015f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:13.725 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.721+0000 7f734effd640 1 -- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7350161f40 msgr2=0x7f73500b8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:13.725 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.721+0000 7f734effd640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7350161f40 0x7f73500b8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:13.725 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.721+0000 7f734effd640 1 -- 172.21.15.44:0/3346013587 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73500b7680 con 0x7f73500d2dd0 2024-04-02T13:20:13.725 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.721+0000 7f734effd640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73500d2dd0 0x7f735014cf20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f734800c4b0 tx=0x7f734800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:13.725 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.721+0000 7f73377fe640 1 -- 172.21.15.44:0/3346013587 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7348017070 con 0x7f73500d2dd0 2024-04-02T13:20:13.725 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.721+0000 7f73377fe640 1 -- 172.21.15.44:0/3346013587 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7348005480 con 0x7f73500d2dd0 2024-04-02T13:20:13.725 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.721+0000 7f735558d640 1 -- 172.21.15.44:0/3346013587 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73500b7960 con 0x7f73500d2dd0 2024-04-02T13:20:13.725 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.721+0000 7f73377fe640 1 -- 172.21.15.44:0/3346013587 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7348003830 con 0x7f73500d2dd0 2024-04-02T13:20:13.725 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.721+0000 7f735558d640 1 -- 172.21.15.44:0/3346013587 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73500b7f20 con 0x7f73500d2dd0 2024-04-02T13:20:13.725 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.722+0000 7f735558d640 1 -- 172.21.15.44:0/3346013587 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f735004f8d0 con 0x7f73500d2dd0 2024-04-02T13:20:13.725 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.723+0000 7f73377fe640 1 -- 172.21.15.44:0/3346013587 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f734801a070 con 0x7f73500d2dd0 2024-04-02T13:20:13.726 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.723+0000 7f73377fe640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f733c06c860 0x7f733c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:13.726 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.723+0000 7f73377fe640 1 -- 172.21.15.44:0/3346013587 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f734808a020 con 0x7f73500d2dd0 2024-04-02T13:20:13.726 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.723+0000 7f734e7fc640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f733c06c860 0x7f733c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:13.726 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.724+0000 7f734e7fc640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f733c06c860 0x7f733c06ed20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f73500b6b30 tx=0x7f7344009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:13.729 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.728+0000 7f73377fe640 1 -- 172.21.15.44:0/3346013587 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7348058c70 con 0x7f73500d2dd0 2024-04-02T13:20:13.921 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.920+0000 7f735558d640 1 -- 172.21.15.44:0/3346013587 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7350154a90 con 0x7f73500d2dd0 2024-04-02T13:20:13.922 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.921+0000 7f73377fe640 1 -- 172.21.15.44:0/3346013587 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10861 (secure 0 0 0) 0x7f734805cf70 con 0x7f73500d2dd0 2024-04-02T13:20:13.922 INFO:teuthology.misc.health.smithi044.stdout: 2024-04-02T13:20:13.923 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":13,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:13.833429+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:13.924 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.923+0000 7f735558d640 1 -- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f733c06c860 msgr2=0x7f733c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:13.924 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.923+0000 7f735558d640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f733c06c860 0x7f733c06ed20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f73500b6b30 tx=0x7f7344009190 comp rx=0 tx=0).stop 2024-04-02T13:20:13.924 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.923+0000 7f735558d640 1 -- 172.21.15.44:0/3346013587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73500d2dd0 msgr2=0x7f735014cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:13.925 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.923+0000 7f735558d640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73500d2dd0 0x7f735014cf20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f734800c4b0 tx=0x7f734800c970 comp rx=0 tx=0).stop 2024-04-02T13:20:13.925 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.923+0000 7f735558d640 1 -- 172.21.15.44:0/3346013587 shutdown_connections 2024-04-02T13:20:13.925 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.923+0000 7f735558d640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f733c06c860 0x7f733c06ed20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:13.925 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.923+0000 7f735558d640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7350161f40 0x7f73500b8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:13.925 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.923+0000 7f735558d640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f735015e3c0 0x7f735015f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:13.925 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.923+0000 7f735558d640 1 --2- 172.21.15.44:0/3346013587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73500d2dd0 0x7f735014cf20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:13.925 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.923+0000 7f735558d640 1 -- 172.21.15.44:0/3346013587 >> 172.21.15.44:0/3346013587 conn(0x7f7350072140 msgr2=0x7f73500d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:13.925 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.923+0000 7f735558d640 1 -- 172.21.15.44:0/3346013587 shutdown_connections 2024-04-02T13:20:13.925 INFO:teuthology.misc.health.smithi044.stderr:2024-04-02T13:20:13.923+0000 7f735558d640 1 -- 172.21.15.44:0/3346013587 wait complete. 2024-04-02T13:20:13.934 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-02T13:20:13.934 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-02T13:20:13.934 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T13:20:13.934 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T13:20:13.937 INFO:tasks.ceph.mds.a:Started 2024-04-02T13:20:13.937 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T13:20:13.937 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T13:20:13.939 INFO:tasks.ceph.mds.c:Started 2024-04-02T13:20:13.939 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T13:20:13.939 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T13:20:13.941 INFO:tasks.ceph.mds.b:Started 2024-04-02T13:20:13.941 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T13:20:13.942 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T13:20:13.944 INFO:tasks.ceph.mds.d:Started 2024-04-02T13:20:13.944 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-02T13:20:13.944 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:14.014 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:20:14.015 INFO:tasks.ceph.mds.c.smithi044.stdout:starting mds.c at 2024-04-02T13:20:14.022 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-04-02T13:20:14.023 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-04-02T13:20:14.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.051+0000 7f84c347b640 1 Processor -- start 2024-04-02T13:20:14.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.051+0000 7f84c347b640 1 -- start start 2024-04-02T13:20:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c347b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f84bc156670 0x7f84bc071a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c347b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84bc0b1e60 0x7f84bc071f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c347b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84bc15d270 con 0x7f84bc0724d0 2024-04-02T13:20:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c347b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f84bc0c5d50 con 0x7f84bc156670 2024-04-02T13:20:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c347b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f84bc15d550 con 0x7f84bc0b1e60 2024-04-02T13:20:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c1217640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f84bc156670 0x7f84bc071a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c1217640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f84bc156670 0x7f84bc071a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56308/0 (socket says 172.21.15.44:56308) 2024-04-02T13:20:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c1217640 1 -- 172.21.15.44:0/1578703792 learned_addr learned my addr 172.21.15.44:0/1578703792 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c0a16640 1 --2- 172.21.15.44:0/1578703792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84bc0b1e60 0x7f84bc071f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c1217640 1 -- 172.21.15.44:0/1578703792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84bc0b1e60 msgr2=0x7f84bc071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c1217640 1 --2- 172.21.15.44:0/1578703792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84bc0b1e60 0x7f84bc071f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c1217640 1 -- 172.21.15.44:0/1578703792 >> v1:172.21.15.69:6789/0 conn(0x7f84bc0724d0 legacy=0x7f84bc15fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c1217640 1 -- 172.21.15.44:0/1578703792 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84bc0c5700 con 0x7f84bc156670 2024-04-02T13:20:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.052+0000 7f84c0a16640 1 --2- 172.21.15.44:0/1578703792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84bc0b1e60 0x7f84bc071f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.053+0000 7f84c1217640 1 --2- 172.21.15.44:0/1578703792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84bc156670 0x7f84bc071a50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009870 tx=0x7f84b402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94376627314c217f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.053+0000 7f84b3fff640 1 -- 172.21.15.44:0/1578703792 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b40317c0 con 0x7f84bc156670 2024-04-02T13:20:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.053+0000 7f84b3fff640 1 -- 172.21.15.44:0/1578703792 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84b4037070 con 0x7f84bc156670 2024-04-02T13:20:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.053+0000 7f84b3fff640 1 -- 172.21.15.44:0/1578703792 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b4031b80 con 0x7f84bc156670 2024-04-02T13:20:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.054+0000 7f84c347b640 1 -- 172.21.15.44:0/1578703792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84bc156670 msgr2=0x7f84bc071a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.054+0000 7f84c347b640 1 --2- 172.21.15.44:0/1578703792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84bc156670 0x7f84bc071a50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009870 tx=0x7f84b402b860 comp rx=0 tx=0).stop 2024-04-02T13:20:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.054+0000 7f84c347b640 1 -- 172.21.15.44:0/1578703792 shutdown_connections 2024-04-02T13:20:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.054+0000 7f84c347b640 1 --2- 172.21.15.44:0/1578703792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84bc0b1e60 0x7f84bc071f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.054+0000 7f84c347b640 1 --2- 172.21.15.44:0/1578703792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84bc156670 0x7f84bc071a50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.054+0000 7f84c347b640 1 -- 172.21.15.44:0/1578703792 >> 172.21.15.44:0/1578703792 conn(0x7f84bc074dc0 msgr2=0x7f84bc06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.054+0000 7f84c347b640 1 -- 172.21.15.44:0/1578703792 shutdown_connections 2024-04-02T13:20:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.054+0000 7f84c347b640 1 -- 172.21.15.44:0/1578703792 wait complete. 2024-04-02T13:20:14.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.055+0000 7f84c347b640 1 Processor -- start 2024-04-02T13:20:14.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.055+0000 7f84c347b640 1 -- start start 2024-04-02T13:20:14.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c347b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84bc0724d0 0x7f84bc141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:14.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c347b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84bc0b1e60 0x7f84bc13fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:14.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c347b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84bc156670 0x7f84bc13f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:14.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c347b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc156a50 con 0x7f84bc0b1e60 2024-04-02T13:20:14.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c347b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc0cdff0 con 0x7f84bc0724d0 2024-04-02T13:20:14.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c347b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f84bc156bd0 con 0x7f84bc156670 2024-04-02T13:20:14.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c0a16640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84bc0b1e60 0x7f84bc13fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:14.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c0a16640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84bc0b1e60 0x7f84bc13fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56324/0 (socket says 172.21.15.44:56324) 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c0a16640 1 -- 172.21.15.44:0/1297580100 learned_addr learned my addr 172.21.15.44:0/1297580100 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c1217640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84bc0724d0 0x7f84bc141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c1a18640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84bc156670 0x7f84bc13f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c0a16640 1 -- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84bc156670 msgr2=0x7f84bc13f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c0a16640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84bc156670 0x7f84bc13f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c0a16640 1 -- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84bc0724d0 msgr2=0x7f84bc141810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c0a16640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84bc0724d0 0x7f84bc141810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c0a16640 1 -- 172.21.15.44:0/1297580100 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84bc140770 con 0x7f84bc0b1e60 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c0a16640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84bc0b1e60 0x7f84bc13fe70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00c7f0 tx=0x7f84ac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84b1ffb640 1 -- 172.21.15.44:0/1297580100 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac004230 con 0x7f84bc0b1e60 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84b1ffb640 1 -- 172.21.15.44:0/1297580100 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84ac0043d0 con 0x7f84bc0b1e60 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84c347b640 1 -- 172.21.15.44:0/1297580100 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84bc140a50 con 0x7f84bc0b1e60 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.056+0000 7f84b1ffb640 1 -- 172.21.15.44:0/1297580100 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac007a70 con 0x7f84bc0b1e60 2024-04-02T13:20:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.057+0000 7f84c347b640 1 -- 172.21.15.44:0/1297580100 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84bc141010 con 0x7f84bc0b1e60 2024-04-02T13:20:14.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.058+0000 7f84c347b640 1 -- 172.21.15.44:0/1297580100 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84bc06c460 con 0x7f84bc0b1e60 2024-04-02T13:20:14.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.058+0000 7f84b1ffb640 1 -- 172.21.15.44:0/1297580100 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f84ac007c10 con 0x7f84bc0b1e60 2024-04-02T13:20:14.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.059+0000 7f84b1ffb640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f84a006c860 0x7f84a006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:14.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.059+0000 7f84b1ffb640 1 -- 172.21.15.44:0/1297580100 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7f84ac08b5f0 con 0x7f84bc0b1e60 2024-04-02T13:20:14.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.059+0000 7f84c1217640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f84a006c860 0x7f84a006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:14.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.059+0000 7f84c1217640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f84a006c860 0x7f84a006ed20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f84b4036000 tx=0x7f84b403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:14.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.063+0000 7f84b1ffb640 1 -- 172.21.15.44:0/1297580100 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f84ac015020 con 0x7f84bc0b1e60 2024-04-02T13:20:14.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.249+0000 7f84c347b640 1 -- 172.21.15.44:0/1297580100 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84bc151980 con 0x7f84bc0b1e60 2024-04-02T13:20:14.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.249+0000 7f84b1ffb640 1 -- 172.21.15.44:0/1297580100 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10861 (secure 0 0 0) 0x7f84ac0598f0 con 0x7f84bc0b1e60 2024-04-02T13:20:14.251 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:14.251 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":13,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:13.833429+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:14.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.251+0000 7f84c347b640 1 -- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f84a006c860 msgr2=0x7f84a006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:14.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.251+0000 7f84c347b640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f84a006c860 0x7f84a006ed20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f84b4036000 tx=0x7f84b403d040 comp rx=0 tx=0).stop 2024-04-02T13:20:14.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.251+0000 7f84c347b640 1 -- 172.21.15.44:0/1297580100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84bc0b1e60 msgr2=0x7f84bc13fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:14.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.251+0000 7f84c347b640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84bc0b1e60 0x7f84bc13fe70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00c7f0 tx=0x7f84ac00ccb0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.251+0000 7f84c347b640 1 -- 172.21.15.44:0/1297580100 shutdown_connections 2024-04-02T13:20:14.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.251+0000 7f84c347b640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f84a006c860 0x7f84a006ed20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.251+0000 7f84c347b640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84bc156670 0x7f84bc13f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.251+0000 7f84c347b640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84bc0b1e60 0x7f84bc13fe70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.251+0000 7f84c347b640 1 --2- 172.21.15.44:0/1297580100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84bc0724d0 0x7f84bc141810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.251+0000 7f84c347b640 1 -- 172.21.15.44:0/1297580100 >> 172.21.15.44:0/1297580100 conn(0x7f84bc074dc0 msgr2=0x7f84bc15eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:14.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.251+0000 7f84c347b640 1 -- 172.21.15.44:0/1297580100 shutdown_connections 2024-04-02T13:20:14.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.251+0000 7f84c347b640 1 -- 172.21.15.44:0/1297580100 wait complete. 2024-04-02T13:20:14.263 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-04-02T13:20:14.263 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:20:14.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.363+0000 7f6f38cf2640 1 Processor -- start 2024-04-02T13:20:14.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.363+0000 7f6f38cf2640 1 -- start start 2024-04-02T13:20:14.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.364+0000 7f6f38cf2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f340d0c50 0x7f6f340d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:14.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.364+0000 7f6f38cf2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6f3406ce40 0x7f6f3406d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:14.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.364+0000 7f6f38cf2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f340cc600 con 0x7f6f3415e4a0 2024-04-02T13:20:14.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.364+0000 7f6f38cf2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f6f340af300 con 0x7f6f3406ce40 2024-04-02T13:20:14.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.364+0000 7f6f38cf2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6f340cc8e0 con 0x7f6f340d0c50 2024-04-02T13:20:14.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.364+0000 7f6f32d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6f3406ce40 0x7f6f3406d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:14.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.364+0000 7f6f32d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6f3406ce40 0x7f6f3406d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56326/0 (socket says 172.21.15.44:56326) 2024-04-02T13:20:14.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.364+0000 7f6f32d9d640 1 -- 172.21.15.44:0/2932777457 learned_addr learned my addr 172.21.15.44:0/2932777457 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:14.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.364+0000 7f6f2bfff640 1 --2- 172.21.15.44:0/2932777457 >> v2:172.21.15.69:3300/0 conn(0x7f6f340d0c50 0x7f6f340d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:14.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.364+0000 7f6f32d9d640 1 -- 172.21.15.44:0/2932777457 >> v2:172.21.15.69:3300/0 conn(0x7f6f340d0c50 msgr2=0x7f6f340d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:14.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.364+0000 7f6f32d9d640 1 --2- 172.21.15.44:0/2932777457 >> v2:172.21.15.69:3300/0 conn(0x7f6f340d0c50 0x7f6f340d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.364+0000 7f6f32d9d640 1 -- 172.21.15.44:0/2932777457 >> v1:172.21.15.69:6789/0 conn(0x7f6f3415e4a0 legacy=0x7f6f340aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.364+0000 7f6f32d9d640 1 -- 172.21.15.44:0/2932777457 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f340ccbc0 con 0x7f6f3406ce40 2024-04-02T13:20:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.365+0000 7f6f32d9d640 1 --2- 172.21.15.44:0/2932777457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f3406ce40 0x7f6f3406d2f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6f340b8060 tx=0x7f6f2c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9dfa1210a2d671 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.365+0000 7f6f31d9b640 1 -- 172.21.15.44:0/2932777457 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c02ca80 con 0x7f6f3406ce40 2024-04-02T13:20:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.365+0000 7f6f31d9b640 1 -- 172.21.15.44:0/2932777457 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f2c02cc20 con 0x7f6f3406ce40 2024-04-02T13:20:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.365+0000 7f6f31d9b640 1 -- 172.21.15.44:0/2932777457 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c0343f0 con 0x7f6f3406ce40 2024-04-02T13:20:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.366+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2932777457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f3406ce40 msgr2=0x7f6f3406d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.366+0000 7f6f38cf2640 1 --2- 172.21.15.44:0/2932777457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f3406ce40 0x7f6f3406d2f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6f340b8060 tx=0x7f6f2c02bee0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.366+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2932777457 shutdown_connections 2024-04-02T13:20:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.366+0000 7f6f38cf2640 1 --2- 172.21.15.44:0/2932777457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f3406ce40 0x7f6f3406d2f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.366+0000 7f6f38cf2640 1 --2- 172.21.15.44:0/2932777457 >> v2:172.21.15.69:3300/0 conn(0x7f6f340d0c50 0x7f6f340d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.366+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2932777457 >> 172.21.15.44:0/2932777457 conn(0x7f6f3409aa20 msgr2=0x7f6f3409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.366+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2932777457 shutdown_connections 2024-04-02T13:20:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.366+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2932777457 wait complete. 2024-04-02T13:20:14.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.367+0000 7f6f38cf2640 1 Processor -- start 2024-04-02T13:20:14.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.367+0000 7f6f38cf2640 1 -- start start 2024-04-02T13:20:14.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f38cf2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f3406ce40 0x7f6f34153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:14.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f38cf2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f340d0c50 0x7f6f340c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:14.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f38cf2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f3415e4a0 0x7f6f340cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:14.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f38cf2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6f340cd460 con 0x7f6f3406ce40 2024-04-02T13:20:14.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f38cf2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f340cd180 con 0x7f6f340d0c50 2024-04-02T13:20:14.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f38cf2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6f3415e880 con 0x7f6f3415e4a0 2024-04-02T13:20:14.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f3259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f3406ce40 0x7f6f34153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:14.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f3259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f3406ce40 0x7f6f34153680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56340/0 (socket says 172.21.15.44:56340) 2024-04-02T13:20:14.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f3259c640 1 -- 172.21.15.44:0/2171442411 learned_addr learned my addr 172.21.15.44:0/2171442411 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:14.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f3259c640 1 -- 172.21.15.44:0/2171442411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f3415e4a0 msgr2=0x7f6f340cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:14.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f2bfff640 1 --2- 172.21.15.44:0/2171442411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f340d0c50 0x7f6f340c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:14.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f3259c640 1 --2- 172.21.15.44:0/2171442411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f3415e4a0 0x7f6f340cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f3259c640 1 -- 172.21.15.44:0/2171442411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f340d0c50 msgr2=0x7f6f340c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:14.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f3259c640 1 --2- 172.21.15.44:0/2171442411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f340d0c50 0x7f6f340c80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f3259c640 1 -- 172.21.15.44:0/2171442411 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f340c8910 con 0x7f6f3406ce40 2024-04-02T13:20:14.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f3259c640 1 --2- 172.21.15.44:0/2171442411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f3406ce40 0x7f6f34153680 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400b3f0 tx=0x7f6f2400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:14.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f2affd640 1 -- 172.21.15.44:0/2171442411 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24015020 con 0x7f6f3406ce40 2024-04-02T13:20:14.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f2affd640 1 -- 172.21.15.44:0/2171442411 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f24011830 con 0x7f6f3406ce40 2024-04-02T13:20:14.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2171442411 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f341f7600 con 0x7f6f3406ce40 2024-04-02T13:20:14.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.368+0000 7f6f2affd640 1 -- 172.21.15.44:0/2171442411 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24011c20 con 0x7f6f3406ce40 2024-04-02T13:20:14.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.369+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2171442411 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f34153bc0 con 0x7f6f3406ce40 2024-04-02T13:20:14.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.369+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2171442411 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f340684f0 con 0x7f6f3406ce40 2024-04-02T13:20:14.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.370+0000 7f6f2affd640 1 -- 172.21.15.44:0/2171442411 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f6f24018020 con 0x7f6f3406ce40 2024-04-02T13:20:14.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.370+0000 7f6f2affd640 1 --2- 172.21.15.44:0/2171442411 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f1406c890 0x7f6f1406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:14.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.370+0000 7f6f2affd640 1 -- 172.21.15.44:0/2171442411 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7f6f2408b1b0 con 0x7f6f3406ce40 2024-04-02T13:20:14.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.370+0000 7f6f2bfff640 1 --2- 172.21.15.44:0/2171442411 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f1406c890 0x7f6f1406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:14.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.371+0000 7f6f2bfff640 1 --2- 172.21.15.44:0/2171442411 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f1406c890 0x7f6f1406ed50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6f3409ff40 tx=0x7f6f1c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:14.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.375+0000 7f6f2affd640 1 -- 172.21.15.44:0/2171442411 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6f2401f030 con 0x7f6f3406ce40 2024-04-02T13:20:14.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.597+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2171442411 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f340cf850 con 0x7f6f3406ce40 2024-04-02T13:20:14.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.598+0000 7f6f2affd640 1 -- 172.21.15.44:0/2171442411 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7f6f240172c0 con 0x7f6f3406ce40 2024-04-02T13:20:14.599 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:14.599 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2772068060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2772068060},{"type":"v1","addr":"172.21.15.44:6833","nonce":2772068060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-02T13:20:14.600 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 2 2024-04-02T13:20:14.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.600+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2171442411 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f1406c890 msgr2=0x7f6f1406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:14.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.600+0000 7f6f38cf2640 1 --2- 172.21.15.44:0/2171442411 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f1406c890 0x7f6f1406ed50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6f3409ff40 tx=0x7f6f1c008040 comp rx=0 tx=0).stop 2024-04-02T13:20:14.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.600+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2171442411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f3406ce40 msgr2=0x7f6f34153680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:14.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.600+0000 7f6f38cf2640 1 --2- 172.21.15.44:0/2171442411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f3406ce40 0x7f6f34153680 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400b3f0 tx=0x7f6f2400b8b0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.600+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2171442411 shutdown_connections 2024-04-02T13:20:14.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.600+0000 7f6f38cf2640 1 --2- 172.21.15.44:0/2171442411 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f1406c890 0x7f6f1406ed50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.600+0000 7f6f38cf2640 1 --2- 172.21.15.44:0/2171442411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f3415e4a0 0x7f6f340cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.600+0000 7f6f38cf2640 1 --2- 172.21.15.44:0/2171442411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f340d0c50 0x7f6f340c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.600+0000 7f6f38cf2640 1 --2- 172.21.15.44:0/2171442411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f3406ce40 0x7f6f34153680 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:14.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.600+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2171442411 >> 172.21.15.44:0/2171442411 conn(0x7f6f3409aa20 msgr2=0x7f6f340aebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:14.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.600+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2171442411 shutdown_connections 2024-04-02T13:20:14.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:14.600+0000 7f6f38cf2640 1 -- 172.21.15.44:0/2171442411 wait complete. 2024-04-02T13:20:16.612 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:20:16.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.725+0000 7f59e8c55640 1 Processor -- start 2024-04-02T13:20:16.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.725+0000 7f59e8c55640 1 -- start start 2024-04-02T13:20:16.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.726+0000 7f59e8c55640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f59e4159db0 0x7f59e415c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:16.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.726+0000 7f59e8c55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59e4071e50 0x7f59e40cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:16.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.726+0000 7f59e8c55640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e40726c0 con 0x7f59e40ccc20 2024-04-02T13:20:16.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.726+0000 7f59e8c55640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f59e4073240 con 0x7f59e4159db0 2024-04-02T13:20:16.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.726+0000 7f59e8c55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f59e40cd050 con 0x7f59e4071e50 2024-04-02T13:20:16.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.726+0000 7f59e259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f59e4159db0 0x7f59e415c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:16.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.726+0000 7f59e259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f59e4159db0 0x7f59e415c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59562/0 (socket says 172.21.15.44:59562) 2024-04-02T13:20:16.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.726+0000 7f59e259c640 1 -- 172.21.15.44:0/3331270951 learned_addr learned my addr 172.21.15.44:0/3331270951 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:16.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.726+0000 7f59dbfff640 1 --2- 172.21.15.44:0/3331270951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59e4071e50 0x7f59e40cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:16.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.726+0000 7f59e259c640 1 -- 172.21.15.44:0/3331270951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59e4071e50 msgr2=0x7f59e40cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:16.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.726+0000 7f59e259c640 1 --2- 172.21.15.44:0/3331270951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59e4071e50 0x7f59e40cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:16.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.726+0000 7f59e259c640 1 -- 172.21.15.44:0/3331270951 >> v1:172.21.15.69:6789/0 conn(0x7f59e40ccc20 legacy=0x7f59e406d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:16.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.726+0000 7f59e259c640 1 -- 172.21.15.44:0/3331270951 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e40729a0 con 0x7f59e4159db0 2024-04-02T13:20:16.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.727+0000 7f59e259c640 1 --2- 172.21.15.44:0/3331270951 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59e4159db0 0x7f59e415c6d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009870 tx=0x7f59d402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d7f075dc7b5e4f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:16.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.727+0000 7f59e1d9b640 1 -- 172.21.15.44:0/3331270951 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d40317c0 con 0x7f59e4159db0 2024-04-02T13:20:16.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.727+0000 7f59e1d9b640 1 -- 172.21.15.44:0/3331270951 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d4037070 con 0x7f59e4159db0 2024-04-02T13:20:16.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.727+0000 7f59e1d9b640 1 -- 172.21.15.44:0/3331270951 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4031b80 con 0x7f59e4159db0 2024-04-02T13:20:16.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.727+0000 7f59e8c55640 1 -- 172.21.15.44:0/3331270951 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59e4159db0 msgr2=0x7f59e415c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:16.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.728+0000 7f59e8c55640 1 --2- 172.21.15.44:0/3331270951 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59e4159db0 0x7f59e415c6d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009870 tx=0x7f59d402b860 comp rx=0 tx=0).stop 2024-04-02T13:20:16.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.728+0000 7f59e8c55640 1 -- 172.21.15.44:0/3331270951 shutdown_connections 2024-04-02T13:20:16.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.728+0000 7f59e8c55640 1 --2- 172.21.15.44:0/3331270951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59e4071e50 0x7f59e40cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:16.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.728+0000 7f59e8c55640 1 --2- 172.21.15.44:0/3331270951 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59e4159db0 0x7f59e415c6d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:16.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.728+0000 7f59e8c55640 1 -- 172.21.15.44:0/3331270951 >> 172.21.15.44:0/3331270951 conn(0x7f59e40cfcd0 msgr2=0x7f59e40d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:16.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.728+0000 7f59e8c55640 1 -- 172.21.15.44:0/3331270951 shutdown_connections 2024-04-02T13:20:16.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.728+0000 7f59e8c55640 1 -- 172.21.15.44:0/3331270951 wait complete. 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.729+0000 7f59e8c55640 1 Processor -- start 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.729+0000 7f59e8c55640 1 -- start start 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e8c55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59e4071e50 0x7f59e414b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e8c55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59e40ccc20 0x7f59e415a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e8c55640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59e4159db0 0x7f59e40b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e8c55640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f59e40722d0 con 0x7f59e4159db0 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e8c55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f59e4072f60 con 0x7f59e4071e50 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e8c55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f59e4072450 con 0x7f59e40ccc20 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59e4159db0 0x7f59e40b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59e4159db0 0x7f59e40b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59570/0 (socket says 172.21.15.44:59570) 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e2d9d640 1 -- 172.21.15.44:0/4112924926 learned_addr learned my addr 172.21.15.44:0/4112924926 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59dbfff640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59e40ccc20 0x7f59e415a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e259c640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59e4071e50 0x7f59e414b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:16.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e2d9d640 1 -- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59e40ccc20 msgr2=0x7f59e415a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:16.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e2d9d640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59e40ccc20 0x7f59e415a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:16.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e2d9d640 1 -- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59e4071e50 msgr2=0x7f59e414b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:16.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e2d9d640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59e4071e50 0x7f59e414b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:16.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e2d9d640 1 -- 172.21.15.44:0/4112924926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e414bbb0 con 0x7f59e4159db0 2024-04-02T13:20:16.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.730+0000 7f59e2d9d640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59e4159db0 0x7f59e40b01a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00c7b0 tx=0x7f59dc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:16.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.731+0000 7f59daffd640 1 -- 172.21.15.44:0/4112924926 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc004b60 con 0x7f59e4159db0 2024-04-02T13:20:16.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.731+0000 7f59daffd640 1 -- 172.21.15.44:0/4112924926 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59dc004d00 con 0x7f59e4159db0 2024-04-02T13:20:16.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.731+0000 7f59daffd640 1 -- 172.21.15.44:0/4112924926 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc00aa30 con 0x7f59e4159db0 2024-04-02T13:20:16.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.731+0000 7f59e8c55640 1 -- 172.21.15.44:0/4112924926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e409c520 con 0x7f59e4159db0 2024-04-02T13:20:16.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.731+0000 7f59e8c55640 1 -- 172.21.15.44:0/4112924926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e409cae0 con 0x7f59e4159db0 2024-04-02T13:20:16.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.732+0000 7f59e8c55640 1 -- 172.21.15.44:0/4112924926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e40684f0 con 0x7f59e4159db0 2024-04-02T13:20:16.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.732+0000 7f59daffd640 1 -- 172.21.15.44:0/4112924926 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f59dc00abd0 con 0x7f59e4159db0 2024-04-02T13:20:16.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.733+0000 7f59daffd640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59c406c860 0x7f59c406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:16.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.733+0000 7f59daffd640 1 -- 172.21.15.44:0/4112924926 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4846+0+0 (secure 0 0 0) 0x7f59dc08b730 con 0x7f59e4159db0 2024-04-02T13:20:16.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.733+0000 7f59e259c640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59c406c860 0x7f59c406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:16.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.733+0000 7f59e259c640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59c406c860 0x7f59c406ed20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f59d4036000 tx=0x7f59d403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:16.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.738+0000 7f59daffd640 1 -- 172.21.15.44:0/4112924926 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f59dc020070 con 0x7f59e4159db0 2024-04-02T13:20:16.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.962+0000 7f59e8c55640 1 -- 172.21.15.44:0/4112924926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59e40aefa0 con 0x7f59e4159db0 2024-04-02T13:20:16.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.963+0000 7f59daffd640 1 -- 172.21.15.44:0/4112924926 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3864 (secure 0 0 0) 0x7f59dc0599e0 con 0x7f59e4159db0 2024-04-02T13:20:16.964 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:16.964 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4040417710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4040417710},{"type":"v1","addr":"172.21.15.69:6836","nonce":4040417710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2772068060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2772068060},{"type":"v1","addr":"172.21.15.44:6833","nonce":2772068060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1985405150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1985405150},{"type":"v1","addr":"172.21.15.69:6837","nonce":1985405150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2282156931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2282156931},{"type":"v1","addr":"172.21.15.44:6835","nonce":2282156931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-02T13:20:16.965 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 3 2024-04-02T13:20:16.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.965+0000 7f59e8c55640 1 -- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59c406c860 msgr2=0x7f59c406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:16.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.965+0000 7f59e8c55640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59c406c860 0x7f59c406ed20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f59d4036000 tx=0x7f59d403d040 comp rx=0 tx=0).stop 2024-04-02T13:20:16.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.965+0000 7f59e8c55640 1 -- 172.21.15.44:0/4112924926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59e4159db0 msgr2=0x7f59e40b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:16.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.965+0000 7f59e8c55640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59e4159db0 0x7f59e40b01a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00c7b0 tx=0x7f59dc00cc70 comp rx=0 tx=0).stop 2024-04-02T13:20:16.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.965+0000 7f59e8c55640 1 -- 172.21.15.44:0/4112924926 shutdown_connections 2024-04-02T13:20:16.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.965+0000 7f59e8c55640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59c406c860 0x7f59c406ed20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:16.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.965+0000 7f59e8c55640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59e4159db0 0x7f59e40b01a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:16.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.965+0000 7f59e8c55640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59e40ccc20 0x7f59e415a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:16.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.965+0000 7f59e8c55640 1 --2- 172.21.15.44:0/4112924926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59e4071e50 0x7f59e414b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:16.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.965+0000 7f59e8c55640 1 -- 172.21.15.44:0/4112924926 >> 172.21.15.44:0/4112924926 conn(0x7f59e40cfcd0 msgr2=0x7f59e40af930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:16.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.965+0000 7f59e8c55640 1 -- 172.21.15.44:0/4112924926 shutdown_connections 2024-04-02T13:20:16.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:16.965+0000 7f59e8c55640 1 -- 172.21.15.44:0/4112924926 wait complete. 2024-04-02T13:20:16.977 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:17.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.076+0000 7f274b952640 1 Processor -- start 2024-04-02T13:20:17.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.077+0000 7f274b952640 1 -- start start 2024-04-02T13:20:17.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.077+0000 7f274b952640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27440b3510 0x7f27440d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.077+0000 7f274b952640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f27440d08d0 0x7f27440b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.077+0000 7f274b952640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27440af420 con 0x7f274415c250 2024-04-02T13:20:17.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.077+0000 7f274b952640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f274415c630 con 0x7f27440d08d0 2024-04-02T13:20:17.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.077+0000 7f274b952640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f27440af700 con 0x7f27440b3510 2024-04-02T13:20:17.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.078+0000 7f2748eed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27440b3510 0x7f27440d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.078+0000 7f2749eef640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f27440d08d0 0x7f27440b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.078+0000 7f2748eed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27440b3510 0x7f27440d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:38388/0 (socket says 172.21.15.44:38388) 2024-04-02T13:20:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.078+0000 7f2748eed640 1 -- 172.21.15.44:0/898961155 learned_addr learned my addr 172.21.15.44:0/898961155 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.078+0000 7f2749eef640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f27440d08d0 0x7f27440b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42864/0 (socket says 172.21.15.44:42864) 2024-04-02T13:20:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.078+0000 7f2749eef640 1 -- 172.21.15.44:0/898961155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27440b3510 msgr2=0x7f27440d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.078+0000 7f2749eef640 1 --2- 172.21.15.44:0/898961155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27440b3510 0x7f27440d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.078+0000 7f2749eef640 1 -- 172.21.15.44:0/898961155 >> v1:172.21.15.69:6789/0 conn(0x7f274415c250 legacy=0x7f27440aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.078+0000 7f2749eef640 1 -- 172.21.15.44:0/898961155 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27440af9e0 con 0x7f27440d08d0 2024-04-02T13:20:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.078+0000 7f2748eed640 1 --2- 172.21.15.44:0/898961155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27440b3510 0x7f27440d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.079+0000 7f2749eef640 1 --2- 172.21.15.44:0/898961155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27440d08d0 0x7f27440b04c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f273c009d90 tx=0x7f273c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f00cc9dcef023940 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.079+0000 7f273bfff640 1 -- 172.21.15.44:0/898961155 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c039070 con 0x7f27440d08d0 2024-04-02T13:20:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.079+0000 7f273bfff640 1 -- 172.21.15.44:0/898961155 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f273c02cbc0 con 0x7f27440d08d0 2024-04-02T13:20:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.079+0000 7f273bfff640 1 -- 172.21.15.44:0/898961155 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c034440 con 0x7f27440d08d0 2024-04-02T13:20:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.079+0000 7f274b952640 1 -- 172.21.15.44:0/898961155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27440d08d0 msgr2=0x7f27440b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.079+0000 7f274b952640 1 --2- 172.21.15.44:0/898961155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27440d08d0 0x7f27440b04c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f273c009d90 tx=0x7f273c02be10 comp rx=0 tx=0).stop 2024-04-02T13:20:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.080+0000 7f274b952640 1 -- 172.21.15.44:0/898961155 shutdown_connections 2024-04-02T13:20:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.080+0000 7f274b952640 1 --2- 172.21.15.44:0/898961155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27440d08d0 0x7f27440b04c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.080+0000 7f274b952640 1 --2- 172.21.15.44:0/898961155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27440b3510 0x7f27440d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.080+0000 7f274b952640 1 -- 172.21.15.44:0/898961155 >> 172.21.15.44:0/898961155 conn(0x7f2744073850 msgr2=0x7f274409aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.080+0000 7f274b952640 1 -- 172.21.15.44:0/898961155 shutdown_connections 2024-04-02T13:20:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.080+0000 7f274b952640 1 -- 172.21.15.44:0/898961155 wait complete. 2024-04-02T13:20:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.081+0000 7f274b952640 1 Processor -- start 2024-04-02T13:20:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.081+0000 7f274b952640 1 -- start start 2024-04-02T13:20:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.081+0000 7f274b952640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27440b3510 0x7f27440c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.081+0000 7f274b952640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27440d08d0 0x7f27440cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.081+0000 7f274b952640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f274415c250 0x7f27440cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.081+0000 7f274b952640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f27440b0b80 con 0x7f274415c250 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.081+0000 7f274b952640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f27440b0a00 con 0x7f27440b3510 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.081+0000 7f274b952640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f27440b0d00 con 0x7f27440d08d0 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f2749eef640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f274415c250 0x7f27440cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f2749eef640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f274415c250 0x7f27440cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59574/0 (socket says 172.21.15.44:59574) 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f2749eef640 1 -- 172.21.15.44:0/2359428036 learned_addr learned my addr 172.21.15.44:0/2359428036 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f27496ee640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27440b3510 0x7f27440c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f2748eed640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27440d08d0 0x7f27440cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f2749eef640 1 -- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27440d08d0 msgr2=0x7f27440cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f2749eef640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27440d08d0 0x7f27440cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f2749eef640 1 -- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27440b3510 msgr2=0x7f27440c7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f2749eef640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27440b3510 0x7f27440c7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f2749eef640 1 -- 172.21.15.44:0/2359428036 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27440c84c0 con 0x7f274415c250 2024-04-02T13:20:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f27496ee640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27440b3510 0x7f27440c7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f2749eef640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f274415c250 0x7f27440cb720 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f273c009eb0 tx=0x7f273c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f2739ffb640 1 -- 172.21.15.44:0/2359428036 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c044020 con 0x7f274415c250 2024-04-02T13:20:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f2739ffb640 1 -- 172.21.15.44:0/2359428036 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f273c03f470 con 0x7f274415c250 2024-04-02T13:20:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f274b952640 1 -- 172.21.15.44:0/2359428036 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27441f6450 con 0x7f274415c250 2024-04-02T13:20:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f2739ffb640 1 -- 172.21.15.44:0/2359428036 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c039040 con 0x7f274415c250 2024-04-02T13:20:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.082+0000 7f274b952640 1 -- 172.21.15.44:0/2359428036 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27441f5970 con 0x7f274415c250 2024-04-02T13:20:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.083+0000 7f274b952640 1 -- 172.21.15.44:0/2359428036 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27440d1420 con 0x7f274415c250 2024-04-02T13:20:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.084+0000 7f2739ffb640 1 -- 172.21.15.44:0/2359428036 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f273c047020 con 0x7f274415c250 2024-04-02T13:20:17.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.084+0000 7f2739ffb640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f272806c860 0x7f272806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.084+0000 7f2739ffb640 1 -- 172.21.15.44:0/2359428036 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f273c0ae130 con 0x7f274415c250 2024-04-02T13:20:17.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.084+0000 7f27496ee640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f272806c860 0x7f272806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.085+0000 7f27496ee640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f272806c860 0x7f272806ed20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2740001660 tx=0x7f2740009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.089+0000 7f2739ffb640 1 -- 172.21.15.44:0/2359428036 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f273c04e030 con 0x7f274415c250 2024-04-02T13:20:17.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.284+0000 7f274b952640 1 -- 172.21.15.44:0/2359428036 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2744068c00 con 0x7f274415c250 2024-04-02T13:20:17.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.285+0000 7f2739ffb640 1 -- 172.21.15.44:0/2359428036 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12849 (secure 0 0 0) 0x7f273c07ca20 con 0x7f274415c250 2024-04-02T13:20:17.286 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:17.286 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":16,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:16.847560+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.288+0000 7f274b952640 1 -- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f272806c860 msgr2=0x7f272806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.288+0000 7f274b952640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f272806c860 0x7f272806ed20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2740001660 tx=0x7f2740009040 comp rx=0 tx=0).stop 2024-04-02T13:20:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.288+0000 7f274b952640 1 -- 172.21.15.44:0/2359428036 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f274415c250 msgr2=0x7f27440cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.288+0000 7f274b952640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f274415c250 0x7f27440cb720 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f273c009eb0 tx=0x7f273c004a40 comp rx=0 tx=0).stop 2024-04-02T13:20:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.288+0000 7f274b952640 1 -- 172.21.15.44:0/2359428036 shutdown_connections 2024-04-02T13:20:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.288+0000 7f274b952640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f272806c860 0x7f272806ed20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.288+0000 7f274b952640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f274415c250 0x7f27440cb720 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.288+0000 7f274b952640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27440d08d0 0x7f27440cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.288+0000 7f274b952640 1 --2- 172.21.15.44:0/2359428036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27440b3510 0x7f27440c7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.288+0000 7f274b952640 1 -- 172.21.15.44:0/2359428036 >> 172.21.15.44:0/2359428036 conn(0x7f2744073850 msgr2=0x7f274409ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.288+0000 7f274b952640 1 -- 172.21.15.44:0/2359428036 shutdown_connections 2024-04-02T13:20:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.288+0000 7f274b952640 1 -- 172.21.15.44:0/2359428036 wait complete. 2024-04-02T13:20:17.301 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:20:17.301 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:20:17.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.407+0000 7f49e6a55640 1 Processor -- start 2024-04-02T13:20:17.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.407+0000 7f49e6a55640 1 -- start start 2024-04-02T13:20:17.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49e6a55640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f49e015bfa0 0x7f49e00b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49e6a55640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f49e00add60 0x7f49e006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49e6a55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49e00ae1f0 0x7f49e006ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49e6a55640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f49e015c380 con 0x7f49e00add60 2024-04-02T13:20:17.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49e6a55640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f49e00d0a10 con 0x7f49e015bfa0 2024-04-02T13:20:17.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49e6a55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f49e00d0790 con 0x7f49e00ae1f0 2024-04-02T13:20:17.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49df7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f49e00add60 0x7f49e006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49df7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f49e00add60 0x7f49e006b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59584/0 (socket says 172.21.15.44:59584) 2024-04-02T13:20:17.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49df7fe640 1 -- 172.21.15.44:0/2052038859 learned_addr learned my addr 172.21.15.44:0/2052038859 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:17.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49e4ff2640 1 --2- 172.21.15.44:0/2052038859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49e00ae1f0 0x7f49e006ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49dffff640 1 --2- 172.21.15.44:0/2052038859 >> v2:172.21.15.69:3300/0 conn(0x7f49e015bfa0 0x7f49e00b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49df7fe640 1 -- 172.21.15.44:0/2052038859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49e00ae1f0 msgr2=0x7f49e006ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:17.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49df7fe640 1 --2- 172.21.15.44:0/2052038859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49e00ae1f0 0x7f49e006ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49df7fe640 1 -- 172.21.15.44:0/2052038859 >> v2:172.21.15.69:3300/0 conn(0x7f49e015bfa0 msgr2=0x7f49e00b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:17.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49df7fe640 1 --2- 172.21.15.44:0/2052038859 >> v2:172.21.15.69:3300/0 conn(0x7f49e015bfa0 0x7f49e00b3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.408+0000 7f49df7fe640 1 -- 172.21.15.44:0/2052038859 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e00afcf0 con 0x7f49e00add60 2024-04-02T13:20:17.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.409+0000 7f49df7fe640 1 --2- 172.21.15.44:0/2052038859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49e00add60 0x7f49e006b4d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f49e0072330 tx=0x7f49d002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d2a359b67d10ca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:17.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.409+0000 7f49deffd640 1 -- 172.21.15.44:0/2052038859 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d00317c0 con 0x7f49e00add60 2024-04-02T13:20:17.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.409+0000 7f49deffd640 1 -- 172.21.15.44:0/2052038859 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d0037070 con 0x7f49e00add60 2024-04-02T13:20:17.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.409+0000 7f49deffd640 1 -- 172.21.15.44:0/2052038859 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0031b80 con 0x7f49e00add60 2024-04-02T13:20:17.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.410+0000 7f49e6a55640 1 -- 172.21.15.44:0/2052038859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49e00add60 msgr2=0x7f49e006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.410+0000 7f49e6a55640 1 --2- 172.21.15.44:0/2052038859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49e00add60 0x7f49e006b4d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f49e0072330 tx=0x7f49d002b860 comp rx=0 tx=0).stop 2024-04-02T13:20:17.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.410+0000 7f49e6a55640 1 -- 172.21.15.44:0/2052038859 shutdown_connections 2024-04-02T13:20:17.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.410+0000 7f49e6a55640 1 --2- 172.21.15.44:0/2052038859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49e00ae1f0 0x7f49e006ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.410+0000 7f49e6a55640 1 --2- 172.21.15.44:0/2052038859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49e00add60 0x7f49e006b4d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.410+0000 7f49e6a55640 1 --2- 172.21.15.44:0/2052038859 >> v2:172.21.15.69:3300/0 conn(0x7f49e015bfa0 0x7f49e00b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.410+0000 7f49e6a55640 1 -- 172.21.15.44:0/2052038859 >> 172.21.15.44:0/2052038859 conn(0x7f49e006c5e0 msgr2=0x7f49e006c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:17.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.410+0000 7f49e6a55640 1 -- 172.21.15.44:0/2052038859 shutdown_connections 2024-04-02T13:20:17.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.410+0000 7f49e6a55640 1 -- 172.21.15.44:0/2052038859 wait complete. 2024-04-02T13:20:17.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.411+0000 7f49e6a55640 1 Processor -- start 2024-04-02T13:20:17.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.411+0000 7f49e6a55640 1 -- start start 2024-04-02T13:20:17.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49e6a55640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49e00add60 0x7f49e0149600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49e6a55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49e00ae1f0 0x7f49e009b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49e6a55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49e015bfa0 0x7f49e015ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49e6a55640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f49e00b0590 con 0x7f49e00add60 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49e6a55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f49e00b02b0 con 0x7f49e015bfa0 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49e6a55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f49e00af8c0 con 0x7f49e00ae1f0 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49e00add60 0x7f49e0149600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49e00add60 0x7f49e0149600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59586/0 (socket says 172.21.15.44:59586) 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49dffff640 1 -- 172.21.15.44:0/1407396667 learned_addr learned my addr 172.21.15.44:0/1407396667 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49e4ff2640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49e015bfa0 0x7f49e015ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49df7fe640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49e00ae1f0 0x7f49e009b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49dffff640 1 -- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49e00ae1f0 msgr2=0x7f49e009b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49dffff640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49e00ae1f0 0x7f49e009b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49dffff640 1 -- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49e015bfa0 msgr2=0x7f49e015ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49dffff640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49e015bfa0 0x7f49e015ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49dffff640 1 -- 172.21.15.44:0/1407396667 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e009de60 con 0x7f49e00add60 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.412+0000 7f49dffff640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49e00add60 0x7f49e0149600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f49d400c820 tx=0x7f49d400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:17.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.413+0000 7f49dcff9640 1 -- 172.21.15.44:0/1407396667 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4015070 con 0x7f49e00add60 2024-04-02T13:20:17.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.413+0000 7f49dcff9640 1 -- 172.21.15.44:0/1407396667 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d4004020 con 0x7f49e00add60 2024-04-02T13:20:17.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.413+0000 7f49e6a55640 1 -- 172.21.15.44:0/1407396667 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49e009e140 con 0x7f49e00add60 2024-04-02T13:20:17.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.413+0000 7f49dcff9640 1 -- 172.21.15.44:0/1407396667 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d40043f0 con 0x7f49e00add60 2024-04-02T13:20:17.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.413+0000 7f49e6a55640 1 -- 172.21.15.44:0/1407396667 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49e009e700 con 0x7f49e00add60 2024-04-02T13:20:17.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.414+0000 7f49e6a55640 1 -- 172.21.15.44:0/1407396667 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49e0068c00 con 0x7f49e00add60 2024-04-02T13:20:17.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.414+0000 7f49dcff9640 1 -- 172.21.15.44:0/1407396667 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f49d4007830 con 0x7f49e00add60 2024-04-02T13:20:17.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.414+0000 7f49dcff9640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f49cc06c860 0x7f49cc06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.415+0000 7f49dcff9640 1 -- 172.21.15.44:0/1407396667 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f49d408b470 con 0x7f49e00add60 2024-04-02T13:20:17.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.419+0000 7f49dcff9640 1 -- 172.21.15.44:0/1407396667 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f49d40595f0 con 0x7f49e00add60 2024-04-02T13:20:17.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.421+0000 7f49df7fe640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f49cc06c860 0x7f49cc06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.422+0000 7f49df7fe640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f49cc06c860 0x7f49cc06ed20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f49d0036000 tx=0x7f49d003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:17.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.607+0000 7f49e6a55640 1 -- 172.21.15.44:0/1407396667 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f49e006cd40 con 0x7f49e00add60 2024-04-02T13:20:17.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.608+0000 7f49dcff9640 1 -- 172.21.15.44:0/1407396667 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f49d405d8f0 con 0x7f49e00add60 2024-04-02T13:20:17.609 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:20:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.610+0000 7f49e6a55640 1 -- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f49cc06c860 msgr2=0x7f49cc06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.610+0000 7f49e6a55640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f49cc06c860 0x7f49cc06ed20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f49d0036000 tx=0x7f49d003d040 comp rx=0 tx=0).stop 2024-04-02T13:20:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.610+0000 7f49e6a55640 1 -- 172.21.15.44:0/1407396667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49e00add60 msgr2=0x7f49e0149600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.610+0000 7f49e6a55640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49e00add60 0x7f49e0149600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f49d400c820 tx=0x7f49d400cce0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.610+0000 7f49e6a55640 1 -- 172.21.15.44:0/1407396667 shutdown_connections 2024-04-02T13:20:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.610+0000 7f49e6a55640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f49cc06c860 0x7f49cc06ed20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.610+0000 7f49e6a55640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49e015bfa0 0x7f49e015ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.610+0000 7f49e6a55640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49e00ae1f0 0x7f49e009b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.610+0000 7f49e6a55640 1 --2- 172.21.15.44:0/1407396667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49e00add60 0x7f49e0149600 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.610+0000 7f49e6a55640 1 -- 172.21.15.44:0/1407396667 >> 172.21.15.44:0/1407396667 conn(0x7f49e006c5e0 msgr2=0x7f49e015a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:17.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.610+0000 7f49e6a55640 1 -- 172.21.15.44:0/1407396667 shutdown_connections 2024-04-02T13:20:17.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.610+0000 7f49e6a55640 1 -- 172.21.15.44:0/1407396667 wait complete. 2024-04-02T13:20:17.622 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T13:20:17.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.723+0000 7fbf19d9d640 1 Processor -- start 2024-04-02T13:20:17.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.723+0000 7fbf19d9d640 1 -- start start 2024-04-02T13:20:17.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.724+0000 7fbf19d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf14162550 0x7fbf140d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.724+0000 7fbf19d9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbf141629e0 0x7fbf140b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.724+0000 7fbf19d9d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf1406d1d0 con 0x7fbf1415e270 2024-04-02T13:20:17.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.724+0000 7fbf19d9d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbf1406dde0 con 0x7fbf141629e0 2024-04-02T13:20:17.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.724+0000 7fbf19d9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbf1406d4b0 con 0x7fbf14162550 2024-04-02T13:20:17.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.724+0000 7fbf13fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf14162550 0x7fbf140d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.724+0000 7fbf1959c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbf141629e0 0x7fbf140b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.724+0000 7fbf18d9b640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbf1415e270 0x7fbf14161e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:46584/0 (socket says 172.21.15.44:46584) 2024-04-02T13:20:17.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.724+0000 7fbf18d9b640 1 -- 172.21.15.44:0/542530128 learned_addr learned my addr 172.21.15.44:0/542530128 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:17.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.725+0000 7fbf13fff640 1 -- 172.21.15.44:0/542530128 >> v2:172.21.15.69:3300/0 conn(0x7fbf141629e0 msgr2=0x7fbf140b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:17.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.725+0000 7fbf13fff640 1 --2- 172.21.15.44:0/542530128 >> v2:172.21.15.69:3300/0 conn(0x7fbf141629e0 0x7fbf140b0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.725+0000 7fbf13fff640 1 -- 172.21.15.44:0/542530128 >> v1:172.21.15.69:6789/0 conn(0x7fbf1415e270 legacy=0x7fbf14161e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.725+0000 7fbf13fff640 1 -- 172.21.15.44:0/542530128 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf1406d790 con 0x7fbf14162550 2024-04-02T13:20:17.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.725+0000 7fbf1959c640 1 --2- 172.21.15.44:0/542530128 >> v2:172.21.15.69:3300/0 conn(0x7fbf141629e0 0x7fbf140b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:17.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.725+0000 7fbf13fff640 1 --2- 172.21.15.44:0/542530128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf14162550 0x7fbf140d4a10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009d90 tx=0x7fbf0402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6c5dcf55546ec42 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:17.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.725+0000 7fbf137fe640 1 -- 172.21.15.44:0/542530128 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04039070 con 0x7fbf14162550 2024-04-02T13:20:17.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.726+0000 7fbf137fe640 1 -- 172.21.15.44:0/542530128 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf0402cc50 con 0x7fbf14162550 2024-04-02T13:20:17.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.726+0000 7fbf137fe640 1 -- 172.21.15.44:0/542530128 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04034440 con 0x7fbf14162550 2024-04-02T13:20:17.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.726+0000 7fbf19d9d640 1 -- 172.21.15.44:0/542530128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf14162550 msgr2=0x7fbf140d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.726+0000 7fbf19d9d640 1 --2- 172.21.15.44:0/542530128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf14162550 0x7fbf140d4a10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009d90 tx=0x7fbf0402bea0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.726+0000 7fbf19d9d640 1 -- 172.21.15.44:0/542530128 shutdown_connections 2024-04-02T13:20:17.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.726+0000 7fbf19d9d640 1 --2- 172.21.15.44:0/542530128 >> v2:172.21.15.69:3300/0 conn(0x7fbf141629e0 0x7fbf140b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.726+0000 7fbf19d9d640 1 --2- 172.21.15.44:0/542530128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf14162550 0x7fbf140d4a10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.726+0000 7fbf19d9d640 1 -- 172.21.15.44:0/542530128 >> 172.21.15.44:0/542530128 conn(0x7fbf140d3ef0 msgr2=0x7fbf140d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:17.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.726+0000 7fbf19d9d640 1 -- 172.21.15.44:0/542530128 shutdown_connections 2024-04-02T13:20:17.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.726+0000 7fbf19d9d640 1 -- 172.21.15.44:0/542530128 wait complete. 2024-04-02T13:20:17.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.727+0000 7fbf19d9d640 1 Processor -- start 2024-04-02T13:20:17.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.727+0000 7fbf19d9d640 1 -- start start 2024-04-02T13:20:17.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf19d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1415e270 0x7fbf1414a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf19d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf14162550 0x7fbf140b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf19d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf141629e0 0x7fbf140cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf19d9d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1415e650 con 0x7fbf1415e270 2024-04-02T13:20:17.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf19d9d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1406cef0 con 0x7fbf141629e0 2024-04-02T13:20:17.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf19d9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbf140b1390 con 0x7fbf14162550 2024-04-02T13:20:17.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf18d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1415e270 0x7fbf1414a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf13fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf14162550 0x7fbf140b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf18d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1415e270 0x7fbf1414a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59592/0 (socket says 172.21.15.44:59592) 2024-04-02T13:20:17.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf1959c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf141629e0 0x7fbf140cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf18d9b640 1 -- 172.21.15.44:0/3986361171 learned_addr learned my addr 172.21.15.44:0/3986361171 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf18d9b640 1 -- 172.21.15.44:0/3986361171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf14162550 msgr2=0x7fbf140b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf18d9b640 1 --2- 172.21.15.44:0/3986361171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf14162550 0x7fbf140b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf18d9b640 1 -- 172.21.15.44:0/3986361171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf141629e0 msgr2=0x7fbf140cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf18d9b640 1 --2- 172.21.15.44:0/3986361171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf141629e0 0x7fbf140cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.728+0000 7fbf18d9b640 1 -- 172.21.15.44:0/3986361171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf140b41c0 con 0x7fbf1415e270 2024-04-02T13:20:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.729+0000 7fbf18d9b640 1 --2- 172.21.15.44:0/3986361171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1415e270 0x7fbf1414a1c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00c400 tx=0x7fbefc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.729+0000 7fbf117fa640 1 -- 172.21.15.44:0/3986361171 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc017070 con 0x7fbf1415e270 2024-04-02T13:20:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.729+0000 7fbf117fa640 1 -- 172.21.15.44:0/3986361171 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbefc005ce0 con 0x7fbf1415e270 2024-04-02T13:20:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.729+0000 7fbf19d9d640 1 -- 172.21.15.44:0/3986361171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf140b44a0 con 0x7fbf1415e270 2024-04-02T13:20:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.729+0000 7fbf117fa640 1 -- 172.21.15.44:0/3986361171 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc0072f0 con 0x7fbf1415e270 2024-04-02T13:20:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.729+0000 7fbf19d9d640 1 -- 172.21.15.44:0/3986361171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf140b4a60 con 0x7fbf1415e270 2024-04-02T13:20:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.730+0000 7fbf19d9d640 1 -- 172.21.15.44:0/3986361171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf1404f8d0 con 0x7fbf1415e270 2024-04-02T13:20:17.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.730+0000 7fbf117fa640 1 -- 172.21.15.44:0/3986361171 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7fbefc01a070 con 0x7fbf1415e270 2024-04-02T13:20:17.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.731+0000 7fbf117fa640 1 --2- 172.21.15.44:0/3986361171 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbef406c890 0x7fbef406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:17.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.731+0000 7fbf117fa640 1 -- 172.21.15.44:0/3986361171 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fbefc08a990 con 0x7fbf1415e270 2024-04-02T13:20:17.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.731+0000 7fbf13fff640 1 --2- 172.21.15.44:0/3986361171 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbef406c890 0x7fbef406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:17.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.731+0000 7fbf13fff640 1 --2- 172.21.15.44:0/3986361171 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbef406c890 0x7fbef406ed50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbf04036000 tx=0x7fbf04030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:17.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.735+0000 7fbf117fa640 1 -- 172.21.15.44:0/3986361171 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbefc058b90 con 0x7fbf1415e270 2024-04-02T13:20:17.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.923+0000 7fbf19d9d640 1 -- 172.21.15.44:0/3986361171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fbf14074e30 con 0x7fbf1415e270 2024-04-02T13:20:17.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.929+0000 7fbf117fa640 1 -- 172.21.15.44:0/3986361171 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7fbefc05ce90 con 0x7fbf1415e270 2024-04-02T13:20:17.930 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:17.931 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-04-02T13:20:17.931 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:20:17.931 INFO:teuthology.orchestra.run.smithi044.stdout: "poolnum": 1, 2024-04-02T13:20:17.931 INFO:teuthology.orchestra.run.smithi044.stdout: "poolname": ".mgr" 2024-04-02T13:20:17.931 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:20:17.931 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-04-02T13:20:17.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.931+0000 7fbf19d9d640 1 -- 172.21.15.44:0/3986361171 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbef406c890 msgr2=0x7fbef406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.931+0000 7fbf19d9d640 1 --2- 172.21.15.44:0/3986361171 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbef406c890 0x7fbef406ed50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbf04036000 tx=0x7fbf04030000 comp rx=0 tx=0).stop 2024-04-02T13:20:17.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.931+0000 7fbf19d9d640 1 -- 172.21.15.44:0/3986361171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1415e270 msgr2=0x7fbf1414a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:17.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.931+0000 7fbf19d9d640 1 --2- 172.21.15.44:0/3986361171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1415e270 0x7fbf1414a1c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00c400 tx=0x7fbefc00c800 comp rx=0 tx=0).stop 2024-04-02T13:20:17.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.931+0000 7fbf19d9d640 1 -- 172.21.15.44:0/3986361171 shutdown_connections 2024-04-02T13:20:17.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.931+0000 7fbf19d9d640 1 --2- 172.21.15.44:0/3986361171 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbef406c890 0x7fbef406ed50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.931+0000 7fbf19d9d640 1 --2- 172.21.15.44:0/3986361171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf141629e0 0x7fbf140cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.931+0000 7fbf19d9d640 1 --2- 172.21.15.44:0/3986361171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf14162550 0x7fbf140b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.931+0000 7fbf19d9d640 1 --2- 172.21.15.44:0/3986361171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1415e270 0x7fbf1414a1c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:17.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.931+0000 7fbf19d9d640 1 -- 172.21.15.44:0/3986361171 >> 172.21.15.44:0/3986361171 conn(0x7fbf140d3ef0 msgr2=0x7fbf140d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:17.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.931+0000 7fbf19d9d640 1 -- 172.21.15.44:0/3986361171 shutdown_connections 2024-04-02T13:20:17.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:17.931+0000 7fbf19d9d640 1 -- 172.21.15.44:0/3986361171 wait complete. 2024-04-02T13:20:17.943 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T13:20:17.943 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T13:20:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.047+0000 7fca3313d640 1 Processor -- start 2024-04-02T13:20:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.047+0000 7fca3313d640 1 -- start start 2024-04-02T13:20:18.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.048+0000 7fca3313d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fca2c06daa0 0x7fca2c0b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:18.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.048+0000 7fca3313d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca2c06de80 con 0x7fca2c0cfdc0 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.048+0000 7fca3313d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca2c0cf000 con 0x7fca2c0d2e00 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.048+0000 7fca3313d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fca2c0cf930 con 0x7fca2c06daa0 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.048+0000 7fca2bfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fca2c0cfdc0 0x7fca2c0d26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54560/0 (socket says 172.21.15.44:54560) 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.048+0000 7fca2bfff640 1 -- 172.21.15.44:0/3944381087 learned_addr learned my addr 172.21.15.44:0/3944381087 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.048+0000 7fca30ed9640 1 --2- 172.21.15.44:0/3944381087 >> v2:172.21.15.69:3300/0 conn(0x7fca2c06daa0 0x7fca2c0b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.048+0000 7fca2b7fe640 1 -- 172.21.15.44:0/3944381087 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1574305084 0 0) 0x7fca2c06de80 con 0x7fca2c0cfdc0 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.048+0000 7fca2b7fe640 1 -- 172.21.15.44:0/3944381087 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca14003520 con 0x7fca2c0cfdc0 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.048+0000 7fca30ed9640 1 -- 172.21.15.44:0/3944381087 >> v1:172.21.15.69:6789/0 conn(0x7fca2c0d2e00 legacy=0x7fca2c15dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.048+0000 7fca30ed9640 1 -- 172.21.15.44:0/3944381087 >> v1:172.21.15.44:6789/0 conn(0x7fca2c0cfdc0 legacy=0x7fca2c0d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.048+0000 7fca2b7fe640 1 -- 172.21.15.44:0/3944381087 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2324419074 0 0) 0x7fca2c0cf000 con 0x7fca2c0d2e00 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.049+0000 7fca30ed9640 1 -- 172.21.15.44:0/3944381087 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2c0cf2e0 con 0x7fca2c06daa0 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.049+0000 7fca30ed9640 1 --2- 172.21.15.44:0/3944381087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2c06daa0 0x7fca2c0b35b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca24009d90 tx=0x7fca2402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d3787eadc9aea8d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.049+0000 7fca2b7fe640 1 -- 172.21.15.44:0/3944381087 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24039070 con 0x7fca2c06daa0 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.049+0000 7fca2b7fe640 1 -- 172.21.15.44:0/3944381087 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca2402cd00 con 0x7fca2c06daa0 2024-04-02T13:20:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.049+0000 7fca2b7fe640 1 -- 172.21.15.44:0/3944381087 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24034640 con 0x7fca2c06daa0 2024-04-02T13:20:18.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.050+0000 7fca3313d640 1 -- 172.21.15.44:0/3944381087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2c06daa0 msgr2=0x7fca2c0b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:18.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.050+0000 7fca3313d640 1 --2- 172.21.15.44:0/3944381087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2c06daa0 0x7fca2c0b35b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca24009d90 tx=0x7fca2402bf50 comp rx=0 tx=0).stop 2024-04-02T13:20:18.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.050+0000 7fca3313d640 1 -- 172.21.15.44:0/3944381087 shutdown_connections 2024-04-02T13:20:18.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.050+0000 7fca3313d640 1 --2- 172.21.15.44:0/3944381087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2c06daa0 0x7fca2c0b35b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:18.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.050+0000 7fca3313d640 1 -- 172.21.15.44:0/3944381087 >> 172.21.15.44:0/3944381087 conn(0x7fca2c09d430 msgr2=0x7fca2c0d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:18.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.050+0000 7fca3313d640 1 -- 172.21.15.44:0/3944381087 shutdown_connections 2024-04-02T13:20:18.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.050+0000 7fca3313d640 1 -- 172.21.15.44:0/3944381087 wait complete. 2024-04-02T13:20:18.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.051+0000 7fca3313d640 1 Processor -- start 2024-04-02T13:20:18.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.051+0000 7fca3313d640 1 -- start start 2024-04-02T13:20:18.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca3313d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2c06daa0 0x7fca2c1f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:18.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca3313d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca2c0cfdc0 0x7fca2c156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:18.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca3313d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2c0d2e00 0x7fca2c0b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:18.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca3313d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c0ceda0 con 0x7fca2c0d2e00 2024-04-02T13:20:18.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca3313d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c0cec20 con 0x7fca2c06daa0 2024-04-02T13:20:18.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca3313d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fca2c0d11c0 con 0x7fca2c0cfdc0 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca316da640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2c0d2e00 0x7fca2c0b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca30ed9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2c06daa0 0x7fca2c1f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca316da640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2c0d2e00 0x7fca2c0b5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59596/0 (socket says 172.21.15.44:59596) 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca316da640 1 -- 172.21.15.44:0/1746748462 learned_addr learned my addr 172.21.15.44:0/1746748462 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca30ed9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2c06daa0 0x7fca2c1f5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42930/0 (socket says 172.21.15.44:42930) 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca2bfff640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca2c0cfdc0 0x7fca2c156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca316da640 1 -- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca2c0cfdc0 msgr2=0x7fca2c156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca316da640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca2c0cfdc0 0x7fca2c156720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca316da640 1 -- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2c06daa0 msgr2=0x7fca2c1f5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca316da640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2c06daa0 0x7fca2c1f5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca316da640 1 -- 172.21.15.44:0/1746748462 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2c0cb1e0 con 0x7fca2c0d2e00 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca2bfff640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca2c0cfdc0 0x7fca2c156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.052+0000 7fca30ed9640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2c06daa0 0x7fca2c1f5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:18.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.053+0000 7fca316da640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2c0d2e00 0x7fca2c0b5ae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00c410 tx=0x7fca1c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:18.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.053+0000 7fca297fa640 1 -- 172.21.15.44:0/1746748462 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c017070 con 0x7fca2c0d2e00 2024-04-02T13:20:18.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.053+0000 7fca297fa640 1 -- 172.21.15.44:0/1746748462 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca1c0035f0 con 0x7fca2c0d2e00 2024-04-02T13:20:18.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.053+0000 7fca3313d640 1 -- 172.21.15.44:0/1746748462 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca2c0cb4c0 con 0x7fca2c0d2e00 2024-04-02T13:20:18.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.053+0000 7fca297fa640 1 -- 172.21.15.44:0/1746748462 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c0039e0 con 0x7fca2c0d2e00 2024-04-02T13:20:18.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.053+0000 7fca3313d640 1 -- 172.21.15.44:0/1746748462 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca2c0cba80 con 0x7fca2c0d2e00 2024-04-02T13:20:18.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.054+0000 7fca3313d640 1 -- 172.21.15.44:0/1746748462 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca2c162f20 con 0x7fca2c0d2e00 2024-04-02T13:20:18.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.054+0000 7fca297fa640 1 -- 172.21.15.44:0/1746748462 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7fca1c01a070 con 0x7fca2c0d2e00 2024-04-02T13:20:18.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.055+0000 7fca297fa640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca1806c860 0x7fca1806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:18.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.055+0000 7fca297fa640 1 -- 172.21.15.44:0/1746748462 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fca1c08ad60 con 0x7fca2c0d2e00 2024-04-02T13:20:18.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.055+0000 7fca30ed9640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca1806c860 0x7fca1806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:18.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.055+0000 7fca30ed9640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca1806c860 0x7fca1806ed20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fca24036000 tx=0x7fca24030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:18.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.059+0000 7fca297fa640 1 -- 172.21.15.44:0/1746748462 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fca2c162f20 con 0x7fca2c0d2e00 2024-04-02T13:20:18.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.243+0000 7fca3313d640 1 -- 172.21.15.44:0/1746748462 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fca2c0684f0 con 0x7fca2c0d2e00 2024-04-02T13:20:18.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.862+0000 7fca297fa640 1 -- 172.21.15.44:0/1746748462 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7fca1c021020 con 0x7fca2c0d2e00 2024-04-02T13:20:18.864 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' created 2024-04-02T13:20:18.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.865+0000 7fca3313d640 1 -- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca1806c860 msgr2=0x7fca1806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:18.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.865+0000 7fca3313d640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca1806c860 0x7fca1806ed20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fca24036000 tx=0x7fca24030000 comp rx=0 tx=0).stop 2024-04-02T13:20:18.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.865+0000 7fca3313d640 1 -- 172.21.15.44:0/1746748462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2c0d2e00 msgr2=0x7fca2c0b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:18.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.865+0000 7fca3313d640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2c0d2e00 0x7fca2c0b5ae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00c410 tx=0x7fca1c00c8d0 comp rx=0 tx=0).stop 2024-04-02T13:20:18.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.865+0000 7fca3313d640 1 -- 172.21.15.44:0/1746748462 shutdown_connections 2024-04-02T13:20:18.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.865+0000 7fca3313d640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca1806c860 0x7fca1806ed20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:18.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.865+0000 7fca3313d640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2c0d2e00 0x7fca2c0b5ae0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:18.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.865+0000 7fca3313d640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca2c0cfdc0 0x7fca2c156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:18.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.865+0000 7fca3313d640 1 --2- 172.21.15.44:0/1746748462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2c06daa0 0x7fca2c1f5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:18.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.865+0000 7fca3313d640 1 -- 172.21.15.44:0/1746748462 >> 172.21.15.44:0/1746748462 conn(0x7fca2c09d430 msgr2=0x7fca2c06c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:18.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.866+0000 7fca3313d640 1 -- 172.21.15.44:0/1746748462 shutdown_connections 2024-04-02T13:20:18.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.866+0000 7fca3313d640 1 -- 172.21.15.44:0/1746748462 wait complete. 2024-04-02T13:20:18.885 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-02T13:20:19.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.997+0000 7f4f9fd5f640 1 Processor -- start 2024-04-02T13:20:19.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.997+0000 7f4f9fd5f640 1 -- start start 2024-04-02T13:20:19.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.998+0000 7f4f9fd5f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4f9815c1a0 0x7f4f9806f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:19.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.998+0000 7f4f9fd5f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4f980d21c0 0x7f4f980cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:19.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.998+0000 7f4f9fd5f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9815c580 con 0x7f4f980cfd60 2024-04-02T13:20:19.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.998+0000 7f4f9fd5f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4f98070ad0 con 0x7f4f980d21c0 2024-04-02T13:20:19.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.998+0000 7f4f9fd5f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4f980d01c0 con 0x7f4f9815c1a0 2024-04-02T13:20:19.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.998+0000 7f4f9e2fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4f980d21c0 0x7f4f980cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:19.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.998+0000 7f4f9d2fa640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4f980cfd60 0x7f4f980d1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54574/0 (socket says 172.21.15.44:54574) 2024-04-02T13:20:19.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.998+0000 7f4f9d2fa640 1 -- 172.21.15.44:0/2292670375 learned_addr learned my addr 172.21.15.44:0/2292670375 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:19.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.998+0000 7f4f9dafb640 1 --2- 172.21.15.44:0/2292670375 >> v2:172.21.15.69:3300/0 conn(0x7f4f9815c1a0 0x7f4f9806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:19.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.999+0000 7f4f9e2fc640 1 -- 172.21.15.44:0/2292670375 >> v2:172.21.15.69:3300/0 conn(0x7f4f9815c1a0 msgr2=0x7f4f9806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:19.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.999+0000 7f4f9e2fc640 1 --2- 172.21.15.44:0/2292670375 >> v2:172.21.15.69:3300/0 conn(0x7f4f9815c1a0 0x7f4f9806f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:19.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.999+0000 7f4f9e2fc640 1 -- 172.21.15.44:0/2292670375 >> v1:172.21.15.44:6789/0 conn(0x7f4f980cfd60 legacy=0x7f4f980d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:19.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.999+0000 7f4f9e2fc640 1 -- 172.21.15.44:0/2292670375 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9806fa30 con 0x7f4f980d21c0 2024-04-02T13:20:19.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.999+0000 7f4f9caf9640 1 -- 172.21.15.44:0/2292670375 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1017736750 0 0) 0x7f4f9815c580 con 0x7f4f980cfd60 2024-04-02T13:20:19.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:18.999+0000 7f4f9e2fc640 1 --2- 172.21.15.44:0/2292670375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f980d21c0 0x7f4f980cc9a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4f9809b780 tx=0x7f4f9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb73d2a5544957fe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:19.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.000+0000 7f4f9caf9640 1 -- 172.21.15.44:0/2292670375 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90035020 con 0x7f4f980d21c0 2024-04-02T13:20:19.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.000+0000 7f4f9caf9640 1 -- 172.21.15.44:0/2292670375 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f90031830 con 0x7f4f980d21c0 2024-04-02T13:20:19.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.000+0000 7f4f9caf9640 1 -- 172.21.15.44:0/2292670375 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90031c00 con 0x7f4f980d21c0 2024-04-02T13:20:19.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.000+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/2292670375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f980d21c0 msgr2=0x7f4f980cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:19.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.000+0000 7f4f9fd5f640 1 --2- 172.21.15.44:0/2292670375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f980d21c0 0x7f4f980cc9a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4f9809b780 tx=0x7f4f9002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:20:19.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.000+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/2292670375 shutdown_connections 2024-04-02T13:20:19.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.000+0000 7f4f9fd5f640 1 --2- 172.21.15.44:0/2292670375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f980d21c0 0x7f4f980cc9a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:19.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.000+0000 7f4f9fd5f640 1 --2- 172.21.15.44:0/2292670375 >> v2:172.21.15.69:3300/0 conn(0x7f4f9815c1a0 0x7f4f9806f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:19.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.000+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/2292670375 >> 172.21.15.44:0/2292670375 conn(0x7f4f9809aa20 msgr2=0x7f4f9809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:19.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.000+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/2292670375 shutdown_connections 2024-04-02T13:20:19.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.000+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/2292670375 wait complete. 2024-04-02T13:20:19.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.001+0000 7f4f9fd5f640 1 Processor -- start 2024-04-02T13:20:19.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.001+0000 7f4f9fd5f640 1 -- start start 2024-04-02T13:20:19.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.002+0000 7f4f9fd5f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f980cfd60 0x7f4f980cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:19.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.002+0000 7f4f9fd5f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f980d21c0 0x7f4f9815ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:19.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.002+0000 7f4f9fd5f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f9815c1a0 0x7f4f98157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:19.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.002+0000 7f4f9fd5f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4f980702d0 con 0x7f4f9815c1a0 2024-04-02T13:20:19.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.002+0000 7f4f9fd5f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9806fff0 con 0x7f4f980d21c0 2024-04-02T13:20:19.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.002+0000 7f4f9fd5f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4f980705b0 con 0x7f4f980cfd60 2024-04-02T13:20:19.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.002+0000 7f4f9e2fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f9815c1a0 0x7f4f98157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:19.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.002+0000 7f4f9e2fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f9815c1a0 0x7f4f98157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59612/0 (socket says 172.21.15.44:59612) 2024-04-02T13:20:19.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.002+0000 7f4f9e2fc640 1 -- 172.21.15.44:0/213986874 learned_addr learned my addr 172.21.15.44:0/213986874 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:19.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.002+0000 7f4f9d2fa640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f980d21c0 0x7f4f9815ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:19.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.002+0000 7f4f9dafb640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f980cfd60 0x7f4f980cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:19.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.003+0000 7f4f9e2fc640 1 -- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f980cfd60 msgr2=0x7f4f980cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:19.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.003+0000 7f4f9e2fc640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f980cfd60 0x7f4f980cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:19.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.003+0000 7f4f9e2fc640 1 -- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f980d21c0 msgr2=0x7f4f9815ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:19.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.003+0000 7f4f9e2fc640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f980d21c0 0x7f4f9815ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:19.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.003+0000 7f4f9e2fc640 1 -- 172.21.15.44:0/213986874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f980cba20 con 0x7f4f9815c1a0 2024-04-02T13:20:19.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.003+0000 7f4f9d2fa640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f980d21c0 0x7f4f9815ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:19.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.003+0000 7f4f9e2fc640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f9815c1a0 0x7f4f98157f60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4f9002bee0 tx=0x7f4f900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:19.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.003+0000 7f4f967fc640 1 -- 172.21.15.44:0/213986874 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90035040 con 0x7f4f9815c1a0 2024-04-02T13:20:19.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.003+0000 7f4f967fc640 1 -- 172.21.15.44:0/213986874 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f9003b860 con 0x7f4f9815c1a0 2024-04-02T13:20:19.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.003+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/213986874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f980ca440 con 0x7f4f9815c1a0 2024-04-02T13:20:19.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.003+0000 7f4f967fc640 1 -- 172.21.15.44:0/213986874 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9003bc50 con 0x7f4f9815c1a0 2024-04-02T13:20:19.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.003+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/213986874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f980cace0 con 0x7f4f9815c1a0 2024-04-02T13:20:19.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.004+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/213986874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f98068c00 con 0x7f4f9815c1a0 2024-04-02T13:20:19.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.004+0000 7f4f967fc640 1 -- 172.21.15.44:0/213986874 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f4f90038020 con 0x7f4f9815c1a0 2024-04-02T13:20:19.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.005+0000 7f4f967fc640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4f8406c860 0x7f4f8406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:19.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.005+0000 7f4f967fc640 1 -- 172.21.15.44:0/213986874 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f4f900ad380 con 0x7f4f9815c1a0 2024-04-02T13:20:19.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.005+0000 7f4f9dafb640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4f8406c860 0x7f4f8406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:19.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.005+0000 7f4f9dafb640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4f8406c860 0x7f4f8406ed20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4f88011340 tx=0x7f4f8800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:19.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.009+0000 7f4f967fc640 1 -- 172.21.15.44:0/213986874 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4f90031800 con 0x7f4f9815c1a0 2024-04-02T13:20:19.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.193+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/213986874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f4f9806d760 con 0x7f4f9815c1a0 2024-04-02T13:20:19.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.869+0000 7f4f967fc640 1 -- 172.21.15.44:0/213986874 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) ==== 171+0+0 (secure 0 0 0) 0x7f4f9007b4b0 con 0x7f4f9815c1a0 2024-04-02T13:20:19.872 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' created 2024-04-02T13:20:19.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.872+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4f8406c860 msgr2=0x7f4f8406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:19.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.872+0000 7f4f9fd5f640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4f8406c860 0x7f4f8406ed20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4f88011340 tx=0x7f4f8800b040 comp rx=0 tx=0).stop 2024-04-02T13:20:19.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.872+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/213986874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f9815c1a0 msgr2=0x7f4f98157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:19.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.872+0000 7f4f9fd5f640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f9815c1a0 0x7f4f98157f60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4f9002bee0 tx=0x7f4f900046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:19.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.872+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/213986874 shutdown_connections 2024-04-02T13:20:19.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.872+0000 7f4f9fd5f640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4f8406c860 0x7f4f8406ed20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:19.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.872+0000 7f4f9fd5f640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f9815c1a0 0x7f4f98157f60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:19.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.872+0000 7f4f9fd5f640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f980d21c0 0x7f4f9815ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:19.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.872+0000 7f4f9fd5f640 1 --2- 172.21.15.44:0/213986874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f980cfd60 0x7f4f980cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:19.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.872+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/213986874 >> 172.21.15.44:0/213986874 conn(0x7f4f9809aa20 msgr2=0x7f4f980af410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:19.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.873+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/213986874 shutdown_connections 2024-04-02T13:20:19.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:19.873+0000 7f4f9fd5f640 1 -- 172.21.15.44:0/213986874 wait complete. 2024-04-02T13:20:19.902 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T13:20:20.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.009+0000 7fbd1d463640 1 Processor -- start 2024-04-02T13:20:20.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.009+0000 7fbd1d463640 1 -- start start 2024-04-02T13:20:20.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd1d463640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbd1815e420 0x7fbd180ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:20.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd1d463640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbd180721f0 0x7fbd180cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:20.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd1d463640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd180cfc40 0x7fbd18161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:20.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd1d463640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fbd180d00b0 con 0x7fbd180721f0 2024-04-02T13:20:20.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd1d463640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbd1806c220 con 0x7fbd1815e420 2024-04-02T13:20:20.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd1d463640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbd1815f7e0 con 0x7fbd180cfc40 2024-04-02T13:20:20.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd0ffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbd180721f0 0x7fbd180cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:20.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd0ffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbd180721f0 0x7fbd180cf700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59628/0 (socket says 172.21.15.44:59628) 2024-04-02T13:20:20.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd0ffff640 1 -- 172.21.15.44:0/3539691353 learned_addr learned my addr 172.21.15.44:0/3539691353 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:20.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd16ffd640 1 --2- 172.21.15.44:0/3539691353 >> v2:172.21.15.69:3300/0 conn(0x7fbd1815e420 0x7fbd180ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:20.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd177fe640 1 --2- 172.21.15.44:0/3539691353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd180cfc40 0x7fbd18161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:20.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd0ffff640 1 -- 172.21.15.44:0/3539691353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd180cfc40 msgr2=0x7fbd18161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:20.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd0ffff640 1 --2- 172.21.15.44:0/3539691353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd180cfc40 0x7fbd18161eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:20.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd0ffff640 1 -- 172.21.15.44:0/3539691353 >> v2:172.21.15.69:3300/0 conn(0x7fbd1815e420 msgr2=0x7fbd180ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:20.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd0ffff640 1 --2- 172.21.15.44:0/3539691353 >> v2:172.21.15.69:3300/0 conn(0x7fbd1815e420 0x7fbd180ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:20.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd0ffff640 1 -- 172.21.15.44:0/3539691353 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1815ec60 con 0x7fbd180721f0 2024-04-02T13:20:20.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd16ffd640 1 --2- 172.21.15.44:0/3539691353 >> v2:172.21.15.69:3300/0 conn(0x7fbd1815e420 0x7fbd180ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:20.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.010+0000 7fbd177fe640 1 --2- 172.21.15.44:0/3539691353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd180cfc40 0x7fbd18161eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:20.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.011+0000 7fbd0ffff640 1 --2- 172.21.15.44:0/3539691353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd180721f0 0x7fbd180cf700 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd1806dc40 tx=0x7fbd0002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b648184115430df4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:20.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.011+0000 7fbd167fc640 1 -- 172.21.15.44:0/3539691353 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd00038070 con 0x7fbd180721f0 2024-04-02T13:20:20.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.011+0000 7fbd167fc640 1 -- 172.21.15.44:0/3539691353 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd0002f050 con 0x7fbd180721f0 2024-04-02T13:20:20.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.011+0000 7fbd167fc640 1 -- 172.21.15.44:0/3539691353 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd00031970 con 0x7fbd180721f0 2024-04-02T13:20:20.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.012+0000 7fbd1d463640 1 -- 172.21.15.44:0/3539691353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd180721f0 msgr2=0x7fbd180cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:20.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.012+0000 7fbd1d463640 1 --2- 172.21.15.44:0/3539691353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd180721f0 0x7fbd180cf700 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd1806dc40 tx=0x7fbd0002b7f0 comp rx=0 tx=0).stop 2024-04-02T13:20:20.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.012+0000 7fbd1d463640 1 -- 172.21.15.44:0/3539691353 shutdown_connections 2024-04-02T13:20:20.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.012+0000 7fbd1d463640 1 --2- 172.21.15.44:0/3539691353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd180cfc40 0x7fbd18161eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:20.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.012+0000 7fbd1d463640 1 --2- 172.21.15.44:0/3539691353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd180721f0 0x7fbd180cf700 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:20.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.012+0000 7fbd1d463640 1 --2- 172.21.15.44:0/3539691353 >> v2:172.21.15.69:3300/0 conn(0x7fbd1815e420 0x7fbd180ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:20.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.012+0000 7fbd1d463640 1 -- 172.21.15.44:0/3539691353 >> 172.21.15.44:0/3539691353 conn(0x7fbd180b2a10 msgr2=0x7fbd180b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:20.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.012+0000 7fbd1d463640 1 -- 172.21.15.44:0/3539691353 shutdown_connections 2024-04-02T13:20:20.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.012+0000 7fbd1d463640 1 -- 172.21.15.44:0/3539691353 wait complete. 2024-04-02T13:20:20.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.013+0000 7fbd1d463640 1 Processor -- start 2024-04-02T13:20:20.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.013+0000 7fbd1d463640 1 -- start start 2024-04-02T13:20:20.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd1d463640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd180721f0 0x7fbd181e4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:20.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd1d463640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd180cfc40 0x7fbd1814a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:20.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd1d463640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd1815e420 0x7fbd180add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:20.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd1d463640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1815f500 con 0x7fbd1815e420 2024-04-02T13:20:20.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd1d463640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1815f220 con 0x7fbd180cfc40 2024-04-02T13:20:20.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd1d463640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbd180cf1e0 con 0x7fbd180721f0 2024-04-02T13:20:20.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd177fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd1815e420 0x7fbd180add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:20.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd177fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd1815e420 0x7fbd180add60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59634/0 (socket says 172.21.15.44:59634) 2024-04-02T13:20:20.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd177fe640 1 -- 172.21.15.44:0/2501502639 learned_addr learned my addr 172.21.15.44:0/2501502639 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:20.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd16ffd640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd180721f0 0x7fbd181e4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:20.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd0ffff640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd180cfc40 0x7fbd1814a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:20.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd177fe640 1 -- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd180721f0 msgr2=0x7fbd181e4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:20.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd177fe640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd180721f0 0x7fbd181e4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:20.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd177fe640 1 -- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd180cfc40 msgr2=0x7fbd1814a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:20.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd177fe640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd180cfc40 0x7fbd1814a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:20.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd177fe640 1 -- 172.21.15.44:0/2501502639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1815d0b0 con 0x7fbd1815e420 2024-04-02T13:20:20.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd0ffff640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd180cfc40 0x7fbd1814a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:20.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd16ffd640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd180721f0 0x7fbd181e4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:20.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd177fe640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd1815e420 0x7fbd180add60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000cf20 tx=0x7fbd1000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:20.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd0f7fe640 1 -- 172.21.15.44:0/2501502639 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1001a070 con 0x7fbd1815e420 2024-04-02T13:20:20.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd0f7fe640 1 -- 172.21.15.44:0/2501502639 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd1000d840 con 0x7fbd1815e420 2024-04-02T13:20:20.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd0f7fe640 1 -- 172.21.15.44:0/2501502639 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1000dc30 con 0x7fbd1815e420 2024-04-02T13:20:20.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.014+0000 7fbd1d463640 1 -- 172.21.15.44:0/2501502639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd1815d350 con 0x7fbd1815e420 2024-04-02T13:20:20.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.015+0000 7fbd1d463640 1 -- 172.21.15.44:0/2501502639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd1815aa60 con 0x7fbd1815e420 2024-04-02T13:20:20.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.015+0000 7fbd1d463640 1 -- 172.21.15.44:0/2501502639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd180684f0 con 0x7fbd1815e420 2024-04-02T13:20:20.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.016+0000 7fbd0f7fe640 1 -- 172.21.15.44:0/2501502639 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbd1001d070 con 0x7fbd1815e420 2024-04-02T13:20:20.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.016+0000 7fbd0f7fe640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcf806c8f0 0x7fbcf806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:20.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.016+0000 7fbd0f7fe640 1 -- 172.21.15.44:0/2501502639 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7fbd1008c290 con 0x7fbd1815e420 2024-04-02T13:20:20.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.016+0000 7fbd16ffd640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcf806c8f0 0x7fbcf806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:20.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.017+0000 7fbd16ffd640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcf806c8f0 0x7fbcf806edb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800aa30 tx=0x7fbd08008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:20.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.021+0000 7fbd0f7fe640 1 -- 172.21.15.44:0/2501502639 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbd1005a7b0 con 0x7fbd1815e420 2024-04-02T13:20:20.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.245+0000 7fbd1d463640 1 -- 172.21.15.44:0/2501502639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fbd180b3770 con 0x7fbd1815e420 2024-04-02T13:20:20.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.871+0000 7fbd0f7fe640 1 -- 172.21.15.44:0/2501502639 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fbd1005a0b0 con 0x7fbd1815e420 2024-04-02T13:20:20.874 INFO:teuthology.orchestra.run.smithi044.stderr:new fs with metadata pool 2 and data pool 3 2024-04-02T13:20:20.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.876+0000 7fbd1d463640 1 -- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcf806c8f0 msgr2=0x7fbcf806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:20.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.876+0000 7fbd1d463640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcf806c8f0 0x7fbcf806edb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800aa30 tx=0x7fbd08008040 comp rx=0 tx=0).stop 2024-04-02T13:20:20.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.876+0000 7fbd1d463640 1 -- 172.21.15.44:0/2501502639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd1815e420 msgr2=0x7fbd180add60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:20.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.876+0000 7fbd1d463640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd1815e420 0x7fbd180add60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000cf20 tx=0x7fbd1000a710 comp rx=0 tx=0).stop 2024-04-02T13:20:20.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.876+0000 7fbd1d463640 1 -- 172.21.15.44:0/2501502639 shutdown_connections 2024-04-02T13:20:20.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.876+0000 7fbd1d463640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcf806c8f0 0x7fbcf806edb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:20.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.876+0000 7fbd1d463640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd1815e420 0x7fbd180add60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:20.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.876+0000 7fbd1d463640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd180cfc40 0x7fbd1814a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:20.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.876+0000 7fbd1d463640 1 --2- 172.21.15.44:0/2501502639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd180721f0 0x7fbd181e4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:20.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.876+0000 7fbd1d463640 1 -- 172.21.15.44:0/2501502639 >> 172.21.15.44:0/2501502639 conn(0x7fbd180b2a10 msgr2=0x7fbd180b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:20.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.877+0000 7fbd1d463640 1 -- 172.21.15.44:0/2501502639 shutdown_connections 2024-04-02T13:20:20.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:20.877+0000 7fbd1d463640 1 -- 172.21.15.44:0/2501502639 wait complete. 2024-04-02T13:20:20.909 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-02T13:20:20.909 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-02T13:20:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.017+0000 7f32b6329640 1 Processor -- start 2024-04-02T13:20:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.017+0000 7f32b6329640 1 -- start start 2024-04-02T13:20:21.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.018+0000 7f32b6329640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b00d00d0 0x7f32b015ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:21.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.018+0000 7f32b6329640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f32b00d2470 0x7f32b015e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:21.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.018+0000 7f32b6329640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32b0070040 con 0x7f32b00d2090 2024-04-02T13:20:21.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.018+0000 7f32b6329640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f32b006c5e0 con 0x7f32b00d2470 2024-04-02T13:20:21.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.018+0000 7f32b6329640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f32b0070320 con 0x7f32b00d00d0 2024-04-02T13:20:21.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.018+0000 7f32affff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f32b00d2090 0x7f32b006f580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54584/0 (socket says 172.21.15.44:54584) 2024-04-02T13:20:21.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.018+0000 7f32affff640 1 -- 172.21.15.44:0/3064487979 learned_addr learned my addr 172.21.15.44:0/3064487979 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:21.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.018+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4011860651 0 0) 0x7f32b0070040 con 0x7f32b00d2090 2024-04-02T13:20:21.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3298003520 con 0x7f32b00d2090 2024-04-02T13:20:21.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32af7fe640 1 --2- 172.21.15.44:0/3064487979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b00d00d0 0x7f32b015ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:21.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32b48c6640 1 --2- 172.21.15.44:0/3064487979 >> v2:172.21.15.69:3300/0 conn(0x7f32b00d2470 0x7f32b015e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:21.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 459441250 0 0) 0x7f3298003520 con 0x7f32b00d2090 2024-04-02T13:20:21.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b00d00d0 msgr2=0x7f32b015ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:21.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 --2- 172.21.15.44:0/3064487979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b00d00d0 0x7f32b015ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:21.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 >> v2:172.21.15.69:3300/0 conn(0x7f32b00d2470 msgr2=0x7f32b015e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:21.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 --2- 172.21.15.44:0/3064487979 >> v2:172.21.15.69:3300/0 conn(0x7f32b00d2470 0x7f32b015e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:21.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b0070600 con 0x7f32b00d2090 2024-04-02T13:20:21.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f32a0001670 con 0x7f32b00d2090 2024-04-02T13:20:21.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32af7fe640 1 --2- 172.21.15.44:0/3064487979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b00d00d0 0x7f32b015ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:21.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 >> v1:172.21.15.44:6789/0 conn(0x7f32b00d2090 legacy=0x7f32b006f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:21.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 --2- 172.21.15.44:0/3064487979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32980039b0 0x7f3298003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:21.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 --2- 172.21.15.44:0/3064487979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b00d00d0 0x7f3298004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:21.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 --2- 172.21.15.44:0/3064487979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3298004d40 0x7f32980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:21.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f32b006c5e0 con 0x7f32980039b0 2024-04-02T13:20:21.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f32b0070320 con 0x7f3298004d40 2024-04-02T13:20:21.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f329800b9c0 con 0x7f32b00d00d0 2024-04-02T13:20:21.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32affff640 1 --2- 172.21.15.44:0/3064487979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32980039b0 0x7f3298003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:21.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32af7fe640 1 --2- 172.21.15.44:0/3064487979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b00d00d0 0x7f3298004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:21.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32b48c6640 1 --2- 172.21.15.44:0/3064487979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3298004d40 0x7f32980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:21.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32affff640 1 -- 172.21.15.44:0/3064487979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b00d00d0 msgr2=0x7f3298004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:21.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32affff640 1 --2- 172.21.15.44:0/3064487979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b00d00d0 0x7f3298004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:21.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32affff640 1 -- 172.21.15.44:0/3064487979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3298004d40 msgr2=0x7f32980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:21.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32affff640 1 --2- 172.21.15.44:0/3064487979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3298004d40 0x7f32980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:21.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.019+0000 7f32affff640 1 -- 172.21.15.44:0/3064487979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3298003520 con 0x7f32980039b0 2024-04-02T13:20:21.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.020+0000 7f32affff640 1 --2- 172.21.15.44:0/3064487979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32980039b0 0x7f3298003dd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f32a000c240 tx=0x7f3298005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d88101fc9d1bd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:21.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.020+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a0007d30 con 0x7f32980039b0 2024-04-02T13:20:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.020+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32a0030070 con 0x7f32980039b0 2024-04-02T13:20:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.020+0000 7f32aeffd640 1 -- 172.21.15.44:0/3064487979 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a002cd80 con 0x7f32980039b0 2024-04-02T13:20:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.021+0000 7f32b6329640 1 -- 172.21.15.44:0/3064487979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32980039b0 msgr2=0x7f3298003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.021+0000 7f32b6329640 1 --2- 172.21.15.44:0/3064487979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32980039b0 0x7f3298003dd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f32a000c240 tx=0x7f3298005830 comp rx=0 tx=0).stop 2024-04-02T13:20:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.021+0000 7f32b48c6640 1 -- 172.21.15.44:0/3064487979 reap_dead start 2024-04-02T13:20:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.021+0000 7f32b6329640 1 -- 172.21.15.44:0/3064487979 shutdown_connections 2024-04-02T13:20:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.021+0000 7f32b6329640 1 -- 172.21.15.44:0/3064487979 >> 172.21.15.44:0/3064487979 conn(0x7f32b009aa20 msgr2=0x7f32b009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.021+0000 7f32b6329640 1 -- 172.21.15.44:0/3064487979 shutdown_connections 2024-04-02T13:20:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.021+0000 7f32b6329640 1 -- 172.21.15.44:0/3064487979 wait complete. 2024-04-02T13:20:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.022+0000 7f32b6329640 1 Processor -- start 2024-04-02T13:20:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.022+0000 7f32b6329640 1 -- start start 2024-04-02T13:20:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b6329640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32980039b0 0x7f32b00b2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b6329640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b015a5a0 0x7f32b0157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b6329640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32b0144300 0x7f32b0144780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b6329640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f32b00cead0 con 0x7f32b0144300 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b6329640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f32b00ce950 con 0x7f32980039b0 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b6329640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f32b006fc90 con 0x7f32b015a5a0 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b48c6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32b0144300 0x7f32b0144780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b48c6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32b0144300 0x7f32b0144780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59660/0 (socket says 172.21.15.44:59660) 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b48c6640 1 -- 172.21.15.44:0/2268451271 learned_addr learned my addr 172.21.15.44:0/2268451271 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32affff640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32980039b0 0x7f32b00b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32af7fe640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b015a5a0 0x7f32b0157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b48c6640 1 -- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b015a5a0 msgr2=0x7f32b0157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b48c6640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b015a5a0 0x7f32b0157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b48c6640 1 -- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32980039b0 msgr2=0x7f32b00b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b48c6640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32980039b0 0x7f32b00b2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b48c6640 1 -- 172.21.15.44:0/2268451271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b00b2f00 con 0x7f32b0144300 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32af7fe640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b015a5a0 0x7f32b0157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b48c6640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32b0144300 0x7f32b0144780 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f32a8007d10 tx=0x7f32a80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32acff9640 1 -- 172.21.15.44:0/2268451271 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8011de0 con 0x7f32b0144300 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32acff9640 1 -- 172.21.15.44:0/2268451271 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32a801b070 con 0x7f32b0144300 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32b6329640 1 -- 172.21.15.44:0/2268451271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32b00b31e0 con 0x7f32b0144300 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.023+0000 7f32acff9640 1 -- 172.21.15.44:0/2268451271 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8003990 con 0x7f32b0144300 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.024+0000 7f32b6329640 1 -- 172.21.15.44:0/2268451271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32b00b37a0 con 0x7f32b0144300 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.025+0000 7f32b6329640 1 -- 172.21.15.44:0/2268451271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32b0068c00 con 0x7f32b0144300 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.025+0000 7f32acff9640 1 -- 172.21.15.44:0/2268451271 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f32a8005d10 con 0x7f32b0144300 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.025+0000 7f32acff9640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f329c06c8f0 0x7f329c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.025+0000 7f32acff9640 1 -- 172.21.15.44:0/2268451271 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f32a808bfe0 con 0x7f32b0144300 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.025+0000 7f32affff640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f329c06c8f0 0x7f329c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.026+0000 7f32affff640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f329c06c8f0 0x7f329c06edb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f32a0034000 tx=0x7f32a0007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.030+0000 7f32acff9640 1 -- 172.21.15.44:0/2268451271 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f32a8025020 con 0x7f32b0144300 2024-04-02T13:20:21.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.223+0000 7f32b6329640 1 -- 172.21.15.44:0/2268451271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f32b015d680 con 0x7f32b0144300 2024-04-02T13:20:21.389 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:20:21.392 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:20:21.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.888+0000 7f32acff9640 1 -- 172.21.15.44:0/2268451271 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) ==== 157+0+0 (secure 0 0 0) 0x7f32a805a410 con 0x7f32b0144300 2024-04-02T13:20:21.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.891+0000 7f32b6329640 1 -- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f329c06c8f0 msgr2=0x7f329c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:21.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.891+0000 7f32b6329640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f329c06c8f0 0x7f329c06edb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f32a0034000 tx=0x7f32a0007000 comp rx=0 tx=0).stop 2024-04-02T13:20:21.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.891+0000 7f32b6329640 1 -- 172.21.15.44:0/2268451271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32b0144300 msgr2=0x7f32b0144780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:21.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.891+0000 7f32b6329640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32b0144300 0x7f32b0144780 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f32a8007d10 tx=0x7f32a80115a0 comp rx=0 tx=0).stop 2024-04-02T13:20:21.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.891+0000 7f32b6329640 1 -- 172.21.15.44:0/2268451271 shutdown_connections 2024-04-02T13:20:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.891+0000 7f32b6329640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f329c06c8f0 0x7f329c06edb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.891+0000 7f32b6329640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32b0144300 0x7f32b0144780 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.891+0000 7f32b6329640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32b015a5a0 0x7f32b0157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.891+0000 7f32b6329640 1 --2- 172.21.15.44:0/2268451271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32980039b0 0x7f32b00b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.891+0000 7f32b6329640 1 -- 172.21.15.44:0/2268451271 >> 172.21.15.44:0/2268451271 conn(0x7f32b009aa20 msgr2=0x7f32b00b4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.891+0000 7f32b6329640 1 -- 172.21.15.44:0/2268451271 shutdown_connections 2024-04-02T13:20:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:21.891+0000 7f32b6329640 1 -- 172.21.15.44:0/2268451271 wait complete. 2024-04-02T13:20:21.905 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T13:20:22.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.015+0000 7f59e1406640 1 Processor -- start 2024-04-02T13:20:22.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.015+0000 7f59e1406640 1 -- start start 2024-04-02T13:20:22.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59e1406640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f59dc0d0aa0 0x7f59dc0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:22.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59e1406640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f59dc0aec60 0x7f59dc0af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:22.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59e1406640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59dc15c5a0 con 0x7f59dc15c1c0 2024-04-02T13:20:22.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59e1406640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f59dc0af9d0 con 0x7f59dc0aec60 2024-04-02T13:20:22.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59e1406640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f59dc0afcb0 con 0x7f59dc0d0aa0 2024-04-02T13:20:22.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59daffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f59dc15c1c0 0x7f59dc0d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54588/0 (socket says 172.21.15.44:54588) 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59daffd640 1 -- 172.21.15.44:0/1803843359 learned_addr learned my addr 172.21.15.44:0/1803843359 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59db7fe640 1 --2- 172.21.15.44:0/1803843359 >> v2:172.21.15.44:3300/0 conn(0x7f59dc0aec60 0x7f59dc0af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59d9ffb640 1 -- 172.21.15.44:0/1803843359 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3551423939 0 0) 0x7f59dc15c5a0 con 0x7f59dc15c1c0 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59d9ffb640 1 -- 172.21.15.44:0/1803843359 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59c8003520 con 0x7f59dc15c1c0 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59da7fc640 1 --2- 172.21.15.44:0/1803843359 >> v2:172.21.15.69:3300/0 conn(0x7f59dc0d0aa0 0x7f59dc0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59db7fe640 1 -- 172.21.15.44:0/1803843359 >> v2:172.21.15.69:3300/0 conn(0x7f59dc0d0aa0 msgr2=0x7f59dc0cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59db7fe640 1 --2- 172.21.15.44:0/1803843359 >> v2:172.21.15.69:3300/0 conn(0x7f59dc0d0aa0 0x7f59dc0cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59db7fe640 1 -- 172.21.15.44:0/1803843359 >> v1:172.21.15.44:6789/0 conn(0x7f59dc15c1c0 legacy=0x7f59dc0d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59db7fe640 1 -- 172.21.15.44:0/1803843359 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59dc0aff90 con 0x7f59dc0aec60 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.016+0000 7f59da7fc640 1 --2- 172.21.15.44:0/1803843359 >> v2:172.21.15.69:3300/0 conn(0x7f59dc0d0aa0 0x7f59dc0cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.017+0000 7f59db7fe640 1 --2- 172.21.15.44:0/1803843359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59dc0aec60 0x7f59dc0af0f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0d1460 tx=0x7f59cc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416a433e4921739e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.017+0000 7f59d9ffb640 1 -- 172.21.15.44:0/1803843359 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc034070 con 0x7f59dc0aec60 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.017+0000 7f59d9ffb640 1 -- 172.21.15.44:0/1803843359 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59cc030610 con 0x7f59dc0aec60 2024-04-02T13:20:22.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.017+0000 7f59d9ffb640 1 -- 172.21.15.44:0/1803843359 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc0309e0 con 0x7f59dc0aec60 2024-04-02T13:20:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.018+0000 7f59e1406640 1 -- 172.21.15.44:0/1803843359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59dc0aec60 msgr2=0x7f59dc0af0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.018+0000 7f59e1406640 1 --2- 172.21.15.44:0/1803843359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59dc0aec60 0x7f59dc0af0f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0d1460 tx=0x7f59cc02b960 comp rx=0 tx=0).stop 2024-04-02T13:20:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.018+0000 7f59e1406640 1 -- 172.21.15.44:0/1803843359 shutdown_connections 2024-04-02T13:20:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.018+0000 7f59e1406640 1 --2- 172.21.15.44:0/1803843359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59dc0aec60 0x7f59dc0af0f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.018+0000 7f59e1406640 1 --2- 172.21.15.44:0/1803843359 >> v2:172.21.15.69:3300/0 conn(0x7f59dc0d0aa0 0x7f59dc0cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.018+0000 7f59e1406640 1 -- 172.21.15.44:0/1803843359 >> 172.21.15.44:0/1803843359 conn(0x7f59dc073850 msgr2=0x7f59dc09aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.018+0000 7f59e1406640 1 -- 172.21.15.44:0/1803843359 shutdown_connections 2024-04-02T13:20:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.018+0000 7f59e1406640 1 -- 172.21.15.44:0/1803843359 wait complete. 2024-04-02T13:20:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.019+0000 7f59e1406640 1 Processor -- start 2024-04-02T13:20:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.019+0000 7f59e1406640 1 -- start start 2024-04-02T13:20:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59e1406640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59dc0aec60 0x7f59dc0c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59e1406640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59dc0d0aa0 0x7f59dc0cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59e1406640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59dc15c1c0 0x7f59dc0c7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59e1406640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc0af630 con 0x7f59dc15c1c0 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59e1406640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc0b0550 con 0x7f59dc0d0aa0 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59e1406640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f59dc0af7b0 con 0x7f59dc0aec60 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59db7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59dc15c1c0 0x7f59dc0c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59db7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59dc15c1c0 0x7f59dc0c7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59670/0 (socket says 172.21.15.44:59670) 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59db7fe640 1 -- 172.21.15.44:0/2480144855 learned_addr learned my addr 172.21.15.44:0/2480144855 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59da7fc640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59dc0d0aa0 0x7f59dc0cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59daffd640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59dc0aec60 0x7f59dc0c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59db7fe640 1 -- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59dc0aec60 msgr2=0x7f59dc0c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59db7fe640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59dc0aec60 0x7f59dc0c9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59db7fe640 1 -- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59dc0d0aa0 msgr2=0x7f59dc0cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59db7fe640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59dc0d0aa0 0x7f59dc0cb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:22.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59db7fe640 1 -- 172.21.15.44:0/2480144855 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59dc0ca0b0 con 0x7f59dc15c1c0 2024-04-02T13:20:22.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59db7fe640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59dc15c1c0 0x7f59dc0c7790 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009890 tx=0x7f59cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:22.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59d37fe640 1 -- 172.21.15.44:0/2480144855 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc034040 con 0x7f59dc15c1c0 2024-04-02T13:20:22.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59d37fe640 1 -- 172.21.15.44:0/2480144855 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59cc03aa20 con 0x7f59dc15c1c0 2024-04-02T13:20:22.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59e1406640 1 -- 172.21.15.44:0/2480144855 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59dc0ca390 con 0x7f59dc15c1c0 2024-04-02T13:20:22.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59d37fe640 1 -- 172.21.15.44:0/2480144855 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc03adf0 con 0x7f59dc15c1c0 2024-04-02T13:20:22.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.020+0000 7f59e1406640 1 -- 172.21.15.44:0/2480144855 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59dc15d8d0 con 0x7f59dc15c1c0 2024-04-02T13:20:22.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.021+0000 7f59e1406640 1 -- 172.21.15.44:0/2480144855 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59dc068c00 con 0x7f59dc15c1c0 2024-04-02T13:20:22.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.022+0000 7f59d37fe640 1 -- 172.21.15.44:0/2480144855 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f59cc037070 con 0x7f59dc15c1c0 2024-04-02T13:20:22.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.022+0000 7f59d37fe640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59bc06c8f0 0x7f59bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:22.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.022+0000 7f59d37fe640 1 -- 172.21.15.44:0/2480144855 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 6007+0+0 (secure 0 0 0) 0x7f59cc0ada90 con 0x7f59dc15c1c0 2024-04-02T13:20:22.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.022+0000 7f59daffd640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59bc06c8f0 0x7f59bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:22.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.023+0000 7f59daffd640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59bc06c8f0 0x7f59bc06edb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f59d4005c00 tx=0x7f59d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:22.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.027+0000 7f59d37fe640 1 -- 172.21.15.44:0/2480144855 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f59cc03f070 con 0x7f59dc15c1c0 2024-04-02T13:20:22.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:22.209+0000 7f59e1406640 1 -- 172.21.15.44:0/2480144855 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f59dc15d6a0 con 0x7f59dc15c1c0 2024-04-02T13:20:23.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:23.899+0000 7f59d37fe640 1 -- 172.21.15.44:0/2480144855 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) ==== 228+0+0 (secure 0 0 0) 0x7f59cc07bd90 con 0x7f59dc15c1c0 2024-04-02T13:20:23.903 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data_ec' created 2024-04-02T13:20:23.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:23.903+0000 7f59e1406640 1 -- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59bc06c8f0 msgr2=0x7f59bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:23.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:23.903+0000 7f59e1406640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59bc06c8f0 0x7f59bc06edb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f59d4005c00 tx=0x7f59d4009040 comp rx=0 tx=0).stop 2024-04-02T13:20:23.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:23.903+0000 7f59e1406640 1 -- 172.21.15.44:0/2480144855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59dc15c1c0 msgr2=0x7f59dc0c7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:23.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:23.903+0000 7f59e1406640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59dc15c1c0 0x7f59dc0c7790 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009890 tx=0x7f59cc0046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:23.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:23.903+0000 7f59e1406640 1 -- 172.21.15.44:0/2480144855 shutdown_connections 2024-04-02T13:20:23.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:23.903+0000 7f59e1406640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59bc06c8f0 0x7f59bc06edb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:23.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:23.903+0000 7f59e1406640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59dc15c1c0 0x7f59dc0c7790 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:23.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:23.903+0000 7f59e1406640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59dc0d0aa0 0x7f59dc0cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:23.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:23.903+0000 7f59e1406640 1 --2- 172.21.15.44:0/2480144855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59dc0aec60 0x7f59dc0c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:23.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:23.903+0000 7f59e1406640 1 -- 172.21.15.44:0/2480144855 >> 172.21.15.44:0/2480144855 conn(0x7f59dc073850 msgr2=0x7f59dc0d0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:23.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:23.904+0000 7f59e1406640 1 -- 172.21.15.44:0/2480144855 shutdown_connections 2024-04-02T13:20:23.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:23.904+0000 7f59e1406640 1 -- 172.21.15.44:0/2480144855 wait complete. 2024-04-02T13:20:23.926 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-02T13:20:24.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.034+0000 7fc84f7d6640 1 Processor -- start 2024-04-02T13:20:24.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.035+0000 7fc84f7d6640 1 -- start start 2024-04-02T13:20:24.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.035+0000 7fc84f7d6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc8480d0a30 0x7fc84806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:24.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.035+0000 7fc84f7d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8480d0000 con 0x7fc8480d2040 2024-04-02T13:20:24.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.035+0000 7fc84f7d6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84806c1d0 con 0x7fc8480d0e10 2024-04-02T13:20:24.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.035+0000 7fc84f7d6640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc84806cb00 con 0x7fc8480d0a30 2024-04-02T13:20:24.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.036+0000 7fc84d572640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc8480d0a30 0x7fc84806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:24.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.036+0000 7fc84dd73640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc8480d2040 0x7fc8480d24f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54594/0 (socket says 172.21.15.44:54594) 2024-04-02T13:20:24.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.036+0000 7fc84dd73640 1 -- 172.21.15.44:0/707958815 learned_addr learned my addr 172.21.15.44:0/707958815 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:24.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.036+0000 7fc84d572640 1 -- 172.21.15.44:0/707958815 >> v1:172.21.15.69:6789/0 conn(0x7fc8480d0e10 legacy=0x7fc8480cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:24.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.036+0000 7fc84d572640 1 -- 172.21.15.44:0/707958815 >> v1:172.21.15.44:6789/0 conn(0x7fc8480d2040 legacy=0x7fc8480d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:24.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.036+0000 7fc84d572640 1 -- 172.21.15.44:0/707958815 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84806c4b0 con 0x7fc8480d0a30 2024-04-02T13:20:24.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.036+0000 7fc83ffff640 1 -- 172.21.15.44:0/707958815 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2916210384 0 0) 0x7fc8480d0000 con 0x7fc8480d2040 2024-04-02T13:20:24.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.037+0000 7fc84d572640 1 --2- 172.21.15.44:0/707958815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8480d0a30 0x7fc84806cf90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc8440097a0 tx=0x7fc84402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6f739cbe644879 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:24.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.037+0000 7fc83ffff640 1 -- 172.21.15.44:0/707958815 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8440315a0 con 0x7fc8480d0a30 2024-04-02T13:20:24.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.037+0000 7fc83ffff640 1 -- 172.21.15.44:0/707958815 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc844037070 con 0x7fc8480d0a30 2024-04-02T13:20:24.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.037+0000 7fc83ffff640 1 -- 172.21.15.44:0/707958815 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844031960 con 0x7fc8480d0a30 2024-04-02T13:20:24.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.037+0000 7fc84f7d6640 1 -- 172.21.15.44:0/707958815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8480d0a30 msgr2=0x7fc84806cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:24.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.037+0000 7fc84f7d6640 1 --2- 172.21.15.44:0/707958815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8480d0a30 0x7fc84806cf90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc8440097a0 tx=0x7fc84402b840 comp rx=0 tx=0).stop 2024-04-02T13:20:24.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.037+0000 7fc84f7d6640 1 -- 172.21.15.44:0/707958815 shutdown_connections 2024-04-02T13:20:24.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.037+0000 7fc84f7d6640 1 --2- 172.21.15.44:0/707958815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8480d0a30 0x7fc84806cf90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:24.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.037+0000 7fc84f7d6640 1 -- 172.21.15.44:0/707958815 >> 172.21.15.44:0/707958815 conn(0x7fc84809d470 msgr2=0x7fc8480d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:24.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.037+0000 7fc84f7d6640 1 -- 172.21.15.44:0/707958815 shutdown_connections 2024-04-02T13:20:24.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.037+0000 7fc84f7d6640 1 -- 172.21.15.44:0/707958815 wait complete. 2024-04-02T13:20:24.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.039+0000 7fc84f7d6640 1 Processor -- start 2024-04-02T13:20:24.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.039+0000 7fc84f7d6640 1 -- start start 2024-04-02T13:20:24.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.039+0000 7fc84f7d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8480d0a30 0x7fc8480cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:24.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.039+0000 7fc84f7d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8480d0e10 0x7fc8480b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:24.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.039+0000 7fc84f7d6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8480d2040 0x7fc8480c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:24.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.039+0000 7fc84f7d6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc8480d2c00 con 0x7fc8480d0a30 2024-04-02T13:20:24.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.039+0000 7fc84f7d6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc8480d1270 con 0x7fc8480d2040 2024-04-02T13:20:24.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.039+0000 7fc84f7d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc84815e6f0 con 0x7fc8480d0e10 2024-04-02T13:20:24.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.039+0000 7fc84d572640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8480d0a30 0x7fc8480cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.039+0000 7fc84d572640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8480d0a30 0x7fc8480cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59690/0 (socket says 172.21.15.44:59690) 2024-04-02T13:20:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.039+0000 7fc84d572640 1 -- 172.21.15.44:0/2142497793 learned_addr learned my addr 172.21.15.44:0/2142497793 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc84dd73640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8480d2040 0x7fc8480c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc84cd71640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8480d0e10 0x7fc8480b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc84d572640 1 -- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8480d0e10 msgr2=0x7fc8480b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc84d572640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8480d0e10 0x7fc8480b3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc84d572640 1 -- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8480d2040 msgr2=0x7fc8480c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc84d572640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8480d2040 0x7fc8480c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc84d572640 1 -- 172.21.15.44:0/2142497793 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8480cc160 con 0x7fc8480d0a30 2024-04-02T13:20:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc84dd73640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8480d2040 0x7fc8480c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc84cd71640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8480d0e10 0x7fc8480b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc84d572640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8480d0a30 0x7fc8480cbc20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc8440097a0 tx=0x7fc8440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc83dffb640 1 -- 172.21.15.44:0/2142497793 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844004230 con 0x7fc8480d0a30 2024-04-02T13:20:24.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc83dffb640 1 -- 172.21.15.44:0/2142497793 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8440043f0 con 0x7fc8480d0a30 2024-04-02T13:20:24.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc84f7d6640 1 -- 172.21.15.44:0/2142497793 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8480b6d30 con 0x7fc8480d0a30 2024-04-02T13:20:24.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc83dffb640 1 -- 172.21.15.44:0/2142497793 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844032880 con 0x7fc8480d0a30 2024-04-02T13:20:24.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.040+0000 7fc84f7d6640 1 -- 172.21.15.44:0/2142497793 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8480b72f0 con 0x7fc8480d0a30 2024-04-02T13:20:24.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.041+0000 7fc84f7d6640 1 -- 172.21.15.44:0/2142497793 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8480684f0 con 0x7fc8480d0a30 2024-04-02T13:20:24.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.042+0000 7fc83dffb640 1 -- 172.21.15.44:0/2142497793 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc844032a20 con 0x7fc8480d0a30 2024-04-02T13:20:24.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.042+0000 7fc83dffb640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc82c06c8f0 0x7fc82c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:24.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.042+0000 7fc83dffb640 1 -- 172.21.15.44:0/2142497793 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7fc84402f030 con 0x7fc8480d0a30 2024-04-02T13:20:24.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.042+0000 7fc84cd71640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc82c06c8f0 0x7fc82c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:24.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.043+0000 7fc84cd71640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc82c06c8f0 0x7fc82c06edb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc8480cd070 tx=0x7fc838008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:24.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.047+0000 7fc83dffb640 1 -- 172.21.15.44:0/2142497793 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc844041070 con 0x7fc8480d0a30 2024-04-02T13:20:24.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.229+0000 7fc84f7d6640 1 -- 172.21.15.44:0/2142497793 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fc8480d2680 con 0x7fc8480d0a30 2024-04-02T13:20:24.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.905+0000 7fc83dffb640 1 -- 172.21.15.44:0/2142497793 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) ==== 169+0+0 (secure 0 0 0) 0x7fc844044340 con 0x7fc8480d0a30 2024-04-02T13:20:24.907 INFO:teuthology.orchestra.run.smithi044.stderr:set pool 4 allow_ec_overwrites to true 2024-04-02T13:20:24.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.907+0000 7fc84f7d6640 1 -- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc82c06c8f0 msgr2=0x7fc82c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:24.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.907+0000 7fc84f7d6640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc82c06c8f0 0x7fc82c06edb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc8480cd070 tx=0x7fc838008040 comp rx=0 tx=0).stop 2024-04-02T13:20:24.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.907+0000 7fc84f7d6640 1 -- 172.21.15.44:0/2142497793 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8480d0a30 msgr2=0x7fc8480cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:24.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.907+0000 7fc84f7d6640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8480d0a30 0x7fc8480cbc20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc8440097a0 tx=0x7fc8440046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:24.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.908+0000 7fc84f7d6640 1 -- 172.21.15.44:0/2142497793 shutdown_connections 2024-04-02T13:20:24.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.908+0000 7fc84f7d6640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc82c06c8f0 0x7fc82c06edb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:24.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.908+0000 7fc84f7d6640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8480d2040 0x7fc8480c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:24.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.908+0000 7fc84f7d6640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8480d0e10 0x7fc8480b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:24.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.908+0000 7fc84f7d6640 1 --2- 172.21.15.44:0/2142497793 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8480d0a30 0x7fc8480cbc20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:24.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.908+0000 7fc84f7d6640 1 -- 172.21.15.44:0/2142497793 >> 172.21.15.44:0/2142497793 conn(0x7fc84809d470 msgr2=0x7fc848160990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:24.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.908+0000 7fc84f7d6640 1 -- 172.21.15.44:0/2142497793 shutdown_connections 2024-04-02T13:20:24.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:24.908+0000 7fc84f7d6640 1 -- 172.21.15.44:0/2142497793 wait complete. 2024-04-02T13:20:24.930 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-02T13:20:25.030 INFO:tasks.ceph.mds.c.smithi044.stderr:2024-04-02T13:20:25.028+0000 7f14fc09d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:20:25.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.044+0000 7f1394035640 1 Processor -- start 2024-04-02T13:20:25.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.045+0000 7f1394035640 1 -- start start 2024-04-02T13:20:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.045+0000 7f1394035640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f138c0d0c90 0x7f138c0d0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.045+0000 7f1394035640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f138c0d1070 0x7f138c15e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.045+0000 7f1394035640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138c0b3aa0 con 0x7f138c0d2990 2024-04-02T13:20:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.045+0000 7f1394035640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f138c0d0680 con 0x7f138c0d0c90 2024-04-02T13:20:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.045+0000 7f1394035640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f138c0d2f50 con 0x7f138c0d1070 2024-04-02T13:20:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.045+0000 7f1391dd1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f138c0d0c90 0x7f138c0d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.045+0000 7f1391dd1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f138c0d0c90 0x7f138c0d0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33688/0 (socket says 172.21.15.44:33688) 2024-04-02T13:20:25.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.045+0000 7f1391dd1640 1 -- 172.21.15.44:0/4294643653 learned_addr learned my addr 172.21.15.44:0/4294643653 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:25.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.046+0000 7f13915d0640 1 --2- 172.21.15.44:0/4294643653 >> v2:172.21.15.69:3300/0 conn(0x7f138c0d1070 0x7f138c15e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:25.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.046+0000 7f1391dd1640 1 -- 172.21.15.44:0/4294643653 >> v2:172.21.15.69:3300/0 conn(0x7f138c0d1070 msgr2=0x7f138c15e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:25.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.046+0000 7f1391dd1640 1 --2- 172.21.15.44:0/4294643653 >> v2:172.21.15.69:3300/0 conn(0x7f138c0d1070 0x7f138c15e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:25.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.046+0000 7f1391dd1640 1 -- 172.21.15.44:0/4294643653 >> v1:172.21.15.69:6789/0 conn(0x7f138c0d2990 legacy=0x7f138c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:25.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.046+0000 7f1391dd1640 1 -- 172.21.15.44:0/4294643653 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138c0b3d40 con 0x7f138c0d0c90 2024-04-02T13:20:25.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.046+0000 7f1391dd1640 1 --2- 172.21.15.44:0/4294643653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f138c0d0c90 0x7f138c0d0140 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f138c08c9b0 tx=0x7f138802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fac82d96ed58eb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.047+0000 7f1390dcf640 1 -- 172.21.15.44:0/4294643653 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388035020 con 0x7f138c0d0c90 2024-04-02T13:20:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.047+0000 7f1390dcf640 1 -- 172.21.15.44:0/4294643653 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1388031870 con 0x7f138c0d0c90 2024-04-02T13:20:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.047+0000 7f1390dcf640 1 -- 172.21.15.44:0/4294643653 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388031c40 con 0x7f138c0d0c90 2024-04-02T13:20:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.047+0000 7f1394035640 1 -- 172.21.15.44:0/4294643653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f138c0d0c90 msgr2=0x7f138c0d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.047+0000 7f1394035640 1 --2- 172.21.15.44:0/4294643653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f138c0d0c90 0x7f138c0d0140 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f138c08c9b0 tx=0x7f138802b910 comp rx=0 tx=0).stop 2024-04-02T13:20:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.047+0000 7f1394035640 1 -- 172.21.15.44:0/4294643653 shutdown_connections 2024-04-02T13:20:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.047+0000 7f1394035640 1 --2- 172.21.15.44:0/4294643653 >> v2:172.21.15.69:3300/0 conn(0x7f138c0d1070 0x7f138c15e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.047+0000 7f1394035640 1 --2- 172.21.15.44:0/4294643653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f138c0d0c90 0x7f138c0d0140 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.047+0000 7f1394035640 1 -- 172.21.15.44:0/4294643653 >> 172.21.15.44:0/4294643653 conn(0x7f138c09d3d0 msgr2=0x7f138c0d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.047+0000 7f1394035640 1 -- 172.21.15.44:0/4294643653 shutdown_connections 2024-04-02T13:20:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.047+0000 7f1394035640 1 -- 172.21.15.44:0/4294643653 wait complete. 2024-04-02T13:20:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.048+0000 7f1394035640 1 Processor -- start 2024-04-02T13:20:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.048+0000 7f1394035640 1 -- start start 2024-04-02T13:20:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.049+0000 7f1394035640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f138c0d0c90 0x7f138c1527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.049+0000 7f1394035640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f138c0d1070 0x7f138c0cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.049+0000 7f1394035640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f138c0d2990 0x7f138c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.049+0000 7f1394035640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f138c0d33b0 con 0x7f138c0d0c90 2024-04-02T13:20:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.049+0000 7f1394035640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f138c0d3230 con 0x7f138c0d1070 2024-04-02T13:20:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.049+0000 7f1394035640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f138c0b5550 con 0x7f138c0d2990 2024-04-02T13:20:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.049+0000 7f1391dd1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f138c0d0c90 0x7f138c1527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.049+0000 7f1391dd1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f138c0d0c90 0x7f138c1527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33692/0 (socket says 172.21.15.44:33692) 2024-04-02T13:20:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.049+0000 7f1391dd1640 1 -- 172.21.15.44:0/2347364781 learned_addr learned my addr 172.21.15.44:0/2347364781 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.049+0000 7f13915d0640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f138c0d1070 0x7f138c0cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.049+0000 7f13925d2640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f138c0d2990 0x7f138c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.049+0000 7f1391dd1640 1 -- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f138c0d2990 msgr2=0x7f138c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.049+0000 7f1391dd1640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f138c0d2990 0x7f138c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.050+0000 7f1391dd1640 1 -- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f138c0d1070 msgr2=0x7f138c0cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.050+0000 7f1391dd1640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f138c0d1070 0x7f138c0cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.050+0000 7f1391dd1640 1 -- 172.21.15.44:0/2347364781 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138c0cb210 con 0x7f138c0d0c90 2024-04-02T13:20:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.050+0000 7f1391dd1640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f138c0d0c90 0x7f138c1527b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f138802b950 tx=0x7f13880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.050+0000 7f13827fc640 1 -- 172.21.15.44:0/2347364781 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388040070 con 0x7f138c0d0c90 2024-04-02T13:20:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.050+0000 7f13827fc640 1 -- 172.21.15.44:0/2347364781 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1388030d00 con 0x7f138c0d0c90 2024-04-02T13:20:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.050+0000 7f1394035640 1 -- 172.21.15.44:0/2347364781 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f138c0cb4f0 con 0x7f138c0d0c90 2024-04-02T13:20:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.050+0000 7f13827fc640 1 -- 172.21.15.44:0/2347364781 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388035040 con 0x7f138c0d0c90 2024-04-02T13:20:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.050+0000 7f1394035640 1 -- 172.21.15.44:0/2347364781 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f138c0cbab0 con 0x7f138c0d0c90 2024-04-02T13:20:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.051+0000 7f1394035640 1 -- 172.21.15.44:0/2347364781 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f138c068c00 con 0x7f138c0d0c90 2024-04-02T13:20:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.051+0000 7f13827fc640 1 -- 172.21.15.44:0/2347364781 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1388038020 con 0x7f138c0d0c90 2024-04-02T13:20:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.052+0000 7f13827fc640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f137006c8f0 0x7f137006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:25.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.052+0000 7f13827fc640 1 -- 172.21.15.44:0/2347364781 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6500+0+0 (secure 0 0 0) 0x7f13880adc20 con 0x7f138c0d0c90 2024-04-02T13:20:25.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.052+0000 7f13915d0640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f137006c8f0 0x7f137006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:25.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.052+0000 7f13915d0640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f137006c8f0 0x7f137006edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f138c0b2fb0 tx=0x7f137c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:25.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.056+0000 7f13827fc640 1 -- 172.21.15.44:0/2347364781 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f138807b6d0 con 0x7f138c0d0c90 2024-04-02T13:20:25.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.280+0000 7f1394035640 1 -- 172.21.15.44:0/2347364781 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f138c0b6600 con 0x7f138c0d0c90 2024-04-02T13:20:25.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.913+0000 7f13827fc640 1 -- 172.21.15.44:0/2347364781 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7f13880ad070 con 0x7f138c0d0c90 2024-04-02T13:20:25.916 INFO:teuthology.orchestra.run.smithi044.stderr:added data pool 4 to fsmap 2024-04-02T13:20:25.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.916+0000 7f1394035640 1 -- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f137006c8f0 msgr2=0x7f137006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:25.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.916+0000 7f1394035640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f137006c8f0 0x7f137006edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f138c0b2fb0 tx=0x7f137c008040 comp rx=0 tx=0).stop 2024-04-02T13:20:25.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.916+0000 7f1394035640 1 -- 172.21.15.44:0/2347364781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f138c0d0c90 msgr2=0x7f138c1527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:25.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.916+0000 7f1394035640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f138c0d0c90 0x7f138c1527b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f138802b950 tx=0x7f13880046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:25.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.916+0000 7f1394035640 1 -- 172.21.15.44:0/2347364781 shutdown_connections 2024-04-02T13:20:25.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.916+0000 7f1394035640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f137006c8f0 0x7f137006edb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:25.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.916+0000 7f1394035640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f138c0d2990 0x7f138c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:25.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.916+0000 7f1394035640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f138c0d1070 0x7f138c0cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:25.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.916+0000 7f1394035640 1 --2- 172.21.15.44:0/2347364781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f138c0d0c90 0x7f138c1527b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:25.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.916+0000 7f1394035640 1 -- 172.21.15.44:0/2347364781 >> 172.21.15.44:0/2347364781 conn(0x7f138c09d3d0 msgr2=0x7f138c15d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:25.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.916+0000 7f1394035640 1 -- 172.21.15.44:0/2347364781 shutdown_connections 2024-04-02T13:20:25.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:25.916+0000 7f1394035640 1 -- 172.21.15.44:0/2347364781 wait complete. 2024-04-02T13:20:25.931 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:20:26.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.037+0000 7f3ce59e9640 1 Processor -- start 2024-04-02T13:20:26.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.037+0000 7f3ce59e9640 1 -- start start 2024-04-02T13:20:26.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.038+0000 7f3ce59e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ce00b34f0 0x7f3ce00d1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.038+0000 7f3ce59e9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce00af4a0 con 0x7f3ce00d2350 2024-04-02T13:20:26.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.038+0000 7f3ce59e9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce00af780 con 0x7f3ce015c2e0 2024-04-02T13:20:26.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.038+0000 7f3ce59e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce00d27d0 con 0x7f3ce00b34f0 2024-04-02T13:20:26.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.038+0000 7f3cdf7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f3ce00d2350 0x7f3ce00b0540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41530/0 (socket says 172.21.15.44:41530) 2024-04-02T13:20:26.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.038+0000 7f3cdf7fe640 1 -- 172.21.15.44:0/3492684332 learned_addr learned my addr 172.21.15.44:0/3492684332 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cde7fc640 1 --2- 172.21.15.44:0/3492684332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ce00b34f0 0x7f3ce00d1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4120495514 0 0) 0x7f3ce00af4a0 con 0x7f3ce00d2350 2024-04-02T13:20:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cc8003520 con 0x7f3ce00d2350 2024-04-02T13:20:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3405081097 0 0) 0x7f3cc8003520 con 0x7f3ce00d2350 2024-04-02T13:20:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ce00b34f0 msgr2=0x7f3ce00d1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 --2- 172.21.15.44:0/3492684332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ce00b34f0 0x7f3ce00d1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 >> v1:172.21.15.69:6789/0 conn(0x7f3ce015c2e0 legacy=0x7f3ce00aec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce00afa60 con 0x7f3ce00d2350 2024-04-02T13:20:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f3cd80019f0 con 0x7f3ce00d2350 2024-04-02T13:20:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 >> v1:172.21.15.44:6789/0 conn(0x7f3ce00d2350 legacy=0x7f3ce00b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 --2- 172.21.15.44:0/3492684332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ce00b34f0 0x7f3cc80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 --2- 172.21.15.44:0/3492684332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3cc8003f60 0x7f3cc8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 --2- 172.21.15.44:0/3492684332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc8004d40 0x7f3cc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc800ba70 con 0x7f3cc8003f60 2024-04-02T13:20:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc800bc20 con 0x7f3cc8004d40 2024-04-02T13:20:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3cc800be20 con 0x7f3ce00b34f0 2024-04-02T13:20:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cde7fc640 1 --2- 172.21.15.44:0/3492684332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3cc8003f60 0x7f3cc8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cdeffd640 1 --2- 172.21.15.44:0/3492684332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ce00b34f0 0x7f3cc80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cdf7fe640 1 --2- 172.21.15.44:0/3492684332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc8004d40 0x7f3cc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cde7fc640 1 -- 172.21.15.44:0/3492684332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ce00b34f0 msgr2=0x7f3cc80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cde7fc640 1 --2- 172.21.15.44:0/3492684332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ce00b34f0 0x7f3cc80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cde7fc640 1 -- 172.21.15.44:0/3492684332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc8004d40 msgr2=0x7f3cc80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cde7fc640 1 --2- 172.21.15.44:0/3492684332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc8004d40 0x7f3cc80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.039+0000 7f3cde7fc640 1 -- 172.21.15.44:0/3492684332 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc8003520 con 0x7f3cc8003f60 2024-04-02T13:20:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.040+0000 7f3cde7fc640 1 --2- 172.21.15.44:0/3492684332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3cc8003f60 0x7f3cc8004800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0009a90 tx=0x7f3cc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff053d5065e7c27e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.040+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0039070 con 0x7f3cc8003f60 2024-04-02T13:20:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.040+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd002c680 con 0x7f3cc8003f60 2024-04-02T13:20:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.040+0000 7f3cddffb640 1 -- 172.21.15.44:0/3492684332 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0009690 con 0x7f3cc8003f60 2024-04-02T13:20:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.041+0000 7f3ce59e9640 1 -- 172.21.15.44:0/3492684332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3cc8003f60 msgr2=0x7f3cc8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.041+0000 7f3ce59e9640 1 --2- 172.21.15.44:0/3492684332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3cc8003f60 0x7f3cc8004800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0009a90 tx=0x7f3cc8005830 comp rx=0 tx=0).stop 2024-04-02T13:20:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.041+0000 7f3cdf7fe640 1 -- 172.21.15.44:0/3492684332 reap_dead start 2024-04-02T13:20:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.041+0000 7f3ce59e9640 1 -- 172.21.15.44:0/3492684332 shutdown_connections 2024-04-02T13:20:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.041+0000 7f3ce59e9640 1 -- 172.21.15.44:0/3492684332 >> 172.21.15.44:0/3492684332 conn(0x7f3ce009aa20 msgr2=0x7f3ce009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.041+0000 7f3ce59e9640 1 -- 172.21.15.44:0/3492684332 shutdown_connections 2024-04-02T13:20:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.041+0000 7f3ce59e9640 1 -- 172.21.15.44:0/3492684332 wait complete. 2024-04-02T13:20:26.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.042+0000 7f3ce59e9640 1 Processor -- start 2024-04-02T13:20:26.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.042+0000 7f3ce59e9640 1 -- start start 2024-04-02T13:20:26.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3ce59e9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3cc8003f60 0x7f3ce015eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3ce59e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce01f79c0 0x7f3ce01f7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3ce59e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ce00ca060 0x7f3ce00ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3ce59e9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00d1e10 con 0x7f3cc8003f60 2024-04-02T13:20:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3ce59e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00b00b0 con 0x7f3ce01f79c0 2024-04-02T13:20:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3ce59e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce00d1f90 con 0x7f3ce00ca060 2024-04-02T13:20:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3cdeffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3cc8003f60 0x7f3ce015eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3cdeffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3cc8003f60 0x7f3ce015eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33704/0 (socket says 172.21.15.44:33704) 2024-04-02T13:20:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3cdeffd640 1 -- 172.21.15.44:0/1540104727 learned_addr learned my addr 172.21.15.44:0/1540104727 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3cdf7fe640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ce00ca060 0x7f3ce00ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3cde7fc640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce01f79c0 0x7f3ce01f7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3cdeffd640 1 -- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ce00ca060 msgr2=0x7f3ce00ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3cdeffd640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ce00ca060 0x7f3ce00ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3cdeffd640 1 -- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce01f79c0 msgr2=0x7f3ce01f7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.043+0000 7f3cdeffd640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce01f79c0 0x7f3ce01f7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.044+0000 7f3cdeffd640 1 -- 172.21.15.44:0/1540104727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce00cad70 con 0x7f3cc8003f60 2024-04-02T13:20:26.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.044+0000 7f3cdeffd640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3cc8003f60 0x7f3ce015eb80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400f850 tx=0x7f3cd400fc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:26.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.044+0000 7f3cc7fff640 1 -- 172.21.15.44:0/1540104727 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40050d0 con 0x7f3cc8003f60 2024-04-02T13:20:26.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.044+0000 7f3cc7fff640 1 -- 172.21.15.44:0/1540104727 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd4018070 con 0x7f3cc8003f60 2024-04-02T13:20:26.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.044+0000 7f3ce59e9640 1 -- 172.21.15.44:0/1540104727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce00cb050 con 0x7f3cc8003f60 2024-04-02T13:20:26.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.044+0000 7f3cc7fff640 1 -- 172.21.15.44:0/1540104727 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd400d6d0 con 0x7f3cc8003f60 2024-04-02T13:20:26.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.044+0000 7f3ce59e9640 1 -- 172.21.15.44:0/1540104727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce00cb610 con 0x7f3cc8003f60 2024-04-02T13:20:26.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.045+0000 7f3ce59e9640 1 -- 172.21.15.44:0/1540104727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce0068c00 con 0x7f3cc8003f60 2024-04-02T13:20:26.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.045+0000 7f3cc7fff640 1 -- 172.21.15.44:0/1540104727 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3cd400d870 con 0x7f3cc8003f60 2024-04-02T13:20:26.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.046+0000 7f3cc7fff640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3ccc06c8f0 0x7f3ccc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.046+0000 7f3cc7fff640 1 -- 172.21.15.44:0/1540104727 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3cd408bb10 con 0x7f3cc8003f60 2024-04-02T13:20:26.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.046+0000 7f3cde7fc640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3ccc06c8f0 0x7f3ccc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.047+0000 7f3cde7fc640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3ccc06c8f0 0x7f3ccc06edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0036000 tx=0x7f3cd0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:26.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.054+0000 7f3cc7fff640 1 -- 172.21.15.44:0/1540104727 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3cd4059c00 con 0x7f3cc8003f60 2024-04-02T13:20:26.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.280+0000 7f3ce59e9640 1 -- 172.21.15.44:0/1540104727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ce015f730 con 0x7f3cc8003f60 2024-04-02T13:20:26.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.281+0000 7f3cc7fff640 1 -- 172.21.15.44:0/1540104727 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5177 (secure 0 0 0) 0x7f3cd4059500 con 0x7f3cc8003f60 2024-04-02T13:20:26.283 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:26.283 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4040417710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4040417710},{"type":"v1","addr":"172.21.15.69:6836","nonce":4040417710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2772068060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2772068060},{"type":"v1","addr":"172.21.15.44:6833","nonce":2772068060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1985405150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1985405150},{"type":"v1","addr":"172.21.15.69:6837","nonce":1985405150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:20:20.247358+0000","modified":"2024-04-02T13:20:25.282737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/2282156931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2282156931},{"type":"v1","addr":"172.21.15.44:6835","nonce":2282156931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-04-02T13:20:26.283 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 7 2024-04-02T13:20:26.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.283+0000 7f3ce59e9640 1 -- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3ccc06c8f0 msgr2=0x7f3ccc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.283+0000 7f3ce59e9640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3ccc06c8f0 0x7f3ccc06edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0036000 tx=0x7f3cd0030000 comp rx=0 tx=0).stop 2024-04-02T13:20:26.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.283+0000 7f3ce59e9640 1 -- 172.21.15.44:0/1540104727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3cc8003f60 msgr2=0x7f3ce015eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.283+0000 7f3ce59e9640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3cc8003f60 0x7f3ce015eb80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400f850 tx=0x7f3cd400fc50 comp rx=0 tx=0).stop 2024-04-02T13:20:26.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.283+0000 7f3ce59e9640 1 -- 172.21.15.44:0/1540104727 shutdown_connections 2024-04-02T13:20:26.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.283+0000 7f3ce59e9640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3ccc06c8f0 0x7f3ccc06edb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.283+0000 7f3ce59e9640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ce00ca060 0x7f3ce00ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.283+0000 7f3ce59e9640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce01f79c0 0x7f3ce01f7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.283+0000 7f3ce59e9640 1 --2- 172.21.15.44:0/1540104727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3cc8003f60 0x7f3ce015eb80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.283+0000 7f3ce59e9640 1 -- 172.21.15.44:0/1540104727 >> 172.21.15.44:0/1540104727 conn(0x7f3ce009aa20 msgr2=0x7f3ce00d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:26.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.283+0000 7f3ce59e9640 1 -- 172.21.15.44:0/1540104727 shutdown_connections 2024-04-02T13:20:26.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.283+0000 7f3ce59e9640 1 -- 172.21.15.44:0/1540104727 wait complete. 2024-04-02T13:20:26.298 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:26.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.399+0000 7f48b20e3640 1 Processor -- start 2024-04-02T13:20:26.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.399+0000 7f48b20e3640 1 -- start start 2024-04-02T13:20:26.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48b20e3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48ac15e4d0 0x7f48ac0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48b20e3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48ac0d1230 0x7f48ac0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48b20e3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48ac15e8b0 con 0x7f48ac0cfd20 2024-04-02T13:20:26.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48b20e3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f48ac0b3a90 con 0x7f48ac0d1230 2024-04-02T13:20:26.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48b20e3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f48ac0b4610 con 0x7f48ac15e4d0 2024-04-02T13:20:26.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48ab7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48ac15e4d0 0x7f48ac0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48abfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48ac0d1230 0x7f48ac0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48ab7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48ac15e4d0 0x7f48ac0cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:60978/0 (socket says 172.21.15.44:60978) 2024-04-02T13:20:26.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48abfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48ac0d1230 0x7f48ac0b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:56798/0 (socket says 172.21.15.44:56798) 2024-04-02T13:20:26.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48ab7fe640 1 -- 172.21.15.44:0/1761124575 learned_addr learned my addr 172.21.15.44:0/1761124575 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:26.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48abfff640 1 -- 172.21.15.44:0/1761124575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48ac15e4d0 msgr2=0x7f48ac0cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:26.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48abfff640 1 --2- 172.21.15.44:0/1761124575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48ac15e4d0 0x7f48ac0cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48abfff640 1 -- 172.21.15.44:0/1761124575 >> v1:172.21.15.69:6789/0 conn(0x7f48ac0cfd20 legacy=0x7f48ac0d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48abfff640 1 -- 172.21.15.44:0/1761124575 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48ac0d2fb0 con 0x7f48ac0d1230 2024-04-02T13:20:26.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.400+0000 7f48ab7fe640 1 --2- 172.21.15.44:0/1761124575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48ac15e4d0 0x7f48ac0cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:26.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.401+0000 7f48abfff640 1 --2- 172.21.15.44:0/1761124575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48ac0d1230 0x7f48ac0b4aa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009d90 tx=0x7f48a402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2847870e3ec9e0a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:26.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.401+0000 7f48aa7fc640 1 -- 172.21.15.44:0/1761124575 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a4039070 con 0x7f48ac0d1230 2024-04-02T13:20:26.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.401+0000 7f48aa7fc640 1 -- 172.21.15.44:0/1761124575 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48a402cc50 con 0x7f48ac0d1230 2024-04-02T13:20:26.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.401+0000 7f48aa7fc640 1 -- 172.21.15.44:0/1761124575 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a4034440 con 0x7f48ac0d1230 2024-04-02T13:20:26.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.402+0000 7f48b20e3640 1 -- 172.21.15.44:0/1761124575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48ac0d1230 msgr2=0x7f48ac0b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.402+0000 7f48b20e3640 1 --2- 172.21.15.44:0/1761124575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48ac0d1230 0x7f48ac0b4aa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009d90 tx=0x7f48a402bea0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.402+0000 7f48b20e3640 1 -- 172.21.15.44:0/1761124575 shutdown_connections 2024-04-02T13:20:26.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.402+0000 7f48b20e3640 1 --2- 172.21.15.44:0/1761124575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48ac0d1230 0x7f48ac0b4aa0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.402+0000 7f48b20e3640 1 --2- 172.21.15.44:0/1761124575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48ac15e4d0 0x7f48ac0cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.402+0000 7f48b20e3640 1 -- 172.21.15.44:0/1761124575 >> 172.21.15.44:0/1761124575 conn(0x7f48ac09d410 msgr2=0x7f48ac0d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:26.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.402+0000 7f48b20e3640 1 -- 172.21.15.44:0/1761124575 shutdown_connections 2024-04-02T13:20:26.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.402+0000 7f48b20e3640 1 -- 172.21.15.44:0/1761124575 wait complete. 2024-04-02T13:20:26.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.403+0000 7f48b20e3640 1 Processor -- start 2024-04-02T13:20:26.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.403+0000 7f48b20e3640 1 -- start start 2024-04-02T13:20:26.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.404+0000 7f48b20e3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48ac0cfd20 0x7f48ac1535c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.404+0000 7f48b20e3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48ac0d1230 0x7f48ac15eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.404+0000 7f48b20e3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48ac15e4d0 0x7f48ac0cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.404+0000 7f48b20e3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac0b42a0 con 0x7f48ac0cfd20 2024-04-02T13:20:26.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.404+0000 7f48b20e3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac0b3fc0 con 0x7f48ac0d1230 2024-04-02T13:20:26.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.404+0000 7f48b20e3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f48ac0b0d00 con 0x7f48ac15e4d0 2024-04-02T13:20:26.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.404+0000 7f48ab7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48ac0cfd20 0x7f48ac1535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.404+0000 7f48ab7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48ac0cfd20 0x7f48ac1535c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33710/0 (socket says 172.21.15.44:33710) 2024-04-02T13:20:26.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.404+0000 7f48ab7fe640 1 -- 172.21.15.44:0/730126085 learned_addr learned my addr 172.21.15.44:0/730126085 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:26.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.404+0000 7f48abfff640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48ac15e4d0 0x7f48ac0cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.404+0000 7f48aaffd640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48ac0d1230 0x7f48ac15eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.405+0000 7f48ab7fe640 1 -- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48ac15e4d0 msgr2=0x7f48ac0cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.405+0000 7f48ab7fe640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48ac15e4d0 0x7f48ac0cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.405+0000 7f48ab7fe640 1 -- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48ac0d1230 msgr2=0x7f48ac15eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.405+0000 7f48ab7fe640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48ac0d1230 0x7f48ac15eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.405+0000 7f48ab7fe640 1 -- 172.21.15.44:0/730126085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48ac15f480 con 0x7f48ac0cfd20 2024-04-02T13:20:26.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.405+0000 7f48ab7fe640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48ac0cfd20 0x7f48ac1535c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f48a000e9b0 tx=0x7f48a000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:26.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.405+0000 7f4893fff640 1 -- 172.21.15.44:0/730126085 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a0017070 con 0x7f48ac0cfd20 2024-04-02T13:20:26.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.405+0000 7f4893fff640 1 -- 172.21.15.44:0/730126085 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48a000c9a0 con 0x7f48ac0cfd20 2024-04-02T13:20:26.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.405+0000 7f48b20e3640 1 -- 172.21.15.44:0/730126085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48ac15f760 con 0x7f48ac0cfd20 2024-04-02T13:20:26.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.405+0000 7f4893fff640 1 -- 172.21.15.44:0/730126085 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a000cd50 con 0x7f48ac0cfd20 2024-04-02T13:20:26.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.405+0000 7f48b20e3640 1 -- 172.21.15.44:0/730126085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48ac15fd20 con 0x7f48ac0cfd20 2024-04-02T13:20:26.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.406+0000 7f48b20e3640 1 -- 172.21.15.44:0/730126085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48ac0d3640 con 0x7f48ac0cfd20 2024-04-02T13:20:26.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.407+0000 7f4893fff640 1 -- 172.21.15.44:0/730126085 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f48a001a070 con 0x7f48ac0cfd20 2024-04-02T13:20:26.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.407+0000 7f4893fff640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f489806c8f0 0x7f489806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.407+0000 7f4893fff640 1 -- 172.21.15.44:0/730126085 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f48a008bc70 con 0x7f48ac0cfd20 2024-04-02T13:20:26.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.407+0000 7f48aaffd640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f489806c8f0 0x7f489806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.408+0000 7f48aaffd640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f489806c8f0 0x7f489806edb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0a1f70 tx=0x7f489c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:26.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.411+0000 7f4893fff640 1 -- 172.21.15.44:0/730126085 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f48ac0d3640 con 0x7f48ac0cfd20 2024-04-02T13:20:26.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.599+0000 7f48b20e3640 1 -- 172.21.15.44:0/730126085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48ac068c00 con 0x7f48ac0cfd20 2024-04-02T13:20:26.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.601+0000 7f4893fff640 1 -- 172.21.15.44:0/730126085 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f48a0059710 con 0x7f48ac0cfd20 2024-04-02T13:20:26.603 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:26.604 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:26.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.603+0000 7f48b20e3640 1 -- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f489806c8f0 msgr2=0x7f489806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.603+0000 7f48b20e3640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f489806c8f0 0x7f489806edb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0a1f70 tx=0x7f489c009040 comp rx=0 tx=0).stop 2024-04-02T13:20:26.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.603+0000 7f48b20e3640 1 -- 172.21.15.44:0/730126085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48ac0cfd20 msgr2=0x7f48ac1535c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.603+0000 7f48b20e3640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48ac0cfd20 0x7f48ac1535c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f48a000e9b0 tx=0x7f48a000ee70 comp rx=0 tx=0).stop 2024-04-02T13:20:26.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.603+0000 7f48b20e3640 1 -- 172.21.15.44:0/730126085 shutdown_connections 2024-04-02T13:20:26.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.603+0000 7f48b20e3640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f489806c8f0 0x7f489806edb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.603+0000 7f48b20e3640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48ac15e4d0 0x7f48ac0cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.603+0000 7f48b20e3640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48ac0d1230 0x7f48ac15eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.603+0000 7f48b20e3640 1 --2- 172.21.15.44:0/730126085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48ac0cfd20 0x7f48ac1535c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.603+0000 7f48b20e3640 1 -- 172.21.15.44:0/730126085 >> 172.21.15.44:0/730126085 conn(0x7f48ac09d410 msgr2=0x7f48ac15d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:26.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.603+0000 7f48b20e3640 1 -- 172.21.15.44:0/730126085 shutdown_connections 2024-04-02T13:20:26.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.603+0000 7f48b20e3640 1 -- 172.21.15.44:0/730126085 wait complete. 2024-04-02T13:20:26.617 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.716+0000 7f0a51ce4640 1 Processor -- start 2024-04-02T13:20:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.716+0000 7f0a51ce4640 1 -- start start 2024-04-02T13:20:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.717+0000 7f0a51ce4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a4c0b3510 0x7f0a4c0d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.717+0000 7f0a51ce4640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0a4c0d08d0 0x7f0a4c0b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.717+0000 7f0a51ce4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a4c0af420 con 0x7f0a4c15c250 2024-04-02T13:20:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.717+0000 7f0a51ce4640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0a4c15c630 con 0x7f0a4c0d08d0 2024-04-02T13:20:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.717+0000 7f0a51ce4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0a4c0af700 con 0x7f0a4c0b3510 2024-04-02T13:20:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.717+0000 7f0a4bfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0a4c0d08d0 0x7f0a4c0b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.717+0000 7f0a4bfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0a4c0d08d0 0x7f0a4c0b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33716/0 (socket says 172.21.15.44:33716) 2024-04-02T13:20:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.717+0000 7f0a4bfff640 1 -- 172.21.15.44:0/3700650584 learned_addr learned my addr 172.21.15.44:0/3700650584 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.717+0000 7f0a4affd640 1 --2- 172.21.15.44:0/3700650584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a4c0b3510 0x7f0a4c0d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.718+0000 7f0a4bfff640 1 -- 172.21.15.44:0/3700650584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a4c0b3510 msgr2=0x7f0a4c0d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:26.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.718+0000 7f0a4bfff640 1 --2- 172.21.15.44:0/3700650584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a4c0b3510 0x7f0a4c0d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.718+0000 7f0a4bfff640 1 -- 172.21.15.44:0/3700650584 >> v1:172.21.15.69:6789/0 conn(0x7f0a4c15c250 legacy=0x7f0a4c0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.718+0000 7f0a4bfff640 1 -- 172.21.15.44:0/3700650584 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4c0af9e0 con 0x7f0a4c0d08d0 2024-04-02T13:20:26.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.719+0000 7f0a4bfff640 1 --2- 172.21.15.44:0/3700650584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a4c0d08d0 0x7f0a4c0b04c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009e50 tx=0x7f0a4402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a8742a1e25bcf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:26.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.719+0000 7f0a4a7fc640 1 -- 172.21.15.44:0/3700650584 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4402c9e0 con 0x7f0a4c0d08d0 2024-04-02T13:20:26.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.719+0000 7f0a4a7fc640 1 -- 172.21.15.44:0/3700650584 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a4402cb80 con 0x7f0a4c0d08d0 2024-04-02T13:20:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.719+0000 7f0a4a7fc640 1 -- 172.21.15.44:0/3700650584 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a440343f0 con 0x7f0a4c0d08d0 2024-04-02T13:20:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.719+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3700650584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a4c0d08d0 msgr2=0x7f0a4c0b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.719+0000 7f0a51ce4640 1 --2- 172.21.15.44:0/3700650584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a4c0d08d0 0x7f0a4c0b04c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009e50 tx=0x7f0a4402be40 comp rx=0 tx=0).stop 2024-04-02T13:20:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.719+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3700650584 shutdown_connections 2024-04-02T13:20:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.719+0000 7f0a51ce4640 1 --2- 172.21.15.44:0/3700650584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a4c0d08d0 0x7f0a4c0b04c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.719+0000 7f0a51ce4640 1 --2- 172.21.15.44:0/3700650584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a4c0b3510 0x7f0a4c0d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.719+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3700650584 >> 172.21.15.44:0/3700650584 conn(0x7f0a4c073850 msgr2=0x7f0a4c09aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.719+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3700650584 shutdown_connections 2024-04-02T13:20:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.719+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3700650584 wait complete. 2024-04-02T13:20:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.721+0000 7f0a51ce4640 1 Processor -- start 2024-04-02T13:20:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.721+0000 7f0a51ce4640 1 -- start start 2024-04-02T13:20:26.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.722+0000 7f0a51ce4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a4c0b3510 0x7f0a4c153710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a51ce4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a4c0d08d0 0x7f0a4c06e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a51ce4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a4c15c250 0x7f0a4c15e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a51ce4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c0b0b80 con 0x7f0a4c0b3510 2024-04-02T13:20:26.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a51ce4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c0b0a00 con 0x7f0a4c0d08d0 2024-04-02T13:20:26.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a51ce4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0a4c0b0d00 con 0x7f0a4c15c250 2024-04-02T13:20:26.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a4b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a4c0b3510 0x7f0a4c153710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a4b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a4c0b3510 0x7f0a4c153710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33722/0 (socket says 172.21.15.44:33722) 2024-04-02T13:20:26.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a4b7fe640 1 -- 172.21.15.44:0/3790035310 learned_addr learned my addr 172.21.15.44:0/3790035310 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:26.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a4affd640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a4c0d08d0 0x7f0a4c06e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a4bfff640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a4c15c250 0x7f0a4c15e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a4b7fe640 1 -- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a4c15c250 msgr2=0x7f0a4c15e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a4b7fe640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a4c15c250 0x7f0a4c15e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a4b7fe640 1 -- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a4c0d08d0 msgr2=0x7f0a4c06e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a4b7fe640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a4c0d08d0 0x7f0a4c06e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a4b7fe640 1 -- 172.21.15.44:0/3790035310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4c06e5e0 con 0x7f0a4c0b3510 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a4b7fe640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a4c0b3510 0x7f0a4c153710 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00c3c0 tx=0x7f0a3c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a33fff640 1 -- 172.21.15.44:0/3790035310 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c011790 con 0x7f0a4c0b3510 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a33fff640 1 -- 172.21.15.44:0/3790035310 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a3c017070 con 0x7f0a4c0b3510 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a33fff640 1 -- 172.21.15.44:0/3790035310 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c011b70 con 0x7f0a4c0b3510 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3790035310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a4c06c950 con 0x7f0a4c0b3510 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.723+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3790035310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a4c06cf10 con 0x7f0a4c0b3510 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.724+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3790035310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a4c0d06c0 con 0x7f0a4c0b3510 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.725+0000 7f0a33fff640 1 -- 172.21.15.44:0/3790035310 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0a3c011d10 con 0x7f0a4c0b3510 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.725+0000 7f0a33fff640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a3806c8f0 0x7f0a3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.725+0000 7f0a33fff640 1 -- 172.21.15.44:0/3790035310 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f0a3c08bbf0 con 0x7f0a4c0b3510 2024-04-02T13:20:26.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.726+0000 7f0a4affd640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a3806c8f0 0x7f0a3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:26.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.726+0000 7f0a4affd640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a3806c8f0 0x7f0a3806edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c09ff40 tx=0x7f0a40008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:26.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.730+0000 7f0a33fff640 1 -- 172.21.15.44:0/3790035310 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0a4c0d06c0 con 0x7f0a4c0b3510 2024-04-02T13:20:26.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.916+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3790035310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a4c068c00 con 0x7f0a4c0b3510 2024-04-02T13:20:26.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.918+0000 7f0a33fff640 1 -- 172.21.15.44:0/3790035310 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f0a3c059d70 con 0x7f0a4c0b3510 2024-04-02T13:20:26.921 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:26.921 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:26.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.920+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a3806c8f0 msgr2=0x7f0a3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.920+0000 7f0a51ce4640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a3806c8f0 0x7f0a3806edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c09ff40 tx=0x7f0a40008040 comp rx=0 tx=0).stop 2024-04-02T13:20:26.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.920+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3790035310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a4c0b3510 msgr2=0x7f0a4c153710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:26.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.920+0000 7f0a51ce4640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a4c0b3510 0x7f0a4c153710 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00c3c0 tx=0x7f0a3c00c880 comp rx=0 tx=0).stop 2024-04-02T13:20:26.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.920+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3790035310 shutdown_connections 2024-04-02T13:20:26.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.920+0000 7f0a51ce4640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a3806c8f0 0x7f0a3806edb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.920+0000 7f0a51ce4640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a4c15c250 0x7f0a4c15e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.920+0000 7f0a51ce4640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a4c0d08d0 0x7f0a4c06e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.920+0000 7f0a51ce4640 1 --2- 172.21.15.44:0/3790035310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a4c0b3510 0x7f0a4c153710 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:26.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.920+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3790035310 >> 172.21.15.44:0/3790035310 conn(0x7f0a4c073850 msgr2=0x7f0a4c06f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:26.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.920+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3790035310 shutdown_connections 2024-04-02T13:20:26.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:26.920+0000 7f0a51ce4640 1 -- 172.21.15.44:0/3790035310 wait complete. 2024-04-02T13:20:26.948 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:20:27.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.046+0000 7fb72774d640 1 Processor -- start 2024-04-02T13:20:27.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.047+0000 7fb72774d640 1 -- start start 2024-04-02T13:20:27.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.047+0000 7fb72774d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb72015c250 0x7fb7200aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:27.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.047+0000 7fb72774d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7200d0aa0 0x7fb7200b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:27.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.047+0000 7fb72774d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7200af420 con 0x7fb7200b3510 2024-04-02T13:20:27.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.047+0000 7fb72774d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb72015c630 con 0x7fb72015c250 2024-04-02T13:20:27.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.047+0000 7fb72774d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb7200af700 con 0x7fb7200d0aa0 2024-04-02T13:20:27.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.047+0000 7fb724ce8640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb7200b3510 0x7fb7200d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41536/0 (socket says 172.21.15.44:41536) 2024-04-02T13:20:27.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.047+0000 7fb724ce8640 1 -- 172.21.15.44:0/1656681437 learned_addr learned my addr 172.21.15.44:0/1656681437 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:27.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.048+0000 7fb7254e9640 1 --2- 172.21.15.44:0/1656681437 >> v2:172.21.15.69:3300/0 conn(0x7fb72015c250 0x7fb7200aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:27.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.048+0000 7fb725cea640 1 --2- 172.21.15.44:0/1656681437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7200d0aa0 0x7fb7200b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:27.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.048+0000 7fb717fff640 1 -- 172.21.15.44:0/1656681437 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1687948161 0 0) 0x7fb7200af420 con 0x7fb7200b3510 2024-04-02T13:20:27.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.048+0000 7fb717fff640 1 -- 172.21.15.44:0/1656681437 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb70c003520 con 0x7fb7200b3510 2024-04-02T13:20:27.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.048+0000 7fb7254e9640 1 -- 172.21.15.44:0/1656681437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7200d0aa0 msgr2=0x7fb7200b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:27.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.048+0000 7fb7254e9640 1 --2- 172.21.15.44:0/1656681437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7200d0aa0 0x7fb7200b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.048+0000 7fb7254e9640 1 -- 172.21.15.44:0/1656681437 >> v1:172.21.15.44:6789/0 conn(0x7fb7200b3510 legacy=0x7fb7200d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:27.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.048+0000 7fb7254e9640 1 -- 172.21.15.44:0/1656681437 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7200af9e0 con 0x7fb72015c250 2024-04-02T13:20:27.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.048+0000 7fb725cea640 1 --2- 172.21.15.44:0/1656681437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7200d0aa0 0x7fb7200b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:27.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.049+0000 7fb7254e9640 1 --2- 172.21.15.44:0/1656681437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb72015c250 0x7fb7200aebf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb71c0097a0 tx=0x7fb71c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51152a2944b2c644 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:27.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.049+0000 7fb717fff640 1 -- 172.21.15.44:0/1656681437 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb71c0315a0 con 0x7fb72015c250 2024-04-02T13:20:27.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.049+0000 7fb717fff640 1 -- 172.21.15.44:0/1656681437 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb71c037070 con 0x7fb72015c250 2024-04-02T13:20:27.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.049+0000 7fb717fff640 1 -- 172.21.15.44:0/1656681437 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb71c031960 con 0x7fb72015c250 2024-04-02T13:20:27.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.049+0000 7fb72774d640 1 -- 172.21.15.44:0/1656681437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb72015c250 msgr2=0x7fb7200aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:27.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.049+0000 7fb72774d640 1 --2- 172.21.15.44:0/1656681437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb72015c250 0x7fb7200aebf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb71c0097a0 tx=0x7fb71c02b840 comp rx=0 tx=0).stop 2024-04-02T13:20:27.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.050+0000 7fb72774d640 1 -- 172.21.15.44:0/1656681437 shutdown_connections 2024-04-02T13:20:27.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.050+0000 7fb72774d640 1 --2- 172.21.15.44:0/1656681437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7200d0aa0 0x7fb7200b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.050+0000 7fb72774d640 1 --2- 172.21.15.44:0/1656681437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb72015c250 0x7fb7200aebf0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.050+0000 7fb72774d640 1 -- 172.21.15.44:0/1656681437 >> 172.21.15.44:0/1656681437 conn(0x7fb720073850 msgr2=0x7fb72009aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:27.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.050+0000 7fb72774d640 1 -- 172.21.15.44:0/1656681437 shutdown_connections 2024-04-02T13:20:27.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.050+0000 7fb72774d640 1 -- 172.21.15.44:0/1656681437 wait complete. 2024-04-02T13:20:27.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.051+0000 7fb72774d640 1 Processor -- start 2024-04-02T13:20:27.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.051+0000 7fb72774d640 1 -- start start 2024-04-02T13:20:27.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.051+0000 7fb72774d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7200b3510 0x7fb720154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.051+0000 7fb72774d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7200d0aa0 0x7fb7200b4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.051+0000 7fb72774d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb72015c250 0x7fb7200b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.051+0000 7fb72774d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb7200b0a00 con 0x7fb72015c250 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.051+0000 7fb72774d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb7200af130 con 0x7fb7200b3510 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.051+0000 7fb72774d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb7200b0b80 con 0x7fb7200d0aa0 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.051+0000 7fb725cea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb72015c250 0x7fb7200b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb725cea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb72015c250 0x7fb7200b50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33730/0 (socket says 172.21.15.44:33730) 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb725cea640 1 -- 172.21.15.44:0/3628556709 learned_addr learned my addr 172.21.15.44:0/3628556709 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb724ce8640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7200d0aa0 0x7fb7200b4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb7254e9640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7200b3510 0x7fb720154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb725cea640 1 -- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7200d0aa0 msgr2=0x7fb7200b4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb725cea640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7200d0aa0 0x7fb7200b4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb725cea640 1 -- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7200b3510 msgr2=0x7fb720154fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:27.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb725cea640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7200b3510 0x7fb720154fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb725cea640 1 -- 172.21.15.44:0/3628556709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb720155500 con 0x7fb72015c250 2024-04-02T13:20:27.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb724ce8640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7200d0aa0 0x7fb7200b4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:27.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb7254e9640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7200b3510 0x7fb720154fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:27.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb725cea640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb72015c250 0x7fb7200b50a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb718009ea0 tx=0x7fb718009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:27.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb715ffb640 1 -- 172.21.15.44:0/3628556709 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718018030 con 0x7fb72015c250 2024-04-02T13:20:27.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb715ffb640 1 -- 172.21.15.44:0/3628556709 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb71800cde0 con 0x7fb72015c250 2024-04-02T13:20:27.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb72774d640 1 -- 172.21.15.44:0/3628556709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7200b6f40 con 0x7fb72015c250 2024-04-02T13:20:27.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb715ffb640 1 -- 172.21.15.44:0/3628556709 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718003bb0 con 0x7fb72015c250 2024-04-02T13:20:27.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.052+0000 7fb72774d640 1 -- 172.21.15.44:0/3628556709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7200b7500 con 0x7fb72015c250 2024-04-02T13:20:27.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.053+0000 7fb72774d640 1 -- 172.21.15.44:0/3628556709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb720068c00 con 0x7fb72015c250 2024-04-02T13:20:27.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.054+0000 7fb715ffb640 1 -- 172.21.15.44:0/3628556709 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb71801b070 con 0x7fb72015c250 2024-04-02T13:20:27.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.054+0000 7fb715ffb640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb70406c8f0 0x7fb70406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:27.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.054+0000 7fb715ffb640 1 -- 172.21.15.44:0/3628556709 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb71808c880 con 0x7fb72015c250 2024-04-02T13:20:27.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.054+0000 7fb7254e9640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb70406c8f0 0x7fb70406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:27.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.055+0000 7fb7254e9640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb70406c8f0 0x7fb70406edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb71c036000 tx=0x7fb71c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:27.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.058+0000 7fb715ffb640 1 -- 172.21.15.44:0/3628556709 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb718010040 con 0x7fb72015c250 2024-04-02T13:20:27.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.282+0000 7fb72774d640 1 -- 172.21.15.44:0/3628556709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7200d2a40 con 0x7fb72015c250 2024-04-02T13:20:27.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.283+0000 7fb715ffb640 1 -- 172.21.15.44:0/3628556709 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5177 (secure 0 0 0) 0x7fb71805a970 con 0x7fb72015c250 2024-04-02T13:20:27.285 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:27.285 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4040417710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4040417710},{"type":"v1","addr":"172.21.15.69:6836","nonce":4040417710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2772068060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2772068060},{"type":"v1","addr":"172.21.15.44:6833","nonce":2772068060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1985405150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1985405150},{"type":"v1","addr":"172.21.15.69:6837","nonce":1985405150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:20:20.247358+0000","modified":"2024-04-02T13:20:25.282737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/2282156931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2282156931},{"type":"v1","addr":"172.21.15.44:6835","nonce":2282156931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-04-02T13:20:27.285 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 7 2024-04-02T13:20:27.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.285+0000 7fb72774d640 1 -- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb70406c8f0 msgr2=0x7fb70406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:27.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.285+0000 7fb72774d640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb70406c8f0 0x7fb70406edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb71c036000 tx=0x7fb71c03d040 comp rx=0 tx=0).stop 2024-04-02T13:20:27.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.285+0000 7fb72774d640 1 -- 172.21.15.44:0/3628556709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb72015c250 msgr2=0x7fb7200b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:27.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.285+0000 7fb72774d640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb72015c250 0x7fb7200b50a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb718009ea0 tx=0x7fb718009f90 comp rx=0 tx=0).stop 2024-04-02T13:20:27.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.285+0000 7fb72774d640 1 -- 172.21.15.44:0/3628556709 shutdown_connections 2024-04-02T13:20:27.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.285+0000 7fb72774d640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb70406c8f0 0x7fb70406edb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.285+0000 7fb72774d640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb72015c250 0x7fb7200b50a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.285+0000 7fb72774d640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7200d0aa0 0x7fb7200b4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.285+0000 7fb72774d640 1 --2- 172.21.15.44:0/3628556709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7200b3510 0x7fb720154fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.285+0000 7fb72774d640 1 -- 172.21.15.44:0/3628556709 >> 172.21.15.44:0/3628556709 conn(0x7fb720073850 msgr2=0x7fb72006f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:27.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.285+0000 7fb72774d640 1 -- 172.21.15.44:0/3628556709 shutdown_connections 2024-04-02T13:20:27.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.285+0000 7fb72774d640 1 -- 172.21.15.44:0/3628556709 wait complete. 2024-04-02T13:20:27.298 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:27.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.397+0000 7ff8fa267640 1 Processor -- start 2024-04-02T13:20:27.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.397+0000 7ff8fa267640 1 -- start start 2024-04-02T13:20:27.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8fa267640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8f40d1150 0x7ff8f40b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:27.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8fa267640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8f40b70b0 con 0x7ff8f415e520 2024-04-02T13:20:27.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8fa267640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8f40b7390 con 0x7ff8f415e140 2024-04-02T13:20:27.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8fa267640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff8f40b9f50 con 0x7ff8f40d1150 2024-04-02T13:20:27.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8f37fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff8f415e520 0x7ff8f40d0a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41552/0 (socket says 172.21.15.44:41552) 2024-04-02T13:20:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8f37fe640 1 -- 172.21.15.44:0/4034475447 learned_addr learned my addr 172.21.15.44:0/4034475447 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8f8804640 1 --2- 172.21.15.44:0/4034475447 >> v2:172.21.15.69:3300/0 conn(0x7ff8f40d1150 0x7ff8f40b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8f2ffd640 1 -- 172.21.15.44:0/4034475447 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1676413091 0 0) 0x7ff8f40b70b0 con 0x7ff8f415e520 2024-04-02T13:20:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8f2ffd640 1 -- 172.21.15.44:0/4034475447 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8dc003520 con 0x7ff8f415e520 2024-04-02T13:20:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8f2ffd640 1 -- 172.21.15.44:0/4034475447 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3684036961 0 0) 0x7ff8f40b7390 con 0x7ff8f415e140 2024-04-02T13:20:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8f2ffd640 1 -- 172.21.15.44:0/4034475447 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8f40b70b0 con 0x7ff8f415e140 2024-04-02T13:20:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8f2ffd640 1 -- 172.21.15.44:0/4034475447 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3082013571 0 0) 0x7ff8dc003520 con 0x7ff8f415e520 2024-04-02T13:20:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8f8804640 1 -- 172.21.15.44:0/4034475447 >> v1:172.21.15.69:6789/0 conn(0x7ff8f415e140 legacy=0x7ff8f40d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8f8804640 1 -- 172.21.15.44:0/4034475447 >> v1:172.21.15.44:6789/0 conn(0x7ff8f415e520 legacy=0x7ff8f40d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.398+0000 7ff8f8804640 1 -- 172.21.15.44:0/4034475447 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f40b7670 con 0x7ff8f40d1150 2024-04-02T13:20:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.399+0000 7ff8f8804640 1 --2- 172.21.15.44:0/4034475447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8f40d1150 0x7ff8f40b4cc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec009d90 tx=0x7ff8ec02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cc4b878db4599ea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.399+0000 7ff8f2ffd640 1 -- 172.21.15.44:0/4034475447 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec039070 con 0x7ff8f40d1150 2024-04-02T13:20:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.399+0000 7ff8f2ffd640 1 -- 172.21.15.44:0/4034475447 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8ec02c8b0 con 0x7ff8f40d1150 2024-04-02T13:20:27.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.399+0000 7ff8f2ffd640 1 -- 172.21.15.44:0/4034475447 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4001670 con 0x7ff8f40d1150 2024-04-02T13:20:27.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.400+0000 7ff8fa267640 1 -- 172.21.15.44:0/4034475447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8f40d1150 msgr2=0x7ff8f40b4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:27.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.400+0000 7ff8fa267640 1 --2- 172.21.15.44:0/4034475447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8f40d1150 0x7ff8f40b4cc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec009d90 tx=0x7ff8ec02bd80 comp rx=0 tx=0).stop 2024-04-02T13:20:27.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.400+0000 7ff8fa267640 1 -- 172.21.15.44:0/4034475447 shutdown_connections 2024-04-02T13:20:27.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.400+0000 7ff8fa267640 1 --2- 172.21.15.44:0/4034475447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8f40d1150 0x7ff8f40b4cc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.400+0000 7ff8fa267640 1 -- 172.21.15.44:0/4034475447 >> 172.21.15.44:0/4034475447 conn(0x7ff8f409d390 msgr2=0x7ff8f40d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:27.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.400+0000 7ff8fa267640 1 -- 172.21.15.44:0/4034475447 shutdown_connections 2024-04-02T13:20:27.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.400+0000 7ff8fa267640 1 -- 172.21.15.44:0/4034475447 wait complete. 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.401+0000 7ff8fa267640 1 Processor -- start 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.401+0000 7ff8fa267640 1 -- start start 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8fa267640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8f40d1150 0x7ff8f414c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8fa267640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8f415e140 0x7ff8f415ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8fa267640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8f415e520 0x7ff8f40cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8fa267640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f40d2d40 con 0x7ff8f40d1150 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8fa267640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f40d4ee0 con 0x7ff8f415e140 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8fa267640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff8f40b8e60 con 0x7ff8f415e520 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8f40d1150 0x7ff8f414c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8f40d1150 0x7ff8f414c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33732/0 (socket says 172.21.15.44:33732) 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8f3fff640 1 -- 172.21.15.44:0/4141852893 learned_addr learned my addr 172.21.15.44:0/4141852893 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8f37fe640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8f415e140 0x7ff8f415ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8f8804640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8f415e520 0x7ff8f40cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8f3fff640 1 -- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8f415e520 msgr2=0x7ff8f40cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8f3fff640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8f415e520 0x7ff8f40cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8f3fff640 1 -- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8f415e140 msgr2=0x7ff8f415ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8f3fff640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8f415e140 0x7ff8f415ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8f3fff640 1 -- 172.21.15.44:0/4141852893 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f4161050 con 0x7ff8f40d1150 2024-04-02T13:20:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.402+0000 7ff8f3fff640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8f40d1150 0x7ff8f414c600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800c490 tx=0x7ff8e800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.403+0000 7ff8f0ff9640 1 -- 172.21.15.44:0/4141852893 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e80052d0 con 0x7ff8f40d1150 2024-04-02T13:20:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.403+0000 7ff8f0ff9640 1 -- 172.21.15.44:0/4141852893 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8e8005470 con 0x7ff8f40d1150 2024-04-02T13:20:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.403+0000 7ff8fa267640 1 -- 172.21.15.44:0/4141852893 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f4160a90 con 0x7ff8f40d1150 2024-04-02T13:20:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.403+0000 7ff8f0ff9640 1 -- 172.21.15.44:0/4141852893 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8012850 con 0x7ff8f40d1150 2024-04-02T13:20:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.403+0000 7ff8fa267640 1 -- 172.21.15.44:0/4141852893 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f40c83e0 con 0x7ff8f40d1150 2024-04-02T13:20:27.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.404+0000 7ff8fa267640 1 -- 172.21.15.44:0/4141852893 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8f40684f0 con 0x7ff8f40d1150 2024-04-02T13:20:27.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.404+0000 7ff8f0ff9640 1 -- 172.21.15.44:0/4141852893 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff8e80129f0 con 0x7ff8f40d1150 2024-04-02T13:20:27.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.405+0000 7ff8f0ff9640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8e006c8f0 0x7ff8e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:27.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.405+0000 7ff8f0ff9640 1 -- 172.21.15.44:0/4141852893 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff8e808bad0 con 0x7ff8f40d1150 2024-04-02T13:20:27.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.405+0000 7ff8f37fe640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8e006c8f0 0x7ff8e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:27.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.405+0000 7ff8f37fe640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8e006c8f0 0x7ff8e006edb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff8f41596a0 tx=0x7ff8e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:27.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.409+0000 7ff8f0ff9640 1 -- 172.21.15.44:0/4141852893 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff8f4161050 con 0x7ff8f40d1150 2024-04-02T13:20:27.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.594+0000 7ff8fa267640 1 -- 172.21.15.44:0/4141852893 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8f4067d70 con 0x7ff8f40d1150 2024-04-02T13:20:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.595+0000 7ff8f0ff9640 1 -- 172.21.15.44:0/4141852893 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7ff8e8059c50 con 0x7ff8f40d1150 2024-04-02T13:20:27.597 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:27.597 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:27.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.597+0000 7ff8fa267640 1 -- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8e006c8f0 msgr2=0x7ff8e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:27.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.597+0000 7ff8fa267640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8e006c8f0 0x7ff8e006edb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff8f41596a0 tx=0x7ff8e4009040 comp rx=0 tx=0).stop 2024-04-02T13:20:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.597+0000 7ff8fa267640 1 -- 172.21.15.44:0/4141852893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8f40d1150 msgr2=0x7ff8f414c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.597+0000 7ff8fa267640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8f40d1150 0x7ff8f414c600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800c490 tx=0x7ff8e800c950 comp rx=0 tx=0).stop 2024-04-02T13:20:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.597+0000 7ff8fa267640 1 -- 172.21.15.44:0/4141852893 shutdown_connections 2024-04-02T13:20:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.597+0000 7ff8fa267640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8e006c8f0 0x7ff8e006edb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.597+0000 7ff8fa267640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8f415e520 0x7ff8f40cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.597+0000 7ff8fa267640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8f415e140 0x7ff8f415ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.597+0000 7ff8fa267640 1 --2- 172.21.15.44:0/4141852893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8f40d1150 0x7ff8f414c600 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.597+0000 7ff8fa267640 1 -- 172.21.15.44:0/4141852893 >> 172.21.15.44:0/4141852893 conn(0x7ff8f409d390 msgr2=0x7ff8f406c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.597+0000 7ff8fa267640 1 -- 172.21.15.44:0/4141852893 shutdown_connections 2024-04-02T13:20:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:27.597+0000 7ff8fa267640 1 -- 172.21.15.44:0/4141852893 wait complete. 2024-04-02T13:20:28.013 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:20:20.247358+0000', 'modified': '2024-04-02T13:20:25.282737+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4266}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4266': {'gid': 4266, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.44:6835/2282156931', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6834', 'nonce': 2282156931}, {'type': 'v1', 'addr': '172.21.15.44:6835', 'nonce': 2282156931}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4266, 'qdb_cluster': [4266]} 2024-04-02T13:20:28.013 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:20:28.013 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-02T13:20:28.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.118+0000 7fc0732dc640 1 Processor -- start 2024-04-02T13:20:28.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.118+0000 7fc0732dc640 1 -- start start 2024-04-02T13:20:28.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.119+0000 7fc0732dc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc06c0d0a30 0x7fc06c06cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.119+0000 7fc0732dc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc06c0d0e10 0x7fc06c0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.119+0000 7fc0732dc640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc06c06c1d0 con 0x7fc06c0cfe30 2024-04-02T13:20:28.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.119+0000 7fc0732dc640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc06c06cb00 con 0x7fc06c0d0a30 2024-04-02T13:20:28.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.119+0000 7fc0732dc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc06c06c4b0 con 0x7fc06c0d0e10 2024-04-02T13:20:28.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.119+0000 7fc071078640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc06c0d0a30 0x7fc06c06cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.119+0000 7fc071078640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc06c0d0a30 0x7fc06c06cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33740/0 (socket says 172.21.15.44:33740) 2024-04-02T13:20:28.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.119+0000 7fc071078640 1 -- 172.21.15.44:0/3438031770 learned_addr learned my addr 172.21.15.44:0/3438031770 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:28.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.119+0000 7fc070877640 1 --2- 172.21.15.44:0/3438031770 >> v2:172.21.15.69:3300/0 conn(0x7fc06c0d0e10 0x7fc06c0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.120+0000 7fc071078640 1 -- 172.21.15.44:0/3438031770 >> v2:172.21.15.69:3300/0 conn(0x7fc06c0d0e10 msgr2=0x7fc06c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:28.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.120+0000 7fc071078640 1 --2- 172.21.15.44:0/3438031770 >> v2:172.21.15.69:3300/0 conn(0x7fc06c0d0e10 0x7fc06c0cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.120+0000 7fc071078640 1 -- 172.21.15.44:0/3438031770 >> v1:172.21.15.69:6789/0 conn(0x7fc06c0cfe30 legacy=0x7fc06c0d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.120+0000 7fc071078640 1 -- 172.21.15.44:0/3438031770 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06c06c790 con 0x7fc06c0d0a30 2024-04-02T13:20:28.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.120+0000 7fc071078640 1 --2- 172.21.15.44:0/3438031770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc06c0d0a30 0x7fc06c06cf90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0b6d30 tx=0x7fc06402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dbb759d45605aa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:28.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.120+0000 7fc06bfff640 1 -- 172.21.15.44:0/3438031770 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc064035020 con 0x7fc06c0d0a30 2024-04-02T13:20:28.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.121+0000 7fc06bfff640 1 -- 172.21.15.44:0/3438031770 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc064031870 con 0x7fc06c0d0a30 2024-04-02T13:20:28.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.121+0000 7fc06bfff640 1 -- 172.21.15.44:0/3438031770 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc064031c40 con 0x7fc06c0d0a30 2024-04-02T13:20:28.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.121+0000 7fc0732dc640 1 -- 172.21.15.44:0/3438031770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc06c0d0a30 msgr2=0x7fc06c06cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.121+0000 7fc0732dc640 1 --2- 172.21.15.44:0/3438031770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc06c0d0a30 0x7fc06c06cf90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0b6d30 tx=0x7fc06402b910 comp rx=0 tx=0).stop 2024-04-02T13:20:28.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.121+0000 7fc0732dc640 1 -- 172.21.15.44:0/3438031770 shutdown_connections 2024-04-02T13:20:28.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.121+0000 7fc0732dc640 1 --2- 172.21.15.44:0/3438031770 >> v2:172.21.15.69:3300/0 conn(0x7fc06c0d0e10 0x7fc06c0cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.121+0000 7fc0732dc640 1 --2- 172.21.15.44:0/3438031770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc06c0d0a30 0x7fc06c06cf90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.121+0000 7fc0732dc640 1 -- 172.21.15.44:0/3438031770 >> 172.21.15.44:0/3438031770 conn(0x7fc06c09d470 msgr2=0x7fc06c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:28.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.121+0000 7fc0732dc640 1 -- 172.21.15.44:0/3438031770 shutdown_connections 2024-04-02T13:20:28.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.121+0000 7fc0732dc640 1 -- 172.21.15.44:0/3438031770 wait complete. 2024-04-02T13:20:28.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.122+0000 7fc0732dc640 1 Processor -- start 2024-04-02T13:20:28.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.122+0000 7fc0732dc640 1 -- start start 2024-04-02T13:20:28.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc0732dc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc06c0cfe30 0x7fc06c156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc0732dc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc06c0d0a30 0x7fc06c0b6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc0732dc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc06c0d0e10 0x7fc06c0cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc0732dc640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c15e870 con 0x7fc06c0cfe30 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc0732dc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c15e6f0 con 0x7fc06c0d0e10 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc0732dc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc06c0d4da0 con 0x7fc06c0d0a30 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc071078640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc06c0cfe30 0x7fc06c156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc071078640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc06c0cfe30 0x7fc06c156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33746/0 (socket says 172.21.15.44:33746) 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc071078640 1 -- 172.21.15.44:0/4195174900 learned_addr learned my addr 172.21.15.44:0/4195174900 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc070877640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc06c0d0a30 0x7fc06c0b6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc071879640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc06c0d0e10 0x7fc06c0cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc071078640 1 -- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc06c0d0a30 msgr2=0x7fc06c0b6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc071078640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc06c0d0a30 0x7fc06c0b6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc071078640 1 -- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc06c0d0e10 msgr2=0x7fc06c0cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc071078640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc06c0d0e10 0x7fc06c0cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc071078640 1 -- 172.21.15.44:0/4195174900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06c0b7580 con 0x7fc06c0cfe30 2024-04-02T13:20:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.123+0000 7fc070877640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc06c0d0a30 0x7fc06c0b6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:28.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.124+0000 7fc071078640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc06c0cfe30 0x7fc06c156790 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc06402b950 tx=0x7fc0640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:28.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.124+0000 7fc069ffb640 1 -- 172.21.15.44:0/4195174900 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc064035040 con 0x7fc06c0cfe30 2024-04-02T13:20:28.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.124+0000 7fc069ffb640 1 -- 172.21.15.44:0/4195174900 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc064030d00 con 0x7fc06c0cfe30 2024-04-02T13:20:28.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.124+0000 7fc0732dc640 1 -- 172.21.15.44:0/4195174900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc06c0b7860 con 0x7fc06c0cfe30 2024-04-02T13:20:28.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.124+0000 7fc069ffb640 1 -- 172.21.15.44:0/4195174900 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0640456a0 con 0x7fc06c0cfe30 2024-04-02T13:20:28.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.124+0000 7fc0732dc640 1 -- 172.21.15.44:0/4195174900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc06c0b7270 con 0x7fc06c0cfe30 2024-04-02T13:20:28.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.125+0000 7fc069ffb640 1 -- 172.21.15.44:0/4195174900 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc064038020 con 0x7fc06c0cfe30 2024-04-02T13:20:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.126+0000 7fc069ffb640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc05806c8f0 0x7fc05806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.126+0000 7fc069ffb640 1 -- 172.21.15.44:0/4195174900 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc0640ae050 con 0x7fc06c0cfe30 2024-04-02T13:20:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.126+0000 7fc070877640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc05806c8f0 0x7fc05806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.127+0000 7fc070877640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc05806c8f0 0x7fc05806edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0b8180 tx=0x7fc060008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.127+0000 7fc075f95740 1 Processor -- start 2024-04-02T13:20:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.127+0000 7fc075f95740 1 -- start start 2024-04-02T13:20:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc075f95740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684965b90 0x558684938b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc075f95740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965f70 0x558684968920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc075f95740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684939070 0x558684968e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc075f95740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5586848a2060 con 0x558684939070 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc075f95740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55868484a650 con 0x558684965f70 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc075f95740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55868489e010 con 0x558684965b90 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc071879640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684939070 0x558684968e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc071879640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684939070 0x558684968e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33758/0 (socket says 172.21.15.44:33758) 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc071879640 1 -- 172.21.15.44:0/2274359789 learned_addr learned my addr 172.21.15.44:0/2274359789 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc071078640 1 --2- 172.21.15.44:0/2274359789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684965b90 0x558684938b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc070877640 1 --2- 172.21.15.44:0/2274359789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965f70 0x558684968920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc071879640 1 -- 172.21.15.44:0/2274359789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684965b90 msgr2=0x558684938b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc071879640 1 --2- 172.21.15.44:0/2274359789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684965b90 0x558684938b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc071879640 1 -- 172.21.15.44:0/2274359789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965f70 msgr2=0x558684968920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc071879640 1 --2- 172.21.15.44:0/2274359789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965f70 0x558684968920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc071879640 1 -- 172.21.15.44:0/2274359789 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586845b62d0 con 0x558684939070 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc071879640 1 --2- 172.21.15.44:0/2274359789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684939070 0x558684968e60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc05c001bd0 tx=0x7fc05c002090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=186aba40ac29747d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.128+0000 7fc052ffd640 1 -- 172.21.15.44:0/2274359789 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c019070 con 0x558684939070 2024-04-02T13:20:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.129+0000 7fc052ffd640 1 -- 172.21.15.44:0/2274359789 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc05c003b80 con 0x558684939070 2024-04-02T13:20:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.129+0000 7fc052ffd640 1 -- 172.21.15.44:0/2274359789 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c00e7a0 con 0x558684939070 2024-04-02T13:20:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.129+0000 7fc075f95740 1 -- 172.21.15.44:0/2274359789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684939070 msgr2=0x558684968e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.129+0000 7fc075f95740 1 --2- 172.21.15.44:0/2274359789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684939070 0x558684968e60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc05c001bd0 tx=0x7fc05c002090 comp rx=0 tx=0).stop 2024-04-02T13:20:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.129+0000 7fc075f95740 1 -- 172.21.15.44:0/2274359789 shutdown_connections 2024-04-02T13:20:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.129+0000 7fc075f95740 1 --2- 172.21.15.44:0/2274359789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684939070 0x558684968e60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.129+0000 7fc075f95740 1 --2- 172.21.15.44:0/2274359789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965f70 0x558684968920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.129+0000 7fc075f95740 1 --2- 172.21.15.44:0/2274359789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684965b90 0x558684938b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.129+0000 7fc075f95740 1 -- 172.21.15.44:0/2274359789 >> 172.21.15.44:0/2274359789 conn(0x55868493ada0 msgr2=0x558684983920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.129+0000 7fc075f95740 1 -- 172.21.15.44:0/2274359789 shutdown_connections 2024-04-02T13:20:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.129+0000 7fc075f95740 1 -- 172.21.15.44:0/2274359789 wait complete. 2024-04-02T13:20:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.130+0000 7fc075f95740 1 -- start start 2024-04-02T13:20:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.130+0000 7fc075f95740 1 Processor -- start 2024-04-02T13:20:28.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc075f95740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965b90 0x558684966dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc075f95740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965f70 0x55868495eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc075f95740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684960be0 0x558684983bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc075f95740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55868493d440 con 0x558684965f70 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc075f95740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55868485da10 con 0x558684965b90 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc075f95740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55868493fc80 con 0x558684960be0 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc075f95740 10 client.0 _ll_register_callbacks cb 0x55868489a0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc070877640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965f70 0x55868495eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc070877640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965f70 0x55868495eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33772/0 (socket says 172.21.15.44:33772) 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc070877640 1 -- 172.21.15.44:0/139774627 learned_addr learned my addr 172.21.15.44:0/139774627 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc071879640 1 --2- 172.21.15.44:0/139774627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684960be0 0x558684983bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc071078640 1 --2- 172.21.15.44:0/139774627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965b90 0x558684966dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc070877640 1 -- 172.21.15.44:0/139774627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684960be0 msgr2=0x558684983bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc070877640 1 --2- 172.21.15.44:0/139774627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684960be0 0x558684983bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc070877640 1 -- 172.21.15.44:0/139774627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965b90 msgr2=0x558684966dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc070877640 1 --2- 172.21.15.44:0/139774627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965b90 0x558684966dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc070877640 1 -- 172.21.15.44:0/139774627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586846806b0 con 0x558684965f70 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc070877640 1 --2- 172.21.15.44:0/139774627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965f70 0x55868495eda0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc060015a50 tx=0x7fc0600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d515c921bfdbe7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:28.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc04bfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc04bfff640 1 -- 172.21.15.44:0/139774627 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc060026070 con 0x558684965f70 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc04bfff640 1 -- 172.21.15.44:0/139774627 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc060016db0 con 0x558684965f70 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.131+0000 7fc075f95740 1 -- 172.21.15.44:0/139774627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc05c003b80 con 0x558684965f70 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc04bfff640 1 -- 172.21.15.44:0/139774627 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fc060031050 con 0x558684965f70 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc075f95740 10 client.4407 fetch_fsmap learned FSMap version 7 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc075f95740 1 -- 172.21.15.44:0/139774627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x558684967ec0 con 0x558684965f70 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc04bfff640 1 -- 172.21.15.44:0/139774627 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc05c003b80 con 0x558684965f70 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc04bfff640 1 -- 172.21.15.44:0/139774627 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7fc060034020 con 0x558684965f70 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc075f95740 10 client.4407 fetch_fsmap finished waiting for FSMap version 7 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc075f95740 10 client.4407 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc075f95740 20 client.4407 populate_metadata read hostname 'smithi044' 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc075f95740 1 --2- 172.21.15.44:0/139774627 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x55868496a8d0 0x558684997800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc075f95740 4 client.4407 mds_command: new command op to 4266 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc075f95740 1 -- 172.21.15.44:0/139774627 --> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558684849cf0 con 0x55868496a8d0 2024-04-02T13:20:28.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc071078640 1 --2- 172.21.15.44:0/139774627 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x55868496a8d0 0x558684997800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc04bfff640 10 client.4407 ms_handle_connect on v2:172.21.15.44:6834/2282156931 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.132+0000 7fc071078640 1 --2- 172.21.15.44:0/139774627 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x55868496a8d0 0x558684997800 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfaa1835719131c1 server_cookie=94c55c54ff4e93b7 in_seq=0 out_seq=0 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc04bfff640 1 -- 172.21.15.44:0/139774627 <== mds.0 v2:172.21.15.44:6834/2282156931 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558684849cf0 con 0x55868496a8d0 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc04bfff640 10 client.4407 handle_command_reply: tid=0 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc075f95740 1 client.4407 shutdown 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc075f95740 1 -- 172.21.15.44:0/139774627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965f70 msgr2=0x55868495eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc075f95740 1 --2- 172.21.15.44:0/139774627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965f70 0x55868495eda0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc060015a50 tx=0x7fc0600046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc075f95740 1 -- 172.21.15.44:0/139774627 shutdown_connections 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc075f95740 1 --2- 172.21.15.44:0/139774627 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x55868496a8d0 0x558684997800 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc075f95740 1 --2- 172.21.15.44:0/139774627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684960be0 0x558684983bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc075f95740 1 --2- 172.21.15.44:0/139774627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965f70 0x55868495eda0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc075f95740 1 --2- 172.21.15.44:0/139774627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965b90 0x558684966dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc04bfff640 0 client.4407 ms_handle_reset on v2:172.21.15.44:6834/2282156931 2024-04-02T13:20:28.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc075f95740 1 -- 172.21.15.44:0/139774627 >> 172.21.15.44:0/139774627 conn(0x558684939070 msgr2=0x55868493cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:28.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc075f95740 1 -- 172.21.15.44:0/139774627 shutdown_connections 2024-04-02T13:20:28.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc075f95740 1 -- 172.21.15.44:0/139774627 wait complete. 2024-04-02T13:20:28.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.133+0000 7fc075f95740 20 client.4407 trim_cache size 0 max 16384 2024-04-02T13:20:28.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.151+0000 7fc075f95740 1 Processor -- start 2024-04-02T13:20:28.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.151+0000 7fc075f95740 1 -- start start 2024-04-02T13:20:28.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc075f95740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684960be0 0x558684969e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc075f95740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965b90 0x558684996480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc075f95740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684965f70 0x558684983620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc075f95740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c201ea0 con 0x558684960be0 2024-04-02T13:20:28.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc075f95740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c200f70 con 0x558684965b90 2024-04-02T13:20:28.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc075f95740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc06c1f8680 con 0x558684965f70 2024-04-02T13:20:28.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc071078640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684960be0 0x558684969e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc071078640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684960be0 0x558684969e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33780/0 (socket says 172.21.15.44:33780) 2024-04-02T13:20:28.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc071078640 1 -- 172.21.15.44:0/2820548558 learned_addr learned my addr 172.21.15.44:0/2820548558 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:28.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc071078640 1 -- 172.21.15.44:0/2820548558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684965f70 msgr2=0x558684983620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:28.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc071078640 1 --2- 172.21.15.44:0/2820548558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684965f70 0x558684983620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc071078640 1 -- 172.21.15.44:0/2820548558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965b90 msgr2=0x558684996480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:28.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc071078640 1 --2- 172.21.15.44:0/2820548558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965b90 0x558684996480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.152+0000 7fc071078640 1 -- 172.21.15.44:0/2820548558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55868495f110 con 0x558684960be0 2024-04-02T13:20:28.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.153+0000 7fc071078640 1 --2- 172.21.15.44:0/2820548558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684960be0 0x558684969e90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc0640ac270 tx=0x7fc06407e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e42acc135ad072ab server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:28.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.153+0000 7fc0527fc640 1 -- 172.21.15.44:0/2820548558 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0640a9070 con 0x558684960be0 2024-04-02T13:20:28.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.153+0000 7fc0527fc640 1 -- 172.21.15.44:0/2820548558 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc06407fe80 con 0x558684960be0 2024-04-02T13:20:28.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.153+0000 7fc0527fc640 1 -- 172.21.15.44:0/2820548558 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0640846b0 con 0x558684960be0 2024-04-02T13:20:28.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.153+0000 7fc075f95740 1 -- 172.21.15.44:0/2820548558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684960be0 msgr2=0x558684969e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.153+0000 7fc075f95740 1 --2- 172.21.15.44:0/2820548558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684960be0 0x558684969e90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc0640ac270 tx=0x7fc06407e040 comp rx=0 tx=0).stop 2024-04-02T13:20:28.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.153+0000 7fc075f95740 1 -- 172.21.15.44:0/2820548558 shutdown_connections 2024-04-02T13:20:28.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.153+0000 7fc075f95740 1 --2- 172.21.15.44:0/2820548558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684965f70 0x558684983620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.153+0000 7fc075f95740 1 --2- 172.21.15.44:0/2820548558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965b90 0x558684996480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.153+0000 7fc075f95740 1 --2- 172.21.15.44:0/2820548558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684960be0 0x558684969e90 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.153+0000 7fc075f95740 1 -- 172.21.15.44:0/2820548558 >> 172.21.15.44:0/2820548558 conn(0x558684939070 msgr2=0x558684966350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:28.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.153+0000 7fc075f95740 1 -- 172.21.15.44:0/2820548558 shutdown_connections 2024-04-02T13:20:28.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.153+0000 7fc075f95740 1 -- 172.21.15.44:0/2820548558 wait complete. 2024-04-02T13:20:28.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.154+0000 7fc075f95740 1 -- start start 2024-04-02T13:20:28.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.154+0000 7fc075f95740 1 Processor -- start 2024-04-02T13:20:28.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc075f95740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684960be0 0x5586849f2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc075f95740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965b90 0x558684964290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc075f95740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965f70 0x5586849647d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc075f95740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558684968760 con 0x558684965b90 2024-04-02T13:20:28.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc075f95740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c200950 con 0x558684965f70 2024-04-02T13:20:28.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc075f95740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55868496acd0 con 0x558684960be0 2024-04-02T13:20:28.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc075f95740 10 client.0 _ll_register_callbacks cb 0x7fc060010900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:20:28.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc070877640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965b90 0x558684964290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc070877640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965b90 0x558684964290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33796/0 (socket says 172.21.15.44:33796) 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc070877640 1 -- 172.21.15.44:0/812335396 learned_addr learned my addr 172.21.15.44:0/812335396 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc071078640 1 --2- 172.21.15.44:0/812335396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684960be0 0x5586849f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc071879640 1 --2- 172.21.15.44:0/812335396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965f70 0x5586849647d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc070877640 1 -- 172.21.15.44:0/812335396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684960be0 msgr2=0x5586849f2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc070877640 1 --2- 172.21.15.44:0/812335396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684960be0 0x5586849f2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc070877640 1 -- 172.21.15.44:0/812335396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965f70 msgr2=0x5586849647d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc070877640 1 --2- 172.21.15.44:0/812335396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965f70 0x5586849647d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc070877640 1 -- 172.21.15.44:0/812335396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586849a73a0 con 0x558684965b90 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc070877640 1 --2- 172.21.15.44:0/812335396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965b90 0x558684964290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc06002b660 tx=0x7fc060009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef999a06dc44dca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.155+0000 7fc04bfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc04bfff640 1 -- 172.21.15.44:0/812335396 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc060026070 con 0x558684965b90 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc04bfff640 1 -- 172.21.15.44:0/812335396 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc06002baf0 con 0x558684965b90 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc075f95740 1 -- 172.21.15.44:0/812335396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc06407fe80 con 0x558684965b90 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc04bfff640 1 -- 172.21.15.44:0/812335396 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fc060042050 con 0x558684965b90 2024-04-02T13:20:28.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc075f95740 10 client.4413 fetch_fsmap learned FSMap version 7 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc075f95740 1 -- 172.21.15.44:0/812335396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x558684996500 con 0x558684965b90 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc04bfff640 1 -- 172.21.15.44:0/812335396 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc06407fe80 con 0x558684965b90 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc04bfff640 1 -- 172.21.15.44:0/812335396 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7fc060045020 con 0x558684965b90 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc075f95740 10 client.4413 fetch_fsmap finished waiting for FSMap version 7 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc075f95740 10 client.4413 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc075f95740 20 client.4413 populate_metadata read hostname 'smithi044' 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc075f95740 1 --2- 172.21.15.44:0/812335396 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x5586849fed30 0x558684937900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc075f95740 4 client.4413 mds_command: new command op to 4266 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc075f95740 1 -- 172.21.15.44:0/812335396 --> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] -- command(tid 0: {"prefix": "status"}) -- 0x5586848af5a0 con 0x5586849fed30 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc071078640 1 --2- 172.21.15.44:0/812335396 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x5586849fed30 0x558684937900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc04bfff640 10 client.4413 ms_handle_connect on v2:172.21.15.44:6834/2282156931 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.156+0000 7fc071078640 1 --2- 172.21.15.44:0/812335396 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x5586849fed30 0x558684937900 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b52792339d49cda server_cookie=5903bd5424f859d5 in_seq=0 out_seq=0 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc04bfff640 1 -- 172.21.15.44:0/812335396 <== mds.0 v2:172.21.15.44:6834/2282156931 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x5586848af5a0 con 0x5586849fed30 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc04bfff640 10 client.4413 handle_command_reply: tid=0 2024-04-02T13:20:28.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc075f95740 1 client.4413 shutdown 2024-04-02T13:20:28.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc075f95740 1 -- 172.21.15.44:0/812335396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965b90 msgr2=0x558684964290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc075f95740 1 --2- 172.21.15.44:0/812335396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965b90 0x558684964290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc06002b660 tx=0x7fc060009190 comp rx=0 tx=0).stop 2024-04-02T13:20:28.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc075f95740 1 -- 172.21.15.44:0/812335396 shutdown_connections 2024-04-02T13:20:28.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc075f95740 1 --2- 172.21.15.44:0/812335396 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x5586849fed30 0x558684937900 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc075f95740 1 --2- 172.21.15.44:0/812335396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558684965f70 0x5586849647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc075f95740 1 --2- 172.21.15.44:0/812335396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558684965b90 0x558684964290 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc075f95740 1 --2- 172.21.15.44:0/812335396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558684960be0 0x5586849f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc04bfff640 0 client.4413 ms_handle_reset on v2:172.21.15.44:6834/2282156931 2024-04-02T13:20:28.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc075f95740 1 -- 172.21.15.44:0/812335396 >> 172.21.15.44:0/812335396 conn(0x558684939070 msgr2=0x558684983620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:28.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc075f95740 1 -- 172.21.15.44:0/812335396 shutdown_connections 2024-04-02T13:20:28.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc075f95740 1 -- 172.21.15.44:0/812335396 wait complete. 2024-04-02T13:20:28.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc075f95740 20 client.4413 trim_cache size 0 max 16384 2024-04-02T13:20:28.166 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:20:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:20:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:20:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 4266, 2024-04-02T13:20:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:20:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:20:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:20:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 7.2851241350000002, 2024-04-02T13:20:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 7, 2024-04-02T13:20:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 25, 2024-04-02T13:20:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 25, 2024-04-02T13:20:28.167 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 14.142227886000001 2024-04-02T13:20:28.167 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:20:28.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc0732dc640 1 -- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc05806c8f0 msgr2=0x7fc05806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc0732dc640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc05806c8f0 0x7fc05806edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0b8180 tx=0x7fc060008040 comp rx=0 tx=0).stop 2024-04-02T13:20:28.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc0732dc640 1 -- 172.21.15.44:0/4195174900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc06c0cfe30 msgr2=0x7fc06c156790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.157+0000 7fc0732dc640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc06c0cfe30 0x7fc06c156790 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc06402b950 tx=0x7fc0640046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.158+0000 7fc0732dc640 1 -- 172.21.15.44:0/4195174900 shutdown_connections 2024-04-02T13:20:28.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.158+0000 7fc0732dc640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc05806c8f0 0x7fc05806edb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.158+0000 7fc0732dc640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc06c0d0e10 0x7fc06c0cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.158+0000 7fc0732dc640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc06c0d0a30 0x7fc06c0b6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.158+0000 7fc0732dc640 1 --2- 172.21.15.44:0/4195174900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc06c0cfe30 0x7fc06c156790 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.158+0000 7fc0732dc640 1 -- 172.21.15.44:0/4195174900 >> 172.21.15.44:0/4195174900 conn(0x7fc06c09d470 msgr2=0x7fc06c0d2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:28.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.158+0000 7fc0732dc640 1 -- 172.21.15.44:0/4195174900 shutdown_connections 2024-04-02T13:20:28.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.158+0000 7fc0732dc640 1 -- 172.21.15.44:0/4195174900 wait complete. 2024-04-02T13:20:28.170 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-02T13:20:28.170 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-02T13:20:28.170 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:20:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.269+0000 7f7fb6b90640 1 Processor -- start 2024-04-02T13:20:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.269+0000 7f7fb6b90640 1 -- start start 2024-04-02T13:20:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.269+0000 7f7fb6b90640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7fb006cf90 0x7f7fb00cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.269+0000 7f7fb6b90640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7fb00cfe30 0x7f7fb00d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.269+0000 7f7fb6b90640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb006c1d0 con 0x7f7fb00d0a30 2024-04-02T13:20:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.269+0000 7f7fb6b90640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7fb006cb00 con 0x7f7fb006cf90 2024-04-02T13:20:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.269+0000 7f7fb6b90640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7fb006d3f0 con 0x7f7fb00cfe30 2024-04-02T13:20:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.270+0000 7f7faffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7fb006cf90 0x7f7fb00cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.270+0000 7f7faffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7fb006cf90 0x7f7fb00cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33810/0 (socket says 172.21.15.44:33810) 2024-04-02T13:20:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.270+0000 7f7faffff640 1 -- 172.21.15.44:0/1804890613 learned_addr learned my addr 172.21.15.44:0/1804890613 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.270+0000 7f7fb512d640 1 --2- 172.21.15.44:0/1804890613 >> v2:172.21.15.69:3300/0 conn(0x7f7fb00cfe30 0x7f7fb00d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.270+0000 7f7faffff640 1 -- 172.21.15.44:0/1804890613 >> v2:172.21.15.69:3300/0 conn(0x7f7fb00cfe30 msgr2=0x7f7fb00d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.270+0000 7f7faffff640 1 --2- 172.21.15.44:0/1804890613 >> v2:172.21.15.69:3300/0 conn(0x7f7fb00cfe30 0x7f7fb00d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.270+0000 7f7faffff640 1 -- 172.21.15.44:0/1804890613 >> v1:172.21.15.69:6789/0 conn(0x7f7fb00d0a30 legacy=0x7f7fb00d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.270+0000 7f7faffff640 1 -- 172.21.15.44:0/1804890613 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb006c4b0 con 0x7f7fb006cf90 2024-04-02T13:20:28.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.270+0000 7f7faffff640 1 --2- 172.21.15.44:0/1804890613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb006cf90 0x7f7fb00cf8f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7fb008c030 tx=0x7f7fa402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df57666148aaf244 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:28.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.271+0000 7f7faf7fe640 1 -- 172.21.15.44:0/1804890613 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa402e070 con 0x7f7fb006cf90 2024-04-02T13:20:28.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.271+0000 7f7faf7fe640 1 -- 172.21.15.44:0/1804890613 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fa4037070 con 0x7f7fb006cf90 2024-04-02T13:20:28.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.271+0000 7f7faf7fe640 1 -- 172.21.15.44:0/1804890613 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4031a00 con 0x7f7fb006cf90 2024-04-02T13:20:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.271+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1804890613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb006cf90 msgr2=0x7f7fb00cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.271+0000 7f7fb6b90640 1 --2- 172.21.15.44:0/1804890613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb006cf90 0x7f7fb00cf8f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7fb008c030 tx=0x7f7fa402b880 comp rx=0 tx=0).stop 2024-04-02T13:20:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.271+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1804890613 shutdown_connections 2024-04-02T13:20:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.271+0000 7f7fb6b90640 1 --2- 172.21.15.44:0/1804890613 >> v2:172.21.15.69:3300/0 conn(0x7f7fb00cfe30 0x7f7fb00d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.271+0000 7f7fb6b90640 1 --2- 172.21.15.44:0/1804890613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb006cf90 0x7f7fb00cf8f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.271+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1804890613 >> 172.21.15.44:0/1804890613 conn(0x7f7fb009d470 msgr2=0x7f7fb00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.271+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1804890613 shutdown_connections 2024-04-02T13:20:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.271+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1804890613 wait complete. 2024-04-02T13:20:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.272+0000 7f7fb6b90640 1 Processor -- start 2024-04-02T13:20:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.272+0000 7f7fb6b90640 1 -- start start 2024-04-02T13:20:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.273+0000 7f7fb6b90640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb006cf90 0x7f7fb00b8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.273+0000 7f7fb6b90640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb00cfe30 0x7f7fb00b6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.273+0000 7f7fb6b90640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb00d0a30 0x7f7fb00b6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.273+0000 7f7fb6b90640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb015e870 con 0x7f7fb006cf90 2024-04-02T13:20:28.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.273+0000 7f7fb6b90640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb015e6f0 con 0x7f7fb00d0a30 2024-04-02T13:20:28.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.273+0000 7f7fb6b90640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb00d4da0 con 0x7f7fb00cfe30 2024-04-02T13:20:28.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.273+0000 7f7fb492c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb006cf90 0x7f7fb00b8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.273+0000 7f7fb492c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb006cf90 0x7f7fb00b8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33814/0 (socket says 172.21.15.44:33814) 2024-04-02T13:20:28.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.273+0000 7f7fb492c640 1 -- 172.21.15.44:0/1398968319 learned_addr learned my addr 172.21.15.44:0/1398968319 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:28.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.273+0000 7f7fb512d640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb00d0a30 0x7f7fb00b6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7faffff640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb00cfe30 0x7f7fb00b6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7fb492c640 1 -- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb00cfe30 msgr2=0x7f7fb00b6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7fb492c640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb00cfe30 0x7f7fb00b6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7fb492c640 1 -- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb00d0a30 msgr2=0x7f7fb00b6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7fb492c640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb00d0a30 0x7f7fb00b6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7fb492c640 1 -- 172.21.15.44:0/1398968319 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb00b4fd0 con 0x7f7fb006cf90 2024-04-02T13:20:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7faffff640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb00cfe30 0x7f7fb00b6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7fb512d640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb00d0a30 0x7f7fb00b6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7fb492c640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb006cf90 0x7f7fb00b8380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7fa000b3f0 tx=0x7f7fa000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7fad7fa640 1 -- 172.21.15.44:0/1398968319 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0015020 con 0x7f7fb006cf90 2024-04-02T13:20:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7fad7fa640 1 -- 172.21.15.44:0/1398968319 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fa0011830 con 0x7f7fb006cf90 2024-04-02T13:20:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7fad7fa640 1 -- 172.21.15.44:0/1398968319 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0011c20 con 0x7f7fb006cf90 2024-04-02T13:20:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1398968319 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb00c9ee0 con 0x7f7fb006cf90 2024-04-02T13:20:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.274+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1398968319 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb00ca4a0 con 0x7f7fb006cf90 2024-04-02T13:20:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.275+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1398968319 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fb00684f0 con 0x7f7fb006cf90 2024-04-02T13:20:28.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.275+0000 7f7fad7fa640 1 -- 172.21.15.44:0/1398968319 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7fa0018020 con 0x7f7fb006cf90 2024-04-02T13:20:28.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.276+0000 7f7fad7fa640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f9c06c8f0 0x7f7f9c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:28.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.276+0000 7f7fad7fa640 1 -- 172.21.15.44:0/1398968319 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7fa008bb30 con 0x7f7fb006cf90 2024-04-02T13:20:28.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.276+0000 7f7faffff640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f9c06c8f0 0x7f7f9c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:28.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.276+0000 7f7faffff640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f9c06c8f0 0x7f7f9c06edb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4036000 tx=0x7f7fa4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:28.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.280+0000 7f7fad7fa640 1 -- 172.21.15.44:0/1398968319 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7fa001f030 con 0x7f7fb006cf90 2024-04-02T13:20:28.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.505+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1398968319 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7fb00d17c0 con 0x7f7fb006cf90 2024-04-02T13:20:28.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.505+0000 7f7fad7fa640 1 -- 172.21.15.44:0/1398968319 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7f7fa0059b90 con 0x7f7fb006cf90 2024-04-02T13:20:28.508 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-02T13:20:28.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.507+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f9c06c8f0 msgr2=0x7f7f9c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.507+0000 7f7fb6b90640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f9c06c8f0 0x7f7f9c06edb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4036000 tx=0x7f7fa4008030 comp rx=0 tx=0).stop 2024-04-02T13:20:28.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.507+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1398968319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb006cf90 msgr2=0x7f7fb00b8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:28.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.507+0000 7f7fb6b90640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb006cf90 0x7f7fb00b8380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7fa000b3f0 tx=0x7f7fa000b8b0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.507+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1398968319 shutdown_connections 2024-04-02T13:20:28.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.507+0000 7f7fb6b90640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f9c06c8f0 0x7f7f9c06edb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.507+0000 7f7fb6b90640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb00d0a30 0x7f7fb00b6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.507+0000 7f7fb6b90640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb00cfe30 0x7f7fb00b6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.507+0000 7f7fb6b90640 1 --2- 172.21.15.44:0/1398968319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb006cf90 0x7f7fb00b8380 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:28.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.507+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1398968319 >> 172.21.15.44:0/1398968319 conn(0x7f7fb009d470 msgr2=0x7f7fb00d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:28.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.507+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1398968319 shutdown_connections 2024-04-02T13:20:28.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28.507+0000 7f7fb6b90640 1 -- 172.21.15.44:0/1398968319 wait complete. 2024-04-02T13:20:28.520 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T13:20:28.520 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:20:28.520 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:20:28.520 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-02T13:20:28.520 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-02T13:20:28.520 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T13:20:28.520 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:20:28.520 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:20:28.520 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-02T13:20:28.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:28.521 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:20:28.535 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:20:28.535 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:20:28.535 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:20:28.535 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42750sec preferred_lft 42750sec 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:20:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:28.537 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:20:28.537 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:28.537 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:20:28.537 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-02T13:20:28.537 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:28.538 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:28.538 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add name ceph-brx type bridge 2024-04-02T13:20:28.538 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr flush dev ceph-brx 2024-04-02T13:20:28.538 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set ceph-brx up 2024-04-02T13:20:28.538 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-02T13:20:28.538 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:28.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:28.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:28.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:28.729 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:20:28.755 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:20:28.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:28.757 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:20:28.822 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:20:28.822 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:20:28.822 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:20:28.822 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:20:28.823 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:20:28.823 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:28.823 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:28.823 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:20:28.823 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:20:28.823 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:20:28.824 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:28.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:29.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:29.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:29.158 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:20:29.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:29.173 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list-id 2024-04-02T13:20:29.227 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:29.227 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:29.227 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-02T13:20:29.227 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-02T13:20:29.227 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:29.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:29.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:29.348 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-02T13:20:29.348 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:29.349 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:29.349 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-02T13:20:29.349 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-02T13:20:29.349 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-02T13:20:29.349 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-02T13:20:29.349 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-02T13:20:29.349 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:29.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:29.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:29.488 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:29.488 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:29.488 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:20:29.488 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set dev brx.0 master ceph-brx 2024-04-02T13:20:29.488 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:29.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:29.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:29.569 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-02T13:20:29.569 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:20:29.569 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-02T13:20:29.586 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-02T13:20:29.586 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:20:29.586 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-02T13:20:29.643 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:20:29.711 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:20:29.768 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:20:29.768 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:20:29.768 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:20:29.768 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:20:29.768 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:20:29.768 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:20:29.768 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:20:29.768 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:20:29.768 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:20:29.769 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:20:29.770 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:20:29.770 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:20:29.770 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:20:29.770 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:20:29.770 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:20:29.770 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:20:29.770 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:20:29.770 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-02T13:20:29.770 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-02T13:20:29.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:29.770 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:20:29.822 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:20:29.822 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-02T13:20:29.865 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:20:29.891 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:20:29.946 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:20:29.947 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:20:29.947 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:20:29.947 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:20:29.947 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:20:29.947 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:20:29.947 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:20:29.947 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:20:29.947 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:20:29.947 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:20:29.947 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:20:29.947 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:20:29.947 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:20:29.947 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:20:29.948 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:20:29.948 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:20:29.948 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:20:29.948 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:20:29.948 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:20:29.948 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:20:29.948 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:20:29.948 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:20:29.948 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:20:29.948 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:20:29.948 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:20:29.948 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:20:29.948 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:20:29.949 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:20:29.949 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:20:29.949 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:20:29.949 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-02T13:20:29.949 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-02T13:20:29.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:29.949 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:20:29.953 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:2024-04-02T13:20:29.951+0000 7ff90da37480 -1 init, newargv = 0x5627e19e3650 newargc=15 2024-04-02T13:20:29.953 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:ceph-fuse[44076]: starting ceph client 2024-04-02T13:20:29.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:ceph-fuse[44076]: starting fuse 2024-04-02T13:20:29.967 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:20:29.967 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:20:29.967 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:20:29.968 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:20:29.969 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:20:29.969 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:20:29.969 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:20:29.969 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:20:29.969 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:20:29.969 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:20:29.969 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:20:29.969 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:20:29.969 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:20:29.969 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.admin")) 2024-04-02T13:20:29.969 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:20:30.089 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.admin.44076.asok 2024-04-02T13:20:30.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:30.099 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.44076.asok 2024-04-02T13:20:30.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:30.100 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.44076.asok status 2024-04-02T13:20:30.237 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:20:30.237 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:20:30.237 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:20:30.237 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:20:30.237 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "admin", 2024-04-02T13:20:30.237 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:20:30.237 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-02T13:20:30.237 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "44076", 2024-04-02T13:20:30.237 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:20:30.237 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:20:30.237 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:20:30.238 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:20:30.238 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 4425, 2024-04-02T13:20:30.238 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:20:30.238 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:20:30.238 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:20:30.238 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 4425 2024-04-02T13:20:30.238 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:20:30.238 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:20:30.238 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:20:30.238 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:20:30.238 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 150788679 2024-04-02T13:20:30.238 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:20:30.238 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 150788679 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.4425 192.168.0.1:0/150788679", 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/150788679", 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 7, 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 25, 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:20:30.239 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:20:30.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:30.245 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:20:30.245 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-02T13:20:30.260 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:20:30.260 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-02T13:20:30.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:30.260 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-02T13:20:30.327 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-02T13:20:30.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-02T13:20:30.435 INFO:teuthology.orchestra.run.smithi044.stdout:# file: . 2024-04-02T13:20:30.435 INFO:teuthology.orchestra.run.smithi044.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-02T13:20:30.436 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:30.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:30.441 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:20:30.441 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:20:30.456 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:30.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.595+0000 7f487f4a5640 1 Processor -- start 2024-04-02T13:20:30.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.595+0000 7f487f4a5640 1 -- start start 2024-04-02T13:20:30.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.596+0000 7f487f4a5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f487815e520 0x7f48780d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:30.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.596+0000 7f487f4a5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48780b70b0 con 0x7f48780d0f80 2024-04-02T13:20:30.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.596+0000 7f487f4a5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48780b7390 con 0x7f487815e140 2024-04-02T13:20:30.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.596+0000 7f487f4a5640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f48780b9f50 con 0x7f487815e520 2024-04-02T13:20:30.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.596+0000 7f487ca40640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f487815e520 0x7f48780d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:30.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.596+0000 7f487da42640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f48780d0f80 0x7f48780b4cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41564/0 (socket says 172.21.15.44:41564) 2024-04-02T13:20:30.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.596+0000 7f487da42640 1 -- 172.21.15.44:0/3506375778 learned_addr learned my addr 172.21.15.44:0/3506375778 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:30.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.596+0000 7f487ca40640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f487815e520 0x7f48780d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33830/0 (socket says 172.21.15.44:33830) 2024-04-02T13:20:30.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.596+0000 7f487ca40640 1 -- 172.21.15.44:0/3506375778 >> v1:172.21.15.69:6789/0 conn(0x7f487815e140 legacy=0x7f48780d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:30.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.596+0000 7f487ca40640 1 -- 172.21.15.44:0/3506375778 >> v1:172.21.15.44:6789/0 conn(0x7f48780d0f80 legacy=0x7f48780b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:30.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.596+0000 7f486ffff640 1 -- 172.21.15.44:0/3506375778 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3596575852 0 0) 0x7f48780b70b0 con 0x7f48780d0f80 2024-04-02T13:20:30.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.596+0000 7f487ca40640 1 -- 172.21.15.44:0/3506375778 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48780b7670 con 0x7f487815e520 2024-04-02T13:20:30.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.597+0000 7f487ca40640 1 --2- 172.21.15.44:0/3506375778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f487815e520 0x7f48780d0a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f48680097a0 tx=0x7f486802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=890b4bc4650bcc6b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:30.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.597+0000 7f486ffff640 1 -- 172.21.15.44:0/3506375778 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4868034070 con 0x7f487815e520 2024-04-02T13:20:30.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.597+0000 7f486ffff640 1 -- 172.21.15.44:0/3506375778 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4868030610 con 0x7f487815e520 2024-04-02T13:20:30.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.597+0000 7f486ffff640 1 -- 172.21.15.44:0/3506375778 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48680309e0 con 0x7f487815e520 2024-04-02T13:20:30.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.598+0000 7f487f4a5640 1 -- 172.21.15.44:0/3506375778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f487815e520 msgr2=0x7f48780d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:30.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.598+0000 7f487f4a5640 1 --2- 172.21.15.44:0/3506375778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f487815e520 0x7f48780d0a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f48680097a0 tx=0x7f486802b960 comp rx=0 tx=0).stop 2024-04-02T13:20:30.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.598+0000 7f487f4a5640 1 -- 172.21.15.44:0/3506375778 shutdown_connections 2024-04-02T13:20:30.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.598+0000 7f487f4a5640 1 --2- 172.21.15.44:0/3506375778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f487815e520 0x7f48780d0a40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:30.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.598+0000 7f487f4a5640 1 -- 172.21.15.44:0/3506375778 >> 172.21.15.44:0/3506375778 conn(0x7f487809d390 msgr2=0x7f48780d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:30.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.598+0000 7f487f4a5640 1 -- 172.21.15.44:0/3506375778 shutdown_connections 2024-04-02T13:20:30.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.598+0000 7f487f4a5640 1 -- 172.21.15.44:0/3506375778 wait complete. 2024-04-02T13:20:30.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.599+0000 7f487f4a5640 1 Processor -- start 2024-04-02T13:20:30.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.599+0000 7f487f4a5640 1 -- start start 2024-04-02T13:20:30.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487f4a5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48780d0f80 0x7f48780cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:30.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487f4a5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f487815e140 0x7f487814f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:30.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487f4a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487815e520 0x7f487814f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:30.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487f4a5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f48780d2d40 con 0x7f48780d0f80 2024-04-02T13:20:30.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487f4a5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f48780d4ee0 con 0x7f487815e520 2024-04-02T13:20:30.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487f4a5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f48780b8e60 con 0x7f487815e140 2024-04-02T13:20:30.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487d241640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48780d0f80 0x7f48780cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:30.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487d241640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48780d0f80 0x7f48780cdd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33838/0 (socket says 172.21.15.44:33838) 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487d241640 1 -- 172.21.15.44:0/1968162488 learned_addr learned my addr 172.21.15.44:0/1968162488 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487da42640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487815e520 0x7f487814f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487d241640 1 -- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f487815e140 msgr2=0x7f487814f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487d241640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f487815e140 0x7f487814f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487d241640 1 -- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487815e520 msgr2=0x7f487814f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487d241640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487815e520 0x7f487814f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487d241640 1 -- 172.21.15.44:0/1968162488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487815b900 con 0x7f48780d0f80 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487da42640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487815e520 0x7f487814f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487d241640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48780d0f80 0x7f48780cdd30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f487400c450 tx=0x7f487400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f486dffb640 1 -- 172.21.15.44:0/1968162488 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48740052d0 con 0x7f48780d0f80 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f486dffb640 1 -- 172.21.15.44:0/1968162488 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4874005470 con 0x7f48780d0f80 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f487f4a5640 1 -- 172.21.15.44:0/1968162488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f487815bbe0 con 0x7f48780d0f80 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.600+0000 7f486dffb640 1 -- 172.21.15.44:0/1968162488 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4874012870 con 0x7f48780d0f80 2024-04-02T13:20:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.601+0000 7f487f4a5640 1 -- 172.21.15.44:0/1968162488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f487815c1a0 con 0x7f48780d0f80 2024-04-02T13:20:30.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.601+0000 7f487f4a5640 1 -- 172.21.15.44:0/1968162488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48780684f0 con 0x7f48780d0f80 2024-04-02T13:20:30.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.602+0000 7f486dffb640 1 -- 172.21.15.44:0/1968162488 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4874012a10 con 0x7f48780d0f80 2024-04-02T13:20:30.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.602+0000 7f486dffb640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f485c06c8f0 0x7f485c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:30.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.602+0000 7f486dffb640 1 -- 172.21.15.44:0/1968162488 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f487408bb20 con 0x7f48780d0f80 2024-04-02T13:20:30.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.602+0000 7f487ca40640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f485c06c8f0 0x7f485c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:30.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.603+0000 7f487ca40640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f485c06c8f0 0x7f485c06edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4868036000 tx=0x7f4868030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:30.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.607+0000 7f486dffb640 1 -- 172.21.15.44:0/1968162488 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4874059c10 con 0x7f48780d0f80 2024-04-02T13:20:30.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.795+0000 7f487f4a5640 1 -- 172.21.15.44:0/1968162488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48780d2990 con 0x7f48780d0f80 2024-04-02T13:20:30.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.796+0000 7f486dffb640 1 -- 172.21.15.44:0/1968162488 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f4874059510 con 0x7f48780d0f80 2024-04-02T13:20:30.798 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:30.799 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:30.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.798+0000 7f487f4a5640 1 -- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f485c06c8f0 msgr2=0x7f485c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.798+0000 7f487f4a5640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f485c06c8f0 0x7f485c06edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4868036000 tx=0x7f4868030000 comp rx=0 tx=0).stop 2024-04-02T13:20:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.798+0000 7f487f4a5640 1 -- 172.21.15.44:0/1968162488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48780d0f80 msgr2=0x7f48780cdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.798+0000 7f487f4a5640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48780d0f80 0x7f48780cdd30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f487400c450 tx=0x7f487400c910 comp rx=0 tx=0).stop 2024-04-02T13:20:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.798+0000 7f487f4a5640 1 -- 172.21.15.44:0/1968162488 shutdown_connections 2024-04-02T13:20:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.798+0000 7f487f4a5640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f485c06c8f0 0x7f485c06edb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.798+0000 7f487f4a5640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487815e520 0x7f487814f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.798+0000 7f487f4a5640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f487815e140 0x7f487814f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.798+0000 7f487f4a5640 1 --2- 172.21.15.44:0/1968162488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48780d0f80 0x7f48780cdd30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.798+0000 7f487f4a5640 1 -- 172.21.15.44:0/1968162488 >> 172.21.15.44:0/1968162488 conn(0x7f487809d390 msgr2=0x7f48780d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.798+0000 7f487f4a5640 1 -- 172.21.15.44:0/1968162488 shutdown_connections 2024-04-02T13:20:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.798+0000 7f487f4a5640 1 -- 172.21.15.44:0/1968162488 wait complete. 2024-04-02T13:20:30.811 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:20:30.811 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:20:30.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.914+0000 7fcfb00b7640 1 Processor -- start 2024-04-02T13:20:30.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.914+0000 7fcfb00b7640 1 -- start start 2024-04-02T13:20:30.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfb00b7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcfa815e1d0 0x7fcfa80d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:30.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfb00b7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcfa80d2dd0 0x7fcfa80b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:30.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfb00b7640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfa80d1640 con 0x7fcfa80cec60 2024-04-02T13:20:30.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfb00b7640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fcfa80cfe00 con 0x7fcfa80d2dd0 2024-04-02T13:20:30.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfb00b7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcfa80cf1f0 con 0x7fcfa815e1d0 2024-04-02T13:20:30.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfae654640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcfa80d2dd0 0x7fcfa80b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:30.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfad652640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fcfa80cec60 0x7fcfa80d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41574/0 (socket says 172.21.15.44:41574) 2024-04-02T13:20:30.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfae654640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcfa80d2dd0 0x7fcfa80b4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33842/0 (socket says 172.21.15.44:33842) 2024-04-02T13:20:30.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfad652640 1 -- 172.21.15.44:0/1053029683 learned_addr learned my addr 172.21.15.44:0/1053029683 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:30.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfade53640 1 --2- 172.21.15.44:0/1053029683 >> v2:172.21.15.69:3300/0 conn(0x7fcfa815e1d0 0x7fcfa80d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:30.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfae654640 1 -- 172.21.15.44:0/1053029683 >> v2:172.21.15.69:3300/0 conn(0x7fcfa815e1d0 msgr2=0x7fcfa80d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:30.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfae654640 1 --2- 172.21.15.44:0/1053029683 >> v2:172.21.15.69:3300/0 conn(0x7fcfa815e1d0 0x7fcfa80d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:30.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfae654640 1 -- 172.21.15.44:0/1053029683 >> v1:172.21.15.44:6789/0 conn(0x7fcfa80cec60 legacy=0x7fcfa80d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:30.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.915+0000 7fcfae654640 1 -- 172.21.15.44:0/1053029683 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa80cf4d0 con 0x7fcfa80d2dd0 2024-04-02T13:20:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.916+0000 7fcfae654640 1 --2- 172.21.15.44:0/1053029683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa80d2dd0 0x7fcfa80b4ce0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcfa808c9c0 tx=0x7fcfa002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874f7a70c8a34c5b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.916+0000 7fcface51640 1 -- 172.21.15.44:0/1053029683 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0039070 con 0x7fcfa80d2dd0 2024-04-02T13:20:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.916+0000 7fcface51640 1 -- 172.21.15.44:0/1053029683 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfa002cbf0 con 0x7fcfa80d2dd0 2024-04-02T13:20:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.916+0000 7fcface51640 1 -- 172.21.15.44:0/1053029683 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0034440 con 0x7fcfa80d2dd0 2024-04-02T13:20:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.917+0000 7fcfb00b7640 1 -- 172.21.15.44:0/1053029683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa80d2dd0 msgr2=0x7fcfa80b4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.917+0000 7fcfb00b7640 1 --2- 172.21.15.44:0/1053029683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa80d2dd0 0x7fcfa80b4ce0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcfa808c9c0 tx=0x7fcfa002be40 comp rx=0 tx=0).stop 2024-04-02T13:20:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.917+0000 7fcfb00b7640 1 -- 172.21.15.44:0/1053029683 shutdown_connections 2024-04-02T13:20:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.917+0000 7fcfb00b7640 1 --2- 172.21.15.44:0/1053029683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa80d2dd0 0x7fcfa80b4ce0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.917+0000 7fcfb00b7640 1 --2- 172.21.15.44:0/1053029683 >> v2:172.21.15.69:3300/0 conn(0x7fcfa815e1d0 0x7fcfa80d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.917+0000 7fcfb00b7640 1 -- 172.21.15.44:0/1053029683 >> 172.21.15.44:0/1053029683 conn(0x7fcfa809d3b0 msgr2=0x7fcfa80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.917+0000 7fcfb00b7640 1 -- 172.21.15.44:0/1053029683 shutdown_connections 2024-04-02T13:20:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.917+0000 7fcfb00b7640 1 -- 172.21.15.44:0/1053029683 wait complete. 2024-04-02T13:20:30.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.918+0000 7fcfb00b7640 1 Processor -- start 2024-04-02T13:20:30.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.918+0000 7fcfb00b7640 1 -- start start 2024-04-02T13:20:30.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfb00b7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa80cec60 0x7fcfa81607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfb00b7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfa80d2dd0 0x7fcfa814d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfb00b7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa815e1d0 0x7fcfa809f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfb00b7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa815e5b0 con 0x7fcfa80cec60 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfb00b7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa80cfa90 con 0x7fcfa815e1d0 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfb00b7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa815e730 con 0x7fcfa80d2dd0 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfade53640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa80cec60 0x7fcfa81607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfade53640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa80cec60 0x7fcfa81607a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33852/0 (socket says 172.21.15.44:33852) 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfade53640 1 -- 172.21.15.44:0/2943162358 learned_addr learned my addr 172.21.15.44:0/2943162358 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfad652640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfa80d2dd0 0x7fcfa814d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfae654640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa815e1d0 0x7fcfa809f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfade53640 1 -- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfa80d2dd0 msgr2=0x7fcfa814d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfade53640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfa80d2dd0 0x7fcfa814d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfade53640 1 -- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa815e1d0 msgr2=0x7fcfa809f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfade53640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa815e1d0 0x7fcfa809f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:30.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.919+0000 7fcfade53640 1 -- 172.21.15.44:0/2943162358 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa809f9f0 con 0x7fcfa80cec60 2024-04-02T13:20:30.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.920+0000 7fcfade53640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa80cec60 0x7fcfa81607a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400e9b0 tx=0x7fcfa400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:30.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.920+0000 7fcf9e7fc640 1 -- 172.21.15.44:0/2943162358 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4017070 con 0x7fcfa80cec60 2024-04-02T13:20:30.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.920+0000 7fcf9e7fc640 1 -- 172.21.15.44:0/2943162358 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfa400c9a0 con 0x7fcfa80cec60 2024-04-02T13:20:30.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.920+0000 7fcfb00b7640 1 -- 172.21.15.44:0/2943162358 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa809fcd0 con 0x7fcfa80cec60 2024-04-02T13:20:30.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.920+0000 7fcf9e7fc640 1 -- 172.21.15.44:0/2943162358 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa400cd90 con 0x7fcfa80cec60 2024-04-02T13:20:30.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.920+0000 7fcfb00b7640 1 -- 172.21.15.44:0/2943162358 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa815ed40 con 0x7fcfa80cec60 2024-04-02T13:20:30.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.921+0000 7fcfb00b7640 1 -- 172.21.15.44:0/2943162358 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa809fea0 con 0x7fcfa80cec60 2024-04-02T13:20:30.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.923+0000 7fcf9e7fc640 1 -- 172.21.15.44:0/2943162358 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcfa401a070 con 0x7fcfa80cec60 2024-04-02T13:20:30.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.923+0000 7fcf9e7fc640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcf8c06c8f0 0x7fcf8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:30.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.923+0000 7fcf9e7fc640 1 -- 172.21.15.44:0/2943162358 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fcfa408c4a0 con 0x7fcfa80cec60 2024-04-02T13:20:30.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.923+0000 7fcfad652640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcf8c06c8f0 0x7fcf8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:30.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.924+0000 7fcfad652640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcf8c06c8f0 0x7fcf8c06edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcfa815aa80 tx=0x7fcf98007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:30.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:30.927+0000 7fcf9e7fc640 1 -- 172.21.15.44:0/2943162358 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fcfa405a0b0 con 0x7fcfa80cec60 2024-04-02T13:20:31.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.108+0000 7fcfb00b7640 1 -- 172.21.15.44:0/2943162358 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcfa80684f0 con 0x7fcfa80cec60 2024-04-02T13:20:31.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.108+0000 7fcf9e7fc640 1 -- 172.21.15.44:0/2943162358 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7fcfa405e3b0 con 0x7fcfa80cec60 2024-04-02T13:20:31.111 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:20:31.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.110+0000 7fcfb00b7640 1 -- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcf8c06c8f0 msgr2=0x7fcf8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.110+0000 7fcfb00b7640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcf8c06c8f0 0x7fcf8c06edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcfa815aa80 tx=0x7fcf98007e80 comp rx=0 tx=0).stop 2024-04-02T13:20:31.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.110+0000 7fcfb00b7640 1 -- 172.21.15.44:0/2943162358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa80cec60 msgr2=0x7fcfa81607a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.110+0000 7fcfb00b7640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa80cec60 0x7fcfa81607a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400e9b0 tx=0x7fcfa400ee70 comp rx=0 tx=0).stop 2024-04-02T13:20:31.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.110+0000 7fcfb00b7640 1 -- 172.21.15.44:0/2943162358 shutdown_connections 2024-04-02T13:20:31.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.110+0000 7fcfb00b7640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcf8c06c8f0 0x7fcf8c06edb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.110+0000 7fcfb00b7640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa815e1d0 0x7fcfa809f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.110+0000 7fcfb00b7640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfa80d2dd0 0x7fcfa814d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.110+0000 7fcfb00b7640 1 --2- 172.21.15.44:0/2943162358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa80cec60 0x7fcfa81607a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.110+0000 7fcfb00b7640 1 -- 172.21.15.44:0/2943162358 >> 172.21.15.44:0/2943162358 conn(0x7fcfa809d3b0 msgr2=0x7fcfa80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:31.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.110+0000 7fcfb00b7640 1 -- 172.21.15.44:0/2943162358 shutdown_connections 2024-04-02T13:20:31.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.110+0000 7fcfb00b7640 1 -- 172.21.15.44:0/2943162358 wait complete. 2024-04-02T13:20:31.123 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:20:31.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.225+0000 7ff2ccb5f640 1 Processor -- start 2024-04-02T13:20:31.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.225+0000 7ff2ccb5f640 1 -- start start 2024-04-02T13:20:31.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.226+0000 7ff2ccb5f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff2c815c1a0 0x7ff2c806f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.226+0000 7ff2ccb5f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff2c80cfd60 0x7ff2c80d1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.226+0000 7ff2ccb5f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2c815c580 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.226+0000 7ff2ccb5f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff2c8070ad0 con 0x7ff2c80cfd60 2024-04-02T13:20:31.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.226+0000 7ff2ccb5f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff2c80d01c0 con 0x7ff2c815c1a0 2024-04-02T13:20:31.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.226+0000 7ff2c5d9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff2c80cfd60 0x7ff2c80d1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.226+0000 7ff2c6d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff2c80d1ff0 0x7ff2c80ce780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41576/0 (socket says 172.21.15.44:41576) 2024-04-02T13:20:31.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.226+0000 7ff2c5d9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff2c80cfd60 0x7ff2c80d1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33866/0 (socket says 172.21.15.44:33866) 2024-04-02T13:20:31.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.226+0000 7ff2c6d9d640 1 -- 172.21.15.44:0/151248563 learned_addr learned my addr 172.21.15.44:0/151248563 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:31.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.226+0000 7ff2c659c640 1 --2- 172.21.15.44:0/151248563 >> v2:172.21.15.69:3300/0 conn(0x7ff2c815c1a0 0x7ff2c806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.227+0000 7ff2c5d9b640 1 -- 172.21.15.44:0/151248563 >> v2:172.21.15.69:3300/0 conn(0x7ff2c815c1a0 msgr2=0x7ff2c806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:31.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.227+0000 7ff2c5d9b640 1 --2- 172.21.15.44:0/151248563 >> v2:172.21.15.69:3300/0 conn(0x7ff2c815c1a0 0x7ff2c806f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.227+0000 7ff2c5d9b640 1 -- 172.21.15.44:0/151248563 >> v1:172.21.15.44:6789/0 conn(0x7ff2c80d1ff0 legacy=0x7ff2c80ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.227+0000 7ff2c559a640 1 -- 172.21.15.44:0/151248563 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4834516 0 0) 0x7ff2c815c580 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.227+0000 7ff2c5d9b640 1 -- 172.21.15.44:0/151248563 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c806fa30 con 0x7ff2c80cfd60 2024-04-02T13:20:31.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.227+0000 7ff2c5d9b640 1 --2- 172.21.15.44:0/151248563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2c80cfd60 0x7ff2c80d1ab0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff2c809b780 tx=0x7ff2b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8484cea246befb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:31.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.228+0000 7ff2c559a640 1 -- 172.21.15.44:0/151248563 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b0035020 con 0x7ff2c80cfd60 2024-04-02T13:20:31.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.228+0000 7ff2c559a640 1 -- 172.21.15.44:0/151248563 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2b0031830 con 0x7ff2c80cfd60 2024-04-02T13:20:31.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.228+0000 7ff2c559a640 1 -- 172.21.15.44:0/151248563 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b0031c00 con 0x7ff2c80cfd60 2024-04-02T13:20:31.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.228+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/151248563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2c80cfd60 msgr2=0x7ff2c80d1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.228+0000 7ff2ccb5f640 1 --2- 172.21.15.44:0/151248563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2c80cfd60 0x7ff2c80d1ab0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff2c809b780 tx=0x7ff2b002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.228+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/151248563 shutdown_connections 2024-04-02T13:20:31.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.228+0000 7ff2ccb5f640 1 --2- 172.21.15.44:0/151248563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2c80cfd60 0x7ff2c80d1ab0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.228+0000 7ff2ccb5f640 1 --2- 172.21.15.44:0/151248563 >> v2:172.21.15.69:3300/0 conn(0x7ff2c815c1a0 0x7ff2c806f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.228+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/151248563 >> 172.21.15.44:0/151248563 conn(0x7ff2c809aa20 msgr2=0x7ff2c809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:31.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.228+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/151248563 shutdown_connections 2024-04-02T13:20:31.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.228+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/151248563 wait complete. 2024-04-02T13:20:31.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.230+0000 7ff2ccb5f640 1 Processor -- start 2024-04-02T13:20:31.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.230+0000 7ff2ccb5f640 1 -- start start 2024-04-02T13:20:31.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.230+0000 7ff2ccb5f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2c80cfd60 0x7ff2c806e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.230+0000 7ff2ccb5f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2c80d1ff0 0x7ff2c814c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.230+0000 7ff2ccb5f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2c815c1a0 0x7ff2c80b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.230+0000 7ff2ccb5f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c80702d0 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.230+0000 7ff2ccb5f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c806fff0 con 0x7ff2c80cfd60 2024-04-02T13:20:31.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.230+0000 7ff2ccb5f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff2c80705b0 con 0x7ff2c815c1a0 2024-04-02T13:20:31.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.230+0000 7ff2c5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2c80d1ff0 0x7ff2c814c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.230+0000 7ff2c5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2c80d1ff0 0x7ff2c814c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33880/0 (socket says 172.21.15.44:33880) 2024-04-02T13:20:31.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.230+0000 7ff2c5d9b640 1 -- 172.21.15.44:0/2908297586 learned_addr learned my addr 172.21.15.44:0/2908297586 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2c659c640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2c80cfd60 0x7ff2c806e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2c6d9d640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2c815c1a0 0x7ff2c80b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2c5d9b640 1 -- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2c815c1a0 msgr2=0x7ff2c80b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2c5d9b640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2c815c1a0 0x7ff2c80b2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2c5d9b640 1 -- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2c80cfd60 msgr2=0x7ff2c806e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2c5d9b640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2c80cfd60 0x7ff2c806e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2c5d9b640 1 -- 172.21.15.44:0/2908297586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c80c75c0 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2c659c640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2c80cfd60 0x7ff2c806e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2c5d9b640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2c80d1ff0 0x7ff2c814c260 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0009890 tx=0x7ff2b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2beffd640 1 -- 172.21.15.44:0/2908297586 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b0041070 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2beffd640 1 -- 172.21.15.44:0/2908297586 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2b003b920 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/2908297586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2c80c78a0 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2beffd640 1 -- 172.21.15.44:0/2908297586 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b0035040 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.231+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/2908297586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2c80c7e60 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.232+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/2908297586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2c8068c00 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.232+0000 7ff2beffd640 1 -- 172.21.15.44:0/2908297586 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff2b0038020 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.233+0000 7ff2beffd640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff2a806c8f0 0x7ff2a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.233+0000 7ff2beffd640 1 -- 172.21.15.44:0/2908297586 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff2b00ad7f0 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.233+0000 7ff2c659c640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff2a806c8f0 0x7ff2a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.233+0000 7ff2c659c640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff2a806c8f0 0x7ff2a806edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009a20 tx=0x7ff2c0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:31.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.237+0000 7ff2beffd640 1 -- 172.21.15.44:0/2908297586 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff2b002f2e0 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.419+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/2908297586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff2c80b0a30 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.419+0000 7ff2beffd640 1 -- 172.21.15.44:0/2908297586 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7ff2b0031800 con 0x7ff2c80d1ff0 2024-04-02T13:20:31.422 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:20:31.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.421+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff2a806c8f0 msgr2=0x7ff2a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.421+0000 7ff2ccb5f640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff2a806c8f0 0x7ff2a806edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009a20 tx=0x7ff2c0008040 comp rx=0 tx=0).stop 2024-04-02T13:20:31.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.421+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/2908297586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2c80d1ff0 msgr2=0x7ff2c814c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.421+0000 7ff2ccb5f640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2c80d1ff0 0x7ff2c814c260 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0009890 tx=0x7ff2b00046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.421+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/2908297586 shutdown_connections 2024-04-02T13:20:31.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.421+0000 7ff2ccb5f640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff2a806c8f0 0x7ff2a806edb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.421+0000 7ff2ccb5f640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2c815c1a0 0x7ff2c80b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.421+0000 7ff2ccb5f640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2c80d1ff0 0x7ff2c814c260 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.421+0000 7ff2ccb5f640 1 --2- 172.21.15.44:0/2908297586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2c80cfd60 0x7ff2c806e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.421+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/2908297586 >> 172.21.15.44:0/2908297586 conn(0x7ff2c809aa20 msgr2=0x7ff2c80af410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:31.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.421+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/2908297586 shutdown_connections 2024-04-02T13:20:31.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.421+0000 7ff2ccb5f640 1 -- 172.21.15.44:0/2908297586 wait complete. 2024-04-02T13:20:31.434 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:20:31.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.536+0000 7f81edc3a640 1 Processor -- start 2024-04-02T13:20:31.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.536+0000 7f81edc3a640 1 -- start start 2024-04-02T13:20:31.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.536+0000 7f81edc3a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f81e806daa0 0x7f81e80b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.536+0000 7f81edc3a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e80cfdc0 0x7f81e80d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.536+0000 7f81edc3a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f81e80d26f0 0x7f81e80d2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.536+0000 7f81edc3a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f81e80cf930 con 0x7f81e80d26f0 2024-04-02T13:20:31.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.536+0000 7f81edc3a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f81e806de80 con 0x7f81e806daa0 2024-04-02T13:20:31.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.537+0000 7f81edc3a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f81e80cf000 con 0x7f81e80cfdc0 2024-04-02T13:20:31.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.537+0000 7f81e7fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f81e80d26f0 0x7f81e80d2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.537+0000 7f81e7fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f81e80d26f0 0x7f81e80d2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33896/0 (socket says 172.21.15.44:33896) 2024-04-02T13:20:31.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.537+0000 7f81e7fff640 1 -- 172.21.15.44:0/2034202377 learned_addr learned my addr 172.21.15.44:0/2034202377 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:31.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.537+0000 7f81e77fe640 1 --2- 172.21.15.44:0/2034202377 >> v2:172.21.15.69:3300/0 conn(0x7f81e806daa0 0x7f81e80b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.537+0000 7f81e6ffd640 1 --2- 172.21.15.44:0/2034202377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e80cfdc0 0x7f81e80d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.537+0000 7f81e7fff640 1 -- 172.21.15.44:0/2034202377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e80cfdc0 msgr2=0x7f81e80d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:31.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.537+0000 7f81e7fff640 1 --2- 172.21.15.44:0/2034202377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e80cfdc0 0x7f81e80d11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.537+0000 7f81e7fff640 1 -- 172.21.15.44:0/2034202377 >> v2:172.21.15.69:3300/0 conn(0x7f81e806daa0 msgr2=0x7f81e80b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:31.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.537+0000 7f81e7fff640 1 --2- 172.21.15.44:0/2034202377 >> v2:172.21.15.69:3300/0 conn(0x7f81e806daa0 0x7f81e80b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.537+0000 7f81e7fff640 1 -- 172.21.15.44:0/2034202377 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e80cf2e0 con 0x7f81e80d26f0 2024-04-02T13:20:31.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.537+0000 7f81e6ffd640 1 --2- 172.21.15.44:0/2034202377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e80cfdc0 0x7f81e80d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:31.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.538+0000 7f81e7fff640 1 --2- 172.21.15.44:0/2034202377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e80d26f0 0x7f81e80d2ba0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f81e808d750 tx=0x7f81e002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67da4e1dbb155fcc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:31.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.538+0000 7f81e67fc640 1 -- 172.21.15.44:0/2034202377 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e002ca70 con 0x7f81e80d26f0 2024-04-02T13:20:31.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.538+0000 7f81e67fc640 1 -- 172.21.15.44:0/2034202377 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81e002cc10 con 0x7f81e80d26f0 2024-04-02T13:20:31.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.538+0000 7f81e67fc640 1 -- 172.21.15.44:0/2034202377 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e00343f0 con 0x7f81e80d26f0 2024-04-02T13:20:31.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.538+0000 7f81edc3a640 1 -- 172.21.15.44:0/2034202377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e80d26f0 msgr2=0x7f81e80d2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.538+0000 7f81edc3a640 1 --2- 172.21.15.44:0/2034202377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e80d26f0 0x7f81e80d2ba0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f81e808d750 tx=0x7f81e002bed0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.538+0000 7f81edc3a640 1 -- 172.21.15.44:0/2034202377 shutdown_connections 2024-04-02T13:20:31.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.538+0000 7f81edc3a640 1 --2- 172.21.15.44:0/2034202377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e80d26f0 0x7f81e80d2ba0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.538+0000 7f81edc3a640 1 --2- 172.21.15.44:0/2034202377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e80cfdc0 0x7f81e80d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.538+0000 7f81edc3a640 1 --2- 172.21.15.44:0/2034202377 >> v2:172.21.15.69:3300/0 conn(0x7f81e806daa0 0x7f81e80b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.538+0000 7f81edc3a640 1 -- 172.21.15.44:0/2034202377 >> 172.21.15.44:0/2034202377 conn(0x7f81e809d430 msgr2=0x7f81e80d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:31.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.538+0000 7f81edc3a640 1 -- 172.21.15.44:0/2034202377 shutdown_connections 2024-04-02T13:20:31.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.539+0000 7f81edc3a640 1 -- 172.21.15.44:0/2034202377 wait complete. 2024-04-02T13:20:31.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.540+0000 7f81edc3a640 1 Processor -- start 2024-04-02T13:20:31.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.540+0000 7f81edc3a640 1 -- start start 2024-04-02T13:20:31.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.540+0000 7f81edc3a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e806daa0 0x7f81e80cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.540+0000 7f81edc3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e80cfdc0 0x7f81e80cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.540+0000 7f81edc3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81e80d26f0 0x7f81e8150c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.540+0000 7f81edc3a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f81e80ceda0 con 0x7f81e80cfdc0 2024-04-02T13:20:31.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.540+0000 7f81edc3a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f81e80cec20 con 0x7f81e80d26f0 2024-04-02T13:20:31.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.540+0000 7f81edc3a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f81e815dd60 con 0x7f81e806daa0 2024-04-02T13:20:31.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81e6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e80cfdc0 0x7f81e80cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81e6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e80cfdc0 0x7f81e80cdb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33912/0 (socket says 172.21.15.44:33912) 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81e6ffd640 1 -- 172.21.15.44:0/2515409682 learned_addr learned my addr 172.21.15.44:0/2515409682 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81e77fe640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e806daa0 0x7f81e80cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81e7fff640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81e80d26f0 0x7f81e8150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81e6ffd640 1 -- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e806daa0 msgr2=0x7f81e80cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81e6ffd640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e806daa0 0x7f81e80cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81e6ffd640 1 -- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81e80d26f0 msgr2=0x7f81e8150c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81e6ffd640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81e80d26f0 0x7f81e8150c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81e6ffd640 1 -- 172.21.15.44:0/2515409682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e815b8c0 con 0x7f81e80cfdc0 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81e6ffd640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e80cfdc0 0x7f81e80cdb10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00e810 tx=0x7f81dc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81cffff640 1 -- 172.21.15.44:0/2515409682 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc00f040 con 0x7f81e80cfdc0 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81cffff640 1 -- 172.21.15.44:0/2515409682 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81dc004030 con 0x7f81e80cfdc0 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81edc3a640 1 -- 172.21.15.44:0/2515409682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e80ca080 con 0x7f81e80cfdc0 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81cffff640 1 -- 172.21.15.44:0/2515409682 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc004400 con 0x7f81e80cfdc0 2024-04-02T13:20:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.541+0000 7f81edc3a640 1 -- 172.21.15.44:0/2515409682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e80b6ed0 con 0x7f81e80cfdc0 2024-04-02T13:20:31.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.542+0000 7f81edc3a640 1 -- 172.21.15.44:0/2515409682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e80c9e10 con 0x7f81e80cfdc0 2024-04-02T13:20:31.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.543+0000 7f81cffff640 1 -- 172.21.15.44:0/2515409682 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f81dc00c6d0 con 0x7f81e80cfdc0 2024-04-02T13:20:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.543+0000 7f81cffff640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f81d406c8f0 0x7f81d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.543+0000 7f81cffff640 1 -- 172.21.15.44:0/2515409682 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f81dc08ba60 con 0x7f81e80cfdc0 2024-04-02T13:20:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.544+0000 7f81e77fe640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f81d406c8f0 0x7f81d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.544+0000 7f81e77fe640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f81d406c8f0 0x7f81d406edb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f81d8009ac0 tx=0x7f81d8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:31.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.548+0000 7f81cffff640 1 -- 172.21.15.44:0/2515409682 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f81dc017020 con 0x7f81e80cfdc0 2024-04-02T13:20:31.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.732+0000 7f81edc3a640 1 -- 172.21.15.44:0/2515409682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f81e80684f0 con 0x7f81e80cfdc0 2024-04-02T13:20:31.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.733+0000 7f81cffff640 1 -- 172.21.15.44:0/2515409682 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f81dc059500 con 0x7f81e80cfdc0 2024-04-02T13:20:31.735 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:20:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.735+0000 7f81edc3a640 1 -- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f81d406c8f0 msgr2=0x7f81d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.735+0000 7f81edc3a640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f81d406c8f0 0x7f81d406edb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f81d8009ac0 tx=0x7f81d8009190 comp rx=0 tx=0).stop 2024-04-02T13:20:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.735+0000 7f81edc3a640 1 -- 172.21.15.44:0/2515409682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e80cfdc0 msgr2=0x7f81e80cdb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.735+0000 7f81edc3a640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e80cfdc0 0x7f81e80cdb10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00e810 tx=0x7f81dc00ecd0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.735+0000 7f81edc3a640 1 -- 172.21.15.44:0/2515409682 shutdown_connections 2024-04-02T13:20:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.735+0000 7f81edc3a640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f81d406c8f0 0x7f81d406edb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.735+0000 7f81edc3a640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81e80d26f0 0x7f81e8150c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.735+0000 7f81edc3a640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e80cfdc0 0x7f81e80cdb10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.735+0000 7f81edc3a640 1 --2- 172.21.15.44:0/2515409682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e806daa0 0x7f81e80cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.735+0000 7f81edc3a640 1 -- 172.21.15.44:0/2515409682 >> 172.21.15.44:0/2515409682 conn(0x7f81e809d430 msgr2=0x7f81e80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.735+0000 7f81edc3a640 1 -- 172.21.15.44:0/2515409682 shutdown_connections 2024-04-02T13:20:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.735+0000 7f81edc3a640 1 -- 172.21.15.44:0/2515409682 wait complete. 2024-04-02T13:20:31.747 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:20:31.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.850+0000 7f8ca539f640 1 Processor -- start 2024-04-02T13:20:31.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.850+0000 7f8ca539f640 1 -- start start 2024-04-02T13:20:31.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.851+0000 7f8ca539f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8ca015e290 0x7f8ca00c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.851+0000 7f8ca539f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ca00d2270 0x7f8ca00b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.851+0000 7f8ca539f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ca00d09e0 con 0x7f8ca00b59c0 2024-04-02T13:20:31.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.851+0000 7f8ca539f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8ca015e670 con 0x7f8ca015e290 2024-04-02T13:20:31.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.851+0000 7f8ca539f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8ca00d2740 con 0x7f8ca00d2270 2024-04-02T13:20:31.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.851+0000 7f8c9ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8ca015e290 0x7f8ca00c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.851+0000 7f8ca4b9e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ca00d2270 0x7f8ca00b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.851+0000 7f8c9ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8ca015e290 0x7f8ca00c7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:57038/0 (socket says 172.21.15.44:57038) 2024-04-02T13:20:31.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.851+0000 7f8c9ffff640 1 -- 172.21.15.44:0/1499586742 learned_addr learned my addr 172.21.15.44:0/1499586742 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:31.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.851+0000 7f8c9f7fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8ca00b59c0 0x7f8ca00d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:33864/0 (socket says 172.21.15.44:33864) 2024-04-02T13:20:31.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.852+0000 7f8c9ffff640 1 -- 172.21.15.44:0/1499586742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ca00d2270 msgr2=0x7f8ca00b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:31.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.852+0000 7f8c9ffff640 1 --2- 172.21.15.44:0/1499586742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ca00d2270 0x7f8ca00b5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.852+0000 7f8c9ffff640 1 -- 172.21.15.44:0/1499586742 >> v1:172.21.15.69:6789/0 conn(0x7f8ca00b59c0 legacy=0x7f8ca00d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.852+0000 7f8c9ffff640 1 -- 172.21.15.44:0/1499586742 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca00d2a20 con 0x7f8ca015e290 2024-04-02T13:20:31.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.852+0000 7f8c9ffff640 1 --2- 172.21.15.44:0/1499586742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ca015e290 0x7f8ca00c7ae0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009870 tx=0x7f8c9002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d217aa24d706312 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:31.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.853+0000 7f8c9effd640 1 -- 172.21.15.44:0/1499586742 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90034070 con 0x7f8ca015e290 2024-04-02T13:20:31.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.853+0000 7f8c9effd640 1 -- 172.21.15.44:0/1499586742 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c90030840 con 0x7f8ca015e290 2024-04-02T13:20:31.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.853+0000 7f8c9effd640 1 -- 172.21.15.44:0/1499586742 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90030c10 con 0x7f8ca015e290 2024-04-02T13:20:31.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.853+0000 7f8ca539f640 1 -- 172.21.15.44:0/1499586742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ca015e290 msgr2=0x7f8ca00c7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.853+0000 7f8ca539f640 1 --2- 172.21.15.44:0/1499586742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ca015e290 0x7f8ca00c7ae0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009870 tx=0x7f8c9002b980 comp rx=0 tx=0).stop 2024-04-02T13:20:31.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.853+0000 7f8ca539f640 1 -- 172.21.15.44:0/1499586742 shutdown_connections 2024-04-02T13:20:31.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.853+0000 7f8ca539f640 1 --2- 172.21.15.44:0/1499586742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ca00d2270 0x7f8ca00b5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.853+0000 7f8ca539f640 1 --2- 172.21.15.44:0/1499586742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ca015e290 0x7f8ca00c7ae0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.853+0000 7f8ca539f640 1 -- 172.21.15.44:0/1499586742 >> 172.21.15.44:0/1499586742 conn(0x7f8ca0074b90 msgr2=0x7f8ca009d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:31.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.853+0000 7f8ca539f640 1 -- 172.21.15.44:0/1499586742 shutdown_connections 2024-04-02T13:20:31.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.853+0000 7f8ca539f640 1 -- 172.21.15.44:0/1499586742 wait complete. 2024-04-02T13:20:31.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.854+0000 7f8ca539f640 1 Processor -- start 2024-04-02T13:20:31.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.855+0000 7f8ca539f640 1 -- start start 2024-04-02T13:20:31.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.855+0000 7f8ca539f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ca00b59c0 0x7f8ca014c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.855+0000 7f8ca539f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ca00d2270 0x7f8ca015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.855+0000 7f8ca539f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ca015e290 0x7f8ca015be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.855+0000 7f8ca539f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca00ceaa0 con 0x7f8ca00b59c0 2024-04-02T13:20:31.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.855+0000 7f8ca539f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca00cee80 con 0x7f8ca015e290 2024-04-02T13:20:31.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.855+0000 7f8ca539f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8ca00cec20 con 0x7f8ca00d2270 2024-04-02T13:20:31.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.855+0000 7f8c9ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ca00b59c0 0x7f8ca014c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.855+0000 7f8c9ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ca00b59c0 0x7f8ca014c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33916/0 (socket says 172.21.15.44:33916) 2024-04-02T13:20:31.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.855+0000 7f8c9ffff640 1 -- 172.21.15.44:0/175007470 learned_addr learned my addr 172.21.15.44:0/175007470 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:31.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.855+0000 7f8ca4b9e640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ca015e290 0x7f8ca015be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.856+0000 7f8c9f7fe640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ca00d2270 0x7f8ca015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.856+0000 7f8c9ffff640 1 -- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ca00d2270 msgr2=0x7f8ca015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.856+0000 7f8c9ffff640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ca00d2270 0x7f8ca015b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.856+0000 7f8c9ffff640 1 -- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ca015e290 msgr2=0x7f8ca015be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:31.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.856+0000 7f8c9ffff640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ca015e290 0x7f8ca015be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:31.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.856+0000 7f8c9ffff640 1 -- 172.21.15.44:0/175007470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca00cd190 con 0x7f8ca00b59c0 2024-04-02T13:20:31.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.856+0000 7f8c9ffff640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ca00b59c0 0x7f8ca014c550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c9002b9c0 tx=0x7f8c900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:31.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.856+0000 7f8c9cff9640 1 -- 172.21.15.44:0/175007470 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90034040 con 0x7f8ca00b59c0 2024-04-02T13:20:31.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.856+0000 7f8c9cff9640 1 -- 172.21.15.44:0/175007470 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c90039d90 con 0x7f8ca00b59c0 2024-04-02T13:20:31.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.856+0000 7f8ca539f640 1 -- 172.21.15.44:0/175007470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ca00cd470 con 0x7f8ca00b59c0 2024-04-02T13:20:31.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.856+0000 7f8c9cff9640 1 -- 172.21.15.44:0/175007470 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90043630 con 0x7f8ca00b59c0 2024-04-02T13:20:31.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.856+0000 7f8ca539f640 1 -- 172.21.15.44:0/175007470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ca00cda30 con 0x7f8ca00b59c0 2024-04-02T13:20:31.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.857+0000 7f8ca539f640 1 -- 172.21.15.44:0/175007470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ca004f860 con 0x7f8ca00b59c0 2024-04-02T13:20:31.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.857+0000 7f8c9cff9640 1 -- 172.21.15.44:0/175007470 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8c9004a060 con 0x7f8ca00b59c0 2024-04-02T13:20:31.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.858+0000 7f8c9cff9640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8c8006c920 0x7f8c8006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:31.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.858+0000 7f8c9cff9640 1 -- 172.21.15.44:0/175007470 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8c900adc50 con 0x7f8ca00b59c0 2024-04-02T13:20:31.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.858+0000 7f8c9f7fe640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8c8006c920 0x7f8c8006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:31.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.858+0000 7f8c9f7fe640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8c8006c920 0x7f8c8006ede0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00d0e60 tx=0x7f8c88007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:31.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:31.862+0000 7f8c9cff9640 1 -- 172.21.15.44:0/175007470 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8c900b3020 con 0x7f8ca00b59c0 2024-04-02T13:20:32.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.047+0000 7f8ca539f640 1 -- 172.21.15.44:0/175007470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8ca015f4f0 con 0x7f8ca00b59c0 2024-04-02T13:20:32.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.047+0000 7f8c9cff9640 1 -- 172.21.15.44:0/175007470 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f8c9003e070 con 0x7f8ca00b59c0 2024-04-02T13:20:32.050 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:20:32.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.049+0000 7f8ca539f640 1 -- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8c8006c920 msgr2=0x7f8c8006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.049+0000 7f8ca539f640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8c8006c920 0x7f8c8006ede0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00d0e60 tx=0x7f8c88007e80 comp rx=0 tx=0).stop 2024-04-02T13:20:32.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.049+0000 7f8ca539f640 1 -- 172.21.15.44:0/175007470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ca00b59c0 msgr2=0x7f8ca014c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.049+0000 7f8ca539f640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ca00b59c0 0x7f8ca014c550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c9002b9c0 tx=0x7f8c900046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.049+0000 7f8ca539f640 1 -- 172.21.15.44:0/175007470 shutdown_connections 2024-04-02T13:20:32.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.049+0000 7f8ca539f640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8c8006c920 0x7f8c8006ede0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.049+0000 7f8ca539f640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ca015e290 0x7f8ca015be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.049+0000 7f8ca539f640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ca00d2270 0x7f8ca015b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.049+0000 7f8ca539f640 1 --2- 172.21.15.44:0/175007470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ca00b59c0 0x7f8ca014c550 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.049+0000 7f8ca539f640 1 -- 172.21.15.44:0/175007470 >> 172.21.15.44:0/175007470 conn(0x7f8ca0074b90 msgr2=0x7f8ca015fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.050+0000 7f8ca539f640 1 -- 172.21.15.44:0/175007470 shutdown_connections 2024-04-02T13:20:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.050+0000 7f8ca539f640 1 -- 172.21.15.44:0/175007470 wait complete. 2024-04-02T13:20:32.063 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:20:32.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.162+0000 7fd4b8faa640 1 Processor -- start 2024-04-02T13:20:32.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.163+0000 7fd4b8faa640 1 -- start start 2024-04-02T13:20:32.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.163+0000 7fd4b8faa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4b415e4d0 0x7fd4b406b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.163+0000 7fd4b8faa640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd4b4072f60 0x7fd4b406ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.163+0000 7fd4b8faa640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4b406dfe0 con 0x7fd4b415e0f0 2024-04-02T13:20:32.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.163+0000 7fd4b8faa640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd4b40aed70 con 0x7fd4b4072f60 2024-04-02T13:20:32.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.163+0000 7fd4b8faa640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd4b406e2c0 con 0x7fd4b415e4d0 2024-04-02T13:20:32.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.163+0000 7fd4b259c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd4b415e0f0 0x7fd4b40d00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41580/0 (socket says 172.21.15.44:41580) 2024-04-02T13:20:32.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.163+0000 7fd4b259c640 1 -- 172.21.15.44:0/590752082 learned_addr learned my addr 172.21.15.44:0/590752082 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:32.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.164+0000 7fd4abfff640 1 --2- 172.21.15.44:0/590752082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4b415e4d0 0x7fd4b406b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.164+0000 7fd4b2d9d640 1 --2- 172.21.15.44:0/590752082 >> v2:172.21.15.69:3300/0 conn(0x7fd4b4072f60 0x7fd4b406ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.164+0000 7fd4b1d9b640 1 -- 172.21.15.44:0/590752082 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 919846454 0 0) 0x7fd4b406dfe0 con 0x7fd4b415e0f0 2024-04-02T13:20:32.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.164+0000 7fd4b1d9b640 1 -- 172.21.15.44:0/590752082 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4a0003520 con 0x7fd4b415e0f0 2024-04-02T13:20:32.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.164+0000 7fd4abfff640 1 -- 172.21.15.44:0/590752082 >> v2:172.21.15.69:3300/0 conn(0x7fd4b4072f60 msgr2=0x7fd4b406ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:32.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.164+0000 7fd4b1d9b640 1 -- 172.21.15.44:0/590752082 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4221963675 0 0) 0x7fd4a0003520 con 0x7fd4b415e0f0 2024-04-02T13:20:32.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.164+0000 7fd4abfff640 1 --2- 172.21.15.44:0/590752082 >> v2:172.21.15.69:3300/0 conn(0x7fd4b4072f60 0x7fd4b406ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.164+0000 7fd4abfff640 1 -- 172.21.15.44:0/590752082 >> v1:172.21.15.44:6789/0 conn(0x7fd4b415e0f0 legacy=0x7fd4b40d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.164+0000 7fd4abfff640 1 -- 172.21.15.44:0/590752082 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b406e5a0 con 0x7fd4b415e4d0 2024-04-02T13:20:32.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.164+0000 7fd4b2d9d640 1 --2- 172.21.15.44:0/590752082 >> v2:172.21.15.69:3300/0 conn(0x7fd4b4072f60 0x7fd4b406ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:32.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.165+0000 7fd4abfff640 1 --2- 172.21.15.44:0/590752082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4b415e4d0 0x7fd4b406b4d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0097a0 tx=0x7fd49c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b60942df5bf0f17 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:32.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.165+0000 7fd4b1d9b640 1 -- 172.21.15.44:0/590752082 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49c038070 con 0x7fd4b415e4d0 2024-04-02T13:20:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.165+0000 7fd4b1d9b640 1 -- 172.21.15.44:0/590752082 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd49c031500 con 0x7fd4b415e4d0 2024-04-02T13:20:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.165+0000 7fd4b1d9b640 1 -- 172.21.15.44:0/590752082 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4001670 con 0x7fd4b415e4d0 2024-04-02T13:20:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.165+0000 7fd4b8faa640 1 -- 172.21.15.44:0/590752082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4b415e4d0 msgr2=0x7fd4b406b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.165+0000 7fd4b8faa640 1 --2- 172.21.15.44:0/590752082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4b415e4d0 0x7fd4b406b4d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0097a0 tx=0x7fd49c02b820 comp rx=0 tx=0).stop 2024-04-02T13:20:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.166+0000 7fd4b8faa640 1 -- 172.21.15.44:0/590752082 shutdown_connections 2024-04-02T13:20:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.166+0000 7fd4b8faa640 1 --2- 172.21.15.44:0/590752082 >> v2:172.21.15.69:3300/0 conn(0x7fd4b4072f60 0x7fd4b406ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.166+0000 7fd4b8faa640 1 --2- 172.21.15.44:0/590752082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4b415e4d0 0x7fd4b406b4d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.166+0000 7fd4b8faa640 1 -- 172.21.15.44:0/590752082 >> 172.21.15.44:0/590752082 conn(0x7fd4b40b2980 msgr2=0x7fd4b40b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.166+0000 7fd4b8faa640 1 -- 172.21.15.44:0/590752082 shutdown_connections 2024-04-02T13:20:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.166+0000 7fd4b8faa640 1 -- 172.21.15.44:0/590752082 wait complete. 2024-04-02T13:20:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.167+0000 7fd4b8faa640 1 Processor -- start 2024-04-02T13:20:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.167+0000 7fd4b8faa640 1 -- start start 2024-04-02T13:20:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.167+0000 7fd4b8faa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b4072f60 0x7fd4b40c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b8faa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4b415e0f0 0x7fd4b40cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b8faa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4b415e4d0 0x7fd4b415abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b8faa640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b40d0800 con 0x7fd4b4072f60 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b8faa640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b40d0af0 con 0x7fd4b415e0f0 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b8faa640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd4b406eda0 con 0x7fd4b415e4d0 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b4072f60 0x7fd4b40c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b4072f60 0x7fd4b40c5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33930/0 (socket says 172.21.15.44:33930) 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b259c640 1 -- 172.21.15.44:0/3127914721 learned_addr learned my addr 172.21.15.44:0/3127914721 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4abfff640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4b415e0f0 0x7fd4b40cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b2d9d640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4b415e4d0 0x7fd4b415abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b259c640 1 -- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4b415e4d0 msgr2=0x7fd4b415abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b259c640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4b415e4d0 0x7fd4b415abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b259c640 1 -- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4b415e0f0 msgr2=0x7fd4b40cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b259c640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4b415e0f0 0x7fd4b40cdf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b259c640 1 -- 172.21.15.44:0/3127914721 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b40ca7f0 con 0x7fd4b4072f60 2024-04-02T13:20:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4abfff640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4b415e0f0 0x7fd4b40cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b259c640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b4072f60 0x7fd4b40c5fa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400c750 tx=0x7fd4a400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4aaffd640 1 -- 172.21.15.44:0/3127914721 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4005ad0 con 0x7fd4b4072f60 2024-04-02T13:20:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4aaffd640 1 -- 172.21.15.44:0/3127914721 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4a4005c90 con 0x7fd4b4072f60 2024-04-02T13:20:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4aaffd640 1 -- 172.21.15.44:0/3127914721 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a40072b0 con 0x7fd4b4072f60 2024-04-02T13:20:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b8faa640 1 -- 172.21.15.44:0/3127914721 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4b40cb940 con 0x7fd4b4072f60 2024-04-02T13:20:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.168+0000 7fd4b8faa640 1 -- 172.21.15.44:0/3127914721 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4b40ce4a0 con 0x7fd4b4072f60 2024-04-02T13:20:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.169+0000 7fd4b8faa640 1 -- 172.21.15.44:0/3127914721 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4b40684f0 con 0x7fd4b4072f60 2024-04-02T13:20:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.170+0000 7fd4aaffd640 1 -- 172.21.15.44:0/3127914721 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd4a4007450 con 0x7fd4b4072f60 2024-04-02T13:20:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.170+0000 7fd4aaffd640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd49406c8f0 0x7fd49406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.170+0000 7fd4aaffd640 1 -- 172.21.15.44:0/3127914721 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd4a408bcb0 con 0x7fd4b4072f60 2024-04-02T13:20:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.170+0000 7fd4abfff640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd49406c8f0 0x7fd49406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.171+0000 7fd4abfff640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd49406c8f0 0x7fd49406edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd49c035000 tx=0x7fd49c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:32.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.175+0000 7fd4aaffd640 1 -- 172.21.15.44:0/3127914721 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd4a4059e30 con 0x7fd4b4072f60 2024-04-02T13:20:32.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.370+0000 7fd4b8faa640 1 -- 172.21.15.44:0/3127914721 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd4b40727f0 con 0x7fd4b4072f60 2024-04-02T13:20:32.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.371+0000 7fd4aaffd640 1 -- 172.21.15.44:0/3127914721 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7fd4a4059730 con 0x7fd4b4072f60 2024-04-02T13:20:32.373 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:20:32.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.372+0000 7fd4b8faa640 1 -- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd49406c8f0 msgr2=0x7fd49406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.372+0000 7fd4b8faa640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd49406c8f0 0x7fd49406edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd49c035000 tx=0x7fd49c008030 comp rx=0 tx=0).stop 2024-04-02T13:20:32.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.372+0000 7fd4b8faa640 1 -- 172.21.15.44:0/3127914721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b4072f60 msgr2=0x7fd4b40c5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.372+0000 7fd4b8faa640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b4072f60 0x7fd4b40c5fa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400c750 tx=0x7fd4a400cc10 comp rx=0 tx=0).stop 2024-04-02T13:20:32.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.372+0000 7fd4b8faa640 1 -- 172.21.15.44:0/3127914721 shutdown_connections 2024-04-02T13:20:32.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.372+0000 7fd4b8faa640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd49406c8f0 0x7fd49406edb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.372+0000 7fd4b8faa640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4b415e4d0 0x7fd4b415abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.372+0000 7fd4b8faa640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4b415e0f0 0x7fd4b40cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.372+0000 7fd4b8faa640 1 --2- 172.21.15.44:0/3127914721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b4072f60 0x7fd4b40c5fa0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.372+0000 7fd4b8faa640 1 -- 172.21.15.44:0/3127914721 >> 172.21.15.44:0/3127914721 conn(0x7fd4b40b2980 msgr2=0x7fd4b40b0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:32.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.373+0000 7fd4b8faa640 1 -- 172.21.15.44:0/3127914721 shutdown_connections 2024-04-02T13:20:32.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.373+0000 7fd4b8faa640 1 -- 172.21.15.44:0/3127914721 wait complete. 2024-04-02T13:20:32.385 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:20:32.385 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:20:32.400 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:32.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.539+0000 7f45e834d640 1 Processor -- start 2024-04-02T13:20:32.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.539+0000 7f45e834d640 1 -- start start 2024-04-02T13:20:32.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.540+0000 7f45e834d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f45e015c130 0x7f45e00d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.540+0000 7f45e834d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45e00d2280 0x7f45e00ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.540+0000 7f45e834d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45e00d2700 con 0x7f45e00cd0b0 2024-04-02T13:20:32.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.540+0000 7f45e834d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f45e00cd8e0 con 0x7f45e015c130 2024-04-02T13:20:32.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.540+0000 7f45e834d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f45e00cdbc0 con 0x7f45e00d2280 2024-04-02T13:20:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.540+0000 7f45e68ea640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f45e00cd0b0 0x7f45e00ce7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41584/0 (socket says 172.21.15.44:41584) 2024-04-02T13:20:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.540+0000 7f45e68ea640 1 -- 172.21.15.44:0/1442052335 learned_addr learned my addr 172.21.15.44:0/1442052335 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.541+0000 7f45e60e9640 1 --2- 172.21.15.44:0/1442052335 >> v2:172.21.15.69:3300/0 conn(0x7f45e015c130 0x7f45e00d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.541+0000 7f45e58e8640 1 --2- 172.21.15.44:0/1442052335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45e00d2280 0x7f45e00ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.541+0000 7f45e50e7640 1 -- 172.21.15.44:0/1442052335 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3818718943 0 0) 0x7f45e00d2700 con 0x7f45e00cd0b0 2024-04-02T13:20:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.541+0000 7f45e50e7640 1 -- 172.21.15.44:0/1442052335 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45cc003520 con 0x7f45e00cd0b0 2024-04-02T13:20:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.541+0000 7f45e60e9640 1 -- 172.21.15.44:0/1442052335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45e00d2280 msgr2=0x7f45e00ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.541+0000 7f45e60e9640 1 --2- 172.21.15.44:0/1442052335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45e00d2280 0x7f45e00ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.541+0000 7f45e60e9640 1 -- 172.21.15.44:0/1442052335 >> v1:172.21.15.44:6789/0 conn(0x7f45e00cd0b0 legacy=0x7f45e00ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.541+0000 7f45e60e9640 1 -- 172.21.15.44:0/1442052335 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e00b1d30 con 0x7f45e015c130 2024-04-02T13:20:32.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.542+0000 7f45e60e9640 1 --2- 172.21.15.44:0/1442052335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45e015c130 0x7f45e00d1d40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0097a0 tx=0x7f45dc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b18fac77d8de7cc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:32.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.542+0000 7f45e50e7640 1 -- 172.21.15.44:0/1442052335 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45dc035020 con 0x7f45e015c130 2024-04-02T13:20:32.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.542+0000 7f45e50e7640 1 -- 172.21.15.44:0/1442052335 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45dc031830 con 0x7f45e015c130 2024-04-02T13:20:32.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.542+0000 7f45e50e7640 1 -- 172.21.15.44:0/1442052335 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45dc031c00 con 0x7f45e015c130 2024-04-02T13:20:32.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.542+0000 7f45e834d640 1 -- 172.21.15.44:0/1442052335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45e015c130 msgr2=0x7f45e00d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.542+0000 7f45e834d640 1 --2- 172.21.15.44:0/1442052335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45e015c130 0x7f45e00d1d40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0097a0 tx=0x7f45dc02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.543+0000 7f45e834d640 1 -- 172.21.15.44:0/1442052335 shutdown_connections 2024-04-02T13:20:32.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.543+0000 7f45e834d640 1 --2- 172.21.15.44:0/1442052335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45e00d2280 0x7f45e00ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.543+0000 7f45e834d640 1 --2- 172.21.15.44:0/1442052335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45e015c130 0x7f45e00d1d40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.543+0000 7f45e834d640 1 -- 172.21.15.44:0/1442052335 >> 172.21.15.44:0/1442052335 conn(0x7f45e009aa20 msgr2=0x7f45e009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:32.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.543+0000 7f45e834d640 1 -- 172.21.15.44:0/1442052335 shutdown_connections 2024-04-02T13:20:32.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.543+0000 7f45e834d640 1 -- 172.21.15.44:0/1442052335 wait complete. 2024-04-02T13:20:32.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.544+0000 7f45e834d640 1 Processor -- start 2024-04-02T13:20:32.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.544+0000 7f45e834d640 1 -- start start 2024-04-02T13:20:32.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.544+0000 7f45e834d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45e00cd0b0 0x7f45e009bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.544+0000 7f45e834d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45e00d2280 0x7f45e00c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.544+0000 7f45e834d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45e015c130 0x7f45e00b68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.544+0000 7f45e834d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f45e00cd540 con 0x7f45e015c130 2024-04-02T13:20:32.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.544+0000 7f45e834d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f45e00b19f0 con 0x7f45e00cd0b0 2024-04-02T13:20:32.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.544+0000 7f45e834d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f45e00cd6c0 con 0x7f45e00d2280 2024-04-02T13:20:32.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.544+0000 7f45e68ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45e015c130 0x7f45e00b68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.544+0000 7f45e68ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45e015c130 0x7f45e00b68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33936/0 (socket says 172.21.15.44:33936) 2024-04-02T13:20:32.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45e68ea640 1 -- 172.21.15.44:0/4028024682 learned_addr learned my addr 172.21.15.44:0/4028024682 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:32.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45e60e9640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45e00cd0b0 0x7f45e009bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45e58e8640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45e00d2280 0x7f45e00c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45e68ea640 1 -- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45e00d2280 msgr2=0x7f45e00c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45e68ea640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45e00d2280 0x7f45e00c9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45e68ea640 1 -- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45e00cd0b0 msgr2=0x7f45e009bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45e68ea640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45e00cd0b0 0x7f45e009bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45e68ea640 1 -- 172.21.15.44:0/4028024682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e00b7290 con 0x7f45e015c130 2024-04-02T13:20:32.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45e68ea640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45e015c130 0x7f45e00b68b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f45d800cb40 tx=0x7f45d800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:32.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45d6ffd640 1 -- 172.21.15.44:0/4028024682 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8017070 con 0x7f45e015c130 2024-04-02T13:20:32.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45d6ffd640 1 -- 172.21.15.44:0/4028024682 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45d800aca0 con 0x7f45e015c130 2024-04-02T13:20:32.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45e834d640 1 -- 172.21.15.44:0/4028024682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e015d950 con 0x7f45e015c130 2024-04-02T13:20:32.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45d6ffd640 1 -- 172.21.15.44:0/4028024682 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8012620 con 0x7f45e015c130 2024-04-02T13:20:32.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.545+0000 7f45e834d640 1 -- 172.21.15.44:0/4028024682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e015df10 con 0x7f45e015c130 2024-04-02T13:20:32.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.546+0000 7f45e834d640 1 -- 172.21.15.44:0/4028024682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45e0068c00 con 0x7f45e015c130 2024-04-02T13:20:32.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.547+0000 7f45d6ffd640 1 -- 172.21.15.44:0/4028024682 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f45d801a070 con 0x7f45e015c130 2024-04-02T13:20:32.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.547+0000 7f45d6ffd640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f45c406c8f0 0x7f45c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.547+0000 7f45d6ffd640 1 -- 172.21.15.44:0/4028024682 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f45d808bc60 con 0x7f45e015c130 2024-04-02T13:20:32.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.547+0000 7f45e60e9640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f45c406c8f0 0x7f45c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.548+0000 7f45e60e9640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f45c406c8f0 0x7f45c406edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f45dc036000 tx=0x7f45dc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:32.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.551+0000 7f45d6ffd640 1 -- 172.21.15.44:0/4028024682 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f45d8059de0 con 0x7f45e015c130 2024-04-02T13:20:32.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.738+0000 7f45e834d640 1 -- 172.21.15.44:0/4028024682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45e00c8a50 con 0x7f45e015c130 2024-04-02T13:20:32.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.739+0000 7f45d6ffd640 1 -- 172.21.15.44:0/4028024682 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f45d80596e0 con 0x7f45e015c130 2024-04-02T13:20:32.742 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:32.742 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:32.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.741+0000 7f45e834d640 1 -- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f45c406c8f0 msgr2=0x7f45c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.741+0000 7f45e834d640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f45c406c8f0 0x7f45c406edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f45dc036000 tx=0x7f45dc03c040 comp rx=0 tx=0).stop 2024-04-02T13:20:32.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.741+0000 7f45e834d640 1 -- 172.21.15.44:0/4028024682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45e015c130 msgr2=0x7f45e00b68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.741+0000 7f45e834d640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45e015c130 0x7f45e00b68b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f45d800cb40 tx=0x7f45d800a060 comp rx=0 tx=0).stop 2024-04-02T13:20:32.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.741+0000 7f45e834d640 1 -- 172.21.15.44:0/4028024682 shutdown_connections 2024-04-02T13:20:32.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.741+0000 7f45e834d640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f45c406c8f0 0x7f45c406edb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.741+0000 7f45e834d640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45e015c130 0x7f45e00b68b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.741+0000 7f45e834d640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45e00d2280 0x7f45e00c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.741+0000 7f45e834d640 1 --2- 172.21.15.44:0/4028024682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45e00cd0b0 0x7f45e009bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.741+0000 7f45e834d640 1 -- 172.21.15.44:0/4028024682 >> 172.21.15.44:0/4028024682 conn(0x7f45e009aa20 msgr2=0x7f45e00b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:32.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.742+0000 7f45e834d640 1 -- 172.21.15.44:0/4028024682 shutdown_connections 2024-04-02T13:20:32.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.742+0000 7f45e834d640 1 -- 172.21.15.44:0/4028024682 wait complete. 2024-04-02T13:20:32.754 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:20:32.755 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:20:32.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.856+0000 7f944abba640 1 Processor -- start 2024-04-02T13:20:32.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.857+0000 7f944abba640 1 -- start start 2024-04-02T13:20:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.857+0000 7f944abba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f94440d1070 0x7f944415e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.857+0000 7f944abba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94440b3aa0 con 0x7f94440d2990 2024-04-02T13:20:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.857+0000 7f944abba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94440d2f50 con 0x7f94440d0c90 2024-04-02T13:20:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.857+0000 7f944abba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f94440b3d40 con 0x7f94440d1070 2024-04-02T13:20:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.857+0000 7f9449157640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f94440d2990 0x7f94440b4540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41596/0 (socket says 172.21.15.44:41596) 2024-04-02T13:20:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.857+0000 7f9449157640 1 -- 172.21.15.44:0/466403912 learned_addr learned my addr 172.21.15.44:0/466403912 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.858+0000 7f9443fff640 1 --2- 172.21.15.44:0/466403912 >> v2:172.21.15.69:3300/0 conn(0x7f94440d1070 0x7f944415e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.858+0000 7f94437fe640 1 -- 172.21.15.44:0/466403912 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3333556441 0 0) 0x7f94440b3aa0 con 0x7f94440d2990 2024-04-02T13:20:32.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.858+0000 7f94437fe640 1 -- 172.21.15.44:0/466403912 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9434003520 con 0x7f94440d2990 2024-04-02T13:20:32.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.858+0000 7f94437fe640 1 -- 172.21.15.44:0/466403912 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 979336487 0 0) 0x7f94440d2f50 con 0x7f94440d0c90 2024-04-02T13:20:32.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.858+0000 7f94437fe640 1 -- 172.21.15.44:0/466403912 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94440b3aa0 con 0x7f94440d0c90 2024-04-02T13:20:32.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.858+0000 7f9443fff640 1 -- 172.21.15.44:0/466403912 >> v1:172.21.15.69:6789/0 conn(0x7f94440d0c90 legacy=0x7f94440d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.858+0000 7f9443fff640 1 -- 172.21.15.44:0/466403912 >> v1:172.21.15.44:6789/0 conn(0x7f94440d2990 legacy=0x7f94440b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.858+0000 7f9443fff640 1 -- 172.21.15.44:0/466403912 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94440b4020 con 0x7f94440d1070 2024-04-02T13:20:32.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.859+0000 7f9443fff640 1 --2- 172.21.15.44:0/466403912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94440d1070 0x7f944415e1b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f94300097a0 tx=0x7f943002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=245441061da1864c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:32.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.859+0000 7f94437fe640 1 -- 172.21.15.44:0/466403912 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9430035020 con 0x7f94440d1070 2024-04-02T13:20:32.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.859+0000 7f94437fe640 1 -- 172.21.15.44:0/466403912 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9430031830 con 0x7f94440d1070 2024-04-02T13:20:32.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.859+0000 7f94437fe640 1 -- 172.21.15.44:0/466403912 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9430031be0 con 0x7f94440d1070 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.859+0000 7f944abba640 1 -- 172.21.15.44:0/466403912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94440d1070 msgr2=0x7f944415e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.859+0000 7f944abba640 1 --2- 172.21.15.44:0/466403912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94440d1070 0x7f944415e1b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f94300097a0 tx=0x7f943002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.859+0000 7f944abba640 1 -- 172.21.15.44:0/466403912 shutdown_connections 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.859+0000 7f944abba640 1 --2- 172.21.15.44:0/466403912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94440d1070 0x7f944415e1b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.859+0000 7f944abba640 1 -- 172.21.15.44:0/466403912 >> 172.21.15.44:0/466403912 conn(0x7f944409d3d0 msgr2=0x7f94440d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.859+0000 7f944abba640 1 -- 172.21.15.44:0/466403912 shutdown_connections 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.859+0000 7f944abba640 1 -- 172.21.15.44:0/466403912 wait complete. 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.861+0000 7f944abba640 1 Processor -- start 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.861+0000 7f944abba640 1 -- start start 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.861+0000 7f944abba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94440d0c90 0x7f9444149b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.861+0000 7f944abba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94440d1070 0x7f94440cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.861+0000 7f944abba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94440d2990 0x7f94440cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.861+0000 7f944abba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f944415e870 con 0x7f94440d2990 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.861+0000 7f944abba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f944415e6f0 con 0x7f94440d1070 2024-04-02T13:20:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.861+0000 7f944abba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f944415e9f0 con 0x7f94440d0c90 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f9449157640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94440d2990 0x7f94440cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f9449157640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94440d2990 0x7f94440cb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33942/0 (socket says 172.21.15.44:33942) 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f9449157640 1 -- 172.21.15.44:0/3626183512 learned_addr learned my addr 172.21.15.44:0/3626183512 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f9443fff640 1 --2- 172.21.15.44:0/3626183512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94440d1070 0x7f94440cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f9449157640 1 -- 172.21.15.44:0/3626183512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94440d0c90 msgr2=0x7f9444149b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f9449157640 1 --2- 172.21.15.44:0/3626183512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94440d0c90 0x7f9444149b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f9449157640 1 -- 172.21.15.44:0/3626183512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94440d1070 msgr2=0x7f94440cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f9449157640 1 --2- 172.21.15.44:0/3626183512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94440d1070 0x7f94440cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f9449157640 1 -- 172.21.15.44:0/3626183512 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94441f8360 con 0x7f94440d2990 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f9449157640 1 --2- 172.21.15.44:0/3626183512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94440d2990 0x7f94440cb6b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f943c00cb10 tx=0x7f943c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f94417fa640 1 -- 172.21.15.44:0/3626183512 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943c017070 con 0x7f94440d2990 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f94417fa640 1 -- 172.21.15.44:0/3626183512 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f943c00ac70 con 0x7f94440d2990 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f944abba640 1 -- 172.21.15.44:0/3626183512 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94441f8640 con 0x7f94440d2990 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f94417fa640 1 -- 172.21.15.44:0/3626183512 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943c00ae60 con 0x7f94440d2990 2024-04-02T13:20:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.862+0000 7f944abba640 1 -- 172.21.15.44:0/3626183512 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94440b8060 con 0x7f94440d2990 2024-04-02T13:20:32.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.863+0000 7f944abba640 1 -- 172.21.15.44:0/3626183512 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9444068c00 con 0x7f94440d2990 2024-04-02T13:20:32.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.864+0000 7f94417fa640 1 -- 172.21.15.44:0/3626183512 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f943c01a070 con 0x7f94440d2990 2024-04-02T13:20:32.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.864+0000 7f94417fa640 1 --2- 172.21.15.44:0/3626183512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f942806c8f0 0x7f942806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:32.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.864+0000 7f94417fa640 1 -- 172.21.15.44:0/3626183512 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f943c08bb20 con 0x7f94440d2990 2024-04-02T13:20:32.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.864+0000 7f9448956640 1 --2- 172.21.15.44:0/3626183512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f942806c8f0 0x7f942806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:32.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.865+0000 7f9448956640 1 --2- 172.21.15.44:0/3626183512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f942806c8f0 0x7f942806edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f94380011e0 tx=0x7f94380078f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:32.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:32.869+0000 7f94417fa640 1 -- 172.21.15.44:0/3626183512 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f943c0595c0 con 0x7f94440d2990 2024-04-02T13:20:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.057+0000 7f944abba640 1 -- 172.21.15.44:0/3626183512 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f944415dbd0 con 0x7f94440d2990 2024-04-02T13:20:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.057+0000 7f94417fa640 1 -- 172.21.15.44:0/3626183512 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f943c05d8c0 con 0x7f94440d2990 2024-04-02T13:20:33.059 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:20:33.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.059+0000 7f944abba640 1 -- 172.21.15.44:0/3626183512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f942806c8f0 msgr2=0x7f942806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.059+0000 7f944abba640 1 --2- 172.21.15.44:0/3626183512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f942806c8f0 0x7f942806edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f94380011e0 tx=0x7f94380078f0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.059+0000 7f944abba640 1 -- 172.21.15.44:0/3626183512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94440d2990 msgr2=0x7f94440cb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.059+0000 7f944abba640 1 --2- 172.21.15.44:0/3626183512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94440d2990 0x7f94440cb6b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f943c00cb10 tx=0x7f943c00a060 comp rx=0 tx=0).stop 2024-04-02T13:20:33.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.059+0000 7f944abba640 1 -- 172.21.15.44:0/3626183512 shutdown_connections 2024-04-02T13:20:33.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.059+0000 7f944abba640 1 --2- 172.21.15.44:0/3626183512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f942806c8f0 0x7f942806edb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.059+0000 7f944abba640 1 --2- 172.21.15.44:0/3626183512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94440d2990 0x7f94440cb6b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.059+0000 7f944abba640 1 --2- 172.21.15.44:0/3626183512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94440d1070 0x7f94440cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.059+0000 7f944abba640 1 --2- 172.21.15.44:0/3626183512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94440d0c90 0x7f9444149b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.059+0000 7f944abba640 1 -- 172.21.15.44:0/3626183512 >> 172.21.15.44:0/3626183512 conn(0x7f944409d3d0 msgr2=0x7f944406d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:33.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.059+0000 7f944abba640 1 -- 172.21.15.44:0/3626183512 shutdown_connections 2024-04-02T13:20:33.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.059+0000 7f944abba640 1 -- 172.21.15.44:0/3626183512 wait complete. 2024-04-02T13:20:33.072 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:20:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.174+0000 7f1462e55640 1 Processor -- start 2024-04-02T13:20:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.174+0000 7f1462e55640 1 -- start start 2024-04-02T13:20:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f1462e55640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f145c15e260 0x7f145c0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f1462e55640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f145c0cec60 0x7f145c168280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f1462e55640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f145c0cf120 con 0x7f145c15e640 2024-04-02T13:20:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f1462e55640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f145c0b5a50 con 0x7f145c0cec60 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f1462e55640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f145c0d01c0 con 0x7f145c15e260 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f14613f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f145c0cec60 0x7f145c168280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f145bfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f145c15e640 0x7f145c0d0ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41604/0 (socket says 172.21.15.44:41604) 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f14613f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f145c0cec60 0x7f145c168280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33948/0 (socket says 172.21.15.44:33948) 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f145bfff640 1 -- 172.21.15.44:0/3469936954 learned_addr learned my addr 172.21.15.44:0/3469936954 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f1460bf1640 1 --2- 172.21.15.44:0/3469936954 >> v2:172.21.15.69:3300/0 conn(0x7f145c15e260 0x7f145c0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f14613f2640 1 -- 172.21.15.44:0/3469936954 >> v2:172.21.15.69:3300/0 conn(0x7f145c15e260 msgr2=0x7f145c0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f14613f2640 1 --2- 172.21.15.44:0/3469936954 >> v2:172.21.15.69:3300/0 conn(0x7f145c15e260 0x7f145c0d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f14613f2640 1 -- 172.21.15.44:0/3469936954 >> v1:172.21.15.44:6789/0 conn(0x7f145c15e640 legacy=0x7f145c0d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f14613f2640 1 -- 172.21.15.44:0/3469936954 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145c0cf400 con 0x7f145c0cec60 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.175+0000 7f145b7fe640 1 -- 172.21.15.44:0/3469936954 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1480515949 0 0) 0x7f145c0cf120 con 0x7f145c15e640 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.176+0000 7f14613f2640 1 --2- 172.21.15.44:0/3469936954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f145c0cec60 0x7f145c168280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f145c09d8f0 tx=0x7f144c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72dc4f76f05a9b76 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.176+0000 7f145b7fe640 1 -- 172.21.15.44:0/3469936954 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c0315a0 con 0x7f145c0cec60 2024-04-02T13:20:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.176+0000 7f145b7fe640 1 -- 172.21.15.44:0/3469936954 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f144c037070 con 0x7f145c0cec60 2024-04-02T13:20:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.176+0000 7f145b7fe640 1 -- 172.21.15.44:0/3469936954 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c031960 con 0x7f145c0cec60 2024-04-02T13:20:33.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.177+0000 7f1462e55640 1 -- 172.21.15.44:0/3469936954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f145c0cec60 msgr2=0x7f145c168280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.177+0000 7f1462e55640 1 --2- 172.21.15.44:0/3469936954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f145c0cec60 0x7f145c168280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f145c09d8f0 tx=0x7f144c02b840 comp rx=0 tx=0).stop 2024-04-02T13:20:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.177+0000 7f1462e55640 1 -- 172.21.15.44:0/3469936954 shutdown_connections 2024-04-02T13:20:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.177+0000 7f1462e55640 1 --2- 172.21.15.44:0/3469936954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f145c0cec60 0x7f145c168280 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.177+0000 7f1462e55640 1 --2- 172.21.15.44:0/3469936954 >> v2:172.21.15.69:3300/0 conn(0x7f145c15e260 0x7f145c0d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.177+0000 7f1462e55640 1 -- 172.21.15.44:0/3469936954 >> 172.21.15.44:0/3469936954 conn(0x7f145c09d3c0 msgr2=0x7f145c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.177+0000 7f1462e55640 1 -- 172.21.15.44:0/3469936954 shutdown_connections 2024-04-02T13:20:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.177+0000 7f1462e55640 1 -- 172.21.15.44:0/3469936954 wait complete. 2024-04-02T13:20:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.178+0000 7f1462e55640 1 Processor -- start 2024-04-02T13:20:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.178+0000 7f1462e55640 1 -- start start 2024-04-02T13:20:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f1462e55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f145c0cec60 0x7f145c0cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f1462e55640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f145c15e260 0x7f145c0cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f1462e55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15e640 0x7f145c151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f1462e55640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f145c0cfca0 con 0x7f145c15e260 2024-04-02T13:20:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f1462e55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f145c0cf9c0 con 0x7f145c15e640 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f1462e55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f145c0d11e0 con 0x7f145c0cec60 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f145bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f145c15e260 0x7f145c0cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f145bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f145c15e260 0x7f145c0cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33952/0 (socket says 172.21.15.44:33952) 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f145bfff640 1 -- 172.21.15.44:0/2316661591 learned_addr learned my addr 172.21.15.44:0/2316661591 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f1460bf1640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f145c0cec60 0x7f145c0cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f14613f2640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15e640 0x7f145c151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f145bfff640 1 -- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f145c0cec60 msgr2=0x7f145c0cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f145bfff640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f145c0cec60 0x7f145c0cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f145bfff640 1 -- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15e640 msgr2=0x7f145c151270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f145bfff640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15e640 0x7f145c151270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f145bfff640 1 -- 172.21.15.44:0/2316661591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145c0cc1a0 con 0x7f145c15e260 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f1460bf1640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f145c0cec60 0x7f145c0cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f14613f2640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15e640 0x7f145c151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.179+0000 7f145bfff640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f145c15e260 0x7f145c0cda10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f145000c410 tx=0x7f145000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:33.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.180+0000 7f14597fa640 1 -- 172.21.15.44:0/2316661591 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14500074e0 con 0x7f145c15e260 2024-04-02T13:20:33.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.180+0000 7f14597fa640 1 -- 172.21.15.44:0/2316661591 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14500047e0 con 0x7f145c15e260 2024-04-02T13:20:33.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.180+0000 7f1462e55640 1 -- 172.21.15.44:0/2316661591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f145c0cc480 con 0x7f145c15e260 2024-04-02T13:20:33.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.180+0000 7f14597fa640 1 -- 172.21.15.44:0/2316661591 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1450004bd0 con 0x7f145c15e260 2024-04-02T13:20:33.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.180+0000 7f1462e55640 1 -- 172.21.15.44:0/2316661591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f145c0cca40 con 0x7f145c15e260 2024-04-02T13:20:33.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.180+0000 7f1462e55640 1 -- 172.21.15.44:0/2316661591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f145c0684f0 con 0x7f145c15e260 2024-04-02T13:20:33.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.181+0000 7f14597fa640 1 -- 172.21.15.44:0/2316661591 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1450011070 con 0x7f145c15e260 2024-04-02T13:20:33.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.181+0000 7f14597fa640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f144806c920 0x7f144806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.182+0000 7f14597fa640 1 -- 172.21.15.44:0/2316661591 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f145008bab0 con 0x7f145c15e260 2024-04-02T13:20:33.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.182+0000 7f1460bf1640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f144806c920 0x7f144806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.182+0000 7f1460bf1640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f144806c920 0x7f144806ede0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1454009b20 tx=0x7f14540039d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:33.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.186+0000 7f14597fa640 1 -- 172.21.15.44:0/2316661591 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f145000e280 con 0x7f145c15e260 2024-04-02T13:20:33.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.370+0000 7f1462e55640 1 -- 172.21.15.44:0/2316661591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f145c0d47f0 con 0x7f145c15e260 2024-04-02T13:20:33.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.370+0000 7f14597fa640 1 -- 172.21.15.44:0/2316661591 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f1450059ba0 con 0x7f145c15e260 2024-04-02T13:20:33.372 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:20:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.372+0000 7f1462e55640 1 -- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f144806c920 msgr2=0x7f144806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.372+0000 7f1462e55640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f144806c920 0x7f144806ede0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1454009b20 tx=0x7f14540039d0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.372+0000 7f1462e55640 1 -- 172.21.15.44:0/2316661591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f145c15e260 msgr2=0x7f145c0cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.372+0000 7f1462e55640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f145c15e260 0x7f145c0cda10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f145000c410 tx=0x7f145000c8d0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.372+0000 7f1462e55640 1 -- 172.21.15.44:0/2316661591 shutdown_connections 2024-04-02T13:20:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.372+0000 7f1462e55640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f144806c920 0x7f144806ede0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.372+0000 7f1462e55640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15e640 0x7f145c151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.372+0000 7f1462e55640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f145c15e260 0x7f145c0cda10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.372+0000 7f1462e55640 1 --2- 172.21.15.44:0/2316661591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f145c0cec60 0x7f145c0cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.372+0000 7f1462e55640 1 -- 172.21.15.44:0/2316661591 >> 172.21.15.44:0/2316661591 conn(0x7f145c09d3c0 msgr2=0x7f145c0ca630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:33.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.373+0000 7f1462e55640 1 -- 172.21.15.44:0/2316661591 shutdown_connections 2024-04-02T13:20:33.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.373+0000 7f1462e55640 1 -- 172.21.15.44:0/2316661591 wait complete. 2024-04-02T13:20:33.385 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:20:33.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.487+0000 7f045bfa8640 1 Processor -- start 2024-04-02T13:20:33.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.487+0000 7f045bfa8640 1 -- start start 2024-04-02T13:20:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.488+0000 7f045bfa8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f045415e1d0 0x7f04540d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.488+0000 7f045bfa8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04540d1640 con 0x7f04540d2dd0 2024-04-02T13:20:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.488+0000 7f045bfa8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04540cf1f0 con 0x7f04540cec60 2024-04-02T13:20:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.488+0000 7f045bfa8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f04540cfe00 con 0x7f045415e1d0 2024-04-02T13:20:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.488+0000 7f0459d44640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f045415e1d0 0x7f04540d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.488+0000 7f0459d44640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f045415e1d0 0x7f04540d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33962/0 (socket says 172.21.15.44:33962) 2024-04-02T13:20:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.488+0000 7f0459d44640 1 -- 172.21.15.44:0/3094920884 learned_addr learned my addr 172.21.15.44:0/3094920884 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.488+0000 7f045a545640 1 --1- 172.21.15.44:0/3094920884 >> v1:172.21.15.44:6789/0 conn(0x7f04540d2dd0 0x7f04540b4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41610/0 (socket says 172.21.15.44:41610) 2024-04-02T13:20:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.489+0000 7f0459d44640 1 -- 172.21.15.44:0/3094920884 >> v1:172.21.15.69:6789/0 conn(0x7f04540cec60 legacy=0x7f04540d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.489+0000 7f0459d44640 1 -- 172.21.15.44:0/3094920884 >> v1:172.21.15.44:6789/0 conn(0x7f04540d2dd0 legacy=0x7f04540b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.489+0000 7f0459d44640 1 -- 172.21.15.44:0/3094920884 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04540cf4d0 con 0x7f045415e1d0 2024-04-02T13:20:33.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.489+0000 7f0458d42640 1 -- 172.21.15.44:0/3094920884 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3813432640 0 0) 0x7f04540d1640 con 0x7f04540d2dd0 2024-04-02T13:20:33.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.489+0000 7f0459d44640 1 --2- 172.21.15.44:0/3094920884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f045415e1d0 0x7f04540d1100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f044c009870 tx=0x7f044c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810316c65f8e371c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:33.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.489+0000 7f0458d42640 1 -- 172.21.15.44:0/3094920884 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c035020 con 0x7f045415e1d0 2024-04-02T13:20:33.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.490+0000 7f0458d42640 1 -- 172.21.15.44:0/3094920884 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f044c031870 con 0x7f045415e1d0 2024-04-02T13:20:33.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.490+0000 7f0458d42640 1 -- 172.21.15.44:0/3094920884 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c031c40 con 0x7f045415e1d0 2024-04-02T13:20:33.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.490+0000 7f045bfa8640 1 -- 172.21.15.44:0/3094920884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f045415e1d0 msgr2=0x7f04540d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.490+0000 7f045bfa8640 1 --2- 172.21.15.44:0/3094920884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f045415e1d0 0x7f04540d1100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f044c009870 tx=0x7f044c02b910 comp rx=0 tx=0).stop 2024-04-02T13:20:33.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.490+0000 7f045bfa8640 1 -- 172.21.15.44:0/3094920884 shutdown_connections 2024-04-02T13:20:33.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.490+0000 7f045bfa8640 1 --2- 172.21.15.44:0/3094920884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f045415e1d0 0x7f04540d1100 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.490+0000 7f045bfa8640 1 -- 172.21.15.44:0/3094920884 >> 172.21.15.44:0/3094920884 conn(0x7f045409d3b0 msgr2=0x7f04540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:33.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.490+0000 7f045bfa8640 1 -- 172.21.15.44:0/3094920884 shutdown_connections 2024-04-02T13:20:33.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.490+0000 7f045bfa8640 1 -- 172.21.15.44:0/3094920884 wait complete. 2024-04-02T13:20:33.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.491+0000 7f045bfa8640 1 Processor -- start 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.491+0000 7f045bfa8640 1 -- start start 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045bfa8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04540cec60 0x7f04541f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045bfa8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540d2dd0 0x7f045414d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045bfa8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f045415e1d0 0x7f04540cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045bfa8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f04540b7fd0 con 0x7f045415e1d0 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045bfa8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04540cfa90 con 0x7f04540d2dd0 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045bfa8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f04540b8150 con 0x7f04540cec60 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045a545640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f045415e1d0 0x7f04540cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045a545640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f045415e1d0 0x7f04540cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33978/0 (socket says 172.21.15.44:33978) 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045a545640 1 -- 172.21.15.44:0/3651872758 learned_addr learned my addr 172.21.15.44:0/3651872758 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f0459543640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540d2dd0 0x7f045414d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f0459d44640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04540cec60 0x7f04541f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045a545640 1 -- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04540cec60 msgr2=0x7f04541f4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045a545640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04540cec60 0x7f04541f4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045a545640 1 -- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540d2dd0 msgr2=0x7f045414d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045a545640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540d2dd0 0x7f045414d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.492+0000 7f045a545640 1 -- 172.21.15.44:0/3651872758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045415fb70 con 0x7f045415e1d0 2024-04-02T13:20:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.493+0000 7f045a545640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f045415e1d0 0x7f04540cc710 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f045000c6d0 tx=0x7f045000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.493+0000 7f044a7fc640 1 -- 172.21.15.44:0/3651872758 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450018070 con 0x7f045415e1d0 2024-04-02T13:20:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.493+0000 7f044a7fc640 1 -- 172.21.15.44:0/3651872758 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0450004d60 con 0x7f045415e1d0 2024-04-02T13:20:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.493+0000 7f045bfa8640 1 -- 172.21.15.44:0/3651872758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f045415fe10 con 0x7f045415e1d0 2024-04-02T13:20:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.493+0000 7f044a7fc640 1 -- 172.21.15.44:0/3651872758 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045000a880 con 0x7f045415e1d0 2024-04-02T13:20:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.493+0000 7f045bfa8640 1 -- 172.21.15.44:0/3651872758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04541603d0 con 0x7f045415e1d0 2024-04-02T13:20:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.494+0000 7f045bfa8640 1 -- 172.21.15.44:0/3651872758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04540684f0 con 0x7f045415e1d0 2024-04-02T13:20:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.494+0000 7f044a7fc640 1 -- 172.21.15.44:0/3651872758 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f045001b070 con 0x7f045415e1d0 2024-04-02T13:20:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.495+0000 7f044a7fc640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f043806c8f0 0x7f043806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.495+0000 7f044a7fc640 1 -- 172.21.15.44:0/3651872758 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f045008b320 con 0x7f045415e1d0 2024-04-02T13:20:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.495+0000 7f0459d44640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f043806c8f0 0x7f043806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.495+0000 7f0459d44640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f043806c8f0 0x7f043806edb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f044c036000 tx=0x7f044c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:33.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.499+0000 7f044a7fc640 1 -- 172.21.15.44:0/3651872758 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f04500594a0 con 0x7f045415e1d0 2024-04-02T13:20:33.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.684+0000 7f045bfa8640 1 -- 172.21.15.44:0/3651872758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f04540d47f0 con 0x7f045415e1d0 2024-04-02T13:20:33.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.684+0000 7f044a7fc640 1 -- 172.21.15.44:0/3651872758 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f0450058da0 con 0x7f045415e1d0 2024-04-02T13:20:33.687 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:20:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.686+0000 7f045bfa8640 1 -- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f043806c8f0 msgr2=0x7f043806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.686+0000 7f045bfa8640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f043806c8f0 0x7f043806edb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f044c036000 tx=0x7f044c008030 comp rx=0 tx=0).stop 2024-04-02T13:20:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.686+0000 7f045bfa8640 1 -- 172.21.15.44:0/3651872758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f045415e1d0 msgr2=0x7f04540cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.686+0000 7f045bfa8640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f045415e1d0 0x7f04540cc710 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f045000c6d0 tx=0x7f045000cb90 comp rx=0 tx=0).stop 2024-04-02T13:20:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.686+0000 7f045bfa8640 1 -- 172.21.15.44:0/3651872758 shutdown_connections 2024-04-02T13:20:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.686+0000 7f045bfa8640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f043806c8f0 0x7f043806edb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.686+0000 7f045bfa8640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f045415e1d0 0x7f04540cc710 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.686+0000 7f045bfa8640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540d2dd0 0x7f045414d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.686+0000 7f045bfa8640 1 --2- 172.21.15.44:0/3651872758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04540cec60 0x7f04541f4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.686+0000 7f045bfa8640 1 -- 172.21.15.44:0/3651872758 >> 172.21.15.44:0/3651872758 conn(0x7f045409d3b0 msgr2=0x7f0454161d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.687+0000 7f045bfa8640 1 -- 172.21.15.44:0/3651872758 shutdown_connections 2024-04-02T13:20:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.687+0000 7f045bfa8640 1 -- 172.21.15.44:0/3651872758 wait complete. 2024-04-02T13:20:33.699 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:20:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.801+0000 7f22f9e55640 1 Processor -- start 2024-04-02T13:20:33.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.801+0000 7f22f9e55640 1 -- start start 2024-04-02T13:20:33.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.802+0000 7f22f9e55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f415c1a0 0x7f22f406f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.802+0000 7f22f9e55640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22f415c580 con 0x7f22f40d21c0 2024-04-02T13:20:33.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.802+0000 7f22f9e55640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22f40d01c0 con 0x7f22f40cfd60 2024-04-02T13:20:33.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.802+0000 7f22f9e55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f22f4070ad0 con 0x7f22f415c1a0 2024-04-02T13:20:33.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.802+0000 7f22f3fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f22f40d21c0 0x7f22f40cc9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41618/0 (socket says 172.21.15.44:41618) 2024-04-02T13:20:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.802+0000 7f22f3fff640 1 -- 172.21.15.44:0/3760925236 learned_addr learned my addr 172.21.15.44:0/3760925236 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.802+0000 7f22f37fe640 1 --2- 172.21.15.44:0/3760925236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f415c1a0 0x7f22f406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.803+0000 7f22f27fc640 1 -- 172.21.15.44:0/3760925236 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3800381117 0 0) 0x7f22f415c580 con 0x7f22f40d21c0 2024-04-02T13:20:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.803+0000 7f22f27fc640 1 -- 172.21.15.44:0/3760925236 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22dc003520 con 0x7f22f40d21c0 2024-04-02T13:20:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.803+0000 7f22f27fc640 1 -- 172.21.15.44:0/3760925236 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3131298173 0 0) 0x7f22f40d01c0 con 0x7f22f40cfd60 2024-04-02T13:20:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.803+0000 7f22f27fc640 1 -- 172.21.15.44:0/3760925236 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22f415c580 con 0x7f22f40cfd60 2024-04-02T13:20:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.803+0000 7f22f37fe640 1 -- 172.21.15.44:0/3760925236 >> v1:172.21.15.69:6789/0 conn(0x7f22f40cfd60 legacy=0x7f22f40d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.803+0000 7f22f37fe640 1 -- 172.21.15.44:0/3760925236 >> v1:172.21.15.44:6789/0 conn(0x7f22f40d21c0 legacy=0x7f22f40cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.803+0000 7f22f37fe640 1 -- 172.21.15.44:0/3760925236 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f406fa30 con 0x7f22f415c1a0 2024-04-02T13:20:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.804+0000 7f22f37fe640 1 --2- 172.21.15.44:0/3760925236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f415c1a0 0x7f22f406f370 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f22e40097d0 tx=0x7f22e402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eaca25714fac972 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.804+0000 7f22f27fc640 1 -- 172.21.15.44:0/3760925236 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e4035020 con 0x7f22f415c1a0 2024-04-02T13:20:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.804+0000 7f22f27fc640 1 -- 172.21.15.44:0/3760925236 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22e4031890 con 0x7f22f415c1a0 2024-04-02T13:20:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.804+0000 7f22f27fc640 1 -- 172.21.15.44:0/3760925236 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e4031c60 con 0x7f22f415c1a0 2024-04-02T13:20:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.804+0000 7f22f9e55640 1 -- 172.21.15.44:0/3760925236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f415c1a0 msgr2=0x7f22f406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.804+0000 7f22f9e55640 1 --2- 172.21.15.44:0/3760925236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f415c1a0 0x7f22f406f370 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f22e40097d0 tx=0x7f22e402b930 comp rx=0 tx=0).stop 2024-04-02T13:20:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.804+0000 7f22f9e55640 1 -- 172.21.15.44:0/3760925236 shutdown_connections 2024-04-02T13:20:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.804+0000 7f22f9e55640 1 --2- 172.21.15.44:0/3760925236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f415c1a0 0x7f22f406f370 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.804+0000 7f22f9e55640 1 -- 172.21.15.44:0/3760925236 >> 172.21.15.44:0/3760925236 conn(0x7f22f409aa20 msgr2=0x7f22f409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.804+0000 7f22f9e55640 1 -- 172.21.15.44:0/3760925236 shutdown_connections 2024-04-02T13:20:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.804+0000 7f22f9e55640 1 -- 172.21.15.44:0/3760925236 wait complete. 2024-04-02T13:20:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.805+0000 7f22f9e55640 1 Processor -- start 2024-04-02T13:20:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.806+0000 7f22f9e55640 1 -- start start 2024-04-02T13:20:33.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.806+0000 7f22f9e55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f40cfd60 0x7f22f40c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.806+0000 7f22f9e55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40d21c0 0x7f22f40c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.806+0000 7f22f9e55640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22f415c1a0 0x7f22f40c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.806+0000 7f22f9e55640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f22f40702d0 con 0x7f22f415c1a0 2024-04-02T13:20:33.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.806+0000 7f22f9e55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f22f406fff0 con 0x7f22f40cfd60 2024-04-02T13:20:33.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.806+0000 7f22f9e55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f22f40705b0 con 0x7f22f40d21c0 2024-04-02T13:20:33.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.806+0000 7f22f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22f415c1a0 0x7f22f40c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.806+0000 7f22f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22f415c1a0 0x7f22f40c98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33988/0 (socket says 172.21.15.44:33988) 2024-04-02T13:20:33.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22f3fff640 1 -- 172.21.15.44:0/3456431254 learned_addr learned my addr 172.21.15.44:0/3456431254 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:33.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22f37fe640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f40cfd60 0x7f22f40c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22f2ffd640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40d21c0 0x7f22f40c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22f3fff640 1 -- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40d21c0 msgr2=0x7f22f40c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22f3fff640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40d21c0 0x7f22f40c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22f3fff640 1 -- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f40cfd60 msgr2=0x7f22f40c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:33.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22f3fff640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f40cfd60 0x7f22f40c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:33.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22f3fff640 1 -- 172.21.15.44:0/3456431254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f406dc30 con 0x7f22f415c1a0 2024-04-02T13:20:33.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22f37fe640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f40cfd60 0x7f22f40c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:33.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22f3fff640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22f415c1a0 0x7f22f40c98d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00cb70 tx=0x7f22ec00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:33.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22dbfff640 1 -- 172.21.15.44:0/3456431254 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec017070 con 0x7f22f415c1a0 2024-04-02T13:20:33.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22dbfff640 1 -- 172.21.15.44:0/3456431254 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22ec00ace0 con 0x7f22f415c1a0 2024-04-02T13:20:33.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22f9e55640 1 -- 172.21.15.44:0/3456431254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f406df10 con 0x7f22f415c1a0 2024-04-02T13:20:33.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22dbfff640 1 -- 172.21.15.44:0/3456431254 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec012600 con 0x7f22f415c1a0 2024-04-02T13:20:33.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.807+0000 7f22f9e55640 1 -- 172.21.15.44:0/3456431254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f415d720 con 0x7f22f415c1a0 2024-04-02T13:20:33.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.808+0000 7f22f9e55640 1 -- 172.21.15.44:0/3456431254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f40c8d50 con 0x7f22f415c1a0 2024-04-02T13:20:33.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.809+0000 7f22dbfff640 1 -- 172.21.15.44:0/3456431254 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f22ec01a070 con 0x7f22f415c1a0 2024-04-02T13:20:33.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.809+0000 7f22dbfff640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f22e006c8f0 0x7f22e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:33.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.809+0000 7f22dbfff640 1 -- 172.21.15.44:0/3456431254 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f22ec08bbf0 con 0x7f22f415c1a0 2024-04-02T13:20:33.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.809+0000 7f22f37fe640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f22e006c8f0 0x7f22e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:33.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.810+0000 7f22f37fe640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f22e006c8f0 0x7f22e006edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f22e4036000 tx=0x7f22e403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:33.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.814+0000 7f22dbfff640 1 -- 172.21.15.44:0/3456431254 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f22ec059680 con 0x7f22f415c1a0 2024-04-02T13:20:33.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.996+0000 7f22f9e55640 1 -- 172.21.15.44:0/3456431254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f22f4068c00 con 0x7f22f415c1a0 2024-04-02T13:20:33.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.996+0000 7f22dbfff640 1 -- 172.21.15.44:0/3456431254 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f22ec05d980 con 0x7f22f415c1a0 2024-04-02T13:20:33.999 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:20:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.998+0000 7f22f9e55640 1 -- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f22e006c8f0 msgr2=0x7f22e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.998+0000 7f22f9e55640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f22e006c8f0 0x7f22e006edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f22e4036000 tx=0x7f22e403c040 comp rx=0 tx=0).stop 2024-04-02T13:20:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.998+0000 7f22f9e55640 1 -- 172.21.15.44:0/3456431254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22f415c1a0 msgr2=0x7f22f40c98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.998+0000 7f22f9e55640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22f415c1a0 0x7f22f40c98d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00cb70 tx=0x7f22ec00a430 comp rx=0 tx=0).stop 2024-04-02T13:20:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.998+0000 7f22f9e55640 1 -- 172.21.15.44:0/3456431254 shutdown_connections 2024-04-02T13:20:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.998+0000 7f22f9e55640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f22e006c8f0 0x7f22e006edb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.998+0000 7f22f9e55640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22f415c1a0 0x7f22f40c98d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.998+0000 7f22f9e55640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40d21c0 0x7f22f40c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.998+0000 7f22f9e55640 1 --2- 172.21.15.44:0/3456431254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f40cfd60 0x7f22f40c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.998+0000 7f22f9e55640 1 -- 172.21.15.44:0/3456431254 >> 172.21.15.44:0/3456431254 conn(0x7f22f409aa20 msgr2=0x7f22f40b04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.998+0000 7f22f9e55640 1 -- 172.21.15.44:0/3456431254 shutdown_connections 2024-04-02T13:20:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:33.999+0000 7f22f9e55640 1 -- 172.21.15.44:0/3456431254 wait complete. 2024-04-02T13:20:34.012 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:20:34.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.112+0000 7f664b5f6640 1 Processor -- start 2024-04-02T13:20:34.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.112+0000 7f664b5f6640 1 -- start start 2024-04-02T13:20:34.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.113+0000 7f664b5f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f664415e550 0x7f664406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:34.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.113+0000 7f664b5f6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f664415f040 0x7f66440d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:34.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.113+0000 7f664b5f6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f664415fb50 con 0x7f664406d530 2024-04-02T13:20:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.113+0000 7f664b5f6640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f664406d970 con 0x7f664415f040 2024-04-02T13:20:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.113+0000 7f664b5f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f664406ca90 con 0x7f664415e550 2024-04-02T13:20:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.113+0000 7f6649b93640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f664415f040 0x7f66440d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.113+0000 7f6649b93640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f664415f040 0x7f66440d0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34000/0 (socket says 172.21.15.44:34000) 2024-04-02T13:20:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.113+0000 7f6649b93640 1 -- 172.21.15.44:0/557012462 learned_addr learned my addr 172.21.15.44:0/557012462 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.113+0000 7f6649392640 1 --2- 172.21.15.44:0/557012462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f664415e550 0x7f664406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.113+0000 7f6649b93640 1 -- 172.21.15.44:0/557012462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f664415e550 msgr2=0x7f664406cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.113+0000 7f6649b93640 1 --2- 172.21.15.44:0/557012462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f664415e550 0x7f664406cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.113+0000 7f6649b93640 1 -- 172.21.15.44:0/557012462 >> v1:172.21.15.69:6789/0 conn(0x7f664406d530 legacy=0x7f664415e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.113+0000 7f6649b93640 1 -- 172.21.15.44:0/557012462 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f664415f590 con 0x7f664415f040 2024-04-02T13:20:34.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.114+0000 7f6649b93640 1 --2- 172.21.15.44:0/557012462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f664415f040 0x7f66440d0ab0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f663c009e50 tx=0x7f663c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d17f3644aa1bcf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:34.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.114+0000 7f663bfff640 1 -- 172.21.15.44:0/557012462 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c02cb00 con 0x7f664415f040 2024-04-02T13:20:34.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.114+0000 7f663bfff640 1 -- 172.21.15.44:0/557012462 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f663c02cca0 con 0x7f664415f040 2024-04-02T13:20:34.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.114+0000 7f663bfff640 1 -- 172.21.15.44:0/557012462 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c034610 con 0x7f664415f040 2024-04-02T13:20:34.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.114+0000 7f664b5f6640 1 -- 172.21.15.44:0/557012462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f664415f040 msgr2=0x7f66440d0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:34.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.114+0000 7f664b5f6640 1 --2- 172.21.15.44:0/557012462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f664415f040 0x7f66440d0ab0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f663c009e50 tx=0x7f663c02bf60 comp rx=0 tx=0).stop 2024-04-02T13:20:34.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.114+0000 7f664b5f6640 1 -- 172.21.15.44:0/557012462 shutdown_connections 2024-04-02T13:20:34.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.114+0000 7f664b5f6640 1 --2- 172.21.15.44:0/557012462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f664415f040 0x7f66440d0ab0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.114+0000 7f664b5f6640 1 --2- 172.21.15.44:0/557012462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f664415e550 0x7f664406cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.114+0000 7f664b5f6640 1 -- 172.21.15.44:0/557012462 >> 172.21.15.44:0/557012462 conn(0x7f664409aa20 msgr2=0x7f664409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:34.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.115+0000 7f664b5f6640 1 -- 172.21.15.44:0/557012462 shutdown_connections 2024-04-02T13:20:34.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.115+0000 7f664b5f6640 1 -- 172.21.15.44:0/557012462 wait complete. 2024-04-02T13:20:34.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.116+0000 7f664b5f6640 1 Processor -- start 2024-04-02T13:20:34.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.116+0000 7f664b5f6640 1 -- start start 2024-04-02T13:20:34.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.116+0000 7f664b5f6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f664406d530 0x7f66440ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:34.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.116+0000 7f664b5f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f664415e550 0x7f66441f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:34.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.116+0000 7f664b5f6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f664415f040 0x7f66441f76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:34.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.116+0000 7f664b5f6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f66440d0590 con 0x7f664406d530 2024-04-02T13:20:34.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.116+0000 7f664b5f6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f66440d02b0 con 0x7f664415f040 2024-04-02T13:20:34.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.116+0000 7f664b5f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f66440cc4b0 con 0x7f664415e550 2024-04-02T13:20:34.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.116+0000 7f6649392640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f664406d530 0x7f66440ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:34.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f6649392640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f664406d530 0x7f66440ca970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34008/0 (socket says 172.21.15.44:34008) 2024-04-02T13:20:34.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f6649392640 1 -- 172.21.15.44:0/1083405850 learned_addr learned my addr 172.21.15.44:0/1083405850 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:34.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f6649b93640 1 --2- 172.21.15.44:0/1083405850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f664415f040 0x7f66441f76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:34.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f6649392640 1 -- 172.21.15.44:0/1083405850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f664415e550 msgr2=0x7f66441f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:34.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f6649392640 1 --2- 172.21.15.44:0/1083405850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f664415e550 0x7f66441f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f6649392640 1 -- 172.21.15.44:0/1083405850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f664415f040 msgr2=0x7f66441f76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:34.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f6649392640 1 --2- 172.21.15.44:0/1083405850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f664415f040 0x7f66441f76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f6649392640 1 -- 172.21.15.44:0/1083405850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f664414bc20 con 0x7f664406d530 2024-04-02T13:20:34.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f6649392640 1 --2- 172.21.15.44:0/1083405850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f664406d530 0x7f66440ca970 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f664000c820 tx=0x7f664000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:34.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f6639ffb640 1 -- 172.21.15.44:0/1083405850 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6640015070 con 0x7f664406d530 2024-04-02T13:20:34.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f6639ffb640 1 -- 172.21.15.44:0/1083405850 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6640004020 con 0x7f664406d530 2024-04-02T13:20:34.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f6639ffb640 1 -- 172.21.15.44:0/1083405850 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66400043f0 con 0x7f664406d530 2024-04-02T13:20:34.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f664b5f6640 1 -- 172.21.15.44:0/1083405850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f664414bf00 con 0x7f664406d530 2024-04-02T13:20:34.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.117+0000 7f664b5f6640 1 -- 172.21.15.44:0/1083405850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f664414c4c0 con 0x7f664406d530 2024-04-02T13:20:34.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.118+0000 7f664b5f6640 1 -- 172.21.15.44:0/1083405850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66440684f0 con 0x7f664406d530 2024-04-02T13:20:34.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.119+0000 7f6639ffb640 1 -- 172.21.15.44:0/1083405850 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6640007830 con 0x7f664406d530 2024-04-02T13:20:34.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.119+0000 7f6639ffb640 1 --2- 172.21.15.44:0/1083405850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f662806c8f0 0x7f662806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:34.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.119+0000 7f6639ffb640 1 -- 172.21.15.44:0/1083405850 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f664008bca0 con 0x7f664406d530 2024-04-02T13:20:34.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.119+0000 7f6648b91640 1 --2- 172.21.15.44:0/1083405850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f662806c8f0 0x7f662806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:34.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.120+0000 7f6648b91640 1 --2- 172.21.15.44:0/1083405850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f662806c8f0 0x7f662806edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6634006fd0 tx=0x7f6634006bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:34.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.123+0000 7f6639ffb640 1 -- 172.21.15.44:0/1083405850 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6640059d90 con 0x7f664406d530 2024-04-02T13:20:34.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.320+0000 7f664b5f6640 1 -- 172.21.15.44:0/1083405850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6644067d70 con 0x7f664406d530 2024-04-02T13:20:34.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.320+0000 7f6639ffb640 1 -- 172.21.15.44:0/1083405850 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7f6640059690 con 0x7f664406d530 2024-04-02T13:20:34.323 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:20:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.322+0000 7f664b5f6640 1 -- 172.21.15.44:0/1083405850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f662806c8f0 msgr2=0x7f662806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.322+0000 7f664b5f6640 1 --2- 172.21.15.44:0/1083405850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f662806c8f0 0x7f662806edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6634006fd0 tx=0x7f6634006bc0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.322+0000 7f664b5f6640 1 -- 172.21.15.44:0/1083405850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f664406d530 msgr2=0x7f66440ca970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.322+0000 7f664b5f6640 1 --2- 172.21.15.44:0/1083405850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f664406d530 0x7f66440ca970 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f664000c820 tx=0x7f664000cce0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.322+0000 7f664b5f6640 1 -- 172.21.15.44:0/1083405850 shutdown_connections 2024-04-02T13:20:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.322+0000 7f664b5f6640 1 --2- 172.21.15.44:0/1083405850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f662806c8f0 0x7f662806edb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.322+0000 7f664b5f6640 1 --2- 172.21.15.44:0/1083405850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f664415f040 0x7f66441f76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.322+0000 7f664b5f6640 1 --2- 172.21.15.44:0/1083405850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f664415e550 0x7f66441f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.322+0000 7f664b5f6640 1 --2- 172.21.15.44:0/1083405850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f664406d530 0x7f66440ca970 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.322+0000 7f664b5f6640 1 -- 172.21.15.44:0/1083405850 >> 172.21.15.44:0/1083405850 conn(0x7f664409aa20 msgr2=0x7f66440ccb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.322+0000 7f664b5f6640 1 -- 172.21.15.44:0/1083405850 shutdown_connections 2024-04-02T13:20:34.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:34.322+0000 7f664b5f6640 1 -- 172.21.15.44:0/1083405850 wait complete. 2024-04-02T13:20:34.335 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:20:34.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:34.335 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-02T13:20:34.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:ceph-fuse[44076]: fuse finished with error 0 and tester_r 0 2024-04-02T13:20:34.362 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:20:40.464 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:20:40.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:40.464 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-02T13:20:40.495 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:40.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.647+0000 7f43ac650640 1 Processor -- start 2024-04-02T13:20:40.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.648+0000 7f43ac650640 1 -- start start 2024-04-02T13:20:40.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.648+0000 7f43ac650640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43a415e4d0 0x7f43a40cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:40.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.648+0000 7f43ac650640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f43a40cfd20 0x7f43a40d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:40.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43ac650640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43a415e8b0 con 0x7f43a40d1060 2024-04-02T13:20:40.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43ac650640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f43a40b3a90 con 0x7f43a40cfd20 2024-04-02T13:20:40.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43ac650640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f43a40b4610 con 0x7f43a415e4d0 2024-04-02T13:20:40.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43a9beb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f43a40cfd20 0x7f43a40d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:40.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43aa3ec640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43a415e4d0 0x7f43a40cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:40.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43aabed640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f43a40d1060 0x7f43a40d1d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:52186/0 (socket says 172.21.15.44:52186) 2024-04-02T13:20:40.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43aabed640 1 -- 172.21.15.44:0/2252029222 learned_addr learned my addr 172.21.15.44:0/2252029222 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:40.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43a93ea640 1 -- 172.21.15.44:0/2252029222 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4243516018 0 0) 0x7f43a415e8b0 con 0x7f43a40d1060 2024-04-02T13:20:40.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43a93ea640 1 -- 172.21.15.44:0/2252029222 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4390003520 con 0x7f43a40d1060 2024-04-02T13:20:40.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43a9beb640 1 -- 172.21.15.44:0/2252029222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43a415e4d0 msgr2=0x7f43a40cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:40.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43a9beb640 1 --2- 172.21.15.44:0/2252029222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43a415e4d0 0x7f43a40cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43a9beb640 1 -- 172.21.15.44:0/2252029222 >> v1:172.21.15.69:6789/0 conn(0x7f43a40d1060 legacy=0x7f43a40d1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:40.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43a9beb640 1 -- 172.21.15.44:0/2252029222 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a40d2fb0 con 0x7f43a40cfd20 2024-04-02T13:20:40.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.649+0000 7f43aa3ec640 1 --2- 172.21.15.44:0/2252029222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43a415e4d0 0x7f43a40cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:40.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.650+0000 7f43a9beb640 1 --2- 172.21.15.44:0/2252029222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43a40cfd20 0x7f43a40d0b20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f43940097a0 tx=0x7f439402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0499109ba7f1ef9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:40.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.650+0000 7f43a93ea640 1 -- 172.21.15.44:0/2252029222 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4394035020 con 0x7f43a40cfd20 2024-04-02T13:20:40.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.650+0000 7f43a93ea640 1 -- 172.21.15.44:0/2252029222 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4394031830 con 0x7f43a40cfd20 2024-04-02T13:20:40.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.650+0000 7f43a93ea640 1 -- 172.21.15.44:0/2252029222 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4394031be0 con 0x7f43a40cfd20 2024-04-02T13:20:40.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.651+0000 7f43ac650640 1 -- 172.21.15.44:0/2252029222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43a40cfd20 msgr2=0x7f43a40d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:40.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.651+0000 7f43ac650640 1 --2- 172.21.15.44:0/2252029222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43a40cfd20 0x7f43a40d0b20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f43940097a0 tx=0x7f439402b8d0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.651+0000 7f43ac650640 1 -- 172.21.15.44:0/2252029222 shutdown_connections 2024-04-02T13:20:40.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.651+0000 7f43ac650640 1 --2- 172.21.15.44:0/2252029222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43a40cfd20 0x7f43a40d0b20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.651+0000 7f43ac650640 1 --2- 172.21.15.44:0/2252029222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43a415e4d0 0x7f43a40cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.651+0000 7f43ac650640 1 -- 172.21.15.44:0/2252029222 >> 172.21.15.44:0/2252029222 conn(0x7f43a409d410 msgr2=0x7f43a40d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:40.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.651+0000 7f43ac650640 1 -- 172.21.15.44:0/2252029222 shutdown_connections 2024-04-02T13:20:40.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.651+0000 7f43ac650640 1 -- 172.21.15.44:0/2252029222 wait complete. 2024-04-02T13:20:40.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.652+0000 7f43ac650640 1 Processor -- start 2024-04-02T13:20:40.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.652+0000 7f43ac650640 1 -- start start 2024-04-02T13:20:40.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.653+0000 7f43ac650640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43a40cfd20 0x7f43a40cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:40.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.653+0000 7f43ac650640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43a40d1060 0x7f43a415eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:40.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.653+0000 7f43ac650640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43a415e4d0 0x7f43a40b8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:40.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.653+0000 7f43ac650640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f43a40b42a0 con 0x7f43a40cfd20 2024-04-02T13:20:40.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.653+0000 7f43ac650640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f43a40b3fc0 con 0x7f43a40d1060 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.653+0000 7f43ac650640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f43a40b0d00 con 0x7f43a415e4d0 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.653+0000 7f43aa3ec640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43a40cfd20 0x7f43a40cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.653+0000 7f43aa3ec640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43a40cfd20 0x7f43a40cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58934/0 (socket says 172.21.15.44:58934) 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.653+0000 7f43aa3ec640 1 -- 172.21.15.44:0/2847320200 learned_addr learned my addr 172.21.15.44:0/2847320200 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.653+0000 7f43a9beb640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43a40d1060 0x7f43a415eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.653+0000 7f43aabed640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43a415e4d0 0x7f43a40b8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.654+0000 7f43aa3ec640 1 -- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43a415e4d0 msgr2=0x7f43a40b8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.654+0000 7f43aa3ec640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43a415e4d0 0x7f43a40b8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.654+0000 7f43aa3ec640 1 -- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43a40d1060 msgr2=0x7f43a415eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.654+0000 7f43aa3ec640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43a40d1060 0x7f43a415eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.654+0000 7f43aa3ec640 1 -- 172.21.15.44:0/2847320200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a41f4a20 con 0x7f43a40cfd20 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.654+0000 7f43aabed640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43a415e4d0 0x7f43a40b8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.654+0000 7f43aa3ec640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43a40cfd20 0x7f43a40cdc60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f43a000e960 tx=0x7f43a000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:40.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.654+0000 7f439affd640 1 -- 172.21.15.44:0/2847320200 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0017070 con 0x7f43a40cfd20 2024-04-02T13:20:40.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.654+0000 7f439affd640 1 -- 172.21.15.44:0/2847320200 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a000ca00 con 0x7f43a40cfd20 2024-04-02T13:20:40.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.654+0000 7f43ac650640 1 -- 172.21.15.44:0/2847320200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a41f4d00 con 0x7f43a40cfd20 2024-04-02T13:20:40.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.654+0000 7f439affd640 1 -- 172.21.15.44:0/2847320200 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a000cdd0 con 0x7f43a40cfd20 2024-04-02T13:20:40.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.654+0000 7f43ac650640 1 -- 172.21.15.44:0/2847320200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a40b8680 con 0x7f43a40cfd20 2024-04-02T13:20:40.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.655+0000 7f43ac650640 1 -- 172.21.15.44:0/2847320200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43a4068c00 con 0x7f43a40cfd20 2024-04-02T13:20:40.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.655+0000 7f439affd640 1 -- 172.21.15.44:0/2847320200 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f43a001a070 con 0x7f43a40cfd20 2024-04-02T13:20:40.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.656+0000 7f439affd640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f438806c8f0 0x7f438806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:40.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.656+0000 7f439affd640 1 -- 172.21.15.44:0/2847320200 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f43a008bfc0 con 0x7f43a40cfd20 2024-04-02T13:20:40.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.656+0000 7f43a9beb640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f438806c8f0 0x7f438806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:40.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.656+0000 7f43a9beb640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f438806c8f0 0x7f438806edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4394036000 tx=0x7f439403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:40.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.660+0000 7f439affd640 1 -- 172.21.15.44:0/2847320200 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f43a0059af0 con 0x7f43a40cfd20 2024-04-02T13:20:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.846+0000 7f43ac650640 1 -- 172.21.15.44:0/2847320200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43a406d000 con 0x7f43a40cfd20 2024-04-02T13:20:40.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.847+0000 7f439affd640 1 -- 172.21.15.44:0/2847320200 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f43a005ddf0 con 0x7f43a40cfd20 2024-04-02T13:20:40.849 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:40.850 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:40.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.849+0000 7f43ac650640 1 -- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f438806c8f0 msgr2=0x7f438806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.849+0000 7f43ac650640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f438806c8f0 0x7f438806edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4394036000 tx=0x7f439403d040 comp rx=0 tx=0).stop 2024-04-02T13:20:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.849+0000 7f43ac650640 1 -- 172.21.15.44:0/2847320200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43a40cfd20 msgr2=0x7f43a40cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.849+0000 7f43ac650640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43a40cfd20 0x7f43a40cdc60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f43a000e960 tx=0x7f43a000ee20 comp rx=0 tx=0).stop 2024-04-02T13:20:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.849+0000 7f43ac650640 1 -- 172.21.15.44:0/2847320200 shutdown_connections 2024-04-02T13:20:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.849+0000 7f43ac650640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f438806c8f0 0x7f438806edb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.849+0000 7f43ac650640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43a415e4d0 0x7f43a40b8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.849+0000 7f43ac650640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43a40d1060 0x7f43a415eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.849+0000 7f43ac650640 1 --2- 172.21.15.44:0/2847320200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43a40cfd20 0x7f43a40cdc60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.849+0000 7f43ac650640 1 -- 172.21.15.44:0/2847320200 >> 172.21.15.44:0/2847320200 conn(0x7f43a409d410 msgr2=0x7f43a4067230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.849+0000 7f43ac650640 1 -- 172.21.15.44:0/2847320200 shutdown_connections 2024-04-02T13:20:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.849+0000 7f43ac650640 1 -- 172.21.15.44:0/2847320200 wait complete. 2024-04-02T13:20:40.862 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:40.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.964+0000 7f90e5e60640 1 Processor -- start 2024-04-02T13:20:40.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.964+0000 7f90e5e60640 1 -- start start 2024-04-02T13:20:40.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.964+0000 7f90e5e60640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f90e00d0a30 0x7f90e006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:40.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.964+0000 7f90e5e60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d0e10 0x7f90e00cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.964+0000 7f90e5e60640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90e006c1d0 con 0x7f90e00cfe30 2024-04-02T13:20:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.964+0000 7f90e5e60640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f90e006cb00 con 0x7f90e00d0a30 2024-04-02T13:20:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.964+0000 7f90e5e60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f90e006c4b0 con 0x7f90e00d0e10 2024-04-02T13:20:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.965+0000 7f90df7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f90e00d0a30 0x7f90e006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.965+0000 7f90df7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f90e00d0a30 0x7f90e006cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58946/0 (socket says 172.21.15.44:58946) 2024-04-02T13:20:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.965+0000 7f90df7fe640 1 -- 172.21.15.44:0/1161944905 learned_addr learned my addr 172.21.15.44:0/1161944905 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.965+0000 7f90deffd640 1 --2- 172.21.15.44:0/1161944905 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d0e10 0x7f90e00cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.965+0000 7f90df7fe640 1 -- 172.21.15.44:0/1161944905 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d0e10 msgr2=0x7f90e00cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.965+0000 7f90df7fe640 1 --2- 172.21.15.44:0/1161944905 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d0e10 0x7f90e00cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.965+0000 7f90df7fe640 1 -- 172.21.15.44:0/1161944905 >> v1:172.21.15.69:6789/0 conn(0x7f90e00cfe30 legacy=0x7f90e00d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.965+0000 7f90df7fe640 1 -- 172.21.15.44:0/1161944905 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e006c790 con 0x7f90e00d0a30 2024-04-02T13:20:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.966+0000 7f90df7fe640 1 --2- 172.21.15.44:0/1161944905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e00d0a30 0x7f90e006cf90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009870 tx=0x7f90d002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a2942074cb75991 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.966+0000 7f90de7fc640 1 -- 172.21.15.44:0/1161944905 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d00317c0 con 0x7f90e00d0a30 2024-04-02T13:20:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.966+0000 7f90de7fc640 1 -- 172.21.15.44:0/1161944905 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90d0037070 con 0x7f90e00d0a30 2024-04-02T13:20:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.966+0000 7f90de7fc640 1 -- 172.21.15.44:0/1161944905 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d0031b80 con 0x7f90e00d0a30 2024-04-02T13:20:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.966+0000 7f90e5e60640 1 -- 172.21.15.44:0/1161944905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e00d0a30 msgr2=0x7f90e006cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.966+0000 7f90e5e60640 1 --2- 172.21.15.44:0/1161944905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e00d0a30 0x7f90e006cf90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009870 tx=0x7f90d002b860 comp rx=0 tx=0).stop 2024-04-02T13:20:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.966+0000 7f90e5e60640 1 -- 172.21.15.44:0/1161944905 shutdown_connections 2024-04-02T13:20:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.966+0000 7f90e5e60640 1 --2- 172.21.15.44:0/1161944905 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d0e10 0x7f90e00cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.966+0000 7f90e5e60640 1 --2- 172.21.15.44:0/1161944905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e00d0a30 0x7f90e006cf90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.966+0000 7f90e5e60640 1 -- 172.21.15.44:0/1161944905 >> 172.21.15.44:0/1161944905 conn(0x7f90e009d470 msgr2=0x7f90e00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:40.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.966+0000 7f90e5e60640 1 -- 172.21.15.44:0/1161944905 shutdown_connections 2024-04-02T13:20:40.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.967+0000 7f90e5e60640 1 -- 172.21.15.44:0/1161944905 wait complete. 2024-04-02T13:20:40.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.968+0000 7f90e5e60640 1 Processor -- start 2024-04-02T13:20:40.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.968+0000 7f90e5e60640 1 -- start start 2024-04-02T13:20:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.968+0000 7f90e5e60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00cfe30 0x7f90e0155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.968+0000 7f90e5e60640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e00d0a30 0x7f90e00ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.969+0000 7f90e5e60640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90e00d0e10 0x7f90e015fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.969+0000 7f90e5e60640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f90e015e870 con 0x7f90e00d0a30 2024-04-02T13:20:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.969+0000 7f90e5e60640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f90e015e6f0 con 0x7f90e00d0e10 2024-04-02T13:20:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.969+0000 7f90e5e60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f90e00d4da0 con 0x7f90e00cfe30 2024-04-02T13:20:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.969+0000 7f90df7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00cfe30 0x7f90e0155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.969+0000 7f90df7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00cfe30 0x7f90e0155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:36160/0 (socket says 172.21.15.44:36160) 2024-04-02T13:20:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.969+0000 7f90df7fe640 1 -- 172.21.15.44:0/1630689688 learned_addr learned my addr 172.21.15.44:0/1630689688 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.969+0000 7f90dffff640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90e00d0e10 0x7f90e015fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.969+0000 7f90deffd640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e00d0a30 0x7f90e00ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.969+0000 7f90deffd640 1 -- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00cfe30 msgr2=0x7f90e0155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.969+0000 7f90deffd640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00cfe30 0x7f90e0155dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.969+0000 7f90deffd640 1 -- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90e00d0e10 msgr2=0x7f90e015fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.969+0000 7f90deffd640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90e00d0e10 0x7f90e015fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.970+0000 7f90deffd640 1 -- 172.21.15.44:0/1630689688 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e00c92d0 con 0x7f90e00d0a30 2024-04-02T13:20:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.970+0000 7f90df7fe640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00cfe30 0x7f90e0155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.970+0000 7f90deffd640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e00d0a30 0x7f90e00ca370 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f90d400c550 tx=0x7f90d400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.970+0000 7f90c7fff640 1 -- 172.21.15.44:0/1630689688 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d4004230 con 0x7f90e00d0a30 2024-04-02T13:20:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.970+0000 7f90c7fff640 1 -- 172.21.15.44:0/1630689688 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90d40043d0 con 0x7f90e00d0a30 2024-04-02T13:20:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.970+0000 7f90c7fff640 1 -- 172.21.15.44:0/1630689688 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d4003830 con 0x7f90e00d0a30 2024-04-02T13:20:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.970+0000 7f90e5e60640 1 -- 172.21.15.44:0/1630689688 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e00c95b0 con 0x7f90e00d0a30 2024-04-02T13:20:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.970+0000 7f90e5e60640 1 -- 172.21.15.44:0/1630689688 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e00c9b70 con 0x7f90e00d0a30 2024-04-02T13:20:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.971+0000 7f90e5e60640 1 -- 172.21.15.44:0/1630689688 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90e00684f0 con 0x7f90e00d0a30 2024-04-02T13:20:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.972+0000 7f90c7fff640 1 -- 172.21.15.44:0/1630689688 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f90d40039d0 con 0x7f90e00d0a30 2024-04-02T13:20:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.972+0000 7f90c7fff640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f90cc06c940 0x7f90cc06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.972+0000 7f90c7fff640 1 -- 172.21.15.44:0/1630689688 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f90d408bbc0 con 0x7f90e00d0a30 2024-04-02T13:20:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.972+0000 7f90df7fe640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f90cc06c940 0x7f90cc06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.973+0000 7f90df7fe640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f90cc06c940 0x7f90cc06ee00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009870 tx=0x7f90d003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:40.977+0000 7f90c7fff640 1 -- 172.21.15.44:0/1630689688 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f90d4015020 con 0x7f90e00d0a30 2024-04-02T13:20:41.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.167+0000 7f90e5e60640 1 -- 172.21.15.44:0/1630689688 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90e015d700 con 0x7f90e00d0a30 2024-04-02T13:20:41.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.168+0000 7f90c7fff640 1 -- 172.21.15.44:0/1630689688 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f90d4059740 con 0x7f90e00d0a30 2024-04-02T13:20:41.170 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:41.170 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:41.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.170+0000 7f90e5e60640 1 -- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f90cc06c940 msgr2=0x7f90cc06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.170+0000 7f90e5e60640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f90cc06c940 0x7f90cc06ee00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009870 tx=0x7f90d003d040 comp rx=0 tx=0).stop 2024-04-02T13:20:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.170+0000 7f90e5e60640 1 -- 172.21.15.44:0/1630689688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e00d0a30 msgr2=0x7f90e00ca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.170+0000 7f90e5e60640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e00d0a30 0x7f90e00ca370 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f90d400c550 tx=0x7f90d400ca10 comp rx=0 tx=0).stop 2024-04-02T13:20:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.170+0000 7f90e5e60640 1 -- 172.21.15.44:0/1630689688 shutdown_connections 2024-04-02T13:20:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.170+0000 7f90e5e60640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f90cc06c940 0x7f90cc06ee00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.170+0000 7f90e5e60640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90e00d0e10 0x7f90e015fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.170+0000 7f90e5e60640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e00d0a30 0x7f90e00ca370 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.170+0000 7f90e5e60640 1 --2- 172.21.15.44:0/1630689688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00cfe30 0x7f90e0155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.170+0000 7f90e5e60640 1 -- 172.21.15.44:0/1630689688 >> 172.21.15.44:0/1630689688 conn(0x7f90e009d470 msgr2=0x7f90e00b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.170+0000 7f90e5e60640 1 -- 172.21.15.44:0/1630689688 shutdown_connections 2024-04-02T13:20:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.170+0000 7f90e5e60640 1 -- 172.21.15.44:0/1630689688 wait complete. 2024-04-02T13:20:41.183 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T13:20:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.286+0000 7fef4ac06640 1 Processor -- start 2024-04-02T13:20:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.286+0000 7fef4ac06640 1 -- start start 2024-04-02T13:20:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef4ac06640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fef440b5540 0x7fef440d1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef4ac06640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef440d21f0 0x7fef440d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef4ac06640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fef440d2b70 0x7fef44161ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef4ac06640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fef4415f920 con 0x7fef440d2b70 2024-04-02T13:20:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef4ac06640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fef440cebd0 con 0x7fef440b5540 2024-04-02T13:20:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef4ac06640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fef440b5920 con 0x7fef440d21f0 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef491a3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fef440d2b70 0x7fef44161ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef491a3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fef440d2b70 0x7fef44161ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58958/0 (socket says 172.21.15.44:58958) 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef491a3640 1 -- 172.21.15.44:0/1088313533 learned_addr learned my addr 172.21.15.44:0/1088313533 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef43fff640 1 --2- 172.21.15.44:0/1088313533 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef440d21f0 0x7fef440d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef489a2640 1 --2- 172.21.15.44:0/1088313533 >> v2:172.21.15.69:3300/0 conn(0x7fef440b5540 0x7fef440d1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef491a3640 1 -- 172.21.15.44:0/1088313533 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef440d21f0 msgr2=0x7fef440d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef491a3640 1 --2- 172.21.15.44:0/1088313533 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef440d21f0 0x7fef440d2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef491a3640 1 -- 172.21.15.44:0/1088313533 >> v2:172.21.15.69:3300/0 conn(0x7fef440b5540 msgr2=0x7fef440d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef491a3640 1 --2- 172.21.15.44:0/1088313533 >> v2:172.21.15.69:3300/0 conn(0x7fef440b5540 0x7fef440d1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef491a3640 1 -- 172.21.15.44:0/1088313533 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4415eda0 con 0x7fef440d2b70 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef43fff640 1 --2- 172.21.15.44:0/1088313533 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef440d21f0 0x7fef440d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.287+0000 7fef489a2640 1 --2- 172.21.15.44:0/1088313533 >> v2:172.21.15.69:3300/0 conn(0x7fef440b5540 0x7fef440d1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.288+0000 7fef491a3640 1 --2- 172.21.15.44:0/1088313533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef440d2b70 0x7fef44161ec0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fef4406e690 tx=0x7fef3c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6319bc20d24a54 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.288+0000 7fef437fe640 1 -- 172.21.15.44:0/1088313533 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c02ca70 con 0x7fef440d2b70 2024-04-02T13:20:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.288+0000 7fef437fe640 1 -- 172.21.15.44:0/1088313533 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef3c02cc10 con 0x7fef440d2b70 2024-04-02T13:20:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.288+0000 7fef437fe640 1 -- 172.21.15.44:0/1088313533 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c0343f0 con 0x7fef440d2b70 2024-04-02T13:20:41.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.288+0000 7fef4ac06640 1 -- 172.21.15.44:0/1088313533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef440d2b70 msgr2=0x7fef44161ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:41.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.288+0000 7fef4ac06640 1 --2- 172.21.15.44:0/1088313533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef440d2b70 0x7fef44161ec0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fef4406e690 tx=0x7fef3c02bed0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.288+0000 7fef4ac06640 1 -- 172.21.15.44:0/1088313533 shutdown_connections 2024-04-02T13:20:41.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.288+0000 7fef4ac06640 1 --2- 172.21.15.44:0/1088313533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef440d2b70 0x7fef44161ec0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.288+0000 7fef4ac06640 1 --2- 172.21.15.44:0/1088313533 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef440d21f0 0x7fef440d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.288+0000 7fef4ac06640 1 --2- 172.21.15.44:0/1088313533 >> v2:172.21.15.69:3300/0 conn(0x7fef440b5540 0x7fef440d1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.288+0000 7fef4ac06640 1 -- 172.21.15.44:0/1088313533 >> 172.21.15.44:0/1088313533 conn(0x7fef4409d390 msgr2=0x7fef440d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.289+0000 7fef4ac06640 1 -- 172.21.15.44:0/1088313533 shutdown_connections 2024-04-02T13:20:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.289+0000 7fef4ac06640 1 -- 172.21.15.44:0/1088313533 wait complete. 2024-04-02T13:20:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.290+0000 7fef4ac06640 1 Processor -- start 2024-04-02T13:20:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.290+0000 7fef4ac06640 1 -- start start 2024-04-02T13:20:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef4ac06640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef440b5540 0x7fef44151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef4ac06640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef440d21f0 0x7fef440c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef4ac06640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef440d2b70 0x7fef440c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef4ac06640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fef4415f640 con 0x7fef440d2b70 2024-04-02T13:20:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef4ac06640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fef4415f360 con 0x7fef440d21f0 2024-04-02T13:20:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef4ac06640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fef4415fcc0 con 0x7fef440b5540 2024-04-02T13:20:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef491a3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef440d2b70 0x7fef440c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef491a3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef440d2b70 0x7fef440c9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58966/0 (socket says 172.21.15.44:58966) 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef491a3640 1 -- 172.21.15.44:0/3188721083 learned_addr learned my addr 172.21.15.44:0/3188721083 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef43fff640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef440d21f0 0x7fef440c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef489a2640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef440b5540 0x7fef44151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef491a3640 1 -- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef440b5540 msgr2=0x7fef44151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef491a3640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef440b5540 0x7fef44151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef491a3640 1 -- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef440d21f0 msgr2=0x7fef440c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef491a3640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef440d21f0 0x7fef440c9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef491a3640 1 -- 172.21.15.44:0/3188721083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef44151a70 con 0x7fef440d2b70 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef489a2640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef440b5540 0x7fef44151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef491a3640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef440d2b70 0x7fef440c9ca0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009f40 tx=0x7fef3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef417fa640 1 -- 172.21.15.44:0/3188721083 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c044020 con 0x7fef440d2b70 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef417fa640 1 -- 172.21.15.44:0/3188721083 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef3c040690 con 0x7fef440d2b70 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef417fa640 1 -- 172.21.15.44:0/3188721083 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c009510 con 0x7fef440d2b70 2024-04-02T13:20:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef4ac06640 1 -- 172.21.15.44:0/3188721083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef441e6c90 con 0x7fef440d2b70 2024-04-02T13:20:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.291+0000 7fef4ac06640 1 -- 172.21.15.44:0/3188721083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef441e7250 con 0x7fef440d2b70 2024-04-02T13:20:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.292+0000 7fef4ac06640 1 -- 172.21.15.44:0/3188721083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef440684f0 con 0x7fef440d2b70 2024-04-02T13:20:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.293+0000 7fef417fa640 1 -- 172.21.15.44:0/3188721083 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fef3c047020 con 0x7fef440d2b70 2024-04-02T13:20:41.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.293+0000 7fef417fa640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fef2806c8f0 0x7fef2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:41.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.293+0000 7fef417fa640 1 -- 172.21.15.44:0/3188721083 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fef3c0aea00 con 0x7fef440d2b70 2024-04-02T13:20:41.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.293+0000 7fef489a2640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fef2806c8f0 0x7fef2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:41.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.294+0000 7fef489a2640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fef2806c8f0 0x7fef2806edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fef38006fd0 tx=0x7fef38008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:41.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.298+0000 7fef417fa640 1 -- 172.21.15.44:0/3188721083 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fef3c04e030 con 0x7fef440d2b70 2024-04-02T13:20:41.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.520+0000 7fef4ac06640 1 -- 172.21.15.44:0/3188721083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fef440d4760 con 0x7fef440d2b70 2024-04-02T13:20:41.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.963+0000 7fef417fa640 1 -- 172.21.15.44:0/3188721083 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7fef3c07cc00 con 0x7fef440d2b70 2024-04-02T13:20:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.966+0000 7fef4ac06640 1 -- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fef2806c8f0 msgr2=0x7fef2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.966+0000 7fef4ac06640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fef2806c8f0 0x7fef2806edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fef38006fd0 tx=0x7fef38008040 comp rx=0 tx=0).stop 2024-04-02T13:20:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.966+0000 7fef4ac06640 1 -- 172.21.15.44:0/3188721083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef440d2b70 msgr2=0x7fef440c9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.966+0000 7fef4ac06640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef440d2b70 0x7fef440c9ca0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009f40 tx=0x7fef3c004a40 comp rx=0 tx=0).stop 2024-04-02T13:20:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.966+0000 7fef4ac06640 1 -- 172.21.15.44:0/3188721083 shutdown_connections 2024-04-02T13:20:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.966+0000 7fef4ac06640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fef2806c8f0 0x7fef2806edb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.966+0000 7fef4ac06640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef440d2b70 0x7fef440c9ca0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.966+0000 7fef4ac06640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef440d21f0 0x7fef440c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.966+0000 7fef4ac06640 1 --2- 172.21.15.44:0/3188721083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef440b5540 0x7fef44151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:41.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.966+0000 7fef4ac06640 1 -- 172.21.15.44:0/3188721083 >> 172.21.15.44:0/3188721083 conn(0x7fef4409d390 msgr2=0x7fef440d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:41.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.966+0000 7fef4ac06640 1 -- 172.21.15.44:0/3188721083 shutdown_connections 2024-04-02T13:20:41.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:41.966+0000 7fef4ac06640 1 -- 172.21.15.44:0/3188721083 wait complete. 2024-04-02T13:20:41.981 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-02T13:20:41.981 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T13:20:42.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.088+0000 7fd53c2f2640 1 Processor -- start 2024-04-02T13:20:42.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.088+0000 7fd53c2f2640 1 -- start start 2024-04-02T13:20:42.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.089+0000 7fd53c2f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd5340d0a30 0x7fd53406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:42.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.089+0000 7fd53c2f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5340d2040 0x7fd5340d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:42.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.089+0000 7fd53c2f2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5340d0000 con 0x7fd5340d0e10 2024-04-02T13:20:42.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.089+0000 7fd53c2f2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd53406cb00 con 0x7fd5340d0a30 2024-04-02T13:20:42.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.089+0000 7fd53c2f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd53406c1d0 con 0x7fd5340d2040 2024-04-02T13:20:42.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.090+0000 7fd53a08e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd5340d0a30 0x7fd53406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:42.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.090+0000 7fd53988d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd5340d0e10 0x7fd5340cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56350/0 (socket says 172.21.15.44:56350) 2024-04-02T13:20:42.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.090+0000 7fd53988d640 1 -- 172.21.15.44:0/639763088 learned_addr learned my addr 172.21.15.44:0/639763088 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:42.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.090+0000 7fd53a88f640 1 --2- 172.21.15.44:0/639763088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5340d2040 0x7fd5340d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:42.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.090+0000 7fd53908c640 1 -- 172.21.15.44:0/639763088 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1729036944 0 0) 0x7fd5340d0000 con 0x7fd5340d0e10 2024-04-02T13:20:42.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.090+0000 7fd53908c640 1 -- 172.21.15.44:0/639763088 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd51c003520 con 0x7fd5340d0e10 2024-04-02T13:20:42.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.090+0000 7fd53a08e640 1 -- 172.21.15.44:0/639763088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5340d2040 msgr2=0x7fd5340d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:42.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.090+0000 7fd53a08e640 1 --2- 172.21.15.44:0/639763088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5340d2040 0x7fd5340d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:42.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.090+0000 7fd53a08e640 1 -- 172.21.15.44:0/639763088 >> v1:172.21.15.44:6789/0 conn(0x7fd5340d0e10 legacy=0x7fd5340cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:42.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.090+0000 7fd53a08e640 1 -- 172.21.15.44:0/639763088 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53406c4b0 con 0x7fd5340d0a30 2024-04-02T13:20:42.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.090+0000 7fd53a88f640 1 --2- 172.21.15.44:0/639763088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5340d2040 0x7fd5340d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:42.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.091+0000 7fd53a08e640 1 --2- 172.21.15.44:0/639763088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5340d0a30 0x7fd53406cf90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0097a0 tx=0x7fd52c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d75c3a38727fa1a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:42.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.091+0000 7fd53908c640 1 -- 172.21.15.44:0/639763088 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c0315a0 con 0x7fd5340d0a30 2024-04-02T13:20:42.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.091+0000 7fd53908c640 1 -- 172.21.15.44:0/639763088 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd52c037070 con 0x7fd5340d0a30 2024-04-02T13:20:42.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.091+0000 7fd53908c640 1 -- 172.21.15.44:0/639763088 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c031960 con 0x7fd5340d0a30 2024-04-02T13:20:42.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.091+0000 7fd53c2f2640 1 -- 172.21.15.44:0/639763088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5340d0a30 msgr2=0x7fd53406cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:42.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.091+0000 7fd53c2f2640 1 --2- 172.21.15.44:0/639763088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5340d0a30 0x7fd53406cf90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0097a0 tx=0x7fd52c02b840 comp rx=0 tx=0).stop 2024-04-02T13:20:42.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.091+0000 7fd53c2f2640 1 -- 172.21.15.44:0/639763088 shutdown_connections 2024-04-02T13:20:42.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.091+0000 7fd53c2f2640 1 --2- 172.21.15.44:0/639763088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5340d2040 0x7fd5340d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:42.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.091+0000 7fd53c2f2640 1 --2- 172.21.15.44:0/639763088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5340d0a30 0x7fd53406cf90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:42.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.091+0000 7fd53c2f2640 1 -- 172.21.15.44:0/639763088 >> 172.21.15.44:0/639763088 conn(0x7fd53409d470 msgr2=0x7fd5340d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:42.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.091+0000 7fd53c2f2640 1 -- 172.21.15.44:0/639763088 shutdown_connections 2024-04-02T13:20:42.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.091+0000 7fd53c2f2640 1 -- 172.21.15.44:0/639763088 wait complete. 2024-04-02T13:20:42.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.093+0000 7fd53c2f2640 1 Processor -- start 2024-04-02T13:20:42.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.093+0000 7fd53c2f2640 1 -- start start 2024-04-02T13:20:42.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.093+0000 7fd53c2f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5340d0a30 0x7fd5340cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:42.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.093+0000 7fd53c2f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5340d0e10 0x7fd53415fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:42.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.093+0000 7fd53c2f2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5340d2040 0x7fd53415ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:42.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.093+0000 7fd53c2f2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd53415e6f0 con 0x7fd5340d0a30 2024-04-02T13:20:42.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.093+0000 7fd53c2f2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd5340d1270 con 0x7fd5340d2040 2024-04-02T13:20:42.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.093+0000 7fd53c2f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd53415e870 con 0x7fd5340d0e10 2024-04-02T13:20:42.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.093+0000 7fd53a08e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5340d0a30 0x7fd5340cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:42.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.093+0000 7fd53a08e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5340d0a30 0x7fd5340cdd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58988/0 (socket says 172.21.15.44:58988) 2024-04-02T13:20:42.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.093+0000 7fd53a08e640 1 -- 172.21.15.44:0/1116161007 learned_addr learned my addr 172.21.15.44:0/1116161007 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:42.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.094+0000 7fd53a08e640 1 -- 172.21.15.44:0/1116161007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5340d0e10 msgr2=0x7fd53415fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:42.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.094+0000 7fd53988d640 1 --2- 172.21.15.44:0/1116161007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5340d0e10 0x7fd53415fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:42.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.094+0000 7fd53a08e640 1 --2- 172.21.15.44:0/1116161007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5340d0e10 0x7fd53415fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:42.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.094+0000 7fd53a08e640 1 -- 172.21.15.44:0/1116161007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5340d2040 msgr2=0x7fd53415ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:42.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.094+0000 7fd53a08e640 1 --2- 172.21.15.44:0/1116161007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5340d2040 0x7fd53415ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:42.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.094+0000 7fd53a08e640 1 -- 172.21.15.44:0/1116161007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5340ce280 con 0x7fd5340d0a30 2024-04-02T13:20:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.094+0000 7fd53a08e640 1 --2- 172.21.15.44:0/1116161007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5340d0a30 0x7fd5340cdd40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd52c034040 tx=0x7fd52c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.094+0000 7fd532ffd640 1 -- 172.21.15.44:0/1116161007 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c032630 con 0x7fd5340d0a30 2024-04-02T13:20:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.094+0000 7fd532ffd640 1 -- 172.21.15.44:0/1116161007 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd52c0327d0 con 0x7fd5340d0a30 2024-04-02T13:20:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.094+0000 7fd53c2f2640 1 -- 172.21.15.44:0/1116161007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5340b5d90 con 0x7fd5340d0a30 2024-04-02T13:20:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.094+0000 7fd532ffd640 1 -- 172.21.15.44:0/1116161007 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c032bc0 con 0x7fd5340d0a30 2024-04-02T13:20:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.094+0000 7fd53c2f2640 1 -- 172.21.15.44:0/1116161007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5340b6350 con 0x7fd5340d0a30 2024-04-02T13:20:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.095+0000 7fd53c2f2640 1 -- 172.21.15.44:0/1116161007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5340684f0 con 0x7fd5340d0a30 2024-04-02T13:20:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.095+0000 7fd532ffd640 1 -- 172.21.15.44:0/1116161007 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd52c03b3f0 con 0x7fd5340d0a30 2024-04-02T13:20:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.096+0000 7fd532ffd640 1 --2- 172.21.15.44:0/1116161007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd52006c8f0 0x7fd52006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.096+0000 7fd532ffd640 1 -- 172.21.15.44:0/1116161007 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd52c02f030 con 0x7fd5340d0a30 2024-04-02T13:20:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.096+0000 7fd53988d640 1 --2- 172.21.15.44:0/1116161007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd52006c8f0 0x7fd52006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:42.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.097+0000 7fd53988d640 1 --2- 172.21.15.44:0/1116161007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd52006c8f0 0x7fd52006edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd5340b6c80 tx=0x7fd528009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:42.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.100+0000 7fd532ffd640 1 -- 172.21.15.44:0/1116161007 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd52c040070 con 0x7fd5340d0a30 2024-04-02T13:20:42.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.330+0000 7fd53c2f2640 1 -- 172.21.15.44:0/1116161007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd5340d17c0 con 0x7fd5340d0a30 2024-04-02T13:20:42.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.971+0000 7fd532ffd640 1 -- 172.21.15.44:0/1116161007 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) ==== 122+0+0 (secure 0 0 0) 0x7fd52c07bfc0 con 0x7fd5340d0a30 2024-04-02T13:20:42.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.974+0000 7fd53c2f2640 1 -- 172.21.15.44:0/1116161007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd52006c8f0 msgr2=0x7fd52006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:42.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.974+0000 7fd53c2f2640 1 --2- 172.21.15.44:0/1116161007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd52006c8f0 0x7fd52006edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd5340b6c80 tx=0x7fd528009040 comp rx=0 tx=0).stop 2024-04-02T13:20:42.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.974+0000 7fd53c2f2640 1 -- 172.21.15.44:0/1116161007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5340d0a30 msgr2=0x7fd5340cdd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:42.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.974+0000 7fd53c2f2640 1 --2- 172.21.15.44:0/1116161007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5340d0a30 0x7fd5340cdd40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd52c034040 tx=0x7fd52c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:42.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.974+0000 7fd53c2f2640 1 -- 172.21.15.44:0/1116161007 shutdown_connections 2024-04-02T13:20:42.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.974+0000 7fd53c2f2640 1 --2- 172.21.15.44:0/1116161007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd52006c8f0 0x7fd52006edb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:42.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.974+0000 7fd53c2f2640 1 --2- 172.21.15.44:0/1116161007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5340d2040 0x7fd53415ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:42.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.974+0000 7fd53c2f2640 1 --2- 172.21.15.44:0/1116161007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5340d0e10 0x7fd53415fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:42.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.974+0000 7fd53c2f2640 1 --2- 172.21.15.44:0/1116161007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5340d0a30 0x7fd5340cdd40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:42.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.974+0000 7fd53c2f2640 1 -- 172.21.15.44:0/1116161007 >> 172.21.15.44:0/1116161007 conn(0x7fd53409d470 msgr2=0x7fd5340b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:42.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.974+0000 7fd53c2f2640 1 -- 172.21.15.44:0/1116161007 shutdown_connections 2024-04-02T13:20:42.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:42.974+0000 7fd53c2f2640 1 -- 172.21.15.44:0/1116161007 wait complete. 2024-04-02T13:20:42.990 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T13:20:42.990 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:20:43.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.095+0000 7fbc81159640 1 Processor -- start 2024-04-02T13:20:43.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.095+0000 7fbc81159640 1 -- start start 2024-04-02T13:20:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc81159640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c0d0a30 0x7fbc7c06cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc81159640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbc7c0d2040 0x7fbc7c0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc81159640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc7c0d0000 con 0x7fbc7c0d0e10 2024-04-02T13:20:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc81159640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbc7c06cb00 con 0x7fbc7c0d2040 2024-04-02T13:20:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc81159640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7c06c1d0 con 0x7fbc7c0d0a30 2024-04-02T13:20:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc7a59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbc7c0d0e10 0x7fbc7c0cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56360/0 (socket says 172.21.15.44:56360) 2024-04-02T13:20:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc7a59c640 1 -- 172.21.15.44:0/500551987 learned_addr learned my addr 172.21.15.44:0/500551987 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 984307289 0 0) 0x7fbc7c0d0000 con 0x7fbc7c0d0e10 2024-04-02T13:20:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc7ad9d640 1 --2- 172.21.15.44:0/500551987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c0d0a30 0x7fbc7c06cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc68003520 con 0x7fbc7c0d0e10 2024-04-02T13:20:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc7b59e640 1 --2- 172.21.15.44:0/500551987 >> v2:172.21.15.69:3300/0 conn(0x7fbc7c0d2040 0x7fbc7c0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2400838746 0 0) 0x7fbc68003520 con 0x7fbc7c0d0e10 2024-04-02T13:20:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c0d0a30 msgr2=0x7fbc7c06cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc79d9b640 1 --2- 172.21.15.44:0/500551987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c0d0a30 0x7fbc7c06cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 >> v2:172.21.15.69:3300/0 conn(0x7fbc7c0d2040 msgr2=0x7fbc7c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc79d9b640 1 --2- 172.21.15.44:0/500551987 >> v2:172.21.15.69:3300/0 conn(0x7fbc7c0d2040 0x7fbc7c0d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c06c4b0 con 0x7fbc7c0d0e10 2024-04-02T13:20:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fbc64001670 con 0x7fbc7c0d0e10 2024-04-02T13:20:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 >> v1:172.21.15.44:6789/0 conn(0x7fbc7c0d0e10 legacy=0x7fbc7c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc79d9b640 1 --2- 172.21.15.44:0/500551987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c0d0a30 0x7fbc680039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc79d9b640 1 --2- 172.21.15.44:0/500551987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc68003f30 0x7fbc68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.096+0000 7fbc79d9b640 1 --2- 172.21.15.44:0/500551987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc68004d40 0x7fbc680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.097+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c06cb00 con 0x7fbc68003f30 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.097+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c06c1d0 con 0x7fbc68004d40 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.097+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbc6800ba10 con 0x7fbc7c0d0a30 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.097+0000 7fbc7a59c640 1 --2- 172.21.15.44:0/500551987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc68003f30 0x7fbc68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.097+0000 7fbc7ad9d640 1 --2- 172.21.15.44:0/500551987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c0d0a30 0x7fbc680039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.097+0000 7fbc7b59e640 1 --2- 172.21.15.44:0/500551987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc68004d40 0x7fbc680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.097+0000 7fbc7a59c640 1 -- 172.21.15.44:0/500551987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c0d0a30 msgr2=0x7fbc680039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.097+0000 7fbc7a59c640 1 --2- 172.21.15.44:0/500551987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c0d0a30 0x7fbc680039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.097+0000 7fbc7a59c640 1 -- 172.21.15.44:0/500551987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc68004d40 msgr2=0x7fbc680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.097+0000 7fbc7a59c640 1 --2- 172.21.15.44:0/500551987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc68004d40 0x7fbc680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.097+0000 7fbc7a59c640 1 -- 172.21.15.44:0/500551987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc68003520 con 0x7fbc68003f30 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.098+0000 7fbc7a59c640 1 --2- 172.21.15.44:0/500551987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc68003f30 0x7fbc68004800 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400c210 tx=0x7fbc68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04202c73a846b1a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:43.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.098+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64007d70 con 0x7fbc68003f30 2024-04-02T13:20:43.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.098+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc64030070 con 0x7fbc68003f30 2024-04-02T13:20:43.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.098+0000 7fbc79d9b640 1 -- 172.21.15.44:0/500551987 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6402cd60 con 0x7fbc68003f30 2024-04-02T13:20:43.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.098+0000 7fbc81159640 1 -- 172.21.15.44:0/500551987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc68003f30 msgr2=0x7fbc68004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.098+0000 7fbc81159640 1 --2- 172.21.15.44:0/500551987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc68003f30 0x7fbc68004800 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400c210 tx=0x7fbc68005830 comp rx=0 tx=0).stop 2024-04-02T13:20:43.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.098+0000 7fbc7b59e640 1 -- 172.21.15.44:0/500551987 reap_dead start 2024-04-02T13:20:43.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.098+0000 7fbc81159640 1 -- 172.21.15.44:0/500551987 shutdown_connections 2024-04-02T13:20:43.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.098+0000 7fbc81159640 1 -- 172.21.15.44:0/500551987 >> 172.21.15.44:0/500551987 conn(0x7fbc7c09d470 msgr2=0x7fbc7c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:43.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.098+0000 7fbc81159640 1 -- 172.21.15.44:0/500551987 shutdown_connections 2024-04-02T13:20:43.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.098+0000 7fbc81159640 1 -- 172.21.15.44:0/500551987 wait complete. 2024-04-02T13:20:43.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.100+0000 7fbc81159640 1 Processor -- start 2024-04-02T13:20:43.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.100+0000 7fbc81159640 1 -- start start 2024-04-02T13:20:43.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc81159640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc68003f30 0x7fbc7c0ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc81159640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c0cce60 0x7fbc7c0c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc81159640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc7c1521b0 0x7fbc7c152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc81159640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c15e6f0 con 0x7fbc7c1521b0 2024-04-02T13:20:43.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc81159640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c0d1270 con 0x7fbc68003f30 2024-04-02T13:20:43.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc81159640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7c15e870 con 0x7fbc7c0cce60 2024-04-02T13:20:43.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc7b59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc7c1521b0 0x7fbc7c152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc7b59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc7c1521b0 0x7fbc7c152660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59002/0 (socket says 172.21.15.44:59002) 2024-04-02T13:20:43.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc7b59e640 1 -- 172.21.15.44:0/3461853570 learned_addr learned my addr 172.21.15.44:0/3461853570 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:43.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc7a59c640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c0cce60 0x7fbc7c0c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc7ad9d640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc68003f30 0x7fbc7c0ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc7b59e640 1 -- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c0cce60 msgr2=0x7fbc7c0c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc7b59e640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c0cce60 0x7fbc7c0c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc7b59e640 1 -- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc68003f30 msgr2=0x7fbc7c0ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc7b59e640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc68003f30 0x7fbc7c0ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc7b59e640 1 -- 172.21.15.44:0/3461853570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c0c85a0 con 0x7fbc7c1521b0 2024-04-02T13:20:43.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc7b59e640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc7c1521b0 0x7fbc7c152660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c0cebf0 tx=0x7fbc6c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc737fe640 1 -- 172.21.15.44:0/3461853570 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00fde0 con 0x7fbc7c1521b0 2024-04-02T13:20:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc737fe640 1 -- 172.21.15.44:0/3461853570 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc6c01d070 con 0x7fbc7c1521b0 2024-04-02T13:20:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc81159640 1 -- 172.21.15.44:0/3461853570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7c0c8880 con 0x7fbc7c1521b0 2024-04-02T13:20:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.101+0000 7fbc737fe640 1 -- 172.21.15.44:0/3461853570 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c015670 con 0x7fbc7c1521b0 2024-04-02T13:20:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.102+0000 7fbc81159640 1 -- 172.21.15.44:0/3461853570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7c15ff30 con 0x7fbc7c1521b0 2024-04-02T13:20:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.103+0000 7fbc81159640 1 -- 172.21.15.44:0/3461853570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc7c15fa40 con 0x7fbc7c1521b0 2024-04-02T13:20:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.103+0000 7fbc737fe640 1 -- 172.21.15.44:0/3461853570 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbc6c015810 con 0x7fbc7c1521b0 2024-04-02T13:20:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.103+0000 7fbc737fe640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbc5c06c8f0 0x7fbc5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.103+0000 7fbc737fe640 1 -- 172.21.15.44:0/3461853570 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbc6c08c8f0 con 0x7fbc7c1521b0 2024-04-02T13:20:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.103+0000 7fbc7ad9d640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbc5c06c8f0 0x7fbc5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.104+0000 7fbc7ad9d640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbc5c06c8f0 0x7fbc5c06edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c0cbb20 tx=0x7fbc74008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:43.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.108+0000 7fbc737fe640 1 -- 172.21.15.44:0/3461853570 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbc6c010040 con 0x7fbc7c1521b0 2024-04-02T13:20:43.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.334+0000 7fbc81159640 1 -- 172.21.15.44:0/3461853570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc7c0684f0 con 0x7fbc7c1521b0 2024-04-02T13:20:43.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.335+0000 7fbc737fe640 1 -- 172.21.15.44:0/3461853570 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7fbc6c05a9e0 con 0x7fbc7c1521b0 2024-04-02T13:20:43.352 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:43.352 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4040417710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4040417710},{"type":"v1","addr":"172.21.15.69:6836","nonce":4040417710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2772068060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2772068060},{"type":"v1","addr":"172.21.15.44:6833","nonce":2772068060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1985405150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1985405150},{"type":"v1","addr":"172.21.15.69:6837","nonce":1985405150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:20:20.247358+0000","modified":"2024-04-02T13:20:42.332570+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/2282156931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2282156931},{"type":"v1","addr":"172.21.15.44:6835","nonce":2282156931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-04-02T13:20:43.354 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 9 2024-04-02T13:20:43.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.336+0000 7fbc81159640 1 -- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbc5c06c8f0 msgr2=0x7fbc5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.336+0000 7fbc81159640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbc5c06c8f0 0x7fbc5c06edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c0cbb20 tx=0x7fbc74008040 comp rx=0 tx=0).stop 2024-04-02T13:20:43.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.336+0000 7fbc81159640 1 -- 172.21.15.44:0/3461853570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc7c1521b0 msgr2=0x7fbc7c152660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.336+0000 7fbc81159640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc7c1521b0 0x7fbc7c152660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c0cebf0 tx=0x7fbc6c00f580 comp rx=0 tx=0).stop 2024-04-02T13:20:43.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.337+0000 7fbc81159640 1 -- 172.21.15.44:0/3461853570 shutdown_connections 2024-04-02T13:20:43.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.337+0000 7fbc81159640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbc5c06c8f0 0x7fbc5c06edb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.337+0000 7fbc81159640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc7c1521b0 0x7fbc7c152660 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.337+0000 7fbc81159640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c0cce60 0x7fbc7c0c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.337+0000 7fbc81159640 1 --2- 172.21.15.44:0/3461853570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc68003f30 0x7fbc7c0ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.337+0000 7fbc81159640 1 -- 172.21.15.44:0/3461853570 >> 172.21.15.44:0/3461853570 conn(0x7fbc7c09d470 msgr2=0x7fbc7c15c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:43.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.337+0000 7fbc81159640 1 -- 172.21.15.44:0/3461853570 shutdown_connections 2024-04-02T13:20:43.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.337+0000 7fbc81159640 1 -- 172.21.15.44:0/3461853570 wait complete. 2024-04-02T13:20:43.355 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:43.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.454+0000 7f2109774640 1 Processor -- start 2024-04-02T13:20:43.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.454+0000 7f2109774640 1 -- start start 2024-04-02T13:20:43.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f2109774640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f210415e140 0x7f21040d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f2109774640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f210415e520 0x7f21040d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f2109774640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21040d2b70 0x7f21040d0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f2109774640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f21040b9f50 con 0x7f210415e520 2024-04-02T13:20:43.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f2109774640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f21040b70b0 con 0x7f210415e140 2024-04-02T13:20:43.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f2109774640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f21040b7390 con 0x7f21040d2b70 2024-04-02T13:20:43.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f20fbfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f210415e520 0x7f21040d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f20fbfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f210415e520 0x7f21040d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59010/0 (socket says 172.21.15.44:59010) 2024-04-02T13:20:43.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f20fbfff640 1 -- 172.21.15.44:0/2575216244 learned_addr learned my addr 172.21.15.44:0/2575216244 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:43.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f2102ffd640 1 --2- 172.21.15.44:0/2575216244 >> v2:172.21.15.69:3300/0 conn(0x7f210415e140 0x7f21040d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f21037fe640 1 --2- 172.21.15.44:0/2575216244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21040d2b70 0x7f21040d0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f20fbfff640 1 -- 172.21.15.44:0/2575216244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21040d2b70 msgr2=0x7f21040d0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f20fbfff640 1 --2- 172.21.15.44:0/2575216244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21040d2b70 0x7f21040d0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f20fbfff640 1 -- 172.21.15.44:0/2575216244 >> v2:172.21.15.69:3300/0 conn(0x7f210415e140 msgr2=0x7f21040d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f20fbfff640 1 --2- 172.21.15.44:0/2575216244 >> v2:172.21.15.69:3300/0 conn(0x7f210415e140 0x7f21040d2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f20fbfff640 1 -- 172.21.15.44:0/2575216244 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21040b7670 con 0x7f210415e520 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.455+0000 7f2102ffd640 1 --2- 172.21.15.44:0/2575216244 >> v2:172.21.15.69:3300/0 conn(0x7f210415e140 0x7f21040d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.456+0000 7f20fbfff640 1 --2- 172.21.15.44:0/2575216244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210415e520 0x7f21040d0a40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f210408e6d0 tx=0x7f20ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722944721df5dc4c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.456+0000 7f21027fc640 1 -- 172.21.15.44:0/2575216244 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec035020 con 0x7f210415e520 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.456+0000 7f21027fc640 1 -- 172.21.15.44:0/2575216244 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20ec031870 con 0x7f210415e520 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.456+0000 7f21027fc640 1 -- 172.21.15.44:0/2575216244 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec031c40 con 0x7f210415e520 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.456+0000 7f2109774640 1 -- 172.21.15.44:0/2575216244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210415e520 msgr2=0x7f21040d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.456+0000 7f2109774640 1 --2- 172.21.15.44:0/2575216244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210415e520 0x7f21040d0a40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f210408e6d0 tx=0x7f20ec02b910 comp rx=0 tx=0).stop 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.457+0000 7f2109774640 1 -- 172.21.15.44:0/2575216244 shutdown_connections 2024-04-02T13:20:43.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.457+0000 7f2109774640 1 --2- 172.21.15.44:0/2575216244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21040d2b70 0x7f21040d0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.457+0000 7f2109774640 1 --2- 172.21.15.44:0/2575216244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210415e520 0x7f21040d0a40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.457+0000 7f2109774640 1 --2- 172.21.15.44:0/2575216244 >> v2:172.21.15.69:3300/0 conn(0x7f210415e140 0x7f21040d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.457+0000 7f2109774640 1 -- 172.21.15.44:0/2575216244 >> 172.21.15.44:0/2575216244 conn(0x7f210409d390 msgr2=0x7f21040d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:43.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.457+0000 7f2109774640 1 -- 172.21.15.44:0/2575216244 shutdown_connections 2024-04-02T13:20:43.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.457+0000 7f2109774640 1 -- 172.21.15.44:0/2575216244 wait complete. 2024-04-02T13:20:43.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.458+0000 7f2109774640 1 Processor -- start 2024-04-02T13:20:43.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.458+0000 7f2109774640 1 -- start start 2024-04-02T13:20:43.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.458+0000 7f2109774640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21040d2b70 0x7f2104158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.458+0000 7f2109774640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f210415e140 0x7f2104157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.458+0000 7f2109774640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f210415e520 0x7f2104153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.458+0000 7f2109774640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f21040b4cc0 con 0x7f21040d2b70 2024-04-02T13:20:43.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.458+0000 7f2109774640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f21040d4ee0 con 0x7f210415e140 2024-04-02T13:20:43.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.458+0000 7f2109774640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f21040b4e40 con 0x7f210415e520 2024-04-02T13:20:43.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f2102ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21040d2b70 0x7f2104158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f2102ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21040d2b70 0x7f2104158490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59020/0 (socket says 172.21.15.44:59020) 2024-04-02T13:20:43.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f2102ffd640 1 -- 172.21.15.44:0/983214141 learned_addr learned my addr 172.21.15.44:0/983214141 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:43.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f21037fe640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f210415e520 0x7f2104153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f20fbfff640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f210415e140 0x7f2104157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f2102ffd640 1 -- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f210415e520 msgr2=0x7f2104153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f2102ffd640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f210415e520 0x7f2104153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f2102ffd640 1 -- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f210415e140 msgr2=0x7f2104157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f2102ffd640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f210415e140 0x7f2104157b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f2102ffd640 1 -- 172.21.15.44:0/983214141 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2104153c20 con 0x7f21040d2b70 2024-04-02T13:20:43.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f2102ffd640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21040d2b70 0x7f2104158490 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f20f400c7f0 tx=0x7f20f400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:43.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f20fb7fe640 1 -- 172.21.15.44:0/983214141 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4015070 con 0x7f21040d2b70 2024-04-02T13:20:43.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f20fb7fe640 1 -- 172.21.15.44:0/983214141 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20f4007a30 con 0x7f21040d2b70 2024-04-02T13:20:43.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f20fb7fe640 1 -- 172.21.15.44:0/983214141 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4007e00 con 0x7f21040d2b70 2024-04-02T13:20:43.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f2109774640 1 -- 172.21.15.44:0/983214141 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21041571d0 con 0x7f21040d2b70 2024-04-02T13:20:43.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.459+0000 7f2109774640 1 -- 172.21.15.44:0/983214141 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2104157790 con 0x7f21040d2b70 2024-04-02T13:20:43.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.460+0000 7f2109774640 1 -- 172.21.15.44:0/983214141 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f210415f6c0 con 0x7f21040d2b70 2024-04-02T13:20:43.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.461+0000 7f20fb7fe640 1 -- 172.21.15.44:0/983214141 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f20f4018070 con 0x7f21040d2b70 2024-04-02T13:20:43.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.461+0000 7f20fb7fe640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20e406c8f0 0x7f20e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.461+0000 7f20fb7fe640 1 -- 172.21.15.44:0/983214141 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f20f408bd40 con 0x7f21040d2b70 2024-04-02T13:20:43.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.461+0000 7f20fbfff640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20e406c8f0 0x7f20e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.462+0000 7f20fbfff640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20e406c8f0 0x7f20e406edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f20ec036000 tx=0x7f20ec008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:43.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.465+0000 7f20fb7fe640 1 -- 172.21.15.44:0/983214141 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f210415f6c0 con 0x7f21040d2b70 2024-04-02T13:20:43.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.650+0000 7f2109774640 1 -- 172.21.15.44:0/983214141 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f210406c030 con 0x7f21040d2b70 2024-04-02T13:20:43.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.651+0000 7f20fb7fe640 1 -- 172.21.15.44:0/983214141 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f20f4059ec0 con 0x7f21040d2b70 2024-04-02T13:20:43.654 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:43.654 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:43.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.653+0000 7f2109774640 1 -- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20e406c8f0 msgr2=0x7f20e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.653+0000 7f2109774640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20e406c8f0 0x7f20e406edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f20ec036000 tx=0x7f20ec008030 comp rx=0 tx=0).stop 2024-04-02T13:20:43.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.653+0000 7f2109774640 1 -- 172.21.15.44:0/983214141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21040d2b70 msgr2=0x7f2104158490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.653+0000 7f2109774640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21040d2b70 0x7f2104158490 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f20f400c7f0 tx=0x7f20f400cbf0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.653+0000 7f2109774640 1 -- 172.21.15.44:0/983214141 shutdown_connections 2024-04-02T13:20:43.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.653+0000 7f2109774640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20e406c8f0 0x7f20e406edb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.653+0000 7f2109774640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f210415e520 0x7f2104153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.653+0000 7f2109774640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f210415e140 0x7f2104157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.653+0000 7f2109774640 1 --2- 172.21.15.44:0/983214141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21040d2b70 0x7f2104158490 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.653+0000 7f2109774640 1 -- 172.21.15.44:0/983214141 >> 172.21.15.44:0/983214141 conn(0x7f210409d390 msgr2=0x7f210406cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:43.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.653+0000 7f2109774640 1 -- 172.21.15.44:0/983214141 shutdown_connections 2024-04-02T13:20:43.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.653+0000 7f2109774640 1 -- 172.21.15.44:0/983214141 wait complete. 2024-04-02T13:20:43.667 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T13:20:43.667 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:20:43.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.766+0000 7ffa0057b640 1 Processor -- start 2024-04-02T13:20:43.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.766+0000 7ffa0057b640 1 -- start start 2024-04-02T13:20:43.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.767+0000 7ffa0057b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff9f815e550 0x7ff9f806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.767+0000 7ffa0057b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f815f040 0x7ff9f80d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.767+0000 7ffa0057b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9f815fb50 con 0x7ff9f806d530 2024-04-02T13:20:43.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.767+0000 7ffa0057b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff9f806d970 con 0x7ff9f815e550 2024-04-02T13:20:43.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.767+0000 7ffa0057b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff9f806ca90 con 0x7ff9f815f040 2024-04-02T13:20:43.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.767+0000 7ff9fdb16640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff9f806d530 0x7ff9f815e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56368/0 (socket says 172.21.15.44:56368) 2024-04-02T13:20:43.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.767+0000 7ff9fdb16640 1 -- 172.21.15.44:0/1697335943 learned_addr learned my addr 172.21.15.44:0/1697335943 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:43.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.767+0000 7ff9feb18640 1 --2- 172.21.15.44:0/1697335943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f815f040 0x7ff9f80d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.767+0000 7ff9fe317640 1 --2- 172.21.15.44:0/1697335943 >> v2:172.21.15.69:3300/0 conn(0x7ff9f815e550 0x7ff9f806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.767+0000 7ff9fd315640 1 -- 172.21.15.44:0/1697335943 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1735256979 0 0) 0x7ff9f815fb50 con 0x7ff9f806d530 2024-04-02T13:20:43.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.767+0000 7ff9fd315640 1 -- 172.21.15.44:0/1697335943 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9e4003520 con 0x7ff9f806d530 2024-04-02T13:20:43.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.768+0000 7ff9fe317640 1 -- 172.21.15.44:0/1697335943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f815f040 msgr2=0x7ff9f80d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:43.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.768+0000 7ff9fe317640 1 --2- 172.21.15.44:0/1697335943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f815f040 0x7ff9f80d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.768+0000 7ff9fe317640 1 -- 172.21.15.44:0/1697335943 >> v1:172.21.15.44:6789/0 conn(0x7ff9f806d530 legacy=0x7ff9f815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.768+0000 7ff9fd315640 1 -- 172.21.15.44:0/1697335943 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1935231804 0 0) 0x7ff9e4003520 con 0x7ff9f806d530 2024-04-02T13:20:43.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.768+0000 7ff9fe317640 1 -- 172.21.15.44:0/1697335943 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f815f590 con 0x7ff9f815e550 2024-04-02T13:20:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.768+0000 7ff9fe317640 1 --2- 172.21.15.44:0/1697335943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f815e550 0x7ff9f806cff0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40097a0 tx=0x7ff9f402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcc84cf6914b733c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.768+0000 7ff9fd315640 1 -- 172.21.15.44:0/1697335943 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4038070 con 0x7ff9f815e550 2024-04-02T13:20:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.769+0000 7ff9fd315640 1 -- 172.21.15.44:0/1697335943 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f4031500 con 0x7ff9f815e550 2024-04-02T13:20:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.769+0000 7ff9fd315640 1 -- 172.21.15.44:0/1697335943 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e8001670 con 0x7ff9f815e550 2024-04-02T13:20:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.769+0000 7ffa0057b640 1 -- 172.21.15.44:0/1697335943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f815e550 msgr2=0x7ff9f806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.769+0000 7ffa0057b640 1 --2- 172.21.15.44:0/1697335943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f815e550 0x7ff9f806cff0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40097a0 tx=0x7ff9f402b820 comp rx=0 tx=0).stop 2024-04-02T13:20:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.769+0000 7ffa0057b640 1 -- 172.21.15.44:0/1697335943 shutdown_connections 2024-04-02T13:20:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.769+0000 7ffa0057b640 1 --2- 172.21.15.44:0/1697335943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f815f040 0x7ff9f80d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.769+0000 7ffa0057b640 1 --2- 172.21.15.44:0/1697335943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f815e550 0x7ff9f806cff0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.769+0000 7ffa0057b640 1 -- 172.21.15.44:0/1697335943 >> 172.21.15.44:0/1697335943 conn(0x7ff9f809aa20 msgr2=0x7ff9f809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.769+0000 7ffa0057b640 1 -- 172.21.15.44:0/1697335943 shutdown_connections 2024-04-02T13:20:43.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.769+0000 7ffa0057b640 1 -- 172.21.15.44:0/1697335943 wait complete. 2024-04-02T13:20:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.770+0000 7ffa0057b640 1 Processor -- start 2024-04-02T13:20:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.770+0000 7ffa0057b640 1 -- start start 2024-04-02T13:20:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ffa0057b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9f806d530 0x7ff9f80cba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ffa0057b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f815e550 0x7ff9f8156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ffa0057b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f815f040 0x7ff9f814f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ffa0057b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f80d0590 con 0x7ff9f806d530 2024-04-02T13:20:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ffa0057b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f80d02b0 con 0x7ff9f815f040 2024-04-02T13:20:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ffa0057b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff9f80cc4b0 con 0x7ff9f815e550 2024-04-02T13:20:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ff9fe317640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9f806d530 0x7ff9f80cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ff9fe317640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9f806d530 0x7ff9f80cba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59034/0 (socket says 172.21.15.44:59034) 2024-04-02T13:20:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ff9fe317640 1 -- 172.21.15.44:0/3805484933 learned_addr learned my addr 172.21.15.44:0/3805484933 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ff9fdb16640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f815e550 0x7ff9f8156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ff9feb18640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f815f040 0x7ff9f814f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ff9fe317640 1 -- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f815e550 msgr2=0x7ff9f8156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ff9fe317640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f815e550 0x7ff9f8156b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ff9fe317640 1 -- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f815f040 msgr2=0x7ff9f814f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ff9fe317640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f815f040 0x7ff9f814f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.771+0000 7ff9fe317640 1 -- 172.21.15.44:0/3805484933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f814f710 con 0x7ff9f806d530 2024-04-02T13:20:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.772+0000 7ff9fe317640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9f806d530 0x7ff9f80cba00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff9f402b860 tx=0x7ff9f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.772+0000 7ff9eeffd640 1 -- 172.21.15.44:0/3805484933 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4038050 con 0x7ff9f806d530 2024-04-02T13:20:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.772+0000 7ff9eeffd640 1 -- 172.21.15.44:0/3805484933 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f4044070 con 0x7ff9f806d530 2024-04-02T13:20:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.772+0000 7ffa0057b640 1 -- 172.21.15.44:0/3805484933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9f8159260 con 0x7ff9f806d530 2024-04-02T13:20:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.772+0000 7ff9eeffd640 1 -- 172.21.15.44:0/3805484933 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4003a30 con 0x7ff9f806d530 2024-04-02T13:20:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.772+0000 7ffa0057b640 1 -- 172.21.15.44:0/3805484933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9f8159820 con 0x7ff9f806d530 2024-04-02T13:20:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.773+0000 7ffa0057b640 1 -- 172.21.15.44:0/3805484933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9f80684f0 con 0x7ff9f806d530 2024-04-02T13:20:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.773+0000 7ff9eeffd640 1 -- 172.21.15.44:0/3805484933 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff9f4003bd0 con 0x7ff9f806d530 2024-04-02T13:20:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.774+0000 7ff9eeffd640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff9dc06c920 0x7ff9dc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.774+0000 7ff9eeffd640 1 -- 172.21.15.44:0/3805484933 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff9f40add60 con 0x7ff9f806d530 2024-04-02T13:20:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.774+0000 7ff9fdb16640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff9dc06c920 0x7ff9dc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.774+0000 7ff9fdb16640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff9dc06c920 0x7ff9dc06ede0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff9f806d500 tx=0x7ff9e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:43.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.778+0000 7ff9eeffd640 1 -- 172.21.15.44:0/3805484933 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff9f4030070 con 0x7ff9f806d530 2024-04-02T13:20:43.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.960+0000 7ffa0057b640 1 -- 172.21.15.44:0/3805484933 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff9f81570c0 con 0x7ff9dc06c920 2024-04-02T13:20:43.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.969+0000 7ff9eeffd640 1 -- 172.21.15.44:0/3805484933 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7ff9f81570c0 con 0x7ff9dc06c920 2024-04-02T13:20:43.976 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:43.978 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:20:43.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.972+0000 7ffa0057b640 1 -- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff9dc06c920 msgr2=0x7ff9dc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.972+0000 7ffa0057b640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff9dc06c920 0x7ff9dc06ede0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff9f806d500 tx=0x7ff9e8009040 comp rx=0 tx=0).stop 2024-04-02T13:20:43.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.973+0000 7ffa0057b640 1 -- 172.21.15.44:0/3805484933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9f806d530 msgr2=0x7ff9f80cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:43.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.973+0000 7ffa0057b640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9f806d530 0x7ff9f80cba00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff9f402b860 tx=0x7ff9f40046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.973+0000 7ffa0057b640 1 -- 172.21.15.44:0/3805484933 shutdown_connections 2024-04-02T13:20:43.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.973+0000 7ffa0057b640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff9dc06c920 0x7ff9dc06ede0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.973+0000 7ffa0057b640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f815f040 0x7ff9f814f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.973+0000 7ffa0057b640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f815e550 0x7ff9f8156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.973+0000 7ffa0057b640 1 --2- 172.21.15.44:0/3805484933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9f806d530 0x7ff9f80cba00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:43.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.973+0000 7ffa0057b640 1 -- 172.21.15.44:0/3805484933 >> 172.21.15.44:0/3805484933 conn(0x7ff9f809aa20 msgr2=0x7ff9f80cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:43.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.973+0000 7ffa0057b640 1 -- 172.21.15.44:0/3805484933 shutdown_connections 2024-04-02T13:20:43.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:43.973+0000 7ffa0057b640 1 -- 172.21.15.44:0/3805484933 wait complete. 2024-04-02T13:20:43.986 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-04-02T13:20:42.878529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219496,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755224,"statfs":{"total":773094113280,"available":772869349376,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003184"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431510+0000","last_change":"2024-04-02T13:20:19.889381+0000","last_active":"2024-04-02T13:20:26.431510+0000","last_peered":"2024-04-02T13:20:26.431510+0000","last_clean":"2024-04-02T13:20:26.431510+0000","last_became_active":"2024-04-02T13:20:19.889109+0000","last_became_peered":"2024-04-02T13:20:19.889109+0000","last_unstale":"2024-04-02T13:20:26.431510+0000","last_undegraded":"2024-04-02T13:20:26.431510+0000","last_fullsized":"2024-04-02T13:20:26.431510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:42:03.725417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911104+0000","last_change":"2024-04-02T13:20:20.899681+0000","last_active":"2024-04-02T13:20:25.911104+0000","last_peered":"2024-04-02T13:20:25.911104+0000","last_clean":"2024-04-02T13:20:25.911104+0000","last_became_active":"2024-04-02T13:20:20.899408+0000","last_became_peered":"2024-04-02T13:20:20.899408+0000","last_unstale":"2024-04-02T13:20:25.911104+0000","last_undegraded":"2024-04-02T13:20:25.911104+0000","last_fullsized":"2024-04-02T13:20:25.911104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910230+0000","last_change":"2024-04-02T13:20:24.974937+0000","last_active":"2024-04-02T13:20:25.910230+0000","last_peered":"2024-04-02T13:20:25.910230+0000","last_clean":"2024-04-02T13:20:25.910230+0000","last_became_active":"2024-04-02T13:20:24.974496+0000","last_became_peered":"2024-04-02T13:20:24.974496+0000","last_unstale":"2024-04-02T13:20:25.910230+0000","last_undegraded":"2024-04-02T13:20:25.910230+0000","last_fullsized":"2024-04-02T13:20:25.910230+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:20:58.050570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911229+0000","last_change":"2024-04-02T13:20:19.896780+0000","last_active":"2024-04-02T13:20:25.911229+0000","last_peered":"2024-04-02T13:20:25.911229+0000","last_clean":"2024-04-02T13:20:25.911229+0000","last_became_active":"2024-04-02T13:20:19.896480+0000","last_became_peered":"2024-04-02T13:20:19.896480+0000","last_unstale":"2024-04-02T13:20:25.911229+0000","last_undegraded":"2024-04-02T13:20:25.911229+0000","last_fullsized":"2024-04-02T13:20:25.911229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431238+0000","last_change":"2024-04-02T13:20:20.905268+0000","last_active":"2024-04-02T13:20:26.431238+0000","last_peered":"2024-04-02T13:20:26.431238+0000","last_clean":"2024-04-02T13:20:26.431238+0000","last_became_active":"2024-04-02T13:20:20.904997+0000","last_became_peered":"2024-04-02T13:20:20.904997+0000","last_unstale":"2024-04-02T13:20:26.431238+0000","last_undegraded":"2024-04-02T13:20:26.431238+0000","last_fullsized":"2024-04-02T13:20:26.431238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912459+0000","last_change":"2024-04-02T13:20:24.933025+0000","last_active":"2024-04-02T13:20:25.912459+0000","last_peered":"2024-04-02T13:20:25.912459+0000","last_clean":"2024-04-02T13:20:25.912459+0000","last_became_active":"2024-04-02T13:20:24.932475+0000","last_became_peered":"2024-04-02T13:20:24.932475+0000","last_unstale":"2024-04-02T13:20:25.912459+0000","last_undegraded":"2024-04-02T13:20:25.912459+0000","last_fullsized":"2024-04-02T13:20:25.912459+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:46.747491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431519+0000","last_change":"2024-04-02T13:20:19.895931+0000","last_active":"2024-04-02T13:20:26.431519+0000","last_peered":"2024-04-02T13:20:26.431519+0000","last_clean":"2024-04-02T13:20:26.431519+0000","last_became_active":"2024-04-02T13:20:19.895304+0000","last_became_peered":"2024-04-02T13:20:19.895304+0000","last_unstale":"2024-04-02T13:20:26.431519+0000","last_undegraded":"2024-04-02T13:20:26.431519+0000","last_fullsized":"2024-04-02T13:20:26.431519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:08.653626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912892+0000","last_change":"2024-04-02T13:20:20.904770+0000","last_active":"2024-04-02T13:20:25.912892+0000","last_peered":"2024-04-02T13:20:25.912892+0000","last_clean":"2024-04-02T13:20:25.912892+0000","last_became_active":"2024-04-02T13:20:20.903916+0000","last_became_peered":"2024-04-02T13:20:20.903916+0000","last_unstale":"2024-04-02T13:20:25.912892+0000","last_undegraded":"2024-04-02T13:20:25.912892+0000","last_fullsized":"2024-04-02T13:20:25.912892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:39.096965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913056+0000","last_change":"2024-04-02T13:20:24.932562+0000","last_active":"2024-04-02T13:20:25.913056+0000","last_peered":"2024-04-02T13:20:25.913056+0000","last_clean":"2024-04-02T13:20:25.913056+0000","last_became_active":"2024-04-02T13:20:24.931896+0000","last_became_peered":"2024-04-02T13:20:24.931896+0000","last_unstale":"2024-04-02T13:20:25.913056+0000","last_undegraded":"2024-04-02T13:20:25.913056+0000","last_fullsized":"2024-04-02T13:20:25.913056+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:43.131658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430952+0000","last_change":"2024-04-02T13:20:19.887808+0000","last_active":"2024-04-02T13:20:26.430952+0000","last_peered":"2024-04-02T13:20:26.430952+0000","last_clean":"2024-04-02T13:20:26.430952+0000","last_became_active":"2024-04-02T13:20:19.887491+0000","last_became_peered":"2024-04-02T13:20:19.887491+0000","last_unstale":"2024-04-02T13:20:26.430952+0000","last_undegraded":"2024-04-02T13:20:26.430952+0000","last_fullsized":"2024-04-02T13:20:26.430952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914136+0000","last_change":"2024-04-02T13:20:20.900447+0000","last_active":"2024-04-02T13:20:25.914136+0000","last_peered":"2024-04-02T13:20:25.914136+0000","last_clean":"2024-04-02T13:20:25.914136+0000","last_became_active":"2024-04-02T13:20:20.900188+0000","last_became_peered":"2024-04-02T13:20:20.900188+0000","last_unstale":"2024-04-02T13:20:25.914136+0000","last_undegraded":"2024-04-02T13:20:25.914136+0000","last_fullsized":"2024-04-02T13:20:25.914136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:37.660148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912721+0000","last_change":"2024-04-02T13:20:24.928309+0000","last_active":"2024-04-02T13:20:25.912721+0000","last_peered":"2024-04-02T13:20:25.912721+0000","last_clean":"2024-04-02T13:20:25.912721+0000","last_became_active":"2024-04-02T13:20:24.927881+0000","last_became_peered":"2024-04-02T13:20:24.927881+0000","last_unstale":"2024-04-02T13:20:25.912721+0000","last_undegraded":"2024-04-02T13:20:25.912721+0000","last_fullsized":"2024-04-02T13:20:25.912721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:38.338080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430504+0000","last_change":"2024-04-02T13:20:20.895522+0000","last_active":"2024-04-02T13:20:26.430504+0000","last_peered":"2024-04-02T13:20:26.430504+0000","last_clean":"2024-04-02T13:20:26.430504+0000","last_became_active":"2024-04-02T13:20:20.895202+0000","last_became_peered":"2024-04-02T13:20:20.895202+0000","last_unstale":"2024-04-02T13:20:26.430504+0000","last_undegraded":"2024-04-02T13:20:26.430504+0000","last_fullsized":"2024-04-02T13:20:26.430504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430987+0000","last_change":"2024-04-02T13:20:19.895121+0000","last_active":"2024-04-02T13:20:26.430987+0000","last_peered":"2024-04-02T13:20:26.430987+0000","last_clean":"2024-04-02T13:20:26.430987+0000","last_became_active":"2024-04-02T13:20:19.894633+0000","last_became_peered":"2024-04-02T13:20:19.894633+0000","last_unstale":"2024-04-02T13:20:26.430987+0000","last_undegraded":"2024-04-02T13:20:26.430987+0000","last_fullsized":"2024-04-02T13:20:26.430987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:05:36.149061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913513+0000","last_change":"2024-04-02T13:20:24.941648+0000","last_active":"2024-04-02T13:20:25.913513+0000","last_peered":"2024-04-02T13:20:25.913513+0000","last_clean":"2024-04-02T13:20:25.913513+0000","last_became_active":"2024-04-02T13:20:24.941281+0000","last_became_peered":"2024-04-02T13:20:24.941281+0000","last_unstale":"2024-04-02T13:20:25.913513+0000","last_undegraded":"2024-04-02T13:20:25.913513+0000","last_fullsized":"2024-04-02T13:20:25.913513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:06.819681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910552+0000","last_change":"2024-04-02T13:20:20.895057+0000","last_active":"2024-04-02T13:20:25.910552+0000","last_peered":"2024-04-02T13:20:25.910552+0000","last_clean":"2024-04-02T13:20:25.910552+0000","last_became_active":"2024-04-02T13:20:20.894189+0000","last_became_peered":"2024-04-02T13:20:20.894189+0000","last_unstale":"2024-04-02T13:20:25.910552+0000","last_undegraded":"2024-04-02T13:20:25.910552+0000","last_fullsized":"2024-04-02T13:20:25.910552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:56:59.954023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432656+0000","last_change":"2024-04-02T13:20:19.898990+0000","last_active":"2024-04-02T13:20:26.432656+0000","last_peered":"2024-04-02T13:20:26.432656+0000","last_clean":"2024-04-02T13:20:26.432656+0000","last_became_active":"2024-04-02T13:20:19.898590+0000","last_became_peered":"2024-04-02T13:20:19.898590+0000","last_unstale":"2024-04-02T13:20:26.432656+0000","last_undegraded":"2024-04-02T13:20:26.432656+0000","last_fullsized":"2024-04-02T13:20:26.432656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912549+0000","last_change":"2024-04-02T13:20:24.943246+0000","last_active":"2024-04-02T13:20:25.912549+0000","last_peered":"2024-04-02T13:20:25.912549+0000","last_clean":"2024-04-02T13:20:25.912549+0000","last_became_active":"2024-04-02T13:20:24.942587+0000","last_became_peered":"2024-04-02T13:20:24.942587+0000","last_unstale":"2024-04-02T13:20:25.912549+0000","last_undegraded":"2024-04-02T13:20:25.912549+0000","last_fullsized":"2024-04-02T13:20:25.912549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:10.369290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910414+0000","last_change":"2024-04-02T13:20:20.896993+0000","last_active":"2024-04-02T13:20:25.910414+0000","last_peered":"2024-04-02T13:20:25.910414+0000","last_clean":"2024-04-02T13:20:25.910414+0000","last_became_active":"2024-04-02T13:20:20.896668+0000","last_became_peered":"2024-04-02T13:20:20.896668+0000","last_unstale":"2024-04-02T13:20:25.910414+0000","last_undegraded":"2024-04-02T13:20:25.910414+0000","last_fullsized":"2024-04-02T13:20:25.910414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430520+0000","last_change":"2024-04-02T13:20:19.892855+0000","last_active":"2024-04-02T13:20:26.430520+0000","last_peered":"2024-04-02T13:20:26.430520+0000","last_clean":"2024-04-02T13:20:26.430520+0000","last_became_active":"2024-04-02T13:20:19.892578+0000","last_became_peered":"2024-04-02T13:20:19.892578+0000","last_unstale":"2024-04-02T13:20:26.430520+0000","last_undegraded":"2024-04-02T13:20:26.430520+0000","last_fullsized":"2024-04-02T13:20:26.430520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:36.358876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432032+0000","last_change":"2024-04-02T13:20:24.975476+0000","last_active":"2024-04-02T13:20:26.432032+0000","last_peered":"2024-04-02T13:20:26.432032+0000","last_clean":"2024-04-02T13:20:26.432032+0000","last_became_active":"2024-04-02T13:20:24.974600+0000","last_became_peered":"2024-04-02T13:20:24.974600+0000","last_unstale":"2024-04-02T13:20:26.432032+0000","last_undegraded":"2024-04-02T13:20:26.432032+0000","last_fullsized":"2024-04-02T13:20:26.432032+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:35.465459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913314+0000","last_change":"2024-04-02T13:20:20.898012+0000","last_active":"2024-04-02T13:20:25.913314+0000","last_peered":"2024-04-02T13:20:25.913314+0000","last_clean":"2024-04-02T13:20:25.913314+0000","last_became_active":"2024-04-02T13:20:20.897732+0000","last_became_peered":"2024-04-02T13:20:20.897732+0000","last_unstale":"2024-04-02T13:20:25.913314+0000","last_undegraded":"2024-04-02T13:20:25.913314+0000","last_fullsized":"2024-04-02T13:20:25.913314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:55.219441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430649+0000","last_change":"2024-04-02T13:20:19.894100+0000","last_active":"2024-04-02T13:20:26.430649+0000","last_peered":"2024-04-02T13:20:26.430649+0000","last_clean":"2024-04-02T13:20:26.430649+0000","last_became_active":"2024-04-02T13:20:19.893628+0000","last_became_peered":"2024-04-02T13:20:19.893628+0000","last_unstale":"2024-04-02T13:20:26.430649+0000","last_undegraded":"2024-04-02T13:20:26.430649+0000","last_fullsized":"2024-04-02T13:20:26.430649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:05.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913916+0000","last_change":"2024-04-02T13:20:24.943183+0000","last_active":"2024-04-02T13:20:25.913916+0000","last_peered":"2024-04-02T13:20:25.913916+0000","last_clean":"2024-04-02T13:20:25.913916+0000","last_became_active":"2024-04-02T13:20:24.942594+0000","last_became_peered":"2024-04-02T13:20:24.942594+0000","last_unstale":"2024-04-02T13:20:25.913916+0000","last_undegraded":"2024-04-02T13:20:25.913916+0000","last_fullsized":"2024-04-02T13:20:25.913916+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:29.504244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431563+0000","last_change":"2024-04-02T13:20:20.898299+0000","last_active":"2024-04-02T13:20:26.431563+0000","last_peered":"2024-04-02T13:20:26.431563+0000","last_clean":"2024-04-02T13:20:26.431563+0000","last_became_active":"2024-04-02T13:20:20.898014+0000","last_became_peered":"2024-04-02T13:20:20.898014+0000","last_unstale":"2024-04-02T13:20:26.431563+0000","last_undegraded":"2024-04-02T13:20:26.431563+0000","last_fullsized":"2024-04-02T13:20:26.431563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:08.738556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431600+0000","last_change":"2024-04-02T13:20:19.895734+0000","last_active":"2024-04-02T13:20:26.431600+0000","last_peered":"2024-04-02T13:20:26.431600+0000","last_clean":"2024-04-02T13:20:26.431600+0000","last_became_active":"2024-04-02T13:20:19.895430+0000","last_became_peered":"2024-04-02T13:20:19.895430+0000","last_unstale":"2024-04-02T13:20:26.431600+0000","last_undegraded":"2024-04-02T13:20:26.431600+0000","last_fullsized":"2024-04-02T13:20:26.431600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912814+0000","last_change":"2024-04-02T13:20:24.972780+0000","last_active":"2024-04-02T13:20:25.912814+0000","last_peered":"2024-04-02T13:20:25.912814+0000","last_clean":"2024-04-02T13:20:25.912814+0000","last_became_active":"2024-04-02T13:20:24.972420+0000","last_became_peered":"2024-04-02T13:20:24.972420+0000","last_unstale":"2024-04-02T13:20:25.912814+0000","last_undegraded":"2024-04-02T13:20:25.912814+0000","last_fullsized":"2024-04-02T13:20:25.912814+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:28.449843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911321+0000","last_change":"2024-04-02T13:20:20.901896+0000","last_active":"2024-04-02T13:20:25.911321+0000","last_peered":"2024-04-02T13:20:25.911321+0000","last_clean":"2024-04-02T13:20:25.911321+0000","last_became_active":"2024-04-02T13:20:20.901593+0000","last_became_peered":"2024-04-02T13:20:20.901593+0000","last_unstale":"2024-04-02T13:20:25.911321+0000","last_undegraded":"2024-04-02T13:20:25.911321+0000","last_fullsized":"2024-04-02T13:20:25.911321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:24.415150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431790+0000","last_change":"2024-04-02T13:20:19.895273+0000","last_active":"2024-04-02T13:20:26.431790+0000","last_peered":"2024-04-02T13:20:26.431790+0000","last_clean":"2024-04-02T13:20:26.431790+0000","last_became_active":"2024-04-02T13:20:19.894962+0000","last_became_peered":"2024-04-02T13:20:19.894962+0000","last_unstale":"2024-04-02T13:20:26.431790+0000","last_undegraded":"2024-04-02T13:20:26.431790+0000","last_fullsized":"2024-04-02T13:20:26.431790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910167+0000","last_change":"2024-04-02T13:20:24.975854+0000","last_active":"2024-04-02T13:20:25.910167+0000","last_peered":"2024-04-02T13:20:25.910167+0000","last_clean":"2024-04-02T13:20:25.910167+0000","last_became_active":"2024-04-02T13:20:24.975476+0000","last_became_peered":"2024-04-02T13:20:24.975476+0000","last_unstale":"2024-04-02T13:20:25.910167+0000","last_undegraded":"2024-04-02T13:20:25.910167+0000","last_fullsized":"2024-04-02T13:20:25.910167+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:00.601346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432365+0000","last_change":"2024-04-02T13:20:20.898220+0000","last_active":"2024-04-02T13:20:26.432365+0000","last_peered":"2024-04-02T13:20:26.432365+0000","last_clean":"2024-04-02T13:20:26.432365+0000","last_became_active":"2024-04-02T13:20:20.897937+0000","last_became_peered":"2024-04-02T13:20:20.897937+0000","last_unstale":"2024-04-02T13:20:26.432365+0000","last_undegraded":"2024-04-02T13:20:26.432365+0000","last_fullsized":"2024-04-02T13:20:26.432365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431659+0000","last_change":"2024-04-02T13:20:19.891525+0000","last_active":"2024-04-02T13:20:26.431659+0000","last_peered":"2024-04-02T13:20:26.431659+0000","last_clean":"2024-04-02T13:20:26.431659+0000","last_became_active":"2024-04-02T13:20:19.891260+0000","last_became_peered":"2024-04-02T13:20:19.891260+0000","last_unstale":"2024-04-02T13:20:26.431659+0000","last_undegraded":"2024-04-02T13:20:26.431659+0000","last_fullsized":"2024-04-02T13:20:26.431659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:01.860021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431840+0000","last_change":"2024-04-02T13:20:24.970679+0000","last_active":"2024-04-02T13:20:26.431840+0000","last_peered":"2024-04-02T13:20:26.431840+0000","last_clean":"2024-04-02T13:20:26.431840+0000","last_became_active":"2024-04-02T13:20:24.970216+0000","last_became_peered":"2024-04-02T13:20:24.970216+0000","last_unstale":"2024-04-02T13:20:26.431840+0000","last_undegraded":"2024-04-02T13:20:26.431840+0000","last_fullsized":"2024-04-02T13:20:26.431840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:46.744270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914241+0000","last_change":"2024-04-02T13:20:20.901035+0000","last_active":"2024-04-02T13:20:25.914241+0000","last_peered":"2024-04-02T13:20:25.914241+0000","last_clean":"2024-04-02T13:20:25.914241+0000","last_became_active":"2024-04-02T13:20:20.900751+0000","last_became_peered":"2024-04-02T13:20:20.900751+0000","last_unstale":"2024-04-02T13:20:25.914241+0000","last_undegraded":"2024-04-02T13:20:25.914241+0000","last_fullsized":"2024-04-02T13:20:25.914241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:31.430767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432453+0000","last_change":"2024-04-02T13:20:19.897166+0000","last_active":"2024-04-02T13:20:26.432453+0000","last_peered":"2024-04-02T13:20:26.432453+0000","last_clean":"2024-04-02T13:20:26.432453+0000","last_became_active":"2024-04-02T13:20:19.896854+0000","last_became_peered":"2024-04-02T13:20:19.896854+0000","last_unstale":"2024-04-02T13:20:26.432453+0000","last_undegraded":"2024-04-02T13:20:26.432453+0000","last_fullsized":"2024-04-02T13:20:26.432453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:08.653626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430806+0000","last_change":"2024-04-02T13:20:24.937584+0000","last_active":"2024-04-02T13:20:26.430806+0000","last_peered":"2024-04-02T13:20:26.430806+0000","last_clean":"2024-04-02T13:20:26.430806+0000","last_became_active":"2024-04-02T13:20:24.937027+0000","last_became_peered":"2024-04-02T13:20:24.937027+0000","last_unstale":"2024-04-02T13:20:26.430806+0000","last_undegraded":"2024-04-02T13:20:26.430806+0000","last_fullsized":"2024-04-02T13:20:26.430806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:29.504244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914109+0000","last_change":"2024-04-02T13:20:20.888461+0000","last_active":"2024-04-02T13:20:25.914109+0000","last_peered":"2024-04-02T13:20:25.914109+0000","last_clean":"2024-04-02T13:20:25.914109+0000","last_became_active":"2024-04-02T13:20:20.887870+0000","last_became_peered":"2024-04-02T13:20:20.887870+0000","last_unstale":"2024-04-02T13:20:25.914109+0000","last_undegraded":"2024-04-02T13:20:25.914109+0000","last_fullsized":"2024-04-02T13:20:25.914109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:06.511257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913085+0000","last_change":"2024-04-02T13:20:19.895704+0000","last_active":"2024-04-02T13:20:25.913085+0000","last_peered":"2024-04-02T13:20:25.913085+0000","last_clean":"2024-04-02T13:20:25.913085+0000","last_became_active":"2024-04-02T13:20:19.895394+0000","last_became_peered":"2024-04-02T13:20:19.895394+0000","last_unstale":"2024-04-02T13:20:25.913085+0000","last_undegraded":"2024-04-02T13:20:25.913085+0000","last_fullsized":"2024-04-02T13:20:25.913085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432394+0000","last_change":"2024-04-02T13:20:24.935887+0000","last_active":"2024-04-02T13:20:26.432394+0000","last_peered":"2024-04-02T13:20:26.432394+0000","last_clean":"2024-04-02T13:20:26.432394+0000","last_became_active":"2024-04-02T13:20:24.935515+0000","last_became_peered":"2024-04-02T13:20:24.935515+0000","last_unstale":"2024-04-02T13:20:26.432394+0000","last_undegraded":"2024-04-02T13:20:26.432394+0000","last_fullsized":"2024-04-02T13:20:26.432394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:46.744270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431762+0000","last_change":"2024-04-02T13:20:20.899067+0000","last_active":"2024-04-02T13:20:26.431762+0000","last_peered":"2024-04-02T13:20:26.431762+0000","last_clean":"2024-04-02T13:20:26.431762+0000","last_became_active":"2024-04-02T13:20:20.898662+0000","last_became_peered":"2024-04-02T13:20:20.898662+0000","last_unstale":"2024-04-02T13:20:26.431762+0000","last_undegraded":"2024-04-02T13:20:26.431762+0000","last_fullsized":"2024-04-02T13:20:26.431762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431752+0000","last_change":"2024-04-02T13:20:19.896149+0000","last_active":"2024-04-02T13:20:26.431752+0000","last_peered":"2024-04-02T13:20:26.431752+0000","last_clean":"2024-04-02T13:20:26.431752+0000","last_became_active":"2024-04-02T13:20:19.895801+0000","last_became_peered":"2024-04-02T13:20:19.895801+0000","last_unstale":"2024-04-02T13:20:26.431752+0000","last_undegraded":"2024-04-02T13:20:26.431752+0000","last_fullsized":"2024-04-02T13:20:26.431752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432170+0000","last_change":"2024-04-02T13:20:24.933627+0000","last_active":"2024-04-02T13:20:26.432170+0000","last_peered":"2024-04-02T13:20:26.432170+0000","last_clean":"2024-04-02T13:20:26.432170+0000","last_became_active":"2024-04-02T13:20:24.932519+0000","last_became_peered":"2024-04-02T13:20:24.932519+0000","last_unstale":"2024-04-02T13:20:26.432170+0000","last_undegraded":"2024-04-02T13:20:26.432170+0000","last_fullsized":"2024-04-02T13:20:26.432170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:41.694841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430241+0000","last_change":"2024-04-02T13:20:20.899704+0000","last_active":"2024-04-02T13:20:26.430241+0000","last_peered":"2024-04-02T13:20:26.430241+0000","last_clean":"2024-04-02T13:20:26.430241+0000","last_became_active":"2024-04-02T13:20:20.894279+0000","last_became_peered":"2024-04-02T13:20:20.894279+0000","last_unstale":"2024-04-02T13:20:26.430241+0000","last_undegraded":"2024-04-02T13:20:26.430241+0000","last_fullsized":"2024-04-02T13:20:26.430241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:48.848558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430741+0000","last_change":"2024-04-02T13:20:19.893903+0000","last_active":"2024-04-02T13:20:26.430741+0000","last_peered":"2024-04-02T13:20:26.430741+0000","last_clean":"2024-04-02T13:20:26.430741+0000","last_became_active":"2024-04-02T13:20:19.893251+0000","last_became_peered":"2024-04-02T13:20:19.893251+0000","last_unstale":"2024-04-02T13:20:26.430741+0000","last_undegraded":"2024-04-02T13:20:26.430741+0000","last_fullsized":"2024-04-02T13:20:26.430741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:36.358876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431391+0000","last_change":"2024-04-02T13:20:24.971364+0000","last_active":"2024-04-02T13:20:26.431391+0000","last_peered":"2024-04-02T13:20:26.431391+0000","last_clean":"2024-04-02T13:20:26.431391+0000","last_became_active":"2024-04-02T13:20:24.970950+0000","last_became_peered":"2024-04-02T13:20:24.970950+0000","last_unstale":"2024-04-02T13:20:26.431391+0000","last_undegraded":"2024-04-02T13:20:26.431391+0000","last_fullsized":"2024-04-02T13:20:26.431391+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:41.694841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910298+0000","last_change":"2024-04-02T13:20:20.906876+0000","last_active":"2024-04-02T13:20:25.910298+0000","last_peered":"2024-04-02T13:20:25.910298+0000","last_clean":"2024-04-02T13:20:25.910298+0000","last_became_active":"2024-04-02T13:20:20.906603+0000","last_became_peered":"2024-04-02T13:20:20.906603+0000","last_unstale":"2024-04-02T13:20:25.910298+0000","last_undegraded":"2024-04-02T13:20:25.910298+0000","last_fullsized":"2024-04-02T13:20:25.910298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:39.096965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913068+0000","last_change":"2024-04-02T13:20:19.889006+0000","last_active":"2024-04-02T13:20:25.913068+0000","last_peered":"2024-04-02T13:20:25.913068+0000","last_clean":"2024-04-02T13:20:25.913068+0000","last_became_active":"2024-04-02T13:20:19.888619+0000","last_became_peered":"2024-04-02T13:20:19.888619+0000","last_unstale":"2024-04-02T13:20:25.913068+0000","last_undegraded":"2024-04-02T13:20:25.913068+0000","last_fullsized":"2024-04-02T13:20:25.913068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430765+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:20:26.430765+0000","last_peered":"2024-04-02T13:20:26.430765+0000","last_clean":"2024-04-02T13:20:26.430765+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:20:26.430765+0000","last_undegraded":"2024-04-02T13:20:26.430765+0000","last_fullsized":"2024-04-02T13:20:26.430765+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430638+0000","last_change":"2024-04-02T13:20:24.972665+0000","last_active":"2024-04-02T13:20:26.430638+0000","last_peered":"2024-04-02T13:20:26.430638+0000","last_clean":"2024-04-02T13:20:26.430638+0000","last_became_active":"2024-04-02T13:20:24.972285+0000","last_became_peered":"2024-04-02T13:20:24.972285+0000","last_unstale":"2024-04-02T13:20:26.430638+0000","last_undegraded":"2024-04-02T13:20:26.430638+0000","last_fullsized":"2024-04-02T13:20:26.430638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:43.131658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431391+0000","last_change":"2024-04-02T13:20:20.901120+0000","last_active":"2024-04-02T13:20:26.431391+0000","last_peered":"2024-04-02T13:20:26.431391+0000","last_clean":"2024-04-02T13:20:26.431391+0000","last_became_active":"2024-04-02T13:20:20.900433+0000","last_became_peered":"2024-04-02T13:20:20.900433+0000","last_unstale":"2024-04-02T13:20:26.431391+0000","last_undegraded":"2024-04-02T13:20:26.431391+0000","last_fullsized":"2024-04-02T13:20:26.431391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:42.709577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431230+0000","last_change":"2024-04-02T13:20:19.893843+0000","last_active":"2024-04-02T13:20:26.431230+0000","last_peered":"2024-04-02T13:20:26.431230+0000","last_clean":"2024-04-02T13:20:26.431230+0000","last_became_active":"2024-04-02T13:20:19.893524+0000","last_became_peered":"2024-04-02T13:20:19.893524+0000","last_unstale":"2024-04-02T13:20:26.431230+0000","last_undegraded":"2024-04-02T13:20:26.431230+0000","last_fullsized":"2024-04-02T13:20:26.431230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:56.359644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430631+0000","last_change":"2024-04-02T13:20:24.971102+0000","last_active":"2024-04-02T13:20:26.430631+0000","last_peered":"2024-04-02T13:20:26.430631+0000","last_clean":"2024-04-02T13:20:26.430631+0000","last_became_active":"2024-04-02T13:20:24.970132+0000","last_became_peered":"2024-04-02T13:20:24.970132+0000","last_unstale":"2024-04-02T13:20:26.430631+0000","last_undegraded":"2024-04-02T13:20:26.430631+0000","last_fullsized":"2024-04-02T13:20:26.430631+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:28.449843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912995+0000","last_change":"2024-04-02T13:20:20.908405+0000","last_active":"2024-04-02T13:20:25.912995+0000","last_peered":"2024-04-02T13:20:25.912995+0000","last_clean":"2024-04-02T13:20:25.912995+0000","last_became_active":"2024-04-02T13:20:20.907986+0000","last_became_peered":"2024-04-02T13:20:20.907986+0000","last_unstale":"2024-04-02T13:20:25.912995+0000","last_undegraded":"2024-04-02T13:20:25.912995+0000","last_fullsized":"2024-04-02T13:20:25.912995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:24.415150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432656+0000","last_change":"2024-04-02T13:20:19.893884+0000","last_active":"2024-04-02T13:20:26.432656+0000","last_peered":"2024-04-02T13:20:26.432656+0000","last_clean":"2024-04-02T13:20:26.432656+0000","last_became_active":"2024-04-02T13:20:19.893605+0000","last_became_peered":"2024-04-02T13:20:19.893605+0000","last_unstale":"2024-04-02T13:20:26.432656+0000","last_undegraded":"2024-04-02T13:20:26.432656+0000","last_fullsized":"2024-04-02T13:20:26.432656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:05.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432026+0000","last_change":"2024-04-02T13:20:24.931897+0000","last_active":"2024-04-02T13:20:26.432026+0000","last_peered":"2024-04-02T13:20:26.432026+0000","last_clean":"2024-04-02T13:20:26.432026+0000","last_became_active":"2024-04-02T13:20:24.931512+0000","last_became_peered":"2024-04-02T13:20:24.931512+0000","last_unstale":"2024-04-02T13:20:26.432026+0000","last_undegraded":"2024-04-02T13:20:26.432026+0000","last_fullsized":"2024-04-02T13:20:26.432026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:26:20.333590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431656+0000","last_change":"2024-04-02T13:20:20.890017+0000","last_active":"2024-04-02T13:20:26.431656+0000","last_peered":"2024-04-02T13:20:26.431656+0000","last_clean":"2024-04-02T13:20:26.431656+0000","last_became_active":"2024-04-02T13:20:20.889739+0000","last_became_peered":"2024-04-02T13:20:20.889739+0000","last_unstale":"2024-04-02T13:20:26.431656+0000","last_undegraded":"2024-04-02T13:20:26.431656+0000","last_fullsized":"2024-04-02T13:20:26.431656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:25.469551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431013+0000","last_change":"2024-04-02T13:20:19.899771+0000","last_active":"2024-04-02T13:20:26.431013+0000","last_peered":"2024-04-02T13:20:26.431013+0000","last_clean":"2024-04-02T13:20:26.431013+0000","last_became_active":"2024-04-02T13:20:19.899415+0000","last_became_peered":"2024-04-02T13:20:19.899415+0000","last_unstale":"2024-04-02T13:20:26.431013+0000","last_undegraded":"2024-04-02T13:20:26.431013+0000","last_fullsized":"2024-04-02T13:20:26.431013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913513+0000","last_change":"2024-04-02T13:20:24.971830+0000","last_active":"2024-04-02T13:20:25.913513+0000","last_peered":"2024-04-02T13:20:25.913513+0000","last_clean":"2024-04-02T13:20:25.913513+0000","last_became_active":"2024-04-02T13:20:24.971442+0000","last_became_peered":"2024-04-02T13:20:24.971442+0000","last_unstale":"2024-04-02T13:20:25.913513+0000","last_undegraded":"2024-04-02T13:20:25.913513+0000","last_fullsized":"2024-04-02T13:20:25.913513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:00.601346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431356+0000","last_change":"2024-04-02T13:20:20.887060+0000","last_active":"2024-04-02T13:20:26.431356+0000","last_peered":"2024-04-02T13:20:26.431356+0000","last_clean":"2024-04-02T13:20:26.431356+0000","last_became_active":"2024-04-02T13:20:20.886492+0000","last_became_peered":"2024-04-02T13:20:20.886492+0000","last_unstale":"2024-04-02T13:20:26.431356+0000","last_undegraded":"2024-04-02T13:20:26.431356+0000","last_fullsized":"2024-04-02T13:20:26.431356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:31.430767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913624+0000","last_change":"2024-04-02T13:20:19.897647+0000","last_active":"2024-04-02T13:20:25.913624+0000","last_peered":"2024-04-02T13:20:25.913624+0000","last_clean":"2024-04-02T13:20:25.913624+0000","last_became_active":"2024-04-02T13:20:19.897244+0000","last_became_peered":"2024-04-02T13:20:19.897244+0000","last_unstale":"2024-04-02T13:20:25.913624+0000","last_undegraded":"2024-04-02T13:20:25.913624+0000","last_fullsized":"2024-04-02T13:20:25.913624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:28:53.732425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912616+0000","last_change":"2024-04-02T13:20:24.932365+0000","last_active":"2024-04-02T13:20:25.912616+0000","last_peered":"2024-04-02T13:20:25.912616+0000","last_clean":"2024-04-02T13:20:25.912616+0000","last_became_active":"2024-04-02T13:20:24.931793+0000","last_became_peered":"2024-04-02T13:20:24.931793+0000","last_unstale":"2024-04-02T13:20:25.912616+0000","last_undegraded":"2024-04-02T13:20:25.912616+0000","last_fullsized":"2024-04-02T13:20:25.912616+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:30.566782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431185+0000","last_change":"2024-04-02T13:20:20.887410+0000","last_active":"2024-04-02T13:20:26.431185+0000","last_peered":"2024-04-02T13:20:26.431185+0000","last_clean":"2024-04-02T13:20:26.431185+0000","last_became_active":"2024-04-02T13:20:20.886414+0000","last_became_peered":"2024-04-02T13:20:20.886414+0000","last_unstale":"2024-04-02T13:20:26.431185+0000","last_undegraded":"2024-04-02T13:20:26.431185+0000","last_fullsized":"2024-04-02T13:20:26.431185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:24.415150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430413+0000","last_change":"2024-04-02T13:20:19.896189+0000","last_active":"2024-04-02T13:20:26.430413+0000","last_peered":"2024-04-02T13:20:26.430413+0000","last_clean":"2024-04-02T13:20:26.430413+0000","last_became_active":"2024-04-02T13:20:19.895844+0000","last_became_peered":"2024-04-02T13:20:19.895844+0000","last_unstale":"2024-04-02T13:20:26.430413+0000","last_undegraded":"2024-04-02T13:20:26.430413+0000","last_fullsized":"2024-04-02T13:20:26.430413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431242+0000","last_change":"2024-04-02T13:20:24.928642+0000","last_active":"2024-04-02T13:20:26.431242+0000","last_peered":"2024-04-02T13:20:26.431242+0000","last_clean":"2024-04-02T13:20:26.431242+0000","last_became_active":"2024-04-02T13:20:24.928080+0000","last_became_peered":"2024-04-02T13:20:24.928080+0000","last_unstale":"2024-04-02T13:20:26.431242+0000","last_undegraded":"2024-04-02T13:20:26.431242+0000","last_fullsized":"2024-04-02T13:20:26.431242+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:35.465459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430479+0000","last_change":"2024-04-02T13:20:20.898225+0000","last_active":"2024-04-02T13:20:26.430479+0000","last_peered":"2024-04-02T13:20:26.430479+0000","last_clean":"2024-04-02T13:20:26.430479+0000","last_became_active":"2024-04-02T13:20:20.897962+0000","last_became_peered":"2024-04-02T13:20:20.897962+0000","last_unstale":"2024-04-02T13:20:26.430479+0000","last_undegraded":"2024-04-02T13:20:26.430479+0000","last_fullsized":"2024-04-02T13:20:26.430479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430618+0000","last_change":"2024-04-02T13:20:19.893578+0000","last_active":"2024-04-02T13:20:26.430618+0000","last_peered":"2024-04-02T13:20:26.430618+0000","last_clean":"2024-04-02T13:20:26.430618+0000","last_became_active":"2024-04-02T13:20:19.893301+0000","last_became_peered":"2024-04-02T13:20:19.893301+0000","last_unstale":"2024-04-02T13:20:26.430618+0000","last_undegraded":"2024-04-02T13:20:26.430618+0000","last_fullsized":"2024-04-02T13:20:26.430618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911170+0000","last_change":"2024-04-02T13:20:24.971783+0000","last_active":"2024-04-02T13:20:25.911170+0000","last_peered":"2024-04-02T13:20:25.911170+0000","last_clean":"2024-04-02T13:20:25.911170+0000","last_became_active":"2024-04-02T13:20:24.971383+0000","last_became_peered":"2024-04-02T13:20:24.971383+0000","last_unstale":"2024-04-02T13:20:25.911170+0000","last_undegraded":"2024-04-02T13:20:25.911170+0000","last_fullsized":"2024-04-02T13:20:25.911170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:01.105712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430867+0000","last_change":"2024-04-02T13:20:20.889512+0000","last_active":"2024-04-02T13:20:26.430867+0000","last_peered":"2024-04-02T13:20:26.430867+0000","last_clean":"2024-04-02T13:20:26.430867+0000","last_became_active":"2024-04-02T13:20:20.886370+0000","last_became_peered":"2024-04-02T13:20:20.886370+0000","last_unstale":"2024-04-02T13:20:26.430867+0000","last_undegraded":"2024-04-02T13:20:26.430867+0000","last_fullsized":"2024-04-02T13:20:26.430867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:15.162527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912998+0000","last_change":"2024-04-02T13:20:19.895006+0000","last_active":"2024-04-02T13:20:25.912998+0000","last_peered":"2024-04-02T13:20:25.912998+0000","last_clean":"2024-04-02T13:20:25.912998+0000","last_became_active":"2024-04-02T13:20:19.894684+0000","last_became_peered":"2024-04-02T13:20:19.894684+0000","last_unstale":"2024-04-02T13:20:25.912998+0000","last_undegraded":"2024-04-02T13:20:25.912998+0000","last_fullsized":"2024-04-02T13:20:25.912998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911073+0000","last_change":"2024-04-02T13:20:24.972147+0000","last_active":"2024-04-02T13:20:25.911073+0000","last_peered":"2024-04-02T13:20:25.911073+0000","last_clean":"2024-04-02T13:20:25.911073+0000","last_became_active":"2024-04-02T13:20:24.971778+0000","last_became_peered":"2024-04-02T13:20:24.971778+0000","last_unstale":"2024-04-02T13:20:25.911073+0000","last_undegraded":"2024-04-02T13:20:25.911073+0000","last_fullsized":"2024-04-02T13:20:25.911073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:22:03.916011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430986+0000","last_change":"2024-04-02T13:20:20.906617+0000","last_active":"2024-04-02T13:20:26.430986+0000","last_peered":"2024-04-02T13:20:26.430986+0000","last_clean":"2024-04-02T13:20:26.430986+0000","last_became_active":"2024-04-02T13:20:20.905272+0000","last_became_peered":"2024-04-02T13:20:20.905272+0000","last_unstale":"2024-04-02T13:20:26.430986+0000","last_undegraded":"2024-04-02T13:20:26.430986+0000","last_fullsized":"2024-04-02T13:20:26.430986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432098+0000","last_change":"2024-04-02T13:20:19.894435+0000","last_active":"2024-04-02T13:20:26.432098+0000","last_peered":"2024-04-02T13:20:26.432098+0000","last_clean":"2024-04-02T13:20:26.432098+0000","last_became_active":"2024-04-02T13:20:19.894163+0000","last_became_peered":"2024-04-02T13:20:19.894163+0000","last_unstale":"2024-04-02T13:20:26.432098+0000","last_undegraded":"2024-04-02T13:20:26.432098+0000","last_fullsized":"2024-04-02T13:20:26.432098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:44:36.097312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914207+0000","last_change":"2024-04-02T13:20:24.970827+0000","last_active":"2024-04-02T13:20:25.914207+0000","last_peered":"2024-04-02T13:20:25.914207+0000","last_clean":"2024-04-02T13:20:25.914207+0000","last_became_active":"2024-04-02T13:20:24.970384+0000","last_became_peered":"2024-04-02T13:20:24.970384+0000","last_unstale":"2024-04-02T13:20:25.914207+0000","last_undegraded":"2024-04-02T13:20:25.914207+0000","last_fullsized":"2024-04-02T13:20:25.914207+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:20:58.050570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431335+0000","last_change":"2024-04-02T13:20:20.899063+0000","last_active":"2024-04-02T13:20:26.431335+0000","last_peered":"2024-04-02T13:20:26.431335+0000","last_clean":"2024-04-02T13:20:26.431335+0000","last_became_active":"2024-04-02T13:20:20.898735+0000","last_became_peered":"2024-04-02T13:20:20.898735+0000","last_unstale":"2024-04-02T13:20:26.431335+0000","last_undegraded":"2024-04-02T13:20:26.431335+0000","last_fullsized":"2024-04-02T13:20:26.431335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:37.660148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:30.433043+0000","last_change":"2024-04-02T13:20:19.898739+0000","last_active":"2024-04-02T13:20:30.433043+0000","last_peered":"2024-04-02T13:20:30.433043+0000","last_clean":"2024-04-02T13:20:30.433043+0000","last_became_active":"2024-04-02T13:20:19.898524+0000","last_became_peered":"2024-04-02T13:20:19.898524+0000","last_unstale":"2024-04-02T13:20:30.433043+0000","last_undegraded":"2024-04-02T13:20:30.433043+0000","last_fullsized":"2024-04-02T13:20:30.433043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432653+0000","last_change":"2024-04-02T13:20:24.941673+0000","last_active":"2024-04-02T13:20:26.432653+0000","last_peered":"2024-04-02T13:20:26.432653+0000","last_clean":"2024-04-02T13:20:26.432653+0000","last_became_active":"2024-04-02T13:20:24.941096+0000","last_became_peered":"2024-04-02T13:20:24.941096+0000","last_unstale":"2024-04-02T13:20:26.432653+0000","last_undegraded":"2024-04-02T13:20:26.432653+0000","last_fullsized":"2024-04-02T13:20:26.432653+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:26:20.333590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913928+0000","last_change":"2024-04-02T13:20:20.900373+0000","last_active":"2024-04-02T13:20:25.913928+0000","last_peered":"2024-04-02T13:20:25.913928+0000","last_clean":"2024-04-02T13:20:25.913928+0000","last_became_active":"2024-04-02T13:20:20.900084+0000","last_became_peered":"2024-04-02T13:20:20.900084+0000","last_unstale":"2024-04-02T13:20:25.913928+0000","last_undegraded":"2024-04-02T13:20:25.913928+0000","last_fullsized":"2024-04-02T13:20:25.913928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:09.661052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432591+0000","last_change":"2024-04-02T13:20:19.896178+0000","last_active":"2024-04-02T13:20:26.432591+0000","last_peered":"2024-04-02T13:20:26.432591+0000","last_clean":"2024-04-02T13:20:26.432591+0000","last_became_active":"2024-04-02T13:20:19.895887+0000","last_became_peered":"2024-04-02T13:20:19.895887+0000","last_unstale":"2024-04-02T13:20:26.432591+0000","last_undegraded":"2024-04-02T13:20:26.432591+0000","last_fullsized":"2024-04-02T13:20:26.432591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912814+0000","last_change":"2024-04-02T13:20:24.932941+0000","last_active":"2024-04-02T13:20:25.912814+0000","last_peered":"2024-04-02T13:20:25.912814+0000","last_clean":"2024-04-02T13:20:25.912814+0000","last_became_active":"2024-04-02T13:20:24.932319+0000","last_became_peered":"2024-04-02T13:20:24.932319+0000","last_unstale":"2024-04-02T13:20:25.912814+0000","last_undegraded":"2024-04-02T13:20:25.912814+0000","last_fullsized":"2024-04-02T13:20:25.912814+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:57:03.988715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431409+0000","last_change":"2024-04-02T13:20:20.902208+0000","last_active":"2024-04-02T13:20:26.431409+0000","last_peered":"2024-04-02T13:20:26.431409+0000","last_clean":"2024-04-02T13:20:26.431409+0000","last_became_active":"2024-04-02T13:20:20.901921+0000","last_became_peered":"2024-04-02T13:20:20.901921+0000","last_unstale":"2024-04-02T13:20:26.431409+0000","last_undegraded":"2024-04-02T13:20:26.431409+0000","last_fullsized":"2024-04-02T13:20:26.431409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913327+0000","last_change":"2024-04-02T13:20:19.893917+0000","last_active":"2024-04-02T13:20:25.913327+0000","last_peered":"2024-04-02T13:20:25.913327+0000","last_clean":"2024-04-02T13:20:25.913327+0000","last_became_active":"2024-04-02T13:20:19.893652+0000","last_became_peered":"2024-04-02T13:20:19.893652+0000","last_unstale":"2024-04-02T13:20:25.913327+0000","last_undegraded":"2024-04-02T13:20:25.913327+0000","last_fullsized":"2024-04-02T13:20:25.913327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:05.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910393+0000","last_change":"2024-04-02T13:20:24.928526+0000","last_active":"2024-04-02T13:20:25.910393+0000","last_peered":"2024-04-02T13:20:25.910393+0000","last_clean":"2024-04-02T13:20:25.910393+0000","last_became_active":"2024-04-02T13:20:24.928135+0000","last_became_peered":"2024-04-02T13:20:24.928135+0000","last_unstale":"2024-04-02T13:20:25.910393+0000","last_undegraded":"2024-04-02T13:20:25.910393+0000","last_fullsized":"2024-04-02T13:20:25.910393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:18.102976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913243+0000","last_change":"2024-04-02T13:20:20.904387+0000","last_active":"2024-04-02T13:20:25.913243+0000","last_peered":"2024-04-02T13:20:25.913243+0000","last_clean":"2024-04-02T13:20:25.913243+0000","last_became_active":"2024-04-02T13:20:20.903444+0000","last_became_peered":"2024-04-02T13:20:20.903444+0000","last_unstale":"2024-04-02T13:20:25.913243+0000","last_undegraded":"2024-04-02T13:20:25.913243+0000","last_fullsized":"2024-04-02T13:20:25.913243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:25.469551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431417+0000","last_change":"2024-04-02T13:20:19.894102+0000","last_active":"2024-04-02T13:20:26.431417+0000","last_peered":"2024-04-02T13:20:26.431417+0000","last_clean":"2024-04-02T13:20:26.431417+0000","last_became_active":"2024-04-02T13:20:19.893813+0000","last_became_peered":"2024-04-02T13:20:19.893813+0000","last_unstale":"2024-04-02T13:20:26.431417+0000","last_undegraded":"2024-04-02T13:20:26.431417+0000","last_fullsized":"2024-04-02T13:20:26.431417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:56.359644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910302+0000","last_change":"2024-04-02T13:20:24.970905+0000","last_active":"2024-04-02T13:20:25.910302+0000","last_peered":"2024-04-02T13:20:25.910302+0000","last_clean":"2024-04-02T13:20:25.910302+0000","last_became_active":"2024-04-02T13:20:24.970252+0000","last_became_peered":"2024-04-02T13:20:24.970252+0000","last_unstale":"2024-04-02T13:20:25.910302+0000","last_undegraded":"2024-04-02T13:20:25.910302+0000","last_fullsized":"2024-04-02T13:20:25.910302+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:36:46.624236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431088+0000","last_change":"2024-04-02T13:20:24.973949+0000","last_active":"2024-04-02T13:20:26.431088+0000","last_peered":"2024-04-02T13:20:26.431088+0000","last_clean":"2024-04-02T13:20:26.431088+0000","last_became_active":"2024-04-02T13:20:24.973471+0000","last_became_peered":"2024-04-02T13:20:24.973471+0000","last_unstale":"2024-04-02T13:20:26.431088+0000","last_undegraded":"2024-04-02T13:20:26.431088+0000","last_fullsized":"2024-04-02T13:20:26.431088+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:29.504244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430330+0000","last_change":"2024-04-02T13:20:20.892943+0000","last_active":"2024-04-02T13:20:26.430330+0000","last_peered":"2024-04-02T13:20:26.430330+0000","last_clean":"2024-04-02T13:20:26.430330+0000","last_became_active":"2024-04-02T13:20:20.892649+0000","last_became_peered":"2024-04-02T13:20:20.892649+0000","last_unstale":"2024-04-02T13:20:26.430330+0000","last_undegraded":"2024-04-02T13:20:26.430330+0000","last_fullsized":"2024-04-02T13:20:26.430330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913844+0000","last_change":"2024-04-02T13:20:19.897517+0000","last_active":"2024-04-02T13:20:25.913844+0000","last_peered":"2024-04-02T13:20:25.913844+0000","last_clean":"2024-04-02T13:20:25.913844+0000","last_became_active":"2024-04-02T13:20:19.897158+0000","last_became_peered":"2024-04-02T13:20:19.897158+0000","last_unstale":"2024-04-02T13:20:25.913844+0000","last_undegraded":"2024-04-02T13:20:25.913844+0000","last_fullsized":"2024-04-02T13:20:25.913844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431290+0000","last_change":"2024-04-02T13:20:24.932523+0000","last_active":"2024-04-02T13:20:26.431290+0000","last_peered":"2024-04-02T13:20:26.431290+0000","last_clean":"2024-04-02T13:20:26.431290+0000","last_became_active":"2024-04-02T13:20:24.932122+0000","last_became_peered":"2024-04-02T13:20:24.932122+0000","last_unstale":"2024-04-02T13:20:26.431290+0000","last_undegraded":"2024-04-02T13:20:26.431290+0000","last_fullsized":"2024-04-02T13:20:26.431290+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:06.819681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910293+0000","last_change":"2024-04-02T13:20:20.900650+0000","last_active":"2024-04-02T13:20:25.910293+0000","last_peered":"2024-04-02T13:20:25.910293+0000","last_clean":"2024-04-02T13:20:25.910293+0000","last_became_active":"2024-04-02T13:20:20.900371+0000","last_became_peered":"2024-04-02T13:20:20.900371+0000","last_unstale":"2024-04-02T13:20:25.910293+0000","last_undegraded":"2024-04-02T13:20:25.910293+0000","last_fullsized":"2024-04-02T13:20:25.910293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910494+0000","last_change":"2024-04-02T13:20:19.894065+0000","last_active":"2024-04-02T13:20:25.910494+0000","last_peered":"2024-04-02T13:20:25.910494+0000","last_clean":"2024-04-02T13:20:25.910494+0000","last_became_active":"2024-04-02T13:20:19.893749+0000","last_became_peered":"2024-04-02T13:20:19.893749+0000","last_unstale":"2024-04-02T13:20:25.910494+0000","last_undegraded":"2024-04-02T13:20:25.910494+0000","last_fullsized":"2024-04-02T13:20:25.910494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:42:03.725417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430909+0000","last_change":"2024-04-02T13:20:19.892884+0000","last_active":"2024-04-02T13:20:26.430909+0000","last_peered":"2024-04-02T13:20:26.430909+0000","last_clean":"2024-04-02T13:20:26.430909+0000","last_became_active":"2024-04-02T13:20:19.892641+0000","last_became_peered":"2024-04-02T13:20:19.892641+0000","last_unstale":"2024-04-02T13:20:26.430909+0000","last_undegraded":"2024-04-02T13:20:26.430909+0000","last_fullsized":"2024-04-02T13:20:26.430909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432078+0000","last_change":"2024-04-02T13:20:20.900588+0000","last_active":"2024-04-02T13:20:26.432078+0000","last_peered":"2024-04-02T13:20:26.432078+0000","last_clean":"2024-04-02T13:20:26.432078+0000","last_became_active":"2024-04-02T13:20:20.900299+0000","last_became_peered":"2024-04-02T13:20:20.900299+0000","last_unstale":"2024-04-02T13:20:26.432078+0000","last_undegraded":"2024-04-02T13:20:26.432078+0000","last_fullsized":"2024-04-02T13:20:26.432078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431129+0000","last_change":"2024-04-02T13:20:24.937767+0000","last_active":"2024-04-02T13:20:26.431129+0000","last_peered":"2024-04-02T13:20:26.431129+0000","last_clean":"2024-04-02T13:20:26.431129+0000","last_became_active":"2024-04-02T13:20:24.937371+0000","last_became_peered":"2024-04-02T13:20:24.937371+0000","last_unstale":"2024-04-02T13:20:26.431129+0000","last_undegraded":"2024-04-02T13:20:26.431129+0000","last_fullsized":"2024-04-02T13:20:26.431129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:52.883251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910649+0000","last_change":"2024-04-02T13:20:19.895568+0000","last_active":"2024-04-02T13:20:25.910649+0000","last_peered":"2024-04-02T13:20:25.910649+0000","last_clean":"2024-04-02T13:20:25.910649+0000","last_became_active":"2024-04-02T13:20:19.895240+0000","last_became_peered":"2024-04-02T13:20:19.895240+0000","last_unstale":"2024-04-02T13:20:25.910649+0000","last_undegraded":"2024-04-02T13:20:25.910649+0000","last_fullsized":"2024-04-02T13:20:25.910649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:56.359644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912900+0000","last_change":"2024-04-02T13:20:20.894414+0000","last_active":"2024-04-02T13:20:25.912900+0000","last_peered":"2024-04-02T13:20:25.912900+0000","last_clean":"2024-04-02T13:20:25.912900+0000","last_became_active":"2024-04-02T13:20:20.894129+0000","last_became_peered":"2024-04-02T13:20:20.894129+0000","last_unstale":"2024-04-02T13:20:25.912900+0000","last_undegraded":"2024-04-02T13:20:25.912900+0000","last_fullsized":"2024-04-02T13:20:25.912900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:01.268837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911029+0000","last_change":"2024-04-02T13:20:24.932278+0000","last_active":"2024-04-02T13:20:25.911029+0000","last_peered":"2024-04-02T13:20:25.911029+0000","last_clean":"2024-04-02T13:20:25.911029+0000","last_became_active":"2024-04-02T13:20:24.931901+0000","last_became_peered":"2024-04-02T13:20:24.931901+0000","last_unstale":"2024-04-02T13:20:25.911029+0000","last_undegraded":"2024-04-02T13:20:25.911029+0000","last_fullsized":"2024-04-02T13:20:25.911029+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:47.846911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430361+0000","last_change":"2024-04-02T13:20:20.898502+0000","last_active":"2024-04-02T13:20:26.430361+0000","last_peered":"2024-04-02T13:20:26.430361+0000","last_clean":"2024-04-02T13:20:26.430361+0000","last_became_active":"2024-04-02T13:20:20.898208+0000","last_became_peered":"2024-04-02T13:20:20.898208+0000","last_unstale":"2024-04-02T13:20:26.430361+0000","last_undegraded":"2024-04-02T13:20:26.430361+0000","last_fullsized":"2024-04-02T13:20:26.430361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431659+0000","last_change":"2024-04-02T13:20:20.905276+0000","last_active":"2024-04-02T13:20:26.431659+0000","last_peered":"2024-04-02T13:20:26.431659+0000","last_clean":"2024-04-02T13:20:26.431659+0000","last_became_active":"2024-04-02T13:20:20.905010+0000","last_became_peered":"2024-04-02T13:20:20.905010+0000","last_unstale":"2024-04-02T13:20:26.431659+0000","last_undegraded":"2024-04-02T13:20:26.431659+0000","last_fullsized":"2024-04-02T13:20:26.431659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:09.661052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912909+0000","last_change":"2024-04-02T13:20:20.908237+0000","last_active":"2024-04-02T13:20:25.912909+0000","last_peered":"2024-04-02T13:20:25.912909+0000","last_clean":"2024-04-02T13:20:25.912909+0000","last_became_active":"2024-04-02T13:20:20.907967+0000","last_became_peered":"2024-04-02T13:20:20.907967+0000","last_unstale":"2024-04-02T13:20:25.912909+0000","last_undegraded":"2024-04-02T13:20:25.912909+0000","last_fullsized":"2024-04-02T13:20:25.912909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:02.784989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431982+0000","last_change":"2024-04-02T13:20:20.901918+0000","last_active":"2024-04-02T13:20:26.431982+0000","last_peered":"2024-04-02T13:20:26.431982+0000","last_clean":"2024-04-02T13:20:26.431982+0000","last_became_active":"2024-04-02T13:20:20.901628+0000","last_became_peered":"2024-04-02T13:20:20.901628+0000","last_unstale":"2024-04-02T13:20:26.431982+0000","last_undegraded":"2024-04-02T13:20:26.431982+0000","last_fullsized":"2024-04-02T13:20:26.431982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911122+0000","last_change":"2024-04-02T13:20:20.899007+0000","last_active":"2024-04-02T13:20:25.911122+0000","last_peered":"2024-04-02T13:20:25.911122+0000","last_clean":"2024-04-02T13:20:25.911122+0000","last_became_active":"2024-04-02T13:20:20.898005+0000","last_became_peered":"2024-04-02T13:20:20.898005+0000","last_unstale":"2024-04-02T13:20:25.911122+0000","last_undegraded":"2024-04-02T13:20:25.911122+0000","last_fullsized":"2024-04-02T13:20:25.911122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432495+0000","last_change":"2024-04-02T13:20:20.896893+0000","last_active":"2024-04-02T13:20:26.432495+0000","last_peered":"2024-04-02T13:20:26.432495+0000","last_clean":"2024-04-02T13:20:26.432495+0000","last_became_active":"2024-04-02T13:20:20.896629+0000","last_became_peered":"2024-04-02T13:20:20.896629+0000","last_unstale":"2024-04-02T13:20:26.432495+0000","last_undegraded":"2024-04-02T13:20:26.432495+0000","last_fullsized":"2024-04-02T13:20:26.432495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431161+0000","last_change":"2024-04-02T13:20:20.900656+0000","last_active":"2024-04-02T13:20:26.431161+0000","last_peered":"2024-04-02T13:20:26.431161+0000","last_clean":"2024-04-02T13:20:26.431161+0000","last_became_active":"2024-04-02T13:20:20.900385+0000","last_became_peered":"2024-04-02T13:20:20.900385+0000","last_unstale":"2024-04-02T13:20:26.431161+0000","last_undegraded":"2024-04-02T13:20:26.431161+0000","last_fullsized":"2024-04-02T13:20:26.431161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911108+0000","last_change":"2024-04-02T13:20:20.897613+0000","last_active":"2024-04-02T13:20:25.911108+0000","last_peered":"2024-04-02T13:20:25.911108+0000","last_clean":"2024-04-02T13:20:25.911108+0000","last_became_active":"2024-04-02T13:20:20.897285+0000","last_became_peered":"2024-04-02T13:20:20.897285+0000","last_unstale":"2024-04-02T13:20:25.911108+0000","last_undegraded":"2024-04-02T13:20:25.911108+0000","last_fullsized":"2024-04-02T13:20:25.911108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913747+0000","last_change":"2024-04-02T13:20:20.907084+0000","last_active":"2024-04-02T13:20:25.913747+0000","last_peered":"2024-04-02T13:20:25.913747+0000","last_clean":"2024-04-02T13:20:25.913747+0000","last_became_active":"2024-04-02T13:20:20.906734+0000","last_became_peered":"2024-04-02T13:20:20.906734+0000","last_unstale":"2024-04-02T13:20:25.913747+0000","last_undegraded":"2024-04-02T13:20:25.913747+0000","last_fullsized":"2024-04-02T13:20:25.913747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910526+0000","last_change":"2024-04-02T13:20:20.897768+0000","last_active":"2024-04-02T13:20:25.910526+0000","last_peered":"2024-04-02T13:20:25.910526+0000","last_clean":"2024-04-02T13:20:25.910526+0000","last_became_active":"2024-04-02T13:20:20.897502+0000","last_became_peered":"2024-04-02T13:20:20.897502+0000","last_unstale":"2024-04-02T13:20:25.910526+0000","last_undegraded":"2024-04-02T13:20:25.910526+0000","last_fullsized":"2024-04-02T13:20:25.910526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:26.532089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912825+0000","last_change":"2024-04-02T13:20:20.903454+0000","last_active":"2024-04-02T13:20:25.912825+0000","last_peered":"2024-04-02T13:20:25.912825+0000","last_clean":"2024-04-02T13:20:25.912825+0000","last_became_active":"2024-04-02T13:20:20.903050+0000","last_became_peered":"2024-04-02T13:20:20.903050+0000","last_unstale":"2024-04-02T13:20:25.912825+0000","last_undegraded":"2024-04-02T13:20:25.912825+0000","last_fullsized":"2024-04-02T13:20:25.912825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:56:59.954023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431891+0000","last_change":"2024-04-02T13:20:20.904681+0000","last_active":"2024-04-02T13:20:26.431891+0000","last_peered":"2024-04-02T13:20:26.431891+0000","last_clean":"2024-04-02T13:20:26.431891+0000","last_became_active":"2024-04-02T13:20:20.904343+0000","last_became_peered":"2024-04-02T13:20:20.904343+0000","last_unstale":"2024-04-02T13:20:26.431891+0000","last_undegraded":"2024-04-02T13:20:26.431891+0000","last_fullsized":"2024-04-02T13:20:26.431891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:22.414825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431123+0000","last_change":"2024-04-02T13:20:20.895363+0000","last_active":"2024-04-02T13:20:26.431123+0000","last_peered":"2024-04-02T13:20:26.431123+0000","last_clean":"2024-04-02T13:20:26.431123+0000","last_became_active":"2024-04-02T13:20:20.895107+0000","last_became_peered":"2024-04-02T13:20:20.895107+0000","last_unstale":"2024-04-02T13:20:26.431123+0000","last_undegraded":"2024-04-02T13:20:26.431123+0000","last_fullsized":"2024-04-02T13:20:26.431123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:22.414825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431618+0000","last_change":"2024-04-02T13:20:20.890501+0000","last_active":"2024-04-02T13:20:26.431618+0000","last_peered":"2024-04-02T13:20:26.431618+0000","last_clean":"2024-04-02T13:20:26.431618+0000","last_became_active":"2024-04-02T13:20:20.890181+0000","last_became_peered":"2024-04-02T13:20:20.890181+0000","last_unstale":"2024-04-02T13:20:26.431618+0000","last_undegraded":"2024-04-02T13:20:26.431618+0000","last_fullsized":"2024-04-02T13:20:26.431618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:02:18.661127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430915+0000","last_change":"2024-04-02T13:20:20.892430+0000","last_active":"2024-04-02T13:20:26.430915+0000","last_peered":"2024-04-02T13:20:26.430915+0000","last_clean":"2024-04-02T13:20:26.430915+0000","last_became_active":"2024-04-02T13:20:20.892117+0000","last_became_peered":"2024-04-02T13:20:20.892117+0000","last_unstale":"2024-04-02T13:20:26.430915+0000","last_undegraded":"2024-04-02T13:20:26.430915+0000","last_fullsized":"2024-04-02T13:20:26.430915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:37.366302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913765+0000","last_change":"2024-04-02T13:20:20.903371+0000","last_active":"2024-04-02T13:20:25.913765+0000","last_peered":"2024-04-02T13:20:25.913765+0000","last_clean":"2024-04-02T13:20:25.913765+0000","last_became_active":"2024-04-02T13:20:20.903078+0000","last_became_peered":"2024-04-02T13:20:20.903078+0000","last_unstale":"2024-04-02T13:20:25.913765+0000","last_undegraded":"2024-04-02T13:20:25.913765+0000","last_fullsized":"2024-04-02T13:20:25.913765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:42.709577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913653+0000","last_change":"2024-04-02T13:20:20.901550+0000","last_active":"2024-04-02T13:20:25.913653+0000","last_peered":"2024-04-02T13:20:25.913653+0000","last_clean":"2024-04-02T13:20:25.913653+0000","last_became_active":"2024-04-02T13:20:20.901234+0000","last_became_peered":"2024-04-02T13:20:20.901234+0000","last_unstale":"2024-04-02T13:20:25.913653+0000","last_undegraded":"2024-04-02T13:20:25.913653+0000","last_fullsized":"2024-04-02T13:20:25.913653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430422+0000","last_change":"2024-04-02T13:20:20.887101+0000","last_active":"2024-04-02T13:20:26.430422+0000","last_peered":"2024-04-02T13:20:26.430422+0000","last_clean":"2024-04-02T13:20:26.430422+0000","last_became_active":"2024-04-02T13:20:20.886810+0000","last_became_peered":"2024-04-02T13:20:20.886810+0000","last_unstale":"2024-04-02T13:20:26.430422+0000","last_undegraded":"2024-04-02T13:20:26.430422+0000","last_fullsized":"2024-04-02T13:20:26.430422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430889+0000","last_change":"2024-04-02T13:20:20.898387+0000","last_active":"2024-04-02T13:20:26.430889+0000","last_peered":"2024-04-02T13:20:26.430889+0000","last_clean":"2024-04-02T13:20:26.430889+0000","last_became_active":"2024-04-02T13:20:20.898075+0000","last_became_peered":"2024-04-02T13:20:20.898075+0000","last_unstale":"2024-04-02T13:20:26.430889+0000","last_undegraded":"2024-04-02T13:20:26.430889+0000","last_fullsized":"2024-04-02T13:20:26.430889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910512+0000","last_change":"2024-04-02T13:20:20.895603+0000","last_active":"2024-04-02T13:20:25.910512+0000","last_peered":"2024-04-02T13:20:25.910512+0000","last_clean":"2024-04-02T13:20:25.910512+0000","last_became_active":"2024-04-02T13:20:20.895316+0000","last_became_peered":"2024-04-02T13:20:20.895316+0000","last_unstale":"2024-04-02T13:20:25.910512+0000","last_undegraded":"2024-04-02T13:20:25.910512+0000","last_fullsized":"2024-04-02T13:20:25.910512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:34.303387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913831+0000","last_change":"2024-04-02T13:20:20.889381+0000","last_active":"2024-04-02T13:20:25.913831+0000","last_peered":"2024-04-02T13:20:25.913831+0000","last_clean":"2024-04-02T13:20:25.913831+0000","last_became_active":"2024-04-02T13:20:20.889102+0000","last_became_peered":"2024-04-02T13:20:20.889102+0000","last_unstale":"2024-04-02T13:20:25.913831+0000","last_undegraded":"2024-04-02T13:20:25.913831+0000","last_fullsized":"2024-04-02T13:20:25.913831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:37.366302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430695+0000","last_change":"2024-04-02T13:20:20.904474+0000","last_active":"2024-04-02T13:20:26.430695+0000","last_peered":"2024-04-02T13:20:26.430695+0000","last_clean":"2024-04-02T13:20:26.430695+0000","last_became_active":"2024-04-02T13:20:20.904212+0000","last_became_peered":"2024-04-02T13:20:20.904212+0000","last_unstale":"2024-04-02T13:20:26.430695+0000","last_undegraded":"2024-04-02T13:20:26.430695+0000","last_fullsized":"2024-04-02T13:20:26.430695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431059+0000","last_change":"2024-04-02T13:20:20.904469+0000","last_active":"2024-04-02T13:20:26.431059+0000","last_peered":"2024-04-02T13:20:26.431059+0000","last_clean":"2024-04-02T13:20:26.431059+0000","last_became_active":"2024-04-02T13:20:20.904195+0000","last_became_peered":"2024-04-02T13:20:20.904195+0000","last_unstale":"2024-04-02T13:20:26.431059+0000","last_undegraded":"2024-04-02T13:20:26.431059+0000","last_fullsized":"2024-04-02T13:20:26.431059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431650+0000","last_change":"2024-04-02T13:20:20.905842+0000","last_active":"2024-04-02T13:20:26.431650+0000","last_peered":"2024-04-02T13:20:26.431650+0000","last_clean":"2024-04-02T13:20:26.431650+0000","last_became_active":"2024-04-02T13:20:20.905558+0000","last_became_peered":"2024-04-02T13:20:20.905558+0000","last_unstale":"2024-04-02T13:20:26.431650+0000","last_undegraded":"2024-04-02T13:20:26.431650+0000","last_fullsized":"2024-04-02T13:20:26.431650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:02.784989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910120+0000","last_change":"2024-04-02T13:20:20.902048+0000","last_active":"2024-04-02T13:20:25.910120+0000","last_peered":"2024-04-02T13:20:25.910120+0000","last_clean":"2024-04-02T13:20:25.910120+0000","last_became_active":"2024-04-02T13:20:20.901733+0000","last_became_peered":"2024-04-02T13:20:20.901733+0000","last_unstale":"2024-04-02T13:20:25.910120+0000","last_undegraded":"2024-04-02T13:20:25.910120+0000","last_fullsized":"2024-04-02T13:20:25.910120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431149+0000","last_change":"2024-04-02T13:20:20.901947+0000","last_active":"2024-04-02T13:20:26.431149+0000","last_peered":"2024-04-02T13:20:26.431149+0000","last_clean":"2024-04-02T13:20:26.431149+0000","last_became_active":"2024-04-02T13:20:20.901655+0000","last_became_peered":"2024-04-02T13:20:20.901655+0000","last_unstale":"2024-04-02T13:20:26.431149+0000","last_undegraded":"2024-04-02T13:20:26.431149+0000","last_fullsized":"2024-04-02T13:20:26.431149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:56:59.954023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912733+0000","last_change":"2024-04-02T13:20:20.905389+0000","last_active":"2024-04-02T13:20:25.912733+0000","last_peered":"2024-04-02T13:20:25.912733+0000","last_clean":"2024-04-02T13:20:25.912733+0000","last_became_active":"2024-04-02T13:20:20.905062+0000","last_became_peered":"2024-04-02T13:20:20.905062+0000","last_unstale":"2024-04-02T13:20:25.912733+0000","last_undegraded":"2024-04-02T13:20:25.912733+0000","last_fullsized":"2024-04-02T13:20:25.912733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:34.303387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430753+0000","last_change":"2024-04-02T13:20:20.897784+0000","last_active":"2024-04-02T13:20:26.430753+0000","last_peered":"2024-04-02T13:20:26.430753+0000","last_clean":"2024-04-02T13:20:26.430753+0000","last_became_active":"2024-04-02T13:20:20.897488+0000","last_became_peered":"2024-04-02T13:20:20.897488+0000","last_unstale":"2024-04-02T13:20:26.430753+0000","last_undegraded":"2024-04-02T13:20:26.430753+0000","last_fullsized":"2024-04-02T13:20:26.430753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:26:16.298897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912800+0000","last_change":"2024-04-02T13:20:20.908726+0000","last_active":"2024-04-02T13:20:25.912800+0000","last_peered":"2024-04-02T13:20:25.912800+0000","last_clean":"2024-04-02T13:20:25.912800+0000","last_became_active":"2024-04-02T13:20:20.908504+0000","last_became_peered":"2024-04-02T13:20:20.908504+0000","last_unstale":"2024-04-02T13:20:25.912800+0000","last_undegraded":"2024-04-02T13:20:25.912800+0000","last_fullsized":"2024-04-02T13:20:25.912800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:26.532089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430330+0000","last_change":"2024-04-02T13:20:20.892134+0000","last_active":"2024-04-02T13:20:26.430330+0000","last_peered":"2024-04-02T13:20:26.430330+0000","last_clean":"2024-04-02T13:20:26.430330+0000","last_became_active":"2024-04-02T13:20:20.891860+0000","last_became_peered":"2024-04-02T13:20:20.891860+0000","last_unstale":"2024-04-02T13:20:26.430330+0000","last_undegraded":"2024-04-02T13:20:26.430330+0000","last_fullsized":"2024-04-02T13:20:26.430330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:17:34.808750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431697+0000","last_change":"2024-04-02T13:20:20.890864+0000","last_active":"2024-04-02T13:20:26.431697+0000","last_peered":"2024-04-02T13:20:26.431697+0000","last_clean":"2024-04-02T13:20:26.431697+0000","last_became_active":"2024-04-02T13:20:20.890585+0000","last_became_peered":"2024-04-02T13:20:20.890585+0000","last_unstale":"2024-04-02T13:20:26.431697+0000","last_undegraded":"2024-04-02T13:20:26.431697+0000","last_fullsized":"2024-04-02T13:20:26.431697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431543+0000","last_change":"2024-04-02T13:20:20.892679+0000","last_active":"2024-04-02T13:20:26.431543+0000","last_peered":"2024-04-02T13:20:26.431543+0000","last_clean":"2024-04-02T13:20:26.431543+0000","last_became_active":"2024-04-02T13:20:20.892440+0000","last_became_peered":"2024-04-02T13:20:20.892440+0000","last_unstale":"2024-04-02T13:20:26.431543+0000","last_undegraded":"2024-04-02T13:20:26.431543+0000","last_fullsized":"2024-04-02T13:20:26.431543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:39.096965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:20:43.992 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:20:44.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.125+0000 7fc3d6c77640 1 Processor -- start 2024-04-02T13:20:44.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.125+0000 7fc3d6c77640 1 -- start start 2024-04-02T13:20:44.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.126+0000 7fc3d6c77640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015e520 0x7fc3d00d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.126+0000 7fc3d6c77640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc3d00d0f80 0x7fc3d00b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.126+0000 7fc3d6c77640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d00b70b0 con 0x7fc3d015e140 2024-04-02T13:20:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.126+0000 7fc3d6c77640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc3d00d13f0 con 0x7fc3d00d0f80 2024-04-02T13:20:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.126+0000 7fc3d6c77640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc3d00b9f50 con 0x7fc3d015e520 2024-04-02T13:20:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.126+0000 7fc3d4a13640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc3d015e140 0x7fc3d00d2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56370/0 (socket says 172.21.15.44:56370) 2024-04-02T13:20:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.126+0000 7fc3d4a13640 1 -- 172.21.15.44:0/1688289063 learned_addr learned my addr 172.21.15.44:0/1688289063 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.126+0000 7fc3cffff640 1 --2- 172.21.15.44:0/1688289063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015e520 0x7fc3d00d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.126+0000 7fc3d5214640 1 --2- 172.21.15.44:0/1688289063 >> v2:172.21.15.69:3300/0 conn(0x7fc3d00d0f80 0x7fc3d00b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.126+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2874417715 0 0) 0x7fc3d00b70b0 con 0x7fc3d015e140 2024-04-02T13:20:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3bc003520 con 0x7fc3d015e140 2024-04-02T13:20:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4294516730 0 0) 0x7fc3bc003520 con 0x7fc3d015e140 2024-04-02T13:20:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015e520 msgr2=0x7fc3d00d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 --2- 172.21.15.44:0/1688289063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015e520 0x7fc3d00d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 >> v2:172.21.15.69:3300/0 conn(0x7fc3d00d0f80 msgr2=0x7fc3d00b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 --2- 172.21.15.44:0/1688289063 >> v2:172.21.15.69:3300/0 conn(0x7fc3d00d0f80 0x7fc3d00b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d00b7390 con 0x7fc3d015e140 2024-04-02T13:20:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fc3c8001670 con 0x7fc3d015e140 2024-04-02T13:20:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3d5214640 1 --2- 172.21.15.44:0/1688289063 >> v2:172.21.15.69:3300/0 conn(0x7fc3d00d0f80 0x7fc3d00b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 >> v1:172.21.15.44:6789/0 conn(0x7fc3d015e140 legacy=0x7fc3d00d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 --2- 172.21.15.44:0/1688289063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015e520 0x7fc3bc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 --2- 172.21.15.44:0/1688289063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc003f30 0x7fc3bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 --2- 172.21.15.44:0/1688289063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3bc004d40 0x7fc3bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d00d13f0 con 0x7fc3bc004d40 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d00b9f50 con 0x7fc3bc003f30 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc3bc00b9c0 con 0x7fc3d015e520 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3d5214640 1 --2- 172.21.15.44:0/1688289063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3bc004d40 0x7fc3bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3cffff640 1 --2- 172.21.15.44:0/1688289063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc003f30 0x7fc3bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3d4a13640 1 --2- 172.21.15.44:0/1688289063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015e520 0x7fc3bc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3d5214640 1 -- 172.21.15.44:0/1688289063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015e520 msgr2=0x7fc3bc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3d5214640 1 --2- 172.21.15.44:0/1688289063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015e520 0x7fc3bc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3d5214640 1 -- 172.21.15.44:0/1688289063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc003f30 msgr2=0x7fc3bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3d5214640 1 --2- 172.21.15.44:0/1688289063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc003f30 0x7fc3bc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3d5214640 1 -- 172.21.15.44:0/1688289063 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3bc003520 con 0x7fc3bc004d40 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.127+0000 7fc3d4a13640 1 --2- 172.21.15.44:0/1688289063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015e520 0x7fc3bc0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.128+0000 7fc3d5214640 1 --2- 172.21.15.44:0/1688289063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3bc004d40 0x7fc3bc0051c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009c90 tx=0x7fc3bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7caac3a56ba8220 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.128+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8039070 con 0x7fc3bc004d40 2024-04-02T13:20:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.128+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3b802c980 con 0x7fc3bc004d40 2024-04-02T13:20:44.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.128+0000 7fc3cf7fe640 1 -- 172.21.15.44:0/1688289063 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8009860 con 0x7fc3bc004d40 2024-04-02T13:20:44.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.129+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1688289063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3bc004d40 msgr2=0x7fc3bc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.129+0000 7fc3d6c77640 1 --2- 172.21.15.44:0/1688289063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3bc004d40 0x7fc3bc0051c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009c90 tx=0x7fc3bc005830 comp rx=0 tx=0).stop 2024-04-02T13:20:44.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.129+0000 7fc3d5214640 1 -- 172.21.15.44:0/1688289063 reap_dead start 2024-04-02T13:20:44.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.129+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1688289063 shutdown_connections 2024-04-02T13:20:44.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.129+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1688289063 >> 172.21.15.44:0/1688289063 conn(0x7fc3d009d390 msgr2=0x7fc3d00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:44.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.129+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1688289063 shutdown_connections 2024-04-02T13:20:44.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.129+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1688289063 wait complete. 2024-04-02T13:20:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.130+0000 7fc3d6c77640 1 Processor -- start 2024-04-02T13:20:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.130+0000 7fc3d6c77640 1 -- start start 2024-04-02T13:20:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d6c77640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015a8c0 0x7fc3d01f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d6c77640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3d00ca410 0x7fc3d00cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d6c77640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3d00cd360 0x7fc3d00cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d6c77640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d00d4ee0 con 0x7fc3d00cd360 2024-04-02T13:20:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d6c77640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d00b7950 con 0x7fc3d00ca410 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d6c77640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc3d00d2d40 con 0x7fc3d015a8c0 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d5214640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3d00cd360 0x7fc3d00cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d5214640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3d00cd360 0x7fc3d00cd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59054/0 (socket says 172.21.15.44:59054) 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d5214640 1 -- 172.21.15.44:0/1827693909 learned_addr learned my addr 172.21.15.44:0/1827693909 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3cffff640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3d00ca410 0x7fc3d00cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d4a13640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015a8c0 0x7fc3d01f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d5214640 1 -- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015a8c0 msgr2=0x7fc3d01f47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d5214640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015a8c0 0x7fc3d01f47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d5214640 1 -- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3d00ca410 msgr2=0x7fc3d00cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d5214640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3d00ca410 0x7fc3d00cb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d5214640 1 -- 172.21.15.44:0/1827693909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d01f4d00 con 0x7fc3d00cd360 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3d5214640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3d00cd360 0x7fc3d00cd7e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009c90 tx=0x7fc3b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.131+0000 7fc3cd7fa640 1 -- 172.21.15.44:0/1827693909 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8039070 con 0x7fc3d00cd360 2024-04-02T13:20:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.132+0000 7fc3cd7fa640 1 -- 172.21.15.44:0/1827693909 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3b8043070 con 0x7fc3d00cd360 2024-04-02T13:20:44.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.132+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1827693909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3d00c7cb0 con 0x7fc3d00cd360 2024-04-02T13:20:44.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.132+0000 7fc3cd7fa640 1 -- 172.21.15.44:0/1827693909 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8009860 con 0x7fc3d00cd360 2024-04-02T13:20:44.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.132+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1827693909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3d00c8270 con 0x7fc3d00cd360 2024-04-02T13:20:44.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.132+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1827693909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3d00684f0 con 0x7fc3d00cd360 2024-04-02T13:20:44.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.133+0000 7fc3cd7fa640 1 -- 172.21.15.44:0/1827693909 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc3b803e440 con 0x7fc3d00cd360 2024-04-02T13:20:44.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.133+0000 7fc3cd7fa640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc3b006c920 0x7fc3b006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.133+0000 7fc3cd7fa640 1 -- 172.21.15.44:0/1827693909 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc3b80ad170 con 0x7fc3d00cd360 2024-04-02T13:20:44.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.134+0000 7fc3d4a13640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc3b006c920 0x7fc3b006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.134+0000 7fc3d4a13640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc3b006c920 0x7fc3b006ede0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8005bd0 tx=0x7fc3c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:44.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.138+0000 7fc3cd7fa640 1 -- 172.21.15.44:0/1827693909 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc3b804c020 con 0x7fc3d00cd360 2024-04-02T13:20:44.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.317+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1827693909 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc3d00ca9b0 con 0x7fc3b006c920 2024-04-02T13:20:44.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.326+0000 7fc3cd7fa640 1 -- 172.21.15.44:0/1827693909 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7fc3d00ca9b0 con 0x7fc3b006c920 2024-04-02T13:20:44.332 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:20:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.329+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc3b006c920 msgr2=0x7fc3b006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.329+0000 7fc3d6c77640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc3b006c920 0x7fc3b006ede0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8005bd0 tx=0x7fc3c8009040 comp rx=0 tx=0).stop 2024-04-02T13:20:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.329+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1827693909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3d00cd360 msgr2=0x7fc3d00cd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.329+0000 7fc3d6c77640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3d00cd360 0x7fc3d00cd7e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009c90 tx=0x7fc3b8004730 comp rx=0 tx=0).stop 2024-04-02T13:20:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.329+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1827693909 shutdown_connections 2024-04-02T13:20:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.329+0000 7fc3d6c77640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc3b006c920 0x7fc3b006ede0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.329+0000 7fc3d6c77640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3d00cd360 0x7fc3d00cd7e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.329+0000 7fc3d6c77640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3d00ca410 0x7fc3d00cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.329+0000 7fc3d6c77640 1 --2- 172.21.15.44:0/1827693909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d015a8c0 0x7fc3d01f47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.329+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1827693909 >> 172.21.15.44:0/1827693909 conn(0x7fc3d009d390 msgr2=0x7fc3d006cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:44.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.329+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1827693909 shutdown_connections 2024-04-02T13:20:44.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.329+0000 7fc3d6c77640 1 -- 172.21.15.44:0/1827693909 wait complete. 2024-04-02T13:20:44.342 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-04-02T13:20:42.878529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219496,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755224,"statfs":{"total":773094113280,"available":772869349376,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003184"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431510+0000","last_change":"2024-04-02T13:20:19.889381+0000","last_active":"2024-04-02T13:20:26.431510+0000","last_peered":"2024-04-02T13:20:26.431510+0000","last_clean":"2024-04-02T13:20:26.431510+0000","last_became_active":"2024-04-02T13:20:19.889109+0000","last_became_peered":"2024-04-02T13:20:19.889109+0000","last_unstale":"2024-04-02T13:20:26.431510+0000","last_undegraded":"2024-04-02T13:20:26.431510+0000","last_fullsized":"2024-04-02T13:20:26.431510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:42:03.725417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911104+0000","last_change":"2024-04-02T13:20:20.899681+0000","last_active":"2024-04-02T13:20:25.911104+0000","last_peered":"2024-04-02T13:20:25.911104+0000","last_clean":"2024-04-02T13:20:25.911104+0000","last_became_active":"2024-04-02T13:20:20.899408+0000","last_became_peered":"2024-04-02T13:20:20.899408+0000","last_unstale":"2024-04-02T13:20:25.911104+0000","last_undegraded":"2024-04-02T13:20:25.911104+0000","last_fullsized":"2024-04-02T13:20:25.911104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910230+0000","last_change":"2024-04-02T13:20:24.974937+0000","last_active":"2024-04-02T13:20:25.910230+0000","last_peered":"2024-04-02T13:20:25.910230+0000","last_clean":"2024-04-02T13:20:25.910230+0000","last_became_active":"2024-04-02T13:20:24.974496+0000","last_became_peered":"2024-04-02T13:20:24.974496+0000","last_unstale":"2024-04-02T13:20:25.910230+0000","last_undegraded":"2024-04-02T13:20:25.910230+0000","last_fullsized":"2024-04-02T13:20:25.910230+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:20:58.050570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911229+0000","last_change":"2024-04-02T13:20:19.896780+0000","last_active":"2024-04-02T13:20:25.911229+0000","last_peered":"2024-04-02T13:20:25.911229+0000","last_clean":"2024-04-02T13:20:25.911229+0000","last_became_active":"2024-04-02T13:20:19.896480+0000","last_became_peered":"2024-04-02T13:20:19.896480+0000","last_unstale":"2024-04-02T13:20:25.911229+0000","last_undegraded":"2024-04-02T13:20:25.911229+0000","last_fullsized":"2024-04-02T13:20:25.911229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431238+0000","last_change":"2024-04-02T13:20:20.905268+0000","last_active":"2024-04-02T13:20:26.431238+0000","last_peered":"2024-04-02T13:20:26.431238+0000","last_clean":"2024-04-02T13:20:26.431238+0000","last_became_active":"2024-04-02T13:20:20.904997+0000","last_became_peered":"2024-04-02T13:20:20.904997+0000","last_unstale":"2024-04-02T13:20:26.431238+0000","last_undegraded":"2024-04-02T13:20:26.431238+0000","last_fullsized":"2024-04-02T13:20:26.431238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912459+0000","last_change":"2024-04-02T13:20:24.933025+0000","last_active":"2024-04-02T13:20:25.912459+0000","last_peered":"2024-04-02T13:20:25.912459+0000","last_clean":"2024-04-02T13:20:25.912459+0000","last_became_active":"2024-04-02T13:20:24.932475+0000","last_became_peered":"2024-04-02T13:20:24.932475+0000","last_unstale":"2024-04-02T13:20:25.912459+0000","last_undegraded":"2024-04-02T13:20:25.912459+0000","last_fullsized":"2024-04-02T13:20:25.912459+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:46.747491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431519+0000","last_change":"2024-04-02T13:20:19.895931+0000","last_active":"2024-04-02T13:20:26.431519+0000","last_peered":"2024-04-02T13:20:26.431519+0000","last_clean":"2024-04-02T13:20:26.431519+0000","last_became_active":"2024-04-02T13:20:19.895304+0000","last_became_peered":"2024-04-02T13:20:19.895304+0000","last_unstale":"2024-04-02T13:20:26.431519+0000","last_undegraded":"2024-04-02T13:20:26.431519+0000","last_fullsized":"2024-04-02T13:20:26.431519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:08.653626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912892+0000","last_change":"2024-04-02T13:20:20.904770+0000","last_active":"2024-04-02T13:20:25.912892+0000","last_peered":"2024-04-02T13:20:25.912892+0000","last_clean":"2024-04-02T13:20:25.912892+0000","last_became_active":"2024-04-02T13:20:20.903916+0000","last_became_peered":"2024-04-02T13:20:20.903916+0000","last_unstale":"2024-04-02T13:20:25.912892+0000","last_undegraded":"2024-04-02T13:20:25.912892+0000","last_fullsized":"2024-04-02T13:20:25.912892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:39.096965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913056+0000","last_change":"2024-04-02T13:20:24.932562+0000","last_active":"2024-04-02T13:20:25.913056+0000","last_peered":"2024-04-02T13:20:25.913056+0000","last_clean":"2024-04-02T13:20:25.913056+0000","last_became_active":"2024-04-02T13:20:24.931896+0000","last_became_peered":"2024-04-02T13:20:24.931896+0000","last_unstale":"2024-04-02T13:20:25.913056+0000","last_undegraded":"2024-04-02T13:20:25.913056+0000","last_fullsized":"2024-04-02T13:20:25.913056+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:43.131658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430952+0000","last_change":"2024-04-02T13:20:19.887808+0000","last_active":"2024-04-02T13:20:26.430952+0000","last_peered":"2024-04-02T13:20:26.430952+0000","last_clean":"2024-04-02T13:20:26.430952+0000","last_became_active":"2024-04-02T13:20:19.887491+0000","last_became_peered":"2024-04-02T13:20:19.887491+0000","last_unstale":"2024-04-02T13:20:26.430952+0000","last_undegraded":"2024-04-02T13:20:26.430952+0000","last_fullsized":"2024-04-02T13:20:26.430952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914136+0000","last_change":"2024-04-02T13:20:20.900447+0000","last_active":"2024-04-02T13:20:25.914136+0000","last_peered":"2024-04-02T13:20:25.914136+0000","last_clean":"2024-04-02T13:20:25.914136+0000","last_became_active":"2024-04-02T13:20:20.900188+0000","last_became_peered":"2024-04-02T13:20:20.900188+0000","last_unstale":"2024-04-02T13:20:25.914136+0000","last_undegraded":"2024-04-02T13:20:25.914136+0000","last_fullsized":"2024-04-02T13:20:25.914136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:37.660148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912721+0000","last_change":"2024-04-02T13:20:24.928309+0000","last_active":"2024-04-02T13:20:25.912721+0000","last_peered":"2024-04-02T13:20:25.912721+0000","last_clean":"2024-04-02T13:20:25.912721+0000","last_became_active":"2024-04-02T13:20:24.927881+0000","last_became_peered":"2024-04-02T13:20:24.927881+0000","last_unstale":"2024-04-02T13:20:25.912721+0000","last_undegraded":"2024-04-02T13:20:25.912721+0000","last_fullsized":"2024-04-02T13:20:25.912721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:38.338080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430504+0000","last_change":"2024-04-02T13:20:20.895522+0000","last_active":"2024-04-02T13:20:26.430504+0000","last_peered":"2024-04-02T13:20:26.430504+0000","last_clean":"2024-04-02T13:20:26.430504+0000","last_became_active":"2024-04-02T13:20:20.895202+0000","last_became_peered":"2024-04-02T13:20:20.895202+0000","last_unstale":"2024-04-02T13:20:26.430504+0000","last_undegraded":"2024-04-02T13:20:26.430504+0000","last_fullsized":"2024-04-02T13:20:26.430504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430987+0000","last_change":"2024-04-02T13:20:19.895121+0000","last_active":"2024-04-02T13:20:26.430987+0000","last_peered":"2024-04-02T13:20:26.430987+0000","last_clean":"2024-04-02T13:20:26.430987+0000","last_became_active":"2024-04-02T13:20:19.894633+0000","last_became_peered":"2024-04-02T13:20:19.894633+0000","last_unstale":"2024-04-02T13:20:26.430987+0000","last_undegraded":"2024-04-02T13:20:26.430987+0000","last_fullsized":"2024-04-02T13:20:26.430987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:05:36.149061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913513+0000","last_change":"2024-04-02T13:20:24.941648+0000","last_active":"2024-04-02T13:20:25.913513+0000","last_peered":"2024-04-02T13:20:25.913513+0000","last_clean":"2024-04-02T13:20:25.913513+0000","last_became_active":"2024-04-02T13:20:24.941281+0000","last_became_peered":"2024-04-02T13:20:24.941281+0000","last_unstale":"2024-04-02T13:20:25.913513+0000","last_undegraded":"2024-04-02T13:20:25.913513+0000","last_fullsized":"2024-04-02T13:20:25.913513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:06.819681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910552+0000","last_change":"2024-04-02T13:20:20.895057+0000","last_active":"2024-04-02T13:20:25.910552+0000","last_peered":"2024-04-02T13:20:25.910552+0000","last_clean":"2024-04-02T13:20:25.910552+0000","last_became_active":"2024-04-02T13:20:20.894189+0000","last_became_peered":"2024-04-02T13:20:20.894189+0000","last_unstale":"2024-04-02T13:20:25.910552+0000","last_undegraded":"2024-04-02T13:20:25.910552+0000","last_fullsized":"2024-04-02T13:20:25.910552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:56:59.954023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432656+0000","last_change":"2024-04-02T13:20:19.898990+0000","last_active":"2024-04-02T13:20:26.432656+0000","last_peered":"2024-04-02T13:20:26.432656+0000","last_clean":"2024-04-02T13:20:26.432656+0000","last_became_active":"2024-04-02T13:20:19.898590+0000","last_became_peered":"2024-04-02T13:20:19.898590+0000","last_unstale":"2024-04-02T13:20:26.432656+0000","last_undegraded":"2024-04-02T13:20:26.432656+0000","last_fullsized":"2024-04-02T13:20:26.432656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912549+0000","last_change":"2024-04-02T13:20:24.943246+0000","last_active":"2024-04-02T13:20:25.912549+0000","last_peered":"2024-04-02T13:20:25.912549+0000","last_clean":"2024-04-02T13:20:25.912549+0000","last_became_active":"2024-04-02T13:20:24.942587+0000","last_became_peered":"2024-04-02T13:20:24.942587+0000","last_unstale":"2024-04-02T13:20:25.912549+0000","last_undegraded":"2024-04-02T13:20:25.912549+0000","last_fullsized":"2024-04-02T13:20:25.912549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:10.369290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910414+0000","last_change":"2024-04-02T13:20:20.896993+0000","last_active":"2024-04-02T13:20:25.910414+0000","last_peered":"2024-04-02T13:20:25.910414+0000","last_clean":"2024-04-02T13:20:25.910414+0000","last_became_active":"2024-04-02T13:20:20.896668+0000","last_became_peered":"2024-04-02T13:20:20.896668+0000","last_unstale":"2024-04-02T13:20:25.910414+0000","last_undegraded":"2024-04-02T13:20:25.910414+0000","last_fullsized":"2024-04-02T13:20:25.910414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430520+0000","last_change":"2024-04-02T13:20:19.892855+0000","last_active":"2024-04-02T13:20:26.430520+0000","last_peered":"2024-04-02T13:20:26.430520+0000","last_clean":"2024-04-02T13:20:26.430520+0000","last_became_active":"2024-04-02T13:20:19.892578+0000","last_became_peered":"2024-04-02T13:20:19.892578+0000","last_unstale":"2024-04-02T13:20:26.430520+0000","last_undegraded":"2024-04-02T13:20:26.430520+0000","last_fullsized":"2024-04-02T13:20:26.430520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:36.358876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432032+0000","last_change":"2024-04-02T13:20:24.975476+0000","last_active":"2024-04-02T13:20:26.432032+0000","last_peered":"2024-04-02T13:20:26.432032+0000","last_clean":"2024-04-02T13:20:26.432032+0000","last_became_active":"2024-04-02T13:20:24.974600+0000","last_became_peered":"2024-04-02T13:20:24.974600+0000","last_unstale":"2024-04-02T13:20:26.432032+0000","last_undegraded":"2024-04-02T13:20:26.432032+0000","last_fullsized":"2024-04-02T13:20:26.432032+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:35.465459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913314+0000","last_change":"2024-04-02T13:20:20.898012+0000","last_active":"2024-04-02T13:20:25.913314+0000","last_peered":"2024-04-02T13:20:25.913314+0000","last_clean":"2024-04-02T13:20:25.913314+0000","last_became_active":"2024-04-02T13:20:20.897732+0000","last_became_peered":"2024-04-02T13:20:20.897732+0000","last_unstale":"2024-04-02T13:20:25.913314+0000","last_undegraded":"2024-04-02T13:20:25.913314+0000","last_fullsized":"2024-04-02T13:20:25.913314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:55.219441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430649+0000","last_change":"2024-04-02T13:20:19.894100+0000","last_active":"2024-04-02T13:20:26.430649+0000","last_peered":"2024-04-02T13:20:26.430649+0000","last_clean":"2024-04-02T13:20:26.430649+0000","last_became_active":"2024-04-02T13:20:19.893628+0000","last_became_peered":"2024-04-02T13:20:19.893628+0000","last_unstale":"2024-04-02T13:20:26.430649+0000","last_undegraded":"2024-04-02T13:20:26.430649+0000","last_fullsized":"2024-04-02T13:20:26.430649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:05.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913916+0000","last_change":"2024-04-02T13:20:24.943183+0000","last_active":"2024-04-02T13:20:25.913916+0000","last_peered":"2024-04-02T13:20:25.913916+0000","last_clean":"2024-04-02T13:20:25.913916+0000","last_became_active":"2024-04-02T13:20:24.942594+0000","last_became_peered":"2024-04-02T13:20:24.942594+0000","last_unstale":"2024-04-02T13:20:25.913916+0000","last_undegraded":"2024-04-02T13:20:25.913916+0000","last_fullsized":"2024-04-02T13:20:25.913916+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:29.504244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431563+0000","last_change":"2024-04-02T13:20:20.898299+0000","last_active":"2024-04-02T13:20:26.431563+0000","last_peered":"2024-04-02T13:20:26.431563+0000","last_clean":"2024-04-02T13:20:26.431563+0000","last_became_active":"2024-04-02T13:20:20.898014+0000","last_became_peered":"2024-04-02T13:20:20.898014+0000","last_unstale":"2024-04-02T13:20:26.431563+0000","last_undegraded":"2024-04-02T13:20:26.431563+0000","last_fullsized":"2024-04-02T13:20:26.431563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:08.738556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431600+0000","last_change":"2024-04-02T13:20:19.895734+0000","last_active":"2024-04-02T13:20:26.431600+0000","last_peered":"2024-04-02T13:20:26.431600+0000","last_clean":"2024-04-02T13:20:26.431600+0000","last_became_active":"2024-04-02T13:20:19.895430+0000","last_became_peered":"2024-04-02T13:20:19.895430+0000","last_unstale":"2024-04-02T13:20:26.431600+0000","last_undegraded":"2024-04-02T13:20:26.431600+0000","last_fullsized":"2024-04-02T13:20:26.431600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912814+0000","last_change":"2024-04-02T13:20:24.972780+0000","last_active":"2024-04-02T13:20:25.912814+0000","last_peered":"2024-04-02T13:20:25.912814+0000","last_clean":"2024-04-02T13:20:25.912814+0000","last_became_active":"2024-04-02T13:20:24.972420+0000","last_became_peered":"2024-04-02T13:20:24.972420+0000","last_unstale":"2024-04-02T13:20:25.912814+0000","last_undegraded":"2024-04-02T13:20:25.912814+0000","last_fullsized":"2024-04-02T13:20:25.912814+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:28.449843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911321+0000","last_change":"2024-04-02T13:20:20.901896+0000","last_active":"2024-04-02T13:20:25.911321+0000","last_peered":"2024-04-02T13:20:25.911321+0000","last_clean":"2024-04-02T13:20:25.911321+0000","last_became_active":"2024-04-02T13:20:20.901593+0000","last_became_peered":"2024-04-02T13:20:20.901593+0000","last_unstale":"2024-04-02T13:20:25.911321+0000","last_undegraded":"2024-04-02T13:20:25.911321+0000","last_fullsized":"2024-04-02T13:20:25.911321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:24.415150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431790+0000","last_change":"2024-04-02T13:20:19.895273+0000","last_active":"2024-04-02T13:20:26.431790+0000","last_peered":"2024-04-02T13:20:26.431790+0000","last_clean":"2024-04-02T13:20:26.431790+0000","last_became_active":"2024-04-02T13:20:19.894962+0000","last_became_peered":"2024-04-02T13:20:19.894962+0000","last_unstale":"2024-04-02T13:20:26.431790+0000","last_undegraded":"2024-04-02T13:20:26.431790+0000","last_fullsized":"2024-04-02T13:20:26.431790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910167+0000","last_change":"2024-04-02T13:20:24.975854+0000","last_active":"2024-04-02T13:20:25.910167+0000","last_peered":"2024-04-02T13:20:25.910167+0000","last_clean":"2024-04-02T13:20:25.910167+0000","last_became_active":"2024-04-02T13:20:24.975476+0000","last_became_peered":"2024-04-02T13:20:24.975476+0000","last_unstale":"2024-04-02T13:20:25.910167+0000","last_undegraded":"2024-04-02T13:20:25.910167+0000","last_fullsized":"2024-04-02T13:20:25.910167+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:00.601346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432365+0000","last_change":"2024-04-02T13:20:20.898220+0000","last_active":"2024-04-02T13:20:26.432365+0000","last_peered":"2024-04-02T13:20:26.432365+0000","last_clean":"2024-04-02T13:20:26.432365+0000","last_became_active":"2024-04-02T13:20:20.897937+0000","last_became_peered":"2024-04-02T13:20:20.897937+0000","last_unstale":"2024-04-02T13:20:26.432365+0000","last_undegraded":"2024-04-02T13:20:26.432365+0000","last_fullsized":"2024-04-02T13:20:26.432365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431659+0000","last_change":"2024-04-02T13:20:19.891525+0000","last_active":"2024-04-02T13:20:26.431659+0000","last_peered":"2024-04-02T13:20:26.431659+0000","last_clean":"2024-04-02T13:20:26.431659+0000","last_became_active":"2024-04-02T13:20:19.891260+0000","last_became_peered":"2024-04-02T13:20:19.891260+0000","last_unstale":"2024-04-02T13:20:26.431659+0000","last_undegraded":"2024-04-02T13:20:26.431659+0000","last_fullsized":"2024-04-02T13:20:26.431659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:01.860021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431840+0000","last_change":"2024-04-02T13:20:24.970679+0000","last_active":"2024-04-02T13:20:26.431840+0000","last_peered":"2024-04-02T13:20:26.431840+0000","last_clean":"2024-04-02T13:20:26.431840+0000","last_became_active":"2024-04-02T13:20:24.970216+0000","last_became_peered":"2024-04-02T13:20:24.970216+0000","last_unstale":"2024-04-02T13:20:26.431840+0000","last_undegraded":"2024-04-02T13:20:26.431840+0000","last_fullsized":"2024-04-02T13:20:26.431840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:46.744270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914241+0000","last_change":"2024-04-02T13:20:20.901035+0000","last_active":"2024-04-02T13:20:25.914241+0000","last_peered":"2024-04-02T13:20:25.914241+0000","last_clean":"2024-04-02T13:20:25.914241+0000","last_became_active":"2024-04-02T13:20:20.900751+0000","last_became_peered":"2024-04-02T13:20:20.900751+0000","last_unstale":"2024-04-02T13:20:25.914241+0000","last_undegraded":"2024-04-02T13:20:25.914241+0000","last_fullsized":"2024-04-02T13:20:25.914241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:31.430767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432453+0000","last_change":"2024-04-02T13:20:19.897166+0000","last_active":"2024-04-02T13:20:26.432453+0000","last_peered":"2024-04-02T13:20:26.432453+0000","last_clean":"2024-04-02T13:20:26.432453+0000","last_became_active":"2024-04-02T13:20:19.896854+0000","last_became_peered":"2024-04-02T13:20:19.896854+0000","last_unstale":"2024-04-02T13:20:26.432453+0000","last_undegraded":"2024-04-02T13:20:26.432453+0000","last_fullsized":"2024-04-02T13:20:26.432453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:08.653626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430806+0000","last_change":"2024-04-02T13:20:24.937584+0000","last_active":"2024-04-02T13:20:26.430806+0000","last_peered":"2024-04-02T13:20:26.430806+0000","last_clean":"2024-04-02T13:20:26.430806+0000","last_became_active":"2024-04-02T13:20:24.937027+0000","last_became_peered":"2024-04-02T13:20:24.937027+0000","last_unstale":"2024-04-02T13:20:26.430806+0000","last_undegraded":"2024-04-02T13:20:26.430806+0000","last_fullsized":"2024-04-02T13:20:26.430806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:29.504244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914109+0000","last_change":"2024-04-02T13:20:20.888461+0000","last_active":"2024-04-02T13:20:25.914109+0000","last_peered":"2024-04-02T13:20:25.914109+0000","last_clean":"2024-04-02T13:20:25.914109+0000","last_became_active":"2024-04-02T13:20:20.887870+0000","last_became_peered":"2024-04-02T13:20:20.887870+0000","last_unstale":"2024-04-02T13:20:25.914109+0000","last_undegraded":"2024-04-02T13:20:25.914109+0000","last_fullsized":"2024-04-02T13:20:25.914109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:06.511257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913085+0000","last_change":"2024-04-02T13:20:19.895704+0000","last_active":"2024-04-02T13:20:25.913085+0000","last_peered":"2024-04-02T13:20:25.913085+0000","last_clean":"2024-04-02T13:20:25.913085+0000","last_became_active":"2024-04-02T13:20:19.895394+0000","last_became_peered":"2024-04-02T13:20:19.895394+0000","last_unstale":"2024-04-02T13:20:25.913085+0000","last_undegraded":"2024-04-02T13:20:25.913085+0000","last_fullsized":"2024-04-02T13:20:25.913085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432394+0000","last_change":"2024-04-02T13:20:24.935887+0000","last_active":"2024-04-02T13:20:26.432394+0000","last_peered":"2024-04-02T13:20:26.432394+0000","last_clean":"2024-04-02T13:20:26.432394+0000","last_became_active":"2024-04-02T13:20:24.935515+0000","last_became_peered":"2024-04-02T13:20:24.935515+0000","last_unstale":"2024-04-02T13:20:26.432394+0000","last_undegraded":"2024-04-02T13:20:26.432394+0000","last_fullsized":"2024-04-02T13:20:26.432394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:46.744270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431762+0000","last_change":"2024-04-02T13:20:20.899067+0000","last_active":"2024-04-02T13:20:26.431762+0000","last_peered":"2024-04-02T13:20:26.431762+0000","last_clean":"2024-04-02T13:20:26.431762+0000","last_became_active":"2024-04-02T13:20:20.898662+0000","last_became_peered":"2024-04-02T13:20:20.898662+0000","last_unstale":"2024-04-02T13:20:26.431762+0000","last_undegraded":"2024-04-02T13:20:26.431762+0000","last_fullsized":"2024-04-02T13:20:26.431762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431752+0000","last_change":"2024-04-02T13:20:19.896149+0000","last_active":"2024-04-02T13:20:26.431752+0000","last_peered":"2024-04-02T13:20:26.431752+0000","last_clean":"2024-04-02T13:20:26.431752+0000","last_became_active":"2024-04-02T13:20:19.895801+0000","last_became_peered":"2024-04-02T13:20:19.895801+0000","last_unstale":"2024-04-02T13:20:26.431752+0000","last_undegraded":"2024-04-02T13:20:26.431752+0000","last_fullsized":"2024-04-02T13:20:26.431752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432170+0000","last_change":"2024-04-02T13:20:24.933627+0000","last_active":"2024-04-02T13:20:26.432170+0000","last_peered":"2024-04-02T13:20:26.432170+0000","last_clean":"2024-04-02T13:20:26.432170+0000","last_became_active":"2024-04-02T13:20:24.932519+0000","last_became_peered":"2024-04-02T13:20:24.932519+0000","last_unstale":"2024-04-02T13:20:26.432170+0000","last_undegraded":"2024-04-02T13:20:26.432170+0000","last_fullsized":"2024-04-02T13:20:26.432170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:41.694841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430241+0000","last_change":"2024-04-02T13:20:20.899704+0000","last_active":"2024-04-02T13:20:26.430241+0000","last_peered":"2024-04-02T13:20:26.430241+0000","last_clean":"2024-04-02T13:20:26.430241+0000","last_became_active":"2024-04-02T13:20:20.894279+0000","last_became_peered":"2024-04-02T13:20:20.894279+0000","last_unstale":"2024-04-02T13:20:26.430241+0000","last_undegraded":"2024-04-02T13:20:26.430241+0000","last_fullsized":"2024-04-02T13:20:26.430241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:48.848558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430741+0000","last_change":"2024-04-02T13:20:19.893903+0000","last_active":"2024-04-02T13:20:26.430741+0000","last_peered":"2024-04-02T13:20:26.430741+0000","last_clean":"2024-04-02T13:20:26.430741+0000","last_became_active":"2024-04-02T13:20:19.893251+0000","last_became_peered":"2024-04-02T13:20:19.893251+0000","last_unstale":"2024-04-02T13:20:26.430741+0000","last_undegraded":"2024-04-02T13:20:26.430741+0000","last_fullsized":"2024-04-02T13:20:26.430741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:36.358876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431391+0000","last_change":"2024-04-02T13:20:24.971364+0000","last_active":"2024-04-02T13:20:26.431391+0000","last_peered":"2024-04-02T13:20:26.431391+0000","last_clean":"2024-04-02T13:20:26.431391+0000","last_became_active":"2024-04-02T13:20:24.970950+0000","last_became_peered":"2024-04-02T13:20:24.970950+0000","last_unstale":"2024-04-02T13:20:26.431391+0000","last_undegraded":"2024-04-02T13:20:26.431391+0000","last_fullsized":"2024-04-02T13:20:26.431391+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:41.694841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910298+0000","last_change":"2024-04-02T13:20:20.906876+0000","last_active":"2024-04-02T13:20:25.910298+0000","last_peered":"2024-04-02T13:20:25.910298+0000","last_clean":"2024-04-02T13:20:25.910298+0000","last_became_active":"2024-04-02T13:20:20.906603+0000","last_became_peered":"2024-04-02T13:20:20.906603+0000","last_unstale":"2024-04-02T13:20:25.910298+0000","last_undegraded":"2024-04-02T13:20:25.910298+0000","last_fullsized":"2024-04-02T13:20:25.910298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:39.096965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913068+0000","last_change":"2024-04-02T13:20:19.889006+0000","last_active":"2024-04-02T13:20:25.913068+0000","last_peered":"2024-04-02T13:20:25.913068+0000","last_clean":"2024-04-02T13:20:25.913068+0000","last_became_active":"2024-04-02T13:20:19.888619+0000","last_became_peered":"2024-04-02T13:20:19.888619+0000","last_unstale":"2024-04-02T13:20:25.913068+0000","last_undegraded":"2024-04-02T13:20:25.913068+0000","last_fullsized":"2024-04-02T13:20:25.913068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430765+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:20:26.430765+0000","last_peered":"2024-04-02T13:20:26.430765+0000","last_clean":"2024-04-02T13:20:26.430765+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:20:26.430765+0000","last_undegraded":"2024-04-02T13:20:26.430765+0000","last_fullsized":"2024-04-02T13:20:26.430765+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430638+0000","last_change":"2024-04-02T13:20:24.972665+0000","last_active":"2024-04-02T13:20:26.430638+0000","last_peered":"2024-04-02T13:20:26.430638+0000","last_clean":"2024-04-02T13:20:26.430638+0000","last_became_active":"2024-04-02T13:20:24.972285+0000","last_became_peered":"2024-04-02T13:20:24.972285+0000","last_unstale":"2024-04-02T13:20:26.430638+0000","last_undegraded":"2024-04-02T13:20:26.430638+0000","last_fullsized":"2024-04-02T13:20:26.430638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:43.131658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431391+0000","last_change":"2024-04-02T13:20:20.901120+0000","last_active":"2024-04-02T13:20:26.431391+0000","last_peered":"2024-04-02T13:20:26.431391+0000","last_clean":"2024-04-02T13:20:26.431391+0000","last_became_active":"2024-04-02T13:20:20.900433+0000","last_became_peered":"2024-04-02T13:20:20.900433+0000","last_unstale":"2024-04-02T13:20:26.431391+0000","last_undegraded":"2024-04-02T13:20:26.431391+0000","last_fullsized":"2024-04-02T13:20:26.431391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:42.709577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431230+0000","last_change":"2024-04-02T13:20:19.893843+0000","last_active":"2024-04-02T13:20:26.431230+0000","last_peered":"2024-04-02T13:20:26.431230+0000","last_clean":"2024-04-02T13:20:26.431230+0000","last_became_active":"2024-04-02T13:20:19.893524+0000","last_became_peered":"2024-04-02T13:20:19.893524+0000","last_unstale":"2024-04-02T13:20:26.431230+0000","last_undegraded":"2024-04-02T13:20:26.431230+0000","last_fullsized":"2024-04-02T13:20:26.431230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:56.359644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430631+0000","last_change":"2024-04-02T13:20:24.971102+0000","last_active":"2024-04-02T13:20:26.430631+0000","last_peered":"2024-04-02T13:20:26.430631+0000","last_clean":"2024-04-02T13:20:26.430631+0000","last_became_active":"2024-04-02T13:20:24.970132+0000","last_became_peered":"2024-04-02T13:20:24.970132+0000","last_unstale":"2024-04-02T13:20:26.430631+0000","last_undegraded":"2024-04-02T13:20:26.430631+0000","last_fullsized":"2024-04-02T13:20:26.430631+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:28.449843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912995+0000","last_change":"2024-04-02T13:20:20.908405+0000","last_active":"2024-04-02T13:20:25.912995+0000","last_peered":"2024-04-02T13:20:25.912995+0000","last_clean":"2024-04-02T13:20:25.912995+0000","last_became_active":"2024-04-02T13:20:20.907986+0000","last_became_peered":"2024-04-02T13:20:20.907986+0000","last_unstale":"2024-04-02T13:20:25.912995+0000","last_undegraded":"2024-04-02T13:20:25.912995+0000","last_fullsized":"2024-04-02T13:20:25.912995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:24.415150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432656+0000","last_change":"2024-04-02T13:20:19.893884+0000","last_active":"2024-04-02T13:20:26.432656+0000","last_peered":"2024-04-02T13:20:26.432656+0000","last_clean":"2024-04-02T13:20:26.432656+0000","last_became_active":"2024-04-02T13:20:19.893605+0000","last_became_peered":"2024-04-02T13:20:19.893605+0000","last_unstale":"2024-04-02T13:20:26.432656+0000","last_undegraded":"2024-04-02T13:20:26.432656+0000","last_fullsized":"2024-04-02T13:20:26.432656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:05.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432026+0000","last_change":"2024-04-02T13:20:24.931897+0000","last_active":"2024-04-02T13:20:26.432026+0000","last_peered":"2024-04-02T13:20:26.432026+0000","last_clean":"2024-04-02T13:20:26.432026+0000","last_became_active":"2024-04-02T13:20:24.931512+0000","last_became_peered":"2024-04-02T13:20:24.931512+0000","last_unstale":"2024-04-02T13:20:26.432026+0000","last_undegraded":"2024-04-02T13:20:26.432026+0000","last_fullsized":"2024-04-02T13:20:26.432026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:26:20.333590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431656+0000","last_change":"2024-04-02T13:20:20.890017+0000","last_active":"2024-04-02T13:20:26.431656+0000","last_peered":"2024-04-02T13:20:26.431656+0000","last_clean":"2024-04-02T13:20:26.431656+0000","last_became_active":"2024-04-02T13:20:20.889739+0000","last_became_peered":"2024-04-02T13:20:20.889739+0000","last_unstale":"2024-04-02T13:20:26.431656+0000","last_undegraded":"2024-04-02T13:20:26.431656+0000","last_fullsized":"2024-04-02T13:20:26.431656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:25.469551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431013+0000","last_change":"2024-04-02T13:20:19.899771+0000","last_active":"2024-04-02T13:20:26.431013+0000","last_peered":"2024-04-02T13:20:26.431013+0000","last_clean":"2024-04-02T13:20:26.431013+0000","last_became_active":"2024-04-02T13:20:19.899415+0000","last_became_peered":"2024-04-02T13:20:19.899415+0000","last_unstale":"2024-04-02T13:20:26.431013+0000","last_undegraded":"2024-04-02T13:20:26.431013+0000","last_fullsized":"2024-04-02T13:20:26.431013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913513+0000","last_change":"2024-04-02T13:20:24.971830+0000","last_active":"2024-04-02T13:20:25.913513+0000","last_peered":"2024-04-02T13:20:25.913513+0000","last_clean":"2024-04-02T13:20:25.913513+0000","last_became_active":"2024-04-02T13:20:24.971442+0000","last_became_peered":"2024-04-02T13:20:24.971442+0000","last_unstale":"2024-04-02T13:20:25.913513+0000","last_undegraded":"2024-04-02T13:20:25.913513+0000","last_fullsized":"2024-04-02T13:20:25.913513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:00.601346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431356+0000","last_change":"2024-04-02T13:20:20.887060+0000","last_active":"2024-04-02T13:20:26.431356+0000","last_peered":"2024-04-02T13:20:26.431356+0000","last_clean":"2024-04-02T13:20:26.431356+0000","last_became_active":"2024-04-02T13:20:20.886492+0000","last_became_peered":"2024-04-02T13:20:20.886492+0000","last_unstale":"2024-04-02T13:20:26.431356+0000","last_undegraded":"2024-04-02T13:20:26.431356+0000","last_fullsized":"2024-04-02T13:20:26.431356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:31.430767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913624+0000","last_change":"2024-04-02T13:20:19.897647+0000","last_active":"2024-04-02T13:20:25.913624+0000","last_peered":"2024-04-02T13:20:25.913624+0000","last_clean":"2024-04-02T13:20:25.913624+0000","last_became_active":"2024-04-02T13:20:19.897244+0000","last_became_peered":"2024-04-02T13:20:19.897244+0000","last_unstale":"2024-04-02T13:20:25.913624+0000","last_undegraded":"2024-04-02T13:20:25.913624+0000","last_fullsized":"2024-04-02T13:20:25.913624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:28:53.732425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912616+0000","last_change":"2024-04-02T13:20:24.932365+0000","last_active":"2024-04-02T13:20:25.912616+0000","last_peered":"2024-04-02T13:20:25.912616+0000","last_clean":"2024-04-02T13:20:25.912616+0000","last_became_active":"2024-04-02T13:20:24.931793+0000","last_became_peered":"2024-04-02T13:20:24.931793+0000","last_unstale":"2024-04-02T13:20:25.912616+0000","last_undegraded":"2024-04-02T13:20:25.912616+0000","last_fullsized":"2024-04-02T13:20:25.912616+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:30.566782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431185+0000","last_change":"2024-04-02T13:20:20.887410+0000","last_active":"2024-04-02T13:20:26.431185+0000","last_peered":"2024-04-02T13:20:26.431185+0000","last_clean":"2024-04-02T13:20:26.431185+0000","last_became_active":"2024-04-02T13:20:20.886414+0000","last_became_peered":"2024-04-02T13:20:20.886414+0000","last_unstale":"2024-04-02T13:20:26.431185+0000","last_undegraded":"2024-04-02T13:20:26.431185+0000","last_fullsized":"2024-04-02T13:20:26.431185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:24.415150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430413+0000","last_change":"2024-04-02T13:20:19.896189+0000","last_active":"2024-04-02T13:20:26.430413+0000","last_peered":"2024-04-02T13:20:26.430413+0000","last_clean":"2024-04-02T13:20:26.430413+0000","last_became_active":"2024-04-02T13:20:19.895844+0000","last_became_peered":"2024-04-02T13:20:19.895844+0000","last_unstale":"2024-04-02T13:20:26.430413+0000","last_undegraded":"2024-04-02T13:20:26.430413+0000","last_fullsized":"2024-04-02T13:20:26.430413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431242+0000","last_change":"2024-04-02T13:20:24.928642+0000","last_active":"2024-04-02T13:20:26.431242+0000","last_peered":"2024-04-02T13:20:26.431242+0000","last_clean":"2024-04-02T13:20:26.431242+0000","last_became_active":"2024-04-02T13:20:24.928080+0000","last_became_peered":"2024-04-02T13:20:24.928080+0000","last_unstale":"2024-04-02T13:20:26.431242+0000","last_undegraded":"2024-04-02T13:20:26.431242+0000","last_fullsized":"2024-04-02T13:20:26.431242+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:35.465459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430479+0000","last_change":"2024-04-02T13:20:20.898225+0000","last_active":"2024-04-02T13:20:26.430479+0000","last_peered":"2024-04-02T13:20:26.430479+0000","last_clean":"2024-04-02T13:20:26.430479+0000","last_became_active":"2024-04-02T13:20:20.897962+0000","last_became_peered":"2024-04-02T13:20:20.897962+0000","last_unstale":"2024-04-02T13:20:26.430479+0000","last_undegraded":"2024-04-02T13:20:26.430479+0000","last_fullsized":"2024-04-02T13:20:26.430479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430618+0000","last_change":"2024-04-02T13:20:19.893578+0000","last_active":"2024-04-02T13:20:26.430618+0000","last_peered":"2024-04-02T13:20:26.430618+0000","last_clean":"2024-04-02T13:20:26.430618+0000","last_became_active":"2024-04-02T13:20:19.893301+0000","last_became_peered":"2024-04-02T13:20:19.893301+0000","last_unstale":"2024-04-02T13:20:26.430618+0000","last_undegraded":"2024-04-02T13:20:26.430618+0000","last_fullsized":"2024-04-02T13:20:26.430618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911170+0000","last_change":"2024-04-02T13:20:24.971783+0000","last_active":"2024-04-02T13:20:25.911170+0000","last_peered":"2024-04-02T13:20:25.911170+0000","last_clean":"2024-04-02T13:20:25.911170+0000","last_became_active":"2024-04-02T13:20:24.971383+0000","last_became_peered":"2024-04-02T13:20:24.971383+0000","last_unstale":"2024-04-02T13:20:25.911170+0000","last_undegraded":"2024-04-02T13:20:25.911170+0000","last_fullsized":"2024-04-02T13:20:25.911170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:01.105712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430867+0000","last_change":"2024-04-02T13:20:20.889512+0000","last_active":"2024-04-02T13:20:26.430867+0000","last_peered":"2024-04-02T13:20:26.430867+0000","last_clean":"2024-04-02T13:20:26.430867+0000","last_became_active":"2024-04-02T13:20:20.886370+0000","last_became_peered":"2024-04-02T13:20:20.886370+0000","last_unstale":"2024-04-02T13:20:26.430867+0000","last_undegraded":"2024-04-02T13:20:26.430867+0000","last_fullsized":"2024-04-02T13:20:26.430867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:15.162527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912998+0000","last_change":"2024-04-02T13:20:19.895006+0000","last_active":"2024-04-02T13:20:25.912998+0000","last_peered":"2024-04-02T13:20:25.912998+0000","last_clean":"2024-04-02T13:20:25.912998+0000","last_became_active":"2024-04-02T13:20:19.894684+0000","last_became_peered":"2024-04-02T13:20:19.894684+0000","last_unstale":"2024-04-02T13:20:25.912998+0000","last_undegraded":"2024-04-02T13:20:25.912998+0000","last_fullsized":"2024-04-02T13:20:25.912998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911073+0000","last_change":"2024-04-02T13:20:24.972147+0000","last_active":"2024-04-02T13:20:25.911073+0000","last_peered":"2024-04-02T13:20:25.911073+0000","last_clean":"2024-04-02T13:20:25.911073+0000","last_became_active":"2024-04-02T13:20:24.971778+0000","last_became_peered":"2024-04-02T13:20:24.971778+0000","last_unstale":"2024-04-02T13:20:25.911073+0000","last_undegraded":"2024-04-02T13:20:25.911073+0000","last_fullsized":"2024-04-02T13:20:25.911073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:22:03.916011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430986+0000","last_change":"2024-04-02T13:20:20.906617+0000","last_active":"2024-04-02T13:20:26.430986+0000","last_peered":"2024-04-02T13:20:26.430986+0000","last_clean":"2024-04-02T13:20:26.430986+0000","last_became_active":"2024-04-02T13:20:20.905272+0000","last_became_peered":"2024-04-02T13:20:20.905272+0000","last_unstale":"2024-04-02T13:20:26.430986+0000","last_undegraded":"2024-04-02T13:20:26.430986+0000","last_fullsized":"2024-04-02T13:20:26.430986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432098+0000","last_change":"2024-04-02T13:20:19.894435+0000","last_active":"2024-04-02T13:20:26.432098+0000","last_peered":"2024-04-02T13:20:26.432098+0000","last_clean":"2024-04-02T13:20:26.432098+0000","last_became_active":"2024-04-02T13:20:19.894163+0000","last_became_peered":"2024-04-02T13:20:19.894163+0000","last_unstale":"2024-04-02T13:20:26.432098+0000","last_undegraded":"2024-04-02T13:20:26.432098+0000","last_fullsized":"2024-04-02T13:20:26.432098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:44:36.097312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914207+0000","last_change":"2024-04-02T13:20:24.970827+0000","last_active":"2024-04-02T13:20:25.914207+0000","last_peered":"2024-04-02T13:20:25.914207+0000","last_clean":"2024-04-02T13:20:25.914207+0000","last_became_active":"2024-04-02T13:20:24.970384+0000","last_became_peered":"2024-04-02T13:20:24.970384+0000","last_unstale":"2024-04-02T13:20:25.914207+0000","last_undegraded":"2024-04-02T13:20:25.914207+0000","last_fullsized":"2024-04-02T13:20:25.914207+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:20:58.050570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431335+0000","last_change":"2024-04-02T13:20:20.899063+0000","last_active":"2024-04-02T13:20:26.431335+0000","last_peered":"2024-04-02T13:20:26.431335+0000","last_clean":"2024-04-02T13:20:26.431335+0000","last_became_active":"2024-04-02T13:20:20.898735+0000","last_became_peered":"2024-04-02T13:20:20.898735+0000","last_unstale":"2024-04-02T13:20:26.431335+0000","last_undegraded":"2024-04-02T13:20:26.431335+0000","last_fullsized":"2024-04-02T13:20:26.431335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:37.660148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:30.433043+0000","last_change":"2024-04-02T13:20:19.898739+0000","last_active":"2024-04-02T13:20:30.433043+0000","last_peered":"2024-04-02T13:20:30.433043+0000","last_clean":"2024-04-02T13:20:30.433043+0000","last_became_active":"2024-04-02T13:20:19.898524+0000","last_became_peered":"2024-04-02T13:20:19.898524+0000","last_unstale":"2024-04-02T13:20:30.433043+0000","last_undegraded":"2024-04-02T13:20:30.433043+0000","last_fullsized":"2024-04-02T13:20:30.433043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432653+0000","last_change":"2024-04-02T13:20:24.941673+0000","last_active":"2024-04-02T13:20:26.432653+0000","last_peered":"2024-04-02T13:20:26.432653+0000","last_clean":"2024-04-02T13:20:26.432653+0000","last_became_active":"2024-04-02T13:20:24.941096+0000","last_became_peered":"2024-04-02T13:20:24.941096+0000","last_unstale":"2024-04-02T13:20:26.432653+0000","last_undegraded":"2024-04-02T13:20:26.432653+0000","last_fullsized":"2024-04-02T13:20:26.432653+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:26:20.333590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913928+0000","last_change":"2024-04-02T13:20:20.900373+0000","last_active":"2024-04-02T13:20:25.913928+0000","last_peered":"2024-04-02T13:20:25.913928+0000","last_clean":"2024-04-02T13:20:25.913928+0000","last_became_active":"2024-04-02T13:20:20.900084+0000","last_became_peered":"2024-04-02T13:20:20.900084+0000","last_unstale":"2024-04-02T13:20:25.913928+0000","last_undegraded":"2024-04-02T13:20:25.913928+0000","last_fullsized":"2024-04-02T13:20:25.913928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:09.661052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432591+0000","last_change":"2024-04-02T13:20:19.896178+0000","last_active":"2024-04-02T13:20:26.432591+0000","last_peered":"2024-04-02T13:20:26.432591+0000","last_clean":"2024-04-02T13:20:26.432591+0000","last_became_active":"2024-04-02T13:20:19.895887+0000","last_became_peered":"2024-04-02T13:20:19.895887+0000","last_unstale":"2024-04-02T13:20:26.432591+0000","last_undegraded":"2024-04-02T13:20:26.432591+0000","last_fullsized":"2024-04-02T13:20:26.432591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912814+0000","last_change":"2024-04-02T13:20:24.932941+0000","last_active":"2024-04-02T13:20:25.912814+0000","last_peered":"2024-04-02T13:20:25.912814+0000","last_clean":"2024-04-02T13:20:25.912814+0000","last_became_active":"2024-04-02T13:20:24.932319+0000","last_became_peered":"2024-04-02T13:20:24.932319+0000","last_unstale":"2024-04-02T13:20:25.912814+0000","last_undegraded":"2024-04-02T13:20:25.912814+0000","last_fullsized":"2024-04-02T13:20:25.912814+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:57:03.988715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431409+0000","last_change":"2024-04-02T13:20:20.902208+0000","last_active":"2024-04-02T13:20:26.431409+0000","last_peered":"2024-04-02T13:20:26.431409+0000","last_clean":"2024-04-02T13:20:26.431409+0000","last_became_active":"2024-04-02T13:20:20.901921+0000","last_became_peered":"2024-04-02T13:20:20.901921+0000","last_unstale":"2024-04-02T13:20:26.431409+0000","last_undegraded":"2024-04-02T13:20:26.431409+0000","last_fullsized":"2024-04-02T13:20:26.431409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913327+0000","last_change":"2024-04-02T13:20:19.893917+0000","last_active":"2024-04-02T13:20:25.913327+0000","last_peered":"2024-04-02T13:20:25.913327+0000","last_clean":"2024-04-02T13:20:25.913327+0000","last_became_active":"2024-04-02T13:20:19.893652+0000","last_became_peered":"2024-04-02T13:20:19.893652+0000","last_unstale":"2024-04-02T13:20:25.913327+0000","last_undegraded":"2024-04-02T13:20:25.913327+0000","last_fullsized":"2024-04-02T13:20:25.913327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:05.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910393+0000","last_change":"2024-04-02T13:20:24.928526+0000","last_active":"2024-04-02T13:20:25.910393+0000","last_peered":"2024-04-02T13:20:25.910393+0000","last_clean":"2024-04-02T13:20:25.910393+0000","last_became_active":"2024-04-02T13:20:24.928135+0000","last_became_peered":"2024-04-02T13:20:24.928135+0000","last_unstale":"2024-04-02T13:20:25.910393+0000","last_undegraded":"2024-04-02T13:20:25.910393+0000","last_fullsized":"2024-04-02T13:20:25.910393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:18.102976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913243+0000","last_change":"2024-04-02T13:20:20.904387+0000","last_active":"2024-04-02T13:20:25.913243+0000","last_peered":"2024-04-02T13:20:25.913243+0000","last_clean":"2024-04-02T13:20:25.913243+0000","last_became_active":"2024-04-02T13:20:20.903444+0000","last_became_peered":"2024-04-02T13:20:20.903444+0000","last_unstale":"2024-04-02T13:20:25.913243+0000","last_undegraded":"2024-04-02T13:20:25.913243+0000","last_fullsized":"2024-04-02T13:20:25.913243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:25.469551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431417+0000","last_change":"2024-04-02T13:20:19.894102+0000","last_active":"2024-04-02T13:20:26.431417+0000","last_peered":"2024-04-02T13:20:26.431417+0000","last_clean":"2024-04-02T13:20:26.431417+0000","last_became_active":"2024-04-02T13:20:19.893813+0000","last_became_peered":"2024-04-02T13:20:19.893813+0000","last_unstale":"2024-04-02T13:20:26.431417+0000","last_undegraded":"2024-04-02T13:20:26.431417+0000","last_fullsized":"2024-04-02T13:20:26.431417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:56.359644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910302+0000","last_change":"2024-04-02T13:20:24.970905+0000","last_active":"2024-04-02T13:20:25.910302+0000","last_peered":"2024-04-02T13:20:25.910302+0000","last_clean":"2024-04-02T13:20:25.910302+0000","last_became_active":"2024-04-02T13:20:24.970252+0000","last_became_peered":"2024-04-02T13:20:24.970252+0000","last_unstale":"2024-04-02T13:20:25.910302+0000","last_undegraded":"2024-04-02T13:20:25.910302+0000","last_fullsized":"2024-04-02T13:20:25.910302+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:36:46.624236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431088+0000","last_change":"2024-04-02T13:20:24.973949+0000","last_active":"2024-04-02T13:20:26.431088+0000","last_peered":"2024-04-02T13:20:26.431088+0000","last_clean":"2024-04-02T13:20:26.431088+0000","last_became_active":"2024-04-02T13:20:24.973471+0000","last_became_peered":"2024-04-02T13:20:24.973471+0000","last_unstale":"2024-04-02T13:20:26.431088+0000","last_undegraded":"2024-04-02T13:20:26.431088+0000","last_fullsized":"2024-04-02T13:20:26.431088+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:29.504244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430330+0000","last_change":"2024-04-02T13:20:20.892943+0000","last_active":"2024-04-02T13:20:26.430330+0000","last_peered":"2024-04-02T13:20:26.430330+0000","last_clean":"2024-04-02T13:20:26.430330+0000","last_became_active":"2024-04-02T13:20:20.892649+0000","last_became_peered":"2024-04-02T13:20:20.892649+0000","last_unstale":"2024-04-02T13:20:26.430330+0000","last_undegraded":"2024-04-02T13:20:26.430330+0000","last_fullsized":"2024-04-02T13:20:26.430330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913844+0000","last_change":"2024-04-02T13:20:19.897517+0000","last_active":"2024-04-02T13:20:25.913844+0000","last_peered":"2024-04-02T13:20:25.913844+0000","last_clean":"2024-04-02T13:20:25.913844+0000","last_became_active":"2024-04-02T13:20:19.897158+0000","last_became_peered":"2024-04-02T13:20:19.897158+0000","last_unstale":"2024-04-02T13:20:25.913844+0000","last_undegraded":"2024-04-02T13:20:25.913844+0000","last_fullsized":"2024-04-02T13:20:25.913844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431290+0000","last_change":"2024-04-02T13:20:24.932523+0000","last_active":"2024-04-02T13:20:26.431290+0000","last_peered":"2024-04-02T13:20:26.431290+0000","last_clean":"2024-04-02T13:20:26.431290+0000","last_became_active":"2024-04-02T13:20:24.932122+0000","last_became_peered":"2024-04-02T13:20:24.932122+0000","last_unstale":"2024-04-02T13:20:26.431290+0000","last_undegraded":"2024-04-02T13:20:26.431290+0000","last_fullsized":"2024-04-02T13:20:26.431290+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:06.819681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910293+0000","last_change":"2024-04-02T13:20:20.900650+0000","last_active":"2024-04-02T13:20:25.910293+0000","last_peered":"2024-04-02T13:20:25.910293+0000","last_clean":"2024-04-02T13:20:25.910293+0000","last_became_active":"2024-04-02T13:20:20.900371+0000","last_became_peered":"2024-04-02T13:20:20.900371+0000","last_unstale":"2024-04-02T13:20:25.910293+0000","last_undegraded":"2024-04-02T13:20:25.910293+0000","last_fullsized":"2024-04-02T13:20:25.910293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910494+0000","last_change":"2024-04-02T13:20:19.894065+0000","last_active":"2024-04-02T13:20:25.910494+0000","last_peered":"2024-04-02T13:20:25.910494+0000","last_clean":"2024-04-02T13:20:25.910494+0000","last_became_active":"2024-04-02T13:20:19.893749+0000","last_became_peered":"2024-04-02T13:20:19.893749+0000","last_unstale":"2024-04-02T13:20:25.910494+0000","last_undegraded":"2024-04-02T13:20:25.910494+0000","last_fullsized":"2024-04-02T13:20:25.910494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:42:03.725417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430909+0000","last_change":"2024-04-02T13:20:19.892884+0000","last_active":"2024-04-02T13:20:26.430909+0000","last_peered":"2024-04-02T13:20:26.430909+0000","last_clean":"2024-04-02T13:20:26.430909+0000","last_became_active":"2024-04-02T13:20:19.892641+0000","last_became_peered":"2024-04-02T13:20:19.892641+0000","last_unstale":"2024-04-02T13:20:26.430909+0000","last_undegraded":"2024-04-02T13:20:26.430909+0000","last_fullsized":"2024-04-02T13:20:26.430909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432078+0000","last_change":"2024-04-02T13:20:20.900588+0000","last_active":"2024-04-02T13:20:26.432078+0000","last_peered":"2024-04-02T13:20:26.432078+0000","last_clean":"2024-04-02T13:20:26.432078+0000","last_became_active":"2024-04-02T13:20:20.900299+0000","last_became_peered":"2024-04-02T13:20:20.900299+0000","last_unstale":"2024-04-02T13:20:26.432078+0000","last_undegraded":"2024-04-02T13:20:26.432078+0000","last_fullsized":"2024-04-02T13:20:26.432078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431129+0000","last_change":"2024-04-02T13:20:24.937767+0000","last_active":"2024-04-02T13:20:26.431129+0000","last_peered":"2024-04-02T13:20:26.431129+0000","last_clean":"2024-04-02T13:20:26.431129+0000","last_became_active":"2024-04-02T13:20:24.937371+0000","last_became_peered":"2024-04-02T13:20:24.937371+0000","last_unstale":"2024-04-02T13:20:26.431129+0000","last_undegraded":"2024-04-02T13:20:26.431129+0000","last_fullsized":"2024-04-02T13:20:26.431129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:52.883251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910649+0000","last_change":"2024-04-02T13:20:19.895568+0000","last_active":"2024-04-02T13:20:25.910649+0000","last_peered":"2024-04-02T13:20:25.910649+0000","last_clean":"2024-04-02T13:20:25.910649+0000","last_became_active":"2024-04-02T13:20:19.895240+0000","last_became_peered":"2024-04-02T13:20:19.895240+0000","last_unstale":"2024-04-02T13:20:25.910649+0000","last_undegraded":"2024-04-02T13:20:25.910649+0000","last_fullsized":"2024-04-02T13:20:25.910649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:56.359644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912900+0000","last_change":"2024-04-02T13:20:20.894414+0000","last_active":"2024-04-02T13:20:25.912900+0000","last_peered":"2024-04-02T13:20:25.912900+0000","last_clean":"2024-04-02T13:20:25.912900+0000","last_became_active":"2024-04-02T13:20:20.894129+0000","last_became_peered":"2024-04-02T13:20:20.894129+0000","last_unstale":"2024-04-02T13:20:25.912900+0000","last_undegraded":"2024-04-02T13:20:25.912900+0000","last_fullsized":"2024-04-02T13:20:25.912900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:01.268837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911029+0000","last_change":"2024-04-02T13:20:24.932278+0000","last_active":"2024-04-02T13:20:25.911029+0000","last_peered":"2024-04-02T13:20:25.911029+0000","last_clean":"2024-04-02T13:20:25.911029+0000","last_became_active":"2024-04-02T13:20:24.931901+0000","last_became_peered":"2024-04-02T13:20:24.931901+0000","last_unstale":"2024-04-02T13:20:25.911029+0000","last_undegraded":"2024-04-02T13:20:25.911029+0000","last_fullsized":"2024-04-02T13:20:25.911029+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:47.846911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430361+0000","last_change":"2024-04-02T13:20:20.898502+0000","last_active":"2024-04-02T13:20:26.430361+0000","last_peered":"2024-04-02T13:20:26.430361+0000","last_clean":"2024-04-02T13:20:26.430361+0000","last_became_active":"2024-04-02T13:20:20.898208+0000","last_became_peered":"2024-04-02T13:20:20.898208+0000","last_unstale":"2024-04-02T13:20:26.430361+0000","last_undegraded":"2024-04-02T13:20:26.430361+0000","last_fullsized":"2024-04-02T13:20:26.430361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431659+0000","last_change":"2024-04-02T13:20:20.905276+0000","last_active":"2024-04-02T13:20:26.431659+0000","last_peered":"2024-04-02T13:20:26.431659+0000","last_clean":"2024-04-02T13:20:26.431659+0000","last_became_active":"2024-04-02T13:20:20.905010+0000","last_became_peered":"2024-04-02T13:20:20.905010+0000","last_unstale":"2024-04-02T13:20:26.431659+0000","last_undegraded":"2024-04-02T13:20:26.431659+0000","last_fullsized":"2024-04-02T13:20:26.431659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:09.661052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912909+0000","last_change":"2024-04-02T13:20:20.908237+0000","last_active":"2024-04-02T13:20:25.912909+0000","last_peered":"2024-04-02T13:20:25.912909+0000","last_clean":"2024-04-02T13:20:25.912909+0000","last_became_active":"2024-04-02T13:20:20.907967+0000","last_became_peered":"2024-04-02T13:20:20.907967+0000","last_unstale":"2024-04-02T13:20:25.912909+0000","last_undegraded":"2024-04-02T13:20:25.912909+0000","last_fullsized":"2024-04-02T13:20:25.912909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:02.784989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431982+0000","last_change":"2024-04-02T13:20:20.901918+0000","last_active":"2024-04-02T13:20:26.431982+0000","last_peered":"2024-04-02T13:20:26.431982+0000","last_clean":"2024-04-02T13:20:26.431982+0000","last_became_active":"2024-04-02T13:20:20.901628+0000","last_became_peered":"2024-04-02T13:20:20.901628+0000","last_unstale":"2024-04-02T13:20:26.431982+0000","last_undegraded":"2024-04-02T13:20:26.431982+0000","last_fullsized":"2024-04-02T13:20:26.431982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911122+0000","last_change":"2024-04-02T13:20:20.899007+0000","last_active":"2024-04-02T13:20:25.911122+0000","last_peered":"2024-04-02T13:20:25.911122+0000","last_clean":"2024-04-02T13:20:25.911122+0000","last_became_active":"2024-04-02T13:20:20.898005+0000","last_became_peered":"2024-04-02T13:20:20.898005+0000","last_unstale":"2024-04-02T13:20:25.911122+0000","last_undegraded":"2024-04-02T13:20:25.911122+0000","last_fullsized":"2024-04-02T13:20:25.911122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432495+0000","last_change":"2024-04-02T13:20:20.896893+0000","last_active":"2024-04-02T13:20:26.432495+0000","last_peered":"2024-04-02T13:20:26.432495+0000","last_clean":"2024-04-02T13:20:26.432495+0000","last_became_active":"2024-04-02T13:20:20.896629+0000","last_became_peered":"2024-04-02T13:20:20.896629+0000","last_unstale":"2024-04-02T13:20:26.432495+0000","last_undegraded":"2024-04-02T13:20:26.432495+0000","last_fullsized":"2024-04-02T13:20:26.432495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431161+0000","last_change":"2024-04-02T13:20:20.900656+0000","last_active":"2024-04-02T13:20:26.431161+0000","last_peered":"2024-04-02T13:20:26.431161+0000","last_clean":"2024-04-02T13:20:26.431161+0000","last_became_active":"2024-04-02T13:20:20.900385+0000","last_became_peered":"2024-04-02T13:20:20.900385+0000","last_unstale":"2024-04-02T13:20:26.431161+0000","last_undegraded":"2024-04-02T13:20:26.431161+0000","last_fullsized":"2024-04-02T13:20:26.431161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911108+0000","last_change":"2024-04-02T13:20:20.897613+0000","last_active":"2024-04-02T13:20:25.911108+0000","last_peered":"2024-04-02T13:20:25.911108+0000","last_clean":"2024-04-02T13:20:25.911108+0000","last_became_active":"2024-04-02T13:20:20.897285+0000","last_became_peered":"2024-04-02T13:20:20.897285+0000","last_unstale":"2024-04-02T13:20:25.911108+0000","last_undegraded":"2024-04-02T13:20:25.911108+0000","last_fullsized":"2024-04-02T13:20:25.911108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913747+0000","last_change":"2024-04-02T13:20:20.907084+0000","last_active":"2024-04-02T13:20:25.913747+0000","last_peered":"2024-04-02T13:20:25.913747+0000","last_clean":"2024-04-02T13:20:25.913747+0000","last_became_active":"2024-04-02T13:20:20.906734+0000","last_became_peered":"2024-04-02T13:20:20.906734+0000","last_unstale":"2024-04-02T13:20:25.913747+0000","last_undegraded":"2024-04-02T13:20:25.913747+0000","last_fullsized":"2024-04-02T13:20:25.913747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910526+0000","last_change":"2024-04-02T13:20:20.897768+0000","last_active":"2024-04-02T13:20:25.910526+0000","last_peered":"2024-04-02T13:20:25.910526+0000","last_clean":"2024-04-02T13:20:25.910526+0000","last_became_active":"2024-04-02T13:20:20.897502+0000","last_became_peered":"2024-04-02T13:20:20.897502+0000","last_unstale":"2024-04-02T13:20:25.910526+0000","last_undegraded":"2024-04-02T13:20:25.910526+0000","last_fullsized":"2024-04-02T13:20:25.910526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:26.532089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912825+0000","last_change":"2024-04-02T13:20:20.903454+0000","last_active":"2024-04-02T13:20:25.912825+0000","last_peered":"2024-04-02T13:20:25.912825+0000","last_clean":"2024-04-02T13:20:25.912825+0000","last_became_active":"2024-04-02T13:20:20.903050+0000","last_became_peered":"2024-04-02T13:20:20.903050+0000","last_unstale":"2024-04-02T13:20:25.912825+0000","last_undegraded":"2024-04-02T13:20:25.912825+0000","last_fullsized":"2024-04-02T13:20:25.912825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:56:59.954023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431891+0000","last_change":"2024-04-02T13:20:20.904681+0000","last_active":"2024-04-02T13:20:26.431891+0000","last_peered":"2024-04-02T13:20:26.431891+0000","last_clean":"2024-04-02T13:20:26.431891+0000","last_became_active":"2024-04-02T13:20:20.904343+0000","last_became_peered":"2024-04-02T13:20:20.904343+0000","last_unstale":"2024-04-02T13:20:26.431891+0000","last_undegraded":"2024-04-02T13:20:26.431891+0000","last_fullsized":"2024-04-02T13:20:26.431891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:22.414825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431123+0000","last_change":"2024-04-02T13:20:20.895363+0000","last_active":"2024-04-02T13:20:26.431123+0000","last_peered":"2024-04-02T13:20:26.431123+0000","last_clean":"2024-04-02T13:20:26.431123+0000","last_became_active":"2024-04-02T13:20:20.895107+0000","last_became_peered":"2024-04-02T13:20:20.895107+0000","last_unstale":"2024-04-02T13:20:26.431123+0000","last_undegraded":"2024-04-02T13:20:26.431123+0000","last_fullsized":"2024-04-02T13:20:26.431123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:22.414825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431618+0000","last_change":"2024-04-02T13:20:20.890501+0000","last_active":"2024-04-02T13:20:26.431618+0000","last_peered":"2024-04-02T13:20:26.431618+0000","last_clean":"2024-04-02T13:20:26.431618+0000","last_became_active":"2024-04-02T13:20:20.890181+0000","last_became_peered":"2024-04-02T13:20:20.890181+0000","last_unstale":"2024-04-02T13:20:26.431618+0000","last_undegraded":"2024-04-02T13:20:26.431618+0000","last_fullsized":"2024-04-02T13:20:26.431618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:02:18.661127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430915+0000","last_change":"2024-04-02T13:20:20.892430+0000","last_active":"2024-04-02T13:20:26.430915+0000","last_peered":"2024-04-02T13:20:26.430915+0000","last_clean":"2024-04-02T13:20:26.430915+0000","last_became_active":"2024-04-02T13:20:20.892117+0000","last_became_peered":"2024-04-02T13:20:20.892117+0000","last_unstale":"2024-04-02T13:20:26.430915+0000","last_undegraded":"2024-04-02T13:20:26.430915+0000","last_fullsized":"2024-04-02T13:20:26.430915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:37.366302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913765+0000","last_change":"2024-04-02T13:20:20.903371+0000","last_active":"2024-04-02T13:20:25.913765+0000","last_peered":"2024-04-02T13:20:25.913765+0000","last_clean":"2024-04-02T13:20:25.913765+0000","last_became_active":"2024-04-02T13:20:20.903078+0000","last_became_peered":"2024-04-02T13:20:20.903078+0000","last_unstale":"2024-04-02T13:20:25.913765+0000","last_undegraded":"2024-04-02T13:20:25.913765+0000","last_fullsized":"2024-04-02T13:20:25.913765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:42.709577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913653+0000","last_change":"2024-04-02T13:20:20.901550+0000","last_active":"2024-04-02T13:20:25.913653+0000","last_peered":"2024-04-02T13:20:25.913653+0000","last_clean":"2024-04-02T13:20:25.913653+0000","last_became_active":"2024-04-02T13:20:20.901234+0000","last_became_peered":"2024-04-02T13:20:20.901234+0000","last_unstale":"2024-04-02T13:20:25.913653+0000","last_undegraded":"2024-04-02T13:20:25.913653+0000","last_fullsized":"2024-04-02T13:20:25.913653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430422+0000","last_change":"2024-04-02T13:20:20.887101+0000","last_active":"2024-04-02T13:20:26.430422+0000","last_peered":"2024-04-02T13:20:26.430422+0000","last_clean":"2024-04-02T13:20:26.430422+0000","last_became_active":"2024-04-02T13:20:20.886810+0000","last_became_peered":"2024-04-02T13:20:20.886810+0000","last_unstale":"2024-04-02T13:20:26.430422+0000","last_undegraded":"2024-04-02T13:20:26.430422+0000","last_fullsized":"2024-04-02T13:20:26.430422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430889+0000","last_change":"2024-04-02T13:20:20.898387+0000","last_active":"2024-04-02T13:20:26.430889+0000","last_peered":"2024-04-02T13:20:26.430889+0000","last_clean":"2024-04-02T13:20:26.430889+0000","last_became_active":"2024-04-02T13:20:20.898075+0000","last_became_peered":"2024-04-02T13:20:20.898075+0000","last_unstale":"2024-04-02T13:20:26.430889+0000","last_undegraded":"2024-04-02T13:20:26.430889+0000","last_fullsized":"2024-04-02T13:20:26.430889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910512+0000","last_change":"2024-04-02T13:20:20.895603+0000","last_active":"2024-04-02T13:20:25.910512+0000","last_peered":"2024-04-02T13:20:25.910512+0000","last_clean":"2024-04-02T13:20:25.910512+0000","last_became_active":"2024-04-02T13:20:20.895316+0000","last_became_peered":"2024-04-02T13:20:20.895316+0000","last_unstale":"2024-04-02T13:20:25.910512+0000","last_undegraded":"2024-04-02T13:20:25.910512+0000","last_fullsized":"2024-04-02T13:20:25.910512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:34.303387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913831+0000","last_change":"2024-04-02T13:20:20.889381+0000","last_active":"2024-04-02T13:20:25.913831+0000","last_peered":"2024-04-02T13:20:25.913831+0000","last_clean":"2024-04-02T13:20:25.913831+0000","last_became_active":"2024-04-02T13:20:20.889102+0000","last_became_peered":"2024-04-02T13:20:20.889102+0000","last_unstale":"2024-04-02T13:20:25.913831+0000","last_undegraded":"2024-04-02T13:20:25.913831+0000","last_fullsized":"2024-04-02T13:20:25.913831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:37.366302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430695+0000","last_change":"2024-04-02T13:20:20.904474+0000","last_active":"2024-04-02T13:20:26.430695+0000","last_peered":"2024-04-02T13:20:26.430695+0000","last_clean":"2024-04-02T13:20:26.430695+0000","last_became_active":"2024-04-02T13:20:20.904212+0000","last_became_peered":"2024-04-02T13:20:20.904212+0000","last_unstale":"2024-04-02T13:20:26.430695+0000","last_undegraded":"2024-04-02T13:20:26.430695+0000","last_fullsized":"2024-04-02T13:20:26.430695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431059+0000","last_change":"2024-04-02T13:20:20.904469+0000","last_active":"2024-04-02T13:20:26.431059+0000","last_peered":"2024-04-02T13:20:26.431059+0000","last_clean":"2024-04-02T13:20:26.431059+0000","last_became_active":"2024-04-02T13:20:20.904195+0000","last_became_peered":"2024-04-02T13:20:20.904195+0000","last_unstale":"2024-04-02T13:20:26.431059+0000","last_undegraded":"2024-04-02T13:20:26.431059+0000","last_fullsized":"2024-04-02T13:20:26.431059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431650+0000","last_change":"2024-04-02T13:20:20.905842+0000","last_active":"2024-04-02T13:20:26.431650+0000","last_peered":"2024-04-02T13:20:26.431650+0000","last_clean":"2024-04-02T13:20:26.431650+0000","last_became_active":"2024-04-02T13:20:20.905558+0000","last_became_peered":"2024-04-02T13:20:20.905558+0000","last_unstale":"2024-04-02T13:20:26.431650+0000","last_undegraded":"2024-04-02T13:20:26.431650+0000","last_fullsized":"2024-04-02T13:20:26.431650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:02.784989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910120+0000","last_change":"2024-04-02T13:20:20.902048+0000","last_active":"2024-04-02T13:20:25.910120+0000","last_peered":"2024-04-02T13:20:25.910120+0000","last_clean":"2024-04-02T13:20:25.910120+0000","last_became_active":"2024-04-02T13:20:20.901733+0000","last_became_peered":"2024-04-02T13:20:20.901733+0000","last_unstale":"2024-04-02T13:20:25.910120+0000","last_undegraded":"2024-04-02T13:20:25.910120+0000","last_fullsized":"2024-04-02T13:20:25.910120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431149+0000","last_change":"2024-04-02T13:20:20.901947+0000","last_active":"2024-04-02T13:20:26.431149+0000","last_peered":"2024-04-02T13:20:26.431149+0000","last_clean":"2024-04-02T13:20:26.431149+0000","last_became_active":"2024-04-02T13:20:20.901655+0000","last_became_peered":"2024-04-02T13:20:20.901655+0000","last_unstale":"2024-04-02T13:20:26.431149+0000","last_undegraded":"2024-04-02T13:20:26.431149+0000","last_fullsized":"2024-04-02T13:20:26.431149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:56:59.954023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912733+0000","last_change":"2024-04-02T13:20:20.905389+0000","last_active":"2024-04-02T13:20:25.912733+0000","last_peered":"2024-04-02T13:20:25.912733+0000","last_clean":"2024-04-02T13:20:25.912733+0000","last_became_active":"2024-04-02T13:20:20.905062+0000","last_became_peered":"2024-04-02T13:20:20.905062+0000","last_unstale":"2024-04-02T13:20:25.912733+0000","last_undegraded":"2024-04-02T13:20:25.912733+0000","last_fullsized":"2024-04-02T13:20:25.912733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:34.303387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430753+0000","last_change":"2024-04-02T13:20:20.897784+0000","last_active":"2024-04-02T13:20:26.430753+0000","last_peered":"2024-04-02T13:20:26.430753+0000","last_clean":"2024-04-02T13:20:26.430753+0000","last_became_active":"2024-04-02T13:20:20.897488+0000","last_became_peered":"2024-04-02T13:20:20.897488+0000","last_unstale":"2024-04-02T13:20:26.430753+0000","last_undegraded":"2024-04-02T13:20:26.430753+0000","last_fullsized":"2024-04-02T13:20:26.430753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:26:16.298897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912800+0000","last_change":"2024-04-02T13:20:20.908726+0000","last_active":"2024-04-02T13:20:25.912800+0000","last_peered":"2024-04-02T13:20:25.912800+0000","last_clean":"2024-04-02T13:20:25.912800+0000","last_became_active":"2024-04-02T13:20:20.908504+0000","last_became_peered":"2024-04-02T13:20:20.908504+0000","last_unstale":"2024-04-02T13:20:25.912800+0000","last_undegraded":"2024-04-02T13:20:25.912800+0000","last_fullsized":"2024-04-02T13:20:25.912800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:26.532089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430330+0000","last_change":"2024-04-02T13:20:20.892134+0000","last_active":"2024-04-02T13:20:26.430330+0000","last_peered":"2024-04-02T13:20:26.430330+0000","last_clean":"2024-04-02T13:20:26.430330+0000","last_became_active":"2024-04-02T13:20:20.891860+0000","last_became_peered":"2024-04-02T13:20:20.891860+0000","last_unstale":"2024-04-02T13:20:26.430330+0000","last_undegraded":"2024-04-02T13:20:26.430330+0000","last_fullsized":"2024-04-02T13:20:26.430330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:17:34.808750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431697+0000","last_change":"2024-04-02T13:20:20.890864+0000","last_active":"2024-04-02T13:20:26.431697+0000","last_peered":"2024-04-02T13:20:26.431697+0000","last_clean":"2024-04-02T13:20:26.431697+0000","last_became_active":"2024-04-02T13:20:20.890585+0000","last_became_peered":"2024-04-02T13:20:20.890585+0000","last_unstale":"2024-04-02T13:20:26.431697+0000","last_undegraded":"2024-04-02T13:20:26.431697+0000","last_fullsized":"2024-04-02T13:20:26.431697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431543+0000","last_change":"2024-04-02T13:20:20.892679+0000","last_active":"2024-04-02T13:20:26.431543+0000","last_peered":"2024-04-02T13:20:26.431543+0000","last_clean":"2024-04-02T13:20:26.431543+0000","last_became_active":"2024-04-02T13:20:20.892440+0000","last_became_peered":"2024-04-02T13:20:20.892440+0000","last_unstale":"2024-04-02T13:20:26.431543+0000","last_undegraded":"2024-04-02T13:20:26.431543+0000","last_fullsized":"2024-04-02T13:20:26.431543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:39.096965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:20:44.347 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T13:20:44.348 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-02T13:20:44.348 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-02T13:20:44.348 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-02T13:20:44.348 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-02T13:20:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.481+0000 7f7d9f552640 1 Processor -- start 2024-04-02T13:20:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.481+0000 7f7d9f552640 1 -- start start 2024-04-02T13:20:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.482+0000 7f7d9f552640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7d980d0b20 0x7f7d980d0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.482+0000 7f7d9f552640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d9815e8b0 con 0x7f7d980b4aa0 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.482+0000 7f7d9f552640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d980d2fb0 con 0x7f7d9815e4d0 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.482+0000 7f7d9f552640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7d980b3a90 con 0x7f7d980d0b20 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.482+0000 7f7d9daef640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7d980b4aa0 0x7f7d980b4f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56386/0 (socket says 172.21.15.44:56386) 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.482+0000 7f7d9daef640 1 -- 172.21.15.44:0/3895758420 learned_addr learned my addr 172.21.15.44:0/3895758420 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d9caed640 1 --2- 172.21.15.44:0/3895758420 >> v2:172.21.15.69:3300/0 conn(0x7f7d980d0b20 0x7f7d980d0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 583292485 0 0) 0x7f7d9815e8b0 con 0x7f7d980b4aa0 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d80003520 con 0x7f7d980b4aa0 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2014370526 0 0) 0x7f7d980d2fb0 con 0x7f7d9815e4d0 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d9815e8b0 con 0x7f7d9815e4d0 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 500569111 0 0) 0x7f7d80003520 con 0x7f7d980b4aa0 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 >> v2:172.21.15.69:3300/0 conn(0x7f7d980d0b20 msgr2=0x7f7d980d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 --2- 172.21.15.44:0/3895758420 >> v2:172.21.15.69:3300/0 conn(0x7f7d980d0b20 0x7f7d980d0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 >> v1:172.21.15.69:6789/0 conn(0x7f7d9815e4d0 legacy=0x7f7d980cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d980b3ce0 con 0x7f7d980b4aa0 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f7d880019f0 con 0x7f7d980b4aa0 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 >> v1:172.21.15.44:6789/0 conn(0x7f7d980b4aa0 legacy=0x7f7d980b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 --2- 172.21.15.44:0/3895758420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d80003b90 0x7f7d80003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 --2- 172.21.15.44:0/3895758420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d80004800 0x7f7d80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 --2- 172.21.15.44:0/3895758420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d800051c0 0x7f7d8000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7d980b3a90 con 0x7f7d80004800 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8000c090 con 0x7f7d800051c0 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8000c290 con 0x7f7d80003b90 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d9caed640 1 --2- 172.21.15.44:0/3895758420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d80004800 0x7f7d80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d9d2ee640 1 --2- 172.21.15.44:0/3895758420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d80003b90 0x7f7d80003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d9daef640 1 --2- 172.21.15.44:0/3895758420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d800051c0 0x7f7d8000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d9caed640 1 -- 172.21.15.44:0/3895758420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d80003b90 msgr2=0x7f7d80003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d9caed640 1 --2- 172.21.15.44:0/3895758420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d80003b90 0x7f7d80003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d9caed640 1 -- 172.21.15.44:0/3895758420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d800051c0 msgr2=0x7f7d8000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:44.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d9caed640 1 --2- 172.21.15.44:0/3895758420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d800051c0 0x7f7d8000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d9caed640 1 -- 172.21.15.44:0/3895758420 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d80003520 con 0x7f7d80004800 2024-04-02T13:20:44.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.483+0000 7f7d9daef640 1 -- 172.21.15.44:0/3895758420 reap_dead start 2024-04-02T13:20:44.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.484+0000 7f7d9caed640 1 --2- 172.21.15.44:0/3895758420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d80004800 0x7f7d80004c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c009ac0 tx=0x7f7d80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d834fec1a197e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:44.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.484+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c039070 con 0x7f7d80004800 2024-04-02T13:20:44.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.484+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d8c02c6b0 con 0x7f7d80004800 2024-04-02T13:20:44.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.484+0000 7f7d97fff640 1 -- 172.21.15.44:0/3895758420 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c009690 con 0x7f7d80004800 2024-04-02T13:20:44.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.485+0000 7f7d9f552640 1 -- 172.21.15.44:0/3895758420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d80004800 msgr2=0x7f7d80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.485+0000 7f7d9f552640 1 --2- 172.21.15.44:0/3895758420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d80004800 0x7f7d80004c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c009ac0 tx=0x7f7d80005830 comp rx=0 tx=0).stop 2024-04-02T13:20:44.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.485+0000 7f7d9f552640 1 -- 172.21.15.44:0/3895758420 shutdown_connections 2024-04-02T13:20:44.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.485+0000 7f7d9f552640 1 --2- 172.21.15.44:0/3895758420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d80004800 0x7f7d80004c80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.485+0000 7f7d9f552640 1 -- 172.21.15.44:0/3895758420 >> 172.21.15.44:0/3895758420 conn(0x7f7d9809d410 msgr2=0x7f7d980d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:44.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.485+0000 7f7d9f552640 1 -- 172.21.15.44:0/3895758420 shutdown_connections 2024-04-02T13:20:44.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.485+0000 7f7d9f552640 1 -- 172.21.15.44:0/3895758420 wait complete. 2024-04-02T13:20:44.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.486+0000 7f7d9f552640 1 Processor -- start 2024-04-02T13:20:44.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.486+0000 7f7d9f552640 1 -- start start 2024-04-02T13:20:44.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9f552640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d80004800 0x7f7d980cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9f552640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d980c9c60 0x7f7d980ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9f552640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9815f830 0x7f7d9815fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9f552640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7d980b42a0 con 0x7f7d980c9c60 2024-04-02T13:20:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9f552640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7d980b4610 con 0x7f7d9815f830 2024-04-02T13:20:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9f552640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7d980b0d00 con 0x7f7d80004800 2024-04-02T13:20:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9caed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d980c9c60 0x7f7d980ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9caed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d980c9c60 0x7f7d980ca0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59080/0 (socket says 172.21.15.44:59080) 2024-04-02T13:20:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9caed640 1 -- 172.21.15.44:0/2299632244 learned_addr learned my addr 172.21.15.44:0/2299632244 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9d2ee640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d80004800 0x7f7d980cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9daef640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9815f830 0x7f7d9815fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9caed640 1 -- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d80004800 msgr2=0x7f7d980cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9caed640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d80004800 0x7f7d980cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9caed640 1 -- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9815f830 msgr2=0x7f7d9815fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9caed640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9815f830 0x7f7d9815fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9caed640 1 -- 172.21.15.44:0/2299632244 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d98155f10 con 0x7f7d980c9c60 2024-04-02T13:20:44.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.487+0000 7f7d9caed640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d980c9c60 0x7f7d980ca0e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c000c00 tx=0x7f7d8c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:44.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.488+0000 7f7d95ffb640 1 -- 172.21.15.44:0/2299632244 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c042020 con 0x7f7d980c9c60 2024-04-02T13:20:44.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.488+0000 7f7d95ffb640 1 -- 172.21.15.44:0/2299632244 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d8c03a040 con 0x7f7d980c9c60 2024-04-02T13:20:44.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.488+0000 7f7d9f552640 1 -- 172.21.15.44:0/2299632244 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d981561c0 con 0x7f7d980c9c60 2024-04-02T13:20:44.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.488+0000 7f7d95ffb640 1 -- 172.21.15.44:0/2299632244 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c039070 con 0x7f7d980c9c60 2024-04-02T13:20:44.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.488+0000 7f7d9f552640 1 -- 172.21.15.44:0/2299632244 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d98156780 con 0x7f7d980c9c60 2024-04-02T13:20:44.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.489+0000 7f7d9f552640 1 -- 172.21.15.44:0/2299632244 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d98068c00 con 0x7f7d980c9c60 2024-04-02T13:20:44.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.489+0000 7f7d95ffb640 1 -- 172.21.15.44:0/2299632244 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7d8c045020 con 0x7f7d980c9c60 2024-04-02T13:20:44.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.490+0000 7f7d95ffb640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7d8406c8f0 0x7f7d8406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.490+0000 7f7d95ffb640 1 -- 172.21.15.44:0/2299632244 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7d8c0ad260 con 0x7f7d980c9c60 2024-04-02T13:20:44.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.490+0000 7f7d9d2ee640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7d8406c8f0 0x7f7d8406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.490+0000 7f7d9d2ee640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7d8406c8f0 0x7f7d8406edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d900055b0 tx=0x7f7d90009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:44.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.494+0000 7f7d95ffb640 1 -- 172.21.15.44:0/2299632244 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7d8c04c070 con 0x7f7d980c9c60 2024-04-02T13:20:44.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.731+0000 7f7d9f552640 1 -- 172.21.15.44:0/2299632244 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f7d980d23d0 con 0x7f7d980c9c60 2024-04-02T13:20:44.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.734+0000 7f7d95ffb640 1 -- 172.21.15.44:0/2299632244 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+174135 (secure 0 0 0) 0x7f7d8c07b350 con 0x7f7d980c9c60 2024-04-02T13:20:44.738 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:44.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.737+0000 7f7d9f552640 1 -- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7d8406c8f0 msgr2=0x7f7d8406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.737+0000 7f7d9f552640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7d8406c8f0 0x7f7d8406edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d900055b0 tx=0x7f7d90009040 comp rx=0 tx=0).stop 2024-04-02T13:20:44.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.737+0000 7f7d9f552640 1 -- 172.21.15.44:0/2299632244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d980c9c60 msgr2=0x7f7d980ca0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.737+0000 7f7d9f552640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d980c9c60 0x7f7d980ca0e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c000c00 tx=0x7f7d8c004730 comp rx=0 tx=0).stop 2024-04-02T13:20:44.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.737+0000 7f7d9f552640 1 -- 172.21.15.44:0/2299632244 shutdown_connections 2024-04-02T13:20:44.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.737+0000 7f7d9f552640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7d8406c8f0 0x7f7d8406edb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.737+0000 7f7d9f552640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9815f830 0x7f7d9815fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.737+0000 7f7d9f552640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d980c9c60 0x7f7d980ca0e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.737+0000 7f7d9f552640 1 --2- 172.21.15.44:0/2299632244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d80004800 0x7f7d980cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.737+0000 7f7d9f552640 1 -- 172.21.15.44:0/2299632244 >> 172.21.15.44:0/2299632244 conn(0x7f7d9809d410 msgr2=0x7f7d98067230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:44.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.737+0000 7f7d9f552640 1 -- 172.21.15.44:0/2299632244 shutdown_connections 2024-04-02T13:20:44.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.737+0000 7f7d9f552640 1 -- 172.21.15.44:0/2299632244 wait complete. 2024-04-02T13:20:44.750 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":3569935720},{"type":"v1","addr":"172.21.15.69:6833","nonce":3569935720}]},"active_addr":"172.21.15.69:6833/3569935720","active_change":"2024-04-02T13:20:06.865758+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4101,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":1964107151}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":172742505}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":321014914}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":2265309164}]}]} 2024-04-02T13:20:44.753 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-02T13:20:44.753 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-02T13:20:44.753 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:44.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.852+0000 7fba5cfbc640 1 Processor -- start 2024-04-02T13:20:44.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.852+0000 7fba5cfbc640 1 -- start start 2024-04-02T13:20:44.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba5cfbc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba5815e0f0 0x7fba580d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba5cfbc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fba580d0630 0x7fba5806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba5cfbc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fba5815e4d0 0x7fba5806ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba5cfbc640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fba5806dfe0 con 0x7fba5815e4d0 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba5cfbc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fba5806e2c0 con 0x7fba580d0630 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba5cfbc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fba5806e5a0 con 0x7fba5815e0f0 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba56d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fba5815e4d0 0x7fba5806ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba56d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fba5815e4d0 0x7fba5806ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54610/0 (socket says 172.21.15.44:54610) 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba56d9d640 1 -- 172.21.15.44:0/622274639 learned_addr learned my addr 172.21.15.44:0/622274639 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba55d9b640 1 --2- 172.21.15.44:0/622274639 >> v2:172.21.15.69:3300/0 conn(0x7fba580d0630 0x7fba5806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba5659c640 1 --2- 172.21.15.44:0/622274639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba5815e0f0 0x7fba580d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba56d9d640 1 -- 172.21.15.44:0/622274639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba5815e0f0 msgr2=0x7fba580d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba56d9d640 1 --2- 172.21.15.44:0/622274639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba5815e0f0 0x7fba580d00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba56d9d640 1 -- 172.21.15.44:0/622274639 >> v2:172.21.15.69:3300/0 conn(0x7fba580d0630 msgr2=0x7fba5806b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba56d9d640 1 --2- 172.21.15.44:0/622274639 >> v2:172.21.15.69:3300/0 conn(0x7fba580d0630 0x7fba5806b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.853+0000 7fba56d9d640 1 -- 172.21.15.44:0/622274639 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba5806e880 con 0x7fba5815e4d0 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.854+0000 7fba56d9d640 1 --2- 172.21.15.44:0/622274639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5815e4d0 0x7fba5806ba10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fba5806e4d0 tx=0x7fba4802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3c64b21cfce013 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:44.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.854+0000 7fba5559a640 1 -- 172.21.15.44:0/622274639 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4802caf0 con 0x7fba5815e4d0 2024-04-02T13:20:44.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.854+0000 7fba5559a640 1 -- 172.21.15.44:0/622274639 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba4802cc90 con 0x7fba5815e4d0 2024-04-02T13:20:44.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.854+0000 7fba5559a640 1 -- 172.21.15.44:0/622274639 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba48034610 con 0x7fba5815e4d0 2024-04-02T13:20:44.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.855+0000 7fba5cfbc640 1 -- 172.21.15.44:0/622274639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5815e4d0 msgr2=0x7fba5806ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.855+0000 7fba5cfbc640 1 --2- 172.21.15.44:0/622274639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5815e4d0 0x7fba5806ba10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fba5806e4d0 tx=0x7fba4802bf50 comp rx=0 tx=0).stop 2024-04-02T13:20:44.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.855+0000 7fba5cfbc640 1 -- 172.21.15.44:0/622274639 shutdown_connections 2024-04-02T13:20:44.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.855+0000 7fba5cfbc640 1 --2- 172.21.15.44:0/622274639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5815e4d0 0x7fba5806ba10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.855+0000 7fba5cfbc640 1 --2- 172.21.15.44:0/622274639 >> v2:172.21.15.69:3300/0 conn(0x7fba580d0630 0x7fba5806b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.855+0000 7fba5cfbc640 1 --2- 172.21.15.44:0/622274639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba5815e0f0 0x7fba580d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.855+0000 7fba5cfbc640 1 -- 172.21.15.44:0/622274639 >> 172.21.15.44:0/622274639 conn(0x7fba580b2980 msgr2=0x7fba580b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:44.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.855+0000 7fba5cfbc640 1 -- 172.21.15.44:0/622274639 shutdown_connections 2024-04-02T13:20:44.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.855+0000 7fba5cfbc640 1 -- 172.21.15.44:0/622274639 wait complete. 2024-04-02T13:20:44.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.856+0000 7fba5cfbc640 1 Processor -- start 2024-04-02T13:20:44.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.856+0000 7fba5cfbc640 1 -- start start 2024-04-02T13:20:44.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba5cfbc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba580d0630 0x7fba5814b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba5cfbc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba5815e0f0 0x7fba580cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba5cfbc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5815e4d0 0x7fba580cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba5cfbc640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fba58072f60 con 0x7fba5815e4d0 2024-04-02T13:20:44.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba5cfbc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fba5806eda0 con 0x7fba5815e0f0 2024-04-02T13:20:44.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba5cfbc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fba580730e0 con 0x7fba580d0630 2024-04-02T13:20:44.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba56d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5815e4d0 0x7fba580cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba56d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5815e4d0 0x7fba580cc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54614/0 (socket says 172.21.15.44:54614) 2024-04-02T13:20:44.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba56d9d640 1 -- 172.21.15.44:0/1206372694 learned_addr learned my addr 172.21.15.44:0/1206372694 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:44.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba5659c640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba580d0630 0x7fba5814b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba55d9b640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba5815e0f0 0x7fba580cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba56d9d640 1 -- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba580d0630 msgr2=0x7fba5814b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba56d9d640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba580d0630 0x7fba5814b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba56d9d640 1 -- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba5815e0f0 msgr2=0x7fba580cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:44.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba56d9d640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba5815e0f0 0x7fba580cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:44.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.857+0000 7fba56d9d640 1 -- 172.21.15.44:0/1206372694 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba58149e40 con 0x7fba5815e4d0 2024-04-02T13:20:44.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.858+0000 7fba56d9d640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5815e4d0 0x7fba580cc430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fba48009f40 tx=0x7fba48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:44.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.858+0000 7fba46ffd640 1 -- 172.21.15.44:0/1206372694 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba48044020 con 0x7fba5815e4d0 2024-04-02T13:20:44.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.858+0000 7fba46ffd640 1 -- 172.21.15.44:0/1206372694 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba48040670 con 0x7fba5815e4d0 2024-04-02T13:20:44.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.858+0000 7fba46ffd640 1 -- 172.21.15.44:0/1206372694 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba48009510 con 0x7fba5815e4d0 2024-04-02T13:20:44.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.858+0000 7fba5cfbc640 1 -- 172.21.15.44:0/1206372694 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba580cca90 con 0x7fba5815e4d0 2024-04-02T13:20:44.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.858+0000 7fba5cfbc640 1 -- 172.21.15.44:0/1206372694 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba580cd050 con 0x7fba5815e4d0 2024-04-02T13:20:44.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.859+0000 7fba5cfbc640 1 -- 172.21.15.44:0/1206372694 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba580684f0 con 0x7fba5815e4d0 2024-04-02T13:20:44.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.859+0000 7fba46ffd640 1 -- 172.21.15.44:0/1206372694 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fba48047020 con 0x7fba5815e4d0 2024-04-02T13:20:44.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.859+0000 7fba46ffd640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fba3806c8f0 0x7fba3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:44.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.860+0000 7fba46ffd640 1 -- 172.21.15.44:0/1206372694 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fba480aeba0 con 0x7fba5815e4d0 2024-04-02T13:20:44.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.860+0000 7fba5659c640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fba3806c8f0 0x7fba3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:44.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.860+0000 7fba5659c640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fba3806c8f0 0x7fba3806edb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fba5000aa40 tx=0x7fba50008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:44.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:44.864+0000 7fba46ffd640 1 -- 172.21.15.44:0/1206372694 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fba4804e020 con 0x7fba5815e4d0 2024-04-02T13:20:45.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.054+0000 7fba5cfbc640 1 -- 172.21.15.44:0/1206372694 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fba580b30a0 con 0x7fba5815e4d0 2024-04-02T13:20:45.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.055+0000 7fba46ffd640 1 -- 172.21.15.44:0/1206372694 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7fba480ae020 con 0x7fba5815e4d0 2024-04-02T13:20:45.058 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:45.058 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:45.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.058+0000 7fba5cfbc640 1 -- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fba3806c8f0 msgr2=0x7fba3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.058+0000 7fba5cfbc640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fba3806c8f0 0x7fba3806edb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fba5000aa40 tx=0x7fba50008040 comp rx=0 tx=0).stop 2024-04-02T13:20:45.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.058+0000 7fba5cfbc640 1 -- 172.21.15.44:0/1206372694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5815e4d0 msgr2=0x7fba580cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.058+0000 7fba5cfbc640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5815e4d0 0x7fba580cc430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fba48009f40 tx=0x7fba48004a40 comp rx=0 tx=0).stop 2024-04-02T13:20:45.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.058+0000 7fba5cfbc640 1 -- 172.21.15.44:0/1206372694 shutdown_connections 2024-04-02T13:20:45.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.058+0000 7fba5cfbc640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fba3806c8f0 0x7fba3806edb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.058+0000 7fba5cfbc640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5815e4d0 0x7fba580cc430 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.058+0000 7fba5cfbc640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba5815e0f0 0x7fba580cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.058+0000 7fba5cfbc640 1 --2- 172.21.15.44:0/1206372694 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba580d0630 0x7fba5814b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.058+0000 7fba5cfbc640 1 -- 172.21.15.44:0/1206372694 >> 172.21.15.44:0/1206372694 conn(0x7fba580b2980 msgr2=0x7fba5806ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.058+0000 7fba5cfbc640 1 -- 172.21.15.44:0/1206372694 shutdown_connections 2024-04-02T13:20:45.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.058+0000 7fba5cfbc640 1 -- 172.21.15.44:0/1206372694 wait complete. 2024-04-02T13:20:45.071 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-02T13:20:45.071 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:45.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.171+0000 7f81a2c6a640 1 Processor -- start 2024-04-02T13:20:45.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.171+0000 7f81a2c6a640 1 -- start start 2024-04-02T13:20:45.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.172+0000 7f81a2c6a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f819c0d2040 0x7f819c0d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.172+0000 7f81a2c6a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f819c0d0000 con 0x7f819c06cf90 2024-04-02T13:20:45.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.172+0000 7f81a2c6a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f819c06c1d0 con 0x7f819c0d0a30 2024-04-02T13:20:45.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.172+0000 7f81a2c6a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f819c06cb00 con 0x7f819c0d2040 2024-04-02T13:20:45.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.172+0000 7f81a1207640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f819c0d2040 0x7f819c0d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.172+0000 7f819bfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f819c06cf90 0x7f819c0cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57374/0 (socket says 172.21.15.44:57374) 2024-04-02T13:20:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.172+0000 7f819bfff640 1 -- 172.21.15.44:0/2560037419 learned_addr learned my addr 172.21.15.44:0/2560037419 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.172+0000 7f81a1207640 1 -- 172.21.15.44:0/2560037419 >> v1:172.21.15.69:6789/0 conn(0x7f819c0d0a30 legacy=0x7f819c0d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.172+0000 7f81a1207640 1 -- 172.21.15.44:0/2560037419 >> v1:172.21.15.44:6789/0 conn(0x7f819c06cf90 legacy=0x7f819c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.172+0000 7f819b7fe640 1 -- 172.21.15.44:0/2560037419 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2129866545 0 0) 0x7f819c0d0000 con 0x7f819c06cf90 2024-04-02T13:20:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.172+0000 7f81a1207640 1 -- 172.21.15.44:0/2560037419 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f819c06c4b0 con 0x7f819c0d2040 2024-04-02T13:20:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.173+0000 7f81a1207640 1 --2- 172.21.15.44:0/2560037419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819c0d2040 0x7f819c0d24f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8194009d90 tx=0x7f819402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff194d7db00f773 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.173+0000 7f819b7fe640 1 -- 172.21.15.44:0/2560037419 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8194039070 con 0x7f819c0d2040 2024-04-02T13:20:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.173+0000 7f819b7fe640 1 -- 172.21.15.44:0/2560037419 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f819402cbd0 con 0x7f819c0d2040 2024-04-02T13:20:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.173+0000 7f819b7fe640 1 -- 172.21.15.44:0/2560037419 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8194034440 con 0x7f819c0d2040 2024-04-02T13:20:45.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.174+0000 7f81a2c6a640 1 -- 172.21.15.44:0/2560037419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819c0d2040 msgr2=0x7f819c0d24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.174+0000 7f81a2c6a640 1 --2- 172.21.15.44:0/2560037419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819c0d2040 0x7f819c0d24f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8194009d90 tx=0x7f819402be20 comp rx=0 tx=0).stop 2024-04-02T13:20:45.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.174+0000 7f81a2c6a640 1 -- 172.21.15.44:0/2560037419 shutdown_connections 2024-04-02T13:20:45.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.174+0000 7f81a2c6a640 1 --2- 172.21.15.44:0/2560037419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819c0d2040 0x7f819c0d24f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.174+0000 7f81a2c6a640 1 -- 172.21.15.44:0/2560037419 >> 172.21.15.44:0/2560037419 conn(0x7f819c09d470 msgr2=0x7f819c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.174+0000 7f81a2c6a640 1 -- 172.21.15.44:0/2560037419 shutdown_connections 2024-04-02T13:20:45.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.174+0000 7f81a2c6a640 1 -- 172.21.15.44:0/2560037419 wait complete. 2024-04-02T13:20:45.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.175+0000 7f81a2c6a640 1 Processor -- start 2024-04-02T13:20:45.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.175+0000 7f81a2c6a640 1 -- start start 2024-04-02T13:20:45.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81a2c6a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f819c06cf90 0x7f819c0cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81a2c6a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819c0d0a30 0x7f819c160a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81a2c6a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f819c0d2040 0x7f819c160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81a2c6a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f819c15e6f0 con 0x7f819c0d0a30 2024-04-02T13:20:45.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81a2c6a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f819c06d3f0 con 0x7f819c06cf90 2024-04-02T13:20:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81a2c6a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f819c15e870 con 0x7f819c0d2040 2024-04-02T13:20:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f819bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819c0d0a30 0x7f819c160a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f819bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819c0d0a30 0x7f819c160a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54622/0 (socket says 172.21.15.44:54622) 2024-04-02T13:20:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f819bfff640 1 -- 172.21.15.44:0/1128715306 learned_addr learned my addr 172.21.15.44:0/1128715306 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81a0a06640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f819c06cf90 0x7f819c0cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f819bfff640 1 -- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f819c0d2040 msgr2=0x7f819c160fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81a1207640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f819c0d2040 0x7f819c160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f819bfff640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f819c0d2040 0x7f819c160fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f819bfff640 1 -- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f819c06cf90 msgr2=0x7f819c0cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f819bfff640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f819c06cf90 0x7f819c0cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f819bfff640 1 -- 172.21.15.44:0/1128715306 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f819c161510 con 0x7f819c0d0a30 2024-04-02T13:20:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81a0a06640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f819c06cf90 0x7f819c0cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:45.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f819bfff640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819c0d0a30 0x7f819c160a90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f818800c440 tx=0x7f818800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81997fa640 1 -- 172.21.15.44:0/1128715306 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188018070 con 0x7f819c0d0a30 2024-04-02T13:20:45.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81997fa640 1 -- 172.21.15.44:0/1128715306 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81880074e0 con 0x7f819c0d0a30 2024-04-02T13:20:45.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81a2c6a640 1 -- 172.21.15.44:0/1128715306 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f819c15fcd0 con 0x7f819c0d0a30 2024-04-02T13:20:45.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81997fa640 1 -- 172.21.15.44:0/1128715306 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188004a20 con 0x7f819c0d0a30 2024-04-02T13:20:45.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.176+0000 7f81a2c6a640 1 -- 172.21.15.44:0/1128715306 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f819c160290 con 0x7f819c0d0a30 2024-04-02T13:20:45.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.177+0000 7f81a2c6a640 1 -- 172.21.15.44:0/1128715306 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f819c0684f0 con 0x7f819c0d0a30 2024-04-02T13:20:45.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.178+0000 7f81997fa640 1 -- 172.21.15.44:0/1128715306 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f818801b070 con 0x7f819c0d0a30 2024-04-02T13:20:45.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.178+0000 7f81997fa640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f818006c920 0x7f818006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.178+0000 7f81997fa640 1 -- 172.21.15.44:0/1128715306 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f818808b120 con 0x7f819c0d0a30 2024-04-02T13:20:45.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.178+0000 7f81a0a06640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f818006c920 0x7f818006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.179+0000 7f81a0a06640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f818006c920 0x7f818006ede0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f819000a830 tx=0x7f8190008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.183+0000 7f81997fa640 1 -- 172.21.15.44:0/1128715306 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8188059200 con 0x7f819c0d0a30 2024-04-02T13:20:45.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.368+0000 7f81a2c6a640 1 -- 172.21.15.44:0/1128715306 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f819c0d17c0 con 0x7f819c0d0a30 2024-04-02T13:20:45.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.369+0000 7f81997fa640 1 -- 172.21.15.44:0/1128715306 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f8188058b00 con 0x7f819c0d0a30 2024-04-02T13:20:45.372 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:45.372 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:45.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.371+0000 7f81a2c6a640 1 -- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f818006c920 msgr2=0x7f818006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.371+0000 7f81a2c6a640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f818006c920 0x7f818006ede0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f819000a830 tx=0x7f8190008040 comp rx=0 tx=0).stop 2024-04-02T13:20:45.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.371+0000 7f81a2c6a640 1 -- 172.21.15.44:0/1128715306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819c0d0a30 msgr2=0x7f819c160a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.371+0000 7f81a2c6a640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819c0d0a30 0x7f819c160a90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f818800c440 tx=0x7f818800c900 comp rx=0 tx=0).stop 2024-04-02T13:20:45.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.371+0000 7f81a2c6a640 1 -- 172.21.15.44:0/1128715306 shutdown_connections 2024-04-02T13:20:45.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.371+0000 7f81a2c6a640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f818006c920 0x7f818006ede0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.371+0000 7f81a2c6a640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f819c0d2040 0x7f819c160fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.371+0000 7f81a2c6a640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819c0d0a30 0x7f819c160a90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.371+0000 7f81a2c6a640 1 --2- 172.21.15.44:0/1128715306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f819c06cf90 0x7f819c0cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.371+0000 7f81a2c6a640 1 -- 172.21.15.44:0/1128715306 >> 172.21.15.44:0/1128715306 conn(0x7f819c09d470 msgr2=0x7f819c152e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.371+0000 7f81a2c6a640 1 -- 172.21.15.44:0/1128715306 shutdown_connections 2024-04-02T13:20:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.371+0000 7f81a2c6a640 1 -- 172.21.15.44:0/1128715306 wait complete. 2024-04-02T13:20:45.385 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-02T13:20:45.385 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-02T13:20:45.386 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-02T13:20:45.386 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-02T13:20:45.387 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-02T13:20:45.387 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-02T13:20:45.387 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-02T13:20:45.388 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-02T13:20:45.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.559+0000 7f4eb0915640 1 Processor -- start 2024-04-02T13:20:45.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.559+0000 7f4eb0915640 1 -- start start 2024-04-02T13:20:45.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.559+0000 7fef47dcd640 1 Processor -- start 2024-04-02T13:20:45.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.559+0000 7fef47dcd640 1 -- start start 2024-04-02T13:20:45.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.560+0000 7f5bb3dcd640 1 Processor -- start 2024-04-02T13:20:45.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.560+0000 7fdda72af640 1 Processor -- start 2024-04-02T13:20:45.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.560+0000 7f5bb3dcd640 1 -- start start 2024-04-02T13:20:45.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.560+0000 7fe8183c7640 1 Processor -- start 2024-04-02T13:20:45.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.560+0000 7fe8183c7640 1 -- start start 2024-04-02T13:20:45.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fe8183c7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe8100d0c00 0x7fe8100ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fe8183c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8100cd290 0x7fe8100c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fe8183c7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8100cdac0 con 0x7fe81015c1b0 2024-04-02T13:20:45.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fe8183c7640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe8100cdda0 con 0x7fe8100d0c00 2024-04-02T13:20:45.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fe8183c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe8100ce080 con 0x7fe8100cd290 2024-04-02T13:20:45.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7f4eb0915640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4eac0ca0f0 0x7f4eac0ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7f4eb0915640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4eac0cd670 0x7f4eac0cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7f4eaa59c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4eac0aedd0 0x7f4eac0cbb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:48624/0 (socket says 172.21.15.44:48624) 2024-04-02T13:20:45.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7f4eaa59c640 1 -- 172.21.15.44:0/2005080665 learned_addr learned my addr 172.21.15.44:0/2005080665 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:45.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7f4eb0915640 1 -- 172.21.15.44:0/2005080665 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4eac06d440 con 0x7f4eac0aedd0 2024-04-02T13:20:45.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7f4eb0915640 1 -- 172.21.15.44:0/2005080665 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4eac0ce7b0 con 0x7f4eac0ca0f0 2024-04-02T13:20:45.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7f4eb0915640 1 -- 172.21.15.44:0/2005080665 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4eac0cea90 con 0x7f4eac0cd670 2024-04-02T13:20:45.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7f4ea9d9b640 1 --2- 172.21.15.44:0/2005080665 >> v2:172.21.15.69:3300/0 conn(0x7f4eac0ca0f0 0x7f4eac0ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.560+0000 7ff3b592b640 1 Processor -- start 2024-04-02T13:20:45.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fe816964640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8100cd290 0x7fe8100c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fe816163640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe81015c1b0 0x7fe8100d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:48636/0 (socket says 172.21.15.44:48636) 2024-04-02T13:20:45.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fe816163640 1 -- 172.21.15.44:0/1435200180 learned_addr learned my addr 172.21.15.44:0/1435200180 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:45.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fef47dcd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fef400b5260 0x7fef4006f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fef47dcd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fef400b5640 0x7fef401635f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fef47dcd640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef4006f690 con 0x7fef40071a50 2024-04-02T13:20:45.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fef47dcd640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fef401672f0 con 0x7fef400b5640 2024-04-02T13:20:45.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fef47dcd640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fef40145af0 con 0x7fef400b5260 2024-04-02T13:20:45.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fef45b69640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fef400b5260 0x7fef4006f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fef4636a640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fef40071a50 0x7fef40071f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57376/0 (socket says 172.21.15.44:57376) 2024-04-02T13:20:45.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fef4636a640 1 -- 172.21.15.44:0/3098438229 learned_addr learned my addr 172.21.15.44:0/3098438229 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:45.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fef44b67640 1 -- 172.21.15.44:0/3098438229 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2509362137 0 0) 0x7fef38001490 con 0x7fef40071a50 2024-04-02T13:20:45.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fef44b67640 1 -- 172.21.15.44:0/3098438229 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef2c003520 con 0x7fef40071a50 2024-04-02T13:20:45.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fef45b69640 1 -- 172.21.15.44:0/3098438229 >> v2:172.21.15.44:3300/0 conn(0x7fef400b5640 msgr2=0x7fef401635f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T13:20:45.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fef45b69640 1 --2- 172.21.15.44:0/3098438229 >> v2:172.21.15.44:3300/0 conn(0x7fef400b5640 0x7fef401635f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fef45b69640 1 -- 172.21.15.44:0/3098438229 >> v1:172.21.15.44:6789/0 conn(0x7fef40071a50 legacy=0x7fef40071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fef45b69640 1 -- 172.21.15.44:0/3098438229 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4006d950 con 0x7fef400b5260 2024-04-02T13:20:45.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.560+0000 7fdda72af640 1 -- start start 2024-04-02T13:20:45.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fdda72af640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdd980ead00 0x7fdd9804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fdda72af640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd9804ffd0 0x7fdd98037bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fdda72af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd98038110 0x7fdd980385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fdda72af640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fdd98054330 con 0x7fdd980ead00 2024-04-02T13:20:45.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fdda72af640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fdd98053720 con 0x7fdd9804ffd0 2024-04-02T13:20:45.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7fdda72af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdd98053a00 con 0x7fdd98038110 2024-04-02T13:20:45.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fdda5aac640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd9804ffd0 0x7fdd98037bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fdda5aac640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd9804ffd0 0x7fdd98037bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34106/0 (socket says 172.21.15.44:34106) 2024-04-02T13:20:45.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fdda5aac640 1 -- 172.21.15.44:0/4049199382 learned_addr learned my addr 172.21.15.44:0/4049199382 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fdda6aae640 1 --2- 172.21.15.44:0/4049199382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd98038110 0x7fdd980385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fdda62ad640 1 --2- 172.21.15.44:0/4049199382 >> v2:172.21.15.44:3300/0 conn(0x7fdd980ead00 0x7fdd9804fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 322459020 0 0) 0x7f4eac06d440 con 0x7f4eac0aedd0 2024-04-02T13:20:45.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e94003520 con 0x7f4eac0aedd0 2024-04-02T13:20:45.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7f4eaad9d640 1 --2- 172.21.15.44:0/2005080665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4eac0cd670 0x7f4eac0cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fe815962640 1 --2- 172.21.15.44:0/1435200180 >> v2:172.21.15.44:3300/0 conn(0x7fe8100d0c00 0x7fe8100ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fe815161640 1 -- 172.21.15.44:0/1435200180 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 259116493 0 0) 0x7fe8100cdac0 con 0x7fe81015c1b0 2024-04-02T13:20:45.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fe815161640 1 -- 172.21.15.44:0/1435200180 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7fc003520 con 0x7fe81015c1b0 2024-04-02T13:20:45.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.560+0000 7ff3b592b640 1 -- start start 2024-04-02T13:20:45.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7ff3b592b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff3b00ccdf0 0x7ff3b006d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7ff3b592b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff3b0166cb0 0x7ff3b00b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7ff3b592b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3b00b3eb0 con 0x7ff3b0162800 2024-04-02T13:20:45.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7ff3b592b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff3b0162be0 con 0x7ff3b00ccdf0 2024-04-02T13:20:45.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.561+0000 7ff3b592b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff3b00b47a0 con 0x7ff3b0166cb0 2024-04-02T13:20:45.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7ff3aeffd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff3b0162800 0x7ff3b00cc6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:48650/0 (socket says 172.21.15.44:48650) 2024-04-02T13:20:45.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7ff3aeffd640 1 -- 172.21.15.44:0/865852313 learned_addr learned my addr 172.21.15.44:0/865852313 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:45.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f5bb3dcd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5bac15e4d0 0x7f5bac0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f5bb3dcd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bac0cfd20 0x7f5bac0d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f5bb3dcd640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bac15e8b0 con 0x7f5bac0d1060 2024-04-02T13:20:45.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f5bb3dcd640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f5bac0b3a90 con 0x7f5bac15e4d0 2024-04-02T13:20:45.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f5bb3dcd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5bac0b4610 con 0x7f5bac0cfd20 2024-04-02T13:20:45.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7ff3ae7fc640 1 --2- 172.21.15.44:0/865852313 >> v2:172.21.15.44:3300/0 conn(0x7ff3b00ccdf0 0x7ff3b006d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f5bb1b69640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5bac15e4d0 0x7f5bac0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f5bb1b69640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5bac15e4d0 0x7f5bac0cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54656/0 (socket says 172.21.15.44:54656) 2024-04-02T13:20:45.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f5bb1b69640 1 -- 172.21.15.44:0/3146961106 learned_addr learned my addr 172.21.15.44:0/3146961106 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 <== mon.1 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 646794397 0 0) 0x7f4e94003520 con 0x7f4eac0aedd0 2024-04-02T13:20:45.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4eac0cd670 msgr2=0x7f4eac0cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 --2- 172.21.15.44:0/2005080665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4eac0cd670 0x7f4eac0cdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 >> v2:172.21.15.69:3300/0 conn(0x7f4eac0ca0f0 msgr2=0x7f4eac0ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 --2- 172.21.15.44:0/2005080665 >> v2:172.21.15.69:3300/0 conn(0x7f4eac0ca0f0 0x7f4eac0ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eac0ced70 con 0x7f4eac0aedd0 2024-04-02T13:20:45.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 <== mon.1 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f4e9c004df0 con 0x7f4eac0aedd0 2024-04-02T13:20:45.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 >> v1:172.21.15.69:6789/0 conn(0x7f4eac0aedd0 legacy=0x7f4eac0cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 --2- 172.21.15.44:0/2005080665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e940039b0 0x7f4e94003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 --2- 172.21.15.44:0/2005080665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94004800 0x7f4e94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 --2- 172.21.15.44:0/2005080665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e940099b0 0x7f4e940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac0ce7b0 con 0x7f4e940099b0 2024-04-02T13:20:45.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac0cea90 con 0x7f4e940039b0 2024-04-02T13:20:45.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f64c5e06640 1 Processor -- start 2024-04-02T13:20:45.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fe816964640 1 -- 172.21.15.44:0/1435200180 >> v2:172.21.15.44:3300/0 conn(0x7fe8100d0c00 msgr2=0x7fe8100ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4e940147a0 con 0x7f4e94004800 2024-04-02T13:20:45.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f4eaa59c640 1 --2- 172.21.15.44:0/2005080665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e940039b0 0x7f4e94003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.562+0000 7fdda5aac640 1 -- 172.21.15.44:0/4049199382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd98038110 msgr2=0x7fdd980385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7fdda5aac640 1 --2- 172.21.15.44:0/4049199382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd98038110 0x7fdd980385a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7fdda5aac640 1 -- 172.21.15.44:0/4049199382 >> v2:172.21.15.44:3300/0 conn(0x7fdd980ead00 msgr2=0x7fdd9804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7fdda5aac640 1 --2- 172.21.15.44:0/4049199382 >> v2:172.21.15.44:3300/0 conn(0x7fdd980ead00 0x7fdd9804fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7fdda5aac640 1 -- 172.21.15.44:0/4049199382 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd98053ce0 con 0x7fdd9804ffd0 2024-04-02T13:20:45.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7fe816964640 1 --2- 172.21.15.44:0/1435200180 >> v2:172.21.15.44:3300/0 conn(0x7fe8100d0c00 0x7fe8100ccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7fe816964640 1 -- 172.21.15.44:0/1435200180 >> v1:172.21.15.69:6789/0 conn(0x7fe81015c1b0 legacy=0x7fe8100d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7fe816964640 1 -- 172.21.15.44:0/1435200180 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8100ce360 con 0x7fe8100cd290 2024-04-02T13:20:45.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1656602823 0 0) 0x7ff3b00b3eb0 con 0x7ff3b0162800 2024-04-02T13:20:45.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff398003520 con 0x7ff3b0162800 2024-04-02T13:20:45.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f4ea9d9b640 1 --2- 172.21.15.44:0/2005080665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94004800 0x7f4e94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f5bb1b69640 1 -- 172.21.15.44:0/3146961106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bac0cfd20 msgr2=0x7f5bac0d0b20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T13:20:45.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f5bb1b69640 1 --2- 172.21.15.44:0/3146961106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bac0cfd20 0x7f5bac0d0b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f5bb1b69640 1 -- 172.21.15.44:0/3146961106 >> v1:172.21.15.69:6789/0 conn(0x7f5bac0d1060 legacy=0x7f5bac0d1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7fe815962640 1 --2- 172.21.15.44:0/1435200180 >> v2:172.21.15.44:3300/0 conn(0x7fe8100d0c00 0x7fe8100ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:45.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f89d97a4640 1 Processor -- start 2024-04-02T13:20:45.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f89d97a4640 1 -- start start 2024-04-02T13:20:45.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f5bb1b69640 1 -- 172.21.15.44:0/3146961106 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bac0d2fb0 con 0x7f5bac15e4d0 2024-04-02T13:20:45.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f64c5e06640 1 -- start start 2024-04-02T13:20:45.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f4ea9d9b640 1 -- 172.21.15.44:0/2005080665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e940039b0 msgr2=0x7f4e94003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f4ea9d9b640 1 --2- 172.21.15.44:0/2005080665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e940039b0 0x7f4e94003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f4ea9d9b640 1 -- 172.21.15.44:0/2005080665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e940099b0 msgr2=0x7f4e940051c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T13:20:45.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f4ea9d9b640 1 --2- 172.21.15.44:0/2005080665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e940099b0 0x7f4e940051c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7f4ea9d9b640 1 -- 172.21.15.44:0/2005080665 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e94003520 con 0x7f4e94004800 2024-04-02T13:20:45.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7ff3af7fe640 1 --2- 172.21.15.44:0/865852313 >> v2:172.21.15.69:3300/0 conn(0x7ff3b0166cb0 0x7ff3b00b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.563+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 <== mon.1 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 557804214 0 0) 0x7ff398003520 con 0x7ff3b0162800 2024-04-02T13:20:45.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 >> v2:172.21.15.69:3300/0 conn(0x7ff3b0166cb0 msgr2=0x7ff3b00b4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3adffb640 1 --2- 172.21.15.44:0/865852313 >> v2:172.21.15.69:3300/0 conn(0x7ff3b0166cb0 0x7ff3b00b4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 >> v2:172.21.15.44:3300/0 conn(0x7ff3b00ccdf0 msgr2=0x7ff3b006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3adffb640 1 --2- 172.21.15.44:0/865852313 >> v2:172.21.15.44:3300/0 conn(0x7ff3b00ccdf0 0x7ff3b006d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b00b4150 con 0x7ff3b0162800 2024-04-02T13:20:45.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 <== mon.1 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7ff3a4001690 con 0x7ff3b0162800 2024-04-02T13:20:45.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 >> v1:172.21.15.69:6789/0 conn(0x7ff3b0162800 legacy=0x7ff3b00cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3adffb640 1 --2- 172.21.15.44:0/865852313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff398003a00 0x7ff398003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3adffb640 1 --2- 172.21.15.44:0/865852313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff398004800 0x7ff398004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3adffb640 1 --2- 172.21.15.44:0/865852313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3980051c0 0x7ff39800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b0162be0 con 0x7ff398004800 2024-04-02T13:20:45.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b00b47a0 con 0x7ff3980051c0 2024-04-02T13:20:45.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff39800c060 con 0x7ff398003a00 2024-04-02T13:20:45.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3aeffd640 1 --2- 172.21.15.44:0/865852313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff398003a00 0x7ff398003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7f64c5e06640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f64c006eca0 0x7f64c00cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7f64c5e06640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f64c0166cf0 0x7f64c00d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7f64c5e06640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c00cec70 con 0x7f64c0162b60 2024-04-02T13:20:45.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7f64c5e06640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f64c0163a20 con 0x7f64c006eca0 2024-04-02T13:20:45.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7f64c5e06640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f64c0163740 con 0x7f64c0166cf0 2024-04-02T13:20:45.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7fdda5aac640 1 --2- 172.21.15.44:0/4049199382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd9804ffd0 0x7fdd98037bd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdda002d900 tx=0x7fdda00a6d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff63567f27fe1775 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3ae7fc640 1 --2- 172.21.15.44:0/865852313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff398004800 0x7ff398004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7fef45b69640 1 --2- 172.21.15.44:0/3098438229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef400b5260 0x7fef4006f150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fef4006c100 tx=0x7fef3c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9081b9f827e5bc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7f4ea9d9b640 1 --2- 172.21.15.44:0/2005080665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94004800 0x7f4e94004c80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ea40094b0 tx=0x7f4e94005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c656c5c3bc4b90f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7f64bf7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f64c006eca0 0x7f64c00cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7f64bf7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f64c006eca0 0x7f64c00cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54670/0 (socket says 172.21.15.44:54670) 2024-04-02T13:20:45.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7f64bf7fe640 1 -- 172.21.15.44:0/2070653831 learned_addr learned my addr 172.21.15.44:0/2070653831 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3af7fe640 1 --2- 172.21.15.44:0/865852313 >> v2:172.21.15.69:3300/0 conn(0x7ff3b0166cb0 0x7ff3b00b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:45.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7f64beffd640 1 --2- 172.21.15.44:0/2070653831 >> v2:172.21.15.69:3300/0 conn(0x7f64c0166cf0 0x7f64c00d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4038070 con 0x7f4e94004800 2024-04-02T13:20:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3ae7fc640 1 -- 172.21.15.44:0/865852313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff398003a00 msgr2=0x7ff398003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3ae7fc640 1 --2- 172.21.15.44:0/865852313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff398003a00 0x7ff398003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3ae7fc640 1 -- 172.21.15.44:0/865852313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3980051c0 msgr2=0x7ff39800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T13:20:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3ae7fc640 1 --2- 172.21.15.44:0/865852313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3980051c0 0x7ff39800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3ae7fc640 1 -- 172.21.15.44:0/865852313 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff398003520 con 0x7ff398004800 2024-04-02T13:20:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7ff3af7fe640 1 -- 172.21.15.44:0/865852313 reap_dead start 2024-04-02T13:20:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d97a4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89d4166190 0x7f89d40cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.564+0000 7fdda52ab640 1 -- 172.21.15.44:0/4049199382 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda00b4070 con 0x7fdd9804ffd0 2024-04-02T13:20:45.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7fdda52ab640 1 -- 172.21.15.44:0/4049199382 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdda00a7ba0 con 0x7fdd9804ffd0 2024-04-02T13:20:45.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d97a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d4166610 0x7f89d40cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d97a4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89d40b3ba0 con 0x7f89d414bd40 2024-04-02T13:20:45.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ea4033600 con 0x7f4e94004800 2024-04-02T13:20:45.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f4ea959a640 1 -- 172.21.15.44:0/2005080665 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea400bd80 con 0x7f4e94004800 2024-04-02T13:20:45.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d97a4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f89d4167130 con 0x7f89d4166190 2024-04-02T13:20:45.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d97a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f89d4166b70 con 0x7f89d4166610 2024-04-02T13:20:45.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d3fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f89d414bd40 0x7f89d40b3490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57398/0 (socket says 172.21.15.44:57398) 2024-04-02T13:20:45.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d3fff640 1 -- 172.21.15.44:0/3862677972 learned_addr learned my addr 172.21.15.44:0/3862677972 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:45.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d37fe640 1 --2- 172.21.15.44:0/3862677972 >> v2:172.21.15.69:3300/0 conn(0x7f89d4166190 0x7f89d40cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d8fa3640 1 --2- 172.21.15.44:0/3862677972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d4166610 0x7f89d40cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f64bf7fe640 1 -- 172.21.15.44:0/2070653831 >> v2:172.21.15.69:3300/0 conn(0x7f64c0166cf0 msgr2=0x7f64c00d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f64bf7fe640 1 --2- 172.21.15.44:0/2070653831 >> v2:172.21.15.69:3300/0 conn(0x7f64c0166cf0 0x7f64c00d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f64bf7fe640 1 -- 172.21.15.44:0/2070653831 >> v1:172.21.15.44:6789/0 conn(0x7f64c0162b60 legacy=0x7f64c00cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f64bf7fe640 1 -- 172.21.15.44:0/2070653831 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c0166210 con 0x7f64c006eca0 2024-04-02T13:20:45.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d2ffd640 1 -- 172.21.15.44:0/3862677972 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592966549 0 0) 0x7f89d40b3ba0 con 0x7f89d414bd40 2024-04-02T13:20:45.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d2ffd640 1 -- 172.21.15.44:0/3862677972 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89bc003520 con 0x7f89d414bd40 2024-04-02T13:20:45.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d37fe640 1 -- 172.21.15.44:0/3862677972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d4166610 msgr2=0x7f89d40cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7fef44b67640 1 -- 172.21.15.44:0/3098438229 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c035020 con 0x7fef400b5260 2024-04-02T13:20:45.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7fef44b67640 1 -- 172.21.15.44:0/3098438229 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef3c031640 con 0x7fef400b5260 2024-04-02T13:20:45.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7fef44b67640 1 -- 172.21.15.44:0/3098438229 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c02d040 con 0x7fef400b5260 2024-04-02T13:20:45.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7fe816964640 1 --2- 172.21.15.44:0/1435200180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8100cd290 0x7fe8100c5e40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe808009d90 tx=0x7fe80802be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58b576ca003f15fb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7fdda52ab640 1 -- 172.21.15.44:0/4049199382 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda00af440 con 0x7fdd9804ffd0 2024-04-02T13:20:45.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7fdda72af640 1 -- 172.21.15.44:0/4049199382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd9804ffd0 msgr2=0x7fdd98037bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7fdda72af640 1 --2- 172.21.15.44:0/4049199382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd9804ffd0 0x7fdd98037bd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdda002d900 tx=0x7fdda00a6d30 comp rx=0 tx=0).stop 2024-04-02T13:20:45.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d37fe640 1 --2- 172.21.15.44:0/3862677972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d4166610 0x7f89d40cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7fe815161640 1 -- 172.21.15.44:0/1435200180 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe808039070 con 0x7fe8100cd290 2024-04-02T13:20:45.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7fe815161640 1 -- 172.21.15.44:0/1435200180 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe80802cbd0 con 0x7fe8100cd290 2024-04-02T13:20:45.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7fe815161640 1 -- 172.21.15.44:0/1435200180 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe808034440 con 0x7fe8100cd290 2024-04-02T13:20:45.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7f4eb0915640 1 -- 172.21.15.44:0/2005080665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94004800 msgr2=0x7f4e94004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7f4eb0915640 1 --2- 172.21.15.44:0/2005080665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94004800 0x7f4e94004c80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ea40094b0 tx=0x7f4e94005830 comp rx=0 tx=0).stop 2024-04-02T13:20:45.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7f4eaad9d640 1 -- 172.21.15.44:0/2005080665 reap_dead start 2024-04-02T13:20:45.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d37fe640 1 -- 172.21.15.44:0/3862677972 >> v1:172.21.15.44:6789/0 conn(0x7f89d414bd40 legacy=0x7f89d40b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.565+0000 7f89d37fe640 1 -- 172.21.15.44:0/3862677972 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d4166e50 con 0x7f89d4166190 2024-04-02T13:20:45.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7f4eb0915640 1 -- 172.21.15.44:0/2005080665 shutdown_connections 2024-04-02T13:20:45.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7fdda72af640 1 -- 172.21.15.44:0/4049199382 shutdown_connections 2024-04-02T13:20:45.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7fdda72af640 1 --2- 172.21.15.44:0/4049199382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd98038110 0x7fdd980385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7fdda72af640 1 --2- 172.21.15.44:0/4049199382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd9804ffd0 0x7fdd98037bd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7fdda72af640 1 --2- 172.21.15.44:0/4049199382 >> v2:172.21.15.44:3300/0 conn(0x7fdd980ead00 0x7fdd9804fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7fdda72af640 1 -- 172.21.15.44:0/4049199382 >> 172.21.15.44:0/4049199382 conn(0x7fdd98009fb0 msgr2=0x7fdd9800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7ff3ae7fc640 1 --2- 172.21.15.44:0/865852313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff398004800 0x7ff398004c80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009fa0 tx=0x7ff398005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fda77c7c08e2d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7f4eb0915640 1 -- 172.21.15.44:0/2005080665 >> 172.21.15.44:0/2005080665 conn(0x7f4eac0cfaa0 msgr2=0x7f4eac0cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8039070 con 0x7ff398004800 2024-04-02T13:20:45.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7f4eb0915640 1 -- 172.21.15.44:0/2005080665 shutdown_connections 2024-04-02T13:20:45.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7fdda72af640 1 -- 172.21.15.44:0/4049199382 shutdown_connections 2024-04-02T13:20:45.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7fdda72af640 1 -- 172.21.15.44:0/4049199382 wait complete. 2024-04-02T13:20:45.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7f5bb1b69640 1 --2- 172.21.15.44:0/3146961106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bac15e4d0 0x7f5bac0cf7e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009870 tx=0x7f5b9c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65115b7b98d6541d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7f4eb0915640 1 -- 172.21.15.44:0/2005080665 wait complete. 2024-04-02T13:20:45.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7fef47dcd640 1 -- 172.21.15.44:0/3098438229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef400b5260 msgr2=0x7fef4006f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7fef47dcd640 1 --2- 172.21.15.44:0/3098438229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef400b5260 0x7fef4006f150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fef4006c100 tx=0x7fef3c02b8f0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a802cc10 con 0x7ff398004800 2024-04-02T13:20:45.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7f89d37fe640 1 --2- 172.21.15.44:0/3862677972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d4166190 0x7f89d40cfa60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f89c80097a0 tx=0x7f89c802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a2cf23450f0a63e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7f64bf7fe640 1 --2- 172.21.15.44:0/2070653831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64c006eca0 0x7f64c00cfdb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f64c0160630 tx=0x7f64b802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f7884a064e9bde server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.566+0000 7ff3adffb640 1 -- 172.21.15.44:0/865852313 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8009bf0 con 0x7ff398004800 2024-04-02T13:20:45.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7fef47dcd640 1 -- 172.21.15.44:0/3098438229 shutdown_connections 2024-04-02T13:20:45.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7fef47dcd640 1 --2- 172.21.15.44:0/3098438229 >> v2:172.21.15.44:3300/0 conn(0x7fef400b5640 0x7fef401635f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7fef47dcd640 1 --2- 172.21.15.44:0/3098438229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef400b5260 0x7fef4006f150 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7fef47dcd640 1 -- 172.21.15.44:0/3098438229 >> 172.21.15.44:0/3098438229 conn(0x7fef40074dc0 msgr2=0x7fef4006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7fe8183c7640 1 -- 172.21.15.44:0/1435200180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8100cd290 msgr2=0x7fe8100c5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7f89d2ffd640 1 -- 172.21.15.44:0/3862677972 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8031780 con 0x7f89d4166190 2024-04-02T13:20:45.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7f89d2ffd640 1 -- 172.21.15.44:0/3862677972 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89c8037070 con 0x7f89d4166190 2024-04-02T13:20:45.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7f64be7fc640 1 -- 172.21.15.44:0/2070653831 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b802c9f0 con 0x7f64c006eca0 2024-04-02T13:20:45.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7f64be7fc640 1 -- 172.21.15.44:0/2070653831 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64b802cb90 con 0x7f64c006eca0 2024-04-02T13:20:45.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7f64be7fc640 1 -- 172.21.15.44:0/2070653831 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b80343f0 con 0x7f64c006eca0 2024-04-02T13:20:45.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7fe8183c7640 1 --2- 172.21.15.44:0/1435200180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8100cd290 0x7fe8100c5e40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe808009d90 tx=0x7fe80802be20 comp rx=0 tx=0).stop 2024-04-02T13:20:45.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7fef47dcd640 1 -- 172.21.15.44:0/3098438229 shutdown_connections 2024-04-02T13:20:45.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7fef47dcd640 1 -- 172.21.15.44:0/3098438229 wait complete. 2024-04-02T13:20:45.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7f89d2ffd640 1 -- 172.21.15.44:0/3862677972 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8031b40 con 0x7f89d4166190 2024-04-02T13:20:45.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7fe8183c7640 1 -- 172.21.15.44:0/1435200180 shutdown_connections 2024-04-02T13:20:45.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7fe8183c7640 1 --2- 172.21.15.44:0/1435200180 >> v2:172.21.15.44:3300/0 conn(0x7fe8100d0c00 0x7fe8100ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7fe8183c7640 1 --2- 172.21.15.44:0/1435200180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8100cd290 0x7fe8100c5e40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7f5bb0b67640 1 -- 172.21.15.44:0/3146961106 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c035020 con 0x7f5bac15e4d0 2024-04-02T13:20:45.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7f5bb0b67640 1 -- 172.21.15.44:0/3146961106 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b9c031870 con 0x7f5bac15e4d0 2024-04-02T13:20:45.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7f5bb0b67640 1 -- 172.21.15.44:0/3146961106 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c031c40 con 0x7f5bac15e4d0 2024-04-02T13:20:45.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7fe8183c7640 1 -- 172.21.15.44:0/1435200180 >> 172.21.15.44:0/1435200180 conn(0x7fe81009aa20 msgr2=0x7fe81009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7ff3b592b640 1 -- 172.21.15.44:0/865852313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff398004800 msgr2=0x7ff398004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7ff3b592b640 1 --2- 172.21.15.44:0/865852313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff398004800 0x7ff398004c80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009fa0 tx=0x7ff398005830 comp rx=0 tx=0).stop 2024-04-02T13:20:45.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7ff3b592b640 1 -- 172.21.15.44:0/865852313 shutdown_connections 2024-04-02T13:20:45.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7ff3b592b640 1 --2- 172.21.15.44:0/865852313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff398004800 0x7ff398004c80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.567+0000 7ff3b592b640 1 -- 172.21.15.44:0/865852313 >> 172.21.15.44:0/865852313 conn(0x7ff3b00cfcd0 msgr2=0x7ff3b00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7fe8183c7640 1 -- 172.21.15.44:0/1435200180 shutdown_connections 2024-04-02T13:20:45.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7fdda72af640 1 Processor -- start 2024-04-02T13:20:45.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f4eb0915640 1 Processor -- start 2024-04-02T13:20:45.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f4eb0915640 1 -- start start 2024-04-02T13:20:45.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/3146961106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bac15e4d0 msgr2=0x7f5bac0cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f5bb3dcd640 1 --2- 172.21.15.44:0/3146961106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bac15e4d0 0x7f5bac0cf7e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009870 tx=0x7f5b9c02b910 comp rx=0 tx=0).stop 2024-04-02T13:20:45.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/3146961106 shutdown_connections 2024-04-02T13:20:45.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f5bb3dcd640 1 --2- 172.21.15.44:0/3146961106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bac0cfd20 0x7f5bac0d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f5bb3dcd640 1 --2- 172.21.15.44:0/3146961106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bac15e4d0 0x7f5bac0cf7e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/3146961106 >> 172.21.15.44:0/3146961106 conn(0x7f5bac09d410 msgr2=0x7f5bac0d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7ff3b592b640 1 -- 172.21.15.44:0/865852313 shutdown_connections 2024-04-02T13:20:45.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f89d97a4640 1 -- 172.21.15.44:0/3862677972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d4166190 msgr2=0x7f89d40cfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f89d97a4640 1 --2- 172.21.15.44:0/3862677972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d4166190 0x7f89d40cfa60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f89c80097a0 tx=0x7f89c802b820 comp rx=0 tx=0).stop 2024-04-02T13:20:45.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/3146961106 shutdown_connections 2024-04-02T13:20:45.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f89d97a4640 1 -- 172.21.15.44:0/3862677972 shutdown_connections 2024-04-02T13:20:45.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f89d97a4640 1 --2- 172.21.15.44:0/3862677972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d4166610 0x7f89d40cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f89d97a4640 1 --2- 172.21.15.44:0/3862677972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d4166190 0x7f89d40cfa60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f89d97a4640 1 -- 172.21.15.44:0/3862677972 >> 172.21.15.44:0/3862677972 conn(0x7f89d406b4d0 msgr2=0x7f89d406b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f89d97a4640 1 -- 172.21.15.44:0/3862677972 shutdown_connections 2024-04-02T13:20:45.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f89d97a4640 1 -- 172.21.15.44:0/3862677972 wait complete. 2024-04-02T13:20:45.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7fe8183c7640 1 -- 172.21.15.44:0/1435200180 wait complete. 2024-04-02T13:20:45.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f64c5e06640 1 -- 172.21.15.44:0/2070653831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64c006eca0 msgr2=0x7f64c00cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f64c5e06640 1 --2- 172.21.15.44:0/2070653831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64c006eca0 0x7f64c00cfdb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f64c0160630 tx=0x7f64b802be50 comp rx=0 tx=0).stop 2024-04-02T13:20:45.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7fef47dcd640 1 Processor -- start 2024-04-02T13:20:45.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f64c5e06640 1 -- 172.21.15.44:0/2070653831 shutdown_connections 2024-04-02T13:20:45.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f64c5e06640 1 --2- 172.21.15.44:0/2070653831 >> v2:172.21.15.69:3300/0 conn(0x7f64c0166cf0 0x7f64c00d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f64c5e06640 1 --2- 172.21.15.44:0/2070653831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64c006eca0 0x7f64c00cfdb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f64c5e06640 1 -- 172.21.15.44:0/2070653831 >> 172.21.15.44:0/2070653831 conn(0x7f64c006b6b0 msgr2=0x7f64c006bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7fef47dcd640 1 -- start start 2024-04-02T13:20:45.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7ff3b592b640 1 -- 172.21.15.44:0/865852313 wait complete. 2024-04-02T13:20:45.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/3146961106 wait complete. 2024-04-02T13:20:45.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f4eb0915640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94004800 0x7f4eac0c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f4eb0915640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eac0c6b70 0x7f4eac0c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f4eb0915640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eac1f3510 0x7f4eac1f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f4eb0915640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac0712a0 con 0x7f4eac0c6b70 2024-04-02T13:20:45.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.568+0000 7fdda72af640 1 -- start start 2024-04-02T13:20:45.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7fdda72af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd98038110 0x7fdd980dc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7fdda72af640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd9804ffd0 0x7fdd980dcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7fdda72af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd980ead00 0x7fdd980f0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7fdda72af640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98054940 con 0x7fdd98038110 2024-04-02T13:20:45.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7fdda72af640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdd980547c0 con 0x7fdd9804ffd0 2024-04-02T13:20:45.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7fdda72af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdd9804f550 con 0x7fdd980ead00 2024-04-02T13:20:45.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f4eb0915640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac0af1b0 con 0x7f4eac1f3510 2024-04-02T13:20:45.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f4eb0915640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4eac0cc240 con 0x7f4e94004800 2024-04-02T13:20:45.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f4eaa59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94004800 0x7f4eac0c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f4eaa59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94004800 0x7f4eac0c8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:41948/0 (socket says 172.21.15.44:41948) 2024-04-02T13:20:45.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f4eaa59c640 1 -- 172.21.15.44:0/796124911 learned_addr learned my addr 172.21.15.44:0/796124911 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f4ea9d9b640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eac0c6b70 0x7f4eac0c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.569+0000 7f4eaad9d640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eac1f3510 0x7f4eac1f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7f4eaa59c640 1 -- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eac1f3510 msgr2=0x7f4eac1f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7f5bb3dcd640 1 Processor -- start 2024-04-02T13:20:45.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7f4eaa59c640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eac1f3510 0x7f4eac1f39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7f4eaa59c640 1 -- 172.21.15.44:0/796124911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eac0c6b70 msgr2=0x7f4eac0c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7f4eaa59c640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eac0c6b70 0x7f4eac0c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7f89d97a4640 1 Processor -- start 2024-04-02T13:20:45.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fe8183c7640 1 Processor -- start 2024-04-02T13:20:45.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7f4eaa59c640 1 -- 172.21.15.44:0/796124911 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eac0c78a0 con 0x7f4e94004800 2024-04-02T13:20:45.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7f4ea9d9b640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eac0c6b70 0x7f4eac0c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:45.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fe8183c7640 1 -- start start 2024-04-02T13:20:45.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fef47dcd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef40071a50 0x7fef4015c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fef47dcd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef400b5260 0x7fef4012fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fef47dcd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef400b5640 0x7fef40143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fef47dcd640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fef40166af0 con 0x7fef400b5640 2024-04-02T13:20:45.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fef47dcd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fef40166810 con 0x7fef400b5260 2024-04-02T13:20:45.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fef47dcd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fef40166dd0 con 0x7fef40071a50 2024-04-02T13:20:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fdda5aac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd9804ffd0 0x7fdd980dcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fdda62ad640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd98038110 0x7fdd980dc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fdda5aac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd9804ffd0 0x7fdd980dcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34172/0 (socket says 172.21.15.44:34172) 2024-04-02T13:20:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fdda5aac640 1 -- 172.21.15.44:0/3437639349 learned_addr learned my addr 172.21.15.44:0/3437639349 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fef45b69640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef40071a50 0x7fef4015c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fef45b69640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef40071a50 0x7fef4015c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:41958/0 (socket says 172.21.15.44:41958) 2024-04-02T13:20:45.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fef45b69640 1 -- 172.21.15.44:0/2272332301 learned_addr learned my addr 172.21.15.44:0/2272332301 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7fef4636a640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef400b5640 0x7fef40143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fef45368640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef400b5260 0x7fef4012fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fdda62ad640 1 -- 172.21.15.44:0/3437639349 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd980ead00 msgr2=0x7fdd980f0a10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T13:20:45.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fdda62ad640 1 --2- 172.21.15.44:0/3437639349 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd980ead00 0x7fdd980f0a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fdda62ad640 1 -- 172.21.15.44:0/3437639349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd9804ffd0 msgr2=0x7fdd980dcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7ff3b592b640 1 Processor -- start 2024-04-02T13:20:45.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7f64c5e06640 1 -- 172.21.15.44:0/2070653831 shutdown_connections 2024-04-02T13:20:45.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fdda62ad640 1 --2- 172.21.15.44:0/3437639349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd9804ffd0 0x7fdd980dcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fdda62ad640 1 -- 172.21.15.44:0/3437639349 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd980f1780 con 0x7fdd98038110 2024-04-02T13:20:45.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fef45b69640 1 -- 172.21.15.44:0/2272332301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef400b5260 msgr2=0x7fef4012fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fef45b69640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef400b5260 0x7fef4012fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fef45b69640 1 -- 172.21.15.44:0/2272332301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef400b5640 msgr2=0x7fef40143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fef45b69640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef400b5640 0x7fef40143ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7ff3b592b640 1 -- start start 2024-04-02T13:20:45.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fef45b69640 1 -- 172.21.15.44:0/2272332301 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef401306b0 con 0x7fef40071a50 2024-04-02T13:20:45.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fef4636a640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef400b5640 0x7fef40143ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:45.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7f5bb3dcd640 1 -- start start 2024-04-02T13:20:45.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fe8183c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8100cd290 0x7fe8101fb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fe8183c7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8100d0c00 0x7fe8101fc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fe8183c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81015c1b0 0x7fe8100b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fdda5aac640 1 --2- 172.21.15.44:0/3437639349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd9804ffd0 0x7fdd980dcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:45.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7f5bb3dcd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bac0cfd20 0x7f5bac0cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7f5bb3dcd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bac0d1060 0x7f5bac15c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7f5bb3dcd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bac15e4d0 0x7f5bac15c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fe8183c7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe8100c6500 con 0x7fe8100d0c00 2024-04-02T13:20:45.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fe8183c7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe8100c6380 con 0x7fe8100cd290 2024-04-02T13:20:45.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fe8183c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe8100c6680 con 0x7fe81015c1b0 2024-04-02T13:20:45.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.570+0000 7f89d97a4640 1 -- start start 2024-04-02T13:20:45.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7f89d97a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d414bd40 0x7f89d4161c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fef45b69640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef40071a50 0x7fef4015c7f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fef3c02bf00 tx=0x7fef3c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fe816163640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8100cd290 0x7fe8101fb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fe816163640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8100cd290 0x7fe8101fb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34180/0 (socket says 172.21.15.44:34180) 2024-04-02T13:20:45.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.571+0000 7fe816163640 1 -- 172.21.15.44:0/1862333177 learned_addr learned my addr 172.21.15.44:0/1862333177 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fe816964640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81015c1b0 0x7fe8100b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fe815962640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8100d0c00 0x7fe8101fc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f89d97a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d4166190 0x7f89d40b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fdda62ad640 1 --2- 172.21.15.44:0/3437639349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd98038110 0x7fdd980dc740 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000e850 tx=0x7fdd9000ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f5bb1b69640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bac0cfd20 0x7f5bac0cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f5bb1b69640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bac0cfd20 0x7f5bac0cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34182/0 (socket says 172.21.15.44:34182) 2024-04-02T13:20:45.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f5bb1b69640 1 -- 172.21.15.44:0/2362851089 learned_addr learned my addr 172.21.15.44:0/2362851089 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f5bb236a640 1 --2- 172.21.15.44:0/2362851089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bac15e4d0 0x7f5bac15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fef367fc640 1 -- 172.21.15.44:0/2272332301 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c040070 con 0x7fef40071a50 2024-04-02T13:20:45.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fef367fc640 1 -- 172.21.15.44:0/2272332301 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef3c039b90 con 0x7fef40071a50 2024-04-02T13:20:45.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fef367fc640 1 -- 172.21.15.44:0/2272332301 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c035040 con 0x7fef40071a50 2024-04-02T13:20:45.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f89d3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d414bd40 0x7f89d4161c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f89d3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d414bd40 0x7f89d4161c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34196/0 (socket says 172.21.15.44:34196) 2024-04-02T13:20:45.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f89d37fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d4166190 0x7f89d40b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac0b42a0 con 0x7f5bac0d1060 2024-04-02T13:20:45.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac0b3fc0 con 0x7f5bac0cfd20 2024-04-02T13:20:45.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5bac0b0d00 con 0x7f5bac15e4d0 2024-04-02T13:20:45.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fef47dcd640 1 -- 172.21.15.44:0/2272332301 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef40130990 con 0x7fef40071a50 2024-04-02T13:20:45.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f89d37fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d4166190 0x7f89d40b1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:41972/0 (socket says 172.21.15.44:41972) 2024-04-02T13:20:45.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f89d97a4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d4166610 0x7f89d40b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f89d97a4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f89d40ced20 con 0x7f89d4166610 2024-04-02T13:20:45.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f89d97a4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89d414c120 con 0x7f89d414bd40 2024-04-02T13:20:45.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f89d97a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f89d40ceea0 con 0x7f89d4166190 2024-04-02T13:20:45.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fef47dcd640 1 -- 172.21.15.44:0/2272332301 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fef40130f50 con 0x7fef40071a50 2024-04-02T13:20:45.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fe816163640 1 -- 172.21.15.44:0/1862333177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81015c1b0 msgr2=0x7fe8100b7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fe816163640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81015c1b0 0x7fe8100b7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fe816163640 1 -- 172.21.15.44:0/1862333177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8100d0c00 msgr2=0x7fe8101fc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fe816163640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8100d0c00 0x7fe8101fc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fe816163640 1 -- 172.21.15.44:0/1862333177 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8100b5770 con 0x7fe8100cd290 2024-04-02T13:20:45.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fdd96ffd640 1 -- 172.21.15.44:0/3437639349 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90017070 con 0x7fdd98038110 2024-04-02T13:20:45.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fdd96ffd640 1 -- 172.21.15.44:0/3437639349 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd9000c940 con 0x7fdd98038110 2024-04-02T13:20:45.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fdd96ffd640 1 -- 172.21.15.44:0/3437639349 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9000cd10 con 0x7fdd98038110 2024-04-02T13:20:45.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fdda72af640 1 -- 172.21.15.44:0/3437639349 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd980f1a60 con 0x7fdd98038110 2024-04-02T13:20:45.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f89d3fff640 1 -- 172.21.15.44:0/3272496562 learned_addr learned my addr 172.21.15.44:0/3272496562 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fe816163640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8100cd290 0x7fe8101fb460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe80000c410 tx=0x7fe80000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7fdda72af640 1 -- 172.21.15.44:0/3437639349 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd98035b40 con 0x7fdd98038110 2024-04-02T13:20:45.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f89d8fa3640 1 --2- 172.21.15.44:0/3272496562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d4166610 0x7f89d40b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7f89d8fa3640 1 -- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d4166190 msgr2=0x7f89d40b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7f89d8fa3640 1 --2- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d4166190 0x7f89d40b1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7fef47dcd640 1 -- 172.21.15.44:0/2272332301 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fef40131330 con 0x7fef40071a50 2024-04-02T13:20:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7f5bb236a640 1 -- 172.21.15.44:0/2362851089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bac0cfd20 msgr2=0x7f5bac0cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7f89d8fa3640 1 -- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d414bd40 msgr2=0x7f89d4161c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7f89d8fa3640 1 --2- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d414bd40 0x7f89d4161c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7f89d8fa3640 1 -- 172.21.15.44:0/3272496562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d41457d0 con 0x7f89d4166610 2024-04-02T13:20:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7f89d8fa3640 1 --2- 172.21.15.44:0/3272496562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d4166610 0x7f89d40b1d80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f89c400d530 tx=0x7f89c400d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f4eaa59c640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94004800 0x7f4eac0c8ab0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0109d0 tx=0x7f4e9c010e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7fe806ffd640 1 -- 172.21.15.44:0/1862333177 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe800017070 con 0x7fe8100cd290 2024-04-02T13:20:45.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7fe806ffd640 1 -- 172.21.15.44:0/1862333177 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe800005ce0 con 0x7fe8100cd290 2024-04-02T13:20:45.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7fe806ffd640 1 -- 172.21.15.44:0/1862333177 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8000072f0 con 0x7fe8100cd290 2024-04-02T13:20:45.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7f5bb236a640 1 --2- 172.21.15.44:0/2362851089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bac0cfd20 0x7f5bac0cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7f5bb236a640 1 -- 172.21.15.44:0/2362851089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bac0d1060 msgr2=0x7f5bac15c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7f5bb236a640 1 --2- 172.21.15.44:0/2362851089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bac0d1060 0x7f5bac15c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7f5bb236a640 1 -- 172.21.15.44:0/2362851089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bac0cd6a0 con 0x7f5bac15e4d0 2024-04-02T13:20:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.572+0000 7f64c5e06640 1 -- 172.21.15.44:0/2070653831 wait complete. 2024-04-02T13:20:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7f5bb236a640 1 --2- 172.21.15.44:0/2362851089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bac15e4d0 0x7f5bac15c800 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400c7b0 tx=0x7f5ba400cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7f5ba27fc640 1 -- 172.21.15.44:0/2362851089 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4016020 con 0x7f5bac15e4d0 2024-04-02T13:20:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7f64c5e06640 1 Processor -- start 2024-04-02T13:20:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7f5ba27fc640 1 -- 172.21.15.44:0/2362851089 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba400aa50 con 0x7f5bac15e4d0 2024-04-02T13:20:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bac0cce00 con 0x7f5bac15e4d0 2024-04-02T13:20:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bac0cd3c0 con 0x7f5bac15e4d0 2024-04-02T13:20:45.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7fe8183c7640 1 -- 172.21.15.44:0/1862333177 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8100ca330 con 0x7fe8100cd290 2024-04-02T13:20:45.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7fe8183c7640 1 -- 172.21.15.44:0/1862333177 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8100ca8f0 con 0x7fe8100cd290 2024-04-02T13:20:45.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7fdda72af640 1 -- 172.21.15.44:0/3437639349 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdd980052c0 con 0x7fdd98038110 2024-04-02T13:20:45.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7ff3b592b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff398004800 0x7ff3b014bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7ff3b592b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3b009c070 0x7ff3b0154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7ff3b592b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b0154a50 0x7ff3b00ad040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7ff3b592b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b006dae0 con 0x7ff3b009c070 2024-04-02T13:20:45.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7ff3b592b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b0167190 con 0x7ff398004800 2024-04-02T13:20:45.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.573+0000 7ff3b592b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b00b4c30 con 0x7ff3b0154a50 2024-04-02T13:20:45.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7ff3aeffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff398004800 0x7ff3b014bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7ff3aeffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff398004800 0x7ff3b014bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34212/0 (socket says 172.21.15.44:34212) 2024-04-02T13:20:45.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7ff3aeffd640 1 -- 172.21.15.44:0/342852530 learned_addr learned my addr 172.21.15.44:0/342852530 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7ff3aeffd640 1 -- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b0154a50 msgr2=0x7ff3b00ad040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T13:20:45.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7ff3aeffd640 1 --2- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b0154a50 0x7ff3b00ad040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7ff3aeffd640 1 -- 172.21.15.44:0/342852530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3b009c070 msgr2=0x7ff3b0154510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T13:20:45.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7ff3aeffd640 1 --2- 172.21.15.44:0/342852530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3b009c070 0x7ff3b0154510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7ff3aeffd640 1 -- 172.21.15.44:0/342852530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b0153260 con 0x7ff398004800 2024-04-02T13:20:45.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7f89d0ff9640 1 -- 172.21.15.44:0/3272496562 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c40125f0 con 0x7f89d4166610 2024-04-02T13:20:45.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7f89d0ff9640 1 -- 172.21.15.44:0/3272496562 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89c4019070 con 0x7f89d4166610 2024-04-02T13:20:45.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7f89d0ff9640 1 -- 172.21.15.44:0/3272496562 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c40129b0 con 0x7f89d4166610 2024-04-02T13:20:45.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7f89d97a4640 1 -- 172.21.15.44:0/3272496562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d4145ab0 con 0x7f89d4166610 2024-04-02T13:20:45.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7f89d97a4640 1 -- 172.21.15.44:0/3272496562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d4146070 con 0x7f89d4166610 2024-04-02T13:20:45.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.574+0000 7f64c5e06640 1 -- start start 2024-04-02T13:20:45.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7fef367fc640 1 -- 172.21.15.44:0/2272332301 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fef3c038020 con 0x7fef40071a50 2024-04-02T13:20:45.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7f4e9affd640 1 -- 172.21.15.44:0/796124911 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c007070 con 0x7f4e94004800 2024-04-02T13:20:45.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7f4eb0915640 1 -- 172.21.15.44:0/796124911 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eac0c7b80 con 0x7f4e94004800 2024-04-02T13:20:45.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7f4eb0915640 1 -- 172.21.15.44:0/796124911 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eac0c8140 con 0x7f4e94004800 2024-04-02T13:20:45.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7fef367fc640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fef2406c9f0 0x7fef2406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7f64c5e06640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c006eca0 0x7f64c015f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7f89d97a4640 1 -- 172.21.15.44:0/3272496562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f89d404f8d0 con 0x7f89d4166610 2024-04-02T13:20:45.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7fef367fc640 1 -- 172.21.15.44:0/2272332301 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fef3c0adf80 con 0x7fef40071a50 2024-04-02T13:20:45.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7f64c5e06640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64c0162b60 0x7f64c00b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7fef367fc640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x7fef24076170 0x7fef240785b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7f64c5e06640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64c0166cf0 0x7f64c00b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7f64c5e06640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f64c006f080 con 0x7f64c0162b60 2024-04-02T13:20:45.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7fdd96ffd640 1 -- 172.21.15.44:0/3437639349 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdd9001a070 con 0x7fdd98038110 2024-04-02T13:20:45.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7f64c5e06640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f64c01667d0 con 0x7f64c006eca0 2024-04-02T13:20:45.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7f64c5e06640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f64c0162fb0 con 0x7f64c0166cf0 2024-04-02T13:20:45.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7ff3aeffd640 1 --2- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff398004800 0x7ff3b014bc90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff3a40072c0 tx=0x7ff3a4007780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7fef367fc640 1 -- 172.21.15.44:0/2272332301 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fef24078c80 con 0x7fef24076170 2024-04-02T13:20:45.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7fef367fc640 1 -- 172.21.15.44:0/2272332301 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fef3c0ae370 con 0x7fef40071a50 2024-04-02T13:20:45.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7fef45368640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fef2406c9f0 0x7fef2406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7f64bf7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c006eca0 0x7f64c015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7f64bf7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c006eca0 0x7f64c015f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34216/0 (socket says 172.21.15.44:34216) 2024-04-02T13:20:45.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7f64bf7fe640 1 -- 172.21.15.44:0/2390708189 learned_addr learned my addr 172.21.15.44:0/2390708189 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7f4e9affd640 1 -- 172.21.15.44:0/796124911 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e9c009040 con 0x7f4e94004800 2024-04-02T13:20:45.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7fdd96ffd640 1 --2- 172.21.15.44:0/3437639349 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdd8406c940 0x7fdd8406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7f4e9affd640 1 -- 172.21.15.44:0/796124911 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c00ecf0 con 0x7f4e94004800 2024-04-02T13:20:45.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7f5ba27fc640 1 -- 172.21.15.44:0/2362851089 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba400ce60 con 0x7f5bac15e4d0 2024-04-02T13:20:45.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.575+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5b94001030 con 0x7f5bac15e4d0 2024-04-02T13:20:45.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7f5ba27fc640 1 -- 172.21.15.44:0/2362851089 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5ba4019020 con 0x7f5bac15e4d0 2024-04-02T13:20:45.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7fdd96ffd640 1 -- 172.21.15.44:0/3437639349 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdd9008bd20 con 0x7fdd98038110 2024-04-02T13:20:45.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7f4e98ff9640 1 -- 172.21.15.44:0/796124911 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4eac0c3270 con 0x7f4e94004800 2024-04-02T13:20:45.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7ff397fff640 1 -- 172.21.15.44:0/342852530 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a400f960 con 0x7ff398004800 2024-04-02T13:20:45.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7ff3b592b640 1 -- 172.21.15.44:0/342852530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3b0153540 con 0x7ff398004800 2024-04-02T13:20:45.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7ff3b592b640 1 -- 172.21.15.44:0/342852530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3b0153b00 con 0x7ff398004800 2024-04-02T13:20:45.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.576+0000 7ff397fff640 1 -- 172.21.15.44:0/342852530 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a4018070 con 0x7ff398004800 2024-04-02T13:20:45.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7fdd96ffd640 1 --2- 172.21.15.44:0/3437639349 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x7fdd840760c0 0x7fdd84078500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7ff397fff640 1 -- 172.21.15.44:0/342852530 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a400fd40 con 0x7ff398004800 2024-04-02T13:20:45.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7fef4636a640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x7fef24076170 0x7fef240785b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7f5ba27fc640 1 --2- 172.21.15.44:0/2362851089 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b9006c9c0 0x7f5b9006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7f5ba27fc640 1 -- 172.21.15.44:0/2362851089 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5ba408c290 con 0x7f5bac15e4d0 2024-04-02T13:20:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7f64beffd640 1 --2- 172.21.15.44:0/2390708189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64c0162b60 0x7f64c00b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7f64bf7fe640 1 -- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64c0166cf0 msgr2=0x7f64c00b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7f64bf7fe640 1 --2- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64c0166cf0 0x7f64c00b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7f64bf7fe640 1 -- 172.21.15.44:0/2390708189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64c0162b60 msgr2=0x7f64c00b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7f64bf7fe640 1 --2- 172.21.15.44:0/2390708189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64c0162b60 0x7f64c00b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7f64bf7fe640 1 -- 172.21.15.44:0/2390708189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c01fcd70 con 0x7f64c006eca0 2024-04-02T13:20:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7f5ba27fc640 1 --2- 172.21.15.44:0/2362851089 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x7f5b90076130 0x7f5b90078570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7f5ba27fc640 1 -- 172.21.15.44:0/2362851089 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ba4012e50 con 0x7f5b90076130 2024-04-02T13:20:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7f5ba27fc640 1 -- 172.21.15.44:0/2362851089 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5ba408c680 con 0x7f5bac15e4d0 2024-04-02T13:20:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7fe806ffd640 1 -- 172.21.15.44:0/1862333177 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe80001a070 con 0x7fe8100cd290 2024-04-02T13:20:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7fef4636a640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x7fef24076170 0x7fef240785b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7fdd96ffd640 1 -- 172.21.15.44:0/3437639349 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdd90015e50 con 0x7fdd840760c0 2024-04-02T13:20:45.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7fdd96ffd640 1 -- 172.21.15.44:0/3437639349 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdd90091070 con 0x7fdd98038110 2024-04-02T13:20:45.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7fe8183c7640 1 -- 172.21.15.44:0/1862333177 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe8100684f0 con 0x7fe8100cd290 2024-04-02T13:20:45.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7f64bf7fe640 1 --2- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c006eca0 0x7f64c015f810 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f64b8036040 tx=0x7f64b8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7fdda5aac640 1 --2- 172.21.15.44:0/3437639349 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdd8406c940 0x7fdd8406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7fe806ffd640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe7f406c9f0 0x7fe7f406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7f89d0ff9640 1 -- 172.21.15.44:0/3272496562 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f89c4012b50 con 0x7f89d4166610 2024-04-02T13:20:45.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.577+0000 7f5bb1b69640 1 --2- 172.21.15.44:0/2362851089 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b9006c9c0 0x7f5b9006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7f5bb1368640 1 --2- 172.21.15.44:0/2362851089 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x7f5b90076130 0x7f5b90078570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7f64a7fff640 1 -- 172.21.15.44:0/2390708189 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b8043020 con 0x7f64c006eca0 2024-04-02T13:20:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7f64a7fff640 1 -- 172.21.15.44:0/2390708189 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64b803e450 con 0x7f64c006eca0 2024-04-02T13:20:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7f64c5e06640 1 -- 172.21.15.44:0/2390708189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64c014d740 con 0x7f64c006eca0 2024-04-02T13:20:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7fdda6aae640 1 --2- 172.21.15.44:0/3437639349 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x7fdd840760c0 0x7fdd84078500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7f64c5e06640 1 -- 172.21.15.44:0/2390708189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64c014dd00 con 0x7f64c006eca0 2024-04-02T13:20:45.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7fef367fc640 1 -- 172.21.15.44:0/2272332301 <== osd.2 v2:172.21.15.44:6816/2091834331 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fef24078c80 con 0x7fef24076170 2024-04-02T13:20:45.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7fe806ffd640 1 -- 172.21.15.44:0/1862333177 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe80008b230 con 0x7fe8100cd290 2024-04-02T13:20:45.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7fe806ffd640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x7fe7f40760f0 0x7fe7f4078530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7fe806ffd640 1 -- 172.21.15.44:0/1862333177 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe800015e70 con 0x7fe7f40760f0 2024-04-02T13:20:45.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7fe815962640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe7f406c9f0 0x7fe7f406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7fe806ffd640 1 -- 172.21.15.44:0/1862333177 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe800058bd0 con 0x7fe8100cd290 2024-04-02T13:20:45.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7f5bb1368640 1 --2- 172.21.15.44:0/2362851089 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x7f5b90076130 0x7f5b90078570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7ff397fff640 1 -- 172.21.15.44:0/342852530 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff3a401d460 con 0x7ff398004800 2024-04-02T13:20:45.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7f4e9affd640 1 -- 172.21.15.44:0/796124911 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4e9c0113f0 con 0x7f4e94004800 2024-04-02T13:20:45.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7f89d0ff9640 1 --2- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f89c006c9c0 0x7f89c006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.578+0000 7f89d3fff640 1 --2- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f89c006c9c0 0x7f89c006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7f89d0ff9640 1 -- 172.21.15.44:0/3272496562 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f89c408cb20 con 0x7f89d4166610 2024-04-02T13:20:45.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7f89d0ff9640 1 --2- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x7f89c0076140 0x7f89c0078580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7f89d0ff9640 1 -- 172.21.15.44:0/3272496562 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89c401ce80 con 0x7f89c0076140 2024-04-02T13:20:45.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7f89d0ff9640 1 -- 172.21.15.44:0/3272496562 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f89c408cf10 con 0x7f89d4166610 2024-04-02T13:20:45.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7f89d3fff640 1 --2- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f89c006c9c0 0x7f89c006ee80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f89cc005c00 tx=0x7f89cc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7f89d37fe640 1 --2- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x7f89c0076140 0x7f89c0078580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7fdda6aae640 1 --2- 172.21.15.44:0/3437639349 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x7fdd840760c0 0x7fdd84078500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7f4e9affd640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e8c06c9c0 0x7f4e8c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7f89d37fe640 1 --2- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x7f89c0076140 0x7f89c0078580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7ff397fff640 1 --2- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff39c06c970 0x7ff39c06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7ff397fff640 1 -- 172.21.15.44:0/342852530 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff3a4057850 con 0x7ff398004800 2024-04-02T13:20:45.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7f64a7fff640 1 -- 172.21.15.44:0/2390708189 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b803e840 con 0x7f64c006eca0 2024-04-02T13:20:45.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7f4e9affd640 1 -- 172.21.15.44:0/796124911 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4e9c08ca70 con 0x7f4e94004800 2024-04-02T13:20:45.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.579+0000 7f5bb1b69640 1 --2- 172.21.15.44:0/2362851089 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b9006c9c0 0x7f5b9006ee80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5bac0ce2c0 tx=0x7f5b9c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7f4ea9d9b640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e8c06c9c0 0x7f4e8c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7ff3ae7fc640 1 --2- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff39c06c970 0x7ff39c06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7f4e9affd640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x7f4e8c076140 0x7f4e8c078580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7f4e9affd640 1 -- 172.21.15.44:0/796124911 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4e9c00ee90 con 0x7f4e8c076140 2024-04-02T13:20:45.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7f4e9affd640 1 -- 172.21.15.44:0/796124911 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4e9c0912a0 con 0x7f4e94004800 2024-04-02T13:20:45.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7f5ba27fc640 1 -- 172.21.15.44:0/2362851089 <== osd.0 v2:172.21.15.44:6800/2303805525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5ba4012e50 con 0x7f5b90076130 2024-04-02T13:20:45.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7fef45368640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fef2406c9f0 0x7fef2406eeb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef400cdb60 tx=0x7fef30009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7f4eaad9d640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x7f4e8c076140 0x7f4e8c078580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.581+0000 7f4ea9d9b640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e8c06c9c0 0x7f4e8c06ee80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4035000 tx=0x7f4ea4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.580+0000 7fe816964640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x7fe7f40760f0 0x7fe7f4078530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.581+0000 7fe816964640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x7fe7f40760f0 0x7fe7f4078530 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.581+0000 7ff3b592b640 1 --2- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x7ff390001650 0x7ff390003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.581+0000 7ff3b592b640 1 -- 172.21.15.44:0/342852530 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff390006bb0 con 0x7ff390001650 2024-04-02T13:20:45.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.581+0000 7f89d0ff9640 1 -- 172.21.15.44:0/3272496562 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89c401ce80 con 0x7f89c0076140 2024-04-02T13:20:45.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.581+0000 7f64a7fff640 1 -- 172.21.15.44:0/2390708189 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f64b8046020 con 0x7f64c006eca0 2024-04-02T13:20:45.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.581+0000 7fdda5aac640 1 --2- 172.21.15.44:0/3437639349 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdd8406c940 0x7fdd8406ee00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdda0051040 tx=0x7fdda00b1040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.581+0000 7f4eaad9d640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x7f4e8c076140 0x7f4e8c078580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.581+0000 7ff3ae7fc640 1 --2- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff39c06c970 0x7ff39c06ee30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8036000 tx=0x7ff3a802e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.581+0000 7fdd96ffd640 1 -- 172.21.15.44:0/3437639349 <== osd.3 v2:172.21.15.44:6824/1588952513 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdd90015e50 con 0x7fdd840760c0 2024-04-02T13:20:45.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.581+0000 7ff3af7fe640 1 --2- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x7ff390001650 0x7ff390003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.581+0000 7f64a7fff640 1 --2- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f64ac06c940 0x7f64ac06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.581+0000 7fe815962640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe7f406c9f0 0x7fe7f406eeb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009560 tx=0x7fe80c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.582+0000 7fe806ffd640 1 -- 172.21.15.44:0/1862333177 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe800015e70 con 0x7fe7f40760f0 2024-04-02T13:20:45.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.582+0000 7f64a7fff640 1 -- 172.21.15.44:0/2390708189 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f64b80ae520 con 0x7f64c006eca0 2024-04-02T13:20:45.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.582+0000 7f64c5e06640 1 -- 172.21.15.44:0/2390708189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f64a0001030 con 0x7f64c006eca0 2024-04-02T13:20:45.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.582+0000 7f64beffd640 1 --2- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f64ac06c940 0x7f64ac06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.582+0000 7f64a7fff640 1 --2- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x7f64ac076040 0x7f64ac078480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.582+0000 7f64a7fff640 1 -- 172.21.15.44:0/2390708189 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64b803e450 con 0x7f64ac076040 2024-04-02T13:20:45.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.582+0000 7f64a7fff640 1 -- 172.21.15.44:0/2390708189 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f64b8078290 con 0x7f64c006eca0 2024-04-02T13:20:45.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.582+0000 7f4e9affd640 1 -- 172.21.15.44:0/796124911 <== osd.4 v2:172.21.15.69:6816/1041350007 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4e9c00ee90 con 0x7f4e8c076140 2024-04-02T13:20:45.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.583+0000 7f64beffd640 1 --2- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f64ac06c940 0x7f64ac06ee00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f64c01508e0 tx=0x7f64b4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.583+0000 7ff3af7fe640 1 --2- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x7ff390001650 0x7ff390003b10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.583+0000 7f64bffff640 1 --2- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x7f64ac076040 0x7f64ac078480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.584+0000 7f64bffff640 1 --2- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x7f64ac076040 0x7f64ac078480 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.584+0000 7ff397fff640 1 -- 172.21.15.44:0/342852530 <== osd.5 v2:172.21.15.69:6800/2841767499 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff390006bb0 con 0x7ff390001650 2024-04-02T13:20:45.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.587+0000 7f64a7fff640 1 -- 172.21.15.44:0/2390708189 <== osd.7 v2:172.21.15.69:6824/1292475712 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f64b803e450 con 0x7f64ac076040 2024-04-02T13:20:45.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.609+0000 7f89d97a4640 1 -- 172.21.15.44:0/3272496562 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f89d40ce4a0 con 0x7f89c0076140 2024-04-02T13:20:45.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.609+0000 7f89d0ff9640 1 -- 172.21.15.44:0/3272496562 <== osd.6 v2:172.21.15.69:6802/2604158175 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f89d40ce4a0 con 0x7f89c0076140 2024-04-02T13:20:45.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.610+0000 7f89ba7fc640 1 -- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x7f89c0076140 msgr2=0x7f89c0078580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.610+0000 7f89ba7fc640 1 --2- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x7f89c0076140 0x7f89c0078580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.610+0000 7f89ba7fc640 1 -- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f89c006c9c0 msgr2=0x7f89c006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.610+0000 7f89ba7fc640 1 --2- 172.21.15.44:0/3272496562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f89c006c9c0 0x7f89c006ee80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f89cc005c00 tx=0x7f89cc009040 comp rx=0 tx=0).stop 2024-04-02T13:20:45.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.610+0000 7f89ba7fc640 1 -- 172.21.15.44:0/3272496562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d4166610 msgr2=0x7f89d40b1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.610+0000 7f89ba7fc640 1 --2- 172.21.15.44:0/3272496562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d4166610 0x7f89d40b1d80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f89c400d530 tx=0x7f89c400d9f0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.611+0000 7f89d8fa3640 1 -- 172.21.15.44:0/3272496562 reap_dead start 2024-04-02T13:20:45.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.612+0000 7f89ba7fc640 1 -- 172.21.15.44:0/3272496562 shutdown_connections 2024-04-02T13:20:45.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.612+0000 7f89ba7fc640 1 -- 172.21.15.44:0/3272496562 >> 172.21.15.44:0/3272496562 conn(0x7f89d406b4d0 msgr2=0x7f89d406ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.612+0000 7f89ba7fc640 1 -- 172.21.15.44:0/3272496562 shutdown_connections 2024-04-02T13:20:45.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.612+0000 7f89ba7fc640 1 -- 172.21.15.44:0/3272496562 wait complete. 2024-04-02T13:20:45.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.612+0000 7fe8183c7640 1 -- 172.21.15.44:0/1862333177 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe8100caac0 con 0x7fe7f40760f0 2024-04-02T13:20:45.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.613+0000 7fe806ffd640 1 -- 172.21.15.44:0/1862333177 <== osd.1 v2:172.21.15.44:6801/704457846 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe8100caac0 con 0x7fe7f40760f0 2024-04-02T13:20:45.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.614+0000 7fe804ff9640 1 -- 172.21.15.44:0/1862333177 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x7fe7f40760f0 msgr2=0x7fe7f4078530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.614+0000 7fe804ff9640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x7fe7f40760f0 0x7fe7f4078530 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.614+0000 7fe804ff9640 1 -- 172.21.15.44:0/1862333177 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe7f406c9f0 msgr2=0x7fe7f406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.614+0000 7fe804ff9640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe7f406c9f0 0x7fe7f406eeb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009560 tx=0x7fe80c009190 comp rx=0 tx=0).stop 2024-04-02T13:20:45.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.614+0000 7fe804ff9640 1 -- 172.21.15.44:0/1862333177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8100cd290 msgr2=0x7fe8101fb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.614+0000 7fe804ff9640 1 --2- 172.21.15.44:0/1862333177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8100cd290 0x7fe8101fb460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe80000c410 tx=0x7fe80000c8d0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.614+0000 7fe816964640 1 -- 172.21.15.44:0/1862333177 reap_dead start 2024-04-02T13:20:45.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.614+0000 7fe804ff9640 1 -- 172.21.15.44:0/1862333177 shutdown_connections 2024-04-02T13:20:45.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.614+0000 7fe804ff9640 1 -- 172.21.15.44:0/1862333177 >> 172.21.15.44:0/1862333177 conn(0x7fe81009aa20 msgr2=0x7fe8100c4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.614+0000 7fe804ff9640 1 -- 172.21.15.44:0/1862333177 shutdown_connections 2024-04-02T13:20:45.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.614+0000 7fef47dcd640 1 -- 172.21.15.44:0/2272332301 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fef40068c00 con 0x7fef24076170 2024-04-02T13:20:45.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.615+0000 7fe804ff9640 1 -- 172.21.15.44:0/1862333177 wait complete. 2024-04-02T13:20:45.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.614+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5b94002dd0 con 0x7f5b90076130 2024-04-02T13:20:45.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.615+0000 7fef367fc640 1 -- 172.21.15.44:0/2272332301 <== osd.2 v2:172.21.15.44:6816/2091834331 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fef40068c00 con 0x7fef24076170 2024-04-02T13:20:45.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.615+0000 7fef47dcd640 1 -- 172.21.15.44:0/2272332301 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x7fef24076170 msgr2=0x7fef240785b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.615+0000 7fef47dcd640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x7fef24076170 0x7fef240785b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.615+0000 7fef47dcd640 1 -- 172.21.15.44:0/2272332301 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fef2406c9f0 msgr2=0x7fef2406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.615+0000 7fdda72af640 1 -- 172.21.15.44:0/3437639349 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdd98005bc0 con 0x7fdd840760c0 2024-04-02T13:20:45.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.615+0000 7fef47dcd640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fef2406c9f0 0x7fef2406eeb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef400cdb60 tx=0x7fef30009060 comp rx=0 tx=0).stop 2024-04-02T13:20:45.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.616+0000 7fef47dcd640 1 -- 172.21.15.44:0/2272332301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef40071a50 msgr2=0x7fef4015c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.616+0000 7fef47dcd640 1 --2- 172.21.15.44:0/2272332301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef40071a50 0x7fef4015c7f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fef3c02bf00 tx=0x7fef3c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.616+0000 7fef4636a640 1 -- 172.21.15.44:0/2272332301 reap_dead start 2024-04-02T13:20:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.616+0000 7f5ba27fc640 1 -- 172.21.15.44:0/2362851089 <== osd.0 v2:172.21.15.44:6800/2303805525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5b94002dd0 con 0x7f5b90076130 2024-04-02T13:20:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.616+0000 7fef47dcd640 1 -- 172.21.15.44:0/2272332301 shutdown_connections 2024-04-02T13:20:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.616+0000 7fef47dcd640 1 -- 172.21.15.44:0/2272332301 >> 172.21.15.44:0/2272332301 conn(0x7fef40074dc0 msgr2=0x7fef40160950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.616+0000 7fdd96ffd640 1 -- 172.21.15.44:0/3437639349 <== osd.3 v2:172.21.15.44:6824/1588952513 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdd98005bc0 con 0x7fdd840760c0 2024-04-02T13:20:45.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.616+0000 7fef47dcd640 1 -- 172.21.15.44:0/2272332301 shutdown_connections 2024-04-02T13:20:45.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.616+0000 7fef47dcd640 1 -- 172.21.15.44:0/2272332301 wait complete. 2024-04-02T13:20:45.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.616+0000 7f4e98ff9640 1 -- 172.21.15.44:0/796124911 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4eac0684f0 con 0x7f4e8c076140 2024-04-02T13:20:45.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7fdda72af640 1 -- 172.21.15.44:0/3437639349 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x7fdd840760c0 msgr2=0x7fdd84078500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7fdda72af640 1 --2- 172.21.15.44:0/3437639349 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x7fdd840760c0 0x7fdd84078500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x7f5b90076130 msgr2=0x7f5b90078570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7f5bb3dcd640 1 --2- 172.21.15.44:0/2362851089 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x7f5b90076130 0x7f5b90078570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b9006c9c0 msgr2=0x7f5b9006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7f5bb3dcd640 1 --2- 172.21.15.44:0/2362851089 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b9006c9c0 0x7f5b9006ee80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5bac0ce2c0 tx=0x7f5b9c03c040 comp rx=0 tx=0).stop 2024-04-02T13:20:45.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7fdda72af640 1 -- 172.21.15.44:0/3437639349 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdd8406c940 msgr2=0x7fdd8406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bac15e4d0 msgr2=0x7f5bac15c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7f5bb3dcd640 1 --2- 172.21.15.44:0/2362851089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bac15e4d0 0x7f5bac15c800 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400c7b0 tx=0x7f5ba400cc70 comp rx=0 tx=0).stop 2024-04-02T13:20:45.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7fdda72af640 1 --2- 172.21.15.44:0/3437639349 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdd8406c940 0x7fdd8406ee00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdda0051040 tx=0x7fdda00b1040 comp rx=0 tx=0).stop 2024-04-02T13:20:45.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7fdda72af640 1 -- 172.21.15.44:0/3437639349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd98038110 msgr2=0x7fdd980dc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7fdda72af640 1 --2- 172.21.15.44:0/3437639349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd98038110 0x7fdd980dc740 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000e850 tx=0x7fdd9000ed10 comp rx=0 tx=0).stop 2024-04-02T13:20:45.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7fdda6aae640 1 -- 172.21.15.44:0/3437639349 reap_dead start 2024-04-02T13:20:45.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7fdda72af640 1 -- 172.21.15.44:0/3437639349 shutdown_connections 2024-04-02T13:20:45.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.617+0000 7fdda72af640 1 -- 172.21.15.44:0/3437639349 >> 172.21.15.44:0/3437639349 conn(0x7fdd98009fb0 msgr2=0x7fdd98048490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.618+0000 7f4e9affd640 1 -- 172.21.15.44:0/796124911 <== osd.4 v2:172.21.15.69:6816/1041350007 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4eac0684f0 con 0x7f4e8c076140 2024-04-02T13:20:45.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.618+0000 7f5bb236a640 1 -- 172.21.15.44:0/2362851089 reap_dead start 2024-04-02T13:20:45.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.618+0000 7fdda72af640 1 -- 172.21.15.44:0/3437639349 shutdown_connections 2024-04-02T13:20:45.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.618+0000 7fdda72af640 1 -- 172.21.15.44:0/3437639349 wait complete. 2024-04-02T13:20:45.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.618+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 shutdown_connections 2024-04-02T13:20:45.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.618+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 >> 172.21.15.44:0/2362851089 conn(0x7f5bac09d410 msgr2=0x7f5bac06d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.618+0000 7ff3b592b640 1 -- 172.21.15.44:0/342852530 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff390005c40 con 0x7ff390001650 2024-04-02T13:20:45.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.618+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 shutdown_connections 2024-04-02T13:20:45.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.618+0000 7f5bb3dcd640 1 -- 172.21.15.44:0/2362851089 wait complete. 2024-04-02T13:20:45.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.619+0000 7ff397fff640 1 -- 172.21.15.44:0/342852530 <== osd.5 v2:172.21.15.69:6800/2841767499 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff390005c40 con 0x7ff390001650 2024-04-02T13:20:45.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.619+0000 7f64c5e06640 1 -- 172.21.15.44:0/2390708189 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f64a0002dd0 con 0x7f64ac076040 2024-04-02T13:20:45.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.619+0000 7f4e98ff9640 1 -- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x7f4e8c076140 msgr2=0x7f4e8c078580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.619+0000 7f4e98ff9640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x7f4e8c076140 0x7f4e8c078580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.619+0000 7f4e98ff9640 1 -- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e8c06c9c0 msgr2=0x7f4e8c06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.619+0000 7f4e98ff9640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e8c06c9c0 0x7f4e8c06ee80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4035000 tx=0x7f4ea4030000 comp rx=0 tx=0).stop 2024-04-02T13:20:45.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.619+0000 7f4e98ff9640 1 -- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94004800 msgr2=0x7f4eac0c8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.619+0000 7f4e98ff9640 1 --2- 172.21.15.44:0/796124911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94004800 0x7f4eac0c8ab0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0109d0 tx=0x7f4e9c010e90 comp rx=0 tx=0).stop 2024-04-02T13:20:45.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7f4eaad9d640 1 -- 172.21.15.44:0/796124911 reap_dead start 2024-04-02T13:20:45.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.619+0000 7f64a7fff640 1 -- 172.21.15.44:0/2390708189 <== osd.7 v2:172.21.15.69:6824/1292475712 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f64a0002dd0 con 0x7f64ac076040 2024-04-02T13:20:45.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7f4e98ff9640 1 -- 172.21.15.44:0/796124911 shutdown_connections 2024-04-02T13:20:45.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7ff3b592b640 1 -- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x7ff390001650 msgr2=0x7ff390003b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7f4e98ff9640 1 -- 172.21.15.44:0/796124911 >> 172.21.15.44:0/796124911 conn(0x7f4eac0cfaa0 msgr2=0x7f4eac0af430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7ff3b592b640 1 --2- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x7ff390001650 0x7ff390003b10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7ff3b592b640 1 -- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff39c06c970 msgr2=0x7ff39c06ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7ff3b592b640 1 --2- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff39c06c970 0x7ff39c06ee30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8036000 tx=0x7ff3a802e000 comp rx=0 tx=0).stop 2024-04-02T13:20:45.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7ff3b592b640 1 -- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff398004800 msgr2=0x7ff3b014bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7ff3b592b640 1 --2- 172.21.15.44:0/342852530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff398004800 0x7ff3b014bc90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff3a40072c0 tx=0x7ff3a4007780 comp rx=0 tx=0).stop 2024-04-02T13:20:45.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7f4e98ff9640 1 -- 172.21.15.44:0/796124911 shutdown_connections 2024-04-02T13:20:45.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7f4e98ff9640 1 -- 172.21.15.44:0/796124911 wait complete. 2024-04-02T13:20:45.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7ff3af7fe640 1 -- 172.21.15.44:0/342852530 reap_dead start 2024-04-02T13:20:45.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7ff3b592b640 1 -- 172.21.15.44:0/342852530 shutdown_connections 2024-04-02T13:20:45.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7ff3b592b640 1 -- 172.21.15.44:0/342852530 >> 172.21.15.44:0/342852530 conn(0x7ff3b00cfcd0 msgr2=0x7ff3b0071190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.620+0000 7f64c5e06640 1 -- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x7f64ac076040 msgr2=0x7f64ac078480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.621+0000 7f64c5e06640 1 --2- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x7f64ac076040 0x7f64ac078480 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.621+0000 7f64c5e06640 1 -- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f64ac06c940 msgr2=0x7f64ac06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.621+0000 7f64c5e06640 1 --2- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f64ac06c940 0x7f64ac06ee00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f64c01508e0 tx=0x7f64b4008040 comp rx=0 tx=0).stop 2024-04-02T13:20:45.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.621+0000 7f64c5e06640 1 -- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c006eca0 msgr2=0x7f64c015f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.621+0000 7f64c5e06640 1 --2- 172.21.15.44:0/2390708189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c006eca0 0x7f64c015f810 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f64b8036040 tx=0x7f64b8004a40 comp rx=0 tx=0).stop 2024-04-02T13:20:45.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.621+0000 7ff3b592b640 1 -- 172.21.15.44:0/342852530 shutdown_connections 2024-04-02T13:20:45.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.621+0000 7f64bffff640 1 -- 172.21.15.44:0/2390708189 reap_dead start 2024-04-02T13:20:45.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.621+0000 7ff3b592b640 1 -- 172.21.15.44:0/342852530 wait complete. 2024-04-02T13:20:45.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.621+0000 7f64c5e06640 1 -- 172.21.15.44:0/2390708189 shutdown_connections 2024-04-02T13:20:45.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.621+0000 7f64c5e06640 1 -- 172.21.15.44:0/2390708189 >> 172.21.15.44:0/2390708189 conn(0x7f64c006b6b0 msgr2=0x7f64c015b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.621+0000 7f64c5e06640 1 -- 172.21.15.44:0/2390708189 shutdown_connections 2024-04-02T13:20:45.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.621+0000 7f64c5e06640 1 -- 172.21.15.44:0/2390708189 wait complete. 2024-04-02T13:20:45.699 INFO:teuthology.orchestra.run.smithi044.stdout:55834574856 2024-04-02T13:20:45.700 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-02T13:20:45.701 INFO:teuthology.orchestra.run.smithi044.stdout:55834574856 2024-04-02T13:20:45.701 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T13:20:45.702 INFO:teuthology.orchestra.run.smithi044.stdout:55834574856 2024-04-02T13:20:45.702 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T13:20:45.703 INFO:teuthology.orchestra.run.smithi044.stdout:55834574857 2024-04-02T13:20:45.703 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T13:20:45.704 INFO:teuthology.orchestra.run.smithi044.stdout:55834574857 2024-04-02T13:20:45.704 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-02T13:20:45.705 INFO:teuthology.orchestra.run.smithi044.stdout:55834574857 2024-04-02T13:20:45.706 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-02T13:20:45.706 INFO:teuthology.orchestra.run.smithi044.stdout:55834574857 2024-04-02T13:20:45.707 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-02T13:20:45.708 INFO:teuthology.orchestra.run.smithi044.stdout:55834574857 2024-04-02T13:20:45.708 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-02T13:20:45.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.909+0000 7fd537c3c640 1 Processor -- start 2024-04-02T13:20:45.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.909+0000 7f2da55d8640 1 Processor -- start 2024-04-02T13:20:45.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.909+0000 7f2da55d8640 1 -- start start 2024-04-02T13:20:45.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.909+0000 7f50a9582640 1 Processor -- start 2024-04-02T13:20:45.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.909+0000 7f50a9582640 1 -- start start 2024-04-02T13:20:45.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.910+0000 7fd537c3c640 1 -- start start 2024-04-02T13:20:45.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7fd537c3c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530162800 0x7fd53006d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7fd537c3c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd530166cb0 0x7fd5300b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7fd537c3c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5300b3eb0 con 0x7fd5300cc6e0 2024-04-02T13:20:45.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7fd537c3c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd530162be0 con 0x7fd530166cb0 2024-04-02T13:20:45.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7fbffae7b640 1 Processor -- start 2024-04-02T13:20:45.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7fd537c3c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd5300b47a0 con 0x7fd530162800 2024-04-02T13:20:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.910+0000 7f2da55d8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2d98048be0 0x7f2d98053fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.910+0000 7f2da55d8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2d98035040 0x7f2d98048fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.910+0000 7f2da55d8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d98034b10 0x7f2d9800b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.910+0000 7f2da55d8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2d9803b2d0 con 0x7f2d98048be0 2024-04-02T13:20:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.910+0000 7f2da55d8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2d9803bae0 con 0x7f2d98035040 2024-04-02T13:20:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.910+0000 7f2da55d8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2d9800b510 con 0x7f2d98034b10 2024-04-02T13:20:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7f2d9ffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2d98048be0 0x7f2d98053fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7f2d9f7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2d98035040 0x7f2d98048fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7f2d9f7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2d98035040 0x7f2d98048fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34228/0 (socket says 172.21.15.44:34228) 2024-04-02T13:20:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7f2d9f7fe640 1 -- 172.21.15.44:0/1621226292 learned_addr learned my addr 172.21.15.44:0/1621226292 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7f2da4dd7640 1 --2- 172.21.15.44:0/1621226292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d98034b10 0x7f2d9800b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7fd5361d9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd5300cc6e0 0x7fd530071190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57404/0 (socket says 172.21.15.44:57404) 2024-04-02T13:20:45.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7fd5361d9640 1 -- 172.21.15.44:0/236897134 learned_addr learned my addr 172.21.15.44:0/236897134 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:45.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7f50a9582640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a4162660 0x7f50a40ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7f50a9582640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f50a406e090 0x7f50a40c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7f50a9582640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f50a40c5a20 0x7f50a41660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7f50a2ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a4162660 0x7f50a40ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7f50a2ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a4162660 0x7f50a40ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:41992/0 (socket says 172.21.15.44:41992) 2024-04-02T13:20:45.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1083207618 0 0) 0x7fd5300b3eb0 con 0x7fd5300cc6e0 2024-04-02T13:20:45.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd51c003520 con 0x7fd5300cc6e0 2024-04-02T13:20:45.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7f50a2ffd640 1 -- 172.21.15.44:0/2845077386 learned_addr learned my addr 172.21.15.44:0/2845077386 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7fd5351d7640 1 --2- 172.21.15.44:0/236897134 >> v2:172.21.15.44:3300/0 conn(0x7fd530166cb0 0x7fd5300b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7f50a27fc640 1 --2- 172.21.15.44:0/2845077386 >> v2:172.21.15.44:3300/0 conn(0x7f50a406e090 0x7f50a40c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7f50a3fff640 1 --2- 172.21.15.44:0/2845077386 >> v2:172.21.15.69:3300/0 conn(0x7f50a40c5a20 0x7f50a41660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7f50a9582640 1 -- 172.21.15.44:0/2845077386 --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f50a40cf630 con 0x7f50a406e090 2024-04-02T13:20:45.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7f50a9582640 1 -- 172.21.15.44:0/2845077386 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f50a40cf910 con 0x7f50a40c5a20 2024-04-02T13:20:45.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7f50a9582640 1 -- 172.21.15.44:0/2845077386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f50a40cfbf0 con 0x7f50a4162660 2024-04-02T13:20:45.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7f2d9ffff640 1 -- 172.21.15.44:0/1621226292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d98034b10 msgr2=0x7f2d9800b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7fbffae7b640 1 -- start start 2024-04-02T13:20:45.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7fbffae7b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbff40b4770 0x7fbff40cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7fbffae7b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff406e180 con 0x7fbff406dcd0 2024-04-02T13:20:45.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7fbffae7b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff415ae50 con 0x7fbff406e3e0 2024-04-02T13:20:45.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.912+0000 7fbffae7b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbff40b5000 con 0x7fbff40b4770 2024-04-02T13:20:45.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7f2d9ffff640 1 --2- 172.21.15.44:0/1621226292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d98034b10 0x7f2d9800b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7f2d9ffff640 1 -- 172.21.15.44:0/1621226292 >> v2:172.21.15.69:3300/0 conn(0x7f2d98035040 msgr2=0x7f2d98048fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7f2d9ffff640 1 --2- 172.21.15.44:0/1621226292 >> v2:172.21.15.69:3300/0 conn(0x7f2d98035040 0x7f2d98048fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fbff3fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbff406dcd0 0x7fbff40b4060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57418/0 (socket says 172.21.15.44:57418) 2024-04-02T13:20:45.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fbff3fff640 1 -- 172.21.15.44:0/3842184315 learned_addr learned my addr 172.21.15.44:0/3842184315 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:45.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fbff8c17640 1 --1- 172.21.15.44:0/3842184315 >> v1:172.21.15.69:6789/0 conn(0x7fbff406e3e0 0x7fbff406e7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:48662/0 (socket says 172.21.15.44:48662) 2024-04-02T13:20:45.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.911+0000 7f2d9ffff640 1 -- 172.21.15.44:0/1621226292 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d98054510 con 0x7f2d98048be0 2024-04-02T13:20:45.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fd5359d8640 1 --2- 172.21.15.44:0/236897134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530162800 0x7fd53006d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3003224741 0 0) 0x7fd51c003520 con 0x7fd5300cc6e0 2024-04-02T13:20:45.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fbff9418640 1 --2- 172.21.15.44:0/3842184315 >> v2:172.21.15.69:3300/0 conn(0x7fbff40b4770 0x7fbff40cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530162800 msgr2=0x7fd53006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fd5349d6640 1 --2- 172.21.15.44:0/236897134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530162800 0x7fd53006d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 >> v2:172.21.15.44:3300/0 conn(0x7fd530166cb0 msgr2=0x7fd5300b4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fd5349d6640 1 --2- 172.21.15.44:0/236897134 >> v2:172.21.15.44:3300/0 conn(0x7fd530166cb0 0x7fd5300b4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5300b4150 con 0x7fd5300cc6e0 2024-04-02T13:20:45.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7f2d9ffff640 1 --2- 172.21.15.44:0/1621226292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d98048be0 0x7f2d98053fd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2d98038710 tx=0x7f2d9402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e290166939331e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fbff37fe640 1 -- 172.21.15.44:0/3842184315 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 226564800 0 0) 0x7fbff415ae50 con 0x7fbff406e3e0 2024-04-02T13:20:45.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fbff37fe640 1 -- 172.21.15.44:0/3842184315 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfdc003520 con 0x7fbff406e3e0 2024-04-02T13:20:45.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fd5280019f0 con 0x7fd5300cc6e0 2024-04-02T13:20:45.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 >> v1:172.21.15.44:6789/0 conn(0x7fd5300cc6e0 legacy=0x7fd530071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fd5349d6640 1 --2- 172.21.15.44:0/236897134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c0039b0 0x7fd51c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fd5349d6640 1 --2- 172.21.15.44:0/236897134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd51c004800 0x7fd51c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7f50a27fc640 1 -- 172.21.15.44:0/2845077386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a4162660 msgr2=0x7f50a40ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7f50a27fc640 1 --2- 172.21.15.44:0/2845077386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a4162660 0x7f50a40ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7f50a27fc640 1 -- 172.21.15.44:0/2845077386 >> v2:172.21.15.69:3300/0 conn(0x7f50a40c5a20 msgr2=0x7f50a41660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7f50a27fc640 1 --2- 172.21.15.44:0/2845077386 >> v2:172.21.15.69:3300/0 conn(0x7f50a40c5a20 0x7f50a41660b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7f50a27fc640 1 -- 172.21.15.44:0/2845077386 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a40ccd00 con 0x7f50a406e090 2024-04-02T13:20:45.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fd5349d6640 1 --2- 172.21.15.44:0/236897134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530162800 0x7fd51c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fbff37fe640 1 -- 172.21.15.44:0/3842184315 <== mon.1 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2758673065 0 0) 0x7fbfdc003520 con 0x7fbff406e3e0 2024-04-02T13:20:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fbff37fe640 1 -- 172.21.15.44:0/3842184315 >> v2:172.21.15.69:3300/0 conn(0x7fbff40b4770 msgr2=0x7fbff40cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fbff37fe640 1 --2- 172.21.15.44:0/3842184315 >> v2:172.21.15.69:3300/0 conn(0x7fbff40b4770 0x7fbff40cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fbff37fe640 1 -- 172.21.15.44:0/3842184315 >> v1:172.21.15.44:6789/0 conn(0x7fbff406dcd0 legacy=0x7fbff40b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.913+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd530162be0 con 0x7fd51c004800 2024-04-02T13:20:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd5300b47a0 con 0x7fd51c0039b0 2024-04-02T13:20:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7fd5361d9640 1 --2- 172.21.15.44:0/236897134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530162800 0x7fd51c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7eff9bfff640 1 Processor -- start 2024-04-02T13:20:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7fd5359d8640 1 --2- 172.21.15.44:0/236897134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c0039b0 0x7fd51c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd51c0055a0 con 0x7fd530162800 2024-04-02T13:20:45.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7eff9bfff640 1 -- start start 2024-04-02T13:20:45.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7f50a27fc640 1 --2- 172.21.15.44:0/2845077386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50a406e090 0x7f50a40c54e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f50a40b1da0 tx=0x7f509802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4439c669493c61ee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7fc125f67640 1 Processor -- start 2024-04-02T13:20:45.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7fd5361d9640 1 -- 172.21.15.44:0/236897134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c0039b0 msgr2=0x7fd51c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7fd5351d7640 1 --2- 172.21.15.44:0/236897134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd51c004800 0x7fd51c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7fd5361d9640 1 --2- 172.21.15.44:0/236897134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c0039b0 0x7fd51c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7fd5361d9640 1 -- 172.21.15.44:0/236897134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd51c004800 msgr2=0x7fd51c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7fd5361d9640 1 --2- 172.21.15.44:0/236897134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd51c004800 0x7fd51c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7fd5361d9640 1 -- 172.21.15.44:0/236897134 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51c003520 con 0x7fd530162800 2024-04-02T13:20:45.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7fc125f67640 1 -- start start 2024-04-02T13:20:45.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.914+0000 7fd5351d7640 1 --2- 172.21.15.44:0/236897134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd51c004800 0x7fd51c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:45.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7f2d9effd640 1 -- 172.21.15.44:0/1621226292 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94034070 con 0x7f2d98048be0 2024-04-02T13:20:45.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7f2d9effd640 1 -- 172.21.15.44:0/1621226292 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d94030650 con 0x7f2d98048be0 2024-04-02T13:20:45.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7f58aa6db640 1 Processor -- start 2024-04-02T13:20:45.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7eff9bfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7eff9c070fc0 0x7eff9c15b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7eff9bfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff9c0cd9d0 0x7eff9c0cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7f2da55d8640 1 -- 172.21.15.44:0/1621226292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d98048be0 msgr2=0x7f2d98053fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7f2da55d8640 1 --2- 172.21.15.44:0/1621226292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d98048be0 0x7f2d98053fd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2d98038710 tx=0x7f2d9402b9a0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7f58aa6db640 1 -- start start 2024-04-02T13:20:45.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7eff9bfff640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff9c0b3000 con 0x7eff9c0b4a70 2024-04-02T13:20:45.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7f2da55d8640 1 -- 172.21.15.44:0/1621226292 shutdown_connections 2024-04-02T13:20:45.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7eff9bfff640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7eff9c0b32c0 con 0x7eff9c070fc0 2024-04-02T13:20:45.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7eff9bfff640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7eff9c0b35a0 con 0x7eff9c0cd9d0 2024-04-02T13:20:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7eff9a7fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7eff9c0b4a70 0x7eff9c0713a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57432/0 (socket says 172.21.15.44:57432) 2024-04-02T13:20:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7f2da55d8640 1 --2- 172.21.15.44:0/1621226292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d98034b10 0x7f2d9800b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7f2da55d8640 1 --2- 172.21.15.44:0/1621226292 >> v2:172.21.15.69:3300/0 conn(0x7f2d98035040 0x7f2d98048fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7f2da55d8640 1 --2- 172.21.15.44:0/1621226292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d98048be0 0x7f2d98053fd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7f2da55d8640 1 -- 172.21.15.44:0/1621226292 >> 172.21.15.44:0/1621226292 conn(0x7f2d9801ef70 msgr2=0x7f2d98055c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.915+0000 7eff9a7fc640 1 -- 172.21.15.44:0/1738942224 learned_addr learned my addr 172.21.15.44:0/1738942224 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7eff9b7fe640 1 --2- 172.21.15.44:0/1738942224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff9c0cd9d0 0x7eff9c0cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fc125f67640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc1180e1290 0x7fc11803a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7f50a1ffb640 1 -- 172.21.15.44:0/2845077386 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50980315a0 con 0x7f50a406e090 2024-04-02T13:20:45.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7f50a1ffb640 1 -- 172.21.15.44:0/2845077386 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5098037070 con 0x7f50a406e090 2024-04-02T13:20:45.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7f50a1ffb640 1 -- 172.21.15.44:0/2845077386 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5098031960 con 0x7f50a406e090 2024-04-02T13:20:45.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7eff99ffb640 1 -- 172.21.15.44:0/1738942224 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1159395831 0 0) 0x7eff9c0b3000 con 0x7eff9c0b4a70 2024-04-02T13:20:45.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7eff99ffb640 1 -- 172.21.15.44:0/1738942224 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff88005730 con 0x7eff9c0b4a70 2024-04-02T13:20:45.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7f2da55d8640 1 -- 172.21.15.44:0/1621226292 shutdown_connections 2024-04-02T13:20:45.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7f2da55d8640 1 -- 172.21.15.44:0/1621226292 wait complete. 2024-04-02T13:20:45.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fbff37fe640 1 -- 172.21.15.44:0/3842184315 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff40b4d20 con 0x7fbff406e3e0 2024-04-02T13:20:45.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fbff37fe640 1 -- 172.21.15.44:0/3842184315 <== mon.1 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fbfe4001670 con 0x7fbff406e3e0 2024-04-02T13:20:45.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fbff37fe640 1 -- 172.21.15.44:0/3842184315 >> v1:172.21.15.69:6789/0 conn(0x7fbff406e3e0 legacy=0x7fbff406e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fbff37fe640 1 --2- 172.21.15.44:0/3842184315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0039b0 0x7fbfdc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fbff37fe640 1 --2- 172.21.15.44:0/3842184315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfdc004800 0x7fbfdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fbff37fe640 1 --2- 172.21.15.44:0/3842184315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfdc0051c0 0x7fbfdc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fbff37fe640 1 -- 172.21.15.44:0/3842184315 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbff40b5000 con 0x7fbfdc004800 2024-04-02T13:20:45.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fbff37fe640 1 -- 172.21.15.44:0/3842184315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbfdc00c160 con 0x7fbfdc0039b0 2024-04-02T13:20:45.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fbff37fe640 1 -- 172.21.15.44:0/3842184315 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbfdc00c360 con 0x7fbfdc0051c0 2024-04-02T13:20:45.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fc11f7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc1180e1290 0x7fc11803a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fc11f7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc1180e1290 0x7fc11803a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34256/0 (socket says 172.21.15.44:34256) 2024-04-02T13:20:45.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fbff9418640 1 --2- 172.21.15.44:0/3842184315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfdc0051c0 0x7fbfdc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fbff3fff640 1 --2- 172.21.15.44:0/3842184315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfdc004800 0x7fbfdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7fc11f7fe640 1 -- 172.21.15.44:0/1859689338 learned_addr learned my addr 172.21.15.44:0/1859689338 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7fc125f67640 1 -- 172.21.15.44:0/1859689338 --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc118054c50 con 0x7fc11803a6a0 2024-04-02T13:20:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7fc125f67640 1 -- 172.21.15.44:0/1859689338 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc118054f30 con 0x7fc118051640 2024-04-02T13:20:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7fc125f67640 1 -- 172.21.15.44:0/1859689338 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc118055860 con 0x7fc1180e1290 2024-04-02T13:20:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7fbff9418640 1 -- 172.21.15.44:0/3842184315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0039b0 msgr2=0x7fbfdc003dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T13:20:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7fbff9418640 1 --2- 172.21.15.44:0/3842184315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0039b0 0x7fbfdc003dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7fbff9418640 1 -- 172.21.15.44:0/3842184315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfdc004800 msgr2=0x7fbfdc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7fbff9418640 1 --2- 172.21.15.44:0/3842184315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfdc004800 0x7fbfdc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7fbff9418640 1 -- 172.21.15.44:0/3842184315 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff40b4d20 con 0x7fbfdc0051c0 2024-04-02T13:20:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f50a9582640 1 -- 172.21.15.44:0/2845077386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50a406e090 msgr2=0x7f50a40c54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f50a9582640 1 --2- 172.21.15.44:0/2845077386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50a406e090 0x7f50a40c54e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f50a40b1da0 tx=0x7f509802b840 comp rx=0 tx=0).stop 2024-04-02T13:20:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7fbff3fff640 1 --2- 172.21.15.44:0/3842184315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfdc004800 0x7fbfdc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f58aa6db640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f58a41596b0 0x7f58a40b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f58aa6db640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58a40d0b20 0x7f58a40b3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f50a9582640 1 -- 172.21.15.44:0/2845077386 shutdown_connections 2024-04-02T13:20:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f50a9582640 1 --2- 172.21.15.44:0/2845077386 >> v2:172.21.15.69:3300/0 conn(0x7f50a40c5a20 0x7f50a41660b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f50a9582640 1 --2- 172.21.15.44:0/2845077386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50a406e090 0x7f50a40c54e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f50a9582640 1 --2- 172.21.15.44:0/2845077386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a4162660 0x7f50a40ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f50a9582640 1 -- 172.21.15.44:0/2845077386 >> 172.21.15.44:0/2845077386 conn(0x7f50a40b74f0 msgr2=0x7f50a40b7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7eff9b7fe640 1 -- 172.21.15.44:0/1738942224 >> v2:172.21.15.69:3300/0 conn(0x7eff9c070fc0 msgr2=0x7eff9c15b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7eff99ffb640 1 -- 172.21.15.44:0/1738942224 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2106985149 0 0) 0x7eff88005730 con 0x7eff9c0b4a70 2024-04-02T13:20:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.916+0000 7eff9affd640 1 --2- 172.21.15.44:0/1738942224 >> v2:172.21.15.69:3300/0 conn(0x7eff9c070fc0 0x7eff9c15b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7eff9b7fe640 1 --2- 172.21.15.44:0/1738942224 >> v2:172.21.15.69:3300/0 conn(0x7eff9c070fc0 0x7eff9c15b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7eff9b7fe640 1 -- 172.21.15.44:0/1738942224 >> v1:172.21.15.44:6789/0 conn(0x7eff9c0b4a70 legacy=0x7eff9c0713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7eff9b7fe640 1 -- 172.21.15.44:0/1738942224 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9c0b3880 con 0x7eff9c0cd9d0 2024-04-02T13:20:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f58a96d9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f58a41592d0 0x7f58a406eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57458/0 (socket says 172.21.15.44:57458) 2024-04-02T13:20:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f50a9582640 1 -- 172.21.15.44:0/2845077386 shutdown_connections 2024-04-02T13:20:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f58a9eda640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58a40d0b20 0x7f58a40b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7eff9affd640 1 --2- 172.21.15.44:0/1738942224 >> v2:172.21.15.69:3300/0 conn(0x7eff9c070fc0 0x7eff9c15b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f50a9582640 1 -- 172.21.15.44:0/2845077386 wait complete. 2024-04-02T13:20:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f58a9eda640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58a40d0b20 0x7f58a40b3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:42028/0 (socket says 172.21.15.44:42028) 2024-04-02T13:20:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f58a9eda640 1 -- 172.21.15.44:0/364161844 learned_addr learned my addr 172.21.15.44:0/364161844 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f58a8ed8640 1 --2- 172.21.15.44:0/364161844 >> v2:172.21.15.69:3300/0 conn(0x7f58a41596b0 0x7f58a40b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f58aa6db640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58a4162b10 con 0x7f58a41592d0 2024-04-02T13:20:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f58aa6db640 1 -- 172.21.15.44:0/364161844 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f58a4162df0 con 0x7f58a41596b0 2024-04-02T13:20:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.917+0000 7f58aa6db640 1 -- 172.21.15.44:0/364161844 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f58a41630d0 con 0x7f58a40d0b20 2024-04-02T13:20:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 518993776 0 0) 0x7fc118054c50 con 0x7fc11803a6a0 2024-04-02T13:20:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc110003520 con 0x7fc11803a6a0 2024-04-02T13:20:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7f58a8ed8640 1 -- 172.21.15.44:0/364161844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58a40d0b20 msgr2=0x7f58a40b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7f58a8ed8640 1 --2- 172.21.15.44:0/364161844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58a40d0b20 0x7f58a40b3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7f58a8ed8640 1 -- 172.21.15.44:0/364161844 >> v1:172.21.15.44:6789/0 conn(0x7f58a41592d0 legacy=0x7f58a406eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7f589bfff640 1 -- 172.21.15.44:0/364161844 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 823093980 0 0) 0x7f58a4162b10 con 0x7f58a41592d0 2024-04-02T13:20:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7f58a8ed8640 1 -- 172.21.15.44:0/364161844 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a41633b0 con 0x7f58a41596b0 2024-04-02T13:20:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fbff9418640 1 -- 172.21.15.44:0/3842184315 reap_dead start 2024-04-02T13:20:45.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7f2da55d8640 1 Processor -- start 2024-04-02T13:20:45.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7f2da55d8640 1 -- start start 2024-04-02T13:20:45.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 357095978 0 0) 0x7fc110003520 con 0x7fc11803a6a0 2024-04-02T13:20:45.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 >> v2:172.21.15.69:3300/0 conn(0x7fc1180e1290 msgr2=0x7fc11803a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 --2- 172.21.15.44:0/1859689338 >> v2:172.21.15.69:3300/0 conn(0x7fc1180e1290 0x7fc11803a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 >> v1:172.21.15.69:6789/0 conn(0x7fc118051640 legacy=0x7fc118009db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc118055210 con 0x7fc11803a6a0 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fc10c001670 con 0x7fc11803a6a0 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 >> v1:172.21.15.44:6789/0 conn(0x7fc11803a6a0 legacy=0x7fc118055cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 --2- 172.21.15.44:0/1859689338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1100039b0 0x7fc110003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 --2- 172.21.15.44:0/1859689338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc110004800 0x7fc110004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 --2- 172.21.15.44:0/1859689338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1100051c0 0x7fc11000fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc118055860 con 0x7fc1100039b0 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc1100059c0 con 0x7fc1100051c0 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc110005bc0 con 0x7fc110004800 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fc11effd640 1 --2- 172.21.15.44:0/1859689338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc110004800 0x7fc110004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fc11f7fe640 1 --2- 172.21.15.44:0/1859689338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1100039b0 0x7fc110003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fc11ffff640 1 --2- 172.21.15.44:0/1859689338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1100051c0 0x7fc11000fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7f9929740640 1 Processor -- start 2024-04-02T13:20:45.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.918+0000 7f9929740640 1 -- start start 2024-04-02T13:20:45.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fd5361d9640 1 --2- 172.21.15.44:0/236897134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530162800 0x7fd51c00b9c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd52800a7a0 tx=0x7fd51c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aebc2bcaccdc15f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fc11f7fe640 1 -- 172.21.15.44:0/1859689338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc110004800 msgr2=0x7fc110004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fc11f7fe640 1 --2- 172.21.15.44:0/1859689338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc110004800 0x7fc110004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fc11f7fe640 1 -- 172.21.15.44:0/1859689338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1100051c0 msgr2=0x7fc11000fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fc11f7fe640 1 --2- 172.21.15.44:0/1859689338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1100051c0 0x7fc11000fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fc11f7fe640 1 -- 172.21.15.44:0/1859689338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc110003520 con 0x7fc1100039b0 2024-04-02T13:20:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fc11ffff640 1 -- 172.21.15.44:0/1859689338 reap_dead start 2024-04-02T13:20:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7eff9b7fe640 1 --2- 172.21.15.44:0/1738942224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff9c0cd9d0 0x7eff9c0cde60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7eff9400a8c0 tx=0x7eff9402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bec7dae26ac9fddb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7f58a8ed8640 1 --2- 172.21.15.44:0/364161844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58a41596b0 0x7f58a40b3450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f58a00097a0 tx=0x7f58a002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21447e8e2a8fe478 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7f50a9582640 1 Processor -- start 2024-04-02T13:20:45.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52800a3f0 con 0x7fd530162800 2024-04-02T13:20:45.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fbff9418640 1 --2- 172.21.15.44:0/3842184315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfdc0051c0 0x7fbfdc00b9c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009ac0 tx=0x7fbfdc005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25a308234e4aed05 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7f50a9582640 1 -- start start 2024-04-02T13:20:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7f9929740640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f99240d2750 0x7f99240d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7f9929740640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f992406d4e0 con 0x7f99240d0990 2024-04-02T13:20:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f9929740640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99240cee00 con 0x7f99240b34c0 2024-04-02T13:20:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f9929740640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f99240d1330 con 0x7f99240d2750 2024-04-02T13:20:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7f2da55d8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d98034b10 0x7f2d980e4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7f2da55d8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d98035040 0x7f2d9804f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7f2da55d8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d98048be0 0x7f2d980e8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7f2da55d8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9803b520 con 0x7f2d98034b10 2024-04-02T13:20:45.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7f2da55d8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2d980e7b00 con 0x7f2d98035040 2024-04-02T13:20:45.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7f2da55d8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2d9803b800 con 0x7f2d98048be0 2024-04-02T13:20:45.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f2da4dd7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d98048be0 0x7f2d980e8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f2da4dd7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d98048be0 0x7f2d980e8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:42046/0 (socket says 172.21.15.44:42046) 2024-04-02T13:20:45.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f2da4dd7640 1 -- 172.21.15.44:0/920157584 learned_addr learned my addr 172.21.15.44:0/920157584 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f2d9f7fe640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d98035040 0x7f2d9804f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f50a9582640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a406e090 0x7f50a4157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f50a9582640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50a40c5a20 0x7f50a4157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f50a9582640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50a4162660 0x7f50a40d0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fc11f7fe640 1 --2- 172.21.15.44:0/1859689338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1100039b0 0x7fc110003dd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc114009b10 tx=0x7fc110005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c38fa01a8675303f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7eff99ffb640 1 -- 172.21.15.44:0/1738942224 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94038070 con 0x7eff9c0cd9d0 2024-04-02T13:20:45.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7eff99ffb640 1 -- 172.21.15.44:0/1738942224 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff94033440 con 0x7eff9c0cd9d0 2024-04-02T13:20:45.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7eff9bfff640 1 -- 172.21.15.44:0/1738942224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff9c0cd9d0 msgr2=0x7eff9c0cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7eff9bfff640 1 --2- 172.21.15.44:0/1738942224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff9c0cd9d0 0x7eff9c0cde60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7eff9400a8c0 tx=0x7eff9402b7a0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd528034630 con 0x7fd530162800 2024-04-02T13:20:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.919+0000 7fd5349d6640 1 -- 172.21.15.44:0/236897134 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd528034a60 con 0x7fd530162800 2024-04-02T13:20:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7fd537c3c640 1 -- 172.21.15.44:0/236897134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530162800 msgr2=0x7fd51c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7fd537c3c640 1 --2- 172.21.15.44:0/236897134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530162800 0x7fd51c00b9c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd52800a7a0 tx=0x7fd51c005830 comp rx=0 tx=0).stop 2024-04-02T13:20:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7fd5361d9640 1 -- 172.21.15.44:0/236897134 reap_dead start 2024-04-02T13:20:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fd537c3c640 1 -- 172.21.15.44:0/236897134 shutdown_connections 2024-04-02T13:20:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fd537c3c640 1 -- 172.21.15.44:0/236897134 >> 172.21.15.44:0/236897134 conn(0x7fd5300cfcd0 msgr2=0x7fd5300d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f99227fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f99240d0990 0x7f99240d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57466/0 (socket says 172.21.15.44:57466) 2024-04-02T13:20:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f99227fc640 1 -- 172.21.15.44:0/1039284267 learned_addr learned my addr 172.21.15.44:0/1039284267 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f99237fe640 1 --2- 172.21.15.44:0/1039284267 >> v2:172.21.15.44:3300/0 conn(0x7f99240d2750 0x7f99240d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f99237fe640 1 -- 172.21.15.44:0/1039284267 >> v1:172.21.15.69:6789/0 conn(0x7f99240b34c0 legacy=0x7f99240cfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f589bfff640 1 -- 172.21.15.44:0/364161844 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a0035020 con 0x7f58a41596b0 2024-04-02T13:20:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f589bfff640 1 -- 172.21.15.44:0/364161844 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58a0031810 con 0x7f58a41596b0 2024-04-02T13:20:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f589bfff640 1 -- 172.21.15.44:0/364161844 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a0031be0 con 0x7f58a41596b0 2024-04-02T13:20:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f58aa6db640 1 -- 172.21.15.44:0/364161844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58a41596b0 msgr2=0x7f58a40b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f58aa6db640 1 --2- 172.21.15.44:0/364161844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58a41596b0 0x7f58a40b3450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f58a00097a0 tx=0x7f58a002b8b0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7eff9bfff640 1 -- 172.21.15.44:0/1738942224 shutdown_connections 2024-04-02T13:20:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7eff9bfff640 1 --2- 172.21.15.44:0/1738942224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff9c0cd9d0 0x7eff9c0cde60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7eff9bfff640 1 --2- 172.21.15.44:0/1738942224 >> v2:172.21.15.69:3300/0 conn(0x7eff9c070fc0 0x7eff9c15b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7eff9bfff640 1 -- 172.21.15.44:0/1738942224 >> 172.21.15.44:0/1738942224 conn(0x7eff9c0b1860 msgr2=0x7eff9c0b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7fbff37fe640 1 -- 172.21.15.44:0/3842184315 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec007990 con 0x7fbfdc0051c0 2024-04-02T13:20:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7fbff37fe640 1 -- 172.21.15.44:0/3842184315 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe8004020 con 0x7fbfdc0051c0 2024-04-02T13:20:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fbffae7b640 1 -- 172.21.15.44:0/3842184315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfdc0051c0 msgr2=0x7fbfdc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fbffae7b640 1 --2- 172.21.15.44:0/3842184315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfdc0051c0 0x7fbfdc00b9c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009ac0 tx=0x7fbfdc005830 comp rx=0 tx=0).stop 2024-04-02T13:20:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f50a9582640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f50a4166770 con 0x7f50a4162660 2024-04-02T13:20:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f2d9ffff640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d98034b10 0x7f2d980e4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f2da4dd7640 1 -- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d98035040 msgr2=0x7f2d9804f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f50a9582640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f50a41665f0 con 0x7f50a40c5a20 2024-04-02T13:20:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fbffae7b640 1 -- 172.21.15.44:0/3842184315 shutdown_connections 2024-04-02T13:20:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fbffae7b640 1 --2- 172.21.15.44:0/3842184315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfdc0051c0 0x7fbfdc00b9c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f58aa6db640 1 -- 172.21.15.44:0/364161844 shutdown_connections 2024-04-02T13:20:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f2da4dd7640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d98035040 0x7f2d9804f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.920+0000 7f2da4dd7640 1 -- 172.21.15.44:0/920157584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d98034b10 msgr2=0x7f2d980e4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f2da4dd7640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d98034b10 0x7f2d980e4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f2da4dd7640 1 -- 172.21.15.44:0/920157584 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d980e8130 con 0x7f2d98048be0 2024-04-02T13:20:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f2d9ffff640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d98034b10 0x7f2d980e4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:45.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f50a9582640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f50a41668f0 con 0x7f50a406e090 2024-04-02T13:20:45.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f99237fe640 1 -- 172.21.15.44:0/1039284267 >> v1:172.21.15.44:6789/0 conn(0x7f99240d0990 legacy=0x7f99240d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f99237fe640 1 -- 172.21.15.44:0/1039284267 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992406d200 con 0x7f99240d2750 2024-04-02T13:20:45.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f9921ffb640 1 -- 172.21.15.44:0/1039284267 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3579605822 0 0) 0x7f99240cee00 con 0x7f99240b34c0 2024-04-02T13:20:45.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f58aa6db640 1 --2- 172.21.15.44:0/364161844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58a40d0b20 0x7f58a40b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc114039070 con 0x7fc1100039b0 2024-04-02T13:20:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc11402c7b0 con 0x7fc1100039b0 2024-04-02T13:20:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fc11e7fc640 1 -- 172.21.15.44:0/1859689338 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11400f040 con 0x7fc1100039b0 2024-04-02T13:20:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fbffae7b640 1 -- 172.21.15.44:0/3842184315 >> 172.21.15.44:0/3842184315 conn(0x7fbff40cfcd0 msgr2=0x7fbff40d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f58aa6db640 1 --2- 172.21.15.44:0/364161844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58a41596b0 0x7f58a40b3450 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f58aa6db640 1 -- 172.21.15.44:0/364161844 >> 172.21.15.44:0/364161844 conn(0x7f58a406b4d0 msgr2=0x7f58a406b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fd537c3c640 1 -- 172.21.15.44:0/236897134 shutdown_connections 2024-04-02T13:20:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fbffae7b640 1 -- 172.21.15.44:0/3842184315 shutdown_connections 2024-04-02T13:20:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f50a3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50a4162660 0x7f50a40d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7eff9bfff640 1 -- 172.21.15.44:0/1738942224 shutdown_connections 2024-04-02T13:20:45.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f50a3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50a4162660 0x7f50a40d0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54844/0 (socket says 172.21.15.44:54844) 2024-04-02T13:20:45.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7eff9bfff640 1 -- 172.21.15.44:0/1738942224 wait complete. 2024-04-02T13:20:45.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fd537c3c640 1 -- 172.21.15.44:0/236897134 wait complete. 2024-04-02T13:20:45.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7fbffae7b640 1 -- 172.21.15.44:0/3842184315 wait complete. 2024-04-02T13:20:45.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f2da4dd7640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d98048be0 0x7f2d980e8ef0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2da0050db0 tx=0x7f2da0075660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f2d9cff9640 1 -- 172.21.15.44:0/920157584 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da006cc40 con 0x7f2d98048be0 2024-04-02T13:20:45.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f2d9cff9640 1 -- 172.21.15.44:0/920157584 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da006ce00 con 0x7f2d98048be0 2024-04-02T13:20:45.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f2d9cff9640 1 -- 172.21.15.44:0/920157584 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da00731d0 con 0x7f2d98048be0 2024-04-02T13:20:45.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f50a3fff640 1 -- 172.21.15.44:0/874288427 learned_addr learned my addr 172.21.15.44:0/874288427 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.921+0000 7f50a27fc640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50a40c5a20 0x7f50a4157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f50a3fff640 1 -- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a406e090 msgr2=0x7f50a4157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f50a2ffd640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a406e090 0x7f50a4157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f50a3fff640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a406e090 0x7f50a4157440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f50a3fff640 1 -- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50a40c5a20 msgr2=0x7f50a4157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f50a3fff640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50a40c5a20 0x7f50a4157980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f50a3fff640 1 -- 172.21.15.44:0/874288427 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a40cbc50 con 0x7f50a4162660 2024-04-02T13:20:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7fc125f67640 1 -- 172.21.15.44:0/1859689338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1100039b0 msgr2=0x7fc110003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7fc125f67640 1 --2- 172.21.15.44:0/1859689338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1100039b0 0x7fc110003dd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc114009b10 tx=0x7fc110005830 comp rx=0 tx=0).stop 2024-04-02T13:20:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f99237fe640 1 --2- 172.21.15.44:0/1039284267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99240d2750 0x7f99240d0df0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f99140097a0 tx=0x7f991402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d2843e54874016 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7fc125f67640 1 -- 172.21.15.44:0/1859689338 shutdown_connections 2024-04-02T13:20:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7fc125f67640 1 --2- 172.21.15.44:0/1859689338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1100039b0 0x7fc110003dd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7fc125f67640 1 -- 172.21.15.44:0/1859689338 >> 172.21.15.44:0/1859689338 conn(0x7fc118056cc0 msgr2=0x7fc1180570d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f9921ffb640 1 -- 172.21.15.44:0/1039284267 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9914034070 con 0x7f99240d2750 2024-04-02T13:20:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f2da55d8640 1 -- 172.21.15.44:0/920157584 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d980e8410 con 0x7f2d98048be0 2024-04-02T13:20:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f2da55d8640 1 -- 172.21.15.44:0/920157584 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d980e5010 con 0x7f2d98048be0 2024-04-02T13:20:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f9921ffb640 1 -- 172.21.15.44:0/1039284267 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9914030610 con 0x7f99240d2750 2024-04-02T13:20:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.922+0000 7f50a3fff640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50a4162660 0x7f50a40d0540 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f509c007e70 tx=0x7f509c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7f508b7fe640 1 -- 172.21.15.44:0/874288427 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509c014620 con 0x7f50a4162660 2024-04-02T13:20:45.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7f508b7fe640 1 -- 172.21.15.44:0/874288427 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f509c019070 con 0x7f50a4162660 2024-04-02T13:20:45.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7f508b7fe640 1 -- 172.21.15.44:0/874288427 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509c014a00 con 0x7f50a4162660 2024-04-02T13:20:45.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7fd537c3c640 1 Processor -- start 2024-04-02T13:20:45.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7f50a9582640 1 -- 172.21.15.44:0/874288427 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50a40cbf30 con 0x7f50a4162660 2024-04-02T13:20:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7fd537c3c640 1 -- start start 2024-04-02T13:20:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7f50a27fc640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50a40c5a20 0x7f50a4157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7f50a9582640 1 -- 172.21.15.44:0/874288427 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a40c6320 con 0x7f50a4162660 2024-04-02T13:20:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7f9921ffb640 1 -- 172.21.15.44:0/1039284267 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99140309e0 con 0x7f99240d2750 2024-04-02T13:20:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7fc125f67640 1 -- 172.21.15.44:0/1859689338 shutdown_connections 2024-04-02T13:20:45.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7eff9bfff640 1 Processor -- start 2024-04-02T13:20:45.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7fc125f67640 1 -- 172.21.15.44:0/1859689338 wait complete. 2024-04-02T13:20:45.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7eff9bfff640 1 -- start start 2024-04-02T13:20:45.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7f9929740640 1 -- 172.21.15.44:0/1039284267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99240d2750 msgr2=0x7f99240d0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.923+0000 7f9929740640 1 --2- 172.21.15.44:0/1039284267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99240d2750 0x7f99240d0df0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f99140097a0 tx=0x7f991402b960 comp rx=0 tx=0).stop 2024-04-02T13:20:45.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7f9929740640 1 -- 172.21.15.44:0/1039284267 shutdown_connections 2024-04-02T13:20:45.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7f9929740640 1 --2- 172.21.15.44:0/1039284267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99240d2750 0x7f99240d0df0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7f9929740640 1 -- 172.21.15.44:0/1039284267 >> 172.21.15.44:0/1039284267 conn(0x7f992409d470 msgr2=0x7f99240d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:45.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7fbffae7b640 1 Processor -- start 2024-04-02T13:20:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7f9929740640 1 -- 172.21.15.44:0/1039284267 shutdown_connections 2024-04-02T13:20:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7fd537c3c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd530162800 0x7fd53009aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7fd537c3c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd53009af60 0x7fd53014fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7fd537c3c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530150420 0x7fd5301539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7fd537c3c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd53006dae0 con 0x7fd530162800 2024-04-02T13:20:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7fd537c3c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd530167190 con 0x7fd53009af60 2024-04-02T13:20:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7fd537c3c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd53006d2d0 con 0x7fd530150420 2024-04-02T13:20:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7fd5361d9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530150420 0x7fd5301539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7fd5361d9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530150420 0x7fd5301539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:42068/0 (socket says 172.21.15.44:42068) 2024-04-02T13:20:45.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7fd5361d9640 1 -- 172.21.15.44:0/1614777386 learned_addr learned my addr 172.21.15.44:0/1614777386 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7eff9bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff9c070fc0 0x7eff9c14e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7f2d9cff9640 1 -- 172.21.15.44:0/920157584 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2da0073390 con 0x7f2d98048be0 2024-04-02T13:20:45.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7fd5359d8640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd530162800 0x7fd53009aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7fd5351d7640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd53009af60 0x7fd53014fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7fd5361d9640 1 -- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd53009af60 msgr2=0x7fd53014fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7fd5361d9640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd53009af60 0x7fd53014fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7fd5361d9640 1 -- 172.21.15.44:0/1614777386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd530162800 msgr2=0x7fd53009aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7fd5361d9640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd530162800 0x7fd53009aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7fd5361d9640 1 -- 172.21.15.44:0/1614777386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53009ba30 con 0x7fd530150420 2024-04-02T13:20:45.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7fd5359d8640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd530162800 0x7fd53009aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:45.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7fbffae7b640 1 -- start start 2024-04-02T13:20:45.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7f2d9cff9640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2d8406c9f0 0x7f2d8406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7f9929740640 1 -- 172.21.15.44:0/1039284267 wait complete. 2024-04-02T13:20:45.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7f2d9cff9640 1 -- 172.21.15.44:0/920157584 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2da0106e30 con 0x7f2d98048be0 2024-04-02T13:20:45.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7f58aa6db640 1 -- 172.21.15.44:0/364161844 shutdown_connections 2024-04-02T13:20:45.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7fd5361d9640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530150420 0x7fd5301539f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd52802cbc0 tx=0x7fd5300d09f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7f2d9ffff640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2d8406c9f0 0x7f2d8406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7fd5267fc640 1 -- 172.21.15.44:0/1614777386 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd528039040 con 0x7fd530150420 2024-04-02T13:20:45.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7fd5267fc640 1 -- 172.21.15.44:0/1614777386 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd528043070 con 0x7fd530150420 2024-04-02T13:20:45.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7fd5267fc640 1 -- 172.21.15.44:0/1614777386 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd528001f30 con 0x7fd530150420 2024-04-02T13:20:45.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7fd537c3c640 1 -- 172.21.15.44:0/1614777386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd53009b470 con 0x7fd530150420 2024-04-02T13:20:45.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.924+0000 7f50a9582640 1 -- 172.21.15.44:0/874288427 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50a4068c00 con 0x7f50a4162660 2024-04-02T13:20:45.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7f508b7fe640 1 -- 172.21.15.44:0/874288427 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f509c014ba0 con 0x7f50a4162660 2024-04-02T13:20:45.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7fd537c3c640 1 -- 172.21.15.44:0/1614777386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5301531f0 con 0x7fd530150420 2024-04-02T13:20:45.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7fc125f67640 1 Processor -- start 2024-04-02T13:20:45.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7f9929740640 1 Processor -- start 2024-04-02T13:20:45.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7f508b7fe640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f509006c9c0 0x7f509006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7fbffae7b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0051c0 0x7fbff409d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7fbffae7b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbff4141f10 0x7fbff40b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7fbffae7b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff4142390 0x7fbff40b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7fbffae7b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbff406d7b0 con 0x7fbff4142390 2024-04-02T13:20:45.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7fbffae7b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbff406d4d0 con 0x7fbfdc0051c0 2024-04-02T13:20:45.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7fbffae7b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbff415b930 con 0x7fbff4141f10 2024-04-02T13:20:45.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7fbff8c17640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0051c0 0x7fbff409d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7f58aa6db640 1 -- 172.21.15.44:0/364161844 wait complete. 2024-04-02T13:20:45.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7f50a2ffd640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f509006c9c0 0x7f509006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7fbff8c17640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0051c0 0x7fbff409d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34318/0 (socket says 172.21.15.44:34318) 2024-04-02T13:20:45.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7fbff8c17640 1 -- 172.21.15.44:0/2102474073 learned_addr learned my addr 172.21.15.44:0/2102474073 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7eff9bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff9c0b4a70 0x7eff9c157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7eff9bfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff9c0cd9d0 0x7eff9c157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7eff9bfff640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c0cc7c0 con 0x7eff9c0b4a70 2024-04-02T13:20:45.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7eff9bfff640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c158350 con 0x7eff9c070fc0 2024-04-02T13:20:45.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7eff9bfff640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7eff9c0cc940 con 0x7eff9c0cd9d0 2024-04-02T13:20:45.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.925+0000 7eff9affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff9c070fc0 0x7eff9c14e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7eff9a7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff9c0b4a70 0x7eff9c157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7eff9a7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff9c0b4a70 0x7eff9c157140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54872/0 (socket says 172.21.15.44:54872) 2024-04-02T13:20:45.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7eff9a7fc640 1 -- 172.21.15.44:0/2003295157 learned_addr learned my addr 172.21.15.44:0/2003295157 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7eff9a7fc640 1 -- 172.21.15.44:0/2003295157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff9c0cd9d0 msgr2=0x7eff9c157680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T13:20:45.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7eff9a7fc640 1 --2- 172.21.15.44:0/2003295157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff9c0cd9d0 0x7eff9c157680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7eff9a7fc640 1 -- 172.21.15.44:0/2003295157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff9c070fc0 msgr2=0x7eff9c14e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7eff9a7fc640 1 --2- 172.21.15.44:0/2003295157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff9c070fc0 0x7eff9c14e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7eff9a7fc640 1 -- 172.21.15.44:0/2003295157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9c14eeb0 con 0x7eff9c0b4a70 2024-04-02T13:20:45.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7eff9a7fc640 1 --2- 172.21.15.44:0/2003295157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff9c0b4a70 0x7eff9c157140 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0021a0 tx=0x7eff8c00a4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7f508b7fe640 1 -- 172.21.15.44:0/874288427 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f509c08c660 con 0x7f50a4162660 2024-04-02T13:20:45.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7f50a2ffd640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f509006c9c0 0x7f509006ee80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f50940096f0 tx=0x7f5094008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7f2da55d8640 1 -- 172.21.15.44:0/920157584 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d98005710 con 0x7f2d98048be0 2024-04-02T13:20:45.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7eff937fe640 1 -- 172.21.15.44:0/2003295157 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c012600 con 0x7eff9c0b4a70 2024-04-02T13:20:45.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7fbff8c17640 1 -- 172.21.15.44:0/2102474073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbff4141f10 msgr2=0x7fbff40b0280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T13:20:45.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7fbff8c17640 1 --2- 172.21.15.44:0/2102474073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbff4141f10 0x7fbff40b0280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7fbff8c17640 1 -- 172.21.15.44:0/2102474073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff4142390 msgr2=0x7fbff40b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7fbff8c17640 1 --2- 172.21.15.44:0/2102474073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff4142390 0x7fbff40b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7fbff8c17640 1 -- 172.21.15.44:0/2102474073 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff41f83f0 con 0x7fbfdc0051c0 2024-04-02T13:20:45.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7eff937fe640 1 -- 172.21.15.44:0/2003295157 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff8c0127a0 con 0x7eff9c0b4a70 2024-04-02T13:20:45.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7eff9bfff640 1 -- 172.21.15.44:0/2003295157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff9c06e070 con 0x7eff9c0b4a70 2024-04-02T13:20:45.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7eff9bfff640 1 -- 172.21.15.44:0/2003295157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff9c06e310 con 0x7eff9c0b4a70 2024-04-02T13:20:45.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7fbff8c17640 1 --2- 172.21.15.44:0/2102474073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0051c0 0x7fbff409d410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400cac0 tx=0x7fbfe400cf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7f2d9ffff640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2d8406c9f0 0x7f2d8406eeb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2d94036000 tx=0x7f2d9403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7fbff17fa640 1 -- 172.21.15.44:0/2102474073 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4017070 con 0x7fbfdc0051c0 2024-04-02T13:20:45.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7fbffae7b640 1 -- 172.21.15.44:0/2102474073 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff4154e00 con 0x7fbfdc0051c0 2024-04-02T13:20:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7fc125f67640 1 -- start start 2024-04-02T13:20:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7eff937fe640 1 -- 172.21.15.44:0/2003295157 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c012b90 con 0x7eff9c0b4a70 2024-04-02T13:20:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7fd537c3c640 1 -- 172.21.15.44:0/1614777386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd518005290 con 0x7fd530150420 2024-04-02T13:20:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7fd5267fc640 1 -- 172.21.15.44:0/1614777386 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd5280349e0 con 0x7fd530150420 2024-04-02T13:20:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7fd5267fc640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd51406c9f0 0x7fd51406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fd5267fc640 1 -- 172.21.15.44:0/1614777386 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd5280adcb0 con 0x7fd530150420 2024-04-02T13:20:45.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7fbffae7b640 1 -- 172.21.15.44:0/2102474073 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff4151f20 con 0x7fbfdc0051c0 2024-04-02T13:20:45.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.926+0000 7f9929740640 1 -- start start 2024-04-02T13:20:45.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7f9929740640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99240b34c0 0x7f9924158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7f9929740640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99240d0990 0x7f99240cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7f9929740640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99240d2750 0x7f99240b93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7f9929740640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f99240cf380 con 0x7f99240b34c0 2024-04-02T13:20:45.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7f9929740640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f99240cf6f0 con 0x7f99240d2750 2024-04-02T13:20:45.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7f9929740640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f99240b38a0 con 0x7f99240d0990 2024-04-02T13:20:45.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7f9922ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99240b34c0 0x7f9924158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7f9922ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99240b34c0 0x7f9924158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54890/0 (socket says 172.21.15.44:54890) 2024-04-02T13:20:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.927+0000 7f9922ffd640 1 -- 172.21.15.44:0/1804507819 learned_addr learned my addr 172.21.15.44:0/1804507819 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7f99237fe640 1 --2- 172.21.15.44:0/1804507819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99240d2750 0x7f99240b93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7f9922ffd640 1 -- 172.21.15.44:0/1804507819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99240d0990 msgr2=0x7f99240cbbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T13:20:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7f9922ffd640 1 --2- 172.21.15.44:0/1804507819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99240d0990 0x7f99240cbbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7f9922ffd640 1 -- 172.21.15.44:0/1804507819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99240d2750 msgr2=0x7f99240b93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7f9922ffd640 1 --2- 172.21.15.44:0/1804507819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99240d2750 0x7f99240b93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7f9922ffd640 1 -- 172.21.15.44:0/1804507819 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99240cc6c0 con 0x7f99240b34c0 2024-04-02T13:20:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.928+0000 7f9922ffd640 1 --2- 172.21.15.44:0/1804507819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99240b34c0 0x7f9924158690 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f991c00c890 tx=0x7f991c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7f58aa6db640 1 Processor -- start 2024-04-02T13:20:45.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fc125f67640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1100039b0 0x7fc1180e0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fc125f67640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1180e2030 0x7fc1180e3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fc125f67640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc11804c700 0x7fc11804cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fc125f67640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc118054a10 con 0x7fc11804c700 2024-04-02T13:20:45.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fc125f67640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc118054890 con 0x7fc1180e2030 2024-04-02T13:20:45.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fc125f67640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc11803a160 con 0x7fc1100039b0 2024-04-02T13:20:45.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fc11f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1100039b0 0x7fc1180e0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fc11f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1100039b0 0x7fc1180e0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:42094/0 (socket says 172.21.15.44:42094) 2024-04-02T13:20:45.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fc11f7fe640 1 -- 172.21.15.44:0/1908095311 learned_addr learned my addr 172.21.15.44:0/1908095311 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fc11ffff640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc11804c700 0x7fc11804cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fc11effd640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1180e2030 0x7fc1180e3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7eff9bfff640 1 -- 172.21.15.44:0/2003295157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff9c04f8d0 con 0x7eff9c0b4a70 2024-04-02T13:20:45.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fd5359d8640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd51406c9f0 0x7fd51406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fbff17fa640 1 -- 172.21.15.44:0/2102474073 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe4005d00 con 0x7fbfdc0051c0 2024-04-02T13:20:45.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7fbff17fa640 1 -- 172.21.15.44:0/2102474073 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4007290 con 0x7fbfdc0051c0 2024-04-02T13:20:45.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.929+0000 7f58aa6db640 1 -- start start 2024-04-02T13:20:45.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7fc11f7fe640 1 -- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1180e2030 msgr2=0x7fc1180e3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7fc11f7fe640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1180e2030 0x7fc1180e3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7fc11f7fe640 1 -- 172.21.15.44:0/1908095311 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc11804c700 msgr2=0x7fc11804cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7fc11f7fe640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc11804c700 0x7fc11804cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7fc11f7fe640 1 -- 172.21.15.44:0/1908095311 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1180e25d0 con 0x7fc1100039b0 2024-04-02T13:20:45.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7fbffae7b640 1 -- 172.21.15.44:0/2102474073 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff40b0d00 con 0x7fbfdc0051c0 2024-04-02T13:20:45.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7fc11f7fe640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1100039b0 0x7fc1180e0260 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc114009c30 tx=0x7fc114004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7fd5359d8640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd51406c9f0 0x7fd51406eeb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd5300cf6d0 tx=0x7fd520008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7f58aa6db640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58a40d0b20 0x7f58a41f8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7f58aa6db640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58a41592d0 0x7f58a414c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7f58aa6db640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58a41596b0 0x7f58a415a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7f58aa6db640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f58a4166410 con 0x7f58a40d0b20 2024-04-02T13:20:45.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7f58a96d9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58a40d0b20 0x7f58a41f8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7f58aa6db640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58a4166290 con 0x7f58a41592d0 2024-04-02T13:20:45.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7f58a8ed8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58a41592d0 0x7f58a414c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7f58a96d9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58a40d0b20 0x7f58a41f8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54900/0 (socket says 172.21.15.44:54900) 2024-04-02T13:20:45.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7f58a96d9640 1 -- 172.21.15.44:0/3578372465 learned_addr learned my addr 172.21.15.44:0/3578372465 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:45.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7fbff17fa640 1 -- 172.21.15.44:0/2102474073 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbfe401a070 con 0x7fbfdc0051c0 2024-04-02T13:20:45.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7f58aa6db640 1 -- 172.21.15.44:0/3578372465 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f58a4166590 con 0x7f58a41596b0 2024-04-02T13:20:45.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.930+0000 7f58a9eda640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58a41596b0 0x7f58a415a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.931+0000 7f58a96d9640 1 -- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58a41596b0 msgr2=0x7f58a415a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.931+0000 7f58a96d9640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58a41596b0 0x7f58a415a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.931+0000 7f58a96d9640 1 -- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58a41592d0 msgr2=0x7f58a414c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:45.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.931+0000 7f58a96d9640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58a41592d0 0x7f58a414c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:45.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.931+0000 7f58a96d9640 1 -- 172.21.15.44:0/3578372465 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a414ca60 con 0x7f58a40d0b20 2024-04-02T13:20:45.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.931+0000 7f58a8ed8640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58a41592d0 0x7f58a414c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:45.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.931+0000 7fbff17fa640 1 --2- 172.21.15.44:0/2102474073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbfe006c940 0x7fbfe006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.931+0000 7fbff17fa640 1 -- 172.21.15.44:0/2102474073 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbfe408b450 con 0x7fbfdc0051c0 2024-04-02T13:20:45.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.931+0000 7f58a96d9640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58a40d0b20 0x7f58a41f8200 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f589400c4a0 tx=0x7f589400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.931+0000 7fbff3fff640 1 --2- 172.21.15.44:0/2102474073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbfe006c940 0x7fbfe006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.932+0000 7fbff3fff640 1 --2- 172.21.15.44:0/2102474073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbfe006c940 0x7fbfe006ee00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbfec0050e0 tx=0x7fbfec005050 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.933+0000 7f2d9cff9640 1 -- 172.21.15.44:0/920157584 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2da00d4f00 con 0x7f2d98048be0 2024-04-02T13:20:45.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.934+0000 7f508b7fe640 1 -- 172.21.15.44:0/874288427 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f509c05a7c0 con 0x7f50a4162660 2024-04-02T13:20:45.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.934+0000 7f5899ffb640 1 -- 172.21.15.44:0/3578372465 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5894018070 con 0x7f58a40d0b20 2024-04-02T13:20:45.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.934+0000 7f990b7fe640 1 -- 172.21.15.44:0/1804507819 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c003b90 con 0x7f99240b34c0 2024-04-02T13:20:45.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.934+0000 7f990b7fe640 1 -- 172.21.15.44:0/1804507819 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f991c003d30 con 0x7f99240b34c0 2024-04-02T13:20:45.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.934+0000 7f9929740640 1 -- 172.21.15.44:0/1804507819 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99240cc9a0 con 0x7f99240b34c0 2024-04-02T13:20:45.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.934+0000 7f58aa6db640 1 -- 172.21.15.44:0/3578372465 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a415e070 con 0x7f58a40d0b20 2024-04-02T13:20:45.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.934+0000 7f990b7fe640 1 -- 172.21.15.44:0/1804507819 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c00aa70 con 0x7f99240b34c0 2024-04-02T13:20:45.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.934+0000 7f58aa6db640 1 -- 172.21.15.44:0/3578372465 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a415e630 con 0x7f58a40d0b20 2024-04-02T13:20:45.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.934+0000 7f9929740640 1 -- 172.21.15.44:0/1804507819 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99240b7140 con 0x7f99240b34c0 2024-04-02T13:20:45.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.936+0000 7eff937fe640 1 -- 172.21.15.44:0/2003295157 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7eff8c025430 con 0x7eff9c0b4a70 2024-04-02T13:20:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.935+0000 7f5899ffb640 1 -- 172.21.15.44:0/3578372465 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58940074e0 con 0x7f58a40d0b20 2024-04-02T13:20:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.936+0000 7f5899ffb640 1 -- 172.21.15.44:0/3578372465 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5894004a20 con 0x7f58a40d0b20 2024-04-02T13:20:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.936+0000 7f9929740640 1 -- 172.21.15.44:0/1804507819 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99240684f0 con 0x7f99240b34c0 2024-04-02T13:20:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.936+0000 7f58aa6db640 1 -- 172.21.15.44:0/3578372465 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5884005290 con 0x7f58a40d0b20 2024-04-02T13:20:45.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.936+0000 7eff937fe640 1 --2- 172.21.15.44:0/2003295157 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff7c06c9c0 0x7eff7c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:45.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.936+0000 7eff937fe640 1 -- 172.21.15.44:0/2003295157 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7eff8c08d1c0 con 0x7eff9c0b4a70 2024-04-02T13:20:45.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.936+0000 7eff9affd640 1 --2- 172.21.15.44:0/2003295157 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff7c06c9c0 0x7eff7c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:45.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.937+0000 7eff9affd640 1 --2- 172.21.15.44:0/2003295157 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff7c06c9c0 0x7eff7c06ee80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7eff84006b10 tx=0x7eff84008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:45.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.940+0000 7eff937fe640 1 -- 172.21.15.44:0/2003295157 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7eff9c14eeb0 con 0x7eff9c0b4a70 2024-04-02T13:20:45.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.943+0000 7fd5267fc640 1 -- 172.21.15.44:0/1614777386 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd52807bdc0 con 0x7fd530150420 2024-04-02T13:20:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.942+0000 7f990b7fe640 1 -- 172.21.15.44:0/1804507819 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f991c00ac10 con 0x7f99240b34c0 2024-04-02T13:20:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.942+0000 7fbff17fa640 1 -- 172.21.15.44:0/2102474073 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbff40b0d00 con 0x7fbfdc0051c0 2024-04-02T13:20:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.943+0000 7f5899ffb640 1 -- 172.21.15.44:0/3578372465 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5894004bc0 con 0x7f58a40d0b20 2024-04-02T13:20:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.943+0000 7f5899ffb640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f588806c8f0 0x7f588806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.943+0000 7f990b7fe640 1 --2- 172.21.15.44:0/1804507819 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f991006c8f0 0x7f991006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.943+0000 7f5899ffb640 1 -- 172.21.15.44:0/3578372465 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f589408ba30 con 0x7f58a40d0b20 2024-04-02T13:20:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.943+0000 7f990b7fe640 1 -- 172.21.15.44:0/1804507819 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f991c08c960 con 0x7f99240b34c0 2024-04-02T13:20:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.943+0000 7f99227fc640 1 --2- 172.21.15.44:0/1804507819 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f991006c8f0 0x7f991006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.944+0000 7f58a8ed8640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f588806c8f0 0x7f588806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:46.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.944+0000 7fc10bfff640 1 -- 172.21.15.44:0/1908095311 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc114039030 con 0x7fc1100039b0 2024-04-02T13:20:46.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.944+0000 7fc10bfff640 1 -- 172.21.15.44:0/1908095311 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc114043070 con 0x7fc1100039b0 2024-04-02T13:20:46.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.944+0000 7fc10bfff640 1 -- 172.21.15.44:0/1908095311 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc114034ca0 con 0x7fc1100039b0 2024-04-02T13:20:46.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.944+0000 7fc125f67640 1 -- 172.21.15.44:0/1908095311 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1180e28b0 con 0x7fc1100039b0 2024-04-02T13:20:46.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.944+0000 7fc125f67640 1 -- 172.21.15.44:0/1908095311 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1180e2e70 con 0x7fc1100039b0 2024-04-02T13:20:46.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.945+0000 7f99227fc640 1 --2- 172.21.15.44:0/1804507819 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f991006c8f0 0x7f991006edb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f99240cd070 tx=0x7f9918007a40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:46.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.946+0000 7fc125f67640 1 -- 172.21.15.44:0/1908095311 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1180052b0 con 0x7fc1100039b0 2024-04-02T13:20:46.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.946+0000 7fc10bfff640 1 -- 172.21.15.44:0/1908095311 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc114048460 con 0x7fc1100039b0 2024-04-02T13:20:46.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.946+0000 7f58a8ed8640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f588806c8f0 0x7f588806edb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f58a0036000 tx=0x7f58a003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.947+0000 7fc10bfff640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc10406c9c0 0x7fc10406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.947+0000 7fc10bfff640 1 -- 172.21.15.44:0/1908095311 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc1140ad690 con 0x7fc1100039b0 2024-04-02T13:20:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.947+0000 7fc11effd640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc10406c9c0 0x7fc10406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.947+0000 7fc11effd640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc10406c9c0 0x7fc10406ee80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc11803c280 tx=0x7fc10c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.948+0000 7f990b7fe640 1 -- 172.21.15.44:0/1804507819 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f991c05ab80 con 0x7f99240b34c0 2024-04-02T13:20:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.951+0000 7fc10bfff640 1 -- 172.21.15.44:0/1908095311 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc11402ce20 con 0x7fc1100039b0 2024-04-02T13:20:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:45.958+0000 7f5899ffb640 1 -- 172.21.15.44:0/3578372465 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5894021070 con 0x7f58a40d0b20 2024-04-02T13:20:46.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.368+0000 7f2da55d8640 1 -- 172.21.15.44:0/920157584 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f2d980093b0 con 0x7f2d98048be0 2024-04-02T13:20:46.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.369+0000 7f2d9cff9640 1 -- 172.21.15.44:0/920157584 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2da00db0d0 con 0x7f2d98048be0 2024-04-02T13:20:46.372 INFO:teuthology.orchestra.run.smithi044.stdout:55834574855 2024-04-02T13:20:46.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.372+0000 7f2d8a7fc640 1 -- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2d8406c9f0 msgr2=0x7f2d8406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.372+0000 7f2d8a7fc640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2d8406c9f0 0x7f2d8406eeb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2d94036000 tx=0x7f2d9403b040 comp rx=0 tx=0).stop 2024-04-02T13:20:46.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.372+0000 7f2d8a7fc640 1 -- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d98048be0 msgr2=0x7f2d980e8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.372+0000 7f2d8a7fc640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d98048be0 0x7f2d980e8ef0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2da0050db0 tx=0x7f2da0075660 comp rx=0 tx=0).stop 2024-04-02T13:20:46.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.373+0000 7f2d8a7fc640 1 -- 172.21.15.44:0/920157584 shutdown_connections 2024-04-02T13:20:46.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.373+0000 7f2d8a7fc640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2d8406c9f0 0x7f2d8406eeb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.373+0000 7f2d8a7fc640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d98048be0 0x7f2d980e8ef0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.373+0000 7f2d8a7fc640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d98035040 0x7f2d9804f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.373+0000 7f2d8a7fc640 1 --2- 172.21.15.44:0/920157584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d98034b10 0x7f2d980e4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.373+0000 7f2d8a7fc640 1 -- 172.21.15.44:0/920157584 >> 172.21.15.44:0/920157584 conn(0x7f2d9801ef70 msgr2=0x7f2d98032860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:46.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.373+0000 7f2d8a7fc640 1 -- 172.21.15.44:0/920157584 shutdown_connections 2024-04-02T13:20:46.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.373+0000 7f2d8a7fc640 1 -- 172.21.15.44:0/920157584 wait complete. 2024-04-02T13:20:46.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.373+0000 7f50a9582640 1 -- 172.21.15.44:0/874288427 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f50a40ce4a0 con 0x7f50a4162660 2024-04-02T13:20:46.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.374+0000 7f508b7fe640 1 -- 172.21.15.44:0/874288427 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f509c05a0c0 con 0x7f50a4162660 2024-04-02T13:20:46.377 INFO:teuthology.orchestra.run.smithi044.stdout:55834574855 2024-04-02T13:20:46.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.374+0000 7fd537c3c640 1 -- 172.21.15.44:0/1614777386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd518005580 con 0x7fd530150420 2024-04-02T13:20:46.377 INFO:teuthology.orchestra.run.smithi044.stdout:55834574856 2024-04-02T13:20:46.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.375+0000 7fd5267fc640 1 -- 172.21.15.44:0/1614777386 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd52807b6c0 con 0x7fd530150420 2024-04-02T13:20:46.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.377+0000 7fbffae7b640 1 -- 172.21.15.44:0/2102474073 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fbff40cd780 con 0x7fbfdc0051c0 2024-04-02T13:20:46.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7f50897fa640 1 -- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f509006c9c0 msgr2=0x7f509006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7f50897fa640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f509006c9c0 0x7f509006ee80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f50940096f0 tx=0x7f5094008040 comp rx=0 tx=0).stop 2024-04-02T13:20:46.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7fbff17fa640 1 -- 172.21.15.44:0/2102474073 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbfe40595b0 con 0x7fbfdc0051c0 2024-04-02T13:20:46.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7f50897fa640 1 -- 172.21.15.44:0/874288427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50a4162660 msgr2=0x7f50a40d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7f50897fa640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50a4162660 0x7f50a40d0540 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f509c007e70 tx=0x7f509c00e4b0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7fd537c3c640 1 -- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd51406c9f0 msgr2=0x7fd51406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7fd537c3c640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd51406c9f0 0x7fd51406eeb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd5300cf6d0 tx=0x7fd520008040 comp rx=0 tx=0).stop 2024-04-02T13:20:46.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7fd537c3c640 1 -- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530150420 msgr2=0x7fd5301539f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7fd537c3c640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530150420 0x7fd5301539f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd52802cbc0 tx=0x7fd5300d09f0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.382 INFO:teuthology.orchestra.run.smithi044.stdout:55834574856 2024-04-02T13:20:46.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7f50897fa640 1 -- 172.21.15.44:0/874288427 shutdown_connections 2024-04-02T13:20:46.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7f50897fa640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f509006c9c0 0x7f509006ee80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.379+0000 7fd537c3c640 1 -- 172.21.15.44:0/1614777386 shutdown_connections 2024-04-02T13:20:46.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.379+0000 7fd537c3c640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd51406c9f0 0x7fd51406eeb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.379+0000 7fd537c3c640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd530150420 0x7fd5301539f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.379+0000 7fd537c3c640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd53009af60 0x7fd53014fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.379+0000 7fd537c3c640 1 --2- 172.21.15.44:0/1614777386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd530162800 0x7fd53009aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.379+0000 7fd537c3c640 1 -- 172.21.15.44:0/1614777386 >> 172.21.15.44:0/1614777386 conn(0x7fd5300cfcd0 msgr2=0x7fd53006dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:46.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7f50897fa640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50a4162660 0x7f50a40d0540 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7f50897fa640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50a40c5a20 0x7f50a4157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7f50897fa640 1 --2- 172.21.15.44:0/874288427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a406e090 0x7f50a4157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.378+0000 7f50897fa640 1 -- 172.21.15.44:0/874288427 >> 172.21.15.44:0/874288427 conn(0x7f50a40b74f0 msgr2=0x7f50a40b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:46.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.379+0000 7f50897fa640 1 -- 172.21.15.44:0/874288427 shutdown_connections 2024-04-02T13:20:46.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.379+0000 7f50897fa640 1 -- 172.21.15.44:0/874288427 wait complete. 2024-04-02T13:20:46.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.379+0000 7fd537c3c640 1 -- 172.21.15.44:0/1614777386 shutdown_connections 2024-04-02T13:20:46.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.379+0000 7fd537c3c640 1 -- 172.21.15.44:0/1614777386 wait complete. 2024-04-02T13:20:46.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.381+0000 7eff9bfff640 1 -- 172.21.15.44:0/2003295157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7eff9c0cd290 con 0x7eff9c0b4a70 2024-04-02T13:20:46.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.381+0000 7eff937fe640 1 -- 172.21.15.44:0/2003295157 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7eff8c05b290 con 0x7eff9c0b4a70 2024-04-02T13:20:46.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.381+0000 7fbffae7b640 1 -- 172.21.15.44:0/2102474073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbfe006c940 msgr2=0x7fbfe006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.381+0000 7fbffae7b640 1 --2- 172.21.15.44:0/2102474073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbfe006c940 0x7fbfe006ee00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbfec0050e0 tx=0x7fbfec005050 comp rx=0 tx=0).stop 2024-04-02T13:20:46.386 INFO:teuthology.orchestra.run.smithi044.stdout:55834574856 2024-04-02T13:20:46.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.382+0000 7fbffae7b640 1 -- 172.21.15.44:0/2102474073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0051c0 msgr2=0x7fbff409d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.382+0000 7fbffae7b640 1 --2- 172.21.15.44:0/2102474073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0051c0 0x7fbff409d410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400cac0 tx=0x7fbfe400cf80 comp rx=0 tx=0).stop 2024-04-02T13:20:46.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.382+0000 7fbffae7b640 1 -- 172.21.15.44:0/2102474073 shutdown_connections 2024-04-02T13:20:46.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.382+0000 7fbffae7b640 1 --2- 172.21.15.44:0/2102474073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbfe006c940 0x7fbfe006ee00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.382+0000 7fbffae7b640 1 --2- 172.21.15.44:0/2102474073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff4142390 0x7fbff40b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.382+0000 7fbffae7b640 1 --2- 172.21.15.44:0/2102474073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbff4141f10 0x7fbff40b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.382+0000 7fbffae7b640 1 --2- 172.21.15.44:0/2102474073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0051c0 0x7fbff409d410 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400cac0 tx=0x7fbfe400cf80 comp rx=0 tx=0).stop 2024-04-02T13:20:46.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.382+0000 7fbffae7b640 1 -- 172.21.15.44:0/2102474073 >> 172.21.15.44:0/2102474073 conn(0x7fbff40cfcd0 msgr2=0x7fbff406e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:46.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.382+0000 7fbffae7b640 1 -- 172.21.15.44:0/2102474073 shutdown_connections 2024-04-02T13:20:46.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.382+0000 7fbffae7b640 1 -- 172.21.15.44:0/2102474073 wait complete. 2024-04-02T13:20:46.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.384+0000 7f9929740640 1 -- 172.21.15.44:0/1804507819 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f99240d4d90 con 0x7f99240b34c0 2024-04-02T13:20:46.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.385+0000 7eff9bfff640 1 -- 172.21.15.44:0/2003295157 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff7c06c9c0 msgr2=0x7eff7c06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.385+0000 7eff9bfff640 1 --2- 172.21.15.44:0/2003295157 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff7c06c9c0 0x7eff7c06ee80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7eff84006b10 tx=0x7eff84008040 comp rx=0 tx=0).stop 2024-04-02T13:20:46.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.385+0000 7f990b7fe640 1 -- 172.21.15.44:0/1804507819 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f991c05a480 con 0x7f99240b34c0 2024-04-02T13:20:46.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.385+0000 7eff9bfff640 1 -- 172.21.15.44:0/2003295157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff9c0b4a70 msgr2=0x7eff9c157140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.385+0000 7eff9bfff640 1 --2- 172.21.15.44:0/2003295157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff9c0b4a70 0x7eff9c157140 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0021a0 tx=0x7eff8c00a4f0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.385+0000 7eff9bfff640 1 -- 172.21.15.44:0/2003295157 shutdown_connections 2024-04-02T13:20:46.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.385+0000 7eff9bfff640 1 --2- 172.21.15.44:0/2003295157 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff7c06c9c0 0x7eff7c06ee80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.385+0000 7eff9bfff640 1 --2- 172.21.15.44:0/2003295157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff9c0cd9d0 0x7eff9c157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.385+0000 7eff9bfff640 1 --2- 172.21.15.44:0/2003295157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff9c0b4a70 0x7eff9c157140 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0021a0 tx=0x7eff8c00a4f0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.385+0000 7eff9bfff640 1 --2- 172.21.15.44:0/2003295157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff9c070fc0 0x7eff9c14e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.385+0000 7eff9bfff640 1 -- 172.21.15.44:0/2003295157 >> 172.21.15.44:0/2003295157 conn(0x7eff9c0b1860 msgr2=0x7eff9c14f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:46.389 INFO:teuthology.orchestra.run.smithi044.stdout:55834574856 2024-04-02T13:20:46.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.385+0000 7eff9bfff640 1 -- 172.21.15.44:0/2003295157 shutdown_connections 2024-04-02T13:20:46.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.385+0000 7eff9bfff640 1 -- 172.21.15.44:0/2003295157 wait complete. 2024-04-02T13:20:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.389+0000 7f99097fa640 1 -- 172.21.15.44:0/1804507819 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f991006c8f0 msgr2=0x7f991006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.389+0000 7f99097fa640 1 --2- 172.21.15.44:0/1804507819 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f991006c8f0 0x7f991006edb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f99240cd070 tx=0x7f9918007a40 comp rx=0 tx=0).stop 2024-04-02T13:20:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.389+0000 7f99097fa640 1 -- 172.21.15.44:0/1804507819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99240b34c0 msgr2=0x7f9924158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.389+0000 7f99097fa640 1 --2- 172.21.15.44:0/1804507819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99240b34c0 0x7f9924158690 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f991c00c890 tx=0x7f991c00cd50 comp rx=0 tx=0).stop 2024-04-02T13:20:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.389+0000 7f99097fa640 1 -- 172.21.15.44:0/1804507819 shutdown_connections 2024-04-02T13:20:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.389+0000 7f99097fa640 1 --2- 172.21.15.44:0/1804507819 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f991006c8f0 0x7f991006edb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.389+0000 7f99097fa640 1 --2- 172.21.15.44:0/1804507819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99240d2750 0x7f99240b93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.389+0000 7f99097fa640 1 --2- 172.21.15.44:0/1804507819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99240d0990 0x7f99240cbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.389+0000 7f99097fa640 1 --2- 172.21.15.44:0/1804507819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99240b34c0 0x7f9924158690 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.389+0000 7f99097fa640 1 -- 172.21.15.44:0/1804507819 >> 172.21.15.44:0/1804507819 conn(0x7f992409d470 msgr2=0x7f99240b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:46.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.390+0000 7f58aa6db640 1 -- 172.21.15.44:0/3578372465 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f5884005850 con 0x7f58a40d0b20 2024-04-02T13:20:46.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.390+0000 7f99097fa640 1 -- 172.21.15.44:0/1804507819 shutdown_connections 2024-04-02T13:20:46.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.390+0000 7f99097fa640 1 -- 172.21.15.44:0/1804507819 wait complete. 2024-04-02T13:20:46.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.390+0000 7f5899ffb640 1 -- 172.21.15.44:0/3578372465 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f58940593a0 con 0x7f58a40d0b20 2024-04-02T13:20:46.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.391+0000 7fc125f67640 1 -- 172.21.15.44:0/1908095311 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc1180385d0 con 0x7fc1100039b0 2024-04-02T13:20:46.394 INFO:teuthology.orchestra.run.smithi044.stdout:55834574856 2024-04-02T13:20:46.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.391+0000 7fc10bfff640 1 -- 172.21.15.44:0/1908095311 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc11400f000 con 0x7fc1100039b0 2024-04-02T13:20:46.394 INFO:teuthology.orchestra.run.smithi044.stdout:55834574855 2024-04-02T13:20:46.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.394+0000 7f58aa6db640 1 -- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f588806c8f0 msgr2=0x7f588806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.394+0000 7f58aa6db640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f588806c8f0 0x7f588806edb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f58a0036000 tx=0x7f58a003c040 comp rx=0 tx=0).stop 2024-04-02T13:20:46.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7f58aa6db640 1 -- 172.21.15.44:0/3578372465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58a40d0b20 msgr2=0x7f58a41f8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7f58aa6db640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58a40d0b20 0x7f58a41f8200 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f589400c4a0 tx=0x7f589400c960 comp rx=0 tx=0).stop 2024-04-02T13:20:46.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7f58aa6db640 1 -- 172.21.15.44:0/3578372465 shutdown_connections 2024-04-02T13:20:46.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7f58aa6db640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f588806c8f0 0x7f588806edb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7f58aa6db640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58a41596b0 0x7f58a415a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7f58aa6db640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58a41592d0 0x7f58a414c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7f58aa6db640 1 --2- 172.21.15.44:0/3578372465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58a40d0b20 0x7f58a41f8200 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7f58aa6db640 1 -- 172.21.15.44:0/3578372465 >> 172.21.15.44:0/3578372465 conn(0x7f58a406b4d0 msgr2=0x7f58a406e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:46.399 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.1 2024-04-02T13:20:46.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7fc125f67640 1 -- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc10406c9c0 msgr2=0x7fc10406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7fc125f67640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc10406c9c0 0x7fc10406ee80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc11803c280 tx=0x7fc10c009040 comp rx=0 tx=0).stop 2024-04-02T13:20:46.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7f58aa6db640 1 -- 172.21.15.44:0/3578372465 shutdown_connections 2024-04-02T13:20:46.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7f58aa6db640 1 -- 172.21.15.44:0/3578372465 wait complete. 2024-04-02T13:20:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7fc125f67640 1 -- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1100039b0 msgr2=0x7fc1180e0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.395+0000 7fc125f67640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1100039b0 0x7fc1180e0260 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc114009c30 tx=0x7fc114004730 comp rx=0 tx=0).stop 2024-04-02T13:20:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.396+0000 7fc125f67640 1 -- 172.21.15.44:0/1908095311 shutdown_connections 2024-04-02T13:20:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.396+0000 7fc125f67640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc10406c9c0 0x7fc10406ee80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.396+0000 7fc125f67640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc11804c700 0x7fc11804cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.396+0000 7fc125f67640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1180e2030 0x7fc1180e3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.396+0000 7fc125f67640 1 --2- 172.21.15.44:0/1908095311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1100039b0 0x7fc1180e0260 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.396+0000 7fc125f67640 1 -- 172.21.15.44:0/1908095311 >> 172.21.15.44:0/1908095311 conn(0x7fc118056cc0 msgr2=0x7fc118009bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.396+0000 7fc125f67640 1 -- 172.21.15.44:0/1908095311 shutdown_connections 2024-04-02T13:20:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:46.396+0000 7fc125f67640 1 -- 172.21.15.44:0/1908095311 wait complete. 2024-04-02T13:20:46.403 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2024-04-02T13:20:46.405 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.3 2024-04-02T13:20:46.407 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.7 2024-04-02T13:20:46.409 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.0 2024-04-02T13:20:46.415 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.4 2024-04-02T13:20:46.415 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.5 2024-04-02T13:20:46.416 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2024-04-02T13:20:47.399 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T13:20:47.404 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-02T13:20:47.406 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-02T13:20:47.408 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-02T13:20:47.409 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T13:20:47.416 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-02T13:20:47.416 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-02T13:20:47.417 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T13:20:47.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.565+0000 7f83b1722640 1 Processor -- start 2024-04-02T13:20:47.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.566+0000 7f83b1722640 1 -- start start 2024-04-02T13:20:47.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.567+0000 7f83b1722640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f83a4050600 0x7f83a40509e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.567+0000 7f83b1722640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f83a4034fd0 0x7f83a4055290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.567+0000 7f83b1722640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a4050f20 con 0x7f83a4053ef0 2024-04-02T13:20:47.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.567+0000 7f83b1722640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f83a4009420 con 0x7f83a4050600 2024-04-02T13:20:47.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.567+0000 7f83b1722640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f83a4008af0 con 0x7f83a4034fd0 2024-04-02T13:20:47.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.568+0000 7f83aa7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f83a4034fd0 0x7f83a4055290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.568+0000 7f83aa7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f83a4034fd0 0x7f83a4055290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34366/0 (socket says 172.21.15.44:34366) 2024-04-02T13:20:47.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.568+0000 7f83aa7fc640 1 -- 172.21.15.44:0/1370592076 learned_addr learned my addr 172.21.15.44:0/1370592076 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.568+0000 7f83aaffd640 1 --2- 172.21.15.44:0/1370592076 >> v2:172.21.15.44:3300/0 conn(0x7f83a4050600 0x7f83a40509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.568+0000 7f83aa7fc640 1 -- 172.21.15.44:0/1370592076 >> v2:172.21.15.44:3300/0 conn(0x7f83a4050600 msgr2=0x7f83a40509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:47.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.568+0000 7f83aa7fc640 1 --2- 172.21.15.44:0/1370592076 >> v2:172.21.15.44:3300/0 conn(0x7f83a4050600 0x7f83a40509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.568+0000 7f83aa7fc640 1 -- 172.21.15.44:0/1370592076 >> v1:172.21.15.44:6789/0 conn(0x7f83a4053ef0 legacy=0x7f83a4054380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T13:20:47.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.568+0000 7f83aa7fc640 1 -- 172.21.15.44:0/1370592076 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a4008dd0 con 0x7f83a4034fd0 2024-04-02T13:20:47.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.570+0000 7f83aa7fc640 1 --2- 172.21.15.44:0/1370592076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83a4034fd0 0x7f83a4055290 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f83ac02d900 tx=0x7f83ac0a6d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bcb08934e160e6d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.570+0000 7f83a9ffb640 1 -- 172.21.15.44:0/1370592076 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac0b4070 con 0x7f83a4034fd0 2024-04-02T13:20:47.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.570+0000 7f83a9ffb640 1 -- 172.21.15.44:0/1370592076 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83ac0a7bc0 con 0x7f83a4034fd0 2024-04-02T13:20:47.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.571+0000 7f83b1722640 1 -- 172.21.15.44:0/1370592076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83a4034fd0 msgr2=0x7f83a4055290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.571+0000 7f83b1722640 1 --2- 172.21.15.44:0/1370592076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83a4034fd0 0x7f83a4055290 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f83ac02d900 tx=0x7f83ac0a6d50 comp rx=0 tx=0).stop 2024-04-02T13:20:47.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.572+0000 7f83b1722640 1 -- 172.21.15.44:0/1370592076 shutdown_connections 2024-04-02T13:20:47.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.572+0000 7f83b1722640 1 --2- 172.21.15.44:0/1370592076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83a4034fd0 0x7f83a4055290 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.572+0000 7f83b1722640 1 --2- 172.21.15.44:0/1370592076 >> v2:172.21.15.44:3300/0 conn(0x7f83a4050600 0x7f83a40509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.572+0000 7f83b1722640 1 -- 172.21.15.44:0/1370592076 >> 172.21.15.44:0/1370592076 conn(0x7f83a4055da0 msgr2=0x7f83a40561b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:47.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.572+0000 7f83b1722640 1 -- 172.21.15.44:0/1370592076 shutdown_connections 2024-04-02T13:20:47.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.572+0000 7f83b1722640 1 -- 172.21.15.44:0/1370592076 wait complete. 2024-04-02T13:20:47.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.574+0000 7f83b1722640 1 Processor -- start 2024-04-02T13:20:47.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.575+0000 7f83b1722640 1 -- start start 2024-04-02T13:20:47.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.576+0000 7f83b1722640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83a4034fd0 0x7f83a40f0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.576+0000 7f83b1722640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83a4050600 0x7f83a404dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.576+0000 7f83aaffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83a4034fd0 0x7f83a40f0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.576+0000 7f83aa7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83a4050600 0x7f83a404dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.576+0000 7f83aa7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83a4050600 0x7f83a404dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54930/0 (socket says 172.21.15.44:54930) 2024-04-02T13:20:47.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.576+0000 7f83aaffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83a4034fd0 0x7f83a40f0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34372/0 (socket says 172.21.15.44:34372) 2024-04-02T13:20:47.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.577+0000 7f83aa7fc640 1 -- 172.21.15.44:0/3067324230 learned_addr learned my addr 172.21.15.44:0/3067324230 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.577+0000 7f83b1722640 1 --2- 172.21.15.44:0/3067324230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83a4053ef0 0x7f83a417d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.577+0000 7f83b1722640 1 -- 172.21.15.44:0/3067324230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f83a40e0270 con 0x7f83a4050600 2024-04-02T13:20:47.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.577+0000 7f83b1722640 1 -- 172.21.15.44:0/3067324230 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f83a40087e0 con 0x7f83a4034fd0 2024-04-02T13:20:47.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.577+0000 7f83b1722640 1 -- 172.21.15.44:0/3067324230 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f83a40e03f0 con 0x7f83a4053ef0 2024-04-02T13:20:47.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.578+0000 7f83aa7fc640 1 -- 172.21.15.44:0/3067324230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83a4053ef0 msgr2=0x7f83a417d7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T13:20:47.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.578+0000 7f83aa7fc640 1 --2- 172.21.15.44:0/3067324230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83a4053ef0 0x7f83a417d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.578+0000 7f83aa7fc640 1 -- 172.21.15.44:0/3067324230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83a4034fd0 msgr2=0x7f83a40f0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.578+0000 7f83aa7fc640 1 --2- 172.21.15.44:0/3067324230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83a4034fd0 0x7f83a40f0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.578+0000 7f83aa7fc640 1 -- 172.21.15.44:0/3067324230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a404fb00 con 0x7f83a4050600 2024-04-02T13:20:47.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.578+0000 7f83aa7fc640 1 --2- 172.21.15.44:0/3067324230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83a4050600 0x7f83a404dbb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f83ac050610 tx=0x7f83ac069ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.579+0000 7f83937fe640 1 -- 172.21.15.44:0/3067324230 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac0b4050 con 0x7f83a4050600 2024-04-02T13:20:47.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.579+0000 7f83b1722640 1 -- 172.21.15.44:0/3067324230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a4036bc0 con 0x7f83a4050600 2024-04-02T13:20:47.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.579+0000 7f83b1722640 1 -- 172.21.15.44:0/3067324230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a4037180 con 0x7f83a4050600 2024-04-02T13:20:47.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.582+0000 7f83937fe640 1 -- 172.21.15.44:0/3067324230 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83ac0b5040 con 0x7f83a4050600 2024-04-02T13:20:47.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.582+0000 7f83937fe640 1 -- 172.21.15.44:0/3067324230 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac0aede0 con 0x7f83a4050600 2024-04-02T13:20:47.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.583+0000 7f83937fe640 1 -- 172.21.15.44:0/3067324230 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f83ac0c2020 con 0x7f83a4050600 2024-04-02T13:20:47.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.584+0000 7f83937fe640 1 --2- 172.21.15.44:0/3067324230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f839406cb30 0x7f839406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.584+0000 7f83937fe640 1 -- 172.21.15.44:0/3067324230 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f83ac128ee0 con 0x7f83a4050600 2024-04-02T13:20:47.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.584+0000 7f83aaffd640 1 --2- 172.21.15.44:0/3067324230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f839406cb30 0x7f839406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.585+0000 7f83aaffd640 1 --2- 172.21.15.44:0/3067324230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f839406cb30 0x7f839406eff0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f839c006fd0 tx=0x7f839c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.586+0000 7f83b1722640 1 -- 172.21.15.44:0/3067324230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f838c005290 con 0x7f83a4050600 2024-04-02T13:20:47.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.596+0000 7f83937fe640 1 -- 172.21.15.44:0/3067324230 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f83a404fb00 con 0x7f83a4050600 2024-04-02T13:20:47.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.618+0000 7f7e57c1a640 1 Processor -- start 2024-04-02T13:20:47.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.618+0000 7f7e57c1a640 1 -- start start 2024-04-02T13:20:47.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.620+0000 7f7e57c1a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5016d970 0x7f7e500cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.620+0000 7f7e57c1a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e50071e50 0x7f7e5016dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.620+0000 7f7e57c1a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e500724c0 con 0x7f7e5016e290 2024-04-02T13:20:47.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.620+0000 7f7e57c1a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7e500727a0 con 0x7f7e50071e50 2024-04-02T13:20:47.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.620+0000 7f7e57c1a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7e50072a80 con 0x7f7e5016d970 2024-04-02T13:20:47.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.620+0000 7f7e561b7640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7e5016e290 0x7f7e5006d1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57470/0 (socket says 172.21.15.44:57470) 2024-04-02T13:20:47.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.620+0000 7f7e561b7640 1 -- 172.21.15.44:0/720071661 learned_addr learned my addr 172.21.15.44:0/720071661 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:47.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.620+0000 7f7e559b6640 1 --2- 172.21.15.44:0/720071661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5016d970 0x7f7e500cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.620+0000 7f7e551b5640 1 --2- 172.21.15.44:0/720071661 >> v2:172.21.15.69:3300/0 conn(0x7f7e50071e50 0x7f7e5016dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.621+0000 7f7e549b4640 1 -- 172.21.15.44:0/720071661 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 767366718 0 0) 0x7f7e500724c0 con 0x7f7e5016e290 2024-04-02T13:20:47.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.621+0000 7f7e549b4640 1 -- 172.21.15.44:0/720071661 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e3c003520 con 0x7f7e5016e290 2024-04-02T13:20:47.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.621+0000 7f7e551b5640 1 -- 172.21.15.44:0/720071661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5016d970 msgr2=0x7f7e500cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:47.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.621+0000 7f7e551b5640 1 --2- 172.21.15.44:0/720071661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5016d970 0x7f7e500cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.621+0000 7f7e551b5640 1 -- 172.21.15.44:0/720071661 >> v1:172.21.15.44:6789/0 conn(0x7f7e5016e290 legacy=0x7f7e5006d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.621+0000 7f7e551b5640 1 -- 172.21.15.44:0/720071661 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e50072d60 con 0x7f7e50071e50 2024-04-02T13:20:47.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.622+0000 7f97b197c640 1 Processor -- start 2024-04-02T13:20:47.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.622+0000 7f8a4e319640 1 Processor -- start 2024-04-02T13:20:47.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.622+0000 7f8a4e319640 1 -- start start 2024-04-02T13:20:47.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.623+0000 7fbb61be7640 1 Processor -- start 2024-04-02T13:20:47.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.623+0000 7f7e551b5640 1 --2- 172.21.15.44:0/720071661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e50071e50 0x7f7e5016dd50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009d90 tx=0x7f7e4802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57babc7e6a7098a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.623+0000 7fbb61be7640 1 -- start start 2024-04-02T13:20:47.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.623+0000 7f7e549b4640 1 -- 172.21.15.44:0/720071661 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48039070 con 0x7f7e50071e50 2024-04-02T13:20:47.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.623+0000 7f7e549b4640 1 -- 172.21.15.44:0/720071661 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e4802cc50 con 0x7f7e50071e50 2024-04-02T13:20:47.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.623+0000 7f3e1df2b640 1 Processor -- start 2024-04-02T13:20:47.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f7e57c1a640 1 -- 172.21.15.44:0/720071661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e50071e50 msgr2=0x7f7e5016dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f7e57c1a640 1 --2- 172.21.15.44:0/720071661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e50071e50 0x7f7e5016dd50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009d90 tx=0x7f7e4802bea0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f8a4e319640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8a400e75c0 0x7f8a40007630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f8a4e319640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a40058990 0x7f8a40053500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f8a4e319640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a40056e60 0x7f8a40053a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f8a4e319640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f8a40007b70 con 0x7f8a400e75c0 2024-04-02T13:20:47.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f8a4e319640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8a400e79a0 con 0x7f8a40056e60 2024-04-02T13:20:47.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f8a4e319640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8a40006590 con 0x7f8a40058990 2024-04-02T13:20:47.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.623+0000 7f97b197c640 1 -- start start 2024-04-02T13:20:47.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f97b197c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f97a40e0050 0x7f97a400b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f97b197c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97a4008f80 0x7f97a403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f97b197c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97a400cb60 con 0x7f97a403ae30 2024-04-02T13:20:47.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f97b197c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f97a400b740 con 0x7f97a40e0050 2024-04-02T13:20:47.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f97b197c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f97a4056600 con 0x7f97a4008f80 2024-04-02T13:20:47.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7fbb61be7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb54006a20 0x7fbb54006eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7fbb61be7640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb54053710 con 0x7fbb54005ed0 2024-04-02T13:20:47.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7fbb61be7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb540539f0 con 0x7fbb540eacf0 2024-04-02T13:20:47.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7fbb61be7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbb54054320 con 0x7fbb54006a20 2024-04-02T13:20:47.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7fbb60be5640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbb540eacf0 0x7fbb5404fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:48690/0 (socket says 172.21.15.44:48690) 2024-04-02T13:20:47.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7fbb60be5640 1 -- 172.21.15.44:0/693359740 learned_addr learned my addr 172.21.15.44:0/693359740 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:47.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f97aa7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97a4008f80 0x7f97a403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f97aa7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97a4008f80 0x7f97a403a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:42122/0 (socket says 172.21.15.44:42122) 2024-04-02T13:20:47.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f97aa7fc640 1 -- 172.21.15.44:0/480748481 learned_addr learned my addr 172.21.15.44:0/480748481 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.623+0000 7f3e1df2b640 1 -- start start 2024-04-02T13:20:47.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f3e1df2b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e100ead00 0x7f3e1004fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.624+0000 7f3e1df2b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3e1004ffd0 0x7f3e10037bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f09e759e640 1 Processor -- start 2024-04-02T13:20:47.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f7e57c1a640 1 -- 172.21.15.44:0/720071661 shutdown_connections 2024-04-02T13:20:47.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f7e57c1a640 1 --2- 172.21.15.44:0/720071661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e50071e50 0x7f7e5016dd50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f7e57c1a640 1 --2- 172.21.15.44:0/720071661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5016d970 0x7f7e500cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f7e57c1a640 1 -- 172.21.15.44:0/720071661 >> 172.21.15.44:0/720071661 conn(0x7f7e500cfcd0 msgr2=0x7f7e500d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:47.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f81b2305640 1 Processor -- start 2024-04-02T13:20:47.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f3e1cf29640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e100ead00 0x7f3e1004fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f3e1df2b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e10053720 con 0x7f3e10038110 2024-04-02T13:20:47.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f3e1df2b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f3e10054330 con 0x7f3e1004ffd0 2024-04-02T13:20:47.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f3e1df2b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3e10053a00 con 0x7f3e100ead00 2024-04-02T13:20:47.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f3e1cf29640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e100ead00 0x7f3e1004fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:42144/0 (socket says 172.21.15.44:42144) 2024-04-02T13:20:47.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f3e1cf29640 1 -- 172.21.15.44:0/4190603851 learned_addr learned my addr 172.21.15.44:0/4190603851 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f8a4d317640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8a400e75c0 0x7f8a40007630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f8a4d317640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8a400e75c0 0x7f8a40007630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54934/0 (socket says 172.21.15.44:54934) 2024-04-02T13:20:47.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f8a4d317640 1 -- 172.21.15.44:0/2783057231 learned_addr learned my addr 172.21.15.44:0/2783057231 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f8a4db18640 1 --2- 172.21.15.44:0/2783057231 >> v2:172.21.15.69:3300/0 conn(0x7f8a40056e60 0x7f8a40053a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f8a4cb16640 1 --2- 172.21.15.44:0/2783057231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a40058990 0x7f8a40053500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f7e57c1a640 1 -- 172.21.15.44:0/720071661 shutdown_connections 2024-04-02T13:20:47.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f7e57c1a640 1 -- 172.21.15.44:0/720071661 wait complete. 2024-04-02T13:20:47.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f3e1cf29640 1 -- 172.21.15.44:0/4190603851 >> v2:172.21.15.44:3300/0 conn(0x7f3e1004ffd0 msgr2=0x7f3e10037bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:47.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f3e1cf29640 1 --2- 172.21.15.44:0/4190603851 >> v2:172.21.15.44:3300/0 conn(0x7f3e1004ffd0 0x7f3e10037bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f3e1cf29640 1 -- 172.21.15.44:0/4190603851 >> v1:172.21.15.44:6789/0 conn(0x7f3e10038110 legacy=0x7f3e100385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:47.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f3e1cf29640 1 -- 172.21.15.44:0/4190603851 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e10053ce0 con 0x7f3e100ead00 2024-04-02T13:20:47.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f09e759e640 1 -- start start 2024-04-02T13:20:47.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f7e57c1a640 1 Processor -- start 2024-04-02T13:20:47.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7f3e1cf29640 1 --2- 172.21.15.44:0/4190603851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e100ead00 0x7f3e1004fa90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009870 tx=0x7f3e0c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a32687afe0ad9bb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f8a4db18640 1 -- 172.21.15.44:0/2783057231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a40058990 msgr2=0x7f8a40053500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:47.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f8a4db18640 1 --2- 172.21.15.44:0/2783057231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a40058990 0x7f8a40053500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f8a4db18640 1 -- 172.21.15.44:0/2783057231 >> v2:172.21.15.44:3300/0 conn(0x7f8a400e75c0 msgr2=0x7f8a40007630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:47.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f8a4db18640 1 --2- 172.21.15.44:0/2783057231 >> v2:172.21.15.44:3300/0 conn(0x7f8a400e75c0 0x7f8a40007630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f8a4db18640 1 -- 172.21.15.44:0/2783057231 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a40006870 con 0x7f8a40056e60 2024-04-02T13:20:47.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f8a4db18640 1 --2- 172.21.15.44:0/2783057231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a40056e60 0x7f8a40053a40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a40011ff0 tx=0x7f8a3802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40d48a9d805ff320 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7f97aaffd640 1 --2- 172.21.15.44:0/480748481 >> v2:172.21.15.44:3300/0 conn(0x7f97a40e0050 0x7f97a400b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7f97aa7fc640 1 -- 172.21.15.44:0/480748481 >> v2:172.21.15.44:3300/0 conn(0x7f97a40e0050 msgr2=0x7f97a400b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:47.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7f97aa7fc640 1 --2- 172.21.15.44:0/480748481 >> v2:172.21.15.44:3300/0 conn(0x7f97a40e0050 0x7f97a400b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7f97aa7fc640 1 -- 172.21.15.44:0/480748481 >> v1:172.21.15.44:6789/0 conn(0x7f97a403ae30 legacy=0x7f97a4007c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:47.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7f97aa7fc640 1 -- 172.21.15.44:0/480748481 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a40568e0 con 0x7f97a4008f80 2024-04-02T13:20:47.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7fbb613e6640 1 --2- 172.21.15.44:0/693359740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb54006a20 0x7fbb54006eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7fbb613e6640 1 -- 172.21.15.44:0/693359740 >> v1:172.21.15.69:6789/0 conn(0x7fbb540eacf0 legacy=0x7fbb5404fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7fbb613e6640 1 -- 172.21.15.44:0/693359740 >> v1:172.21.15.44:6789/0 conn(0x7fbb54005ed0 legacy=0x7fbb54006310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:47.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7fbb5b7fe640 1 -- 172.21.15.44:0/693359740 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1093733539 0 0) 0x7fbb540539f0 con 0x7fbb540eacf0 2024-04-02T13:20:47.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7fbb613e6640 1 -- 172.21.15.44:0/693359740 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb54053cd0 con 0x7fbb54006a20 2024-04-02T13:20:47.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7f09e759e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815bb20 0x7f09e815aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7f09e759e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09e815b040 con 0x7f09e8072bf0 2024-04-02T13:20:47.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7f09e759e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09e815b320 con 0x7f09e8073940 2024-04-02T13:20:47.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.626+0000 7f09e759e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f09e8074820 con 0x7f09e815bb20 2024-04-02T13:20:47.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f09e5d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815bb20 0x7f09e815aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f09e5d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815bb20 0x7f09e815aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:42170/0 (socket says 172.21.15.44:42170) 2024-04-02T13:20:47.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f09e5d9b640 1 -- 172.21.15.44:0/3906555922 learned_addr learned my addr 172.21.15.44:0/3906555922 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f8a3ffff640 1 -- 172.21.15.44:0/2783057231 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a380315a0 con 0x7f8a40056e60 2024-04-02T13:20:47.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f97aa7fc640 1 --2- 172.21.15.44:0/480748481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97a4008f80 0x7f97a403a8f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f97ac000b60 tx=0x7f97ac0a6cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e51253b49829673 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f8a3ffff640 1 -- 172.21.15.44:0/2783057231 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a38037070 con 0x7f8a40056e60 2024-04-02T13:20:47.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.625+0000 7f81b2305640 1 -- start start 2024-04-02T13:20:47.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f81b2305640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f81ac0cbdb0 0x7f81ac15f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f81b2305640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81ac15e930 con 0x7f81ac072070 2024-04-02T13:20:47.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f81b2305640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81ac0ccbd0 con 0x7f81ac15e230 2024-04-02T13:20:47.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f81b2305640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f81ac15ef60 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f3e177fe640 1 -- 172.21.15.44:0/4190603851 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c035020 con 0x7f3e100ead00 2024-04-02T13:20:47.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f81b0b02640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f81ac0cbdb0 0x7f81ac15f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f81b0b02640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f81ac0cbdb0 0x7f81ac15f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54956/0 (socket says 172.21.15.44:54956) 2024-04-02T13:20:47.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f81b0b02640 1 -- 172.21.15.44:0/3646271674 learned_addr learned my addr 172.21.15.44:0/3646271674 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f3e177fe640 1 -- 172.21.15.44:0/4190603851 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e0c031640 con 0x7f3e100ead00 2024-04-02T13:20:47.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7fbb613e6640 1 --2- 172.21.15.44:0/693359740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb54006a20 0x7fbb54006eb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c04fd80 tx=0x7fbb5c0a6cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d23a293887ec844 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f3e177fe640 1 -- 172.21.15.44:0/4190603851 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c02d040 con 0x7f3e100ead00 2024-04-02T13:20:47.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f97a9ffb640 1 -- 172.21.15.44:0/480748481 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0b4070 con 0x7f97a4008f80 2024-04-02T13:20:47.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f97a9ffb640 1 -- 172.21.15.44:0/480748481 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97ac0a7b30 con 0x7f97a4008f80 2024-04-02T13:20:47.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f97a9ffb640 1 -- 172.21.15.44:0/480748481 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0af440 con 0x7f97a4008f80 2024-04-02T13:20:47.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f8a3ffff640 1 -- 172.21.15.44:0/2783057231 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38031960 con 0x7f8a40056e60 2024-04-02T13:20:47.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7fbb5b7fe640 1 -- 172.21.15.44:0/693359740 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c0b4070 con 0x7fbb54006a20 2024-04-02T13:20:47.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7fbb5b7fe640 1 -- 172.21.15.44:0/693359740 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb5c0a7b60 con 0x7fbb54006a20 2024-04-02T13:20:47.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7fbb5b7fe640 1 -- 172.21.15.44:0/693359740 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c0af440 con 0x7fbb54006a20 2024-04-02T13:20:47.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f8a4e319640 1 -- 172.21.15.44:0/2783057231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a40056e60 msgr2=0x7f8a40053a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f8a4e319640 1 --2- 172.21.15.44:0/2783057231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a40056e60 0x7f8a40053a40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a40011ff0 tx=0x7f8a3802b840 comp rx=0 tx=0).stop 2024-04-02T13:20:47.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f3e1df2b640 1 -- 172.21.15.44:0/4190603851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e100ead00 msgr2=0x7f3e1004fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f3e1df2b640 1 --2- 172.21.15.44:0/4190603851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e100ead00 0x7f3e1004fa90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009870 tx=0x7f3e0c02b8f0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f3e1df2b640 1 -- 172.21.15.44:0/4190603851 shutdown_connections 2024-04-02T13:20:47.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f3e1df2b640 1 --2- 172.21.15.44:0/4190603851 >> v2:172.21.15.44:3300/0 conn(0x7f3e1004ffd0 0x7f3e10037bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f3e1df2b640 1 --2- 172.21.15.44:0/4190603851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e100ead00 0x7f3e1004fa90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f3e1df2b640 1 -- 172.21.15.44:0/4190603851 >> 172.21.15.44:0/4190603851 conn(0x7f3e10009fb0 msgr2=0x7f3e1000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:47.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f8a4e319640 1 -- 172.21.15.44:0/2783057231 shutdown_connections 2024-04-02T13:20:47.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f8a4e319640 1 --2- 172.21.15.44:0/2783057231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a40056e60 0x7f8a40053a40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f8a4e319640 1 --2- 172.21.15.44:0/2783057231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a40058990 0x7f8a40053500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f8a4e319640 1 --2- 172.21.15.44:0/2783057231 >> v2:172.21.15.44:3300/0 conn(0x7f8a400e75c0 0x7f8a40007630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f8a4e319640 1 -- 172.21.15.44:0/2783057231 >> 172.21.15.44:0/2783057231 conn(0x7f8a40058eb0 msgr2=0x7f8a400592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:47.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.627+0000 7f7e57c1a640 1 -- start start 2024-04-02T13:20:47.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f7e57c1a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e50071e50 0x7f7e5009d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f7e57c1a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5016d970 0x7f7e50170e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f7e57c1a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e5016e290 0x7f7e5009aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f7e57c1a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7e500cc980 con 0x7f7e5016e290 2024-04-02T13:20:47.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f7e57c1a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7e500cc800 con 0x7f7e50071e50 2024-04-02T13:20:47.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f7e57c1a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7e50171c90 con 0x7f7e5016d970 2024-04-02T13:20:47.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f7e559b6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e50071e50 0x7f7e5009d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f7e559b6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e50071e50 0x7f7e5009d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34394/0 (socket says 172.21.15.44:34394) 2024-04-02T13:20:47.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f7e559b6640 1 -- 172.21.15.44:0/2317860512 learned_addr learned my addr 172.21.15.44:0/2317860512 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f7e551b5640 1 --2- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5016d970 0x7f7e50170e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f3e1df2b640 1 -- 172.21.15.44:0/4190603851 shutdown_connections 2024-04-02T13:20:47.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f3e1df2b640 1 -- 172.21.15.44:0/4190603851 wait complete. 2024-04-02T13:20:47.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 418473799 0 0) 0x7f09e815b320 con 0x7f09e8073940 2024-04-02T13:20:47.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09d0003520 con 0x7f09e8073940 2024-04-02T13:20:47.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7fbb61be7640 1 -- 172.21.15.44:0/693359740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb54006a20 msgr2=0x7fbb54006eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7fbb61be7640 1 --2- 172.21.15.44:0/693359740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb54006a20 0x7fbb54006eb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c04fd80 tx=0x7fbb5c0a6cf0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 <== mon.1 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 572278342 0 0) 0x7f09d0003520 con 0x7f09e8073940 2024-04-02T13:20:47.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7fbb61be7640 1 -- 172.21.15.44:0/693359740 shutdown_connections 2024-04-02T13:20:47.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7fbb61be7640 1 --2- 172.21.15.44:0/693359740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb54006a20 0x7fbb54006eb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7fbb61be7640 1 -- 172.21.15.44:0/693359740 >> 172.21.15.44:0/693359740 conn(0x7fbb54009fa0 msgr2=0x7fbb5400a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:47.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815bb20 msgr2=0x7f09e815aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:47.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f09e559a640 1 --2- 172.21.15.44:0/3906555922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815bb20 0x7f09e815aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7fbb61be7640 1 -- 172.21.15.44:0/693359740 shutdown_connections 2024-04-02T13:20:47.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 >> v1:172.21.15.44:6789/0 conn(0x7f09e8072bf0 legacy=0x7f09e806d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e815b600 con 0x7f09e8073940 2024-04-02T13:20:47.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f8a4e319640 1 -- 172.21.15.44:0/2783057231 shutdown_connections 2024-04-02T13:20:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 <== mon.1 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f09e00019f0 con 0x7f09e8073940 2024-04-02T13:20:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 >> v1:172.21.15.69:6789/0 conn(0x7f09e8073940 legacy=0x7f09e8074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f09e559a640 1 --2- 172.21.15.44:0/3906555922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09d00039b0 0x7f09d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f09e559a640 1 --2- 172.21.15.44:0/3906555922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09d0004800 0x7f09d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f09e559a640 1 --2- 172.21.15.44:0/3906555922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815bb20 0x7f09d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09e8074820 con 0x7f09d00039b0 2024-04-02T13:20:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09d0009a90 con 0x7f09d0004800 2024-04-02T13:20:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f09d00145c0 con 0x7f09e815bb20 2024-04-02T13:20:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f09e659c640 1 --2- 172.21.15.44:0/3906555922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09d00039b0 0x7f09d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f09e5d9b640 1 --2- 172.21.15.44:0/3906555922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09d0004800 0x7f09d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f8a4e319640 1 -- 172.21.15.44:0/2783057231 wait complete. 2024-04-02T13:20:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7fbb61be7640 1 -- 172.21.15.44:0/693359740 wait complete. 2024-04-02T13:20:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f7e559b6640 1 -- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5016d970 msgr2=0x7f7e50170e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f81b0b02640 1 -- 172.21.15.44:0/3646271674 >> v1:172.21.15.69:6789/0 conn(0x7f81ac15e230 legacy=0x7f81ac0cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f81b0b02640 1 -- 172.21.15.44:0/3646271674 >> v1:172.21.15.44:6789/0 conn(0x7f81ac072070 legacy=0x7f81ac06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.628+0000 7f81b0b02640 1 -- 172.21.15.44:0/3646271674 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ac0cceb0 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.630+0000 7f09e6d9d640 1 --2- 172.21.15.44:0/3906555922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815bb20 0x7f09d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f7e559b6640 1 --2- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5016d970 0x7f7e50170e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f7e559b6640 1 -- 172.21.15.44:0/2317860512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e5016e290 msgr2=0x7f7e5009aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f7e559b6640 1 --2- 172.21.15.44:0/2317860512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e5016e290 0x7f7e5009aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f7e559b6640 1 -- 172.21.15.44:0/2317860512 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e5016a9f0 con 0x7f7e50071e50 2024-04-02T13:20:47.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f7e559b6640 1 --2- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e50071e50 0x7f7e5009d160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c00e9b0 tx=0x7f7e4c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f7e467fc640 1 -- 172.21.15.44:0/2317860512 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c017070 con 0x7f7e50071e50 2024-04-02T13:20:47.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f7e57c1a640 1 -- 172.21.15.44:0/2317860512 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e5016cc30 con 0x7f7e50071e50 2024-04-02T13:20:47.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.630+0000 7f7e57c1a640 1 -- 172.21.15.44:0/2317860512 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e5016d1d0 con 0x7f7e50071e50 2024-04-02T13:20:47.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.630+0000 7f09e659c640 1 -- 172.21.15.44:0/3906555922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815bb20 msgr2=0x7f09d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:47.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.630+0000 7f09e659c640 1 --2- 172.21.15.44:0/3906555922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815bb20 0x7f09d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.630+0000 7f09e659c640 1 -- 172.21.15.44:0/3906555922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09d0004800 msgr2=0x7f09d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:47.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.630+0000 7f09e659c640 1 --2- 172.21.15.44:0/3906555922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09d0004800 0x7f09d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.630+0000 7f09e659c640 1 -- 172.21.15.44:0/3906555922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d0003520 con 0x7f09d00039b0 2024-04-02T13:20:47.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f97b197c640 1 -- 172.21.15.44:0/480748481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97a4008f80 msgr2=0x7f97a403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.629+0000 7f97b197c640 1 --2- 172.21.15.44:0/480748481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97a4008f80 0x7f97a403a8f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f97ac000b60 tx=0x7f97ac0a6cc0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.630+0000 7f81b0b02640 1 --2- 172.21.15.44:0/3646271674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ac0cbdb0 0x7f81ac15f3f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f81ac15f0e0 tx=0x7f81a802e010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b444390b04ebe447 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.630+0000 7f3e1df2b640 1 Processor -- start 2024-04-02T13:20:47.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.630+0000 7f81a3fff640 1 -- 172.21.15.44:0/3646271674 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a803b070 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.630+0000 7f3e1df2b640 1 -- start start 2024-04-02T13:20:47.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.630+0000 7f81a3fff640 1 -- 172.21.15.44:0/3646271674 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a802ed40 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.630+0000 7f81a3fff640 1 -- 172.21.15.44:0/3646271674 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a8036660 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.631+0000 7f7e467fc640 1 -- 172.21.15.44:0/2317860512 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e4c00c9a0 con 0x7f7e50071e50 2024-04-02T13:20:47.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.631+0000 7f7e467fc640 1 -- 172.21.15.44:0/2317860512 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c00cd90 con 0x7f7e50071e50 2024-04-02T13:20:47.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.631+0000 7fbb61be7640 1 Processor -- start 2024-04-02T13:20:47.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.631+0000 7f8a4e319640 1 Processor -- start 2024-04-02T13:20:47.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.631+0000 7f8a4e319640 1 -- start start 2024-04-02T13:20:47.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.631+0000 7f97b197c640 1 -- 172.21.15.44:0/480748481 shutdown_connections 2024-04-02T13:20:47.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.631+0000 7f97b197c640 1 --2- 172.21.15.44:0/480748481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97a4008f80 0x7f97a403a8f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.631+0000 7f97b197c640 1 --2- 172.21.15.44:0/480748481 >> v2:172.21.15.44:3300/0 conn(0x7f97a40e0050 0x7f97a400b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.631+0000 7f97b197c640 1 -- 172.21.15.44:0/480748481 >> 172.21.15.44:0/480748481 conn(0x7f97a401efa0 msgr2=0x7f97a4055c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:47.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.631+0000 7fbb61be7640 1 -- start start 2024-04-02T13:20:47.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.631+0000 7f81b2305640 1 -- 172.21.15.44:0/3646271674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ac0cbdb0 msgr2=0x7f81ac15f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.631+0000 7f81b2305640 1 --2- 172.21.15.44:0/3646271674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ac0cbdb0 0x7f81ac15f3f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f81ac15f0e0 tx=0x7f81a802e010 comp rx=0 tx=0).stop 2024-04-02T13:20:47.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.631+0000 7f09e659c640 1 --2- 172.21.15.44:0/3906555922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09d00039b0 0x7f09d0003dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f09e000c540 tx=0x7f09d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d509ebe1a8a72d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0039070 con 0x7f09d00039b0 2024-04-02T13:20:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f3e1df2b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e10038110 0x7f3e100d8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f81b2305640 1 -- 172.21.15.44:0/3646271674 shutdown_connections 2024-04-02T13:20:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f81b2305640 1 --2- 172.21.15.44:0/3646271674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ac0cbdb0 0x7f81ac15f3f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f81b2305640 1 -- 172.21.15.44:0/3646271674 >> 172.21.15.44:0/3646271674 conn(0x7f81ac0d0bc0 msgr2=0x7f81ac0d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f7e57c1a640 1 -- 172.21.15.44:0/2317860512 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e500684f0 con 0x7f7e50071e50 2024-04-02T13:20:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f3e1df2b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e1004ffd0 0x7f3e100dc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f3e1df2b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e100dc8d0 0x7f3e100d69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f3e1cf29640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e10038110 0x7f3e100d8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f3e1cf29640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e10038110 0x7f3e100d8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34408/0 (socket says 172.21.15.44:34408) 2024-04-02T13:20:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f3e17fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e1004ffd0 0x7f3e100dc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f3e17fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e1004ffd0 0x7f3e100dc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:42214/0 (socket says 172.21.15.44:42214) 2024-04-02T13:20:47.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f8a4e319640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a40056e60 0x7f8a400e90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f8a4e319640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a40058990 0x7f8a400ea450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f8a4e319640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a400e75c0 0x7f8a40051760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f8a4e319640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8a40007110 con 0x7f8a400e75c0 2024-04-02T13:20:47.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f8a4e319640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a40006e30 con 0x7f8a40058990 2024-04-02T13:20:47.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f8a4e319640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8a400061b0 con 0x7f8a40056e60 2024-04-02T13:20:47.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f8a4d317640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a40056e60 0x7f8a400e90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f8a4d317640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a40056e60 0x7f8a400e90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:42198/0 (socket says 172.21.15.44:42198) 2024-04-02T13:20:47.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f8a4d317640 1 -- 172.21.15.44:0/1978969681 learned_addr learned my addr 172.21.15.44:0/1978969681 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f8a4cb16640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a40058990 0x7f8a400ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f8a4db18640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a400e75c0 0x7f8a40051760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f97b197c640 1 -- 172.21.15.44:0/480748481 shutdown_connections 2024-04-02T13:20:47.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7fbb61be7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb54005ed0 0x7fbb5404e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e002cda0 con 0x7f09d00039b0 2024-04-02T13:20:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f09e559a640 1 -- 172.21.15.44:0/3906555922 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0039070 con 0x7f09d00039b0 2024-04-02T13:20:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f3e1cf29640 1 -- 172.21.15.44:0/1487180064 learned_addr learned my addr 172.21.15.44:0/1487180064 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f3e1df2b640 1 -- 172.21.15.44:0/1487180064 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3e100547c0 con 0x7f3e100dc8d0 2024-04-02T13:20:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.632+0000 7f97b197c640 1 -- 172.21.15.44:0/480748481 wait complete. 2024-04-02T13:20:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f3e1d72a640 1 --2- 172.21.15.44:0/1487180064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e100dc8d0 0x7f3e100d69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f3e1df2b640 1 -- 172.21.15.44:0/1487180064 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3e10038cb0 con 0x7f3e10038110 2024-04-02T13:20:47.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f09e759e640 1 -- 172.21.15.44:0/3906555922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09d00039b0 msgr2=0x7f09d0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f09e759e640 1 --2- 172.21.15.44:0/3906555922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09d00039b0 0x7f09d0003dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f09e000c540 tx=0x7f09d0005830 comp rx=0 tx=0).stop 2024-04-02T13:20:47.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f3e1df2b640 1 -- 172.21.15.44:0/1487180064 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3e10054940 con 0x7f3e1004ffd0 2024-04-02T13:20:47.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f09e6d9d640 1 -- 172.21.15.44:0/3906555922 reap_dead start 2024-04-02T13:20:47.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f8a4d317640 1 -- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a40058990 msgr2=0x7f8a400ea450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f81b2305640 1 -- 172.21.15.44:0/3646271674 shutdown_connections 2024-04-02T13:20:47.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f09e759e640 1 -- 172.21.15.44:0/3906555922 shutdown_connections 2024-04-02T13:20:47.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f09e759e640 1 -- 172.21.15.44:0/3906555922 >> 172.21.15.44:0/3906555922 conn(0x7f09e80d0310 msgr2=0x7f09e80d0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f8a4d317640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a40058990 0x7f8a400ea450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f8a4d317640 1 -- 172.21.15.44:0/1978969681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a400e75c0 msgr2=0x7f8a40051760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f8a4d317640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a400e75c0 0x7f8a40051760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f8a4d317640 1 -- 172.21.15.44:0/1978969681 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a400d9d60 con 0x7f8a40056e60 2024-04-02T13:20:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7fbb60be5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb54005ed0 0x7fbb5404e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f3e1d72a640 1 -- 172.21.15.44:0/1487180064 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e1004ffd0 msgr2=0x7f3e100dc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f3e1d72a640 1 --2- 172.21.15.44:0/1487180064 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e1004ffd0 0x7f3e100dc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f3e1d72a640 1 -- 172.21.15.44:0/1487180064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e10038110 msgr2=0x7f3e100d8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f3e1d72a640 1 --2- 172.21.15.44:0/1487180064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e10038110 0x7f3e100d8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7f3e1d72a640 1 -- 172.21.15.44:0/1487180064 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e100d7730 con 0x7f3e100dc8d0 2024-04-02T13:20:47.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7f8a4d317640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a40056e60 0x7f8a400e90a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8a48050db0 tx=0x7f8a4806c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f81b2305640 1 -- 172.21.15.44:0/3646271674 wait complete. 2024-04-02T13:20:47.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7fbb61be7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb54006a20 0x7fbb540d7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7fbb61be7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb540eacf0 0x7fbb540e4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7f8a3dffb640 1 -- 172.21.15.44:0/1978969681 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a480731b0 con 0x7f8a40056e60 2024-04-02T13:20:47.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f7e467fc640 1 -- 172.21.15.44:0/2317860512 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7e4c01a070 con 0x7f7e50071e50 2024-04-02T13:20:47.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.633+0000 7f7e467fc640 1 --2- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7e3406c9f0 0x7f7e3406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7f7e467fc640 1 -- 172.21.15.44:0/2317860512 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7e4c08bdb0 con 0x7f7e50071e50 2024-04-02T13:20:47.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7f3e1d72a640 1 --2- 172.21.15.44:0/1487180064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e100dc8d0 0x7f3e100d69c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3e18068ba0 tx=0x7f3e18075660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7f7e551b5640 1 --2- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7e3406c9f0 0x7f7e3406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7f97b197c640 1 Processor -- start 2024-04-02T13:20:47.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7fbb61be7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbb540547b0 con 0x7fbb54005ed0 2024-04-02T13:20:47.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7fbb60be5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb54005ed0 0x7fbb5404e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54978/0 (socket says 172.21.15.44:54978) 2024-04-02T13:20:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7fbb60be5640 1 -- 172.21.15.44:0/4116501277 learned_addr learned my addr 172.21.15.44:0/4116501277 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7fbb61be7640 1 -- 172.21.15.44:0/4116501277 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbb54050190 con 0x7fbb540eacf0 2024-04-02T13:20:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7fbb61be7640 1 -- 172.21.15.44:0/4116501277 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbb54054930 con 0x7fbb54006a20 2024-04-02T13:20:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7fbb5bfff640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb54006a20 0x7fbb540d7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7f09e759e640 1 -- 172.21.15.44:0/3906555922 shutdown_connections 2024-04-02T13:20:47.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7fbb613e6640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb540eacf0 0x7fbb540e4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7f3e157fa640 1 -- 172.21.15.44:0/1487180064 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1806cca0 con 0x7f3e100dc8d0 2024-04-02T13:20:47.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7f3e157fa640 1 -- 172.21.15.44:0/1487180064 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e180731c0 con 0x7f3e100dc8d0 2024-04-02T13:20:47.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7f97b197c640 1 -- start start 2024-04-02T13:20:47.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7fbb5bfff640 1 -- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb540eacf0 msgr2=0x7fbb540e4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7fbb5bfff640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb540eacf0 0x7fbb540e4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7fbb5bfff640 1 -- 172.21.15.44:0/4116501277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb54005ed0 msgr2=0x7fbb5404e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7fbb5bfff640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb54005ed0 0x7fbb5404e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7fbb5bfff640 1 -- 172.21.15.44:0/4116501277 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5404d860 con 0x7fbb54006a20 2024-04-02T13:20:47.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7fbb613e6640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb540eacf0 0x7fbb540e4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:47.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7fbb60be5640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb54005ed0 0x7fbb5404e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7f7e551b5640 1 --2- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7e3406c9f0 0x7f7e3406eeb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7e48036000 tx=0x7f7e48030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7fbb5bfff640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb54006a20 0x7fbb540d7ef0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800c370 tx=0x7fbb4800c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7f81b2305640 1 Processor -- start 2024-04-02T13:20:47.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7f81b2305640 1 -- start start 2024-04-02T13:20:47.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7f8a3dffb640 1 -- 172.21.15.44:0/1978969681 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a48073350 con 0x7f8a40056e60 2024-04-02T13:20:47.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7f8a3dffb640 1 -- 172.21.15.44:0/1978969681 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a48073540 con 0x7f8a40056e60 2024-04-02T13:20:47.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.634+0000 7f09e759e640 1 -- 172.21.15.44:0/3906555922 wait complete. 2024-04-02T13:20:47.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f97b197c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97a4008f80 0x7f97a404e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f97b197c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97a403ae30 0x7f97a40d9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f09e759e640 1 Processor -- start 2024-04-02T13:20:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f97b197c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97a40e0050 0x7f97a40e2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f09e759e640 1 -- start start 2024-04-02T13:20:47.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7f3e1df2b640 1 -- 172.21.15.44:0/1487180064 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e100d7a10 con 0x7f3e100dc8d0 2024-04-02T13:20:47.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7f3e1df2b640 1 -- 172.21.15.44:0/1487180064 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e100d7fd0 con 0x7f3e100dc8d0 2024-04-02T13:20:47.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.635+0000 7f3e157fa640 1 -- 172.21.15.44:0/1487180064 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1806ce60 con 0x7f3e100dc8d0 2024-04-02T13:20:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f97b197c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f97a40e0430 con 0x7f97a403ae30 2024-04-02T13:20:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f97b197c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f97a40534d0 con 0x7f97a4008f80 2024-04-02T13:20:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f97b197c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f97a403b640 con 0x7f97a40e0050 2024-04-02T13:20:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f97aaffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97a4008f80 0x7f97a404e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f97aaffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97a4008f80 0x7f97a404e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34422/0 (socket says 172.21.15.44:34422) 2024-04-02T13:20:47.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f97aaffd640 1 -- 172.21.15.44:0/3832816695 learned_addr learned my addr 172.21.15.44:0/3832816695 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f3e02ffd640 1 -- 172.21.15.44:0/1487180064 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3df0005290 con 0x7f3e100dc8d0 2024-04-02T13:20:47.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f8a4e319640 1 -- 172.21.15.44:0/1978969681 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a400e8000 con 0x7f8a40056e60 2024-04-02T13:20:47.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f81b2305640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ac072070 0x7f81ac0a9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f8a4e319640 1 -- 172.21.15.44:0/1978969681 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a400e88a0 con 0x7f8a40056e60 2024-04-02T13:20:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f97ab7fe640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97a40e0050 0x7f97a40e2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.636+0000 7f97aa7fc640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97a403ae30 0x7f97a40d9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f81b2305640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81ac0cbdb0 0x7f81ac0aa7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f81b2305640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81ac15e230 0x7f81ac0aad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f81b2305640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f81ac0724c0 con 0x7f81ac072070 2024-04-02T13:20:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f81b2305640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f81ac15e610 con 0x7f81ac15e230 2024-04-02T13:20:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f81b2305640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f81ac072640 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f81b1303640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ac072070 0x7f81ac0a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f81b0b02640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81ac0cbdb0 0x7f81ac0aa7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f81b1303640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ac072070 0x7f81ac0a9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55006/0 (socket says 172.21.15.44:55006) 2024-04-02T13:20:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f81b1303640 1 -- 172.21.15.44:0/45036759 learned_addr learned my addr 172.21.15.44:0/45036759 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f97aa7fc640 1 -- 172.21.15.44:0/3832816695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97a40e0050 msgr2=0x7f97a40e2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f97aa7fc640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97a40e0050 0x7f97a40e2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f97aa7fc640 1 -- 172.21.15.44:0/3832816695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97a4008f80 msgr2=0x7f97a404e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f97aa7fc640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97a4008f80 0x7f97a404e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f97aa7fc640 1 -- 172.21.15.44:0/3832816695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a41687f0 con 0x7f97a403ae30 2024-04-02T13:20:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7fbb597fa640 1 -- 172.21.15.44:0/4116501277 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480052d0 con 0x7fbb54006a20 2024-04-02T13:20:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7fbb597fa640 1 -- 172.21.15.44:0/4116501277 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb48005490 con 0x7fbb54006a20 2024-04-02T13:20:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7fbb597fa640 1 -- 172.21.15.44:0/4116501277 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48007260 con 0x7fbb54006a20 2024-04-02T13:20:47.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f81b1b04640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81ac15e230 0x7f81ac0aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f97aa7fc640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97a403ae30 0x7f97a40d9ea0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f97ac069c00 tx=0x7f97ac073de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7fbb61be7640 1 -- 172.21.15.44:0/4116501277 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb5404db20 con 0x7fbb54006a20 2024-04-02T13:20:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f09e759e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09d00039b0 0x7f09e80c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f81b0b02640 1 -- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81ac15e230 msgr2=0x7f81ac0aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f81b0b02640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81ac15e230 0x7f81ac0aad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f81b0b02640 1 -- 172.21.15.44:0/45036759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ac072070 msgr2=0x7f81ac0a9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7fbb61be7640 1 -- 172.21.15.44:0/4116501277 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb5404e0e0 con 0x7fbb54006a20 2024-04-02T13:20:47.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f09e759e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80c5470 0x7f09e8155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.637+0000 7f3e157fa640 1 -- 172.21.15.44:0/1487180064 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3e18073470 con 0x7f3e100dc8d0 2024-04-02T13:20:47.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f81b0b02640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ac072070 0x7f81ac0a9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f81b0b02640 1 -- 172.21.15.44:0/45036759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ac0a9f30 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f81b1303640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ac072070 0x7f81ac0a9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:47.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f8a4e319640 1 -- 172.21.15.44:0/1978969681 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a400056d0 con 0x7f8a40056e60 2024-04-02T13:20:47.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f3e157fa640 1 --2- 172.21.15.44:0/1487180064 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3dfc06c920 0x7f3dfc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f3e157fa640 1 -- 172.21.15.44:0/1487180064 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3e18106ab0 con 0x7f3e100dc8d0 2024-04-02T13:20:47.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f09e5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80c5470 0x7f09e8155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f09e5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80c5470 0x7f09e8155660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55020/0 (socket says 172.21.15.44:55020) 2024-04-02T13:20:47.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f81b0b02640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81ac0cbdb0 0x7f81ac0aa7d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81a8035000 tx=0x7f81a80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f09e659c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09d00039b0 0x7f09e80c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f09e659c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09d00039b0 0x7f09e80c6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:42236/0 (socket says 172.21.15.44:42236) 2024-04-02T13:20:47.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f09e5d9b640 1 -- 172.21.15.44:0/2203403994 learned_addr learned my addr 172.21.15.44:0/2203403994 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:47.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f3e1cf29640 1 --2- 172.21.15.44:0/1487180064 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3dfc06c920 0x7f3dfc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f09e759e640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09e808fff0 0x7f09e8090480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f09e759e640 1 -- 172.21.15.44:0/2203403994 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09e80af670 con 0x7f09e80c5470 2024-04-02T13:20:47.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f09e759e640 1 -- 172.21.15.44:0/2203403994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09e806c320 con 0x7f09e808fff0 2024-04-02T13:20:47.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.638+0000 7f09e759e640 1 -- 172.21.15.44:0/2203403994 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f09e8071a30 con 0x7f09d00039b0 2024-04-02T13:20:47.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.639+0000 7f09e6d9d640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09e808fff0 0x7f09e8090480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.639+0000 7f7e467fc640 1 -- 172.21.15.44:0/2317860512 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7e5016a9f0 con 0x7f7e50071e50 2024-04-02T13:20:47.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.639+0000 7f81a1ffb640 1 -- 172.21.15.44:0/45036759 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a803b040 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.639+0000 7f81b2305640 1 -- 172.21.15.44:0/45036759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81ac0aa210 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.639+0000 7f09e5d9b640 1 -- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09d00039b0 msgr2=0x7f09e80c6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.639+0000 7f09e5d9b640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09d00039b0 0x7f09e80c6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.639+0000 7f09e5d9b640 1 -- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09e808fff0 msgr2=0x7f09e8090480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:47.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.639+0000 7f81b2305640 1 -- 172.21.15.44:0/45036759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f81ac158c50 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.639+0000 7f09e5d9b640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09e808fff0 0x7f09e8090480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:47.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.639+0000 7f09e5d9b640 1 -- 172.21.15.44:0/2203403994 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e80c6c20 con 0x7f09e80c5470 2024-04-02T13:20:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.640+0000 7f3e1cf29640 1 --2- 172.21.15.44:0/1487180064 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3dfc06c920 0x7f3dfc06ede0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c02b930 tx=0x7f3e0c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.640+0000 7fbb61be7640 1 -- 172.21.15.44:0/4116501277 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb540052b0 con 0x7fbb54006a20 2024-04-02T13:20:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.640+0000 7fbb597fa640 1 -- 172.21.15.44:0/4116501277 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbb48007400 con 0x7fbb54006a20 2024-04-02T13:20:47.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.640+0000 7f09e5d9b640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80c5470 0x7f09e8155660 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0119a0 tx=0x7f09dc011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.640+0000 7fbb597fa640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbb4006c9f0 0x7fbb4006eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.640+0000 7f8a3dffb640 1 -- 172.21.15.44:0/1978969681 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8a480736e0 con 0x7f8a40056e60 2024-04-02T13:20:47.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.640+0000 7fbb597fa640 1 -- 172.21.15.44:0/4116501277 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbb4808b2a0 con 0x7fbb54006a20 2024-04-02T13:20:47.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.640+0000 7fbb60be5640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbb4006c9f0 0x7fbb4006eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.641+0000 7f8a3dffb640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8a2c06c940 0x7f8a2c06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.641+0000 7f81b2305640 1 -- 172.21.15.44:0/45036759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81ac04f8d0 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.641+0000 7f8a3dffb640 1 -- 172.21.15.44:0/1978969681 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8a48106da0 con 0x7f8a40056e60 2024-04-02T13:20:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.641+0000 7f8a4cb16640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8a2c06c940 0x7f8a2c06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.641+0000 7fbb60be5640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbb4006c9f0 0x7fbb4006eeb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbb50000f60 tx=0x7fbb50007ea0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.641+0000 7f8a4cb16640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8a2c06c940 0x7f8a2c06ee00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8a40054230 tx=0x7f8a44008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.644+0000 7f81a1ffb640 1 -- 172.21.15.44:0/45036759 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a802ed40 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.644+0000 7f8a3dffb640 1 -- 172.21.15.44:0/1978969681 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8a48090070 con 0x7f8a40056e60 2024-04-02T13:20:47.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.645+0000 7f81a1ffb640 1 -- 172.21.15.44:0/45036759 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a8034970 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.648+0000 7f3e157fa640 1 -- 172.21.15.44:0/1487180064 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3e1808f030 con 0x7f3e100dc8d0 2024-04-02T13:20:47.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.647+0000 7f9793fff640 1 -- 172.21.15.44:0/3832816695 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0b4050 con 0x7f97a403ae30 2024-04-02T13:20:47.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.648+0000 7f97b197c640 1 -- 172.21.15.44:0/3832816695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a4168a90 con 0x7f97a403ae30 2024-04-02T13:20:47.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.648+0000 7f97b197c640 1 -- 172.21.15.44:0/3832816695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a4169050 con 0x7f97a403ae30 2024-04-02T13:20:47.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.647+0000 7f09d6ffd640 1 -- 172.21.15.44:0/2203403994 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc00faf0 con 0x7f09e80c5470 2024-04-02T13:20:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.648+0000 7f09e759e640 1 -- 172.21.15.44:0/2203403994 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e80b3200 con 0x7f09e80c5470 2024-04-02T13:20:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.648+0000 7f09e759e640 1 -- 172.21.15.44:0/2203403994 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e80b37c0 con 0x7f09e80c5470 2024-04-02T13:20:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.649+0000 7f09e759e640 1 -- 172.21.15.44:0/2203403994 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09e804f8d0 con 0x7f09e80c5470 2024-04-02T13:20:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.649+0000 7f9793fff640 1 -- 172.21.15.44:0/3832816695 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97ac0c0070 con 0x7f97a403ae30 2024-04-02T13:20:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.649+0000 7f9793fff640 1 -- 172.21.15.44:0/3832816695 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0b9a10 con 0x7f97a403ae30 2024-04-02T13:20:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.649+0000 7fbb597fa640 1 -- 172.21.15.44:0/4116501277 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbb48058d20 con 0x7fbb54006a20 2024-04-02T13:20:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.650+0000 7f97b197c640 1 -- 172.21.15.44:0/3832816695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a4005740 con 0x7f97a403ae30 2024-04-02T13:20:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.651+0000 7f81a1ffb640 1 -- 172.21.15.44:0/45036759 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f81a8034b10 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.651+0000 7f09d6ffd640 1 -- 172.21.15.44:0/2203403994 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09dc00fc90 con 0x7f09e80c5470 2024-04-02T13:20:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.651+0000 7f09d6ffd640 1 -- 172.21.15.44:0/2203403994 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc014440 con 0x7f09e80c5470 2024-04-02T13:20:47.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.651+0000 7f81a1ffb640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f819006c8f0 0x7f819006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.651+0000 7f81a1ffb640 1 -- 172.21.15.44:0/45036759 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f81a80aeb30 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.652+0000 7f81b1303640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f819006c8f0 0x7f819006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.652+0000 7f09d6ffd640 1 -- 172.21.15.44:0/2203403994 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f09dc012040 con 0x7f09e80c5470 2024-04-02T13:20:47.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.653+0000 7f09d6ffd640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09c806c920 0x7f09c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.653+0000 7f09d6ffd640 1 -- 172.21.15.44:0/2203403994 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f09dc08c3d0 con 0x7f09e80c5470 2024-04-02T13:20:47.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.654+0000 7f09e659c640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09c806c920 0x7f09c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.654+0000 7f81b1303640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f819006c8f0 0x7f819006edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f819c001720 tx=0x7f819c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.654+0000 7f9793fff640 1 -- 172.21.15.44:0/3832816695 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f97ac0b9bb0 con 0x7f97a403ae30 2024-04-02T13:20:47.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.654+0000 7f9793fff640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f979806c8f0 0x7f979806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:47.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.654+0000 7f09e659c640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09c806c920 0x7f09c806ede0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f09e0036000 tx=0x7f09e0009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.654+0000 7f9793fff640 1 -- 172.21.15.44:0/3832816695 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f97ac129fb0 con 0x7f97a403ae30 2024-04-02T13:20:47.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.655+0000 7f97aaffd640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f979806c8f0 0x7f979806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:47.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.655+0000 7f81a1ffb640 1 -- 172.21.15.44:0/45036759 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f81a807cd30 con 0x7f81ac0cbdb0 2024-04-02T13:20:47.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.655+0000 7f97aaffd640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f979806c8f0 0x7f979806edb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f979c00bb10 tx=0x7f979c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:47.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.662+0000 7f9793fff640 1 -- 172.21.15.44:0/3832816695 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f97a400cb60 con 0x7f97a403ae30 2024-04-02T13:20:47.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:47.671+0000 7f09d6ffd640 1 -- 172.21.15.44:0/2203403994 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f09dc019030 con 0x7f09e80c5470 2024-04-02T13:20:48.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.009+0000 7f83b1722640 1 -- 172.21.15.44:0/3067324230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f838c005580 con 0x7f83a4050600 2024-04-02T13:20:48.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.010+0000 7f83937fe640 1 -- 172.21.15.44:0/3067324230 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f83ac0f7040 con 0x7f83a4050600 2024-04-02T13:20:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:55834574856 2024-04-02T13:20:48.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.013+0000 7f83917fa640 1 -- 172.21.15.44:0/3067324230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f839406cb30 msgr2=0x7f839406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.014+0000 7f83917fa640 1 --2- 172.21.15.44:0/3067324230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f839406cb30 0x7f839406eff0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f839c006fd0 tx=0x7f839c008040 comp rx=0 tx=0).stop 2024-04-02T13:20:48.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.014+0000 7f83917fa640 1 -- 172.21.15.44:0/3067324230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83a4050600 msgr2=0x7f83a404dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.014+0000 7f83917fa640 1 --2- 172.21.15.44:0/3067324230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83a4050600 0x7f83a404dbb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f83ac050610 tx=0x7f83ac069ee0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.014+0000 7f83917fa640 1 -- 172.21.15.44:0/3067324230 shutdown_connections 2024-04-02T13:20:48.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.014+0000 7f83917fa640 1 --2- 172.21.15.44:0/3067324230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f839406cb30 0x7f839406eff0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.014+0000 7f83917fa640 1 --2- 172.21.15.44:0/3067324230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83a4053ef0 0x7f83a417d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.015+0000 7f83917fa640 1 --2- 172.21.15.44:0/3067324230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83a4050600 0x7f83a404dbb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.015+0000 7f83917fa640 1 --2- 172.21.15.44:0/3067324230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83a4034fd0 0x7f83a40f0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.015+0000 7f83917fa640 1 -- 172.21.15.44:0/3067324230 >> 172.21.15.44:0/3067324230 conn(0x7f83a4055da0 msgr2=0x7f83a40e0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:48.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.015+0000 7f83917fa640 1 -- 172.21.15.44:0/3067324230 shutdown_connections 2024-04-02T13:20:48.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.015+0000 7f83917fa640 1 -- 172.21.15.44:0/3067324230 wait complete. 2024-04-02T13:20:48.034 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2024-04-02T13:20:48.034 DEBUG:teuthology.parallel:result is None 2024-04-02T13:20:48.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.048+0000 7f8a4e319640 1 -- 172.21.15.44:0/1978969681 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8a4000e0c0 con 0x7f8a40056e60 2024-04-02T13:20:48.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.048+0000 7f8a3dffb640 1 -- 172.21.15.44:0/1978969681 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8a480d48b0 con 0x7f8a40056e60 2024-04-02T13:20:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:55834574857 2024-04-02T13:20:48.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.051+0000 7f8a4e319640 1 -- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8a2c06c940 msgr2=0x7f8a2c06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.051+0000 7f8a4e319640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8a2c06c940 0x7f8a2c06ee00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8a40054230 tx=0x7f8a44008040 comp rx=0 tx=0).stop 2024-04-02T13:20:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.051+0000 7f8a4e319640 1 -- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a40056e60 msgr2=0x7f8a400e90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.051+0000 7f8a4e319640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a40056e60 0x7f8a400e90a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8a48050db0 tx=0x7f8a4806c8b0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.051+0000 7f8a4e319640 1 -- 172.21.15.44:0/1978969681 shutdown_connections 2024-04-02T13:20:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.051+0000 7f8a4e319640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8a2c06c940 0x7f8a2c06ee00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.051+0000 7f8a4e319640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a400e75c0 0x7f8a40051760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.051+0000 7f8a4e319640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a40058990 0x7f8a400ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.051+0000 7f8a4e319640 1 --2- 172.21.15.44:0/1978969681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a40056e60 0x7f8a400e90a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.051+0000 7f8a4e319640 1 -- 172.21.15.44:0/1978969681 >> 172.21.15.44:0/1978969681 conn(0x7f8a40058eb0 msgr2=0x7f8a400e3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.051+0000 7f8a4e319640 1 -- 172.21.15.44:0/1978969681 shutdown_connections 2024-04-02T13:20:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.051+0000 7f8a4e319640 1 -- 172.21.15.44:0/1978969681 wait complete. 2024-04-02T13:20:48.064 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.0 2024-04-02T13:20:48.064 DEBUG:teuthology.parallel:result is None 2024-04-02T13:20:48.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.070+0000 7f7e57c1a640 1 -- 172.21.15.44:0/2317860512 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f7e5008b0a0 con 0x7f7e50071e50 2024-04-02T13:20:48.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.070+0000 7f7e467fc640 1 -- 172.21.15.44:0/2317860512 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7e4c0598c0 con 0x7f7e50071e50 2024-04-02T13:20:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:55834574857 2024-04-02T13:20:48.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.072+0000 7f7e57c1a640 1 -- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7e3406c9f0 msgr2=0x7f7e3406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.072+0000 7f7e57c1a640 1 --2- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7e3406c9f0 0x7f7e3406eeb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7e48036000 tx=0x7f7e48030000 comp rx=0 tx=0).stop 2024-04-02T13:20:48.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.072+0000 7f7e57c1a640 1 -- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e50071e50 msgr2=0x7f7e5009d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.072+0000 7f7e57c1a640 1 --2- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e50071e50 0x7f7e5009d160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c00e9b0 tx=0x7f7e4c00ee70 comp rx=0 tx=0).stop 2024-04-02T13:20:48.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.073+0000 7f7e57c1a640 1 -- 172.21.15.44:0/2317860512 shutdown_connections 2024-04-02T13:20:48.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.073+0000 7f7e57c1a640 1 --2- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7e3406c9f0 0x7f7e3406eeb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.073+0000 7f7e57c1a640 1 --2- 172.21.15.44:0/2317860512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e5016e290 0x7f7e5009aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.073+0000 7f7e57c1a640 1 --2- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5016d970 0x7f7e50170e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.073+0000 7f7e57c1a640 1 --2- 172.21.15.44:0/2317860512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e50071e50 0x7f7e5009d160 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.073+0000 7f7e57c1a640 1 -- 172.21.15.44:0/2317860512 >> 172.21.15.44:0/2317860512 conn(0x7f7e500cfcd0 msgr2=0x7f7e500b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:48.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.073+0000 7f7e57c1a640 1 -- 172.21.15.44:0/2317860512 shutdown_connections 2024-04-02T13:20:48.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.073+0000 7f7e57c1a640 1 -- 172.21.15.44:0/2317860512 wait complete. 2024-04-02T13:20:48.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.077+0000 7fbb61be7640 1 -- 172.21.15.44:0/4116501277 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fbb54038900 con 0x7fbb54006a20 2024-04-02T13:20:48.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.078+0000 7fbb597fa640 1 -- 172.21.15.44:0/4116501277 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbb4805f020 con 0x7fbb54006a20 2024-04-02T13:20:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:55834574857 2024-04-02T13:20:48.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.080+0000 7f3e02ffd640 1 -- 172.21.15.44:0/1487180064 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f3df0005580 con 0x7f3e100dc8d0 2024-04-02T13:20:48.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.081+0000 7f3e157fa640 1 -- 172.21.15.44:0/1487180064 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3e180d4a70 con 0x7f3e100dc8d0 2024-04-02T13:20:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:55834574857 2024-04-02T13:20:48.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.081+0000 7fbb61be7640 1 -- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbb4006c9f0 msgr2=0x7fbb4006eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.081+0000 7fbb61be7640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbb4006c9f0 0x7fbb4006eeb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbb50000f60 tx=0x7fbb50007ea0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.081+0000 7fbb61be7640 1 -- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb54006a20 msgr2=0x7fbb540d7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.081+0000 7fbb61be7640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb54006a20 0x7fbb540d7ef0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800c370 tx=0x7fbb4800c830 comp rx=0 tx=0).stop 2024-04-02T13:20:48.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.081+0000 7fbb61be7640 1 -- 172.21.15.44:0/4116501277 shutdown_connections 2024-04-02T13:20:48.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.082+0000 7fbb61be7640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbb4006c9f0 0x7fbb4006eeb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.082+0000 7fbb61be7640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb540eacf0 0x7fbb540e4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.082+0000 7fbb61be7640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb54006a20 0x7fbb540d7ef0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.082+0000 7fbb61be7640 1 --2- 172.21.15.44:0/4116501277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb54005ed0 0x7fbb5404e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.082+0000 7fbb61be7640 1 -- 172.21.15.44:0/4116501277 >> 172.21.15.44:0/4116501277 conn(0x7fbb54009fa0 msgr2=0x7fbb54006e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:48.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.082+0000 7fbb61be7640 1 -- 172.21.15.44:0/4116501277 shutdown_connections 2024-04-02T13:20:48.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.082+0000 7fbb61be7640 1 -- 172.21.15.44:0/4116501277 wait complete. 2024-04-02T13:20:48.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.083+0000 7f97b197c640 1 -- 172.21.15.44:0/3832816695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f97a4000b90 con 0x7f97a403ae30 2024-04-02T13:20:48.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.083+0000 7f9793fff640 1 -- 172.21.15.44:0/3832816695 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f97ac0f7ac0 con 0x7f97a403ae30 2024-04-02T13:20:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:55834574857 2024-04-02T13:20:48.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f3e02ffd640 1 -- 172.21.15.44:0/1487180064 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3dfc06c920 msgr2=0x7f3dfc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f3e02ffd640 1 --2- 172.21.15.44:0/1487180064 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3dfc06c920 0x7f3dfc06ede0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c02b930 tx=0x7f3e0c03c040 comp rx=0 tx=0).stop 2024-04-02T13:20:48.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f3e02ffd640 1 -- 172.21.15.44:0/1487180064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e100dc8d0 msgr2=0x7f3e100d69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f3e02ffd640 1 --2- 172.21.15.44:0/1487180064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e100dc8d0 0x7f3e100d69c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3e18068ba0 tx=0x7f3e18075660 comp rx=0 tx=0).stop 2024-04-02T13:20:48.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f3e02ffd640 1 -- 172.21.15.44:0/1487180064 shutdown_connections 2024-04-02T13:20:48.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f3e02ffd640 1 --2- 172.21.15.44:0/1487180064 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3dfc06c920 0x7f3dfc06ede0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f3e02ffd640 1 --2- 172.21.15.44:0/1487180064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e100dc8d0 0x7f3e100d69c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f3e02ffd640 1 --2- 172.21.15.44:0/1487180064 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e1004ffd0 0x7f3e100dc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f3e02ffd640 1 --2- 172.21.15.44:0/1487180064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e10038110 0x7f3e100d8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f3e02ffd640 1 -- 172.21.15.44:0/1487180064 >> 172.21.15.44:0/1487180064 conn(0x7f3e10009fb0 msgr2=0x7f3e10036430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:48.088 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2024-04-02T13:20:48.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f3e02ffd640 1 -- 172.21.15.44:0/1487180064 shutdown_connections 2024-04-02T13:20:48.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f3e02ffd640 1 -- 172.21.15.44:0/1487180064 wait complete. 2024-04-02T13:20:48.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f81b2305640 1 -- 172.21.15.44:0/45036759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f81ac0b1000 con 0x7f81ac0cbdb0 2024-04-02T13:20:48.089 DEBUG:teuthology.parallel:result is None 2024-04-02T13:20:48.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f97b197c640 1 -- 172.21.15.44:0/3832816695 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f979806c8f0 msgr2=0x7f979806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f97b197c640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f979806c8f0 0x7f979806edb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f979c00bb10 tx=0x7f979c008040 comp rx=0 tx=0).stop 2024-04-02T13:20:48.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f97b197c640 1 -- 172.21.15.44:0/3832816695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97a403ae30 msgr2=0x7f97a40d9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f97b197c640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97a403ae30 0x7f97a40d9ea0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f97ac069c00 tx=0x7f97ac073de0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f97b197c640 1 -- 172.21.15.44:0/3832816695 shutdown_connections 2024-04-02T13:20:48.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f97b197c640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f979806c8f0 0x7f979806edb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f97b197c640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97a40e0050 0x7f97a40e2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f97b197c640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97a403ae30 0x7f97a40d9ea0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f97b197c640 1 --2- 172.21.15.44:0/3832816695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97a4008f80 0x7f97a404e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.085+0000 7f97b197c640 1 -- 172.21.15.44:0/3832816695 >> 172.21.15.44:0/3832816695 conn(0x7f97a401efa0 msgr2=0x7f97a4055220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:48.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.086+0000 7f97b197c640 1 -- 172.21.15.44:0/3832816695 shutdown_connections 2024-04-02T13:20:48.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.086+0000 7f81a1ffb640 1 -- 172.21.15.44:0/45036759 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f81a807c630 con 0x7f81ac0cbdb0 2024-04-02T13:20:48.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.086+0000 7f97b197c640 1 -- 172.21.15.44:0/3832816695 wait complete. 2024-04-02T13:20:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:55834574856 2024-04-02T13:20:48.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.088+0000 7f81b2305640 1 -- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f819006c8f0 msgr2=0x7f819006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.088+0000 7f81b2305640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f819006c8f0 0x7f819006edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f819c001720 tx=0x7f819c009040 comp rx=0 tx=0).stop 2024-04-02T13:20:48.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.088+0000 7f81b2305640 1 -- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81ac0cbdb0 msgr2=0x7f81ac0aa7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.088+0000 7f81b2305640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81ac0cbdb0 0x7f81ac0aa7d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81a8035000 tx=0x7f81a80046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.088+0000 7f81b2305640 1 -- 172.21.15.44:0/45036759 shutdown_connections 2024-04-02T13:20:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.088+0000 7f81b2305640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f819006c8f0 0x7f819006edb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.088+0000 7f81b2305640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81ac15e230 0x7f81ac0aad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.088+0000 7f81b2305640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81ac0cbdb0 0x7f81ac0aa7d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.088+0000 7f81b2305640 1 --2- 172.21.15.44:0/45036759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ac072070 0x7f81ac0a9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.088+0000 7f81b2305640 1 -- 172.21.15.44:0/45036759 >> 172.21.15.44:0/45036759 conn(0x7f81ac0d0bc0 msgr2=0x7f81ac0b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.088+0000 7f81b2305640 1 -- 172.21.15.44:0/45036759 shutdown_connections 2024-04-02T13:20:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.088+0000 7f81b2305640 1 -- 172.21.15.44:0/45036759 wait complete. 2024-04-02T13:20:48.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.094+0000 7f09e759e640 1 -- 172.21.15.44:0/2203403994 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f09e80cdc20 con 0x7f09e80c5470 2024-04-02T13:20:48.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.094+0000 7f09d6ffd640 1 -- 172.21.15.44:0/2203403994 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f09dc05a4b0 con 0x7f09e80c5470 2024-04-02T13:20:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:55834574856 2024-04-02T13:20:48.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.097+0000 7f09e759e640 1 -- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09c806c920 msgr2=0x7f09c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.097+0000 7f09e759e640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09c806c920 0x7f09c806ede0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f09e0036000 tx=0x7f09e0009030 comp rx=0 tx=0).stop 2024-04-02T13:20:48.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.097+0000 7f09e759e640 1 -- 172.21.15.44:0/2203403994 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80c5470 msgr2=0x7f09e8155660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.097+0000 7f09e759e640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80c5470 0x7f09e8155660 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0119a0 tx=0x7f09dc011da0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.097+0000 7f09e759e640 1 -- 172.21.15.44:0/2203403994 shutdown_connections 2024-04-02T13:20:48.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.097+0000 7f09e759e640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09c806c920 0x7f09c806ede0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.097+0000 7f09e759e640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09e808fff0 0x7f09e8090480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.097+0000 7f09e759e640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80c5470 0x7f09e8155660 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0119a0 tx=0x7f09dc011da0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.097+0000 7f09e759e640 1 --2- 172.21.15.44:0/2203403994 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09d00039b0 0x7f09e80c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.097+0000 7f09e759e640 1 -- 172.21.15.44:0/2203403994 >> 172.21.15.44:0/2203403994 conn(0x7f09e80d0310 msgr2=0x7f09e80d06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:48.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.097+0000 7f09e759e640 1 -- 172.21.15.44:0/2203403994 shutdown_connections 2024-04-02T13:20:48.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.097+0000 7f09e759e640 1 -- 172.21.15.44:0/2203403994 wait complete. 2024-04-02T13:20:48.101 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.5 2024-04-02T13:20:48.101 DEBUG:teuthology.parallel:result is None 2024-04-02T13:20:48.105 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2024-04-02T13:20:48.105 DEBUG:teuthology.parallel:result is None 2024-04-02T13:20:48.107 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2024-04-02T13:20:48.108 DEBUG:teuthology.parallel:result is None 2024-04-02T13:20:48.108 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2024-04-02T13:20:48.108 DEBUG:teuthology.parallel:result is None 2024-04-02T13:20:48.112 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2024-04-02T13:20:48.112 DEBUG:teuthology.parallel:result is None 2024-04-02T13:20:48.112 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-02T13:20:48.112 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:20:48.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.250+0000 7f4c57eb1640 1 Processor -- start 2024-04-02T13:20:48.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.251+0000 7f4c57eb1640 1 -- start start 2024-04-02T13:20:48.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.251+0000 7f4c57eb1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4c500d0a30 0x7f4c5006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.251+0000 7f4c57eb1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c500d0000 con 0x7f4c500d2040 2024-04-02T13:20:48.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.251+0000 7f4c57eb1640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c5006c1d0 con 0x7f4c500d0e10 2024-04-02T13:20:48.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.251+0000 7f4c57eb1640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4c5006cb00 con 0x7f4c500d0a30 2024-04-02T13:20:48.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.251+0000 7f4c5644e640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4c500d2040 0x7f4c500d24f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57520/0 (socket says 172.21.15.44:57520) 2024-04-02T13:20:48.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.251+0000 7f4c5644e640 1 -- 172.21.15.44:0/1792877368 learned_addr learned my addr 172.21.15.44:0/1792877368 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:48.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c55c4d640 1 --2- 172.21.15.44:0/1792877368 >> v2:172.21.15.69:3300/0 conn(0x7f4c500d0a30 0x7f4c5006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3373923056 0 0) 0x7f4c500d0000 con 0x7f4c500d2040 2024-04-02T13:20:48.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c3c003520 con 0x7f4c500d2040 2024-04-02T13:20:48.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2889094464 0 0) 0x7f4c3c003520 con 0x7f4c500d2040 2024-04-02T13:20:48.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 >> v2:172.21.15.69:3300/0 conn(0x7f4c500d0a30 msgr2=0x7f4c5006cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:48.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 --2- 172.21.15.44:0/1792877368 >> v2:172.21.15.69:3300/0 conn(0x7f4c500d0a30 0x7f4c5006cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 >> v1:172.21.15.69:6789/0 conn(0x7f4c500d0e10 legacy=0x7f4c500cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5006c4b0 con 0x7f4c500d2040 2024-04-02T13:20:48.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f4c480019f0 con 0x7f4c500d2040 2024-04-02T13:20:48.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 >> v1:172.21.15.44:6789/0 conn(0x7f4c500d2040 legacy=0x7f4c500d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 --2- 172.21.15.44:0/1792877368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c0039b0 0x7f4c3c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 --2- 172.21.15.44:0/1792877368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c0043b0 0x7f4c3c004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 --2- 172.21.15.44:0/1792877368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c3c005120 0x7f4c3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5006cb00 con 0x7f4c3c0039b0 2024-04-02T13:20:48.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3c00c060 con 0x7f4c3c0043b0 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4c3c00c260 con 0x7f4c3c005120 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c55c4d640 1 --2- 172.21.15.44:0/1792877368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c0039b0 0x7f4c3c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c5544c640 1 --2- 172.21.15.44:0/1792877368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c0043b0 0x7f4c3c004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.252+0000 7f4c5644e640 1 --2- 172.21.15.44:0/1792877368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c3c005120 0x7f4c3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.253+0000 7f4c55c4d640 1 -- 172.21.15.44:0/1792877368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c3c005120 msgr2=0x7f4c3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.253+0000 7f4c55c4d640 1 --2- 172.21.15.44:0/1792877368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c3c005120 0x7f4c3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.253+0000 7f4c55c4d640 1 -- 172.21.15.44:0/1792877368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c0043b0 msgr2=0x7f4c3c004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.253+0000 7f4c55c4d640 1 --2- 172.21.15.44:0/1792877368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c0043b0 0x7f4c3c004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.253+0000 7f4c55c4d640 1 -- 172.21.15.44:0/1792877368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c3c003520 con 0x7f4c3c0039b0 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.253+0000 7f4c5644e640 1 -- 172.21.15.44:0/1792877368 reap_dead start 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.253+0000 7f4c55c4d640 1 --2- 172.21.15.44:0/1792877368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c0039b0 0x7f4c3c003d90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009ac0 tx=0x7f4c3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a4eac953d36b12 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.253+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c039070 con 0x7f4c3c0039b0 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.253+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c4c02c6b0 con 0x7f4c3c0039b0 2024-04-02T13:20:48.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.253+0000 7f4c54c4b640 1 -- 172.21.15.44:0/1792877368 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c009690 con 0x7f4c3c0039b0 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.254+0000 7f4c57eb1640 1 -- 172.21.15.44:0/1792877368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c0039b0 msgr2=0x7f4c3c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.254+0000 7f4c57eb1640 1 --2- 172.21.15.44:0/1792877368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c0039b0 0x7f4c3c003d90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009ac0 tx=0x7f4c3c005830 comp rx=0 tx=0).stop 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.254+0000 7f4c57eb1640 1 -- 172.21.15.44:0/1792877368 shutdown_connections 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.254+0000 7f4c57eb1640 1 --2- 172.21.15.44:0/1792877368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c0039b0 0x7f4c3c003d90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.254+0000 7f4c57eb1640 1 -- 172.21.15.44:0/1792877368 >> 172.21.15.44:0/1792877368 conn(0x7f4c5009d470 msgr2=0x7f4c500d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.254+0000 7f4c57eb1640 1 -- 172.21.15.44:0/1792877368 shutdown_connections 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.254+0000 7f4c57eb1640 1 -- 172.21.15.44:0/1792877368 wait complete. 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.255+0000 7f4c57eb1640 1 Processor -- start 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.255+0000 7f4c57eb1640 1 -- start start 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c57eb1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c0039b0 0x7f4c50151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c57eb1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c500b4590 0x7f4c500cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c57eb1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c500b4a10 0x7f4c500cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c57eb1640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5015e6f0 con 0x7f4c500b4590 2024-04-02T13:20:48.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c57eb1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4c500d1270 con 0x7f4c3c0039b0 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c57eb1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4c5015e870 con 0x7f4c500b4a10 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c5544c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c500b4590 0x7f4c500cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c5544c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c500b4590 0x7f4c500cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55050/0 (socket says 172.21.15.44:55050) 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c5544c640 1 -- 172.21.15.44:0/2234432495 learned_addr learned my addr 172.21.15.44:0/2234432495 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c55c4d640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c0039b0 0x7f4c50151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c5644e640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c500b4a10 0x7f4c500cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c5544c640 1 -- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c500b4a10 msgr2=0x7f4c500cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c5544c640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c500b4a10 0x7f4c500cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c5544c640 1 -- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c0039b0 msgr2=0x7f4c50151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c5544c640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c0039b0 0x7f4c50151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.256+0000 7f4c5544c640 1 -- 172.21.15.44:0/2234432495 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c500cb390 con 0x7f4c500b4590 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.257+0000 7f4c5544c640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c500b4590 0x7f4c500cd4a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000f8a0 tx=0x7f4c4000fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.257+0000 7f4c467fc640 1 -- 172.21.15.44:0/2234432495 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4001a070 con 0x7f4c500b4590 2024-04-02T13:20:48.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.257+0000 7f4c467fc640 1 -- 172.21.15.44:0/2234432495 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c4000fe90 con 0x7f4c500b4590 2024-04-02T13:20:48.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.257+0000 7f4c57eb1640 1 -- 172.21.15.44:0/2234432495 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c500cb670 con 0x7f4c500b4590 2024-04-02T13:20:48.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.257+0000 7f4c467fc640 1 -- 172.21.15.44:0/2234432495 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c40007050 con 0x7f4c500b4590 2024-04-02T13:20:48.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.257+0000 7f4c57eb1640 1 -- 172.21.15.44:0/2234432495 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c500cbc30 con 0x7f4c500b4590 2024-04-02T13:20:48.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.258+0000 7f4c57eb1640 1 -- 172.21.15.44:0/2234432495 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c500684f0 con 0x7f4c500b4590 2024-04-02T13:20:48.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.258+0000 7f4c467fc640 1 -- 172.21.15.44:0/2234432495 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4c4001d070 con 0x7f4c500b4590 2024-04-02T13:20:48.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.258+0000 7f4c467fc640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4c3406c8f0 0x7f4c3406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.259+0000 7f4c467fc640 1 -- 172.21.15.44:0/2234432495 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4c4008b510 con 0x7f4c500b4590 2024-04-02T13:20:48.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.259+0000 7f4c55c4d640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4c3406c8f0 0x7f4c3406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.259+0000 7f4c55c4d640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4c3406c8f0 0x7f4c3406edb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c036000 tx=0x7f4c4c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:48.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.263+0000 7f4c467fc640 1 -- 172.21.15.44:0/2234432495 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4c40059040 con 0x7f4c500b4590 2024-04-02T13:20:48.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.444+0000 7f4c57eb1640 1 -- 172.21.15.44:0/2234432495 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4c500cdf20 con 0x7f4c3406c8f0 2024-04-02T13:20:48.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.453+0000 7f4c467fc640 1 -- 172.21.15.44:0/2234432495 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7f4c500cdf20 con 0x7f4c3406c8f0 2024-04-02T13:20:48.458 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:48.460 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:20:48.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.456+0000 7f4c57eb1640 1 -- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4c3406c8f0 msgr2=0x7f4c3406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.456+0000 7f4c57eb1640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4c3406c8f0 0x7f4c3406edb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c036000 tx=0x7f4c4c030000 comp rx=0 tx=0).stop 2024-04-02T13:20:48.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.456+0000 7f4c57eb1640 1 -- 172.21.15.44:0/2234432495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c500b4590 msgr2=0x7f4c500cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.456+0000 7f4c57eb1640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c500b4590 0x7f4c500cd4a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000f8a0 tx=0x7f4c4000fca0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.456+0000 7f4c57eb1640 1 -- 172.21.15.44:0/2234432495 shutdown_connections 2024-04-02T13:20:48.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.456+0000 7f4c57eb1640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4c3406c8f0 0x7f4c3406edb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.456+0000 7f4c57eb1640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c500b4a10 0x7f4c500cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.456+0000 7f4c57eb1640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c500b4590 0x7f4c500cd4a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.456+0000 7f4c57eb1640 1 --2- 172.21.15.44:0/2234432495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c0039b0 0x7f4c50151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.456+0000 7f4c57eb1640 1 -- 172.21.15.44:0/2234432495 >> 172.21.15.44:0/2234432495 conn(0x7f4c5009d470 msgr2=0x7f4c500b2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:48.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.456+0000 7f4c57eb1640 1 -- 172.21.15.44:0/2234432495 shutdown_connections 2024-04-02T13:20:48.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.456+0000 7f4c57eb1640 1 -- 172.21.15.44:0/2234432495 wait complete. 2024-04-02T13:20:48.469 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-04-02T13:20:46.879424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219496,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755224,"statfs":{"total":773094113280,"available":772869349376,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003122"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431510+0000","last_change":"2024-04-02T13:20:19.889381+0000","last_active":"2024-04-02T13:20:26.431510+0000","last_peered":"2024-04-02T13:20:26.431510+0000","last_clean":"2024-04-02T13:20:26.431510+0000","last_became_active":"2024-04-02T13:20:19.889109+0000","last_became_peered":"2024-04-02T13:20:19.889109+0000","last_unstale":"2024-04-02T13:20:26.431510+0000","last_undegraded":"2024-04-02T13:20:26.431510+0000","last_fullsized":"2024-04-02T13:20:26.431510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:42:03.725417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911104+0000","last_change":"2024-04-02T13:20:20.899681+0000","last_active":"2024-04-02T13:20:25.911104+0000","last_peered":"2024-04-02T13:20:25.911104+0000","last_clean":"2024-04-02T13:20:25.911104+0000","last_became_active":"2024-04-02T13:20:20.899408+0000","last_became_peered":"2024-04-02T13:20:20.899408+0000","last_unstale":"2024-04-02T13:20:25.911104+0000","last_undegraded":"2024-04-02T13:20:25.911104+0000","last_fullsized":"2024-04-02T13:20:25.911104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910230+0000","last_change":"2024-04-02T13:20:24.974937+0000","last_active":"2024-04-02T13:20:25.910230+0000","last_peered":"2024-04-02T13:20:25.910230+0000","last_clean":"2024-04-02T13:20:25.910230+0000","last_became_active":"2024-04-02T13:20:24.974496+0000","last_became_peered":"2024-04-02T13:20:24.974496+0000","last_unstale":"2024-04-02T13:20:25.910230+0000","last_undegraded":"2024-04-02T13:20:25.910230+0000","last_fullsized":"2024-04-02T13:20:25.910230+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:20:58.050570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911229+0000","last_change":"2024-04-02T13:20:19.896780+0000","last_active":"2024-04-02T13:20:25.911229+0000","last_peered":"2024-04-02T13:20:25.911229+0000","last_clean":"2024-04-02T13:20:25.911229+0000","last_became_active":"2024-04-02T13:20:19.896480+0000","last_became_peered":"2024-04-02T13:20:19.896480+0000","last_unstale":"2024-04-02T13:20:25.911229+0000","last_undegraded":"2024-04-02T13:20:25.911229+0000","last_fullsized":"2024-04-02T13:20:25.911229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431238+0000","last_change":"2024-04-02T13:20:20.905268+0000","last_active":"2024-04-02T13:20:26.431238+0000","last_peered":"2024-04-02T13:20:26.431238+0000","last_clean":"2024-04-02T13:20:26.431238+0000","last_became_active":"2024-04-02T13:20:20.904997+0000","last_became_peered":"2024-04-02T13:20:20.904997+0000","last_unstale":"2024-04-02T13:20:26.431238+0000","last_undegraded":"2024-04-02T13:20:26.431238+0000","last_fullsized":"2024-04-02T13:20:26.431238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912459+0000","last_change":"2024-04-02T13:20:24.933025+0000","last_active":"2024-04-02T13:20:25.912459+0000","last_peered":"2024-04-02T13:20:25.912459+0000","last_clean":"2024-04-02T13:20:25.912459+0000","last_became_active":"2024-04-02T13:20:24.932475+0000","last_became_peered":"2024-04-02T13:20:24.932475+0000","last_unstale":"2024-04-02T13:20:25.912459+0000","last_undegraded":"2024-04-02T13:20:25.912459+0000","last_fullsized":"2024-04-02T13:20:25.912459+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:46.747491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431519+0000","last_change":"2024-04-02T13:20:19.895931+0000","last_active":"2024-04-02T13:20:26.431519+0000","last_peered":"2024-04-02T13:20:26.431519+0000","last_clean":"2024-04-02T13:20:26.431519+0000","last_became_active":"2024-04-02T13:20:19.895304+0000","last_became_peered":"2024-04-02T13:20:19.895304+0000","last_unstale":"2024-04-02T13:20:26.431519+0000","last_undegraded":"2024-04-02T13:20:26.431519+0000","last_fullsized":"2024-04-02T13:20:26.431519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:08.653626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912892+0000","last_change":"2024-04-02T13:20:20.904770+0000","last_active":"2024-04-02T13:20:25.912892+0000","last_peered":"2024-04-02T13:20:25.912892+0000","last_clean":"2024-04-02T13:20:25.912892+0000","last_became_active":"2024-04-02T13:20:20.903916+0000","last_became_peered":"2024-04-02T13:20:20.903916+0000","last_unstale":"2024-04-02T13:20:25.912892+0000","last_undegraded":"2024-04-02T13:20:25.912892+0000","last_fullsized":"2024-04-02T13:20:25.912892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:39.096965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913056+0000","last_change":"2024-04-02T13:20:24.932562+0000","last_active":"2024-04-02T13:20:25.913056+0000","last_peered":"2024-04-02T13:20:25.913056+0000","last_clean":"2024-04-02T13:20:25.913056+0000","last_became_active":"2024-04-02T13:20:24.931896+0000","last_became_peered":"2024-04-02T13:20:24.931896+0000","last_unstale":"2024-04-02T13:20:25.913056+0000","last_undegraded":"2024-04-02T13:20:25.913056+0000","last_fullsized":"2024-04-02T13:20:25.913056+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:43.131658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430952+0000","last_change":"2024-04-02T13:20:19.887808+0000","last_active":"2024-04-02T13:20:26.430952+0000","last_peered":"2024-04-02T13:20:26.430952+0000","last_clean":"2024-04-02T13:20:26.430952+0000","last_became_active":"2024-04-02T13:20:19.887491+0000","last_became_peered":"2024-04-02T13:20:19.887491+0000","last_unstale":"2024-04-02T13:20:26.430952+0000","last_undegraded":"2024-04-02T13:20:26.430952+0000","last_fullsized":"2024-04-02T13:20:26.430952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914136+0000","last_change":"2024-04-02T13:20:20.900447+0000","last_active":"2024-04-02T13:20:25.914136+0000","last_peered":"2024-04-02T13:20:25.914136+0000","last_clean":"2024-04-02T13:20:25.914136+0000","last_became_active":"2024-04-02T13:20:20.900188+0000","last_became_peered":"2024-04-02T13:20:20.900188+0000","last_unstale":"2024-04-02T13:20:25.914136+0000","last_undegraded":"2024-04-02T13:20:25.914136+0000","last_fullsized":"2024-04-02T13:20:25.914136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:37.660148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912721+0000","last_change":"2024-04-02T13:20:24.928309+0000","last_active":"2024-04-02T13:20:25.912721+0000","last_peered":"2024-04-02T13:20:25.912721+0000","last_clean":"2024-04-02T13:20:25.912721+0000","last_became_active":"2024-04-02T13:20:24.927881+0000","last_became_peered":"2024-04-02T13:20:24.927881+0000","last_unstale":"2024-04-02T13:20:25.912721+0000","last_undegraded":"2024-04-02T13:20:25.912721+0000","last_fullsized":"2024-04-02T13:20:25.912721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:38.338080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430504+0000","last_change":"2024-04-02T13:20:20.895522+0000","last_active":"2024-04-02T13:20:26.430504+0000","last_peered":"2024-04-02T13:20:26.430504+0000","last_clean":"2024-04-02T13:20:26.430504+0000","last_became_active":"2024-04-02T13:20:20.895202+0000","last_became_peered":"2024-04-02T13:20:20.895202+0000","last_unstale":"2024-04-02T13:20:26.430504+0000","last_undegraded":"2024-04-02T13:20:26.430504+0000","last_fullsized":"2024-04-02T13:20:26.430504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430987+0000","last_change":"2024-04-02T13:20:19.895121+0000","last_active":"2024-04-02T13:20:26.430987+0000","last_peered":"2024-04-02T13:20:26.430987+0000","last_clean":"2024-04-02T13:20:26.430987+0000","last_became_active":"2024-04-02T13:20:19.894633+0000","last_became_peered":"2024-04-02T13:20:19.894633+0000","last_unstale":"2024-04-02T13:20:26.430987+0000","last_undegraded":"2024-04-02T13:20:26.430987+0000","last_fullsized":"2024-04-02T13:20:26.430987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:05:36.149061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913513+0000","last_change":"2024-04-02T13:20:24.941648+0000","last_active":"2024-04-02T13:20:25.913513+0000","last_peered":"2024-04-02T13:20:25.913513+0000","last_clean":"2024-04-02T13:20:25.913513+0000","last_became_active":"2024-04-02T13:20:24.941281+0000","last_became_peered":"2024-04-02T13:20:24.941281+0000","last_unstale":"2024-04-02T13:20:25.913513+0000","last_undegraded":"2024-04-02T13:20:25.913513+0000","last_fullsized":"2024-04-02T13:20:25.913513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:06.819681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910552+0000","last_change":"2024-04-02T13:20:20.895057+0000","last_active":"2024-04-02T13:20:25.910552+0000","last_peered":"2024-04-02T13:20:25.910552+0000","last_clean":"2024-04-02T13:20:25.910552+0000","last_became_active":"2024-04-02T13:20:20.894189+0000","last_became_peered":"2024-04-02T13:20:20.894189+0000","last_unstale":"2024-04-02T13:20:25.910552+0000","last_undegraded":"2024-04-02T13:20:25.910552+0000","last_fullsized":"2024-04-02T13:20:25.910552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:56:59.954023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432656+0000","last_change":"2024-04-02T13:20:19.898990+0000","last_active":"2024-04-02T13:20:26.432656+0000","last_peered":"2024-04-02T13:20:26.432656+0000","last_clean":"2024-04-02T13:20:26.432656+0000","last_became_active":"2024-04-02T13:20:19.898590+0000","last_became_peered":"2024-04-02T13:20:19.898590+0000","last_unstale":"2024-04-02T13:20:26.432656+0000","last_undegraded":"2024-04-02T13:20:26.432656+0000","last_fullsized":"2024-04-02T13:20:26.432656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912549+0000","last_change":"2024-04-02T13:20:24.943246+0000","last_active":"2024-04-02T13:20:25.912549+0000","last_peered":"2024-04-02T13:20:25.912549+0000","last_clean":"2024-04-02T13:20:25.912549+0000","last_became_active":"2024-04-02T13:20:24.942587+0000","last_became_peered":"2024-04-02T13:20:24.942587+0000","last_unstale":"2024-04-02T13:20:25.912549+0000","last_undegraded":"2024-04-02T13:20:25.912549+0000","last_fullsized":"2024-04-02T13:20:25.912549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:10.369290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910414+0000","last_change":"2024-04-02T13:20:20.896993+0000","last_active":"2024-04-02T13:20:25.910414+0000","last_peered":"2024-04-02T13:20:25.910414+0000","last_clean":"2024-04-02T13:20:25.910414+0000","last_became_active":"2024-04-02T13:20:20.896668+0000","last_became_peered":"2024-04-02T13:20:20.896668+0000","last_unstale":"2024-04-02T13:20:25.910414+0000","last_undegraded":"2024-04-02T13:20:25.910414+0000","last_fullsized":"2024-04-02T13:20:25.910414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430520+0000","last_change":"2024-04-02T13:20:19.892855+0000","last_active":"2024-04-02T13:20:26.430520+0000","last_peered":"2024-04-02T13:20:26.430520+0000","last_clean":"2024-04-02T13:20:26.430520+0000","last_became_active":"2024-04-02T13:20:19.892578+0000","last_became_peered":"2024-04-02T13:20:19.892578+0000","last_unstale":"2024-04-02T13:20:26.430520+0000","last_undegraded":"2024-04-02T13:20:26.430520+0000","last_fullsized":"2024-04-02T13:20:26.430520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:36.358876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432032+0000","last_change":"2024-04-02T13:20:24.975476+0000","last_active":"2024-04-02T13:20:26.432032+0000","last_peered":"2024-04-02T13:20:26.432032+0000","last_clean":"2024-04-02T13:20:26.432032+0000","last_became_active":"2024-04-02T13:20:24.974600+0000","last_became_peered":"2024-04-02T13:20:24.974600+0000","last_unstale":"2024-04-02T13:20:26.432032+0000","last_undegraded":"2024-04-02T13:20:26.432032+0000","last_fullsized":"2024-04-02T13:20:26.432032+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:35.465459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913314+0000","last_change":"2024-04-02T13:20:20.898012+0000","last_active":"2024-04-02T13:20:25.913314+0000","last_peered":"2024-04-02T13:20:25.913314+0000","last_clean":"2024-04-02T13:20:25.913314+0000","last_became_active":"2024-04-02T13:20:20.897732+0000","last_became_peered":"2024-04-02T13:20:20.897732+0000","last_unstale":"2024-04-02T13:20:25.913314+0000","last_undegraded":"2024-04-02T13:20:25.913314+0000","last_fullsized":"2024-04-02T13:20:25.913314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:55.219441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430649+0000","last_change":"2024-04-02T13:20:19.894100+0000","last_active":"2024-04-02T13:20:26.430649+0000","last_peered":"2024-04-02T13:20:26.430649+0000","last_clean":"2024-04-02T13:20:26.430649+0000","last_became_active":"2024-04-02T13:20:19.893628+0000","last_became_peered":"2024-04-02T13:20:19.893628+0000","last_unstale":"2024-04-02T13:20:26.430649+0000","last_undegraded":"2024-04-02T13:20:26.430649+0000","last_fullsized":"2024-04-02T13:20:26.430649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:05.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913916+0000","last_change":"2024-04-02T13:20:24.943183+0000","last_active":"2024-04-02T13:20:25.913916+0000","last_peered":"2024-04-02T13:20:25.913916+0000","last_clean":"2024-04-02T13:20:25.913916+0000","last_became_active":"2024-04-02T13:20:24.942594+0000","last_became_peered":"2024-04-02T13:20:24.942594+0000","last_unstale":"2024-04-02T13:20:25.913916+0000","last_undegraded":"2024-04-02T13:20:25.913916+0000","last_fullsized":"2024-04-02T13:20:25.913916+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:29.504244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431563+0000","last_change":"2024-04-02T13:20:20.898299+0000","last_active":"2024-04-02T13:20:26.431563+0000","last_peered":"2024-04-02T13:20:26.431563+0000","last_clean":"2024-04-02T13:20:26.431563+0000","last_became_active":"2024-04-02T13:20:20.898014+0000","last_became_peered":"2024-04-02T13:20:20.898014+0000","last_unstale":"2024-04-02T13:20:26.431563+0000","last_undegraded":"2024-04-02T13:20:26.431563+0000","last_fullsized":"2024-04-02T13:20:26.431563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:08.738556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431600+0000","last_change":"2024-04-02T13:20:19.895734+0000","last_active":"2024-04-02T13:20:26.431600+0000","last_peered":"2024-04-02T13:20:26.431600+0000","last_clean":"2024-04-02T13:20:26.431600+0000","last_became_active":"2024-04-02T13:20:19.895430+0000","last_became_peered":"2024-04-02T13:20:19.895430+0000","last_unstale":"2024-04-02T13:20:26.431600+0000","last_undegraded":"2024-04-02T13:20:26.431600+0000","last_fullsized":"2024-04-02T13:20:26.431600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912814+0000","last_change":"2024-04-02T13:20:24.972780+0000","last_active":"2024-04-02T13:20:25.912814+0000","last_peered":"2024-04-02T13:20:25.912814+0000","last_clean":"2024-04-02T13:20:25.912814+0000","last_became_active":"2024-04-02T13:20:24.972420+0000","last_became_peered":"2024-04-02T13:20:24.972420+0000","last_unstale":"2024-04-02T13:20:25.912814+0000","last_undegraded":"2024-04-02T13:20:25.912814+0000","last_fullsized":"2024-04-02T13:20:25.912814+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:28.449843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911321+0000","last_change":"2024-04-02T13:20:20.901896+0000","last_active":"2024-04-02T13:20:25.911321+0000","last_peered":"2024-04-02T13:20:25.911321+0000","last_clean":"2024-04-02T13:20:25.911321+0000","last_became_active":"2024-04-02T13:20:20.901593+0000","last_became_peered":"2024-04-02T13:20:20.901593+0000","last_unstale":"2024-04-02T13:20:25.911321+0000","last_undegraded":"2024-04-02T13:20:25.911321+0000","last_fullsized":"2024-04-02T13:20:25.911321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:24.415150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431790+0000","last_change":"2024-04-02T13:20:19.895273+0000","last_active":"2024-04-02T13:20:26.431790+0000","last_peered":"2024-04-02T13:20:26.431790+0000","last_clean":"2024-04-02T13:20:26.431790+0000","last_became_active":"2024-04-02T13:20:19.894962+0000","last_became_peered":"2024-04-02T13:20:19.894962+0000","last_unstale":"2024-04-02T13:20:26.431790+0000","last_undegraded":"2024-04-02T13:20:26.431790+0000","last_fullsized":"2024-04-02T13:20:26.431790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910167+0000","last_change":"2024-04-02T13:20:24.975854+0000","last_active":"2024-04-02T13:20:25.910167+0000","last_peered":"2024-04-02T13:20:25.910167+0000","last_clean":"2024-04-02T13:20:25.910167+0000","last_became_active":"2024-04-02T13:20:24.975476+0000","last_became_peered":"2024-04-02T13:20:24.975476+0000","last_unstale":"2024-04-02T13:20:25.910167+0000","last_undegraded":"2024-04-02T13:20:25.910167+0000","last_fullsized":"2024-04-02T13:20:25.910167+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:00.601346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432365+0000","last_change":"2024-04-02T13:20:20.898220+0000","last_active":"2024-04-02T13:20:26.432365+0000","last_peered":"2024-04-02T13:20:26.432365+0000","last_clean":"2024-04-02T13:20:26.432365+0000","last_became_active":"2024-04-02T13:20:20.897937+0000","last_became_peered":"2024-04-02T13:20:20.897937+0000","last_unstale":"2024-04-02T13:20:26.432365+0000","last_undegraded":"2024-04-02T13:20:26.432365+0000","last_fullsized":"2024-04-02T13:20:26.432365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431659+0000","last_change":"2024-04-02T13:20:19.891525+0000","last_active":"2024-04-02T13:20:26.431659+0000","last_peered":"2024-04-02T13:20:26.431659+0000","last_clean":"2024-04-02T13:20:26.431659+0000","last_became_active":"2024-04-02T13:20:19.891260+0000","last_became_peered":"2024-04-02T13:20:19.891260+0000","last_unstale":"2024-04-02T13:20:26.431659+0000","last_undegraded":"2024-04-02T13:20:26.431659+0000","last_fullsized":"2024-04-02T13:20:26.431659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:01.860021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431840+0000","last_change":"2024-04-02T13:20:24.970679+0000","last_active":"2024-04-02T13:20:26.431840+0000","last_peered":"2024-04-02T13:20:26.431840+0000","last_clean":"2024-04-02T13:20:26.431840+0000","last_became_active":"2024-04-02T13:20:24.970216+0000","last_became_peered":"2024-04-02T13:20:24.970216+0000","last_unstale":"2024-04-02T13:20:26.431840+0000","last_undegraded":"2024-04-02T13:20:26.431840+0000","last_fullsized":"2024-04-02T13:20:26.431840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:46.744270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914241+0000","last_change":"2024-04-02T13:20:20.901035+0000","last_active":"2024-04-02T13:20:25.914241+0000","last_peered":"2024-04-02T13:20:25.914241+0000","last_clean":"2024-04-02T13:20:25.914241+0000","last_became_active":"2024-04-02T13:20:20.900751+0000","last_became_peered":"2024-04-02T13:20:20.900751+0000","last_unstale":"2024-04-02T13:20:25.914241+0000","last_undegraded":"2024-04-02T13:20:25.914241+0000","last_fullsized":"2024-04-02T13:20:25.914241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:31.430767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432453+0000","last_change":"2024-04-02T13:20:19.897166+0000","last_active":"2024-04-02T13:20:26.432453+0000","last_peered":"2024-04-02T13:20:26.432453+0000","last_clean":"2024-04-02T13:20:26.432453+0000","last_became_active":"2024-04-02T13:20:19.896854+0000","last_became_peered":"2024-04-02T13:20:19.896854+0000","last_unstale":"2024-04-02T13:20:26.432453+0000","last_undegraded":"2024-04-02T13:20:26.432453+0000","last_fullsized":"2024-04-02T13:20:26.432453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:08.653626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430806+0000","last_change":"2024-04-02T13:20:24.937584+0000","last_active":"2024-04-02T13:20:26.430806+0000","last_peered":"2024-04-02T13:20:26.430806+0000","last_clean":"2024-04-02T13:20:26.430806+0000","last_became_active":"2024-04-02T13:20:24.937027+0000","last_became_peered":"2024-04-02T13:20:24.937027+0000","last_unstale":"2024-04-02T13:20:26.430806+0000","last_undegraded":"2024-04-02T13:20:26.430806+0000","last_fullsized":"2024-04-02T13:20:26.430806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:29.504244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914109+0000","last_change":"2024-04-02T13:20:20.888461+0000","last_active":"2024-04-02T13:20:25.914109+0000","last_peered":"2024-04-02T13:20:25.914109+0000","last_clean":"2024-04-02T13:20:25.914109+0000","last_became_active":"2024-04-02T13:20:20.887870+0000","last_became_peered":"2024-04-02T13:20:20.887870+0000","last_unstale":"2024-04-02T13:20:25.914109+0000","last_undegraded":"2024-04-02T13:20:25.914109+0000","last_fullsized":"2024-04-02T13:20:25.914109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:06.511257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913085+0000","last_change":"2024-04-02T13:20:19.895704+0000","last_active":"2024-04-02T13:20:25.913085+0000","last_peered":"2024-04-02T13:20:25.913085+0000","last_clean":"2024-04-02T13:20:25.913085+0000","last_became_active":"2024-04-02T13:20:19.895394+0000","last_became_peered":"2024-04-02T13:20:19.895394+0000","last_unstale":"2024-04-02T13:20:25.913085+0000","last_undegraded":"2024-04-02T13:20:25.913085+0000","last_fullsized":"2024-04-02T13:20:25.913085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432394+0000","last_change":"2024-04-02T13:20:24.935887+0000","last_active":"2024-04-02T13:20:26.432394+0000","last_peered":"2024-04-02T13:20:26.432394+0000","last_clean":"2024-04-02T13:20:26.432394+0000","last_became_active":"2024-04-02T13:20:24.935515+0000","last_became_peered":"2024-04-02T13:20:24.935515+0000","last_unstale":"2024-04-02T13:20:26.432394+0000","last_undegraded":"2024-04-02T13:20:26.432394+0000","last_fullsized":"2024-04-02T13:20:26.432394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:46.744270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431762+0000","last_change":"2024-04-02T13:20:20.899067+0000","last_active":"2024-04-02T13:20:26.431762+0000","last_peered":"2024-04-02T13:20:26.431762+0000","last_clean":"2024-04-02T13:20:26.431762+0000","last_became_active":"2024-04-02T13:20:20.898662+0000","last_became_peered":"2024-04-02T13:20:20.898662+0000","last_unstale":"2024-04-02T13:20:26.431762+0000","last_undegraded":"2024-04-02T13:20:26.431762+0000","last_fullsized":"2024-04-02T13:20:26.431762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431752+0000","last_change":"2024-04-02T13:20:19.896149+0000","last_active":"2024-04-02T13:20:26.431752+0000","last_peered":"2024-04-02T13:20:26.431752+0000","last_clean":"2024-04-02T13:20:26.431752+0000","last_became_active":"2024-04-02T13:20:19.895801+0000","last_became_peered":"2024-04-02T13:20:19.895801+0000","last_unstale":"2024-04-02T13:20:26.431752+0000","last_undegraded":"2024-04-02T13:20:26.431752+0000","last_fullsized":"2024-04-02T13:20:26.431752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432170+0000","last_change":"2024-04-02T13:20:24.933627+0000","last_active":"2024-04-02T13:20:26.432170+0000","last_peered":"2024-04-02T13:20:26.432170+0000","last_clean":"2024-04-02T13:20:26.432170+0000","last_became_active":"2024-04-02T13:20:24.932519+0000","last_became_peered":"2024-04-02T13:20:24.932519+0000","last_unstale":"2024-04-02T13:20:26.432170+0000","last_undegraded":"2024-04-02T13:20:26.432170+0000","last_fullsized":"2024-04-02T13:20:26.432170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:41.694841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430241+0000","last_change":"2024-04-02T13:20:20.899704+0000","last_active":"2024-04-02T13:20:26.430241+0000","last_peered":"2024-04-02T13:20:26.430241+0000","last_clean":"2024-04-02T13:20:26.430241+0000","last_became_active":"2024-04-02T13:20:20.894279+0000","last_became_peered":"2024-04-02T13:20:20.894279+0000","last_unstale":"2024-04-02T13:20:26.430241+0000","last_undegraded":"2024-04-02T13:20:26.430241+0000","last_fullsized":"2024-04-02T13:20:26.430241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:48.848558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430741+0000","last_change":"2024-04-02T13:20:19.893903+0000","last_active":"2024-04-02T13:20:26.430741+0000","last_peered":"2024-04-02T13:20:26.430741+0000","last_clean":"2024-04-02T13:20:26.430741+0000","last_became_active":"2024-04-02T13:20:19.893251+0000","last_became_peered":"2024-04-02T13:20:19.893251+0000","last_unstale":"2024-04-02T13:20:26.430741+0000","last_undegraded":"2024-04-02T13:20:26.430741+0000","last_fullsized":"2024-04-02T13:20:26.430741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:36.358876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431391+0000","last_change":"2024-04-02T13:20:24.971364+0000","last_active":"2024-04-02T13:20:26.431391+0000","last_peered":"2024-04-02T13:20:26.431391+0000","last_clean":"2024-04-02T13:20:26.431391+0000","last_became_active":"2024-04-02T13:20:24.970950+0000","last_became_peered":"2024-04-02T13:20:24.970950+0000","last_unstale":"2024-04-02T13:20:26.431391+0000","last_undegraded":"2024-04-02T13:20:26.431391+0000","last_fullsized":"2024-04-02T13:20:26.431391+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:41.694841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910298+0000","last_change":"2024-04-02T13:20:20.906876+0000","last_active":"2024-04-02T13:20:25.910298+0000","last_peered":"2024-04-02T13:20:25.910298+0000","last_clean":"2024-04-02T13:20:25.910298+0000","last_became_active":"2024-04-02T13:20:20.906603+0000","last_became_peered":"2024-04-02T13:20:20.906603+0000","last_unstale":"2024-04-02T13:20:25.910298+0000","last_undegraded":"2024-04-02T13:20:25.910298+0000","last_fullsized":"2024-04-02T13:20:25.910298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:39.096965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913068+0000","last_change":"2024-04-02T13:20:19.889006+0000","last_active":"2024-04-02T13:20:25.913068+0000","last_peered":"2024-04-02T13:20:25.913068+0000","last_clean":"2024-04-02T13:20:25.913068+0000","last_became_active":"2024-04-02T13:20:19.888619+0000","last_became_peered":"2024-04-02T13:20:19.888619+0000","last_unstale":"2024-04-02T13:20:25.913068+0000","last_undegraded":"2024-04-02T13:20:25.913068+0000","last_fullsized":"2024-04-02T13:20:25.913068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430765+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:20:26.430765+0000","last_peered":"2024-04-02T13:20:26.430765+0000","last_clean":"2024-04-02T13:20:26.430765+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:20:26.430765+0000","last_undegraded":"2024-04-02T13:20:26.430765+0000","last_fullsized":"2024-04-02T13:20:26.430765+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430638+0000","last_change":"2024-04-02T13:20:24.972665+0000","last_active":"2024-04-02T13:20:26.430638+0000","last_peered":"2024-04-02T13:20:26.430638+0000","last_clean":"2024-04-02T13:20:26.430638+0000","last_became_active":"2024-04-02T13:20:24.972285+0000","last_became_peered":"2024-04-02T13:20:24.972285+0000","last_unstale":"2024-04-02T13:20:26.430638+0000","last_undegraded":"2024-04-02T13:20:26.430638+0000","last_fullsized":"2024-04-02T13:20:26.430638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:43.131658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431391+0000","last_change":"2024-04-02T13:20:20.901120+0000","last_active":"2024-04-02T13:20:26.431391+0000","last_peered":"2024-04-02T13:20:26.431391+0000","last_clean":"2024-04-02T13:20:26.431391+0000","last_became_active":"2024-04-02T13:20:20.900433+0000","last_became_peered":"2024-04-02T13:20:20.900433+0000","last_unstale":"2024-04-02T13:20:26.431391+0000","last_undegraded":"2024-04-02T13:20:26.431391+0000","last_fullsized":"2024-04-02T13:20:26.431391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:42.709577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431230+0000","last_change":"2024-04-02T13:20:19.893843+0000","last_active":"2024-04-02T13:20:26.431230+0000","last_peered":"2024-04-02T13:20:26.431230+0000","last_clean":"2024-04-02T13:20:26.431230+0000","last_became_active":"2024-04-02T13:20:19.893524+0000","last_became_peered":"2024-04-02T13:20:19.893524+0000","last_unstale":"2024-04-02T13:20:26.431230+0000","last_undegraded":"2024-04-02T13:20:26.431230+0000","last_fullsized":"2024-04-02T13:20:26.431230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:56.359644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430631+0000","last_change":"2024-04-02T13:20:24.971102+0000","last_active":"2024-04-02T13:20:26.430631+0000","last_peered":"2024-04-02T13:20:26.430631+0000","last_clean":"2024-04-02T13:20:26.430631+0000","last_became_active":"2024-04-02T13:20:24.970132+0000","last_became_peered":"2024-04-02T13:20:24.970132+0000","last_unstale":"2024-04-02T13:20:26.430631+0000","last_undegraded":"2024-04-02T13:20:26.430631+0000","last_fullsized":"2024-04-02T13:20:26.430631+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:28.449843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912995+0000","last_change":"2024-04-02T13:20:20.908405+0000","last_active":"2024-04-02T13:20:25.912995+0000","last_peered":"2024-04-02T13:20:25.912995+0000","last_clean":"2024-04-02T13:20:25.912995+0000","last_became_active":"2024-04-02T13:20:20.907986+0000","last_became_peered":"2024-04-02T13:20:20.907986+0000","last_unstale":"2024-04-02T13:20:25.912995+0000","last_undegraded":"2024-04-02T13:20:25.912995+0000","last_fullsized":"2024-04-02T13:20:25.912995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:24.415150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432656+0000","last_change":"2024-04-02T13:20:19.893884+0000","last_active":"2024-04-02T13:20:26.432656+0000","last_peered":"2024-04-02T13:20:26.432656+0000","last_clean":"2024-04-02T13:20:26.432656+0000","last_became_active":"2024-04-02T13:20:19.893605+0000","last_became_peered":"2024-04-02T13:20:19.893605+0000","last_unstale":"2024-04-02T13:20:26.432656+0000","last_undegraded":"2024-04-02T13:20:26.432656+0000","last_fullsized":"2024-04-02T13:20:26.432656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:05.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432026+0000","last_change":"2024-04-02T13:20:24.931897+0000","last_active":"2024-04-02T13:20:26.432026+0000","last_peered":"2024-04-02T13:20:26.432026+0000","last_clean":"2024-04-02T13:20:26.432026+0000","last_became_active":"2024-04-02T13:20:24.931512+0000","last_became_peered":"2024-04-02T13:20:24.931512+0000","last_unstale":"2024-04-02T13:20:26.432026+0000","last_undegraded":"2024-04-02T13:20:26.432026+0000","last_fullsized":"2024-04-02T13:20:26.432026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:26:20.333590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431656+0000","last_change":"2024-04-02T13:20:20.890017+0000","last_active":"2024-04-02T13:20:26.431656+0000","last_peered":"2024-04-02T13:20:26.431656+0000","last_clean":"2024-04-02T13:20:26.431656+0000","last_became_active":"2024-04-02T13:20:20.889739+0000","last_became_peered":"2024-04-02T13:20:20.889739+0000","last_unstale":"2024-04-02T13:20:26.431656+0000","last_undegraded":"2024-04-02T13:20:26.431656+0000","last_fullsized":"2024-04-02T13:20:26.431656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:25.469551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431013+0000","last_change":"2024-04-02T13:20:19.899771+0000","last_active":"2024-04-02T13:20:26.431013+0000","last_peered":"2024-04-02T13:20:26.431013+0000","last_clean":"2024-04-02T13:20:26.431013+0000","last_became_active":"2024-04-02T13:20:19.899415+0000","last_became_peered":"2024-04-02T13:20:19.899415+0000","last_unstale":"2024-04-02T13:20:26.431013+0000","last_undegraded":"2024-04-02T13:20:26.431013+0000","last_fullsized":"2024-04-02T13:20:26.431013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913513+0000","last_change":"2024-04-02T13:20:24.971830+0000","last_active":"2024-04-02T13:20:25.913513+0000","last_peered":"2024-04-02T13:20:25.913513+0000","last_clean":"2024-04-02T13:20:25.913513+0000","last_became_active":"2024-04-02T13:20:24.971442+0000","last_became_peered":"2024-04-02T13:20:24.971442+0000","last_unstale":"2024-04-02T13:20:25.913513+0000","last_undegraded":"2024-04-02T13:20:25.913513+0000","last_fullsized":"2024-04-02T13:20:25.913513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:00.601346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431356+0000","last_change":"2024-04-02T13:20:20.887060+0000","last_active":"2024-04-02T13:20:26.431356+0000","last_peered":"2024-04-02T13:20:26.431356+0000","last_clean":"2024-04-02T13:20:26.431356+0000","last_became_active":"2024-04-02T13:20:20.886492+0000","last_became_peered":"2024-04-02T13:20:20.886492+0000","last_unstale":"2024-04-02T13:20:26.431356+0000","last_undegraded":"2024-04-02T13:20:26.431356+0000","last_fullsized":"2024-04-02T13:20:26.431356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:31.430767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913624+0000","last_change":"2024-04-02T13:20:19.897647+0000","last_active":"2024-04-02T13:20:25.913624+0000","last_peered":"2024-04-02T13:20:25.913624+0000","last_clean":"2024-04-02T13:20:25.913624+0000","last_became_active":"2024-04-02T13:20:19.897244+0000","last_became_peered":"2024-04-02T13:20:19.897244+0000","last_unstale":"2024-04-02T13:20:25.913624+0000","last_undegraded":"2024-04-02T13:20:25.913624+0000","last_fullsized":"2024-04-02T13:20:25.913624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:28:53.732425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912616+0000","last_change":"2024-04-02T13:20:24.932365+0000","last_active":"2024-04-02T13:20:25.912616+0000","last_peered":"2024-04-02T13:20:25.912616+0000","last_clean":"2024-04-02T13:20:25.912616+0000","last_became_active":"2024-04-02T13:20:24.931793+0000","last_became_peered":"2024-04-02T13:20:24.931793+0000","last_unstale":"2024-04-02T13:20:25.912616+0000","last_undegraded":"2024-04-02T13:20:25.912616+0000","last_fullsized":"2024-04-02T13:20:25.912616+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:30.566782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431185+0000","last_change":"2024-04-02T13:20:20.887410+0000","last_active":"2024-04-02T13:20:26.431185+0000","last_peered":"2024-04-02T13:20:26.431185+0000","last_clean":"2024-04-02T13:20:26.431185+0000","last_became_active":"2024-04-02T13:20:20.886414+0000","last_became_peered":"2024-04-02T13:20:20.886414+0000","last_unstale":"2024-04-02T13:20:26.431185+0000","last_undegraded":"2024-04-02T13:20:26.431185+0000","last_fullsized":"2024-04-02T13:20:26.431185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:24.415150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430413+0000","last_change":"2024-04-02T13:20:19.896189+0000","last_active":"2024-04-02T13:20:26.430413+0000","last_peered":"2024-04-02T13:20:26.430413+0000","last_clean":"2024-04-02T13:20:26.430413+0000","last_became_active":"2024-04-02T13:20:19.895844+0000","last_became_peered":"2024-04-02T13:20:19.895844+0000","last_unstale":"2024-04-02T13:20:26.430413+0000","last_undegraded":"2024-04-02T13:20:26.430413+0000","last_fullsized":"2024-04-02T13:20:26.430413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431242+0000","last_change":"2024-04-02T13:20:24.928642+0000","last_active":"2024-04-02T13:20:26.431242+0000","last_peered":"2024-04-02T13:20:26.431242+0000","last_clean":"2024-04-02T13:20:26.431242+0000","last_became_active":"2024-04-02T13:20:24.928080+0000","last_became_peered":"2024-04-02T13:20:24.928080+0000","last_unstale":"2024-04-02T13:20:26.431242+0000","last_undegraded":"2024-04-02T13:20:26.431242+0000","last_fullsized":"2024-04-02T13:20:26.431242+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:35.465459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430479+0000","last_change":"2024-04-02T13:20:20.898225+0000","last_active":"2024-04-02T13:20:26.430479+0000","last_peered":"2024-04-02T13:20:26.430479+0000","last_clean":"2024-04-02T13:20:26.430479+0000","last_became_active":"2024-04-02T13:20:20.897962+0000","last_became_peered":"2024-04-02T13:20:20.897962+0000","last_unstale":"2024-04-02T13:20:26.430479+0000","last_undegraded":"2024-04-02T13:20:26.430479+0000","last_fullsized":"2024-04-02T13:20:26.430479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430618+0000","last_change":"2024-04-02T13:20:19.893578+0000","last_active":"2024-04-02T13:20:26.430618+0000","last_peered":"2024-04-02T13:20:26.430618+0000","last_clean":"2024-04-02T13:20:26.430618+0000","last_became_active":"2024-04-02T13:20:19.893301+0000","last_became_peered":"2024-04-02T13:20:19.893301+0000","last_unstale":"2024-04-02T13:20:26.430618+0000","last_undegraded":"2024-04-02T13:20:26.430618+0000","last_fullsized":"2024-04-02T13:20:26.430618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911170+0000","last_change":"2024-04-02T13:20:24.971783+0000","last_active":"2024-04-02T13:20:25.911170+0000","last_peered":"2024-04-02T13:20:25.911170+0000","last_clean":"2024-04-02T13:20:25.911170+0000","last_became_active":"2024-04-02T13:20:24.971383+0000","last_became_peered":"2024-04-02T13:20:24.971383+0000","last_unstale":"2024-04-02T13:20:25.911170+0000","last_undegraded":"2024-04-02T13:20:25.911170+0000","last_fullsized":"2024-04-02T13:20:25.911170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:01.105712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430867+0000","last_change":"2024-04-02T13:20:20.889512+0000","last_active":"2024-04-02T13:20:26.430867+0000","last_peered":"2024-04-02T13:20:26.430867+0000","last_clean":"2024-04-02T13:20:26.430867+0000","last_became_active":"2024-04-02T13:20:20.886370+0000","last_became_peered":"2024-04-02T13:20:20.886370+0000","last_unstale":"2024-04-02T13:20:26.430867+0000","last_undegraded":"2024-04-02T13:20:26.430867+0000","last_fullsized":"2024-04-02T13:20:26.430867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:15.162527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912998+0000","last_change":"2024-04-02T13:20:19.895006+0000","last_active":"2024-04-02T13:20:25.912998+0000","last_peered":"2024-04-02T13:20:25.912998+0000","last_clean":"2024-04-02T13:20:25.912998+0000","last_became_active":"2024-04-02T13:20:19.894684+0000","last_became_peered":"2024-04-02T13:20:19.894684+0000","last_unstale":"2024-04-02T13:20:25.912998+0000","last_undegraded":"2024-04-02T13:20:25.912998+0000","last_fullsized":"2024-04-02T13:20:25.912998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911073+0000","last_change":"2024-04-02T13:20:24.972147+0000","last_active":"2024-04-02T13:20:25.911073+0000","last_peered":"2024-04-02T13:20:25.911073+0000","last_clean":"2024-04-02T13:20:25.911073+0000","last_became_active":"2024-04-02T13:20:24.971778+0000","last_became_peered":"2024-04-02T13:20:24.971778+0000","last_unstale":"2024-04-02T13:20:25.911073+0000","last_undegraded":"2024-04-02T13:20:25.911073+0000","last_fullsized":"2024-04-02T13:20:25.911073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:22:03.916011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430986+0000","last_change":"2024-04-02T13:20:20.906617+0000","last_active":"2024-04-02T13:20:26.430986+0000","last_peered":"2024-04-02T13:20:26.430986+0000","last_clean":"2024-04-02T13:20:26.430986+0000","last_became_active":"2024-04-02T13:20:20.905272+0000","last_became_peered":"2024-04-02T13:20:20.905272+0000","last_unstale":"2024-04-02T13:20:26.430986+0000","last_undegraded":"2024-04-02T13:20:26.430986+0000","last_fullsized":"2024-04-02T13:20:26.430986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432098+0000","last_change":"2024-04-02T13:20:19.894435+0000","last_active":"2024-04-02T13:20:26.432098+0000","last_peered":"2024-04-02T13:20:26.432098+0000","last_clean":"2024-04-02T13:20:26.432098+0000","last_became_active":"2024-04-02T13:20:19.894163+0000","last_became_peered":"2024-04-02T13:20:19.894163+0000","last_unstale":"2024-04-02T13:20:26.432098+0000","last_undegraded":"2024-04-02T13:20:26.432098+0000","last_fullsized":"2024-04-02T13:20:26.432098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:44:36.097312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914207+0000","last_change":"2024-04-02T13:20:24.970827+0000","last_active":"2024-04-02T13:20:25.914207+0000","last_peered":"2024-04-02T13:20:25.914207+0000","last_clean":"2024-04-02T13:20:25.914207+0000","last_became_active":"2024-04-02T13:20:24.970384+0000","last_became_peered":"2024-04-02T13:20:24.970384+0000","last_unstale":"2024-04-02T13:20:25.914207+0000","last_undegraded":"2024-04-02T13:20:25.914207+0000","last_fullsized":"2024-04-02T13:20:25.914207+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:20:58.050570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431335+0000","last_change":"2024-04-02T13:20:20.899063+0000","last_active":"2024-04-02T13:20:26.431335+0000","last_peered":"2024-04-02T13:20:26.431335+0000","last_clean":"2024-04-02T13:20:26.431335+0000","last_became_active":"2024-04-02T13:20:20.898735+0000","last_became_peered":"2024-04-02T13:20:20.898735+0000","last_unstale":"2024-04-02T13:20:26.431335+0000","last_undegraded":"2024-04-02T13:20:26.431335+0000","last_fullsized":"2024-04-02T13:20:26.431335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:37.660148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:30.433043+0000","last_change":"2024-04-02T13:20:19.898739+0000","last_active":"2024-04-02T13:20:30.433043+0000","last_peered":"2024-04-02T13:20:30.433043+0000","last_clean":"2024-04-02T13:20:30.433043+0000","last_became_active":"2024-04-02T13:20:19.898524+0000","last_became_peered":"2024-04-02T13:20:19.898524+0000","last_unstale":"2024-04-02T13:20:30.433043+0000","last_undegraded":"2024-04-02T13:20:30.433043+0000","last_fullsized":"2024-04-02T13:20:30.433043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432653+0000","last_change":"2024-04-02T13:20:24.941673+0000","last_active":"2024-04-02T13:20:26.432653+0000","last_peered":"2024-04-02T13:20:26.432653+0000","last_clean":"2024-04-02T13:20:26.432653+0000","last_became_active":"2024-04-02T13:20:24.941096+0000","last_became_peered":"2024-04-02T13:20:24.941096+0000","last_unstale":"2024-04-02T13:20:26.432653+0000","last_undegraded":"2024-04-02T13:20:26.432653+0000","last_fullsized":"2024-04-02T13:20:26.432653+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:26:20.333590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913928+0000","last_change":"2024-04-02T13:20:20.900373+0000","last_active":"2024-04-02T13:20:25.913928+0000","last_peered":"2024-04-02T13:20:25.913928+0000","last_clean":"2024-04-02T13:20:25.913928+0000","last_became_active":"2024-04-02T13:20:20.900084+0000","last_became_peered":"2024-04-02T13:20:20.900084+0000","last_unstale":"2024-04-02T13:20:25.913928+0000","last_undegraded":"2024-04-02T13:20:25.913928+0000","last_fullsized":"2024-04-02T13:20:25.913928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:09.661052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432591+0000","last_change":"2024-04-02T13:20:19.896178+0000","last_active":"2024-04-02T13:20:26.432591+0000","last_peered":"2024-04-02T13:20:26.432591+0000","last_clean":"2024-04-02T13:20:26.432591+0000","last_became_active":"2024-04-02T13:20:19.895887+0000","last_became_peered":"2024-04-02T13:20:19.895887+0000","last_unstale":"2024-04-02T13:20:26.432591+0000","last_undegraded":"2024-04-02T13:20:26.432591+0000","last_fullsized":"2024-04-02T13:20:26.432591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912814+0000","last_change":"2024-04-02T13:20:24.932941+0000","last_active":"2024-04-02T13:20:25.912814+0000","last_peered":"2024-04-02T13:20:25.912814+0000","last_clean":"2024-04-02T13:20:25.912814+0000","last_became_active":"2024-04-02T13:20:24.932319+0000","last_became_peered":"2024-04-02T13:20:24.932319+0000","last_unstale":"2024-04-02T13:20:25.912814+0000","last_undegraded":"2024-04-02T13:20:25.912814+0000","last_fullsized":"2024-04-02T13:20:25.912814+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:57:03.988715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431409+0000","last_change":"2024-04-02T13:20:20.902208+0000","last_active":"2024-04-02T13:20:26.431409+0000","last_peered":"2024-04-02T13:20:26.431409+0000","last_clean":"2024-04-02T13:20:26.431409+0000","last_became_active":"2024-04-02T13:20:20.901921+0000","last_became_peered":"2024-04-02T13:20:20.901921+0000","last_unstale":"2024-04-02T13:20:26.431409+0000","last_undegraded":"2024-04-02T13:20:26.431409+0000","last_fullsized":"2024-04-02T13:20:26.431409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913327+0000","last_change":"2024-04-02T13:20:19.893917+0000","last_active":"2024-04-02T13:20:25.913327+0000","last_peered":"2024-04-02T13:20:25.913327+0000","last_clean":"2024-04-02T13:20:25.913327+0000","last_became_active":"2024-04-02T13:20:19.893652+0000","last_became_peered":"2024-04-02T13:20:19.893652+0000","last_unstale":"2024-04-02T13:20:25.913327+0000","last_undegraded":"2024-04-02T13:20:25.913327+0000","last_fullsized":"2024-04-02T13:20:25.913327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:05.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910393+0000","last_change":"2024-04-02T13:20:24.928526+0000","last_active":"2024-04-02T13:20:25.910393+0000","last_peered":"2024-04-02T13:20:25.910393+0000","last_clean":"2024-04-02T13:20:25.910393+0000","last_became_active":"2024-04-02T13:20:24.928135+0000","last_became_peered":"2024-04-02T13:20:24.928135+0000","last_unstale":"2024-04-02T13:20:25.910393+0000","last_undegraded":"2024-04-02T13:20:25.910393+0000","last_fullsized":"2024-04-02T13:20:25.910393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:18.102976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913243+0000","last_change":"2024-04-02T13:20:20.904387+0000","last_active":"2024-04-02T13:20:25.913243+0000","last_peered":"2024-04-02T13:20:25.913243+0000","last_clean":"2024-04-02T13:20:25.913243+0000","last_became_active":"2024-04-02T13:20:20.903444+0000","last_became_peered":"2024-04-02T13:20:20.903444+0000","last_unstale":"2024-04-02T13:20:25.913243+0000","last_undegraded":"2024-04-02T13:20:25.913243+0000","last_fullsized":"2024-04-02T13:20:25.913243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:25.469551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431417+0000","last_change":"2024-04-02T13:20:19.894102+0000","last_active":"2024-04-02T13:20:26.431417+0000","last_peered":"2024-04-02T13:20:26.431417+0000","last_clean":"2024-04-02T13:20:26.431417+0000","last_became_active":"2024-04-02T13:20:19.893813+0000","last_became_peered":"2024-04-02T13:20:19.893813+0000","last_unstale":"2024-04-02T13:20:26.431417+0000","last_undegraded":"2024-04-02T13:20:26.431417+0000","last_fullsized":"2024-04-02T13:20:26.431417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:56.359644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910302+0000","last_change":"2024-04-02T13:20:24.970905+0000","last_active":"2024-04-02T13:20:25.910302+0000","last_peered":"2024-04-02T13:20:25.910302+0000","last_clean":"2024-04-02T13:20:25.910302+0000","last_became_active":"2024-04-02T13:20:24.970252+0000","last_became_peered":"2024-04-02T13:20:24.970252+0000","last_unstale":"2024-04-02T13:20:25.910302+0000","last_undegraded":"2024-04-02T13:20:25.910302+0000","last_fullsized":"2024-04-02T13:20:25.910302+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:36:46.624236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431088+0000","last_change":"2024-04-02T13:20:24.973949+0000","last_active":"2024-04-02T13:20:26.431088+0000","last_peered":"2024-04-02T13:20:26.431088+0000","last_clean":"2024-04-02T13:20:26.431088+0000","last_became_active":"2024-04-02T13:20:24.973471+0000","last_became_peered":"2024-04-02T13:20:24.973471+0000","last_unstale":"2024-04-02T13:20:26.431088+0000","last_undegraded":"2024-04-02T13:20:26.431088+0000","last_fullsized":"2024-04-02T13:20:26.431088+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:29.504244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430330+0000","last_change":"2024-04-02T13:20:20.892943+0000","last_active":"2024-04-02T13:20:26.430330+0000","last_peered":"2024-04-02T13:20:26.430330+0000","last_clean":"2024-04-02T13:20:26.430330+0000","last_became_active":"2024-04-02T13:20:20.892649+0000","last_became_peered":"2024-04-02T13:20:20.892649+0000","last_unstale":"2024-04-02T13:20:26.430330+0000","last_undegraded":"2024-04-02T13:20:26.430330+0000","last_fullsized":"2024-04-02T13:20:26.430330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913844+0000","last_change":"2024-04-02T13:20:19.897517+0000","last_active":"2024-04-02T13:20:25.913844+0000","last_peered":"2024-04-02T13:20:25.913844+0000","last_clean":"2024-04-02T13:20:25.913844+0000","last_became_active":"2024-04-02T13:20:19.897158+0000","last_became_peered":"2024-04-02T13:20:19.897158+0000","last_unstale":"2024-04-02T13:20:25.913844+0000","last_undegraded":"2024-04-02T13:20:25.913844+0000","last_fullsized":"2024-04-02T13:20:25.913844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431290+0000","last_change":"2024-04-02T13:20:24.932523+0000","last_active":"2024-04-02T13:20:26.431290+0000","last_peered":"2024-04-02T13:20:26.431290+0000","last_clean":"2024-04-02T13:20:26.431290+0000","last_became_active":"2024-04-02T13:20:24.932122+0000","last_became_peered":"2024-04-02T13:20:24.932122+0000","last_unstale":"2024-04-02T13:20:26.431290+0000","last_undegraded":"2024-04-02T13:20:26.431290+0000","last_fullsized":"2024-04-02T13:20:26.431290+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:06.819681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910293+0000","last_change":"2024-04-02T13:20:20.900650+0000","last_active":"2024-04-02T13:20:25.910293+0000","last_peered":"2024-04-02T13:20:25.910293+0000","last_clean":"2024-04-02T13:20:25.910293+0000","last_became_active":"2024-04-02T13:20:20.900371+0000","last_became_peered":"2024-04-02T13:20:20.900371+0000","last_unstale":"2024-04-02T13:20:25.910293+0000","last_undegraded":"2024-04-02T13:20:25.910293+0000","last_fullsized":"2024-04-02T13:20:25.910293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910494+0000","last_change":"2024-04-02T13:20:19.894065+0000","last_active":"2024-04-02T13:20:25.910494+0000","last_peered":"2024-04-02T13:20:25.910494+0000","last_clean":"2024-04-02T13:20:25.910494+0000","last_became_active":"2024-04-02T13:20:19.893749+0000","last_became_peered":"2024-04-02T13:20:19.893749+0000","last_unstale":"2024-04-02T13:20:25.910494+0000","last_undegraded":"2024-04-02T13:20:25.910494+0000","last_fullsized":"2024-04-02T13:20:25.910494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:42:03.725417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430909+0000","last_change":"2024-04-02T13:20:19.892884+0000","last_active":"2024-04-02T13:20:26.430909+0000","last_peered":"2024-04-02T13:20:26.430909+0000","last_clean":"2024-04-02T13:20:26.430909+0000","last_became_active":"2024-04-02T13:20:19.892641+0000","last_became_peered":"2024-04-02T13:20:19.892641+0000","last_unstale":"2024-04-02T13:20:26.430909+0000","last_undegraded":"2024-04-02T13:20:26.430909+0000","last_fullsized":"2024-04-02T13:20:26.430909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432078+0000","last_change":"2024-04-02T13:20:20.900588+0000","last_active":"2024-04-02T13:20:26.432078+0000","last_peered":"2024-04-02T13:20:26.432078+0000","last_clean":"2024-04-02T13:20:26.432078+0000","last_became_active":"2024-04-02T13:20:20.900299+0000","last_became_peered":"2024-04-02T13:20:20.900299+0000","last_unstale":"2024-04-02T13:20:26.432078+0000","last_undegraded":"2024-04-02T13:20:26.432078+0000","last_fullsized":"2024-04-02T13:20:26.432078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431129+0000","last_change":"2024-04-02T13:20:24.937767+0000","last_active":"2024-04-02T13:20:26.431129+0000","last_peered":"2024-04-02T13:20:26.431129+0000","last_clean":"2024-04-02T13:20:26.431129+0000","last_became_active":"2024-04-02T13:20:24.937371+0000","last_became_peered":"2024-04-02T13:20:24.937371+0000","last_unstale":"2024-04-02T13:20:26.431129+0000","last_undegraded":"2024-04-02T13:20:26.431129+0000","last_fullsized":"2024-04-02T13:20:26.431129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:52.883251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910649+0000","last_change":"2024-04-02T13:20:19.895568+0000","last_active":"2024-04-02T13:20:25.910649+0000","last_peered":"2024-04-02T13:20:25.910649+0000","last_clean":"2024-04-02T13:20:25.910649+0000","last_became_active":"2024-04-02T13:20:19.895240+0000","last_became_peered":"2024-04-02T13:20:19.895240+0000","last_unstale":"2024-04-02T13:20:25.910649+0000","last_undegraded":"2024-04-02T13:20:25.910649+0000","last_fullsized":"2024-04-02T13:20:25.910649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:56.359644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912900+0000","last_change":"2024-04-02T13:20:20.894414+0000","last_active":"2024-04-02T13:20:25.912900+0000","last_peered":"2024-04-02T13:20:25.912900+0000","last_clean":"2024-04-02T13:20:25.912900+0000","last_became_active":"2024-04-02T13:20:20.894129+0000","last_became_peered":"2024-04-02T13:20:20.894129+0000","last_unstale":"2024-04-02T13:20:25.912900+0000","last_undegraded":"2024-04-02T13:20:25.912900+0000","last_fullsized":"2024-04-02T13:20:25.912900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:01.268837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911029+0000","last_change":"2024-04-02T13:20:24.932278+0000","last_active":"2024-04-02T13:20:25.911029+0000","last_peered":"2024-04-02T13:20:25.911029+0000","last_clean":"2024-04-02T13:20:25.911029+0000","last_became_active":"2024-04-02T13:20:24.931901+0000","last_became_peered":"2024-04-02T13:20:24.931901+0000","last_unstale":"2024-04-02T13:20:25.911029+0000","last_undegraded":"2024-04-02T13:20:25.911029+0000","last_fullsized":"2024-04-02T13:20:25.911029+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:47.846911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430361+0000","last_change":"2024-04-02T13:20:20.898502+0000","last_active":"2024-04-02T13:20:26.430361+0000","last_peered":"2024-04-02T13:20:26.430361+0000","last_clean":"2024-04-02T13:20:26.430361+0000","last_became_active":"2024-04-02T13:20:20.898208+0000","last_became_peered":"2024-04-02T13:20:20.898208+0000","last_unstale":"2024-04-02T13:20:26.430361+0000","last_undegraded":"2024-04-02T13:20:26.430361+0000","last_fullsized":"2024-04-02T13:20:26.430361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431659+0000","last_change":"2024-04-02T13:20:20.905276+0000","last_active":"2024-04-02T13:20:26.431659+0000","last_peered":"2024-04-02T13:20:26.431659+0000","last_clean":"2024-04-02T13:20:26.431659+0000","last_became_active":"2024-04-02T13:20:20.905010+0000","last_became_peered":"2024-04-02T13:20:20.905010+0000","last_unstale":"2024-04-02T13:20:26.431659+0000","last_undegraded":"2024-04-02T13:20:26.431659+0000","last_fullsized":"2024-04-02T13:20:26.431659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:09.661052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912909+0000","last_change":"2024-04-02T13:20:20.908237+0000","last_active":"2024-04-02T13:20:25.912909+0000","last_peered":"2024-04-02T13:20:25.912909+0000","last_clean":"2024-04-02T13:20:25.912909+0000","last_became_active":"2024-04-02T13:20:20.907967+0000","last_became_peered":"2024-04-02T13:20:20.907967+0000","last_unstale":"2024-04-02T13:20:25.912909+0000","last_undegraded":"2024-04-02T13:20:25.912909+0000","last_fullsized":"2024-04-02T13:20:25.912909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:02.784989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431982+0000","last_change":"2024-04-02T13:20:20.901918+0000","last_active":"2024-04-02T13:20:26.431982+0000","last_peered":"2024-04-02T13:20:26.431982+0000","last_clean":"2024-04-02T13:20:26.431982+0000","last_became_active":"2024-04-02T13:20:20.901628+0000","last_became_peered":"2024-04-02T13:20:20.901628+0000","last_unstale":"2024-04-02T13:20:26.431982+0000","last_undegraded":"2024-04-02T13:20:26.431982+0000","last_fullsized":"2024-04-02T13:20:26.431982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911122+0000","last_change":"2024-04-02T13:20:20.899007+0000","last_active":"2024-04-02T13:20:25.911122+0000","last_peered":"2024-04-02T13:20:25.911122+0000","last_clean":"2024-04-02T13:20:25.911122+0000","last_became_active":"2024-04-02T13:20:20.898005+0000","last_became_peered":"2024-04-02T13:20:20.898005+0000","last_unstale":"2024-04-02T13:20:25.911122+0000","last_undegraded":"2024-04-02T13:20:25.911122+0000","last_fullsized":"2024-04-02T13:20:25.911122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432495+0000","last_change":"2024-04-02T13:20:20.896893+0000","last_active":"2024-04-02T13:20:26.432495+0000","last_peered":"2024-04-02T13:20:26.432495+0000","last_clean":"2024-04-02T13:20:26.432495+0000","last_became_active":"2024-04-02T13:20:20.896629+0000","last_became_peered":"2024-04-02T13:20:20.896629+0000","last_unstale":"2024-04-02T13:20:26.432495+0000","last_undegraded":"2024-04-02T13:20:26.432495+0000","last_fullsized":"2024-04-02T13:20:26.432495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431161+0000","last_change":"2024-04-02T13:20:20.900656+0000","last_active":"2024-04-02T13:20:26.431161+0000","last_peered":"2024-04-02T13:20:26.431161+0000","last_clean":"2024-04-02T13:20:26.431161+0000","last_became_active":"2024-04-02T13:20:20.900385+0000","last_became_peered":"2024-04-02T13:20:20.900385+0000","last_unstale":"2024-04-02T13:20:26.431161+0000","last_undegraded":"2024-04-02T13:20:26.431161+0000","last_fullsized":"2024-04-02T13:20:26.431161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911108+0000","last_change":"2024-04-02T13:20:20.897613+0000","last_active":"2024-04-02T13:20:25.911108+0000","last_peered":"2024-04-02T13:20:25.911108+0000","last_clean":"2024-04-02T13:20:25.911108+0000","last_became_active":"2024-04-02T13:20:20.897285+0000","last_became_peered":"2024-04-02T13:20:20.897285+0000","last_unstale":"2024-04-02T13:20:25.911108+0000","last_undegraded":"2024-04-02T13:20:25.911108+0000","last_fullsized":"2024-04-02T13:20:25.911108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913747+0000","last_change":"2024-04-02T13:20:20.907084+0000","last_active":"2024-04-02T13:20:25.913747+0000","last_peered":"2024-04-02T13:20:25.913747+0000","last_clean":"2024-04-02T13:20:25.913747+0000","last_became_active":"2024-04-02T13:20:20.906734+0000","last_became_peered":"2024-04-02T13:20:20.906734+0000","last_unstale":"2024-04-02T13:20:25.913747+0000","last_undegraded":"2024-04-02T13:20:25.913747+0000","last_fullsized":"2024-04-02T13:20:25.913747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910526+0000","last_change":"2024-04-02T13:20:20.897768+0000","last_active":"2024-04-02T13:20:25.910526+0000","last_peered":"2024-04-02T13:20:25.910526+0000","last_clean":"2024-04-02T13:20:25.910526+0000","last_became_active":"2024-04-02T13:20:20.897502+0000","last_became_peered":"2024-04-02T13:20:20.897502+0000","last_unstale":"2024-04-02T13:20:25.910526+0000","last_undegraded":"2024-04-02T13:20:25.910526+0000","last_fullsized":"2024-04-02T13:20:25.910526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:26.532089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912825+0000","last_change":"2024-04-02T13:20:20.903454+0000","last_active":"2024-04-02T13:20:25.912825+0000","last_peered":"2024-04-02T13:20:25.912825+0000","last_clean":"2024-04-02T13:20:25.912825+0000","last_became_active":"2024-04-02T13:20:20.903050+0000","last_became_peered":"2024-04-02T13:20:20.903050+0000","last_unstale":"2024-04-02T13:20:25.912825+0000","last_undegraded":"2024-04-02T13:20:25.912825+0000","last_fullsized":"2024-04-02T13:20:25.912825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:56:59.954023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431891+0000","last_change":"2024-04-02T13:20:20.904681+0000","last_active":"2024-04-02T13:20:26.431891+0000","last_peered":"2024-04-02T13:20:26.431891+0000","last_clean":"2024-04-02T13:20:26.431891+0000","last_became_active":"2024-04-02T13:20:20.904343+0000","last_became_peered":"2024-04-02T13:20:20.904343+0000","last_unstale":"2024-04-02T13:20:26.431891+0000","last_undegraded":"2024-04-02T13:20:26.431891+0000","last_fullsized":"2024-04-02T13:20:26.431891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:22.414825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431123+0000","last_change":"2024-04-02T13:20:20.895363+0000","last_active":"2024-04-02T13:20:26.431123+0000","last_peered":"2024-04-02T13:20:26.431123+0000","last_clean":"2024-04-02T13:20:26.431123+0000","last_became_active":"2024-04-02T13:20:20.895107+0000","last_became_peered":"2024-04-02T13:20:20.895107+0000","last_unstale":"2024-04-02T13:20:26.431123+0000","last_undegraded":"2024-04-02T13:20:26.431123+0000","last_fullsized":"2024-04-02T13:20:26.431123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:22.414825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431618+0000","last_change":"2024-04-02T13:20:20.890501+0000","last_active":"2024-04-02T13:20:26.431618+0000","last_peered":"2024-04-02T13:20:26.431618+0000","last_clean":"2024-04-02T13:20:26.431618+0000","last_became_active":"2024-04-02T13:20:20.890181+0000","last_became_peered":"2024-04-02T13:20:20.890181+0000","last_unstale":"2024-04-02T13:20:26.431618+0000","last_undegraded":"2024-04-02T13:20:26.431618+0000","last_fullsized":"2024-04-02T13:20:26.431618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:02:18.661127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430915+0000","last_change":"2024-04-02T13:20:20.892430+0000","last_active":"2024-04-02T13:20:26.430915+0000","last_peered":"2024-04-02T13:20:26.430915+0000","last_clean":"2024-04-02T13:20:26.430915+0000","last_became_active":"2024-04-02T13:20:20.892117+0000","last_became_peered":"2024-04-02T13:20:20.892117+0000","last_unstale":"2024-04-02T13:20:26.430915+0000","last_undegraded":"2024-04-02T13:20:26.430915+0000","last_fullsized":"2024-04-02T13:20:26.430915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:37.366302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913765+0000","last_change":"2024-04-02T13:20:20.903371+0000","last_active":"2024-04-02T13:20:25.913765+0000","last_peered":"2024-04-02T13:20:25.913765+0000","last_clean":"2024-04-02T13:20:25.913765+0000","last_became_active":"2024-04-02T13:20:20.903078+0000","last_became_peered":"2024-04-02T13:20:20.903078+0000","last_unstale":"2024-04-02T13:20:25.913765+0000","last_undegraded":"2024-04-02T13:20:25.913765+0000","last_fullsized":"2024-04-02T13:20:25.913765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:42.709577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913653+0000","last_change":"2024-04-02T13:20:20.901550+0000","last_active":"2024-04-02T13:20:25.913653+0000","last_peered":"2024-04-02T13:20:25.913653+0000","last_clean":"2024-04-02T13:20:25.913653+0000","last_became_active":"2024-04-02T13:20:20.901234+0000","last_became_peered":"2024-04-02T13:20:20.901234+0000","last_unstale":"2024-04-02T13:20:25.913653+0000","last_undegraded":"2024-04-02T13:20:25.913653+0000","last_fullsized":"2024-04-02T13:20:25.913653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430422+0000","last_change":"2024-04-02T13:20:20.887101+0000","last_active":"2024-04-02T13:20:26.430422+0000","last_peered":"2024-04-02T13:20:26.430422+0000","last_clean":"2024-04-02T13:20:26.430422+0000","last_became_active":"2024-04-02T13:20:20.886810+0000","last_became_peered":"2024-04-02T13:20:20.886810+0000","last_unstale":"2024-04-02T13:20:26.430422+0000","last_undegraded":"2024-04-02T13:20:26.430422+0000","last_fullsized":"2024-04-02T13:20:26.430422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430889+0000","last_change":"2024-04-02T13:20:20.898387+0000","last_active":"2024-04-02T13:20:26.430889+0000","last_peered":"2024-04-02T13:20:26.430889+0000","last_clean":"2024-04-02T13:20:26.430889+0000","last_became_active":"2024-04-02T13:20:20.898075+0000","last_became_peered":"2024-04-02T13:20:20.898075+0000","last_unstale":"2024-04-02T13:20:26.430889+0000","last_undegraded":"2024-04-02T13:20:26.430889+0000","last_fullsized":"2024-04-02T13:20:26.430889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910512+0000","last_change":"2024-04-02T13:20:20.895603+0000","last_active":"2024-04-02T13:20:25.910512+0000","last_peered":"2024-04-02T13:20:25.910512+0000","last_clean":"2024-04-02T13:20:25.910512+0000","last_became_active":"2024-04-02T13:20:20.895316+0000","last_became_peered":"2024-04-02T13:20:20.895316+0000","last_unstale":"2024-04-02T13:20:25.910512+0000","last_undegraded":"2024-04-02T13:20:25.910512+0000","last_fullsized":"2024-04-02T13:20:25.910512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:34.303387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913831+0000","last_change":"2024-04-02T13:20:20.889381+0000","last_active":"2024-04-02T13:20:25.913831+0000","last_peered":"2024-04-02T13:20:25.913831+0000","last_clean":"2024-04-02T13:20:25.913831+0000","last_became_active":"2024-04-02T13:20:20.889102+0000","last_became_peered":"2024-04-02T13:20:20.889102+0000","last_unstale":"2024-04-02T13:20:25.913831+0000","last_undegraded":"2024-04-02T13:20:25.913831+0000","last_fullsized":"2024-04-02T13:20:25.913831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:37.366302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430695+0000","last_change":"2024-04-02T13:20:20.904474+0000","last_active":"2024-04-02T13:20:26.430695+0000","last_peered":"2024-04-02T13:20:26.430695+0000","last_clean":"2024-04-02T13:20:26.430695+0000","last_became_active":"2024-04-02T13:20:20.904212+0000","last_became_peered":"2024-04-02T13:20:20.904212+0000","last_unstale":"2024-04-02T13:20:26.430695+0000","last_undegraded":"2024-04-02T13:20:26.430695+0000","last_fullsized":"2024-04-02T13:20:26.430695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431059+0000","last_change":"2024-04-02T13:20:20.904469+0000","last_active":"2024-04-02T13:20:26.431059+0000","last_peered":"2024-04-02T13:20:26.431059+0000","last_clean":"2024-04-02T13:20:26.431059+0000","last_became_active":"2024-04-02T13:20:20.904195+0000","last_became_peered":"2024-04-02T13:20:20.904195+0000","last_unstale":"2024-04-02T13:20:26.431059+0000","last_undegraded":"2024-04-02T13:20:26.431059+0000","last_fullsized":"2024-04-02T13:20:26.431059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431650+0000","last_change":"2024-04-02T13:20:20.905842+0000","last_active":"2024-04-02T13:20:26.431650+0000","last_peered":"2024-04-02T13:20:26.431650+0000","last_clean":"2024-04-02T13:20:26.431650+0000","last_became_active":"2024-04-02T13:20:20.905558+0000","last_became_peered":"2024-04-02T13:20:20.905558+0000","last_unstale":"2024-04-02T13:20:26.431650+0000","last_undegraded":"2024-04-02T13:20:26.431650+0000","last_fullsized":"2024-04-02T13:20:26.431650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:02.784989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910120+0000","last_change":"2024-04-02T13:20:20.902048+0000","last_active":"2024-04-02T13:20:25.910120+0000","last_peered":"2024-04-02T13:20:25.910120+0000","last_clean":"2024-04-02T13:20:25.910120+0000","last_became_active":"2024-04-02T13:20:20.901733+0000","last_became_peered":"2024-04-02T13:20:20.901733+0000","last_unstale":"2024-04-02T13:20:25.910120+0000","last_undegraded":"2024-04-02T13:20:25.910120+0000","last_fullsized":"2024-04-02T13:20:25.910120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431149+0000","last_change":"2024-04-02T13:20:20.901947+0000","last_active":"2024-04-02T13:20:26.431149+0000","last_peered":"2024-04-02T13:20:26.431149+0000","last_clean":"2024-04-02T13:20:26.431149+0000","last_became_active":"2024-04-02T13:20:20.901655+0000","last_became_peered":"2024-04-02T13:20:20.901655+0000","last_unstale":"2024-04-02T13:20:26.431149+0000","last_undegraded":"2024-04-02T13:20:26.431149+0000","last_fullsized":"2024-04-02T13:20:26.431149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:56:59.954023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912733+0000","last_change":"2024-04-02T13:20:20.905389+0000","last_active":"2024-04-02T13:20:25.912733+0000","last_peered":"2024-04-02T13:20:25.912733+0000","last_clean":"2024-04-02T13:20:25.912733+0000","last_became_active":"2024-04-02T13:20:20.905062+0000","last_became_peered":"2024-04-02T13:20:20.905062+0000","last_unstale":"2024-04-02T13:20:25.912733+0000","last_undegraded":"2024-04-02T13:20:25.912733+0000","last_fullsized":"2024-04-02T13:20:25.912733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:34.303387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430753+0000","last_change":"2024-04-02T13:20:20.897784+0000","last_active":"2024-04-02T13:20:26.430753+0000","last_peered":"2024-04-02T13:20:26.430753+0000","last_clean":"2024-04-02T13:20:26.430753+0000","last_became_active":"2024-04-02T13:20:20.897488+0000","last_became_peered":"2024-04-02T13:20:20.897488+0000","last_unstale":"2024-04-02T13:20:26.430753+0000","last_undegraded":"2024-04-02T13:20:26.430753+0000","last_fullsized":"2024-04-02T13:20:26.430753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:26:16.298897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912800+0000","last_change":"2024-04-02T13:20:20.908726+0000","last_active":"2024-04-02T13:20:25.912800+0000","last_peered":"2024-04-02T13:20:25.912800+0000","last_clean":"2024-04-02T13:20:25.912800+0000","last_became_active":"2024-04-02T13:20:20.908504+0000","last_became_peered":"2024-04-02T13:20:20.908504+0000","last_unstale":"2024-04-02T13:20:25.912800+0000","last_undegraded":"2024-04-02T13:20:25.912800+0000","last_fullsized":"2024-04-02T13:20:25.912800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:26.532089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430330+0000","last_change":"2024-04-02T13:20:20.892134+0000","last_active":"2024-04-02T13:20:26.430330+0000","last_peered":"2024-04-02T13:20:26.430330+0000","last_clean":"2024-04-02T13:20:26.430330+0000","last_became_active":"2024-04-02T13:20:20.891860+0000","last_became_peered":"2024-04-02T13:20:20.891860+0000","last_unstale":"2024-04-02T13:20:26.430330+0000","last_undegraded":"2024-04-02T13:20:26.430330+0000","last_fullsized":"2024-04-02T13:20:26.430330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:17:34.808750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431697+0000","last_change":"2024-04-02T13:20:20.890864+0000","last_active":"2024-04-02T13:20:26.431697+0000","last_peered":"2024-04-02T13:20:26.431697+0000","last_clean":"2024-04-02T13:20:26.431697+0000","last_became_active":"2024-04-02T13:20:20.890585+0000","last_became_peered":"2024-04-02T13:20:20.890585+0000","last_unstale":"2024-04-02T13:20:26.431697+0000","last_undegraded":"2024-04-02T13:20:26.431697+0000","last_fullsized":"2024-04-02T13:20:26.431697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431543+0000","last_change":"2024-04-02T13:20:20.892679+0000","last_active":"2024-04-02T13:20:26.431543+0000","last_peered":"2024-04-02T13:20:26.431543+0000","last_clean":"2024-04-02T13:20:26.431543+0000","last_became_active":"2024-04-02T13:20:20.892440+0000","last_became_peered":"2024-04-02T13:20:20.892440+0000","last_unstale":"2024-04-02T13:20:26.431543+0000","last_undegraded":"2024-04-02T13:20:26.431543+0000","last_fullsized":"2024-04-02T13:20:26.431543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:39.096965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:20:48.475 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:20:48.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.609+0000 7f000577b640 1 Processor -- start 2024-04-02T13:20:48.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.609+0000 7f000577b640 1 -- start start 2024-04-02T13:20:48.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.610+0000 7f000577b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f000015e550 0x7f000006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.610+0000 7f000577b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f000015f040 0x7f00000d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.610+0000 7f000577b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f000015fb50 con 0x7f000006d530 2024-04-02T13:20:48.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.610+0000 7f000577b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f000006d970 con 0x7f000015f040 2024-04-02T13:20:48.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.610+0000 7f000577b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f000006ca90 con 0x7f000015e550 2024-04-02T13:20:48.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.610+0000 7effff7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f000015f040 0x7f00000d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.610+0000 7effff7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f000015f040 0x7f00000d0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55062/0 (socket says 172.21.15.44:55062) 2024-04-02T13:20:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.610+0000 7effff7fe640 1 -- 172.21.15.44:0/2285295609 learned_addr learned my addr 172.21.15.44:0/2285295609 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.610+0000 7efffeffd640 1 --2- 172.21.15.44:0/2285295609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f000015e550 0x7f000006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.611+0000 7effff7fe640 1 -- 172.21.15.44:0/2285295609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f000015e550 msgr2=0x7f000006cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.611+0000 7effff7fe640 1 --2- 172.21.15.44:0/2285295609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f000015e550 0x7f000006cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.611+0000 7effff7fe640 1 -- 172.21.15.44:0/2285295609 >> v1:172.21.15.69:6789/0 conn(0x7f000006d530 legacy=0x7f000015e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.611+0000 7effff7fe640 1 -- 172.21.15.44:0/2285295609 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f000015f590 con 0x7f000015f040 2024-04-02T13:20:48.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.611+0000 7effff7fe640 1 --2- 172.21.15.44:0/2285295609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f000015f040 0x7f00000d0ab0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efff0009e50 tx=0x7efff002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f602866867862230 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:48.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.612+0000 7efffdffb640 1 -- 172.21.15.44:0/2285295609 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff002cb90 con 0x7f000015f040 2024-04-02T13:20:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.612+0000 7efffdffb640 1 -- 172.21.15.44:0/2285295609 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efff002cd30 con 0x7f000015f040 2024-04-02T13:20:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.612+0000 7efffdffb640 1 -- 172.21.15.44:0/2285295609 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff0034610 con 0x7f000015f040 2024-04-02T13:20:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.612+0000 7f000577b640 1 -- 172.21.15.44:0/2285295609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f000015f040 msgr2=0x7f00000d0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.612+0000 7f000577b640 1 --2- 172.21.15.44:0/2285295609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f000015f040 0x7f00000d0ab0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efff0009e50 tx=0x7efff002bff0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.612+0000 7f000577b640 1 -- 172.21.15.44:0/2285295609 shutdown_connections 2024-04-02T13:20:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.612+0000 7f000577b640 1 --2- 172.21.15.44:0/2285295609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f000015f040 0x7f00000d0ab0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.612+0000 7f000577b640 1 --2- 172.21.15.44:0/2285295609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f000015e550 0x7f000006cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.612+0000 7f000577b640 1 -- 172.21.15.44:0/2285295609 >> 172.21.15.44:0/2285295609 conn(0x7f000009aa20 msgr2=0x7f000009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.612+0000 7f000577b640 1 -- 172.21.15.44:0/2285295609 shutdown_connections 2024-04-02T13:20:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.612+0000 7f000577b640 1 -- 172.21.15.44:0/2285295609 wait complete. 2024-04-02T13:20:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.613+0000 7f000577b640 1 Processor -- start 2024-04-02T13:20:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.614+0000 7f000577b640 1 -- start start 2024-04-02T13:20:48.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.614+0000 7f000577b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f000006d530 0x7f00000c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.614+0000 7f000577b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f000015e550 0x7f000009c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.614+0000 7f000577b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f000015f040 0x7f000009ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.614+0000 7f000577b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f00000d0590 con 0x7f000006d530 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.614+0000 7f000577b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f00000d02b0 con 0x7f000015f040 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.614+0000 7f000577b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f00000cc4b0 con 0x7f000015e550 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.614+0000 7efffeffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f000006d530 0x7f00000c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.614+0000 7efffeffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f000006d530 0x7f00000c9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55078/0 (socket says 172.21.15.44:55078) 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.614+0000 7efffeffd640 1 -- 172.21.15.44:0/940874935 learned_addr learned my addr 172.21.15.44:0/940874935 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.615+0000 7efffe7fc640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f000015e550 0x7f000009c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.615+0000 7effff7fe640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f000015f040 0x7f000009ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.615+0000 7efffeffd640 1 -- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f000015e550 msgr2=0x7f000009c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.615+0000 7efffeffd640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f000015e550 0x7f000009c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.615+0000 7efffeffd640 1 -- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f000015f040 msgr2=0x7f000009ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.615+0000 7efffeffd640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f000015f040 0x7f000009ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.615+0000 7efffeffd640 1 -- 172.21.15.44:0/940874935 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00000b4b80 con 0x7f000006d530 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.615+0000 7efffeffd640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f000006d530 0x7f00000c9940 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efff800e870 tx=0x7efff800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.615+0000 7efff77fe640 1 -- 172.21.15.44:0/940874935 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff8017070 con 0x7f000006d530 2024-04-02T13:20:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.615+0000 7efff77fe640 1 -- 172.21.15.44:0/940874935 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efff800c940 con 0x7f000006d530 2024-04-02T13:20:48.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.615+0000 7f000577b640 1 -- 172.21.15.44:0/940874935 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00000b4e60 con 0x7f000006d530 2024-04-02T13:20:48.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.615+0000 7efff77fe640 1 -- 172.21.15.44:0/940874935 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff800cd30 con 0x7f000006d530 2024-04-02T13:20:48.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.615+0000 7f000577b640 1 -- 172.21.15.44:0/940874935 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00000b5420 con 0x7f000006d530 2024-04-02T13:20:48.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.616+0000 7f000577b640 1 -- 172.21.15.44:0/940874935 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00000684f0 con 0x7f000006d530 2024-04-02T13:20:48.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.616+0000 7efff77fe640 1 -- 172.21.15.44:0/940874935 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efff801a070 con 0x7f000006d530 2024-04-02T13:20:48.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.617+0000 7efff77fe640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7effe006c8f0 0x7effe006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.617+0000 7efff77fe640 1 -- 172.21.15.44:0/940874935 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7efff808c180 con 0x7f000006d530 2024-04-02T13:20:48.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.617+0000 7efffe7fc640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7effe006c8f0 0x7effe006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.617+0000 7efffe7fc640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7effe006c8f0 0x7effe006edb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7effe8007fd0 tx=0x7effe8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:48.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.621+0000 7efff77fe640 1 -- 172.21.15.44:0/940874935 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7efff805a1d0 con 0x7f000006d530 2024-04-02T13:20:48.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.803+0000 7f000577b640 1 -- 172.21.15.44:0/940874935 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f00000b3530 con 0x7effe006c8f0 2024-04-02T13:20:48.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.812+0000 7efff77fe640 1 -- 172.21.15.44:0/940874935 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7f00000b3530 con 0x7effe006c8f0 2024-04-02T13:20:48.817 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:20:48.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.815+0000 7f000577b640 1 -- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7effe006c8f0 msgr2=0x7effe006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.815+0000 7f000577b640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7effe006c8f0 0x7effe006edb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7effe8007fd0 tx=0x7effe8009040 comp rx=0 tx=0).stop 2024-04-02T13:20:48.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.815+0000 7f000577b640 1 -- 172.21.15.44:0/940874935 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f000006d530 msgr2=0x7f00000c9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.815+0000 7f000577b640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f000006d530 0x7f00000c9940 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efff800e870 tx=0x7efff800ed30 comp rx=0 tx=0).stop 2024-04-02T13:20:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.815+0000 7f000577b640 1 -- 172.21.15.44:0/940874935 shutdown_connections 2024-04-02T13:20:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.815+0000 7f000577b640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7effe006c8f0 0x7effe006edb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.815+0000 7f000577b640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f000015f040 0x7f000009ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.815+0000 7f000577b640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f000015e550 0x7f000009c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.815+0000 7f000577b640 1 --2- 172.21.15.44:0/940874935 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f000006d530 0x7f00000c9940 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.815+0000 7f000577b640 1 -- 172.21.15.44:0/940874935 >> 172.21.15.44:0/940874935 conn(0x7f000009aa20 msgr2=0x7f00000ccfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.815+0000 7f000577b640 1 -- 172.21.15.44:0/940874935 shutdown_connections 2024-04-02T13:20:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.815+0000 7f000577b640 1 -- 172.21.15.44:0/940874935 wait complete. 2024-04-02T13:20:48.828 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-04-02T13:20:46.879424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219496,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755224,"statfs":{"total":773094113280,"available":772869349376,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003122"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431510+0000","last_change":"2024-04-02T13:20:19.889381+0000","last_active":"2024-04-02T13:20:26.431510+0000","last_peered":"2024-04-02T13:20:26.431510+0000","last_clean":"2024-04-02T13:20:26.431510+0000","last_became_active":"2024-04-02T13:20:19.889109+0000","last_became_peered":"2024-04-02T13:20:19.889109+0000","last_unstale":"2024-04-02T13:20:26.431510+0000","last_undegraded":"2024-04-02T13:20:26.431510+0000","last_fullsized":"2024-04-02T13:20:26.431510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:42:03.725417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911104+0000","last_change":"2024-04-02T13:20:20.899681+0000","last_active":"2024-04-02T13:20:25.911104+0000","last_peered":"2024-04-02T13:20:25.911104+0000","last_clean":"2024-04-02T13:20:25.911104+0000","last_became_active":"2024-04-02T13:20:20.899408+0000","last_became_peered":"2024-04-02T13:20:20.899408+0000","last_unstale":"2024-04-02T13:20:25.911104+0000","last_undegraded":"2024-04-02T13:20:25.911104+0000","last_fullsized":"2024-04-02T13:20:25.911104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910230+0000","last_change":"2024-04-02T13:20:24.974937+0000","last_active":"2024-04-02T13:20:25.910230+0000","last_peered":"2024-04-02T13:20:25.910230+0000","last_clean":"2024-04-02T13:20:25.910230+0000","last_became_active":"2024-04-02T13:20:24.974496+0000","last_became_peered":"2024-04-02T13:20:24.974496+0000","last_unstale":"2024-04-02T13:20:25.910230+0000","last_undegraded":"2024-04-02T13:20:25.910230+0000","last_fullsized":"2024-04-02T13:20:25.910230+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:20:58.050570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911229+0000","last_change":"2024-04-02T13:20:19.896780+0000","last_active":"2024-04-02T13:20:25.911229+0000","last_peered":"2024-04-02T13:20:25.911229+0000","last_clean":"2024-04-02T13:20:25.911229+0000","last_became_active":"2024-04-02T13:20:19.896480+0000","last_became_peered":"2024-04-02T13:20:19.896480+0000","last_unstale":"2024-04-02T13:20:25.911229+0000","last_undegraded":"2024-04-02T13:20:25.911229+0000","last_fullsized":"2024-04-02T13:20:25.911229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431238+0000","last_change":"2024-04-02T13:20:20.905268+0000","last_active":"2024-04-02T13:20:26.431238+0000","last_peered":"2024-04-02T13:20:26.431238+0000","last_clean":"2024-04-02T13:20:26.431238+0000","last_became_active":"2024-04-02T13:20:20.904997+0000","last_became_peered":"2024-04-02T13:20:20.904997+0000","last_unstale":"2024-04-02T13:20:26.431238+0000","last_undegraded":"2024-04-02T13:20:26.431238+0000","last_fullsized":"2024-04-02T13:20:26.431238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912459+0000","last_change":"2024-04-02T13:20:24.933025+0000","last_active":"2024-04-02T13:20:25.912459+0000","last_peered":"2024-04-02T13:20:25.912459+0000","last_clean":"2024-04-02T13:20:25.912459+0000","last_became_active":"2024-04-02T13:20:24.932475+0000","last_became_peered":"2024-04-02T13:20:24.932475+0000","last_unstale":"2024-04-02T13:20:25.912459+0000","last_undegraded":"2024-04-02T13:20:25.912459+0000","last_fullsized":"2024-04-02T13:20:25.912459+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:46.747491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431519+0000","last_change":"2024-04-02T13:20:19.895931+0000","last_active":"2024-04-02T13:20:26.431519+0000","last_peered":"2024-04-02T13:20:26.431519+0000","last_clean":"2024-04-02T13:20:26.431519+0000","last_became_active":"2024-04-02T13:20:19.895304+0000","last_became_peered":"2024-04-02T13:20:19.895304+0000","last_unstale":"2024-04-02T13:20:26.431519+0000","last_undegraded":"2024-04-02T13:20:26.431519+0000","last_fullsized":"2024-04-02T13:20:26.431519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:08.653626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912892+0000","last_change":"2024-04-02T13:20:20.904770+0000","last_active":"2024-04-02T13:20:25.912892+0000","last_peered":"2024-04-02T13:20:25.912892+0000","last_clean":"2024-04-02T13:20:25.912892+0000","last_became_active":"2024-04-02T13:20:20.903916+0000","last_became_peered":"2024-04-02T13:20:20.903916+0000","last_unstale":"2024-04-02T13:20:25.912892+0000","last_undegraded":"2024-04-02T13:20:25.912892+0000","last_fullsized":"2024-04-02T13:20:25.912892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:39.096965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913056+0000","last_change":"2024-04-02T13:20:24.932562+0000","last_active":"2024-04-02T13:20:25.913056+0000","last_peered":"2024-04-02T13:20:25.913056+0000","last_clean":"2024-04-02T13:20:25.913056+0000","last_became_active":"2024-04-02T13:20:24.931896+0000","last_became_peered":"2024-04-02T13:20:24.931896+0000","last_unstale":"2024-04-02T13:20:25.913056+0000","last_undegraded":"2024-04-02T13:20:25.913056+0000","last_fullsized":"2024-04-02T13:20:25.913056+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:43.131658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430952+0000","last_change":"2024-04-02T13:20:19.887808+0000","last_active":"2024-04-02T13:20:26.430952+0000","last_peered":"2024-04-02T13:20:26.430952+0000","last_clean":"2024-04-02T13:20:26.430952+0000","last_became_active":"2024-04-02T13:20:19.887491+0000","last_became_peered":"2024-04-02T13:20:19.887491+0000","last_unstale":"2024-04-02T13:20:26.430952+0000","last_undegraded":"2024-04-02T13:20:26.430952+0000","last_fullsized":"2024-04-02T13:20:26.430952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914136+0000","last_change":"2024-04-02T13:20:20.900447+0000","last_active":"2024-04-02T13:20:25.914136+0000","last_peered":"2024-04-02T13:20:25.914136+0000","last_clean":"2024-04-02T13:20:25.914136+0000","last_became_active":"2024-04-02T13:20:20.900188+0000","last_became_peered":"2024-04-02T13:20:20.900188+0000","last_unstale":"2024-04-02T13:20:25.914136+0000","last_undegraded":"2024-04-02T13:20:25.914136+0000","last_fullsized":"2024-04-02T13:20:25.914136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:37.660148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912721+0000","last_change":"2024-04-02T13:20:24.928309+0000","last_active":"2024-04-02T13:20:25.912721+0000","last_peered":"2024-04-02T13:20:25.912721+0000","last_clean":"2024-04-02T13:20:25.912721+0000","last_became_active":"2024-04-02T13:20:24.927881+0000","last_became_peered":"2024-04-02T13:20:24.927881+0000","last_unstale":"2024-04-02T13:20:25.912721+0000","last_undegraded":"2024-04-02T13:20:25.912721+0000","last_fullsized":"2024-04-02T13:20:25.912721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:38.338080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430504+0000","last_change":"2024-04-02T13:20:20.895522+0000","last_active":"2024-04-02T13:20:26.430504+0000","last_peered":"2024-04-02T13:20:26.430504+0000","last_clean":"2024-04-02T13:20:26.430504+0000","last_became_active":"2024-04-02T13:20:20.895202+0000","last_became_peered":"2024-04-02T13:20:20.895202+0000","last_unstale":"2024-04-02T13:20:26.430504+0000","last_undegraded":"2024-04-02T13:20:26.430504+0000","last_fullsized":"2024-04-02T13:20:26.430504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430987+0000","last_change":"2024-04-02T13:20:19.895121+0000","last_active":"2024-04-02T13:20:26.430987+0000","last_peered":"2024-04-02T13:20:26.430987+0000","last_clean":"2024-04-02T13:20:26.430987+0000","last_became_active":"2024-04-02T13:20:19.894633+0000","last_became_peered":"2024-04-02T13:20:19.894633+0000","last_unstale":"2024-04-02T13:20:26.430987+0000","last_undegraded":"2024-04-02T13:20:26.430987+0000","last_fullsized":"2024-04-02T13:20:26.430987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:05:36.149061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913513+0000","last_change":"2024-04-02T13:20:24.941648+0000","last_active":"2024-04-02T13:20:25.913513+0000","last_peered":"2024-04-02T13:20:25.913513+0000","last_clean":"2024-04-02T13:20:25.913513+0000","last_became_active":"2024-04-02T13:20:24.941281+0000","last_became_peered":"2024-04-02T13:20:24.941281+0000","last_unstale":"2024-04-02T13:20:25.913513+0000","last_undegraded":"2024-04-02T13:20:25.913513+0000","last_fullsized":"2024-04-02T13:20:25.913513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:06.819681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910552+0000","last_change":"2024-04-02T13:20:20.895057+0000","last_active":"2024-04-02T13:20:25.910552+0000","last_peered":"2024-04-02T13:20:25.910552+0000","last_clean":"2024-04-02T13:20:25.910552+0000","last_became_active":"2024-04-02T13:20:20.894189+0000","last_became_peered":"2024-04-02T13:20:20.894189+0000","last_unstale":"2024-04-02T13:20:25.910552+0000","last_undegraded":"2024-04-02T13:20:25.910552+0000","last_fullsized":"2024-04-02T13:20:25.910552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:56:59.954023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432656+0000","last_change":"2024-04-02T13:20:19.898990+0000","last_active":"2024-04-02T13:20:26.432656+0000","last_peered":"2024-04-02T13:20:26.432656+0000","last_clean":"2024-04-02T13:20:26.432656+0000","last_became_active":"2024-04-02T13:20:19.898590+0000","last_became_peered":"2024-04-02T13:20:19.898590+0000","last_unstale":"2024-04-02T13:20:26.432656+0000","last_undegraded":"2024-04-02T13:20:26.432656+0000","last_fullsized":"2024-04-02T13:20:26.432656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912549+0000","last_change":"2024-04-02T13:20:24.943246+0000","last_active":"2024-04-02T13:20:25.912549+0000","last_peered":"2024-04-02T13:20:25.912549+0000","last_clean":"2024-04-02T13:20:25.912549+0000","last_became_active":"2024-04-02T13:20:24.942587+0000","last_became_peered":"2024-04-02T13:20:24.942587+0000","last_unstale":"2024-04-02T13:20:25.912549+0000","last_undegraded":"2024-04-02T13:20:25.912549+0000","last_fullsized":"2024-04-02T13:20:25.912549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:10.369290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910414+0000","last_change":"2024-04-02T13:20:20.896993+0000","last_active":"2024-04-02T13:20:25.910414+0000","last_peered":"2024-04-02T13:20:25.910414+0000","last_clean":"2024-04-02T13:20:25.910414+0000","last_became_active":"2024-04-02T13:20:20.896668+0000","last_became_peered":"2024-04-02T13:20:20.896668+0000","last_unstale":"2024-04-02T13:20:25.910414+0000","last_undegraded":"2024-04-02T13:20:25.910414+0000","last_fullsized":"2024-04-02T13:20:25.910414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430520+0000","last_change":"2024-04-02T13:20:19.892855+0000","last_active":"2024-04-02T13:20:26.430520+0000","last_peered":"2024-04-02T13:20:26.430520+0000","last_clean":"2024-04-02T13:20:26.430520+0000","last_became_active":"2024-04-02T13:20:19.892578+0000","last_became_peered":"2024-04-02T13:20:19.892578+0000","last_unstale":"2024-04-02T13:20:26.430520+0000","last_undegraded":"2024-04-02T13:20:26.430520+0000","last_fullsized":"2024-04-02T13:20:26.430520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:36.358876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432032+0000","last_change":"2024-04-02T13:20:24.975476+0000","last_active":"2024-04-02T13:20:26.432032+0000","last_peered":"2024-04-02T13:20:26.432032+0000","last_clean":"2024-04-02T13:20:26.432032+0000","last_became_active":"2024-04-02T13:20:24.974600+0000","last_became_peered":"2024-04-02T13:20:24.974600+0000","last_unstale":"2024-04-02T13:20:26.432032+0000","last_undegraded":"2024-04-02T13:20:26.432032+0000","last_fullsized":"2024-04-02T13:20:26.432032+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:35.465459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913314+0000","last_change":"2024-04-02T13:20:20.898012+0000","last_active":"2024-04-02T13:20:25.913314+0000","last_peered":"2024-04-02T13:20:25.913314+0000","last_clean":"2024-04-02T13:20:25.913314+0000","last_became_active":"2024-04-02T13:20:20.897732+0000","last_became_peered":"2024-04-02T13:20:20.897732+0000","last_unstale":"2024-04-02T13:20:25.913314+0000","last_undegraded":"2024-04-02T13:20:25.913314+0000","last_fullsized":"2024-04-02T13:20:25.913314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:55.219441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430649+0000","last_change":"2024-04-02T13:20:19.894100+0000","last_active":"2024-04-02T13:20:26.430649+0000","last_peered":"2024-04-02T13:20:26.430649+0000","last_clean":"2024-04-02T13:20:26.430649+0000","last_became_active":"2024-04-02T13:20:19.893628+0000","last_became_peered":"2024-04-02T13:20:19.893628+0000","last_unstale":"2024-04-02T13:20:26.430649+0000","last_undegraded":"2024-04-02T13:20:26.430649+0000","last_fullsized":"2024-04-02T13:20:26.430649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:05.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913916+0000","last_change":"2024-04-02T13:20:24.943183+0000","last_active":"2024-04-02T13:20:25.913916+0000","last_peered":"2024-04-02T13:20:25.913916+0000","last_clean":"2024-04-02T13:20:25.913916+0000","last_became_active":"2024-04-02T13:20:24.942594+0000","last_became_peered":"2024-04-02T13:20:24.942594+0000","last_unstale":"2024-04-02T13:20:25.913916+0000","last_undegraded":"2024-04-02T13:20:25.913916+0000","last_fullsized":"2024-04-02T13:20:25.913916+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:29.504244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431563+0000","last_change":"2024-04-02T13:20:20.898299+0000","last_active":"2024-04-02T13:20:26.431563+0000","last_peered":"2024-04-02T13:20:26.431563+0000","last_clean":"2024-04-02T13:20:26.431563+0000","last_became_active":"2024-04-02T13:20:20.898014+0000","last_became_peered":"2024-04-02T13:20:20.898014+0000","last_unstale":"2024-04-02T13:20:26.431563+0000","last_undegraded":"2024-04-02T13:20:26.431563+0000","last_fullsized":"2024-04-02T13:20:26.431563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:08.738556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431600+0000","last_change":"2024-04-02T13:20:19.895734+0000","last_active":"2024-04-02T13:20:26.431600+0000","last_peered":"2024-04-02T13:20:26.431600+0000","last_clean":"2024-04-02T13:20:26.431600+0000","last_became_active":"2024-04-02T13:20:19.895430+0000","last_became_peered":"2024-04-02T13:20:19.895430+0000","last_unstale":"2024-04-02T13:20:26.431600+0000","last_undegraded":"2024-04-02T13:20:26.431600+0000","last_fullsized":"2024-04-02T13:20:26.431600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912814+0000","last_change":"2024-04-02T13:20:24.972780+0000","last_active":"2024-04-02T13:20:25.912814+0000","last_peered":"2024-04-02T13:20:25.912814+0000","last_clean":"2024-04-02T13:20:25.912814+0000","last_became_active":"2024-04-02T13:20:24.972420+0000","last_became_peered":"2024-04-02T13:20:24.972420+0000","last_unstale":"2024-04-02T13:20:25.912814+0000","last_undegraded":"2024-04-02T13:20:25.912814+0000","last_fullsized":"2024-04-02T13:20:25.912814+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:28.449843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911321+0000","last_change":"2024-04-02T13:20:20.901896+0000","last_active":"2024-04-02T13:20:25.911321+0000","last_peered":"2024-04-02T13:20:25.911321+0000","last_clean":"2024-04-02T13:20:25.911321+0000","last_became_active":"2024-04-02T13:20:20.901593+0000","last_became_peered":"2024-04-02T13:20:20.901593+0000","last_unstale":"2024-04-02T13:20:25.911321+0000","last_undegraded":"2024-04-02T13:20:25.911321+0000","last_fullsized":"2024-04-02T13:20:25.911321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:24.415150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431790+0000","last_change":"2024-04-02T13:20:19.895273+0000","last_active":"2024-04-02T13:20:26.431790+0000","last_peered":"2024-04-02T13:20:26.431790+0000","last_clean":"2024-04-02T13:20:26.431790+0000","last_became_active":"2024-04-02T13:20:19.894962+0000","last_became_peered":"2024-04-02T13:20:19.894962+0000","last_unstale":"2024-04-02T13:20:26.431790+0000","last_undegraded":"2024-04-02T13:20:26.431790+0000","last_fullsized":"2024-04-02T13:20:26.431790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910167+0000","last_change":"2024-04-02T13:20:24.975854+0000","last_active":"2024-04-02T13:20:25.910167+0000","last_peered":"2024-04-02T13:20:25.910167+0000","last_clean":"2024-04-02T13:20:25.910167+0000","last_became_active":"2024-04-02T13:20:24.975476+0000","last_became_peered":"2024-04-02T13:20:24.975476+0000","last_unstale":"2024-04-02T13:20:25.910167+0000","last_undegraded":"2024-04-02T13:20:25.910167+0000","last_fullsized":"2024-04-02T13:20:25.910167+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:00.601346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432365+0000","last_change":"2024-04-02T13:20:20.898220+0000","last_active":"2024-04-02T13:20:26.432365+0000","last_peered":"2024-04-02T13:20:26.432365+0000","last_clean":"2024-04-02T13:20:26.432365+0000","last_became_active":"2024-04-02T13:20:20.897937+0000","last_became_peered":"2024-04-02T13:20:20.897937+0000","last_unstale":"2024-04-02T13:20:26.432365+0000","last_undegraded":"2024-04-02T13:20:26.432365+0000","last_fullsized":"2024-04-02T13:20:26.432365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431659+0000","last_change":"2024-04-02T13:20:19.891525+0000","last_active":"2024-04-02T13:20:26.431659+0000","last_peered":"2024-04-02T13:20:26.431659+0000","last_clean":"2024-04-02T13:20:26.431659+0000","last_became_active":"2024-04-02T13:20:19.891260+0000","last_became_peered":"2024-04-02T13:20:19.891260+0000","last_unstale":"2024-04-02T13:20:26.431659+0000","last_undegraded":"2024-04-02T13:20:26.431659+0000","last_fullsized":"2024-04-02T13:20:26.431659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:01.860021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431840+0000","last_change":"2024-04-02T13:20:24.970679+0000","last_active":"2024-04-02T13:20:26.431840+0000","last_peered":"2024-04-02T13:20:26.431840+0000","last_clean":"2024-04-02T13:20:26.431840+0000","last_became_active":"2024-04-02T13:20:24.970216+0000","last_became_peered":"2024-04-02T13:20:24.970216+0000","last_unstale":"2024-04-02T13:20:26.431840+0000","last_undegraded":"2024-04-02T13:20:26.431840+0000","last_fullsized":"2024-04-02T13:20:26.431840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:46.744270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914241+0000","last_change":"2024-04-02T13:20:20.901035+0000","last_active":"2024-04-02T13:20:25.914241+0000","last_peered":"2024-04-02T13:20:25.914241+0000","last_clean":"2024-04-02T13:20:25.914241+0000","last_became_active":"2024-04-02T13:20:20.900751+0000","last_became_peered":"2024-04-02T13:20:20.900751+0000","last_unstale":"2024-04-02T13:20:25.914241+0000","last_undegraded":"2024-04-02T13:20:25.914241+0000","last_fullsized":"2024-04-02T13:20:25.914241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:31.430767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432453+0000","last_change":"2024-04-02T13:20:19.897166+0000","last_active":"2024-04-02T13:20:26.432453+0000","last_peered":"2024-04-02T13:20:26.432453+0000","last_clean":"2024-04-02T13:20:26.432453+0000","last_became_active":"2024-04-02T13:20:19.896854+0000","last_became_peered":"2024-04-02T13:20:19.896854+0000","last_unstale":"2024-04-02T13:20:26.432453+0000","last_undegraded":"2024-04-02T13:20:26.432453+0000","last_fullsized":"2024-04-02T13:20:26.432453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:08.653626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430806+0000","last_change":"2024-04-02T13:20:24.937584+0000","last_active":"2024-04-02T13:20:26.430806+0000","last_peered":"2024-04-02T13:20:26.430806+0000","last_clean":"2024-04-02T13:20:26.430806+0000","last_became_active":"2024-04-02T13:20:24.937027+0000","last_became_peered":"2024-04-02T13:20:24.937027+0000","last_unstale":"2024-04-02T13:20:26.430806+0000","last_undegraded":"2024-04-02T13:20:26.430806+0000","last_fullsized":"2024-04-02T13:20:26.430806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:29.504244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914109+0000","last_change":"2024-04-02T13:20:20.888461+0000","last_active":"2024-04-02T13:20:25.914109+0000","last_peered":"2024-04-02T13:20:25.914109+0000","last_clean":"2024-04-02T13:20:25.914109+0000","last_became_active":"2024-04-02T13:20:20.887870+0000","last_became_peered":"2024-04-02T13:20:20.887870+0000","last_unstale":"2024-04-02T13:20:25.914109+0000","last_undegraded":"2024-04-02T13:20:25.914109+0000","last_fullsized":"2024-04-02T13:20:25.914109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:06.511257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913085+0000","last_change":"2024-04-02T13:20:19.895704+0000","last_active":"2024-04-02T13:20:25.913085+0000","last_peered":"2024-04-02T13:20:25.913085+0000","last_clean":"2024-04-02T13:20:25.913085+0000","last_became_active":"2024-04-02T13:20:19.895394+0000","last_became_peered":"2024-04-02T13:20:19.895394+0000","last_unstale":"2024-04-02T13:20:25.913085+0000","last_undegraded":"2024-04-02T13:20:25.913085+0000","last_fullsized":"2024-04-02T13:20:25.913085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432394+0000","last_change":"2024-04-02T13:20:24.935887+0000","last_active":"2024-04-02T13:20:26.432394+0000","last_peered":"2024-04-02T13:20:26.432394+0000","last_clean":"2024-04-02T13:20:26.432394+0000","last_became_active":"2024-04-02T13:20:24.935515+0000","last_became_peered":"2024-04-02T13:20:24.935515+0000","last_unstale":"2024-04-02T13:20:26.432394+0000","last_undegraded":"2024-04-02T13:20:26.432394+0000","last_fullsized":"2024-04-02T13:20:26.432394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:46.744270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431762+0000","last_change":"2024-04-02T13:20:20.899067+0000","last_active":"2024-04-02T13:20:26.431762+0000","last_peered":"2024-04-02T13:20:26.431762+0000","last_clean":"2024-04-02T13:20:26.431762+0000","last_became_active":"2024-04-02T13:20:20.898662+0000","last_became_peered":"2024-04-02T13:20:20.898662+0000","last_unstale":"2024-04-02T13:20:26.431762+0000","last_undegraded":"2024-04-02T13:20:26.431762+0000","last_fullsized":"2024-04-02T13:20:26.431762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431752+0000","last_change":"2024-04-02T13:20:19.896149+0000","last_active":"2024-04-02T13:20:26.431752+0000","last_peered":"2024-04-02T13:20:26.431752+0000","last_clean":"2024-04-02T13:20:26.431752+0000","last_became_active":"2024-04-02T13:20:19.895801+0000","last_became_peered":"2024-04-02T13:20:19.895801+0000","last_unstale":"2024-04-02T13:20:26.431752+0000","last_undegraded":"2024-04-02T13:20:26.431752+0000","last_fullsized":"2024-04-02T13:20:26.431752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432170+0000","last_change":"2024-04-02T13:20:24.933627+0000","last_active":"2024-04-02T13:20:26.432170+0000","last_peered":"2024-04-02T13:20:26.432170+0000","last_clean":"2024-04-02T13:20:26.432170+0000","last_became_active":"2024-04-02T13:20:24.932519+0000","last_became_peered":"2024-04-02T13:20:24.932519+0000","last_unstale":"2024-04-02T13:20:26.432170+0000","last_undegraded":"2024-04-02T13:20:26.432170+0000","last_fullsized":"2024-04-02T13:20:26.432170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:41.694841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430241+0000","last_change":"2024-04-02T13:20:20.899704+0000","last_active":"2024-04-02T13:20:26.430241+0000","last_peered":"2024-04-02T13:20:26.430241+0000","last_clean":"2024-04-02T13:20:26.430241+0000","last_became_active":"2024-04-02T13:20:20.894279+0000","last_became_peered":"2024-04-02T13:20:20.894279+0000","last_unstale":"2024-04-02T13:20:26.430241+0000","last_undegraded":"2024-04-02T13:20:26.430241+0000","last_fullsized":"2024-04-02T13:20:26.430241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:48.848558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430741+0000","last_change":"2024-04-02T13:20:19.893903+0000","last_active":"2024-04-02T13:20:26.430741+0000","last_peered":"2024-04-02T13:20:26.430741+0000","last_clean":"2024-04-02T13:20:26.430741+0000","last_became_active":"2024-04-02T13:20:19.893251+0000","last_became_peered":"2024-04-02T13:20:19.893251+0000","last_unstale":"2024-04-02T13:20:26.430741+0000","last_undegraded":"2024-04-02T13:20:26.430741+0000","last_fullsized":"2024-04-02T13:20:26.430741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:36.358876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431391+0000","last_change":"2024-04-02T13:20:24.971364+0000","last_active":"2024-04-02T13:20:26.431391+0000","last_peered":"2024-04-02T13:20:26.431391+0000","last_clean":"2024-04-02T13:20:26.431391+0000","last_became_active":"2024-04-02T13:20:24.970950+0000","last_became_peered":"2024-04-02T13:20:24.970950+0000","last_unstale":"2024-04-02T13:20:26.431391+0000","last_undegraded":"2024-04-02T13:20:26.431391+0000","last_fullsized":"2024-04-02T13:20:26.431391+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:41.694841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910298+0000","last_change":"2024-04-02T13:20:20.906876+0000","last_active":"2024-04-02T13:20:25.910298+0000","last_peered":"2024-04-02T13:20:25.910298+0000","last_clean":"2024-04-02T13:20:25.910298+0000","last_became_active":"2024-04-02T13:20:20.906603+0000","last_became_peered":"2024-04-02T13:20:20.906603+0000","last_unstale":"2024-04-02T13:20:25.910298+0000","last_undegraded":"2024-04-02T13:20:25.910298+0000","last_fullsized":"2024-04-02T13:20:25.910298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:39.096965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913068+0000","last_change":"2024-04-02T13:20:19.889006+0000","last_active":"2024-04-02T13:20:25.913068+0000","last_peered":"2024-04-02T13:20:25.913068+0000","last_clean":"2024-04-02T13:20:25.913068+0000","last_became_active":"2024-04-02T13:20:19.888619+0000","last_became_peered":"2024-04-02T13:20:19.888619+0000","last_unstale":"2024-04-02T13:20:25.913068+0000","last_undegraded":"2024-04-02T13:20:25.913068+0000","last_fullsized":"2024-04-02T13:20:25.913068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430765+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:20:26.430765+0000","last_peered":"2024-04-02T13:20:26.430765+0000","last_clean":"2024-04-02T13:20:26.430765+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:20:26.430765+0000","last_undegraded":"2024-04-02T13:20:26.430765+0000","last_fullsized":"2024-04-02T13:20:26.430765+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430638+0000","last_change":"2024-04-02T13:20:24.972665+0000","last_active":"2024-04-02T13:20:26.430638+0000","last_peered":"2024-04-02T13:20:26.430638+0000","last_clean":"2024-04-02T13:20:26.430638+0000","last_became_active":"2024-04-02T13:20:24.972285+0000","last_became_peered":"2024-04-02T13:20:24.972285+0000","last_unstale":"2024-04-02T13:20:26.430638+0000","last_undegraded":"2024-04-02T13:20:26.430638+0000","last_fullsized":"2024-04-02T13:20:26.430638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:43.131658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431391+0000","last_change":"2024-04-02T13:20:20.901120+0000","last_active":"2024-04-02T13:20:26.431391+0000","last_peered":"2024-04-02T13:20:26.431391+0000","last_clean":"2024-04-02T13:20:26.431391+0000","last_became_active":"2024-04-02T13:20:20.900433+0000","last_became_peered":"2024-04-02T13:20:20.900433+0000","last_unstale":"2024-04-02T13:20:26.431391+0000","last_undegraded":"2024-04-02T13:20:26.431391+0000","last_fullsized":"2024-04-02T13:20:26.431391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:42.709577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431230+0000","last_change":"2024-04-02T13:20:19.893843+0000","last_active":"2024-04-02T13:20:26.431230+0000","last_peered":"2024-04-02T13:20:26.431230+0000","last_clean":"2024-04-02T13:20:26.431230+0000","last_became_active":"2024-04-02T13:20:19.893524+0000","last_became_peered":"2024-04-02T13:20:19.893524+0000","last_unstale":"2024-04-02T13:20:26.431230+0000","last_undegraded":"2024-04-02T13:20:26.431230+0000","last_fullsized":"2024-04-02T13:20:26.431230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:56.359644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430631+0000","last_change":"2024-04-02T13:20:24.971102+0000","last_active":"2024-04-02T13:20:26.430631+0000","last_peered":"2024-04-02T13:20:26.430631+0000","last_clean":"2024-04-02T13:20:26.430631+0000","last_became_active":"2024-04-02T13:20:24.970132+0000","last_became_peered":"2024-04-02T13:20:24.970132+0000","last_unstale":"2024-04-02T13:20:26.430631+0000","last_undegraded":"2024-04-02T13:20:26.430631+0000","last_fullsized":"2024-04-02T13:20:26.430631+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:28.449843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912995+0000","last_change":"2024-04-02T13:20:20.908405+0000","last_active":"2024-04-02T13:20:25.912995+0000","last_peered":"2024-04-02T13:20:25.912995+0000","last_clean":"2024-04-02T13:20:25.912995+0000","last_became_active":"2024-04-02T13:20:20.907986+0000","last_became_peered":"2024-04-02T13:20:20.907986+0000","last_unstale":"2024-04-02T13:20:25.912995+0000","last_undegraded":"2024-04-02T13:20:25.912995+0000","last_fullsized":"2024-04-02T13:20:25.912995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:24.415150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432656+0000","last_change":"2024-04-02T13:20:19.893884+0000","last_active":"2024-04-02T13:20:26.432656+0000","last_peered":"2024-04-02T13:20:26.432656+0000","last_clean":"2024-04-02T13:20:26.432656+0000","last_became_active":"2024-04-02T13:20:19.893605+0000","last_became_peered":"2024-04-02T13:20:19.893605+0000","last_unstale":"2024-04-02T13:20:26.432656+0000","last_undegraded":"2024-04-02T13:20:26.432656+0000","last_fullsized":"2024-04-02T13:20:26.432656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:05.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432026+0000","last_change":"2024-04-02T13:20:24.931897+0000","last_active":"2024-04-02T13:20:26.432026+0000","last_peered":"2024-04-02T13:20:26.432026+0000","last_clean":"2024-04-02T13:20:26.432026+0000","last_became_active":"2024-04-02T13:20:24.931512+0000","last_became_peered":"2024-04-02T13:20:24.931512+0000","last_unstale":"2024-04-02T13:20:26.432026+0000","last_undegraded":"2024-04-02T13:20:26.432026+0000","last_fullsized":"2024-04-02T13:20:26.432026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:26:20.333590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431656+0000","last_change":"2024-04-02T13:20:20.890017+0000","last_active":"2024-04-02T13:20:26.431656+0000","last_peered":"2024-04-02T13:20:26.431656+0000","last_clean":"2024-04-02T13:20:26.431656+0000","last_became_active":"2024-04-02T13:20:20.889739+0000","last_became_peered":"2024-04-02T13:20:20.889739+0000","last_unstale":"2024-04-02T13:20:26.431656+0000","last_undegraded":"2024-04-02T13:20:26.431656+0000","last_fullsized":"2024-04-02T13:20:26.431656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:25.469551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431013+0000","last_change":"2024-04-02T13:20:19.899771+0000","last_active":"2024-04-02T13:20:26.431013+0000","last_peered":"2024-04-02T13:20:26.431013+0000","last_clean":"2024-04-02T13:20:26.431013+0000","last_became_active":"2024-04-02T13:20:19.899415+0000","last_became_peered":"2024-04-02T13:20:19.899415+0000","last_unstale":"2024-04-02T13:20:26.431013+0000","last_undegraded":"2024-04-02T13:20:26.431013+0000","last_fullsized":"2024-04-02T13:20:26.431013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913513+0000","last_change":"2024-04-02T13:20:24.971830+0000","last_active":"2024-04-02T13:20:25.913513+0000","last_peered":"2024-04-02T13:20:25.913513+0000","last_clean":"2024-04-02T13:20:25.913513+0000","last_became_active":"2024-04-02T13:20:24.971442+0000","last_became_peered":"2024-04-02T13:20:24.971442+0000","last_unstale":"2024-04-02T13:20:25.913513+0000","last_undegraded":"2024-04-02T13:20:25.913513+0000","last_fullsized":"2024-04-02T13:20:25.913513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:00.601346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431356+0000","last_change":"2024-04-02T13:20:20.887060+0000","last_active":"2024-04-02T13:20:26.431356+0000","last_peered":"2024-04-02T13:20:26.431356+0000","last_clean":"2024-04-02T13:20:26.431356+0000","last_became_active":"2024-04-02T13:20:20.886492+0000","last_became_peered":"2024-04-02T13:20:20.886492+0000","last_unstale":"2024-04-02T13:20:26.431356+0000","last_undegraded":"2024-04-02T13:20:26.431356+0000","last_fullsized":"2024-04-02T13:20:26.431356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:31.430767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913624+0000","last_change":"2024-04-02T13:20:19.897647+0000","last_active":"2024-04-02T13:20:25.913624+0000","last_peered":"2024-04-02T13:20:25.913624+0000","last_clean":"2024-04-02T13:20:25.913624+0000","last_became_active":"2024-04-02T13:20:19.897244+0000","last_became_peered":"2024-04-02T13:20:19.897244+0000","last_unstale":"2024-04-02T13:20:25.913624+0000","last_undegraded":"2024-04-02T13:20:25.913624+0000","last_fullsized":"2024-04-02T13:20:25.913624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:28:53.732425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912616+0000","last_change":"2024-04-02T13:20:24.932365+0000","last_active":"2024-04-02T13:20:25.912616+0000","last_peered":"2024-04-02T13:20:25.912616+0000","last_clean":"2024-04-02T13:20:25.912616+0000","last_became_active":"2024-04-02T13:20:24.931793+0000","last_became_peered":"2024-04-02T13:20:24.931793+0000","last_unstale":"2024-04-02T13:20:25.912616+0000","last_undegraded":"2024-04-02T13:20:25.912616+0000","last_fullsized":"2024-04-02T13:20:25.912616+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:30.566782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431185+0000","last_change":"2024-04-02T13:20:20.887410+0000","last_active":"2024-04-02T13:20:26.431185+0000","last_peered":"2024-04-02T13:20:26.431185+0000","last_clean":"2024-04-02T13:20:26.431185+0000","last_became_active":"2024-04-02T13:20:20.886414+0000","last_became_peered":"2024-04-02T13:20:20.886414+0000","last_unstale":"2024-04-02T13:20:26.431185+0000","last_undegraded":"2024-04-02T13:20:26.431185+0000","last_fullsized":"2024-04-02T13:20:26.431185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:24.415150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430413+0000","last_change":"2024-04-02T13:20:19.896189+0000","last_active":"2024-04-02T13:20:26.430413+0000","last_peered":"2024-04-02T13:20:26.430413+0000","last_clean":"2024-04-02T13:20:26.430413+0000","last_became_active":"2024-04-02T13:20:19.895844+0000","last_became_peered":"2024-04-02T13:20:19.895844+0000","last_unstale":"2024-04-02T13:20:26.430413+0000","last_undegraded":"2024-04-02T13:20:26.430413+0000","last_fullsized":"2024-04-02T13:20:26.430413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431242+0000","last_change":"2024-04-02T13:20:24.928642+0000","last_active":"2024-04-02T13:20:26.431242+0000","last_peered":"2024-04-02T13:20:26.431242+0000","last_clean":"2024-04-02T13:20:26.431242+0000","last_became_active":"2024-04-02T13:20:24.928080+0000","last_became_peered":"2024-04-02T13:20:24.928080+0000","last_unstale":"2024-04-02T13:20:26.431242+0000","last_undegraded":"2024-04-02T13:20:26.431242+0000","last_fullsized":"2024-04-02T13:20:26.431242+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:35.465459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430479+0000","last_change":"2024-04-02T13:20:20.898225+0000","last_active":"2024-04-02T13:20:26.430479+0000","last_peered":"2024-04-02T13:20:26.430479+0000","last_clean":"2024-04-02T13:20:26.430479+0000","last_became_active":"2024-04-02T13:20:20.897962+0000","last_became_peered":"2024-04-02T13:20:20.897962+0000","last_unstale":"2024-04-02T13:20:26.430479+0000","last_undegraded":"2024-04-02T13:20:26.430479+0000","last_fullsized":"2024-04-02T13:20:26.430479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430618+0000","last_change":"2024-04-02T13:20:19.893578+0000","last_active":"2024-04-02T13:20:26.430618+0000","last_peered":"2024-04-02T13:20:26.430618+0000","last_clean":"2024-04-02T13:20:26.430618+0000","last_became_active":"2024-04-02T13:20:19.893301+0000","last_became_peered":"2024-04-02T13:20:19.893301+0000","last_unstale":"2024-04-02T13:20:26.430618+0000","last_undegraded":"2024-04-02T13:20:26.430618+0000","last_fullsized":"2024-04-02T13:20:26.430618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:54.808816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911170+0000","last_change":"2024-04-02T13:20:24.971783+0000","last_active":"2024-04-02T13:20:25.911170+0000","last_peered":"2024-04-02T13:20:25.911170+0000","last_clean":"2024-04-02T13:20:25.911170+0000","last_became_active":"2024-04-02T13:20:24.971383+0000","last_became_peered":"2024-04-02T13:20:24.971383+0000","last_unstale":"2024-04-02T13:20:25.911170+0000","last_undegraded":"2024-04-02T13:20:25.911170+0000","last_fullsized":"2024-04-02T13:20:25.911170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:01.105712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430867+0000","last_change":"2024-04-02T13:20:20.889512+0000","last_active":"2024-04-02T13:20:26.430867+0000","last_peered":"2024-04-02T13:20:26.430867+0000","last_clean":"2024-04-02T13:20:26.430867+0000","last_became_active":"2024-04-02T13:20:20.886370+0000","last_became_peered":"2024-04-02T13:20:20.886370+0000","last_unstale":"2024-04-02T13:20:26.430867+0000","last_undegraded":"2024-04-02T13:20:26.430867+0000","last_fullsized":"2024-04-02T13:20:26.430867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:15.162527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912998+0000","last_change":"2024-04-02T13:20:19.895006+0000","last_active":"2024-04-02T13:20:25.912998+0000","last_peered":"2024-04-02T13:20:25.912998+0000","last_clean":"2024-04-02T13:20:25.912998+0000","last_became_active":"2024-04-02T13:20:19.894684+0000","last_became_peered":"2024-04-02T13:20:19.894684+0000","last_unstale":"2024-04-02T13:20:25.912998+0000","last_undegraded":"2024-04-02T13:20:25.912998+0000","last_fullsized":"2024-04-02T13:20:25.912998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911073+0000","last_change":"2024-04-02T13:20:24.972147+0000","last_active":"2024-04-02T13:20:25.911073+0000","last_peered":"2024-04-02T13:20:25.911073+0000","last_clean":"2024-04-02T13:20:25.911073+0000","last_became_active":"2024-04-02T13:20:24.971778+0000","last_became_peered":"2024-04-02T13:20:24.971778+0000","last_unstale":"2024-04-02T13:20:25.911073+0000","last_undegraded":"2024-04-02T13:20:25.911073+0000","last_fullsized":"2024-04-02T13:20:25.911073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:22:03.916011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430986+0000","last_change":"2024-04-02T13:20:20.906617+0000","last_active":"2024-04-02T13:20:26.430986+0000","last_peered":"2024-04-02T13:20:26.430986+0000","last_clean":"2024-04-02T13:20:26.430986+0000","last_became_active":"2024-04-02T13:20:20.905272+0000","last_became_peered":"2024-04-02T13:20:20.905272+0000","last_unstale":"2024-04-02T13:20:26.430986+0000","last_undegraded":"2024-04-02T13:20:26.430986+0000","last_fullsized":"2024-04-02T13:20:26.430986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432098+0000","last_change":"2024-04-02T13:20:19.894435+0000","last_active":"2024-04-02T13:20:26.432098+0000","last_peered":"2024-04-02T13:20:26.432098+0000","last_clean":"2024-04-02T13:20:26.432098+0000","last_became_active":"2024-04-02T13:20:19.894163+0000","last_became_peered":"2024-04-02T13:20:19.894163+0000","last_unstale":"2024-04-02T13:20:26.432098+0000","last_undegraded":"2024-04-02T13:20:26.432098+0000","last_fullsized":"2024-04-02T13:20:26.432098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:44:36.097312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.914207+0000","last_change":"2024-04-02T13:20:24.970827+0000","last_active":"2024-04-02T13:20:25.914207+0000","last_peered":"2024-04-02T13:20:25.914207+0000","last_clean":"2024-04-02T13:20:25.914207+0000","last_became_active":"2024-04-02T13:20:24.970384+0000","last_became_peered":"2024-04-02T13:20:24.970384+0000","last_unstale":"2024-04-02T13:20:25.914207+0000","last_undegraded":"2024-04-02T13:20:25.914207+0000","last_fullsized":"2024-04-02T13:20:25.914207+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:20:58.050570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431335+0000","last_change":"2024-04-02T13:20:20.899063+0000","last_active":"2024-04-02T13:20:26.431335+0000","last_peered":"2024-04-02T13:20:26.431335+0000","last_clean":"2024-04-02T13:20:26.431335+0000","last_became_active":"2024-04-02T13:20:20.898735+0000","last_became_peered":"2024-04-02T13:20:20.898735+0000","last_unstale":"2024-04-02T13:20:26.431335+0000","last_undegraded":"2024-04-02T13:20:26.431335+0000","last_fullsized":"2024-04-02T13:20:26.431335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:30:37.660148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:30.433043+0000","last_change":"2024-04-02T13:20:19.898739+0000","last_active":"2024-04-02T13:20:30.433043+0000","last_peered":"2024-04-02T13:20:30.433043+0000","last_clean":"2024-04-02T13:20:30.433043+0000","last_became_active":"2024-04-02T13:20:19.898524+0000","last_became_peered":"2024-04-02T13:20:19.898524+0000","last_unstale":"2024-04-02T13:20:30.433043+0000","last_undegraded":"2024-04-02T13:20:30.433043+0000","last_fullsized":"2024-04-02T13:20:30.433043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432653+0000","last_change":"2024-04-02T13:20:24.941673+0000","last_active":"2024-04-02T13:20:26.432653+0000","last_peered":"2024-04-02T13:20:26.432653+0000","last_clean":"2024-04-02T13:20:26.432653+0000","last_became_active":"2024-04-02T13:20:24.941096+0000","last_became_peered":"2024-04-02T13:20:24.941096+0000","last_unstale":"2024-04-02T13:20:26.432653+0000","last_undegraded":"2024-04-02T13:20:26.432653+0000","last_fullsized":"2024-04-02T13:20:26.432653+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:26:20.333590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913928+0000","last_change":"2024-04-02T13:20:20.900373+0000","last_active":"2024-04-02T13:20:25.913928+0000","last_peered":"2024-04-02T13:20:25.913928+0000","last_clean":"2024-04-02T13:20:25.913928+0000","last_became_active":"2024-04-02T13:20:20.900084+0000","last_became_peered":"2024-04-02T13:20:20.900084+0000","last_unstale":"2024-04-02T13:20:25.913928+0000","last_undegraded":"2024-04-02T13:20:25.913928+0000","last_fullsized":"2024-04-02T13:20:25.913928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:09.661052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432591+0000","last_change":"2024-04-02T13:20:19.896178+0000","last_active":"2024-04-02T13:20:26.432591+0000","last_peered":"2024-04-02T13:20:26.432591+0000","last_clean":"2024-04-02T13:20:26.432591+0000","last_became_active":"2024-04-02T13:20:19.895887+0000","last_became_peered":"2024-04-02T13:20:19.895887+0000","last_unstale":"2024-04-02T13:20:26.432591+0000","last_undegraded":"2024-04-02T13:20:26.432591+0000","last_fullsized":"2024-04-02T13:20:26.432591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:00.261412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912814+0000","last_change":"2024-04-02T13:20:24.932941+0000","last_active":"2024-04-02T13:20:25.912814+0000","last_peered":"2024-04-02T13:20:25.912814+0000","last_clean":"2024-04-02T13:20:25.912814+0000","last_became_active":"2024-04-02T13:20:24.932319+0000","last_became_peered":"2024-04-02T13:20:24.932319+0000","last_unstale":"2024-04-02T13:20:25.912814+0000","last_undegraded":"2024-04-02T13:20:25.912814+0000","last_fullsized":"2024-04-02T13:20:25.912814+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:57:03.988715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431409+0000","last_change":"2024-04-02T13:20:20.902208+0000","last_active":"2024-04-02T13:20:26.431409+0000","last_peered":"2024-04-02T13:20:26.431409+0000","last_clean":"2024-04-02T13:20:26.431409+0000","last_became_active":"2024-04-02T13:20:20.901921+0000","last_became_peered":"2024-04-02T13:20:20.901921+0000","last_unstale":"2024-04-02T13:20:26.431409+0000","last_undegraded":"2024-04-02T13:20:26.431409+0000","last_fullsized":"2024-04-02T13:20:26.431409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913327+0000","last_change":"2024-04-02T13:20:19.893917+0000","last_active":"2024-04-02T13:20:25.913327+0000","last_peered":"2024-04-02T13:20:25.913327+0000","last_clean":"2024-04-02T13:20:25.913327+0000","last_became_active":"2024-04-02T13:20:19.893652+0000","last_became_peered":"2024-04-02T13:20:19.893652+0000","last_unstale":"2024-04-02T13:20:25.913327+0000","last_undegraded":"2024-04-02T13:20:25.913327+0000","last_fullsized":"2024-04-02T13:20:25.913327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:05.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910393+0000","last_change":"2024-04-02T13:20:24.928526+0000","last_active":"2024-04-02T13:20:25.910393+0000","last_peered":"2024-04-02T13:20:25.910393+0000","last_clean":"2024-04-02T13:20:25.910393+0000","last_became_active":"2024-04-02T13:20:24.928135+0000","last_became_peered":"2024-04-02T13:20:24.928135+0000","last_unstale":"2024-04-02T13:20:25.910393+0000","last_undegraded":"2024-04-02T13:20:25.910393+0000","last_fullsized":"2024-04-02T13:20:25.910393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:18.102976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913243+0000","last_change":"2024-04-02T13:20:20.904387+0000","last_active":"2024-04-02T13:20:25.913243+0000","last_peered":"2024-04-02T13:20:25.913243+0000","last_clean":"2024-04-02T13:20:25.913243+0000","last_became_active":"2024-04-02T13:20:20.903444+0000","last_became_peered":"2024-04-02T13:20:20.903444+0000","last_unstale":"2024-04-02T13:20:25.913243+0000","last_undegraded":"2024-04-02T13:20:25.913243+0000","last_fullsized":"2024-04-02T13:20:25.913243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:25.469551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431417+0000","last_change":"2024-04-02T13:20:19.894102+0000","last_active":"2024-04-02T13:20:26.431417+0000","last_peered":"2024-04-02T13:20:26.431417+0000","last_clean":"2024-04-02T13:20:26.431417+0000","last_became_active":"2024-04-02T13:20:19.893813+0000","last_became_peered":"2024-04-02T13:20:19.893813+0000","last_unstale":"2024-04-02T13:20:26.431417+0000","last_undegraded":"2024-04-02T13:20:26.431417+0000","last_fullsized":"2024-04-02T13:20:26.431417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:56.359644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910302+0000","last_change":"2024-04-02T13:20:24.970905+0000","last_active":"2024-04-02T13:20:25.910302+0000","last_peered":"2024-04-02T13:20:25.910302+0000","last_clean":"2024-04-02T13:20:25.910302+0000","last_became_active":"2024-04-02T13:20:24.970252+0000","last_became_peered":"2024-04-02T13:20:24.970252+0000","last_unstale":"2024-04-02T13:20:25.910302+0000","last_undegraded":"2024-04-02T13:20:25.910302+0000","last_fullsized":"2024-04-02T13:20:25.910302+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:36:46.624236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431088+0000","last_change":"2024-04-02T13:20:24.973949+0000","last_active":"2024-04-02T13:20:26.431088+0000","last_peered":"2024-04-02T13:20:26.431088+0000","last_clean":"2024-04-02T13:20:26.431088+0000","last_became_active":"2024-04-02T13:20:24.973471+0000","last_became_peered":"2024-04-02T13:20:24.973471+0000","last_unstale":"2024-04-02T13:20:26.431088+0000","last_undegraded":"2024-04-02T13:20:26.431088+0000","last_fullsized":"2024-04-02T13:20:26.431088+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:29.504244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430330+0000","last_change":"2024-04-02T13:20:20.892943+0000","last_active":"2024-04-02T13:20:26.430330+0000","last_peered":"2024-04-02T13:20:26.430330+0000","last_clean":"2024-04-02T13:20:26.430330+0000","last_became_active":"2024-04-02T13:20:20.892649+0000","last_became_peered":"2024-04-02T13:20:20.892649+0000","last_unstale":"2024-04-02T13:20:26.430330+0000","last_undegraded":"2024-04-02T13:20:26.430330+0000","last_fullsized":"2024-04-02T13:20:26.430330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913844+0000","last_change":"2024-04-02T13:20:19.897517+0000","last_active":"2024-04-02T13:20:25.913844+0000","last_peered":"2024-04-02T13:20:25.913844+0000","last_clean":"2024-04-02T13:20:25.913844+0000","last_became_active":"2024-04-02T13:20:19.897158+0000","last_became_peered":"2024-04-02T13:20:19.897158+0000","last_unstale":"2024-04-02T13:20:25.913844+0000","last_undegraded":"2024-04-02T13:20:25.913844+0000","last_fullsized":"2024-04-02T13:20:25.913844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:38:54.212016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431290+0000","last_change":"2024-04-02T13:20:24.932523+0000","last_active":"2024-04-02T13:20:26.431290+0000","last_peered":"2024-04-02T13:20:26.431290+0000","last_clean":"2024-04-02T13:20:26.431290+0000","last_became_active":"2024-04-02T13:20:24.932122+0000","last_became_peered":"2024-04-02T13:20:24.932122+0000","last_unstale":"2024-04-02T13:20:26.431290+0000","last_undegraded":"2024-04-02T13:20:26.431290+0000","last_fullsized":"2024-04-02T13:20:26.431290+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:06.819681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910293+0000","last_change":"2024-04-02T13:20:20.900650+0000","last_active":"2024-04-02T13:20:25.910293+0000","last_peered":"2024-04-02T13:20:25.910293+0000","last_clean":"2024-04-02T13:20:25.910293+0000","last_became_active":"2024-04-02T13:20:20.900371+0000","last_became_peered":"2024-04-02T13:20:20.900371+0000","last_unstale":"2024-04-02T13:20:25.910293+0000","last_undegraded":"2024-04-02T13:20:25.910293+0000","last_fullsized":"2024-04-02T13:20:25.910293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:31.960019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910494+0000","last_change":"2024-04-02T13:20:19.894065+0000","last_active":"2024-04-02T13:20:25.910494+0000","last_peered":"2024-04-02T13:20:25.910494+0000","last_clean":"2024-04-02T13:20:25.910494+0000","last_became_active":"2024-04-02T13:20:19.893749+0000","last_became_peered":"2024-04-02T13:20:19.893749+0000","last_unstale":"2024-04-02T13:20:25.910494+0000","last_undegraded":"2024-04-02T13:20:25.910494+0000","last_fullsized":"2024-04-02T13:20:25.910494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:42:03.725417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430909+0000","last_change":"2024-04-02T13:20:19.892884+0000","last_active":"2024-04-02T13:20:26.430909+0000","last_peered":"2024-04-02T13:20:26.430909+0000","last_clean":"2024-04-02T13:20:26.430909+0000","last_became_active":"2024-04-02T13:20:19.892641+0000","last_became_peered":"2024-04-02T13:20:19.892641+0000","last_unstale":"2024-04-02T13:20:26.430909+0000","last_undegraded":"2024-04-02T13:20:26.430909+0000","last_fullsized":"2024-04-02T13:20:26.430909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:58.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432078+0000","last_change":"2024-04-02T13:20:20.900588+0000","last_active":"2024-04-02T13:20:26.432078+0000","last_peered":"2024-04-02T13:20:26.432078+0000","last_clean":"2024-04-02T13:20:26.432078+0000","last_became_active":"2024-04-02T13:20:20.900299+0000","last_became_peered":"2024-04-02T13:20:20.900299+0000","last_unstale":"2024-04-02T13:20:26.432078+0000","last_undegraded":"2024-04-02T13:20:26.432078+0000","last_fullsized":"2024-04-02T13:20:26.432078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431129+0000","last_change":"2024-04-02T13:20:24.937767+0000","last_active":"2024-04-02T13:20:26.431129+0000","last_peered":"2024-04-02T13:20:26.431129+0000","last_clean":"2024-04-02T13:20:26.431129+0000","last_became_active":"2024-04-02T13:20:24.937371+0000","last_became_peered":"2024-04-02T13:20:24.937371+0000","last_unstale":"2024-04-02T13:20:26.431129+0000","last_undegraded":"2024-04-02T13:20:26.431129+0000","last_fullsized":"2024-04-02T13:20:26.431129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:52.883251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910649+0000","last_change":"2024-04-02T13:20:19.895568+0000","last_active":"2024-04-02T13:20:25.910649+0000","last_peered":"2024-04-02T13:20:25.910649+0000","last_clean":"2024-04-02T13:20:25.910649+0000","last_became_active":"2024-04-02T13:20:19.895240+0000","last_became_peered":"2024-04-02T13:20:19.895240+0000","last_unstale":"2024-04-02T13:20:25.910649+0000","last_undegraded":"2024-04-02T13:20:25.910649+0000","last_fullsized":"2024-04-02T13:20:25.910649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:18.857266+0000","last_clean_scrub_stamp":"2024-04-02T13:20:18.857266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:56.359644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912900+0000","last_change":"2024-04-02T13:20:20.894414+0000","last_active":"2024-04-02T13:20:25.912900+0000","last_peered":"2024-04-02T13:20:25.912900+0000","last_clean":"2024-04-02T13:20:25.912900+0000","last_became_active":"2024-04-02T13:20:20.894129+0000","last_became_peered":"2024-04-02T13:20:20.894129+0000","last_unstale":"2024-04-02T13:20:25.912900+0000","last_undegraded":"2024-04-02T13:20:25.912900+0000","last_fullsized":"2024-04-02T13:20:25.912900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:01.268837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911029+0000","last_change":"2024-04-02T13:20:24.932278+0000","last_active":"2024-04-02T13:20:25.911029+0000","last_peered":"2024-04-02T13:20:25.911029+0000","last_clean":"2024-04-02T13:20:25.911029+0000","last_became_active":"2024-04-02T13:20:24.931901+0000","last_became_peered":"2024-04-02T13:20:24.931901+0000","last_unstale":"2024-04-02T13:20:25.911029+0000","last_undegraded":"2024-04-02T13:20:25.911029+0000","last_fullsized":"2024-04-02T13:20:25.911029+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:23.899384+0000","last_clean_scrub_stamp":"2024-04-02T13:20:23.899384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:47.846911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430361+0000","last_change":"2024-04-02T13:20:20.898502+0000","last_active":"2024-04-02T13:20:26.430361+0000","last_peered":"2024-04-02T13:20:26.430361+0000","last_clean":"2024-04-02T13:20:26.430361+0000","last_became_active":"2024-04-02T13:20:20.898208+0000","last_became_peered":"2024-04-02T13:20:20.898208+0000","last_unstale":"2024-04-02T13:20:26.430361+0000","last_undegraded":"2024-04-02T13:20:26.430361+0000","last_fullsized":"2024-04-02T13:20:26.430361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431659+0000","last_change":"2024-04-02T13:20:20.905276+0000","last_active":"2024-04-02T13:20:26.431659+0000","last_peered":"2024-04-02T13:20:26.431659+0000","last_clean":"2024-04-02T13:20:26.431659+0000","last_became_active":"2024-04-02T13:20:20.905010+0000","last_became_peered":"2024-04-02T13:20:20.905010+0000","last_unstale":"2024-04-02T13:20:26.431659+0000","last_undegraded":"2024-04-02T13:20:26.431659+0000","last_fullsized":"2024-04-02T13:20:26.431659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:09.661052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912909+0000","last_change":"2024-04-02T13:20:20.908237+0000","last_active":"2024-04-02T13:20:25.912909+0000","last_peered":"2024-04-02T13:20:25.912909+0000","last_clean":"2024-04-02T13:20:25.912909+0000","last_became_active":"2024-04-02T13:20:20.907967+0000","last_became_peered":"2024-04-02T13:20:20.907967+0000","last_unstale":"2024-04-02T13:20:25.912909+0000","last_undegraded":"2024-04-02T13:20:25.912909+0000","last_fullsized":"2024-04-02T13:20:25.912909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:02.784989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431982+0000","last_change":"2024-04-02T13:20:20.901918+0000","last_active":"2024-04-02T13:20:26.431982+0000","last_peered":"2024-04-02T13:20:26.431982+0000","last_clean":"2024-04-02T13:20:26.431982+0000","last_became_active":"2024-04-02T13:20:20.901628+0000","last_became_peered":"2024-04-02T13:20:20.901628+0000","last_unstale":"2024-04-02T13:20:26.431982+0000","last_undegraded":"2024-04-02T13:20:26.431982+0000","last_fullsized":"2024-04-02T13:20:26.431982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911122+0000","last_change":"2024-04-02T13:20:20.899007+0000","last_active":"2024-04-02T13:20:25.911122+0000","last_peered":"2024-04-02T13:20:25.911122+0000","last_clean":"2024-04-02T13:20:25.911122+0000","last_became_active":"2024-04-02T13:20:20.898005+0000","last_became_peered":"2024-04-02T13:20:20.898005+0000","last_unstale":"2024-04-02T13:20:25.911122+0000","last_undegraded":"2024-04-02T13:20:25.911122+0000","last_fullsized":"2024-04-02T13:20:25.911122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.432495+0000","last_change":"2024-04-02T13:20:20.896893+0000","last_active":"2024-04-02T13:20:26.432495+0000","last_peered":"2024-04-02T13:20:26.432495+0000","last_clean":"2024-04-02T13:20:26.432495+0000","last_became_active":"2024-04-02T13:20:20.896629+0000","last_became_peered":"2024-04-02T13:20:20.896629+0000","last_unstale":"2024-04-02T13:20:26.432495+0000","last_undegraded":"2024-04-02T13:20:26.432495+0000","last_fullsized":"2024-04-02T13:20:26.432495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431161+0000","last_change":"2024-04-02T13:20:20.900656+0000","last_active":"2024-04-02T13:20:26.431161+0000","last_peered":"2024-04-02T13:20:26.431161+0000","last_clean":"2024-04-02T13:20:26.431161+0000","last_became_active":"2024-04-02T13:20:20.900385+0000","last_became_peered":"2024-04-02T13:20:20.900385+0000","last_unstale":"2024-04-02T13:20:26.431161+0000","last_undegraded":"2024-04-02T13:20:26.431161+0000","last_fullsized":"2024-04-02T13:20:26.431161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.911108+0000","last_change":"2024-04-02T13:20:20.897613+0000","last_active":"2024-04-02T13:20:25.911108+0000","last_peered":"2024-04-02T13:20:25.911108+0000","last_clean":"2024-04-02T13:20:25.911108+0000","last_became_active":"2024-04-02T13:20:20.897285+0000","last_became_peered":"2024-04-02T13:20:20.897285+0000","last_unstale":"2024-04-02T13:20:25.911108+0000","last_undegraded":"2024-04-02T13:20:25.911108+0000","last_fullsized":"2024-04-02T13:20:25.911108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:30.719314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913747+0000","last_change":"2024-04-02T13:20:20.907084+0000","last_active":"2024-04-02T13:20:25.913747+0000","last_peered":"2024-04-02T13:20:25.913747+0000","last_clean":"2024-04-02T13:20:25.913747+0000","last_became_active":"2024-04-02T13:20:20.906734+0000","last_became_peered":"2024-04-02T13:20:20.906734+0000","last_unstale":"2024-04-02T13:20:25.913747+0000","last_undegraded":"2024-04-02T13:20:25.913747+0000","last_fullsized":"2024-04-02T13:20:25.913747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910526+0000","last_change":"2024-04-02T13:20:20.897768+0000","last_active":"2024-04-02T13:20:25.910526+0000","last_peered":"2024-04-02T13:20:25.910526+0000","last_clean":"2024-04-02T13:20:25.910526+0000","last_became_active":"2024-04-02T13:20:20.897502+0000","last_became_peered":"2024-04-02T13:20:20.897502+0000","last_unstale":"2024-04-02T13:20:25.910526+0000","last_undegraded":"2024-04-02T13:20:25.910526+0000","last_fullsized":"2024-04-02T13:20:25.910526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:26.532089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912825+0000","last_change":"2024-04-02T13:20:20.903454+0000","last_active":"2024-04-02T13:20:25.912825+0000","last_peered":"2024-04-02T13:20:25.912825+0000","last_clean":"2024-04-02T13:20:25.912825+0000","last_became_active":"2024-04-02T13:20:20.903050+0000","last_became_peered":"2024-04-02T13:20:20.903050+0000","last_unstale":"2024-04-02T13:20:25.912825+0000","last_undegraded":"2024-04-02T13:20:25.912825+0000","last_fullsized":"2024-04-02T13:20:25.912825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:56:59.954023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431891+0000","last_change":"2024-04-02T13:20:20.904681+0000","last_active":"2024-04-02T13:20:26.431891+0000","last_peered":"2024-04-02T13:20:26.431891+0000","last_clean":"2024-04-02T13:20:26.431891+0000","last_became_active":"2024-04-02T13:20:20.904343+0000","last_became_peered":"2024-04-02T13:20:20.904343+0000","last_unstale":"2024-04-02T13:20:26.431891+0000","last_undegraded":"2024-04-02T13:20:26.431891+0000","last_fullsized":"2024-04-02T13:20:26.431891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:22.414825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431123+0000","last_change":"2024-04-02T13:20:20.895363+0000","last_active":"2024-04-02T13:20:26.431123+0000","last_peered":"2024-04-02T13:20:26.431123+0000","last_clean":"2024-04-02T13:20:26.431123+0000","last_became_active":"2024-04-02T13:20:20.895107+0000","last_became_peered":"2024-04-02T13:20:20.895107+0000","last_unstale":"2024-04-02T13:20:26.431123+0000","last_undegraded":"2024-04-02T13:20:26.431123+0000","last_fullsized":"2024-04-02T13:20:26.431123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:22.414825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431618+0000","last_change":"2024-04-02T13:20:20.890501+0000","last_active":"2024-04-02T13:20:26.431618+0000","last_peered":"2024-04-02T13:20:26.431618+0000","last_clean":"2024-04-02T13:20:26.431618+0000","last_became_active":"2024-04-02T13:20:20.890181+0000","last_became_peered":"2024-04-02T13:20:20.890181+0000","last_unstale":"2024-04-02T13:20:26.431618+0000","last_undegraded":"2024-04-02T13:20:26.431618+0000","last_fullsized":"2024-04-02T13:20:26.431618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:02:18.661127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430915+0000","last_change":"2024-04-02T13:20:20.892430+0000","last_active":"2024-04-02T13:20:26.430915+0000","last_peered":"2024-04-02T13:20:26.430915+0000","last_clean":"2024-04-02T13:20:26.430915+0000","last_became_active":"2024-04-02T13:20:20.892117+0000","last_became_peered":"2024-04-02T13:20:20.892117+0000","last_unstale":"2024-04-02T13:20:26.430915+0000","last_undegraded":"2024-04-02T13:20:26.430915+0000","last_fullsized":"2024-04-02T13:20:26.430915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:37.366302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913765+0000","last_change":"2024-04-02T13:20:20.903371+0000","last_active":"2024-04-02T13:20:25.913765+0000","last_peered":"2024-04-02T13:20:25.913765+0000","last_clean":"2024-04-02T13:20:25.913765+0000","last_became_active":"2024-04-02T13:20:20.903078+0000","last_became_peered":"2024-04-02T13:20:20.903078+0000","last_unstale":"2024-04-02T13:20:25.913765+0000","last_undegraded":"2024-04-02T13:20:25.913765+0000","last_fullsized":"2024-04-02T13:20:25.913765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:42.709577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913653+0000","last_change":"2024-04-02T13:20:20.901550+0000","last_active":"2024-04-02T13:20:25.913653+0000","last_peered":"2024-04-02T13:20:25.913653+0000","last_clean":"2024-04-02T13:20:25.913653+0000","last_became_active":"2024-04-02T13:20:20.901234+0000","last_became_peered":"2024-04-02T13:20:20.901234+0000","last_unstale":"2024-04-02T13:20:25.913653+0000","last_undegraded":"2024-04-02T13:20:25.913653+0000","last_fullsized":"2024-04-02T13:20:25.913653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430422+0000","last_change":"2024-04-02T13:20:20.887101+0000","last_active":"2024-04-02T13:20:26.430422+0000","last_peered":"2024-04-02T13:20:26.430422+0000","last_clean":"2024-04-02T13:20:26.430422+0000","last_became_active":"2024-04-02T13:20:20.886810+0000","last_became_peered":"2024-04-02T13:20:20.886810+0000","last_unstale":"2024-04-02T13:20:26.430422+0000","last_undegraded":"2024-04-02T13:20:26.430422+0000","last_fullsized":"2024-04-02T13:20:26.430422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:21:06.334598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430889+0000","last_change":"2024-04-02T13:20:20.898387+0000","last_active":"2024-04-02T13:20:26.430889+0000","last_peered":"2024-04-02T13:20:26.430889+0000","last_clean":"2024-04-02T13:20:26.430889+0000","last_became_active":"2024-04-02T13:20:20.898075+0000","last_became_peered":"2024-04-02T13:20:20.898075+0000","last_unstale":"2024-04-02T13:20:26.430889+0000","last_undegraded":"2024-04-02T13:20:26.430889+0000","last_fullsized":"2024-04-02T13:20:26.430889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:26.614243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910512+0000","last_change":"2024-04-02T13:20:20.895603+0000","last_active":"2024-04-02T13:20:25.910512+0000","last_peered":"2024-04-02T13:20:25.910512+0000","last_clean":"2024-04-02T13:20:25.910512+0000","last_became_active":"2024-04-02T13:20:20.895316+0000","last_became_peered":"2024-04-02T13:20:20.895316+0000","last_unstale":"2024-04-02T13:20:25.910512+0000","last_undegraded":"2024-04-02T13:20:25.910512+0000","last_fullsized":"2024-04-02T13:20:25.910512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:34.303387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.913831+0000","last_change":"2024-04-02T13:20:20.889381+0000","last_active":"2024-04-02T13:20:25.913831+0000","last_peered":"2024-04-02T13:20:25.913831+0000","last_clean":"2024-04-02T13:20:25.913831+0000","last_became_active":"2024-04-02T13:20:20.889102+0000","last_became_peered":"2024-04-02T13:20:20.889102+0000","last_unstale":"2024-04-02T13:20:25.913831+0000","last_undegraded":"2024-04-02T13:20:25.913831+0000","last_fullsized":"2024-04-02T13:20:25.913831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:37.366302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430695+0000","last_change":"2024-04-02T13:20:20.904474+0000","last_active":"2024-04-02T13:20:26.430695+0000","last_peered":"2024-04-02T13:20:26.430695+0000","last_clean":"2024-04-02T13:20:26.430695+0000","last_became_active":"2024-04-02T13:20:20.904212+0000","last_became_peered":"2024-04-02T13:20:20.904212+0000","last_unstale":"2024-04-02T13:20:26.430695+0000","last_undegraded":"2024-04-02T13:20:26.430695+0000","last_fullsized":"2024-04-02T13:20:26.430695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431059+0000","last_change":"2024-04-02T13:20:20.904469+0000","last_active":"2024-04-02T13:20:26.431059+0000","last_peered":"2024-04-02T13:20:26.431059+0000","last_clean":"2024-04-02T13:20:26.431059+0000","last_became_active":"2024-04-02T13:20:20.904195+0000","last_became_peered":"2024-04-02T13:20:20.904195+0000","last_unstale":"2024-04-02T13:20:26.431059+0000","last_undegraded":"2024-04-02T13:20:26.431059+0000","last_fullsized":"2024-04-02T13:20:26.431059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431650+0000","last_change":"2024-04-02T13:20:20.905842+0000","last_active":"2024-04-02T13:20:26.431650+0000","last_peered":"2024-04-02T13:20:26.431650+0000","last_clean":"2024-04-02T13:20:26.431650+0000","last_became_active":"2024-04-02T13:20:20.905558+0000","last_became_peered":"2024-04-02T13:20:20.905558+0000","last_unstale":"2024-04-02T13:20:26.431650+0000","last_undegraded":"2024-04-02T13:20:26.431650+0000","last_fullsized":"2024-04-02T13:20:26.431650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:22:02.784989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.910120+0000","last_change":"2024-04-02T13:20:20.902048+0000","last_active":"2024-04-02T13:20:25.910120+0000","last_peered":"2024-04-02T13:20:25.910120+0000","last_clean":"2024-04-02T13:20:25.910120+0000","last_became_active":"2024-04-02T13:20:20.901733+0000","last_became_peered":"2024-04-02T13:20:20.901733+0000","last_unstale":"2024-04-02T13:20:25.910120+0000","last_undegraded":"2024-04-02T13:20:25.910120+0000","last_fullsized":"2024-04-02T13:20:25.910120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:14.996480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431149+0000","last_change":"2024-04-02T13:20:20.901947+0000","last_active":"2024-04-02T13:20:26.431149+0000","last_peered":"2024-04-02T13:20:26.431149+0000","last_clean":"2024-04-02T13:20:26.431149+0000","last_became_active":"2024-04-02T13:20:20.901655+0000","last_became_peered":"2024-04-02T13:20:20.901655+0000","last_unstale":"2024-04-02T13:20:26.431149+0000","last_undegraded":"2024-04-02T13:20:26.431149+0000","last_fullsized":"2024-04-02T13:20:26.431149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:56:59.954023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912733+0000","last_change":"2024-04-02T13:20:20.905389+0000","last_active":"2024-04-02T13:20:25.912733+0000","last_peered":"2024-04-02T13:20:25.912733+0000","last_clean":"2024-04-02T13:20:25.912733+0000","last_became_active":"2024-04-02T13:20:20.905062+0000","last_became_peered":"2024-04-02T13:20:20.905062+0000","last_unstale":"2024-04-02T13:20:25.912733+0000","last_undegraded":"2024-04-02T13:20:25.912733+0000","last_fullsized":"2024-04-02T13:20:25.912733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:34.303387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430753+0000","last_change":"2024-04-02T13:20:20.897784+0000","last_active":"2024-04-02T13:20:26.430753+0000","last_peered":"2024-04-02T13:20:26.430753+0000","last_clean":"2024-04-02T13:20:26.430753+0000","last_became_active":"2024-04-02T13:20:20.897488+0000","last_became_peered":"2024-04-02T13:20:20.897488+0000","last_unstale":"2024-04-02T13:20:26.430753+0000","last_undegraded":"2024-04-02T13:20:26.430753+0000","last_fullsized":"2024-04-02T13:20:26.430753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:26:16.298897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:25.912800+0000","last_change":"2024-04-02T13:20:20.908726+0000","last_active":"2024-04-02T13:20:25.912800+0000","last_peered":"2024-04-02T13:20:25.912800+0000","last_clean":"2024-04-02T13:20:25.912800+0000","last_became_active":"2024-04-02T13:20:20.908504+0000","last_became_peered":"2024-04-02T13:20:20.908504+0000","last_unstale":"2024-04-02T13:20:25.912800+0000","last_undegraded":"2024-04-02T13:20:25.912800+0000","last_fullsized":"2024-04-02T13:20:25.912800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:26.532089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.430330+0000","last_change":"2024-04-02T13:20:20.892134+0000","last_active":"2024-04-02T13:20:26.430330+0000","last_peered":"2024-04-02T13:20:26.430330+0000","last_clean":"2024-04-02T13:20:26.430330+0000","last_became_active":"2024-04-02T13:20:20.891860+0000","last_became_peered":"2024-04-02T13:20:20.891860+0000","last_unstale":"2024-04-02T13:20:26.430330+0000","last_undegraded":"2024-04-02T13:20:26.430330+0000","last_fullsized":"2024-04-02T13:20:26.430330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:17:34.808750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431697+0000","last_change":"2024-04-02T13:20:20.890864+0000","last_active":"2024-04-02T13:20:26.431697+0000","last_peered":"2024-04-02T13:20:26.431697+0000","last_clean":"2024-04-02T13:20:26.431697+0000","last_became_active":"2024-04-02T13:20:20.890585+0000","last_became_peered":"2024-04-02T13:20:20.890585+0000","last_unstale":"2024-04-02T13:20:26.431697+0000","last_undegraded":"2024-04-02T13:20:26.431697+0000","last_fullsized":"2024-04-02T13:20:26.431697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:37:42.712799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-02T13:20:26.431543+0000","last_change":"2024-04-02T13:20:20.892679+0000","last_active":"2024-04-02T13:20:26.431543+0000","last_peered":"2024-04-02T13:20:26.431543+0000","last_clean":"2024-04-02T13:20:26.431543+0000","last_became_active":"2024-04-02T13:20:20.892440+0000","last_became_peered":"2024-04-02T13:20:20.892440+0000","last_unstale":"2024-04-02T13:20:26.431543+0000","last_undegraded":"2024-04-02T13:20:26.431543+0000","last_fullsized":"2024-04-02T13:20:26.431543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:19.864691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:19.864691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:39.096965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:20:48.833 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-02T13:20:48.833 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-02T13:20:48.833 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-02T13:20:48.833 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-02T13:20:48.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.968+0000 7f86829d4640 1 Processor -- start 2024-04-02T13:20:48.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.968+0000 7f86829d4640 1 -- start start 2024-04-02T13:20:48.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.969+0000 7f86829d4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f867c0cdd40 0x7f867c0b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.969+0000 7f86829d4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f867c0d0ab0 con 0x7f867c15c080 2024-04-02T13:20:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.969+0000 7f86829d4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f867c0d0d90 con 0x7f867c15c460 2024-04-02T13:20:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.970+0000 7f86829d4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f867c0cce30 con 0x7f867c0cdd40 2024-04-02T13:20:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.970+0000 7f867bfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f867c15c080 0x7f867c06f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57534/0 (socket says 172.21.15.44:57534) 2024-04-02T13:20:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.970+0000 7f867bfff640 1 -- 172.21.15.44:0/230825080 learned_addr learned my addr 172.21.15.44:0/230825080 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.970+0000 7f8680f71640 1 --2- 172.21.15.44:0/230825080 >> v2:172.21.15.69:3300/0 conn(0x7f867c0cdd40 0x7f867c0b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.970+0000 7f867affd640 1 -- 172.21.15.44:0/230825080 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3006218926 0 0) 0x7f867c0d0ab0 con 0x7f867c15c080 2024-04-02T13:20:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.970+0000 7f867affd640 1 -- 172.21.15.44:0/230825080 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8664003520 con 0x7f867c15c080 2024-04-02T13:20:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.970+0000 7f867affd640 1 -- 172.21.15.44:0/230825080 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1055158003 0 0) 0x7f867c0d0d90 con 0x7f867c15c460 2024-04-02T13:20:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.970+0000 7f867affd640 1 -- 172.21.15.44:0/230825080 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f867c0d0ab0 con 0x7f867c15c460 2024-04-02T13:20:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.970+0000 7f8680f71640 1 -- 172.21.15.44:0/230825080 >> v1:172.21.15.69:6789/0 conn(0x7f867c15c460 legacy=0x7f867c0cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.970+0000 7f867affd640 1 -- 172.21.15.44:0/230825080 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2962163946 0 0) 0x7f8664003520 con 0x7f867c15c080 2024-04-02T13:20:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.970+0000 7f8680f71640 1 -- 172.21.15.44:0/230825080 >> v1:172.21.15.44:6789/0 conn(0x7f867c15c080 legacy=0x7f867c06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.970+0000 7f8680f71640 1 -- 172.21.15.44:0/230825080 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f867c0cd110 con 0x7f867c0cdd40 2024-04-02T13:20:48.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.971+0000 7f8680f71640 1 --2- 172.21.15.44:0/230825080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f867c0cdd40 0x7f867c0b3a30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8674009d90 tx=0x7f867402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6980697935436c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:48.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.972+0000 7f867affd640 1 -- 172.21.15.44:0/230825080 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674039070 con 0x7f867c0cdd40 2024-04-02T13:20:48.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.972+0000 7f867affd640 1 -- 172.21.15.44:0/230825080 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f867402c940 con 0x7f867c0cdd40 2024-04-02T13:20:48.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.972+0000 7f867affd640 1 -- 172.21.15.44:0/230825080 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8670001670 con 0x7f867c0cdd40 2024-04-02T13:20:48.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.972+0000 7f86829d4640 1 -- 172.21.15.44:0/230825080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f867c0cdd40 msgr2=0x7f867c0b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.972+0000 7f86829d4640 1 --2- 172.21.15.44:0/230825080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f867c0cdd40 0x7f867c0b3a30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8674009d90 tx=0x7f867402be10 comp rx=0 tx=0).stop 2024-04-02T13:20:48.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.972+0000 7f86829d4640 1 -- 172.21.15.44:0/230825080 shutdown_connections 2024-04-02T13:20:48.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.972+0000 7f86829d4640 1 --2- 172.21.15.44:0/230825080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f867c0cdd40 0x7f867c0b3a30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.972+0000 7f86829d4640 1 -- 172.21.15.44:0/230825080 >> 172.21.15.44:0/230825080 conn(0x7f867c073870 msgr2=0x7f867c09aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:48.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.972+0000 7f86829d4640 1 -- 172.21.15.44:0/230825080 shutdown_connections 2024-04-02T13:20:48.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.972+0000 7f86829d4640 1 -- 172.21.15.44:0/230825080 wait complete. 2024-04-02T13:20:48.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.974+0000 7f86829d4640 1 Processor -- start 2024-04-02T13:20:48.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.974+0000 7f86829d4640 1 -- start start 2024-04-02T13:20:48.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.974+0000 7f86829d4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f867c0cdd40 0x7f867c0c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.974+0000 7f86829d4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f867c15c080 0x7f867c155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.974+0000 7f86829d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f867c15c460 0x7f867c06dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f86829d4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f867c0ccbd0 con 0x7f867c0cdd40 2024-04-02T13:20:48.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f86829d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f867c0cca50 con 0x7f867c15c460 2024-04-02T13:20:48.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f86829d4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f867c0b3f70 con 0x7f867c15c080 2024-04-02T13:20:48.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f867bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f867c0cdd40 0x7f867c0c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f867bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f867c0cdd40 0x7f867c0c7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55094/0 (socket says 172.21.15.44:55094) 2024-04-02T13:20:48.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f867bfff640 1 -- 172.21.15.44:0/943886 learned_addr learned my addr 172.21.15.44:0/943886 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:48.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f8680f71640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f867c15c460 0x7f867c06dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f867b7fe640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f867c15c080 0x7f867c155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f867bfff640 1 -- 172.21.15.44:0/943886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f867c15c080 msgr2=0x7f867c155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f867bfff640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f867c15c080 0x7f867c155510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f867bfff640 1 -- 172.21.15.44:0/943886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f867c15c460 msgr2=0x7f867c06dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f867bfff640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f867c15c460 0x7f867c06dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f867bfff640 1 -- 172.21.15.44:0/943886 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f867c06cd40 con 0x7f867c0cdd40 2024-04-02T13:20:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f867b7fe640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f867c15c080 0x7f867c155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f867bfff640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f867c0cdd40 0x7f867c0c7820 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f867000c780 tx=0x7f867000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f8678ff9640 1 -- 172.21.15.44:0/943886 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8670005ad0 con 0x7f867c0cdd40 2024-04-02T13:20:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f8678ff9640 1 -- 172.21.15.44:0/943886 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8670005c90 con 0x7f867c0cdd40 2024-04-02T13:20:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f8678ff9640 1 -- 172.21.15.44:0/943886 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86700072b0 con 0x7f867c0cdd40 2024-04-02T13:20:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f86829d4640 1 -- 172.21.15.44:0/943886 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f867c06d020 con 0x7f867c0cdd40 2024-04-02T13:20:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.975+0000 7f86829d4640 1 -- 172.21.15.44:0/943886 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f867c06d5e0 con 0x7f867c0cdd40 2024-04-02T13:20:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.976+0000 7f86829d4640 1 -- 172.21.15.44:0/943886 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f867c068c00 con 0x7f867c0cdd40 2024-04-02T13:20:48.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.977+0000 7f8678ff9640 1 -- 172.21.15.44:0/943886 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8670007450 con 0x7f867c0cdd40 2024-04-02T13:20:48.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.977+0000 7f8678ff9640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f866806c8f0 0x7f866806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:48.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.977+0000 7f8678ff9640 1 -- 172.21.15.44:0/943886 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f867008bcb0 con 0x7f867c0cdd40 2024-04-02T13:20:48.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.977+0000 7f867b7fe640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f866806c8f0 0x7f866806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:48.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.978+0000 7f867b7fe640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f866806c8f0 0x7f866806edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f867c06faf0 tx=0x7f866c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:48.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:48.981+0000 7f8678ff9640 1 -- 172.21.15.44:0/943886 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8670059da0 con 0x7f867c0cdd40 2024-04-02T13:20:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.205+0000 7f86829d4640 1 -- 172.21.15.44:0/943886 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f867c158100 con 0x7f867c0cdd40 2024-04-02T13:20:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.206+0000 7f8678ff9640 1 -- 172.21.15.44:0/943886 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f86700596a0 con 0x7f867c0cdd40 2024-04-02T13:20:49.208 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:49.208 INFO:teuthology.orchestra.run.smithi044.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-02T13:20:49.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.208+0000 7f86829d4640 1 -- 172.21.15.44:0/943886 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f866806c8f0 msgr2=0x7f866806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.208+0000 7f86829d4640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f866806c8f0 0x7f866806edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f867c06faf0 tx=0x7f866c009040 comp rx=0 tx=0).stop 2024-04-02T13:20:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.208+0000 7f86829d4640 1 -- 172.21.15.44:0/943886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f867c0cdd40 msgr2=0x7f867c0c7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.208+0000 7f86829d4640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f867c0cdd40 0x7f867c0c7820 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f867000c780 tx=0x7f867000cc40 comp rx=0 tx=0).stop 2024-04-02T13:20:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.208+0000 7f86829d4640 1 -- 172.21.15.44:0/943886 shutdown_connections 2024-04-02T13:20:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.208+0000 7f86829d4640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f866806c8f0 0x7f866806edb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.208+0000 7f86829d4640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f867c15c460 0x7f867c06dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.208+0000 7f86829d4640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f867c15c080 0x7f867c155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.208+0000 7f86829d4640 1 --2- 172.21.15.44:0/943886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f867c0cdd40 0x7f867c0c7820 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.208+0000 7f86829d4640 1 -- 172.21.15.44:0/943886 >> 172.21.15.44:0/943886 conn(0x7f867c073870 msgr2=0x7f867c0cd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.208+0000 7f86829d4640 1 -- 172.21.15.44:0/943886 shutdown_connections 2024-04-02T13:20:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.208+0000 7f86829d4640 1 -- 172.21.15.44:0/943886 wait complete. 2024-04-02T13:20:49.221 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-02T13:20:49.221 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:49.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.320+0000 7f56bcb7c640 1 Processor -- start 2024-04-02T13:20:49.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.320+0000 7f56bcb7c640 1 -- start start 2024-04-02T13:20:49.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.321+0000 7f56bcb7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56b80d28f0 0x7f56b80d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.321+0000 7f56bcb7c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b80d4d70 con 0x7f56b80b47b0 2024-04-02T13:20:49.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.321+0000 7f56bcb7c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b80d2030 con 0x7f56b80b5200 2024-04-02T13:20:49.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.321+0000 7f56bcb7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f56b80d5050 con 0x7f56b80d28f0 2024-04-02T13:20:49.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.321+0000 7f56b6d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f56b80b47b0 0x7f56b80b7120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57538/0 (socket says 172.21.15.44:57538) 2024-04-02T13:20:49.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.321+0000 7f56b6d9d640 1 -- 172.21.15.44:0/2606251034 learned_addr learned my addr 172.21.15.44:0/2606251034 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:49.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.321+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3310076656 0 0) 0x7f56b80d4d70 con 0x7f56b80b47b0 2024-04-02T13:20:49.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.321+0000 7f56b659c640 1 --2- 172.21.15.44:0/2606251034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56b80d28f0 0x7f56b80d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.321+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56a4003520 con 0x7f56b80b47b0 2024-04-02T13:20:49.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1812759220 0 0) 0x7f56a4003520 con 0x7f56b80b47b0 2024-04-02T13:20:49.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56b80d28f0 msgr2=0x7f56b80d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b559a640 1 --2- 172.21.15.44:0/2606251034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56b80d28f0 0x7f56b80d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 >> v1:172.21.15.69:6789/0 conn(0x7f56b80b5200 legacy=0x7f56b80b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b80b4cb0 con 0x7f56b80b47b0 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f56a80019f0 con 0x7f56b80b47b0 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 >> v1:172.21.15.44:6789/0 conn(0x7f56b80b47b0 legacy=0x7f56b80b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b559a640 1 --2- 172.21.15.44:0/2606251034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56a40039e0 0x7f56a4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b559a640 1 --2- 172.21.15.44:0/2606251034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56a4004800 0x7f56a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b559a640 1 --2- 172.21.15.44:0/2606251034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56b80d28f0 0x7f56a4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f56b80d5050 con 0x7f56a4004800 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f56a400baf0 con 0x7f56a40039e0 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f56a400bcf0 con 0x7f56b80d28f0 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b5d9b640 1 --2- 172.21.15.44:0/2606251034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56a4004800 0x7f56a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b659c640 1 --2- 172.21.15.44:0/2606251034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56a40039e0 0x7f56a4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b6d9d640 1 --2- 172.21.15.44:0/2606251034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56b80d28f0 0x7f56a4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b5d9b640 1 -- 172.21.15.44:0/2606251034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56b80d28f0 msgr2=0x7f56a4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:49.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b5d9b640 1 --2- 172.21.15.44:0/2606251034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56b80d28f0 0x7f56a4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b5d9b640 1 -- 172.21.15.44:0/2606251034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56a40039e0 msgr2=0x7f56a4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:49.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b5d9b640 1 --2- 172.21.15.44:0/2606251034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56a40039e0 0x7f56a4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.322+0000 7f56b5d9b640 1 -- 172.21.15.44:0/2606251034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a4003520 con 0x7f56a4004800 2024-04-02T13:20:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.323+0000 7f56b5d9b640 1 --2- 172.21.15.44:0/2606251034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56a4004800 0x7f56a4004c80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f56a0001910 tx=0x7f56a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7181fe9fd4e136ed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.323+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a00014e0 con 0x7f56a4004800 2024-04-02T13:20:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.323+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56a0037070 con 0x7f56a4004800 2024-04-02T13:20:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.323+0000 7f56b559a640 1 -- 172.21.15.44:0/2606251034 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a0032840 con 0x7f56a4004800 2024-04-02T13:20:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.324+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2606251034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56a4004800 msgr2=0x7f56a4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.324+0000 7f56bcb7c640 1 --2- 172.21.15.44:0/2606251034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56a4004800 0x7f56a4004c80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f56a0001910 tx=0x7f56a4005830 comp rx=0 tx=0).stop 2024-04-02T13:20:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.324+0000 7f56b6d9d640 1 -- 172.21.15.44:0/2606251034 reap_dead start 2024-04-02T13:20:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.324+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2606251034 shutdown_connections 2024-04-02T13:20:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.324+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2606251034 >> 172.21.15.44:0/2606251034 conn(0x7f56b80d4120 msgr2=0x7f56b80d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.324+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2606251034 shutdown_connections 2024-04-02T13:20:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.324+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2606251034 wait complete. 2024-04-02T13:20:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.325+0000 7f56bcb7c640 1 Processor -- start 2024-04-02T13:20:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.325+0000 7f56bcb7c640 1 -- start start 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56bcb7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56a4004800 0x7f56b80b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56bcb7c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b815edd0 0x7f56b815f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56bcb7c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b81612b0 0x7f56b8161730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56bcb7c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f56b80d23a0 con 0x7f56b815edd0 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56bcb7c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f56b80d14f0 con 0x7f56b81612b0 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56bcb7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f56b80b5020 con 0x7f56a4004800 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56b5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b815edd0 0x7f56b815f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56b5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b815edd0 0x7f56b815f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55124/0 (socket says 172.21.15.44:55124) 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56b5d9b640 1 -- 172.21.15.44:0/2527638396 learned_addr learned my addr 172.21.15.44:0/2527638396 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56b659c640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56a4004800 0x7f56b80b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56b6d9d640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b81612b0 0x7f56b8161730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56b5d9b640 1 -- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56a4004800 msgr2=0x7f56b80b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56b5d9b640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56a4004800 0x7f56b80b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56b5d9b640 1 -- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b81612b0 msgr2=0x7f56b8161730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56b5d9b640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b81612b0 0x7f56b8161730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56b5d9b640 1 -- 172.21.15.44:0/2527638396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b80b2550 con 0x7f56b815edd0 2024-04-02T13:20:49.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56b659c640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56a4004800 0x7f56b80b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:49.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56b5d9b640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b815edd0 0x7f56b815f230 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f56a0034040 tx=0x7f56a00012b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:49.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56aeffd640 1 -- 172.21.15.44:0/2527638396 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a0006e50 con 0x7f56b815edd0 2024-04-02T13:20:49.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56aeffd640 1 -- 172.21.15.44:0/2527638396 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56a0040070 con 0x7f56b815edd0 2024-04-02T13:20:49.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2527638396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b80b2830 con 0x7f56b815edd0 2024-04-02T13:20:49.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.326+0000 7f56aeffd640 1 -- 172.21.15.44:0/2527638396 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a0007240 con 0x7f56b815edd0 2024-04-02T13:20:49.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.327+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2527638396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b80b2df0 con 0x7f56b815edd0 2024-04-02T13:20:49.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.328+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2527638396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56b80684f0 con 0x7f56b815edd0 2024-04-02T13:20:49.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.328+0000 7f56aeffd640 1 -- 172.21.15.44:0/2527638396 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f56a00073e0 con 0x7f56b815edd0 2024-04-02T13:20:49.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.328+0000 7f56aeffd640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f569806ca90 0x7f569806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.328+0000 7f56aeffd640 1 -- 172.21.15.44:0/2527638396 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f56a00ade80 con 0x7f56b815edd0 2024-04-02T13:20:49.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.329+0000 7f56b659c640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f569806ca90 0x7f569806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.329+0000 7f56b659c640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f569806ca90 0x7f569806ef50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009b40 tx=0x7f56b0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:49.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.333+0000 7f56aeffd640 1 -- 172.21.15.44:0/2527638396 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f56a007bee0 con 0x7f56b815edd0 2024-04-02T13:20:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.516+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2527638396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f56b80d2e50 con 0x7f56b815edd0 2024-04-02T13:20:49.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.518+0000 7f56aeffd640 1 -- 172.21.15.44:0/2527638396 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f56a007b7e0 con 0x7f56b815edd0 2024-04-02T13:20:49.520 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:49.520 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:49.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.519+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f569806ca90 msgr2=0x7f569806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.519+0000 7f56bcb7c640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f569806ca90 0x7f569806ef50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009b40 tx=0x7f56b0008040 comp rx=0 tx=0).stop 2024-04-02T13:20:49.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.519+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2527638396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b815edd0 msgr2=0x7f56b815f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.519+0000 7f56bcb7c640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b815edd0 0x7f56b815f230 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f56a0034040 tx=0x7f56a00012b0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.519+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2527638396 shutdown_connections 2024-04-02T13:20:49.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.519+0000 7f56bcb7c640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f569806ca90 0x7f569806ef50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.519+0000 7f56bcb7c640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b81612b0 0x7f56b8161730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.519+0000 7f56bcb7c640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b815edd0 0x7f56b815f230 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.519+0000 7f56bcb7c640 1 --2- 172.21.15.44:0/2527638396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56a4004800 0x7f56b80b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.519+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2527638396 >> 172.21.15.44:0/2527638396 conn(0x7f56b80d4120 msgr2=0x7f56b80b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:49.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.520+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2527638396 shutdown_connections 2024-04-02T13:20:49.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.520+0000 7f56bcb7c640 1 -- 172.21.15.44:0/2527638396 wait complete. 2024-04-02T13:20:49.533 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:20:49.533 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:20:49.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.636+0000 7fc0d7dd1640 1 Processor -- start 2024-04-02T13:20:49.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.636+0000 7fc0d7dd1640 1 -- start start 2024-04-02T13:20:49.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.637+0000 7fc0d7dd1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d2e00 0x7fc0d00b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.637+0000 7fc0d7dd1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0d006de80 con 0x7fc0d00cfdc0 2024-04-02T13:20:49.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.637+0000 7fc0d7dd1640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0d00cf000 con 0x7fc0d006daa0 2024-04-02T13:20:49.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.637+0000 7fc0d7dd1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc0d00cf930 con 0x7fc0d00d2e00 2024-04-02T13:20:49.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.637+0000 7fc0d636e640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc0d00cfdc0 0x7fc0d015dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57554/0 (socket says 172.21.15.44:57554) 2024-04-02T13:20:49.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.637+0000 7fc0d636e640 1 -- 172.21.15.44:0/513985187 learned_addr learned my addr 172.21.15.44:0/513985187 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:49.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.637+0000 7fc0d536c640 1 --2- 172.21.15.44:0/513985187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d2e00 0x7fc0d00b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.637+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1993672323 0 0) 0x7fc0d006de80 con 0x7fc0d00cfdc0 2024-04-02T13:20:49.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.637+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0bc003520 con 0x7fc0d00cfdc0 2024-04-02T13:20:49.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.637+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 996798815 0 0) 0x7fc0bc003520 con 0x7fc0d00cfdc0 2024-04-02T13:20:49.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.637+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d2e00 msgr2=0x7fc0d00b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:49.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.637+0000 7fc0d4b6b640 1 --2- 172.21.15.44:0/513985187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d2e00 0x7fc0d00b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.637+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 >> v1:172.21.15.69:6789/0 conn(0x7fc0d006daa0 legacy=0x7fc0d00d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0d00cf2e0 con 0x7fc0d00cfdc0 2024-04-02T13:20:49.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fc0c80019f0 con 0x7fc0d00cfdc0 2024-04-02T13:20:49.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 >> v1:172.21.15.44:6789/0 conn(0x7fc0d00cfdc0 legacy=0x7fc0d015dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d4b6b640 1 --2- 172.21.15.44:0/513985187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d2e00 0x7fc0bc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d4b6b640 1 --2- 172.21.15.44:0/513985187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0bc003f20 0x7fc0bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d4b6b640 1 --2- 172.21.15.44:0/513985187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0bc004d40 0x7fc0bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d00cf930 con 0x7fc0bc003f20 2024-04-02T13:20:49.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc00ba20 con 0x7fc0bc004d40 2024-04-02T13:20:49.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc0bc00bc20 con 0x7fc0d00d2e00 2024-04-02T13:20:49.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d536c640 1 --2- 172.21.15.44:0/513985187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0bc003f20 0x7fc0bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d636e640 1 --2- 172.21.15.44:0/513985187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0bc004d40 0x7fc0bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d5b6d640 1 --2- 172.21.15.44:0/513985187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d2e00 0x7fc0bc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d536c640 1 -- 172.21.15.44:0/513985187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d2e00 msgr2=0x7fc0bc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:49.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d536c640 1 --2- 172.21.15.44:0/513985187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0d00d2e00 0x7fc0bc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d536c640 1 -- 172.21.15.44:0/513985187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0bc004d40 msgr2=0x7fc0bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:49.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d536c640 1 --2- 172.21.15.44:0/513985187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0bc004d40 0x7fc0bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.638+0000 7fc0d536c640 1 -- 172.21.15.44:0/513985187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0bc003520 con 0x7fc0bc003f20 2024-04-02T13:20:49.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.639+0000 7fc0d536c640 1 --2- 172.21.15.44:0/513985187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0bc003f20 0x7fc0bc004800 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0009aa0 tx=0x7fc0bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863f8f7cd4ce84bc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:49.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.639+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0039070 con 0x7fc0bc003f20 2024-04-02T13:20:49.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.639+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0c002c600 con 0x7fc0bc003f20 2024-04-02T13:20:49.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.639+0000 7fc0d4b6b640 1 -- 172.21.15.44:0/513985187 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0009670 con 0x7fc0bc003f20 2024-04-02T13:20:49.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.639+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/513985187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0bc003f20 msgr2=0x7fc0bc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.639+0000 7fc0d7dd1640 1 --2- 172.21.15.44:0/513985187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0bc003f20 0x7fc0bc004800 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0009aa0 tx=0x7fc0bc005830 comp rx=0 tx=0).stop 2024-04-02T13:20:49.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.639+0000 7fc0d636e640 1 -- 172.21.15.44:0/513985187 reap_dead start 2024-04-02T13:20:49.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.639+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/513985187 shutdown_connections 2024-04-02T13:20:49.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.639+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/513985187 >> 172.21.15.44:0/513985187 conn(0x7fc0d009d430 msgr2=0x7fc0d00d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:49.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.640+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/513985187 shutdown_connections 2024-04-02T13:20:49.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.640+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/513985187 wait complete. 2024-04-02T13:20:49.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.641+0000 7fc0d7dd1640 1 Processor -- start 2024-04-02T13:20:49.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.641+0000 7fc0d7dd1640 1 -- start start 2024-04-02T13:20:49.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.641+0000 7fc0d7dd1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0bc003f20 0x7fc0d0155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.641+0000 7fc0d7dd1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0d00b8e40 0x7fc0d00b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.641+0000 7fc0d7dd1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0d01541c0 0x7fc0d0154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.641+0000 7fc0d7dd1640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d00ceda0 con 0x7fc0d00b8e40 2024-04-02T13:20:49.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.641+0000 7fc0d7dd1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d00cec20 con 0x7fc0d01541c0 2024-04-02T13:20:49.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.641+0000 7fc0d7dd1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc0d00b35b0 con 0x7fc0bc003f20 2024-04-02T13:20:49.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.641+0000 7fc0d536c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0d00b8e40 0x7fc0d00b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0d536c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0d00b8e40 0x7fc0d00b92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55144/0 (socket says 172.21.15.44:55144) 2024-04-02T13:20:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0d536c640 1 -- 172.21.15.44:0/152480106 learned_addr learned my addr 172.21.15.44:0/152480106 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0d636e640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0d01541c0 0x7fc0d0154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0d5b6d640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0bc003f20 0x7fc0d0155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0d536c640 1 -- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0bc003f20 msgr2=0x7fc0d0155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0d536c640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0bc003f20 0x7fc0d0155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0d536c640 1 -- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0d01541c0 msgr2=0x7fc0d0154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0d536c640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0d01541c0 0x7fc0d0154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0d536c640 1 -- 172.21.15.44:0/152480106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0d0156290 con 0x7fc0d00b8e40 2024-04-02T13:20:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0d536c640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0d00b8e40 0x7fc0d00b92a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0037040 tx=0x7fc0c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0c67fc640 1 -- 172.21.15.44:0/152480106 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0044020 con 0x7fc0d00b8e40 2024-04-02T13:20:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0c67fc640 1 -- 172.21.15.44:0/152480106 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0c003f730 con 0x7fc0d00b8e40 2024-04-02T13:20:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0c67fc640 1 -- 172.21.15.44:0/152480106 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0039040 con 0x7fc0d00b8e40 2024-04-02T13:20:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/152480106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0d0154cb0 con 0x7fc0d00b8e40 2024-04-02T13:20:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.642+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/152480106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0d0155270 con 0x7fc0d00b8e40 2024-04-02T13:20:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.643+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/152480106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0d00684f0 con 0x7fc0d00b8e40 2024-04-02T13:20:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.644+0000 7fc0c67fc640 1 -- 172.21.15.44:0/152480106 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc0c0047020 con 0x7fc0d00b8e40 2024-04-02T13:20:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.644+0000 7fc0c67fc640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0b406c8f0 0x7fc0b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.644+0000 7fc0c67fc640 1 -- 172.21.15.44:0/152480106 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc0c00adef0 con 0x7fc0d00b8e40 2024-04-02T13:20:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.644+0000 7fc0d5b6d640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0b406c8f0 0x7fc0b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.645+0000 7fc0d5b6d640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0b406c8f0 0x7fc0b406edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc003700 tx=0x7fc0cc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:49.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.648+0000 7fc0c67fc640 1 -- 172.21.15.44:0/152480106 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc0c004e030 con 0x7fc0d00b8e40 2024-04-02T13:20:49.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.831+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/152480106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc0d00d24d0 con 0x7fc0d00b8e40 2024-04-02T13:20:49.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.831+0000 7fc0c67fc640 1 -- 172.21.15.44:0/152480106 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7fc0c0080e20 con 0x7fc0d00b8e40 2024-04-02T13:20:49.834 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:20:49.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.833+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0b406c8f0 msgr2=0x7fc0b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.833+0000 7fc0d7dd1640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0b406c8f0 0x7fc0b406edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc003700 tx=0x7fc0cc009040 comp rx=0 tx=0).stop 2024-04-02T13:20:49.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.833+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/152480106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0d00b8e40 msgr2=0x7fc0d00b92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.833+0000 7fc0d7dd1640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0d00b8e40 0x7fc0d00b92a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0037040 tx=0x7fc0c0004730 comp rx=0 tx=0).stop 2024-04-02T13:20:49.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.833+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/152480106 shutdown_connections 2024-04-02T13:20:49.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.833+0000 7fc0d7dd1640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0b406c8f0 0x7fc0b406edb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.833+0000 7fc0d7dd1640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0d01541c0 0x7fc0d0154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.833+0000 7fc0d7dd1640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0d00b8e40 0x7fc0d00b92a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.833+0000 7fc0d7dd1640 1 --2- 172.21.15.44:0/152480106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0bc003f20 0x7fc0d0155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.833+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/152480106 >> 172.21.15.44:0/152480106 conn(0x7fc0d009d430 msgr2=0x7fc0d00b3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:49.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.833+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/152480106 shutdown_connections 2024-04-02T13:20:49.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.833+0000 7fc0d7dd1640 1 -- 172.21.15.44:0/152480106 wait complete. 2024-04-02T13:20:49.846 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:20:49.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.949+0000 7fe987be9640 1 Processor -- start 2024-04-02T13:20:49.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.949+0000 7fe987be9640 1 -- start start 2024-04-02T13:20:49.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe987be9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe98015e5d0 0x7fe9800d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe987be9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9800b4180 0x7fe9800b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe987be9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9800d1630 con 0x7fe98006cff0 2024-04-02T13:20:49.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe987be9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe9800d1bf0 con 0x7fe98015e5d0 2024-04-02T13:20:49.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe987be9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe9800d1910 con 0x7fe9800b4180 2024-04-02T13:20:49.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe985985640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe98006cff0 0x7fe98006d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57558/0 (socket says 172.21.15.44:57558) 2024-04-02T13:20:49.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe985985640 1 -- 172.21.15.44:0/3532981364 learned_addr learned my addr 172.21.15.44:0/3532981364 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:49.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe986186640 1 --2- 172.21.15.44:0/3532981364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9800b4180 0x7fe9800b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe985184640 1 --2- 172.21.15.44:0/3532981364 >> v2:172.21.15.69:3300/0 conn(0x7fe98015e5d0 0x7fe9800d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 582209327 0 0) 0x7fe9800d1630 con 0x7fe98006cff0 2024-04-02T13:20:49.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe96c003520 con 0x7fe98006cff0 2024-04-02T13:20:49.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 246210180 0 0) 0x7fe96c003520 con 0x7fe98006cff0 2024-04-02T13:20:49.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9800b4180 msgr2=0x7fe9800b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:49.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe984983640 1 --2- 172.21.15.44:0/3532981364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9800b4180 0x7fe9800b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 >> v2:172.21.15.69:3300/0 conn(0x7fe98015e5d0 msgr2=0x7fe9800d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:49.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe984983640 1 --2- 172.21.15.44:0/3532981364 >> v2:172.21.15.69:3300/0 conn(0x7fe98015e5d0 0x7fe9800d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9800b1540 con 0x7fe98006cff0 2024-04-02T13:20:49.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.950+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fe97c001670 con 0x7fe98006cff0 2024-04-02T13:20:49.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 >> v1:172.21.15.44:6789/0 conn(0x7fe98006cff0 legacy=0x7fe98006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe984983640 1 --2- 172.21.15.44:0/3532981364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe96c0039b0 0x7fe96c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe984983640 1 --2- 172.21.15.44:0/3532981364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9800b4180 0x7fe96c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe984983640 1 --2- 172.21.15.44:0/3532981364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe96c004d40 0x7fe96c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe9800d1bf0 con 0x7fe96c0039b0 2024-04-02T13:20:49.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe9800d1910 con 0x7fe96c004d40 2024-04-02T13:20:49.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe96c00b9c0 con 0x7fe9800b4180 2024-04-02T13:20:49.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe985985640 1 --2- 172.21.15.44:0/3532981364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe96c0039b0 0x7fe96c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe985184640 1 --2- 172.21.15.44:0/3532981364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9800b4180 0x7fe96c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe986186640 1 --2- 172.21.15.44:0/3532981364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe96c004d40 0x7fe96c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe985985640 1 -- 172.21.15.44:0/3532981364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9800b4180 msgr2=0x7fe96c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:49.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe985985640 1 --2- 172.21.15.44:0/3532981364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9800b4180 0x7fe96c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe985985640 1 -- 172.21.15.44:0/3532981364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe96c004d40 msgr2=0x7fe96c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:49.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe985985640 1 --2- 172.21.15.44:0/3532981364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe96c004d40 0x7fe96c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe985985640 1 -- 172.21.15.44:0/3532981364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe96c003520 con 0x7fe96c0039b0 2024-04-02T13:20:49.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.951+0000 7fe986186640 1 --2- 172.21.15.44:0/3532981364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe96c004d40 0x7fe96c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:49.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.952+0000 7fe985985640 1 --2- 172.21.15.44:0/3532981364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe96c0039b0 0x7fe96c003dd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00c290 tx=0x7fe96c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e2728dc2172dcd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:49.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.952+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c007d50 con 0x7fe96c0039b0 2024-04-02T13:20:49.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.952+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe97c030070 con 0x7fe96c0039b0 2024-04-02T13:20:49.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.952+0000 7fe984983640 1 -- 172.21.15.44:0/3532981364 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c02cd80 con 0x7fe96c0039b0 2024-04-02T13:20:49.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.952+0000 7fe987be9640 1 -- 172.21.15.44:0/3532981364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe96c0039b0 msgr2=0x7fe96c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.952+0000 7fe987be9640 1 --2- 172.21.15.44:0/3532981364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe96c0039b0 0x7fe96c003dd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00c290 tx=0x7fe96c005830 comp rx=0 tx=0).stop 2024-04-02T13:20:49.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.952+0000 7fe986186640 1 -- 172.21.15.44:0/3532981364 reap_dead start 2024-04-02T13:20:49.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.952+0000 7fe987be9640 1 -- 172.21.15.44:0/3532981364 shutdown_connections 2024-04-02T13:20:49.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.952+0000 7fe987be9640 1 -- 172.21.15.44:0/3532981364 >> 172.21.15.44:0/3532981364 conn(0x7fe98009aa20 msgr2=0x7fe98009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:49.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.952+0000 7fe987be9640 1 -- 172.21.15.44:0/3532981364 shutdown_connections 2024-04-02T13:20:49.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.952+0000 7fe987be9640 1 -- 172.21.15.44:0/3532981364 wait complete. 2024-04-02T13:20:49.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.954+0000 7fe987be9640 1 Processor -- start 2024-04-02T13:20:49.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.954+0000 7fe987be9640 1 -- start start 2024-04-02T13:20:49.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.954+0000 7fe987be9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe96c0039b0 0x7fe9800b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.954+0000 7fe987be9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9800b6da0 0x7fe980159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.954+0000 7fe987be9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe98015a3d0 0x7fe980151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.954+0000 7fe987be9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe9800d13f0 con 0x7fe9800b6da0 2024-04-02T13:20:49.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.954+0000 7fe987be9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe9800d1270 con 0x7fe98015a3d0 2024-04-02T13:20:49.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.954+0000 7fe987be9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe9800b1d40 con 0x7fe96c0039b0 2024-04-02T13:20:49.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.954+0000 7fe985184640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9800b6da0 0x7fe980159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.954+0000 7fe985184640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9800b6da0 0x7fe980159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55158/0 (socket says 172.21.15.44:55158) 2024-04-02T13:20:49.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.954+0000 7fe985184640 1 -- 172.21.15.44:0/1117095521 learned_addr learned my addr 172.21.15.44:0/1117095521 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:49.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.955+0000 7fe986186640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe98015a3d0 0x7fe980151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.955+0000 7fe985985640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe96c0039b0 0x7fe9800b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.955+0000 7fe985184640 1 -- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe96c0039b0 msgr2=0x7fe9800b6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.955+0000 7fe985184640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe96c0039b0 0x7fe9800b6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.955+0000 7fe985184640 1 -- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe98015a3d0 msgr2=0x7fe980151040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:49.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.955+0000 7fe985184640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe98015a3d0 0x7fe980151040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:49.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.955+0000 7fe985184640 1 -- 172.21.15.44:0/1117095521 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9800b7ea0 con 0x7fe9800b6da0 2024-04-02T13:20:49.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.955+0000 7fe985184640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9800b6da0 0x7fe980159e90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe9700119b0 tx=0x7fe970011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:49.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.955+0000 7fe9767fc640 1 -- 172.21.15.44:0/1117095521 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe970019070 con 0x7fe9800b6da0 2024-04-02T13:20:49.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.955+0000 7fe9767fc640 1 -- 172.21.15.44:0/1117095521 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe970004240 con 0x7fe9800b6da0 2024-04-02T13:20:49.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.955+0000 7fe987be9640 1 -- 172.21.15.44:0/1117095521 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9800b4d70 con 0x7fe9800b6da0 2024-04-02T13:20:49.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.955+0000 7fe9767fc640 1 -- 172.21.15.44:0/1117095521 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97000f8e0 con 0x7fe9800b6da0 2024-04-02T13:20:49.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.955+0000 7fe987be9640 1 -- 172.21.15.44:0/1117095521 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9800b73c0 con 0x7fe9800b6da0 2024-04-02T13:20:49.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.956+0000 7fe987be9640 1 -- 172.21.15.44:0/1117095521 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9800684f0 con 0x7fe9800b6da0 2024-04-02T13:20:49.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.956+0000 7fe9767fc640 1 -- 172.21.15.44:0/1117095521 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe97001c070 con 0x7fe9800b6da0 2024-04-02T13:20:49.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.957+0000 7fe9767fc640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe96406c920 0x7fe96406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:49.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.957+0000 7fe9767fc640 1 -- 172.21.15.44:0/1117095521 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe97008c2c0 con 0x7fe9800b6da0 2024-04-02T13:20:49.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.957+0000 7fe985985640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe96406c920 0x7fe96406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:49.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.957+0000 7fe985985640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe96406c920 0x7fe96406ede0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe97c034000 tx=0x7fe97c007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:49.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:49.961+0000 7fe9767fc640 1 -- 172.21.15.44:0/1117095521 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe97005a3b0 con 0x7fe9800b6da0 2024-04-02T13:20:50.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.145+0000 7fe987be9640 1 -- 172.21.15.44:0/1117095521 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe98006cdd0 con 0x7fe9800b6da0 2024-04-02T13:20:50.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.145+0000 7fe9767fc640 1 -- 172.21.15.44:0/1117095521 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fe970059cb0 con 0x7fe9800b6da0 2024-04-02T13:20:50.148 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:20:50.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.147+0000 7fe987be9640 1 -- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe96406c920 msgr2=0x7fe96406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.147+0000 7fe987be9640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe96406c920 0x7fe96406ede0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe97c034000 tx=0x7fe97c007000 comp rx=0 tx=0).stop 2024-04-02T13:20:50.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.147+0000 7fe987be9640 1 -- 172.21.15.44:0/1117095521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9800b6da0 msgr2=0x7fe980159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.147+0000 7fe987be9640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9800b6da0 0x7fe980159e90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe9700119b0 tx=0x7fe970011e70 comp rx=0 tx=0).stop 2024-04-02T13:20:50.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.147+0000 7fe987be9640 1 -- 172.21.15.44:0/1117095521 shutdown_connections 2024-04-02T13:20:50.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.147+0000 7fe987be9640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe96406c920 0x7fe96406ede0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.147+0000 7fe987be9640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe98015a3d0 0x7fe980151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.147+0000 7fe987be9640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9800b6da0 0x7fe980159e90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.147+0000 7fe987be9640 1 --2- 172.21.15.44:0/1117095521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe96c0039b0 0x7fe9800b6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.147+0000 7fe987be9640 1 -- 172.21.15.44:0/1117095521 >> 172.21.15.44:0/1117095521 conn(0x7fe98009aa20 msgr2=0x7fe98015cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:50.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.147+0000 7fe987be9640 1 -- 172.21.15.44:0/1117095521 shutdown_connections 2024-04-02T13:20:50.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.147+0000 7fe987be9640 1 -- 172.21.15.44:0/1117095521 wait complete. 2024-04-02T13:20:50.161 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:20:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.262+0000 7f7ba33cd640 1 Processor -- start 2024-04-02T13:20:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.263+0000 7f7ba33cd640 1 -- start start 2024-04-02T13:20:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.263+0000 7f7ba33cd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7b9c15c1a0 0x7f7b9c06f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.263+0000 7f7ba33cd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7b9c0cfd60 0x7f7b9c0d1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.263+0000 7f7ba33cd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b9c0d1ff0 0x7f7b9c0ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.263+0000 7f7ba33cd640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7b9c070ad0 con 0x7f7b9c15c1a0 2024-04-02T13:20:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.263+0000 7f7ba33cd640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7b9c15c580 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.263+0000 7f7ba33cd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7b9c0d01c0 con 0x7f7b9c0d1ff0 2024-04-02T13:20:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.263+0000 7f7ba1169640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7b9c15c1a0 0x7f7b9c06f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.263+0000 7f7ba1169640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7b9c15c1a0 0x7f7b9c06f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55164/0 (socket says 172.21.15.44:55164) 2024-04-02T13:20:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.263+0000 7f7ba1169640 1 -- 172.21.15.44:0/2643401396 learned_addr learned my addr 172.21.15.44:0/2643401396 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.264+0000 7f7ba196a640 1 --2- 172.21.15.44:0/2643401396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b9c0d1ff0 0x7f7b9c0ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.264+0000 7f7ba0968640 1 --2- 172.21.15.44:0/2643401396 >> v2:172.21.15.69:3300/0 conn(0x7f7b9c0cfd60 0x7f7b9c0d1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.264+0000 7f7ba1169640 1 -- 172.21.15.44:0/2643401396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b9c0d1ff0 msgr2=0x7f7b9c0ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.264+0000 7f7ba1169640 1 --2- 172.21.15.44:0/2643401396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b9c0d1ff0 0x7f7b9c0ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.264+0000 7f7ba1169640 1 -- 172.21.15.44:0/2643401396 >> v2:172.21.15.69:3300/0 conn(0x7f7b9c0cfd60 msgr2=0x7f7b9c0d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.264+0000 7f7ba1169640 1 --2- 172.21.15.44:0/2643401396 >> v2:172.21.15.69:3300/0 conn(0x7f7b9c0cfd60 0x7f7b9c0d1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.264+0000 7f7ba1169640 1 -- 172.21.15.44:0/2643401396 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9c0d2480 con 0x7f7b9c15c1a0 2024-04-02T13:20:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.264+0000 7f7ba0968640 1 --2- 172.21.15.44:0/2643401396 >> v2:172.21.15.69:3300/0 conn(0x7f7b9c0cfd60 0x7f7b9c0d1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.264+0000 7f7ba196a640 1 --2- 172.21.15.44:0/2643401396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b9c0d1ff0 0x7f7b9c0ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:50.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.264+0000 7f7ba1169640 1 --2- 172.21.15.44:0/2643401396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b9c15c1a0 0x7f7b9c06f370 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0d2f10 tx=0x7f7b9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77a6431eaa9f9f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.265+0000 7f7b93fff640 1 -- 172.21.15.44:0/2643401396 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98035020 con 0x7f7b9c15c1a0 2024-04-02T13:20:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.265+0000 7f7b93fff640 1 -- 172.21.15.44:0/2643401396 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b98031870 con 0x7f7b9c15c1a0 2024-04-02T13:20:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.265+0000 7f7b93fff640 1 -- 172.21.15.44:0/2643401396 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98031c40 con 0x7f7b9c15c1a0 2024-04-02T13:20:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.265+0000 7f7ba33cd640 1 -- 172.21.15.44:0/2643401396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b9c15c1a0 msgr2=0x7f7b9c06f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.265+0000 7f7ba33cd640 1 --2- 172.21.15.44:0/2643401396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b9c15c1a0 0x7f7b9c06f370 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0d2f10 tx=0x7f7b9802b910 comp rx=0 tx=0).stop 2024-04-02T13:20:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.265+0000 7f7ba33cd640 1 -- 172.21.15.44:0/2643401396 shutdown_connections 2024-04-02T13:20:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.265+0000 7f7ba33cd640 1 --2- 172.21.15.44:0/2643401396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b9c0d1ff0 0x7f7b9c0ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.265+0000 7f7ba33cd640 1 --2- 172.21.15.44:0/2643401396 >> v2:172.21.15.69:3300/0 conn(0x7f7b9c0cfd60 0x7f7b9c0d1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.265+0000 7f7ba33cd640 1 --2- 172.21.15.44:0/2643401396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b9c15c1a0 0x7f7b9c06f370 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.265+0000 7f7ba33cd640 1 -- 172.21.15.44:0/2643401396 >> 172.21.15.44:0/2643401396 conn(0x7f7b9c09aa20 msgr2=0x7f7b9c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.265+0000 7f7ba33cd640 1 -- 172.21.15.44:0/2643401396 shutdown_connections 2024-04-02T13:20:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.265+0000 7f7ba33cd640 1 -- 172.21.15.44:0/2643401396 wait complete. 2024-04-02T13:20:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.266+0000 7f7ba33cd640 1 Processor -- start 2024-04-02T13:20:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.266+0000 7f7ba33cd640 1 -- start start 2024-04-02T13:20:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.267+0000 7f7ba33cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b9c0cfd60 0x7f7b9c15c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.267+0000 7f7ba33cd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b9c0d1ff0 0x7f7b9c0c4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.267+0000 7f7ba33cd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b9c15c1a0 0x7f7b9c1f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.267+0000 7f7ba33cd640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c06fff0 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.267+0000 7f7ba33cd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c06fd10 con 0x7f7b9c0d1ff0 2024-04-02T13:20:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.267+0000 7f7ba33cd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7b9c0702d0 con 0x7f7b9c15c1a0 2024-04-02T13:20:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.267+0000 7f7ba1169640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b9c0cfd60 0x7f7b9c15c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.267+0000 7f7ba1169640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b9c0cfd60 0x7f7b9c15c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55174/0 (socket says 172.21.15.44:55174) 2024-04-02T13:20:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.267+0000 7f7ba1169640 1 -- 172.21.15.44:0/485639176 learned_addr learned my addr 172.21.15.44:0/485639176 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.267+0000 7f7ba0968640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b9c0d1ff0 0x7f7b9c0c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.267+0000 7f7ba196a640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b9c15c1a0 0x7f7b9c1f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.267+0000 7f7ba1169640 1 -- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b9c15c1a0 msgr2=0x7f7b9c1f5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.268+0000 7f7ba1169640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b9c15c1a0 0x7f7b9c1f5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.268+0000 7f7ba1169640 1 -- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b9c0d1ff0 msgr2=0x7f7b9c0c4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.268+0000 7f7ba1169640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b9c0d1ff0 0x7f7b9c0c4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.268+0000 7f7ba1169640 1 -- 172.21.15.44:0/485639176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9c0c61b0 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.268+0000 7f7ba1169640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b9c0cfd60 0x7f7b9c15c730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7b9802b950 tx=0x7f7b980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.268+0000 7f7b91ffb640 1 -- 172.21.15.44:0/485639176 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98035040 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.268+0000 7f7b91ffb640 1 -- 172.21.15.44:0/485639176 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b98030d00 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.268+0000 7f7ba33cd640 1 -- 172.21.15.44:0/485639176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b9c0c6490 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.268+0000 7f7b91ffb640 1 -- 172.21.15.44:0/485639176 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b980456a0 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.268+0000 7f7ba33cd640 1 -- 172.21.15.44:0/485639176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b9c0c6a50 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.269+0000 7f7ba33cd640 1 -- 172.21.15.44:0/485639176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b9c068c00 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.270+0000 7f7b91ffb640 1 -- 172.21.15.44:0/485639176 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7b98038020 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.270+0000 7f7b91ffb640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b8006c8f0 0x7f7b8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.270+0000 7f7b91ffb640 1 -- 172.21.15.44:0/485639176 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7b980ae000 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.270+0000 7f7ba0968640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b8006c8f0 0x7f7b8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.271+0000 7f7ba0968640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b8006c8f0 0x7f7b8006edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0c7c20 tx=0x7f7b8c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.274+0000 7f7b91ffb640 1 -- 172.21.15.44:0/485639176 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7b9807bb30 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.460+0000 7f7ba33cd640 1 -- 172.21.15.44:0/485639176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7b9c0ccd30 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.460+0000 7f7b91ffb640 1 -- 172.21.15.44:0/485639176 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f7b9807fe30 con 0x7f7b9c0cfd60 2024-04-02T13:20:50.463 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:20:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.462+0000 7f7ba33cd640 1 -- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b8006c8f0 msgr2=0x7f7b8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.462+0000 7f7ba33cd640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b8006c8f0 0x7f7b8006edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0c7c20 tx=0x7f7b8c008040 comp rx=0 tx=0).stop 2024-04-02T13:20:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.462+0000 7f7ba33cd640 1 -- 172.21.15.44:0/485639176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b9c0cfd60 msgr2=0x7f7b9c15c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.462+0000 7f7ba33cd640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b9c0cfd60 0x7f7b9c15c730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7b9802b950 tx=0x7f7b980046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.462+0000 7f7ba33cd640 1 -- 172.21.15.44:0/485639176 shutdown_connections 2024-04-02T13:20:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.462+0000 7f7ba33cd640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b8006c8f0 0x7f7b8006edb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.462+0000 7f7ba33cd640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b9c15c1a0 0x7f7b9c1f5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.462+0000 7f7ba33cd640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b9c0d1ff0 0x7f7b9c0c4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.462+0000 7f7ba33cd640 1 --2- 172.21.15.44:0/485639176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b9c0cfd60 0x7f7b9c15c730 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.462+0000 7f7ba33cd640 1 -- 172.21.15.44:0/485639176 >> 172.21.15.44:0/485639176 conn(0x7f7b9c09aa20 msgr2=0x7f7b9c0aeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.463+0000 7f7ba33cd640 1 -- 172.21.15.44:0/485639176 shutdown_connections 2024-04-02T13:20:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.463+0000 7f7ba33cd640 1 -- 172.21.15.44:0/485639176 wait complete. 2024-04-02T13:20:50.475 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:20:50.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.577+0000 7fa96fee4640 1 Processor -- start 2024-04-02T13:20:50.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.577+0000 7fa96fee4640 1 -- start start 2024-04-02T13:20:50.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.578+0000 7fa96fee4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa96806daa0 0x7fa9680b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.578+0000 7fa96fee4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa96806de80 con 0x7fa9680cfdc0 2024-04-02T13:20:50.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.578+0000 7fa96fee4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9680cf000 con 0x7fa9680d2e00 2024-04-02T13:20:50.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.578+0000 7fa96fee4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa9680cf930 con 0x7fa96806daa0 2024-04-02T13:20:50.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.578+0000 7fa96d47f640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa9680cfdc0 0x7fa9680d26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57560/0 (socket says 172.21.15.44:57560) 2024-04-02T13:20:50.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.578+0000 7fa96d47f640 1 -- 172.21.15.44:0/1927963806 learned_addr learned my addr 172.21.15.44:0/1927963806 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:50.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.578+0000 7fa96dc80640 1 --2- 172.21.15.44:0/1927963806 >> v2:172.21.15.69:3300/0 conn(0x7fa96806daa0 0x7fa9680b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.578+0000 7fa96cc7e640 1 -- 172.21.15.44:0/1927963806 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2709350398 0 0) 0x7fa96806de80 con 0x7fa9680cfdc0 2024-04-02T13:20:50.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.578+0000 7fa96cc7e640 1 -- 172.21.15.44:0/1927963806 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa950003520 con 0x7fa9680cfdc0 2024-04-02T13:20:50.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.579+0000 7fa96cc7e640 1 -- 172.21.15.44:0/1927963806 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1243113186 0 0) 0x7fa950003520 con 0x7fa9680cfdc0 2024-04-02T13:20:50.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.579+0000 7fa96dc80640 1 -- 172.21.15.44:0/1927963806 >> v1:172.21.15.69:6789/0 conn(0x7fa9680d2e00 legacy=0x7fa96815dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.579+0000 7fa96dc80640 1 -- 172.21.15.44:0/1927963806 >> v1:172.21.15.44:6789/0 conn(0x7fa9680cfdc0 legacy=0x7fa9680d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.579+0000 7fa96dc80640 1 -- 172.21.15.44:0/1927963806 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9680cf2e0 con 0x7fa96806daa0 2024-04-02T13:20:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.579+0000 7fa96dc80640 1 --2- 172.21.15.44:0/1927963806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96806daa0 0x7fa9680b35b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa9600097a0 tx=0x7fa96002b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=661869b001254dd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.580+0000 7fa96cc7e640 1 -- 172.21.15.44:0/1927963806 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960037070 con 0x7fa96806daa0 2024-04-02T13:20:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.580+0000 7fa96cc7e640 1 -- 172.21.15.44:0/1927963806 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa960030510 con 0x7fa96806daa0 2024-04-02T13:20:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.580+0000 7fa96cc7e640 1 -- 172.21.15.44:0/1927963806 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c001670 con 0x7fa96806daa0 2024-04-02T13:20:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.580+0000 7fa96fee4640 1 -- 172.21.15.44:0/1927963806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96806daa0 msgr2=0x7fa9680b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.580+0000 7fa96fee4640 1 --2- 172.21.15.44:0/1927963806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96806daa0 0x7fa9680b35b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa9600097a0 tx=0x7fa96002b680 comp rx=0 tx=0).stop 2024-04-02T13:20:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.580+0000 7fa96fee4640 1 -- 172.21.15.44:0/1927963806 shutdown_connections 2024-04-02T13:20:50.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.580+0000 7fa96fee4640 1 --2- 172.21.15.44:0/1927963806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96806daa0 0x7fa9680b35b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.580+0000 7fa96fee4640 1 -- 172.21.15.44:0/1927963806 >> 172.21.15.44:0/1927963806 conn(0x7fa96809d430 msgr2=0x7fa9680d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:50.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.580+0000 7fa96fee4640 1 -- 172.21.15.44:0/1927963806 shutdown_connections 2024-04-02T13:20:50.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.580+0000 7fa96fee4640 1 -- 172.21.15.44:0/1927963806 wait complete. 2024-04-02T13:20:50.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.581+0000 7fa96fee4640 1 Processor -- start 2024-04-02T13:20:50.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.581+0000 7fa96fee4640 1 -- start start 2024-04-02T13:20:50.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.582+0000 7fa96fee4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96806daa0 0x7fa96815f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.582+0000 7fa96fee4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9680cfdc0 0x7fa9681566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.582+0000 7fa96fee4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9680d2e00 0x7fa9680b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.582+0000 7fa96fee4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa9680ceda0 con 0x7fa9680cfdc0 2024-04-02T13:20:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.582+0000 7fa96fee4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9680cec20 con 0x7fa96806daa0 2024-04-02T13:20:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.582+0000 7fa96fee4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa9680d11c0 con 0x7fa9680d2e00 2024-04-02T13:20:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.582+0000 7fa96d47f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9680cfdc0 0x7fa9681566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.582+0000 7fa96d47f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9680cfdc0 0x7fa9681566f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55182/0 (socket says 172.21.15.44:55182) 2024-04-02T13:20:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.582+0000 7fa96d47f640 1 -- 172.21.15.44:0/2651107273 learned_addr learned my addr 172.21.15.44:0/2651107273 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.582+0000 7fa96dc80640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96806daa0 0x7fa96815f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.582+0000 7fa96e481640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9680d2e00 0x7fa9680b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.583+0000 7fa96d47f640 1 -- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9680d2e00 msgr2=0x7fa9680b14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.583+0000 7fa96d47f640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9680d2e00 0x7fa9680b14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.583+0000 7fa96d47f640 1 -- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96806daa0 msgr2=0x7fa96815f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.583+0000 7fa96d47f640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96806daa0 0x7fa96815f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.583+0000 7fa96d47f640 1 -- 172.21.15.44:0/2651107273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96815eb90 con 0x7fa9680cfdc0 2024-04-02T13:20:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.583+0000 7fa96d47f640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9680cfdc0 0x7fa9681566f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa95c0017e0 tx=0x7fa95c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.583+0000 7fa9667fc640 1 -- 172.21.15.44:0/2651107273 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c017070 con 0x7fa9680cfdc0 2024-04-02T13:20:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.583+0000 7fa9667fc640 1 -- 172.21.15.44:0/2651107273 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa95c005ad0 con 0x7fa9680cfdc0 2024-04-02T13:20:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.583+0000 7fa9667fc640 1 -- 172.21.15.44:0/2651107273 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c007070 con 0x7fa9680cfdc0 2024-04-02T13:20:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.583+0000 7fa96fee4640 1 -- 172.21.15.44:0/2651107273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa96815ee70 con 0x7fa9680cfdc0 2024-04-02T13:20:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.583+0000 7fa96fee4640 1 -- 172.21.15.44:0/2651107273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa968156c30 con 0x7fa9680cfdc0 2024-04-02T13:20:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.584+0000 7fa96fee4640 1 -- 172.21.15.44:0/2651107273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9680684f0 con 0x7fa9680cfdc0 2024-04-02T13:20:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.584+0000 7fa9667fc640 1 -- 172.21.15.44:0/2651107273 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa95c01a070 con 0x7fa9680cfdc0 2024-04-02T13:20:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.585+0000 7fa9667fc640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa95406c920 0x7fa95406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.585+0000 7fa9667fc640 1 -- 172.21.15.44:0/2651107273 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa95c08b2b0 con 0x7fa9680cfdc0 2024-04-02T13:20:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.585+0000 7fa96dc80640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa95406c920 0x7fa95406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.585+0000 7fa96dc80640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa95406c920 0x7fa95406ede0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa960034000 tx=0x7fa960008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.589+0000 7fa9667fc640 1 -- 172.21.15.44:0/2651107273 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa95c058cc0 con 0x7fa9680cfdc0 2024-04-02T13:20:50.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.771+0000 7fa96fee4640 1 -- 172.21.15.44:0/2651107273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa9680d24d0 con 0x7fa9680cfdc0 2024-04-02T13:20:50.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.771+0000 7fa9667fc640 1 -- 172.21.15.44:0/2651107273 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fa95c05cfc0 con 0x7fa9680cfdc0 2024-04-02T13:20:50.774 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:20:50.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.773+0000 7fa96fee4640 1 -- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa95406c920 msgr2=0x7fa95406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.773+0000 7fa96fee4640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa95406c920 0x7fa95406ede0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa960034000 tx=0x7fa960008030 comp rx=0 tx=0).stop 2024-04-02T13:20:50.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.773+0000 7fa96fee4640 1 -- 172.21.15.44:0/2651107273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9680cfdc0 msgr2=0x7fa9681566f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.773+0000 7fa96fee4640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9680cfdc0 0x7fa9681566f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa95c0017e0 tx=0x7fa95c00cbe0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.773+0000 7fa96fee4640 1 -- 172.21.15.44:0/2651107273 shutdown_connections 2024-04-02T13:20:50.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.773+0000 7fa96fee4640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa95406c920 0x7fa95406ede0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.773+0000 7fa96fee4640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9680d2e00 0x7fa9680b14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.773+0000 7fa96fee4640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9680cfdc0 0x7fa9681566f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.773+0000 7fa96fee4640 1 --2- 172.21.15.44:0/2651107273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96806daa0 0x7fa96815f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.773+0000 7fa96fee4640 1 -- 172.21.15.44:0/2651107273 >> 172.21.15.44:0/2651107273 conn(0x7fa96809d430 msgr2=0x7fa9680b2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:50.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.773+0000 7fa96fee4640 1 -- 172.21.15.44:0/2651107273 shutdown_connections 2024-04-02T13:20:50.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.773+0000 7fa96fee4640 1 -- 172.21.15.44:0/2651107273 wait complete. 2024-04-02T13:20:50.786 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:20:50.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.885+0000 7fd4a4374640 1 Processor -- start 2024-04-02T13:20:50.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.885+0000 7fd4a4374640 1 -- start start 2024-04-02T13:20:50.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a4374640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd49c15e140 0x7fd49c0d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a4374640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd49c15e520 0x7fd49c0d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a4374640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd49c0d2b70 0x7fd49c0d0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a4374640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd49c0b9f50 con 0x7fd49c15e140 2024-04-02T13:20:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a4374640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd49c0b70b0 con 0x7fd49c15e520 2024-04-02T13:20:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a4374640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd49c0b7390 con 0x7fd49c0d2b70 2024-04-02T13:20:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a2110640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd49c15e140 0x7fd49c0d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a2110640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd49c15e140 0x7fd49c0d2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55196/0 (socket says 172.21.15.44:55196) 2024-04-02T13:20:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a2110640 1 -- 172.21.15.44:0/3993762491 learned_addr learned my addr 172.21.15.44:0/3993762491 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a2110640 1 -- 172.21.15.44:0/3993762491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd49c0d2b70 msgr2=0x7fd49c0d0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a2911640 1 --2- 172.21.15.44:0/3993762491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd49c0d2b70 0x7fd49c0d0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a190f640 1 --2- 172.21.15.44:0/3993762491 >> v2:172.21.15.69:3300/0 conn(0x7fd49c15e520 0x7fd49c0d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a2110640 1 --2- 172.21.15.44:0/3993762491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd49c0d2b70 0x7fd49c0d0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a2110640 1 -- 172.21.15.44:0/3993762491 >> v2:172.21.15.69:3300/0 conn(0x7fd49c15e520 msgr2=0x7fd49c0d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a2110640 1 --2- 172.21.15.44:0/3993762491 >> v2:172.21.15.69:3300/0 conn(0x7fd49c15e520 0x7fd49c0d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a2110640 1 -- 172.21.15.44:0/3993762491 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49c0b7670 con 0x7fd49c15e140 2024-04-02T13:20:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a2911640 1 --2- 172.21.15.44:0/3993762491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd49c0d2b70 0x7fd49c0d0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.886+0000 7fd4a190f640 1 --2- 172.21.15.44:0/3993762491 >> v2:172.21.15.69:3300/0 conn(0x7fd49c15e520 0x7fd49c0d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.887+0000 7fd4a2110640 1 --2- 172.21.15.44:0/3993762491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49c15e140 0x7fd49c0d2630 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0ce6d0 tx=0x7fd49402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72454e0f7efc3c70 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.887+0000 7fd4a110e640 1 -- 172.21.15.44:0/3993762491 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd494035020 con 0x7fd49c15e140 2024-04-02T13:20:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.887+0000 7fd4a110e640 1 -- 172.21.15.44:0/3993762491 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd49402d040 con 0x7fd49c15e140 2024-04-02T13:20:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.887+0000 7fd4a110e640 1 -- 172.21.15.44:0/3993762491 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd494031a80 con 0x7fd49c15e140 2024-04-02T13:20:50.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.888+0000 7fd4a4374640 1 -- 172.21.15.44:0/3993762491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49c15e140 msgr2=0x7fd49c0d2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.888+0000 7fd4a4374640 1 --2- 172.21.15.44:0/3993762491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49c15e140 0x7fd49c0d2630 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0ce6d0 tx=0x7fd49402b900 comp rx=0 tx=0).stop 2024-04-02T13:20:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.888+0000 7fd4a4374640 1 -- 172.21.15.44:0/3993762491 shutdown_connections 2024-04-02T13:20:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.888+0000 7fd4a4374640 1 --2- 172.21.15.44:0/3993762491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd49c0d2b70 0x7fd49c0d0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.888+0000 7fd4a4374640 1 --2- 172.21.15.44:0/3993762491 >> v2:172.21.15.69:3300/0 conn(0x7fd49c15e520 0x7fd49c0d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.888+0000 7fd4a4374640 1 --2- 172.21.15.44:0/3993762491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49c15e140 0x7fd49c0d2630 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.888+0000 7fd4a4374640 1 -- 172.21.15.44:0/3993762491 >> 172.21.15.44:0/3993762491 conn(0x7fd49c09d390 msgr2=0x7fd49c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.888+0000 7fd4a4374640 1 -- 172.21.15.44:0/3993762491 shutdown_connections 2024-04-02T13:20:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.888+0000 7fd4a4374640 1 -- 172.21.15.44:0/3993762491 wait complete. 2024-04-02T13:20:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.889+0000 7fd4a4374640 1 Processor -- start 2024-04-02T13:20:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.889+0000 7fd4a4374640 1 -- start start 2024-04-02T13:20:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a4374640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd49c0d2b70 0x7fd49c0cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a4374640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd49c15e140 0x7fd49c1f0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a4374640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49c15e520 0x7fd49c1f0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a4374640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd49c0b4cc0 con 0x7fd49c15e520 2024-04-02T13:20:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a4374640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd49c0d4ee0 con 0x7fd49c0d2b70 2024-04-02T13:20:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a4374640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd49c0b4e40 con 0x7fd49c15e140 2024-04-02T13:20:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a2911640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49c15e520 0x7fd49c1f0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a2911640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49c15e520 0x7fd49c1f0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55208/0 (socket says 172.21.15.44:55208) 2024-04-02T13:20:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a2911640 1 -- 172.21.15.44:0/3731576928 learned_addr learned my addr 172.21.15.44:0/3731576928 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a2110640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd49c0d2b70 0x7fd49c0cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a190f640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd49c15e140 0x7fd49c1f0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a2911640 1 -- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd49c15e140 msgr2=0x7fd49c1f0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a2911640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd49c15e140 0x7fd49c1f0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a2911640 1 -- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd49c0d2b70 msgr2=0x7fd49c0cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a2911640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd49c0d2b70 0x7fd49c0cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:50.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.890+0000 7fd4a2911640 1 -- 172.21.15.44:0/3731576928 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49c1f1500 con 0x7fd49c15e520 2024-04-02T13:20:50.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.891+0000 7fd4a2911640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49c15e520 0x7fd49c1f0fc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd49800cea0 tx=0x7fd49800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:50.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.891+0000 7fd492ffd640 1 -- 172.21.15.44:0/3731576928 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd498003b90 con 0x7fd49c15e520 2024-04-02T13:20:50.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.891+0000 7fd492ffd640 1 -- 172.21.15.44:0/3731576928 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd498003d50 con 0x7fd49c15e520 2024-04-02T13:20:50.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.891+0000 7fd4a4374640 1 -- 172.21.15.44:0/3731576928 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd49c0ce0c0 con 0x7fd49c15e520 2024-04-02T13:20:50.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.891+0000 7fd492ffd640 1 -- 172.21.15.44:0/3731576928 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd498004260 con 0x7fd49c15e520 2024-04-02T13:20:50.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.891+0000 7fd4a4374640 1 -- 172.21.15.44:0/3731576928 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd49c15ff00 con 0x7fd49c15e520 2024-04-02T13:20:50.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.892+0000 7fd4a4374640 1 -- 172.21.15.44:0/3731576928 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd49c15fa10 con 0x7fd49c15e520 2024-04-02T13:20:50.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.892+0000 7fd492ffd640 1 -- 172.21.15.44:0/3731576928 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd498004400 con 0x7fd49c15e520 2024-04-02T13:20:50.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.893+0000 7fd492ffd640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd48006c8f0 0x7fd48006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:50.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.893+0000 7fd492ffd640 1 -- 172.21.15.44:0/3731576928 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd49808c750 con 0x7fd49c15e520 2024-04-02T13:20:50.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.893+0000 7fd4a2110640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd48006c8f0 0x7fd48006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:50.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.894+0000 7fd4a2110640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd48006c8f0 0x7fd48006edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd494036000 tx=0x7fd49403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:50.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:50.897+0000 7fd492ffd640 1 -- 172.21.15.44:0/3731576928 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd49805a280 con 0x7fd49c15e520 2024-04-02T13:20:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.123+0000 7fd4a4374640 1 -- 172.21.15.44:0/3731576928 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd49c0caec0 con 0x7fd49c15e520 2024-04-02T13:20:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.123+0000 7fd492ffd640 1 -- 172.21.15.44:0/3731576928 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7fd49805e580 con 0x7fd49c15e520 2024-04-02T13:20:51.126 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:51.126 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4040417710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4040417710},{"type":"v1","addr":"172.21.15.69:6836","nonce":4040417710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2772068060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2772068060},{"type":"v1","addr":"172.21.15.44:6833","nonce":2772068060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1985405150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1985405150},{"type":"v1","addr":"172.21.15.69:6837","nonce":1985405150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:20:20.247358+0000","modified":"2024-04-02T13:20:42.332570+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/2282156931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2282156931},{"type":"v1","addr":"172.21.15.44:6835","nonce":2282156931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-04-02T13:20:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 9 2024-04-02T13:20:51.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.125+0000 7fd4a4374640 1 -- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd48006c8f0 msgr2=0x7fd48006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.125+0000 7fd4a4374640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd48006c8f0 0x7fd48006edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd494036000 tx=0x7fd49403c040 comp rx=0 tx=0).stop 2024-04-02T13:20:51.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.125+0000 7fd4a4374640 1 -- 172.21.15.44:0/3731576928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49c15e520 msgr2=0x7fd49c1f0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.125+0000 7fd4a4374640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49c15e520 0x7fd49c1f0fc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd49800cea0 tx=0x7fd49800cf90 comp rx=0 tx=0).stop 2024-04-02T13:20:51.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.125+0000 7fd4a4374640 1 -- 172.21.15.44:0/3731576928 shutdown_connections 2024-04-02T13:20:51.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.125+0000 7fd4a4374640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd48006c8f0 0x7fd48006edb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.125+0000 7fd4a4374640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49c15e520 0x7fd49c1f0fc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.125+0000 7fd4a4374640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd49c15e140 0x7fd49c1f0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.125+0000 7fd4a4374640 1 --2- 172.21.15.44:0/3731576928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd49c0d2b70 0x7fd49c0cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.125+0000 7fd4a4374640 1 -- 172.21.15.44:0/3731576928 >> 172.21.15.44:0/3731576928 conn(0x7fd49c09d390 msgr2=0x7fd49c0b9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:51.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.125+0000 7fd4a4374640 1 -- 172.21.15.44:0/3731576928 shutdown_connections 2024-04-02T13:20:51.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.125+0000 7fd4a4374640 1 -- 172.21.15.44:0/3731576928 wait complete. 2024-04-02T13:20:51.139 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:51.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.237+0000 7f64edc93640 1 Processor -- start 2024-04-02T13:20:51.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.238+0000 7f64edc93640 1 -- start start 2024-04-02T13:20:51.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.238+0000 7f64edc93640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e815e4d0 0x7f64e80cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.238+0000 7f64edc93640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64e815e8b0 con 0x7f64e80cfd20 2024-04-02T13:20:51.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.238+0000 7f64edc93640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64e80d2fb0 con 0x7f64e80d1230 2024-04-02T13:20:51.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.238+0000 7f64edc93640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f64e80b3a90 con 0x7f64e815e4d0 2024-04-02T13:20:51.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.238+0000 7f64e6ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f64e80cfd20 0x7f64e80d0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57570/0 (socket says 172.21.15.44:57570) 2024-04-02T13:20:51.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.238+0000 7f64e6ffd640 1 -- 172.21.15.44:0/1417429046 learned_addr learned my addr 172.21.15.44:0/1417429046 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:51.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e77fe640 1 --2- 172.21.15.44:0/1417429046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e815e4d0 0x7f64e80cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1377360489 0 0) 0x7f64e815e8b0 con 0x7f64e80cfd20 2024-04-02T13:20:51.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64d0003520 con 0x7f64e80cfd20 2024-04-02T13:20:51.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 967364324 0 0) 0x7f64e80d2fb0 con 0x7f64e80d1230 2024-04-02T13:20:51.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64e815e8b0 con 0x7f64e80d1230 2024-04-02T13:20:51.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2856513422 0 0) 0x7f64d0003520 con 0x7f64e80cfd20 2024-04-02T13:20:51.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e815e4d0 msgr2=0x7f64e80cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:51.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 --2- 172.21.15.44:0/1417429046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e815e4d0 0x7f64e80cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 >> v1:172.21.15.69:6789/0 conn(0x7f64e80d1230 legacy=0x7f64e80b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e80b3ce0 con 0x7f64e80cfd20 2024-04-02T13:20:51.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f64dc001670 con 0x7f64e80cfd20 2024-04-02T13:20:51.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 >> v1:172.21.15.44:6789/0 conn(0x7f64e80cfd20 legacy=0x7f64e80d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 --2- 172.21.15.44:0/1417429046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d0003b70 0x7f64d0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 --2- 172.21.15.44:0/1417429046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64d0004800 0x7f64d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 --2- 172.21.15.44:0/1417429046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e815e4d0 0x7f64d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f64e80b3a90 con 0x7f64d0004800 2024-04-02T13:20:51.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f64d000ba70 con 0x7f64d0003b70 2024-04-02T13:20:51.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f64d000bc40 con 0x7f64e815e4d0 2024-04-02T13:20:51.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e6ffd640 1 --2- 172.21.15.44:0/1417429046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64d0004800 0x7f64d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e77fe640 1 --2- 172.21.15.44:0/1417429046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d0003b70 0x7f64d0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e7fff640 1 --2- 172.21.15.44:0/1417429046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e815e4d0 0x7f64d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e6ffd640 1 -- 172.21.15.44:0/1417429046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e815e4d0 msgr2=0x7f64d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:51.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e6ffd640 1 --2- 172.21.15.44:0/1417429046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e815e4d0 0x7f64d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e6ffd640 1 -- 172.21.15.44:0/1417429046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d0003b70 msgr2=0x7f64d0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:51.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e6ffd640 1 --2- 172.21.15.44:0/1417429046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d0003b70 0x7f64d0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.239+0000 7f64e6ffd640 1 -- 172.21.15.44:0/1417429046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d0003520 con 0x7f64d0004800 2024-04-02T13:20:51.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.240+0000 7f64e6ffd640 1 --2- 172.21.15.44:0/1417429046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64d0004800 0x7f64d0004c80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00c190 tx=0x7f64d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2494a80024f138d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:51.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.240+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc007cc0 con 0x7f64d0004800 2024-04-02T13:20:51.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.240+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64dc030070 con 0x7f64d0004800 2024-04-02T13:20:51.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.240+0000 7f64e67fc640 1 -- 172.21.15.44:0/1417429046 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc005d30 con 0x7f64d0004800 2024-04-02T13:20:51.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.241+0000 7f64edc93640 1 -- 172.21.15.44:0/1417429046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64d0004800 msgr2=0x7f64d0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.241+0000 7f64edc93640 1 --2- 172.21.15.44:0/1417429046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64d0004800 0x7f64d0004c80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00c190 tx=0x7f64d0005830 comp rx=0 tx=0).stop 2024-04-02T13:20:51.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.241+0000 7f64e7fff640 1 -- 172.21.15.44:0/1417429046 reap_dead start 2024-04-02T13:20:51.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.241+0000 7f64edc93640 1 -- 172.21.15.44:0/1417429046 shutdown_connections 2024-04-02T13:20:51.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.241+0000 7f64edc93640 1 -- 172.21.15.44:0/1417429046 >> 172.21.15.44:0/1417429046 conn(0x7f64e809d410 msgr2=0x7f64e80d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:51.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.241+0000 7f64edc93640 1 -- 172.21.15.44:0/1417429046 shutdown_connections 2024-04-02T13:20:51.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.241+0000 7f64edc93640 1 -- 172.21.15.44:0/1417429046 wait complete. 2024-04-02T13:20:51.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.242+0000 7f64edc93640 1 Processor -- start 2024-04-02T13:20:51.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.242+0000 7f64edc93640 1 -- start start 2024-04-02T13:20:51.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64edc93640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d0004800 0x7f64e8151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64edc93640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e80c7cc0 0x7f64e81f4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64edc93640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64e80ca7c0 0x7f64e80cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64edc93640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f64e80b42a0 con 0x7f64e80ca7c0 2024-04-02T13:20:51.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64edc93640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f64e80b4610 con 0x7f64d0004800 2024-04-02T13:20:51.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64edc93640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f64e80b0d00 con 0x7f64e80c7cc0 2024-04-02T13:20:51.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64e7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64e80ca7c0 0x7f64e80cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64e7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64e80ca7c0 0x7f64e80cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55238/0 (socket says 172.21.15.44:55238) 2024-04-02T13:20:51.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64e7fff640 1 -- 172.21.15.44:0/4012751667 learned_addr learned my addr 172.21.15.44:0/4012751667 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:51.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64e77fe640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d0004800 0x7f64e8151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64e6ffd640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e80c7cc0 0x7f64e81f4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64e7fff640 1 -- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e80c7cc0 msgr2=0x7f64e81f4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64e7fff640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e80c7cc0 0x7f64e81f4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64e7fff640 1 -- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d0004800 msgr2=0x7f64e8151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64e7fff640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d0004800 0x7f64e8151680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64e7fff640 1 -- 172.21.15.44:0/4012751667 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e81f4d60 con 0x7f64e80ca7c0 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64e77fe640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d0004800 0x7f64e8151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.243+0000 7f64e7fff640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64e80ca7c0 0x7f64e80cd210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f64e80b6d00 tx=0x7f64e000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.244+0000 7f64cffff640 1 -- 172.21.15.44:0/4012751667 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0003970 con 0x7f64e80ca7c0 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.244+0000 7f64cffff640 1 -- 172.21.15.44:0/4012751667 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64e0018070 con 0x7f64e80ca7c0 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.244+0000 7f64edc93640 1 -- 172.21.15.44:0/4012751667 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64e80c9480 con 0x7f64e80ca7c0 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.244+0000 7f64cffff640 1 -- 172.21.15.44:0/4012751667 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0003d50 con 0x7f64e80ca7c0 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.244+0000 7f64edc93640 1 -- 172.21.15.44:0/4012751667 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64e80c9a00 con 0x7f64e80ca7c0 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.245+0000 7f64edc93640 1 -- 172.21.15.44:0/4012751667 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64e8068c00 con 0x7f64e80ca7c0 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.245+0000 7f64cffff640 1 -- 172.21.15.44:0/4012751667 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f64e00050d0 con 0x7f64e80ca7c0 2024-04-02T13:20:51.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.246+0000 7f64cffff640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f64d406c920 0x7f64d406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.246+0000 7f64cffff640 1 -- 172.21.15.44:0/4012751667 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f64e008bb80 con 0x7f64e80ca7c0 2024-04-02T13:20:51.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.246+0000 7f64e77fe640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f64d406c920 0x7f64d406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.246+0000 7f64e77fe640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f64d406c920 0x7f64d406ede0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f64e80cf650 tx=0x7f64d8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:51.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.250+0000 7f64cffff640 1 -- 172.21.15.44:0/4012751667 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f64e81f4d60 con 0x7f64e80ca7c0 2024-04-02T13:20:51.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.434+0000 7f64edc93640 1 -- 172.21.15.44:0/4012751667 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64e80b2eb0 con 0x7f64e80ca7c0 2024-04-02T13:20:51.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.435+0000 7f64cffff640 1 -- 172.21.15.44:0/4012751667 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f64e0059c70 con 0x7f64e80ca7c0 2024-04-02T13:20:51.437 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:51.437 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.437+0000 7f64edc93640 1 -- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f64d406c920 msgr2=0x7f64d406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.437+0000 7f64edc93640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f64d406c920 0x7f64d406ede0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f64e80cf650 tx=0x7f64d8008040 comp rx=0 tx=0).stop 2024-04-02T13:20:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.437+0000 7f64edc93640 1 -- 172.21.15.44:0/4012751667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64e80ca7c0 msgr2=0x7f64e80cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.437+0000 7f64edc93640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64e80ca7c0 0x7f64e80cd210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f64e80b6d00 tx=0x7f64e000eb20 comp rx=0 tx=0).stop 2024-04-02T13:20:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.437+0000 7f64edc93640 1 -- 172.21.15.44:0/4012751667 shutdown_connections 2024-04-02T13:20:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.437+0000 7f64edc93640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f64d406c920 0x7f64d406ede0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.437+0000 7f64edc93640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64e80ca7c0 0x7f64e80cd210 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.437+0000 7f64edc93640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e80c7cc0 0x7f64e81f4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.437+0000 7f64edc93640 1 --2- 172.21.15.44:0/4012751667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d0004800 0x7f64e8151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.437+0000 7f64edc93640 1 -- 172.21.15.44:0/4012751667 >> 172.21.15.44:0/4012751667 conn(0x7f64e809d410 msgr2=0x7f64e815d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.437+0000 7f64edc93640 1 -- 172.21.15.44:0/4012751667 shutdown_connections 2024-04-02T13:20:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.437+0000 7f64edc93640 1 -- 172.21.15.44:0/4012751667 wait complete. 2024-04-02T13:20:51.451 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:20:51.451 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:20:51.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.553+0000 7f36e9b3d640 1 Processor -- start 2024-04-02T13:20:51.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.554+0000 7f36e9b3d640 1 -- start start 2024-04-02T13:20:51.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.554+0000 7f36e9b3d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36e40d2090 0x7f36e406f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.554+0000 7f36e9b3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36e415e4d0 0x7f36e406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.554+0000 7f36e9b3d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36e4070040 con 0x7f36e40d00d0 2024-04-02T13:20:51.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.554+0000 7f36e9b3d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f36e406c5e0 con 0x7f36e40d2090 2024-04-02T13:20:51.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.554+0000 7f36e9b3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f36e4070320 con 0x7f36e415e4d0 2024-04-02T13:20:51.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.555+0000 7f36e37fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36e40d2090 0x7f36e406f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.555+0000 7f36e3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36e415e4d0 0x7f36e406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.555+0000 7f36e37fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36e40d2090 0x7f36e406f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34666/0 (socket says 172.21.15.44:34666) 2024-04-02T13:20:51.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.555+0000 7f36e37fe640 1 -- 172.21.15.44:0/120758324 learned_addr learned my addr 172.21.15.44:0/120758324 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:51.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.555+0000 7f36e3fff640 1 --2- 172.21.15.44:0/120758324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36e415e4d0 0x7f36e406cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:42410/0 (socket says 172.21.15.44:42410) 2024-04-02T13:20:51.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.555+0000 7f36e37fe640 1 -- 172.21.15.44:0/120758324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36e415e4d0 msgr2=0x7f36e406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:51.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.555+0000 7f36e37fe640 1 --2- 172.21.15.44:0/120758324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36e415e4d0 0x7f36e406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.555+0000 7f36e37fe640 1 -- 172.21.15.44:0/120758324 >> v1:172.21.15.69:6789/0 conn(0x7f36e40d00d0 legacy=0x7f36e415ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.555+0000 7f36e37fe640 1 -- 172.21.15.44:0/120758324 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e4070600 con 0x7f36e40d2090 2024-04-02T13:20:51.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.555+0000 7f36e27fc640 1 -- 172.21.15.44:0/120758324 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 264584329 0 0) 0x7f36e4070040 con 0x7f36e40d00d0 2024-04-02T13:20:51.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.556+0000 7f36e37fe640 1 --2- 172.21.15.44:0/120758324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36e40d2090 0x7f36e406f580 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f36d4009870 tx=0x7f36d402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16f12b7c0af75f07 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:51.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.556+0000 7f36e27fc640 1 -- 172.21.15.44:0/120758324 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d4034070 con 0x7f36e40d2090 2024-04-02T13:20:51.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.556+0000 7f36e27fc640 1 -- 172.21.15.44:0/120758324 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36d4030840 con 0x7f36e40d2090 2024-04-02T13:20:51.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.556+0000 7f36e27fc640 1 -- 172.21.15.44:0/120758324 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d4030c10 con 0x7f36e40d2090 2024-04-02T13:20:51.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.556+0000 7f36e9b3d640 1 -- 172.21.15.44:0/120758324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36e40d2090 msgr2=0x7f36e406f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.556+0000 7f36e9b3d640 1 --2- 172.21.15.44:0/120758324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36e40d2090 0x7f36e406f580 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f36d4009870 tx=0x7f36d402b980 comp rx=0 tx=0).stop 2024-04-02T13:20:51.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.556+0000 7f36e9b3d640 1 -- 172.21.15.44:0/120758324 shutdown_connections 2024-04-02T13:20:51.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.556+0000 7f36e9b3d640 1 --2- 172.21.15.44:0/120758324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36e415e4d0 0x7f36e406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.556+0000 7f36e9b3d640 1 --2- 172.21.15.44:0/120758324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36e40d2090 0x7f36e406f580 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.557+0000 7f36e9b3d640 1 -- 172.21.15.44:0/120758324 >> 172.21.15.44:0/120758324 conn(0x7f36e409aa20 msgr2=0x7f36e409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:51.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.557+0000 7f36e9b3d640 1 -- 172.21.15.44:0/120758324 shutdown_connections 2024-04-02T13:20:51.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.557+0000 7f36e9b3d640 1 -- 172.21.15.44:0/120758324 wait complete. 2024-04-02T13:20:51.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.558+0000 7f36e9b3d640 1 Processor -- start 2024-04-02T13:20:51.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.558+0000 7f36e9b3d640 1 -- start start 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e9b3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36e40d00d0 0x7f36e415e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e9b3d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36e40d2090 0x7f36e40b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e9b3d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36e415e4d0 0x7f36e4154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e9b3d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f36e40cead0 con 0x7f36e415e4d0 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e9b3d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f36e40ce950 con 0x7f36e40d2090 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e9b3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f36e406fac0 con 0x7f36e40d00d0 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36e415e4d0 0x7f36e4154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36e415e4d0 0x7f36e4154af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55250/0 (socket says 172.21.15.44:55250) 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e3fff640 1 -- 172.21.15.44:0/3249796274 learned_addr learned my addr 172.21.15.44:0/3249796274 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e2ffd640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36e40d2090 0x7f36e40b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e37fe640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36e40d00d0 0x7f36e415e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e3fff640 1 -- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36e40d00d0 msgr2=0x7f36e415e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e3fff640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36e40d00d0 0x7f36e415e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e3fff640 1 -- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36e40d2090 msgr2=0x7f36e40b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e3fff640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36e40d2090 0x7f36e40b0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e3fff640 1 -- 172.21.15.44:0/3249796274 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e415eeb0 con 0x7f36e415e4d0 2024-04-02T13:20:51.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e2ffd640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36e40d2090 0x7f36e40b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:51.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e37fe640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36e40d00d0 0x7f36e415e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:51.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e3fff640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36e415e4d0 0x7f36e4154af0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00efc0 tx=0x7f36dc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:51.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36cbfff640 1 -- 172.21.15.44:0/3249796274 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc018020 con 0x7f36e415e4d0 2024-04-02T13:20:51.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36cbfff640 1 -- 172.21.15.44:0/3249796274 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36dc014480 con 0x7f36e415e4d0 2024-04-02T13:20:51.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36cbfff640 1 -- 172.21.15.44:0/3249796274 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc009e60 con 0x7f36e415e4d0 2024-04-02T13:20:51.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.559+0000 7f36e9b3d640 1 -- 172.21.15.44:0/3249796274 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e415f130 con 0x7f36e415e4d0 2024-04-02T13:20:51.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.560+0000 7f36e9b3d640 1 -- 172.21.15.44:0/3249796274 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e415f6f0 con 0x7f36e415e4d0 2024-04-02T13:20:51.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.560+0000 7f36e9b3d640 1 -- 172.21.15.44:0/3249796274 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36e4068c00 con 0x7f36e415e4d0 2024-04-02T13:20:51.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.561+0000 7f36cbfff640 1 -- 172.21.15.44:0/3249796274 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f36dc005d10 con 0x7f36e415e4d0 2024-04-02T13:20:51.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.561+0000 7f36cbfff640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36d006c920 0x7f36d006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.561+0000 7f36cbfff640 1 -- 172.21.15.44:0/3249796274 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f36dc08cb20 con 0x7f36e415e4d0 2024-04-02T13:20:51.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.562+0000 7f36e37fe640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36d006c920 0x7f36d006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.562+0000 7f36e37fe640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36d006c920 0x7f36d006ede0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f36d4036000 tx=0x7f36d403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:51.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.566+0000 7f36cbfff640 1 -- 172.21.15.44:0/3249796274 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f36dc022030 con 0x7f36e415e4d0 2024-04-02T13:20:51.700 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:20:51.702 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:20:51.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.749+0000 7f36e9b3d640 1 -- 172.21.15.44:0/3249796274 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f36e40d2b90 con 0x7f36e415e4d0 2024-04-02T13:20:51.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.749+0000 7f36cbfff640 1 -- 172.21.15.44:0/3249796274 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f36e415eeb0 con 0x7f36e415e4d0 2024-04-02T13:20:51.751 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:20:51.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.751+0000 7f36e9b3d640 1 -- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36d006c920 msgr2=0x7f36d006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.751+0000 7f36e9b3d640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36d006c920 0x7f36d006ede0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f36d4036000 tx=0x7f36d403b040 comp rx=0 tx=0).stop 2024-04-02T13:20:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.751+0000 7f36e9b3d640 1 -- 172.21.15.44:0/3249796274 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36e415e4d0 msgr2=0x7f36e4154af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.751+0000 7f36e9b3d640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36e415e4d0 0x7f36e4154af0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00efc0 tx=0x7f36dc00c490 comp rx=0 tx=0).stop 2024-04-02T13:20:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.751+0000 7f36e9b3d640 1 -- 172.21.15.44:0/3249796274 shutdown_connections 2024-04-02T13:20:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.751+0000 7f36e9b3d640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36d006c920 0x7f36d006ede0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.751+0000 7f36e9b3d640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36e415e4d0 0x7f36e4154af0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.751+0000 7f36e9b3d640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36e40d2090 0x7f36e40b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.751+0000 7f36e9b3d640 1 --2- 172.21.15.44:0/3249796274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36e40d00d0 0x7f36e415e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.751+0000 7f36e9b3d640 1 -- 172.21.15.44:0/3249796274 >> 172.21.15.44:0/3249796274 conn(0x7f36e409aa20 msgr2=0x7f36e409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.751+0000 7f36e9b3d640 1 -- 172.21.15.44:0/3249796274 shutdown_connections 2024-04-02T13:20:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.751+0000 7f36e9b3d640 1 -- 172.21.15.44:0/3249796274 wait complete. 2024-04-02T13:20:51.764 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:20:51.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.907+0000 7f23beb60640 1 Processor -- start 2024-04-02T13:20:51.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.907+0000 7f23beb60640 1 -- start start 2024-04-02T13:20:51.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.908+0000 7f23beb60640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f23b80cd770 0x7f23b806eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.908+0000 7f23beb60640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f23b80ceec0 0x7f23b80cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.908+0000 7f23beb60640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23b806f3f0 con 0x7f23b806b4d0 2024-04-02T13:20:51.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.908+0000 7f23beb60640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f23b806f6d0 con 0x7f23b80ceec0 2024-04-02T13:20:51.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.908+0000 7f23beb60640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f23b806f9b0 con 0x7f23b80cd770 2024-04-02T13:20:51.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.908+0000 7f23bd0fd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f23b80ceec0 0x7f23b80cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.908+0000 7f23bd0fd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f23b80ceec0 0x7f23b80cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55256/0 (socket says 172.21.15.44:55256) 2024-04-02T13:20:51.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.908+0000 7f23bd0fd640 1 -- 172.21.15.44:0/2366798234 learned_addr learned my addr 172.21.15.44:0/2366798234 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:51.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.908+0000 7f23b7fff640 1 --2- 172.21.15.44:0/2366798234 >> v2:172.21.15.69:3300/0 conn(0x7f23b80cd770 0x7f23b806eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.908+0000 7f23bd0fd640 1 -- 172.21.15.44:0/2366798234 >> v2:172.21.15.69:3300/0 conn(0x7f23b80cd770 msgr2=0x7f23b806eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:51.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.908+0000 7f23bd0fd640 1 --2- 172.21.15.44:0/2366798234 >> v2:172.21.15.69:3300/0 conn(0x7f23b80cd770 0x7f23b806eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.908+0000 7f23bd0fd640 1 -- 172.21.15.44:0/2366798234 >> v1:172.21.15.69:6789/0 conn(0x7f23b806b4d0 legacy=0x7f23b806b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:51.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.908+0000 7f23bd0fd640 1 -- 172.21.15.44:0/2366798234 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b806fc90 con 0x7f23b80ceec0 2024-04-02T13:20:51.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.909+0000 7f23bd0fd640 1 --2- 172.21.15.44:0/2366798234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23b80ceec0 0x7f23b80cf310 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f23b8075360 tx=0x7f23b002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42fabbf82aab88fd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:51.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.909+0000 7f23b77fe640 1 -- 172.21.15.44:0/2366798234 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b002ca80 con 0x7f23b80ceec0 2024-04-02T13:20:51.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.909+0000 7f23b77fe640 1 -- 172.21.15.44:0/2366798234 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23b002cc20 con 0x7f23b80ceec0 2024-04-02T13:20:51.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.909+0000 7f23b77fe640 1 -- 172.21.15.44:0/2366798234 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b00343f0 con 0x7f23b80ceec0 2024-04-02T13:20:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.909+0000 7f23beb60640 1 -- 172.21.15.44:0/2366798234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23b80ceec0 msgr2=0x7f23b80cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.909+0000 7f23beb60640 1 --2- 172.21.15.44:0/2366798234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23b80ceec0 0x7f23b80cf310 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f23b8075360 tx=0x7f23b002bee0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.909+0000 7f23beb60640 1 -- 172.21.15.44:0/2366798234 shutdown_connections 2024-04-02T13:20:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.909+0000 7f23beb60640 1 --2- 172.21.15.44:0/2366798234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23b80ceec0 0x7f23b80cf310 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.909+0000 7f23beb60640 1 --2- 172.21.15.44:0/2366798234 >> v2:172.21.15.69:3300/0 conn(0x7f23b80cd770 0x7f23b806eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.909+0000 7f23beb60640 1 -- 172.21.15.44:0/2366798234 >> 172.21.15.44:0/2366798234 conn(0x7f23b80b2a10 msgr2=0x7f23b80b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.910+0000 7f23beb60640 1 -- 172.21.15.44:0/2366798234 shutdown_connections 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.910+0000 7f23beb60640 1 -- 172.21.15.44:0/2366798234 wait complete. 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.911+0000 7f23beb60640 1 Processor -- start 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.911+0000 7f23beb60640 1 -- start start 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23beb60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23b806b4d0 0x7f23b8147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23beb60640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23b80cd770 0x7f23b815c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23beb60640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23b80ceec0 0x7f23b80b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23beb60640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f23b815c470 con 0x7f23b80cd770 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23beb60640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f23b806f0e0 con 0x7f23b80ceec0 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23beb60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f23b815c5f0 con 0x7f23b806b4d0 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23b7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23b80cd770 0x7f23b815c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23b7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23b80cd770 0x7f23b815c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55258/0 (socket says 172.21.15.44:55258) 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23b7fff640 1 -- 172.21.15.44:0/3821965203 learned_addr learned my addr 172.21.15.44:0/3821965203 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23bd0fd640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23b80ceec0 0x7f23b80b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23bc8fc640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23b806b4d0 0x7f23b8147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23b7fff640 1 -- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23b806b4d0 msgr2=0x7f23b8147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23b7fff640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23b806b4d0 0x7f23b8147890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23b7fff640 1 -- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23b80ceec0 msgr2=0x7f23b80b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23b7fff640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23b80ceec0 0x7f23b80b1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23b7fff640 1 -- 172.21.15.44:0/3821965203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b815d0e0 con 0x7f23b80cd770 2024-04-02T13:20:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23b7fff640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23b80cd770 0x7f23b815c770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f23a400c7f0 tx=0x7f23a400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23b57fa640 1 -- 172.21.15.44:0/3821965203 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a4015070 con 0x7f23b80cd770 2024-04-02T13:20:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23b57fa640 1 -- 172.21.15.44:0/3821965203 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23a4004240 con 0x7f23b80cd770 2024-04-02T13:20:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.912+0000 7f23b57fa640 1 -- 172.21.15.44:0/3821965203 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a4007620 con 0x7f23b80cd770 2024-04-02T13:20:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.913+0000 7f23beb60640 1 -- 172.21.15.44:0/3821965203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23b815d3c0 con 0x7f23b80cd770 2024-04-02T13:20:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.913+0000 7f23beb60640 1 -- 172.21.15.44:0/3821965203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23b815d980 con 0x7f23b80cd770 2024-04-02T13:20:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.914+0000 7f23beb60640 1 -- 172.21.15.44:0/3821965203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23b815dd60 con 0x7f23b80cd770 2024-04-02T13:20:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.914+0000 7f23b57fa640 1 -- 172.21.15.44:0/3821965203 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f23a40079d0 con 0x7f23b80cd770 2024-04-02T13:20:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.914+0000 7f23b57fa640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f239c06c8f0 0x7f239c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.915+0000 7f23b57fa640 1 -- 172.21.15.44:0/3821965203 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f23a408c280 con 0x7f23b80cd770 2024-04-02T13:20:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.915+0000 7f23bc8fc640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f239c06c8f0 0x7f239c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:51.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.915+0000 7f23bc8fc640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f239c06c8f0 0x7f239c06edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0017f0 tx=0x7f23ac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:51.919+0000 7f23b57fa640 1 -- 172.21.15.44:0/3821965203 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f23b815dd60 con 0x7f23b80cd770 2024-04-02T13:20:52.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.105+0000 7f23beb60640 1 -- 172.21.15.44:0/3821965203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f23b80684f0 con 0x7f23b80cd770 2024-04-02T13:20:52.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.105+0000 7f23b57fa640 1 -- 172.21.15.44:0/3821965203 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f23a4059d10 con 0x7f23b80cd770 2024-04-02T13:20:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:20:52.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.107+0000 7f23beb60640 1 -- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f239c06c8f0 msgr2=0x7f239c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.107+0000 7f23beb60640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f239c06c8f0 0x7f239c06edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0017f0 tx=0x7f23ac009040 comp rx=0 tx=0).stop 2024-04-02T13:20:52.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.107+0000 7f23beb60640 1 -- 172.21.15.44:0/3821965203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23b80cd770 msgr2=0x7f23b815c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.107+0000 7f23beb60640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23b80cd770 0x7f23b815c770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f23a400c7f0 tx=0x7f23a400ccb0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.107+0000 7f23beb60640 1 -- 172.21.15.44:0/3821965203 shutdown_connections 2024-04-02T13:20:52.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.107+0000 7f23beb60640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f239c06c8f0 0x7f239c06edb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.107+0000 7f23beb60640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23b80ceec0 0x7f23b80b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.107+0000 7f23beb60640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23b80cd770 0x7f23b815c770 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.107+0000 7f23beb60640 1 --2- 172.21.15.44:0/3821965203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23b806b4d0 0x7f23b8147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.107+0000 7f23beb60640 1 -- 172.21.15.44:0/3821965203 >> 172.21.15.44:0/3821965203 conn(0x7f23b80b2a10 msgr2=0x7f23b80b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:52.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.107+0000 7f23beb60640 1 -- 172.21.15.44:0/3821965203 shutdown_connections 2024-04-02T13:20:52.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.107+0000 7f23beb60640 1 -- 172.21.15.44:0/3821965203 wait complete. 2024-04-02T13:20:52.120 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:20:52.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.222+0000 7f10fefa8640 1 Processor -- start 2024-04-02T13:20:52.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.223+0000 7f10fefa8640 1 -- start start 2024-04-02T13:20:52.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.223+0000 7f10fefa8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80d21c0 0x7f10f806f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.223+0000 7f10fefa8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f10f80cfd60 0x7f10f80cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.223+0000 7f10fefa8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10f815c580 con 0x7f10f815c1a0 2024-04-02T13:20:52.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.223+0000 7f10fefa8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f10f8070ad0 con 0x7f10f80cfd60 2024-04-02T13:20:52.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.223+0000 7f10fefa8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f10f80d01f0 con 0x7f10f80d21c0 2024-04-02T13:20:52.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.223+0000 7f10fd545640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f10f80cfd60 0x7f10f80cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.223+0000 7f10fcd44640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f10f815c1a0 0x7f10f80d1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57576/0 (socket says 172.21.15.44:57576) 2024-04-02T13:20:52.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.223+0000 7f10fcd44640 1 -- 172.21.15.44:0/3277871577 learned_addr learned my addr 172.21.15.44:0/3277871577 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:52.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.224+0000 7f10f7fff640 1 --2- 172.21.15.44:0/3277871577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80d21c0 0x7f10f806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.224+0000 7f10fd545640 1 -- 172.21.15.44:0/3277871577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80d21c0 msgr2=0x7f10f806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:52.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.224+0000 7f10fd545640 1 --2- 172.21.15.44:0/3277871577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80d21c0 0x7f10f806f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.224+0000 7f10fd545640 1 -- 172.21.15.44:0/3277871577 >> v1:172.21.15.44:6789/0 conn(0x7f10f815c1a0 legacy=0x7f10f80d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.224+0000 7f10f77fe640 1 -- 172.21.15.44:0/3277871577 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4005600288 0 0) 0x7f10f815c580 con 0x7f10f815c1a0 2024-04-02T13:20:52.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.224+0000 7f10fd545640 1 -- 172.21.15.44:0/3277871577 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f806fa30 con 0x7f10f80cfd60 2024-04-02T13:20:52.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.224+0000 7f10fd545640 1 --2- 172.21.15.44:0/3277871577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f80cfd60 0x7f10f80cc9a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009dc0 tx=0x7f10f002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda4a325b116bcc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:52.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.225+0000 7f10f77fe640 1 -- 172.21.15.44:0/3277871577 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0039070 con 0x7f10f80cfd60 2024-04-02T13:20:52.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.225+0000 7f10f77fe640 1 -- 172.21.15.44:0/3277871577 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f002cc40 con 0x7f10f80cfd60 2024-04-02T13:20:52.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.225+0000 7f10f77fe640 1 -- 172.21.15.44:0/3277871577 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0034440 con 0x7f10f80cfd60 2024-04-02T13:20:52.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.225+0000 7f10fefa8640 1 -- 172.21.15.44:0/3277871577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f80cfd60 msgr2=0x7f10f80cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.225+0000 7f10fefa8640 1 --2- 172.21.15.44:0/3277871577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f80cfd60 0x7f10f80cc9a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009dc0 tx=0x7f10f002be90 comp rx=0 tx=0).stop 2024-04-02T13:20:52.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.225+0000 7f10fefa8640 1 -- 172.21.15.44:0/3277871577 shutdown_connections 2024-04-02T13:20:52.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.225+0000 7f10fefa8640 1 --2- 172.21.15.44:0/3277871577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f80cfd60 0x7f10f80cc9a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.225+0000 7f10fefa8640 1 --2- 172.21.15.44:0/3277871577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80d21c0 0x7f10f806f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.225+0000 7f10fefa8640 1 -- 172.21.15.44:0/3277871577 >> 172.21.15.44:0/3277871577 conn(0x7f10f809aa20 msgr2=0x7f10f809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:52.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.225+0000 7f10fefa8640 1 -- 172.21.15.44:0/3277871577 shutdown_connections 2024-04-02T13:20:52.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.225+0000 7f10fefa8640 1 -- 172.21.15.44:0/3277871577 wait complete. 2024-04-02T13:20:52.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.226+0000 7f10fefa8640 1 Processor -- start 2024-04-02T13:20:52.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.226+0000 7f10fefa8640 1 -- start start 2024-04-02T13:20:52.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.227+0000 7f10fefa8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f80cfd60 0x7f10f815e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.227+0000 7f10fefa8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80d21c0 0x7f10f80ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.227+0000 7f10fefa8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f815c1a0 0x7f10f814fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.227+0000 7f10fefa8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f10f80702d0 con 0x7f10f815c1a0 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.227+0000 7f10fefa8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f10f806fff0 con 0x7f10f80cfd60 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.227+0000 7f10fefa8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f10f80705b0 con 0x7f10f80d21c0 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.227+0000 7f10fd545640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f815c1a0 0x7f10f814fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.227+0000 7f10fd545640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f815c1a0 0x7f10f814fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55290/0 (socket says 172.21.15.44:55290) 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.227+0000 7f10fd545640 1 -- 172.21.15.44:0/3886346072 learned_addr learned my addr 172.21.15.44:0/3886346072 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.227+0000 7f10fcd44640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f80cfd60 0x7f10f815e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.227+0000 7f10f7fff640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80d21c0 0x7f10f80ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.228+0000 7f10fd545640 1 -- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80d21c0 msgr2=0x7f10f80ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.228+0000 7f10fd545640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80d21c0 0x7f10f80ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.228+0000 7f10fd545640 1 -- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f80cfd60 msgr2=0x7f10f815e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.228+0000 7f10fd545640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f80cfd60 0x7f10f815e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.228+0000 7f10fd545640 1 -- 172.21.15.44:0/3886346072 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f815ea10 con 0x7f10f815c1a0 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.228+0000 7f10fd545640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f815c1a0 0x7f10f814fc50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f10f0037040 tx=0x7f10f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.228+0000 7f10f57fa640 1 -- 172.21.15.44:0/3886346072 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0044020 con 0x7f10f815c1a0 2024-04-02T13:20:52.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.228+0000 7f10f57fa640 1 -- 172.21.15.44:0/3886346072 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f0034d40 con 0x7f10f815c1a0 2024-04-02T13:20:52.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.228+0000 7f10fefa8640 1 -- 172.21.15.44:0/3886346072 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f815ec90 con 0x7f10f815c1a0 2024-04-02T13:20:52.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.228+0000 7f10f57fa640 1 -- 172.21.15.44:0/3886346072 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0039040 con 0x7f10f815c1a0 2024-04-02T13:20:52.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.228+0000 7f10fefa8640 1 -- 172.21.15.44:0/3886346072 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f81f26b0 con 0x7f10f815c1a0 2024-04-02T13:20:52.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.229+0000 7f10fefa8640 1 -- 172.21.15.44:0/3886346072 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10f8068c00 con 0x7f10f815c1a0 2024-04-02T13:20:52.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.229+0000 7f10f57fa640 1 -- 172.21.15.44:0/3886346072 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f10f0047020 con 0x7f10f815c1a0 2024-04-02T13:20:52.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.230+0000 7f10f57fa640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f10dc06c920 0x7f10dc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.230+0000 7f10f57fa640 1 -- 172.21.15.44:0/3886346072 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f10f00ae300 con 0x7f10f815c1a0 2024-04-02T13:20:52.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.230+0000 7f10fcd44640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f10dc06c920 0x7f10dc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.231+0000 7f10fcd44640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f10dc06c920 0x7f10dc06ede0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f10ec007fd0 tx=0x7f10ec007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:52.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.234+0000 7f10f57fa640 1 -- 172.21.15.44:0/3886346072 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f10f004e070 con 0x7f10f815c1a0 2024-04-02T13:20:52.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.418+0000 7f10fefa8640 1 -- 172.21.15.44:0/3886346072 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f10f80b2610 con 0x7f10f815c1a0 2024-04-02T13:20:52.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.419+0000 7f10f57fa640 1 -- 172.21.15.44:0/3886346072 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f10f815ea10 con 0x7f10f815c1a0 2024-04-02T13:20:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:20:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.420+0000 7f10fefa8640 1 -- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f10dc06c920 msgr2=0x7f10dc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.420+0000 7f10fefa8640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f10dc06c920 0x7f10dc06ede0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f10ec007fd0 tx=0x7f10ec007de0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.420+0000 7f10fefa8640 1 -- 172.21.15.44:0/3886346072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f815c1a0 msgr2=0x7f10f814fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.420+0000 7f10fefa8640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f815c1a0 0x7f10f814fc50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f10f0037040 tx=0x7f10f0004a40 comp rx=0 tx=0).stop 2024-04-02T13:20:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.421+0000 7f10fefa8640 1 -- 172.21.15.44:0/3886346072 shutdown_connections 2024-04-02T13:20:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.421+0000 7f10fefa8640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f10dc06c920 0x7f10dc06ede0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.421+0000 7f10fefa8640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f815c1a0 0x7f10f814fc50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.421+0000 7f10fefa8640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80d21c0 0x7f10f80ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.421+0000 7f10fefa8640 1 --2- 172.21.15.44:0/3886346072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f80cfd60 0x7f10f815e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.421+0000 7f10fefa8640 1 -- 172.21.15.44:0/3886346072 >> 172.21.15.44:0/3886346072 conn(0x7f10f809aa20 msgr2=0x7f10f80b0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:52.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.421+0000 7f10fefa8640 1 -- 172.21.15.44:0/3886346072 shutdown_connections 2024-04-02T13:20:52.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.421+0000 7f10fefa8640 1 -- 172.21.15.44:0/3886346072 wait complete. 2024-04-02T13:20:52.434 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:20:52.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.535+0000 7f3dee140640 1 Processor -- start 2024-04-02T13:20:52.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.535+0000 7f3dee140640 1 -- start start 2024-04-02T13:20:52.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.536+0000 7f3dee140640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80c6b40 0x7f3de8166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.536+0000 7f3dee140640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3de80cd760 con 0x7f3de80b4050 2024-04-02T13:20:52.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.536+0000 7f3dee140640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3de80cda40 con 0x7f3de815c260 2024-04-02T13:20:52.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.536+0000 7f3dee140640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3de80cdd20 con 0x7f3de80c6b40 2024-04-02T13:20:52.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.536+0000 7f3de6ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f3de80b4050 0x7f3de80ccaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57582/0 (socket says 172.21.15.44:57582) 2024-04-02T13:20:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.536+0000 7f3de6ffd640 1 -- 172.21.15.44:0/1352730291 learned_addr learned my addr 172.21.15.44:0/1352730291 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.536+0000 7f3de7fff640 1 --2- 172.21.15.44:0/1352730291 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80c6b40 0x7f3de8166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.537+0000 7f3de67fc640 1 -- 172.21.15.44:0/1352730291 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2745058423 0 0) 0x7f3de80cd760 con 0x7f3de80b4050 2024-04-02T13:20:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.537+0000 7f3de67fc640 1 -- 172.21.15.44:0/1352730291 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dd0003520 con 0x7f3de80b4050 2024-04-02T13:20:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.537+0000 7f3de67fc640 1 -- 172.21.15.44:0/1352730291 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4049911497 0 0) 0x7f3de80cda40 con 0x7f3de815c260 2024-04-02T13:20:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.537+0000 7f3de67fc640 1 -- 172.21.15.44:0/1352730291 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3de80cd760 con 0x7f3de815c260 2024-04-02T13:20:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.537+0000 7f3de7fff640 1 -- 172.21.15.44:0/1352730291 >> v1:172.21.15.69:6789/0 conn(0x7f3de815c260 legacy=0x7f3de80b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.537+0000 7f3de7fff640 1 -- 172.21.15.44:0/1352730291 >> v1:172.21.15.44:6789/0 conn(0x7f3de80b4050 legacy=0x7f3de80ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.537+0000 7f3de7fff640 1 -- 172.21.15.44:0/1352730291 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de80ce000 con 0x7f3de80c6b40 2024-04-02T13:20:52.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.538+0000 7f3de7fff640 1 --2- 172.21.15.44:0/1352730291 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80c6b40 0x7f3de8166cf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3de0009dc0 tx=0x7f3de002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f65ec9bfb8afd6c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.538+0000 7f3de67fc640 1 -- 172.21.15.44:0/1352730291 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de0039070 con 0x7f3de80c6b40 2024-04-02T13:20:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.538+0000 7f3de67fc640 1 -- 172.21.15.44:0/1352730291 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3de002ccd0 con 0x7f3de80c6b40 2024-04-02T13:20:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.538+0000 7f3de67fc640 1 -- 172.21.15.44:0/1352730291 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de0034640 con 0x7f3de80c6b40 2024-04-02T13:20:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.538+0000 7f3dee140640 1 -- 172.21.15.44:0/1352730291 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80c6b40 msgr2=0x7f3de8166cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.538+0000 7f3dee140640 1 --2- 172.21.15.44:0/1352730291 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80c6b40 0x7f3de8166cf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3de0009dc0 tx=0x7f3de002bf20 comp rx=0 tx=0).stop 2024-04-02T13:20:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.538+0000 7f3dee140640 1 -- 172.21.15.44:0/1352730291 shutdown_connections 2024-04-02T13:20:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.538+0000 7f3dee140640 1 --2- 172.21.15.44:0/1352730291 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80c6b40 0x7f3de8166cf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.538+0000 7f3dee140640 1 -- 172.21.15.44:0/1352730291 >> 172.21.15.44:0/1352730291 conn(0x7f3de809aa20 msgr2=0x7f3de809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.538+0000 7f3dee140640 1 -- 172.21.15.44:0/1352730291 shutdown_connections 2024-04-02T13:20:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.538+0000 7f3dee140640 1 -- 172.21.15.44:0/1352730291 wait complete. 2024-04-02T13:20:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.540+0000 7f3dee140640 1 Processor -- start 2024-04-02T13:20:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.540+0000 7f3dee140640 1 -- start start 2024-04-02T13:20:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.540+0000 7f3dee140640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80b4050 0x7f3de80ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.540+0000 7f3dee140640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3de80c6b40 0x7f3de80c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.540+0000 7f3dee140640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3de815c260 0x7f3de814a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.540+0000 7f3dee140640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3de80cd380 con 0x7f3de80c6b40 2024-04-02T13:20:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.540+0000 7f3dee140640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3de80cd200 con 0x7f3de815c260 2024-04-02T13:20:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.540+0000 7f3dee140640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3de80cd500 con 0x7f3de80b4050 2024-04-02T13:20:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3de6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3de80c6b40 0x7f3de80c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3de6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3de80c6b40 0x7f3de80c90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55298/0 (socket says 172.21.15.44:55298) 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3de6ffd640 1 -- 172.21.15.44:0/4000387112 learned_addr learned my addr 172.21.15.44:0/4000387112 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3de77fe640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80b4050 0x7f3de80ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3de7fff640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3de815c260 0x7f3de814a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3de6ffd640 1 -- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80b4050 msgr2=0x7f3de80ca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3de6ffd640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80b4050 0x7f3de80ca9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3de6ffd640 1 -- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3de815c260 msgr2=0x7f3de814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3de6ffd640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3de815c260 0x7f3de814a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3de6ffd640 1 -- 172.21.15.44:0/4000387112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de80c9900 con 0x7f3de80c6b40 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3de7fff640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3de815c260 0x7f3de814a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3de77fe640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80b4050 0x7f3de80ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3de6ffd640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3de80c6b40 0x7f3de80c90b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc00c500 tx=0x7f3ddc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3dcffff640 1 -- 172.21.15.44:0/4000387112 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc017070 con 0x7f3de80c6b40 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3dcffff640 1 -- 172.21.15.44:0/4000387112 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ddc005ce0 con 0x7f3de80c6b40 2024-04-02T13:20:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3dee140640 1 -- 172.21.15.44:0/4000387112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de80c9be0 con 0x7f3de80c6b40 2024-04-02T13:20:52.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3dcffff640 1 -- 172.21.15.44:0/4000387112 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc0072f0 con 0x7f3de80c6b40 2024-04-02T13:20:52.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.541+0000 7f3dee140640 1 -- 172.21.15.44:0/4000387112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de80ca1a0 con 0x7f3de80c6b40 2024-04-02T13:20:52.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.542+0000 7f3dee140640 1 -- 172.21.15.44:0/4000387112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de80684f0 con 0x7f3de80c6b40 2024-04-02T13:20:52.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.543+0000 7f3dcffff640 1 -- 172.21.15.44:0/4000387112 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3ddc01a070 con 0x7f3de80c6b40 2024-04-02T13:20:52.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.543+0000 7f3dcffff640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3dd406c8f0 0x7f3dd406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.543+0000 7f3dcffff640 1 -- 172.21.15.44:0/4000387112 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3ddc08b1a0 con 0x7f3de80c6b40 2024-04-02T13:20:52.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.544+0000 7f3de77fe640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3dd406c8f0 0x7f3dd406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.544+0000 7f3de77fe640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3dd406c8f0 0x7f3dd406edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8001610 tx=0x7f3dd8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:52.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.548+0000 7f3dcffff640 1 -- 172.21.15.44:0/4000387112 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3ddc058cd0 con 0x7f3de80c6b40 2024-04-02T13:20:52.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.732+0000 7f3dee140640 1 -- 172.21.15.44:0/4000387112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3de80c7330 con 0x7f3de80c6b40 2024-04-02T13:20:52.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.732+0000 7f3dcffff640 1 -- 172.21.15.44:0/4000387112 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f3ddc05cfd0 con 0x7f3de80c6b40 2024-04-02T13:20:52.735 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:20:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.734+0000 7f3dee140640 1 -- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3dd406c8f0 msgr2=0x7f3dd406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.734+0000 7f3dee140640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3dd406c8f0 0x7f3dd406edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8001610 tx=0x7f3dd8009040 comp rx=0 tx=0).stop 2024-04-02T13:20:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.734+0000 7f3dee140640 1 -- 172.21.15.44:0/4000387112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3de80c6b40 msgr2=0x7f3de80c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.734+0000 7f3dee140640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3de80c6b40 0x7f3de80c90b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc00c500 tx=0x7f3ddc00c9c0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.734+0000 7f3dee140640 1 -- 172.21.15.44:0/4000387112 shutdown_connections 2024-04-02T13:20:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.734+0000 7f3dee140640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3dd406c8f0 0x7f3dd406edb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.734+0000 7f3dee140640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3de815c260 0x7f3de814a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.734+0000 7f3dee140640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3de80c6b40 0x7f3de80c90b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.734+0000 7f3dee140640 1 --2- 172.21.15.44:0/4000387112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80b4050 0x7f3de80ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.734+0000 7f3dee140640 1 -- 172.21.15.44:0/4000387112 >> 172.21.15.44:0/4000387112 conn(0x7f3de809aa20 msgr2=0x7f3de80c8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:52.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.734+0000 7f3dee140640 1 -- 172.21.15.44:0/4000387112 shutdown_connections 2024-04-02T13:20:52.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.734+0000 7f3dee140640 1 -- 172.21.15.44:0/4000387112 wait complete. 2024-04-02T13:20:52.747 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:20:52.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.847+0000 7f4aaef9f640 1 Processor -- start 2024-04-02T13:20:52.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.847+0000 7f4aaef9f640 1 -- start start 2024-04-02T13:20:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aaef9f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4aa815e430 0x7f4aa80d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aaef9f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa80d0f70 0x7f4aa80d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aaef9f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4aa80b3f50 0x7f4aa80b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aaef9f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4aa815e810 con 0x7f4aa815e430 2024-04-02T13:20:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aaef9f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4aa80b37e0 con 0x7f4aa80b3f50 2024-04-02T13:20:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aaef9f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa80d1e00 con 0x7f4aa80d0f70 2024-04-02T13:20:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aadf9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4aa815e430 0x7f4aa80d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aadf9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4aa815e430 0x7f4aa80d2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55308/0 (socket says 172.21.15.44:55308) 2024-04-02T13:20:52.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aadf9d640 1 -- 172.21.15.44:0/3233130660 learned_addr learned my addr 172.21.15.44:0/3233130660 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:52.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aae79e640 1 --2- 172.21.15.44:0/3233130660 >> v2:172.21.15.69:3300/0 conn(0x7f4aa80b3f50 0x7f4aa80b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aad79c640 1 --2- 172.21.15.44:0/3233130660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa80d0f70 0x7f4aa80d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aadf9d640 1 -- 172.21.15.44:0/3233130660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa80d0f70 msgr2=0x7f4aa80d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:52.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aadf9d640 1 --2- 172.21.15.44:0/3233130660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa80d0f70 0x7f4aa80d29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aadf9d640 1 -- 172.21.15.44:0/3233130660 >> v2:172.21.15.69:3300/0 conn(0x7f4aa80b3f50 msgr2=0x7f4aa80b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:52.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aadf9d640 1 --2- 172.21.15.44:0/3233130660 >> v2:172.21.15.69:3300/0 conn(0x7f4aa80b3f50 0x7f4aa80b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aadf9d640 1 -- 172.21.15.44:0/3233130660 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa80d20e0 con 0x7f4aa815e430 2024-04-02T13:20:52.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.848+0000 7f4aad79c640 1 --2- 172.21.15.44:0/3233130660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa80d0f70 0x7f4aa80d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:52.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.849+0000 7f4aadf9d640 1 --2- 172.21.15.44:0/3233130660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4aa815e430 0x7f4aa80d2460 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4aa80b9f00 tx=0x7f4aa402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980f55af4d49fac0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:52.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.849+0000 7f4aacf9b640 1 -- 172.21.15.44:0/3233130660 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4035020 con 0x7f4aa815e430 2024-04-02T13:20:52.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.849+0000 7f4aacf9b640 1 -- 172.21.15.44:0/3233130660 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4aa4031870 con 0x7f4aa815e430 2024-04-02T13:20:52.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.849+0000 7f4aacf9b640 1 -- 172.21.15.44:0/3233130660 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4031c40 con 0x7f4aa815e430 2024-04-02T13:20:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.850+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3233130660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4aa815e430 msgr2=0x7f4aa80d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.850+0000 7f4aaef9f640 1 --2- 172.21.15.44:0/3233130660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4aa815e430 0x7f4aa80d2460 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4aa80b9f00 tx=0x7f4aa402b910 comp rx=0 tx=0).stop 2024-04-02T13:20:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.850+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3233130660 shutdown_connections 2024-04-02T13:20:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.850+0000 7f4aaef9f640 1 --2- 172.21.15.44:0/3233130660 >> v2:172.21.15.69:3300/0 conn(0x7f4aa80b3f50 0x7f4aa80b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.850+0000 7f4aaef9f640 1 --2- 172.21.15.44:0/3233130660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa80d0f70 0x7f4aa80d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.850+0000 7f4aaef9f640 1 --2- 172.21.15.44:0/3233130660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4aa815e430 0x7f4aa80d2460 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.850+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3233130660 >> 172.21.15.44:0/3233130660 conn(0x7f4aa80d3e80 msgr2=0x7f4aa80d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.850+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3233130660 shutdown_connections 2024-04-02T13:20:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.850+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3233130660 wait complete. 2024-04-02T13:20:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.851+0000 7f4aaef9f640 1 Processor -- start 2024-04-02T13:20:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.851+0000 7f4aaef9f640 1 -- start start 2024-04-02T13:20:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.851+0000 7f4aaef9f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4aa80b3f50 0x7f4aa80c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.851+0000 7f4aaef9f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aa80d0f70 0x7f4aa81f7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.851+0000 7f4aaef9f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa815e430 0x7f4aa815fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.851+0000 7f4aaef9f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa80d4b20 con 0x7f4aa80b3f50 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.851+0000 7f4aaef9f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa80d49a0 con 0x7f4aa80d0f70 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.851+0000 7f4aaef9f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa80b4920 con 0x7f4aa815e430 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4aadf9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4aa80b3f50 0x7f4aa80c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4aadf9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4aa80b3f50 0x7f4aa80c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55320/0 (socket says 172.21.15.44:55320) 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4aadf9d640 1 -- 172.21.15.44:0/3755321172 learned_addr learned my addr 172.21.15.44:0/3755321172 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4aadf9d640 1 -- 172.21.15.44:0/3755321172 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa815e430 msgr2=0x7f4aa815fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4aadf9d640 1 --2- 172.21.15.44:0/3755321172 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa815e430 0x7f4aa815fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4aadf9d640 1 -- 172.21.15.44:0/3755321172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aa80d0f70 msgr2=0x7f4aa81f7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4aadf9d640 1 --2- 172.21.15.44:0/3755321172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aa80d0f70 0x7f4aa81f7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4aadf9d640 1 -- 172.21.15.44:0/3755321172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa815ea50 con 0x7f4aa80b3f50 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4aadf9d640 1 --2- 172.21.15.44:0/3755321172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4aa80b3f50 0x7f4aa80c9b90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4000c00 tx=0x7f4aa40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4a9e7fc640 1 -- 172.21.15.44:0/3755321172 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa403ad60 con 0x7f4aa80b3f50 2024-04-02T13:20:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4a9e7fc640 1 -- 172.21.15.44:0/3755321172 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4aa40443f0 con 0x7f4aa80b3f50 2024-04-02T13:20:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3755321172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa81f8060 con 0x7f4aa80b3f50 2024-04-02T13:20:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4a9e7fc640 1 -- 172.21.15.44:0/3755321172 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa40447e0 con 0x7f4aa80b3f50 2024-04-02T13:20:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.852+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3755321172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa815efa0 con 0x7f4aa80b3f50 2024-04-02T13:20:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.853+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3755321172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa804f8d0 con 0x7f4aa80b3f50 2024-04-02T13:20:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.854+0000 7f4a9e7fc640 1 -- 172.21.15.44:0/3755321172 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4aa404b060 con 0x7f4aa80b3f50 2024-04-02T13:20:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.854+0000 7f4a9e7fc640 1 --2- 172.21.15.44:0/3755321172 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4a8c06c920 0x7f4a8c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.854+0000 7f4a9e7fc640 1 -- 172.21.15.44:0/3755321172 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4aa40adf10 con 0x7f4aa80b3f50 2024-04-02T13:20:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.854+0000 7f4aad79c640 1 --2- 172.21.15.44:0/3755321172 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4a8c06c920 0x7f4a8c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.855+0000 7f4aad79c640 1 --2- 172.21.15.44:0/3755321172 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4a8c06c920 0x7f4a8c06ede0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4a98006fd0 tx=0x7f4a98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:52.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:52.858+0000 7f4a9e7fc640 1 -- 172.21.15.44:0/3755321172 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4aa81f8060 con 0x7f4aa80b3f50 2024-04-02T13:20:53.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.091+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3755321172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4aa80b3280 con 0x7f4aa80b3f50 2024-04-02T13:20:53.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.092+0000 7f4a9e7fc640 1 -- 172.21.15.44:0/3755321172 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7f4aa403f070 con 0x7f4aa80b3f50 2024-04-02T13:20:53.094 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:53.094 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4040417710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4040417710},{"type":"v1","addr":"172.21.15.69:6836","nonce":4040417710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2772068060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2772068060},{"type":"v1","addr":"172.21.15.44:6833","nonce":2772068060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1985405150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1985405150},{"type":"v1","addr":"172.21.15.69:6837","nonce":1985405150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:20:20.247358+0000","modified":"2024-04-02T13:20:42.332570+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/2282156931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2282156931},{"type":"v1","addr":"172.21.15.44:6835","nonce":2282156931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-04-02T13:20:53.094 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 9 2024-04-02T13:20:53.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.094+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3755321172 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4a8c06c920 msgr2=0x7f4a8c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.094+0000 7f4aaef9f640 1 --2- 172.21.15.44:0/3755321172 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4a8c06c920 0x7f4a8c06ede0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4a98006fd0 tx=0x7f4a98008040 comp rx=0 tx=0).stop 2024-04-02T13:20:53.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.094+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3755321172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4aa80b3f50 msgr2=0x7f4aa80c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.094+0000 7f4aaef9f640 1 --2- 172.21.15.44:0/3755321172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4aa80b3f50 0x7f4aa80c9b90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4000c00 tx=0x7f4aa40046c0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.094+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3755321172 shutdown_connections 2024-04-02T13:20:53.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.094+0000 7f4aaef9f640 1 --2- 172.21.15.44:0/3755321172 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4a8c06c920 0x7f4a8c06ede0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.094+0000 7f4aaef9f640 1 --2- 172.21.15.44:0/3755321172 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa815e430 0x7f4aa815fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.094+0000 7f4aaef9f640 1 --2- 172.21.15.44:0/3755321172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aa80d0f70 0x7f4aa81f7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.094+0000 7f4aaef9f640 1 --2- 172.21.15.44:0/3755321172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4aa80b3f50 0x7f4aa80c9b90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.094+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3755321172 >> 172.21.15.44:0/3755321172 conn(0x7f4aa80d3e80 msgr2=0x7f4aa806d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:53.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.094+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3755321172 shutdown_connections 2024-04-02T13:20:53.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.094+0000 7f4aaef9f640 1 -- 172.21.15.44:0/3755321172 wait complete. 2024-04-02T13:20:53.107 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:20:53.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.206+0000 7fed4d174640 1 Processor -- start 2024-04-02T13:20:53.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.206+0000 7fed4d174640 1 -- start start 2024-04-02T13:20:53.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.207+0000 7fed4d174640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed480b47b0 0x7fed480d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:53.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.207+0000 7fed4d174640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fed480b5200 0x7fed480b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:53.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.207+0000 7fed4d174640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed480d4d70 con 0x7fed480d28f0 2024-04-02T13:20:53.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.207+0000 7fed4d174640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fed480d5050 con 0x7fed480b5200 2024-04-02T13:20:53.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.207+0000 7fed4d174640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fed480d23a0 con 0x7fed480b47b0 2024-04-02T13:20:53.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.207+0000 7fed4759e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fed480b5200 0x7fed480b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.207+0000 7fed4759e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fed480b5200 0x7fed480b7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55324/0 (socket says 172.21.15.44:55324) 2024-04-02T13:20:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.207+0000 7fed4759e640 1 -- 172.21.15.44:0/4274546438 learned_addr learned my addr 172.21.15.44:0/4274546438 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.208+0000 7fed3ffff640 1 --2- 172.21.15.44:0/4274546438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed480b47b0 0x7fed480d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.208+0000 7fed4759e640 1 -- 172.21.15.44:0/4274546438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed480b47b0 msgr2=0x7fed480d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.208+0000 7fed4759e640 1 --2- 172.21.15.44:0/4274546438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed480b47b0 0x7fed480d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.208+0000 7fed4759e640 1 -- 172.21.15.44:0/4274546438 >> v1:172.21.15.69:6789/0 conn(0x7fed480d28f0 legacy=0x7fed480b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.208+0000 7fed4759e640 1 -- 172.21.15.44:0/4274546438 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed480d2030 con 0x7fed480b5200 2024-04-02T13:20:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.208+0000 7fed3ffff640 1 --2- 172.21.15.44:0/4274546438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed480b47b0 0x7fed480d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:53.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.208+0000 7fed4759e640 1 --2- 172.21.15.44:0/4274546438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed480b5200 0x7fed480b7120 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fed40009e50 tx=0x7fed4002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f81835c81a90244 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:53.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.209+0000 7fed4659c640 1 -- 172.21.15.44:0/4274546438 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4002c9e0 con 0x7fed480b5200 2024-04-02T13:20:53.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.209+0000 7fed4659c640 1 -- 172.21.15.44:0/4274546438 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed4002cb80 con 0x7fed480b5200 2024-04-02T13:20:53.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.209+0000 7fed4659c640 1 -- 172.21.15.44:0/4274546438 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed400343f0 con 0x7fed480b5200 2024-04-02T13:20:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.209+0000 7fed4d174640 1 -- 172.21.15.44:0/4274546438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed480b5200 msgr2=0x7fed480b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.209+0000 7fed4d174640 1 --2- 172.21.15.44:0/4274546438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed480b5200 0x7fed480b7120 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fed40009e50 tx=0x7fed4002be40 comp rx=0 tx=0).stop 2024-04-02T13:20:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.209+0000 7fed4d174640 1 -- 172.21.15.44:0/4274546438 shutdown_connections 2024-04-02T13:20:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.209+0000 7fed4d174640 1 --2- 172.21.15.44:0/4274546438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed480b5200 0x7fed480b7120 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.209+0000 7fed4d174640 1 --2- 172.21.15.44:0/4274546438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed480b47b0 0x7fed480d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.209+0000 7fed4d174640 1 -- 172.21.15.44:0/4274546438 >> 172.21.15.44:0/4274546438 conn(0x7fed480d4120 msgr2=0x7fed480d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.209+0000 7fed4d174640 1 -- 172.21.15.44:0/4274546438 shutdown_connections 2024-04-02T13:20:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.209+0000 7fed4d174640 1 -- 172.21.15.44:0/4274546438 wait complete. 2024-04-02T13:20:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.210+0000 7fed4d174640 1 Processor -- start 2024-04-02T13:20:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.211+0000 7fed4d174640 1 -- start start 2024-04-02T13:20:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.211+0000 7fed4d174640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed480b47b0 0x7fed48156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.211+0000 7fed4d174640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed480b5200 0x7fed4809e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.211+0000 7fed4d174640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed480d28f0 0x7fed4809e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.211+0000 7fed4d174640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fed480d1260 con 0x7fed480b47b0 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.211+0000 7fed4d174640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed480b5020 con 0x7fed480d28f0 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.211+0000 7fed4d174640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fed480d13e0 con 0x7fed480b5200 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.211+0000 7fed46d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed480b47b0 0x7fed48156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.211+0000 7fed46d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed480b47b0 0x7fed48156a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55340/0 (socket says 172.21.15.44:55340) 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.211+0000 7fed46d9d640 1 -- 172.21.15.44:0/2696822118 learned_addr learned my addr 172.21.15.44:0/2696822118 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.211+0000 7fed3ffff640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed480b5200 0x7fed4809e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.212+0000 7fed4759e640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed480d28f0 0x7fed4809e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.212+0000 7fed46d9d640 1 -- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed480b5200 msgr2=0x7fed4809e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.212+0000 7fed46d9d640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed480b5200 0x7fed4809e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.212+0000 7fed46d9d640 1 -- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed480d28f0 msgr2=0x7fed4809e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.212+0000 7fed46d9d640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed480d28f0 0x7fed4809e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.212+0000 7fed46d9d640 1 -- 172.21.15.44:0/2696822118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed480a1e30 con 0x7fed480b47b0 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.212+0000 7fed46d9d640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed480b47b0 0x7fed48156a00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fed3800c3c0 tx=0x7fed3800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.212+0000 7fed3f7fe640 1 -- 172.21.15.44:0/2696822118 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380052d0 con 0x7fed480b47b0 2024-04-02T13:20:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.212+0000 7fed3f7fe640 1 -- 172.21.15.44:0/2696822118 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed38005490 con 0x7fed480b47b0 2024-04-02T13:20:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.212+0000 7fed3f7fe640 1 -- 172.21.15.44:0/2696822118 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38007260 con 0x7fed480b47b0 2024-04-02T13:20:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.212+0000 7fed4d174640 1 -- 172.21.15.44:0/2696822118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4809ec40 con 0x7fed480b47b0 2024-04-02T13:20:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.212+0000 7fed4d174640 1 -- 172.21.15.44:0/2696822118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed48157240 con 0x7fed480b47b0 2024-04-02T13:20:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.213+0000 7fed4d174640 1 -- 172.21.15.44:0/2696822118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed480684f0 con 0x7fed480b47b0 2024-04-02T13:20:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.213+0000 7fed3f7fe640 1 -- 172.21.15.44:0/2696822118 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fed38007400 con 0x7fed480b47b0 2024-04-02T13:20:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.214+0000 7fed3f7fe640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fed2806c8f0 0x7fed2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.214+0000 7fed3f7fe640 1 -- 172.21.15.44:0/2696822118 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fed3808b250 con 0x7fed480b47b0 2024-04-02T13:20:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.214+0000 7fed3ffff640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fed2806c8f0 0x7fed2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.215+0000 7fed3ffff640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fed2806c8f0 0x7fed2806edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fed4815ffd0 tx=0x7fed30008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:53.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.218+0000 7fed3f7fe640 1 -- 172.21.15.44:0/2696822118 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fed38058d80 con 0x7fed480b47b0 2024-04-02T13:20:53.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.404+0000 7fed4d174640 1 -- 172.21.15.44:0/2696822118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed4806d720 con 0x7fed480b47b0 2024-04-02T13:20:53.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.405+0000 7fed3f7fe640 1 -- 172.21.15.44:0/2696822118 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7fed3805d080 con 0x7fed480b47b0 2024-04-02T13:20:53.407 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:53.408 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:20:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.407+0000 7fed4d174640 1 -- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fed2806c8f0 msgr2=0x7fed2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.407+0000 7fed4d174640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fed2806c8f0 0x7fed2806edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fed4815ffd0 tx=0x7fed30008040 comp rx=0 tx=0).stop 2024-04-02T13:20:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.407+0000 7fed4d174640 1 -- 172.21.15.44:0/2696822118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed480b47b0 msgr2=0x7fed48156a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.407+0000 7fed4d174640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed480b47b0 0x7fed48156a00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fed3800c3c0 tx=0x7fed3800c880 comp rx=0 tx=0).stop 2024-04-02T13:20:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.407+0000 7fed4d174640 1 -- 172.21.15.44:0/2696822118 shutdown_connections 2024-04-02T13:20:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.407+0000 7fed4d174640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fed2806c8f0 0x7fed2806edb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.407+0000 7fed4d174640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed480d28f0 0x7fed4809e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.407+0000 7fed4d174640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed480b5200 0x7fed4809e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.407+0000 7fed4d174640 1 --2- 172.21.15.44:0/2696822118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed480b47b0 0x7fed48156a00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.407+0000 7fed4d174640 1 -- 172.21.15.44:0/2696822118 >> 172.21.15.44:0/2696822118 conn(0x7fed480d4120 msgr2=0x7fed480d0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.407+0000 7fed4d174640 1 -- 172.21.15.44:0/2696822118 shutdown_connections 2024-04-02T13:20:53.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.407+0000 7fed4d174640 1 -- 172.21.15.44:0/2696822118 wait complete. 2024-04-02T13:20:53.421 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:20:53.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.520+0000 7f987e093640 1 Processor -- start 2024-04-02T13:20:53.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.520+0000 7f987e093640 1 -- start start 2024-04-02T13:20:53.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.521+0000 7f987e093640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f987815e0f0 0x7f98780d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:53.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.521+0000 7f987e093640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f987806bbe0 0x7f98780add60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:53.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.521+0000 7f987e093640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987806dfe0 con 0x7f98780d0630 2024-04-02T13:20:53.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.521+0000 7f987e093640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f987806e2c0 con 0x7f987806bbe0 2024-04-02T13:20:53.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.521+0000 7f987e093640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f987806e5a0 con 0x7f987815e0f0 2024-04-02T13:20:53.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.521+0000 7f9877fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f987806bbe0 0x7f98780add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:53.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.521+0000 7f98777fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f987815e0f0 0x7f98780d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:53.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.521+0000 7f9877fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f987806bbe0 0x7f98780add60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34748/0 (socket says 172.21.15.44:34748) 2024-04-02T13:20:53.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.521+0000 7f9877fff640 1 -- 172.21.15.44:0/802485255 learned_addr learned my addr 172.21.15.44:0/802485255 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:53.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.521+0000 7f9876ffd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f98780d0630 0x7f987806b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:48792/0 (socket says 172.21.15.44:48792) 2024-04-02T13:20:53.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.522+0000 7f9877fff640 1 -- 172.21.15.44:0/802485255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f987815e0f0 msgr2=0x7f98780d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:53.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.522+0000 7f9877fff640 1 --2- 172.21.15.44:0/802485255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f987815e0f0 0x7f98780d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.522+0000 7f9877fff640 1 -- 172.21.15.44:0/802485255 >> v1:172.21.15.69:6789/0 conn(0x7f98780d0630 legacy=0x7f987806b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.522+0000 7f9877fff640 1 -- 172.21.15.44:0/802485255 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987806e880 con 0x7f987806bbe0 2024-04-02T13:20:53.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.522+0000 7f98777fe640 1 --2- 172.21.15.44:0/802485255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f987815e0f0 0x7f98780d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:53.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.523+0000 7f9877fff640 1 --2- 172.21.15.44:0/802485255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f987806bbe0 0x7f98780add60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9870009e50 tx=0x7f987002bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce32b29f8c14703 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:53.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.523+0000 7f98767fc640 1 -- 172.21.15.44:0/802485255 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987002cb90 con 0x7f987806bbe0 2024-04-02T13:20:53.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.523+0000 7f98767fc640 1 -- 172.21.15.44:0/802485255 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f987002cd30 con 0x7f987806bbe0 2024-04-02T13:20:53.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.523+0000 7f98767fc640 1 -- 172.21.15.44:0/802485255 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870034610 con 0x7f987806bbe0 2024-04-02T13:20:53.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.523+0000 7f987e093640 1 -- 172.21.15.44:0/802485255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f987806bbe0 msgr2=0x7f98780add60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.523+0000 7f987e093640 1 --2- 172.21.15.44:0/802485255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f987806bbe0 0x7f98780add60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9870009e50 tx=0x7f987002bff0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.523+0000 7f987e093640 1 -- 172.21.15.44:0/802485255 shutdown_connections 2024-04-02T13:20:53.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.523+0000 7f987e093640 1 --2- 172.21.15.44:0/802485255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f987806bbe0 0x7f98780add60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.523+0000 7f987e093640 1 --2- 172.21.15.44:0/802485255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f987815e0f0 0x7f98780d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.523+0000 7f987e093640 1 -- 172.21.15.44:0/802485255 >> 172.21.15.44:0/802485255 conn(0x7f98780b2980 msgr2=0x7f98780b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:53.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.523+0000 7f987e093640 1 -- 172.21.15.44:0/802485255 shutdown_connections 2024-04-02T13:20:53.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.523+0000 7f987e093640 1 -- 172.21.15.44:0/802485255 wait complete. 2024-04-02T13:20:53.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.524+0000 7f987e093640 1 Processor -- start 2024-04-02T13:20:53.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.525+0000 7f987e093640 1 -- start start 2024-04-02T13:20:53.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.525+0000 7f987e093640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f987806bbe0 0x7f98781f56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:53.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.525+0000 7f987e093640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98780d0630 0x7f98781606b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:53.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.525+0000 7f987e093640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f987815e0f0 0x7f98781f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:53.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.525+0000 7f987e093640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f987815e4d0 con 0x7f987806bbe0 2024-04-02T13:20:53.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.525+0000 7f987e093640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f987806eda0 con 0x7f98780d0630 2024-04-02T13:20:53.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.525+0000 7f987e093640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f987815e650 con 0x7f987815e0f0 2024-04-02T13:20:53.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f98777fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f987806bbe0 0x7f98781f56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:53.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f98777fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f987806bbe0 0x7f98781f56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55348/0 (socket says 172.21.15.44:55348) 2024-04-02T13:20:53.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f98777fe640 1 -- 172.21.15.44:0/846586228 learned_addr learned my addr 172.21.15.44:0/846586228 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:53.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f9876ffd640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98780d0630 0x7f98781606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:53.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f9877fff640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f987815e0f0 0x7f98781f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:53.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f98777fe640 1 -- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f987815e0f0 msgr2=0x7f98781f4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f98777fe640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f987815e0f0 0x7f98781f4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f98777fe640 1 -- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98780d0630 msgr2=0x7f98781606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f98777fe640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98780d0630 0x7f98781606b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f98777fe640 1 -- 172.21.15.44:0/846586228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98781f4e70 con 0x7f987806bbe0 2024-04-02T13:20:53.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f9876ffd640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98780d0630 0x7f98781606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:53.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f98777fe640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f987806bbe0 0x7f98781f56c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f986800e960 tx=0x7f986800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:53.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f985ffff640 1 -- 172.21.15.44:0/846586228 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9868017070 con 0x7f987806bbe0 2024-04-02T13:20:53.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f985ffff640 1 -- 172.21.15.44:0/846586228 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f986800ca00 con 0x7f987806bbe0 2024-04-02T13:20:53.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f987e093640 1 -- 172.21.15.44:0/846586228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98781f5150 con 0x7f987806bbe0 2024-04-02T13:20:53.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f985ffff640 1 -- 172.21.15.44:0/846586228 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986800cdd0 con 0x7f987806bbe0 2024-04-02T13:20:53.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.526+0000 7f987e093640 1 -- 172.21.15.44:0/846586228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f987815e990 con 0x7f987806bbe0 2024-04-02T13:20:53.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.527+0000 7f987e093640 1 -- 172.21.15.44:0/846586228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98780684f0 con 0x7f987806bbe0 2024-04-02T13:20:53.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.528+0000 7f985ffff640 1 -- 172.21.15.44:0/846586228 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f986801a070 con 0x7f987806bbe0 2024-04-02T13:20:53.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.528+0000 7f985ffff640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f986406c8f0 0x7f986406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:53.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.528+0000 7f985ffff640 1 -- 172.21.15.44:0/846586228 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f986808c100 con 0x7f987806bbe0 2024-04-02T13:20:53.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.528+0000 7f9876ffd640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f986406c8f0 0x7f986406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:53.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.529+0000 7f9876ffd640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f986406c8f0 0x7f986406edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f98780d0580 tx=0x7f986c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:53.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.533+0000 7f985ffff640 1 -- 172.21.15.44:0/846586228 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f986805a1f0 con 0x7f987806bbe0 2024-04-02T13:20:53.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.755+0000 7f987e093640 1 -- 172.21.15.44:0/846586228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98780c7bd0 con 0x7f987806bbe0 2024-04-02T13:20:53.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.755+0000 7f985ffff640 1 -- 172.21.15.44:0/846586228 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7f9868059af0 con 0x7f987806bbe0 2024-04-02T13:20:53.758 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:20:53.758 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4040417710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4040417710},{"type":"v1","addr":"172.21.15.69:6836","nonce":4040417710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2772068060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2772068060},{"type":"v1","addr":"172.21.15.44:6833","nonce":2772068060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1985405150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1985405150},{"type":"v1","addr":"172.21.15.69:6837","nonce":1985405150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:20:20.247358+0000","modified":"2024-04-02T13:20:42.332570+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/2282156931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2282156931},{"type":"v1","addr":"172.21.15.44:6835","nonce":2282156931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-04-02T13:20:53.758 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 9 2024-04-02T13:20:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.757+0000 7f987e093640 1 -- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f986406c8f0 msgr2=0x7f986406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.757+0000 7f987e093640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f986406c8f0 0x7f986406edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f98780d0580 tx=0x7f986c009040 comp rx=0 tx=0).stop 2024-04-02T13:20:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.757+0000 7f987e093640 1 -- 172.21.15.44:0/846586228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f987806bbe0 msgr2=0x7f98781f56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.757+0000 7f987e093640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f987806bbe0 0x7f98781f56c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f986800e960 tx=0x7f986800ee20 comp rx=0 tx=0).stop 2024-04-02T13:20:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.757+0000 7f987e093640 1 -- 172.21.15.44:0/846586228 shutdown_connections 2024-04-02T13:20:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.757+0000 7f987e093640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f986406c8f0 0x7f986406edb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.757+0000 7f987e093640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f987815e0f0 0x7f98781f4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.757+0000 7f987e093640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98780d0630 0x7f98781606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.757+0000 7f987e093640 1 --2- 172.21.15.44:0/846586228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f987806bbe0 0x7f98781f56c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:53.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.757+0000 7f987e093640 1 -- 172.21.15.44:0/846586228 >> 172.21.15.44:0/846586228 conn(0x7f98780b2980 msgr2=0x7f98780b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:53.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.757+0000 7f987e093640 1 -- 172.21.15.44:0/846586228 shutdown_connections 2024-04-02T13:20:53.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:53.757+0000 7f987e093640 1 -- 172.21.15.44:0/846586228 wait complete. 2024-04-02T13:20:54.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:20:20.247358+0000', 'modified': '2024-04-02T13:20:42.332570+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4266}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4266': {'gid': 4266, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.44:6835/2282156931', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6834', 'nonce': 2282156931}, {'type': 'v1', 'addr': '172.21.15.44:6835', 'nonce': 2282156931}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4266, 'qdb_cluster': [4266]} 2024-04-02T13:20:54.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:20:54.177 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-02T13:20:54.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.284+0000 7f90acefb640 1 Processor -- start 2024-04-02T13:20:54.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.284+0000 7f90acefb640 1 -- start start 2024-04-02T13:20:54.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90acefb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f90a815e4a0 0x7f90a80aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90acefb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f90a80d0c50 0x7f90a80d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90acefb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90a80cc600 con 0x7f90a806ce40 2024-04-02T13:20:54.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90acefb640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f90a80cc8e0 con 0x7f90a80d0c50 2024-04-02T13:20:54.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90acefb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f90a80ccbc0 con 0x7f90a815e4a0 2024-04-02T13:20:54.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90a5d9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f90a80d0c50 0x7f90a80d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90a6d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f90a806ce40 0x7f90a80cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57586/0 (socket says 172.21.15.44:57586) 2024-04-02T13:20:54.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90a6d9d640 1 -- 172.21.15.44:0/3731404699 learned_addr learned my addr 172.21.15.44:0/3731404699 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:54.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90a659c640 1 --2- 172.21.15.44:0/3731404699 >> v2:172.21.15.69:3300/0 conn(0x7f90a815e4a0 0x7f90a80aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90a5d9b640 1 -- 172.21.15.44:0/3731404699 >> v2:172.21.15.69:3300/0 conn(0x7f90a815e4a0 msgr2=0x7f90a80aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:54.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90a5d9b640 1 --2- 172.21.15.44:0/3731404699 >> v2:172.21.15.69:3300/0 conn(0x7f90a815e4a0 0x7f90a80aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90a5d9b640 1 -- 172.21.15.44:0/3731404699 >> v1:172.21.15.44:6789/0 conn(0x7f90a806ce40 legacy=0x7f90a80cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90a5d9b640 1 -- 172.21.15.44:0/3731404699 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a80ccea0 con 0x7f90a80d0c50 2024-04-02T13:20:54.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.285+0000 7f90a559a640 1 -- 172.21.15.44:0/3731404699 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1538775912 0 0) 0x7f90a80cc600 con 0x7f90a806ce40 2024-04-02T13:20:54.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.286+0000 7f90a5d9b640 1 --2- 172.21.15.44:0/3731404699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90a80d0c50 0x7f90a80d10b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f90a80d1f80 tx=0x7f909002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6a854464ff5fb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:54.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.286+0000 7f90a559a640 1 -- 172.21.15.44:0/3731404699 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090035020 con 0x7f90a80d0c50 2024-04-02T13:20:54.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.286+0000 7f90a559a640 1 -- 172.21.15.44:0/3731404699 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9090031830 con 0x7f90a80d0c50 2024-04-02T13:20:54.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.286+0000 7f90a559a640 1 -- 172.21.15.44:0/3731404699 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090031c00 con 0x7f90a80d0c50 2024-04-02T13:20:54.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.287+0000 7f90acefb640 1 -- 172.21.15.44:0/3731404699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90a80d0c50 msgr2=0x7f90a80d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.287+0000 7f90acefb640 1 --2- 172.21.15.44:0/3731404699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90a80d0c50 0x7f90a80d10b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f90a80d1f80 tx=0x7f909002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.287+0000 7f90acefb640 1 -- 172.21.15.44:0/3731404699 shutdown_connections 2024-04-02T13:20:54.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.287+0000 7f90acefb640 1 --2- 172.21.15.44:0/3731404699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90a80d0c50 0x7f90a80d10b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.287+0000 7f90acefb640 1 --2- 172.21.15.44:0/3731404699 >> v2:172.21.15.69:3300/0 conn(0x7f90a815e4a0 0x7f90a80aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.287+0000 7f90acefb640 1 -- 172.21.15.44:0/3731404699 >> 172.21.15.44:0/3731404699 conn(0x7f90a809aa20 msgr2=0x7f90a809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:54.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.287+0000 7f90acefb640 1 -- 172.21.15.44:0/3731404699 shutdown_connections 2024-04-02T13:20:54.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.287+0000 7f90acefb640 1 -- 172.21.15.44:0/3731404699 wait complete. 2024-04-02T13:20:54.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.288+0000 7f90acefb640 1 Processor -- start 2024-04-02T13:20:54.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.288+0000 7f90acefb640 1 -- start start 2024-04-02T13:20:54.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90acefb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90a806ce40 0x7f90a8154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90acefb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90a80d0c50 0x7f90a80caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90acefb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90a815e4a0 0x7f90a80b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90acefb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f90a815e880 con 0x7f90a806ce40 2024-04-02T13:20:54.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90acefb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f90a80cd460 con 0x7f90a815e4a0 2024-04-02T13:20:54.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90acefb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f90a80af130 con 0x7f90a80d0c50 2024-04-02T13:20:54.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90a659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90a806ce40 0x7f90a8154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90a659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90a806ce40 0x7f90a8154570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55360/0 (socket says 172.21.15.44:55360) 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90a659c640 1 -- 172.21.15.44:0/3829206876 learned_addr learned my addr 172.21.15.44:0/3829206876 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90a5d9b640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90a80d0c50 0x7f90a80caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90a6d9d640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90a815e4a0 0x7f90a80b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90a659c640 1 -- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90a80d0c50 msgr2=0x7f90a80caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90a659c640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90a80d0c50 0x7f90a80caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90a659c640 1 -- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90a815e4a0 msgr2=0x7f90a80b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90a659c640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90a815e4a0 0x7f90a80b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90a659c640 1 -- 172.21.15.44:0/3829206876 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a80c6a20 con 0x7f90a806ce40 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.289+0000 7f90a6d9d640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90a815e4a0 0x7f90a80b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.290+0000 7f90a659c640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90a806ce40 0x7f90a8154570 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f90a000e870 tx=0x7f90a000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.290+0000 7f909effd640 1 -- 172.21.15.44:0/3829206876 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a0017070 con 0x7f90a806ce40 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.290+0000 7f909effd640 1 -- 172.21.15.44:0/3829206876 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a000c940 con 0x7f90a806ce40 2024-04-02T13:20:54.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.290+0000 7f90acefb640 1 -- 172.21.15.44:0/3829206876 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90a80c5ea0 con 0x7f90a806ce40 2024-04-02T13:20:54.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.290+0000 7f909effd640 1 -- 172.21.15.44:0/3829206876 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a000cd30 con 0x7f90a806ce40 2024-04-02T13:20:54.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.290+0000 7f90acefb640 1 -- 172.21.15.44:0/3829206876 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90a80c6460 con 0x7f90a806ce40 2024-04-02T13:20:54.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.291+0000 7f909effd640 1 -- 172.21.15.44:0/3829206876 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f90a001a070 con 0x7f90a806ce40 2024-04-02T13:20:54.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.292+0000 7f909effd640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f908806c8f0 0x7f908806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.292+0000 7f909effd640 1 -- 172.21.15.44:0/3829206876 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f90a008bca0 con 0x7f90a806ce40 2024-04-02T13:20:54.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.292+0000 7f90a5d9b640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f908806c8f0 0x7f908806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.293+0000 7f90a5d9b640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f908806c8f0 0x7f908806edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9090036000 tx=0x7f9090008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:54.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.293+0000 7f90afbb6740 1 Processor -- start 2024-04-02T13:20:54.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.293+0000 7f90afbb6740 1 -- start start 2024-04-02T13:20:54.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90afbb6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 0x5632e3b81b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90afbb6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3bb1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90afbb6740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3b82070 0x5632e3bb1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90afbb6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5632e3aeb060 con 0x5632e3baeb90 2024-04-02T13:20:54.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90afbb6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5632e3a93650 con 0x5632e3baef70 2024-04-02T13:20:54.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90afbb6740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5632e3ae7010 con 0x5632e3b82070 2024-04-02T13:20:54.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90a659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 0x5632e3b81b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90a659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 0x5632e3b81b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55376/0 (socket says 172.21.15.44:55376) 2024-04-02T13:20:54.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90a659c640 1 -- 172.21.15.44:0/2847705508 learned_addr learned my addr 172.21.15.44:0/2847705508 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:54.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90a5d9b640 1 --2- 172.21.15.44:0/2847705508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3bb1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90a6d9d640 1 --2- 172.21.15.44:0/2847705508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3b82070 0x5632e3bb1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90a659c640 1 -- 172.21.15.44:0/2847705508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3b82070 msgr2=0x5632e3bb1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:54.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90a659c640 1 --2- 172.21.15.44:0/2847705508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3b82070 0x5632e3bb1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90a659c640 1 -- 172.21.15.44:0/2847705508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 msgr2=0x5632e3bb1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90a659c640 1 --2- 172.21.15.44:0/2847705508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3bb1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90a659c640 1 -- 172.21.15.44:0/2847705508 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632e37ff2d0 con 0x5632e3baeb90 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f90a659c640 1 --2- 172.21.15.44:0/2847705508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 0x5632e3b81b30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f90a0061d40 tx=0x7f90a0059160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d684a33fe30f60 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f9087fff640 1 -- 172.21.15.44:0/2847705508 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a0021070 con 0x5632e3baeb90 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f9087fff640 1 -- 172.21.15.44:0/2847705508 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a0062270 con 0x5632e3baeb90 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.294+0000 7f9087fff640 1 -- 172.21.15.44:0/2847705508 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a0055db0 con 0x5632e3baeb90 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.295+0000 7f90afbb6740 1 -- 172.21.15.44:0/2847705508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 msgr2=0x5632e3b81b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.295+0000 7f90afbb6740 1 --2- 172.21.15.44:0/2847705508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 0x5632e3b81b30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f90a0061d40 tx=0x7f90a0059160 comp rx=0 tx=0).stop 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.295+0000 7f90afbb6740 1 -- 172.21.15.44:0/2847705508 shutdown_connections 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.295+0000 7f90afbb6740 1 --2- 172.21.15.44:0/2847705508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3b82070 0x5632e3bb1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.295+0000 7f90afbb6740 1 --2- 172.21.15.44:0/2847705508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3bb1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.295+0000 7f90afbb6740 1 --2- 172.21.15.44:0/2847705508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 0x5632e3b81b30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.295+0000 7f90afbb6740 1 -- 172.21.15.44:0/2847705508 >> 172.21.15.44:0/2847705508 conn(0x5632e3b83da0 msgr2=0x5632e3bcc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.295+0000 7f90afbb6740 1 -- 172.21.15.44:0/2847705508 shutdown_connections 2024-04-02T13:20:54.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.295+0000 7f90afbb6740 1 -- 172.21.15.44:0/2847705508 wait complete. 2024-04-02T13:20:54.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.296+0000 7f90afbb6740 1 -- start start 2024-04-02T13:20:54.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.296+0000 7f90afbb6740 1 Processor -- start 2024-04-02T13:20:54.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90afbb6740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3baeb90 0x5632e3be1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90afbb6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3bdc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90afbb6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 0x5632e3be09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90afbb6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5632e3b86440 con 0x5632e3be0550 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90afbb6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5632e3aa6a10 con 0x5632e3baef70 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90afbb6740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5632e3b88c80 con 0x5632e3baeb90 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90afbb6740 10 client.0 _ll_register_callbacks cb 0x5632e3ae30b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90a6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 0x5632e3be09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90a6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 0x5632e3be09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55386/0 (socket says 172.21.15.44:55386) 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90a6d9d640 1 -- 172.21.15.44:0/811104849 learned_addr learned my addr 172.21.15.44:0/811104849 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90a659c640 1 --2- 172.21.15.44:0/811104849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3baeb90 0x5632e3be1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90a5d9b640 1 --2- 172.21.15.44:0/811104849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3bdc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90a6d9d640 1 -- 172.21.15.44:0/811104849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3baeb90 msgr2=0x5632e3be1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90a6d9d640 1 --2- 172.21.15.44:0/811104849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3baeb90 0x5632e3be1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90a6d9d640 1 -- 172.21.15.44:0/811104849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 msgr2=0x5632e3bdc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90a6d9d640 1 --2- 172.21.15.44:0/811104849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3bdc980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90a6d9d640 1 -- 172.21.15.44:0/811104849 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632e38c96b0 con 0x5632e3be0550 2024-04-02T13:20:54.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90a6d9d640 1 --2- 172.21.15.44:0/811104849 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 0x5632e3be09b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f90980018b0 tx=0x7f9098001d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b66827f2095d32 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90857fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.297+0000 7f90857fa640 1 -- 172.21.15.44:0/811104849 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9098019020 con 0x5632e3be0550 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90857fa640 1 -- 172.21.15.44:0/811104849 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9098011830 con 0x5632e3be0550 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90afbb6740 1 -- 172.21.15.44:0/811104849 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90a0062270 con 0x5632e3be0550 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90857fa640 1 -- 172.21.15.44:0/811104849 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f9098023050 con 0x5632e3be0550 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90afbb6740 10 client.4821 fetch_fsmap learned FSMap version 9 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90afbb6740 1 -- 172.21.15.44:0/811104849 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x5632e3bb2d50 con 0x5632e3be0550 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90857fa640 1 -- 172.21.15.44:0/811104849 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a0062270 con 0x5632e3be0550 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90857fa640 1 -- 172.21.15.44:0/811104849 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f9098026020 con 0x5632e3be0550 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90afbb6740 10 client.4821 fetch_fsmap finished waiting for FSMap version 9 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90afbb6740 10 client.4821 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90afbb6740 20 client.4821 populate_metadata read hostname 'smithi044' 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90afbb6740 1 --2- 172.21.15.44:0/811104849 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x5632e3bcb510 0x5632e3beb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90afbb6740 4 client.4821 mds_command: new command op to 4266 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90afbb6740 1 -- 172.21.15.44:0/811104849 --> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632e3a92cf0 con 0x5632e3bcb510 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90a659c640 1 --2- 172.21.15.44:0/811104849 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x5632e3bcb510 0x5632e3beb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90857fa640 10 client.4821 ms_handle_connect on v2:172.21.15.44:6834/2282156931 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.298+0000 7f90a659c640 1 --2- 172.21.15.44:0/811104849 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x5632e3bcb510 0x5632e3beb9e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbecdb07068fb992 server_cookie=33ac2a50096be36d in_seq=0 out_seq=0 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90857fa640 1 -- 172.21.15.44:0/811104849 <== mds.0 v2:172.21.15.44:6834/2282156931 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632e3a92cf0 con 0x5632e3bcb510 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90857fa640 10 client.4821 handle_command_reply: tid=0 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90afbb6740 1 client.4821 shutdown 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90afbb6740 1 -- 172.21.15.44:0/811104849 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 msgr2=0x5632e3be09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90afbb6740 1 --2- 172.21.15.44:0/811104849 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 0x5632e3be09b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f90980018b0 tx=0x7f9098001d70 comp rx=0 tx=0).stop 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90afbb6740 1 -- 172.21.15.44:0/811104849 shutdown_connections 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90afbb6740 1 --2- 172.21.15.44:0/811104849 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x5632e3bcb510 0x5632e3beb9e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90afbb6740 1 --2- 172.21.15.44:0/811104849 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 0x5632e3be09b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90afbb6740 1 --2- 172.21.15.44:0/811104849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3bdc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90afbb6740 1 --2- 172.21.15.44:0/811104849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3baeb90 0x5632e3be1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90857fa640 0 client.4821 ms_handle_reset on v2:172.21.15.44:6834/2282156931 2024-04-02T13:20:54.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90afbb6740 1 -- 172.21.15.44:0/811104849 >> 172.21.15.44:0/811104849 conn(0x5632e3b82070 msgr2=0x5632e3b85a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:54.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90afbb6740 1 -- 172.21.15.44:0/811104849 shutdown_connections 2024-04-02T13:20:54.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.299+0000 7f90afbb6740 1 -- 172.21.15.44:0/811104849 wait complete. 2024-04-02T13:20:54.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.300+0000 7f90afbb6740 20 client.4821 trim_cache size 0 max 16384 2024-04-02T13:20:54.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.318+0000 7f90afbb6740 1 Processor -- start 2024-04-02T13:20:54.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.318+0000 7f90afbb6740 1 -- start start 2024-04-02T13:20:54.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90afbb6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 0x5632e3bb1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90afbb6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3bf0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90afbb6740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3be0550 0x5632e3bf0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90afbb6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f90a8201bc0 con 0x5632e3baeb90 2024-04-02T13:20:54.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90afbb6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f90a8200bf0 con 0x5632e3baef70 2024-04-02T13:20:54.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90afbb6740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f90a82000e0 con 0x5632e3be0550 2024-04-02T13:20:54.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90a659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 0x5632e3bb1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90a659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 0x5632e3bb1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55394/0 (socket says 172.21.15.44:55394) 2024-04-02T13:20:54.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90a659c640 1 -- 172.21.15.44:0/1064685479 learned_addr learned my addr 172.21.15.44:0/1064685479 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:54.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90a659c640 1 -- 172.21.15.44:0/1064685479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3be0550 msgr2=0x5632e3bf0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:54.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90a659c640 1 --2- 172.21.15.44:0/1064685479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3be0550 0x5632e3bf0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90a659c640 1 -- 172.21.15.44:0/1064685479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 msgr2=0x5632e3bf0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:54.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90a659c640 1 --2- 172.21.15.44:0/1064685479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3bf0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90a659c640 1 -- 172.21.15.44:0/1064685479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632e3a8ba80 con 0x5632e3baeb90 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90a659c640 1 --2- 172.21.15.44:0/1064685479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 0x5632e3bb1550 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f90a0055b50 tx=0x7f90a001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b081460b2c446e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90877fe640 1 -- 172.21.15.44:0/1064685479 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a0077070 con 0x5632e3baeb90 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90877fe640 1 -- 172.21.15.44:0/1064685479 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a0062270 con 0x5632e3baeb90 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.319+0000 7f90877fe640 1 -- 172.21.15.44:0/1064685479 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a0061de0 con 0x5632e3baeb90 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.320+0000 7f90afbb6740 1 -- 172.21.15.44:0/1064685479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 msgr2=0x5632e3bb1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.320+0000 7f90afbb6740 1 --2- 172.21.15.44:0/1064685479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 0x5632e3bb1550 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f90a0055b50 tx=0x7f90a001d000 comp rx=0 tx=0).stop 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.320+0000 7f90afbb6740 1 -- 172.21.15.44:0/1064685479 shutdown_connections 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.320+0000 7f90afbb6740 1 --2- 172.21.15.44:0/1064685479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3be0550 0x5632e3bf0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.320+0000 7f90afbb6740 1 --2- 172.21.15.44:0/1064685479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3bf0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.320+0000 7f90afbb6740 1 --2- 172.21.15.44:0/1064685479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3baeb90 0x5632e3bb1550 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.320+0000 7f90afbb6740 1 -- 172.21.15.44:0/1064685479 >> 172.21.15.44:0/1064685479 conn(0x5632e3b82070 msgr2=0x5632e3bccb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.320+0000 7f90afbb6740 1 -- 172.21.15.44:0/1064685479 shutdown_connections 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.320+0000 7f90afbb6740 1 -- 172.21.15.44:0/1064685479 wait complete. 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.321+0000 7f90afbb6740 1 -- start start 2024-04-02T13:20:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.321+0000 7f90afbb6740 1 Processor -- start 2024-04-02T13:20:54.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.321+0000 7f90afbb6740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3baeb90 0x5632e3c46550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.321+0000 7f90afbb6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3b813b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.321+0000 7f90afbb6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 0x5632e3be2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.321+0000 7f90afbb6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5632e3bb48b0 con 0x5632e3be0550 2024-04-02T13:20:54.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.321+0000 7f90afbb6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f90a82005d0 con 0x5632e3baef70 2024-04-02T13:20:54.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.321+0000 7f90afbb6740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5632e3bcaef0 con 0x5632e3baeb90 2024-04-02T13:20:54.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90afbb6740 10 client.0 _ll_register_callbacks cb 0x7f9098007050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:20:54.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90a6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 0x5632e3be2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90a6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 0x5632e3be2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55406/0 (socket says 172.21.15.44:55406) 2024-04-02T13:20:54.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90a6d9d640 1 -- 172.21.15.44:0/2377116282 learned_addr learned my addr 172.21.15.44:0/2377116282 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90a659c640 1 --2- 172.21.15.44:0/2377116282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3baeb90 0x5632e3c46550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90a5d9b640 1 --2- 172.21.15.44:0/2377116282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3b813b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90a6d9d640 1 -- 172.21.15.44:0/2377116282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3baeb90 msgr2=0x5632e3c46550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90a6d9d640 1 --2- 172.21.15.44:0/2377116282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3baeb90 0x5632e3c46550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90a6d9d640 1 -- 172.21.15.44:0/2377116282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 msgr2=0x5632e3b813b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90a6d9d640 1 --2- 172.21.15.44:0/2377116282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3b813b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90a6d9d640 1 -- 172.21.15.44:0/2377116282 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632e3bd0600 con 0x5632e3be0550 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90a6d9d640 1 --2- 172.21.15.44:0/2377116282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 0x5632e3be2c70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9098001220 tx=0x7f9098007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b830535df919f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90857fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90857fa640 1 -- 172.21.15.44:0/2377116282 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9098019030 con 0x5632e3be0550 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90857fa640 1 -- 172.21.15.44:0/2377116282 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9098028de0 con 0x5632e3be0550 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90afbb6740 1 -- 172.21.15.44:0/2377116282 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90a0062270 con 0x5632e3be0550 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90857fa640 1 -- 172.21.15.44:0/2377116282 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f9098011b80 con 0x5632e3be0550 2024-04-02T13:20:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90afbb6740 10 client.4827 fetch_fsmap learned FSMap version 9 2024-04-02T13:20:54.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90afbb6740 1 -- 172.21.15.44:0/2377116282 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x5632e3be72c0 con 0x5632e3be0550 2024-04-02T13:20:54.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.322+0000 7f90857fa640 1 -- 172.21.15.44:0/2377116282 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a0062270 con 0x5632e3be0550 2024-04-02T13:20:54.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90857fa640 1 -- 172.21.15.44:0/2377116282 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f909802f050 con 0x5632e3be0550 2024-04-02T13:20:54.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 10 client.4827 fetch_fsmap finished waiting for FSMap version 9 2024-04-02T13:20:54.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 10 client.4827 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T13:20:54.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 20 client.4827 populate_metadata read hostname 'smithi044' 2024-04-02T13:20:54.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 1 --2- 172.21.15.44:0/2377116282 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x5632e3befa00 0x5632e3befde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 4 client.4827 mds_command: new command op to 4266 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:20:54.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 1 -- 172.21.15.44:0/2377116282 --> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] -- command(tid 0: {"prefix": "status"}) -- 0x5632e3bb3f30 con 0x5632e3befa00 2024-04-02T13:20:54.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90a659c640 1 --2- 172.21.15.44:0/2377116282 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x5632e3befa00 0x5632e3befde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90857fa640 10 client.4827 ms_handle_connect on v2:172.21.15.44:6834/2282156931 2024-04-02T13:20:54.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90a659c640 1 --2- 172.21.15.44:0/2377116282 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x5632e3befa00 0x5632e3befde0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ee44e49020fc681 server_cookie=8050904d51236c6b in_seq=0 out_seq=0 2024-04-02T13:20:54.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90857fa640 1 -- 172.21.15.44:0/2377116282 <== mds.0 v2:172.21.15.44:6834/2282156931 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5632e3bb3f30 con 0x5632e3befa00 2024-04-02T13:20:54.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90857fa640 10 client.4827 handle_command_reply: tid=0 2024-04-02T13:20:54.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 1 client.4827 shutdown 2024-04-02T13:20:54.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 1 -- 172.21.15.44:0/2377116282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 msgr2=0x5632e3be2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 1 --2- 172.21.15.44:0/2377116282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 0x5632e3be2c70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9098001220 tx=0x7f9098007d90 comp rx=0 tx=0).stop 2024-04-02T13:20:54.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 1 -- 172.21.15.44:0/2377116282 shutdown_connections 2024-04-02T13:20:54.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 1 --2- 172.21.15.44:0/2377116282 >> [v2:172.21.15.44:6834/2282156931,v1:172.21.15.44:6835/2282156931] conn(0x5632e3befa00 0x5632e3befde0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 1 --2- 172.21.15.44:0/2377116282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5632e3be0550 0x5632e3be2c70 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 1 --2- 172.21.15.44:0/2377116282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5632e3baef70 0x5632e3b813b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90857fa640 0 client.4827 ms_handle_reset on v2:172.21.15.44:6834/2282156931 2024-04-02T13:20:54.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 1 --2- 172.21.15.44:0/2377116282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5632e3baeb90 0x5632e3c46550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 1 -- 172.21.15.44:0/2377116282 >> 172.21.15.44:0/2377116282 conn(0x5632e3b82070 msgr2=0x5632e3bb4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:54.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.323+0000 7f90afbb6740 1 -- 172.21.15.44:0/2377116282 shutdown_connections 2024-04-02T13:20:54.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90afbb6740 1 -- 172.21.15.44:0/2377116282 wait complete. 2024-04-02T13:20:54.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90afbb6740 20 client.4827 trim_cache size 0 max 16384 2024-04-02T13:20:54.332 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:20:54.332 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:20:54.332 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:20:54.332 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 4266, 2024-04-02T13:20:54.332 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:20:54.332 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:20:54.332 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:20:54.332 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 33.451703145000003, 2024-04-02T13:20:54.332 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 9, 2024-04-02T13:20:54.333 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 25, 2024-04-02T13:20:54.333 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 25, 2024-04-02T13:20:54.333 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 40.3088069 2024-04-02T13:20:54.333 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:20:54.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90acefb640 1 -- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f908806c8f0 msgr2=0x7f908806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90acefb640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f908806c8f0 0x7f908806edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9090036000 tx=0x7f9090008030 comp rx=0 tx=0).stop 2024-04-02T13:20:54.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90acefb640 1 -- 172.21.15.44:0/3829206876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90a806ce40 msgr2=0x7f90a8154570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90acefb640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90a806ce40 0x7f90a8154570 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f90a000e870 tx=0x7f90a000ed30 comp rx=0 tx=0).stop 2024-04-02T13:20:54.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90acefb640 1 -- 172.21.15.44:0/3829206876 shutdown_connections 2024-04-02T13:20:54.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90acefb640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f908806c8f0 0x7f908806edb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90acefb640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90a815e4a0 0x7f90a80b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90acefb640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90a80d0c50 0x7f90a80caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90acefb640 1 --2- 172.21.15.44:0/3829206876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90a806ce40 0x7f90a8154570 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90acefb640 1 -- 172.21.15.44:0/3829206876 >> 172.21.15.44:0/3829206876 conn(0x7f90a809aa20 msgr2=0x7f90a806c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:54.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90acefb640 1 -- 172.21.15.44:0/3829206876 shutdown_connections 2024-04-02T13:20:54.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.324+0000 7f90acefb640 1 -- 172.21.15.44:0/3829206876 wait complete. 2024-04-02T13:20:54.337 INFO:teuthology.run_tasks:Running task sequential... 2024-04-02T13:20:54.346 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-02T13:20:54.346 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-02T13:20:54.347 INFO:teuthology.task.print:Enabling mgr modules 2024-04-02T13:20:54.347 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-02T13:20:54.356 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-02T13:20:54.356 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-02T13:20:54.356 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-02T13:20:54.356 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-02T13:20:54.356 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:20:54.356 INFO:tasks.ceph_fuse:client.2 config is {} 2024-04-02T13:20:54.357 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-02T13:20:54.357 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-02T13:20:54.357 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-02T13:20:54.357 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-02T13:20:54.357 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-02T13:20:54.357 INFO:tasks.ceph_fuse:client.3 config is {} 2024-04-02T13:20:54.357 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-02T13:20:54.357 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-02T13:20:54.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:54.357 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-02T13:20:54.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:54.376 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link delete ceph-brx 2024-04-02T13:20:54.450 INFO:teuthology.orchestra.run.smithi069.stderr:Cannot find device "ceph-brx" 2024-04-02T13:20:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:20:54.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:54.452 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:20:54.466 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-02T13:20:54.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:54.467 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-02T13:20:54.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:54.534 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link delete ceph-brx 2024-04-02T13:20:54.632 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-02T13:20:54.633 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:20:54.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.733+0000 7fd799df2640 1 Processor -- start 2024-04-02T13:20:54.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.733+0000 7fd799df2640 1 -- start start 2024-04-02T13:20:54.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.734+0000 7fd799df2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7940d0c50 0x7fd7940d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.734+0000 7fd799df2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd79406ce40 0x7fd79406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.734+0000 7fd799df2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7940cc600 con 0x7fd79415e4a0 2024-04-02T13:20:54.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.734+0000 7fd799df2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd7940af300 con 0x7fd79406ce40 2024-04-02T13:20:54.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.734+0000 7fd799df2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd7940cc8e0 con 0x7fd7940d0c50 2024-04-02T13:20:54.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.734+0000 7fd792ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7940d0c50 0x7fd7940d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.734+0000 7fd7937fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd79415e4a0 0x7fd7940aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:42820/0 (socket says 172.21.15.44:42820) 2024-04-02T13:20:54.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.734+0000 7fd793fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd79406ce40 0x7fd79406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.734+0000 7fd7937fe640 1 -- 172.21.15.44:0/3045183008 learned_addr learned my addr 172.21.15.44:0/3045183008 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:54.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.735+0000 7fd7927fc640 1 -- 172.21.15.44:0/3045183008 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3056658094 0 0) 0x7fd7940cc600 con 0x7fd79415e4a0 2024-04-02T13:20:54.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.735+0000 7fd7927fc640 1 -- 172.21.15.44:0/3045183008 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd77c003520 con 0x7fd79415e4a0 2024-04-02T13:20:54.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.735+0000 7fd792ffd640 1 -- 172.21.15.44:0/3045183008 >> v2:172.21.15.69:3300/0 conn(0x7fd79406ce40 msgr2=0x7fd79406d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:54.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.735+0000 7fd792ffd640 1 --2- 172.21.15.44:0/3045183008 >> v2:172.21.15.69:3300/0 conn(0x7fd79406ce40 0x7fd79406d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.735+0000 7fd792ffd640 1 -- 172.21.15.44:0/3045183008 >> v1:172.21.15.69:6789/0 conn(0x7fd79415e4a0 legacy=0x7fd7940aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.735+0000 7fd792ffd640 1 -- 172.21.15.44:0/3045183008 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7940ccbc0 con 0x7fd7940d0c50 2024-04-02T13:20:54.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.735+0000 7fd793fff640 1 --2- 172.21.15.44:0/3045183008 >> v2:172.21.15.69:3300/0 conn(0x7fd79406ce40 0x7fd79406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:20:54.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.736+0000 7fd792ffd640 1 --2- 172.21.15.44:0/3045183008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7940d0c50 0x7fd7940d10b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd7880097a0 tx=0x7fd78802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=febb98ca216da177 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:54.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.736+0000 7fd7927fc640 1 -- 172.21.15.44:0/3045183008 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd788035020 con 0x7fd7940d0c50 2024-04-02T13:20:54.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.736+0000 7fd7927fc640 1 -- 172.21.15.44:0/3045183008 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd788031830 con 0x7fd7940d0c50 2024-04-02T13:20:54.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.736+0000 7fd7927fc640 1 -- 172.21.15.44:0/3045183008 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd788031be0 con 0x7fd7940d0c50 2024-04-02T13:20:54.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.736+0000 7fd799df2640 1 -- 172.21.15.44:0/3045183008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7940d0c50 msgr2=0x7fd7940d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.736+0000 7fd799df2640 1 --2- 172.21.15.44:0/3045183008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7940d0c50 0x7fd7940d10b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd7880097a0 tx=0x7fd78802b8d0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.736+0000 7fd799df2640 1 -- 172.21.15.44:0/3045183008 shutdown_connections 2024-04-02T13:20:54.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.736+0000 7fd799df2640 1 --2- 172.21.15.44:0/3045183008 >> v2:172.21.15.69:3300/0 conn(0x7fd79406ce40 0x7fd79406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.736+0000 7fd799df2640 1 --2- 172.21.15.44:0/3045183008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7940d0c50 0x7fd7940d10b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.736+0000 7fd799df2640 1 -- 172.21.15.44:0/3045183008 >> 172.21.15.44:0/3045183008 conn(0x7fd79409aa20 msgr2=0x7fd79409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:54.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.736+0000 7fd799df2640 1 -- 172.21.15.44:0/3045183008 shutdown_connections 2024-04-02T13:20:54.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.736+0000 7fd799df2640 1 -- 172.21.15.44:0/3045183008 wait complete. 2024-04-02T13:20:54.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.737+0000 7fd799df2640 1 Processor -- start 2024-04-02T13:20:54.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.738+0000 7fd799df2640 1 -- start start 2024-04-02T13:20:54.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.738+0000 7fd799df2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd79406ce40 0x7fd794160970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.738+0000 7fd799df2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7940d0c50 0x7fd7940c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.738+0000 7fd799df2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd79415e4a0 0x7fd79415c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.738+0000 7fd799df2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd7940cd460 con 0x7fd79415e4a0 2024-04-02T13:20:54.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.738+0000 7fd799df2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd7940cd180 con 0x7fd79406ce40 2024-04-02T13:20:54.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.738+0000 7fd799df2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd79415e880 con 0x7fd7940d0c50 2024-04-02T13:20:54.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.738+0000 7fd793fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd79415e4a0 0x7fd79415c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.738+0000 7fd793fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd79415e4a0 0x7fd79415c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45750/0 (socket says 172.21.15.44:45750) 2024-04-02T13:20:54.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.738+0000 7fd793fff640 1 -- 172.21.15.44:0/1840971039 learned_addr learned my addr 172.21.15.44:0/1840971039 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:54.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd7937fe640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd79406ce40 0x7fd794160970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd792ffd640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7940d0c50 0x7fd7940c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd793fff640 1 -- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7940d0c50 msgr2=0x7fd7940c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd793fff640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7940d0c50 0x7fd7940c8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd793fff640 1 -- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd79406ce40 msgr2=0x7fd794160970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd793fff640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd79406ce40 0x7fd794160970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd793fff640 1 -- 172.21.15.44:0/1840971039 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79415f8d0 con 0x7fd79415e4a0 2024-04-02T13:20:54.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd792ffd640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7940d0c50 0x7fd7940c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:20:54.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd793fff640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd79415e4a0 0x7fd79415c440 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009f00 tx=0x7fd78c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:54.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd77bfff640 1 -- 172.21.15.44:0/1840971039 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78c018030 con 0x7fd79415e4a0 2024-04-02T13:20:54.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd77bfff640 1 -- 172.21.15.44:0/1840971039 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd78c00ce50 con 0x7fd79415e4a0 2024-04-02T13:20:54.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd77bfff640 1 -- 172.21.15.44:0/1840971039 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78c003bb0 con 0x7fd79415e4a0 2024-04-02T13:20:54.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd799df2640 1 -- 172.21.15.44:0/1840971039 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd79415fbb0 con 0x7fd79415e4a0 2024-04-02T13:20:54.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.739+0000 7fd799df2640 1 -- 172.21.15.44:0/1840971039 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd794160170 con 0x7fd79415e4a0 2024-04-02T13:20:54.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.740+0000 7fd799df2640 1 -- 172.21.15.44:0/1840971039 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7940684f0 con 0x7fd79415e4a0 2024-04-02T13:20:54.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.741+0000 7fd77bfff640 1 -- 172.21.15.44:0/1840971039 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd78c01b070 con 0x7fd79415e4a0 2024-04-02T13:20:54.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.741+0000 7fd77bfff640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd78006c8f0 0x7fd78006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:54.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.741+0000 7fd77bfff640 1 -- 172.21.15.44:0/1840971039 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd78c08c870 con 0x7fd79415e4a0 2024-04-02T13:20:54.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.741+0000 7fd7937fe640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd78006c8f0 0x7fd78006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:54.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.742+0000 7fd7937fe640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd78006c8f0 0x7fd78006edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd784005880 tx=0x7fd784009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:54.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.745+0000 7fd77bfff640 1 -- 172.21.15.44:0/1840971039 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd78c010040 con 0x7fd79415e4a0 2024-04-02T13:20:54.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.969+0000 7fd799df2640 1 -- 172.21.15.44:0/1840971039 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd79415dba0 con 0x7fd79415e4a0 2024-04-02T13:20:54.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.969+0000 7fd77bfff640 1 -- 172.21.15.44:0/1840971039 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7fd78c012340 con 0x7fd79415e4a0 2024-04-02T13:20:54.972 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-02T13:20:54.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.971+0000 7fd799df2640 1 -- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd78006c8f0 msgr2=0x7fd78006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.971+0000 7fd799df2640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd78006c8f0 0x7fd78006edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd784005880 tx=0x7fd784009040 comp rx=0 tx=0).stop 2024-04-02T13:20:54.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.971+0000 7fd799df2640 1 -- 172.21.15.44:0/1840971039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd79415e4a0 msgr2=0x7fd79415c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:54.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.971+0000 7fd799df2640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd79415e4a0 0x7fd79415c440 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009f00 tx=0x7fd78c00c490 comp rx=0 tx=0).stop 2024-04-02T13:20:54.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.971+0000 7fd799df2640 1 -- 172.21.15.44:0/1840971039 shutdown_connections 2024-04-02T13:20:54.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.971+0000 7fd799df2640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd78006c8f0 0x7fd78006edb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.971+0000 7fd799df2640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd79415e4a0 0x7fd79415c440 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.971+0000 7fd799df2640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7940d0c50 0x7fd7940c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.971+0000 7fd799df2640 1 --2- 172.21.15.44:0/1840971039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd79406ce40 0x7fd794160970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:54.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.971+0000 7fd799df2640 1 -- 172.21.15.44:0/1840971039 >> 172.21.15.44:0/1840971039 conn(0x7fd79409aa20 msgr2=0x7fd79406d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:54.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.971+0000 7fd799df2640 1 -- 172.21.15.44:0/1840971039 shutdown_connections 2024-04-02T13:20:54.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:54.971+0000 7fd799df2640 1 -- 172.21.15.44:0/1840971039 wait complete. 2024-04-02T13:20:54.984 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:20:54.984 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:20:54.984 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:20:54.984 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:20:54.985 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:20:54.985 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:20:54.985 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:20:54.985 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:20:54.985 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:20:54.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:54.985 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:20:55.000 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:20:55.000 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:20:55.000 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:20:55.000 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:55.000 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:20:55.000 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:55.000 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:20:55.000 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:55.000 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:20:55.000 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:20:55.001 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:55.001 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:20:55.001 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:20:55.001 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:55.001 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:20:55.001 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:20:55.001 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42724sec preferred_lft 42724sec 2024-04-02T13:20:55.001 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:20:55.001 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:55.001 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:20:55.001 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:55.001 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:20:55.001 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-02T13:20:55.002 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:55.002 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:55.002 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add name ceph-brx type bridge 2024-04-02T13:20:55.002 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr flush dev ceph-brx 2024-04-02T13:20:55.002 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set ceph-brx up 2024-04-02T13:20:55.002 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-02T13:20:55.002 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:55.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:55.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:55.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:55.149 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:20:55.175 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:20:55.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:55.176 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:20:55.233 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:20:55.233 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:20:55.233 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:20:55.233 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:20:55.233 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:20:55.233 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:55.233 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:55.233 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:20:55.234 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:20:55.234 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:20:55.234 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:55.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:55.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:55.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:55.368 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:20:55.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:55.383 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list-id 2024-04-02T13:20:55.437 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:55.438 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:55.438 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:20:55.438 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-02T13:20:55.438 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:55.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:55.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:55.560 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-02T13:20:55.561 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:55.561 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:55.561 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-02T13:20:55.561 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-02T13:20:55.561 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-02T13:20:55.561 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-02T13:20:55.561 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-02T13:20:55.561 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:55.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:55.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:55.686 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:55.686 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:55.686 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:20:55.686 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set dev brx.0 master ceph-brx 2024-04-02T13:20:55.686 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:55.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:55.767 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:20:55.767 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:20:55.767 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:20:55.781 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:20:55.781 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:20:55.781 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:20:55.835 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:20:55.902 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:20:55.957 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:20:55.957 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:20:55.957 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:20:55.957 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:20:55.957 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:20:55.957 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:20:55.957 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:20:55.958 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:20:55.959 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:20:55.959 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:20:55.959 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:20:55.959 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:20:55.959 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:20:55.959 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:20:55.959 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:20:55.959 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:20:55.959 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:20:55.959 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:20:55.959 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:20:55.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:55.960 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:20:56.014 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:20:56.015 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:20:56.057 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:20:56.085 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:20:56.141 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:20:56.141 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:20:56.141 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:20:56.141 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:20:56.141 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:20:56.141 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:20:56.141 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:20:56.141 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:20:56.142 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:20:56.143 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:20:56.143 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:20:56.143 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:20:56.143 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:20:56.143 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:20:56.143 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:20:56.143 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:20:56.143 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:20:56.143 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:20:56.143 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:20:56.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:56.144 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:20:56.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:20:56.145+0000 7fc1ffe3c480 -1 init, newargv = 0x5619bfc63220 newargc=15 2024-04-02T13:20:56.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[47546]: starting ceph client 2024-04-02T13:20:56.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[47546]: starting fuse 2024-04-02T13:20:56.163 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:20:56.163 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:20:56.163 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:20:56.164 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:20:56.165 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:20:56.165 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:20:56.165 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:20:56.165 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:20:56.165 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:20:56.165 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:20:56.165 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:20:56.165 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:20:56.165 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:20:56.165 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:20:56.165 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:20:56.285 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.47546.asok 2024-04-02T13:20:56.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:56.296 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47546.asok 2024-04-02T13:20:56.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:56.296 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47546.asok status 2024-04-02T13:20:56.434 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:20:56.435 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:20:56.435 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:20:56.435 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:20:56.435 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:20:56.435 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:20:56.435 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:20:56.435 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "47546", 2024-04-02T13:20:56.435 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:20:56.435 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:20:56.435 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:20:56.435 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:20:56.435 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 4836, 2024-04-02T13:20:56.435 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 4836 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2376580 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2376580 2024-04-02T13:20:56.436 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:20:56.437 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.4836 192.168.0.1:0/2376580", 2024-04-02T13:20:56.437 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/2376580", 2024-04-02T13:20:56.437 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:20:56.437 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 9, 2024-04-02T13:20:56.437 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 25, 2024-04-02T13:20:56.437 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:20:56.437 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:20:56.437 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:20:56.437 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:20:56.443 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:20:56.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.545+0000 7f75cf288640 1 Processor -- start 2024-04-02T13:20:56.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.545+0000 7f75cf288640 1 -- start start 2024-04-02T13:20:56.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.545+0000 7f75cf288640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f75c80ce2e0 0x7f75c8165820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:56.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.545+0000 7f75cf288640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75c80b3160 con 0x7f75c815aa00 2024-04-02T13:20:56.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.546+0000 7f75cf288640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75c815ade0 con 0x7f75c80b2d00 2024-04-02T13:20:56.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.546+0000 7f75cf288640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f75c80721a0 con 0x7f75c80ce2e0 2024-04-02T13:20:56.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.546+0000 7f75cd825640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f75c80ce2e0 0x7f75c8165820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:56.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.546+0000 7f75cd024640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f75c815aa00 0x7f75c8072bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:55288/0 (socket says 172.21.15.44:55288) 2024-04-02T13:20:56.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.546+0000 7f75cd024640 1 -- 172.21.15.44:0/3491551122 learned_addr learned my addr 172.21.15.44:0/3491551122 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:20:56.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.546+0000 7f75cd825640 1 --2- 172.21.15.44:0/3491551122 >> v2:172.21.15.44:3300/0 conn(0x7f75c80ce2e0 0x7f75c8165820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45756/0 (socket says 172.21.15.44:45756) 2024-04-02T13:20:56.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.546+0000 7f75cd825640 1 -- 172.21.15.44:0/3491551122 >> v1:172.21.15.69:6789/0 conn(0x7f75c80b2d00 legacy=0x7f75c80cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:56.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.546+0000 7f75cd825640 1 -- 172.21.15.44:0/3491551122 >> v1:172.21.15.44:6789/0 conn(0x7f75c815aa00 legacy=0x7f75c8072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:56.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.546+0000 7f75cd825640 1 -- 172.21.15.44:0/3491551122 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c80af1b0 con 0x7f75c80ce2e0 2024-04-02T13:20:56.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.546+0000 7f75bffff640 1 -- 172.21.15.44:0/3491551122 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 977065199 0 0) 0x7f75c80b3160 con 0x7f75c815aa00 2024-04-02T13:20:56.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.547+0000 7f75cd825640 1 --2- 172.21.15.44:0/3491551122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c80ce2e0 0x7f75c8165820 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009d90 tx=0x7f75c002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7c09a2e8c9e519 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:56.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.547+0000 7f75bffff640 1 -- 172.21.15.44:0/3491551122 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c0039070 con 0x7f75c80ce2e0 2024-04-02T13:20:56.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.547+0000 7f75bffff640 1 -- 172.21.15.44:0/3491551122 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c002cc60 con 0x7f75c80ce2e0 2024-04-02T13:20:56.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.547+0000 7f75bffff640 1 -- 172.21.15.44:0/3491551122 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c0034660 con 0x7f75c80ce2e0 2024-04-02T13:20:56.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.547+0000 7f75cf288640 1 -- 172.21.15.44:0/3491551122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c80ce2e0 msgr2=0x7f75c8165820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:56.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.547+0000 7f75cf288640 1 --2- 172.21.15.44:0/3491551122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c80ce2e0 0x7f75c8165820 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009d90 tx=0x7f75c002beb0 comp rx=0 tx=0).stop 2024-04-02T13:20:56.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.548+0000 7f75cf288640 1 -- 172.21.15.44:0/3491551122 shutdown_connections 2024-04-02T13:20:56.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.548+0000 7f75cf288640 1 --2- 172.21.15.44:0/3491551122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c80ce2e0 0x7f75c8165820 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:56.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.548+0000 7f75cf288640 1 -- 172.21.15.44:0/3491551122 >> 172.21.15.44:0/3491551122 conn(0x7f75c80cfcd0 msgr2=0x7f75c80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:56.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.548+0000 7f75cf288640 1 -- 172.21.15.44:0/3491551122 shutdown_connections 2024-04-02T13:20:56.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.548+0000 7f75cf288640 1 -- 172.21.15.44:0/3491551122 wait complete. 2024-04-02T13:20:56.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.549+0000 7f75cf288640 1 Processor -- start 2024-04-02T13:20:56.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.549+0000 7f75cf288640 1 -- start start 2024-04-02T13:20:56.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.549+0000 7f75cf288640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75c80b2d00 0x7f75c814d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:56.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.549+0000 7f75cf288640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c80ce2e0 0x7f75c8150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:56.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.549+0000 7f75cf288640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75c815aa00 0x7f75c814e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:56.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.549+0000 7f75cf288640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f75c80726d0 con 0x7f75c80ce2e0 2024-04-02T13:20:56.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.549+0000 7f75cf288640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f75c80723f0 con 0x7f75c80b2d00 2024-04-02T13:20:56.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.549+0000 7f75cf288640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f75c80cc750 con 0x7f75c815aa00 2024-04-02T13:20:56.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75cc823640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c80ce2e0 0x7f75c8150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:56.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75cc823640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c80ce2e0 0x7f75c8150770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45768/0 (socket says 172.21.15.44:45768) 2024-04-02T13:20:56.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75cc823640 1 -- 172.21.15.44:0/927661189 learned_addr learned my addr 172.21.15.44:0/927661189 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75cd024640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75c80b2d00 0x7f75c814d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75cd825640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75c815aa00 0x7f75c814e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75cc823640 1 -- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75c815aa00 msgr2=0x7f75c814e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75cc823640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75c815aa00 0x7f75c814e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75cc823640 1 -- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75c80b2d00 msgr2=0x7f75c814d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75cc823640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75c80b2d00 0x7f75c814d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75cc823640 1 -- 172.21.15.44:0/927661189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c814f0d0 con 0x7f75c80ce2e0 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75cd825640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75c815aa00 0x7f75c814e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75cc823640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c80ce2e0 0x7f75c8150770 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f75c400c450 tx=0x7f75c400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75bdffb640 1 -- 172.21.15.44:0/927661189 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c40052d0 con 0x7f75c80ce2e0 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75bdffb640 1 -- 172.21.15.44:0/927661189 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c4005470 con 0x7f75c80ce2e0 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75bdffb640 1 -- 172.21.15.44:0/927661189 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c4012870 con 0x7f75c80ce2e0 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.550+0000 7f75cf288640 1 -- 172.21.15.44:0/927661189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c814f3b0 con 0x7f75c80ce2e0 2024-04-02T13:20:56.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.551+0000 7f75cf288640 1 -- 172.21.15.44:0/927661189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c814f970 con 0x7f75c80ce2e0 2024-04-02T13:20:56.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.552+0000 7f75cf288640 1 -- 172.21.15.44:0/927661189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75c8074f20 con 0x7f75c80ce2e0 2024-04-02T13:20:56.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.552+0000 7f75bdffb640 1 -- 172.21.15.44:0/927661189 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f75c4012a10 con 0x7f75c80ce2e0 2024-04-02T13:20:56.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.552+0000 7f75bdffb640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75ac06c8f0 0x7f75ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:56.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.552+0000 7f75bdffb640 1 -- 172.21.15.44:0/927661189 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f75c408ba70 con 0x7f75c80ce2e0 2024-04-02T13:20:56.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.553+0000 7f75cd024640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75ac06c8f0 0x7f75ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:56.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.553+0000 7f75cd024640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75ac06c8f0 0x7f75ac06edb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f75b8000f60 tx=0x7f75b8007e40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:56.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.557+0000 7f75bdffb640 1 -- 172.21.15.44:0/927661189 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f75c4059ad0 con 0x7f75c80ce2e0 2024-04-02T13:20:56.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.784+0000 7f75cf288640 1 -- 172.21.15.44:0/927661189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f75c80684f0 con 0x7f75c80ce2e0 2024-04-02T13:20:56.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.785+0000 7f75bdffb640 1 -- 172.21.15.44:0/927661189 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f75c40593d0 con 0x7f75c80ce2e0 2024-04-02T13:20:56.786 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-02T13:20:56.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.787+0000 7f75cf288640 1 -- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75ac06c8f0 msgr2=0x7f75ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:56.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.787+0000 7f75cf288640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75ac06c8f0 0x7f75ac06edb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f75b8000f60 tx=0x7f75b8007e40 comp rx=0 tx=0).stop 2024-04-02T13:20:56.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.787+0000 7f75cf288640 1 -- 172.21.15.44:0/927661189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c80ce2e0 msgr2=0x7f75c8150770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:56.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.787+0000 7f75cf288640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c80ce2e0 0x7f75c8150770 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f75c400c450 tx=0x7f75c400c910 comp rx=0 tx=0).stop 2024-04-02T13:20:56.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.787+0000 7f75cf288640 1 -- 172.21.15.44:0/927661189 shutdown_connections 2024-04-02T13:20:56.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.787+0000 7f75cf288640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75ac06c8f0 0x7f75ac06edb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:56.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.787+0000 7f75cf288640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75c815aa00 0x7f75c814e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:56.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.787+0000 7f75cf288640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c80ce2e0 0x7f75c8150770 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:56.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.787+0000 7f75cf288640 1 --2- 172.21.15.44:0/927661189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75c80b2d00 0x7f75c814d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:56.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.787+0000 7f75cf288640 1 -- 172.21.15.44:0/927661189 >> 172.21.15.44:0/927661189 conn(0x7f75c80cfcd0 msgr2=0x7f75c80b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:56.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.787+0000 7f75cf288640 1 -- 172.21.15.44:0/927661189 shutdown_connections 2024-04-02T13:20:56.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:56.787+0000 7f75cf288640 1 -- 172.21.15.44:0/927661189 wait complete. 2024-04-02T13:20:56.799 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:20:56.799 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:20:56.799 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:20:56.799 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-02T13:20:56.799 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-02T13:20:56.799 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:20:56.799 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:20:56.799 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:20:56.800 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-02T13:20:56.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:56.800 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:20:56.815 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:20:56.815 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:20:56.815 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:20:56.815 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:56.815 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:20:56.815 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:56.815 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:20:56.815 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:56.815 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42722sec preferred_lft 42722sec 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:20:56.816 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:20:56.817 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:56.817 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:20:56.817 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:56.817 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link tentative 2024-04-02T13:20:56.817 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:56.817 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:20:56.817 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:20:56.817 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link tentative 2024-04-02T13:20:56.817 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:56.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:56.817 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:20:56.881 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:20:56.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:56.883 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:20:56.939 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:20:56.940 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:20:56.940 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:20:56.940 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:20:56.940 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:20:56.941 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:56.941 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:56.941 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:20:56.941 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:20:56.941 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:20:56.941 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:57.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:57.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:57.078 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:20:57.093 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:20:57.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:57.094 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list-id 2024-04-02T13:20:57.149 INFO:teuthology.orchestra.run.smithi044.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-02T13:20:57.149 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:57.149 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:57.150 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:20:57.150 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-02T13:20:57.150 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:57.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:57.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:57.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:57.272 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-02T13:20:57.299 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:20:57.299 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:20:57.299 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:20:57.299 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:57.299 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:20:57.299 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:57.299 INFO:teuthology.orchestra.run.smithi044.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:20:57.299 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 8a:15:07:ec:e2:77 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-02T13:20:57.299 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-02T13:20:57.299 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:57.299 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::8815:7ff:feec:e277/64 scope link tentative 2024-04-02T13:20:57.300 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:57.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:57.301 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-02T13:20:57.368 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:20:57.368 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:20:57.368 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:20:57.368 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:57.368 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:20:57.368 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:57.369 INFO:teuthology.orchestra.run.smithi044.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:20:57.369 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 8a:15:07:ec:e2:77 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-02T13:20:57.369 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-02T13:20:57.369 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:57.369 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::8815:7ff:feec:e277/64 scope link tentative 2024-04-02T13:20:57.369 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:57.371 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-02T13:20:57.371 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:57.371 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:57.371 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-02T13:20:57.371 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-02T13:20:57.371 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-02T13:20:57.371 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-02T13:20:57.372 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-02T13:20:57.372 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:57.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:57.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:57.536 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:57.536 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:20:57.536 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.1 up 2024-04-02T13:20:57.536 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set dev brx.1 master ceph-brx 2024-04-02T13:20:57.536 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:20:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:57.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:57.618 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-02T13:20:57.618 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:20:57.618 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-02T13:20:57.632 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-02T13:20:57.632 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:20:57.632 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-02T13:20:57.691 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:20:57.758 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:20:57.814 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:20:57.814 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:20:57.814 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:20:57.815 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:20:57.816 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:20:57.817 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:20:57.817 INFO:teuthology.orchestra.run.smithi044.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-02T13:20:57.817 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:20:57.817 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:20:57.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:57.817 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:20:57.871 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:20:57.871 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-04-02T13:20:57.872 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-04-02T13:20:57.914 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:20:57.941 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:20:57.997 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:20:57.997 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:20:57.997 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:20:57.997 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:20:57.997 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:20:57.997 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:20:57.997 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:20:57.997 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:20:57.997 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:20:57.997 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:20:57.997 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:20:57.998 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:20:57.998 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:20:57.998 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:20:57.998 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:20:57.998 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:20:57.998 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:20:57.998 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:20:57.998 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:20:57.998 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:20:57.998 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:20:57.998 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:20:57.998 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:20:57.998 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:20:57.999 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:20:57.999 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:20:57.999 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:20:57.999 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:20:57.999 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:20:57.999 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:20:57.999 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:20:57.999 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:20:57.999 INFO:teuthology.orchestra.run.smithi044.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-02T13:20:57.999 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:20:57.999 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:20:57.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:58.000 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:20:58.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi044.stderr:2024-04-02T13:20:57.999+0000 7fe4c3abb480 -1 init, newargv = 0x5571a509fa40 newargc=15 2024-04-02T13:20:58.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi044.stderr:ceph-fuse[48055]: starting ceph client 2024-04-02T13:20:58.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi044.stderr:ceph-fuse[48055]: starting fuse 2024-04-02T13:20:58.015 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:20:58.015 INFO:teuthology.orchestra.run.smithi044.stdout:44 2024-04-02T13:20:58.015 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-04-02T13:20:58.015 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:20:58.015 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:20:58.015 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:20:58.015 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:20:58.016 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:20:58.017 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:20:58.017 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:20:58.017 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:20:58.017 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:20:58.017 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:20:58.017 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:20:58.017 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:20:58.017 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.2")) 2024-04-02T13:20:58.017 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:20:58.136 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.2.48055.asok 2024-04-02T13:20:58.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:58.146 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.48055.asok 2024-04-02T13:20:58.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:58.147 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.48055.asok status 2024-04-02T13:20:58.283 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:20:58.283 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:20:58.283 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:20:58.283 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:20:58.283 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "2", 2024-04-02T13:20:58.283 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:20:58.283 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-02T13:20:58.283 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "48055", 2024-04-02T13:20:58.283 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:20:58.283 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:20:58.283 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 4854, 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 4854 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.2:0", 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 509085946 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:20:58.284 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:20:58.285 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.2:0", 2024-04-02T13:20:58.285 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 509085946 2024-04-02T13:20:58.285 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:20:58.285 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.4854 192.168.0.2:0/509085946", 2024-04-02T13:20:58.285 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.2:0/509085946", 2024-04-02T13:20:58.285 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:20:58.285 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 9, 2024-04-02T13:20:58.285 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 25, 2024-04-02T13:20:58.285 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:20:58.285 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:20:58.285 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:20:58.285 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:20:58.291 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:20:58.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.391+0000 7f48d8469640 1 Processor -- start 2024-04-02T13:20:58.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.391+0000 7f48d8469640 1 -- start start 2024-04-02T13:20:58.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.392+0000 7f48d8469640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48d015e4a0 0x7f48d00aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:58.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.392+0000 7f48d8469640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48d006d550 0x7f48d00d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:58.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.392+0000 7f48d8469640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48d00cc600 con 0x7f48d00d0c50 2024-04-02T13:20:58.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.392+0000 7f48d8469640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f48d00cc8e0 con 0x7f48d015e4a0 2024-04-02T13:20:58.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.392+0000 7f48d8469640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f48d00ccbc0 con 0x7f48d006d550 2024-04-02T13:20:58.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.392+0000 7f48d6205640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48d015e4a0 0x7f48d00aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:58.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.392+0000 7f48d6a06640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48d006d550 0x7f48d00d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:58.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.392+0000 7f48d6205640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48d015e4a0 0x7f48d00aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:33768/0 (socket says 172.21.15.44:33768) 2024-04-02T13:20:58.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.392+0000 7f48d6a06640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48d006d550 0x7f48d00d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:42042/0 (socket says 172.21.15.44:42042) 2024-04-02T13:20:58.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.392+0000 7f48d6205640 1 -- 172.21.15.44:0/3066925677 learned_addr learned my addr 172.21.15.44:0/3066925677 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:58.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.393+0000 7f48d6205640 1 -- 172.21.15.44:0/3066925677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48d006d550 msgr2=0x7f48d00d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:20:58.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.393+0000 7f48d6205640 1 --2- 172.21.15.44:0/3066925677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48d006d550 0x7f48d00d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:58.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.393+0000 7f48d6205640 1 -- 172.21.15.44:0/3066925677 >> v1:172.21.15.69:6789/0 conn(0x7f48d00d0c50 legacy=0x7f48d006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:58.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.393+0000 7f48d6205640 1 -- 172.21.15.44:0/3066925677 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d00ccea0 con 0x7f48d015e4a0 2024-04-02T13:20:58.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.394+0000 7f48d6205640 1 --2- 172.21.15.44:0/3066925677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48d015e4a0 0x7f48d00aebf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f48cc009870 tx=0x7f48cc02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b8727bc9e679186 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:58.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.394+0000 7f48d5203640 1 -- 172.21.15.44:0/3066925677 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc034070 con 0x7f48d015e4a0 2024-04-02T13:20:58.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.394+0000 7f48d5203640 1 -- 172.21.15.44:0/3066925677 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48cc030840 con 0x7f48d015e4a0 2024-04-02T13:20:58.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.394+0000 7f48d5203640 1 -- 172.21.15.44:0/3066925677 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc030c10 con 0x7f48d015e4a0 2024-04-02T13:20:58.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.394+0000 7f48d8469640 1 -- 172.21.15.44:0/3066925677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48d015e4a0 msgr2=0x7f48d00aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:58.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.394+0000 7f48d8469640 1 --2- 172.21.15.44:0/3066925677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48d015e4a0 0x7f48d00aebf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f48cc009870 tx=0x7f48cc02b980 comp rx=0 tx=0).stop 2024-04-02T13:20:58.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.394+0000 7f48d8469640 1 -- 172.21.15.44:0/3066925677 shutdown_connections 2024-04-02T13:20:58.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.394+0000 7f48d8469640 1 --2- 172.21.15.44:0/3066925677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48d006d550 0x7f48d00d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:58.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.394+0000 7f48d8469640 1 --2- 172.21.15.44:0/3066925677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48d015e4a0 0x7f48d00aebf0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:58.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.394+0000 7f48d8469640 1 -- 172.21.15.44:0/3066925677 >> 172.21.15.44:0/3066925677 conn(0x7f48d009aa20 msgr2=0x7f48d009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:58.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.395+0000 7f48d8469640 1 -- 172.21.15.44:0/3066925677 shutdown_connections 2024-04-02T13:20:58.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.395+0000 7f48d8469640 1 -- 172.21.15.44:0/3066925677 wait complete. 2024-04-02T13:20:58.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.396+0000 7f48d8469640 1 Processor -- start 2024-04-02T13:20:58.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.396+0000 7f48d8469640 1 -- start start 2024-04-02T13:20:58.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.396+0000 7f48d8469640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48d006d550 0x7f48d0155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:58.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.396+0000 7f48d8469640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48d00d0c50 0x7f48d00c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:58.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.396+0000 7f48d8469640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48d015e4a0 0x7f48d00c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:58.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.396+0000 7f48d8469640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f48d015e880 con 0x7f48d00d0c50 2024-04-02T13:20:58.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.396+0000 7f48d8469640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f48d00cd460 con 0x7f48d006d550 2024-04-02T13:20:58.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.396+0000 7f48d8469640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f48d00af130 con 0x7f48d015e4a0 2024-04-02T13:20:58.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48d5a04640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48d00d0c50 0x7f48d00c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:58.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48d5a04640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48d00d0c50 0x7f48d00c5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45772/0 (socket says 172.21.15.44:45772) 2024-04-02T13:20:58.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48d5a04640 1 -- 172.21.15.44:0/3783334022 learned_addr learned my addr 172.21.15.44:0/3783334022 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:20:58.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48d6a06640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48d015e4a0 0x7f48d00c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:58.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48d6205640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48d006d550 0x7f48d0155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:58.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48d5a04640 1 -- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48d015e4a0 msgr2=0x7f48d00c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48d5a04640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48d015e4a0 0x7f48d00c7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48d5a04640 1 -- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48d006d550 msgr2=0x7f48d0155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48d5a04640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48d006d550 0x7f48d0155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48d5a04640 1 -- 172.21.15.44:0/3783334022 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d00c6eb0 con 0x7f48d00d0c50 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48d5a04640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48d00d0c50 0x7f48d00c5b50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f48c000c470 tx=0x7f48c0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48c6ffd640 1 -- 172.21.15.44:0/3783334022 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c00052d0 con 0x7f48d00d0c50 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48c6ffd640 1 -- 172.21.15.44:0/3783334022 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48c0005470 con 0x7f48d00d0c50 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48d8469640 1 -- 172.21.15.44:0/3783334022 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d00c6610 con 0x7f48d00d0c50 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48c6ffd640 1 -- 172.21.15.44:0/3783334022 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c0007260 con 0x7f48d00d0c50 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.397+0000 7f48d8469640 1 -- 172.21.15.44:0/3783334022 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d00c6bd0 con 0x7f48d00d0c50 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.398+0000 7f48d8469640 1 -- 172.21.15.44:0/3783334022 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d00684f0 con 0x7f48d00d0c50 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.399+0000 7f48c6ffd640 1 -- 172.21.15.44:0/3783334022 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f48c0007400 con 0x7f48d00d0c50 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.399+0000 7f48c6ffd640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48b406c8f0 0x7f48b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:20:58.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.399+0000 7f48c6ffd640 1 -- 172.21.15.44:0/3783334022 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f48c008b2b0 con 0x7f48d00d0c50 2024-04-02T13:20:58.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.399+0000 7f48d6205640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48b406c8f0 0x7f48b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:20:58.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.400+0000 7f48d6205640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48b406c8f0 0x7f48b406edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f48cc036000 tx=0x7f48cc03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:20:58.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.403+0000 7f48c6ffd640 1 -- 172.21.15.44:0/3783334022 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f48c0058de0 con 0x7f48d00d0c50 2024-04-02T13:20:58.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.625+0000 7f48d8469640 1 -- 172.21.15.44:0/3783334022 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f48d00b7e40 con 0x7f48d00d0c50 2024-04-02T13:20:58.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.625+0000 7f48c6ffd640 1 -- 172.21.15.44:0/3783334022 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f48c005d0e0 con 0x7f48d00d0c50 2024-04-02T13:20:58.627 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-02T13:20:58.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.627+0000 7f48d8469640 1 -- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48b406c8f0 msgr2=0x7f48b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:58.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.627+0000 7f48d8469640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48b406c8f0 0x7f48b406edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f48cc036000 tx=0x7f48cc03b040 comp rx=0 tx=0).stop 2024-04-02T13:20:58.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.627+0000 7f48d8469640 1 -- 172.21.15.44:0/3783334022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48d00d0c50 msgr2=0x7f48d00c5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:20:58.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.627+0000 7f48d8469640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48d00d0c50 0x7f48d00c5b50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f48c000c470 tx=0x7f48c0001e70 comp rx=0 tx=0).stop 2024-04-02T13:20:58.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.627+0000 7f48d8469640 1 -- 172.21.15.44:0/3783334022 shutdown_connections 2024-04-02T13:20:58.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.627+0000 7f48d8469640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48b406c8f0 0x7f48b406edb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:58.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.627+0000 7f48d8469640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48d015e4a0 0x7f48d00c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:58.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.627+0000 7f48d8469640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48d00d0c50 0x7f48d00c5b50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:58.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.627+0000 7f48d8469640 1 --2- 172.21.15.44:0/3783334022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48d006d550 0x7f48d0155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:20:58.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.627+0000 7f48d8469640 1 -- 172.21.15.44:0/3783334022 >> 172.21.15.44:0/3783334022 conn(0x7f48d009aa20 msgr2=0x7f48d00d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:20:58.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.627+0000 7f48d8469640 1 -- 172.21.15.44:0/3783334022 shutdown_connections 2024-04-02T13:20:58.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:20:58.627+0000 7f48d8469640 1 -- 172.21.15.44:0/3783334022 wait complete. 2024-04-02T13:20:58.639 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:20:58.639 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:20:58.639 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-02T13:20:58.639 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-02T13:20:58.639 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-02T13:20:58.639 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:20:58.639 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:20:58.639 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:20:58.639 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-02T13:20:58.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:58.640 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-02T13:20:58.659 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:20:58.659 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:20:58.659 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:20:58.659 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:58.659 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-02T13:20:58.659 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:58.659 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:20:58.659 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:58.660 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-02T13:20:58.660 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:20:58.660 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:58.660 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-02T13:20:58.660 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:20:58.660 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:58.660 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:20:58.660 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:20:58.660 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T13:20:58.660 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42720sec preferred_lft 42720sec 2024-04-02T13:20:58.660 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-02T13:20:58.660 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:20:58.660 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-02T13:20:58.661 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:58.661 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-02T13:20:58.661 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add name ceph-brx type bridge 2024-04-02T13:20:58.661 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip addr flush dev ceph-brx 2024-04-02T13:20:58.661 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set ceph-brx up 2024-04-02T13:20:58.661 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-02T13:20:58.661 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-02T13:20:58.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:58.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:58.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:58.994 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:20:59.022 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-02T13:20:59.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:59.023 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-02T13:20:59.104 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-02T13:20:59.104 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:20:59.104 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T13:20:59.104 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T13:20:59.104 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:20:59.105 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:59.105 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-02T13:20:59.105 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T13:20:59.105 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T13:20:59.105 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T13:20:59.105 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-02T13:20:59.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:59.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:59.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:59.412 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-02T13:20:59.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:20:59.427 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list-id 2024-04-02T13:20:59.483 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:59.484 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-02T13:20:59.484 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T13:20:59.484 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-02T13:20:59.484 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-02T13:20:59.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:59.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:59.606 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-02T13:20:59.606 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:59.607 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-02T13:20:59.607 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-02T13:20:59.607 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-02T13:20:59.607 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-02T13:20:59.607 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-02T13:20:59.607 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-02T13:20:59.607 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-02T13:20:59.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:59.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:59.748 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:20:59.748 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-02T13:20:59.748 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-02T13:20:59.749 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set dev brx.0 master ceph-brx 2024-04-02T13:20:59.749 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-02T13:20:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:20:59.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:20:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:20:59.830 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-02T13:20:59.830 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:20:59.830 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-02T13:20:59.844 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-02T13:20:59.844 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:20:59.844 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-02T13:20:59.902 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-02T13:20:59.970 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:21:00.026 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-02T13:21:00.027 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-02T13:21:00.027 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-02T13:21:00.027 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-02T13:21:00.027 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-02T13:21:00.027 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-02T13:21:00.027 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-02T13:21:00.027 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-02T13:21:00.027 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-02T13:21:00.027 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-02T13:21:00.027 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-02T13:21:00.027 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-02T13:21:00.027 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:21:00.027 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-02T13:21:00.028 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-02T13:21:00.028 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-02T13:21:00.028 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-02T13:21:00.028 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:21:00.028 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-02T13:21:00.028 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-02T13:21:00.028 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:21:00.028 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:21:00.028 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:21:00.028 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-02T13:21:00.028 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-02T13:21:00.028 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-04-02T13:21:00.029 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-04-02T13:21:00.029 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-04-02T13:21:00.029 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-04-02T13:21:00.029 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-02T13:21:00.029 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T13:21:00.029 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T13:21:00.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:00.030 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-02T13:21:00.081 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:21:00.081 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-02T13:21:00.124 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-02T13:21:00.151 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:21:00.207 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-02T13:21:00.208 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-02T13:21:00.208 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-02T13:21:00.208 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-02T13:21:00.208 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-02T13:21:00.208 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-02T13:21:00.208 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-02T13:21:00.208 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-02T13:21:00.208 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-02T13:21:00.208 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-02T13:21:00.208 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-02T13:21:00.208 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-02T13:21:00.208 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:21:00.209 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-02T13:21:00.209 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-02T13:21:00.209 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-02T13:21:00.209 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-02T13:21:00.209 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:21:00.209 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-02T13:21:00.209 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-02T13:21:00.209 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:21:00.209 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:21:00.209 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:21:00.210 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-02T13:21:00.210 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-02T13:21:00.210 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-04-02T13:21:00.210 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-04-02T13:21:00.210 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-04-02T13:21:00.210 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-04-02T13:21:00.210 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-02T13:21:00.210 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T13:21:00.210 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T13:21:00.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:00.211 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-02T13:21:00.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:2024-04-02T13:21:00.208+0000 7fd3f9a4d480 -1 init, newargv = 0x5631fd685170 newargc=15 2024-04-02T13:21:00.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[43756]: starting ceph client 2024-04-02T13:21:00.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[43756]: starting fuse 2024-04-02T13:21:00.226 INFO:teuthology.orchestra.run.smithi069.stdout:41 2024-04-02T13:21:00.227 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:21:00.227 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:21:00.227 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-02T13:21:00.227 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-02T13:21:00.227 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-02T13:21:00.227 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-02T13:21:00.227 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-02T13:21:00.227 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-02T13:21:00.227 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-02T13:21:00.227 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-02T13:21:00.227 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-02T13:21:00.227 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-02T13:21:00.228 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.1")) 2024-04-02T13:21:00.229 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-02T13:21:00.346 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.1.43756.asok 2024-04-02T13:21:00.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:21:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:00.356 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43756.asok 2024-04-02T13:21:00.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:00.356 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43756.asok status 2024-04-02T13:21:00.492 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T13:21:00.492 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-02T13:21:00.492 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:21:00.492 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:21:00.492 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "1", 2024-04-02T13:21:00.492 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-02T13:21:00.492 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-02T13:21:00.492 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "43756", 2024-04-02T13:21:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-02T13:21:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T13:21:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-02T13:21:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-02T13:21:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 4619, 2024-04-02T13:21:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-02T13:21:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-02T13:21:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-02T13:21:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 4619 2024-04-02T13:21:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T13:21:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-02T13:21:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T13:21:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:21:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 3516426947 2024-04-02T13:21:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T13:21:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T13:21:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-02T13:21:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T13:21:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:21:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 3516426947 2024-04-02T13:21:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T13:21:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.4619 192.168.0.1:0/3516426947", 2024-04-02T13:21:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/3516426947", 2024-04-02T13:21:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-02T13:21:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 9, 2024-04-02T13:21:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 25, 2024-04-02T13:21:00.495 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:21:00.495 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-02T13:21:00.495 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-02T13:21:00.495 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T13:21:00.500 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:21:00.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.613+0000 7ff05b54a640 1 Processor -- start 2024-04-02T13:21:00.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.613+0000 7ff05b54a640 1 -- start start 2024-04-02T13:21:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.614+0000 7ff05b54a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff0540d28f0 0x7ff0540d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.614+0000 7ff05b54a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0540b47b0 0x7ff0540b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.614+0000 7ff05b54a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0540d4d70 con 0x7ff0540b5200 2024-04-02T13:21:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.614+0000 7ff05b54a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff0540d5050 con 0x7ff0540d28f0 2024-04-02T13:21:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.614+0000 7ff05b54a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0540d23a0 con 0x7ff0540b47b0 2024-04-02T13:21:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.614+0000 7ff058ae5640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff0540b5200 0x7ff0540b40a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:55294/0 (socket says 172.21.15.44:55294) 2024-04-02T13:21:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.614+0000 7ff058ae5640 1 -- 172.21.15.44:0/735178092 learned_addr learned my addr 172.21.15.44:0/735178092 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.614+0000 7ff0592e6640 1 --2- 172.21.15.44:0/735178092 >> v2:172.21.15.69:3300/0 conn(0x7ff0540d28f0 0x7ff0540d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.614+0000 7ff059ae7640 1 --2- 172.21.15.44:0/735178092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0540b47b0 0x7ff0540b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.614+0000 7ff04bfff640 1 -- 172.21.15.44:0/735178092 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4277102513 0 0) 0x7ff0540d4d70 con 0x7ff0540b5200 2024-04-02T13:21:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.615+0000 7ff04bfff640 1 -- 172.21.15.44:0/735178092 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff040003520 con 0x7ff0540b5200 2024-04-02T13:21:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.615+0000 7ff059ae7640 1 -- 172.21.15.44:0/735178092 >> v2:172.21.15.69:3300/0 conn(0x7ff0540d28f0 msgr2=0x7ff0540d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.615+0000 7ff059ae7640 1 --2- 172.21.15.44:0/735178092 >> v2:172.21.15.69:3300/0 conn(0x7ff0540d28f0 0x7ff0540d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.615+0000 7ff059ae7640 1 -- 172.21.15.44:0/735178092 >> v1:172.21.15.44:6789/0 conn(0x7ff0540b5200 legacy=0x7ff0540b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.615+0000 7ff059ae7640 1 -- 172.21.15.44:0/735178092 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0540d2030 con 0x7ff0540b47b0 2024-04-02T13:21:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.615+0000 7ff0592e6640 1 --2- 172.21.15.44:0/735178092 >> v2:172.21.15.69:3300/0 conn(0x7ff0540d28f0 0x7ff0540d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.615+0000 7ff059ae7640 1 --2- 172.21.15.44:0/735178092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0540b47b0 0x7ff0540b7120 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff050009d90 tx=0x7ff05002bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88f8ee5879ee8720 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.616+0000 7ff04bfff640 1 -- 172.21.15.44:0/735178092 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff050039070 con 0x7ff0540b47b0 2024-04-02T13:21:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.616+0000 7ff04bfff640 1 -- 172.21.15.44:0/735178092 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff05002cb30 con 0x7ff0540b47b0 2024-04-02T13:21:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.616+0000 7ff04bfff640 1 -- 172.21.15.44:0/735178092 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff050034440 con 0x7ff0540b47b0 2024-04-02T13:21:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.616+0000 7ff05b54a640 1 -- 172.21.15.44:0/735178092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0540b47b0 msgr2=0x7ff0540b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.616+0000 7ff05b54a640 1 --2- 172.21.15.44:0/735178092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0540b47b0 0x7ff0540b7120 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff050009d90 tx=0x7ff05002bd80 comp rx=0 tx=0).stop 2024-04-02T13:21:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.616+0000 7ff05b54a640 1 -- 172.21.15.44:0/735178092 shutdown_connections 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.616+0000 7ff05b54a640 1 --2- 172.21.15.44:0/735178092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0540b47b0 0x7ff0540b7120 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.616+0000 7ff05b54a640 1 --2- 172.21.15.44:0/735178092 >> v2:172.21.15.69:3300/0 conn(0x7ff0540d28f0 0x7ff0540d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.616+0000 7ff05b54a640 1 -- 172.21.15.44:0/735178092 >> 172.21.15.44:0/735178092 conn(0x7ff0540d4120 msgr2=0x7ff0540d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.616+0000 7ff05b54a640 1 -- 172.21.15.44:0/735178092 shutdown_connections 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.616+0000 7ff05b54a640 1 -- 172.21.15.44:0/735178092 wait complete. 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.617+0000 7ff05b54a640 1 Processor -- start 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.618+0000 7ff05b54a640 1 -- start start 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.618+0000 7ff05b54a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0540b47b0 0x7ff0540b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.618+0000 7ff05b54a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0540b5200 0x7ff0540cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.618+0000 7ff05b54a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0540d28f0 0x7ff05415f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.618+0000 7ff05b54a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff0540d1260 con 0x7ff0540b5200 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.618+0000 7ff05b54a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0540b5020 con 0x7ff0540b47b0 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.618+0000 7ff05b54a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0540d13e0 con 0x7ff0540d28f0 2024-04-02T13:21:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.618+0000 7ff058ae5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0540b5200 0x7ff0540cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.618+0000 7ff058ae5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0540b5200 0x7ff0540cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45774/0 (socket says 172.21.15.44:45774) 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.618+0000 7ff058ae5640 1 -- 172.21.15.44:0/544431334 learned_addr learned my addr 172.21.15.44:0/544431334 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.619+0000 7ff059ae7640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0540d28f0 0x7ff05415f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.619+0000 7ff0592e6640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0540b47b0 0x7ff0540b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.619+0000 7ff058ae5640 1 -- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0540d28f0 msgr2=0x7ff05415f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.619+0000 7ff058ae5640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0540d28f0 0x7ff05415f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.619+0000 7ff058ae5640 1 -- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0540b47b0 msgr2=0x7ff0540b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.619+0000 7ff058ae5640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0540b47b0 0x7ff0540b29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.619+0000 7ff058ae5640 1 -- 172.21.15.44:0/544431334 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0540cb920 con 0x7ff0540b5200 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.619+0000 7ff058ae5640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0540b5200 0x7ff0540cb1f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff04400c3c0 tx=0x7ff04400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.619+0000 7ff049ffb640 1 -- 172.21.15.44:0/544431334 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff044017070 con 0x7ff0540b5200 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.619+0000 7ff049ffb640 1 -- 172.21.15.44:0/544431334 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff044005ce0 con 0x7ff0540b5200 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.619+0000 7ff05b54a640 1 -- 172.21.15.44:0/544431334 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0540cbc00 con 0x7ff0540b5200 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.619+0000 7ff049ffb640 1 -- 172.21.15.44:0/544431334 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0440072f0 con 0x7ff0540b5200 2024-04-02T13:21:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.619+0000 7ff05b54a640 1 -- 172.21.15.44:0/544431334 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0540cc1c0 con 0x7ff0540b5200 2024-04-02T13:21:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.620+0000 7ff05b54a640 1 -- 172.21.15.44:0/544431334 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0540684f0 con 0x7ff0540b5200 2024-04-02T13:21:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.621+0000 7ff049ffb640 1 -- 172.21.15.44:0/544431334 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff04401a070 con 0x7ff0540b5200 2024-04-02T13:21:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.621+0000 7ff049ffb640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff03806c8f0 0x7ff03806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.621+0000 7ff049ffb640 1 -- 172.21.15.44:0/544431334 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff04408b0f0 con 0x7ff0540b5200 2024-04-02T13:21:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.621+0000 7ff0592e6640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff03806c8f0 0x7ff03806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.622+0000 7ff0592e6640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff03806c8f0 0x7ff03806edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff04c009560 tx=0x7ff04c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.626+0000 7ff049ffb640 1 -- 172.21.15.44:0/544431334 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff044058c20 con 0x7ff0540b5200 2024-04-02T13:21:00.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.848+0000 7ff05b54a640 1 -- 172.21.15.44:0/544431334 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff0540b8110 con 0x7ff0540b5200 2024-04-02T13:21:00.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.848+0000 7ff049ffb640 1 -- 172.21.15.44:0/544431334 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7ff04405cf20 con 0x7ff0540b5200 2024-04-02T13:21:00.850 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-02T13:21:00.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.850+0000 7ff05b54a640 1 -- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff03806c8f0 msgr2=0x7ff03806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:00.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.850+0000 7ff05b54a640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff03806c8f0 0x7ff03806edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff04c009560 tx=0x7ff04c009190 comp rx=0 tx=0).stop 2024-04-02T13:21:00.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.850+0000 7ff05b54a640 1 -- 172.21.15.44:0/544431334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0540b5200 msgr2=0x7ff0540cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:00.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.850+0000 7ff05b54a640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0540b5200 0x7ff0540cb1f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff04400c3c0 tx=0x7ff04400c880 comp rx=0 tx=0).stop 2024-04-02T13:21:00.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.850+0000 7ff05b54a640 1 -- 172.21.15.44:0/544431334 shutdown_connections 2024-04-02T13:21:00.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.850+0000 7ff05b54a640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff03806c8f0 0x7ff03806edb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:00.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.850+0000 7ff05b54a640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0540d28f0 0x7ff05415f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:00.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.850+0000 7ff05b54a640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0540b5200 0x7ff0540cb1f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:00.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.850+0000 7ff05b54a640 1 --2- 172.21.15.44:0/544431334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0540b47b0 0x7ff0540b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:00.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.850+0000 7ff05b54a640 1 -- 172.21.15.44:0/544431334 >> 172.21.15.44:0/544431334 conn(0x7ff0540d4120 msgr2=0x7ff0540b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:00.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.850+0000 7ff05b54a640 1 -- 172.21.15.44:0/544431334 shutdown_connections 2024-04-02T13:21:00.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:00.850+0000 7ff05b54a640 1 -- 172.21.15.44:0/544431334 wait complete. 2024-04-02T13:21:00.862 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:21:00.862 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:21:00.862 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-02T13:21:00.862 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-02T13:21:00.862 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-02T13:21:00.863 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:21:00.863 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:21:00.863 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:21:00.863 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-02T13:21:00.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:00.863 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-02T13:21:00.879 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:21:00.880 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:21:00.880 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:21:00.880 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:00.880 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-02T13:21:00.880 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:00.880 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:21:00.880 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:21:00.880 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-02T13:21:00.880 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:21:00.880 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:21:00.880 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-02T13:21:00.880 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:21:00.880 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42718sec preferred_lft 42718sec 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether c6:37:bc:f0:53:98 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::8082:d7ff:feaa:f4aa/64 scope link 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether c6:37:bc:f0:53:98 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T13:21:00.881 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::c437:bcff:fef0:5398/64 scope link tentative 2024-04-02T13:21:00.882 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:00.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:00.882 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:21:00.946 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-02T13:21:00.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:00.948 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-02T13:21:01.003 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-02T13:21:01.003 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:21:01.004 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T13:21:01.004 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T13:21:01.004 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:21:01.004 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:21:01.004 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-02T13:21:01.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T13:21:01.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T13:21:01.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T13:21:01.004 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-02T13:21:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:21:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:21:01.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:21:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:01.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:01.138 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-02T13:21:01.153 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-02T13:21:01.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:01.154 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list-id 2024-04-02T13:21:01.208 INFO:teuthology.orchestra.run.smithi069.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-02T13:21:01.208 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:21:01.209 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-02T13:21:01.209 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-02T13:21:01.209 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-02T13:21:01.209 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-02T13:21:01.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:21:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:21:01.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:21:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:01.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:01.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-02T13:21:01.357 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:21:01.357 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:21:01.357 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:21:01.357 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:01.358 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-02T13:21:01.358 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:01.358 INFO:teuthology.orchestra.run.smithi069.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:21:01.358 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 76:90:bd:16:ea:a4 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-02T13:21:01.358 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-02T13:21:01.358 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:01.358 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::7490:bdff:fe16:eaa4/64 scope link 2024-04-02T13:21:01.358 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:01.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:01.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-02T13:21:01.426 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:21:01.426 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:21:01.426 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:21:01.426 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:01.426 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-02T13:21:01.426 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:01.426 INFO:teuthology.orchestra.run.smithi069.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:21:01.426 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 76:90:bd:16:ea:a4 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-02T13:21:01.426 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-02T13:21:01.427 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:01.427 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::7490:bdff:fe16:eaa4/64 scope link 2024-04-02T13:21:01.427 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:01.428 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-02T13:21:01.428 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:21:01.428 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-02T13:21:01.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-02T13:21:01.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-02T13:21:01.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-02T13:21:01.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-02T13:21:01.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-02T13:21:01.428 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-02T13:21:01.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:21:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:21:01.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:21:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:01.592 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:21:01.592 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-02T13:21:01.592 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.1 up 2024-04-02T13:21:01.592 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set dev brx.1 master ceph-brx 2024-04-02T13:21:01.593 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-02T13:21:01.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:21:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:21:01.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:21:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:01.674 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-02T13:21:01.674 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:21:01.674 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-02T13:21:01.688 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-02T13:21:01.688 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:21:01.688 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-02T13:21:01.743 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-02T13:21:01.809 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:21:01.865 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-02T13:21:01.865 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-02T13:21:01.866 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-02T13:21:01.866 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-02T13:21:01.866 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-02T13:21:01.866 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-02T13:21:01.866 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-02T13:21:01.866 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-02T13:21:01.866 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-02T13:21:01.866 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-02T13:21:01.866 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-02T13:21:01.866 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-02T13:21:01.866 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:21:01.866 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-02T13:21:01.866 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-04-02T13:21:01.867 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-04-02T13:21:01.868 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-02T13:21:01.868 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T13:21:01.868 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T13:21:01.868 INFO:teuthology.orchestra.run.smithi069.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-02T13:21:01.868 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-02T13:21:01.868 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-02T13:21:01.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:01.868 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-02T13:21:01.920 INFO:teuthology.orchestra.run.smithi069.stdout:41 2024-04-02T13:21:01.921 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-04-02T13:21:01.921 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-04-02T13:21:01.963 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-02T13:21:01.989 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:21:02.045 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-02T13:21:02.045 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-02T13:21:02.045 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-02T13:21:02.045 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-02T13:21:02.045 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-02T13:21:02.045 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-02T13:21:02.045 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-02T13:21:02.045 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-02T13:21:02.045 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-02T13:21:02.045 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-02T13:21:02.045 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-02T13:21:02.045 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-02T13:21:02.046 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-04-02T13:21:02.047 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-04-02T13:21:02.047 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-04-02T13:21:02.047 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-04-02T13:21:02.047 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-02T13:21:02.047 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T13:21:02.047 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T13:21:02.047 INFO:teuthology.orchestra.run.smithi069.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-02T13:21:02.047 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-02T13:21:02.047 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-02T13:21:02.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:02.048 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-02T13:21:02.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi069.stderr:2024-04-02T13:21:02.046+0000 7f616d20b480 -1 init, newargv = 0x55c468d71260 newargc=15 2024-04-02T13:21:02.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi069.stderr:ceph-fuse[44213]: starting ceph client 2024-04-02T13:21:02.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi069.stderr:ceph-fuse[44213]: starting fuse 2024-04-02T13:21:02.067 INFO:teuthology.orchestra.run.smithi069.stdout:41 2024-04-02T13:21:02.067 INFO:teuthology.orchestra.run.smithi069.stdout:44 2024-04-02T13:21:02.067 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-04-02T13:21:02.067 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:21:02.068 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-02T13:21:02.068 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-02T13:21:02.068 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-02T13:21:02.068 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-02T13:21:02.068 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-02T13:21:02.068 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-02T13:21:02.068 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-02T13:21:02.068 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-02T13:21:02.068 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-02T13:21:02.068 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-02T13:21:02.068 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-02T13:21:02.068 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-02T13:21:02.069 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-02T13:21:02.069 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-02T13:21:02.069 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-02T13:21:02.069 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-02T13:21:02.069 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:21:02.069 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:21:02.069 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:21:02.069 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-02T13:21:02.069 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-02T13:21:02.069 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-02T13:21:02.069 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:21:02.069 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-02T13:21:02.069 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.3")) 2024-04-02T13:21:02.070 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-02T13:21:02.188 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.3.44213.asok 2024-04-02T13:21:02.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T13:21:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:02.198 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44213.asok 2024-04-02T13:21:02.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:02.199 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44213.asok status 2024-04-02T13:21:02.334 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T13:21:02.334 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-02T13:21:02.334 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:21:02.334 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:21:02.334 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "3", 2024-04-02T13:21:02.334 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-02T13:21:02.334 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-02T13:21:02.334 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "44213", 2024-04-02T13:21:02.334 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-02T13:21:02.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T13:21:02.334 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 4631, 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 4631 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.2:0", 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 832070827 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-02T13:21:02.335 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T13:21:02.336 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.2:0", 2024-04-02T13:21:02.336 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 832070827 2024-04-02T13:21:02.336 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T13:21:02.336 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.4631 192.168.0.2:0/832070827", 2024-04-02T13:21:02.336 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.2:0/832070827", 2024-04-02T13:21:02.336 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-02T13:21:02.336 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 9, 2024-04-02T13:21:02.336 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 25, 2024-04-02T13:21:02.336 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:21:02.336 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-02T13:21:02.336 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-02T13:21:02.336 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T13:21:02.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:02.342 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:21:02.343 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:21:02.362 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:21:02.362 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:21:02.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:02.362 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:21:02.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:02.433 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:21:02.433 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.2 2024-04-02T13:21:02.490 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:21:02.491 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-04-02T13:21:02.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:02.491 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-02T13:21:02.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:02.561 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-02T13:21:02.561 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-02T13:21:02.577 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-02T13:21:02.577 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-02T13:21:02.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:02.577 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-02T13:21:02.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:02.644 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-02T13:21:02.644 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.3 2024-04-02T13:21:02.700 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-02T13:21:02.700 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-04-02T13:21:02.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:02.700 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-02T13:21:02.768 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-02T13:21:02.787 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:02.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.898+0000 7f8a1eb32640 1 Processor -- start 2024-04-02T13:21:02.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.898+0000 7f8a1eb32640 1 -- start start 2024-04-02T13:21:02.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1eb32640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8a180cece0 0x7f8a180cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:02.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1eb32640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a180d09f0 0x7f8a1806d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:02.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1eb32640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a1806ece0 0x7f8a1815eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:02.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1eb32640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f8a180cf600 con 0x7f8a180cece0 2024-04-02T13:21:02.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1eb32640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8a1806dc40 con 0x7f8a1806ece0 2024-04-02T13:21:02.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1eb32640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8a1806df20 con 0x7f8a180d09f0 2024-04-02T13:21:02.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1c8ce640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8a180cece0 0x7f8a180cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:02.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1c8ce640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8a180cece0 0x7f8a180cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45782/0 (socket says 172.21.15.44:45782) 2024-04-02T13:21:02.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1c8ce640 1 -- 172.21.15.44:0/3794164756 learned_addr learned my addr 172.21.15.44:0/3794164756 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:02.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1d0cf640 1 --2- 172.21.15.44:0/3794164756 >> v2:172.21.15.69:3300/0 conn(0x7f8a1806ece0 0x7f8a1815eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:02.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a17fff640 1 --2- 172.21.15.44:0/3794164756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a180d09f0 0x7f8a1806d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:02.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1c8ce640 1 -- 172.21.15.44:0/3794164756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a180d09f0 msgr2=0x7f8a1806d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:02.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1c8ce640 1 --2- 172.21.15.44:0/3794164756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a180d09f0 0x7f8a1806d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:02.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1c8ce640 1 -- 172.21.15.44:0/3794164756 >> v2:172.21.15.69:3300/0 conn(0x7f8a1806ece0 msgr2=0x7f8a1815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:02.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1c8ce640 1 --2- 172.21.15.44:0/3794164756 >> v2:172.21.15.69:3300/0 conn(0x7f8a1806ece0 0x7f8a1815eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:02.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.899+0000 7f8a1c8ce640 1 -- 172.21.15.44:0/3794164756 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a1806e200 con 0x7f8a180cece0 2024-04-02T13:21:02.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.900+0000 7f8a1c8ce640 1 --2- 172.21.15.44:0/3794164756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a180cece0 0x7f8a180cf0c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8a180cf7e0 tx=0x7f8a0c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=740ddbd6a3820eca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:02.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.900+0000 7f8a177fe640 1 -- 172.21.15.44:0/3794164756 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c035020 con 0x7f8a180cece0 2024-04-02T13:21:02.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.900+0000 7f8a177fe640 1 -- 172.21.15.44:0/3794164756 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a0c031640 con 0x7f8a180cece0 2024-04-02T13:21:02.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.900+0000 7f8a177fe640 1 -- 172.21.15.44:0/3794164756 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c02d040 con 0x7f8a180cece0 2024-04-02T13:21:02.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.901+0000 7f8a1eb32640 1 -- 172.21.15.44:0/3794164756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a180cece0 msgr2=0x7f8a180cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:02.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.901+0000 7f8a1eb32640 1 --2- 172.21.15.44:0/3794164756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a180cece0 0x7f8a180cf0c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8a180cf7e0 tx=0x7f8a0c02b8f0 comp rx=0 tx=0).stop 2024-04-02T13:21:02.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.901+0000 7f8a1eb32640 1 -- 172.21.15.44:0/3794164756 shutdown_connections 2024-04-02T13:21:02.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.901+0000 7f8a1eb32640 1 --2- 172.21.15.44:0/3794164756 >> v2:172.21.15.69:3300/0 conn(0x7f8a1806ece0 0x7f8a1815eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:02.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.901+0000 7f8a1eb32640 1 --2- 172.21.15.44:0/3794164756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a180d09f0 0x7f8a1806d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:02.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.901+0000 7f8a1eb32640 1 --2- 172.21.15.44:0/3794164756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a180cece0 0x7f8a180cf0c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:02.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.901+0000 7f8a1eb32640 1 -- 172.21.15.44:0/3794164756 >> 172.21.15.44:0/3794164756 conn(0x7f8a180cfcd0 msgr2=0x7f8a180d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:02.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.901+0000 7f8a1eb32640 1 -- 172.21.15.44:0/3794164756 shutdown_connections 2024-04-02T13:21:02.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.901+0000 7f8a1eb32640 1 -- 172.21.15.44:0/3794164756 wait complete. 2024-04-02T13:21:02.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.902+0000 7f8a1eb32640 1 Processor -- start 2024-04-02T13:21:02.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.902+0000 7f8a1eb32640 1 -- start start 2024-04-02T13:21:02.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1eb32640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a1806ece0 0x7f8a1814c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:02.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1eb32640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a180cece0 0x7f8a1809d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:02.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1eb32640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a180d09f0 0x7f8a181f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:02.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1eb32640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1806d810 con 0x7f8a180d09f0 2024-04-02T13:21:02.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1eb32640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1806e7c0 con 0x7f8a180cece0 2024-04-02T13:21:02.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1eb32640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8a1806d990 con 0x7f8a1806ece0 2024-04-02T13:21:02.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1d0cf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a180d09f0 0x7f8a181f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:02.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1d0cf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a180d09f0 0x7f8a181f7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45794/0 (socket says 172.21.15.44:45794) 2024-04-02T13:21:02.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1d0cf640 1 -- 172.21.15.44:0/2034805739 learned_addr learned my addr 172.21.15.44:0/2034805739 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a17fff640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a180cece0 0x7f8a1809d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1c8ce640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a1806ece0 0x7f8a1814c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1d0cf640 1 -- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a1806ece0 msgr2=0x7f8a1814c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1d0cf640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a1806ece0 0x7f8a1814c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1d0cf640 1 -- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a180cece0 msgr2=0x7f8a1809d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1d0cf640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a180cece0 0x7f8a1809d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.903+0000 7f8a1d0cf640 1 -- 172.21.15.44:0/2034805739 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a1814ccf0 con 0x7f8a180d09f0 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.904+0000 7f8a1d0cf640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a180d09f0 0x7f8a181f7aa0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8a10007e60 tx=0x7f8a1000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.904+0000 7f8a157fa640 1 -- 172.21.15.44:0/2034805739 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a10014420 con 0x7f8a180d09f0 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.904+0000 7f8a157fa640 1 -- 172.21.15.44:0/2034805739 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a10019070 con 0x7f8a180d09f0 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.904+0000 7f8a1eb32640 1 -- 172.21.15.44:0/2034805739 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a1809d860 con 0x7f8a180d09f0 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.904+0000 7f8a157fa640 1 -- 172.21.15.44:0/2034805739 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a10014800 con 0x7f8a180d09f0 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.904+0000 7f8a1eb32640 1 -- 172.21.15.44:0/2034805739 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a181f8840 con 0x7f8a180d09f0 2024-04-02T13:21:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.905+0000 7f8a1eb32640 1 -- 172.21.15.44:0/2034805739 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a180684f0 con 0x7f8a180d09f0 2024-04-02T13:21:02.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.905+0000 7f8a157fa640 1 -- 172.21.15.44:0/2034805739 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8a100149a0 con 0x7f8a180d09f0 2024-04-02T13:21:02.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.906+0000 7f8a157fa640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f89fc06c8f0 0x7f89fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:02.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.906+0000 7f8a157fa640 1 -- 172.21.15.44:0/2034805739 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8a1008c6b0 con 0x7f8a180d09f0 2024-04-02T13:21:02.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.906+0000 7f8a1c8ce640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f89fc06c8f0 0x7f89fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:02.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.906+0000 7f8a1c8ce640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f89fc06c8f0 0x7f89fc06edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c036000 tx=0x7f8a0c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:02.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:02.911+0000 7f8a157fa640 1 -- 172.21.15.44:0/2034805739 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8a1814ccf0 con 0x7f8a180d09f0 2024-04-02T13:21:03.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.096+0000 7f8a1eb32640 1 -- 172.21.15.44:0/2034805739 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a180ce230 con 0x7f8a180d09f0 2024-04-02T13:21:03.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.098+0000 7f8a157fa640 1 -- 172.21.15.44:0/2034805739 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f8a1005a7a0 con 0x7f8a180d09f0 2024-04-02T13:21:03.099 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:03.100 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.100+0000 7f8a1eb32640 1 -- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f89fc06c8f0 msgr2=0x7f89fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.100+0000 7f8a1eb32640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f89fc06c8f0 0x7f89fc06edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c036000 tx=0x7f8a0c03c040 comp rx=0 tx=0).stop 2024-04-02T13:21:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.100+0000 7f8a1eb32640 1 -- 172.21.15.44:0/2034805739 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a180d09f0 msgr2=0x7f8a181f7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.100+0000 7f8a1eb32640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a180d09f0 0x7f8a181f7aa0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8a10007e60 tx=0x7f8a1000e4b0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.100+0000 7f8a1eb32640 1 -- 172.21.15.44:0/2034805739 shutdown_connections 2024-04-02T13:21:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.100+0000 7f8a1eb32640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f89fc06c8f0 0x7f89fc06edb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.100+0000 7f8a1eb32640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a180d09f0 0x7f8a181f7aa0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.100+0000 7f8a1eb32640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a180cece0 0x7f8a1809d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.100+0000 7f8a1eb32640 1 --2- 172.21.15.44:0/2034805739 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a1806ece0 0x7f8a1814c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.100+0000 7f8a1eb32640 1 -- 172.21.15.44:0/2034805739 >> 172.21.15.44:0/2034805739 conn(0x7f8a180cfcd0 msgr2=0x7f8a18157670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:03.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.100+0000 7f8a1eb32640 1 -- 172.21.15.44:0/2034805739 shutdown_connections 2024-04-02T13:21:03.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.100+0000 7f8a1eb32640 1 -- 172.21.15.44:0/2034805739 wait complete. 2024-04-02T13:21:03.112 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:03.113 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:03.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.217+0000 7fc9e9808640 1 Processor -- start 2024-04-02T13:21:03.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.217+0000 7fc9e9808640 1 -- start start 2024-04-02T13:21:03.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.218+0000 7fc9e9808640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc9e415e640 0x7fc9e40d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.218+0000 7fc9e9808640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc9e40cec60 0x7fc9e40d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.218+0000 7fc9e9808640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e40cf120 con 0x7fc9e415e260 2024-04-02T13:21:03.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.218+0000 7fc9e9808640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc9e40b5a50 con 0x7fc9e415e640 2024-04-02T13:21:03.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.218+0000 7fc9e9808640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc9e40d01c0 con 0x7fc9e40cec60 2024-04-02T13:21:03.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.218+0000 7fc9e27fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc9e415e640 0x7fc9e40d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.218+0000 7fc9e27fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc9e415e640 0x7fc9e40d0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45802/0 (socket says 172.21.15.44:45802) 2024-04-02T13:21:03.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.218+0000 7fc9e27fc640 1 -- 172.21.15.44:0/1999409235 learned_addr learned my addr 172.21.15.44:0/1999409235 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:03.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.218+0000 7fc9e37fe640 1 --2- 172.21.15.44:0/1999409235 >> v2:172.21.15.69:3300/0 conn(0x7fc9e40cec60 0x7fc9e40d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.218+0000 7fc9e27fc640 1 -- 172.21.15.44:0/1999409235 >> v2:172.21.15.69:3300/0 conn(0x7fc9e40cec60 msgr2=0x7fc9e40d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:03.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.218+0000 7fc9e27fc640 1 --2- 172.21.15.44:0/1999409235 >> v2:172.21.15.69:3300/0 conn(0x7fc9e40cec60 0x7fc9e40d1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.218+0000 7fc9e27fc640 1 -- 172.21.15.44:0/1999409235 >> v1:172.21.15.69:6789/0 conn(0x7fc9e415e260 legacy=0x7fc9e40d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.218+0000 7fc9e27fc640 1 -- 172.21.15.44:0/1999409235 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e40cf400 con 0x7fc9e415e640 2024-04-02T13:21:03.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.219+0000 7fc9e27fc640 1 --2- 172.21.15.44:0/1999409235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e415e640 0x7fc9e40d0ad0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc9e40ce6c0 tx=0x7fc9cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2943347752aec8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:03.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.219+0000 7fc9e1ffb640 1 -- 172.21.15.44:0/1999409235 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc035020 con 0x7fc9e415e640 2024-04-02T13:21:03.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.219+0000 7fc9e1ffb640 1 -- 172.21.15.44:0/1999409235 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9cc031870 con 0x7fc9e415e640 2024-04-02T13:21:03.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.219+0000 7fc9e1ffb640 1 -- 172.21.15.44:0/1999409235 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc031c40 con 0x7fc9e415e640 2024-04-02T13:21:03.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.220+0000 7fc9e9808640 1 -- 172.21.15.44:0/1999409235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e415e640 msgr2=0x7fc9e40d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.220+0000 7fc9e9808640 1 --2- 172.21.15.44:0/1999409235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e415e640 0x7fc9e40d0ad0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc9e40ce6c0 tx=0x7fc9cc02b910 comp rx=0 tx=0).stop 2024-04-02T13:21:03.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.220+0000 7fc9e9808640 1 -- 172.21.15.44:0/1999409235 shutdown_connections 2024-04-02T13:21:03.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.220+0000 7fc9e9808640 1 --2- 172.21.15.44:0/1999409235 >> v2:172.21.15.69:3300/0 conn(0x7fc9e40cec60 0x7fc9e40d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.220+0000 7fc9e9808640 1 --2- 172.21.15.44:0/1999409235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e415e640 0x7fc9e40d0ad0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.220+0000 7fc9e9808640 1 -- 172.21.15.44:0/1999409235 >> 172.21.15.44:0/1999409235 conn(0x7fc9e409d3c0 msgr2=0x7fc9e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.220+0000 7fc9e9808640 1 -- 172.21.15.44:0/1999409235 shutdown_connections 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.220+0000 7fc9e9808640 1 -- 172.21.15.44:0/1999409235 wait complete. 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.221+0000 7fc9e9808640 1 Processor -- start 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.221+0000 7fc9e9808640 1 -- start start 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e9808640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9e40cec60 0x7fc9e4157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e9808640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9e415e260 0x7fc9e40cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e9808640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e415e640 0x7fc9e40cc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e9808640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e40cfca0 con 0x7fc9e415e640 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e9808640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e40cf9c0 con 0x7fc9e40cec60 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e9808640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e40d3070 con 0x7fc9e415e260 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e415e640 0x7fc9e40cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e415e640 0x7fc9e40cc570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45818/0 (socket says 172.21.15.44:45818) 2024-04-02T13:21:03.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e37fe640 1 -- 172.21.15.44:0/3604053037 learned_addr learned my addr 172.21.15.44:0/3604053037 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:03.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e27fc640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9e415e260 0x7fc9e40cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e2ffd640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9e40cec60 0x7fc9e4157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e37fe640 1 -- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9e415e260 msgr2=0x7fc9e40cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e37fe640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9e415e260 0x7fc9e40cc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e37fe640 1 -- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9e40cec60 msgr2=0x7fc9e4157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e37fe640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9e40cec60 0x7fc9e4157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e37fe640 1 -- 172.21.15.44:0/3604053037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e41604b0 con 0x7fc9e415e640 2024-04-02T13:21:03.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9e37fe640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e415e640 0x7fc9e40cc570 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4007e40 tx=0x7fc9d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:03.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.222+0000 7fc9dbfff640 1 -- 172.21.15.44:0/3604053037 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d4014620 con 0x7fc9e415e640 2024-04-02T13:21:03.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.223+0000 7fc9dbfff640 1 -- 172.21.15.44:0/3604053037 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9d4019070 con 0x7fc9e415e640 2024-04-02T13:21:03.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.223+0000 7fc9e9808640 1 -- 172.21.15.44:0/3604053037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e40b6e30 con 0x7fc9e415e640 2024-04-02T13:21:03.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.223+0000 7fc9dbfff640 1 -- 172.21.15.44:0/3604053037 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d4014a20 con 0x7fc9e415e640 2024-04-02T13:21:03.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.223+0000 7fc9e9808640 1 -- 172.21.15.44:0/3604053037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e40b73f0 con 0x7fc9e415e640 2024-04-02T13:21:03.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.223+0000 7fc9e9808640 1 -- 172.21.15.44:0/3604053037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9e408d820 con 0x7fc9e415e640 2024-04-02T13:21:03.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.224+0000 7fc9dbfff640 1 -- 172.21.15.44:0/3604053037 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc9d4014bc0 con 0x7fc9e415e640 2024-04-02T13:21:03.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.224+0000 7fc9dbfff640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9c406c8f0 0x7fc9c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.224+0000 7fc9dbfff640 1 -- 172.21.15.44:0/3604053037 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc9d408c680 con 0x7fc9e415e640 2024-04-02T13:21:03.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.224+0000 7fc9e2ffd640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9c406c8f0 0x7fc9c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.225+0000 7fc9e2ffd640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9c406c8f0 0x7fc9c406edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc001780 tx=0x7fc9dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:03.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.229+0000 7fc9dbfff640 1 -- 172.21.15.44:0/3604053037 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc9e408d820 con 0x7fc9e415e640 2024-04-02T13:21:03.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.411+0000 7fc9e9808640 1 -- 172.21.15.44:0/3604053037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc9e40684f0 con 0x7fc9e415e640 2024-04-02T13:21:03.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.412+0000 7fc9dbfff640 1 -- 172.21.15.44:0/3604053037 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7fc9d405a120 con 0x7fc9e415e640 2024-04-02T13:21:03.413 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:03.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.413+0000 7fc9e9808640 1 -- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9c406c8f0 msgr2=0x7fc9c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.413+0000 7fc9e9808640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9c406c8f0 0x7fc9c406edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc001780 tx=0x7fc9dc009040 comp rx=0 tx=0).stop 2024-04-02T13:21:03.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.413+0000 7fc9e9808640 1 -- 172.21.15.44:0/3604053037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e415e640 msgr2=0x7fc9e40cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.414+0000 7fc9e9808640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e415e640 0x7fc9e40cc570 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4007e40 tx=0x7fc9d400e4b0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.414+0000 7fc9e9808640 1 -- 172.21.15.44:0/3604053037 shutdown_connections 2024-04-02T13:21:03.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.414+0000 7fc9e9808640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9c406c8f0 0x7fc9c406edb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.414+0000 7fc9e9808640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e415e640 0x7fc9e40cc570 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.414+0000 7fc9e9808640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9e415e260 0x7fc9e40cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.414+0000 7fc9e9808640 1 --2- 172.21.15.44:0/3604053037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9e40cec60 0x7fc9e4157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.414+0000 7fc9e9808640 1 -- 172.21.15.44:0/3604053037 >> 172.21.15.44:0/3604053037 conn(0x7fc9e409d3c0 msgr2=0x7fc9e40d0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:03.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.414+0000 7fc9e9808640 1 -- 172.21.15.44:0/3604053037 shutdown_connections 2024-04-02T13:21:03.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.414+0000 7fc9e9808640 1 -- 172.21.15.44:0/3604053037 wait complete. 2024-04-02T13:21:03.426 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:03.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.529+0000 7f370b847640 1 Processor -- start 2024-04-02T13:21:03.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.529+0000 7f370b847640 1 -- start start 2024-04-02T13:21:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.529+0000 7f370b847640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f37040b1440 0x7f37040b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.529+0000 7f370b847640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37040d03d0 0x7f37040d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f370b847640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37040cca50 con 0x7f370415c220 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f370b847640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f37040ccd60 con 0x7f37040b1440 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f370b847640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f37040cef40 con 0x7f37040d03d0 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f3709de4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37040d03d0 0x7f37040d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f3708de2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f37040b1440 0x7f37040b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f3709de4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37040d03d0 0x7f37040d0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:42090/0 (socket says 172.21.15.44:42090) 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f37095e3640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f370415c220 0x7f37040ce780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:42860/0 (socket says 172.21.15.44:42860) 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f3708de2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f37040b1440 0x7f37040b1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:33802/0 (socket says 172.21.15.44:33802) 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f3709de4640 1 -- 172.21.15.44:0/2532012277 learned_addr learned my addr 172.21.15.44:0/2532012277 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f3708de2640 1 -- 172.21.15.44:0/2532012277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37040d03d0 msgr2=0x7f37040d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f3708de2640 1 --2- 172.21.15.44:0/2532012277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37040d03d0 0x7f37040d0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f3708de2640 1 -- 172.21.15.44:0/2532012277 >> v1:172.21.15.69:6789/0 conn(0x7f370415c220 legacy=0x7f37040ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f3708de2640 1 -- 172.21.15.44:0/2532012277 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37040cf220 con 0x7f37040b1440 2024-04-02T13:21:03.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f36fbfff640 1 -- 172.21.15.44:0/2532012277 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 296937444 0 0) 0x7f3700001110 con 0x7f370415c220 2024-04-02T13:21:03.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.530+0000 7f3709de4640 1 --2- 172.21.15.44:0/2532012277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37040d03d0 0x7f37040d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:03.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.531+0000 7f3708de2640 1 --2- 172.21.15.44:0/2532012277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37040b1440 0x7f37040b1db0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f36f40097a0 tx=0x7f36f402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62ea1b0e6385a9c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:03.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.531+0000 7f36fbfff640 1 -- 172.21.15.44:0/2532012277 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f4035020 con 0x7f37040b1440 2024-04-02T13:21:03.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.531+0000 7f36fbfff640 1 -- 172.21.15.44:0/2532012277 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f4031830 con 0x7f37040b1440 2024-04-02T13:21:03.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.531+0000 7f36fbfff640 1 -- 172.21.15.44:0/2532012277 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f4031c00 con 0x7f37040b1440 2024-04-02T13:21:03.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.532+0000 7f370b847640 1 -- 172.21.15.44:0/2532012277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37040b1440 msgr2=0x7f37040b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.532+0000 7f370b847640 1 --2- 172.21.15.44:0/2532012277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37040b1440 0x7f37040b1db0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f36f40097a0 tx=0x7f36f402b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.532+0000 7f370b847640 1 -- 172.21.15.44:0/2532012277 shutdown_connections 2024-04-02T13:21:03.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.532+0000 7f370b847640 1 --2- 172.21.15.44:0/2532012277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37040b1440 0x7f37040b1db0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.532+0000 7f370b847640 1 --2- 172.21.15.44:0/2532012277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37040d03d0 0x7f37040d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.532+0000 7f370b847640 1 -- 172.21.15.44:0/2532012277 >> 172.21.15.44:0/2532012277 conn(0x7f370409aa20 msgr2=0x7f370409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:03.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.532+0000 7f370b847640 1 -- 172.21.15.44:0/2532012277 shutdown_connections 2024-04-02T13:21:03.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.532+0000 7f370b847640 1 -- 172.21.15.44:0/2532012277 wait complete. 2024-04-02T13:21:03.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.533+0000 7f370b847640 1 Processor -- start 2024-04-02T13:21:03.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.533+0000 7f370b847640 1 -- start start 2024-04-02T13:21:03.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f370b847640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37040b1440 0x7f37041561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f370b847640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37040d03d0 0x7f370415f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f370b847640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370415c220 0x7f37041580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f370b847640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f37040cd1f0 con 0x7f37040d03d0 2024-04-02T13:21:03.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f370b847640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f370415c600 con 0x7f37040b1440 2024-04-02T13:21:03.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f370b847640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f37040cd370 con 0x7f370415c220 2024-04-02T13:21:03.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f3708de2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37040d03d0 0x7f370415f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f3708de2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37040d03d0 0x7f370415f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45824/0 (socket says 172.21.15.44:45824) 2024-04-02T13:21:03.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f3708de2640 1 -- 172.21.15.44:0/4122046803 learned_addr learned my addr 172.21.15.44:0/4122046803 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:03.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f3709de4640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370415c220 0x7f37041580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f37095e3640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37040b1440 0x7f37041561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f3708de2640 1 -- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370415c220 msgr2=0x7f37041580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f3708de2640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370415c220 0x7f37041580f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f3708de2640 1 -- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37040b1440 msgr2=0x7f37041561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f3708de2640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37040b1440 0x7f37041561a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f3708de2640 1 -- 172.21.15.44:0/4122046803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370415d560 con 0x7f37040d03d0 2024-04-02T13:21:03.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f3708de2640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37040d03d0 0x7f370415f060 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f36f4000c00 tx=0x7f36f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:03.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.534+0000 7f36f9ffb640 1 -- 172.21.15.44:0/4122046803 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f4035070 con 0x7f37040d03d0 2024-04-02T13:21:03.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.535+0000 7f36f9ffb640 1 -- 172.21.15.44:0/4122046803 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f4004230 con 0x7f37040d03d0 2024-04-02T13:21:03.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.535+0000 7f370b847640 1 -- 172.21.15.44:0/4122046803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f370415d840 con 0x7f37040d03d0 2024-04-02T13:21:03.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.535+0000 7f36f9ffb640 1 -- 172.21.15.44:0/4122046803 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f403a630 con 0x7f37040d03d0 2024-04-02T13:21:03.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.535+0000 7f370b847640 1 -- 172.21.15.44:0/4122046803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3704157bd0 con 0x7f37040d03d0 2024-04-02T13:21:03.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.536+0000 7f370b847640 1 -- 172.21.15.44:0/4122046803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37040684f0 con 0x7f37040d03d0 2024-04-02T13:21:03.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.536+0000 7f36f9ffb640 1 -- 172.21.15.44:0/4122046803 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f36f4038020 con 0x7f37040d03d0 2024-04-02T13:21:03.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.537+0000 7f36f9ffb640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36e806c8f0 0x7f36e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.537+0000 7f36f9ffb640 1 -- 172.21.15.44:0/4122046803 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f36f40adbc0 con 0x7f37040d03d0 2024-04-02T13:21:03.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.537+0000 7f37095e3640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36e806c8f0 0x7f36e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.537+0000 7f37095e3640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36e806c8f0 0x7f36e806edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3700000f60 tx=0x7f3700007c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:03.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.541+0000 7f36f9ffb640 1 -- 172.21.15.44:0/4122046803 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f36f4040070 con 0x7f37040d03d0 2024-04-02T13:21:03.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.724+0000 7f370b847640 1 -- 172.21.15.44:0/4122046803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f37040b1060 con 0x7f37040d03d0 2024-04-02T13:21:03.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.724+0000 7f36f9ffb640 1 -- 172.21.15.44:0/4122046803 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f36f4042e40 con 0x7f37040d03d0 2024-04-02T13:21:03.726 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:03.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.726+0000 7f370b847640 1 -- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36e806c8f0 msgr2=0x7f36e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.726+0000 7f370b847640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36e806c8f0 0x7f36e806edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3700000f60 tx=0x7f3700007c00 comp rx=0 tx=0).stop 2024-04-02T13:21:03.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.726+0000 7f370b847640 1 -- 172.21.15.44:0/4122046803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37040d03d0 msgr2=0x7f370415f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.726+0000 7f370b847640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37040d03d0 0x7f370415f060 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f36f4000c00 tx=0x7f36f40046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.726+0000 7f370b847640 1 -- 172.21.15.44:0/4122046803 shutdown_connections 2024-04-02T13:21:03.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.726+0000 7f370b847640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36e806c8f0 0x7f36e806edb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.726+0000 7f370b847640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370415c220 0x7f37041580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.726+0000 7f370b847640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37040d03d0 0x7f370415f060 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.726+0000 7f370b847640 1 --2- 172.21.15.44:0/4122046803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37040b1440 0x7f37041561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.726+0000 7f370b847640 1 -- 172.21.15.44:0/4122046803 >> 172.21.15.44:0/4122046803 conn(0x7f370409aa20 msgr2=0x7f37040cd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:03.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.726+0000 7f370b847640 1 -- 172.21.15.44:0/4122046803 shutdown_connections 2024-04-02T13:21:03.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.726+0000 7f370b847640 1 -- 172.21.15.44:0/4122046803 wait complete. 2024-04-02T13:21:03.738 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:03.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.841+0000 7f88ff911640 1 Processor -- start 2024-04-02T13:21:03.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.842+0000 7f88ff911640 1 -- start start 2024-04-02T13:21:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.842+0000 7f88ff911640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f88f815e640 0x7f88f80d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.842+0000 7f88ff911640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88f80cec60 0x7f88f80d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.842+0000 7f88ff911640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f80cf120 con 0x7f88f815e260 2024-04-02T13:21:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.842+0000 7f88ff911640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f88f80b5a50 con 0x7f88f815e640 2024-04-02T13:21:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.842+0000 7f88ff911640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f88f80d01c0 con 0x7f88f80cec60 2024-04-02T13:21:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.842+0000 7f88fceac640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f88f815e640 0x7f88f80d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.843+0000 7f88fceac640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f88f815e640 0x7f88f80d0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45838/0 (socket says 172.21.15.44:45838) 2024-04-02T13:21:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.843+0000 7f88fceac640 1 -- 172.21.15.44:0/1244540883 learned_addr learned my addr 172.21.15.44:0/1244540883 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.843+0000 7f88fdeae640 1 --2- 172.21.15.44:0/1244540883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88f80cec60 0x7f88f80d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.843+0000 7f88fceac640 1 -- 172.21.15.44:0/1244540883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88f80cec60 msgr2=0x7f88f80d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.843+0000 7f88fceac640 1 --2- 172.21.15.44:0/1244540883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88f80cec60 0x7f88f80d1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.843+0000 7f88fceac640 1 -- 172.21.15.44:0/1244540883 >> v1:172.21.15.69:6789/0 conn(0x7f88f815e260 legacy=0x7f88f80d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.843+0000 7f88fceac640 1 -- 172.21.15.44:0/1244540883 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f80cf400 con 0x7f88f815e640 2024-04-02T13:21:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.844+0000 7f88fceac640 1 --2- 172.21.15.44:0/1244540883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88f815e640 0x7f88f80d0ad0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f88e8009870 tx=0x7f88e802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ee41b8a481205f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.844+0000 7f88effff640 1 -- 172.21.15.44:0/1244540883 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8035020 con 0x7f88f815e640 2024-04-02T13:21:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.844+0000 7f88effff640 1 -- 172.21.15.44:0/1244540883 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e8031640 con 0x7f88f815e640 2024-04-02T13:21:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.844+0000 7f88effff640 1 -- 172.21.15.44:0/1244540883 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e802d040 con 0x7f88f815e640 2024-04-02T13:21:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.844+0000 7f88ff911640 1 -- 172.21.15.44:0/1244540883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88f815e640 msgr2=0x7f88f80d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.844+0000 7f88ff911640 1 --2- 172.21.15.44:0/1244540883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88f815e640 0x7f88f80d0ad0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f88e8009870 tx=0x7f88e802b8f0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.844+0000 7f88ff911640 1 -- 172.21.15.44:0/1244540883 shutdown_connections 2024-04-02T13:21:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.844+0000 7f88ff911640 1 --2- 172.21.15.44:0/1244540883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88f815e640 0x7f88f80d0ad0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.844+0000 7f88ff911640 1 --2- 172.21.15.44:0/1244540883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88f80cec60 0x7f88f80d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.844+0000 7f88ff911640 1 -- 172.21.15.44:0/1244540883 >> 172.21.15.44:0/1244540883 conn(0x7f88f809d3c0 msgr2=0x7f88f80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.844+0000 7f88ff911640 1 -- 172.21.15.44:0/1244540883 shutdown_connections 2024-04-02T13:21:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.844+0000 7f88ff911640 1 -- 172.21.15.44:0/1244540883 wait complete. 2024-04-02T13:21:03.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.845+0000 7f88ff911640 1 Processor -- start 2024-04-02T13:21:03.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.846+0000 7f88ff911640 1 -- start start 2024-04-02T13:21:03.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.846+0000 7f88ff911640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88f80cec60 0x7f88f80b89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.846+0000 7f88ff911640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88f815e260 0x7f88f81f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.846+0000 7f88ff911640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88f815e640 0x7f88f81f8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.846+0000 7f88ff911640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f88f80cfca0 con 0x7f88f815e260 2024-04-02T13:21:03.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.846+0000 7f88ff911640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f88f80cf9c0 con 0x7f88f80cec60 2024-04-02T13:21:03.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.846+0000 7f88ff911640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f88f80d3070 con 0x7f88f815e640 2024-04-02T13:21:03.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.846+0000 7f88fceac640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88f815e260 0x7f88f81f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.846+0000 7f88fceac640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88f815e260 0x7f88f81f4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45850/0 (socket says 172.21.15.44:45850) 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.846+0000 7f88fceac640 1 -- 172.21.15.44:0/335947255 learned_addr learned my addr 172.21.15.44:0/335947255 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88fdeae640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88f815e640 0x7f88f81f8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88fd6ad640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88f80cec60 0x7f88f80b89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88fceac640 1 -- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88f815e640 msgr2=0x7f88f81f8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88fceac640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88f815e640 0x7f88f81f8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88fceac640 1 -- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88f80cec60 msgr2=0x7f88f80b89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88fceac640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88f80cec60 0x7f88f80b89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88fceac640 1 -- 172.21.15.44:0/335947255 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f80b6e80 con 0x7f88f815e260 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88fdeae640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88f815e640 0x7f88f81f8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88fd6ad640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88f80cec60 0x7f88f80b89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88fceac640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88f815e260 0x7f88f81f4b70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f88e802b930 tx=0x7f88e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88edffb640 1 -- 172.21.15.44:0/335947255 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8040070 con 0x7f88f815e260 2024-04-02T13:21:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88edffb640 1 -- 172.21.15.44:0/335947255 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e8039bf0 con 0x7f88f815e260 2024-04-02T13:21:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88ff911640 1 -- 172.21.15.44:0/335947255 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f81f8b80 con 0x7f88f815e260 2024-04-02T13:21:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88edffb640 1 -- 172.21.15.44:0/335947255 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8035040 con 0x7f88f815e260 2024-04-02T13:21:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.847+0000 7f88ff911640 1 -- 172.21.15.44:0/335947255 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f80b8ef0 con 0x7f88f815e260 2024-04-02T13:21:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.848+0000 7f88ff911640 1 -- 172.21.15.44:0/335947255 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f80684f0 con 0x7f88f815e260 2024-04-02T13:21:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.849+0000 7f88edffb640 1 -- 172.21.15.44:0/335947255 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f88e804b060 con 0x7f88f815e260 2024-04-02T13:21:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.849+0000 7f88edffb640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f88dc06c920 0x7f88dc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:03.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.849+0000 7f88edffb640 1 -- 172.21.15.44:0/335947255 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f88e80ad950 con 0x7f88f815e260 2024-04-02T13:21:03.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.849+0000 7f88fd6ad640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f88dc06c920 0x7f88dc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:03.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.850+0000 7f88fd6ad640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f88dc06c920 0x7f88dc06ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f88f4003eb0 tx=0x7f88f4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:03.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:03.854+0000 7f88edffb640 1 -- 172.21.15.44:0/335947255 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f88e807ba30 con 0x7f88f815e260 2024-04-02T13:21:04.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.040+0000 7f88ff911640 1 -- 172.21.15.44:0/335947255 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f88f80b6ca0 con 0x7f88f815e260 2024-04-02T13:21:04.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.040+0000 7f88edffb640 1 -- 172.21.15.44:0/335947255 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f88f80b6ca0 con 0x7f88f815e260 2024-04-02T13:21:04.042 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:04.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.042+0000 7f88ff911640 1 -- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f88dc06c920 msgr2=0x7f88dc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.042+0000 7f88ff911640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f88dc06c920 0x7f88dc06ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f88f4003eb0 tx=0x7f88f4009040 comp rx=0 tx=0).stop 2024-04-02T13:21:04.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.042+0000 7f88ff911640 1 -- 172.21.15.44:0/335947255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88f815e260 msgr2=0x7f88f81f4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.042+0000 7f88ff911640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88f815e260 0x7f88f81f4b70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f88e802b930 tx=0x7f88e80046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.042+0000 7f88ff911640 1 -- 172.21.15.44:0/335947255 shutdown_connections 2024-04-02T13:21:04.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.042+0000 7f88ff911640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f88dc06c920 0x7f88dc06ede0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.042+0000 7f88ff911640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88f815e640 0x7f88f81f8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.042+0000 7f88ff911640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88f815e260 0x7f88f81f4b70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.042+0000 7f88ff911640 1 --2- 172.21.15.44:0/335947255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88f80cec60 0x7f88f80b89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.042+0000 7f88ff911640 1 -- 172.21.15.44:0/335947255 >> 172.21.15.44:0/335947255 conn(0x7f88f809d3c0 msgr2=0x7f88f80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:04.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.042+0000 7f88ff911640 1 -- 172.21.15.44:0/335947255 shutdown_connections 2024-04-02T13:21:04.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.042+0000 7f88ff911640 1 -- 172.21.15.44:0/335947255 wait complete. 2024-04-02T13:21:04.056 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.164+0000 7f78faaa5640 1 Processor -- start 2024-04-02T13:21:04.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.164+0000 7f78faaa5640 1 -- start start 2024-04-02T13:21:04.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.165+0000 7f78faaa5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78f415e1d0 0x7f78f40d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.165+0000 7f78faaa5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f78f40d2dd0 0x7f78f40b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.165+0000 7f78faaa5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f40d1640 con 0x7f78f40cec60 2024-04-02T13:21:04.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.165+0000 7f78faaa5640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f78f40cfe00 con 0x7f78f40d2dd0 2024-04-02T13:21:04.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.165+0000 7f78faaa5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f78f40cf1f0 con 0x7f78f415e1d0 2024-04-02T13:21:04.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.165+0000 7f78f9042640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f78f40d2dd0 0x7f78f40b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.165+0000 7f78f3fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f78f40cec60 0x7f78f40d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:55306/0 (socket says 172.21.15.44:55306) 2024-04-02T13:21:04.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.165+0000 7f78f3fff640 1 -- 172.21.15.44:0/1101001877 learned_addr learned my addr 172.21.15.44:0/1101001877 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:04.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.166+0000 7f78f8841640 1 --2- 172.21.15.44:0/1101001877 >> v2:172.21.15.69:3300/0 conn(0x7f78f415e1d0 0x7f78f40d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.166+0000 7f78f9042640 1 -- 172.21.15.44:0/1101001877 >> v2:172.21.15.69:3300/0 conn(0x7f78f415e1d0 msgr2=0x7f78f40d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:04.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.166+0000 7f78f9042640 1 --2- 172.21.15.44:0/1101001877 >> v2:172.21.15.69:3300/0 conn(0x7f78f415e1d0 0x7f78f40d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.166+0000 7f78f9042640 1 -- 172.21.15.44:0/1101001877 >> v1:172.21.15.44:6789/0 conn(0x7f78f40cec60 legacy=0x7f78f40d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.166+0000 7f78f37fe640 1 -- 172.21.15.44:0/1101001877 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2289055880 0 0) 0x7f78f40d1640 con 0x7f78f40cec60 2024-04-02T13:21:04.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.166+0000 7f78f9042640 1 -- 172.21.15.44:0/1101001877 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f40cf4d0 con 0x7f78f40d2dd0 2024-04-02T13:21:04.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.166+0000 7f78f9042640 1 --2- 172.21.15.44:0/1101001877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78f40d2dd0 0x7f78f40b4ce0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f78f408c9c0 tx=0x7f78dc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3f93a546f1d2cd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:04.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.167+0000 7f78f37fe640 1 -- 172.21.15.44:0/1101001877 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0315a0 con 0x7f78f40d2dd0 2024-04-02T13:21:04.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.167+0000 7f78f37fe640 1 -- 172.21.15.44:0/1101001877 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78dc037070 con 0x7f78f40d2dd0 2024-04-02T13:21:04.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.167+0000 7f78f37fe640 1 -- 172.21.15.44:0/1101001877 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc031960 con 0x7f78f40d2dd0 2024-04-02T13:21:04.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.167+0000 7f78faaa5640 1 -- 172.21.15.44:0/1101001877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78f40d2dd0 msgr2=0x7f78f40b4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.167+0000 7f78faaa5640 1 --2- 172.21.15.44:0/1101001877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78f40d2dd0 0x7f78f40b4ce0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f78f408c9c0 tx=0x7f78dc02b840 comp rx=0 tx=0).stop 2024-04-02T13:21:04.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.167+0000 7f78faaa5640 1 -- 172.21.15.44:0/1101001877 shutdown_connections 2024-04-02T13:21:04.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.167+0000 7f78faaa5640 1 --2- 172.21.15.44:0/1101001877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78f40d2dd0 0x7f78f40b4ce0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.167+0000 7f78faaa5640 1 --2- 172.21.15.44:0/1101001877 >> v2:172.21.15.69:3300/0 conn(0x7f78f415e1d0 0x7f78f40d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.167+0000 7f78faaa5640 1 -- 172.21.15.44:0/1101001877 >> 172.21.15.44:0/1101001877 conn(0x7f78f409d3b0 msgr2=0x7f78f40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.167+0000 7f78faaa5640 1 -- 172.21.15.44:0/1101001877 shutdown_connections 2024-04-02T13:21:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.167+0000 7f78faaa5640 1 -- 172.21.15.44:0/1101001877 wait complete. 2024-04-02T13:21:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.168+0000 7f78faaa5640 1 Processor -- start 2024-04-02T13:21:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.168+0000 7f78faaa5640 1 -- start start 2024-04-02T13:21:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.169+0000 7f78faaa5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78f40cec60 0x7f78f4150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.169+0000 7f78faaa5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f40d2dd0 0x7f78f40a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.169+0000 7f78faaa5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78f415e1d0 0x7f78f4150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.169+0000 7f78faaa5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f78f415e5b0 con 0x7f78f40cec60 2024-04-02T13:21:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.169+0000 7f78faaa5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f78f40cfa90 con 0x7f78f40d2dd0 2024-04-02T13:21:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.169+0000 7f78faaa5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f78f415e730 con 0x7f78f415e1d0 2024-04-02T13:21:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.169+0000 7f78f8841640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78f40cec60 0x7f78f4150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.169+0000 7f78f8841640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78f40cec60 0x7f78f4150410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45860/0 (socket says 172.21.15.44:45860) 2024-04-02T13:21:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.169+0000 7f78f8841640 1 -- 172.21.15.44:0/707313395 learned_addr learned my addr 172.21.15.44:0/707313395 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.169+0000 7f78f9042640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78f415e1d0 0x7f78f4150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.169+0000 7f78f3fff640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f40d2dd0 0x7f78f40a12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.170+0000 7f78f8841640 1 -- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78f415e1d0 msgr2=0x7f78f4150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.170+0000 7f78f8841640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78f415e1d0 0x7f78f4150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.170+0000 7f78f8841640 1 -- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f40d2dd0 msgr2=0x7f78f40a12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.170+0000 7f78f8841640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f40d2dd0 0x7f78f40a12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.170+0000 7f78f8841640 1 -- 172.21.15.44:0/707313395 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f40a0230 con 0x7f78f40cec60 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.170+0000 7f78f3fff640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f40d2dd0 0x7f78f40a12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.170+0000 7f78f8841640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78f40cec60 0x7f78f4150410 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f78ec007ef0 tx=0x7f78ec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.170+0000 7f78f17fa640 1 -- 172.21.15.44:0/707313395 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec014620 con 0x7f78f40cec60 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.170+0000 7f78f17fa640 1 -- 172.21.15.44:0/707313395 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78ec019070 con 0x7f78f40cec60 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.170+0000 7f78faaa5640 1 -- 172.21.15.44:0/707313395 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f40a0510 con 0x7f78f40cec60 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.170+0000 7f78f17fa640 1 -- 172.21.15.44:0/707313395 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec014a00 con 0x7f78f40cec60 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.170+0000 7f78faaa5640 1 -- 172.21.15.44:0/707313395 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f40a0ad0 con 0x7f78f40cec60 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.171+0000 7f78faaa5640 1 -- 172.21.15.44:0/707313395 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f40684f0 con 0x7f78f40cec60 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.171+0000 7f78f17fa640 1 -- 172.21.15.44:0/707313395 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f78ec014ba0 con 0x7f78f40cec60 2024-04-02T13:21:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.172+0000 7f78f17fa640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f78d406c8f0 0x7f78d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.172+0000 7f78f17fa640 1 -- 172.21.15.44:0/707313395 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f78ec08c740 con 0x7f78f40cec60 2024-04-02T13:21:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.172+0000 7f78f3fff640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f78d406c8f0 0x7f78d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.172+0000 7f78f3fff640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f78d406c8f0 0x7f78d406edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f78f41541b0 tx=0x7f78e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.176+0000 7f78f17fa640 1 -- 172.21.15.44:0/707313395 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f78ec05a8c0 con 0x7f78f40cec60 2024-04-02T13:21:04.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.359+0000 7f78faaa5640 1 -- 172.21.15.44:0/707313395 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f78f40d47f0 con 0x7f78f40cec60 2024-04-02T13:21:04.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.360+0000 7f78f17fa640 1 -- 172.21.15.44:0/707313395 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f78ec05a1c0 con 0x7f78f40cec60 2024-04-02T13:21:04.361 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:04.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.362+0000 7f78faaa5640 1 -- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f78d406c8f0 msgr2=0x7f78d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.362+0000 7f78faaa5640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f78d406c8f0 0x7f78d406edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f78f41541b0 tx=0x7f78e4009040 comp rx=0 tx=0).stop 2024-04-02T13:21:04.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.362+0000 7f78faaa5640 1 -- 172.21.15.44:0/707313395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78f40cec60 msgr2=0x7f78f4150410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.362+0000 7f78faaa5640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78f40cec60 0x7f78f4150410 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f78ec007ef0 tx=0x7f78ec00e4b0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.362+0000 7f78faaa5640 1 -- 172.21.15.44:0/707313395 shutdown_connections 2024-04-02T13:21:04.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.362+0000 7f78faaa5640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f78d406c8f0 0x7f78d406edb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.362+0000 7f78faaa5640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78f415e1d0 0x7f78f4150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.362+0000 7f78faaa5640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f40d2dd0 0x7f78f40a12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.362+0000 7f78faaa5640 1 --2- 172.21.15.44:0/707313395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78f40cec60 0x7f78f4150410 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.362+0000 7f78faaa5640 1 -- 172.21.15.44:0/707313395 >> 172.21.15.44:0/707313395 conn(0x7f78f409d3b0 msgr2=0x7f78f40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:04.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.362+0000 7f78faaa5640 1 -- 172.21.15.44:0/707313395 shutdown_connections 2024-04-02T13:21:04.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.362+0000 7f78faaa5640 1 -- 172.21.15.44:0/707313395 wait complete. 2024-04-02T13:21:04.375 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.476+0000 7fb0da293640 1 Processor -- start 2024-04-02T13:21:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.476+0000 7fb0da293640 1 -- start start 2024-04-02T13:21:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.477+0000 7fb0da293640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d40d0a30 0x7fb0d406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.477+0000 7fb0da293640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb0d40d0e10 0x7fb0d40cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.477+0000 7fb0da293640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d406c1d0 con 0x7fb0d40cfe30 2024-04-02T13:21:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.477+0000 7fb0da293640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb0d406cb00 con 0x7fb0d40d0e10 2024-04-02T13:21:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.477+0000 7fb0da293640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d406c4b0 con 0x7fb0d40d0a30 2024-04-02T13:21:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.477+0000 7fb0d8830640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb0d40cfe30 0x7fb0d40d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:55316/0 (socket says 172.21.15.44:55316) 2024-04-02T13:21:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.477+0000 7fb0d8830640 1 -- 172.21.15.44:0/757241569 learned_addr learned my addr 172.21.15.44:0/757241569 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.477+0000 7fb0d37fe640 1 --2- 172.21.15.44:0/757241569 >> v2:172.21.15.69:3300/0 conn(0x7fb0d40d0e10 0x7fb0d40cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.477+0000 7fb0d3fff640 1 --2- 172.21.15.44:0/757241569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d40d0a30 0x7fb0d406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.478+0000 7fb0d2ffd640 1 -- 172.21.15.44:0/757241569 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 554142769 0 0) 0x7fb0d406c1d0 con 0x7fb0d40cfe30 2024-04-02T13:21:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.478+0000 7fb0d2ffd640 1 -- 172.21.15.44:0/757241569 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0bc003520 con 0x7fb0d40cfe30 2024-04-02T13:21:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.478+0000 7fb0d3fff640 1 -- 172.21.15.44:0/757241569 >> v2:172.21.15.69:3300/0 conn(0x7fb0d40d0e10 msgr2=0x7fb0d40cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.478+0000 7fb0d3fff640 1 --2- 172.21.15.44:0/757241569 >> v2:172.21.15.69:3300/0 conn(0x7fb0d40d0e10 0x7fb0d40cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.478+0000 7fb0d3fff640 1 -- 172.21.15.44:0/757241569 >> v1:172.21.15.44:6789/0 conn(0x7fb0d40cfe30 legacy=0x7fb0d40d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.478+0000 7fb0d3fff640 1 -- 172.21.15.44:0/757241569 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d406c790 con 0x7fb0d40d0a30 2024-04-02T13:21:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.478+0000 7fb0d37fe640 1 --2- 172.21.15.44:0/757241569 >> v2:172.21.15.69:3300/0 conn(0x7fb0d40d0e10 0x7fb0d40cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.479+0000 7fb0d3fff640 1 --2- 172.21.15.44:0/757241569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d40d0a30 0x7fb0d406cf90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40097a0 tx=0x7fb0c402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7baf3cc56cc268b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.479+0000 7fb0d2ffd640 1 -- 172.21.15.44:0/757241569 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c40315a0 con 0x7fb0d40d0a30 2024-04-02T13:21:04.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.479+0000 7fb0d2ffd640 1 -- 172.21.15.44:0/757241569 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0c4037070 con 0x7fb0d40d0a30 2024-04-02T13:21:04.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.479+0000 7fb0d2ffd640 1 -- 172.21.15.44:0/757241569 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4031940 con 0x7fb0d40d0a30 2024-04-02T13:21:04.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.479+0000 7fb0da293640 1 -- 172.21.15.44:0/757241569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d40d0a30 msgr2=0x7fb0d406cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.479+0000 7fb0da293640 1 --2- 172.21.15.44:0/757241569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d40d0a30 0x7fb0d406cf90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40097a0 tx=0x7fb0c402b840 comp rx=0 tx=0).stop 2024-04-02T13:21:04.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.479+0000 7fb0da293640 1 -- 172.21.15.44:0/757241569 shutdown_connections 2024-04-02T13:21:04.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.479+0000 7fb0da293640 1 --2- 172.21.15.44:0/757241569 >> v2:172.21.15.69:3300/0 conn(0x7fb0d40d0e10 0x7fb0d40cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.479+0000 7fb0da293640 1 --2- 172.21.15.44:0/757241569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d40d0a30 0x7fb0d406cf90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.479+0000 7fb0da293640 1 -- 172.21.15.44:0/757241569 >> 172.21.15.44:0/757241569 conn(0x7fb0d409d470 msgr2=0x7fb0d40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:04.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.479+0000 7fb0da293640 1 -- 172.21.15.44:0/757241569 shutdown_connections 2024-04-02T13:21:04.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.479+0000 7fb0da293640 1 -- 172.21.15.44:0/757241569 wait complete. 2024-04-02T13:21:04.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.480+0000 7fb0da293640 1 Processor -- start 2024-04-02T13:21:04.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.481+0000 7fb0da293640 1 -- start start 2024-04-02T13:21:04.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.481+0000 7fb0da293640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d40cfe30 0x7fb0d41e9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.481+0000 7fb0da293640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d40d0a30 0x7fb0d4159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.481+0000 7fb0da293640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d40d0e10 0x7fb0d40b6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.481+0000 7fb0da293640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d415e870 con 0x7fb0d40d0e10 2024-04-02T13:21:04.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.481+0000 7fb0da293640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d415e6f0 con 0x7fb0d40d0a30 2024-04-02T13:21:04.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.481+0000 7fb0da293640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d40d4da0 con 0x7fb0d40cfe30 2024-04-02T13:21:04.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.481+0000 7fb0d8830640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d40d0e10 0x7fb0d40b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0d8830640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d40d0e10 0x7fb0d40b6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45864/0 (socket says 172.21.15.44:45864) 2024-04-02T13:21:04.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0d8830640 1 -- 172.21.15.44:0/1934556995 learned_addr learned my addr 172.21.15.44:0/1934556995 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:04.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0d3fff640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d40cfe30 0x7fb0d41e9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0d37fe640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d40d0a30 0x7fb0d4159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0d8830640 1 -- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d40cfe30 msgr2=0x7fb0d41e9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0d8830640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d40cfe30 0x7fb0d41e9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0d8830640 1 -- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d40d0a30 msgr2=0x7fb0d4159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0d8830640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d40d0a30 0x7fb0d4159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0d8830640 1 -- 172.21.15.44:0/1934556995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d40b6aa0 con 0x7fb0d40d0e10 2024-04-02T13:21:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0d8830640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d40d0e10 0x7fb0d40b6560 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00ca20 tx=0x7fb0cc00cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0d0ff9640 1 -- 172.21.15.44:0/1934556995 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc017070 con 0x7fb0d40d0e10 2024-04-02T13:21:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0d0ff9640 1 -- 172.21.15.44:0/1934556995 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0cc00abd0 con 0x7fb0d40d0e10 2024-04-02T13:21:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0da293640 1 -- 172.21.15.44:0/1934556995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d40b7a30 con 0x7fb0d40d0e10 2024-04-02T13:21:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0d0ff9640 1 -- 172.21.15.44:0/1934556995 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc012400 con 0x7fb0d40d0e10 2024-04-02T13:21:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.482+0000 7fb0da293640 1 -- 172.21.15.44:0/1934556995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d40b7ff0 con 0x7fb0d40d0e10 2024-04-02T13:21:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.483+0000 7fb0da293640 1 -- 172.21.15.44:0/1934556995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0d40684f0 con 0x7fb0d40d0e10 2024-04-02T13:21:04.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.484+0000 7fb0d0ff9640 1 -- 172.21.15.44:0/1934556995 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb0cc01a070 con 0x7fb0d40d0e10 2024-04-02T13:21:04.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.484+0000 7fb0d0ff9640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb0c006c8f0 0x7fb0c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.484+0000 7fb0d0ff9640 1 -- 172.21.15.44:0/1934556995 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb0cc08bb20 con 0x7fb0d40d0e10 2024-04-02T13:21:04.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.484+0000 7fb0d3fff640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb0c006c8f0 0x7fb0c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.485+0000 7fb0d3fff640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb0c006c8f0 0x7fb0c006edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4036000 tx=0x7fb0c403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:04.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.488+0000 7fb0d0ff9640 1 -- 172.21.15.44:0/1934556995 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb0d40b6aa0 con 0x7fb0d40d0e10 2024-04-02T13:21:04.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.679+0000 7fb0da293640 1 -- 172.21.15.44:0/1934556995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb0d40d17c0 con 0x7fb0d40d0e10 2024-04-02T13:21:04.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.680+0000 7fb0d0ff9640 1 -- 172.21.15.44:0/1934556995 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7fb0cc0595c0 con 0x7fb0d40d0e10 2024-04-02T13:21:04.682 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:04.682 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.682+0000 7fb0da293640 1 -- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb0c006c8f0 msgr2=0x7fb0c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.682+0000 7fb0da293640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb0c006c8f0 0x7fb0c006edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4036000 tx=0x7fb0c403d040 comp rx=0 tx=0).stop 2024-04-02T13:21:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.682+0000 7fb0da293640 1 -- 172.21.15.44:0/1934556995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d40d0e10 msgr2=0x7fb0d40b6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.682+0000 7fb0da293640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d40d0e10 0x7fb0d40b6560 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00ca20 tx=0x7fb0cc00cee0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.682+0000 7fb0da293640 1 -- 172.21.15.44:0/1934556995 shutdown_connections 2024-04-02T13:21:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.682+0000 7fb0da293640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb0c006c8f0 0x7fb0c006edb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.682+0000 7fb0da293640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d40d0e10 0x7fb0d40b6560 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.682+0000 7fb0da293640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d40d0a30 0x7fb0d4159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.682+0000 7fb0da293640 1 --2- 172.21.15.44:0/1934556995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d40cfe30 0x7fb0d41e9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.682+0000 7fb0da293640 1 -- 172.21.15.44:0/1934556995 >> 172.21.15.44:0/1934556995 conn(0x7fb0d409d470 msgr2=0x7fb0d40b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.682+0000 7fb0da293640 1 -- 172.21.15.44:0/1934556995 shutdown_connections 2024-04-02T13:21:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.682+0000 7fb0da293640 1 -- 172.21.15.44:0/1934556995 wait complete. 2024-04-02T13:21:04.694 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:04.695 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:04.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.798+0000 7f31f9693640 1 Processor -- start 2024-04-02T13:21:04.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.798+0000 7f31f9693640 1 -- start start 2024-04-02T13:21:04.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.799+0000 7f31f9693640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f31f40d00d0 0x7f31f415ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.799+0000 7f31f9693640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31f4070040 con 0x7f31f40d2470 2024-04-02T13:21:04.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.799+0000 7f31f9693640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31f4070320 con 0x7f31f40d2090 2024-04-02T13:21:04.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.799+0000 7f31f9693640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f31f406c5e0 con 0x7f31f40d00d0 2024-04-02T13:21:04.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.799+0000 7f31f37fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f31f40d2470 0x7f31f406cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54642/0 (socket says 172.21.15.44:54642) 2024-04-02T13:21:04.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.799+0000 7f31f27fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f31f40d00d0 0x7f31f415ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.799+0000 7f31f37fe640 1 -- 172.21.15.44:0/1221642490 learned_addr learned my addr 172.21.15.44:0/1221642490 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:04.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.800+0000 7f31f27fc640 1 -- 172.21.15.44:0/1221642490 >> v1:172.21.15.69:6789/0 conn(0x7f31f40d2090 legacy=0x7f31f406f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.800+0000 7f31f27fc640 1 -- 172.21.15.44:0/1221642490 >> v1:172.21.15.44:6789/0 conn(0x7f31f40d2470 legacy=0x7f31f406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.800+0000 7f31f27fc640 1 -- 172.21.15.44:0/1221642490 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f4070600 con 0x7f31f40d00d0 2024-04-02T13:21:04.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.800+0000 7f31f27fc640 1 --2- 172.21.15.44:0/1221642490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31f40d00d0 0x7f31f415ddc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0097d0 tx=0x7f31dc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a7351894849d35 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:04.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.800+0000 7f31f1ffb640 1 -- 172.21.15.44:0/1221642490 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31dc034070 con 0x7f31f40d00d0 2024-04-02T13:21:04.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.801+0000 7f31f1ffb640 1 -- 172.21.15.44:0/1221642490 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31dc030610 con 0x7f31f40d00d0 2024-04-02T13:21:04.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.801+0000 7f31f1ffb640 1 -- 172.21.15.44:0/1221642490 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31dc0309e0 con 0x7f31f40d00d0 2024-04-02T13:21:04.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.801+0000 7f31f9693640 1 -- 172.21.15.44:0/1221642490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31f40d00d0 msgr2=0x7f31f415ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.801+0000 7f31f9693640 1 --2- 172.21.15.44:0/1221642490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31f40d00d0 0x7f31f415ddc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0097d0 tx=0x7f31dc02b960 comp rx=0 tx=0).stop 2024-04-02T13:21:04.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.801+0000 7f31f9693640 1 -- 172.21.15.44:0/1221642490 shutdown_connections 2024-04-02T13:21:04.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.801+0000 7f31f9693640 1 --2- 172.21.15.44:0/1221642490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31f40d00d0 0x7f31f415ddc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.801+0000 7f31f9693640 1 -- 172.21.15.44:0/1221642490 >> 172.21.15.44:0/1221642490 conn(0x7f31f409aa20 msgr2=0x7f31f409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:04.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.801+0000 7f31f9693640 1 -- 172.21.15.44:0/1221642490 shutdown_connections 2024-04-02T13:21:04.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.801+0000 7f31f9693640 1 -- 172.21.15.44:0/1221642490 wait complete. 2024-04-02T13:21:04.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.802+0000 7f31f9693640 1 Processor -- start 2024-04-02T13:21:04.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.802+0000 7f31f9693640 1 -- start start 2024-04-02T13:21:04.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f9693640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31f40d00d0 0x7f31f4151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f9693640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31f40d2090 0x7f31f409bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f9693640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31f40d2470 0x7f31f409c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f9693640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f31f40cead0 con 0x7f31f40d2090 2024-04-02T13:21:04.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f9693640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31f40ce950 con 0x7f31f40d2470 2024-04-02T13:21:04.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f9693640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f31f406fc90 con 0x7f31f40d00d0 2024-04-02T13:21:04.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f27fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31f40d2090 0x7f31f409bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f27fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31f40d2090 0x7f31f409bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44924/0 (socket says 172.21.15.44:44924) 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f27fc640 1 -- 172.21.15.44:0/946446448 learned_addr learned my addr 172.21.15.44:0/946446448 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f37fe640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31f40d2470 0x7f31f409c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f2ffd640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31f40d00d0 0x7f31f4151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f27fc640 1 -- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31f40d00d0 msgr2=0x7f31f4151320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f27fc640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31f40d00d0 0x7f31f4151320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f27fc640 1 -- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31f40d2470 msgr2=0x7f31f409c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.803+0000 7f31f27fc640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31f40d2470 0x7f31f409c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.804+0000 7f31f27fc640 1 -- 172.21.15.44:0/946446448 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f409ec30 con 0x7f31f40d2090 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.804+0000 7f31f2ffd640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31f40d00d0 0x7f31f4151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.804+0000 7f31f27fc640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31f40d2090 0x7f31f409bc00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f31dc02bf70 tx=0x7f31dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.804+0000 7f31eb7fe640 1 -- 172.21.15.44:0/946446448 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31dc034040 con 0x7f31f40d2090 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.804+0000 7f31eb7fe640 1 -- 172.21.15.44:0/946446448 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31dc03acc0 con 0x7f31f40d2090 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.804+0000 7f31eb7fe640 1 -- 172.21.15.44:0/946446448 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31dc044630 con 0x7f31f40d2090 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.804+0000 7f31f9693640 1 -- 172.21.15.44:0/946446448 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f409ef10 con 0x7f31f40d2090 2024-04-02T13:21:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.804+0000 7f31f9693640 1 -- 172.21.15.44:0/946446448 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f409f4d0 con 0x7f31f40d2090 2024-04-02T13:21:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.805+0000 7f31f9693640 1 -- 172.21.15.44:0/946446448 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31f4068c00 con 0x7f31f40d2090 2024-04-02T13:21:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.805+0000 7f31eb7fe640 1 -- 172.21.15.44:0/946446448 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f31dc04b060 con 0x7f31f40d2090 2024-04-02T13:21:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.806+0000 7f31eb7fe640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f31d406c8f0 0x7f31d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:04.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.806+0000 7f31eb7fe640 1 -- 172.21.15.44:0/946446448 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f31dc0add20 con 0x7f31f40d2090 2024-04-02T13:21:04.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.806+0000 7f31f2ffd640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f31d406c8f0 0x7f31d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:04.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.806+0000 7f31f2ffd640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f31d406c8f0 0x7f31d406edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0015e0 tx=0x7f31ec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:04.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.810+0000 7f31eb7fe640 1 -- 172.21.15.44:0/946446448 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f31dc03f070 con 0x7f31f40d2090 2024-04-02T13:21:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.997+0000 7f31f9693640 1 -- 172.21.15.44:0/946446448 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f31f40d1890 con 0x7f31f40d2090 2024-04-02T13:21:05.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.997+0000 7f31eb7fe640 1 -- 172.21.15.44:0/946446448 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f31dc07bea0 con 0x7f31f40d2090 2024-04-02T13:21:05.002 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.999+0000 7f31f9693640 1 -- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f31d406c8f0 msgr2=0x7f31d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.999+0000 7f31f9693640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f31d406c8f0 0x7f31d406edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0015e0 tx=0x7f31ec009040 comp rx=0 tx=0).stop 2024-04-02T13:21:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.999+0000 7f31f9693640 1 -- 172.21.15.44:0/946446448 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31f40d2090 msgr2=0x7f31f409bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.999+0000 7f31f9693640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31f40d2090 0x7f31f409bc00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f31dc02bf70 tx=0x7f31dc0046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.999+0000 7f31f9693640 1 -- 172.21.15.44:0/946446448 shutdown_connections 2024-04-02T13:21:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.999+0000 7f31f9693640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f31d406c8f0 0x7f31d406edb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.999+0000 7f31f9693640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31f40d2470 0x7f31f409c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.999+0000 7f31f9693640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31f40d2090 0x7f31f409bc00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.999+0000 7f31f9693640 1 --2- 172.21.15.44:0/946446448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31f40d00d0 0x7f31f4151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.999+0000 7f31f9693640 1 -- 172.21.15.44:0/946446448 >> 172.21.15.44:0/946446448 conn(0x7f31f409aa20 msgr2=0x7f31f40b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.999+0000 7f31f9693640 1 -- 172.21.15.44:0/946446448 shutdown_connections 2024-04-02T13:21:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:04.999+0000 7f31f9693640 1 -- 172.21.15.44:0/946446448 wait complete. 2024-04-02T13:21:05.012 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:05.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.115+0000 7f726bec5640 1 Processor -- start 2024-04-02T13:21:05.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.115+0000 7f726bec5640 1 -- start start 2024-04-02T13:21:05.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.116+0000 7f726bec5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f72640d00d0 0x7f726415ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.116+0000 7f726bec5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f72640d2470 0x7f726415e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.116+0000 7f726bec5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7264070040 con 0x7f72640d2090 2024-04-02T13:21:05.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.116+0000 7f726bec5640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f726406c5e0 con 0x7f72640d2470 2024-04-02T13:21:05.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.116+0000 7f726bec5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7264070320 con 0x7f72640d00d0 2024-04-02T13:21:05.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.116+0000 7f726a462640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f72640d2470 0x7f726415e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.116+0000 7f726a462640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f72640d2470 0x7f726415e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44928/0 (socket says 172.21.15.44:44928) 2024-04-02T13:21:05.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.116+0000 7f726a462640 1 -- 172.21.15.44:0/1338196208 learned_addr learned my addr 172.21.15.44:0/1338196208 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:05.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.116+0000 7f7269460640 1 --2- 172.21.15.44:0/1338196208 >> v2:172.21.15.69:3300/0 conn(0x7f72640d00d0 0x7f726415ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.117+0000 7f726a462640 1 -- 172.21.15.44:0/1338196208 >> v2:172.21.15.69:3300/0 conn(0x7f72640d00d0 msgr2=0x7f726415ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.117+0000 7f726a462640 1 --2- 172.21.15.44:0/1338196208 >> v2:172.21.15.69:3300/0 conn(0x7f72640d00d0 0x7f726415ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.117+0000 7f726a462640 1 -- 172.21.15.44:0/1338196208 >> v1:172.21.15.69:6789/0 conn(0x7f72640d2090 legacy=0x7f726406f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.117+0000 7f7268c5f640 1 -- 172.21.15.44:0/1338196208 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4280441758 0 0) 0x7f7264070040 con 0x7f72640d2090 2024-04-02T13:21:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.117+0000 7f726a462640 1 -- 172.21.15.44:0/1338196208 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7264070600 con 0x7f72640d2470 2024-04-02T13:21:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.117+0000 7f726a462640 1 --2- 172.21.15.44:0/1338196208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72640d2470 0x7f726415e300 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f725c009870 tx=0x7f725c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e53970c2a9cfe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.118+0000 7f7268c5f640 1 -- 172.21.15.44:0/1338196208 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c034070 con 0x7f72640d2470 2024-04-02T13:21:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.118+0000 7f7268c5f640 1 -- 172.21.15.44:0/1338196208 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f725c030860 con 0x7f72640d2470 2024-04-02T13:21:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.118+0000 7f7268c5f640 1 -- 172.21.15.44:0/1338196208 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c030c30 con 0x7f72640d2470 2024-04-02T13:21:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.118+0000 7f726bec5640 1 -- 172.21.15.44:0/1338196208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72640d2470 msgr2=0x7f726415e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.118+0000 7f726bec5640 1 --2- 172.21.15.44:0/1338196208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72640d2470 0x7f726415e300 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f725c009870 tx=0x7f725c02b9a0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.118+0000 7f726bec5640 1 -- 172.21.15.44:0/1338196208 shutdown_connections 2024-04-02T13:21:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.118+0000 7f726bec5640 1 --2- 172.21.15.44:0/1338196208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72640d2470 0x7f726415e300 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.118+0000 7f726bec5640 1 --2- 172.21.15.44:0/1338196208 >> v2:172.21.15.69:3300/0 conn(0x7f72640d00d0 0x7f726415ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.118+0000 7f726bec5640 1 -- 172.21.15.44:0/1338196208 >> 172.21.15.44:0/1338196208 conn(0x7f726409aa20 msgr2=0x7f726409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.118+0000 7f726bec5640 1 -- 172.21.15.44:0/1338196208 shutdown_connections 2024-04-02T13:21:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.118+0000 7f726bec5640 1 -- 172.21.15.44:0/1338196208 wait complete. 2024-04-02T13:21:05.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.119+0000 7f726bec5640 1 Processor -- start 2024-04-02T13:21:05.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.119+0000 7f726bec5640 1 -- start start 2024-04-02T13:21:05.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.120+0000 7f726bec5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72640d00d0 0x7f72641e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.120+0000 7f726bec5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72640d2090 0x7f72640b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.120+0000 7f726bec5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72640d2470 0x7f726414ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.120+0000 7f726bec5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72640cead0 con 0x7f72640d00d0 2024-04-02T13:21:05.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.120+0000 7f726bec5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72640ce950 con 0x7f72640d2090 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.120+0000 7f726bec5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f726406fc90 con 0x7f72640d2470 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.120+0000 7f7269c61640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72640d00d0 0x7f72641e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.120+0000 7f7269c61640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72640d00d0 0x7f72641e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44940/0 (socket says 172.21.15.44:44940) 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.120+0000 7f7269c61640 1 -- 172.21.15.44:0/1089955051 learned_addr learned my addr 172.21.15.44:0/1089955051 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.120+0000 7f726a462640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72640d2470 0x7f726414ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.120+0000 7f7269460640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72640d2090 0x7f72640b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.121+0000 7f7269c61640 1 -- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72640d2470 msgr2=0x7f726414ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.121+0000 7f7269c61640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72640d2470 0x7f726414ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.121+0000 7f7269c61640 1 -- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72640d2090 msgr2=0x7f72640b64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.121+0000 7f7269c61640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72640d2090 0x7f72640b64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.121+0000 7f7269c61640 1 -- 172.21.15.44:0/1089955051 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72641f44a0 con 0x7f72640d00d0 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.121+0000 7f726a462640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72640d2470 0x7f726414ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.121+0000 7f7269c61640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72640d00d0 0x7f72641e4090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f725400c7f0 tx=0x7f725400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.121+0000 7f72627fc640 1 -- 172.21.15.44:0/1089955051 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7254017070 con 0x7f72640d00d0 2024-04-02T13:21:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.121+0000 7f72627fc640 1 -- 172.21.15.44:0/1089955051 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7254003da0 con 0x7f72640d00d0 2024-04-02T13:21:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.121+0000 7f726bec5640 1 -- 172.21.15.44:0/1089955051 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72641e4950 con 0x7f72640d00d0 2024-04-02T13:21:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.121+0000 7f72627fc640 1 -- 172.21.15.44:0/1089955051 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7254005290 con 0x7f72640d00d0 2024-04-02T13:21:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.121+0000 7f726bec5640 1 -- 172.21.15.44:0/1089955051 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72641e4f10 con 0x7f72640d00d0 2024-04-02T13:21:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.122+0000 7f726bec5640 1 -- 172.21.15.44:0/1089955051 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72641f4670 con 0x7f72640d00d0 2024-04-02T13:21:05.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.122+0000 7f72627fc640 1 -- 172.21.15.44:0/1089955051 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f725401a070 con 0x7f72640d00d0 2024-04-02T13:21:05.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.123+0000 7f72627fc640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f725006c8f0 0x7f725006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.123+0000 7f72627fc640 1 -- 172.21.15.44:0/1089955051 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f725408b540 con 0x7f72640d00d0 2024-04-02T13:21:05.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.123+0000 7f7269460640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f725006c8f0 0x7f725006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.124+0000 7f7269460640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f725006c8f0 0x7f725006edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f726406bb40 tx=0x7f7258008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:05.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.127+0000 7f72627fc640 1 -- 172.21.15.44:0/1089955051 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7254059630 con 0x7f72640d00d0 2024-04-02T13:21:05.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.313+0000 7f726bec5640 1 -- 172.21.15.44:0/1089955051 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7264068c00 con 0x7f72640d00d0 2024-04-02T13:21:05.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.313+0000 7f72627fc640 1 -- 172.21.15.44:0/1089955051 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f7254058f30 con 0x7f72640d00d0 2024-04-02T13:21:05.315 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:05.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.315+0000 7f726bec5640 1 -- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f725006c8f0 msgr2=0x7f725006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.315+0000 7f726bec5640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f725006c8f0 0x7f725006edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f726406bb40 tx=0x7f7258008040 comp rx=0 tx=0).stop 2024-04-02T13:21:05.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.315+0000 7f726bec5640 1 -- 172.21.15.44:0/1089955051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72640d00d0 msgr2=0x7f72641e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.315+0000 7f726bec5640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72640d00d0 0x7f72641e4090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f725400c7f0 tx=0x7f725400ccb0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.315+0000 7f726bec5640 1 -- 172.21.15.44:0/1089955051 shutdown_connections 2024-04-02T13:21:05.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.315+0000 7f726bec5640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f725006c8f0 0x7f725006edb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.315+0000 7f726bec5640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72640d2470 0x7f726414ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.315+0000 7f726bec5640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72640d2090 0x7f72640b64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.315+0000 7f726bec5640 1 --2- 172.21.15.44:0/1089955051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72640d00d0 0x7f72641e4090 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.315+0000 7f726bec5640 1 -- 172.21.15.44:0/1089955051 >> 172.21.15.44:0/1089955051 conn(0x7f726409aa20 msgr2=0x7f72640d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:05.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.315+0000 7f726bec5640 1 -- 172.21.15.44:0/1089955051 shutdown_connections 2024-04-02T13:21:05.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.316+0000 7f726bec5640 1 -- 172.21.15.44:0/1089955051 wait complete. 2024-04-02T13:21:05.328 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:05.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.431+0000 7f571179d640 1 Processor -- start 2024-04-02T13:21:05.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.431+0000 7f571179d640 1 -- start start 2024-04-02T13:21:05.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.431+0000 7f571179d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f570c1565d0 0x7f570c0d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.431+0000 7f571179d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f570c0cf5d0 0x7f570c15c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.431+0000 7f571179d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570c160b10 con 0x7f570c0d05e0 2024-04-02T13:21:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.431+0000 7f571179d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f570c0cfad0 con 0x7f570c0cf5d0 2024-04-02T13:21:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.431+0000 7f571179d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f570c15fd50 con 0x7f570c1565d0 2024-04-02T13:21:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.432+0000 7f570b7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f570c0cf5d0 0x7f570c15c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.432+0000 7f570a7fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f570c0d05e0 0x7f570c0ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54656/0 (socket says 172.21.15.44:54656) 2024-04-02T13:21:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.432+0000 7f570a7fc640 1 -- 172.21.15.44:0/787629540 learned_addr learned my addr 172.21.15.44:0/787629540 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.432+0000 7f570affd640 1 --2- 172.21.15.44:0/787629540 >> v2:172.21.15.69:3300/0 conn(0x7f570c1565d0 0x7f570c0d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.432+0000 7f570b7fe640 1 -- 172.21.15.44:0/787629540 >> v2:172.21.15.69:3300/0 conn(0x7f570c1565d0 msgr2=0x7f570c0d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.432+0000 7f570b7fe640 1 --2- 172.21.15.44:0/787629540 >> v2:172.21.15.69:3300/0 conn(0x7f570c1565d0 0x7f570c0d00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.432+0000 7f570b7fe640 1 -- 172.21.15.44:0/787629540 >> v1:172.21.15.44:6789/0 conn(0x7f570c0d05e0 legacy=0x7f570c0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.432+0000 7f570b7fe640 1 -- 172.21.15.44:0/787629540 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570c160030 con 0x7f570c0cf5d0 2024-04-02T13:21:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.432+0000 7f5709ffb640 1 -- 172.21.15.44:0/787629540 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4040924252 0 0) 0x7f570c160b10 con 0x7f570c0d05e0 2024-04-02T13:21:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.433+0000 7f570b7fe640 1 --2- 172.21.15.44:0/787629540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f570c0cf5d0 0x7f570c15c7a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f570c0d0cd0 tx=0x7f570402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5758ba2da723190 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.433+0000 7f5709ffb640 1 -- 172.21.15.44:0/787629540 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5704039070 con 0x7f570c0cf5d0 2024-04-02T13:21:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.433+0000 7f5709ffb640 1 -- 172.21.15.44:0/787629540 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f570402cbf0 con 0x7f570c0cf5d0 2024-04-02T13:21:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.433+0000 7f5709ffb640 1 -- 172.21.15.44:0/787629540 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5704034440 con 0x7f570c0cf5d0 2024-04-02T13:21:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.433+0000 7f571179d640 1 -- 172.21.15.44:0/787629540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f570c0cf5d0 msgr2=0x7f570c15c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.433+0000 7f571179d640 1 --2- 172.21.15.44:0/787629540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f570c0cf5d0 0x7f570c15c7a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f570c0d0cd0 tx=0x7f570402be40 comp rx=0 tx=0).stop 2024-04-02T13:21:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.433+0000 7f571179d640 1 -- 172.21.15.44:0/787629540 shutdown_connections 2024-04-02T13:21:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.433+0000 7f571179d640 1 --2- 172.21.15.44:0/787629540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f570c0cf5d0 0x7f570c15c7a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.433+0000 7f571179d640 1 --2- 172.21.15.44:0/787629540 >> v2:172.21.15.69:3300/0 conn(0x7f570c1565d0 0x7f570c0d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.433+0000 7f571179d640 1 -- 172.21.15.44:0/787629540 >> 172.21.15.44:0/787629540 conn(0x7f570c06b920 msgr2=0x7f570c06bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.433+0000 7f571179d640 1 -- 172.21.15.44:0/787629540 shutdown_connections 2024-04-02T13:21:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.433+0000 7f571179d640 1 -- 172.21.15.44:0/787629540 wait complete. 2024-04-02T13:21:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.435+0000 7f571179d640 1 Processor -- start 2024-04-02T13:21:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.435+0000 7f571179d640 1 -- start start 2024-04-02T13:21:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.435+0000 7f571179d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f570c0cf5d0 0x7f570c14d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.435+0000 7f571179d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f570c0d05e0 0x7f570c071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.435+0000 7f571179d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f570c1565d0 0x7f570c071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.435+0000 7f571179d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f570c1569b0 con 0x7f570c0cf5d0 2024-04-02T13:21:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.435+0000 7f571179d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f570c1605f0 con 0x7f570c0d05e0 2024-04-02T13:21:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.435+0000 7f571179d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f570c156b30 con 0x7f570c1565d0 2024-04-02T13:21:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f570affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f570c0cf5d0 0x7f570c14d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f570affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f570c0cf5d0 0x7f570c14d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44964/0 (socket says 172.21.15.44:44964) 2024-04-02T13:21:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f570affd640 1 -- 172.21.15.44:0/2164547519 learned_addr learned my addr 172.21.15.44:0/2164547519 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f570affd640 1 -- 172.21.15.44:0/2164547519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f570c1565d0 msgr2=0x7f570c071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f570affd640 1 --2- 172.21.15.44:0/2164547519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f570c1565d0 0x7f570c071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f570affd640 1 -- 172.21.15.44:0/2164547519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f570c0d05e0 msgr2=0x7f570c071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f570a7fc640 1 --2- 172.21.15.44:0/2164547519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f570c0d05e0 0x7f570c071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f570affd640 1 --2- 172.21.15.44:0/2164547519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f570c0d05e0 0x7f570c071970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f570affd640 1 -- 172.21.15.44:0/2164547519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570c1e7580 con 0x7f570c0cf5d0 2024-04-02T13:21:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f570affd640 1 --2- 172.21.15.44:0/2164547519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f570c0cf5d0 0x7f570c14d900 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f56fc00c7f0 tx=0x7f56fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f56f37fe640 1 -- 172.21.15.44:0/2164547519 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc015070 con 0x7f570c0cf5d0 2024-04-02T13:21:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f56f37fe640 1 -- 172.21.15.44:0/2164547519 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56fc004240 con 0x7f570c0cf5d0 2024-04-02T13:21:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f571179d640 1 -- 172.21.15.44:0/2164547519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570c1e5af0 con 0x7f570c0cf5d0 2024-04-02T13:21:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f56f37fe640 1 -- 172.21.15.44:0/2164547519 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc007620 con 0x7f570c0cf5d0 2024-04-02T13:21:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.436+0000 7f571179d640 1 -- 172.21.15.44:0/2164547519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f570c1e60b0 con 0x7f570c0cf5d0 2024-04-02T13:21:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.437+0000 7f571179d640 1 -- 172.21.15.44:0/2164547519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f570c0684f0 con 0x7f570c0cf5d0 2024-04-02T13:21:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.438+0000 7f56f37fe640 1 -- 172.21.15.44:0/2164547519 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f56fc0079d0 con 0x7f570c0cf5d0 2024-04-02T13:21:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.438+0000 7f56f37fe640 1 --2- 172.21.15.44:0/2164547519 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56f806c920 0x7f56f806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.438+0000 7f56f37fe640 1 -- 172.21.15.44:0/2164547519 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f56fc08bf90 con 0x7f570c0cf5d0 2024-04-02T13:21:05.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.438+0000 7f570a7fc640 1 --2- 172.21.15.44:0/2164547519 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56f806c920 0x7f56f806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.439+0000 7f570a7fc640 1 --2- 172.21.15.44:0/2164547519 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56f806c920 0x7f56f806ede0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f570c159db0 tx=0x7f5700009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.443+0000 7f56f37fe640 1 -- 172.21.15.44:0/2164547519 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f56fc059ff0 con 0x7f570c0cf5d0 2024-04-02T13:21:05.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.627+0000 7f571179d640 1 -- 172.21.15.44:0/2164547519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f570c06c1c0 con 0x7f570c0cf5d0 2024-04-02T13:21:05.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.627+0000 7f56f37fe640 1 -- 172.21.15.44:0/2164547519 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f56fc0598f0 con 0x7f570c0cf5d0 2024-04-02T13:21:05.629 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.629+0000 7f571179d640 1 -- 172.21.15.44:0/2164547519 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56f806c920 msgr2=0x7f56f806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.629+0000 7f571179d640 1 --2- 172.21.15.44:0/2164547519 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56f806c920 0x7f56f806ede0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f570c159db0 tx=0x7f5700009040 comp rx=0 tx=0).stop 2024-04-02T13:21:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.629+0000 7f571179d640 1 -- 172.21.15.44:0/2164547519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f570c0cf5d0 msgr2=0x7f570c14d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.629+0000 7f571179d640 1 --2- 172.21.15.44:0/2164547519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f570c0cf5d0 0x7f570c14d900 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f56fc00c7f0 tx=0x7f56fc00ccb0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.629+0000 7f571179d640 1 -- 172.21.15.44:0/2164547519 shutdown_connections 2024-04-02T13:21:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.629+0000 7f571179d640 1 --2- 172.21.15.44:0/2164547519 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56f806c920 0x7f56f806ede0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.629+0000 7f571179d640 1 --2- 172.21.15.44:0/2164547519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f570c1565d0 0x7f570c071eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.629+0000 7f571179d640 1 --2- 172.21.15.44:0/2164547519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f570c0d05e0 0x7f570c071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.629+0000 7f571179d640 1 --2- 172.21.15.44:0/2164547519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f570c0cf5d0 0x7f570c14d900 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.629+0000 7f571179d640 1 -- 172.21.15.44:0/2164547519 >> 172.21.15.44:0/2164547519 conn(0x7f570c06b920 msgr2=0x7f570c071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.629+0000 7f571179d640 1 -- 172.21.15.44:0/2164547519 shutdown_connections 2024-04-02T13:21:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.629+0000 7f571179d640 1 -- 172.21.15.44:0/2164547519 wait complete. 2024-04-02T13:21:05.641 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:05.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.744+0000 7fa414fc8640 1 Processor -- start 2024-04-02T13:21:05.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.744+0000 7fa414fc8640 1 -- start start 2024-04-02T13:21:05.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.745+0000 7fa414fc8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa41015e260 0x7fa4100d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.745+0000 7fa414fc8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41015e640 0x7fa4100d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.745+0000 7fa414fc8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4100cf120 con 0x7fa4100cec60 2024-04-02T13:21:05.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.745+0000 7fa414fc8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa4100b5a50 con 0x7fa41015e260 2024-04-02T13:21:05.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.745+0000 7fa414fc8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa4100d01c0 con 0x7fa41015e640 2024-04-02T13:21:05.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.745+0000 7fa40e59c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa41015e260 0x7fa4100d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.745+0000 7fa40ed9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa4100cec60 0x7fa410161f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54662/0 (socket says 172.21.15.44:54662) 2024-04-02T13:21:05.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.745+0000 7fa40ed9d640 1 -- 172.21.15.44:0/2843684298 learned_addr learned my addr 172.21.15.44:0/2843684298 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:05.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.745+0000 7fa40dd9b640 1 --2- 172.21.15.44:0/2843684298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41015e640 0x7fa4100d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.746+0000 7fa40e59c640 1 -- 172.21.15.44:0/2843684298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41015e640 msgr2=0x7fa4100d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:05.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.746+0000 7fa40e59c640 1 --2- 172.21.15.44:0/2843684298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41015e640 0x7fa4100d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.746+0000 7fa40e59c640 1 -- 172.21.15.44:0/2843684298 >> v1:172.21.15.44:6789/0 conn(0x7fa4100cec60 legacy=0x7fa410161f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.746+0000 7fa40e59c640 1 -- 172.21.15.44:0/2843684298 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4100cf400 con 0x7fa41015e260 2024-04-02T13:21:05.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.746+0000 7fa40d59a640 1 -- 172.21.15.44:0/2843684298 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3863305349 0 0) 0x7fa4100cf120 con 0x7fa4100cec60 2024-04-02T13:21:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.746+0000 7fa40e59c640 1 --2- 172.21.15.44:0/2843684298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41015e260 0x7fa4100d2960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa4080097a0 tx=0x7fa40802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7ca3d68de130b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.747+0000 7fa40d59a640 1 -- 172.21.15.44:0/2843684298 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408035020 con 0x7fa41015e260 2024-04-02T13:21:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.747+0000 7fa40d59a640 1 -- 172.21.15.44:0/2843684298 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa408031830 con 0x7fa41015e260 2024-04-02T13:21:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.747+0000 7fa40d59a640 1 -- 172.21.15.44:0/2843684298 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408031c00 con 0x7fa41015e260 2024-04-02T13:21:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.747+0000 7fa414fc8640 1 -- 172.21.15.44:0/2843684298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41015e260 msgr2=0x7fa4100d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.747+0000 7fa414fc8640 1 --2- 172.21.15.44:0/2843684298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41015e260 0x7fa4100d2960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa4080097a0 tx=0x7fa40802b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.747+0000 7fa414fc8640 1 -- 172.21.15.44:0/2843684298 shutdown_connections 2024-04-02T13:21:05.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.747+0000 7fa414fc8640 1 --2- 172.21.15.44:0/2843684298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41015e640 0x7fa4100d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.747+0000 7fa414fc8640 1 --2- 172.21.15.44:0/2843684298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41015e260 0x7fa4100d2960 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.747+0000 7fa414fc8640 1 -- 172.21.15.44:0/2843684298 >> 172.21.15.44:0/2843684298 conn(0x7fa41009d3c0 msgr2=0x7fa4100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:05.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.747+0000 7fa414fc8640 1 -- 172.21.15.44:0/2843684298 shutdown_connections 2024-04-02T13:21:05.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.747+0000 7fa414fc8640 1 -- 172.21.15.44:0/2843684298 wait complete. 2024-04-02T13:21:05.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.749+0000 7fa414fc8640 1 Processor -- start 2024-04-02T13:21:05.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.749+0000 7fa414fc8640 1 -- start start 2024-04-02T13:21:05.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.749+0000 7fa414fc8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4100cec60 0x7fa4101535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.749+0000 7fa414fc8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41015e260 0x7fa4100c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.749+0000 7fa414fc8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41015e640 0x7fa4100c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.749+0000 7fa414fc8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa4100cfca0 con 0x7fa41015e640 2024-04-02T13:21:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.749+0000 7fa414fc8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa4100cf9c0 con 0x7fa4100cec60 2024-04-02T13:21:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.749+0000 7fa414fc8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa4100d2ea0 con 0x7fa41015e260 2024-04-02T13:21:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.749+0000 7fa40ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41015e640 0x7fa4100c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.749+0000 7fa40ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41015e640 0x7fa4100c99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44970/0 (socket says 172.21.15.44:44970) 2024-04-02T13:21:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.749+0000 7fa40ed9d640 1 -- 172.21.15.44:0/3650800030 learned_addr learned my addr 172.21.15.44:0/3650800030 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa40e59c640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4100cec60 0x7fa4101535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa40dd9b640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41015e260 0x7fa4100c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa40ed9d640 1 -- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41015e260 msgr2=0x7fa4100c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa40ed9d640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41015e260 0x7fa4100c9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa40ed9d640 1 -- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4100cec60 msgr2=0x7fa4101535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa40ed9d640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4100cec60 0x7fa4101535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa40ed9d640 1 -- 172.21.15.44:0/3650800030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4100b78b0 con 0x7fa41015e640 2024-04-02T13:21:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa40e59c640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4100cec60 0x7fa4101535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa40ed9d640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41015e640 0x7fa4100c99c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa40000c730 tx=0x7fa40000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa3feffd640 1 -- 172.21.15.44:0/3650800030 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa400018070 con 0x7fa41015e640 2024-04-02T13:21:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa3feffd640 1 -- 172.21.15.44:0/3650800030 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa40000ce40 con 0x7fa41015e640 2024-04-02T13:21:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa3feffd640 1 -- 172.21.15.44:0/3650800030 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40000a860 con 0x7fa41015e640 2024-04-02T13:21:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa414fc8640 1 -- 172.21.15.44:0/3650800030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4100b7b90 con 0x7fa41015e640 2024-04-02T13:21:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.750+0000 7fa414fc8640 1 -- 172.21.15.44:0/3650800030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4100b8150 con 0x7fa41015e640 2024-04-02T13:21:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.751+0000 7fa414fc8640 1 -- 172.21.15.44:0/3650800030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa41008d820 con 0x7fa41015e640 2024-04-02T13:21:05.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.752+0000 7fa3feffd640 1 -- 172.21.15.44:0/3650800030 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa40001b070 con 0x7fa41015e640 2024-04-02T13:21:05.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.752+0000 7fa3feffd640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa3f006c8f0 0x7fa3f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:05.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.752+0000 7fa3feffd640 1 -- 172.21.15.44:0/3650800030 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa40008b480 con 0x7fa41015e640 2024-04-02T13:21:05.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.752+0000 7fa40e59c640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa3f006c8f0 0x7fa3f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:05.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.753+0000 7fa40e59c640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa3f006c8f0 0x7fa3f006edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa408036000 tx=0x7fa408008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:05.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.757+0000 7fa3feffd640 1 -- 172.21.15.44:0/3650800030 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa41008d820 con 0x7fa41015e640 2024-04-02T13:21:05.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.941+0000 7fa414fc8640 1 -- 172.21.15.44:0/3650800030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa4100684f0 con 0x7fa41015e640 2024-04-02T13:21:05.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.941+0000 7fa3feffd640 1 -- 172.21.15.44:0/3650800030 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fa400059570 con 0x7fa41015e640 2024-04-02T13:21:05.943 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:05.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.943+0000 7fa414fc8640 1 -- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa3f006c8f0 msgr2=0x7fa3f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.943+0000 7fa414fc8640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa3f006c8f0 0x7fa3f006edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa408036000 tx=0x7fa408008030 comp rx=0 tx=0).stop 2024-04-02T13:21:05.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.943+0000 7fa414fc8640 1 -- 172.21.15.44:0/3650800030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41015e640 msgr2=0x7fa4100c99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:05.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.943+0000 7fa414fc8640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41015e640 0x7fa4100c99c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa40000c730 tx=0x7fa40000cbf0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.943+0000 7fa414fc8640 1 -- 172.21.15.44:0/3650800030 shutdown_connections 2024-04-02T13:21:05.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.943+0000 7fa414fc8640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa3f006c8f0 0x7fa3f006edb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.943+0000 7fa414fc8640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41015e640 0x7fa4100c99c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.943+0000 7fa414fc8640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41015e260 0x7fa4100c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.943+0000 7fa414fc8640 1 --2- 172.21.15.44:0/3650800030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4100cec60 0x7fa4101535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:05.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.943+0000 7fa414fc8640 1 -- 172.21.15.44:0/3650800030 >> 172.21.15.44:0/3650800030 conn(0x7fa41009d3c0 msgr2=0x7fa41006d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:05.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.943+0000 7fa414fc8640 1 -- 172.21.15.44:0/3650800030 shutdown_connections 2024-04-02T13:21:05.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:05.943+0000 7fa414fc8640 1 -- 172.21.15.44:0/3650800030 wait complete. 2024-04-02T13:21:05.955 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:06.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.056+0000 7f5718a70640 1 Processor -- start 2024-04-02T13:21:06.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.056+0000 7f5718a70640 1 -- start start 2024-04-02T13:21:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5718a70640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57140729b0 0x7f571415f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5718a70640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f571415c2e0 con 0x7f57140b5670 2024-04-02T13:21:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5718a70640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57140b45d0 con 0x7f571415aef0 2024-04-02T13:21:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5718a70640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f57140af8f0 con 0x7f57140729b0 2024-04-02T13:21:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f570bfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f57140b5670 0x7f57140722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54664/0 (socket says 172.21.15.44:54664) 2024-04-02T13:21:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f570bfff640 1 -- 172.21.15.44:0/2564989930 learned_addr learned my addr 172.21.15.44:0/2564989930 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5712d9d640 1 --2- 172.21.15.44:0/2564989930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57140729b0 0x7f571415f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3559248306 0 0) 0x7f571415c2e0 con 0x7f57140b5670 2024-04-02T13:21:06.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5700003520 con 0x7f57140b5670 2024-04-02T13:21:06.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2826516544 0 0) 0x7f5700003520 con 0x7f57140b5670 2024-04-02T13:21:06.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57140729b0 msgr2=0x7f571415f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:06.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5711d9b640 1 --2- 172.21.15.44:0/2564989930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57140729b0 0x7f571415f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 >> v1:172.21.15.69:6789/0 conn(0x7f571415aef0 legacy=0x7f57140b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57140b48b0 con 0x7f57140b5670 2024-04-02T13:21:06.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f56fc001670 con 0x7f57140b5670 2024-04-02T13:21:06.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 >> v1:172.21.15.44:6789/0 conn(0x7f57140b5670 legacy=0x7f57140722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5711d9b640 1 --2- 172.21.15.44:0/2564989930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57000039e0 0x7f5700003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5711d9b640 1 --2- 172.21.15.44:0/2564989930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5700004800 0x7f5700004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.057+0000 7f5711d9b640 1 --2- 172.21.15.44:0/2564989930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57140729b0 0x7f57000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.058+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f57140af8f0 con 0x7f5700004800 2024-04-02T13:21:06.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.058+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f570000baa0 con 0x7f57000039e0 2024-04-02T13:21:06.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.058+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f570000bca0 con 0x7f57140729b0 2024-04-02T13:21:06.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.058+0000 7f570bfff640 1 --2- 172.21.15.44:0/2564989930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5700004800 0x7f5700004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.058+0000 7f571259c640 1 --2- 172.21.15.44:0/2564989930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57000039e0 0x7f5700003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.058+0000 7f5712d9d640 1 --2- 172.21.15.44:0/2564989930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57140729b0 0x7f57000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.058+0000 7f570bfff640 1 -- 172.21.15.44:0/2564989930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57140729b0 msgr2=0x7f57000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:06.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.058+0000 7f570bfff640 1 --2- 172.21.15.44:0/2564989930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57140729b0 0x7f57000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.058+0000 7f570bfff640 1 -- 172.21.15.44:0/2564989930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57000039e0 msgr2=0x7f5700003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:06.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.058+0000 7f570bfff640 1 --2- 172.21.15.44:0/2564989930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57000039e0 0x7f5700003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.058+0000 7f570bfff640 1 -- 172.21.15.44:0/2564989930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5700003520 con 0x7f5700004800 2024-04-02T13:21:06.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.059+0000 7f570bfff640 1 --2- 172.21.15.44:0/2564989930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5700004800 0x7f5700004c80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f56fc00c210 tx=0x7f5700005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4080695c5b0ad80 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:06.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.059+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc007cf0 con 0x7f5700004800 2024-04-02T13:21:06.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.059+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56fc030070 con 0x7f5700004800 2024-04-02T13:21:06.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.059+0000 7f5711d9b640 1 -- 172.21.15.44:0/2564989930 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc005d30 con 0x7f5700004800 2024-04-02T13:21:06.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.059+0000 7f5718a70640 1 -- 172.21.15.44:0/2564989930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5700004800 msgr2=0x7f5700004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.059+0000 7f5718a70640 1 --2- 172.21.15.44:0/2564989930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5700004800 0x7f5700004c80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f56fc00c210 tx=0x7f5700005830 comp rx=0 tx=0).stop 2024-04-02T13:21:06.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.059+0000 7f5712d9d640 1 -- 172.21.15.44:0/2564989930 reap_dead start 2024-04-02T13:21:06.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.059+0000 7f5718a70640 1 -- 172.21.15.44:0/2564989930 shutdown_connections 2024-04-02T13:21:06.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.059+0000 7f5718a70640 1 -- 172.21.15.44:0/2564989930 >> 172.21.15.44:0/2564989930 conn(0x7f57140b2af0 msgr2=0x7f57140b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:06.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.059+0000 7f5718a70640 1 -- 172.21.15.44:0/2564989930 shutdown_connections 2024-04-02T13:21:06.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.059+0000 7f5718a70640 1 -- 172.21.15.44:0/2564989930 wait complete. 2024-04-02T13:21:06.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.061+0000 7f5718a70640 1 Processor -- start 2024-04-02T13:21:06.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.061+0000 7f5718a70640 1 -- start start 2024-04-02T13:21:06.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.061+0000 7f5718a70640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5700004800 0x7f5714145850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.061+0000 7f5718a70640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5714159280 0x7f5714159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.061+0000 7f5718a70640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57140cd820 0x7f57140cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.061+0000 7f5718a70640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f57140b5150 con 0x7f5714159280 2024-04-02T13:21:06.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.061+0000 7f5718a70640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f57140b4e70 con 0x7f5700004800 2024-04-02T13:21:06.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.061+0000 7f5718a70640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f571415f5d0 con 0x7f57140cd820 2024-04-02T13:21:06.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f570bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5714159280 0x7f5714159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f570bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5714159280 0x7f5714159700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44986/0 (socket says 172.21.15.44:44986) 2024-04-02T13:21:06.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f570bfff640 1 -- 172.21.15.44:0/2519013746 learned_addr learned my addr 172.21.15.44:0/2519013746 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:06.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f571259c640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5700004800 0x7f5714145850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f5712d9d640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57140cd820 0x7f57140cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f570bfff640 1 -- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57140cd820 msgr2=0x7f57140cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f570bfff640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57140cd820 0x7f57140cdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f570bfff640 1 -- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5700004800 msgr2=0x7f5714145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f570bfff640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5700004800 0x7f5714145850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f570bfff640 1 -- 172.21.15.44:0/2519013746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57140ce4c0 con 0x7f5714159280 2024-04-02T13:21:06.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f571259c640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5700004800 0x7f5714145850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:06.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f570bfff640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5714159280 0x7f5714159700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f56fc00c210 tx=0x7f57140b47a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:06.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f570affd640 1 -- 172.21.15.44:0/2519013746 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc001c30 con 0x7f5714159280 2024-04-02T13:21:06.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f570affd640 1 -- 172.21.15.44:0/2519013746 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56fc004bf0 con 0x7f5714159280 2024-04-02T13:21:06.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f5718a70640 1 -- 172.21.15.44:0/2519013746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57141571b0 con 0x7f5714159280 2024-04-02T13:21:06.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f570affd640 1 -- 172.21.15.44:0/2519013746 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc007070 con 0x7f5714159280 2024-04-02T13:21:06.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.062+0000 7f5718a70640 1 -- 172.21.15.44:0/2519013746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5714157770 con 0x7f5714159280 2024-04-02T13:21:06.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.063+0000 7f5718a70640 1 -- 172.21.15.44:0/2519013746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5714068700 con 0x7f5714159280 2024-04-02T13:21:06.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.064+0000 7f570affd640 1 -- 172.21.15.44:0/2519013746 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f56fc047020 con 0x7f5714159280 2024-04-02T13:21:06.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.064+0000 7f570affd640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56f406c920 0x7f56f406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.064+0000 7f570affd640 1 -- 172.21.15.44:0/2519013746 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f56fc0acfb0 con 0x7f5714159280 2024-04-02T13:21:06.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.064+0000 7f571259c640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56f406c920 0x7f56f406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.065+0000 7f571259c640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56f406c920 0x7f56f406ede0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f57141e53a0 tx=0x7f5704009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:06.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.069+0000 7f570affd640 1 -- 172.21.15.44:0/2519013746 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5714068700 con 0x7f5714159280 2024-04-02T13:21:06.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.253+0000 7f5718a70640 1 -- 172.21.15.44:0/2519013746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5714070060 con 0x7f5714159280 2024-04-02T13:21:06.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.254+0000 7f570affd640 1 -- 172.21.15.44:0/2519013746 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f5714070060 con 0x7f5714159280 2024-04-02T13:21:06.256 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:06.256 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:20:25.282583+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.257+0000 7f5718a70640 1 -- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56f406c920 msgr2=0x7f56f406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.257+0000 7f5718a70640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56f406c920 0x7f56f406ede0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f57141e53a0 tx=0x7f5704009040 comp rx=0 tx=0).stop 2024-04-02T13:21:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.257+0000 7f5718a70640 1 -- 172.21.15.44:0/2519013746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5714159280 msgr2=0x7f5714159700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.257+0000 7f5718a70640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5714159280 0x7f5714159700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f56fc00c210 tx=0x7f57140b47a0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.257+0000 7f5718a70640 1 -- 172.21.15.44:0/2519013746 shutdown_connections 2024-04-02T13:21:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.257+0000 7f5718a70640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56f406c920 0x7f56f406ede0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.257+0000 7f5718a70640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57140cd820 0x7f57140cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.257+0000 7f5718a70640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5714159280 0x7f5714159700 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.257+0000 7f5718a70640 1 --2- 172.21.15.44:0/2519013746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5700004800 0x7f5714145850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.257+0000 7f5718a70640 1 -- 172.21.15.44:0/2519013746 >> 172.21.15.44:0/2519013746 conn(0x7f57140b2af0 msgr2=0x7f571415e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.257+0000 7f5718a70640 1 -- 172.21.15.44:0/2519013746 shutdown_connections 2024-04-02T13:21:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.257+0000 7f5718a70640 1 -- 172.21.15.44:0/2519013746 wait complete. 2024-04-02T13:21:06.269 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:06.270 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:06.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.373+0000 7f18dd97c640 1 Processor -- start 2024-04-02T13:21:06.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.373+0000 7f18dd97c640 1 -- start start 2024-04-02T13:21:06.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.374+0000 7f18dd97c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f18d815e260 0x7f18d80d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.374+0000 7f18dd97c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f18d80cec60 0x7f18d8168280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.374+0000 7f18dd97c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18d80cf120 con 0x7f18d815e640 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.374+0000 7f18dd97c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f18d80b5a50 con 0x7f18d815e260 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.374+0000 7f18dd97c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f18d80d01c0 con 0x7f18d80cec60 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.374+0000 7f18d6ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f18d815e260 0x7f18d80d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.374+0000 7f18d6ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f18d815e260 0x7f18d80d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44988/0 (socket says 172.21.15.44:44988) 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.374+0000 7f18d6ffd640 1 -- 172.21.15.44:0/1339206651 learned_addr learned my addr 172.21.15.44:0/1339206651 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.374+0000 7f18d77fe640 1 --2- 172.21.15.44:0/1339206651 >> v2:172.21.15.69:3300/0 conn(0x7f18d80cec60 0x7f18d8168280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.374+0000 7f18d6ffd640 1 -- 172.21.15.44:0/1339206651 >> v2:172.21.15.69:3300/0 conn(0x7f18d80cec60 msgr2=0x7f18d8168280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.374+0000 7f18d6ffd640 1 --2- 172.21.15.44:0/1339206651 >> v2:172.21.15.69:3300/0 conn(0x7f18d80cec60 0x7f18d8168280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.374+0000 7f18d6ffd640 1 -- 172.21.15.44:0/1339206651 >> v1:172.21.15.69:6789/0 conn(0x7f18d815e640 legacy=0x7f18d80d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.374+0000 7f18d6ffd640 1 -- 172.21.15.44:0/1339206651 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d80cf400 con 0x7f18d815e260 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.375+0000 7f18d6ffd640 1 --2- 172.21.15.44:0/1339206651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f18d815e260 0x7f18d80d2960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f18d80ce6c0 tx=0x7f18c802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a88593e7121739 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.375+0000 7f18d5ffb640 1 -- 172.21.15.44:0/1339206651 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c802ca80 con 0x7f18d815e260 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.375+0000 7f18d5ffb640 1 -- 172.21.15.44:0/1339206651 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c802cc20 con 0x7f18d815e260 2024-04-02T13:21:06.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.375+0000 7f18d5ffb640 1 -- 172.21.15.44:0/1339206651 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c80343f0 con 0x7f18d815e260 2024-04-02T13:21:06.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.376+0000 7f18dd97c640 1 -- 172.21.15.44:0/1339206651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f18d815e260 msgr2=0x7f18d80d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.376+0000 7f18dd97c640 1 --2- 172.21.15.44:0/1339206651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f18d815e260 0x7f18d80d2960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f18d80ce6c0 tx=0x7f18c802bee0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.376+0000 7f18dd97c640 1 -- 172.21.15.44:0/1339206651 shutdown_connections 2024-04-02T13:21:06.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.376+0000 7f18dd97c640 1 --2- 172.21.15.44:0/1339206651 >> v2:172.21.15.69:3300/0 conn(0x7f18d80cec60 0x7f18d8168280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.376+0000 7f18dd97c640 1 --2- 172.21.15.44:0/1339206651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f18d815e260 0x7f18d80d2960 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.376+0000 7f18dd97c640 1 -- 172.21.15.44:0/1339206651 >> 172.21.15.44:0/1339206651 conn(0x7f18d809d3c0 msgr2=0x7f18d80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:06.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.376+0000 7f18dd97c640 1 -- 172.21.15.44:0/1339206651 shutdown_connections 2024-04-02T13:21:06.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.376+0000 7f18dd97c640 1 -- 172.21.15.44:0/1339206651 wait complete. 2024-04-02T13:21:06.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.377+0000 7f18dd97c640 1 Processor -- start 2024-04-02T13:21:06.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.377+0000 7f18dd97c640 1 -- start start 2024-04-02T13:21:06.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18dd97c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18d80cec60 0x7f18d80c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18dd97c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18d815e260 0x7f18d80cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18dd97c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f18d815e640 0x7f18d806c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18dd97c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f18d80cfca0 con 0x7f18d815e640 2024-04-02T13:21:06.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18dd97c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f18d80cf9c0 con 0x7f18d815e260 2024-04-02T13:21:06.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18dd97c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f18d80d11e0 con 0x7f18d80cec60 2024-04-02T13:21:06.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18d77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f18d815e640 0x7f18d806c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18d77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f18d815e640 0x7f18d806c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45000/0 (socket says 172.21.15.44:45000) 2024-04-02T13:21:06.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18d77fe640 1 -- 172.21.15.44:0/936543200 learned_addr learned my addr 172.21.15.44:0/936543200 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:06.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18d6ffd640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18d80cec60 0x7f18d80c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18d67fc640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18d815e260 0x7f18d80cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18d77fe640 1 -- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18d80cec60 msgr2=0x7f18d80c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18d77fe640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18d80cec60 0x7f18d80c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18d77fe640 1 -- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18d815e260 msgr2=0x7f18d80cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18d77fe640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18d815e260 0x7f18d80cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.378+0000 7f18d77fe640 1 -- 172.21.15.44:0/936543200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d80ca4c0 con 0x7f18d815e640 2024-04-02T13:21:06.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.379+0000 7f18d77fe640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f18d815e640 0x7f18d806c360 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f18d000d800 tx=0x7f18d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:06.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.379+0000 7f18cffff640 1 -- 172.21.15.44:0/936543200 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d000b6c0 con 0x7f18d815e640 2024-04-02T13:21:06.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.379+0000 7f18cffff640 1 -- 172.21.15.44:0/936543200 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18d0019070 con 0x7f18d815e640 2024-04-02T13:21:06.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.379+0000 7f18dd97c640 1 -- 172.21.15.44:0/936543200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18d80cd230 con 0x7f18d815e640 2024-04-02T13:21:06.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.379+0000 7f18cffff640 1 -- 172.21.15.44:0/936543200 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d000bac0 con 0x7f18d815e640 2024-04-02T13:21:06.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.379+0000 7f18dd97c640 1 -- 172.21.15.44:0/936543200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18d80cd7f0 con 0x7f18d815e640 2024-04-02T13:21:06.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.380+0000 7f18dd97c640 1 -- 172.21.15.44:0/936543200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18d80684f0 con 0x7f18d815e640 2024-04-02T13:21:06.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.380+0000 7f18cffff640 1 -- 172.21.15.44:0/936543200 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f18d000bc60 con 0x7f18d815e640 2024-04-02T13:21:06.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.380+0000 7f18cffff640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f18b806c8f0 0x7f18b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.381+0000 7f18cffff640 1 -- 172.21.15.44:0/936543200 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f18d008c200 con 0x7f18d815e640 2024-04-02T13:21:06.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.381+0000 7f18d6ffd640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f18b806c8f0 0x7f18b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.381+0000 7f18d6ffd640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f18b806c8f0 0x7f18b806edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f18c8035000 tx=0x7f18c8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:06.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.385+0000 7f18cffff640 1 -- 172.21.15.44:0/936543200 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f18d0059ca0 con 0x7f18d815e640 2024-04-02T13:21:06.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.567+0000 7f18dd97c640 1 -- 172.21.15.44:0/936543200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f18d81ea630 con 0x7f18d815e640 2024-04-02T13:21:06.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.567+0000 7f18cffff640 1 -- 172.21.15.44:0/936543200 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f18d005dfa0 con 0x7f18d815e640 2024-04-02T13:21:06.569 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:06.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.569+0000 7f18dd97c640 1 -- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f18b806c8f0 msgr2=0x7f18b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.569+0000 7f18dd97c640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f18b806c8f0 0x7f18b806edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f18c8035000 tx=0x7f18c8030000 comp rx=0 tx=0).stop 2024-04-02T13:21:06.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.569+0000 7f18dd97c640 1 -- 172.21.15.44:0/936543200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f18d815e640 msgr2=0x7f18d806c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.569+0000 7f18dd97c640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f18d815e640 0x7f18d806c360 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f18d000d800 tx=0x7f18d000dcc0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.569+0000 7f18dd97c640 1 -- 172.21.15.44:0/936543200 shutdown_connections 2024-04-02T13:21:06.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.569+0000 7f18dd97c640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f18b806c8f0 0x7f18b806edb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.569+0000 7f18dd97c640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f18d815e640 0x7f18d806c360 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.569+0000 7f18dd97c640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18d815e260 0x7f18d80cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.569+0000 7f18dd97c640 1 --2- 172.21.15.44:0/936543200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18d80cec60 0x7f18d80c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.569+0000 7f18dd97c640 1 -- 172.21.15.44:0/936543200 >> 172.21.15.44:0/936543200 conn(0x7f18d809d3c0 msgr2=0x7f18d80b2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:06.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.569+0000 7f18dd97c640 1 -- 172.21.15.44:0/936543200 shutdown_connections 2024-04-02T13:21:06.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.569+0000 7f18dd97c640 1 -- 172.21.15.44:0/936543200 wait complete. 2024-04-02T13:21:06.582 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.685+0000 7fc6eb9a8640 1 Processor -- start 2024-04-02T13:21:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.686+0000 7fc6eb9a8640 1 -- start start 2024-04-02T13:21:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.686+0000 7fc6eb9a8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc6e415e5d0 0x7fc6e40d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.686+0000 7fc6eb9a8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6e40b4180 0x7fc6e40b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.686+0000 7fc6eb9a8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6e40d1630 con 0x7fc6e406cff0 2024-04-02T13:21:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.686+0000 7fc6eb9a8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc6e40d1bf0 con 0x7fc6e415e5d0 2024-04-02T13:21:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.686+0000 7fc6eb9a8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e40d1910 con 0x7fc6e40b4180 2024-04-02T13:21:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.687+0000 7fc6e8f43640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc6e415e5d0 0x7fc6e40d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.687+0000 7fc6e8f43640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc6e415e5d0 0x7fc6e40d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45002/0 (socket says 172.21.15.44:45002) 2024-04-02T13:21:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.687+0000 7fc6e8f43640 1 -- 172.21.15.44:0/3920343986 learned_addr learned my addr 172.21.15.44:0/3920343986 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:06.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.687+0000 7fc6e9f45640 1 --2- 172.21.15.44:0/3920343986 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6e40b4180 0x7fc6e40b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.687+0000 7fc6e8f43640 1 -- 172.21.15.44:0/3920343986 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6e40b4180 msgr2=0x7fc6e40b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:06.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.687+0000 7fc6e8f43640 1 --2- 172.21.15.44:0/3920343986 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6e40b4180 0x7fc6e40b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.687+0000 7fc6e8f43640 1 -- 172.21.15.44:0/3920343986 >> v1:172.21.15.69:6789/0 conn(0x7fc6e406cff0 legacy=0x7fc6e406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.687+0000 7fc6e8f43640 1 -- 172.21.15.44:0/3920343986 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e40b1540 con 0x7fc6e415e5d0 2024-04-02T13:21:06.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.688+0000 7fc6e8f43640 1 --2- 172.21.15.44:0/3920343986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6e415e5d0 0x7fc6e40d0d30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009870 tx=0x7fc6d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba53ae4eae99693 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:06.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.688+0000 7fc6dbfff640 1 -- 172.21.15.44:0/3920343986 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4035020 con 0x7fc6e415e5d0 2024-04-02T13:21:06.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.688+0000 7fc6dbfff640 1 -- 172.21.15.44:0/3920343986 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d4031640 con 0x7fc6e415e5d0 2024-04-02T13:21:06.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.688+0000 7fc6dbfff640 1 -- 172.21.15.44:0/3920343986 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d402d040 con 0x7fc6e415e5d0 2024-04-02T13:21:06.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.688+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/3920343986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6e415e5d0 msgr2=0x7fc6e40d0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.688+0000 7fc6eb9a8640 1 --2- 172.21.15.44:0/3920343986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6e415e5d0 0x7fc6e40d0d30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009870 tx=0x7fc6d402b8f0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.688+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/3920343986 shutdown_connections 2024-04-02T13:21:06.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.688+0000 7fc6eb9a8640 1 --2- 172.21.15.44:0/3920343986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6e415e5d0 0x7fc6e40d0d30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.688+0000 7fc6eb9a8640 1 --2- 172.21.15.44:0/3920343986 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6e40b4180 0x7fc6e40b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.688+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/3920343986 >> 172.21.15.44:0/3920343986 conn(0x7fc6e409aa20 msgr2=0x7fc6e409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:06.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.688+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/3920343986 shutdown_connections 2024-04-02T13:21:06.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.688+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/3920343986 wait complete. 2024-04-02T13:21:06.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.689+0000 7fc6eb9a8640 1 Processor -- start 2024-04-02T13:21:06.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.690+0000 7fc6eb9a8640 1 -- start start 2024-04-02T13:21:06.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.690+0000 7fc6eb9a8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6e406cff0 0x7fc6e415b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.690+0000 7fc6eb9a8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6e40b4180 0x7fc6e41f8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.690+0000 7fc6eb9a8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6e415e5d0 0x7fc6e40b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.690+0000 7fc6eb9a8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e40d13f0 con 0x7fc6e415e5d0 2024-04-02T13:21:06.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.690+0000 7fc6eb9a8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e40d1270 con 0x7fc6e406cff0 2024-04-02T13:21:06.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.690+0000 7fc6eb9a8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e40b2280 con 0x7fc6e40b4180 2024-04-02T13:21:06.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.690+0000 7fc6e9f45640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6e415e5d0 0x7fc6e40b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.690+0000 7fc6e9f45640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6e415e5d0 0x7fc6e40b4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45004/0 (socket says 172.21.15.44:45004) 2024-04-02T13:21:06.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.690+0000 7fc6e9f45640 1 -- 172.21.15.44:0/1314363619 learned_addr learned my addr 172.21.15.44:0/1314363619 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:06.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.690+0000 7fc6e8f43640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6e40b4180 0x7fc6e41f8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.690+0000 7fc6e9744640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6e406cff0 0x7fc6e415b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.691+0000 7fc6e9f45640 1 -- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6e40b4180 msgr2=0x7fc6e41f8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.691+0000 7fc6e9f45640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6e40b4180 0x7fc6e41f8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.691+0000 7fc6e9f45640 1 -- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6e406cff0 msgr2=0x7fc6e415b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.691+0000 7fc6e9f45640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6e406cff0 0x7fc6e415b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.691+0000 7fc6e9f45640 1 -- 172.21.15.44:0/1314363619 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e4161100 con 0x7fc6e415e5d0 2024-04-02T13:21:06.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.691+0000 7fc6e9f45640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6e415e5d0 0x7fc6e40b4b70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009e30 tx=0x7fc6dc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:06.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.691+0000 7fc6d9ffb640 1 -- 172.21.15.44:0/1314363619 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc018020 con 0x7fc6e415e5d0 2024-04-02T13:21:06.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.691+0000 7fc6d9ffb640 1 -- 172.21.15.44:0/1314363619 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6dc014480 con 0x7fc6e415e5d0 2024-04-02T13:21:06.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.691+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/1314363619 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6e41613e0 con 0x7fc6e415e5d0 2024-04-02T13:21:06.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.691+0000 7fc6d9ffb640 1 -- 172.21.15.44:0/1314363619 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc014870 con 0x7fc6e415e5d0 2024-04-02T13:21:06.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.691+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/1314363619 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6e4157010 con 0x7fc6e415e5d0 2024-04-02T13:21:06.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.692+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/1314363619 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6e40684f0 con 0x7fc6e415e5d0 2024-04-02T13:21:06.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.693+0000 7fc6d9ffb640 1 -- 172.21.15.44:0/1314363619 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc6dc005d10 con 0x7fc6e415e5d0 2024-04-02T13:21:06.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.693+0000 7fc6d9ffb640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc6c806c8f0 0x7fc6c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:06.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.693+0000 7fc6d9ffb640 1 -- 172.21.15.44:0/1314363619 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc6dc08cb70 con 0x7fc6e415e5d0 2024-04-02T13:21:06.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.693+0000 7fc6e9744640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc6c806c8f0 0x7fc6c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:06.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.694+0000 7fc6e9744640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc6c806c8f0 0x7fc6c806edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0001780 tx=0x7fc6e0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:06.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.697+0000 7fc6d9ffb640 1 -- 172.21.15.44:0/1314363619 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc6dc022030 con 0x7fc6e415e5d0 2024-04-02T13:21:06.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.883+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/1314363619 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc6e406cdd0 con 0x7fc6e415e5d0 2024-04-02T13:21:06.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.884+0000 7fc6d9ffb640 1 -- 172.21.15.44:0/1314363619 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fc6dc010040 con 0x7fc6e415e5d0 2024-04-02T13:21:06.885 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:06.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.886+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc6c806c8f0 msgr2=0x7fc6c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.886+0000 7fc6eb9a8640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc6c806c8f0 0x7fc6c806edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0001780 tx=0x7fc6e0009040 comp rx=0 tx=0).stop 2024-04-02T13:21:06.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.886+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/1314363619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6e415e5d0 msgr2=0x7fc6e40b4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:06.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.886+0000 7fc6eb9a8640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6e415e5d0 0x7fc6e40b4b70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009e30 tx=0x7fc6dc00c490 comp rx=0 tx=0).stop 2024-04-02T13:21:06.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.886+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/1314363619 shutdown_connections 2024-04-02T13:21:06.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.886+0000 7fc6eb9a8640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc6c806c8f0 0x7fc6c806edb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.886+0000 7fc6eb9a8640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6e415e5d0 0x7fc6e40b4b70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.886+0000 7fc6eb9a8640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6e40b4180 0x7fc6e41f8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.886+0000 7fc6eb9a8640 1 --2- 172.21.15.44:0/1314363619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6e406cff0 0x7fc6e415b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:06.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.886+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/1314363619 >> 172.21.15.44:0/1314363619 conn(0x7fc6e409aa20 msgr2=0x7fc6e406ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:06.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.886+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/1314363619 shutdown_connections 2024-04-02T13:21:06.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:06.886+0000 7fc6eb9a8640 1 -- 172.21.15.44:0/1314363619 wait complete. 2024-04-02T13:21:06.899 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:07.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.003+0000 7f2fba8c5640 1 Processor -- start 2024-04-02T13:21:07.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.003+0000 7f2fba8c5640 1 -- start start 2024-04-02T13:21:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.004+0000 7f2fba8c5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2fb415bfa0 0x7f2fb40b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.004+0000 7f2fba8c5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2fb40add60 0x7f2fb406b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.004+0000 7f2fba8c5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb40afcf0 con 0x7f2fb40ae1f0 2024-04-02T13:21:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.004+0000 7f2fba8c5640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2fb415c380 con 0x7f2fb40add60 2024-04-02T13:21:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.004+0000 7f2fba8c5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2fb40d0a10 con 0x7f2fb415bfa0 2024-04-02T13:21:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.004+0000 7f2fb37fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2fb40add60 0x7f2fb406b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.004+0000 7f2fb37fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2fb40add60 0x7f2fb406b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45016/0 (socket says 172.21.15.44:45016) 2024-04-02T13:21:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.004+0000 7f2fb37fe640 1 -- 172.21.15.44:0/938593517 learned_addr learned my addr 172.21.15.44:0/938593517 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:07.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.004+0000 7f2fb3fff640 1 --2- 172.21.15.44:0/938593517 >> v2:172.21.15.69:3300/0 conn(0x7f2fb415bfa0 0x7f2fb40b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.004+0000 7f2fb37fe640 1 -- 172.21.15.44:0/938593517 >> v2:172.21.15.69:3300/0 conn(0x7f2fb415bfa0 msgr2=0x7f2fb40b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:07.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.004+0000 7f2fb37fe640 1 --2- 172.21.15.44:0/938593517 >> v2:172.21.15.69:3300/0 conn(0x7f2fb415bfa0 0x7f2fb40b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.005+0000 7f2fb37fe640 1 -- 172.21.15.44:0/938593517 >> v1:172.21.15.69:6789/0 conn(0x7f2fb40ae1f0 legacy=0x7f2fb40d0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.005+0000 7f2fb37fe640 1 -- 172.21.15.44:0/938593517 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb40affd0 con 0x7f2fb40add60 2024-04-02T13:21:07.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.005+0000 7f2fb37fe640 1 --2- 172.21.15.44:0/938593517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb40add60 0x7f2fb406b4d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4072330 tx=0x7f2fa402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba848060380149e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:07.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.005+0000 7f2fb2ffd640 1 -- 172.21.15.44:0/938593517 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4035020 con 0x7f2fb40add60 2024-04-02T13:21:07.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.005+0000 7f2fb2ffd640 1 -- 172.21.15.44:0/938593517 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa4031870 con 0x7f2fb40add60 2024-04-02T13:21:07.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.005+0000 7f2fb2ffd640 1 -- 172.21.15.44:0/938593517 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4031c40 con 0x7f2fb40add60 2024-04-02T13:21:07.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.006+0000 7f2fba8c5640 1 -- 172.21.15.44:0/938593517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb40add60 msgr2=0x7f2fb406b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.006+0000 7f2fba8c5640 1 --2- 172.21.15.44:0/938593517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb40add60 0x7f2fb406b4d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4072330 tx=0x7f2fa402b910 comp rx=0 tx=0).stop 2024-04-02T13:21:07.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.006+0000 7f2fba8c5640 1 -- 172.21.15.44:0/938593517 shutdown_connections 2024-04-02T13:21:07.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.006+0000 7f2fba8c5640 1 --2- 172.21.15.44:0/938593517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb40add60 0x7f2fb406b4d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.006+0000 7f2fba8c5640 1 --2- 172.21.15.44:0/938593517 >> v2:172.21.15.69:3300/0 conn(0x7f2fb415bfa0 0x7f2fb40b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.006+0000 7f2fba8c5640 1 -- 172.21.15.44:0/938593517 >> 172.21.15.44:0/938593517 conn(0x7f2fb406c5e0 msgr2=0x7f2fb406c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:07.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.006+0000 7f2fba8c5640 1 -- 172.21.15.44:0/938593517 shutdown_connections 2024-04-02T13:21:07.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.006+0000 7f2fba8c5640 1 -- 172.21.15.44:0/938593517 wait complete. 2024-04-02T13:21:07.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.007+0000 7f2fba8c5640 1 Processor -- start 2024-04-02T13:21:07.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.007+0000 7f2fba8c5640 1 -- start start 2024-04-02T13:21:07.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fba8c5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb40add60 0x7f2fb4152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fba8c5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb40ae1f0 0x7f2fb415e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fba8c5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb415bfa0 0x7f2fb40b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fba8c5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb40b0590 con 0x7f2fb40add60 2024-04-02T13:21:07.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fba8c5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb40d0790 con 0x7f2fb40ae1f0 2024-04-02T13:21:07.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fba8c5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb40af8c0 con 0x7f2fb415bfa0 2024-04-02T13:21:07.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fb3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb40add60 0x7f2fb4152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fb3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb40add60 0x7f2fb4152eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45022/0 (socket says 172.21.15.44:45022) 2024-04-02T13:21:07.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fb3fff640 1 -- 172.21.15.44:0/525016566 learned_addr learned my addr 172.21.15.44:0/525016566 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:07.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fb37fe640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb40ae1f0 0x7f2fb415e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fb8e62640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb415bfa0 0x7f2fb40b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fb3fff640 1 -- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb415bfa0 msgr2=0x7f2fb40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fb3fff640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb415bfa0 0x7f2fb40b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fb3fff640 1 -- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb40ae1f0 msgr2=0x7f2fb415e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fb3fff640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb40ae1f0 0x7f2fb415e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.008+0000 7f2fb3fff640 1 -- 172.21.15.44:0/525016566 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb41533f0 con 0x7f2fb40add60 2024-04-02T13:21:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.009+0000 7f2fb3fff640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb40add60 0x7f2fb4152eb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800d5f0 tx=0x7f2fa800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.009+0000 7f2fb0ff9640 1 -- 172.21.15.44:0/525016566 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa80125f0 con 0x7f2fb40add60 2024-04-02T13:21:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.009+0000 7f2fb0ff9640 1 -- 172.21.15.44:0/525016566 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa8019070 con 0x7f2fb40add60 2024-04-02T13:21:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.009+0000 7f2fba8c5640 1 -- 172.21.15.44:0/525016566 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb415d860 con 0x7f2fb40add60 2024-04-02T13:21:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.009+0000 7f2fb0ff9640 1 -- 172.21.15.44:0/525016566 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa80129d0 con 0x7f2fb40add60 2024-04-02T13:21:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.009+0000 7f2fba8c5640 1 -- 172.21.15.44:0/525016566 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb415de20 con 0x7f2fb40add60 2024-04-02T13:21:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.010+0000 7f2fba8c5640 1 -- 172.21.15.44:0/525016566 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb4068c00 con 0x7f2fb40add60 2024-04-02T13:21:07.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.010+0000 7f2fb0ff9640 1 -- 172.21.15.44:0/525016566 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2fa8012b70 con 0x7f2fb40add60 2024-04-02T13:21:07.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.011+0000 7f2fb0ff9640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2fa006c8f0 0x7f2fa006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.011+0000 7f2fb0ff9640 1 -- 172.21.15.44:0/525016566 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2fa808c1d0 con 0x7f2fb40add60 2024-04-02T13:21:07.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.011+0000 7f2fb37fe640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2fa006c8f0 0x7f2fa006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.012+0000 7f2fb37fe640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2fa006c8f0 0x7f2fa006edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4036000 tx=0x7f2fa403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:07.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.015+0000 7f2fb0ff9640 1 -- 172.21.15.44:0/525016566 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2fa8059be0 con 0x7f2fb40add60 2024-04-02T13:21:07.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.205+0000 7f2fba8c5640 1 -- 172.21.15.44:0/525016566 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2fb4159bb0 con 0x7f2fb40add60 2024-04-02T13:21:07.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.205+0000 7f2fb0ff9640 1 -- 172.21.15.44:0/525016566 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f2fa805dee0 con 0x7f2fb40add60 2024-04-02T13:21:07.206 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:07.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.207+0000 7f2fba8c5640 1 -- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2fa006c8f0 msgr2=0x7f2fa006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.207+0000 7f2fba8c5640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2fa006c8f0 0x7f2fa006edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4036000 tx=0x7f2fa403c040 comp rx=0 tx=0).stop 2024-04-02T13:21:07.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.207+0000 7f2fba8c5640 1 -- 172.21.15.44:0/525016566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb40add60 msgr2=0x7f2fb4152eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.207+0000 7f2fba8c5640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb40add60 0x7f2fb4152eb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800d5f0 tx=0x7f2fa800dab0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.207+0000 7f2fba8c5640 1 -- 172.21.15.44:0/525016566 shutdown_connections 2024-04-02T13:21:07.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.207+0000 7f2fba8c5640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2fa006c8f0 0x7f2fa006edb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.207+0000 7f2fba8c5640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb415bfa0 0x7f2fb40b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.207+0000 7f2fba8c5640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb40ae1f0 0x7f2fb415e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.207+0000 7f2fba8c5640 1 --2- 172.21.15.44:0/525016566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb40add60 0x7f2fb4152eb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.207+0000 7f2fba8c5640 1 -- 172.21.15.44:0/525016566 >> 172.21.15.44:0/525016566 conn(0x7f2fb406c5e0 msgr2=0x7f2fb4067230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:07.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.207+0000 7f2fba8c5640 1 -- 172.21.15.44:0/525016566 shutdown_connections 2024-04-02T13:21:07.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.207+0000 7f2fba8c5640 1 -- 172.21.15.44:0/525016566 wait complete. 2024-04-02T13:21:07.219 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:07.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.339+0000 7f0b5dc0d640 1 Processor -- start 2024-04-02T13:21:07.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.339+0000 7f0b5dc0d640 1 -- start start 2024-04-02T13:21:07.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.341+0000 7f0b5dc0d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b5815c1a0 0x7f0b5806f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.341+0000 7f0b5dc0d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0b580d21c0 0x7f0b580cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.341+0000 7f0b5dc0d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b5815c580 con 0x7f0b580cfd60 2024-04-02T13:21:07.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.341+0000 7f0b5dc0d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0b58070ad0 con 0x7f0b580d21c0 2024-04-02T13:21:07.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.341+0000 7f0b5dc0d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0b580d01c0 con 0x7f0b5815c1a0 2024-04-02T13:21:07.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.341+0000 7f0b57fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0b580d21c0 0x7f0b580cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.341+0000 7f0b57fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0b580d21c0 0x7f0b580cc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45028/0 (socket says 172.21.15.44:45028) 2024-04-02T13:21:07.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.341+0000 7f0b57fff640 1 -- 172.21.15.44:0/1279781447 learned_addr learned my addr 172.21.15.44:0/1279781447 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:07.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.341+0000 7f0b577fe640 1 --2- 172.21.15.44:0/1279781447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b5815c1a0 0x7f0b5806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.341+0000 7f0b57fff640 1 -- 172.21.15.44:0/1279781447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b5815c1a0 msgr2=0x7f0b5806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:07.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.341+0000 7f0b57fff640 1 --2- 172.21.15.44:0/1279781447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b5815c1a0 0x7f0b5806f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.341+0000 7f0b57fff640 1 -- 172.21.15.44:0/1279781447 >> v1:172.21.15.69:6789/0 conn(0x7f0b580cfd60 legacy=0x7f0b580d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.341+0000 7f0b57fff640 1 -- 172.21.15.44:0/1279781447 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b5806fa30 con 0x7f0b580d21c0 2024-04-02T13:21:07.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.342+0000 7f0b57fff640 1 --2- 172.21.15.44:0/1279781447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b580d21c0 0x7f0b580cc9a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009e50 tx=0x7f0b5002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54097a372448d5b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:07.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.342+0000 7f0b567fc640 1 -- 172.21.15.44:0/1279781447 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5002cb00 con 0x7f0b580d21c0 2024-04-02T13:21:07.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.342+0000 7f0b567fc640 1 -- 172.21.15.44:0/1279781447 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5002cca0 con 0x7f0b580d21c0 2024-04-02T13:21:07.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.342+0000 7f0b567fc640 1 -- 172.21.15.44:0/1279781447 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b50034610 con 0x7f0b580d21c0 2024-04-02T13:21:07.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.343+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/1279781447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b580d21c0 msgr2=0x7f0b580cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.343+0000 7f0b5dc0d640 1 --2- 172.21.15.44:0/1279781447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b580d21c0 0x7f0b580cc9a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009e50 tx=0x7f0b5002bf60 comp rx=0 tx=0).stop 2024-04-02T13:21:07.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.343+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/1279781447 shutdown_connections 2024-04-02T13:21:07.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.343+0000 7f0b5dc0d640 1 --2- 172.21.15.44:0/1279781447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b580d21c0 0x7f0b580cc9a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.343+0000 7f0b5dc0d640 1 --2- 172.21.15.44:0/1279781447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b5815c1a0 0x7f0b5806f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.343+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/1279781447 >> 172.21.15.44:0/1279781447 conn(0x7f0b5809aa20 msgr2=0x7f0b5809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:07.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.343+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/1279781447 shutdown_connections 2024-04-02T13:21:07.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.343+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/1279781447 wait complete. 2024-04-02T13:21:07.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.344+0000 7f0b5dc0d640 1 Processor -- start 2024-04-02T13:21:07.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.345+0000 7f0b5dc0d640 1 -- start start 2024-04-02T13:21:07.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b5dc0d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b580cfd60 0x7f0b58151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b5dc0d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d21c0 0x7f0b581f5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b5dc0d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b5815c1a0 0x7f0b581f5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b5dc0d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0b580702d0 con 0x7f0b580cfd60 2024-04-02T13:21:07.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b5dc0d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5806fff0 con 0x7f0b5815c1a0 2024-04-02T13:21:07.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b5dc0d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0b580705b0 con 0x7f0b580d21c0 2024-04-02T13:21:07.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b580cfd60 0x7f0b58151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b580cfd60 0x7f0b58151750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45038/0 (socket says 172.21.15.44:45038) 2024-04-02T13:21:07.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b577fe640 1 -- 172.21.15.44:0/93646651 learned_addr learned my addr 172.21.15.44:0/93646651 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:07.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b56ffd640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d21c0 0x7f0b581f5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b57fff640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b5815c1a0 0x7f0b581f5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b577fe640 1 -- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d21c0 msgr2=0x7f0b581f5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b577fe640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d21c0 0x7f0b581f5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b577fe640 1 -- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b5815c1a0 msgr2=0x7f0b581f5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b577fe640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b5815c1a0 0x7f0b581f5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b577fe640 1 -- 172.21.15.44:0/93646651 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b580c6e20 con 0x7f0b580cfd60 2024-04-02T13:21:07.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b56ffd640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d21c0 0x7f0b581f5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:07.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b577fe640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b580cfd60 0x7f0b58151750 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800d5f0 tx=0x7f0b4800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:07.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b4ffff640 1 -- 172.21.15.44:0/93646651 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b480125f0 con 0x7f0b580cfd60 2024-04-02T13:21:07.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/93646651 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b580c9da0 con 0x7f0b580cfd60 2024-04-02T13:21:07.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b4ffff640 1 -- 172.21.15.44:0/93646651 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b48019070 con 0x7f0b580cfd60 2024-04-02T13:21:07.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.346+0000 7f0b4ffff640 1 -- 172.21.15.44:0/93646651 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b480129d0 con 0x7f0b580cfd60 2024-04-02T13:21:07.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.347+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/93646651 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b580ca360 con 0x7f0b580cfd60 2024-04-02T13:21:07.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.347+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/93646651 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b58068c00 con 0x7f0b580cfd60 2024-04-02T13:21:07.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.348+0000 7f0b4ffff640 1 -- 172.21.15.44:0/93646651 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0b48012b70 con 0x7f0b580cfd60 2024-04-02T13:21:07.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.348+0000 7f0b4ffff640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b3806c8f0 0x7f0b3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.348+0000 7f0b4ffff640 1 -- 172.21.15.44:0/93646651 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6764+0+0 (secure 0 0 0) 0x7f0b4808c360 con 0x7f0b580cfd60 2024-04-02T13:21:07.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.348+0000 7f0b56ffd640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b3806c8f0 0x7f0b3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.349+0000 7f0b56ffd640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b3806c8f0 0x7f0b3806edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0b581524b0 tx=0x7f0b40009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:07.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.353+0000 7f0b4ffff640 1 -- 172.21.15.44:0/93646651 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0b4805a380 con 0x7f0b580cfd60 2024-04-02T13:21:07.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.536+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/93646651 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0b580d1e10 con 0x7f0b580cfd60 2024-04-02T13:21:07.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.536+0000 7f0b4ffff640 1 -- 172.21.15.44:0/93646651 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f0b48059c80 con 0x7f0b580cfd60 2024-04-02T13:21:07.538 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:07.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.538+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b3806c8f0 msgr2=0x7f0b3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.538+0000 7f0b5dc0d640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b3806c8f0 0x7f0b3806edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0b581524b0 tx=0x7f0b40009040 comp rx=0 tx=0).stop 2024-04-02T13:21:07.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.538+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/93646651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b580cfd60 msgr2=0x7f0b58151750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.538+0000 7f0b5dc0d640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b580cfd60 0x7f0b58151750 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800d5f0 tx=0x7f0b4800dab0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.538+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/93646651 shutdown_connections 2024-04-02T13:21:07.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.538+0000 7f0b5dc0d640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b3806c8f0 0x7f0b3806edb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.538+0000 7f0b5dc0d640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b5815c1a0 0x7f0b581f5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.538+0000 7f0b5dc0d640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d21c0 0x7f0b581f5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.538+0000 7f0b5dc0d640 1 --2- 172.21.15.44:0/93646651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b580cfd60 0x7f0b58151750 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.538+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/93646651 >> 172.21.15.44:0/93646651 conn(0x7f0b5809aa20 msgr2=0x7f0b580b24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:07.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.538+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/93646651 shutdown_connections 2024-04-02T13:21:07.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.538+0000 7f0b5dc0d640 1 -- 172.21.15.44:0/93646651 wait complete. 2024-04-02T13:21:07.551 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:07.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.651+0000 7fb3f5688640 1 Processor -- start 2024-04-02T13:21:07.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.651+0000 7fb3f5688640 1 -- start start 2024-04-02T13:21:07.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.652+0000 7fb3f5688640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb3f015e430 0x7fb3f00ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.652+0000 7fb3f5688640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3f006c7f0 0x7fb3f00ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.652+0000 7fb3f5688640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb3f00cf240 0x7fb3f00d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.652+0000 7fb3f5688640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fb3f00d1410 con 0x7fb3f00cf240 2024-04-02T13:21:07.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.652+0000 7fb3f5688640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb3f006cff0 con 0x7fb3f015e430 2024-04-02T13:21:07.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.652+0000 7fb3f5688640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb3f015e810 con 0x7fb3f006c7f0 2024-04-02T13:21:07.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.652+0000 7fb3ef7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb3f00cf240 0x7fb3f00d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.652+0000 7fb3ef7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb3f00cf240 0x7fb3f00d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45050/0 (socket says 172.21.15.44:45050) 2024-04-02T13:21:07.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.652+0000 7fb3ef7fe640 1 -- 172.21.15.44:0/3907377886 learned_addr learned my addr 172.21.15.44:0/3907377886 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:07.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.652+0000 7fb3ee7fc640 1 --2- 172.21.15.44:0/3907377886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3f006c7f0 0x7fb3f00ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.652+0000 7fb3eeffd640 1 --2- 172.21.15.44:0/3907377886 >> v2:172.21.15.69:3300/0 conn(0x7fb3f015e430 0x7fb3f00ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.653+0000 7fb3ef7fe640 1 -- 172.21.15.44:0/3907377886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3f006c7f0 msgr2=0x7fb3f00ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:07.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.653+0000 7fb3ef7fe640 1 --2- 172.21.15.44:0/3907377886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3f006c7f0 0x7fb3f00ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.653+0000 7fb3ef7fe640 1 -- 172.21.15.44:0/3907377886 >> v2:172.21.15.69:3300/0 conn(0x7fb3f015e430 msgr2=0x7fb3f00ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:07.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.653+0000 7fb3ef7fe640 1 --2- 172.21.15.44:0/3907377886 >> v2:172.21.15.69:3300/0 conn(0x7fb3f015e430 0x7fb3f00ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.653+0000 7fb3ef7fe640 1 -- 172.21.15.44:0/3907377886 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3f006d240 con 0x7fb3f00cf240 2024-04-02T13:21:07.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.653+0000 7fb3ef7fe640 1 --2- 172.21.15.44:0/3907377886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f00cf240 0x7fb3f00d1af0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00cf9f0 tx=0x7fb3e002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5df38d9eca15879 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:07.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.654+0000 7fb3edffb640 1 -- 172.21.15.44:0/3907377886 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e002c9e0 con 0x7fb3f00cf240 2024-04-02T13:21:07.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.654+0000 7fb3edffb640 1 -- 172.21.15.44:0/3907377886 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3e002cb80 con 0x7fb3f00cf240 2024-04-02T13:21:07.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.654+0000 7fb3edffb640 1 -- 172.21.15.44:0/3907377886 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e00343f0 con 0x7fb3f00cf240 2024-04-02T13:21:07.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.654+0000 7fb3f5688640 1 -- 172.21.15.44:0/3907377886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f00cf240 msgr2=0x7fb3f00d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.654+0000 7fb3f5688640 1 --2- 172.21.15.44:0/3907377886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f00cf240 0x7fb3f00d1af0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00cf9f0 tx=0x7fb3e002be40 comp rx=0 tx=0).stop 2024-04-02T13:21:07.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.654+0000 7fb3f5688640 1 -- 172.21.15.44:0/3907377886 shutdown_connections 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.654+0000 7fb3f5688640 1 --2- 172.21.15.44:0/3907377886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f00cf240 0x7fb3f00d1af0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.654+0000 7fb3f5688640 1 --2- 172.21.15.44:0/3907377886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3f006c7f0 0x7fb3f00ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.654+0000 7fb3f5688640 1 --2- 172.21.15.44:0/3907377886 >> v2:172.21.15.69:3300/0 conn(0x7fb3f015e430 0x7fb3f00ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.654+0000 7fb3f5688640 1 -- 172.21.15.44:0/3907377886 >> 172.21.15.44:0/3907377886 conn(0x7fb3f0073850 msgr2=0x7fb3f009aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.654+0000 7fb3f5688640 1 -- 172.21.15.44:0/3907377886 shutdown_connections 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.654+0000 7fb3f5688640 1 -- 172.21.15.44:0/3907377886 wait complete. 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.655+0000 7fb3f5688640 1 Processor -- start 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.656+0000 7fb3f5688640 1 -- start start 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.656+0000 7fb3f5688640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f006c7f0 0x7fb3f0153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.656+0000 7fb3f5688640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3f00cf240 0x7fb3f015ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.656+0000 7fb3f5688640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3f015e430 0x7fb3f00c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.656+0000 7fb3f5688640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f00d0ef0 con 0x7fb3f006c7f0 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.656+0000 7fb3f5688640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f00d0c10 con 0x7fb3f015e430 2024-04-02T13:21:07.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.656+0000 7fb3f5688640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb3f006d800 con 0x7fb3f00cf240 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.656+0000 7fb3eeffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f006c7f0 0x7fb3f0153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.656+0000 7fb3eeffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f006c7f0 0x7fb3f0153d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45066/0 (socket says 172.21.15.44:45066) 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.656+0000 7fb3eeffd640 1 -- 172.21.15.44:0/332211391 learned_addr learned my addr 172.21.15.44:0/332211391 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.656+0000 7fb3ee7fc640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3f00cf240 0x7fb3f015ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.657+0000 7fb3ef7fe640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3f015e430 0x7fb3f00c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.657+0000 7fb3eeffd640 1 -- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3f00cf240 msgr2=0x7fb3f015ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.657+0000 7fb3eeffd640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3f00cf240 0x7fb3f015ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.657+0000 7fb3eeffd640 1 -- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3f015e430 msgr2=0x7fb3f00c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.657+0000 7fb3eeffd640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3f015e430 0x7fb3f00c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.657+0000 7fb3eeffd640 1 -- 172.21.15.44:0/332211391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3f01542c0 con 0x7fb3f006c7f0 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.657+0000 7fb3ef7fe640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3f015e430 0x7fb3f00c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.657+0000 7fb3eeffd640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f006c7f0 0x7fb3f0153d80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800e870 tx=0x7fb3e800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.657+0000 7fb3e77fe640 1 -- 172.21.15.44:0/332211391 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e8017070 con 0x7fb3f006c7f0 2024-04-02T13:21:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.657+0000 7fb3e77fe640 1 -- 172.21.15.44:0/332211391 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3e800c940 con 0x7fb3f006c7f0 2024-04-02T13:21:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.657+0000 7fb3e77fe640 1 -- 172.21.15.44:0/332211391 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e800cd30 con 0x7fb3f006c7f0 2024-04-02T13:21:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.657+0000 7fb3f5688640 1 -- 172.21.15.44:0/332211391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3f015fb10 con 0x7fb3f006c7f0 2024-04-02T13:21:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.657+0000 7fb3f5688640 1 -- 172.21.15.44:0/332211391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3f015f550 con 0x7fb3f006c7f0 2024-04-02T13:21:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.658+0000 7fb3f5688640 1 -- 172.21.15.44:0/332211391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3f0068c00 con 0x7fb3f006c7f0 2024-04-02T13:21:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.659+0000 7fb3e77fe640 1 -- 172.21.15.44:0/332211391 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb3e801a070 con 0x7fb3f006c7f0 2024-04-02T13:21:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.659+0000 7fb3e77fe640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb3d006c8f0 0x7fb3d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.659+0000 7fb3e77fe640 1 -- 172.21.15.44:0/332211391 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6764+0+0 (secure 0 0 0) 0x7fb3e808bd60 con 0x7fb3f006c7f0 2024-04-02T13:21:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.659+0000 7fb3ee7fc640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb3d006c8f0 0x7fb3d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.660+0000 7fb3ee7fc640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb3d006c8f0 0x7fb3d006edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00c8a30 tx=0x7fb3d8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.663+0000 7fb3e77fe640 1 -- 172.21.15.44:0/332211391 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb3f015fb10 con 0x7fb3f006c7f0 2024-04-02T13:21:07.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.849+0000 7fb3f5688640 1 -- 172.21.15.44:0/332211391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb3f00b2cf0 con 0x7fb3f006c7f0 2024-04-02T13:21:07.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.850+0000 7fb3e77fe640 1 -- 172.21.15.44:0/332211391 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18899 (secure 0 0 0) 0x7fb3e80597b0 con 0x7fb3f006c7f0 2024-04-02T13:21:07.852 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:07.852 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:07.053205+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:07.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.852+0000 7fb3f5688640 1 -- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb3d006c8f0 msgr2=0x7fb3d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.852+0000 7fb3f5688640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb3d006c8f0 0x7fb3d006edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00c8a30 tx=0x7fb3d8009190 comp rx=0 tx=0).stop 2024-04-02T13:21:07.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.852+0000 7fb3f5688640 1 -- 172.21.15.44:0/332211391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f006c7f0 msgr2=0x7fb3f0153d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.852+0000 7fb3f5688640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f006c7f0 0x7fb3f0153d80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800e870 tx=0x7fb3e800ed30 comp rx=0 tx=0).stop 2024-04-02T13:21:07.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.852+0000 7fb3f5688640 1 -- 172.21.15.44:0/332211391 shutdown_connections 2024-04-02T13:21:07.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.852+0000 7fb3f5688640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb3d006c8f0 0x7fb3d006edb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.852+0000 7fb3f5688640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3f015e430 0x7fb3f00c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.852+0000 7fb3f5688640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3f00cf240 0x7fb3f015ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.852+0000 7fb3f5688640 1 --2- 172.21.15.44:0/332211391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f006c7f0 0x7fb3f0153d80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.852+0000 7fb3f5688640 1 -- 172.21.15.44:0/332211391 >> 172.21.15.44:0/332211391 conn(0x7fb3f0073850 msgr2=0x7fb3f00b3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:07.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.852+0000 7fb3f5688640 1 -- 172.21.15.44:0/332211391 shutdown_connections 2024-04-02T13:21:07.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.852+0000 7fb3f5688640 1 -- 172.21.15.44:0/332211391 wait complete. 2024-04-02T13:21:07.867 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:07.867 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:07.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.972+0000 7f61855ba640 1 Processor -- start 2024-04-02T13:21:07.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.972+0000 7f61855ba640 1 -- start start 2024-04-02T13:21:07.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f61855ba640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f618015e260 0x7f61800d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f61855ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f618015e640 0x7f61800d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f61855ba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61800cf120 con 0x7f61800cec60 2024-04-02T13:21:07.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f61855ba640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f61800b5a50 con 0x7f618015e260 2024-04-02T13:21:07.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f61855ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f61800d01c0 con 0x7f618015e640 2024-04-02T13:21:07.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f617effd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f618015e260 0x7f61800d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f617f7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f61800cec60 0x7f6180161f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54672/0 (socket says 172.21.15.44:54672) 2024-04-02T13:21:07.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f617f7fe640 1 -- 172.21.15.44:0/190604482 learned_addr learned my addr 172.21.15.44:0/190604482 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:07.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f617e7fc640 1 --2- 172.21.15.44:0/190604482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f618015e640 0x7f61800d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f617effd640 1 -- 172.21.15.44:0/190604482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f618015e640 msgr2=0x7f61800d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:07.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f617effd640 1 --2- 172.21.15.44:0/190604482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f618015e640 0x7f61800d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f617effd640 1 -- 172.21.15.44:0/190604482 >> v1:172.21.15.44:6789/0 conn(0x7f61800cec60 legacy=0x7f6180161f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f617effd640 1 -- 172.21.15.44:0/190604482 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61800cf400 con 0x7f618015e260 2024-04-02T13:21:07.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.973+0000 7f617dffb640 1 -- 172.21.15.44:0/190604482 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 57028438 0 0) 0x7f61800cf120 con 0x7f61800cec60 2024-04-02T13:21:07.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.974+0000 7f617effd640 1 --2- 172.21.15.44:0/190604482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f618015e260 0x7f61800d2960 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6170009d90 tx=0x7f617002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f0f59ffd57e6583 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:07.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.974+0000 7f617dffb640 1 -- 172.21.15.44:0/190604482 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170039070 con 0x7f618015e260 2024-04-02T13:21:07.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.974+0000 7f617dffb640 1 -- 172.21.15.44:0/190604482 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f617002cc50 con 0x7f618015e260 2024-04-02T13:21:07.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.974+0000 7f617dffb640 1 -- 172.21.15.44:0/190604482 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170034440 con 0x7f618015e260 2024-04-02T13:21:07.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.975+0000 7f61855ba640 1 -- 172.21.15.44:0/190604482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f618015e260 msgr2=0x7f61800d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.975+0000 7f61855ba640 1 --2- 172.21.15.44:0/190604482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f618015e260 0x7f61800d2960 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6170009d90 tx=0x7f617002bea0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.975+0000 7f61855ba640 1 -- 172.21.15.44:0/190604482 shutdown_connections 2024-04-02T13:21:07.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.975+0000 7f61855ba640 1 --2- 172.21.15.44:0/190604482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f618015e640 0x7f61800d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.975+0000 7f61855ba640 1 --2- 172.21.15.44:0/190604482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f618015e260 0x7f61800d2960 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.975+0000 7f61855ba640 1 -- 172.21.15.44:0/190604482 >> 172.21.15.44:0/190604482 conn(0x7f618009d3c0 msgr2=0x7f61800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:07.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.975+0000 7f61855ba640 1 -- 172.21.15.44:0/190604482 shutdown_connections 2024-04-02T13:21:07.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.975+0000 7f61855ba640 1 -- 172.21.15.44:0/190604482 wait complete. 2024-04-02T13:21:07.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.976+0000 7f61855ba640 1 Processor -- start 2024-04-02T13:21:07.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.976+0000 7f61855ba640 1 -- start start 2024-04-02T13:21:07.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f61855ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61800cec60 0x7f61800cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f61855ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f618015e260 0x7f61800b8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f61855ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f618015e640 0x7f618015c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f61855ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f61800cfca0 con 0x7f618015e640 2024-04-02T13:21:07.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f61855ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f61800cf9c0 con 0x7f61800cec60 2024-04-02T13:21:07.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f61855ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f61800d2ea0 con 0x7f618015e260 2024-04-02T13:21:07.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f617f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f618015e640 0x7f618015c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f617f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f618015e640 0x7f618015c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45088/0 (socket says 172.21.15.44:45088) 2024-04-02T13:21:07.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f617f7fe640 1 -- 172.21.15.44:0/2599653997 learned_addr learned my addr 172.21.15.44:0/2599653997 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:07.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f617e7fc640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f618015e260 0x7f61800b8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f617effd640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61800cec60 0x7f61800cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f617f7fe640 1 -- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f618015e260 msgr2=0x7f61800b8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f617f7fe640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f618015e260 0x7f61800b8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f617f7fe640 1 -- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61800cec60 msgr2=0x7f61800cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:07.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f617f7fe640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61800cec60 0x7f61800cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:07.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f617f7fe640 1 -- 172.21.15.44:0/2599653997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61800b6f50 con 0x7f618015e640 2024-04-02T13:21:07.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f617f7fe640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f618015e640 0x7f618015c440 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f617800c410 tx=0x7f617800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:07.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.977+0000 7f61777fe640 1 -- 172.21.15.44:0/2599653997 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61780074e0 con 0x7f618015e640 2024-04-02T13:21:07.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.978+0000 7f61777fe640 1 -- 172.21.15.44:0/2599653997 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61780047e0 con 0x7f618015e640 2024-04-02T13:21:07.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.978+0000 7f61855ba640 1 -- 172.21.15.44:0/2599653997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61800b7230 con 0x7f618015e640 2024-04-02T13:21:07.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.978+0000 7f61777fe640 1 -- 172.21.15.44:0/2599653997 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178004bd0 con 0x7f618015e640 2024-04-02T13:21:07.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.978+0000 7f61855ba640 1 -- 172.21.15.44:0/2599653997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61800b77f0 con 0x7f618015e640 2024-04-02T13:21:07.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.979+0000 7f61855ba640 1 -- 172.21.15.44:0/2599653997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61800684f0 con 0x7f618015e640 2024-04-02T13:21:07.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.979+0000 7f61777fe640 1 -- 172.21.15.44:0/2599653997 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6178011070 con 0x7f618015e640 2024-04-02T13:21:07.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.979+0000 7f61777fe640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f616006c8f0 0x7f616006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:07.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.979+0000 7f61777fe640 1 -- 172.21.15.44:0/2599653997 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6764+0+0 (secure 0 0 0) 0x7f617808bab0 con 0x7f618015e640 2024-04-02T13:21:07.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.980+0000 7f617effd640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f616006c8f0 0x7f616006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:07.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.980+0000 7f617effd640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f616006c8f0 0x7f616006edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6170036000 tx=0x7f6170030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:07.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:07.984+0000 7f61777fe640 1 -- 172.21.15.44:0/2599653997 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6178059b50 con 0x7f618015e640 2024-04-02T13:21:08.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.172+0000 7f61855ba640 1 -- 172.21.15.44:0/2599653997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f61800d47f0 con 0x7f618015e640 2024-04-02T13:21:08.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.173+0000 7f61777fe640 1 -- 172.21.15.44:0/2599653997 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f6178059450 con 0x7f618015e640 2024-04-02T13:21:08.174 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:08.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.174+0000 7f61855ba640 1 -- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f616006c8f0 msgr2=0x7f616006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.174+0000 7f61855ba640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f616006c8f0 0x7f616006edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6170036000 tx=0x7f6170030000 comp rx=0 tx=0).stop 2024-04-02T13:21:08.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.174+0000 7f61855ba640 1 -- 172.21.15.44:0/2599653997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f618015e640 msgr2=0x7f618015c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.174+0000 7f61855ba640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f618015e640 0x7f618015c440 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f617800c410 tx=0x7f617800c8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.174+0000 7f61855ba640 1 -- 172.21.15.44:0/2599653997 shutdown_connections 2024-04-02T13:21:08.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.174+0000 7f61855ba640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f616006c8f0 0x7f616006edb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.174+0000 7f61855ba640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f618015e640 0x7f618015c440 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.174+0000 7f61855ba640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f618015e260 0x7f61800b8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.174+0000 7f61855ba640 1 --2- 172.21.15.44:0/2599653997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61800cec60 0x7f61800cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.174+0000 7f61855ba640 1 -- 172.21.15.44:0/2599653997 >> 172.21.15.44:0/2599653997 conn(0x7f618009d3c0 msgr2=0x7f618006c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:08.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.175+0000 7f61855ba640 1 -- 172.21.15.44:0/2599653997 shutdown_connections 2024-04-02T13:21:08.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.175+0000 7f61855ba640 1 -- 172.21.15.44:0/2599653997 wait complete. 2024-04-02T13:21:08.187 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:08.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.290+0000 7fad6ae55640 1 Processor -- start 2024-04-02T13:21:08.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.290+0000 7fad6ae55640 1 -- start start 2024-04-02T13:21:08.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.291+0000 7fad6ae55640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fad640cec60 0x7fad64169290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.291+0000 7fad6ae55640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad640cf120 con 0x7fad6415e260 2024-04-02T13:21:08.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.291+0000 7fad6ae55640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad640cf400 con 0x7fad6415e640 2024-04-02T13:21:08.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.291+0000 7fad6ae55640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fad640b5a50 con 0x7fad640cec60 2024-04-02T13:21:08.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.291+0000 7fad68bf1640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fad6415e260 0x7fad640d2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54682/0 (socket says 172.21.15.44:54682) 2024-04-02T13:21:08.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.291+0000 7fad68bf1640 1 -- 172.21.15.44:0/1974487397 learned_addr learned my addr 172.21.15.44:0/1974487397 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.291+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1276924780 0 0) 0x7fad640cf120 con 0x7fad6415e260 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.291+0000 7fad693f2640 1 --2- 172.21.15.44:0/1974487397 >> v2:172.21.15.69:3300/0 conn(0x7fad640cec60 0x7fad64169290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.291+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad4c003520 con 0x7fad6415e260 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3500247490 0 0) 0x7fad4c003520 con 0x7fad6415e260 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 >> v2:172.21.15.69:3300/0 conn(0x7fad640cec60 msgr2=0x7fad64169290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad637fe640 1 --2- 172.21.15.44:0/1974487397 >> v2:172.21.15.69:3300/0 conn(0x7fad640cec60 0x7fad64169290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 >> v1:172.21.15.69:6789/0 conn(0x7fad6415e640 legacy=0x7fad640d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad640cf6e0 con 0x7fad6415e260 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fad54001670 con 0x7fad6415e260 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 >> v1:172.21.15.44:6789/0 conn(0x7fad6415e260 legacy=0x7fad640d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad637fe640 1 --2- 172.21.15.44:0/1974487397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0039b0 0x7fad4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad637fe640 1 --2- 172.21.15.44:0/1974487397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c004800 0x7fad4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad637fe640 1 --2- 172.21.15.44:0/1974487397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad4c0051c0 0x7fad4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fad640b5a50 con 0x7fad4c0051c0 2024-04-02T13:21:08.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fad4c00c160 con 0x7fad4c0039b0 2024-04-02T13:21:08.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fad4c00c360 con 0x7fad4c004800 2024-04-02T13:21:08.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad693f2640 1 --2- 172.21.15.44:0/1974487397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad4c0051c0 0x7fad4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad68bf1640 1 --2- 172.21.15.44:0/1974487397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0039b0 0x7fad4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad63fff640 1 --2- 172.21.15.44:0/1974487397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c004800 0x7fad4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad693f2640 1 -- 172.21.15.44:0/1974487397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c004800 msgr2=0x7fad4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:08.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad693f2640 1 --2- 172.21.15.44:0/1974487397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c004800 0x7fad4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad693f2640 1 -- 172.21.15.44:0/1974487397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0039b0 msgr2=0x7fad4c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:08.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad693f2640 1 --2- 172.21.15.44:0/1974487397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0039b0 0x7fad4c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.292+0000 7fad693f2640 1 -- 172.21.15.44:0/1974487397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4c003520 con 0x7fad4c0051c0 2024-04-02T13:21:08.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.293+0000 7fad693f2640 1 -- 172.21.15.44:0/1974487397 reap_dead start 2024-04-02T13:21:08.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.293+0000 7fad693f2640 1 --2- 172.21.15.44:0/1974487397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad4c0051c0 0x7fad4c00b9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fad5c009fc0 tx=0x7fad4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69597f23402809c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:08.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.293+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5c009c10 con 0x7fad4c0051c0 2024-04-02T13:21:08.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.293+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad5c02cca0 con 0x7fad4c0051c0 2024-04-02T13:21:08.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.293+0000 7fad637fe640 1 -- 172.21.15.44:0/1974487397 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5c003bd0 con 0x7fad4c0051c0 2024-04-02T13:21:08.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.294+0000 7fad6ae55640 1 -- 172.21.15.44:0/1974487397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad4c0051c0 msgr2=0x7fad4c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.294+0000 7fad6ae55640 1 --2- 172.21.15.44:0/1974487397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad4c0051c0 0x7fad4c00b9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fad5c009fc0 tx=0x7fad4c005830 comp rx=0 tx=0).stop 2024-04-02T13:21:08.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.294+0000 7fad6ae55640 1 -- 172.21.15.44:0/1974487397 shutdown_connections 2024-04-02T13:21:08.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.294+0000 7fad6ae55640 1 --2- 172.21.15.44:0/1974487397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad4c0051c0 0x7fad4c00b9c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.294+0000 7fad6ae55640 1 -- 172.21.15.44:0/1974487397 >> 172.21.15.44:0/1974487397 conn(0x7fad6409d3c0 msgr2=0x7fad640d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:08.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.294+0000 7fad6ae55640 1 -- 172.21.15.44:0/1974487397 shutdown_connections 2024-04-02T13:21:08.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.294+0000 7fad6ae55640 1 -- 172.21.15.44:0/1974487397 wait complete. 2024-04-02T13:21:08.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.295+0000 7fad6ae55640 1 Processor -- start 2024-04-02T13:21:08.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.295+0000 7fad6ae55640 1 -- start start 2024-04-02T13:21:08.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad6ae55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0051c0 0x7fad640b9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad6ae55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad6415ea50 0x7fad641552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad6ae55640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad64160160 0x7fad640b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad6ae55640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fad640cfca0 con 0x7fad64160160 2024-04-02T13:21:08.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad6ae55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fad640d2960 con 0x7fad4c0051c0 2024-04-02T13:21:08.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad6ae55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fad640d2ae0 con 0x7fad6415ea50 2024-04-02T13:21:08.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad693f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad64160160 0x7fad640b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad693f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad64160160 0x7fad640b4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45106/0 (socket says 172.21.15.44:45106) 2024-04-02T13:21:08.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad693f2640 1 -- 172.21.15.44:0/1006222669 learned_addr learned my addr 172.21.15.44:0/1006222669 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:08.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad68bf1640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0051c0 0x7fad640b9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad63fff640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad6415ea50 0x7fad641552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad693f2640 1 -- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad6415ea50 msgr2=0x7fad641552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad693f2640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad6415ea50 0x7fad641552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad693f2640 1 -- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0051c0 msgr2=0x7fad640b9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad693f2640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0051c0 0x7fad640b9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad693f2640 1 -- 172.21.15.44:0/1006222669 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad640b8cb0 con 0x7fad64160160 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad693f2640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad64160160 0x7fad640b4df0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fad5c02c6d0 tx=0x7fad5c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad617fa640 1 -- 172.21.15.44:0/1006222669 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5c003990 con 0x7fad64160160 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad617fa640 1 -- 172.21.15.44:0/1006222669 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad5c044070 con 0x7fad64160160 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad6ae55640 1 -- 172.21.15.44:0/1006222669 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad6415f0c0 con 0x7fad64160160 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad617fa640 1 -- 172.21.15.44:0/1006222669 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5c003d90 con 0x7fad64160160 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.296+0000 7fad6ae55640 1 -- 172.21.15.44:0/1006222669 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad6415f680 con 0x7fad64160160 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.297+0000 7fad6ae55640 1 -- 172.21.15.44:0/1006222669 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad640684f0 con 0x7fad64160160 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.298+0000 7fad617fa640 1 -- 172.21.15.44:0/1006222669 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fad5c005d10 con 0x7fad64160160 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.298+0000 7fad617fa640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fad5006c8f0 0x7fad5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.298+0000 7fad617fa640 1 -- 172.21.15.44:0/1006222669 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fad5c0ae6b0 con 0x7fad64160160 2024-04-02T13:21:08.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.299+0000 7fad68bf1640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fad5006c8f0 0x7fad5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.299+0000 7fad68bf1640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fad5006c8f0 0x7fad5006edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fad54001eb0 tx=0x7fad54009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:08.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.303+0000 7fad617fa640 1 -- 172.21.15.44:0/1006222669 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fad5c07c6c0 con 0x7fad64160160 2024-04-02T13:21:08.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.488+0000 7fad6ae55640 1 -- 172.21.15.44:0/1006222669 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fad640d47f0 con 0x7fad64160160 2024-04-02T13:21:08.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.489+0000 7fad617fa640 1 -- 172.21.15.44:0/1006222669 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fad5c07bfc0 con 0x7fad64160160 2024-04-02T13:21:08.490 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:08.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.490+0000 7fad6ae55640 1 -- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fad5006c8f0 msgr2=0x7fad5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.490+0000 7fad6ae55640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fad5006c8f0 0x7fad5006edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fad54001eb0 tx=0x7fad54009040 comp rx=0 tx=0).stop 2024-04-02T13:21:08.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.490+0000 7fad6ae55640 1 -- 172.21.15.44:0/1006222669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad64160160 msgr2=0x7fad640b4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.490+0000 7fad6ae55640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad64160160 0x7fad640b4df0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fad5c02c6d0 tx=0x7fad5c009750 comp rx=0 tx=0).stop 2024-04-02T13:21:08.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.491+0000 7fad6ae55640 1 -- 172.21.15.44:0/1006222669 shutdown_connections 2024-04-02T13:21:08.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.491+0000 7fad6ae55640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fad5006c8f0 0x7fad5006edb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.491+0000 7fad6ae55640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad64160160 0x7fad640b4df0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.491+0000 7fad6ae55640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad6415ea50 0x7fad641552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.491+0000 7fad6ae55640 1 --2- 172.21.15.44:0/1006222669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0051c0 0x7fad640b9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.491+0000 7fad6ae55640 1 -- 172.21.15.44:0/1006222669 >> 172.21.15.44:0/1006222669 conn(0x7fad6409d3c0 msgr2=0x7fad6406d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:08.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.491+0000 7fad6ae55640 1 -- 172.21.15.44:0/1006222669 shutdown_connections 2024-04-02T13:21:08.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.491+0000 7fad6ae55640 1 -- 172.21.15.44:0/1006222669 wait complete. 2024-04-02T13:21:08.503 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:08.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.607+0000 7f975e455640 1 Processor -- start 2024-04-02T13:21:08.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.608+0000 7f975e455640 1 -- start start 2024-04-02T13:21:08.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.608+0000 7f975e455640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f97580ce9a0 0x7f97580d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.608+0000 7f975e455640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97580d0670 0x7f9758164c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.608+0000 7f975e455640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97580cdc20 con 0x7f97581554c0 2024-04-02T13:21:08.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.608+0000 7f975e455640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f97580d0ae0 con 0x7f97580ce9a0 2024-04-02T13:21:08.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.608+0000 7f975e455640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f97580cdec0 con 0x7f97580d0670 2024-04-02T13:21:08.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.608+0000 7f97577fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f97580ce9a0 0x7f97580d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.608+0000 7f9757fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f97581554c0 0x7f9758071120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54690/0 (socket says 172.21.15.44:54690) 2024-04-02T13:21:08.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.608+0000 7f9757fff640 1 -- 172.21.15.44:0/3224459108 learned_addr learned my addr 172.21.15.44:0/3224459108 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:08.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.609+0000 7f975c9f2640 1 --2- 172.21.15.44:0/3224459108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97580d0670 0x7f9758164c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.609+0000 7f97577fe640 1 -- 172.21.15.44:0/3224459108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97580d0670 msgr2=0x7f9758164c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:08.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.609+0000 7f97577fe640 1 --2- 172.21.15.44:0/3224459108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97580d0670 0x7f9758164c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.609+0000 7f97577fe640 1 -- 172.21.15.44:0/3224459108 >> v1:172.21.15.44:6789/0 conn(0x7f97581554c0 legacy=0x7f9758071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.609+0000 7f97577fe640 1 -- 172.21.15.44:0/3224459108 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97580ce1a0 con 0x7f97580ce9a0 2024-04-02T13:21:08.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.610+0000 7f97577fe640 1 --2- 172.21.15.44:0/3224459108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97580ce9a0 0x7f97580d0130 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f974c0097a0 tx=0x7f974c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac053d7be7fcf77c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:08.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.610+0000 7f9756ffd640 1 -- 172.21.15.44:0/3224459108 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c035020 con 0x7f97580ce9a0 2024-04-02T13:21:08.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.610+0000 7f9756ffd640 1 -- 172.21.15.44:0/3224459108 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f974c031830 con 0x7f97580ce9a0 2024-04-02T13:21:08.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.610+0000 7f9756ffd640 1 -- 172.21.15.44:0/3224459108 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c031c00 con 0x7f97580ce9a0 2024-04-02T13:21:08.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.610+0000 7f975e455640 1 -- 172.21.15.44:0/3224459108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97580ce9a0 msgr2=0x7f97580d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.610+0000 7f975e455640 1 --2- 172.21.15.44:0/3224459108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97580ce9a0 0x7f97580d0130 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f974c0097a0 tx=0x7f974c02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.610+0000 7f975e455640 1 -- 172.21.15.44:0/3224459108 shutdown_connections 2024-04-02T13:21:08.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.610+0000 7f975e455640 1 --2- 172.21.15.44:0/3224459108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97580d0670 0x7f9758164c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.610+0000 7f975e455640 1 --2- 172.21.15.44:0/3224459108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97580ce9a0 0x7f97580d0130 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.610+0000 7f975e455640 1 -- 172.21.15.44:0/3224459108 >> 172.21.15.44:0/3224459108 conn(0x7f975806b920 msgr2=0x7f975806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:08.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.610+0000 7f975e455640 1 -- 172.21.15.44:0/3224459108 shutdown_connections 2024-04-02T13:21:08.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.610+0000 7f975e455640 1 -- 172.21.15.44:0/3224459108 wait complete. 2024-04-02T13:21:08.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.611+0000 7f975e455640 1 Processor -- start 2024-04-02T13:21:08.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.612+0000 7f975e455640 1 -- start start 2024-04-02T13:21:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.612+0000 7f975e455640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97580ce9a0 0x7f975814f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.612+0000 7f975e455640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97580d0670 0x7f975809c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.612+0000 7f975e455640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97581554c0 0x7f975815d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.612+0000 7f975e455640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f975815c290 con 0x7f97581554c0 2024-04-02T13:21:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.612+0000 7f975e455640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f97581558a0 con 0x7f97580d0670 2024-04-02T13:21:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.612+0000 7f975e455640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f97581f7610 con 0x7f97580ce9a0 2024-04-02T13:21:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f975c9f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97581554c0 0x7f975815d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f975c9f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97581554c0 0x7f975815d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45134/0 (socket says 172.21.15.44:45134) 2024-04-02T13:21:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f975c9f2640 1 -- 172.21.15.44:0/392190987 learned_addr learned my addr 172.21.15.44:0/392190987 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f9757fff640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97580ce9a0 0x7f975814f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f97577fe640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97580d0670 0x7f975809c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f975c9f2640 1 -- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97580ce9a0 msgr2=0x7f975814f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f975c9f2640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97580ce9a0 0x7f975814f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f975c9f2640 1 -- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97580d0670 msgr2=0x7f975809c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f975c9f2640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97580d0670 0x7f975809c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f975c9f2640 1 -- 172.21.15.44:0/392190987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97581f78b0 con 0x7f97581554c0 2024-04-02T13:21:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f97577fe640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97580d0670 0x7f975809c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f975c9f2640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97581554c0 0x7f975815d640 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f975000cd70 tx=0x7f975000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f9754ff9640 1 -- 172.21.15.44:0/392190987 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9750003b90 con 0x7f97581554c0 2024-04-02T13:21:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f9754ff9640 1 -- 172.21.15.44:0/392190987 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9750003d30 con 0x7f97581554c0 2024-04-02T13:21:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f975e455640 1 -- 172.21.15.44:0/392190987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97581f7b90 con 0x7f97581554c0 2024-04-02T13:21:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f9754ff9640 1 -- 172.21.15.44:0/392190987 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9750004240 con 0x7f97581554c0 2024-04-02T13:21:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.613+0000 7f975e455640 1 -- 172.21.15.44:0/392190987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97580b2070 con 0x7f97581554c0 2024-04-02T13:21:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.614+0000 7f975e455640 1 -- 172.21.15.44:0/392190987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97580684f0 con 0x7f97581554c0 2024-04-02T13:21:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.615+0000 7f9754ff9640 1 -- 172.21.15.44:0/392190987 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f97500043e0 con 0x7f97581554c0 2024-04-02T13:21:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.615+0000 7f9754ff9640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f974406c8f0 0x7f974406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.615+0000 7f9754ff9640 1 -- 172.21.15.44:0/392190987 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f975008cac0 con 0x7f97581554c0 2024-04-02T13:21:08.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.615+0000 7f9757fff640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f974406c8f0 0x7f974406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.616+0000 7f9757fff640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f974406c8f0 0x7f974406edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9748000f60 tx=0x7f9748009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:08.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.620+0000 7f9754ff9640 1 -- 172.21.15.44:0/392190987 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f975005aad0 con 0x7f97581554c0 2024-04-02T13:21:08.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.803+0000 7f975e455640 1 -- 172.21.15.44:0/392190987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f97580b5170 con 0x7f97581554c0 2024-04-02T13:21:08.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.804+0000 7f9754ff9640 1 -- 172.21.15.44:0/392190987 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f975005a3d0 con 0x7f97581554c0 2024-04-02T13:21:08.805 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:08.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.805+0000 7f975e455640 1 -- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f974406c8f0 msgr2=0x7f974406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.805+0000 7f975e455640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f974406c8f0 0x7f974406edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9748000f60 tx=0x7f9748009040 comp rx=0 tx=0).stop 2024-04-02T13:21:08.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.805+0000 7f975e455640 1 -- 172.21.15.44:0/392190987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97581554c0 msgr2=0x7f975815d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.805+0000 7f975e455640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97581554c0 0x7f975815d640 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f975000cd70 tx=0x7f975000a660 comp rx=0 tx=0).stop 2024-04-02T13:21:08.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.805+0000 7f975e455640 1 -- 172.21.15.44:0/392190987 shutdown_connections 2024-04-02T13:21:08.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.805+0000 7f975e455640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f974406c8f0 0x7f974406edb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.805+0000 7f975e455640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97581554c0 0x7f975815d640 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.805+0000 7f975e455640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97580d0670 0x7f975809c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.805+0000 7f975e455640 1 --2- 172.21.15.44:0/392190987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97580ce9a0 0x7f975814f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.805+0000 7f975e455640 1 -- 172.21.15.44:0/392190987 >> 172.21.15.44:0/392190987 conn(0x7f975806b920 msgr2=0x7f975806bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:08.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.806+0000 7f975e455640 1 -- 172.21.15.44:0/392190987 shutdown_connections 2024-04-02T13:21:08.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.806+0000 7f975e455640 1 -- 172.21.15.44:0/392190987 wait complete. 2024-04-02T13:21:08.818 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:08.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.924+0000 7f3f9a574640 1 Processor -- start 2024-04-02T13:21:08.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.924+0000 7f3f9a574640 1 -- start start 2024-04-02T13:21:08.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.924+0000 7f3f9a574640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3f9415e470 0x7f3f940b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.924+0000 7f3f9a574640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3f940ce790 0x7f3f9406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.924+0000 7f3f9a574640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f940ced00 con 0x7f3f940b1540 2024-04-02T13:21:08.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.924+0000 7f3f9a574640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f3f940cefe0 con 0x7f3f940ce790 2024-04-02T13:21:08.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.924+0000 7f3f9a574640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3f940cf2c0 con 0x7f3f9415e470 2024-04-02T13:21:08.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.925+0000 7f3f98b11640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3f940ce790 0x7f3f9406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.925+0000 7f3f98b11640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3f940ce790 0x7f3f9406d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45144/0 (socket says 172.21.15.44:45144) 2024-04-02T13:21:08.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.925+0000 7f3f98b11640 1 -- 172.21.15.44:0/2163197397 learned_addr learned my addr 172.21.15.44:0/2163197397 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:08.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.925+0000 7f3f93fff640 1 --2- 172.21.15.44:0/2163197397 >> v2:172.21.15.69:3300/0 conn(0x7f3f9415e470 0x7f3f940b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.925+0000 7f3f98b11640 1 -- 172.21.15.44:0/2163197397 >> v2:172.21.15.69:3300/0 conn(0x7f3f9415e470 msgr2=0x7f3f940b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:08.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.925+0000 7f3f98b11640 1 --2- 172.21.15.44:0/2163197397 >> v2:172.21.15.69:3300/0 conn(0x7f3f9415e470 0x7f3f940b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.925+0000 7f3f98b11640 1 -- 172.21.15.44:0/2163197397 >> v1:172.21.15.69:6789/0 conn(0x7f3f940b1540 legacy=0x7f3f940d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.925+0000 7f3f98b11640 1 -- 172.21.15.44:0/2163197397 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f940cf5a0 con 0x7f3f940ce790 2024-04-02T13:21:08.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.926+0000 7f3f98b11640 1 --2- 172.21.15.44:0/2163197397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f940ce790 0x7f3f9406d0f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3f9406f1c0 tx=0x7f3f8c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bee034cde7db98f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:08.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.926+0000 7f3f92ffd640 1 -- 172.21.15.44:0/2163197397 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c02ca80 con 0x7f3f940ce790 2024-04-02T13:21:08.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.926+0000 7f3f92ffd640 1 -- 172.21.15.44:0/2163197397 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f8c02cc20 con 0x7f3f940ce790 2024-04-02T13:21:08.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.926+0000 7f3f92ffd640 1 -- 172.21.15.44:0/2163197397 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c0343f0 con 0x7f3f940ce790 2024-04-02T13:21:08.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.926+0000 7f3f9a574640 1 -- 172.21.15.44:0/2163197397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f940ce790 msgr2=0x7f3f9406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.926+0000 7f3f9a574640 1 --2- 172.21.15.44:0/2163197397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f940ce790 0x7f3f9406d0f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3f9406f1c0 tx=0x7f3f8c02bee0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.926+0000 7f3f9a574640 1 -- 172.21.15.44:0/2163197397 shutdown_connections 2024-04-02T13:21:08.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.926+0000 7f3f9a574640 1 --2- 172.21.15.44:0/2163197397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f940ce790 0x7f3f9406d0f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.926+0000 7f3f9a574640 1 --2- 172.21.15.44:0/2163197397 >> v2:172.21.15.69:3300/0 conn(0x7f3f9415e470 0x7f3f940b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.926+0000 7f3f9a574640 1 -- 172.21.15.44:0/2163197397 >> 172.21.15.44:0/2163197397 conn(0x7f3f9409aa20 msgr2=0x7f3f9409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:08.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.926+0000 7f3f9a574640 1 -- 172.21.15.44:0/2163197397 shutdown_connections 2024-04-02T13:21:08.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.926+0000 7f3f9a574640 1 -- 172.21.15.44:0/2163197397 wait complete. 2024-04-02T13:21:08.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.928+0000 7f3f9a574640 1 Processor -- start 2024-04-02T13:21:08.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.928+0000 7f3f9a574640 1 -- start start 2024-04-02T13:21:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.928+0000 7f3f9a574640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f940b1540 0x7f3f9415f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.928+0000 7f3f9a574640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f940ce790 0x7f3f941617b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.928+0000 7f3f9a574640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f9415e470 0x7f3f940c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.928+0000 7f3f9a574640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f940d22c0 con 0x7f3f940ce790 2024-04-02T13:21:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.928+0000 7f3f9a574640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9406cb60 con 0x7f3f940b1540 2024-04-02T13:21:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.928+0000 7f3f9a574640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3f940d2440 con 0x7f3f9415e470 2024-04-02T13:21:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f937fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f940ce790 0x7f3f941617b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f937fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f940ce790 0x7f3f941617b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45154/0 (socket says 172.21.15.44:45154) 2024-04-02T13:21:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f937fe640 1 -- 172.21.15.44:0/3335402236 learned_addr learned my addr 172.21.15.44:0/3335402236 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f98b11640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f9415e470 0x7f3f940c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f93fff640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f940b1540 0x7f3f9415f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f937fe640 1 -- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f9415e470 msgr2=0x7f3f940c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f937fe640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f9415e470 0x7f3f940c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f937fe640 1 -- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f940b1540 msgr2=0x7f3f9415f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f937fe640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f940b1540 0x7f3f9415f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f937fe640 1 -- 172.21.15.44:0/3335402236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9415fb70 con 0x7f3f940ce790 2024-04-02T13:21:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f937fe640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f940ce790 0x7f3f941617b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3f8800c400 tx=0x7f3f8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f90ff9640 1 -- 172.21.15.44:0/3335402236 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f880052d0 con 0x7f3f940ce790 2024-04-02T13:21:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f90ff9640 1 -- 172.21.15.44:0/3335402236 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f88005490 con 0x7f3f940ce790 2024-04-02T13:21:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f9a574640 1 -- 172.21.15.44:0/3335402236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f9415fe50 con 0x7f3f940ce790 2024-04-02T13:21:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f90ff9640 1 -- 172.21.15.44:0/3335402236 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88007260 con 0x7f3f940ce790 2024-04-02T13:21:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.929+0000 7f3f9a574640 1 -- 172.21.15.44:0/3335402236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f94160410 con 0x7f3f940ce790 2024-04-02T13:21:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.930+0000 7f3f9a574640 1 -- 172.21.15.44:0/3335402236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f94160610 con 0x7f3f940ce790 2024-04-02T13:21:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.931+0000 7f3f90ff9640 1 -- 172.21.15.44:0/3335402236 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3f88007400 con 0x7f3f940ce790 2024-04-02T13:21:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.931+0000 7f3f90ff9640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3f8006c920 0x7f3f8006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.931+0000 7f3f90ff9640 1 -- 172.21.15.44:0/3335402236 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f3f8808b340 con 0x7f3f940ce790 2024-04-02T13:21:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.931+0000 7f3f93fff640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3f8006c920 0x7f3f8006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:08.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.932+0000 7f3f93fff640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3f8006c920 0x7f3f8006ede0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3f840079a0 tx=0x7f3f84008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:08.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:08.935+0000 7f3f90ff9640 1 -- 172.21.15.44:0/3335402236 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3f88058d90 con 0x7f3f940ce790 2024-04-02T13:21:09.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.120+0000 7f3f9a574640 1 -- 172.21.15.44:0/3335402236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3f940684f0 con 0x7f3f940ce790 2024-04-02T13:21:09.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.120+0000 7f3f90ff9640 1 -- 172.21.15.44:0/3335402236 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f3f8805d090 con 0x7f3f940ce790 2024-04-02T13:21:09.122 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:09.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.122+0000 7f3f9a574640 1 -- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3f8006c920 msgr2=0x7f3f8006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.122+0000 7f3f9a574640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3f8006c920 0x7f3f8006ede0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3f840079a0 tx=0x7f3f84008040 comp rx=0 tx=0).stop 2024-04-02T13:21:09.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.122+0000 7f3f9a574640 1 -- 172.21.15.44:0/3335402236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f940ce790 msgr2=0x7f3f941617b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.122+0000 7f3f9a574640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f940ce790 0x7f3f941617b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3f8800c400 tx=0x7f3f8800c8c0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.122+0000 7f3f9a574640 1 -- 172.21.15.44:0/3335402236 shutdown_connections 2024-04-02T13:21:09.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.122+0000 7f3f9a574640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3f8006c920 0x7f3f8006ede0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.122+0000 7f3f9a574640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f9415e470 0x7f3f940c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.122+0000 7f3f9a574640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f940ce790 0x7f3f941617b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.122+0000 7f3f9a574640 1 --2- 172.21.15.44:0/3335402236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f940b1540 0x7f3f9415f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.122+0000 7f3f9a574640 1 -- 172.21.15.44:0/3335402236 >> 172.21.15.44:0/3335402236 conn(0x7f3f9409aa20 msgr2=0x7f3f94161f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.122+0000 7f3f9a574640 1 -- 172.21.15.44:0/3335402236 shutdown_connections 2024-04-02T13:21:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.122+0000 7f3f9a574640 1 -- 172.21.15.44:0/3335402236 wait complete. 2024-04-02T13:21:09.173 INFO:tasks.cephfs_test_runner:Starting test: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-02T13:21:09.174 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:09.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.274+0000 7f9178a1b640 1 Processor -- start 2024-04-02T13:21:09.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.274+0000 7f9178a1b640 1 -- start start 2024-04-02T13:21:09.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.275+0000 7f9178a1b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f917406e650 0x7f917415c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.275+0000 7f9178a1b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f91740cc6e0 0x7f91740ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.275+0000 7f9178a1b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91740726c0 con 0x7f9174159db0 2024-04-02T13:21:09.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.275+0000 7f9178a1b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f91740cd070 con 0x7f91740cc6e0 2024-04-02T13:21:09.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.275+0000 7f9178a1b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9174073240 con 0x7f917406e650 2024-04-02T13:21:09.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.275+0000 7f9171d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f917406e650 0x7f917415c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.275+0000 7f9172d9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f91740cc6e0 0x7f91740ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.275+0000 7f9171d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f917406e650 0x7f917415c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:45358/0 (socket says 172.21.15.44:45358) 2024-04-02T13:21:09.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.275+0000 7f9172d9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f91740cc6e0 0x7f91740ccb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:37310/0 (socket says 172.21.15.44:37310) 2024-04-02T13:21:09.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.275+0000 7f917259c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9174159db0 0x7f9174071e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:55332/0 (socket says 172.21.15.44:55332) 2024-04-02T13:21:09.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.275+0000 7f9171d9b640 1 -- 172.21.15.44:0/936146614 learned_addr learned my addr 172.21.15.44:0/936146614 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:09.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.276+0000 7f9171d9b640 1 -- 172.21.15.44:0/936146614 >> v2:172.21.15.69:3300/0 conn(0x7f91740cc6e0 msgr2=0x7f91740ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:09.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.276+0000 7f9171d9b640 1 --2- 172.21.15.44:0/936146614 >> v2:172.21.15.69:3300/0 conn(0x7f91740cc6e0 0x7f91740ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.276+0000 7f9171d9b640 1 -- 172.21.15.44:0/936146614 >> v1:172.21.15.69:6789/0 conn(0x7f9174159db0 legacy=0x7f9174071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.276+0000 7f9171d9b640 1 -- 172.21.15.44:0/936146614 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91740729a0 con 0x7f917406e650 2024-04-02T13:21:09.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.277+0000 7f9171d9b640 1 --2- 172.21.15.44:0/936146614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f917406e650 0x7f917415c6d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9168009870 tx=0x7f916802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22440befb9c7a82d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:09.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.277+0000 7f917159a640 1 -- 172.21.15.44:0/936146614 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9168035020 con 0x7f917406e650 2024-04-02T13:21:09.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.277+0000 7f917159a640 1 -- 172.21.15.44:0/936146614 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9168031640 con 0x7f917406e650 2024-04-02T13:21:09.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.277+0000 7f917159a640 1 -- 172.21.15.44:0/936146614 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916802d040 con 0x7f917406e650 2024-04-02T13:21:09.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.277+0000 7f9178a1b640 1 -- 172.21.15.44:0/936146614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f917406e650 msgr2=0x7f917415c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.277+0000 7f9178a1b640 1 --2- 172.21.15.44:0/936146614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f917406e650 0x7f917415c6d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9168009870 tx=0x7f916802b8f0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.277+0000 7f9178a1b640 1 -- 172.21.15.44:0/936146614 shutdown_connections 2024-04-02T13:21:09.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.277+0000 7f9178a1b640 1 --2- 172.21.15.44:0/936146614 >> v2:172.21.15.69:3300/0 conn(0x7f91740cc6e0 0x7f91740ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.277+0000 7f9178a1b640 1 --2- 172.21.15.44:0/936146614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f917406e650 0x7f917415c6d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.277+0000 7f9178a1b640 1 -- 172.21.15.44:0/936146614 >> 172.21.15.44:0/936146614 conn(0x7f91740cfcd0 msgr2=0x7f91740d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:09.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.277+0000 7f9178a1b640 1 -- 172.21.15.44:0/936146614 shutdown_connections 2024-04-02T13:21:09.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.277+0000 7f9178a1b640 1 -- 172.21.15.44:0/936146614 wait complete. 2024-04-02T13:21:09.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.279+0000 7f9178a1b640 1 Processor -- start 2024-04-02T13:21:09.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.279+0000 7f9178a1b640 1 -- start start 2024-04-02T13:21:09.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.279+0000 7f9178a1b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f917406e650 0x7f91740cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.279+0000 7f9178a1b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91740cc6e0 0x7f91740cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.279+0000 7f9178a1b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9174159db0 0x7f9174163ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.279+0000 7f9178a1b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9174165cf0 con 0x7f9174159db0 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.279+0000 7f9178a1b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9174072f60 con 0x7f91740cc6e0 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.279+0000 7f9178a1b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9174165e70 con 0x7f917406e650 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9172d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9174159db0 0x7f9174163ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9172d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9174159db0 0x7f9174163ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45164/0 (socket says 172.21.15.44:45164) 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9172d9d640 1 -- 172.21.15.44:0/2346794844 learned_addr learned my addr 172.21.15.44:0/2346794844 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9171d9b640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91740cc6e0 0x7f91740cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f917259c640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f917406e650 0x7f91740cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9172d9d640 1 -- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f917406e650 msgr2=0x7f91740cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9172d9d640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f917406e650 0x7f91740cecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9172d9d640 1 -- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91740cc6e0 msgr2=0x7f91740cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9172d9d640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91740cc6e0 0x7f91740cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9172d9d640 1 -- 172.21.15.44:0/2346794844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917415db10 con 0x7f9174159db0 2024-04-02T13:21:09.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9172d9d640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9174159db0 0x7f9174163ed0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f916400efc0 tx=0x7f916400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:09.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9162ffd640 1 -- 172.21.15.44:0/2346794844 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164018020 con 0x7f9174159db0 2024-04-02T13:21:09.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9162ffd640 1 -- 172.21.15.44:0/2346794844 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9164014480 con 0x7f9174159db0 2024-04-02T13:21:09.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9162ffd640 1 -- 172.21.15.44:0/2346794844 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164014870 con 0x7f9174159db0 2024-04-02T13:21:09.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9178a1b640 1 -- 172.21.15.44:0/2346794844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f917415ddf0 con 0x7f9174159db0 2024-04-02T13:21:09.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.280+0000 7f9178a1b640 1 -- 172.21.15.44:0/2346794844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f917415e3b0 con 0x7f9174159db0 2024-04-02T13:21:09.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.281+0000 7f9178a1b640 1 -- 172.21.15.44:0/2346794844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9174074f20 con 0x7f9174159db0 2024-04-02T13:21:09.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.282+0000 7f9162ffd640 1 -- 172.21.15.44:0/2346794844 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9164005d10 con 0x7f9174159db0 2024-04-02T13:21:09.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.282+0000 7f9162ffd640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f915406c8f0 0x7f915406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.282+0000 7f9162ffd640 1 -- 172.21.15.44:0/2346794844 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f916408cc30 con 0x7f9174159db0 2024-04-02T13:21:09.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.282+0000 7f917259c640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f915406c8f0 0x7f915406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.283+0000 7f917259c640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f915406c8f0 0x7f915406edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f916c007fd0 tx=0x7f916c007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:09.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.287+0000 7f9162ffd640 1 -- 172.21.15.44:0/2346794844 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9164022030 con 0x7f9174159db0 2024-04-02T13:21:09.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.471+0000 7f9178a1b640 1 -- 172.21.15.44:0/2346794844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f917415d4b0 con 0x7f9174159db0 2024-04-02T13:21:09.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.473+0000 7f9162ffd640 1 -- 172.21.15.44:0/2346794844 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18899 (secure 0 0 0) 0x7f9164010040 con 0x7f9174159db0 2024-04-02T13:21:09.474 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:09.474 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:08.061253+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:09.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.474+0000 7f9178a1b640 1 -- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f915406c8f0 msgr2=0x7f915406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.474+0000 7f9178a1b640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f915406c8f0 0x7f915406edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f916c007fd0 tx=0x7f916c007e80 comp rx=0 tx=0).stop 2024-04-02T13:21:09.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.474+0000 7f9178a1b640 1 -- 172.21.15.44:0/2346794844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9174159db0 msgr2=0x7f9174163ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.474+0000 7f9178a1b640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9174159db0 0x7f9174163ed0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f916400efc0 tx=0x7f916400c490 comp rx=0 tx=0).stop 2024-04-02T13:21:09.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.475+0000 7f9178a1b640 1 -- 172.21.15.44:0/2346794844 shutdown_connections 2024-04-02T13:21:09.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.475+0000 7f9178a1b640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f915406c8f0 0x7f915406edb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.475+0000 7f9178a1b640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9174159db0 0x7f9174163ed0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.475+0000 7f9178a1b640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91740cc6e0 0x7f91740cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.475+0000 7f9178a1b640 1 --2- 172.21.15.44:0/2346794844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f917406e650 0x7f91740cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.475+0000 7f9178a1b640 1 -- 172.21.15.44:0/2346794844 >> 172.21.15.44:0/2346794844 conn(0x7f91740cfcd0 msgr2=0x7f91740cdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:09.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.475+0000 7f9178a1b640 1 -- 172.21.15.44:0/2346794844 shutdown_connections 2024-04-02T13:21:09.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.475+0000 7f9178a1b640 1 -- 172.21.15.44:0/2346794844 wait complete. 2024-04-02T13:21:09.487 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:09.488 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:09.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.591+0000 7ff382cba640 1 Processor -- start 2024-04-02T13:21:09.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.591+0000 7ff382cba640 1 -- start start 2024-04-02T13:21:09.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.591+0000 7ff382cba640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff37c0721f0 0x7ff37c0cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.591+0000 7ff382cba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff37c06c220 con 0x7ff37c15e420 2024-04-02T13:21:09.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.591+0000 7ff382cba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff37c15f7e0 con 0x7ff37c0cfc40 2024-04-02T13:21:09.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.591+0000 7ff382cba640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff37c0d00b0 con 0x7ff37c0721f0 2024-04-02T13:21:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.592+0000 7ff37bfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff37c0721f0 0x7ff37c0cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.592+0000 7ff380a56640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff37c15e420 0x7ff37c0ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54698/0 (socket says 172.21.15.44:54698) 2024-04-02T13:21:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.592+0000 7ff380a56640 1 -- 172.21.15.44:0/1472856693 learned_addr learned my addr 172.21.15.44:0/1472856693 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.592+0000 7ff37bfff640 1 -- 172.21.15.44:0/1472856693 >> v1:172.21.15.69:6789/0 conn(0x7ff37c0cfc40 legacy=0x7ff37c0cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.592+0000 7ff37bfff640 1 -- 172.21.15.44:0/1472856693 >> v1:172.21.15.44:6789/0 conn(0x7ff37c15e420 legacy=0x7ff37c0ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.592+0000 7ff37bfff640 1 -- 172.21.15.44:0/1472856693 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37c15ec60 con 0x7ff37c0721f0 2024-04-02T13:21:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.592+0000 7ff37b7fe640 1 -- 172.21.15.44:0/1472856693 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 737129344 0 0) 0x7ff364001110 con 0x7ff37c15e420 2024-04-02T13:21:09.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.593+0000 7ff37bfff640 1 --2- 172.21.15.44:0/1472856693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37c0721f0 0x7ff37c0cf700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff36c0097a0 tx=0x7ff36c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d249e5189d43f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:09.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.593+0000 7ff37b7fe640 1 -- 172.21.15.44:0/1472856693 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c038070 con 0x7ff37c0721f0 2024-04-02T13:21:09.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.593+0000 7ff37b7fe640 1 -- 172.21.15.44:0/1472856693 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff36c031700 con 0x7ff37c0721f0 2024-04-02T13:21:09.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.593+0000 7ff37b7fe640 1 -- 172.21.15.44:0/1472856693 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c031ad0 con 0x7ff37c0721f0 2024-04-02T13:21:09.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.593+0000 7ff382cba640 1 -- 172.21.15.44:0/1472856693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37c0721f0 msgr2=0x7ff37c0cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.593+0000 7ff382cba640 1 --2- 172.21.15.44:0/1472856693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37c0721f0 0x7ff37c0cf700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff36c0097a0 tx=0x7ff36c02b7a0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.593+0000 7ff382cba640 1 -- 172.21.15.44:0/1472856693 shutdown_connections 2024-04-02T13:21:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.593+0000 7ff382cba640 1 --2- 172.21.15.44:0/1472856693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37c0721f0 0x7ff37c0cf700 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.593+0000 7ff382cba640 1 -- 172.21.15.44:0/1472856693 >> 172.21.15.44:0/1472856693 conn(0x7ff37c0b2a10 msgr2=0x7ff37c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.593+0000 7ff382cba640 1 -- 172.21.15.44:0/1472856693 shutdown_connections 2024-04-02T13:21:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.593+0000 7ff382cba640 1 -- 172.21.15.44:0/1472856693 wait complete. 2024-04-02T13:21:09.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.594+0000 7ff382cba640 1 Processor -- start 2024-04-02T13:21:09.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.595+0000 7ff382cba640 1 -- start start 2024-04-02T13:21:09.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.595+0000 7ff382cba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37c0721f0 0x7ff37c14a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.595+0000 7ff382cba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff37c0cfc40 0x7ff37c1570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.595+0000 7ff382cba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff37c15e420 0x7ff37c154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.595+0000 7ff382cba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c15f220 con 0x7ff37c0721f0 2024-04-02T13:21:09.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.595+0000 7ff382cba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c0cf3b0 con 0x7ff37c0cfc40 2024-04-02T13:21:09.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.595+0000 7ff382cba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff37c15f500 con 0x7ff37c15e420 2024-04-02T13:21:09.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.595+0000 7ff380a56640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37c0721f0 0x7ff37c14a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff380a56640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37c0721f0 0x7ff37c14a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45178/0 (socket says 172.21.15.44:45178) 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff380a56640 1 -- 172.21.15.44:0/3709340585 learned_addr learned my addr 172.21.15.44:0/3709340585 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff37bfff640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff37c0cfc40 0x7ff37c1570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff381257640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff37c15e420 0x7ff37c154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff380a56640 1 -- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff37c15e420 msgr2=0x7ff37c154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff380a56640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff37c15e420 0x7ff37c154050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff380a56640 1 -- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff37c0cfc40 msgr2=0x7ff37c1570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff380a56640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff37c0cfc40 0x7ff37c1570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff380a56640 1 -- 172.21.15.44:0/3709340585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37c156030 con 0x7ff37c0721f0 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff380a56640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37c0721f0 0x7ff37c14a850 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff36400c430 tx=0x7ff36400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff3797fa640 1 -- 172.21.15.44:0/3709340585 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3640074e0 con 0x7ff37c0721f0 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff3797fa640 1 -- 172.21.15.44:0/3709340585 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3640047e0 con 0x7ff37c0721f0 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff3797fa640 1 -- 172.21.15.44:0/3709340585 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff364004bd0 con 0x7ff37c0721f0 2024-04-02T13:21:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff382cba640 1 -- 172.21.15.44:0/3709340585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff37c156310 con 0x7ff37c0721f0 2024-04-02T13:21:09.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.596+0000 7ff382cba640 1 -- 172.21.15.44:0/3709340585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff37c1568d0 con 0x7ff37c0721f0 2024-04-02T13:21:09.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.597+0000 7ff382cba640 1 -- 172.21.15.44:0/3709340585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff37c0684f0 con 0x7ff37c0721f0 2024-04-02T13:21:09.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.598+0000 7ff3797fa640 1 -- 172.21.15.44:0/3709340585 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff364011070 con 0x7ff37c0721f0 2024-04-02T13:21:09.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.598+0000 7ff3797fa640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff35c06c8f0 0x7ff35c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.598+0000 7ff3797fa640 1 -- 172.21.15.44:0/3709340585 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7ff36408bb10 con 0x7ff37c0721f0 2024-04-02T13:21:09.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.598+0000 7ff37bfff640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff35c06c8f0 0x7ff35c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.599+0000 7ff37bfff640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff35c06c8f0 0x7ff35c06edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff36c035000 tx=0x7ff36c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:09.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.603+0000 7ff3797fa640 1 -- 172.21.15.44:0/3709340585 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff37c06c220 con 0x7ff37c0721f0 2024-04-02T13:21:09.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.785+0000 7ff382cba640 1 -- 172.21.15.44:0/3709340585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff37c0ceeb0 con 0x7ff37c0721f0 2024-04-02T13:21:09.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.786+0000 7ff3797fa640 1 -- 172.21.15.44:0/3709340585 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7ff364059440 con 0x7ff37c0721f0 2024-04-02T13:21:09.787 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:09.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.788+0000 7ff382cba640 1 -- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff35c06c8f0 msgr2=0x7ff35c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.788+0000 7ff382cba640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff35c06c8f0 0x7ff35c06edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff36c035000 tx=0x7ff36c008030 comp rx=0 tx=0).stop 2024-04-02T13:21:09.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.788+0000 7ff382cba640 1 -- 172.21.15.44:0/3709340585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37c0721f0 msgr2=0x7ff37c14a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.788+0000 7ff382cba640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37c0721f0 0x7ff37c14a850 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff36400c430 tx=0x7ff36400c8f0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.788+0000 7ff382cba640 1 -- 172.21.15.44:0/3709340585 shutdown_connections 2024-04-02T13:21:09.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.788+0000 7ff382cba640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff35c06c8f0 0x7ff35c06edb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.788+0000 7ff382cba640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff37c15e420 0x7ff37c154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.788+0000 7ff382cba640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff37c0cfc40 0x7ff37c1570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.788+0000 7ff382cba640 1 --2- 172.21.15.44:0/3709340585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37c0721f0 0x7ff37c14a850 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.788+0000 7ff382cba640 1 -- 172.21.15.44:0/3709340585 >> 172.21.15.44:0/3709340585 conn(0x7ff37c0b2a10 msgr2=0x7ff37c0b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:09.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.788+0000 7ff382cba640 1 -- 172.21.15.44:0/3709340585 shutdown_connections 2024-04-02T13:21:09.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.788+0000 7ff382cba640 1 -- 172.21.15.44:0/3709340585 wait complete. 2024-04-02T13:21:09.801 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:09.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.904+0000 7fa75b041640 1 Processor -- start 2024-04-02T13:21:09.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.904+0000 7fa75b041640 1 -- start start 2024-04-02T13:21:09.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.905+0000 7fa75b041640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa7540b4050 0x7fa7540ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.905+0000 7fa75b041640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7540cd4a0 con 0x7fa7540cd030 2024-04-02T13:21:09.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.905+0000 7fa75b041640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7540cd760 con 0x7fa75415c260 2024-04-02T13:21:09.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.905+0000 7fa75b041640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa7540cda40 con 0x7fa7540b4050 2024-04-02T13:21:09.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.905+0000 7fa7595de640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa7540cd030 0x7fa7540c5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54710/0 (socket says 172.21.15.44:54710) 2024-04-02T13:21:09.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.905+0000 7fa7595de640 1 -- 172.21.15.44:0/2560268735 learned_addr learned my addr 172.21.15.44:0/2560268735 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:09.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.905+0000 7fa753fff640 1 --2- 172.21.15.44:0/2560268735 >> v2:172.21.15.69:3300/0 conn(0x7fa7540b4050 0x7fa7540ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.905+0000 7fa7537fe640 1 -- 172.21.15.44:0/2560268735 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4259071666 0 0) 0x7fa7540cd4a0 con 0x7fa7540cd030 2024-04-02T13:21:09.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.905+0000 7fa7537fe640 1 -- 172.21.15.44:0/2560268735 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa73c003520 con 0x7fa7540cd030 2024-04-02T13:21:09.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.905+0000 7fa7537fe640 1 -- 172.21.15.44:0/2560268735 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1617732662 0 0) 0x7fa7540cd760 con 0x7fa75415c260 2024-04-02T13:21:09.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.905+0000 7fa7537fe640 1 -- 172.21.15.44:0/2560268735 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7540cd4a0 con 0x7fa75415c260 2024-04-02T13:21:09.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.906+0000 7fa753fff640 1 -- 172.21.15.44:0/2560268735 >> v1:172.21.15.69:6789/0 conn(0x7fa75415c260 legacy=0x7fa7540b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.906+0000 7fa753fff640 1 -- 172.21.15.44:0/2560268735 >> v1:172.21.15.44:6789/0 conn(0x7fa7540cd030 legacy=0x7fa7540c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.906+0000 7fa753fff640 1 -- 172.21.15.44:0/2560268735 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7540cdd20 con 0x7fa7540b4050 2024-04-02T13:21:09.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.906+0000 7fa753fff640 1 --2- 172.21.15.44:0/2560268735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7540b4050 0x7fa7540ccaf0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa7480097a0 tx=0x7fa74802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdb84541a2b79ae0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:09.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.906+0000 7fa7537fe640 1 -- 172.21.15.44:0/2560268735 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa748034070 con 0x7fa7540b4050 2024-04-02T13:21:09.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.906+0000 7fa7537fe640 1 -- 172.21.15.44:0/2560268735 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa748030610 con 0x7fa7540b4050 2024-04-02T13:21:09.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.906+0000 7fa7537fe640 1 -- 172.21.15.44:0/2560268735 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7480309c0 con 0x7fa7540b4050 2024-04-02T13:21:09.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.907+0000 7fa75b041640 1 -- 172.21.15.44:0/2560268735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7540b4050 msgr2=0x7fa7540ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.907+0000 7fa75b041640 1 --2- 172.21.15.44:0/2560268735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7540b4050 0x7fa7540ccaf0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa7480097a0 tx=0x7fa74802b960 comp rx=0 tx=0).stop 2024-04-02T13:21:09.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.907+0000 7fa75b041640 1 -- 172.21.15.44:0/2560268735 shutdown_connections 2024-04-02T13:21:09.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.907+0000 7fa75b041640 1 --2- 172.21.15.44:0/2560268735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7540b4050 0x7fa7540ccaf0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.907+0000 7fa75b041640 1 -- 172.21.15.44:0/2560268735 >> 172.21.15.44:0/2560268735 conn(0x7fa75409aa20 msgr2=0x7fa75409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:09.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.907+0000 7fa75b041640 1 -- 172.21.15.44:0/2560268735 shutdown_connections 2024-04-02T13:21:09.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.907+0000 7fa75b041640 1 -- 172.21.15.44:0/2560268735 wait complete. 2024-04-02T13:21:09.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.908+0000 7fa75b041640 1 Processor -- start 2024-04-02T13:21:09.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.908+0000 7fa75b041640 1 -- start start 2024-04-02T13:21:09.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.909+0000 7fa75b041640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7540b4050 0x7fa7541f1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.909+0000 7fa75b041640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7540cd030 0x7fa75415c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.909+0000 7fa75b041640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa75415c260 0x7fa75415e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.909+0000 7fa75b041640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa7540c6240 con 0x7fa75415c260 2024-04-02T13:21:09.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.909+0000 7fa75b041640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa7540cfe60 con 0x7fa7540cd030 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.909+0000 7fa75b041640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa7540c63c0 con 0x7fa7540b4050 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.909+0000 7fa7595de640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa75415c260 0x7fa75415e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.909+0000 7fa7595de640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa75415c260 0x7fa75415e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45184/0 (socket says 172.21.15.44:45184) 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.909+0000 7fa7595de640 1 -- 172.21.15.44:0/223094788 learned_addr learned my addr 172.21.15.44:0/223094788 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.909+0000 7fa753fff640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7540cd030 0x7fa75415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.909+0000 7fa758ddd640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7540b4050 0x7fa7541f1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.910+0000 7fa7595de640 1 -- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7540b4050 msgr2=0x7fa7541f1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.910+0000 7fa7595de640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7540b4050 0x7fa7541f1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.910+0000 7fa7595de640 1 -- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7540cd030 msgr2=0x7fa75415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.910+0000 7fa7595de640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7540cd030 0x7fa75415c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.910+0000 7fa7595de640 1 -- 172.21.15.44:0/223094788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7540b5440 con 0x7fa75415c260 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.910+0000 7fa7595de640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa75415c260 0x7fa75415e320 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa74400c430 tx=0x7fa74400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.910+0000 7fa7517fa640 1 -- 172.21.15.44:0/223094788 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa744017070 con 0x7fa75415c260 2024-04-02T13:21:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.910+0000 7fa7517fa640 1 -- 172.21.15.44:0/223094788 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa744005ce0 con 0x7fa75415c260 2024-04-02T13:21:09.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.910+0000 7fa7517fa640 1 -- 172.21.15.44:0/223094788 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7440072f0 con 0x7fa75415c260 2024-04-02T13:21:09.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.910+0000 7fa75b041640 1 -- 172.21.15.44:0/223094788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7540b5720 con 0x7fa75415c260 2024-04-02T13:21:09.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.910+0000 7fa75b041640 1 -- 172.21.15.44:0/223094788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7540b5ce0 con 0x7fa75415c260 2024-04-02T13:21:09.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.911+0000 7fa75b041640 1 -- 172.21.15.44:0/223094788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7540684f0 con 0x7fa75415c260 2024-04-02T13:21:09.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.911+0000 7fa7517fa640 1 -- 172.21.15.44:0/223094788 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa74401a070 con 0x7fa75415c260 2024-04-02T13:21:09.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.912+0000 7fa7517fa640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa74006c8f0 0x7fa74006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:09.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.912+0000 7fa7517fa640 1 -- 172.21.15.44:0/223094788 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fa74408b350 con 0x7fa75415c260 2024-04-02T13:21:09.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.912+0000 7fa758ddd640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa74006c8f0 0x7fa74006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:09.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.913+0000 7fa758ddd640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa74006c8f0 0x7fa74006edb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa74c001e00 tx=0x7fa74c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:09.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:09.916+0000 7fa7517fa640 1 -- 172.21.15.44:0/223094788 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa744058d10 con 0x7fa75415c260 2024-04-02T13:21:10.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.101+0000 7fa75b041640 1 -- 172.21.15.44:0/223094788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa75406fad0 con 0x7fa75415c260 2024-04-02T13:21:10.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.101+0000 7fa7517fa640 1 -- 172.21.15.44:0/223094788 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fa74405d010 con 0x7fa75415c260 2024-04-02T13:21:10.104 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:10.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.103+0000 7fa75b041640 1 -- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa74006c8f0 msgr2=0x7fa74006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.103+0000 7fa75b041640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa74006c8f0 0x7fa74006edb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa74c001e00 tx=0x7fa74c009040 comp rx=0 tx=0).stop 2024-04-02T13:21:10.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.103+0000 7fa75b041640 1 -- 172.21.15.44:0/223094788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa75415c260 msgr2=0x7fa75415e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.103+0000 7fa75b041640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa75415c260 0x7fa75415e320 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa74400c430 tx=0x7fa74400c8f0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.103+0000 7fa75b041640 1 -- 172.21.15.44:0/223094788 shutdown_connections 2024-04-02T13:21:10.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.103+0000 7fa75b041640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa74006c8f0 0x7fa74006edb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.103+0000 7fa75b041640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa75415c260 0x7fa75415e320 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.103+0000 7fa75b041640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7540cd030 0x7fa75415c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.103+0000 7fa75b041640 1 --2- 172.21.15.44:0/223094788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7540b4050 0x7fa7541f1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.103+0000 7fa75b041640 1 -- 172.21.15.44:0/223094788 >> 172.21.15.44:0/223094788 conn(0x7fa75409aa20 msgr2=0x7fa7540c80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:10.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.103+0000 7fa75b041640 1 -- 172.21.15.44:0/223094788 shutdown_connections 2024-04-02T13:21:10.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.103+0000 7fa75b041640 1 -- 172.21.15.44:0/223094788 wait complete. 2024-04-02T13:21:10.115 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:10.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.218+0000 7f6b94435640 1 Processor -- start 2024-04-02T13:21:10.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.218+0000 7f6b94435640 1 -- start start 2024-04-02T13:21:10.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.219+0000 7f6b94435640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c15e1d0 0x7f6b8c0d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.219+0000 7f6b94435640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6b8c0d2dd0 0x7f6b8c0b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.219+0000 7f6b94435640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8c0d1640 con 0x7f6b8c0cec60 2024-04-02T13:21:10.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.219+0000 7f6b94435640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6b8c0cfe00 con 0x7f6b8c0d2dd0 2024-04-02T13:21:10.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.219+0000 7f6b94435640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6b8c0cf1f0 con 0x7f6b8c15e1d0 2024-04-02T13:21:10.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.219+0000 7f6b919d0640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6b8c0cec60 0x7f6b8c0d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54722/0 (socket says 172.21.15.44:54722) 2024-04-02T13:21:10.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.219+0000 7f6b919d0640 1 -- 172.21.15.44:0/58361468 learned_addr learned my addr 172.21.15.44:0/58361468 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:10.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b921d1640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c15e1d0 0x7f6b8c0d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b929d2640 1 --2- 172.21.15.44:0/58361468 >> v2:172.21.15.69:3300/0 conn(0x7f6b8c0d2dd0 0x7f6b8c0b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 746160979 0 0) 0x7f6b8c0d1640 con 0x7f6b8c0cec60 2024-04-02T13:21:10.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b74003520 con 0x7f6b8c0cec60 2024-04-02T13:21:10.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2838794014 0 0) 0x7f6b74003520 con 0x7f6b8c0cec60 2024-04-02T13:21:10.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c15e1d0 msgr2=0x7f6b8c0d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c15e1d0 0x7f6b8c0d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 >> v2:172.21.15.69:3300/0 conn(0x7f6b8c0d2dd0 msgr2=0x7f6b8c0b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 --2- 172.21.15.44:0/58361468 >> v2:172.21.15.69:3300/0 conn(0x7f6b8c0d2dd0 0x7f6b8c0b4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8c0cf4d0 con 0x7f6b8c0cec60 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f6b80001670 con 0x7f6b8c0cec60 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b929d2640 1 --2- 172.21.15.44:0/58361468 >> v2:172.21.15.69:3300/0 conn(0x7f6b8c0d2dd0 0x7f6b8c0b4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 >> v1:172.21.15.44:6789/0 conn(0x7f6b8c0cec60 legacy=0x7f6b8c0d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c15e1d0 0x7f6b740039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b74003f30 0x7f6b74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b74004d40 0x7f6b740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c0cfe00 con 0x7f6b74003f30 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c0cf1f0 con 0x7f6b74004d40 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6b7400ba10 con 0x7f6b8c15e1d0 2024-04-02T13:21:10.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b919d0640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b74003f30 0x7f6b74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b921d1640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c15e1d0 0x7f6b740039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b929d2640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b74004d40 0x7f6b740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b919d0640 1 -- 172.21.15.44:0/58361468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c15e1d0 msgr2=0x7f6b740039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:10.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b919d0640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c15e1d0 0x7f6b740039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b919d0640 1 -- 172.21.15.44:0/58361468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b74004d40 msgr2=0x7f6b740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:10.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.220+0000 7f6b919d0640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b74004d40 0x7f6b740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.221+0000 7f6b919d0640 1 -- 172.21.15.44:0/58361468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b74003520 con 0x7f6b74003f30 2024-04-02T13:21:10.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.221+0000 7f6b921d1640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c15e1d0 0x7f6b740039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:10.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.221+0000 7f6b929d2640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b74004d40 0x7f6b740051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:10.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.221+0000 7f6b919d0640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b74003f30 0x7f6b74004800 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000c240 tx=0x7f6b74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb6148f792ad2dd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:10.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.221+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b80007d50 con 0x7f6b74003f30 2024-04-02T13:21:10.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.221+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b80030070 con 0x7f6b74003f30 2024-04-02T13:21:10.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.221+0000 7f6b911cf640 1 -- 172.21.15.44:0/58361468 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8002cdc0 con 0x7f6b74003f30 2024-04-02T13:21:10.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.222+0000 7f6b94435640 1 -- 172.21.15.44:0/58361468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b74003f30 msgr2=0x7f6b74004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.222+0000 7f6b94435640 1 --2- 172.21.15.44:0/58361468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b74003f30 0x7f6b74004800 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000c240 tx=0x7f6b74005830 comp rx=0 tx=0).stop 2024-04-02T13:21:10.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.222+0000 7f6b929d2640 1 -- 172.21.15.44:0/58361468 reap_dead start 2024-04-02T13:21:10.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.222+0000 7f6b94435640 1 -- 172.21.15.44:0/58361468 shutdown_connections 2024-04-02T13:21:10.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.222+0000 7f6b94435640 1 -- 172.21.15.44:0/58361468 >> 172.21.15.44:0/58361468 conn(0x7f6b8c09d3b0 msgr2=0x7f6b8c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.222+0000 7f6b94435640 1 -- 172.21.15.44:0/58361468 shutdown_connections 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.222+0000 7f6b94435640 1 -- 172.21.15.44:0/58361468 wait complete. 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.223+0000 7f6b94435640 1 Processor -- start 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.223+0000 7f6b94435640 1 -- start start 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b94435640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b74003f30 0x7f6b8c15b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b94435640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c09de20 0x7f6b8c0d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b94435640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b8c0d10f0 0x7f6b8c09f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b94435640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c15e5b0 con 0x7f6b74003f30 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b94435640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c0cfa90 con 0x7f6b8c0d10f0 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b94435640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6b8c15e730 con 0x7f6b8c09de20 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b921d1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b74003f30 0x7f6b8c15b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b921d1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b74003f30 0x7f6b8c15b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45212/0 (socket says 172.21.15.44:45212) 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b921d1640 1 -- 172.21.15.44:0/19426003 learned_addr learned my addr 172.21.15.44:0/19426003 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:10.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b919d0640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c09de20 0x7f6b8c0d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b929d2640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b8c0d10f0 0x7f6b8c09f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b921d1640 1 -- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c09de20 msgr2=0x7f6b8c0d0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b921d1640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c09de20 0x7f6b8c0d0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b921d1640 1 -- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b8c0d10f0 msgr2=0x7f6b8c09f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b921d1640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b8c0d10f0 0x7f6b8c09f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.224+0000 7f6b921d1640 1 -- 172.21.15.44:0/19426003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8c15b6e0 con 0x7f6b74003f30 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.225+0000 7f6b921d1640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b74003f30 0x7f6b8c15b1a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00f460 tx=0x7f6b7c00f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.225+0000 7f6b8affd640 1 -- 172.21.15.44:0/19426003 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c003b90 con 0x7f6b74003f30 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.225+0000 7f6b8affd640 1 -- 172.21.15.44:0/19426003 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b7c003d30 con 0x7f6b74003f30 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.225+0000 7f6b8affd640 1 -- 172.21.15.44:0/19426003 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c004240 con 0x7f6b74003f30 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.225+0000 7f6b94435640 1 -- 172.21.15.44:0/19426003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8c09e320 con 0x7f6b74003f30 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.225+0000 7f6b94435640 1 -- 172.21.15.44:0/19426003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b8c09e8e0 con 0x7f6b74003f30 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.226+0000 7f6b94435640 1 -- 172.21.15.44:0/19426003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b8c0684f0 con 0x7f6b74003f30 2024-04-02T13:21:10.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.226+0000 7f6b8affd640 1 -- 172.21.15.44:0/19426003 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6b7c0043e0 con 0x7f6b74003f30 2024-04-02T13:21:10.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.227+0000 7f6b8affd640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b7806c8f0 0x7f6b7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.227+0000 7f6b8affd640 1 -- 172.21.15.44:0/19426003 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f6b7c08cec0 con 0x7f6b74003f30 2024-04-02T13:21:10.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.227+0000 7f6b919d0640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b7806c8f0 0x7f6b7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.227+0000 7f6b919d0640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b7806c8f0 0x7f6b7806edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6b80034000 tx=0x7f6b80007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:10.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.231+0000 7f6b8affd640 1 -- 172.21.15.44:0/19426003 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6b7c019030 con 0x7f6b74003f30 2024-04-02T13:21:10.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.419+0000 7f6b94435640 1 -- 172.21.15.44:0/19426003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6b8c0d47f0 con 0x7f6b74003f30 2024-04-02T13:21:10.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.419+0000 7f6b8affd640 1 -- 172.21.15.44:0/19426003 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f6b7c010040 con 0x7f6b74003f30 2024-04-02T13:21:10.421 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:10.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.421+0000 7f6b94435640 1 -- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b7806c8f0 msgr2=0x7f6b7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.421+0000 7f6b94435640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b7806c8f0 0x7f6b7806edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6b80034000 tx=0x7f6b80007000 comp rx=0 tx=0).stop 2024-04-02T13:21:10.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.421+0000 7f6b94435640 1 -- 172.21.15.44:0/19426003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b74003f30 msgr2=0x7f6b8c15b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.421+0000 7f6b94435640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b74003f30 0x7f6b8c15b1a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00f460 tx=0x7f6b7c00f920 comp rx=0 tx=0).stop 2024-04-02T13:21:10.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.421+0000 7f6b94435640 1 -- 172.21.15.44:0/19426003 shutdown_connections 2024-04-02T13:21:10.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.421+0000 7f6b94435640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b7806c8f0 0x7f6b7806edb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.421+0000 7f6b94435640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b8c0d10f0 0x7f6b8c09f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.421+0000 7f6b94435640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b8c09de20 0x7f6b8c0d0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.421+0000 7f6b94435640 1 --2- 172.21.15.44:0/19426003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b74003f30 0x7f6b8c15b1a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.421+0000 7f6b94435640 1 -- 172.21.15.44:0/19426003 >> 172.21.15.44:0/19426003 conn(0x7f6b8c09d3b0 msgr2=0x7f6b8c0b8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:10.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.421+0000 7f6b94435640 1 -- 172.21.15.44:0/19426003 shutdown_connections 2024-04-02T13:21:10.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.421+0000 7f6b94435640 1 -- 172.21.15.44:0/19426003 wait complete. 2024-04-02T13:21:10.433 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:10.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.536+0000 7fe7b4511640 1 Processor -- start 2024-04-02T13:21:10.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.536+0000 7fe7b4511640 1 -- start start 2024-04-02T13:21:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.537+0000 7fe7b4511640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe7ac0b1540 0x7fe7ac0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.537+0000 7fe7b4511640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe7ac0ce790 0x7fe7ac06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.537+0000 7fe7b4511640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7ac0ced00 con 0x7fe7ac15e470 2024-04-02T13:21:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.537+0000 7fe7b4511640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe7ac0cefe0 con 0x7fe7ac0ce790 2024-04-02T13:21:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.537+0000 7fe7b4511640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe7ac0cf2c0 con 0x7fe7ac0b1540 2024-04-02T13:21:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.537+0000 7fe7b2aae640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe7ac0ce790 0x7fe7ac06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.537+0000 7fe7b2aae640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe7ac0ce790 0x7fe7ac06d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45220/0 (socket says 172.21.15.44:45220) 2024-04-02T13:21:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.537+0000 7fe7b2aae640 1 -- 172.21.15.44:0/1400428283 learned_addr learned my addr 172.21.15.44:0/1400428283 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.537+0000 7fe7b1aac640 1 --2- 172.21.15.44:0/1400428283 >> v2:172.21.15.69:3300/0 conn(0x7fe7ac0b1540 0x7fe7ac0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.537+0000 7fe7b2aae640 1 -- 172.21.15.44:0/1400428283 >> v2:172.21.15.69:3300/0 conn(0x7fe7ac0b1540 msgr2=0x7fe7ac0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.538+0000 7fe7b2aae640 1 --2- 172.21.15.44:0/1400428283 >> v2:172.21.15.69:3300/0 conn(0x7fe7ac0b1540 0x7fe7ac0b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.538+0000 7fe7b2aae640 1 -- 172.21.15.44:0/1400428283 >> v1:172.21.15.69:6789/0 conn(0x7fe7ac15e470 legacy=0x7fe7ac0d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.538+0000 7fe7b2aae640 1 -- 172.21.15.44:0/1400428283 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ac0cf5a0 con 0x7fe7ac0ce790 2024-04-02T13:21:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.538+0000 7fe7b1aac640 1 --2- 172.21.15.44:0/1400428283 >> v2:172.21.15.69:3300/0 conn(0x7fe7ac0b1540 0x7fe7ac0b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:10.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.538+0000 7fe7b2aae640 1 --2- 172.21.15.44:0/1400428283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7ac0ce790 0x7fe7ac06d0f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac163070 tx=0x7fe7a402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a57c4f32728fc91 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:10.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.538+0000 7fe7b12ab640 1 -- 172.21.15.44:0/1400428283 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a402e070 con 0x7fe7ac0ce790 2024-04-02T13:21:10.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.538+0000 7fe7b12ab640 1 -- 172.21.15.44:0/1400428283 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a4037070 con 0x7fe7ac0ce790 2024-04-02T13:21:10.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.538+0000 7fe7b12ab640 1 -- 172.21.15.44:0/1400428283 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4031a00 con 0x7fe7ac0ce790 2024-04-02T13:21:10.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.539+0000 7fe7b4511640 1 -- 172.21.15.44:0/1400428283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7ac0ce790 msgr2=0x7fe7ac06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.539+0000 7fe7b4511640 1 --2- 172.21.15.44:0/1400428283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7ac0ce790 0x7fe7ac06d0f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac163070 tx=0x7fe7a402b880 comp rx=0 tx=0).stop 2024-04-02T13:21:10.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.539+0000 7fe7b4511640 1 -- 172.21.15.44:0/1400428283 shutdown_connections 2024-04-02T13:21:10.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.539+0000 7fe7b4511640 1 --2- 172.21.15.44:0/1400428283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7ac0ce790 0x7fe7ac06d0f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.539+0000 7fe7b4511640 1 --2- 172.21.15.44:0/1400428283 >> v2:172.21.15.69:3300/0 conn(0x7fe7ac0b1540 0x7fe7ac0b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.539+0000 7fe7b4511640 1 -- 172.21.15.44:0/1400428283 >> 172.21.15.44:0/1400428283 conn(0x7fe7ac09aa20 msgr2=0x7fe7ac09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:10.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.539+0000 7fe7b4511640 1 -- 172.21.15.44:0/1400428283 shutdown_connections 2024-04-02T13:21:10.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.539+0000 7fe7b4511640 1 -- 172.21.15.44:0/1400428283 wait complete. 2024-04-02T13:21:10.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.540+0000 7fe7b4511640 1 Processor -- start 2024-04-02T13:21:10.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.540+0000 7fe7b4511640 1 -- start start 2024-04-02T13:21:10.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b4511640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7ac0b1540 0x7fe7ac0c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b4511640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7ac0ce790 0x7fe7ac1f6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b4511640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac15e470 0x7fe7ac15c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b4511640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac0d22c0 con 0x7fe7ac0b1540 2024-04-02T13:21:10.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b4511640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac06cb60 con 0x7fe7ac15e470 2024-04-02T13:21:10.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b4511640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe7ac0d2440 con 0x7fe7ac0ce790 2024-04-02T13:21:10.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b22ad640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7ac0b1540 0x7fe7ac0c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b22ad640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7ac0b1540 0x7fe7ac0c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45230/0 (socket says 172.21.15.44:45230) 2024-04-02T13:21:10.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b22ad640 1 -- 172.21.15.44:0/3694608631 learned_addr learned my addr 172.21.15.44:0/3694608631 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:10.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b2aae640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac15e470 0x7fe7ac15c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b1aac640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7ac0ce790 0x7fe7ac1f6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b22ad640 1 -- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7ac0ce790 msgr2=0x7fe7ac1f6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b22ad640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7ac0ce790 0x7fe7ac1f6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b22ad640 1 -- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac15e470 msgr2=0x7fe7ac15c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b22ad640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac15e470 0x7fe7ac15c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b22ad640 1 -- 172.21.15.44:0/3694608631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ac0b5da0 con 0x7fe7ac0b1540 2024-04-02T13:21:10.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b2aae640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac15e470 0x7fe7ac15c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:10.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.541+0000 7fe7b1aac640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7ac0ce790 0x7fe7ac1f6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:10.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.542+0000 7fe7b22ad640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7ac0b1540 0x7fe7ac0c7b80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00ba80 tx=0x7fe79c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:10.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.542+0000 7fe7aaffd640 1 -- 172.21.15.44:0/3694608631 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79c00cc00 con 0x7fe7ac0b1540 2024-04-02T13:21:10.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.542+0000 7fe7aaffd640 1 -- 172.21.15.44:0/3694608631 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe79c00cda0 con 0x7fe7ac0b1540 2024-04-02T13:21:10.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.542+0000 7fe7b4511640 1 -- 172.21.15.44:0/3694608631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7ac0c5d90 con 0x7fe7ac0b1540 2024-04-02T13:21:10.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.542+0000 7fe7aaffd640 1 -- 172.21.15.44:0/3694608631 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79c003bb0 con 0x7fe7ac0b1540 2024-04-02T13:21:10.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.542+0000 7fe7b4511640 1 -- 172.21.15.44:0/3694608631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7ac0c6350 con 0x7fe7ac0b1540 2024-04-02T13:21:10.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.543+0000 7fe7b4511640 1 -- 172.21.15.44:0/3694608631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7ac0684f0 con 0x7fe7ac0b1540 2024-04-02T13:21:10.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.543+0000 7fe7aaffd640 1 -- 172.21.15.44:0/3694608631 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe79c014440 con 0x7fe7ac0b1540 2024-04-02T13:21:10.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.544+0000 7fe7aaffd640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe79806c8f0 0x7fe79806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.544+0000 7fe7aaffd640 1 -- 172.21.15.44:0/3694608631 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fe79c08be30 con 0x7fe7ac0b1540 2024-04-02T13:21:10.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.544+0000 7fe7b1aac640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe79806c8f0 0x7fe79806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.545+0000 7fe7b1aac640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe79806c8f0 0x7fe79806edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0006fd0 tx=0x7fe7a0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:10.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.548+0000 7fe7aaffd640 1 -- 172.21.15.44:0/3694608631 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe79c059ed0 con 0x7fe7ac0b1540 2024-04-02T13:21:10.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.730+0000 7fe7b4511640 1 -- 172.21.15.44:0/3694608631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe7ac0c6a60 con 0x7fe7ac0b1540 2024-04-02T13:21:10.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.731+0000 7fe7aaffd640 1 -- 172.21.15.44:0/3694608631 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fe79c0597d0 con 0x7fe7ac0b1540 2024-04-02T13:21:10.732 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:10.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.732+0000 7fe7b4511640 1 -- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe79806c8f0 msgr2=0x7fe79806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.732+0000 7fe7b4511640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe79806c8f0 0x7fe79806edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0006fd0 tx=0x7fe7a0008040 comp rx=0 tx=0).stop 2024-04-02T13:21:10.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.732+0000 7fe7b4511640 1 -- 172.21.15.44:0/3694608631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7ac0b1540 msgr2=0x7fe7ac0c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.732+0000 7fe7b4511640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7ac0b1540 0x7fe7ac0c7b80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00ba80 tx=0x7fe79c00bf40 comp rx=0 tx=0).stop 2024-04-02T13:21:10.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.732+0000 7fe7b4511640 1 -- 172.21.15.44:0/3694608631 shutdown_connections 2024-04-02T13:21:10.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.732+0000 7fe7b4511640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe79806c8f0 0x7fe79806edb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.732+0000 7fe7b4511640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac15e470 0x7fe7ac15c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.732+0000 7fe7b4511640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7ac0ce790 0x7fe7ac1f6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.732+0000 7fe7b4511640 1 --2- 172.21.15.44:0/3694608631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7ac0b1540 0x7fe7ac0c7b80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.732+0000 7fe7b4511640 1 -- 172.21.15.44:0/3694608631 >> 172.21.15.44:0/3694608631 conn(0x7fe7ac09aa20 msgr2=0x7fe7ac06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:10.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.733+0000 7fe7b4511640 1 -- 172.21.15.44:0/3694608631 shutdown_connections 2024-04-02T13:21:10.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.733+0000 7fe7b4511640 1 -- 172.21.15.44:0/3694608631 wait complete. 2024-04-02T13:21:10.745 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2024-04-02T13:21:10.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.845+0000 7fd8ab7ba640 1 Processor -- start 2024-04-02T13:21:10.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.845+0000 7fd8ab7ba640 1 -- start start 2024-04-02T13:21:10.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.846+0000 7fd8ab7ba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd8a40d0a30 0x7fd8a406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.846+0000 7fd8ab7ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a40d0e10 0x7fd8a40cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.846+0000 7fd8ab7ba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a406c1d0 con 0x7fd8a40cfe30 2024-04-02T13:21:10.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.846+0000 7fd8ab7ba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd8a406cb00 con 0x7fd8a40d0a30 2024-04-02T13:21:10.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.846+0000 7fd8ab7ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a406c4b0 con 0x7fd8a40d0e10 2024-04-02T13:21:10.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.846+0000 7fd8a9d57640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd8a40cfe30 0x7fd8a40d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54730/0 (socket says 172.21.15.44:54730) 2024-04-02T13:21:10.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.846+0000 7fd8a9d57640 1 -- 172.21.15.44:0/3956489490 learned_addr learned my addr 172.21.15.44:0/3956489490 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:10.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.846+0000 7fd8a8d55640 1 --2- 172.21.15.44:0/3956489490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a40d0e10 0x7fd8a40cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.846+0000 7fd8a9556640 1 --2- 172.21.15.44:0/3956489490 >> v2:172.21.15.69:3300/0 conn(0x7fd8a40d0a30 0x7fd8a406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3622981670 0 0) 0x7fd8a406c1d0 con 0x7fd8a40cfe30 2024-04-02T13:21:10.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd88c003520 con 0x7fd8a40cfe30 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1363724646 0 0) 0x7fd88c003520 con 0x7fd8a40cfe30 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a40d0e10 msgr2=0x7fd8a40cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 --2- 172.21.15.44:0/3956489490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a40d0e10 0x7fd8a40cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 >> v2:172.21.15.69:3300/0 conn(0x7fd8a40d0a30 msgr2=0x7fd8a406cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 --2- 172.21.15.44:0/3956489490 >> v2:172.21.15.69:3300/0 conn(0x7fd8a40d0a30 0x7fd8a406cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a406c790 con 0x7fd8a40cfe30 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fd894001670 con 0x7fd8a40cfe30 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a9556640 1 --2- 172.21.15.44:0/3956489490 >> v2:172.21.15.69:3300/0 conn(0x7fd8a40d0a30 0x7fd8a406cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 >> v1:172.21.15.44:6789/0 conn(0x7fd8a40cfe30 legacy=0x7fd8a40d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 --2- 172.21.15.44:0/3956489490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd88c0039b0 0x7fd88c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 --2- 172.21.15.44:0/3956489490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88c004800 0x7fd88c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 --2- 172.21.15.44:0/3956489490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd88c0051c0 0x7fd88c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a406cb00 con 0x7fd88c004800 2024-04-02T13:21:10.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a406c4b0 con 0x7fd88c0039b0 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd88c00bf30 con 0x7fd88c0051c0 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a8d55640 1 --2- 172.21.15.44:0/3956489490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a40d0e10 0x7fd8a40cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a9556640 1 --2- 172.21.15.44:0/3956489490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd88c0039b0 0x7fd88c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a8d55640 1 --2- 172.21.15.44:0/3956489490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88c004800 0x7fd88c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a9d57640 1 --2- 172.21.15.44:0/3956489490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd88c0051c0 0x7fd88c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a8d55640 1 -- 172.21.15.44:0/3956489490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd88c0051c0 msgr2=0x7fd88c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.847+0000 7fd8a8d55640 1 --2- 172.21.15.44:0/3956489490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd88c0051c0 0x7fd88c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.848+0000 7fd8a8d55640 1 -- 172.21.15.44:0/3956489490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd88c0039b0 msgr2=0x7fd88c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.848+0000 7fd8a8d55640 1 --2- 172.21.15.44:0/3956489490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd88c0039b0 0x7fd88c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.848+0000 7fd8a8d55640 1 -- 172.21.15.44:0/3956489490 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88c003520 con 0x7fd88c004800 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.848+0000 7fd8a8d55640 1 --2- 172.21.15.44:0/3956489490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88c004800 0x7fd88c004c80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd8a40cb1c0 tx=0x7fd88c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5501f46092a2356 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.848+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8980097f0 con 0x7fd88c004800 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.848+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd898004030 con 0x7fd88c004800 2024-04-02T13:21:10.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.848+0000 7fd8a3fff640 1 -- 172.21.15.44:0/3956489490 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898004400 con 0x7fd88c004800 2024-04-02T13:21:10.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.849+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/3956489490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88c004800 msgr2=0x7fd88c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.849+0000 7fd8ab7ba640 1 --2- 172.21.15.44:0/3956489490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88c004800 0x7fd88c004c80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd8a40cb1c0 tx=0x7fd88c005830 comp rx=0 tx=0).stop 2024-04-02T13:21:10.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.849+0000 7fd8a9d57640 1 -- 172.21.15.44:0/3956489490 reap_dead start 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.849+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/3956489490 shutdown_connections 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.849+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/3956489490 >> 172.21.15.44:0/3956489490 conn(0x7fd8a409d470 msgr2=0x7fd8a40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.849+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/3956489490 shutdown_connections 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.849+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/3956489490 wait complete. 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.850+0000 7fd8ab7ba640 1 Processor -- start 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.850+0000 7fd8ab7ba640 1 -- start start 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8ab7ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd88c004800 0x7fd8a41595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8ab7ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8a415f650 0x7fd8a415fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8ab7ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8a41e8d30 0x7fd8a41e91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8ab7ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a415e870 con 0x7fd8a41e8d30 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8ab7ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a415e6f0 con 0x7fd8a415f650 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8ab7ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a40d4da0 con 0x7fd88c004800 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8a9d57640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8a41e8d30 0x7fd8a41e91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8a9d57640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8a41e8d30 0x7fd8a41e91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45240/0 (socket says 172.21.15.44:45240) 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8a9d57640 1 -- 172.21.15.44:0/95295598 learned_addr learned my addr 172.21.15.44:0/95295598 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8a9556640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd88c004800 0x7fd8a41595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8a8d55640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8a415f650 0x7fd8a415fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8a9d57640 1 -- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd88c004800 msgr2=0x7fd8a41595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8a9d57640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd88c004800 0x7fd8a41595a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8a9d57640 1 -- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8a415f650 msgr2=0x7fd8a415fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8a9d57640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8a415f650 0x7fd8a415fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8a9d57640 1 -- 172.21.15.44:0/95295598 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a41e9720 con 0x7fd8a41e8d30 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8a8d55640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8a415f650 0x7fd8a415fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8a9556640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd88c004800 0x7fd8a41595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.851+0000 7fd8a9d57640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8a41e8d30 0x7fd8a41e91e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd8a406bdc0 tx=0x7fd894007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.852+0000 7fd8a1ffb640 1 -- 172.21.15.44:0/95295598 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89400f600 con 0x7fd8a41e8d30 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.852+0000 7fd8a1ffb640 1 -- 172.21.15.44:0/95295598 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd894018070 con 0x7fd8a41e8d30 2024-04-02T13:21:10.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.852+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/95295598 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a41e7f70 con 0x7fd8a41e8d30 2024-04-02T13:21:10.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.852+0000 7fd8a1ffb640 1 -- 172.21.15.44:0/95295598 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89400fa00 con 0x7fd8a41e8d30 2024-04-02T13:21:10.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.852+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/95295598 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a41e8530 con 0x7fd8a41e8d30 2024-04-02T13:21:10.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.853+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/95295598 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8a40684f0 con 0x7fd8a41e8d30 2024-04-02T13:21:10.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.853+0000 7fd8a1ffb640 1 -- 172.21.15.44:0/95295598 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd89400fba0 con 0x7fd8a41e8d30 2024-04-02T13:21:10.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.854+0000 7fd8a1ffb640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd89006c8f0 0x7fd89006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:10.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.854+0000 7fd8a1ffb640 1 -- 172.21.15.44:0/95295598 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fd89408bcf0 con 0x7fd8a41e8d30 2024-04-02T13:21:10.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.854+0000 7fd8a9556640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd89006c8f0 0x7fd89006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:10.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.855+0000 7fd8a9556640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd89006c8f0 0x7fd89006edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd89c007560 tx=0x7fd89c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:10.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:10.858+0000 7fd8a1ffb640 1 -- 172.21.15.44:0/95295598 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd894090070 con 0x7fd8a41e8d30 2024-04-02T13:21:11.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:11.117+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/95295598 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) -- 0x7fd8a40d17c0 con 0x7fd8a41e8d30 2024-04-02T13:21:12.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.082+0000 7fd8a1ffb640 1 -- 172.21.15.44:0/95295598 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v67) ==== 145+0+0 (secure 0 0 0) 0x7fd894059740 con 0x7fd8a41e8d30 2024-04-02T13:21:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.085+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd89006c8f0 msgr2=0x7fd89006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.085+0000 7fd8ab7ba640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd89006c8f0 0x7fd89006edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd89c007560 tx=0x7fd89c00b040 comp rx=0 tx=0).stop 2024-04-02T13:21:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.085+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/95295598 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8a41e8d30 msgr2=0x7fd8a41e91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.085+0000 7fd8ab7ba640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8a41e8d30 0x7fd8a41e91e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd8a406bdc0 tx=0x7fd894007430 comp rx=0 tx=0).stop 2024-04-02T13:21:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.085+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/95295598 shutdown_connections 2024-04-02T13:21:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.085+0000 7fd8ab7ba640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd89006c8f0 0x7fd89006edb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.085+0000 7fd8ab7ba640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8a41e8d30 0x7fd8a41e91e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.085+0000 7fd8ab7ba640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8a415f650 0x7fd8a415fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.085+0000 7fd8ab7ba640 1 --2- 172.21.15.44:0/95295598 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd88c004800 0x7fd8a41595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.085+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/95295598 >> 172.21.15.44:0/95295598 conn(0x7fd8a409d470 msgr2=0x7fd8a40d3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:12.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.085+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/95295598 shutdown_connections 2024-04-02T13:21:12.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.085+0000 7fd8ab7ba640 1 -- 172.21.15.44:0/95295598 wait complete. 2024-04-02T13:21:12.100 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T13:21:12.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.209+0000 7f6ae30d3640 1 Processor -- start 2024-04-02T13:21:12.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.209+0000 7f6ae30d3640 1 -- start start 2024-04-02T13:21:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.210+0000 7f6ae30d3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6adc0b1c30 0x7f6adc0b2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.210+0000 7f6ae30d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6adc0cd810 0x7f6adc0b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.210+0000 7f6ae30d3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6adc0b2550 con 0x7f6adc0b3bf0 2024-04-02T13:21:12.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.210+0000 7f6ae30d3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6adc0ce800 con 0x7f6adc0b1c30 2024-04-02T13:21:12.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.210+0000 7f6ae30d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6adc06e530 con 0x7f6adc0cd810 2024-04-02T13:21:12.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.210+0000 7f6ae1670640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6adc0b3bf0 0x7f6adc0b40a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54746/0 (socket says 172.21.15.44:54746) 2024-04-02T13:21:12.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.210+0000 7f6ae1670640 1 -- 172.21.15.44:0/934324611 learned_addr learned my addr 172.21.15.44:0/934324611 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:12.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.210+0000 7f6ae0e6f640 1 --2- 172.21.15.44:0/934324611 >> v2:172.21.15.69:3300/0 conn(0x7f6adc0b1c30 0x7f6adc0b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.210+0000 7f6adbfff640 1 --2- 172.21.15.44:0/934324611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6adc0cd810 0x7f6adc0b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.210+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1072134880 0 0) 0x7f6adc0b2550 con 0x7f6adc0b3bf0 2024-04-02T13:21:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.210+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ac8003520 con 0x7f6adc0b3bf0 2024-04-02T13:21:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4207861307 0 0) 0x7f6ac8003520 con 0x7f6adc0b3bf0 2024-04-02T13:21:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6adc0cd810 msgr2=0x7f6adc0b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 --2- 172.21.15.44:0/934324611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6adc0cd810 0x7f6adc0b3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 >> v2:172.21.15.69:3300/0 conn(0x7f6adc0b1c30 msgr2=0x7f6adc0b2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 --2- 172.21.15.44:0/934324611 >> v2:172.21.15.69:3300/0 conn(0x7f6adc0b1c30 0x7f6adc0b2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6adc0cea30 con 0x7f6adc0b3bf0 2024-04-02T13:21:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f6ac4001670 con 0x7f6adc0b3bf0 2024-04-02T13:21:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 >> v1:172.21.15.44:6789/0 conn(0x7f6adc0b3bf0 legacy=0x7f6adc0b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 --2- 172.21.15.44:0/934324611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac80039b0 0x7f6ac8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 --2- 172.21.15.44:0/934324611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ac8004800 0x7f6ac8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 --2- 172.21.15.44:0/934324611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6adc0cd810 0x7f6ac80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc06e530 con 0x7f6ac80039b0 2024-04-02T13:21:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac800b9c0 con 0x7f6ac8004800 2024-04-02T13:21:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac800bb40 con 0x7f6adc0cd810 2024-04-02T13:21:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6ae0e6f640 1 --2- 172.21.15.44:0/934324611 >> v2:172.21.15.69:3300/0 conn(0x7f6adc0b1c30 0x7f6adc0b2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6ae0e6f640 1 --2- 172.21.15.44:0/934324611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac80039b0 0x7f6ac8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6adbfff640 1 --2- 172.21.15.44:0/934324611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ac8004800 0x7f6ac8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6ae1670640 1 --2- 172.21.15.44:0/934324611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6adc0cd810 0x7f6ac80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6ae0e6f640 1 -- 172.21.15.44:0/934324611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6adc0cd810 msgr2=0x7f6ac80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6ae0e6f640 1 --2- 172.21.15.44:0/934324611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6adc0cd810 0x7f6ac80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6ae0e6f640 1 -- 172.21.15.44:0/934324611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ac8004800 msgr2=0x7f6ac8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:12.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6ae0e6f640 1 --2- 172.21.15.44:0/934324611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ac8004800 0x7f6ac8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.211+0000 7f6ae0e6f640 1 -- 172.21.15.44:0/934324611 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac8003520 con 0x7f6ac80039b0 2024-04-02T13:21:12.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.212+0000 7f6ae0e6f640 1 --2- 172.21.15.44:0/934324611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac80039b0 0x7f6ac8003dd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009c70 tx=0x7f6ac8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=696ff6efc5bba420 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:12.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.212+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4009840 con 0x7f6ac80039b0 2024-04-02T13:21:12.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.212+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ad4004020 con 0x7f6ac80039b0 2024-04-02T13:21:12.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.212+0000 7f6adb7fe640 1 -- 172.21.15.44:0/934324611 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4004410 con 0x7f6ac80039b0 2024-04-02T13:21:12.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.213+0000 7f6ae30d3640 1 -- 172.21.15.44:0/934324611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac80039b0 msgr2=0x7f6ac8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.213+0000 7f6ae30d3640 1 --2- 172.21.15.44:0/934324611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac80039b0 0x7f6ac8003dd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009c70 tx=0x7f6ac8005830 comp rx=0 tx=0).stop 2024-04-02T13:21:12.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.213+0000 7f6ae1670640 1 -- 172.21.15.44:0/934324611 reap_dead start 2024-04-02T13:21:12.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.213+0000 7f6ae30d3640 1 -- 172.21.15.44:0/934324611 shutdown_connections 2024-04-02T13:21:12.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.213+0000 7f6ae30d3640 1 -- 172.21.15.44:0/934324611 >> 172.21.15.44:0/934324611 conn(0x7f6adc06b4d0 msgr2=0x7f6adc06b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:12.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.213+0000 7f6ae30d3640 1 -- 172.21.15.44:0/934324611 shutdown_connections 2024-04-02T13:21:12.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.213+0000 7f6ae30d3640 1 -- 172.21.15.44:0/934324611 wait complete. 2024-04-02T13:21:12.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.214+0000 7f6ae30d3640 1 Processor -- start 2024-04-02T13:21:12.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.214+0000 7f6ae30d3640 1 -- start start 2024-04-02T13:21:12.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae30d3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac80039b0 0x7f6adc0c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae30d3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6adc156470 0x7f6adc09ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae30d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6adc1568e0 0x7f6adc09d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae30d3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc0cf2d0 con 0x7f6ac80039b0 2024-04-02T13:21:12.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae30d3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc0ceff0 con 0x7f6adc156470 2024-04-02T13:21:12.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae30d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6adc0cf5b0 con 0x7f6adc1568e0 2024-04-02T13:21:12.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae0e6f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac80039b0 0x7f6adc0c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae0e6f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac80039b0 0x7f6adc0c7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45256/0 (socket says 172.21.15.44:45256) 2024-04-02T13:21:12.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae0e6f640 1 -- 172.21.15.44:0/2519751808 learned_addr learned my addr 172.21.15.44:0/2519751808 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:12.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae1670640 1 --2- 172.21.15.44:0/2519751808 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6adc1568e0 0x7f6adc09d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae0e6f640 1 -- 172.21.15.44:0/2519751808 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6adc1568e0 msgr2=0x7f6adc09d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae0e6f640 1 --2- 172.21.15.44:0/2519751808 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6adc1568e0 0x7f6adc09d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae0e6f640 1 -- 172.21.15.44:0/2519751808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6adc156470 msgr2=0x7f6adc09ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae0e6f640 1 --2- 172.21.15.44:0/2519751808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6adc156470 0x7f6adc09ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae0e6f640 1 -- 172.21.15.44:0/2519751808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6adc0adaf0 con 0x7f6ac80039b0 2024-04-02T13:21:12.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.215+0000 7f6ae0e6f640 1 --2- 172.21.15.44:0/2519751808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac80039b0 0x7f6adc0c7ec0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009b90 tx=0x7f6ad4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:12.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.216+0000 7f6ad97fa640 1 -- 172.21.15.44:0/2519751808 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4043020 con 0x7f6ac80039b0 2024-04-02T13:21:12.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.216+0000 7f6ad97fa640 1 -- 172.21.15.44:0/2519751808 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ad402c960 con 0x7f6ac80039b0 2024-04-02T13:21:12.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.216+0000 7f6ae30d3640 1 -- 172.21.15.44:0/2519751808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6adc0c8630 con 0x7f6ac80039b0 2024-04-02T13:21:12.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.216+0000 7f6ad97fa640 1 -- 172.21.15.44:0/2519751808 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4006e10 con 0x7f6ac80039b0 2024-04-02T13:21:12.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.216+0000 7f6ae30d3640 1 -- 172.21.15.44:0/2519751808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6adc0ae070 con 0x7f6ac80039b0 2024-04-02T13:21:12.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.217+0000 7f6ae30d3640 1 -- 172.21.15.44:0/2519751808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6adc0684f0 con 0x7f6ac80039b0 2024-04-02T13:21:12.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.217+0000 7f6ad97fa640 1 -- 172.21.15.44:0/2519751808 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ad4046020 con 0x7f6ac80039b0 2024-04-02T13:21:12.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.217+0000 7f6ad97fa640 1 --2- 172.21.15.44:0/2519751808 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6abc06c920 0x7f6abc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.218+0000 7f6ad97fa640 1 -- 172.21.15.44:0/2519751808 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f6ad40ade50 con 0x7f6ac80039b0 2024-04-02T13:21:12.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.218+0000 7f6adbfff640 1 --2- 172.21.15.44:0/2519751808 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6abc06c920 0x7f6abc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.218+0000 7f6adbfff640 1 --2- 172.21.15.44:0/2519751808 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6abc06c920 0x7f6abc06ede0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6adc156890 tx=0x7f6acc009750 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:12.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.222+0000 7f6ad97fa640 1 -- 172.21.15.44:0/2519751808 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6ad404d030 con 0x7f6ac80039b0 2024-04-02T13:21:12.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.405+0000 7f6ae30d3640 1 -- 172.21.15.44:0/2519751808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f6adc06e780 con 0x7f6ac80039b0 2024-04-02T13:21:12.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.408+0000 7f6ad97fa640 1 -- 172.21.15.44:0/2519751808 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f6ad407bef0 con 0x7f6ac80039b0 2024-04-02T13:21:12.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.410+0000 7f6ae30d3640 1 -- 172.21.15.44:0/2519751808 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6abc06c920 msgr2=0x7f6abc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.410+0000 7f6ae30d3640 1 --2- 172.21.15.44:0/2519751808 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6abc06c920 0x7f6abc06ede0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6adc156890 tx=0x7f6acc009750 comp rx=0 tx=0).stop 2024-04-02T13:21:12.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.410+0000 7f6ae30d3640 1 -- 172.21.15.44:0/2519751808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac80039b0 msgr2=0x7f6adc0c7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.410+0000 7f6ae30d3640 1 --2- 172.21.15.44:0/2519751808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac80039b0 0x7f6adc0c7ec0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009b90 tx=0x7f6ad4004730 comp rx=0 tx=0).stop 2024-04-02T13:21:12.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.410+0000 7f6ae30d3640 1 -- 172.21.15.44:0/2519751808 shutdown_connections 2024-04-02T13:21:12.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.410+0000 7f6ae30d3640 1 --2- 172.21.15.44:0/2519751808 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6abc06c920 0x7f6abc06ede0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.410+0000 7f6ae30d3640 1 --2- 172.21.15.44:0/2519751808 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6adc1568e0 0x7f6adc09d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.410+0000 7f6ae30d3640 1 --2- 172.21.15.44:0/2519751808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6adc156470 0x7f6adc09ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.410+0000 7f6ae30d3640 1 --2- 172.21.15.44:0/2519751808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac80039b0 0x7f6adc0c7ec0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.410+0000 7f6ae30d3640 1 -- 172.21.15.44:0/2519751808 >> 172.21.15.44:0/2519751808 conn(0x7f6adc06b4d0 msgr2=0x7f6adc0b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:12.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.410+0000 7f6ae30d3640 1 -- 172.21.15.44:0/2519751808 shutdown_connections 2024-04-02T13:21:12.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.410+0000 7f6ae30d3640 1 -- 172.21.15.44:0/2519751808 wait complete. 2024-04-02T13:21:12.422 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:21:12.423 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:21:12.461 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:21:12.515 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:21:12.516 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:21:12.516 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:12.531 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:21:12.531 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:12.586 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:12.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.726+0000 7f721a2f2640 1 Processor -- start 2024-04-02T13:21:12.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.726+0000 7f721a2f2640 1 -- start start 2024-04-02T13:21:12.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.727+0000 7f721a2f2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f721415e4d0 0x7f72140cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.727+0000 7f721a2f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72140cfd20 0x7f72140d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.727+0000 7f721a2f2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f721415e8b0 con 0x7f72140d1060 2024-04-02T13:21:12.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.727+0000 7f721a2f2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f72140b3a90 con 0x7f721415e4d0 2024-04-02T13:21:12.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.727+0000 7f721a2f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f72140b4610 con 0x7f72140cfd20 2024-04-02T13:21:12.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.727+0000 7f721888f640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f72140d1060 0x7f72140d1d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54760/0 (socket says 172.21.15.44:54760) 2024-04-02T13:21:12.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.727+0000 7f721888f640 1 -- 172.21.15.44:0/3935791616 learned_addr learned my addr 172.21.15.44:0/3935791616 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:12.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f72137fe640 1 --2- 172.21.15.44:0/3935791616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72140cfd20 0x7f72140d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7213fff640 1 --2- 172.21.15.44:0/3935791616 >> v2:172.21.15.69:3300/0 conn(0x7f721415e4d0 0x7f72140cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 744494691 0 0) 0x7f721415e8b0 con 0x7f72140d1060 2024-04-02T13:21:12.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71fc003520 con 0x7f72140d1060 2024-04-02T13:21:12.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3057964444 0 0) 0x7f71fc003520 con 0x7f72140d1060 2024-04-02T13:21:12.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72140cfd20 msgr2=0x7f72140d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:12.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 --2- 172.21.15.44:0/3935791616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72140cfd20 0x7f72140d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 >> v2:172.21.15.69:3300/0 conn(0x7f721415e4d0 msgr2=0x7f72140cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:12.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 --2- 172.21.15.44:0/3935791616 >> v2:172.21.15.69:3300/0 conn(0x7f721415e4d0 0x7f72140cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72140d2fb0 con 0x7f72140d1060 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f720c0019f0 con 0x7f72140d1060 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 >> v1:172.21.15.44:6789/0 conn(0x7f72140d1060 legacy=0x7f72140d1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 --2- 172.21.15.44:0/3935791616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72140cfd20 0x7f71fc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 --2- 172.21.15.44:0/3935791616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71fc003f30 0x7f71fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 --2- 172.21.15.44:0/3935791616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71fc004d40 0x7f71fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72140b3a90 con 0x7f71fc003f30 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72140b4610 con 0x7f71fc004d40 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f71fc00b9c0 con 0x7f72140cfd20 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f72137fe640 1 --2- 172.21.15.44:0/3935791616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71fc003f30 0x7f71fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f7213fff640 1 --2- 172.21.15.44:0/3935791616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72140cfd20 0x7f71fc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f721888f640 1 --2- 172.21.15.44:0/3935791616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71fc004d40 0x7f71fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f72137fe640 1 -- 172.21.15.44:0/3935791616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72140cfd20 msgr2=0x7f71fc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f72137fe640 1 --2- 172.21.15.44:0/3935791616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72140cfd20 0x7f71fc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f72137fe640 1 -- 172.21.15.44:0/3935791616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71fc004d40 msgr2=0x7f71fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:12.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f72137fe640 1 --2- 172.21.15.44:0/3935791616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71fc004d40 0x7f71fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.728+0000 7f72137fe640 1 -- 172.21.15.44:0/3935791616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71fc003520 con 0x7f71fc003f30 2024-04-02T13:21:12.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.729+0000 7f72137fe640 1 --2- 172.21.15.44:0/3935791616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71fc003f30 0x7f71fc004800 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7208009ac0 tx=0x7f71fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45428f21147c902e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:12.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.729+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7208039070 con 0x7f71fc003f30 2024-04-02T13:21:12.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.729+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f720802c6b0 con 0x7f71fc003f30 2024-04-02T13:21:12.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.729+0000 7f7212ffd640 1 -- 172.21.15.44:0/3935791616 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7208009690 con 0x7f71fc003f30 2024-04-02T13:21:12.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.730+0000 7f721a2f2640 1 -- 172.21.15.44:0/3935791616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71fc003f30 msgr2=0x7f71fc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.730+0000 7f721a2f2640 1 --2- 172.21.15.44:0/3935791616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71fc003f30 0x7f71fc004800 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7208009ac0 tx=0x7f71fc005830 comp rx=0 tx=0).stop 2024-04-02T13:21:12.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.730+0000 7f721888f640 1 -- 172.21.15.44:0/3935791616 reap_dead start 2024-04-02T13:21:12.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.730+0000 7f721a2f2640 1 -- 172.21.15.44:0/3935791616 shutdown_connections 2024-04-02T13:21:12.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.730+0000 7f721a2f2640 1 -- 172.21.15.44:0/3935791616 >> 172.21.15.44:0/3935791616 conn(0x7f721409d410 msgr2=0x7f72140d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:12.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.730+0000 7f721a2f2640 1 -- 172.21.15.44:0/3935791616 shutdown_connections 2024-04-02T13:21:12.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.730+0000 7f721a2f2640 1 -- 172.21.15.44:0/3935791616 wait complete. 2024-04-02T13:21:12.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.731+0000 7f721a2f2640 1 Processor -- start 2024-04-02T13:21:12.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.731+0000 7f721a2f2640 1 -- start start 2024-04-02T13:21:12.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f721a2f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71fc003f30 0x7f7214161590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f721a2f2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f721414e260 0x7f72140cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f721a2f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721414d610 0x7f721414da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f721a2f2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72140b42a0 con 0x7f71fc003f30 2024-04-02T13:21:12.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f721a2f2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72140b3fc0 con 0x7f721414e260 2024-04-02T13:21:12.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f721a2f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f72140b0d00 con 0x7f721414d610 2024-04-02T13:21:12.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f7213fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71fc003f30 0x7f7214161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f7213fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71fc003f30 0x7f7214161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45284/0 (socket says 172.21.15.44:45284) 2024-04-02T13:21:12.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f7213fff640 1 -- 172.21.15.44:0/1216675255 learned_addr learned my addr 172.21.15.44:0/1216675255 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:12.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f72137fe640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f721414e260 0x7f72140cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f721888f640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721414d610 0x7f721414da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f7213fff640 1 -- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721414d610 msgr2=0x7f721414da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f7213fff640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721414d610 0x7f721414da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f7213fff640 1 -- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f721414e260 msgr2=0x7f72140cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f7213fff640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f721414e260 0x7f72140cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f7213fff640 1 -- 172.21.15.44:0/1216675255 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72140b6ef0 con 0x7f71fc003f30 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.732+0000 7f721888f640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721414d610 0x7f721414da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.733+0000 7f7213fff640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71fc003f30 0x7f7214161590 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7204011a70 tx=0x7f7204011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.733+0000 7f7210ff9640 1 -- 172.21.15.44:0/1216675255 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f720400fb50 con 0x7f71fc003f30 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.733+0000 7f7210ff9640 1 -- 172.21.15.44:0/1216675255 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f720400fcf0 con 0x7f71fc003f30 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.733+0000 7f721a2f2640 1 -- 172.21.15.44:0/1216675255 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72140b71d0 con 0x7f71fc003f30 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.733+0000 7f7210ff9640 1 -- 172.21.15.44:0/1216675255 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f720400fea0 con 0x7f71fc003f30 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.733+0000 7f721a2f2640 1 -- 172.21.15.44:0/1216675255 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72140cdb20 con 0x7f71fc003f30 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.734+0000 7f721a2f2640 1 -- 172.21.15.44:0/1216675255 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7214068c00 con 0x7f71fc003f30 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.734+0000 7f7210ff9640 1 -- 172.21.15.44:0/1216675255 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7204012040 con 0x7f71fc003f30 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.735+0000 7f7210ff9640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f720006c8f0 0x7f720006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.735+0000 7f7210ff9640 1 -- 172.21.15.44:0/1216675255 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f720408c660 con 0x7f71fc003f30 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.735+0000 7f72137fe640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f720006c8f0 0x7f720006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:12.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.735+0000 7f72137fe640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f720006c8f0 0x7f720006edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7208036000 tx=0x7f7208030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:12.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.739+0000 7f7210ff9640 1 -- 172.21.15.44:0/1216675255 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7204019030 con 0x7f71fc003f30 2024-04-02T13:21:12.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.925+0000 7f721a2f2640 1 -- 172.21.15.44:0/1216675255 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f72140b73a0 con 0x7f71fc003f30 2024-04-02T13:21:12.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.926+0000 7f7210ff9640 1 -- 172.21.15.44:0/1216675255 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18899 (secure 0 0 0) 0x7f720405a670 con 0x7f71fc003f30 2024-04-02T13:21:12.928 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:12.928 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:08.061253+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:12.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.928+0000 7f721a2f2640 1 -- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f720006c8f0 msgr2=0x7f720006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.928+0000 7f721a2f2640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f720006c8f0 0x7f720006edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7208036000 tx=0x7f7208030000 comp rx=0 tx=0).stop 2024-04-02T13:21:12.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.928+0000 7f721a2f2640 1 -- 172.21.15.44:0/1216675255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71fc003f30 msgr2=0x7f7214161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:12.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.928+0000 7f721a2f2640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71fc003f30 0x7f7214161590 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7204011a70 tx=0x7f7204011f30 comp rx=0 tx=0).stop 2024-04-02T13:21:12.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.929+0000 7f721a2f2640 1 -- 172.21.15.44:0/1216675255 shutdown_connections 2024-04-02T13:21:12.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.929+0000 7f721a2f2640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f720006c8f0 0x7f720006edb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.929+0000 7f721a2f2640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721414d610 0x7f721414da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.929+0000 7f721a2f2640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f721414e260 0x7f72140cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.929+0000 7f721a2f2640 1 --2- 172.21.15.44:0/1216675255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71fc003f30 0x7f7214161590 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:12.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.929+0000 7f721a2f2640 1 -- 172.21.15.44:0/1216675255 >> 172.21.15.44:0/1216675255 conn(0x7f721409d410 msgr2=0x7f72140b2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:12.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.929+0000 7f721a2f2640 1 -- 172.21.15.44:0/1216675255 shutdown_connections 2024-04-02T13:21:12.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:12.929+0000 7f721a2f2640 1 -- 172.21.15.44:0/1216675255 wait complete. 2024-04-02T13:21:12.942 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:12.942 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:13.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.046+0000 7f10934a5640 1 Processor -- start 2024-04-02T13:21:13.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.046+0000 7f10934a5640 1 -- start start 2024-04-02T13:21:13.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.047+0000 7f10934a5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f108c0ce9a0 0x7f108c0d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.047+0000 7f10934a5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f108c0cdc20 con 0x7f108c0d0670 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.047+0000 7f10934a5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f108c0cdec0 con 0x7f108c1554c0 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.047+0000 7f10934a5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f108c0d0aa0 con 0x7f108c0ce9a0 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.047+0000 7f1091a42640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f108c0d0670 0x7f108c164c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54768/0 (socket says 172.21.15.44:54768) 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.047+0000 7f1091a42640 1 -- 172.21.15.44:0/846941902 learned_addr learned my addr 172.21.15.44:0/846941902 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.047+0000 7f1090a40640 1 --2- 172.21.15.44:0/846941902 >> v2:172.21.15.69:3300/0 conn(0x7f108c0ce9a0 0x7f108c0d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.047+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3553961348 0 0) 0x7f108c0cdc20 con 0x7f108c0d0670 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.047+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1074003520 con 0x7f108c0d0670 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1399782033 0 0) 0x7f108c0cdec0 con 0x7f108c1554c0 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f108c0cdc20 con 0x7f108c1554c0 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1263022093 0 0) 0x7f1074003520 con 0x7f108c0d0670 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 >> v2:172.21.15.69:3300/0 conn(0x7f108c0ce9a0 msgr2=0x7f108c0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 --2- 172.21.15.44:0/846941902 >> v2:172.21.15.69:3300/0 conn(0x7f108c0ce9a0 0x7f108c0d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 >> v1:172.21.15.69:6789/0 conn(0x7f108c1554c0 legacy=0x7f108c071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108c0ce1a0 con 0x7f108c0d0670 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f107c001a10 con 0x7f108c0d0670 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f1090a40640 1 --2- 172.21.15.44:0/846941902 >> v2:172.21.15.69:3300/0 conn(0x7f108c0ce9a0 0x7f108c0d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 >> v1:172.21.15.44:6789/0 conn(0x7f108c0d0670 legacy=0x7f108c164c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 --2- 172.21.15.44:0/846941902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1074003b40 0x7f1074003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 --2- 172.21.15.44:0/846941902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1074004800 0x7f1074004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 --2- 172.21.15.44:0/846941902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10740051c0 0x7f107400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f108c0d0aa0 con 0x7f10740051c0 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f107400c040 con 0x7f1074004800 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f107400c240 con 0x7f1074003b40 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f1091a42640 1 --2- 172.21.15.44:0/846941902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10740051c0 0x7f107400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f1091241640 1 --2- 172.21.15.44:0/846941902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1074003b40 0x7f1074003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f1090a40640 1 --2- 172.21.15.44:0/846941902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1074004800 0x7f1074004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f1091a42640 1 -- 172.21.15.44:0/846941902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1074003b40 msgr2=0x7f1074003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f1091a42640 1 --2- 172.21.15.44:0/846941902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1074003b40 0x7f1074003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f1091a42640 1 -- 172.21.15.44:0/846941902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1074004800 msgr2=0x7f1074004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:13.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f1091a42640 1 --2- 172.21.15.44:0/846941902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1074004800 0x7f1074004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.048+0000 7f1091a42640 1 -- 172.21.15.44:0/846941902 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1074003520 con 0x7f10740051c0 2024-04-02T13:21:13.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.049+0000 7f1091a42640 1 -- 172.21.15.44:0/846941902 reap_dead start 2024-04-02T13:21:13.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.049+0000 7f1091a42640 1 --2- 172.21.15.44:0/846941902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10740051c0 0x7f107400b9c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f107c00a7f0 tx=0x7f1074005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c554406a2754378a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:13.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.049+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c00a450 con 0x7f10740051c0 2024-04-02T13:21:13.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.049+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f107c0072f0 con 0x7f10740051c0 2024-04-02T13:21:13.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.049+0000 7f108bfff640 1 -- 172.21.15.44:0/846941902 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c0076e0 con 0x7f10740051c0 2024-04-02T13:21:13.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.050+0000 7f10934a5640 1 -- 172.21.15.44:0/846941902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10740051c0 msgr2=0x7f107400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.050+0000 7f10934a5640 1 --2- 172.21.15.44:0/846941902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10740051c0 0x7f107400b9c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f107c00a7f0 tx=0x7f1074005830 comp rx=0 tx=0).stop 2024-04-02T13:21:13.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.050+0000 7f10934a5640 1 -- 172.21.15.44:0/846941902 shutdown_connections 2024-04-02T13:21:13.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.050+0000 7f10934a5640 1 --2- 172.21.15.44:0/846941902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10740051c0 0x7f107400b9c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.050+0000 7f10934a5640 1 -- 172.21.15.44:0/846941902 >> 172.21.15.44:0/846941902 conn(0x7f108c06b920 msgr2=0x7f108c06bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.050+0000 7f10934a5640 1 -- 172.21.15.44:0/846941902 shutdown_connections 2024-04-02T13:21:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.050+0000 7f10934a5640 1 -- 172.21.15.44:0/846941902 wait complete. 2024-04-02T13:21:13.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.051+0000 7f10934a5640 1 Processor -- start 2024-04-02T13:21:13.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.051+0000 7f10934a5640 1 -- start start 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f10934a5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10740051c0 0x7f108c14a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f10934a5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f108c09aa20 0x7f108c0c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f10934a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f108c0c9530 0x7f108c15c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f10934a5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f108c15c290 con 0x7f108c09aa20 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f10934a5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f108c1558a0 con 0x7f108c0c9530 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f10934a5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f108c0ad360 con 0x7f10740051c0 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f1090a40640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f108c09aa20 0x7f108c0c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f1090a40640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f108c09aa20 0x7f108c0c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45292/0 (socket says 172.21.15.44:45292) 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f1090a40640 1 -- 172.21.15.44:0/2593198873 learned_addr learned my addr 172.21.15.44:0/2593198873 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f1091241640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10740051c0 0x7f108c14a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f1091a42640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f108c0c9530 0x7f108c15c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f1090a40640 1 -- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10740051c0 msgr2=0x7f108c14a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f1090a40640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10740051c0 0x7f108c14a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f1090a40640 1 -- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f108c0c9530 msgr2=0x7f108c15c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f1090a40640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f108c0c9530 0x7f108c15c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f1090a40640 1 -- 172.21.15.44:0/2593198873 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108c0acd50 con 0x7f108c09aa20 2024-04-02T13:21:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f1091a42640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f108c0c9530 0x7f108c15c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f1090a40640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f108c09aa20 0x7f108c0c8ff0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1080011a90 tx=0x7f1080011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.052+0000 7f1089ffb640 1 -- 172.21.15.44:0/2593198873 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108000faf0 con 0x7f108c09aa20 2024-04-02T13:21:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.053+0000 7f1089ffb640 1 -- 172.21.15.44:0/2593198873 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f108000fc90 con 0x7f108c09aa20 2024-04-02T13:21:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.053+0000 7f10934a5640 1 -- 172.21.15.44:0/2593198873 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f108c0c9a40 con 0x7f108c09aa20 2024-04-02T13:21:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.053+0000 7f1089ffb640 1 -- 172.21.15.44:0/2593198873 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080004020 con 0x7f108c09aa20 2024-04-02T13:21:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.053+0000 7f10934a5640 1 -- 172.21.15.44:0/2593198873 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f108c0acff0 con 0x7f108c09aa20 2024-04-02T13:21:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.054+0000 7f10934a5640 1 -- 172.21.15.44:0/2593198873 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f108c0684f0 con 0x7f108c09aa20 2024-04-02T13:21:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.054+0000 7f1089ffb640 1 -- 172.21.15.44:0/2593198873 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1080012040 con 0x7f108c09aa20 2024-04-02T13:21:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.055+0000 7f1089ffb640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f107806c920 0x7f107806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.055+0000 7f1089ffb640 1 -- 172.21.15.44:0/2593198873 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f108008c410 con 0x7f108c09aa20 2024-04-02T13:21:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.055+0000 7f1091241640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f107806c920 0x7f107806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.055+0000 7f1091241640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f107806c920 0x7f107806ede0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1084000c00 tx=0x7f1084009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.059+0000 7f1089ffb640 1 -- 172.21.15.44:0/2593198873 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1080019030 con 0x7f108c09aa20 2024-04-02T13:21:13.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.242+0000 7f10934a5640 1 -- 172.21.15.44:0/2593198873 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f108c0ad4e0 con 0x7f108c09aa20 2024-04-02T13:21:13.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.243+0000 7f1089ffb640 1 -- 172.21.15.44:0/2593198873 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f108005a380 con 0x7f108c09aa20 2024-04-02T13:21:13.244 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:13.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.245+0000 7f10934a5640 1 -- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f107806c920 msgr2=0x7f107806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.245+0000 7f10934a5640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f107806c920 0x7f107806ede0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1084000c00 tx=0x7f1084009040 comp rx=0 tx=0).stop 2024-04-02T13:21:13.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.245+0000 7f10934a5640 1 -- 172.21.15.44:0/2593198873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f108c09aa20 msgr2=0x7f108c0c8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.245+0000 7f10934a5640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f108c09aa20 0x7f108c0c8ff0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1080011a90 tx=0x7f1080011e90 comp rx=0 tx=0).stop 2024-04-02T13:21:13.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.245+0000 7f10934a5640 1 -- 172.21.15.44:0/2593198873 shutdown_connections 2024-04-02T13:21:13.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.245+0000 7f10934a5640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f107806c920 0x7f107806ede0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.245+0000 7f10934a5640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f108c0c9530 0x7f108c15c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.245+0000 7f10934a5640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f108c09aa20 0x7f108c0c8ff0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.245+0000 7f10934a5640 1 --2- 172.21.15.44:0/2593198873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10740051c0 0x7f108c14a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.245+0000 7f10934a5640 1 -- 172.21.15.44:0/2593198873 >> 172.21.15.44:0/2593198873 conn(0x7f108c06b920 msgr2=0x7f108c15d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:13.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.245+0000 7f10934a5640 1 -- 172.21.15.44:0/2593198873 shutdown_connections 2024-04-02T13:21:13.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.245+0000 7f10934a5640 1 -- 172.21.15.44:0/2593198873 wait complete. 2024-04-02T13:21:13.257 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:13.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.361+0000 7f92636b1640 1 Processor -- start 2024-04-02T13:21:13.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.361+0000 7f92636b1640 1 -- start start 2024-04-02T13:21:13.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.361+0000 7f92636b1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f925c0b4180 0x7f925c168f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.361+0000 7f92636b1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f925c0d1630 con 0x7f925c06cff0 2024-04-02T13:21:13.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.361+0000 7f92636b1640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f925c0d1910 con 0x7f925c15e5d0 2024-04-02T13:21:13.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.361+0000 7f92636b1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f925c0d1bf0 con 0x7f925c0b4180 2024-04-02T13:21:13.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.362+0000 7f926144d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f925c06cff0 0x7f925c06d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54782/0 (socket says 172.21.15.44:54782) 2024-04-02T13:21:13.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.362+0000 7f926144d640 1 -- 172.21.15.44:0/4007402105 learned_addr learned my addr 172.21.15.44:0/4007402105 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:13.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.362+0000 7f9261c4e640 1 --2- 172.21.15.44:0/4007402105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f925c0b4180 0x7f925c168f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.362+0000 7f9253fff640 1 -- 172.21.15.44:0/4007402105 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1731326200 0 0) 0x7f925c0d1630 con 0x7f925c06cff0 2024-04-02T13:21:13.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.362+0000 7f9253fff640 1 -- 172.21.15.44:0/4007402105 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9248003520 con 0x7f925c06cff0 2024-04-02T13:21:13.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.362+0000 7f9261c4e640 1 -- 172.21.15.44:0/4007402105 >> v1:172.21.15.69:6789/0 conn(0x7f925c15e5d0 legacy=0x7f925c0d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.362+0000 7f9253fff640 1 -- 172.21.15.44:0/4007402105 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3194003501 0 0) 0x7f925c0d1910 con 0x7f925c15e5d0 2024-04-02T13:21:13.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.362+0000 7f9261c4e640 1 -- 172.21.15.44:0/4007402105 >> v1:172.21.15.44:6789/0 conn(0x7f925c06cff0 legacy=0x7f925c06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.362+0000 7f9261c4e640 1 -- 172.21.15.44:0/4007402105 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925c0b1540 con 0x7f925c0b4180 2024-04-02T13:21:13.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.363+0000 7f9261c4e640 1 --2- 172.21.15.44:0/4007402105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f925c0b4180 0x7f925c168f20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f925c0d1d70 tx=0x7f925402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b770357171cb50a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:13.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.363+0000 7f9253fff640 1 -- 172.21.15.44:0/4007402105 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9254039070 con 0x7f925c0b4180 2024-04-02T13:21:13.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.363+0000 7f9253fff640 1 -- 172.21.15.44:0/4007402105 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f925402cc40 con 0x7f925c0b4180 2024-04-02T13:21:13.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.363+0000 7f9253fff640 1 -- 172.21.15.44:0/4007402105 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9254034440 con 0x7f925c0b4180 2024-04-02T13:21:13.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.364+0000 7f92636b1640 1 -- 172.21.15.44:0/4007402105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f925c0b4180 msgr2=0x7f925c168f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.364+0000 7f92636b1640 1 --2- 172.21.15.44:0/4007402105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f925c0b4180 0x7f925c168f20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f925c0d1d70 tx=0x7f925402be90 comp rx=0 tx=0).stop 2024-04-02T13:21:13.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.364+0000 7f92636b1640 1 -- 172.21.15.44:0/4007402105 shutdown_connections 2024-04-02T13:21:13.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.364+0000 7f92636b1640 1 --2- 172.21.15.44:0/4007402105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f925c0b4180 0x7f925c168f20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.364+0000 7f92636b1640 1 -- 172.21.15.44:0/4007402105 >> 172.21.15.44:0/4007402105 conn(0x7f925c09aa20 msgr2=0x7f925c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:13.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.364+0000 7f92636b1640 1 -- 172.21.15.44:0/4007402105 shutdown_connections 2024-04-02T13:21:13.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.364+0000 7f92636b1640 1 -- 172.21.15.44:0/4007402105 wait complete. 2024-04-02T13:21:13.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.365+0000 7f92636b1640 1 Processor -- start 2024-04-02T13:21:13.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.365+0000 7f92636b1640 1 -- start start 2024-04-02T13:21:13.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f92636b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f925c06cff0 0x7f925c0c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f92636b1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f925c0b4180 0x7f925c0c5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f92636b1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f925c15e5d0 0x7f925c1f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f92636b1640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f925c0b2ed0 con 0x7f925c06cff0 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f92636b1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f925c0b2d50 con 0x7f925c0b4180 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f92636b1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f925c0b3050 con 0x7f925c15e5d0 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f926144d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f925c06cff0 0x7f925c0c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f926144d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f925c06cff0 0x7f925c0c6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45294/0 (socket says 172.21.15.44:45294) 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f926144d640 1 -- 172.21.15.44:0/335621391 learned_addr learned my addr 172.21.15.44:0/335621391 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f9261c4e640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f925c15e5d0 0x7f925c1f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f926144d640 1 -- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f925c15e5d0 msgr2=0x7f925c1f3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f9260c4c640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f925c0b4180 0x7f925c0c5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f926144d640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f925c15e5d0 0x7f925c1f3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f926144d640 1 -- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f925c0b4180 msgr2=0x7f925c0c5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f926144d640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f925c0b4180 0x7f925c0c5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f926144d640 1 -- 172.21.15.44:0/335621391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925c0b59e0 con 0x7f925c06cff0 2024-04-02T13:21:13.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f9261c4e640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f925c15e5d0 0x7f925c1f3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:13.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f9260c4c640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f925c0b4180 0x7f925c0c5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:13.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f926144d640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f925c06cff0 0x7f925c0c6d60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f925800c3c0 tx=0x7f925800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:13.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f9251ffb640 1 -- 172.21.15.44:0/335621391 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258017070 con 0x7f925c06cff0 2024-04-02T13:21:13.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f9251ffb640 1 -- 172.21.15.44:0/335621391 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9258005ce0 con 0x7f925c06cff0 2024-04-02T13:21:13.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f92636b1640 1 -- 172.21.15.44:0/335621391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f925c1f4770 con 0x7f925c06cff0 2024-04-02T13:21:13.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.366+0000 7f9251ffb640 1 -- 172.21.15.44:0/335621391 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580072f0 con 0x7f925c06cff0 2024-04-02T13:21:13.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.367+0000 7f92636b1640 1 -- 172.21.15.44:0/335621391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f925c0b5c60 con 0x7f925c06cff0 2024-04-02T13:21:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.367+0000 7f92636b1640 1 -- 172.21.15.44:0/335621391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f925c0684f0 con 0x7f925c06cff0 2024-04-02T13:21:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.368+0000 7f9251ffb640 1 -- 172.21.15.44:0/335621391 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f925801a070 con 0x7f925c06cff0 2024-04-02T13:21:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.368+0000 7f9251ffb640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f924006c8f0 0x7f924006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.368+0000 7f9251ffb640 1 -- 172.21.15.44:0/335621391 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f925808b340 con 0x7f925c06cff0 2024-04-02T13:21:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.368+0000 7f9260c4c640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f924006c8f0 0x7f924006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.369+0000 7f9260c4c640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f924006c8f0 0x7f924006edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f925c161030 tx=0x7f924c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:13.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.372+0000 7f9251ffb640 1 -- 172.21.15.44:0/335621391 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f925c0b59e0 con 0x7f925c06cff0 2024-04-02T13:21:13.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.559+0000 7f92636b1640 1 -- 172.21.15.44:0/335621391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f925c06cdd0 con 0x7f925c06cff0 2024-04-02T13:21:13.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.559+0000 7f9251ffb640 1 -- 172.21.15.44:0/335621391 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f9258059350 con 0x7f925c06cff0 2024-04-02T13:21:13.561 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:13.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.561+0000 7f92636b1640 1 -- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f924006c8f0 msgr2=0x7f924006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.561+0000 7f92636b1640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f924006c8f0 0x7f924006edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f925c161030 tx=0x7f924c009040 comp rx=0 tx=0).stop 2024-04-02T13:21:13.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.561+0000 7f92636b1640 1 -- 172.21.15.44:0/335621391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f925c06cff0 msgr2=0x7f925c0c6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.561+0000 7f92636b1640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f925c06cff0 0x7f925c0c6d60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f925800c3c0 tx=0x7f925800c880 comp rx=0 tx=0).stop 2024-04-02T13:21:13.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.561+0000 7f92636b1640 1 -- 172.21.15.44:0/335621391 shutdown_connections 2024-04-02T13:21:13.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.561+0000 7f92636b1640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f924006c8f0 0x7f924006edb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.561+0000 7f92636b1640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f925c15e5d0 0x7f925c1f3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.561+0000 7f92636b1640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f925c0b4180 0x7f925c0c5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.561+0000 7f92636b1640 1 --2- 172.21.15.44:0/335621391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f925c06cff0 0x7f925c0c6d60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.561+0000 7f92636b1640 1 -- 172.21.15.44:0/335621391 >> 172.21.15.44:0/335621391 conn(0x7f925c09aa20 msgr2=0x7f925c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:13.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.561+0000 7f92636b1640 1 -- 172.21.15.44:0/335621391 shutdown_connections 2024-04-02T13:21:13.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.561+0000 7f92636b1640 1 -- 172.21.15.44:0/335621391 wait complete. 2024-04-02T13:21:13.573 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:13.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.676+0000 7f7cef035640 1 Processor -- start 2024-04-02T13:21:13.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.676+0000 7f7cef035640 1 -- start start 2024-04-02T13:21:13.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.677+0000 7f7cef035640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7ce815e1d0 0x7f7ce80d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.677+0000 7f7cef035640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ce80d1640 con 0x7f7ce80d2dd0 2024-04-02T13:21:13.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.677+0000 7f7cef035640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ce80cf1f0 con 0x7f7ce80cec60 2024-04-02T13:21:13.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.677+0000 7f7cef035640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7ce80cfe00 con 0x7f7ce815e1d0 2024-04-02T13:21:13.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.677+0000 7f7cecdd1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7ce815e1d0 0x7f7ce80d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.677+0000 7f7ced5d2640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7ce80d2dd0 0x7f7ce80b4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54794/0 (socket says 172.21.15.44:54794) 2024-04-02T13:21:13.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.677+0000 7f7ced5d2640 1 -- 172.21.15.44:0/2094743146 learned_addr learned my addr 172.21.15.44:0/2094743146 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:13.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.678+0000 7f7cecdd1640 1 -- 172.21.15.44:0/2094743146 >> v1:172.21.15.69:6789/0 conn(0x7f7ce80cec60 legacy=0x7f7ce80d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.678+0000 7f7cecdd1640 1 -- 172.21.15.44:0/2094743146 >> v1:172.21.15.44:6789/0 conn(0x7f7ce80d2dd0 legacy=0x7f7ce80b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.678+0000 7f7ce77fe640 1 -- 172.21.15.44:0/2094743146 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3282370599 0 0) 0x7f7ce80d1640 con 0x7f7ce80d2dd0 2024-04-02T13:21:13.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.678+0000 7f7cecdd1640 1 -- 172.21.15.44:0/2094743146 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce80cf4d0 con 0x7f7ce815e1d0 2024-04-02T13:21:13.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.678+0000 7f7cecdd1640 1 --2- 172.21.15.44:0/2094743146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ce815e1d0 0x7f7ce80d1100 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0009d90 tx=0x7f7ce002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b144dfef7edf7b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:13.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.679+0000 7f7ce77fe640 1 -- 172.21.15.44:0/2094743146 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0039070 con 0x7f7ce815e1d0 2024-04-02T13:21:13.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.679+0000 7f7ce77fe640 1 -- 172.21.15.44:0/2094743146 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce002cc60 con 0x7f7ce815e1d0 2024-04-02T13:21:13.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.679+0000 7f7ce77fe640 1 -- 172.21.15.44:0/2094743146 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0034660 con 0x7f7ce815e1d0 2024-04-02T13:21:13.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.679+0000 7f7cef035640 1 -- 172.21.15.44:0/2094743146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ce815e1d0 msgr2=0x7f7ce80d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.679+0000 7f7cef035640 1 --2- 172.21.15.44:0/2094743146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ce815e1d0 0x7f7ce80d1100 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0009d90 tx=0x7f7ce002beb0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.679+0000 7f7cef035640 1 -- 172.21.15.44:0/2094743146 shutdown_connections 2024-04-02T13:21:13.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.679+0000 7f7cef035640 1 --2- 172.21.15.44:0/2094743146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ce815e1d0 0x7f7ce80d1100 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.679+0000 7f7cef035640 1 -- 172.21.15.44:0/2094743146 >> 172.21.15.44:0/2094743146 conn(0x7f7ce809d3b0 msgr2=0x7f7ce80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:13.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.679+0000 7f7cef035640 1 -- 172.21.15.44:0/2094743146 shutdown_connections 2024-04-02T13:21:13.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.679+0000 7f7cef035640 1 -- 172.21.15.44:0/2094743146 wait complete. 2024-04-02T13:21:13.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.680+0000 7f7cef035640 1 Processor -- start 2024-04-02T13:21:13.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.680+0000 7f7cef035640 1 -- start start 2024-04-02T13:21:13.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.681+0000 7f7cef035640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ce80cec60 0x7f7ce809e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.681+0000 7f7cef035640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ce80d2dd0 0x7f7ce80a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.681+0000 7f7cef035640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ce815e1d0 0x7f7ce80a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.681+0000 7f7cef035640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce80b7fd0 con 0x7f7ce815e1d0 2024-04-02T13:21:13.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.681+0000 7f7cef035640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce80cfa90 con 0x7f7ce80d2dd0 2024-04-02T13:21:13.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.681+0000 7f7cef035640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7ce80b8150 con 0x7f7ce80cec60 2024-04-02T13:21:13.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.681+0000 7f7ced5d2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ce815e1d0 0x7f7ce80a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.681+0000 7f7ced5d2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ce815e1d0 0x7f7ce80a11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45300/0 (socket says 172.21.15.44:45300) 2024-04-02T13:21:13.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.681+0000 7f7ced5d2640 1 -- 172.21.15.44:0/2562193468 learned_addr learned my addr 172.21.15.44:0/2562193468 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:13.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.681+0000 7f7ce7fff640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ce80d2dd0 0x7f7ce80a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.681+0000 7f7cecdd1640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ce80cec60 0x7f7ce809e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.682+0000 7f7ced5d2640 1 -- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ce80cec60 msgr2=0x7f7ce809e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.682+0000 7f7ced5d2640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ce80cec60 0x7f7ce809e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.682+0000 7f7ced5d2640 1 -- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ce80d2dd0 msgr2=0x7f7ce80a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.682+0000 7f7ced5d2640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ce80d2dd0 0x7f7ce80a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.682+0000 7f7ced5d2640 1 -- 172.21.15.44:0/2562193468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce809ea10 con 0x7f7ce815e1d0 2024-04-02T13:21:13.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.682+0000 7f7ce7fff640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ce80d2dd0 0x7f7ce80a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:13.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.682+0000 7f7ced5d2640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ce815e1d0 0x7f7ce80a11c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800c3e0 tx=0x7f7cd800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:13.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.682+0000 7f7ce57fa640 1 -- 172.21.15.44:0/2562193468 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd80074e0 con 0x7f7ce815e1d0 2024-04-02T13:21:13.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.682+0000 7f7ce57fa640 1 -- 172.21.15.44:0/2562193468 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd80047e0 con 0x7f7ce815e1d0 2024-04-02T13:21:13.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.682+0000 7f7cef035640 1 -- 172.21.15.44:0/2562193468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce81f7a60 con 0x7f7ce815e1d0 2024-04-02T13:21:13.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.682+0000 7f7ce57fa640 1 -- 172.21.15.44:0/2562193468 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8004bd0 con 0x7f7ce815e1d0 2024-04-02T13:21:13.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.682+0000 7f7cef035640 1 -- 172.21.15.44:0/2562193468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce81f8020 con 0x7f7ce815e1d0 2024-04-02T13:21:13.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.683+0000 7f7cef035640 1 -- 172.21.15.44:0/2562193468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce80684f0 con 0x7f7ce815e1d0 2024-04-02T13:21:13.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.683+0000 7f7ce57fa640 1 -- 172.21.15.44:0/2562193468 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7cd8011070 con 0x7f7ce815e1d0 2024-04-02T13:21:13.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.684+0000 7f7ce57fa640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7cd406c8f0 0x7f7cd406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.684+0000 7f7ce57fa640 1 -- 172.21.15.44:0/2562193468 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f7cd808bad0 con 0x7f7ce815e1d0 2024-04-02T13:21:13.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.684+0000 7f7cecdd1640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7cd406c8f0 0x7f7cd406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.685+0000 7f7cecdd1640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7cd406c8f0 0x7f7cd406edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0036000 tx=0x7f7ce0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:13.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.688+0000 7f7ce57fa640 1 -- 172.21.15.44:0/2562193468 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7cd8059ae0 con 0x7f7ce815e1d0 2024-04-02T13:21:13.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.873+0000 7f7cef035640 1 -- 172.21.15.44:0/2562193468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7ce80d47f0 con 0x7f7ce815e1d0 2024-04-02T13:21:13.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.873+0000 7f7ce57fa640 1 -- 172.21.15.44:0/2562193468 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f7cd80593e0 con 0x7f7ce815e1d0 2024-04-02T13:21:13.875 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.875+0000 7f7cef035640 1 -- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7cd406c8f0 msgr2=0x7f7cd406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.875+0000 7f7cef035640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7cd406c8f0 0x7f7cd406edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0036000 tx=0x7f7ce0030000 comp rx=0 tx=0).stop 2024-04-02T13:21:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.875+0000 7f7cef035640 1 -- 172.21.15.44:0/2562193468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ce815e1d0 msgr2=0x7f7ce80a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.875+0000 7f7cef035640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ce815e1d0 0x7f7ce80a11c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800c3e0 tx=0x7f7cd800c8a0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.875+0000 7f7cef035640 1 -- 172.21.15.44:0/2562193468 shutdown_connections 2024-04-02T13:21:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.875+0000 7f7cef035640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7cd406c8f0 0x7f7cd406edb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.875+0000 7f7cef035640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ce815e1d0 0x7f7ce80a11c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.875+0000 7f7cef035640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ce80d2dd0 0x7f7ce80a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.875+0000 7f7cef035640 1 --2- 172.21.15.44:0/2562193468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ce80cec60 0x7f7ce809e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.875+0000 7f7cef035640 1 -- 172.21.15.44:0/2562193468 >> 172.21.15.44:0/2562193468 conn(0x7f7ce809d3b0 msgr2=0x7f7ce8161d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.875+0000 7f7cef035640 1 -- 172.21.15.44:0/2562193468 shutdown_connections 2024-04-02T13:21:13.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.875+0000 7f7cef035640 1 -- 172.21.15.44:0/2562193468 wait complete. 2024-04-02T13:21:13.887 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:13.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.990+0000 7f8deeed6640 1 Processor -- start 2024-04-02T13:21:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.990+0000 7f8deeed6640 1 -- start start 2024-04-02T13:21:13.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8deeed6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8de81554c0 0x7f8de8071120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8deeed6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8de80d0840 0x7f8de8164c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8deeed6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8de80cdc20 con 0x7f8de80ce9a0 2024-04-02T13:21:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8deeed6640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f8de80cdec0 con 0x7f8de80d0840 2024-04-02T13:21:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8deeed6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8de80ce1a0 con 0x7f8de81554c0 2024-04-02T13:21:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8ded473640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8de80d0840 0x7f8de8164c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8de7fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f8de80ce9a0 0x7f8de80d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54808/0 (socket says 172.21.15.44:54808) 2024-04-02T13:21:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8de7fff640 1 -- 172.21.15.44:0/150083175 learned_addr learned my addr 172.21.15.44:0/150083175 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8decc72640 1 --2- 172.21.15.44:0/150083175 >> v2:172.21.15.69:3300/0 conn(0x7f8de81554c0 0x7f8de8071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8ded473640 1 -- 172.21.15.44:0/150083175 >> v2:172.21.15.69:3300/0 conn(0x7f8de81554c0 msgr2=0x7f8de8071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8ded473640 1 --2- 172.21.15.44:0/150083175 >> v2:172.21.15.69:3300/0 conn(0x7f8de81554c0 0x7f8de8071120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8ded473640 1 -- 172.21.15.44:0/150083175 >> v1:172.21.15.44:6789/0 conn(0x7f8de80ce9a0 legacy=0x7f8de80d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8ded473640 1 -- 172.21.15.44:0/150083175 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de80ce480 con 0x7f8de80d0840 2024-04-02T13:21:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8de77fe640 1 -- 172.21.15.44:0/150083175 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778617648 0 0) 0x7f8de80cdc20 con 0x7f8de80ce9a0 2024-04-02T13:21:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.991+0000 7f8decc72640 1 --2- 172.21.15.44:0/150083175 >> v2:172.21.15.69:3300/0 conn(0x7f8de81554c0 0x7f8de8071120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.992+0000 7f8ded473640 1 --2- 172.21.15.44:0/150083175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8de80d0840 0x7f8de8164c10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8de80af8b0 tx=0x7f8dd802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a77d27a5770f4e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:13.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.992+0000 7f8de77fe640 1 -- 172.21.15.44:0/150083175 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd8035020 con 0x7f8de80d0840 2024-04-02T13:21:13.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.992+0000 7f8de77fe640 1 -- 172.21.15.44:0/150083175 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dd8031830 con 0x7f8de80d0840 2024-04-02T13:21:13.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.992+0000 7f8de77fe640 1 -- 172.21.15.44:0/150083175 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd8031c00 con 0x7f8de80d0840 2024-04-02T13:21:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.993+0000 7f8deeed6640 1 -- 172.21.15.44:0/150083175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8de80d0840 msgr2=0x7f8de8164c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.993+0000 7f8deeed6640 1 --2- 172.21.15.44:0/150083175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8de80d0840 0x7f8de8164c10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8de80af8b0 tx=0x7f8dd802b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.993+0000 7f8deeed6640 1 -- 172.21.15.44:0/150083175 shutdown_connections 2024-04-02T13:21:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.993+0000 7f8deeed6640 1 --2- 172.21.15.44:0/150083175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8de80d0840 0x7f8de8164c10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.993+0000 7f8deeed6640 1 --2- 172.21.15.44:0/150083175 >> v2:172.21.15.69:3300/0 conn(0x7f8de81554c0 0x7f8de8071120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.993+0000 7f8deeed6640 1 -- 172.21.15.44:0/150083175 >> 172.21.15.44:0/150083175 conn(0x7f8de806b920 msgr2=0x7f8de806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.993+0000 7f8deeed6640 1 -- 172.21.15.44:0/150083175 shutdown_connections 2024-04-02T13:21:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.993+0000 7f8deeed6640 1 -- 172.21.15.44:0/150083175 wait complete. 2024-04-02T13:21:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.994+0000 7f8deeed6640 1 Processor -- start 2024-04-02T13:21:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.994+0000 7f8deeed6640 1 -- start start 2024-04-02T13:21:13.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8deeed6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8de80ce9a0 0x7f8de814f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8deeed6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8de80d0840 0x7f8de815a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8deeed6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8de81554c0 0x7f8de815c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:13.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8deeed6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8de815c290 con 0x7f8de80ce9a0 2024-04-02T13:21:13.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8deeed6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8de8071be0 con 0x7f8de80d0840 2024-04-02T13:21:13.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8deeed6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8de815abf0 con 0x7f8de81554c0 2024-04-02T13:21:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8decc72640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8de80ce9a0 0x7f8de814f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8decc72640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8de80ce9a0 0x7f8de814f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45324/0 (socket says 172.21.15.44:45324) 2024-04-02T13:21:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8decc72640 1 -- 172.21.15.44:0/51329045 learned_addr learned my addr 172.21.15.44:0/51329045 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8de7fff640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8de80d0840 0x7f8de815a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8ded473640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8de81554c0 0x7f8de815c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8decc72640 1 -- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8de81554c0 msgr2=0x7f8de815c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8decc72640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8de81554c0 0x7f8de815c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8decc72640 1 -- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8de80d0840 msgr2=0x7f8de815a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8decc72640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8de80d0840 0x7f8de815a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8decc72640 1 -- 172.21.15.44:0/51329045 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de8158ff0 con 0x7f8de80ce9a0 2024-04-02T13:21:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.995+0000 7f8ded473640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8de81554c0 0x7f8de815c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.996+0000 7f8decc72640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8de80ce9a0 0x7f8de814f720 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009fa0 tx=0x7f8de000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.996+0000 7f8de57fa640 1 -- 172.21.15.44:0/51329045 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de001a070 con 0x7f8de80ce9a0 2024-04-02T13:21:14.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.996+0000 7f8de57fa640 1 -- 172.21.15.44:0/51329045 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de0003bb0 con 0x7f8de80ce9a0 2024-04-02T13:21:14.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.996+0000 7f8de57fa640 1 -- 172.21.15.44:0/51329045 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0004030 con 0x7f8de80ce9a0 2024-04-02T13:21:14.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.996+0000 7f8deeed6640 1 -- 172.21.15.44:0/51329045 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de81592d0 con 0x7f8de80ce9a0 2024-04-02T13:21:14.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.996+0000 7f8deeed6640 1 -- 172.21.15.44:0/51329045 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de81586e0 con 0x7f8de80ce9a0 2024-04-02T13:21:14.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.997+0000 7f8deeed6640 1 -- 172.21.15.44:0/51329045 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de80cdb80 con 0x7f8de80ce9a0 2024-04-02T13:21:14.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.997+0000 7f8de57fa640 1 -- 172.21.15.44:0/51329045 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8de001d070 con 0x7f8de80ce9a0 2024-04-02T13:21:14.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.998+0000 7f8de57fa640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8dd406c8f0 0x7f8dd406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:14.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.998+0000 7f8de57fa640 1 -- 172.21.15.44:0/51329045 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f8de008c140 con 0x7f8de80ce9a0 2024-04-02T13:21:14.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.998+0000 7f8de7fff640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8dd406c8f0 0x7f8dd406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:13.998+0000 7f8de7fff640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8dd406c8f0 0x7f8dd406edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8de815d140 tx=0x7f8ddc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:14.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.002+0000 7f8de57fa640 1 -- 172.21.15.44:0/51329045 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8de80cdb80 con 0x7f8de80ce9a0 2024-04-02T13:21:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.185+0000 7f8deeed6640 1 -- 172.21.15.44:0/51329045 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8de80684f0 con 0x7f8de80ce9a0 2024-04-02T13:21:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.185+0000 7f8de57fa640 1 -- 172.21.15.44:0/51329045 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f8de0010070 con 0x7f8de80ce9a0 2024-04-02T13:21:14.187 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:14.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.187+0000 7f8deeed6640 1 -- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8dd406c8f0 msgr2=0x7f8dd406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.187+0000 7f8deeed6640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8dd406c8f0 0x7f8dd406edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8de815d140 tx=0x7f8ddc009040 comp rx=0 tx=0).stop 2024-04-02T13:21:14.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.187+0000 7f8deeed6640 1 -- 172.21.15.44:0/51329045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8de80ce9a0 msgr2=0x7f8de814f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.187+0000 7f8deeed6640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8de80ce9a0 0x7f8de814f720 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009fa0 tx=0x7f8de000d560 comp rx=0 tx=0).stop 2024-04-02T13:21:14.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.187+0000 7f8deeed6640 1 -- 172.21.15.44:0/51329045 shutdown_connections 2024-04-02T13:21:14.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.187+0000 7f8deeed6640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8dd406c8f0 0x7f8dd406edb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.187+0000 7f8deeed6640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8de81554c0 0x7f8de815c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.187+0000 7f8deeed6640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8de80d0840 0x7f8de815a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.187+0000 7f8deeed6640 1 --2- 172.21.15.44:0/51329045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8de80ce9a0 0x7f8de814f720 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.187+0000 7f8deeed6640 1 -- 172.21.15.44:0/51329045 >> 172.21.15.44:0/51329045 conn(0x7f8de806b920 msgr2=0x7f8de80b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:14.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.187+0000 7f8deeed6640 1 -- 172.21.15.44:0/51329045 shutdown_connections 2024-04-02T13:21:14.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.187+0000 7f8deeed6640 1 -- 172.21.15.44:0/51329045 wait complete. 2024-04-02T13:21:14.202 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:21:14.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.303+0000 7f8775cba640 1 Processor -- start 2024-04-02T13:21:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.303+0000 7f8775cba640 1 -- start start 2024-04-02T13:21:14.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.303+0000 7f8775cba640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f87700b5200 0x7f87700d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:14.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.303+0000 7f8775cba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87700d4d70 con 0x7f87700d28f0 2024-04-02T13:21:14.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.303+0000 7f8775cba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87700d2030 con 0x7f87700b47b0 2024-04-02T13:21:14.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.303+0000 7f8775cba640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f87700d5050 con 0x7f87700b5200 2024-04-02T13:21:14.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.304+0000 7f876ffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f87700b5200 0x7f87700d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.304+0000 7f876f7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f87700d28f0 0x7f87700b40a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54824/0 (socket says 172.21.15.44:54824) 2024-04-02T13:21:14.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.304+0000 7f876f7fe640 1 -- 172.21.15.44:0/3477293461 learned_addr learned my addr 172.21.15.44:0/3477293461 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:14.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.304+0000 7f876ffff640 1 -- 172.21.15.44:0/3477293461 >> v1:172.21.15.69:6789/0 conn(0x7f87700b47b0 legacy=0x7f87700b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.304+0000 7f876ffff640 1 -- 172.21.15.44:0/3477293461 >> v1:172.21.15.44:6789/0 conn(0x7f87700d28f0 legacy=0x7f87700b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.304+0000 7f876ffff640 1 -- 172.21.15.44:0/3477293461 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87700b4cb0 con 0x7f87700b5200 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.305+0000 7f876ffff640 1 --2- 172.21.15.44:0/3477293461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87700b5200 0x7f87700d2cd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8768009d90 tx=0x7f876802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d76f7b08524a30c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.305+0000 7f876e7fc640 1 -- 172.21.15.44:0/3477293461 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8768039070 con 0x7f87700b5200 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.305+0000 7f876e7fc640 1 -- 172.21.15.44:0/3477293461 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f876802cb40 con 0x7f87700b5200 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.305+0000 7f876e7fc640 1 -- 172.21.15.44:0/3477293461 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8768034440 con 0x7f87700b5200 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.305+0000 7f8775cba640 1 -- 172.21.15.44:0/3477293461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87700b5200 msgr2=0x7f87700d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.305+0000 7f8775cba640 1 --2- 172.21.15.44:0/3477293461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87700b5200 0x7f87700d2cd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8768009d90 tx=0x7f876802bd90 comp rx=0 tx=0).stop 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.305+0000 7f8775cba640 1 -- 172.21.15.44:0/3477293461 shutdown_connections 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.305+0000 7f8775cba640 1 --2- 172.21.15.44:0/3477293461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87700b5200 0x7f87700d2cd0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.305+0000 7f8775cba640 1 -- 172.21.15.44:0/3477293461 >> 172.21.15.44:0/3477293461 conn(0x7f87700d4120 msgr2=0x7f87700d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.305+0000 7f8775cba640 1 -- 172.21.15.44:0/3477293461 shutdown_connections 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.305+0000 7f8775cba640 1 -- 172.21.15.44:0/3477293461 wait complete. 2024-04-02T13:21:14.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.306+0000 7f8775cba640 1 Processor -- start 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.307+0000 7f8775cba640 1 -- start start 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.307+0000 7f8775cba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87700b47b0 0x7f8770153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.307+0000 7f8775cba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87700b5200 0x7f87700b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.307+0000 7f8775cba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87700d28f0 0x7f87700b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.307+0000 7f8775cba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f87700d23a0 con 0x7f87700b47b0 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.307+0000 7f8775cba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f87700d14f0 con 0x7f87700d28f0 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.307+0000 7f8775cba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f87700b5020 con 0x7f87700b5200 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.307+0000 7f876f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87700b47b0 0x7f8770153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.307+0000 7f876f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87700b47b0 0x7f8770153f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45342/0 (socket says 172.21.15.44:45342) 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.307+0000 7f876f7fe640 1 -- 172.21.15.44:0/1500599828 learned_addr learned my addr 172.21.15.44:0/1500599828 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f876ffff640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87700d28f0 0x7f87700b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f876effd640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87700b5200 0x7f87700b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f876f7fe640 1 -- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87700b5200 msgr2=0x7f87700b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f876f7fe640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87700b5200 0x7f87700b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f876f7fe640 1 -- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87700d28f0 msgr2=0x7f87700b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f876f7fe640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87700d28f0 0x7f87700b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f876f7fe640 1 -- 172.21.15.44:0/1500599828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87700cd1c0 con 0x7f87700b47b0 2024-04-02T13:21:14.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f876ffff640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87700d28f0 0x7f87700b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:14.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f876f7fe640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87700b47b0 0x7f8770153f40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f876000c3c0 tx=0x7f876000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:14.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f8757fff640 1 -- 172.21.15.44:0/1500599828 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760017070 con 0x7f87700b47b0 2024-04-02T13:21:14.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f8757fff640 1 -- 172.21.15.44:0/1500599828 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8760005ce0 con 0x7f87700b47b0 2024-04-02T13:21:14.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f8775cba640 1 -- 172.21.15.44:0/1500599828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87700cd4a0 con 0x7f87700b47b0 2024-04-02T13:21:14.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f8757fff640 1 -- 172.21.15.44:0/1500599828 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87600072f0 con 0x7f87700b47b0 2024-04-02T13:21:14.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.308+0000 7f8775cba640 1 -- 172.21.15.44:0/1500599828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87700cda60 con 0x7f87700b47b0 2024-04-02T13:21:14.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.309+0000 7f8775cba640 1 -- 172.21.15.44:0/1500599828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f877006b7c0 con 0x7f87700b47b0 2024-04-02T13:21:14.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.310+0000 7f8757fff640 1 -- 172.21.15.44:0/1500599828 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f876001a070 con 0x7f87700b47b0 2024-04-02T13:21:14.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.310+0000 7f8757fff640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f875c06c8f0 0x7f875c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:14.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.310+0000 7f8757fff640 1 -- 172.21.15.44:0/1500599828 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f876008b320 con 0x7f87700b47b0 2024-04-02T13:21:14.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.310+0000 7f876effd640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f875c06c8f0 0x7f875c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.311+0000 7f876effd640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f875c06c8f0 0x7f875c06edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f87701f7b00 tx=0x7f8764008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:14.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.314+0000 7f8757fff640 1 -- 172.21.15.44:0/1500599828 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f877006b7c0 con 0x7f87700b47b0 2024-04-02T13:21:14.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.511+0000 7f8775cba640 1 -- 172.21.15.44:0/1500599828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f87700b6aa0 con 0x7f87700b47b0 2024-04-02T13:21:14.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.511+0000 7f8757fff640 1 -- 172.21.15.44:0/1500599828 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f87700b6aa0 con 0x7f87700b47b0 2024-04-02T13:21:14.512 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:21:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.513+0000 7f8775cba640 1 -- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f875c06c8f0 msgr2=0x7f875c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.513+0000 7f8775cba640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f875c06c8f0 0x7f875c06edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f87701f7b00 tx=0x7f8764008040 comp rx=0 tx=0).stop 2024-04-02T13:21:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.513+0000 7f8775cba640 1 -- 172.21.15.44:0/1500599828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87700b47b0 msgr2=0x7f8770153f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.513+0000 7f8775cba640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87700b47b0 0x7f8770153f40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f876000c3c0 tx=0x7f876000c880 comp rx=0 tx=0).stop 2024-04-02T13:21:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.513+0000 7f8775cba640 1 -- 172.21.15.44:0/1500599828 shutdown_connections 2024-04-02T13:21:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.513+0000 7f8775cba640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f875c06c8f0 0x7f875c06edb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.513+0000 7f8775cba640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87700d28f0 0x7f87700b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.513+0000 7f8775cba640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87700b5200 0x7f87700b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.513+0000 7f8775cba640 1 --2- 172.21.15.44:0/1500599828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87700b47b0 0x7f8770153f40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.513+0000 7f8775cba640 1 -- 172.21.15.44:0/1500599828 >> 172.21.15.44:0/1500599828 conn(0x7f87700d4120 msgr2=0x7f87700cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.513+0000 7f8775cba640 1 -- 172.21.15.44:0/1500599828 shutdown_connections 2024-04-02T13:21:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.513+0000 7f8775cba640 1 -- 172.21.15.44:0/1500599828 wait complete. 2024-04-02T13:21:14.525 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:21:14.525 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:14.540 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:14.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.680+0000 7f1fb102c640 1 Processor -- start 2024-04-02T13:21:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.680+0000 7f1fb102c640 1 -- start start 2024-04-02T13:21:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.681+0000 7f1fb102c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fac06d700 0x7f1fac15e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.681+0000 7f1fb102c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fac15fb50 con 0x7f1fac15ee70 2024-04-02T13:21:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.681+0000 7f1fb102c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fac15f320 con 0x7f1fac15e550 2024-04-02T13:21:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.681+0000 7f1fb102c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1fac06ca90 con 0x7f1fac06d700 2024-04-02T13:21:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.681+0000 7f1fab59e640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f1fac15ee70 0x7f1fac0d0ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48834/0 (socket says 172.21.15.44:48834) 2024-04-02T13:21:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.681+0000 7f1fab59e640 1 -- 172.21.15.44:0/1648403066 learned_addr learned my addr 172.21.15.44:0/1648403066 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.681+0000 7f1fa3fff640 1 --2- 172.21.15.44:0/1648403066 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fac06d700 0x7f1fac15e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.681+0000 7f1faa59c640 1 -- 172.21.15.44:0/1648403066 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2733994042 0 0) 0x7f1fac15fb50 con 0x7f1fac15ee70 2024-04-02T13:21:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.681+0000 7f1faa59c640 1 -- 172.21.15.44:0/1648403066 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f98003520 con 0x7f1fac15ee70 2024-04-02T13:21:14.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.681+0000 7f1faa59c640 1 -- 172.21.15.44:0/1648403066 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1217509658 0 0) 0x7f1fac15f320 con 0x7f1fac15e550 2024-04-02T13:21:14.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.681+0000 7f1fa3fff640 1 -- 172.21.15.44:0/1648403066 >> v1:172.21.15.69:6789/0 conn(0x7f1fac15e550 legacy=0x7f1fac06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.681+0000 7f1fa3fff640 1 -- 172.21.15.44:0/1648403066 >> v1:172.21.15.44:6789/0 conn(0x7f1fac15ee70 legacy=0x7f1fac0d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.681+0000 7f1fa3fff640 1 -- 172.21.15.44:0/1648403066 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fac15f590 con 0x7f1fac06d700 2024-04-02T13:21:14.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.682+0000 7f1fa3fff640 1 --2- 172.21.15.44:0/1648403066 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fac06d700 0x7f1fac15e930 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f940097d0 tx=0x7f1f9402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acb7077e95c0a46b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:14.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.682+0000 7f1faa59c640 1 -- 172.21.15.44:0/1648403066 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f94034070 con 0x7f1fac06d700 2024-04-02T13:21:14.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.682+0000 7f1faa59c640 1 -- 172.21.15.44:0/1648403066 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f94030880 con 0x7f1fac06d700 2024-04-02T13:21:14.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.682+0000 7f1faa59c640 1 -- 172.21.15.44:0/1648403066 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f94030c30 con 0x7f1fac06d700 2024-04-02T13:21:14.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.683+0000 7f1fb102c640 1 -- 172.21.15.44:0/1648403066 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fac06d700 msgr2=0x7f1fac15e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.683+0000 7f1fb102c640 1 --2- 172.21.15.44:0/1648403066 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fac06d700 0x7f1fac15e930 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f940097d0 tx=0x7f1f9402b9c0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.683+0000 7f1fb102c640 1 -- 172.21.15.44:0/1648403066 shutdown_connections 2024-04-02T13:21:14.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.683+0000 7f1fb102c640 1 --2- 172.21.15.44:0/1648403066 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fac06d700 0x7f1fac15e930 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.683+0000 7f1fb102c640 1 -- 172.21.15.44:0/1648403066 >> 172.21.15.44:0/1648403066 conn(0x7f1fac09aa20 msgr2=0x7f1fac09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:14.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.683+0000 7f1fb102c640 1 -- 172.21.15.44:0/1648403066 shutdown_connections 2024-04-02T13:21:14.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.683+0000 7f1fb102c640 1 -- 172.21.15.44:0/1648403066 wait complete. 2024-04-02T13:21:14.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.684+0000 7f1fb102c640 1 Processor -- start 2024-04-02T13:21:14.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.684+0000 7f1fb102c640 1 -- start start 2024-04-02T13:21:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1fb102c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fac06d700 0x7f1fac0c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1fb102c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fac15e550 0x7f1fac0caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1fb102c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fac15ee70 0x7f1fac0b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1fb102c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac0d0590 con 0x7f1fac06d700 2024-04-02T13:21:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1fb102c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac0d02b0 con 0x7f1fac15e550 2024-04-02T13:21:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1fb102c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1fac0cc4b0 con 0x7f1fac15ee70 2024-04-02T13:21:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1faad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fac06d700 0x7f1fac0c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1faad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fac06d700 0x7f1fac0c9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45670/0 (socket says 172.21.15.44:45670) 2024-04-02T13:21:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1faad9d640 1 -- 172.21.15.44:0/1128276433 learned_addr learned my addr 172.21.15.44:0/1128276433 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1fab59e640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fac15ee70 0x7f1fac0b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1fa3fff640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fac15e550 0x7f1fac0caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1faad9d640 1 -- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fac15ee70 msgr2=0x7f1fac0b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1faad9d640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fac15ee70 0x7f1fac0b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1faad9d640 1 -- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fac15e550 msgr2=0x7f1fac0caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1faad9d640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fac15e550 0x7f1fac0caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1faad9d640 1 -- 172.21.15.44:0/1128276433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fac0cb1b0 con 0x7f1fac06d700 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1fab59e640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fac15ee70 0x7f1fac0b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1fa3fff640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fac15e550 0x7f1fac0caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.685+0000 7f1faad9d640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fac06d700 0x7f1fac0c9ad0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c00c410 tx=0x7f1f9c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.686+0000 7f1fa37fe640 1 -- 172.21.15.44:0/1128276433 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c017070 con 0x7f1fac06d700 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.686+0000 7f1fa37fe640 1 -- 172.21.15.44:0/1128276433 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f9c0035f0 con 0x7f1fac06d700 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.686+0000 7f1fb102c640 1 -- 172.21.15.44:0/1128276433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fac0cb490 con 0x7f1fac06d700 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.686+0000 7f1fa37fe640 1 -- 172.21.15.44:0/1128276433 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c0039e0 con 0x7f1fac06d700 2024-04-02T13:21:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.686+0000 7f1fb102c640 1 -- 172.21.15.44:0/1128276433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fac0cba50 con 0x7f1fac06d700 2024-04-02T13:21:14.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.687+0000 7f1fb102c640 1 -- 172.21.15.44:0/1128276433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fac0684f0 con 0x7f1fac06d700 2024-04-02T13:21:14.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.687+0000 7f1fa37fe640 1 -- 172.21.15.44:0/1128276433 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1f9c01a070 con 0x7f1fac06d700 2024-04-02T13:21:14.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.688+0000 7f1fa37fe640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1f8c06c8f0 0x7f1f8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:14.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.688+0000 7f1fa37fe640 1 -- 172.21.15.44:0/1128276433 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f1f9c08b2f0 con 0x7f1fac06d700 2024-04-02T13:21:14.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.688+0000 7f1fa3fff640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1f8c06c8f0 0x7f1f8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:14.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.688+0000 7f1fa3fff640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1f8c06c8f0 0x7f1f8c06edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1f94036000 tx=0x7f1f94008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:14.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.692+0000 7f1fa37fe640 1 -- 172.21.15.44:0/1128276433 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1f9c059390 con 0x7f1fac06d700 2024-04-02T13:21:14.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.878+0000 7f1fb102c640 1 -- 172.21.15.44:0/1128276433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1fac15ead0 con 0x7f1fac06d700 2024-04-02T13:21:14.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.879+0000 7f1fa37fe640 1 -- 172.21.15.44:0/1128276433 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18899 (secure 0 0 0) 0x7f1f9c058c90 con 0x7f1fac06d700 2024-04-02T13:21:14.882 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:14.882 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:08.061253+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:14.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.881+0000 7f1fb102c640 1 -- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1f8c06c8f0 msgr2=0x7f1f8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.881+0000 7f1fb102c640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1f8c06c8f0 0x7f1f8c06edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1f94036000 tx=0x7f1f94008030 comp rx=0 tx=0).stop 2024-04-02T13:21:14.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.881+0000 7f1fb102c640 1 -- 172.21.15.44:0/1128276433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fac06d700 msgr2=0x7f1fac0c9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:14.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.881+0000 7f1fb102c640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fac06d700 0x7f1fac0c9ad0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c00c410 tx=0x7f1f9c00c8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.881+0000 7f1fb102c640 1 -- 172.21.15.44:0/1128276433 shutdown_connections 2024-04-02T13:21:14.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.881+0000 7f1fb102c640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1f8c06c8f0 0x7f1f8c06edb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.881+0000 7f1fb102c640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fac15ee70 0x7f1fac0b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.881+0000 7f1fb102c640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fac15e550 0x7f1fac0caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.881+0000 7f1fb102c640 1 --2- 172.21.15.44:0/1128276433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fac06d700 0x7f1fac0c9ad0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:14.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.881+0000 7f1fb102c640 1 -- 172.21.15.44:0/1128276433 >> 172.21.15.44:0/1128276433 conn(0x7f1fac09aa20 msgr2=0x7f1fac0cd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:14.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.881+0000 7f1fb102c640 1 -- 172.21.15.44:0/1128276433 shutdown_connections 2024-04-02T13:21:14.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.881+0000 7f1fb102c640 1 -- 172.21.15.44:0/1128276433 wait complete. 2024-04-02T13:21:14.894 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:14.894 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:14.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.997+0000 7f9d50c0f640 1 Processor -- start 2024-04-02T13:21:14.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.997+0000 7f9d50c0f640 1 -- start start 2024-04-02T13:21:15.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.998+0000 7f9d50c0f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9d4c15a940 0x7f9d4c0ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.998+0000 7f9d50c0f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9d4c0cc640 0x7f9d4c0cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.998+0000 7f9d50c0f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d4c0ccf90 con 0x7f9d4c071200 2024-04-02T13:21:15.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.998+0000 7f9d50c0f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f9d4c06ef80 con 0x7f9d4c0cc640 2024-04-02T13:21:15.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.998+0000 7f9d50c0f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9d4c0ce020 con 0x7f9d4c15a940 2024-04-02T13:21:15.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.999+0000 7f9d4ad9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9d4c0cc640 0x7f9d4c0cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.999+0000 7f9d4ad9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9d4c0cc640 0x7f9d4c0cca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45682/0 (socket says 172.21.15.44:45682) 2024-04-02T13:21:15.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.999+0000 7f9d4ad9d640 1 -- 172.21.15.44:0/1189548919 learned_addr learned my addr 172.21.15.44:0/1189548919 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:15.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.999+0000 7f9d4a59c640 1 --2- 172.21.15.44:0/1189548919 >> v2:172.21.15.69:3300/0 conn(0x7f9d4c15a940 0x7f9d4c0ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.999+0000 7f9d4ad9d640 1 -- 172.21.15.44:0/1189548919 >> v2:172.21.15.69:3300/0 conn(0x7f9d4c15a940 msgr2=0x7f9d4c0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:15.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.999+0000 7f9d4ad9d640 1 --2- 172.21.15.44:0/1189548919 >> v2:172.21.15.69:3300/0 conn(0x7f9d4c15a940 0x7f9d4c0ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.999+0000 7f9d4ad9d640 1 -- 172.21.15.44:0/1189548919 >> v1:172.21.15.69:6789/0 conn(0x7f9d4c071200 legacy=0x7f9d4c06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:15.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.999+0000 7f9d4ad9d640 1 -- 172.21.15.44:0/1189548919 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4c0ce9f0 con 0x7f9d4c0cc640 2024-04-02T13:21:15.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:14.999+0000 7f9d4a59c640 1 --2- 172.21.15.44:0/1189548919 >> v2:172.21.15.69:3300/0 conn(0x7f9d4c15a940 0x7f9d4c0ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:15.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.000+0000 7f9d4ad9d640 1 --2- 172.21.15.44:0/1189548919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d4c0cc640 0x7f9d4c0cca50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0d0d80 tx=0x7f9d3c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5915aea858a6182f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:15.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.000+0000 7f9d4959a640 1 -- 172.21.15.44:0/1189548919 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c02c9f0 con 0x7f9d4c0cc640 2024-04-02T13:21:15.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.000+0000 7f9d4959a640 1 -- 172.21.15.44:0/1189548919 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d3c02cb90 con 0x7f9d4c0cc640 2024-04-02T13:21:15.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.000+0000 7f9d4959a640 1 -- 172.21.15.44:0/1189548919 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c0343f0 con 0x7f9d4c0cc640 2024-04-02T13:21:15.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.000+0000 7f9d50c0f640 1 -- 172.21.15.44:0/1189548919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d4c0cc640 msgr2=0x7f9d4c0cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.000+0000 7f9d50c0f640 1 --2- 172.21.15.44:0/1189548919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d4c0cc640 0x7f9d4c0cca50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0d0d80 tx=0x7f9d3c02be50 comp rx=0 tx=0).stop 2024-04-02T13:21:15.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.000+0000 7f9d50c0f640 1 -- 172.21.15.44:0/1189548919 shutdown_connections 2024-04-02T13:21:15.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.000+0000 7f9d50c0f640 1 --2- 172.21.15.44:0/1189548919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d4c0cc640 0x7f9d4c0cca50 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.000+0000 7f9d50c0f640 1 --2- 172.21.15.44:0/1189548919 >> v2:172.21.15.69:3300/0 conn(0x7f9d4c15a940 0x7f9d4c0ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.000+0000 7f9d50c0f640 1 -- 172.21.15.44:0/1189548919 >> 172.21.15.44:0/1189548919 conn(0x7f9d4c06b920 msgr2=0x7f9d4c06bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:15.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.000+0000 7f9d50c0f640 1 -- 172.21.15.44:0/1189548919 shutdown_connections 2024-04-02T13:21:15.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.001+0000 7f9d50c0f640 1 -- 172.21.15.44:0/1189548919 wait complete. 2024-04-02T13:21:15.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.002+0000 7f9d50c0f640 1 Processor -- start 2024-04-02T13:21:15.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.002+0000 7f9d50c0f640 1 -- start start 2024-04-02T13:21:15.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.002+0000 7f9d50c0f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d4c071200 0x7f9d4c148720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d50c0f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d4c0cc640 0x7f9d4c09aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d50c0f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d4c15a940 0x7f9d4c09c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d50c0f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c0cdcb0 con 0x7f9d4c0cc640 2024-04-02T13:21:15.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d50c0f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c06eca0 con 0x7f9d4c15a940 2024-04-02T13:21:15.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d50c0f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9d4c0acd50 con 0x7f9d4c071200 2024-04-02T13:21:15.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d49d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d4c0cc640 0x7f9d4c09aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d49d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d4c0cc640 0x7f9d4c09aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45694/0 (socket says 172.21.15.44:45694) 2024-04-02T13:21:15.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d49d9b640 1 -- 172.21.15.44:0/3498839215 learned_addr learned my addr 172.21.15.44:0/3498839215 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:15.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d4a59c640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d4c071200 0x7f9d4c148720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d4ad9d640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d4c15a940 0x7f9d4c09c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d49d9b640 1 -- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d4c071200 msgr2=0x7f9d4c148720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d49d9b640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d4c071200 0x7f9d4c148720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d49d9b640 1 -- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d4c15a940 msgr2=0x7f9d4c09c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d49d9b640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d4c15a940 0x7f9d4c09c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d49d9b640 1 -- 172.21.15.44:0/3498839215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4c148c60 con 0x7f9d4c0cc640 2024-04-02T13:21:15.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d4a59c640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d4c071200 0x7f9d4c148720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:15.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d4ad9d640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d4c15a940 0x7f9d4c09c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:15.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d49d9b640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d4c0cc640 0x7f9d4c09aa20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400c430 tx=0x7f9d3400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:15.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d42ffd640 1 -- 172.21.15.44:0/3498839215 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d34011790 con 0x7f9d4c0cc640 2024-04-02T13:21:15.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d42ffd640 1 -- 172.21.15.44:0/3498839215 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d34017070 con 0x7f9d4c0cc640 2024-04-02T13:21:15.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d50c0f640 1 -- 172.21.15.44:0/3498839215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4c09b250 con 0x7f9d4c0cc640 2024-04-02T13:21:15.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d42ffd640 1 -- 172.21.15.44:0/3498839215 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d34011bb0 con 0x7f9d4c0cc640 2024-04-02T13:21:15.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.003+0000 7f9d50c0f640 1 -- 172.21.15.44:0/3498839215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d4c09b810 con 0x7f9d4c0cc640 2024-04-02T13:21:15.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.004+0000 7f9d50c0f640 1 -- 172.21.15.44:0/3498839215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d4c0684f0 con 0x7f9d4c0cc640 2024-04-02T13:21:15.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.005+0000 7f9d42ffd640 1 -- 172.21.15.44:0/3498839215 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9d3401c460 con 0x7f9d4c0cc640 2024-04-02T13:21:15.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.005+0000 7f9d42ffd640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9d2c06c8f0 0x7f9d2c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.005+0000 7f9d42ffd640 1 -- 172.21.15.44:0/3498839215 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f9d3408bd40 con 0x7f9d4c0cc640 2024-04-02T13:21:15.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.005+0000 7f9d4a59c640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9d2c06c8f0 0x7f9d2c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.006+0000 7f9d4a59c640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9d2c06c8f0 0x7f9d2c06edb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0cc0c0 tx=0x7f9d44008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:15.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.010+0000 7f9d42ffd640 1 -- 172.21.15.44:0/3498839215 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9d4c148c60 con 0x7f9d4c0cc640 2024-04-02T13:21:15.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.194+0000 7f9d50c0f640 1 -- 172.21.15.44:0/3498839215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9d4c073250 con 0x7f9d4c0cc640 2024-04-02T13:21:15.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.194+0000 7f9d42ffd640 1 -- 172.21.15.44:0/3498839215 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f9d34059de0 con 0x7f9d4c0cc640 2024-04-02T13:21:15.196 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:15.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.196+0000 7f9d50c0f640 1 -- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9d2c06c8f0 msgr2=0x7f9d2c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.196+0000 7f9d50c0f640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9d2c06c8f0 0x7f9d2c06edb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0cc0c0 tx=0x7f9d44008040 comp rx=0 tx=0).stop 2024-04-02T13:21:15.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.196+0000 7f9d50c0f640 1 -- 172.21.15.44:0/3498839215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d4c0cc640 msgr2=0x7f9d4c09aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.196+0000 7f9d50c0f640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d4c0cc640 0x7f9d4c09aa20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400c430 tx=0x7f9d3400c8f0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.196+0000 7f9d50c0f640 1 -- 172.21.15.44:0/3498839215 shutdown_connections 2024-04-02T13:21:15.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.196+0000 7f9d50c0f640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9d2c06c8f0 0x7f9d2c06edb0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.196+0000 7f9d50c0f640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d4c15a940 0x7f9d4c09c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.196+0000 7f9d50c0f640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d4c0cc640 0x7f9d4c09aa20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.196+0000 7f9d50c0f640 1 --2- 172.21.15.44:0/3498839215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d4c071200 0x7f9d4c148720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.196+0000 7f9d50c0f640 1 -- 172.21.15.44:0/3498839215 >> 172.21.15.44:0/3498839215 conn(0x7f9d4c06b920 msgr2=0x7f9d4c06bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:15.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.196+0000 7f9d50c0f640 1 -- 172.21.15.44:0/3498839215 shutdown_connections 2024-04-02T13:21:15.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.196+0000 7f9d50c0f640 1 -- 172.21.15.44:0/3498839215 wait complete. 2024-04-02T13:21:15.208 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:15.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.311+0000 7f5cf521b640 1 Processor -- start 2024-04-02T13:21:15.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.311+0000 7f5cf521b640 1 -- start start 2024-04-02T13:21:15.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.312+0000 7f5cf521b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cf015c260 0x7f5cf00b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.312+0000 7f5cf521b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5cf00b3e80 0x7f5cf00ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.312+0000 7f5cf521b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf00cd4c0 con 0x7f5cf00cd030 2024-04-02T13:21:15.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.312+0000 7f5cf521b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f5cf00b42e0 con 0x7f5cf00b3e80 2024-04-02T13:21:15.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.312+0000 7f5cf521b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf00cd760 con 0x7f5cf015c260 2024-04-02T13:21:15.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.312+0000 7f5ce659c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5cf00b3e80 0x7f5cf00ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.312+0000 7f5ce659c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5cf00b3e80 0x7f5cf00ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45704/0 (socket says 172.21.15.44:45704) 2024-04-02T13:21:15.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.312+0000 7f5ce659c640 1 -- 172.21.15.44:0/3207919324 learned_addr learned my addr 172.21.15.44:0/3207919324 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:15.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.312+0000 7f5ceed9d640 1 --2- 172.21.15.44:0/3207919324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cf015c260 0x7f5cf00b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.312+0000 7f5ce659c640 1 -- 172.21.15.44:0/3207919324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cf015c260 msgr2=0x7f5cf00b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:15.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.312+0000 7f5ce659c640 1 --2- 172.21.15.44:0/3207919324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cf015c260 0x7f5cf00b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.312+0000 7f5ce659c640 1 -- 172.21.15.44:0/3207919324 >> v1:172.21.15.69:6789/0 conn(0x7f5cf00cd030 legacy=0x7f5cf00ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.312+0000 7f5ce659c640 1 -- 172.21.15.44:0/3207919324 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf00cda40 con 0x7f5cf00b3e80 2024-04-02T13:21:15.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.313+0000 7f5ce659c640 1 --2- 172.21.15.44:0/3207919324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cf00b3e80 0x7f5cf00ccaf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009870 tx=0x7f5cd802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f63cf472628681 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:15.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.313+0000 7f5cee59c640 1 -- 172.21.15.44:0/3207919324 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8034070 con 0x7f5cf00b3e80 2024-04-02T13:21:15.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.313+0000 7f5cee59c640 1 -- 172.21.15.44:0/3207919324 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cd8030840 con 0x7f5cf00b3e80 2024-04-02T13:21:15.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.313+0000 7f5cee59c640 1 -- 172.21.15.44:0/3207919324 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8030c10 con 0x7f5cf00b3e80 2024-04-02T13:21:15.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.313+0000 7f5cf521b640 1 -- 172.21.15.44:0/3207919324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cf00b3e80 msgr2=0x7f5cf00ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.313+0000 7f5cf521b640 1 --2- 172.21.15.44:0/3207919324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cf00b3e80 0x7f5cf00ccaf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009870 tx=0x7f5cd802b980 comp rx=0 tx=0).stop 2024-04-02T13:21:15.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.313+0000 7f5cf521b640 1 -- 172.21.15.44:0/3207919324 shutdown_connections 2024-04-02T13:21:15.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.313+0000 7f5cf521b640 1 --2- 172.21.15.44:0/3207919324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cf00b3e80 0x7f5cf00ccaf0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.313+0000 7f5cf521b640 1 --2- 172.21.15.44:0/3207919324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cf015c260 0x7f5cf00b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.313+0000 7f5cf521b640 1 -- 172.21.15.44:0/3207919324 >> 172.21.15.44:0/3207919324 conn(0x7f5cf009aa20 msgr2=0x7f5cf009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:15.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.313+0000 7f5cf521b640 1 -- 172.21.15.44:0/3207919324 shutdown_connections 2024-04-02T13:21:15.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.314+0000 7f5cf521b640 1 -- 172.21.15.44:0/3207919324 wait complete. 2024-04-02T13:21:15.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.315+0000 7f5cf521b640 1 Processor -- start 2024-04-02T13:21:15.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.315+0000 7f5cf521b640 1 -- start start 2024-04-02T13:21:15.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cf521b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cf00b3e80 0x7f5cf015f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cf521b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cf00cd030 0x7f5cf015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cf521b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cf015c260 0x7f5cf00706c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cf521b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf00cfe60 con 0x7f5cf015c260 2024-04-02T13:21:15.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cf521b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf00ce000 con 0x7f5cf00cd030 2024-04-02T13:21:15.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cf521b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf00c6b40 con 0x7f5cf00b3e80 2024-04-02T13:21:15.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cef59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cf015c260 0x7f5cf00706c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5ceed9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cf00b3e80 0x7f5cf015f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cef59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cf015c260 0x7f5cf00706c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45706/0 (socket says 172.21.15.44:45706) 2024-04-02T13:21:15.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cef59e640 1 -- 172.21.15.44:0/2308619316 learned_addr learned my addr 172.21.15.44:0/2308619316 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:15.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5ce659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cf00cd030 0x7f5cf015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cef59e640 1 -- 172.21.15.44:0/2308619316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cf00b3e80 msgr2=0x7f5cf015f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cef59e640 1 --2- 172.21.15.44:0/2308619316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cf00b3e80 0x7f5cf015f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cef59e640 1 -- 172.21.15.44:0/2308619316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cf00cd030 msgr2=0x7f5cf015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cef59e640 1 --2- 172.21.15.44:0/2308619316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cf00cd030 0x7f5cf015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cef59e640 1 -- 172.21.15.44:0/2308619316 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf015d2b0 con 0x7f5cf015c260 2024-04-02T13:21:15.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.316+0000 7f5cef59e640 1 --2- 172.21.15.44:0/2308619316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cf015c260 0x7f5cf00706c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c7a0 tx=0x7f5ce800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:15.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.317+0000 7f5ce7fff640 1 -- 172.21.15.44:0/2308619316 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8003b70 con 0x7f5cf015c260 2024-04-02T13:21:15.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.317+0000 7f5ce7fff640 1 -- 172.21.15.44:0/2308619316 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce8003d30 con 0x7f5cf015c260 2024-04-02T13:21:15.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.317+0000 7f5cf521b640 1 -- 172.21.15.44:0/2308619316 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf006e160 con 0x7f5cf015c260 2024-04-02T13:21:15.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.317+0000 7f5ce7fff640 1 -- 172.21.15.44:0/2308619316 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce800aa90 con 0x7f5cf015c260 2024-04-02T13:21:15.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.317+0000 7f5cf521b640 1 -- 172.21.15.44:0/2308619316 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf015c850 con 0x7f5cf015c260 2024-04-02T13:21:15.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.318+0000 7f5cf521b640 1 -- 172.21.15.44:0/2308619316 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf015c640 con 0x7f5cf015c260 2024-04-02T13:21:15.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.318+0000 7f5ce7fff640 1 -- 172.21.15.44:0/2308619316 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5ce800ac30 con 0x7f5cf015c260 2024-04-02T13:21:15.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.318+0000 7f5ce7fff640 1 --2- 172.21.15.44:0/2308619316 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cd006c8f0 0x7f5cd006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.318+0000 7f5ce7fff640 1 -- 172.21.15.44:0/2308619316 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f5ce808bcf0 con 0x7f5cf015c260 2024-04-02T13:21:15.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.319+0000 7f5ceed9d640 1 --2- 172.21.15.44:0/2308619316 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cd006c8f0 0x7f5cd006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.319+0000 7f5ceed9d640 1 --2- 172.21.15.44:0/2308619316 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cd006c8f0 0x7f5cd006edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5ce00079a0 tx=0x7f5ce0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:15.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.323+0000 7f5ce7fff640 1 -- 172.21.15.44:0/2308619316 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5ce8090070 con 0x7f5cf015c260 2024-04-02T13:21:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.511+0000 7f5cf521b640 1 -- 172.21.15.44:0/2308619316 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5cf00684f0 con 0x7f5cf015c260 2024-04-02T13:21:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.511+0000 7f5ce7fff640 1 -- 172.21.15.44:0/2308619316 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f5ce8059740 con 0x7f5cf015c260 2024-04-02T13:21:15.513 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.513+0000 7f5cf521b640 1 -- 172.21.15.44:0/2308619316 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cd006c8f0 msgr2=0x7f5cd006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.513+0000 7f5cf521b640 1 --2- 172.21.15.44:0/2308619316 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cd006c8f0 0x7f5cd006edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5ce00079a0 tx=0x7f5ce0008040 comp rx=0 tx=0).stop 2024-04-02T13:21:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.513+0000 7f5cf521b640 1 -- 172.21.15.44:0/2308619316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cf015c260 msgr2=0x7f5cf00706c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.513+0000 7f5cf521b640 1 --2- 172.21.15.44:0/2308619316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cf015c260 0x7f5cf00706c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c7a0 tx=0x7f5ce800cc60 comp rx=0 tx=0).stop 2024-04-02T13:21:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.513+0000 7f5cf521b640 1 -- 172.21.15.44:0/2308619316 shutdown_connections 2024-04-02T13:21:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.513+0000 7f5cf521b640 1 --2- 172.21.15.44:0/2308619316 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cd006c8f0 0x7f5cd006edb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.513+0000 7f5cf521b640 1 --2- 172.21.15.44:0/2308619316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cf015c260 0x7f5cf00706c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.513+0000 7f5cf521b640 1 --2- 172.21.15.44:0/2308619316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cf00cd030 0x7f5cf015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.513+0000 7f5cf521b640 1 --2- 172.21.15.44:0/2308619316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cf00b3e80 0x7f5cf015f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.513+0000 7f5cf521b640 1 -- 172.21.15.44:0/2308619316 >> 172.21.15.44:0/2308619316 conn(0x7f5cf009aa20 msgr2=0x7f5cf00c6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.513+0000 7f5cf521b640 1 -- 172.21.15.44:0/2308619316 shutdown_connections 2024-04-02T13:21:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.513+0000 7f5cf521b640 1 -- 172.21.15.44:0/2308619316 wait complete. 2024-04-02T13:21:15.526 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:15.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.629+0000 7f5f3a5e2640 1 Processor -- start 2024-04-02T13:21:15.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.629+0000 7f5f3a5e2640 1 -- start start 2024-04-02T13:21:15.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.630+0000 7f5f3a5e2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5f340cfdc0 0x7f5f340b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.630+0000 7f5f3a5e2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f3406de80 con 0x7f5f3406daa0 2024-04-02T13:21:15.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.630+0000 7f5f3a5e2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f340cf000 con 0x7f5f340d2e00 2024-04-02T13:21:15.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.630+0000 7f5f3a5e2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f5f340cf930 con 0x7f5f340cfdc0 2024-04-02T13:21:15.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.630+0000 7f5f33fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f5f3406daa0 0x7f5f340d26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48836/0 (socket says 172.21.15.44:48836) 2024-04-02T13:21:15.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.630+0000 7f5f38b7f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5f340cfdc0 0x7f5f340b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.630+0000 7f5f33fff640 1 -- 172.21.15.44:0/3676167073 learned_addr learned my addr 172.21.15.44:0/3676167073 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:15.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.630+0000 7f5f38b7f640 1 -- 172.21.15.44:0/3676167073 >> v1:172.21.15.69:6789/0 conn(0x7f5f340d2e00 legacy=0x7f5f3415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:15.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.630+0000 7f5f38b7f640 1 -- 172.21.15.44:0/3676167073 >> v1:172.21.15.44:6789/0 conn(0x7f5f3406daa0 legacy=0x7f5f340d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.630+0000 7f5f32ffd640 1 -- 172.21.15.44:0/3676167073 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4180754368 0 0) 0x7f5f3406de80 con 0x7f5f3406daa0 2024-04-02T13:21:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.630+0000 7f5f38b7f640 1 -- 172.21.15.44:0/3676167073 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f340cf2e0 con 0x7f5f340cfdc0 2024-04-02T13:21:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.631+0000 7f5f38b7f640 1 --2- 172.21.15.44:0/3676167073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f340cfdc0 0x7f5f340b35b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009d90 tx=0x7f5f2c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809d0bec0a8185b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.631+0000 7f5f32ffd640 1 -- 172.21.15.44:0/3676167073 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c039070 con 0x7f5f340cfdc0 2024-04-02T13:21:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.631+0000 7f5f32ffd640 1 -- 172.21.15.44:0/3676167073 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f2c02cc60 con 0x7f5f340cfdc0 2024-04-02T13:21:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.631+0000 7f5f32ffd640 1 -- 172.21.15.44:0/3676167073 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c034660 con 0x7f5f340cfdc0 2024-04-02T13:21:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.632+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/3676167073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f340cfdc0 msgr2=0x7f5f340b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.632+0000 7f5f3a5e2640 1 --2- 172.21.15.44:0/3676167073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f340cfdc0 0x7f5f340b35b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009d90 tx=0x7f5f2c02beb0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.632+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/3676167073 shutdown_connections 2024-04-02T13:21:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.632+0000 7f5f3a5e2640 1 --2- 172.21.15.44:0/3676167073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f340cfdc0 0x7f5f340b35b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.632+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/3676167073 >> 172.21.15.44:0/3676167073 conn(0x7f5f3409d430 msgr2=0x7f5f340d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.632+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/3676167073 shutdown_connections 2024-04-02T13:21:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.632+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/3676167073 wait complete. 2024-04-02T13:21:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.633+0000 7f5f3a5e2640 1 Processor -- start 2024-04-02T13:21:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.633+0000 7f5f3a5e2640 1 -- start start 2024-04-02T13:21:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f3a5e2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f3406daa0 0x7f5f341591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f3a5e2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f340cfdc0 0x7f5f340cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f3a5e2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f340d2e00 0x7f5f340c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f3a5e2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5f340ceda0 con 0x7f5f340d2e00 2024-04-02T13:21:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f3a5e2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5f340cec20 con 0x7f5f340cfdc0 2024-04-02T13:21:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f3a5e2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5f340d11c0 con 0x7f5f3406daa0 2024-04-02T13:21:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f38b7f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f340d2e00 0x7f5f340c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f38b7f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f340d2e00 0x7f5f340c9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45726/0 (socket says 172.21.15.44:45726) 2024-04-02T13:21:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f38b7f640 1 -- 172.21.15.44:0/2797864966 learned_addr learned my addr 172.21.15.44:0/2797864966 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f33fff640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f3406daa0 0x7f5f341591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f337fe640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f340cfdc0 0x7f5f340cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f38b7f640 1 -- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f3406daa0 msgr2=0x7f5f341591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f38b7f640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f3406daa0 0x7f5f341591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f38b7f640 1 -- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f340cfdc0 msgr2=0x7f5f340cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f38b7f640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f340cfdc0 0x7f5f340cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f38b7f640 1 -- 172.21.15.44:0/2797864966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f340ca320 con 0x7f5f340d2e00 2024-04-02T13:21:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f38b7f640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f340d2e00 0x7f5f340c9de0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009eb0 tx=0x7f5f2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f30ff9640 1 -- 172.21.15.44:0/2797864966 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c044030 con 0x7f5f340d2e00 2024-04-02T13:21:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f30ff9640 1 -- 172.21.15.44:0/2797864966 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f2c02ca60 con 0x7f5f340d2e00 2024-04-02T13:21:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.634+0000 7f5f30ff9640 1 -- 172.21.15.44:0/2797864966 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c039040 con 0x7f5f340d2e00 2024-04-02T13:21:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.635+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/2797864966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f340cd780 con 0x7f5f340d2e00 2024-04-02T13:21:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.635+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/2797864966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f340ccde0 con 0x7f5f340d2e00 2024-04-02T13:21:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.635+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/2797864966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f340684f0 con 0x7f5f340d2e00 2024-04-02T13:21:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.636+0000 7f5f30ff9640 1 -- 172.21.15.44:0/2797864966 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5f2c047070 con 0x7f5f340d2e00 2024-04-02T13:21:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.636+0000 7f5f30ff9640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f1406c920 0x7f5f1406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.636+0000 7f5f30ff9640 1 -- 172.21.15.44:0/2797864966 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f5f2c0ae900 con 0x7f5f340d2e00 2024-04-02T13:21:15.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.636+0000 7f5f33fff640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f1406c920 0x7f5f1406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.637+0000 7f5f33fff640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f1406c920 0x7f5f1406ede0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c000f60 tx=0x7f5f1c007b50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:15.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.641+0000 7f5f30ff9640 1 -- 172.21.15.44:0/2797864966 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5f2c04e070 con 0x7f5f340d2e00 2024-04-02T13:21:15.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.825+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/2797864966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5f340d24d0 con 0x7f5f340d2e00 2024-04-02T13:21:15.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.826+0000 7f5f30ff9640 1 -- 172.21.15.44:0/2797864966 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f5f340cd780 con 0x7f5f340d2e00 2024-04-02T13:21:15.827 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.827+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f1406c920 msgr2=0x7f5f1406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.827+0000 7f5f3a5e2640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f1406c920 0x7f5f1406ede0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c000f60 tx=0x7f5f1c007b50 comp rx=0 tx=0).stop 2024-04-02T13:21:15.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.827+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/2797864966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f340d2e00 msgr2=0x7f5f340c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.827+0000 7f5f3a5e2640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f340d2e00 0x7f5f340c9de0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009eb0 tx=0x7f5f2c004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:15.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.827+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/2797864966 shutdown_connections 2024-04-02T13:21:15.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.827+0000 7f5f3a5e2640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f1406c920 0x7f5f1406ede0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.827+0000 7f5f3a5e2640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f340d2e00 0x7f5f340c9de0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.828+0000 7f5f3a5e2640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f340cfdc0 0x7f5f340cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.828+0000 7f5f3a5e2640 1 --2- 172.21.15.44:0/2797864966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f3406daa0 0x7f5f341591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.828+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/2797864966 >> 172.21.15.44:0/2797864966 conn(0x7f5f3409d430 msgr2=0x7f5f340b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:15.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.828+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/2797864966 shutdown_connections 2024-04-02T13:21:15.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.828+0000 7f5f3a5e2640 1 -- 172.21.15.44:0/2797864966 wait complete. 2024-04-02T13:21:15.840 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:15.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.944+0000 7f9661b1b640 1 Processor -- start 2024-04-02T13:21:15.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.944+0000 7f9661b1b640 1 -- start start 2024-04-02T13:21:15.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.945+0000 7f9661b1b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c15e640 0x7f965c0d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.945+0000 7f9661b1b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f965c0cec60 0x7f965c0d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.945+0000 7f9661b1b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f965c0cf120 con 0x7f965c15e260 2024-04-02T13:21:15.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.945+0000 7f9661b1b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f965c0b5a50 con 0x7f965c0cec60 2024-04-02T13:21:15.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.945+0000 7f9661b1b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f965c0d01c0 con 0x7f965c15e640 2024-04-02T13:21:15.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.945+0000 7f965affd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c15e640 0x7f965c0d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.945+0000 7f965bfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f965c0cec60 0x7f965c0d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.945+0000 7f965b7fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f965c15e260 0x7f965c0d2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:54422/0 (socket says 172.21.15.44:54422) 2024-04-02T13:21:15.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.945+0000 7f965b7fe640 1 -- 172.21.15.44:0/3402563255 learned_addr learned my addr 172.21.15.44:0/3402563255 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:15.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.946+0000 7f965a7fc640 1 -- 172.21.15.44:0/3402563255 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1042657684 0 0) 0x7f965c0cf120 con 0x7f965c15e260 2024-04-02T13:21:15.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.946+0000 7f965a7fc640 1 -- 172.21.15.44:0/3402563255 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9644003520 con 0x7f965c15e260 2024-04-02T13:21:15.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.946+0000 7f965affd640 1 -- 172.21.15.44:0/3402563255 >> v2:172.21.15.69:3300/0 conn(0x7f965c0cec60 msgr2=0x7f965c0d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:15.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.946+0000 7f965affd640 1 --2- 172.21.15.44:0/3402563255 >> v2:172.21.15.69:3300/0 conn(0x7f965c0cec60 0x7f965c0d1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.946+0000 7f965affd640 1 -- 172.21.15.44:0/3402563255 >> v1:172.21.15.69:6789/0 conn(0x7f965c15e260 legacy=0x7f965c0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.946+0000 7f965affd640 1 -- 172.21.15.44:0/3402563255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c0cf400 con 0x7f965c15e640 2024-04-02T13:21:15.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.946+0000 7f965bfff640 1 --2- 172.21.15.44:0/3402563255 >> v2:172.21.15.69:3300/0 conn(0x7f965c0cec60 0x7f965c0d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:15.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.947+0000 7f965affd640 1 --2- 172.21.15.44:0/3402563255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c15e640 0x7f965c0d0ad0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f96500097a0 tx=0x7f965002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2811cd62b169c26 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:15.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.947+0000 7f965a7fc640 1 -- 172.21.15.44:0/3402563255 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650035020 con 0x7f965c15e640 2024-04-02T13:21:15.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.947+0000 7f965a7fc640 1 -- 172.21.15.44:0/3402563255 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9650031830 con 0x7f965c15e640 2024-04-02T13:21:15.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.947+0000 7f965a7fc640 1 -- 172.21.15.44:0/3402563255 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650031be0 con 0x7f965c15e640 2024-04-02T13:21:15.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.947+0000 7f9661b1b640 1 -- 172.21.15.44:0/3402563255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c15e640 msgr2=0x7f965c0d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.947+0000 7f9661b1b640 1 --2- 172.21.15.44:0/3402563255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c15e640 0x7f965c0d0ad0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f96500097a0 tx=0x7f965002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.947+0000 7f9661b1b640 1 -- 172.21.15.44:0/3402563255 shutdown_connections 2024-04-02T13:21:15.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.947+0000 7f9661b1b640 1 --2- 172.21.15.44:0/3402563255 >> v2:172.21.15.69:3300/0 conn(0x7f965c0cec60 0x7f965c0d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.947+0000 7f9661b1b640 1 --2- 172.21.15.44:0/3402563255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c15e640 0x7f965c0d0ad0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.947+0000 7f9661b1b640 1 -- 172.21.15.44:0/3402563255 >> 172.21.15.44:0/3402563255 conn(0x7f965c09d3c0 msgr2=0x7f965c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:15.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.947+0000 7f9661b1b640 1 -- 172.21.15.44:0/3402563255 shutdown_connections 2024-04-02T13:21:15.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.947+0000 7f9661b1b640 1 -- 172.21.15.44:0/3402563255 wait complete. 2024-04-02T13:21:15.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.948+0000 7f9661b1b640 1 Processor -- start 2024-04-02T13:21:15.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.949+0000 7f9661b1b640 1 -- start start 2024-04-02T13:21:15.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.949+0000 7f9661b1b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f965c0cec60 0x7f965c154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.949+0000 7f9661b1b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965c15e260 0x7f965c0d0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.949+0000 7f9661b1b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c15e640 0x7f965c0d0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.949+0000 7f9661b1b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f965c0cfca0 con 0x7f965c0cec60 2024-04-02T13:21:15.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.949+0000 7f9661b1b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f965c0cf9c0 con 0x7f965c15e260 2024-04-02T13:21:15.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.949+0000 7f9661b1b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f965c0d3070 con 0x7f965c15e640 2024-04-02T13:21:15.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.949+0000 7f965b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f965c0cec60 0x7f965c154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.949+0000 7f965b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f965c0cec60 0x7f965c154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45730/0 (socket says 172.21.15.44:45730) 2024-04-02T13:21:15.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.949+0000 7f965b7fe640 1 -- 172.21.15.44:0/58417532 learned_addr learned my addr 172.21.15.44:0/58417532 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:15.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.950+0000 7f965affd640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965c15e260 0x7f965c0d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.950+0000 7f965bfff640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c15e640 0x7f965c0d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.950+0000 7f965b7fe640 1 -- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c15e640 msgr2=0x7f965c0d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.950+0000 7f965b7fe640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c15e640 0x7f965c0d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.950+0000 7f965b7fe640 1 -- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965c15e260 msgr2=0x7f965c0d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:15.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.950+0000 7f965b7fe640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965c15e260 0x7f965c0d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:15.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.950+0000 7f965b7fe640 1 -- 172.21.15.44:0/58417532 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c0c8910 con 0x7f965c0cec60 2024-04-02T13:21:15.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.950+0000 7f965b7fe640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f965c0cec60 0x7f965c154f90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f964c00c430 tx=0x7f964c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:15.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.950+0000 7f9643fff640 1 -- 172.21.15.44:0/58417532 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c017070 con 0x7f965c0cec60 2024-04-02T13:21:15.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.950+0000 7f9643fff640 1 -- 172.21.15.44:0/58417532 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f964c005ce0 con 0x7f965c0cec60 2024-04-02T13:21:15.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.950+0000 7f9643fff640 1 -- 172.21.15.44:0/58417532 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c0072f0 con 0x7f965c0cec60 2024-04-02T13:21:15.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.950+0000 7f9661b1b640 1 -- 172.21.15.44:0/58417532 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965c15f4f0 con 0x7f965c0cec60 2024-04-02T13:21:15.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.950+0000 7f9661b1b640 1 -- 172.21.15.44:0/58417532 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f965c15fab0 con 0x7f965c0cec60 2024-04-02T13:21:15.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.951+0000 7f9661b1b640 1 -- 172.21.15.44:0/58417532 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f965c0684f0 con 0x7f965c0cec60 2024-04-02T13:21:15.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.951+0000 7f9643fff640 1 -- 172.21.15.44:0/58417532 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f964c01a070 con 0x7f965c0cec60 2024-04-02T13:21:15.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.952+0000 7f9643fff640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f964806c8f0 0x7f964806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:15.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.952+0000 7f9643fff640 1 -- 172.21.15.44:0/58417532 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f964c08b2e0 con 0x7f965c0cec60 2024-04-02T13:21:15.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.952+0000 7f965affd640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f964806c8f0 0x7f964806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:15.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.952+0000 7f965affd640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f964806c8f0 0x7f964806edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9650036000 tx=0x7f965003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:15.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:15.956+0000 7f9643fff640 1 -- 172.21.15.44:0/58417532 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f964c058ca0 con 0x7f965c0cec60 2024-04-02T13:21:16.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.142+0000 7f9661b1b640 1 -- 172.21.15.44:0/58417532 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f965c0d1090 con 0x7f965c0cec60 2024-04-02T13:21:16.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.142+0000 7f9643fff640 1 -- 172.21.15.44:0/58417532 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f964c05cfa0 con 0x7f965c0cec60 2024-04-02T13:21:16.144 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:16.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.145+0000 7f9661b1b640 1 -- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f964806c8f0 msgr2=0x7f964806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.145+0000 7f9661b1b640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f964806c8f0 0x7f964806edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9650036000 tx=0x7f965003c040 comp rx=0 tx=0).stop 2024-04-02T13:21:16.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.145+0000 7f9661b1b640 1 -- 172.21.15.44:0/58417532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f965c0cec60 msgr2=0x7f965c154f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.145+0000 7f9661b1b640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f965c0cec60 0x7f965c154f90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f964c00c430 tx=0x7f964c00c830 comp rx=0 tx=0).stop 2024-04-02T13:21:16.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.145+0000 7f9661b1b640 1 -- 172.21.15.44:0/58417532 shutdown_connections 2024-04-02T13:21:16.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.145+0000 7f9661b1b640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f964806c8f0 0x7f964806edb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.145+0000 7f9661b1b640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c15e640 0x7f965c0d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.145+0000 7f9661b1b640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965c15e260 0x7f965c0d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.145+0000 7f9661b1b640 1 --2- 172.21.15.44:0/58417532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f965c0cec60 0x7f965c154f90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.145+0000 7f9661b1b640 1 -- 172.21.15.44:0/58417532 >> 172.21.15.44:0/58417532 conn(0x7f965c09d3c0 msgr2=0x7f965c15d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:16.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.145+0000 7f9661b1b640 1 -- 172.21.15.44:0/58417532 shutdown_connections 2024-04-02T13:21:16.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.145+0000 7f9661b1b640 1 -- 172.21.15.44:0/58417532 wait complete. 2024-04-02T13:21:16.157 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:21:16.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.258+0000 7f6f80cd3640 1 Processor -- start 2024-04-02T13:21:16.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.258+0000 7f6f80cd3640 1 -- start start 2024-04-02T13:21:16.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.259+0000 7f6f80cd3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6f7c0b47b0 0x7f6f7c0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:16.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.259+0000 7f6f80cd3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f7c0d4d70 con 0x7f6f7c0d28f0 2024-04-02T13:21:16.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.259+0000 7f6f80cd3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f7c0d2030 con 0x7f6f7c0b5200 2024-04-02T13:21:16.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.259+0000 7f6f80cd3640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f6f7c0d5050 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.259+0000 7f6f73fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6f7c0b47b0 0x7f6f7c0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:16.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.259+0000 7f6f7a59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6f7c0d28f0 0x7f6f7c0b40a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48838/0 (socket says 172.21.15.44:48838) 2024-04-02T13:21:16.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.259+0000 7f6f7a59c640 1 -- 172.21.15.44:0/566360727 learned_addr learned my addr 172.21.15.44:0/566360727 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:16.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.259+0000 7f6f79d9b640 1 -- 172.21.15.44:0/566360727 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3530578633 0 0) 0x7f6f7c0d4d70 con 0x7f6f7c0d28f0 2024-04-02T13:21:16.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.259+0000 7f6f79d9b640 1 -- 172.21.15.44:0/566360727 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f68003520 con 0x7f6f7c0d28f0 2024-04-02T13:21:16.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.259+0000 7f6f73fff640 1 -- 172.21.15.44:0/566360727 >> v1:172.21.15.69:6789/0 conn(0x7f6f7c0b5200 legacy=0x7f6f7c0b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:16.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.259+0000 7f6f73fff640 1 -- 172.21.15.44:0/566360727 >> v1:172.21.15.44:6789/0 conn(0x7f6f7c0d28f0 legacy=0x7f6f7c0b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.259+0000 7f6f73fff640 1 -- 172.21.15.44:0/566360727 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7c0b4cb0 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.260+0000 7f6f73fff640 1 --2- 172.21.15.44:0/566360727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7c0b47b0 0x7f6f7c0d2cd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6f640097a0 tx=0x7f6f6402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3084ec240e932e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:16.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.260+0000 7f6f79d9b640 1 -- 172.21.15.44:0/566360727 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64038070 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.260+0000 7f6f79d9b640 1 -- 172.21.15.44:0/566360727 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f64031700 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.260+0000 7f6f79d9b640 1 -- 172.21.15.44:0/566360727 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64031ab0 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.260+0000 7f6f80cd3640 1 -- 172.21.15.44:0/566360727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7c0b47b0 msgr2=0x7f6f7c0d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.260+0000 7f6f80cd3640 1 --2- 172.21.15.44:0/566360727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7c0b47b0 0x7f6f7c0d2cd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6f640097a0 tx=0x7f6f6402b7a0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.260+0000 7f6f80cd3640 1 -- 172.21.15.44:0/566360727 shutdown_connections 2024-04-02T13:21:16.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.260+0000 7f6f80cd3640 1 --2- 172.21.15.44:0/566360727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7c0b47b0 0x7f6f7c0d2cd0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.260+0000 7f6f80cd3640 1 -- 172.21.15.44:0/566360727 >> 172.21.15.44:0/566360727 conn(0x7f6f7c0d4120 msgr2=0x7f6f7c0d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:16.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.261+0000 7f6f80cd3640 1 -- 172.21.15.44:0/566360727 shutdown_connections 2024-04-02T13:21:16.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.261+0000 7f6f80cd3640 1 -- 172.21.15.44:0/566360727 wait complete. 2024-04-02T13:21:16.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.262+0000 7f6f80cd3640 1 Processor -- start 2024-04-02T13:21:16.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.262+0000 7f6f80cd3640 1 -- start start 2024-04-02T13:21:16.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.262+0000 7f6f80cd3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7c0b47b0 0x7f6f7c1f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:16.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f80cd3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f7c0b5200 0x7f6f7c157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:16.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f80cd3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f7c0d28f0 0x7f6f7c0b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:16.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f80cd3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c0d23a0 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f80cd3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c0d14f0 con 0x7f6f7c0b5200 2024-04-02T13:21:16.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f80cd3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6f7c0b5020 con 0x7f6f7c0d28f0 2024-04-02T13:21:16.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f7a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7c0b47b0 0x7f6f7c1f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:16.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f7a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7c0b47b0 0x7f6f7c1f4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45754/0 (socket says 172.21.15.44:45754) 2024-04-02T13:21:16.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f7a59c640 1 -- 172.21.15.44:0/2249263711 learned_addr learned my addr 172.21.15.44:0/2249263711 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:16.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f7a59c640 1 -- 172.21.15.44:0/2249263711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f7c0d28f0 msgr2=0x7f6f7c0b2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f7a59c640 1 --2- 172.21.15.44:0/2249263711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f7c0d28f0 0x7f6f7c0b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f7a59c640 1 -- 172.21.15.44:0/2249263711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f7c0b5200 msgr2=0x7f6f7c157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f7a59c640 1 --2- 172.21.15.44:0/2249263711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f7c0b5200 0x7f6f7c157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f7a59c640 1 -- 172.21.15.44:0/2249263711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7c0b2820 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f7a59c640 1 --2- 172.21.15.44:0/2249263711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7c0b47b0 0x7f6f7c1f4600 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c390 tx=0x7f6f6c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:16.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f72ffd640 1 -- 172.21.15.44:0/2249263711 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c017070 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f72ffd640 1 -- 172.21.15.44:0/2249263711 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f6c005ce0 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f80cd3640 1 -- 172.21.15.44:0/2249263711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f7c0cbd10 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f72ffd640 1 -- 172.21.15.44:0/2249263711 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c0072f0 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.263+0000 7f6f80cd3640 1 -- 172.21.15.44:0/2249263711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f7c0b2ac0 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.264+0000 7f6f80cd3640 1 -- 172.21.15.44:0/2249263711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f7c0684f0 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.265+0000 7f6f72ffd640 1 -- 172.21.15.44:0/2249263711 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6f6c01a070 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.265+0000 7f6f72ffd640 1 --2- 172.21.15.44:0/2249263711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f5c06c8f0 0x7f6f5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:16.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.265+0000 7f6f72ffd640 1 -- 172.21.15.44:0/2249263711 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f6f6c08b390 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.265+0000 7f6f73fff640 1 --2- 172.21.15.44:0/2249263711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f5c06c8f0 0x7f6f5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:16.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.266+0000 7f6f73fff640 1 --2- 172.21.15.44:0/2249263711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f5c06c8f0 0x7f6f5c06edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6f64035000 tx=0x7f6f64030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:16.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.270+0000 7f6f72ffd640 1 -- 172.21.15.44:0/2249263711 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6f6c0593a0 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.465+0000 7f6f80cd3640 1 -- 172.21.15.44:0/2249263711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6f7c0b70b0 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.465+0000 7f6f72ffd640 1 -- 172.21.15.44:0/2249263711 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f6f6c058ca0 con 0x7f6f7c0b47b0 2024-04-02T13:21:16.467 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:21:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.467+0000 7f6f80cd3640 1 -- 172.21.15.44:0/2249263711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f5c06c8f0 msgr2=0x7f6f5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.467+0000 7f6f80cd3640 1 --2- 172.21.15.44:0/2249263711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f5c06c8f0 0x7f6f5c06edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6f64035000 tx=0x7f6f64030000 comp rx=0 tx=0).stop 2024-04-02T13:21:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.467+0000 7f6f80cd3640 1 -- 172.21.15.44:0/2249263711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7c0b47b0 msgr2=0x7f6f7c1f4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.467+0000 7f6f80cd3640 1 --2- 172.21.15.44:0/2249263711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7c0b47b0 0x7f6f7c1f4600 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c390 tx=0x7f6f6c00c850 comp rx=0 tx=0).stop 2024-04-02T13:21:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.467+0000 7f6f80cd3640 1 -- 172.21.15.44:0/2249263711 shutdown_connections 2024-04-02T13:21:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.467+0000 7f6f80cd3640 1 --2- 172.21.15.44:0/2249263711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f5c06c8f0 0x7f6f5c06edb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.467+0000 7f6f80cd3640 1 --2- 172.21.15.44:0/2249263711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f7c0d28f0 0x7f6f7c0b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.467+0000 7f6f80cd3640 1 --2- 172.21.15.44:0/2249263711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f7c0b5200 0x7f6f7c157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.467+0000 7f6f80cd3640 1 --2- 172.21.15.44:0/2249263711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7c0b47b0 0x7f6f7c1f4600 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.467+0000 7f6f80cd3640 1 -- 172.21.15.44:0/2249263711 >> 172.21.15.44:0/2249263711 conn(0x7f6f7c0d4120 msgr2=0x7f6f7c0cf580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.467+0000 7f6f80cd3640 1 -- 172.21.15.44:0/2249263711 shutdown_connections 2024-04-02T13:21:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.467+0000 7f6f80cd3640 1 -- 172.21.15.44:0/2249263711 wait complete. 2024-04-02T13:21:16.479 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:21:16.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:16.479 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:21:16.506 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:21:16.506 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:21:16.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:16.506 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:21:16.561 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:21:16.561 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:16.580 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:21:16.580 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:16.638 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:16.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.738+0000 7f5161904640 1 Processor -- start 2024-04-02T13:21:16.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.738+0000 7f5161904640 1 -- start start 2024-04-02T13:21:16.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.739+0000 7f5161904640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f515c15e4d0 0x7f515c06b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:16.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.739+0000 7f5161904640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f515c06dfe0 con 0x7f515c15e0f0 2024-04-02T13:21:16.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.739+0000 7f5161904640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f515c06e2c0 con 0x7f515c072f60 2024-04-02T13:21:16.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.739+0000 7f5161904640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f515c06e5a0 con 0x7f515c15e4d0 2024-04-02T13:21:16.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.739+0000 7f515affd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f515c15e0f0 0x7f515c0d00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48850/0 (socket says 172.21.15.44:48850) 2024-04-02T13:21:16.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.739+0000 7f515affd640 1 -- 172.21.15.44:0/2221855135 learned_addr learned my addr 172.21.15.44:0/2221855135 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:16.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.739+0000 7f515a7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f515c15e4d0 0x7f515c06b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:16.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.739+0000 7f5159ffb640 1 -- 172.21.15.44:0/2221855135 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4064317986 0 0) 0x7f515c06dfe0 con 0x7f515c15e0f0 2024-04-02T13:21:16.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.739+0000 7f5159ffb640 1 -- 172.21.15.44:0/2221855135 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5148003520 con 0x7f515c15e0f0 2024-04-02T13:21:16.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.739+0000 7f515a7fc640 1 -- 172.21.15.44:0/2221855135 >> v1:172.21.15.69:6789/0 conn(0x7f515c072f60 legacy=0x7f515c0add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:16.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.739+0000 7f515a7fc640 1 -- 172.21.15.44:0/2221855135 >> v1:172.21.15.44:6789/0 conn(0x7f515c15e0f0 legacy=0x7f515c0d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.739+0000 7f515a7fc640 1 -- 172.21.15.44:0/2221855135 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515c06e880 con 0x7f515c15e4d0 2024-04-02T13:21:16.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.740+0000 7f515a7fc640 1 --2- 172.21.15.44:0/2221855135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f515c15e4d0 0x7f515c06b4d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f51440097a0 tx=0x7f514402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5c62b96f3a364f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:16.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.740+0000 7f5159ffb640 1 -- 172.21.15.44:0/2221855135 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5144034070 con 0x7f515c15e4d0 2024-04-02T13:21:16.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.740+0000 7f5159ffb640 1 -- 172.21.15.44:0/2221855135 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5144030610 con 0x7f515c15e4d0 2024-04-02T13:21:16.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.740+0000 7f5159ffb640 1 -- 172.21.15.44:0/2221855135 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51440309c0 con 0x7f515c15e4d0 2024-04-02T13:21:16.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.741+0000 7f5161904640 1 -- 172.21.15.44:0/2221855135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f515c15e4d0 msgr2=0x7f515c06b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.741+0000 7f5161904640 1 --2- 172.21.15.44:0/2221855135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f515c15e4d0 0x7f515c06b4d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f51440097a0 tx=0x7f514402b960 comp rx=0 tx=0).stop 2024-04-02T13:21:16.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.741+0000 7f5161904640 1 -- 172.21.15.44:0/2221855135 shutdown_connections 2024-04-02T13:21:16.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.741+0000 7f5161904640 1 --2- 172.21.15.44:0/2221855135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f515c15e4d0 0x7f515c06b4d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.741+0000 7f5161904640 1 -- 172.21.15.44:0/2221855135 >> 172.21.15.44:0/2221855135 conn(0x7f515c0b2980 msgr2=0x7f515c0b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:16.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.741+0000 7f5161904640 1 -- 172.21.15.44:0/2221855135 shutdown_connections 2024-04-02T13:21:16.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.741+0000 7f5161904640 1 -- 172.21.15.44:0/2221855135 wait complete. 2024-04-02T13:21:16.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.742+0000 7f5161904640 1 Processor -- start 2024-04-02T13:21:16.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.742+0000 7f5161904640 1 -- start start 2024-04-02T13:21:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f5161904640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f515c072f60 0x7f515c143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f5161904640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f515c15e0f0 0x7f515c0b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f5161904640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f515c15e4d0 0x7f515c147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f5161904640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f515c06eda0 con 0x7f515c15e4d0 2024-04-02T13:21:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f5161904640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f515c0d0800 con 0x7f515c15e0f0 2024-04-02T13:21:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f5161904640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f515c06ba10 con 0x7f515c072f60 2024-04-02T13:21:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f515b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f515c15e4d0 0x7f515c147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f515b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f515c15e4d0 0x7f515c147b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45776/0 (socket says 172.21.15.44:45776) 2024-04-02T13:21:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f515b7fe640 1 -- 172.21.15.44:0/297335916 learned_addr learned my addr 172.21.15.44:0/297335916 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f515a7fc640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f515c15e0f0 0x7f515c0b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f515affd640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f515c072f60 0x7f515c143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f515b7fe640 1 -- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f515c072f60 msgr2=0x7f515c143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f515b7fe640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f515c072f60 0x7f515c143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f515b7fe640 1 -- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f515c15e0f0 msgr2=0x7f515c0b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f515b7fe640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f515c15e0f0 0x7f515c0b63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f515b7fe640 1 -- 172.21.15.44:0/297335916 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515c0b6900 con 0x7f515c15e4d0 2024-04-02T13:21:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f515a7fc640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f515c15e0f0 0x7f515c0b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f515affd640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f515c072f60 0x7f515c143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f515b7fe640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f515c15e4d0 0x7f515c147b90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f515400c410 tx=0x7f515400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.743+0000 7f5153fff640 1 -- 172.21.15.44:0/297335916 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5154011790 con 0x7f515c15e4d0 2024-04-02T13:21:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.744+0000 7f5153fff640 1 -- 172.21.15.44:0/297335916 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5154017070 con 0x7f515c15e4d0 2024-04-02T13:21:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.744+0000 7f5161904640 1 -- 172.21.15.44:0/297335916 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f515c1480d0 con 0x7f515c15e4d0 2024-04-02T13:21:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.744+0000 7f5153fff640 1 -- 172.21.15.44:0/297335916 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5154011bb0 con 0x7f515c15e4d0 2024-04-02T13:21:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.744+0000 7f5161904640 1 -- 172.21.15.44:0/297335916 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f515c144510 con 0x7f515c15e4d0 2024-04-02T13:21:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.745+0000 7f5161904640 1 -- 172.21.15.44:0/297335916 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f515c0684f0 con 0x7f515c15e4d0 2024-04-02T13:21:16.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.745+0000 7f5153fff640 1 -- 172.21.15.44:0/297335916 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f515401c460 con 0x7f515c15e4d0 2024-04-02T13:21:16.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.745+0000 7f5153fff640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f513c06c8f0 0x7f513c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:16.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.746+0000 7f5153fff640 1 -- 172.21.15.44:0/297335916 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f515408bd20 con 0x7f515c15e4d0 2024-04-02T13:21:16.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.746+0000 7f515affd640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f513c06c8f0 0x7f513c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:16.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.746+0000 7f515affd640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f513c06c8f0 0x7f513c06edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f514c005c00 tx=0x7f514c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:16.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.750+0000 7f5153fff640 1 -- 172.21.15.44:0/297335916 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5154059dc0 con 0x7f515c15e4d0 2024-04-02T13:21:16.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.937+0000 7f5161904640 1 -- 172.21.15.44:0/297335916 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f515c0afd80 con 0x7f515c15e4d0 2024-04-02T13:21:16.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.938+0000 7f5153fff640 1 -- 172.21.15.44:0/297335916 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18899 (secure 0 0 0) 0x7f51540596c0 con 0x7f515c15e4d0 2024-04-02T13:21:16.939 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:16.940 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:08.061253+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.940+0000 7f5161904640 1 -- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f513c06c8f0 msgr2=0x7f513c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.940+0000 7f5161904640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f513c06c8f0 0x7f513c06edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f514c005c00 tx=0x7f514c009040 comp rx=0 tx=0).stop 2024-04-02T13:21:16.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.940+0000 7f5161904640 1 -- 172.21.15.44:0/297335916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f515c15e4d0 msgr2=0x7f515c147b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:16.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.940+0000 7f5161904640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f515c15e4d0 0x7f515c147b90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f515400c410 tx=0x7f515400c8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.940+0000 7f5161904640 1 -- 172.21.15.44:0/297335916 shutdown_connections 2024-04-02T13:21:16.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.940+0000 7f5161904640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f513c06c8f0 0x7f513c06edb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.940+0000 7f5161904640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f515c15e4d0 0x7f515c147b90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.940+0000 7f5161904640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f515c15e0f0 0x7f515c0b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.940+0000 7f5161904640 1 --2- 172.21.15.44:0/297335916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f515c072f60 0x7f515c143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:16.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.940+0000 7f5161904640 1 -- 172.21.15.44:0/297335916 >> 172.21.15.44:0/297335916 conn(0x7f515c0b2980 msgr2=0x7f515c0b1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:16.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.940+0000 7f5161904640 1 -- 172.21.15.44:0/297335916 shutdown_connections 2024-04-02T13:21:16.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:16.940+0000 7f5161904640 1 -- 172.21.15.44:0/297335916 wait complete. 2024-04-02T13:21:16.953 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:16.953 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:17.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.056+0000 7f998c082640 1 Processor -- start 2024-04-02T13:21:17.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.056+0000 7f998c082640 1 -- start start 2024-04-02T13:21:17.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f998c082640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f998415e5d0 0x7f99840d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f998c082640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f99840b4180 0x7f99840b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f998c082640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99840d1630 con 0x7f998406cff0 2024-04-02T13:21:17.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f998c082640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f99840d1bf0 con 0x7f99840b4180 2024-04-02T13:21:17.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f998c082640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f99840d1910 con 0x7f998415e5d0 2024-04-02T13:21:17.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f9989e1e640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f998406cff0 0x7f998406d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48852/0 (socket says 172.21.15.44:48852) 2024-04-02T13:21:17.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f9989e1e640 1 -- 172.21.15.44:0/2222360145 learned_addr learned my addr 172.21.15.44:0/2222360145 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:17.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f998961d640 1 --2- 172.21.15.44:0/2222360145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f998415e5d0 0x7f99840d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f998a61f640 1 --2- 172.21.15.44:0/2222360145 >> v2:172.21.15.69:3300/0 conn(0x7f99840b4180 0x7f99840b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1749895579 0 0) 0x7f99840d1630 con 0x7f998406cff0 2024-04-02T13:21:17.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9970003520 con 0x7f998406cff0 2024-04-02T13:21:17.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1824082629 0 0) 0x7f9970003520 con 0x7f998406cff0 2024-04-02T13:21:17.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f998415e5d0 msgr2=0x7f99840d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:17.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f9988e1c640 1 --2- 172.21.15.44:0/2222360145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f998415e5d0 0x7f99840d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 >> v2:172.21.15.69:3300/0 conn(0x7f99840b4180 msgr2=0x7f99840b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:17.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.057+0000 7f9988e1c640 1 --2- 172.21.15.44:0/2222360145 >> v2:172.21.15.69:3300/0 conn(0x7f99840b4180 0x7f99840b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99840b1540 con 0x7f998406cff0 2024-04-02T13:21:17.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f9980001670 con 0x7f998406cff0 2024-04-02T13:21:17.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 >> v1:172.21.15.44:6789/0 conn(0x7f998406cff0 legacy=0x7f998406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f9988e1c640 1 --2- 172.21.15.44:0/2222360145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99700039b0 0x7f9970003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f9988e1c640 1 --2- 172.21.15.44:0/2222360145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9970004800 0x7f9970004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f9988e1c640 1 --2- 172.21.15.44:0/2222360145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f998415e5d0 0x7f99700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f99840d1bf0 con 0x7f9970004800 2024-04-02T13:21:17.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f99840d1910 con 0x7f99700039b0 2024-04-02T13:21:17.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f997000b9c0 con 0x7f998415e5d0 2024-04-02T13:21:17.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f998961d640 1 --2- 172.21.15.44:0/2222360145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9970004800 0x7f9970004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f9989e1e640 1 --2- 172.21.15.44:0/2222360145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99700039b0 0x7f9970003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f998a61f640 1 --2- 172.21.15.44:0/2222360145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f998415e5d0 0x7f99700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f998961d640 1 -- 172.21.15.44:0/2222360145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f998415e5d0 msgr2=0x7f99700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:17.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f998961d640 1 --2- 172.21.15.44:0/2222360145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f998415e5d0 0x7f99700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f998961d640 1 -- 172.21.15.44:0/2222360145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99700039b0 msgr2=0x7f9970003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:17.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f998961d640 1 --2- 172.21.15.44:0/2222360145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99700039b0 0x7f9970003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.058+0000 7f998961d640 1 -- 172.21.15.44:0/2222360145 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9970003520 con 0x7f9970004800 2024-04-02T13:21:17.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.059+0000 7f998961d640 1 --2- 172.21.15.44:0/2222360145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9970004800 0x7f9970004c80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9974009ac0 tx=0x7f9970005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c3f695d9d6c4b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:17.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.059+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974039070 con 0x7f9970004800 2024-04-02T13:21:17.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.059+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f997402c680 con 0x7f9970004800 2024-04-02T13:21:17.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.059+0000 7f9988e1c640 1 -- 172.21.15.44:0/2222360145 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974009690 con 0x7f9970004800 2024-04-02T13:21:17.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.059+0000 7f998c082640 1 -- 172.21.15.44:0/2222360145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9970004800 msgr2=0x7f9970004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.059+0000 7f998c082640 1 --2- 172.21.15.44:0/2222360145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9970004800 0x7f9970004c80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9974009ac0 tx=0x7f9970005830 comp rx=0 tx=0).stop 2024-04-02T13:21:17.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.059+0000 7f998a61f640 1 -- 172.21.15.44:0/2222360145 reap_dead start 2024-04-02T13:21:17.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.059+0000 7f998c082640 1 -- 172.21.15.44:0/2222360145 shutdown_connections 2024-04-02T13:21:17.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.059+0000 7f998c082640 1 -- 172.21.15.44:0/2222360145 >> 172.21.15.44:0/2222360145 conn(0x7f998409aa20 msgr2=0x7f998409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:17.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.060+0000 7f998c082640 1 -- 172.21.15.44:0/2222360145 shutdown_connections 2024-04-02T13:21:17.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.060+0000 7f998c082640 1 -- 172.21.15.44:0/2222360145 wait complete. 2024-04-02T13:21:17.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.061+0000 7f998c082640 1 Processor -- start 2024-04-02T13:21:17.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.061+0000 7f998c082640 1 -- start start 2024-04-02T13:21:17.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.061+0000 7f998c082640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9970004800 0x7f998415e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.061+0000 7f998c082640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9984153340 0x7f9984153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.061+0000 7f998c082640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99840b6e00 0x7f99840b7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.061+0000 7f998c082640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f99840d0eb0 con 0x7f9984153340 2024-04-02T13:21:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.061+0000 7f998c082640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f99840d0d30 con 0x7f99840b6e00 2024-04-02T13:21:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.061+0000 7f998c082640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f99840d1030 con 0x7f9970004800 2024-04-02T13:21:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.061+0000 7f998961d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9984153340 0x7f9984153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.061+0000 7f998961d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9984153340 0x7f9984153740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45784/0 (socket says 172.21.15.44:45784) 2024-04-02T13:21:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.062+0000 7f998961d640 1 -- 172.21.15.44:0/3999675049 learned_addr learned my addr 172.21.15.44:0/3999675049 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.062+0000 7f998961d640 1 -- 172.21.15.44:0/3999675049 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9970004800 msgr2=0x7f998415e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.062+0000 7f998961d640 1 --2- 172.21.15.44:0/3999675049 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9970004800 0x7f998415e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.062+0000 7f998961d640 1 -- 172.21.15.44:0/3999675049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99840b6e00 msgr2=0x7f99840b7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.062+0000 7f998961d640 1 --2- 172.21.15.44:0/3999675049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99840b6e00 0x7f99840b7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.062+0000 7f998961d640 1 -- 172.21.15.44:0/3999675049 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99840b6360 con 0x7f9984153340 2024-04-02T13:21:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.062+0000 7f998961d640 1 --2- 172.21.15.44:0/3999675049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9984153340 0x7f9984153740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9974000c00 tx=0x7f9974004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.062+0000 7f997a7fc640 1 -- 172.21.15.44:0/3999675049 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974042020 con 0x7f9984153340 2024-04-02T13:21:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.062+0000 7f997a7fc640 1 -- 172.21.15.44:0/3999675049 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f997403a040 con 0x7f9984153340 2024-04-02T13:21:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.062+0000 7f998c082640 1 -- 172.21.15.44:0/3999675049 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99840c7790 con 0x7f9984153340 2024-04-02T13:21:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.062+0000 7f997a7fc640 1 -- 172.21.15.44:0/3999675049 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974039070 con 0x7f9984153340 2024-04-02T13:21:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.062+0000 7f998c082640 1 -- 172.21.15.44:0/3999675049 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9984152d80 con 0x7f9984153340 2024-04-02T13:21:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.063+0000 7f998c082640 1 -- 172.21.15.44:0/3999675049 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f998406c7c0 con 0x7f9984153340 2024-04-02T13:21:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.064+0000 7f997a7fc640 1 -- 172.21.15.44:0/3999675049 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9974045020 con 0x7f9984153340 2024-04-02T13:21:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.064+0000 7f997a7fc640 1 --2- 172.21.15.44:0/3999675049 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f996806c920 0x7f996806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.064+0000 7f997a7fc640 1 -- 172.21.15.44:0/3999675049 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f99740ad2c0 con 0x7f9984153340 2024-04-02T13:21:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.064+0000 7f9989e1e640 1 --2- 172.21.15.44:0/3999675049 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f996806c920 0x7f996806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.065+0000 7f9989e1e640 1 --2- 172.21.15.44:0/3999675049 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f996806c920 0x7f996806ede0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9980001be0 tx=0x7f9980001b70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:17.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.069+0000 7f997a7fc640 1 -- 172.21.15.44:0/3999675049 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f998406c7c0 con 0x7f9984153340 2024-04-02T13:21:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.252+0000 7f998c082640 1 -- 172.21.15.44:0/3999675049 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f998406cdd0 con 0x7f9984153340 2024-04-02T13:21:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.252+0000 7f997a7fc640 1 -- 172.21.15.44:0/3999675049 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f997404c070 con 0x7f9984153340 2024-04-02T13:21:17.254 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:17.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.254+0000 7f998c082640 1 -- 172.21.15.44:0/3999675049 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f996806c920 msgr2=0x7f996806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.254+0000 7f998c082640 1 --2- 172.21.15.44:0/3999675049 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f996806c920 0x7f996806ede0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9980001be0 tx=0x7f9980001b70 comp rx=0 tx=0).stop 2024-04-02T13:21:17.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.254+0000 7f998c082640 1 -- 172.21.15.44:0/3999675049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9984153340 msgr2=0x7f9984153740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.254+0000 7f998c082640 1 --2- 172.21.15.44:0/3999675049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9984153340 0x7f9984153740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9974000c00 tx=0x7f9974004730 comp rx=0 tx=0).stop 2024-04-02T13:21:17.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.254+0000 7f998c082640 1 -- 172.21.15.44:0/3999675049 shutdown_connections 2024-04-02T13:21:17.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.254+0000 7f998c082640 1 --2- 172.21.15.44:0/3999675049 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f996806c920 0x7f996806ede0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.254+0000 7f998c082640 1 --2- 172.21.15.44:0/3999675049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99840b6e00 0x7f99840b7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.254+0000 7f998c082640 1 --2- 172.21.15.44:0/3999675049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9984153340 0x7f9984153740 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.254+0000 7f998c082640 1 --2- 172.21.15.44:0/3999675049 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9970004800 0x7f998415e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.254+0000 7f998c082640 1 -- 172.21.15.44:0/3999675049 >> 172.21.15.44:0/3999675049 conn(0x7f998409aa20 msgr2=0x7f998415d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.254+0000 7f998c082640 1 -- 172.21.15.44:0/3999675049 shutdown_connections 2024-04-02T13:21:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.254+0000 7f998c082640 1 -- 172.21.15.44:0/3999675049 wait complete. 2024-04-02T13:21:17.267 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.370+0000 7fdc60b15640 1 Processor -- start 2024-04-02T13:21:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.370+0000 7fdc60b15640 1 -- start start 2024-04-02T13:21:17.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc60b15640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdc5c15e470 0x7fdc5c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc60b15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c0ce790 0x7fdc5c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc60b15640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc5c0ced00 con 0x7fdc5c0b1540 2024-04-02T13:21:17.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc60b15640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fdc5c0cefe0 con 0x7fdc5c15e470 2024-04-02T13:21:17.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc60b15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdc5c0cf2c0 con 0x7fdc5c0ce790 2024-04-02T13:21:17.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc5a59c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdc5c15e470 0x7fdc5c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc59d9b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fdc5c0b1540 0x7fdc5c0d1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48864/0 (socket says 172.21.15.44:48864) 2024-04-02T13:21:17.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc59d9b640 1 -- 172.21.15.44:0/2346146314 learned_addr learned my addr 172.21.15.44:0/2346146314 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:17.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc5ad9d640 1 --2- 172.21.15.44:0/2346146314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c0ce790 0x7fdc5c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc5a59c640 1 -- 172.21.15.44:0/2346146314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c0ce790 msgr2=0x7fdc5c06d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:17.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc5a59c640 1 --2- 172.21.15.44:0/2346146314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c0ce790 0x7fdc5c06d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc5a59c640 1 -- 172.21.15.44:0/2346146314 >> v1:172.21.15.44:6789/0 conn(0x7fdc5c0b1540 legacy=0x7fdc5c0d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc5a59c640 1 -- 172.21.15.44:0/2346146314 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc5c0cf5a0 con 0x7fdc5c15e470 2024-04-02T13:21:17.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.371+0000 7fdc5959a640 1 -- 172.21.15.44:0/2346146314 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 407303724 0 0) 0x7fdc5c0ced00 con 0x7fdc5c0b1540 2024-04-02T13:21:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.372+0000 7fdc5a59c640 1 --2- 172.21.15.44:0/2346146314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc5c15e470 0x7fdc5c0b32c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0097a0 tx=0x7fdc4c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720aaeaa6e797a02 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.372+0000 7fdc5959a640 1 -- 172.21.15.44:0/2346146314 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c0315a0 con 0x7fdc5c15e470 2024-04-02T13:21:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.372+0000 7fdc5959a640 1 -- 172.21.15.44:0/2346146314 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc4c037070 con 0x7fdc5c15e470 2024-04-02T13:21:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.372+0000 7fdc5959a640 1 -- 172.21.15.44:0/2346146314 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c031960 con 0x7fdc5c15e470 2024-04-02T13:21:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.372+0000 7fdc60b15640 1 -- 172.21.15.44:0/2346146314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc5c15e470 msgr2=0x7fdc5c0b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.372+0000 7fdc60b15640 1 --2- 172.21.15.44:0/2346146314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc5c15e470 0x7fdc5c0b32c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0097a0 tx=0x7fdc4c02b840 comp rx=0 tx=0).stop 2024-04-02T13:21:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.373+0000 7fdc60b15640 1 -- 172.21.15.44:0/2346146314 shutdown_connections 2024-04-02T13:21:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.373+0000 7fdc60b15640 1 --2- 172.21.15.44:0/2346146314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c0ce790 0x7fdc5c06d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.373+0000 7fdc60b15640 1 --2- 172.21.15.44:0/2346146314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc5c15e470 0x7fdc5c0b32c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.373+0000 7fdc60b15640 1 -- 172.21.15.44:0/2346146314 >> 172.21.15.44:0/2346146314 conn(0x7fdc5c09aa20 msgr2=0x7fdc5c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.373+0000 7fdc60b15640 1 -- 172.21.15.44:0/2346146314 shutdown_connections 2024-04-02T13:21:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.373+0000 7fdc60b15640 1 -- 172.21.15.44:0/2346146314 wait complete. 2024-04-02T13:21:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.374+0000 7fdc60b15640 1 Processor -- start 2024-04-02T13:21:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.374+0000 7fdc60b15640 1 -- start start 2024-04-02T13:21:17.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc60b15640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc5c0b1540 0x7fdc5c0b7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc60b15640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5c0ce790 0x7fdc5c0b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc60b15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c15e470 0x7fdc5c0b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc60b15640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5c0d22c0 con 0x7fdc5c0b1540 2024-04-02T13:21:17.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc60b15640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5c06cb60 con 0x7fdc5c0ce790 2024-04-02T13:21:17.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc60b15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdc5c0d2440 con 0x7fdc5c15e470 2024-04-02T13:21:17.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc5a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc5c0b1540 0x7fdc5c0b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc5a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc5c0b1540 0x7fdc5c0b7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45806/0 (socket says 172.21.15.44:45806) 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc5a59c640 1 -- 172.21.15.44:0/2566245230 learned_addr learned my addr 172.21.15.44:0/2566245230 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc5a59c640 1 -- 172.21.15.44:0/2566245230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c15e470 msgr2=0x7fdc5c0b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc5a59c640 1 --2- 172.21.15.44:0/2566245230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c15e470 0x7fdc5c0b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc5a59c640 1 -- 172.21.15.44:0/2566245230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5c0ce790 msgr2=0x7fdc5c0b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc5a59c640 1 --2- 172.21.15.44:0/2566245230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5c0ce790 0x7fdc5c0b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc5a59c640 1 -- 172.21.15.44:0/2566245230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc5c0c9ac0 con 0x7fdc5c0b1540 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc5a59c640 1 --2- 172.21.15.44:0/2566245230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc5c0b1540 0x7fdc5c0b7210 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0097a0 tx=0x7fdc4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc4affd640 1 -- 172.21.15.44:0/2566245230 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c004230 con 0x7fdc5c0b1540 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc4affd640 1 -- 172.21.15.44:0/2566245230 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc4c0043d0 con 0x7fdc5c0b1540 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc60b15640 1 -- 172.21.15.44:0/2566245230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc5c0c9da0 con 0x7fdc5c0b1540 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc4affd640 1 -- 172.21.15.44:0/2566245230 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c003850 con 0x7fdc5c0b1540 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.375+0000 7fdc60b15640 1 -- 172.21.15.44:0/2566245230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc5c0b7750 con 0x7fdc5c0b1540 2024-04-02T13:21:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.376+0000 7fdc60b15640 1 -- 172.21.15.44:0/2566245230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc5c0d1110 con 0x7fdc5c0b1540 2024-04-02T13:21:17.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.377+0000 7fdc4affd640 1 -- 172.21.15.44:0/2566245230 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdc4c0039f0 con 0x7fdc5c0b1540 2024-04-02T13:21:17.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.377+0000 7fdc4affd640 1 --2- 172.21.15.44:0/2566245230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc3c06c8f0 0x7fdc3c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.377+0000 7fdc4affd640 1 -- 172.21.15.44:0/2566245230 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fdc4c02f030 con 0x7fdc5c0b1540 2024-04-02T13:21:17.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.377+0000 7fdc59d9b640 1 --2- 172.21.15.44:0/2566245230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc3c06c8f0 0x7fdc3c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.378+0000 7fdc59d9b640 1 --2- 172.21.15.44:0/2566245230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc3c06c8f0 0x7fdc3c06edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdc50007fd0 tx=0x7fdc50007960 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:17.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.381+0000 7fdc4affd640 1 -- 172.21.15.44:0/2566245230 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fdc5c0d1110 con 0x7fdc5c0b1540 2024-04-02T13:21:17.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.563+0000 7fdc60b15640 1 -- 172.21.15.44:0/2566245230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdc5c06b8a0 con 0x7fdc5c0b1540 2024-04-02T13:21:17.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.564+0000 7fdc4affd640 1 -- 172.21.15.44:0/2566245230 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fdc5c06b8a0 con 0x7fdc5c0b1540 2024-04-02T13:21:17.565 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:17.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.565+0000 7fdc60b15640 1 -- 172.21.15.44:0/2566245230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc3c06c8f0 msgr2=0x7fdc3c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.565+0000 7fdc60b15640 1 --2- 172.21.15.44:0/2566245230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc3c06c8f0 0x7fdc3c06edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdc50007fd0 tx=0x7fdc50007960 comp rx=0 tx=0).stop 2024-04-02T13:21:17.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.565+0000 7fdc60b15640 1 -- 172.21.15.44:0/2566245230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc5c0b1540 msgr2=0x7fdc5c0b7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.565+0000 7fdc60b15640 1 --2- 172.21.15.44:0/2566245230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc5c0b1540 0x7fdc5c0b7210 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0097a0 tx=0x7fdc4c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.565+0000 7fdc60b15640 1 -- 172.21.15.44:0/2566245230 shutdown_connections 2024-04-02T13:21:17.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.565+0000 7fdc60b15640 1 --2- 172.21.15.44:0/2566245230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc3c06c8f0 0x7fdc3c06edb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.565+0000 7fdc60b15640 1 --2- 172.21.15.44:0/2566245230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c15e470 0x7fdc5c0b3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.565+0000 7fdc60b15640 1 --2- 172.21.15.44:0/2566245230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5c0ce790 0x7fdc5c0b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.565+0000 7fdc60b15640 1 --2- 172.21.15.44:0/2566245230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc5c0b1540 0x7fdc5c0b7210 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.565+0000 7fdc60b15640 1 -- 172.21.15.44:0/2566245230 >> 172.21.15.44:0/2566245230 conn(0x7fdc5c09aa20 msgr2=0x7fdc5c0af520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:17.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.565+0000 7fdc60b15640 1 -- 172.21.15.44:0/2566245230 shutdown_connections 2024-04-02T13:21:17.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.565+0000 7fdc60b15640 1 -- 172.21.15.44:0/2566245230 wait complete. 2024-04-02T13:21:17.577 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:17.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.681+0000 7f4bde87b640 1 Processor -- start 2024-04-02T13:21:17.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.681+0000 7f4bde87b640 1 -- start start 2024-04-02T13:21:17.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.681+0000 7f4bde87b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4bd815e260 0x7f4bd80d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.681+0000 7f4bde87b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4bd80cec60 0x7f4bd8168280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.681+0000 7f4bde87b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bd80cf120 con 0x7f4bd815e640 2024-04-02T13:21:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.681+0000 7f4bde87b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4bd80b5a50 con 0x7f4bd80cec60 2024-04-02T13:21:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.682+0000 7f4bde87b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4bd80d01c0 con 0x7f4bd815e260 2024-04-02T13:21:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.682+0000 7f4bdce18640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4bd80cec60 0x7f4bd8168280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.682+0000 7f4bd77fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4bd815e640 0x7f4bd80d0ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48866/0 (socket says 172.21.15.44:48866) 2024-04-02T13:21:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.682+0000 7f4bd77fe640 1 -- 172.21.15.44:0/450798797 learned_addr learned my addr 172.21.15.44:0/450798797 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.682+0000 7f4bd7fff640 1 --2- 172.21.15.44:0/450798797 >> v2:172.21.15.69:3300/0 conn(0x7f4bd815e260 0x7f4bd80d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.682+0000 7f4bdce18640 1 -- 172.21.15.44:0/450798797 >> v2:172.21.15.69:3300/0 conn(0x7f4bd815e260 msgr2=0x7f4bd80d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.682+0000 7f4bdce18640 1 --2- 172.21.15.44:0/450798797 >> v2:172.21.15.69:3300/0 conn(0x7f4bd815e260 0x7f4bd80d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.682+0000 7f4bdce18640 1 -- 172.21.15.44:0/450798797 >> v1:172.21.15.44:6789/0 conn(0x7f4bd815e640 legacy=0x7f4bd80d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.682+0000 7f4bdce18640 1 -- 172.21.15.44:0/450798797 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd80cf400 con 0x7f4bd80cec60 2024-04-02T13:21:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.682+0000 7f4bd6ffd640 1 -- 172.21.15.44:0/450798797 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 908504667 0 0) 0x7f4bd80cf120 con 0x7f4bd815e640 2024-04-02T13:21:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.683+0000 7f4bdce18640 1 --2- 172.21.15.44:0/450798797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd80cec60 0x7f4bd8168280 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4bd809d8f0 tx=0x7f4bd002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5448bd7f963c228 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.683+0000 7f4bd6ffd640 1 -- 172.21.15.44:0/450798797 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0039070 con 0x7f4bd80cec60 2024-04-02T13:21:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.683+0000 7f4bd6ffd640 1 -- 172.21.15.44:0/450798797 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd002cbf0 con 0x7f4bd80cec60 2024-04-02T13:21:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.683+0000 7f4bd6ffd640 1 -- 172.21.15.44:0/450798797 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0034440 con 0x7f4bd80cec60 2024-04-02T13:21:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.683+0000 7f4bde87b640 1 -- 172.21.15.44:0/450798797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd80cec60 msgr2=0x7f4bd8168280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.683+0000 7f4bde87b640 1 --2- 172.21.15.44:0/450798797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd80cec60 0x7f4bd8168280 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4bd809d8f0 tx=0x7f4bd002be40 comp rx=0 tx=0).stop 2024-04-02T13:21:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.683+0000 7f4bde87b640 1 -- 172.21.15.44:0/450798797 shutdown_connections 2024-04-02T13:21:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.683+0000 7f4bde87b640 1 --2- 172.21.15.44:0/450798797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd80cec60 0x7f4bd8168280 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.683+0000 7f4bde87b640 1 --2- 172.21.15.44:0/450798797 >> v2:172.21.15.69:3300/0 conn(0x7f4bd815e260 0x7f4bd80d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.683+0000 7f4bde87b640 1 -- 172.21.15.44:0/450798797 >> 172.21.15.44:0/450798797 conn(0x7f4bd809d3c0 msgr2=0x7f4bd80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.683+0000 7f4bde87b640 1 -- 172.21.15.44:0/450798797 shutdown_connections 2024-04-02T13:21:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.683+0000 7f4bde87b640 1 -- 172.21.15.44:0/450798797 wait complete. 2024-04-02T13:21:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.685+0000 7f4bde87b640 1 Processor -- start 2024-04-02T13:21:17.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.685+0000 7f4bde87b640 1 -- start start 2024-04-02T13:21:17.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.685+0000 7f4bde87b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bd80cec60 0x7f4bd8161700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.685+0000 7f4bde87b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd815e260 0x7f4bd80cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.685+0000 7f4bde87b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd815e640 0x7f4bd80b7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.685+0000 7f4bde87b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd80cfca0 con 0x7f4bd815e640 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.685+0000 7f4bde87b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd80cf9c0 con 0x7f4bd815e260 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.685+0000 7f4bde87b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4bd80d11e0 con 0x7f4bd80cec60 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bdce18640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd815e640 0x7f4bd80b7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bdce18640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd815e640 0x7f4bd80b7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45830/0 (socket says 172.21.15.44:45830) 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bdce18640 1 -- 172.21.15.44:0/2068200965 learned_addr learned my addr 172.21.15.44:0/2068200965 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bd77fe640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd815e260 0x7f4bd80cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bd7fff640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bd80cec60 0x7f4bd8161700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bdce18640 1 -- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bd80cec60 msgr2=0x7f4bd8161700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bdce18640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bd80cec60 0x7f4bd8161700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bdce18640 1 -- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd815e260 msgr2=0x7f4bd80cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bdce18640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd815e260 0x7f4bd80cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bdce18640 1 -- 172.21.15.44:0/2068200965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd80cd2e0 con 0x7f4bd815e640 2024-04-02T13:21:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bdce18640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd815e640 0x7f4bd80b7d50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0037040 tx=0x7f4bd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bd4ff9640 1 -- 172.21.15.44:0/2068200965 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0039050 con 0x7f4bd815e640 2024-04-02T13:21:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bd4ff9640 1 -- 172.21.15.44:0/2068200965 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd0045070 con 0x7f4bd815e640 2024-04-02T13:21:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bde87b640 1 -- 172.21.15.44:0/2068200965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd80cd5c0 con 0x7f4bd815e640 2024-04-02T13:21:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bd4ff9640 1 -- 172.21.15.44:0/2068200965 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd003e810 con 0x7f4bd815e640 2024-04-02T13:21:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.686+0000 7f4bde87b640 1 -- 172.21.15.44:0/2068200965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd80ccf40 con 0x7f4bd815e640 2024-04-02T13:21:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.687+0000 7f4bde87b640 1 -- 172.21.15.44:0/2068200965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd80cd9b0 con 0x7f4bd815e640 2024-04-02T13:21:17.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.688+0000 7f4bd4ff9640 1 -- 172.21.15.44:0/2068200965 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4bd003e9b0 con 0x7f4bd815e640 2024-04-02T13:21:17.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.688+0000 7f4bd4ff9640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4bc406c8f0 0x7f4bc406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.688+0000 7f4bd4ff9640 1 -- 172.21.15.44:0/2068200965 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f4bd00aed30 con 0x7f4bd815e640 2024-04-02T13:21:17.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.688+0000 7f4bd7fff640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4bc406c8f0 0x7f4bc406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.689+0000 7f4bd7fff640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4bc406c8f0 0x7f4bc406edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8006fd0 tx=0x7f4bc8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:17.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.693+0000 7f4bd4ff9640 1 -- 172.21.15.44:0/2068200965 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4bd007cdd0 con 0x7f4bd815e640 2024-04-02T13:21:17.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.876+0000 7f4bde87b640 1 -- 172.21.15.44:0/2068200965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4bd80cd790 con 0x7f4bd815e640 2024-04-02T13:21:17.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.876+0000 7f4bd4ff9640 1 -- 172.21.15.44:0/2068200965 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f4bd007c6d0 con 0x7f4bd815e640 2024-04-02T13:21:17.878 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:17.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.878+0000 7f4bde87b640 1 -- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4bc406c8f0 msgr2=0x7f4bc406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.878+0000 7f4bde87b640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4bc406c8f0 0x7f4bc406edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8006fd0 tx=0x7f4bc8008040 comp rx=0 tx=0).stop 2024-04-02T13:21:17.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.878+0000 7f4bde87b640 1 -- 172.21.15.44:0/2068200965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd815e640 msgr2=0x7f4bd80b7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.878+0000 7f4bde87b640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd815e640 0x7f4bd80b7d50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0037040 tx=0x7f4bd0004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:17.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.878+0000 7f4bde87b640 1 -- 172.21.15.44:0/2068200965 shutdown_connections 2024-04-02T13:21:17.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.878+0000 7f4bde87b640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4bc406c8f0 0x7f4bc406edb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.878+0000 7f4bde87b640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd815e640 0x7f4bd80b7d50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.878+0000 7f4bde87b640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd815e260 0x7f4bd80cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.878+0000 7f4bde87b640 1 --2- 172.21.15.44:0/2068200965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bd80cec60 0x7f4bd8161700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.878+0000 7f4bde87b640 1 -- 172.21.15.44:0/2068200965 >> 172.21.15.44:0/2068200965 conn(0x7f4bd809d3c0 msgr2=0x7f4bd815d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:17.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.878+0000 7f4bde87b640 1 -- 172.21.15.44:0/2068200965 shutdown_connections 2024-04-02T13:21:17.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.878+0000 7f4bde87b640 1 -- 172.21.15.44:0/2068200965 wait complete. 2024-04-02T13:21:17.891 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:17.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.993+0000 7f328e4f2640 1 Processor -- start 2024-04-02T13:21:17.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.993+0000 7f328e4f2640 1 -- start start 2024-04-02T13:21:17.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.994+0000 7f328e4f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32880d2dd0 0x7f32880b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.994+0000 7f328e4f2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32880cf1f0 con 0x7f328815e1d0 2024-04-02T13:21:17.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.994+0000 7f328e4f2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32880cf4d0 con 0x7f32880cec60 2024-04-02T13:21:17.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.994+0000 7f328e4f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f32880cfe00 con 0x7f32880d2dd0 2024-04-02T13:21:17.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.994+0000 7f3287fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f328815e1d0 0x7f32880d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48880/0 (socket says 172.21.15.44:48880) 2024-04-02T13:21:17.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.994+0000 7f3287fff640 1 -- 172.21.15.44:0/1504614378 learned_addr learned my addr 172.21.15.44:0/1504614378 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:17.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2162134371 0 0) 0x7f32880cf1f0 con 0x7f328815e1d0 2024-04-02T13:21:17.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3270003520 con 0x7f328815e1d0 2024-04-02T13:21:17.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f328ca8f640 1 --2- 172.21.15.44:0/1504614378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32880d2dd0 0x7f32880b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1567827619 0 0) 0x7f3270003520 con 0x7f328815e1d0 2024-04-02T13:21:17.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32880d2dd0 msgr2=0x7f32880b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:17.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 --2- 172.21.15.44:0/1504614378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32880d2dd0 0x7f32880b4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:17.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 >> v1:172.21.15.69:6789/0 conn(0x7f32880cec60 legacy=0x7f32880d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32880cf7b0 con 0x7f328815e1d0 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f3278001670 con 0x7f328815e1d0 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f328ca8f640 1 --2- 172.21.15.44:0/1504614378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32880d2dd0 0x7f32880b4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 >> v1:172.21.15.44:6789/0 conn(0x7f328815e1d0 legacy=0x7f32880d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 --2- 172.21.15.44:0/1504614378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32700039e0 0x7f3270003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 --2- 172.21.15.44:0/1504614378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32880d2dd0 0x7f3270004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 --2- 172.21.15.44:0/1504614378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3270004d40 0x7f32700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f32880cfe00 con 0x7f3270004d40 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f327000bb80 con 0x7f32700039e0 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f327000bd80 con 0x7f32880d2dd0 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f328ca8f640 1 --2- 172.21.15.44:0/1504614378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3270004d40 0x7f32700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f3287fff640 1 --2- 172.21.15.44:0/1504614378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32700039e0 0x7f3270003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f32877fe640 1 --2- 172.21.15.44:0/1504614378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32880d2dd0 0x7f3270004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:17.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f328ca8f640 1 -- 172.21.15.44:0/1504614378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32880d2dd0 msgr2=0x7f3270004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:18.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f328ca8f640 1 --2- 172.21.15.44:0/1504614378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32880d2dd0 0x7f3270004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f328ca8f640 1 -- 172.21.15.44:0/1504614378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32700039e0 msgr2=0x7f3270003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:18.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f328ca8f640 1 --2- 172.21.15.44:0/1504614378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32700039e0 0x7f3270003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.995+0000 7f328ca8f640 1 -- 172.21.15.44:0/1504614378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3270003520 con 0x7f3270004d40 2024-04-02T13:21:18.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.996+0000 7f328ca8f640 1 --2- 172.21.15.44:0/1504614378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3270004d40 0x7f32700051c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f32800073e0 tx=0x7f3270005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f897290735d7e75 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:18.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.996+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3280039070 con 0x7f3270004d40 2024-04-02T13:21:18.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.996+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f328002cd30 con 0x7f3270004d40 2024-04-02T13:21:18.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.996+0000 7f3286ffd640 1 -- 172.21.15.44:0/1504614378 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3280009c80 con 0x7f3270004d40 2024-04-02T13:21:18.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.997+0000 7f328e4f2640 1 -- 172.21.15.44:0/1504614378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3270004d40 msgr2=0x7f32700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.997+0000 7f328e4f2640 1 --2- 172.21.15.44:0/1504614378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3270004d40 0x7f32700051c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f32800073e0 tx=0x7f3270005830 comp rx=0 tx=0).stop 2024-04-02T13:21:18.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.997+0000 7f328ca8f640 1 -- 172.21.15.44:0/1504614378 reap_dead start 2024-04-02T13:21:18.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.997+0000 7f328e4f2640 1 -- 172.21.15.44:0/1504614378 shutdown_connections 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.997+0000 7f328e4f2640 1 -- 172.21.15.44:0/1504614378 >> 172.21.15.44:0/1504614378 conn(0x7f328809d3b0 msgr2=0x7f32880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.997+0000 7f328e4f2640 1 -- 172.21.15.44:0/1504614378 shutdown_connections 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.997+0000 7f328e4f2640 1 -- 172.21.15.44:0/1504614378 wait complete. 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.998+0000 7f328e4f2640 1 Processor -- start 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.998+0000 7f328e4f2640 1 -- start start 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328e4f2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3288153b30 0x7f3288158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328e4f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3288153f10 0x7f328815a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328e4f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328815a7b0 0x7f32880cc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328e4f2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f32880b8150 con 0x7f328815a7b0 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328e4f2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f32880b7fd0 con 0x7f3288153b30 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328e4f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f32880b82d0 con 0x7f3288153f10 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328ca8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328815a7b0 0x7f32880cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328ca8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328815a7b0 0x7f32880cc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45834/0 (socket says 172.21.15.44:45834) 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328ca8f640 1 -- 172.21.15.44:0/1146519361 learned_addr learned my addr 172.21.15.44:0/1146519361 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:18.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f3287fff640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3288153b30 0x7f3288158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f32877fe640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3288153f10 0x7f328815a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328ca8f640 1 -- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3288153f10 msgr2=0x7f328815a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328ca8f640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3288153f10 0x7f328815a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328ca8f640 1 -- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3288153b30 msgr2=0x7f3288158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328ca8f640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3288153b30 0x7f3288158270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328ca8f640 1 -- 172.21.15.44:0/1146519361 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3288158be0 con 0x7f328815a7b0 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f3287fff640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3288153b30 0x7f3288158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:17.999+0000 7f328ca8f640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328815a7b0 0x7f32880cc460 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f32800073e0 tx=0x7f3280004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.000+0000 7f3284ff9640 1 -- 172.21.15.44:0/1146519361 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3280039030 con 0x7f328815a7b0 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.000+0000 7f3284ff9640 1 -- 172.21.15.44:0/1146519361 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f328003a040 con 0x7f328815a7b0 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.000+0000 7f328e4f2640 1 -- 172.21.15.44:0/1146519361 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f328815f6a0 con 0x7f328815a7b0 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.000+0000 7f3284ff9640 1 -- 172.21.15.44:0/1146519361 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3280009c80 con 0x7f328815a7b0 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.000+0000 7f328e4f2640 1 -- 172.21.15.44:0/1146519361 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f328815fc60 con 0x7f328815a7b0 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.001+0000 7f328e4f2640 1 -- 172.21.15.44:0/1146519361 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32880684f0 con 0x7f328815a7b0 2024-04-02T13:21:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.001+0000 7f3284ff9640 1 -- 172.21.15.44:0/1146519361 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3280045020 con 0x7f328815a7b0 2024-04-02T13:21:18.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.002+0000 7f3284ff9640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f327406c920 0x7f327406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.002+0000 7f3284ff9640 1 -- 172.21.15.44:0/1146519361 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f32800ad3f0 con 0x7f328815a7b0 2024-04-02T13:21:18.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.002+0000 7f3287fff640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f327406c920 0x7f327406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.002+0000 7f3287fff640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f327406c920 0x7f327406ede0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3278000c00 tx=0x7f3278009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:18.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.006+0000 7f3284ff9640 1 -- 172.21.15.44:0/1146519361 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f328004c020 con 0x7f328815a7b0 2024-04-02T13:21:18.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.192+0000 7f328e4f2640 1 -- 172.21.15.44:0/1146519361 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f32880d47f0 con 0x7f328815a7b0 2024-04-02T13:21:18.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.192+0000 7f3284ff9640 1 -- 172.21.15.44:0/1146519361 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f328007b400 con 0x7f328815a7b0 2024-04-02T13:21:18.194 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:18.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.194+0000 7f328e4f2640 1 -- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f327406c920 msgr2=0x7f327406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.194+0000 7f328e4f2640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f327406c920 0x7f327406ede0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3278000c00 tx=0x7f3278009040 comp rx=0 tx=0).stop 2024-04-02T13:21:18.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.194+0000 7f328e4f2640 1 -- 172.21.15.44:0/1146519361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328815a7b0 msgr2=0x7f32880cc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.194+0000 7f328e4f2640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328815a7b0 0x7f32880cc460 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f32800073e0 tx=0x7f3280004ab0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.194+0000 7f328e4f2640 1 -- 172.21.15.44:0/1146519361 shutdown_connections 2024-04-02T13:21:18.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.194+0000 7f328e4f2640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f327406c920 0x7f327406ede0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.194+0000 7f328e4f2640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328815a7b0 0x7f32880cc460 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.194+0000 7f328e4f2640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3288153f10 0x7f328815a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.194+0000 7f328e4f2640 1 --2- 172.21.15.44:0/1146519361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3288153b30 0x7f3288158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.194+0000 7f328e4f2640 1 -- 172.21.15.44:0/1146519361 >> 172.21.15.44:0/1146519361 conn(0x7f328809d3b0 msgr2=0x7f3288161d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:18.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.194+0000 7f328e4f2640 1 -- 172.21.15.44:0/1146519361 shutdown_connections 2024-04-02T13:21:18.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.194+0000 7f328e4f2640 1 -- 172.21.15.44:0/1146519361 wait complete. 2024-04-02T13:21:18.207 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:21:18.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.308+0000 7ff29ef9d640 1 Processor -- start 2024-04-02T13:21:18.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.308+0000 7ff29ef9d640 1 -- start start 2024-04-02T13:21:18.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.309+0000 7ff29ef9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff2981629e0 0x7ff2980d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.309+0000 7ff29ef9d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff29806d1d0 con 0x7ff298162550 2024-04-02T13:21:18.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.309+0000 7ff29ef9d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff29806d4b0 con 0x7ff29815e270 2024-04-02T13:21:18.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.309+0000 7ff29ef9d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff29806dde0 con 0x7ff2981629e0 2024-04-02T13:21:18.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.309+0000 7ff29e79c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff2981629e0 0x7ff2980d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.309+0000 7ff29d79a640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff298162550 0x7ff2980b0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48894/0 (socket says 172.21.15.44:48894) 2024-04-02T13:21:18.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.309+0000 7ff29d79a640 1 -- 172.21.15.44:0/1977471785 learned_addr learned my addr 172.21.15.44:0/1977471785 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:18.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.309+0000 7ff29e79c640 1 -- 172.21.15.44:0/1977471785 >> v1:172.21.15.69:6789/0 conn(0x7ff29815e270 legacy=0x7ff298161e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:18.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.309+0000 7ff29e79c640 1 -- 172.21.15.44:0/1977471785 >> v1:172.21.15.44:6789/0 conn(0x7ff298162550 legacy=0x7ff2980b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.309+0000 7ff29cf99640 1 -- 172.21.15.44:0/1977471785 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2970304065 0 0) 0x7ff29806d1d0 con 0x7ff298162550 2024-04-02T13:21:18.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.309+0000 7ff29e79c640 1 -- 172.21.15.44:0/1977471785 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29806d790 con 0x7ff2981629e0 2024-04-02T13:21:18.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.310+0000 7ff29e79c640 1 --2- 172.21.15.44:0/1977471785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2981629e0 0x7ff2980d4a10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff2900097a0 tx=0x7ff29002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d9d7cb98673f53 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:18.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.310+0000 7ff29cf99640 1 -- 172.21.15.44:0/1977471785 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff290034070 con 0x7ff2981629e0 2024-04-02T13:21:18.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.310+0000 7ff29cf99640 1 -- 172.21.15.44:0/1977471785 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff290030610 con 0x7ff2981629e0 2024-04-02T13:21:18.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.310+0000 7ff29cf99640 1 -- 172.21.15.44:0/1977471785 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2900309e0 con 0x7ff2981629e0 2024-04-02T13:21:18.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.311+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1977471785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2981629e0 msgr2=0x7ff2980d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.311+0000 7ff29ef9d640 1 --2- 172.21.15.44:0/1977471785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2981629e0 0x7ff2980d4a10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff2900097a0 tx=0x7ff29002b960 comp rx=0 tx=0).stop 2024-04-02T13:21:18.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.311+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1977471785 shutdown_connections 2024-04-02T13:21:18.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.311+0000 7ff29ef9d640 1 --2- 172.21.15.44:0/1977471785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2981629e0 0x7ff2980d4a10 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.311+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1977471785 >> 172.21.15.44:0/1977471785 conn(0x7ff2980d3ef0 msgr2=0x7ff2980d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:18.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.311+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1977471785 shutdown_connections 2024-04-02T13:21:18.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.311+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1977471785 wait complete. 2024-04-02T13:21:18.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.312+0000 7ff29ef9d640 1 Processor -- start 2024-04-02T13:21:18.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.312+0000 7ff29ef9d640 1 -- start start 2024-04-02T13:21:18.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29ef9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff29815e270 0x7ff298158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29ef9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff298162550 0x7ff298160f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29ef9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2981629e0 0x7ff2980b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29ef9d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff29815e650 con 0x7ff29815e270 2024-04-02T13:21:18.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29ef9d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff29806cef0 con 0x7ff2981629e0 2024-04-02T13:21:18.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29ef9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff2980b1560 con 0x7ff298162550 2024-04-02T13:21:18.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29df9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff29815e270 0x7ff298158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29df9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff29815e270 0x7ff298158f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45858/0 (socket says 172.21.15.44:45858) 2024-04-02T13:21:18.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29df9b640 1 -- 172.21.15.44:0/1937959353 learned_addr learned my addr 172.21.15.44:0/1937959353 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:18.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29df9b640 1 -- 172.21.15.44:0/1937959353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff298162550 msgr2=0x7ff298160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29df9b640 1 --2- 172.21.15.44:0/1937959353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff298162550 0x7ff298160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29df9b640 1 -- 172.21.15.44:0/1937959353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2981629e0 msgr2=0x7ff2980b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29e79c640 1 --2- 172.21.15.44:0/1937959353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2981629e0 0x7ff2980b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29df9b640 1 --2- 172.21.15.44:0/1937959353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2981629e0 0x7ff2980b4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29df9b640 1 -- 172.21.15.44:0/1937959353 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2980b3bf0 con 0x7ff29815e270 2024-04-02T13:21:18.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29df9b640 1 --2- 172.21.15.44:0/1937959353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff29815e270 0x7ff298158f90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff28800c870 tx=0x7ff28800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:18.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff2967fc640 1 -- 172.21.15.44:0/1937959353 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288003b70 con 0x7ff29815e270 2024-04-02T13:21:18.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff2967fc640 1 -- 172.21.15.44:0/1937959353 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff288003d30 con 0x7ff29815e270 2024-04-02T13:21:18.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff2967fc640 1 -- 172.21.15.44:0/1937959353 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28800aa90 con 0x7ff29815e270 2024-04-02T13:21:18.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.313+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1937959353 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2980b3ed0 con 0x7ff29815e270 2024-04-02T13:21:18.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.314+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1937959353 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2980b4490 con 0x7ff29815e270 2024-04-02T13:21:18.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.315+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1937959353 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff29804f8d0 con 0x7ff29815e270 2024-04-02T13:21:18.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.315+0000 7ff2967fc640 1 -- 172.21.15.44:0/1937959353 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff28800ac30 con 0x7ff29815e270 2024-04-02T13:21:18.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.315+0000 7ff2967fc640 1 --2- 172.21.15.44:0/1937959353 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff28406c8f0 0x7ff28406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.315+0000 7ff2967fc640 1 -- 172.21.15.44:0/1937959353 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7ff28808bf40 con 0x7ff29815e270 2024-04-02T13:21:18.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.315+0000 7ff29d79a640 1 --2- 172.21.15.44:0/1937959353 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff28406c8f0 0x7ff28406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.316+0000 7ff29d79a640 1 --2- 172.21.15.44:0/1937959353 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff28406c8f0 0x7ff28406edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff28c007fd0 tx=0x7ff28c007900 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:18.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.320+0000 7ff2967fc640 1 -- 172.21.15.44:0/1937959353 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff288059f40 con 0x7ff29815e270 2024-04-02T13:21:18.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.517+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1937959353 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff2980d0870 con 0x7ff29815e270 2024-04-02T13:21:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.517+0000 7ff2967fc640 1 -- 172.21.15.44:0/1937959353 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7ff288059840 con 0x7ff29815e270 2024-04-02T13:21:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:21:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.519+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1937959353 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff28406c8f0 msgr2=0x7ff28406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.519+0000 7ff29ef9d640 1 --2- 172.21.15.44:0/1937959353 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff28406c8f0 0x7ff28406edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff28c007fd0 tx=0x7ff28c007900 comp rx=0 tx=0).stop 2024-04-02T13:21:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.519+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1937959353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff29815e270 msgr2=0x7ff298158f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.519+0000 7ff29ef9d640 1 --2- 172.21.15.44:0/1937959353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff29815e270 0x7ff298158f90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff28800c870 tx=0x7ff28800cd30 comp rx=0 tx=0).stop 2024-04-02T13:21:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.519+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1937959353 shutdown_connections 2024-04-02T13:21:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.519+0000 7ff29ef9d640 1 --2- 172.21.15.44:0/1937959353 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff28406c8f0 0x7ff28406edb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.519+0000 7ff29ef9d640 1 --2- 172.21.15.44:0/1937959353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2981629e0 0x7ff2980b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.519+0000 7ff29ef9d640 1 --2- 172.21.15.44:0/1937959353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff298162550 0x7ff298160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.519+0000 7ff29ef9d640 1 --2- 172.21.15.44:0/1937959353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff29815e270 0x7ff298158f90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.519+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1937959353 >> 172.21.15.44:0/1937959353 conn(0x7ff2980d3ef0 msgr2=0x7ff2980d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:18.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.519+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1937959353 shutdown_connections 2024-04-02T13:21:18.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.519+0000 7ff29ef9d640 1 -- 172.21.15.44:0/1937959353 wait complete. 2024-04-02T13:21:18.531 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:21:18.531 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:18.551 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:18.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.651+0000 7f252e160640 1 Processor -- start 2024-04-02T13:21:18.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.651+0000 7f252e160640 1 -- start start 2024-04-02T13:21:18.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.652+0000 7f252e160640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252815e430 0x7f25280ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.652+0000 7f252e160640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f25280b1ef0 0x7f25280b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.652+0000 7f252e160640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252815e810 con 0x7f252806c7f0 2024-04-02T13:21:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.652+0000 7f252e160640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f25280d1410 con 0x7f25280b1ef0 2024-04-02T13:21:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.652+0000 7f252e160640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f252806cff0 con 0x7f252815e430 2024-04-02T13:21:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.652+0000 7f2527fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f25280b1ef0 0x7f25280b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.652+0000 7f25277fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252815e430 0x7f25280ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.653+0000 7f2527fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f25280b1ef0 0x7f25280b2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:41372/0 (socket says 172.21.15.44:41372) 2024-04-02T13:21:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.653+0000 7f2527fff640 1 -- 172.21.15.44:0/3089565924 learned_addr learned my addr 172.21.15.44:0/3089565924 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.653+0000 7f2526ffd640 1 --1- 172.21.15.44:0/3089565924 >> v1:172.21.15.69:6789/0 conn(0x7f252806c7f0 0x7f25280ced00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:54454/0 (socket says 172.21.15.44:54454) 2024-04-02T13:21:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.653+0000 7f2527fff640 1 -- 172.21.15.44:0/3089565924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252815e430 msgr2=0x7f25280ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.653+0000 7f2527fff640 1 --2- 172.21.15.44:0/3089565924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252815e430 0x7f25280ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.653+0000 7f2527fff640 1 -- 172.21.15.44:0/3089565924 >> v1:172.21.15.69:6789/0 conn(0x7f252806c7f0 legacy=0x7f25280ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.653+0000 7f25267fc640 1 -- 172.21.15.44:0/3089565924 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 728048209 0 0) 0x7f252815e810 con 0x7f252806c7f0 2024-04-02T13:21:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.653+0000 7f2527fff640 1 -- 172.21.15.44:0/3089565924 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252806d240 con 0x7f25280b1ef0 2024-04-02T13:21:18.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.653+0000 7f25277fe640 1 --2- 172.21.15.44:0/3089565924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252815e430 0x7f25280ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:18.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.654+0000 7f2527fff640 1 --2- 172.21.15.44:0/3089565924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25280b1ef0 0x7f25280b2360 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2520009e50 tx=0x7f252002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e09a0ace40124211 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:18.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.654+0000 7f25267fc640 1 -- 172.21.15.44:0/3089565924 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252002ca70 con 0x7f25280b1ef0 2024-04-02T13:21:18.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.654+0000 7f25267fc640 1 -- 172.21.15.44:0/3089565924 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f252002cc10 con 0x7f25280b1ef0 2024-04-02T13:21:18.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.654+0000 7f25267fc640 1 -- 172.21.15.44:0/3089565924 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25200343f0 con 0x7f25280b1ef0 2024-04-02T13:21:18.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.654+0000 7f252e160640 1 -- 172.21.15.44:0/3089565924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25280b1ef0 msgr2=0x7f25280b2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.654+0000 7f252e160640 1 --2- 172.21.15.44:0/3089565924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25280b1ef0 0x7f25280b2360 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2520009e50 tx=0x7f252002bed0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.654+0000 7f252e160640 1 -- 172.21.15.44:0/3089565924 shutdown_connections 2024-04-02T13:21:18.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.654+0000 7f252e160640 1 --2- 172.21.15.44:0/3089565924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25280b1ef0 0x7f25280b2360 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.654+0000 7f252e160640 1 --2- 172.21.15.44:0/3089565924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252815e430 0x7f25280ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.654+0000 7f252e160640 1 -- 172.21.15.44:0/3089565924 >> 172.21.15.44:0/3089565924 conn(0x7f2528073850 msgr2=0x7f252809aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.654+0000 7f252e160640 1 -- 172.21.15.44:0/3089565924 shutdown_connections 2024-04-02T13:21:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.654+0000 7f252e160640 1 -- 172.21.15.44:0/3089565924 wait complete. 2024-04-02T13:21:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.656+0000 7f252e160640 1 Processor -- start 2024-04-02T13:21:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.656+0000 7f252e160640 1 -- start start 2024-04-02T13:21:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.656+0000 7f252e160640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252806c7f0 0x7f252815ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.656+0000 7f252e160640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25280b1ef0 0x7f2528157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.656+0000 7f252e160640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f252815e430 0x7f25280b65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.656+0000 7f252e160640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f25280d0ef0 con 0x7f252815e430 2024-04-02T13:21:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.656+0000 7f252e160640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25280d0c10 con 0x7f25280b1ef0 2024-04-02T13:21:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.656+0000 7f252e160640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f252806d800 con 0x7f252806c7f0 2024-04-02T13:21:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f2527fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f252815e430 0x7f25280b65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f2527fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f252815e430 0x7f25280b65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45862/0 (socket says 172.21.15.44:45862) 2024-04-02T13:21:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f2527fff640 1 -- 172.21.15.44:0/338396800 learned_addr learned my addr 172.21.15.44:0/338396800 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f25277fe640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252806c7f0 0x7f252815ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f2526ffd640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25280b1ef0 0x7f2528157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f2527fff640 1 -- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252806c7f0 msgr2=0x7f252815ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f2527fff640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252806c7f0 0x7f252815ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f2527fff640 1 -- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25280b1ef0 msgr2=0x7f2528157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f2527fff640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25280b1ef0 0x7f2528157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f2527fff640 1 -- 172.21.15.44:0/338396800 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25280c9220 con 0x7f252815e430 2024-04-02T13:21:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f2527fff640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f252815e430 0x7f25280b65c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2520009e50 tx=0x7f2520004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:18.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f250ffff640 1 -- 172.21.15.44:0/338396800 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520043030 con 0x7f252815e430 2024-04-02T13:21:18.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f250ffff640 1 -- 172.21.15.44:0/338396800 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f252003e450 con 0x7f252815e430 2024-04-02T13:21:18.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f252e160640 1 -- 172.21.15.44:0/338396800 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25280c9500 con 0x7f252815e430 2024-04-02T13:21:18.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f250ffff640 1 -- 172.21.15.44:0/338396800 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520009510 con 0x7f252815e430 2024-04-02T13:21:18.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.657+0000 7f252e160640 1 -- 172.21.15.44:0/338396800 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25280c9ac0 con 0x7f252815e430 2024-04-02T13:21:18.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.658+0000 7f252e160640 1 -- 172.21.15.44:0/338396800 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2528068c00 con 0x7f252815e430 2024-04-02T13:21:18.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.659+0000 7f250ffff640 1 -- 172.21.15.44:0/338396800 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2520046070 con 0x7f252815e430 2024-04-02T13:21:18.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.659+0000 7f250ffff640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f251406c8f0 0x7f251406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.659+0000 7f250ffff640 1 -- 172.21.15.44:0/338396800 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f25200ae190 con 0x7f252815e430 2024-04-02T13:21:18.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.659+0000 7f25277fe640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f251406c8f0 0x7f251406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.660+0000 7f25277fe640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f251406c8f0 0x7f251406edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f25280b0520 tx=0x7f2518009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.663+0000 7f250ffff640 1 -- 172.21.15.44:0/338396800 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f252004d070 con 0x7f252815e430 2024-04-02T13:21:18.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.848+0000 7f252e160640 1 -- 172.21.15.44:0/338396800 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f25280b08d0 con 0x7f252815e430 2024-04-02T13:21:18.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.850+0000 7f250ffff640 1 -- 172.21.15.44:0/338396800 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18899 (secure 0 0 0) 0x7f252007bb60 con 0x7f252815e430 2024-04-02T13:21:18.851 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:18.851 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:08.061253+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:18.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.851+0000 7f252e160640 1 -- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f251406c8f0 msgr2=0x7f251406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.851+0000 7f252e160640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f251406c8f0 0x7f251406edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f25280b0520 tx=0x7f2518009190 comp rx=0 tx=0).stop 2024-04-02T13:21:18.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.852+0000 7f252e160640 1 -- 172.21.15.44:0/338396800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f252815e430 msgr2=0x7f25280b65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.852+0000 7f252e160640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f252815e430 0x7f25280b65c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2520009e50 tx=0x7f2520004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:18.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.852+0000 7f252e160640 1 -- 172.21.15.44:0/338396800 shutdown_connections 2024-04-02T13:21:18.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.852+0000 7f252e160640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f251406c8f0 0x7f251406edb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.852+0000 7f252e160640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f252815e430 0x7f25280b65c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.852+0000 7f252e160640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25280b1ef0 0x7f2528157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.852+0000 7f252e160640 1 --2- 172.21.15.44:0/338396800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252806c7f0 0x7f252815ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.852+0000 7f252e160640 1 -- 172.21.15.44:0/338396800 >> 172.21.15.44:0/338396800 conn(0x7f2528073850 msgr2=0x7f25280aeda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:18.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.852+0000 7f252e160640 1 -- 172.21.15.44:0/338396800 shutdown_connections 2024-04-02T13:21:18.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.852+0000 7f252e160640 1 -- 172.21.15.44:0/338396800 wait complete. 2024-04-02T13:21:18.866 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:18.867 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:18.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.970+0000 7faf87359640 1 Processor -- start 2024-04-02T13:21:18.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.970+0000 7faf87359640 1 -- start start 2024-04-02T13:21:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.971+0000 7faf87359640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7faf800b4050 0x7faf800ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.971+0000 7faf87359640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf800cd4a0 con 0x7faf800cd030 2024-04-02T13:21:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.971+0000 7faf87359640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf800cd760 con 0x7faf8015c260 2024-04-02T13:21:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.971+0000 7faf87359640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7faf800cda40 con 0x7faf800b4050 2024-04-02T13:21:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.971+0000 7faf848f4640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7faf800b4050 0x7faf800ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.971+0000 7faf858f6640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7faf800cd030 0x7faf800c5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48904/0 (socket says 172.21.15.44:48904) 2024-04-02T13:21:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.971+0000 7faf858f6640 1 -- 172.21.15.44:0/1430460243 learned_addr learned my addr 172.21.15.44:0/1430460243 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.971+0000 7faf848f4640 1 -- 172.21.15.44:0/1430460243 >> v1:172.21.15.69:6789/0 conn(0x7faf8015c260 legacy=0x7faf800b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.971+0000 7faf848f4640 1 -- 172.21.15.44:0/1430460243 >> v1:172.21.15.44:6789/0 conn(0x7faf800cd030 legacy=0x7faf800c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.971+0000 7faf848f4640 1 -- 172.21.15.44:0/1430460243 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf800cdd20 con 0x7faf800b4050 2024-04-02T13:21:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.971+0000 7faf77fff640 1 -- 172.21.15.44:0/1430460243 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1633618512 0 0) 0x7faf800cd4a0 con 0x7faf800cd030 2024-04-02T13:21:18.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.972+0000 7faf848f4640 1 --2- 172.21.15.44:0/1430460243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800b4050 0x7faf800ccaf0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7faf700097a0 tx=0x7faf7002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b46a06a26bc892 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:18.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.972+0000 7faf77fff640 1 -- 172.21.15.44:0/1430460243 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf70034070 con 0x7faf800b4050 2024-04-02T13:21:18.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.972+0000 7faf77fff640 1 -- 172.21.15.44:0/1430460243 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf70030610 con 0x7faf800b4050 2024-04-02T13:21:18.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.972+0000 7faf77fff640 1 -- 172.21.15.44:0/1430460243 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf700309e0 con 0x7faf800b4050 2024-04-02T13:21:18.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.973+0000 7faf87359640 1 -- 172.21.15.44:0/1430460243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800b4050 msgr2=0x7faf800ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.973+0000 7faf87359640 1 --2- 172.21.15.44:0/1430460243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800b4050 0x7faf800ccaf0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7faf700097a0 tx=0x7faf7002b960 comp rx=0 tx=0).stop 2024-04-02T13:21:18.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.973+0000 7faf87359640 1 -- 172.21.15.44:0/1430460243 shutdown_connections 2024-04-02T13:21:18.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.973+0000 7faf87359640 1 --2- 172.21.15.44:0/1430460243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800b4050 0x7faf800ccaf0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.973+0000 7faf87359640 1 -- 172.21.15.44:0/1430460243 >> 172.21.15.44:0/1430460243 conn(0x7faf8009aa20 msgr2=0x7faf8009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:18.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.973+0000 7faf87359640 1 -- 172.21.15.44:0/1430460243 shutdown_connections 2024-04-02T13:21:18.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.973+0000 7faf87359640 1 -- 172.21.15.44:0/1430460243 wait complete. 2024-04-02T13:21:18.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.974+0000 7faf87359640 1 Processor -- start 2024-04-02T13:21:18.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.974+0000 7faf87359640 1 -- start start 2024-04-02T13:21:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf87359640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf800b4050 0x7faf800b7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf87359640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800cd030 0x7faf8015ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf87359640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf8015c260 0x7faf80150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf87359640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faf800c6240 con 0x7faf800cd030 2024-04-02T13:21:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf87359640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faf800cfe60 con 0x7faf800b4050 2024-04-02T13:21:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf87359640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faf800c63c0 con 0x7faf8015c260 2024-04-02T13:21:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf848f4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800cd030 0x7faf8015ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf848f4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800cd030 0x7faf8015ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45880/0 (socket says 172.21.15.44:45880) 2024-04-02T13:21:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf848f4640 1 -- 172.21.15.44:0/3194472078 learned_addr learned my addr 172.21.15.44:0/3194472078 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf850f5640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf800b4050 0x7faf800b7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf858f6640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf8015c260 0x7faf80150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf848f4640 1 -- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf8015c260 msgr2=0x7faf80150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf848f4640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf8015c260 0x7faf80150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf848f4640 1 -- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf800b4050 msgr2=0x7faf800b7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf848f4640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf800b4050 0x7faf800b7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf848f4640 1 -- 172.21.15.44:0/3194472078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf800b64e0 con 0x7faf800cd030 2024-04-02T13:21:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf850f5640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf800b4050 0x7faf800b7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf848f4640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800cd030 0x7faf8015ee40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7faf700098c0 tx=0x7faf700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf75ffb640 1 -- 172.21.15.44:0/3194472078 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf70034040 con 0x7faf800cd030 2024-04-02T13:21:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf75ffb640 1 -- 172.21.15.44:0/3194472078 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf70039a10 con 0x7faf800cd030 2024-04-02T13:21:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf87359640 1 -- 172.21.15.44:0/3194472078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf800b67c0 con 0x7faf800cd030 2024-04-02T13:21:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.975+0000 7faf75ffb640 1 -- 172.21.15.44:0/3194472078 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf70039de0 con 0x7faf800cd030 2024-04-02T13:21:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.976+0000 7faf87359640 1 -- 172.21.15.44:0/3194472078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf800b6d80 con 0x7faf800cd030 2024-04-02T13:21:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.976+0000 7faf87359640 1 -- 172.21.15.44:0/3194472078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf8006c0b0 con 0x7faf800cd030 2024-04-02T13:21:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.977+0000 7faf75ffb640 1 -- 172.21.15.44:0/3194472078 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faf70037070 con 0x7faf800cd030 2024-04-02T13:21:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.977+0000 7faf75ffb640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf6406c8f0 0x7faf6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.977+0000 7faf75ffb640 1 -- 172.21.15.44:0/3194472078 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7faf700ae060 con 0x7faf800cd030 2024-04-02T13:21:18.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.978+0000 7faf850f5640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf6406c8f0 0x7faf6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:18.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.978+0000 7faf850f5640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf6406c8f0 0x7faf6406edb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faf7c007fd0 tx=0x7faf7c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:18.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:18.982+0000 7faf75ffb640 1 -- 172.21.15.44:0/3194472078 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7faf8006c0b0 con 0x7faf800cd030 2024-04-02T13:21:19.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.166+0000 7faf87359640 1 -- 172.21.15.44:0/3194472078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faf8015b740 con 0x7faf800cd030 2024-04-02T13:21:19.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.166+0000 7faf75ffb640 1 -- 172.21.15.44:0/3194472078 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7faf7007c070 con 0x7faf800cd030 2024-04-02T13:21:19.168 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:19.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.168+0000 7faf87359640 1 -- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf6406c8f0 msgr2=0x7faf6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.168+0000 7faf87359640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf6406c8f0 0x7faf6406edb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faf7c007fd0 tx=0x7faf7c009040 comp rx=0 tx=0).stop 2024-04-02T13:21:19.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.168+0000 7faf87359640 1 -- 172.21.15.44:0/3194472078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800cd030 msgr2=0x7faf8015ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.168+0000 7faf87359640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800cd030 0x7faf8015ee40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7faf700098c0 tx=0x7faf700046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.168+0000 7faf87359640 1 -- 172.21.15.44:0/3194472078 shutdown_connections 2024-04-02T13:21:19.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.168+0000 7faf87359640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf6406c8f0 0x7faf6406edb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.168+0000 7faf87359640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf8015c260 0x7faf80150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.168+0000 7faf87359640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800cd030 0x7faf8015ee40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.168+0000 7faf87359640 1 --2- 172.21.15.44:0/3194472078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf800b4050 0x7faf800b7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.168+0000 7faf87359640 1 -- 172.21.15.44:0/3194472078 >> 172.21.15.44:0/3194472078 conn(0x7faf8009aa20 msgr2=0x7faf800c6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:19.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.168+0000 7faf87359640 1 -- 172.21.15.44:0/3194472078 shutdown_connections 2024-04-02T13:21:19.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.168+0000 7faf87359640 1 -- 172.21.15.44:0/3194472078 wait complete. 2024-04-02T13:21:19.180 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:19.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.283+0000 7f1fd6ef2640 1 Processor -- start 2024-04-02T13:21:19.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.283+0000 7f1fd6ef2640 1 -- start start 2024-04-02T13:21:19.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fd6ef2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1fd00aeca0 0x7f1fd006e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fd6ef2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd00d07b0 0x7f1fd00acde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fd6ef2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fd00cdd60 con 0x7f1fd006eb10 2024-04-02T13:21:19.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fd6ef2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f1fd015b3b0 con 0x7f1fd00aeca0 2024-04-02T13:21:19.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fd6ef2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1fd00ce040 con 0x7f1fd00d07b0 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fd4c8e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1fd00aeca0 0x7f1fd006e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fd4c8e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1fd00aeca0 0x7f1fd006e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45894/0 (socket says 172.21.15.44:45894) 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fcffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f1fd006eb10 0x7f1fd00d00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48910/0 (socket says 172.21.15.44:48910) 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fd4c8e640 1 -- 172.21.15.44:0/2027936427 learned_addr learned my addr 172.21.15.44:0/2027936427 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fd548f640 1 --2- 172.21.15.44:0/2027936427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd00d07b0 0x7f1fd00acde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fd4c8e640 1 -- 172.21.15.44:0/2027936427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd00d07b0 msgr2=0x7f1fd00acde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fd4c8e640 1 --2- 172.21.15.44:0/2027936427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd00d07b0 0x7f1fd00acde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fd4c8e640 1 -- 172.21.15.44:0/2027936427 >> v1:172.21.15.44:6789/0 conn(0x7f1fd006eb10 legacy=0x7f1fd00d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.284+0000 7f1fd4c8e640 1 -- 172.21.15.44:0/2027936427 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd00ce320 con 0x7f1fd00aeca0 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.285+0000 7f1fd4c8e640 1 --2- 172.21.15.44:0/2027936427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fd00aeca0 0x7f1fd006e5d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009870 tx=0x7f1fc402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec646497b49a79e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.285+0000 7f1fcf7fe640 1 -- 172.21.15.44:0/2027936427 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4034070 con 0x7f1fd00aeca0 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.285+0000 7f1fcf7fe640 1 -- 172.21.15.44:0/2027936427 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fc4030840 con 0x7f1fd00aeca0 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.285+0000 7f1fcf7fe640 1 -- 172.21.15.44:0/2027936427 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4030c10 con 0x7f1fd00aeca0 2024-04-02T13:21:19.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.286+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/2027936427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fd00aeca0 msgr2=0x7f1fd006e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.286+0000 7f1fd6ef2640 1 --2- 172.21.15.44:0/2027936427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fd00aeca0 0x7f1fd006e5d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009870 tx=0x7f1fc402b980 comp rx=0 tx=0).stop 2024-04-02T13:21:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.286+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/2027936427 shutdown_connections 2024-04-02T13:21:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.286+0000 7f1fd6ef2640 1 --2- 172.21.15.44:0/2027936427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd00d07b0 0x7f1fd00acde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.286+0000 7f1fd6ef2640 1 --2- 172.21.15.44:0/2027936427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fd00aeca0 0x7f1fd006e5d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.286+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/2027936427 >> 172.21.15.44:0/2027936427 conn(0x7f1fd006b920 msgr2=0x7f1fd006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.286+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/2027936427 shutdown_connections 2024-04-02T13:21:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.286+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/2027936427 wait complete. 2024-04-02T13:21:19.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.287+0000 7f1fd6ef2640 1 Processor -- start 2024-04-02T13:21:19.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.287+0000 7f1fd6ef2640 1 -- start start 2024-04-02T13:21:19.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fd6ef2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd006eb10 0x7f1fd0154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fd6ef2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fd00aeca0 0x7f1fd014a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fd6ef2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd00d07b0 0x7f1fd014a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fd6ef2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd00ad320 con 0x7f1fd00aeca0 2024-04-02T13:21:19.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fd6ef2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd00ce8e0 con 0x7f1fd006eb10 2024-04-02T13:21:19.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fd6ef2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1fd00ad4a0 con 0x7f1fd00d07b0 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fcffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fd00aeca0 0x7f1fd014a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fcffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fd00aeca0 0x7f1fd014a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45900/0 (socket says 172.21.15.44:45900) 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fcffff640 1 -- 172.21.15.44:0/3824126247 learned_addr learned my addr 172.21.15.44:0/3824126247 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fd548f640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd00d07b0 0x7f1fd014a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fd4c8e640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd006eb10 0x7f1fd0154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fcffff640 1 -- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd00d07b0 msgr2=0x7f1fd014a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fcffff640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd00d07b0 0x7f1fd014a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fcffff640 1 -- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd006eb10 msgr2=0x7f1fd0154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fcffff640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd006eb10 0x7f1fd0154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fcffff640 1 -- 172.21.15.44:0/3824126247 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd00b2f30 con 0x7f1fd00aeca0 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fd4c8e640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd006eb10 0x7f1fd0154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fcffff640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fd00aeca0 0x7f1fd014a1f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00c400 tx=0x7f1fbc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fcd7fa640 1 -- 172.21.15.44:0/3824126247 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc0074e0 con 0x7f1fd00aeca0 2024-04-02T13:21:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fcd7fa640 1 -- 172.21.15.44:0/3824126247 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fbc007680 con 0x7f1fd00aeca0 2024-04-02T13:21:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fcd7fa640 1 -- 172.21.15.44:0/3824126247 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc004a00 con 0x7f1fd00aeca0 2024-04-02T13:21:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.288+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/3824126247 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd01546d0 con 0x7f1fd00aeca0 2024-04-02T13:21:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.289+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/3824126247 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fd009d4c0 con 0x7f1fd00aeca0 2024-04-02T13:21:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.290+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/3824126247 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fd00684f0 con 0x7f1fd00aeca0 2024-04-02T13:21:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.290+0000 7f1fcd7fa640 1 -- 172.21.15.44:0/3824126247 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1fbc01a020 con 0x7f1fd00aeca0 2024-04-02T13:21:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.290+0000 7f1fcd7fa640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1fb406c8f0 0x7f1fb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.290+0000 7f1fcd7fa640 1 -- 172.21.15.44:0/3824126247 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f1fbc08b870 con 0x7f1fd00aeca0 2024-04-02T13:21:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.291+0000 7f1fd4c8e640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1fb406c8f0 0x7f1fb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.291+0000 7f1fd4c8e640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1fb406c8f0 0x7f1fb406edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4036000 tx=0x7f1fc403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:19.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.295+0000 7f1fcd7fa640 1 -- 172.21.15.44:0/3824126247 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1fbc0597e0 con 0x7f1fd00aeca0 2024-04-02T13:21:19.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.476+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/3824126247 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1fd00754e0 con 0x7f1fd00aeca0 2024-04-02T13:21:19.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.477+0000 7f1fcd7fa640 1 -- 172.21.15.44:0/3824126247 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f1fbc0590e0 con 0x7f1fd00aeca0 2024-04-02T13:21:19.478 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:19.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.479+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1fb406c8f0 msgr2=0x7f1fb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.479+0000 7f1fd6ef2640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1fb406c8f0 0x7f1fb406edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4036000 tx=0x7f1fc403b040 comp rx=0 tx=0).stop 2024-04-02T13:21:19.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.479+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/3824126247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fd00aeca0 msgr2=0x7f1fd014a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.479+0000 7f1fd6ef2640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fd00aeca0 0x7f1fd014a1f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00c400 tx=0x7f1fbc00c8c0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.479+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/3824126247 shutdown_connections 2024-04-02T13:21:19.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.479+0000 7f1fd6ef2640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1fb406c8f0 0x7f1fb406edb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.479+0000 7f1fd6ef2640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd00d07b0 0x7f1fd014a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.479+0000 7f1fd6ef2640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fd00aeca0 0x7f1fd014a1f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.479+0000 7f1fd6ef2640 1 --2- 172.21.15.44:0/3824126247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd006eb10 0x7f1fd0154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.479+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/3824126247 >> 172.21.15.44:0/3824126247 conn(0x7f1fd006b920 msgr2=0x7f1fd0154900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:19.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.479+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/3824126247 shutdown_connections 2024-04-02T13:21:19.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.479+0000 7f1fd6ef2640 1 -- 172.21.15.44:0/3824126247 wait complete. 2024-04-02T13:21:19.491 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:19.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.594+0000 7f2ced79d640 1 Processor -- start 2024-04-02T13:21:19.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.594+0000 7f2ced79d640 1 -- start start 2024-04-02T13:21:19.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.594+0000 7f2ced79d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2ce806daa0 0x7f2ce80b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.594+0000 7f2ced79d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2ce80cfdc0 0x7f2ce80d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.594+0000 7f2ced79d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ce806de80 con 0x7f2ce80d26f0 2024-04-02T13:21:19.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.594+0000 7f2ced79d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2ce80cf930 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.594+0000 7f2ced79d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2ce80cf000 con 0x7f2ce806daa0 2024-04-02T13:21:19.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.595+0000 7f2cdffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2ce80cfdc0 0x7f2ce80d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.595+0000 7f2ce77fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f2ce80d26f0 0x7f2ce815dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48916/0 (socket says 172.21.15.44:48916) 2024-04-02T13:21:19.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.595+0000 7f2ce77fe640 1 -- 172.21.15.44:0/3935773676 learned_addr learned my addr 172.21.15.44:0/3935773676 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:19.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.595+0000 7f2ce6ffd640 1 --2- 172.21.15.44:0/3935773676 >> v2:172.21.15.69:3300/0 conn(0x7f2ce806daa0 0x7f2ce80b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.595+0000 7f2cdffff640 1 -- 172.21.15.44:0/3935773676 >> v2:172.21.15.69:3300/0 conn(0x7f2ce806daa0 msgr2=0x7f2ce80b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:19.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.595+0000 7f2cdffff640 1 --2- 172.21.15.44:0/3935773676 >> v2:172.21.15.69:3300/0 conn(0x7f2ce806daa0 0x7f2ce80b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.595+0000 7f2cdffff640 1 -- 172.21.15.44:0/3935773676 >> v1:172.21.15.44:6789/0 conn(0x7f2ce80d26f0 legacy=0x7f2ce815dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.595+0000 7f2cdffff640 1 -- 172.21.15.44:0/3935773676 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce80cf2e0 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.595+0000 7f2ce67fc640 1 -- 172.21.15.44:0/3935773676 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2468863117 0 0) 0x7f2ce806de80 con 0x7f2ce80d26f0 2024-04-02T13:21:19.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.596+0000 7f2cdffff640 1 --2- 172.21.15.44:0/3935773676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ce80cfdc0 0x7f2ce80d11c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2ce808d750 tx=0x7f2cd002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9daf7bcc63a96c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:19.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.596+0000 7f2ce67fc640 1 -- 172.21.15.44:0/3935773676 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0035020 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.596+0000 7f2ce67fc640 1 -- 172.21.15.44:0/3935773676 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cd0031830 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.596+0000 7f2ce67fc640 1 -- 172.21.15.44:0/3935773676 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0031c00 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.596+0000 7f2ced79d640 1 -- 172.21.15.44:0/3935773676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ce80cfdc0 msgr2=0x7f2ce80d11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.596+0000 7f2ced79d640 1 --2- 172.21.15.44:0/3935773676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ce80cfdc0 0x7f2ce80d11c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2ce808d750 tx=0x7f2cd002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.596+0000 7f2ced79d640 1 -- 172.21.15.44:0/3935773676 shutdown_connections 2024-04-02T13:21:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.596+0000 7f2ced79d640 1 --2- 172.21.15.44:0/3935773676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ce80cfdc0 0x7f2ce80d11c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.596+0000 7f2ced79d640 1 --2- 172.21.15.44:0/3935773676 >> v2:172.21.15.69:3300/0 conn(0x7f2ce806daa0 0x7f2ce80b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.596+0000 7f2ced79d640 1 -- 172.21.15.44:0/3935773676 >> 172.21.15.44:0/3935773676 conn(0x7f2ce809d430 msgr2=0x7f2ce80d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.596+0000 7f2ced79d640 1 -- 172.21.15.44:0/3935773676 shutdown_connections 2024-04-02T13:21:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.596+0000 7f2ced79d640 1 -- 172.21.15.44:0/3935773676 wait complete. 2024-04-02T13:21:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.598+0000 7f2ced79d640 1 Processor -- start 2024-04-02T13:21:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.598+0000 7f2ced79d640 1 -- start start 2024-04-02T13:21:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.598+0000 7f2ced79d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce806daa0 0x7f2ce80ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.598+0000 7f2ced79d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ce80cfdc0 0x7f2ce80cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.598+0000 7f2ced79d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce80d26f0 0x7f2ce80b8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.598+0000 7f2ced79d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce80ceda0 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.598+0000 7f2ced79d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce80cec20 con 0x7f2ce806daa0 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.598+0000 7f2ced79d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2ce80d2b60 con 0x7f2ce80d26f0 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.598+0000 7f2cdffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ce80cfdc0 0x7f2ce80cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.598+0000 7f2cdffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ce80cfdc0 0x7f2ce80cb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45916/0 (socket says 172.21.15.44:45916) 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.598+0000 7f2cdffff640 1 -- 172.21.15.44:0/968569158 learned_addr learned my addr 172.21.15.44:0/968569158 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2ce6ffd640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce806daa0 0x7f2ce80ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2ce77fe640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce80d26f0 0x7f2ce80b8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2cdffff640 1 -- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce80d26f0 msgr2=0x7f2ce80b8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2cdffff640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce80d26f0 0x7f2ce80b8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2cdffff640 1 -- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce806daa0 msgr2=0x7f2ce80ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2cdffff640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce806daa0 0x7f2ce80ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2cdffff640 1 -- 172.21.15.44:0/968569158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce80ccb80 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2ce6ffd640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce806daa0 0x7f2ce80ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2ce77fe640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce80d26f0 0x7f2ce80b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2cdffff640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ce80cfdc0 0x7f2ce80cb1e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0009890 tx=0x7f2cd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:19.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2cdf7fe640 1 -- 172.21.15.44:0/968569158 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0041070 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2cdf7fe640 1 -- 172.21.15.44:0/968569158 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cd003aa30 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2ced79d640 1 -- 172.21.15.44:0/968569158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ce80cbae0 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2cdf7fe640 1 -- 172.21.15.44:0/968569158 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0035040 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.599+0000 7f2ced79d640 1 -- 172.21.15.44:0/968569158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ce80cc0a0 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.600+0000 7f2ced79d640 1 -- 172.21.15.44:0/968569158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ce80684f0 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.601+0000 7f2cdf7fe640 1 -- 172.21.15.44:0/968569158 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2cd0038020 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.601+0000 7f2cdf7fe640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2cc806c8f0 0x7f2cc806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.601+0000 7f2cdf7fe640 1 -- 172.21.15.44:0/968569158 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f2cd00ad9d0 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.601+0000 7f2ce6ffd640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2cc806c8f0 0x7f2cc806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.602+0000 7f2ce6ffd640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2cc806c8f0 0x7f2cc806edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8006fd0 tx=0x7f2cd8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.606+0000 7f2cdf7fe640 1 -- 172.21.15.44:0/968569158 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2cd007b9e0 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.789+0000 7f2ced79d640 1 -- 172.21.15.44:0/968569158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2ce80d24d0 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.789+0000 7f2cdf7fe640 1 -- 172.21.15.44:0/968569158 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f2cd007b2e0 con 0x7f2ce80cfdc0 2024-04-02T13:21:19.791 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:19.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.791+0000 7f2ced79d640 1 -- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2cc806c8f0 msgr2=0x7f2cc806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.791+0000 7f2ced79d640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2cc806c8f0 0x7f2cc806edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8006fd0 tx=0x7f2cd8008040 comp rx=0 tx=0).stop 2024-04-02T13:21:19.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.791+0000 7f2ced79d640 1 -- 172.21.15.44:0/968569158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ce80cfdc0 msgr2=0x7f2ce80cb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.791+0000 7f2ced79d640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ce80cfdc0 0x7f2ce80cb1e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0009890 tx=0x7f2cd00046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.791+0000 7f2ced79d640 1 -- 172.21.15.44:0/968569158 shutdown_connections 2024-04-02T13:21:19.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.791+0000 7f2ced79d640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2cc806c8f0 0x7f2cc806edb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.791+0000 7f2ced79d640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce80d26f0 0x7f2ce80b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.791+0000 7f2ced79d640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ce80cfdc0 0x7f2ce80cb1e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.791+0000 7f2ced79d640 1 --2- 172.21.15.44:0/968569158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce806daa0 0x7f2ce80ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.791+0000 7f2ced79d640 1 -- 172.21.15.44:0/968569158 >> 172.21.15.44:0/968569158 conn(0x7f2ce809d430 msgr2=0x7f2ce80d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:19.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.791+0000 7f2ced79d640 1 -- 172.21.15.44:0/968569158 shutdown_connections 2024-04-02T13:21:19.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.791+0000 7f2ced79d640 1 -- 172.21.15.44:0/968569158 wait complete. 2024-04-02T13:21:19.803 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:19.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.907+0000 7fad126d3640 1 Processor -- start 2024-04-02T13:21:19.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.907+0000 7fad126d3640 1 -- start start 2024-04-02T13:21:19.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.907+0000 7fad126d3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fad0c0ce790 0x7fad0c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.907+0000 7fad126d3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad0c0ced00 con 0x7fad0c0b1540 2024-04-02T13:21:19.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.907+0000 7fad126d3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad0c0cefe0 con 0x7fad0c15e470 2024-04-02T13:21:19.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.907+0000 7fad126d3640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fad0c0cf2c0 con 0x7fad0c0ce790 2024-04-02T13:21:19.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.908+0000 7fad0b7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fad0c0b1540 0x7fad0c06d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48920/0 (socket says 172.21.15.44:48920) 2024-04-02T13:21:19.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.908+0000 7fad10c70640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fad0c0ce790 0x7fad0c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.908+0000 7fad0b7fe640 1 -- 172.21.15.44:0/1871560859 learned_addr learned my addr 172.21.15.44:0/1871560859 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:19.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.908+0000 7fad10c70640 1 -- 172.21.15.44:0/1871560859 >> v1:172.21.15.69:6789/0 conn(0x7fad0c15e470 legacy=0x7fad0c0d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.908+0000 7fad10c70640 1 -- 172.21.15.44:0/1871560859 >> v1:172.21.15.44:6789/0 conn(0x7fad0c0b1540 legacy=0x7fad0c06d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.908+0000 7fad10c70640 1 -- 172.21.15.44:0/1871560859 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c0cf5a0 con 0x7fad0c0ce790 2024-04-02T13:21:19.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.908+0000 7fad0affd640 1 -- 172.21.15.44:0/1871560859 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 459251184 0 0) 0x7fad0c0ced00 con 0x7fad0c0b1540 2024-04-02T13:21:19.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.909+0000 7fad10c70640 1 --2- 172.21.15.44:0/1871560859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad0c0ce790 0x7fad0c0b32c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fad04009d90 tx=0x7fad0402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd4193abc3aeb8b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:19.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.909+0000 7fad0affd640 1 -- 172.21.15.44:0/1871560859 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04039070 con 0x7fad0c0ce790 2024-04-02T13:21:19.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.909+0000 7fad0affd640 1 -- 172.21.15.44:0/1871560859 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad0402cc60 con 0x7fad0c0ce790 2024-04-02T13:21:19.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.909+0000 7fad0affd640 1 -- 172.21.15.44:0/1871560859 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04034660 con 0x7fad0c0ce790 2024-04-02T13:21:19.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.909+0000 7fad126d3640 1 -- 172.21.15.44:0/1871560859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad0c0ce790 msgr2=0x7fad0c0b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.909+0000 7fad126d3640 1 --2- 172.21.15.44:0/1871560859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad0c0ce790 0x7fad0c0b32c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fad04009d90 tx=0x7fad0402beb0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.909+0000 7fad126d3640 1 -- 172.21.15.44:0/1871560859 shutdown_connections 2024-04-02T13:21:19.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.909+0000 7fad126d3640 1 --2- 172.21.15.44:0/1871560859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad0c0ce790 0x7fad0c0b32c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.909+0000 7fad126d3640 1 -- 172.21.15.44:0/1871560859 >> 172.21.15.44:0/1871560859 conn(0x7fad0c09aa20 msgr2=0x7fad0c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:19.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.909+0000 7fad126d3640 1 -- 172.21.15.44:0/1871560859 shutdown_connections 2024-04-02T13:21:19.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.909+0000 7fad126d3640 1 -- 172.21.15.44:0/1871560859 wait complete. 2024-04-02T13:21:19.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.911+0000 7fad126d3640 1 Processor -- start 2024-04-02T13:21:19.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.911+0000 7fad126d3640 1 -- start start 2024-04-02T13:21:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.911+0000 7fad126d3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad0c0b1540 0x7fad0c153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.911+0000 7fad126d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0c0ce790 0x7fad0c1f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.911+0000 7fad126d3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad0c15e470 0x7fad0c1f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.911+0000 7fad126d3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c0d22c0 con 0x7fad0c15e470 2024-04-02T13:21:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.911+0000 7fad126d3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c06cb60 con 0x7fad0c0b1540 2024-04-02T13:21:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.911+0000 7fad126d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fad0c0d2440 con 0x7fad0c0ce790 2024-04-02T13:21:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad10c70640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad0c15e470 0x7fad0c1f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad10c70640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad0c15e470 0x7fad0c1f79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45928/0 (socket says 172.21.15.44:45928) 2024-04-02T13:21:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad10c70640 1 -- 172.21.15.44:0/1300284577 learned_addr learned my addr 172.21.15.44:0/1300284577 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad0b7fe640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0c0ce790 0x7fad0c1f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad0bfff640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad0c0b1540 0x7fad0c153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad10c70640 1 -- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0c0ce790 msgr2=0x7fad0c1f74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad10c70640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0c0ce790 0x7fad0c1f74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad10c70640 1 -- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad0c0b1540 msgr2=0x7fad0c153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad10c70640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad0c0b1540 0x7fad0c153c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad10c70640 1 -- 172.21.15.44:0/1300284577 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c157bb0 con 0x7fad0c15e470 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad10c70640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad0c15e470 0x7fad0c1f79f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fad04009d90 tx=0x7fad04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad08ff9640 1 -- 172.21.15.44:0/1300284577 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04039050 con 0x7fad0c15e470 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad08ff9640 1 -- 172.21.15.44:0/1300284577 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad0403a040 con 0x7fad0c15e470 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad08ff9640 1 -- 172.21.15.44:0/1300284577 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0403e610 con 0x7fad0c15e470 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad126d3640 1 -- 172.21.15.44:0/1300284577 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad0c157e90 con 0x7fad0c15e470 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.912+0000 7fad126d3640 1 -- 172.21.15.44:0/1300284577 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad0c15a5c0 con 0x7fad0c15e470 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.913+0000 7fad126d3640 1 -- 172.21.15.44:0/1300284577 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad0c0684f0 con 0x7fad0c15e470 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.914+0000 7fad08ff9640 1 -- 172.21.15.44:0/1300284577 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fad04047020 con 0x7fad0c15e470 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.914+0000 7fad08ff9640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7facf806c8f0 0x7facf806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.914+0000 7fad08ff9640 1 -- 172.21.15.44:0/1300284577 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fad040ae1e0 con 0x7fad0c15e470 2024-04-02T13:21:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.914+0000 7fad0bfff640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7facf806c8f0 0x7facf806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:19.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.915+0000 7fad0bfff640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7facf806c8f0 0x7facf806edb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7facfc000f60 tx=0x7facfc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:19.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:19.919+0000 7fad08ff9640 1 -- 172.21.15.44:0/1300284577 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fad0404e020 con 0x7fad0c15e470 2024-04-02T13:21:20.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.102+0000 7fad126d3640 1 -- 172.21.15.44:0/1300284577 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fad0c0d2d10 con 0x7fad0c15e470 2024-04-02T13:21:20.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.103+0000 7fad08ff9640 1 -- 172.21.15.44:0/1300284577 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fad0407bc60 con 0x7fad0c15e470 2024-04-02T13:21:20.104 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:20.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.105+0000 7fad126d3640 1 -- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7facf806c8f0 msgr2=0x7facf806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:20.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.105+0000 7fad126d3640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7facf806c8f0 0x7facf806edb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7facfc000f60 tx=0x7facfc009040 comp rx=0 tx=0).stop 2024-04-02T13:21:20.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.105+0000 7fad126d3640 1 -- 172.21.15.44:0/1300284577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad0c15e470 msgr2=0x7fad0c1f79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:20.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.105+0000 7fad126d3640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad0c15e470 0x7fad0c1f79f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fad04009d90 tx=0x7fad04004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:20.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.105+0000 7fad126d3640 1 -- 172.21.15.44:0/1300284577 shutdown_connections 2024-04-02T13:21:20.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.105+0000 7fad126d3640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7facf806c8f0 0x7facf806edb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.105+0000 7fad126d3640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad0c15e470 0x7fad0c1f79f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.105+0000 7fad126d3640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0c0ce790 0x7fad0c1f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.105+0000 7fad126d3640 1 --2- 172.21.15.44:0/1300284577 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad0c0b1540 0x7fad0c153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.105+0000 7fad126d3640 1 -- 172.21.15.44:0/1300284577 >> 172.21.15.44:0/1300284577 conn(0x7fad0c09aa20 msgr2=0x7fad0c070b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:20.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.105+0000 7fad126d3640 1 -- 172.21.15.44:0/1300284577 shutdown_connections 2024-04-02T13:21:20.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.105+0000 7fad126d3640 1 -- 172.21.15.44:0/1300284577 wait complete. 2024-04-02T13:21:20.117 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:21:20.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.218+0000 7f67c7b4a640 1 Processor -- start 2024-04-02T13:21:20.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.218+0000 7f67c7b4a640 1 -- start start 2024-04-02T13:21:20.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.219+0000 7f67c7b4a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67c015c250 0x7f67c00aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:20.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.219+0000 7f67c7b4a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f67c00b3510 0x7f67c00d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:20.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.219+0000 7f67c7b4a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67c00af420 con 0x7f67c00d08d0 2024-04-02T13:21:20.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.219+0000 7f67c7b4a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f67c015c630 con 0x7f67c00b3510 2024-04-02T13:21:20.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.219+0000 7f67c7b4a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f67c00d0d20 con 0x7f67c015c250 2024-04-02T13:21:20.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.219+0000 7f67c50e5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f67c00b3510 0x7f67c00d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:20.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.219+0000 7f67c50e5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f67c00b3510 0x7f67c00d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45934/0 (socket says 172.21.15.44:45934) 2024-04-02T13:21:20.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.219+0000 7f67c50e5640 1 -- 172.21.15.44:0/2912776333 learned_addr learned my addr 172.21.15.44:0/2912776333 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:20.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.220+0000 7f67c58e6640 1 --2- 172.21.15.44:0/2912776333 >> v2:172.21.15.69:3300/0 conn(0x7f67c015c250 0x7f67c00aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:20.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.220+0000 7f67c50e5640 1 -- 172.21.15.44:0/2912776333 >> v2:172.21.15.69:3300/0 conn(0x7f67c015c250 msgr2=0x7f67c00aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:20.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.220+0000 7f67c50e5640 1 --2- 172.21.15.44:0/2912776333 >> v2:172.21.15.69:3300/0 conn(0x7f67c015c250 0x7f67c00aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.220+0000 7f67c50e5640 1 -- 172.21.15.44:0/2912776333 >> v1:172.21.15.69:6789/0 conn(0x7f67c00d08d0 legacy=0x7f67c00b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:20.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.220+0000 7f67c50e5640 1 -- 172.21.15.44:0/2912776333 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c00af700 con 0x7f67c00b3510 2024-04-02T13:21:20.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.220+0000 7f67c50e5640 1 --2- 172.21.15.44:0/2912776333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67c00b3510 0x7f67c00d0390 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f67c00cf9f0 tx=0x7f67b002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4146241f42c39783 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:20.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.221+0000 7f67c48e4640 1 -- 172.21.15.44:0/2912776333 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b002e070 con 0x7f67c00b3510 2024-04-02T13:21:20.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.221+0000 7f67c48e4640 1 -- 172.21.15.44:0/2912776333 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67b0037070 con 0x7f67c00b3510 2024-04-02T13:21:20.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.221+0000 7f67c48e4640 1 -- 172.21.15.44:0/2912776333 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b0031a00 con 0x7f67c00b3510 2024-04-02T13:21:20.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.221+0000 7f67c7b4a640 1 -- 172.21.15.44:0/2912776333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67c00b3510 msgr2=0x7f67c00d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:20.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.221+0000 7f67c7b4a640 1 --2- 172.21.15.44:0/2912776333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67c00b3510 0x7f67c00d0390 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f67c00cf9f0 tx=0x7f67b002b880 comp rx=0 tx=0).stop 2024-04-02T13:21:20.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.221+0000 7f67c7b4a640 1 -- 172.21.15.44:0/2912776333 shutdown_connections 2024-04-02T13:21:20.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.221+0000 7f67c7b4a640 1 --2- 172.21.15.44:0/2912776333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67c00b3510 0x7f67c00d0390 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.221+0000 7f67c7b4a640 1 --2- 172.21.15.44:0/2912776333 >> v2:172.21.15.69:3300/0 conn(0x7f67c015c250 0x7f67c00aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.221+0000 7f67c7b4a640 1 -- 172.21.15.44:0/2912776333 >> 172.21.15.44:0/2912776333 conn(0x7f67c0073850 msgr2=0x7f67c009aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:20.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.221+0000 7f67c7b4a640 1 -- 172.21.15.44:0/2912776333 shutdown_connections 2024-04-02T13:21:20.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.221+0000 7f67c7b4a640 1 -- 172.21.15.44:0/2912776333 wait complete. 2024-04-02T13:21:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.222+0000 7f67c7b4a640 1 Processor -- start 2024-04-02T13:21:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.222+0000 7f67c7b4a640 1 -- start start 2024-04-02T13:21:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.223+0000 7f67c7b4a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67c00b3510 0x7f67c00b65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.223+0000 7f67c7b4a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c00d08d0 0x7f67c00b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.223+0000 7f67c7b4a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67c015c250 0x7f67c015e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.223+0000 7f67c7b4a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f67c00af130 con 0x7f67c00b3510 2024-04-02T13:21:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.223+0000 7f67c7b4a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f67c00afcc0 con 0x7f67c00d08d0 2024-04-02T13:21:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.223+0000 7f67c7b4a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f67c00b14d0 con 0x7f67c015c250 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.223+0000 7f67c58e6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67c00b3510 0x7f67c00b65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.223+0000 7f67c58e6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67c00b3510 0x7f67c00b65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45946/0 (socket says 172.21.15.44:45946) 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.223+0000 7f67c58e6640 1 -- 172.21.15.44:0/96423011 learned_addr learned my addr 172.21.15.44:0/96423011 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.223+0000 7f67c50e5640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c00d08d0 0x7f67c00b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.224+0000 7f67c60e7640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67c015c250 0x7f67c015e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.224+0000 7f67c58e6640 1 -- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67c015c250 msgr2=0x7f67c015e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.224+0000 7f67c58e6640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67c015c250 0x7f67c015e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.224+0000 7f67c58e6640 1 -- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c00d08d0 msgr2=0x7f67c00b7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.224+0000 7f67c58e6640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c00d08d0 0x7f67c00b7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.224+0000 7f67c58e6640 1 -- 172.21.15.44:0/96423011 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c01f5380 con 0x7f67c00b3510 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.224+0000 7f67c58e6640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67c00b3510 0x7f67c00b65c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f67b800c820 tx=0x7f67b800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.224+0000 7f67b67fc640 1 -- 172.21.15.44:0/96423011 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b8015070 con 0x7f67c00b3510 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.224+0000 7f67b67fc640 1 -- 172.21.15.44:0/96423011 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67b8004020 con 0x7f67c00b3510 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.224+0000 7f67c7b4a640 1 -- 172.21.15.44:0/96423011 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67c01f5660 con 0x7f67c00b3510 2024-04-02T13:21:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.224+0000 7f67b67fc640 1 -- 172.21.15.44:0/96423011 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b80043f0 con 0x7f67c00b3510 2024-04-02T13:21:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.224+0000 7f67c7b4a640 1 -- 172.21.15.44:0/96423011 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67c01f5c20 con 0x7f67c00b3510 2024-04-02T13:21:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.225+0000 7f67c7b4a640 1 -- 172.21.15.44:0/96423011 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67c0068c00 con 0x7f67c00b3510 2024-04-02T13:21:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.225+0000 7f67b67fc640 1 -- 172.21.15.44:0/96423011 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f67b8007830 con 0x7f67c00b3510 2024-04-02T13:21:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.226+0000 7f67b67fc640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f67a406c8f0 0x7f67a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.226+0000 7f67b67fc640 1 -- 172.21.15.44:0/96423011 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f67b808bd80 con 0x7f67c00b3510 2024-04-02T13:21:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.226+0000 7f67c50e5640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f67a406c8f0 0x7f67a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.226+0000 7f67c50e5640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f67a406c8f0 0x7f67a406edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f67b0036000 tx=0x7f67b003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:20.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.230+0000 7f67b67fc640 1 -- 172.21.15.44:0/96423011 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f67b8059d90 con 0x7f67c00b3510 2024-04-02T13:21:20.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.430+0000 7f67c7b4a640 1 -- 172.21.15.44:0/96423011 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f67c0070010 con 0x7f67c00b3510 2024-04-02T13:21:20.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.431+0000 7f67b67fc640 1 -- 172.21.15.44:0/96423011 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f67b8059690 con 0x7f67c00b3510 2024-04-02T13:21:20.432 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:21:20.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.432+0000 7f67c7b4a640 1 -- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f67a406c8f0 msgr2=0x7f67a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:20.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.432+0000 7f67c7b4a640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f67a406c8f0 0x7f67a406edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f67b0036000 tx=0x7f67b003d040 comp rx=0 tx=0).stop 2024-04-02T13:21:20.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.432+0000 7f67c7b4a640 1 -- 172.21.15.44:0/96423011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67c00b3510 msgr2=0x7f67c00b65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.432+0000 7f67c7b4a640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67c00b3510 0x7f67c00b65c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f67b800c820 tx=0x7f67b800cce0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.432+0000 7f67c7b4a640 1 -- 172.21.15.44:0/96423011 shutdown_connections 2024-04-02T13:21:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.432+0000 7f67c7b4a640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f67a406c8f0 0x7f67a406edb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.432+0000 7f67c7b4a640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67c015c250 0x7f67c015e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.432+0000 7f67c7b4a640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c00d08d0 0x7f67c00b7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.432+0000 7f67c7b4a640 1 --2- 172.21.15.44:0/96423011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67c00b3510 0x7f67c00b65c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.432+0000 7f67c7b4a640 1 -- 172.21.15.44:0/96423011 >> 172.21.15.44:0/96423011 conn(0x7f67c0073850 msgr2=0x7f67c015b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.433+0000 7f67c7b4a640 1 -- 172.21.15.44:0/96423011 shutdown_connections 2024-04-02T13:21:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.433+0000 7f67c7b4a640 1 -- 172.21.15.44:0/96423011 wait complete. 2024-04-02T13:21:20.444 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-02T13:21:20.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:20.445 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-02T13:21:20.477 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:21:20.477 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:21:20.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:20.477 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-02T13:21:20.612 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:21:20.612 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:20.627 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:21:20.627 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:20.682 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:20.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.822+0000 7fdcb2642640 1 Processor -- start 2024-04-02T13:21:20.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.822+0000 7fdcb2642640 1 -- start start 2024-04-02T13:21:20.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.823+0000 7fdcb2642640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcac15c1c0 0x7fdcac0affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:20.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.823+0000 7fdcb2642640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdcac0b3d70 0x7fdcac0b0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:20.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.823+0000 7fdcb2642640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcac0b1640 con 0x7fdcac0b0a40 2024-04-02T13:21:20.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.823+0000 7fdcb2642640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fdcac0aef20 con 0x7fdcac0b3d70 2024-04-02T13:21:20.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.823+0000 7fdcb2642640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdcac0af200 con 0x7fdcac15c1c0 2024-04-02T13:21:20.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.823+0000 7fdcab7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdcac0b3d70 0x7fdcac0b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:20.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.823+0000 7fdcabfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcac15c1c0 0x7fdcac0affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:20.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.823+0000 7fdcb0bdf640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdcac0b0a40 0x7fdcac164e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:54488/0 (socket says 172.21.15.44:54488) 2024-04-02T13:21:20.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.823+0000 7fdcb0bdf640 1 -- 172.21.15.44:0/702907508 learned_addr learned my addr 172.21.15.44:0/702907508 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:20.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.824+0000 7fdcaaffd640 1 -- 172.21.15.44:0/702907508 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 790472377 0 0) 0x7fdca4001490 con 0x7fdcac0b0a40 2024-04-02T13:21:20.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.824+0000 7fdcabfff640 1 -- 172.21.15.44:0/702907508 >> v2:172.21.15.69:3300/0 conn(0x7fdcac0b3d70 msgr2=0x7fdcac0b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:20.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.824+0000 7fdcabfff640 1 --2- 172.21.15.44:0/702907508 >> v2:172.21.15.69:3300/0 conn(0x7fdcac0b3d70 0x7fdcac0b0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.824+0000 7fdcabfff640 1 -- 172.21.15.44:0/702907508 >> v1:172.21.15.69:6789/0 conn(0x7fdcac0b0a40 legacy=0x7fdcac164e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:20.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.824+0000 7fdcabfff640 1 -- 172.21.15.44:0/702907508 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcac0af4e0 con 0x7fdcac15c1c0 2024-04-02T13:21:20.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.825+0000 7fdcabfff640 1 --2- 172.21.15.44:0/702907508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcac15c1c0 0x7fdcac0affc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c0097a0 tx=0x7fdc9c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ae1cfbf32961d9e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:20.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.825+0000 7fdcaaffd640 1 -- 172.21.15.44:0/702907508 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c035020 con 0x7fdcac15c1c0 2024-04-02T13:21:20.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.825+0000 7fdcaaffd640 1 -- 172.21.15.44:0/702907508 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc9c031830 con 0x7fdcac15c1c0 2024-04-02T13:21:20.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.825+0000 7fdcaaffd640 1 -- 172.21.15.44:0/702907508 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c031c00 con 0x7fdcac15c1c0 2024-04-02T13:21:20.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.825+0000 7fdcb2642640 1 -- 172.21.15.44:0/702907508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcac15c1c0 msgr2=0x7fdcac0affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:20.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.825+0000 7fdcb2642640 1 --2- 172.21.15.44:0/702907508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcac15c1c0 0x7fdcac0affc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c0097a0 tx=0x7fdc9c02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.825+0000 7fdcb2642640 1 -- 172.21.15.44:0/702907508 shutdown_connections 2024-04-02T13:21:20.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.825+0000 7fdcb2642640 1 --2- 172.21.15.44:0/702907508 >> v2:172.21.15.69:3300/0 conn(0x7fdcac0b3d70 0x7fdcac0b0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.825+0000 7fdcb2642640 1 --2- 172.21.15.44:0/702907508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcac15c1c0 0x7fdcac0affc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.825+0000 7fdcb2642640 1 -- 172.21.15.44:0/702907508 >> 172.21.15.44:0/702907508 conn(0x7fdcac09aa20 msgr2=0x7fdcac09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:20.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.825+0000 7fdcb2642640 1 -- 172.21.15.44:0/702907508 shutdown_connections 2024-04-02T13:21:20.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.825+0000 7fdcb2642640 1 -- 172.21.15.44:0/702907508 wait complete. 2024-04-02T13:21:20.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.827+0000 7fdcb2642640 1 Processor -- start 2024-04-02T13:21:20.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.827+0000 7fdcb2642640 1 -- start start 2024-04-02T13:21:20.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.827+0000 7fdcb2642640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcac0b0a40 0x7fdcac0c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:20.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.827+0000 7fdcb2642640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcac0b3d70 0x7fdcac070670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.827+0000 7fdcb2642640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcac15c1c0 0x7fdcac1f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.827+0000 7fdcb2642640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdcac0cc990 con 0x7fdcac0b0a40 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.827+0000 7fdcb2642640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdcac0afaa0 con 0x7fdcac15c1c0 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.827+0000 7fdcb2642640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdcac0aebf0 con 0x7fdcac0b3d70 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdcabfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcac0b0a40 0x7fdcac0c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdcabfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcac0b0a40 0x7fdcac0c7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45956/0 (socket says 172.21.15.44:45956) 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdcabfff640 1 -- 172.21.15.44:0/1024722048 learned_addr learned my addr 172.21.15.44:0/1024722048 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdcab7fe640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcac0b3d70 0x7fdcac070670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdcb0bdf640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcac15c1c0 0x7fdcac1f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdcabfff640 1 -- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcac0b3d70 msgr2=0x7fdcac070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdcabfff640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcac0b3d70 0x7fdcac070670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdcabfff640 1 -- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcac15c1c0 msgr2=0x7fdcac1f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdcabfff640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcac15c1c0 0x7fdcac1f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:20.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdcabfff640 1 -- 172.21.15.44:0/1024722048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcac1f7d10 con 0x7fdcac0b0a40 2024-04-02T13:21:20.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdcabfff640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcac0b0a40 0x7fdcac0c7300 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c02bee0 tx=0x7fdc9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:20.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdca8ff9640 1 -- 172.21.15.44:0/1024722048 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c035040 con 0x7fdcac0b0a40 2024-04-02T13:21:20.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdca8ff9640 1 -- 172.21.15.44:0/1024722048 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc9c03b920 con 0x7fdcac0b0a40 2024-04-02T13:21:20.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdca8ff9640 1 -- 172.21.15.44:0/1024722048 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c03bd10 con 0x7fdcac0b0a40 2024-04-02T13:21:20.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdcb2642640 1 -- 172.21.15.44:0/1024722048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcac1f7ff0 con 0x7fdcac0b0a40 2024-04-02T13:21:20.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.828+0000 7fdcb2642640 1 -- 172.21.15.44:0/1024722048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcac0c5f80 con 0x7fdcac0b0a40 2024-04-02T13:21:20.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.829+0000 7fdcb2642640 1 -- 172.21.15.44:0/1024722048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcac0684f0 con 0x7fdcac0b0a40 2024-04-02T13:21:20.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.830+0000 7fdca8ff9640 1 -- 172.21.15.44:0/1024722048 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdc9c038020 con 0x7fdcac0b0a40 2024-04-02T13:21:20.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.830+0000 7fdca8ff9640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc8c06c8f0 0x7fdc8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:20.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.830+0000 7fdca8ff9640 1 -- 172.21.15.44:0/1024722048 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fdc9c0adf20 con 0x7fdcac0b0a40 2024-04-02T13:21:20.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.830+0000 7fdcab7fe640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc8c06c8f0 0x7fdc8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:20.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.831+0000 7fdcab7fe640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc8c06c8f0 0x7fdc8c06edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdcac06e340 tx=0x7fdc94009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:20.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:20.834+0000 7fdca8ff9640 1 -- 172.21.15.44:0/1024722048 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fdc9c031800 con 0x7fdcac0b0a40 2024-04-02T13:21:21.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.026+0000 7fdcb2642640 1 -- 172.21.15.44:0/1024722048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdcac0b27d0 con 0x7fdcac0b0a40 2024-04-02T13:21:21.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.028+0000 7fdca8ff9640 1 -- 172.21.15.44:0/1024722048 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18899 (secure 0 0 0) 0x7fdc9c07b8f0 con 0x7fdcac0b0a40 2024-04-02T13:21:21.029 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:21.029 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:08.061253+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:21.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.030+0000 7fdcb2642640 1 -- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc8c06c8f0 msgr2=0x7fdc8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.030+0000 7fdcb2642640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc8c06c8f0 0x7fdc8c06edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdcac06e340 tx=0x7fdc94009190 comp rx=0 tx=0).stop 2024-04-02T13:21:21.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.030+0000 7fdcb2642640 1 -- 172.21.15.44:0/1024722048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcac0b0a40 msgr2=0x7fdcac0c7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.030+0000 7fdcb2642640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcac0b0a40 0x7fdcac0c7300 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c02bee0 tx=0x7fdc9c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.030+0000 7fdcb2642640 1 -- 172.21.15.44:0/1024722048 shutdown_connections 2024-04-02T13:21:21.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.030+0000 7fdcb2642640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc8c06c8f0 0x7fdc8c06edb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.030+0000 7fdcb2642640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcac15c1c0 0x7fdcac1f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.030+0000 7fdcb2642640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcac0b3d70 0x7fdcac070670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.030+0000 7fdcb2642640 1 --2- 172.21.15.44:0/1024722048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcac0b0a40 0x7fdcac0c7300 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.030+0000 7fdcb2642640 1 -- 172.21.15.44:0/1024722048 >> 172.21.15.44:0/1024722048 conn(0x7fdcac09aa20 msgr2=0x7fdcac0b2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:21.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.030+0000 7fdcb2642640 1 -- 172.21.15.44:0/1024722048 shutdown_connections 2024-04-02T13:21:21.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.030+0000 7fdcb2642640 1 -- 172.21.15.44:0/1024722048 wait complete. 2024-04-02T13:21:21.042 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:21.043 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:21.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.146+0000 7ff13681b640 1 Processor -- start 2024-04-02T13:21:21.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.146+0000 7ff13681b640 1 -- start start 2024-04-02T13:21:21.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.147+0000 7ff13681b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff13015e5d0 0x7ff1300d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.147+0000 7ff13681b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1300d1630 con 0x7ff13006cff0 2024-04-02T13:21:21.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.147+0000 7ff13681b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1300d1910 con 0x7ff1300b4180 2024-04-02T13:21:21.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.147+0000 7ff13681b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff1300d1bf0 con 0x7ff13015e5d0 2024-04-02T13:21:21.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.147+0000 7ff12ffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff13006cff0 0x7ff13006d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48936/0 (socket says 172.21.15.44:48936) 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.147+0000 7ff12ffff640 1 -- 172.21.15.44:0/3339389111 learned_addr learned my addr 172.21.15.44:0/3339389111 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.147+0000 7ff12f7fe640 1 --2- 172.21.15.44:0/3339389111 >> v2:172.21.15.69:3300/0 conn(0x7ff13015e5d0 0x7ff1300d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.147+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4147286698 0 0) 0x7ff1300d1630 con 0x7ff13006cff0 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.147+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff118003520 con 0x7ff13006cff0 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.147+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3493817256 0 0) 0x7ff1300d1910 con 0x7ff1300b4180 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1300d1630 con 0x7ff1300b4180 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 397740088 0 0) 0x7ff118003520 con 0x7ff13006cff0 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 >> v2:172.21.15.69:3300/0 conn(0x7ff13015e5d0 msgr2=0x7ff1300d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 --2- 172.21.15.44:0/3339389111 >> v2:172.21.15.69:3300/0 conn(0x7ff13015e5d0 0x7ff1300d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 >> v1:172.21.15.69:6789/0 conn(0x7ff1300b4180 legacy=0x7ff1300b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1300b1540 con 0x7ff13006cff0 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7ff120001670 con 0x7ff13006cff0 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 >> v1:172.21.15.44:6789/0 conn(0x7ff13006cff0 legacy=0x7ff13006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 --2- 172.21.15.44:0/3339389111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff118003b40 0x7ff118003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 --2- 172.21.15.44:0/3339389111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff118004800 0x7ff118004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 --2- 172.21.15.44:0/3339389111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1180051c0 0x7ff11800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff1300d1bf0 con 0x7ff118003b40 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff11800c0f0 con 0x7ff1180051c0 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff11800c2f0 con 0x7ff118004800 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12ffff640 1 --2- 172.21.15.44:0/3339389111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff118003b40 0x7ff118003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff134db8640 1 --2- 172.21.15.44:0/3339389111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1180051c0 0x7ff11800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12f7fe640 1 --2- 172.21.15.44:0/3339389111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff118004800 0x7ff118004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12ffff640 1 -- 172.21.15.44:0/3339389111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff118004800 msgr2=0x7ff118004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12ffff640 1 --2- 172.21.15.44:0/3339389111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff118004800 0x7ff118004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12ffff640 1 -- 172.21.15.44:0/3339389111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1180051c0 msgr2=0x7ff11800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12ffff640 1 --2- 172.21.15.44:0/3339389111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1180051c0 0x7ff11800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff12ffff640 1 -- 172.21.15.44:0/3339389111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff118003520 con 0x7ff118003b40 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.148+0000 7ff134db8640 1 -- 172.21.15.44:0/3339389111 reap_dead start 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.149+0000 7ff12ffff640 1 --2- 172.21.15.44:0/3339389111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff118003b40 0x7ff118003f40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff12000c190 tx=0x7ff118005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b699c600f8c887c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:21.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.149+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120007cc0 con 0x7ff118003b40 2024-04-02T13:21:21.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.149+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff120030070 con 0x7ff118003b40 2024-04-02T13:21:21.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.149+0000 7ff12effd640 1 -- 172.21.15.44:0/3339389111 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120005d70 con 0x7ff118003b40 2024-04-02T13:21:21.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.149+0000 7ff13681b640 1 -- 172.21.15.44:0/3339389111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff118003b40 msgr2=0x7ff118003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.149+0000 7ff13681b640 1 --2- 172.21.15.44:0/3339389111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff118003b40 0x7ff118003f40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff12000c190 tx=0x7ff118005830 comp rx=0 tx=0).stop 2024-04-02T13:21:21.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.149+0000 7ff13681b640 1 -- 172.21.15.44:0/3339389111 shutdown_connections 2024-04-02T13:21:21.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.150+0000 7ff13681b640 1 --2- 172.21.15.44:0/3339389111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff118003b40 0x7ff118003f40 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.150+0000 7ff13681b640 1 -- 172.21.15.44:0/3339389111 >> 172.21.15.44:0/3339389111 conn(0x7ff13009aa20 msgr2=0x7ff13009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:21.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.150+0000 7ff13681b640 1 -- 172.21.15.44:0/3339389111 shutdown_connections 2024-04-02T13:21:21.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.150+0000 7ff13681b640 1 -- 172.21.15.44:0/3339389111 wait complete. 2024-04-02T13:21:21.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.151+0000 7ff13681b640 1 Processor -- start 2024-04-02T13:21:21.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.151+0000 7ff13681b640 1 -- start start 2024-04-02T13:21:21.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff13681b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff118003b40 0x7ff1300b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff13681b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff130161330 0x7ff1301f6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff13681b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff13006fec0 0x7ff130070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff13681b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff13006d550 con 0x7ff118003b40 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff13681b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff13006d3d0 con 0x7ff13006fec0 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff13681b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff13006d6d0 con 0x7ff130161330 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff12ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff118003b40 0x7ff1300b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff12ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff118003b40 0x7ff1300b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45978/0 (socket says 172.21.15.44:45978) 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff12ffff640 1 -- 172.21.15.44:0/1375527558 learned_addr learned my addr 172.21.15.44:0/1375527558 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff134db8640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff13006fec0 0x7ff130070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff12f7fe640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff130161330 0x7ff1301f6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff12ffff640 1 -- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff130161330 msgr2=0x7ff1301f6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff12ffff640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff130161330 0x7ff1301f6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff12ffff640 1 -- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff13006fec0 msgr2=0x7ff130070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff12ffff640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff13006fec0 0x7ff130070340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff12ffff640 1 -- 172.21.15.44:0/1375527558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1300b4cf0 con 0x7ff118003b40 2024-04-02T13:21:21.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff12ffff640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff118003b40 0x7ff1300b4660 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff1200018f0 tx=0x7ff1200019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:21.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff12cff9640 1 -- 172.21.15.44:0/1375527558 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120045070 con 0x7ff118003b40 2024-04-02T13:21:21.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff12cff9640 1 -- 172.21.15.44:0/1375527558 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff12002c9c0 con 0x7ff118003b40 2024-04-02T13:21:21.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff13681b640 1 -- 172.21.15.44:0/1375527558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1300b4fd0 con 0x7ff118003b40 2024-04-02T13:21:21.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff12cff9640 1 -- 172.21.15.44:0/1375527558 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12002cd70 con 0x7ff118003b40 2024-04-02T13:21:21.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.152+0000 7ff13681b640 1 -- 172.21.15.44:0/1375527558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1300b5590 con 0x7ff118003b40 2024-04-02T13:21:21.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.153+0000 7ff13681b640 1 -- 172.21.15.44:0/1375527558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1300684f0 con 0x7ff118003b40 2024-04-02T13:21:21.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.154+0000 7ff12cff9640 1 -- 172.21.15.44:0/1375527558 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff120007070 con 0x7ff118003b40 2024-04-02T13:21:21.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.154+0000 7ff12cff9640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff11c06c920 0x7ff11c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.154+0000 7ff12cff9640 1 -- 172.21.15.44:0/1375527558 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7ff1200acdf0 con 0x7ff118003b40 2024-04-02T13:21:21.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.155+0000 7ff12f7fe640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff11c06c920 0x7ff11c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.155+0000 7ff12f7fe640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff11c06c920 0x7ff11c06ede0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff1300b26c0 tx=0x7ff124008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:21.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.159+0000 7ff12cff9640 1 -- 172.21.15.44:0/1375527558 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff12007adf0 con 0x7ff118003b40 2024-04-02T13:21:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.343+0000 7ff13681b640 1 -- 172.21.15.44:0/1375527558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff1300d0950 con 0x7ff118003b40 2024-04-02T13:21:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.343+0000 7ff12cff9640 1 -- 172.21.15.44:0/1375527558 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7ff12007a6f0 con 0x7ff118003b40 2024-04-02T13:21:21.345 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.345+0000 7ff13681b640 1 -- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff11c06c920 msgr2=0x7ff11c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.345+0000 7ff13681b640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff11c06c920 0x7ff11c06ede0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff1300b26c0 tx=0x7ff124008040 comp rx=0 tx=0).stop 2024-04-02T13:21:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.345+0000 7ff13681b640 1 -- 172.21.15.44:0/1375527558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff118003b40 msgr2=0x7ff1300b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.345+0000 7ff13681b640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff118003b40 0x7ff1300b4660 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff1200018f0 tx=0x7ff1200019c0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.345+0000 7ff13681b640 1 -- 172.21.15.44:0/1375527558 shutdown_connections 2024-04-02T13:21:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.345+0000 7ff13681b640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff11c06c920 0x7ff11c06ede0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.345+0000 7ff13681b640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff13006fec0 0x7ff130070340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.345+0000 7ff13681b640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff130161330 0x7ff1301f6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.345+0000 7ff13681b640 1 --2- 172.21.15.44:0/1375527558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff118003b40 0x7ff1300b4660 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.345+0000 7ff13681b640 1 -- 172.21.15.44:0/1375527558 >> 172.21.15.44:0/1375527558 conn(0x7ff13009aa20 msgr2=0x7ff1300b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.345+0000 7ff13681b640 1 -- 172.21.15.44:0/1375527558 shutdown_connections 2024-04-02T13:21:21.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.345+0000 7ff13681b640 1 -- 172.21.15.44:0/1375527558 wait complete. 2024-04-02T13:21:21.357 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:21.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.460+0000 7fb5ac1c8640 1 Processor -- start 2024-04-02T13:21:21.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.460+0000 7fb5ac1c8640 1 -- start start 2024-04-02T13:21:21.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5ac1c8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb5a415e470 0x7fb5a40b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5ac1c8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb5a40ce790 0x7fb5a406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5ac1c8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5a40ced00 con 0x7fb5a40b1540 2024-04-02T13:21:21.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5ac1c8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fb5a40cefe0 con 0x7fb5a415e470 2024-04-02T13:21:21.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5ac1c8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb5a40cf2c0 con 0x7fb5a40ce790 2024-04-02T13:21:21.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5a9f64640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb5a415e470 0x7fb5a40b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5a9763640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb5a40b1540 0x7fb5a40d1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48938/0 (socket says 172.21.15.44:48938) 2024-04-02T13:21:21.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5a9f64640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb5a415e470 0x7fb5a40b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45982/0 (socket says 172.21.15.44:45982) 2024-04-02T13:21:21.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5a9763640 1 -- 172.21.15.44:0/2567032605 learned_addr learned my addr 172.21.15.44:0/2567032605 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:21.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5aa765640 1 --2- 172.21.15.44:0/2567032605 >> v2:172.21.15.69:3300/0 conn(0x7fb5a40ce790 0x7fb5a406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5a9f64640 1 -- 172.21.15.44:0/2567032605 >> v2:172.21.15.69:3300/0 conn(0x7fb5a40ce790 msgr2=0x7fb5a406d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:21.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5a9f64640 1 --2- 172.21.15.44:0/2567032605 >> v2:172.21.15.69:3300/0 conn(0x7fb5a40ce790 0x7fb5a406d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5a9f64640 1 -- 172.21.15.44:0/2567032605 >> v1:172.21.15.44:6789/0 conn(0x7fb5a40b1540 legacy=0x7fb5a40d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5a8f62640 1 -- 172.21.15.44:0/2567032605 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1733150118 0 0) 0x7fb5a40ced00 con 0x7fb5a40b1540 2024-04-02T13:21:21.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.461+0000 7fb5a9f64640 1 -- 172.21.15.44:0/2567032605 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a40cf5a0 con 0x7fb5a415e470 2024-04-02T13:21:21.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.462+0000 7fb5a9f64640 1 --2- 172.21.15.44:0/2567032605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5a415e470 0x7fb5a40b32c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb5a40cc460 tx=0x7fb59c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67abde22ea5bd1b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:21.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.462+0000 7fb5a8f62640 1 -- 172.21.15.44:0/2567032605 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c039070 con 0x7fb5a415e470 2024-04-02T13:21:21.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.462+0000 7fb5a8f62640 1 -- 172.21.15.44:0/2567032605 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb59c02cbf0 con 0x7fb5a415e470 2024-04-02T13:21:21.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.462+0000 7fb5a8f62640 1 -- 172.21.15.44:0/2567032605 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c034440 con 0x7fb5a415e470 2024-04-02T13:21:21.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.463+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/2567032605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5a415e470 msgr2=0x7fb5a40b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.463+0000 7fb5ac1c8640 1 --2- 172.21.15.44:0/2567032605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5a415e470 0x7fb5a40b32c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb5a40cc460 tx=0x7fb59c02be40 comp rx=0 tx=0).stop 2024-04-02T13:21:21.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.463+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/2567032605 shutdown_connections 2024-04-02T13:21:21.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.463+0000 7fb5ac1c8640 1 --2- 172.21.15.44:0/2567032605 >> v2:172.21.15.69:3300/0 conn(0x7fb5a40ce790 0x7fb5a406d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.463+0000 7fb5ac1c8640 1 --2- 172.21.15.44:0/2567032605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5a415e470 0x7fb5a40b32c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.463+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/2567032605 >> 172.21.15.44:0/2567032605 conn(0x7fb5a409aa20 msgr2=0x7fb5a409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:21.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.463+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/2567032605 shutdown_connections 2024-04-02T13:21:21.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.463+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/2567032605 wait complete. 2024-04-02T13:21:21.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.464+0000 7fb5ac1c8640 1 Processor -- start 2024-04-02T13:21:21.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.464+0000 7fb5ac1c8640 1 -- start start 2024-04-02T13:21:21.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5ac1c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5a40b1540 0x7fb5a414cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5ac1c8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5a40ce790 0x7fb5a414d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5ac1c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5a415e470 0x7fb5a40c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5ac1c8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a40d22c0 con 0x7fb5a40b1540 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5ac1c8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a406cb60 con 0x7fb5a40ce790 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5ac1c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a40d2440 con 0x7fb5a415e470 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5a9f64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5a40b1540 0x7fb5a414cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5a9f64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5a40b1540 0x7fb5a414cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45994/0 (socket says 172.21.15.44:45994) 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5a9f64640 1 -- 172.21.15.44:0/160996463 learned_addr learned my addr 172.21.15.44:0/160996463 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5a9763640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5a40ce790 0x7fb5a414d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5aa765640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5a415e470 0x7fb5a40c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5a9f64640 1 -- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5a415e470 msgr2=0x7fb5a40c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5a9f64640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5a415e470 0x7fb5a40c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5a9f64640 1 -- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5a40ce790 msgr2=0x7fb5a414d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5a9f64640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5a40ce790 0x7fb5a414d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5a9f64640 1 -- 172.21.15.44:0/160996463 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a41617f0 con 0x7fb5a40b1540 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.465+0000 7fb5a9f64640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5a40b1540 0x7fb5a414cc50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb59c037040 tx=0x7fb59c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.466+0000 7fb5a27fc640 1 -- 172.21.15.44:0/160996463 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c039050 con 0x7fb5a40b1540 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.466+0000 7fb5a27fc640 1 -- 172.21.15.44:0/160996463 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb59c045070 con 0x7fb5a40b1540 2024-04-02T13:21:21.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.466+0000 7fb5a27fc640 1 -- 172.21.15.44:0/160996463 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c03e850 con 0x7fb5a40b1540 2024-04-02T13:21:21.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.466+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/160996463 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a40cb9f0 con 0x7fb5a40b1540 2024-04-02T13:21:21.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.466+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/160996463 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a40c6560 con 0x7fb5a40b1540 2024-04-02T13:21:21.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.467+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/160996463 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a40684f0 con 0x7fb5a40b1540 2024-04-02T13:21:21.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.467+0000 7fb5a27fc640 1 -- 172.21.15.44:0/160996463 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb59c03e9f0 con 0x7fb5a40b1540 2024-04-02T13:21:21.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.467+0000 7fb5a27fc640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb59006c920 0x7fb59006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.468+0000 7fb5a27fc640 1 -- 172.21.15.44:0/160996463 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fb59c0aed50 con 0x7fb5a40b1540 2024-04-02T13:21:21.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.468+0000 7fb5a9763640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb59006c920 0x7fb59006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.468+0000 7fb5a9763640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb59006c920 0x7fb59006ede0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb5a40c8820 tx=0x7fb598007b90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:21.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.472+0000 7fb5a27fc640 1 -- 172.21.15.44:0/160996463 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb59c0b2030 con 0x7fb5a40b1540 2024-04-02T13:21:21.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.657+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/160996463 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb5a40d2d10 con 0x7fb5a40b1540 2024-04-02T13:21:21.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.658+0000 7fb5a27fc640 1 -- 172.21.15.44:0/160996463 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fb59c07cf10 con 0x7fb5a40b1540 2024-04-02T13:21:21.659 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:21.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.660+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb59006c920 msgr2=0x7fb59006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.660+0000 7fb5ac1c8640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb59006c920 0x7fb59006ede0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb5a40c8820 tx=0x7fb598007b90 comp rx=0 tx=0).stop 2024-04-02T13:21:21.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.660+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/160996463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5a40b1540 msgr2=0x7fb5a414cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.660+0000 7fb5ac1c8640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5a40b1540 0x7fb5a414cc50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb59c037040 tx=0x7fb59c004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:21.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.660+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/160996463 shutdown_connections 2024-04-02T13:21:21.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.660+0000 7fb5ac1c8640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb59006c920 0x7fb59006ede0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.660+0000 7fb5ac1c8640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5a415e470 0x7fb5a40c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.660+0000 7fb5ac1c8640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5a40ce790 0x7fb5a414d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.660+0000 7fb5ac1c8640 1 --2- 172.21.15.44:0/160996463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5a40b1540 0x7fb5a414cc50 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.660+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/160996463 >> 172.21.15.44:0/160996463 conn(0x7fb5a409aa20 msgr2=0x7fb5a409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:21.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.660+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/160996463 shutdown_connections 2024-04-02T13:21:21.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.660+0000 7fb5ac1c8640 1 -- 172.21.15.44:0/160996463 wait complete. 2024-04-02T13:21:21.672 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:21.763 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:21:21.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:21:21.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.775+0000 7f56fdb40640 1 Processor -- start 2024-04-02T13:21:21.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.775+0000 7f56fdb40640 1 -- start start 2024-04-02T13:21:21.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.775+0000 7f56fdb40640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f56f815e260 0x7f56f80d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.775+0000 7f56fdb40640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f56f815e640 0x7f56f80d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.775+0000 7f56fdb40640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f80cf120 con 0x7f56f80cec60 2024-04-02T13:21:21.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.775+0000 7f56fdb40640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f56f80b5a50 con 0x7f56f815e260 2024-04-02T13:21:21.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.776+0000 7f56fdb40640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f56f80d01c0 con 0x7f56f815e640 2024-04-02T13:21:21.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.776+0000 7f56f7fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f56f80cec60 0x7f56f8161f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48950/0 (socket says 172.21.15.44:48950) 2024-04-02T13:21:21.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.776+0000 7f56f77fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f56f815e260 0x7f56f80d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.776+0000 7f56f7fff640 1 -- 172.21.15.44:0/1723615689 learned_addr learned my addr 172.21.15.44:0/1723615689 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:21.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.776+0000 7f56f6ffd640 1 --2- 172.21.15.44:0/1723615689 >> v2:172.21.15.69:3300/0 conn(0x7f56f815e640 0x7f56f80d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.776+0000 7f56f77fe640 1 -- 172.21.15.44:0/1723615689 >> v2:172.21.15.69:3300/0 conn(0x7f56f815e640 msgr2=0x7f56f80d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:21.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.776+0000 7f56f77fe640 1 --2- 172.21.15.44:0/1723615689 >> v2:172.21.15.69:3300/0 conn(0x7f56f815e640 0x7f56f80d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.776+0000 7f56f77fe640 1 -- 172.21.15.44:0/1723615689 >> v1:172.21.15.44:6789/0 conn(0x7f56f80cec60 legacy=0x7f56f8161f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.776+0000 7f56f77fe640 1 -- 172.21.15.44:0/1723615689 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f80cf400 con 0x7f56f815e260 2024-04-02T13:21:21.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.776+0000 7f56f67fc640 1 -- 172.21.15.44:0/1723615689 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1616878933 0 0) 0x7f56f80cf120 con 0x7f56f80cec60 2024-04-02T13:21:21.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.777+0000 7f56f77fe640 1 --2- 172.21.15.44:0/1723615689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f815e260 0x7f56f80d2960 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f56f80b6c70 tx=0x7f56e802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b3ec756e04ce21 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:21.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.777+0000 7f56f67fc640 1 -- 172.21.15.44:0/1723615689 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e80315a0 con 0x7f56f815e260 2024-04-02T13:21:21.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.777+0000 7f56f67fc640 1 -- 172.21.15.44:0/1723615689 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56e8037070 con 0x7f56f815e260 2024-04-02T13:21:21.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.777+0000 7f56f67fc640 1 -- 172.21.15.44:0/1723615689 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e8031960 con 0x7f56f815e260 2024-04-02T13:21:21.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.778+0000 7f56fdb40640 1 -- 172.21.15.44:0/1723615689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f815e260 msgr2=0x7f56f80d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.778+0000 7f56fdb40640 1 --2- 172.21.15.44:0/1723615689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f815e260 0x7f56f80d2960 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f56f80b6c70 tx=0x7f56e802b840 comp rx=0 tx=0).stop 2024-04-02T13:21:21.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.778+0000 7f56fdb40640 1 -- 172.21.15.44:0/1723615689 shutdown_connections 2024-04-02T13:21:21.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.778+0000 7f56fdb40640 1 --2- 172.21.15.44:0/1723615689 >> v2:172.21.15.69:3300/0 conn(0x7f56f815e640 0x7f56f80d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.778+0000 7f56fdb40640 1 --2- 172.21.15.44:0/1723615689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f815e260 0x7f56f80d2960 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.778+0000 7f56fdb40640 1 -- 172.21.15.44:0/1723615689 >> 172.21.15.44:0/1723615689 conn(0x7f56f809d3c0 msgr2=0x7f56f80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:21.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.778+0000 7f56fdb40640 1 -- 172.21.15.44:0/1723615689 shutdown_connections 2024-04-02T13:21:21.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.778+0000 7f56fdb40640 1 -- 172.21.15.44:0/1723615689 wait complete. 2024-04-02T13:21:21.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.779+0000 7f56fdb40640 1 Processor -- start 2024-04-02T13:21:21.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.779+0000 7f56fdb40640 1 -- start start 2024-04-02T13:21:21.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.780+0000 7f56fdb40640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80cec60 0x7f56f80b96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.780+0000 7f56fdb40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f815e260 0x7f56f80b7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.780+0000 7f56fdb40640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f815e640 0x7f56f815ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.780+0000 7f56fdb40640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f56f80cfca0 con 0x7f56f815e640 2024-04-02T13:21:21.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.780+0000 7f56fdb40640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f56f80cf9c0 con 0x7f56f815e260 2024-04-02T13:21:21.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.780+0000 7f56fdb40640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f56f80d2ea0 con 0x7f56f80cec60 2024-04-02T13:21:21.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.780+0000 7f56f7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f815e640 0x7f56f815ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.780+0000 7f56f7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f815e640 0x7f56f815ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46020/0 (socket says 172.21.15.44:46020) 2024-04-02T13:21:21.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.780+0000 7f56f77fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80cec60 0x7f56f80b96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.780+0000 7f56f6ffd640 1 --2- 172.21.15.44:0/1633039397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f815e260 0x7f56f80b7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.780+0000 7f56f7fff640 1 -- 172.21.15.44:0/1633039397 learned_addr learned my addr 172.21.15.44:0/1633039397 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:21.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.781+0000 7f56f7fff640 1 -- 172.21.15.44:0/1633039397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80cec60 msgr2=0x7f56f80b96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.781+0000 7f56f7fff640 1 --2- 172.21.15.44:0/1633039397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80cec60 0x7f56f80b96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.781+0000 7f56f7fff640 1 -- 172.21.15.44:0/1633039397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f815e260 msgr2=0x7f56f80b7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.781+0000 7f56f7fff640 1 --2- 172.21.15.44:0/1633039397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f815e260 0x7f56f80b7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.781+0000 7f56f7fff640 1 -- 172.21.15.44:0/1633039397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f80cc140 con 0x7f56f815e640 2024-04-02T13:21:21.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.781+0000 7f56f7fff640 1 --2- 172.21.15.44:0/1633039397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f815e640 0x7f56f815ea50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f56f000c7a0 tx=0x7f56f000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:21.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.781+0000 7f56dffff640 1 -- 172.21.15.44:0/1633039397 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f0018070 con 0x7f56f815e640 2024-04-02T13:21:21.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.781+0000 7f56dffff640 1 -- 172.21.15.44:0/1633039397 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f000a640 con 0x7f56f815e640 2024-04-02T13:21:21.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.781+0000 7f56fdb40640 1 -- 172.21.15.44:0/1633039397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f80cc3e0 con 0x7f56f815e640 2024-04-02T13:21:21.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.781+0000 7f56dffff640 1 -- 172.21.15.44:0/1633039397 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f000aab0 con 0x7f56f815e640 2024-04-02T13:21:21.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.781+0000 7f56fdb40640 1 -- 172.21.15.44:0/1633039397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f80ccca0 con 0x7f56f815e640 2024-04-02T13:21:21.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.782+0000 7f56fdb40640 1 -- 172.21.15.44:0/1633039397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f80684f0 con 0x7f56f815e640 2024-04-02T13:21:21.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.782+0000 7f56dffff640 1 -- 172.21.15.44:0/1633039397 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f56f001b070 con 0x7f56f815e640 2024-04-02T13:21:21.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.783+0000 7f56dffff640 1 --2- 172.21.15.44:0/1633039397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56e406c920 0x7f56e406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:21.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.783+0000 7f56dffff640 1 -- 172.21.15.44:0/1633039397 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f56f008b410 con 0x7f56f815e640 2024-04-02T13:21:21.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.783+0000 7f56f77fe640 1 --2- 172.21.15.44:0/1633039397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56e406c920 0x7f56e406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:21.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.784+0000 7f56f77fe640 1 --2- 172.21.15.44:0/1633039397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56e406c920 0x7f56e406ede0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f56e8036000 tx=0x7f56e8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:21.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.788+0000 7f56dffff640 1 -- 172.21.15.44:0/1633039397 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f56f0058e60 con 0x7f56f815e640 2024-04-02T13:21:21.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.973+0000 7f56fdb40640 1 -- 172.21.15.44:0/1633039397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f56f80d47f0 con 0x7f56f815e640 2024-04-02T13:21:21.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.973+0000 7f56dffff640 1 -- 172.21.15.44:0/1633039397 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f56f005d160 con 0x7f56f815e640 2024-04-02T13:21:21.974 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:21.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.975+0000 7f56fdb40640 1 -- 172.21.15.44:0/1633039397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56e406c920 msgr2=0x7f56e406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.975+0000 7f56fdb40640 1 --2- 172.21.15.44:0/1633039397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56e406c920 0x7f56e406ede0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f56e8036000 tx=0x7f56e8008030 comp rx=0 tx=0).stop 2024-04-02T13:21:21.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.975+0000 7f56fdb40640 1 -- 172.21.15.44:0/1633039397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f815e640 msgr2=0x7f56f815ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:21.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.975+0000 7f56fdb40640 1 --2- 172.21.15.44:0/1633039397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f815e640 0x7f56f815ea50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f56f000c7a0 tx=0x7f56f000cc60 comp rx=0 tx=0).stop 2024-04-02T13:21:21.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.975+0000 7f56fdb40640 1 -- 172.21.15.44:0/1633039397 shutdown_connections 2024-04-02T13:21:21.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.975+0000 7f56fdb40640 1 --2- 172.21.15.44:0/1633039397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56e406c920 0x7f56e406ede0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.975+0000 7f56fdb40640 1 --2- 172.21.15.44:0/1633039397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f815e640 0x7f56f815ea50 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.975+0000 7f56fdb40640 1 --2- 172.21.15.44:0/1633039397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f815e260 0x7f56f80b7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.975+0000 7f56fdb40640 1 --2- 172.21.15.44:0/1633039397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80cec60 0x7f56f80b96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:21.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.975+0000 7f56fdb40640 1 -- 172.21.15.44:0/1633039397 >> 172.21.15.44:0/1633039397 conn(0x7f56f809d3c0 msgr2=0x7f56f806c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:21.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.975+0000 7f56fdb40640 1 -- 172.21.15.44:0/1633039397 shutdown_connections 2024-04-02T13:21:21.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:21.975+0000 7f56fdb40640 1 -- 172.21.15.44:0/1633039397 wait complete. 2024-04-02T13:21:21.987 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:22.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.090+0000 7f6b80c9a640 1 Processor -- start 2024-04-02T13:21:22.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.090+0000 7f6b80c9a640 1 -- start start 2024-04-02T13:21:22.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.091+0000 7f6b80c9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b7c1554c0 0x7f6b7c071120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.091+0000 7f6b80c9a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6b7c0ce9a0 0x7f6b7c0d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.091+0000 7f6b80c9a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b7c0cdc20 con 0x7f6b7c071660 2024-04-02T13:21:22.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.091+0000 7f6b80c9a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6b7c0cdec0 con 0x7f6b7c0ce9a0 2024-04-02T13:21:22.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.091+0000 7f6b80c9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6b7c0ce1a0 con 0x7f6b7c1554c0 2024-04-02T13:21:22.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.091+0000 7f6b7a59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b7c1554c0 0x7f6b7c071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.091+0000 7f6b79d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6b7c0ce9a0 0x7f6b7c0d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.091+0000 7f6b7ad9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6b7c071660 0x7f6b7c15eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:54504/0 (socket says 172.21.15.44:54504) 2024-04-02T13:21:22.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.091+0000 7f6b7ad9d640 1 -- 172.21.15.44:0/3352273611 learned_addr learned my addr 172.21.15.44:0/3352273611 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:22.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.092+0000 7f6b7a59c640 1 -- 172.21.15.44:0/3352273611 >> v2:172.21.15.69:3300/0 conn(0x7f6b7c0ce9a0 msgr2=0x7f6b7c0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:22.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.092+0000 7f6b7a59c640 1 --2- 172.21.15.44:0/3352273611 >> v2:172.21.15.69:3300/0 conn(0x7f6b7c0ce9a0 0x7f6b7c0d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.092+0000 7f6b7a59c640 1 -- 172.21.15.44:0/3352273611 >> v1:172.21.15.69:6789/0 conn(0x7f6b7c071660 legacy=0x7f6b7c15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.092+0000 7f6b7a59c640 1 -- 172.21.15.44:0/3352273611 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7c0ce480 con 0x7f6b7c1554c0 2024-04-02T13:21:22.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.092+0000 7f6b79d9b640 1 --2- 172.21.15.44:0/3352273611 >> v2:172.21.15.69:3300/0 conn(0x7f6b7c0ce9a0 0x7f6b7c0d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:22.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.093+0000 7f6b7a59c640 1 --2- 172.21.15.44:0/3352273611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b7c1554c0 0x7f6b7c071120 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6b740097a0 tx=0x7f6b7402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20232e4c4adcea3b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:22.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.093+0000 7f6b7959a640 1 -- 172.21.15.44:0/3352273611 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b74035020 con 0x7f6b7c1554c0 2024-04-02T13:21:22.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.093+0000 7f6b7959a640 1 -- 172.21.15.44:0/3352273611 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b74031830 con 0x7f6b7c1554c0 2024-04-02T13:21:22.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.093+0000 7f6b7959a640 1 -- 172.21.15.44:0/3352273611 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b74031c00 con 0x7f6b7c1554c0 2024-04-02T13:21:22.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.093+0000 7f6b80c9a640 1 -- 172.21.15.44:0/3352273611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b7c1554c0 msgr2=0x7f6b7c071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.093+0000 7f6b80c9a640 1 --2- 172.21.15.44:0/3352273611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b7c1554c0 0x7f6b7c071120 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6b740097a0 tx=0x7f6b7402b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.093+0000 7f6b80c9a640 1 -- 172.21.15.44:0/3352273611 shutdown_connections 2024-04-02T13:21:22.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.093+0000 7f6b80c9a640 1 --2- 172.21.15.44:0/3352273611 >> v2:172.21.15.69:3300/0 conn(0x7f6b7c0ce9a0 0x7f6b7c0d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.093+0000 7f6b80c9a640 1 --2- 172.21.15.44:0/3352273611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b7c1554c0 0x7f6b7c071120 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.093+0000 7f6b80c9a640 1 -- 172.21.15.44:0/3352273611 >> 172.21.15.44:0/3352273611 conn(0x7f6b7c06b920 msgr2=0x7f6b7c06bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:22.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.093+0000 7f6b80c9a640 1 -- 172.21.15.44:0/3352273611 shutdown_connections 2024-04-02T13:21:22.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.093+0000 7f6b80c9a640 1 -- 172.21.15.44:0/3352273611 wait complete. 2024-04-02T13:21:22.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.095+0000 7f6b80c9a640 1 Processor -- start 2024-04-02T13:21:22.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.095+0000 7f6b80c9a640 1 -- start start 2024-04-02T13:21:22.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.095+0000 7f6b80c9a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b7c071660 0x7f6b7c0b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.095+0000 7f6b80c9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b7c0ce9a0 0x7f6b7c14f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.095+0000 7f6b80c9a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b7c1554c0 0x7f6b7c151120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.095+0000 7f6b80c9a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7c15c290 con 0x7f6b7c1554c0 2024-04-02T13:21:22.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.095+0000 7f6b80c9a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7c071be0 con 0x7f6b7c071660 2024-04-02T13:21:22.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.095+0000 7f6b80c9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6b7c1f0900 con 0x7f6b7c0ce9a0 2024-04-02T13:21:22.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b7ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b7c1554c0 0x7f6b7c151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b7ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b7c1554c0 0x7f6b7c151120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46022/0 (socket says 172.21.15.44:46022) 2024-04-02T13:21:22.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b7ad9d640 1 -- 172.21.15.44:0/745365754 learned_addr learned my addr 172.21.15.44:0/745365754 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:22.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b79d9b640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b7c0ce9a0 0x7f6b7c14f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b7a59c640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b7c071660 0x7f6b7c0b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b7ad9d640 1 -- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b7c0ce9a0 msgr2=0x7f6b7c14f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b7ad9d640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b7c0ce9a0 0x7f6b7c14f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b7ad9d640 1 -- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b7c071660 msgr2=0x7f6b7c0b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b7ad9d640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b7c071660 0x7f6b7c0b2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b7ad9d640 1 -- 172.21.15.44:0/745365754 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7c1f0be0 con 0x7f6b7c1554c0 2024-04-02T13:21:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b7a59c640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b7c071660 0x7f6b7c0b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b7ad9d640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b7c1554c0 0x7f6b7c151120 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00cbc0 tx=0x7f6b6c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b72ffd640 1 -- 172.21.15.44:0/745365754 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c017070 con 0x7f6b7c1554c0 2024-04-02T13:21:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b72ffd640 1 -- 172.21.15.44:0/745365754 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b6c00ad40 con 0x7f6b7c1554c0 2024-04-02T13:21:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b80c9a640 1 -- 172.21.15.44:0/745365754 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b7c1f0ec0 con 0x7f6b7c1554c0 2024-04-02T13:21:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b72ffd640 1 -- 172.21.15.44:0/745365754 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c012600 con 0x7f6b7c1554c0 2024-04-02T13:21:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.096+0000 7f6b80c9a640 1 -- 172.21.15.44:0/745365754 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b7c0b2f70 con 0x7f6b7c1554c0 2024-04-02T13:21:22.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.097+0000 7f6b80c9a640 1 -- 172.21.15.44:0/745365754 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b7c0684f0 con 0x7f6b7c1554c0 2024-04-02T13:21:22.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.098+0000 7f6b72ffd640 1 -- 172.21.15.44:0/745365754 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6b6c01a070 con 0x7f6b7c1554c0 2024-04-02T13:21:22.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.098+0000 7f6b72ffd640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b5c06c8f0 0x7f6b5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.098+0000 7f6b72ffd640 1 -- 172.21.15.44:0/745365754 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f6b6c08bc80 con 0x7f6b7c1554c0 2024-04-02T13:21:22.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.099+0000 7f6b7a59c640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b5c06c8f0 0x7f6b5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.099+0000 7f6b7a59c640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b5c06c8f0 0x7f6b5c06edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6b74036000 tx=0x7f6b74008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:22.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.103+0000 7f6b72ffd640 1 -- 172.21.15.44:0/745365754 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6b6c0595a0 con 0x7f6b7c1554c0 2024-04-02T13:21:22.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.286+0000 7f6b80c9a640 1 -- 172.21.15.44:0/745365754 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6b7c0cc280 con 0x7f6b7c1554c0 2024-04-02T13:21:22.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.286+0000 7f6b72ffd640 1 -- 172.21.15.44:0/745365754 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f6b6c05d8a0 con 0x7f6b7c1554c0 2024-04-02T13:21:22.288 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:22.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.288+0000 7f6b80c9a640 1 -- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b5c06c8f0 msgr2=0x7f6b5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.288+0000 7f6b80c9a640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b5c06c8f0 0x7f6b5c06edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6b74036000 tx=0x7f6b74008030 comp rx=0 tx=0).stop 2024-04-02T13:21:22.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.288+0000 7f6b80c9a640 1 -- 172.21.15.44:0/745365754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b7c1554c0 msgr2=0x7f6b7c151120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.288+0000 7f6b80c9a640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b7c1554c0 0x7f6b7c151120 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00cbc0 tx=0x7f6b6c00a430 comp rx=0 tx=0).stop 2024-04-02T13:21:22.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.288+0000 7f6b80c9a640 1 -- 172.21.15.44:0/745365754 shutdown_connections 2024-04-02T13:21:22.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.288+0000 7f6b80c9a640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b5c06c8f0 0x7f6b5c06edb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.288+0000 7f6b80c9a640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b7c1554c0 0x7f6b7c151120 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.288+0000 7f6b80c9a640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b7c0ce9a0 0x7f6b7c14f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.288+0000 7f6b80c9a640 1 --2- 172.21.15.44:0/745365754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b7c071660 0x7f6b7c0b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.288+0000 7f6b80c9a640 1 -- 172.21.15.44:0/745365754 >> 172.21.15.44:0/745365754 conn(0x7f6b7c06b920 msgr2=0x7f6b7c06d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:22.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.288+0000 7f6b80c9a640 1 -- 172.21.15.44:0/745365754 shutdown_connections 2024-04-02T13:21:22.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.288+0000 7f6b80c9a640 1 -- 172.21.15.44:0/745365754 wait complete. 2024-04-02T13:21:22.301 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:21:22.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.403+0000 7f9622152640 1 Processor -- start 2024-04-02T13:21:22.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.403+0000 7f9622152640 1 -- start start 2024-04-02T13:21:22.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f9622152640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f961c15e4a0 0x7f961c0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f9622152640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f961c06d550 0x7f961c0d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f9622152640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f961c0cc600 con 0x7f961c0d0c50 2024-04-02T13:21:22.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f9622152640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f961c0cc8e0 con 0x7f961c06d550 2024-04-02T13:21:22.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f9622152640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f961c0ccbc0 con 0x7f961c15e4a0 2024-04-02T13:21:22.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f961bfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f961c06d550 0x7f961c0d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f961b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f961c15e4a0 0x7f961c0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f961affd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f961c0d0c50 0x7f961c06ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:54506/0 (socket says 172.21.15.44:54506) 2024-04-02T13:21:22.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f961affd640 1 -- 172.21.15.44:0/2505956233 learned_addr learned my addr 172.21.15.44:0/2505956233 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:22.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f961a7fc640 1 -- 172.21.15.44:0/2505956233 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2008110040 0 0) 0x7f961c0cc600 con 0x7f961c0d0c50 2024-04-02T13:21:22.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f961a7fc640 1 -- 172.21.15.44:0/2505956233 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9604003520 con 0x7f961c0d0c50 2024-04-02T13:21:22.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f961b7fe640 1 -- 172.21.15.44:0/2505956233 >> v2:172.21.15.69:3300/0 conn(0x7f961c06d550 msgr2=0x7f961c0d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:22.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f961b7fe640 1 --2- 172.21.15.44:0/2505956233 >> v2:172.21.15.69:3300/0 conn(0x7f961c06d550 0x7f961c0d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.404+0000 7f961b7fe640 1 -- 172.21.15.44:0/2505956233 >> v1:172.21.15.69:6789/0 conn(0x7f961c0d0c50 legacy=0x7f961c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.405+0000 7f961b7fe640 1 -- 172.21.15.44:0/2505956233 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f961c0ccea0 con 0x7f961c15e4a0 2024-04-02T13:21:22.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.405+0000 7f961bfff640 1 --2- 172.21.15.44:0/2505956233 >> v2:172.21.15.69:3300/0 conn(0x7f961c06d550 0x7f961c0d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:22.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.405+0000 7f961b7fe640 1 --2- 172.21.15.44:0/2505956233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f961c15e4a0 0x7f961c0aebf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f96100097a0 tx=0x7f961002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e38bc1684a6a520 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:22.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.405+0000 7f961a7fc640 1 -- 172.21.15.44:0/2505956233 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610035020 con 0x7f961c15e4a0 2024-04-02T13:21:22.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.406+0000 7f961a7fc640 1 -- 172.21.15.44:0/2505956233 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9610031830 con 0x7f961c15e4a0 2024-04-02T13:21:22.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.406+0000 7f961a7fc640 1 -- 172.21.15.44:0/2505956233 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610031c00 con 0x7f961c15e4a0 2024-04-02T13:21:22.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.406+0000 7f9622152640 1 -- 172.21.15.44:0/2505956233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f961c15e4a0 msgr2=0x7f961c0aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.406+0000 7f9622152640 1 --2- 172.21.15.44:0/2505956233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f961c15e4a0 0x7f961c0aebf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f96100097a0 tx=0x7f961002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.406+0000 7f9622152640 1 -- 172.21.15.44:0/2505956233 shutdown_connections 2024-04-02T13:21:22.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.406+0000 7f9622152640 1 --2- 172.21.15.44:0/2505956233 >> v2:172.21.15.69:3300/0 conn(0x7f961c06d550 0x7f961c0d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.406+0000 7f9622152640 1 --2- 172.21.15.44:0/2505956233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f961c15e4a0 0x7f961c0aebf0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.406+0000 7f9622152640 1 -- 172.21.15.44:0/2505956233 >> 172.21.15.44:0/2505956233 conn(0x7f961c09aa20 msgr2=0x7f961c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:22.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.406+0000 7f9622152640 1 -- 172.21.15.44:0/2505956233 shutdown_connections 2024-04-02T13:21:22.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.406+0000 7f9622152640 1 -- 172.21.15.44:0/2505956233 wait complete. 2024-04-02T13:21:22.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.407+0000 7f9622152640 1 Processor -- start 2024-04-02T13:21:22.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.407+0000 7f9622152640 1 -- start start 2024-04-02T13:21:22.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.408+0000 7f9622152640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f961c06d550 0x7f961c15f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.408+0000 7f9622152640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f961c0d0c50 0x7f961c1f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.408+0000 7f9622152640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f961c15e4a0 0x7f961c161470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.408+0000 7f9622152640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f961c15e880 con 0x7f961c06d550 2024-04-02T13:21:22.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.408+0000 7f9622152640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f961c0cd460 con 0x7f961c0d0c50 2024-04-02T13:21:22.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.408+0000 7f9622152640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f961c0af130 con 0x7f961c15e4a0 2024-04-02T13:21:22.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.408+0000 7f961b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f961c06d550 0x7f961c15f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.408+0000 7f961b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f961c06d550 0x7f961c15f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46030/0 (socket says 172.21.15.44:46030) 2024-04-02T13:21:22.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.408+0000 7f961b7fe640 1 -- 172.21.15.44:0/2859172869 learned_addr learned my addr 172.21.15.44:0/2859172869 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:22.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.408+0000 7f961affd640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f961c0d0c50 0x7f961c1f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.408+0000 7f961bfff640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f961c15e4a0 0x7f961c161470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.409+0000 7f961b7fe640 1 -- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f961c15e4a0 msgr2=0x7f961c161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.409+0000 7f961b7fe640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f961c15e4a0 0x7f961c161470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.409+0000 7f961b7fe640 1 -- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f961c0d0c50 msgr2=0x7f961c1f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.409+0000 7f961b7fe640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f961c0d0c50 0x7f961c1f3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.409+0000 7f961b7fe640 1 -- 172.21.15.44:0/2859172869 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f961c15ed30 con 0x7f961c06d550 2024-04-02T13:21:22.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.409+0000 7f961b7fe640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f961c06d550 0x7f961c15f2a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9610034040 tx=0x7f96100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:22.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.409+0000 7f9603fff640 1 -- 172.21.15.44:0/2859172869 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610040070 con 0x7f961c06d550 2024-04-02T13:21:22.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.409+0000 7f9603fff640 1 -- 172.21.15.44:0/2859172869 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f961003a850 con 0x7f961c06d550 2024-04-02T13:21:22.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.409+0000 7f9603fff640 1 -- 172.21.15.44:0/2859172869 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610035040 con 0x7f961c06d550 2024-04-02T13:21:22.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.409+0000 7f9622152640 1 -- 172.21.15.44:0/2859172869 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f961c1f4d10 con 0x7f961c06d550 2024-04-02T13:21:22.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.409+0000 7f9622152640 1 -- 172.21.15.44:0/2859172869 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f961c1f4750 con 0x7f961c06d550 2024-04-02T13:21:22.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.410+0000 7f9622152640 1 -- 172.21.15.44:0/2859172869 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f961c0684f0 con 0x7f961c06d550 2024-04-02T13:21:22.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.410+0000 7f9603fff640 1 -- 172.21.15.44:0/2859172869 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9610038020 con 0x7f961c06d550 2024-04-02T13:21:22.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.411+0000 7f9603fff640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f960806c8f0 0x7f960806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.411+0000 7f9603fff640 1 -- 172.21.15.44:0/2859172869 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f96100ad5f0 con 0x7f961c06d550 2024-04-02T13:21:22.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.411+0000 7f961affd640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f960806c8f0 0x7f960806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.412+0000 7f961affd640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f960806c8f0 0x7f960806edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f961c0b0250 tx=0x7f960c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:22.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.415+0000 7f9603fff640 1 -- 172.21.15.44:0/2859172869 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f961c15ed30 con 0x7f961c06d550 2024-04-02T13:21:22.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.610+0000 7f9622152640 1 -- 172.21.15.44:0/2859172869 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f961c0ce890 con 0x7f961c06d550 2024-04-02T13:21:22.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.610+0000 7f9603fff640 1 -- 172.21.15.44:0/2859172869 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f961007b610 con 0x7f961c06d550 2024-04-02T13:21:22.612 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:21:22.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.612+0000 7f9622152640 1 -- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f960806c8f0 msgr2=0x7f960806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.612+0000 7f9622152640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f960806c8f0 0x7f960806edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f961c0b0250 tx=0x7f960c009040 comp rx=0 tx=0).stop 2024-04-02T13:21:22.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.612+0000 7f9622152640 1 -- 172.21.15.44:0/2859172869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f961c06d550 msgr2=0x7f961c15f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.612+0000 7f9622152640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f961c06d550 0x7f961c15f2a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9610034040 tx=0x7f96100046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.612+0000 7f9622152640 1 -- 172.21.15.44:0/2859172869 shutdown_connections 2024-04-02T13:21:22.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.612+0000 7f9622152640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f960806c8f0 0x7f960806edb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.612+0000 7f9622152640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f961c15e4a0 0x7f961c161470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.612+0000 7f9622152640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f961c0d0c50 0x7f961c1f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.612+0000 7f9622152640 1 --2- 172.21.15.44:0/2859172869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f961c06d550 0x7f961c15f2a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.612+0000 7f9622152640 1 -- 172.21.15.44:0/2859172869 >> 172.21.15.44:0/2859172869 conn(0x7f961c09aa20 msgr2=0x7f961c0b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:22.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.612+0000 7f9622152640 1 -- 172.21.15.44:0/2859172869 shutdown_connections 2024-04-02T13:21:22.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.612+0000 7f9622152640 1 -- 172.21.15.44:0/2859172869 wait complete. 2024-04-02T13:21:22.624 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:21:22.624 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:22.639 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:22.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.779+0000 7fe78808f640 1 Processor -- start 2024-04-02T13:21:22.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.779+0000 7fe78808f640 1 -- start start 2024-04-02T13:21:22.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.780+0000 7fe78808f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe78015e430 0x7fe7800ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.780+0000 7fe78808f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe7800b1ef0 0x7fe7800b2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.780+0000 7fe78808f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe78015e810 con 0x7fe78006c7f0 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.780+0000 7fe78808f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe7800d1410 con 0x7fe78015e430 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.780+0000 7fe78808f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe78006cff0 con 0x7fe7800b1ef0 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.780+0000 7fe785e2b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe78015e430 0x7fe7800ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.780+0000 7fe785e2b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe78015e430 0x7fe7800ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46032/0 (socket says 172.21.15.44:46032) 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.780+0000 7fe785e2b640 1 -- 172.21.15.44:0/2728950005 learned_addr learned my addr 172.21.15.44:0/2728950005 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.780+0000 7fe78662c640 1 --2- 172.21.15.44:0/2728950005 >> v2:172.21.15.69:3300/0 conn(0x7fe7800b1ef0 0x7fe7800b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.781+0000 7fe785e2b640 1 -- 172.21.15.44:0/2728950005 >> v2:172.21.15.69:3300/0 conn(0x7fe7800b1ef0 msgr2=0x7fe7800b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.781+0000 7fe785e2b640 1 --2- 172.21.15.44:0/2728950005 >> v2:172.21.15.69:3300/0 conn(0x7fe7800b1ef0 0x7fe7800b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.781+0000 7fe785e2b640 1 -- 172.21.15.44:0/2728950005 >> v1:172.21.15.69:6789/0 conn(0x7fe78006c7f0 legacy=0x7fe7800ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.781+0000 7fe785e2b640 1 -- 172.21.15.44:0/2728950005 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78006d240 con 0x7fe78015e430 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.781+0000 7fe785e2b640 1 --2- 172.21.15.44:0/2728950005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe78015e430 0x7fe7800ce7c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe7800cc520 tx=0x7fe77002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467665797427d07f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.782+0000 7fe784e29640 1 -- 172.21.15.44:0/2728950005 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77002ca80 con 0x7fe78015e430 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.782+0000 7fe784e29640 1 -- 172.21.15.44:0/2728950005 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe77002cc20 con 0x7fe78015e430 2024-04-02T13:21:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.782+0000 7fe784e29640 1 -- 172.21.15.44:0/2728950005 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7700343f0 con 0x7fe78015e430 2024-04-02T13:21:22.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.782+0000 7fe78808f640 1 -- 172.21.15.44:0/2728950005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe78015e430 msgr2=0x7fe7800ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.782+0000 7fe78808f640 1 --2- 172.21.15.44:0/2728950005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe78015e430 0x7fe7800ce7c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe7800cc520 tx=0x7fe77002bee0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.782+0000 7fe78808f640 1 -- 172.21.15.44:0/2728950005 shutdown_connections 2024-04-02T13:21:22.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.782+0000 7fe78808f640 1 --2- 172.21.15.44:0/2728950005 >> v2:172.21.15.69:3300/0 conn(0x7fe7800b1ef0 0x7fe7800b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.782+0000 7fe78808f640 1 --2- 172.21.15.44:0/2728950005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe78015e430 0x7fe7800ce7c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.782+0000 7fe78808f640 1 -- 172.21.15.44:0/2728950005 >> 172.21.15.44:0/2728950005 conn(0x7fe780073850 msgr2=0x7fe78009aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:22.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.782+0000 7fe78808f640 1 -- 172.21.15.44:0/2728950005 shutdown_connections 2024-04-02T13:21:22.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.782+0000 7fe78808f640 1 -- 172.21.15.44:0/2728950005 wait complete. 2024-04-02T13:21:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.783+0000 7fe78808f640 1 Processor -- start 2024-04-02T13:21:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.783+0000 7fe78808f640 1 -- start start 2024-04-02T13:21:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78808f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78006c7f0 0x7fe7800ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78808f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7800b1ef0 0x7fe780153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78808f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe78015e430 0x7fe7800c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78808f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe7800d0ef0 con 0x7fe7800b1ef0 2024-04-02T13:21:22.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78808f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe7800d0c10 con 0x7fe78006c7f0 2024-04-02T13:21:22.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78808f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe78006d800 con 0x7fe78015e430 2024-04-02T13:21:22.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78562a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7800b1ef0 0x7fe780153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78562a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7800b1ef0 0x7fe780153250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46036/0 (socket says 172.21.15.44:46036) 2024-04-02T13:21:22.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78562a640 1 -- 172.21.15.44:0/3206032631 learned_addr learned my addr 172.21.15.44:0/3206032631 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:22.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78662c640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe78015e430 0x7fe7800c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe785e2b640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78006c7f0 0x7fe7800ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78562a640 1 -- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe78015e430 msgr2=0x7fe7800c72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78562a640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe78015e430 0x7fe7800c72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78562a640 1 -- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78006c7f0 msgr2=0x7fe7800ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78562a640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78006c7f0 0x7fe7800ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.784+0000 7fe78562a640 1 -- 172.21.15.44:0/3206032631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7800c9470 con 0x7fe7800b1ef0 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.785+0000 7fe78562a640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7800b1ef0 0x7fe780153250 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe77400c3c0 tx=0x7fe77400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.785+0000 7fe77e7fc640 1 -- 172.21.15.44:0/3206032631 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7740052d0 con 0x7fe7800b1ef0 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.785+0000 7fe77e7fc640 1 -- 172.21.15.44:0/3206032631 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe774005490 con 0x7fe7800b1ef0 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.785+0000 7fe78808f640 1 -- 172.21.15.44:0/3206032631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7800c9750 con 0x7fe7800b1ef0 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.785+0000 7fe77e7fc640 1 -- 172.21.15.44:0/3206032631 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe774007260 con 0x7fe7800b1ef0 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.785+0000 7fe78808f640 1 -- 172.21.15.44:0/3206032631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7800ca770 con 0x7fe7800b1ef0 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.786+0000 7fe78808f640 1 -- 172.21.15.44:0/3206032631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe780068c00 con 0x7fe7800b1ef0 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.786+0000 7fe77e7fc640 1 -- 172.21.15.44:0/3206032631 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe774007400 con 0x7fe7800b1ef0 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.787+0000 7fe77e7fc640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe76c06c8f0 0x7fe76c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.787+0000 7fe77e7fc640 1 -- 172.21.15.44:0/3206032631 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fe77408b2f0 con 0x7fe7800b1ef0 2024-04-02T13:21:22.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.787+0000 7fe785e2b640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe76c06c8f0 0x7fe76c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:22.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.787+0000 7fe785e2b640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe76c06c8f0 0x7fe76c06edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe770035000 tx=0x7fe770030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:22.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.791+0000 7fe77e7fc640 1 -- 172.21.15.44:0/3206032631 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe774058d40 con 0x7fe7800b1ef0 2024-04-02T13:21:22.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.978+0000 7fe78808f640 1 -- 172.21.15.44:0/3206032631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe78006f600 con 0x7fe7800b1ef0 2024-04-02T13:21:22.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.979+0000 7fe77e7fc640 1 -- 172.21.15.44:0/3206032631 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18899 (secure 0 0 0) 0x7fe77405d040 con 0x7fe7800b1ef0 2024-04-02T13:21:22.981 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:22.981 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:08.061253+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:22.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.981+0000 7fe78808f640 1 -- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe76c06c8f0 msgr2=0x7fe76c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.981+0000 7fe78808f640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe76c06c8f0 0x7fe76c06edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe770035000 tx=0x7fe770030000 comp rx=0 tx=0).stop 2024-04-02T13:21:22.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.981+0000 7fe78808f640 1 -- 172.21.15.44:0/3206032631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7800b1ef0 msgr2=0x7fe780153250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:22.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.981+0000 7fe78808f640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7800b1ef0 0x7fe780153250 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe77400c3c0 tx=0x7fe77400c880 comp rx=0 tx=0).stop 2024-04-02T13:21:22.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.981+0000 7fe78808f640 1 -- 172.21.15.44:0/3206032631 shutdown_connections 2024-04-02T13:21:22.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.981+0000 7fe78808f640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe76c06c8f0 0x7fe76c06edb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.981+0000 7fe78808f640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe78015e430 0x7fe7800c72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.981+0000 7fe78808f640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7800b1ef0 0x7fe780153250 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.981+0000 7fe78808f640 1 --2- 172.21.15.44:0/3206032631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78006c7f0 0x7fe7800ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:22.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.981+0000 7fe78808f640 1 -- 172.21.15.44:0/3206032631 >> 172.21.15.44:0/3206032631 conn(0x7fe780073850 msgr2=0x7fe7800aff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:22.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.981+0000 7fe78808f640 1 -- 172.21.15.44:0/3206032631 shutdown_connections 2024-04-02T13:21:22.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:22.981+0000 7fe78808f640 1 -- 172.21.15.44:0/3206032631 wait complete. 2024-04-02T13:21:22.994 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:22.994 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:23.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.098+0000 7f6599aba640 1 Processor -- start 2024-04-02T13:21:23.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.098+0000 7f6599aba640 1 -- start start 2024-04-02T13:21:23.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f6599aba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f659406daa0 0x7f65940b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f6599aba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f65940cfdc0 0x7f65940d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f6599aba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f659406de80 con 0x7f65940d26f0 2024-04-02T13:21:23.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f6599aba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f65940cf930 con 0x7f65940cfdc0 2024-04-02T13:21:23.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f6599aba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f65940cf000 con 0x7f659406daa0 2024-04-02T13:21:23.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f6593fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f65940d26f0 0x7f659415dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48952/0 (socket says 172.21.15.44:48952) 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f6593fff640 1 -- 172.21.15.44:0/2721367380 learned_addr learned my addr 172.21.15.44:0/2721367380 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f6592ffd640 1 --2- 172.21.15.44:0/2721367380 >> v2:172.21.15.69:3300/0 conn(0x7f65940cfdc0 0x7f65940d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f65937fe640 1 --2- 172.21.15.44:0/2721367380 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f659406daa0 0x7f65940b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f65927fc640 1 -- 172.21.15.44:0/2721367380 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 949988317 0 0) 0x7f659406de80 con 0x7f65940d26f0 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f65927fc640 1 -- 172.21.15.44:0/2721367380 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f657c003520 con 0x7f65940d26f0 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f65927fc640 1 -- 172.21.15.44:0/2721367380 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3851693740 0 0) 0x7f657c003520 con 0x7f65940d26f0 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f6592ffd640 1 -- 172.21.15.44:0/2721367380 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f659406daa0 msgr2=0x7f65940b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f6592ffd640 1 --2- 172.21.15.44:0/2721367380 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f659406daa0 0x7f65940b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f6592ffd640 1 -- 172.21.15.44:0/2721367380 >> v1:172.21.15.44:6789/0 conn(0x7f65940d26f0 legacy=0x7f659415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f6592ffd640 1 -- 172.21.15.44:0/2721367380 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65940cf2e0 con 0x7f65940cfdc0 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.099+0000 7f65937fe640 1 --2- 172.21.15.44:0/2721367380 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f659406daa0 0x7f65940b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.100+0000 7f6592ffd640 1 --2- 172.21.15.44:0/2721367380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65940cfdc0 0x7f65940d11c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f65880097a0 tx=0x7f658802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=268c6fc981cd1796 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.100+0000 7f65927fc640 1 -- 172.21.15.44:0/2721367380 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6588037070 con 0x7f65940cfdc0 2024-04-02T13:21:23.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.100+0000 7f65927fc640 1 -- 172.21.15.44:0/2721367380 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6588030510 con 0x7f65940cfdc0 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.100+0000 7f65927fc640 1 -- 172.21.15.44:0/2721367380 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c0019f0 con 0x7f65940cfdc0 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.101+0000 7f6599aba640 1 -- 172.21.15.44:0/2721367380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65940cfdc0 msgr2=0x7f65940d11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.101+0000 7f6599aba640 1 --2- 172.21.15.44:0/2721367380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65940cfdc0 0x7f65940d11c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f65880097a0 tx=0x7f658802b790 comp rx=0 tx=0).stop 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.101+0000 7f6599aba640 1 -- 172.21.15.44:0/2721367380 shutdown_connections 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.101+0000 7f6599aba640 1 --2- 172.21.15.44:0/2721367380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65940cfdc0 0x7f65940d11c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.101+0000 7f6599aba640 1 --2- 172.21.15.44:0/2721367380 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f659406daa0 0x7f65940b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.101+0000 7f6599aba640 1 -- 172.21.15.44:0/2721367380 >> 172.21.15.44:0/2721367380 conn(0x7f659409d430 msgr2=0x7f65940d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.101+0000 7f6599aba640 1 -- 172.21.15.44:0/2721367380 shutdown_connections 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.101+0000 7f6599aba640 1 -- 172.21.15.44:0/2721367380 wait complete. 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.102+0000 7f6599aba640 1 Processor -- start 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.102+0000 7f6599aba640 1 -- start start 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6599aba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f659406daa0 0x7f6594159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6599aba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65940cfdc0 0x7f65940b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6599aba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65940d26f0 0x7f65940b8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6599aba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f65940ceda0 con 0x7f65940d26f0 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6599aba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f65940cec20 con 0x7f659406daa0 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6599aba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f65940d2ba0 con 0x7f65940cfdc0 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6593fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65940d26f0 0x7f65940b8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6593fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65940d26f0 0x7f65940b8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46052/0 (socket says 172.21.15.44:46052) 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6593fff640 1 -- 172.21.15.44:0/1474711741 learned_addr learned my addr 172.21.15.44:0/1474711741 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f65937fe640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f659406daa0 0x7f6594159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6592ffd640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65940cfdc0 0x7f65940b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6593fff640 1 -- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65940cfdc0 msgr2=0x7f65940b7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6593fff640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65940cfdc0 0x7f65940b7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6593fff640 1 -- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f659406daa0 msgr2=0x7f6594159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6593fff640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f659406daa0 0x7f6594159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6593fff640 1 -- 172.21.15.44:0/1474711741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65940cbf60 con 0x7f65940d26f0 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.103+0000 7f6593fff640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65940d26f0 0x7f65940b8360 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f658c00ccc0 tx=0x7f658c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:23.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.104+0000 7f657bfff640 1 -- 172.21.15.44:0/1474711741 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c012400 con 0x7f65940d26f0 2024-04-02T13:21:23.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.104+0000 7f657bfff640 1 -- 172.21.15.44:0/1474711741 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f658c0125c0 con 0x7f65940d26f0 2024-04-02T13:21:23.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.104+0000 7f6599aba640 1 -- 172.21.15.44:0/1474711741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65940cc240 con 0x7f65940d26f0 2024-04-02T13:21:23.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.104+0000 7f657bfff640 1 -- 172.21.15.44:0/1474711741 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c0129b0 con 0x7f65940d26f0 2024-04-02T13:21:23.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.104+0000 7f6599aba640 1 -- 172.21.15.44:0/1474711741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65940cc800 con 0x7f65940d26f0 2024-04-02T13:21:23.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.105+0000 7f6599aba640 1 -- 172.21.15.44:0/1474711741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65940684f0 con 0x7f65940d26f0 2024-04-02T13:21:23.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.105+0000 7f657bfff640 1 -- 172.21.15.44:0/1474711741 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f658c012b50 con 0x7f65940d26f0 2024-04-02T13:21:23.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.105+0000 7f657bfff640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f658006c8f0 0x7f658006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.106+0000 7f657bfff640 1 -- 172.21.15.44:0/1474711741 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f658c08c7c0 con 0x7f65940d26f0 2024-04-02T13:21:23.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.106+0000 7f65937fe640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f658006c8f0 0x7f658006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.106+0000 7f65937fe640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f658006c8f0 0x7f658006edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6584009560 tx=0x7f6584008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:23.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.110+0000 7f657bfff640 1 -- 172.21.15.44:0/1474711741 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f658c0182b0 con 0x7f65940d26f0 2024-04-02T13:21:23.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.293+0000 7f6599aba640 1 -- 172.21.15.44:0/1474711741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f65940d24d0 con 0x7f65940d26f0 2024-04-02T13:21:23.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.293+0000 7f657bfff640 1 -- 172.21.15.44:0/1474711741 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f658c05a7d0 con 0x7f65940d26f0 2024-04-02T13:21:23.295 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:23.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.295+0000 7f6599aba640 1 -- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f658006c8f0 msgr2=0x7f658006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.295+0000 7f6599aba640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f658006c8f0 0x7f658006edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6584009560 tx=0x7f6584008040 comp rx=0 tx=0).stop 2024-04-02T13:21:23.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.295+0000 7f6599aba640 1 -- 172.21.15.44:0/1474711741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65940d26f0 msgr2=0x7f65940b8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.295+0000 7f6599aba640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65940d26f0 0x7f65940b8360 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f658c00ccc0 tx=0x7f658c00a430 comp rx=0 tx=0).stop 2024-04-02T13:21:23.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.295+0000 7f6599aba640 1 -- 172.21.15.44:0/1474711741 shutdown_connections 2024-04-02T13:21:23.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.295+0000 7f6599aba640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f658006c8f0 0x7f658006edb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.295+0000 7f6599aba640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65940d26f0 0x7f65940b8360 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.295+0000 7f6599aba640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65940cfdc0 0x7f65940b7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.295+0000 7f6599aba640 1 --2- 172.21.15.44:0/1474711741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f659406daa0 0x7f6594159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.295+0000 7f6599aba640 1 -- 172.21.15.44:0/1474711741 >> 172.21.15.44:0/1474711741 conn(0x7f659409d430 msgr2=0x7f65941601d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:23.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.295+0000 7f6599aba640 1 -- 172.21.15.44:0/1474711741 shutdown_connections 2024-04-02T13:21:23.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.295+0000 7f6599aba640 1 -- 172.21.15.44:0/1474711741 wait complete. 2024-04-02T13:21:23.307 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:23.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.411+0000 7fd7814b2640 1 Processor -- start 2024-04-02T13:21:23.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.411+0000 7fd7814b2640 1 -- start start 2024-04-02T13:21:23.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd7814b2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd77c0d0a30 0x7fd77c0b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd7814b2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd77c0b3990 0x7fd77c143a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd7814b2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd77c0d0e10 con 0x7fd77c143f60 2024-04-02T13:21:23.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd7814b2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd77c0b3df0 con 0x7fd77c0d0a30 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd7814b2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd77c161e10 con 0x7fd77c0b3990 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd77affd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd77c0d0a30 0x7fd77c0b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd77affd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd77c0d0a30 0x7fd77c0b3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46056/0 (socket says 172.21.15.44:46056) 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd77affd640 1 -- 172.21.15.44:0/3840114325 learned_addr learned my addr 172.21.15.44:0/3840114325 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd77a7fc640 1 --2- 172.21.15.44:0/3840114325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd77c0b3990 0x7fd77c143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd77affd640 1 -- 172.21.15.44:0/3840114325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd77c0b3990 msgr2=0x7fd77c143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd77affd640 1 --2- 172.21.15.44:0/3840114325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd77c0b3990 0x7fd77c143a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd77affd640 1 -- 172.21.15.44:0/3840114325 >> v1:172.21.15.69:6789/0 conn(0x7fd77c143f60 legacy=0x7fd77c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd77affd640 1 -- 172.21.15.44:0/3840114325 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77c144750 con 0x7fd77c0d0a30 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.412+0000 7fd77a7fc640 1 --2- 172.21.15.44:0/3840114325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd77c0b3990 0x7fd77c143a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.413+0000 7fd77affd640 1 --2- 172.21.15.44:0/3840114325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd77c0d0a30 0x7fd77c0b3450 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009870 tx=0x7fd76c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b71cf6e3c0b8d2ed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.413+0000 7fd779ffb640 1 -- 172.21.15.44:0/3840114325 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c035020 con 0x7fd77c0d0a30 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.413+0000 7fd779ffb640 1 -- 172.21.15.44:0/3840114325 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd76c031640 con 0x7fd77c0d0a30 2024-04-02T13:21:23.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.413+0000 7fd779ffb640 1 -- 172.21.15.44:0/3840114325 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c02d040 con 0x7fd77c0d0a30 2024-04-02T13:21:23.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.413+0000 7fd7814b2640 1 -- 172.21.15.44:0/3840114325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd77c0d0a30 msgr2=0x7fd77c0b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.413+0000 7fd7814b2640 1 --2- 172.21.15.44:0/3840114325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd77c0d0a30 0x7fd77c0b3450 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009870 tx=0x7fd76c02b8f0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.414+0000 7fd7814b2640 1 -- 172.21.15.44:0/3840114325 shutdown_connections 2024-04-02T13:21:23.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.414+0000 7fd7814b2640 1 --2- 172.21.15.44:0/3840114325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd77c0b3990 0x7fd77c143a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.414+0000 7fd7814b2640 1 --2- 172.21.15.44:0/3840114325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd77c0d0a30 0x7fd77c0b3450 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.414+0000 7fd7814b2640 1 -- 172.21.15.44:0/3840114325 >> 172.21.15.44:0/3840114325 conn(0x7fd77c0b5d90 msgr2=0x7fd77c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:23.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.414+0000 7fd7814b2640 1 -- 172.21.15.44:0/3840114325 shutdown_connections 2024-04-02T13:21:23.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.414+0000 7fd7814b2640 1 -- 172.21.15.44:0/3840114325 wait complete. 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.415+0000 7fd7814b2640 1 Processor -- start 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.415+0000 7fd7814b2640 1 -- start start 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd7814b2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd77c0b3990 0x7fd77c0aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd7814b2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd77c0d0a30 0x7fd77c140aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd7814b2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd77c143f60 0x7fd77c145650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd7814b2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c162620 con 0x7fd77c0b3990 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd7814b2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c162340 con 0x7fd77c143f60 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd7814b2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd77c144390 con 0x7fd77c0d0a30 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd77affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd77c0b3990 0x7fd77c0aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd77affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd77c0b3990 0x7fd77c0aeff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46066/0 (socket says 172.21.15.44:46066) 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd77affd640 1 -- 172.21.15.44:0/1016420575 learned_addr learned my addr 172.21.15.44:0/1016420575 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd77b7fe640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd77c143f60 0x7fd77c145650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd77a7fc640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd77c0d0a30 0x7fd77c140aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd77affd640 1 -- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd77c0d0a30 msgr2=0x7fd77c140aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd77affd640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd77c0d0a30 0x7fd77c140aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd77affd640 1 -- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd77c143f60 msgr2=0x7fd77c145650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd77affd640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd77c143f60 0x7fd77c145650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd77affd640 1 -- 172.21.15.44:0/1016420575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77c15c420 con 0x7fd77c0b3990 2024-04-02T13:21:23.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd77a7fc640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd77c0d0a30 0x7fd77c140aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:23.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd77affd640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd77c0b3990 0x7fd77c0aeff0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd76c02b930 tx=0x7fd76c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:23.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd7637fe640 1 -- 172.21.15.44:0/1016420575 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c040070 con 0x7fd77c0b3990 2024-04-02T13:21:23.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd7637fe640 1 -- 172.21.15.44:0/1016420575 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd76c03e070 con 0x7fd77c0b3990 2024-04-02T13:21:23.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd7637fe640 1 -- 172.21.15.44:0/1016420575 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c035040 con 0x7fd77c0b3990 2024-04-02T13:21:23.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd7814b2640 1 -- 172.21.15.44:0/1016420575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd77c0af600 con 0x7fd77c0b3990 2024-04-02T13:21:23.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.416+0000 7fd7814b2640 1 -- 172.21.15.44:0/1016420575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77c15c6e0 con 0x7fd77c0b3990 2024-04-02T13:21:23.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.418+0000 7fd7814b2640 1 -- 172.21.15.44:0/1016420575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd77c0684f0 con 0x7fd77c0b3990 2024-04-02T13:21:23.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.418+0000 7fd7637fe640 1 -- 172.21.15.44:0/1016420575 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd76c038020 con 0x7fd77c0b3990 2024-04-02T13:21:23.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.418+0000 7fd7637fe640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd76806c8f0 0x7fd76806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.418+0000 7fd7637fe640 1 -- 172.21.15.44:0/1016420575 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fd76c0ad8e0 con 0x7fd77c0b3990 2024-04-02T13:21:23.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.418+0000 7fd77a7fc640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd76806c8f0 0x7fd76806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.419+0000 7fd77a7fc640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd76806c8f0 0x7fd76806edb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0af530 tx=0x7fd770008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:23.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.423+0000 7fd7637fe640 1 -- 172.21.15.44:0/1016420575 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd76c07b900 con 0x7fd77c0b3990 2024-04-02T13:21:23.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.607+0000 7fd7814b2640 1 -- 172.21.15.44:0/1016420575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd77c06bcb0 con 0x7fd77c0b3990 2024-04-02T13:21:23.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.608+0000 7fd7637fe640 1 -- 172.21.15.44:0/1016420575 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fd76c07b200 con 0x7fd77c0b3990 2024-04-02T13:21:23.610 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:23.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.610+0000 7fd7814b2640 1 -- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd76806c8f0 msgr2=0x7fd76806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.610+0000 7fd7814b2640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd76806c8f0 0x7fd76806edb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0af530 tx=0x7fd770008040 comp rx=0 tx=0).stop 2024-04-02T13:21:23.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.610+0000 7fd7814b2640 1 -- 172.21.15.44:0/1016420575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd77c0b3990 msgr2=0x7fd77c0aeff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.610+0000 7fd7814b2640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd77c0b3990 0x7fd77c0aeff0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd76c02b930 tx=0x7fd76c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.610+0000 7fd7814b2640 1 -- 172.21.15.44:0/1016420575 shutdown_connections 2024-04-02T13:21:23.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.610+0000 7fd7814b2640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd76806c8f0 0x7fd76806edb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.610+0000 7fd7814b2640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd77c143f60 0x7fd77c145650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.610+0000 7fd7814b2640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd77c0d0a30 0x7fd77c140aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.610+0000 7fd7814b2640 1 --2- 172.21.15.44:0/1016420575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd77c0b3990 0x7fd77c0aeff0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.610+0000 7fd7814b2640 1 -- 172.21.15.44:0/1016420575 >> 172.21.15.44:0/1016420575 conn(0x7fd77c0b5d90 msgr2=0x7fd77c15f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:23.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.610+0000 7fd7814b2640 1 -- 172.21.15.44:0/1016420575 shutdown_connections 2024-04-02T13:21:23.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.610+0000 7fd7814b2640 1 -- 172.21.15.44:0/1016420575 wait complete. 2024-04-02T13:21:23.622 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.725+0000 7f6265b82640 1 Processor -- start 2024-04-02T13:21:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.725+0000 7f6265b82640 1 -- start start 2024-04-02T13:21:23.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.726+0000 7f6265b82640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f626015a940 0x7f62600ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.726+0000 7f6265b82640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62600cc640 0x7f62600cca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.726+0000 7f6265b82640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62600ccfd0 con 0x7f6260071200 2024-04-02T13:21:23.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.726+0000 7f6265b82640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f626006ef80 con 0x7f626015a940 2024-04-02T13:21:23.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.726+0000 7f6265b82640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f62600ce020 con 0x7f62600cc640 2024-04-02T13:21:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.726+0000 7f625effd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6260071200 0x7f626006e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48954/0 (socket says 172.21.15.44:48954) 2024-04-02T13:21:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.726+0000 7f625effd640 1 -- 172.21.15.44:0/2319401329 learned_addr learned my addr 172.21.15.44:0/2319401329 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.726+0000 7f625f7fe640 1 --2- 172.21.15.44:0/2319401329 >> v2:172.21.15.69:3300/0 conn(0x7f626015a940 0x7f62600ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.726+0000 7f625ffff640 1 --2- 172.21.15.44:0/2319401329 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62600cc640 0x7f62600cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.727+0000 7f625e7fc640 1 -- 172.21.15.44:0/2319401329 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3696955594 0 0) 0x7f62600ccfd0 con 0x7f6260071200 2024-04-02T13:21:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.727+0000 7f625e7fc640 1 -- 172.21.15.44:0/2319401329 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6248003520 con 0x7f6260071200 2024-04-02T13:21:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.727+0000 7f625ffff640 1 -- 172.21.15.44:0/2319401329 >> v2:172.21.15.69:3300/0 conn(0x7f626015a940 msgr2=0x7f62600ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.727+0000 7f625ffff640 1 --2- 172.21.15.44:0/2319401329 >> v2:172.21.15.69:3300/0 conn(0x7f626015a940 0x7f62600ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.727+0000 7f625ffff640 1 -- 172.21.15.44:0/2319401329 >> v1:172.21.15.44:6789/0 conn(0x7f6260071200 legacy=0x7f626006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.727+0000 7f625ffff640 1 -- 172.21.15.44:0/2319401329 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62600ce9f0 con 0x7f62600cc640 2024-04-02T13:21:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.727+0000 7f625f7fe640 1 --2- 172.21.15.44:0/2319401329 >> v2:172.21.15.69:3300/0 conn(0x7f626015a940 0x7f62600ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.728+0000 7f625ffff640 1 --2- 172.21.15.44:0/2319401329 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62600cc640 0x7f62600cca90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6258009d90 tx=0x7f625802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66cf0e71ac42004 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.728+0000 7f625e7fc640 1 -- 172.21.15.44:0/2319401329 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258039070 con 0x7f62600cc640 2024-04-02T13:21:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.728+0000 7f625e7fc640 1 -- 172.21.15.44:0/2319401329 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f625802cbc0 con 0x7f62600cc640 2024-04-02T13:21:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.728+0000 7f625e7fc640 1 -- 172.21.15.44:0/2319401329 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258034440 con 0x7f62600cc640 2024-04-02T13:21:23.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.728+0000 7f6265b82640 1 -- 172.21.15.44:0/2319401329 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62600cc640 msgr2=0x7f62600cca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.728+0000 7f6265b82640 1 --2- 172.21.15.44:0/2319401329 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62600cc640 0x7f62600cca90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6258009d90 tx=0x7f625802be10 comp rx=0 tx=0).stop 2024-04-02T13:21:23.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.728+0000 7f6265b82640 1 -- 172.21.15.44:0/2319401329 shutdown_connections 2024-04-02T13:21:23.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.728+0000 7f6265b82640 1 --2- 172.21.15.44:0/2319401329 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62600cc640 0x7f62600cca90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.728+0000 7f6265b82640 1 --2- 172.21.15.44:0/2319401329 >> v2:172.21.15.69:3300/0 conn(0x7f626015a940 0x7f62600ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.728+0000 7f6265b82640 1 -- 172.21.15.44:0/2319401329 >> 172.21.15.44:0/2319401329 conn(0x7f626006b920 msgr2=0x7f626006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:23.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.728+0000 7f6265b82640 1 -- 172.21.15.44:0/2319401329 shutdown_connections 2024-04-02T13:21:23.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.728+0000 7f6265b82640 1 -- 172.21.15.44:0/2319401329 wait complete. 2024-04-02T13:21:23.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.729+0000 7f6265b82640 1 Processor -- start 2024-04-02T13:21:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.730+0000 7f6265b82640 1 -- start start 2024-04-02T13:21:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.730+0000 7f6265b82640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6260071200 0x7f626009aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.730+0000 7f6265b82640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62600cc640 0x7f62600b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.730+0000 7f6265b82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f626015a940 0x7f62600b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.730+0000 7f6265b82640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f62600cdcb0 con 0x7f626015a940 2024-04-02T13:21:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.730+0000 7f6265b82640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f626006eca0 con 0x7f6260071200 2024-04-02T13:21:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.730+0000 7f6265b82640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f62600d00a0 con 0x7f62600cc640 2024-04-02T13:21:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f625ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f626015a940 0x7f62600b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f625ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f626015a940 0x7f62600b1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46080/0 (socket says 172.21.15.44:46080) 2024-04-02T13:21:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f625ffff640 1 -- 172.21.15.44:0/2500859461 learned_addr learned my addr 172.21.15.44:0/2500859461 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:23.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f625effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62600cc640 0x7f62600b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f625f7fe640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6260071200 0x7f626009aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f625ffff640 1 -- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62600cc640 msgr2=0x7f62600b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f625ffff640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62600cc640 0x7f62600b0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f625ffff640 1 -- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6260071200 msgr2=0x7f626009aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f625ffff640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6260071200 0x7f626009aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f625ffff640 1 -- 172.21.15.44:0/2500859461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62601f50a0 con 0x7f626015a940 2024-04-02T13:21:23.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f625effd640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62600cc640 0x7f62600b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:23.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f625ffff640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f626015a940 0x7f62600b1150 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6258009d90 tx=0x7f6258004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:23.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f6247fff640 1 -- 172.21.15.44:0/2500859461 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258039050 con 0x7f626015a940 2024-04-02T13:21:23.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f6247fff640 1 -- 172.21.15.44:0/2500859461 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6258045070 con 0x7f626015a940 2024-04-02T13:21:23.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f6265b82640 1 -- 172.21.15.44:0/2500859461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62601f5380 con 0x7f626015a940 2024-04-02T13:21:23.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f6247fff640 1 -- 172.21.15.44:0/2500859461 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258009450 con 0x7f626015a940 2024-04-02T13:21:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.731+0000 7f6265b82640 1 -- 172.21.15.44:0/2500859461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62601f5940 con 0x7f626015a940 2024-04-02T13:21:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.732+0000 7f6265b82640 1 -- 172.21.15.44:0/2500859461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62600684f0 con 0x7f626015a940 2024-04-02T13:21:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.733+0000 7f6247fff640 1 -- 172.21.15.44:0/2500859461 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f625803e830 con 0x7f626015a940 2024-04-02T13:21:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.733+0000 7f6247fff640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f624c06c8f0 0x7f624c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.733+0000 7f6247fff640 1 -- 172.21.15.44:0/2500859461 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f62580aead0 con 0x7f626015a940 2024-04-02T13:21:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.733+0000 7f625f7fe640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f624c06c8f0 0x7f624c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.734+0000 7f625f7fe640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f624c06c8f0 0x7f624c06edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f626009d3a0 tx=0x7f6250009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:23.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.737+0000 7f6247fff640 1 -- 172.21.15.44:0/2500859461 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f625807c5f0 con 0x7f626015a940 2024-04-02T13:21:23.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.922+0000 7f6265b82640 1 -- 172.21.15.44:0/2500859461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f62600c8ff0 con 0x7f626015a940 2024-04-02T13:21:23.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.922+0000 7f6247fff640 1 -- 172.21.15.44:0/2500859461 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f62580808f0 con 0x7f626015a940 2024-04-02T13:21:23.924 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:23.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.924+0000 7f6265b82640 1 -- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f624c06c8f0 msgr2=0x7f624c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.924+0000 7f6265b82640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f624c06c8f0 0x7f624c06edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f626009d3a0 tx=0x7f6250009190 comp rx=0 tx=0).stop 2024-04-02T13:21:23.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.924+0000 7f6265b82640 1 -- 172.21.15.44:0/2500859461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f626015a940 msgr2=0x7f62600b1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:23.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.924+0000 7f6265b82640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f626015a940 0x7f62600b1150 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6258009d90 tx=0x7f6258004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:23.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.924+0000 7f6265b82640 1 -- 172.21.15.44:0/2500859461 shutdown_connections 2024-04-02T13:21:23.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.924+0000 7f6265b82640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f624c06c8f0 0x7f624c06edb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.924+0000 7f6265b82640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f626015a940 0x7f62600b1150 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.924+0000 7f6265b82640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62600cc640 0x7f62600b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.924+0000 7f6265b82640 1 --2- 172.21.15.44:0/2500859461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6260071200 0x7f626009aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:23.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.924+0000 7f6265b82640 1 -- 172.21.15.44:0/2500859461 >> 172.21.15.44:0/2500859461 conn(0x7f626006b920 msgr2=0x7f62600af560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:23.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.924+0000 7f6265b82640 1 -- 172.21.15.44:0/2500859461 shutdown_connections 2024-04-02T13:21:23.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:23.924+0000 7f6265b82640 1 -- 172.21.15.44:0/2500859461 wait complete. 2024-04-02T13:21:23.936 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:24.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.040+0000 7f6c999d6640 1 Processor -- start 2024-04-02T13:21:24.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.040+0000 7f6c999d6640 1 -- start start 2024-04-02T13:21:24.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.041+0000 7f6c999d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c9415c1a0 0x7f6c9406f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.041+0000 7f6c999d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c9415c580 con 0x7f6c940d21c0 2024-04-02T13:21:24.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.041+0000 7f6c999d6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c940d01c0 con 0x7f6c940cfd60 2024-04-02T13:21:24.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.041+0000 7f6c999d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6c94070ad0 con 0x7f6c9415c1a0 2024-04-02T13:21:24.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.041+0000 7f6c937fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6c940d21c0 0x7f6c940cc9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48956/0 (socket says 172.21.15.44:48956) 2024-04-02T13:21:24.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.041+0000 7f6c937fe640 1 -- 172.21.15.44:0/4226894354 learned_addr learned my addr 172.21.15.44:0/4226894354 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:24.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.041+0000 7f6c92ffd640 1 --2- 172.21.15.44:0/4226894354 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c9415c1a0 0x7f6c9406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.042+0000 7f6c91ffb640 1 -- 172.21.15.44:0/4226894354 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3234062127 0 0) 0x7f6c9415c580 con 0x7f6c940d21c0 2024-04-02T13:21:24.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.042+0000 7f6c91ffb640 1 -- 172.21.15.44:0/4226894354 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c80003520 con 0x7f6c940d21c0 2024-04-02T13:21:24.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.042+0000 7f6c91ffb640 1 -- 172.21.15.44:0/4226894354 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 604781795 0 0) 0x7f6c940d01c0 con 0x7f6c940cfd60 2024-04-02T13:21:24.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.042+0000 7f6c91ffb640 1 -- 172.21.15.44:0/4226894354 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c9415c580 con 0x7f6c940cfd60 2024-04-02T13:21:24.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.042+0000 7f6c92ffd640 1 -- 172.21.15.44:0/4226894354 >> v1:172.21.15.69:6789/0 conn(0x7f6c940cfd60 legacy=0x7f6c940d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.042+0000 7f6c92ffd640 1 -- 172.21.15.44:0/4226894354 >> v1:172.21.15.44:6789/0 conn(0x7f6c940d21c0 legacy=0x7f6c940cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.042+0000 7f6c92ffd640 1 -- 172.21.15.44:0/4226894354 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9406fa30 con 0x7f6c9415c1a0 2024-04-02T13:21:24.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.042+0000 7f6c92ffd640 1 --2- 172.21.15.44:0/4226894354 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c9415c1a0 0x7f6c9406f370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0097d0 tx=0x7f6c8c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=370706bc807b23bb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:24.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.043+0000 7f6c91ffb640 1 -- 172.21.15.44:0/4226894354 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c035020 con 0x7f6c9415c1a0 2024-04-02T13:21:24.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.043+0000 7f6c91ffb640 1 -- 172.21.15.44:0/4226894354 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c8c031890 con 0x7f6c9415c1a0 2024-04-02T13:21:24.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.043+0000 7f6c91ffb640 1 -- 172.21.15.44:0/4226894354 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c031c40 con 0x7f6c9415c1a0 2024-04-02T13:21:24.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.043+0000 7f6c999d6640 1 -- 172.21.15.44:0/4226894354 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c9415c1a0 msgr2=0x7f6c9406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.043+0000 7f6c999d6640 1 --2- 172.21.15.44:0/4226894354 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c9415c1a0 0x7f6c9406f370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0097d0 tx=0x7f6c8c02b930 comp rx=0 tx=0).stop 2024-04-02T13:21:24.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.043+0000 7f6c999d6640 1 -- 172.21.15.44:0/4226894354 shutdown_connections 2024-04-02T13:21:24.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.043+0000 7f6c999d6640 1 --2- 172.21.15.44:0/4226894354 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c9415c1a0 0x7f6c9406f370 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.043+0000 7f6c999d6640 1 -- 172.21.15.44:0/4226894354 >> 172.21.15.44:0/4226894354 conn(0x7f6c9409aa20 msgr2=0x7f6c9409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:24.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.043+0000 7f6c999d6640 1 -- 172.21.15.44:0/4226894354 shutdown_connections 2024-04-02T13:21:24.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.043+0000 7f6c999d6640 1 -- 172.21.15.44:0/4226894354 wait complete. 2024-04-02T13:21:24.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.044+0000 7f6c999d6640 1 Processor -- start 2024-04-02T13:21:24.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.045+0000 7f6c999d6640 1 -- start start 2024-04-02T13:21:24.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.045+0000 7f6c999d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c940cfd60 0x7f6c941ea4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.045+0000 7f6c999d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c940d21c0 0x7f6c9415dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.045+0000 7f6c999d6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c9415c1a0 0x7f6c9415e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.045+0000 7f6c999d6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6c940702d0 con 0x7f6c940cfd60 2024-04-02T13:21:24.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.045+0000 7f6c999d6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9406fff0 con 0x7f6c9415c1a0 2024-04-02T13:21:24.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.045+0000 7f6c999d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6c940705b0 con 0x7f6c940d21c0 2024-04-02T13:21:24.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.045+0000 7f6c92ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c940cfd60 0x7f6c941ea4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.045+0000 7f6c92ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c940cfd60 0x7f6c941ea4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46088/0 (socket says 172.21.15.44:46088) 2024-04-02T13:21:24.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.045+0000 7f6c92ffd640 1 -- 172.21.15.44:0/3661255399 learned_addr learned my addr 172.21.15.44:0/3661255399 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:24.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.045+0000 7f6c927fc640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c940d21c0 0x7f6c9415dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.046+0000 7f6c937fe640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c9415c1a0 0x7f6c9415e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.046+0000 7f6c92ffd640 1 -- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c940d21c0 msgr2=0x7f6c9415dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.046+0000 7f6c92ffd640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c940d21c0 0x7f6c9415dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.046+0000 7f6c92ffd640 1 -- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c9415c1a0 msgr2=0x7f6c9415e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.046+0000 7f6c92ffd640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c9415c1a0 0x7f6c9415e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.046+0000 7f6c92ffd640 1 -- 172.21.15.44:0/3661255399 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c94151130 con 0x7f6c940cfd60 2024-04-02T13:21:24.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.046+0000 7f6c92ffd640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c940cfd60 0x7f6c941ea4e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c034040 tx=0x7f6c8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:24.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.046+0000 7f6c8bfff640 1 -- 172.21.15.44:0/3661255399 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c035040 con 0x7f6c940cfd60 2024-04-02T13:21:24.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.046+0000 7f6c8bfff640 1 -- 172.21.15.44:0/3661255399 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c8c030d20 con 0x7f6c940cfd60 2024-04-02T13:21:24.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.046+0000 7f6c999d6640 1 -- 172.21.15.44:0/3661255399 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c94150e50 con 0x7f6c940cfd60 2024-04-02T13:21:24.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.046+0000 7f6c8bfff640 1 -- 172.21.15.44:0/3661255399 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0456a0 con 0x7f6c940cfd60 2024-04-02T13:21:24.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.046+0000 7f6c999d6640 1 -- 172.21.15.44:0/3661255399 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c9415ecc0 con 0x7f6c940cfd60 2024-04-02T13:21:24.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.047+0000 7f6c999d6640 1 -- 172.21.15.44:0/3661255399 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c94068c00 con 0x7f6c940cfd60 2024-04-02T13:21:24.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.048+0000 7f6c8bfff640 1 -- 172.21.15.44:0/3661255399 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6c8c038020 con 0x7f6c940cfd60 2024-04-02T13:21:24.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.048+0000 7f6c8bfff640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c7406c8f0 0x7f6c7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.048+0000 7f6c8bfff640 1 -- 172.21.15.44:0/3661255399 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f6c8c0ae1d0 con 0x7f6c940cfd60 2024-04-02T13:21:24.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.048+0000 7f6c927fc640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c7406c8f0 0x7f6c7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.048+0000 7f6c927fc640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c7406c8f0 0x7f6c7406edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c007fd0 tx=0x7f6c7c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:24.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.052+0000 7f6c8bfff640 1 -- 172.21.15.44:0/3661255399 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6c8c02d040 con 0x7f6c940cfd60 2024-04-02T13:21:24.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.236+0000 7f6c999d6640 1 -- 172.21.15.44:0/3661255399 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6c940c53b0 con 0x7f6c940cfd60 2024-04-02T13:21:24.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.236+0000 7f6c8bfff640 1 -- 172.21.15.44:0/3661255399 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f6c8c07bb90 con 0x7f6c940cfd60 2024-04-02T13:21:24.238 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:24.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.238+0000 7f6c999d6640 1 -- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c7406c8f0 msgr2=0x7f6c7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.238+0000 7f6c999d6640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c7406c8f0 0x7f6c7406edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c007fd0 tx=0x7f6c7c009040 comp rx=0 tx=0).stop 2024-04-02T13:21:24.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.238+0000 7f6c999d6640 1 -- 172.21.15.44:0/3661255399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c940cfd60 msgr2=0x7f6c941ea4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.238+0000 7f6c999d6640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c940cfd60 0x7f6c941ea4e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c034040 tx=0x7f6c8c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.238+0000 7f6c999d6640 1 -- 172.21.15.44:0/3661255399 shutdown_connections 2024-04-02T13:21:24.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.238+0000 7f6c999d6640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c7406c8f0 0x7f6c7406edb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.238+0000 7f6c999d6640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c9415c1a0 0x7f6c9415e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.238+0000 7f6c999d6640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c940d21c0 0x7f6c9415dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.238+0000 7f6c999d6640 1 --2- 172.21.15.44:0/3661255399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c940cfd60 0x7f6c941ea4e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.238+0000 7f6c999d6640 1 -- 172.21.15.44:0/3661255399 >> 172.21.15.44:0/3661255399 conn(0x7f6c9409aa20 msgr2=0x7f6c940cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:24.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.238+0000 7f6c999d6640 1 -- 172.21.15.44:0/3661255399 shutdown_connections 2024-04-02T13:21:24.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.238+0000 7f6c999d6640 1 -- 172.21.15.44:0/3661255399 wait complete. 2024-04-02T13:21:24.251 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:21:24.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.352+0000 7fbc0962c640 1 Processor -- start 2024-04-02T13:21:24.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.353+0000 7fbc0962c640 1 -- start start 2024-04-02T13:21:24.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.353+0000 7fbc0962c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbc040d28f0 0x7fbc040d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.353+0000 7fbc0962c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc040b5200 0x7fbc040b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.353+0000 7fbc0962c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc040d4d70 con 0x7fbc040d1260 2024-04-02T13:21:24.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.353+0000 7fbc0962c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbc040d5050 con 0x7fbc040d28f0 2024-04-02T13:21:24.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.353+0000 7fbc0962c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbc040d23a0 con 0x7fbc040b5200 2024-04-02T13:21:24.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.353+0000 7fbc037fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbc040d1260 0x7fbc040ced30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48970/0 (socket says 172.21.15.44:48970) 2024-04-02T13:21:24.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.353+0000 7fbc037fe640 1 -- 172.21.15.44:0/1427611950 learned_addr learned my addr 172.21.15.44:0/1427611950 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:24.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.354+0000 7fbc027fc640 1 --2- 172.21.15.44:0/1427611950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc040b5200 0x7fbc040b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.354+0000 7fbc02ffd640 1 --2- 172.21.15.44:0/1427611950 >> v2:172.21.15.69:3300/0 conn(0x7fbc040d28f0 0x7fbc040d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.354+0000 7fbc01ffb640 1 -- 172.21.15.44:0/1427611950 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 595765042 0 0) 0x7fbc040d4d70 con 0x7fbc040d1260 2024-04-02T13:21:24.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.354+0000 7fbc01ffb640 1 -- 172.21.15.44:0/1427611950 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbec003520 con 0x7fbc040d1260 2024-04-02T13:21:24.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.354+0000 7fbc027fc640 1 -- 172.21.15.44:0/1427611950 >> v2:172.21.15.69:3300/0 conn(0x7fbc040d28f0 msgr2=0x7fbc040d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:24.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.354+0000 7fbc027fc640 1 --2- 172.21.15.44:0/1427611950 >> v2:172.21.15.69:3300/0 conn(0x7fbc040d28f0 0x7fbc040d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.354+0000 7fbc027fc640 1 -- 172.21.15.44:0/1427611950 >> v1:172.21.15.44:6789/0 conn(0x7fbc040d1260 legacy=0x7fbc040ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.354+0000 7fbc027fc640 1 -- 172.21.15.44:0/1427611950 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc040d2030 con 0x7fbc040b5200 2024-04-02T13:21:24.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.354+0000 7fbc02ffd640 1 --2- 172.21.15.44:0/1427611950 >> v2:172.21.15.69:3300/0 conn(0x7fbc040d28f0 0x7fbc040d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:24.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.355+0000 7fbc027fc640 1 --2- 172.21.15.44:0/1427611950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc040b5200 0x7fbc040b40a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009d90 tx=0x7fbbf802bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8072e9e96eec27a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:24.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.355+0000 7fbc01ffb640 1 -- 172.21.15.44:0/1427611950 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8039070 con 0x7fbc040b5200 2024-04-02T13:21:24.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.355+0000 7fbc01ffb640 1 -- 172.21.15.44:0/1427611950 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf802cb30 con 0x7fbc040b5200 2024-04-02T13:21:24.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.355+0000 7fbc01ffb640 1 -- 172.21.15.44:0/1427611950 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8034440 con 0x7fbc040b5200 2024-04-02T13:21:24.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.355+0000 7fbc0962c640 1 -- 172.21.15.44:0/1427611950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc040b5200 msgr2=0x7fbc040b40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.355+0000 7fbc0962c640 1 --2- 172.21.15.44:0/1427611950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc040b5200 0x7fbc040b40a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009d90 tx=0x7fbbf802bd80 comp rx=0 tx=0).stop 2024-04-02T13:21:24.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.355+0000 7fbc0962c640 1 -- 172.21.15.44:0/1427611950 shutdown_connections 2024-04-02T13:21:24.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.355+0000 7fbc0962c640 1 --2- 172.21.15.44:0/1427611950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc040b5200 0x7fbc040b40a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.355+0000 7fbc0962c640 1 --2- 172.21.15.44:0/1427611950 >> v2:172.21.15.69:3300/0 conn(0x7fbc040d28f0 0x7fbc040d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.355+0000 7fbc0962c640 1 -- 172.21.15.44:0/1427611950 >> 172.21.15.44:0/1427611950 conn(0x7fbc040d4120 msgr2=0x7fbc040d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:24.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.355+0000 7fbc0962c640 1 -- 172.21.15.44:0/1427611950 shutdown_connections 2024-04-02T13:21:24.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.355+0000 7fbc0962c640 1 -- 172.21.15.44:0/1427611950 wait complete. 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.357+0000 7fbc0962c640 1 Processor -- start 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.357+0000 7fbc0962c640 1 -- start start 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.357+0000 7fbc0962c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc040b5200 0x7fbc040cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.357+0000 7fbc0962c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc040d1260 0x7fbc041604a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.357+0000 7fbc0962c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc040d28f0 0x7fbc0409fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.357+0000 7fbc0962c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbc040b8130 con 0x7fbc040d1260 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.357+0000 7fbc0962c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbc040b5020 con 0x7fbc040d28f0 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.357+0000 7fbc0962c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbc040b82b0 con 0x7fbc040b5200 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc027fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc040d1260 0x7fbc041604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc027fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc040d1260 0x7fbc041604a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46096/0 (socket says 172.21.15.44:46096) 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc027fc640 1 -- 172.21.15.44:0/2394766040 learned_addr learned my addr 172.21.15.44:0/2394766040 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc02ffd640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc040b5200 0x7fbc040cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc037fe640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc040d28f0 0x7fbc0409fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc027fc640 1 -- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc040b5200 msgr2=0x7fbc040cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc027fc640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc040b5200 0x7fbc040cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc027fc640 1 -- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc040d28f0 msgr2=0x7fbc0409fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc027fc640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc040d28f0 0x7fbc0409fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc027fc640 1 -- 172.21.15.44:0/2394766040 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc040ce330 con 0x7fbc040d1260 2024-04-02T13:21:24.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc02ffd640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc040b5200 0x7fbc040cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:24.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc027fc640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc040d1260 0x7fbc041604a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8037040 tx=0x7fbbf8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:24.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbbeb7fe640 1 -- 172.21.15.44:0/2394766040 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8039050 con 0x7fbc040d1260 2024-04-02T13:21:24.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbbeb7fe640 1 -- 172.21.15.44:0/2394766040 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf8045070 con 0x7fbc040d1260 2024-04-02T13:21:24.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc0962c640 1 -- 172.21.15.44:0/2394766040 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc041609e0 con 0x7fbc040d1260 2024-04-02T13:21:24.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbbeb7fe640 1 -- 172.21.15.44:0/2394766040 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8009450 con 0x7fbc040d1260 2024-04-02T13:21:24.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.358+0000 7fbc0962c640 1 -- 172.21.15.44:0/2394766040 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc0409ed30 con 0x7fbc040d1260 2024-04-02T13:21:24.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.359+0000 7fbc0962c640 1 -- 172.21.15.44:0/2394766040 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc0409ef70 con 0x7fbc040d1260 2024-04-02T13:21:24.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.360+0000 7fbbeb7fe640 1 -- 172.21.15.44:0/2394766040 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbbf803e850 con 0x7fbc040d1260 2024-04-02T13:21:24.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.360+0000 7fbbeb7fe640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbbf006c8f0 0x7fbbf006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.360+0000 7fbbeb7fe640 1 -- 172.21.15.44:0/2394766040 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fbbf80aed20 con 0x7fbc040d1260 2024-04-02T13:21:24.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.360+0000 7fbc02ffd640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbbf006c8f0 0x7fbbf006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.361+0000 7fbc02ffd640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbbf006c8f0 0x7fbbf006edb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009a20 tx=0x7fbbfc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:24.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.365+0000 7fbbeb7fe640 1 -- 172.21.15.44:0/2394766040 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbbf8032e20 con 0x7fbc040d1260 2024-04-02T13:21:24.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.561+0000 7fbc0962c640 1 -- 172.21.15.44:0/2394766040 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbc040d0da0 con 0x7fbc040d1260 2024-04-02T13:21:24.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.561+0000 7fbbeb7fe640 1 -- 172.21.15.44:0/2394766040 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fbbf807cee0 con 0x7fbc040d1260 2024-04-02T13:21:24.563 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:21:24.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.563+0000 7fbc0962c640 1 -- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbbf006c8f0 msgr2=0x7fbbf006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.563+0000 7fbc0962c640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbbf006c8f0 0x7fbbf006edb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009a20 tx=0x7fbbfc008040 comp rx=0 tx=0).stop 2024-04-02T13:21:24.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.563+0000 7fbc0962c640 1 -- 172.21.15.44:0/2394766040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc040d1260 msgr2=0x7fbc041604a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.563+0000 7fbc0962c640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc040d1260 0x7fbc041604a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8037040 tx=0x7fbbf8004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:24.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.563+0000 7fbc0962c640 1 -- 172.21.15.44:0/2394766040 shutdown_connections 2024-04-02T13:21:24.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.563+0000 7fbc0962c640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbbf006c8f0 0x7fbbf006edb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.563+0000 7fbc0962c640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc040d28f0 0x7fbc0409fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.563+0000 7fbc0962c640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc040d1260 0x7fbc041604a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.563+0000 7fbc0962c640 1 --2- 172.21.15.44:0/2394766040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc040b5200 0x7fbc040cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.563+0000 7fbc0962c640 1 -- 172.21.15.44:0/2394766040 >> 172.21.15.44:0/2394766040 conn(0x7fbc040d4120 msgr2=0x7fbc041615f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:24.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.563+0000 7fbc0962c640 1 -- 172.21.15.44:0/2394766040 shutdown_connections 2024-04-02T13:21:24.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.563+0000 7fbc0962c640 1 -- 172.21.15.44:0/2394766040 wait complete. 2024-04-02T13:21:24.575 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:21:24.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:24.576 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-02T13:21:24.604 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:21:24.604 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:21:24.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:24.605 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-02T13:21:24.669 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:21:24.669 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:24.685 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:21:24.686 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:24.741 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:24.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.842+0000 7f14b034a640 1 Processor -- start 2024-04-02T13:21:24.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.842+0000 7f14b034a640 1 -- start start 2024-04-02T13:21:24.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14b034a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f14a80d0c50 0x7f14a80d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14b034a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f14a806ce40 0x7f14a806d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14b034a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14a80cc600 con 0x7f14a815e4a0 2024-04-02T13:21:24.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14b034a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f14a80af300 con 0x7f14a806ce40 2024-04-02T13:21:24.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14b034a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f14a80cc8e0 con 0x7f14a80d0c50 2024-04-02T13:21:24.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14ae8e7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f14a806ce40 0x7f14a806d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14ae0e6640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f14a815e4a0 0x7f14a80aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46262/0 (socket says 172.21.15.44:46262) 2024-04-02T13:21:24.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14ae0e6640 1 -- 172.21.15.44:0/1728802309 learned_addr learned my addr 172.21.15.44:0/1728802309 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:24.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14ad8e5640 1 --2- 172.21.15.44:0/1728802309 >> v2:172.21.15.69:3300/0 conn(0x7f14a80d0c50 0x7f14a80d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14ae8e7640 1 -- 172.21.15.44:0/1728802309 >> v2:172.21.15.69:3300/0 conn(0x7f14a80d0c50 msgr2=0x7f14a80d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:24.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14ae8e7640 1 --2- 172.21.15.44:0/1728802309 >> v2:172.21.15.69:3300/0 conn(0x7f14a80d0c50 0x7f14a80d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14ae8e7640 1 -- 172.21.15.44:0/1728802309 >> v1:172.21.15.44:6789/0 conn(0x7f14a815e4a0 legacy=0x7f14a80aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14ae8e7640 1 -- 172.21.15.44:0/1728802309 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a80ccbc0 con 0x7f14a806ce40 2024-04-02T13:21:24.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14ad0e4640 1 -- 172.21.15.44:0/1728802309 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2715125149 0 0) 0x7f14a80cc600 con 0x7f14a815e4a0 2024-04-02T13:21:24.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.843+0000 7f14ad8e5640 1 --2- 172.21.15.44:0/1728802309 >> v2:172.21.15.69:3300/0 conn(0x7f14a80d0c50 0x7f14a80d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.844+0000 7f14ae8e7640 1 --2- 172.21.15.44:0/1728802309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14a806ce40 0x7f14a806d2f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f14a80b8060 tx=0x7f14a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8eacfd59c3d1f15 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.844+0000 7f14ad0e4640 1 -- 172.21.15.44:0/1728802309 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a4039070 con 0x7f14a806ce40 2024-04-02T13:21:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.844+0000 7f14ad0e4640 1 -- 172.21.15.44:0/1728802309 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14a402cc80 con 0x7f14a806ce40 2024-04-02T13:21:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.844+0000 7f14ad0e4640 1 -- 172.21.15.44:0/1728802309 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a4034660 con 0x7f14a806ce40 2024-04-02T13:21:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.845+0000 7f14b034a640 1 -- 172.21.15.44:0/1728802309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14a806ce40 msgr2=0x7f14a806d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.845+0000 7f14b034a640 1 --2- 172.21.15.44:0/1728802309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14a806ce40 0x7f14a806d2f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f14a80b8060 tx=0x7f14a402bed0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.845+0000 7f14b034a640 1 -- 172.21.15.44:0/1728802309 shutdown_connections 2024-04-02T13:21:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.845+0000 7f14b034a640 1 --2- 172.21.15.44:0/1728802309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14a806ce40 0x7f14a806d2f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.845+0000 7f14b034a640 1 --2- 172.21.15.44:0/1728802309 >> v2:172.21.15.69:3300/0 conn(0x7f14a80d0c50 0x7f14a80d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.845+0000 7f14b034a640 1 -- 172.21.15.44:0/1728802309 >> 172.21.15.44:0/1728802309 conn(0x7f14a809aa20 msgr2=0x7f14a809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.845+0000 7f14b034a640 1 -- 172.21.15.44:0/1728802309 shutdown_connections 2024-04-02T13:21:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.845+0000 7f14b034a640 1 -- 172.21.15.44:0/1728802309 wait complete. 2024-04-02T13:21:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.846+0000 7f14b034a640 1 Processor -- start 2024-04-02T13:21:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.846+0000 7f14b034a640 1 -- start start 2024-04-02T13:21:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14b034a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a806ce40 0x7f14a80b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14b034a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a80d0c50 0x7f14a815a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14b034a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14a815e4a0 0x7f14a8161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14b034a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f14a80cd460 con 0x7f14a815e4a0 2024-04-02T13:21:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14b034a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f14a80cd180 con 0x7f14a80d0c50 2024-04-02T13:21:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14b034a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f14a815e880 con 0x7f14a806ce40 2024-04-02T13:21:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14ae8e7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14a815e4a0 0x7f14a8161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14ae8e7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14a815e4a0 0x7f14a8161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54220/0 (socket says 172.21.15.44:54220) 2024-04-02T13:21:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14ae8e7640 1 -- 172.21.15.44:0/442812795 learned_addr learned my addr 172.21.15.44:0/442812795 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14ad8e5640 1 --2- 172.21.15.44:0/442812795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a80d0c50 0x7f14a815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14ae8e7640 1 -- 172.21.15.44:0/442812795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a806ce40 msgr2=0x7f14a80b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14ae8e7640 1 --2- 172.21.15.44:0/442812795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a806ce40 0x7f14a80b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14ae8e7640 1 -- 172.21.15.44:0/442812795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a80d0c50 msgr2=0x7f14a815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14ae8e7640 1 --2- 172.21.15.44:0/442812795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a80d0c50 0x7f14a815a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14ae8e7640 1 -- 172.21.15.44:0/442812795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a80af8d0 con 0x7f14a815e4a0 2024-04-02T13:21:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14ae8e7640 1 --2- 172.21.15.44:0/442812795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14a815e4a0 0x7f14a8161180 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009e80 tx=0x7f14a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f149effd640 1 -- 172.21.15.44:0/442812795 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a4039050 con 0x7f14a815e4a0 2024-04-02T13:21:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f149effd640 1 -- 172.21.15.44:0/442812795 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14a4045070 con 0x7f14a815e4a0 2024-04-02T13:21:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f149effd640 1 -- 172.21.15.44:0/442812795 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a403e830 con 0x7f14a815e4a0 2024-04-02T13:21:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.847+0000 7f14b034a640 1 -- 172.21.15.44:0/442812795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14a80afbb0 con 0x7f14a815e4a0 2024-04-02T13:21:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.848+0000 7f14b034a640 1 -- 172.21.15.44:0/442812795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14a80b0170 con 0x7f14a815e4a0 2024-04-02T13:21:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.848+0000 7f14b034a640 1 -- 172.21.15.44:0/442812795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14a80684f0 con 0x7f14a815e4a0 2024-04-02T13:21:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.849+0000 7f149effd640 1 -- 172.21.15.44:0/442812795 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f14a403e9d0 con 0x7f14a815e4a0 2024-04-02T13:21:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.849+0000 7f149effd640 1 --2- 172.21.15.44:0/442812795 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f148c06c8f0 0x7f148c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.849+0000 7f149effd640 1 -- 172.21.15.44:0/442812795 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f14a40aec60 con 0x7f14a815e4a0 2024-04-02T13:21:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.850+0000 7f14ae0e6640 1 --2- 172.21.15.44:0/442812795 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f148c06c8f0 0x7f148c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.850+0000 7f14ae0e6640 1 --2- 172.21.15.44:0/442812795 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f148c06c8f0 0x7f148c06edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f14a0007fd0 tx=0x7f14a0007960 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:24.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:24.854+0000 7f149effd640 1 -- 172.21.15.44:0/442812795 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f14a407ce20 con 0x7f14a815e4a0 2024-04-02T13:21:25.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.041+0000 7f14b034a640 1 -- 172.21.15.44:0/442812795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14a806f650 con 0x7f14a815e4a0 2024-04-02T13:21:25.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.042+0000 7f149effd640 1 -- 172.21.15.44:0/442812795 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18899 (secure 0 0 0) 0x7f14a407c720 con 0x7f14a815e4a0 2024-04-02T13:21:25.045 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:25.045 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:08.061253+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:25.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.044+0000 7f14b034a640 1 -- 172.21.15.44:0/442812795 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f148c06c8f0 msgr2=0x7f148c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.044+0000 7f14b034a640 1 --2- 172.21.15.44:0/442812795 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f148c06c8f0 0x7f148c06edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f14a0007fd0 tx=0x7f14a0007960 comp rx=0 tx=0).stop 2024-04-02T13:21:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.044+0000 7f14b034a640 1 -- 172.21.15.44:0/442812795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14a815e4a0 msgr2=0x7f14a8161180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.044+0000 7f14b034a640 1 --2- 172.21.15.44:0/442812795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14a815e4a0 0x7f14a8161180 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009e80 tx=0x7f14a4004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.044+0000 7f14b034a640 1 -- 172.21.15.44:0/442812795 shutdown_connections 2024-04-02T13:21:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.044+0000 7f14b034a640 1 --2- 172.21.15.44:0/442812795 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f148c06c8f0 0x7f148c06edb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.044+0000 7f14b034a640 1 --2- 172.21.15.44:0/442812795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14a815e4a0 0x7f14a8161180 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.044+0000 7f14b034a640 1 --2- 172.21.15.44:0/442812795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a80d0c50 0x7f14a815a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.044+0000 7f14b034a640 1 --2- 172.21.15.44:0/442812795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a806ce40 0x7f14a80b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.044+0000 7f14b034a640 1 -- 172.21.15.44:0/442812795 >> 172.21.15.44:0/442812795 conn(0x7f14a809aa20 msgr2=0x7f14a815d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:25.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.044+0000 7f14b034a640 1 -- 172.21.15.44:0/442812795 shutdown_connections 2024-04-02T13:21:25.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.044+0000 7f14b034a640 1 -- 172.21.15.44:0/442812795 wait complete. 2024-04-02T13:21:25.058 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:25.058 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:25.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.162+0000 7ff15d740640 1 Processor -- start 2024-04-02T13:21:25.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.162+0000 7ff15d740640 1 -- start start 2024-04-02T13:21:25.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.163+0000 7ff15d740640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff1580d2dd0 0x7ff1580b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.163+0000 7ff15d740640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1580cf1f0 con 0x7ff1580cec60 2024-04-02T13:21:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.163+0000 7ff15d740640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1580cf4d0 con 0x7ff15815e1d0 2024-04-02T13:21:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.163+0000 7ff15d740640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff1580cfe00 con 0x7ff1580d2dd0 2024-04-02T13:21:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.163+0000 7ff1567fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff1580cec60 0x7ff1580d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46264/0 (socket says 172.21.15.44:46264) 2024-04-02T13:21:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.163+0000 7ff1567fc640 1 -- 172.21.15.44:0/1783456161 learned_addr learned my addr 172.21.15.44:0/1783456161 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.163+0000 7ff1577fe640 1 --2- 172.21.15.44:0/1783456161 >> v2:172.21.15.69:3300/0 conn(0x7ff1580d2dd0 0x7ff1580b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.163+0000 7ff155ffb640 1 -- 172.21.15.44:0/1783456161 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2369780408 0 0) 0x7ff1580cf1f0 con 0x7ff1580cec60 2024-04-02T13:21:25.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.163+0000 7ff155ffb640 1 -- 172.21.15.44:0/1783456161 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff144003520 con 0x7ff1580cec60 2024-04-02T13:21:25.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.163+0000 7ff155ffb640 1 -- 172.21.15.44:0/1783456161 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3494436073 0 0) 0x7ff1580cf4d0 con 0x7ff15815e1d0 2024-04-02T13:21:25.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.163+0000 7ff155ffb640 1 -- 172.21.15.44:0/1783456161 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1580cf1f0 con 0x7ff15815e1d0 2024-04-02T13:21:25.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.164+0000 7ff1577fe640 1 -- 172.21.15.44:0/1783456161 >> v1:172.21.15.69:6789/0 conn(0x7ff15815e1d0 legacy=0x7ff1580d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.164+0000 7ff1577fe640 1 -- 172.21.15.44:0/1783456161 >> v1:172.21.15.44:6789/0 conn(0x7ff1580cec60 legacy=0x7ff1580d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.164+0000 7ff1577fe640 1 -- 172.21.15.44:0/1783456161 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1580cf7b0 con 0x7ff1580d2dd0 2024-04-02T13:21:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.164+0000 7ff1577fe640 1 --2- 172.21.15.44:0/1783456161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1580d2dd0 0x7ff1580b4ce0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff1500097a0 tx=0x7ff15002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57b8602f34857ccf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.164+0000 7ff155ffb640 1 -- 172.21.15.44:0/1783456161 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff150035020 con 0x7ff1580d2dd0 2024-04-02T13:21:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.164+0000 7ff155ffb640 1 -- 172.21.15.44:0/1783456161 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff150031830 con 0x7ff1580d2dd0 2024-04-02T13:21:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.164+0000 7ff155ffb640 1 -- 172.21.15.44:0/1783456161 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff150031be0 con 0x7ff1580d2dd0 2024-04-02T13:21:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.165+0000 7ff15d740640 1 -- 172.21.15.44:0/1783456161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1580d2dd0 msgr2=0x7ff1580b4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.165+0000 7ff15d740640 1 --2- 172.21.15.44:0/1783456161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1580d2dd0 0x7ff1580b4ce0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff1500097a0 tx=0x7ff15002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.165+0000 7ff15d740640 1 -- 172.21.15.44:0/1783456161 shutdown_connections 2024-04-02T13:21:25.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.165+0000 7ff15d740640 1 --2- 172.21.15.44:0/1783456161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1580d2dd0 0x7ff1580b4ce0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.165+0000 7ff15d740640 1 -- 172.21.15.44:0/1783456161 >> 172.21.15.44:0/1783456161 conn(0x7ff15809d3b0 msgr2=0x7ff1580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.165+0000 7ff15d740640 1 -- 172.21.15.44:0/1783456161 shutdown_connections 2024-04-02T13:21:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.165+0000 7ff15d740640 1 -- 172.21.15.44:0/1783456161 wait complete. 2024-04-02T13:21:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.166+0000 7ff15d740640 1 Processor -- start 2024-04-02T13:21:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.166+0000 7ff15d740640 1 -- start start 2024-04-02T13:21:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.167+0000 7ff15d740640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1580cec60 0x7ff1581511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.167+0000 7ff15d740640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1580d2dd0 0x7ff158160290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.167+0000 7ff15d740640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff15815e1d0 0x7ff15814d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.167+0000 7ff15d740640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff1580b8150 con 0x7ff1580d2dd0 2024-04-02T13:21:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.167+0000 7ff15d740640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff1580b7fd0 con 0x7ff1580cec60 2024-04-02T13:21:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.167+0000 7ff15d740640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff1580b82d0 con 0x7ff15815e1d0 2024-04-02T13:21:25.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.167+0000 7ff1567fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1580d2dd0 0x7ff158160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.167+0000 7ff1567fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1580d2dd0 0x7ff158160290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54224/0 (socket says 172.21.15.44:54224) 2024-04-02T13:21:25.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.167+0000 7ff1567fc640 1 -- 172.21.15.44:0/3954512774 learned_addr learned my addr 172.21.15.44:0/3954512774 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:25.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.167+0000 7ff156ffd640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1580cec60 0x7ff1581511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.167+0000 7ff1577fe640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff15815e1d0 0x7ff15814d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.168+0000 7ff1567fc640 1 -- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff15815e1d0 msgr2=0x7ff15814d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.168+0000 7ff1567fc640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff15815e1d0 0x7ff15814d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.168+0000 7ff1567fc640 1 -- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1580cec60 msgr2=0x7ff1581511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.168+0000 7ff1567fc640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1580cec60 0x7ff1581511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.168+0000 7ff1567fc640 1 -- 172.21.15.44:0/3954512774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff158151ae0 con 0x7ff1580d2dd0 2024-04-02T13:21:25.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.168+0000 7ff1567fc640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1580d2dd0 0x7ff158160290 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff14000c410 tx=0x7ff14000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:25.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.168+0000 7ff14f7fe640 1 -- 172.21.15.44:0/3954512774 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140017070 con 0x7ff1580d2dd0 2024-04-02T13:21:25.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.168+0000 7ff14f7fe640 1 -- 172.21.15.44:0/3954512774 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff140005ce0 con 0x7ff1580d2dd0 2024-04-02T13:21:25.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.168+0000 7ff15d740640 1 -- 172.21.15.44:0/3954512774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff158151dc0 con 0x7ff1580d2dd0 2024-04-02T13:21:25.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.168+0000 7ff14f7fe640 1 -- 172.21.15.44:0/3954512774 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1400072f0 con 0x7ff1580d2dd0 2024-04-02T13:21:25.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.168+0000 7ff15d740640 1 -- 172.21.15.44:0/3954512774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff158152380 con 0x7ff1580d2dd0 2024-04-02T13:21:25.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.169+0000 7ff15d740640 1 -- 172.21.15.44:0/3954512774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1580684f0 con 0x7ff1580d2dd0 2024-04-02T13:21:25.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.170+0000 7ff14f7fe640 1 -- 172.21.15.44:0/3954512774 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff14001a070 con 0x7ff1580d2dd0 2024-04-02T13:21:25.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.170+0000 7ff14f7fe640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff13806c8f0 0x7ff13806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.170+0000 7ff14f7fe640 1 -- 172.21.15.44:0/3954512774 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7ff14008b2a0 con 0x7ff1580d2dd0 2024-04-02T13:21:25.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.170+0000 7ff156ffd640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff13806c8f0 0x7ff13806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.171+0000 7ff156ffd640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff13806c8f0 0x7ff13806edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff148001ea0 tx=0x7ff148009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:25.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.174+0000 7ff14f7fe640 1 -- 172.21.15.44:0/3954512774 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff140058db0 con 0x7ff1580d2dd0 2024-04-02T13:21:25.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.358+0000 7ff15d740640 1 -- 172.21.15.44:0/3954512774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff1580d47f0 con 0x7ff1580d2dd0 2024-04-02T13:21:25.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.359+0000 7ff14f7fe640 1 -- 172.21.15.44:0/3954512774 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7ff14005d0b0 con 0x7ff1580d2dd0 2024-04-02T13:21:25.361 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:25.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.360+0000 7ff15d740640 1 -- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff13806c8f0 msgr2=0x7ff13806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.360+0000 7ff15d740640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff13806c8f0 0x7ff13806edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff148001ea0 tx=0x7ff148009040 comp rx=0 tx=0).stop 2024-04-02T13:21:25.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.360+0000 7ff15d740640 1 -- 172.21.15.44:0/3954512774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1580d2dd0 msgr2=0x7ff158160290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.360+0000 7ff15d740640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1580d2dd0 0x7ff158160290 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff14000c410 tx=0x7ff14000c8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.360+0000 7ff15d740640 1 -- 172.21.15.44:0/3954512774 shutdown_connections 2024-04-02T13:21:25.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.360+0000 7ff15d740640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff13806c8f0 0x7ff13806edb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.360+0000 7ff15d740640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff15815e1d0 0x7ff15814d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.360+0000 7ff15d740640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1580d2dd0 0x7ff158160290 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.360+0000 7ff15d740640 1 --2- 172.21.15.44:0/3954512774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1580cec60 0x7ff1581511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.360+0000 7ff15d740640 1 -- 172.21.15.44:0/3954512774 >> 172.21.15.44:0/3954512774 conn(0x7ff15809d3b0 msgr2=0x7ff15806d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:25.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.361+0000 7ff15d740640 1 -- 172.21.15.44:0/3954512774 shutdown_connections 2024-04-02T13:21:25.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.361+0000 7ff15d740640 1 -- 172.21.15.44:0/3954512774 wait complete. 2024-04-02T13:21:25.373 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:25.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.478+0000 7fe5c493d640 1 Processor -- start 2024-04-02T13:21:25.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.478+0000 7fe5c493d640 1 -- start start 2024-04-02T13:21:25.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5c493d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe5c015a940 0x7fe5c00ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5c493d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5c0071200 0x7fe5c006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5c493d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c00ce9f0 con 0x7fe5c00cc640 2024-04-02T13:21:25.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5c493d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe5c006ef80 con 0x7fe5c015a940 2024-04-02T13:21:25.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5c493d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c00ce020 con 0x7fe5c0071200 2024-04-02T13:21:25.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5be59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe5c015a940 0x7fe5c00ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5b7fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5c0071200 0x7fe5c006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5bed9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe5c00cc640 0x7fe5c00acd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:44790/0 (socket says 172.21.15.44:44790) 2024-04-02T13:21:25.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5bed9d640 1 -- 172.21.15.44:0/3572583644 learned_addr learned my addr 172.21.15.44:0/3572583644 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:25.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5bdd9b640 1 -- 172.21.15.44:0/3572583644 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 791197014 0 0) 0x7fe5c00ce9f0 con 0x7fe5c00cc640 2024-04-02T13:21:25.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5bdd9b640 1 -- 172.21.15.44:0/3572583644 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5ac005730 con 0x7fe5c00cc640 2024-04-02T13:21:25.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5be59c640 1 -- 172.21.15.44:0/3572583644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5c0071200 msgr2=0x7fe5c006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:25.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5be59c640 1 --2- 172.21.15.44:0/3572583644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5c0071200 0x7fe5c006e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5be59c640 1 -- 172.21.15.44:0/3572583644 >> v1:172.21.15.69:6789/0 conn(0x7fe5c00cc640 legacy=0x7fe5c00acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.479+0000 7fe5be59c640 1 -- 172.21.15.44:0/3572583644 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c006e9c0 con 0x7fe5c015a940 2024-04-02T13:21:25.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.480+0000 7fe5be59c640 1 --2- 172.21.15.44:0/3572583644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c015a940 0x7fe5c00ce4b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe5b00097a0 tx=0x7fe5b002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc6df3a1bb77214d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:25.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.480+0000 7fe5bdd9b640 1 -- 172.21.15.44:0/3572583644 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b00315a0 con 0x7fe5c015a940 2024-04-02T13:21:25.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.480+0000 7fe5bdd9b640 1 -- 172.21.15.44:0/3572583644 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5b0037070 con 0x7fe5c015a940 2024-04-02T13:21:25.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.480+0000 7fe5bdd9b640 1 -- 172.21.15.44:0/3572583644 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0031960 con 0x7fe5c015a940 2024-04-02T13:21:25.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.481+0000 7fe5c493d640 1 -- 172.21.15.44:0/3572583644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c015a940 msgr2=0x7fe5c00ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.481+0000 7fe5c493d640 1 --2- 172.21.15.44:0/3572583644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c015a940 0x7fe5c00ce4b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe5b00097a0 tx=0x7fe5b002b840 comp rx=0 tx=0).stop 2024-04-02T13:21:25.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.481+0000 7fe5c493d640 1 -- 172.21.15.44:0/3572583644 shutdown_connections 2024-04-02T13:21:25.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.481+0000 7fe5c493d640 1 --2- 172.21.15.44:0/3572583644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5c0071200 0x7fe5c006e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.481+0000 7fe5c493d640 1 --2- 172.21.15.44:0/3572583644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c015a940 0x7fe5c00ce4b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.481+0000 7fe5c493d640 1 -- 172.21.15.44:0/3572583644 >> 172.21.15.44:0/3572583644 conn(0x7fe5c006b920 msgr2=0x7fe5c006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:25.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.481+0000 7fe5c493d640 1 -- 172.21.15.44:0/3572583644 shutdown_connections 2024-04-02T13:21:25.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.481+0000 7fe5c493d640 1 -- 172.21.15.44:0/3572583644 wait complete. 2024-04-02T13:21:25.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.482+0000 7fe5c493d640 1 Processor -- start 2024-04-02T13:21:25.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.482+0000 7fe5c493d640 1 -- start start 2024-04-02T13:21:25.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5c493d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5c0071200 0x7fe5c014e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5c493d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c00cc640 0x7fe5c009aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5c493d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5c015a940 0x7fe5c014aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5c493d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c006e600 con 0x7fe5c0071200 2024-04-02T13:21:25.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5c493d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c00cdcb0 con 0x7fe5c00cc640 2024-04-02T13:21:25.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5c493d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c006e780 con 0x7fe5c015a940 2024-04-02T13:21:25.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5be59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5c0071200 0x7fe5c014e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5be59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5c0071200 0x7fe5c014e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54232/0 (socket says 172.21.15.44:54232) 2024-04-02T13:21:25.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5be59c640 1 -- 172.21.15.44:0/2943995218 learned_addr learned my addr 172.21.15.44:0/2943995218 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:25.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5b7fff640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c00cc640 0x7fe5c009aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5bed9d640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5c015a940 0x7fe5c014aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5be59c640 1 -- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5c015a940 msgr2=0x7fe5c014aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5be59c640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5c015a940 0x7fe5c014aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5be59c640 1 -- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c00cc640 msgr2=0x7fe5c009aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5be59c640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c00cc640 0x7fe5c009aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5be59c640 1 -- 172.21.15.44:0/2943995218 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c009af60 con 0x7fe5c0071200 2024-04-02T13:21:25.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.483+0000 7fe5bed9d640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5c015a940 0x7fe5c014aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:25.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.484+0000 7fe5be59c640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5c0071200 0x7fe5c014e050 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0034040 tx=0x7fe5b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:25.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.484+0000 7fe5b6ffd640 1 -- 172.21.15.44:0/2943995218 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0032630 con 0x7fe5c0071200 2024-04-02T13:21:25.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.484+0000 7fe5b6ffd640 1 -- 172.21.15.44:0/2943995218 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5b00327f0 con 0x7fe5c0071200 2024-04-02T13:21:25.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.484+0000 7fe5c493d640 1 -- 172.21.15.44:0/2943995218 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c009b230 con 0x7fe5c0071200 2024-04-02T13:21:25.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.484+0000 7fe5b6ffd640 1 -- 172.21.15.44:0/2943995218 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0032be0 con 0x7fe5c0071200 2024-04-02T13:21:25.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.484+0000 7fe5c493d640 1 -- 172.21.15.44:0/2943995218 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c009b7f0 con 0x7fe5c0071200 2024-04-02T13:21:25.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.485+0000 7fe5c493d640 1 -- 172.21.15.44:0/2943995218 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5c00684f0 con 0x7fe5c0071200 2024-04-02T13:21:25.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.485+0000 7fe5b6ffd640 1 -- 172.21.15.44:0/2943995218 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe5b004e430 con 0x7fe5c0071200 2024-04-02T13:21:25.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.486+0000 7fe5b6ffd640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe5a006c920 0x7fe5a006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.486+0000 7fe5b6ffd640 1 -- 172.21.15.44:0/2943995218 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fe5b002f030 con 0x7fe5c0071200 2024-04-02T13:21:25.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.486+0000 7fe5b7fff640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe5a006c920 0x7fe5a006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.486+0000 7fe5b7fff640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe5a006c920 0x7fe5a006ede0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe5c01f76b0 tx=0x7fe5a800a2a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:25.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.490+0000 7fe5b6ffd640 1 -- 172.21.15.44:0/2943995218 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe5b0040070 con 0x7fe5c0071200 2024-04-02T13:21:25.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.678+0000 7fe5c493d640 1 -- 172.21.15.44:0/2943995218 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe5c006df80 con 0x7fe5c0071200 2024-04-02T13:21:25.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.678+0000 7fe5b6ffd640 1 -- 172.21.15.44:0/2943995218 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fe5b007bca0 con 0x7fe5c0071200 2024-04-02T13:21:25.680 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:25.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.680+0000 7fe5c493d640 1 -- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe5a006c920 msgr2=0x7fe5a006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.680+0000 7fe5c493d640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe5a006c920 0x7fe5a006ede0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe5c01f76b0 tx=0x7fe5a800a2a0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.680+0000 7fe5c493d640 1 -- 172.21.15.44:0/2943995218 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5c0071200 msgr2=0x7fe5c014e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.680+0000 7fe5c493d640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5c0071200 0x7fe5c014e050 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0034040 tx=0x7fe5b00046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.680+0000 7fe5c493d640 1 -- 172.21.15.44:0/2943995218 shutdown_connections 2024-04-02T13:21:25.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.680+0000 7fe5c493d640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe5a006c920 0x7fe5a006ede0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.680+0000 7fe5c493d640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5c015a940 0x7fe5c014aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.680+0000 7fe5c493d640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c00cc640 0x7fe5c009aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.680+0000 7fe5c493d640 1 --2- 172.21.15.44:0/2943995218 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5c0071200 0x7fe5c014e050 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.680+0000 7fe5c493d640 1 -- 172.21.15.44:0/2943995218 >> 172.21.15.44:0/2943995218 conn(0x7fe5c006b920 msgr2=0x7fe5c00b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:25.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.680+0000 7fe5c493d640 1 -- 172.21.15.44:0/2943995218 shutdown_connections 2024-04-02T13:21:25.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.680+0000 7fe5c493d640 1 -- 172.21.15.44:0/2943995218 wait complete. 2024-04-02T13:21:25.693 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:25.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.799+0000 7f0aaec86640 1 Processor -- start 2024-04-02T13:21:25.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.799+0000 7f0aaec86640 1 -- start start 2024-04-02T13:21:25.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aaec86640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aa8070690 0x7f0aa80b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aaec86640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0aa80722a0 0x7f0aa8072710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aaec86640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0aa80b44b0 con 0x7f0aa8070a70 2024-04-02T13:21:25.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aaec86640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0aa8072c50 con 0x7f0aa80722a0 2024-04-02T13:21:25.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aaec86640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa80b4900 con 0x7f0aa8070690 2024-04-02T13:21:25.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aad223640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0aa80722a0 0x7f0aa8072710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aad223640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0aa80722a0 0x7f0aa8072710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54236/0 (socket says 172.21.15.44:54236) 2024-04-02T13:21:25.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aad223640 1 -- 172.21.15.44:0/2877336736 learned_addr learned my addr 172.21.15.44:0/2877336736 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:25.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aaca22640 1 --2- 172.21.15.44:0/2877336736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aa8070690 0x7f0aa80b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aad223640 1 -- 172.21.15.44:0/2877336736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aa8070690 msgr2=0x7f0aa80b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:25.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aad223640 1 --2- 172.21.15.44:0/2877336736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aa8070690 0x7f0aa80b3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aad223640 1 -- 172.21.15.44:0/2877336736 >> v1:172.21.15.69:6789/0 conn(0x7f0aa8070a70 legacy=0x7f0aa815c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aad223640 1 -- 172.21.15.44:0/2877336736 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa80b4be0 con 0x7f0aa80722a0 2024-04-02T13:21:25.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.800+0000 7f0aaca22640 1 --2- 172.21.15.44:0/2877336736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aa8070690 0x7f0aa80b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:25.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.801+0000 7f0aad223640 1 --2- 172.21.15.44:0/2877336736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aa80722a0 0x7f0aa8072710 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0009e50 tx=0x7f0aa002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe1551e05b198ac server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:25.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.801+0000 7f0aa77fe640 1 -- 172.21.15.44:0/2877336736 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa002cb00 con 0x7f0aa80722a0 2024-04-02T13:21:25.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.801+0000 7f0aa77fe640 1 -- 172.21.15.44:0/2877336736 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0aa002cca0 con 0x7f0aa80722a0 2024-04-02T13:21:25.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.801+0000 7f0aa77fe640 1 -- 172.21.15.44:0/2877336736 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0034610 con 0x7f0aa80722a0 2024-04-02T13:21:25.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.802+0000 7f0aaec86640 1 -- 172.21.15.44:0/2877336736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aa80722a0 msgr2=0x7f0aa8072710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.802+0000 7f0aaec86640 1 --2- 172.21.15.44:0/2877336736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aa80722a0 0x7f0aa8072710 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0009e50 tx=0x7f0aa002bf60 comp rx=0 tx=0).stop 2024-04-02T13:21:25.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.802+0000 7f0aaec86640 1 -- 172.21.15.44:0/2877336736 shutdown_connections 2024-04-02T13:21:25.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.802+0000 7f0aaec86640 1 --2- 172.21.15.44:0/2877336736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aa80722a0 0x7f0aa8072710 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.802+0000 7f0aaec86640 1 --2- 172.21.15.44:0/2877336736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aa8070690 0x7f0aa80b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.802+0000 7f0aaec86640 1 -- 172.21.15.44:0/2877336736 >> 172.21.15.44:0/2877336736 conn(0x7f0aa806c5e0 msgr2=0x7f0aa806c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:25.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.802+0000 7f0aaec86640 1 -- 172.21.15.44:0/2877336736 shutdown_connections 2024-04-02T13:21:25.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.802+0000 7f0aaec86640 1 -- 172.21.15.44:0/2877336736 wait complete. 2024-04-02T13:21:25.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.803+0000 7f0aaec86640 1 Processor -- start 2024-04-02T13:21:25.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.803+0000 7f0aaec86640 1 -- start start 2024-04-02T13:21:25.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aaec86640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aa8070690 0x7f0aa80b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aaec86640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aa8070a70 0x7f0aa81f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aaec86640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aa80722a0 0x7f0aa81556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aaec86640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa8155230 con 0x7f0aa80722a0 2024-04-02T13:21:25.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aaec86640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa81550b0 con 0x7f0aa8070a70 2024-04-02T13:21:25.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aaec86640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa81553b0 con 0x7f0aa8070690 2024-04-02T13:21:25.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aad223640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aa80722a0 0x7f0aa81556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aad223640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aa80722a0 0x7f0aa81556d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54240/0 (socket says 172.21.15.44:54240) 2024-04-02T13:21:25.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aad223640 1 -- 172.21.15.44:0/4204074241 learned_addr learned my addr 172.21.15.44:0/4204074241 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:25.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aaca22640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aa8070690 0x7f0aa80b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aa7fff640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aa8070a70 0x7f0aa81f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aad223640 1 -- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aa8070690 msgr2=0x7f0aa80b03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aad223640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aa8070690 0x7f0aa80b03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aad223640 1 -- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aa8070a70 msgr2=0x7f0aa81f53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:25.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aad223640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aa8070a70 0x7f0aa81f53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:25.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.804+0000 7f0aad223640 1 -- 172.21.15.44:0/4204074241 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa81f5b40 con 0x7f0aa80722a0 2024-04-02T13:21:25.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.805+0000 7f0aad223640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aa80722a0 0x7f0aa81556d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0009e50 tx=0x7f0aa0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:25.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.805+0000 7f0aa57fa640 1 -- 172.21.15.44:0/4204074241 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0043020 con 0x7f0aa80722a0 2024-04-02T13:21:25.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.805+0000 7f0aa57fa640 1 -- 172.21.15.44:0/4204074241 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0aa002ce50 con 0x7f0aa80722a0 2024-04-02T13:21:25.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.805+0000 7f0aaec86640 1 -- 172.21.15.44:0/4204074241 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aa81f5e20 con 0x7f0aa80722a0 2024-04-02T13:21:25.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.805+0000 7f0aa57fa640 1 -- 172.21.15.44:0/4204074241 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0009510 con 0x7f0aa80722a0 2024-04-02T13:21:25.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.805+0000 7f0aaec86640 1 -- 172.21.15.44:0/4204074241 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aa8151ea0 con 0x7f0aa80722a0 2024-04-02T13:21:25.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.806+0000 7f0aaec86640 1 -- 172.21.15.44:0/4204074241 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a88005290 con 0x7f0aa80722a0 2024-04-02T13:21:25.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.807+0000 7f0aa57fa640 1 -- 172.21.15.44:0/4204074241 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0aa0046020 con 0x7f0aa80722a0 2024-04-02T13:21:25.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.807+0000 7f0aa57fa640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a9406c8f0 0x7f0a9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:25.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.807+0000 7f0aa57fa640 1 -- 172.21.15.44:0/4204074241 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f0aa00ae400 con 0x7f0aa80722a0 2024-04-02T13:21:25.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.808+0000 7f0aaca22640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a9406c8f0 0x7f0a9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:25.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.808+0000 7f0aaca22640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a9406c8f0 0x7f0a9406edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0a98006fd0 tx=0x7f0a98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:25.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:25.816+0000 7f0aa57fa640 1 -- 172.21.15.44:0/4204074241 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0aa004d030 con 0x7f0aa80722a0 2024-04-02T13:21:26.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.000+0000 7f0aaec86640 1 -- 172.21.15.44:0/4204074241 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0a88005850 con 0x7f0aa80722a0 2024-04-02T13:21:26.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.000+0000 7f0aa57fa640 1 -- 172.21.15.44:0/4204074241 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f0aa007bdc0 con 0x7f0aa80722a0 2024-04-02T13:21:26.003 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:26.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.002+0000 7f0aaec86640 1 -- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a9406c8f0 msgr2=0x7f0a9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.002+0000 7f0aaec86640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a9406c8f0 0x7f0a9406edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0a98006fd0 tx=0x7f0a98008040 comp rx=0 tx=0).stop 2024-04-02T13:21:26.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.002+0000 7f0aaec86640 1 -- 172.21.15.44:0/4204074241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aa80722a0 msgr2=0x7f0aa81556d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.002+0000 7f0aaec86640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aa80722a0 0x7f0aa81556d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0009e50 tx=0x7f0aa0004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:26.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.003+0000 7f0aaec86640 1 -- 172.21.15.44:0/4204074241 shutdown_connections 2024-04-02T13:21:26.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.003+0000 7f0aaec86640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a9406c8f0 0x7f0a9406edb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.003+0000 7f0aaec86640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aa80722a0 0x7f0aa81556d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.003+0000 7f0aaec86640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aa8070a70 0x7f0aa81f53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.003+0000 7f0aaec86640 1 --2- 172.21.15.44:0/4204074241 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aa8070690 0x7f0aa80b03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.003+0000 7f0aaec86640 1 -- 172.21.15.44:0/4204074241 >> 172.21.15.44:0/4204074241 conn(0x7f0aa806c5e0 msgr2=0x7f0aa80cf100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:26.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.003+0000 7f0aaec86640 1 -- 172.21.15.44:0/4204074241 shutdown_connections 2024-04-02T13:21:26.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.003+0000 7f0aaec86640 1 -- 172.21.15.44:0/4204074241 wait complete. 2024-04-02T13:21:26.018 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:26.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.127+0000 7f6f4ce0d640 1 Processor -- start 2024-04-02T13:21:26.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.127+0000 7f6f4ce0d640 1 -- start start 2024-04-02T13:21:26.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.127+0000 7f6f4ce0d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6f4815fd30 0x7f6f4806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.128+0000 7f6f4ce0d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f481601b0 0x7f6f4806ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.128+0000 7f6f4ce0d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f48160710 con 0x7f6f480ce4b0 2024-04-02T13:21:26.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.128+0000 7f6f4ce0d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f6f48160cd0 con 0x7f6f4815fd30 2024-04-02T13:21:26.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.128+0000 7f6f4ce0d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6f4815ccc0 con 0x7f6f481601b0 2024-04-02T13:21:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.128+0000 7f6f3ffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6f4815fd30 0x7f6f4806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.128+0000 7f6f4659c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6f480ce4b0 0x7f6f4815d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46280/0 (socket says 172.21.15.44:46280) 2024-04-02T13:21:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.128+0000 7f6f4659c640 1 -- 172.21.15.44:0/1635298753 learned_addr learned my addr 172.21.15.44:0/1635298753 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.128+0000 7f6f46d9d640 1 --2- 172.21.15.44:0/1635298753 >> v2:172.21.15.69:3300/0 conn(0x7f6f481601b0 0x7f6f4806ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.128+0000 7f6f3ffff640 1 -- 172.21.15.44:0/1635298753 >> v2:172.21.15.69:3300/0 conn(0x7f6f481601b0 msgr2=0x7f6f4806ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.128+0000 7f6f3ffff640 1 --2- 172.21.15.44:0/1635298753 >> v2:172.21.15.69:3300/0 conn(0x7f6f481601b0 0x7f6f4806ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.128+0000 7f6f3ffff640 1 -- 172.21.15.44:0/1635298753 >> v1:172.21.15.44:6789/0 conn(0x7f6f480ce4b0 legacy=0x7f6f4815d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.128+0000 7f6f3ffff640 1 -- 172.21.15.44:0/1635298753 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f481609f0 con 0x7f6f4815fd30 2024-04-02T13:21:26.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.128+0000 7f6f46d9d640 1 --2- 172.21.15.44:0/1635298753 >> v2:172.21.15.69:3300/0 conn(0x7f6f481601b0 0x7f6f4806ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:26.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.129+0000 7f6f3ffff640 1 --2- 172.21.15.44:0/1635298753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f4815fd30 0x7f6f4806b4d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6f480d0c60 tx=0x7f6f3002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6ed3054e06ea09 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:26.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.129+0000 7f6f45d9b640 1 -- 172.21.15.44:0/1635298753 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f30035020 con 0x7f6f4815fd30 2024-04-02T13:21:26.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.129+0000 7f6f45d9b640 1 -- 172.21.15.44:0/1635298753 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f30031830 con 0x7f6f4815fd30 2024-04-02T13:21:26.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.129+0000 7f6f45d9b640 1 -- 172.21.15.44:0/1635298753 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f30031c00 con 0x7f6f4815fd30 2024-04-02T13:21:26.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.130+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/1635298753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f4815fd30 msgr2=0x7f6f4806b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.130+0000 7f6f4ce0d640 1 --2- 172.21.15.44:0/1635298753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f4815fd30 0x7f6f4806b4d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6f480d0c60 tx=0x7f6f3002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.130+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/1635298753 shutdown_connections 2024-04-02T13:21:26.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.130+0000 7f6f4ce0d640 1 --2- 172.21.15.44:0/1635298753 >> v2:172.21.15.69:3300/0 conn(0x7f6f481601b0 0x7f6f4806ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.130+0000 7f6f4ce0d640 1 --2- 172.21.15.44:0/1635298753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f4815fd30 0x7f6f4806b4d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.130+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/1635298753 >> 172.21.15.44:0/1635298753 conn(0x7f6f480d0de0 msgr2=0x7f6f480d11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:26.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.130+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/1635298753 shutdown_connections 2024-04-02T13:21:26.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.130+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/1635298753 wait complete. 2024-04-02T13:21:26.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.131+0000 7f6f4ce0d640 1 Processor -- start 2024-04-02T13:21:26.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.131+0000 7f6f4ce0d640 1 -- start start 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f4ce0d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f480ce4b0 0x7f6f4815e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f4ce0d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4815fd30 0x7f6f4809b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f4ce0d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f481601b0 0x7f6f4814f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f4ce0d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6f480cdb10 con 0x7f6f481601b0 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f4ce0d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f480cd990 con 0x7f6f480ce4b0 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f4ce0d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6f480b4d50 con 0x7f6f4815fd30 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f46d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f481601b0 0x7f6f4814f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f46d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f481601b0 0x7f6f4814f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54250/0 (socket says 172.21.15.44:54250) 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f46d9d640 1 -- 172.21.15.44:0/3413742746 learned_addr learned my addr 172.21.15.44:0/3413742746 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f3ffff640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4815fd30 0x7f6f4809b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f4659c640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f480ce4b0 0x7f6f4815e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f46d9d640 1 -- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4815fd30 msgr2=0x7f6f4809b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f46d9d640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4815fd30 0x7f6f4809b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f46d9d640 1 -- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f480ce4b0 msgr2=0x7f6f4815e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f46d9d640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f480ce4b0 0x7f6f4815e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f46d9d640 1 -- 172.21.15.44:0/3413742746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f4809bef0 con 0x7f6f481601b0 2024-04-02T13:21:26.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f4659c640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f480ce4b0 0x7f6f4815e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:26.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.132+0000 7f6f46d9d640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f481601b0 0x7f6f4814f4b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000cca0 tx=0x7f6f4000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:26.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.133+0000 7f6f3effd640 1 -- 172.21.15.44:0/3413742746 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4000f680 con 0x7f6f481601b0 2024-04-02T13:21:26.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.133+0000 7f6f3effd640 1 -- 172.21.15.44:0/3413742746 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f4000f840 con 0x7f6f481601b0 2024-04-02T13:21:26.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.133+0000 7f6f3effd640 1 -- 172.21.15.44:0/3413742746 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4000fc30 con 0x7f6f481601b0 2024-04-02T13:21:26.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.133+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/3413742746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f4809c1d0 con 0x7f6f481601b0 2024-04-02T13:21:26.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.133+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/3413742746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f4809c790 con 0x7f6f481601b0 2024-04-02T13:21:26.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.134+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/3413742746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f48068c00 con 0x7f6f481601b0 2024-04-02T13:21:26.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.135+0000 7f6f3effd640 1 -- 172.21.15.44:0/3413742746 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6f4000e3e0 con 0x7f6f481601b0 2024-04-02T13:21:26.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.135+0000 7f6f3effd640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f2806c8f0 0x7f6f2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.135+0000 7f6f3effd640 1 -- 172.21.15.44:0/3413742746 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f6f4008cbd0 con 0x7f6f481601b0 2024-04-02T13:21:26.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.136+0000 7f6f4659c640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f2806c8f0 0x7f6f2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.136+0000 7f6f4659c640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f2806c8f0 0x7f6f2806edb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6f38000f60 tx=0x7f6f38009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:26.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.140+0000 7f6f3effd640 1 -- 172.21.15.44:0/3413742746 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6f40014030 con 0x7f6f481601b0 2024-04-02T13:21:26.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.328+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/3413742746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6f480cd550 con 0x7f6f481601b0 2024-04-02T13:21:26.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.328+0000 7f6f3effd640 1 -- 172.21.15.44:0/3413742746 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f6f4005abe0 con 0x7f6f481601b0 2024-04-02T13:21:26.331 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.331+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f2806c8f0 msgr2=0x7f6f2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.331+0000 7f6f4ce0d640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f2806c8f0 0x7f6f2806edb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6f38000f60 tx=0x7f6f38009040 comp rx=0 tx=0).stop 2024-04-02T13:21:26.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.331+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/3413742746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f481601b0 msgr2=0x7f6f4814f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.331+0000 7f6f4ce0d640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f481601b0 0x7f6f4814f4b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000cca0 tx=0x7f6f4000a650 comp rx=0 tx=0).stop 2024-04-02T13:21:26.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.331+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/3413742746 shutdown_connections 2024-04-02T13:21:26.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.331+0000 7f6f4ce0d640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f2806c8f0 0x7f6f2806edb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.331+0000 7f6f4ce0d640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f481601b0 0x7f6f4814f4b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.331+0000 7f6f4ce0d640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4815fd30 0x7f6f4809b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.331+0000 7f6f4ce0d640 1 --2- 172.21.15.44:0/3413742746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f480ce4b0 0x7f6f4815e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.331+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/3413742746 >> 172.21.15.44:0/3413742746 conn(0x7f6f480d0de0 msgr2=0x7f6f4815a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:26.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.331+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/3413742746 shutdown_connections 2024-04-02T13:21:26.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.331+0000 7f6f4ce0d640 1 -- 172.21.15.44:0/3413742746 wait complete. 2024-04-02T13:21:26.344 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:21:26.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.445+0000 7f6926031640 1 Processor -- start 2024-04-02T13:21:26.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.445+0000 7f6926031640 1 -- start start 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.446+0000 7f6926031640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6920166f50 0x7f6920169340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.446+0000 7f6926031640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69200cf5e0 con 0x7f692006c7f0 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.446+0000 7f6926031640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f692015e810 con 0x7f692015e430 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.446+0000 7f6926031640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f69200d1410 con 0x7f6920166f50 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.446+0000 7f691effd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f692006c7f0 0x7f69200ceed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46282/0 (socket says 172.21.15.44:46282) 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.446+0000 7f691effd640 1 -- 172.21.15.44:0/832867595 learned_addr learned my addr 172.21.15.44:0/832867595 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.446+0000 7f691ffff640 1 --2- 172.21.15.44:0/832867595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6920166f50 0x7f6920169340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.446+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2922313806 0 0) 0x7f69200cf5e0 con 0x7f692006c7f0 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.446+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f690c003520 con 0x7f692006c7f0 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2403667511 0 0) 0x7f690c003520 con 0x7f692006c7f0 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6920166f50 msgr2=0x7f6920169340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691e7fc640 1 --2- 172.21.15.44:0/832867595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6920166f50 0x7f6920169340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 >> v1:172.21.15.69:6789/0 conn(0x7f692015e430 legacy=0x7f69200ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692006d240 con 0x7f692006c7f0 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f6908001670 con 0x7f692006c7f0 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 >> v1:172.21.15.44:6789/0 conn(0x7f692006c7f0 legacy=0x7f69200ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691e7fc640 1 --2- 172.21.15.44:0/832867595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f690c0039e0 0x7f690c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691e7fc640 1 --2- 172.21.15.44:0/832867595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6920166f50 0x7f690c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691e7fc640 1 --2- 172.21.15.44:0/832867595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f690c004d40 0x7f690c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f690c00ba70 con 0x7f690c0039e0 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f690c00bbf0 con 0x7f690c004d40 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f690c00bdf0 con 0x7f6920166f50 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691f7fe640 1 --2- 172.21.15.44:0/832867595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f690c0039e0 0x7f690c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691ffff640 1 --2- 172.21.15.44:0/832867595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f690c004d40 0x7f690c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691effd640 1 --2- 172.21.15.44:0/832867595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6920166f50 0x7f690c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691f7fe640 1 -- 172.21.15.44:0/832867595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6920166f50 msgr2=0x7f690c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691f7fe640 1 --2- 172.21.15.44:0/832867595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6920166f50 0x7f690c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691f7fe640 1 -- 172.21.15.44:0/832867595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f690c004d40 msgr2=0x7f690c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:26.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691f7fe640 1 --2- 172.21.15.44:0/832867595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f690c004d40 0x7f690c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691f7fe640 1 -- 172.21.15.44:0/832867595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f690c003520 con 0x7f690c0039e0 2024-04-02T13:21:26.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.447+0000 7f691ffff640 1 --2- 172.21.15.44:0/832867595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f690c004d40 0x7f690c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:26.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.448+0000 7f691f7fe640 1 --2- 172.21.15.44:0/832867595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f690c0039e0 0x7f690c003dc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6910001860 tx=0x7f690c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1aa89bd7fca7c4c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:26.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.448+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691002f050 con 0x7f690c0039e0 2024-04-02T13:21:26.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.448+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6910033650 con 0x7f690c0039e0 2024-04-02T13:21:26.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.448+0000 7f691e7fc640 1 -- 172.21.15.44:0/832867595 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910001430 con 0x7f690c0039e0 2024-04-02T13:21:26.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.449+0000 7f6926031640 1 -- 172.21.15.44:0/832867595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f690c0039e0 msgr2=0x7f690c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.449+0000 7f6926031640 1 --2- 172.21.15.44:0/832867595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f690c0039e0 0x7f690c003dc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6910001860 tx=0x7f690c005830 comp rx=0 tx=0).stop 2024-04-02T13:21:26.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.449+0000 7f691ffff640 1 -- 172.21.15.44:0/832867595 reap_dead start 2024-04-02T13:21:26.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.449+0000 7f6926031640 1 -- 172.21.15.44:0/832867595 shutdown_connections 2024-04-02T13:21:26.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.449+0000 7f6926031640 1 -- 172.21.15.44:0/832867595 >> 172.21.15.44:0/832867595 conn(0x7f6920073850 msgr2=0x7f692009aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:26.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.449+0000 7f6926031640 1 -- 172.21.15.44:0/832867595 shutdown_connections 2024-04-02T13:21:26.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.449+0000 7f6926031640 1 -- 172.21.15.44:0/832867595 wait complete. 2024-04-02T13:21:26.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.450+0000 7f6926031640 1 Processor -- start 2024-04-02T13:21:26.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.450+0000 7f6926031640 1 -- start start 2024-04-02T13:21:26.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f6926031640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f690c0039e0 0x7f69200c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f6926031640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6920150f30 0x7f69200b0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f6926031640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69200c9010 0x7f69200c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f6926031640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f69200d0c10 con 0x7f690c0039e0 2024-04-02T13:21:26.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f6926031640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f692006cff0 con 0x7f6920150f30 2024-04-02T13:21:26.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f6926031640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f69200d0ef0 con 0x7f69200c9010 2024-04-02T13:21:26.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f691f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f690c0039e0 0x7f69200c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f691f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f690c0039e0 0x7f69200c5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54276/0 (socket says 172.21.15.44:54276) 2024-04-02T13:21:26.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f691f7fe640 1 -- 172.21.15.44:0/557328023 learned_addr learned my addr 172.21.15.44:0/557328023 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:26.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f691effd640 1 --2- 172.21.15.44:0/557328023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6920150f30 0x7f69200b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f691f7fe640 1 -- 172.21.15.44:0/557328023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69200c9010 msgr2=0x7f69200c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f691f7fe640 1 --2- 172.21.15.44:0/557328023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69200c9010 0x7f69200c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f691f7fe640 1 -- 172.21.15.44:0/557328023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6920150f30 msgr2=0x7f69200b0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f691f7fe640 1 --2- 172.21.15.44:0/557328023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6920150f30 0x7f69200b0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f691f7fe640 1 -- 172.21.15.44:0/557328023 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69200c5f90 con 0x7f690c0039e0 2024-04-02T13:21:26.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.451+0000 7f691f7fe640 1 --2- 172.21.15.44:0/557328023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f690c0039e0 0x7f69200c5a50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6910001980 tx=0x7f6910001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:26.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.452+0000 7f6917fff640 1 -- 172.21.15.44:0/557328023 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910001430 con 0x7f690c0039e0 2024-04-02T13:21:26.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.452+0000 7f6917fff640 1 -- 172.21.15.44:0/557328023 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69100035f0 con 0x7f690c0039e0 2024-04-02T13:21:26.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.452+0000 7f6926031640 1 -- 172.21.15.44:0/557328023 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69200cc570 con 0x7f690c0039e0 2024-04-02T13:21:26.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.452+0000 7f6917fff640 1 -- 172.21.15.44:0/557328023 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69100039a0 con 0x7f690c0039e0 2024-04-02T13:21:26.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.452+0000 7f6926031640 1 -- 172.21.15.44:0/557328023 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69200c64f0 con 0x7f690c0039e0 2024-04-02T13:21:26.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.453+0000 7f6926031640 1 -- 172.21.15.44:0/557328023 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6920068c00 con 0x7f690c0039e0 2024-04-02T13:21:26.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.453+0000 7f6917fff640 1 -- 172.21.15.44:0/557328023 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6910003b40 con 0x7f690c0039e0 2024-04-02T13:21:26.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.454+0000 7f6917fff640 1 --2- 172.21.15.44:0/557328023 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f690006c920 0x7f690006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.454+0000 7f6917fff640 1 -- 172.21.15.44:0/557328023 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f69100ae040 con 0x7f690c0039e0 2024-04-02T13:21:26.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.454+0000 7f691effd640 1 --2- 172.21.15.44:0/557328023 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f690006c920 0x7f690006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.454+0000 7f691effd640 1 --2- 172.21.15.44:0/557328023 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f690006c920 0x7f690006ede0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6908001ce0 tx=0x7f6908001e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:26.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.458+0000 7f6917fff640 1 -- 172.21.15.44:0/557328023 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f691004a060 con 0x7f690c0039e0 2024-04-02T13:21:26.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.656+0000 7f6926031640 1 -- 172.21.15.44:0/557328023 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f69200afaa0 con 0x7f690c0039e0 2024-04-02T13:21:26.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.656+0000 7f6917fff640 1 -- 172.21.15.44:0/557328023 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f69200c5f90 con 0x7f690c0039e0 2024-04-02T13:21:26.659 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:21:26.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.658+0000 7f6926031640 1 -- 172.21.15.44:0/557328023 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f690006c920 msgr2=0x7f690006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.658+0000 7f6926031640 1 --2- 172.21.15.44:0/557328023 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f690006c920 0x7f690006ede0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6908001ce0 tx=0x7f6908001e30 comp rx=0 tx=0).stop 2024-04-02T13:21:26.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.658+0000 7f6926031640 1 -- 172.21.15.44:0/557328023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f690c0039e0 msgr2=0x7f69200c5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.658+0000 7f6926031640 1 --2- 172.21.15.44:0/557328023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f690c0039e0 0x7f69200c5a50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6910001980 tx=0x7f6910001230 comp rx=0 tx=0).stop 2024-04-02T13:21:26.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.658+0000 7f6926031640 1 -- 172.21.15.44:0/557328023 shutdown_connections 2024-04-02T13:21:26.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.658+0000 7f6926031640 1 --2- 172.21.15.44:0/557328023 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f690006c920 0x7f690006ede0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.658+0000 7f6926031640 1 --2- 172.21.15.44:0/557328023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69200c9010 0x7f69200c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.658+0000 7f6926031640 1 --2- 172.21.15.44:0/557328023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6920150f30 0x7f69200b0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.658+0000 7f6926031640 1 --2- 172.21.15.44:0/557328023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f690c0039e0 0x7f69200c5a50 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.658+0000 7f6926031640 1 -- 172.21.15.44:0/557328023 >> 172.21.15.44:0/557328023 conn(0x7f6920073850 msgr2=0x7f69200aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:26.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.659+0000 7f6926031640 1 -- 172.21.15.44:0/557328023 shutdown_connections 2024-04-02T13:21:26.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.659+0000 7f6926031640 1 -- 172.21.15.44:0/557328023 wait complete. 2024-04-02T13:21:26.672 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:21:26.672 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:26.691 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:26.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.790+0000 7f4663670640 1 Processor -- start 2024-04-02T13:21:26.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.790+0000 7f4663670640 1 -- start start 2024-04-02T13:21:26.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.790+0000 7f4663670640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f465c15f040 0x7f465c168f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.790+0000 7f4663670640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f465c15fb50 con 0x7f465c06d700 2024-04-02T13:21:26.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.790+0000 7f4663670640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f465c0d1ac0 con 0x7f465c15e550 2024-04-02T13:21:26.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.790+0000 7f4663670640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f465c06ca90 con 0x7f465c15f040 2024-04-02T13:21:26.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4660c0b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f465c06d700 0x7f465c15e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46288/0 (socket says 172.21.15.44:46288) 2024-04-02T13:21:26.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4660c0b640 1 -- 172.21.15.44:0/2486886535 learned_addr learned my addr 172.21.15.44:0/2486886535 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:26.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4661c0d640 1 --2- 172.21.15.44:0/2486886535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f465c15f040 0x7f465c168f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2823383944 0 0) 0x7f465c15fb50 con 0x7f465c06d700 2024-04-02T13:21:26.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4648003520 con 0x7f465c06d700 2024-04-02T13:21:26.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3739796540 0 0) 0x7f4648003520 con 0x7f465c06d700 2024-04-02T13:21:26.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f465c15f040 msgr2=0x7f465c168f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:26.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 --2- 172.21.15.44:0/2486886535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f465c15f040 0x7f465c168f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 >> v1:172.21.15.69:6789/0 conn(0x7f465c15e550 legacy=0x7f465c06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f465c15f590 con 0x7f465c06d700 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f464c001670 con 0x7f465c06d700 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 >> v1:172.21.15.44:6789/0 conn(0x7f465c06d700 legacy=0x7f465c15e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 --2- 172.21.15.44:0/2486886535 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46480039e0 0x7f4648003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 --2- 172.21.15.44:0/2486886535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f465c15f040 0x7f4648004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 --2- 172.21.15.44:0/2486886535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4648004d40 0x7f46480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f465c06ca90 con 0x7f46480039e0 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f464800bb30 con 0x7f4648004d40 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f464800bd30 con 0x7f465c15f040 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f466140c640 1 --2- 172.21.15.44:0/2486886535 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46480039e0 0x7f4648003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4660c0b640 1 --2- 172.21.15.44:0/2486886535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f465c15f040 0x7f4648004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.791+0000 7f4661c0d640 1 --2- 172.21.15.44:0/2486886535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4648004d40 0x7f46480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.792+0000 7f466140c640 1 -- 172.21.15.44:0/2486886535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f465c15f040 msgr2=0x7f4648004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:26.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.792+0000 7f466140c640 1 --2- 172.21.15.44:0/2486886535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f465c15f040 0x7f4648004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.792+0000 7f466140c640 1 -- 172.21.15.44:0/2486886535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4648004d40 msgr2=0x7f46480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:26.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.792+0000 7f466140c640 1 --2- 172.21.15.44:0/2486886535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4648004d40 0x7f46480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.792+0000 7f466140c640 1 -- 172.21.15.44:0/2486886535 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4648003520 con 0x7f46480039e0 2024-04-02T13:21:26.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.792+0000 7f466140c640 1 --2- 172.21.15.44:0/2486886535 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46480039e0 0x7f4648003e00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4658001720 tx=0x7f4648005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcc841d64b973e76 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:26.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.793+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4658001370 con 0x7f46480039e0 2024-04-02T13:21:26.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.793+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46580052d0 con 0x7f46480039e0 2024-04-02T13:21:26.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.793+0000 7f4653fff640 1 -- 172.21.15.44:0/2486886535 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4658003830 con 0x7f46480039e0 2024-04-02T13:21:26.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.793+0000 7f4663670640 1 -- 172.21.15.44:0/2486886535 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46480039e0 msgr2=0x7f4648003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.793+0000 7f4663670640 1 --2- 172.21.15.44:0/2486886535 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46480039e0 0x7f4648003e00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4658001720 tx=0x7f4648005830 comp rx=0 tx=0).stop 2024-04-02T13:21:26.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.793+0000 7f4661c0d640 1 -- 172.21.15.44:0/2486886535 reap_dead start 2024-04-02T13:21:26.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.793+0000 7f4663670640 1 -- 172.21.15.44:0/2486886535 shutdown_connections 2024-04-02T13:21:26.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.793+0000 7f4663670640 1 -- 172.21.15.44:0/2486886535 >> 172.21.15.44:0/2486886535 conn(0x7f465c09aa20 msgr2=0x7f465c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:26.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.793+0000 7f4663670640 1 -- 172.21.15.44:0/2486886535 shutdown_connections 2024-04-02T13:21:26.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.793+0000 7f4663670640 1 -- 172.21.15.44:0/2486886535 wait complete. 2024-04-02T13:21:26.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.794+0000 7f4663670640 1 Processor -- start 2024-04-02T13:21:26.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.794+0000 7f4663670640 1 -- start start 2024-04-02T13:21:26.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4663670640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480039e0 0x7f465c0b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4663670640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f465c155ba0 0x7f465c0caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4663670640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f465c0cafa0 0x7f465c154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4663670640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f465c0d0590 con 0x7f465c155ba0 2024-04-02T13:21:26.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4663670640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f465c0d02b0 con 0x7f46480039e0 2024-04-02T13:21:26.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4663670640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f465c0cc4b0 con 0x7f465c0cafa0 2024-04-02T13:21:26.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4660c0b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f465c155ba0 0x7f465c0caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4660c0b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f465c155ba0 0x7f465c0caa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54298/0 (socket says 172.21.15.44:54298) 2024-04-02T13:21:26.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4660c0b640 1 -- 172.21.15.44:0/3030464870 learned_addr learned my addr 172.21.15.44:0/3030464870 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:26.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f466140c640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480039e0 0x7f465c0b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4661c0d640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f465c0cafa0 0x7f465c154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4660c0b640 1 -- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f465c0cafa0 msgr2=0x7f465c154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4660c0b640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f465c0cafa0 0x7f465c154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4660c0b640 1 -- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480039e0 msgr2=0x7f465c0b77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4660c0b640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480039e0 0x7f465c0b77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.795+0000 7f4660c0b640 1 -- 172.21.15.44:0/3030464870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f465c0cba20 con 0x7f465c155ba0 2024-04-02T13:21:26.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.796+0000 7f4660c0b640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f465c155ba0 0x7f465c0caa60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f464c0011e0 tx=0x7f464c007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:26.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.796+0000 7f4651ffb640 1 -- 172.21.15.44:0/3030464870 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464c00f960 con 0x7f465c155ba0 2024-04-02T13:21:26.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.796+0000 7f4651ffb640 1 -- 172.21.15.44:0/3030464870 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f464c018070 con 0x7f465c155ba0 2024-04-02T13:21:26.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.796+0000 7f4663670640 1 -- 172.21.15.44:0/3030464870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f465c0cb460 con 0x7f465c155ba0 2024-04-02T13:21:26.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.796+0000 7f4651ffb640 1 -- 172.21.15.44:0/3030464870 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464c00fd60 con 0x7f465c155ba0 2024-04-02T13:21:26.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.796+0000 7f4663670640 1 -- 172.21.15.44:0/3030464870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f465c0b0ab0 con 0x7f465c155ba0 2024-04-02T13:21:26.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.797+0000 7f4663670640 1 -- 172.21.15.44:0/3030464870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f465c15e930 con 0x7f465c155ba0 2024-04-02T13:21:26.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.797+0000 7f4651ffb640 1 -- 172.21.15.44:0/3030464870 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f464c01d460 con 0x7f465c155ba0 2024-04-02T13:21:26.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.797+0000 7f4651ffb640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f464006c8f0 0x7f464006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:26.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.798+0000 7f4651ffb640 1 -- 172.21.15.44:0/3030464870 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f464c08be00 con 0x7f465c155ba0 2024-04-02T13:21:26.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.798+0000 7f466140c640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f464006c8f0 0x7f464006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:26.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.798+0000 7f466140c640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f464006c8f0 0x7f464006edb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4658033000 tx=0x7f4658009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:26.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.802+0000 7f4651ffb640 1 -- 172.21.15.44:0/3030464870 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f465c15e930 con 0x7f465c155ba0 2024-04-02T13:21:26.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.991+0000 7f4663670640 1 -- 172.21.15.44:0/3030464870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f465c0b2b50 con 0x7f465c155ba0 2024-04-02T13:21:26.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.992+0000 7f4651ffb640 1 -- 172.21.15.44:0/3030464870 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18899 (secure 0 0 0) 0x7f464c059850 con 0x7f465c155ba0 2024-04-02T13:21:26.995 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:26.995 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:08.061253+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:26.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.994+0000 7f4663670640 1 -- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f464006c8f0 msgr2=0x7f464006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.994+0000 7f4663670640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f464006c8f0 0x7f464006edb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4658033000 tx=0x7f4658009030 comp rx=0 tx=0).stop 2024-04-02T13:21:26.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.994+0000 7f4663670640 1 -- 172.21.15.44:0/3030464870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f465c155ba0 msgr2=0x7f465c0caa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:26.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.994+0000 7f4663670640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f465c155ba0 0x7f465c0caa60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f464c0011e0 tx=0x7f464c007720 comp rx=0 tx=0).stop 2024-04-02T13:21:26.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.994+0000 7f4663670640 1 -- 172.21.15.44:0/3030464870 shutdown_connections 2024-04-02T13:21:26.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.994+0000 7f4663670640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f464006c8f0 0x7f464006edb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.994+0000 7f4663670640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f465c0cafa0 0x7f465c154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.994+0000 7f4663670640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f465c155ba0 0x7f465c0caa60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.994+0000 7f4663670640 1 --2- 172.21.15.44:0/3030464870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480039e0 0x7f465c0b77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:26.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.994+0000 7f4663670640 1 -- 172.21.15.44:0/3030464870 >> 172.21.15.44:0/3030464870 conn(0x7f465c09aa20 msgr2=0x7f465c0c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:26.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.994+0000 7f4663670640 1 -- 172.21.15.44:0/3030464870 shutdown_connections 2024-04-02T13:21:26.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:26.994+0000 7f4663670640 1 -- 172.21.15.44:0/3030464870 wait complete. 2024-04-02T13:21:27.008 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:27.008 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:27.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.110+0000 7f71cf4c8640 1 Processor -- start 2024-04-02T13:21:27.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.110+0000 7f71cf4c8640 1 -- start start 2024-04-02T13:21:27.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.111+0000 7f71cf4c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c80721f0 0x7f71c80cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.111+0000 7f71cf4c8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f71c80cfc40 0x7f71c80cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.111+0000 7f71cf4c8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71c806c220 con 0x7f71c815e420 2024-04-02T13:21:27.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.111+0000 7f71cf4c8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f71c80d00b0 con 0x7f71c80cfc40 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.111+0000 7f71cf4c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f71c80cf3b0 con 0x7f71c80721f0 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.111+0000 7f71cd264640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f71c815e420 0x7f71c80ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46300/0 (socket says 172.21.15.44:46300) 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.111+0000 7f71cd264640 1 -- 172.21.15.44:0/1612746576 learned_addr learned my addr 172.21.15.44:0/1612746576 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.111+0000 7f71cca63640 1 --2- 172.21.15.44:0/1612746576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c80721f0 0x7f71c80cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.111+0000 7f71cda65640 1 --2- 172.21.15.44:0/1612746576 >> v2:172.21.15.69:3300/0 conn(0x7f71c80cfc40 0x7f71c80cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.111+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2741951529 0 0) 0x7f71b8001490 con 0x7f71c815e420 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.111+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71b0003520 con 0x7f71c815e420 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 434760872 0 0) 0x7f71b0003520 con 0x7f71c815e420 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c80721f0 msgr2=0x7f71c80cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 --2- 172.21.15.44:0/1612746576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c80721f0 0x7f71c80cf700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 >> v2:172.21.15.69:3300/0 conn(0x7f71c80cfc40 msgr2=0x7f71c80cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 --2- 172.21.15.44:0/1612746576 >> v2:172.21.15.69:3300/0 conn(0x7f71c80cfc40 0x7f71c80cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c815f7e0 con 0x7f71c815e420 2024-04-02T13:21:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f71b8001bc0 con 0x7f71c815e420 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 >> v1:172.21.15.44:6789/0 conn(0x7f71c815e420 legacy=0x7f71c80ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 --2- 172.21.15.44:0/1612746576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71b00039b0 0x7f71b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 --2- 172.21.15.44:0/1612746576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c80721f0 0x7f71b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 --2- 172.21.15.44:0/1612746576 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71b0004d40 0x7f71b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f71c80d00b0 con 0x7f71b0004d40 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f71c80cf3b0 con 0x7f71b00039b0 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f71b000ba40 con 0x7f71c80721f0 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71cda65640 1 --2- 172.21.15.44:0/1612746576 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71b0004d40 0x7f71b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71cd264640 1 --2- 172.21.15.44:0/1612746576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71b00039b0 0x7f71b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71cca63640 1 --2- 172.21.15.44:0/1612746576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c80721f0 0x7f71b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71cda65640 1 -- 172.21.15.44:0/1612746576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c80721f0 msgr2=0x7f71b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71cda65640 1 --2- 172.21.15.44:0/1612746576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c80721f0 0x7f71b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71cda65640 1 -- 172.21.15.44:0/1612746576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71b00039b0 msgr2=0x7f71b0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71cda65640 1 --2- 172.21.15.44:0/1612746576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71b00039b0 0x7f71b0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71cda65640 1 -- 172.21.15.44:0/1612746576 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b0003520 con 0x7f71b0004d40 2024-04-02T13:21:27.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.112+0000 7f71cd264640 1 --2- 172.21.15.44:0/1612746576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71b00039b0 0x7f71b0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:27.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.113+0000 7f71cda65640 1 --2- 172.21.15.44:0/1612746576 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71b0004d40 0x7f71b00051c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009ac0 tx=0x7f71b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f50d52aca9612b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:27.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.113+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c0039070 con 0x7f71b0004d40 2024-04-02T13:21:27.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.113+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c002c710 con 0x7f71b0004d40 2024-04-02T13:21:27.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.113+0000 7f71c7fff640 1 -- 172.21.15.44:0/1612746576 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c0009690 con 0x7f71b0004d40 2024-04-02T13:21:27.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.113+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1612746576 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71b0004d40 msgr2=0x7f71b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.113+0000 7f71cf4c8640 1 --2- 172.21.15.44:0/1612746576 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71b0004d40 0x7f71b00051c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009ac0 tx=0x7f71b0005830 comp rx=0 tx=0).stop 2024-04-02T13:21:27.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.113+0000 7f71cda65640 1 -- 172.21.15.44:0/1612746576 reap_dead start 2024-04-02T13:21:27.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.113+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1612746576 shutdown_connections 2024-04-02T13:21:27.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.113+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1612746576 >> 172.21.15.44:0/1612746576 conn(0x7f71c80b2a10 msgr2=0x7f71c80b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:27.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.114+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1612746576 shutdown_connections 2024-04-02T13:21:27.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.114+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1612746576 wait complete. 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.115+0000 7f71cf4c8640 1 Processor -- start 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.115+0000 7f71cf4c8640 1 -- start start 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.115+0000 7f71cf4c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71b0004d40 0x7f71c8156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.115+0000 7f71cf4c8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71c8156b50 0x7f71c8072840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.115+0000 7f71cf4c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c81582f0 0x7f71c8158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.115+0000 7f71cf4c8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f71c815f220 con 0x7f71b0004d40 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.115+0000 7f71cf4c8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f71c815ef40 con 0x7f71c8156b50 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.115+0000 7f71cf4c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f71c815f500 con 0x7f71c81582f0 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cd264640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71b0004d40 0x7f71c8156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cd264640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71b0004d40 0x7f71c8156170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54314/0 (socket says 172.21.15.44:54314) 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cd264640 1 -- 172.21.15.44:0/1286218437 learned_addr learned my addr 172.21.15.44:0/1286218437 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cda65640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c81582f0 0x7f71c8158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cca63640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71c8156b50 0x7f71c8072840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cd264640 1 -- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c81582f0 msgr2=0x7f71c8158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cd264640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c81582f0 0x7f71c8158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cd264640 1 -- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71c8156b50 msgr2=0x7f71c8072840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cd264640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71c8156b50 0x7f71c8072840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cd264640 1 -- 172.21.15.44:0/1286218437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c8156fd0 con 0x7f71b0004d40 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cca63640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71c8156b50 0x7f71c8072840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cda65640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c81582f0 0x7f71c8158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cd264640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71b0004d40 0x7f71c8156170 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f71b8001330 tx=0x7f71b800a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71c5ffb640 1 -- 172.21.15.44:0/1286218437 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b800ae60 con 0x7f71b0004d40 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71c5ffb640 1 -- 172.21.15.44:0/1286218437 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71b8018070 con 0x7f71b0004d40 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1286218437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71c8157250 con 0x7f71b0004d40 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71c5ffb640 1 -- 172.21.15.44:0/1286218437 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b8013620 con 0x7f71b0004d40 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.116+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1286218437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71c8157810 con 0x7f71b0004d40 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.117+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1286218437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71c80684f0 con 0x7f71b0004d40 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.118+0000 7f71c5ffb640 1 -- 172.21.15.44:0/1286218437 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f71b80137c0 con 0x7f71b0004d40 2024-04-02T13:21:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.118+0000 7f71c5ffb640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f71b406c8f0 0x7f71b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.118+0000 7f71c5ffb640 1 -- 172.21.15.44:0/1286218437 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f71b808c610 con 0x7f71b0004d40 2024-04-02T13:21:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.118+0000 7f71cca63640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f71b406c8f0 0x7f71b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.119+0000 7f71cca63640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f71b406c8f0 0x7f71b406edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f71c80cf950 tx=0x7f71bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:27.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.123+0000 7f71c5ffb640 1 -- 172.21.15.44:0/1286218437 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f71c8156fd0 con 0x7f71b0004d40 2024-04-02T13:21:27.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.306+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1286218437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f71c80b48f0 con 0x7f71b0004d40 2024-04-02T13:21:27.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.306+0000 7f71c5ffb640 1 -- 172.21.15.44:0/1286218437 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f71c806c220 con 0x7f71b0004d40 2024-04-02T13:21:27.309 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:27.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.308+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f71b406c8f0 msgr2=0x7f71b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.308+0000 7f71cf4c8640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f71b406c8f0 0x7f71b406edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f71c80cf950 tx=0x7f71bc008040 comp rx=0 tx=0).stop 2024-04-02T13:21:27.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.308+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1286218437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71b0004d40 msgr2=0x7f71c8156170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.308+0000 7f71cf4c8640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71b0004d40 0x7f71c8156170 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f71b8001330 tx=0x7f71b800a570 comp rx=0 tx=0).stop 2024-04-02T13:21:27.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.308+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1286218437 shutdown_connections 2024-04-02T13:21:27.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.308+0000 7f71cf4c8640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f71b406c8f0 0x7f71b406edb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.308+0000 7f71cf4c8640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71c81582f0 0x7f71c8158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.308+0000 7f71cf4c8640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71c8156b50 0x7f71c8072840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.308+0000 7f71cf4c8640 1 --2- 172.21.15.44:0/1286218437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71b0004d40 0x7f71c8156170 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.308+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1286218437 >> 172.21.15.44:0/1286218437 conn(0x7f71c80b2a10 msgr2=0x7f71c80b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.308+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1286218437 shutdown_connections 2024-04-02T13:21:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.308+0000 7f71cf4c8640 1 -- 172.21.15.44:0/1286218437 wait complete. 2024-04-02T13:21:27.321 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:27.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.423+0000 7f4e794d9640 1 Processor -- start 2024-04-02T13:21:27.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.423+0000 7f4e794d9640 1 -- start start 2024-04-02T13:21:27.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.424+0000 7f4e794d9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e741565d0 0x7f4e740d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.424+0000 7f4e794d9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4e740cf5d0 0x7f4e7415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.424+0000 7f4e794d9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e74160b10 con 0x7f4e740d05e0 2024-04-02T13:21:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.424+0000 7f4e794d9640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4e740cfad0 con 0x7f4e740cf5d0 2024-04-02T13:21:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.424+0000 7f4e794d9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4e7415fd50 con 0x7f4e741565d0 2024-04-02T13:21:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.424+0000 7f4e737fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4e740cf5d0 0x7f4e7415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.424+0000 7f4e737fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4e740cf5d0 0x7f4e7415c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54328/0 (socket says 172.21.15.44:54328) 2024-04-02T13:21:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.424+0000 7f4e737fe640 1 -- 172.21.15.44:0/859105580 learned_addr learned my addr 172.21.15.44:0/859105580 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.424+0000 7f4e72ffd640 1 --2- 172.21.15.44:0/859105580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e741565d0 0x7f4e740d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.424+0000 7f4e737fe640 1 -- 172.21.15.44:0/859105580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e741565d0 msgr2=0x7f4e740d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.424+0000 7f4e737fe640 1 --2- 172.21.15.44:0/859105580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e741565d0 0x7f4e740d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.424+0000 7f4e737fe640 1 -- 172.21.15.44:0/859105580 >> v1:172.21.15.69:6789/0 conn(0x7f4e740d05e0 legacy=0x7f4e740ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.424+0000 7f4e737fe640 1 -- 172.21.15.44:0/859105580 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e74160030 con 0x7f4e740cf5d0 2024-04-02T13:21:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.425+0000 7f4e737fe640 1 --2- 172.21.15.44:0/859105580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e740cf5d0 0x7f4e7415c7a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009870 tx=0x7f4e6c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8668b9bc71537b67 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.425+0000 7f4e71ffb640 1 -- 172.21.15.44:0/859105580 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0317c0 con 0x7f4e740cf5d0 2024-04-02T13:21:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.425+0000 7f4e71ffb640 1 -- 172.21.15.44:0/859105580 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e6c037070 con 0x7f4e740cf5d0 2024-04-02T13:21:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.425+0000 7f4e71ffb640 1 -- 172.21.15.44:0/859105580 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c031b80 con 0x7f4e740cf5d0 2024-04-02T13:21:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.426+0000 7f4e794d9640 1 -- 172.21.15.44:0/859105580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e740cf5d0 msgr2=0x7f4e7415c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.426+0000 7f4e794d9640 1 --2- 172.21.15.44:0/859105580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e740cf5d0 0x7f4e7415c7a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009870 tx=0x7f4e6c02b860 comp rx=0 tx=0).stop 2024-04-02T13:21:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.426+0000 7f4e794d9640 1 -- 172.21.15.44:0/859105580 shutdown_connections 2024-04-02T13:21:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.426+0000 7f4e794d9640 1 --2- 172.21.15.44:0/859105580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e740cf5d0 0x7f4e7415c7a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.426+0000 7f4e794d9640 1 --2- 172.21.15.44:0/859105580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e741565d0 0x7f4e740d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.426+0000 7f4e794d9640 1 -- 172.21.15.44:0/859105580 >> 172.21.15.44:0/859105580 conn(0x7f4e7406b920 msgr2=0x7f4e7406bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:27.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.426+0000 7f4e794d9640 1 -- 172.21.15.44:0/859105580 shutdown_connections 2024-04-02T13:21:27.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.426+0000 7f4e794d9640 1 -- 172.21.15.44:0/859105580 wait complete. 2024-04-02T13:21:27.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.427+0000 7f4e794d9640 1 Processor -- start 2024-04-02T13:21:27.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.427+0000 7f4e794d9640 1 -- start start 2024-04-02T13:21:27.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e794d9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e740cf5d0 0x7f4e7415c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e794d9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e740d05e0 0x7f4e7415f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e794d9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e741565d0 0x7f4e74141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e794d9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741569b0 con 0x7f4e740d05e0 2024-04-02T13:21:27.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e794d9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741605f0 con 0x7f4e740cf5d0 2024-04-02T13:21:27.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e794d9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4e74156b30 con 0x7f4e741565d0 2024-04-02T13:21:27.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e727fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e740d05e0 0x7f4e7415f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e727fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e740d05e0 0x7f4e7415f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54332/0 (socket says 172.21.15.44:54332) 2024-04-02T13:21:27.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e727fc640 1 -- 172.21.15.44:0/338635966 learned_addr learned my addr 172.21.15.44:0/338635966 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:27.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e72ffd640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e740cf5d0 0x7f4e7415c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e737fe640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e741565d0 0x7f4e74141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e727fc640 1 -- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e741565d0 msgr2=0x7f4e74141560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e727fc640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e741565d0 0x7f4e74141560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e727fc640 1 -- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e740cf5d0 msgr2=0x7f4e7415c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e727fc640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e740cf5d0 0x7f4e7415c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e727fc640 1 -- 172.21.15.44:0/338635966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e74142020 con 0x7f4e740d05e0 2024-04-02T13:21:27.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e72ffd640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e740cf5d0 0x7f4e7415c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:27.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e727fc640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e740d05e0 0x7f4e7415f000 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00c430 tx=0x7f4e5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:27.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e6b7fe640 1 -- 172.21.15.44:0/338635966 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c011790 con 0x7f4e740d05e0 2024-04-02T13:21:27.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.428+0000 7f4e6b7fe640 1 -- 172.21.15.44:0/338635966 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e5c017070 con 0x7f4e740d05e0 2024-04-02T13:21:27.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.429+0000 7f4e794d9640 1 -- 172.21.15.44:0/338635966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e74142300 con 0x7f4e740d05e0 2024-04-02T13:21:27.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.429+0000 7f4e6b7fe640 1 -- 172.21.15.44:0/338635966 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c011b70 con 0x7f4e740d05e0 2024-04-02T13:21:27.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.429+0000 7f4e794d9640 1 -- 172.21.15.44:0/338635966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e7415e4f0 con 0x7f4e740d05e0 2024-04-02T13:21:27.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.430+0000 7f4e794d9640 1 -- 172.21.15.44:0/338635966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e740684f0 con 0x7f4e740d05e0 2024-04-02T13:21:27.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.430+0000 7f4e6b7fe640 1 -- 172.21.15.44:0/338635966 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4e5c011d10 con 0x7f4e740d05e0 2024-04-02T13:21:27.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.430+0000 7f4e6b7fe640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e5406c8f0 0x7f4e5406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.431+0000 7f4e6b7fe640 1 -- 172.21.15.44:0/338635966 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f4e5c08bdd0 con 0x7f4e740d05e0 2024-04-02T13:21:27.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.431+0000 7f4e72ffd640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e5406c8f0 0x7f4e5406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.431+0000 7f4e72ffd640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e5406c8f0 0x7f4e5406edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4e64007330 tx=0x7f4e6400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:27.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.435+0000 7f4e6b7fe640 1 -- 172.21.15.44:0/338635966 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4e5c059de0 con 0x7f4e740d05e0 2024-04-02T13:21:27.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.617+0000 7f4e794d9640 1 -- 172.21.15.44:0/338635966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4e740ce540 con 0x7f4e740d05e0 2024-04-02T13:21:27.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.617+0000 7f4e6b7fe640 1 -- 172.21.15.44:0/338635966 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f4e5c08d0a0 con 0x7f4e740d05e0 2024-04-02T13:21:27.621 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.619+0000 7f4e794d9640 1 -- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e5406c8f0 msgr2=0x7f4e5406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.619+0000 7f4e794d9640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e5406c8f0 0x7f4e5406edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4e64007330 tx=0x7f4e6400b040 comp rx=0 tx=0).stop 2024-04-02T13:21:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.619+0000 7f4e794d9640 1 -- 172.21.15.44:0/338635966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e740d05e0 msgr2=0x7f4e7415f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.619+0000 7f4e794d9640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e740d05e0 0x7f4e7415f000 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00c430 tx=0x7f4e5c00c8f0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.619+0000 7f4e794d9640 1 -- 172.21.15.44:0/338635966 shutdown_connections 2024-04-02T13:21:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.619+0000 7f4e794d9640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e5406c8f0 0x7f4e5406edb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.619+0000 7f4e794d9640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e741565d0 0x7f4e74141560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.619+0000 7f4e794d9640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e740d05e0 0x7f4e7415f000 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.619+0000 7f4e794d9640 1 --2- 172.21.15.44:0/338635966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e740cf5d0 0x7f4e7415c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.619+0000 7f4e794d9640 1 -- 172.21.15.44:0/338635966 >> 172.21.15.44:0/338635966 conn(0x7f4e7406b920 msgr2=0x7f4e7406bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.619+0000 7f4e794d9640 1 -- 172.21.15.44:0/338635966 shutdown_connections 2024-04-02T13:21:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.619+0000 7f4e794d9640 1 -- 172.21.15.44:0/338635966 wait complete. 2024-04-02T13:21:27.632 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:27.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.734+0000 7f788f23c640 1 Processor -- start 2024-04-02T13:21:27.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.734+0000 7f788f23c640 1 -- start start 2024-04-02T13:21:27.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.735+0000 7f788f23c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f788815e470 0x7f78880b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.735+0000 7f788f23c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78880ced00 con 0x7f78880b1540 2024-04-02T13:21:27.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.735+0000 7f788f23c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78880cefe0 con 0x7f78880ce790 2024-04-02T13:21:27.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.735+0000 7f788f23c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f78880cf2c0 con 0x7f788815e470 2024-04-02T13:21:27.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.735+0000 7f7887fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f78880b1540 0x7f78880d1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46302/0 (socket says 172.21.15.44:46302) 2024-04-02T13:21:27.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.735+0000 7f7887fff640 1 -- 172.21.15.44:0/1896254700 learned_addr learned my addr 172.21.15.44:0/1896254700 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.735+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317604190 0 0) 0x7f78880ced00 con 0x7f78880b1540 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.735+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7870003520 con 0x7f78880b1540 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.735+0000 7f788cfd8640 1 --2- 172.21.15.44:0/1896254700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f788815e470 0x7f78880b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.735+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4229246434 0 0) 0x7f7870003520 con 0x7f78880b1540 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f788815e470 msgr2=0x7f78880b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f78877fe640 1 --2- 172.21.15.44:0/1896254700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f788815e470 0x7f78880b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 >> v1:172.21.15.69:6789/0 conn(0x7f78880ce790 legacy=0x7f788806d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78880cf5a0 con 0x7f78880b1540 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f787c001670 con 0x7f78880b1540 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 >> v1:172.21.15.44:6789/0 conn(0x7f78880b1540 legacy=0x7f78880d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f78877fe640 1 --2- 172.21.15.44:0/1896254700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78700039e0 0x7f7870003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f78877fe640 1 --2- 172.21.15.44:0/1896254700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7870004800 0x7f7870004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f78877fe640 1 --2- 172.21.15.44:0/1896254700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f788815e470 0x7f787000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f78880cf2c0 con 0x7f7870004800 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7870005520 con 0x7f78700039e0 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f787000bf00 con 0x7f788815e470 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f7887fff640 1 --2- 172.21.15.44:0/1896254700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7870004800 0x7f7870004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f788cfd8640 1 --2- 172.21.15.44:0/1896254700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78700039e0 0x7f7870003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f788d7d9640 1 --2- 172.21.15.44:0/1896254700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f788815e470 0x7f787000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f7887fff640 1 -- 172.21.15.44:0/1896254700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f788815e470 msgr2=0x7f787000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f7887fff640 1 --2- 172.21.15.44:0/1896254700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f788815e470 0x7f787000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f7887fff640 1 -- 172.21.15.44:0/1896254700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78700039e0 msgr2=0x7f7870003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f7887fff640 1 --2- 172.21.15.44:0/1896254700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78700039e0 0x7f7870003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f7887fff640 1 -- 172.21.15.44:0/1896254700 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7870003520 con 0x7f7870004800 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.736+0000 7f788d7d9640 1 --2- 172.21.15.44:0/1896254700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f788815e470 0x7f787000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.737+0000 7f7887fff640 1 --2- 172.21.15.44:0/1896254700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7870004800 0x7f7870004c80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f78880b1960 tx=0x7f7870005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e79949a26317c14 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.737+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c007d30 con 0x7f7870004800 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.737+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f787c030070 con 0x7f7870004800 2024-04-02T13:21:27.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.737+0000 7f78877fe640 1 -- 172.21.15.44:0/1896254700 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c005d30 con 0x7f7870004800 2024-04-02T13:21:27.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.737+0000 7f788f23c640 1 -- 172.21.15.44:0/1896254700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7870004800 msgr2=0x7f7870004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.737+0000 7f788f23c640 1 --2- 172.21.15.44:0/1896254700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7870004800 0x7f7870004c80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f78880b1960 tx=0x7f7870005830 comp rx=0 tx=0).stop 2024-04-02T13:21:27.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.738+0000 7f788d7d9640 1 -- 172.21.15.44:0/1896254700 reap_dead start 2024-04-02T13:21:27.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.738+0000 7f788f23c640 1 -- 172.21.15.44:0/1896254700 shutdown_connections 2024-04-02T13:21:27.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.738+0000 7f788f23c640 1 -- 172.21.15.44:0/1896254700 >> 172.21.15.44:0/1896254700 conn(0x7f788809aa20 msgr2=0x7f788809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:27.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.738+0000 7f788f23c640 1 -- 172.21.15.44:0/1896254700 shutdown_connections 2024-04-02T13:21:27.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.738+0000 7f788f23c640 1 -- 172.21.15.44:0/1896254700 wait complete. 2024-04-02T13:21:27.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.739+0000 7f788f23c640 1 Processor -- start 2024-04-02T13:21:27.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.739+0000 7f788f23c640 1 -- start start 2024-04-02T13:21:27.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788f23c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7870004800 0x7f788815a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788f23c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788815afb0 0x7f78881591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788f23c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f788806f350 0x7f788806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788f23c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f78880d22c0 con 0x7f7870004800 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788f23c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f788806cb60 con 0x7f788815afb0 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788f23c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f78880d2440 con 0x7f788806f350 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788cfd8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7870004800 0x7f788815a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788cfd8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7870004800 0x7f788815a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54338/0 (socket says 172.21.15.44:54338) 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788cfd8640 1 -- 172.21.15.44:0/99075081 learned_addr learned my addr 172.21.15.44:0/99075081 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788cfd8640 1 -- 172.21.15.44:0/99075081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f788806f350 msgr2=0x7f788806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788cfd8640 1 --2- 172.21.15.44:0/99075081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f788806f350 0x7f788806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788cfd8640 1 -- 172.21.15.44:0/99075081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788815afb0 msgr2=0x7f78881591d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f7887fff640 1 --2- 172.21.15.44:0/99075081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788815afb0 0x7f78881591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788cfd8640 1 --2- 172.21.15.44:0/99075081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788815afb0 0x7f78881591d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788cfd8640 1 -- 172.21.15.44:0/99075081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78880b7c50 con 0x7f7870004800 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788cfd8640 1 --2- 172.21.15.44:0/99075081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7870004800 0x7f788815a7e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f78800046d0 tx=0x7f7880007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:27.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f78857fa640 1 -- 172.21.15.44:0/99075081 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880015420 con 0x7f7870004800 2024-04-02T13:21:27.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f78857fa640 1 -- 172.21.15.44:0/99075081 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f788001a070 con 0x7f7870004800 2024-04-02T13:21:27.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f78857fa640 1 -- 172.21.15.44:0/99075081 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880015800 con 0x7f7870004800 2024-04-02T13:21:27.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.740+0000 7f788f23c640 1 -- 172.21.15.44:0/99075081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78880c9160 con 0x7f7870004800 2024-04-02T13:21:27.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.741+0000 7f788f23c640 1 -- 172.21.15.44:0/99075081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78880c9720 con 0x7f7870004800 2024-04-02T13:21:27.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.741+0000 7f788f23c640 1 -- 172.21.15.44:0/99075081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78880684f0 con 0x7f7870004800 2024-04-02T13:21:27.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.742+0000 7f78857fa640 1 -- 172.21.15.44:0/99075081 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f78800159a0 con 0x7f7870004800 2024-04-02T13:21:27.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.742+0000 7f78857fa640 1 --2- 172.21.15.44:0/99075081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f787406c8f0 0x7f787406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:27.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.742+0000 7f78857fa640 1 -- 172.21.15.44:0/99075081 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f788008c870 con 0x7f7870004800 2024-04-02T13:21:27.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.742+0000 7f7887fff640 1 --2- 172.21.15.44:0/99075081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f787406c8f0 0x7f787406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:27.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.743+0000 7f7887fff640 1 --2- 172.21.15.44:0/99075081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f787406c8f0 0x7f787406edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f787c034000 tx=0x7f787c009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:27.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.747+0000 7f78857fa640 1 -- 172.21.15.44:0/99075081 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f788005a910 con 0x7f7870004800 2024-04-02T13:21:27.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.930+0000 7f788f23c640 1 -- 172.21.15.44:0/99075081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f78880d2d10 con 0x7f7870004800 2024-04-02T13:21:27.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.931+0000 7f78857fa640 1 -- 172.21.15.44:0/99075081 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f788005a210 con 0x7f7870004800 2024-04-02T13:21:27.933 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.933+0000 7f788f23c640 1 -- 172.21.15.44:0/99075081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f787406c8f0 msgr2=0x7f787406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.933+0000 7f788f23c640 1 --2- 172.21.15.44:0/99075081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f787406c8f0 0x7f787406edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f787c034000 tx=0x7f787c009030 comp rx=0 tx=0).stop 2024-04-02T13:21:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.933+0000 7f788f23c640 1 -- 172.21.15.44:0/99075081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7870004800 msgr2=0x7f788815a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.933+0000 7f788f23c640 1 --2- 172.21.15.44:0/99075081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7870004800 0x7f788815a7e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f78800046d0 tx=0x7f7880007700 comp rx=0 tx=0).stop 2024-04-02T13:21:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.933+0000 7f788f23c640 1 -- 172.21.15.44:0/99075081 shutdown_connections 2024-04-02T13:21:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.933+0000 7f788f23c640 1 --2- 172.21.15.44:0/99075081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f787406c8f0 0x7f787406edb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.933+0000 7f788f23c640 1 --2- 172.21.15.44:0/99075081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f788806f350 0x7f788806f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.933+0000 7f788f23c640 1 --2- 172.21.15.44:0/99075081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788815afb0 0x7f78881591d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.933+0000 7f788f23c640 1 --2- 172.21.15.44:0/99075081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7870004800 0x7f788815a7e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.933+0000 7f788f23c640 1 -- 172.21.15.44:0/99075081 >> 172.21.15.44:0/99075081 conn(0x7f788809aa20 msgr2=0x7f788806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.933+0000 7f788f23c640 1 -- 172.21.15.44:0/99075081 shutdown_connections 2024-04-02T13:21:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:27.933+0000 7f788f23c640 1 -- 172.21.15.44:0/99075081 wait complete. 2024-04-02T13:21:27.946 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.048+0000 7f5542315640 1 Processor -- start 2024-04-02T13:21:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.048+0000 7f5542315640 1 -- start start 2024-04-02T13:21:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.049+0000 7f5542315640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f553c0cec60 0x7f553c0d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.049+0000 7f5542315640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f553c15e5b0 0x7f553c0d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.049+0000 7f5542315640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f553c0cf1f0 con 0x7f553c15e1d0 2024-04-02T13:21:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.049+0000 7f5542315640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f553c0cfe00 con 0x7f553c15e5b0 2024-04-02T13:21:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.049+0000 7f5542315640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f553c0cf4d0 con 0x7f553c0cec60 2024-04-02T13:21:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.049+0000 7f55408b2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f553c15e5b0 0x7f553c0d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.049+0000 7f55408b2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f553c15e5b0 0x7f553c0d2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54342/0 (socket says 172.21.15.44:54342) 2024-04-02T13:21:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.049+0000 7f55408b2640 1 -- 172.21.15.44:0/2765764578 learned_addr learned my addr 172.21.15.44:0/2765764578 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.049+0000 7f553b7fe640 1 --2- 172.21.15.44:0/2765764578 >> v2:172.21.15.69:3300/0 conn(0x7f553c0cec60 0x7f553c0d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.050+0000 7f55408b2640 1 -- 172.21.15.44:0/2765764578 >> v2:172.21.15.69:3300/0 conn(0x7f553c0cec60 msgr2=0x7f553c0d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.050+0000 7f55408b2640 1 --2- 172.21.15.44:0/2765764578 >> v2:172.21.15.69:3300/0 conn(0x7f553c0cec60 0x7f553c0d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.050+0000 7f55408b2640 1 -- 172.21.15.44:0/2765764578 >> v1:172.21.15.69:6789/0 conn(0x7f553c15e1d0 legacy=0x7f553c0d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.050+0000 7f55408b2640 1 -- 172.21.15.44:0/2765764578 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f553c0cf7b0 con 0x7f553c15e5b0 2024-04-02T13:21:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.050+0000 7f55408b2640 1 --2- 172.21.15.44:0/2765764578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f553c15e5b0 0x7f553c0d2c00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f553c161f20 tx=0x7f553402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0f1788a96ab8bf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.051+0000 7f553affd640 1 -- 172.21.15.44:0/2765764578 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553402ca80 con 0x7f553c15e5b0 2024-04-02T13:21:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.051+0000 7f553affd640 1 -- 172.21.15.44:0/2765764578 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f553402cc20 con 0x7f553c15e5b0 2024-04-02T13:21:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.051+0000 7f553affd640 1 -- 172.21.15.44:0/2765764578 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55340343f0 con 0x7f553c15e5b0 2024-04-02T13:21:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.051+0000 7f5542315640 1 -- 172.21.15.44:0/2765764578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f553c15e5b0 msgr2=0x7f553c0d2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.051+0000 7f5542315640 1 --2- 172.21.15.44:0/2765764578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f553c15e5b0 0x7f553c0d2c00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f553c161f20 tx=0x7f553402bee0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.051+0000 7f5542315640 1 -- 172.21.15.44:0/2765764578 shutdown_connections 2024-04-02T13:21:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.051+0000 7f5542315640 1 --2- 172.21.15.44:0/2765764578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f553c15e5b0 0x7f553c0d2c00 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.051+0000 7f5542315640 1 --2- 172.21.15.44:0/2765764578 >> v2:172.21.15.69:3300/0 conn(0x7f553c0cec60 0x7f553c0d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.051+0000 7f5542315640 1 -- 172.21.15.44:0/2765764578 >> 172.21.15.44:0/2765764578 conn(0x7f553c09d3b0 msgr2=0x7f553c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.051+0000 7f5542315640 1 -- 172.21.15.44:0/2765764578 shutdown_connections 2024-04-02T13:21:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.051+0000 7f5542315640 1 -- 172.21.15.44:0/2765764578 wait complete. 2024-04-02T13:21:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.052+0000 7f5542315640 1 Processor -- start 2024-04-02T13:21:28.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.053+0000 7f5542315640 1 -- start start 2024-04-02T13:21:28.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.053+0000 7f5542315640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f553c0cec60 0x7f553c1573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.053+0000 7f5542315640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f553c15e1d0 0x7f553c15e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.053+0000 7f5542315640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f553c15e5b0 0x7f553c1533b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.053+0000 7f5542315640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f553c0b4e60 con 0x7f553c15e1d0 2024-04-02T13:21:28.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.053+0000 7f5542315640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f553c0b4ce0 con 0x7f553c15e5b0 2024-04-02T13:21:28.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.053+0000 7f5542315640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f553c0b4fe0 con 0x7f553c0cec60 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.053+0000 7f553b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f553c15e1d0 0x7f553c15e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.053+0000 7f553b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f553c15e1d0 0x7f553c15e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54350/0 (socket says 172.21.15.44:54350) 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.053+0000 7f553b7fe640 1 -- 172.21.15.44:0/1688057130 learned_addr learned my addr 172.21.15.44:0/1688057130 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.053+0000 7f55408b2640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f553c15e5b0 0x7f553c1533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.053+0000 7f553bfff640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f553c0cec60 0x7f553c1573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.054+0000 7f553b7fe640 1 -- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f553c0cec60 msgr2=0x7f553c1573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.054+0000 7f553b7fe640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f553c0cec60 0x7f553c1573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.054+0000 7f553b7fe640 1 -- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f553c15e5b0 msgr2=0x7f553c1533b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.054+0000 7f553b7fe640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f553c15e5b0 0x7f553c1533b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.054+0000 7f553b7fe640 1 -- 172.21.15.44:0/1688057130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f553c15ef00 con 0x7f553c15e1d0 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.054+0000 7f55408b2640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f553c15e5b0 0x7f553c1533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.054+0000 7f553b7fe640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f553c15e1d0 0x7f553c15e9c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f552400e9e0 tx=0x7f552400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.054+0000 7f5538ff9640 1 -- 172.21.15.44:0/1688057130 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524017070 con 0x7f553c15e1d0 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.054+0000 7f5538ff9640 1 -- 172.21.15.44:0/1688057130 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f552400c9d0 con 0x7f553c15e1d0 2024-04-02T13:21:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.054+0000 7f5542315640 1 -- 172.21.15.44:0/1688057130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f553c15f1a0 con 0x7f553c15e1d0 2024-04-02T13:21:28.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.054+0000 7f5538ff9640 1 -- 172.21.15.44:0/1688057130 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552400cdc0 con 0x7f553c15e1d0 2024-04-02T13:21:28.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.054+0000 7f5542315640 1 -- 172.21.15.44:0/1688057130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f553c15f760 con 0x7f553c15e1d0 2024-04-02T13:21:28.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.055+0000 7f5542315640 1 -- 172.21.15.44:0/1688057130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f553c0684f0 con 0x7f553c15e1d0 2024-04-02T13:21:28.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.056+0000 7f5538ff9640 1 -- 172.21.15.44:0/1688057130 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f552401a070 con 0x7f553c15e1d0 2024-04-02T13:21:28.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.056+0000 7f5538ff9640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f551c06c8f0 0x7f551c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.056+0000 7f5538ff9640 1 -- 172.21.15.44:0/1688057130 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f552408bf70 con 0x7f553c15e1d0 2024-04-02T13:21:28.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.056+0000 7f553bfff640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f551c06c8f0 0x7f551c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.057+0000 7f553bfff640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f551c06c8f0 0x7f551c06edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f552c007fd0 tx=0x7f552c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:28.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.060+0000 7f5538ff9640 1 -- 172.21.15.44:0/1688057130 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5524059f80 con 0x7f553c15e1d0 2024-04-02T13:21:28.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.245+0000 7f5542315640 1 -- 172.21.15.44:0/1688057130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f553c0b8220 con 0x7f553c15e1d0 2024-04-02T13:21:28.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.245+0000 7f5538ff9640 1 -- 172.21.15.44:0/1688057130 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f5524059880 con 0x7f553c15e1d0 2024-04-02T13:21:28.248 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:28.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.247+0000 7f5542315640 1 -- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f551c06c8f0 msgr2=0x7f551c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.247+0000 7f5542315640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f551c06c8f0 0x7f551c06edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f552c007fd0 tx=0x7f552c009040 comp rx=0 tx=0).stop 2024-04-02T13:21:28.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.247+0000 7f5542315640 1 -- 172.21.15.44:0/1688057130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f553c15e1d0 msgr2=0x7f553c15e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.247+0000 7f5542315640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f553c15e1d0 0x7f553c15e9c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f552400e9e0 tx=0x7f552400eea0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.247+0000 7f5542315640 1 -- 172.21.15.44:0/1688057130 shutdown_connections 2024-04-02T13:21:28.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.247+0000 7f5542315640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f551c06c8f0 0x7f551c06edb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.247+0000 7f5542315640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f553c15e5b0 0x7f553c1533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.247+0000 7f5542315640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f553c15e1d0 0x7f553c15e9c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.247+0000 7f5542315640 1 --2- 172.21.15.44:0/1688057130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f553c0cec60 0x7f553c1573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.247+0000 7f5542315640 1 -- 172.21.15.44:0/1688057130 >> 172.21.15.44:0/1688057130 conn(0x7f553c09d3b0 msgr2=0x7f553c0d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:28.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.248+0000 7f5542315640 1 -- 172.21.15.44:0/1688057130 shutdown_connections 2024-04-02T13:21:28.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.248+0000 7f5542315640 1 -- 172.21.15.44:0/1688057130 wait complete. 2024-04-02T13:21:28.261 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:21:28.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.361+0000 7f299cd15640 1 Processor -- start 2024-04-02T13:21:28.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.361+0000 7f299cd15640 1 -- start start 2024-04-02T13:21:28.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299cd15640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2998159db0 0x7f299815c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299cd15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2998071e50 0x7f29980cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299cd15640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f29980ccc20 0x7f299806d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299cd15640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2998073240 con 0x7f2998159db0 2024-04-02T13:21:28.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299cd15640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f29980cd050 con 0x7f29980ccc20 2024-04-02T13:21:28.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299cd15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f29980726c0 con 0x7f2998071e50 2024-04-02T13:21:28.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299659c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2998159db0 0x7f299815c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299659c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2998159db0 0x7f299815c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54360/0 (socket says 172.21.15.44:54360) 2024-04-02T13:21:28.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299659c640 1 -- 172.21.15.44:0/199610507 learned_addr learned my addr 172.21.15.44:0/199610507 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:28.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f298ffff640 1 --2- 172.21.15.44:0/199610507 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2998071e50 0x7f29980cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f2996d9d640 1 --2- 172.21.15.44:0/199610507 >> v2:172.21.15.69:3300/0 conn(0x7f29980ccc20 0x7f299806d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299659c640 1 -- 172.21.15.44:0/199610507 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2998071e50 msgr2=0x7f29980cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:28.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299659c640 1 --2- 172.21.15.44:0/199610507 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2998071e50 0x7f29980cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299659c640 1 -- 172.21.15.44:0/199610507 >> v2:172.21.15.69:3300/0 conn(0x7f29980ccc20 msgr2=0x7f299806d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:28.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299659c640 1 --2- 172.21.15.44:0/199610507 >> v2:172.21.15.69:3300/0 conn(0x7f29980ccc20 0x7f299806d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f299659c640 1 -- 172.21.15.44:0/199610507 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29980729a0 con 0x7f2998159db0 2024-04-02T13:21:28.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.362+0000 7f2996d9d640 1 --2- 172.21.15.44:0/199610507 >> v2:172.21.15.69:3300/0 conn(0x7f29980ccc20 0x7f299806d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:28.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.363+0000 7f299659c640 1 --2- 172.21.15.44:0/199610507 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2998159db0 0x7f299815c6d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2998071120 tx=0x7f298802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd6d1071911be7a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:28.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.363+0000 7f2995d9b640 1 -- 172.21.15.44:0/199610507 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298802e070 con 0x7f2998159db0 2024-04-02T13:21:28.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.363+0000 7f2995d9b640 1 -- 172.21.15.44:0/199610507 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2988037070 con 0x7f2998159db0 2024-04-02T13:21:28.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.363+0000 7f2995d9b640 1 -- 172.21.15.44:0/199610507 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988031a00 con 0x7f2998159db0 2024-04-02T13:21:28.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.364+0000 7f299cd15640 1 -- 172.21.15.44:0/199610507 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2998159db0 msgr2=0x7f299815c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.364+0000 7f299cd15640 1 --2- 172.21.15.44:0/199610507 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2998159db0 0x7f299815c6d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2998071120 tx=0x7f298802b880 comp rx=0 tx=0).stop 2024-04-02T13:21:28.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.364+0000 7f299cd15640 1 -- 172.21.15.44:0/199610507 shutdown_connections 2024-04-02T13:21:28.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.364+0000 7f299cd15640 1 --2- 172.21.15.44:0/199610507 >> v2:172.21.15.69:3300/0 conn(0x7f29980ccc20 0x7f299806d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.364+0000 7f299cd15640 1 --2- 172.21.15.44:0/199610507 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2998071e50 0x7f29980cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.364+0000 7f299cd15640 1 --2- 172.21.15.44:0/199610507 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2998159db0 0x7f299815c6d0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.364+0000 7f299cd15640 1 -- 172.21.15.44:0/199610507 >> 172.21.15.44:0/199610507 conn(0x7f29980cfcd0 msgr2=0x7f29980d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:28.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.364+0000 7f299cd15640 1 -- 172.21.15.44:0/199610507 shutdown_connections 2024-04-02T13:21:28.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.364+0000 7f299cd15640 1 -- 172.21.15.44:0/199610507 wait complete. 2024-04-02T13:21:28.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.365+0000 7f299cd15640 1 Processor -- start 2024-04-02T13:21:28.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.365+0000 7f299cd15640 1 -- start start 2024-04-02T13:21:28.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299cd15640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2998071e50 0x7f2998158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299cd15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29980ccc20 0x7f299814b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299cd15640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2998159db0 0x7f29981f5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299cd15640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f29980722d0 con 0x7f2998071e50 2024-04-02T13:21:28.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299cd15640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2998072f60 con 0x7f2998159db0 2024-04-02T13:21:28.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299cd15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2998072450 con 0x7f29980ccc20 2024-04-02T13:21:28.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2998071e50 0x7f2998158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2998071e50 0x7f2998158c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54374/0 (socket says 172.21.15.44:54374) 2024-04-02T13:21:28.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299659c640 1 -- 172.21.15.44:0/1454323870 learned_addr learned my addr 172.21.15.44:0/1454323870 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:28.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f298ffff640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29980ccc20 0x7f299814b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f2996d9d640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2998159db0 0x7f29981f5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299659c640 1 -- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29980ccc20 msgr2=0x7f299814b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299659c640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29980ccc20 0x7f299814b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299659c640 1 -- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2998159db0 msgr2=0x7f29981f5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299659c640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2998159db0 0x7f29981f5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299659c640 1 -- 172.21.15.44:0/1454323870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299809bf50 con 0x7f2998071e50 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f2996d9d640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2998159db0 0x7f29981f5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.366+0000 7f299659c640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2998071e50 0x7f2998158c00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f298802b8c0 tx=0x7f29880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.367+0000 7f298effd640 1 -- 172.21.15.44:0/1454323870 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988032880 con 0x7f2998071e50 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.367+0000 7f298effd640 1 -- 172.21.15.44:0/1454323870 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2988032a20 con 0x7f2998071e50 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.367+0000 7f299cd15640 1 -- 172.21.15.44:0/1454323870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299809c230 con 0x7f2998071e50 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.367+0000 7f298effd640 1 -- 172.21.15.44:0/1454323870 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988032e10 con 0x7f2998071e50 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.367+0000 7f299cd15640 1 -- 172.21.15.44:0/1454323870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2998155870 con 0x7f2998071e50 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.368+0000 7f299cd15640 1 -- 172.21.15.44:0/1454323870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2998074f20 con 0x7f2998071e50 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.368+0000 7f298effd640 1 -- 172.21.15.44:0/1454323870 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f298804c060 con 0x7f2998071e50 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.368+0000 7f298effd640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f297806c920 0x7f297806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.369+0000 7f298effd640 1 -- 172.21.15.44:0/1454323870 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f29880adba0 con 0x7f2998071e50 2024-04-02T13:21:28.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.369+0000 7f298ffff640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f297806c920 0x7f297806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.369+0000 7f298ffff640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f297806c920 0x7f297806ede0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2980006fd0 tx=0x7f2980008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:28.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.373+0000 7f298effd640 1 -- 172.21.15.44:0/1454323870 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2988040070 con 0x7f2998071e50 2024-04-02T13:21:28.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.568+0000 7f299cd15640 1 -- 172.21.15.44:0/1454323870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f29981f53f0 con 0x7f2998071e50 2024-04-02T13:21:28.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.569+0000 7f298effd640 1 -- 172.21.15.44:0/1454323870 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f298807bba0 con 0x7f2998071e50 2024-04-02T13:21:28.571 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:21:28.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.570+0000 7f299cd15640 1 -- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f297806c920 msgr2=0x7f297806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.570+0000 7f299cd15640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f297806c920 0x7f297806ede0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2980006fd0 tx=0x7f2980008040 comp rx=0 tx=0).stop 2024-04-02T13:21:28.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.570+0000 7f299cd15640 1 -- 172.21.15.44:0/1454323870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2998071e50 msgr2=0x7f2998158c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.570+0000 7f299cd15640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2998071e50 0x7f2998158c00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f298802b8c0 tx=0x7f29880046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.570+0000 7f299cd15640 1 -- 172.21.15.44:0/1454323870 shutdown_connections 2024-04-02T13:21:28.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.570+0000 7f299cd15640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f297806c920 0x7f297806ede0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.570+0000 7f299cd15640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2998159db0 0x7f29981f5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.570+0000 7f299cd15640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29980ccc20 0x7f299814b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.570+0000 7f299cd15640 1 --2- 172.21.15.44:0/1454323870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2998071e50 0x7f2998158c00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.570+0000 7f299cd15640 1 -- 172.21.15.44:0/1454323870 >> 172.21.15.44:0/1454323870 conn(0x7f29980cfcd0 msgr2=0x7f29980cf210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:28.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.570+0000 7f299cd15640 1 -- 172.21.15.44:0/1454323870 shutdown_connections 2024-04-02T13:21:28.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.570+0000 7f299cd15640 1 -- 172.21.15.44:0/1454323870 wait complete. 2024-04-02T13:21:28.583 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-02T13:21:28.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:28.584 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-02T13:21:28.617 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:21:28.617 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:21:28.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:28.617 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-02T13:21:28.676 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T13:21:28.676 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:21:28.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.776+0000 7fa2eb15f640 1 Processor -- start 2024-04-02T13:21:28.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.776+0000 7fa2eb15f640 1 -- start start 2024-04-02T13:21:28.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.776+0000 7fa2eb15f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40d0c50 0x7fa2e40d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.776+0000 7fa2eb15f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e40cc600 con 0x7fa2e415e4a0 2024-04-02T13:21:28.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.776+0000 7fa2eb15f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e40cc8e0 con 0x7fa2e406ce40 2024-04-02T13:21:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.776+0000 7fa2eb15f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e40af300 con 0x7fa2e40d0c50 2024-04-02T13:21:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e8efb640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa2e415e4a0 0x7fa2e40aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46306/0 (socket says 172.21.15.44:46306) 2024-04-02T13:21:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e8efb640 1 -- 172.21.15.44:0/3552081781 learned_addr learned my addr 172.21.15.44:0/3552081781 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e3fff640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40d0c50 0x7fa2e40d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 369670825 0 0) 0x7fa2e40cc600 con 0x7fa2e415e4a0 2024-04-02T13:21:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2d4003520 con 0x7fa2e415e4a0 2024-04-02T13:21:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 746106935 0 0) 0x7fa2e40cc8e0 con 0x7fa2e406ce40 2024-04-02T13:21:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2e40cc600 con 0x7fa2e406ce40 2024-04-02T13:21:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3351056787 0 0) 0x7fa2d4003520 con 0x7fa2e415e4a0 2024-04-02T13:21:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40d0c50 msgr2=0x7fa2e40d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40d0c50 0x7fa2e40d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 >> v1:172.21.15.69:6789/0 conn(0x7fa2e406ce40 legacy=0x7fa2e40cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e40ccbc0 con 0x7fa2e415e4a0 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fa2d8001670 con 0x7fa2e415e4a0 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 >> v1:172.21.15.44:6789/0 conn(0x7fa2e415e4a0 legacy=0x7fa2e40aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2d4003b70 0x7fa2d4003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2d4004800 0x7fa2d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40d0c50 0x7fa2d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e40af300 con 0x7fa2d4004800 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d400ba70 con 0x7fa2d4003b70 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa2d400bc70 con 0x7fa2e40d0c50 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e3fff640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2d4004800 0x7fa2d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e8efb640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2d4003b70 0x7fa2d4003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.777+0000 7fa2e96fc640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40d0c50 0x7fa2d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.778+0000 7fa2e3fff640 1 -- 172.21.15.44:0/3552081781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40d0c50 msgr2=0x7fa2d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.778+0000 7fa2e3fff640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40d0c50 0x7fa2d40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.778+0000 7fa2e3fff640 1 -- 172.21.15.44:0/3552081781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2d4003b70 msgr2=0x7fa2d4003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.778+0000 7fa2e3fff640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2d4003b70 0x7fa2d4003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.778+0000 7fa2e3fff640 1 -- 172.21.15.44:0/3552081781 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d4003520 con 0x7fa2d4004800 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.778+0000 7fa2e8efb640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2d4003b70 0x7fa2d4003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.778+0000 7fa2e96fc640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40d0c50 0x7fa2d40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.778+0000 7fa2e3fff640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2d4004800 0x7fa2d4004c80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009a90 tx=0x7fa2d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56df5319644f248 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.778+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0039070 con 0x7fa2d4004800 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.778+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d002c680 con 0x7fa2d4004800 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.778+0000 7fa2e37fe640 1 -- 172.21.15.44:0/3552081781 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0009690 con 0x7fa2d4004800 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.779+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3552081781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2d4004800 msgr2=0x7fa2d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.779+0000 7fa2eb15f640 1 --2- 172.21.15.44:0/3552081781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2d4004800 0x7fa2d4004c80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009a90 tx=0x7fa2d4005830 comp rx=0 tx=0).stop 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.779+0000 7fa2e96fc640 1 -- 172.21.15.44:0/3552081781 reap_dead start 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.779+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3552081781 shutdown_connections 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.779+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3552081781 >> 172.21.15.44:0/3552081781 conn(0x7fa2e409aa20 msgr2=0x7fa2e409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:28.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.779+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3552081781 shutdown_connections 2024-04-02T13:21:28.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.779+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3552081781 wait complete. 2024-04-02T13:21:28.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.780+0000 7fa2eb15f640 1 Processor -- start 2024-04-02T13:21:28.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.780+0000 7fa2eb15f640 1 -- start start 2024-04-02T13:21:28.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2eb15f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2d4004800 0x7fa2e41605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2eb15f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2e415ea60 0x7fa2e415a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2eb15f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40b3350 0x7fa2e40b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2eb15f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e40cd460 con 0x7fa2e415ea60 2024-04-02T13:21:28.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2eb15f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e40cd180 con 0x7fa2d4004800 2024-04-02T13:21:28.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2eb15f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e415e880 con 0x7fa2e40b3350 2024-04-02T13:21:28.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2e3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2e415ea60 0x7fa2e415a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2e3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2e415ea60 0x7fa2e415a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54390/0 (socket says 172.21.15.44:54390) 2024-04-02T13:21:28.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2e3fff640 1 -- 172.21.15.44:0/3233954252 learned_addr learned my addr 172.21.15.44:0/3233954252 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2e96fc640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40b3350 0x7fa2e40b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2e8efb640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2d4004800 0x7fa2e41605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2e3fff640 1 -- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40b3350 msgr2=0x7fa2e40b37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2e3fff640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40b3350 0x7fa2e40b37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2e3fff640 1 -- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2d4004800 msgr2=0x7fa2e41605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2e3fff640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2d4004800 0x7fa2e41605a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.781+0000 7fa2e3fff640 1 -- 172.21.15.44:0/3233954252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e415ae10 con 0x7fa2e415ea60 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.782+0000 7fa2e3fff640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2e415ea60 0x7fa2e415a8d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009bb0 tx=0x7fa2d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.782+0000 7fa2e17fa640 1 -- 172.21.15.44:0/3233954252 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0039050 con 0x7fa2e415ea60 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.782+0000 7fa2e17fa640 1 -- 172.21.15.44:0/3233954252 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d0045070 con 0x7fa2e415ea60 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.782+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3233954252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e415f1d0 con 0x7fa2e415ea60 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.782+0000 7fa2e17fa640 1 -- 172.21.15.44:0/3233954252 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0034a70 con 0x7fa2e415ea60 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.782+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3233954252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e415f790 con 0x7fa2e415ea60 2024-04-02T13:21:28.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.783+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3233954252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2e415f990 con 0x7fa2e415ea60 2024-04-02T13:21:28.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.783+0000 7fa2e17fa640 1 -- 172.21.15.44:0/3233954252 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa2d0034c10 con 0x7fa2e415ea60 2024-04-02T13:21:28.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.784+0000 7fa2e17fa640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa2c806c8f0 0x7fa2c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:28.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.784+0000 7fa2e17fa640 1 -- 172.21.15.44:0/3233954252 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fa2d00ae030 con 0x7fa2e415ea60 2024-04-02T13:21:28.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.784+0000 7fa2e8efb640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa2c806c8f0 0x7fa2c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:28.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.784+0000 7fa2e8efb640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa2c806c8f0 0x7fa2c806edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8001eb0 tx=0x7fa2d8001d20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:28.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:28.788+0000 7fa2e17fa640 1 -- 172.21.15.44:0/3233954252 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa2e415f990 con 0x7fa2e415ea60 2024-04-02T13:21:29.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.016+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3233954252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2e40d0090 con 0x7fa2e415ea60 2024-04-02T13:21:29.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.017+0000 7fa2e17fa640 1 -- 172.21.15.44:0/3233954252 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7fa2e40d0090 con 0x7fa2e415ea60 2024-04-02T13:21:29.020 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:29.020 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4040417710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4040417710},{"type":"v1","addr":"172.21.15.69:6836","nonce":4040417710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2772068060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2772068060},{"type":"v1","addr":"172.21.15.44:6833","nonce":2772068060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1985405150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1985405150},{"type":"v1","addr":"172.21.15.69:6837","nonce":1985405150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:20:20.247358+0000","modified":"2024-04-02T13:20:42.332570+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/2282156931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2282156931},{"type":"v1","addr":"172.21.15.44:6835","nonce":2282156931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-04-02T13:21:29.020 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 9 2024-04-02T13:21:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.019+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa2c806c8f0 msgr2=0x7fa2c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.019+0000 7fa2eb15f640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa2c806c8f0 0x7fa2c806edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8001eb0 tx=0x7fa2d8001d20 comp rx=0 tx=0).stop 2024-04-02T13:21:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.019+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3233954252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2e415ea60 msgr2=0x7fa2e415a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.019+0000 7fa2eb15f640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2e415ea60 0x7fa2e415a8d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009bb0 tx=0x7fa2d0004730 comp rx=0 tx=0).stop 2024-04-02T13:21:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.019+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3233954252 shutdown_connections 2024-04-02T13:21:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.019+0000 7fa2eb15f640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa2c806c8f0 0x7fa2c806edb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.019+0000 7fa2eb15f640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2e40b3350 0x7fa2e40b37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.019+0000 7fa2eb15f640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2e415ea60 0x7fa2e415a8d0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.019+0000 7fa2eb15f640 1 --2- 172.21.15.44:0/3233954252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2d4004800 0x7fa2e41605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.019+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3233954252 >> 172.21.15.44:0/3233954252 conn(0x7fa2e409aa20 msgr2=0x7fa2e40704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:29.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.019+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3233954252 shutdown_connections 2024-04-02T13:21:29.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.019+0000 7fa2eb15f640 1 -- 172.21.15.44:0/3233954252 wait complete. 2024-04-02T13:21:29.033 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:29.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.132+0000 7fce7c7f0640 1 Processor -- start 2024-04-02T13:21:29.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.132+0000 7fce7c7f0640 1 -- start start 2024-04-02T13:21:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.133+0000 7fce7c7f0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fce740d0a30 0x7fce7406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.133+0000 7fce7c7f0640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce740d0000 con 0x7fce740d2040 2024-04-02T13:21:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.133+0000 7fce7c7f0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce7406c1d0 con 0x7fce740d0e10 2024-04-02T13:21:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.133+0000 7fce7c7f0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fce7406cb00 con 0x7fce740d0a30 2024-04-02T13:21:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.133+0000 7fce7ad8d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fce740d2040 0x7fce740d24f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46314/0 (socket says 172.21.15.44:46314) 2024-04-02T13:21:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.133+0000 7fce7ad8d640 1 -- 172.21.15.44:0/1610016657 learned_addr learned my addr 172.21.15.44:0/1610016657 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.133+0000 7fce7a58c640 1 --2- 172.21.15.44:0/1610016657 >> v2:172.21.15.69:3300/0 conn(0x7fce740d0a30 0x7fce7406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.133+0000 7fce7a58c640 1 -- 172.21.15.44:0/1610016657 >> v1:172.21.15.69:6789/0 conn(0x7fce740d0e10 legacy=0x7fce740cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.133+0000 7fce7a58c640 1 -- 172.21.15.44:0/1610016657 >> v1:172.21.15.44:6789/0 conn(0x7fce740d2040 legacy=0x7fce740d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.133+0000 7fce7a58c640 1 -- 172.21.15.44:0/1610016657 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7406c4b0 con 0x7fce740d0a30 2024-04-02T13:21:29.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.134+0000 7fce7a58c640 1 --2- 172.21.15.44:0/1610016657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce740d0a30 0x7fce7406cf90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fce700097a0 tx=0x7fce7002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a55da37237b74bc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:29.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.136+0000 7fce7958a640 1 -- 172.21.15.44:0/1610016657 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce70035020 con 0x7fce740d0a30 2024-04-02T13:21:29.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.136+0000 7fce7958a640 1 -- 172.21.15.44:0/1610016657 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce70031830 con 0x7fce740d0a30 2024-04-02T13:21:29.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.136+0000 7fce7958a640 1 -- 172.21.15.44:0/1610016657 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce70031c00 con 0x7fce740d0a30 2024-04-02T13:21:29.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.136+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1610016657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce740d0a30 msgr2=0x7fce7406cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.136+0000 7fce7c7f0640 1 --2- 172.21.15.44:0/1610016657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce740d0a30 0x7fce7406cf90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fce700097a0 tx=0x7fce7002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.136+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1610016657 shutdown_connections 2024-04-02T13:21:29.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.136+0000 7fce7c7f0640 1 --2- 172.21.15.44:0/1610016657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce740d0a30 0x7fce7406cf90 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.136+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1610016657 >> 172.21.15.44:0/1610016657 conn(0x7fce7409d470 msgr2=0x7fce740d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:29.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.137+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1610016657 shutdown_connections 2024-04-02T13:21:29.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.137+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1610016657 wait complete. 2024-04-02T13:21:29.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.138+0000 7fce7c7f0640 1 Processor -- start 2024-04-02T13:21:29.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.138+0000 7fce7c7f0640 1 -- start start 2024-04-02T13:21:29.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.138+0000 7fce7c7f0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce740d0a30 0x7fce7415f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.138+0000 7fce7c7f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce740d0e10 0x7fce740b6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.138+0000 7fce7c7f0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce740d2040 0x7fce740b6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.138+0000 7fce7c7f0640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fce740d2c00 con 0x7fce740d0e10 2024-04-02T13:21:29.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.138+0000 7fce7c7f0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fce740d1270 con 0x7fce740d0a30 2024-04-02T13:21:29.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.138+0000 7fce7c7f0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fce7415e6f0 con 0x7fce740d2040 2024-04-02T13:21:29.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.138+0000 7fce79d8b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce740d0e10 0x7fce740b6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce79d8b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce740d0e10 0x7fce740b6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54394/0 (socket says 172.21.15.44:54394) 2024-04-02T13:21:29.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce79d8b640 1 -- 172.21.15.44:0/1160063711 learned_addr learned my addr 172.21.15.44:0/1160063711 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:29.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce7ad8d640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce740d2040 0x7fce740b6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce7a58c640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce740d0a30 0x7fce7415f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce79d8b640 1 -- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce740d2040 msgr2=0x7fce740b6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce79d8b640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce740d2040 0x7fce740b6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce79d8b640 1 -- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce740d0a30 msgr2=0x7fce7415f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce79d8b640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce740d0a30 0x7fce7415f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce79d8b640 1 -- 172.21.15.44:0/1160063711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce740b4730 con 0x7fce740d0e10 2024-04-02T13:21:29.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce7a58c640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce740d0a30 0x7fce7415f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:29.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce79d8b640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce740d0e10 0x7fce740b6d30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fce6400c440 tx=0x7fce6400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:29.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce6affd640 1 -- 172.21.15.44:0/1160063711 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce640052d0 con 0x7fce740d0e10 2024-04-02T13:21:29.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce6affd640 1 -- 172.21.15.44:0/1160063711 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce64005470 con 0x7fce740d0e10 2024-04-02T13:21:29.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce6affd640 1 -- 172.21.15.44:0/1160063711 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64007260 con 0x7fce740d0e10 2024-04-02T13:21:29.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1160063711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce740b4a10 con 0x7fce740d0e10 2024-04-02T13:21:29.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.139+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1160063711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce740b67d0 con 0x7fce740d0e10 2024-04-02T13:21:29.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.140+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1160063711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce740684f0 con 0x7fce740d0e10 2024-04-02T13:21:29.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.141+0000 7fce6affd640 1 -- 172.21.15.44:0/1160063711 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fce640074c0 con 0x7fce740d0e10 2024-04-02T13:21:29.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.141+0000 7fce6affd640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce5806c8f0 0x7fce5806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.141+0000 7fce6affd640 1 -- 172.21.15.44:0/1160063711 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fce6408b2a0 con 0x7fce740d0e10 2024-04-02T13:21:29.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.141+0000 7fce7a58c640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce5806c8f0 0x7fce5806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.142+0000 7fce7a58c640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce5806c8f0 0x7fce5806edb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fce70036000 tx=0x7fce7003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:29.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.145+0000 7fce6affd640 1 -- 172.21.15.44:0/1160063711 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fce64058cf0 con 0x7fce740d0e10 2024-04-02T13:21:29.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.334+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1160063711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce740c77d0 con 0x7fce740d0e10 2024-04-02T13:21:29.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.335+0000 7fce6affd640 1 -- 172.21.15.44:0/1160063711 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18899 (secure 0 0 0) 0x7fce6408b0c0 con 0x7fce740d0e10 2024-04-02T13:21:29.338 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:29.338 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:08.061253+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:29.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.337+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce5806c8f0 msgr2=0x7fce5806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.337+0000 7fce7c7f0640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce5806c8f0 0x7fce5806edb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fce70036000 tx=0x7fce7003c040 comp rx=0 tx=0).stop 2024-04-02T13:21:29.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.337+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1160063711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce740d0e10 msgr2=0x7fce740b6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.337+0000 7fce7c7f0640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce740d0e10 0x7fce740b6d30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fce6400c440 tx=0x7fce6400c900 comp rx=0 tx=0).stop 2024-04-02T13:21:29.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.337+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1160063711 shutdown_connections 2024-04-02T13:21:29.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.337+0000 7fce7c7f0640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce5806c8f0 0x7fce5806edb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.337+0000 7fce7c7f0640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce740d2040 0x7fce740b6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.337+0000 7fce7c7f0640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce740d0e10 0x7fce740b6d30 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.337+0000 7fce7c7f0640 1 --2- 172.21.15.44:0/1160063711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce740d0a30 0x7fce7415f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.337+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1160063711 >> 172.21.15.44:0/1160063711 conn(0x7fce7409d470 msgr2=0x7fce7415d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:29.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.337+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1160063711 shutdown_connections 2024-04-02T13:21:29.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.337+0000 7fce7c7f0640 1 -- 172.21.15.44:0/1160063711 wait complete. 2024-04-02T13:21:29.351 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:20:18.247349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:20:19.197936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-02T13:20:22.905008+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-02T13:21:29.351 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:29.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.453+0000 7f5314c2c640 1 Processor -- start 2024-04-02T13:21:29.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.453+0000 7f5314c2c640 1 -- start start 2024-04-02T13:21:29.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.454+0000 7f5314c2c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f53100cec60 0x7f53100d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.454+0000 7f5314c2c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53100cf1f0 con 0x7f531015e1d0 2024-04-02T13:21:29.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.454+0000 7f5314c2c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53100cf4d0 con 0x7f531015e5b0 2024-04-02T13:21:29.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.454+0000 7f5314c2c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f53100cfe00 con 0x7f53100cec60 2024-04-02T13:21:29.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.454+0000 7f530dd9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f53100cec60 0x7f53100d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.454+0000 7f530e59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f531015e1d0 0x7f53100d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46328/0 (socket says 172.21.15.44:46328) 2024-04-02T13:21:29.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.454+0000 7f530e59c640 1 -- 172.21.15.44:0/549576280 learned_addr learned my addr 172.21.15.44:0/549576280 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:29.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.454+0000 7f530dd9b640 1 -- 172.21.15.44:0/549576280 >> v1:172.21.15.69:6789/0 conn(0x7f531015e5b0 legacy=0x7f53100b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:29.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.454+0000 7f530dd9b640 1 -- 172.21.15.44:0/549576280 >> v1:172.21.15.44:6789/0 conn(0x7f531015e1d0 legacy=0x7f53100d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.454+0000 7f530dd9b640 1 -- 172.21.15.44:0/549576280 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53100cf7b0 con 0x7f53100cec60 2024-04-02T13:21:29.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.454+0000 7f530d59a640 1 -- 172.21.15.44:0/549576280 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3330605037 0 0) 0x7f53100cf1f0 con 0x7f531015e1d0 2024-04-02T13:21:29.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.455+0000 7f530dd9b640 1 --2- 172.21.15.44:0/549576280 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53100cec60 0x7f53100d26c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f53040097a0 tx=0x7f530402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c8206f6c747dc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:29.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.455+0000 7f530d59a640 1 -- 172.21.15.44:0/549576280 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5304035020 con 0x7f53100cec60 2024-04-02T13:21:29.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.455+0000 7f530d59a640 1 -- 172.21.15.44:0/549576280 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5304031830 con 0x7f53100cec60 2024-04-02T13:21:29.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.455+0000 7f530d59a640 1 -- 172.21.15.44:0/549576280 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5304031c00 con 0x7f53100cec60 2024-04-02T13:21:29.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.456+0000 7f5314c2c640 1 -- 172.21.15.44:0/549576280 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53100cec60 msgr2=0x7f53100d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.456+0000 7f5314c2c640 1 --2- 172.21.15.44:0/549576280 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53100cec60 0x7f53100d26c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f53040097a0 tx=0x7f530402b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.456+0000 7f5314c2c640 1 -- 172.21.15.44:0/549576280 shutdown_connections 2024-04-02T13:21:29.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.456+0000 7f5314c2c640 1 --2- 172.21.15.44:0/549576280 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53100cec60 0x7f53100d26c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.456+0000 7f5314c2c640 1 -- 172.21.15.44:0/549576280 >> 172.21.15.44:0/549576280 conn(0x7f531009d3b0 msgr2=0x7f53100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:29.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.456+0000 7f5314c2c640 1 -- 172.21.15.44:0/549576280 shutdown_connections 2024-04-02T13:21:29.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.456+0000 7f5314c2c640 1 -- 172.21.15.44:0/549576280 wait complete. 2024-04-02T13:21:29.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.457+0000 7f5314c2c640 1 Processor -- start 2024-04-02T13:21:29.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.457+0000 7f5314c2c640 1 -- start start 2024-04-02T13:21:29.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f5314c2c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53100cec60 0x7f53100cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f5314c2c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f531015e1d0 0x7f53100cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f5314c2c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f531015e5b0 0x7f531009fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f5314c2c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f53100b8150 con 0x7f531015e1d0 2024-04-02T13:21:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f5314c2c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f53100b7fd0 con 0x7f531015e5b0 2024-04-02T13:21:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f5314c2c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f53100b82d0 con 0x7f53100cec60 2024-04-02T13:21:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f530dd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f531015e1d0 0x7f53100cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f530dd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f531015e1d0 0x7f53100cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54402/0 (socket says 172.21.15.44:54402) 2024-04-02T13:21:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f530dd9b640 1 -- 172.21.15.44:0/2930332650 learned_addr learned my addr 172.21.15.44:0/2930332650 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f530e59c640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53100cec60 0x7f53100cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f530ed9d640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f531015e5b0 0x7f531009fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f530dd9b640 1 -- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53100cec60 msgr2=0x7f53100cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f530dd9b640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53100cec60 0x7f53100cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f530dd9b640 1 -- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f531015e5b0 msgr2=0x7f531009fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f530dd9b640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f531015e5b0 0x7f531009fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f530dd9b640 1 -- 172.21.15.44:0/2930332650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53101565a0 con 0x7f531015e1d0 2024-04-02T13:21:29.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f530ed9d640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f531015e5b0 0x7f531009fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:29.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f530dd9b640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f531015e1d0 0x7f53100cc870 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f53040097a0 tx=0x7f53040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:29.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f52feffd640 1 -- 172.21.15.44:0/2930332650 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5304041070 con 0x7f531015e1d0 2024-04-02T13:21:29.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f52feffd640 1 -- 172.21.15.44:0/2930332650 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5304004230 con 0x7f531015e1d0 2024-04-02T13:21:29.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.458+0000 7f5314c2c640 1 -- 172.21.15.44:0/2930332650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53100ccff0 con 0x7f531015e1d0 2024-04-02T13:21:29.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.459+0000 7f52feffd640 1 -- 172.21.15.44:0/2930332650 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5304035040 con 0x7f531015e1d0 2024-04-02T13:21:29.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.459+0000 7f5314c2c640 1 -- 172.21.15.44:0/2930332650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53100cd5b0 con 0x7f531015e1d0 2024-04-02T13:21:29.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.459+0000 7f5314c2c640 1 -- 172.21.15.44:0/2930332650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53100684f0 con 0x7f531015e1d0 2024-04-02T13:21:29.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.460+0000 7f52feffd640 1 -- 172.21.15.44:0/2930332650 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5304038020 con 0x7f531015e1d0 2024-04-02T13:21:29.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.460+0000 7f52feffd640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f52f006c920 0x7f52f006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.460+0000 7f52feffd640 1 -- 172.21.15.44:0/2930332650 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f53040adb10 con 0x7f531015e1d0 2024-04-02T13:21:29.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.460+0000 7f530e59c640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f52f006c920 0x7f52f006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.461+0000 7f530e59c640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f52f006c920 0x7f52f006ede0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5300000f60 tx=0x7f5300007d00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:29.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.465+0000 7f52feffd640 1 -- 172.21.15.44:0/2930332650 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f530407baa0 con 0x7f531015e1d0 2024-04-02T13:21:29.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.647+0000 7f5314c2c640 1 -- 172.21.15.44:0/2930332650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5310067d30 con 0x7f531015e1d0 2024-04-02T13:21:29.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.647+0000 7f52feffd640 1 -- 172.21.15.44:0/2930332650 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f530407b3a0 con 0x7f531015e1d0 2024-04-02T13:21:29.650 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.649+0000 7f5314c2c640 1 -- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f52f006c920 msgr2=0x7f52f006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.649+0000 7f5314c2c640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f52f006c920 0x7f52f006ede0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5300000f60 tx=0x7f5300007d00 comp rx=0 tx=0).stop 2024-04-02T13:21:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.649+0000 7f5314c2c640 1 -- 172.21.15.44:0/2930332650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f531015e1d0 msgr2=0x7f53100cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.649+0000 7f5314c2c640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f531015e1d0 0x7f53100cc870 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f53040097a0 tx=0x7f53040046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.649+0000 7f5314c2c640 1 -- 172.21.15.44:0/2930332650 shutdown_connections 2024-04-02T13:21:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.649+0000 7f5314c2c640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f52f006c920 0x7f52f006ede0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.649+0000 7f5314c2c640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f531015e5b0 0x7f531009fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.649+0000 7f5314c2c640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f531015e1d0 0x7f53100cc870 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.649+0000 7f5314c2c640 1 --2- 172.21.15.44:0/2930332650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53100cec60 0x7f53100cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.649+0000 7f5314c2c640 1 -- 172.21.15.44:0/2930332650 >> 172.21.15.44:0/2930332650 conn(0x7f531009d3b0 msgr2=0x7f53100b8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:29.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.649+0000 7f5314c2c640 1 -- 172.21.15.44:0/2930332650 shutdown_connections 2024-04-02T13:21:29.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.649+0000 7f5314c2c640 1 -- 172.21.15.44:0/2930332650 wait complete. 2024-04-02T13:21:29.663 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:29.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.764+0000 7fa1f0cf2640 1 Processor -- start 2024-04-02T13:21:29.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.764+0000 7fa1f0cf2640 1 -- start start 2024-04-02T13:21:29.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.765+0000 7fa1f0cf2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa1ec15e1d0 0x7fa1ec0d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.765+0000 7fa1f0cf2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec0cec60 0x7fa1ec0d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.765+0000 7fa1f0cf2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa1ec15e5b0 0x7fa1ec0d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.765+0000 7fa1f0cf2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa1ec0cfe00 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.765+0000 7fa1f0cf2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa1ec0d1640 con 0x7fa1ec15e1d0 2024-04-02T13:21:29.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.765+0000 7fa1f0cf2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa1ec0cf1f0 con 0x7fa1ec0cec60 2024-04-02T13:21:29.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.765+0000 7fa1ead9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa1ec15e5b0 0x7fa1ec0d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.765+0000 7fa1ead9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa1ec15e5b0 0x7fa1ec0d2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54406/0 (socket says 172.21.15.44:54406) 2024-04-02T13:21:29.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.765+0000 7fa1ead9d640 1 -- 172.21.15.44:0/158341114 learned_addr learned my addr 172.21.15.44:0/158341114 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:29.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.766+0000 7fa1e9d9b640 1 --2- 172.21.15.44:0/158341114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec0cec60 0x7fa1ec0d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.766+0000 7fa1ea59c640 1 --2- 172.21.15.44:0/158341114 >> v2:172.21.15.69:3300/0 conn(0x7fa1ec15e1d0 0x7fa1ec0d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.766+0000 7fa1ead9d640 1 -- 172.21.15.44:0/158341114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec0cec60 msgr2=0x7fa1ec0d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:29.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.766+0000 7fa1ead9d640 1 --2- 172.21.15.44:0/158341114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec0cec60 0x7fa1ec0d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.766+0000 7fa1ead9d640 1 -- 172.21.15.44:0/158341114 >> v2:172.21.15.69:3300/0 conn(0x7fa1ec15e1d0 msgr2=0x7fa1ec0d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:29.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.766+0000 7fa1ead9d640 1 --2- 172.21.15.44:0/158341114 >> v2:172.21.15.69:3300/0 conn(0x7fa1ec15e1d0 0x7fa1ec0d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.766+0000 7fa1ead9d640 1 -- 172.21.15.44:0/158341114 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ec0cf4d0 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.766+0000 7fa1ead9d640 1 --2- 172.21.15.44:0/158341114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5b0 0x7fa1ec0d2c00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec08c9c0 tx=0x7fa1dc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4adb3bc3d328293 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:29.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.767+0000 7fa1e959a640 1 -- 172.21.15.44:0/158341114 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc02c9e0 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.767+0000 7fa1e959a640 1 -- 172.21.15.44:0/158341114 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1dc02cb80 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.767+0000 7fa1e959a640 1 -- 172.21.15.44:0/158341114 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc0343f0 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.767+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/158341114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5b0 msgr2=0x7fa1ec0d2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.767+0000 7fa1f0cf2640 1 --2- 172.21.15.44:0/158341114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5b0 0x7fa1ec0d2c00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec08c9c0 tx=0x7fa1dc02be40 comp rx=0 tx=0).stop 2024-04-02T13:21:29.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.767+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/158341114 shutdown_connections 2024-04-02T13:21:29.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.767+0000 7fa1f0cf2640 1 --2- 172.21.15.44:0/158341114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5b0 0x7fa1ec0d2c00 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.767+0000 7fa1f0cf2640 1 --2- 172.21.15.44:0/158341114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec0cec60 0x7fa1ec0d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.767+0000 7fa1f0cf2640 1 --2- 172.21.15.44:0/158341114 >> v2:172.21.15.69:3300/0 conn(0x7fa1ec15e1d0 0x7fa1ec0d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.767+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/158341114 >> 172.21.15.44:0/158341114 conn(0x7fa1ec09d3b0 msgr2=0x7fa1ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:29.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.767+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/158341114 shutdown_connections 2024-04-02T13:21:29.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.767+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/158341114 wait complete. 2024-04-02T13:21:29.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.768+0000 7fa1f0cf2640 1 Processor -- start 2024-04-02T13:21:29.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.769+0000 7fa1f0cf2640 1 -- start start 2024-04-02T13:21:29.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.769+0000 7fa1f0cf2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1ec0cec60 0x7fa1ec1526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.769+0000 7fa1f0cf2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec15e1d0 0x7fa1ec15fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.769+0000 7fa1f0cf2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5b0 0x7fa1ec0b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.769+0000 7fa1f0cf2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec0b4ce0 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.769+0000 7fa1f0cf2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec0cfa90 con 0x7fa1ec0cec60 2024-04-02T13:21:29.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.769+0000 7fa1f0cf2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa1ec0b4e60 con 0x7fa1ec15e1d0 2024-04-02T13:21:29.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.769+0000 7fa1ead9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5b0 0x7fa1ec0b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.769+0000 7fa1ead9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5b0 0x7fa1ec0b8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54412/0 (socket says 172.21.15.44:54412) 2024-04-02T13:21:29.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.769+0000 7fa1ead9d640 1 -- 172.21.15.44:0/3040416012 learned_addr learned my addr 172.21.15.44:0/3040416012 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:29.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1ea59c640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1ec0cec60 0x7fa1ec1526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1e9d9b640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec15e1d0 0x7fa1ec15fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1ead9d640 1 -- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec15e1d0 msgr2=0x7fa1ec15fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1ead9d640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec15e1d0 0x7fa1ec15fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1ead9d640 1 -- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1ec0cec60 msgr2=0x7fa1ec1526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1ead9d640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1ec0cec60 0x7fa1ec1526f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1ead9d640 1 -- 172.21.15.44:0/3040416012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ec0cb590 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1ea59c640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1ec0cec60 0x7fa1ec1526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:29.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1ead9d640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5b0 0x7fa1ec0b8010 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc036040 tx=0x7fa1dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:29.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1e2ffd640 1 -- 172.21.15.44:0/3040416012 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc044020 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1e2ffd640 1 -- 172.21.15.44:0/3040416012 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1dc040670 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/3040416012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1ec0cb870 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1e2ffd640 1 -- 172.21.15.44:0/3040416012 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc009510 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.770+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/3040416012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1ec0cbe30 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.771+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/3040416012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1ec0684f0 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.772+0000 7fa1e2ffd640 1 -- 172.21.15.44:0/3040416012 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa1dc047020 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.772+0000 7fa1e2ffd640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa1cc06c8f0 0x7fa1cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:29.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.772+0000 7fa1e2ffd640 1 -- 172.21.15.44:0/3040416012 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fa1dc0aec00 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.772+0000 7fa1ea59c640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa1cc06c8f0 0x7fa1cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:29.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.773+0000 7fa1ea59c640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa1cc06c8f0 0x7fa1cc06edb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400b840 tx=0x7fa1e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:29.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.776+0000 7fa1e2ffd640 1 -- 172.21.15.44:0/3040416012 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa1dc0b5080 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.958+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/3040416012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa1ec0d47f0 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.959+0000 7fa1e2ffd640 1 -- 172.21.15.44:0/3040416012 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fa1dc04e030 con 0x7fa1ec15e5b0 2024-04-02T13:21:29.962 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:29.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.961+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa1cc06c8f0 msgr2=0x7fa1cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.961+0000 7fa1f0cf2640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa1cc06c8f0 0x7fa1cc06edb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400b840 tx=0x7fa1e4008040 comp rx=0 tx=0).stop 2024-04-02T13:21:29.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.961+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/3040416012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5b0 msgr2=0x7fa1ec0b8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:29.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.961+0000 7fa1f0cf2640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5b0 0x7fa1ec0b8010 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc036040 tx=0x7fa1dc004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:29.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.961+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/3040416012 shutdown_connections 2024-04-02T13:21:29.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.961+0000 7fa1f0cf2640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa1cc06c8f0 0x7fa1cc06edb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.961+0000 7fa1f0cf2640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5b0 0x7fa1ec0b8010 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.961+0000 7fa1f0cf2640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec15e1d0 0x7fa1ec15fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.961+0000 7fa1f0cf2640 1 --2- 172.21.15.44:0/3040416012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1ec0cec60 0x7fa1ec1526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:29.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.961+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/3040416012 >> 172.21.15.44:0/3040416012 conn(0x7fa1ec09d3b0 msgr2=0x7fa1ec0d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:29.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.961+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/3040416012 shutdown_connections 2024-04-02T13:21:29.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:29.961+0000 7fa1f0cf2640 1 -- 172.21.15.44:0/3040416012 wait complete. 2024-04-02T13:21:29.974 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:30.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.076+0000 7f7a6e255640 1 Processor -- start 2024-04-02T13:21:30.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.076+0000 7f7a6e255640 1 -- start start 2024-04-02T13:21:30.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.076+0000 7f7a6e255640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a680c6b40 0x7f7a68166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.076+0000 7f7a6e255640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a680cd760 con 0x7f7a680b4050 2024-04-02T13:21:30.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.076+0000 7f7a6e255640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a680cda40 con 0x7f7a6815c260 2024-04-02T13:21:30.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.076+0000 7f7a6e255640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7a680cdd20 con 0x7f7a680c6b40 2024-04-02T13:21:30.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.077+0000 7f7a66ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7a680b4050 0x7f7a680ccaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46340/0 (socket says 172.21.15.44:46340) 2024-04-02T13:21:30.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.077+0000 7f7a66ffd640 1 -- 172.21.15.44:0/3163802396 learned_addr learned my addr 172.21.15.44:0/3163802396 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:30.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.077+0000 7f7a67fff640 1 --2- 172.21.15.44:0/3163802396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a680c6b40 0x7f7a68166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.077+0000 7f7a667fc640 1 -- 172.21.15.44:0/3163802396 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1690063628 0 0) 0x7f7a680cd760 con 0x7f7a680b4050 2024-04-02T13:21:30.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.077+0000 7f7a667fc640 1 -- 172.21.15.44:0/3163802396 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a50003520 con 0x7f7a680b4050 2024-04-02T13:21:30.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.077+0000 7f7a667fc640 1 -- 172.21.15.44:0/3163802396 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2896375124 0 0) 0x7f7a680cda40 con 0x7f7a6815c260 2024-04-02T13:21:30.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.077+0000 7f7a67fff640 1 -- 172.21.15.44:0/3163802396 >> v1:172.21.15.69:6789/0 conn(0x7f7a6815c260 legacy=0x7f7a680b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.077+0000 7f7a67fff640 1 -- 172.21.15.44:0/3163802396 >> v1:172.21.15.44:6789/0 conn(0x7f7a680b4050 legacy=0x7f7a680ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.077+0000 7f7a67fff640 1 -- 172.21.15.44:0/3163802396 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a680ce000 con 0x7f7a680c6b40 2024-04-02T13:21:30.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.078+0000 7f7a67fff640 1 --2- 172.21.15.44:0/3163802396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a680c6b40 0x7f7a68166cf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7a60009dc0 tx=0x7f7a6002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65dab33c5af13410 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:30.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.078+0000 7f7a667fc640 1 -- 172.21.15.44:0/3163802396 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a60039070 con 0x7f7a680c6b40 2024-04-02T13:21:30.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.078+0000 7f7a667fc640 1 -- 172.21.15.44:0/3163802396 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a6002cb10 con 0x7f7a680c6b40 2024-04-02T13:21:30.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.078+0000 7f7a667fc640 1 -- 172.21.15.44:0/3163802396 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a60034440 con 0x7f7a680c6b40 2024-04-02T13:21:30.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.079+0000 7f7a6e255640 1 -- 172.21.15.44:0/3163802396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a680c6b40 msgr2=0x7f7a68166cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.079+0000 7f7a6e255640 1 --2- 172.21.15.44:0/3163802396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a680c6b40 0x7f7a68166cf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7a60009dc0 tx=0x7f7a6002bde0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.079+0000 7f7a6e255640 1 -- 172.21.15.44:0/3163802396 shutdown_connections 2024-04-02T13:21:30.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.079+0000 7f7a6e255640 1 --2- 172.21.15.44:0/3163802396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a680c6b40 0x7f7a68166cf0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.079+0000 7f7a6e255640 1 -- 172.21.15.44:0/3163802396 >> 172.21.15.44:0/3163802396 conn(0x7f7a6809aa20 msgr2=0x7f7a6809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:30.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.079+0000 7f7a6e255640 1 -- 172.21.15.44:0/3163802396 shutdown_connections 2024-04-02T13:21:30.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.079+0000 7f7a6e255640 1 -- 172.21.15.44:0/3163802396 wait complete. 2024-04-02T13:21:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.080+0000 7f7a6e255640 1 Processor -- start 2024-04-02T13:21:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.080+0000 7f7a6e255640 1 -- start start 2024-04-02T13:21:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a6e255640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680b4050 0x7f7a6814e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a6e255640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a680c6b40 0x7f7a6806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a6e255640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a6815c260 0x7f7a680d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a6e255640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7a680cd380 con 0x7f7a680c6b40 2024-04-02T13:21:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a6e255640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a680cd200 con 0x7f7a680b4050 2024-04-02T13:21:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a6e255640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7a680cd500 con 0x7f7a6815c260 2024-04-02T13:21:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a66ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a680c6b40 0x7f7a6806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a66ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a680c6b40 0x7f7a6806d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54422/0 (socket says 172.21.15.44:54422) 2024-04-02T13:21:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a66ffd640 1 -- 172.21.15.44:0/2700173300 learned_addr learned my addr 172.21.15.44:0/2700173300 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:30.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a67fff640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a6815c260 0x7f7a680d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a677fe640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680b4050 0x7f7a6814e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a66ffd640 1 -- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a6815c260 msgr2=0x7f7a680d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a66ffd640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a6815c260 0x7f7a680d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a66ffd640 1 -- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680b4050 msgr2=0x7f7a6814e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a66ffd640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680b4050 0x7f7a6814e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a66ffd640 1 -- 172.21.15.44:0/2700173300 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a6814e980 con 0x7f7a680c6b40 2024-04-02T13:21:30.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.081+0000 7f7a66ffd640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a680c6b40 0x7f7a6806d330 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7a5800c700 tx=0x7f7a5800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:30.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.082+0000 7f7a4ffff640 1 -- 172.21.15.44:0/2700173300 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a58018070 con 0x7f7a680c6b40 2024-04-02T13:21:30.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.082+0000 7f7a4ffff640 1 -- 172.21.15.44:0/2700173300 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a580074e0 con 0x7f7a680c6b40 2024-04-02T13:21:30.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.082+0000 7f7a4ffff640 1 -- 172.21.15.44:0/2700173300 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a58004a20 con 0x7f7a680c6b40 2024-04-02T13:21:30.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.082+0000 7f7a6e255640 1 -- 172.21.15.44:0/2700173300 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a6806ddf0 con 0x7f7a680c6b40 2024-04-02T13:21:30.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.082+0000 7f7a6e255640 1 -- 172.21.15.44:0/2700173300 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a6806e3b0 con 0x7f7a680c6b40 2024-04-02T13:21:30.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.083+0000 7f7a6e255640 1 -- 172.21.15.44:0/2700173300 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a6806e5b0 con 0x7f7a680c6b40 2024-04-02T13:21:30.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.083+0000 7f7a4ffff640 1 -- 172.21.15.44:0/2700173300 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7a58004bc0 con 0x7f7a680c6b40 2024-04-02T13:21:30.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.083+0000 7f7a4ffff640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7a5406c920 0x7f7a5406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.083+0000 7f7a4ffff640 1 -- 172.21.15.44:0/2700173300 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f7a5808b480 con 0x7f7a680c6b40 2024-04-02T13:21:30.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.084+0000 7f7a677fe640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7a5406c920 0x7f7a5406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.084+0000 7f7a677fe640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7a5406c920 0x7f7a5406ede0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c000f60 tx=0x7f7a5c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.088+0000 7f7a4ffff640 1 -- 172.21.15.44:0/2700173300 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7a6806e5b0 con 0x7f7a680c6b40 2024-04-02T13:21:30.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.271+0000 7f7a6e255640 1 -- 172.21.15.44:0/2700173300 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7a680684f0 con 0x7f7a680c6b40 2024-04-02T13:21:30.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.272+0000 7f7a4ffff640 1 -- 172.21.15.44:0/2700173300 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f7a6814e980 con 0x7f7a680c6b40 2024-04-02T13:21:30.274 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:30.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.274+0000 7f7a6e255640 1 -- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7a5406c920 msgr2=0x7f7a5406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.274+0000 7f7a6e255640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7a5406c920 0x7f7a5406ede0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c000f60 tx=0x7f7a5c009040 comp rx=0 tx=0).stop 2024-04-02T13:21:30.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.274+0000 7f7a6e255640 1 -- 172.21.15.44:0/2700173300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a680c6b40 msgr2=0x7f7a6806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.274+0000 7f7a6e255640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a680c6b40 0x7f7a6806d330 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7a5800c700 tx=0x7f7a5800cbc0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.274+0000 7f7a6e255640 1 -- 172.21.15.44:0/2700173300 shutdown_connections 2024-04-02T13:21:30.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.274+0000 7f7a6e255640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7a5406c920 0x7f7a5406ede0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.274+0000 7f7a6e255640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a6815c260 0x7f7a680d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.274+0000 7f7a6e255640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a680c6b40 0x7f7a6806d330 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.274+0000 7f7a6e255640 1 --2- 172.21.15.44:0/2700173300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680b4050 0x7f7a6814e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.274+0000 7f7a6e255640 1 -- 172.21.15.44:0/2700173300 >> 172.21.15.44:0/2700173300 conn(0x7f7a6809aa20 msgr2=0x7f7a680cf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:30.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.274+0000 7f7a6e255640 1 -- 172.21.15.44:0/2700173300 shutdown_connections 2024-04-02T13:21:30.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.274+0000 7f7a6e255640 1 -- 172.21.15.44:0/2700173300 wait complete. 2024-04-02T13:21:30.287 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-02T13:21:30.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.389+0000 7f1301c2b640 1 Processor -- start 2024-04-02T13:21:30.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.389+0000 7f1301c2b640 1 -- start start 2024-04-02T13:21:30.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.389+0000 7f1301c2b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f12fc06cff0 0x7f12fc06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.389+0000 7f1301c2b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f12fc15e5d0 0x7f12fc0d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.389+0000 7f1301c2b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12fc06d910 con 0x7f12fc0b4180 2024-04-02T13:21:30.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.389+0000 7f1301c2b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f12fc0d1bf0 con 0x7f12fc15e5d0 2024-04-02T13:21:30.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.389+0000 7f1301c2b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f12fc0d1630 con 0x7f12fc06cff0 2024-04-02T13:21:30.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.390+0000 7f12faffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f12fc15e5d0 0x7f12fc0d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.390+0000 7f12faffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f12fc15e5d0 0x7f12fc0d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54432/0 (socket says 172.21.15.44:54432) 2024-04-02T13:21:30.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.390+0000 7f12faffd640 1 -- 172.21.15.44:0/4148288636 learned_addr learned my addr 172.21.15.44:0/4148288636 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:30.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.390+0000 7f12faffd640 1 -- 172.21.15.44:0/4148288636 >> v2:172.21.15.69:3300/0 conn(0x7f12fc06cff0 msgr2=0x7f12fc06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:30.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.390+0000 7f12faffd640 1 --2- 172.21.15.44:0/4148288636 >> v2:172.21.15.69:3300/0 conn(0x7f12fc06cff0 0x7f12fc06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.390+0000 7f12faffd640 1 -- 172.21.15.44:0/4148288636 >> v1:172.21.15.69:6789/0 conn(0x7f12fc0b4180 legacy=0x7f12fc0ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:30.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.390+0000 7f12faffd640 1 -- 172.21.15.44:0/4148288636 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12fc0d1910 con 0x7f12fc15e5d0 2024-04-02T13:21:30.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.391+0000 7f12faffd640 1 --2- 172.21.15.44:0/4148288636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc15e5d0 0x7f12fc0d0d30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f12fc071d40 tx=0x7f12f002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18774a859d9e35d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:30.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.391+0000 7f12fa7fc640 1 -- 172.21.15.44:0/4148288636 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f002e070 con 0x7f12fc15e5d0 2024-04-02T13:21:30.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.391+0000 7f12fa7fc640 1 -- 172.21.15.44:0/4148288636 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12f0037070 con 0x7f12fc15e5d0 2024-04-02T13:21:30.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.391+0000 7f12fa7fc640 1 -- 172.21.15.44:0/4148288636 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f0031a00 con 0x7f12fc15e5d0 2024-04-02T13:21:30.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.391+0000 7f1301c2b640 1 -- 172.21.15.44:0/4148288636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc15e5d0 msgr2=0x7f12fc0d0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.391+0000 7f1301c2b640 1 --2- 172.21.15.44:0/4148288636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc15e5d0 0x7f12fc0d0d30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f12fc071d40 tx=0x7f12f002b880 comp rx=0 tx=0).stop 2024-04-02T13:21:30.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.391+0000 7f1301c2b640 1 -- 172.21.15.44:0/4148288636 shutdown_connections 2024-04-02T13:21:30.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.391+0000 7f1301c2b640 1 --2- 172.21.15.44:0/4148288636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc15e5d0 0x7f12fc0d0d30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.391+0000 7f1301c2b640 1 --2- 172.21.15.44:0/4148288636 >> v2:172.21.15.69:3300/0 conn(0x7f12fc06cff0 0x7f12fc06d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.391+0000 7f1301c2b640 1 -- 172.21.15.44:0/4148288636 >> 172.21.15.44:0/4148288636 conn(0x7f12fc09aa20 msgr2=0x7f12fc09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:30.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.391+0000 7f1301c2b640 1 -- 172.21.15.44:0/4148288636 shutdown_connections 2024-04-02T13:21:30.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.391+0000 7f1301c2b640 1 -- 172.21.15.44:0/4148288636 wait complete. 2024-04-02T13:21:30.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.392+0000 7f1301c2b640 1 Processor -- start 2024-04-02T13:21:30.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.393+0000 7f1301c2b640 1 -- start start 2024-04-02T13:21:30.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.393+0000 7f1301c2b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12fc06cff0 0x7f12fc14f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.393+0000 7f1301c2b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc0b4180 0x7f12fc0b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.393+0000 7f1301c2b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc15e5d0 0x7f12fc0af450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.393+0000 7f1301c2b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f12fc0d1270 con 0x7f12fc0b4180 2024-04-02T13:21:30.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.393+0000 7f1301c2b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f12fc0b1820 con 0x7f12fc15e5d0 2024-04-02T13:21:30.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.393+0000 7f1301c2b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f12fc0d13f0 con 0x7f12fc06cff0 2024-04-02T13:21:30.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.393+0000 7f12faffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc0b4180 0x7f12fc0b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.393+0000 7f12faffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc0b4180 0x7f12fc0b4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54440/0 (socket says 172.21.15.44:54440) 2024-04-02T13:21:30.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.393+0000 7f12faffd640 1 -- 172.21.15.44:0/2257255214 learned_addr learned my addr 172.21.15.44:0/2257255214 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:30.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.394+0000 7f12fb7fe640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12fc06cff0 0x7f12fc14f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.394+0000 7f12fbfff640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc15e5d0 0x7f12fc0af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.394+0000 7f12faffd640 1 -- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12fc06cff0 msgr2=0x7f12fc14f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.394+0000 7f12faffd640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12fc06cff0 0x7f12fc14f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.394+0000 7f12faffd640 1 -- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc15e5d0 msgr2=0x7f12fc0af450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.394+0000 7f12faffd640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc15e5d0 0x7f12fc0af450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.394+0000 7f12faffd640 1 -- 172.21.15.44:0/2257255214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12fc1f4190 con 0x7f12fc0b4180 2024-04-02T13:21:30.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.394+0000 7f12faffd640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc0b4180 0x7f12fc0b4b70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f12f002b8c0 tx=0x7f12f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:30.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.394+0000 7f12e3fff640 1 -- 172.21.15.44:0/2257255214 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f0032880 con 0x7f12fc0b4180 2024-04-02T13:21:30.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.394+0000 7f12e3fff640 1 -- 172.21.15.44:0/2257255214 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12f0032a20 con 0x7f12fc0b4180 2024-04-02T13:21:30.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.394+0000 7f1301c2b640 1 -- 172.21.15.44:0/2257255214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12fc1f4400 con 0x7f12fc0b4180 2024-04-02T13:21:30.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.394+0000 7f12e3fff640 1 -- 172.21.15.44:0/2257255214 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f0032e10 con 0x7f12fc0b4180 2024-04-02T13:21:30.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.394+0000 7f1301c2b640 1 -- 172.21.15.44:0/2257255214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12fc1f49c0 con 0x7f12fc0b4180 2024-04-02T13:21:30.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.395+0000 7f1301c2b640 1 -- 172.21.15.44:0/2257255214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12fc0684f0 con 0x7f12fc0b4180 2024-04-02T13:21:30.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.395+0000 7f12e3fff640 1 -- 172.21.15.44:0/2257255214 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f12f00453f0 con 0x7f12fc0b4180 2024-04-02T13:21:30.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.396+0000 7f12e3fff640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f12e806c920 0x7f12e806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.396+0000 7f12e3fff640 1 -- 172.21.15.44:0/2257255214 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f12f00ae0a0 con 0x7f12fc0b4180 2024-04-02T13:21:30.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.396+0000 7f12fb7fe640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f12e806c920 0x7f12e806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.396+0000 7f12fb7fe640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f12e806c920 0x7f12e806ede0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009890 tx=0x7f12ec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:30.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.400+0000 7f12e3fff640 1 -- 172.21.15.44:0/2257255214 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f12f0040070 con 0x7f12fc0b4180 2024-04-02T13:21:30.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.586+0000 7f1301c2b640 1 -- 172.21.15.44:0/2257255214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f12fc06cdd0 con 0x7f12fc0b4180 2024-04-02T13:21:30.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.586+0000 7f12e3fff640 1 -- 172.21.15.44:0/2257255214 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f12f007b9d0 con 0x7f12fc0b4180 2024-04-02T13:21:30.589 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:30.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.588+0000 7f1301c2b640 1 -- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f12e806c920 msgr2=0x7f12e806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.588+0000 7f1301c2b640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f12e806c920 0x7f12e806ede0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009890 tx=0x7f12ec008040 comp rx=0 tx=0).stop 2024-04-02T13:21:30.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.588+0000 7f1301c2b640 1 -- 172.21.15.44:0/2257255214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc0b4180 msgr2=0x7f12fc0b4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.588+0000 7f1301c2b640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc0b4180 0x7f12fc0b4b70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f12f002b8c0 tx=0x7f12f00046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.589+0000 7f1301c2b640 1 -- 172.21.15.44:0/2257255214 shutdown_connections 2024-04-02T13:21:30.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.589+0000 7f1301c2b640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f12e806c920 0x7f12e806ede0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.589+0000 7f1301c2b640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc15e5d0 0x7f12fc0af450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.589+0000 7f1301c2b640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc0b4180 0x7f12fc0b4b70 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.589+0000 7f1301c2b640 1 --2- 172.21.15.44:0/2257255214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12fc06cff0 0x7f12fc14f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.589+0000 7f1301c2b640 1 -- 172.21.15.44:0/2257255214 >> 172.21.15.44:0/2257255214 conn(0x7f12fc09aa20 msgr2=0x7f12fc0cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:30.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.589+0000 7f1301c2b640 1 -- 172.21.15.44:0/2257255214 shutdown_connections 2024-04-02T13:21:30.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.589+0000 7f1301c2b640 1 -- 172.21.15.44:0/2257255214 wait complete. 2024-04-02T13:21:30.602 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:21:30.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.701+0000 7f98c9550640 1 Processor -- start 2024-04-02T13:21:30.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.701+0000 7f98c9550640 1 -- start start 2024-04-02T13:21:30.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.701+0000 7f98c9550640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f98c40cffd0 0x7f98c40ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.701+0000 7f98c9550640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98c40cef30 con 0x7f98c41617e0 2024-04-02T13:21:30.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.701+0000 7f98c9550640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98c40cf210 con 0x7f98c40cd3a0 2024-04-02T13:21:30.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.701+0000 7f98c9550640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f98c40cf4f0 con 0x7f98c40cffd0 2024-04-02T13:21:30.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.701+0000 7f98c27fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f98c40cffd0 0x7f98c40ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.701+0000 7f98c2ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f98c41617e0 0x7f98c40ccc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46356/0 (socket says 172.21.15.44:46356) 2024-04-02T13:21:30.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.701+0000 7f98c2ffd640 1 -- 172.21.15.44:0/517148700 learned_addr learned my addr 172.21.15.44:0/517148700 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:30.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.702+0000 7f98c27fc640 1 -- 172.21.15.44:0/517148700 >> v1:172.21.15.69:6789/0 conn(0x7f98c40cd3a0 legacy=0x7f98c40ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:30.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.702+0000 7f98c27fc640 1 -- 172.21.15.44:0/517148700 >> v1:172.21.15.44:6789/0 conn(0x7f98c41617e0 legacy=0x7f98c40ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.702+0000 7f98c27fc640 1 -- 172.21.15.44:0/517148700 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c40cf7d0 con 0x7f98c40cffd0 2024-04-02T13:21:30.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.702+0000 7f98c27fc640 1 --2- 172.21.15.44:0/517148700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c40cffd0 0x7f98c40ce8c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0097a0 tx=0x7f98ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e260eddf09c199ce server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:30.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.703+0000 7f98c1ffb640 1 -- 172.21.15.44:0/517148700 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac035020 con 0x7f98c40cffd0 2024-04-02T13:21:30.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.703+0000 7f98c1ffb640 1 -- 172.21.15.44:0/517148700 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98ac031830 con 0x7f98c40cffd0 2024-04-02T13:21:30.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.703+0000 7f98c1ffb640 1 -- 172.21.15.44:0/517148700 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac031c00 con 0x7f98c40cffd0 2024-04-02T13:21:30.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.703+0000 7f98c9550640 1 -- 172.21.15.44:0/517148700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c40cffd0 msgr2=0x7f98c40ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.703+0000 7f98c9550640 1 --2- 172.21.15.44:0/517148700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c40cffd0 0x7f98c40ce8c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0097a0 tx=0x7f98ac02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.703+0000 7f98c9550640 1 -- 172.21.15.44:0/517148700 shutdown_connections 2024-04-02T13:21:30.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.703+0000 7f98c9550640 1 --2- 172.21.15.44:0/517148700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c40cffd0 0x7f98c40ce8c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.703+0000 7f98c9550640 1 -- 172.21.15.44:0/517148700 >> 172.21.15.44:0/517148700 conn(0x7f98c40b20e0 msgr2=0x7f98c40b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.703+0000 7f98c9550640 1 -- 172.21.15.44:0/517148700 shutdown_connections 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.703+0000 7f98c9550640 1 -- 172.21.15.44:0/517148700 wait complete. 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.705+0000 7f98c9550640 1 Processor -- start 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.705+0000 7f98c9550640 1 -- start start 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.705+0000 7f98c9550640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c40cd3a0 0x7f98c41e7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.705+0000 7f98c9550640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98c40cffd0 0x7f98c406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.705+0000 7f98c9550640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c41617e0 0x7f98c40aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.705+0000 7f98c9550640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4161d40 con 0x7f98c41617e0 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.705+0000 7f98c9550640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4161bc0 con 0x7f98c40cd3a0 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.705+0000 7f98c9550640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f98c41629c0 con 0x7f98c40cffd0 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.705+0000 7f98c37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c41617e0 0x7f98c40aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.705+0000 7f98c37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c41617e0 0x7f98c40aee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54454/0 (socket says 172.21.15.44:54454) 2024-04-02T13:21:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.705+0000 7f98c37fe640 1 -- 172.21.15.44:0/1090423630 learned_addr learned my addr 172.21.15.44:0/1090423630 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.706+0000 7f98c2ffd640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c40cd3a0 0x7f98c41e7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.706+0000 7f98c27fc640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98c40cffd0 0x7f98c406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.706+0000 7f98c37fe640 1 -- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98c40cffd0 msgr2=0x7f98c406ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.706+0000 7f98c37fe640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98c40cffd0 0x7f98c406ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.706+0000 7f98c37fe640 1 -- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c40cd3a0 msgr2=0x7f98c41e7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.706+0000 7f98c37fe640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c40cd3a0 0x7f98c41e7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.706+0000 7f98c37fe640 1 -- 172.21.15.44:0/1090423630 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c41e8920 con 0x7f98c41617e0 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.706+0000 7f98c37fe640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c41617e0 0x7f98c40aee30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f98b400c780 tx=0x7f98b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.706+0000 7f98bb7fe640 1 -- 172.21.15.44:0/1090423630 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b4004b60 con 0x7f98c41617e0 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.706+0000 7f98bb7fe640 1 -- 172.21.15.44:0/1090423630 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98b4004d00 con 0x7f98c41617e0 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.706+0000 7f98bb7fe640 1 -- 172.21.15.44:0/1090423630 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b400aa90 con 0x7f98c41617e0 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.706+0000 7f98c9550640 1 -- 172.21.15.44:0/1090423630 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c41e8bc0 con 0x7f98c41617e0 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.706+0000 7f98c9550640 1 -- 172.21.15.44:0/1090423630 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c41e9180 con 0x7f98c41617e0 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.707+0000 7f98c9550640 1 -- 172.21.15.44:0/1090423630 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c406f570 con 0x7f98c41617e0 2024-04-02T13:21:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.708+0000 7f98bb7fe640 1 -- 172.21.15.44:0/1090423630 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f98b400ac30 con 0x7f98c41617e0 2024-04-02T13:21:30.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.708+0000 7f98bb7fe640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98a406c8f0 0x7f98a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:30.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.708+0000 7f98bb7fe640 1 -- 172.21.15.44:0/1090423630 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f98b408c0e0 con 0x7f98c41617e0 2024-04-02T13:21:30.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.708+0000 7f98c2ffd640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98a406c8f0 0x7f98a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:30.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.709+0000 7f98c2ffd640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98a406c8f0 0x7f98a406edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f98bc007fd0 tx=0x7f98bc007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:30.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.712+0000 7f98bb7fe640 1 -- 172.21.15.44:0/1090423630 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f98b4020070 con 0x7f98c41617e0 2024-04-02T13:21:30.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.937+0000 7f98c9550640 1 -- 172.21.15.44:0/1090423630 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98c40684f0 con 0x7f98c41617e0 2024-04-02T13:21:30.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.937+0000 7f98bb7fe640 1 -- 172.21.15.44:0/1090423630 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7f98b4059aa0 con 0x7f98c41617e0 2024-04-02T13:21:30.940 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:30.940 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4040417710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4040417710},{"type":"v1","addr":"172.21.15.69:6836","nonce":4040417710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2772068060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2772068060},{"type":"v1","addr":"172.21.15.44:6833","nonce":2772068060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1985405150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1985405150},{"type":"v1","addr":"172.21.15.69:6837","nonce":1985405150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:20:20.247358+0000","modified":"2024-04-02T13:20:42.332570+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/2282156931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2282156931},{"type":"v1","addr":"172.21.15.44:6835","nonce":2282156931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-04-02T13:21:30.940 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 9 2024-04-02T13:21:30.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.939+0000 7f98c9550640 1 -- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98a406c8f0 msgr2=0x7f98a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.939+0000 7f98c9550640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98a406c8f0 0x7f98a406edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f98bc007fd0 tx=0x7f98bc007e80 comp rx=0 tx=0).stop 2024-04-02T13:21:30.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.939+0000 7f98c9550640 1 -- 172.21.15.44:0/1090423630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c41617e0 msgr2=0x7f98c40aee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:30.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.939+0000 7f98c9550640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c41617e0 0x7f98c40aee30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f98b400c780 tx=0x7f98b400cc40 comp rx=0 tx=0).stop 2024-04-02T13:21:30.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.939+0000 7f98c9550640 1 -- 172.21.15.44:0/1090423630 shutdown_connections 2024-04-02T13:21:30.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.939+0000 7f98c9550640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98a406c8f0 0x7f98a406edb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.939+0000 7f98c9550640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c41617e0 0x7f98c40aee30 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.939+0000 7f98c9550640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98c40cffd0 0x7f98c406ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.939+0000 7f98c9550640 1 --2- 172.21.15.44:0/1090423630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c40cd3a0 0x7f98c41e7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:30.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.939+0000 7f98c9550640 1 -- 172.21.15.44:0/1090423630 >> 172.21.15.44:0/1090423630 conn(0x7f98c40b20e0 msgr2=0x7f98c4071870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:30.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.939+0000 7f98c9550640 1 -- 172.21.15.44:0/1090423630 shutdown_connections 2024-04-02T13:21:30.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:30.939+0000 7f98c9550640 1 -- 172.21.15.44:0/1090423630 wait complete. 2024-04-02T13:21:30.953 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:31.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.051+0000 7f52c8396640 1 Processor -- start 2024-04-02T13:21:31.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.052+0000 7f52c8396640 1 -- start start 2024-04-02T13:21:31.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.052+0000 7f52c8396640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f52c015e0f0 0x7f52c00d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.052+0000 7f52c8396640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52c006bbe0 0x7f52c00add60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.052+0000 7f52c8396640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52c006dfe0 con 0x7f52c00d0630 2024-04-02T13:21:31.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.052+0000 7f52c8396640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f52c006e2c0 con 0x7f52c015e0f0 2024-04-02T13:21:31.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.052+0000 7f52c8396640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f52c006e5a0 con 0x7f52c006bbe0 2024-04-02T13:21:31.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.053+0000 7f52c6132640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f52c015e0f0 0x7f52c00d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.053+0000 7f52c6132640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f52c015e0f0 0x7f52c00d00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54458/0 (socket says 172.21.15.44:54458) 2024-04-02T13:21:31.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.053+0000 7f52c6132640 1 -- 172.21.15.44:0/3182076934 learned_addr learned my addr 172.21.15.44:0/3182076934 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:31.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.053+0000 7f52c6132640 1 -- 172.21.15.44:0/3182076934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52c006bbe0 msgr2=0x7f52c00add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:31.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.053+0000 7f52c6933640 1 --2- 172.21.15.44:0/3182076934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52c006bbe0 0x7f52c00add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.053+0000 7f52c6132640 1 --2- 172.21.15.44:0/3182076934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52c006bbe0 0x7f52c00add60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.053+0000 7f52c6132640 1 -- 172.21.15.44:0/3182076934 >> v1:172.21.15.69:6789/0 conn(0x7f52c00d0630 legacy=0x7f52c006b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.053+0000 7f52c6132640 1 -- 172.21.15.44:0/3182076934 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c006e880 con 0x7f52c015e0f0 2024-04-02T13:21:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.053+0000 7f52c6933640 1 --2- 172.21.15.44:0/3182076934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52c006bbe0 0x7f52c00add60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.054+0000 7f52c6132640 1 --2- 172.21.15.44:0/3182076934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52c015e0f0 0x7f52c00d00f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009870 tx=0x7f52bc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b918a04a9b274016 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.054+0000 7f52c5130640 1 -- 172.21.15.44:0/3182076934 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52bc034070 con 0x7f52c015e0f0 2024-04-02T13:21:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.054+0000 7f52c5130640 1 -- 172.21.15.44:0/3182076934 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52bc030840 con 0x7f52c015e0f0 2024-04-02T13:21:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.054+0000 7f52c5130640 1 -- 172.21.15.44:0/3182076934 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52bc030c10 con 0x7f52c015e0f0 2024-04-02T13:21:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.054+0000 7f52c8396640 1 -- 172.21.15.44:0/3182076934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52c015e0f0 msgr2=0x7f52c00d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.054+0000 7f52c8396640 1 --2- 172.21.15.44:0/3182076934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52c015e0f0 0x7f52c00d00f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009870 tx=0x7f52bc02b980 comp rx=0 tx=0).stop 2024-04-02T13:21:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.054+0000 7f52c8396640 1 -- 172.21.15.44:0/3182076934 shutdown_connections 2024-04-02T13:21:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.054+0000 7f52c8396640 1 --2- 172.21.15.44:0/3182076934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52c006bbe0 0x7f52c00add60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.054+0000 7f52c8396640 1 --2- 172.21.15.44:0/3182076934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52c015e0f0 0x7f52c00d00f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.054+0000 7f52c8396640 1 -- 172.21.15.44:0/3182076934 >> 172.21.15.44:0/3182076934 conn(0x7f52c00b2980 msgr2=0x7f52c00b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:31.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.055+0000 7f52c8396640 1 -- 172.21.15.44:0/3182076934 shutdown_connections 2024-04-02T13:21:31.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.055+0000 7f52c8396640 1 -- 172.21.15.44:0/3182076934 wait complete. 2024-04-02T13:21:31.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.056+0000 7f52c8396640 1 Processor -- start 2024-04-02T13:21:31.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.056+0000 7f52c8396640 1 -- start start 2024-04-02T13:21:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.056+0000 7f52c8396640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52c006bbe0 0x7f52c00c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.056+0000 7f52c8396640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52c00d0630 0x7f52c00c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.056+0000 7f52c8396640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52c015e0f0 0x7f52c00ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.056+0000 7f52c8396640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f52c015e4d0 con 0x7f52c00d0630 2024-04-02T13:21:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.056+0000 7f52c8396640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f52c006eda0 con 0x7f52c015e0f0 2024-04-02T13:21:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.056+0000 7f52c8396640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f52c015e650 con 0x7f52c006bbe0 2024-04-02T13:21:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.056+0000 7f52c5931640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52c00d0630 0x7f52c00c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.056+0000 7f52c5931640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52c00d0630 0x7f52c00c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54464/0 (socket says 172.21.15.44:54464) 2024-04-02T13:21:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.056+0000 7f52c5931640 1 -- 172.21.15.44:0/1450015807 learned_addr learned my addr 172.21.15.44:0/1450015807 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.057+0000 7f52c6132640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52c006bbe0 0x7f52c00c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.057+0000 7f52c6933640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52c015e0f0 0x7f52c00ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.057+0000 7f52c5931640 1 -- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52c006bbe0 msgr2=0x7f52c00c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.057+0000 7f52c5931640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52c006bbe0 0x7f52c00c7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.057+0000 7f52c5931640 1 -- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52c015e0f0 msgr2=0x7f52c00ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.057+0000 7f52c5931640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52c015e0f0 0x7f52c00ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.057+0000 7f52c5931640 1 -- 172.21.15.44:0/1450015807 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c00c8580 con 0x7f52c00d0630 2024-04-02T13:21:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.057+0000 7f52c5931640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52c00d0630 0x7f52c00c9620 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f52b000c430 tx=0x7f52b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.057+0000 7f52b6ffd640 1 -- 172.21.15.44:0/1450015807 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0011790 con 0x7f52c00d0630 2024-04-02T13:21:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.057+0000 7f52b6ffd640 1 -- 172.21.15.44:0/1450015807 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b0017070 con 0x7f52c00d0630 2024-04-02T13:21:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.057+0000 7f52c8396640 1 -- 172.21.15.44:0/1450015807 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c00c8860 con 0x7f52c00d0630 2024-04-02T13:21:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.057+0000 7f52b6ffd640 1 -- 172.21.15.44:0/1450015807 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0011bb0 con 0x7f52c00d0630 2024-04-02T13:21:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.057+0000 7f52c8396640 1 -- 172.21.15.44:0/1450015807 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c00c8e20 con 0x7f52c00d0630 2024-04-02T13:21:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.058+0000 7f52c8396640 1 -- 172.21.15.44:0/1450015807 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52c00684f0 con 0x7f52c00d0630 2024-04-02T13:21:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.059+0000 7f52b6ffd640 1 -- 172.21.15.44:0/1450015807 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f52b001c460 con 0x7f52c00d0630 2024-04-02T13:21:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.059+0000 7f52b6ffd640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f52a406c920 0x7f52a406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.059+0000 7f52b6ffd640 1 -- 172.21.15.44:0/1450015807 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f52b008bda0 con 0x7f52c00d0630 2024-04-02T13:21:31.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.059+0000 7f52c6132640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f52a406c920 0x7f52a406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.060+0000 7f52c6132640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f52a406c920 0x7f52a406ede0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f52bc036000 tx=0x7f52bc03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:31.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.063+0000 7f52b6ffd640 1 -- 172.21.15.44:0/1450015807 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f52b0059db0 con 0x7f52c00d0630 2024-04-02T13:21:31.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.253+0000 7f52c8396640 1 -- 172.21.15.44:0/1450015807 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f52c00727f0 con 0x7f52c00d0630 2024-04-02T13:21:31.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.254+0000 7f52b6ffd640 1 -- 172.21.15.44:0/1450015807 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18899 (secure 0 0 0) 0x7f52b00596b0 con 0x7f52c00d0630 2024-04-02T13:21:31.257 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:31.257 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:08.061253+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:31.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.256+0000 7f52c8396640 1 -- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f52a406c920 msgr2=0x7f52a406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.256+0000 7f52c8396640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f52a406c920 0x7f52a406ede0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f52bc036000 tx=0x7f52bc03b040 comp rx=0 tx=0).stop 2024-04-02T13:21:31.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.256+0000 7f52c8396640 1 -- 172.21.15.44:0/1450015807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52c00d0630 msgr2=0x7f52c00c9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.256+0000 7f52c8396640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52c00d0630 0x7f52c00c9620 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f52b000c430 tx=0x7f52b000c8f0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.256+0000 7f52c8396640 1 -- 172.21.15.44:0/1450015807 shutdown_connections 2024-04-02T13:21:31.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.256+0000 7f52c8396640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f52a406c920 0x7f52a406ede0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.256+0000 7f52c8396640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52c015e0f0 0x7f52c00ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.256+0000 7f52c8396640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52c00d0630 0x7f52c00c9620 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.256+0000 7f52c8396640 1 --2- 172.21.15.44:0/1450015807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52c006bbe0 0x7f52c00c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.257+0000 7f52c8396640 1 -- 172.21.15.44:0/1450015807 >> 172.21.15.44:0/1450015807 conn(0x7f52c00b2980 msgr2=0x7f52c00b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:31.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.257+0000 7f52c8396640 1 -- 172.21.15.44:0/1450015807 shutdown_connections 2024-04-02T13:21:31.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.257+0000 7f52c8396640 1 -- 172.21.15.44:0/1450015807 wait complete. 2024-04-02T13:21:31.270 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T13:21:31.270 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:21:31.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.369+0000 7f360bc50640 1 Processor -- start 2024-04-02T13:21:31.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.369+0000 7f360bc50640 1 -- start start 2024-04-02T13:21:31.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.370+0000 7f360bc50640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f36040d0a30 0x7f360406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.370+0000 7f360bc50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36040d2040 0x7f36040d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.370+0000 7f360bc50640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36040d0000 con 0x7f36040d0e10 2024-04-02T13:21:31.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.370+0000 7f360bc50640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f360406cb00 con 0x7f36040d0a30 2024-04-02T13:21:31.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.370+0000 7f360bc50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f360406c1d0 con 0x7f36040d2040 2024-04-02T13:21:31.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.370+0000 7f36099ec640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f36040d0a30 0x7f360406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.370+0000 7f36091eb640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f36040d0e10 0x7f36040cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46360/0 (socket says 172.21.15.44:46360) 2024-04-02T13:21:31.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.370+0000 7f36091eb640 1 -- 172.21.15.44:0/3484464307 learned_addr learned my addr 172.21.15.44:0/3484464307 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:31.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.371+0000 7f360a1ed640 1 --2- 172.21.15.44:0/3484464307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36040d2040 0x7f36040d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.371+0000 7f36099ec640 1 -- 172.21.15.44:0/3484464307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36040d2040 msgr2=0x7f36040d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:31.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.371+0000 7f36099ec640 1 --2- 172.21.15.44:0/3484464307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36040d2040 0x7f36040d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.371+0000 7f36099ec640 1 -- 172.21.15.44:0/3484464307 >> v1:172.21.15.44:6789/0 conn(0x7f36040d0e10 legacy=0x7f36040cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.371+0000 7f36099ec640 1 -- 172.21.15.44:0/3484464307 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f360406c4b0 con 0x7f36040d0a30 2024-04-02T13:21:31.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.371+0000 7f36089ea640 1 -- 172.21.15.44:0/3484464307 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3464242976 0 0) 0x7f36040d0000 con 0x7f36040d0e10 2024-04-02T13:21:31.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.371+0000 7f360a1ed640 1 --2- 172.21.15.44:0/3484464307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36040d2040 0x7f36040d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.371+0000 7f36099ec640 1 --2- 172.21.15.44:0/3484464307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36040d0a30 0x7f360406cf90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f36000097a0 tx=0x7f360002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da6b3125fd3a182 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.372+0000 7f36089ea640 1 -- 172.21.15.44:0/3484464307 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36000315a0 con 0x7f36040d0a30 2024-04-02T13:21:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.372+0000 7f36089ea640 1 -- 172.21.15.44:0/3484464307 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3600037070 con 0x7f36040d0a30 2024-04-02T13:21:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.372+0000 7f36089ea640 1 -- 172.21.15.44:0/3484464307 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3600031960 con 0x7f36040d0a30 2024-04-02T13:21:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.372+0000 7f360bc50640 1 -- 172.21.15.44:0/3484464307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36040d0a30 msgr2=0x7f360406cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.372+0000 7f360bc50640 1 --2- 172.21.15.44:0/3484464307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36040d0a30 0x7f360406cf90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f36000097a0 tx=0x7f360002b840 comp rx=0 tx=0).stop 2024-04-02T13:21:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.372+0000 7f360bc50640 1 -- 172.21.15.44:0/3484464307 shutdown_connections 2024-04-02T13:21:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.372+0000 7f360bc50640 1 --2- 172.21.15.44:0/3484464307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36040d2040 0x7f36040d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.372+0000 7f360bc50640 1 --2- 172.21.15.44:0/3484464307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36040d0a30 0x7f360406cf90 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.372+0000 7f360bc50640 1 -- 172.21.15.44:0/3484464307 >> 172.21.15.44:0/3484464307 conn(0x7f360409d470 msgr2=0x7f36040d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.372+0000 7f360bc50640 1 -- 172.21.15.44:0/3484464307 shutdown_connections 2024-04-02T13:21:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.372+0000 7f360bc50640 1 -- 172.21.15.44:0/3484464307 wait complete. 2024-04-02T13:21:31.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.373+0000 7f360bc50640 1 Processor -- start 2024-04-02T13:21:31.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.373+0000 7f360bc50640 1 -- start start 2024-04-02T13:21:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f360bc50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36040d0a30 0x7f36040b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f360bc50640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36040d0e10 0x7f360414f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f360bc50640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36040d2040 0x7f36040b6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f360bc50640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f360415e6f0 con 0x7f36040d0e10 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f360bc50640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f36040d1270 con 0x7f36040d2040 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f360bc50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f360415e870 con 0x7f36040d0a30 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f36091eb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36040d0e10 0x7f360414f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f36091eb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36040d0e10 0x7f360414f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54478/0 (socket says 172.21.15.44:54478) 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f36091eb640 1 -- 172.21.15.44:0/2268107021 learned_addr learned my addr 172.21.15.44:0/2268107021 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f360a1ed640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36040d2040 0x7f36040b6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f36099ec640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36040d0a30 0x7f36040b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f36091eb640 1 -- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36040d0a30 msgr2=0x7f36040b6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f36091eb640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36040d0a30 0x7f36040b6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f36091eb640 1 -- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36040d2040 msgr2=0x7f36040b6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f36091eb640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36040d2040 0x7f36040b6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.374+0000 7f36091eb640 1 -- 172.21.15.44:0/2268107021 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36040c8e00 con 0x7f36040d0e10 2024-04-02T13:21:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.375+0000 7f360a1ed640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36040d2040 0x7f36040b6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.375+0000 7f36091eb640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36040d0e10 0x7f360414f620 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f35f400c440 tx=0x7f35f400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.375+0000 7f35fa7fc640 1 -- 172.21.15.44:0/2268107021 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4018070 con 0x7f36040d0e10 2024-04-02T13:21:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.375+0000 7f35fa7fc640 1 -- 172.21.15.44:0/2268107021 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35f40074e0 con 0x7f36040d0e10 2024-04-02T13:21:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.375+0000 7f360bc50640 1 -- 172.21.15.44:0/2268107021 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36040c90e0 con 0x7f36040d0e10 2024-04-02T13:21:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.375+0000 7f35fa7fc640 1 -- 172.21.15.44:0/2268107021 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4004a20 con 0x7f36040d0e10 2024-04-02T13:21:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.375+0000 7f360bc50640 1 -- 172.21.15.44:0/2268107021 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36040c96a0 con 0x7f36040d0e10 2024-04-02T13:21:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.376+0000 7f360bc50640 1 -- 172.21.15.44:0/2268107021 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36040684f0 con 0x7f36040d0e10 2024-04-02T13:21:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.376+0000 7f35fa7fc640 1 -- 172.21.15.44:0/2268107021 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f35f401b070 con 0x7f36040d0e10 2024-04-02T13:21:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.377+0000 7f35fa7fc640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35e806c8f0 0x7f35e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.377+0000 7f35fa7fc640 1 -- 172.21.15.44:0/2268107021 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f35f408b2e0 con 0x7f36040d0e10 2024-04-02T13:21:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.377+0000 7f36099ec640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35e806c8f0 0x7f35e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.377+0000 7f36099ec640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35e806c8f0 0x7f35e806edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3600036000 tx=0x7f3600008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.381+0000 7f35fa7fc640 1 -- 172.21.15.44:0/2268107021 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f35f4059250 con 0x7f36040d0e10 2024-04-02T13:21:31.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.605+0000 7f360bc50640 1 -- 172.21.15.44:0/2268107021 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36040d17c0 con 0x7f36040d0e10 2024-04-02T13:21:31.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.606+0000 7f35fa7fc640 1 -- 172.21.15.44:0/2268107021 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7f35f4058b50 con 0x7f36040d0e10 2024-04-02T13:21:31.609 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:31.609 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4040417710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4040417710},{"type":"v1","addr":"172.21.15.69:6836","nonce":4040417710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2772068060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2772068060},{"type":"v1","addr":"172.21.15.44:6833","nonce":2772068060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1985405150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1985405150},{"type":"v1","addr":"172.21.15.69:6837","nonce":1985405150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:20:20.247358+0000","modified":"2024-04-02T13:20:42.332570+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/2282156931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2282156931},{"type":"v1","addr":"172.21.15.44:6835","nonce":2282156931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-04-02T13:21:31.609 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 9 2024-04-02T13:21:31.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.608+0000 7f360bc50640 1 -- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35e806c8f0 msgr2=0x7f35e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.608+0000 7f360bc50640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35e806c8f0 0x7f35e806edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3600036000 tx=0x7f3600008030 comp rx=0 tx=0).stop 2024-04-02T13:21:31.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.608+0000 7f360bc50640 1 -- 172.21.15.44:0/2268107021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36040d0e10 msgr2=0x7f360414f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.608+0000 7f360bc50640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36040d0e10 0x7f360414f620 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f35f400c440 tx=0x7f35f400c900 comp rx=0 tx=0).stop 2024-04-02T13:21:31.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.608+0000 7f360bc50640 1 -- 172.21.15.44:0/2268107021 shutdown_connections 2024-04-02T13:21:31.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.608+0000 7f360bc50640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35e806c8f0 0x7f35e806edb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.608+0000 7f360bc50640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36040d2040 0x7f36040b6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.608+0000 7f360bc50640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36040d0e10 0x7f360414f620 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.608+0000 7f360bc50640 1 --2- 172.21.15.44:0/2268107021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36040d0a30 0x7f36040b6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.608+0000 7f360bc50640 1 -- 172.21.15.44:0/2268107021 >> 172.21.15.44:0/2268107021 conn(0x7f360409d470 msgr2=0x7f36040b2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:31.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.608+0000 7f360bc50640 1 -- 172.21.15.44:0/2268107021 shutdown_connections 2024-04-02T13:21:31.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.608+0000 7f360bc50640 1 -- 172.21.15.44:0/2268107021 wait complete. 2024-04-02T13:21:31.622 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:21:31.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.720+0000 7feefffe2640 1 Processor -- start 2024-04-02T13:21:31.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.721+0000 7feefffe2640 1 -- start start 2024-04-02T13:21:31.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.721+0000 7feefffe2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7feef815e0f0 0x7feef80d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.721+0000 7feefffe2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feef806dfe0 con 0x7feef80d0630 2024-04-02T13:21:31.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.721+0000 7feefffe2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feef806e2c0 con 0x7feef806bbe0 2024-04-02T13:21:31.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.721+0000 7feefffe2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7feef806e5a0 con 0x7feef815e0f0 2024-04-02T13:21:31.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.722+0000 7feefdd7e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7feef815e0f0 0x7feef80d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.722+0000 7feefd57d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7feef80d0630 0x7feef806b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46362/0 (socket says 172.21.15.44:46362) 2024-04-02T13:21:31.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.722+0000 7feefd57d640 1 -- 172.21.15.44:0/3862565947 learned_addr learned my addr 172.21.15.44:0/3862565947 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:31.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.722+0000 7feefdd7e640 1 -- 172.21.15.44:0/3862565947 >> v1:172.21.15.69:6789/0 conn(0x7feef806bbe0 legacy=0x7feef80add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:31.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.722+0000 7feefdd7e640 1 -- 172.21.15.44:0/3862565947 >> v1:172.21.15.44:6789/0 conn(0x7feef80d0630 legacy=0x7feef806b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.722+0000 7feefcd7c640 1 -- 172.21.15.44:0/3862565947 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 702263753 0 0) 0x7feef806dfe0 con 0x7feef80d0630 2024-04-02T13:21:31.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.722+0000 7feefdd7e640 1 -- 172.21.15.44:0/3862565947 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef806e880 con 0x7feef815e0f0 2024-04-02T13:21:31.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.723+0000 7feefdd7e640 1 --2- 172.21.15.44:0/3862565947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef815e0f0 0x7feef80d00f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7feef40097a0 tx=0x7feef402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93f56a06da349e56 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:31.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.723+0000 7feefcd7c640 1 -- 172.21.15.44:0/3862565947 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef402f810 con 0x7feef815e0f0 2024-04-02T13:21:31.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.723+0000 7feefcd7c640 1 -- 172.21.15.44:0/3862565947 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feef402f9b0 con 0x7feef815e0f0 2024-04-02T13:21:31.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.723+0000 7feefcd7c640 1 -- 172.21.15.44:0/3862565947 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef402fd80 con 0x7feef815e0f0 2024-04-02T13:21:31.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.723+0000 7feefffe2640 1 -- 172.21.15.44:0/3862565947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef815e0f0 msgr2=0x7feef80d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.723+0000 7feefffe2640 1 --2- 172.21.15.44:0/3862565947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef815e0f0 0x7feef80d00f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7feef40097a0 tx=0x7feef402b9f0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.723+0000 7feefffe2640 1 -- 172.21.15.44:0/3862565947 shutdown_connections 2024-04-02T13:21:31.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.723+0000 7feefffe2640 1 --2- 172.21.15.44:0/3862565947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef815e0f0 0x7feef80d00f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.723+0000 7feefffe2640 1 -- 172.21.15.44:0/3862565947 >> 172.21.15.44:0/3862565947 conn(0x7feef80b2980 msgr2=0x7feef80b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:31.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.723+0000 7feefffe2640 1 -- 172.21.15.44:0/3862565947 shutdown_connections 2024-04-02T13:21:31.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.723+0000 7feefffe2640 1 -- 172.21.15.44:0/3862565947 wait complete. 2024-04-02T13:21:31.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.725+0000 7feefffe2640 1 Processor -- start 2024-04-02T13:21:31.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.725+0000 7feefffe2640 1 -- start start 2024-04-02T13:21:31.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.725+0000 7feefffe2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feef806bbe0 0x7feef80ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.725+0000 7feefffe2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feef80d0630 0x7feef81f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.725+0000 7feefffe2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef815e0f0 0x7feef80cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.725+0000 7feefffe2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feef815e4d0 con 0x7feef815e0f0 2024-04-02T13:21:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.725+0000 7feefffe2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7feef806eda0 con 0x7feef80d0630 2024-04-02T13:21:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.725+0000 7feefffe2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7feef815e650 con 0x7feef806bbe0 2024-04-02T13:21:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.725+0000 7feefe57f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef815e0f0 0x7feef80cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.725+0000 7feefe57f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef815e0f0 0x7feef80cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54502/0 (socket says 172.21.15.44:54502) 2024-04-02T13:21:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.725+0000 7feefe57f640 1 -- 172.21.15.44:0/749290114 learned_addr learned my addr 172.21.15.44:0/749290114 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.725+0000 7feefdd7e640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feef806bbe0 0x7feef80ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.726+0000 7feefd57d640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feef80d0630 0x7feef81f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.726+0000 7feefe57f640 1 -- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feef806bbe0 msgr2=0x7feef80ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.726+0000 7feefe57f640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feef806bbe0 0x7feef80ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.726+0000 7feefe57f640 1 -- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feef80d0630 msgr2=0x7feef81f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.726+0000 7feefe57f640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feef80d0630 0x7feef81f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.726+0000 7feefe57f640 1 -- 172.21.15.44:0/749290114 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef80cc640 con 0x7feef815e0f0 2024-04-02T13:21:31.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.726+0000 7feefd57d640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feef80d0630 0x7feef81f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:31.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.726+0000 7feefe57f640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef815e0f0 0x7feef80cbdf0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7feef000c6b0 tx=0x7feef000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:31.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.726+0000 7feeee7fc640 1 -- 172.21.15.44:0/749290114 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef0003b70 con 0x7feef815e0f0 2024-04-02T13:21:31.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.726+0000 7feeee7fc640 1 -- 172.21.15.44:0/749290114 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feef0003d30 con 0x7feef815e0f0 2024-04-02T13:21:31.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.726+0000 7feefffe2640 1 -- 172.21.15.44:0/749290114 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef80cc920 con 0x7feef815e0f0 2024-04-02T13:21:31.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.726+0000 7feeee7fc640 1 -- 172.21.15.44:0/749290114 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef0005310 con 0x7feef815e0f0 2024-04-02T13:21:31.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.726+0000 7feefffe2640 1 -- 172.21.15.44:0/749290114 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef80c83b0 con 0x7feef815e0f0 2024-04-02T13:21:31.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.727+0000 7feefffe2640 1 -- 172.21.15.44:0/749290114 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feef80c85b0 con 0x7feef815e0f0 2024-04-02T13:21:31.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.727+0000 7feeee7fc640 1 -- 172.21.15.44:0/749290114 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feef00054b0 con 0x7feef815e0f0 2024-04-02T13:21:31.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.728+0000 7feeee7fc640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feedc06c8f0 0x7feedc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:31.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.728+0000 7feeee7fc640 1 -- 172.21.15.44:0/749290114 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7feef008bca0 con 0x7feef815e0f0 2024-04-02T13:21:31.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.728+0000 7feefdd7e640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feedc06c8f0 0x7feedc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:31.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.729+0000 7feefdd7e640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feedc06c8f0 0x7feedc06edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7feef4036000 tx=0x7feef4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:31.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.732+0000 7feeee7fc640 1 -- 172.21.15.44:0/749290114 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7feef80c85b0 con 0x7feef815e0f0 2024-04-02T13:21:31.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.956+0000 7feefffe2640 1 -- 172.21.15.44:0/749290114 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feef80684f0 con 0x7feef815e0f0 2024-04-02T13:21:31.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.956+0000 7feeee7fc640 1 -- 172.21.15.44:0/749290114 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7feef0059d40 con 0x7feef815e0f0 2024-04-02T13:21:31.959 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:31.959 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4040417710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4040417710},{"type":"v1","addr":"172.21.15.69:6836","nonce":4040417710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2772068060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2772068060},{"type":"v1","addr":"172.21.15.44:6833","nonce":2772068060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1985405150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1985405150},{"type":"v1","addr":"172.21.15.69:6837","nonce":1985405150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:20:20.247358+0000","modified":"2024-04-02T13:20:42.332570+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/2282156931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2282156931},{"type":"v1","addr":"172.21.15.44:6835","nonce":2282156931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-04-02T13:21:31.959 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 9 2024-04-02T13:21:31.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.958+0000 7feefffe2640 1 -- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feedc06c8f0 msgr2=0x7feedc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.958+0000 7feefffe2640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feedc06c8f0 0x7feedc06edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7feef4036000 tx=0x7feef4008030 comp rx=0 tx=0).stop 2024-04-02T13:21:31.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.958+0000 7feefffe2640 1 -- 172.21.15.44:0/749290114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef815e0f0 msgr2=0x7feef80cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:31.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.958+0000 7feefffe2640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef815e0f0 0x7feef80cbdf0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7feef000c6b0 tx=0x7feef000cb70 comp rx=0 tx=0).stop 2024-04-02T13:21:31.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.958+0000 7feefffe2640 1 -- 172.21.15.44:0/749290114 shutdown_connections 2024-04-02T13:21:31.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.958+0000 7feefffe2640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feedc06c8f0 0x7feedc06edb0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.958+0000 7feefffe2640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef815e0f0 0x7feef80cbdf0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.958+0000 7feefffe2640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feef80d0630 0x7feef81f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.958+0000 7feefffe2640 1 --2- 172.21.15.44:0/749290114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feef806bbe0 0x7feef80ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:31.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.958+0000 7feefffe2640 1 -- 172.21.15.44:0/749290114 >> 172.21.15.44:0/749290114 conn(0x7feef80b2980 msgr2=0x7feef80b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:31.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.958+0000 7feefffe2640 1 -- 172.21.15.44:0/749290114 shutdown_connections 2024-04-02T13:21:31.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:31.958+0000 7feefffe2640 1 -- 172.21.15.44:0/749290114 wait complete. 2024-04-02T13:21:31.971 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:32.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.070+0000 7f4e908d6640 1 Processor -- start 2024-04-02T13:21:32.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.070+0000 7f4e908d6640 1 -- start start 2024-04-02T13:21:32.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.071+0000 7f4e908d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e8c06cf90 0x7f4e8c0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.071+0000 7f4e908d6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4e8c0cfe30 0x7f4e8c0d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.071+0000 7f4e908d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e8c06c1d0 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.071+0000 7f4e908d6640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4e8c06cb00 con 0x7f4e8c0cfe30 2024-04-02T13:21:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.071+0000 7f4e908d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8c06d3f0 con 0x7f4e8c06cf90 2024-04-02T13:21:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.071+0000 7f4e8ad9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4e8c0cfe30 0x7f4e8c0d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.071+0000 7f4e8a59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4e8c0d0a30 0x7f4e8c0d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46378/0 (socket says 172.21.15.44:46378) 2024-04-02T13:21:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.071+0000 7f4e8a59c640 1 -- 172.21.15.44:0/3253814522 learned_addr learned my addr 172.21.15.44:0/3253814522 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.072+0000 7f4e89d9b640 1 --2- 172.21.15.44:0/3253814522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e8c06cf90 0x7f4e8c0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.072+0000 7f4e8ad9d640 1 -- 172.21.15.44:0/3253814522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e8c06cf90 msgr2=0x7f4e8c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.072+0000 7f4e8ad9d640 1 --2- 172.21.15.44:0/3253814522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e8c06cf90 0x7f4e8c0cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.072+0000 7f4e8ad9d640 1 -- 172.21.15.44:0/3253814522 >> v1:172.21.15.44:6789/0 conn(0x7f4e8c0d0a30 legacy=0x7f4e8c0d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.072+0000 7f4e8959a640 1 -- 172.21.15.44:0/3253814522 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1591905079 0 0) 0x7f4e8c06c1d0 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.072+0000 7f4e8ad9d640 1 -- 172.21.15.44:0/3253814522 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8c06c4b0 con 0x7f4e8c0cfe30 2024-04-02T13:21:32.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.073+0000 7f4e8ad9d640 1 --2- 172.21.15.44:0/3253814522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e8c0cfe30 0x7f4e8c0d2040 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0097a0 tx=0x7f4e7c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e8816ae16f24b53 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:32.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.073+0000 7f4e8959a640 1 -- 172.21.15.44:0/3253814522 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c0315a0 con 0x7f4e8c0cfe30 2024-04-02T13:21:32.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.073+0000 7f4e8959a640 1 -- 172.21.15.44:0/3253814522 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e7c037070 con 0x7f4e8c0cfe30 2024-04-02T13:21:32.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.073+0000 7f4e8959a640 1 -- 172.21.15.44:0/3253814522 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c031960 con 0x7f4e8c0cfe30 2024-04-02T13:21:32.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.073+0000 7f4e908d6640 1 -- 172.21.15.44:0/3253814522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e8c0cfe30 msgr2=0x7f4e8c0d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.073+0000 7f4e908d6640 1 --2- 172.21.15.44:0/3253814522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e8c0cfe30 0x7f4e8c0d2040 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0097a0 tx=0x7f4e7c02b840 comp rx=0 tx=0).stop 2024-04-02T13:21:32.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.073+0000 7f4e908d6640 1 -- 172.21.15.44:0/3253814522 shutdown_connections 2024-04-02T13:21:32.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.073+0000 7f4e908d6640 1 --2- 172.21.15.44:0/3253814522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e8c0cfe30 0x7f4e8c0d2040 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.073+0000 7f4e908d6640 1 --2- 172.21.15.44:0/3253814522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e8c06cf90 0x7f4e8c0cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.073+0000 7f4e908d6640 1 -- 172.21.15.44:0/3253814522 >> 172.21.15.44:0/3253814522 conn(0x7f4e8c09d470 msgr2=0x7f4e8c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:32.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.073+0000 7f4e908d6640 1 -- 172.21.15.44:0/3253814522 shutdown_connections 2024-04-02T13:21:32.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.073+0000 7f4e908d6640 1 -- 172.21.15.44:0/3253814522 wait complete. 2024-04-02T13:21:32.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.075+0000 7f4e908d6640 1 Processor -- start 2024-04-02T13:21:32.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.075+0000 7f4e908d6640 1 -- start start 2024-04-02T13:21:32.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.075+0000 7f4e908d6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e8c06cf90 0x7f4e8c15ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.075+0000 7f4e908d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e8c0cfe30 0x7f4e8c0b5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.075+0000 7f4e908d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e8c0d0a30 0x7f4e8c0c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.075+0000 7f4e908d6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8c15e870 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.075+0000 7f4e908d6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8c15e6f0 con 0x7f4e8c06cf90 2024-04-02T13:21:32.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.075+0000 7f4e908d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8c0d4da0 con 0x7f4e8c0cfe30 2024-04-02T13:21:32.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e8ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e8c0d0a30 0x7f4e8c0c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e8ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e8c0d0a30 0x7f4e8c0c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54520/0 (socket says 172.21.15.44:54520) 2024-04-02T13:21:32.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e8ad9d640 1 -- 172.21.15.44:0/2081879697 learned_addr learned my addr 172.21.15.44:0/2081879697 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:32.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e8a59c640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e8c06cf90 0x7f4e8c15ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e89d9b640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e8c0cfe30 0x7f4e8c0b5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e8ad9d640 1 -- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e8c0cfe30 msgr2=0x7f4e8c0b5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e8ad9d640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e8c0cfe30 0x7f4e8c0b5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e8ad9d640 1 -- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e8c06cf90 msgr2=0x7f4e8c15ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e8ad9d640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e8c06cf90 0x7f4e8c15ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e8ad9d640 1 -- 172.21.15.44:0/2081879697 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8c1f5150 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e8ad9d640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e8c0d0a30 0x7f4e8c0c9830 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0098c0 tx=0x7f4e7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:32.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e82ffd640 1 -- 172.21.15.44:0/2081879697 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c032630 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e82ffd640 1 -- 172.21.15.44:0/2081879697 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e7c0327d0 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e82ffd640 1 -- 172.21.15.44:0/2081879697 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c032bc0 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e908d6640 1 -- 172.21.15.44:0/2081879697 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e8c1f53d0 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.076+0000 7f4e908d6640 1 -- 172.21.15.44:0/2081879697 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e8c15ef60 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.077+0000 7f4e908d6640 1 -- 172.21.15.44:0/2081879697 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e8c0684f0 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.078+0000 7f4e82ffd640 1 -- 172.21.15.44:0/2081879697 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4e7c04c020 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.078+0000 7f4e82ffd640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e6c06c8f0 0x7f4e6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.078+0000 7f4e82ffd640 1 -- 172.21.15.44:0/2081879697 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7f4e7c02f020 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.078+0000 7f4e8a59c640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e6c06c8f0 0x7f4e6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.079+0000 7f4e8a59c640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e6c06c8f0 0x7f4e6c06edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4e840012e0 tx=0x7f4e84007ed0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:32.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.083+0000 7f4e82ffd640 1 -- 172.21.15.44:0/2081879697 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4e7c0afe20 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.268+0000 7f4e908d6640 1 -- 172.21.15.44:0/2081879697 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e8c1f55a0 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.269+0000 7f4e82ffd640 1 -- 172.21.15.44:0/2081879697 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18899 (secure 0 0 0) 0x7f4e8c1f5150 con 0x7f4e8c0d0a30 2024-04-02T13:21:32.272 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:32.272 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:08.061253+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:20:18.247349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T13:20:19.197936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-02T13:20:22.905008+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:32.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.271+0000 7f4e908d6640 1 -- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e6c06c8f0 msgr2=0x7f4e6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.271+0000 7f4e908d6640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e6c06c8f0 0x7f4e6c06edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4e840012e0 tx=0x7f4e84007ed0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.271+0000 7f4e908d6640 1 -- 172.21.15.44:0/2081879697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e8c0d0a30 msgr2=0x7f4e8c0c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.271+0000 7f4e908d6640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e8c0d0a30 0x7f4e8c0c9830 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0098c0 tx=0x7f4e7c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.271+0000 7f4e908d6640 1 -- 172.21.15.44:0/2081879697 shutdown_connections 2024-04-02T13:21:32.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.271+0000 7f4e908d6640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e6c06c8f0 0x7f4e6c06edb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.271+0000 7f4e908d6640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e8c0d0a30 0x7f4e8c0c9830 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.271+0000 7f4e908d6640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e8c0cfe30 0x7f4e8c0b5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.271+0000 7f4e908d6640 1 --2- 172.21.15.44:0/2081879697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e8c06cf90 0x7f4e8c15ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.271+0000 7f4e908d6640 1 -- 172.21.15.44:0/2081879697 >> 172.21.15.44:0/2081879697 conn(0x7f4e8c09d470 msgr2=0x7f4e8c0cf3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:32.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.271+0000 7f4e908d6640 1 -- 172.21.15.44:0/2081879697 shutdown_connections 2024-04-02T13:21:32.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.271+0000 7f4e908d6640 1 -- 172.21.15.44:0/2081879697 wait complete. 2024-04-02T13:21:32.285 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:21:32.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.385+0000 7fea46fd6640 1 Processor -- start 2024-04-02T13:21:32.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.385+0000 7fea46fd6640 1 -- start start 2024-04-02T13:21:32.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.386+0000 7fea46fd6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fea400cd5e0 0x7fea400adc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.386+0000 7fea46fd6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea400ae1a0 0x7fea400cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.386+0000 7fea46fd6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea400ae9d0 con 0x7fea4015e3b0 2024-04-02T13:21:32.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.386+0000 7fea46fd6640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fea400ce950 con 0x7fea400cd5e0 2024-04-02T13:21:32.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.386+0000 7fea46fd6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fea400aecb0 con 0x7fea400ae1a0 2024-04-02T13:21:32.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.386+0000 7fea37fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fea400cd5e0 0x7fea400adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.386+0000 7fea37fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fea400cd5e0 0x7fea400adc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54526/0 (socket says 172.21.15.44:54526) 2024-04-02T13:21:32.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.386+0000 7fea37fff640 1 -- 172.21.15.44:0/933446205 learned_addr learned my addr 172.21.15.44:0/933446205 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:32.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.386+0000 7fea45573640 1 --2- 172.21.15.44:0/933446205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea400ae1a0 0x7fea400cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.386+0000 7fea37fff640 1 -- 172.21.15.44:0/933446205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea400ae1a0 msgr2=0x7fea400cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:32.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.386+0000 7fea37fff640 1 --2- 172.21.15.44:0/933446205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea400ae1a0 0x7fea400cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.386+0000 7fea37fff640 1 -- 172.21.15.44:0/933446205 >> v1:172.21.15.69:6789/0 conn(0x7fea4015e3b0 legacy=0x7fea400cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.386+0000 7fea37fff640 1 -- 172.21.15.44:0/933446205 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea400aef90 con 0x7fea400cd5e0 2024-04-02T13:21:32.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.387+0000 7fea37fff640 1 --2- 172.21.15.44:0/933446205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea400cd5e0 0x7fea400adc60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fea30009870 tx=0x7fea3002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58bc875e1ffd00e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:32.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.387+0000 7fea377fe640 1 -- 172.21.15.44:0/933446205 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea30034070 con 0x7fea400cd5e0 2024-04-02T13:21:32.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.387+0000 7fea377fe640 1 -- 172.21.15.44:0/933446205 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea30030840 con 0x7fea400cd5e0 2024-04-02T13:21:32.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.387+0000 7fea377fe640 1 -- 172.21.15.44:0/933446205 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea30030c10 con 0x7fea400cd5e0 2024-04-02T13:21:32.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.387+0000 7fea46fd6640 1 -- 172.21.15.44:0/933446205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea400cd5e0 msgr2=0x7fea400adc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.387+0000 7fea46fd6640 1 --2- 172.21.15.44:0/933446205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea400cd5e0 0x7fea400adc60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fea30009870 tx=0x7fea3002b980 comp rx=0 tx=0).stop 2024-04-02T13:21:32.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.388+0000 7fea46fd6640 1 -- 172.21.15.44:0/933446205 shutdown_connections 2024-04-02T13:21:32.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.388+0000 7fea46fd6640 1 --2- 172.21.15.44:0/933446205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea400cd5e0 0x7fea400adc60 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.388+0000 7fea46fd6640 1 --2- 172.21.15.44:0/933446205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea400ae1a0 0x7fea400cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.388+0000 7fea46fd6640 1 -- 172.21.15.44:0/933446205 >> 172.21.15.44:0/933446205 conn(0x7fea400b20e0 msgr2=0x7fea400b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:32.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.388+0000 7fea46fd6640 1 -- 172.21.15.44:0/933446205 shutdown_connections 2024-04-02T13:21:32.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.388+0000 7fea46fd6640 1 -- 172.21.15.44:0/933446205 wait complete. 2024-04-02T13:21:32.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.389+0000 7fea46fd6640 1 Processor -- start 2024-04-02T13:21:32.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.389+0000 7fea46fd6640 1 -- start start 2024-04-02T13:21:32.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea46fd6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea400ae1a0 0x7fea400b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea46fd6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea400cd5e0 0x7fea400c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea46fd6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea4015e3b0 0x7fea40154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea46fd6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fea4015e910 con 0x7fea4015e3b0 2024-04-02T13:21:32.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea46fd6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fea4015e790 con 0x7fea400ae1a0 2024-04-02T13:21:32.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea46fd6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fea400ae5c0 con 0x7fea400cd5e0 2024-04-02T13:21:32.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea45573640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea4015e3b0 0x7fea40154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea45573640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea4015e3b0 0x7fea40154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54536/0 (socket says 172.21.15.44:54536) 2024-04-02T13:21:32.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea45573640 1 -- 172.21.15.44:0/389452005 learned_addr learned my addr 172.21.15.44:0/389452005 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:32.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea37fff640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea400cd5e0 0x7fea400c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea44d72640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea400ae1a0 0x7fea400b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea45573640 1 -- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea400cd5e0 msgr2=0x7fea400c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea45573640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea400cd5e0 0x7fea400c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea45573640 1 -- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea400ae1a0 msgr2=0x7fea400b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea45573640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea400ae1a0 0x7fea400b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea45573640 1 -- 172.21.15.44:0/389452005 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea400ca180 con 0x7fea4015e3b0 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea37fff640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea400cd5e0 0x7fea400c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea44d72640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea400ae1a0 0x7fea400b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.390+0000 7fea45573640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea4015e3b0 0x7fea40154df0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fea38007e60 tx=0x7fea3800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.391+0000 7fea357fa640 1 -- 172.21.15.44:0/389452005 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38014420 con 0x7fea4015e3b0 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.391+0000 7fea357fa640 1 -- 172.21.15.44:0/389452005 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea38019070 con 0x7fea4015e3b0 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.391+0000 7fea46fd6640 1 -- 172.21.15.44:0/389452005 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea400ca400 con 0x7fea4015e3b0 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.391+0000 7fea357fa640 1 -- 172.21.15.44:0/389452005 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38014820 con 0x7fea4015e3b0 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.391+0000 7fea46fd6640 1 -- 172.21.15.44:0/389452005 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea400ca9c0 con 0x7fea4015e3b0 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.391+0000 7fea46fd6640 1 -- 172.21.15.44:0/389452005 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea400684f0 con 0x7fea4015e3b0 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.392+0000 7fea357fa640 1 -- 172.21.15.44:0/389452005 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fea380149c0 con 0x7fea4015e3b0 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.392+0000 7fea357fa640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fea2c06c920 0x7fea2c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.393+0000 7fea357fa640 1 -- 172.21.15.44:0/389452005 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6764+0+0 (secure 0 0 0) 0x7fea3808c7a0 con 0x7fea4015e3b0 2024-04-02T13:21:32.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.393+0000 7fea44d72640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fea2c06c920 0x7fea2c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.393+0000 7fea44d72640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fea2c06c920 0x7fea2c06ede0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fea3c000f60 tx=0x7fea3c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:32.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.397+0000 7fea357fa640 1 -- 172.21.15.44:0/389452005 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fea400ca180 con 0x7fea4015e3b0 2024-04-02T13:21:32.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.619+0000 7fea46fd6640 1 -- 172.21.15.44:0/389452005 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fea4015d310 con 0x7fea4015e3b0 2024-04-02T13:21:32.622 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:21:32.620+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:21:32.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.623+0000 7fea357fa640 1 -- 172.21.15.44:0/389452005 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) ==== 161+0+0 (secure 0 0 0) 0x7fea3805a7b0 con 0x7fea4015e3b0 2024-04-02T13:21:32.626 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:21:32.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.626+0000 7fea46fd6640 1 -- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fea2c06c920 msgr2=0x7fea2c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.626+0000 7fea46fd6640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fea2c06c920 0x7fea2c06ede0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fea3c000f60 tx=0x7fea3c009040 comp rx=0 tx=0).stop 2024-04-02T13:21:32.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.626+0000 7fea46fd6640 1 -- 172.21.15.44:0/389452005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea4015e3b0 msgr2=0x7fea40154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.626+0000 7fea46fd6640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea4015e3b0 0x7fea40154df0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fea38007e60 tx=0x7fea3800e4b0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.626+0000 7fea46fd6640 1 -- 172.21.15.44:0/389452005 shutdown_connections 2024-04-02T13:21:32.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.626+0000 7fea46fd6640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fea2c06c920 0x7fea2c06ede0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.626+0000 7fea46fd6640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea4015e3b0 0x7fea40154df0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.626+0000 7fea46fd6640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea400cd5e0 0x7fea400c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.626+0000 7fea46fd6640 1 --2- 172.21.15.44:0/389452005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea400ae1a0 0x7fea400b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.626+0000 7fea46fd6640 1 -- 172.21.15.44:0/389452005 >> 172.21.15.44:0/389452005 conn(0x7fea400b20e0 msgr2=0x7fea4006dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:32.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.626+0000 7fea46fd6640 1 -- 172.21.15.44:0/389452005 shutdown_connections 2024-04-02T13:21:32.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.626+0000 7fea46fd6640 1 -- 172.21.15.44:0/389452005 wait complete. 2024-04-02T13:21:32.631 INFO:tasks.ceph.mds.c.smithi044.stderr: -4748> 2024-04-02T13:20:25.028+0000 7f14fc09d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:21:32.640 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T13:21:32.660 INFO:tasks.ceph.mds.c.smithi044.stdout:starting mds.c at 2024-04-02T13:21:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.743+0000 7fd096069640 1 Processor -- start 2024-04-02T13:21:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.743+0000 7fd096069640 1 -- start start 2024-04-02T13:21:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd096069640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd090071200 0x7fd0900ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd096069640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0900d00a0 0x7fd0900cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd096069640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0900ce9f0 con 0x7fd09015a940 2024-04-02T13:21:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd096069640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd09006ef80 con 0x7fd0900d00a0 2024-04-02T13:21:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd096069640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd0900ce020 con 0x7fd090071200 2024-04-02T13:21:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd08ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0900d00a0 0x7fd0900cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd08f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd090071200 0x7fd0900ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd08effd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd09015a940 0x7fd09006e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:44952/0 (socket says 172.21.15.44:44952) 2024-04-02T13:21:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd08effd640 1 -- 172.21.15.44:0/1575583737 learned_addr learned my addr 172.21.15.44:0/1575583737 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd08e7fc640 1 -- 172.21.15.44:0/1575583737 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1474263879 0 0) 0x7fd0900ce9f0 con 0x7fd09015a940 2024-04-02T13:21:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd08e7fc640 1 -- 172.21.15.44:0/1575583737 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd078003520 con 0x7fd09015a940 2024-04-02T13:21:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd08ffff640 1 -- 172.21.15.44:0/1575583737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd090071200 msgr2=0x7fd0900ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd08ffff640 1 --2- 172.21.15.44:0/1575583737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd090071200 0x7fd0900ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd08ffff640 1 -- 172.21.15.44:0/1575583737 >> v1:172.21.15.69:6789/0 conn(0x7fd09015a940 legacy=0x7fd09006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.744+0000 7fd08ffff640 1 -- 172.21.15.44:0/1575583737 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09006e9c0 con 0x7fd0900d00a0 2024-04-02T13:21:32.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.745+0000 7fd08ffff640 1 --2- 172.21.15.44:0/1575583737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0900d00a0 0x7fd0900cc640 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd088009d90 tx=0x7fd08802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71b134b48b33568a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:32.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.745+0000 7fd08e7fc640 1 -- 172.21.15.44:0/1575583737 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088039070 con 0x7fd0900d00a0 2024-04-02T13:21:32.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.745+0000 7fd08e7fc640 1 -- 172.21.15.44:0/1575583737 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd08802cbc0 con 0x7fd0900d00a0 2024-04-02T13:21:32.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.745+0000 7fd08e7fc640 1 -- 172.21.15.44:0/1575583737 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088034440 con 0x7fd0900d00a0 2024-04-02T13:21:32.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.746+0000 7fd096069640 1 -- 172.21.15.44:0/1575583737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0900d00a0 msgr2=0x7fd0900cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.746+0000 7fd096069640 1 --2- 172.21.15.44:0/1575583737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0900d00a0 0x7fd0900cc640 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd088009d90 tx=0x7fd08802be10 comp rx=0 tx=0).stop 2024-04-02T13:21:32.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.746+0000 7fd096069640 1 -- 172.21.15.44:0/1575583737 shutdown_connections 2024-04-02T13:21:32.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.746+0000 7fd096069640 1 --2- 172.21.15.44:0/1575583737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0900d00a0 0x7fd0900cc640 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.746+0000 7fd096069640 1 --2- 172.21.15.44:0/1575583737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd090071200 0x7fd0900ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.746+0000 7fd096069640 1 -- 172.21.15.44:0/1575583737 >> 172.21.15.44:0/1575583737 conn(0x7fd09006b920 msgr2=0x7fd09006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:32.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.746+0000 7fd096069640 1 -- 172.21.15.44:0/1575583737 shutdown_connections 2024-04-02T13:21:32.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.746+0000 7fd096069640 1 -- 172.21.15.44:0/1575583737 wait complete. 2024-04-02T13:21:32.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.747+0000 7fd096069640 1 Processor -- start 2024-04-02T13:21:32.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.747+0000 7fd096069640 1 -- start start 2024-04-02T13:21:32.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd096069640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd090071200 0x7fd0900b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd096069640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0900d00a0 0x7fd0901500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd096069640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd09015a940 0x7fd090156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd096069640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd0900d0550 con 0x7fd090071200 2024-04-02T13:21:32.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd096069640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd0900cdcb0 con 0x7fd09015a940 2024-04-02T13:21:32.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd096069640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd0900d06d0 con 0x7fd0900d00a0 2024-04-02T13:21:32.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd08f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd090071200 0x7fd0900b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd08f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd090071200 0x7fd0900b4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54558/0 (socket says 172.21.15.44:54558) 2024-04-02T13:21:32.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd08f7fe640 1 -- 172.21.15.44:0/3714993924 learned_addr learned my addr 172.21.15.44:0/3714993924 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:32.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd08ffff640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd09015a940 0x7fd090156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd08effd640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0900d00a0 0x7fd0901500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd08f7fe640 1 -- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0900d00a0 msgr2=0x7fd0901500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd08f7fe640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0900d00a0 0x7fd0901500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd08f7fe640 1 -- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd09015a940 msgr2=0x7fd090156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd08f7fe640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd09015a940 0x7fd090156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd08f7fe640 1 -- 172.21.15.44:0/3714993924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09009cdd0 con 0x7fd090071200 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.748+0000 7fd08ffff640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd09015a940 0x7fd090156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.749+0000 7fd08f7fe640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd090071200 0x7fd0900b4560 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd08000e8a0 tx=0x7fd08000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.749+0000 7fd077fff640 1 -- 172.21.15.44:0/3714993924 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080017070 con 0x7fd090071200 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.749+0000 7fd077fff640 1 -- 172.21.15.44:0/3714993924 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd08000c940 con 0x7fd090071200 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.749+0000 7fd096069640 1 -- 172.21.15.44:0/3714993924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09009d0b0 con 0x7fd090071200 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.749+0000 7fd077fff640 1 -- 172.21.15.44:0/3714993924 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08000cd30 con 0x7fd090071200 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.749+0000 7fd096069640 1 -- 172.21.15.44:0/3714993924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd09015ad20 con 0x7fd090071200 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.750+0000 7fd096069640 1 -- 172.21.15.44:0/3714993924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0900684f0 con 0x7fd090071200 2024-04-02T13:21:32.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.750+0000 7fd077fff640 1 -- 172.21.15.44:0/3714993924 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd08001a070 con 0x7fd090071200 2024-04-02T13:21:32.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.751+0000 7fd077fff640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd07c06c920 0x7fd07c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:32.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.751+0000 7fd077fff640 1 -- 172.21.15.44:0/3714993924 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6850+0+0 (secure 0 0 0) 0x7fd08008c200 con 0x7fd090071200 2024-04-02T13:21:32.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.751+0000 7fd08effd640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd07c06c920 0x7fd07c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:32.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.751+0000 7fd08effd640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd07c06c920 0x7fd07c06ede0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd090154630 tx=0x7fd084009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:32.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.755+0000 7fd077fff640 1 -- 172.21.15.44:0/3714993924 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd08005a1b0 con 0x7fd090071200 2024-04-02T13:21:32.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:32.981+0000 7fd096069640 1 -- 172.21.15.44:0/3714993924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd0900b2a70 con 0x7fd090071200 2024-04-02T13:21:33.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.627+0000 7fd077fff640 1 -- 172.21.15.44:0/3714993924 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7fd080059ab0 con 0x7fd090071200 2024-04-02T13:21:33.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.630+0000 7fd096069640 1 -- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd07c06c920 msgr2=0x7fd07c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:33.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.630+0000 7fd096069640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd07c06c920 0x7fd07c06ede0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd090154630 tx=0x7fd084009040 comp rx=0 tx=0).stop 2024-04-02T13:21:33.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.631+0000 7fd096069640 1 -- 172.21.15.44:0/3714993924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd090071200 msgr2=0x7fd0900b4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:33.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.631+0000 7fd096069640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd090071200 0x7fd0900b4560 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd08000e8a0 tx=0x7fd08000ed60 comp rx=0 tx=0).stop 2024-04-02T13:21:33.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.631+0000 7fd096069640 1 -- 172.21.15.44:0/3714993924 shutdown_connections 2024-04-02T13:21:33.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.631+0000 7fd096069640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd07c06c920 0x7fd07c06ede0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:33.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.631+0000 7fd096069640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd09015a940 0x7fd090156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:33.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.631+0000 7fd096069640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0900d00a0 0x7fd0901500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:33.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.631+0000 7fd096069640 1 --2- 172.21.15.44:0/3714993924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd090071200 0x7fd0900b4560 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:33.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.631+0000 7fd096069640 1 -- 172.21.15.44:0/3714993924 >> 172.21.15.44:0/3714993924 conn(0x7fd09006b920 msgr2=0x7fd0900b0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:33.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.631+0000 7fd096069640 1 -- 172.21.15.44:0/3714993924 shutdown_connections 2024-04-02T13:21:33.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.631+0000 7fd096069640 1 -- 172.21.15.44:0/3714993924 wait complete. 2024-04-02T13:21:33.647 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T13:21:33.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.799+0000 7fcb99360640 1 Processor -- start 2024-04-02T13:21:33.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.799+0000 7fcb99360640 1 -- start start 2024-04-02T13:21:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.799+0000 7fcb99360640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcb9415e5d0 0x7fcb940d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.799+0000 7fcb99360640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcb940b4180 0x7fcb940b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.799+0000 7fcb99360640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb940d1630 con 0x7fcb9406cff0 2024-04-02T13:21:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.799+0000 7fcb99360640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fcb940d1bf0 con 0x7fcb9415e5d0 2024-04-02T13:21:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.799+0000 7fcb99360640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcb940d1910 con 0x7fcb940b4180 2024-04-02T13:21:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.800+0000 7fcb927fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcb9415e5d0 0x7fcb940d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.800+0000 7fcb927fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcb9415e5d0 0x7fcb940d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54564/0 (socket says 172.21.15.44:54564) 2024-04-02T13:21:33.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.800+0000 7fcb927fc640 1 -- 172.21.15.44:0/640826379 learned_addr learned my addr 172.21.15.44:0/640826379 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:33.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.800+0000 7fcb937fe640 1 --2- 172.21.15.44:0/640826379 >> v2:172.21.15.69:3300/0 conn(0x7fcb940b4180 0x7fcb940b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:33.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.800+0000 7fcb927fc640 1 -- 172.21.15.44:0/640826379 >> v2:172.21.15.69:3300/0 conn(0x7fcb940b4180 msgr2=0x7fcb940b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:33.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.800+0000 7fcb927fc640 1 --2- 172.21.15.44:0/640826379 >> v2:172.21.15.69:3300/0 conn(0x7fcb940b4180 0x7fcb940b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:33.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.800+0000 7fcb927fc640 1 -- 172.21.15.44:0/640826379 >> v1:172.21.15.69:6789/0 conn(0x7fcb9406cff0 legacy=0x7fcb9406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:33.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.800+0000 7fcb927fc640 1 -- 172.21.15.44:0/640826379 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb940b1540 con 0x7fcb9415e5d0 2024-04-02T13:21:33.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.801+0000 7fcb927fc640 1 --2- 172.21.15.44:0/640826379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb9415e5d0 0x7fcb940d0d30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fcb9409b680 tx=0x7fcb8802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f52e83fbad0a6c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:33.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.801+0000 7fcb91ffb640 1 -- 172.21.15.44:0/640826379 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88035020 con 0x7fcb9415e5d0 2024-04-02T13:21:33.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.801+0000 7fcb91ffb640 1 -- 172.21.15.44:0/640826379 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb88031870 con 0x7fcb9415e5d0 2024-04-02T13:21:33.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.801+0000 7fcb91ffb640 1 -- 172.21.15.44:0/640826379 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88031c40 con 0x7fcb9415e5d0 2024-04-02T13:21:33.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.801+0000 7fcb99360640 1 -- 172.21.15.44:0/640826379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb9415e5d0 msgr2=0x7fcb940d0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:33.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.801+0000 7fcb99360640 1 --2- 172.21.15.44:0/640826379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb9415e5d0 0x7fcb940d0d30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fcb9409b680 tx=0x7fcb8802b910 comp rx=0 tx=0).stop 2024-04-02T13:21:33.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.801+0000 7fcb99360640 1 -- 172.21.15.44:0/640826379 shutdown_connections 2024-04-02T13:21:33.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.801+0000 7fcb99360640 1 --2- 172.21.15.44:0/640826379 >> v2:172.21.15.69:3300/0 conn(0x7fcb940b4180 0x7fcb940b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.801+0000 7fcb99360640 1 --2- 172.21.15.44:0/640826379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb9415e5d0 0x7fcb940d0d30 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.801+0000 7fcb99360640 1 -- 172.21.15.44:0/640826379 >> 172.21.15.44:0/640826379 conn(0x7fcb9409aa20 msgr2=0x7fcb9409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.801+0000 7fcb99360640 1 -- 172.21.15.44:0/640826379 shutdown_connections 2024-04-02T13:21:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.801+0000 7fcb99360640 1 -- 172.21.15.44:0/640826379 wait complete. 2024-04-02T13:21:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.803+0000 7fcb99360640 1 Processor -- start 2024-04-02T13:21:33.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.803+0000 7fcb99360640 1 -- start start 2024-04-02T13:21:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb99360640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb9406cff0 0x7fcb940c8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb99360640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb940b4180 0x7fcb941545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb99360640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb9415e5d0 0x7fcb94158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb99360640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcb940d13f0 con 0x7fcb9406cff0 2024-04-02T13:21:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb99360640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcb940d1270 con 0x7fcb940b4180 2024-04-02T13:21:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb99360640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcb940b2280 con 0x7fcb9415e5d0 2024-04-02T13:21:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb92ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb9406cff0 0x7fcb940c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb92ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb9406cff0 0x7fcb940c8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54578/0 (socket says 172.21.15.44:54578) 2024-04-02T13:21:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb92ffd640 1 -- 172.21.15.44:0/586265954 learned_addr learned my addr 172.21.15.44:0/586265954 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb927fc640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb940b4180 0x7fcb941545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb937fe640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb9415e5d0 0x7fcb94158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb92ffd640 1 -- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb9415e5d0 msgr2=0x7fcb94158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb92ffd640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb9415e5d0 0x7fcb94158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb92ffd640 1 -- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb940b4180 msgr2=0x7fcb941545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb92ffd640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb940b4180 0x7fcb941545a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb92ffd640 1 -- 172.21.15.44:0/586265954 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb940b6e00 con 0x7fcb9406cff0 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb92ffd640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb9406cff0 0x7fcb940c8450 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400c400 tx=0x7fcb8400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb7b7fe640 1 -- 172.21.15.44:0/586265954 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb840052d0 con 0x7fcb9406cff0 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb7b7fe640 1 -- 172.21.15.44:0/586265954 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb84005490 con 0x7fcb9406cff0 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb99360640 1 -- 172.21.15.44:0/586265954 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb940c7970 con 0x7fcb9406cff0 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.804+0000 7fcb7b7fe640 1 -- 172.21.15.44:0/586265954 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb84007260 con 0x7fcb9406cff0 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.805+0000 7fcb99360640 1 -- 172.21.15.44:0/586265954 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb940c7f30 con 0x7fcb9406cff0 2024-04-02T13:21:33.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.806+0000 7fcb99360640 1 -- 172.21.15.44:0/586265954 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb940684f0 con 0x7fcb9406cff0 2024-04-02T13:21:33.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.806+0000 7fcb7b7fe640 1 -- 172.21.15.44:0/586265954 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcb84007400 con 0x7fcb9406cff0 2024-04-02T13:21:33.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.806+0000 7fcb7b7fe640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb8006c8f0 0x7fcb8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:33.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.807+0000 7fcb7b7fe640 1 -- 172.21.15.44:0/586265954 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6850+0+0 (secure 0 0 0) 0x7fcb8408b4b0 con 0x7fcb9406cff0 2024-04-02T13:21:33.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.807+0000 7fcb927fc640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb8006c8f0 0x7fcb8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:33.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.807+0000 7fcb927fc640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb8006c8f0 0x7fcb8006edb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcb88036000 tx=0x7fcb8803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:33.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.811+0000 7fcb7b7fe640 1 -- 172.21.15.44:0/586265954 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fcb84058e00 con 0x7fcb9406cff0 2024-04-02T13:21:33.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:33.996+0000 7fcb99360640 1 -- 172.21.15.44:0/586265954 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fcb9406cdd0 con 0x7fcb9406cff0 2024-04-02T13:21:34.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.629+0000 7fcb7b7fe640 1 -- 172.21.15.44:0/586265954 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) ==== 181+0+0 (secure 0 0 0) 0x7fcb8405d100 con 0x7fcb9406cff0 2024-04-02T13:21:34.632 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' removed 2024-04-02T13:21:34.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.633+0000 7fcb99360640 1 -- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb8006c8f0 msgr2=0x7fcb8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:34.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.633+0000 7fcb99360640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb8006c8f0 0x7fcb8006edb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcb88036000 tx=0x7fcb8803c040 comp rx=0 tx=0).stop 2024-04-02T13:21:34.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.633+0000 7fcb99360640 1 -- 172.21.15.44:0/586265954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb9406cff0 msgr2=0x7fcb940c8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:34.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.633+0000 7fcb99360640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb9406cff0 0x7fcb940c8450 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400c400 tx=0x7fcb8400c8c0 comp rx=0 tx=0).stop 2024-04-02T13:21:34.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.633+0000 7fcb99360640 1 -- 172.21.15.44:0/586265954 shutdown_connections 2024-04-02T13:21:34.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.633+0000 7fcb99360640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb8006c8f0 0x7fcb8006edb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:34.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.633+0000 7fcb99360640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb9415e5d0 0x7fcb94158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:34.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.633+0000 7fcb99360640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb940b4180 0x7fcb941545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:34.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.633+0000 7fcb99360640 1 --2- 172.21.15.44:0/586265954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb9406cff0 0x7fcb940c8450 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:34.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.633+0000 7fcb99360640 1 -- 172.21.15.44:0/586265954 >> 172.21.15.44:0/586265954 conn(0x7fcb9409aa20 msgr2=0x7fcb940b2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:34.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.633+0000 7fcb99360640 1 -- 172.21.15.44:0/586265954 shutdown_connections 2024-04-02T13:21:34.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.633+0000 7fcb99360640 1 -- 172.21.15.44:0/586265954 wait complete. 2024-04-02T13:21:34.649 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T13:21:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.758+0000 7f077a32b640 1 Processor -- start 2024-04-02T13:21:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.758+0000 7f077a32b640 1 -- start start 2024-04-02T13:21:34.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.759+0000 7f077a32b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07740d2e00 0x7f07740b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:34.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.759+0000 7f077a32b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f07740cfdc0 0x7f07740d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:34.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.759+0000 7f077a32b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f077406de80 con 0x7f077406daa0 2024-04-02T13:21:34.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.759+0000 7f077a32b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f07740cf930 con 0x7f07740cfdc0 2024-04-02T13:21:34.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.759+0000 7f077a32b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f07740cf000 con 0x7f07740d2e00 2024-04-02T13:21:34.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.759+0000 7f07788c8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f07740cfdc0 0x7f07740d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:34.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.759+0000 7f0773fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f077406daa0 0x7f07740d26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42276/0 (socket says 172.21.15.44:42276) 2024-04-02T13:21:34.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.759+0000 7f0773fff640 1 -- 172.21.15.44:0/1972399530 learned_addr learned my addr 172.21.15.44:0/1972399530 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:34.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.760+0000 7f07737fe640 1 --2- 172.21.15.44:0/1972399530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07740d2e00 0x7f07740b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:34.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.760+0000 7f07788c8640 1 -- 172.21.15.44:0/1972399530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07740d2e00 msgr2=0x7f07740b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:34.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.760+0000 7f07788c8640 1 --2- 172.21.15.44:0/1972399530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07740d2e00 0x7f07740b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:34.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.760+0000 7f07788c8640 1 -- 172.21.15.44:0/1972399530 >> v1:172.21.15.44:6789/0 conn(0x7f077406daa0 legacy=0x7f07740d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:34.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.760+0000 7f0772ffd640 1 -- 172.21.15.44:0/1972399530 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2255610297 0 0) 0x7f077406de80 con 0x7f077406daa0 2024-04-02T13:21:34.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.760+0000 7f07788c8640 1 -- 172.21.15.44:0/1972399530 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07740cf2e0 con 0x7f07740cfdc0 2024-04-02T13:21:34.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.760+0000 7f07737fe640 1 --2- 172.21.15.44:0/1972399530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07740d2e00 0x7f07740b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:34.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.760+0000 7f07788c8640 1 --2- 172.21.15.44:0/1972399530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07740cfdc0 0x7f07740d11c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f076c009d90 tx=0x7f076c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16635f45527a7335 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:34.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.760+0000 7f0772ffd640 1 -- 172.21.15.44:0/1972399530 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076c039070 con 0x7f07740cfdc0 2024-04-02T13:21:34.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.761+0000 7f0772ffd640 1 -- 172.21.15.44:0/1972399530 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f076c02cc50 con 0x7f07740cfdc0 2024-04-02T13:21:34.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.761+0000 7f0772ffd640 1 -- 172.21.15.44:0/1972399530 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076c034440 con 0x7f07740cfdc0 2024-04-02T13:21:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.761+0000 7f077a32b640 1 -- 172.21.15.44:0/1972399530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07740cfdc0 msgr2=0x7f07740d11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.761+0000 7f077a32b640 1 --2- 172.21.15.44:0/1972399530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07740cfdc0 0x7f07740d11c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f076c009d90 tx=0x7f076c02bea0 comp rx=0 tx=0).stop 2024-04-02T13:21:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.761+0000 7f077a32b640 1 -- 172.21.15.44:0/1972399530 shutdown_connections 2024-04-02T13:21:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.761+0000 7f077a32b640 1 --2- 172.21.15.44:0/1972399530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07740cfdc0 0x7f07740d11c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.761+0000 7f077a32b640 1 --2- 172.21.15.44:0/1972399530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07740d2e00 0x7f07740b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.761+0000 7f077a32b640 1 -- 172.21.15.44:0/1972399530 >> 172.21.15.44:0/1972399530 conn(0x7f077409d430 msgr2=0x7f07740d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.761+0000 7f077a32b640 1 -- 172.21.15.44:0/1972399530 shutdown_connections 2024-04-02T13:21:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.761+0000 7f077a32b640 1 -- 172.21.15.44:0/1972399530 wait complete. 2024-04-02T13:21:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.762+0000 7f077a32b640 1 Processor -- start 2024-04-02T13:21:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.763+0000 7f077a32b640 1 -- start start 2024-04-02T13:21:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.763+0000 7f077a32b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f077406daa0 0x7f07740cda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.763+0000 7f077a32b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07740cfdc0 0x7f077415e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.763+0000 7f077a32b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07740d2e00 0x7f07740b5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.763+0000 7f077a32b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f07740ceda0 con 0x7f07740d2e00 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.763+0000 7f077a32b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f07740cec20 con 0x7f077406daa0 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.763+0000 7f077a32b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f077415dd60 con 0x7f07740cfdc0 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.763+0000 7f07788c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07740d2e00 0x7f07740b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.763+0000 7f07788c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07740d2e00 0x7f07740b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38004/0 (socket says 172.21.15.44:38004) 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.763+0000 7f07788c8640 1 -- 172.21.15.44:0/206515217 learned_addr learned my addr 172.21.15.44:0/206515217 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.763+0000 7f07737fe640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07740cfdc0 0x7f077415e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.763+0000 7f0773fff640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f077406daa0 0x7f07740cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.764+0000 7f07788c8640 1 -- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07740cfdc0 msgr2=0x7f077415e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.764+0000 7f07788c8640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07740cfdc0 0x7f077415e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.764+0000 7f07788c8640 1 -- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f077406daa0 msgr2=0x7f07740cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.764+0000 7f07788c8640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f077406daa0 0x7f07740cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.764+0000 7f07788c8640 1 -- 172.21.15.44:0/206515217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0774160820 con 0x7f07740d2e00 2024-04-02T13:21:34.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.764+0000 7f07788c8640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07740d2e00 0x7f07740b5960 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f076c009eb0 tx=0x7f076c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:34.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.764+0000 7f0770ff9640 1 -- 172.21.15.44:0/206515217 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076c039050 con 0x7f07740d2e00 2024-04-02T13:21:34.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.764+0000 7f0770ff9640 1 -- 172.21.15.44:0/206515217 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f076c045070 con 0x7f07740d2e00 2024-04-02T13:21:34.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.764+0000 7f077a32b640 1 -- 172.21.15.44:0/206515217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0774160b00 con 0x7f07740d2e00 2024-04-02T13:21:34.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.764+0000 7f0770ff9640 1 -- 172.21.15.44:0/206515217 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076c03e610 con 0x7f07740d2e00 2024-04-02T13:21:34.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.764+0000 7f077a32b640 1 -- 172.21.15.44:0/206515217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f077415bf00 con 0x7f07740d2e00 2024-04-02T13:21:34.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.765+0000 7f077a32b640 1 -- 172.21.15.44:0/206515217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07740d4b50 con 0x7f07740d2e00 2024-04-02T13:21:34.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.766+0000 7f0770ff9640 1 -- 172.21.15.44:0/206515217 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f076c03e7b0 con 0x7f07740d2e00 2024-04-02T13:21:34.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.766+0000 7f0770ff9640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f076006c8f0 0x7f076006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:34.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.766+0000 7f0770ff9640 1 -- 172.21.15.44:0/206515217 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6412+0+0 (secure 0 0 0) 0x7f076c0ae8d0 con 0x7f07740d2e00 2024-04-02T13:21:34.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.766+0000 7f0773fff640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f076006c8f0 0x7f076006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:34.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.767+0000 7f0773fff640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f076006c8f0 0x7f076006edb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0768000f60 tx=0x7f0768007d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:34.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.770+0000 7f0770ff9640 1 -- 172.21.15.44:0/206515217 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f07740d4b50 con 0x7f07740d2e00 2024-04-02T13:21:34.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:34.956+0000 7f077a32b640 1 -- 172.21.15.44:0/206515217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f07740684f0 con 0x7f07740d2e00 2024-04-02T13:21:35.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.631+0000 7f0770ff9640 1 -- 172.21.15.44:0/206515217 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) ==== 169+0+0 (secure 0 0 0) 0x7f076c07c510 con 0x7f07740d2e00 2024-04-02T13:21:35.634 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' removed 2024-04-02T13:21:35.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.634+0000 7f077a32b640 1 -- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f076006c8f0 msgr2=0x7f076006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:35.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.634+0000 7f077a32b640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f076006c8f0 0x7f076006edb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0768000f60 tx=0x7f0768007d50 comp rx=0 tx=0).stop 2024-04-02T13:21:35.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.634+0000 7f077a32b640 1 -- 172.21.15.44:0/206515217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07740d2e00 msgr2=0x7f07740b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:35.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.634+0000 7f077a32b640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07740d2e00 0x7f07740b5960 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f076c009eb0 tx=0x7f076c004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:35.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.635+0000 7f077a32b640 1 -- 172.21.15.44:0/206515217 shutdown_connections 2024-04-02T13:21:35.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.635+0000 7f077a32b640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f076006c8f0 0x7f076006edb0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:35.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.635+0000 7f077a32b640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07740d2e00 0x7f07740b5960 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:35.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.635+0000 7f077a32b640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07740cfdc0 0x7f077415e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:35.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.635+0000 7f077a32b640 1 --2- 172.21.15.44:0/206515217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f077406daa0 0x7f07740cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:35.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.635+0000 7f077a32b640 1 -- 172.21.15.44:0/206515217 >> 172.21.15.44:0/206515217 conn(0x7f077409d430 msgr2=0x7f07740b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:35.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.635+0000 7f077a32b640 1 -- 172.21.15.44:0/206515217 shutdown_connections 2024-04-02T13:21:35.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.635+0000 7f077a32b640 1 -- 172.21.15.44:0/206515217 wait complete. 2024-04-02T13:21:35.651 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-02T13:21:35.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.760+0000 7fa1f173d640 1 Processor -- start 2024-04-02T13:21:35.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.760+0000 7fa1f173d640 1 -- start start 2024-04-02T13:21:35.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1f173d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec15e5d0 0x7fa1ec0d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:35.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1f173d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa1ec0b4180 0x7fa1ec0b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1f173d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1ec0d1630 con 0x7fa1ec06cff0 2024-04-02T13:21:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1f173d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa1ec0d1bf0 con 0x7fa1ec0b4180 2024-04-02T13:21:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1f173d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa1ec0d1910 con 0x7fa1ec15e5d0 2024-04-02T13:21:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1eaffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa1ec06cff0 0x7fa1ec06d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42278/0 (socket says 172.21.15.44:42278) 2024-04-02T13:21:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1eaffd640 1 -- 172.21.15.44:0/3549333219 learned_addr learned my addr 172.21.15.44:0/3549333219 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1eb7fe640 1 --2- 172.21.15.44:0/3549333219 >> v2:172.21.15.44:3300/0 conn(0x7fa1ec0b4180 0x7fa1ec0b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1ea7fc640 1 -- 172.21.15.44:0/3549333219 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2849508388 0 0) 0x7fa1ec0d1630 con 0x7fa1ec06cff0 2024-04-02T13:21:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1e3fff640 1 --2- 172.21.15.44:0/3549333219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec15e5d0 0x7fa1ec0d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1ea7fc640 1 -- 172.21.15.44:0/3549333219 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1d8003520 con 0x7fa1ec06cff0 2024-04-02T13:21:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1eb7fe640 1 -- 172.21.15.44:0/3549333219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec15e5d0 msgr2=0x7fa1ec0d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1eb7fe640 1 --2- 172.21.15.44:0/3549333219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec15e5d0 0x7fa1ec0d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1eb7fe640 1 -- 172.21.15.44:0/3549333219 >> v1:172.21.15.44:6789/0 conn(0x7fa1ec06cff0 legacy=0x7fa1ec06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.761+0000 7fa1eb7fe640 1 -- 172.21.15.44:0/3549333219 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ec0b1540 con 0x7fa1ec0b4180 2024-04-02T13:21:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.762+0000 7fa1eb7fe640 1 --2- 172.21.15.44:0/3549333219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec0b4180 0x7fa1ec0b1d40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009d90 tx=0x7fa1e402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd0f1643feea0f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.762+0000 7fa1ea7fc640 1 -- 172.21.15.44:0/3549333219 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4039070 con 0x7fa1ec0b4180 2024-04-02T13:21:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.762+0000 7fa1ea7fc640 1 -- 172.21.15.44:0/3549333219 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1e402cbc0 con 0x7fa1ec0b4180 2024-04-02T13:21:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.762+0000 7fa1ea7fc640 1 -- 172.21.15.44:0/3549333219 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4034440 con 0x7fa1ec0b4180 2024-04-02T13:21:35.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.763+0000 7fa1f173d640 1 -- 172.21.15.44:0/3549333219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec0b4180 msgr2=0x7fa1ec0b1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:35.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.763+0000 7fa1f173d640 1 --2- 172.21.15.44:0/3549333219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec0b4180 0x7fa1ec0b1d40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009d90 tx=0x7fa1e402be10 comp rx=0 tx=0).stop 2024-04-02T13:21:35.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.763+0000 7fa1f173d640 1 -- 172.21.15.44:0/3549333219 shutdown_connections 2024-04-02T13:21:35.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.763+0000 7fa1f173d640 1 --2- 172.21.15.44:0/3549333219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec0b4180 0x7fa1ec0b1d40 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:35.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.763+0000 7fa1f173d640 1 --2- 172.21.15.44:0/3549333219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec15e5d0 0x7fa1ec0d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:35.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.763+0000 7fa1f173d640 1 -- 172.21.15.44:0/3549333219 >> 172.21.15.44:0/3549333219 conn(0x7fa1ec09aa20 msgr2=0x7fa1ec09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:35.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.763+0000 7fa1f173d640 1 -- 172.21.15.44:0/3549333219 shutdown_connections 2024-04-02T13:21:35.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.763+0000 7fa1f173d640 1 -- 172.21.15.44:0/3549333219 wait complete. 2024-04-02T13:21:35.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.764+0000 7fa1f173d640 1 Processor -- start 2024-04-02T13:21:35.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.764+0000 7fa1f173d640 1 -- start start 2024-04-02T13:21:35.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1f173d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec06cff0 0x7fa1ec1611f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:35.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1f173d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1ec0b4180 0x7fa1ec0c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:35.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1f173d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5d0 0x7fa1ec150030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:35.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1f173d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec0d13f0 con 0x7fa1ec15e5d0 2024-04-02T13:21:35.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1f173d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec0d1270 con 0x7fa1ec0b4180 2024-04-02T13:21:35.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1f173d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa1ec0b2280 con 0x7fa1ec06cff0 2024-04-02T13:21:35.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1eb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5d0 0x7fa1ec150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:35.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1eb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5d0 0x7fa1ec150030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38030/0 (socket says 172.21.15.44:38030) 2024-04-02T13:21:35.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1eb7fe640 1 -- 172.21.15.44:0/3036970176 learned_addr learned my addr 172.21.15.44:0/3036970176 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:35.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1eaffd640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec06cff0 0x7fa1ec1611f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:35.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1eb7fe640 1 -- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec06cff0 msgr2=0x7fa1ec1611f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:35.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1eb7fe640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec06cff0 0x7fa1ec1611f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:35.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1e3fff640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1ec0b4180 0x7fa1ec0c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:35.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1eb7fe640 1 -- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1ec0b4180 msgr2=0x7fa1ec0c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:35.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1eb7fe640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1ec0b4180 0x7fa1ec0c7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:35.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.765+0000 7fa1eb7fe640 1 -- 172.21.15.44:0/3036970176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ec0b4d00 con 0x7fa1ec15e5d0 2024-04-02T13:21:35.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.766+0000 7fa1eb7fe640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5d0 0x7fa1ec150030 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009d90 tx=0x7fa1e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:35.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.766+0000 7fa1e37fe640 1 -- 172.21.15.44:0/3036970176 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4039050 con 0x7fa1ec15e5d0 2024-04-02T13:21:35.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.766+0000 7fa1e37fe640 1 -- 172.21.15.44:0/3036970176 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1e4045070 con 0x7fa1ec15e5d0 2024-04-02T13:21:35.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.766+0000 7fa1f173d640 1 -- 172.21.15.44:0/3036970176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1ec0b4fe0 con 0x7fa1ec15e5d0 2024-04-02T13:21:35.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.766+0000 7fa1e37fe640 1 -- 172.21.15.44:0/3036970176 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4009450 con 0x7fa1ec15e5d0 2024-04-02T13:21:35.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.766+0000 7fa1f173d640 1 -- 172.21.15.44:0/3036970176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1ec0b55a0 con 0x7fa1ec15e5d0 2024-04-02T13:21:35.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.767+0000 7fa1f173d640 1 -- 172.21.15.44:0/3036970176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1ec0684f0 con 0x7fa1ec15e5d0 2024-04-02T13:21:35.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.767+0000 7fa1e37fe640 1 -- 172.21.15.44:0/3036970176 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa1e403e850 con 0x7fa1ec15e5d0 2024-04-02T13:21:35.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.768+0000 7fa1e37fe640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa1cc06c8f0 0x7fa1cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:35.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.768+0000 7fa1e37fe640 1 -- 172.21.15.44:0/3036970176 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5853+0+0 (secure 0 0 0) 0x7fa1e40ae880 con 0x7fa1ec15e5d0 2024-04-02T13:21:35.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.768+0000 7fa1eaffd640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa1cc06c8f0 0x7fa1cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:35.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.769+0000 7fa1eaffd640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa1cc06c8f0 0x7fa1cc06edb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc000f60 tx=0x7fa1dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:35.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.772+0000 7fa1e37fe640 1 -- 172.21.15.44:0/3036970176 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa1e4032e20 con 0x7fa1ec15e5d0 2024-04-02T13:21:35.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:35.957+0000 7fa1f173d640 1 -- 172.21.15.44:0/3036970176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa1ec06cdd0 con 0x7fa1ec15e5d0 2024-04-02T13:21:36.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.642+0000 7fa1e37fe640 1 -- 172.21.15.44:0/3036970176 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) ==== 178+0+0 (secure 0 0 0) 0x7fa1e407cc20 con 0x7fa1ec15e5d0 2024-04-02T13:21:36.645 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data_ec' removed 2024-04-02T13:21:36.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.646+0000 7fa1f173d640 1 -- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa1cc06c8f0 msgr2=0x7fa1cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:36.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.646+0000 7fa1f173d640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa1cc06c8f0 0x7fa1cc06edb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc000f60 tx=0x7fa1dc009040 comp rx=0 tx=0).stop 2024-04-02T13:21:36.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.646+0000 7fa1f173d640 1 -- 172.21.15.44:0/3036970176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5d0 msgr2=0x7fa1ec150030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:36.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.646+0000 7fa1f173d640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5d0 0x7fa1ec150030 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009d90 tx=0x7fa1e4004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:36.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.647+0000 7fa1f173d640 1 -- 172.21.15.44:0/3036970176 shutdown_connections 2024-04-02T13:21:36.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.647+0000 7fa1f173d640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa1cc06c8f0 0x7fa1cc06edb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:36.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.647+0000 7fa1f173d640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1ec15e5d0 0x7fa1ec150030 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:36.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.647+0000 7fa1f173d640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1ec0b4180 0x7fa1ec0c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:36.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.647+0000 7fa1f173d640 1 --2- 172.21.15.44:0/3036970176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1ec06cff0 0x7fa1ec1611f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:36.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.647+0000 7fa1f173d640 1 -- 172.21.15.44:0/3036970176 >> 172.21.15.44:0/3036970176 conn(0x7fa1ec09aa20 msgr2=0x7fa1ec0b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:36.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.648+0000 7fa1f173d640 1 -- 172.21.15.44:0/3036970176 shutdown_connections 2024-04-02T13:21:36.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.648+0000 7fa1f173d640 1 -- 172.21.15.44:0/3036970176 wait complete. 2024-04-02T13:21:36.665 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T13:21:36.665 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T13:21:36.665 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T13:21:36.665 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:21:36.665 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T13:21:36.665 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T13:21:36.666 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T13:21:36.666 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:21:36.666 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T13:21:36.666 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T13:21:36.666 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T13:21:36.666 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:21:36.666 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T13:21:36.666 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T13:21:36.667 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T13:21:36.667 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:21:36.678 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T13:21:36.679 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T13:21:36.680 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T13:21:36.680 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T13:21:36.681 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T13:21:36.681 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T13:21:36.682 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T13:21:36.682 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T13:21:36.722 INFO:tasks.ceph.mds.a:Started 2024-04-02T13:21:36.723 DEBUG:teuthology.parallel:result is None 2024-04-02T13:21:36.724 INFO:tasks.ceph.mds.c:Started 2024-04-02T13:21:36.724 DEBUG:teuthology.parallel:result is None 2024-04-02T13:21:36.725 INFO:tasks.ceph.mds.b:Started 2024-04-02T13:21:36.725 DEBUG:teuthology.parallel:result is None 2024-04-02T13:21:36.725 INFO:tasks.ceph.mds.d:Started 2024-04-02T13:21:36.726 DEBUG:teuthology.parallel:result is None 2024-04-02T13:21:36.726 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T13:21:36.800 INFO:tasks.ceph.mds.c.smithi044.stdout:starting mds.c at 2024-04-02T13:21:36.800 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:21:36.806 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-04-02T13:21:36.806 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-04-02T13:21:36.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.830+0000 7f04b3b3d640 1 Processor -- start 2024-04-02T13:21:36.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.830+0000 7f04b3b3d640 1 -- start start 2024-04-02T13:21:36.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.831+0000 7f04b3b3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ac0d2040 0x7f04ac0d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:36.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.831+0000 7f04b3b3d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04ac0d0000 con 0x7f04ac06cf90 2024-04-02T13:21:36.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.831+0000 7f04b3b3d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04ac06c1d0 con 0x7f04ac0d0a30 2024-04-02T13:21:36.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.831+0000 7f04b3b3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f04ac06cb00 con 0x7f04ac0d2040 2024-04-02T13:21:36.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.831+0000 7f04b10d8640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f04ac06cf90 0x7f04ac0cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42314/0 (socket says 172.21.15.44:42314) 2024-04-02T13:21:36.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.831+0000 7f04b10d8640 1 -- 172.21.15.44:0/4229661555 learned_addr learned my addr 172.21.15.44:0/4229661555 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:36.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.831+0000 7f04b20da640 1 --2- 172.21.15.44:0/4229661555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ac0d2040 0x7f04ac0d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:36.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.832+0000 7f04b08d7640 1 -- 172.21.15.44:0/4229661555 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1139197210 0 0) 0x7f04ac0d0000 con 0x7f04ac06cf90 2024-04-02T13:21:36.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.832+0000 7f04b08d7640 1 -- 172.21.15.44:0/4229661555 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0498003520 con 0x7f04ac06cf90 2024-04-02T13:21:36.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.832+0000 7f04b08d7640 1 -- 172.21.15.44:0/4229661555 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 750090785 0 0) 0x7f04ac06c1d0 con 0x7f04ac0d0a30 2024-04-02T13:21:36.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.832+0000 7f04b08d7640 1 -- 172.21.15.44:0/4229661555 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04ac0d0000 con 0x7f04ac0d0a30 2024-04-02T13:21:36.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.832+0000 7f04b20da640 1 -- 172.21.15.44:0/4229661555 >> v1:172.21.15.69:6789/0 conn(0x7f04ac0d0a30 legacy=0x7f04ac0d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:36.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.832+0000 7f04b08d7640 1 -- 172.21.15.44:0/4229661555 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3840531926 0 0) 0x7f0498003520 con 0x7f04ac06cf90 2024-04-02T13:21:36.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.832+0000 7f04b20da640 1 -- 172.21.15.44:0/4229661555 >> v1:172.21.15.44:6789/0 conn(0x7f04ac06cf90 legacy=0x7f04ac0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:36.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.832+0000 7f04b20da640 1 -- 172.21.15.44:0/4229661555 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac06c4b0 con 0x7f04ac0d2040 2024-04-02T13:21:36.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.833+0000 7f04b20da640 1 --2- 172.21.15.44:0/4229661555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ac0d2040 0x7f04ac0d24f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009dc0 tx=0x7f04a402bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f978123b0e0d4815 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:36.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.833+0000 7f04b08d7640 1 -- 172.21.15.44:0/4229661555 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4039070 con 0x7f04ac0d2040 2024-04-02T13:21:36.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.833+0000 7f04b08d7640 1 -- 172.21.15.44:0/4229661555 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a402c880 con 0x7f04ac0d2040 2024-04-02T13:21:36.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.833+0000 7f04b08d7640 1 -- 172.21.15.44:0/4229661555 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a8001670 con 0x7f04ac0d2040 2024-04-02T13:21:36.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.833+0000 7f04b3b3d640 1 -- 172.21.15.44:0/4229661555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ac0d2040 msgr2=0x7f04ac0d24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:36.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.833+0000 7f04b3b3d640 1 --2- 172.21.15.44:0/4229661555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ac0d2040 0x7f04ac0d24f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009dc0 tx=0x7f04a402bd50 comp rx=0 tx=0).stop 2024-04-02T13:21:36.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.833+0000 7f04b3b3d640 1 -- 172.21.15.44:0/4229661555 shutdown_connections 2024-04-02T13:21:36.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.833+0000 7f04b3b3d640 1 --2- 172.21.15.44:0/4229661555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ac0d2040 0x7f04ac0d24f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:36.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.833+0000 7f04b3b3d640 1 -- 172.21.15.44:0/4229661555 >> 172.21.15.44:0/4229661555 conn(0x7f04ac09d470 msgr2=0x7f04ac0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:36.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.833+0000 7f04b3b3d640 1 -- 172.21.15.44:0/4229661555 shutdown_connections 2024-04-02T13:21:36.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.833+0000 7f04b3b3d640 1 -- 172.21.15.44:0/4229661555 wait complete. 2024-04-02T13:21:36.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.834+0000 7f04b3b3d640 1 Processor -- start 2024-04-02T13:21:36.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.834+0000 7f04b3b3d640 1 -- start start 2024-04-02T13:21:36.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.835+0000 7f04b3b3d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ac06cf90 0x7f04ac14e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:36.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.835+0000 7f04b3b3d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04ac0d0a30 0x7f04ac0b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:36.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.835+0000 7f04b3b3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ac0d2040 0x7f04ac160bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:36.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.835+0000 7f04b3b3d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac15e6f0 con 0x7f04ac0d0a30 2024-04-02T13:21:36.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.835+0000 7f04b3b3d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac06d3f0 con 0x7f04ac06cf90 2024-04-02T13:21:36.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.835+0000 7f04b3b3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f04ac15e870 con 0x7f04ac0d2040 2024-04-02T13:21:36.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.835+0000 7f04b10d8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04ac0d0a30 0x7f04ac0b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.835+0000 7f04b10d8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04ac0d0a30 0x7f04ac0b3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38068/0 (socket says 172.21.15.44:38068) 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.835+0000 7f04b10d8640 1 -- 172.21.15.44:0/1519646233 learned_addr learned my addr 172.21.15.44:0/1519646233 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.835+0000 7f04b20da640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ac0d2040 0x7f04ac160bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.835+0000 7f04b18d9640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ac06cf90 0x7f04ac14e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.836+0000 7f04b10d8640 1 -- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ac0d2040 msgr2=0x7f04ac160bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.836+0000 7f04b10d8640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ac0d2040 0x7f04ac160bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.836+0000 7f04b10d8640 1 -- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ac06cf90 msgr2=0x7f04ac14e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.836+0000 7f04b10d8640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ac06cf90 0x7f04ac14e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.836+0000 7f04b10d8640 1 -- 172.21.15.44:0/1519646233 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac0b44a0 con 0x7f04ac0d0a30 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.836+0000 7f04b10d8640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04ac0d0a30 0x7f04ac0b3ba0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f04a8001eb0 tx=0x7f04a800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.836+0000 7f04a27fc640 1 -- 172.21.15.44:0/1519646233 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a8005ad0 con 0x7f04ac0d0a30 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.836+0000 7f04a27fc640 1 -- 172.21.15.44:0/1519646233 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a8005c70 con 0x7f04ac0d0a30 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.836+0000 7f04b3b3d640 1 -- 172.21.15.44:0/1519646233 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ac0b4780 con 0x7f04ac0d0a30 2024-04-02T13:21:36.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.836+0000 7f04a27fc640 1 -- 172.21.15.44:0/1519646233 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a80072b0 con 0x7f04ac0d0a30 2024-04-02T13:21:36.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.836+0000 7f04b3b3d640 1 -- 172.21.15.44:0/1519646233 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ac0b4d40 con 0x7f04ac0d0a30 2024-04-02T13:21:36.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.837+0000 7f04b3b3d640 1 -- 172.21.15.44:0/1519646233 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ac0684f0 con 0x7f04ac0d0a30 2024-04-02T13:21:36.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.837+0000 7f04a27fc640 1 -- 172.21.15.44:0/1519646233 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f04a8007450 con 0x7f04ac0d0a30 2024-04-02T13:21:36.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.838+0000 7f04a27fc640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f049006c8f0 0x7f049006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:36.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.838+0000 7f04a27fc640 1 -- 172.21.15.44:0/1519646233 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5335+0+0 (secure 0 0 0) 0x7f04a808b610 con 0x7f04ac0d0a30 2024-04-02T13:21:36.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.838+0000 7f04b18d9640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f049006c8f0 0x7f049006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:36.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.838+0000 7f04b18d9640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f049006c8f0 0x7f049006edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f049c0011e0 tx=0x7f049c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:36.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:36.842+0000 7f04a27fc640 1 -- 172.21.15.44:0/1519646233 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f04a8005e20 con 0x7f04ac0d0a30 2024-04-02T13:21:37.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.040+0000 7f04b3b3d640 1 -- 172.21.15.44:0/1519646233 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f04ac162d40 con 0x7f04ac0d0a30 2024-04-02T13:21:37.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.654+0000 7f04a27fc640 1 -- 172.21.15.44:0/1519646233 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) ==== 97+0+0 (secure 0 0 0) 0x7f04a80595f0 con 0x7f04ac0d0a30 2024-04-02T13:21:37.657 INFO:teuthology.orchestra.run.smithi044.stderr: removed all blocklist entries 2024-04-02T13:21:37.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.658+0000 7f04b3b3d640 1 -- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f049006c8f0 msgr2=0x7f049006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:37.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.658+0000 7f04b3b3d640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f049006c8f0 0x7f049006edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f049c0011e0 tx=0x7f049c009040 comp rx=0 tx=0).stop 2024-04-02T13:21:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.658+0000 7f04b3b3d640 1 -- 172.21.15.44:0/1519646233 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04ac0d0a30 msgr2=0x7f04ac0b3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.658+0000 7f04b3b3d640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04ac0d0a30 0x7f04ac0b3ba0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f04a8001eb0 tx=0x7f04a800cbe0 comp rx=0 tx=0).stop 2024-04-02T13:21:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.658+0000 7f04b3b3d640 1 -- 172.21.15.44:0/1519646233 shutdown_connections 2024-04-02T13:21:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.658+0000 7f04b3b3d640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f049006c8f0 0x7f049006edb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.658+0000 7f04b3b3d640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ac0d2040 0x7f04ac160bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.658+0000 7f04b3b3d640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04ac0d0a30 0x7f04ac0b3ba0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.658+0000 7f04b3b3d640 1 --2- 172.21.15.44:0/1519646233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ac06cf90 0x7f04ac14e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.658+0000 7f04b3b3d640 1 -- 172.21.15.44:0/1519646233 >> 172.21.15.44:0/1519646233 conn(0x7f04ac09d470 msgr2=0x7f04ac0b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.658+0000 7f04b3b3d640 1 -- 172.21.15.44:0/1519646233 shutdown_connections 2024-04-02T13:21:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.658+0000 7f04b3b3d640 1 -- 172.21.15.44:0/1519646233 wait complete. 2024-04-02T13:21:37.677 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T13:21:37.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.784+0000 7f23e21ba640 1 Processor -- start 2024-04-02T13:21:37.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.784+0000 7f23e21ba640 1 -- start start 2024-04-02T13:21:37.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.785+0000 7f23e21ba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f23dc0b47b0 0x7f23dc0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:37.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.785+0000 7f23e21ba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23dc0d4d70 con 0x7f23dc0d28f0 2024-04-02T13:21:37.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.785+0000 7f23e21ba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23dc0d2030 con 0x7f23dc0b5200 2024-04-02T13:21:37.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.785+0000 7f23e21ba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f23dc0d5050 con 0x7f23dc0b47b0 2024-04-02T13:21:37.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.785+0000 7f23db7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f23dc0d28f0 0x7f23dc0b40a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42330/0 (socket says 172.21.15.44:42330) 2024-04-02T13:21:37.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.785+0000 7f23db7fe640 1 -- 172.21.15.44:0/953962406 learned_addr learned my addr 172.21.15.44:0/953962406 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:37.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.785+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1719669104 0 0) 0x7f23dc0d4d70 con 0x7f23dc0d28f0 2024-04-02T13:21:37.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.785+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23c4003520 con 0x7f23dc0d28f0 2024-04-02T13:21:37.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.785+0000 7f23daffd640 1 --2- 172.21.15.44:0/953962406 >> v2:172.21.15.69:3300/0 conn(0x7f23dc0b47b0 0x7f23dc0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:37.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 697274052 0 0) 0x7f23c4003520 con 0x7f23dc0d28f0 2024-04-02T13:21:37.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 >> v2:172.21.15.69:3300/0 conn(0x7f23dc0b47b0 msgr2=0x7f23dc0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:37.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23da7fc640 1 --2- 172.21.15.44:0/953962406 >> v2:172.21.15.69:3300/0 conn(0x7f23dc0b47b0 0x7f23dc0d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:37.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 >> v1:172.21.15.69:6789/0 conn(0x7f23dc0b5200 legacy=0x7f23dc0b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:37.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23dc0b4cb0 con 0x7f23dc0d28f0 2024-04-02T13:21:37.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f23cc001670 con 0x7f23dc0d28f0 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 >> v1:172.21.15.44:6789/0 conn(0x7f23dc0d28f0 legacy=0x7f23dc0b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23da7fc640 1 --2- 172.21.15.44:0/953962406 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23c40039b0 0x7f23c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23da7fc640 1 --2- 172.21.15.44:0/953962406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23c4004800 0x7f23c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23da7fc640 1 --2- 172.21.15.44:0/953962406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23c40052e0 0x7f23c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f23dc0d5050 con 0x7f23c4004800 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f23c400c1b0 con 0x7f23c40052e0 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f23c400c3b0 con 0x7f23c40039b0 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23daffd640 1 --2- 172.21.15.44:0/953962406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23c4004800 0x7f23c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23dbfff640 1 --2- 172.21.15.44:0/953962406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23c40052e0 0x7f23c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23db7fe640 1 --2- 172.21.15.44:0/953962406 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23c40039b0 0x7f23c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23daffd640 1 -- 172.21.15.44:0/953962406 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23c40039b0 msgr2=0x7f23c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23daffd640 1 --2- 172.21.15.44:0/953962406 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23c40039b0 0x7f23c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23daffd640 1 -- 172.21.15.44:0/953962406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23c40052e0 msgr2=0x7f23c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:37.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23daffd640 1 --2- 172.21.15.44:0/953962406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23c40052e0 0x7f23c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23daffd640 1 -- 172.21.15.44:0/953962406 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c4003520 con 0x7f23c4004800 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23dbfff640 1 -- 172.21.15.44:0/953962406 reap_dead start 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.786+0000 7f23db7fe640 1 --2- 172.21.15.44:0/953962406 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23c40039b0 0x7f23c4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.787+0000 7f23daffd640 1 --2- 172.21.15.44:0/953962406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23c4004800 0x7f23c4004c80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009a90 tx=0x7f23c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4983ce8a32cba3e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.787+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d0039070 con 0x7f23c4004800 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.787+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23d002c680 con 0x7f23c4004800 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.787+0000 7f23da7fc640 1 -- 172.21.15.44:0/953962406 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d0009690 con 0x7f23c4004800 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.788+0000 7f23e21ba640 1 -- 172.21.15.44:0/953962406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23c4004800 msgr2=0x7f23c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.788+0000 7f23e21ba640 1 --2- 172.21.15.44:0/953962406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23c4004800 0x7f23c4004c80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009a90 tx=0x7f23c4005830 comp rx=0 tx=0).stop 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.788+0000 7f23e21ba640 1 -- 172.21.15.44:0/953962406 shutdown_connections 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.788+0000 7f23e21ba640 1 --2- 172.21.15.44:0/953962406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23c4004800 0x7f23c4004c80 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.788+0000 7f23e21ba640 1 -- 172.21.15.44:0/953962406 >> 172.21.15.44:0/953962406 conn(0x7f23dc0d4120 msgr2=0x7f23dc0d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.788+0000 7f23e21ba640 1 -- 172.21.15.44:0/953962406 shutdown_connections 2024-04-02T13:21:37.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.788+0000 7f23e21ba640 1 -- 172.21.15.44:0/953962406 wait complete. 2024-04-02T13:21:37.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.789+0000 7f23e21ba640 1 Processor -- start 2024-04-02T13:21:37.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.789+0000 7f23e21ba640 1 -- start start 2024-04-02T13:21:37.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23e21ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23c4004800 0x7f23dc160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:37.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23e21ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23dc0b16c0 0x7f23dc0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:37.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23e21ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23dc0b30a0 0x7f23dc0b3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:37.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23e21ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f23dc0d23a0 con 0x7f23dc0b16c0 2024-04-02T13:21:37.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23e21ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f23dc0d1470 con 0x7f23dc0b30a0 2024-04-02T13:21:37.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23e21ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f23dc0b5020 con 0x7f23c4004800 2024-04-02T13:21:37.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23daffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23dc0b16c0 0x7f23dc0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:37.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23daffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23dc0b16c0 0x7f23dc0b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38084/0 (socket says 172.21.15.44:38084) 2024-04-02T13:21:37.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23daffd640 1 -- 172.21.15.44:0/3942609376 learned_addr learned my addr 172.21.15.44:0/3942609376 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:37.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23db7fe640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23c4004800 0x7f23dc160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:37.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23daffd640 1 -- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23c4004800 msgr2=0x7f23dc160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:37.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23dbfff640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23dc0b30a0 0x7f23dc0b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:37.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23daffd640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23c4004800 0x7f23dc160160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:37.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23daffd640 1 -- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23dc0b30a0 msgr2=0x7f23dc0b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:37.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23daffd640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23dc0b30a0 0x7f23dc0b3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:37.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23daffd640 1 -- 172.21.15.44:0/3942609376 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23dc0cc410 con 0x7f23dc0b16c0 2024-04-02T13:21:37.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.790+0000 7f23daffd640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23dc0b16c0 0x7f23dc0b1b40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f23d0037040 tx=0x7f23d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:37.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.791+0000 7f23c3fff640 1 -- 172.21.15.44:0/3942609376 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d0042020 con 0x7f23dc0b16c0 2024-04-02T13:21:37.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.791+0000 7f23c3fff640 1 -- 172.21.15.44:0/3942609376 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23d003a040 con 0x7f23dc0b16c0 2024-04-02T13:21:37.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.791+0000 7f23e21ba640 1 -- 172.21.15.44:0/3942609376 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23dc0cc6b0 con 0x7f23dc0b16c0 2024-04-02T13:21:37.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.791+0000 7f23c3fff640 1 -- 172.21.15.44:0/3942609376 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d0039070 con 0x7f23dc0b16c0 2024-04-02T13:21:37.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.791+0000 7f23e21ba640 1 -- 172.21.15.44:0/3942609376 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23dc0ccc70 con 0x7f23dc0b16c0 2024-04-02T13:21:37.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.792+0000 7f23e21ba640 1 -- 172.21.15.44:0/3942609376 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23dc0684f0 con 0x7f23dc0b16c0 2024-04-02T13:21:37.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.792+0000 7f23c3fff640 1 -- 172.21.15.44:0/3942609376 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f23d0045020 con 0x7f23dc0b16c0 2024-04-02T13:21:37.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.792+0000 7f23c3fff640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f23c806c8f0 0x7f23c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:37.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.793+0000 7f23c3fff640 1 -- 172.21.15.44:0/3942609376 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f23d00ad130 con 0x7f23dc0b16c0 2024-04-02T13:21:37.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.793+0000 7f23db7fe640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f23c806c8f0 0x7f23c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:37.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.793+0000 7f23db7fe640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f23c806c8f0 0x7f23c806edb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f23cc001bf0 tx=0x7f23cc001b80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:37.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:37.797+0000 7f23c3fff640 1 -- 172.21.15.44:0/3942609376 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f23d004c030 con 0x7f23dc0b16c0 2024-04-02T13:21:38.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.056+0000 7f23e21ba640 1 -- 172.21.15.44:0/3942609376 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f23dc0caf60 con 0x7f23dc0b16c0 2024-04-02T13:21:38.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.057+0000 7f23c3fff640 1 -- 172.21.15.44:0/3942609376 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7f23d007b720 con 0x7f23dc0b16c0 2024-04-02T13:21:38.060 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:38.060 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:21:38.060 INFO:teuthology.orchestra.run.smithi044.stdout: "auth_dump": [ 2024-04-02T13:21:38.060 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.060 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.a", 2024-04-02T13:21:38.060 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmVhZrMhAAklqx5uKCFA5kiqYwyLBKpg==", 2024-04-02T13:21:38.060 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.060 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:21:38.060 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:21:38.060 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:21:38.060 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.b", 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmalQ+AxAA2SqH9CX/66OIXR7SDID97w==", 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.c", 2024-04-02T13:21:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxm8BLKORAAjMIrmdQ/bXbPk2sAnYJWTw==", 2024-04-02T13:21:38.062 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.062 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:21:38.062 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:21:38.062 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:21:38.062 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.062 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.062 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.062 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.062 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.d", 2024-04-02T13:21:38.062 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmNoaRChAAEqxWmBjBWG79Tkj6dU8FQg==", 2024-04-02T13:21:38.062 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.062 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:21:38.062 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.0", 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDfBQxm4GP9KxAA/AmOunAfVaczAWRGVXvFVg==", 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.063 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.1", 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDiBQxmDL/nKRAADF2C30+IGAXL9XwDQDKCUA==", 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.2", 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDlBQxm1gOCKBAAuYxN9jNHuRYEqc8WrsYErQ==", 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.064 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.3", 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDoBQxmLLFQJxAAFMCzphGU7TOt4nC0Nt+y+g==", 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.4", 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDuBQxmtk8QLhAAR4gm0irR6C3D2SkhFjvDKQ==", 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.065 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.5", 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDxBQxm6Ee/LBAAX5h+9ML70ztw2QH/50OzOg==", 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.066 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.067 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.6", 2024-04-02T13:21:38.067 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD0BQxm35hwKxAAvsgMmiXdLo+c7wlvb9RtVQ==", 2024-04-02T13:21:38.067 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.067 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:21:38.067 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:21:38.067 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.067 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.067 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.068 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.068 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.7", 2024-04-02T13:21:38.068 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD3BQxmYLBOKhAA1vuSOBDcFf2zT78eoTHtHQ==", 2024-04-02T13:21:38.068 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.068 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:21:38.068 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:21:38.068 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.068 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.068 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.068 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.0", 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A==", 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.1", 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxm3sYsGBAAeRCNbP5qiQ8YnRVsDoaaFQ==", 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:21:38.069 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:21:38.070 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:21:38.070 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:21:38.070 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.070 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.070 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.070 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.2", 2024-04-02T13:21:38.070 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmKNS5FBAAKvsCNmuG+fI1D/vwu6MC5w==", 2024-04-02T13:21:38.070 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.070 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:21:38.070 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:21:38.070 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:21:38.070 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:21:38.070 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.071 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.071 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.071 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.3", 2024-04-02T13:21:38.071 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmslsSHhAAPsAfALW7RimJDyXdOa2wbQ==", 2024-04-02T13:21:38.071 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.071 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:21:38.071 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:21:38.071 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:21:38.071 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:21:38.071 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.071 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.071 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.admin", 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmq+VcFRAA9CgeH75Q0sXf27cZdJthJg==", 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.bootstrap-mds", 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQACBgxmSwWOIxAAbRufv3Jo1ADa4AhnLWP+zw==", 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.072 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile bootstrap-mds" 2024-04-02T13:21:38.073 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.073 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.073 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.073 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.bootstrap-mgr", 2024-04-02T13:21:38.073 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQACBgxm7BKOIxAAhYxwSDHBxrjbtZ63Qblo3w==", 2024-04-02T13:21:38.073 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.073 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-02T13:21:38.073 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.073 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.073 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.073 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.bootstrap-osd", 2024-04-02T13:21:38.073 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQACBgxm9yaOIxAAH5tb5pBgzUw46w4eOTYcww==", 2024-04-02T13:21:38.073 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.074 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile bootstrap-osd" 2024-04-02T13:21:38.074 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.074 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.074 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.074 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.bootstrap-rbd", 2024-04-02T13:21:38.074 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQACBgxmGDGOIxAALeQYAQ4hEWfJ5byDT+FJ6g==", 2024-04-02T13:21:38.074 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.074 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-02T13:21:38.074 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.074 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.074 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.074 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-02T13:21:38.074 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQACBgxmZjiOIxAAGf2Pg+2ASlV/KIJFtOKwnQ==", 2024-04-02T13:21:38.075 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.075 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-02T13:21:38.075 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.075 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.075 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.075 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.bootstrap-rgw", 2024-04-02T13:21:38.075 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQACBgxmiD+OIxAAK9x7R6VtRdQiE2xBpBHoRg==", 2024-04-02T13:21:38.075 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.075 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-02T13:21:38.075 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.075 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.075 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.075 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.x", 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmR5ICLxAAtQeg/gq0OAq6e0mroBZVxQ==", 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.y", 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmxoquKhAAE5fMnJIuCy5hGJ1aEAx8Vg==", 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:21:38.076 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.077 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:38.077 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-02T13:21:38.077 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:21:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.059+0000 7f23e21ba640 1 -- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f23c806c8f0 msgr2=0x7f23c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.059+0000 7f23e21ba640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f23c806c8f0 0x7f23c806edb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f23cc001bf0 tx=0x7f23cc001b80 comp rx=0 tx=0).stop 2024-04-02T13:21:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.059+0000 7f23e21ba640 1 -- 172.21.15.44:0/3942609376 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23dc0b16c0 msgr2=0x7f23dc0b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.059+0000 7f23e21ba640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23dc0b16c0 0x7f23dc0b1b40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f23d0037040 tx=0x7f23d0004730 comp rx=0 tx=0).stop 2024-04-02T13:21:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.059+0000 7f23e21ba640 1 -- 172.21.15.44:0/3942609376 shutdown_connections 2024-04-02T13:21:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.059+0000 7f23e21ba640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f23c806c8f0 0x7f23c806edb0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.059+0000 7f23e21ba640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23dc0b30a0 0x7f23dc0b3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.059+0000 7f23e21ba640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23dc0b16c0 0x7f23dc0b1b40 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.059+0000 7f23e21ba640 1 --2- 172.21.15.44:0/3942609376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23c4004800 0x7f23dc160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.059+0000 7f23e21ba640 1 -- 172.21.15.44:0/3942609376 >> 172.21.15.44:0/3942609376 conn(0x7f23dc0d4120 msgr2=0x7f23dc06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.059+0000 7f23e21ba640 1 -- 172.21.15.44:0/3942609376 shutdown_connections 2024-04-02T13:21:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.059+0000 7f23e21ba640 1 -- 172.21.15.44:0/3942609376 wait complete. 2024-04-02T13:21:38.079 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-02T13:21:38.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.179+0000 7f6144d7b640 1 Processor -- start 2024-04-02T13:21:38.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.179+0000 7f6144d7b640 1 -- start start 2024-04-02T13:21:38.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f6144d7b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f614015e4a0 0x7f61400aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f6144d7b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f61400d0c50 0x7f61400d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f6144d7b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f614006ce40 0x7f614006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f6144d7b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f61400cc600 con 0x7f61400d0c50 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f6144d7b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f61400cc8e0 con 0x7f614015e4a0 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f6144d7b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f61400ccbc0 con 0x7f614006ce40 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f613dd9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f61400d0c50 0x7f61400d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f613dd9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f61400d0c50 0x7f61400d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38094/0 (socket says 172.21.15.44:38094) 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f613dd9b640 1 -- 172.21.15.44:0/6366437 learned_addr learned my addr 172.21.15.44:0/6366437 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f613ed9d640 1 --2- 172.21.15.44:0/6366437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f614006ce40 0x7f614006d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f613e59c640 1 --2- 172.21.15.44:0/6366437 >> v2:172.21.15.69:3300/0 conn(0x7f614015e4a0 0x7f61400aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f613dd9b640 1 -- 172.21.15.44:0/6366437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f614006ce40 msgr2=0x7f614006d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f613dd9b640 1 --2- 172.21.15.44:0/6366437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f614006ce40 0x7f614006d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f613dd9b640 1 -- 172.21.15.44:0/6366437 >> v2:172.21.15.69:3300/0 conn(0x7f614015e4a0 msgr2=0x7f61400aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f613dd9b640 1 --2- 172.21.15.44:0/6366437 >> v2:172.21.15.69:3300/0 conn(0x7f614015e4a0 0x7f61400aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f613dd9b640 1 -- 172.21.15.44:0/6366437 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61400ccea0 con 0x7f61400d0c50 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f613ed9d640 1 --2- 172.21.15.44:0/6366437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f614006ce40 0x7f614006d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.180+0000 7f613e59c640 1 --2- 172.21.15.44:0/6366437 >> v2:172.21.15.69:3300/0 conn(0x7f614015e4a0 0x7f61400aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:38.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.181+0000 7f613dd9b640 1 --2- 172.21.15.44:0/6366437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61400d0c50 0x7f61400d10b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f61400d1f80 tx=0x7f613402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d360886b7fbfc0d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:38.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.181+0000 7f613d59a640 1 -- 172.21.15.44:0/6366437 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6134035020 con 0x7f61400d0c50 2024-04-02T13:21:38.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.181+0000 7f613d59a640 1 -- 172.21.15.44:0/6366437 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6134031870 con 0x7f61400d0c50 2024-04-02T13:21:38.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.181+0000 7f613d59a640 1 -- 172.21.15.44:0/6366437 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6134031c40 con 0x7f61400d0c50 2024-04-02T13:21:38.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.182+0000 7f6144d7b640 1 -- 172.21.15.44:0/6366437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61400d0c50 msgr2=0x7f61400d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.182+0000 7f6144d7b640 1 --2- 172.21.15.44:0/6366437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61400d0c50 0x7f61400d10b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f61400d1f80 tx=0x7f613402b910 comp rx=0 tx=0).stop 2024-04-02T13:21:38.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.182+0000 7f6144d7b640 1 -- 172.21.15.44:0/6366437 shutdown_connections 2024-04-02T13:21:38.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.182+0000 7f6144d7b640 1 --2- 172.21.15.44:0/6366437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f614006ce40 0x7f614006d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.182+0000 7f6144d7b640 1 --2- 172.21.15.44:0/6366437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61400d0c50 0x7f61400d10b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.182+0000 7f6144d7b640 1 --2- 172.21.15.44:0/6366437 >> v2:172.21.15.69:3300/0 conn(0x7f614015e4a0 0x7f61400aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.182+0000 7f6144d7b640 1 -- 172.21.15.44:0/6366437 >> 172.21.15.44:0/6366437 conn(0x7f614009aa20 msgr2=0x7f614009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:38.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.182+0000 7f6144d7b640 1 -- 172.21.15.44:0/6366437 shutdown_connections 2024-04-02T13:21:38.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.182+0000 7f6144d7b640 1 -- 172.21.15.44:0/6366437 wait complete. 2024-04-02T13:21:38.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.183+0000 7f6144d7b640 1 Processor -- start 2024-04-02T13:21:38.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.183+0000 7f6144d7b640 1 -- start start 2024-04-02T13:21:38.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f6144d7b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f614006ce40 0x7f614015bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f6144d7b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61400d0c50 0x7f61400b7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f6144d7b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f614015e4a0 0x7f6140160910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f6144d7b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f614015e880 con 0x7f61400d0c50 2024-04-02T13:21:38.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f6144d7b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f61400cd460 con 0x7f614006ce40 2024-04-02T13:21:38.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f6144d7b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f61400af130 con 0x7f614015e4a0 2024-04-02T13:21:38.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f613dd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61400d0c50 0x7f61400b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f613dd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61400d0c50 0x7f61400b7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38106/0 (socket says 172.21.15.44:38106) 2024-04-02T13:21:38.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f613dd9b640 1 -- 172.21.15.44:0/1986452575 learned_addr learned my addr 172.21.15.44:0/1986452575 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:38.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f613ed9d640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f614015e4a0 0x7f6140160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f613e59c640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f614006ce40 0x7f614015bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f613dd9b640 1 -- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f614015e4a0 msgr2=0x7f6140160910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f613dd9b640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f614015e4a0 0x7f6140160910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f613dd9b640 1 -- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f614006ce40 msgr2=0x7f614015bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f613dd9b640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f614006ce40 0x7f614015bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.184+0000 7f613dd9b640 1 -- 172.21.15.44:0/1986452575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614015ce80 con 0x7f61400d0c50 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.185+0000 7f613dd9b640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61400d0c50 0x7f61400b7550 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f613402b950 tx=0x7f61340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.185+0000 7f612effd640 1 -- 172.21.15.44:0/1986452575 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6134040070 con 0x7f61400d0c50 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.185+0000 7f612effd640 1 -- 172.21.15.44:0/1986452575 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f613403a420 con 0x7f61400d0c50 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.185+0000 7f6144d7b640 1 -- 172.21.15.44:0/1986452575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f614015c300 con 0x7f61400d0c50 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.185+0000 7f612effd640 1 -- 172.21.15.44:0/1986452575 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6134035040 con 0x7f61400d0c50 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.185+0000 7f6144d7b640 1 -- 172.21.15.44:0/1986452575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f614015c8c0 con 0x7f61400d0c50 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.186+0000 7f6144d7b640 1 -- 172.21.15.44:0/1986452575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61400684f0 con 0x7f61400d0c50 2024-04-02T13:21:38.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.186+0000 7f612effd640 1 -- 172.21.15.44:0/1986452575 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6134038020 con 0x7f61400d0c50 2024-04-02T13:21:38.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.187+0000 7f612effd640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f612006c8f0 0x7f612006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.187+0000 7f612effd640 1 -- 172.21.15.44:0/1986452575 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f61340ada00 con 0x7f61400d0c50 2024-04-02T13:21:38.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.187+0000 7f613e59c640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f612006c8f0 0x7f612006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.187+0000 7f613e59c640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f612006c8f0 0x7f612006edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f613800a8d0 tx=0x7f6138008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:38.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.191+0000 7f612effd640 1 -- 172.21.15.44:0/1986452575 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f613407bf70 con 0x7f61400d0c50 2024-04-02T13:21:38.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.447+0000 7f6144d7b640 1 -- 172.21.15.44:0/1986452575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f614006d490 con 0x7f61400d0c50 2024-04-02T13:21:38.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.449+0000 7f612effd640 1 -- 172.21.15.44:0/1986452575 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f613407b870 con 0x7f61400d0c50 2024-04-02T13:21:38.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.451+0000 7f6144d7b640 1 -- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f612006c8f0 msgr2=0x7f612006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.451+0000 7f6144d7b640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f612006c8f0 0x7f612006edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f613800a8d0 tx=0x7f6138008040 comp rx=0 tx=0).stop 2024-04-02T13:21:38.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.451+0000 7f6144d7b640 1 -- 172.21.15.44:0/1986452575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61400d0c50 msgr2=0x7f61400b7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.451+0000 7f6144d7b640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61400d0c50 0x7f61400b7550 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f613402b950 tx=0x7f61340046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.451+0000 7f6144d7b640 1 -- 172.21.15.44:0/1986452575 shutdown_connections 2024-04-02T13:21:38.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.451+0000 7f6144d7b640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f612006c8f0 0x7f612006edb0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.451+0000 7f6144d7b640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f614015e4a0 0x7f6140160910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.451+0000 7f6144d7b640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61400d0c50 0x7f61400b7550 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.451+0000 7f6144d7b640 1 --2- 172.21.15.44:0/1986452575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f614006ce40 0x7f614015bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.451+0000 7f6144d7b640 1 -- 172.21.15.44:0/1986452575 >> 172.21.15.44:0/1986452575 conn(0x7f614009aa20 msgr2=0x7f614009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:38.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.451+0000 7f6144d7b640 1 -- 172.21.15.44:0/1986452575 shutdown_connections 2024-04-02T13:21:38.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.451+0000 7f6144d7b640 1 -- 172.21.15.44:0/1986452575 wait complete. 2024-04-02T13:21:38.464 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-02T13:21:38.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.564+0000 7fb53c60d640 1 Processor -- start 2024-04-02T13:21:38.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.564+0000 7fb53c60d640 1 -- start start 2024-04-02T13:21:38.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.564+0000 7fb53c60d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340ca920 0x7fb534161e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.564+0000 7fb53c60d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb5340cdcb0 0x7fb5341623d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.564+0000 7fb53c60d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5340cd510 con 0x7fb53415e210 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.564+0000 7fb53c60d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb5340716c0 con 0x7fb5340cdcb0 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.564+0000 7fb53c60d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb5340719a0 con 0x7fb5340ca920 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb53a3a9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb53415e210 0x7fb5340ccc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42334/0 (socket says 172.21.15.44:42334) 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb53a3a9640 1 -- 172.21.15.44:0/1758566607 learned_addr learned my addr 172.21.15.44:0/1758566607 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3626446480 0 0) 0x7fb5340cd510 con 0x7fb53415e210 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb53abaa640 1 --2- 172.21.15.44:0/1758566607 >> v2:172.21.15.69:3300/0 conn(0x7fb5340cdcb0 0x7fb5341623d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb539ba8640 1 --2- 172.21.15.44:0/1758566607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340ca920 0x7fb534161e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb520003520 con 0x7fb53415e210 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 731142249 0 0) 0x7fb520003520 con 0x7fb53415e210 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340ca920 msgr2=0x7fb534161e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 --2- 172.21.15.44:0/1758566607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340ca920 0x7fb534161e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 >> v2:172.21.15.69:3300/0 conn(0x7fb5340cdcb0 msgr2=0x7fb5341623d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:38.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 --2- 172.21.15.44:0/1758566607 >> v2:172.21.15.69:3300/0 conn(0x7fb5340cdcb0 0x7fb5341623d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb534071c80 con 0x7fb53415e210 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fb530001670 con 0x7fb53415e210 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 >> v1:172.21.15.44:6789/0 conn(0x7fb53415e210 legacy=0x7fb5340ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 --2- 172.21.15.44:0/1758566607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340ca920 0x7fb5200039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 --2- 172.21.15.44:0/1758566607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb520003f30 0x7fb520004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 --2- 172.21.15.44:0/1758566607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb520004d40 0x7fb5200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb5340716c0 con 0x7fb520004d40 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb5340719a0 con 0x7fb520003f30 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb52000b9f0 con 0x7fb5340ca920 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb53abaa640 1 --2- 172.21.15.44:0/1758566607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb520004d40 0x7fb5200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb53a3a9640 1 --2- 172.21.15.44:0/1758566607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340ca920 0x7fb5200039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.565+0000 7fb539ba8640 1 --2- 172.21.15.44:0/1758566607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb520003f30 0x7fb520004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.566+0000 7fb53abaa640 1 -- 172.21.15.44:0/1758566607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340ca920 msgr2=0x7fb5200039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:38.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.566+0000 7fb53abaa640 1 --2- 172.21.15.44:0/1758566607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340ca920 0x7fb5200039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.566+0000 7fb53abaa640 1 -- 172.21.15.44:0/1758566607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb520003f30 msgr2=0x7fb520004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.566+0000 7fb53abaa640 1 --2- 172.21.15.44:0/1758566607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb520003f30 0x7fb520004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.566+0000 7fb53abaa640 1 -- 172.21.15.44:0/1758566607 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb520003520 con 0x7fb520004d40 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.566+0000 7fb53abaa640 1 --2- 172.21.15.44:0/1758566607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb520004d40 0x7fb5200051c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009f90 tx=0x7fb520005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c0abd927a1e1872 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.567+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c039070 con 0x7fb520004d40 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.567+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb52c02cc20 con 0x7fb520004d40 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.567+0000 7fb5393a7640 1 -- 172.21.15.44:0/1758566607 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c009c10 con 0x7fb520004d40 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.567+0000 7fb53c60d640 1 -- 172.21.15.44:0/1758566607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb520004d40 msgr2=0x7fb5200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.567+0000 7fb53c60d640 1 --2- 172.21.15.44:0/1758566607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb520004d40 0x7fb5200051c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009f90 tx=0x7fb520005830 comp rx=0 tx=0).stop 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.567+0000 7fb53abaa640 1 -- 172.21.15.44:0/1758566607 reap_dead start 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.567+0000 7fb53c60d640 1 -- 172.21.15.44:0/1758566607 shutdown_connections 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.567+0000 7fb53c60d640 1 -- 172.21.15.44:0/1758566607 >> 172.21.15.44:0/1758566607 conn(0x7fb5340b1c30 msgr2=0x7fb5340b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.567+0000 7fb53c60d640 1 -- 172.21.15.44:0/1758566607 shutdown_connections 2024-04-02T13:21:38.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.567+0000 7fb53c60d640 1 -- 172.21.15.44:0/1758566607 wait complete. 2024-04-02T13:21:38.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.568+0000 7fb53c60d640 1 Processor -- start 2024-04-02T13:21:38.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.568+0000 7fb53c60d640 1 -- start start 2024-04-02T13:21:38.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53c60d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb520004d40 0x7fb53415b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53c60d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb53415fb10 0x7fb5340b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53c60d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340d0440 0x7fb5340d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53c60d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb534161e90 con 0x7fb520004d40 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53c60d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb534162010 con 0x7fb53415fb10 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53c60d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb534162190 con 0x7fb5340d0440 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53a3a9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb520004d40 0x7fb53415b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53a3a9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb520004d40 0x7fb53415b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38130/0 (socket says 172.21.15.44:38130) 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53a3a9640 1 -- 172.21.15.44:0/2946353650 learned_addr learned my addr 172.21.15.44:0/2946353650 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb539ba8640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb53415fb10 0x7fb5340b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53abaa640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340d0440 0x7fb5340d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53a3a9640 1 -- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340d0440 msgr2=0x7fb5340d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53a3a9640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340d0440 0x7fb5340d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53a3a9640 1 -- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb53415fb10 msgr2=0x7fb5340b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53a3a9640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb53415fb10 0x7fb5340b2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53a3a9640 1 -- 172.21.15.44:0/2946353650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5340b3530 con 0x7fb520004d40 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.569+0000 7fb53abaa640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340d0440 0x7fb5340d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:38.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.570+0000 7fb53a3a9640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb520004d40 0x7fb53415b130 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb530007340 tx=0x7fb530007740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:38.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.570+0000 7fb52affd640 1 -- 172.21.15.44:0/2946353650 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53000f8e0 con 0x7fb520004d40 2024-04-02T13:21:38.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.570+0000 7fb52affd640 1 -- 172.21.15.44:0/2946353650 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb530018070 con 0x7fb520004d40 2024-04-02T13:21:38.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.570+0000 7fb52affd640 1 -- 172.21.15.44:0/2946353650 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53000fce0 con 0x7fb520004d40 2024-04-02T13:21:38.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.570+0000 7fb53c60d640 1 -- 172.21.15.44:0/2946353650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5340b3810 con 0x7fb520004d40 2024-04-02T13:21:38.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.570+0000 7fb53c60d640 1 -- 172.21.15.44:0/2946353650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5340b3dd0 con 0x7fb520004d40 2024-04-02T13:21:38.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.571+0000 7fb53c60d640 1 -- 172.21.15.44:0/2946353650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5340684f0 con 0x7fb520004d40 2024-04-02T13:21:38.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.571+0000 7fb52affd640 1 -- 172.21.15.44:0/2946353650 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb53001d460 con 0x7fb520004d40 2024-04-02T13:21:38.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.572+0000 7fb52affd640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb51806c8f0 0x7fb51806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.572+0000 7fb52affd640 1 -- 172.21.15.44:0/2946353650 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fb53008b920 con 0x7fb520004d40 2024-04-02T13:21:38.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.572+0000 7fb539ba8640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb51806c8f0 0x7fb51806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.572+0000 7fb539ba8640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb51806c8f0 0x7fb51806edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb5340c83a0 tx=0x7fb524009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:38.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.576+0000 7fb52affd640 1 -- 172.21.15.44:0/2946353650 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb530059fa0 con 0x7fb520004d40 2024-04-02T13:21:38.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.834+0000 7fb53c60d640 1 -- 172.21.15.44:0/2946353650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fb53406eae0 con 0x7fb520004d40 2024-04-02T13:21:38.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.836+0000 7fb52affd640 1 -- 172.21.15.44:0/2946353650 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7fb5300598a0 con 0x7fb520004d40 2024-04-02T13:21:38.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.838+0000 7fb53c60d640 1 -- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb51806c8f0 msgr2=0x7fb51806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.838+0000 7fb53c60d640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb51806c8f0 0x7fb51806edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb5340c83a0 tx=0x7fb524009190 comp rx=0 tx=0).stop 2024-04-02T13:21:38.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.838+0000 7fb53c60d640 1 -- 172.21.15.44:0/2946353650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb520004d40 msgr2=0x7fb53415b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.838+0000 7fb53c60d640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb520004d40 0x7fb53415b130 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb530007340 tx=0x7fb530007740 comp rx=0 tx=0).stop 2024-04-02T13:21:38.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.838+0000 7fb53c60d640 1 -- 172.21.15.44:0/2946353650 shutdown_connections 2024-04-02T13:21:38.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.838+0000 7fb53c60d640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb51806c8f0 0x7fb51806edb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.838+0000 7fb53c60d640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5340d0440 0x7fb5340d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.838+0000 7fb53c60d640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb53415fb10 0x7fb5340b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.838+0000 7fb53c60d640 1 --2- 172.21.15.44:0/2946353650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb520004d40 0x7fb53415b130 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.838+0000 7fb53c60d640 1 -- 172.21.15.44:0/2946353650 >> 172.21.15.44:0/2946353650 conn(0x7fb5340b1c30 msgr2=0x7fb53415d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:38.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.838+0000 7fb53c60d640 1 -- 172.21.15.44:0/2946353650 shutdown_connections 2024-04-02T13:21:38.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.838+0000 7fb53c60d640 1 -- 172.21.15.44:0/2946353650 wait complete. 2024-04-02T13:21:38.851 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-02T13:21:38.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.951+0000 7f3e86b59640 1 Processor -- start 2024-04-02T13:21:38.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.952+0000 7f3e86b59640 1 -- start start 2024-04-02T13:21:38.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.952+0000 7f3e86b59640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3e800d2040 0x7f3e800d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.952+0000 7f3e86b59640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e800d0000 con 0x7f3e800d0a30 2024-04-02T13:21:38.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.952+0000 7f3e86b59640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e8006c1d0 con 0x7f3e8006cf90 2024-04-02T13:21:38.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.952+0000 7f3e86b59640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3e8006cb00 con 0x7f3e800d2040 2024-04-02T13:21:38.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.952+0000 7f3e848f5640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f3e800d0a30 0x7f3e800d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42344/0 (socket says 172.21.15.44:42344) 2024-04-02T13:21:38.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.952+0000 7f3e848f5640 1 -- 172.21.15.44:0/3560869863 learned_addr learned my addr 172.21.15.44:0/3560869863 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:38.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e850f6640 1 --2- 172.21.15.44:0/3560869863 >> v2:172.21.15.69:3300/0 conn(0x7f3e800d2040 0x7f3e800d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 541442317 0 0) 0x7f3e800d0000 con 0x7f3e800d0a30 2024-04-02T13:21:38.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e70003520 con 0x7f3e800d0a30 2024-04-02T13:21:38.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2183460418 0 0) 0x7f3e70003520 con 0x7f3e800d0a30 2024-04-02T13:21:38.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 >> v2:172.21.15.69:3300/0 conn(0x7f3e800d2040 msgr2=0x7f3e800d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:38.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 --2- 172.21.15.44:0/3560869863 >> v2:172.21.15.69:3300/0 conn(0x7f3e800d2040 0x7f3e800d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 >> v1:172.21.15.69:6789/0 conn(0x7f3e8006cf90 legacy=0x7f3e800cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8006c4b0 con 0x7f3e800d0a30 2024-04-02T13:21:38.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f3e74001670 con 0x7f3e800d0a30 2024-04-02T13:21:38.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 >> v1:172.21.15.44:6789/0 conn(0x7f3e800d0a30 legacy=0x7f3e800d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 --2- 172.21.15.44:0/3560869863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e700039b0 0x7f3e70003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 --2- 172.21.15.44:0/3560869863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e700043b0 0x7f3e70004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 --2- 172.21.15.44:0/3560869863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e70005120 0x7f3e7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8006cb00 con 0x7f3e700043b0 2024-04-02T13:21:38.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7000c060 con 0x7f3e70005120 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7000c260 con 0x7f3e700039b0 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7ffff640 1 --2- 172.21.15.44:0/3560869863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e700043b0 0x7f3e70004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e848f5640 1 --2- 172.21.15.44:0/3560869863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e700039b0 0x7f3e70003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e850f6640 1 --2- 172.21.15.44:0/3560869863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e70005120 0x7f3e7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7ffff640 1 -- 172.21.15.44:0/3560869863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e700039b0 msgr2=0x7f3e70003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7ffff640 1 --2- 172.21.15.44:0/3560869863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e700039b0 0x7f3e70003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7ffff640 1 -- 172.21.15.44:0/3560869863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e70005120 msgr2=0x7f3e7000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.953+0000 7f3e7ffff640 1 --2- 172.21.15.44:0/3560869863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e70005120 0x7f3e7000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.954+0000 7f3e7ffff640 1 -- 172.21.15.44:0/3560869863 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e70003520 con 0x7f3e700043b0 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.954+0000 7f3e850f6640 1 -- 172.21.15.44:0/3560869863 reap_dead start 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.954+0000 7f3e7ffff640 1 --2- 172.21.15.44:0/3560869863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e700043b0 0x7f3e70004880 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c001830 tx=0x7f3e70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4587a6de83d7a92 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.954+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c001430 con 0x7f3e700043b0 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.954+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e6c032620 con 0x7f3e700043b0 2024-04-02T13:21:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.954+0000 7f3e7f7fe640 1 -- 172.21.15.44:0/3560869863 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c032a10 con 0x7f3e700043b0 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.955+0000 7f3e86b59640 1 -- 172.21.15.44:0/3560869863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e700043b0 msgr2=0x7f3e70004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.955+0000 7f3e86b59640 1 --2- 172.21.15.44:0/3560869863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e700043b0 0x7f3e70004880 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c001830 tx=0x7f3e70005830 comp rx=0 tx=0).stop 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.955+0000 7f3e86b59640 1 -- 172.21.15.44:0/3560869863 shutdown_connections 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.955+0000 7f3e86b59640 1 --2- 172.21.15.44:0/3560869863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e700043b0 0x7f3e70004880 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.955+0000 7f3e86b59640 1 -- 172.21.15.44:0/3560869863 >> 172.21.15.44:0/3560869863 conn(0x7f3e8009d470 msgr2=0x7f3e800d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.955+0000 7f3e86b59640 1 -- 172.21.15.44:0/3560869863 shutdown_connections 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.955+0000 7f3e86b59640 1 -- 172.21.15.44:0/3560869863 wait complete. 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.956+0000 7f3e86b59640 1 Processor -- start 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.956+0000 7f3e86b59640 1 -- start start 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e86b59640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e700043b0 0x7f3e8015fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e86b59640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e800d2fe0 0x7f3e800d4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e86b59640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e8014fc00 0x7f3e8015c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e86b59640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8015e6f0 con 0x7f3e800d2fe0 2024-04-02T13:21:38.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e86b59640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8006d3f0 con 0x7f3e8014fc00 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e86b59640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3e8015e870 con 0x7f3e700043b0 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e7ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e800d2fe0 0x7f3e800d4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e7ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e800d2fe0 0x7f3e800d4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38144/0 (socket says 172.21.15.44:38144) 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e7ffff640 1 -- 172.21.15.44:0/3897769341 learned_addr learned my addr 172.21.15.44:0/3897769341 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e848f5640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e700043b0 0x7f3e8015fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e850f6640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e8014fc00 0x7f3e8015c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e7ffff640 1 -- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e700043b0 msgr2=0x7f3e8015fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e7ffff640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e700043b0 0x7f3e8015fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e7ffff640 1 -- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e8014fc00 msgr2=0x7f3e8015c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e7ffff640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e8014fc00 0x7f3e8015c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.957+0000 7f3e7ffff640 1 -- 172.21.15.44:0/3897769341 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e800d4ac0 con 0x7f3e800d2fe0 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.958+0000 7f3e7ffff640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e800d2fe0 0x7f3e800d4580 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c02bd90 tx=0x7f3e6c001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.958+0000 7f3e7d7fa640 1 -- 172.21.15.44:0/3897769341 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c007220 con 0x7f3e800d2fe0 2024-04-02T13:21:38.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.958+0000 7f3e7d7fa640 1 -- 172.21.15.44:0/3897769341 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e6c0073c0 con 0x7f3e800d2fe0 2024-04-02T13:21:38.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.958+0000 7f3e86b59640 1 -- 172.21.15.44:0/3897769341 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e800d34e0 con 0x7f3e800d2fe0 2024-04-02T13:21:38.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.958+0000 7f3e7d7fa640 1 -- 172.21.15.44:0/3897769341 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c0067d0 con 0x7f3e800d2fe0 2024-04-02T13:21:38.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.958+0000 7f3e86b59640 1 -- 172.21.15.44:0/3897769341 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e800d3aa0 con 0x7f3e800d2fe0 2024-04-02T13:21:38.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.959+0000 7f3e86b59640 1 -- 172.21.15.44:0/3897769341 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e800684f0 con 0x7f3e800d2fe0 2024-04-02T13:21:38.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.959+0000 7f3e7d7fa640 1 -- 172.21.15.44:0/3897769341 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3e6c044030 con 0x7f3e800d2fe0 2024-04-02T13:21:38.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.960+0000 7f3e7d7fa640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e6406c920 0x7f3e6406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:38.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.960+0000 7f3e7d7fa640 1 -- 172.21.15.44:0/3897769341 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f3e6c0ad4e0 con 0x7f3e800d2fe0 2024-04-02T13:21:38.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.960+0000 7f3e848f5640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e6406c920 0x7f3e6406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:38.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.960+0000 7f3e848f5640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e6406c920 0x7f3e6406ede0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3e74007cb0 tx=0x7f3e74000f80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:38.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:38.964+0000 7f3e7d7fa640 1 -- 172.21.15.44:0/3897769341 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3e6c07b480 con 0x7f3e800d2fe0 2024-04-02T13:21:39.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.220+0000 7f3e86b59640 1 -- 172.21.15.44:0/3897769341 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f3e800d17c0 con 0x7f3e800d2fe0 2024-04-02T13:21:39.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.222+0000 7f3e7d7fa640 1 -- 172.21.15.44:0/3897769341 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f3e6c07f780 con 0x7f3e800d2fe0 2024-04-02T13:21:39.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.224+0000 7f3e86b59640 1 -- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e6406c920 msgr2=0x7f3e6406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.224+0000 7f3e86b59640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e6406c920 0x7f3e6406ede0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3e74007cb0 tx=0x7f3e74000f80 comp rx=0 tx=0).stop 2024-04-02T13:21:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.224+0000 7f3e86b59640 1 -- 172.21.15.44:0/3897769341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e800d2fe0 msgr2=0x7f3e800d4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.224+0000 7f3e86b59640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e800d2fe0 0x7f3e800d4580 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c02bd90 tx=0x7f3e6c001230 comp rx=0 tx=0).stop 2024-04-02T13:21:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.224+0000 7f3e86b59640 1 -- 172.21.15.44:0/3897769341 shutdown_connections 2024-04-02T13:21:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.224+0000 7f3e86b59640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e6406c920 0x7f3e6406ede0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.224+0000 7f3e86b59640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e8014fc00 0x7f3e8015c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.224+0000 7f3e86b59640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e800d2fe0 0x7f3e800d4580 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.224+0000 7f3e86b59640 1 --2- 172.21.15.44:0/3897769341 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e700043b0 0x7f3e8015fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.224+0000 7f3e86b59640 1 -- 172.21.15.44:0/3897769341 >> 172.21.15.44:0/3897769341 conn(0x7f3e8009d470 msgr2=0x7f3e8015a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.224+0000 7f3e86b59640 1 -- 172.21.15.44:0/3897769341 shutdown_connections 2024-04-02T13:21:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.224+0000 7f3e86b59640 1 -- 172.21.15.44:0/3897769341 wait complete. 2024-04-02T13:21:39.237 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-02T13:21:39.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.337+0000 7f3e92cd9640 1 Processor -- start 2024-04-02T13:21:39.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.337+0000 7f3e92cd9640 1 -- start start 2024-04-02T13:21:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.338+0000 7f3e92cd9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3e8c15e0f0 0x7f3e8c0d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.338+0000 7f3e92cd9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e8c06dfe0 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.338+0000 7f3e92cd9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e8c06e2c0 con 0x7f3e8c0d0630 2024-04-02T13:21:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.338+0000 7f3e92cd9640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f3e8c06e5a0 con 0x7f3e8c15e0f0 2024-04-02T13:21:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.338+0000 7f3e90a75640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3e8c15e0f0 0x7f3e8c0d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.338+0000 7f3e90a75640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3e8c15e0f0 0x7f3e8c0d00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38150/0 (socket says 172.21.15.44:38150) 2024-04-02T13:21:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.338+0000 7f3e90a75640 1 -- 172.21.15.44:0/1606437112 learned_addr learned my addr 172.21.15.44:0/1606437112 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.338+0000 7f3e91276640 1 --1- 172.21.15.44:0/1606437112 >> v1:172.21.15.44:6789/0 conn(0x7f3e8c06bbe0 0x7f3e8c0add60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42356/0 (socket says 172.21.15.44:42356) 2024-04-02T13:21:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.338+0000 7f3e90a75640 1 -- 172.21.15.44:0/1606437112 >> v1:172.21.15.69:6789/0 conn(0x7f3e8c0d0630 legacy=0x7f3e8c06b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.338+0000 7f3e90a75640 1 -- 172.21.15.44:0/1606437112 >> v1:172.21.15.44:6789/0 conn(0x7f3e8c06bbe0 legacy=0x7f3e8c0add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:39.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.338+0000 7f3e8b7fe640 1 -- 172.21.15.44:0/1606437112 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1874692462 0 0) 0x7f3e8c06dfe0 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.338+0000 7f3e90a75640 1 -- 172.21.15.44:0/1606437112 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8c06e880 con 0x7f3e8c15e0f0 2024-04-02T13:21:39.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.339+0000 7f3e90a75640 1 --2- 172.21.15.44:0/1606437112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e8c15e0f0 0x7f3e8c0d00f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009e50 tx=0x7f3e8402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a310f2924f0988a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:39.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.339+0000 7f3e8b7fe640 1 -- 172.21.15.44:0/1606437112 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8402cbb0 con 0x7f3e8c15e0f0 2024-04-02T13:21:39.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.339+0000 7f3e8b7fe640 1 -- 172.21.15.44:0/1606437112 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e8402cd50 con 0x7f3e8c15e0f0 2024-04-02T13:21:39.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.339+0000 7f3e8b7fe640 1 -- 172.21.15.44:0/1606437112 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e84034610 con 0x7f3e8c15e0f0 2024-04-02T13:21:39.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.340+0000 7f3e92cd9640 1 -- 172.21.15.44:0/1606437112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e8c15e0f0 msgr2=0x7f3e8c0d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:39.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.340+0000 7f3e92cd9640 1 --2- 172.21.15.44:0/1606437112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e8c15e0f0 0x7f3e8c0d00f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009e50 tx=0x7f3e8402c010 comp rx=0 tx=0).stop 2024-04-02T13:21:39.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.340+0000 7f3e92cd9640 1 -- 172.21.15.44:0/1606437112 shutdown_connections 2024-04-02T13:21:39.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.340+0000 7f3e92cd9640 1 --2- 172.21.15.44:0/1606437112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e8c15e0f0 0x7f3e8c0d00f0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.340+0000 7f3e92cd9640 1 -- 172.21.15.44:0/1606437112 >> 172.21.15.44:0/1606437112 conn(0x7f3e8c0b2980 msgr2=0x7f3e8c0b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:39.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.340+0000 7f3e92cd9640 1 -- 172.21.15.44:0/1606437112 shutdown_connections 2024-04-02T13:21:39.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.340+0000 7f3e92cd9640 1 -- 172.21.15.44:0/1606437112 wait complete. 2024-04-02T13:21:39.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.341+0000 7f3e92cd9640 1 Processor -- start 2024-04-02T13:21:39.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.341+0000 7f3e92cd9640 1 -- start start 2024-04-02T13:21:39.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e92cd9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e8c06bbe0 0x7f3e8c0c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e92cd9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e8c0d0630 0x7f3e8c15c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e92cd9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e8c15e0f0 0x7f3e8c0c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e92cd9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c15e4d0 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e92cd9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c06eda0 con 0x7f3e8c15e0f0 2024-04-02T13:21:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e92cd9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3e8c15e650 con 0x7f3e8c0d0630 2024-04-02T13:21:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e90a75640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e8c06bbe0 0x7f3e8c0c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e90a75640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e8c06bbe0 0x7f3e8c0c8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38158/0 (socket says 172.21.15.44:38158) 2024-04-02T13:21:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e90a75640 1 -- 172.21.15.44:0/43688772 learned_addr learned my addr 172.21.15.44:0/43688772 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e8bfff640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e8c0d0630 0x7f3e8c15c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e91276640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e8c15e0f0 0x7f3e8c0c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e90a75640 1 -- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e8c0d0630 msgr2=0x7f3e8c15c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e90a75640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e8c0d0630 0x7f3e8c15c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e90a75640 1 -- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e8c15e0f0 msgr2=0x7f3e8c0c7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e90a75640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e8c15e0f0 0x7f3e8c0c7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e90a75640 1 -- 172.21.15.44:0/43688772 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8c15c9e0 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e8bfff640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e8c0d0630 0x7f3e8c15c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e91276640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e8c15e0f0 0x7f3e8c0c7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e90a75640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e8c06bbe0 0x7f3e8c0c8f50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009e50 tx=0x7f3e84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.342+0000 7f3e897fa640 1 -- 172.21.15.44:0/43688772 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e84043020 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.343+0000 7f3e897fa640 1 -- 172.21.15.44:0/43688772 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e8403e450 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.343+0000 7f3e92cd9640 1 -- 172.21.15.44:0/43688772 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8c1f4600 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.343+0000 7f3e897fa640 1 -- 172.21.15.44:0/43688772 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8403e840 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.343+0000 7f3e92cd9640 1 -- 172.21.15.44:0/43688772 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e8c1f4ea0 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.344+0000 7f3e92cd9640 1 -- 172.21.15.44:0/43688772 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e8c1f49f0 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.344+0000 7f3e897fa640 1 -- 172.21.15.44:0/43688772 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3e84046020 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.344+0000 7f3e897fa640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e7806c8f0 0x7f3e7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.344+0000 7f3e897fa640 1 -- 172.21.15.44:0/43688772 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f3e840ae3b0 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.344+0000 7f3e8bfff640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e7806c8f0 0x7f3e7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:39.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.345+0000 7f3e8bfff640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e7806c8f0 0x7f3e7806edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0b2f40 tx=0x7f3e80009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:39.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.349+0000 7f3e897fa640 1 -- 172.21.15.44:0/43688772 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3e8404d030 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.606+0000 7f3e92cd9640 1 -- 172.21.15.44:0/43688772 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f3e8c1f47d0 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.608+0000 7f3e897fa640 1 -- 172.21.15.44:0/43688772 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f3e8407c910 con 0x7f3e8c06bbe0 2024-04-02T13:21:39.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.610+0000 7f3e92cd9640 1 -- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e7806c8f0 msgr2=0x7f3e7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:39.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.610+0000 7f3e92cd9640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e7806c8f0 0x7f3e7806edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0b2f40 tx=0x7f3e80009040 comp rx=0 tx=0).stop 2024-04-02T13:21:39.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.610+0000 7f3e92cd9640 1 -- 172.21.15.44:0/43688772 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e8c06bbe0 msgr2=0x7f3e8c0c8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:39.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.610+0000 7f3e92cd9640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e8c06bbe0 0x7f3e8c0c8f50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009e50 tx=0x7f3e84004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:39.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.610+0000 7f3e92cd9640 1 -- 172.21.15.44:0/43688772 shutdown_connections 2024-04-02T13:21:39.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.610+0000 7f3e92cd9640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e7806c8f0 0x7f3e7806edb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.610+0000 7f3e92cd9640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e8c15e0f0 0x7f3e8c0c7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.610+0000 7f3e92cd9640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e8c0d0630 0x7f3e8c15c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.610+0000 7f3e92cd9640 1 --2- 172.21.15.44:0/43688772 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e8c06bbe0 0x7f3e8c0c8f50 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.610+0000 7f3e92cd9640 1 -- 172.21.15.44:0/43688772 >> 172.21.15.44:0/43688772 conn(0x7f3e8c0b2980 msgr2=0x7f3e8c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:39.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.610+0000 7f3e92cd9640 1 -- 172.21.15.44:0/43688772 shutdown_connections 2024-04-02T13:21:39.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.610+0000 7f3e92cd9640 1 -- 172.21.15.44:0/43688772 wait complete. 2024-04-02T13:21:39.623 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-02T13:21:39.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.723+0000 7f0b32f1c640 1 Processor -- start 2024-04-02T13:21:39.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.723+0000 7f0b32f1c640 1 -- start start 2024-04-02T13:21:39.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.724+0000 7f0b32f1c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0b2c15e140 0x7f0b2c0d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:39.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.724+0000 7f0b32f1c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b2c0b70b0 con 0x7f0b2c0d1150 2024-04-02T13:21:39.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.724+0000 7f0b32f1c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b2c0b7390 con 0x7f0b2c15e520 2024-04-02T13:21:39.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.724+0000 7f0b32f1c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0b2c0b9f50 con 0x7f0b2c15e140 2024-04-02T13:21:39.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.724+0000 7f0b30cb8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0b2c15e140 0x7f0b2c0d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:39.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.724+0000 7f0b30cb8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0b2c15e140 0x7f0b2c0d2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38172/0 (socket says 172.21.15.44:38172) 2024-04-02T13:21:39.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.724+0000 7f0b314b9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0b2c0d1150 0x7f0b2c0b4cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42372/0 (socket says 172.21.15.44:42372) 2024-04-02T13:21:39.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.724+0000 7f0b30cb8640 1 -- 172.21.15.44:0/1054496587 learned_addr learned my addr 172.21.15.44:0/1054496587 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:39.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.724+0000 7f0b30cb8640 1 -- 172.21.15.44:0/1054496587 >> v1:172.21.15.69:6789/0 conn(0x7f0b2c15e520 legacy=0x7f0b2c0d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:39.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.724+0000 7f0b30cb8640 1 -- 172.21.15.44:0/1054496587 >> v1:172.21.15.44:6789/0 conn(0x7f0b2c0d1150 legacy=0x7f0b2c0b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:39.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.724+0000 7f0b30cb8640 1 -- 172.21.15.44:0/1054496587 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b2c0b7670 con 0x7f0b2c15e140 2024-04-02T13:21:39.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.724+0000 7f0b2b7fe640 1 -- 172.21.15.44:0/1054496587 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1648432297 0 0) 0x7f0b2c0b70b0 con 0x7f0b2c0d1150 2024-04-02T13:21:39.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.725+0000 7f0b30cb8640 1 --2- 172.21.15.44:0/1054496587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b2c15e140 0x7f0b2c0d2630 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0b24009e50 tx=0x7f0b2402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3db45e9d22cd06 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:39.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.725+0000 7f0b2b7fe640 1 -- 172.21.15.44:0/1054496587 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2402ca80 con 0x7f0b2c15e140 2024-04-02T13:21:39.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.725+0000 7f0b2b7fe640 1 -- 172.21.15.44:0/1054496587 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b2402cc20 con 0x7f0b2c15e140 2024-04-02T13:21:39.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.725+0000 7f0b2b7fe640 1 -- 172.21.15.44:0/1054496587 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b240343f0 con 0x7f0b2c15e140 2024-04-02T13:21:39.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.726+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1054496587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b2c15e140 msgr2=0x7f0b2c0d2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:39.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.726+0000 7f0b32f1c640 1 --2- 172.21.15.44:0/1054496587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b2c15e140 0x7f0b2c0d2630 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0b24009e50 tx=0x7f0b2402bee0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.726+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1054496587 shutdown_connections 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.726+0000 7f0b32f1c640 1 --2- 172.21.15.44:0/1054496587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b2c15e140 0x7f0b2c0d2630 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.726+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1054496587 >> 172.21.15.44:0/1054496587 conn(0x7f0b2c09d390 msgr2=0x7f0b2c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.726+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1054496587 shutdown_connections 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.726+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1054496587 wait complete. 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.727+0000 7f0b32f1c640 1 Processor -- start 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.727+0000 7f0b32f1c640 1 -- start start 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b32f1c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b2c0d1150 0x7f0b2c15a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b32f1c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b2c15e140 0x7f0b2c1588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b32f1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b2c15e520 0x7f0b2c0c8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b32f1c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0b2c0d2b70 con 0x7f0b2c0d1150 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b32f1c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0b2c0d4ee0 con 0x7f0b2c15e140 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b32f1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0b2c0d2cf0 con 0x7f0b2c15e520 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b30cb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b2c0d1150 0x7f0b2c15a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b30cb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b2c0d1150 0x7f0b2c15a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38180/0 (socket says 172.21.15.44:38180) 2024-04-02T13:21:39.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b30cb8640 1 -- 172.21.15.44:0/1755878548 learned_addr learned my addr 172.21.15.44:0/1755878548 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b314b9640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b2c15e520 0x7f0b2c0c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b2bfff640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b2c15e140 0x7f0b2c1588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b30cb8640 1 -- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b2c15e520 msgr2=0x7f0b2c0c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b30cb8640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b2c15e520 0x7f0b2c0c8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b30cb8640 1 -- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b2c15e140 msgr2=0x7f0b2c1588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b30cb8640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b2c15e140 0x7f0b2c1588f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b30cb8640 1 -- 172.21.15.44:0/1755878548 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b2c15ab60 con 0x7f0b2c0d1150 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b314b9640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b2c15e520 0x7f0b2c0c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.728+0000 7f0b2bfff640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b2c15e140 0x7f0b2c1588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.729+0000 7f0b30cb8640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b2c0d1150 0x7f0b2c15a620 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0b24009e50 tx=0x7f0b24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.729+0000 7f0b297fa640 1 -- 172.21.15.44:0/1755878548 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b24043020 con 0x7f0b2c0d1150 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.729+0000 7f0b297fa640 1 -- 172.21.15.44:0/1755878548 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b2403e450 con 0x7f0b2c0d1150 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.729+0000 7f0b297fa640 1 -- 172.21.15.44:0/1755878548 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2403e840 con 0x7f0b2c0d1150 2024-04-02T13:21:39.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.729+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1755878548 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b2c0cb2c0 con 0x7f0b2c0d1150 2024-04-02T13:21:39.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.729+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1755878548 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b2c0cb880 con 0x7f0b2c0d1150 2024-04-02T13:21:39.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.730+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1755878548 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b2c0684f0 con 0x7f0b2c0d1150 2024-04-02T13:21:39.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.730+0000 7f0b297fa640 1 -- 172.21.15.44:0/1755878548 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0b24046020 con 0x7f0b2c0d1150 2024-04-02T13:21:39.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.731+0000 7f0b297fa640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b0c06c8f0 0x7f0b0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:39.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.731+0000 7f0b297fa640 1 -- 172.21.15.44:0/1755878548 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f0b240ae2c0 con 0x7f0b2c0d1150 2024-04-02T13:21:39.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.731+0000 7f0b2bfff640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b0c06c8f0 0x7f0b0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:39.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.731+0000 7f0b2bfff640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b0c06c8f0 0x7f0b0c06edb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0ce190 tx=0x7f0b14008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:39.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.735+0000 7f0b297fa640 1 -- 172.21.15.44:0/1755878548 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0b2404d030 con 0x7f0b2c0d1150 2024-04-02T13:21:39.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.994+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1755878548 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f0b2c067d70 con 0x7f0b2c0d1150 2024-04-02T13:21:39.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.996+0000 7f0b297fa640 1 -- 172.21.15.44:0/1755878548 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7f0b2407c940 con 0x7f0b2c0d1150 2024-04-02T13:21:40.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.999+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b0c06c8f0 msgr2=0x7f0b0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.999+0000 7f0b32f1c640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b0c06c8f0 0x7f0b0c06edb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0ce190 tx=0x7f0b14008040 comp rx=0 tx=0).stop 2024-04-02T13:21:40.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.999+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1755878548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b2c0d1150 msgr2=0x7f0b2c15a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.999+0000 7f0b32f1c640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b2c0d1150 0x7f0b2c15a620 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0b24009e50 tx=0x7f0b24004a40 comp rx=0 tx=0).stop 2024-04-02T13:21:40.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.999+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1755878548 shutdown_connections 2024-04-02T13:21:40.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.999+0000 7f0b32f1c640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b0c06c8f0 0x7f0b0c06edb0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.999+0000 7f0b32f1c640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b2c15e520 0x7f0b2c0c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.999+0000 7f0b32f1c640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b2c15e140 0x7f0b2c1588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.999+0000 7f0b32f1c640 1 --2- 172.21.15.44:0/1755878548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b2c0d1150 0x7f0b2c15a620 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.999+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1755878548 >> 172.21.15.44:0/1755878548 conn(0x7f0b2c09d390 msgr2=0x7f0b2c0ceb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:40.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.999+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1755878548 shutdown_connections 2024-04-02T13:21:40.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:39.999+0000 7f0b32f1c640 1 -- 172.21.15.44:0/1755878548 wait complete. 2024-04-02T13:21:40.012 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-02T13:21:40.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.113+0000 7f70ecd9d640 1 Processor -- start 2024-04-02T13:21:40.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.113+0000 7f70ecd9d640 1 -- start start 2024-04-02T13:21:40.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.114+0000 7f70ecd9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f70e80cece0 0x7f70e80cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.114+0000 7f70ecd9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70e806ece0 0x7f70e80acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.114+0000 7f70ecd9d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70e80cf600 con 0x7f70e80d09f0 2024-04-02T13:21:40.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.114+0000 7f70ecd9d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f70e806d9e0 con 0x7f70e80cece0 2024-04-02T13:21:40.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.114+0000 7f70ecd9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f70e806dc40 con 0x7f70e806ece0 2024-04-02T13:21:40.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.114+0000 7f70e659c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f70e80cece0 0x7f70e80cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.114+0000 7f70e659c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f70e80cece0 0x7f70e80cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38186/0 (socket says 172.21.15.44:38186) 2024-04-02T13:21:40.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.114+0000 7f70e659c640 1 -- 172.21.15.44:0/2774028570 learned_addr learned my addr 172.21.15.44:0/2774028570 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:40.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.114+0000 7f70e659c640 1 -- 172.21.15.44:0/2774028570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70e806ece0 msgr2=0x7f70e80acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:40.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.114+0000 7f70e659c640 1 --2- 172.21.15.44:0/2774028570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70e806ece0 0x7f70e80acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.114+0000 7f70e659c640 1 -- 172.21.15.44:0/2774028570 >> v1:172.21.15.69:6789/0 conn(0x7f70e80d09f0 legacy=0x7f70e806d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:40.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.114+0000 7f70e659c640 1 -- 172.21.15.44:0/2774028570 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e806df20 con 0x7f70e80cece0 2024-04-02T13:21:40.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.115+0000 7f70e659c640 1 --2- 172.21.15.44:0/2774028570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e80cece0 0x7f70e80cf0c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f70d800bb10 tx=0x7f70d802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12d0687215eb122 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:40.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.115+0000 7f70e559a640 1 -- 172.21.15.44:0/2774028570 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d803b070 con 0x7f70e80cece0 2024-04-02T13:21:40.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.115+0000 7f70e559a640 1 -- 172.21.15.44:0/2774028570 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d8036440 con 0x7f70e80cece0 2024-04-02T13:21:40.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.115+0000 7f70e559a640 1 -- 172.21.15.44:0/2774028570 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d8036810 con 0x7f70e80cece0 2024-04-02T13:21:40.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.115+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2774028570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e80cece0 msgr2=0x7f70e80cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.115+0000 7f70ecd9d640 1 --2- 172.21.15.44:0/2774028570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e80cece0 0x7f70e80cf0c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f70d800bb10 tx=0x7f70d802e0c0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.116+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2774028570 shutdown_connections 2024-04-02T13:21:40.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.116+0000 7f70ecd9d640 1 --2- 172.21.15.44:0/2774028570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70e806ece0 0x7f70e80acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.116+0000 7f70ecd9d640 1 --2- 172.21.15.44:0/2774028570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e80cece0 0x7f70e80cf0c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.116+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2774028570 >> 172.21.15.44:0/2774028570 conn(0x7f70e80cfcd0 msgr2=0x7f70e80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:40.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.116+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2774028570 shutdown_connections 2024-04-02T13:21:40.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.116+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2774028570 wait complete. 2024-04-02T13:21:40.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.117+0000 7f70ecd9d640 1 Processor -- start 2024-04-02T13:21:40.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.117+0000 7f70ecd9d640 1 -- start start 2024-04-02T13:21:40.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70ecd9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e806ece0 0x7f70e81533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70ecd9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e80cece0 0x7f70e815bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70ecd9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70e80d09f0 0x7f70e81e5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70ecd9d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f70e806e7c0 con 0x7f70e806ece0 2024-04-02T13:21:40.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70ecd9d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f70e806e4e0 con 0x7f70e80cece0 2024-04-02T13:21:40.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70ecd9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f70e815b220 con 0x7f70e80d09f0 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70e659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e806ece0 0x7f70e81533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70e659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e806ece0 0x7f70e81533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38192/0 (socket says 172.21.15.44:38192) 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70e659c640 1 -- 172.21.15.44:0/2292036863 learned_addr learned my addr 172.21.15.44:0/2292036863 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70e659c640 1 -- 172.21.15.44:0/2292036863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70e80d09f0 msgr2=0x7f70e81e5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70e659c640 1 --2- 172.21.15.44:0/2292036863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70e80d09f0 0x7f70e81e5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70e659c640 1 -- 172.21.15.44:0/2292036863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e80cece0 msgr2=0x7f70e815bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70e659c640 1 --2- 172.21.15.44:0/2292036863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e80cece0 0x7f70e815bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70e659c640 1 -- 172.21.15.44:0/2292036863 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e81e6270 con 0x7f70e806ece0 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70e659c640 1 --2- 172.21.15.44:0/2292036863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e806ece0 0x7f70e81533d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f70d800b2c0 tx=0x7f70d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70d6ffd640 1 -- 172.21.15.44:0/2292036863 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d8045070 con 0x7f70e806ece0 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70d6ffd640 1 -- 172.21.15.44:0/2292036863 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d8040890 con 0x7f70e806ece0 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2292036863 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70e81e6550 con 0x7f70e806ece0 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70d6ffd640 1 -- 172.21.15.44:0/2292036863 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d803b040 con 0x7f70e806ece0 2024-04-02T13:21:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.118+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2292036863 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70e809ad00 con 0x7f70e806ece0 2024-04-02T13:21:40.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.119+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2292036863 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70e8074f20 con 0x7f70e806ece0 2024-04-02T13:21:40.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.120+0000 7f70d6ffd640 1 -- 172.21.15.44:0/2292036863 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f70d8048070 con 0x7f70e806ece0 2024-04-02T13:21:40.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.120+0000 7f70d6ffd640 1 --2- 172.21.15.44:0/2292036863 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f70c806c8f0 0x7f70c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.120+0000 7f70d6ffd640 1 -- 172.21.15.44:0/2292036863 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f70d80b0740 con 0x7f70e806ece0 2024-04-02T13:21:40.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.120+0000 7f70e5d9b640 1 --2- 172.21.15.44:0/2292036863 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f70c806c8f0 0x7f70c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.121+0000 7f70e5d9b640 1 --2- 172.21.15.44:0/2292036863 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f70c806c8f0 0x7f70c806edb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f70e81e4880 tx=0x7f70dc006ba0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:40.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.125+0000 7f70d6ffd640 1 -- 172.21.15.44:0/2292036863 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f70d807edc0 con 0x7f70e806ece0 2024-04-02T13:21:40.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.386+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2292036863 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f70e80678a0 con 0x7f70e806ece0 2024-04-02T13:21:40.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.388+0000 7f70d6ffd640 1 -- 172.21.15.44:0/2292036863 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f70d807e6c0 con 0x7f70e806ece0 2024-04-02T13:21:40.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.390+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2292036863 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f70c806c8f0 msgr2=0x7f70c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.390+0000 7f70ecd9d640 1 --2- 172.21.15.44:0/2292036863 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f70c806c8f0 0x7f70c806edb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f70e81e4880 tx=0x7f70dc006ba0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.390+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2292036863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e806ece0 msgr2=0x7f70e81533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.390+0000 7f70ecd9d640 1 --2- 172.21.15.44:0/2292036863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e806ece0 0x7f70e81533d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f70d800b2c0 tx=0x7f70d80046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.390+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2292036863 shutdown_connections 2024-04-02T13:21:40.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.390+0000 7f70ecd9d640 1 --2- 172.21.15.44:0/2292036863 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f70c806c8f0 0x7f70c806edb0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.390+0000 7f70ecd9d640 1 --2- 172.21.15.44:0/2292036863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70e80d09f0 0x7f70e81e5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.390+0000 7f70ecd9d640 1 --2- 172.21.15.44:0/2292036863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e80cece0 0x7f70e815bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.390+0000 7f70ecd9d640 1 --2- 172.21.15.44:0/2292036863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e806ece0 0x7f70e81533d0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.390+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2292036863 >> 172.21.15.44:0/2292036863 conn(0x7f70e80cfcd0 msgr2=0x7f70e8071550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:40.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.390+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2292036863 shutdown_connections 2024-04-02T13:21:40.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.390+0000 7f70ecd9d640 1 -- 172.21.15.44:0/2292036863 wait complete. 2024-04-02T13:21:40.403 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:40.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.502+0000 7f3b6abe9640 1 Processor -- start 2024-04-02T13:21:40.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.502+0000 7f3b6abe9640 1 -- start start 2024-04-02T13:21:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.503+0000 7f3b6abe9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b640d0a30 0x7f3b6406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.503+0000 7f3b6abe9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b640d0000 con 0x7f3b640d0e10 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.503+0000 7f3b6abe9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b6406c1d0 con 0x7f3b640d2040 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.503+0000 7f3b6abe9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3b6406cb00 con 0x7f3b640d0a30 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.503+0000 7f3b63fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f3b640d0e10 0x7f3b640cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42374/0 (socket says 172.21.15.44:42374) 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.503+0000 7f3b63fff640 1 -- 172.21.15.44:0/2236796804 learned_addr learned my addr 172.21.15.44:0/2236796804 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b68985640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b640d0a30 0x7f3b6406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4080683153 0 0) 0x7f3b640d0000 con 0x7f3b640d0e10 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b54003520 con 0x7f3b640d0e10 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3136766381 0 0) 0x7f3b54003520 con 0x7f3b640d0e10 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b640d0a30 msgr2=0x7f3b6406cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b640d0a30 0x7f3b6406cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 >> v1:172.21.15.69:6789/0 conn(0x7f3b640d2040 legacy=0x7f3b640d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b6406c4b0 con 0x7f3b640d0e10 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f3b50001670 con 0x7f3b640d0e10 2024-04-02T13:21:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 >> v1:172.21.15.44:6789/0 conn(0x7f3b640d0e10 legacy=0x7f3b640cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b540039e0 0x7f3b54003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b54004800 0x7f3b54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b640d0a30 0x7f3b54005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6406cb00 con 0x7f3b540039e0 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5400baf0 con 0x7f3b54004800 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3b5400bcf0 con 0x7f3b640d0a30 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b68985640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b540039e0 0x7f3b54003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b63fff640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b54004800 0x7f3b54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b69186640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b640d0a30 0x7f3b54005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b68985640 1 -- 172.21.15.44:0/2236796804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b640d0a30 msgr2=0x7f3b54005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b68985640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b640d0a30 0x7f3b54005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b68985640 1 -- 172.21.15.44:0/2236796804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b54004800 msgr2=0x7f3b54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b68985640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b54004800 0x7f3b54004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b68985640 1 -- 172.21.15.44:0/2236796804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b54003520 con 0x7f3b540039e0 2024-04-02T13:21:40.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b69186640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b640d0a30 0x7f3b54005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:40.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.504+0000 7f3b63fff640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b54004800 0x7f3b54004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:40.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.505+0000 7f3b68985640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b540039e0 0x7f3b54003e00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009ac0 tx=0x7f3b54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8cf92f3043610f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:40.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.505+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b58039070 con 0x7f3b540039e0 2024-04-02T13:21:40.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.505+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b5802c6b0 con 0x7f3b540039e0 2024-04-02T13:21:40.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.505+0000 7f3b637fe640 1 -- 172.21.15.44:0/2236796804 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b58009690 con 0x7f3b540039e0 2024-04-02T13:21:40.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.506+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2236796804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b540039e0 msgr2=0x7f3b54003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.506+0000 7f3b6abe9640 1 --2- 172.21.15.44:0/2236796804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b540039e0 0x7f3b54003e00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009ac0 tx=0x7f3b54005830 comp rx=0 tx=0).stop 2024-04-02T13:21:40.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.506+0000 7f3b69186640 1 -- 172.21.15.44:0/2236796804 reap_dead start 2024-04-02T13:21:40.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.506+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2236796804 shutdown_connections 2024-04-02T13:21:40.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.506+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2236796804 >> 172.21.15.44:0/2236796804 conn(0x7f3b6409d470 msgr2=0x7f3b640d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:40.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.506+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2236796804 shutdown_connections 2024-04-02T13:21:40.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.506+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2236796804 wait complete. 2024-04-02T13:21:40.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.507+0000 7f3b6abe9640 1 Processor -- start 2024-04-02T13:21:40.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.507+0000 7f3b6abe9640 1 -- start start 2024-04-02T13:21:40.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b6abe9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b540039e0 0x7f3b64152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b6abe9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b6415ff50 0x7f3b641603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b6abe9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b640b8180 0x7f3b640b8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b6abe9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6415e6f0 con 0x7f3b640b8180 2024-04-02T13:21:40.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b6abe9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3b640d1270 con 0x7f3b6415ff50 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b6abe9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3b6415e870 con 0x7f3b540039e0 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b69186640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b640b8180 0x7f3b640b8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b69186640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b640b8180 0x7f3b640b8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38208/0 (socket says 172.21.15.44:38208) 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b69186640 1 -- 172.21.15.44:0/2657744356 learned_addr learned my addr 172.21.15.44:0/2657744356 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b68985640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b540039e0 0x7f3b64152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b63fff640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b6415ff50 0x7f3b641603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b69186640 1 -- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b540039e0 msgr2=0x7f3b64152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b69186640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b540039e0 0x7f3b64152f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b69186640 1 -- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b6415ff50 msgr2=0x7f3b641603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b69186640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b6415ff50 0x7f3b641603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.508+0000 7f3b69186640 1 -- 172.21.15.44:0/2657744356 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b640b7de0 con 0x7f3b640b8180 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.509+0000 7f3b69186640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b640b8180 0x7f3b640b8630 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3b640cebf0 tx=0x7f3b5c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.509+0000 7f3b617fa640 1 -- 172.21.15.44:0/2657744356 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c006b50 con 0x7f3b640b8180 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.509+0000 7f3b617fa640 1 -- 172.21.15.44:0/2657744356 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b5c006d10 con 0x7f3b640b8180 2024-04-02T13:21:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.509+0000 7f3b617fa640 1 -- 172.21.15.44:0/2657744356 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c003970 con 0x7f3b640b8180 2024-04-02T13:21:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.509+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2657744356 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b640b8ea0 con 0x7f3b640b8180 2024-04-02T13:21:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.509+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2657744356 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b640b9460 con 0x7f3b640b8180 2024-04-02T13:21:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.510+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2657744356 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b640684f0 con 0x7f3b640b8180 2024-04-02T13:21:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.510+0000 7f3b617fa640 1 -- 172.21.15.44:0/2657744356 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3b5c01a020 con 0x7f3b640b8180 2024-04-02T13:21:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.511+0000 7f3b617fa640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b4806c920 0x7f3b4806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.511+0000 7f3b617fa640 1 -- 172.21.15.44:0/2657744356 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f3b5c08af20 con 0x7f3b640b8180 2024-04-02T13:21:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.511+0000 7f3b68985640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b4806c920 0x7f3b4806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.511+0000 7f3b68985640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b4806c920 0x7f3b4806ede0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3b58036000 tx=0x7f3b58030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:40.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.515+0000 7f3b617fa640 1 -- 172.21.15.44:0/2657744356 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3b5c006f00 con 0x7f3b640b8180 2024-04-02T13:21:40.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.701+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2657744356 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b640d17c0 con 0x7f3b640b8180 2024-04-02T13:21:40.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.702+0000 7f3b617fa640 1 -- 172.21.15.44:0/2657744356 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+13116 (secure 0 0 0) 0x7f3b5c059510 con 0x7f3b640b8180 2024-04-02T13:21:40.705 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:40.705 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":35,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:37.668630+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.704+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b4806c920 msgr2=0x7f3b4806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.704+0000 7f3b6abe9640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b4806c920 0x7f3b4806ede0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3b58036000 tx=0x7f3b58030000 comp rx=0 tx=0).stop 2024-04-02T13:21:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.704+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2657744356 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b640b8180 msgr2=0x7f3b640b8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.704+0000 7f3b6abe9640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b640b8180 0x7f3b640b8630 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3b640cebf0 tx=0x7f3b5c00eb90 comp rx=0 tx=0).stop 2024-04-02T13:21:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.704+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2657744356 shutdown_connections 2024-04-02T13:21:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.704+0000 7f3b6abe9640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b4806c920 0x7f3b4806ede0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.704+0000 7f3b6abe9640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b640b8180 0x7f3b640b8630 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.704+0000 7f3b6abe9640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b6415ff50 0x7f3b641603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.704+0000 7f3b6abe9640 1 --2- 172.21.15.44:0/2657744356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b540039e0 0x7f3b64152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.704+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2657744356 >> 172.21.15.44:0/2657744356 conn(0x7f3b6409d470 msgr2=0x7f3b640b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:40.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.704+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2657744356 shutdown_connections 2024-04-02T13:21:40.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.704+0000 7f3b6abe9640 1 -- 172.21.15.44:0/2657744356 wait complete. 2024-04-02T13:21:40.718 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:21:40.718 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:40.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.821+0000 7f2644281640 1 Processor -- start 2024-04-02T13:21:40.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.821+0000 7f2644281640 1 -- start start 2024-04-02T13:21:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.822+0000 7f2644281640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f263c0aeca0 0x7f263c06e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.822+0000 7f2644281640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f263c0cdd60 con 0x7f263c0d07b0 2024-04-02T13:21:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.822+0000 7f2644281640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f263c0ce040 con 0x7f263c06eb10 2024-04-02T13:21:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.822+0000 7f2644281640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f263c15b3b0 con 0x7f263c0aeca0 2024-04-02T13:21:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.822+0000 7f264201d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f263c0aeca0 0x7f263c06e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.822+0000 7f264281e640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f263c0d07b0 0x7f263c0acde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42384/0 (socket says 172.21.15.44:42384) 2024-04-02T13:21:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.822+0000 7f264281e640 1 -- 172.21.15.44:0/1942573756 learned_addr learned my addr 172.21.15.44:0/1942573756 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.822+0000 7f264201d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f263c0aeca0 0x7f263c06e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38214/0 (socket says 172.21.15.44:38214) 2024-04-02T13:21:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.822+0000 7f264101b640 1 -- 172.21.15.44:0/1942573756 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 230514718 0 0) 0x7f263c0cdd60 con 0x7f263c0d07b0 2024-04-02T13:21:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.822+0000 7f264101b640 1 -- 172.21.15.44:0/1942573756 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2628003520 con 0x7f263c0d07b0 2024-04-02T13:21:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.822+0000 7f264201d640 1 -- 172.21.15.44:0/1942573756 >> v1:172.21.15.69:6789/0 conn(0x7f263c06eb10 legacy=0x7f263c0d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.822+0000 7f264201d640 1 -- 172.21.15.44:0/1942573756 >> v1:172.21.15.44:6789/0 conn(0x7f263c0d07b0 legacy=0x7f263c0acde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.822+0000 7f264201d640 1 -- 172.21.15.44:0/1942573756 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263c0ce320 con 0x7f263c0aeca0 2024-04-02T13:21:40.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.823+0000 7f264201d640 1 --2- 172.21.15.44:0/1942573756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f263c0aeca0 0x7f263c06e5d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f26380097a0 tx=0x7f263802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451a69127d682454 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:40.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.823+0000 7f264101b640 1 -- 172.21.15.44:0/1942573756 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2638034070 con 0x7f263c0aeca0 2024-04-02T13:21:40.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.823+0000 7f264101b640 1 -- 172.21.15.44:0/1942573756 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2638030610 con 0x7f263c0aeca0 2024-04-02T13:21:40.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.823+0000 7f264101b640 1 -- 172.21.15.44:0/1942573756 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26380309e0 con 0x7f263c0aeca0 2024-04-02T13:21:40.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.824+0000 7f2644281640 1 -- 172.21.15.44:0/1942573756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f263c0aeca0 msgr2=0x7f263c06e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.824+0000 7f2644281640 1 --2- 172.21.15.44:0/1942573756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f263c0aeca0 0x7f263c06e5d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f26380097a0 tx=0x7f263802b960 comp rx=0 tx=0).stop 2024-04-02T13:21:40.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.824+0000 7f2644281640 1 -- 172.21.15.44:0/1942573756 shutdown_connections 2024-04-02T13:21:40.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.824+0000 7f2644281640 1 --2- 172.21.15.44:0/1942573756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f263c0aeca0 0x7f263c06e5d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.824+0000 7f2644281640 1 -- 172.21.15.44:0/1942573756 >> 172.21.15.44:0/1942573756 conn(0x7f263c06b920 msgr2=0x7f263c06bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:40.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.824+0000 7f2644281640 1 -- 172.21.15.44:0/1942573756 shutdown_connections 2024-04-02T13:21:40.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.824+0000 7f2644281640 1 -- 172.21.15.44:0/1942573756 wait complete. 2024-04-02T13:21:40.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.825+0000 7f2644281640 1 Processor -- start 2024-04-02T13:21:40.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.825+0000 7f2644281640 1 -- start start 2024-04-02T13:21:40.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f2644281640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f263c06eb10 0x7f263c0b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f2644281640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f263c0aeca0 0x7f263c147dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f2644281640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f263c0d07b0 0x7f263c1e34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f2644281640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f263c0ad4f0 con 0x7f263c0aeca0 2024-04-02T13:21:40.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f2644281640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f263c0ce8e0 con 0x7f263c06eb10 2024-04-02T13:21:40.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f2644281640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f263c0ad670 con 0x7f263c0d07b0 2024-04-02T13:21:40.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f264181c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f263c0aeca0 0x7f263c147dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f264181c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f263c0aeca0 0x7f263c147dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38216/0 (socket says 172.21.15.44:38216) 2024-04-02T13:21:40.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f264181c640 1 -- 172.21.15.44:0/502387997 learned_addr learned my addr 172.21.15.44:0/502387997 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f264281e640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f263c0d07b0 0x7f263c1e34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f264201d640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f263c06eb10 0x7f263c0b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f264181c640 1 -- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f263c0d07b0 msgr2=0x7f263c1e34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f264181c640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f263c0d07b0 0x7f263c1e34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f264181c640 1 -- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f263c06eb10 msgr2=0x7f263c0b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f264181c640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f263c06eb10 0x7f263c0b1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f264181c640 1 -- 172.21.15.44:0/502387997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263c145c60 con 0x7f263c0aeca0 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f264281e640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f263c0d07b0 0x7f263c1e34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f264181c640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f263c0aeca0 0x7f263c147dc0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f262c00c400 tx=0x7f262c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f2632ffd640 1 -- 172.21.15.44:0/502387997 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262c0052d0 con 0x7f263c0aeca0 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f2632ffd640 1 -- 172.21.15.44:0/502387997 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f262c005490 con 0x7f263c0aeca0 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f2644281640 1 -- 172.21.15.44:0/502387997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f263c145f40 con 0x7f263c0aeca0 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f2632ffd640 1 -- 172.21.15.44:0/502387997 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262c007260 con 0x7f263c0aeca0 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.826+0000 7f2644281640 1 -- 172.21.15.44:0/502387997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f263c146500 con 0x7f263c0aeca0 2024-04-02T13:21:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.827+0000 7f2644281640 1 -- 172.21.15.44:0/502387997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f263c0684f0 con 0x7f263c0aeca0 2024-04-02T13:21:40.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.828+0000 7f2632ffd640 1 -- 172.21.15.44:0/502387997 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f262c007400 con 0x7f263c0aeca0 2024-04-02T13:21:40.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.828+0000 7f2632ffd640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f262006c8f0 0x7f262006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:40.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.828+0000 7f2632ffd640 1 -- 172.21.15.44:0/502387997 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f262c08af40 con 0x7f263c0aeca0 2024-04-02T13:21:40.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.828+0000 7f264201d640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f262006c8f0 0x7f262006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:40.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.829+0000 7f264201d640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f262006c8f0 0x7f262006edb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2638036000 tx=0x7f2638008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:40.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:40.832+0000 7f2632ffd640 1 -- 172.21.15.44:0/502387997 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f262c0594b0 con 0x7f263c0aeca0 2024-04-02T13:21:41.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.018+0000 7f2644281640 1 -- 172.21.15.44:0/502387997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f263c06e510 con 0x7f263c0aeca0 2024-04-02T13:21:41.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.018+0000 7f2632ffd640 1 -- 172.21.15.44:0/502387997 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) ==== 93+0+10 (secure 0 0 0) 0x7f262c058db0 con 0x7f263c0aeca0 2024-04-02T13:21:41.021 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:41.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.020+0000 7f2644281640 1 -- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f262006c8f0 msgr2=0x7f262006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.020+0000 7f2644281640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f262006c8f0 0x7f262006edb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2638036000 tx=0x7f2638008030 comp rx=0 tx=0).stop 2024-04-02T13:21:41.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.020+0000 7f2644281640 1 -- 172.21.15.44:0/502387997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f263c0aeca0 msgr2=0x7f263c147dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.020+0000 7f2644281640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f263c0aeca0 0x7f263c147dc0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f262c00c400 tx=0x7f262c00c8c0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.020+0000 7f2644281640 1 -- 172.21.15.44:0/502387997 shutdown_connections 2024-04-02T13:21:41.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.020+0000 7f2644281640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f262006c8f0 0x7f262006edb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.020+0000 7f2644281640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f263c0d07b0 0x7f263c1e34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.020+0000 7f2644281640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f263c0aeca0 0x7f263c147dc0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.020+0000 7f2644281640 1 --2- 172.21.15.44:0/502387997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f263c06eb10 0x7f263c0b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.020+0000 7f2644281640 1 -- 172.21.15.44:0/502387997 >> 172.21.15.44:0/502387997 conn(0x7f263c06b920 msgr2=0x7f263c0b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:41.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.020+0000 7f2644281640 1 -- 172.21.15.44:0/502387997 shutdown_connections 2024-04-02T13:21:41.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.020+0000 7f2644281640 1 -- 172.21.15.44:0/502387997 wait complete. 2024-04-02T13:21:41.033 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T13:21:41.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.132+0000 7fd0b820d640 1 Processor -- start 2024-04-02T13:21:41.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.132+0000 7fd0b820d640 1 -- start start 2024-04-02T13:21:41.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.133+0000 7fd0b820d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd0b00d0a30 0x7fd0b006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.133+0000 7fd0b820d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0b00d2040 0x7fd0b00d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.133+0000 7fd0b820d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0b00d0000 con 0x7fd0b00d0e10 2024-04-02T13:21:41.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.133+0000 7fd0b820d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd0b006cb00 con 0x7fd0b00d0a30 2024-04-02T13:21:41.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.133+0000 7fd0b820d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd0b006c1d0 con 0x7fd0b00d2040 2024-04-02T13:21:41.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.133+0000 7fd0b5fa9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd0b00d0a30 0x7fd0b006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.133+0000 7fd0b57a8640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd0b00d0e10 0x7fd0b00cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42392/0 (socket says 172.21.15.44:42392) 2024-04-02T13:21:41.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.133+0000 7fd0b5fa9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd0b00d0a30 0x7fd0b006cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38228/0 (socket says 172.21.15.44:38228) 2024-04-02T13:21:41.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.133+0000 7fd0b57a8640 1 -- 172.21.15.44:0/4275121245 learned_addr learned my addr 172.21.15.44:0/4275121245 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:41.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.134+0000 7fd0b5fa9640 1 -- 172.21.15.44:0/4275121245 >> v2:172.21.15.69:3300/0 conn(0x7fd0b00d2040 msgr2=0x7fd0b00d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:41.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.134+0000 7fd0b67aa640 1 --2- 172.21.15.44:0/4275121245 >> v2:172.21.15.69:3300/0 conn(0x7fd0b00d2040 0x7fd0b00d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.134+0000 7fd0b5fa9640 1 --2- 172.21.15.44:0/4275121245 >> v2:172.21.15.69:3300/0 conn(0x7fd0b00d2040 0x7fd0b00d24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.134+0000 7fd0b5fa9640 1 -- 172.21.15.44:0/4275121245 >> v1:172.21.15.44:6789/0 conn(0x7fd0b00d0e10 legacy=0x7fd0b00cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.134+0000 7fd0b4fa7640 1 -- 172.21.15.44:0/4275121245 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1113702265 0 0) 0x7fd0b00d0000 con 0x7fd0b00d0e10 2024-04-02T13:21:41.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.134+0000 7fd0b5fa9640 1 -- 172.21.15.44:0/4275121245 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b006c4b0 con 0x7fd0b00d0a30 2024-04-02T13:21:41.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.134+0000 7fd0b5fa9640 1 --2- 172.21.15.44:0/4275121245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0b00d0a30 0x7fd0b006cf90 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd0b008c000 tx=0x7fd0a002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267af6470b92304b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:41.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.135+0000 7fd0b4fa7640 1 -- 172.21.15.44:0/4275121245 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0039070 con 0x7fd0b00d0a30 2024-04-02T13:21:41.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.135+0000 7fd0b4fa7640 1 -- 172.21.15.44:0/4275121245 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a002cbf0 con 0x7fd0b00d0a30 2024-04-02T13:21:41.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.135+0000 7fd0b4fa7640 1 -- 172.21.15.44:0/4275121245 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0034440 con 0x7fd0b00d0a30 2024-04-02T13:21:41.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.135+0000 7fd0b820d640 1 -- 172.21.15.44:0/4275121245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0b00d0a30 msgr2=0x7fd0b006cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.135+0000 7fd0b820d640 1 --2- 172.21.15.44:0/4275121245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0b00d0a30 0x7fd0b006cf90 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd0b008c000 tx=0x7fd0a002be40 comp rx=0 tx=0).stop 2024-04-02T13:21:41.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.135+0000 7fd0b820d640 1 -- 172.21.15.44:0/4275121245 shutdown_connections 2024-04-02T13:21:41.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.135+0000 7fd0b820d640 1 --2- 172.21.15.44:0/4275121245 >> v2:172.21.15.69:3300/0 conn(0x7fd0b00d2040 0x7fd0b00d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.135+0000 7fd0b820d640 1 --2- 172.21.15.44:0/4275121245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0b00d0a30 0x7fd0b006cf90 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.135+0000 7fd0b820d640 1 -- 172.21.15.44:0/4275121245 >> 172.21.15.44:0/4275121245 conn(0x7fd0b009d470 msgr2=0x7fd0b00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:41.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.135+0000 7fd0b820d640 1 -- 172.21.15.44:0/4275121245 shutdown_connections 2024-04-02T13:21:41.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.135+0000 7fd0b820d640 1 -- 172.21.15.44:0/4275121245 wait complete. 2024-04-02T13:21:41.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.136+0000 7fd0b820d640 1 Processor -- start 2024-04-02T13:21:41.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.136+0000 7fd0b820d640 1 -- start start 2024-04-02T13:21:41.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b820d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0b00d0a30 0x7fd0b00cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b820d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0b00d0e10 0x7fd0b015fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b820d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0b00d2040 0x7fd0b00c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b820d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b015e6f0 con 0x7fd0b00d2040 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b820d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b00d1270 con 0x7fd0b00d0a30 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b820d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd0b015e870 con 0x7fd0b00d0e10 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b67aa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0b00d2040 0x7fd0b00c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b67aa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0b00d2040 0x7fd0b00c8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38244/0 (socket says 172.21.15.44:38244) 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b67aa640 1 -- 172.21.15.44:0/1564732393 learned_addr learned my addr 172.21.15.44:0/1564732393 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b57a8640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0b00d0e10 0x7fd0b015fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b5fa9640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0b00d0a30 0x7fd0b00cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b67aa640 1 -- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0b00d0e10 msgr2=0x7fd0b015fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b67aa640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0b00d0e10 0x7fd0b015fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b67aa640 1 -- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0b00d0a30 msgr2=0x7fd0b00cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b67aa640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0b00d0a30 0x7fd0b00cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b67aa640 1 -- 172.21.15.44:0/1564732393 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b00b3d40 con 0x7fd0b00d2040 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.137+0000 7fd0b5fa9640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0b00d0a30 0x7fd0b00cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.138+0000 7fd0b67aa640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0b00d2040 0x7fd0b00c8930 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800c550 tx=0x7fd0a800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:41.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.138+0000 7fd0ae7fc640 1 -- 172.21.15.44:0/1564732393 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8015070 con 0x7fd0b00d2040 2024-04-02T13:21:41.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.138+0000 7fd0ae7fc640 1 -- 172.21.15.44:0/1564732393 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a8004220 con 0x7fd0b00d2040 2024-04-02T13:21:41.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.138+0000 7fd0ae7fc640 1 -- 172.21.15.44:0/1564732393 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8003610 con 0x7fd0b00d2040 2024-04-02T13:21:41.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.138+0000 7fd0b820d640 1 -- 172.21.15.44:0/1564732393 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b00b4020 con 0x7fd0b00d2040 2024-04-02T13:21:41.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.138+0000 7fd0b820d640 1 -- 172.21.15.44:0/1564732393 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b00b45e0 con 0x7fd0b00d2040 2024-04-02T13:21:41.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.139+0000 7fd0b820d640 1 -- 172.21.15.44:0/1564732393 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0b00684f0 con 0x7fd0b00d2040 2024-04-02T13:21:41.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.139+0000 7fd0ae7fc640 1 -- 172.21.15.44:0/1564732393 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd0a80037b0 con 0x7fd0b00d2040 2024-04-02T13:21:41.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.140+0000 7fd0ae7fc640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd09c06c8f0 0x7fd09c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.140+0000 7fd0ae7fc640 1 -- 172.21.15.44:0/1564732393 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fd0a808b760 con 0x7fd0b00d2040 2024-04-02T13:21:41.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.140+0000 7fd0b5fa9640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd09c06c8f0 0x7fd09c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.140+0000 7fd0b5fa9640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd09c06c8f0 0x7fd09c06edb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0036000 tx=0x7fd0a0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:41.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.144+0000 7fd0ae7fc640 1 -- 172.21.15.44:0/1564732393 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd0a8059d50 con 0x7fd0b00d2040 2024-04-02T13:21:41.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.327+0000 7fd0b820d640 1 -- 172.21.15.44:0/1564732393 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd0b015d640 con 0x7fd0b00d2040 2024-04-02T13:21:41.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.328+0000 7fd0ae7fc640 1 -- 172.21.15.44:0/1564732393 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) ==== 84+0+65 (secure 0 0 0) 0x7fd0a8059650 con 0x7fd0b00d2040 2024-04-02T13:21:41.330 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:41.331 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-04-02T13:21:41.331 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:21:41.331 INFO:teuthology.orchestra.run.smithi044.stdout: "poolnum": 1, 2024-04-02T13:21:41.331 INFO:teuthology.orchestra.run.smithi044.stdout: "poolname": ".mgr" 2024-04-02T13:21:41.331 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:41.331 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-04-02T13:21:41.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.330+0000 7fd0b820d640 1 -- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd09c06c8f0 msgr2=0x7fd09c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.330+0000 7fd0b820d640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd09c06c8f0 0x7fd09c06edb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0036000 tx=0x7fd0a0030000 comp rx=0 tx=0).stop 2024-04-02T13:21:41.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.330+0000 7fd0b820d640 1 -- 172.21.15.44:0/1564732393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0b00d2040 msgr2=0x7fd0b00c8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.330+0000 7fd0b820d640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0b00d2040 0x7fd0b00c8930 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800c550 tx=0x7fd0a800ca10 comp rx=0 tx=0).stop 2024-04-02T13:21:41.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.330+0000 7fd0b820d640 1 -- 172.21.15.44:0/1564732393 shutdown_connections 2024-04-02T13:21:41.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.330+0000 7fd0b820d640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd09c06c8f0 0x7fd09c06edb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.330+0000 7fd0b820d640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0b00d2040 0x7fd0b00c8930 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.330+0000 7fd0b820d640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0b00d0e10 0x7fd0b015fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.330+0000 7fd0b820d640 1 --2- 172.21.15.44:0/1564732393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0b00d0a30 0x7fd0b00cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.330+0000 7fd0b820d640 1 -- 172.21.15.44:0/1564732393 >> 172.21.15.44:0/1564732393 conn(0x7fd0b009d470 msgr2=0x7fd0b00b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:41.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.330+0000 7fd0b820d640 1 -- 172.21.15.44:0/1564732393 shutdown_connections 2024-04-02T13:21:41.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.330+0000 7fd0b820d640 1 -- 172.21.15.44:0/1564732393 wait complete. 2024-04-02T13:21:41.343 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T13:21:41.343 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T13:21:41.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.446+0000 7f7e93c7b640 1 Processor -- start 2024-04-02T13:21:41.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.446+0000 7f7e93c7b640 1 -- start start 2024-04-02T13:21:41.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e93c7b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e8c0b1540 0x7f7e8c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e93c7b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e8c0ced00 con 0x7f7e8c0ce790 2024-04-02T13:21:41.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e93c7b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e8c0cefe0 con 0x7f7e8c15e470 2024-04-02T13:21:41.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e93c7b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7e8c0cf2c0 con 0x7f7e8c0b1540 2024-04-02T13:21:41.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e92218640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7e8c0ce790 0x7f7e8c06d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42400/0 (socket says 172.21.15.44:42400) 2024-04-02T13:21:41.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e92218640 1 -- 172.21.15.44:0/1582761375 learned_addr learned my addr 172.21.15.44:0/1582761375 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:41.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1659541241 0 0) 0x7f7e8c0ced00 con 0x7f7e8c0ce790 2024-04-02T13:21:41.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e78003520 con 0x7f7e8c0ce790 2024-04-02T13:21:41.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e91216640 1 --2- 172.21.15.44:0/1582761375 >> v2:172.21.15.69:3300/0 conn(0x7f7e8c0b1540 0x7f7e8c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3598318704 0 0) 0x7f7e78003520 con 0x7f7e8c0ce790 2024-04-02T13:21:41.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 >> v2:172.21.15.69:3300/0 conn(0x7f7e8c0b1540 msgr2=0x7f7e8c0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:41.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e90a15640 1 --2- 172.21.15.44:0/1582761375 >> v2:172.21.15.69:3300/0 conn(0x7f7e8c0b1540 0x7f7e8c0b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 >> v1:172.21.15.69:6789/0 conn(0x7f7e8c15e470 legacy=0x7f7e8c0d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8c0cf5a0 con 0x7f7e8c0ce790 2024-04-02T13:21:41.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f7e840019f0 con 0x7f7e8c0ce790 2024-04-02T13:21:41.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 >> v1:172.21.15.44:6789/0 conn(0x7f7e8c0ce790 legacy=0x7f7e8c06d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e90a15640 1 --2- 172.21.15.44:0/1582761375 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e780039b0 0x7f7e78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.447+0000 7f7e90a15640 1 --2- 172.21.15.44:0/1582761375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e78004800 0x7f7e78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.448+0000 7f7e90a15640 1 --2- 172.21.15.44:0/1582761375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e780051c0 0x7f7e7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.448+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8c0cf2c0 con 0x7f7e780051c0 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.448+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7800c160 con 0x7f7e78004800 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.448+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7e7800c360 con 0x7f7e780039b0 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.448+0000 7f7e92218640 1 --2- 172.21.15.44:0/1582761375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e780051c0 0x7f7e7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.448+0000 7f7e91a17640 1 --2- 172.21.15.44:0/1582761375 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e780039b0 0x7f7e78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.448+0000 7f7e91216640 1 --2- 172.21.15.44:0/1582761375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e78004800 0x7f7e78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.448+0000 7f7e92218640 1 -- 172.21.15.44:0/1582761375 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e780039b0 msgr2=0x7f7e78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.448+0000 7f7e92218640 1 --2- 172.21.15.44:0/1582761375 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e780039b0 0x7f7e78003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.448+0000 7f7e92218640 1 -- 172.21.15.44:0/1582761375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e78004800 msgr2=0x7f7e78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.448+0000 7f7e92218640 1 --2- 172.21.15.44:0/1582761375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e78004800 0x7f7e78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.448+0000 7f7e92218640 1 -- 172.21.15.44:0/1582761375 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e78003520 con 0x7f7e780051c0 2024-04-02T13:21:41.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.449+0000 7f7e92218640 1 -- 172.21.15.44:0/1582761375 reap_dead start 2024-04-02T13:21:41.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.449+0000 7f7e92218640 1 --2- 172.21.15.44:0/1582761375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e780051c0 0x7f7e7800b9c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400a840 tx=0x7f7e78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec73221f43ea44c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:41.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.449+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8400a460 con 0x7f7e780051c0 2024-04-02T13:21:41.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.449+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e84034650 con 0x7f7e780051c0 2024-04-02T13:21:41.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.449+0000 7f7e90a15640 1 -- 172.21.15.44:0/1582761375 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e84034a40 con 0x7f7e780051c0 2024-04-02T13:21:41.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.449+0000 7f7e93c7b640 1 -- 172.21.15.44:0/1582761375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e780051c0 msgr2=0x7f7e7800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.449+0000 7f7e93c7b640 1 --2- 172.21.15.44:0/1582761375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e780051c0 0x7f7e7800b9c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400a840 tx=0x7f7e78005830 comp rx=0 tx=0).stop 2024-04-02T13:21:41.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.449+0000 7f7e93c7b640 1 -- 172.21.15.44:0/1582761375 shutdown_connections 2024-04-02T13:21:41.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.449+0000 7f7e93c7b640 1 --2- 172.21.15.44:0/1582761375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e780051c0 0x7f7e7800b9c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.449+0000 7f7e93c7b640 1 -- 172.21.15.44:0/1582761375 >> 172.21.15.44:0/1582761375 conn(0x7f7e8c09aa20 msgr2=0x7f7e8c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:41.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.449+0000 7f7e93c7b640 1 -- 172.21.15.44:0/1582761375 shutdown_connections 2024-04-02T13:21:41.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.449+0000 7f7e93c7b640 1 -- 172.21.15.44:0/1582761375 wait complete. 2024-04-02T13:21:41.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.451+0000 7f7e93c7b640 1 Processor -- start 2024-04-02T13:21:41.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.451+0000 7f7e93c7b640 1 -- start start 2024-04-02T13:21:41.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.451+0000 7f7e93c7b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e780051c0 0x7f7e8c150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.451+0000 7f7e93c7b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e8c14d3d0 0x7f7e8c14ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.451+0000 7f7e93c7b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e8c14d830 0x7f7e8c14f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.451+0000 7f7e93c7b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8c0b32c0 con 0x7f7e780051c0 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.451+0000 7f7e93c7b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8c06cb60 con 0x7f7e8c14d830 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.451+0000 7f7e93c7b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7e8c0b3440 con 0x7f7e8c14d3d0 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.451+0000 7f7e91a17640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e780051c0 0x7f7e8c150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.451+0000 7f7e91a17640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e780051c0 0x7f7e8c150b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38252/0 (socket says 172.21.15.44:38252) 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.451+0000 7f7e91a17640 1 -- 172.21.15.44:0/3013070456 learned_addr learned my addr 172.21.15.44:0/3013070456 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e91216640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e8c14d3d0 0x7f7e8c14ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e92218640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e8c14d830 0x7f7e8c14f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e91a17640 1 -- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e8c14d3d0 msgr2=0x7f7e8c14ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e91a17640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e8c14d3d0 0x7f7e8c14ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e91a17640 1 -- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e8c14d830 msgr2=0x7f7e8c14f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e91a17640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e8c14d830 0x7f7e8c14f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e91a17640 1 -- 172.21.15.44:0/3013070456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8c14fa90 con 0x7f7e780051c0 2024-04-02T13:21:41.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e92218640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e8c14d830 0x7f7e8c14f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:41.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e91a17640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e780051c0 0x7f7e8c150b30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800f620 tx=0x7f7e8800fa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:41.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e827fc640 1 -- 172.21.15.44:0/3013070456 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e880052e0 con 0x7f7e780051c0 2024-04-02T13:21:41.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e827fc640 1 -- 172.21.15.44:0/3013070456 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e88018070 con 0x7f7e780051c0 2024-04-02T13:21:41.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e93c7b640 1 -- 172.21.15.44:0/3013070456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e8c14fd70 con 0x7f7e780051c0 2024-04-02T13:21:41.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e827fc640 1 -- 172.21.15.44:0/3013070456 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e88003830 con 0x7f7e780051c0 2024-04-02T13:21:41.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.452+0000 7f7e93c7b640 1 -- 172.21.15.44:0/3013070456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e8c150330 con 0x7f7e780051c0 2024-04-02T13:21:41.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.453+0000 7f7e93c7b640 1 -- 172.21.15.44:0/3013070456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e8c0684f0 con 0x7f7e780051c0 2024-04-02T13:21:41.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.454+0000 7f7e827fc640 1 -- 172.21.15.44:0/3013070456 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7e880039d0 con 0x7f7e780051c0 2024-04-02T13:21:41.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.454+0000 7f7e827fc640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7e7006c8f0 0x7f7e7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.454+0000 7f7e827fc640 1 -- 172.21.15.44:0/3013070456 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f7e8808b5a0 con 0x7f7e780051c0 2024-04-02T13:21:41.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.454+0000 7f7e91216640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7e7006c8f0 0x7f7e7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.454+0000 7f7e91216640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7e7006c8f0 0x7f7e7006edb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c1f77e0 tx=0x7f7e7c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:41.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.458+0000 7f7e827fc640 1 -- 172.21.15.44:0/3013070456 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7e88059ba0 con 0x7f7e780051c0 2024-04-02T13:21:41.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.642+0000 7f7e93c7b640 1 -- 172.21.15.44:0/3013070456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f7e8c15d7d0 con 0x7f7e780051c0 2024-04-02T13:21:41.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.843+0000 7f7e827fc640 1 -- 172.21.15.44:0/3013070456 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) ==== 138+0+0 (secure 0 0 0) 0x7f7e880594a0 con 0x7f7e780051c0 2024-04-02T13:21:41.846 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' created 2024-04-02T13:21:41.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.846+0000 7f7e93c7b640 1 -- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7e7006c8f0 msgr2=0x7f7e7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.846+0000 7f7e93c7b640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7e7006c8f0 0x7f7e7006edb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c1f77e0 tx=0x7f7e7c009190 comp rx=0 tx=0).stop 2024-04-02T13:21:41.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.846+0000 7f7e93c7b640 1 -- 172.21.15.44:0/3013070456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e780051c0 msgr2=0x7f7e8c150b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.846+0000 7f7e93c7b640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e780051c0 0x7f7e8c150b30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800f620 tx=0x7f7e8800fa20 comp rx=0 tx=0).stop 2024-04-02T13:21:41.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.846+0000 7f7e93c7b640 1 -- 172.21.15.44:0/3013070456 shutdown_connections 2024-04-02T13:21:41.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.846+0000 7f7e93c7b640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7e7006c8f0 0x7f7e7006edb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.846+0000 7f7e93c7b640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e8c14d830 0x7f7e8c14f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.846+0000 7f7e93c7b640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e8c14d3d0 0x7f7e8c14ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.846+0000 7f7e93c7b640 1 --2- 172.21.15.44:0/3013070456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e780051c0 0x7f7e8c150b30 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.846+0000 7f7e93c7b640 1 -- 172.21.15.44:0/3013070456 >> 172.21.15.44:0/3013070456 conn(0x7f7e8c09aa20 msgr2=0x7f7e8c06f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:41.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.846+0000 7f7e93c7b640 1 -- 172.21.15.44:0/3013070456 shutdown_connections 2024-04-02T13:21:41.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.846+0000 7f7e93c7b640 1 -- 172.21.15.44:0/3013070456 wait complete. 2024-04-02T13:21:41.862 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T13:21:41.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.968+0000 7fa4ee293640 1 Processor -- start 2024-04-02T13:21:41.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.968+0000 7fa4ee293640 1 -- start start 2024-04-02T13:21:41.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.969+0000 7fa4ee293640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa4e80d00a0 0x7fa4e80d0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.969+0000 7fa4ee293640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa4e80acde0 0x7fa4e80ad210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.969+0000 7fa4ee293640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e80cdd60 con 0x7fa4e80aeca0 2024-04-02T13:21:41.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.969+0000 7fa4ee293640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa4e815b3b0 con 0x7fa4e80d00a0 2024-04-02T13:21:41.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.969+0000 7fa4ee293640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa4e80ce040 con 0x7fa4e80acde0 2024-04-02T13:21:41.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.969+0000 7fa4e77fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa4e80d00a0 0x7fa4e80d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.969+0000 7fa4e77fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa4e80d00a0 0x7fa4e80d0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38260/0 (socket says 172.21.15.44:38260) 2024-04-02T13:21:41.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.969+0000 7fa4e77fe640 1 -- 172.21.15.44:0/1374774748 learned_addr learned my addr 172.21.15.44:0/1374774748 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:41.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.969+0000 7fa4ec830640 1 --2- 172.21.15.44:0/1374774748 >> v2:172.21.15.69:3300/0 conn(0x7fa4e80acde0 0x7fa4e80ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.970+0000 7fa4e77fe640 1 -- 172.21.15.44:0/1374774748 >> v2:172.21.15.69:3300/0 conn(0x7fa4e80acde0 msgr2=0x7fa4e80ad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:41.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.970+0000 7fa4e77fe640 1 --2- 172.21.15.44:0/1374774748 >> v2:172.21.15.69:3300/0 conn(0x7fa4e80acde0 0x7fa4e80ad210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.970+0000 7fa4e77fe640 1 -- 172.21.15.44:0/1374774748 >> v1:172.21.15.69:6789/0 conn(0x7fa4e80aeca0 legacy=0x7fa4e806e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.970+0000 7fa4e77fe640 1 -- 172.21.15.44:0/1374774748 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e80ce320 con 0x7fa4e80d00a0 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.970+0000 7fa4ec830640 1 --2- 172.21.15.44:0/1374774748 >> v2:172.21.15.69:3300/0 conn(0x7fa4e80acde0 0x7fa4e80ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.970+0000 7fa4e77fe640 1 --2- 172.21.15.44:0/1374774748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4e80d00a0 0x7fa4e80d0530 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80735b0 tx=0x7fa4dc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78c4805b76a5b24 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.970+0000 7fa4e6ffd640 1 -- 172.21.15.44:0/1374774748 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc034070 con 0x7fa4e80d00a0 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.970+0000 7fa4e6ffd640 1 -- 172.21.15.44:0/1374774748 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4dc030860 con 0x7fa4e80d00a0 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.970+0000 7fa4e6ffd640 1 -- 172.21.15.44:0/1374774748 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc030c30 con 0x7fa4e80d00a0 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.971+0000 7fa4ee293640 1 -- 172.21.15.44:0/1374774748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4e80d00a0 msgr2=0x7fa4e80d0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.971+0000 7fa4ee293640 1 --2- 172.21.15.44:0/1374774748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4e80d00a0 0x7fa4e80d0530 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80735b0 tx=0x7fa4dc02b9a0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.971+0000 7fa4ee293640 1 -- 172.21.15.44:0/1374774748 shutdown_connections 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.971+0000 7fa4ee293640 1 --2- 172.21.15.44:0/1374774748 >> v2:172.21.15.69:3300/0 conn(0x7fa4e80acde0 0x7fa4e80ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.971+0000 7fa4ee293640 1 --2- 172.21.15.44:0/1374774748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4e80d00a0 0x7fa4e80d0530 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.971+0000 7fa4ee293640 1 -- 172.21.15.44:0/1374774748 >> 172.21.15.44:0/1374774748 conn(0x7fa4e806b920 msgr2=0x7fa4e806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.971+0000 7fa4ee293640 1 -- 172.21.15.44:0/1374774748 shutdown_connections 2024-04-02T13:21:41.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.971+0000 7fa4ee293640 1 -- 172.21.15.44:0/1374774748 wait complete. 2024-04-02T13:21:41.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.972+0000 7fa4ee293640 1 Processor -- start 2024-04-02T13:21:41.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.972+0000 7fa4ee293640 1 -- start start 2024-04-02T13:21:41.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.973+0000 7fa4ee293640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4e80acde0 0x7fa4e81482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.973+0000 7fa4ee293640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4e80aeca0 0x7fa4e81495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.973+0000 7fa4ee293640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4e80d00a0 0x7fa4e815b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.973+0000 7fa4ee293640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e806ece0 con 0x7fa4e80aeca0 2024-04-02T13:21:41.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.973+0000 7fa4ee293640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e80ce8e0 con 0x7fa4e80d00a0 2024-04-02T13:21:41.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.973+0000 7fa4ee293640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa4e806ee60 con 0x7fa4e80acde0 2024-04-02T13:21:41.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.973+0000 7fa4e77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4e80aeca0 0x7fa4e81495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.973+0000 7fa4e77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4e80aeca0 0x7fa4e81495c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38276/0 (socket says 172.21.15.44:38276) 2024-04-02T13:21:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.973+0000 7fa4e77fe640 1 -- 172.21.15.44:0/2354666582 learned_addr learned my addr 172.21.15.44:0/2354666582 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.973+0000 7fa4ec830640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4e80d00a0 0x7fa4e815b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.973+0000 7fa4e7fff640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4e80acde0 0x7fa4e81482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.974+0000 7fa4e77fe640 1 -- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4e80acde0 msgr2=0x7fa4e81482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.974+0000 7fa4e77fe640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4e80acde0 0x7fa4e81482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.974+0000 7fa4e77fe640 1 -- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4e80d00a0 msgr2=0x7fa4e815b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.974+0000 7fa4e77fe640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4e80d00a0 0x7fa4e815b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.974+0000 7fa4e77fe640 1 -- 172.21.15.44:0/2354666582 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e8148810 con 0x7fa4e80aeca0 2024-04-02T13:21:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.974+0000 7fa4e77fe640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4e80aeca0 0x7fa4e81495c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc000c00 tx=0x7fa4dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.974+0000 7fa4e4ff9640 1 -- 172.21.15.44:0/2354666582 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc034040 con 0x7fa4e80aeca0 2024-04-02T13:21:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.974+0000 7fa4e4ff9640 1 -- 172.21.15.44:0/2354666582 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4dc030660 con 0x7fa4e80aeca0 2024-04-02T13:21:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.974+0000 7fa4ee293640 1 -- 172.21.15.44:0/2354666582 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4e815dbd0 con 0x7fa4e80aeca0 2024-04-02T13:21:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.974+0000 7fa4e4ff9640 1 -- 172.21.15.44:0/2354666582 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0423f0 con 0x7fa4e80aeca0 2024-04-02T13:21:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.974+0000 7fa4ee293640 1 -- 172.21.15.44:0/2354666582 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4e815e190 con 0x7fa4e80aeca0 2024-04-02T13:21:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.975+0000 7fa4ee293640 1 -- 172.21.15.44:0/2354666582 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4e80684f0 con 0x7fa4e80aeca0 2024-04-02T13:21:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.975+0000 7fa4e4ff9640 1 -- 172.21.15.44:0/2354666582 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa4dc037070 con 0x7fa4e80aeca0 2024-04-02T13:21:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.976+0000 7fa4e4ff9640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa4d406c8f0 0x7fa4d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.976+0000 7fa4e4ff9640 1 -- 172.21.15.44:0/2354666582 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5619+0+0 (secure 0 0 0) 0x7fa4dc0add80 con 0x7fa4e80aeca0 2024-04-02T13:21:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.976+0000 7fa4e7fff640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa4d406c8f0 0x7fa4d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.977+0000 7fa4e7fff640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa4d406c8f0 0x7fa4d406edb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8154f90 tx=0x7fa4d8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:41.980+0000 7fa4e4ff9640 1 -- 172.21.15.44:0/2354666582 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa4e8148810 con 0x7fa4e80aeca0 2024-04-02T13:21:42.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.163+0000 7fa4ee293640 1 -- 172.21.15.44:0/2354666582 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fa4e806bda0 con 0x7fa4e80aeca0 2024-04-02T13:21:42.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.846+0000 7fa4e4ff9640 1 -- 172.21.15.44:0/2354666582 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) ==== 170+0+0 (secure 0 0 0) 0x7fa4dc07bb10 con 0x7fa4e80aeca0 2024-04-02T13:21:42.849 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' created 2024-04-02T13:21:42.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.848+0000 7fa4ee293640 1 -- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa4d406c8f0 msgr2=0x7fa4d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:42.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.848+0000 7fa4ee293640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa4d406c8f0 0x7fa4d406edb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8154f90 tx=0x7fa4d8009040 comp rx=0 tx=0).stop 2024-04-02T13:21:42.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.848+0000 7fa4ee293640 1 -- 172.21.15.44:0/2354666582 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4e80aeca0 msgr2=0x7fa4e81495c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:42.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.848+0000 7fa4ee293640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4e80aeca0 0x7fa4e81495c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc000c00 tx=0x7fa4dc0046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:42.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.849+0000 7fa4ee293640 1 -- 172.21.15.44:0/2354666582 shutdown_connections 2024-04-02T13:21:42.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.849+0000 7fa4ee293640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa4d406c8f0 0x7fa4d406edb0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:42.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.849+0000 7fa4ee293640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4e80d00a0 0x7fa4e815b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:42.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.849+0000 7fa4ee293640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4e80aeca0 0x7fa4e81495c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:42.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.849+0000 7fa4ee293640 1 --2- 172.21.15.44:0/2354666582 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4e80acde0 0x7fa4e81482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:42.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.849+0000 7fa4ee293640 1 -- 172.21.15.44:0/2354666582 >> 172.21.15.44:0/2354666582 conn(0x7fa4e806b920 msgr2=0x7fa4e80add40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:42.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.849+0000 7fa4ee293640 1 -- 172.21.15.44:0/2354666582 shutdown_connections 2024-04-02T13:21:42.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.849+0000 7fa4ee293640 1 -- 172.21.15.44:0/2354666582 wait complete. 2024-04-02T13:21:42.874 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T13:21:42.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.979+0000 7fe001b60640 1 Processor -- start 2024-04-02T13:21:42.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.979+0000 7fe001b60640 1 -- start start 2024-04-02T13:21:42.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fe001b60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc15e640 0x7fdffc0d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:42.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fe001b60640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdffc0cf120 con 0x7fdffc15e260 2024-04-02T13:21:42.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fe001b60640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdffc0cf400 con 0x7fdffc0cec60 2024-04-02T13:21:42.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fe001b60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdffc0b5a50 con 0x7fdffc15e640 2024-04-02T13:21:42.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffb7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fdffc15e260 0x7fdffc0d2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42404/0 (socket says 172.21.15.44:42404) 2024-04-02T13:21:42.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffb7fe640 1 -- 172.21.15.44:0/4150271191 learned_addr learned my addr 172.21.15.44:0/4150271191 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:42.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffaffd640 1 --2- 172.21.15.44:0/4150271191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc15e640 0x7fdffc0d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:42.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3702304674 0 0) 0x7fdffc0cf120 con 0x7fdffc15e260 2024-04-02T13:21:42.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfe4003520 con 0x7fdffc15e260 2024-04-02T13:21:42.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4548271 0 0) 0x7fdffc0cf400 con 0x7fdffc0cec60 2024-04-02T13:21:42.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdffc0cf120 con 0x7fdffc0cec60 2024-04-02T13:21:42.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 774953761 0 0) 0x7fdfe4003520 con 0x7fdffc15e260 2024-04-02T13:21:42.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc15e640 msgr2=0x7fdffc0d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:42.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffa7fc640 1 --2- 172.21.15.44:0/4150271191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc15e640 0x7fdffc0d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:42.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 >> v1:172.21.15.69:6789/0 conn(0x7fdffc0cec60 legacy=0x7fdffc168280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:42.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdffc0cf6e0 con 0x7fdffc15e260 2024-04-02T13:21:42.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fdfec001670 con 0x7fdffc15e260 2024-04-02T13:21:42.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.980+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 >> v1:172.21.15.44:6789/0 conn(0x7fdffc15e260 legacy=0x7fdffc0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:42.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffa7fc640 1 --2- 172.21.15.44:0/4150271191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc15e640 0x7fdfe4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:42.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffa7fc640 1 --2- 172.21.15.44:0/4150271191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfe40040d0 0x7fdfe4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:42.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffa7fc640 1 --2- 172.21.15.44:0/4150271191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdfe4004d40 0x7fdfe40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:42.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc0b5a50 con 0x7fdfe4004d40 2024-04-02T13:21:42.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe400baa0 con 0x7fdfe40040d0 2024-04-02T13:21:42.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdfe400bca0 con 0x7fdffc15e640 2024-04-02T13:21:42.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffbfff640 1 --2- 172.21.15.44:0/4150271191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdfe4004d40 0x7fdfe40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:42.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffb7fe640 1 --2- 172.21.15.44:0/4150271191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc15e640 0x7fdfe4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:42.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffaffd640 1 --2- 172.21.15.44:0/4150271191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfe40040d0 0x7fdfe4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:42.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffbfff640 1 -- 172.21.15.44:0/4150271191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc15e640 msgr2=0x7fdfe4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:42.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffbfff640 1 --2- 172.21.15.44:0/4150271191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc15e640 0x7fdfe4003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffbfff640 1 -- 172.21.15.44:0/4150271191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfe40040d0 msgr2=0x7fdfe4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffbfff640 1 --2- 172.21.15.44:0/4150271191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfe40040d0 0x7fdfe4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffbfff640 1 -- 172.21.15.44:0/4150271191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe4003520 con 0x7fdfe4004d40 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.981+0000 7fdffb7fe640 1 --2- 172.21.15.44:0/4150271191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc15e640 0x7fdfe4003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.982+0000 7fdffbfff640 1 --2- 172.21.15.44:0/4150271191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdfe4004d40 0x7fdfe40051c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0ce9d0 tx=0x7fdfe4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b99ba4f04ccfea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.982+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4001bf0 con 0x7fdfe4004d40 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.982+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff4003b70 con 0x7fdfe4004d40 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.982+0000 7fdffa7fc640 1 -- 172.21.15.44:0/4150271191 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff40050d0 con 0x7fdfe4004d40 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.982+0000 7fe001b60640 1 -- 172.21.15.44:0/4150271191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdfe4004d40 msgr2=0x7fdfe40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.982+0000 7fe001b60640 1 --2- 172.21.15.44:0/4150271191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdfe4004d40 0x7fdfe40051c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0ce9d0 tx=0x7fdfe4005830 comp rx=0 tx=0).stop 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.982+0000 7fdffbfff640 1 -- 172.21.15.44:0/4150271191 reap_dead start 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.982+0000 7fe001b60640 1 -- 172.21.15.44:0/4150271191 shutdown_connections 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.982+0000 7fe001b60640 1 -- 172.21.15.44:0/4150271191 >> 172.21.15.44:0/4150271191 conn(0x7fdffc09d3c0 msgr2=0x7fdffc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:42.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.983+0000 7fe001b60640 1 -- 172.21.15.44:0/4150271191 shutdown_connections 2024-04-02T13:21:42.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.983+0000 7fe001b60640 1 -- 172.21.15.44:0/4150271191 wait complete. 2024-04-02T13:21:42.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.984+0000 7fe001b60640 1 Processor -- start 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.984+0000 7fe001b60640 1 -- start start 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.984+0000 7fe001b60640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdffc154130 0x7fdffc0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.984+0000 7fe001b60640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdffc1548b0 0x7fdffc0cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fe001b60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc0cd230 0x7fdffc15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fe001b60640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc0cfca0 con 0x7fdffc154130 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fe001b60640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc0d3070 con 0x7fdffc1548b0 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fe001b60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdffc0d0ad0 con 0x7fdffc0cd230 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdffb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdffc154130 0x7fdffc0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdffb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdffc154130 0x7fdffc0cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38288/0 (socket says 172.21.15.44:38288) 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdffb7fe640 1 -- 172.21.15.44:0/2444350623 learned_addr learned my addr 172.21.15.44:0/2444350623 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdffaffd640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdffc1548b0 0x7fdffc0cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdffbfff640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc0cd230 0x7fdffc15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdffb7fe640 1 -- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc0cd230 msgr2=0x7fdffc15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:42.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdffb7fe640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc0cd230 0x7fdffc15ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdffb7fe640 1 -- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdffc1548b0 msgr2=0x7fdffc0cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdffb7fe640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdffc1548b0 0x7fdffc0cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdffb7fe640 1 -- 172.21.15.44:0/2444350623 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdffc160190 con 0x7fdffc154130 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdffb7fe640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdffc154130 0x7fdffc0cb1b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdfec0017e0 tx=0x7fdfec0076e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdfe3fff640 1 -- 172.21.15.44:0/2444350623 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec00fa20 con 0x7fdffc154130 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdfe3fff640 1 -- 172.21.15.44:0/2444350623 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfec018070 con 0x7fdffc154130 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fe001b60640 1 -- 172.21.15.44:0/2444350623 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdffc160470 con 0x7fdffc154130 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fdfe3fff640 1 -- 172.21.15.44:0/2444350623 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec00fdc0 con 0x7fdffc154130 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.985+0000 7fe001b60640 1 -- 172.21.15.44:0/2444350623 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdffc0c7e40 con 0x7fdffc154130 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.986+0000 7fe001b60640 1 -- 172.21.15.44:0/2444350623 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdffc0684f0 con 0x7fdffc154130 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.987+0000 7fdfe3fff640 1 -- 172.21.15.44:0/2444350623 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdfec01d460 con 0x7fdffc154130 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.987+0000 7fdfe3fff640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdfe806c8f0 0x7fdfe806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.987+0000 7fdfe3fff640 1 -- 172.21.15.44:0/2444350623 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6001+0+0 (secure 0 0 0) 0x7fdfec08be30 con 0x7fdffc154130 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.987+0000 7fdffaffd640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdfe806c8f0 0x7fdfe806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:42.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.988+0000 7fdffaffd640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdfe806c8f0 0x7fdfe806edb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdffc1614c0 tx=0x7fdff0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:42.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:42.992+0000 7fdfe3fff640 1 -- 172.21.15.44:0/2444350623 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fdfec05a1c0 con 0x7fdffc154130 2024-04-02T13:21:43.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.213+0000 7fe001b60640 1 -- 172.21.15.44:0/2444350623 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fdffc0c8010 con 0x7fdffc154130 2024-04-02T13:21:43.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.872+0000 7fdfe3fff640 1 -- 172.21.15.44:0/2444350623 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) ==== 172+0+0 (secure 0 0 0) 0x7fdfec059ac0 con 0x7fdffc154130 2024-04-02T13:21:43.876 INFO:teuthology.orchestra.run.smithi044.stderr:new fs with metadata pool 5 and data pool 6 2024-04-02T13:21:43.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.880+0000 7fe001b60640 1 -- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdfe806c8f0 msgr2=0x7fdfe806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:43.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.880+0000 7fe001b60640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdfe806c8f0 0x7fdfe806edb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdffc1614c0 tx=0x7fdff0009190 comp rx=0 tx=0).stop 2024-04-02T13:21:43.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.881+0000 7fe001b60640 1 -- 172.21.15.44:0/2444350623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdffc154130 msgr2=0x7fdffc0cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:43.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.881+0000 7fe001b60640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdffc154130 0x7fdffc0cb1b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdfec0017e0 tx=0x7fdfec0076e0 comp rx=0 tx=0).stop 2024-04-02T13:21:43.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.882+0000 7fe001b60640 1 -- 172.21.15.44:0/2444350623 shutdown_connections 2024-04-02T13:21:43.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.882+0000 7fe001b60640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdfe806c8f0 0x7fdfe806edb0 secure :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdffc1614c0 tx=0x7fdff0009190 comp rx=0 tx=0).stop 2024-04-02T13:21:43.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.882+0000 7fe001b60640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdffc0cd230 0x7fdffc15ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:43.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.882+0000 7fe001b60640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdffc1548b0 0x7fdffc0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:43.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.882+0000 7fe001b60640 1 --2- 172.21.15.44:0/2444350623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdffc154130 0x7fdffc0cb1b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:43.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.882+0000 7fe001b60640 1 -- 172.21.15.44:0/2444350623 >> 172.21.15.44:0/2444350623 conn(0x7fdffc09d3c0 msgr2=0x7fdffc0b8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:43.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.885+0000 7fe001b60640 1 -- 172.21.15.44:0/2444350623 shutdown_connections 2024-04-02T13:21:43.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:43.885+0000 7fe001b60640 1 -- 172.21.15.44:0/2444350623 wait complete. 2024-04-02T13:21:43.908 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:44.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.011+0000 7fca2bc2c640 1 Processor -- start 2024-04-02T13:21:44.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.011+0000 7fca2bc2c640 1 -- start start 2024-04-02T13:21:44.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca2bc2c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fca2415e4a0 0x7fca240aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca2bc2c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fca240d0c50 0x7fca240d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca2bc2c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca2406ce40 0x7fca2406d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca2bc2c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fca240cc600 con 0x7fca240d0c50 2024-04-02T13:21:44.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca2bc2c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fca240cc8e0 con 0x7fca2415e4a0 2024-04-02T13:21:44.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca2bc2c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fca240ccbc0 con 0x7fca2406ce40 2024-04-02T13:21:44.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca291c7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fca240d0c50 0x7fca240d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca291c7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fca240d0c50 0x7fca240d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38304/0 (socket says 172.21.15.44:38304) 2024-04-02T13:21:44.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca291c7640 1 -- 172.21.15.44:0/2370220290 learned_addr learned my addr 172.21.15.44:0/2370220290 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:44.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca291c7640 1 -- 172.21.15.44:0/2370220290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca2406ce40 msgr2=0x7fca2406d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:44.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca2a1c9640 1 --2- 172.21.15.44:0/2370220290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca2406ce40 0x7fca2406d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca299c8640 1 --2- 172.21.15.44:0/2370220290 >> v2:172.21.15.69:3300/0 conn(0x7fca2415e4a0 0x7fca240aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca291c7640 1 --2- 172.21.15.44:0/2370220290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca2406ce40 0x7fca2406d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca291c7640 1 -- 172.21.15.44:0/2370220290 >> v2:172.21.15.69:3300/0 conn(0x7fca2415e4a0 msgr2=0x7fca240aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:44.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca291c7640 1 --2- 172.21.15.44:0/2370220290 >> v2:172.21.15.69:3300/0 conn(0x7fca2415e4a0 0x7fca240aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca291c7640 1 -- 172.21.15.44:0/2370220290 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca240ccea0 con 0x7fca240d0c50 2024-04-02T13:21:44.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.012+0000 7fca2a1c9640 1 --2- 172.21.15.44:0/2370220290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca2406ce40 0x7fca2406d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:44.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.013+0000 7fca291c7640 1 --2- 172.21.15.44:0/2370220290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca240d0c50 0x7fca240d10b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fca240d1f80 tx=0x7fca1402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230dc79caf216ffe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:44.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.013+0000 7fca289c6640 1 -- 172.21.15.44:0/2370220290 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca14035020 con 0x7fca240d0c50 2024-04-02T13:21:44.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.013+0000 7fca289c6640 1 -- 172.21.15.44:0/2370220290 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca14031870 con 0x7fca240d0c50 2024-04-02T13:21:44.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.013+0000 7fca289c6640 1 -- 172.21.15.44:0/2370220290 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca14031c40 con 0x7fca240d0c50 2024-04-02T13:21:44.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.014+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2370220290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca240d0c50 msgr2=0x7fca240d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.014+0000 7fca2bc2c640 1 --2- 172.21.15.44:0/2370220290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca240d0c50 0x7fca240d10b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fca240d1f80 tx=0x7fca1402b910 comp rx=0 tx=0).stop 2024-04-02T13:21:44.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.014+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2370220290 shutdown_connections 2024-04-02T13:21:44.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.014+0000 7fca2bc2c640 1 --2- 172.21.15.44:0/2370220290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca2406ce40 0x7fca2406d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.014+0000 7fca2bc2c640 1 --2- 172.21.15.44:0/2370220290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca240d0c50 0x7fca240d10b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.014+0000 7fca2bc2c640 1 --2- 172.21.15.44:0/2370220290 >> v2:172.21.15.69:3300/0 conn(0x7fca2415e4a0 0x7fca240aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.014+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2370220290 >> 172.21.15.44:0/2370220290 conn(0x7fca2409aa20 msgr2=0x7fca2409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:44.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.014+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2370220290 shutdown_connections 2024-04-02T13:21:44.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.014+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2370220290 wait complete. 2024-04-02T13:21:44.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.015+0000 7fca2bc2c640 1 Processor -- start 2024-04-02T13:21:44.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.015+0000 7fca2bc2c640 1 -- start start 2024-04-02T13:21:44.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2bc2c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2406ce40 0x7fca241f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2bc2c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca240d0c50 0x7fca240c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2bc2c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2415e4a0 0x7fca240c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2bc2c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fca2415e880 con 0x7fca2415e4a0 2024-04-02T13:21:44.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2bc2c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fca240cd460 con 0x7fca2406ce40 2024-04-02T13:21:44.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2bc2c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fca240af130 con 0x7fca240d0c50 2024-04-02T13:21:44.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2a1c9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2415e4a0 0x7fca240c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2a1c9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2415e4a0 0x7fca240c79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38308/0 (socket says 172.21.15.44:38308) 2024-04-02T13:21:44.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2a1c9640 1 -- 172.21.15.44:0/2835066224 learned_addr learned my addr 172.21.15.44:0/2835066224 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:44.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca291c7640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca240d0c50 0x7fca240c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca299c8640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2406ce40 0x7fca241f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2a1c9640 1 -- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca240d0c50 msgr2=0x7fca240c74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2a1c9640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca240d0c50 0x7fca240c74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2a1c9640 1 -- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2406ce40 msgr2=0x7fca241f3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2a1c9640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2406ce40 0x7fca241f3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2a1c9640 1 -- 172.21.15.44:0/2835066224 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca240c8000 con 0x7fca2415e4a0 2024-04-02T13:21:44.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2a1c9640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2415e4a0 0x7fca240c79f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00cf20 tx=0x7fca1c00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:44.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca1a7fc640 1 -- 172.21.15.44:0/2835066224 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c01a070 con 0x7fca2415e4a0 2024-04-02T13:21:44.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca1a7fc640 1 -- 172.21.15.44:0/2835066224 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca1c00d840 con 0x7fca2415e4a0 2024-04-02T13:21:44.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.016+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2835066224 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca240c82e0 con 0x7fca2415e4a0 2024-04-02T13:21:44.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.017+0000 7fca1a7fc640 1 -- 172.21.15.44:0/2835066224 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c00dc30 con 0x7fca2415e4a0 2024-04-02T13:21:44.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.017+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2835066224 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca240c88a0 con 0x7fca2415e4a0 2024-04-02T13:21:44.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.017+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2835066224 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca240684f0 con 0x7fca2415e4a0 2024-04-02T13:21:44.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.018+0000 7fca1a7fc640 1 -- 172.21.15.44:0/2835066224 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fca1c01d070 con 0x7fca2415e4a0 2024-04-02T13:21:44.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.018+0000 7fca1a7fc640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca0806c8f0 0x7fca0806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.018+0000 7fca1a7fc640 1 -- 172.21.15.44:0/2835066224 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fca1c08c4f0 con 0x7fca2415e4a0 2024-04-02T13:21:44.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.018+0000 7fca299c8640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca0806c8f0 0x7fca0806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.019+0000 7fca299c8640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca0806c8f0 0x7fca0806edb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fca2000a9a0 tx=0x7fca20008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:44.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.023+0000 7fca1a7fc640 1 -- 172.21.15.44:0/2835066224 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fca1c05a140 con 0x7fca2415e4a0 2024-04-02T13:21:44.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.208+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2835066224 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca24067e50 con 0x7fca2415e4a0 2024-04-02T13:21:44.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.209+0000 7fca1a7fc640 1 -- 172.21.15.44:0/2835066224 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16907 (secure 0 0 0) 0x7fca1c05e440 con 0x7fca2415e4a0 2024-04-02T13:21:44.212 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:44.212 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":38,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:43.215471+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:21:41.647497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-02T13:21:42.168299+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:44.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.211+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca0806c8f0 msgr2=0x7fca0806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.211+0000 7fca2bc2c640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca0806c8f0 0x7fca0806edb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fca2000a9a0 tx=0x7fca20008040 comp rx=0 tx=0).stop 2024-04-02T13:21:44.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.211+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2835066224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2415e4a0 msgr2=0x7fca240c79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.211+0000 7fca2bc2c640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2415e4a0 0x7fca240c79f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00cf20 tx=0x7fca1c00a7d0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.211+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2835066224 shutdown_connections 2024-04-02T13:21:44.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.211+0000 7fca2bc2c640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca0806c8f0 0x7fca0806edb0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.211+0000 7fca2bc2c640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca2415e4a0 0x7fca240c79f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.211+0000 7fca2bc2c640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca240d0c50 0x7fca240c74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.211+0000 7fca2bc2c640 1 --2- 172.21.15.44:0/2835066224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca2406ce40 0x7fca241f3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.211+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2835066224 >> 172.21.15.44:0/2835066224 conn(0x7fca2409aa20 msgr2=0x7fca2406d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:44.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.211+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2835066224 shutdown_connections 2024-04-02T13:21:44.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.211+0000 7fca2bc2c640 1 -- 172.21.15.44:0/2835066224 wait complete. 2024-04-02T13:21:44.225 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:44.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.324+0000 7f86d5c4a640 1 Processor -- start 2024-04-02T13:21:44.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.324+0000 7f86d5c4a640 1 -- start start 2024-04-02T13:21:44.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86d5c4a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d00b2d00 0x7f86d00cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86d5c4a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f86d00ce110 0x7f86d0071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86d5c4a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86d00b3160 con 0x7f86d015aa00 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86d5c4a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f86d015ade0 con 0x7f86d00ce110 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86d5c4a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f86d00af1b0 con 0x7f86d00b2d00 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86cffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f86d00ce110 0x7f86d0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86cf7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f86d015aa00 0x7f86d0072bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42418/0 (socket says 172.21.15.44:42418) 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86cf7fe640 1 -- 172.21.15.44:0/1613543244 learned_addr learned my addr 172.21.15.44:0/1613543244 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86ceffd640 1 --2- 172.21.15.44:0/1613543244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d00b2d00 0x7f86d00cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86cffff640 1 -- 172.21.15.44:0/1613543244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d00b2d00 msgr2=0x7f86d00cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86cffff640 1 --2- 172.21.15.44:0/1613543244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d00b2d00 0x7f86d00cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86cffff640 1 -- 172.21.15.44:0/1613543244 >> v1:172.21.15.44:6789/0 conn(0x7f86d015aa00 legacy=0x7f86d0072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86ce7fc640 1 -- 172.21.15.44:0/1613543244 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2468507838 0 0) 0x7f86d00b3160 con 0x7f86d015aa00 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.325+0000 7f86cffff640 1 -- 172.21.15.44:0/1613543244 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d00af490 con 0x7f86d00ce110 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.326+0000 7f86cffff640 1 --2- 172.21.15.44:0/1613543244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86d00ce110 0x7f86d0071190 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009d90 tx=0x7f86c802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd73bec97e191244 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.326+0000 7f86ce7fc640 1 -- 172.21.15.44:0/1613543244 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c8039070 con 0x7f86d00ce110 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.326+0000 7f86ce7fc640 1 -- 172.21.15.44:0/1613543244 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86c802cbc0 con 0x7f86d00ce110 2024-04-02T13:21:44.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.326+0000 7f86ce7fc640 1 -- 172.21.15.44:0/1613543244 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c8034440 con 0x7f86d00ce110 2024-04-02T13:21:44.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.327+0000 7f86d5c4a640 1 -- 172.21.15.44:0/1613543244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86d00ce110 msgr2=0x7f86d0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.327+0000 7f86d5c4a640 1 --2- 172.21.15.44:0/1613543244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86d00ce110 0x7f86d0071190 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009d90 tx=0x7f86c802be10 comp rx=0 tx=0).stop 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.327+0000 7f86d5c4a640 1 -- 172.21.15.44:0/1613543244 shutdown_connections 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.327+0000 7f86d5c4a640 1 --2- 172.21.15.44:0/1613543244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86d00ce110 0x7f86d0071190 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.327+0000 7f86d5c4a640 1 --2- 172.21.15.44:0/1613543244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d00b2d00 0x7f86d00cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.327+0000 7f86d5c4a640 1 -- 172.21.15.44:0/1613543244 >> 172.21.15.44:0/1613543244 conn(0x7f86d00cfcd0 msgr2=0x7f86d00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.327+0000 7f86d5c4a640 1 -- 172.21.15.44:0/1613543244 shutdown_connections 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.327+0000 7f86d5c4a640 1 -- 172.21.15.44:0/1613543244 wait complete. 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.328+0000 7f86d5c4a640 1 Processor -- start 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.328+0000 7f86d5c4a640 1 -- start start 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86d5c4a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86d00b2d00 0x7f86d01596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86d5c4a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86d00ce110 0x7f86d01f4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86d5c4a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d015aa00 0x7f86d006e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86d5c4a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f86d00ce5a0 con 0x7f86d00b2d00 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86d5c4a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f86d00726d0 con 0x7f86d00ce110 2024-04-02T13:21:44.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86d5c4a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f86d00cc750 con 0x7f86d015aa00 2024-04-02T13:21:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86cf7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86d00b2d00 0x7f86d01596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86cf7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86d00b2d00 0x7f86d01596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38324/0 (socket says 172.21.15.44:38324) 2024-04-02T13:21:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86cf7fe640 1 -- 172.21.15.44:0/3058358782 learned_addr learned my addr 172.21.15.44:0/3058358782 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86ceffd640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86d00ce110 0x7f86d01f4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86cf7fe640 1 -- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d015aa00 msgr2=0x7f86d006e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86cffff640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d015aa00 0x7f86d006e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86cf7fe640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d015aa00 0x7f86d006e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86cf7fe640 1 -- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86d00ce110 msgr2=0x7f86d01f4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86cf7fe640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86d00ce110 0x7f86d01f4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86cf7fe640 1 -- 172.21.15.44:0/3058358782 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d015a020 con 0x7f86d00b2d00 2024-04-02T13:21:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86ceffd640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86d00ce110 0x7f86d01f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86cffff640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d015aa00 0x7f86d006e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:44.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86cf7fe640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86d00b2d00 0x7f86d01596b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f86c000c500 tx=0x7f86c000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:44.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.329+0000 7f86b7fff640 1 -- 172.21.15.44:0/3058358782 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c0017070 con 0x7f86d00b2d00 2024-04-02T13:21:44.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.330+0000 7f86b7fff640 1 -- 172.21.15.44:0/3058358782 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86c0006520 con 0x7f86d00b2d00 2024-04-02T13:21:44.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.330+0000 7f86d5c4a640 1 -- 172.21.15.44:0/3058358782 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86d00c9220 con 0x7f86d00b2d00 2024-04-02T13:21:44.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.330+0000 7f86b7fff640 1 -- 172.21.15.44:0/3058358782 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c0004a00 con 0x7f86d00b2d00 2024-04-02T13:21:44.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.330+0000 7f86d5c4a640 1 -- 172.21.15.44:0/3058358782 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86d00c97e0 con 0x7f86d00b2d00 2024-04-02T13:21:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.331+0000 7f86d5c4a640 1 -- 172.21.15.44:0/3058358782 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86d0074f20 con 0x7f86d00b2d00 2024-04-02T13:21:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.331+0000 7f86b7fff640 1 -- 172.21.15.44:0/3058358782 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f86c001a070 con 0x7f86d00b2d00 2024-04-02T13:21:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.331+0000 7f86b7fff640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f86bc06c8f0 0x7f86bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.332+0000 7f86b7fff640 1 -- 172.21.15.44:0/3058358782 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f86c008b0a0 con 0x7f86d00b2d00 2024-04-02T13:21:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.332+0000 7f86ceffd640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f86bc06c8f0 0x7f86bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.332+0000 7f86ceffd640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f86bc06c8f0 0x7f86bc06edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f86d00b4ee0 tx=0x7f86c4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:44.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.336+0000 7f86b7fff640 1 -- 172.21.15.44:0/3058358782 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f86c00593d0 con 0x7f86d00b2d00 2024-04-02T13:21:44.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.521+0000 7f86d5c4a640 1 -- 172.21.15.44:0/3058358782 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86d00b3950 con 0x7f86d00b2d00 2024-04-02T13:21:44.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.522+0000 7f86b7fff640 1 -- 172.21.15.44:0/3058358782 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16907 (secure 0 0 0) 0x7f86d00b3950 con 0x7f86d00b2d00 2024-04-02T13:21:44.525 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:44.525 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":38,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:43.215471+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:21:41.647497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-02T13:21:42.168299+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:44.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.524+0000 7f86d5c4a640 1 -- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f86bc06c8f0 msgr2=0x7f86bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.524+0000 7f86d5c4a640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f86bc06c8f0 0x7f86bc06edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f86d00b4ee0 tx=0x7f86c4008040 comp rx=0 tx=0).stop 2024-04-02T13:21:44.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.524+0000 7f86d5c4a640 1 -- 172.21.15.44:0/3058358782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86d00b2d00 msgr2=0x7f86d01596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.524+0000 7f86d5c4a640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86d00b2d00 0x7f86d01596b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f86c000c500 tx=0x7f86c000c9c0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.524+0000 7f86d5c4a640 1 -- 172.21.15.44:0/3058358782 shutdown_connections 2024-04-02T13:21:44.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.524+0000 7f86d5c4a640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f86bc06c8f0 0x7f86bc06edb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.524+0000 7f86d5c4a640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d015aa00 0x7f86d006e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.524+0000 7f86d5c4a640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86d00ce110 0x7f86d01f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.524+0000 7f86d5c4a640 1 --2- 172.21.15.44:0/3058358782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86d00b2d00 0x7f86d01596b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.524+0000 7f86d5c4a640 1 -- 172.21.15.44:0/3058358782 >> 172.21.15.44:0/3058358782 conn(0x7f86d00cfcd0 msgr2=0x7f86d00b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:44.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.524+0000 7f86d5c4a640 1 -- 172.21.15.44:0/3058358782 shutdown_connections 2024-04-02T13:21:44.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.524+0000 7f86d5c4a640 1 -- 172.21.15.44:0/3058358782 wait complete. 2024-04-02T13:21:44.538 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T13:21:44.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.640+0000 7f48e14c5640 1 Processor -- start 2024-04-02T13:21:44.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.640+0000 7f48e14c5640 1 -- start start 2024-04-02T13:21:44.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.641+0000 7f48e14c5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48dc161780 0x7f48dc06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.641+0000 7f48e14c5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48dc0cc6c0 con 0x7f48dc0cf5b0 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.641+0000 7f48e14c5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48dc0cc9a0 con 0x7f48dc0ceed0 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.641+0000 7f48e14c5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f48dc0ccc80 con 0x7f48dc161780 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.641+0000 7f48da7fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f48dc0cf5b0 0x7f48dc0ce7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32962/0 (socket says 172.21.15.44:32962) 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.641+0000 7f48da7fc640 1 -- 172.21.15.44:0/270199339 learned_addr learned my addr 172.21.15.44:0/270199339 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.641+0000 7f48daffd640 1 --2- 172.21.15.44:0/270199339 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48dc161780 0x7f48dc06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.641+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2756031189 0 0) 0x7f48dc0cc6c0 con 0x7f48dc0cf5b0 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.641+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48c8003520 con 0x7f48dc0cf5b0 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3895893919 0 0) 0x7f48c8003520 con 0x7f48dc0cf5b0 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48dc161780 msgr2=0x7f48dc06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48d9ffb640 1 --2- 172.21.15.44:0/270199339 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48dc161780 0x7f48dc06cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 >> v1:172.21.15.69:6789/0 conn(0x7f48dc0ceed0 legacy=0x7f48dc0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48dc0ccf60 con 0x7f48dc0cf5b0 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f48c4001670 con 0x7f48dc0cf5b0 2024-04-02T13:21:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 >> v1:172.21.15.44:6789/0 conn(0x7f48dc0cf5b0 legacy=0x7f48dc0ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48d9ffb640 1 --2- 172.21.15.44:0/270199339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48c80039e0 0x7f48c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48d9ffb640 1 --2- 172.21.15.44:0/270199339 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48c8004390 0x7f48c8004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48d9ffb640 1 --2- 172.21.15.44:0/270199339 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c8005100 0x7f48c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc0ccc80 con 0x7f48c80039e0 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f48c800bfe0 con 0x7f48c8004390 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f48c800c1e0 con 0x7f48c8005100 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48daffd640 1 --2- 172.21.15.44:0/270199339 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48dc161780 0x7f48dc06cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48da7fc640 1 --2- 172.21.15.44:0/270199339 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48c8004390 0x7f48c8004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48daffd640 1 --2- 172.21.15.44:0/270199339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48c80039e0 0x7f48c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48db7fe640 1 --2- 172.21.15.44:0/270199339 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c8005100 0x7f48c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48daffd640 1 -- 172.21.15.44:0/270199339 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c8005100 msgr2=0x7f48c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48daffd640 1 --2- 172.21.15.44:0/270199339 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c8005100 0x7f48c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48daffd640 1 -- 172.21.15.44:0/270199339 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48c8004390 msgr2=0x7f48c8004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:44.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48daffd640 1 --2- 172.21.15.44:0/270199339 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48c8004390 0x7f48c8004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.642+0000 7f48daffd640 1 -- 172.21.15.44:0/270199339 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c8003520 con 0x7f48c80039e0 2024-04-02T13:21:44.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.643+0000 7f48daffd640 1 --2- 172.21.15.44:0/270199339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48c80039e0 0x7f48c8003dc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0b4aa0 tx=0x7f48c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bea1c0dde08aaed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:44.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.643+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d40097d0 con 0x7f48c80039e0 2024-04-02T13:21:44.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.643+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48d4004030 con 0x7f48c80039e0 2024-04-02T13:21:44.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.643+0000 7f48d9ffb640 1 -- 172.21.15.44:0/270199339 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4004400 con 0x7f48c80039e0 2024-04-02T13:21:44.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.644+0000 7f48e14c5640 1 -- 172.21.15.44:0/270199339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48c80039e0 msgr2=0x7f48c8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.644+0000 7f48e14c5640 1 --2- 172.21.15.44:0/270199339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48c80039e0 0x7f48c8003dc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0b4aa0 tx=0x7f48c8005830 comp rx=0 tx=0).stop 2024-04-02T13:21:44.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.644+0000 7f48db7fe640 1 -- 172.21.15.44:0/270199339 reap_dead start 2024-04-02T13:21:44.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.644+0000 7f48e14c5640 1 -- 172.21.15.44:0/270199339 shutdown_connections 2024-04-02T13:21:44.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.644+0000 7f48e14c5640 1 -- 172.21.15.44:0/270199339 >> 172.21.15.44:0/270199339 conn(0x7f48dc073850 msgr2=0x7f48dc09aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:44.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.644+0000 7f48e14c5640 1 -- 172.21.15.44:0/270199339 shutdown_connections 2024-04-02T13:21:44.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.644+0000 7f48e14c5640 1 -- 172.21.15.44:0/270199339 wait complete. 2024-04-02T13:21:44.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.645+0000 7f48e14c5640 1 Processor -- start 2024-04-02T13:21:44.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.645+0000 7f48e14c5640 1 -- start start 2024-04-02T13:21:44.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48e14c5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48c80039e0 0x7f48dc1542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48e14c5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48dc0ca0d0 0x7f48dc1f4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48e14c5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48dc1f48b0 0x7f48dc15f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48e14c5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc0d0180 con 0x7f48c80039e0 2024-04-02T13:21:44.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48e14c5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc161b60 con 0x7f48dc1f48b0 2024-04-02T13:21:44.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48e14c5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f48dc0d0300 con 0x7f48dc0ca0d0 2024-04-02T13:21:44.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48daffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48c80039e0 0x7f48dc1542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48daffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48c80039e0 0x7f48dc1542b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53808/0 (socket says 172.21.15.44:53808) 2024-04-02T13:21:44.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48daffd640 1 -- 172.21.15.44:0/1817575805 learned_addr learned my addr 172.21.15.44:0/1817575805 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:44.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48db7fe640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48dc1f48b0 0x7f48dc15f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48da7fc640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48dc0ca0d0 0x7f48dc1f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48daffd640 1 -- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48dc0ca0d0 msgr2=0x7f48dc1f4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48daffd640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48dc0ca0d0 0x7f48dc1f4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48daffd640 1 -- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48dc1f48b0 msgr2=0x7f48dc15f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:44.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48daffd640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48dc1f48b0 0x7f48dc15f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:44.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48daffd640 1 -- 172.21.15.44:0/1817575805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48dc0cac40 con 0x7f48c80039e0 2024-04-02T13:21:44.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.646+0000 7f48db7fe640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48dc1f48b0 0x7f48dc15f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:44.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.647+0000 7f48daffd640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48c80039e0 0x7f48dc1542b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f48d4036040 tx=0x7f48d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:44.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.647+0000 7f48d37fe640 1 -- 172.21.15.44:0/1817575805 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4042070 con 0x7f48c80039e0 2024-04-02T13:21:44.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.647+0000 7f48d37fe640 1 -- 172.21.15.44:0/1817575805 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48d402c910 con 0x7f48c80039e0 2024-04-02T13:21:44.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.647+0000 7f48e14c5640 1 -- 172.21.15.44:0/1817575805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48dc15d8f0 con 0x7f48c80039e0 2024-04-02T13:21:44.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.647+0000 7f48d37fe640 1 -- 172.21.15.44:0/1817575805 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d402ccc0 con 0x7f48c80039e0 2024-04-02T13:21:44.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.647+0000 7f48e14c5640 1 -- 172.21.15.44:0/1817575805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48dc0b7ba0 con 0x7f48c80039e0 2024-04-02T13:21:44.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.648+0000 7f48e14c5640 1 -- 172.21.15.44:0/1817575805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48dc160ff0 con 0x7f48c80039e0 2024-04-02T13:21:44.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.648+0000 7f48d37fe640 1 -- 172.21.15.44:0/1817575805 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f48d4045070 con 0x7f48c80039e0 2024-04-02T13:21:44.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.648+0000 7f48d37fe640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48bc06c8f0 0x7f48bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:44.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.649+0000 7f48d37fe640 1 -- 172.21.15.44:0/1817575805 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f48d40adba0 con 0x7f48c80039e0 2024-04-02T13:21:44.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.649+0000 7f48da7fc640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48bc06c8f0 0x7f48bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:44.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.649+0000 7f48da7fc640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48bc06c8f0 0x7f48bc06edb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f48dc070ee0 tx=0x7f48c4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:44.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.653+0000 7f48d37fe640 1 -- 172.21.15.44:0/1817575805 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f48dc160ff0 con 0x7f48c80039e0 2024-04-02T13:21:44.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:44.882+0000 7f48e14c5640 1 -- 172.21.15.44:0/1817575805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f48dc068c00 con 0x7f48c80039e0 2024-04-02T13:21:45.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:45.882+0000 7f48d37fe640 1 -- 172.21.15.44:0/1817575805 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f48d407bff0 con 0x7f48c80039e0 2024-04-02T13:21:45.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:45.885+0000 7f48e14c5640 1 -- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48bc06c8f0 msgr2=0x7f48bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:45.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:45.885+0000 7f48e14c5640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48bc06c8f0 0x7f48bc06edb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f48dc070ee0 tx=0x7f48c4009040 comp rx=0 tx=0).stop 2024-04-02T13:21:45.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:45.885+0000 7f48e14c5640 1 -- 172.21.15.44:0/1817575805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48c80039e0 msgr2=0x7f48dc1542b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:45.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:45.885+0000 7f48e14c5640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48c80039e0 0x7f48dc1542b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f48d4036040 tx=0x7f48d4004730 comp rx=0 tx=0).stop 2024-04-02T13:21:45.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:45.885+0000 7f48e14c5640 1 -- 172.21.15.44:0/1817575805 shutdown_connections 2024-04-02T13:21:45.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:45.885+0000 7f48e14c5640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48bc06c8f0 0x7f48bc06edb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:45.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:45.885+0000 7f48e14c5640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48dc1f48b0 0x7f48dc15f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:45.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:45.885+0000 7f48e14c5640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48dc0ca0d0 0x7f48dc1f4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:45.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:45.885+0000 7f48e14c5640 1 --2- 172.21.15.44:0/1817575805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48c80039e0 0x7f48dc1542b0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:45.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:45.885+0000 7f48e14c5640 1 -- 172.21.15.44:0/1817575805 >> 172.21.15.44:0/1817575805 conn(0x7f48dc073850 msgr2=0x7f48dc06ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:45.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:45.885+0000 7f48e14c5640 1 -- 172.21.15.44:0/1817575805 shutdown_connections 2024-04-02T13:21:45.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:45.885+0000 7f48e14c5640 1 -- 172.21.15.44:0/1817575805 wait complete. 2024-04-02T13:21:45.901 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T13:21:45.901 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T13:21:46.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.009+0000 7fce82bcd640 1 Processor -- start 2024-04-02T13:21:46.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.009+0000 7fce82bcd640 1 -- start start 2024-04-02T13:21:46.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.010+0000 7fce82bcd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fce7c15e4a0 0x7fce7c070680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:46.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.010+0000 7fce82bcd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fce7c0cbb60 0x7fce7c161ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:46.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.010+0000 7fce82bcd640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce7c15e880 con 0x7fce7c0d03b0 2024-04-02T13:21:46.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.010+0000 7fce82bcd640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fce7c15f8a0 con 0x7fce7c0cbb60 2024-04-02T13:21:46.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.010+0000 7fce82bcd640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fce7c0cebf0 con 0x7fce7c15e4a0 2024-04-02T13:21:46.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.010+0000 7fce8116a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fce7c0cbb60 0x7fce7c161ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:46.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.010+0000 7fce8116a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fce7c0cbb60 0x7fce7c161ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53810/0 (socket says 172.21.15.44:53810) 2024-04-02T13:21:46.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.010+0000 7fce8116a640 1 -- 172.21.15.44:0/1021259922 learned_addr learned my addr 172.21.15.44:0/1021259922 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:46.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.010+0000 7fce8116a640 1 -- 172.21.15.44:0/1021259922 >> v2:172.21.15.69:3300/0 conn(0x7fce7c15e4a0 msgr2=0x7fce7c070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:46.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.010+0000 7fce80969640 1 --2- 172.21.15.44:0/1021259922 >> v2:172.21.15.69:3300/0 conn(0x7fce7c15e4a0 0x7fce7c070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:46.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.010+0000 7fce8116a640 1 --2- 172.21.15.44:0/1021259922 >> v2:172.21.15.69:3300/0 conn(0x7fce7c15e4a0 0x7fce7c070680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:46.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.010+0000 7fce8116a640 1 -- 172.21.15.44:0/1021259922 >> v1:172.21.15.69:6789/0 conn(0x7fce7c0d03b0 legacy=0x7fce7c0cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:46.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.011+0000 7fce8116a640 1 -- 172.21.15.44:0/1021259922 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7c15eb00 con 0x7fce7c0cbb60 2024-04-02T13:21:46.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.011+0000 7fce8116a640 1 --2- 172.21.15.44:0/1021259922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce7c0cbb60 0x7fce7c161ea0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fce7c071dd0 tx=0x7fce6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58783c0b197c5d86 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:46.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.011+0000 7fce7b7fe640 1 -- 172.21.15.44:0/1021259922 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c035020 con 0x7fce7c0cbb60 2024-04-02T13:21:46.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.011+0000 7fce7b7fe640 1 -- 172.21.15.44:0/1021259922 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce6c031870 con 0x7fce7c0cbb60 2024-04-02T13:21:46.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.011+0000 7fce7b7fe640 1 -- 172.21.15.44:0/1021259922 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c031c40 con 0x7fce7c0cbb60 2024-04-02T13:21:46.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.012+0000 7fce82bcd640 1 -- 172.21.15.44:0/1021259922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce7c0cbb60 msgr2=0x7fce7c161ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:46.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.012+0000 7fce82bcd640 1 --2- 172.21.15.44:0/1021259922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce7c0cbb60 0x7fce7c161ea0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fce7c071dd0 tx=0x7fce6c02b910 comp rx=0 tx=0).stop 2024-04-02T13:21:46.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.012+0000 7fce82bcd640 1 -- 172.21.15.44:0/1021259922 shutdown_connections 2024-04-02T13:21:46.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.012+0000 7fce82bcd640 1 --2- 172.21.15.44:0/1021259922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce7c0cbb60 0x7fce7c161ea0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:46.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.012+0000 7fce82bcd640 1 --2- 172.21.15.44:0/1021259922 >> v2:172.21.15.69:3300/0 conn(0x7fce7c15e4a0 0x7fce7c070680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:46.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.012+0000 7fce82bcd640 1 -- 172.21.15.44:0/1021259922 >> 172.21.15.44:0/1021259922 conn(0x7fce7c0d0e00 msgr2=0x7fce7c0d1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:46.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.012+0000 7fce82bcd640 1 -- 172.21.15.44:0/1021259922 shutdown_connections 2024-04-02T13:21:46.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.012+0000 7fce82bcd640 1 -- 172.21.15.44:0/1021259922 wait complete. 2024-04-02T13:21:46.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.013+0000 7fce82bcd640 1 Processor -- start 2024-04-02T13:21:46.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.013+0000 7fce82bcd640 1 -- start start 2024-04-02T13:21:46.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.014+0000 7fce82bcd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce7c0cbb60 0x7fce7c152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:46.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.014+0000 7fce82bcd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce7c0d03b0 0x7fce7c0b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:46.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.014+0000 7fce82bcd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce7c15e4a0 0x7fce7c0b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:46.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.014+0000 7fce82bcd640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fce7c15f3a0 con 0x7fce7c0cbb60 2024-04-02T13:21:46.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.014+0000 7fce82bcd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fce7c15f0c0 con 0x7fce7c0d03b0 2024-04-02T13:21:46.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.014+0000 7fce82bcd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fce7c0cd5b0 con 0x7fce7c15e4a0 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.014+0000 7fce80969640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce7c0cbb60 0x7fce7c152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.014+0000 7fce80969640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce7c0cbb60 0x7fce7c152560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53820/0 (socket says 172.21.15.44:53820) 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.014+0000 7fce80969640 1 -- 172.21.15.44:0/554422189 learned_addr learned my addr 172.21.15.44:0/554422189 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.014+0000 7fce7bfff640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce7c0d03b0 0x7fce7c0b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.014+0000 7fce8116a640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce7c15e4a0 0x7fce7c0b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.015+0000 7fce80969640 1 -- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce7c15e4a0 msgr2=0x7fce7c0b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.015+0000 7fce80969640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce7c15e4a0 0x7fce7c0b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.015+0000 7fce80969640 1 -- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce7c0d03b0 msgr2=0x7fce7c0b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.015+0000 7fce80969640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce7c0d03b0 0x7fce7c0b3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.015+0000 7fce80969640 1 -- 172.21.15.44:0/554422189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7c152aa0 con 0x7fce7c0cbb60 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.015+0000 7fce80969640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce7c0cbb60 0x7fce7c152560 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fce7400cd20 tx=0x7fce7400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.015+0000 7fce797fa640 1 -- 172.21.15.44:0/554422189 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce74003990 con 0x7fce7c0cbb60 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.015+0000 7fce797fa640 1 -- 172.21.15.44:0/554422189 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce74003b30 con 0x7fce7c0cbb60 2024-04-02T13:21:46.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.015+0000 7fce82bcd640 1 -- 172.21.15.44:0/554422189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce7c0c6bf0 con 0x7fce7c0cbb60 2024-04-02T13:21:46.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.015+0000 7fce797fa640 1 -- 172.21.15.44:0/554422189 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce74004020 con 0x7fce7c0cbb60 2024-04-02T13:21:46.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.015+0000 7fce82bcd640 1 -- 172.21.15.44:0/554422189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce7c0c71b0 con 0x7fce7c0cbb60 2024-04-02T13:21:46.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.016+0000 7fce82bcd640 1 -- 172.21.15.44:0/554422189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce7c0684f0 con 0x7fce7c0cbb60 2024-04-02T13:21:46.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.016+0000 7fce797fa640 1 -- 172.21.15.44:0/554422189 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fce740041c0 con 0x7fce7c0cbb60 2024-04-02T13:21:46.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.017+0000 7fce797fa640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce6806c920 0x7fce6806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:46.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.017+0000 7fce797fa640 1 -- 172.21.15.44:0/554422189 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fce7408c140 con 0x7fce7c0cbb60 2024-04-02T13:21:46.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.017+0000 7fce7bfff640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce6806c920 0x7fce6806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:46.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.017+0000 7fce7bfff640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce6806c920 0x7fce6806ede0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fce7c1561c0 tx=0x7fce70007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:46.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.021+0000 7fce797fa640 1 -- 172.21.15.44:0/554422189 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fce74014030 con 0x7fce7c0cbb60 2024-04-02T13:21:46.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.246+0000 7fce82bcd640 1 -- 172.21.15.44:0/554422189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fce7c0aff30 con 0x7fce7c0cbb60 2024-04-02T13:21:46.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.883+0000 7fce797fa640 1 -- 172.21.15.44:0/554422189 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7fce74059d90 con 0x7fce7c0cbb60 2024-04-02T13:21:46.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.886+0000 7fce82bcd640 1 -- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce6806c920 msgr2=0x7fce6806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:46.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.886+0000 7fce82bcd640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce6806c920 0x7fce6806ede0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fce7c1561c0 tx=0x7fce70007e30 comp rx=0 tx=0).stop 2024-04-02T13:21:46.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.886+0000 7fce82bcd640 1 -- 172.21.15.44:0/554422189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce7c0cbb60 msgr2=0x7fce7c152560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:46.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.886+0000 7fce82bcd640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce7c0cbb60 0x7fce7c152560 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fce7400cd20 tx=0x7fce7400a5c0 comp rx=0 tx=0).stop 2024-04-02T13:21:46.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.886+0000 7fce82bcd640 1 -- 172.21.15.44:0/554422189 shutdown_connections 2024-04-02T13:21:46.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.886+0000 7fce82bcd640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce6806c920 0x7fce6806ede0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:46.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.886+0000 7fce82bcd640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce7c15e4a0 0x7fce7c0b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:46.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.886+0000 7fce82bcd640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce7c0d03b0 0x7fce7c0b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:46.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.886+0000 7fce82bcd640 1 --2- 172.21.15.44:0/554422189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce7c0cbb60 0x7fce7c152560 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:46.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.886+0000 7fce82bcd640 1 -- 172.21.15.44:0/554422189 >> 172.21.15.44:0/554422189 conn(0x7fce7c0d0e00 msgr2=0x7fce7c0add60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:46.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.886+0000 7fce82bcd640 1 -- 172.21.15.44:0/554422189 shutdown_connections 2024-04-02T13:21:46.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:46.886+0000 7fce82bcd640 1 -- 172.21.15.44:0/554422189 wait complete. 2024-04-02T13:21:46.902 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T13:21:46.903 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:21:47.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.008+0000 7fd879b8f640 1 Processor -- start 2024-04-02T13:21:47.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.008+0000 7fd879b8f640 1 -- start start 2024-04-02T13:21:47.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.009+0000 7fd879b8f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd8740b2d00 0x7fd8740cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.009+0000 7fd879b8f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8740b3160 con 0x7fd87415aa00 2024-04-02T13:21:47.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.009+0000 7fd879b8f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd87415ade0 con 0x7fd8740ce110 2024-04-02T13:21:47.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.009+0000 7fd879b8f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd8740af1b0 con 0x7fd8740b2d00 2024-04-02T13:21:47.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.009+0000 7fd872ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd8740b2d00 0x7fd8740cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.009+0000 7fd8737fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd87415aa00 0x7fd874072bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32976/0 (socket says 172.21.15.44:32976) 2024-04-02T13:21:47.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.009+0000 7fd8737fe640 1 -- 172.21.15.44:0/1676646113 learned_addr learned my addr 172.21.15.44:0/1676646113 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:47.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.009+0000 7fd872ffd640 1 -- 172.21.15.44:0/1676646113 >> v1:172.21.15.69:6789/0 conn(0x7fd8740ce110 legacy=0x7fd874071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:47.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.009+0000 7fd872ffd640 1 -- 172.21.15.44:0/1676646113 >> v1:172.21.15.44:6789/0 conn(0x7fd87415aa00 legacy=0x7fd874072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.009+0000 7fd872ffd640 1 -- 172.21.15.44:0/1676646113 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8740af490 con 0x7fd8740b2d00 2024-04-02T13:21:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.010+0000 7fd872ffd640 1 --2- 172.21.15.44:0/1676646113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8740b2d00 0x7fd8740cdbd0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd86400a940 tx=0x7fd86402cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a73a8ff94ea579c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.010+0000 7fd8727fc640 1 -- 172.21.15.44:0/1676646113 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86403a070 con 0x7fd8740b2d00 2024-04-02T13:21:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.010+0000 7fd8727fc640 1 -- 172.21.15.44:0/1676646113 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd86402dab0 con 0x7fd8740b2d00 2024-04-02T13:21:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.010+0000 7fd8727fc640 1 -- 172.21.15.44:0/1676646113 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86400b040 con 0x7fd8740b2d00 2024-04-02T13:21:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.011+0000 7fd879b8f640 1 -- 172.21.15.44:0/1676646113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8740b2d00 msgr2=0x7fd8740cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.011+0000 7fd879b8f640 1 --2- 172.21.15.44:0/1676646113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8740b2d00 0x7fd8740cdbd0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd86400a940 tx=0x7fd86402cd00 comp rx=0 tx=0).stop 2024-04-02T13:21:47.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.011+0000 7fd879b8f640 1 -- 172.21.15.44:0/1676646113 shutdown_connections 2024-04-02T13:21:47.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.011+0000 7fd879b8f640 1 --2- 172.21.15.44:0/1676646113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8740b2d00 0x7fd8740cdbd0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.011+0000 7fd879b8f640 1 -- 172.21.15.44:0/1676646113 >> 172.21.15.44:0/1676646113 conn(0x7fd8740cfcd0 msgr2=0x7fd8740d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:47.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.011+0000 7fd879b8f640 1 -- 172.21.15.44:0/1676646113 shutdown_connections 2024-04-02T13:21:47.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.011+0000 7fd879b8f640 1 -- 172.21.15.44:0/1676646113 wait complete. 2024-04-02T13:21:47.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.012+0000 7fd879b8f640 1 Processor -- start 2024-04-02T13:21:47.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.012+0000 7fd879b8f640 1 -- start start 2024-04-02T13:21:47.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd879b8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8740b2d00 0x7fd8741435f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd879b8f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8740ce110 0x7fd874144a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd879b8f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd87415aa00 0x7fd87414fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd879b8f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8740ce560 con 0x7fd8740b2d00 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd879b8f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8740726d0 con 0x7fd87415aa00 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd879b8f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd8740cc750 con 0x7fd8740ce110 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd8737fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8740b2d00 0x7fd8741435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd8737fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8740b2d00 0x7fd8741435f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53844/0 (socket says 172.21.15.44:53844) 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd8737fe640 1 -- 172.21.15.44:0/233552910 learned_addr learned my addr 172.21.15.44:0/233552910 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd872ffd640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8740ce110 0x7fd874144a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd873fff640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd87415aa00 0x7fd87414fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd8737fe640 1 -- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8740ce110 msgr2=0x7fd874144a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd8737fe640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8740ce110 0x7fd874144a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd8737fe640 1 -- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd87415aa00 msgr2=0x7fd87414fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd8737fe640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd87415aa00 0x7fd87414fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd8737fe640 1 -- 172.21.15.44:0/233552910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87414bed0 con 0x7fd8740b2d00 2024-04-02T13:21:47.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.013+0000 7fd8737fe640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8740b2d00 0x7fd8741435f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd86800c3c0 tx=0x7fd86800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:47.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.014+0000 7fd85bfff640 1 -- 172.21.15.44:0/233552910 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd868017070 con 0x7fd8740b2d00 2024-04-02T13:21:47.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.014+0000 7fd85bfff640 1 -- 172.21.15.44:0/233552910 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd868005ce0 con 0x7fd8740b2d00 2024-04-02T13:21:47.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.014+0000 7fd879b8f640 1 -- 172.21.15.44:0/233552910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd87414ae30 con 0x7fd8740b2d00 2024-04-02T13:21:47.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.014+0000 7fd85bfff640 1 -- 172.21.15.44:0/233552910 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8680072f0 con 0x7fd8740b2d00 2024-04-02T13:21:47.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.014+0000 7fd879b8f640 1 -- 172.21.15.44:0/233552910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd87414b3f0 con 0x7fd8740b2d00 2024-04-02T13:21:47.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.015+0000 7fd879b8f640 1 -- 172.21.15.44:0/233552910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd874074f20 con 0x7fd8740b2d00 2024-04-02T13:21:47.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.015+0000 7fd85bfff640 1 -- 172.21.15.44:0/233552910 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd86801a070 con 0x7fd8740b2d00 2024-04-02T13:21:47.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.015+0000 7fd85bfff640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd86006c8f0 0x7fd86006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.016+0000 7fd85bfff640 1 -- 172.21.15.44:0/233552910 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fd86808b080 con 0x7fd8740b2d00 2024-04-02T13:21:47.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.016+0000 7fd872ffd640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd86006c8f0 0x7fd86006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.016+0000 7fd872ffd640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd86006c8f0 0x7fd86006edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd864037000 tx=0x7fd86400c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:47.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.020+0000 7fd85bfff640 1 -- 172.21.15.44:0/233552910 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd868059320 con 0x7fd8740b2d00 2024-04-02T13:21:47.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.243+0000 7fd879b8f640 1 -- 172.21.15.44:0/233552910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8740684f0 con 0x7fd8740b2d00 2024-04-02T13:21:47.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.244+0000 7fd85bfff640 1 -- 172.21.15.44:0/233552910 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5181 (secure 0 0 0) 0x7fd868058c20 con 0x7fd8740b2d00 2024-04-02T13:21:47.246 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:47.246 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3050514527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3050514527},{"type":"v1","addr":"172.21.15.69:6836","nonce":3050514527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4973,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1949024012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1949024012},{"type":"v1","addr":"172.21.15.69:6837","nonce":1949024012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/2817873198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2817873198},{"type":"v1","addr":"172.21.15.44:6834","nonce":2817873198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:21:43.215796+0000","modified":"2024-04-02T13:21:46.248294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/1969465285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":1969465285},{"type":"v1","addr":"172.21.15.44:6835","nonce":1969465285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":2}]} 2024-04-02T13:21:47.247 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 18 2024-04-02T13:21:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.245+0000 7fd879b8f640 1 -- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd86006c8f0 msgr2=0x7fd86006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.245+0000 7fd879b8f640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd86006c8f0 0x7fd86006edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd864037000 tx=0x7fd86400c040 comp rx=0 tx=0).stop 2024-04-02T13:21:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.245+0000 7fd879b8f640 1 -- 172.21.15.44:0/233552910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8740b2d00 msgr2=0x7fd8741435f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.245+0000 7fd879b8f640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8740b2d00 0x7fd8741435f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd86800c3c0 tx=0x7fd86800c880 comp rx=0 tx=0).stop 2024-04-02T13:21:47.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.246+0000 7fd879b8f640 1 -- 172.21.15.44:0/233552910 shutdown_connections 2024-04-02T13:21:47.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.246+0000 7fd879b8f640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd86006c8f0 0x7fd86006edb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.246+0000 7fd879b8f640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd87415aa00 0x7fd87414fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.246+0000 7fd879b8f640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8740ce110 0x7fd874144a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.246+0000 7fd879b8f640 1 --2- 172.21.15.44:0/233552910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8740b2d00 0x7fd8741435f0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.246+0000 7fd879b8f640 1 -- 172.21.15.44:0/233552910 >> 172.21.15.44:0/233552910 conn(0x7fd8740cfcd0 msgr2=0x7fd87414d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:47.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.246+0000 7fd879b8f640 1 -- 172.21.15.44:0/233552910 shutdown_connections 2024-04-02T13:21:47.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.246+0000 7fd879b8f640 1 -- 172.21.15.44:0/233552910 wait complete. 2024-04-02T13:21:47.259 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:47.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.358+0000 7fb35ca55640 1 Processor -- start 2024-04-02T13:21:47.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.358+0000 7fb35ca55640 1 -- start start 2024-04-02T13:21:47.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.359+0000 7fb35ca55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb35815c1c0 0x7fb3580affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.359+0000 7fb35ca55640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb3580b0c10 0x7fb358165e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.359+0000 7fb35ca55640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3580b1640 con 0x7fb3580b3d70 2024-04-02T13:21:47.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.359+0000 7fb35ca55640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb3580aef20 con 0x7fb3580b0c10 2024-04-02T13:21:47.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.359+0000 7fb35ca55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb3580af200 con 0x7fb35815c1c0 2024-04-02T13:21:47.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.359+0000 7fb355d9b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb3580b3d70 0x7fb3580b0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32992/0 (socket says 172.21.15.44:32992) 2024-04-02T13:21:47.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.359+0000 7fb355d9b640 1 -- 172.21.15.44:0/699312417 learned_addr learned my addr 172.21.15.44:0/699312417 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:47.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.359+0000 7fb356d9d640 1 --2- 172.21.15.44:0/699312417 >> v2:172.21.15.69:3300/0 conn(0x7fb3580b0c10 0x7fb358165e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.359+0000 7fb35659c640 1 --2- 172.21.15.44:0/699312417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb35815c1c0 0x7fb3580affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3730604741 0 0) 0x7fb340001110 con 0x7fb3580b3d70 2024-04-02T13:21:47.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb344003520 con 0x7fb3580b3d70 2024-04-02T13:21:47.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 811091554 0 0) 0x7fb344003520 con 0x7fb3580b3d70 2024-04-02T13:21:47.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb35815c1c0 msgr2=0x7fb3580affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:47.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 --2- 172.21.15.44:0/699312417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb35815c1c0 0x7fb3580affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 >> v2:172.21.15.69:3300/0 conn(0x7fb3580b0c10 msgr2=0x7fb358165e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:47.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 --2- 172.21.15.44:0/699312417 >> v2:172.21.15.69:3300/0 conn(0x7fb3580b0c10 0x7fb358165e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3580af4e0 con 0x7fb3580b3d70 2024-04-02T13:21:47.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fb340001840 con 0x7fb3580b3d70 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 >> v1:172.21.15.44:6789/0 conn(0x7fb3580b3d70 legacy=0x7fb3580b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 --2- 172.21.15.44:0/699312417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb35815c1c0 0x7fb3440039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 --2- 172.21.15.44:0/699312417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb344003f30 0x7fb344004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 --2- 172.21.15.44:0/699312417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb344004d40 0x7fb3440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb3580aef20 con 0x7fb344003f30 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb3580af200 con 0x7fb344004d40 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb34400b9c0 con 0x7fb35815c1c0 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb355d9b640 1 --2- 172.21.15.44:0/699312417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb344003f30 0x7fb344004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb35659c640 1 --2- 172.21.15.44:0/699312417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb35815c1c0 0x7fb3440039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb356d9d640 1 --2- 172.21.15.44:0/699312417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb344004d40 0x7fb3440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb355d9b640 1 -- 172.21.15.44:0/699312417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb35815c1c0 msgr2=0x7fb3440039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb355d9b640 1 --2- 172.21.15.44:0/699312417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb35815c1c0 0x7fb3440039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb355d9b640 1 -- 172.21.15.44:0/699312417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb344004d40 msgr2=0x7fb3440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:47.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb355d9b640 1 --2- 172.21.15.44:0/699312417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb344004d40 0x7fb3440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.360+0000 7fb355d9b640 1 -- 172.21.15.44:0/699312417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb344003520 con 0x7fb344003f30 2024-04-02T13:21:47.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.361+0000 7fb355d9b640 1 --2- 172.21.15.44:0/699312417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb344003f30 0x7fb344004800 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb34000c3b0 tx=0x7fb344005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ee939bb682ec0d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:47.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.361+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34000c0e0 con 0x7fb344003f30 2024-04-02T13:21:47.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.361+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb340030070 con 0x7fb344003f30 2024-04-02T13:21:47.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.361+0000 7fb35559a640 1 -- 172.21.15.44:0/699312417 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3400058d0 con 0x7fb344003f30 2024-04-02T13:21:47.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.362+0000 7fb35ca55640 1 -- 172.21.15.44:0/699312417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb344003f30 msgr2=0x7fb344004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.362+0000 7fb35ca55640 1 --2- 172.21.15.44:0/699312417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb344003f30 0x7fb344004800 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb34000c3b0 tx=0x7fb344005830 comp rx=0 tx=0).stop 2024-04-02T13:21:47.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.362+0000 7fb356d9d640 1 -- 172.21.15.44:0/699312417 reap_dead start 2024-04-02T13:21:47.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.362+0000 7fb35ca55640 1 -- 172.21.15.44:0/699312417 shutdown_connections 2024-04-02T13:21:47.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.362+0000 7fb35ca55640 1 -- 172.21.15.44:0/699312417 >> 172.21.15.44:0/699312417 conn(0x7fb35809aa20 msgr2=0x7fb35809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:47.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.362+0000 7fb35ca55640 1 -- 172.21.15.44:0/699312417 shutdown_connections 2024-04-02T13:21:47.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.362+0000 7fb35ca55640 1 -- 172.21.15.44:0/699312417 wait complete. 2024-04-02T13:21:47.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.363+0000 7fb35ca55640 1 Processor -- start 2024-04-02T13:21:47.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.363+0000 7fb35ca55640 1 -- start start 2024-04-02T13:21:47.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb35ca55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb344003f30 0x7fb3581f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb35ca55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb35806c700 0x7fb3580b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb35ca55640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3580c6600 0x7fb3580c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb35ca55640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb3580b1080 con 0x7fb3580c6600 2024-04-02T13:21:47.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb35ca55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb3580afaa0 con 0x7fb344003f30 2024-04-02T13:21:47.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb35ca55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb3580cc990 con 0x7fb35806c700 2024-04-02T13:21:47.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb356d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3580c6600 0x7fb3580c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb356d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3580c6600 0x7fb3580c6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53870/0 (socket says 172.21.15.44:53870) 2024-04-02T13:21:47.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb356d9d640 1 -- 172.21.15.44:0/1874828774 learned_addr learned my addr 172.21.15.44:0/1874828774 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:47.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb355d9b640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb35806c700 0x7fb3580b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb35659c640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb344003f30 0x7fb3581f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb356d9d640 1 -- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb35806c700 msgr2=0x7fb3580b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb356d9d640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb35806c700 0x7fb3580b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb356d9d640 1 -- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb344003f30 msgr2=0x7fb3581f74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb356d9d640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb344003f30 0x7fb3581f74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.364+0000 7fb356d9d640 1 -- 172.21.15.44:0/1874828774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35806cf00 con 0x7fb3580c6600 2024-04-02T13:21:47.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.365+0000 7fb356d9d640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3580c6600 0x7fb3580c6a60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb3580b8060 tx=0x7fb34800f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:47.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.365+0000 7fb34effd640 1 -- 172.21.15.44:0/1874828774 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34800fd10 con 0x7fb3580c6600 2024-04-02T13:21:47.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.365+0000 7fb34effd640 1 -- 172.21.15.44:0/1874828774 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb348015430 con 0x7fb3580c6600 2024-04-02T13:21:47.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.365+0000 7fb35ca55640 1 -- 172.21.15.44:0/1874828774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3581f7a80 con 0x7fb3580c6600 2024-04-02T13:21:47.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.365+0000 7fb34effd640 1 -- 172.21.15.44:0/1874828774 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348015820 con 0x7fb3580c6600 2024-04-02T13:21:47.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.365+0000 7fb35ca55640 1 -- 172.21.15.44:0/1874828774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3581f8040 con 0x7fb3580c6600 2024-04-02T13:21:47.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.366+0000 7fb35ca55640 1 -- 172.21.15.44:0/1874828774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3580684f0 con 0x7fb3580c6600 2024-04-02T13:21:47.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.366+0000 7fb34effd640 1 -- 172.21.15.44:0/1874828774 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb3480159c0 con 0x7fb3580c6600 2024-04-02T13:21:47.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.367+0000 7fb34effd640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb33806c8f0 0x7fb33806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.367+0000 7fb34effd640 1 -- 172.21.15.44:0/1874828774 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fb34808c960 con 0x7fb3580c6600 2024-04-02T13:21:47.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.367+0000 7fb35659c640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb33806c8f0 0x7fb33806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.367+0000 7fb35659c640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb33806c8f0 0x7fb33806edb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb350009ae0 tx=0x7fb350009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:47.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.371+0000 7fb34effd640 1 -- 172.21.15.44:0/1874828774 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb348019030 con 0x7fb3580c6600 2024-04-02T13:21:47.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.557+0000 7fb35ca55640 1 -- 172.21.15.44:0/1874828774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb35815b600 con 0x7fb3580c6600 2024-04-02T13:21:47.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.558+0000 7fb34effd640 1 -- 172.21.15.44:0/1874828774 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16894 (secure 0 0 0) 0x7fb348010040 con 0x7fb3580c6600 2024-04-02T13:21:47.561 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:47.561 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":39,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:44.874189+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:21:41.647497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-02T13:21:42.168299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:47.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.560+0000 7fb35ca55640 1 -- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb33806c8f0 msgr2=0x7fb33806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.560+0000 7fb35ca55640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb33806c8f0 0x7fb33806edb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb350009ae0 tx=0x7fb350009190 comp rx=0 tx=0).stop 2024-04-02T13:21:47.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.560+0000 7fb35ca55640 1 -- 172.21.15.44:0/1874828774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3580c6600 msgr2=0x7fb3580c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.560+0000 7fb35ca55640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3580c6600 0x7fb3580c6a60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb3580b8060 tx=0x7fb34800f1b0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.560+0000 7fb35ca55640 1 -- 172.21.15.44:0/1874828774 shutdown_connections 2024-04-02T13:21:47.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.560+0000 7fb35ca55640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb33806c8f0 0x7fb33806edb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.560+0000 7fb35ca55640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3580c6600 0x7fb3580c6a60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.560+0000 7fb35ca55640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb35806c700 0x7fb3580b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.560+0000 7fb35ca55640 1 --2- 172.21.15.44:0/1874828774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb344003f30 0x7fb3581f74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.560+0000 7fb35ca55640 1 -- 172.21.15.44:0/1874828774 >> 172.21.15.44:0/1874828774 conn(0x7fb35809aa20 msgr2=0x7fb3580cdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:47.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.560+0000 7fb35ca55640 1 -- 172.21.15.44:0/1874828774 shutdown_connections 2024-04-02T13:21:47.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.560+0000 7fb35ca55640 1 -- 172.21.15.44:0/1874828774 wait complete. 2024-04-02T13:21:47.574 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T13:21:47.574 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:21:47.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.673+0000 7fd8a232b640 1 Processor -- start 2024-04-02T13:21:47.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.673+0000 7fd8a232b640 1 -- start start 2024-04-02T13:21:47.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.674+0000 7fd8a232b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd89c0b3d70 0x7fd89c0b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.674+0000 7fd8a232b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd89c0b0c10 0x7fd89c165e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.674+0000 7fd8a232b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89c0b1640 con 0x7fd89c15c1c0 2024-04-02T13:21:47.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.674+0000 7fd8a232b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd89c0aef20 con 0x7fd89c0b3d70 2024-04-02T13:21:47.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.674+0000 7fd8a232b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd89c0af200 con 0x7fd89c0b0c10 2024-04-02T13:21:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.674+0000 7fd89b7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd89c0b3d70 0x7fd89c0b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.674+0000 7fd89b7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd89c0b3d70 0x7fd89c0b06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53880/0 (socket says 172.21.15.44:53880) 2024-04-02T13:21:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.674+0000 7fd89b7fe640 1 -- 172.21.15.44:0/1674744942 learned_addr learned my addr 172.21.15.44:0/1674744942 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.674+0000 7fd8a08c8640 1 --2- 172.21.15.44:0/1674744942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd89c0b0c10 0x7fd89c165e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.674+0000 7fd89b7fe640 1 -- 172.21.15.44:0/1674744942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd89c0b0c10 msgr2=0x7fd89c165e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.674+0000 7fd89b7fe640 1 --2- 172.21.15.44:0/1674744942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd89c0b0c10 0x7fd89c165e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.674+0000 7fd89b7fe640 1 -- 172.21.15.44:0/1674744942 >> v1:172.21.15.69:6789/0 conn(0x7fd89c15c1c0 legacy=0x7fd89c0affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.674+0000 7fd89b7fe640 1 -- 172.21.15.44:0/1674744942 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89c0af4e0 con 0x7fd89c0b3d70 2024-04-02T13:21:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.675+0000 7fd89b7fe640 1 --2- 172.21.15.44:0/1674744942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd89c0b3d70 0x7fd89c0b06d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd88c009870 tx=0x7fd88c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d979b99027294d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.675+0000 7fd89affd640 1 -- 172.21.15.44:0/1674744942 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88c034070 con 0x7fd89c0b3d70 2024-04-02T13:21:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.675+0000 7fd89affd640 1 -- 172.21.15.44:0/1674744942 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd88c030840 con 0x7fd89c0b3d70 2024-04-02T13:21:47.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.675+0000 7fd89affd640 1 -- 172.21.15.44:0/1674744942 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88c030c10 con 0x7fd89c0b3d70 2024-04-02T13:21:47.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.675+0000 7fd8a232b640 1 -- 172.21.15.44:0/1674744942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd89c0b3d70 msgr2=0x7fd89c0b06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.675+0000 7fd8a232b640 1 --2- 172.21.15.44:0/1674744942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd89c0b3d70 0x7fd89c0b06d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd88c009870 tx=0x7fd88c02b980 comp rx=0 tx=0).stop 2024-04-02T13:21:47.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.676+0000 7fd8a232b640 1 -- 172.21.15.44:0/1674744942 shutdown_connections 2024-04-02T13:21:47.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.676+0000 7fd8a232b640 1 --2- 172.21.15.44:0/1674744942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd89c0b3d70 0x7fd89c0b06d0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.676+0000 7fd8a232b640 1 --2- 172.21.15.44:0/1674744942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd89c0b0c10 0x7fd89c165e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.676+0000 7fd8a232b640 1 -- 172.21.15.44:0/1674744942 >> 172.21.15.44:0/1674744942 conn(0x7fd89c09aa20 msgr2=0x7fd89c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:47.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.676+0000 7fd8a232b640 1 -- 172.21.15.44:0/1674744942 shutdown_connections 2024-04-02T13:21:47.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.676+0000 7fd8a232b640 1 -- 172.21.15.44:0/1674744942 wait complete. 2024-04-02T13:21:47.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.677+0000 7fd8a232b640 1 Processor -- start 2024-04-02T13:21:47.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.677+0000 7fd8a232b640 1 -- start start 2024-04-02T13:21:47.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd8a232b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd89c0b0c10 0x7fd89c150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd8a232b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd89c0b3d70 0x7fd89c0c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd8a232b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd89c15c1c0 0x7fd89c0cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd8a232b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd89c0cc990 con 0x7fd89c0b3d70 2024-04-02T13:21:47.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd8a232b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd89c0afaa0 con 0x7fd89c0b0c10 2024-04-02T13:21:47.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd8a232b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd89c0aebf0 con 0x7fd89c15c1c0 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd89b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd89c0b3d70 0x7fd89c0c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd89b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd89c0b3d70 0x7fd89c0c9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53896/0 (socket says 172.21.15.44:53896) 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd89b7fe640 1 -- 172.21.15.44:0/1586575765 learned_addr learned my addr 172.21.15.44:0/1586575765 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd8a08c8640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd89c15c1c0 0x7fd89c0cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd89bfff640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd89c0b0c10 0x7fd89c150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd89b7fe640 1 -- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd89c15c1c0 msgr2=0x7fd89c0cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd89b7fe640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd89c15c1c0 0x7fd89c0cab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd89b7fe640 1 -- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd89c0b0c10 msgr2=0x7fd89c150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd89b7fe640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd89c0b0c10 0x7fd89c150980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd89b7fe640 1 -- 172.21.15.44:0/1586575765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89c0c9b40 con 0x7fd89c0b3d70 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd89b7fe640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd89c0b3d70 0x7fd89c0c9080 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd88c000c00 tx=0x7fd88c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd898ff9640 1 -- 172.21.15.44:0/1586575765 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88c048070 con 0x7fd89c0b3d70 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd898ff9640 1 -- 172.21.15.44:0/1586575765 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd88c030640 con 0x7fd89c0b3d70 2024-04-02T13:21:47.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd8a232b640 1 -- 172.21.15.44:0/1586575765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd89c0c9e20 con 0x7fd89c0b3d70 2024-04-02T13:21:47.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd898ff9640 1 -- 172.21.15.44:0/1586575765 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88c034040 con 0x7fd89c0b3d70 2024-04-02T13:21:47.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.678+0000 7fd8a232b640 1 -- 172.21.15.44:0/1586575765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd89c0c5bd0 con 0x7fd89c0b3d70 2024-04-02T13:21:47.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.679+0000 7fd8a232b640 1 -- 172.21.15.44:0/1586575765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd89c0684f0 con 0x7fd89c0b3d70 2024-04-02T13:21:47.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.680+0000 7fd898ff9640 1 -- 172.21.15.44:0/1586575765 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd88c04b070 con 0x7fd89c0b3d70 2024-04-02T13:21:47.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.680+0000 7fd898ff9640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd87c06c8f0 0x7fd87c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:47.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.680+0000 7fd898ff9640 1 -- 172.21.15.44:0/1586575765 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fd88c0adb00 con 0x7fd89c0b3d70 2024-04-02T13:21:47.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.680+0000 7fd89bfff640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd87c06c8f0 0x7fd87c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:47.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.681+0000 7fd89bfff640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd87c06c8f0 0x7fd87c06edb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd884000f60 tx=0x7fd884009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:47.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.685+0000 7fd898ff9640 1 -- 172.21.15.44:0/1586575765 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd88c030810 con 0x7fd89c0b3d70 2024-04-02T13:21:47.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.866+0000 7fd8a232b640 1 -- 172.21.15.44:0/1586575765 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd89c0cec60 con 0x7fd87c06c8f0 2024-04-02T13:21:47.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.873+0000 7fd898ff9640 1 -- 172.21.15.44:0/1586575765 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251882 (secure 0 0 0) 0x7fd89c0cec60 con 0x7fd87c06c8f0 2024-04-02T13:21:47.878 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:47.879 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:21:47.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.876+0000 7fd8a232b640 1 -- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd87c06c8f0 msgr2=0x7fd87c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.876+0000 7fd8a232b640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd87c06c8f0 0x7fd87c06edb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd884000f60 tx=0x7fd884009040 comp rx=0 tx=0).stop 2024-04-02T13:21:47.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.876+0000 7fd8a232b640 1 -- 172.21.15.44:0/1586575765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd89c0b3d70 msgr2=0x7fd89c0c9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:47.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.876+0000 7fd8a232b640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd89c0b3d70 0x7fd89c0c9080 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd88c000c00 tx=0x7fd88c004730 comp rx=0 tx=0).stop 2024-04-02T13:21:47.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.876+0000 7fd8a232b640 1 -- 172.21.15.44:0/1586575765 shutdown_connections 2024-04-02T13:21:47.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.876+0000 7fd8a232b640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd87c06c8f0 0x7fd87c06edb0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.876+0000 7fd8a232b640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd89c15c1c0 0x7fd89c0cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.876+0000 7fd8a232b640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd89c0b3d70 0x7fd89c0c9080 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.876+0000 7fd8a232b640 1 --2- 172.21.15.44:0/1586575765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd89c0b0c10 0x7fd89c150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:47.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.876+0000 7fd8a232b640 1 -- 172.21.15.44:0/1586575765 >> 172.21.15.44:0/1586575765 conn(0x7fd89c09aa20 msgr2=0x7fd89c09b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:47.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.876+0000 7fd8a232b640 1 -- 172.21.15.44:0/1586575765 shutdown_connections 2024-04-02T13:21:47.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:47.876+0000 7fd8a232b640 1 -- 172.21.15.44:0/1586575765 wait complete. 2024-04-02T13:21:47.889 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-04-02T13:21:46.893405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591838,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":202,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220664,"kb_used_data":4500,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754056,"statfs":{"total":773094113280,"available":772868153344,"internally_reserved":0,"allocated":4608000,"data_stored":2635919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1470,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.045243"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887540+0000","last_change":"2024-04-02T13:21:43.883508+0000","last_active":"2024-04-02T13:21:43.887540+0000","last_peered":"2024-04-02T13:21:43.887540+0000","last_clean":"2024-04-02T13:21:43.887540+0000","last_became_active":"2024-04-02T13:21:43.883200+0000","last_became_peered":"2024-04-02T13:21:43.883200+0000","last_unstale":"2024-04-02T13:21:43.887540+0000","last_undegraded":"2024-04-02T13:21:43.887540+0000","last_fullsized":"2024-04-02T13:21:43.887540+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:22.467118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887693+0000","last_change":"2024-04-02T13:21:43.884004+0000","last_active":"2024-04-02T13:21:43.887693+0000","last_peered":"2024-04-02T13:21:43.887693+0000","last_clean":"2024-04-02T13:21:43.887693+0000","last_became_active":"2024-04-02T13:21:43.883760+0000","last_became_peered":"2024-04-02T13:21:43.883760+0000","last_unstale":"2024-04-02T13:21:43.887693+0000","last_undegraded":"2024-04-02T13:21:43.887693+0000","last_fullsized":"2024-04-02T13:21:43.887693+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:33.147235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897360+0000","last_change":"2024-04-02T13:21:43.889721+0000","last_active":"2024-04-02T13:21:43.897360+0000","last_peered":"2024-04-02T13:21:43.897360+0000","last_clean":"2024-04-02T13:21:43.897360+0000","last_became_active":"2024-04-02T13:21:43.889139+0000","last_became_peered":"2024-04-02T13:21:43.889139+0000","last_unstale":"2024-04-02T13:21:43.897360+0000","last_undegraded":"2024-04-02T13:21:43.897360+0000","last_fullsized":"2024-04-02T13:21:43.897360+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:42:32.884966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.896070+0000","last_change":"2024-04-02T13:21:43.891625+0000","last_active":"2024-04-02T13:21:43.896070+0000","last_peered":"2024-04-02T13:21:43.896070+0000","last_clean":"2024-04-02T13:21:43.896070+0000","last_became_active":"2024-04-02T13:21:43.890952+0000","last_became_peered":"2024-04-02T13:21:43.890952+0000","last_unstale":"2024-04-02T13:21:43.896070+0000","last_undegraded":"2024-04-02T13:21:43.896070+0000","last_fullsized":"2024-04-02T13:21:43.896070+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:26.646577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.893389+0000","last_change":"2024-04-02T13:21:43.890367+0000","last_active":"2024-04-02T13:21:43.893389+0000","last_peered":"2024-04-02T13:21:43.893389+0000","last_clean":"2024-04-02T13:21:43.893389+0000","last_became_active":"2024-04-02T13:21:43.890087+0000","last_became_peered":"2024-04-02T13:21:43.890087+0000","last_unstale":"2024-04-02T13:21:43.893389+0000","last_undegraded":"2024-04-02T13:21:43.893389+0000","last_fullsized":"2024-04-02T13:21:43.893389+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:20.667849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.893989+0000","last_change":"2024-04-02T13:21:43.888470+0000","last_active":"2024-04-02T13:21:43.893989+0000","last_peered":"2024-04-02T13:21:43.893989+0000","last_clean":"2024-04-02T13:21:43.893989+0000","last_became_active":"2024-04-02T13:21:43.888141+0000","last_became_peered":"2024-04-02T13:21:43.888141+0000","last_unstale":"2024-04-02T13:21:43.893989+0000","last_undegraded":"2024-04-02T13:21:43.893989+0000","last_fullsized":"2024-04-02T13:21:43.893989+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:10.725145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888968+0000","last_change":"2024-04-02T13:21:43.888055+0000","last_active":"2024-04-02T13:21:43.888968+0000","last_peered":"2024-04-02T13:21:43.888968+0000","last_clean":"2024-04-02T13:21:43.888968+0000","last_became_active":"2024-04-02T13:21:43.887783+0000","last_became_peered":"2024-04-02T13:21:43.887783+0000","last_unstale":"2024-04-02T13:21:43.888968+0000","last_undegraded":"2024-04-02T13:21:43.888968+0000","last_fullsized":"2024-04-02T13:21:43.888968+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:33.147235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886428+0000","last_change":"2024-04-02T13:21:43.877806+0000","last_active":"2024-04-02T13:21:43.886428+0000","last_peered":"2024-04-02T13:21:43.886428+0000","last_clean":"2024-04-02T13:21:43.886428+0000","last_became_active":"2024-04-02T13:21:43.877524+0000","last_became_peered":"2024-04-02T13:21:43.877524+0000","last_unstale":"2024-04-02T13:21:43.886428+0000","last_undegraded":"2024-04-02T13:21:43.886428+0000","last_fullsized":"2024-04-02T13:21:43.886428+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:40:40.996848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897209+0000","last_change":"2024-04-02T13:21:43.889363+0000","last_active":"2024-04-02T13:21:43.897209+0000","last_peered":"2024-04-02T13:21:43.897209+0000","last_clean":"2024-04-02T13:21:43.897209+0000","last_became_active":"2024-04-02T13:21:43.889089+0000","last_became_peered":"2024-04-02T13:21:43.889089+0000","last_unstale":"2024-04-02T13:21:43.897209+0000","last_undegraded":"2024-04-02T13:21:43.897209+0000","last_fullsized":"2024-04-02T13:21:43.897209+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:34.646859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.877502+0000","last_change":"2024-04-02T13:21:43.873747+0000","last_active":"2024-04-02T13:21:43.877502+0000","last_peered":"2024-04-02T13:21:43.877502+0000","last_clean":"2024-04-02T13:21:43.877502+0000","last_became_active":"2024-04-02T13:21:43.872056+0000","last_became_peered":"2024-04-02T13:21:43.872056+0000","last_unstale":"2024-04-02T13:21:43.877502+0000","last_undegraded":"2024-04-02T13:21:43.877502+0000","last_fullsized":"2024-04-02T13:21:43.877502+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:56:22.065177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888414+0000","last_change":"2024-04-02T13:21:43.883290+0000","last_active":"2024-04-02T13:21:43.888414+0000","last_peered":"2024-04-02T13:21:43.888414+0000","last_clean":"2024-04-02T13:21:43.888414+0000","last_became_active":"2024-04-02T13:21:43.882955+0000","last_became_peered":"2024-04-02T13:21:43.882955+0000","last_unstale":"2024-04-02T13:21:43.888414+0000","last_undegraded":"2024-04-02T13:21:43.888414+0000","last_fullsized":"2024-04-02T13:21:43.888414+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:58.834076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897446+0000","last_change":"2024-04-02T13:21:43.894628+0000","last_active":"2024-04-02T13:21:43.897446+0000","last_peered":"2024-04-02T13:21:43.897446+0000","last_clean":"2024-04-02T13:21:43.897446+0000","last_became_active":"2024-04-02T13:21:43.894282+0000","last_became_peered":"2024-04-02T13:21:43.894282+0000","last_unstale":"2024-04-02T13:21:43.897446+0000","last_undegraded":"2024-04-02T13:21:43.897446+0000","last_fullsized":"2024-04-02T13:21:43.897446+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:43:09.586928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.876606+0000","last_change":"2024-04-02T13:21:43.872575+0000","last_active":"2024-04-02T13:21:43.876606+0000","last_peered":"2024-04-02T13:21:43.876606+0000","last_clean":"2024-04-02T13:21:43.876606+0000","last_became_active":"2024-04-02T13:21:43.872316+0000","last_became_peered":"2024-04-02T13:21:43.872316+0000","last_unstale":"2024-04-02T13:21:43.876606+0000","last_undegraded":"2024-04-02T13:21:43.876606+0000","last_fullsized":"2024-04-02T13:21:43.876606+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:43:09.586928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891124+0000","last_change":"2024-04-02T13:21:43.885777+0000","last_active":"2024-04-02T13:21:43.891124+0000","last_peered":"2024-04-02T13:21:43.891124+0000","last_clean":"2024-04-02T13:21:43.891124+0000","last_became_active":"2024-04-02T13:21:43.885024+0000","last_became_peered":"2024-04-02T13:21:43.885024+0000","last_unstale":"2024-04-02T13:21:43.891124+0000","last_undegraded":"2024-04-02T13:21:43.891124+0000","last_fullsized":"2024-04-02T13:21:43.891124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:49.117575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886051+0000","last_change":"2024-04-02T13:21:43.877278+0000","last_active":"2024-04-02T13:21:43.886051+0000","last_peered":"2024-04-02T13:21:43.886051+0000","last_clean":"2024-04-02T13:21:43.886051+0000","last_became_active":"2024-04-02T13:21:43.876746+0000","last_became_peered":"2024-04-02T13:21:43.876746+0000","last_unstale":"2024-04-02T13:21:43.886051+0000","last_undegraded":"2024-04-02T13:21:43.886051+0000","last_fullsized":"2024-04-02T13:21:43.886051+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:13.466624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886640+0000","last_change":"2024-04-02T13:21:43.877474+0000","last_active":"2024-04-02T13:21:43.886640+0000","last_peered":"2024-04-02T13:21:43.886640+0000","last_clean":"2024-04-02T13:21:43.886640+0000","last_became_active":"2024-04-02T13:21:43.877199+0000","last_became_peered":"2024-04-02T13:21:43.877199+0000","last_unstale":"2024-04-02T13:21:43.886640+0000","last_undegraded":"2024-04-02T13:21:43.886640+0000","last_fullsized":"2024-04-02T13:21:43.886640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:43:09.586928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883542+0000","last_change":"2024-04-02T13:21:43.879759+0000","last_active":"2024-04-02T13:21:43.883542+0000","last_peered":"2024-04-02T13:21:43.883542+0000","last_clean":"2024-04-02T13:21:43.883542+0000","last_became_active":"2024-04-02T13:21:43.879462+0000","last_became_peered":"2024-04-02T13:21:43.879462+0000","last_unstale":"2024-04-02T13:21:43.883542+0000","last_undegraded":"2024-04-02T13:21:43.883542+0000","last_fullsized":"2024-04-02T13:21:43.883542+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:56:22.065177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.884063+0000","last_change":"2024-04-02T13:21:43.881236+0000","last_active":"2024-04-02T13:21:43.884063+0000","last_peered":"2024-04-02T13:21:43.884063+0000","last_clean":"2024-04-02T13:21:43.884063+0000","last_became_active":"2024-04-02T13:21:43.880958+0000","last_became_peered":"2024-04-02T13:21:43.880958+0000","last_unstale":"2024-04-02T13:21:43.884063+0000","last_undegraded":"2024-04-02T13:21:43.884063+0000","last_fullsized":"2024-04-02T13:21:43.884063+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:05:19.791693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891796+0000","last_change":"2024-04-02T13:21:43.885592+0000","last_active":"2024-04-02T13:21:43.891796+0000","last_peered":"2024-04-02T13:21:43.891796+0000","last_clean":"2024-04-02T13:21:43.891796+0000","last_became_active":"2024-04-02T13:21:43.884169+0000","last_became_peered":"2024-04-02T13:21:43.884169+0000","last_unstale":"2024-04-02T13:21:43.891796+0000","last_undegraded":"2024-04-02T13:21:43.891796+0000","last_fullsized":"2024-04-02T13:21:43.891796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:47:16.117390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887124+0000","last_change":"2024-04-02T13:21:43.884294+0000","last_active":"2024-04-02T13:21:43.887124+0000","last_peered":"2024-04-02T13:21:43.887124+0000","last_clean":"2024-04-02T13:21:43.887124+0000","last_became_active":"2024-04-02T13:21:43.884029+0000","last_became_peered":"2024-04-02T13:21:43.884029+0000","last_unstale":"2024-04-02T13:21:43.887124+0000","last_undegraded":"2024-04-02T13:21:43.887124+0000","last_fullsized":"2024-04-02T13:21:43.887124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:38.212652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.877694+0000","last_change":"2024-04-02T13:21:43.875390+0000","last_active":"2024-04-02T13:21:43.877694+0000","last_peered":"2024-04-02T13:21:43.877694+0000","last_clean":"2024-04-02T13:21:43.877694+0000","last_became_active":"2024-04-02T13:21:43.875118+0000","last_became_peered":"2024-04-02T13:21:43.875118+0000","last_unstale":"2024-04-02T13:21:43.877694+0000","last_undegraded":"2024-04-02T13:21:43.877694+0000","last_fullsized":"2024-04-02T13:21:43.877694+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:32:03.790521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.898971+0000","last_change":"2024-04-02T13:21:43.892131+0000","last_active":"2024-04-02T13:21:43.898971+0000","last_peered":"2024-04-02T13:21:43.898971+0000","last_clean":"2024-04-02T13:21:43.898971+0000","last_became_active":"2024-04-02T13:21:43.891816+0000","last_became_peered":"2024-04-02T13:21:43.891816+0000","last_unstale":"2024-04-02T13:21:43.898971+0000","last_undegraded":"2024-04-02T13:21:43.898971+0000","last_fullsized":"2024-04-02T13:21:43.898971+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:47.913991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883161+0000","last_change":"2024-04-02T13:21:43.880541+0000","last_active":"2024-04-02T13:21:43.883161+0000","last_peered":"2024-04-02T13:21:43.883161+0000","last_clean":"2024-04-02T13:21:43.883161+0000","last_became_active":"2024-04-02T13:21:43.879795+0000","last_became_peered":"2024-04-02T13:21:43.879795+0000","last_unstale":"2024-04-02T13:21:43.883161+0000","last_undegraded":"2024-04-02T13:21:43.883161+0000","last_fullsized":"2024-04-02T13:21:43.883161+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:46.012704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.890753+0000","last_change":"2024-04-02T13:21:43.881876+0000","last_active":"2024-04-02T13:21:43.890753+0000","last_peered":"2024-04-02T13:21:43.890753+0000","last_clean":"2024-04-02T13:21:43.890753+0000","last_became_active":"2024-04-02T13:21:43.881579+0000","last_became_peered":"2024-04-02T13:21:43.881579+0000","last_unstale":"2024-04-02T13:21:43.890753+0000","last_undegraded":"2024-04-02T13:21:43.890753+0000","last_fullsized":"2024-04-02T13:21:43.890753+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:26:56.782064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886741+0000","last_change":"2024-04-02T13:21:43.882505+0000","last_active":"2024-04-02T13:21:43.886741+0000","last_peered":"2024-04-02T13:21:43.886741+0000","last_clean":"2024-04-02T13:21:43.886741+0000","last_became_active":"2024-04-02T13:21:43.882124+0000","last_became_peered":"2024-04-02T13:21:43.882124+0000","last_unstale":"2024-04-02T13:21:43.886741+0000","last_undegraded":"2024-04-02T13:21:43.886741+0000","last_fullsized":"2024-04-02T13:21:43.886741+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:50:26.272689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887953+0000","last_change":"2024-04-02T13:21:43.880686+0000","last_active":"2024-04-02T13:21:43.887953+0000","last_peered":"2024-04-02T13:21:43.887953+0000","last_clean":"2024-04-02T13:21:43.887953+0000","last_became_active":"2024-04-02T13:21:43.880399+0000","last_became_peered":"2024-04-02T13:21:43.880399+0000","last_unstale":"2024-04-02T13:21:43.887953+0000","last_undegraded":"2024-04-02T13:21:43.887953+0000","last_fullsized":"2024-04-02T13:21:43.887953+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:34.646859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.884062+0000","last_change":"2024-04-02T13:21:43.878844+0000","last_active":"2024-04-02T13:21:43.884062+0000","last_peered":"2024-04-02T13:21:43.884062+0000","last_clean":"2024-04-02T13:21:43.884062+0000","last_became_active":"2024-04-02T13:21:43.878605+0000","last_became_peered":"2024-04-02T13:21:43.878605+0000","last_unstale":"2024-04-02T13:21:43.884062+0000","last_undegraded":"2024-04-02T13:21:43.884062+0000","last_fullsized":"2024-04-02T13:21:43.884062+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:26.646577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888942+0000","last_change":"2024-04-02T13:21:43.888058+0000","last_active":"2024-04-02T13:21:43.888942+0000","last_peered":"2024-04-02T13:21:43.888942+0000","last_clean":"2024-04-02T13:21:43.888942+0000","last_became_active":"2024-04-02T13:21:43.887696+0000","last_became_peered":"2024-04-02T13:21:43.887696+0000","last_unstale":"2024-04-02T13:21:43.888942+0000","last_undegraded":"2024-04-02T13:21:43.888942+0000","last_fullsized":"2024-04-02T13:21:43.888942+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:48:14.205602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883665+0000","last_change":"2024-04-02T13:21:43.878318+0000","last_active":"2024-04-02T13:21:43.883665+0000","last_peered":"2024-04-02T13:21:43.883665+0000","last_clean":"2024-04-02T13:21:43.883665+0000","last_became_active":"2024-04-02T13:21:43.878035+0000","last_became_peered":"2024-04-02T13:21:43.878035+0000","last_unstale":"2024-04-02T13:21:43.883665+0000","last_undegraded":"2024-04-02T13:21:43.883665+0000","last_fullsized":"2024-04-02T13:21:43.883665+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:20.667849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888857+0000","last_change":"2024-04-02T13:21:43.885523+0000","last_active":"2024-04-02T13:21:43.888857+0000","last_peered":"2024-04-02T13:21:43.888857+0000","last_clean":"2024-04-02T13:21:43.888857+0000","last_became_active":"2024-04-02T13:21:43.884744+0000","last_became_peered":"2024-04-02T13:21:43.884744+0000","last_unstale":"2024-04-02T13:21:43.888857+0000","last_undegraded":"2024-04-02T13:21:43.888857+0000","last_fullsized":"2024-04-02T13:21:43.888857+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:15:37.587150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.877210+0000","last_change":"2024-04-02T13:21:43.875957+0000","last_active":"2024-04-02T13:21:43.877210+0000","last_peered":"2024-04-02T13:21:43.877210+0000","last_clean":"2024-04-02T13:21:43.877210+0000","last_became_active":"2024-04-02T13:21:43.875630+0000","last_became_peered":"2024-04-02T13:21:43.875630+0000","last_unstale":"2024-04-02T13:21:43.877210+0000","last_undegraded":"2024-04-02T13:21:43.877210+0000","last_fullsized":"2024-04-02T13:21:43.877210+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:47.913991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883756+0000","last_change":"2024-04-02T13:21:43.878610+0000","last_active":"2024-04-02T13:21:43.883756+0000","last_peered":"2024-04-02T13:21:43.883756+0000","last_clean":"2024-04-02T13:21:43.883756+0000","last_became_active":"2024-04-02T13:21:43.878156+0000","last_became_peered":"2024-04-02T13:21:43.878156+0000","last_unstale":"2024-04-02T13:21:43.883756+0000","last_undegraded":"2024-04-02T13:21:43.883756+0000","last_fullsized":"2024-04-02T13:21:43.883756+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:49.117575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883205+0000","last_change":"2024-04-02T13:21:43.880230+0000","last_active":"2024-04-02T13:21:43.883205+0000","last_peered":"2024-04-02T13:21:43.883205+0000","last_clean":"2024-04-02T13:21:43.883205+0000","last_became_active":"2024-04-02T13:21:43.879948+0000","last_became_peered":"2024-04-02T13:21:43.879948+0000","last_unstale":"2024-04-02T13:21:43.883205+0000","last_undegraded":"2024-04-02T13:21:43.883205+0000","last_fullsized":"2024-04-02T13:21:43.883205+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:19:33.449996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.896632+0000","last_change":"2024-04-02T13:21:42.870063+0000","last_active":"2024-04-02T13:21:43.896632+0000","last_peered":"2024-04-02T13:21:43.896632+0000","last_clean":"2024-04-02T13:21:43.896632+0000","last_became_active":"2024-04-02T13:21:42.869787+0000","last_became_peered":"2024-04-02T13:21:42.869787+0000","last_unstale":"2024-04-02T13:21:43.896632+0000","last_undegraded":"2024-04-02T13:21:43.896632+0000","last_fullsized":"2024-04-02T13:21:43.896632+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:35.866655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888088+0000","last_change":"2024-04-02T13:21:43.882527+0000","last_active":"2024-04-02T13:21:43.888088+0000","last_peered":"2024-04-02T13:21:43.888088+0000","last_clean":"2024-04-02T13:21:43.888088+0000","last_became_active":"2024-04-02T13:21:43.882237+0000","last_became_peered":"2024-04-02T13:21:43.882237+0000","last_unstale":"2024-04-02T13:21:43.888088+0000","last_undegraded":"2024-04-02T13:21:43.888088+0000","last_fullsized":"2024-04-02T13:21:43.888088+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:47.913991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.879336+0000","last_change":"2024-04-02T13:21:42.866397+0000","last_active":"2024-04-02T13:21:43.879336+0000","last_peered":"2024-04-02T13:21:43.879336+0000","last_clean":"2024-04-02T13:21:43.879336+0000","last_became_active":"2024-04-02T13:21:42.866129+0000","last_became_peered":"2024-04-02T13:21:42.866129+0000","last_unstale":"2024-04-02T13:21:43.879336+0000","last_undegraded":"2024-04-02T13:21:43.879336+0000","last_fullsized":"2024-04-02T13:21:43.879336+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:25.641837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888627+0000","last_change":"2024-04-02T13:21:43.885483+0000","last_active":"2024-04-02T13:21:43.888627+0000","last_peered":"2024-04-02T13:21:43.888627+0000","last_clean":"2024-04-02T13:21:43.888627+0000","last_became_active":"2024-04-02T13:21:43.885047+0000","last_became_peered":"2024-04-02T13:21:43.885047+0000","last_unstale":"2024-04-02T13:21:43.888627+0000","last_undegraded":"2024-04-02T13:21:43.888627+0000","last_fullsized":"2024-04-02T13:21:43.888627+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:15:37.587150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.872089+0000","last_change":"2024-04-02T13:21:42.863185+0000","last_active":"2024-04-02T13:21:43.872089+0000","last_peered":"2024-04-02T13:21:43.872089+0000","last_clean":"2024-04-02T13:21:43.872089+0000","last_became_active":"2024-04-02T13:21:42.862944+0000","last_became_peered":"2024-04-02T13:21:42.862944+0000","last_unstale":"2024-04-02T13:21:43.872089+0000","last_undegraded":"2024-04-02T13:21:43.872089+0000","last_fullsized":"2024-04-02T13:21:43.872089+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:37.207912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887575+0000","last_change":"2024-04-02T13:21:43.884619+0000","last_active":"2024-04-02T13:21:43.887575+0000","last_peered":"2024-04-02T13:21:43.887575+0000","last_clean":"2024-04-02T13:21:43.887575+0000","last_became_active":"2024-04-02T13:21:43.884286+0000","last_became_peered":"2024-04-02T13:21:43.884286+0000","last_unstale":"2024-04-02T13:21:43.887575+0000","last_undegraded":"2024-04-02T13:21:43.887575+0000","last_fullsized":"2024-04-02T13:21:43.887575+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:48:14.205602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.874745+0000","last_change":"2024-04-02T13:21:42.865049+0000","last_active":"2024-04-02T13:21:43.874745+0000","last_peered":"2024-04-02T13:21:43.874745+0000","last_clean":"2024-04-02T13:21:43.874745+0000","last_became_active":"2024-04-02T13:21:42.863891+0000","last_became_peered":"2024-04-02T13:21:42.863891+0000","last_unstale":"2024-04-02T13:21:43.874745+0000","last_undegraded":"2024-04-02T13:21:43.874745+0000","last_fullsized":"2024-04-02T13:21:43.874745+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:05:18.786953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897383+0000","last_change":"2024-04-02T13:21:43.890395+0000","last_active":"2024-04-02T13:21:43.897383+0000","last_peered":"2024-04-02T13:21:43.897383+0000","last_clean":"2024-04-02T13:21:43.897383+0000","last_became_active":"2024-04-02T13:21:43.890130+0000","last_became_peered":"2024-04-02T13:21:43.890130+0000","last_unstale":"2024-04-02T13:21:43.897383+0000","last_undegraded":"2024-04-02T13:21:43.897383+0000","last_fullsized":"2024-04-02T13:21:43.897383+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:47.913991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897694+0000","last_change":"2024-04-02T13:21:42.872282+0000","last_active":"2024-04-02T13:21:43.897694+0000","last_peered":"2024-04-02T13:21:43.897694+0000","last_clean":"2024-04-02T13:21:43.897694+0000","last_became_active":"2024-04-02T13:21:42.871960+0000","last_became_peered":"2024-04-02T13:21:42.871960+0000","last_unstale":"2024-04-02T13:21:43.897694+0000","last_undegraded":"2024-04-02T13:21:43.897694+0000","last_fullsized":"2024-04-02T13:21:43.897694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:33.642119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886073+0000","last_change":"2024-04-02T13:21:43.878315+0000","last_active":"2024-04-02T13:21:43.886073+0000","last_peered":"2024-04-02T13:21:43.886073+0000","last_clean":"2024-04-02T13:21:43.886073+0000","last_became_active":"2024-04-02T13:21:43.877294+0000","last_became_peered":"2024-04-02T13:21:43.877294+0000","last_unstale":"2024-04-02T13:21:43.886073+0000","last_undegraded":"2024-04-02T13:21:43.886073+0000","last_fullsized":"2024-04-02T13:21:43.886073+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:40:40.996848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.869091+0000","last_change":"2024-04-02T13:21:42.862800+0000","last_active":"2024-04-02T13:21:43.869091+0000","last_peered":"2024-04-02T13:21:43.869091+0000","last_clean":"2024-04-02T13:21:43.869091+0000","last_became_active":"2024-04-02T13:21:42.862510+0000","last_became_peered":"2024-04-02T13:21:42.862510+0000","last_unstale":"2024-04-02T13:21:43.869091+0000","last_undegraded":"2024-04-02T13:21:43.869091+0000","last_fullsized":"2024-04-02T13:21:43.869091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:15:36.582410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891145+0000","last_change":"2024-04-02T13:21:43.887181+0000","last_active":"2024-04-02T13:21:43.891145+0000","last_peered":"2024-04-02T13:21:43.891145+0000","last_clean":"2024-04-02T13:21:43.891145+0000","last_became_active":"2024-04-02T13:21:43.886877+0000","last_became_peered":"2024-04-02T13:21:43.886877+0000","last_unstale":"2024-04-02T13:21:43.891145+0000","last_undegraded":"2024-04-02T13:21:43.891145+0000","last_fullsized":"2024-04-02T13:21:43.891145+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:34.646859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871640+0000","last_change":"2024-04-02T13:21:42.871820+0000","last_active":"2024-04-02T13:21:43.871640+0000","last_peered":"2024-04-02T13:21:43.871640+0000","last_clean":"2024-04-02T13:21:43.871640+0000","last_became_active":"2024-04-02T13:21:42.871528+0000","last_became_peered":"2024-04-02T13:21:42.871528+0000","last_unstale":"2024-04-02T13:21:43.871640+0000","last_undegraded":"2024-04-02T13:21:43.871640+0000","last_fullsized":"2024-04-02T13:21:43.871640+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:05:18.786953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.895123+0000","last_change":"2024-04-02T13:21:43.885724+0000","last_active":"2024-04-02T13:21:43.895123+0000","last_peered":"2024-04-02T13:21:43.895123+0000","last_clean":"2024-04-02T13:21:43.895123+0000","last_became_active":"2024-04-02T13:21:43.885310+0000","last_became_peered":"2024-04-02T13:21:43.885310+0000","last_unstale":"2024-04-02T13:21:43.895123+0000","last_undegraded":"2024-04-02T13:21:43.895123+0000","last_fullsized":"2024-04-02T13:21:43.895123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:05:08.926884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.872563+0000","last_change":"2024-04-02T13:21:42.869469+0000","last_active":"2024-04-02T13:21:43.872563+0000","last_peered":"2024-04-02T13:21:43.872563+0000","last_clean":"2024-04-02T13:21:43.872563+0000","last_became_active":"2024-04-02T13:21:42.869003+0000","last_became_peered":"2024-04-02T13:21:42.869003+0000","last_unstale":"2024-04-02T13:21:43.872563+0000","last_undegraded":"2024-04-02T13:21:43.872563+0000","last_fullsized":"2024-04-02T13:21:43.872563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:19.663109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891315+0000","last_change":"2024-04-02T13:21:43.889932+0000","last_active":"2024-04-02T13:21:43.891315+0000","last_peered":"2024-04-02T13:21:43.891315+0000","last_clean":"2024-04-02T13:21:43.891315+0000","last_became_active":"2024-04-02T13:21:43.889649+0000","last_became_peered":"2024-04-02T13:21:43.889649+0000","last_unstale":"2024-04-02T13:21:43.891315+0000","last_undegraded":"2024-04-02T13:21:43.891315+0000","last_fullsized":"2024-04-02T13:21:43.891315+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:36.871395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.872889+0000","last_change":"2024-04-02T13:21:42.861003+0000","last_active":"2024-04-02T13:21:43.872889+0000","last_peered":"2024-04-02T13:21:43.872889+0000","last_clean":"2024-04-02T13:21:43.872889+0000","last_became_active":"2024-04-02T13:21:42.860721+0000","last_became_peered":"2024-04-02T13:21:42.860721+0000","last_unstale":"2024-04-02T13:21:43.872889+0000","last_undegraded":"2024-04-02T13:21:43.872889+0000","last_fullsized":"2024-04-02T13:21:43.872889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:45.007964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883769+0000","last_change":"2024-04-02T13:21:43.880770+0000","last_active":"2024-04-02T13:21:43.883769+0000","last_peered":"2024-04-02T13:21:43.883769+0000","last_clean":"2024-04-02T13:21:43.883769+0000","last_became_active":"2024-04-02T13:21:43.880237+0000","last_became_peered":"2024-04-02T13:21:43.880237+0000","last_unstale":"2024-04-02T13:21:43.883769+0000","last_undegraded":"2024-04-02T13:21:43.883769+0000","last_fullsized":"2024-04-02T13:21:43.883769+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:13.466624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.866440+0000","last_change":"2024-04-02T13:21:42.864488+0000","last_active":"2024-04-02T13:21:43.866440+0000","last_peered":"2024-04-02T13:21:43.866440+0000","last_clean":"2024-04-02T13:21:43.866440+0000","last_became_active":"2024-04-02T13:21:42.864203+0000","last_became_peered":"2024-04-02T13:21:42.864203+0000","last_unstale":"2024-04-02T13:21:43.866440+0000","last_undegraded":"2024-04-02T13:21:43.866440+0000","last_fullsized":"2024-04-02T13:21:43.866440+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:48.112835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886807+0000","last_change":"2024-04-02T13:21:43.884306+0000","last_active":"2024-04-02T13:21:43.886807+0000","last_peered":"2024-04-02T13:21:43.886807+0000","last_clean":"2024-04-02T13:21:43.886807+0000","last_became_active":"2024-04-02T13:21:43.883955+0000","last_became_peered":"2024-04-02T13:21:43.883955+0000","last_unstale":"2024-04-02T13:21:43.886807+0000","last_undegraded":"2024-04-02T13:21:43.886807+0000","last_fullsized":"2024-04-02T13:21:43.886807+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:38.212652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.899186+0000","last_change":"2024-04-02T13:21:42.870155+0000","last_active":"2024-04-02T13:21:43.899186+0000","last_peered":"2024-04-02T13:21:43.899186+0000","last_clean":"2024-04-02T13:21:43.899186+0000","last_became_active":"2024-04-02T13:21:42.869902+0000","last_became_peered":"2024-04-02T13:21:42.869902+0000","last_unstale":"2024-04-02T13:21:43.899186+0000","last_undegraded":"2024-04-02T13:21:43.899186+0000","last_fullsized":"2024-04-02T13:21:43.899186+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:42:31.880226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":8,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.893865+0000","last_change":"2024-04-02T13:21:43.893865+0000","last_active":"2024-04-02T13:21:43.893865+0000","last_peered":"2024-04-02T13:21:43.893865+0000","last_clean":"2024-04-02T13:21:43.893865+0000","last_became_active":"2024-04-02T13:21:43.893601+0000","last_became_peered":"2024-04-02T13:21:43.893601+0000","last_unstale":"2024-04-02T13:21:43.893865+0000","last_undegraded":"2024-04-02T13:21:43.893865+0000","last_fullsized":"2024-04-02T13:21:43.893865+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:10.725145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'5","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.874107+0000","last_change":"2024-04-02T13:21:42.863541+0000","last_active":"2024-04-02T13:21:43.874107+0000","last_peered":"2024-04-02T13:21:43.874107+0000","last_clean":"2024-04-02T13:21:43.874107+0000","last_became_active":"2024-04-02T13:21:42.863081+0000","last_became_peered":"2024-04-02T13:21:42.863081+0000","last_unstale":"2024-04-02T13:21:43.874107+0000","last_undegraded":"2024-04-02T13:21:43.874107+0000","last_fullsized":"2024-04-02T13:21:43.874107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:12.461884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897413+0000","last_change":"2024-04-02T13:21:43.892311+0000","last_active":"2024-04-02T13:21:43.897413+0000","last_peered":"2024-04-02T13:21:43.897413+0000","last_clean":"2024-04-02T13:21:43.897413+0000","last_became_active":"2024-04-02T13:21:43.889327+0000","last_became_peered":"2024-04-02T13:21:43.889327+0000","last_unstale":"2024-04-02T13:21:43.897413+0000","last_undegraded":"2024-04-02T13:21:43.897413+0000","last_fullsized":"2024-04-02T13:21:43.897413+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:10.725145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.872151+0000","last_change":"2024-04-02T13:21:42.868191+0000","last_active":"2024-04-02T13:21:43.872151+0000","last_peered":"2024-04-02T13:21:43.872151+0000","last_clean":"2024-04-02T13:21:43.872151+0000","last_became_active":"2024-04-02T13:21:42.867897+0000","last_became_peered":"2024-04-02T13:21:42.867897+0000","last_unstale":"2024-04-02T13:21:43.872151+0000","last_undegraded":"2024-04-02T13:21:43.872151+0000","last_fullsized":"2024-04-02T13:21:43.872151+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:32:02.785781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.895101+0000","last_change":"2024-04-02T13:21:43.885167+0000","last_active":"2024-04-02T13:21:43.895101+0000","last_peered":"2024-04-02T13:21:43.895101+0000","last_clean":"2024-04-02T13:21:43.895101+0000","last_became_active":"2024-04-02T13:21:43.884844+0000","last_became_peered":"2024-04-02T13:21:43.884844+0000","last_unstale":"2024-04-02T13:21:43.895101+0000","last_undegraded":"2024-04-02T13:21:43.895101+0000","last_fullsized":"2024-04-02T13:21:43.895101+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:10:20.429411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871825+0000","last_change":"2024-04-02T13:21:42.872412+0000","last_active":"2024-04-02T13:21:43.871825+0000","last_peered":"2024-04-02T13:21:43.871825+0000","last_clean":"2024-04-02T13:21:43.871825+0000","last_became_active":"2024-04-02T13:21:42.872160+0000","last_became_peered":"2024-04-02T13:21:42.872160+0000","last_unstale":"2024-04-02T13:21:43.871825+0000","last_undegraded":"2024-04-02T13:21:43.871825+0000","last_fullsized":"2024-04-02T13:21:43.871825+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:50:25.267949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.878071+0000","last_change":"2024-04-02T13:21:43.876405+0000","last_active":"2024-04-02T13:21:43.878071+0000","last_peered":"2024-04-02T13:21:43.878071+0000","last_clean":"2024-04-02T13:21:43.878071+0000","last_became_active":"2024-04-02T13:21:43.875443+0000","last_became_peered":"2024-04-02T13:21:43.875443+0000","last_unstale":"2024-04-02T13:21:43.878071+0000","last_undegraded":"2024-04-02T13:21:43.878071+0000","last_fullsized":"2024-04-02T13:21:43.878071+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:10.725145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.865679+0000","last_change":"2024-04-02T13:21:42.870037+0000","last_active":"2024-04-02T13:21:43.865679+0000","last_peered":"2024-04-02T13:21:43.865679+0000","last_clean":"2024-04-02T13:21:43.865679+0000","last_became_active":"2024-04-02T13:21:42.869753+0000","last_became_peered":"2024-04-02T13:21:42.869753+0000","last_unstale":"2024-04-02T13:21:43.865679+0000","last_undegraded":"2024-04-02T13:21:43.865679+0000","last_fullsized":"2024-04-02T13:21:43.865679+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:47:15.112650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886820+0000","last_change":"2024-04-02T13:21:43.883404+0000","last_active":"2024-04-02T13:21:43.886820+0000","last_peered":"2024-04-02T13:21:43.886820+0000","last_clean":"2024-04-02T13:21:43.886820+0000","last_became_active":"2024-04-02T13:21:43.882899+0000","last_became_peered":"2024-04-02T13:21:43.882899+0000","last_unstale":"2024-04-02T13:21:43.886820+0000","last_undegraded":"2024-04-02T13:21:43.886820+0000","last_fullsized":"2024-04-02T13:21:43.886820+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:58.834076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.896382+0000","last_change":"2024-04-02T13:21:42.863744+0000","last_active":"2024-04-02T13:21:43.896382+0000","last_peered":"2024-04-02T13:21:43.896382+0000","last_clean":"2024-04-02T13:21:43.896382+0000","last_became_active":"2024-04-02T13:21:42.863198+0000","last_became_peered":"2024-04-02T13:21:42.863198+0000","last_unstale":"2024-04-02T13:21:43.896382+0000","last_undegraded":"2024-04-02T13:21:43.896382+0000","last_fullsized":"2024-04-02T13:21:43.896382+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:42:31.880226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887782+0000","last_change":"2024-04-02T13:21:43.884934+0000","last_active":"2024-04-02T13:21:43.887782+0000","last_peered":"2024-04-02T13:21:43.887782+0000","last_clean":"2024-04-02T13:21:43.887782+0000","last_became_active":"2024-04-02T13:21:43.884630+0000","last_became_peered":"2024-04-02T13:21:43.884630+0000","last_unstale":"2024-04-02T13:21:43.887782+0000","last_undegraded":"2024-04-02T13:21:43.887782+0000","last_fullsized":"2024-04-02T13:21:43.887782+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:42.322235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.868413+0000","last_change":"2024-04-02T13:21:42.863458+0000","last_active":"2024-04-02T13:21:43.868413+0000","last_peered":"2024-04-02T13:21:43.868413+0000","last_clean":"2024-04-02T13:21:43.868413+0000","last_became_active":"2024-04-02T13:21:42.863196+0000","last_became_peered":"2024-04-02T13:21:42.863196+0000","last_unstale":"2024-04-02T13:21:43.868413+0000","last_undegraded":"2024-04-02T13:21:43.868413+0000","last_fullsized":"2024-04-02T13:21:43.868413+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:46.909251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.876133+0000","last_change":"2024-04-02T13:21:43.874576+0000","last_active":"2024-04-02T13:21:43.876133+0000","last_peered":"2024-04-02T13:21:43.876133+0000","last_clean":"2024-04-02T13:21:43.876133+0000","last_became_active":"2024-04-02T13:21:43.874255+0000","last_became_peered":"2024-04-02T13:21:43.874255+0000","last_unstale":"2024-04-02T13:21:43.876133+0000","last_undegraded":"2024-04-02T13:21:43.876133+0000","last_fullsized":"2024-04-02T13:21:43.876133+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:19:33.449996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.874853+0000","last_change":"2024-04-02T13:21:42.872412+0000","last_active":"2024-04-02T13:21:43.874853+0000","last_peered":"2024-04-02T13:21:43.874853+0000","last_clean":"2024-04-02T13:21:43.874853+0000","last_became_active":"2024-04-02T13:21:42.872147+0000","last_became_peered":"2024-04-02T13:21:42.872147+0000","last_unstale":"2024-04-02T13:21:43.874853+0000","last_undegraded":"2024-04-02T13:21:43.874853+0000","last_fullsized":"2024-04-02T13:21:43.874853+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:50:25.267949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.882605+0000","last_change":"2024-04-02T13:21:43.879554+0000","last_active":"2024-04-02T13:21:43.882605+0000","last_peered":"2024-04-02T13:21:43.882605+0000","last_clean":"2024-04-02T13:21:43.882605+0000","last_became_active":"2024-04-02T13:21:43.879279+0000","last_became_peered":"2024-04-02T13:21:43.879279+0000","last_unstale":"2024-04-02T13:21:43.882605+0000","last_undegraded":"2024-04-02T13:21:43.882605+0000","last_fullsized":"2024-04-02T13:21:43.882605+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:38.212652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.873146+0000","last_change":"2024-04-02T13:21:42.863953+0000","last_active":"2024-04-02T13:21:43.873146+0000","last_peered":"2024-04-02T13:21:43.873146+0000","last_clean":"2024-04-02T13:21:43.873146+0000","last_became_active":"2024-04-02T13:21:42.863672+0000","last_became_peered":"2024-04-02T13:21:42.863672+0000","last_unstale":"2024-04-02T13:21:43.873146+0000","last_undegraded":"2024-04-02T13:21:43.873146+0000","last_fullsized":"2024-04-02T13:21:43.873146+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:02:51.970866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891171+0000","last_change":"2024-04-02T13:21:43.887379+0000","last_active":"2024-04-02T13:21:43.891171+0000","last_peered":"2024-04-02T13:21:43.891171+0000","last_clean":"2024-04-02T13:21:43.891171+0000","last_became_active":"2024-04-02T13:21:43.887007+0000","last_became_peered":"2024-04-02T13:21:43.887007+0000","last_unstale":"2024-04-02T13:21:43.891171+0000","last_undegraded":"2024-04-02T13:21:43.891171+0000","last_fullsized":"2024-04-02T13:21:43.891171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:56:07.414303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871640+0000","last_change":"2024-04-02T13:21:42.868600+0000","last_active":"2024-04-02T13:21:43.871640+0000","last_peered":"2024-04-02T13:21:43.871640+0000","last_clean":"2024-04-02T13:21:43.871640+0000","last_became_active":"2024-04-02T13:21:42.868113+0000","last_became_peered":"2024-04-02T13:21:42.868113+0000","last_unstale":"2024-04-02T13:21:43.871640+0000","last_undegraded":"2024-04-02T13:21:43.871640+0000","last_fullsized":"2024-04-02T13:21:43.871640+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:25.641837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886617+0000","last_change":"2024-04-02T13:21:43.882139+0000","last_active":"2024-04-02T13:21:43.886617+0000","last_peered":"2024-04-02T13:21:43.886617+0000","last_clean":"2024-04-02T13:21:43.886617+0000","last_became_active":"2024-04-02T13:21:43.881025+0000","last_became_peered":"2024-04-02T13:21:43.881025+0000","last_unstale":"2024-04-02T13:21:43.886617+0000","last_undegraded":"2024-04-02T13:21:43.886617+0000","last_fullsized":"2024-04-02T13:21:43.886617+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:58.834076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871612+0000","last_change":"2024-04-02T13:21:42.866632+0000","last_active":"2024-04-02T13:21:43.871612+0000","last_peered":"2024-04-02T13:21:43.871612+0000","last_clean":"2024-04-02T13:21:43.871612+0000","last_became_active":"2024-04-02T13:21:42.866363+0000","last_became_peered":"2024-04-02T13:21:42.866363+0000","last_unstale":"2024-04-02T13:21:43.871612+0000","last_undegraded":"2024-04-02T13:21:43.871612+0000","last_fullsized":"2024-04-02T13:21:43.871612+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:58:01.695741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891636+0000","last_change":"2024-04-02T13:21:43.885397+0000","last_active":"2024-04-02T13:21:43.891636+0000","last_peered":"2024-04-02T13:21:43.891636+0000","last_clean":"2024-04-02T13:21:43.891636+0000","last_became_active":"2024-04-02T13:21:43.885108+0000","last_became_peered":"2024-04-02T13:21:43.885108+0000","last_unstale":"2024-04-02T13:21:43.891636+0000","last_undegraded":"2024-04-02T13:21:43.891636+0000","last_fullsized":"2024-04-02T13:21:43.891636+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:15:37.587150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871715+0000","last_change":"2024-04-02T13:21:42.864454+0000","last_active":"2024-04-02T13:21:43.871715+0000","last_peered":"2024-04-02T13:21:43.871715+0000","last_clean":"2024-04-02T13:21:43.871715+0000","last_became_active":"2024-04-02T13:21:42.864126+0000","last_became_peered":"2024-04-02T13:21:42.864126+0000","last_unstale":"2024-04-02T13:21:43.871715+0000","last_undegraded":"2024-04-02T13:21:43.871715+0000","last_fullsized":"2024-04-02T13:21:43.871715+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:45:29.122779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.882589+0000","last_change":"2024-04-02T13:21:43.876666+0000","last_active":"2024-04-02T13:21:43.882589+0000","last_peered":"2024-04-02T13:21:43.882589+0000","last_clean":"2024-04-02T13:21:43.882589+0000","last_became_active":"2024-04-02T13:21:43.876405+0000","last_became_peered":"2024-04-02T13:21:43.876405+0000","last_unstale":"2024-04-02T13:21:43.882589+0000","last_undegraded":"2024-04-02T13:21:43.882589+0000","last_fullsized":"2024-04-02T13:21:43.882589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:46.012704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.869629+0000","last_change":"2024-04-02T13:21:42.867209+0000","last_active":"2024-04-02T13:21:43.869629+0000","last_peered":"2024-04-02T13:21:43.869629+0000","last_clean":"2024-04-02T13:21:43.869629+0000","last_became_active":"2024-04-02T13:21:42.866938+0000","last_became_peered":"2024-04-02T13:21:42.866938+0000","last_unstale":"2024-04-02T13:21:43.869629+0000","last_undegraded":"2024-04-02T13:21:43.869629+0000","last_fullsized":"2024-04-02T13:21:43.869629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:33.642119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.895095+0000","last_change":"2024-04-02T13:21:43.885621+0000","last_active":"2024-04-02T13:21:43.895095+0000","last_peered":"2024-04-02T13:21:43.895095+0000","last_clean":"2024-04-02T13:21:43.895095+0000","last_became_active":"2024-04-02T13:21:43.884937+0000","last_became_peered":"2024-04-02T13:21:43.884937+0000","last_unstale":"2024-04-02T13:21:43.895095+0000","last_undegraded":"2024-04-02T13:21:43.895095+0000","last_fullsized":"2024-04-02T13:21:43.895095+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:52:41.672416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871838+0000","last_change":"2024-04-02T13:21:42.865961+0000","last_active":"2024-04-02T13:21:43.871838+0000","last_peered":"2024-04-02T13:21:43.871838+0000","last_clean":"2024-04-02T13:21:43.871838+0000","last_became_active":"2024-04-02T13:21:42.865702+0000","last_became_peered":"2024-04-02T13:21:42.865702+0000","last_unstale":"2024-04-02T13:21:43.871838+0000","last_undegraded":"2024-04-02T13:21:43.871838+0000","last_fullsized":"2024-04-02T13:21:43.871838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:26:09.307417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888012+0000","last_change":"2024-04-02T13:21:43.881163+0000","last_active":"2024-04-02T13:21:43.888012+0000","last_peered":"2024-04-02T13:21:43.888012+0000","last_clean":"2024-04-02T13:21:43.888012+0000","last_became_active":"2024-04-02T13:21:43.880878+0000","last_became_peered":"2024-04-02T13:21:43.880878+0000","last_unstale":"2024-04-02T13:21:43.888012+0000","last_undegraded":"2024-04-02T13:21:43.888012+0000","last_fullsized":"2024-04-02T13:21:43.888012+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:42:32.884966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.899503+0000","last_change":"2024-04-02T13:21:42.869522+0000","last_active":"2024-04-02T13:21:43.899503+0000","last_peered":"2024-04-02T13:21:43.899503+0000","last_clean":"2024-04-02T13:21:43.899503+0000","last_became_active":"2024-04-02T13:21:42.869256+0000","last_became_peered":"2024-04-02T13:21:42.869256+0000","last_unstale":"2024-04-02T13:21:43.899503+0000","last_undegraded":"2024-04-02T13:21:43.899503+0000","last_fullsized":"2024-04-02T13:21:43.899503+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:09.720405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883790+0000","last_change":"2024-04-02T13:21:43.881002+0000","last_active":"2024-04-02T13:21:43.883790+0000","last_peered":"2024-04-02T13:21:43.883790+0000","last_clean":"2024-04-02T13:21:43.883790+0000","last_became_active":"2024-04-02T13:21:43.880726+0000","last_became_peered":"2024-04-02T13:21:43.880726+0000","last_unstale":"2024-04-02T13:21:43.883790+0000","last_undegraded":"2024-04-02T13:21:43.883790+0000","last_fullsized":"2024-04-02T13:21:43.883790+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:40:40.996848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.872459+0000","last_change":"2024-04-02T13:21:42.867407+0000","last_active":"2024-04-02T13:21:43.872459+0000","last_peered":"2024-04-02T13:21:43.872459+0000","last_clean":"2024-04-02T13:21:43.872459+0000","last_became_active":"2024-04-02T13:21:42.867155+0000","last_became_peered":"2024-04-02T13:21:42.867155+0000","last_unstale":"2024-04-02T13:21:43.872459+0000","last_undegraded":"2024-04-02T13:21:43.872459+0000","last_fullsized":"2024-04-02T13:21:43.872459+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:36:08.445273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":8,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887540+0000","last_change":"2024-04-02T13:21:43.887540+0000","last_active":"2024-04-02T13:21:43.887540+0000","last_peered":"2024-04-02T13:21:43.887540+0000","last_clean":"2024-04-02T13:21:43.887540+0000","last_became_active":"2024-04-02T13:21:43.887198+0000","last_became_peered":"2024-04-02T13:21:43.887198+0000","last_unstale":"2024-04-02T13:21:43.887540+0000","last_undegraded":"2024-04-02T13:21:43.887540+0000","last_fullsized":"2024-04-02T13:21:43.887540+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:51:30.559674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.881397+0000","last_change":"2024-04-02T13:21:42.865779+0000","last_active":"2024-04-02T13:21:43.881397+0000","last_peered":"2024-04-02T13:21:43.881397+0000","last_clean":"2024-04-02T13:21:43.881397+0000","last_became_active":"2024-04-02T13:21:42.865523+0000","last_became_peered":"2024-04-02T13:21:42.865523+0000","last_unstale":"2024-04-02T13:21:43.881397+0000","last_undegraded":"2024-04-02T13:21:43.881397+0000","last_fullsized":"2024-04-02T13:21:43.881397+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:50:25.267949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897184+0000","last_change":"2024-04-02T13:21:43.889026+0000","last_active":"2024-04-02T13:21:43.897184+0000","last_peered":"2024-04-02T13:21:43.897184+0000","last_clean":"2024-04-02T13:21:43.897184+0000","last_became_active":"2024-04-02T13:21:43.888729+0000","last_became_peered":"2024-04-02T13:21:43.888729+0000","last_unstale":"2024-04-02T13:21:43.897184+0000","last_undegraded":"2024-04-02T13:21:43.897184+0000","last_fullsized":"2024-04-02T13:21:43.897184+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:49.117575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.880880+0000","last_change":"2024-04-02T13:21:42.869452+0000","last_active":"2024-04-02T13:21:43.880880+0000","last_peered":"2024-04-02T13:21:43.880880+0000","last_clean":"2024-04-02T13:21:43.880880+0000","last_became_active":"2024-04-02T13:21:42.869042+0000","last_became_peered":"2024-04-02T13:21:42.869042+0000","last_unstale":"2024-04-02T13:21:43.880880+0000","last_undegraded":"2024-04-02T13:21:43.880880+0000","last_fullsized":"2024-04-02T13:21:43.880880+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:05:18.786953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.881120+0000","last_change":"2024-04-02T13:21:43.876850+0000","last_active":"2024-04-02T13:21:43.881120+0000","last_peered":"2024-04-02T13:21:43.881120+0000","last_clean":"2024-04-02T13:21:43.881120+0000","last_became_active":"2024-04-02T13:21:43.876507+0000","last_became_peered":"2024-04-02T13:21:43.876507+0000","last_unstale":"2024-04-02T13:21:43.881120+0000","last_undegraded":"2024-04-02T13:21:43.881120+0000","last_fullsized":"2024-04-02T13:21:43.881120+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:15:37.587150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.866263+0000","last_change":"2024-04-02T13:21:42.866521+0000","last_active":"2024-04-02T13:21:43.866263+0000","last_peered":"2024-04-02T13:21:43.866263+0000","last_clean":"2024-04-02T13:21:43.866263+0000","last_became_active":"2024-04-02T13:21:42.866185+0000","last_became_peered":"2024-04-02T13:21:42.866185+0000","last_unstale":"2024-04-02T13:21:43.866263+0000","last_undegraded":"2024-04-02T13:21:43.866263+0000","last_fullsized":"2024-04-02T13:21:43.866263+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:56:06.409563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891164+0000","last_change":"2024-04-02T13:21:43.888269+0000","last_active":"2024-04-02T13:21:43.891164+0000","last_peered":"2024-04-02T13:21:43.891164+0000","last_clean":"2024-04-02T13:21:43.891164+0000","last_became_active":"2024-04-02T13:21:43.887677+0000","last_became_peered":"2024-04-02T13:21:43.887677+0000","last_unstale":"2024-04-02T13:21:43.891164+0000","last_undegraded":"2024-04-02T13:21:43.891164+0000","last_fullsized":"2024-04-02T13:21:43.891164+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:42:32.884966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.874248+0000","last_change":"2024-04-02T13:21:42.868817+0000","last_active":"2024-04-02T13:21:43.874248+0000","last_peered":"2024-04-02T13:21:43.874248+0000","last_clean":"2024-04-02T13:21:43.874248+0000","last_became_active":"2024-04-02T13:21:42.868541+0000","last_became_peered":"2024-04-02T13:21:42.868541+0000","last_unstale":"2024-04-02T13:21:43.874248+0000","last_undegraded":"2024-04-02T13:21:43.874248+0000","last_fullsized":"2024-04-02T13:21:43.874248+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:25.641837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883617+0000","last_change":"2024-04-02T13:21:43.880463+0000","last_active":"2024-04-02T13:21:43.883617+0000","last_peered":"2024-04-02T13:21:43.883617+0000","last_clean":"2024-04-02T13:21:43.883617+0000","last_became_active":"2024-04-02T13:21:43.880220+0000","last_became_peered":"2024-04-02T13:21:43.880220+0000","last_unstale":"2024-04-02T13:21:43.883617+0000","last_undegraded":"2024-04-02T13:21:43.883617+0000","last_fullsized":"2024-04-02T13:21:43.883617+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:32:03.790521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871008+0000","last_change":"2024-04-02T13:21:42.861463+0000","last_active":"2024-04-02T13:21:43.871008+0000","last_peered":"2024-04-02T13:21:43.871008+0000","last_clean":"2024-04-02T13:21:43.871008+0000","last_became_active":"2024-04-02T13:21:42.861191+0000","last_became_peered":"2024-04-02T13:21:42.861191+0000","last_unstale":"2024-04-02T13:21:43.871008+0000","last_undegraded":"2024-04-02T13:21:43.871008+0000","last_fullsized":"2024-04-02T13:21:43.871008+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:19:32.445256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888917+0000","last_change":"2024-04-02T13:21:43.887970+0000","last_active":"2024-04-02T13:21:43.888917+0000","last_peered":"2024-04-02T13:21:43.888917+0000","last_clean":"2024-04-02T13:21:43.888917+0000","last_became_active":"2024-04-02T13:21:43.887632+0000","last_became_peered":"2024-04-02T13:21:43.887632+0000","last_unstale":"2024-04-02T13:21:43.888917+0000","last_undegraded":"2024-04-02T13:21:43.888917+0000","last_fullsized":"2024-04-02T13:21:43.888917+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:22.467118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.869418+0000","last_change":"2024-04-02T13:21:42.869673+0000","last_active":"2024-04-02T13:21:43.869418+0000","last_peered":"2024-04-02T13:21:43.869418+0000","last_clean":"2024-04-02T13:21:43.869418+0000","last_became_active":"2024-04-02T13:21:42.869401+0000","last_became_peered":"2024-04-02T13:21:42.869401+0000","last_unstale":"2024-04-02T13:21:43.869418+0000","last_undegraded":"2024-04-02T13:21:43.869418+0000","last_fullsized":"2024-04-02T13:21:43.869418+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:48.112835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":202,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.869832+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:21:43.869832+0000","last_peered":"2024-04-02T13:21:43.869832+0000","last_clean":"2024-04-02T13:21:43.869832+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:21:43.869832+0000","last_undegraded":"2024-04-02T13:21:43.869832+0000","last_fullsized":"2024-04-02T13:21:43.869832+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1470,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":2940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37,"ondisk_log_size":37,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1019904,"data_stored":771370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343812,"statfs":{"total":96636764160,"available":96608063488,"internally_reserved":0,"allocated":1032192,"data_stored":778537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:21:47.893 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:21:48.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.028+0000 7f4c18e0d640 1 Processor -- start 2024-04-02T13:21:48.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.028+0000 7f4c18e0d640 1 -- start start 2024-04-02T13:21:48.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c18e0d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4c1415e550 0x7f4c1406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c18e0d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4c1406d530 0x7f4c1415e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c18e0d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c1415fb50 con 0x7f4c1415ee70 2024-04-02T13:21:48.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c18e0d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4c1406d970 con 0x7f4c1415e550 2024-04-02T13:21:48.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c18e0d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4c1415f300 con 0x7f4c1406d530 2024-04-02T13:21:48.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c1259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4c1415e550 0x7f4c1406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c12d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4c1415ee70 0x7f4c140ce6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33008/0 (socket says 172.21.15.44:33008) 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c1259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4c1415e550 0x7f4c1406cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53910/0 (socket says 172.21.15.44:53910) 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c12d9d640 1 -- 172.21.15.44:0/3593244172 learned_addr learned my addr 172.21.15.44:0/3593244172 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c0bfff640 1 --2- 172.21.15.44:0/3593244172 >> v2:172.21.15.69:3300/0 conn(0x7f4c1406d530 0x7f4c1415e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c1259c640 1 -- 172.21.15.44:0/3593244172 >> v2:172.21.15.69:3300/0 conn(0x7f4c1406d530 msgr2=0x7f4c1415e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c1259c640 1 --2- 172.21.15.44:0/3593244172 >> v2:172.21.15.69:3300/0 conn(0x7f4c1406d530 0x7f4c1415e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c1259c640 1 -- 172.21.15.44:0/3593244172 >> v1:172.21.15.44:6789/0 conn(0x7f4c1415ee70 legacy=0x7f4c140ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c11d9b640 1 -- 172.21.15.44:0/3593244172 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 927228990 0 0) 0x7f4c1415fb50 con 0x7f4c1415ee70 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.029+0000 7f4c1259c640 1 -- 172.21.15.44:0/3593244172 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c140d1ac0 con 0x7f4c1415e550 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.030+0000 7f4c1259c640 1 --2- 172.21.15.44:0/3593244172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c1415e550 0x7f4c1406cff0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4c140b4a20 tx=0x7f4c0402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b41470b683a1b7e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.030+0000 7f4c11d9b640 1 -- 172.21.15.44:0/3593244172 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c04034070 con 0x7f4c1415e550 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.030+0000 7f4c11d9b640 1 -- 172.21.15.44:0/3593244172 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c04030610 con 0x7f4c1415e550 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.030+0000 7f4c11d9b640 1 -- 172.21.15.44:0/3593244172 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c040309e0 con 0x7f4c1415e550 2024-04-02T13:21:48.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.031+0000 7f4c18e0d640 1 -- 172.21.15.44:0/3593244172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c1415e550 msgr2=0x7f4c1406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.031+0000 7f4c18e0d640 1 --2- 172.21.15.44:0/3593244172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c1415e550 0x7f4c1406cff0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4c140b4a20 tx=0x7f4c0402b960 comp rx=0 tx=0).stop 2024-04-02T13:21:48.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.031+0000 7f4c18e0d640 1 -- 172.21.15.44:0/3593244172 shutdown_connections 2024-04-02T13:21:48.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.031+0000 7f4c18e0d640 1 --2- 172.21.15.44:0/3593244172 >> v2:172.21.15.69:3300/0 conn(0x7f4c1406d530 0x7f4c1415e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.031+0000 7f4c18e0d640 1 --2- 172.21.15.44:0/3593244172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c1415e550 0x7f4c1406cff0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.031+0000 7f4c18e0d640 1 -- 172.21.15.44:0/3593244172 >> 172.21.15.44:0/3593244172 conn(0x7f4c1409aa20 msgr2=0x7f4c1409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:48.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.031+0000 7f4c18e0d640 1 -- 172.21.15.44:0/3593244172 shutdown_connections 2024-04-02T13:21:48.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.031+0000 7f4c18e0d640 1 -- 172.21.15.44:0/3593244172 wait complete. 2024-04-02T13:21:48.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.032+0000 7f4c18e0d640 1 Processor -- start 2024-04-02T13:21:48.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.032+0000 7f4c18e0d640 1 -- start start 2024-04-02T13:21:48.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c18e0d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c1406d530 0x7f4c140b73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c18e0d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c1415e550 0x7f4c140cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c18e0d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c1415ee70 0x7f4c1414b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c18e0d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1415f870 con 0x7f4c1415e550 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c18e0d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1415f590 con 0x7f4c1406d530 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c18e0d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4c140d02b0 con 0x7f4c1415ee70 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c0bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c1415e550 0x7f4c140cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c0bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c1415e550 0x7f4c140cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53918/0 (socket says 172.21.15.44:53918) 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c0bfff640 1 -- 172.21.15.44:0/2190966194 learned_addr learned my addr 172.21.15.44:0/2190966194 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c12d9d640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c1415ee70 0x7f4c1414b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c1259c640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c1406d530 0x7f4c140b73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c0bfff640 1 -- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c1415ee70 msgr2=0x7f4c1414b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c0bfff640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c1415ee70 0x7f4c1414b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c0bfff640 1 -- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c1406d530 msgr2=0x7f4c140b73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c0bfff640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c1406d530 0x7f4c140b73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c0bfff640 1 -- 172.21.15.44:0/2190966194 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c140b7920 con 0x7f4c1415e550 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c1259c640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c1406d530 0x7f4c140b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c0bfff640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c1415e550 0x7f4c140cb3e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00c820 tx=0x7f4bfc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:48.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c0affd640 1 -- 172.21.15.44:0/2190966194 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc015070 con 0x7f4c1415e550 2024-04-02T13:21:48.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c0affd640 1 -- 172.21.15.44:0/2190966194 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bfc004020 con 0x7f4c1415e550 2024-04-02T13:21:48.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.033+0000 7f4c0affd640 1 -- 172.21.15.44:0/2190966194 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc0043f0 con 0x7f4c1415e550 2024-04-02T13:21:48.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.034+0000 7f4c18e0d640 1 -- 172.21.15.44:0/2190966194 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c140ca940 con 0x7f4c1415e550 2024-04-02T13:21:48.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.034+0000 7f4c18e0d640 1 -- 172.21.15.44:0/2190966194 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c14157360 con 0x7f4c1415e550 2024-04-02T13:21:48.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.035+0000 7f4c18e0d640 1 -- 172.21.15.44:0/2190966194 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c140684f0 con 0x7f4c1415e550 2024-04-02T13:21:48.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.035+0000 7f4c0affd640 1 -- 172.21.15.44:0/2190966194 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4bfc007830 con 0x7f4c1415e550 2024-04-02T13:21:48.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.035+0000 7f4c0affd640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4bf406c920 0x7f4bf406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.035+0000 7f4c0affd640 1 -- 172.21.15.44:0/2190966194 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f4bfc08bae0 con 0x7f4c1415e550 2024-04-02T13:21:48.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.036+0000 7f4c1259c640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4bf406c920 0x7f4bf406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.036+0000 7f4c1259c640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4bf406c920 0x7f4bf406ede0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4c04036000 tx=0x7f4c04008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:48.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.040+0000 7f4c0affd640 1 -- 172.21.15.44:0/2190966194 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4bfc059cf0 con 0x7f4c1415e550 2024-04-02T13:21:48.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.220+0000 7f4c18e0d640 1 -- 172.21.15.44:0/2190966194 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4c140d04f0 con 0x7f4bf406c920 2024-04-02T13:21:48.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.226+0000 7f4c0affd640 1 -- 172.21.15.44:0/2190966194 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251882 (secure 0 0 0) 0x7f4c140d04f0 con 0x7f4bf406c920 2024-04-02T13:21:48.231 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:48.232 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:21:48.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.229+0000 7f4c18e0d640 1 -- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4bf406c920 msgr2=0x7f4bf406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.229+0000 7f4c18e0d640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4bf406c920 0x7f4bf406ede0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4c04036000 tx=0x7f4c04008030 comp rx=0 tx=0).stop 2024-04-02T13:21:48.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.229+0000 7f4c18e0d640 1 -- 172.21.15.44:0/2190966194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c1415e550 msgr2=0x7f4c140cb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.229+0000 7f4c18e0d640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c1415e550 0x7f4c140cb3e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00c820 tx=0x7f4bfc00cce0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.229+0000 7f4c18e0d640 1 -- 172.21.15.44:0/2190966194 shutdown_connections 2024-04-02T13:21:48.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.229+0000 7f4c18e0d640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4bf406c920 0x7f4bf406ede0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.229+0000 7f4c18e0d640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c1415ee70 0x7f4c1414b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.229+0000 7f4c18e0d640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c1415e550 0x7f4c140cb3e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.229+0000 7f4c18e0d640 1 --2- 172.21.15.44:0/2190966194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c1406d530 0x7f4c140b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.229+0000 7f4c18e0d640 1 -- 172.21.15.44:0/2190966194 >> 172.21.15.44:0/2190966194 conn(0x7f4c1409aa20 msgr2=0x7f4c140cda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:48.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.229+0000 7f4c18e0d640 1 -- 172.21.15.44:0/2190966194 shutdown_connections 2024-04-02T13:21:48.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.229+0000 7f4c18e0d640 1 -- 172.21.15.44:0/2190966194 wait complete. 2024-04-02T13:21:48.242 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-04-02T13:21:46.893405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591838,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":202,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220664,"kb_used_data":4500,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754056,"statfs":{"total":773094113280,"available":772868153344,"internally_reserved":0,"allocated":4608000,"data_stored":2635919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1470,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.045243"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887540+0000","last_change":"2024-04-02T13:21:43.883508+0000","last_active":"2024-04-02T13:21:43.887540+0000","last_peered":"2024-04-02T13:21:43.887540+0000","last_clean":"2024-04-02T13:21:43.887540+0000","last_became_active":"2024-04-02T13:21:43.883200+0000","last_became_peered":"2024-04-02T13:21:43.883200+0000","last_unstale":"2024-04-02T13:21:43.887540+0000","last_undegraded":"2024-04-02T13:21:43.887540+0000","last_fullsized":"2024-04-02T13:21:43.887540+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:22.467118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887693+0000","last_change":"2024-04-02T13:21:43.884004+0000","last_active":"2024-04-02T13:21:43.887693+0000","last_peered":"2024-04-02T13:21:43.887693+0000","last_clean":"2024-04-02T13:21:43.887693+0000","last_became_active":"2024-04-02T13:21:43.883760+0000","last_became_peered":"2024-04-02T13:21:43.883760+0000","last_unstale":"2024-04-02T13:21:43.887693+0000","last_undegraded":"2024-04-02T13:21:43.887693+0000","last_fullsized":"2024-04-02T13:21:43.887693+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:33.147235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897360+0000","last_change":"2024-04-02T13:21:43.889721+0000","last_active":"2024-04-02T13:21:43.897360+0000","last_peered":"2024-04-02T13:21:43.897360+0000","last_clean":"2024-04-02T13:21:43.897360+0000","last_became_active":"2024-04-02T13:21:43.889139+0000","last_became_peered":"2024-04-02T13:21:43.889139+0000","last_unstale":"2024-04-02T13:21:43.897360+0000","last_undegraded":"2024-04-02T13:21:43.897360+0000","last_fullsized":"2024-04-02T13:21:43.897360+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:42:32.884966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.896070+0000","last_change":"2024-04-02T13:21:43.891625+0000","last_active":"2024-04-02T13:21:43.896070+0000","last_peered":"2024-04-02T13:21:43.896070+0000","last_clean":"2024-04-02T13:21:43.896070+0000","last_became_active":"2024-04-02T13:21:43.890952+0000","last_became_peered":"2024-04-02T13:21:43.890952+0000","last_unstale":"2024-04-02T13:21:43.896070+0000","last_undegraded":"2024-04-02T13:21:43.896070+0000","last_fullsized":"2024-04-02T13:21:43.896070+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:26.646577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.893389+0000","last_change":"2024-04-02T13:21:43.890367+0000","last_active":"2024-04-02T13:21:43.893389+0000","last_peered":"2024-04-02T13:21:43.893389+0000","last_clean":"2024-04-02T13:21:43.893389+0000","last_became_active":"2024-04-02T13:21:43.890087+0000","last_became_peered":"2024-04-02T13:21:43.890087+0000","last_unstale":"2024-04-02T13:21:43.893389+0000","last_undegraded":"2024-04-02T13:21:43.893389+0000","last_fullsized":"2024-04-02T13:21:43.893389+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:20.667849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.893989+0000","last_change":"2024-04-02T13:21:43.888470+0000","last_active":"2024-04-02T13:21:43.893989+0000","last_peered":"2024-04-02T13:21:43.893989+0000","last_clean":"2024-04-02T13:21:43.893989+0000","last_became_active":"2024-04-02T13:21:43.888141+0000","last_became_peered":"2024-04-02T13:21:43.888141+0000","last_unstale":"2024-04-02T13:21:43.893989+0000","last_undegraded":"2024-04-02T13:21:43.893989+0000","last_fullsized":"2024-04-02T13:21:43.893989+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:10.725145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888968+0000","last_change":"2024-04-02T13:21:43.888055+0000","last_active":"2024-04-02T13:21:43.888968+0000","last_peered":"2024-04-02T13:21:43.888968+0000","last_clean":"2024-04-02T13:21:43.888968+0000","last_became_active":"2024-04-02T13:21:43.887783+0000","last_became_peered":"2024-04-02T13:21:43.887783+0000","last_unstale":"2024-04-02T13:21:43.888968+0000","last_undegraded":"2024-04-02T13:21:43.888968+0000","last_fullsized":"2024-04-02T13:21:43.888968+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:33.147235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886428+0000","last_change":"2024-04-02T13:21:43.877806+0000","last_active":"2024-04-02T13:21:43.886428+0000","last_peered":"2024-04-02T13:21:43.886428+0000","last_clean":"2024-04-02T13:21:43.886428+0000","last_became_active":"2024-04-02T13:21:43.877524+0000","last_became_peered":"2024-04-02T13:21:43.877524+0000","last_unstale":"2024-04-02T13:21:43.886428+0000","last_undegraded":"2024-04-02T13:21:43.886428+0000","last_fullsized":"2024-04-02T13:21:43.886428+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:40:40.996848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897209+0000","last_change":"2024-04-02T13:21:43.889363+0000","last_active":"2024-04-02T13:21:43.897209+0000","last_peered":"2024-04-02T13:21:43.897209+0000","last_clean":"2024-04-02T13:21:43.897209+0000","last_became_active":"2024-04-02T13:21:43.889089+0000","last_became_peered":"2024-04-02T13:21:43.889089+0000","last_unstale":"2024-04-02T13:21:43.897209+0000","last_undegraded":"2024-04-02T13:21:43.897209+0000","last_fullsized":"2024-04-02T13:21:43.897209+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:34.646859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.877502+0000","last_change":"2024-04-02T13:21:43.873747+0000","last_active":"2024-04-02T13:21:43.877502+0000","last_peered":"2024-04-02T13:21:43.877502+0000","last_clean":"2024-04-02T13:21:43.877502+0000","last_became_active":"2024-04-02T13:21:43.872056+0000","last_became_peered":"2024-04-02T13:21:43.872056+0000","last_unstale":"2024-04-02T13:21:43.877502+0000","last_undegraded":"2024-04-02T13:21:43.877502+0000","last_fullsized":"2024-04-02T13:21:43.877502+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:56:22.065177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888414+0000","last_change":"2024-04-02T13:21:43.883290+0000","last_active":"2024-04-02T13:21:43.888414+0000","last_peered":"2024-04-02T13:21:43.888414+0000","last_clean":"2024-04-02T13:21:43.888414+0000","last_became_active":"2024-04-02T13:21:43.882955+0000","last_became_peered":"2024-04-02T13:21:43.882955+0000","last_unstale":"2024-04-02T13:21:43.888414+0000","last_undegraded":"2024-04-02T13:21:43.888414+0000","last_fullsized":"2024-04-02T13:21:43.888414+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:58.834076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897446+0000","last_change":"2024-04-02T13:21:43.894628+0000","last_active":"2024-04-02T13:21:43.897446+0000","last_peered":"2024-04-02T13:21:43.897446+0000","last_clean":"2024-04-02T13:21:43.897446+0000","last_became_active":"2024-04-02T13:21:43.894282+0000","last_became_peered":"2024-04-02T13:21:43.894282+0000","last_unstale":"2024-04-02T13:21:43.897446+0000","last_undegraded":"2024-04-02T13:21:43.897446+0000","last_fullsized":"2024-04-02T13:21:43.897446+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:43:09.586928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.876606+0000","last_change":"2024-04-02T13:21:43.872575+0000","last_active":"2024-04-02T13:21:43.876606+0000","last_peered":"2024-04-02T13:21:43.876606+0000","last_clean":"2024-04-02T13:21:43.876606+0000","last_became_active":"2024-04-02T13:21:43.872316+0000","last_became_peered":"2024-04-02T13:21:43.872316+0000","last_unstale":"2024-04-02T13:21:43.876606+0000","last_undegraded":"2024-04-02T13:21:43.876606+0000","last_fullsized":"2024-04-02T13:21:43.876606+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:43:09.586928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891124+0000","last_change":"2024-04-02T13:21:43.885777+0000","last_active":"2024-04-02T13:21:43.891124+0000","last_peered":"2024-04-02T13:21:43.891124+0000","last_clean":"2024-04-02T13:21:43.891124+0000","last_became_active":"2024-04-02T13:21:43.885024+0000","last_became_peered":"2024-04-02T13:21:43.885024+0000","last_unstale":"2024-04-02T13:21:43.891124+0000","last_undegraded":"2024-04-02T13:21:43.891124+0000","last_fullsized":"2024-04-02T13:21:43.891124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:49.117575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886051+0000","last_change":"2024-04-02T13:21:43.877278+0000","last_active":"2024-04-02T13:21:43.886051+0000","last_peered":"2024-04-02T13:21:43.886051+0000","last_clean":"2024-04-02T13:21:43.886051+0000","last_became_active":"2024-04-02T13:21:43.876746+0000","last_became_peered":"2024-04-02T13:21:43.876746+0000","last_unstale":"2024-04-02T13:21:43.886051+0000","last_undegraded":"2024-04-02T13:21:43.886051+0000","last_fullsized":"2024-04-02T13:21:43.886051+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:13.466624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886640+0000","last_change":"2024-04-02T13:21:43.877474+0000","last_active":"2024-04-02T13:21:43.886640+0000","last_peered":"2024-04-02T13:21:43.886640+0000","last_clean":"2024-04-02T13:21:43.886640+0000","last_became_active":"2024-04-02T13:21:43.877199+0000","last_became_peered":"2024-04-02T13:21:43.877199+0000","last_unstale":"2024-04-02T13:21:43.886640+0000","last_undegraded":"2024-04-02T13:21:43.886640+0000","last_fullsized":"2024-04-02T13:21:43.886640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:43:09.586928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883542+0000","last_change":"2024-04-02T13:21:43.879759+0000","last_active":"2024-04-02T13:21:43.883542+0000","last_peered":"2024-04-02T13:21:43.883542+0000","last_clean":"2024-04-02T13:21:43.883542+0000","last_became_active":"2024-04-02T13:21:43.879462+0000","last_became_peered":"2024-04-02T13:21:43.879462+0000","last_unstale":"2024-04-02T13:21:43.883542+0000","last_undegraded":"2024-04-02T13:21:43.883542+0000","last_fullsized":"2024-04-02T13:21:43.883542+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:56:22.065177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.884063+0000","last_change":"2024-04-02T13:21:43.881236+0000","last_active":"2024-04-02T13:21:43.884063+0000","last_peered":"2024-04-02T13:21:43.884063+0000","last_clean":"2024-04-02T13:21:43.884063+0000","last_became_active":"2024-04-02T13:21:43.880958+0000","last_became_peered":"2024-04-02T13:21:43.880958+0000","last_unstale":"2024-04-02T13:21:43.884063+0000","last_undegraded":"2024-04-02T13:21:43.884063+0000","last_fullsized":"2024-04-02T13:21:43.884063+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:05:19.791693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891796+0000","last_change":"2024-04-02T13:21:43.885592+0000","last_active":"2024-04-02T13:21:43.891796+0000","last_peered":"2024-04-02T13:21:43.891796+0000","last_clean":"2024-04-02T13:21:43.891796+0000","last_became_active":"2024-04-02T13:21:43.884169+0000","last_became_peered":"2024-04-02T13:21:43.884169+0000","last_unstale":"2024-04-02T13:21:43.891796+0000","last_undegraded":"2024-04-02T13:21:43.891796+0000","last_fullsized":"2024-04-02T13:21:43.891796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:47:16.117390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887124+0000","last_change":"2024-04-02T13:21:43.884294+0000","last_active":"2024-04-02T13:21:43.887124+0000","last_peered":"2024-04-02T13:21:43.887124+0000","last_clean":"2024-04-02T13:21:43.887124+0000","last_became_active":"2024-04-02T13:21:43.884029+0000","last_became_peered":"2024-04-02T13:21:43.884029+0000","last_unstale":"2024-04-02T13:21:43.887124+0000","last_undegraded":"2024-04-02T13:21:43.887124+0000","last_fullsized":"2024-04-02T13:21:43.887124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:38.212652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.877694+0000","last_change":"2024-04-02T13:21:43.875390+0000","last_active":"2024-04-02T13:21:43.877694+0000","last_peered":"2024-04-02T13:21:43.877694+0000","last_clean":"2024-04-02T13:21:43.877694+0000","last_became_active":"2024-04-02T13:21:43.875118+0000","last_became_peered":"2024-04-02T13:21:43.875118+0000","last_unstale":"2024-04-02T13:21:43.877694+0000","last_undegraded":"2024-04-02T13:21:43.877694+0000","last_fullsized":"2024-04-02T13:21:43.877694+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:32:03.790521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.898971+0000","last_change":"2024-04-02T13:21:43.892131+0000","last_active":"2024-04-02T13:21:43.898971+0000","last_peered":"2024-04-02T13:21:43.898971+0000","last_clean":"2024-04-02T13:21:43.898971+0000","last_became_active":"2024-04-02T13:21:43.891816+0000","last_became_peered":"2024-04-02T13:21:43.891816+0000","last_unstale":"2024-04-02T13:21:43.898971+0000","last_undegraded":"2024-04-02T13:21:43.898971+0000","last_fullsized":"2024-04-02T13:21:43.898971+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:47.913991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883161+0000","last_change":"2024-04-02T13:21:43.880541+0000","last_active":"2024-04-02T13:21:43.883161+0000","last_peered":"2024-04-02T13:21:43.883161+0000","last_clean":"2024-04-02T13:21:43.883161+0000","last_became_active":"2024-04-02T13:21:43.879795+0000","last_became_peered":"2024-04-02T13:21:43.879795+0000","last_unstale":"2024-04-02T13:21:43.883161+0000","last_undegraded":"2024-04-02T13:21:43.883161+0000","last_fullsized":"2024-04-02T13:21:43.883161+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:46.012704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.890753+0000","last_change":"2024-04-02T13:21:43.881876+0000","last_active":"2024-04-02T13:21:43.890753+0000","last_peered":"2024-04-02T13:21:43.890753+0000","last_clean":"2024-04-02T13:21:43.890753+0000","last_became_active":"2024-04-02T13:21:43.881579+0000","last_became_peered":"2024-04-02T13:21:43.881579+0000","last_unstale":"2024-04-02T13:21:43.890753+0000","last_undegraded":"2024-04-02T13:21:43.890753+0000","last_fullsized":"2024-04-02T13:21:43.890753+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:26:56.782064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886741+0000","last_change":"2024-04-02T13:21:43.882505+0000","last_active":"2024-04-02T13:21:43.886741+0000","last_peered":"2024-04-02T13:21:43.886741+0000","last_clean":"2024-04-02T13:21:43.886741+0000","last_became_active":"2024-04-02T13:21:43.882124+0000","last_became_peered":"2024-04-02T13:21:43.882124+0000","last_unstale":"2024-04-02T13:21:43.886741+0000","last_undegraded":"2024-04-02T13:21:43.886741+0000","last_fullsized":"2024-04-02T13:21:43.886741+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:50:26.272689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887953+0000","last_change":"2024-04-02T13:21:43.880686+0000","last_active":"2024-04-02T13:21:43.887953+0000","last_peered":"2024-04-02T13:21:43.887953+0000","last_clean":"2024-04-02T13:21:43.887953+0000","last_became_active":"2024-04-02T13:21:43.880399+0000","last_became_peered":"2024-04-02T13:21:43.880399+0000","last_unstale":"2024-04-02T13:21:43.887953+0000","last_undegraded":"2024-04-02T13:21:43.887953+0000","last_fullsized":"2024-04-02T13:21:43.887953+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:34.646859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.884062+0000","last_change":"2024-04-02T13:21:43.878844+0000","last_active":"2024-04-02T13:21:43.884062+0000","last_peered":"2024-04-02T13:21:43.884062+0000","last_clean":"2024-04-02T13:21:43.884062+0000","last_became_active":"2024-04-02T13:21:43.878605+0000","last_became_peered":"2024-04-02T13:21:43.878605+0000","last_unstale":"2024-04-02T13:21:43.884062+0000","last_undegraded":"2024-04-02T13:21:43.884062+0000","last_fullsized":"2024-04-02T13:21:43.884062+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:26.646577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888942+0000","last_change":"2024-04-02T13:21:43.888058+0000","last_active":"2024-04-02T13:21:43.888942+0000","last_peered":"2024-04-02T13:21:43.888942+0000","last_clean":"2024-04-02T13:21:43.888942+0000","last_became_active":"2024-04-02T13:21:43.887696+0000","last_became_peered":"2024-04-02T13:21:43.887696+0000","last_unstale":"2024-04-02T13:21:43.888942+0000","last_undegraded":"2024-04-02T13:21:43.888942+0000","last_fullsized":"2024-04-02T13:21:43.888942+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:48:14.205602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883665+0000","last_change":"2024-04-02T13:21:43.878318+0000","last_active":"2024-04-02T13:21:43.883665+0000","last_peered":"2024-04-02T13:21:43.883665+0000","last_clean":"2024-04-02T13:21:43.883665+0000","last_became_active":"2024-04-02T13:21:43.878035+0000","last_became_peered":"2024-04-02T13:21:43.878035+0000","last_unstale":"2024-04-02T13:21:43.883665+0000","last_undegraded":"2024-04-02T13:21:43.883665+0000","last_fullsized":"2024-04-02T13:21:43.883665+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:20.667849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888857+0000","last_change":"2024-04-02T13:21:43.885523+0000","last_active":"2024-04-02T13:21:43.888857+0000","last_peered":"2024-04-02T13:21:43.888857+0000","last_clean":"2024-04-02T13:21:43.888857+0000","last_became_active":"2024-04-02T13:21:43.884744+0000","last_became_peered":"2024-04-02T13:21:43.884744+0000","last_unstale":"2024-04-02T13:21:43.888857+0000","last_undegraded":"2024-04-02T13:21:43.888857+0000","last_fullsized":"2024-04-02T13:21:43.888857+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:15:37.587150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.877210+0000","last_change":"2024-04-02T13:21:43.875957+0000","last_active":"2024-04-02T13:21:43.877210+0000","last_peered":"2024-04-02T13:21:43.877210+0000","last_clean":"2024-04-02T13:21:43.877210+0000","last_became_active":"2024-04-02T13:21:43.875630+0000","last_became_peered":"2024-04-02T13:21:43.875630+0000","last_unstale":"2024-04-02T13:21:43.877210+0000","last_undegraded":"2024-04-02T13:21:43.877210+0000","last_fullsized":"2024-04-02T13:21:43.877210+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:47.913991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883756+0000","last_change":"2024-04-02T13:21:43.878610+0000","last_active":"2024-04-02T13:21:43.883756+0000","last_peered":"2024-04-02T13:21:43.883756+0000","last_clean":"2024-04-02T13:21:43.883756+0000","last_became_active":"2024-04-02T13:21:43.878156+0000","last_became_peered":"2024-04-02T13:21:43.878156+0000","last_unstale":"2024-04-02T13:21:43.883756+0000","last_undegraded":"2024-04-02T13:21:43.883756+0000","last_fullsized":"2024-04-02T13:21:43.883756+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:49.117575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883205+0000","last_change":"2024-04-02T13:21:43.880230+0000","last_active":"2024-04-02T13:21:43.883205+0000","last_peered":"2024-04-02T13:21:43.883205+0000","last_clean":"2024-04-02T13:21:43.883205+0000","last_became_active":"2024-04-02T13:21:43.879948+0000","last_became_peered":"2024-04-02T13:21:43.879948+0000","last_unstale":"2024-04-02T13:21:43.883205+0000","last_undegraded":"2024-04-02T13:21:43.883205+0000","last_fullsized":"2024-04-02T13:21:43.883205+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:19:33.449996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.896632+0000","last_change":"2024-04-02T13:21:42.870063+0000","last_active":"2024-04-02T13:21:43.896632+0000","last_peered":"2024-04-02T13:21:43.896632+0000","last_clean":"2024-04-02T13:21:43.896632+0000","last_became_active":"2024-04-02T13:21:42.869787+0000","last_became_peered":"2024-04-02T13:21:42.869787+0000","last_unstale":"2024-04-02T13:21:43.896632+0000","last_undegraded":"2024-04-02T13:21:43.896632+0000","last_fullsized":"2024-04-02T13:21:43.896632+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:35.866655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888088+0000","last_change":"2024-04-02T13:21:43.882527+0000","last_active":"2024-04-02T13:21:43.888088+0000","last_peered":"2024-04-02T13:21:43.888088+0000","last_clean":"2024-04-02T13:21:43.888088+0000","last_became_active":"2024-04-02T13:21:43.882237+0000","last_became_peered":"2024-04-02T13:21:43.882237+0000","last_unstale":"2024-04-02T13:21:43.888088+0000","last_undegraded":"2024-04-02T13:21:43.888088+0000","last_fullsized":"2024-04-02T13:21:43.888088+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:47.913991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.879336+0000","last_change":"2024-04-02T13:21:42.866397+0000","last_active":"2024-04-02T13:21:43.879336+0000","last_peered":"2024-04-02T13:21:43.879336+0000","last_clean":"2024-04-02T13:21:43.879336+0000","last_became_active":"2024-04-02T13:21:42.866129+0000","last_became_peered":"2024-04-02T13:21:42.866129+0000","last_unstale":"2024-04-02T13:21:43.879336+0000","last_undegraded":"2024-04-02T13:21:43.879336+0000","last_fullsized":"2024-04-02T13:21:43.879336+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:25.641837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888627+0000","last_change":"2024-04-02T13:21:43.885483+0000","last_active":"2024-04-02T13:21:43.888627+0000","last_peered":"2024-04-02T13:21:43.888627+0000","last_clean":"2024-04-02T13:21:43.888627+0000","last_became_active":"2024-04-02T13:21:43.885047+0000","last_became_peered":"2024-04-02T13:21:43.885047+0000","last_unstale":"2024-04-02T13:21:43.888627+0000","last_undegraded":"2024-04-02T13:21:43.888627+0000","last_fullsized":"2024-04-02T13:21:43.888627+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:15:37.587150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.872089+0000","last_change":"2024-04-02T13:21:42.863185+0000","last_active":"2024-04-02T13:21:43.872089+0000","last_peered":"2024-04-02T13:21:43.872089+0000","last_clean":"2024-04-02T13:21:43.872089+0000","last_became_active":"2024-04-02T13:21:42.862944+0000","last_became_peered":"2024-04-02T13:21:42.862944+0000","last_unstale":"2024-04-02T13:21:43.872089+0000","last_undegraded":"2024-04-02T13:21:43.872089+0000","last_fullsized":"2024-04-02T13:21:43.872089+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:37.207912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887575+0000","last_change":"2024-04-02T13:21:43.884619+0000","last_active":"2024-04-02T13:21:43.887575+0000","last_peered":"2024-04-02T13:21:43.887575+0000","last_clean":"2024-04-02T13:21:43.887575+0000","last_became_active":"2024-04-02T13:21:43.884286+0000","last_became_peered":"2024-04-02T13:21:43.884286+0000","last_unstale":"2024-04-02T13:21:43.887575+0000","last_undegraded":"2024-04-02T13:21:43.887575+0000","last_fullsized":"2024-04-02T13:21:43.887575+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:48:14.205602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.874745+0000","last_change":"2024-04-02T13:21:42.865049+0000","last_active":"2024-04-02T13:21:43.874745+0000","last_peered":"2024-04-02T13:21:43.874745+0000","last_clean":"2024-04-02T13:21:43.874745+0000","last_became_active":"2024-04-02T13:21:42.863891+0000","last_became_peered":"2024-04-02T13:21:42.863891+0000","last_unstale":"2024-04-02T13:21:43.874745+0000","last_undegraded":"2024-04-02T13:21:43.874745+0000","last_fullsized":"2024-04-02T13:21:43.874745+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:05:18.786953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897383+0000","last_change":"2024-04-02T13:21:43.890395+0000","last_active":"2024-04-02T13:21:43.897383+0000","last_peered":"2024-04-02T13:21:43.897383+0000","last_clean":"2024-04-02T13:21:43.897383+0000","last_became_active":"2024-04-02T13:21:43.890130+0000","last_became_peered":"2024-04-02T13:21:43.890130+0000","last_unstale":"2024-04-02T13:21:43.897383+0000","last_undegraded":"2024-04-02T13:21:43.897383+0000","last_fullsized":"2024-04-02T13:21:43.897383+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:47.913991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897694+0000","last_change":"2024-04-02T13:21:42.872282+0000","last_active":"2024-04-02T13:21:43.897694+0000","last_peered":"2024-04-02T13:21:43.897694+0000","last_clean":"2024-04-02T13:21:43.897694+0000","last_became_active":"2024-04-02T13:21:42.871960+0000","last_became_peered":"2024-04-02T13:21:42.871960+0000","last_unstale":"2024-04-02T13:21:43.897694+0000","last_undegraded":"2024-04-02T13:21:43.897694+0000","last_fullsized":"2024-04-02T13:21:43.897694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:33.642119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886073+0000","last_change":"2024-04-02T13:21:43.878315+0000","last_active":"2024-04-02T13:21:43.886073+0000","last_peered":"2024-04-02T13:21:43.886073+0000","last_clean":"2024-04-02T13:21:43.886073+0000","last_became_active":"2024-04-02T13:21:43.877294+0000","last_became_peered":"2024-04-02T13:21:43.877294+0000","last_unstale":"2024-04-02T13:21:43.886073+0000","last_undegraded":"2024-04-02T13:21:43.886073+0000","last_fullsized":"2024-04-02T13:21:43.886073+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:40:40.996848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.869091+0000","last_change":"2024-04-02T13:21:42.862800+0000","last_active":"2024-04-02T13:21:43.869091+0000","last_peered":"2024-04-02T13:21:43.869091+0000","last_clean":"2024-04-02T13:21:43.869091+0000","last_became_active":"2024-04-02T13:21:42.862510+0000","last_became_peered":"2024-04-02T13:21:42.862510+0000","last_unstale":"2024-04-02T13:21:43.869091+0000","last_undegraded":"2024-04-02T13:21:43.869091+0000","last_fullsized":"2024-04-02T13:21:43.869091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:15:36.582410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891145+0000","last_change":"2024-04-02T13:21:43.887181+0000","last_active":"2024-04-02T13:21:43.891145+0000","last_peered":"2024-04-02T13:21:43.891145+0000","last_clean":"2024-04-02T13:21:43.891145+0000","last_became_active":"2024-04-02T13:21:43.886877+0000","last_became_peered":"2024-04-02T13:21:43.886877+0000","last_unstale":"2024-04-02T13:21:43.891145+0000","last_undegraded":"2024-04-02T13:21:43.891145+0000","last_fullsized":"2024-04-02T13:21:43.891145+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:34.646859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871640+0000","last_change":"2024-04-02T13:21:42.871820+0000","last_active":"2024-04-02T13:21:43.871640+0000","last_peered":"2024-04-02T13:21:43.871640+0000","last_clean":"2024-04-02T13:21:43.871640+0000","last_became_active":"2024-04-02T13:21:42.871528+0000","last_became_peered":"2024-04-02T13:21:42.871528+0000","last_unstale":"2024-04-02T13:21:43.871640+0000","last_undegraded":"2024-04-02T13:21:43.871640+0000","last_fullsized":"2024-04-02T13:21:43.871640+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:05:18.786953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.895123+0000","last_change":"2024-04-02T13:21:43.885724+0000","last_active":"2024-04-02T13:21:43.895123+0000","last_peered":"2024-04-02T13:21:43.895123+0000","last_clean":"2024-04-02T13:21:43.895123+0000","last_became_active":"2024-04-02T13:21:43.885310+0000","last_became_peered":"2024-04-02T13:21:43.885310+0000","last_unstale":"2024-04-02T13:21:43.895123+0000","last_undegraded":"2024-04-02T13:21:43.895123+0000","last_fullsized":"2024-04-02T13:21:43.895123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:05:08.926884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.872563+0000","last_change":"2024-04-02T13:21:42.869469+0000","last_active":"2024-04-02T13:21:43.872563+0000","last_peered":"2024-04-02T13:21:43.872563+0000","last_clean":"2024-04-02T13:21:43.872563+0000","last_became_active":"2024-04-02T13:21:42.869003+0000","last_became_peered":"2024-04-02T13:21:42.869003+0000","last_unstale":"2024-04-02T13:21:43.872563+0000","last_undegraded":"2024-04-02T13:21:43.872563+0000","last_fullsized":"2024-04-02T13:21:43.872563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:19.663109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891315+0000","last_change":"2024-04-02T13:21:43.889932+0000","last_active":"2024-04-02T13:21:43.891315+0000","last_peered":"2024-04-02T13:21:43.891315+0000","last_clean":"2024-04-02T13:21:43.891315+0000","last_became_active":"2024-04-02T13:21:43.889649+0000","last_became_peered":"2024-04-02T13:21:43.889649+0000","last_unstale":"2024-04-02T13:21:43.891315+0000","last_undegraded":"2024-04-02T13:21:43.891315+0000","last_fullsized":"2024-04-02T13:21:43.891315+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:36.871395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.872889+0000","last_change":"2024-04-02T13:21:42.861003+0000","last_active":"2024-04-02T13:21:43.872889+0000","last_peered":"2024-04-02T13:21:43.872889+0000","last_clean":"2024-04-02T13:21:43.872889+0000","last_became_active":"2024-04-02T13:21:42.860721+0000","last_became_peered":"2024-04-02T13:21:42.860721+0000","last_unstale":"2024-04-02T13:21:43.872889+0000","last_undegraded":"2024-04-02T13:21:43.872889+0000","last_fullsized":"2024-04-02T13:21:43.872889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:45.007964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883769+0000","last_change":"2024-04-02T13:21:43.880770+0000","last_active":"2024-04-02T13:21:43.883769+0000","last_peered":"2024-04-02T13:21:43.883769+0000","last_clean":"2024-04-02T13:21:43.883769+0000","last_became_active":"2024-04-02T13:21:43.880237+0000","last_became_peered":"2024-04-02T13:21:43.880237+0000","last_unstale":"2024-04-02T13:21:43.883769+0000","last_undegraded":"2024-04-02T13:21:43.883769+0000","last_fullsized":"2024-04-02T13:21:43.883769+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:13.466624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.866440+0000","last_change":"2024-04-02T13:21:42.864488+0000","last_active":"2024-04-02T13:21:43.866440+0000","last_peered":"2024-04-02T13:21:43.866440+0000","last_clean":"2024-04-02T13:21:43.866440+0000","last_became_active":"2024-04-02T13:21:42.864203+0000","last_became_peered":"2024-04-02T13:21:42.864203+0000","last_unstale":"2024-04-02T13:21:43.866440+0000","last_undegraded":"2024-04-02T13:21:43.866440+0000","last_fullsized":"2024-04-02T13:21:43.866440+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:48.112835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886807+0000","last_change":"2024-04-02T13:21:43.884306+0000","last_active":"2024-04-02T13:21:43.886807+0000","last_peered":"2024-04-02T13:21:43.886807+0000","last_clean":"2024-04-02T13:21:43.886807+0000","last_became_active":"2024-04-02T13:21:43.883955+0000","last_became_peered":"2024-04-02T13:21:43.883955+0000","last_unstale":"2024-04-02T13:21:43.886807+0000","last_undegraded":"2024-04-02T13:21:43.886807+0000","last_fullsized":"2024-04-02T13:21:43.886807+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:38.212652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.899186+0000","last_change":"2024-04-02T13:21:42.870155+0000","last_active":"2024-04-02T13:21:43.899186+0000","last_peered":"2024-04-02T13:21:43.899186+0000","last_clean":"2024-04-02T13:21:43.899186+0000","last_became_active":"2024-04-02T13:21:42.869902+0000","last_became_peered":"2024-04-02T13:21:42.869902+0000","last_unstale":"2024-04-02T13:21:43.899186+0000","last_undegraded":"2024-04-02T13:21:43.899186+0000","last_fullsized":"2024-04-02T13:21:43.899186+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:42:31.880226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":8,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.893865+0000","last_change":"2024-04-02T13:21:43.893865+0000","last_active":"2024-04-02T13:21:43.893865+0000","last_peered":"2024-04-02T13:21:43.893865+0000","last_clean":"2024-04-02T13:21:43.893865+0000","last_became_active":"2024-04-02T13:21:43.893601+0000","last_became_peered":"2024-04-02T13:21:43.893601+0000","last_unstale":"2024-04-02T13:21:43.893865+0000","last_undegraded":"2024-04-02T13:21:43.893865+0000","last_fullsized":"2024-04-02T13:21:43.893865+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:10.725145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'5","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.874107+0000","last_change":"2024-04-02T13:21:42.863541+0000","last_active":"2024-04-02T13:21:43.874107+0000","last_peered":"2024-04-02T13:21:43.874107+0000","last_clean":"2024-04-02T13:21:43.874107+0000","last_became_active":"2024-04-02T13:21:42.863081+0000","last_became_peered":"2024-04-02T13:21:42.863081+0000","last_unstale":"2024-04-02T13:21:43.874107+0000","last_undegraded":"2024-04-02T13:21:43.874107+0000","last_fullsized":"2024-04-02T13:21:43.874107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:12.461884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897413+0000","last_change":"2024-04-02T13:21:43.892311+0000","last_active":"2024-04-02T13:21:43.897413+0000","last_peered":"2024-04-02T13:21:43.897413+0000","last_clean":"2024-04-02T13:21:43.897413+0000","last_became_active":"2024-04-02T13:21:43.889327+0000","last_became_peered":"2024-04-02T13:21:43.889327+0000","last_unstale":"2024-04-02T13:21:43.897413+0000","last_undegraded":"2024-04-02T13:21:43.897413+0000","last_fullsized":"2024-04-02T13:21:43.897413+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:10.725145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.872151+0000","last_change":"2024-04-02T13:21:42.868191+0000","last_active":"2024-04-02T13:21:43.872151+0000","last_peered":"2024-04-02T13:21:43.872151+0000","last_clean":"2024-04-02T13:21:43.872151+0000","last_became_active":"2024-04-02T13:21:42.867897+0000","last_became_peered":"2024-04-02T13:21:42.867897+0000","last_unstale":"2024-04-02T13:21:43.872151+0000","last_undegraded":"2024-04-02T13:21:43.872151+0000","last_fullsized":"2024-04-02T13:21:43.872151+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:32:02.785781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.895101+0000","last_change":"2024-04-02T13:21:43.885167+0000","last_active":"2024-04-02T13:21:43.895101+0000","last_peered":"2024-04-02T13:21:43.895101+0000","last_clean":"2024-04-02T13:21:43.895101+0000","last_became_active":"2024-04-02T13:21:43.884844+0000","last_became_peered":"2024-04-02T13:21:43.884844+0000","last_unstale":"2024-04-02T13:21:43.895101+0000","last_undegraded":"2024-04-02T13:21:43.895101+0000","last_fullsized":"2024-04-02T13:21:43.895101+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:10:20.429411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871825+0000","last_change":"2024-04-02T13:21:42.872412+0000","last_active":"2024-04-02T13:21:43.871825+0000","last_peered":"2024-04-02T13:21:43.871825+0000","last_clean":"2024-04-02T13:21:43.871825+0000","last_became_active":"2024-04-02T13:21:42.872160+0000","last_became_peered":"2024-04-02T13:21:42.872160+0000","last_unstale":"2024-04-02T13:21:43.871825+0000","last_undegraded":"2024-04-02T13:21:43.871825+0000","last_fullsized":"2024-04-02T13:21:43.871825+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:50:25.267949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.878071+0000","last_change":"2024-04-02T13:21:43.876405+0000","last_active":"2024-04-02T13:21:43.878071+0000","last_peered":"2024-04-02T13:21:43.878071+0000","last_clean":"2024-04-02T13:21:43.878071+0000","last_became_active":"2024-04-02T13:21:43.875443+0000","last_became_peered":"2024-04-02T13:21:43.875443+0000","last_unstale":"2024-04-02T13:21:43.878071+0000","last_undegraded":"2024-04-02T13:21:43.878071+0000","last_fullsized":"2024-04-02T13:21:43.878071+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:10.725145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.865679+0000","last_change":"2024-04-02T13:21:42.870037+0000","last_active":"2024-04-02T13:21:43.865679+0000","last_peered":"2024-04-02T13:21:43.865679+0000","last_clean":"2024-04-02T13:21:43.865679+0000","last_became_active":"2024-04-02T13:21:42.869753+0000","last_became_peered":"2024-04-02T13:21:42.869753+0000","last_unstale":"2024-04-02T13:21:43.865679+0000","last_undegraded":"2024-04-02T13:21:43.865679+0000","last_fullsized":"2024-04-02T13:21:43.865679+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:47:15.112650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886820+0000","last_change":"2024-04-02T13:21:43.883404+0000","last_active":"2024-04-02T13:21:43.886820+0000","last_peered":"2024-04-02T13:21:43.886820+0000","last_clean":"2024-04-02T13:21:43.886820+0000","last_became_active":"2024-04-02T13:21:43.882899+0000","last_became_peered":"2024-04-02T13:21:43.882899+0000","last_unstale":"2024-04-02T13:21:43.886820+0000","last_undegraded":"2024-04-02T13:21:43.886820+0000","last_fullsized":"2024-04-02T13:21:43.886820+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:58.834076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.896382+0000","last_change":"2024-04-02T13:21:42.863744+0000","last_active":"2024-04-02T13:21:43.896382+0000","last_peered":"2024-04-02T13:21:43.896382+0000","last_clean":"2024-04-02T13:21:43.896382+0000","last_became_active":"2024-04-02T13:21:42.863198+0000","last_became_peered":"2024-04-02T13:21:42.863198+0000","last_unstale":"2024-04-02T13:21:43.896382+0000","last_undegraded":"2024-04-02T13:21:43.896382+0000","last_fullsized":"2024-04-02T13:21:43.896382+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:42:31.880226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887782+0000","last_change":"2024-04-02T13:21:43.884934+0000","last_active":"2024-04-02T13:21:43.887782+0000","last_peered":"2024-04-02T13:21:43.887782+0000","last_clean":"2024-04-02T13:21:43.887782+0000","last_became_active":"2024-04-02T13:21:43.884630+0000","last_became_peered":"2024-04-02T13:21:43.884630+0000","last_unstale":"2024-04-02T13:21:43.887782+0000","last_undegraded":"2024-04-02T13:21:43.887782+0000","last_fullsized":"2024-04-02T13:21:43.887782+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:42.322235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.868413+0000","last_change":"2024-04-02T13:21:42.863458+0000","last_active":"2024-04-02T13:21:43.868413+0000","last_peered":"2024-04-02T13:21:43.868413+0000","last_clean":"2024-04-02T13:21:43.868413+0000","last_became_active":"2024-04-02T13:21:42.863196+0000","last_became_peered":"2024-04-02T13:21:42.863196+0000","last_unstale":"2024-04-02T13:21:43.868413+0000","last_undegraded":"2024-04-02T13:21:43.868413+0000","last_fullsized":"2024-04-02T13:21:43.868413+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:46.909251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.876133+0000","last_change":"2024-04-02T13:21:43.874576+0000","last_active":"2024-04-02T13:21:43.876133+0000","last_peered":"2024-04-02T13:21:43.876133+0000","last_clean":"2024-04-02T13:21:43.876133+0000","last_became_active":"2024-04-02T13:21:43.874255+0000","last_became_peered":"2024-04-02T13:21:43.874255+0000","last_unstale":"2024-04-02T13:21:43.876133+0000","last_undegraded":"2024-04-02T13:21:43.876133+0000","last_fullsized":"2024-04-02T13:21:43.876133+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:19:33.449996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.874853+0000","last_change":"2024-04-02T13:21:42.872412+0000","last_active":"2024-04-02T13:21:43.874853+0000","last_peered":"2024-04-02T13:21:43.874853+0000","last_clean":"2024-04-02T13:21:43.874853+0000","last_became_active":"2024-04-02T13:21:42.872147+0000","last_became_peered":"2024-04-02T13:21:42.872147+0000","last_unstale":"2024-04-02T13:21:43.874853+0000","last_undegraded":"2024-04-02T13:21:43.874853+0000","last_fullsized":"2024-04-02T13:21:43.874853+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:50:25.267949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.882605+0000","last_change":"2024-04-02T13:21:43.879554+0000","last_active":"2024-04-02T13:21:43.882605+0000","last_peered":"2024-04-02T13:21:43.882605+0000","last_clean":"2024-04-02T13:21:43.882605+0000","last_became_active":"2024-04-02T13:21:43.879279+0000","last_became_peered":"2024-04-02T13:21:43.879279+0000","last_unstale":"2024-04-02T13:21:43.882605+0000","last_undegraded":"2024-04-02T13:21:43.882605+0000","last_fullsized":"2024-04-02T13:21:43.882605+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:38.212652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.873146+0000","last_change":"2024-04-02T13:21:42.863953+0000","last_active":"2024-04-02T13:21:43.873146+0000","last_peered":"2024-04-02T13:21:43.873146+0000","last_clean":"2024-04-02T13:21:43.873146+0000","last_became_active":"2024-04-02T13:21:42.863672+0000","last_became_peered":"2024-04-02T13:21:42.863672+0000","last_unstale":"2024-04-02T13:21:43.873146+0000","last_undegraded":"2024-04-02T13:21:43.873146+0000","last_fullsized":"2024-04-02T13:21:43.873146+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:02:51.970866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891171+0000","last_change":"2024-04-02T13:21:43.887379+0000","last_active":"2024-04-02T13:21:43.891171+0000","last_peered":"2024-04-02T13:21:43.891171+0000","last_clean":"2024-04-02T13:21:43.891171+0000","last_became_active":"2024-04-02T13:21:43.887007+0000","last_became_peered":"2024-04-02T13:21:43.887007+0000","last_unstale":"2024-04-02T13:21:43.891171+0000","last_undegraded":"2024-04-02T13:21:43.891171+0000","last_fullsized":"2024-04-02T13:21:43.891171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:56:07.414303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871640+0000","last_change":"2024-04-02T13:21:42.868600+0000","last_active":"2024-04-02T13:21:43.871640+0000","last_peered":"2024-04-02T13:21:43.871640+0000","last_clean":"2024-04-02T13:21:43.871640+0000","last_became_active":"2024-04-02T13:21:42.868113+0000","last_became_peered":"2024-04-02T13:21:42.868113+0000","last_unstale":"2024-04-02T13:21:43.871640+0000","last_undegraded":"2024-04-02T13:21:43.871640+0000","last_fullsized":"2024-04-02T13:21:43.871640+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:25.641837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.886617+0000","last_change":"2024-04-02T13:21:43.882139+0000","last_active":"2024-04-02T13:21:43.886617+0000","last_peered":"2024-04-02T13:21:43.886617+0000","last_clean":"2024-04-02T13:21:43.886617+0000","last_became_active":"2024-04-02T13:21:43.881025+0000","last_became_peered":"2024-04-02T13:21:43.881025+0000","last_unstale":"2024-04-02T13:21:43.886617+0000","last_undegraded":"2024-04-02T13:21:43.886617+0000","last_fullsized":"2024-04-02T13:21:43.886617+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:58.834076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871612+0000","last_change":"2024-04-02T13:21:42.866632+0000","last_active":"2024-04-02T13:21:43.871612+0000","last_peered":"2024-04-02T13:21:43.871612+0000","last_clean":"2024-04-02T13:21:43.871612+0000","last_became_active":"2024-04-02T13:21:42.866363+0000","last_became_peered":"2024-04-02T13:21:42.866363+0000","last_unstale":"2024-04-02T13:21:43.871612+0000","last_undegraded":"2024-04-02T13:21:43.871612+0000","last_fullsized":"2024-04-02T13:21:43.871612+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:58:01.695741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891636+0000","last_change":"2024-04-02T13:21:43.885397+0000","last_active":"2024-04-02T13:21:43.891636+0000","last_peered":"2024-04-02T13:21:43.891636+0000","last_clean":"2024-04-02T13:21:43.891636+0000","last_became_active":"2024-04-02T13:21:43.885108+0000","last_became_peered":"2024-04-02T13:21:43.885108+0000","last_unstale":"2024-04-02T13:21:43.891636+0000","last_undegraded":"2024-04-02T13:21:43.891636+0000","last_fullsized":"2024-04-02T13:21:43.891636+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:15:37.587150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871715+0000","last_change":"2024-04-02T13:21:42.864454+0000","last_active":"2024-04-02T13:21:43.871715+0000","last_peered":"2024-04-02T13:21:43.871715+0000","last_clean":"2024-04-02T13:21:43.871715+0000","last_became_active":"2024-04-02T13:21:42.864126+0000","last_became_peered":"2024-04-02T13:21:42.864126+0000","last_unstale":"2024-04-02T13:21:43.871715+0000","last_undegraded":"2024-04-02T13:21:43.871715+0000","last_fullsized":"2024-04-02T13:21:43.871715+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:45:29.122779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.882589+0000","last_change":"2024-04-02T13:21:43.876666+0000","last_active":"2024-04-02T13:21:43.882589+0000","last_peered":"2024-04-02T13:21:43.882589+0000","last_clean":"2024-04-02T13:21:43.882589+0000","last_became_active":"2024-04-02T13:21:43.876405+0000","last_became_peered":"2024-04-02T13:21:43.876405+0000","last_unstale":"2024-04-02T13:21:43.882589+0000","last_undegraded":"2024-04-02T13:21:43.882589+0000","last_fullsized":"2024-04-02T13:21:43.882589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:46.012704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.869629+0000","last_change":"2024-04-02T13:21:42.867209+0000","last_active":"2024-04-02T13:21:43.869629+0000","last_peered":"2024-04-02T13:21:43.869629+0000","last_clean":"2024-04-02T13:21:43.869629+0000","last_became_active":"2024-04-02T13:21:42.866938+0000","last_became_peered":"2024-04-02T13:21:42.866938+0000","last_unstale":"2024-04-02T13:21:43.869629+0000","last_undegraded":"2024-04-02T13:21:43.869629+0000","last_fullsized":"2024-04-02T13:21:43.869629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:33.642119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.895095+0000","last_change":"2024-04-02T13:21:43.885621+0000","last_active":"2024-04-02T13:21:43.895095+0000","last_peered":"2024-04-02T13:21:43.895095+0000","last_clean":"2024-04-02T13:21:43.895095+0000","last_became_active":"2024-04-02T13:21:43.884937+0000","last_became_peered":"2024-04-02T13:21:43.884937+0000","last_unstale":"2024-04-02T13:21:43.895095+0000","last_undegraded":"2024-04-02T13:21:43.895095+0000","last_fullsized":"2024-04-02T13:21:43.895095+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:52:41.672416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871838+0000","last_change":"2024-04-02T13:21:42.865961+0000","last_active":"2024-04-02T13:21:43.871838+0000","last_peered":"2024-04-02T13:21:43.871838+0000","last_clean":"2024-04-02T13:21:43.871838+0000","last_became_active":"2024-04-02T13:21:42.865702+0000","last_became_peered":"2024-04-02T13:21:42.865702+0000","last_unstale":"2024-04-02T13:21:43.871838+0000","last_undegraded":"2024-04-02T13:21:43.871838+0000","last_fullsized":"2024-04-02T13:21:43.871838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:26:09.307417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888012+0000","last_change":"2024-04-02T13:21:43.881163+0000","last_active":"2024-04-02T13:21:43.888012+0000","last_peered":"2024-04-02T13:21:43.888012+0000","last_clean":"2024-04-02T13:21:43.888012+0000","last_became_active":"2024-04-02T13:21:43.880878+0000","last_became_peered":"2024-04-02T13:21:43.880878+0000","last_unstale":"2024-04-02T13:21:43.888012+0000","last_undegraded":"2024-04-02T13:21:43.888012+0000","last_fullsized":"2024-04-02T13:21:43.888012+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:42:32.884966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.899503+0000","last_change":"2024-04-02T13:21:42.869522+0000","last_active":"2024-04-02T13:21:43.899503+0000","last_peered":"2024-04-02T13:21:43.899503+0000","last_clean":"2024-04-02T13:21:43.899503+0000","last_became_active":"2024-04-02T13:21:42.869256+0000","last_became_peered":"2024-04-02T13:21:42.869256+0000","last_unstale":"2024-04-02T13:21:43.899503+0000","last_undegraded":"2024-04-02T13:21:43.899503+0000","last_fullsized":"2024-04-02T13:21:43.899503+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:09.720405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883790+0000","last_change":"2024-04-02T13:21:43.881002+0000","last_active":"2024-04-02T13:21:43.883790+0000","last_peered":"2024-04-02T13:21:43.883790+0000","last_clean":"2024-04-02T13:21:43.883790+0000","last_became_active":"2024-04-02T13:21:43.880726+0000","last_became_peered":"2024-04-02T13:21:43.880726+0000","last_unstale":"2024-04-02T13:21:43.883790+0000","last_undegraded":"2024-04-02T13:21:43.883790+0000","last_fullsized":"2024-04-02T13:21:43.883790+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:40:40.996848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.872459+0000","last_change":"2024-04-02T13:21:42.867407+0000","last_active":"2024-04-02T13:21:43.872459+0000","last_peered":"2024-04-02T13:21:43.872459+0000","last_clean":"2024-04-02T13:21:43.872459+0000","last_became_active":"2024-04-02T13:21:42.867155+0000","last_became_peered":"2024-04-02T13:21:42.867155+0000","last_unstale":"2024-04-02T13:21:43.872459+0000","last_undegraded":"2024-04-02T13:21:43.872459+0000","last_fullsized":"2024-04-02T13:21:43.872459+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:36:08.445273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":8,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.887540+0000","last_change":"2024-04-02T13:21:43.887540+0000","last_active":"2024-04-02T13:21:43.887540+0000","last_peered":"2024-04-02T13:21:43.887540+0000","last_clean":"2024-04-02T13:21:43.887540+0000","last_became_active":"2024-04-02T13:21:43.887198+0000","last_became_peered":"2024-04-02T13:21:43.887198+0000","last_unstale":"2024-04-02T13:21:43.887540+0000","last_undegraded":"2024-04-02T13:21:43.887540+0000","last_fullsized":"2024-04-02T13:21:43.887540+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:51:30.559674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.881397+0000","last_change":"2024-04-02T13:21:42.865779+0000","last_active":"2024-04-02T13:21:43.881397+0000","last_peered":"2024-04-02T13:21:43.881397+0000","last_clean":"2024-04-02T13:21:43.881397+0000","last_became_active":"2024-04-02T13:21:42.865523+0000","last_became_peered":"2024-04-02T13:21:42.865523+0000","last_unstale":"2024-04-02T13:21:43.881397+0000","last_undegraded":"2024-04-02T13:21:43.881397+0000","last_fullsized":"2024-04-02T13:21:43.881397+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:50:25.267949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.897184+0000","last_change":"2024-04-02T13:21:43.889026+0000","last_active":"2024-04-02T13:21:43.897184+0000","last_peered":"2024-04-02T13:21:43.897184+0000","last_clean":"2024-04-02T13:21:43.897184+0000","last_became_active":"2024-04-02T13:21:43.888729+0000","last_became_peered":"2024-04-02T13:21:43.888729+0000","last_unstale":"2024-04-02T13:21:43.897184+0000","last_undegraded":"2024-04-02T13:21:43.897184+0000","last_fullsized":"2024-04-02T13:21:43.897184+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:49.117575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.880880+0000","last_change":"2024-04-02T13:21:42.869452+0000","last_active":"2024-04-02T13:21:43.880880+0000","last_peered":"2024-04-02T13:21:43.880880+0000","last_clean":"2024-04-02T13:21:43.880880+0000","last_became_active":"2024-04-02T13:21:42.869042+0000","last_became_peered":"2024-04-02T13:21:42.869042+0000","last_unstale":"2024-04-02T13:21:43.880880+0000","last_undegraded":"2024-04-02T13:21:43.880880+0000","last_fullsized":"2024-04-02T13:21:43.880880+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:05:18.786953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.881120+0000","last_change":"2024-04-02T13:21:43.876850+0000","last_active":"2024-04-02T13:21:43.881120+0000","last_peered":"2024-04-02T13:21:43.881120+0000","last_clean":"2024-04-02T13:21:43.881120+0000","last_became_active":"2024-04-02T13:21:43.876507+0000","last_became_peered":"2024-04-02T13:21:43.876507+0000","last_unstale":"2024-04-02T13:21:43.881120+0000","last_undegraded":"2024-04-02T13:21:43.881120+0000","last_fullsized":"2024-04-02T13:21:43.881120+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:15:37.587150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.866263+0000","last_change":"2024-04-02T13:21:42.866521+0000","last_active":"2024-04-02T13:21:43.866263+0000","last_peered":"2024-04-02T13:21:43.866263+0000","last_clean":"2024-04-02T13:21:43.866263+0000","last_became_active":"2024-04-02T13:21:42.866185+0000","last_became_peered":"2024-04-02T13:21:42.866185+0000","last_unstale":"2024-04-02T13:21:43.866263+0000","last_undegraded":"2024-04-02T13:21:43.866263+0000","last_fullsized":"2024-04-02T13:21:43.866263+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:56:06.409563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.891164+0000","last_change":"2024-04-02T13:21:43.888269+0000","last_active":"2024-04-02T13:21:43.891164+0000","last_peered":"2024-04-02T13:21:43.891164+0000","last_clean":"2024-04-02T13:21:43.891164+0000","last_became_active":"2024-04-02T13:21:43.887677+0000","last_became_peered":"2024-04-02T13:21:43.887677+0000","last_unstale":"2024-04-02T13:21:43.891164+0000","last_undegraded":"2024-04-02T13:21:43.891164+0000","last_fullsized":"2024-04-02T13:21:43.891164+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:42:32.884966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.874248+0000","last_change":"2024-04-02T13:21:42.868817+0000","last_active":"2024-04-02T13:21:43.874248+0000","last_peered":"2024-04-02T13:21:43.874248+0000","last_clean":"2024-04-02T13:21:43.874248+0000","last_became_active":"2024-04-02T13:21:42.868541+0000","last_became_peered":"2024-04-02T13:21:42.868541+0000","last_unstale":"2024-04-02T13:21:43.874248+0000","last_undegraded":"2024-04-02T13:21:43.874248+0000","last_fullsized":"2024-04-02T13:21:43.874248+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:25.641837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.883617+0000","last_change":"2024-04-02T13:21:43.880463+0000","last_active":"2024-04-02T13:21:43.883617+0000","last_peered":"2024-04-02T13:21:43.883617+0000","last_clean":"2024-04-02T13:21:43.883617+0000","last_became_active":"2024-04-02T13:21:43.880220+0000","last_became_peered":"2024-04-02T13:21:43.880220+0000","last_unstale":"2024-04-02T13:21:43.883617+0000","last_undegraded":"2024-04-02T13:21:43.883617+0000","last_fullsized":"2024-04-02T13:21:43.883617+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:32:03.790521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.871008+0000","last_change":"2024-04-02T13:21:42.861463+0000","last_active":"2024-04-02T13:21:43.871008+0000","last_peered":"2024-04-02T13:21:43.871008+0000","last_clean":"2024-04-02T13:21:43.871008+0000","last_became_active":"2024-04-02T13:21:42.861191+0000","last_became_peered":"2024-04-02T13:21:42.861191+0000","last_unstale":"2024-04-02T13:21:43.871008+0000","last_undegraded":"2024-04-02T13:21:43.871008+0000","last_fullsized":"2024-04-02T13:21:43.871008+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:19:32.445256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.888917+0000","last_change":"2024-04-02T13:21:43.887970+0000","last_active":"2024-04-02T13:21:43.888917+0000","last_peered":"2024-04-02T13:21:43.888917+0000","last_clean":"2024-04-02T13:21:43.888917+0000","last_became_active":"2024-04-02T13:21:43.887632+0000","last_became_peered":"2024-04-02T13:21:43.887632+0000","last_unstale":"2024-04-02T13:21:43.888917+0000","last_undegraded":"2024-04-02T13:21:43.888917+0000","last_fullsized":"2024-04-02T13:21:43.888917+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:42.845343+0000","last_clean_scrub_stamp":"2024-04-02T13:21:42.845343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:22.467118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.869418+0000","last_change":"2024-04-02T13:21:42.869673+0000","last_active":"2024-04-02T13:21:43.869418+0000","last_peered":"2024-04-02T13:21:43.869418+0000","last_clean":"2024-04-02T13:21:43.869418+0000","last_became_active":"2024-04-02T13:21:42.869401+0000","last_became_peered":"2024-04-02T13:21:42.869401+0000","last_unstale":"2024-04-02T13:21:43.869418+0000","last_undegraded":"2024-04-02T13:21:43.869418+0000","last_fullsized":"2024-04-02T13:21:43.869418+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:21:41.840603+0000","last_clean_scrub_stamp":"2024-04-02T13:21:41.840603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:48.112835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":202,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-02T13:21:43.869832+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:21:43.869832+0000","last_peered":"2024-04-02T13:21:43.869832+0000","last_clean":"2024-04-02T13:21:43.869832+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:21:43.869832+0000","last_undegraded":"2024-04-02T13:21:43.869832+0000","last_fullsized":"2024-04-02T13:21:43.869832+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1470,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":2940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37,"ondisk_log_size":37,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1019904,"data_stored":771370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343812,"statfs":{"total":96636764160,"available":96608063488,"internally_reserved":0,"allocated":1032192,"data_stored":778537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:21:48.246 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T13:21:48.246 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T13:21:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.389+0000 7f3652350640 1 Processor -- start 2024-04-02T13:21:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.389+0000 7f3652350640 1 -- start start 2024-04-02T13:21:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.389+0000 7f3652350640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f364c15e140 0x7f364c0d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.389+0000 7f3652350640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f364c0b70b0 con 0x7f364c0d1150 2024-04-02T13:21:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.389+0000 7f3652350640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f364c0b7390 con 0x7f364c15e520 2024-04-02T13:21:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.389+0000 7f3652350640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f364c0b9f50 con 0x7f364c15e140 2024-04-02T13:21:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.389+0000 7f364bfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f364c15e140 0x7f364c0d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.390+0000 7f364bfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f364c15e140 0x7f364c0d2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53930/0 (socket says 172.21.15.44:53930) 2024-04-02T13:21:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.390+0000 7f364bfff640 1 -- 172.21.15.44:0/2107520347 learned_addr learned my addr 172.21.15.44:0/2107520347 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.390+0000 7f364bfff640 1 -- 172.21.15.44:0/2107520347 >> v1:172.21.15.69:6789/0 conn(0x7f364c15e520 legacy=0x7f364c0d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.390+0000 7f364bfff640 1 -- 172.21.15.44:0/2107520347 >> v1:172.21.15.44:6789/0 conn(0x7f364c0d1150 legacy=0x7f364c0b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.390+0000 7f364bfff640 1 -- 172.21.15.44:0/2107520347 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f364c0b7670 con 0x7f364c15e140 2024-04-02T13:21:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.390+0000 7f364affd640 1 -- 172.21.15.44:0/2107520347 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3648578777 0 0) 0x7f364c0b70b0 con 0x7f364c0d1150 2024-04-02T13:21:48.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.391+0000 7f364bfff640 1 --2- 172.21.15.44:0/2107520347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f364c15e140 0x7f364c0d2630 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f363c009870 tx=0x7f363c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4abf65d98ff5e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:48.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.391+0000 7f364affd640 1 -- 172.21.15.44:0/2107520347 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c035020 con 0x7f364c15e140 2024-04-02T13:21:48.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.391+0000 7f364affd640 1 -- 172.21.15.44:0/2107520347 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f363c031870 con 0x7f364c15e140 2024-04-02T13:21:48.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.391+0000 7f364affd640 1 -- 172.21.15.44:0/2107520347 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c031c40 con 0x7f364c15e140 2024-04-02T13:21:48.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.391+0000 7f3652350640 1 -- 172.21.15.44:0/2107520347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f364c15e140 msgr2=0x7f364c0d2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.391+0000 7f3652350640 1 --2- 172.21.15.44:0/2107520347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f364c15e140 0x7f364c0d2630 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f363c009870 tx=0x7f363c02b910 comp rx=0 tx=0).stop 2024-04-02T13:21:48.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.391+0000 7f3652350640 1 -- 172.21.15.44:0/2107520347 shutdown_connections 2024-04-02T13:21:48.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.391+0000 7f3652350640 1 --2- 172.21.15.44:0/2107520347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f364c15e140 0x7f364c0d2630 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.391+0000 7f3652350640 1 -- 172.21.15.44:0/2107520347 >> 172.21.15.44:0/2107520347 conn(0x7f364c09d390 msgr2=0x7f364c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:48.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.391+0000 7f3652350640 1 -- 172.21.15.44:0/2107520347 shutdown_connections 2024-04-02T13:21:48.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.391+0000 7f3652350640 1 -- 172.21.15.44:0/2107520347 wait complete. 2024-04-02T13:21:48.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.392+0000 7f3652350640 1 Processor -- start 2024-04-02T13:21:48.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.393+0000 7f3652350640 1 -- start start 2024-04-02T13:21:48.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.393+0000 7f3652350640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f364c0d1150 0x7f364c151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.393+0000 7f3652350640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f364c15e140 0x7f364c0c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.393+0000 7f3652350640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f364c15e520 0x7f364c0ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.393+0000 7f3652350640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f364c0d2b70 con 0x7f364c15e520 2024-04-02T13:21:48.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.393+0000 7f3652350640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f364c0d4ee0 con 0x7f364c15e140 2024-04-02T13:21:48.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.393+0000 7f3652350640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f364c0d2cf0 con 0x7f364c0d1150 2024-04-02T13:21:48.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.393+0000 7f36508ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f364c15e520 0x7f364c0ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f36508ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f364c15e520 0x7f364c0ca200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53940/0 (socket says 172.21.15.44:53940) 2024-04-02T13:21:48.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f36508ed640 1 -- 172.21.15.44:0/679276112 learned_addr learned my addr 172.21.15.44:0/679276112 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:48.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f364bfff640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f364c0d1150 0x7f364c151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f364b7fe640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f364c15e140 0x7f364c0c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f36508ed640 1 -- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f364c0d1150 msgr2=0x7f364c151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f36508ed640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f364c0d1150 0x7f364c151450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f36508ed640 1 -- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f364c15e140 msgr2=0x7f364c0c9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f36508ed640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f364c15e140 0x7f364c0c9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f36508ed640 1 -- 172.21.15.44:0/679276112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f364c0cc010 con 0x7f364c15e520 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f36508ed640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f364c15e520 0x7f364c0ca200 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f364400c730 tx=0x7f364400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f3648ff9640 1 -- 172.21.15.44:0/679276112 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3644003da0 con 0x7f364c15e520 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f3648ff9640 1 -- 172.21.15.44:0/679276112 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f364400ce40 con 0x7f364c15e520 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f3648ff9640 1 -- 172.21.15.44:0/679276112 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36440052b0 con 0x7f364c15e520 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f3652350640 1 -- 172.21.15.44:0/679276112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f364c0c7d20 con 0x7f364c15e520 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.394+0000 7f3652350640 1 -- 172.21.15.44:0/679276112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f364c151990 con 0x7f364c15e520 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.395+0000 7f3652350640 1 -- 172.21.15.44:0/679276112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f364c0684f0 con 0x7f364c15e520 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.396+0000 7f3648ff9640 1 -- 172.21.15.44:0/679276112 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3644005450 con 0x7f364c15e520 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.396+0000 7f3648ff9640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f363806c8f0 0x7f363806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.396+0000 7f3648ff9640 1 -- 172.21.15.44:0/679276112 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f364408bc30 con 0x7f364c15e520 2024-04-02T13:21:48.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.396+0000 7f364bfff640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f363806c8f0 0x7f363806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.397+0000 7f364bfff640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f363806c8f0 0x7f363806edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f363c036000 tx=0x7f363c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:48.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.400+0000 7f3648ff9640 1 -- 172.21.15.44:0/679276112 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3644059ed0 con 0x7f364c15e520 2024-04-02T13:21:48.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.655+0000 7f3652350640 1 -- 172.21.15.44:0/679276112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f364c068c90 con 0x7f364c15e520 2024-04-02T13:21:48.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.657+0000 7f3648ff9640 1 -- 172.21.15.44:0/679276112 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f36440597d0 con 0x7f364c15e520 2024-04-02T13:21:48.659 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-04-02T13:21:48.660 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A== 2024-04-02T13:21:48.660 INFO:teuthology.orchestra.run.smithi044.stdout: caps mds = "allow" 2024-04-02T13:21:48.660 INFO:teuthology.orchestra.run.smithi044.stdout: caps mon = "allow r" 2024-04-02T13:21:48.660 INFO:teuthology.orchestra.run.smithi044.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-02T13:21:48.660 INFO:teuthology.orchestra.run.smithi044.stderr:updated caps for client.0 2024-04-02T13:21:48.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.659+0000 7f3652350640 1 -- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f363806c8f0 msgr2=0x7f363806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.659+0000 7f3652350640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f363806c8f0 0x7f363806edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f363c036000 tx=0x7f363c008030 comp rx=0 tx=0).stop 2024-04-02T13:21:48.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.659+0000 7f3652350640 1 -- 172.21.15.44:0/679276112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f364c15e520 msgr2=0x7f364c0ca200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.659+0000 7f3652350640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f364c15e520 0x7f364c0ca200 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f364400c730 tx=0x7f364400cbf0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.659+0000 7f3652350640 1 -- 172.21.15.44:0/679276112 shutdown_connections 2024-04-02T13:21:48.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.659+0000 7f3652350640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f363806c8f0 0x7f363806edb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.659+0000 7f3652350640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f364c15e520 0x7f364c0ca200 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.659+0000 7f3652350640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f364c15e140 0x7f364c0c9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.659+0000 7f3652350640 1 --2- 172.21.15.44:0/679276112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f364c0d1150 0x7f364c151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.659+0000 7f3652350640 1 -- 172.21.15.44:0/679276112 >> 172.21.15.44:0/679276112 conn(0x7f364c09d390 msgr2=0x7f364c0ceb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:48.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.659+0000 7f3652350640 1 -- 172.21.15.44:0/679276112 shutdown_connections 2024-04-02T13:21:48.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.659+0000 7f3652350640 1 -- 172.21.15.44:0/679276112 wait complete. 2024-04-02T13:21:48.663 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:21:48.661+0000 7f130a6cc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:21:48.673 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:21:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.771+0000 7f4b40fa3640 1 Processor -- start 2024-04-02T13:21:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.772+0000 7f4b40fa3640 1 -- start start 2024-04-02T13:21:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.772+0000 7f4b40fa3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b3c1625a0 0x7f4b3c071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.772+0000 7f4b40fa3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b3c06d560 con 0x7f4b3c1588e0 2024-04-02T13:21:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.772+0000 7f4b40fa3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b3c06d820 con 0x7f4b3c0b47c0 2024-04-02T13:21:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.772+0000 7f4b40fa3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4b3c06ecf0 con 0x7f4b3c1625a0 2024-04-02T13:21:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.773+0000 7f4b3a59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4b3c1588e0 0x7f4b3c06e5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33030/0 (socket says 172.21.15.44:33030) 2024-04-02T13:21:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.773+0000 7f4b3a59c640 1 -- 172.21.15.44:0/2041008218 learned_addr learned my addr 172.21.15.44:0/2041008218 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.773+0000 7f4b3ad9d640 1 --2- 172.21.15.44:0/2041008218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b3c1625a0 0x7f4b3c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.773+0000 7f4b3959a640 1 -- 172.21.15.44:0/2041008218 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588852334 0 0) 0x7f4b3c06d560 con 0x7f4b3c1588e0 2024-04-02T13:21:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.773+0000 7f4b3959a640 1 -- 172.21.15.44:0/2041008218 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b28003520 con 0x7f4b3c1588e0 2024-04-02T13:21:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.773+0000 7f4b3959a640 1 -- 172.21.15.44:0/2041008218 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1979785503 0 0) 0x7f4b3c06d820 con 0x7f4b3c0b47c0 2024-04-02T13:21:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.773+0000 7f4b3959a640 1 -- 172.21.15.44:0/2041008218 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b3c06d560 con 0x7f4b3c0b47c0 2024-04-02T13:21:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.773+0000 7f4b3959a640 1 -- 172.21.15.44:0/2041008218 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2204947481 0 0) 0x7f4b28003520 con 0x7f4b3c1588e0 2024-04-02T13:21:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.773+0000 7f4b3ad9d640 1 -- 172.21.15.44:0/2041008218 >> v1:172.21.15.69:6789/0 conn(0x7f4b3c0b47c0 legacy=0x7f4b3c161e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.773+0000 7f4b3ad9d640 1 -- 172.21.15.44:0/2041008218 >> v1:172.21.15.44:6789/0 conn(0x7f4b3c1588e0 legacy=0x7f4b3c06e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.773+0000 7f4b3ad9d640 1 -- 172.21.15.44:0/2041008218 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b3c06db00 con 0x7f4b3c1625a0 2024-04-02T13:21:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.774+0000 7f4b3ad9d640 1 --2- 172.21.15.44:0/2041008218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b3c1625a0 0x7f4b3c071190 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009dc0 tx=0x7f4b2c02bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38423d35a879109e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.774+0000 7f4b3959a640 1 -- 172.21.15.44:0/2041008218 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c039070 con 0x7f4b3c1625a0 2024-04-02T13:21:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.774+0000 7f4b3959a640 1 -- 172.21.15.44:0/2041008218 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b2c02c910 con 0x7f4b3c1625a0 2024-04-02T13:21:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.774+0000 7f4b3959a640 1 -- 172.21.15.44:0/2041008218 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34001670 con 0x7f4b3c1625a0 2024-04-02T13:21:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.775+0000 7f4b40fa3640 1 -- 172.21.15.44:0/2041008218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b3c1625a0 msgr2=0x7f4b3c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.775+0000 7f4b40fa3640 1 --2- 172.21.15.44:0/2041008218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b3c1625a0 0x7f4b3c071190 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009dc0 tx=0x7f4b2c02bde0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.775+0000 7f4b40fa3640 1 -- 172.21.15.44:0/2041008218 shutdown_connections 2024-04-02T13:21:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.775+0000 7f4b40fa3640 1 --2- 172.21.15.44:0/2041008218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b3c1625a0 0x7f4b3c071190 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.775+0000 7f4b40fa3640 1 -- 172.21.15.44:0/2041008218 >> 172.21.15.44:0/2041008218 conn(0x7f4b3c0cfcd0 msgr2=0x7f4b3c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:48.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.775+0000 7f4b40fa3640 1 -- 172.21.15.44:0/2041008218 shutdown_connections 2024-04-02T13:21:48.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.775+0000 7f4b40fa3640 1 -- 172.21.15.44:0/2041008218 wait complete. 2024-04-02T13:21:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.776+0000 7f4b40fa3640 1 Processor -- start 2024-04-02T13:21:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.776+0000 7f4b40fa3640 1 -- start start 2024-04-02T13:21:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b40fa3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b3c0b47c0 0x7f4b3c157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b40fa3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b3c1588e0 0x7f4b3c09cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b40fa3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b3c1625a0 0x7f4b3c1e9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b40fa3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3c158cc0 con 0x7f4b3c0b47c0 2024-04-02T13:21:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b40fa3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3c06e0c0 con 0x7f4b3c1588e0 2024-04-02T13:21:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b40fa3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4b3c162ae0 con 0x7f4b3c1625a0 2024-04-02T13:21:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b3a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b3c0b47c0 0x7f4b3c157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b3a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b3c0b47c0 0x7f4b3c157740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53950/0 (socket says 172.21.15.44:53950) 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b3a59c640 1 -- 172.21.15.44:0/4027312152 learned_addr learned my addr 172.21.15.44:0/4027312152 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b3ad9d640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b3c1625a0 0x7f4b3c1e9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b39d9b640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b3c1588e0 0x7f4b3c09cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b3a59c640 1 -- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b3c1625a0 msgr2=0x7f4b3c1e9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b3a59c640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b3c1625a0 0x7f4b3c1e9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b3a59c640 1 -- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b3c1588e0 msgr2=0x7f4b3c09cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b3a59c640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b3c1588e0 0x7f4b3c09cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b3a59c640 1 -- 172.21.15.44:0/4027312152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b3c09d350 con 0x7f4b3c0b47c0 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b3ad9d640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b3c1625a0 0x7f4b3c1e9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b39d9b640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b3c1588e0 0x7f4b3c09cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.777+0000 7f4b3a59c640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b3c0b47c0 0x7f4b3c157740 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400c780 tx=0x7f4b3400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.778+0000 7f4b32ffd640 1 -- 172.21.15.44:0/4027312152 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34005ce0 con 0x7f4b3c0b47c0 2024-04-02T13:21:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.778+0000 7f4b32ffd640 1 -- 172.21.15.44:0/4027312152 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b3400ce30 con 0x7f4b3c0b47c0 2024-04-02T13:21:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.778+0000 7f4b32ffd640 1 -- 172.21.15.44:0/4027312152 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34007290 con 0x7f4b3c0b47c0 2024-04-02T13:21:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.778+0000 7f4b40fa3640 1 -- 172.21.15.44:0/4027312152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b3c09d830 con 0x7f4b3c0b47c0 2024-04-02T13:21:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.778+0000 7f4b40fa3640 1 -- 172.21.15.44:0/4027312152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b3c09b840 con 0x7f4b3c0b47c0 2024-04-02T13:21:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.779+0000 7f4b40fa3640 1 -- 172.21.15.44:0/4027312152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b3c0684f0 con 0x7f4b3c0b47c0 2024-04-02T13:21:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.779+0000 7f4b32ffd640 1 -- 172.21.15.44:0/4027312152 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4b34007430 con 0x7f4b3c0b47c0 2024-04-02T13:21:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.779+0000 7f4b32ffd640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4b1c06c8f0 0x7f4b1c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.779+0000 7f4b32ffd640 1 -- 172.21.15.44:0/4027312152 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f4b3408baf0 con 0x7f4b3c0b47c0 2024-04-02T13:21:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.780+0000 7f4b39d9b640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4b1c06c8f0 0x7f4b1c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.780+0000 7f4b39d9b640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4b1c06c8f0 0x7f4b1c06edb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c1e8e50 tx=0x7f4b24009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:48.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:48.784+0000 7f4b32ffd640 1 -- 172.21.15.44:0/4027312152 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4b34059ee0 con 0x7f4b3c0b47c0 2024-04-02T13:21:49.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.008+0000 7f4b40fa3640 1 -- 172.21.15.44:0/4027312152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b3c071610 con 0x7f4b3c0b47c0 2024-04-02T13:21:49.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.009+0000 7f4b32ffd640 1 -- 172.21.15.44:0/4027312152 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5181 (secure 0 0 0) 0x7f4b340597e0 con 0x7f4b3c0b47c0 2024-04-02T13:21:49.012 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:49.012 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3050514527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3050514527},{"type":"v1","addr":"172.21.15.69:6836","nonce":3050514527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4973,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1949024012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1949024012},{"type":"v1","addr":"172.21.15.69:6837","nonce":1949024012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/2817873198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2817873198},{"type":"v1","addr":"172.21.15.44:6834","nonce":2817873198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:21:43.215796+0000","modified":"2024-04-02T13:21:46.248294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/1969465285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":1969465285},{"type":"v1","addr":"172.21.15.44:6835","nonce":1969465285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":2}]} 2024-04-02T13:21:49.012 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 18 2024-04-02T13:21:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.011+0000 7f4b40fa3640 1 -- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4b1c06c8f0 msgr2=0x7f4b1c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.011+0000 7f4b40fa3640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4b1c06c8f0 0x7f4b1c06edb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c1e8e50 tx=0x7f4b24009040 comp rx=0 tx=0).stop 2024-04-02T13:21:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.011+0000 7f4b40fa3640 1 -- 172.21.15.44:0/4027312152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b3c0b47c0 msgr2=0x7f4b3c157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.011+0000 7f4b40fa3640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b3c0b47c0 0x7f4b3c157740 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400c780 tx=0x7f4b3400cc40 comp rx=0 tx=0).stop 2024-04-02T13:21:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.011+0000 7f4b40fa3640 1 -- 172.21.15.44:0/4027312152 shutdown_connections 2024-04-02T13:21:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.011+0000 7f4b40fa3640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4b1c06c8f0 0x7f4b1c06edb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.011+0000 7f4b40fa3640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b3c1625a0 0x7f4b3c1e9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.011+0000 7f4b40fa3640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b3c1588e0 0x7f4b3c09cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.011+0000 7f4b40fa3640 1 --2- 172.21.15.44:0/4027312152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b3c0b47c0 0x7f4b3c157740 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.011+0000 7f4b40fa3640 1 -- 172.21.15.44:0/4027312152 >> 172.21.15.44:0/4027312152 conn(0x7f4b3c0cfcd0 msgr2=0x7f4b3c072750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.011+0000 7f4b40fa3640 1 -- 172.21.15.44:0/4027312152 shutdown_connections 2024-04-02T13:21:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.011+0000 7f4b40fa3640 1 -- 172.21.15.44:0/4027312152 wait complete. 2024-04-02T13:21:49.427 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:21:43.215796+0000', 'modified': '2024-04-02T13:21:46.248294+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5511}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5511': {'gid': 5511, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.44:6835/1969465285', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6833', 'nonce': 1969465285}, {'type': 'v1', 'addr': '172.21.15.44:6835', 'nonce': 1969465285}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5511, 'qdb_cluster': [5511]} 2024-04-02T13:21:49.427 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:21:49.427 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T13:21:49.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.530+0000 7fcd7cd3d640 1 Processor -- start 2024-04-02T13:21:49.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.530+0000 7fcd7cd3d640 1 -- start start 2024-04-02T13:21:49.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7cd3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd7806bbe0 0x7fcd78169140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7cd3d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd780aed70 con 0x7fcd7815e0f0 2024-04-02T13:21:49.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7cd3d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd7806dfe0 con 0x7fcd7815e4d0 2024-04-02T13:21:49.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7cd3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcd7806e2c0 con 0x7fcd7806bbe0 2024-04-02T13:21:49.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7659c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fcd7815e0f0 0x7fcd780d00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33040/0 (socket says 172.21.15.44:33040) 2024-04-02T13:21:49.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7659c640 1 -- 172.21.15.44:0/11787021 learned_addr learned my addr 172.21.15.44:0/11787021 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:49.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd76d9d640 1 --2- 172.21.15.44:0/11787021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd7806bbe0 0x7fcd78169140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 950472551 0 0) 0x7fcd780aed70 con 0x7fcd7815e0f0 2024-04-02T13:21:49.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd60003520 con 0x7fcd7815e0f0 2024-04-02T13:21:49.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3024823620 0 0) 0x7fcd60003520 con 0x7fcd7815e0f0 2024-04-02T13:21:49.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd7806bbe0 msgr2=0x7fcd78169140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:49.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 --2- 172.21.15.44:0/11787021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd7806bbe0 0x7fcd78169140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 >> v1:172.21.15.69:6789/0 conn(0x7fcd7815e4d0 legacy=0x7fcd7806b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7806e5a0 con 0x7fcd7815e0f0 2024-04-02T13:21:49.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fcd68001670 con 0x7fcd7815e0f0 2024-04-02T13:21:49.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 >> v1:172.21.15.44:6789/0 conn(0x7fcd7815e0f0 legacy=0x7fcd780d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 --2- 172.21.15.44:0/11787021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd600039e0 0x7fcd60003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 --2- 172.21.15.44:0/11787021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd7806bbe0 0x7fcd60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 --2- 172.21.15.44:0/11787021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd60004d40 0x7fcd600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7806e2c0 con 0x7fcd600039e0 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcd6000bb30 con 0x7fcd60004d40 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.531+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcd6000bd30 con 0x7fcd7806bbe0 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.532+0000 7fcd7659c640 1 --2- 172.21.15.44:0/11787021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd600039e0 0x7fcd60003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.532+0000 7fcd75d9b640 1 --2- 172.21.15.44:0/11787021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd7806bbe0 0x7fcd60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.532+0000 7fcd76d9d640 1 --2- 172.21.15.44:0/11787021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd60004d40 0x7fcd600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.532+0000 7fcd7659c640 1 -- 172.21.15.44:0/11787021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd7806bbe0 msgr2=0x7fcd60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.532+0000 7fcd7659c640 1 --2- 172.21.15.44:0/11787021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd7806bbe0 0x7fcd60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.532+0000 7fcd7659c640 1 -- 172.21.15.44:0/11787021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd60004d40 msgr2=0x7fcd600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.532+0000 7fcd7659c640 1 --2- 172.21.15.44:0/11787021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd60004d40 0x7fcd600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.532+0000 7fcd7659c640 1 -- 172.21.15.44:0/11787021 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd60003520 con 0x7fcd600039e0 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.532+0000 7fcd76d9d640 1 --2- 172.21.15.44:0/11787021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd60004d40 0x7fcd600051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.533+0000 7fcd7659c640 1 --2- 172.21.15.44:0/11787021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd600039e0 0x7fcd60003e00 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fcd6800c210 tx=0x7fcd60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0b7bd611b09ad7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:49.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.533+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd68007cf0 con 0x7fcd600039e0 2024-04-02T13:21:49.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.533+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd68030070 con 0x7fcd600039e0 2024-04-02T13:21:49.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.533+0000 7fcd7559a640 1 -- 172.21.15.44:0/11787021 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd68005d30 con 0x7fcd600039e0 2024-04-02T13:21:49.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.533+0000 7fcd7cd3d640 1 -- 172.21.15.44:0/11787021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd600039e0 msgr2=0x7fcd60003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.533+0000 7fcd7cd3d640 1 --2- 172.21.15.44:0/11787021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd600039e0 0x7fcd60003e00 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fcd6800c210 tx=0x7fcd60005830 comp rx=0 tx=0).stop 2024-04-02T13:21:49.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.533+0000 7fcd76d9d640 1 -- 172.21.15.44:0/11787021 reap_dead start 2024-04-02T13:21:49.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.533+0000 7fcd7cd3d640 1 -- 172.21.15.44:0/11787021 shutdown_connections 2024-04-02T13:21:49.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.533+0000 7fcd7cd3d640 1 -- 172.21.15.44:0/11787021 >> 172.21.15.44:0/11787021 conn(0x7fcd780b2980 msgr2=0x7fcd780b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:49.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.533+0000 7fcd7cd3d640 1 -- 172.21.15.44:0/11787021 shutdown_connections 2024-04-02T13:21:49.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.533+0000 7fcd7cd3d640 1 -- 172.21.15.44:0/11787021 wait complete. 2024-04-02T13:21:49.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.535+0000 7fcd7cd3d640 1 Processor -- start 2024-04-02T13:21:49.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.535+0000 7fcd7cd3d640 1 -- start start 2024-04-02T13:21:49.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.535+0000 7fcd7cd3d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd600039e0 0x7fcd780c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.535+0000 7fcd7cd3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd780add60 0x7fcd780ae1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.535+0000 7fcd7cd3d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd780cc370 0x7fcd780cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.535+0000 7fcd7cd3d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcd780d0800 con 0x7fcd600039e0 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.535+0000 7fcd7cd3d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcd780d0af0 con 0x7fcd780cc370 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.535+0000 7fcd7cd3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcd7806eda0 con 0x7fcd780add60 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.535+0000 7fcd7659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd600039e0 0x7fcd780c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.535+0000 7fcd7659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd600039e0 0x7fcd780c7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53974/0 (socket says 172.21.15.44:53974) 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.535+0000 7fcd7659c640 1 -- 172.21.15.44:0/3909674560 learned_addr learned my addr 172.21.15.44:0/3909674560 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.536+0000 7fcd76d9d640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd780cc370 0x7fcd780cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.536+0000 7fcd75d9b640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd780add60 0x7fcd780ae1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.536+0000 7fcd7659c640 1 -- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd780add60 msgr2=0x7fcd780ae1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.536+0000 7fcd7659c640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd780add60 0x7fcd780ae1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.536+0000 7fcd7659c640 1 -- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd780cc370 msgr2=0x7fcd780cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.536+0000 7fcd7659c640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd780cc370 0x7fcd780cc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.536+0000 7fcd7659c640 1 -- 172.21.15.44:0/3909674560 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd780cd0a0 con 0x7fcd600039e0 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.536+0000 7fcd7659c640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd600039e0 0x7fcd780c7490 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fcd6802c500 tx=0x7fcd780d0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:49.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.536+0000 7fcd66ffd640 1 -- 172.21.15.44:0/3909674560 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd68001c30 con 0x7fcd600039e0 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.536+0000 7fcd66ffd640 1 -- 172.21.15.44:0/3909674560 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd68004bf0 con 0x7fcd600039e0 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.536+0000 7fcd66ffd640 1 -- 172.21.15.44:0/3909674560 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd68007070 con 0x7fcd600039e0 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.536+0000 7fcd7cd3d640 1 -- 172.21.15.44:0/3909674560 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd780cd380 con 0x7fcd600039e0 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.536+0000 7fcd7cd3d640 1 -- 172.21.15.44:0/3909674560 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd78156460 con 0x7fcd600039e0 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.538+0000 7fcd66ffd640 1 -- 172.21.15.44:0/3909674560 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcd68047020 con 0x7fcd600039e0 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.538+0000 7fcd66ffd640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcd5806c8f0 0x7fcd5806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.538+0000 7fcd66ffd640 1 -- 172.21.15.44:0/3909674560 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fcd680acf10 con 0x7fcd600039e0 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.538+0000 7fcd75d9b640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcd5806c8f0 0x7fcd5806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.539+0000 7fcd75d9b640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcd5806c8f0 0x7fcd5806edb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcd781e9620 tx=0x7fcd6c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.539+0000 7fcd7fa00740 1 Processor -- start 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.539+0000 7fcd7fa00740 1 -- start start 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7fa00740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 0x557cb7e86b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7fa00740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3f70 0x557cb7eb6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7fa00740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7e87070 0x557cb7eb6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7fa00740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557cb7df0060 con 0x557cb7eb3b90 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7fa00740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557cb7d98650 con 0x557cb7e87070 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7fa00740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x557cb7dec010 con 0x557cb7eb3f70 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 0x557cb7e86b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 0x557cb7e86b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53980/0 (socket says 172.21.15.44:53980) 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7659c640 1 -- 172.21.15.44:0/2677665132 learned_addr learned my addr 172.21.15.44:0/2677665132 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd75d9b640 1 --2- 172.21.15.44:0/2677665132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3f70 0x557cb7eb6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd76d9d640 1 --2- 172.21.15.44:0/2677665132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7e87070 0x557cb7eb6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7659c640 1 -- 172.21.15.44:0/2677665132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3f70 msgr2=0x557cb7eb6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7659c640 1 --2- 172.21.15.44:0/2677665132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3f70 0x557cb7eb6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7659c640 1 -- 172.21.15.44:0/2677665132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7e87070 msgr2=0x557cb7eb6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7659c640 1 --2- 172.21.15.44:0/2677665132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7e87070 0x557cb7eb6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7659c640 1 -- 172.21.15.44:0/2677665132 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cb7b042d0 con 0x557cb7eb3b90 2024-04-02T13:21:49.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.540+0000 7fcd7659c640 1 --2- 172.21.15.44:0/2677665132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 0x557cb7e86b30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcd68001840 tx=0x7fcd68076df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57843be576e12135 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.541+0000 7fcd57fff640 1 -- 172.21.15.44:0/2677665132 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6807edd0 con 0x557cb7eb3b90 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.541+0000 7fcd57fff640 1 -- 172.21.15.44:0/2677665132 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd68030070 con 0x557cb7eb3b90 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.541+0000 7fcd57fff640 1 -- 172.21.15.44:0/2677665132 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd68083690 con 0x557cb7eb3b90 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.541+0000 7fcd7fa00740 1 -- 172.21.15.44:0/2677665132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 msgr2=0x557cb7e86b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.541+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/2677665132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 0x557cb7e86b30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcd68001840 tx=0x7fcd68076df0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.541+0000 7fcd7fa00740 1 -- 172.21.15.44:0/2677665132 shutdown_connections 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.541+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/2677665132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7e87070 0x557cb7eb6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.541+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/2677665132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3f70 0x557cb7eb6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.541+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/2677665132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 0x557cb7e86b30 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.541+0000 7fcd7fa00740 1 -- 172.21.15.44:0/2677665132 >> 172.21.15.44:0/2677665132 conn(0x557cb7e88da0 msgr2=0x557cb7ed1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.541+0000 7fcd7fa00740 1 -- 172.21.15.44:0/2677665132 shutdown_connections 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.541+0000 7fcd7fa00740 1 -- 172.21.15.44:0/2677665132 wait complete. 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.542+0000 7fcd7fa00740 1 -- start start 2024-04-02T13:21:49.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.542+0000 7fcd7fa00740 1 Processor -- start 2024-04-02T13:21:49.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd7fa00740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3b90 0x557cb7ee51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd7fa00740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 0x557cb7ee5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd7fa00740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eaf260 0x557cb7ee5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd7fa00740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557cb7d98650 con 0x557cb7eb3f70 2024-04-02T13:21:49.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd7fa00740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557cb7daba10 con 0x557cb7eaf260 2024-04-02T13:21:49.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd7fa00740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x557cb7df0060 con 0x557cb7eb3b90 2024-04-02T13:21:49.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd7fa00740 10 client.0 _ll_register_callbacks cb 0x557cb7de80b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:21:49.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd75d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 0x557cb7ee5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd75d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 0x557cb7ee5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53988/0 (socket says 172.21.15.44:53988) 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd75d9b640 1 -- 172.21.15.44:0/3662707365 learned_addr learned my addr 172.21.15.44:0/3662707365 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd7659c640 1 --2- 172.21.15.44:0/3662707365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3b90 0x557cb7ee51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd76d9d640 1 --2- 172.21.15.44:0/3662707365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eaf260 0x557cb7ee5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd75d9b640 1 -- 172.21.15.44:0/3662707365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3b90 msgr2=0x557cb7ee51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd75d9b640 1 --2- 172.21.15.44:0/3662707365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3b90 0x557cb7ee51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd75d9b640 1 -- 172.21.15.44:0/3662707365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eaf260 msgr2=0x557cb7ee5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd75d9b640 1 --2- 172.21.15.44:0/3662707365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eaf260 0x557cb7ee5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd75d9b640 1 -- 172.21.15.44:0/3662707365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cb7bce6b0 con 0x557cb7eb3f70 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd75d9b640 1 --2- 172.21.15.44:0/3662707365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 0x557cb7ee5700 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c015550 tx=0x7fcd6c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1dc0630e0c85554 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.543+0000 7fcd557fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd557fa640 1 -- 172.21.15.44:0/3662707365 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c0145e0 con 0x557cb7eb3f70 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd557fa640 1 -- 172.21.15.44:0/3662707365 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fcd6c014780 con 0x557cb7eb3f70 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3662707365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd68030070 con 0x557cb7eb3f70 2024-04-02T13:21:49.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd557fa640 1 -- 172.21.15.44:0/3662707365 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fcd6c030050 con 0x557cb7eb3f70 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd7fa00740 10 client.5697 fetch_fsmap learned FSMap version 18 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3662707365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x557cb7eb80c0 con 0x557cb7eb3f70 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd557fa640 1 -- 172.21.15.44:0/3662707365 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd68030070 con 0x557cb7eb3f70 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd557fa640 1 -- 172.21.15.44:0/3662707365 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7fcd6c033020 con 0x557cb7eb3f70 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd7fa00740 10 client.5697 fetch_fsmap finished waiting for FSMap version 18 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd7fa00740 10 client.5697 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd7fa00740 20 client.5697 populate_metadata read hostname 'smithi044' 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/3662707365 >> [v2:172.21.15.44:6833/1969465285,v1:172.21.15.44:6835/1969465285] conn(0x557cb7eb9380 0x557cb7ef0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd7fa00740 4 client.5697 mds_command: new command op to 5511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3662707365 --> [v2:172.21.15.44:6833/1969465285,v1:172.21.15.44:6835/1969465285] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557cb7d97cf0 con 0x557cb7eb9380 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.544+0000 7fcd7659c640 1 --2- 172.21.15.44:0/3662707365 >> [v2:172.21.15.44:6833/1969465285,v1:172.21.15.44:6835/1969465285] conn(0x557cb7eb9380 0x557cb7ef0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd7659c640 1 --2- 172.21.15.44:0/3662707365 >> [v2:172.21.15.44:6833/1969465285,v1:172.21.15.44:6835/1969465285] conn(0x557cb7eb9380 0x557cb7ef0650 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86e88cbaff3eb4fb server_cookie=e0e1e0b3974a8732 in_seq=0 out_seq=0 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd557fa640 10 client.5697 ms_handle_connect on v2:172.21.15.44:6833/1969465285 2024-04-02T13:21:49.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd557fa640 1 -- 172.21.15.44:0/3662707365 <== mds.0 v2:172.21.15.44:6833/1969465285 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557cb7d97cf0 con 0x557cb7eb9380 2024-04-02T13:21:49.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd557fa640 10 client.5697 handle_command_reply: tid=0 2024-04-02T13:21:49.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd7fa00740 1 client.5697 shutdown 2024-04-02T13:21:49.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3662707365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 msgr2=0x557cb7ee5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/3662707365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 0x557cb7ee5700 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c015550 tx=0x7fcd6c001120 comp rx=0 tx=0).stop 2024-04-02T13:21:49.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3662707365 shutdown_connections 2024-04-02T13:21:49.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/3662707365 >> [v2:172.21.15.44:6833/1969465285,v1:172.21.15.44:6835/1969465285] conn(0x557cb7eb9380 0x557cb7ef0650 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/3662707365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eaf260 0x557cb7ee5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/3662707365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 0x557cb7ee5700 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/3662707365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3b90 0x557cb7ee51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd557fa640 0 client.5697 ms_handle_reset on v2:172.21.15.44:6833/1969465285 2024-04-02T13:21:49.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3662707365 >> 172.21.15.44:0/3662707365 conn(0x557cb7e87070 msgr2=0x557cb7e8ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:49.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3662707365 shutdown_connections 2024-04-02T13:21:49.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.545+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3662707365 wait complete. 2024-04-02T13:21:49.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.546+0000 7fcd7fa00740 20 client.5697 trim_cache size 0 max 16384 2024-04-02T13:21:49.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.564+0000 7fcd7fa00740 1 Processor -- start 2024-04-02T13:21:49.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.564+0000 7fcd7fa00740 1 -- start start 2024-04-02T13:21:49.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd7fa00740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eaf260 0x557cb7eb5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd7fa00740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eb3b90 0x557cb7eb9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd7fa00740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 0x557cb7eb9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd7fa00740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcd782016c0 con 0x557cb7eb3f70 2024-04-02T13:21:49.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd7fa00740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcd782006a0 con 0x557cb7eb3b90 2024-04-02T13:21:49.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd7fa00740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcd780c8830 con 0x557cb7eaf260 2024-04-02T13:21:49.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd76d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 0x557cb7eb9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd76d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 0x557cb7eb9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54004/0 (socket says 172.21.15.44:54004) 2024-04-02T13:21:49.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd76d9d640 1 -- 172.21.15.44:0/2273050585 learned_addr learned my addr 172.21.15.44:0/2273050585 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:49.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd7659c640 1 --2- 172.21.15.44:0/2273050585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eaf260 0x557cb7eb5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd75d9b640 1 --2- 172.21.15.44:0/2273050585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eb3b90 0x557cb7eb9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd76d9d640 1 -- 172.21.15.44:0/2273050585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eaf260 msgr2=0x557cb7eb5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd76d9d640 1 --2- 172.21.15.44:0/2273050585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eaf260 0x557cb7eb5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd76d9d640 1 -- 172.21.15.44:0/2273050585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eb3b90 msgr2=0x557cb7eb9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd76d9d640 1 --2- 172.21.15.44:0/2273050585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eb3b90 0x557cb7eb9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd76d9d640 1 -- 172.21.15.44:0/2273050585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cb7eb71c0 con 0x557cb7eb3f70 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd76d9d640 1 --2- 172.21.15.44:0/2273050585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 0x557cb7eb9ca0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fcd70003ca0 tx=0x7fcd70009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880bbcc154fda9e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.565+0000 7fcd577fe640 1 -- 172.21.15.44:0/2273050585 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70018830 con 0x557cb7eb3f70 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.566+0000 7fcd577fe640 1 -- 172.21.15.44:0/2273050585 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd70018a10 con 0x557cb7eb3f70 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.566+0000 7fcd577fe640 1 -- 172.21.15.44:0/2273050585 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70018e00 con 0x557cb7eb3f70 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.566+0000 7fcd7fa00740 1 -- 172.21.15.44:0/2273050585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 msgr2=0x557cb7eb9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.566+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/2273050585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 0x557cb7eb9ca0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fcd70003ca0 tx=0x7fcd70009750 comp rx=0 tx=0).stop 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.566+0000 7fcd7fa00740 1 -- 172.21.15.44:0/2273050585 shutdown_connections 2024-04-02T13:21:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.566+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/2273050585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3f70 0x557cb7eb9ca0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.566+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/2273050585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eb3b90 0x557cb7eb9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.566+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/2273050585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eaf260 0x557cb7eb5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.566+0000 7fcd7fa00740 1 -- 172.21.15.44:0/2273050585 >> 172.21.15.44:0/2273050585 conn(0x557cb7e87070 msgr2=0x557cb7ed3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:49.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.566+0000 7fcd7fa00740 1 -- 172.21.15.44:0/2273050585 shutdown_connections 2024-04-02T13:21:49.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.566+0000 7fcd7fa00740 1 -- 172.21.15.44:0/2273050585 wait complete. 2024-04-02T13:21:49.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.567+0000 7fcd7fa00740 1 -- start start 2024-04-02T13:21:49.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.567+0000 7fcd7fa00740 1 Processor -- start 2024-04-02T13:21:49.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd7fa00740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eaf260 0x557cb7e86aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd7fa00740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 0x557cb7e8acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd7fa00740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3f70 0x557cb7e8b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd7fa00740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557cb7ebb250 con 0x557cb7eb3b90 2024-04-02T13:21:49.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd7fa00740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcd780c8e80 con 0x557cb7eaf260 2024-04-02T13:21:49.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd7fa00740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x557cb7eba550 con 0x557cb7eb3f70 2024-04-02T13:21:49.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd7fa00740 10 client.0 _ll_register_callbacks cb 0x7fcd6c005380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:21:49.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd75d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 0x557cb7e8acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd75d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 0x557cb7e8acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54010/0 (socket says 172.21.15.44:54010) 2024-04-02T13:21:49.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd75d9b640 1 -- 172.21.15.44:0/3882919995 learned_addr learned my addr 172.21.15.44:0/3882919995 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:49.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd7659c640 1 --2- 172.21.15.44:0/3882919995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eaf260 0x557cb7e86aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd76d9d640 1 --2- 172.21.15.44:0/3882919995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3f70 0x557cb7e8b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd75d9b640 1 -- 172.21.15.44:0/3882919995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3f70 msgr2=0x557cb7e8b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd75d9b640 1 --2- 172.21.15.44:0/3882919995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3f70 0x557cb7e8b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd75d9b640 1 -- 172.21.15.44:0/3882919995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eaf260 msgr2=0x557cb7e86aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd75d9b640 1 --2- 172.21.15.44:0/3882919995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eaf260 0x557cb7e86aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd75d9b640 1 -- 172.21.15.44:0/3882919995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cb7eab430 con 0x557cb7eb3b90 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd75d9b640 1 --2- 172.21.15.44:0/3882919995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 0x557cb7e8acb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009010 tx=0x7fcd6c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3395d982979f5f6f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd557fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd557fa640 1 -- 172.21.15.44:0/3882919995 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c02d850 con 0x557cb7eb3b90 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd557fa640 1 -- 172.21.15.44:0/3882919995 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fcd6c03ec60 con 0x557cb7eb3b90 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.568+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3882919995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd70018a10 con 0x557cb7eb3b90 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd557fa640 1 -- 172.21.15.44:0/3882919995 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fcd6c03ee60 con 0x557cb7eb3b90 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd7fa00740 10 client.5703 fetch_fsmap learned FSMap version 18 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3882919995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x557cb7e86580 con 0x557cb7eb3b90 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd557fa640 1 -- 172.21.15.44:0/3882919995 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd70018a10 con 0x557cb7eb3b90 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd557fa640 1 -- 172.21.15.44:0/3882919995 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7fcd6c033040 con 0x557cb7eb3b90 2024-04-02T13:21:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd7fa00740 10 client.5703 fetch_fsmap finished waiting for FSMap version 18 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd7fa00740 10 client.5703 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd7fa00740 20 client.5703 populate_metadata read hostname 'smithi044' 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/3882919995 >> [v2:172.21.15.44:6833/1969465285,v1:172.21.15.44:6835/1969465285] conn(0x557cb7ed2e00 0x557cb7ed31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd7fa00740 4 client.5703 mds_command: new command op to 5511 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3882919995 --> [v2:172.21.15.44:6833/1969465285,v1:172.21.15.44:6835/1969465285] -- command(tid 0: {"prefix": "status"}) -- 0x557cb7dfd5a0 con 0x557cb7ed2e00 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd7659c640 1 --2- 172.21.15.44:0/3882919995 >> [v2:172.21.15.44:6833/1969465285,v1:172.21.15.44:6835/1969465285] conn(0x557cb7ed2e00 0x557cb7ed31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd557fa640 10 client.5703 ms_handle_connect on v2:172.21.15.44:6833/1969465285 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.569+0000 7fcd7659c640 1 --2- 172.21.15.44:0/3882919995 >> [v2:172.21.15.44:6833/1969465285,v1:172.21.15.44:6835/1969465285] conn(0x557cb7ed2e00 0x557cb7ed31e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=342bdd3606e14c6c server_cookie=d09ee3f29c966214 in_seq=0 out_seq=0 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd557fa640 1 -- 172.21.15.44:0/3882919995 <== mds.0 v2:172.21.15.44:6833/1969465285 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x557cb7dfd5a0 con 0x557cb7ed2e00 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd557fa640 10 client.5703 handle_command_reply: tid=0 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7fa00740 1 client.5703 shutdown 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3882919995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 msgr2=0x557cb7e8acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/3882919995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 0x557cb7e8acb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009010 tx=0x7fcd6c015000 comp rx=0 tx=0).stop 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3882919995 shutdown_connections 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/3882919995 >> [v2:172.21.15.44:6833/1969465285,v1:172.21.15.44:6835/1969465285] conn(0x557cb7ed2e00 0x557cb7ed31e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/3882919995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cb7eb3f70 0x557cb7e8b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/3882919995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cb7eb3b90 0x557cb7e8acb0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7fa00740 1 --2- 172.21.15.44:0/3882919995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cb7eaf260 0x557cb7e86aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd557fa640 0 client.5703 ms_handle_reset on v2:172.21.15.44:6833/1969465285 2024-04-02T13:21:49.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3882919995 >> 172.21.15.44:0/3882919995 conn(0x557cb7e87070 msgr2=0x557cb7ef5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:49.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3882919995 shutdown_connections 2024-04-02T13:21:49.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7fa00740 1 -- 172.21.15.44:0/3882919995 wait complete. 2024-04-02T13:21:49.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7fa00740 20 client.5703 trim_cache size 0 max 16384 2024-04-02T13:21:49.578 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:21:49.578 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:21:49.578 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:21:49.579 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 5511, 2024-04-02T13:21:49.579 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:21:49.579 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:21:49.579 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:21:49.579 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 5.6960466500000004, 2024-04-02T13:21:49.579 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 18, 2024-04-02T13:21:49.579 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 39, 2024-04-02T13:21:49.579 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 39, 2024-04-02T13:21:49.579 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 12.770304629 2024-04-02T13:21:49.579 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:21:49.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7cd3d640 1 -- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcd5806c8f0 msgr2=0x7fcd5806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7cd3d640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcd5806c8f0 0x7fcd5806edb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcd781e9620 tx=0x7fcd6c009040 comp rx=0 tx=0).stop 2024-04-02T13:21:49.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7cd3d640 1 -- 172.21.15.44:0/3909674560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd600039e0 msgr2=0x7fcd780c7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.570+0000 7fcd7cd3d640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd600039e0 0x7fcd780c7490 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fcd6802c500 tx=0x7fcd780d0040 comp rx=0 tx=0).stop 2024-04-02T13:21:49.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.571+0000 7fcd7cd3d640 1 -- 172.21.15.44:0/3909674560 shutdown_connections 2024-04-02T13:21:49.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.571+0000 7fcd7cd3d640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcd5806c8f0 0x7fcd5806edb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.571+0000 7fcd7cd3d640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd780cc370 0x7fcd780cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.571+0000 7fcd7cd3d640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd780add60 0x7fcd780ae1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.571+0000 7fcd7cd3d640 1 --2- 172.21.15.44:0/3909674560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd600039e0 0x7fcd780c7490 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.571+0000 7fcd7cd3d640 1 -- 172.21.15.44:0/3909674560 >> 172.21.15.44:0/3909674560 conn(0x7fcd780b2980 msgr2=0x7fcd780b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:49.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.571+0000 7fcd7cd3d640 1 -- 172.21.15.44:0/3909674560 shutdown_connections 2024-04-02T13:21:49.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.571+0000 7fcd7cd3d640 1 -- 172.21.15.44:0/3909674560 wait complete. 2024-04-02T13:21:49.584 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:21:49.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.683+0000 7fbd9ec4a640 1 Processor -- start 2024-04-02T13:21:49.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.683+0000 7fbd9ec4a640 1 -- start start 2024-04-02T13:21:49.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.683+0000 7fbd9ec4a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd9806b4d0 0x7fbd980d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.684+0000 7fbd9ec4a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd9806bab0 con 0x7fbd980d06c0 2024-04-02T13:21:49.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.684+0000 7fbd9ec4a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd9806bd90 con 0x7fbd9815c160 2024-04-02T13:21:49.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.684+0000 7fbd9ec4a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbd980ccec0 con 0x7fbd9806b4d0 2024-04-02T13:21:49.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.684+0000 7fbd9d1e7640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbd980d06c0 0x7fbd980cd6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33042/0 (socket says 172.21.15.44:33042) 2024-04-02T13:21:49.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.684+0000 7fbd9d1e7640 1 -- 172.21.15.44:0/3548211685 learned_addr learned my addr 172.21.15.44:0/3548211685 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:49.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.684+0000 7fbd97fff640 1 --2- 172.21.15.44:0/3548211685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd9806b4d0 0x7fbd980d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.684+0000 7fbd977fe640 1 -- 172.21.15.44:0/3548211685 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1961212711 0 0) 0x7fbd9806bab0 con 0x7fbd980d06c0 2024-04-02T13:21:49.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.684+0000 7fbd977fe640 1 -- 172.21.15.44:0/3548211685 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd80003520 con 0x7fbd980d06c0 2024-04-02T13:21:49.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.684+0000 7fbd977fe640 1 -- 172.21.15.44:0/3548211685 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 918673323 0 0) 0x7fbd9806bd90 con 0x7fbd9815c160 2024-04-02T13:21:49.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.684+0000 7fbd977fe640 1 -- 172.21.15.44:0/3548211685 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd9806bab0 con 0x7fbd9815c160 2024-04-02T13:21:49.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.684+0000 7fbd97fff640 1 -- 172.21.15.44:0/3548211685 >> v1:172.21.15.69:6789/0 conn(0x7fbd9815c160 legacy=0x7fbd98160600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.684+0000 7fbd97fff640 1 -- 172.21.15.44:0/3548211685 >> v1:172.21.15.44:6789/0 conn(0x7fbd980d06c0 legacy=0x7fbd980cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.684+0000 7fbd97fff640 1 -- 172.21.15.44:0/3548211685 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd980cd1a0 con 0x7fbd9806b4d0 2024-04-02T13:21:49.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.685+0000 7fbd97fff640 1 --2- 172.21.15.44:0/3548211685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd9806b4d0 0x7fbd980d0180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0097d0 tx=0x7fbd8c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c01f0bbcebfe2f8b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:49.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.685+0000 7fbd977fe640 1 -- 172.21.15.44:0/3548211685 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c035020 con 0x7fbd9806b4d0 2024-04-02T13:21:49.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.685+0000 7fbd977fe640 1 -- 172.21.15.44:0/3548211685 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd8c031800 con 0x7fbd9806b4d0 2024-04-02T13:21:49.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.685+0000 7fbd977fe640 1 -- 172.21.15.44:0/3548211685 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c031bb0 con 0x7fbd9806b4d0 2024-04-02T13:21:49.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.686+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/3548211685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd9806b4d0 msgr2=0x7fbd980d0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.686+0000 7fbd9ec4a640 1 --2- 172.21.15.44:0/3548211685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd9806b4d0 0x7fbd980d0180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0097d0 tx=0x7fbd8c02b8a0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.686+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/3548211685 shutdown_connections 2024-04-02T13:21:49.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.686+0000 7fbd9ec4a640 1 --2- 172.21.15.44:0/3548211685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd9806b4d0 0x7fbd980d0180 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.686+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/3548211685 >> 172.21.15.44:0/3548211685 conn(0x7fbd980b2980 msgr2=0x7fbd980b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:49.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.686+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/3548211685 shutdown_connections 2024-04-02T13:21:49.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.686+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/3548211685 wait complete. 2024-04-02T13:21:49.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.687+0000 7fbd9ec4a640 1 Processor -- start 2024-04-02T13:21:49.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.687+0000 7fbd9ec4a640 1 -- start start 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9ec4a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd9806b4d0 0x7fbd980c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9ec4a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd980d06c0 0x7fbd980ae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9ec4a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9815c160 0x7fbd980aef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9ec4a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd980adee0 con 0x7fbd9815c160 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9ec4a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbd980add60 con 0x7fbd9806b4d0 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9ec4a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbd980cddd0 con 0x7fbd980d06c0 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9d1e7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9815c160 0x7fbd980aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9d1e7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9815c160 0x7fbd980aef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54024/0 (socket says 172.21.15.44:54024) 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9d1e7640 1 -- 172.21.15.44:0/2467347477 learned_addr learned my addr 172.21.15.44:0/2467347477 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd97fff640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd980d06c0 0x7fbd980ae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9c9e6640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd9806b4d0 0x7fbd980c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9d1e7640 1 -- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd980d06c0 msgr2=0x7fbd980ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9d1e7640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd980d06c0 0x7fbd980ae9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9d1e7640 1 -- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd9806b4d0 msgr2=0x7fbd980c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9d1e7640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd9806b4d0 0x7fbd980c7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9d1e7640 1 -- 172.21.15.44:0/2467347477 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd9809ca00 con 0x7fbd9815c160 2024-04-02T13:21:49.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9d1e7640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9815c160 0x7fbd980aef10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbd8800c480 tx=0x7fbd8800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:49.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd957fa640 1 -- 172.21.15.44:0/2467347477 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88017070 con 0x7fbd9815c160 2024-04-02T13:21:49.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd957fa640 1 -- 172.21.15.44:0/2467347477 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd88005ce0 con 0x7fbd9815c160 2024-04-02T13:21:49.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/2467347477 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd9809cce0 con 0x7fbd9815c160 2024-04-02T13:21:49.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd957fa640 1 -- 172.21.15.44:0/2467347477 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88007270 con 0x7fbd9815c160 2024-04-02T13:21:49.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.688+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/2467347477 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd9809d2a0 con 0x7fbd9815c160 2024-04-02T13:21:49.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.689+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/2467347477 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd980684f0 con 0x7fbd9815c160 2024-04-02T13:21:49.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.690+0000 7fbd957fa640 1 -- 172.21.15.44:0/2467347477 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbd8801a070 con 0x7fbd9815c160 2024-04-02T13:21:49.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.690+0000 7fbd957fa640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd8406c8f0 0x7fbd8406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:49.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.690+0000 7fbd957fa640 1 -- 172.21.15.44:0/2467347477 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbd8808af80 con 0x7fbd9815c160 2024-04-02T13:21:49.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.691+0000 7fbd9c9e6640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd8406c8f0 0x7fbd8406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:49.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.691+0000 7fbd9c9e6640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd8406c8f0 0x7fbd8406edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbd90001460 tx=0x7fbd90009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:49.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.695+0000 7fbd957fa640 1 -- 172.21.15.44:0/2467347477 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbd88058d20 con 0x7fbd9815c160 2024-04-02T13:21:49.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.917+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/2467347477 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbd9809e840 con 0x7fbd9815c160 2024-04-02T13:21:49.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.917+0000 7fbd957fa640 1 -- 172.21.15.44:0/2467347477 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+73 (secure 0 0 0) 0x7fbd8805f030 con 0x7fbd9815c160 2024-04-02T13:21:49.920 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:21:49.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.919+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd8406c8f0 msgr2=0x7fbd8406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.919+0000 7fbd9ec4a640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd8406c8f0 0x7fbd8406edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbd90001460 tx=0x7fbd90009040 comp rx=0 tx=0).stop 2024-04-02T13:21:49.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.919+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/2467347477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9815c160 msgr2=0x7fbd980aef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:49.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.919+0000 7fbd9ec4a640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9815c160 0x7fbd980aef10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbd8800c480 tx=0x7fbd8800c940 comp rx=0 tx=0).stop 2024-04-02T13:21:49.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.919+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/2467347477 shutdown_connections 2024-04-02T13:21:49.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.919+0000 7fbd9ec4a640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd8406c8f0 0x7fbd8406edb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.919+0000 7fbd9ec4a640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9815c160 0x7fbd980aef10 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.919+0000 7fbd9ec4a640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd980d06c0 0x7fbd980ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.919+0000 7fbd9ec4a640 1 --2- 172.21.15.44:0/2467347477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd9806b4d0 0x7fbd980c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:49.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.919+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/2467347477 >> 172.21.15.44:0/2467347477 conn(0x7fbd980b2980 msgr2=0x7fbd980b1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:49.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.919+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/2467347477 shutdown_connections 2024-04-02T13:21:49.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:49.919+0000 7fbd9ec4a640 1 -- 172.21.15.44:0/2467347477 wait complete. 2024-04-02T13:21:49.932 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:21:49.932 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:21:49.932 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:21:49.932 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:21:49.932 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:21:49.932 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:21:49.933 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:21:49.933 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:21:49.933 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:21:49.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:49.933 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:21:49.948 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:21:49.948 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:21:49.948 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:21:49.948 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:49.948 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:21:49.948 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:49.948 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:21:49.948 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42669sec preferred_lft 42669sec 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:21:49.949 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:21:49.950 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:21:49.950 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:21:49.950 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:49.950 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:21:49.950 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:49.950 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:21:49.950 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:21:49.950 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:21:49.950 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:49.950 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:21:49.950 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:21:49.950 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:21:49.950 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:21:49.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:49.951 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:21:50.013 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:21:50.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:50.015 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:21:50.070 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:21:50.070 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:21:50.070 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:21:50.070 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:21:50.071 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:21:50.071 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:21:50.071 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:21:50.071 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:21:50.071 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:21:50.071 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:21:50.071 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:21:50.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:21:50.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:50.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:50.205 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:21:50.220 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:21:50.220 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:21:50.220 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:21:50.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:50.220 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:21:50.287 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:21:50.287 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:21:50.287 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:21:50.341 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:21:50.342 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:21:50.342 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:21:50.397 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:21:50.464 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:21:50.520 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:21:50.520 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:21:50.520 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:21:50.520 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:21:50.520 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:21:50.520 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:21:50.520 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:21:50.520 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:21:50.520 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:21:50.520 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:21:50.520 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:21:50.521 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:21:50.522 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:21:50.522 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:21:50.522 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:21:50.522 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:21:50.522 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:21:50.522 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:21:50.522 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:21:50.522 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:21:50.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:50.522 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:21:50.574 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:21:50.574 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:21:50.617 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:21:50.643 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:21:50.698 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:21:50.698 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:21:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:21:50.700 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:21:50.701 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:21:50.701 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:21:50.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:50.701 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:21:50.702 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:21:50.700+0000 7f6820e4d480 -1 init, newargv = 0x55e11ded73f0 newargc=15 2024-04-02T13:21:50.702 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[53818]: starting ceph client 2024-04-02T13:21:50.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[53818]: starting fuse 2024-04-02T13:21:50.717 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:21:50.717 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:21:50.717 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:21:50.718 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:21:50.718 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:21:50.718 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:21:50.718 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:21:50.718 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:21:50.718 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:21:50.718 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:21:50.718 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:21:50.718 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:21:50.718 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:21:50.718 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:21:50.718 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:21:50.719 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:21:50.838 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.53818.asok 2024-04-02T13:21:50.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:50.848 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.53818.asok 2024-04-02T13:21:50.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:50.849 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.53818.asok status 2024-04-02T13:21:50.985 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:21:50.986 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:21:50.986 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:21:50.986 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:21:50.986 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:21:50.986 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:21:50.986 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:21:50.986 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "53818", 2024-04-02T13:21:50.986 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:21:50.986 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:50.986 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:21:50.986 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:21:50.986 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 5712, 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 5712 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3523535107 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:21:50.987 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3523535107 2024-04-02T13:21:50.988 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:21:50.988 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.5712 192.168.0.1:0/3523535107", 2024-04-02T13:21:50.988 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/3523535107", 2024-04-02T13:21:50.988 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:21:50.988 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 18, 2024-04-02T13:21:50.988 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 39, 2024-04-02T13:21:50.988 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:21:50.988 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:21:50.988 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:21:50.988 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:21:50.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:50.994 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:21:50.994 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:21:51.009 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:21:51.009 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:21:51.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:51.009 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:21:51.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:51.077 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir) 2024-04-02T13:21:51.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:51.137 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir/childdir) 2024-04-02T13:21:51.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:51.194 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch rfile) 2024-04-02T13:21:51.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:51.251 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/pfile) 2024-04-02T13:21:51.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:51.310 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/childdir/cfile) 2024-04-02T13:21:51.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:51.367 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-04-02T13:21:51.430 INFO:teuthology.orchestra.run.smithi044.stdout:./ 2024-04-02T13:21:51.430 INFO:teuthology.orchestra.run.smithi044.stdout:./rfile 2024-04-02T13:21:51.430 INFO:teuthology.orchestra.run.smithi044.stdout:./parentdir 2024-04-02T13:21:51.430 INFO:teuthology.orchestra.run.smithi044.stdout:./parentdir/pfile 2024-04-02T13:21:51.430 INFO:teuthology.orchestra.run.smithi044.stdout:./parentdir/childdir 2024-04-02T13:21:51.431 INFO:teuthology.orchestra.run.smithi044.stdout:./parentdir/childdir/cfile 2024-04-02T13:21:51.431 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:21:51.431 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:21:51.431 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:21:51.431 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:21:51.431 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-04-02T13:21:51.431 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:21:51.538 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:21:51.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:51.547 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:21:51.547 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:21:51.547 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:21:51.547 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:21:51.547 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./rfile").st_ino) 2024-04-02T13:21:51.547 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:21:51.653 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627778 2024-04-02T13:21:51.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:51.661 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:21:51.661 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:21:51.661 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:21:51.661 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:21:51.661 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir").st_ino) 2024-04-02T13:21:51.662 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:21:51.766 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627776 2024-04-02T13:21:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:51.775 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:21:51.775 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:21:51.775 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:21:51.775 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:21:51.775 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/pfile").st_ino) 2024-04-02T13:21:51.775 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:21:51.831 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:21:51.832 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:21:51.882 INFO:teuthology.orchestra.run.smithi044.stdout:1099511628279 2024-04-02T13:21:51.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:51.892 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:21:51.892 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:21:51.892 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:21:51.892 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:21:51.892 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir").st_ino) 2024-04-02T13:21:51.892 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:21:51.997 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627777 2024-04-02T13:21:52.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:52.006 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:21:52.006 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:21:52.006 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:21:52.006 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:21:52.006 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir/cfile").st_ino) 2024-04-02T13:21:52.006 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:21:52.112 INFO:teuthology.orchestra.run.smithi044.stdout:1099511628280 2024-04-02T13:21:52.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:21:52.120 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:21:52.120 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:52.176 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.315+0000 7f8f4fd7b640 1 Processor -- start 2024-04-02T13:21:52.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.315+0000 7f8f4fd7b640 1 -- start start 2024-04-02T13:21:52.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.316+0000 7f8f4fd7b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8f480d09f0 0x7f8f4806d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.316+0000 7f8f4fd7b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f4806ece0 0x7f8f480acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.316+0000 7f8f4fd7b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f4806dc40 con 0x7f8f480cece0 2024-04-02T13:21:52.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.316+0000 7f8f4fd7b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f8f4806df20 con 0x7f8f480d09f0 2024-04-02T13:21:52.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.316+0000 7f8f4fd7b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8f4806e200 con 0x7f8f4806ece0 2024-04-02T13:21:52.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.316+0000 7f8f4d316640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8f480d09f0 0x7f8f4806d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.316+0000 7f8f4d316640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8f480d09f0 0x7f8f4806d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54030/0 (socket says 172.21.15.44:54030) 2024-04-02T13:21:52.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.316+0000 7f8f4d316640 1 -- 172.21.15.44:0/2105533921 learned_addr learned my addr 172.21.15.44:0/2105533921 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:52.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.316+0000 7f8f4e318640 1 --2- 172.21.15.44:0/2105533921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f4806ece0 0x7f8f480acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.316+0000 7f8f4d316640 1 -- 172.21.15.44:0/2105533921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f4806ece0 msgr2=0x7f8f480acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:52.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.316+0000 7f8f4d316640 1 --2- 172.21.15.44:0/2105533921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f4806ece0 0x7f8f480acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.316+0000 7f8f4d316640 1 -- 172.21.15.44:0/2105533921 >> v1:172.21.15.69:6789/0 conn(0x7f8f480cece0 legacy=0x7f8f480cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.316+0000 7f8f4d316640 1 -- 172.21.15.44:0/2105533921 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f4806e4e0 con 0x7f8f480d09f0 2024-04-02T13:21:52.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.317+0000 7f8f4d316640 1 --2- 172.21.15.44:0/2105533921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f480d09f0 0x7f8f4806d2d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800bb10 tx=0x7f8f3802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212fd763130c90f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:52.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.317+0000 7f8f4cb15640 1 -- 172.21.15.44:0/2105533921 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3803b070 con 0x7f8f480d09f0 2024-04-02T13:21:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.317+0000 7f8f4cb15640 1 -- 172.21.15.44:0/2105533921 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f38036440 con 0x7f8f480d09f0 2024-04-02T13:21:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.317+0000 7f8f4cb15640 1 -- 172.21.15.44:0/2105533921 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f38036810 con 0x7f8f480d09f0 2024-04-02T13:21:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.318+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/2105533921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f480d09f0 msgr2=0x7f8f4806d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.318+0000 7f8f4fd7b640 1 --2- 172.21.15.44:0/2105533921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f480d09f0 0x7f8f4806d2d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800bb10 tx=0x7f8f3802e150 comp rx=0 tx=0).stop 2024-04-02T13:21:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.318+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/2105533921 shutdown_connections 2024-04-02T13:21:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.318+0000 7f8f4fd7b640 1 --2- 172.21.15.44:0/2105533921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f480d09f0 0x7f8f4806d2d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.318+0000 7f8f4fd7b640 1 --2- 172.21.15.44:0/2105533921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f4806ece0 0x7f8f480acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.318+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/2105533921 >> 172.21.15.44:0/2105533921 conn(0x7f8f480cfcd0 msgr2=0x7f8f480d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.318+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/2105533921 shutdown_connections 2024-04-02T13:21:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.318+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/2105533921 wait complete. 2024-04-02T13:21:52.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.319+0000 7f8f4fd7b640 1 Processor -- start 2024-04-02T13:21:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.319+0000 7f8f4fd7b640 1 -- start start 2024-04-02T13:21:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.319+0000 7f8f4fd7b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f4806ece0 0x7f8f4814e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.319+0000 7f8f4fd7b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f480cece0 0x7f8f4809c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.319+0000 7f8f4fd7b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f480d09f0 0x7f8f480c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.319+0000 7f8f4fd7b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4806d990 con 0x7f8f480d09f0 2024-04-02T13:21:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.319+0000 7f8f4fd7b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4806d810 con 0x7f8f480cece0 2024-04-02T13:21:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.319+0000 7f8f4fd7b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8f4815b220 con 0x7f8f4806ece0 2024-04-02T13:21:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f4e318640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f480d09f0 0x7f8f480c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f4e318640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f480d09f0 0x7f8f480c4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54042/0 (socket says 172.21.15.44:54042) 2024-04-02T13:21:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f4e318640 1 -- 172.21.15.44:0/3131284777 learned_addr learned my addr 172.21.15.44:0/3131284777 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f4e318640 1 -- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f4806ece0 msgr2=0x7f8f4814e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f4db17640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f4806ece0 0x7f8f4814e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f4e318640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f4806ece0 0x7f8f4814e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f4e318640 1 -- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f480cece0 msgr2=0x7f8f4809c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f4d316640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f480cece0 0x7f8f4809c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f4e318640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f480cece0 0x7f8f4809c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f4e318640 1 -- 172.21.15.44:0/3131284777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f4814ed10 con 0x7f8f480d09f0 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f4e318640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f480d09f0 0x7f8f480c4c70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8f40007e30 tx=0x7f8f4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f3e7fc640 1 -- 172.21.15.44:0/3131284777 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f40014620 con 0x7f8f480d09f0 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f3e7fc640 1 -- 172.21.15.44:0/3131284777 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f40019070 con 0x7f8f480d09f0 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/3131284777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f4809cd70 con 0x7f8f480d09f0 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f3e7fc640 1 -- 172.21.15.44:0/3131284777 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f40014a00 con 0x7f8f480d09f0 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.320+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/3131284777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f4815c310 con 0x7f8f480d09f0 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.321+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/3131284777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f48074f20 con 0x7f8f480d09f0 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.322+0000 7f8f3e7fc640 1 -- 172.21.15.44:0/3131284777 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8f40014ba0 con 0x7f8f480d09f0 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.322+0000 7f8f3e7fc640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8f2c06c8f0 0x7f8f2c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.322+0000 7f8f3e7fc640 1 -- 172.21.15.44:0/3131284777 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f8f4008c570 con 0x7f8f480d09f0 2024-04-02T13:21:52.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.322+0000 7f8f4db17640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8f2c06c8f0 0x7f8f2c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.323+0000 7f8f4db17640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8f2c06c8f0 0x7f8f2c06edb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8f440015a0 tx=0x7f8f44009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:52.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.327+0000 7f8f3e7fc640 1 -- 172.21.15.44:0/3131284777 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8f4814ed10 con 0x7f8f480d09f0 2024-04-02T13:21:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.517+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/3131284777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8f480684f0 con 0x7f8f480d09f0 2024-04-02T13:21:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.518+0000 7f8f3e7fc640 1 -- 172.21.15.44:0/3131284777 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16894 (secure 0 0 0) 0x7f8f4005a8a0 con 0x7f8f480d09f0 2024-04-02T13:21:52.521 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:52.521 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":39,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:44.874189+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:21:41.647497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-02T13:21:42.168299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:52.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.520+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8f2c06c8f0 msgr2=0x7f8f2c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.520+0000 7f8f4fd7b640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8f2c06c8f0 0x7f8f2c06edb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8f440015a0 tx=0x7f8f44009040 comp rx=0 tx=0).stop 2024-04-02T13:21:52.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.520+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/3131284777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f480d09f0 msgr2=0x7f8f480c4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.520+0000 7f8f4fd7b640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f480d09f0 0x7f8f480c4c70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8f40007e30 tx=0x7f8f4000e4b0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.520+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/3131284777 shutdown_connections 2024-04-02T13:21:52.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.520+0000 7f8f4fd7b640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8f2c06c8f0 0x7f8f2c06edb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.520+0000 7f8f4fd7b640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f480d09f0 0x7f8f480c4c70 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.520+0000 7f8f4fd7b640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f480cece0 0x7f8f4809c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.520+0000 7f8f4fd7b640 1 --2- 172.21.15.44:0/3131284777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f4806ece0 0x7f8f4814e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.520+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/3131284777 >> 172.21.15.44:0/3131284777 conn(0x7f8f480cfcd0 msgr2=0x7f8f480acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:52.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.520+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/3131284777 shutdown_connections 2024-04-02T13:21:52.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.520+0000 7f8f4fd7b640 1 -- 172.21.15.44:0/3131284777 wait complete. 2024-04-02T13:21:52.533 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:21:41.647497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:21:42.168299+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:21:52.534 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:52.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.636+0000 7fc1144d3640 1 Processor -- start 2024-04-02T13:21:52.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.636+0000 7fc1144d3640 1 -- start start 2024-04-02T13:21:52.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc1144d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc10c15c1a0 0x7fc10c06f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc1144d3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc10c0cfd60 0x7fc10c0d1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc1144d3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc10c0d1ff0 0x7fc10c0ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc1144d3640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc10c070ad0 con 0x7fc10c0cfd60 2024-04-02T13:21:52.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc1144d3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc10c15c580 con 0x7fc10c0d1ff0 2024-04-02T13:21:52.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc1144d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc10c0d01c0 con 0x7fc10c15c1a0 2024-04-02T13:21:52.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc111a6e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc10c0cfd60 0x7fc10c0d1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc111a6e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc10c0cfd60 0x7fc10c0d1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54044/0 (socket says 172.21.15.44:54044) 2024-04-02T13:21:52.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc111a6e640 1 -- 172.21.15.44:0/819813235 learned_addr learned my addr 172.21.15.44:0/819813235 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:52.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc11226f640 1 --2- 172.21.15.44:0/819813235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc10c15c1a0 0x7fc10c06f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc112a70640 1 --2- 172.21.15.44:0/819813235 >> v2:172.21.15.69:3300/0 conn(0x7fc10c0d1ff0 0x7fc10c0ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc111a6e640 1 -- 172.21.15.44:0/819813235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc10c15c1a0 msgr2=0x7fc10c06f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:52.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc111a6e640 1 --2- 172.21.15.44:0/819813235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc10c15c1a0 0x7fc10c06f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc111a6e640 1 -- 172.21.15.44:0/819813235 >> v2:172.21.15.69:3300/0 conn(0x7fc10c0d1ff0 msgr2=0x7fc10c0ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:52.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc111a6e640 1 --2- 172.21.15.44:0/819813235 >> v2:172.21.15.69:3300/0 conn(0x7fc10c0d1ff0 0x7fc10c0ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.637+0000 7fc111a6e640 1 -- 172.21.15.44:0/819813235 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10c0d2480 con 0x7fc10c0cfd60 2024-04-02T13:21:52.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.638+0000 7fc111a6e640 1 --2- 172.21.15.44:0/819813235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc10c0cfd60 0x7fc10c0d1ab0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc10c09b990 tx=0x7fc0fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5d39609352ca57 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:52.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.638+0000 7fc11126d640 1 -- 172.21.15.44:0/819813235 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc035020 con 0x7fc10c0cfd60 2024-04-02T13:21:52.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.638+0000 7fc11126d640 1 -- 172.21.15.44:0/819813235 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0fc031870 con 0x7fc10c0cfd60 2024-04-02T13:21:52.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.638+0000 7fc11126d640 1 -- 172.21.15.44:0/819813235 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc031c40 con 0x7fc10c0cfd60 2024-04-02T13:21:52.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.639+0000 7fc1144d3640 1 -- 172.21.15.44:0/819813235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc10c0cfd60 msgr2=0x7fc10c0d1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.639+0000 7fc1144d3640 1 --2- 172.21.15.44:0/819813235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc10c0cfd60 0x7fc10c0d1ab0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc10c09b990 tx=0x7fc0fc02b910 comp rx=0 tx=0).stop 2024-04-02T13:21:52.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.639+0000 7fc1144d3640 1 -- 172.21.15.44:0/819813235 shutdown_connections 2024-04-02T13:21:52.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.639+0000 7fc1144d3640 1 --2- 172.21.15.44:0/819813235 >> v2:172.21.15.69:3300/0 conn(0x7fc10c0d1ff0 0x7fc10c0ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.639+0000 7fc1144d3640 1 --2- 172.21.15.44:0/819813235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc10c0cfd60 0x7fc10c0d1ab0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.639+0000 7fc1144d3640 1 --2- 172.21.15.44:0/819813235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc10c15c1a0 0x7fc10c06f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.639+0000 7fc1144d3640 1 -- 172.21.15.44:0/819813235 >> 172.21.15.44:0/819813235 conn(0x7fc10c09aa20 msgr2=0x7fc10c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:52.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.639+0000 7fc1144d3640 1 -- 172.21.15.44:0/819813235 shutdown_connections 2024-04-02T13:21:52.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.639+0000 7fc1144d3640 1 -- 172.21.15.44:0/819813235 wait complete. 2024-04-02T13:21:52.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.640+0000 7fc1144d3640 1 Processor -- start 2024-04-02T13:21:52.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.640+0000 7fc1144d3640 1 -- start start 2024-04-02T13:21:52.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc1144d3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc10c0cfd60 0x7fc10c0cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc1144d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc10c0d1ff0 0x7fc10c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc1144d3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc10c15c1a0 0x7fc10c0c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc1144d3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c06fff0 con 0x7fc10c0cfd60 2024-04-02T13:21:52.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc1144d3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c06fd10 con 0x7fc10c15c1a0 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc1144d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc10c0702d0 con 0x7fc10c0d1ff0 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc11226f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc10c0cfd60 0x7fc10c0cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc11226f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc10c0cfd60 0x7fc10c0cb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54050/0 (socket says 172.21.15.44:54050) 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc11226f640 1 -- 172.21.15.44:0/3756074104 learned_addr learned my addr 172.21.15.44:0/3756074104 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc112a70640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc10c15c1a0 0x7fc10c0c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc111a6e640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc10c0d1ff0 0x7fc10c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc11226f640 1 -- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc10c0d1ff0 msgr2=0x7fc10c15f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc11226f640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc10c0d1ff0 0x7fc10c15f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc11226f640 1 -- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc10c15c1a0 msgr2=0x7fc10c0c6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc11226f640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc10c15c1a0 0x7fc10c0c6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc11226f640 1 -- 172.21.15.44:0/3756074104 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10c15f590 con 0x7fc10c0cfd60 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc112a70640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc10c15c1a0 0x7fc10c0c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc11226f640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc10c0cfd60 0x7fc10c0cb870 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc104007e40 tx=0x7fc10400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:52.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc102ffd640 1 -- 172.21.15.44:0/3756074104 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc104014620 con 0x7fc10c0cfd60 2024-04-02T13:21:52.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc102ffd640 1 -- 172.21.15.44:0/3756074104 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc104019070 con 0x7fc10c0cfd60 2024-04-02T13:21:52.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc1144d3640 1 -- 172.21.15.44:0/3756074104 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc10c0c6c70 con 0x7fc10c0cfd60 2024-04-02T13:21:52.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.641+0000 7fc102ffd640 1 -- 172.21.15.44:0/3756074104 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc104014a00 con 0x7fc10c0cfd60 2024-04-02T13:21:52.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.642+0000 7fc1144d3640 1 -- 172.21.15.44:0/3756074104 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc10c0cb070 con 0x7fc10c0cfd60 2024-04-02T13:21:52.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.642+0000 7fc1144d3640 1 -- 172.21.15.44:0/3756074104 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc10c068c00 con 0x7fc10c0cfd60 2024-04-02T13:21:52.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.643+0000 7fc102ffd640 1 -- 172.21.15.44:0/3756074104 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc104014ba0 con 0x7fc10c0cfd60 2024-04-02T13:21:52.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.643+0000 7fc102ffd640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0f006c920 0x7fc0f006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.643+0000 7fc102ffd640 1 -- 172.21.15.44:0/3756074104 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fc10408c4d0 con 0x7fc10c0cfd60 2024-04-02T13:21:52.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.643+0000 7fc111a6e640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0f006c920 0x7fc0f006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.644+0000 7fc111a6e640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0f006c920 0x7fc0f006ede0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc036000 tx=0x7fc0fc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:52.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.648+0000 7fc102ffd640 1 -- 172.21.15.44:0/3756074104 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc10405a800 con 0x7fc10c0cfd60 2024-04-02T13:21:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.831+0000 7fc1144d3640 1 -- 172.21.15.44:0/3756074104 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc10c0cdd00 con 0x7fc10c0cfd60 2024-04-02T13:21:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.831+0000 7fc102ffd640 1 -- 172.21.15.44:0/3756074104 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7fc10405a100 con 0x7fc10c0cfd60 2024-04-02T13:21:52.834 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.833+0000 7fc1144d3640 1 -- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0f006c920 msgr2=0x7fc0f006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.833+0000 7fc1144d3640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0f006c920 0x7fc0f006ede0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc036000 tx=0x7fc0fc03c040 comp rx=0 tx=0).stop 2024-04-02T13:21:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.833+0000 7fc1144d3640 1 -- 172.21.15.44:0/3756074104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc10c0cfd60 msgr2=0x7fc10c0cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.833+0000 7fc1144d3640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc10c0cfd60 0x7fc10c0cb870 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc104007e40 tx=0x7fc10400e4b0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.833+0000 7fc1144d3640 1 -- 172.21.15.44:0/3756074104 shutdown_connections 2024-04-02T13:21:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.833+0000 7fc1144d3640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0f006c920 0x7fc0f006ede0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.833+0000 7fc1144d3640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc10c15c1a0 0x7fc10c0c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.833+0000 7fc1144d3640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc10c0d1ff0 0x7fc10c15f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.833+0000 7fc1144d3640 1 --2- 172.21.15.44:0/3756074104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc10c0cfd60 0x7fc10c0cb870 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.833+0000 7fc1144d3640 1 -- 172.21.15.44:0/3756074104 >> 172.21.15.44:0/3756074104 conn(0x7fc10c09aa20 msgr2=0x7fc10c0b2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.833+0000 7fc1144d3640 1 -- 172.21.15.44:0/3756074104 shutdown_connections 2024-04-02T13:21:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.833+0000 7fc1144d3640 1 -- 172.21.15.44:0/3756074104 wait complete. 2024-04-02T13:21:52.846 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:52.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.948+0000 7f07dafd6640 1 Processor -- start 2024-04-02T13:21:52.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.948+0000 7f07dafd6640 1 -- start start 2024-04-02T13:21:52.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.949+0000 7f07dafd6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f07d415e470 0x7f07d40b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.949+0000 7f07dafd6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07d40b1540 0x7f07d40d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.949+0000 7f07dafd6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d40ced00 con 0x7f07d40ce790 2024-04-02T13:21:52.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.949+0000 7f07dafd6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f07d40cefe0 con 0x7f07d415e470 2024-04-02T13:21:52.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.949+0000 7f07dafd6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f07d40cf2c0 con 0x7f07d40b1540 2024-04-02T13:21:52.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.949+0000 7f07d3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07d40b1540 0x7f07d40d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.949+0000 7f07d3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07d40b1540 0x7f07d40d1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:51636/0 (socket says 172.21.15.44:51636) 2024-04-02T13:21:52.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.949+0000 7f07d3fff640 1 -- 172.21.15.44:0/627975094 learned_addr learned my addr 172.21.15.44:0/627975094 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:52.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.949+0000 7f07d8d72640 1 --2- 172.21.15.44:0/627975094 >> v2:172.21.15.69:3300/0 conn(0x7f07d415e470 0x7f07d40b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.950+0000 7f07d8d72640 1 -- 172.21.15.44:0/627975094 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07d40b1540 msgr2=0x7f07d40d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:52.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.950+0000 7f07d8d72640 1 --2- 172.21.15.44:0/627975094 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07d40b1540 0x7f07d40d1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.950+0000 7f07d8d72640 1 -- 172.21.15.44:0/627975094 >> v1:172.21.15.69:6789/0 conn(0x7f07d40ce790 legacy=0x7f07d4161fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.950+0000 7f07d8d72640 1 -- 172.21.15.44:0/627975094 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d40cf5a0 con 0x7f07d415e470 2024-04-02T13:21:52.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.950+0000 7f07d3fff640 1 --2- 172.21.15.44:0/627975094 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07d40b1540 0x7f07d40d1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:21:52.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.951+0000 7f07d8d72640 1 --2- 172.21.15.44:0/627975094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07d415e470 0x7f07d40b32c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009d90 tx=0x7f07cc02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbc5243ee88c2921 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:52.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.951+0000 7f07d37fe640 1 -- 172.21.15.44:0/627975094 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc039070 con 0x7f07d415e470 2024-04-02T13:21:52.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.951+0000 7f07d37fe640 1 -- 172.21.15.44:0/627975094 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07cc02cbc0 con 0x7f07d415e470 2024-04-02T13:21:52.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.951+0000 7f07d37fe640 1 -- 172.21.15.44:0/627975094 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc034440 con 0x7f07d415e470 2024-04-02T13:21:52.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.951+0000 7f07dafd6640 1 -- 172.21.15.44:0/627975094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07d415e470 msgr2=0x7f07d40b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.951+0000 7f07dafd6640 1 --2- 172.21.15.44:0/627975094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07d415e470 0x7f07d40b32c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009d90 tx=0x7f07cc02be10 comp rx=0 tx=0).stop 2024-04-02T13:21:52.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.951+0000 7f07dafd6640 1 -- 172.21.15.44:0/627975094 shutdown_connections 2024-04-02T13:21:52.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.951+0000 7f07dafd6640 1 --2- 172.21.15.44:0/627975094 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07d40b1540 0x7f07d40d1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.951+0000 7f07dafd6640 1 --2- 172.21.15.44:0/627975094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07d415e470 0x7f07d40b32c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.951+0000 7f07dafd6640 1 -- 172.21.15.44:0/627975094 >> 172.21.15.44:0/627975094 conn(0x7f07d409aa20 msgr2=0x7f07d409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:52.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.951+0000 7f07dafd6640 1 -- 172.21.15.44:0/627975094 shutdown_connections 2024-04-02T13:21:52.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.951+0000 7f07dafd6640 1 -- 172.21.15.44:0/627975094 wait complete. 2024-04-02T13:21:52.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.953+0000 7f07dafd6640 1 Processor -- start 2024-04-02T13:21:52.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.953+0000 7f07dafd6640 1 -- start start 2024-04-02T13:21:52.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.953+0000 7f07dafd6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07d40b1540 0x7f07d40b59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.953+0000 7f07dafd6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07d40ce790 0x7f07d40b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.953+0000 7f07dafd6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07d415e470 0x7f07d406f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.953+0000 7f07dafd6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f07d40d20f0 con 0x7f07d415e470 2024-04-02T13:21:52.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.953+0000 7f07dafd6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f07d406cb60 con 0x7f07d40ce790 2024-04-02T13:21:52.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.953+0000 7f07dafd6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f07d40d2270 con 0x7f07d40b1540 2024-04-02T13:21:52.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d9573640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07d415e470 0x7f07d406f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d9573640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07d415e470 0x7f07d406f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54052/0 (socket says 172.21.15.44:54052) 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d9573640 1 -- 172.21.15.44:0/3696067683 learned_addr learned my addr 172.21.15.44:0/3696067683 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d3fff640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07d40ce790 0x7f07d40b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d8d72640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07d40b1540 0x7f07d40b59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d9573640 1 -- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07d40b1540 msgr2=0x7f07d40b59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d9573640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07d40b1540 0x7f07d40b59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d9573640 1 -- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07d40ce790 msgr2=0x7f07d40b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d9573640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07d40ce790 0x7f07d40b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d9573640 1 -- 172.21.15.44:0/3696067683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d40b67f0 con 0x7f07d415e470 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d9573640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07d415e470 0x7f07d406f900 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f07c400c400 tx=0x7f07c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d17fa640 1 -- 172.21.15.44:0/3696067683 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c4017070 con 0x7f07d415e470 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d17fa640 1 -- 172.21.15.44:0/3696067683 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07c40052d0 con 0x7f07d415e470 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07dafd6640 1 -- 172.21.15.44:0/3696067683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d40b6ad0 con 0x7f07d415e470 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07d17fa640 1 -- 172.21.15.44:0/3696067683 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c4003810 con 0x7f07d415e470 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.954+0000 7f07dafd6640 1 -- 172.21.15.44:0/3696067683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d40b7090 con 0x7f07d415e470 2024-04-02T13:21:52.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.955+0000 7f07dafd6640 1 -- 172.21.15.44:0/3696067683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07d40684f0 con 0x7f07d415e470 2024-04-02T13:21:52.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.956+0000 7f07d17fa640 1 -- 172.21.15.44:0/3696067683 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f07c401a070 con 0x7f07d415e470 2024-04-02T13:21:52.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.956+0000 7f07d17fa640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f07c006c8f0 0x7f07c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:52.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.956+0000 7f07d17fa640 1 -- 172.21.15.44:0/3696067683 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f07c408aec0 con 0x7f07d415e470 2024-04-02T13:21:52.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.956+0000 7f07d8d72640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f07c006c8f0 0x7f07c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:52.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.957+0000 7f07d8d72640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f07c006c8f0 0x7f07c006edb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f07cc036000 tx=0x7f07cc030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:52.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:52.961+0000 7f07d17fa640 1 -- 172.21.15.44:0/3696067683 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f07c4058ba0 con 0x7f07d415e470 2024-04-02T13:21:53.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.143+0000 7f07dafd6640 1 -- 172.21.15.44:0/3696067683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f07d40b7290 con 0x7f07d415e470 2024-04-02T13:21:53.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.143+0000 7f07d17fa640 1 -- 172.21.15.44:0/3696067683 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7f07c405cea0 con 0x7f07d415e470 2024-04-02T13:21:53.146 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:53.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.145+0000 7f07dafd6640 1 -- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f07c006c8f0 msgr2=0x7f07c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.145+0000 7f07dafd6640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f07c006c8f0 0x7f07c006edb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f07cc036000 tx=0x7f07cc030000 comp rx=0 tx=0).stop 2024-04-02T13:21:53.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.145+0000 7f07dafd6640 1 -- 172.21.15.44:0/3696067683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07d415e470 msgr2=0x7f07d406f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.145+0000 7f07dafd6640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07d415e470 0x7f07d406f900 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f07c400c400 tx=0x7f07c400c8c0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.145+0000 7f07dafd6640 1 -- 172.21.15.44:0/3696067683 shutdown_connections 2024-04-02T13:21:53.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.145+0000 7f07dafd6640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f07c006c8f0 0x7f07c006edb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.145+0000 7f07dafd6640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07d415e470 0x7f07d406f900 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.145+0000 7f07dafd6640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07d40ce790 0x7f07d40b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.145+0000 7f07dafd6640 1 --2- 172.21.15.44:0/3696067683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07d40b1540 0x7f07d40b59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.145+0000 7f07dafd6640 1 -- 172.21.15.44:0/3696067683 >> 172.21.15.44:0/3696067683 conn(0x7f07d409aa20 msgr2=0x7f07d40aee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:53.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.145+0000 7f07dafd6640 1 -- 172.21.15.44:0/3696067683 shutdown_connections 2024-04-02T13:21:53.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.146+0000 7f07dafd6640 1 -- 172.21.15.44:0/3696067683 wait complete. 2024-04-02T13:21:53.159 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.261+0000 7fa23759e640 1 Processor -- start 2024-04-02T13:21:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.261+0000 7fa23759e640 1 -- start start 2024-04-02T13:21:53.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.262+0000 7fa23759e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa23815e1d0 0x7fa23806d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.262+0000 7fa23759e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa2380d2030 0x7fa2380cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.262+0000 7fa23759e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2380d0d30 con 0x7fa2380d08a0 2024-04-02T13:21:53.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.262+0000 7fa23759e640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa2380b1970 con 0x7fa23815e1d0 2024-04-02T13:21:53.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.262+0000 7fa23759e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa2380d1010 con 0x7fa2380d2030 2024-04-02T13:21:53.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.262+0000 7fa23659c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa23815e1d0 0x7fa23806d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.262+0000 7fa23659c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa23815e1d0 0x7fa23806d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54062/0 (socket says 172.21.15.44:54062) 2024-04-02T13:21:53.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.262+0000 7fa23659c640 1 -- 172.21.15.44:0/2093113958 learned_addr learned my addr 172.21.15.44:0/2093113958 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:53.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.263+0000 7fa236d9d640 1 --2- 172.21.15.44:0/2093113958 >> v2:172.21.15.69:3300/0 conn(0x7fa2380d2030 0x7fa2380cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.263+0000 7fa23659c640 1 -- 172.21.15.44:0/2093113958 >> v2:172.21.15.69:3300/0 conn(0x7fa2380d2030 msgr2=0x7fa2380cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:53.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.263+0000 7fa23659c640 1 --2- 172.21.15.44:0/2093113958 >> v2:172.21.15.69:3300/0 conn(0x7fa2380d2030 0x7fa2380cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.263+0000 7fa23659c640 1 -- 172.21.15.44:0/2093113958 >> v1:172.21.15.69:6789/0 conn(0x7fa2380d08a0 legacy=0x7fa2380d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:53.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.263+0000 7fa23659c640 1 -- 172.21.15.44:0/2093113958 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2380b1bc0 con 0x7fa23815e1d0 2024-04-02T13:21:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.263+0000 7fa23659c640 1 --2- 172.21.15.44:0/2093113958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa23815e1d0 0x7fa23806d060 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa2380b7db0 tx=0x7fa23002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec812a89761e6cfa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.264+0000 7fa23559a640 1 -- 172.21.15.44:0/2093113958 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230035020 con 0x7fa23815e1d0 2024-04-02T13:21:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.264+0000 7fa23559a640 1 -- 172.21.15.44:0/2093113958 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa230031870 con 0x7fa23815e1d0 2024-04-02T13:21:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.264+0000 7fa23559a640 1 -- 172.21.15.44:0/2093113958 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230031c40 con 0x7fa23815e1d0 2024-04-02T13:21:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.264+0000 7fa23759e640 1 -- 172.21.15.44:0/2093113958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa23815e1d0 msgr2=0x7fa23806d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.264+0000 7fa23759e640 1 --2- 172.21.15.44:0/2093113958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa23815e1d0 0x7fa23806d060 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa2380b7db0 tx=0x7fa23002b910 comp rx=0 tx=0).stop 2024-04-02T13:21:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.264+0000 7fa23759e640 1 -- 172.21.15.44:0/2093113958 shutdown_connections 2024-04-02T13:21:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.264+0000 7fa23759e640 1 --2- 172.21.15.44:0/2093113958 >> v2:172.21.15.69:3300/0 conn(0x7fa2380d2030 0x7fa2380cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.264+0000 7fa23759e640 1 --2- 172.21.15.44:0/2093113958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa23815e1d0 0x7fa23806d060 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.264+0000 7fa23759e640 1 -- 172.21.15.44:0/2093113958 >> 172.21.15.44:0/2093113958 conn(0x7fa238073520 msgr2=0x7fa238074b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.264+0000 7fa23759e640 1 -- 172.21.15.44:0/2093113958 shutdown_connections 2024-04-02T13:21:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.264+0000 7fa23759e640 1 -- 172.21.15.44:0/2093113958 wait complete. 2024-04-02T13:21:53.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.266+0000 7fa23759e640 1 Processor -- start 2024-04-02T13:21:53.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.266+0000 7fa23759e640 1 -- start start 2024-04-02T13:21:53.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.266+0000 7fa23759e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2380d2030 0x7fa2380ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.266+0000 7fa23759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa23815e1d0 0x7fa23815fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.266+0000 7fa23759e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa238160080 0x7fa238153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.266+0000 7fa23759e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa23815e730 con 0x7fa2380d2030 2024-04-02T13:21:53.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.266+0000 7fa23759e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa23815e5b0 con 0x7fa23815e1d0 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.266+0000 7fa23759e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa23806d5a0 con 0x7fa238160080 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.266+0000 7fa23659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2380d2030 0x7fa2380ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.266+0000 7fa23659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2380d2030 0x7fa2380ca510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54064/0 (socket says 172.21.15.44:54064) 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.266+0000 7fa23659c640 1 -- 172.21.15.44:0/2344792975 learned_addr learned my addr 172.21.15.44:0/2344792975 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.267+0000 7fa236d9d640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa238160080 0x7fa238153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.267+0000 7fa235d9b640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa23815e1d0 0x7fa23815fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.267+0000 7fa23659c640 1 -- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa238160080 msgr2=0x7fa238153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.267+0000 7fa23659c640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa238160080 0x7fa238153020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.267+0000 7fa23659c640 1 -- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa23815e1d0 msgr2=0x7fa23815fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.267+0000 7fa23659c640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa23815e1d0 0x7fa23815fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.267+0000 7fa23659c640 1 -- 172.21.15.44:0/2344792975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2380b7750 con 0x7fa2380d2030 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.267+0000 7fa23659c640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2380d2030 0x7fa2380ca510 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa230000c00 tx=0x7fa2300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.267+0000 7fa22effd640 1 -- 172.21.15.44:0/2344792975 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230040070 con 0x7fa2380d2030 2024-04-02T13:21:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.267+0000 7fa22effd640 1 -- 172.21.15.44:0/2344792975 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa23003a420 con 0x7fa2380d2030 2024-04-02T13:21:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.267+0000 7fa22effd640 1 -- 172.21.15.44:0/2344792975 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230035040 con 0x7fa2380d2030 2024-04-02T13:21:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.267+0000 7fa23759e640 1 -- 172.21.15.44:0/2344792975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2381606b0 con 0x7fa2380d2030 2024-04-02T13:21:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.267+0000 7fa23759e640 1 -- 172.21.15.44:0/2344792975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa238160c70 con 0x7fa2380d2030 2024-04-02T13:21:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.268+0000 7fa23759e640 1 -- 172.21.15.44:0/2344792975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa23804f860 con 0x7fa2380d2030 2024-04-02T13:21:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.269+0000 7fa22effd640 1 -- 172.21.15.44:0/2344792975 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa23004b060 con 0x7fa2380d2030 2024-04-02T13:21:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.269+0000 7fa22effd640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa21806c8f0 0x7fa21806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.269+0000 7fa22effd640 1 -- 172.21.15.44:0/2344792975 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fa2300ada60 con 0x7fa2380d2030 2024-04-02T13:21:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.269+0000 7fa235d9b640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa21806c8f0 0x7fa21806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.270+0000 7fa235d9b640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa21806c8f0 0x7fa21806edb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa2380cb7c0 tx=0x7fa220007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:53.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.274+0000 7fa22effd640 1 -- 172.21.15.44:0/2344792975 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa23007b740 con 0x7fa2380d2030 2024-04-02T13:21:53.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.456+0000 7fa23759e640 1 -- 172.21.15.44:0/2344792975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa23806cdd0 con 0x7fa2380d2030 2024-04-02T13:21:53.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.457+0000 7fa22effd640 1 -- 172.21.15.44:0/2344792975 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7fa23007fa40 con 0x7fa2380d2030 2024-04-02T13:21:53.460 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:53.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.459+0000 7fa23759e640 1 -- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa21806c8f0 msgr2=0x7fa21806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.459+0000 7fa23759e640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa21806c8f0 0x7fa21806edb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa2380cb7c0 tx=0x7fa220007e30 comp rx=0 tx=0).stop 2024-04-02T13:21:53.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.459+0000 7fa23759e640 1 -- 172.21.15.44:0/2344792975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2380d2030 msgr2=0x7fa2380ca510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.459+0000 7fa23759e640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2380d2030 0x7fa2380ca510 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa230000c00 tx=0x7fa2300046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.459+0000 7fa23759e640 1 -- 172.21.15.44:0/2344792975 shutdown_connections 2024-04-02T13:21:53.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.459+0000 7fa23759e640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa21806c8f0 0x7fa21806edb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.459+0000 7fa23759e640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa238160080 0x7fa238153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.459+0000 7fa23759e640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa23815e1d0 0x7fa23815fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.459+0000 7fa23759e640 1 --2- 172.21.15.44:0/2344792975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2380d2030 0x7fa2380ca510 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.459+0000 7fa23759e640 1 -- 172.21.15.44:0/2344792975 >> 172.21.15.44:0/2344792975 conn(0x7fa238073520 msgr2=0x7fa2380aeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:53.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.459+0000 7fa23759e640 1 -- 172.21.15.44:0/2344792975 shutdown_connections 2024-04-02T13:21:53.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.459+0000 7fa23759e640 1 -- 172.21.15.44:0/2344792975 wait complete. 2024-04-02T13:21:53.472 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:21:53.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.572+0000 7f00469e2640 1 Processor -- start 2024-04-02T13:21:53.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.572+0000 7f00469e2640 1 -- start start 2024-04-02T13:21:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f00469e2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f004015e0f0 0x7f00400d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f00469e2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f00400d0630 0x7f004006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f00469e2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00400aed70 con 0x7f004015e4d0 2024-04-02T13:21:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f00469e2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f004006dfe0 con 0x7f00400d0630 2024-04-02T13:21:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f00469e2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f004006e2c0 con 0x7f004015e0f0 2024-04-02T13:21:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f003f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f00400d0630 0x7f004006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f003f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f00400d0630 0x7f004006b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54072/0 (socket says 172.21.15.44:54072) 2024-04-02T13:21:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f003f7fe640 1 -- 172.21.15.44:0/3068301547 learned_addr learned my addr 172.21.15.44:0/3068301547 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f003f7fe640 1 -- 172.21.15.44:0/3068301547 >> v2:172.21.15.69:3300/0 conn(0x7f004015e0f0 msgr2=0x7f00400d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f003ffff640 1 --2- 172.21.15.44:0/3068301547 >> v2:172.21.15.69:3300/0 conn(0x7f004015e0f0 0x7f00400d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f003f7fe640 1 --2- 172.21.15.44:0/3068301547 >> v2:172.21.15.69:3300/0 conn(0x7f004015e0f0 0x7f00400d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f003f7fe640 1 -- 172.21.15.44:0/3068301547 >> v1:172.21.15.69:6789/0 conn(0x7f004015e4d0 legacy=0x7f0040161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:53.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f003f7fe640 1 -- 172.21.15.44:0/3068301547 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004006e5a0 con 0x7f00400d0630 2024-04-02T13:21:53.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.573+0000 7f003ffff640 1 --2- 172.21.15.44:0/3068301547 >> v2:172.21.15.69:3300/0 conn(0x7f004015e0f0 0x7f00400d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:53.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.574+0000 7f003f7fe640 1 --2- 172.21.15.44:0/3068301547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00400d0630 0x7f004006b4d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0040072520 tx=0x7f003002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a263e3d40e7e32a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:53.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.574+0000 7f003effd640 1 -- 172.21.15.44:0/3068301547 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0030034070 con 0x7f00400d0630 2024-04-02T13:21:53.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.574+0000 7f003effd640 1 -- 172.21.15.44:0/3068301547 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0030030860 con 0x7f00400d0630 2024-04-02T13:21:53.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.574+0000 7f003effd640 1 -- 172.21.15.44:0/3068301547 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0030030c30 con 0x7f00400d0630 2024-04-02T13:21:53.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.575+0000 7f00469e2640 1 -- 172.21.15.44:0/3068301547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00400d0630 msgr2=0x7f004006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.575+0000 7f00469e2640 1 --2- 172.21.15.44:0/3068301547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00400d0630 0x7f004006b4d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0040072520 tx=0x7f003002b9a0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.575+0000 7f00469e2640 1 -- 172.21.15.44:0/3068301547 shutdown_connections 2024-04-02T13:21:53.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.575+0000 7f00469e2640 1 --2- 172.21.15.44:0/3068301547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00400d0630 0x7f004006b4d0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.575+0000 7f00469e2640 1 --2- 172.21.15.44:0/3068301547 >> v2:172.21.15.69:3300/0 conn(0x7f004015e0f0 0x7f00400d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.575+0000 7f00469e2640 1 -- 172.21.15.44:0/3068301547 >> 172.21.15.44:0/3068301547 conn(0x7f00400b2980 msgr2=0x7f00400b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:53.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.575+0000 7f00469e2640 1 -- 172.21.15.44:0/3068301547 shutdown_connections 2024-04-02T13:21:53.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.575+0000 7f00469e2640 1 -- 172.21.15.44:0/3068301547 wait complete. 2024-04-02T13:21:53.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.576+0000 7f00469e2640 1 Processor -- start 2024-04-02T13:21:53.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.576+0000 7f00469e2640 1 -- start start 2024-04-02T13:21:53.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f00469e2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00400d0630 0x7f00400c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f00469e2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e0f0 0x7f004015b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f00469e2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004015e4d0 0x7f00400cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f00469e2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f004006eda0 con 0x7f004015e4d0 2024-04-02T13:21:53.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f00469e2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f00400d0af0 con 0x7f00400d0630 2024-04-02T13:21:53.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f00469e2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0040072f60 con 0x7f004015e0f0 2024-04-02T13:21:53.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f0044f7f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004015e4d0 0x7f00400cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f0044f7f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004015e4d0 0x7f00400cc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54080/0 (socket says 172.21.15.44:54080) 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f0044f7f640 1 -- 172.21.15.44:0/3748728895 learned_addr learned my addr 172.21.15.44:0/3748728895 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f003f7fe640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e0f0 0x7f004015b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f003ffff640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00400d0630 0x7f00400c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f0044f7f640 1 -- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e0f0 msgr2=0x7f004015b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f0044f7f640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e0f0 0x7f004015b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f0044f7f640 1 -- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00400d0630 msgr2=0x7f00400c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f0044f7f640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00400d0630 0x7f00400c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.577+0000 7f0044f7f640 1 -- 172.21.15.44:0/3748728895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004014a060 con 0x7f004015e4d0 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.578+0000 7f0044f7f640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004015e4d0 0x7f00400cc430 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f003800b9e0 tx=0x7f003800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.578+0000 7f003cff9640 1 -- 172.21.15.44:0/3748728895 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0038003b70 con 0x7f004015e4d0 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.578+0000 7f003cff9640 1 -- 172.21.15.44:0/3748728895 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0038003d10 con 0x7f004015e4d0 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.578+0000 7f00469e2640 1 -- 172.21.15.44:0/3748728895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f004015adb0 con 0x7f004015e4d0 2024-04-02T13:21:53.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.578+0000 7f003cff9640 1 -- 172.21.15.44:0/3748728895 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003800cb60 con 0x7f004015e4d0 2024-04-02T13:21:53.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.578+0000 7f00469e2640 1 -- 172.21.15.44:0/3748728895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f004015b370 con 0x7f004015e4d0 2024-04-02T13:21:53.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.579+0000 7f00469e2640 1 -- 172.21.15.44:0/3748728895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00400684f0 con 0x7f004015e4d0 2024-04-02T13:21:53.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.579+0000 7f003cff9640 1 -- 172.21.15.44:0/3748728895 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f003800cd00 con 0x7f004015e4d0 2024-04-02T13:21:53.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.580+0000 7f003cff9640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f002006c8f0 0x7f002006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.580+0000 7f003cff9640 1 -- 172.21.15.44:0/3748728895 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f003808ba60 con 0x7f004015e4d0 2024-04-02T13:21:53.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.580+0000 7f003ffff640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f002006c8f0 0x7f002006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.580+0000 7f003ffff640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f002006c8f0 0x7f002006edb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f002800ab60 tx=0x7f0028008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:53.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.584+0000 7f003cff9640 1 -- 172.21.15.44:0/3748728895 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f00380596b0 con 0x7f004015e4d0 2024-04-02T13:21:53.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.780+0000 7f00469e2640 1 -- 172.21.15.44:0/3748728895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f00400727f0 con 0x7f004015e4d0 2024-04-02T13:21:53.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.780+0000 7f003cff9640 1 -- 172.21.15.44:0/3748728895 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) ==== 80+0+0 (secure 0 0 0) 0x7f003805d9b0 con 0x7f004015e4d0 2024-04-02T13:21:53.783 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:21:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.782+0000 7f00469e2640 1 -- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f002006c8f0 msgr2=0x7f002006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.782+0000 7f00469e2640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f002006c8f0 0x7f002006edb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f002800ab60 tx=0x7f0028008040 comp rx=0 tx=0).stop 2024-04-02T13:21:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.782+0000 7f00469e2640 1 -- 172.21.15.44:0/3748728895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004015e4d0 msgr2=0x7f00400cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.782+0000 7f00469e2640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004015e4d0 0x7f00400cc430 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f003800b9e0 tx=0x7f003800bea0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.782+0000 7f00469e2640 1 -- 172.21.15.44:0/3748728895 shutdown_connections 2024-04-02T13:21:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.782+0000 7f00469e2640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f002006c8f0 0x7f002006edb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.782+0000 7f00469e2640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004015e4d0 0x7f00400cc430 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.782+0000 7f00469e2640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e0f0 0x7f004015b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.782+0000 7f00469e2640 1 --2- 172.21.15.44:0/3748728895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00400d0630 0x7f00400c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.782+0000 7f00469e2640 1 -- 172.21.15.44:0/3748728895 >> 172.21.15.44:0/3748728895 conn(0x7f00400b2980 msgr2=0x7f00400ca100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:53.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.782+0000 7f00469e2640 1 -- 172.21.15.44:0/3748728895 shutdown_connections 2024-04-02T13:21:53.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.782+0000 7f00469e2640 1 -- 172.21.15.44:0/3748728895 wait complete. 2024-04-02T13:21:53.795 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:21:53.796 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:21:53.810 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:21:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.949+0000 7f7cf4d1c640 1 Processor -- start 2024-04-02T13:21:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.949+0000 7f7cf4d1c640 1 -- start start 2024-04-02T13:21:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.950+0000 7f7cf4d1c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7cf00cece0 0x7f7cf00cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.950+0000 7f7cf4d1c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7cf00d09f0 0x7f7cf006d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.950+0000 7f7cf4d1c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cf00cf600 con 0x7f7cf006ece0 2024-04-02T13:21:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.950+0000 7f7cf4d1c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7cf006dc40 con 0x7f7cf00cece0 2024-04-02T13:21:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.950+0000 7f7cf4d1c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7cf006df20 con 0x7f7cf00d09f0 2024-04-02T13:21:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.950+0000 7f7cee59c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7cf00cece0 0x7f7cf00cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.950+0000 7f7cee59c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7cf00cece0 0x7f7cf00cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54096/0 (socket says 172.21.15.44:54096) 2024-04-02T13:21:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.950+0000 7f7cee59c640 1 -- 172.21.15.44:0/2495565602 learned_addr learned my addr 172.21.15.44:0/2495565602 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.950+0000 7f7cedd9b640 1 --2- 172.21.15.44:0/2495565602 >> v2:172.21.15.69:3300/0 conn(0x7f7cf00d09f0 0x7f7cf006d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.950+0000 7f7cee59c640 1 -- 172.21.15.44:0/2495565602 >> v2:172.21.15.69:3300/0 conn(0x7f7cf00d09f0 msgr2=0x7f7cf006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.951+0000 7f7cee59c640 1 --2- 172.21.15.44:0/2495565602 >> v2:172.21.15.69:3300/0 conn(0x7f7cf00d09f0 0x7f7cf006d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.951+0000 7f7cee59c640 1 -- 172.21.15.44:0/2495565602 >> v1:172.21.15.69:6789/0 conn(0x7f7cf006ece0 legacy=0x7f7cf015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.951+0000 7f7cee59c640 1 -- 172.21.15.44:0/2495565602 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf006e200 con 0x7f7cf00cece0 2024-04-02T13:21:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.951+0000 7f7cee59c640 1 --2- 172.21.15.44:0/2495565602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7cf00cece0 0x7f7cf00cf0c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00b44f0 tx=0x7f7ce802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c3fdbfc98417b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.951+0000 7f7ced59a640 1 -- 172.21.15.44:0/2495565602 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8034070 con 0x7f7cf00cece0 2024-04-02T13:21:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.951+0000 7f7ced59a640 1 -- 172.21.15.44:0/2495565602 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce8030860 con 0x7f7cf00cece0 2024-04-02T13:21:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.951+0000 7f7ced59a640 1 -- 172.21.15.44:0/2495565602 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8030c30 con 0x7f7cf00cece0 2024-04-02T13:21:53.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.952+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/2495565602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7cf00cece0 msgr2=0x7f7cf00cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.952+0000 7f7cf4d1c640 1 --2- 172.21.15.44:0/2495565602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7cf00cece0 0x7f7cf00cf0c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00b44f0 tx=0x7f7ce802b9a0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.952+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/2495565602 shutdown_connections 2024-04-02T13:21:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.952+0000 7f7cf4d1c640 1 --2- 172.21.15.44:0/2495565602 >> v2:172.21.15.69:3300/0 conn(0x7f7cf00d09f0 0x7f7cf006d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.952+0000 7f7cf4d1c640 1 --2- 172.21.15.44:0/2495565602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7cf00cece0 0x7f7cf00cf0c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.952+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/2495565602 >> 172.21.15.44:0/2495565602 conn(0x7f7cf00cfcd0 msgr2=0x7f7cf00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.952+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/2495565602 shutdown_connections 2024-04-02T13:21:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.952+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/2495565602 wait complete. 2024-04-02T13:21:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.953+0000 7f7cf4d1c640 1 Processor -- start 2024-04-02T13:21:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.953+0000 7f7cf4d1c640 1 -- start start 2024-04-02T13:21:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.954+0000 7f7cf4d1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7cf006ece0 0x7f7cf009c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.954+0000 7f7cf4d1c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cf00cece0 0x7f7cf01e7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.954+0000 7f7cf4d1c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7cf00d09f0 0x7f7cf01e8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.954+0000 7f7cf4d1c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf006d810 con 0x7f7cf00d09f0 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.954+0000 7f7cf4d1c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf006e7c0 con 0x7f7cf00cece0 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.954+0000 7f7cf4d1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf006d990 con 0x7f7cf006ece0 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.954+0000 7f7ceed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7cf00d09f0 0x7f7cf01e8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.954+0000 7f7ceed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7cf00d09f0 0x7f7cf01e8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54110/0 (socket says 172.21.15.44:54110) 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.954+0000 7f7ceed9d640 1 -- 172.21.15.44:0/3202405126 learned_addr learned my addr 172.21.15.44:0/3202405126 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.954+0000 7f7cee59c640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7cf006ece0 0x7f7cf009c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.954+0000 7f7cedd9b640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cf00cece0 0x7f7cf01e7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.955+0000 7f7ceed9d640 1 -- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7cf006ece0 msgr2=0x7f7cf009c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.955+0000 7f7ceed9d640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7cf006ece0 0x7f7cf009c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.955+0000 7f7ceed9d640 1 -- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cf00cece0 msgr2=0x7f7cf01e7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.955+0000 7f7ceed9d640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cf00cece0 0x7f7cf01e7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.955+0000 7f7ceed9d640 1 -- 172.21.15.44:0/3202405126 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf01f85a0 con 0x7f7cf00d09f0 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.955+0000 7f7ceed9d640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7cf00d09f0 0x7f7cf01e8150 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000c7a0 tx=0x7f7ce000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.955+0000 7f7cdeffd640 1 -- 172.21.15.44:0/3202405126 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0003b70 con 0x7f7cf00d09f0 2024-04-02T13:21:53.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.955+0000 7f7cdeffd640 1 -- 172.21.15.44:0/3202405126 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce0003d30 con 0x7f7cf00d09f0 2024-04-02T13:21:53.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.955+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/3202405126 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf01f77e0 con 0x7f7cf00d09f0 2024-04-02T13:21:53.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.955+0000 7f7cdeffd640 1 -- 172.21.15.44:0/3202405126 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce000aa90 con 0x7f7cf00d09f0 2024-04-02T13:21:53.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.955+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/3202405126 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf01f7da0 con 0x7f7cf00d09f0 2024-04-02T13:21:53.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.956+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/3202405126 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf00684f0 con 0x7f7cf00d09f0 2024-04-02T13:21:53.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.956+0000 7f7cdeffd640 1 -- 172.21.15.44:0/3202405126 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7ce000ac30 con 0x7f7cf00d09f0 2024-04-02T13:21:53.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.957+0000 7f7cdeffd640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7cd006c8f0 0x7f7cd006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:53.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.957+0000 7f7cdeffd640 1 -- 172.21.15.44:0/3202405126 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f7ce008bb90 con 0x7f7cf00d09f0 2024-04-02T13:21:53.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.957+0000 7f7cee59c640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7cd006c8f0 0x7f7cd006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:53.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.957+0000 7f7cee59c640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7cd006c8f0 0x7f7cd006edb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8036000 tx=0x7f7ce8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:53.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:53.961+0000 7f7cdeffd640 1 -- 172.21.15.44:0/3202405126 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7ce0059740 con 0x7f7cf00d09f0 2024-04-02T13:21:54.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.146+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/3202405126 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7cf00b2bf0 con 0x7f7cf00d09f0 2024-04-02T13:21:54.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.147+0000 7f7cdeffd640 1 -- 172.21.15.44:0/3202405126 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16894 (secure 0 0 0) 0x7f7ce005da40 con 0x7f7cf00d09f0 2024-04-02T13:21:54.150 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:54.150 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":39,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:21:44.874189+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:21:41.647497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-02T13:21:42.168299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:21:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.149+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7cd006c8f0 msgr2=0x7f7cd006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.149+0000 7f7cf4d1c640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7cd006c8f0 0x7f7cd006edb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8036000 tx=0x7f7ce8008030 comp rx=0 tx=0).stop 2024-04-02T13:21:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.149+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/3202405126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7cf00d09f0 msgr2=0x7f7cf01e8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.149+0000 7f7cf4d1c640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7cf00d09f0 0x7f7cf01e8150 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000c7a0 tx=0x7f7ce000cc60 comp rx=0 tx=0).stop 2024-04-02T13:21:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.149+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/3202405126 shutdown_connections 2024-04-02T13:21:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.149+0000 7f7cf4d1c640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7cd006c8f0 0x7f7cd006edb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.149+0000 7f7cf4d1c640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7cf00d09f0 0x7f7cf01e8150 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.149+0000 7f7cf4d1c640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cf00cece0 0x7f7cf01e7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.149+0000 7f7cf4d1c640 1 --2- 172.21.15.44:0/3202405126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7cf006ece0 0x7f7cf009c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.149+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/3202405126 >> 172.21.15.44:0/3202405126 conn(0x7f7cf00cfcd0 msgr2=0x7f7cf00b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.149+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/3202405126 shutdown_connections 2024-04-02T13:21:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.149+0000 7f7cf4d1c640 1 -- 172.21.15.44:0/3202405126 wait complete. 2024-04-02T13:21:54.163 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:21:41.647497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:21:42.168299+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:21:54.164 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:21:54.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.265+0000 7f4f0abe9640 1 Processor -- start 2024-04-02T13:21:54.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.265+0000 7f4f0abe9640 1 -- start start 2024-04-02T13:21:54.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.266+0000 7f4f0abe9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4f0415e1d0 0x7f4f040d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.266+0000 7f4f0abe9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f040d1640 con 0x7f4f040cec60 2024-04-02T13:21:54.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.266+0000 7f4f0abe9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f040cf1f0 con 0x7f4f040d2dd0 2024-04-02T13:21:54.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.266+0000 7f4f0abe9640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4f040cfe00 con 0x7f4f0415e1d0 2024-04-02T13:21:54.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.266+0000 7f4f08985640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4f0415e1d0 0x7f4f040d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.266+0000 7f4f03fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4f040cec60 0x7f4f040d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33044/0 (socket says 172.21.15.44:33044) 2024-04-02T13:21:54.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.266+0000 7f4f03fff640 1 -- 172.21.15.44:0/3031955889 learned_addr learned my addr 172.21.15.44:0/3031955889 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:54.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.267+0000 7f4f08985640 1 -- 172.21.15.44:0/3031955889 >> v1:172.21.15.69:6789/0 conn(0x7f4f040d2dd0 legacy=0x7f4f040b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.267+0000 7f4f08985640 1 -- 172.21.15.44:0/3031955889 >> v1:172.21.15.44:6789/0 conn(0x7f4f040cec60 legacy=0x7f4f040d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.267+0000 7f4f037fe640 1 -- 172.21.15.44:0/3031955889 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3477286211 0 0) 0x7f4f040d1640 con 0x7f4f040cec60 2024-04-02T13:21:54.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.267+0000 7f4f08985640 1 -- 172.21.15.44:0/3031955889 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f040cf4d0 con 0x7f4f0415e1d0 2024-04-02T13:21:54.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.267+0000 7f4f08985640 1 --2- 172.21.15.44:0/3031955889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f0415e1d0 0x7f4f040d1100 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f4efc009d90 tx=0x7f4efc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95a6cde79dcb2d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:54.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.267+0000 7f4f037fe640 1 -- 172.21.15.44:0/3031955889 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4efc039070 con 0x7f4f0415e1d0 2024-04-02T13:21:54.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.267+0000 7f4f037fe640 1 -- 172.21.15.44:0/3031955889 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4efc02cc60 con 0x7f4f0415e1d0 2024-04-02T13:21:54.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.267+0000 7f4f037fe640 1 -- 172.21.15.44:0/3031955889 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4efc034660 con 0x7f4f0415e1d0 2024-04-02T13:21:54.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.268+0000 7f4f0abe9640 1 -- 172.21.15.44:0/3031955889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f0415e1d0 msgr2=0x7f4f040d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.268+0000 7f4f0abe9640 1 --2- 172.21.15.44:0/3031955889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f0415e1d0 0x7f4f040d1100 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f4efc009d90 tx=0x7f4efc02beb0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.268+0000 7f4f0abe9640 1 -- 172.21.15.44:0/3031955889 shutdown_connections 2024-04-02T13:21:54.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.268+0000 7f4f0abe9640 1 --2- 172.21.15.44:0/3031955889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f0415e1d0 0x7f4f040d1100 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.268+0000 7f4f0abe9640 1 -- 172.21.15.44:0/3031955889 >> 172.21.15.44:0/3031955889 conn(0x7f4f0409d3b0 msgr2=0x7f4f040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:54.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.268+0000 7f4f0abe9640 1 -- 172.21.15.44:0/3031955889 shutdown_connections 2024-04-02T13:21:54.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.268+0000 7f4f0abe9640 1 -- 172.21.15.44:0/3031955889 wait complete. 2024-04-02T13:21:54.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.269+0000 7f4f0abe9640 1 Processor -- start 2024-04-02T13:21:54.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.269+0000 7f4f0abe9640 1 -- start start 2024-04-02T13:21:54.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.270+0000 7f4f0abe9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f040cec60 0x7f4f0409ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.270+0000 7f4f0abe9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f040d2dd0 0x7f4f0415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.270+0000 7f4f0abe9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f0415e1d0 0x7f4f040b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.270+0000 7f4f0abe9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4f040b7fd0 con 0x7f4f040d2dd0 2024-04-02T13:21:54.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.270+0000 7f4f0abe9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4f040cfa90 con 0x7f4f040cec60 2024-04-02T13:21:54.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.270+0000 7f4f0abe9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4f040b8150 con 0x7f4f0415e1d0 2024-04-02T13:21:54.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.270+0000 7f4f03fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f040d2dd0 0x7f4f0415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.270+0000 7f4f03fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f040d2dd0 0x7f4f0415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54130/0 (socket says 172.21.15.44:54130) 2024-04-02T13:21:54.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.270+0000 7f4f03fff640 1 -- 172.21.15.44:0/2313516037 learned_addr learned my addr 172.21.15.44:0/2313516037 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:54.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.270+0000 7f4f09186640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f0415e1d0 0x7f4f040b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.270+0000 7f4f08985640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f040cec60 0x7f4f0409ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.271+0000 7f4f03fff640 1 -- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f0415e1d0 msgr2=0x7f4f040b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.271+0000 7f4f03fff640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f0415e1d0 0x7f4f040b0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.271+0000 7f4f03fff640 1 -- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f040cec60 msgr2=0x7f4f0409ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.271+0000 7f4f03fff640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f040cec60 0x7f4f0409ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.271+0000 7f4f03fff640 1 -- 172.21.15.44:0/2313516037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f040b1300 con 0x7f4f040d2dd0 2024-04-02T13:21:54.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.271+0000 7f4f03fff640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f040d2dd0 0x7f4f0415cf90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4eec00c3f0 tx=0x7f4eec00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:54.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.271+0000 7f4f017fa640 1 -- 172.21.15.44:0/2313516037 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec0074e0 con 0x7f4f040d2dd0 2024-04-02T13:21:54.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.271+0000 7f4f017fa640 1 -- 172.21.15.44:0/2313516037 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eec0047e0 con 0x7f4f040d2dd0 2024-04-02T13:21:54.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.271+0000 7f4f0abe9640 1 -- 172.21.15.44:0/2313516037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f040a1060 con 0x7f4f040d2dd0 2024-04-02T13:21:54.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.271+0000 7f4f017fa640 1 -- 172.21.15.44:0/2313516037 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec004c10 con 0x7f4f040d2dd0 2024-04-02T13:21:54.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.271+0000 7f4f0abe9640 1 -- 172.21.15.44:0/2313516037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f040a1620 con 0x7f4f040d2dd0 2024-04-02T13:21:54.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.272+0000 7f4f0abe9640 1 -- 172.21.15.44:0/2313516037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f040684f0 con 0x7f4f040d2dd0 2024-04-02T13:21:54.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.272+0000 7f4f017fa640 1 -- 172.21.15.44:0/2313516037 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4eec011070 con 0x7f4f040d2dd0 2024-04-02T13:21:54.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.273+0000 7f4f017fa640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ee406c920 0x7f4ee406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.273+0000 7f4f017fa640 1 -- 172.21.15.44:0/2313516037 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f4eec08b7b0 con 0x7f4f040d2dd0 2024-04-02T13:21:54.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.273+0000 7f4f08985640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ee406c920 0x7f4ee406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.273+0000 7f4f08985640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ee406c920 0x7f4ee406ede0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4efc036000 tx=0x7f4efc030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:54.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.277+0000 7f4f017fa640 1 -- 172.21.15.44:0/2313516037 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4f040b1300 con 0x7f4f040d2dd0 2024-04-02T13:21:54.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.460+0000 7f4f0abe9640 1 -- 172.21.15.44:0/2313516037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4f040d47f0 con 0x7f4f040d2dd0 2024-04-02T13:21:54.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.460+0000 7f4f017fa640 1 -- 172.21.15.44:0/2313516037 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7f4eec059380 con 0x7f4f040d2dd0 2024-04-02T13:21:54.463 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:21:54.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.462+0000 7f4f0abe9640 1 -- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ee406c920 msgr2=0x7f4ee406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.462+0000 7f4f0abe9640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ee406c920 0x7f4ee406ede0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4efc036000 tx=0x7f4efc030000 comp rx=0 tx=0).stop 2024-04-02T13:21:54.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.462+0000 7f4f0abe9640 1 -- 172.21.15.44:0/2313516037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f040d2dd0 msgr2=0x7f4f0415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.462+0000 7f4f0abe9640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f040d2dd0 0x7f4f0415cf90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4eec00c3f0 tx=0x7f4eec00c8b0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.462+0000 7f4f0abe9640 1 -- 172.21.15.44:0/2313516037 shutdown_connections 2024-04-02T13:21:54.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.462+0000 7f4f0abe9640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ee406c920 0x7f4ee406ede0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.462+0000 7f4f0abe9640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f0415e1d0 0x7f4f040b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.462+0000 7f4f0abe9640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f040d2dd0 0x7f4f0415cf90 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.462+0000 7f4f0abe9640 1 --2- 172.21.15.44:0/2313516037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f040cec60 0x7f4f0409ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.462+0000 7f4f0abe9640 1 -- 172.21.15.44:0/2313516037 >> 172.21.15.44:0/2313516037 conn(0x7f4f0409d3b0 msgr2=0x7f4f0406c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:54.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.462+0000 7f4f0abe9640 1 -- 172.21.15.44:0/2313516037 shutdown_connections 2024-04-02T13:21:54.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.462+0000 7f4f0abe9640 1 -- 172.21.15.44:0/2313516037 wait complete. 2024-04-02T13:21:54.476 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:21:54.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.577+0000 7f928bbdb640 1 Processor -- start 2024-04-02T13:21:54.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.577+0000 7f928bbdb640 1 -- start start 2024-04-02T13:21:54.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.578+0000 7f928bbdb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f92840cec60 0x7f92840d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.578+0000 7f928bbdb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f928415e5b0 0x7f92840d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.578+0000 7f928bbdb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92840cf1f0 con 0x7f928415e1d0 2024-04-02T13:21:54.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.578+0000 7f928bbdb640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f92840cfe00 con 0x7f92840cec60 2024-04-02T13:21:54.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.578+0000 7f928bbdb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f92840cf4d0 con 0x7f928415e5b0 2024-04-02T13:21:54.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.578+0000 7f9289176640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f92840cec60 0x7f92840d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.578+0000 7f9289977640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f928415e1d0 0x7f92840d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51120/0 (socket says 172.21.15.44:51120) 2024-04-02T13:21:54.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.578+0000 7f9289176640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f92840cec60 0x7f92840d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46410/0 (socket says 172.21.15.44:46410) 2024-04-02T13:21:54.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.578+0000 7f9289977640 1 -- 172.21.15.44:0/2747105812 learned_addr learned my addr 172.21.15.44:0/2747105812 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:54.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.579+0000 7f928a178640 1 --2- 172.21.15.44:0/2747105812 >> v2:172.21.15.69:3300/0 conn(0x7f928415e5b0 0x7f92840d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.579+0000 7f9289176640 1 -- 172.21.15.44:0/2747105812 >> v2:172.21.15.69:3300/0 conn(0x7f928415e5b0 msgr2=0x7f92840d2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:54.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.579+0000 7f9289176640 1 --2- 172.21.15.44:0/2747105812 >> v2:172.21.15.69:3300/0 conn(0x7f928415e5b0 0x7f92840d2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.579+0000 7f9289176640 1 -- 172.21.15.44:0/2747105812 >> v1:172.21.15.44:6789/0 conn(0x7f928415e1d0 legacy=0x7f92840d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.579+0000 7f9289176640 1 -- 172.21.15.44:0/2747105812 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92840cf7b0 con 0x7f92840cec60 2024-04-02T13:21:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.579+0000 7f9289176640 1 --2- 172.21.15.44:0/2747105812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92840cec60 0x7f92840d26c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f92840d0350 tx=0x7f927802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b836a0f9343eb09 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.579+0000 7f9288975640 1 -- 172.21.15.44:0/2747105812 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92780315a0 con 0x7f92840cec60 2024-04-02T13:21:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.579+0000 7f9288975640 1 -- 172.21.15.44:0/2747105812 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9278037070 con 0x7f92840cec60 2024-04-02T13:21:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.579+0000 7f9288975640 1 -- 172.21.15.44:0/2747105812 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9278031960 con 0x7f92840cec60 2024-04-02T13:21:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.580+0000 7f928bbdb640 1 -- 172.21.15.44:0/2747105812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92840cec60 msgr2=0x7f92840d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.580+0000 7f928bbdb640 1 --2- 172.21.15.44:0/2747105812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92840cec60 0x7f92840d26c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f92840d0350 tx=0x7f927802b840 comp rx=0 tx=0).stop 2024-04-02T13:21:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.580+0000 7f928bbdb640 1 -- 172.21.15.44:0/2747105812 shutdown_connections 2024-04-02T13:21:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.580+0000 7f928bbdb640 1 --2- 172.21.15.44:0/2747105812 >> v2:172.21.15.69:3300/0 conn(0x7f928415e5b0 0x7f92840d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.580+0000 7f928bbdb640 1 --2- 172.21.15.44:0/2747105812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92840cec60 0x7f92840d26c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.580+0000 7f928bbdb640 1 -- 172.21.15.44:0/2747105812 >> 172.21.15.44:0/2747105812 conn(0x7f928409d3b0 msgr2=0x7f92840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.580+0000 7f928bbdb640 1 -- 172.21.15.44:0/2747105812 shutdown_connections 2024-04-02T13:21:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.580+0000 7f928bbdb640 1 -- 172.21.15.44:0/2747105812 wait complete. 2024-04-02T13:21:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.581+0000 7f928bbdb640 1 Processor -- start 2024-04-02T13:21:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.581+0000 7f928bbdb640 1 -- start start 2024-04-02T13:21:54.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928bbdb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92840cec60 0x7f9284160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928bbdb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f928415e1d0 0x7f9284154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928bbdb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928415e5b0 0x7f9284154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928bbdb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f92840b4e60 con 0x7f928415e5b0 2024-04-02T13:21:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928bbdb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f92840b4ce0 con 0x7f928415e1d0 2024-04-02T13:21:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928bbdb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f92840b4fe0 con 0x7f92840cec60 2024-04-02T13:21:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928a178640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928415e5b0 0x7f9284154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928a178640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928415e5b0 0x7f9284154c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46420/0 (socket says 172.21.15.44:46420) 2024-04-02T13:21:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928a178640 1 -- 172.21.15.44:0/2437964744 learned_addr learned my addr 172.21.15.44:0/2437964744 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f9289176640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f928415e1d0 0x7f9284154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f9289977640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92840cec60 0x7f9284160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928a178640 1 -- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92840cec60 msgr2=0x7f9284160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928a178640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92840cec60 0x7f9284160dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928a178640 1 -- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f928415e1d0 msgr2=0x7f9284154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928a178640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f928415e1d0 0x7f9284154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f928a178640 1 -- 172.21.15.44:0/2437964744 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928415fa90 con 0x7f928415e5b0 2024-04-02T13:21:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.582+0000 7f9289977640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92840cec60 0x7f9284160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.583+0000 7f928a178640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928415e5b0 0x7f9284154c70 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f927400cc30 tx=0x7f927400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.583+0000 7f92827fc640 1 -- 172.21.15.44:0/2437964744 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927400ce40 con 0x7f928415e5b0 2024-04-02T13:21:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.583+0000 7f92827fc640 1 -- 172.21.15.44:0/2437964744 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9274003990 con 0x7f928415e5b0 2024-04-02T13:21:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.583+0000 7f928bbdb640 1 -- 172.21.15.44:0/2437964744 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f928415fd30 con 0x7f928415e5b0 2024-04-02T13:21:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.583+0000 7f92827fc640 1 -- 172.21.15.44:0/2437964744 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9274003d40 con 0x7f928415e5b0 2024-04-02T13:21:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.583+0000 7f928bbdb640 1 -- 172.21.15.44:0/2437964744 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92841602f0 con 0x7f928415e5b0 2024-04-02T13:21:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.584+0000 7f928bbdb640 1 -- 172.21.15.44:0/2437964744 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92840684f0 con 0x7f928415e5b0 2024-04-02T13:21:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.584+0000 7f92827fc640 1 -- 172.21.15.44:0/2437964744 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9274003fa0 con 0x7f928415e5b0 2024-04-02T13:21:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.584+0000 7f92827fc640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f927006c8f0 0x7f927006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.585+0000 7f92827fc640 1 -- 172.21.15.44:0/2437964744 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f927408c150 con 0x7f928415e5b0 2024-04-02T13:21:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.585+0000 7f9289977640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f927006c8f0 0x7f927006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.585+0000 7f9289977640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f927006c8f0 0x7f927006edb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f927c000f60 tx=0x7f927c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:54.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.589+0000 7f92827fc640 1 -- 172.21.15.44:0/2437964744 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9274014030 con 0x7f928415e5b0 2024-04-02T13:21:54.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.772+0000 7f928bbdb640 1 -- 172.21.15.44:0/2437964744 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f928415d240 con 0x7f928415e5b0 2024-04-02T13:21:54.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.773+0000 7f92827fc640 1 -- 172.21.15.44:0/2437964744 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7f927405a3f0 con 0x7f928415e5b0 2024-04-02T13:21:54.775 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:21:54.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.775+0000 7f928bbdb640 1 -- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f927006c8f0 msgr2=0x7f927006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.775+0000 7f928bbdb640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f927006c8f0 0x7f927006edb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f927c000f60 tx=0x7f927c009040 comp rx=0 tx=0).stop 2024-04-02T13:21:54.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.775+0000 7f928bbdb640 1 -- 172.21.15.44:0/2437964744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928415e5b0 msgr2=0x7f9284154c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.775+0000 7f928bbdb640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928415e5b0 0x7f9284154c70 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f927400cc30 tx=0x7f927400a5c0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.775+0000 7f928bbdb640 1 -- 172.21.15.44:0/2437964744 shutdown_connections 2024-04-02T13:21:54.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.775+0000 7f928bbdb640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f927006c8f0 0x7f927006edb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.775+0000 7f928bbdb640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928415e5b0 0x7f9284154c70 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.775+0000 7f928bbdb640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f928415e1d0 0x7f9284154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.775+0000 7f928bbdb640 1 --2- 172.21.15.44:0/2437964744 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92840cec60 0x7f9284160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.775+0000 7f928bbdb640 1 -- 172.21.15.44:0/2437964744 >> 172.21.15.44:0/2437964744 conn(0x7f928409d3b0 msgr2=0x7f92840ca590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:54.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.775+0000 7f928bbdb640 1 -- 172.21.15.44:0/2437964744 shutdown_connections 2024-04-02T13:21:54.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.775+0000 7f928bbdb640 1 -- 172.21.15.44:0/2437964744 wait complete. 2024-04-02T13:21:54.788 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:21:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.890+0000 7fe66e1a3640 1 Processor -- start 2024-04-02T13:21:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.890+0000 7fe66e1a3640 1 -- start start 2024-04-02T13:21:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.891+0000 7fe66e1a3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe6680cfe10 0x7fe668169240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.891+0000 7fe66e1a3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe66806c220 con 0x7fe6680721f0 2024-04-02T13:21:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.891+0000 7fe66e1a3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe66815f7e0 con 0x7fe66815e420 2024-04-02T13:21:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.891+0000 7fe66e1a3640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe6680cf3b0 con 0x7fe6680cfe10 2024-04-02T13:21:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.891+0000 7fe667fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe6680cfe10 0x7fe668169240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.891+0000 7fe666ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe6680721f0 0x7fe6680cf700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51130/0 (socket says 172.21.15.44:51130) 2024-04-02T13:21:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.891+0000 7fe666ffd640 1 -- 172.21.15.44:0/1690162281 learned_addr learned my addr 172.21.15.44:0/1690162281 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.891+0000 7fe667fff640 1 -- 172.21.15.44:0/1690162281 >> v1:172.21.15.69:6789/0 conn(0x7fe66815e420 legacy=0x7fe6680ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.891+0000 7fe667fff640 1 -- 172.21.15.44:0/1690162281 >> v1:172.21.15.44:6789/0 conn(0x7fe6680721f0 legacy=0x7fe6680cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.891+0000 7fe667fff640 1 -- 172.21.15.44:0/1690162281 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe66815ec60 con 0x7fe6680cfe10 2024-04-02T13:21:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.891+0000 7fe6667fc640 1 -- 172.21.15.44:0/1690162281 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2840866755 0 0) 0x7fe65c001110 con 0x7fe6680721f0 2024-04-02T13:21:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.892+0000 7fe667fff640 1 --2- 172.21.15.44:0/1690162281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe6680cfe10 0x7fe668169240 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe6600097d0 tx=0x7fe66002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52ce65ab8dd9c47 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.892+0000 7fe6667fc640 1 -- 172.21.15.44:0/1690162281 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6600315a0 con 0x7fe6680cfe10 2024-04-02T13:21:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.892+0000 7fe6667fc640 1 -- 172.21.15.44:0/1690162281 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe660037070 con 0x7fe6680cfe10 2024-04-02T13:21:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.892+0000 7fe6667fc640 1 -- 172.21.15.44:0/1690162281 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe660031960 con 0x7fe6680cfe10 2024-04-02T13:21:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.893+0000 7fe66e1a3640 1 -- 172.21.15.44:0/1690162281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe6680cfe10 msgr2=0x7fe668169240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.893+0000 7fe66e1a3640 1 --2- 172.21.15.44:0/1690162281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe6680cfe10 0x7fe668169240 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe6600097d0 tx=0x7fe66002b840 comp rx=0 tx=0).stop 2024-04-02T13:21:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.893+0000 7fe66e1a3640 1 -- 172.21.15.44:0/1690162281 shutdown_connections 2024-04-02T13:21:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.893+0000 7fe66e1a3640 1 --2- 172.21.15.44:0/1690162281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe6680cfe10 0x7fe668169240 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.893+0000 7fe66e1a3640 1 -- 172.21.15.44:0/1690162281 >> 172.21.15.44:0/1690162281 conn(0x7fe6680b2a10 msgr2=0x7fe6680b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.893+0000 7fe66e1a3640 1 -- 172.21.15.44:0/1690162281 shutdown_connections 2024-04-02T13:21:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.893+0000 7fe66e1a3640 1 -- 172.21.15.44:0/1690162281 wait complete. 2024-04-02T13:21:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.894+0000 7fe66e1a3640 1 Processor -- start 2024-04-02T13:21:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.894+0000 7fe66e1a3640 1 -- start start 2024-04-02T13:21:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe66e1a3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6680721f0 0x7fe668151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe66e1a3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680cfe10 0x7fe668150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe66e1a3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66815e420 0x7fe668072650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe66e1a3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe66815f500 con 0x7fe66815e420 2024-04-02T13:21:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe66e1a3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe66815f2a0 con 0x7fe6680721f0 2024-04-02T13:21:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe66e1a3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe66815e800 con 0x7fe6680cfe10 2024-04-02T13:21:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe667fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66815e420 0x7fe668072650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe667fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66815e420 0x7fe668072650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46444/0 (socket says 172.21.15.44:46444) 2024-04-02T13:21:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe667fff640 1 -- 172.21.15.44:0/786367405 learned_addr learned my addr 172.21.15.44:0/786367405 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe666ffd640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680cfe10 0x7fe668150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe6677fe640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6680721f0 0x7fe668151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe667fff640 1 -- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680cfe10 msgr2=0x7fe668150920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe667fff640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680cfe10 0x7fe668150920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe667fff640 1 -- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6680721f0 msgr2=0x7fe668151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe667fff640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6680721f0 0x7fe668151d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe667fff640 1 -- 172.21.15.44:0/786367405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe668152280 con 0x7fe66815e420 2024-04-02T13:21:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe6677fe640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6680721f0 0x7fe668151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.895+0000 7fe667fff640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66815e420 0x7fe668072650 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe660009720 tx=0x7fe6600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.896+0000 7fe64ffff640 1 -- 172.21.15.44:0/786367405 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe660032840 con 0x7fe66815e420 2024-04-02T13:21:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.896+0000 7fe64ffff640 1 -- 172.21.15.44:0/786367405 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6600329e0 con 0x7fe66815e420 2024-04-02T13:21:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.896+0000 7fe66e1a3640 1 -- 172.21.15.44:0/786367405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe668150e60 con 0x7fe66815e420 2024-04-02T13:21:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.896+0000 7fe64ffff640 1 -- 172.21.15.44:0/786367405 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe660032dd0 con 0x7fe66815e420 2024-04-02T13:21:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.896+0000 7fe66e1a3640 1 -- 172.21.15.44:0/786367405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6681f4c40 con 0x7fe66815e420 2024-04-02T13:21:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.897+0000 7fe66e1a3640 1 -- 172.21.15.44:0/786367405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6680684f0 con 0x7fe66815e420 2024-04-02T13:21:54.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.897+0000 7fe64ffff640 1 -- 172.21.15.44:0/786367405 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe66004d3f0 con 0x7fe66815e420 2024-04-02T13:21:54.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.897+0000 7fe64ffff640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe65406c8f0 0x7fe65406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:54.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.897+0000 7fe64ffff640 1 -- 172.21.15.44:0/786367405 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fe66002f030 con 0x7fe66815e420 2024-04-02T13:21:54.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.898+0000 7fe6677fe640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe65406c8f0 0x7fe65406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:54.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.898+0000 7fe6677fe640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe65406c8f0 0x7fe65406edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe6580012e0 tx=0x7fe658009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:54.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:54.902+0000 7fe64ffff640 1 -- 172.21.15.44:0/786367405 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe668150e60 con 0x7fe66815e420 2024-04-02T13:21:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.086+0000 7fe66e1a3640 1 -- 172.21.15.44:0/786367405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe6680b3770 con 0x7fe66815e420 2024-04-02T13:21:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.086+0000 7fe64ffff640 1 -- 172.21.15.44:0/786367405 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7fe668152280 con 0x7fe66815e420 2024-04-02T13:21:55.089 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:21:55.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.088+0000 7fe66e1a3640 1 -- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe65406c8f0 msgr2=0x7fe65406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.088+0000 7fe66e1a3640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe65406c8f0 0x7fe65406edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe6580012e0 tx=0x7fe658009040 comp rx=0 tx=0).stop 2024-04-02T13:21:55.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.088+0000 7fe66e1a3640 1 -- 172.21.15.44:0/786367405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66815e420 msgr2=0x7fe668072650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.088+0000 7fe66e1a3640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66815e420 0x7fe668072650 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe660009720 tx=0x7fe6600046c0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.088+0000 7fe66e1a3640 1 -- 172.21.15.44:0/786367405 shutdown_connections 2024-04-02T13:21:55.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.088+0000 7fe66e1a3640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe65406c8f0 0x7fe65406edb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.088+0000 7fe66e1a3640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66815e420 0x7fe668072650 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.088+0000 7fe66e1a3640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680cfe10 0x7fe668150920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.088+0000 7fe66e1a3640 1 --2- 172.21.15.44:0/786367405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6680721f0 0x7fe668151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.088+0000 7fe66e1a3640 1 -- 172.21.15.44:0/786367405 >> 172.21.15.44:0/786367405 conn(0x7fe6680b2a10 msgr2=0x7fe6680b0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.088+0000 7fe66e1a3640 1 -- 172.21.15.44:0/786367405 shutdown_connections 2024-04-02T13:21:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.088+0000 7fe66e1a3640 1 -- 172.21.15.44:0/786367405 wait complete. 2024-04-02T13:21:55.102 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:21:55.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.201+0000 7fe027029640 1 Processor -- start 2024-04-02T13:21:55.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.201+0000 7fe027029640 1 -- start start 2024-04-02T13:21:55.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.202+0000 7fe027029640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe0200d09f0 0x7fe02006d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.202+0000 7fe027029640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe02006dc40 con 0x7fe0200cece0 2024-04-02T13:21:55.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.202+0000 7fe027029640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe02006df20 con 0x7fe02006ece0 2024-04-02T13:21:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.202+0000 7fe027029640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe02006e200 con 0x7fe0200d09f0 2024-04-02T13:21:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.202+0000 7fe024dc5640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe0200cece0 0x7fe0200cf0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51138/0 (socket says 172.21.15.44:51138) 2024-04-02T13:21:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.202+0000 7fe024dc5640 1 -- 172.21.15.44:0/880344363 learned_addr learned my addr 172.21.15.44:0/880344363 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.202+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1697573052 0 0) 0x7fe02006dc40 con 0x7fe0200cece0 2024-04-02T13:21:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.202+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe00c005730 con 0x7fe0200cece0 2024-04-02T13:21:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.202+0000 7fe01ffff640 1 --2- 172.21.15.44:0/880344363 >> v2:172.21.15.69:3300/0 conn(0x7fe0200d09f0 0x7fe02006d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3963773557 0 0) 0x7fe00c005730 con 0x7fe0200cece0 2024-04-02T13:21:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 >> v2:172.21.15.69:3300/0 conn(0x7fe0200d09f0 msgr2=0x7fe02006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01f7fe640 1 --2- 172.21.15.44:0/880344363 >> v2:172.21.15.69:3300/0 conn(0x7fe0200d09f0 0x7fe02006d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 >> v1:172.21.15.69:6789/0 conn(0x7fe02006ece0 legacy=0x7fe0200acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe02006e4e0 con 0x7fe0200cece0 2024-04-02T13:21:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fe018001670 con 0x7fe0200cece0 2024-04-02T13:21:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 >> v1:172.21.15.44:6789/0 conn(0x7fe0200cece0 legacy=0x7fe0200cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01f7fe640 1 --2- 172.21.15.44:0/880344363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe00c005bc0 0x7fe00c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01f7fe640 1 --2- 172.21.15.44:0/880344363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe00c006a10 0x7fe00c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01f7fe640 1 --2- 172.21.15.44:0/880344363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe00c0073d0 0x7fe00c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe02006e200 con 0x7fe00c005bc0 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe00c00e270 con 0x7fe00c0073d0 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe00c00e470 con 0x7fe00c006a10 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe024dc5640 1 --2- 172.21.15.44:0/880344363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe00c005bc0 0x7fe00c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe0255c6640 1 --2- 172.21.15.44:0/880344363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe00c0073d0 0x7fe00c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe01ffff640 1 --2- 172.21.15.44:0/880344363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe00c006a10 0x7fe00c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe024dc5640 1 -- 172.21.15.44:0/880344363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe00c006a10 msgr2=0x7fe00c006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe024dc5640 1 --2- 172.21.15.44:0/880344363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe00c006a10 0x7fe00c006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe024dc5640 1 -- 172.21.15.44:0/880344363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe00c0073d0 msgr2=0x7fe00c00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe024dc5640 1 --2- 172.21.15.44:0/880344363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe00c0073d0 0x7fe00c00dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe0255c6640 1 -- 172.21.15.44:0/880344363 reap_dead start 2024-04-02T13:21:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.203+0000 7fe024dc5640 1 -- 172.21.15.44:0/880344363 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe00c005730 con 0x7fe00c005bc0 2024-04-02T13:21:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.204+0000 7fe024dc5640 1 --2- 172.21.15.44:0/880344363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe00c005bc0 0x7fe00c005fe0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe01800c240 tx=0x7fe00c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd0f4235f558185 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.204+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe018007cf0 con 0x7fe00c005bc0 2024-04-02T13:21:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.204+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe018030070 con 0x7fe00c005bc0 2024-04-02T13:21:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.204+0000 7fe01f7fe640 1 -- 172.21.15.44:0/880344363 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe018005d70 con 0x7fe00c005bc0 2024-04-02T13:21:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.205+0000 7fe027029640 1 -- 172.21.15.44:0/880344363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe00c005bc0 msgr2=0x7fe00c005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.205+0000 7fe027029640 1 --2- 172.21.15.44:0/880344363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe00c005bc0 0x7fe00c005fe0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe01800c240 tx=0x7fe00c007a40 comp rx=0 tx=0).stop 2024-04-02T13:21:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.205+0000 7fe027029640 1 -- 172.21.15.44:0/880344363 shutdown_connections 2024-04-02T13:21:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.205+0000 7fe027029640 1 --2- 172.21.15.44:0/880344363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe00c005bc0 0x7fe00c005fe0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.205+0000 7fe027029640 1 -- 172.21.15.44:0/880344363 >> 172.21.15.44:0/880344363 conn(0x7fe0200cfcd0 msgr2=0x7fe0200d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.205+0000 7fe027029640 1 -- 172.21.15.44:0/880344363 shutdown_connections 2024-04-02T13:21:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.205+0000 7fe027029640 1 -- 172.21.15.44:0/880344363 wait complete. 2024-04-02T13:21:55.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.206+0000 7fe027029640 1 Processor -- start 2024-04-02T13:21:55.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.206+0000 7fe027029640 1 -- start start 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe027029640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe00c005bc0 0x7fe0201f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe027029640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe020153f00 0x7fe020158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe027029640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe020155700 0x7fe020155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe027029640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe02015b3a0 con 0x7fe00c005bc0 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe027029640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe02015b220 con 0x7fe020155700 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe027029640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe0200cf0c0 con 0x7fe020153f00 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe024dc5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe00c005bc0 0x7fe0201f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe024dc5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe00c005bc0 0x7fe0201f41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46462/0 (socket says 172.21.15.44:46462) 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe024dc5640 1 -- 172.21.15.44:0/3154056002 learned_addr learned my addr 172.21.15.44:0/3154056002 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe0255c6640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe020155700 0x7fe020155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe024dc5640 1 -- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe020153f00 msgr2=0x7fe020158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe01ffff640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe020153f00 0x7fe020158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe024dc5640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe020153f00 0x7fe020158740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe024dc5640 1 -- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe020155700 msgr2=0x7fe020155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe024dc5640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe020155700 0x7fe020155b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe024dc5640 1 -- 172.21.15.44:0/3154056002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0201544c0 con 0x7fe00c005bc0 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe0255c6640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe020155700 0x7fe020155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe024dc5640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe00c005bc0 0x7fe0201f41e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe01800c240 tx=0x7fe02006e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe01d7fa640 1 -- 172.21.15.44:0/3154056002 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe018045070 con 0x7fe00c005bc0 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe01d7fa640 1 -- 172.21.15.44:0/3154056002 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe018004bf0 con 0x7fe00c005bc0 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe01d7fa640 1 -- 172.21.15.44:0/3154056002 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe018001c30 con 0x7fe00c005bc0 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.207+0000 7fe027029640 1 -- 172.21.15.44:0/3154056002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0201547a0 con 0x7fe00c005bc0 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.208+0000 7fe027029640 1 -- 172.21.15.44:0/3154056002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe020154d60 con 0x7fe00c005bc0 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.208+0000 7fe027029640 1 -- 172.21.15.44:0/3154056002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe020074f20 con 0x7fe00c005bc0 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.209+0000 7fe01d7fa640 1 -- 172.21.15.44:0/3154056002 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe018007070 con 0x7fe00c005bc0 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.209+0000 7fe01d7fa640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe00006c920 0x7fe00006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.209+0000 7fe01d7fa640 1 -- 172.21.15.44:0/3154056002 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fe0180acc30 con 0x7fe00c005bc0 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.210+0000 7fe01ffff640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe00006c920 0x7fe00006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.210+0000 7fe01ffff640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe00006c920 0x7fe00006ede0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe02015e760 tx=0x7fe01000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:55.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.214+0000 7fe01d7fa640 1 -- 172.21.15.44:0/3154056002 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe01807ae30 con 0x7fe00c005bc0 2024-04-02T13:21:55.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.413+0000 7fe027029640 1 -- 172.21.15.44:0/3154056002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe0200684f0 con 0x7fe00c005bc0 2024-04-02T13:21:55.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.413+0000 7fe01d7fa640 1 -- 172.21.15.44:0/3154056002 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) ==== 80+0+0 (secure 0 0 0) 0x7fe01807a730 con 0x7fe00c005bc0 2024-04-02T13:21:55.416 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:21:55.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.415+0000 7fe027029640 1 -- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe00006c920 msgr2=0x7fe00006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.415+0000 7fe027029640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe00006c920 0x7fe00006ede0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe02015e760 tx=0x7fe01000b040 comp rx=0 tx=0).stop 2024-04-02T13:21:55.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.415+0000 7fe027029640 1 -- 172.21.15.44:0/3154056002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe00c005bc0 msgr2=0x7fe0201f41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.415+0000 7fe027029640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe00c005bc0 0x7fe0201f41e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe01800c240 tx=0x7fe02006e0f0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.415+0000 7fe027029640 1 -- 172.21.15.44:0/3154056002 shutdown_connections 2024-04-02T13:21:55.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.415+0000 7fe027029640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe00006c920 0x7fe00006ede0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.415+0000 7fe027029640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe020155700 0x7fe020155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.415+0000 7fe027029640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe020153f00 0x7fe020158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.415+0000 7fe027029640 1 --2- 172.21.15.44:0/3154056002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe00c005bc0 0x7fe0201f41e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.415+0000 7fe027029640 1 -- 172.21.15.44:0/3154056002 >> 172.21.15.44:0/3154056002 conn(0x7fe0200cfcd0 msgr2=0x7fe020072a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:55.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.415+0000 7fe027029640 1 -- 172.21.15.44:0/3154056002 shutdown_connections 2024-04-02T13:21:55.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.415+0000 7fe027029640 1 -- 172.21.15.44:0/3154056002 wait complete. 2024-04-02T13:21:55.429 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:21:55.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:55.429 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:21:55.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[53818]: fuse finished with error 0 and tester_r 0 2024-04-02T13:21:55.456 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:21:55.500 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:21:55.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:21:55.501 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:21:55.516 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:21:55.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.657+0000 7f0e4eafb640 1 Processor -- start 2024-04-02T13:21:55.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.657+0000 7f0e4eafb640 1 -- start start 2024-04-02T13:21:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.658+0000 7f0e4eafb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0e480d0aa0 0x7f0e48166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.658+0000 7f0e4eafb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e480af420 con 0x7f0e4815c250 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.658+0000 7f0e4eafb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e480af700 con 0x7f0e480b3510 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.658+0000 7f0e4eafb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0e4815c630 con 0x7f0e480d0aa0 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.658+0000 7f0e4c897640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0e4815c250 0x7f0e480aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51154/0 (socket says 172.21.15.44:51154) 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.658+0000 7f0e4c897640 1 -- 172.21.15.44:0/2973676670 learned_addr learned my addr 172.21.15.44:0/2973676670 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e4d098640 1 --2- 172.21.15.44:0/2973676670 >> v2:172.21.15.69:3300/0 conn(0x7f0e480d0aa0 0x7f0e48166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3747583786 0 0) 0x7f0e480af420 con 0x7f0e4815c250 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e30003520 con 0x7f0e4815c250 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 208168971 0 0) 0x7f0e480af700 con 0x7f0e480b3510 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e480af420 con 0x7f0e480b3510 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3925903310 0 0) 0x7f0e30003520 con 0x7f0e4815c250 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 >> v2:172.21.15.69:3300/0 conn(0x7f0e480d0aa0 msgr2=0x7f0e48166cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 --2- 172.21.15.44:0/2973676670 >> v2:172.21.15.69:3300/0 conn(0x7f0e480d0aa0 0x7f0e48166cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 >> v1:172.21.15.69:6789/0 conn(0x7f0e480b3510 legacy=0x7f0e480d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e480af9e0 con 0x7f0e4815c250 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f0e400019f0 con 0x7f0e4815c250 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 >> v1:172.21.15.44:6789/0 conn(0x7f0e4815c250 legacy=0x7f0e480aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 --2- 172.21.15.44:0/2973676670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e30003b90 0x7f0e30003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 --2- 172.21.15.44:0/2973676670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e30004800 0x7f0e30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 --2- 172.21.15.44:0/2973676670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e30005210 0x7f0e3000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4815c630 con 0x7f0e30005210 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3000c140 con 0x7f0e30003b90 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0e3000c340 con 0x7f0e30004800 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e4d098640 1 --2- 172.21.15.44:0/2973676670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e30005210 0x7f0e3000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e4c897640 1 --2- 172.21.15.44:0/2973676670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e30003b90 0x7f0e30003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e47fff640 1 --2- 172.21.15.44:0/2973676670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e30004800 0x7f0e30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e4d098640 1 -- 172.21.15.44:0/2973676670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e30004800 msgr2=0x7f0e30004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e4d098640 1 --2- 172.21.15.44:0/2973676670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e30004800 0x7f0e30004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e4d098640 1 -- 172.21.15.44:0/2973676670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e30003b90 msgr2=0x7f0e30003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e4d098640 1 --2- 172.21.15.44:0/2973676670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e30003b90 0x7f0e30003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.659+0000 7f0e4d098640 1 -- 172.21.15.44:0/2973676670 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e30003520 con 0x7f0e30005210 2024-04-02T13:21:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.660+0000 7f0e4d098640 1 -- 172.21.15.44:0/2973676670 reap_dead start 2024-04-02T13:21:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.660+0000 7f0e4d098640 1 --2- 172.21.15.44:0/2973676670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e30005210 0x7f0e3000b9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0e38009a90 tx=0x7f0e30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f5ed418780c93d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.660+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e38009690 con 0x7f0e30005210 2024-04-02T13:21:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.660+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e38004020 con 0x7f0e30005210 2024-04-02T13:21:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.660+0000 7f0e477fe640 1 -- 172.21.15.44:0/2973676670 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e38004410 con 0x7f0e30005210 2024-04-02T13:21:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.661+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2973676670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e30005210 msgr2=0x7f0e3000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.661+0000 7f0e4eafb640 1 --2- 172.21.15.44:0/2973676670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e30005210 0x7f0e3000b9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0e38009a90 tx=0x7f0e30005830 comp rx=0 tx=0).stop 2024-04-02T13:21:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.661+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2973676670 shutdown_connections 2024-04-02T13:21:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.661+0000 7f0e4eafb640 1 --2- 172.21.15.44:0/2973676670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e30005210 0x7f0e3000b9c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.661+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2973676670 >> 172.21.15.44:0/2973676670 conn(0x7f0e48073850 msgr2=0x7f0e4809aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.661+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2973676670 shutdown_connections 2024-04-02T13:21:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.661+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2973676670 wait complete. 2024-04-02T13:21:55.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.662+0000 7f0e4eafb640 1 Processor -- start 2024-04-02T13:21:55.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.662+0000 7f0e4eafb640 1 -- start start 2024-04-02T13:21:55.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4eafb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e30005210 0x7f0e480b5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4eafb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e480cf090 0x7f0e480c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4eafb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e480b6df0 0x7f0e480b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4eafb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0e480d0510 con 0x7f0e480b6df0 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4eafb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0e480d0390 con 0x7f0e480cf090 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4eafb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0e480d0690 con 0x7f0e30005210 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4d098640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e480b6df0 0x7f0e480b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4d098640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e480b6df0 0x7f0e480b72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46492/0 (socket says 172.21.15.44:46492) 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4d098640 1 -- 172.21.15.44:0/2206635987 learned_addr learned my addr 172.21.15.44:0/2206635987 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4c897640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e30005210 0x7f0e480b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e47fff640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e480cf090 0x7f0e480c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4d098640 1 -- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e30005210 msgr2=0x7f0e480b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4d098640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e30005210 0x7f0e480b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4d098640 1 -- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e480cf090 msgr2=0x7f0e480c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.663+0000 7f0e4d098640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e480cf090 0x7f0e480c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.664+0000 7f0e4d098640 1 -- 172.21.15.44:0/2206635987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e480c8b40 con 0x7f0e480b6df0 2024-04-02T13:21:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.664+0000 7f0e47fff640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e480cf090 0x7f0e480c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:21:55.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.664+0000 7f0e4d098640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e480b6df0 0x7f0e480b72a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0e38009bb0 tx=0x7f0e38004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:55.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.664+0000 7f0e457fa640 1 -- 172.21.15.44:0/2206635987 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3802cbf0 con 0x7f0e480b6df0 2024-04-02T13:21:55.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.664+0000 7f0e457fa640 1 -- 172.21.15.44:0/2206635987 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e3802cd90 con 0x7f0e480b6df0 2024-04-02T13:21:55.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.664+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2206635987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e480cd320 con 0x7f0e480b6df0 2024-04-02T13:21:55.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.664+0000 7f0e457fa640 1 -- 172.21.15.44:0/2206635987 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e38003850 con 0x7f0e480b6df0 2024-04-02T13:21:55.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.664+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2206635987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e480cd5c0 con 0x7f0e480b6df0 2024-04-02T13:21:55.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.665+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2206635987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e48068c00 con 0x7f0e480b6df0 2024-04-02T13:21:55.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.665+0000 7f0e457fa640 1 -- 172.21.15.44:0/2206635987 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0e380039f0 con 0x7f0e480b6df0 2024-04-02T13:21:55.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.666+0000 7f0e457fa640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e3406c8f0 0x7f0e3406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:55.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.666+0000 7f0e457fa640 1 -- 172.21.15.44:0/2206635987 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f0e380acde0 con 0x7f0e480b6df0 2024-04-02T13:21:55.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.666+0000 7f0e4c897640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e3406c8f0 0x7f0e3406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:55.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.666+0000 7f0e4c897640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e3406c8f0 0x7f0e3406edb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0e40000f80 tx=0x7f0e40006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:55.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.670+0000 7f0e457fa640 1 -- 172.21.15.44:0/2206635987 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0e3807b0f0 con 0x7f0e480b6df0 2024-04-02T13:21:55.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.893+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2206635987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e480b0cd0 con 0x7f0e480b6df0 2024-04-02T13:21:55.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.894+0000 7f0e457fa640 1 -- 172.21.15.44:0/2206635987 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5181 (secure 0 0 0) 0x7f0e3807a9f0 con 0x7f0e480b6df0 2024-04-02T13:21:55.897 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:55.897 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3050514527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3050514527},{"type":"v1","addr":"172.21.15.69:6836","nonce":3050514527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4973,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1949024012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1949024012},{"type":"v1","addr":"172.21.15.69:6837","nonce":1949024012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/2817873198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2817873198},{"type":"v1","addr":"172.21.15.44:6834","nonce":2817873198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:21:43.215796+0000","modified":"2024-04-02T13:21:46.248294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/1969465285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":1969465285},{"type":"v1","addr":"172.21.15.44:6835","nonce":1969465285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":2}]} 2024-04-02T13:21:55.897 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 18 2024-04-02T13:21:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.896+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e3406c8f0 msgr2=0x7f0e3406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.896+0000 7f0e4eafb640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e3406c8f0 0x7f0e3406edb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0e40000f80 tx=0x7f0e40006040 comp rx=0 tx=0).stop 2024-04-02T13:21:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.896+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2206635987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e480b6df0 msgr2=0x7f0e480b72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.896+0000 7f0e4eafb640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e480b6df0 0x7f0e480b72a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0e38009bb0 tx=0x7f0e38004730 comp rx=0 tx=0).stop 2024-04-02T13:21:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.896+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2206635987 shutdown_connections 2024-04-02T13:21:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.896+0000 7f0e4eafb640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e3406c8f0 0x7f0e3406edb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.896+0000 7f0e4eafb640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e480b6df0 0x7f0e480b72a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.896+0000 7f0e4eafb640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e480cf090 0x7f0e480c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.896+0000 7f0e4eafb640 1 --2- 172.21.15.44:0/2206635987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e30005210 0x7f0e480b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:55.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.896+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2206635987 >> 172.21.15.44:0/2206635987 conn(0x7f0e48073850 msgr2=0x7f0e480b14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:55.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.896+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2206635987 shutdown_connections 2024-04-02T13:21:55.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:55.896+0000 7f0e4eafb640 1 -- 172.21.15.44:0/2206635987 wait complete. 2024-04-02T13:21:55.910 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-04-02T13:21:56.038 INFO:teuthology.orchestra.run.smithi044.stdout:{"message":"","return_code":0} 2024-04-02T13:21:56.038 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-02T13:21:56.038 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:21:56.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.138+0000 7ff0da0c7640 1 Processor -- start 2024-04-02T13:21:56.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.138+0000 7ff0da0c7640 1 -- start start 2024-04-02T13:21:56.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.139+0000 7ff0da0c7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0d40d28f0 0x7ff0d40d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:56.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.139+0000 7ff0da0c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d40b5200 0x7ff0d40b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:56.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.139+0000 7ff0da0c7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d40d4d70 con 0x7ff0d40d1260 2024-04-02T13:21:56.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.139+0000 7ff0da0c7640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff0d40d5050 con 0x7ff0d40d28f0 2024-04-02T13:21:56.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.139+0000 7ff0da0c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d40d23a0 con 0x7ff0d40b5200 2024-04-02T13:21:56.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.139+0000 7ff0d37fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0d40d28f0 0x7ff0d40d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:56.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.139+0000 7ff0d37fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0d40d28f0 0x7ff0d40d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46504/0 (socket says 172.21.15.44:46504) 2024-04-02T13:21:56.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.139+0000 7ff0d37fe640 1 -- 172.21.15.44:0/1550118342 learned_addr learned my addr 172.21.15.44:0/1550118342 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:56.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.139+0000 7ff0d2ffd640 1 --2- 172.21.15.44:0/1550118342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d40b5200 0x7ff0d40b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:56.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.139+0000 7ff0d37fe640 1 -- 172.21.15.44:0/1550118342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d40b5200 msgr2=0x7ff0d40b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:56.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.139+0000 7ff0d37fe640 1 --2- 172.21.15.44:0/1550118342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d40b5200 0x7ff0d40b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:56.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.139+0000 7ff0d37fe640 1 -- 172.21.15.44:0/1550118342 >> v1:172.21.15.69:6789/0 conn(0x7ff0d40d1260 legacy=0x7ff0d40ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:21:56.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.139+0000 7ff0d37fe640 1 -- 172.21.15.44:0/1550118342 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d40d2030 con 0x7ff0d40d28f0 2024-04-02T13:21:56.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.140+0000 7ff0d37fe640 1 --2- 172.21.15.44:0/1550118342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d40d28f0 0x7ff0d40d2cd0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009870 tx=0x7ff0c402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628dfdbfa5e60531 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:56.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.140+0000 7ff0d27fc640 1 -- 172.21.15.44:0/1550118342 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4038070 con 0x7ff0d40d28f0 2024-04-02T13:21:56.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.140+0000 7ff0d27fc640 1 -- 172.21.15.44:0/1550118342 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c4031740 con 0x7ff0d40d28f0 2024-04-02T13:21:56.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.140+0000 7ff0d27fc640 1 -- 172.21.15.44:0/1550118342 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c402f050 con 0x7ff0d40d28f0 2024-04-02T13:21:56.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.141+0000 7ff0da0c7640 1 -- 172.21.15.44:0/1550118342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d40d28f0 msgr2=0x7ff0d40d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:56.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.141+0000 7ff0da0c7640 1 --2- 172.21.15.44:0/1550118342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d40d28f0 0x7ff0d40d2cd0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009870 tx=0x7ff0c402b7e0 comp rx=0 tx=0).stop 2024-04-02T13:21:56.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.141+0000 7ff0da0c7640 1 -- 172.21.15.44:0/1550118342 shutdown_connections 2024-04-02T13:21:56.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.141+0000 7ff0da0c7640 1 --2- 172.21.15.44:0/1550118342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d40b5200 0x7ff0d40b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:56.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.141+0000 7ff0da0c7640 1 --2- 172.21.15.44:0/1550118342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d40d28f0 0x7ff0d40d2cd0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:56.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.141+0000 7ff0da0c7640 1 -- 172.21.15.44:0/1550118342 >> 172.21.15.44:0/1550118342 conn(0x7ff0d40d4120 msgr2=0x7ff0d40d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:56.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.141+0000 7ff0da0c7640 1 -- 172.21.15.44:0/1550118342 shutdown_connections 2024-04-02T13:21:56.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.141+0000 7ff0da0c7640 1 -- 172.21.15.44:0/1550118342 wait complete. 2024-04-02T13:21:56.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.142+0000 7ff0da0c7640 1 Processor -- start 2024-04-02T13:21:56.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.142+0000 7ff0da0c7640 1 -- start start 2024-04-02T13:21:56.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0da0c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d40b5200 0x7ff0d4156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:56.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0da0c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d40d1260 0x7ff0d40b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:56.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0da0c7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d40d28f0 0x7ff0d41e7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:56.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0da0c7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d40b8130 con 0x7ff0d40d28f0 2024-04-02T13:21:56.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0da0c7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d40b5020 con 0x7ff0d40b5200 2024-04-02T13:21:56.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0da0c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d40b82b0 con 0x7ff0d40d1260 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0d3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d40d28f0 0x7ff0d41e7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0d3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d40d28f0 0x7ff0d41e7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46506/0 (socket says 172.21.15.44:46506) 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0d3fff640 1 -- 172.21.15.44:0/3728310933 learned_addr learned my addr 172.21.15.44:0/3728310933 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0d2ffd640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d40d1260 0x7ff0d40b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0d37fe640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d40b5200 0x7ff0d4156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0d3fff640 1 -- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d40d1260 msgr2=0x7ff0d40b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0d3fff640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d40d1260 0x7ff0d40b35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0d3fff640 1 -- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d40b5200 msgr2=0x7ff0d4156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0d3fff640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d40b5200 0x7ff0d4156f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0d3fff640 1 -- 172.21.15.44:0/3728310933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d40b2500 con 0x7ff0d40d28f0 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0d2ffd640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d40d1260 0x7ff0d40b35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0d37fe640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d40b5200 0x7ff0d4156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.143+0000 7ff0d3fff640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d40d28f0 0x7ff0d41e7fe0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00b9a0 tx=0x7ff0cc00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.144+0000 7ff0bbfff640 1 -- 172.21.15.44:0/3728310933 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc019070 con 0x7ff0d40d28f0 2024-04-02T13:21:56.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.144+0000 7ff0bbfff640 1 -- 172.21.15.44:0/3728310933 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0cc00cba0 con 0x7ff0d40d28f0 2024-04-02T13:21:56.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.144+0000 7ff0da0c7640 1 -- 172.21.15.44:0/3728310933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d40b27e0 con 0x7ff0d40d28f0 2024-04-02T13:21:56.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.144+0000 7ff0bbfff640 1 -- 172.21.15.44:0/3728310933 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc014440 con 0x7ff0d40d28f0 2024-04-02T13:21:56.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.144+0000 7ff0da0c7640 1 -- 172.21.15.44:0/3728310933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d40b2da0 con 0x7ff0d40d28f0 2024-04-02T13:21:56.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.145+0000 7ff0da0c7640 1 -- 172.21.15.44:0/3728310933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0d40684f0 con 0x7ff0d40d28f0 2024-04-02T13:21:56.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.145+0000 7ff0bbfff640 1 -- 172.21.15.44:0/3728310933 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0cc01c070 con 0x7ff0d40d28f0 2024-04-02T13:21:56.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.146+0000 7ff0bbfff640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0c006c920 0x7ff0c006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:21:56.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.146+0000 7ff0bbfff640 1 -- 172.21.15.44:0/3728310933 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7ff0cc08b8d0 con 0x7ff0d40d28f0 2024-04-02T13:21:56.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.146+0000 7ff0d37fe640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0c006c920 0x7ff0c006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:21:56.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.146+0000 7ff0d37fe640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0c006c920 0x7ff0c006ede0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4035000 tx=0x7ff0c4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:21:56.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.150+0000 7ff0bbfff640 1 -- 172.21.15.44:0/3728310933 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff0cc059b70 con 0x7ff0d40d28f0 2024-04-02T13:21:56.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.373+0000 7ff0da0c7640 1 -- 172.21.15.44:0/3728310933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0d40b6580 con 0x7ff0d40d28f0 2024-04-02T13:21:56.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.373+0000 7ff0bbfff640 1 -- 172.21.15.44:0/3728310933 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5181 (secure 0 0 0) 0x7ff0cc059470 con 0x7ff0d40d28f0 2024-04-02T13:21:56.376 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:21:56.376 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3050514527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3050514527},{"type":"v1","addr":"172.21.15.69:6836","nonce":3050514527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4973,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1949024012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1949024012},{"type":"v1","addr":"172.21.15.69:6837","nonce":1949024012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/2817873198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2817873198},{"type":"v1","addr":"172.21.15.44:6834","nonce":2817873198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:21:43.215796+0000","modified":"2024-04-02T13:21:46.248294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/1969465285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":1969465285},{"type":"v1","addr":"172.21.15.44:6835","nonce":1969465285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":2}]} 2024-04-02T13:21:56.376 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 18 2024-04-02T13:21:56.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.375+0000 7ff0da0c7640 1 -- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0c006c920 msgr2=0x7ff0c006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:56.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.375+0000 7ff0da0c7640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0c006c920 0x7ff0c006ede0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4035000 tx=0x7ff0c4008030 comp rx=0 tx=0).stop 2024-04-02T13:21:56.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.375+0000 7ff0da0c7640 1 -- 172.21.15.44:0/3728310933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d40d28f0 msgr2=0x7ff0d41e7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:21:56.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.375+0000 7ff0da0c7640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d40d28f0 0x7ff0d41e7fe0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00b9a0 tx=0x7ff0cc00be60 comp rx=0 tx=0).stop 2024-04-02T13:21:56.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.375+0000 7ff0da0c7640 1 -- 172.21.15.44:0/3728310933 shutdown_connections 2024-04-02T13:21:56.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.375+0000 7ff0da0c7640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0c006c920 0x7ff0c006ede0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:56.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.375+0000 7ff0da0c7640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d40d28f0 0x7ff0d41e7fe0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:56.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.375+0000 7ff0da0c7640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d40d1260 0x7ff0d40b35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:56.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.375+0000 7ff0da0c7640 1 --2- 172.21.15.44:0/3728310933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d40b5200 0x7ff0d4156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:21:56.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.375+0000 7ff0da0c7640 1 -- 172.21.15.44:0/3728310933 >> 172.21.15.44:0/3728310933 conn(0x7ff0d40d4120 msgr2=0x7ff0d40b8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:21:56.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.375+0000 7ff0da0c7640 1 -- 172.21.15.44:0/3728310933 shutdown_connections 2024-04-02T13:21:56.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:21:56.375+0000 7ff0da0c7640 1 -- 172.21.15.44:0/3728310933 wait complete. 2024-04-02T13:21:56.389 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json tag path /parentdir mytag 2024-04-02T13:21:56.512 INFO:teuthology.orchestra.run.smithi044.stdout:{"return_code":0,"scrub_tag":"mytag","mode":"asynchronous"} 2024-04-02T13:21:56.512 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag" } 2024-04-02T13:22:06.514 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000000.00000000 scrub_tag 2024-04-02T13:22:06.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.561+0000 7fad96366f40 1 Processor -- start 2024-04-02T13:22:06.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.561+0000 7fad96366f40 1 -- start start 2024-04-02T13:22:06.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.562+0000 7fad96366f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55be5fbfc780 0x55be5fc860e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.562+0000 7fad96366f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55be5fbfe220 0x55be5fc81750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.562+0000 7fad96366f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5fc88430 0x55be5fbb6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.562+0000 7fad96366f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55be5fb5b3e0 con 0x55be5fc88430 2024-04-02T13:22:06.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.562+0000 7fad96366f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55be5fb5f430 con 0x55be5fbfe220 2024-04-02T13:22:06.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.562+0000 7fad96366f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55be5fc88a20 con 0x55be5fbfc780 2024-04-02T13:22:06.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.562+0000 7fad9466a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5fc88430 0x55be5fbb6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.562+0000 7fad9466a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5fc88430 0x55be5fbb6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42820/0 (socket says 172.21.15.44:42820) 2024-04-02T13:22:06.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.562+0000 7fad9466a640 1 -- 172.21.15.44:0/4257480113 learned_addr learned my addr 172.21.15.44:0/4257480113 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:06.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.562+0000 7fad93e69640 1 --2- 172.21.15.44:0/4257480113 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55be5fbfc780 0x55be5fc860e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.562+0000 7fad93668640 1 --2- 172.21.15.44:0/4257480113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55be5fbfe220 0x55be5fc81750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.563+0000 7fad9466a640 1 -- 172.21.15.44:0/4257480113 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55be5fbfc780 msgr2=0x55be5fc860e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.563+0000 7fad9466a640 1 --2- 172.21.15.44:0/4257480113 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55be5fbfc780 0x55be5fc860e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.563+0000 7fad9466a640 1 -- 172.21.15.44:0/4257480113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55be5fbfe220 msgr2=0x55be5fc81750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.563+0000 7fad9466a640 1 --2- 172.21.15.44:0/4257480113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55be5fbfe220 0x55be5fc81750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.563+0000 7fad9466a640 1 -- 172.21.15.44:0/4257480113 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be5fc78230 con 0x55be5fc88430 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.563+0000 7fad93668640 1 --2- 172.21.15.44:0/4257480113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55be5fbfe220 0x55be5fc81750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.563+0000 7fad9466a640 1 --2- 172.21.15.44:0/4257480113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5fc88430 0x55be5fbb6610 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fad8c001650 tx=0x7fad8c001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2e78aca132edd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.563+0000 7fad92666640 1 -- 172.21.15.44:0/4257480113 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c016070 con 0x55be5fc88430 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.563+0000 7fad92666640 1 -- 172.21.15.44:0/4257480113 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad8c00f730 con 0x55be5fc88430 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.563+0000 7fad92666640 1 -- 172.21.15.44:0/4257480113 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c00fb20 con 0x55be5fc88430 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.564+0000 7fad96366f40 1 -- 172.21.15.44:0/4257480113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5fc88430 msgr2=0x55be5fbb6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.564+0000 7fad96366f40 1 --2- 172.21.15.44:0/4257480113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5fc88430 0x55be5fbb6610 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fad8c001650 tx=0x7fad8c001b10 comp rx=0 tx=0).stop 2024-04-02T13:22:06.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.564+0000 7fad96366f40 1 -- 172.21.15.44:0/4257480113 shutdown_connections 2024-04-02T13:22:06.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.564+0000 7fad96366f40 1 --2- 172.21.15.44:0/4257480113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5fc88430 0x55be5fbb6610 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.564+0000 7fad96366f40 1 --2- 172.21.15.44:0/4257480113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55be5fbfe220 0x55be5fc81750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.564+0000 7fad96366f40 1 --2- 172.21.15.44:0/4257480113 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55be5fbfc780 0x55be5fc860e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.564+0000 7fad96366f40 1 -- 172.21.15.44:0/4257480113 >> 172.21.15.44:0/4257480113 conn(0x55be5fbe1500 msgr2=0x55be5fbe2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:06.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.564+0000 7fad96366f40 1 -- 172.21.15.44:0/4257480113 shutdown_connections 2024-04-02T13:22:06.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.564+0000 7fad96366f40 1 -- 172.21.15.44:0/4257480113 wait complete. 2024-04-02T13:22:06.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.564+0000 7fad96366f40 1 Processor -- start 2024-04-02T13:22:06.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.564+0000 7fad96366f40 1 -- start start 2024-04-02T13:22:06.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.565+0000 7fad96366f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55be5fbfc780 0x55be5fc8a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.565+0000 7fad96366f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55be5fbfe220 0x55be5fbdf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.565+0000 7fad96366f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5fc88430 0x55be5fbdff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.565+0000 7fad96366f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55be5fc81c90 con 0x55be5fc88430 2024-04-02T13:22:06.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.565+0000 7fad96366f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55be5fc77ff0 con 0x55be5fbfc780 2024-04-02T13:22:06.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.565+0000 7fad96366f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55be5fc88e70 con 0x55be5fbfe220 2024-04-02T13:22:06.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.565+0000 7fad9466a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5fc88430 0x55be5fbdff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.565+0000 7fad9466a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5fc88430 0x55be5fbdff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42826/0 (socket says 172.21.15.44:42826) 2024-04-02T13:22:06.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.565+0000 7fad9466a640 1 -- 172.21.15.44:0/1073506007 learned_addr learned my addr 172.21.15.44:0/1073506007 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.565+0000 7fad93668640 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55be5fbfe220 0x55be5fbdf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.565+0000 7fad93e69640 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55be5fbfc780 0x55be5fc8a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.566+0000 7fad9466a640 1 -- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55be5fbfe220 msgr2=0x55be5fbdf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.566+0000 7fad9466a640 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55be5fbfe220 0x55be5fbdf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.566+0000 7fad9466a640 1 -- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55be5fbfc780 msgr2=0x55be5fc8a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.566+0000 7fad9466a640 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55be5fbfc780 0x55be5fc8a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.566+0000 7fad9466a640 1 -- 172.21.15.44:0/1073506007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be5fd28480 con 0x55be5fc88430 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.566+0000 7fad93e69640 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55be5fbfc780 0x55be5fc8a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.566+0000 7fad9466a640 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5fc88430 0x55be5fbdff10 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fad8c002010 tx=0x7fad8c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.566+0000 7fad91664640 1 -- 172.21.15.44:0/1073506007 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c021020 con 0x55be5fc88430 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.566+0000 7fad91664640 1 -- 172.21.15.44:0/1073506007 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad8c00fea0 con 0x55be5fc88430 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.566+0000 7fad96366f40 1 -- 172.21.15.44:0/1073506007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55be5fd28760 con 0x55be5fc88430 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.566+0000 7fad91664640 1 -- 172.21.15.44:0/1073506007 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c016030 con 0x55be5fc88430 2024-04-02T13:22:06.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.566+0000 7fad96366f40 1 -- 172.21.15.44:0/1073506007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be5fc82db0 con 0x55be5fc88430 2024-04-02T13:22:06.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.568+0000 7fad91664640 1 -- 172.21.15.44:0/1073506007 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fad8c024020 con 0x55be5fc88430 2024-04-02T13:22:06.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.568+0000 7fad91664640 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fad7406b2d0 0x7fad7406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.568+0000 7fad91664640 1 -- 172.21.15.44:0/1073506007 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fad8c08bdd0 con 0x55be5fc88430 2024-04-02T13:22:06.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.568+0000 7fad93e69640 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fad7406b2d0 0x7fad7406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.568+0000 7fad96366f40 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55be5fc8ad20 0x55be5fbe0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.568+0000 7fad96366f40 1 -- 172.21.15.44:0/1073506007 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:1 5.10 5:0cf6ad0f:::10000000000.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55be5fc832d0 con 0x55be5fc8ad20 2024-04-02T13:22:06.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.569+0000 7fad93668640 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55be5fc8ad20 0x55be5fbe0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.569+0000 7fad93e69640 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fad7406b2d0 0x7fad7406d790 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fad84054000 tx=0x7fad8404e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.569+0000 7fad93668640 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55be5fc8ad20 0x55be5fbe0450 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.569+0000 7fad93668640 1 -- 172.21.15.44:0/1073506007 <== osd.7 v2:172.21.15.69:6824/1292475712 1 ==== osd_op_reply(1 10000000000.00000000 [getxattr (9) out=9b] v0'0 uv11 ondisk = 0) ==== 164+0+9 (crc 0 0 0) 0x7fad7c0107f0 con 0x55be5fc8ad20 2024-04-02T13:22:06.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.569+0000 7fad96366f40 1 -- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55be5fc8ad20 msgr2=0x55be5fbe0450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.569+0000 7fad96366f40 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55be5fc8ad20 0x55be5fbe0450 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.570+0000 7fad96366f40 1 -- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fad7406b2d0 msgr2=0x7fad7406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.570+0000 7fad96366f40 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fad7406b2d0 0x7fad7406d790 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fad84054000 tx=0x7fad8404e000 comp rx=0 tx=0).stop 2024-04-02T13:22:06.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.570+0000 7fad96366f40 1 -- 172.21.15.44:0/1073506007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5fc88430 msgr2=0x55be5fbdff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.570+0000 7fad96366f40 1 --2- 172.21.15.44:0/1073506007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5fc88430 0x55be5fbdff10 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fad8c002010 tx=0x7fad8c000f40 comp rx=0 tx=0).stop 2024-04-02T13:22:06.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.570+0000 7fad9466a640 1 -- 172.21.15.44:0/1073506007 reap_dead start 2024-04-02T13:22:06.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.570+0000 7fad96366f40 1 -- 172.21.15.44:0/1073506007 shutdown_connections 2024-04-02T13:22:06.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.570+0000 7fad96366f40 1 -- 172.21.15.44:0/1073506007 >> 172.21.15.44:0/1073506007 conn(0x55be5fbe1500 msgr2=0x55be5fc84b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:06.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.570+0000 7fad96366f40 1 -- 172.21.15.44:0/1073506007 shutdown_connections 2024-04-02T13:22:06.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.570+0000 7fad96366f40 1 -- 172.21.15.44:0/1073506007 wait complete. 2024-04-02T13:22:06.576 INFO:teuthology.orchestra.run.smithi044.stdout:mytag 2024-04-02T13:22:06.576 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000001.00000000 scrub_tag 2024-04-02T13:22:06.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.619+0000 7fb9a244cf40 1 Processor -- start 2024-04-02T13:22:06.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.619+0000 7fb9a244cf40 1 -- start start 2024-04-02T13:22:06.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb9a244cf40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563adebb79a0 0x563adeb718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb9a244cf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563adec2d9b0 0x563adeb71e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb9a244cf40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563adec2dd90 0x563adec28220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb9a244cf40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563adec2d4b0 con 0x563adec2dd90 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb9a244cf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563adeb173e0 con 0x563adec2d9b0 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb9a244cf40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x563adebb7f90 con 0x563adebb79a0 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb99feb5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563adec2dd90 0x563adec28220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb99feb5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563adec2dd90 0x563adec28220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42844/0 (socket says 172.21.15.44:42844) 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb99feb5640 1 -- 172.21.15.44:0/1691498104 learned_addr learned my addr 172.21.15.44:0/1691498104 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb99f6b4640 1 --2- 172.21.15.44:0/1691498104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563adebb79a0 0x563adeb718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb99eeb3640 1 --2- 172.21.15.44:0/1691498104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563adec2d9b0 0x563adeb71e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb99feb5640 1 -- 172.21.15.44:0/1691498104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563adebb79a0 msgr2=0x563adeb718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb99feb5640 1 --2- 172.21.15.44:0/1691498104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563adebb79a0 0x563adeb718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb99feb5640 1 -- 172.21.15.44:0/1691498104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563adec2d9b0 msgr2=0x563adeb71e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:06.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb99feb5640 1 --2- 172.21.15.44:0/1691498104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563adec2d9b0 0x563adeb71e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb99feb5640 1 -- 172.21.15.44:0/1691498104 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563adec2a600 con 0x563adec2dd90 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.620+0000 7fb99feb5640 1 --2- 172.21.15.44:0/1691498104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563adec2dd90 0x563adec28220 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb998001530 tx=0x7fb9980019f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc05f708074f6d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.621+0000 7fb99deb1640 1 -- 172.21.15.44:0/1691498104 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99800f710 con 0x563adec2dd90 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.621+0000 7fb99deb1640 1 -- 172.21.15.44:0/1691498104 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb99800f8d0 con 0x563adec2dd90 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.621+0000 7fb99deb1640 1 -- 172.21.15.44:0/1691498104 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99800fcc0 con 0x563adec2dd90 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.621+0000 7fb9a244cf40 1 -- 172.21.15.44:0/1691498104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563adec2dd90 msgr2=0x563adec28220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.621+0000 7fb9a244cf40 1 --2- 172.21.15.44:0/1691498104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563adec2dd90 0x563adec28220 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb998001530 tx=0x7fb9980019f0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.621+0000 7fb9a244cf40 1 -- 172.21.15.44:0/1691498104 shutdown_connections 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.621+0000 7fb9a244cf40 1 --2- 172.21.15.44:0/1691498104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563adec2dd90 0x563adec28220 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.621+0000 7fb9a244cf40 1 --2- 172.21.15.44:0/1691498104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563adec2d9b0 0x563adeb71e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.621+0000 7fb9a244cf40 1 --2- 172.21.15.44:0/1691498104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563adebb79a0 0x563adeb718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.621+0000 7fb9a244cf40 1 -- 172.21.15.44:0/1691498104 >> 172.21.15.44:0/1691498104 conn(0x563adeb701c0 msgr2=0x563adec4a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.621+0000 7fb9a244cf40 1 -- 172.21.15.44:0/1691498104 shutdown_connections 2024-04-02T13:22:06.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.621+0000 7fb9a244cf40 1 -- 172.21.15.44:0/1691498104 wait complete. 2024-04-02T13:22:06.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.622+0000 7fb9a244cf40 1 Processor -- start 2024-04-02T13:22:06.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.622+0000 7fb9a244cf40 1 -- start start 2024-04-02T13:22:06.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.622+0000 7fb9a244cf40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563adebb79a0 0x563adec42590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.622+0000 7fb9a244cf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563adec2d9b0 0x563adec46fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.622+0000 7fb9a244cf40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563adec2dd90 0x563adec491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb9a244cf40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563adeb72f50 con 0x563adebb79a0 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb9a244cf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563adeb9c3c0 con 0x563adec2d9b0 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb9a244cf40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x563adec2aae0 con 0x563adec2dd90 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99f6b4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563adebb79a0 0x563adec42590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99f6b4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563adebb79a0 0x563adec42590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42846/0 (socket says 172.21.15.44:42846) 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99f6b4640 1 -- 172.21.15.44:0/2071234115 learned_addr learned my addr 172.21.15.44:0/2071234115 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99eeb3640 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563adec2d9b0 0x563adec46fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99feb5640 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563adec2dd90 0x563adec491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99f6b4640 1 -- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563adec2dd90 msgr2=0x563adec491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99f6b4640 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563adec2dd90 0x563adec491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99f6b4640 1 -- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563adec2d9b0 msgr2=0x563adec46fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99f6b4640 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563adec2d9b0 0x563adec46fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99f6b4640 1 -- 172.21.15.44:0/2071234115 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563adec27690 con 0x563adebb79a0 2024-04-02T13:22:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99f6b4640 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563adebb79a0 0x563adec42590 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb990008810 tx=0x7fb990008900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99ceaf640 1 -- 172.21.15.44:0/2071234115 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9900147e0 con 0x563adebb79a0 2024-04-02T13:22:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99ceaf640 1 -- 172.21.15.44:0/2071234115 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb99001a070 con 0x563adebb79a0 2024-04-02T13:22:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb99ceaf640 1 -- 172.21.15.44:0/2071234115 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990014b80 con 0x563adebb79a0 2024-04-02T13:22:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb9a244cf40 1 -- 172.21.15.44:0/2071234115 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563adeb99fd0 con 0x563adebb79a0 2024-04-02T13:22:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.623+0000 7fb9a244cf40 1 -- 172.21.15.44:0/2071234115 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563adebb5bc0 con 0x563adebb79a0 2024-04-02T13:22:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.625+0000 7fb99ceaf640 1 -- 172.21.15.44:0/2071234115 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb990015470 con 0x563adebb79a0 2024-04-02T13:22:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.625+0000 7fb99ceaf640 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb98006b2d0 0x7fb98006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.625+0000 7fb99ceaf640 1 -- 172.21.15.44:0/2071234115 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fb99008b210 con 0x563adebb79a0 2024-04-02T13:22:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.625+0000 7fb99eeb3640 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb98006b2d0 0x7fb98006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.625+0000 7fb9a244cf40 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563adec496e0 0x563adec40780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.625+0000 7fb9a244cf40 1 -- 172.21.15.44:0/2071234115 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:1 5.7 5:e2e07930:::10000000001.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x563adec3f5d0 con 0x563adec496e0 2024-04-02T13:22:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.626+0000 7fb99feb5640 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563adec496e0 0x563adec40780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.626+0000 7fb99eeb3640 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb98006b2d0 0x7fb98006d790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb988054000 tx=0x7fb98804e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.626+0000 7fb99feb5640 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563adec496e0 0x563adec40780 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.627+0000 7fb99feb5640 1 -- 172.21.15.44:0/2071234115 <== osd.5 v2:172.21.15.69:6800/2841767499 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (9) out=9b] v0'0 uv5 ondisk = 0) ==== 164+0+9 (crc 0 0 0) 0x7fb99800f070 con 0x563adec496e0 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.627+0000 7fb9a244cf40 1 -- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563adec496e0 msgr2=0x563adec40780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.627+0000 7fb9a244cf40 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563adec496e0 0x563adec40780 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.627+0000 7fb9a244cf40 1 -- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb98006b2d0 msgr2=0x7fb98006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.627+0000 7fb9a244cf40 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb98006b2d0 0x7fb98006d790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb988054000 tx=0x7fb98804e000 comp rx=0 tx=0).stop 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.627+0000 7fb9a244cf40 1 -- 172.21.15.44:0/2071234115 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563adebb79a0 msgr2=0x563adec42590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.627+0000 7fb9a244cf40 1 --2- 172.21.15.44:0/2071234115 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563adebb79a0 0x563adec42590 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb990008810 tx=0x7fb990008900 comp rx=0 tx=0).stop 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.627+0000 7fb99feb5640 1 -- 172.21.15.44:0/2071234115 reap_dead start 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.627+0000 7fb9a244cf40 1 -- 172.21.15.44:0/2071234115 shutdown_connections 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.627+0000 7fb9a244cf40 1 -- 172.21.15.44:0/2071234115 >> 172.21.15.44:0/2071234115 conn(0x563adeb701c0 msgr2=0x563adec4a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.627+0000 7fb9a244cf40 1 -- 172.21.15.44:0/2071234115 shutdown_connections 2024-04-02T13:22:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.627+0000 7fb9a244cf40 1 -- 172.21.15.44:0/2071234115 wait complete. 2024-04-02T13:22:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:mytag 2024-04-02T13:22:06.634 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2024-04-02T13:22:06.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.674+0000 7fb21c83cf40 1 Processor -- start 2024-04-02T13:22:06.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.675+0000 7fb21c83cf40 1 -- start start 2024-04-02T13:22:06.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.675+0000 7fb21c83cf40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563355c8c440 0x563355c8a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.675+0000 7fb21c83cf40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563355d43640 0x563355d48a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.675+0000 7fb21c83cf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563355d46330 0x563355d48f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.675+0000 7fb21c83cf40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563355d5b1f0 con 0x563355d43640 2024-04-02T13:22:06.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.675+0000 7fb21c83cf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563355c323e0 con 0x563355d46330 2024-04-02T13:22:06.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.675+0000 7fb21c83cf40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x563355d446a0 con 0x563355c8c440 2024-04-02T13:22:06.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.675+0000 7fb219399640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563355d43640 0x563355d48a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.675+0000 7fb219399640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563355d43640 0x563355d48a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42850/0 (socket says 172.21.15.44:42850) 2024-04-02T13:22:06.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.675+0000 7fb219399640 1 -- 172.21.15.44:0/3536306319 learned_addr learned my addr 172.21.15.44:0/3536306319 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:06.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.675+0000 7fb219b9a640 1 --2- 172.21.15.44:0/3536306319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563355c8c440 0x563355c8a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb21a39b640 1 --2- 172.21.15.44:0/3536306319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563355d46330 0x563355d48f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb219399640 1 -- 172.21.15.44:0/3536306319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563355c8c440 msgr2=0x563355c8a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:06.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb219399640 1 --2- 172.21.15.44:0/3536306319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563355c8c440 0x563355c8a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb219399640 1 -- 172.21.15.44:0/3536306319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563355d46330 msgr2=0x563355d48f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:06.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb219399640 1 --2- 172.21.15.44:0/3536306319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563355d46330 0x563355d48f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb219399640 1 -- 172.21.15.44:0/3536306319 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563355cd4290 con 0x563355d43640 2024-04-02T13:22:06.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb219b9a640 1 --2- 172.21.15.44:0/3536306319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563355c8c440 0x563355c8a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:06.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb219399640 1 --2- 172.21.15.44:0/3536306319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563355d43640 0x563355d48a40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb204006ec0 tx=0x7fb20400b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ef72fc0d0ed5ed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb213fff640 1 -- 172.21.15.44:0/3536306319 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb204012070 con 0x563355d43640 2024-04-02T13:22:06.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb213fff640 1 -- 172.21.15.44:0/3536306319 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb20400e690 con 0x563355d43640 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb213fff640 1 -- 172.21.15.44:0/3536306319 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20400eb00 con 0x563355d43640 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb21c83cf40 1 -- 172.21.15.44:0/3536306319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563355d43640 msgr2=0x563355d48a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb21c83cf40 1 --2- 172.21.15.44:0/3536306319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563355d43640 0x563355d48a40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb204006ec0 tx=0x7fb20400b6c0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb21c83cf40 1 -- 172.21.15.44:0/3536306319 shutdown_connections 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb21c83cf40 1 --2- 172.21.15.44:0/3536306319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563355d46330 0x563355d48f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb21c83cf40 1 --2- 172.21.15.44:0/3536306319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563355d43640 0x563355d48a40 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb21c83cf40 1 --2- 172.21.15.44:0/3536306319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563355c8c440 0x563355c8a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb21c83cf40 1 -- 172.21.15.44:0/3536306319 >> 172.21.15.44:0/3536306319 conn(0x563355cb7c80 msgr2=0x563355c89f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb21c83cf40 1 -- 172.21.15.44:0/3536306319 shutdown_connections 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.676+0000 7fb21c83cf40 1 -- 172.21.15.44:0/3536306319 wait complete. 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.677+0000 7fb21c83cf40 1 Processor -- start 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.677+0000 7fb21c83cf40 1 -- start start 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb21c83cf40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563355c8c440 0x563355d60000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb21c83cf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563355d43640 0x563355cba0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb21c83cf40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563355d46330 0x563355d49300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb21c83cf40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563355d49eb0 con 0x563355c8c440 2024-04-02T13:22:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb21c83cf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563355d623c0 con 0x563355d43640 2024-04-02T13:22:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb21c83cf40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x563355d430a0 con 0x563355d46330 2024-04-02T13:22:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb219b9a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563355c8c440 0x563355d60000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb219b9a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563355c8c440 0x563355d60000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42854/0 (socket says 172.21.15.44:42854) 2024-04-02T13:22:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb219b9a640 1 -- 172.21.15.44:0/164768627 learned_addr learned my addr 172.21.15.44:0/164768627 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb21a39b640 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563355d46330 0x563355d49300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb219399640 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563355d43640 0x563355cba0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb219b9a640 1 -- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563355d46330 msgr2=0x563355d49300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb219b9a640 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563355d46330 0x563355d49300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb219b9a640 1 -- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563355d43640 msgr2=0x563355cba0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb219b9a640 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563355d43640 0x563355cba0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb219b9a640 1 -- 172.21.15.44:0/164768627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563355cb8590 con 0x563355c8c440 2024-04-02T13:22:06.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb219399640 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563355d43640 0x563355cba0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:06.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb219b9a640 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563355c8c440 0x563355d60000 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00a4a0 tx=0x7fb20c00a960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb212ffd640 1 -- 172.21.15.44:0/164768627 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c015030 con 0x563355c8c440 2024-04-02T13:22:06.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb212ffd640 1 -- 172.21.15.44:0/164768627 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb20c011690 con 0x563355c8c440 2024-04-02T13:22:06.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb212ffd640 1 -- 172.21.15.44:0/164768627 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c011a60 con 0x563355c8c440 2024-04-02T13:22:06.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.678+0000 7fb21c83cf40 1 -- 172.21.15.44:0/164768627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563355cb8870 con 0x563355c8c440 2024-04-02T13:22:06.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.679+0000 7fb21c83cf40 1 -- 172.21.15.44:0/164768627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563355d5f880 con 0x563355c8c440 2024-04-02T13:22:06.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.680+0000 7fb212ffd640 1 -- 172.21.15.44:0/164768627 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb20c018070 con 0x563355c8c440 2024-04-02T13:22:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.680+0000 7fb212ffd640 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb1fc06b2d0 0x7fb1fc06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.680+0000 7fb212ffd640 1 -- 172.21.15.44:0/164768627 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fb20c08af10 con 0x563355c8c440 2024-04-02T13:22:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.680+0000 7fb219399640 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb1fc06b2d0 0x7fb1fc06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.680+0000 7fb21c83cf40 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563355d49840 0x563355dfbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.680+0000 7fb21c83cf40 1 -- 172.21.15.44:0/164768627 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:1 6.36 6:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x563355dfc1d0 con 0x563355d49840 2024-04-02T13:22:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.681+0000 7fb21a39b640 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563355d49840 0x563355dfbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.681+0000 7fb219399640 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb1fc06b2d0 0x7fb1fc06d790 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x563355d405d0 tx=0x7fb204019040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.681+0000 7fb21a39b640 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563355d49840 0x563355dfbc90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.681+0000 7fb21a39b640 1 -- 172.21.15.44:0/164768627 <== osd.5 v2:172.21.15.69:6800/2841767499 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr (9) out=9b] v0'0 uv17 ondisk = 0) ==== 164+0+9 (crc 0 0 0) 0x7fb214052880 con 0x563355d49840 2024-04-02T13:22:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.681+0000 7fb21c83cf40 1 -- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563355d49840 msgr2=0x563355dfbc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.681+0000 7fb21c83cf40 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563355d49840 0x563355dfbc90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.681+0000 7fb21c83cf40 1 -- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb1fc06b2d0 msgr2=0x7fb1fc06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.681+0000 7fb21c83cf40 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb1fc06b2d0 0x7fb1fc06d790 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x563355d405d0 tx=0x7fb204019040 comp rx=0 tx=0).stop 2024-04-02T13:22:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.681+0000 7fb21c83cf40 1 -- 172.21.15.44:0/164768627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563355c8c440 msgr2=0x563355d60000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.681+0000 7fb21c83cf40 1 --2- 172.21.15.44:0/164768627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563355c8c440 0x563355d60000 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00a4a0 tx=0x7fb20c00a960 comp rx=0 tx=0).stop 2024-04-02T13:22:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.682+0000 7fb21a39b640 1 -- 172.21.15.44:0/164768627 reap_dead start 2024-04-02T13:22:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.682+0000 7fb21c83cf40 1 -- 172.21.15.44:0/164768627 shutdown_connections 2024-04-02T13:22:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.682+0000 7fb21c83cf40 1 -- 172.21.15.44:0/164768627 >> 172.21.15.44:0/164768627 conn(0x563355cb7c80 msgr2=0x563355d48a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.682+0000 7fb21c83cf40 1 -- 172.21.15.44:0/164768627 shutdown_connections 2024-04-02T13:22:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.682+0000 7fb21c83cf40 1 -- 172.21.15.44:0/164768627 wait complete. 2024-04-02T13:22:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:mytag 2024-04-02T13:22:06.689 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2024-04-02T13:22:06.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.770+0000 7f5a11835f40 1 Processor -- start 2024-04-02T13:22:06.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.770+0000 7f5a11835f40 1 -- start start 2024-04-02T13:22:06.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a11835f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59fc003630 0x55b45156d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a11835f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b45156d820 0x55b451573670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a11835f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b451573bb0 0x55b45149ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a11835f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b4514e3ce0 con 0x55b45156d820 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a11835f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b4514433e0 con 0x55b451573bb0 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a11835f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55b45149d200 con 0x7f59fc003630 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0e29c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b45156d820 0x55b451573670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0e29c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b45156d820 0x55b451573670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42870/0 (socket says 172.21.15.44:42870) 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0e29c640 1 -- 172.21.15.44:0/869217570 learned_addr learned my addr 172.21.15.44:0/869217570 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0f29e640 1 --2- 172.21.15.44:0/869217570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b451573bb0 0x55b45149ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0ea9d640 1 --2- 172.21.15.44:0/869217570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59fc003630 0x55b45156d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0e29c640 1 -- 172.21.15.44:0/869217570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59fc003630 msgr2=0x55b45156d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0e29c640 1 --2- 172.21.15.44:0/869217570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59fc003630 0x55b45156d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0e29c640 1 -- 172.21.15.44:0/869217570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b451573bb0 msgr2=0x55b45149ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0e29c640 1 --2- 172.21.15.44:0/869217570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b451573bb0 0x55b45149ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0e29c640 1 -- 172.21.15.44:0/869217570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4514e20f0 con 0x55b45156d820 2024-04-02T13:22:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0f29e640 1 --2- 172.21.15.44:0/869217570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b451573bb0 0x55b45149ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0ea9d640 1 --2- 172.21.15.44:0/869217570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59fc003630 0x55b45156d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0e29c640 1 --2- 172.21.15.44:0/869217570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b45156d820 0x55b451573670 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f59f80097f0 tx=0x7f59f8005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b662fb69df46b2ea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.771+0000 7f5a0d29a640 1 -- 172.21.15.44:0/869217570 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f80086d0 con 0x55b45156d820 2024-04-02T13:22:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.772+0000 7f5a0d29a640 1 -- 172.21.15.44:0/869217570 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f8008870 con 0x55b45156d820 2024-04-02T13:22:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.772+0000 7f5a0d29a640 1 -- 172.21.15.44:0/869217570 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8008c60 con 0x55b45156d820 2024-04-02T13:22:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.772+0000 7f5a11835f40 1 -- 172.21.15.44:0/869217570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b45156d820 msgr2=0x55b451573670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.772+0000 7f5a11835f40 1 --2- 172.21.15.44:0/869217570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b45156d820 0x55b451573670 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f59f80097f0 tx=0x7f59f8005a00 comp rx=0 tx=0).stop 2024-04-02T13:22:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.772+0000 7f5a11835f40 1 -- 172.21.15.44:0/869217570 shutdown_connections 2024-04-02T13:22:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.772+0000 7f5a11835f40 1 --2- 172.21.15.44:0/869217570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b451573bb0 0x55b45149ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.772+0000 7f5a11835f40 1 --2- 172.21.15.44:0/869217570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b45156d820 0x55b451573670 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.772+0000 7f5a11835f40 1 --2- 172.21.15.44:0/869217570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59fc003630 0x55b45156d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.772+0000 7f5a11835f40 1 -- 172.21.15.44:0/869217570 >> 172.21.15.44:0/869217570 conn(0x55b45149c1c0 msgr2=0x55b45149c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.772+0000 7f5a11835f40 1 -- 172.21.15.44:0/869217570 shutdown_connections 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.772+0000 7f5a11835f40 1 -- 172.21.15.44:0/869217570 wait complete. 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.773+0000 7f5a11835f40 1 Processor -- start 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.773+0000 7f5a11835f40 1 -- start start 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.773+0000 7f5a11835f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b45156d820 0x55b45156caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.773+0000 7f5a11835f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b451573bb0 0x55b45155b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.773+0000 7f5a11835f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59fc003630 0x55b45160a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.773+0000 7f5a11835f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b45156dcd0 con 0x7f59fc003630 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.773+0000 7f5a11835f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b45156fb10 con 0x55b451573bb0 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.773+0000 7f5a11835f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55b45149e060 con 0x55b45156d820 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.773+0000 7f5a0f29e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59fc003630 0x55b45160a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.773+0000 7f5a0f29e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59fc003630 0x55b45160a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42876/0 (socket says 172.21.15.44:42876) 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.773+0000 7f5a0f29e640 1 -- 172.21.15.44:0/2704556450 learned_addr learned my addr 172.21.15.44:0/2704556450 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a0ea9d640 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b45156d820 0x55b45156caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a0e29c640 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b451573bb0 0x55b45155b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a0f29e640 1 -- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b45156d820 msgr2=0x55b45156caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a0f29e640 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b45156d820 0x55b45156caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a0f29e640 1 -- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b451573bb0 msgr2=0x55b45155b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a0f29e640 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b451573bb0 0x55b45155b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a0f29e640 1 -- 172.21.15.44:0/2704556450 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4514dfd70 con 0x7f59fc003630 2024-04-02T13:22:06.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a0ea9d640 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b45156d820 0x55b45156caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:06.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a0f29e640 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59fc003630 0x55b45160a760 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5a0001d050 tx=0x7f5a00005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a07fff640 1 -- 172.21.15.44:0/2704556450 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0000a3b0 con 0x7f59fc003630 2024-04-02T13:22:06.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a07fff640 1 -- 172.21.15.44:0/2704556450 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a00004820 con 0x7f59fc003630 2024-04-02T13:22:06.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a11835f40 1 -- 172.21.15.44:0/2704556450 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b4514e0050 con 0x7f59fc003630 2024-04-02T13:22:06.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a07fff640 1 -- 172.21.15.44:0/2704556450 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00005a80 con 0x7f59fc003630 2024-04-02T13:22:06.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.774+0000 7f5a11835f40 1 -- 172.21.15.44:0/2704556450 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4514e0610 con 0x7f59fc003630 2024-04-02T13:22:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.776+0000 7f5a07fff640 1 -- 172.21.15.44:0/2704556450 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5a00004c10 con 0x7f59fc003630 2024-04-02T13:22:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.776+0000 7f5a07fff640 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59f006b2d0 0x7f59f006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.776+0000 7f5a07fff640 1 -- 172.21.15.44:0/2704556450 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f5a000cd3f0 con 0x7f59fc003630 2024-04-02T13:22:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.776+0000 7f5a0ea9d640 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59f006b2d0 0x7f59f006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.776+0000 7f5a11835f40 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55b451572e70 0x55b4514c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.776+0000 7f5a11835f40 1 -- 172.21.15.44:0/2704556450 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:1 6.19 6:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55b4514e2700 con 0x55b451572e70 2024-04-02T13:22:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.776+0000 7f5a0e29c640 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55b451572e70 0x55b4514c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a0ea9d640 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59f006b2d0 0x7f59f006d790 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5a080017e0 tx=0x7f5a08001df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a0e29c640 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55b451572e70 0x55b4514c7970 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a0e29c640 1 -- 172.21.15.44:0/2704556450 <== osd.5 v2:172.21.15.69:6800/2841767499 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (9) out=9b] v0'0 uv12 ondisk = 0) ==== 164+0+9 (crc 0 0 0) 0x7f59f8008b20 con 0x55b451572e70 2024-04-02T13:22:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a11835f40 1 -- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55b451572e70 msgr2=0x55b4514c7970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a11835f40 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55b451572e70 0x55b4514c7970 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a11835f40 1 -- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59f006b2d0 msgr2=0x7f59f006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a11835f40 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59f006b2d0 0x7f59f006d790 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5a080017e0 tx=0x7f5a08001df0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a11835f40 1 -- 172.21.15.44:0/2704556450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59fc003630 msgr2=0x55b45160a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a11835f40 1 --2- 172.21.15.44:0/2704556450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59fc003630 0x55b45160a760 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5a0001d050 tx=0x7f5a00005d80 comp rx=0 tx=0).stop 2024-04-02T13:22:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a0f29e640 1 -- 172.21.15.44:0/2704556450 reap_dead start 2024-04-02T13:22:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a11835f40 1 -- 172.21.15.44:0/2704556450 shutdown_connections 2024-04-02T13:22:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a11835f40 1 -- 172.21.15.44:0/2704556450 >> 172.21.15.44:0/2704556450 conn(0x55b45149c1c0 msgr2=0x55b45149c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a11835f40 1 -- 172.21.15.44:0/2704556450 shutdown_connections 2024-04-02T13:22:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.777+0000 7f5a11835f40 1 -- 172.21.15.44:0/2704556450 wait complete. 2024-04-02T13:22:06.785 INFO:teuthology.orchestra.run.smithi044.stdout:mytag 2024-04-02T13:22:06.785 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2024-04-02T13:22:06.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.866+0000 7f44dfc37f40 1 Processor -- start 2024-04-02T13:22:06.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.866+0000 7f44dfc37f40 1 -- start start 2024-04-02T13:22:06.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dfc37f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3d28c1210 0x55b3d28ab530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dfc37f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3d28a9b80 0x55b3d28ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dfc37f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3d28cac80 0x55b3d27f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dfc37f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b3d279a3e0 con 0x55b3d28cac80 2024-04-02T13:22:06.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dfc37f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b3d279e430 con 0x55b3d28a9b80 2024-04-02T13:22:06.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dfc37f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55b3d28227a0 con 0x55b3d28c1210 2024-04-02T13:22:06.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dd77b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3d28cac80 0x55b3d27f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dcf7a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3d28c1210 0x55b3d28ab530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dd77b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3d28cac80 0x55b3d27f15f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42892/0 (socket says 172.21.15.44:42892) 2024-04-02T13:22:06.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dd77b640 1 -- 172.21.15.44:0/1050969402 learned_addr learned my addr 172.21.15.44:0/1050969402 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:06.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dcf7a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3d28c1210 0x55b3d28ab530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:39812/0 (socket says 172.21.15.44:39812) 2024-04-02T13:22:06.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44d7fff640 1 --2- 172.21.15.44:0/1050969402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3d28a9b80 0x55b3d28ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dd77b640 1 -- 172.21.15.44:0/1050969402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3d28c1210 msgr2=0x55b3d28ab530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:06.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dd77b640 1 --2- 172.21.15.44:0/1050969402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3d28c1210 0x55b3d28ab530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dd77b640 1 -- 172.21.15.44:0/1050969402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3d28a9b80 msgr2=0x55b3d28ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dd77b640 1 --2- 172.21.15.44:0/1050969402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3d28a9b80 0x55b3d28ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dd77b640 1 -- 172.21.15.44:0/1050969402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3d28c9e30 con 0x55b3d28cac80 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.867+0000 7f44dd77b640 1 --2- 172.21.15.44:0/1050969402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3d28cac80 0x55b3d27f15f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f44d8057040 tx=0x7f44d8003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1baf82a6d0cf3d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.868+0000 7f44d6ffd640 1 -- 172.21.15.44:0/1050969402 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d8059070 con 0x55b3d28cac80 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.868+0000 7f44d6ffd640 1 -- 172.21.15.44:0/1050969402 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44d80545e0 con 0x55b3d28cac80 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.868+0000 7f44d6ffd640 1 -- 172.21.15.44:0/1050969402 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d800a3b0 con 0x55b3d28cac80 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.868+0000 7f44dfc37f40 1 -- 172.21.15.44:0/1050969402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3d28cac80 msgr2=0x55b3d27f15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.868+0000 7f44dfc37f40 1 --2- 172.21.15.44:0/1050969402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3d28cac80 0x55b3d27f15f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f44d8057040 tx=0x7f44d8003330 comp rx=0 tx=0).stop 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.868+0000 7f44dfc37f40 1 -- 172.21.15.44:0/1050969402 shutdown_connections 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.868+0000 7f44dfc37f40 1 --2- 172.21.15.44:0/1050969402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3d28cac80 0x55b3d27f15f0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.868+0000 7f44dfc37f40 1 --2- 172.21.15.44:0/1050969402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3d28a9b80 0x55b3d28ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.868+0000 7f44dfc37f40 1 --2- 172.21.15.44:0/1050969402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3d28c1210 0x55b3d28ab530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.868+0000 7f44dfc37f40 1 -- 172.21.15.44:0/1050969402 >> 172.21.15.44:0/1050969402 conn(0x55b3d281fc80 msgr2=0x55b3d28c9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.868+0000 7f44dfc37f40 1 -- 172.21.15.44:0/1050969402 shutdown_connections 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.868+0000 7f44dfc37f40 1 -- 172.21.15.44:0/1050969402 wait complete. 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.869+0000 7f44dfc37f40 1 Processor -- start 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.869+0000 7f44dfc37f40 1 -- start start 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44dfc37f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3d28a9b80 0x55b3d28aab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44dfc37f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3d28c1210 0x55b3d28bd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44dfc37f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3d28cac80 0x55b3d2958450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44dfc37f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b3d28c15f0 con 0x55b3d28c1210 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44dfc37f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b3d281cce0 con 0x55b3d28a9b80 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44dfc37f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55b3d28c1770 con 0x55b3d28cac80 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44d7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3d28c1210 0x55b3d28bd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44d7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3d28c1210 0x55b3d28bd950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42904/0 (socket says 172.21.15.44:42904) 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44d7fff640 1 -- 172.21.15.44:0/3849283295 learned_addr learned my addr 172.21.15.44:0/3849283295 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:06.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44dcf7a640 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3d28a9b80 0x55b3d28aab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44dd77b640 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3d28cac80 0x55b3d2958450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44d7fff640 1 -- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3d28cac80 msgr2=0x55b3d2958450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44d7fff640 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3d28cac80 0x55b3d2958450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44d7fff640 1 -- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3d28a9b80 msgr2=0x55b3d28aab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44d7fff640 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3d28a9b80 0x55b3d28aab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44d7fff640 1 -- 172.21.15.44:0/3849283295 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3d28ab1b0 con 0x55b3d28c1210 2024-04-02T13:22:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44d7fff640 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3d28c1210 0x55b3d28bd950 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f44c8002eb0 tx=0x7f44c800f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44d5ffb640 1 -- 172.21.15.44:0/3849283295 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8012640 con 0x55b3d28c1210 2024-04-02T13:22:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44d5ffb640 1 -- 172.21.15.44:0/3849283295 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44c8016030 con 0x55b3d28c1210 2024-04-02T13:22:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44dfc37f40 1 -- 172.21.15.44:0/3849283295 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b3d28ab490 con 0x55b3d28c1210 2024-04-02T13:22:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44d5ffb640 1 -- 172.21.15.44:0/3849283295 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c80129e0 con 0x55b3d28c1210 2024-04-02T13:22:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.870+0000 7f44dfc37f40 1 -- 172.21.15.44:0/3849283295 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3d28aba50 con 0x55b3d28c1210 2024-04-02T13:22:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:22:06.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.872+0000 7f44d5ffb640 1 -- 172.21.15.44:0/3849283295 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f44c800e050 con 0x55b3d28c1210 2024-04-02T13:22:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.872+0000 7f44d5ffb640 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f44c006b2d0 0x7f44c006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.872+0000 7f44d5ffb640 1 -- 172.21.15.44:0/3849283295 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f44c808acd0 con 0x55b3d28c1210 2024-04-02T13:22:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.872+0000 7f44dcf7a640 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f44c006b2d0 0x7f44c006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.872+0000 7f44dfc37f40 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55b3d28a7320 0x55b3d28a7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.873+0000 7f44dfc37f40 1 -- 172.21.15.44:0/3849283295 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:1 6.12 6:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55b3d28a7c90 con 0x55b3d28a7320 2024-04-02T13:22:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.873+0000 7f44dd77b640 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55b3d28a7320 0x55b3d28a7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.873+0000 7f44dcf7a640 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f44c006b2d0 0x7f44c006d790 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55b3d28cab30 tx=0x7f44d0012040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.873+0000 7f44dd77b640 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55b3d28a7320 0x55b3d28a7750 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.873+0000 7f44dd77b640 1 -- 172.21.15.44:0/3849283295 <== osd.7 v2:172.21.15.69:6824/1292475712 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) ==== 164+0+0 (crc 0 0 0) 0x7f44d8050040 con 0x55b3d28a7320 2024-04-02T13:22:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:error getting xattr cephfs_data/10000000002.00000000/scrub_tag: (61) No data available 2024-04-02T13:22:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.873+0000 7f44dfc37f40 1 -- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55b3d28a7320 msgr2=0x55b3d28a7750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.873+0000 7f44dfc37f40 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55b3d28a7320 0x55b3d28a7750 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.873+0000 7f44dfc37f40 1 -- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f44c006b2d0 msgr2=0x7f44c006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.873+0000 7f44dfc37f40 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f44c006b2d0 0x7f44c006d790 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55b3d28cab30 tx=0x7f44d0012040 comp rx=0 tx=0).stop 2024-04-02T13:22:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.874+0000 7f44dfc37f40 1 -- 172.21.15.44:0/3849283295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3d28c1210 msgr2=0x55b3d28bd950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.874+0000 7f44dfc37f40 1 --2- 172.21.15.44:0/3849283295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3d28c1210 0x55b3d28bd950 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f44c8002eb0 tx=0x7f44c800f710 comp rx=0 tx=0).stop 2024-04-02T13:22:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.874+0000 7f44dd77b640 1 -- 172.21.15.44:0/3849283295 reap_dead start 2024-04-02T13:22:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.874+0000 7f44dfc37f40 1 -- 172.21.15.44:0/3849283295 shutdown_connections 2024-04-02T13:22:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.874+0000 7f44dfc37f40 1 -- 172.21.15.44:0/3849283295 >> 172.21.15.44:0/3849283295 conn(0x55b3d281fc80 msgr2=0x55b3d27f4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.874+0000 7f44dfc37f40 1 -- 172.21.15.44:0/3849283295 shutdown_connections 2024-04-02T13:22:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:06.874+0000 7f44dfc37f40 1 -- 172.21.15.44:0/3849283295 wait complete. 2024-04-02T13:22:06.882 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:22:06.882 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:22:06.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:22:07.004 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:22:07.005 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:22:07.005 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:22:07.005 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:22:07.024 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:22:07.025 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:22:07.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:07.025 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:22:07.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:22:07.080 INFO:teuthology.orchestra.run.smithi044.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-02T13:22:07.080 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:22:07.080 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:22:07.080 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:22:07.098 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:22:07.098 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:22:07.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:07.098 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-02T13:22:07.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:22:07.155 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-02T13:22:07.155 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:22:07.155 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:22:07.155 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:22:07.171 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:22:07.171 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:22:07.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:07.171 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-02T13:22:07.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:22:07.225 INFO:teuthology.orchestra.run.smithi044.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-02T13:22:07.225 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:22:07.226 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:22:07.226 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:22:07.243 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:22:07.243 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:22:07.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:07.243 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-02T13:22:07.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:22:07.299 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-02T13:22:07.299 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:22:07.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.399+0000 7f92ce38f640 1 Processor -- start 2024-04-02T13:22:07.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.399+0000 7f92ce38f640 1 -- start start 2024-04-02T13:22:07.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.400+0000 7f92ce38f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f92c806d700 0x7f92c815e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:07.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.400+0000 7f92ce38f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f92c815ee70 0x7f92c80d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:07.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.400+0000 7f92ce38f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92c815fb50 con 0x7f92c815e550 2024-04-02T13:22:07.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.400+0000 7f92ce38f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f92c806ca90 con 0x7f92c806d700 2024-04-02T13:22:07.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.400+0000 7f92ce38f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f92c815f590 con 0x7f92c815ee70 2024-04-02T13:22:07.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.400+0000 7f92c77fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f92c806d700 0x7f92c815e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:07.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.400+0000 7f92c77fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f92c806d700 0x7f92c815e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42906/0 (socket says 172.21.15.44:42906) 2024-04-02T13:22:07.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.400+0000 7f92c77fe640 1 -- 172.21.15.44:0/1492566503 learned_addr learned my addr 172.21.15.44:0/1492566503 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:07.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.400+0000 7f92cc92c640 1 --2- 172.21.15.44:0/1492566503 >> v2:172.21.15.69:3300/0 conn(0x7f92c815ee70 0x7f92c80d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:07.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.400+0000 7f92c77fe640 1 -- 172.21.15.44:0/1492566503 >> v2:172.21.15.69:3300/0 conn(0x7f92c815ee70 msgr2=0x7f92c80d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:07.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.400+0000 7f92c77fe640 1 --2- 172.21.15.44:0/1492566503 >> v2:172.21.15.69:3300/0 conn(0x7f92c815ee70 0x7f92c80d0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.400+0000 7f92c77fe640 1 -- 172.21.15.44:0/1492566503 >> v1:172.21.15.69:6789/0 conn(0x7f92c815e550 legacy=0x7f92c806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:07.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.400+0000 7f92c77fe640 1 -- 172.21.15.44:0/1492566503 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c815f320 con 0x7f92c806d700 2024-04-02T13:22:07.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.401+0000 7f92c77fe640 1 --2- 172.21.15.44:0/1492566503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92c806d700 0x7f92c815e930 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f92c809b870 tx=0x7f92b802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46c5cdb3ada4246 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:07.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.401+0000 7f92c6ffd640 1 -- 172.21.15.44:0/1492566503 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b8034070 con 0x7f92c806d700 2024-04-02T13:22:07.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.401+0000 7f92c6ffd640 1 -- 172.21.15.44:0/1492566503 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92b8030860 con 0x7f92c806d700 2024-04-02T13:22:07.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.401+0000 7f92c6ffd640 1 -- 172.21.15.44:0/1492566503 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b8030c30 con 0x7f92c806d700 2024-04-02T13:22:07.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.402+0000 7f92ce38f640 1 -- 172.21.15.44:0/1492566503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92c806d700 msgr2=0x7f92c815e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:07.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.402+0000 7f92ce38f640 1 --2- 172.21.15.44:0/1492566503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92c806d700 0x7f92c815e930 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f92c809b870 tx=0x7f92b802b9a0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.402+0000 7f92ce38f640 1 -- 172.21.15.44:0/1492566503 shutdown_connections 2024-04-02T13:22:07.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.402+0000 7f92ce38f640 1 --2- 172.21.15.44:0/1492566503 >> v2:172.21.15.69:3300/0 conn(0x7f92c815ee70 0x7f92c80d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.402+0000 7f92ce38f640 1 --2- 172.21.15.44:0/1492566503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92c806d700 0x7f92c815e930 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.402+0000 7f92ce38f640 1 -- 172.21.15.44:0/1492566503 >> 172.21.15.44:0/1492566503 conn(0x7f92c809aa20 msgr2=0x7f92c809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:07.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.402+0000 7f92ce38f640 1 -- 172.21.15.44:0/1492566503 shutdown_connections 2024-04-02T13:22:07.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.402+0000 7f92ce38f640 1 -- 172.21.15.44:0/1492566503 wait complete. 2024-04-02T13:22:07.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.403+0000 7f92ce38f640 1 Processor -- start 2024-04-02T13:22:07.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.403+0000 7f92ce38f640 1 -- start start 2024-04-02T13:22:07.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92ce38f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c806d700 0x7f92c815b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:07.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92ce38f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92c815e550 0x7f92c80ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:07.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92ce38f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c815ee70 0x7f92c80c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:07.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92ce38f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f92c80d0590 con 0x7f92c815e550 2024-04-02T13:22:07.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92ce38f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f92c80d02b0 con 0x7f92c815ee70 2024-04-02T13:22:07.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92ce38f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f92c80cc4b0 con 0x7f92c806d700 2024-04-02T13:22:07.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92c77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92c815e550 0x7f92c80ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:07.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92c77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92c815e550 0x7f92c80ca880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42918/0 (socket says 172.21.15.44:42918) 2024-04-02T13:22:07.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92c77fe640 1 -- 172.21.15.44:0/4001185299 learned_addr learned my addr 172.21.15.44:0/4001185299 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:07.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92cc92c640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c815ee70 0x7f92c80c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:07.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92c7fff640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c806d700 0x7f92c815b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:07.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92c77fe640 1 -- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c806d700 msgr2=0x7f92c815b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:07.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92c77fe640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c806d700 0x7f92c815b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92c77fe640 1 -- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c815ee70 msgr2=0x7f92c80c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:07.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92c77fe640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c815ee70 0x7f92c80c79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92c77fe640 1 -- 172.21.15.44:0/4001185299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c815bad0 con 0x7f92c815e550 2024-04-02T13:22:07.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.404+0000 7f92c7fff640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c806d700 0x7f92c815b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:07.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.405+0000 7f92c77fe640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92c815e550 0x7f92c80ca880 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f92b8000c00 tx=0x7f92b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:07.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.405+0000 7f92c4ff9640 1 -- 172.21.15.44:0/4001185299 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b8034040 con 0x7f92c815e550 2024-04-02T13:22:07.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.405+0000 7f92c4ff9640 1 -- 172.21.15.44:0/4001185299 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92b80433f0 con 0x7f92c815e550 2024-04-02T13:22:07.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.405+0000 7f92ce38f640 1 -- 172.21.15.44:0/4001185299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92c80cb2d0 con 0x7f92c815e550 2024-04-02T13:22:07.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.405+0000 7f92c4ff9640 1 -- 172.21.15.44:0/4001185299 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b80437e0 con 0x7f92c815e550 2024-04-02T13:22:07.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.405+0000 7f92ce38f640 1 -- 172.21.15.44:0/4001185299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92c80cb890 con 0x7f92c815e550 2024-04-02T13:22:07.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.406+0000 7f92ce38f640 1 -- 172.21.15.44:0/4001185299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92c80684f0 con 0x7f92c815e550 2024-04-02T13:22:07.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.406+0000 7f92c4ff9640 1 -- 172.21.15.44:0/4001185299 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f92b804a060 con 0x7f92c815e550 2024-04-02T13:22:07.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.407+0000 7f92c4ff9640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f92a806c8f0 0x7f92a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:07.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.407+0000 7f92c4ff9640 1 -- 172.21.15.44:0/4001185299 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f92b80adfe0 con 0x7f92c815e550 2024-04-02T13:22:07.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.407+0000 7f92c7fff640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f92a806c8f0 0x7f92a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:07.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.407+0000 7f92c7fff640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f92a806c8f0 0x7f92a806edb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f92b0006fd0 tx=0x7f92b0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:07.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.411+0000 7f92c4ff9640 1 -- 172.21.15.44:0/4001185299 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f92c815bad0 con 0x7f92c815e550 2024-04-02T13:22:07.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.632+0000 7f92ce38f640 1 -- 172.21.15.44:0/4001185299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92c80cd600 con 0x7f92c815e550 2024-04-02T13:22:07.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.633+0000 7f92c4ff9640 1 -- 172.21.15.44:0/4001185299 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5181 (secure 0 0 0) 0x7f92b803e070 con 0x7f92c815e550 2024-04-02T13:22:07.635 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:07.635 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3050514527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3050514527},{"type":"v1","addr":"172.21.15.69:6836","nonce":3050514527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4973,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1949024012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1949024012},{"type":"v1","addr":"172.21.15.69:6837","nonce":1949024012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/2817873198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2817873198},{"type":"v1","addr":"172.21.15.44:6834","nonce":2817873198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:21:43.215796+0000","modified":"2024-04-02T13:21:46.248294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/1969465285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":1969465285},{"type":"v1","addr":"172.21.15.44:6835","nonce":1969465285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":2}]} 2024-04-02T13:22:07.635 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 18 2024-04-02T13:22:07.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.635+0000 7f92ce38f640 1 -- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f92a806c8f0 msgr2=0x7f92a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:07.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.635+0000 7f92ce38f640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f92a806c8f0 0x7f92a806edb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f92b0006fd0 tx=0x7f92b0008040 comp rx=0 tx=0).stop 2024-04-02T13:22:07.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.635+0000 7f92ce38f640 1 -- 172.21.15.44:0/4001185299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92c815e550 msgr2=0x7f92c80ca880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:07.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.635+0000 7f92ce38f640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92c815e550 0x7f92c80ca880 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f92b8000c00 tx=0x7f92b80046c0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.635+0000 7f92ce38f640 1 -- 172.21.15.44:0/4001185299 shutdown_connections 2024-04-02T13:22:07.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.635+0000 7f92ce38f640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f92a806c8f0 0x7f92a806edb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.635+0000 7f92ce38f640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c815ee70 0x7f92c80c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.635+0000 7f92ce38f640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92c815e550 0x7f92c80ca880 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.635+0000 7f92ce38f640 1 --2- 172.21.15.44:0/4001185299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c806d700 0x7f92c815b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.635+0000 7f92ce38f640 1 -- 172.21.15.44:0/4001185299 >> 172.21.15.44:0/4001185299 conn(0x7f92c809aa20 msgr2=0x7f92c80d14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:07.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.635+0000 7f92ce38f640 1 -- 172.21.15.44:0/4001185299 shutdown_connections 2024-04-02T13:22:07.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.635+0000 7f92ce38f640 1 -- 172.21.15.44:0/4001185299 wait complete. 2024-04-02T13:22:07.647 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:22:07.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.747+0000 7f43402f2640 1 Processor -- start 2024-04-02T13:22:07.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.747+0000 7f43402f2640 1 -- start start 2024-04-02T13:22:07.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.748+0000 7f43402f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f433806d700 0x7f433815e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:07.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.748+0000 7f43402f2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f433815fb50 con 0x7f433815ee70 2024-04-02T13:22:07.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.748+0000 7f43402f2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f433815f320 con 0x7f433815e550 2024-04-02T13:22:07.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.748+0000 7f43402f2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f433806ca90 con 0x7f433806d700 2024-04-02T13:22:07.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.748+0000 7f433d88d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f433806d700 0x7f433815e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:07.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.748+0000 7f433e88f640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f433815ee70 0x7f43380d0ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:47152/0 (socket says 172.21.15.44:47152) 2024-04-02T13:22:07.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.748+0000 7f433e88f640 1 -- 172.21.15.44:0/1619259353 learned_addr learned my addr 172.21.15.44:0/1619259353 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:07.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.749+0000 7f433d88d640 1 -- 172.21.15.44:0/1619259353 >> v1:172.21.15.69:6789/0 conn(0x7f433815e550 legacy=0x7f433806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:07.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.749+0000 7f433d88d640 1 -- 172.21.15.44:0/1619259353 >> v1:172.21.15.44:6789/0 conn(0x7f433815ee70 legacy=0x7f43380d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:07.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.749+0000 7f433d88d640 1 -- 172.21.15.44:0/1619259353 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f433815f590 con 0x7f433806d700 2024-04-02T13:22:07.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.749+0000 7f433d08c640 1 -- 172.21.15.44:0/1619259353 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3797104432 0 0) 0x7f433815fb50 con 0x7f433815ee70 2024-04-02T13:22:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.749+0000 7f433d88d640 1 --2- 172.21.15.44:0/1619259353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f433806d700 0x7f433815e930 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f43280097a0 tx=0x7f432802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17f4014cc96bcdd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.749+0000 7f433d08c640 1 -- 172.21.15.44:0/1619259353 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4328034070 con 0x7f433806d700 2024-04-02T13:22:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.750+0000 7f433d08c640 1 -- 172.21.15.44:0/1619259353 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4328030610 con 0x7f433806d700 2024-04-02T13:22:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.750+0000 7f433d08c640 1 -- 172.21.15.44:0/1619259353 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43280309e0 con 0x7f433806d700 2024-04-02T13:22:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.750+0000 7f43402f2640 1 -- 172.21.15.44:0/1619259353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f433806d700 msgr2=0x7f433815e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.750+0000 7f43402f2640 1 --2- 172.21.15.44:0/1619259353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f433806d700 0x7f433815e930 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f43280097a0 tx=0x7f432802b960 comp rx=0 tx=0).stop 2024-04-02T13:22:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.750+0000 7f43402f2640 1 -- 172.21.15.44:0/1619259353 shutdown_connections 2024-04-02T13:22:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.750+0000 7f43402f2640 1 --2- 172.21.15.44:0/1619259353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f433806d700 0x7f433815e930 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.750+0000 7f43402f2640 1 -- 172.21.15.44:0/1619259353 >> 172.21.15.44:0/1619259353 conn(0x7f433809aa20 msgr2=0x7f433809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.750+0000 7f43402f2640 1 -- 172.21.15.44:0/1619259353 shutdown_connections 2024-04-02T13:22:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.750+0000 7f43402f2640 1 -- 172.21.15.44:0/1619259353 wait complete. 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.751+0000 7f43402f2640 1 Processor -- start 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.751+0000 7f43402f2640 1 -- start start 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f43402f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f433806d700 0x7f4338158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f43402f2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f433815e550 0x7f43380cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f43402f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f433815ee70 0x7f433806fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f43402f2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f43380d0590 con 0x7f433815ee70 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f43402f2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f43380d02b0 con 0x7f433815e550 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f43402f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f43380cc4b0 con 0x7f433806d700 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f433e88f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f433815ee70 0x7f433806fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f433e88f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f433815ee70 0x7f433806fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42936/0 (socket says 172.21.15.44:42936) 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f433e88f640 1 -- 172.21.15.44:0/1828901859 learned_addr learned my addr 172.21.15.44:0/1828901859 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f433e08e640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f433806d700 0x7f4338158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f433d88d640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f433815e550 0x7f43380cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:07.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f433e88f640 1 -- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f433806d700 msgr2=0x7f4338158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:07.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f433e88f640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f433806d700 0x7f4338158710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f433e88f640 1 -- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f433815e550 msgr2=0x7f43380cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:07.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.752+0000 7f433e88f640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f433815e550 0x7f43380cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.753+0000 7f433e88f640 1 -- 172.21.15.44:0/1828901859 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4338158c50 con 0x7f433815ee70 2024-04-02T13:22:07.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.753+0000 7f433e88f640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f433815ee70 0x7f433806fe60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f433400c780 tx=0x7f433400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:07.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.753+0000 7f432effd640 1 -- 172.21.15.44:0/1828901859 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334018070 con 0x7f433815ee70 2024-04-02T13:22:07.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.753+0000 7f432effd640 1 -- 172.21.15.44:0/1828901859 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f433400a640 con 0x7f433815ee70 2024-04-02T13:22:07.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.753+0000 7f432effd640 1 -- 172.21.15.44:0/1828901859 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433400aa30 con 0x7f433815ee70 2024-04-02T13:22:07.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.753+0000 7f43402f2640 1 -- 172.21.15.44:0/1828901859 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43380ca2d0 con 0x7f433815ee70 2024-04-02T13:22:07.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.753+0000 7f43402f2640 1 -- 172.21.15.44:0/1828901859 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43380ca890 con 0x7f433815ee70 2024-04-02T13:22:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.754+0000 7f43402f2640 1 -- 172.21.15.44:0/1828901859 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43380684f0 con 0x7f433815ee70 2024-04-02T13:22:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.754+0000 7f432effd640 1 -- 172.21.15.44:0/1828901859 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f433401b070 con 0x7f433815ee70 2024-04-02T13:22:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.755+0000 7f432effd640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f431c06c8f0 0x7f431c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.755+0000 7f432effd640 1 -- 172.21.15.44:0/1828901859 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f433408b210 con 0x7f433815ee70 2024-04-02T13:22:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.755+0000 7f433e08e640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f431c06c8f0 0x7f431c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.755+0000 7f433e08e640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f431c06c8f0 0x7f431c06edb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4330004ea0 tx=0x7f4330009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:07.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.759+0000 7f432effd640 1 -- 172.21.15.44:0/1828901859 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4334059540 con 0x7f433815ee70 2024-04-02T13:22:07.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.950+0000 7f43402f2640 1 -- 172.21.15.44:0/1828901859 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43380b3390 con 0x7f433815ee70 2024-04-02T13:22:07.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.951+0000 7f432effd640 1 -- 172.21.15.44:0/1828901859 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) ==== 74+0+16943 (secure 0 0 0) 0x7f4334058e40 con 0x7f433815ee70 2024-04-02T13:22:07.953 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:07.954 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":40,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:22:07.936506+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:21:41.647497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-02T13:21:42.168299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:22:07.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.954+0000 7f43402f2640 1 -- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f431c06c8f0 msgr2=0x7f431c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:07.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.954+0000 7f43402f2640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f431c06c8f0 0x7f431c06edb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4330004ea0 tx=0x7f4330009040 comp rx=0 tx=0).stop 2024-04-02T13:22:07.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.954+0000 7f43402f2640 1 -- 172.21.15.44:0/1828901859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f433815ee70 msgr2=0x7f433806fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:07.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.954+0000 7f43402f2640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f433815ee70 0x7f433806fe60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f433400c780 tx=0x7f433400cc40 comp rx=0 tx=0).stop 2024-04-02T13:22:07.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.954+0000 7f43402f2640 1 -- 172.21.15.44:0/1828901859 shutdown_connections 2024-04-02T13:22:07.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.954+0000 7f43402f2640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f431c06c8f0 0x7f431c06edb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.954+0000 7f43402f2640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f433815ee70 0x7f433806fe60 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.954+0000 7f43402f2640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f433815e550 0x7f43380cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.954+0000 7f43402f2640 1 --2- 172.21.15.44:0/1828901859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f433806d700 0x7f4338158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:07.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.954+0000 7f43402f2640 1 -- 172.21.15.44:0/1828901859 >> 172.21.15.44:0/1828901859 conn(0x7f433809aa20 msgr2=0x7f433806cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:07.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.954+0000 7f43402f2640 1 -- 172.21.15.44:0/1828901859 shutdown_connections 2024-04-02T13:22:07.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:07.955+0000 7f43402f2640 1 -- 172.21.15.44:0/1828901859 wait complete. 2024-04-02T13:22:07.967 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:21:41.647497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:21:42.168299+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:22:07.967 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:22:08.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.070+0000 7f7fb7e3a640 1 Processor -- start 2024-04-02T13:22:08.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.070+0000 7f7fb7e3a640 1 -- start start 2024-04-02T13:22:08.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.071+0000 7f7fb7e3a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7fb015e420 0x7f7fb00ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.071+0000 7f7fb7e3a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7fb00cfe10 0x7f7fb00cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.071+0000 7f7fb7e3a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb006c220 con 0x7f7fb00721f0 2024-04-02T13:22:08.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.071+0000 7f7fb7e3a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7fb015f7e0 con 0x7f7fb00cfe10 2024-04-02T13:22:08.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.071+0000 7f7fb7e3a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7fb015ec60 con 0x7f7fb015e420 2024-04-02T13:22:08.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.071+0000 7f7fb63d7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7fb00cfe10 0x7f7fb00cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.071+0000 7f7fb63d7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7fb00cfe10 0x7f7fb00cdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42948/0 (socket says 172.21.15.44:42948) 2024-04-02T13:22:08.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.071+0000 7f7fb63d7640 1 -- 172.21.15.44:0/1562357887 learned_addr learned my addr 172.21.15.44:0/1562357887 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:08.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.071+0000 7f7fb63d7640 1 -- 172.21.15.44:0/1562357887 >> v2:172.21.15.69:3300/0 conn(0x7f7fb015e420 msgr2=0x7f7fb00ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:08.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.071+0000 7f7fb63d7640 1 --2- 172.21.15.44:0/1562357887 >> v2:172.21.15.69:3300/0 conn(0x7f7fb015e420 0x7f7fb00ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.071+0000 7f7fb63d7640 1 -- 172.21.15.44:0/1562357887 >> v1:172.21.15.69:6789/0 conn(0x7f7fb00721f0 legacy=0x7f7fb00cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:08.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.071+0000 7f7fb63d7640 1 -- 172.21.15.44:0/1562357887 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb015ef40 con 0x7f7fb00cfe10 2024-04-02T13:22:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.072+0000 7f7fb63d7640 1 --2- 172.21.15.44:0/1562357887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb00cfe10 0x7f7fb00cdae0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f7fb006dc40 tx=0x7f7fa802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e15fccfb2f68a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.072+0000 7f7fb4bd4640 1 -- 172.21.15.44:0/1562357887 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa802c960 con 0x7f7fb00cfe10 2024-04-02T13:22:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.072+0000 7f7fb4bd4640 1 -- 172.21.15.44:0/1562357887 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fa802cb00 con 0x7f7fb00cfe10 2024-04-02T13:22:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.072+0000 7f7fb4bd4640 1 -- 172.21.15.44:0/1562357887 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa80343f0 con 0x7f7fb00cfe10 2024-04-02T13:22:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.073+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/1562357887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb00cfe10 msgr2=0x7f7fb00cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.073+0000 7f7fb7e3a640 1 --2- 172.21.15.44:0/1562357887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb00cfe10 0x7f7fb00cdae0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f7fb006dc40 tx=0x7f7fa802bdc0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.073+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/1562357887 shutdown_connections 2024-04-02T13:22:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.073+0000 7f7fb7e3a640 1 --2- 172.21.15.44:0/1562357887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb00cfe10 0x7f7fb00cdae0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.073+0000 7f7fb7e3a640 1 --2- 172.21.15.44:0/1562357887 >> v2:172.21.15.69:3300/0 conn(0x7f7fb015e420 0x7f7fb00ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.073+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/1562357887 >> 172.21.15.44:0/1562357887 conn(0x7f7fb00b2a10 msgr2=0x7f7fb00b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.073+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/1562357887 shutdown_connections 2024-04-02T13:22:08.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.073+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/1562357887 wait complete. 2024-04-02T13:22:08.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.074+0000 7f7fb7e3a640 1 Processor -- start 2024-04-02T13:22:08.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.074+0000 7f7fb7e3a640 1 -- start start 2024-04-02T13:22:08.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.075+0000 7f7fb7e3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb00721f0 0x7f7fb01552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.075+0000 7f7fb7e3a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb00cfe10 0x7f7fb0151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.075+0000 7f7fb7e3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb015e420 0x7f7fb0154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.075+0000 7f7fb7e3a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb00cf1e0 con 0x7f7fb015e420 2024-04-02T13:22:08.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.075+0000 7f7fb7e3a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb015f500 con 0x7f7fb00721f0 2024-04-02T13:22:08.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.075+0000 7f7fb7e3a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb00cf360 con 0x7f7fb00cfe10 2024-04-02T13:22:08.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.075+0000 7f7fb63d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb015e420 0x7f7fb0154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.075+0000 7f7fb63d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb015e420 0x7f7fb0154a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42950/0 (socket says 172.21.15.44:42950) 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.075+0000 7f7fb63d7640 1 -- 172.21.15.44:0/74159254 learned_addr learned my addr 172.21.15.44:0/74159254 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.075+0000 7f7fb53d5640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb00cfe10 0x7f7fb0151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.075+0000 7f7fb5bd6640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb00721f0 0x7f7fb01552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.076+0000 7f7fb63d7640 1 -- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb00cfe10 msgr2=0x7f7fb0151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.076+0000 7f7fb63d7640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb00cfe10 0x7f7fb0151e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.076+0000 7f7fb63d7640 1 -- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb00721f0 msgr2=0x7f7fb01552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.076+0000 7f7fb63d7640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb00721f0 0x7f7fb01552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.076+0000 7f7fb63d7640 1 -- 172.21.15.44:0/74159254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb0154fc0 con 0x7f7fb015e420 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.076+0000 7f7fb63d7640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb015e420 0x7f7fb0154a80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009f40 tx=0x7f7fa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.076+0000 7f7fa67fc640 1 -- 172.21.15.44:0/74159254 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8043020 con 0x7f7fb015e420 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.076+0000 7f7fa67fc640 1 -- 172.21.15.44:0/74159254 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fa803e450 con 0x7f7fb015e420 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.076+0000 7f7fa67fc640 1 -- 172.21.15.44:0/74159254 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa803e820 con 0x7f7fb015e420 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.076+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/74159254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb01513b0 con 0x7f7fb015e420 2024-04-02T13:22:08.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.076+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/74159254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb0155bf0 con 0x7f7fb015e420 2024-04-02T13:22:08.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.077+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/74159254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fb00684f0 con 0x7f7fb015e420 2024-04-02T13:22:08.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.077+0000 7f7fa67fc640 1 -- 172.21.15.44:0/74159254 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7fa8046020 con 0x7f7fb015e420 2024-04-02T13:22:08.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.078+0000 7f7fa67fc640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f9406c920 0x7f7f9406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.078+0000 7f7fa67fc640 1 -- 172.21.15.44:0/74159254 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f7fa80ae1e0 con 0x7f7fb015e420 2024-04-02T13:22:08.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.078+0000 7f7fb5bd6640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f9406c920 0x7f7f9406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.079+0000 7f7fb5bd6640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f9406c920 0x7f7f9406ede0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0098e0 tx=0x7f7fac009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:08.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.082+0000 7f7fa67fc640 1 -- 172.21.15.44:0/74159254 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7fa804d070 con 0x7f7fb015e420 2024-04-02T13:22:08.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.267+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/74159254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7fb00b3770 con 0x7f7fb015e420 2024-04-02T13:22:08.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.268+0000 7f7fa67fc640 1 -- 172.21.15.44:0/74159254 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) ==== 93+0+10 (secure 0 0 0) 0x7f7fb006c220 con 0x7f7fb015e420 2024-04-02T13:22:08.269 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:22:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.269+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f9406c920 msgr2=0x7f7f9406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.269+0000 7f7fb7e3a640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f9406c920 0x7f7f9406ede0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0098e0 tx=0x7f7fac009060 comp rx=0 tx=0).stop 2024-04-02T13:22:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.269+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/74159254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb015e420 msgr2=0x7f7fb0154a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.269+0000 7f7fb7e3a640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb015e420 0x7f7fb0154a80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009f40 tx=0x7f7fa8004a40 comp rx=0 tx=0).stop 2024-04-02T13:22:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.269+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/74159254 shutdown_connections 2024-04-02T13:22:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.269+0000 7f7fb7e3a640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f9406c920 0x7f7f9406ede0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.269+0000 7f7fb7e3a640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fb015e420 0x7f7fb0154a80 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.269+0000 7f7fb7e3a640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb00cfe10 0x7f7fb0151e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.269+0000 7f7fb7e3a640 1 --2- 172.21.15.44:0/74159254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb00721f0 0x7f7fb01552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.269+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/74159254 >> 172.21.15.44:0/74159254 conn(0x7f7fb00b2a10 msgr2=0x7f7fb00b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.269+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/74159254 shutdown_connections 2024-04-02T13:22:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.269+0000 7f7fb7e3a640 1 -- 172.21.15.44:0/74159254 wait complete. 2024-04-02T13:22:08.282 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:22:08.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.385+0000 7f6220a0d640 1 Processor -- start 2024-04-02T13:22:08.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.385+0000 7f6220a0d640 1 -- start start 2024-04-02T13:22:08.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f6220a0d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f621c0aeca0 0x7f621c06e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f6220a0d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f621c0d07b0 0x7f621c0acde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f6220a0d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f621c0cdd60 con 0x7f621c06eb10 2024-04-02T13:22:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f6220a0d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f621c15b3b0 con 0x7f621c0d07b0 2024-04-02T13:22:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f6220a0d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f621c0ce040 con 0x7f621c0aeca0 2024-04-02T13:22:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f621ad9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f621c0d07b0 0x7f621c0acde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f621ad9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f621c0d07b0 0x7f621c0acde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42966/0 (socket says 172.21.15.44:42966) 2024-04-02T13:22:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f621ad9d640 1 -- 172.21.15.44:0/2066140452 learned_addr learned my addr 172.21.15.44:0/2066140452 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f621a59c640 1 --2- 172.21.15.44:0/2066140452 >> v2:172.21.15.69:3300/0 conn(0x7f621c0aeca0 0x7f621c06e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f621ad9d640 1 -- 172.21.15.44:0/2066140452 >> v2:172.21.15.69:3300/0 conn(0x7f621c0aeca0 msgr2=0x7f621c06e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f621ad9d640 1 --2- 172.21.15.44:0/2066140452 >> v2:172.21.15.69:3300/0 conn(0x7f621c0aeca0 0x7f621c06e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f621ad9d640 1 -- 172.21.15.44:0/2066140452 >> v1:172.21.15.69:6789/0 conn(0x7f621c06eb10 legacy=0x7f621c0d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f621ad9d640 1 -- 172.21.15.44:0/2066140452 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621c0ce320 con 0x7f621c0d07b0 2024-04-02T13:22:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.386+0000 7f621a59c640 1 --2- 172.21.15.44:0/2066140452 >> v2:172.21.15.69:3300/0 conn(0x7f621c0aeca0 0x7f621c06e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:08.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.387+0000 7f621ad9d640 1 --2- 172.21.15.44:0/2066140452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0d07b0 0x7f621c0acde0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f621c0d0ce0 tx=0x7f621402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a45582394280e8c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:08.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.387+0000 7f621959a640 1 -- 172.21.15.44:0/2066140452 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621402ca80 con 0x7f621c0d07b0 2024-04-02T13:22:08.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.387+0000 7f621959a640 1 -- 172.21.15.44:0/2066140452 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f621402cc20 con 0x7f621c0d07b0 2024-04-02T13:22:08.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.387+0000 7f621959a640 1 -- 172.21.15.44:0/2066140452 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62140343f0 con 0x7f621c0d07b0 2024-04-02T13:22:08.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.388+0000 7f6220a0d640 1 -- 172.21.15.44:0/2066140452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0d07b0 msgr2=0x7f621c0acde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.388+0000 7f6220a0d640 1 --2- 172.21.15.44:0/2066140452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0d07b0 0x7f621c0acde0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f621c0d0ce0 tx=0x7f621402bee0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.388+0000 7f6220a0d640 1 -- 172.21.15.44:0/2066140452 shutdown_connections 2024-04-02T13:22:08.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.388+0000 7f6220a0d640 1 --2- 172.21.15.44:0/2066140452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0d07b0 0x7f621c0acde0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.388+0000 7f6220a0d640 1 --2- 172.21.15.44:0/2066140452 >> v2:172.21.15.69:3300/0 conn(0x7f621c0aeca0 0x7f621c06e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.388+0000 7f6220a0d640 1 -- 172.21.15.44:0/2066140452 >> 172.21.15.44:0/2066140452 conn(0x7f621c06b920 msgr2=0x7f621c06bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:08.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.388+0000 7f6220a0d640 1 -- 172.21.15.44:0/2066140452 shutdown_connections 2024-04-02T13:22:08.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.388+0000 7f6220a0d640 1 -- 172.21.15.44:0/2066140452 wait complete. 2024-04-02T13:22:08.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.389+0000 7f6220a0d640 1 Processor -- start 2024-04-02T13:22:08.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.389+0000 7f6220a0d640 1 -- start start 2024-04-02T13:22:08.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6220a0d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f621c06eb10 0x7f621c150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6220a0d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0aeca0 0x7f621c1592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6220a0d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f621c0d07b0 0x7f621c1597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6220a0d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f621c0ad320 con 0x7f621c0aeca0 2024-04-02T13:22:08.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6220a0d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f621c0ce8e0 con 0x7f621c06eb10 2024-04-02T13:22:08.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6220a0d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f621c0ad4a0 con 0x7f621c0d07b0 2024-04-02T13:22:08.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6219d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0aeca0 0x7f621c1592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6219d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0aeca0 0x7f621c1592a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42976/0 (socket says 172.21.15.44:42976) 2024-04-02T13:22:08.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6219d9b640 1 -- 172.21.15.44:0/1840674838 learned_addr learned my addr 172.21.15.44:0/1840674838 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:08.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f621ad9d640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f621c0d07b0 0x7f621c1597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f621a59c640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f621c06eb10 0x7f621c150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6219d9b640 1 -- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f621c0d07b0 msgr2=0x7f621c1597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6219d9b640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f621c0d07b0 0x7f621c1597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6219d9b640 1 -- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f621c06eb10 msgr2=0x7f621c150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6219d9b640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f621c06eb10 0x7f621c150830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6219d9b640 1 -- 172.21.15.44:0/1840674838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621c1f1ba0 con 0x7f621c0aeca0 2024-04-02T13:22:08.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f621ad9d640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f621c0d07b0 0x7f621c1597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:08.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f621a59c640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f621c06eb10 0x7f621c150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:08.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.390+0000 7f6219d9b640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0aeca0 0x7f621c1592a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f621000c430 tx=0x7f621000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:08.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.391+0000 7f620affd640 1 -- 172.21.15.44:0/1840674838 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6210011790 con 0x7f621c0aeca0 2024-04-02T13:22:08.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.391+0000 7f620affd640 1 -- 172.21.15.44:0/1840674838 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6210017070 con 0x7f621c0aeca0 2024-04-02T13:22:08.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.391+0000 7f6220a0d640 1 -- 172.21.15.44:0/1840674838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f621c1f1e20 con 0x7f621c0aeca0 2024-04-02T13:22:08.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.391+0000 7f620affd640 1 -- 172.21.15.44:0/1840674838 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6210011bb0 con 0x7f621c0aeca0 2024-04-02T13:22:08.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.391+0000 7f6220a0d640 1 -- 172.21.15.44:0/1840674838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f621c1f23e0 con 0x7f621c0aeca0 2024-04-02T13:22:08.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.392+0000 7f6220a0d640 1 -- 172.21.15.44:0/1840674838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f621c0684f0 con 0x7f621c0aeca0 2024-04-02T13:22:08.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.392+0000 7f620affd640 1 -- 172.21.15.44:0/1840674838 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f621001c460 con 0x7f621c0aeca0 2024-04-02T13:22:08.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.392+0000 7f620affd640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f61fc06c8f0 0x7f61fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.393+0000 7f620affd640 1 -- 172.21.15.44:0/1840674838 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f621008bbf0 con 0x7f621c0aeca0 2024-04-02T13:22:08.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.393+0000 7f621a59c640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f61fc06c8f0 0x7f61fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.393+0000 7f621a59c640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f61fc06c8f0 0x7f61fc06edb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f621c1e5690 tx=0x7f620c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:08.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.397+0000 7f620affd640 1 -- 172.21.15.44:0/1840674838 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f621c1f1ba0 con 0x7f621c0aeca0 2024-04-02T13:22:08.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.583+0000 7f6220a0d640 1 -- 172.21.15.44:0/1840674838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f621c06e5d0 con 0x7f621c0aeca0 2024-04-02T13:22:08.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.583+0000 7f620affd640 1 -- 172.21.15.44:0/1840674838 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v40) ==== 104+0+11 (secure 0 0 0) 0x7f6210059e80 con 0x7f621c0aeca0 2024-04-02T13:22:08.585 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:22:08.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.585+0000 7f6220a0d640 1 -- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f61fc06c8f0 msgr2=0x7f61fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.585+0000 7f6220a0d640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f61fc06c8f0 0x7f61fc06edb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f621c1e5690 tx=0x7f620c008040 comp rx=0 tx=0).stop 2024-04-02T13:22:08.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.585+0000 7f6220a0d640 1 -- 172.21.15.44:0/1840674838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0aeca0 msgr2=0x7f621c1592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.585+0000 7f6220a0d640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0aeca0 0x7f621c1592a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f621000c430 tx=0x7f621000c8f0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.585+0000 7f6220a0d640 1 -- 172.21.15.44:0/1840674838 shutdown_connections 2024-04-02T13:22:08.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.585+0000 7f6220a0d640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f61fc06c8f0 0x7f61fc06edb0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.585+0000 7f6220a0d640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f621c0d07b0 0x7f621c1597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.585+0000 7f6220a0d640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0aeca0 0x7f621c1592a0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.585+0000 7f6220a0d640 1 --2- 172.21.15.44:0/1840674838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f621c06eb10 0x7f621c150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.585+0000 7f6220a0d640 1 -- 172.21.15.44:0/1840674838 >> 172.21.15.44:0/1840674838 conn(0x7f621c06b920 msgr2=0x7f621c0d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:08.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.585+0000 7f6220a0d640 1 -- 172.21.15.44:0/1840674838 shutdown_connections 2024-04-02T13:22:08.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.585+0000 7f6220a0d640 1 -- 172.21.15.44:0/1840674838 wait complete. 2024-04-02T13:22:08.598 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:22:08.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.701+0000 7f4aec750640 1 Processor -- start 2024-04-02T13:22:08.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.701+0000 7f4aec750640 1 -- start start 2024-04-02T13:22:08.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4aec750640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4ae40d0c90 0x7f4ae40d0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4aec750640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4ae40d1070 0x7f4ae415e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4aec750640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ae40b3aa0 con 0x7f4ae40d2990 2024-04-02T13:22:08.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4aec750640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4ae40d0680 con 0x7f4ae40d1070 2024-04-02T13:22:08.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4aec750640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4ae40d2f50 con 0x7f4ae40d0c90 2024-04-02T13:22:08.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4ae9ceb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4ae40d1070 0x7f4ae415e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4aeaced640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4ae40d2990 0x7f4ae40cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:47160/0 (socket says 172.21.15.44:47160) 2024-04-02T13:22:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4aeaced640 1 -- 172.21.15.44:0/2969197776 learned_addr learned my addr 172.21.15.44:0/2969197776 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4aea4ec640 1 --2- 172.21.15.44:0/2969197776 >> v2:172.21.15.69:3300/0 conn(0x7f4ae40d0c90 0x7f4ae40d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4ae9ceb640 1 -- 172.21.15.44:0/2969197776 >> v2:172.21.15.69:3300/0 conn(0x7f4ae40d0c90 msgr2=0x7f4ae40d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4ae9ceb640 1 --2- 172.21.15.44:0/2969197776 >> v2:172.21.15.69:3300/0 conn(0x7f4ae40d0c90 0x7f4ae40d0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4ae9ceb640 1 -- 172.21.15.44:0/2969197776 >> v1:172.21.15.44:6789/0 conn(0x7f4ae40d2990 legacy=0x7f4ae40cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4ae9ceb640 1 -- 172.21.15.44:0/2969197776 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae40b3d40 con 0x7f4ae40d1070 2024-04-02T13:22:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.702+0000 7f4ae94ea640 1 -- 172.21.15.44:0/2969197776 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2640515576 0 0) 0x7f4ae40b3aa0 con 0x7f4ae40d2990 2024-04-02T13:22:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.703+0000 7f4ae9ceb640 1 --2- 172.21.15.44:0/2969197776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ae40d1070 0x7f4ae415e1b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4ae408e6c0 tx=0x7f4ad802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c1c59d2c2295ee4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.703+0000 7f4ae94ea640 1 -- 172.21.15.44:0/2969197776 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad80315a0 con 0x7f4ae40d1070 2024-04-02T13:22:08.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.703+0000 7f4ae94ea640 1 -- 172.21.15.44:0/2969197776 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ad8037070 con 0x7f4ae40d1070 2024-04-02T13:22:08.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.703+0000 7f4ae94ea640 1 -- 172.21.15.44:0/2969197776 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8031960 con 0x7f4ae40d1070 2024-04-02T13:22:08.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.704+0000 7f4aec750640 1 -- 172.21.15.44:0/2969197776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ae40d1070 msgr2=0x7f4ae415e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.704+0000 7f4aec750640 1 --2- 172.21.15.44:0/2969197776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ae40d1070 0x7f4ae415e1b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4ae408e6c0 tx=0x7f4ad802b840 comp rx=0 tx=0).stop 2024-04-02T13:22:08.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.704+0000 7f4aec750640 1 -- 172.21.15.44:0/2969197776 shutdown_connections 2024-04-02T13:22:08.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.704+0000 7f4aec750640 1 --2- 172.21.15.44:0/2969197776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ae40d1070 0x7f4ae415e1b0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.704+0000 7f4aec750640 1 --2- 172.21.15.44:0/2969197776 >> v2:172.21.15.69:3300/0 conn(0x7f4ae40d0c90 0x7f4ae40d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.704+0000 7f4aec750640 1 -- 172.21.15.44:0/2969197776 >> 172.21.15.44:0/2969197776 conn(0x7f4ae409d3d0 msgr2=0x7f4ae40d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:08.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.704+0000 7f4aec750640 1 -- 172.21.15.44:0/2969197776 shutdown_connections 2024-04-02T13:22:08.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.704+0000 7f4aec750640 1 -- 172.21.15.44:0/2969197776 wait complete. 2024-04-02T13:22:08.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.705+0000 7f4aec750640 1 Processor -- start 2024-04-02T13:22:08.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.705+0000 7f4aec750640 1 -- start start 2024-04-02T13:22:08.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aec750640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ae40d0c90 0x7f4ae41f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aec750640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ae40d1070 0x7f4ae40b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aec750640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ae40d2990 0x7f4ae415ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aec750640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae40d33b0 con 0x7f4ae40d2990 2024-04-02T13:22:08.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aec750640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae40d3230 con 0x7f4ae40d0c90 2024-04-02T13:22:08.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aec750640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4ae40b5550 con 0x7f4ae40d1070 2024-04-02T13:22:08.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aeaced640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ae40d2990 0x7f4ae415ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aeaced640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ae40d2990 0x7f4ae415ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42990/0 (socket says 172.21.15.44:42990) 2024-04-02T13:22:08.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aeaced640 1 -- 172.21.15.44:0/3694758900 learned_addr learned my addr 172.21.15.44:0/3694758900 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:08.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4ae9ceb640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ae40d1070 0x7f4ae40b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aea4ec640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ae40d0c90 0x7f4ae41f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aeaced640 1 -- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ae40d1070 msgr2=0x7f4ae40b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aeaced640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ae40d1070 0x7f4ae40b6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aeaced640 1 -- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ae40d0c90 msgr2=0x7f4ae41f3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aeaced640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ae40d0c90 0x7f4ae41f3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aeaced640 1 -- 172.21.15.44:0/3694758900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae40b7320 con 0x7f4ae40d2990 2024-04-02T13:22:08.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.706+0000 7f4aeaced640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ae40d2990 0x7f4ae415ba20 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400c700 tx=0x7f4ad400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:08.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.707+0000 7f4ae2ffd640 1 -- 172.21.15.44:0/3694758900 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad4018070 con 0x7f4ae40d2990 2024-04-02T13:22:08.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.707+0000 7f4ae2ffd640 1 -- 172.21.15.44:0/3694758900 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ad400ce10 con 0x7f4ae40d2990 2024-04-02T13:22:08.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.707+0000 7f4aec750640 1 -- 172.21.15.44:0/3694758900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae40b7600 con 0x7f4ae40d2990 2024-04-02T13:22:08.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.707+0000 7f4ae2ffd640 1 -- 172.21.15.44:0/3694758900 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad400a860 con 0x7f4ae40d2990 2024-04-02T13:22:08.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.707+0000 7f4aec750640 1 -- 172.21.15.44:0/3694758900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae40b7bc0 con 0x7f4ae40d2990 2024-04-02T13:22:08.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.708+0000 7f4aec750640 1 -- 172.21.15.44:0/3694758900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ae4068c00 con 0x7f4ae40d2990 2024-04-02T13:22:08.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.708+0000 7f4ae2ffd640 1 -- 172.21.15.44:0/3694758900 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4ad401b070 con 0x7f4ae40d2990 2024-04-02T13:22:08.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.709+0000 7f4ae2ffd640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ad006c8f0 0x7f4ad006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:08.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.709+0000 7f4ae2ffd640 1 -- 172.21.15.44:0/3694758900 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f4ad408b240 con 0x7f4ae40d2990 2024-04-02T13:22:08.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.709+0000 7f4aea4ec640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ad006c8f0 0x7f4ad006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:08.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.709+0000 7f4aea4ec640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ad006c8f0 0x7f4ad006edb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00a970 tx=0x7f4adc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:08.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.713+0000 7f4ae2ffd640 1 -- 172.21.15.44:0/3694758900 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4ad4059550 con 0x7f4ae40d2990 2024-04-02T13:22:08.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.897+0000 7f4aec750640 1 -- 172.21.15.44:0/3694758900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4ae40d0400 con 0x7f4ae40d2990 2024-04-02T13:22:08.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.897+0000 7f4ae2ffd640 1 -- 172.21.15.44:0/3694758900 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) ==== 100+0+11 (secure 0 0 0) 0x7f4ad4058e50 con 0x7f4ae40d2990 2024-04-02T13:22:08.899 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:22:08.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.899+0000 7f4aec750640 1 -- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ad006c8f0 msgr2=0x7f4ad006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.899+0000 7f4aec750640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ad006c8f0 0x7f4ad006edb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00a970 tx=0x7f4adc008040 comp rx=0 tx=0).stop 2024-04-02T13:22:08.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.899+0000 7f4aec750640 1 -- 172.21.15.44:0/3694758900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ae40d2990 msgr2=0x7f4ae415ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:08.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.899+0000 7f4aec750640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ae40d2990 0x7f4ae415ba20 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400c700 tx=0x7f4ad400cbc0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.899+0000 7f4aec750640 1 -- 172.21.15.44:0/3694758900 shutdown_connections 2024-04-02T13:22:08.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.899+0000 7f4aec750640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ad006c8f0 0x7f4ad006edb0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.899+0000 7f4aec750640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ae40d2990 0x7f4ae415ba20 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.899+0000 7f4aec750640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ae40d1070 0x7f4ae40b6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.899+0000 7f4aec750640 1 --2- 172.21.15.44:0/3694758900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ae40d0c90 0x7f4ae41f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:08.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.899+0000 7f4aec750640 1 -- 172.21.15.44:0/3694758900 >> 172.21.15.44:0/3694758900 conn(0x7f4ae409d3d0 msgr2=0x7f4ae415d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:08.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.899+0000 7f4aec750640 1 -- 172.21.15.44:0/3694758900 shutdown_connections 2024-04-02T13:22:08.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:08.899+0000 7f4aec750640 1 -- 172.21.15.44:0/3694758900 wait complete. 2024-04-02T13:22:08.912 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:22:09.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.014+0000 7f373e64a640 1 Processor -- start 2024-04-02T13:22:09.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.015+0000 7f373e64a640 1 -- start start 2024-04-02T13:22:09.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.015+0000 7f373e64a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f373806cf90 0x7f37380cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.015+0000 7f373e64a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f37380cfe30 0x7f37380d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.015+0000 7f373e64a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f373806c1d0 con 0x7f37380d0a30 2024-04-02T13:22:09.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.015+0000 7f373e64a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f373806cb00 con 0x7f37380cfe30 2024-04-02T13:22:09.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.015+0000 7f373e64a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f373806d3f0 con 0x7f373806cf90 2024-04-02T13:22:09.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.015+0000 7f373cbe7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f37380cfe30 0x7f37380d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.015+0000 7f373cbe7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f37380cfe30 0x7f37380d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43000/0 (socket says 172.21.15.44:43000) 2024-04-02T13:22:09.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.015+0000 7f373cbe7640 1 -- 172.21.15.44:0/1746439398 learned_addr learned my addr 172.21.15.44:0/1746439398 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:09.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.016+0000 7f37377fe640 1 --2- 172.21.15.44:0/1746439398 >> v2:172.21.15.69:3300/0 conn(0x7f373806cf90 0x7f37380cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.016+0000 7f373cbe7640 1 -- 172.21.15.44:0/1746439398 >> v2:172.21.15.69:3300/0 conn(0x7f373806cf90 msgr2=0x7f37380cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:09.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.016+0000 7f373cbe7640 1 --2- 172.21.15.44:0/1746439398 >> v2:172.21.15.69:3300/0 conn(0x7f373806cf90 0x7f37380cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.016+0000 7f373cbe7640 1 -- 172.21.15.44:0/1746439398 >> v1:172.21.15.69:6789/0 conn(0x7f37380d0a30 legacy=0x7f37380d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:09.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.016+0000 7f373cbe7640 1 -- 172.21.15.44:0/1746439398 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373806c4b0 con 0x7f37380cfe30 2024-04-02T13:22:09.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.016+0000 7f37377fe640 1 --2- 172.21.15.44:0/1746439398 >> v2:172.21.15.69:3300/0 conn(0x7f373806cf90 0x7f37380cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:09.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.016+0000 7f373cbe7640 1 --2- 172.21.15.44:0/1746439398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37380cfe30 0x7f37380d2040 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f37380b6d30 tx=0x7f373002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ad82b13c0c12c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:09.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.017+0000 7f3736ffd640 1 -- 172.21.15.44:0/1746439398 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373002c9f0 con 0x7f37380cfe30 2024-04-02T13:22:09.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.017+0000 7f3736ffd640 1 -- 172.21.15.44:0/1746439398 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373002cb90 con 0x7f37380cfe30 2024-04-02T13:22:09.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.017+0000 7f3736ffd640 1 -- 172.21.15.44:0/1746439398 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37300343f0 con 0x7f37380cfe30 2024-04-02T13:22:09.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.017+0000 7f373e64a640 1 -- 172.21.15.44:0/1746439398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37380cfe30 msgr2=0x7f37380d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.017+0000 7f373e64a640 1 --2- 172.21.15.44:0/1746439398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37380cfe30 0x7f37380d2040 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f37380b6d30 tx=0x7f373002be50 comp rx=0 tx=0).stop 2024-04-02T13:22:09.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.017+0000 7f373e64a640 1 -- 172.21.15.44:0/1746439398 shutdown_connections 2024-04-02T13:22:09.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.017+0000 7f373e64a640 1 --2- 172.21.15.44:0/1746439398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37380cfe30 0x7f37380d2040 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.017+0000 7f373e64a640 1 --2- 172.21.15.44:0/1746439398 >> v2:172.21.15.69:3300/0 conn(0x7f373806cf90 0x7f37380cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.017+0000 7f373e64a640 1 -- 172.21.15.44:0/1746439398 >> 172.21.15.44:0/1746439398 conn(0x7f373809d470 msgr2=0x7f37380d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:09.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.017+0000 7f373e64a640 1 -- 172.21.15.44:0/1746439398 shutdown_connections 2024-04-02T13:22:09.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.017+0000 7f373e64a640 1 -- 172.21.15.44:0/1746439398 wait complete. 2024-04-02T13:22:09.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.018+0000 7f373e64a640 1 Processor -- start 2024-04-02T13:22:09.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.019+0000 7f373e64a640 1 -- start start 2024-04-02T13:22:09.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.019+0000 7f373e64a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f373806cf90 0x7f37380b9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.019+0000 7f373e64a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37380cfe30 0x7f37380b1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.019+0000 7f373e64a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37380d0a30 0x7f3738155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.019+0000 7f373e64a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f373815e870 con 0x7f37380d0a30 2024-04-02T13:22:09.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.019+0000 7f373e64a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f373815e6f0 con 0x7f373806cf90 2024-04-02T13:22:09.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.019+0000 7f373e64a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f37380d4da0 con 0x7f37380cfe30 2024-04-02T13:22:09.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.019+0000 7f373cbe7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37380d0a30 0x7f3738155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f373cbe7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37380d0a30 0x7f3738155250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43006/0 (socket says 172.21.15.44:43006) 2024-04-02T13:22:09.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f373cbe7640 1 -- 172.21.15.44:0/423098692 learned_addr learned my addr 172.21.15.44:0/423098692 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:09.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f37377fe640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37380cfe30 0x7f37380b1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f3737fff640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f373806cf90 0x7f37380b9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f373cbe7640 1 -- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37380cfe30 msgr2=0x7f37380b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f373cbe7640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37380cfe30 0x7f37380b1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f373cbe7640 1 -- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f373806cf90 msgr2=0x7f37380b9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f373cbe7640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f373806cf90 0x7f37380b9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f373cbe7640 1 -- 172.21.15.44:0/423098692 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37380b53d0 con 0x7f37380d0a30 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f3737fff640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f373806cf90 0x7f37380b9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f373cbe7640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37380d0a30 0x7f3738155250 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3730036040 tx=0x7f3730004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f3734ff9640 1 -- 172.21.15.44:0/423098692 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3730043020 con 0x7f37380d0a30 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f3734ff9640 1 -- 172.21.15.44:0/423098692 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373003e450 con 0x7f37380d0a30 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f373e64a640 1 -- 172.21.15.44:0/423098692 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37380b56b0 con 0x7f37380d0a30 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f3734ff9640 1 -- 172.21.15.44:0/423098692 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373003e860 con 0x7f37380d0a30 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.020+0000 7f373e64a640 1 -- 172.21.15.44:0/423098692 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37380b5f50 con 0x7f37380d0a30 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.021+0000 7f373e64a640 1 -- 172.21.15.44:0/423098692 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37380b5880 con 0x7f37380d0a30 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.022+0000 7f3734ff9640 1 -- 172.21.15.44:0/423098692 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3730046020 con 0x7f37380d0a30 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.022+0000 7f3734ff9640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f372406c8f0 0x7f372406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.022+0000 7f3734ff9640 1 -- 172.21.15.44:0/423098692 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f37300ae660 con 0x7f37380d0a30 2024-04-02T13:22:09.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.022+0000 7f3737fff640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f372406c8f0 0x7f372406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.023+0000 7f3737fff640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f372406c8f0 0x7f372406edb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f372800a830 tx=0x7f3728008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:09.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.026+0000 7f3734ff9640 1 -- 172.21.15.44:0/423098692 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f373004d030 con 0x7f37380d0a30 2024-04-02T13:22:09.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.248+0000 7f373e64a640 1 -- 172.21.15.44:0/423098692 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37380684f0 con 0x7f37380d0a30 2024-04-02T13:22:09.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.249+0000 7f3734ff9640 1 -- 172.21.15.44:0/423098692 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5181 (secure 0 0 0) 0x7f373007c8d0 con 0x7f37380d0a30 2024-04-02T13:22:09.250 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:09.250 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3050514527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3050514527},{"type":"v1","addr":"172.21.15.69:6836","nonce":3050514527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4973,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1949024012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1949024012},{"type":"v1","addr":"172.21.15.69:6837","nonce":1949024012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/2817873198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2817873198},{"type":"v1","addr":"172.21.15.44:6834","nonce":2817873198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:21:43.215796+0000","modified":"2024-04-02T13:21:46.248294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/1969465285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":1969465285},{"type":"v1","addr":"172.21.15.44:6835","nonce":1969465285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":2}]} 2024-04-02T13:22:09.251 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 18 2024-04-02T13:22:09.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.250+0000 7f373e64a640 1 -- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f372406c8f0 msgr2=0x7f372406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.250+0000 7f373e64a640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f372406c8f0 0x7f372406edb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f372800a830 tx=0x7f3728008040 comp rx=0 tx=0).stop 2024-04-02T13:22:09.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.250+0000 7f373e64a640 1 -- 172.21.15.44:0/423098692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37380d0a30 msgr2=0x7f3738155250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.250+0000 7f373e64a640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37380d0a30 0x7f3738155250 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3730036040 tx=0x7f3730004a40 comp rx=0 tx=0).stop 2024-04-02T13:22:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.250+0000 7f373e64a640 1 -- 172.21.15.44:0/423098692 shutdown_connections 2024-04-02T13:22:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.250+0000 7f373e64a640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f372406c8f0 0x7f372406edb0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.250+0000 7f373e64a640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37380d0a30 0x7f3738155250 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.250+0000 7f373e64a640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37380cfe30 0x7f37380b1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.250+0000 7f373e64a640 1 --2- 172.21.15.44:0/423098692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f373806cf90 0x7f37380b9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.250+0000 7f373e64a640 1 -- 172.21.15.44:0/423098692 >> 172.21.15.44:0/423098692 conn(0x7f373809d470 msgr2=0x7f3738160d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.251+0000 7f373e64a640 1 -- 172.21.15.44:0/423098692 shutdown_connections 2024-04-02T13:22:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.251+0000 7f373e64a640 1 -- 172.21.15.44:0/423098692 wait complete. 2024-04-02T13:22:09.263 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:22:09.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.363+0000 7fc12cae3640 1 Processor -- start 2024-04-02T13:22:09.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.363+0000 7fc12cae3640 1 -- start start 2024-04-02T13:22:09.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.364+0000 7fc12cae3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc1280d09f0 0x7fc12806d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.364+0000 7fc12cae3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc12806ece0 0x7fc1280acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.364+0000 7fc12cae3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc12806dc40 con 0x7fc1280cece0 2024-04-02T13:22:09.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.364+0000 7fc12cae3640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc12806df20 con 0x7fc1280d09f0 2024-04-02T13:22:09.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.364+0000 7fc12cae3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc12806e200 con 0x7fc12806ece0 2024-04-02T13:22:09.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.364+0000 7fc125d9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc1280d09f0 0x7fc12806d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.364+0000 7fc125d9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc1280d09f0 0x7fc12806d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43010/0 (socket says 172.21.15.44:43010) 2024-04-02T13:22:09.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.364+0000 7fc125d9b640 1 -- 172.21.15.44:0/3437748707 learned_addr learned my addr 172.21.15.44:0/3437748707 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:09.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.364+0000 7fc126d9d640 1 --2- 172.21.15.44:0/3437748707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc12806ece0 0x7fc1280acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.364+0000 7fc125d9b640 1 -- 172.21.15.44:0/3437748707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc12806ece0 msgr2=0x7fc1280acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:09.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.364+0000 7fc125d9b640 1 --2- 172.21.15.44:0/3437748707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc12806ece0 0x7fc1280acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.364+0000 7fc125d9b640 1 -- 172.21.15.44:0/3437748707 >> v1:172.21.15.69:6789/0 conn(0x7fc1280cece0 legacy=0x7fc1280cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.364+0000 7fc125d9b640 1 -- 172.21.15.44:0/3437748707 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc12806e4e0 con 0x7fc1280d09f0 2024-04-02T13:22:09.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.365+0000 7fc125d9b640 1 --2- 172.21.15.44:0/3437748707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1280d09f0 0x7fc12806d2d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc110009870 tx=0x7fc11002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655181d37f0071cf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:09.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.365+0000 7fc12559a640 1 -- 172.21.15.44:0/3437748707 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc110034070 con 0x7fc1280d09f0 2024-04-02T13:22:09.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.365+0000 7fc12559a640 1 -- 172.21.15.44:0/3437748707 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc110030840 con 0x7fc1280d09f0 2024-04-02T13:22:09.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.365+0000 7fc12559a640 1 -- 172.21.15.44:0/3437748707 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc110030c10 con 0x7fc1280d09f0 2024-04-02T13:22:09.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.366+0000 7fc12cae3640 1 -- 172.21.15.44:0/3437748707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1280d09f0 msgr2=0x7fc12806d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.366+0000 7fc12cae3640 1 --2- 172.21.15.44:0/3437748707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1280d09f0 0x7fc12806d2d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc110009870 tx=0x7fc11002b980 comp rx=0 tx=0).stop 2024-04-02T13:22:09.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.366+0000 7fc12cae3640 1 -- 172.21.15.44:0/3437748707 shutdown_connections 2024-04-02T13:22:09.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.366+0000 7fc12cae3640 1 --2- 172.21.15.44:0/3437748707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1280d09f0 0x7fc12806d2d0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.366+0000 7fc12cae3640 1 --2- 172.21.15.44:0/3437748707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc12806ece0 0x7fc1280acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.366+0000 7fc12cae3640 1 -- 172.21.15.44:0/3437748707 >> 172.21.15.44:0/3437748707 conn(0x7fc1280cfcd0 msgr2=0x7fc1280d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:09.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.366+0000 7fc12cae3640 1 -- 172.21.15.44:0/3437748707 shutdown_connections 2024-04-02T13:22:09.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.366+0000 7fc12cae3640 1 -- 172.21.15.44:0/3437748707 wait complete. 2024-04-02T13:22:09.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.367+0000 7fc12cae3640 1 Processor -- start 2024-04-02T13:22:09.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.367+0000 7fc12cae3640 1 -- start start 2024-04-02T13:22:09.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.368+0000 7fc12cae3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc12806ece0 0x7fc12815ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.368+0000 7fc12cae3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1280cece0 0x7fc1280b1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.368+0000 7fc12cae3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1280d09f0 0x7fc1281574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.368+0000 7fc12cae3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc12806d990 con 0x7fc12806ece0 2024-04-02T13:22:09.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.368+0000 7fc12cae3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc12806d810 con 0x7fc1280d09f0 2024-04-02T13:22:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.368+0000 7fc12cae3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc12815b220 con 0x7fc1280cece0 2024-04-02T13:22:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.368+0000 7fc12659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc12806ece0 0x7fc12815ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.368+0000 7fc12659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc12806ece0 0x7fc12815ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43014/0 (socket says 172.21.15.44:43014) 2024-04-02T13:22:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.368+0000 7fc12659c640 1 -- 172.21.15.44:0/667931880 learned_addr learned my addr 172.21.15.44:0/667931880 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.368+0000 7fc125d9b640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1280cece0 0x7fc1280b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.368+0000 7fc126d9d640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1280d09f0 0x7fc1281574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.368+0000 7fc12659c640 1 -- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1280cece0 msgr2=0x7fc1280b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.369+0000 7fc12659c640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1280cece0 0x7fc1280b1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.369+0000 7fc12659c640 1 -- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1280d09f0 msgr2=0x7fc1281574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.369+0000 7fc12659c640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1280d09f0 0x7fc1281574d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.369+0000 7fc12659c640 1 -- 172.21.15.44:0/667931880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1280b1760 con 0x7fc12806ece0 2024-04-02T13:22:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.369+0000 7fc12659c640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc12806ece0 0x7fc12815ccd0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc12000c400 tx=0x7fc12000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:09.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.369+0000 7fc11effd640 1 -- 172.21.15.44:0/667931880 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1200052d0 con 0x7fc12806ece0 2024-04-02T13:22:09.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.369+0000 7fc11effd640 1 -- 172.21.15.44:0/667931880 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc120005490 con 0x7fc12806ece0 2024-04-02T13:22:09.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.369+0000 7fc11effd640 1 -- 172.21.15.44:0/667931880 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc120007260 con 0x7fc12806ece0 2024-04-02T13:22:09.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.369+0000 7fc12cae3640 1 -- 172.21.15.44:0/667931880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc12809d500 con 0x7fc12806ece0 2024-04-02T13:22:09.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.369+0000 7fc12cae3640 1 -- 172.21.15.44:0/667931880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc12815c1f0 con 0x7fc12806ece0 2024-04-02T13:22:09.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.370+0000 7fc12cae3640 1 -- 172.21.15.44:0/667931880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc128074f20 con 0x7fc12806ece0 2024-04-02T13:22:09.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.370+0000 7fc11effd640 1 -- 172.21.15.44:0/667931880 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc120007400 con 0x7fc12806ece0 2024-04-02T13:22:09.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.371+0000 7fc11effd640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc10806c8f0 0x7fc10806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.371+0000 7fc11effd640 1 -- 172.21.15.44:0/667931880 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fc12008b200 con 0x7fc12806ece0 2024-04-02T13:22:09.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.371+0000 7fc125d9b640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc10806c8f0 0x7fc10806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.371+0000 7fc125d9b640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc10806c8f0 0x7fc10806edb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc110036000 tx=0x7fc11003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:09.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.375+0000 7fc11effd640 1 -- 172.21.15.44:0/667931880 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc120058e10 con 0x7fc12806ece0 2024-04-02T13:22:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.560+0000 7fc12cae3640 1 -- 172.21.15.44:0/667931880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc128075550 con 0x7fc12806ece0 2024-04-02T13:22:09.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.561+0000 7fc11effd640 1 -- 172.21.15.44:0/667931880 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+16943 (secure 0 0 0) 0x7fc12005d110 con 0x7fc12806ece0 2024-04-02T13:22:09.563 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:09.563 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":41,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:22:08.943287+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:21:41.647497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-02T13:21:42.168299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:22:09.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.563+0000 7fc12cae3640 1 -- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc10806c8f0 msgr2=0x7fc10806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.563+0000 7fc12cae3640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc10806c8f0 0x7fc10806edb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc110036000 tx=0x7fc11003b040 comp rx=0 tx=0).stop 2024-04-02T13:22:09.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.563+0000 7fc12cae3640 1 -- 172.21.15.44:0/667931880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc12806ece0 msgr2=0x7fc12815ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.563+0000 7fc12cae3640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc12806ece0 0x7fc12815ccd0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc12000c400 tx=0x7fc12000c8c0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.563+0000 7fc12cae3640 1 -- 172.21.15.44:0/667931880 shutdown_connections 2024-04-02T13:22:09.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.563+0000 7fc12cae3640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc10806c8f0 0x7fc10806edb0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.563+0000 7fc12cae3640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1280d09f0 0x7fc1281574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.563+0000 7fc12cae3640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1280cece0 0x7fc1280b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.563+0000 7fc12cae3640 1 --2- 172.21.15.44:0/667931880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc12806ece0 0x7fc12815ccd0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.563+0000 7fc12cae3640 1 -- 172.21.15.44:0/667931880 >> 172.21.15.44:0/667931880 conn(0x7fc1280cfcd0 msgr2=0x7fc1281547b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:09.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.563+0000 7fc12cae3640 1 -- 172.21.15.44:0/667931880 shutdown_connections 2024-04-02T13:22:09.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.563+0000 7fc12cae3640 1 -- 172.21.15.44:0/667931880 wait complete. 2024-04-02T13:22:09.576 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T13:22:09.576 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:22:09.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.676+0000 7fdf94888640 1 Processor -- start 2024-04-02T13:22:09.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.676+0000 7fdf94888640 1 -- start start 2024-04-02T13:22:09.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf94888640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdf900cece0 0x7fdf900cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf94888640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf900d09f0 0x7fdf9006d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf94888640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdf9006ece0 0x7fdf9015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf94888640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fdf900cf600 con 0x7fdf900cece0 2024-04-02T13:22:09.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf94888640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fdf9006dc40 con 0x7fdf9006ece0 2024-04-02T13:22:09.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf94888640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9006df20 con 0x7fdf900d09f0 2024-04-02T13:22:09.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf8e59c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdf900cece0 0x7fdf900cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf8e59c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdf900cece0 0x7fdf900cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43018/0 (socket says 172.21.15.44:43018) 2024-04-02T13:22:09.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf8e59c640 1 -- 172.21.15.44:0/907346409 learned_addr learned my addr 172.21.15.44:0/907346409 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:09.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf8dd9b640 1 --2- 172.21.15.44:0/907346409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf900d09f0 0x7fdf9006d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf8ed9d640 1 --2- 172.21.15.44:0/907346409 >> v2:172.21.15.69:3300/0 conn(0x7fdf9006ece0 0x7fdf9015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf8e59c640 1 -- 172.21.15.44:0/907346409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf900d09f0 msgr2=0x7fdf9006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:09.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf8e59c640 1 --2- 172.21.15.44:0/907346409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf900d09f0 0x7fdf9006d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf8e59c640 1 -- 172.21.15.44:0/907346409 >> v2:172.21.15.69:3300/0 conn(0x7fdf9006ece0 msgr2=0x7fdf9015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:09.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf8e59c640 1 --2- 172.21.15.44:0/907346409 >> v2:172.21.15.69:3300/0 conn(0x7fdf9006ece0 0x7fdf9015eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.677+0000 7fdf8e59c640 1 -- 172.21.15.44:0/907346409 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9006e200 con 0x7fdf900cece0 2024-04-02T13:22:09.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.678+0000 7fdf8e59c640 1 --2- 172.21.15.44:0/907346409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf900cece0 0x7fdf900cf0c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdf900cf7e0 tx=0x7fdf8802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf0964346731905 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:09.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.678+0000 7fdf8d59a640 1 -- 172.21.15.44:0/907346409 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88035020 con 0x7fdf900cece0 2024-04-02T13:22:09.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.678+0000 7fdf8d59a640 1 -- 172.21.15.44:0/907346409 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf88031640 con 0x7fdf900cece0 2024-04-02T13:22:09.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.678+0000 7fdf8d59a640 1 -- 172.21.15.44:0/907346409 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8802d040 con 0x7fdf900cece0 2024-04-02T13:22:09.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.679+0000 7fdf94888640 1 -- 172.21.15.44:0/907346409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf900cece0 msgr2=0x7fdf900cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.679+0000 7fdf94888640 1 --2- 172.21.15.44:0/907346409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf900cece0 0x7fdf900cf0c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdf900cf7e0 tx=0x7fdf8802b8f0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.679+0000 7fdf94888640 1 -- 172.21.15.44:0/907346409 shutdown_connections 2024-04-02T13:22:09.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.679+0000 7fdf94888640 1 --2- 172.21.15.44:0/907346409 >> v2:172.21.15.69:3300/0 conn(0x7fdf9006ece0 0x7fdf9015eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.679+0000 7fdf94888640 1 --2- 172.21.15.44:0/907346409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf900d09f0 0x7fdf9006d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.679+0000 7fdf94888640 1 --2- 172.21.15.44:0/907346409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf900cece0 0x7fdf900cf0c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.679+0000 7fdf94888640 1 -- 172.21.15.44:0/907346409 >> 172.21.15.44:0/907346409 conn(0x7fdf900cfcd0 msgr2=0x7fdf900d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:09.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.679+0000 7fdf94888640 1 -- 172.21.15.44:0/907346409 shutdown_connections 2024-04-02T13:22:09.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.679+0000 7fdf94888640 1 -- 172.21.15.44:0/907346409 wait complete. 2024-04-02T13:22:09.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.680+0000 7fdf94888640 1 Processor -- start 2024-04-02T13:22:09.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.680+0000 7fdf94888640 1 -- start start 2024-04-02T13:22:09.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf94888640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf9006ece0 0x7fdf9014c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf94888640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf900cece0 0x7fdf9009d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf94888640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf900d09f0 0x7fdf901f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf94888640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9006d810 con 0x7fdf900cece0 2024-04-02T13:22:09.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf94888640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9006e7c0 con 0x7fdf900d09f0 2024-04-02T13:22:09.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf94888640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9006d990 con 0x7fdf9006ece0 2024-04-02T13:22:09.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf8dd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf900cece0 0x7fdf9009d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf8dd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf900cece0 0x7fdf9009d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43028/0 (socket says 172.21.15.44:43028) 2024-04-02T13:22:09.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf8dd9b640 1 -- 172.21.15.44:0/2488714885 learned_addr learned my addr 172.21.15.44:0/2488714885 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:09.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf8e59c640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf9006ece0 0x7fdf9014c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf8ed9d640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf900d09f0 0x7fdf901f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf8dd9b640 1 -- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf9006ece0 msgr2=0x7fdf9014c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf8dd9b640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf9006ece0 0x7fdf9014c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf8dd9b640 1 -- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf900d09f0 msgr2=0x7fdf901f7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf8dd9b640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf900d09f0 0x7fdf901f7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.681+0000 7fdf8dd9b640 1 -- 172.21.15.44:0/2488714885 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9014ccf0 con 0x7fdf900cece0 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.682+0000 7fdf8e59c640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf9006ece0 0x7fdf9014c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.682+0000 7fdf8ed9d640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf900d09f0 0x7fdf901f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.682+0000 7fdf8dd9b640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf900cece0 0x7fdf9009d320 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800e870 tx=0x7fdf7800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.682+0000 7fdf86ffd640 1 -- 172.21.15.44:0/2488714885 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78017070 con 0x7fdf900cece0 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.682+0000 7fdf86ffd640 1 -- 172.21.15.44:0/2488714885 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf7800c940 con 0x7fdf900cece0 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.682+0000 7fdf94888640 1 -- 172.21.15.44:0/2488714885 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9009d860 con 0x7fdf900cece0 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.682+0000 7fdf86ffd640 1 -- 172.21.15.44:0/2488714885 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7800cd10 con 0x7fdf900cece0 2024-04-02T13:22:09.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.682+0000 7fdf94888640 1 -- 172.21.15.44:0/2488714885 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf901f8840 con 0x7fdf900cece0 2024-04-02T13:22:09.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.683+0000 7fdf94888640 1 -- 172.21.15.44:0/2488714885 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf900684f0 con 0x7fdf900cece0 2024-04-02T13:22:09.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.683+0000 7fdf86ffd640 1 -- 172.21.15.44:0/2488714885 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdf7801a070 con 0x7fdf900cece0 2024-04-02T13:22:09.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.684+0000 7fdf86ffd640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdf7006c8f0 0x7fdf7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:09.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.684+0000 7fdf86ffd640 1 -- 172.21.15.44:0/2488714885 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fdf7808bbe0 con 0x7fdf900cece0 2024-04-02T13:22:09.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.684+0000 7fdf8e59c640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdf7006c8f0 0x7fdf7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:09.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.684+0000 7fdf8e59c640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdf7006c8f0 0x7fdf7006edb0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdf88036000 tx=0x7fdf8803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:09.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.688+0000 7fdf86ffd640 1 -- 172.21.15.44:0/2488714885 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fdf9014ccf0 con 0x7fdf900cece0 2024-04-02T13:22:09.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.910+0000 7fdf94888640 1 -- 172.21.15.44:0/2488714885 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf900ce230 con 0x7fdf900cece0 2024-04-02T13:22:09.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.910+0000 7fdf86ffd640 1 -- 172.21.15.44:0/2488714885 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5181 (secure 0 0 0) 0x7fdf78059e60 con 0x7fdf900cece0 2024-04-02T13:22:09.912 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:09.912 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3050514527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3050514527},{"type":"v1","addr":"172.21.15.69:6836","nonce":3050514527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4973,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1949024012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1949024012},{"type":"v1","addr":"172.21.15.69:6837","nonce":1949024012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/2817873198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2817873198},{"type":"v1","addr":"172.21.15.44:6834","nonce":2817873198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:21:43.215796+0000","modified":"2024-04-02T13:21:46.248294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/1969465285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":1969465285},{"type":"v1","addr":"172.21.15.44:6835","nonce":1969465285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":2}]} 2024-04-02T13:22:09.913 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 18 2024-04-02T13:22:09.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.912+0000 7fdf94888640 1 -- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdf7006c8f0 msgr2=0x7fdf7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.912+0000 7fdf94888640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdf7006c8f0 0x7fdf7006edb0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdf88036000 tx=0x7fdf8803c040 comp rx=0 tx=0).stop 2024-04-02T13:22:09.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.912+0000 7fdf94888640 1 -- 172.21.15.44:0/2488714885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf900cece0 msgr2=0x7fdf9009d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:09.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.912+0000 7fdf94888640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf900cece0 0x7fdf9009d320 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800e870 tx=0x7fdf7800ed30 comp rx=0 tx=0).stop 2024-04-02T13:22:09.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.912+0000 7fdf94888640 1 -- 172.21.15.44:0/2488714885 shutdown_connections 2024-04-02T13:22:09.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.912+0000 7fdf94888640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdf7006c8f0 0x7fdf7006edb0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.912+0000 7fdf94888640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf900d09f0 0x7fdf901f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.912+0000 7fdf94888640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf900cece0 0x7fdf9009d320 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.912+0000 7fdf94888640 1 --2- 172.21.15.44:0/2488714885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf9006ece0 0x7fdf9014c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.912+0000 7fdf94888640 1 -- 172.21.15.44:0/2488714885 >> 172.21.15.44:0/2488714885 conn(0x7fdf900cfcd0 msgr2=0x7fdf90157670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.912+0000 7fdf94888640 1 -- 172.21.15.44:0/2488714885 shutdown_connections 2024-04-02T13:22:09.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:09.912+0000 7fdf94888640 1 -- 172.21.15.44:0/2488714885 wait complete. 2024-04-02T13:22:09.926 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:22:10.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.027+0000 7f8740b04640 1 Processor -- start 2024-04-02T13:22:10.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.027+0000 7f8740b04640 1 -- start start 2024-04-02T13:22:10.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.028+0000 7f8740b04640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f873c15e520 0x7f873c0d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.028+0000 7f8740b04640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f873c0d0f80 0x7f873c0b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.028+0000 7f8740b04640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f873c0b70b0 con 0x7f873c15e140 2024-04-02T13:22:10.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.028+0000 7f8740b04640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f873c0d13f0 con 0x7f873c0d0f80 2024-04-02T13:22:10.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.028+0000 7f8740b04640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f873c0b9f50 con 0x7f873c15e520 2024-04-02T13:22:10.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.028+0000 7f873ad9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f873c0d0f80 0x7f873c0b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.028+0000 7f873ad9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f873c0d0f80 0x7f873c0b4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43032/0 (socket says 172.21.15.44:43032) 2024-04-02T13:22:10.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.028+0000 7f873ad9d640 1 -- 172.21.15.44:0/1437349822 learned_addr learned my addr 172.21.15.44:0/1437349822 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:10.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.028+0000 7f8739d9b640 1 --2- 172.21.15.44:0/1437349822 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f873c15e520 0x7f873c0d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.028+0000 7f873ad9d640 1 -- 172.21.15.44:0/1437349822 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f873c15e520 msgr2=0x7f873c0d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:10.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.028+0000 7f873ad9d640 1 --2- 172.21.15.44:0/1437349822 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f873c15e520 0x7f873c0d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.028+0000 7f873ad9d640 1 -- 172.21.15.44:0/1437349822 >> v1:172.21.15.69:6789/0 conn(0x7f873c15e140 legacy=0x7f873c0d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:10.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.028+0000 7f873ad9d640 1 -- 172.21.15.44:0/1437349822 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f873c0b7390 con 0x7f873c0d0f80 2024-04-02T13:22:10.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.029+0000 7f873ad9d640 1 --2- 172.21.15.44:0/1437349822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f873c0d0f80 0x7f873c0b4cc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8734009870 tx=0x7f873402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf46b815bae1e27 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:10.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.029+0000 7f873959a640 1 -- 172.21.15.44:0/1437349822 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8734034070 con 0x7f873c0d0f80 2024-04-02T13:22:10.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.029+0000 7f873959a640 1 -- 172.21.15.44:0/1437349822 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8734030840 con 0x7f873c0d0f80 2024-04-02T13:22:10.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.029+0000 7f873959a640 1 -- 172.21.15.44:0/1437349822 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8734030c10 con 0x7f873c0d0f80 2024-04-02T13:22:10.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.030+0000 7f8740b04640 1 -- 172.21.15.44:0/1437349822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f873c0d0f80 msgr2=0x7f873c0b4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.030+0000 7f8740b04640 1 --2- 172.21.15.44:0/1437349822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f873c0d0f80 0x7f873c0b4cc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8734009870 tx=0x7f873402b980 comp rx=0 tx=0).stop 2024-04-02T13:22:10.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.030+0000 7f8740b04640 1 -- 172.21.15.44:0/1437349822 shutdown_connections 2024-04-02T13:22:10.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.030+0000 7f8740b04640 1 --2- 172.21.15.44:0/1437349822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f873c0d0f80 0x7f873c0b4cc0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.030+0000 7f8740b04640 1 --2- 172.21.15.44:0/1437349822 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f873c15e520 0x7f873c0d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.030+0000 7f8740b04640 1 -- 172.21.15.44:0/1437349822 >> 172.21.15.44:0/1437349822 conn(0x7f873c09d390 msgr2=0x7f873c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:10.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.030+0000 7f8740b04640 1 -- 172.21.15.44:0/1437349822 shutdown_connections 2024-04-02T13:22:10.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.030+0000 7f8740b04640 1 -- 172.21.15.44:0/1437349822 wait complete. 2024-04-02T13:22:10.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.031+0000 7f8740b04640 1 Processor -- start 2024-04-02T13:22:10.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.031+0000 7f8740b04640 1 -- start start 2024-04-02T13:22:10.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f8740b04640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f873c0d0f80 0x7f873c148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f8740b04640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f873c15e140 0x7f873c0ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f8740b04640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f873c15e520 0x7f873c0cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f8740b04640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f873c0d4ee0 con 0x7f873c0d0f80 2024-04-02T13:22:10.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f8740b04640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f873c0b7950 con 0x7f873c15e520 2024-04-02T13:22:10.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f8740b04640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f873c0d2d40 con 0x7f873c15e140 2024-04-02T13:22:10.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f873a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f873c0d0f80 0x7f873c148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f873a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f873c0d0f80 0x7f873c148ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43046/0 (socket says 172.21.15.44:43046) 2024-04-02T13:22:10.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f873a59c640 1 -- 172.21.15.44:0/3872407884 learned_addr learned my addr 172.21.15.44:0/3872407884 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:10.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f8739d9b640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f873c15e140 0x7f873c0ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f873ad9d640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f873c15e520 0x7f873c0cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f873a59c640 1 -- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f873c15e140 msgr2=0x7f873c0ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f873a59c640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f873c15e140 0x7f873c0ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f873a59c640 1 -- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f873c15e520 msgr2=0x7f873c0cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f873a59c640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f873c15e520 0x7f873c0cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.032+0000 7f873a59c640 1 -- 172.21.15.44:0/3872407884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f873c0c9580 con 0x7f873c0d0f80 2024-04-02T13:22:10.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.033+0000 7f873a59c640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f873c0d0f80 0x7f873c148ff0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f872c00c790 tx=0x7f872c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:10.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.033+0000 7f8732ffd640 1 -- 172.21.15.44:0/3872407884 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c016020 con 0x7f873c0d0f80 2024-04-02T13:22:10.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.033+0000 7f8732ffd640 1 -- 172.21.15.44:0/3872407884 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f872c006d50 con 0x7f873c0d0f80 2024-04-02T13:22:10.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.033+0000 7f8740b04640 1 -- 172.21.15.44:0/3872407884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f873c0c9860 con 0x7f873c0d0f80 2024-04-02T13:22:10.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.033+0000 7f8732ffd640 1 -- 172.21.15.44:0/3872407884 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c003bb0 con 0x7f873c0d0f80 2024-04-02T13:22:10.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.033+0000 7f8740b04640 1 -- 172.21.15.44:0/3872407884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f873c0c9e20 con 0x7f873c0d0f80 2024-04-02T13:22:10.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.034+0000 7f8740b04640 1 -- 172.21.15.44:0/3872407884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f873c0684f0 con 0x7f873c0d0f80 2024-04-02T13:22:10.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.034+0000 7f8732ffd640 1 -- 172.21.15.44:0/3872407884 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f872c00a850 con 0x7f873c0d0f80 2024-04-02T13:22:10.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.035+0000 7f8732ffd640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f871c06c8f0 0x7f871c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.035+0000 7f8732ffd640 1 -- 172.21.15.44:0/3872407884 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f872c08b840 con 0x7f873c0d0f80 2024-04-02T13:22:10.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.035+0000 7f8739d9b640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f871c06c8f0 0x7f871c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.035+0000 7f8739d9b640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f871c06c8f0 0x7f871c06edb0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f873c1f7650 tx=0x7f8724008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:10.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.039+0000 7f8732ffd640 1 -- 172.21.15.44:0/3872407884 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f872c020020 con 0x7f873c0d0f80 2024-04-02T13:22:10.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.261+0000 7f8740b04640 1 -- 172.21.15.44:0/3872407884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f873c0d0080 con 0x7f873c0d0f80 2024-04-02T13:22:10.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.261+0000 7f8732ffd640 1 -- 172.21.15.44:0/3872407884 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5181 (secure 0 0 0) 0x7f872c059500 con 0x7f873c0d0f80 2024-04-02T13:22:10.263 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:10.263 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3050514527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3050514527},{"type":"v1","addr":"172.21.15.69:6836","nonce":3050514527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4973,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1949024012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1949024012},{"type":"v1","addr":"172.21.15.69:6837","nonce":1949024012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/2817873198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2817873198},{"type":"v1","addr":"172.21.15.44:6834","nonce":2817873198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:21:43.215796+0000","modified":"2024-04-02T13:21:46.248294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/1969465285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":1969465285},{"type":"v1","addr":"172.21.15.44:6835","nonce":1969465285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":2}]} 2024-04-02T13:22:10.263 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 18 2024-04-02T13:22:10.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.263+0000 7f8740b04640 1 -- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f871c06c8f0 msgr2=0x7f871c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.263+0000 7f8740b04640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f871c06c8f0 0x7f871c06edb0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f873c1f7650 tx=0x7f8724008040 comp rx=0 tx=0).stop 2024-04-02T13:22:10.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.263+0000 7f8740b04640 1 -- 172.21.15.44:0/3872407884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f873c0d0f80 msgr2=0x7f873c148ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.263+0000 7f8740b04640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f873c0d0f80 0x7f873c148ff0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f872c00c790 tx=0x7f872c00cc50 comp rx=0 tx=0).stop 2024-04-02T13:22:10.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.264+0000 7f8740b04640 1 -- 172.21.15.44:0/3872407884 shutdown_connections 2024-04-02T13:22:10.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.264+0000 7f8740b04640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f871c06c8f0 0x7f871c06edb0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.264+0000 7f8740b04640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f873c15e520 0x7f873c0cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.264+0000 7f8740b04640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f873c15e140 0x7f873c0ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.264+0000 7f8740b04640 1 --2- 172.21.15.44:0/3872407884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f873c0d0f80 0x7f873c148ff0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.264+0000 7f8740b04640 1 -- 172.21.15.44:0/3872407884 >> 172.21.15.44:0/3872407884 conn(0x7f873c09d390 msgr2=0x7f873c1613a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:10.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.264+0000 7f8740b04640 1 -- 172.21.15.44:0/3872407884 shutdown_connections 2024-04-02T13:22:10.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.264+0000 7f8740b04640 1 -- 172.21.15.44:0/3872407884 wait complete. 2024-04-02T13:22:10.276 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:22:10.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.375+0000 7ff6e4df2640 1 Processor -- start 2024-04-02T13:22:10.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.375+0000 7ff6e4df2640 1 -- start start 2024-04-02T13:22:10.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.376+0000 7ff6e4df2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6e00cece0 0x7ff6e00cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.376+0000 7ff6e4df2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff6e006ece0 0x7ff6e00acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.376+0000 7ff6e4df2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6e00cf600 con 0x7ff6e00d09f0 2024-04-02T13:22:10.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.376+0000 7ff6e4df2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff6e006d9e0 con 0x7ff6e006ece0 2024-04-02T13:22:10.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.376+0000 7ff6e4df2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff6e006dc40 con 0x7ff6e00cece0 2024-04-02T13:22:10.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.376+0000 7ff6ded9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff6e006ece0 0x7ff6e00acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.376+0000 7ff6ded9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff6e006ece0 0x7ff6e00acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43062/0 (socket says 172.21.15.44:43062) 2024-04-02T13:22:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.376+0000 7ff6ded9d640 1 -- 172.21.15.44:0/2471018366 learned_addr learned my addr 172.21.15.44:0/2471018366 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.376+0000 7ff6de59c640 1 --2- 172.21.15.44:0/2471018366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6e00cece0 0x7ff6e00cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.376+0000 7ff6ded9d640 1 -- 172.21.15.44:0/2471018366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6e00cece0 msgr2=0x7ff6e00cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.377+0000 7ff6ded9d640 1 --2- 172.21.15.44:0/2471018366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6e00cece0 0x7ff6e00cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.377+0000 7ff6ded9d640 1 -- 172.21.15.44:0/2471018366 >> v1:172.21.15.69:6789/0 conn(0x7ff6e00d09f0 legacy=0x7ff6e006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.377+0000 7ff6ded9d640 1 -- 172.21.15.44:0/2471018366 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e006df20 con 0x7ff6e006ece0 2024-04-02T13:22:10.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.377+0000 7ff6ded9d640 1 --2- 172.21.15.44:0/2471018366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e006ece0 0x7ff6e00acd50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000d180 tx=0x7ff6d002e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d8663718dd5f8f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:10.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.377+0000 7ff6dd59a640 1 -- 172.21.15.44:0/2471018366 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d002ea60 con 0x7ff6e006ece0 2024-04-02T13:22:10.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.377+0000 7ff6dd59a640 1 -- 172.21.15.44:0/2471018366 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6d002ec00 con 0x7ff6e006ece0 2024-04-02T13:22:10.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.377+0000 7ff6dd59a640 1 -- 172.21.15.44:0/2471018366 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0009600 con 0x7ff6e006ece0 2024-04-02T13:22:10.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.378+0000 7ff6e4df2640 1 -- 172.21.15.44:0/2471018366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e006ece0 msgr2=0x7ff6e00acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.378+0000 7ff6e4df2640 1 --2- 172.21.15.44:0/2471018366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e006ece0 0x7ff6e00acd50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000d180 tx=0x7ff6d002e0c0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.378+0000 7ff6e4df2640 1 -- 172.21.15.44:0/2471018366 shutdown_connections 2024-04-02T13:22:10.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.378+0000 7ff6e4df2640 1 --2- 172.21.15.44:0/2471018366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e006ece0 0x7ff6e00acd50 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.378+0000 7ff6e4df2640 1 --2- 172.21.15.44:0/2471018366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6e00cece0 0x7ff6e00cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.378+0000 7ff6e4df2640 1 -- 172.21.15.44:0/2471018366 >> 172.21.15.44:0/2471018366 conn(0x7ff6e00cfcd0 msgr2=0x7ff6e00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:10.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.378+0000 7ff6e4df2640 1 -- 172.21.15.44:0/2471018366 shutdown_connections 2024-04-02T13:22:10.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.378+0000 7ff6e4df2640 1 -- 172.21.15.44:0/2471018366 wait complete. 2024-04-02T13:22:10.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.379+0000 7ff6e4df2640 1 Processor -- start 2024-04-02T13:22:10.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.379+0000 7ff6e4df2640 1 -- start start 2024-04-02T13:22:10.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.380+0000 7ff6e4df2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6e006ece0 0x7ff6e01ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.380+0000 7ff6e4df2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e00cece0 0x7ff6e00b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.380+0000 7ff6e4df2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e00d09f0 0x7ff6e00b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.380+0000 7ff6e4df2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e006e7c0 con 0x7ff6e00d09f0 2024-04-02T13:22:10.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.380+0000 7ff6e4df2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e006e4e0 con 0x7ff6e00cece0 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.380+0000 7ff6e4df2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff6e015b220 con 0x7ff6e006ece0 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.380+0000 7ff6ded9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e00d09f0 0x7ff6e00b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.380+0000 7ff6ded9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e00d09f0 0x7ff6e00b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43070/0 (socket says 172.21.15.44:43070) 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.380+0000 7ff6ded9d640 1 -- 172.21.15.44:0/259588010 learned_addr learned my addr 172.21.15.44:0/259588010 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.380+0000 7ff6de59c640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6e006ece0 0x7ff6e01ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.380+0000 7ff6ddd9b640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e00cece0 0x7ff6e00b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.381+0000 7ff6ded9d640 1 -- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6e006ece0 msgr2=0x7ff6e01ef7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.381+0000 7ff6ded9d640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6e006ece0 0x7ff6e01ef7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.381+0000 7ff6ded9d640 1 -- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e00cece0 msgr2=0x7ff6e00b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.381+0000 7ff6ded9d640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e00cece0 0x7ff6e00b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.381+0000 7ff6ded9d640 1 -- 172.21.15.44:0/259588010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e009d830 con 0x7ff6e00d09f0 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.381+0000 7ff6ddd9b640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e00cece0 0x7ff6e00b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.381+0000 7ff6ded9d640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e00d09f0 0x7ff6e00b12f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0038040 tx=0x7ff6d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:10.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.381+0000 7ff6d6ffd640 1 -- 172.21.15.44:0/259588010 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0045020 con 0x7ff6e00d09f0 2024-04-02T13:22:10.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.381+0000 7ff6d6ffd640 1 -- 172.21.15.44:0/259588010 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6d0040660 con 0x7ff6e00d09f0 2024-04-02T13:22:10.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.381+0000 7ff6e4df2640 1 -- 172.21.15.44:0/259588010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6e009c790 con 0x7ff6e00d09f0 2024-04-02T13:22:10.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.381+0000 7ff6d6ffd640 1 -- 172.21.15.44:0/259588010 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d000b720 con 0x7ff6e00d09f0 2024-04-02T13:22:10.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.381+0000 7ff6e4df2640 1 -- 172.21.15.44:0/259588010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e009cd50 con 0x7ff6e00d09f0 2024-04-02T13:22:10.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.382+0000 7ff6e4df2640 1 -- 172.21.15.44:0/259588010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6e00684f0 con 0x7ff6e00d09f0 2024-04-02T13:22:10.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.382+0000 7ff6d6ffd640 1 -- 172.21.15.44:0/259588010 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff6d0048020 con 0x7ff6e00d09f0 2024-04-02T13:22:10.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.383+0000 7ff6d6ffd640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff6c006c8f0 0x7ff6c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.383+0000 7ff6d6ffd640 1 -- 172.21.15.44:0/259588010 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7ff6d00b08f0 con 0x7ff6e00d09f0 2024-04-02T13:22:10.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.383+0000 7ff6de59c640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff6c006c8f0 0x7ff6c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.383+0000 7ff6de59c640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff6c006c8f0 0x7ff6c006edb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800aac0 tx=0x7ff6d8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:10.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.387+0000 7ff6d6ffd640 1 -- 172.21.15.44:0/259588010 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff6d004f030 con 0x7ff6e00d09f0 2024-04-02T13:22:10.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.573+0000 7ff6e4df2640 1 -- 172.21.15.44:0/259588010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff6e015a290 con 0x7ff6e00d09f0 2024-04-02T13:22:10.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.574+0000 7ff6d6ffd640 1 -- 172.21.15.44:0/259588010 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+16943 (secure 0 0 0) 0x7ff6d007eae0 con 0x7ff6e00d09f0 2024-04-02T13:22:10.575 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:10.576 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":41,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:22:08.943287+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:21:41.647497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-02T13:21:42.168299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:22:10.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.575+0000 7ff6e4df2640 1 -- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff6c006c8f0 msgr2=0x7ff6c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.575+0000 7ff6e4df2640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff6c006c8f0 0x7ff6c006edb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800aac0 tx=0x7ff6d8008040 comp rx=0 tx=0).stop 2024-04-02T13:22:10.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.575+0000 7ff6e4df2640 1 -- 172.21.15.44:0/259588010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e00d09f0 msgr2=0x7ff6e00b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.575+0000 7ff6e4df2640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e00d09f0 0x7ff6e00b12f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0038040 tx=0x7ff6d0004a40 comp rx=0 tx=0).stop 2024-04-02T13:22:10.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.576+0000 7ff6e4df2640 1 -- 172.21.15.44:0/259588010 shutdown_connections 2024-04-02T13:22:10.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.576+0000 7ff6e4df2640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff6c006c8f0 0x7ff6c006edb0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.576+0000 7ff6e4df2640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e00d09f0 0x7ff6e00b12f0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.576+0000 7ff6e4df2640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e00cece0 0x7ff6e00b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.576+0000 7ff6e4df2640 1 --2- 172.21.15.44:0/259588010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6e006ece0 0x7ff6e01ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.576+0000 7ff6e4df2640 1 -- 172.21.15.44:0/259588010 >> 172.21.15.44:0/259588010 conn(0x7ff6e00cfcd0 msgr2=0x7ff6e0071c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:10.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.576+0000 7ff6e4df2640 1 -- 172.21.15.44:0/259588010 shutdown_connections 2024-04-02T13:22:10.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.576+0000 7ff6e4df2640 1 -- 172.21.15.44:0/259588010 wait complete. 2024-04-02T13:22:10.588 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:22:10.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.689+0000 7f8ed7055640 1 Processor -- start 2024-04-02T13:22:10.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.689+0000 7f8ed7055640 1 -- start start 2024-04-02T13:22:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.689+0000 7f8ed7055640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed015e550 0x7f8ed006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.689+0000 7f8ed7055640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8ed006d530 0x7f8ed015e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.689+0000 7f8ed7055640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ed015fb50 con 0x7f8ed015ee70 2024-04-02T13:22:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.689+0000 7f8ed7055640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8ed006d970 con 0x7f8ed006d530 2024-04-02T13:22:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.689+0000 7f8ed7055640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed015f300 con 0x7f8ed015e550 2024-04-02T13:22:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.690+0000 7f8ecffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8ed006d530 0x7f8ed015e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.690+0000 7f8ed4df1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed015e550 0x7f8ed006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.690+0000 7f8ed55f2640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8ed015ee70 0x7f8ed00ce6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:60932/0 (socket says 172.21.15.44:60932) 2024-04-02T13:22:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.690+0000 7f8ed55f2640 1 -- 172.21.15.44:0/3948995740 learned_addr learned my addr 172.21.15.44:0/3948995740 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.690+0000 7f8ecf7fe640 1 -- 172.21.15.44:0/3948995740 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2416289506 0 0) 0x7f8ed015fb50 con 0x7f8ed015ee70 2024-04-02T13:22:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.690+0000 7f8ecf7fe640 1 -- 172.21.15.44:0/3948995740 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ec0003520 con 0x7f8ed015ee70 2024-04-02T13:22:10.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.690+0000 7f8ed4df1640 1 -- 172.21.15.44:0/3948995740 >> v2:172.21.15.69:3300/0 conn(0x7f8ed006d530 msgr2=0x7f8ed015e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:10.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.690+0000 7f8ed4df1640 1 --2- 172.21.15.44:0/3948995740 >> v2:172.21.15.69:3300/0 conn(0x7f8ed006d530 0x7f8ed015e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.690+0000 7f8ed4df1640 1 -- 172.21.15.44:0/3948995740 >> v1:172.21.15.69:6789/0 conn(0x7f8ed015ee70 legacy=0x7f8ed00ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.690+0000 7f8ed4df1640 1 -- 172.21.15.44:0/3948995740 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed00d1ac0 con 0x7f8ed015e550 2024-04-02T13:22:10.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.690+0000 7f8ecffff640 1 --2- 172.21.15.44:0/3948995740 >> v2:172.21.15.69:3300/0 conn(0x7f8ed006d530 0x7f8ed015e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:22:10.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.691+0000 7f8ed4df1640 1 --2- 172.21.15.44:0/3948995740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed015e550 0x7f8ed006cff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40097a0 tx=0x7f8ec402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=593578b5263b2712 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:10.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.691+0000 7f8ecf7fe640 1 -- 172.21.15.44:0/3948995740 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4034070 con 0x7f8ed015e550 2024-04-02T13:22:10.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.691+0000 7f8ecf7fe640 1 -- 172.21.15.44:0/3948995740 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ec4030610 con 0x7f8ed015e550 2024-04-02T13:22:10.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.691+0000 7f8ecf7fe640 1 -- 172.21.15.44:0/3948995740 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec40309c0 con 0x7f8ed015e550 2024-04-02T13:22:10.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.692+0000 7f8ed7055640 1 -- 172.21.15.44:0/3948995740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed015e550 msgr2=0x7f8ed006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.692+0000 7f8ed7055640 1 --2- 172.21.15.44:0/3948995740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed015e550 0x7f8ed006cff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40097a0 tx=0x7f8ec402b960 comp rx=0 tx=0).stop 2024-04-02T13:22:10.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.692+0000 7f8ed7055640 1 -- 172.21.15.44:0/3948995740 shutdown_connections 2024-04-02T13:22:10.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.692+0000 7f8ed7055640 1 --2- 172.21.15.44:0/3948995740 >> v2:172.21.15.69:3300/0 conn(0x7f8ed006d530 0x7f8ed015e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.692+0000 7f8ed7055640 1 --2- 172.21.15.44:0/3948995740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed015e550 0x7f8ed006cff0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.692+0000 7f8ed7055640 1 -- 172.21.15.44:0/3948995740 >> 172.21.15.44:0/3948995740 conn(0x7f8ed009aa20 msgr2=0x7f8ed009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:10.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.692+0000 7f8ed7055640 1 -- 172.21.15.44:0/3948995740 shutdown_connections 2024-04-02T13:22:10.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.692+0000 7f8ed7055640 1 -- 172.21.15.44:0/3948995740 wait complete. 2024-04-02T13:22:10.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.693+0000 7f8ed7055640 1 Processor -- start 2024-04-02T13:22:10.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.693+0000 7f8ed7055640 1 -- start start 2024-04-02T13:22:10.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed7055640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ed006d530 0x7f8ed00cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed7055640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed015e550 0x7f8ed009f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed7055640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed015ee70 0x7f8ed00affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed7055640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed015f870 con 0x7f8ed006d530 2024-04-02T13:22:10.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed7055640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed015f590 con 0x7f8ed015e550 2024-04-02T13:22:10.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed7055640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed00d02b0 con 0x7f8ed015ee70 2024-04-02T13:22:10.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed4df1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ed006d530 0x7f8ed00cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed4df1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ed006d530 0x7f8ed00cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43084/0 (socket says 172.21.15.44:43084) 2024-04-02T13:22:10.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed4df1640 1 -- 172.21.15.44:0/3011308518 learned_addr learned my addr 172.21.15.44:0/3011308518 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:10.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ecffff640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed015e550 0x7f8ed009f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed55f2640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed015ee70 0x7f8ed00affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed4df1640 1 -- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed015ee70 msgr2=0x7f8ed00affa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed4df1640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed015ee70 0x7f8ed00affa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed4df1640 1 -- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed015e550 msgr2=0x7f8ed009f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed4df1640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed015e550 0x7f8ed009f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.694+0000 7f8ed4df1640 1 -- 172.21.15.44:0/3011308518 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed015d240 con 0x7f8ed006d530 2024-04-02T13:22:10.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.695+0000 7f8ed4df1640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ed006d530 0x7f8ed00cb630 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8ec402bf70 tx=0x7f8ec40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:10.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.695+0000 7f8ecd7fa640 1 -- 172.21.15.44:0/3011308518 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4034040 con 0x7f8ed006d530 2024-04-02T13:22:10.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.695+0000 7f8ecd7fa640 1 -- 172.21.15.44:0/3011308518 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ec403aaa0 con 0x7f8ed006d530 2024-04-02T13:22:10.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.695+0000 7f8ed7055640 1 -- 172.21.15.44:0/3011308518 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed00ca590 con 0x7f8ed006d530 2024-04-02T13:22:10.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.695+0000 7f8ecd7fa640 1 -- 172.21.15.44:0/3011308518 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4006e10 con 0x7f8ed006d530 2024-04-02T13:22:10.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.695+0000 7f8ed7055640 1 -- 172.21.15.44:0/3011308518 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed00cab50 con 0x7f8ed006d530 2024-04-02T13:22:10.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.696+0000 7f8ed7055640 1 -- 172.21.15.44:0/3011308518 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ed00684f0 con 0x7f8ed006d530 2024-04-02T13:22:10.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.696+0000 7f8ecd7fa640 1 -- 172.21.15.44:0/3011308518 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8ec4037070 con 0x7f8ed006d530 2024-04-02T13:22:10.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.697+0000 7f8ecd7fa640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8eb406c920 0x7f8eb406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:10.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.697+0000 7f8ecd7fa640 1 -- 172.21.15.44:0/3011308518 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f8ec40ad890 con 0x7f8ed006d530 2024-04-02T13:22:10.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.697+0000 7f8ecffff640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8eb406c920 0x7f8eb406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:10.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.697+0000 7f8ecffff640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8eb406c920 0x7f8eb406ede0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009560 tx=0x7f8ebc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:10.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.701+0000 7f8ecd7fa640 1 -- 172.21.15.44:0/3011308518 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8ec403f070 con 0x7f8ed006d530 2024-04-02T13:22:10.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.927+0000 7f8ed7055640 1 -- 172.21.15.44:0/3011308518 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f8ed00cd6b0 con 0x7f8ed006d530 2024-04-02T13:22:10.929 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:22:10.927+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:22:10.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.929+0000 7f8ecd7fa640 1 -- 172.21.15.44:0/3011308518 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v19) ==== 161+0+0 (secure 0 0 0) 0x7f8ec407b550 con 0x7f8ed006d530 2024-04-02T13:22:10.930 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:22:10.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.932+0000 7f8ed7055640 1 -- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8eb406c920 msgr2=0x7f8eb406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.932+0000 7f8ed7055640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8eb406c920 0x7f8eb406ede0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009560 tx=0x7f8ebc009190 comp rx=0 tx=0).stop 2024-04-02T13:22:10.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.932+0000 7f8ed7055640 1 -- 172.21.15.44:0/3011308518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ed006d530 msgr2=0x7f8ed00cb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:10.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.932+0000 7f8ed7055640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ed006d530 0x7f8ed00cb630 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8ec402bf70 tx=0x7f8ec40046c0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.932+0000 7f8ed7055640 1 -- 172.21.15.44:0/3011308518 shutdown_connections 2024-04-02T13:22:10.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.932+0000 7f8ed7055640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8eb406c920 0x7f8eb406ede0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.932+0000 7f8ed7055640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed015ee70 0x7f8ed00affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.932+0000 7f8ed7055640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed015e550 0x7f8ed009f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.932+0000 7f8ed7055640 1 --2- 172.21.15.44:0/3011308518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ed006d530 0x7f8ed00cb630 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:10.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.932+0000 7f8ed7055640 1 -- 172.21.15.44:0/3011308518 >> 172.21.15.44:0/3011308518 conn(0x7f8ed009aa20 msgr2=0x7f8ed00cd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:10.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.932+0000 7f8ed7055640 1 -- 172.21.15.44:0/3011308518 shutdown_connections 2024-04-02T13:22:10.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:10.932+0000 7f8ed7055640 1 -- 172.21.15.44:0/3011308518 wait complete. 2024-04-02T13:22:10.936 INFO:tasks.ceph.mds.a.smithi044.stderr: -5694> 2024-04-02T13:21:48.661+0000 7f130a6cc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:22:10.945 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T13:22:10.966 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:22:11.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.048+0000 7fa87f5f6640 1 Processor -- start 2024-04-02T13:22:11.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.048+0000 7fa87f5f6640 1 -- start start 2024-04-02T13:22:11.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.049+0000 7fa87f5f6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa87815e5d0 0x7fa8780d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:11.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.049+0000 7fa87f5f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8780b4180 0x7fa8780b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:11.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.049+0000 7fa87f5f6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8780d1630 con 0x7fa87806cff0 2024-04-02T13:22:11.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.049+0000 7fa87f5f6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa8780d1bf0 con 0x7fa87815e5d0 2024-04-02T13:22:11.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.049+0000 7fa87f5f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa8780d1910 con 0x7fa8780b4180 2024-04-02T13:22:11.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.049+0000 7fa87d392640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa87806cff0 0x7fa87806d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:47166/0 (socket says 172.21.15.44:47166) 2024-04-02T13:22:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.049+0000 7fa87d392640 1 -- 172.21.15.44:0/1095431424 learned_addr learned my addr 172.21.15.44:0/1095431424 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.049+0000 7fa87db93640 1 --2- 172.21.15.44:0/1095431424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8780b4180 0x7fa8780b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.049+0000 7fa87cb91640 1 --2- 172.21.15.44:0/1095431424 >> v2:172.21.15.69:3300/0 conn(0x7fa87815e5d0 0x7fa8780d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.049+0000 7fa86bfff640 1 -- 172.21.15.44:0/1095431424 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2772443331 0 0) 0x7fa8780d1630 con 0x7fa87806cff0 2024-04-02T13:22:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.049+0000 7fa86bfff640 1 -- 172.21.15.44:0/1095431424 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa864003520 con 0x7fa87806cff0 2024-04-02T13:22:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.050+0000 7fa86bfff640 1 -- 172.21.15.44:0/1095431424 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1587108284 0 0) 0x7fa864003520 con 0x7fa87806cff0 2024-04-02T13:22:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.050+0000 7fa87db93640 1 -- 172.21.15.44:0/1095431424 >> v2:172.21.15.69:3300/0 conn(0x7fa87815e5d0 msgr2=0x7fa8780d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.050+0000 7fa87db93640 1 --2- 172.21.15.44:0/1095431424 >> v2:172.21.15.69:3300/0 conn(0x7fa87815e5d0 0x7fa8780d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.050+0000 7fa87db93640 1 -- 172.21.15.44:0/1095431424 >> v1:172.21.15.44:6789/0 conn(0x7fa87806cff0 legacy=0x7fa87806d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.050+0000 7fa87db93640 1 -- 172.21.15.44:0/1095431424 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8780b1540 con 0x7fa8780b4180 2024-04-02T13:22:11.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.050+0000 7fa87db93640 1 --2- 172.21.15.44:0/1095431424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8780b4180 0x7fa8780b1d40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa8700097a0 tx=0x7fa87002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45b76f3584efd413 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:11.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.051+0000 7fa86bfff640 1 -- 172.21.15.44:0/1095431424 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870037070 con 0x7fa8780b4180 2024-04-02T13:22:11.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.051+0000 7fa86bfff640 1 -- 172.21.15.44:0/1095431424 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa870030510 con 0x7fa8780b4180 2024-04-02T13:22:11.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.051+0000 7fa86bfff640 1 -- 172.21.15.44:0/1095431424 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c001670 con 0x7fa8780b4180 2024-04-02T13:22:11.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.051+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1095431424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8780b4180 msgr2=0x7fa8780b1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:11.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.051+0000 7fa87f5f6640 1 --2- 172.21.15.44:0/1095431424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8780b4180 0x7fa8780b1d40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa8700097a0 tx=0x7fa87002b710 comp rx=0 tx=0).stop 2024-04-02T13:22:11.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.051+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1095431424 shutdown_connections 2024-04-02T13:22:11.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.051+0000 7fa87f5f6640 1 --2- 172.21.15.44:0/1095431424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8780b4180 0x7fa8780b1d40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:11.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.051+0000 7fa87f5f6640 1 --2- 172.21.15.44:0/1095431424 >> v2:172.21.15.69:3300/0 conn(0x7fa87815e5d0 0x7fa8780d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:11.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.051+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1095431424 >> 172.21.15.44:0/1095431424 conn(0x7fa87809aa20 msgr2=0x7fa87809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:11.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.051+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1095431424 shutdown_connections 2024-04-02T13:22:11.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.051+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1095431424 wait complete. 2024-04-02T13:22:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.052+0000 7fa87f5f6640 1 Processor -- start 2024-04-02T13:22:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.052+0000 7fa87f5f6640 1 -- start start 2024-04-02T13:22:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.053+0000 7fa87f5f6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa87806cff0 0x7fa8781553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.053+0000 7fa87f5f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8780b4180 0x7fa878156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.053+0000 7fa87f5f6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa87815e5d0 0x7fa878153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.053+0000 7fa87f5f6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa8780d13f0 con 0x7fa87815e5d0 2024-04-02T13:22:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.053+0000 7fa87f5f6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa8780d1270 con 0x7fa87806cff0 2024-04-02T13:22:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.053+0000 7fa87f5f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa8780b2280 con 0x7fa8780b4180 2024-04-02T13:22:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.053+0000 7fa87db93640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa87815e5d0 0x7fa878153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.053+0000 7fa87db93640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa87815e5d0 0x7fa878153820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43106/0 (socket says 172.21.15.44:43106) 2024-04-02T13:22:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.053+0000 7fa87db93640 1 -- 172.21.15.44:0/1393932348 learned_addr learned my addr 172.21.15.44:0/1393932348 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.053+0000 7fa87cb91640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8780b4180 0x7fa878156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.053+0000 7fa87d392640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa87806cff0 0x7fa8781553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.054+0000 7fa87db93640 1 -- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8780b4180 msgr2=0x7fa878156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.054+0000 7fa87db93640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8780b4180 0x7fa878156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.054+0000 7fa87db93640 1 -- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa87806cff0 msgr2=0x7fa8781553e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.054+0000 7fa87db93640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa87806cff0 0x7fa8781553e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.054+0000 7fa87db93640 1 -- 172.21.15.44:0/1393932348 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa878153d60 con 0x7fa87815e5d0 2024-04-02T13:22:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.054+0000 7fa87d392640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa87806cff0 0x7fa8781553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.054+0000 7fa87db93640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa87815e5d0 0x7fa878153820 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa87002b750 tx=0x7fa8700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.054+0000 7fa869ffb640 1 -- 172.21.15.44:0/1393932348 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870042020 con 0x7fa87815e5d0 2024-04-02T13:22:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.054+0000 7fa869ffb640 1 -- 172.21.15.44:0/1393932348 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa870003610 con 0x7fa87815e5d0 2024-04-02T13:22:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.054+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1393932348 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa878156380 con 0x7fa87815e5d0 2024-04-02T13:22:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.054+0000 7fa869ffb640 1 -- 172.21.15.44:0/1393932348 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870037040 con 0x7fa87815e5d0 2024-04-02T13:22:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.054+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1393932348 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa878155dc0 con 0x7fa87815e5d0 2024-04-02T13:22:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.055+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1393932348 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8780684f0 con 0x7fa87815e5d0 2024-04-02T13:22:11.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.056+0000 7fa869ffb640 1 -- 172.21.15.44:0/1393932348 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa870045020 con 0x7fa87815e5d0 2024-04-02T13:22:11.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.056+0000 7fa869ffb640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa85c06c8f0 0x7fa85c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:11.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.056+0000 7fa869ffb640 1 -- 172.21.15.44:0/1393932348 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fa8700ad9a0 con 0x7fa87815e5d0 2024-04-02T13:22:11.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.056+0000 7fa87d392640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa85c06c8f0 0x7fa85c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:11.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.057+0000 7fa87d392640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa85c06c8f0 0x7fa85c06edb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0079b0 tx=0x7fa86c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:11.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.061+0000 7fa869ffb640 1 -- 172.21.15.44:0/1393932348 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa87004c030 con 0x7fa87815e5d0 2024-04-02T13:22:11.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.286+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1393932348 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa87806cdd0 con 0x7fa87815e5d0 2024-04-02T13:22:11.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.963+0000 7fa869ffb640 1 -- 172.21.15.44:0/1393932348 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v20) ==== 104+0+0 (secure 0 0 0) 0x7fa87007bc60 con 0x7fa87815e5d0 2024-04-02T13:22:11.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.966+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa85c06c8f0 msgr2=0x7fa85c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:11.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.966+0000 7fa87f5f6640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa85c06c8f0 0x7fa85c06edb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0079b0 tx=0x7fa86c009040 comp rx=0 tx=0).stop 2024-04-02T13:22:11.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.966+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1393932348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa87815e5d0 msgr2=0x7fa878153820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:11.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.966+0000 7fa87f5f6640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa87815e5d0 0x7fa878153820 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa87002b750 tx=0x7fa8700046c0 comp rx=0 tx=0).stop 2024-04-02T13:22:11.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.966+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1393932348 shutdown_connections 2024-04-02T13:22:11.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.966+0000 7fa87f5f6640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa85c06c8f0 0x7fa85c06edb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:11.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.966+0000 7fa87f5f6640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa87815e5d0 0x7fa878153820 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:11.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.966+0000 7fa87f5f6640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8780b4180 0x7fa878156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:11.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.966+0000 7fa87f5f6640 1 --2- 172.21.15.44:0/1393932348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa87806cff0 0x7fa8781553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:11.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.966+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1393932348 >> 172.21.15.44:0/1393932348 conn(0x7fa87809aa20 msgr2=0x7fa8780ce860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:11.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.966+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1393932348 shutdown_connections 2024-04-02T13:22:11.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:11.966+0000 7fa87f5f6640 1 -- 172.21.15.44:0/1393932348 wait complete. 2024-04-02T13:22:11.982 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T13:22:12.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.133+0000 7fc9d62f2640 1 Processor -- start 2024-04-02T13:22:12.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.133+0000 7fc9d62f2640 1 -- start start 2024-04-02T13:22:12.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.134+0000 7fc9d62f2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc9d015e050 0x7fc9d00cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:12.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.134+0000 7fc9d62f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc9d0162b40 0x7fc9d00b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:12.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.134+0000 7fc9d62f2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9d00b2580 con 0x7fc9d00b2170 2024-04-02T13:22:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.134+0000 7fc9d62f2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc9d00cd280 con 0x7fc9d0162b40 2024-04-02T13:22:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.134+0000 7fc9d62f2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc9d00cd560 con 0x7fc9d015e050 2024-04-02T13:22:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.134+0000 7fc9d488f640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc9d00b2170 0x7fc9d00b3c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:47182/0 (socket says 172.21.15.44:47182) 2024-04-02T13:22:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.134+0000 7fc9d488f640 1 -- 172.21.15.44:0/3293434951 learned_addr learned my addr 172.21.15.44:0/3293434951 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.134+0000 7fc9cf7fe640 1 --2- 172.21.15.44:0/3293434951 >> v2:172.21.15.44:3300/0 conn(0x7fc9d0162b40 0x7fc9d00b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.134+0000 7fc9cffff640 1 --2- 172.21.15.44:0/3293434951 >> v2:172.21.15.69:3300/0 conn(0x7fc9d015e050 0x7fc9d00cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.135+0000 7fc9cf7fe640 1 -- 172.21.15.44:0/3293434951 >> v2:172.21.15.69:3300/0 conn(0x7fc9d015e050 msgr2=0x7fc9d00cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.135+0000 7fc9cf7fe640 1 --2- 172.21.15.44:0/3293434951 >> v2:172.21.15.69:3300/0 conn(0x7fc9d015e050 0x7fc9d00cca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.135+0000 7fc9cf7fe640 1 -- 172.21.15.44:0/3293434951 >> v1:172.21.15.44:6789/0 conn(0x7fc9d00b2170 legacy=0x7fc9d00b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.135+0000 7fc9cf7fe640 1 -- 172.21.15.44:0/3293434951 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d00cd840 con 0x7fc9d0162b40 2024-04-02T13:22:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.135+0000 7fc9ceffd640 1 -- 172.21.15.44:0/3293434951 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1701518273 0 0) 0x7fc9c8001490 con 0x7fc9d00b2170 2024-04-02T13:22:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.135+0000 7fc9cf7fe640 1 --2- 172.21.15.44:0/3293434951 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9d0162b40 0x7fc9d00b1c30 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc9d00b2da0 tx=0x7fc9c402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef6ec0aee2b963f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.135+0000 7fc9ceffd640 1 -- 172.21.15.44:0/3293434951 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c40315a0 con 0x7fc9d0162b40 2024-04-02T13:22:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.135+0000 7fc9ceffd640 1 -- 172.21.15.44:0/3293434951 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9c4037070 con 0x7fc9d0162b40 2024-04-02T13:22:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.135+0000 7fc9ceffd640 1 -- 172.21.15.44:0/3293434951 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4031960 con 0x7fc9d0162b40 2024-04-02T13:22:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.136+0000 7fc9d62f2640 1 -- 172.21.15.44:0/3293434951 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9d0162b40 msgr2=0x7fc9d00b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.136+0000 7fc9d62f2640 1 --2- 172.21.15.44:0/3293434951 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9d0162b40 0x7fc9d00b1c30 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc9d00b2da0 tx=0x7fc9c402b840 comp rx=0 tx=0).stop 2024-04-02T13:22:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.136+0000 7fc9d62f2640 1 -- 172.21.15.44:0/3293434951 shutdown_connections 2024-04-02T13:22:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.136+0000 7fc9d62f2640 1 --2- 172.21.15.44:0/3293434951 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9d0162b40 0x7fc9d00b1c30 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.136+0000 7fc9d62f2640 1 --2- 172.21.15.44:0/3293434951 >> v2:172.21.15.69:3300/0 conn(0x7fc9d015e050 0x7fc9d00cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.136+0000 7fc9d62f2640 1 -- 172.21.15.44:0/3293434951 >> 172.21.15.44:0/3293434951 conn(0x7fc9d006b4d0 msgr2=0x7fc9d006b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.136+0000 7fc9d62f2640 1 -- 172.21.15.44:0/3293434951 shutdown_connections 2024-04-02T13:22:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.136+0000 7fc9d62f2640 1 -- 172.21.15.44:0/3293434951 wait complete. 2024-04-02T13:22:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.137+0000 7fc9d62f2640 1 Processor -- start 2024-04-02T13:22:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.138+0000 7fc9d62f2640 1 -- start start 2024-04-02T13:22:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.138+0000 7fc9d62f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9d00b2170 0x7fc9d0152980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.138+0000 7fc9d62f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9d015e050 0x7fc9d00cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.138+0000 7fc9d62f2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d0162b40 0x7fc9d00b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.138+0000 7fc9d62f2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d015f5c0 con 0x7fc9d00b2170 2024-04-02T13:22:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.138+0000 7fc9d62f2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d015f440 con 0x7fc9d0162b40 2024-04-02T13:22:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.138+0000 7fc9d62f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc9d015f740 con 0x7fc9d015e050 2024-04-02T13:22:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.138+0000 7fc9cffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9d00b2170 0x7fc9d0152980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.138+0000 7fc9cffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9d00b2170 0x7fc9d0152980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43130/0 (socket says 172.21.15.44:43130) 2024-04-02T13:22:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.138+0000 7fc9cffff640 1 -- 172.21.15.44:0/405988910 learned_addr learned my addr 172.21.15.44:0/405988910 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9cf7fe640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9d015e050 0x7fc9d00cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9d488f640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d0162b40 0x7fc9d00b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9cffff640 1 -- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9d015e050 msgr2=0x7fc9d00cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9cffff640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9d015e050 0x7fc9d00cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9cffff640 1 -- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d0162b40 msgr2=0x7fc9d00b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9cffff640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d0162b40 0x7fc9d00b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9cffff640 1 -- 172.21.15.44:0/405988910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d01f7a30 con 0x7fc9d00b2170 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9cf7fe640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9d015e050 0x7fc9d00cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9cffff640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9d00b2170 0x7fc9d0152980 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc9c000d800 tx=0x7fc9c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9ccff9640 1 -- 172.21.15.44:0/405988910 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c000b6c0 con 0x7fc9d00b2170 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9ccff9640 1 -- 172.21.15.44:0/405988910 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9c0019070 con 0x7fc9d00b2170 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9d62f2640 1 -- 172.21.15.44:0/405988910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d01f87b0 con 0x7fc9d00b2170 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9ccff9640 1 -- 172.21.15.44:0/405988910 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c000ba80 con 0x7fc9d00b2170 2024-04-02T13:22:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.139+0000 7fc9d62f2640 1 -- 172.21.15.44:0/405988910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d01f7fb0 con 0x7fc9d00b2170 2024-04-02T13:22:12.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.140+0000 7fc9d62f2640 1 -- 172.21.15.44:0/405988910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9d00684f0 con 0x7fc9d00b2170 2024-04-02T13:22:12.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.141+0000 7fc9ccff9640 1 -- 172.21.15.44:0/405988910 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc9c000bc20 con 0x7fc9d00b2170 2024-04-02T13:22:12.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.141+0000 7fc9ccff9640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9bc06c8f0 0x7fc9bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:12.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.141+0000 7fc9ccff9640 1 -- 172.21.15.44:0/405988910 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fc9c008c240 con 0x7fc9d00b2170 2024-04-02T13:22:12.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.141+0000 7fc9cf7fe640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9bc06c8f0 0x7fc9bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:12.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.142+0000 7fc9cf7fe640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9bc06c8f0 0x7fc9bc06edb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4036000 tx=0x7fc9c403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.146+0000 7fc9ccff9640 1 -- 172.21.15.44:0/405988910 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc9c001c360 con 0x7fc9d00b2170 2024-04-02T13:22:12.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.330+0000 7fc9d62f2640 1 -- 172.21.15.44:0/405988910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc9d00b33a0 con 0x7fc9d00b2170 2024-04-02T13:22:12.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.968+0000 7fc9ccff9640 1 -- 172.21.15.44:0/405988910 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v43) ==== 181+0+0 (secure 0 0 0) 0x7fc9c005a470 con 0x7fc9d00b2170 2024-04-02T13:22:12.970 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' removed 2024-04-02T13:22:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.971+0000 7fc9d62f2640 1 -- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9bc06c8f0 msgr2=0x7fc9bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.971+0000 7fc9d62f2640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9bc06c8f0 0x7fc9bc06edb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4036000 tx=0x7fc9c403d040 comp rx=0 tx=0).stop 2024-04-02T13:22:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.971+0000 7fc9d62f2640 1 -- 172.21.15.44:0/405988910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9d00b2170 msgr2=0x7fc9d0152980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.971+0000 7fc9d62f2640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9d00b2170 0x7fc9d0152980 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc9c000d800 tx=0x7fc9c000dcc0 comp rx=0 tx=0).stop 2024-04-02T13:22:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.972+0000 7fc9d62f2640 1 -- 172.21.15.44:0/405988910 shutdown_connections 2024-04-02T13:22:12.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.972+0000 7fc9d62f2640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9bc06c8f0 0x7fc9bc06edb0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:12.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.972+0000 7fc9d62f2640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d0162b40 0x7fc9d00b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:12.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.972+0000 7fc9d62f2640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9d015e050 0x7fc9d00cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:12.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.972+0000 7fc9d62f2640 1 --2- 172.21.15.44:0/405988910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9d00b2170 0x7fc9d0152980 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:12.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.972+0000 7fc9d62f2640 1 -- 172.21.15.44:0/405988910 >> 172.21.15.44:0/405988910 conn(0x7fc9d006b4d0 msgr2=0x7fc9d006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:12.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.972+0000 7fc9d62f2640 1 -- 172.21.15.44:0/405988910 shutdown_connections 2024-04-02T13:22:12.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:12.972+0000 7fc9d62f2640 1 -- 172.21.15.44:0/405988910 wait complete. 2024-04-02T13:22:12.988 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T13:22:13.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.095+0000 7f1df95a8640 1 Processor -- start 2024-04-02T13:22:13.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.095+0000 7f1df95a8640 1 -- start start 2024-04-02T13:22:13.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.096+0000 7f1df95a8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415e420 0x7f1df40ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:13.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.096+0000 7f1df95a8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1df406c220 con 0x7f1df40cfe10 2024-04-02T13:22:13.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.096+0000 7f1df95a8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1df415f7e0 con 0x7f1df40721f0 2024-04-02T13:22:13.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.096+0000 7f1df95a8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1df415ec60 con 0x7f1df415e420 2024-04-02T13:22:13.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.096+0000 7f1df37fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f1df40cfe10 0x7f1df40cdae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:47192/0 (socket says 172.21.15.44:47192) 2024-04-02T13:22:13.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.096+0000 7f1df37fe640 1 -- 172.21.15.44:0/3115416845 learned_addr learned my addr 172.21.15.44:0/3115416845 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:13.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.096+0000 7f1df2ffd640 1 --2- 172.21.15.44:0/3115416845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415e420 0x7f1df40ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.096+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1482339718 0 0) 0x7f1dec001110 con 0x7f1df40cfe10 2024-04-02T13:22:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.096+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1de0003520 con 0x7f1df40cfe10 2024-04-02T13:22:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.096+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3925739891 0 0) 0x7f1de0003520 con 0x7f1df40cfe10 2024-04-02T13:22:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.096+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415e420 msgr2=0x7f1df40ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.096+0000 7f1df1ffb640 1 --2- 172.21.15.44:0/3115416845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415e420 0x7f1df40ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 >> v1:172.21.15.69:6789/0 conn(0x7f1df40721f0 legacy=0x7f1df40cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df415ef40 con 0x7f1df40cfe10 2024-04-02T13:22:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f1dec001840 con 0x7f1df40cfe10 2024-04-02T13:22:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 >> v1:172.21.15.44:6789/0 conn(0x7f1df40cfe10 legacy=0x7f1df40cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df1ffb640 1 --2- 172.21.15.44:0/3115416845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415e420 0x7f1de00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df1ffb640 1 --2- 172.21.15.44:0/3115416845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de0003f60 0x7f1de0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df1ffb640 1 --2- 172.21.15.44:0/3115416845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1de0004d40 0x7f1de00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1df415ec60 con 0x7f1de0003f60 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1de000bc00 con 0x7f1de0004d40 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1de000be00 con 0x7f1df415e420 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df2ffd640 1 --2- 172.21.15.44:0/3115416845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415e420 0x7f1de00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df27fc640 1 --2- 172.21.15.44:0/3115416845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de0003f60 0x7f1de0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df37fe640 1 --2- 172.21.15.44:0/3115416845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1de0004d40 0x7f1de00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df2ffd640 1 -- 172.21.15.44:0/3115416845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1de0004d40 msgr2=0x7f1de00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df2ffd640 1 --2- 172.21.15.44:0/3115416845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1de0004d40 0x7f1de00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df2ffd640 1 -- 172.21.15.44:0/3115416845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de0003f60 msgr2=0x7f1de0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df2ffd640 1 --2- 172.21.15.44:0/3115416845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de0003f60 0x7f1de0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df2ffd640 1 -- 172.21.15.44:0/3115416845 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de0003520 con 0x7f1df415e420 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.097+0000 7f1df27fc640 1 --2- 172.21.15.44:0/3115416845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de0003f60 0x7f1de0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.098+0000 7f1df2ffd640 1 --2- 172.21.15.44:0/3115416845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415e420 0x7f1de00039e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009fc0 tx=0x7f1de0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeaf0b97118b659d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.098+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de4039070 con 0x7f1df415e420 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.098+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1de402cc60 con 0x7f1df415e420 2024-04-02T13:22:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.098+0000 7f1df1ffb640 1 -- 172.21.15.44:0/3115416845 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de4009c10 con 0x7f1df415e420 2024-04-02T13:22:13.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.099+0000 7f1df95a8640 1 -- 172.21.15.44:0/3115416845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415e420 msgr2=0x7f1de00039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:13.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.099+0000 7f1df95a8640 1 --2- 172.21.15.44:0/3115416845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415e420 0x7f1de00039e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009fc0 tx=0x7f1de0005830 comp rx=0 tx=0).stop 2024-04-02T13:22:13.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.099+0000 7f1df37fe640 1 -- 172.21.15.44:0/3115416845 reap_dead start 2024-04-02T13:22:13.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.099+0000 7f1df95a8640 1 -- 172.21.15.44:0/3115416845 shutdown_connections 2024-04-02T13:22:13.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.099+0000 7f1df95a8640 1 -- 172.21.15.44:0/3115416845 >> 172.21.15.44:0/3115416845 conn(0x7f1df40b2a10 msgr2=0x7f1df40b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:13.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.099+0000 7f1df95a8640 1 -- 172.21.15.44:0/3115416845 shutdown_connections 2024-04-02T13:22:13.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.099+0000 7f1df95a8640 1 -- 172.21.15.44:0/3115416845 wait complete. 2024-04-02T13:22:13.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.100+0000 7f1df95a8640 1 Processor -- start 2024-04-02T13:22:13.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.100+0000 7f1df95a8640 1 -- start start 2024-04-02T13:22:13.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df95a8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1df415e420 0x7f1df4151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:13.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df95a8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df41f8200 0x7f1df4145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:13.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df95a8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1df40b4c80 0x7f1df40b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:13.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df95a8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1df415e800 con 0x7f1df40b4c80 2024-04-02T13:22:13.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df95a8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1df415f500 con 0x7f1df415e420 2024-04-02T13:22:13.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df95a8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1df415e980 con 0x7f1df41f8200 2024-04-02T13:22:13.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1df40b4c80 0x7f1df40b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:13.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1df40b4c80 0x7f1df40b5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43144/0 (socket says 172.21.15.44:43144) 2024-04-02T13:22:13.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df37fe640 1 -- 172.21.15.44:0/2791051185 learned_addr learned my addr 172.21.15.44:0/2791051185 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:13.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df27fc640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df41f8200 0x7f1df4145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df2ffd640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1df415e420 0x7f1df4151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df37fe640 1 -- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df41f8200 msgr2=0x7f1df4145800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df37fe640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df41f8200 0x7f1df4145800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df37fe640 1 -- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1df415e420 msgr2=0x7f1df4151370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df37fe640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1df415e420 0x7f1df4151370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df37fe640 1 -- 172.21.15.44:0/2791051185 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df4145d40 con 0x7f1df40b4c80 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df27fc640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df41f8200 0x7f1df4145800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df37fe640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1df40b4c80 0x7f1df40b5100 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f1df406c420 tx=0x7f1dec007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1deb7fe640 1 -- 172.21.15.44:0/2791051185 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec01a070 con 0x7f1df40b4c80 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1deb7fe640 1 -- 172.21.15.44:0/2791051185 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dec00f600 con 0x7f1df40b4c80 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1deb7fe640 1 -- 172.21.15.44:0/2791051185 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc001370 con 0x7f1df40b4c80 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.101+0000 7f1df95a8640 1 -- 172.21.15.44:0/2791051185 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df41f89a0 con 0x7f1df40b4c80 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.102+0000 7f1df95a8640 1 -- 172.21.15.44:0/2791051185 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df41f8f60 con 0x7f1df40b4c80 2024-04-02T13:22:13.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.102+0000 7f1df95a8640 1 -- 172.21.15.44:0/2791051185 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df40684f0 con 0x7f1df40b4c80 2024-04-02T13:22:13.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.103+0000 7f1deb7fe640 1 -- 172.21.15.44:0/2791051185 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1dec00f9f0 con 0x7f1df40b4c80 2024-04-02T13:22:13.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.103+0000 7f1deb7fe640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1dd406c8f0 0x7f1dd406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:13.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.103+0000 7f1deb7fe640 1 -- 172.21.15.44:0/2791051185 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5778+0+0 (secure 0 0 0) 0x7f1dec08ade0 con 0x7f1df40b4c80 2024-04-02T13:22:13.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.104+0000 7f1df2ffd640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1dd406c8f0 0x7f1dd406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:13.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.104+0000 7f1df2ffd640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1dd406c8f0 0x7f1dd406edb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1de4036000 tx=0x7f1de402e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:13.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.108+0000 7f1deb7fe640 1 -- 172.21.15.44:0/2791051185 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1df4145d40 con 0x7f1df40b4c80 2024-04-02T13:22:13.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.295+0000 7f1df95a8640 1 -- 172.21.15.44:0/2791051185 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1df40b27f0 con 0x7f1df40b4c80 2024-04-02T13:22:13.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.973+0000 7f1deb7fe640 1 -- 172.21.15.44:0/2791051185 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v44) ==== 169+0+0 (secure 0 0 0) 0x7f1df406c220 con 0x7f1df40b4c80 2024-04-02T13:22:13.975 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' removed 2024-04-02T13:22:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.976+0000 7f1df95a8640 1 -- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1dd406c8f0 msgr2=0x7f1dd406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.976+0000 7f1df95a8640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1dd406c8f0 0x7f1dd406edb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1de4036000 tx=0x7f1de402e000 comp rx=0 tx=0).stop 2024-04-02T13:22:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.976+0000 7f1df95a8640 1 -- 172.21.15.44:0/2791051185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1df40b4c80 msgr2=0x7f1df40b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.976+0000 7f1df95a8640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1df40b4c80 0x7f1df40b5100 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f1df406c420 tx=0x7f1dec007430 comp rx=0 tx=0).stop 2024-04-02T13:22:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.976+0000 7f1df95a8640 1 -- 172.21.15.44:0/2791051185 shutdown_connections 2024-04-02T13:22:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.976+0000 7f1df95a8640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1dd406c8f0 0x7f1dd406edb0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.976+0000 7f1df95a8640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1df40b4c80 0x7f1df40b5100 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.976+0000 7f1df95a8640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df41f8200 0x7f1df4145800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.976+0000 7f1df95a8640 1 --2- 172.21.15.44:0/2791051185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1df415e420 0x7f1df4151370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.976+0000 7f1df95a8640 1 -- 172.21.15.44:0/2791051185 >> 172.21.15.44:0/2791051185 conn(0x7f1df40b2a10 msgr2=0x7f1df40b19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.976+0000 7f1df95a8640 1 -- 172.21.15.44:0/2791051185 shutdown_connections 2024-04-02T13:22:13.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:13.976+0000 7f1df95a8640 1 -- 172.21.15.44:0/2791051185 wait complete. 2024-04-02T13:22:13.991 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T13:22:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.099+0000 7f5148f1c640 1 Processor -- start 2024-04-02T13:22:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.099+0000 7f5148f1c640 1 -- start start 2024-04-02T13:22:14.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.100+0000 7f5148f1c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f514415e420 0x7f51440ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.100+0000 7f5148f1c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f514406c220 con 0x7f51440cfe10 2024-04-02T13:22:14.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.100+0000 7f5148f1c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f514415f7e0 con 0x7f51440721f0 2024-04-02T13:22:14.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.100+0000 7f5148f1c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f514415ec60 con 0x7f514415e420 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.100+0000 7f5142d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f51440cfe10 0x7f51440cdae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:47202/0 (socket says 172.21.15.44:47202) 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.100+0000 7f5142d9d640 1 -- 172.21.15.44:0/1274225528 learned_addr learned my addr 172.21.15.44:0/1274225528 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.100+0000 7f514259c640 1 --2- 172.21.15.44:0/1274225528 >> v2:172.21.15.69:3300/0 conn(0x7f514415e420 0x7f51440ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.100+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 731448099 0 0) 0x7f5134001490 con 0x7f51440cfe10 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.100+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f512c003520 con 0x7f51440cfe10 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3639716125 0 0) 0x7f512c003520 con 0x7f51440cfe10 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 >> v2:172.21.15.69:3300/0 conn(0x7f514415e420 msgr2=0x7f51440ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514159a640 1 --2- 172.21.15.44:0/1274225528 >> v2:172.21.15.69:3300/0 conn(0x7f514415e420 0x7f51440ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 >> v1:172.21.15.69:6789/0 conn(0x7f51440721f0 legacy=0x7f51440cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514415ef40 con 0x7f51440cfe10 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f5134001bc0 con 0x7f51440cfe10 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 >> v1:172.21.15.44:6789/0 conn(0x7f51440cfe10 legacy=0x7f51440cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514159a640 1 --2- 172.21.15.44:0/1274225528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512c0039b0 0x7f512c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514159a640 1 --2- 172.21.15.44:0/1274225528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f512c004800 0x7f512c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514159a640 1 --2- 172.21.15.44:0/1274225528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f512c0052e0 0x7f512c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f514415ec60 con 0x7f512c004800 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f512c00c1b0 con 0x7f512c0039b0 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f512c00c3b0 con 0x7f512c0052e0 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f5141d9b640 1 --2- 172.21.15.44:0/1274225528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f512c004800 0x7f512c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f514259c640 1 --2- 172.21.15.44:0/1274225528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512c0039b0 0x7f512c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f5142d9d640 1 --2- 172.21.15.44:0/1274225528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f512c0052e0 0x7f512c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f5141d9b640 1 -- 172.21.15.44:0/1274225528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f512c0052e0 msgr2=0x7f512c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f5141d9b640 1 --2- 172.21.15.44:0/1274225528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f512c0052e0 0x7f512c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f5141d9b640 1 -- 172.21.15.44:0/1274225528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512c0039b0 msgr2=0x7f512c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f5141d9b640 1 --2- 172.21.15.44:0/1274225528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512c0039b0 0x7f512c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f5141d9b640 1 -- 172.21.15.44:0/1274225528 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f512c003520 con 0x7f512c004800 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.101+0000 7f5142d9d640 1 -- 172.21.15.44:0/1274225528 reap_dead start 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.102+0000 7f5141d9b640 1 --2- 172.21.15.44:0/1274225528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f512c004800 0x7f512c004c80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5138001720 tx=0x7f512c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f66fad4e620347e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.102+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138001370 con 0x7f512c004800 2024-04-02T13:22:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.102+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51380052d0 con 0x7f512c004800 2024-04-02T13:22:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.102+0000 7f514159a640 1 -- 172.21.15.44:0/1274225528 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138003830 con 0x7f512c004800 2024-04-02T13:22:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.103+0000 7f5148f1c640 1 -- 172.21.15.44:0/1274225528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f512c004800 msgr2=0x7f512c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.103+0000 7f5148f1c640 1 --2- 172.21.15.44:0/1274225528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f512c004800 0x7f512c004c80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5138001720 tx=0x7f512c005830 comp rx=0 tx=0).stop 2024-04-02T13:22:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.103+0000 7f5148f1c640 1 -- 172.21.15.44:0/1274225528 shutdown_connections 2024-04-02T13:22:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.103+0000 7f5148f1c640 1 --2- 172.21.15.44:0/1274225528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f512c004800 0x7f512c004c80 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.103+0000 7f5148f1c640 1 -- 172.21.15.44:0/1274225528 >> 172.21.15.44:0/1274225528 conn(0x7f51440b2a10 msgr2=0x7f51440b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.103+0000 7f5148f1c640 1 -- 172.21.15.44:0/1274225528 shutdown_connections 2024-04-02T13:22:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.103+0000 7f5148f1c640 1 -- 172.21.15.44:0/1274225528 wait complete. 2024-04-02T13:22:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.104+0000 7f5148f1c640 1 Processor -- start 2024-04-02T13:22:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.104+0000 7f5148f1c640 1 -- start start 2024-04-02T13:22:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5148f1c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512c004800 0x7f51441f5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5148f1c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5144141140 0x7f5144158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5148f1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51441f8100 0x7f51441f8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5148f1c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f514415e800 con 0x7f5144141140 2024-04-02T13:22:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5148f1c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f514415f500 con 0x7f512c004800 2024-04-02T13:22:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5148f1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f514415e980 con 0x7f51441f8100 2024-04-02T13:22:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5141d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5144141140 0x7f5144158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5141d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5144141140 0x7f5144158b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43154/0 (socket says 172.21.15.44:43154) 2024-04-02T13:22:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5141d9b640 1 -- 172.21.15.44:0/787611895 learned_addr learned my addr 172.21.15.44:0/787611895 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5142d9d640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51441f8100 0x7f51441f8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f514259c640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512c004800 0x7f51441f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5141d9b640 1 -- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51441f8100 msgr2=0x7f51441f8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5141d9b640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51441f8100 0x7f51441f8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5141d9b640 1 -- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512c004800 msgr2=0x7f51441f5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5141d9b640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512c004800 0x7f51441f5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.105+0000 7f5141d9b640 1 -- 172.21.15.44:0/787611895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5144159470 con 0x7f5144141140 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.106+0000 7f5141d9b640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5144141140 0x7f5144158b40 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5138005040 tx=0x7f5138001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.106+0000 7f5132ffd640 1 -- 172.21.15.44:0/787611895 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138041070 con 0x7f5144141140 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.106+0000 7f5132ffd640 1 -- 172.21.15.44:0/787611895 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5138005480 con 0x7f5144141140 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.106+0000 7f5148f1c640 1 -- 172.21.15.44:0/787611895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5144159750 con 0x7f5144141140 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.106+0000 7f5132ffd640 1 -- 172.21.15.44:0/787611895 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138007a30 con 0x7f5144141140 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.106+0000 7f5148f1c640 1 -- 172.21.15.44:0/787611895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5144159d10 con 0x7f5144141140 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.107+0000 7f5148f1c640 1 -- 172.21.15.44:0/787611895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51440d0b00 con 0x7f5144141140 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.107+0000 7f5132ffd640 1 -- 172.21.15.44:0/787611895 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5138044070 con 0x7f5144141140 2024-04-02T13:22:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.108+0000 7f5132ffd640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f512406c8f0 0x7f512406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.108+0000 7f5132ffd640 1 -- 172.21.15.44:0/787611895 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5335+0+0 (secure 0 0 0) 0x7f51380ad380 con 0x7f5144141140 2024-04-02T13:22:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.108+0000 7f514259c640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f512406c8f0 0x7f512406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.108+0000 7f514259c640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f512406c8f0 0x7f512406edb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f514406ad00 tx=0x7f513c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:14.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.112+0000 7f5132ffd640 1 -- 172.21.15.44:0/787611895 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f51440d0b00 con 0x7f5144141140 2024-04-02T13:22:14.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.302+0000 7f5148f1c640 1 -- 172.21.15.44:0/787611895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f51440684f0 con 0x7f5144141140 2024-04-02T13:22:14.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.305+0000 7f5132ffd640 1 -- 172.21.15.44:0/787611895 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f513807b480 con 0x7f5144141140 2024-04-02T13:22:14.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.307+0000 7f5148f1c640 1 -- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f512406c8f0 msgr2=0x7f512406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.307+0000 7f5148f1c640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f512406c8f0 0x7f512406edb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f514406ad00 tx=0x7f513c009190 comp rx=0 tx=0).stop 2024-04-02T13:22:14.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.307+0000 7f5148f1c640 1 -- 172.21.15.44:0/787611895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5144141140 msgr2=0x7f5144158b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.307+0000 7f5148f1c640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5144141140 0x7f5144158b40 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5138005040 tx=0x7f5138001120 comp rx=0 tx=0).stop 2024-04-02T13:22:14.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.307+0000 7f5148f1c640 1 -- 172.21.15.44:0/787611895 shutdown_connections 2024-04-02T13:22:14.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.307+0000 7f5148f1c640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f512406c8f0 0x7f512406edb0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.307+0000 7f5148f1c640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51441f8100 0x7f51441f8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.307+0000 7f5148f1c640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5144141140 0x7f5144158b40 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.307+0000 7f5148f1c640 1 --2- 172.21.15.44:0/787611895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512c004800 0x7f51441f5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.307+0000 7f5148f1c640 1 -- 172.21.15.44:0/787611895 >> 172.21.15.44:0/787611895 conn(0x7f51440b2a10 msgr2=0x7f514406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:14.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.307+0000 7f5148f1c640 1 -- 172.21.15.44:0/787611895 shutdown_connections 2024-04-02T13:22:14.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.307+0000 7f5148f1c640 1 -- 172.21.15.44:0/787611895 wait complete. 2024-04-02T13:22:14.320 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2024-04-02T13:22:14.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.419+0000 7f55f0fe4640 1 Processor -- start 2024-04-02T13:22:14.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.419+0000 7f55f0fe4640 1 -- start start 2024-04-02T13:22:14.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.420+0000 7f55f0fe4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f55ec15f040 0x7f55ec168f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.420+0000 7f55f0fe4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55ec15fb50 con 0x7f55ec15e550 2024-04-02T13:22:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.420+0000 7f55f0fe4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55ec0d1ac0 con 0x7f55ec06d700 2024-04-02T13:22:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.420+0000 7f55f0fe4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f55ec06ca90 con 0x7f55ec15f040 2024-04-02T13:22:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.420+0000 7f55ea59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f55ec15e550 0x7f55ec06cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:47216/0 (socket says 172.21.15.44:47216) 2024-04-02T13:22:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.420+0000 7f55ea59c640 1 -- 172.21.15.44:0/2709668134 learned_addr learned my addr 172.21.15.44:0/2709668134 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.420+0000 7f55ead9d640 1 --2- 172.21.15.44:0/2709668134 >> v2:172.21.15.69:3300/0 conn(0x7f55ec15f040 0x7f55ec168f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1384512133 0 0) 0x7f55ec15fb50 con 0x7f55ec15e550 2024-04-02T13:22:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55d8003520 con 0x7f55ec15e550 2024-04-02T13:22:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4144690922 0 0) 0x7f55d8003520 con 0x7f55ec15e550 2024-04-02T13:22:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 >> v2:172.21.15.69:3300/0 conn(0x7f55ec15f040 msgr2=0x7f55ec168f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 --2- 172.21.15.44:0/2709668134 >> v2:172.21.15.69:3300/0 conn(0x7f55ec15f040 0x7f55ec168f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 >> v1:172.21.15.69:6789/0 conn(0x7f55ec06d700 legacy=0x7f55ec15e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ec15f590 con 0x7f55ec15e550 2024-04-02T13:22:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f55e4001670 con 0x7f55ec15e550 2024-04-02T13:22:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 >> v1:172.21.15.44:6789/0 conn(0x7f55ec15e550 legacy=0x7f55ec06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 --2- 172.21.15.44:0/2709668134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55d80039b0 0x7f55d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 --2- 172.21.15.44:0/2709668134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55d8004800 0x7f55d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 --2- 172.21.15.44:0/2709668134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55d80051c0 0x7f55d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec06ca90 con 0x7f55d8004800 2024-04-02T13:22:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f55d800c070 con 0x7f55d80039b0 2024-04-02T13:22:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f55d800c270 con 0x7f55d80051c0 2024-04-02T13:22:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e9d9b640 1 --2- 172.21.15.44:0/2709668134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55d8004800 0x7f55d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55ea59c640 1 --2- 172.21.15.44:0/2709668134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55d80039b0 0x7f55d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55ead9d640 1 --2- 172.21.15.44:0/2709668134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55d80051c0 0x7f55d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e9d9b640 1 -- 172.21.15.44:0/2709668134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55d80051c0 msgr2=0x7f55d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e9d9b640 1 --2- 172.21.15.44:0/2709668134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55d80051c0 0x7f55d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e9d9b640 1 -- 172.21.15.44:0/2709668134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55d80039b0 msgr2=0x7f55d8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e9d9b640 1 --2- 172.21.15.44:0/2709668134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55d80039b0 0x7f55d8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55e9d9b640 1 -- 172.21.15.44:0/2709668134 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d8003520 con 0x7f55d8004800 2024-04-02T13:22:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55ead9d640 1 --2- 172.21.15.44:0/2709668134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55d80051c0 0x7f55d800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.421+0000 7f55ead9d640 1 -- 172.21.15.44:0/2709668134 reap_dead start 2024-04-02T13:22:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.422+0000 7f55e9d9b640 1 --2- 172.21.15.44:0/2709668134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55d8004800 0x7f55d8004c80 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f55d4001750 tx=0x7f55d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26cd8f6e67d1ee4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.422+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d4001370 con 0x7f55d8004800 2024-04-02T13:22:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.422+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55d40052d0 con 0x7f55d8004800 2024-04-02T13:22:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.422+0000 7f55e959a640 1 -- 172.21.15.44:0/2709668134 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d4003830 con 0x7f55d8004800 2024-04-02T13:22:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.423+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2709668134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55d8004800 msgr2=0x7f55d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.423+0000 7f55f0fe4640 1 --2- 172.21.15.44:0/2709668134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55d8004800 0x7f55d8004c80 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f55d4001750 tx=0x7f55d8005830 comp rx=0 tx=0).stop 2024-04-02T13:22:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.423+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2709668134 shutdown_connections 2024-04-02T13:22:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.423+0000 7f55f0fe4640 1 --2- 172.21.15.44:0/2709668134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55d8004800 0x7f55d8004c80 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.423+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2709668134 >> 172.21.15.44:0/2709668134 conn(0x7f55ec09aa20 msgr2=0x7f55ec09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.423+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2709668134 shutdown_connections 2024-04-02T13:22:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.423+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2709668134 wait complete. 2024-04-02T13:22:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.424+0000 7f55f0fe4640 1 Processor -- start 2024-04-02T13:22:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.424+0000 7f55f0fe4640 1 -- start start 2024-04-02T13:22:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55f0fe4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55d8004800 0x7f55ec09bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55f0fe4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55ec09c0d0 0x7f55ec0c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55f0fe4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55ec0c80a0 0x7f55ec09ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55f0fe4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec0d0590 con 0x7f55ec09c0d0 2024-04-02T13:22:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55f0fe4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec0d02b0 con 0x7f55ec0c80a0 2024-04-02T13:22:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55f0fe4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f55ec0cc4b0 con 0x7f55d8004800 2024-04-02T13:22:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55e9d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55ec09c0d0 0x7f55ec0c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55e9d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55ec09c0d0 0x7f55ec0c7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43178/0 (socket says 172.21.15.44:43178) 2024-04-02T13:22:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55e9d9b640 1 -- 172.21.15.44:0/2884069068 learned_addr learned my addr 172.21.15.44:0/2884069068 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55ead9d640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55ec0c80a0 0x7f55ec09ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55ea59c640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55d8004800 0x7f55ec09bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55e9d9b640 1 -- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55d8004800 msgr2=0x7f55ec09bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55e9d9b640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55d8004800 0x7f55ec09bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55e9d9b640 1 -- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55ec0c80a0 msgr2=0x7f55ec09ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55e9d9b640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55ec0c80a0 0x7f55ec09ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55e9d9b640 1 -- 172.21.15.44:0/2884069068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ec09c570 con 0x7f55ec09c0d0 2024-04-02T13:22:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55e9d9b640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55ec09c0d0 0x7f55ec0c7b60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f55d4000c00 tx=0x7f55d4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:14.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.425+0000 7f55e2ffd640 1 -- 172.21.15.44:0/2884069068 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d4041020 con 0x7f55ec09c0d0 2024-04-02T13:22:14.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.426+0000 7f55e2ffd640 1 -- 172.21.15.44:0/2884069068 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55d4005480 con 0x7f55ec09c0d0 2024-04-02T13:22:14.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.426+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2884069068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55ec09c850 con 0x7f55ec09c0d0 2024-04-02T13:22:14.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.426+0000 7f55e2ffd640 1 -- 172.21.15.44:0/2884069068 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d4007a30 con 0x7f55ec09c0d0 2024-04-02T13:22:14.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.426+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2884069068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55ec09e4f0 con 0x7f55ec09c0d0 2024-04-02T13:22:14.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.427+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2884069068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55ec0c84b0 con 0x7f55ec09c0d0 2024-04-02T13:22:14.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.427+0000 7f55e2ffd640 1 -- 172.21.15.44:0/2884069068 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f55d4044020 con 0x7f55ec09c0d0 2024-04-02T13:22:14.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.427+0000 7f55e2ffd640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f55cc06c920 0x7f55cc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:14.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.428+0000 7f55e2ffd640 1 -- 172.21.15.44:0/2884069068 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5335+0+0 (secure 0 0 0) 0x7f55d40ad420 con 0x7f55ec09c0d0 2024-04-02T13:22:14.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.428+0000 7f55ea59c640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f55cc06c920 0x7f55cc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:14.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.428+0000 7f55ea59c640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f55cc06c920 0x7f55cc06ede0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f55e4001050 tx=0x7f55e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:14.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.432+0000 7f55e2ffd640 1 -- 172.21.15.44:0/2884069068 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f55ec0c84b0 con 0x7f55ec09c0d0 2024-04-02T13:22:14.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.697+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2884069068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) -- 0x7f55ec0684f0 con 0x7f55ec09c0d0 2024-04-02T13:22:14.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.975+0000 7f55e2ffd640 1 -- 172.21.15.44:0/2884069068 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v124) ==== 142+0+0 (secure 0 0 0) 0x7f55d404b030 con 0x7f55ec09c0d0 2024-04-02T13:22:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.977+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f55cc06c920 msgr2=0x7f55cc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.977+0000 7f55f0fe4640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f55cc06c920 0x7f55cc06ede0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f55e4001050 tx=0x7f55e4009040 comp rx=0 tx=0).stop 2024-04-02T13:22:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.977+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2884069068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55ec09c0d0 msgr2=0x7f55ec0c7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.977+0000 7f55f0fe4640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55ec09c0d0 0x7f55ec0c7b60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f55d4000c00 tx=0x7f55d4001120 comp rx=0 tx=0).stop 2024-04-02T13:22:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.977+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2884069068 shutdown_connections 2024-04-02T13:22:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.977+0000 7f55f0fe4640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f55cc06c920 0x7f55cc06ede0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.977+0000 7f55f0fe4640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55ec0c80a0 0x7f55ec09ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.977+0000 7f55f0fe4640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55ec09c0d0 0x7f55ec0c7b60 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.977+0000 7f55f0fe4640 1 --2- 172.21.15.44:0/2884069068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55d8004800 0x7f55ec09bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.977+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2884069068 >> 172.21.15.44:0/2884069068 conn(0x7f55ec09aa20 msgr2=0x7f55ec15d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.977+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2884069068 shutdown_connections 2024-04-02T13:22:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:14.977+0000 7f55f0fe4640 1 -- 172.21.15.44:0/2884069068 wait complete. 2024-04-02T13:22:14.990 INFO:tasks.cephfs_test_runner:test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-04-02T13:22:14.990 INFO:tasks.cephfs_test_runner:Starting test: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) That the MDS can repair an inodes backtrace in the data pool 2024-04-02T13:22:14.990 INFO:tasks.cephfs_test_runner:test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-02T13:22:14.990 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:22:15.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.091+0000 7fe8acbcd640 1 Processor -- start 2024-04-02T13:22:15.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.091+0000 7fe8acbcd640 1 -- start start 2024-04-02T13:22:15.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.091+0000 7fe8acbcd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8a815e550 0x7fe8a806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.091+0000 7fe8acbcd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe8a806d530 0x7fe8a815e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.091+0000 7fe8acbcd640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8a815fb50 con 0x7fe8a815ee70 2024-04-02T13:22:15.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.091+0000 7fe8acbcd640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe8a806d970 con 0x7fe8a806d530 2024-04-02T13:22:15.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.091+0000 7fe8acbcd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a815f300 con 0x7fe8a815e550 2024-04-02T13:22:15.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.092+0000 7fe8a659c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8a815e550 0x7fe8a806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.092+0000 7fe89ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe8a806d530 0x7fe8a815e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.092+0000 7fe89ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe8a806d530 0x7fe8a815e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:43806/0 (socket says 172.21.15.44:43806) 2024-04-02T13:22:15.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.092+0000 7fe89ffff640 1 -- 172.21.15.44:0/1611641735 learned_addr learned my addr 172.21.15.44:0/1611641735 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:15.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.092+0000 7fe8a6d9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe8a815ee70 0x7fe8a80ce6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:39002/0 (socket says 172.21.15.44:39002) 2024-04-02T13:22:15.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.092+0000 7fe89ffff640 1 -- 172.21.15.44:0/1611641735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8a815e550 msgr2=0x7fe8a806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:15.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.092+0000 7fe89ffff640 1 --2- 172.21.15.44:0/1611641735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8a815e550 0x7fe8a806cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.092+0000 7fe89ffff640 1 -- 172.21.15.44:0/1611641735 >> v1:172.21.15.69:6789/0 conn(0x7fe8a815ee70 legacy=0x7fe8a80ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.092+0000 7fe89ffff640 1 -- 172.21.15.44:0/1611641735 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a80d1ac0 con 0x7fe8a806d530 2024-04-02T13:22:15.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.093+0000 7fe89ffff640 1 --2- 172.21.15.44:0/1611641735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a806d530 0x7fe8a815e930 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe890009870 tx=0x7fe89002ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=542049e4d726e816 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:15.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.093+0000 7fe8a5d9b640 1 -- 172.21.15.44:0/1611641735 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89002f840 con 0x7fe8a806d530 2024-04-02T13:22:15.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.093+0000 7fe8a5d9b640 1 -- 172.21.15.44:0/1611641735 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe89002f9e0 con 0x7fe8a806d530 2024-04-02T13:22:15.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.093+0000 7fe8a5d9b640 1 -- 172.21.15.44:0/1611641735 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89002fdb0 con 0x7fe8a806d530 2024-04-02T13:22:15.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.094+0000 7fe8acbcd640 1 -- 172.21.15.44:0/1611641735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a806d530 msgr2=0x7fe8a815e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.094+0000 7fe8acbcd640 1 --2- 172.21.15.44:0/1611641735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a806d530 0x7fe8a815e930 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe890009870 tx=0x7fe89002ba20 comp rx=0 tx=0).stop 2024-04-02T13:22:15.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.094+0000 7fe8acbcd640 1 -- 172.21.15.44:0/1611641735 shutdown_connections 2024-04-02T13:22:15.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.094+0000 7fe8acbcd640 1 --2- 172.21.15.44:0/1611641735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a806d530 0x7fe8a815e930 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.094+0000 7fe8acbcd640 1 --2- 172.21.15.44:0/1611641735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8a815e550 0x7fe8a806cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.094+0000 7fe8acbcd640 1 -- 172.21.15.44:0/1611641735 >> 172.21.15.44:0/1611641735 conn(0x7fe8a809aa20 msgr2=0x7fe8a809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:15.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.094+0000 7fe8acbcd640 1 -- 172.21.15.44:0/1611641735 shutdown_connections 2024-04-02T13:22:15.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.094+0000 7fe8acbcd640 1 -- 172.21.15.44:0/1611641735 wait complete. 2024-04-02T13:22:15.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.095+0000 7fe8acbcd640 1 Processor -- start 2024-04-02T13:22:15.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.095+0000 7fe8acbcd640 1 -- start start 2024-04-02T13:22:15.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8acbcd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a806d530 0x7fe8a80b7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8acbcd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a815e550 0x7fe8a809bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8acbcd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8a815ee70 0x7fe8a809d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8acbcd640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a815f870 con 0x7fe8a806d530 2024-04-02T13:22:15.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8acbcd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a815f590 con 0x7fe8a815e550 2024-04-02T13:22:15.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8acbcd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a80d02b0 con 0x7fe8a815ee70 2024-04-02T13:22:15.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8a659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a806d530 0x7fe8a80b7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8a659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a806d530 0x7fe8a80b7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60012/0 (socket says 172.21.15.44:60012) 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8a659c640 1 -- 172.21.15.44:0/3992521828 learned_addr learned my addr 172.21.15.44:0/3992521828 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8a6d9d640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8a815ee70 0x7fe8a809d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe89ffff640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a815e550 0x7fe8a809bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8a659c640 1 -- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8a815ee70 msgr2=0x7fe8a809d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8a659c640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8a815ee70 0x7fe8a809d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8a659c640 1 -- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a815e550 msgr2=0x7fe8a809bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8a659c640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a815e550 0x7fe8a809bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8a659c640 1 -- 172.21.15.44:0/3992521828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a809c2e0 con 0x7fe8a806d530 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe8a659c640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a806d530 0x7fe8a80b7880 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe89800e870 tx=0x7fe89800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.096+0000 7fe89effd640 1 -- 172.21.15.44:0/3992521828 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe898017070 con 0x7fe8a806d530 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.097+0000 7fe89effd640 1 -- 172.21.15.44:0/3992521828 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe89800c940 con 0x7fe8a806d530 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.097+0000 7fe8acbcd640 1 -- 172.21.15.44:0/3992521828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a809c5c0 con 0x7fe8a806d530 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.097+0000 7fe89effd640 1 -- 172.21.15.44:0/3992521828 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89800cd30 con 0x7fe8a806d530 2024-04-02T13:22:15.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.097+0000 7fe8acbcd640 1 -- 172.21.15.44:0/3992521828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a809cb80 con 0x7fe8a806d530 2024-04-02T13:22:15.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.097+0000 7fe8acbcd640 1 -- 172.21.15.44:0/3992521828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a80684f0 con 0x7fe8a806d530 2024-04-02T13:22:15.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.098+0000 7fe89effd640 1 -- 172.21.15.44:0/3992521828 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe89801a070 con 0x7fe8a806d530 2024-04-02T13:22:15.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.098+0000 7fe89effd640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe88806c8f0 0x7fe88806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.098+0000 7fe89effd640 1 -- 172.21.15.44:0/3992521828 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5335+0+0 (secure 0 0 0) 0x7fe89808bb80 con 0x7fe8a806d530 2024-04-02T13:22:15.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.099+0000 7fe89ffff640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe88806c8f0 0x7fe88806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.099+0000 7fe89ffff640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe88806c8f0 0x7fe88806edb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe890036000 tx=0x7fe890008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:15.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.103+0000 7fe89effd640 1 -- 172.21.15.44:0/3992521828 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe898059ad0 con 0x7fe8a806d530 2024-04-02T13:22:15.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.287+0000 7fe8acbcd640 1 -- 172.21.15.44:0/3992521828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe8a80d0580 con 0x7fe8a806d530 2024-04-02T13:22:15.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.288+0000 7fe89effd640 1 -- 172.21.15.44:0/3992521828 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+13245 (secure 0 0 0) 0x7fe89805ddd0 con 0x7fe8a806d530 2024-04-02T13:22:15.290 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:15.290 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":44,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:22:13.969286+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6835/1969465285":"2024-04-03T13:22:10.928093+0000","172.21.15.44:6833/1969465285":"2024-04-03T13:22:10.928093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:22:15.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.290+0000 7fe8acbcd640 1 -- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe88806c8f0 msgr2=0x7fe88806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.290+0000 7fe8acbcd640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe88806c8f0 0x7fe88806edb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe890036000 tx=0x7fe890008030 comp rx=0 tx=0).stop 2024-04-02T13:22:15.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.290+0000 7fe8acbcd640 1 -- 172.21.15.44:0/3992521828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a806d530 msgr2=0x7fe8a80b7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.290+0000 7fe8acbcd640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a806d530 0x7fe8a80b7880 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe89800e870 tx=0x7fe89800ed30 comp rx=0 tx=0).stop 2024-04-02T13:22:15.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.290+0000 7fe8acbcd640 1 -- 172.21.15.44:0/3992521828 shutdown_connections 2024-04-02T13:22:15.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.290+0000 7fe8acbcd640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe88806c8f0 0x7fe88806edb0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.290+0000 7fe8acbcd640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8a815ee70 0x7fe8a809d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.290+0000 7fe8acbcd640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a815e550 0x7fe8a809bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.290+0000 7fe8acbcd640 1 --2- 172.21.15.44:0/3992521828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a806d530 0x7fe8a80b7880 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.290+0000 7fe8acbcd640 1 -- 172.21.15.44:0/3992521828 >> 172.21.15.44:0/3992521828 conn(0x7fe8a809aa20 msgr2=0x7fe8a80cdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:15.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.290+0000 7fe8acbcd640 1 -- 172.21.15.44:0/3992521828 shutdown_connections 2024-04-02T13:22:15.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.290+0000 7fe8acbcd640 1 -- 172.21.15.44:0/3992521828 wait complete. 2024-04-02T13:22:15.303 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:22:15.303 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:22:15.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.405+0000 7f936d8d9640 1 Processor -- start 2024-04-02T13:22:15.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.405+0000 7f936d8d9640 1 -- start start 2024-04-02T13:22:15.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.406+0000 7f936d8d9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f936815e4d0 0x7f93681691a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.406+0000 7f936d8d9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9368070040 con 0x7f93680d2090 2024-04-02T13:22:15.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.406+0000 7f936d8d9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9368070320 con 0x7f93680d00d0 2024-04-02T13:22:15.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.406+0000 7f936d8d9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f936806c5e0 con 0x7f936815e4d0 2024-04-02T13:22:15.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.406+0000 7f9366ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f93680d2090 0x7f936806f580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53304/0 (socket says 172.21.15.44:53304) 2024-04-02T13:22:15.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.406+0000 7f9366ffd640 1 -- 172.21.15.44:0/2050802912 learned_addr learned my addr 172.21.15.44:0/2050802912 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:15.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.406+0000 7f93677fe640 1 --2- 172.21.15.44:0/2050802912 >> v2:172.21.15.69:3300/0 conn(0x7f936815e4d0 0x7f93681691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.406+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1776347311 0 0) 0x7f9368070040 con 0x7f93680d2090 2024-04-02T13:22:15.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9354003520 con 0x7f93680d2090 2024-04-02T13:22:15.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2173645270 0 0) 0x7f9354003520 con 0x7f93680d2090 2024-04-02T13:22:15.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 >> v2:172.21.15.69:3300/0 conn(0x7f936815e4d0 msgr2=0x7f93681691a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 --2- 172.21.15.44:0/2050802912 >> v2:172.21.15.69:3300/0 conn(0x7f936815e4d0 0x7f93681691a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 >> v1:172.21.15.69:6789/0 conn(0x7f93680d00d0 legacy=0x7f936815ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9368070600 con 0x7f93680d2090 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f9360001670 con 0x7f93680d2090 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 >> v1:172.21.15.44:6789/0 conn(0x7f93680d2090 legacy=0x7f936806f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 --2- 172.21.15.44:0/2050802912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93540039b0 0x7f9354003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 --2- 172.21.15.44:0/2050802912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9354004800 0x7f9354004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 --2- 172.21.15.44:0/2050802912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93540051c0 0x7f935400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f936806c5e0 con 0x7f93540039b0 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f935400c090 con 0x7f93540051c0 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f935400c290 con 0x7f9354004800 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9366ffd640 1 --2- 172.21.15.44:0/2050802912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93540039b0 0x7f9354003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f93677fe640 1 --2- 172.21.15.44:0/2050802912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93540051c0 0x7f935400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f93667fc640 1 --2- 172.21.15.44:0/2050802912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9354004800 0x7f9354004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9366ffd640 1 -- 172.21.15.44:0/2050802912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9354004800 msgr2=0x7f9354004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9366ffd640 1 --2- 172.21.15.44:0/2050802912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9354004800 0x7f9354004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9366ffd640 1 -- 172.21.15.44:0/2050802912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93540051c0 msgr2=0x7f935400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9366ffd640 1 --2- 172.21.15.44:0/2050802912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93540051c0 0x7f935400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f9366ffd640 1 -- 172.21.15.44:0/2050802912 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9354003520 con 0x7f93540039b0 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f93677fe640 1 --2- 172.21.15.44:0/2050802912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93540051c0 0x7f935400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.407+0000 7f93677fe640 1 -- 172.21.15.44:0/2050802912 reap_dead start 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.408+0000 7f9366ffd640 1 --2- 172.21.15.44:0/2050802912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93540039b0 0x7f9354003df0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f936000c240 tx=0x7f9354005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d67eb6ded849d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.408+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9360007cf0 con 0x7f93540039b0 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.408+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9360030070 con 0x7f93540039b0 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.408+0000 7f9365ffb640 1 -- 172.21.15.44:0/2050802912 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9360005d30 con 0x7f93540039b0 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.409+0000 7f936d8d9640 1 -- 172.21.15.44:0/2050802912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93540039b0 msgr2=0x7f9354003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.409+0000 7f936d8d9640 1 --2- 172.21.15.44:0/2050802912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93540039b0 0x7f9354003df0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f936000c240 tx=0x7f9354005830 comp rx=0 tx=0).stop 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.409+0000 7f936d8d9640 1 -- 172.21.15.44:0/2050802912 shutdown_connections 2024-04-02T13:22:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.409+0000 7f936d8d9640 1 --2- 172.21.15.44:0/2050802912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93540039b0 0x7f9354003df0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.409+0000 7f936d8d9640 1 -- 172.21.15.44:0/2050802912 >> 172.21.15.44:0/2050802912 conn(0x7f936809aa20 msgr2=0x7f936809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:15.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.409+0000 7f936d8d9640 1 -- 172.21.15.44:0/2050802912 shutdown_connections 2024-04-02T13:22:15.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.409+0000 7f936d8d9640 1 -- 172.21.15.44:0/2050802912 wait complete. 2024-04-02T13:22:15.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.410+0000 7f936d8d9640 1 Processor -- start 2024-04-02T13:22:15.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.410+0000 7f936d8d9640 1 -- start start 2024-04-02T13:22:15.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f936d8d9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93540039b0 0x7f93680b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f936d8d9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9368154f40 0x7f936815feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f936d8d9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9368160b40 0x7f9368160fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f936d8d9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f93680cead0 con 0x7f93540039b0 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f936d8d9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f93680ce950 con 0x7f9368154f40 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f936d8d9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f93680d2470 con 0x7f9368160b40 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f9366ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93540039b0 0x7f93680b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f9366ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93540039b0 0x7f93680b40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60026/0 (socket says 172.21.15.44:60026) 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f9366ffd640 1 -- 172.21.15.44:0/954192835 learned_addr learned my addr 172.21.15.44:0/954192835 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f9366ffd640 1 -- 172.21.15.44:0/954192835 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9368160b40 msgr2=0x7f9368160fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f9366ffd640 1 --2- 172.21.15.44:0/954192835 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9368160b40 0x7f9368160fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f9366ffd640 1 -- 172.21.15.44:0/954192835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9368154f40 msgr2=0x7f936815feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f9366ffd640 1 --2- 172.21.15.44:0/954192835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9368154f40 0x7f936815feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f9366ffd640 1 -- 172.21.15.44:0/954192835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9368153470 con 0x7f93540039b0 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f9366ffd640 1 --2- 172.21.15.44:0/954192835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93540039b0 0x7f93680b40a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f936002c500 tx=0x7f9368070210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f935ffff640 1 -- 172.21.15.44:0/954192835 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9360001c30 con 0x7f93540039b0 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f935ffff640 1 -- 172.21.15.44:0/954192835 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9360004bf0 con 0x7f93540039b0 2024-04-02T13:22:15.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f936d8d9640 1 -- 172.21.15.44:0/954192835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93680b45e0 con 0x7f93540039b0 2024-04-02T13:22:15.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f935ffff640 1 -- 172.21.15.44:0/954192835 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9360007070 con 0x7f93540039b0 2024-04-02T13:22:15.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.411+0000 7f936d8d9640 1 -- 172.21.15.44:0/954192835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93681557e0 con 0x7f93540039b0 2024-04-02T13:22:15.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.413+0000 7f936d8d9640 1 -- 172.21.15.44:0/954192835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93680d1580 con 0x7f93540039b0 2024-04-02T13:22:15.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.413+0000 7f935ffff640 1 -- 172.21.15.44:0/954192835 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9360048020 con 0x7f93540039b0 2024-04-02T13:22:15.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.413+0000 7f935ffff640 1 --2- 172.21.15.44:0/954192835 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f934806c920 0x7f934806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.414+0000 7f935ffff640 1 -- 172.21.15.44:0/954192835 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5335+0+0 (secure 0 0 0) 0x7f93600accf0 con 0x7f93540039b0 2024-04-02T13:22:15.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.414+0000 7f93667fc640 1 --2- 172.21.15.44:0/954192835 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f934806c920 0x7f934806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.414+0000 7f93667fc640 1 --2- 172.21.15.44:0/954192835 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f934806c920 0x7f934806ede0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9350007fd0 tx=0x7f9350009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:15.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.418+0000 7f935ffff640 1 -- 172.21.15.44:0/954192835 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f93680d1580 con 0x7f93540039b0 2024-04-02T13:22:15.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.600+0000 7f936d8d9640 1 -- 172.21.15.44:0/954192835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9368068c00 con 0x7f93540039b0 2024-04-02T13:22:15.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.600+0000 7f935ffff640 1 -- 172.21.15.44:0/954192835 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) ==== 93+0+10 (secure 0 0 0) 0x7f93600b1070 con 0x7f93540039b0 2024-04-02T13:22:15.602 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:22:15.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.603+0000 7f936d8d9640 1 -- 172.21.15.44:0/954192835 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f934806c920 msgr2=0x7f934806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.603+0000 7f936d8d9640 1 --2- 172.21.15.44:0/954192835 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f934806c920 0x7f934806ede0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9350007fd0 tx=0x7f9350009040 comp rx=0 tx=0).stop 2024-04-02T13:22:15.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.603+0000 7f936d8d9640 1 -- 172.21.15.44:0/954192835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93540039b0 msgr2=0x7f93680b40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.603+0000 7f936d8d9640 1 --2- 172.21.15.44:0/954192835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93540039b0 0x7f93680b40a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f936002c500 tx=0x7f9368070210 comp rx=0 tx=0).stop 2024-04-02T13:22:15.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.603+0000 7f936d8d9640 1 -- 172.21.15.44:0/954192835 shutdown_connections 2024-04-02T13:22:15.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.603+0000 7f936d8d9640 1 --2- 172.21.15.44:0/954192835 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f934806c920 0x7f934806ede0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.603+0000 7f936d8d9640 1 --2- 172.21.15.44:0/954192835 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9368160b40 0x7f9368160fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.603+0000 7f936d8d9640 1 --2- 172.21.15.44:0/954192835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9368154f40 0x7f936815feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.603+0000 7f936d8d9640 1 --2- 172.21.15.44:0/954192835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93540039b0 0x7f93680b40a0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.603+0000 7f936d8d9640 1 -- 172.21.15.44:0/954192835 >> 172.21.15.44:0/954192835 conn(0x7f936809aa20 msgr2=0x7f93680b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:15.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.603+0000 7f936d8d9640 1 -- 172.21.15.44:0/954192835 shutdown_connections 2024-04-02T13:22:15.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.603+0000 7f936d8d9640 1 -- 172.21.15.44:0/954192835 wait complete. 2024-04-02T13:22:15.615 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2024-04-02T13:22:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.716+0000 7f0f88042640 1 Processor -- start 2024-04-02T13:22:15.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.716+0000 7f0f88042640 1 -- start start 2024-04-02T13:22:15.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f88042640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f8006d700 0x7f0f8015e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f88042640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f8015fb50 con 0x7f0f8015ee70 2024-04-02T13:22:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f88042640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f8015f320 con 0x7f0f8015e550 2024-04-02T13:22:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f88042640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0f8006ca90 con 0x7f0f8006d700 2024-04-02T13:22:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f865df640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0f8015ee70 0x7f0f800d0ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53306/0 (socket says 172.21.15.44:53306) 2024-04-02T13:22:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f865df640 1 -- 172.21.15.44:0/4244529351 learned_addr learned my addr 172.21.15.44:0/4244529351 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f855dd640 1 --2- 172.21.15.44:0/4244529351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f8006d700 0x7f0f8015e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3206826415 0 0) 0x7f0f8015fb50 con 0x7f0f8015ee70 2024-04-02T13:22:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f6c003520 con 0x7f0f8015ee70 2024-04-02T13:22:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 94953166 0 0) 0x7f0f6c003520 con 0x7f0f8015ee70 2024-04-02T13:22:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f8006d700 msgr2=0x7f0f8015e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 --2- 172.21.15.44:0/4244529351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f8006d700 0x7f0f8015e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 >> v1:172.21.15.69:6789/0 conn(0x7f0f8015e550 legacy=0x7f0f8006cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f8015f590 con 0x7f0f8015ee70 2024-04-02T13:22:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f0f780019f0 con 0x7f0f8015ee70 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 >> v1:172.21.15.44:6789/0 conn(0x7f0f8015ee70 legacy=0x7f0f800d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 --2- 172.21.15.44:0/4244529351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f6c0039e0 0x7f0f6c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 --2- 172.21.15.44:0/4244529351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f8006d700 0x7f0f6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 --2- 172.21.15.44:0/4244529351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f6c004d40 0x7f0f6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8006ca90 con 0x7f0f6c004d40 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c00b9c0 con 0x7f0f6c0039e0 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0f6c00bbc0 con 0x7f0f8006d700 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f865df640 1 --2- 172.21.15.44:0/4244529351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f6c004d40 0x7f0f6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.717+0000 7f0f85dde640 1 --2- 172.21.15.44:0/4244529351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f6c0039e0 0x7f0f6c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.718+0000 7f0f855dd640 1 --2- 172.21.15.44:0/4244529351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f8006d700 0x7f0f6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.718+0000 7f0f865df640 1 -- 172.21.15.44:0/4244529351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f8006d700 msgr2=0x7f0f6c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.718+0000 7f0f865df640 1 --2- 172.21.15.44:0/4244529351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f8006d700 0x7f0f6c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.718+0000 7f0f865df640 1 -- 172.21.15.44:0/4244529351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f6c0039e0 msgr2=0x7f0f6c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.718+0000 7f0f865df640 1 --2- 172.21.15.44:0/4244529351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f6c0039e0 0x7f0f6c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.718+0000 7f0f865df640 1 -- 172.21.15.44:0/4244529351 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f6c003520 con 0x7f0f6c004d40 2024-04-02T13:22:15.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.718+0000 7f0f865df640 1 --2- 172.21.15.44:0/4244529351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f6c004d40 0x7f0f6c0051c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800a920 tx=0x7f0f6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1393d5a04fafcd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:15.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.719+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7800a4c0 con 0x7f0f6c004d40 2024-04-02T13:22:15.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.719+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f78034650 con 0x7f0f6c004d40 2024-04-02T13:22:15.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.719+0000 7f0f84ddc640 1 -- 172.21.15.44:0/4244529351 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78034a40 con 0x7f0f6c004d40 2024-04-02T13:22:15.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.719+0000 7f0f88042640 1 -- 172.21.15.44:0/4244529351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f6c004d40 msgr2=0x7f0f6c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.719+0000 7f0f88042640 1 --2- 172.21.15.44:0/4244529351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f6c004d40 0x7f0f6c0051c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800a920 tx=0x7f0f6c005830 comp rx=0 tx=0).stop 2024-04-02T13:22:15.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.719+0000 7f0f865df640 1 -- 172.21.15.44:0/4244529351 reap_dead start 2024-04-02T13:22:15.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.719+0000 7f0f88042640 1 -- 172.21.15.44:0/4244529351 shutdown_connections 2024-04-02T13:22:15.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.719+0000 7f0f88042640 1 -- 172.21.15.44:0/4244529351 >> 172.21.15.44:0/4244529351 conn(0x7f0f8009aa20 msgr2=0x7f0f8009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:15.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.719+0000 7f0f88042640 1 -- 172.21.15.44:0/4244529351 shutdown_connections 2024-04-02T13:22:15.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.719+0000 7f0f88042640 1 -- 172.21.15.44:0/4244529351 wait complete. 2024-04-02T13:22:15.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.720+0000 7f0f88042640 1 Processor -- start 2024-04-02T13:22:15.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.720+0000 7f0f88042640 1 -- start start 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.721+0000 7f0f88042640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f80154300 0x7f0f800b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.721+0000 7f0f88042640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f800b6730 0x7f0f800b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.721+0000 7f0f88042640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f800b24e0 0x7f0f801f7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.721+0000 7f0f88042640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0f800d0590 con 0x7f0f800b24e0 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.721+0000 7f0f88042640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0f800d02b0 con 0x7f0f80154300 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.721+0000 7f0f88042640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0f800cc4b0 con 0x7f0f800b6730 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.721+0000 7f0f865df640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f800b24e0 0x7f0f801f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.721+0000 7f0f865df640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f800b24e0 0x7f0f801f7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60044/0 (socket says 172.21.15.44:60044) 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.721+0000 7f0f865df640 1 -- 172.21.15.44:0/1586872580 learned_addr learned my addr 172.21.15.44:0/1586872580 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.721+0000 7f0f855dd640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f800b6730 0x7f0f800b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.721+0000 7f0f85dde640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f80154300 0x7f0f800b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.722+0000 7f0f865df640 1 -- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f800b6730 msgr2=0x7f0f800b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.722+0000 7f0f865df640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f800b6730 0x7f0f800b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.722+0000 7f0f865df640 1 -- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f80154300 msgr2=0x7f0f800b61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:15.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.722+0000 7f0f865df640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f80154300 0x7f0f800b61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:15.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.722+0000 7f0f865df640 1 -- 172.21.15.44:0/1586872580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f800aeed0 con 0x7f0f800b24e0 2024-04-02T13:22:15.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.722+0000 7f0f85dde640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f80154300 0x7f0f800b61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:15.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.722+0000 7f0f855dd640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f800b6730 0x7f0f800b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:15.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.722+0000 7f0f865df640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f800b24e0 0x7f0f801f7410 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0f78001cf0 tx=0x7f0f78001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:15.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.722+0000 7f0f767fc640 1 -- 172.21.15.44:0/1586872580 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78045070 con 0x7f0f800b24e0 2024-04-02T13:22:15.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.722+0000 7f0f767fc640 1 -- 172.21.15.44:0/1586872580 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f7803a040 con 0x7f0f800b24e0 2024-04-02T13:22:15.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.722+0000 7f0f88042640 1 -- 172.21.15.44:0/1586872580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f800af1b0 con 0x7f0f800b24e0 2024-04-02T13:22:15.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.722+0000 7f0f767fc640 1 -- 172.21.15.44:0/1586872580 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7800a4c0 con 0x7f0f800b24e0 2024-04-02T13:22:15.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.722+0000 7f0f88042640 1 -- 172.21.15.44:0/1586872580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f800af770 con 0x7f0f800b24e0 2024-04-02T13:22:15.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.723+0000 7f0f88042640 1 -- 172.21.15.44:0/1586872580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f800684f0 con 0x7f0f800b24e0 2024-04-02T13:22:15.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.723+0000 7f0f767fc640 1 -- 172.21.15.44:0/1586872580 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0f78048070 con 0x7f0f800b24e0 2024-04-02T13:22:15.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.724+0000 7f0f767fc640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0f6406c8f0 0x7f0f6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:15.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.724+0000 7f0f767fc640 1 -- 172.21.15.44:0/1586872580 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5335+0+0 (secure 0 0 0) 0x7f0f780ad5b0 con 0x7f0f800b24e0 2024-04-02T13:22:15.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.724+0000 7f0f85dde640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0f6406c8f0 0x7f0f6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:15.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.725+0000 7f0f85dde640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0f6406c8f0 0x7f0f6406edb0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c003700 tx=0x7f0f7c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:15.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.729+0000 7f0f767fc640 1 -- 172.21.15.44:0/1586872580 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0f7807bac0 con 0x7f0f800b24e0 2024-04-02T13:22:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:15.985+0000 7f0f88042640 1 -- 172.21.15.44:0/1586872580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) -- 0x7f0f800b3420 con 0x7f0f800b24e0 2024-04-02T13:22:16.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:16.984+0000 7f0f767fc640 1 -- 172.21.15.44:0/1586872580 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v126) ==== 152+0+0 (secure 0 0 0) 0x7f0f7807b3c0 con 0x7f0f800b24e0 2024-04-02T13:22:16.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:16.986+0000 7f0f88042640 1 -- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0f6406c8f0 msgr2=0x7f0f6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:16.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:16.986+0000 7f0f88042640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0f6406c8f0 0x7f0f6406edb0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c003700 tx=0x7f0f7c009040 comp rx=0 tx=0).stop 2024-04-02T13:22:16.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:16.986+0000 7f0f88042640 1 -- 172.21.15.44:0/1586872580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f800b24e0 msgr2=0x7f0f801f7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:16.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:16.986+0000 7f0f88042640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f800b24e0 0x7f0f801f7410 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0f78001cf0 tx=0x7f0f78001dc0 comp rx=0 tx=0).stop 2024-04-02T13:22:16.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:16.986+0000 7f0f88042640 1 -- 172.21.15.44:0/1586872580 shutdown_connections 2024-04-02T13:22:16.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:16.986+0000 7f0f88042640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0f6406c8f0 0x7f0f6406edb0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:16.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:16.986+0000 7f0f88042640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f800b24e0 0x7f0f801f7410 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:16.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:16.986+0000 7f0f88042640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f800b6730 0x7f0f800b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:16.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:16.986+0000 7f0f88042640 1 --2- 172.21.15.44:0/1586872580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f80154300 0x7f0f800b61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:16.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:16.986+0000 7f0f88042640 1 -- 172.21.15.44:0/1586872580 >> 172.21.15.44:0/1586872580 conn(0x7f0f8009aa20 msgr2=0x7f0f80070800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:16.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:16.987+0000 7f0f88042640 1 -- 172.21.15.44:0/1586872580 shutdown_connections 2024-04-02T13:22:16.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:16.987+0000 7f0f88042640 1 -- 172.21.15.44:0/1586872580 wait complete. 2024-04-02T13:22:17.002 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T13:22:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.111+0000 7f27be759640 1 Processor -- start 2024-04-02T13:22:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.111+0000 7f27be759640 1 -- start start 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27be759640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f27b815c260 0x7f27b80b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27be759640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f27b80b3e80 0x7f27b80ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27be759640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27b80cd4a0 con 0x7f27b80cd030 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27be759640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f27b80cd760 con 0x7f27b815c260 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27be759640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f27b80cda40 con 0x7f27b80b3e80 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27b7fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f27b815c260 0x7f27b80b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27bccf6640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f27b80cd030 0x7f27b80ce9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53320/0 (socket says 172.21.15.44:53320) 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27bccf6640 1 -- 172.21.15.44:0/188052139 learned_addr learned my addr 172.21.15.44:0/188052139 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27b77fe640 1 --2- 172.21.15.44:0/188052139 >> v2:172.21.15.69:3300/0 conn(0x7f27b80b3e80 0x7f27b80ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27b7fff640 1 -- 172.21.15.44:0/188052139 >> v2:172.21.15.69:3300/0 conn(0x7f27b80b3e80 msgr2=0x7f27b80ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27b7fff640 1 --2- 172.21.15.44:0/188052139 >> v2:172.21.15.69:3300/0 conn(0x7f27b80b3e80 0x7f27b80ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27b7fff640 1 -- 172.21.15.44:0/188052139 >> v1:172.21.15.44:6789/0 conn(0x7f27b80cd030 legacy=0x7f27b80ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27b7fff640 1 -- 172.21.15.44:0/188052139 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b80cdd20 con 0x7f27b815c260 2024-04-02T13:22:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27b6ffd640 1 -- 172.21.15.44:0/188052139 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404154743 0 0) 0x7f27b80cd4a0 con 0x7f27b80cd030 2024-04-02T13:22:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.112+0000 7f27b77fe640 1 --2- 172.21.15.44:0/188052139 >> v2:172.21.15.69:3300/0 conn(0x7f27b80b3e80 0x7f27b80ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.113+0000 7f27b7fff640 1 --2- 172.21.15.44:0/188052139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b815c260 0x7f27b80b3940 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f27b809b6a0 tx=0x7f27ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28923fbb90e40b9d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.113+0000 7f27b6ffd640 1 -- 172.21.15.44:0/188052139 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac035020 con 0x7f27b815c260 2024-04-02T13:22:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.113+0000 7f27b6ffd640 1 -- 172.21.15.44:0/188052139 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27ac031830 con 0x7f27b815c260 2024-04-02T13:22:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.113+0000 7f27b6ffd640 1 -- 172.21.15.44:0/188052139 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac031c00 con 0x7f27b815c260 2024-04-02T13:22:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.114+0000 7f27be759640 1 -- 172.21.15.44:0/188052139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b815c260 msgr2=0x7f27b80b3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.114+0000 7f27be759640 1 --2- 172.21.15.44:0/188052139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b815c260 0x7f27b80b3940 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f27b809b6a0 tx=0x7f27ac02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.114+0000 7f27be759640 1 -- 172.21.15.44:0/188052139 shutdown_connections 2024-04-02T13:22:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.114+0000 7f27be759640 1 --2- 172.21.15.44:0/188052139 >> v2:172.21.15.69:3300/0 conn(0x7f27b80b3e80 0x7f27b80ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.114+0000 7f27be759640 1 --2- 172.21.15.44:0/188052139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b815c260 0x7f27b80b3940 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.114+0000 7f27be759640 1 -- 172.21.15.44:0/188052139 >> 172.21.15.44:0/188052139 conn(0x7f27b809aa20 msgr2=0x7f27b809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.114+0000 7f27be759640 1 -- 172.21.15.44:0/188052139 shutdown_connections 2024-04-02T13:22:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.114+0000 7f27be759640 1 -- 172.21.15.44:0/188052139 wait complete. 2024-04-02T13:22:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.115+0000 7f27be759640 1 Processor -- start 2024-04-02T13:22:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.115+0000 7f27be759640 1 -- start start 2024-04-02T13:22:17.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27be759640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27b80b3e80 0x7f27b81f3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:17.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27be759640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80cd030 0x7f27b80d09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27be759640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27b815c260 0x7f27b80b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27be759640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f27b80c6b40 con 0x7f27b80cd030 2024-04-02T13:22:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27be759640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f27b80cfe60 con 0x7f27b815c260 2024-04-02T13:22:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27be759640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f27b80c6cc0 con 0x7f27b80b3e80 2024-04-02T13:22:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27b77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80cd030 0x7f27b80d09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27b77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80cd030 0x7f27b80d09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60056/0 (socket says 172.21.15.44:60056) 2024-04-02T13:22:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27b77fe640 1 -- 172.21.15.44:0/3261909074 learned_addr learned my addr 172.21.15.44:0/3261909074 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27bccf6640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27b815c260 0x7f27b80b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27b7fff640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27b80b3e80 0x7f27b81f3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27b77fe640 1 -- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27b80b3e80 msgr2=0x7f27b81f3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27b77fe640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27b80b3e80 0x7f27b81f3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27b77fe640 1 -- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27b815c260 msgr2=0x7f27b80b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27b77fe640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27b815c260 0x7f27b80b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27b77fe640 1 -- 172.21.15.44:0/3261909074 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b80d0f00 con 0x7f27b80cd030 2024-04-02T13:22:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.116+0000 7f27b77fe640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80cd030 0x7f27b80d09c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f27a800c7f0 tx=0x7f27a800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.117+0000 7f27b4ff9640 1 -- 172.21.15.44:0/3261909074 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a8015070 con 0x7f27b80cd030 2024-04-02T13:22:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.117+0000 7f27b4ff9640 1 -- 172.21.15.44:0/3261909074 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27a8007820 con 0x7f27b80cd030 2024-04-02T13:22:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.117+0000 7f27be759640 1 -- 172.21.15.44:0/3261909074 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b81f45d0 con 0x7f27b80cd030 2024-04-02T13:22:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.117+0000 7f27b4ff9640 1 -- 172.21.15.44:0/3261909074 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a8007bd0 con 0x7f27b80cd030 2024-04-02T13:22:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.117+0000 7f27be759640 1 -- 172.21.15.44:0/3261909074 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b80b6e20 con 0x7f27b80cd030 2024-04-02T13:22:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.118+0000 7f27be759640 1 -- 172.21.15.44:0/3261909074 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27b80684f0 con 0x7f27b80cd030 2024-04-02T13:22:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.118+0000 7f27b4ff9640 1 -- 172.21.15.44:0/3261909074 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f27a8018070 con 0x7f27b80cd030 2024-04-02T13:22:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.119+0000 7f27b4ff9640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27a406c8f0 0x7f27a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.119+0000 7f27b4ff9640 1 -- 172.21.15.44:0/3261909074 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5335+0+0 (secure 0 0 0) 0x7f27a808b940 con 0x7f27b80cd030 2024-04-02T13:22:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.119+0000 7f27b7fff640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27a406c8f0 0x7f27a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.119+0000 7f27b7fff640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27a406c8f0 0x7f27a406edb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f27ac036000 tx=0x7f27ac008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:17.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.123+0000 7f27b4ff9640 1 -- 172.21.15.44:0/3261909074 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f27a8059920 con 0x7f27b80cd030 2024-04-02T13:22:17.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.306+0000 7f27be759640 1 -- 172.21.15.44:0/3261909074 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f27b8070300 con 0x7f27b80cd030 2024-04-02T13:22:17.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.309+0000 7f27b4ff9640 1 -- 172.21.15.44:0/3261909074 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f27a805dc20 con 0x7f27b80cd030 2024-04-02T13:22:17.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.311+0000 7f27be759640 1 -- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27a406c8f0 msgr2=0x7f27a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:17.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.311+0000 7f27be759640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27a406c8f0 0x7f27a406edb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f27ac036000 tx=0x7f27ac008030 comp rx=0 tx=0).stop 2024-04-02T13:22:17.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.311+0000 7f27be759640 1 -- 172.21.15.44:0/3261909074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80cd030 msgr2=0x7f27b80d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:17.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.311+0000 7f27be759640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80cd030 0x7f27b80d09c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f27a800c7f0 tx=0x7f27a800ccb0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.311+0000 7f27be759640 1 -- 172.21.15.44:0/3261909074 shutdown_connections 2024-04-02T13:22:17.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.311+0000 7f27be759640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27a406c8f0 0x7f27a406edb0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.311+0000 7f27be759640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27b815c260 0x7f27b80b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.311+0000 7f27be759640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80cd030 0x7f27b80d09c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.311+0000 7f27be759640 1 --2- 172.21.15.44:0/3261909074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27b80b3e80 0x7f27b81f3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.311+0000 7f27be759640 1 -- 172.21.15.44:0/3261909074 >> 172.21.15.44:0/3261909074 conn(0x7f27b809aa20 msgr2=0x7f27b80c7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:17.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.312+0000 7f27be759640 1 -- 172.21.15.44:0/3261909074 shutdown_connections 2024-04-02T13:22:17.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.312+0000 7f27be759640 1 -- 172.21.15.44:0/3261909074 wait complete. 2024-04-02T13:22:17.324 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:22:17.324 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:22:17.362 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:22:17.408 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:22:17.408 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:22:17.408 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:22:17.423 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:22:17.424 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:22:17.424 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:22:17.441 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:22:17.442 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:22:17.442 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:22:17.478 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:22:17.479 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:22:17.479 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:22:17.500 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:22:17.501 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T13:22:17.501 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:22:17.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.618+0000 7f7c2b00d640 1 Processor -- start 2024-04-02T13:22:17.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.618+0000 7f7c2b00d640 1 -- start start 2024-04-02T13:22:17.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c2b00d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7c2415e140 0x7f7c240d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:17.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c2b00d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7c2415e520 0x7f7c240d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:17.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c2b00d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c240d2b70 0x7f7c240d0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:17.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c2b00d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7c240b9f50 con 0x7f7c2415e520 2024-04-02T13:22:17.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c2b00d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7c240b70b0 con 0x7f7c2415e140 2024-04-02T13:22:17.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c2b00d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7c240b7390 con 0x7f7c240d2b70 2024-04-02T13:22:17.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c23fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7c2415e520 0x7f7c240d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:17.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c23fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7c2415e520 0x7f7c240d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60066/0 (socket says 172.21.15.44:60066) 2024-04-02T13:22:17.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c23fff640 1 -- 172.21.15.44:0/3265934476 learned_addr learned my addr 172.21.15.44:0/3265934476 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:17.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c295aa640 1 --2- 172.21.15.44:0/3265934476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c240d2b70 0x7f7c240d0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:17.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c28da9640 1 --2- 172.21.15.44:0/3265934476 >> v2:172.21.15.69:3300/0 conn(0x7f7c2415e140 0x7f7c240d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:17.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c23fff640 1 -- 172.21.15.44:0/3265934476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c240d2b70 msgr2=0x7f7c240d0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:17.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c23fff640 1 --2- 172.21.15.44:0/3265934476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c240d2b70 0x7f7c240d0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c23fff640 1 -- 172.21.15.44:0/3265934476 >> v2:172.21.15.69:3300/0 conn(0x7f7c2415e140 msgr2=0x7f7c240d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:17.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c23fff640 1 --2- 172.21.15.44:0/3265934476 >> v2:172.21.15.69:3300/0 conn(0x7f7c2415e140 0x7f7c240d2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.619+0000 7f7c23fff640 1 -- 172.21.15.44:0/3265934476 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c240b7670 con 0x7f7c2415e520 2024-04-02T13:22:17.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.620+0000 7f7c23fff640 1 --2- 172.21.15.44:0/3265934476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c2415e520 0x7f7c240d0a40 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f7c2408e6d0 tx=0x7f7c1002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea9174de3d11b6e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:17.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.620+0000 7f7c237fe640 1 -- 172.21.15.44:0/3265934476 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c10035020 con 0x7f7c2415e520 2024-04-02T13:22:17.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.620+0000 7f7c237fe640 1 -- 172.21.15.44:0/3265934476 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c10031870 con 0x7f7c2415e520 2024-04-02T13:22:17.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.620+0000 7f7c237fe640 1 -- 172.21.15.44:0/3265934476 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c10031c40 con 0x7f7c2415e520 2024-04-02T13:22:17.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.620+0000 7f7c2b00d640 1 -- 172.21.15.44:0/3265934476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c2415e520 msgr2=0x7f7c240d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:17.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.620+0000 7f7c2b00d640 1 --2- 172.21.15.44:0/3265934476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c2415e520 0x7f7c240d0a40 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f7c2408e6d0 tx=0x7f7c1002b910 comp rx=0 tx=0).stop 2024-04-02T13:22:17.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.621+0000 7f7c2b00d640 1 -- 172.21.15.44:0/3265934476 shutdown_connections 2024-04-02T13:22:17.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.621+0000 7f7c2b00d640 1 --2- 172.21.15.44:0/3265934476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c240d2b70 0x7f7c240d0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.621+0000 7f7c2b00d640 1 --2- 172.21.15.44:0/3265934476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c2415e520 0x7f7c240d0a40 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.621+0000 7f7c2b00d640 1 --2- 172.21.15.44:0/3265934476 >> v2:172.21.15.69:3300/0 conn(0x7f7c2415e140 0x7f7c240d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.621+0000 7f7c2b00d640 1 -- 172.21.15.44:0/3265934476 >> 172.21.15.44:0/3265934476 conn(0x7f7c2409d390 msgr2=0x7f7c240d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:17.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.621+0000 7f7c2b00d640 1 -- 172.21.15.44:0/3265934476 shutdown_connections 2024-04-02T13:22:17.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.621+0000 7f7c2b00d640 1 -- 172.21.15.44:0/3265934476 wait complete. 2024-04-02T13:22:17.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.622+0000 7f7c2b00d640 1 Processor -- start 2024-04-02T13:22:17.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.622+0000 7f7c2b00d640 1 -- start start 2024-04-02T13:22:17.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c2b00d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c240d2b70 0x7f7c2415f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:17.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c2b00d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c2415e140 0x7f7c240b5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:17.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c2b00d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c2415e520 0x7f7c24152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:17.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c2b00d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7c240b4cc0 con 0x7f7c240d2b70 2024-04-02T13:22:17.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c2b00d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7c240d4ee0 con 0x7f7c2415e520 2024-04-02T13:22:17.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c2b00d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7c240b4e40 con 0x7f7c2415e140 2024-04-02T13:22:17.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c28da9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c240d2b70 0x7f7c2415f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:17.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c28da9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c240d2b70 0x7f7c2415f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60072/0 (socket says 172.21.15.44:60072) 2024-04-02T13:22:17.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c28da9640 1 -- 172.21.15.44:0/4163075617 learned_addr learned my addr 172.21.15.44:0/4163075617 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:17.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c23fff640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c2415e140 0x7f7c240b5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:17.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c295aa640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c2415e520 0x7f7c24152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:17.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c28da9640 1 -- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c2415e140 msgr2=0x7f7c240b5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c28da9640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c2415e140 0x7f7c240b5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c28da9640 1 -- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c2415e520 msgr2=0x7f7c24152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c28da9640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c2415e520 0x7f7c24152030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c28da9640 1 -- 172.21.15.44:0/4163075617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c24161860 con 0x7f7c240d2b70 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c28da9640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c240d2b70 0x7f7c2415f840 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800d800 tx=0x7f7c1800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c217fa640 1 -- 172.21.15.44:0/4163075617 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1800b6c0 con 0x7f7c240d2b70 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c217fa640 1 -- 172.21.15.44:0/4163075617 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c18019070 con 0x7f7c240d2b70 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c2b00d640 1 -- 172.21.15.44:0/4163075617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c241607c0 con 0x7f7c240d2b70 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c217fa640 1 -- 172.21.15.44:0/4163075617 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1800ba80 con 0x7f7c240d2b70 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.623+0000 7f7c2b00d640 1 -- 172.21.15.44:0/4163075617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c24161060 con 0x7f7c240d2b70 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.624+0000 7f7c2b00d640 1 -- 172.21.15.44:0/4163075617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c240684f0 con 0x7f7c240d2b70 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.625+0000 7f7c217fa640 1 -- 172.21.15.44:0/4163075617 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7c1800bc20 con 0x7f7c240d2b70 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.625+0000 7f7c217fa640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7c0806c8f0 0x7f7c0806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.625+0000 7f7c217fa640 1 -- 172.21.15.44:0/4163075617 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5335+0+0 (secure 0 0 0) 0x7f7c1808bca0 con 0x7f7c240d2b70 2024-04-02T13:22:17.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.625+0000 7f7c23fff640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7c0806c8f0 0x7f7c0806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:17.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.626+0000 7f7c23fff640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7c0806c8f0 0x7f7c0806edb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7c10036000 tx=0x7f7c1003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:17.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.630+0000 7f7c217fa640 1 -- 172.21.15.44:0/4163075617 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7c18059bf0 con 0x7f7c240d2b70 2024-04-02T13:22:17.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.857+0000 7f7c2b00d640 1 -- 172.21.15.44:0/4163075617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c2406c030 con 0x7f7c240d2b70 2024-04-02T13:22:17.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.857+0000 7f7c217fa640 1 -- 172.21.15.44:0/4163075617 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+3866 (secure 0 0 0) 0x7f7c1805def0 con 0x7f7c240d2b70 2024-04-02T13:22:17.859 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:17.859 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3050514527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3050514527},{"type":"v1","addr":"172.21.15.69:6836","nonce":3050514527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4973,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1949024012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1949024012},{"type":"v1","addr":"172.21.15.69:6837","nonce":1949024012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/2817873198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2817873198},{"type":"v1","addr":"172.21.15.44:6834","nonce":2817873198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/567140000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":567140000},{"type":"v1","addr":"172.21.15.44:6835","nonce":567140000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[]} 2024-04-02T13:22:17.860 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 20 2024-04-02T13:22:17.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.859+0000 7f7c2b00d640 1 -- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7c0806c8f0 msgr2=0x7f7c0806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:17.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.859+0000 7f7c2b00d640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7c0806c8f0 0x7f7c0806edb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7c10036000 tx=0x7f7c1003c040 comp rx=0 tx=0).stop 2024-04-02T13:22:17.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.859+0000 7f7c2b00d640 1 -- 172.21.15.44:0/4163075617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c240d2b70 msgr2=0x7f7c2415f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:17.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.859+0000 7f7c2b00d640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c240d2b70 0x7f7c2415f840 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800d800 tx=0x7f7c1800dcc0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.860+0000 7f7c2b00d640 1 -- 172.21.15.44:0/4163075617 shutdown_connections 2024-04-02T13:22:17.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.860+0000 7f7c2b00d640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7c0806c8f0 0x7f7c0806edb0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.860+0000 7f7c2b00d640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c2415e520 0x7f7c24152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.860+0000 7f7c2b00d640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c2415e140 0x7f7c240b5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.860+0000 7f7c2b00d640 1 --2- 172.21.15.44:0/4163075617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c240d2b70 0x7f7c2415f840 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:17.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.860+0000 7f7c2b00d640 1 -- 172.21.15.44:0/4163075617 >> 172.21.15.44:0/4163075617 conn(0x7f7c2409d390 msgr2=0x7f7c240b8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:17.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.860+0000 7f7c2b00d640 1 -- 172.21.15.44:0/4163075617 shutdown_connections 2024-04-02T13:22:17.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:17.860+0000 7f7c2b00d640 1 -- 172.21.15.44:0/4163075617 wait complete. 2024-04-02T13:22:17.872 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T13:22:17.873 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T13:22:17.873 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T13:22:17.873 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:22:17.873 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T13:22:17.873 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T13:22:17.874 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T13:22:17.874 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:22:17.874 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T13:22:17.874 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T13:22:17.875 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T13:22:17.875 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:22:17.875 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T13:22:17.875 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T13:22:17.876 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T13:22:17.876 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:22:17.882 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T13:22:17.883 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T13:22:17.883 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T13:22:17.884 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T13:22:17.887 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T13:22:17.887 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T13:22:17.888 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T13:22:17.888 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T13:22:17.925 INFO:tasks.ceph.mds.a:Started 2024-04-02T13:22:17.925 DEBUG:teuthology.parallel:result is None 2024-04-02T13:22:17.925 INFO:tasks.ceph.mds.c:Started 2024-04-02T13:22:17.926 DEBUG:teuthology.parallel:result is None 2024-04-02T13:22:17.929 INFO:tasks.ceph.mds.b:Started 2024-04-02T13:22:17.929 DEBUG:teuthology.parallel:result is None 2024-04-02T13:22:17.930 INFO:tasks.ceph.mds.d:Started 2024-04-02T13:22:17.930 DEBUG:teuthology.parallel:result is None 2024-04-02T13:22:17.931 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T13:22:18.002 INFO:tasks.ceph.mds.c.smithi044.stdout:starting mds.c at 2024-04-02T13:22:18.003 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:22:18.012 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-04-02T13:22:18.012 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-04-02T13:22:18.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.037+0000 7f6ad958f640 1 Processor -- start 2024-04-02T13:22:18.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.037+0000 7f6ad958f640 1 -- start start 2024-04-02T13:22:18.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.038+0000 7f6ad958f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6ad40b5200 0x7f6ad40d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:18.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.038+0000 7f6ad958f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ad40d4d70 con 0x7f6ad40d28f0 2024-04-02T13:22:18.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.038+0000 7f6ad958f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ad40d2030 con 0x7f6ad40b47b0 2024-04-02T13:22:18.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.038+0000 7f6ad958f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6ad40d5050 con 0x7f6ad40b5200 2024-04-02T13:22:18.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.038+0000 7f6ad2ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6ad40d28f0 0x7f6ad40b40a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53324/0 (socket says 172.21.15.44:53324) 2024-04-02T13:22:18.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.038+0000 7f6ad2ffd640 1 -- 172.21.15.44:0/2057403759 learned_addr learned my addr 172.21.15.44:0/2057403759 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:18.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.038+0000 7f6ad37fe640 1 --2- 172.21.15.44:0/2057403759 >> v2:172.21.15.69:3300/0 conn(0x7f6ad40b5200 0x7f6ad40d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:18.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.038+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 52255964 0 0) 0x7f6ad40d4d70 con 0x7f6ad40d28f0 2024-04-02T13:22:18.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.038+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ac0003520 con 0x7f6ad40d28f0 2024-04-02T13:22:18.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.038+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2595372977 0 0) 0x7f6ac0003520 con 0x7f6ad40d28f0 2024-04-02T13:22:18.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.038+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 >> v2:172.21.15.69:3300/0 conn(0x7f6ad40b5200 msgr2=0x7f6ad40d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:18.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad27fc640 1 --2- 172.21.15.44:0/2057403759 >> v2:172.21.15.69:3300/0 conn(0x7f6ad40b5200 0x7f6ad40d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 >> v1:172.21.15.69:6789/0 conn(0x7f6ad40b47b0 legacy=0x7f6ad40b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad40b4cb0 con 0x7f6ad40d28f0 2024-04-02T13:22:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f6ac4001670 con 0x7f6ad40d28f0 2024-04-02T13:22:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 >> v1:172.21.15.44:6789/0 conn(0x7f6ad40d28f0 legacy=0x7f6ad40b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad27fc640 1 --2- 172.21.15.44:0/2057403759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ac00039b0 0x7f6ac0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad27fc640 1 --2- 172.21.15.44:0/2057403759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac0004800 0x7f6ac0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad27fc640 1 --2- 172.21.15.44:0/2057403759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ac00052e0 0x7f6ac000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad40d5050 con 0x7f6ac0004800 2024-04-02T13:22:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac000c1b0 con 0x7f6ac00052e0 2024-04-02T13:22:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac000c3b0 con 0x7f6ac00039b0 2024-04-02T13:22:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad37fe640 1 --2- 172.21.15.44:0/2057403759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ac00052e0 0x7f6ac000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6aca7fc640 1 --2- 172.21.15.44:0/2057403759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac0004800 0x7f6ac0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad2ffd640 1 --2- 172.21.15.44:0/2057403759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ac00039b0 0x7f6ac0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad37fe640 1 -- 172.21.15.44:0/2057403759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ac00039b0 msgr2=0x7f6ac0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad37fe640 1 --2- 172.21.15.44:0/2057403759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ac00039b0 0x7f6ac0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad37fe640 1 -- 172.21.15.44:0/2057403759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac0004800 msgr2=0x7f6ac0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad37fe640 1 --2- 172.21.15.44:0/2057403759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac0004800 0x7f6ac0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.039+0000 7f6ad37fe640 1 -- 172.21.15.44:0/2057403759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac0003520 con 0x7f6ac00052e0 2024-04-02T13:22:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.040+0000 7f6ad37fe640 1 -- 172.21.15.44:0/2057403759 reap_dead start 2024-04-02T13:22:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.040+0000 7f6ad37fe640 1 --2- 172.21.15.44:0/2057403759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ac00052e0 0x7f6ac000b9c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6acc009f90 tx=0x7f6ac0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e86cd2d2df5a68f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.040+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc009c10 con 0x7f6ac00052e0 2024-04-02T13:22:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.040+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6acc02cc20 con 0x7f6ac00052e0 2024-04-02T13:22:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.040+0000 7f6ad27fc640 1 -- 172.21.15.44:0/2057403759 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc003990 con 0x7f6ac00052e0 2024-04-02T13:22:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.040+0000 7f6ad958f640 1 -- 172.21.15.44:0/2057403759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ac00052e0 msgr2=0x7f6ac000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.040+0000 7f6ad958f640 1 --2- 172.21.15.44:0/2057403759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ac00052e0 0x7f6ac000b9c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6acc009f90 tx=0x7f6ac0005830 comp rx=0 tx=0).stop 2024-04-02T13:22:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.040+0000 7f6ad958f640 1 -- 172.21.15.44:0/2057403759 shutdown_connections 2024-04-02T13:22:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.040+0000 7f6ad958f640 1 --2- 172.21.15.44:0/2057403759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ac00052e0 0x7f6ac000b9c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.040+0000 7f6ad958f640 1 -- 172.21.15.44:0/2057403759 >> 172.21.15.44:0/2057403759 conn(0x7f6ad40d4120 msgr2=0x7f6ad40d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.040+0000 7f6ad958f640 1 -- 172.21.15.44:0/2057403759 shutdown_connections 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.040+0000 7f6ad958f640 1 -- 172.21.15.44:0/2057403759 wait complete. 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.042+0000 7f6ad958f640 1 Processor -- start 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.042+0000 7f6ad958f640 1 -- start start 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.042+0000 7f6ad958f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac00052e0 0x7f6ad40a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.042+0000 7f6ad958f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ad40a1950 0x7f6ad40b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.042+0000 7f6ad958f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ad40b33c0 0x7f6ad409ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.042+0000 7f6ad958f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad40d23a0 con 0x7f6ac00052e0 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.042+0000 7f6ad958f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad40d1470 con 0x7f6ad40a1950 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.042+0000 7f6ad958f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6ad40b5020 con 0x7f6ad40b33c0 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6ad2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac00052e0 0x7f6ad40a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6ad2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac00052e0 0x7f6ad40a0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60108/0 (socket says 172.21.15.44:60108) 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6ad2ffd640 1 -- 172.21.15.44:0/1965441979 learned_addr learned my addr 172.21.15.44:0/1965441979 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6aca7fc640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ad40a1950 0x7f6ad40b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6ad37fe640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ad40b33c0 0x7f6ad409ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6ad2ffd640 1 -- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ad40b33c0 msgr2=0x7f6ad409ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6ad2ffd640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ad40b33c0 0x7f6ad409ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6ad2ffd640 1 -- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ad40a1950 msgr2=0x7f6ad40b2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6ad2ffd640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ad40a1950 0x7f6ad40b2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6ad2ffd640 1 -- 172.21.15.44:0/1965441979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad41f3b60 con 0x7f6ac00052e0 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6ad2ffd640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac00052e0 0x7f6ad40a0ca0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4004ea0 tx=0x7f6ac4004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6acbfff640 1 -- 172.21.15.44:0/1965441979 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac400f680 con 0x7f6ac00052e0 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6acbfff640 1 -- 172.21.15.44:0/1965441979 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ac4018070 con 0x7f6ac00052e0 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6ad958f640 1 -- 172.21.15.44:0/1965441979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ad41f3e40 con 0x7f6ac00052e0 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6acbfff640 1 -- 172.21.15.44:0/1965441979 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac400fa60 con 0x7f6ac00052e0 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.043+0000 7f6ad958f640 1 -- 172.21.15.44:0/1965441979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ad41f4400 con 0x7f6ac00052e0 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.044+0000 7f6ad958f640 1 -- 172.21.15.44:0/1965441979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ad40684f0 con 0x7f6ac00052e0 2024-04-02T13:22:18.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.045+0000 7f6acbfff640 1 -- 172.21.15.44:0/1965441979 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ac400fc00 con 0x7f6ac00052e0 2024-04-02T13:22:18.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.045+0000 7f6acbfff640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6ab406c920 0x7f6ab406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:18.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.045+0000 7f6acbfff640 1 -- 172.21.15.44:0/1965441979 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5335+0+0 (secure 0 0 0) 0x7f6ac408b6f0 con 0x7f6ac00052e0 2024-04-02T13:22:18.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.045+0000 7f6aca7fc640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6ab406c920 0x7f6ab406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:18.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.046+0000 7f6aca7fc640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6ab406c920 0x7f6ab406ede0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6abc000c00 tx=0x7f6abc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:18.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.050+0000 7f6acbfff640 1 -- 172.21.15.44:0/1965441979 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6ac40596d0 con 0x7f6ac00052e0 2024-04-02T13:22:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.246+0000 7f6ad958f640 1 -- 172.21.15.44:0/1965441979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f6ad40b37a0 con 0x7f6ac00052e0 2024-04-02T13:22:19.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:18.999+0000 7f6acbfff640 1 -- 172.21.15.44:0/1965441979 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v46) ==== 97+0+0 (secure 0 0 0) 0x7f6ac408b030 con 0x7f6ac00052e0 2024-04-02T13:22:19.001 INFO:teuthology.orchestra.run.smithi044.stderr: removed all blocklist entries 2024-04-02T13:22:19.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.003+0000 7f6ad958f640 1 -- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6ab406c920 msgr2=0x7f6ab406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.003+0000 7f6ad958f640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6ab406c920 0x7f6ab406ede0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6abc000c00 tx=0x7f6abc009040 comp rx=0 tx=0).stop 2024-04-02T13:22:19.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.003+0000 7f6ad958f640 1 -- 172.21.15.44:0/1965441979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac00052e0 msgr2=0x7f6ad40a0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.003+0000 7f6ad958f640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac00052e0 0x7f6ad40a0ca0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4004ea0 tx=0x7f6ac4004f90 comp rx=0 tx=0).stop 2024-04-02T13:22:19.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.003+0000 7f6ad958f640 1 -- 172.21.15.44:0/1965441979 shutdown_connections 2024-04-02T13:22:19.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.003+0000 7f6ad958f640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6ab406c920 0x7f6ab406ede0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.003+0000 7f6ad958f640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ad40b33c0 0x7f6ad409ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.003+0000 7f6ad958f640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ad40a1950 0x7f6ad40b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.003+0000 7f6ad958f640 1 --2- 172.21.15.44:0/1965441979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ac00052e0 0x7f6ad40a0ca0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.003+0000 7f6ad958f640 1 -- 172.21.15.44:0/1965441979 >> 172.21.15.44:0/1965441979 conn(0x7f6ad40d4120 msgr2=0x7f6ad406d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:19.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.004+0000 7f6ad958f640 1 -- 172.21.15.44:0/1965441979 shutdown_connections 2024-04-02T13:22:19.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.004+0000 7f6ad958f640 1 -- 172.21.15.44:0/1965441979 wait complete. 2024-04-02T13:22:19.022 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T13:22:19.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.127+0000 7fcb7d2f2640 1 Processor -- start 2024-04-02T13:22:19.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.127+0000 7fcb7d2f2640 1 -- start start 2024-04-02T13:22:19.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.128+0000 7fcb7d2f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcb7815c250 0x7fcb780aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.128+0000 7fcb7d2f2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcb780b3510 0x7fcb780d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.128+0000 7fcb7d2f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb780d08d0 0x7fcb780b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.128+0000 7fcb7d2f2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fcb7815c630 con 0x7fcb7815c250 2024-04-02T13:22:19.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.128+0000 7fcb7d2f2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcb780af420 con 0x7fcb780b3510 2024-04-02T13:22:19.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.128+0000 7fcb7d2f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcb780af700 con 0x7fcb780d08d0 2024-04-02T13:22:19.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.128+0000 7fcb76ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcb7815c250 0x7fcb780aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.128+0000 7fcb76ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcb7815c250 0x7fcb780aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60120/0 (socket says 172.21.15.44:60120) 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.128+0000 7fcb76ffd640 1 -- 172.21.15.44:0/4029199632 learned_addr learned my addr 172.21.15.44:0/4029199632 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.129+0000 7fcb767fc640 1 --2- 172.21.15.44:0/4029199632 >> v2:172.21.15.69:3300/0 conn(0x7fcb780b3510 0x7fcb780d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.129+0000 7fcb777fe640 1 --2- 172.21.15.44:0/4029199632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb780d08d0 0x7fcb780b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.129+0000 7fcb76ffd640 1 -- 172.21.15.44:0/4029199632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb780d08d0 msgr2=0x7fcb780b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.129+0000 7fcb76ffd640 1 --2- 172.21.15.44:0/4029199632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb780d08d0 0x7fcb780b1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.129+0000 7fcb76ffd640 1 -- 172.21.15.44:0/4029199632 >> v2:172.21.15.69:3300/0 conn(0x7fcb780b3510 msgr2=0x7fcb780d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.129+0000 7fcb76ffd640 1 --2- 172.21.15.44:0/4029199632 >> v2:172.21.15.69:3300/0 conn(0x7fcb780b3510 0x7fcb780d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.129+0000 7fcb76ffd640 1 -- 172.21.15.44:0/4029199632 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb780af9e0 con 0x7fcb7815c250 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.129+0000 7fcb777fe640 1 --2- 172.21.15.44:0/4029199632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb780d08d0 0x7fcb780b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.129+0000 7fcb767fc640 1 --2- 172.21.15.44:0/4029199632 >> v2:172.21.15.69:3300/0 conn(0x7fcb780b3510 0x7fcb780d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.129+0000 7fcb76ffd640 1 --2- 172.21.15.44:0/4029199632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb7815c250 0x7fcb780aebf0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcb780c8b70 tx=0x7fcb7002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0a9dc6d5d18e9c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.130+0000 7fcb75ffb640 1 -- 172.21.15.44:0/4029199632 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7002e070 con 0x7fcb7815c250 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.130+0000 7fcb75ffb640 1 -- 172.21.15.44:0/4029199632 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb70037070 con 0x7fcb7815c250 2024-04-02T13:22:19.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.130+0000 7fcb75ffb640 1 -- 172.21.15.44:0/4029199632 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb70031a00 con 0x7fcb7815c250 2024-04-02T13:22:19.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.130+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/4029199632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb7815c250 msgr2=0x7fcb780aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.130+0000 7fcb7d2f2640 1 --2- 172.21.15.44:0/4029199632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb7815c250 0x7fcb780aebf0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcb780c8b70 tx=0x7fcb7002b880 comp rx=0 tx=0).stop 2024-04-02T13:22:19.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.130+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/4029199632 shutdown_connections 2024-04-02T13:22:19.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.130+0000 7fcb7d2f2640 1 --2- 172.21.15.44:0/4029199632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb780d08d0 0x7fcb780b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.130+0000 7fcb7d2f2640 1 --2- 172.21.15.44:0/4029199632 >> v2:172.21.15.69:3300/0 conn(0x7fcb780b3510 0x7fcb780d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.130+0000 7fcb7d2f2640 1 --2- 172.21.15.44:0/4029199632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb7815c250 0x7fcb780aebf0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.130+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/4029199632 >> 172.21.15.44:0/4029199632 conn(0x7fcb78073850 msgr2=0x7fcb7809aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:19.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.130+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/4029199632 shutdown_connections 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.130+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/4029199632 wait complete. 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.132+0000 7fcb7d2f2640 1 Processor -- start 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.132+0000 7fcb7d2f2640 1 -- start start 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.132+0000 7fcb7d2f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb780b3510 0x7fcb78151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.132+0000 7fcb7d2f2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb780d08d0 0x7fcb780b73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.132+0000 7fcb7d2f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb7815c250 0x7fcb780ce910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.132+0000 7fcb7d2f2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcb780b2380 con 0x7fcb7815c250 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.132+0000 7fcb7d2f2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcb780af130 con 0x7fcb780d08d0 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.132+0000 7fcb7d2f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcb780b2500 con 0x7fcb780b3510 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.132+0000 7fcb777fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb7815c250 0x7fcb780ce910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.132+0000 7fcb777fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb7815c250 0x7fcb780ce910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60124/0 (socket says 172.21.15.44:60124) 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.132+0000 7fcb777fe640 1 -- 172.21.15.44:0/3846979617 learned_addr learned my addr 172.21.15.44:0/3846979617 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb76ffd640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb780b3510 0x7fcb78151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb767fc640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb780d08d0 0x7fcb780b73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb777fe640 1 -- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb780b3510 msgr2=0x7fcb78151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb777fe640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb780b3510 0x7fcb78151780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb777fe640 1 -- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb780d08d0 msgr2=0x7fcb780b73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb777fe640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb780d08d0 0x7fcb780b73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb777fe640 1 -- 172.21.15.44:0/3846979617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb78151cc0 con 0x7fcb7815c250 2024-04-02T13:22:19.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb767fc640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb780d08d0 0x7fcb780b73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:19.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb777fe640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb7815c250 0x7fcb780ce910 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcb68007ef0 tx=0x7fcb6800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:19.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb6f7fe640 1 -- 172.21.15.44:0/3846979617 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6801a070 con 0x7fcb7815c250 2024-04-02T13:22:19.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb6f7fe640 1 -- 172.21.15.44:0/3846979617 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb68003b90 con 0x7fcb7815c250 2024-04-02T13:22:19.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/3846979617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb780cca60 con 0x7fcb7815c250 2024-04-02T13:22:19.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb6f7fe640 1 -- 172.21.15.44:0/3846979617 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb68004030 con 0x7fcb7815c250 2024-04-02T13:22:19.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.133+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/3846979617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb780c7b50 con 0x7fcb7815c250 2024-04-02T13:22:19.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.134+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/3846979617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb780c7d50 con 0x7fcb7815c250 2024-04-02T13:22:19.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.135+0000 7fcb6f7fe640 1 -- 172.21.15.44:0/3846979617 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcb6801d070 con 0x7fcb7815c250 2024-04-02T13:22:19.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.135+0000 7fcb6f7fe640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb5806c8f0 0x7fcb5806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.135+0000 7fcb6f7fe640 1 -- 172.21.15.44:0/3846979617 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7fcb6808ba90 con 0x7fcb7815c250 2024-04-02T13:22:19.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.135+0000 7fcb76ffd640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb5806c8f0 0x7fcb5806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.136+0000 7fcb76ffd640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb5806c8f0 0x7fcb5806edb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcb70036000 tx=0x7fcb7003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:19.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.139+0000 7fcb6f7fe640 1 -- 172.21.15.44:0/3846979617 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fcb68010070 con 0x7fcb7815c250 2024-04-02T13:22:19.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.399+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/3846979617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fcb78068c00 con 0x7fcb7815c250 2024-04-02T13:22:19.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.400+0000 7fcb6f7fe640 1 -- 172.21.15.44:0/3846979617 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) ==== 80+0+5304 (secure 0 0 0) 0x7fcb68059ac0 con 0x7fcb7815c250 2024-04-02T13:22:19.402 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:19.402 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:22:19.402 INFO:teuthology.orchestra.run.smithi044.stdout: "auth_dump": [ 2024-04-02T13:22:19.402 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.402 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.a", 2024-04-02T13:22:19.402 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmVhZrMhAAklqx5uKCFA5kiqYwyLBKpg==", 2024-04-02T13:22:19.402 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.402 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:22:19.403 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:22:19.403 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:22:19.403 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.403 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.403 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.403 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.403 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.b", 2024-04-02T13:22:19.403 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmalQ+AxAA2SqH9CX/66OIXR7SDID97w==", 2024-04-02T13:22:19.403 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.403 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:22:19.403 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:22:19.403 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:22:19.403 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.404 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.404 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.404 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.404 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.c", 2024-04-02T13:22:19.404 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxm8BLKORAAjMIrmdQ/bXbPk2sAnYJWTw==", 2024-04-02T13:22:19.404 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.404 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:22:19.404 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:22:19.404 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:22:19.404 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.404 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.404 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.404 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.405 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.d", 2024-04-02T13:22:19.405 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmNoaRChAAEqxWmBjBWG79Tkj6dU8FQg==", 2024-04-02T13:22:19.405 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.405 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:22:19.405 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:22:19.405 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:22:19.405 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.405 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.405 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.405 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.405 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.0", 2024-04-02T13:22:19.405 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDfBQxm4GP9KxAA/AmOunAfVaczAWRGVXvFVg==", 2024-04-02T13:22:19.405 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.406 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:22:19.406 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:22:19.406 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.406 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.406 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.406 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.406 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.1", 2024-04-02T13:22:19.406 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDiBQxmDL/nKRAADF2C30+IGAXL9XwDQDKCUA==", 2024-04-02T13:22:19.406 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.406 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:22:19.406 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:22:19.406 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.407 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.407 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.407 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.407 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.2", 2024-04-02T13:22:19.407 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDlBQxm1gOCKBAAuYxN9jNHuRYEqc8WrsYErQ==", 2024-04-02T13:22:19.407 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.407 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:22:19.407 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:22:19.407 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.407 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.407 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.407 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.408 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.3", 2024-04-02T13:22:19.408 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDoBQxmLLFQJxAAFMCzphGU7TOt4nC0Nt+y+g==", 2024-04-02T13:22:19.408 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.408 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:22:19.408 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:22:19.408 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.408 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.408 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.408 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.408 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.4", 2024-04-02T13:22:19.408 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDuBQxmtk8QLhAAR4gm0irR6C3D2SkhFjvDKQ==", 2024-04-02T13:22:19.408 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.408 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:22:19.409 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:22:19.409 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.409 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.409 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.409 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.409 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.5", 2024-04-02T13:22:19.409 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDxBQxm6Ee/LBAAX5h+9ML70ztw2QH/50OzOg==", 2024-04-02T13:22:19.409 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.409 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:22:19.409 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:22:19.409 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.409 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.410 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.410 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.410 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.6", 2024-04-02T13:22:19.410 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD0BQxm35hwKxAAvsgMmiXdLo+c7wlvb9RtVQ==", 2024-04-02T13:22:19.410 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.410 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:22:19.410 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:22:19.410 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.410 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.410 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.410 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.410 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.7", 2024-04-02T13:22:19.410 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD3BQxmYLBOKhAA1vuSOBDcFf2zT78eoTHtHQ==", 2024-04-02T13:22:19.411 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.411 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:22:19.411 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:22:19.411 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.411 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.411 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.411 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.411 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.0", 2024-04-02T13:22:19.411 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A==", 2024-04-02T13:22:19.411 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.411 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:22:19.411 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow r", 2024-04-02T13:22:19.411 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T13:22:19.412 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.412 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.412 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.412 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.1", 2024-04-02T13:22:19.412 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxm3sYsGBAAeRCNbP5qiQ8YnRVsDoaaFQ==", 2024-04-02T13:22:19.412 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.412 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:22:19.412 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:22:19.412 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:22:19.412 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:22:19.412 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.412 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.412 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.413 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.2", 2024-04-02T13:22:19.413 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmKNS5FBAAKvsCNmuG+fI1D/vwu6MC5w==", 2024-04-02T13:22:19.413 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.413 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:22:19.413 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:22:19.413 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:22:19.413 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:22:19.413 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.413 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.413 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.413 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.3", 2024-04-02T13:22:19.413 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmslsSHhAAPsAfALW7RimJDyXdOa2wbQ==", 2024-04-02T13:22:19.413 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.414 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:22:19.414 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:22:19.414 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:22:19.414 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:22:19.414 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.414 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.414 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.414 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.admin", 2024-04-02T13:22:19.414 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmq+VcFRAA9CgeH75Q0sXf27cZdJthJg==", 2024-04-02T13:22:19.414 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.414 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:22:19.414 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:22:19.415 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:22:19.415 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.415 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.415 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.415 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.415 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.x", 2024-04-02T13:22:19.415 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmR5ICLxAAtQeg/gq0OAq6e0mroBZVxQ==", 2024-04-02T13:22:19.415 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.415 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:22:19.415 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:22:19.415 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.415 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.416 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:19.416 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:19.416 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.y", 2024-04-02T13:22:19.416 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmxoquKhAAE5fMnJIuCy5hGJ1aEAx8Vg==", 2024-04-02T13:22:19.416 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:22:19.416 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:22:19.416 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:22:19.416 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:22:19.416 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.416 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:19.416 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-02T13:22:19.416 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:22:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.402+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb5806c8f0 msgr2=0x7fcb5806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.402+0000 7fcb7d2f2640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb5806c8f0 0x7fcb5806edb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcb70036000 tx=0x7fcb7003d040 comp rx=0 tx=0).stop 2024-04-02T13:22:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.402+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/3846979617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb7815c250 msgr2=0x7fcb780ce910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.402+0000 7fcb7d2f2640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb7815c250 0x7fcb780ce910 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcb68007ef0 tx=0x7fcb6800d4a0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.402+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/3846979617 shutdown_connections 2024-04-02T13:22:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.402+0000 7fcb7d2f2640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb5806c8f0 0x7fcb5806edb0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.402+0000 7fcb7d2f2640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb7815c250 0x7fcb780ce910 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.402+0000 7fcb7d2f2640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb780d08d0 0x7fcb780b73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.402+0000 7fcb7d2f2640 1 --2- 172.21.15.44:0/3846979617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb780b3510 0x7fcb78151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.402+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/3846979617 >> 172.21.15.44:0/3846979617 conn(0x7fcb78073850 msgr2=0x7fcb780b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:19.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.402+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/3846979617 shutdown_connections 2024-04-02T13:22:19.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.402+0000 7fcb7d2f2640 1 -- 172.21.15.44:0/3846979617 wait complete. 2024-04-02T13:22:19.419 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:22:19.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.519+0000 7f95ec9fe640 1 Processor -- start 2024-04-02T13:22:19.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.519+0000 7f95ec9fe640 1 -- start start 2024-04-02T13:22:19.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.520+0000 7f95ec9fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f95e80d0980 0x7f95e80d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.520+0000 7f95ec9fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e80b0c70 0x7f95e80af060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.520+0000 7f95ec9fe640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e80cfda0 con 0x7f95e815c2b0 2024-04-02T13:22:19.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.520+0000 7f95ec9fe640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f95e80cee00 con 0x7f95e80d0980 2024-04-02T13:22:19.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.520+0000 7f95ec9fe640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f95e80cf0a0 con 0x7f95e80b0c70 2024-04-02T13:22:19.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.520+0000 7f95e659c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f95e80d0980 0x7f95e80d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.520+0000 7f95e6d9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f95e815c2b0 0x7f95e80d0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:39054/0 (socket says 172.21.15.44:39054) 2024-04-02T13:22:19.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.520+0000 7f95e759e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e80b0c70 0x7f95e80af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.520+0000 7f95e6d9d640 1 -- 172.21.15.44:0/1212949779 learned_addr learned my addr 172.21.15.44:0/1212949779 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:19.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.521+0000 7f95e5d9b640 1 -- 172.21.15.44:0/1212949779 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3155136613 0 0) 0x7f95d8001110 con 0x7f95e815c2b0 2024-04-02T13:22:19.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.521+0000 7f95e5d9b640 1 -- 172.21.15.44:0/1212949779 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95d0003520 con 0x7f95e815c2b0 2024-04-02T13:22:19.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.521+0000 7f95e759e640 1 -- 172.21.15.44:0/1212949779 >> v2:172.21.15.69:3300/0 conn(0x7f95e80d0980 msgr2=0x7f95e80d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:19.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.521+0000 7f95e759e640 1 --2- 172.21.15.44:0/1212949779 >> v2:172.21.15.69:3300/0 conn(0x7f95e80d0980 0x7f95e80d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.521+0000 7f95e759e640 1 -- 172.21.15.44:0/1212949779 >> v1:172.21.15.69:6789/0 conn(0x7f95e815c2b0 legacy=0x7f95e80d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.521+0000 7f95e759e640 1 -- 172.21.15.44:0/1212949779 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e80cf380 con 0x7f95e80b0c70 2024-04-02T13:22:19.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.521+0000 7f95e659c640 1 --2- 172.21.15.44:0/1212949779 >> v2:172.21.15.69:3300/0 conn(0x7f95e80d0980 0x7f95e80d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:22:19.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.522+0000 7f95e759e640 1 --2- 172.21.15.44:0/1212949779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e80b0c70 0x7f95e80af060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95e0009d90 tx=0x7f95e002bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd9afd6c3e7ee89b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:19.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.522+0000 7f95e5d9b640 1 -- 172.21.15.44:0/1212949779 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0039070 con 0x7f95e80b0c70 2024-04-02T13:22:19.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.522+0000 7f95e5d9b640 1 -- 172.21.15.44:0/1212949779 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e002cb30 con 0x7f95e80b0c70 2024-04-02T13:22:19.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.522+0000 7f95e5d9b640 1 -- 172.21.15.44:0/1212949779 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0034440 con 0x7f95e80b0c70 2024-04-02T13:22:19.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.522+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1212949779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e80b0c70 msgr2=0x7f95e80af060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.522+0000 7f95ec9fe640 1 --2- 172.21.15.44:0/1212949779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e80b0c70 0x7f95e80af060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95e0009d90 tx=0x7f95e002bd80 comp rx=0 tx=0).stop 2024-04-02T13:22:19.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.522+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1212949779 shutdown_connections 2024-04-02T13:22:19.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.522+0000 7f95ec9fe640 1 --2- 172.21.15.44:0/1212949779 >> v2:172.21.15.69:3300/0 conn(0x7f95e80d0980 0x7f95e80d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.522+0000 7f95ec9fe640 1 --2- 172.21.15.44:0/1212949779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e80b0c70 0x7f95e80af060 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.522+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1212949779 >> 172.21.15.44:0/1212949779 conn(0x7f95e80d1c90 msgr2=0x7f95e80d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:19.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.522+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1212949779 shutdown_connections 2024-04-02T13:22:19.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.522+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1212949779 wait complete. 2024-04-02T13:22:19.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.524+0000 7f95ec9fe640 1 Processor -- start 2024-04-02T13:22:19.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.524+0000 7f95ec9fe640 1 -- start start 2024-04-02T13:22:19.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95ec9fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e80b0c70 0x7f95e80b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95ec9fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95e80d0980 0x7f95e80b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95ec9fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e815c2b0 0x7f95e815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95ec9fe640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f95e80af930 con 0x7f95e80b0c70 2024-04-02T13:22:19.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95ec9fe640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f95e80af7b0 con 0x7f95e80d0980 2024-04-02T13:22:19.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95ec9fe640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f95e80afab0 con 0x7f95e815c2b0 2024-04-02T13:22:19.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95e6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e80b0c70 0x7f95e80b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95e6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e80b0c70 0x7f95e80b51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60140/0 (socket says 172.21.15.44:60140) 2024-04-02T13:22:19.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95e6d9d640 1 -- 172.21.15.44:0/1599489122 learned_addr learned my addr 172.21.15.44:0/1599489122 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:19.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95e759e640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e815c2b0 0x7f95e815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95e659c640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95e80d0980 0x7f95e80b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95e6d9d640 1 -- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e815c2b0 msgr2=0x7f95e815cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95e6d9d640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e815c2b0 0x7f95e815cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95e6d9d640 1 -- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95e80d0980 msgr2=0x7f95e80b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95e6d9d640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95e80d0980 0x7f95e80b2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95e6d9d640 1 -- 172.21.15.44:0/1599489122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e80b6090 con 0x7f95e80b0c70 2024-04-02T13:22:19.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95e759e640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e815c2b0 0x7f95e815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:19.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95e659c640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95e80d0980 0x7f95e80b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:19.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.525+0000 7f95e6d9d640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e80b0c70 0x7f95e80b51d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f95d800c3d0 tx=0x7f95d800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:19.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.526+0000 7f95cf7fe640 1 -- 172.21.15.44:0/1599489122 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d8017070 con 0x7f95e80b0c70 2024-04-02T13:22:19.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.526+0000 7f95cf7fe640 1 -- 172.21.15.44:0/1599489122 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d8005ce0 con 0x7f95e80b0c70 2024-04-02T13:22:19.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.526+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1599489122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e80b5ad0 con 0x7f95e80b0c70 2024-04-02T13:22:19.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.526+0000 7f95cf7fe640 1 -- 172.21.15.44:0/1599489122 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d8007310 con 0x7f95e80b0c70 2024-04-02T13:22:19.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.526+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1599489122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e815c730 con 0x7f95e80b0c70 2024-04-02T13:22:19.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.527+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1599489122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e804f8d0 con 0x7f95e80b0c70 2024-04-02T13:22:19.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.527+0000 7f95cf7fe640 1 -- 172.21.15.44:0/1599489122 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f95d801a070 con 0x7f95e80b0c70 2024-04-02T13:22:19.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.528+0000 7f95cf7fe640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f95d406c8f0 0x7f95d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.528+0000 7f95cf7fe640 1 -- 172.21.15.44:0/1599489122 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7f95d808ace0 con 0x7f95e80b0c70 2024-04-02T13:22:19.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.528+0000 7f95e659c640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f95d406c8f0 0x7f95d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.528+0000 7f95e659c640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f95d406c8f0 0x7f95d406edb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f95e8158460 tx=0x7f95dc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:19.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.532+0000 7f95cf7fe640 1 -- 172.21.15.44:0/1599489122 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f95e80cfda0 con 0x7f95e80b0c70 2024-04-02T13:22:19.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.717+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1599489122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f95e80673d0 con 0x7f95e80b0c70 2024-04-02T13:22:19.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.717+0000 7f95cf7fe640 1 -- 172.21.15.44:0/1599489122 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+13116 (secure 0 0 0) 0x7f95d8059360 con 0x7f95e80b0c70 2024-04-02T13:22:19.719 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:19.719 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":48,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:22:19.011538+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:22:19.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.719+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f95d406c8f0 msgr2=0x7f95d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.719+0000 7f95ec9fe640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f95d406c8f0 0x7f95d406edb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f95e8158460 tx=0x7f95dc008040 comp rx=0 tx=0).stop 2024-04-02T13:22:19.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.719+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1599489122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e80b0c70 msgr2=0x7f95e80b51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.719+0000 7f95ec9fe640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e80b0c70 0x7f95e80b51d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f95d800c3d0 tx=0x7f95d800c890 comp rx=0 tx=0).stop 2024-04-02T13:22:19.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.719+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1599489122 shutdown_connections 2024-04-02T13:22:19.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.719+0000 7f95ec9fe640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f95d406c8f0 0x7f95d406edb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.719+0000 7f95ec9fe640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e815c2b0 0x7f95e815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.719+0000 7f95ec9fe640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95e80d0980 0x7f95e80b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.719+0000 7f95ec9fe640 1 --2- 172.21.15.44:0/1599489122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e80b0c70 0x7f95e80b51d0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.719+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1599489122 >> 172.21.15.44:0/1599489122 conn(0x7f95e80d1c90 msgr2=0x7f95e80d23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:19.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.720+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1599489122 shutdown_connections 2024-04-02T13:22:19.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.720+0000 7f95ec9fe640 1 -- 172.21.15.44:0/1599489122 wait complete. 2024-04-02T13:22:19.732 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:22:19.733 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:22:19.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.835+0000 7f2f5bed3640 1 Processor -- start 2024-04-02T13:22:19.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.835+0000 7f2f5bed3640 1 -- start start 2024-04-02T13:22:19.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.836+0000 7f2f5bed3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2f5415e1d0 0x7f2f540d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.836+0000 7f2f5bed3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f540d1640 con 0x7f2f540d2dd0 2024-04-02T13:22:19.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.836+0000 7f2f5bed3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f540cf1f0 con 0x7f2f540cec60 2024-04-02T13:22:19.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.836+0000 7f2f5bed3640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2f540cfe00 con 0x7f2f5415e1d0 2024-04-02T13:22:19.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.836+0000 7f2f59c6f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2f5415e1d0 0x7f2f540d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.836+0000 7f2f5a470640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f2f540d2dd0 0x7f2f540b4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53340/0 (socket says 172.21.15.44:53340) 2024-04-02T13:22:19.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.836+0000 7f2f5a470640 1 -- 172.21.15.44:0/3529102809 learned_addr learned my addr 172.21.15.44:0/3529102809 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:19.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.837+0000 7f2f59c6f640 1 -- 172.21.15.44:0/3529102809 >> v1:172.21.15.69:6789/0 conn(0x7f2f540cec60 legacy=0x7f2f540d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.837+0000 7f2f59c6f640 1 -- 172.21.15.44:0/3529102809 >> v1:172.21.15.44:6789/0 conn(0x7f2f540d2dd0 legacy=0x7f2f540b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.837+0000 7f2f58c6d640 1 -- 172.21.15.44:0/3529102809 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 424830167 0 0) 0x7f2f540d1640 con 0x7f2f540d2dd0 2024-04-02T13:22:19.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.837+0000 7f2f59c6f640 1 -- 172.21.15.44:0/3529102809 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f540cf4d0 con 0x7f2f5415e1d0 2024-04-02T13:22:19.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.837+0000 7f2f59c6f640 1 --2- 172.21.15.44:0/3529102809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f5415e1d0 0x7f2f540d1100 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2f44009d90 tx=0x7f2f4402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798337a9456da5a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:19.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.838+0000 7f2f58c6d640 1 -- 172.21.15.44:0/3529102809 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f44039070 con 0x7f2f5415e1d0 2024-04-02T13:22:19.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.838+0000 7f2f58c6d640 1 -- 172.21.15.44:0/3529102809 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f4402cc60 con 0x7f2f5415e1d0 2024-04-02T13:22:19.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.838+0000 7f2f58c6d640 1 -- 172.21.15.44:0/3529102809 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f44034660 con 0x7f2f5415e1d0 2024-04-02T13:22:19.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.838+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3529102809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f5415e1d0 msgr2=0x7f2f540d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.838+0000 7f2f5bed3640 1 --2- 172.21.15.44:0/3529102809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f5415e1d0 0x7f2f540d1100 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2f44009d90 tx=0x7f2f4402beb0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.838+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3529102809 shutdown_connections 2024-04-02T13:22:19.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.838+0000 7f2f5bed3640 1 --2- 172.21.15.44:0/3529102809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f5415e1d0 0x7f2f540d1100 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.838+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3529102809 >> 172.21.15.44:0/3529102809 conn(0x7f2f5409d3b0 msgr2=0x7f2f540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:19.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.838+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3529102809 shutdown_connections 2024-04-02T13:22:19.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.838+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3529102809 wait complete. 2024-04-02T13:22:19.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.839+0000 7f2f5bed3640 1 Processor -- start 2024-04-02T13:22:19.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.839+0000 7f2f5bed3640 1 -- start start 2024-04-02T13:22:19.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.840+0000 7f2f5bed3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f540cec60 0x7f2f54157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.840+0000 7f2f5bed3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f540d2dd0 0x7f2f540cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.840+0000 7f2f5bed3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f541587e0 0x7f2f54157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.840+0000 7f2f5bed3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2f540b7fd0 con 0x7f2f540d2dd0 2024-04-02T13:22:19.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.840+0000 7f2f5bed3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2f540cfa90 con 0x7f2f540cec60 2024-04-02T13:22:19.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.840+0000 7f2f5bed3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2f540b8150 con 0x7f2f541587e0 2024-04-02T13:22:19.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.840+0000 7f2f5946e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f540d2dd0 0x7f2f540cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.840+0000 7f2f5946e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f540d2dd0 0x7f2f540cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60156/0 (socket says 172.21.15.44:60156) 2024-04-02T13:22:19.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.840+0000 7f2f5946e640 1 -- 172.21.15.44:0/3168327158 learned_addr learned my addr 172.21.15.44:0/3168327158 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:19.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.840+0000 7f2f59c6f640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f540cec60 0x7f2f54157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.840+0000 7f2f5a470640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f541587e0 0x7f2f54157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.841+0000 7f2f5946e640 1 -- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f541587e0 msgr2=0x7f2f54157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.841+0000 7f2f5946e640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f541587e0 0x7f2f54157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.841+0000 7f2f5946e640 1 -- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f540cec60 msgr2=0x7f2f54157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.841+0000 7f2f5946e640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f540cec60 0x7f2f54157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.841+0000 7f2f5946e640 1 -- 172.21.15.44:0/3168327158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f54158330 con 0x7f2f540d2dd0 2024-04-02T13:22:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.841+0000 7f2f5946e640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f540d2dd0 0x7f2f540cc8b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800c400 tx=0x7f2f4800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.841+0000 7f2f527fc640 1 -- 172.21.15.44:0/3168327158 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f48017070 con 0x7f2f540d2dd0 2024-04-02T13:22:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.841+0000 7f2f527fc640 1 -- 172.21.15.44:0/3168327158 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f480052d0 con 0x7f2f540d2dd0 2024-04-02T13:22:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.841+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3168327158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f541f3f30 con 0x7f2f540d2dd0 2024-04-02T13:22:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.841+0000 7f2f527fc640 1 -- 172.21.15.44:0/3168327158 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f48003830 con 0x7f2f540d2dd0 2024-04-02T13:22:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.841+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3168327158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f541f44d0 con 0x7f2f540d2dd0 2024-04-02T13:22:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.842+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3168327158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f540684f0 con 0x7f2f540d2dd0 2024-04-02T13:22:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.842+0000 7f2f527fc640 1 -- 172.21.15.44:0/3168327158 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2f4801a070 con 0x7f2f540d2dd0 2024-04-02T13:22:19.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.843+0000 7f2f527fc640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f4006c920 0x7f2f4006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:19.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.843+0000 7f2f527fc640 1 -- 172.21.15.44:0/3168327158 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7f2f4808ac10 con 0x7f2f540d2dd0 2024-04-02T13:22:19.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.843+0000 7f2f59c6f640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f4006c920 0x7f2f4006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:19.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.843+0000 7f2f59c6f640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f4006c920 0x7f2f4006ede0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2f44036000 tx=0x7f2f44030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:19.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:19.847+0000 7f2f527fc640 1 -- 172.21.15.44:0/3168327158 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2f48058bb0 con 0x7f2f540d2dd0 2024-04-02T13:22:20.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.032+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3168327158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2f540d36c0 con 0x7f2f540d2dd0 2024-04-02T13:22:20.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.032+0000 7f2f527fc640 1 -- 172.21.15.44:0/3168327158 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7f2f4805ceb0 con 0x7f2f540d2dd0 2024-04-02T13:22:20.034 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:22:20.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.034+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f4006c920 msgr2=0x7f2f4006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:20.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.034+0000 7f2f5bed3640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f4006c920 0x7f2f4006ede0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2f44036000 tx=0x7f2f44030000 comp rx=0 tx=0).stop 2024-04-02T13:22:20.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.034+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3168327158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f540d2dd0 msgr2=0x7f2f540cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:20.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.034+0000 7f2f5bed3640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f540d2dd0 0x7f2f540cc8b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800c400 tx=0x7f2f4800c8c0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.034+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3168327158 shutdown_connections 2024-04-02T13:22:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.034+0000 7f2f5bed3640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f4006c920 0x7f2f4006ede0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.034+0000 7f2f5bed3640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f541587e0 0x7f2f54157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.034+0000 7f2f5bed3640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f540d2dd0 0x7f2f540cc8b0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.034+0000 7f2f5bed3640 1 --2- 172.21.15.44:0/3168327158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f540cec60 0x7f2f54157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.034+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3168327158 >> 172.21.15.44:0/3168327158 conn(0x7f2f5409d3b0 msgr2=0x7f2f5406c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.034+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3168327158 shutdown_connections 2024-04-02T13:22:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.034+0000 7f2f5bed3640 1 -- 172.21.15.44:0/3168327158 wait complete. 2024-04-02T13:22:20.047 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T13:22:20.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.147+0000 7f3d21ddb640 1 Processor -- start 2024-04-02T13:22:20.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.147+0000 7f3d21ddb640 1 -- start start 2024-04-02T13:22:20.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.148+0000 7f3d21ddb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d1c15c240 0x7f3d1c0d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:20.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.148+0000 7f3d21ddb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3d1c0cfc10 0x7f3d1c0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:20.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.148+0000 7f3d21ddb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d1c0d2c80 con 0x7f3d1c0ce4f0 2024-04-02T13:22:20.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.148+0000 7f3d21ddb640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f3d1c0c5980 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.148+0000 7f3d21ddb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3d1c0c5c60 con 0x7f3d1c15c240 2024-04-02T13:22:20.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.148+0000 7f3d1affd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3d1c0cfc10 0x7f3d1c0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:20.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.148+0000 7f3d1affd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3d1c0cfc10 0x7f3d1c0d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60158/0 (socket says 172.21.15.44:60158) 2024-04-02T13:22:20.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.148+0000 7f3d1affd640 1 -- 172.21.15.44:0/491093184 learned_addr learned my addr 172.21.15.44:0/491093184 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:20.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.148+0000 7f3d1b7fe640 1 --2- 172.21.15.44:0/491093184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d1c15c240 0x7f3d1c0d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:20.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.148+0000 7f3d1affd640 1 -- 172.21.15.44:0/491093184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d1c15c240 msgr2=0x7f3d1c0d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:20.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.148+0000 7f3d1affd640 1 --2- 172.21.15.44:0/491093184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d1c15c240 0x7f3d1c0d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.148+0000 7f3d1affd640 1 -- 172.21.15.44:0/491093184 >> v1:172.21.15.69:6789/0 conn(0x7f3d1c0ce4f0 legacy=0x7f3d1c0ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:20.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.148+0000 7f3d1affd640 1 -- 172.21.15.44:0/491093184 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1c0c5f40 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.149+0000 7f3d1affd640 1 --2- 172.21.15.44:0/491093184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d1c0cfc10 0x7f3d1c0d0950 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009870 tx=0x7f3d1002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd7f9afe5350634 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:20.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.149+0000 7f3d1a7fc640 1 -- 172.21.15.44:0/491093184 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10034070 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.149+0000 7f3d1a7fc640 1 -- 172.21.15.44:0/491093184 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d10030840 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.149+0000 7f3d1a7fc640 1 -- 172.21.15.44:0/491093184 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10030c10 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.150+0000 7f3d21ddb640 1 -- 172.21.15.44:0/491093184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d1c0cfc10 msgr2=0x7f3d1c0d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:20.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.150+0000 7f3d21ddb640 1 --2- 172.21.15.44:0/491093184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d1c0cfc10 0x7f3d1c0d0950 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009870 tx=0x7f3d1002b980 comp rx=0 tx=0).stop 2024-04-02T13:22:20.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.150+0000 7f3d21ddb640 1 -- 172.21.15.44:0/491093184 shutdown_connections 2024-04-02T13:22:20.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.150+0000 7f3d21ddb640 1 --2- 172.21.15.44:0/491093184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d1c0cfc10 0x7f3d1c0d0950 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.150+0000 7f3d21ddb640 1 --2- 172.21.15.44:0/491093184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d1c15c240 0x7f3d1c0d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.150+0000 7f3d21ddb640 1 -- 172.21.15.44:0/491093184 >> 172.21.15.44:0/491093184 conn(0x7f3d1c0d1e30 msgr2=0x7f3d1c0d2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:20.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.150+0000 7f3d21ddb640 1 -- 172.21.15.44:0/491093184 shutdown_connections 2024-04-02T13:22:20.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.150+0000 7f3d21ddb640 1 -- 172.21.15.44:0/491093184 wait complete. 2024-04-02T13:22:20.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.151+0000 7f3d21ddb640 1 Processor -- start 2024-04-02T13:22:20.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.151+0000 7f3d21ddb640 1 -- start start 2024-04-02T13:22:20.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d21ddb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d1c0ce4f0 0x7f3d1c09d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:20.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d21ddb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d1c0cfc10 0x7f3d1c09db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:20.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d21ddb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d1c15c240 0x7f3d1c09e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:20.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d21ddb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c15c620 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d21ddb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c0d2950 con 0x7f3d1c15c240 2024-04-02T13:22:20.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d21ddb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3d1c0cf090 con 0x7f3d1c0ce4f0 2024-04-02T13:22:20.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d1affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d1c0cfc10 0x7f3d1c09db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:20.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d1affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d1c0cfc10 0x7f3d1c09db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60166/0 (socket says 172.21.15.44:60166) 2024-04-02T13:22:20.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d1affd640 1 -- 172.21.15.44:0/4226358459 learned_addr learned my addr 172.21.15.44:0/4226358459 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:20.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d1affd640 1 -- 172.21.15.44:0/4226358459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d1c0ce4f0 msgr2=0x7f3d1c09d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:20.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d1affd640 1 --2- 172.21.15.44:0/4226358459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d1c0ce4f0 0x7f3d1c09d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d1affd640 1 -- 172.21.15.44:0/4226358459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d1c15c240 msgr2=0x7f3d1c09e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:20.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d1affd640 1 --2- 172.21.15.44:0/4226358459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d1c15c240 0x7f3d1c09e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d1affd640 1 -- 172.21.15.44:0/4226358459 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1c1f4470 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d1affd640 1 --2- 172.21.15.44:0/4226358459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d1c0cfc10 0x7f3d1c09db70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3d10000c00 tx=0x7f3d100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:20.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d03fff640 1 -- 172.21.15.44:0/4226358459 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10034040 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d03fff640 1 -- 172.21.15.44:0/4226358459 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d10039d90 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d21ddb640 1 -- 172.21.15.44:0/4226358459 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d1c09ea20 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.152+0000 7f3d03fff640 1 -- 172.21.15.44:0/4226358459 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10043610 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.153+0000 7f3d21ddb640 1 -- 172.21.15.44:0/4226358459 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d1c09efe0 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.154+0000 7f3d21ddb640 1 -- 172.21.15.44:0/4226358459 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d1c0d1620 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.154+0000 7f3d03fff640 1 -- 172.21.15.44:0/4226358459 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d1004a060 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.154+0000 7f3d03fff640 1 --2- 172.21.15.44:0/4226358459 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d0806c8f0 0x7f3d0806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:20.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.155+0000 7f3d03fff640 1 -- 172.21.15.44:0/4226358459 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7f3d100ad9e0 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.155+0000 7f3d1b7fe640 1 --2- 172.21.15.44:0/4226358459 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d0806c8f0 0x7f3d0806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:20.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.155+0000 7f3d1b7fe640 1 --2- 172.21.15.44:0/4226358459 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d0806c8f0 0x7f3d0806edb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c006fd0 tx=0x7f3d0c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:20.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.159+0000 7f3d03fff640 1 -- 172.21.15.44:0/4226358459 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3d1c0d1620 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.344+0000 7f3d21ddb640 1 -- 172.21.15.44:0/4226358459 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f3d1c1f4640 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.344+0000 7f3d03fff640 1 -- 172.21.15.44:0/4226358459 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v48) ==== 84+0+65 (secure 0 0 0) 0x7f3d1003e070 con 0x7f3d1c0cfc10 2024-04-02T13:22:20.346 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:20.346 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-04-02T13:22:20.346 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:20.346 INFO:teuthology.orchestra.run.smithi044.stdout: "poolnum": 1, 2024-04-02T13:22:20.346 INFO:teuthology.orchestra.run.smithi044.stdout: "poolname": ".mgr" 2024-04-02T13:22:20.347 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:20.347 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-04-02T13:22:20.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.346+0000 7f3d21ddb640 1 -- 172.21.15.44:0/4226358459 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d0806c8f0 msgr2=0x7f3d0806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:20.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.346+0000 7f3d21ddb640 1 --2- 172.21.15.44:0/4226358459 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d0806c8f0 0x7f3d0806edb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c006fd0 tx=0x7f3d0c008040 comp rx=0 tx=0).stop 2024-04-02T13:22:20.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.346+0000 7f3d21ddb640 1 -- 172.21.15.44:0/4226358459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d1c0cfc10 msgr2=0x7f3d1c09db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:20.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.346+0000 7f3d21ddb640 1 --2- 172.21.15.44:0/4226358459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d1c0cfc10 0x7f3d1c09db70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3d10000c00 tx=0x7f3d100046c0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.346+0000 7f3d21ddb640 1 -- 172.21.15.44:0/4226358459 shutdown_connections 2024-04-02T13:22:20.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.346+0000 7f3d21ddb640 1 --2- 172.21.15.44:0/4226358459 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d0806c8f0 0x7f3d0806edb0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.346+0000 7f3d21ddb640 1 --2- 172.21.15.44:0/4226358459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d1c15c240 0x7f3d1c09e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.346+0000 7f3d21ddb640 1 --2- 172.21.15.44:0/4226358459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d1c0cfc10 0x7f3d1c09db70 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.346+0000 7f3d21ddb640 1 --2- 172.21.15.44:0/4226358459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d1c0ce4f0 0x7f3d1c09d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.346+0000 7f3d21ddb640 1 -- 172.21.15.44:0/4226358459 >> 172.21.15.44:0/4226358459 conn(0x7f3d1c0d1e30 msgr2=0x7f3d1c070c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:20.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.347+0000 7f3d21ddb640 1 -- 172.21.15.44:0/4226358459 shutdown_connections 2024-04-02T13:22:20.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.347+0000 7f3d21ddb640 1 -- 172.21.15.44:0/4226358459 wait complete. 2024-04-02T13:22:20.359 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T13:22:20.359 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T13:22:20.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.463+0000 7f3db4c86640 1 Processor -- start 2024-04-02T13:22:20.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.463+0000 7f3db4c86640 1 -- start start 2024-04-02T13:22:20.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.463+0000 7f3db4c86640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3db01554c0 0x7f3db0071120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:20.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.464+0000 7f3db4c86640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3db00d0840 0x7f3db0164c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:20.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.464+0000 7f3db4c86640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3db00cdc20 con 0x7f3db00ce9a0 2024-04-02T13:22:20.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.464+0000 7f3db4c86640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f3db00cdec0 con 0x7f3db00d0840 2024-04-02T13:22:20.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.464+0000 7f3db4c86640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3db00ce1a0 con 0x7f3db01554c0 2024-04-02T13:22:20.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.464+0000 7f3daed9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3db00d0840 0x7f3db0164c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:20.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.464+0000 7f3daed9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3db00d0840 0x7f3db0164c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60172/0 (socket says 172.21.15.44:60172) 2024-04-02T13:22:20.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.464+0000 7f3daed9d640 1 -- 172.21.15.44:0/430790223 learned_addr learned my addr 172.21.15.44:0/430790223 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:20.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.464+0000 7f3dae59c640 1 --2- 172.21.15.44:0/430790223 >> v2:172.21.15.69:3300/0 conn(0x7f3db01554c0 0x7f3db0071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:20.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.464+0000 7f3daed9d640 1 -- 172.21.15.44:0/430790223 >> v2:172.21.15.69:3300/0 conn(0x7f3db01554c0 msgr2=0x7f3db0071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:20.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.464+0000 7f3daed9d640 1 --2- 172.21.15.44:0/430790223 >> v2:172.21.15.69:3300/0 conn(0x7f3db01554c0 0x7f3db0071120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.464+0000 7f3daed9d640 1 -- 172.21.15.44:0/430790223 >> v1:172.21.15.69:6789/0 conn(0x7f3db00ce9a0 legacy=0x7f3db00d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.464+0000 7f3daed9d640 1 -- 172.21.15.44:0/430790223 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db00ce480 con 0x7f3db00d0840 2024-04-02T13:22:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.464+0000 7f3dae59c640 1 --2- 172.21.15.44:0/430790223 >> v2:172.21.15.69:3300/0 conn(0x7f3db01554c0 0x7f3db0071120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:20.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.465+0000 7f3daed9d640 1 --2- 172.21.15.44:0/430790223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3db00d0840 0x7f3db0164c10 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f3db00af8b0 tx=0x7f3da802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e99392bc0bb9c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:20.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.465+0000 7f3dad59a640 1 -- 172.21.15.44:0/430790223 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da802cb20 con 0x7f3db00d0840 2024-04-02T13:22:20.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.465+0000 7f3dad59a640 1 -- 172.21.15.44:0/430790223 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da802ccc0 con 0x7f3db00d0840 2024-04-02T13:22:20.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.465+0000 7f3dad59a640 1 -- 172.21.15.44:0/430790223 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8034610 con 0x7f3db00d0840 2024-04-02T13:22:20.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.465+0000 7f3db4c86640 1 -- 172.21.15.44:0/430790223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3db00d0840 msgr2=0x7f3db0164c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:20.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.465+0000 7f3db4c86640 1 --2- 172.21.15.44:0/430790223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3db00d0840 0x7f3db0164c10 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f3db00af8b0 tx=0x7f3da802bf80 comp rx=0 tx=0).stop 2024-04-02T13:22:20.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.465+0000 7f3db4c86640 1 -- 172.21.15.44:0/430790223 shutdown_connections 2024-04-02T13:22:20.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.465+0000 7f3db4c86640 1 --2- 172.21.15.44:0/430790223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3db00d0840 0x7f3db0164c10 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.465+0000 7f3db4c86640 1 --2- 172.21.15.44:0/430790223 >> v2:172.21.15.69:3300/0 conn(0x7f3db01554c0 0x7f3db0071120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.465+0000 7f3db4c86640 1 -- 172.21.15.44:0/430790223 >> 172.21.15.44:0/430790223 conn(0x7f3db006b920 msgr2=0x7f3db006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:20.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.466+0000 7f3db4c86640 1 -- 172.21.15.44:0/430790223 shutdown_connections 2024-04-02T13:22:20.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.466+0000 7f3db4c86640 1 -- 172.21.15.44:0/430790223 wait complete. 2024-04-02T13:22:20.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.467+0000 7f3db4c86640 1 Processor -- start 2024-04-02T13:22:20.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.467+0000 7f3db4c86640 1 -- start start 2024-04-02T13:22:20.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.467+0000 7f3db4c86640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3db00ce9a0 0x7f3db0156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:20.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.467+0000 7f3db4c86640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3db00d0840 0x7f3db0158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:20.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.467+0000 7f3db4c86640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3db01554c0 0x7f3db014d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:20.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3db4c86640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3db015c290 con 0x7f3db00d0840 2024-04-02T13:22:20.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3db4c86640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3db0071be0 con 0x7f3db00ce9a0 2024-04-02T13:22:20.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3db4c86640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3db01f50a0 con 0x7f3db01554c0 2024-04-02T13:22:20.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3dadd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3db00d0840 0x7f3db0158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:20.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3dadd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3db00d0840 0x7f3db0158580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60176/0 (socket says 172.21.15.44:60176) 2024-04-02T13:22:20.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3dadd9b640 1 -- 172.21.15.44:0/3682404631 learned_addr learned my addr 172.21.15.44:0/3682404631 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:20.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3dadd9b640 1 -- 172.21.15.44:0/3682404631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3db01554c0 msgr2=0x7f3db014d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:20.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3dadd9b640 1 --2- 172.21.15.44:0/3682404631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3db01554c0 0x7f3db014d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3dadd9b640 1 -- 172.21.15.44:0/3682404631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3db00ce9a0 msgr2=0x7f3db0156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:20.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3dadd9b640 1 --2- 172.21.15.44:0/3682404631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3db00ce9a0 0x7f3db0156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:20.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3dadd9b640 1 -- 172.21.15.44:0/3682404631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db01f4420 con 0x7f3db00d0840 2024-04-02T13:22:20.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3dadd9b640 1 --2- 172.21.15.44:0/3682404631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3db00d0840 0x7f3db0158580 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3da400c430 tx=0x7f3da400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:20.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3d9effd640 1 -- 172.21.15.44:0/3682404631 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da4017070 con 0x7f3db00d0840 2024-04-02T13:22:20.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3d9effd640 1 -- 172.21.15.44:0/3682404631 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da40052d0 con 0x7f3db00d0840 2024-04-02T13:22:20.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3db4c86640 1 -- 172.21.15.44:0/3682404631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db01f4700 con 0x7f3db00d0840 2024-04-02T13:22:20.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3d9effd640 1 -- 172.21.15.44:0/3682404631 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da4003850 con 0x7f3db00d0840 2024-04-02T13:22:20.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.468+0000 7f3db4c86640 1 -- 172.21.15.44:0/3682404631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db014dbb0 con 0x7f3db00d0840 2024-04-02T13:22:20.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.469+0000 7f3db4c86640 1 -- 172.21.15.44:0/3682404631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3db00684f0 con 0x7f3db00d0840 2024-04-02T13:22:20.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.470+0000 7f3d9effd640 1 -- 172.21.15.44:0/3682404631 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3da401a070 con 0x7f3db00d0840 2024-04-02T13:22:20.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.470+0000 7f3d9effd640 1 --2- 172.21.15.44:0/3682404631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d9006c8f0 0x7f3d9006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:20.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.470+0000 7f3d9effd640 1 -- 172.21.15.44:0/3682404631 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7f3da408ad60 con 0x7f3db00d0840 2024-04-02T13:22:20.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.471+0000 7f3dae59c640 1 --2- 172.21.15.44:0/3682404631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d9006c8f0 0x7f3d9006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:20.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.471+0000 7f3dae59c640 1 --2- 172.21.15.44:0/3682404631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d9006c8f0 0x7f3d9006edb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3da0006e80 tx=0x7f3da0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:20.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.475+0000 7f3d9effd640 1 -- 172.21.15.44:0/3682404631 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3da4059340 con 0x7f3db00d0840 2024-04-02T13:22:20.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:20.659+0000 7f3db4c86640 1 -- 172.21.15.44:0/3682404631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3db00b5170 con 0x7f3db00d0840 2024-04-02T13:22:21.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.005+0000 7f3d9effd640 1 -- 172.21.15.44:0/3682404631 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v49) ==== 138+0+0 (secure 0 0 0) 0x7f3da4058c40 con 0x7f3db00d0840 2024-04-02T13:22:21.007 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' created 2024-04-02T13:22:21.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.007+0000 7f3db4c86640 1 -- 172.21.15.44:0/3682404631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d9006c8f0 msgr2=0x7f3d9006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:21.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.007+0000 7f3db4c86640 1 --2- 172.21.15.44:0/3682404631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d9006c8f0 0x7f3d9006edb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3da0006e80 tx=0x7f3da0008040 comp rx=0 tx=0).stop 2024-04-02T13:22:21.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.007+0000 7f3db4c86640 1 -- 172.21.15.44:0/3682404631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3db00d0840 msgr2=0x7f3db0158580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:21.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.007+0000 7f3db4c86640 1 --2- 172.21.15.44:0/3682404631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3db00d0840 0x7f3db0158580 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3da400c430 tx=0x7f3da400c8f0 comp rx=0 tx=0).stop 2024-04-02T13:22:21.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.007+0000 7f3db4c86640 1 -- 172.21.15.44:0/3682404631 shutdown_connections 2024-04-02T13:22:21.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.007+0000 7f3db4c86640 1 --2- 172.21.15.44:0/3682404631 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d9006c8f0 0x7f3d9006edb0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:21.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.007+0000 7f3db4c86640 1 --2- 172.21.15.44:0/3682404631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3db01554c0 0x7f3db014d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:21.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.007+0000 7f3db4c86640 1 --2- 172.21.15.44:0/3682404631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3db00d0840 0x7f3db0158580 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:21.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.007+0000 7f3db4c86640 1 --2- 172.21.15.44:0/3682404631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3db00ce9a0 0x7f3db0156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:21.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.008+0000 7f3db4c86640 1 -- 172.21.15.44:0/3682404631 >> 172.21.15.44:0/3682404631 conn(0x7f3db006b920 msgr2=0x7f3db00cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:21.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.008+0000 7f3db4c86640 1 -- 172.21.15.44:0/3682404631 shutdown_connections 2024-04-02T13:22:21.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.008+0000 7f3db4c86640 1 -- 172.21.15.44:0/3682404631 wait complete. 2024-04-02T13:22:21.024 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T13:22:21.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.134+0000 7f59a1055640 1 Processor -- start 2024-04-02T13:22:21.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.134+0000 7f59a1055640 1 -- start start 2024-04-02T13:22:21.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.134+0000 7f59a1055640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f599c15e5d0 0x7f599c0d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:21.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.134+0000 7f59a1055640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f599c0b4180 0x7f599c0b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:21.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.134+0000 7f59a1055640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f599c0d1630 con 0x7f599c06cff0 2024-04-02T13:22:21.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.134+0000 7f59a1055640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f599c0d1bf0 con 0x7f599c0b4180 2024-04-02T13:22:21.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.134+0000 7f59a1055640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f599c0d1910 con 0x7f599c15e5d0 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.135+0000 7f599b59e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f599c0b4180 0x7f599c0b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.135+0000 7f599ad9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f599c06cff0 0x7f599c06d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53344/0 (socket says 172.21.15.44:53344) 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.135+0000 7f599ad9d640 1 -- 172.21.15.44:0/1500931720 learned_addr learned my addr 172.21.15.44:0/1500931720 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.135+0000 7f5993fff640 1 --2- 172.21.15.44:0/1500931720 >> v2:172.21.15.69:3300/0 conn(0x7f599c15e5d0 0x7f599c0d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.135+0000 7f599b59e640 1 -- 172.21.15.44:0/1500931720 >> v2:172.21.15.69:3300/0 conn(0x7f599c15e5d0 msgr2=0x7f599c0d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.135+0000 7f599b59e640 1 --2- 172.21.15.44:0/1500931720 >> v2:172.21.15.69:3300/0 conn(0x7f599c15e5d0 0x7f599c0d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.135+0000 7f599b59e640 1 -- 172.21.15.44:0/1500931720 >> v1:172.21.15.44:6789/0 conn(0x7f599c06cff0 legacy=0x7f599c06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.135+0000 7f599b59e640 1 -- 172.21.15.44:0/1500931720 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f599c0b1540 con 0x7f599c0b4180 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.135+0000 7f599a59c640 1 -- 172.21.15.44:0/1500931720 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 124102461 0 0) 0x7f599c0d1630 con 0x7f599c06cff0 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.135+0000 7f5993fff640 1 --2- 172.21.15.44:0/1500931720 >> v2:172.21.15.69:3300/0 conn(0x7f599c15e5d0 0x7f599c0d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.136+0000 7f599b59e640 1 --2- 172.21.15.44:0/1500931720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0b4180 0x7f599c0b1d40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f599c161e40 tx=0x7f599402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b928c90364c5a45 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.136+0000 7f599a59c640 1 -- 172.21.15.44:0/1500931720 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5994039070 con 0x7f599c0b4180 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.136+0000 7f599a59c640 1 -- 172.21.15.44:0/1500931720 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f599402cbf0 con 0x7f599c0b4180 2024-04-02T13:22:21.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.136+0000 7f599a59c640 1 -- 172.21.15.44:0/1500931720 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5994034440 con 0x7f599c0b4180 2024-04-02T13:22:21.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.136+0000 7f59a1055640 1 -- 172.21.15.44:0/1500931720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0b4180 msgr2=0x7f599c0b1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:21.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.136+0000 7f59a1055640 1 --2- 172.21.15.44:0/1500931720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0b4180 0x7f599c0b1d40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f599c161e40 tx=0x7f599402be40 comp rx=0 tx=0).stop 2024-04-02T13:22:21.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.136+0000 7f59a1055640 1 -- 172.21.15.44:0/1500931720 shutdown_connections 2024-04-02T13:22:21.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.136+0000 7f59a1055640 1 --2- 172.21.15.44:0/1500931720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0b4180 0x7f599c0b1d40 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:21.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.136+0000 7f59a1055640 1 --2- 172.21.15.44:0/1500931720 >> v2:172.21.15.69:3300/0 conn(0x7f599c15e5d0 0x7f599c0d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:21.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.136+0000 7f59a1055640 1 -- 172.21.15.44:0/1500931720 >> 172.21.15.44:0/1500931720 conn(0x7f599c09aa20 msgr2=0x7f599c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:21.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.136+0000 7f59a1055640 1 -- 172.21.15.44:0/1500931720 shutdown_connections 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.137+0000 7f59a1055640 1 -- 172.21.15.44:0/1500931720 wait complete. 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.138+0000 7f59a1055640 1 Processor -- start 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.138+0000 7f59a1055640 1 -- start start 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.138+0000 7f59a1055640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f599c06cff0 0x7f599c14f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.138+0000 7f59a1055640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0b4180 0x7f599c0c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.138+0000 7f59a1055640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f599c15e5d0 0x7f599c15fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.138+0000 7f59a1055640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f599c0d13f0 con 0x7f599c0b4180 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.138+0000 7f59a1055640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f599c0d1270 con 0x7f599c06cff0 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.138+0000 7f59a1055640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f599c0b2280 con 0x7f599c15e5d0 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f5993fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0b4180 0x7f599c0c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f5993fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0b4180 0x7f599c0c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60182/0 (socket says 172.21.15.44:60182) 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f5993fff640 1 -- 172.21.15.44:0/2898487594 learned_addr learned my addr 172.21.15.44:0/2898487594 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f599ad9d640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f599c06cff0 0x7f599c14f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:21.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f599b59e640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f599c15e5d0 0x7f599c15fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f5993fff640 1 -- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f599c15e5d0 msgr2=0x7f599c15fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f5993fff640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f599c15e5d0 0x7f599c15fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f5993fff640 1 -- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f599c06cff0 msgr2=0x7f599c14f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f5993fff640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f599c06cff0 0x7f599c14f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f5993fff640 1 -- 172.21.15.44:0/2898487594 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f599c0c6ae0 con 0x7f599c0b4180 2024-04-02T13:22:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f5993fff640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0b4180 0x7f599c0c7b80 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f598400c7c0 tx=0x7f598400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f59937fe640 1 -- 172.21.15.44:0/2898487594 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5984015070 con 0x7f599c0b4180 2024-04-02T13:22:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f59937fe640 1 -- 172.21.15.44:0/2898487594 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5984007820 con 0x7f599c0b4180 2024-04-02T13:22:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f59a1055640 1 -- 172.21.15.44:0/2898487594 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f599c0c6dc0 con 0x7f599c0b4180 2024-04-02T13:22:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f59937fe640 1 -- 172.21.15.44:0/2898487594 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5984007bf0 con 0x7f599c0b4180 2024-04-02T13:22:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.139+0000 7f59a1055640 1 -- 172.21.15.44:0/2898487594 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f599c0c7380 con 0x7f599c0b4180 2024-04-02T13:22:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.140+0000 7f59a1055640 1 -- 172.21.15.44:0/2898487594 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f599c0684f0 con 0x7f599c0b4180 2024-04-02T13:22:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.141+0000 7f59937fe640 1 -- 172.21.15.44:0/2898487594 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5984018070 con 0x7f599c0b4180 2024-04-02T13:22:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.141+0000 7f59937fe640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f597c06c8f0 0x7f597c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.141+0000 7f59937fe640 1 -- 172.21.15.44:0/2898487594 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5619+0+0 (secure 0 0 0) 0x7f598408b8a0 con 0x7f599c0b4180 2024-04-02T13:22:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.142+0000 7f599ad9d640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f597c06c8f0 0x7f597c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:21.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.142+0000 7f599ad9d640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f597c06c8f0 0x7f597c06edb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f598c000f60 tx=0x7f598c007d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:21.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.146+0000 7f59937fe640 1 -- 172.21.15.44:0/2898487594 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5984059760 con 0x7f599c0b4180 2024-04-02T13:22:21.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:21.328+0000 7f59a1055640 1 -- 172.21.15.44:0/2898487594 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f599c06cdd0 con 0x7f599c0b4180 2024-04-02T13:22:21.954 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:22:21.957 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:22:22.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.007+0000 7f59937fe640 1 -- 172.21.15.44:0/2898487594 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v50) ==== 170+0+0 (secure 0 0 0) 0x7f598405da60 con 0x7f599c0b4180 2024-04-02T13:22:22.009 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' created 2024-04-02T13:22:22.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.010+0000 7f59a1055640 1 -- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f597c06c8f0 msgr2=0x7f597c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:22.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.010+0000 7f59a1055640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f597c06c8f0 0x7f597c06edb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f598c000f60 tx=0x7f598c007d50 comp rx=0 tx=0).stop 2024-04-02T13:22:22.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.010+0000 7f59a1055640 1 -- 172.21.15.44:0/2898487594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0b4180 msgr2=0x7f599c0c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:22.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.010+0000 7f59a1055640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0b4180 0x7f599c0c7b80 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f598400c7c0 tx=0x7f598400cc80 comp rx=0 tx=0).stop 2024-04-02T13:22:22.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.010+0000 7f59a1055640 1 -- 172.21.15.44:0/2898487594 shutdown_connections 2024-04-02T13:22:22.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.010+0000 7f59a1055640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f597c06c8f0 0x7f597c06edb0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:22.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.010+0000 7f59a1055640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f599c15e5d0 0x7f599c15fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:22.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.010+0000 7f59a1055640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0b4180 0x7f599c0c7b80 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:22.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.010+0000 7f59a1055640 1 --2- 172.21.15.44:0/2898487594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f599c06cff0 0x7f599c14f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:22.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.010+0000 7f59a1055640 1 -- 172.21.15.44:0/2898487594 >> 172.21.15.44:0/2898487594 conn(0x7f599c09aa20 msgr2=0x7f599c0b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:22.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.010+0000 7f59a1055640 1 -- 172.21.15.44:0/2898487594 shutdown_connections 2024-04-02T13:22:22.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.011+0000 7f59a1055640 1 -- 172.21.15.44:0/2898487594 wait complete. 2024-04-02T13:22:22.038 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T13:22:22.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.183+0000 7f7b7839d640 1 Processor -- start 2024-04-02T13:22:22.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.183+0000 7f7b7839d640 1 -- start start 2024-04-02T13:22:22.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.184+0000 7f7b7839d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7b700b1540 0x7f7b700b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:22.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.184+0000 7f7b7839d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7b700ce790 0x7f7b7006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:22.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.184+0000 7f7b7839d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b700ced00 con 0x7f7b7015e470 2024-04-02T13:22:22.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.184+0000 7f7b7839d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7b700cefe0 con 0x7f7b700ce790 2024-04-02T13:22:22.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.184+0000 7f7b7839d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7b700cf2c0 con 0x7f7b700b1540 2024-04-02T13:22:22.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.184+0000 7f7b7693a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7b700ce790 0x7f7b7006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:22.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.184+0000 7f7b7693a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7b700ce790 0x7f7b7006d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60198/0 (socket says 172.21.15.44:60198) 2024-04-02T13:22:22.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.184+0000 7f7b7693a640 1 -- 172.21.15.44:0/699432222 learned_addr learned my addr 172.21.15.44:0/699432222 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:22.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.184+0000 7f7b75938640 1 --2- 172.21.15.44:0/699432222 >> v2:172.21.15.69:3300/0 conn(0x7f7b700b1540 0x7f7b700b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:22.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.185+0000 7f7b7693a640 1 -- 172.21.15.44:0/699432222 >> v2:172.21.15.69:3300/0 conn(0x7f7b700b1540 msgr2=0x7f7b700b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:22.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.185+0000 7f7b7693a640 1 --2- 172.21.15.44:0/699432222 >> v2:172.21.15.69:3300/0 conn(0x7f7b700b1540 0x7f7b700b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:22.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.185+0000 7f7b7693a640 1 -- 172.21.15.44:0/699432222 >> v1:172.21.15.69:6789/0 conn(0x7f7b7015e470 legacy=0x7f7b700d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:22.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.185+0000 7f7b7693a640 1 -- 172.21.15.44:0/699432222 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b700cf5a0 con 0x7f7b700ce790 2024-04-02T13:22:22.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.185+0000 7f7b7693a640 1 --2- 172.21.15.44:0/699432222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b700ce790 0x7f7b7006d0f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7b70163070 tx=0x7f7b6002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d5e794fd598763 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:22.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.185+0000 7f7b75137640 1 -- 172.21.15.44:0/699432222 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6002c9f0 con 0x7f7b700ce790 2024-04-02T13:22:22.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.185+0000 7f7b75137640 1 -- 172.21.15.44:0/699432222 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b6002cb90 con 0x7f7b700ce790 2024-04-02T13:22:22.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.185+0000 7f7b75137640 1 -- 172.21.15.44:0/699432222 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b600343f0 con 0x7f7b700ce790 2024-04-02T13:22:22.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.186+0000 7f7b7839d640 1 -- 172.21.15.44:0/699432222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b700ce790 msgr2=0x7f7b7006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:22.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.186+0000 7f7b7839d640 1 --2- 172.21.15.44:0/699432222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b700ce790 0x7f7b7006d0f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7b70163070 tx=0x7f7b6002be50 comp rx=0 tx=0).stop 2024-04-02T13:22:22.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.186+0000 7f7b7839d640 1 -- 172.21.15.44:0/699432222 shutdown_connections 2024-04-02T13:22:22.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.186+0000 7f7b7839d640 1 --2- 172.21.15.44:0/699432222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b700ce790 0x7f7b7006d0f0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:22.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.186+0000 7f7b7839d640 1 --2- 172.21.15.44:0/699432222 >> v2:172.21.15.69:3300/0 conn(0x7f7b700b1540 0x7f7b700b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:22.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.186+0000 7f7b7839d640 1 -- 172.21.15.44:0/699432222 >> 172.21.15.44:0/699432222 conn(0x7f7b7009aa20 msgr2=0x7f7b7009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:22.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.186+0000 7f7b7839d640 1 -- 172.21.15.44:0/699432222 shutdown_connections 2024-04-02T13:22:22.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.186+0000 7f7b7839d640 1 -- 172.21.15.44:0/699432222 wait complete. 2024-04-02T13:22:22.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.187+0000 7f7b7839d640 1 Processor -- start 2024-04-02T13:22:22.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.187+0000 7f7b7839d640 1 -- start start 2024-04-02T13:22:22.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b7839d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b700b1540 0x7f7b70152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:22.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b7839d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b700ce790 0x7f7b700c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:22.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b7839d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b7015e470 0x7f7b7006d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:22.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b7839d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7b700d22c0 con 0x7f7b700b1540 2024-04-02T13:22:22.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b7839d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7006cb60 con 0x7f7b700ce790 2024-04-02T13:22:22.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b7839d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7b700d2440 con 0x7f7b7015e470 2024-04-02T13:22:22.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b76139640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b700b1540 0x7f7b70152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b76139640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b700b1540 0x7f7b70152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60200/0 (socket says 172.21.15.44:60200) 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b76139640 1 -- 172.21.15.44:0/2461530513 learned_addr learned my addr 172.21.15.44:0/2461530513 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b75938640 1 --2- 172.21.15.44:0/2461530513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b700ce790 0x7f7b700c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b76139640 1 -- 172.21.15.44:0/2461530513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b7015e470 msgr2=0x7f7b7006d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b76139640 1 --2- 172.21.15.44:0/2461530513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b7015e470 0x7f7b7006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b76139640 1 -- 172.21.15.44:0/2461530513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b700ce790 msgr2=0x7f7b700c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b76139640 1 --2- 172.21.15.44:0/2461530513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b700ce790 0x7f7b700c59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b76139640 1 -- 172.21.15.44:0/2461530513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b70152860 con 0x7f7b700b1540 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.188+0000 7f7b76139640 1 --2- 172.21.15.44:0/2461530513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b700b1540 0x7f7b70152320 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800b3f0 tx=0x7f7b6800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.189+0000 7f7b6effd640 1 -- 172.21.15.44:0/2461530513 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b68015020 con 0x7f7b700b1540 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.189+0000 7f7b6effd640 1 -- 172.21.15.44:0/2461530513 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b68011830 con 0x7f7b700b1540 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.189+0000 7f7b7839d640 1 -- 172.21.15.44:0/2461530513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b7015ecb0 con 0x7f7b700b1540 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.189+0000 7f7b6effd640 1 -- 172.21.15.44:0/2461530513 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b68011c20 con 0x7f7b700b1540 2024-04-02T13:22:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.189+0000 7f7b7839d640 1 -- 172.21.15.44:0/2461530513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b7015f270 con 0x7f7b700b1540 2024-04-02T13:22:22.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.190+0000 7f7b7839d640 1 -- 172.21.15.44:0/2461530513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b700684f0 con 0x7f7b700b1540 2024-04-02T13:22:22.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.190+0000 7f7b6effd640 1 -- 172.21.15.44:0/2461530513 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7b68018020 con 0x7f7b700b1540 2024-04-02T13:22:22.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.191+0000 7f7b6effd640 1 --2- 172.21.15.44:0/2461530513 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b5c06c8f0 0x7f7b5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:22.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.191+0000 7f7b6effd640 1 -- 172.21.15.44:0/2461530513 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6001+0+0 (secure 0 0 0) 0x7f7b6808b830 con 0x7f7b700b1540 2024-04-02T13:22:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.191+0000 7f7b75938640 1 --2- 172.21.15.44:0/2461530513 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b5c06c8f0 0x7f7b5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.192+0000 7f7b75938640 1 --2- 172.21.15.44:0/2461530513 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b5c06c8f0 0x7f7b5c06edb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7b7015bf60 tx=0x7f7b64008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:22.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.195+0000 7f7b6effd640 1 -- 172.21.15.44:0/2461530513 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7b6801f030 con 0x7f7b700b1540 2024-04-02T13:22:22.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:22.421+0000 7f7b7839d640 1 -- 172.21.15.44:0/2461530513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f7b700d2d10 con 0x7f7b700b1540 2024-04-02T13:22:23.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.043+0000 7f7b6effd640 1 -- 172.21.15.44:0/2461530513 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v23) ==== 172+0+0 (secure 0 0 0) 0x7f7b680172c0 con 0x7f7b700b1540 2024-04-02T13:22:23.045 INFO:teuthology.orchestra.run.smithi044.stderr:new fs with metadata pool 7 and data pool 8 2024-04-02T13:22:23.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.046+0000 7f7b7839d640 1 -- 172.21.15.44:0/2461530513 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b5c06c8f0 msgr2=0x7f7b5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.046+0000 7f7b7839d640 1 --2- 172.21.15.44:0/2461530513 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b5c06c8f0 0x7f7b5c06edb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7b7015bf60 tx=0x7f7b64008040 comp rx=0 tx=0).stop 2024-04-02T13:22:23.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.046+0000 7f7b7839d640 1 -- 172.21.15.44:0/2461530513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b700b1540 msgr2=0x7f7b70152320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.047+0000 7f7b7839d640 1 --2- 172.21.15.44:0/2461530513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b700b1540 0x7f7b70152320 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800b3f0 tx=0x7f7b6800b8b0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.047+0000 7f7b7839d640 1 -- 172.21.15.44:0/2461530513 shutdown_connections 2024-04-02T13:22:23.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.047+0000 7f7b7839d640 1 --2- 172.21.15.44:0/2461530513 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b5c06c8f0 0x7f7b5c06edb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.047+0000 7f7b7839d640 1 --2- 172.21.15.44:0/2461530513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b7015e470 0x7f7b7006d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.047+0000 7f7b7839d640 1 --2- 172.21.15.44:0/2461530513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b700ce790 0x7f7b700c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.047+0000 7f7b7839d640 1 --2- 172.21.15.44:0/2461530513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b700b1540 0x7f7b70152320 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.047+0000 7f7b7839d640 1 -- 172.21.15.44:0/2461530513 >> 172.21.15.44:0/2461530513 conn(0x7f7b7009aa20 msgr2=0x7f7b700b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:23.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.047+0000 7f7b7839d640 1 -- 172.21.15.44:0/2461530513 shutdown_connections 2024-04-02T13:22:23.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.047+0000 7f7b7839d640 1 -- 172.21.15.44:0/2461530513 wait complete. 2024-04-02T13:22:23.066 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:22:23.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.170+0000 7f070d6c8640 1 Processor -- start 2024-04-02T13:22:23.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.170+0000 7f070d6c8640 1 -- start start 2024-04-02T13:22:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f070d6c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f070815e550 0x7f070806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f070d6c8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f070815fb50 con 0x7f070806d530 2024-04-02T13:22:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f070d6c8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f070806ca90 con 0x7f070815f040 2024-04-02T13:22:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f070d6c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f070806d970 con 0x7f070815e550 2024-04-02T13:22:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f07067fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f070806d530 0x7f070815e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53350/0 (socket says 172.21.15.44:53350) 2024-04-02T13:22:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f07067fc640 1 -- 172.21.15.44:0/3472006773 learned_addr learned my addr 172.21.15.44:0/3472006773 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f0706ffd640 1 --2- 172.21.15.44:0/3472006773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f070815e550 0x7f070806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f0705ffb640 1 -- 172.21.15.44:0/3472006773 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2701731848 0 0) 0x7f070815fb50 con 0x7f070806d530 2024-04-02T13:22:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f0705ffb640 1 -- 172.21.15.44:0/3472006773 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06f4003520 con 0x7f070806d530 2024-04-02T13:22:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f0705ffb640 1 -- 172.21.15.44:0/3472006773 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2548785331 0 0) 0x7f070806ca90 con 0x7f070815f040 2024-04-02T13:22:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f0705ffb640 1 -- 172.21.15.44:0/3472006773 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f070815fb50 con 0x7f070815f040 2024-04-02T13:22:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f0706ffd640 1 -- 172.21.15.44:0/3472006773 >> v1:172.21.15.69:6789/0 conn(0x7f070815f040 legacy=0x7f07080d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f0706ffd640 1 -- 172.21.15.44:0/3472006773 >> v1:172.21.15.44:6789/0 conn(0x7f070806d530 legacy=0x7f070815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.171+0000 7f0706ffd640 1 -- 172.21.15.44:0/3472006773 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f070815f590 con 0x7f070815e550 2024-04-02T13:22:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.172+0000 7f0706ffd640 1 --2- 172.21.15.44:0/3472006773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f070815e550 0x7f070806cff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f06f8009dc0 tx=0x7f06f802bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65296d61710290 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.172+0000 7f0705ffb640 1 -- 172.21.15.44:0/3472006773 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f8039070 con 0x7f070815e550 2024-04-02T13:22:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.172+0000 7f0705ffb640 1 -- 172.21.15.44:0/3472006773 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06f802cd50 con 0x7f070815e550 2024-04-02T13:22:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.172+0000 7f0705ffb640 1 -- 172.21.15.44:0/3472006773 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f8034640 con 0x7f070815e550 2024-04-02T13:22:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.173+0000 7f070d6c8640 1 -- 172.21.15.44:0/3472006773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f070815e550 msgr2=0x7f070806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.173+0000 7f070d6c8640 1 --2- 172.21.15.44:0/3472006773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f070815e550 0x7f070806cff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f06f8009dc0 tx=0x7f06f802bfa0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.173+0000 7f070d6c8640 1 -- 172.21.15.44:0/3472006773 shutdown_connections 2024-04-02T13:22:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.173+0000 7f070d6c8640 1 --2- 172.21.15.44:0/3472006773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f070815e550 0x7f070806cff0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.173+0000 7f070d6c8640 1 -- 172.21.15.44:0/3472006773 >> 172.21.15.44:0/3472006773 conn(0x7f070809aa20 msgr2=0x7f070809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:23.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.173+0000 7f070d6c8640 1 -- 172.21.15.44:0/3472006773 shutdown_connections 2024-04-02T13:22:23.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.173+0000 7f070d6c8640 1 -- 172.21.15.44:0/3472006773 wait complete. 2024-04-02T13:22:23.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.174+0000 7f070d6c8640 1 Processor -- start 2024-04-02T13:22:23.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.174+0000 7f070d6c8640 1 -- start start 2024-04-02T13:22:23.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f070d6c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f070806d530 0x7f07081484e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f070d6c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f070815e550 0x7f0708157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f070d6c8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f070815f040 0x7f07081577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f070d6c8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f07080d0590 con 0x7f070815e550 2024-04-02T13:22:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f070d6c8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f07080d02b0 con 0x7f070815f040 2024-04-02T13:22:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f070d6c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f07080cc4b0 con 0x7f070806d530 2024-04-02T13:22:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f07067fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f070815e550 0x7f0708157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f07067fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f070815e550 0x7f0708157280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60208/0 (socket says 172.21.15.44:60208) 2024-04-02T13:22:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f07067fc640 1 -- 172.21.15.44:0/2425043369 learned_addr learned my addr 172.21.15.44:0/2425043369 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f0706ffd640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f070806d530 0x7f07081484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f07077fe640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f070815f040 0x7f07081577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f07067fc640 1 -- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f070806d530 msgr2=0x7f07081484e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f07067fc640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f070806d530 0x7f07081484e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f07067fc640 1 -- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f070815f040 msgr2=0x7f07081577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f07067fc640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f070815f040 0x7f07081577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f07067fc640 1 -- 172.21.15.44:0/2425043369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07080b7240 con 0x7f070815e550 2024-04-02T13:22:23.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.175+0000 7f07077fe640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f070815f040 0x7f07081577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:23.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.176+0000 7f07067fc640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f070815e550 0x7f0708157280 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f06f000c440 tx=0x7f06f000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:23.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.176+0000 7f06ff7fe640 1 -- 172.21.15.44:0/2425043369 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f0017070 con 0x7f070815e550 2024-04-02T13:22:23.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.176+0000 7f06ff7fe640 1 -- 172.21.15.44:0/2425043369 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06f0005ce0 con 0x7f070815e550 2024-04-02T13:22:23.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.176+0000 7f070d6c8640 1 -- 172.21.15.44:0/2425043369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0708145db0 con 0x7f070815e550 2024-04-02T13:22:23.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.176+0000 7f06ff7fe640 1 -- 172.21.15.44:0/2425043369 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f0007270 con 0x7f070815e550 2024-04-02T13:22:23.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.176+0000 7f070d6c8640 1 -- 172.21.15.44:0/2425043369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0708146370 con 0x7f070815e550 2024-04-02T13:22:23.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.177+0000 7f070d6c8640 1 -- 172.21.15.44:0/2425043369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0708146570 con 0x7f070815e550 2024-04-02T13:22:23.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.177+0000 7f06ff7fe640 1 -- 172.21.15.44:0/2425043369 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f06f001a070 con 0x7f070815e550 2024-04-02T13:22:23.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.178+0000 7f06ff7fe640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f06e806c8f0 0x7f06e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.178+0000 7f06ff7fe640 1 -- 172.21.15.44:0/2425043369 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f06f008af20 con 0x7f070815e550 2024-04-02T13:22:23.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.178+0000 7f0706ffd640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f06e806c8f0 0x7f06e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.178+0000 7f0706ffd640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f06e806c8f0 0x7f06e806edb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f06f8036000 tx=0x7f06f8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:23.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.182+0000 7f06ff7fe640 1 -- 172.21.15.44:0/2425043369 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0708146570 con 0x7f070815e550 2024-04-02T13:22:23.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.368+0000 7f070d6c8640 1 -- 172.21.15.44:0/2425043369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0708145bc0 con 0x7f070815e550 2024-04-02T13:22:23.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.369+0000 7f06ff7fe640 1 -- 172.21.15.44:0/2425043369 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+16907 (secure 0 0 0) 0x7f06f0058b60 con 0x7f070815e550 2024-04-02T13:22:23.371 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:23.371 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":51,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:22:22.422084+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:22:20.662776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-02T13:22:21.332577+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:22:23.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.371+0000 7f070d6c8640 1 -- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f06e806c8f0 msgr2=0x7f06e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.371+0000 7f070d6c8640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f06e806c8f0 0x7f06e806edb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f06f8036000 tx=0x7f06f8030000 comp rx=0 tx=0).stop 2024-04-02T13:22:23.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.371+0000 7f070d6c8640 1 -- 172.21.15.44:0/2425043369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f070815e550 msgr2=0x7f0708157280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.371+0000 7f070d6c8640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f070815e550 0x7f0708157280 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f06f000c440 tx=0x7f06f000c900 comp rx=0 tx=0).stop 2024-04-02T13:22:23.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.371+0000 7f070d6c8640 1 -- 172.21.15.44:0/2425043369 shutdown_connections 2024-04-02T13:22:23.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.371+0000 7f070d6c8640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f06e806c8f0 0x7f06e806edb0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.371+0000 7f070d6c8640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f070815f040 0x7f07081577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.371+0000 7f070d6c8640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f070815e550 0x7f0708157280 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.371+0000 7f070d6c8640 1 --2- 172.21.15.44:0/2425043369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f070806d530 0x7f07081484e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.371+0000 7f070d6c8640 1 -- 172.21.15.44:0/2425043369 >> 172.21.15.44:0/2425043369 conn(0x7f070809aa20 msgr2=0x7f07080cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:23.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.371+0000 7f070d6c8640 1 -- 172.21.15.44:0/2425043369 shutdown_connections 2024-04-02T13:22:23.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.371+0000 7f070d6c8640 1 -- 172.21.15.44:0/2425043369 wait complete. 2024-04-02T13:22:23.384 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:22:23.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.484+0000 7faf00ad6640 1 Processor -- start 2024-04-02T13:22:23.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.484+0000 7faf00ad6640 1 -- start start 2024-04-02T13:22:23.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.485+0000 7faf00ad6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7faefc0cc8b0 0x7faefc0cab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.485+0000 7faf00ad6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faefc06fc60 con 0x7faefc0ccf00 2024-04-02T13:22:23.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.485+0000 7faf00ad6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faefc06ff00 con 0x7faefc0cbb50 2024-04-02T13:22:23.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.485+0000 7faf00ad6640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7faefc0cb100 con 0x7faefc0cc8b0 2024-04-02T13:22:23.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.485+0000 7faef3fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7faefc0cc8b0 0x7faefc0cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.485+0000 7faefa59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7faefc0ccf00 0x7faefc0cd2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53366/0 (socket says 172.21.15.44:53366) 2024-04-02T13:22:23.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.485+0000 7faefa59c640 1 -- 172.21.15.44:0/2301181314 learned_addr learned my addr 172.21.15.44:0/2301181314 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:23.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.485+0000 7faef3fff640 1 -- 172.21.15.44:0/2301181314 >> v1:172.21.15.69:6789/0 conn(0x7faefc0cbb50 legacy=0x7faefc0b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:23.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.485+0000 7faef3fff640 1 -- 172.21.15.44:0/2301181314 >> v1:172.21.15.44:6789/0 conn(0x7faefc0ccf00 legacy=0x7faefc0cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.485+0000 7faef3fff640 1 -- 172.21.15.44:0/2301181314 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faefc0701e0 con 0x7faefc0cc8b0 2024-04-02T13:22:23.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.485+0000 7faef9d9b640 1 -- 172.21.15.44:0/2301181314 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3872045164 0 0) 0x7faefc06fc60 con 0x7faefc0ccf00 2024-04-02T13:22:23.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.486+0000 7faef3fff640 1 --2- 172.21.15.44:0/2301181314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc0cc8b0 0x7faefc0cab40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7faee40097a0 tx=0x7faee402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1990c7a67a044581 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:23.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.486+0000 7faef9d9b640 1 -- 172.21.15.44:0/2301181314 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee4035020 con 0x7faefc0cc8b0 2024-04-02T13:22:23.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.486+0000 7faef9d9b640 1 -- 172.21.15.44:0/2301181314 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faee4031830 con 0x7faefc0cc8b0 2024-04-02T13:22:23.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.486+0000 7faef9d9b640 1 -- 172.21.15.44:0/2301181314 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee4031c00 con 0x7faefc0cc8b0 2024-04-02T13:22:23.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.487+0000 7faf00ad6640 1 -- 172.21.15.44:0/2301181314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc0cc8b0 msgr2=0x7faefc0cab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.487+0000 7faf00ad6640 1 --2- 172.21.15.44:0/2301181314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc0cc8b0 0x7faefc0cab40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7faee40097a0 tx=0x7faee402b8d0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.487+0000 7faf00ad6640 1 -- 172.21.15.44:0/2301181314 shutdown_connections 2024-04-02T13:22:23.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.487+0000 7faf00ad6640 1 --2- 172.21.15.44:0/2301181314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc0cc8b0 0x7faefc0cab40 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.487+0000 7faf00ad6640 1 -- 172.21.15.44:0/2301181314 >> 172.21.15.44:0/2301181314 conn(0x7faefc0b20e0 msgr2=0x7faefc0b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:23.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.487+0000 7faf00ad6640 1 -- 172.21.15.44:0/2301181314 shutdown_connections 2024-04-02T13:22:23.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.487+0000 7faf00ad6640 1 -- 172.21.15.44:0/2301181314 wait complete. 2024-04-02T13:22:23.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.488+0000 7faf00ad6640 1 Processor -- start 2024-04-02T13:22:23.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.488+0000 7faf00ad6640 1 -- start start 2024-04-02T13:22:23.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faf00ad6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc0cbb50 0x7faefc14d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faf00ad6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faefc0cc8b0 0x7faefc0c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faf00ad6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faefc0ccf00 0x7faefc0b5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faf00ad6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faefc0cc610 con 0x7faefc0cbb50 2024-04-02T13:22:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faf00ad6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faefc0cb6c0 con 0x7faefc0cc8b0 2024-04-02T13:22:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faf00ad6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faefc15c550 con 0x7faefc0ccf00 2024-04-02T13:22:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faefa59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc0cbb50 0x7faefc14d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faefa59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc0cbb50 0x7faefc14d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60230/0 (socket says 172.21.15.44:60230) 2024-04-02T13:22:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faefa59c640 1 -- 172.21.15.44:0/1322174363 learned_addr learned my addr 172.21.15.44:0/1322174363 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faef3fff640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faefc0cc8b0 0x7faefc0c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faefad9d640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faefc0ccf00 0x7faefc0b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faefa59c640 1 -- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faefc0ccf00 msgr2=0x7faefc0b5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faefa59c640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faefc0ccf00 0x7faefc0b5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faefa59c640 1 -- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faefc0cc8b0 msgr2=0x7faefc0c8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faefa59c640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faefc0cc8b0 0x7faefc0c8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faefa59c640 1 -- 172.21.15.44:0/1322174363 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faefc14dd70 con 0x7faefc0cbb50 2024-04-02T13:22:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faef3fff640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faefc0cc8b0 0x7faefc0c8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faefad9d640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faefc0ccf00 0x7faefc0b5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faefa59c640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc0cbb50 0x7faefc14d830 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7faeec00c3f0 tx=0x7faeec00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.489+0000 7faef2ffd640 1 -- 172.21.15.44:0/1322174363 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec0074e0 con 0x7faefc0cbb50 2024-04-02T13:22:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.490+0000 7faef2ffd640 1 -- 172.21.15.44:0/1322174363 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeec0047e0 con 0x7faefc0cbb50 2024-04-02T13:22:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.490+0000 7faf00ad6640 1 -- 172.21.15.44:0/1322174363 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faefc0c9060 con 0x7faefc0cbb50 2024-04-02T13:22:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.490+0000 7faef2ffd640 1 -- 172.21.15.44:0/1322174363 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec004c10 con 0x7faefc0cbb50 2024-04-02T13:22:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.490+0000 7faf00ad6640 1 -- 172.21.15.44:0/1322174363 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faefc0c9620 con 0x7faefc0cbb50 2024-04-02T13:22:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.490+0000 7faf00ad6640 1 -- 172.21.15.44:0/1322174363 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faefc0684f0 con 0x7faefc0cbb50 2024-04-02T13:22:23.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.491+0000 7faef2ffd640 1 -- 172.21.15.44:0/1322174363 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faeec011070 con 0x7faefc0cbb50 2024-04-02T13:22:23.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.491+0000 7faef2ffd640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faedc06c8f0 0x7faedc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.492+0000 7faef2ffd640 1 -- 172.21.15.44:0/1322174363 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7faeec08b8b0 con 0x7faefc0cbb50 2024-04-02T13:22:23.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.492+0000 7faef3fff640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faedc06c8f0 0x7faedc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.492+0000 7faef3fff640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faedc06c8f0 0x7faedc06edb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7faee4036000 tx=0x7faee4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:23.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.496+0000 7faef2ffd640 1 -- 172.21.15.44:0/1322174363 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7faeec01a2b0 con 0x7faefc0cbb50 2024-04-02T13:22:23.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.682+0000 7faf00ad6640 1 -- 172.21.15.44:0/1322174363 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faefc1599d0 con 0x7faefc0cbb50 2024-04-02T13:22:23.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.683+0000 7faef2ffd640 1 -- 172.21.15.44:0/1322174363 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+16907 (secure 0 0 0) 0x7faeec059b50 con 0x7faefc0cbb50 2024-04-02T13:22:23.685 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:23.685 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":51,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:22:22.422084+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:22:20.662776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-02T13:22:21.332577+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:22:23.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.685+0000 7faf00ad6640 1 -- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faedc06c8f0 msgr2=0x7faedc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.685+0000 7faf00ad6640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faedc06c8f0 0x7faedc06edb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7faee4036000 tx=0x7faee4008030 comp rx=0 tx=0).stop 2024-04-02T13:22:23.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.685+0000 7faf00ad6640 1 -- 172.21.15.44:0/1322174363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc0cbb50 msgr2=0x7faefc14d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.685+0000 7faf00ad6640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc0cbb50 0x7faefc14d830 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7faeec00c3f0 tx=0x7faeec00c8b0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.685+0000 7faf00ad6640 1 -- 172.21.15.44:0/1322174363 shutdown_connections 2024-04-02T13:22:23.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.685+0000 7faf00ad6640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faedc06c8f0 0x7faedc06edb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.685+0000 7faf00ad6640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faefc0ccf00 0x7faefc0b5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.685+0000 7faf00ad6640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faefc0cc8b0 0x7faefc0c8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.685+0000 7faf00ad6640 1 --2- 172.21.15.44:0/1322174363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc0cbb50 0x7faefc14d830 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.685+0000 7faf00ad6640 1 -- 172.21.15.44:0/1322174363 >> 172.21.15.44:0/1322174363 conn(0x7faefc0b20e0 msgr2=0x7faefc0b16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:23.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.685+0000 7faf00ad6640 1 -- 172.21.15.44:0/1322174363 shutdown_connections 2024-04-02T13:22:23.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.685+0000 7faf00ad6640 1 -- 172.21.15.44:0/1322174363 wait complete. 2024-04-02T13:22:23.698 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T13:22:23.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.802+0000 7f6e239a8640 1 Processor -- start 2024-04-02T13:22:23.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.802+0000 7f6e239a8640 1 -- start start 2024-04-02T13:22:23.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.803+0000 7f6e239a8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6e1c0d0c50 0x7f6e1c0d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.803+0000 7f6e239a8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6e1c06ce40 0x7f6e1c06d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.803+0000 7f6e239a8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e1c0cc600 con 0x7f6e1c15e4a0 2024-04-02T13:22:23.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.803+0000 7f6e239a8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f6e1c0af300 con 0x7f6e1c06ce40 2024-04-02T13:22:23.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.803+0000 7f6e239a8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6e1c0cc8e0 con 0x7f6e1c0d0c50 2024-04-02T13:22:23.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.803+0000 7f6e21f45640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6e1c06ce40 0x7f6e1c06d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.803+0000 7f6e21f45640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6e1c06ce40 0x7f6e1c06d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60244/0 (socket says 172.21.15.44:60244) 2024-04-02T13:22:23.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.803+0000 7f6e21f45640 1 -- 172.21.15.44:0/1134300842 learned_addr learned my addr 172.21.15.44:0/1134300842 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:23.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.803+0000 7f6e20f43640 1 --2- 172.21.15.44:0/1134300842 >> v2:172.21.15.69:3300/0 conn(0x7f6e1c0d0c50 0x7f6e1c0d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.803+0000 7f6e21f45640 1 -- 172.21.15.44:0/1134300842 >> v2:172.21.15.69:3300/0 conn(0x7f6e1c0d0c50 msgr2=0x7f6e1c0d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:23.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.803+0000 7f6e21f45640 1 --2- 172.21.15.44:0/1134300842 >> v2:172.21.15.69:3300/0 conn(0x7f6e1c0d0c50 0x7f6e1c0d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.804+0000 7f6e21f45640 1 -- 172.21.15.44:0/1134300842 >> v1:172.21.15.69:6789/0 conn(0x7f6e1c15e4a0 legacy=0x7f6e1c0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.804+0000 7f6e21f45640 1 -- 172.21.15.44:0/1134300842 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e1c0ccbc0 con 0x7f6e1c06ce40 2024-04-02T13:22:23.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.805+0000 7f6e21f45640 1 --2- 172.21.15.44:0/1134300842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e1c06ce40 0x7f6e1c06d2f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0b8060 tx=0x7f6e1802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1bdb452d43038f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:23.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.805+0000 7f6e13fff640 1 -- 172.21.15.44:0/1134300842 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1802cb20 con 0x7f6e1c06ce40 2024-04-02T13:22:23.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.805+0000 7f6e13fff640 1 -- 172.21.15.44:0/1134300842 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e1802ccc0 con 0x7f6e1c06ce40 2024-04-02T13:22:23.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.805+0000 7f6e13fff640 1 -- 172.21.15.44:0/1134300842 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e18034610 con 0x7f6e1c06ce40 2024-04-02T13:22:23.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.805+0000 7f6e239a8640 1 -- 172.21.15.44:0/1134300842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e1c06ce40 msgr2=0x7f6e1c06d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.805+0000 7f6e239a8640 1 --2- 172.21.15.44:0/1134300842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e1c06ce40 0x7f6e1c06d2f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0b8060 tx=0x7f6e1802bf80 comp rx=0 tx=0).stop 2024-04-02T13:22:23.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.805+0000 7f6e239a8640 1 -- 172.21.15.44:0/1134300842 shutdown_connections 2024-04-02T13:22:23.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.805+0000 7f6e239a8640 1 --2- 172.21.15.44:0/1134300842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e1c06ce40 0x7f6e1c06d2f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.805+0000 7f6e239a8640 1 --2- 172.21.15.44:0/1134300842 >> v2:172.21.15.69:3300/0 conn(0x7f6e1c0d0c50 0x7f6e1c0d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.805+0000 7f6e239a8640 1 -- 172.21.15.44:0/1134300842 >> 172.21.15.44:0/1134300842 conn(0x7f6e1c09aa20 msgr2=0x7f6e1c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:23.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.805+0000 7f6e239a8640 1 -- 172.21.15.44:0/1134300842 shutdown_connections 2024-04-02T13:22:23.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.805+0000 7f6e239a8640 1 -- 172.21.15.44:0/1134300842 wait complete. 2024-04-02T13:22:23.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.807+0000 7f6e239a8640 1 Processor -- start 2024-04-02T13:22:23.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.807+0000 7f6e239a8640 1 -- start start 2024-04-02T13:22:23.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.807+0000 7f6e239a8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e1c06ce40 0x7f6e1c1517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.807+0000 7f6e239a8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e1c0d0c50 0x7f6e1c0b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.807+0000 7f6e239a8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e1c15e4a0 0x7f6e1c0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.807+0000 7f6e239a8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1c0cd460 con 0x7f6e1c15e4a0 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.807+0000 7f6e239a8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1c0cd180 con 0x7f6e1c0d0c50 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.807+0000 7f6e239a8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6e1c15e880 con 0x7f6e1c06ce40 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.807+0000 7f6e21f45640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e1c15e4a0 0x7f6e1c0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.807+0000 7f6e21f45640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e1c15e4a0 0x7f6e1c0b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60256/0 (socket says 172.21.15.44:60256) 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.807+0000 7f6e21f45640 1 -- 172.21.15.44:0/3975910601 learned_addr learned my addr 172.21.15.44:0/3975910601 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e21f45640 1 -- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e1c06ce40 msgr2=0x7f6e1c1517f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e20f43640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e1c0d0c50 0x7f6e1c0b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e21744640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e1c06ce40 0x7f6e1c1517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e21f45640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e1c06ce40 0x7f6e1c1517f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e21f45640 1 -- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e1c0d0c50 msgr2=0x7f6e1c0b5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e21f45640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e1c0d0c50 0x7f6e1c0b5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e21f45640 1 -- 172.21.15.44:0/3975910601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e1c151d30 con 0x7f6e1c15e4a0 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e20f43640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e1c0d0c50 0x7f6e1c0b5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:23.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e21744640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e1c06ce40 0x7f6e1c1517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:23.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e21f45640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e1c15e4a0 0x7f6e1c0b5640 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6e18009f40 tx=0x7f6e18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:23.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e11ffb640 1 -- 172.21.15.44:0/3975910601 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e18043020 con 0x7f6e1c15e4a0 2024-04-02T13:22:23.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e11ffb640 1 -- 172.21.15.44:0/3975910601 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e1803e450 con 0x7f6e1c15e4a0 2024-04-02T13:22:23.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e239a8640 1 -- 172.21.15.44:0/3975910601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e1c15ed90 con 0x7f6e1c15e4a0 2024-04-02T13:22:23.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e11ffb640 1 -- 172.21.15.44:0/3975910601 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e18009510 con 0x7f6e1c15e4a0 2024-04-02T13:22:23.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.808+0000 7f6e239a8640 1 -- 172.21.15.44:0/3975910601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e1c0b5f50 con 0x7f6e1c15e4a0 2024-04-02T13:22:23.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.809+0000 7f6e239a8640 1 -- 172.21.15.44:0/3975910601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e1c0684f0 con 0x7f6e1c15e4a0 2024-04-02T13:22:23.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.810+0000 7f6e11ffb640 1 -- 172.21.15.44:0/3975910601 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6e18005d10 con 0x7f6e1c15e4a0 2024-04-02T13:22:23.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.810+0000 7f6e11ffb640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6e0006c8f0 0x7f6e0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:23.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.810+0000 7f6e11ffb640 1 -- 172.21.15.44:0/3975910601 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f6e180ae760 con 0x7f6e1c15e4a0 2024-04-02T13:22:23.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.810+0000 7f6e21744640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6e0006c8f0 0x7f6e0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:23.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.811+0000 7f6e21744640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6e0006c8f0 0x7f6e0006edb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6e140015e0 tx=0x7f6e14009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:23.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:23.814+0000 7f6e11ffb640 1 -- 172.21.15.44:0/3975910601 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6e1807ca90 con 0x7f6e1c15e4a0 2024-04-02T13:22:24.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.038+0000 7f6e239a8640 1 -- 172.21.15.44:0/3975910601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6e1c06f5c0 con 0x7f6e1c15e4a0 2024-04-02T13:22:24.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.046+0000 7f6e11ffb640 1 -- 172.21.15.44:0/3975910601 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f6e1807c390 con 0x7f6e1c15e4a0 2024-04-02T13:22:24.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.048+0000 7f6e239a8640 1 -- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6e0006c8f0 msgr2=0x7f6e0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:24.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.048+0000 7f6e239a8640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6e0006c8f0 0x7f6e0006edb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6e140015e0 tx=0x7f6e14009040 comp rx=0 tx=0).stop 2024-04-02T13:22:24.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.048+0000 7f6e239a8640 1 -- 172.21.15.44:0/3975910601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e1c15e4a0 msgr2=0x7f6e1c0b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:24.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.048+0000 7f6e239a8640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e1c15e4a0 0x7f6e1c0b5640 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6e18009f40 tx=0x7f6e18004a40 comp rx=0 tx=0).stop 2024-04-02T13:22:24.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.048+0000 7f6e239a8640 1 -- 172.21.15.44:0/3975910601 shutdown_connections 2024-04-02T13:22:24.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.048+0000 7f6e239a8640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6e0006c8f0 0x7f6e0006edb0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:24.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.048+0000 7f6e239a8640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e1c15e4a0 0x7f6e1c0b5640 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:24.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.048+0000 7f6e239a8640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e1c0d0c50 0x7f6e1c0b5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:24.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.048+0000 7f6e239a8640 1 --2- 172.21.15.44:0/3975910601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e1c06ce40 0x7f6e1c1517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:24.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.048+0000 7f6e239a8640 1 -- 172.21.15.44:0/3975910601 >> 172.21.15.44:0/3975910601 conn(0x7f6e1c09aa20 msgr2=0x7f6e1c067230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:24.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.048+0000 7f6e239a8640 1 -- 172.21.15.44:0/3975910601 shutdown_connections 2024-04-02T13:22:24.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.048+0000 7f6e239a8640 1 -- 172.21.15.44:0/3975910601 wait complete. 2024-04-02T13:22:24.060 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T13:22:24.061 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T13:22:24.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.163+0000 7fd03af7c640 1 Processor -- start 2024-04-02T13:22:24.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.163+0000 7fd03af7c640 1 -- start start 2024-04-02T13:22:24.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd03af7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0341565f0 0x7fd0340acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:24.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd03af7c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd03415c5b0 0x7fd0340cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:24.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd03af7c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03415cb70 con 0x7fd0340aea70 2024-04-02T13:22:24.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd03af7c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd0340ca380 con 0x7fd03415c5b0 2024-04-02T13:22:24.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd03af7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd03415ce50 con 0x7fd0341565f0 2024-04-02T13:22:24.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd033fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd03415c5b0 0x7fd0340cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:24.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd039519640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd0340aea70 0x7fd0340ca810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53374/0 (socket says 172.21.15.44:53374) 2024-04-02T13:22:24.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd033fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd03415c5b0 0x7fd0340cbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60258/0 (socket says 172.21.15.44:60258) 2024-04-02T13:22:24.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd039519640 1 -- 172.21.15.44:0/978869013 learned_addr learned my addr 172.21.15.44:0/978869013 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:24.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd038d18640 1 --2- 172.21.15.44:0/978869013 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0341565f0 0x7fd0340acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:24.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd033fff640 1 -- 172.21.15.44:0/978869013 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0341565f0 msgr2=0x7fd0340acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:24.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd033fff640 1 --2- 172.21.15.44:0/978869013 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0341565f0 0x7fd0340acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:24.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd033fff640 1 -- 172.21.15.44:0/978869013 >> v1:172.21.15.44:6789/0 conn(0x7fd0340aea70 legacy=0x7fd0340ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:24.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd033fff640 1 -- 172.21.15.44:0/978869013 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03415d130 con 0x7fd03415c5b0 2024-04-02T13:22:24.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd0337fe640 1 -- 172.21.15.44:0/978869013 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3584355730 0 0) 0x7fd03415cb70 con 0x7fd0340aea70 2024-04-02T13:22:24.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.164+0000 7fd038d18640 1 --2- 172.21.15.44:0/978869013 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0341565f0 0x7fd0340acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:24.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.165+0000 7fd033fff640 1 --2- 172.21.15.44:0/978869013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd03415c5b0 0x7fd0340cbed0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd0280097a0 tx=0x7fd02802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a037080b7908a5b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:24.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.165+0000 7fd0337fe640 1 -- 172.21.15.44:0/978869013 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd028035020 con 0x7fd03415c5b0 2024-04-02T13:22:24.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.165+0000 7fd0337fe640 1 -- 172.21.15.44:0/978869013 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd028031830 con 0x7fd03415c5b0 2024-04-02T13:22:24.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.165+0000 7fd0337fe640 1 -- 172.21.15.44:0/978869013 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd028031c00 con 0x7fd03415c5b0 2024-04-02T13:22:24.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.166+0000 7fd03af7c640 1 -- 172.21.15.44:0/978869013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd03415c5b0 msgr2=0x7fd0340cbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:24.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.166+0000 7fd03af7c640 1 --2- 172.21.15.44:0/978869013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd03415c5b0 0x7fd0340cbed0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd0280097a0 tx=0x7fd02802b8d0 comp rx=0 tx=0).stop 2024-04-02T13:22:24.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.166+0000 7fd03af7c640 1 -- 172.21.15.44:0/978869013 shutdown_connections 2024-04-02T13:22:24.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.166+0000 7fd03af7c640 1 --2- 172.21.15.44:0/978869013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd03415c5b0 0x7fd0340cbed0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:24.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.166+0000 7fd03af7c640 1 --2- 172.21.15.44:0/978869013 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0341565f0 0x7fd0340acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:24.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.166+0000 7fd03af7c640 1 -- 172.21.15.44:0/978869013 >> 172.21.15.44:0/978869013 conn(0x7fd0340cfca0 msgr2=0x7fd0340d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:24.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.166+0000 7fd03af7c640 1 -- 172.21.15.44:0/978869013 shutdown_connections 2024-04-02T13:22:24.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.166+0000 7fd03af7c640 1 -- 172.21.15.44:0/978869013 wait complete. 2024-04-02T13:22:24.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.167+0000 7fd03af7c640 1 Processor -- start 2024-04-02T13:22:24.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.167+0000 7fd03af7c640 1 -- start start 2024-04-02T13:22:24.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd03af7c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0340aea70 0x7fd0341f5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:24.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd03af7c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0341565f0 0x7fd0341527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:24.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd03af7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03415c5b0 0x7fd03409cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:24.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd03af7c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd0340cc410 con 0x7fd0340aea70 2024-04-02T13:22:24.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd03af7c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd0341569d0 con 0x7fd0341565f0 2024-04-02T13:22:24.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd03af7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd0340aef00 con 0x7fd03415c5b0 2024-04-02T13:22:24.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd038d18640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0340aea70 0x7fd0341f5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:24.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd038d18640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0340aea70 0x7fd0341f5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60268/0 (socket says 172.21.15.44:60268) 2024-04-02T13:22:24.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd038d18640 1 -- 172.21.15.44:0/1428853837 learned_addr learned my addr 172.21.15.44:0/1428853837 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:24.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd039519640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03415c5b0 0x7fd03409cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:24.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd033fff640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0341565f0 0x7fd0341527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:24.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd038d18640 1 -- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03415c5b0 msgr2=0x7fd03409cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:24.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd038d18640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03415c5b0 0x7fd03409cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:24.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd038d18640 1 -- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0341565f0 msgr2=0x7fd0341527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:24.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd038d18640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0341565f0 0x7fd0341527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:24.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.168+0000 7fd038d18640 1 -- 172.21.15.44:0/1428853837 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd034152d20 con 0x7fd0340aea70 2024-04-02T13:22:24.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.169+0000 7fd038d18640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0340aea70 0x7fd0341f5b60 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd02c00cf90 tx=0x7fd02c00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:24.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.169+0000 7fd0317fa640 1 -- 172.21.15.44:0/1428853837 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c01a070 con 0x7fd0340aea70 2024-04-02T13:22:24.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.169+0000 7fd0317fa640 1 -- 172.21.15.44:0/1428853837 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd02c00d840 con 0x7fd0340aea70 2024-04-02T13:22:24.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.169+0000 7fd03af7c640 1 -- 172.21.15.44:0/1428853837 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd03409b100 con 0x7fd0340aea70 2024-04-02T13:22:24.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.169+0000 7fd0317fa640 1 -- 172.21.15.44:0/1428853837 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c00dc10 con 0x7fd0340aea70 2024-04-02T13:22:24.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.169+0000 7fd03af7c640 1 -- 172.21.15.44:0/1428853837 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd03409b6c0 con 0x7fd0340aea70 2024-04-02T13:22:24.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.170+0000 7fd03af7c640 1 -- 172.21.15.44:0/1428853837 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd034074f20 con 0x7fd0340aea70 2024-04-02T13:22:24.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.170+0000 7fd0317fa640 1 -- 172.21.15.44:0/1428853837 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd02c01d070 con 0x7fd0340aea70 2024-04-02T13:22:24.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.171+0000 7fd0317fa640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd02006c920 0x7fd02006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:24.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.171+0000 7fd0317fa640 1 -- 172.21.15.44:0/1428853837 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fd02c08c4f0 con 0x7fd0340aea70 2024-04-02T13:22:24.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.171+0000 7fd033fff640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd02006c920 0x7fd02006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:24.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.171+0000 7fd033fff640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd02006c920 0x7fd02006ede0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd028036000 tx=0x7fd02803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:24.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.175+0000 7fd0317fa640 1 -- 172.21.15.44:0/1428853837 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd02c05a0a0 con 0x7fd0340aea70 2024-04-02T13:22:24.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:24.397+0000 7fd03af7c640 1 -- 172.21.15.44:0/1428853837 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd0340684f0 con 0x7fd0340aea70 2024-04-02T13:22:25.004 INFO:tasks.ceph.mds.d.smithi069.stderr:2024-04-02T13:22:25.002+0000 7fbf4d51c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:22:25.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.051+0000 7fd0317fa640 1 -- 172.21.15.44:0/1428853837 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) ==== 122+0+0 (secure 0 0 0) 0x7fd02c05e3a0 con 0x7fd0340aea70 2024-04-02T13:22:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.053+0000 7fd03af7c640 1 -- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd02006c920 msgr2=0x7fd02006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.053+0000 7fd03af7c640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd02006c920 0x7fd02006ede0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd028036000 tx=0x7fd02803d040 comp rx=0 tx=0).stop 2024-04-02T13:22:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.053+0000 7fd03af7c640 1 -- 172.21.15.44:0/1428853837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0340aea70 msgr2=0x7fd0341f5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.053+0000 7fd03af7c640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0340aea70 0x7fd0341f5b60 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd02c00cf90 tx=0x7fd02c00a7d0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.053+0000 7fd03af7c640 1 -- 172.21.15.44:0/1428853837 shutdown_connections 2024-04-02T13:22:25.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.053+0000 7fd03af7c640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd02006c920 0x7fd02006ede0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.053+0000 7fd03af7c640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03415c5b0 0x7fd03409cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.053+0000 7fd03af7c640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0341565f0 0x7fd0341527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.053+0000 7fd03af7c640 1 --2- 172.21.15.44:0/1428853837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0340aea70 0x7fd0341f5b60 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.053+0000 7fd03af7c640 1 -- 172.21.15.44:0/1428853837 >> 172.21.15.44:0/1428853837 conn(0x7fd0340cfca0 msgr2=0x7fd0340acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:25.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.054+0000 7fd03af7c640 1 -- 172.21.15.44:0/1428853837 shutdown_connections 2024-04-02T13:22:25.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.054+0000 7fd03af7c640 1 -- 172.21.15.44:0/1428853837 wait complete. 2024-04-02T13:22:25.067 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T13:22:25.068 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:22:25.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.171+0000 7f176df50640 1 Processor -- start 2024-04-02T13:22:25.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.171+0000 7f176df50640 1 -- start start 2024-04-02T13:22:25.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.171+0000 7f176df50640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f17680d0a30 0x7f176806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.171+0000 7f176df50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17680d2040 0x7f17680d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.171+0000 7f176df50640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17680d0000 con 0x7f17680d0e10 2024-04-02T13:22:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.171+0000 7f176df50640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f176806cb00 con 0x7f17680d0a30 2024-04-02T13:22:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.171+0000 7f176df50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f176806c1d0 con 0x7f17680d2040 2024-04-02T13:22:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.172+0000 7f17677fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f17680d0a30 0x7f176806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.172+0000 7f1766ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f17680d0e10 0x7f17680cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:58440/0 (socket says 172.21.15.44:58440) 2024-04-02T13:22:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.172+0000 7f1766ffd640 1 -- 172.21.15.44:0/3954268299 learned_addr learned my addr 172.21.15.44:0/3954268299 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.172+0000 7f17677fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f17680d0a30 0x7f176806cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43464/0 (socket says 172.21.15.44:43464) 2024-04-02T13:22:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.172+0000 7f1767fff640 1 --2- 172.21.15.44:0/3954268299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17680d2040 0x7f17680d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.172+0000 7f17677fe640 1 -- 172.21.15.44:0/3954268299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17680d2040 msgr2=0x7f17680d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.172+0000 7f17677fe640 1 --2- 172.21.15.44:0/3954268299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17680d2040 0x7f17680d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.172+0000 7f17677fe640 1 -- 172.21.15.44:0/3954268299 >> v1:172.21.15.44:6789/0 conn(0x7f17680d0e10 legacy=0x7f17680cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.172+0000 7f17677fe640 1 -- 172.21.15.44:0/3954268299 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f176806c4b0 con 0x7f17680d0a30 2024-04-02T13:22:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.172+0000 7f17667fc640 1 -- 172.21.15.44:0/3954268299 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1994018910 0 0) 0x7f17680d0000 con 0x7f17680d0e10 2024-04-02T13:22:25.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.172+0000 7f1767fff640 1 --2- 172.21.15.44:0/3954268299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17680d2040 0x7f17680d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.173+0000 7f17677fe640 1 --2- 172.21.15.44:0/3954268299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680d0a30 0x7f176806cf90 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f17580097a0 tx=0x7f175802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9f8acc68288c8b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.173+0000 7f17667fc640 1 -- 172.21.15.44:0/3954268299 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17580315a0 con 0x7f17680d0a30 2024-04-02T13:22:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.173+0000 7f17667fc640 1 -- 172.21.15.44:0/3954268299 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1758037070 con 0x7f17680d0a30 2024-04-02T13:22:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.173+0000 7f17667fc640 1 -- 172.21.15.44:0/3954268299 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1758031960 con 0x7f17680d0a30 2024-04-02T13:22:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.173+0000 7f176df50640 1 -- 172.21.15.44:0/3954268299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680d0a30 msgr2=0x7f176806cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.173+0000 7f176df50640 1 --2- 172.21.15.44:0/3954268299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680d0a30 0x7f176806cf90 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f17580097a0 tx=0x7f175802b840 comp rx=0 tx=0).stop 2024-04-02T13:22:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.173+0000 7f176df50640 1 -- 172.21.15.44:0/3954268299 shutdown_connections 2024-04-02T13:22:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.173+0000 7f176df50640 1 --2- 172.21.15.44:0/3954268299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17680d2040 0x7f17680d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.173+0000 7f176df50640 1 --2- 172.21.15.44:0/3954268299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680d0a30 0x7f176806cf90 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.173+0000 7f176df50640 1 -- 172.21.15.44:0/3954268299 >> 172.21.15.44:0/3954268299 conn(0x7f176809d470 msgr2=0x7f17680d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:25.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.173+0000 7f176df50640 1 -- 172.21.15.44:0/3954268299 shutdown_connections 2024-04-02T13:22:25.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.173+0000 7f176df50640 1 -- 172.21.15.44:0/3954268299 wait complete. 2024-04-02T13:22:25.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.174+0000 7f176df50640 1 Processor -- start 2024-04-02T13:22:25.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.175+0000 7f176df50640 1 -- start start 2024-04-02T13:22:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.175+0000 7f176df50640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680d0a30 0x7f17680b9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.175+0000 7f176df50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17680d0e10 0x7f17680c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.175+0000 7f176df50640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17680d2040 0x7f176815feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.175+0000 7f176df50640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f176815e6f0 con 0x7f17680d0a30 2024-04-02T13:22:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.175+0000 7f176df50640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f17680d1270 con 0x7f17680d2040 2024-04-02T13:22:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.175+0000 7f176df50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f176815e870 con 0x7f17680d0e10 2024-04-02T13:22:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.175+0000 7f17677fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680d0a30 0x7f17680b9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.175+0000 7f17677fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680d0a30 0x7f17680b9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43474/0 (socket says 172.21.15.44:43474) 2024-04-02T13:22:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.175+0000 7f17677fe640 1 -- 172.21.15.44:0/143147854 learned_addr learned my addr 172.21.15.44:0/143147854 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f1766ffd640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17680d0e10 0x7f17680c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f1767fff640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17680d2040 0x7f176815feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f17677fe640 1 -- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17680d0e10 msgr2=0x7f17680c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f17677fe640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17680d0e10 0x7f17680c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f17677fe640 1 -- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17680d2040 msgr2=0x7f176815feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f17677fe640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17680d2040 0x7f176815feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f17677fe640 1 -- 172.21.15.44:0/143147854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17680b9c80 con 0x7f17680d0a30 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f1767fff640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17680d2040 0x7f176815feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f17677fe640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680d0a30 0x7f17680b9740 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1758034040 tx=0x7f17580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f174ffff640 1 -- 172.21.15.44:0/143147854 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1758032840 con 0x7f17680d0a30 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f174ffff640 1 -- 172.21.15.44:0/143147854 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1758032a00 con 0x7f17680d0a30 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f176df50640 1 -- 172.21.15.44:0/143147854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17680c9ab0 con 0x7f17680d0a30 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f174ffff640 1 -- 172.21.15.44:0/143147854 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1758032df0 con 0x7f17680d0a30 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.176+0000 7f176df50640 1 -- 172.21.15.44:0/143147854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17680ca070 con 0x7f17680d0a30 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.177+0000 7f176df50640 1 -- 172.21.15.44:0/143147854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17680684f0 con 0x7f17680d0a30 2024-04-02T13:22:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.178+0000 7f174ffff640 1 -- 172.21.15.44:0/143147854 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f175803b3f0 con 0x7f17680d0a30 2024-04-02T13:22:25.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.178+0000 7f174ffff640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f175406c8f0 0x7f175406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.178+0000 7f174ffff640 1 -- 172.21.15.44:0/143147854 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f175802f030 con 0x7f17680d0a30 2024-04-02T13:22:25.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.178+0000 7f1766ffd640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f175406c8f0 0x7f175406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.179+0000 7f1766ffd640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f175406c8f0 0x7f175406edb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f17680b64f0 tx=0x7f175c007c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:25.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.182+0000 7f174ffff640 1 -- 172.21.15.44:0/143147854 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1758040070 con 0x7f17680d0a30 2024-04-02T13:22:25.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.407+0000 7f176df50640 1 -- 172.21.15.44:0/143147854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17680d17c0 con 0x7f17680d0a30 2024-04-02T13:22:25.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.408+0000 7f174ffff640 1 -- 172.21.15.44:0/143147854 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5181 (secure 0 0 0) 0x7f175807bf50 con 0x7f17680d0a30 2024-04-02T13:22:25.410 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:25.410 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/4131284806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":4131284806},{"type":"v1","addr":"172.21.15.44:6833","nonce":4131284806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/4262427145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":4262427145},{"type":"v1","addr":"172.21.15.44:6835","nonce":4262427145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4112565894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4112565894},{"type":"v1","addr":"172.21.15.69:6836","nonce":4112565894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:22:22.422387+0000","modified":"2024-04-02T13:22:24.398291+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6837/1745145720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1745145720},{"type":"v1","addr":"172.21.15.69:6837","nonce":1745145720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":3}]} 2024-04-02T13:22:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-02T13:22:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.410+0000 7f176df50640 1 -- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f175406c8f0 msgr2=0x7f175406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.410+0000 7f176df50640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f175406c8f0 0x7f175406edb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f17680b64f0 tx=0x7f175c007c00 comp rx=0 tx=0).stop 2024-04-02T13:22:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.410+0000 7f176df50640 1 -- 172.21.15.44:0/143147854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680d0a30 msgr2=0x7f17680b9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.410+0000 7f176df50640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680d0a30 0x7f17680b9740 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1758034040 tx=0x7f17580046c0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.410+0000 7f176df50640 1 -- 172.21.15.44:0/143147854 shutdown_connections 2024-04-02T13:22:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.410+0000 7f176df50640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f175406c8f0 0x7f175406edb0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.410+0000 7f176df50640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17680d2040 0x7f176815feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.410+0000 7f176df50640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17680d0e10 0x7f17680c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.410+0000 7f176df50640 1 --2- 172.21.15.44:0/143147854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680d0a30 0x7f17680b9740 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.410+0000 7f176df50640 1 -- 172.21.15.44:0/143147854 >> 172.21.15.44:0/143147854 conn(0x7f176809d470 msgr2=0x7f176815c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.410+0000 7f176df50640 1 -- 172.21.15.44:0/143147854 shutdown_connections 2024-04-02T13:22:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.410+0000 7f176df50640 1 -- 172.21.15.44:0/143147854 wait complete. 2024-04-02T13:22:25.423 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:22:25.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.523+0000 7f5cd8b47640 1 Processor -- start 2024-04-02T13:22:25.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.523+0000 7f5cd8b47640 1 -- start start 2024-04-02T13:22:25.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.524+0000 7f5cd8b47640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5cd415e550 0x7f5cd406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.524+0000 7f5cd8b47640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd415f040 0x7f5cd40d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.524+0000 7f5cd8b47640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cd415fb50 con 0x7f5cd406d530 2024-04-02T13:22:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.524+0000 7f5cd8b47640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5cd406d970 con 0x7f5cd415e550 2024-04-02T13:22:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.524+0000 7f5cd8b47640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5cd406ca90 con 0x7f5cd415f040 2024-04-02T13:22:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.524+0000 7f5cd2d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd415f040 0x7f5cd40d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.524+0000 7f5cd259c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5cd415e550 0x7f5cd406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.524+0000 7f5cd259c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5cd415e550 0x7f5cd406cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:50342/0 (socket says 172.21.15.44:50342) 2024-04-02T13:22:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.524+0000 7f5cd259c640 1 -- 172.21.15.44:0/729270543 learned_addr learned my addr 172.21.15.44:0/729270543 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.524+0000 7f5cd2d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd415f040 0x7f5cd40d0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:52818/0 (socket says 172.21.15.44:52818) 2024-04-02T13:22:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.524+0000 7f5cd1d9b640 1 --1- 172.21.15.44:0/729270543 >> v1:172.21.15.69:6789/0 conn(0x7f5cd406d530 0x7f5cd415e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:34744/0 (socket says 172.21.15.44:34744) 2024-04-02T13:22:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.525+0000 7f5cd259c640 1 -- 172.21.15.44:0/729270543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd415f040 msgr2=0x7f5cd40d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.525+0000 7f5cd259c640 1 --2- 172.21.15.44:0/729270543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd415f040 0x7f5cd40d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.525+0000 7f5cd259c640 1 -- 172.21.15.44:0/729270543 >> v1:172.21.15.69:6789/0 conn(0x7f5cd406d530 legacy=0x7f5cd415e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.525+0000 7f5cd259c640 1 -- 172.21.15.44:0/729270543 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd415f590 con 0x7f5cd415e550 2024-04-02T13:22:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.525+0000 7f5cd2d9d640 1 --2- 172.21.15.44:0/729270543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd415f040 0x7f5cd40d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:22:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.525+0000 7f5cd259c640 1 --2- 172.21.15.44:0/729270543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd415e550 0x7f5cd406cff0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc009870 tx=0x7f5ccc02ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e678d04871915c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.526+0000 7f5cd159a640 1 -- 172.21.15.44:0/729270543 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc02f840 con 0x7f5cd415e550 2024-04-02T13:22:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.526+0000 7f5cd159a640 1 -- 172.21.15.44:0/729270543 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ccc02f9e0 con 0x7f5cd415e550 2024-04-02T13:22:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.526+0000 7f5cd159a640 1 -- 172.21.15.44:0/729270543 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc02fdb0 con 0x7f5cd415e550 2024-04-02T13:22:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.526+0000 7f5cd8b47640 1 -- 172.21.15.44:0/729270543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd415e550 msgr2=0x7f5cd406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.526+0000 7f5cd8b47640 1 --2- 172.21.15.44:0/729270543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd415e550 0x7f5cd406cff0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc009870 tx=0x7f5ccc02ba20 comp rx=0 tx=0).stop 2024-04-02T13:22:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.526+0000 7f5cd8b47640 1 -- 172.21.15.44:0/729270543 shutdown_connections 2024-04-02T13:22:25.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.526+0000 7f5cd8b47640 1 --2- 172.21.15.44:0/729270543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd415f040 0x7f5cd40d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.526+0000 7f5cd8b47640 1 --2- 172.21.15.44:0/729270543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd415e550 0x7f5cd406cff0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.526+0000 7f5cd8b47640 1 -- 172.21.15.44:0/729270543 >> 172.21.15.44:0/729270543 conn(0x7f5cd409aa20 msgr2=0x7f5cd409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:25.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.526+0000 7f5cd8b47640 1 -- 172.21.15.44:0/729270543 shutdown_connections 2024-04-02T13:22:25.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.526+0000 7f5cd8b47640 1 -- 172.21.15.44:0/729270543 wait complete. 2024-04-02T13:22:25.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.528+0000 7f5cd8b47640 1 Processor -- start 2024-04-02T13:22:25.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.528+0000 7f5cd8b47640 1 -- start start 2024-04-02T13:22:25.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.528+0000 7f5cd8b47640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd406d530 0x7f5cd414c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.528+0000 7f5cd8b47640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cd415e550 0x7f5cd40b7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.528+0000 7f5cd8b47640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd415f040 0x7f5cd40b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.528+0000 7f5cd8b47640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd40d0590 con 0x7f5cd415e550 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.528+0000 7f5cd8b47640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd40d02b0 con 0x7f5cd415f040 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.528+0000 7f5cd8b47640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5cd40cc4b0 con 0x7f5cd406d530 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.528+0000 7f5cd1d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cd415e550 0x7f5cd40b7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5cd1d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cd415e550 0x7f5cd40b7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43480/0 (socket says 172.21.15.44:43480) 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5cd1d9b640 1 -- 172.21.15.44:0/2842364556 learned_addr learned my addr 172.21.15.44:0/2842364556 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5cd2d9d640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd415f040 0x7f5cd40b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5cd259c640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd406d530 0x7f5cd414c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5cd1d9b640 1 -- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd406d530 msgr2=0x7f5cd414c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5cd1d9b640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd406d530 0x7f5cd414c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5cd1d9b640 1 -- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd415f040 msgr2=0x7f5cd40b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5cd1d9b640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd415f040 0x7f5cd40b5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5cd1d9b640 1 -- 172.21.15.44:0/2842364556 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd414c7e0 con 0x7f5cd415e550 2024-04-02T13:22:25.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5cd1d9b640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cd415e550 0x7f5cd40b7680 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00c3c0 tx=0x7f5cbc00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:25.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5ccaffd640 1 -- 172.21.15.44:0/2842364556 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc0074e0 con 0x7f5cd415e550 2024-04-02T13:22:25.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5ccaffd640 1 -- 172.21.15.44:0/2842364556 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cbc007680 con 0x7f5cd415e550 2024-04-02T13:22:25.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5ccaffd640 1 -- 172.21.15.44:0/2842364556 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc004a00 con 0x7f5cd415e550 2024-04-02T13:22:25.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5cd8b47640 1 -- 172.21.15.44:0/2842364556 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cd4156910 con 0x7f5cd415e550 2024-04-02T13:22:25.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.529+0000 7f5cd8b47640 1 -- 172.21.15.44:0/2842364556 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cd40cb330 con 0x7f5cd415e550 2024-04-02T13:22:25.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.530+0000 7f5cd8b47640 1 -- 172.21.15.44:0/2842364556 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cd40684f0 con 0x7f5cd415e550 2024-04-02T13:22:25.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.531+0000 7f5ccaffd640 1 -- 172.21.15.44:0/2842364556 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5cbc01a020 con 0x7f5cd415e550 2024-04-02T13:22:25.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.531+0000 7f5ccaffd640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cb406c8f0 0x7f5cb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.532+0000 7f5ccaffd640 1 -- 172.21.15.44:0/2842364556 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f5cbc08b470 con 0x7f5cd415e550 2024-04-02T13:22:25.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.532+0000 7f5cd259c640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cb406c8f0 0x7f5cb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.532+0000 7f5cd259c640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cb406c8f0 0x7f5cb406edb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc036000 tx=0x7f5ccc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:25.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.536+0000 7f5ccaffd640 1 -- 172.21.15.44:0/2842364556 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5cd414c7e0 con 0x7f5cd415e550 2024-04-02T13:22:25.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.722+0000 7f5cd8b47640 1 -- 172.21.15.44:0/2842364556 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5cd409b460 con 0x7f5cd415e550 2024-04-02T13:22:25.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.723+0000 7f5ccaffd640 1 -- 172.21.15.44:0/2842364556 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+16894 (secure 0 0 0) 0x7f5cbc0595e0 con 0x7f5cd415e550 2024-04-02T13:22:25.725 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:25.725 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":52,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:22:24.017854+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:22:20.662776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-02T13:22:21.332577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:22:25.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.725+0000 7f5cd8b47640 1 -- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cb406c8f0 msgr2=0x7f5cb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.725+0000 7f5cd8b47640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cb406c8f0 0x7f5cb406edb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc036000 tx=0x7f5ccc008030 comp rx=0 tx=0).stop 2024-04-02T13:22:25.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.725+0000 7f5cd8b47640 1 -- 172.21.15.44:0/2842364556 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cd415e550 msgr2=0x7f5cd40b7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.725+0000 7f5cd8b47640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cd415e550 0x7f5cd40b7680 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00c3c0 tx=0x7f5cbc00c7c0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.725+0000 7f5cd8b47640 1 -- 172.21.15.44:0/2842364556 shutdown_connections 2024-04-02T13:22:25.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.725+0000 7f5cd8b47640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cb406c8f0 0x7f5cb406edb0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.725+0000 7f5cd8b47640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd415f040 0x7f5cd40b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.725+0000 7f5cd8b47640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5cd415e550 0x7f5cd40b7680 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.725+0000 7f5cd8b47640 1 --2- 172.21.15.44:0/2842364556 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd406d530 0x7f5cd414c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.725+0000 7f5cd8b47640 1 -- 172.21.15.44:0/2842364556 >> 172.21.15.44:0/2842364556 conn(0x7f5cd409aa20 msgr2=0x7f5cd406fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:25.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.725+0000 7f5cd8b47640 1 -- 172.21.15.44:0/2842364556 shutdown_connections 2024-04-02T13:22:25.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.725+0000 7f5cd8b47640 1 -- 172.21.15.44:0/2842364556 wait complete. 2024-04-02T13:22:25.738 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T13:22:25.738 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:22:25.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.838+0000 7f99c26a5640 1 Processor -- start 2024-04-02T13:22:25.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.838+0000 7f99c26a5640 1 -- start start 2024-04-02T13:22:25.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.839+0000 7f99c26a5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f99bc0d09f0 0x7f99bc06d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.839+0000 7f99c26a5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99bc06ece0 0x7f99bc0acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.839+0000 7f99c26a5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99bc06dc40 con 0x7f99bc0cece0 2024-04-02T13:22:25.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.839+0000 7f99c26a5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f99bc06df20 con 0x7f99bc0d09f0 2024-04-02T13:22:25.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.839+0000 7f99c26a5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f99bc06e200 con 0x7f99bc06ece0 2024-04-02T13:22:25.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.839+0000 7f99bbfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f99bc0cece0 0x7f99bc0cf0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:58456/0 (socket says 172.21.15.44:58456) 2024-04-02T13:22:25.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.839+0000 7f99bbfff640 1 -- 172.21.15.44:0/1257674535 learned_addr learned my addr 172.21.15.44:0/1257674535 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:25.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.839+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 504694920 0 0) 0x7f99bc06dc40 con 0x7f99bc0cece0 2024-04-02T13:22:25.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.839+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99a8003520 con 0x7f99bc0cece0 2024-04-02T13:22:25.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.839+0000 7f99bb7fe640 1 --2- 172.21.15.44:0/1257674535 >> v2:172.21.15.69:3300/0 conn(0x7f99bc0d09f0 0x7f99bc06d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.839+0000 7f99c0c42640 1 --2- 172.21.15.44:0/1257674535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99bc06ece0 0x7f99bc0acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1047884615 0 0) 0x7f99a8003520 con 0x7f99bc0cece0 2024-04-02T13:22:25.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99bc06ece0 msgr2=0x7f99bc0acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:25.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 --2- 172.21.15.44:0/1257674535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99bc06ece0 0x7f99bc0acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 >> v2:172.21.15.69:3300/0 conn(0x7f99bc0d09f0 msgr2=0x7f99bc06d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:25.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 --2- 172.21.15.44:0/1257674535 >> v2:172.21.15.69:3300/0 conn(0x7f99bc0d09f0 0x7f99bc06d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99bc06e4e0 con 0x7f99bc0cece0 2024-04-02T13:22:25.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f99a4001670 con 0x7f99bc0cece0 2024-04-02T13:22:25.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 >> v1:172.21.15.44:6789/0 conn(0x7f99bc0cece0 legacy=0x7f99bc0cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 --2- 172.21.15.44:0/1257674535 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a80039b0 0x7f99a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 --2- 172.21.15.44:0/1257674535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a8004800 0x7f99a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 --2- 172.21.15.44:0/1257674535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99bc06ece0 0x7f99a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc06df20 con 0x7f99a80039b0 2024-04-02T13:22:25.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc06e200 con 0x7f99a8004800 2024-04-02T13:22:25.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f99a800b9c0 con 0x7f99bc06ece0 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99bbfff640 1 --2- 172.21.15.44:0/1257674535 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a80039b0 0x7f99a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99bb7fe640 1 --2- 172.21.15.44:0/1257674535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a8004800 0x7f99a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99c0c42640 1 --2- 172.21.15.44:0/1257674535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99bc06ece0 0x7f99a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99bbfff640 1 -- 172.21.15.44:0/1257674535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99bc06ece0 msgr2=0x7f99a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99bbfff640 1 --2- 172.21.15.44:0/1257674535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99bc06ece0 0x7f99a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99bbfff640 1 -- 172.21.15.44:0/1257674535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a8004800 msgr2=0x7f99a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99bbfff640 1 --2- 172.21.15.44:0/1257674535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a8004800 0x7f99a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.840+0000 7f99bbfff640 1 -- 172.21.15.44:0/1257674535 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a8003520 con 0x7f99a80039b0 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.841+0000 7f99bbfff640 1 --2- 172.21.15.44:0/1257674535 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a80039b0 0x7f99a8003dd0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f99a4007fd0 tx=0x7f99a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=691616d12e113f73 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.841+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a4007d50 con 0x7f99a80039b0 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.841+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99a4030070 con 0x7f99a80039b0 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.841+0000 7f99baffd640 1 -- 172.21.15.44:0/1257674535 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a402cd40 con 0x7f99a80039b0 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.841+0000 7f99c26a5640 1 -- 172.21.15.44:0/1257674535 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a80039b0 msgr2=0x7f99a8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.841+0000 7f99c26a5640 1 --2- 172.21.15.44:0/1257674535 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a80039b0 0x7f99a8003dd0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f99a4007fd0 tx=0x7f99a8005830 comp rx=0 tx=0).stop 2024-04-02T13:22:25.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.841+0000 7f99c0c42640 1 -- 172.21.15.44:0/1257674535 reap_dead start 2024-04-02T13:22:25.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.842+0000 7f99c26a5640 1 -- 172.21.15.44:0/1257674535 shutdown_connections 2024-04-02T13:22:25.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.842+0000 7f99c26a5640 1 -- 172.21.15.44:0/1257674535 >> 172.21.15.44:0/1257674535 conn(0x7f99bc0cfcd0 msgr2=0x7f99bc0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:25.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.842+0000 7f99c26a5640 1 -- 172.21.15.44:0/1257674535 shutdown_connections 2024-04-02T13:22:25.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.842+0000 7f99c26a5640 1 -- 172.21.15.44:0/1257674535 wait complete. 2024-04-02T13:22:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.843+0000 7f99c26a5640 1 Processor -- start 2024-04-02T13:22:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.843+0000 7f99c26a5640 1 -- start start 2024-04-02T13:22:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.843+0000 7f99c26a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a80039b0 0x7f99bc145c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c26a5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99bc150770 0x7f99bc09b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c26a5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99bc09b7e0 0x7f99bc071870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c26a5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc06d990 con 0x7f99bc09b7e0 2024-04-02T13:22:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c26a5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc06d810 con 0x7f99a80039b0 2024-04-02T13:22:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c26a5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f99bc15b220 con 0x7f99bc150770 2024-04-02T13:22:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c0c42640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99bc09b7e0 0x7f99bc071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c0c42640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99bc09b7e0 0x7f99bc071870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43498/0 (socket says 172.21.15.44:43498) 2024-04-02T13:22:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c0c42640 1 -- 172.21.15.44:0/3032879150 learned_addr learned my addr 172.21.15.44:0/3032879150 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99bb7fe640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99bc150770 0x7f99bc09b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99bbfff640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a80039b0 0x7f99bc145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c0c42640 1 -- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99bc150770 msgr2=0x7f99bc09b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c0c42640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99bc150770 0x7f99bc09b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c0c42640 1 -- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a80039b0 msgr2=0x7f99bc145c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c0c42640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a80039b0 0x7f99bc145c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c0c42640 1 -- 172.21.15.44:0/3032879150 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99bc14f6d0 con 0x7f99bc09b7e0 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c0c42640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99bc09b7e0 0x7f99bc071870 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f99bc070e80 tx=0x7f99b40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99b8ff9640 1 -- 172.21.15.44:0/3032879150 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4011db0 con 0x7f99bc09b7e0 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99b8ff9640 1 -- 172.21.15.44:0/3032879150 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b401b070 con 0x7f99bc09b7e0 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99b8ff9640 1 -- 172.21.15.44:0/3032879150 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4003990 con 0x7f99bc09b7e0 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c26a5640 1 -- 172.21.15.44:0/3032879150 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99bc14f9b0 con 0x7f99bc09b7e0 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.844+0000 7f99c26a5640 1 -- 172.21.15.44:0/3032879150 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99bc14ff70 con 0x7f99bc09b7e0 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.845+0000 7f99c26a5640 1 -- 172.21.15.44:0/3032879150 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99bc074f20 con 0x7f99bc09b7e0 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.846+0000 7f99b8ff9640 1 -- 172.21.15.44:0/3032879150 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f99b4005d10 con 0x7f99bc09b7e0 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.846+0000 7f99b8ff9640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f999c06c920 0x7f999c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.846+0000 7f99b8ff9640 1 -- 172.21.15.44:0/3032879150 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f99b408c230 con 0x7f99bc09b7e0 2024-04-02T13:22:25.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.846+0000 7f99bbfff640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f999c06c920 0x7f999c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:25.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.847+0000 7f99bbfff640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f999c06c920 0x7f999c06ede0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f99a4034000 tx=0x7f99a4007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:25.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:25.850+0000 7f99b8ff9640 1 -- 172.21.15.44:0/3032879150 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f99b4025020 con 0x7f99bc09b7e0 2024-04-02T13:22:26.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.033+0000 7f99c26a5640 1 -- 172.21.15.44:0/3032879150 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f99bc073450 con 0x7f999c06c920 2024-04-02T13:22:26.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.040+0000 7f99b8ff9640 1 -- 172.21.15.44:0/3032879150 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251891 (secure 0 0 0) 0x7f99bc073450 con 0x7f999c06c920 2024-04-02T13:22:26.044 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:22:26.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.042+0000 7f99c26a5640 1 -- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f999c06c920 msgr2=0x7f999c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.042+0000 7f99c26a5640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f999c06c920 0x7f999c06ede0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f99a4034000 tx=0x7f99a4007000 comp rx=0 tx=0).stop 2024-04-02T13:22:26.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.042+0000 7f99c26a5640 1 -- 172.21.15.44:0/3032879150 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99bc09b7e0 msgr2=0x7f99bc071870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.042+0000 7f99c26a5640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99bc09b7e0 0x7f99bc071870 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f99bc070e80 tx=0x7f99b40115a0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.042+0000 7f99c26a5640 1 -- 172.21.15.44:0/3032879150 shutdown_connections 2024-04-02T13:22:26.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.042+0000 7f99c26a5640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f999c06c920 0x7f999c06ede0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.042+0000 7f99c26a5640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99bc09b7e0 0x7f99bc071870 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.043+0000 7f99c26a5640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99bc150770 0x7f99bc09b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.043+0000 7f99c26a5640 1 --2- 172.21.15.44:0/3032879150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a80039b0 0x7f99bc145c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.043+0000 7f99c26a5640 1 -- 172.21.15.44:0/3032879150 >> 172.21.15.44:0/3032879150 conn(0x7f99bc0cfcd0 msgr2=0x7f99bc0cf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:26.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.043+0000 7f99c26a5640 1 -- 172.21.15.44:0/3032879150 shutdown_connections 2024-04-02T13:22:26.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.043+0000 7f99c26a5640 1 -- 172.21.15.44:0/3032879150 wait complete. 2024-04-02T13:22:26.055 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-04-02T13:22:24.902312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222100,"kb_used_data":5916,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752620,"statfs":{"total":773094113280,"available":772866682880,"internally_reserved":0,"allocated":6057984,"data_stored":3317178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000993"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.044805+0000","last_change":"2024-04-02T13:22:23.038062+0000","last_active":"2024-04-02T13:22:23.044805+0000","last_peered":"2024-04-02T13:22:23.044805+0000","last_clean":"2024-04-02T13:22:23.044805+0000","last_became_active":"2024-04-02T13:22:23.037713+0000","last_became_peered":"2024-04-02T13:22:23.037713+0000","last_unstale":"2024-04-02T13:22:23.044805+0000","last_undegraded":"2024-04-02T13:22:23.044805+0000","last_fullsized":"2024-04-02T13:22:23.044805+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:17.264918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.041501+0000","last_change":"2024-04-02T13:22:23.039918+0000","last_active":"2024-04-02T13:22:23.041501+0000","last_peered":"2024-04-02T13:22:23.041501+0000","last_clean":"2024-04-02T13:22:23.041501+0000","last_became_active":"2024-04-02T13:22:23.038737+0000","last_became_peered":"2024-04-02T13:22:23.038737+0000","last_unstale":"2024-04-02T13:22:23.041501+0000","last_undegraded":"2024-04-02T13:22:23.041501+0000","last_fullsized":"2024-04-02T13:22:23.041501+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:35.277058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045422+0000","last_change":"2024-04-02T13:22:23.039151+0000","last_active":"2024-04-02T13:22:23.045422+0000","last_peered":"2024-04-02T13:22:23.045422+0000","last_clean":"2024-04-02T13:22:23.045422+0000","last_became_active":"2024-04-02T13:22:23.038711+0000","last_became_peered":"2024-04-02T13:22:23.038711+0000","last_unstale":"2024-04-02T13:22:23.045422+0000","last_undegraded":"2024-04-02T13:22:23.045422+0000","last_fullsized":"2024-04-02T13:22:23.045422+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:38.210075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.033592+0000","last_change":"2024-04-02T13:22:23.030782+0000","last_active":"2024-04-02T13:22:23.033592+0000","last_peered":"2024-04-02T13:22:23.033592+0000","last_clean":"2024-04-02T13:22:23.033592+0000","last_became_active":"2024-04-02T13:22:23.030502+0000","last_became_peered":"2024-04-02T13:22:23.030502+0000","last_unstale":"2024-04-02T13:22:23.033592+0000","last_undegraded":"2024-04-02T13:22:23.033592+0000","last_fullsized":"2024-04-02T13:22:23.033592+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:18:33.463457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.041602+0000","last_change":"2024-04-02T13:22:23.037131+0000","last_active":"2024-04-02T13:22:23.041602+0000","last_peered":"2024-04-02T13:22:23.041602+0000","last_clean":"2024-04-02T13:22:23.041602+0000","last_became_active":"2024-04-02T13:22:23.036685+0000","last_became_peered":"2024-04-02T13:22:23.036685+0000","last_unstale":"2024-04-02T13:22:23.041602+0000","last_undegraded":"2024-04-02T13:22:23.041602+0000","last_fullsized":"2024-04-02T13:22:23.041602+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:48.881381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.044840+0000","last_change":"2024-04-02T13:22:23.036926+0000","last_active":"2024-04-02T13:22:23.044840+0000","last_peered":"2024-04-02T13:22:23.044840+0000","last_clean":"2024-04-02T13:22:23.044840+0000","last_became_active":"2024-04-02T13:22:23.036606+0000","last_became_peered":"2024-04-02T13:22:23.036606+0000","last_unstale":"2024-04-02T13:22:23.044840+0000","last_undegraded":"2024-04-02T13:22:23.044840+0000","last_fullsized":"2024-04-02T13:22:23.044840+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:59:33.138858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045393+0000","last_change":"2024-04-02T13:22:23.039377+0000","last_active":"2024-04-02T13:22:23.045393+0000","last_peered":"2024-04-02T13:22:23.045393+0000","last_clean":"2024-04-02T13:22:23.045393+0000","last_became_active":"2024-04-02T13:22:23.039110+0000","last_became_peered":"2024-04-02T13:22:23.039110+0000","last_unstale":"2024-04-02T13:22:23.045393+0000","last_undegraded":"2024-04-02T13:22:23.045393+0000","last_fullsized":"2024-04-02T13:22:23.045393+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:54.364791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.050308+0000","last_change":"2024-04-02T13:22:23.046347+0000","last_active":"2024-04-02T13:22:23.050308+0000","last_peered":"2024-04-02T13:22:23.050308+0000","last_clean":"2024-04-02T13:22:23.050308+0000","last_became_active":"2024-04-02T13:22:23.046061+0000","last_became_peered":"2024-04-02T13:22:23.046061+0000","last_unstale":"2024-04-02T13:22:23.050308+0000","last_undegraded":"2024-04-02T13:22:23.050308+0000","last_fullsized":"2024-04-02T13:22:23.050308+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:49:32.798661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.050369+0000","last_change":"2024-04-02T13:22:23.042203+0000","last_active":"2024-04-02T13:22:23.050369+0000","last_peered":"2024-04-02T13:22:23.050369+0000","last_clean":"2024-04-02T13:22:23.050369+0000","last_became_active":"2024-04-02T13:22:23.041607+0000","last_became_peered":"2024-04-02T13:22:23.041607+0000","last_unstale":"2024-04-02T13:22:23.050369+0000","last_undegraded":"2024-04-02T13:22:23.050369+0000","last_fullsized":"2024-04-02T13:22:23.050369+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:56:23.553708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.044597+0000","last_change":"2024-04-02T13:22:23.034698+0000","last_active":"2024-04-02T13:22:23.044597+0000","last_peered":"2024-04-02T13:22:23.044597+0000","last_clean":"2024-04-02T13:22:23.044597+0000","last_became_active":"2024-04-02T13:22:23.034323+0000","last_became_peered":"2024-04-02T13:22:23.034323+0000","last_unstale":"2024-04-02T13:22:23.044597+0000","last_undegraded":"2024-04-02T13:22:23.044597+0000","last_fullsized":"2024-04-02T13:22:23.044597+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:18:05.259057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.043118+0000","last_change":"2024-04-02T13:22:23.035966+0000","last_active":"2024-04-02T13:22:23.043118+0000","last_peered":"2024-04-02T13:22:23.043118+0000","last_clean":"2024-04-02T13:22:23.043118+0000","last_became_active":"2024-04-02T13:22:23.035715+0000","last_became_peered":"2024-04-02T13:22:23.035715+0000","last_unstale":"2024-04-02T13:22:23.043118+0000","last_undegraded":"2024-04-02T13:22:23.043118+0000","last_fullsized":"2024-04-02T13:22:23.043118+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:31:21.359318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.050317+0000","last_change":"2024-04-02T13:22:23.042316+0000","last_active":"2024-04-02T13:22:23.050317+0000","last_peered":"2024-04-02T13:22:23.050317+0000","last_clean":"2024-04-02T13:22:23.050317+0000","last_became_active":"2024-04-02T13:22:23.042013+0000","last_became_peered":"2024-04-02T13:22:23.042013+0000","last_unstale":"2024-04-02T13:22:23.050317+0000","last_undegraded":"2024-04-02T13:22:23.050317+0000","last_fullsized":"2024-04-02T13:22:23.050317+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:22:35.034740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045513+0000","last_change":"2024-04-02T13:22:23.035938+0000","last_active":"2024-04-02T13:22:23.045513+0000","last_peered":"2024-04-02T13:22:23.045513+0000","last_clean":"2024-04-02T13:22:23.045513+0000","last_became_active":"2024-04-02T13:22:23.034813+0000","last_became_peered":"2024-04-02T13:22:23.034813+0000","last_unstale":"2024-04-02T13:22:23.045513+0000","last_undegraded":"2024-04-02T13:22:23.045513+0000","last_fullsized":"2024-04-02T13:22:23.045513+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:53.950028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045199+0000","last_change":"2024-04-02T13:22:23.038717+0000","last_active":"2024-04-02T13:22:23.045199+0000","last_peered":"2024-04-02T13:22:23.045199+0000","last_clean":"2024-04-02T13:22:23.045199+0000","last_became_active":"2024-04-02T13:22:23.038424+0000","last_became_peered":"2024-04-02T13:22:23.038424+0000","last_unstale":"2024-04-02T13:22:23.045199+0000","last_undegraded":"2024-04-02T13:22:23.045199+0000","last_fullsized":"2024-04-02T13:22:23.045199+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:30:33.169862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042349+0000","last_change":"2024-04-02T13:22:23.040984+0000","last_active":"2024-04-02T13:22:23.042349+0000","last_peered":"2024-04-02T13:22:23.042349+0000","last_clean":"2024-04-02T13:22:23.042349+0000","last_became_active":"2024-04-02T13:22:23.040645+0000","last_became_peered":"2024-04-02T13:22:23.040645+0000","last_unstale":"2024-04-02T13:22:23.042349+0000","last_undegraded":"2024-04-02T13:22:23.042349+0000","last_fullsized":"2024-04-02T13:22:23.042349+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:17:04.245119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.044750+0000","last_change":"2024-04-02T13:22:23.035200+0000","last_active":"2024-04-02T13:22:23.044750+0000","last_peered":"2024-04-02T13:22:23.044750+0000","last_clean":"2024-04-02T13:22:23.044750+0000","last_became_active":"2024-04-02T13:22:23.034896+0000","last_became_peered":"2024-04-02T13:22:23.034896+0000","last_unstale":"2024-04-02T13:22:23.044750+0000","last_undegraded":"2024-04-02T13:22:23.044750+0000","last_fullsized":"2024-04-02T13:22:23.044750+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:38:50.523334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047621+0000","last_change":"2024-04-02T13:22:23.043885+0000","last_active":"2024-04-02T13:22:23.047621+0000","last_peered":"2024-04-02T13:22:23.047621+0000","last_clean":"2024-04-02T13:22:23.047621+0000","last_became_active":"2024-04-02T13:22:23.043378+0000","last_became_peered":"2024-04-02T13:22:23.043378+0000","last_unstale":"2024-04-02T13:22:23.047621+0000","last_undegraded":"2024-04-02T13:22:23.047621+0000","last_fullsized":"2024-04-02T13:22:23.047621+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:41:22.513266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.041522+0000","last_change":"2024-04-02T13:22:23.040144+0000","last_active":"2024-04-02T13:22:23.041522+0000","last_peered":"2024-04-02T13:22:23.041522+0000","last_clean":"2024-04-02T13:22:23.041522+0000","last_became_active":"2024-04-02T13:22:23.039298+0000","last_became_peered":"2024-04-02T13:22:23.039298+0000","last_unstale":"2024-04-02T13:22:23.041522+0000","last_undegraded":"2024-04-02T13:22:23.041522+0000","last_fullsized":"2024-04-02T13:22:23.041522+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:10:58.898565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042331+0000","last_change":"2024-04-02T13:22:23.037200+0000","last_active":"2024-04-02T13:22:23.042331+0000","last_peered":"2024-04-02T13:22:23.042331+0000","last_clean":"2024-04-02T13:22:23.042331+0000","last_became_active":"2024-04-02T13:22:23.036883+0000","last_became_peered":"2024-04-02T13:22:23.036883+0000","last_unstale":"2024-04-02T13:22:23.042331+0000","last_undegraded":"2024-04-02T13:22:23.042331+0000","last_fullsized":"2024-04-02T13:22:23.042331+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:17.264918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042484+0000","last_change":"2024-04-02T13:22:23.040208+0000","last_active":"2024-04-02T13:22:23.042484+0000","last_peered":"2024-04-02T13:22:23.042484+0000","last_clean":"2024-04-02T13:22:23.042484+0000","last_became_active":"2024-04-02T13:22:23.039883+0000","last_became_peered":"2024-04-02T13:22:23.039883+0000","last_unstale":"2024-04-02T13:22:23.042484+0000","last_undegraded":"2024-04-02T13:22:23.042484+0000","last_fullsized":"2024-04-02T13:22:23.042484+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:17.264918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039730+0000","last_change":"2024-04-02T13:22:23.036840+0000","last_active":"2024-04-02T13:22:23.039730+0000","last_peered":"2024-04-02T13:22:23.039730+0000","last_clean":"2024-04-02T13:22:23.039730+0000","last_became_active":"2024-04-02T13:22:23.036568+0000","last_became_peered":"2024-04-02T13:22:23.036568+0000","last_unstale":"2024-04-02T13:22:23.039730+0000","last_undegraded":"2024-04-02T13:22:23.039730+0000","last_fullsized":"2024-04-02T13:22:23.039730+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:38:50.523334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047658+0000","last_change":"2024-04-02T13:22:23.041137+0000","last_active":"2024-04-02T13:22:23.047658+0000","last_peered":"2024-04-02T13:22:23.047658+0000","last_clean":"2024-04-02T13:22:23.047658+0000","last_became_active":"2024-04-02T13:22:23.040811+0000","last_became_peered":"2024-04-02T13:22:23.040811+0000","last_unstale":"2024-04-02T13:22:23.047658+0000","last_undegraded":"2024-04-02T13:22:23.047658+0000","last_fullsized":"2024-04-02T13:22:23.047658+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:04.281077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042889+0000","last_change":"2024-04-02T13:22:23.035328+0000","last_active":"2024-04-02T13:22:23.042889+0000","last_peered":"2024-04-02T13:22:23.042889+0000","last_clean":"2024-04-02T13:22:23.042889+0000","last_became_active":"2024-04-02T13:22:23.035041+0000","last_became_peered":"2024-04-02T13:22:23.035041+0000","last_unstale":"2024-04-02T13:22:23.042889+0000","last_undegraded":"2024-04-02T13:22:23.042889+0000","last_fullsized":"2024-04-02T13:22:23.042889+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:58:34.769523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045734+0000","last_change":"2024-04-02T13:22:23.040898+0000","last_active":"2024-04-02T13:22:23.045734+0000","last_peered":"2024-04-02T13:22:23.045734+0000","last_clean":"2024-04-02T13:22:23.045734+0000","last_became_active":"2024-04-02T13:22:23.040612+0000","last_became_peered":"2024-04-02T13:22:23.040612+0000","last_unstale":"2024-04-02T13:22:23.045734+0000","last_undegraded":"2024-04-02T13:22:23.045734+0000","last_fullsized":"2024-04-02T13:22:23.045734+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:10:23.686213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047648+0000","last_change":"2024-04-02T13:22:23.044186+0000","last_active":"2024-04-02T13:22:23.047648+0000","last_peered":"2024-04-02T13:22:23.047648+0000","last_clean":"2024-04-02T13:22:23.047648+0000","last_became_active":"2024-04-02T13:22:23.043772+0000","last_became_peered":"2024-04-02T13:22:23.043772+0000","last_unstale":"2024-04-02T13:22:23.047648+0000","last_undegraded":"2024-04-02T13:22:23.047648+0000","last_fullsized":"2024-04-02T13:22:23.047648+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:59.544978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042052+0000","last_change":"2024-04-02T13:22:23.036634+0000","last_active":"2024-04-02T13:22:23.042052+0000","last_peered":"2024-04-02T13:22:23.042052+0000","last_clean":"2024-04-02T13:22:23.042052+0000","last_became_active":"2024-04-02T13:22:23.036205+0000","last_became_peered":"2024-04-02T13:22:23.036205+0000","last_unstale":"2024-04-02T13:22:23.042052+0000","last_undegraded":"2024-04-02T13:22:23.042052+0000","last_fullsized":"2024-04-02T13:22:23.042052+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:37:25.803096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045050+0000","last_change":"2024-04-02T13:22:23.040147+0000","last_active":"2024-04-02T13:22:23.045050+0000","last_peered":"2024-04-02T13:22:23.045050+0000","last_clean":"2024-04-02T13:22:23.045050+0000","last_became_active":"2024-04-02T13:22:23.039316+0000","last_became_peered":"2024-04-02T13:22:23.039316+0000","last_unstale":"2024-04-02T13:22:23.045050+0000","last_undegraded":"2024-04-02T13:22:23.045050+0000","last_fullsized":"2024-04-02T13:22:23.045050+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:38.210075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042462+0000","last_change":"2024-04-02T13:22:23.040164+0000","last_active":"2024-04-02T13:22:23.042462+0000","last_peered":"2024-04-02T13:22:23.042462+0000","last_clean":"2024-04-02T13:22:23.042462+0000","last_became_active":"2024-04-02T13:22:23.039870+0000","last_became_peered":"2024-04-02T13:22:23.039870+0000","last_unstale":"2024-04-02T13:22:23.042462+0000","last_undegraded":"2024-04-02T13:22:23.042462+0000","last_fullsized":"2024-04-02T13:22:23.042462+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:53.950028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.053898+0000","last_change":"2024-04-02T13:22:23.045746+0000","last_active":"2024-04-02T13:22:23.053898+0000","last_peered":"2024-04-02T13:22:23.053898+0000","last_clean":"2024-04-02T13:22:23.053898+0000","last_became_active":"2024-04-02T13:22:23.045455+0000","last_became_peered":"2024-04-02T13:22:23.045455+0000","last_unstale":"2024-04-02T13:22:23.053898+0000","last_undegraded":"2024-04-02T13:22:23.053898+0000","last_fullsized":"2024-04-02T13:22:23.053898+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:35.277058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.035653+0000","last_change":"2024-04-02T13:22:23.033659+0000","last_active":"2024-04-02T13:22:23.035653+0000","last_peered":"2024-04-02T13:22:23.035653+0000","last_clean":"2024-04-02T13:22:23.035653+0000","last_became_active":"2024-04-02T13:22:23.033351+0000","last_became_peered":"2024-04-02T13:22:23.033351+0000","last_unstale":"2024-04-02T13:22:23.035653+0000","last_undegraded":"2024-04-02T13:22:23.035653+0000","last_fullsized":"2024-04-02T13:22:23.035653+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:04.247519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045370+0000","last_change":"2024-04-02T13:22:23.039319+0000","last_active":"2024-04-02T13:22:23.045370+0000","last_peered":"2024-04-02T13:22:23.045370+0000","last_clean":"2024-04-02T13:22:23.045370+0000","last_became_active":"2024-04-02T13:22:23.038317+0000","last_became_peered":"2024-04-02T13:22:23.038317+0000","last_unstale":"2024-04-02T13:22:23.045370+0000","last_undegraded":"2024-04-02T13:22:23.045370+0000","last_fullsized":"2024-04-02T13:22:23.045370+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:55:36.337222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039805+0000","last_change":"2024-04-02T13:22:23.038665+0000","last_active":"2024-04-02T13:22:23.039805+0000","last_peered":"2024-04-02T13:22:23.039805+0000","last_clean":"2024-04-02T13:22:23.039805+0000","last_became_active":"2024-04-02T13:22:23.038394+0000","last_became_peered":"2024-04-02T13:22:23.038394+0000","last_unstale":"2024-04-02T13:22:23.039805+0000","last_undegraded":"2024-04-02T13:22:23.039805+0000","last_fullsized":"2024-04-02T13:22:23.039805+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:51.961849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.037368+0000","last_change":"2024-04-02T13:22:23.035212+0000","last_active":"2024-04-02T13:22:23.037368+0000","last_peered":"2024-04-02T13:22:23.037368+0000","last_clean":"2024-04-02T13:22:23.037368+0000","last_became_active":"2024-04-02T13:22:23.034895+0000","last_became_peered":"2024-04-02T13:22:23.034895+0000","last_unstale":"2024-04-02T13:22:23.037368+0000","last_undegraded":"2024-04-02T13:22:23.037368+0000","last_fullsized":"2024-04-02T13:22:23.037368+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:59.544978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.054821+0000","last_change":"2024-04-02T13:22:22.023171+0000","last_active":"2024-04-02T13:22:23.054821+0000","last_peered":"2024-04-02T13:22:23.054821+0000","last_clean":"2024-04-02T13:22:23.054821+0000","last_became_active":"2024-04-02T13:22:22.022887+0000","last_became_peered":"2024-04-02T13:22:22.022887+0000","last_unstale":"2024-04-02T13:22:23.054821+0000","last_undegraded":"2024-04-02T13:22:23.054821+0000","last_fullsized":"2024-04-02T13:22:23.054821+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:32:17.085706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039365+0000","last_change":"2024-04-02T13:22:23.035768+0000","last_active":"2024-04-02T13:22:23.039365+0000","last_peered":"2024-04-02T13:22:23.039365+0000","last_clean":"2024-04-02T13:22:23.039365+0000","last_became_active":"2024-04-02T13:22:23.035005+0000","last_became_peered":"2024-04-02T13:22:23.035005+0000","last_unstale":"2024-04-02T13:22:23.039365+0000","last_undegraded":"2024-04-02T13:22:23.039365+0000","last_fullsized":"2024-04-02T13:22:23.039365+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:41:22.513266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.056444+0000","last_change":"2024-04-02T13:22:22.022403+0000","last_active":"2024-04-02T13:22:23.056444+0000","last_peered":"2024-04-02T13:22:23.056444+0000","last_clean":"2024-04-02T13:22:23.056444+0000","last_became_active":"2024-04-02T13:22:22.022136+0000","last_became_peered":"2024-04-02T13:22:22.022136+0000","last_unstale":"2024-04-02T13:22:23.056444+0000","last_undegraded":"2024-04-02T13:22:23.056444+0000","last_fullsized":"2024-04-02T13:22:23.056444+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:24:04.737397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042523+0000","last_change":"2024-04-02T13:22:23.040354+0000","last_active":"2024-04-02T13:22:23.042523+0000","last_peered":"2024-04-02T13:22:23.042523+0000","last_clean":"2024-04-02T13:22:23.042523+0000","last_became_active":"2024-04-02T13:22:23.039514+0000","last_became_peered":"2024-04-02T13:22:23.039514+0000","last_unstale":"2024-04-02T13:22:23.042523+0000","last_undegraded":"2024-04-02T13:22:23.042523+0000","last_fullsized":"2024-04-02T13:22:23.042523+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:30:33.169862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.055728+0000","last_change":"2024-04-02T13:22:22.023345+0000","last_active":"2024-04-02T13:22:23.055728+0000","last_peered":"2024-04-02T13:22:23.055728+0000","last_clean":"2024-04-02T13:22:23.055728+0000","last_became_active":"2024-04-02T13:22:22.022938+0000","last_became_peered":"2024-04-02T13:22:22.022938+0000","last_unstale":"2024-04-02T13:22:23.055728+0000","last_undegraded":"2024-04-02T13:22:23.055728+0000","last_fullsized":"2024-04-02T13:22:23.055728+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:29:41.604642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.037748+0000","last_change":"2024-04-02T13:22:23.035616+0000","last_active":"2024-04-02T13:22:23.037748+0000","last_peered":"2024-04-02T13:22:23.037748+0000","last_clean":"2024-04-02T13:22:23.037748+0000","last_became_active":"2024-04-02T13:22:23.035013+0000","last_became_peered":"2024-04-02T13:22:23.035013+0000","last_unstale":"2024-04-02T13:22:23.037748+0000","last_undegraded":"2024-04-02T13:22:23.037748+0000","last_fullsized":"2024-04-02T13:22:23.037748+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:35:42.066923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.058058+0000","last_change":"2024-04-02T13:22:22.023284+0000","last_active":"2024-04-02T13:22:23.058058+0000","last_peered":"2024-04-02T13:22:23.058058+0000","last_clean":"2024-04-02T13:22:23.058058+0000","last_became_active":"2024-04-02T13:22:22.023017+0000","last_became_peered":"2024-04-02T13:22:22.023017+0000","last_unstale":"2024-04-02T13:22:23.058058+0000","last_undegraded":"2024-04-02T13:22:23.058058+0000","last_fullsized":"2024-04-02T13:22:23.058058+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:51.321428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.041938+0000","last_change":"2024-04-02T13:22:23.038949+0000","last_active":"2024-04-02T13:22:23.041938+0000","last_peered":"2024-04-02T13:22:23.041938+0000","last_clean":"2024-04-02T13:22:23.041938+0000","last_became_active":"2024-04-02T13:22:23.038640+0000","last_became_peered":"2024-04-02T13:22:23.038640+0000","last_unstale":"2024-04-02T13:22:23.041938+0000","last_undegraded":"2024-04-02T13:22:23.041938+0000","last_fullsized":"2024-04-02T13:22:23.041938+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:38:50.523334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.060104+0000","last_change":"2024-04-02T13:22:22.031515+0000","last_active":"2024-04-02T13:22:23.060104+0000","last_peered":"2024-04-02T13:22:23.060104+0000","last_clean":"2024-04-02T13:22:23.060104+0000","last_became_active":"2024-04-02T13:22:22.031230+0000","last_became_peered":"2024-04-02T13:22:22.031230+0000","last_unstale":"2024-04-02T13:22:23.060104+0000","last_undegraded":"2024-04-02T13:22:23.060104+0000","last_fullsized":"2024-04-02T13:22:23.060104+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:51.321428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.035753+0000","last_change":"2024-04-02T13:22:23.034001+0000","last_active":"2024-04-02T13:22:23.035753+0000","last_peered":"2024-04-02T13:22:23.035753+0000","last_clean":"2024-04-02T13:22:23.035753+0000","last_became_active":"2024-04-02T13:22:23.033723+0000","last_became_peered":"2024-04-02T13:22:23.033723+0000","last_unstale":"2024-04-02T13:22:23.035753+0000","last_undegraded":"2024-04-02T13:22:23.035753+0000","last_fullsized":"2024-04-02T13:22:23.035753+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:41:22.513266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.053761+0000","last_change":"2024-04-02T13:22:22.019210+0000","last_active":"2024-04-02T13:22:23.053761+0000","last_peered":"2024-04-02T13:22:23.053761+0000","last_clean":"2024-04-02T13:22:23.053761+0000","last_became_active":"2024-04-02T13:22:22.018868+0000","last_became_peered":"2024-04-02T13:22:22.018868+0000","last_unstale":"2024-04-02T13:22:23.053761+0000","last_undegraded":"2024-04-02T13:22:23.053761+0000","last_fullsized":"2024-04-02T13:22:23.053761+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:12.453015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047676+0000","last_change":"2024-04-02T13:22:23.036999+0000","last_active":"2024-04-02T13:22:23.047676+0000","last_peered":"2024-04-02T13:22:23.047676+0000","last_clean":"2024-04-02T13:22:23.047676+0000","last_became_active":"2024-04-02T13:22:23.036602+0000","last_became_peered":"2024-04-02T13:22:23.036602+0000","last_unstale":"2024-04-02T13:22:23.047676+0000","last_undegraded":"2024-04-02T13:22:23.047676+0000","last_fullsized":"2024-04-02T13:22:23.047676+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:56:56.678078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.029459+0000","last_change":"2024-04-02T13:22:22.024170+0000","last_active":"2024-04-02T13:22:23.029459+0000","last_peered":"2024-04-02T13:22:23.029459+0000","last_clean":"2024-04-02T13:22:23.029459+0000","last_became_active":"2024-04-02T13:22:22.023880+0000","last_became_peered":"2024-04-02T13:22:22.023880+0000","last_unstale":"2024-04-02T13:22:23.029459+0000","last_undegraded":"2024-04-02T13:22:23.029459+0000","last_fullsized":"2024-04-02T13:22:23.029459+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:29:41.604642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.040937+0000","last_change":"2024-04-02T13:22:23.039340+0000","last_active":"2024-04-02T13:22:23.040937+0000","last_peered":"2024-04-02T13:22:23.040937+0000","last_clean":"2024-04-02T13:22:23.040937+0000","last_became_active":"2024-04-02T13:22:23.039090+0000","last_became_peered":"2024-04-02T13:22:23.039090+0000","last_unstale":"2024-04-02T13:22:23.040937+0000","last_undegraded":"2024-04-02T13:22:23.040937+0000","last_fullsized":"2024-04-02T13:22:23.040937+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:51.961849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.040333+0000","last_change":"2024-04-02T13:22:22.030883+0000","last_active":"2024-04-02T13:22:23.040333+0000","last_peered":"2024-04-02T13:22:23.040333+0000","last_clean":"2024-04-02T13:22:23.040333+0000","last_became_active":"2024-04-02T13:22:22.030643+0000","last_became_peered":"2024-04-02T13:22:22.030643+0000","last_unstale":"2024-04-02T13:22:23.040333+0000","last_undegraded":"2024-04-02T13:22:23.040333+0000","last_fullsized":"2024-04-02T13:22:23.040333+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:25.658945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.036158+0000","last_change":"2024-04-02T13:22:23.031013+0000","last_active":"2024-04-02T13:22:23.036158+0000","last_peered":"2024-04-02T13:22:23.036158+0000","last_clean":"2024-04-02T13:22:23.036158+0000","last_became_active":"2024-04-02T13:22:23.030716+0000","last_became_peered":"2024-04-02T13:22:23.030716+0000","last_unstale":"2024-04-02T13:22:23.036158+0000","last_undegraded":"2024-04-02T13:22:23.036158+0000","last_fullsized":"2024-04-02T13:22:23.036158+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:18:00.483711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.029075+0000","last_change":"2024-04-02T13:22:22.023831+0000","last_active":"2024-04-02T13:22:23.029075+0000","last_peered":"2024-04-02T13:22:23.029075+0000","last_clean":"2024-04-02T13:22:23.029075+0000","last_became_active":"2024-04-02T13:22:22.023496+0000","last_became_peered":"2024-04-02T13:22:22.023496+0000","last_unstale":"2024-04-02T13:22:23.029075+0000","last_undegraded":"2024-04-02T13:22:23.029075+0000","last_fullsized":"2024-04-02T13:22:23.029075+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:49:32.982039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039195+0000","last_change":"2024-04-02T13:22:23.035333+0000","last_active":"2024-04-02T13:22:23.039195+0000","last_peered":"2024-04-02T13:22:23.039195+0000","last_clean":"2024-04-02T13:22:23.039195+0000","last_became_active":"2024-04-02T13:22:23.034614+0000","last_became_peered":"2024-04-02T13:22:23.034614+0000","last_unstale":"2024-04-02T13:22:23.039195+0000","last_undegraded":"2024-04-02T13:22:23.039195+0000","last_fullsized":"2024-04-02T13:22:23.039195+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:10:23.686213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.025408+0000","last_change":"2024-04-02T13:22:22.026761+0000","last_active":"2024-04-02T13:22:23.025408+0000","last_peered":"2024-04-02T13:22:23.025408+0000","last_clean":"2024-04-02T13:22:23.025408+0000","last_became_active":"2024-04-02T13:22:22.026460+0000","last_became_peered":"2024-04-02T13:22:22.026460+0000","last_unstale":"2024-04-02T13:22:23.025408+0000","last_undegraded":"2024-04-02T13:22:23.025408+0000","last_fullsized":"2024-04-02T13:22:23.025408+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:24:04.737397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045792+0000","last_change":"2024-04-02T13:22:23.041432+0000","last_active":"2024-04-02T13:22:23.045792+0000","last_peered":"2024-04-02T13:22:23.045792+0000","last_clean":"2024-04-02T13:22:23.045792+0000","last_became_active":"2024-04-02T13:22:23.041174+0000","last_became_peered":"2024-04-02T13:22:23.041174+0000","last_unstale":"2024-04-02T13:22:23.045792+0000","last_undegraded":"2024-04-02T13:22:23.045792+0000","last_fullsized":"2024-04-02T13:22:23.045792+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:18:33.463457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.057937+0000","last_change":"2024-04-02T13:22:22.027465+0000","last_active":"2024-04-02T13:22:23.057937+0000","last_peered":"2024-04-02T13:22:23.057937+0000","last_clean":"2024-04-02T13:22:23.057937+0000","last_became_active":"2024-04-02T13:22:22.027213+0000","last_became_peered":"2024-04-02T13:22:22.027213+0000","last_unstale":"2024-04-02T13:22:23.057937+0000","last_undegraded":"2024-04-02T13:22:23.057937+0000","last_fullsized":"2024-04-02T13:22:23.057937+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:25.658945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042405+0000","last_change":"2024-04-02T13:22:23.041462+0000","last_active":"2024-04-02T13:22:23.042405+0000","last_peered":"2024-04-02T13:22:23.042405+0000","last_clean":"2024-04-02T13:22:23.042405+0000","last_became_active":"2024-04-02T13:22:23.041190+0000","last_became_peered":"2024-04-02T13:22:23.041190+0000","last_unstale":"2024-04-02T13:22:23.042405+0000","last_undegraded":"2024-04-02T13:22:23.042405+0000","last_fullsized":"2024-04-02T13:22:23.042405+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:53.950028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":18,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.061790+0000","last_change":"2024-04-02T13:22:22.025433+0000","last_active":"2024-04-02T13:22:23.061790+0000","last_peered":"2024-04-02T13:22:23.061790+0000","last_clean":"2024-04-02T13:22:23.061790+0000","last_became_active":"2024-04-02T13:22:22.025190+0000","last_became_peered":"2024-04-02T13:22:22.025190+0000","last_unstale":"2024-04-02T13:22:23.061790+0000","last_undegraded":"2024-04-02T13:22:23.061790+0000","last_fullsized":"2024-04-02T13:22:23.061790+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:10:57.894038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039768+0000","last_change":"2024-04-02T13:22:23.035548+0000","last_active":"2024-04-02T13:22:23.039768+0000","last_peered":"2024-04-02T13:22:23.039768+0000","last_clean":"2024-04-02T13:22:23.039768+0000","last_became_active":"2024-04-02T13:22:23.035057+0000","last_became_peered":"2024-04-02T13:22:23.035057+0000","last_unstale":"2024-04-02T13:22:23.039768+0000","last_undegraded":"2024-04-02T13:22:23.039768+0000","last_fullsized":"2024-04-02T13:22:23.039768+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:04.247519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.058099+0000","last_change":"2024-04-02T13:22:22.030861+0000","last_active":"2024-04-02T13:22:23.058099+0000","last_peered":"2024-04-02T13:22:23.058099+0000","last_clean":"2024-04-02T13:22:23.058099+0000","last_became_active":"2024-04-02T13:22:22.030416+0000","last_became_peered":"2024-04-02T13:22:22.030416+0000","last_unstale":"2024-04-02T13:22:23.058099+0000","last_undegraded":"2024-04-02T13:22:23.058099+0000","last_fullsized":"2024-04-02T13:22:23.058099+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:29:35.357053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.037365+0000","last_change":"2024-04-02T13:22:23.035360+0000","last_active":"2024-04-02T13:22:23.037365+0000","last_peered":"2024-04-02T13:22:23.037365+0000","last_clean":"2024-04-02T13:22:23.037365+0000","last_became_active":"2024-04-02T13:22:23.035070+0000","last_became_peered":"2024-04-02T13:22:23.035070+0000","last_unstale":"2024-04-02T13:22:23.037365+0000","last_undegraded":"2024-04-02T13:22:23.037365+0000","last_fullsized":"2024-04-02T13:22:23.037365+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:59:33.138858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.031945+0000","last_change":"2024-04-02T13:22:22.032136+0000","last_active":"2024-04-02T13:22:23.031945+0000","last_peered":"2024-04-02T13:22:23.031945+0000","last_clean":"2024-04-02T13:22:23.031945+0000","last_became_active":"2024-04-02T13:22:22.031828+0000","last_became_peered":"2024-04-02T13:22:22.031828+0000","last_unstale":"2024-04-02T13:22:23.031945+0000","last_undegraded":"2024-04-02T13:22:23.031945+0000","last_fullsized":"2024-04-02T13:22:23.031945+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:12.453015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.037740+0000","last_change":"2024-04-02T13:22:23.033476+0000","last_active":"2024-04-02T13:22:23.037740+0000","last_peered":"2024-04-02T13:22:23.037740+0000","last_clean":"2024-04-02T13:22:23.037740+0000","last_became_active":"2024-04-02T13:22:23.033170+0000","last_became_peered":"2024-04-02T13:22:23.033170+0000","last_unstale":"2024-04-02T13:22:23.037740+0000","last_undegraded":"2024-04-02T13:22:23.037740+0000","last_fullsized":"2024-04-02T13:22:23.037740+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:17:04.245119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.027168+0000","last_change":"2024-04-02T13:22:22.024652+0000","last_active":"2024-04-02T13:22:23.027168+0000","last_peered":"2024-04-02T13:22:23.027168+0000","last_clean":"2024-04-02T13:22:23.027168+0000","last_became_active":"2024-04-02T13:22:22.024371+0000","last_became_peered":"2024-04-02T13:22:22.024371+0000","last_unstale":"2024-04-02T13:22:23.027168+0000","last_undegraded":"2024-04-02T13:22:23.027168+0000","last_fullsized":"2024-04-02T13:22:23.027168+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:51.321428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047668+0000","last_change":"2024-04-02T13:22:23.044787+0000","last_active":"2024-04-02T13:22:23.047668+0000","last_peered":"2024-04-02T13:22:23.047668+0000","last_clean":"2024-04-02T13:22:23.047668+0000","last_became_active":"2024-04-02T13:22:23.044295+0000","last_became_peered":"2024-04-02T13:22:23.044295+0000","last_unstale":"2024-04-02T13:22:23.047668+0000","last_undegraded":"2024-04-02T13:22:23.047668+0000","last_fullsized":"2024-04-02T13:22:23.047668+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:51.961849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":20,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.060163+0000","last_change":"2024-04-02T13:22:22.022925+0000","last_active":"2024-04-02T13:22:23.060163+0000","last_peered":"2024-04-02T13:22:23.060163+0000","last_clean":"2024-04-02T13:22:23.060163+0000","last_became_active":"2024-04-02T13:22:22.022662+0000","last_became_peered":"2024-04-02T13:22:22.022662+0000","last_unstale":"2024-04-02T13:22:23.060163+0000","last_undegraded":"2024-04-02T13:22:23.060163+0000","last_fullsized":"2024-04-02T13:22:23.060163+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:34.272531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.037724+0000","last_change":"2024-04-02T13:22:23.035631+0000","last_active":"2024-04-02T13:22:23.037724+0000","last_peered":"2024-04-02T13:22:23.037724+0000","last_clean":"2024-04-02T13:22:23.037724+0000","last_became_active":"2024-04-02T13:22:23.035330+0000","last_became_peered":"2024-04-02T13:22:23.035330+0000","last_unstale":"2024-04-02T13:22:23.037724+0000","last_undegraded":"2024-04-02T13:22:23.037724+0000","last_fullsized":"2024-04-02T13:22:23.037724+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:49:36.074639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.058015+0000","last_change":"2024-04-02T13:22:22.031587+0000","last_active":"2024-04-02T13:22:23.058015+0000","last_peered":"2024-04-02T13:22:23.058015+0000","last_clean":"2024-04-02T13:22:23.058015+0000","last_became_active":"2024-04-02T13:22:22.031325+0000","last_became_peered":"2024-04-02T13:22:22.031325+0000","last_unstale":"2024-04-02T13:22:23.058015+0000","last_undegraded":"2024-04-02T13:22:23.058015+0000","last_fullsized":"2024-04-02T13:22:23.058015+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:18:04.254530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045019+0000","last_change":"2024-04-02T13:22:23.039729+0000","last_active":"2024-04-02T13:22:23.045019+0000","last_peered":"2024-04-02T13:22:23.045019+0000","last_clean":"2024-04-02T13:22:23.045019+0000","last_became_active":"2024-04-02T13:22:23.039281+0000","last_became_peered":"2024-04-02T13:22:23.039281+0000","last_unstale":"2024-04-02T13:22:23.045019+0000","last_undegraded":"2024-04-02T13:22:23.045019+0000","last_fullsized":"2024-04-02T13:22:23.045019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:57:23.743511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.056301+0000","last_change":"2024-04-02T13:22:22.023564+0000","last_active":"2024-04-02T13:22:23.056301+0000","last_peered":"2024-04-02T13:22:23.056301+0000","last_clean":"2024-04-02T13:22:23.056301+0000","last_became_active":"2024-04-02T13:22:22.023273+0000","last_became_peered":"2024-04-02T13:22:22.023273+0000","last_unstale":"2024-04-02T13:22:23.056301+0000","last_undegraded":"2024-04-02T13:22:23.056301+0000","last_fullsized":"2024-04-02T13:22:23.056301+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:52:08.715533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.041702+0000","last_change":"2024-04-02T13:22:23.038304+0000","last_active":"2024-04-02T13:22:23.041702+0000","last_peered":"2024-04-02T13:22:23.041702+0000","last_clean":"2024-04-02T13:22:23.041702+0000","last_became_active":"2024-04-02T13:22:23.038037+0000","last_became_peered":"2024-04-02T13:22:23.038037+0000","last_unstale":"2024-04-02T13:22:23.041702+0000","last_undegraded":"2024-04-02T13:22:23.041702+0000","last_fullsized":"2024-04-02T13:22:23.041702+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:18:05.259057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.055601+0000","last_change":"2024-04-02T13:22:22.021325+0000","last_active":"2024-04-02T13:22:23.055601+0000","last_peered":"2024-04-02T13:22:23.055601+0000","last_clean":"2024-04-02T13:22:23.055601+0000","last_became_active":"2024-04-02T13:22:22.020527+0000","last_became_peered":"2024-04-02T13:22:22.020527+0000","last_unstale":"2024-04-02T13:22:23.055601+0000","last_undegraded":"2024-04-02T13:22:23.055601+0000","last_fullsized":"2024-04-02T13:22:23.055601+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:47.876854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045554+0000","last_change":"2024-04-02T13:22:23.038942+0000","last_active":"2024-04-02T13:22:23.045554+0000","last_peered":"2024-04-02T13:22:23.045554+0000","last_clean":"2024-04-02T13:22:23.045554+0000","last_became_active":"2024-04-02T13:22:23.038606+0000","last_became_peered":"2024-04-02T13:22:23.038606+0000","last_unstale":"2024-04-02T13:22:23.045554+0000","last_undegraded":"2024-04-02T13:22:23.045554+0000","last_fullsized":"2024-04-02T13:22:23.045554+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:30:33.169862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.032246+0000","last_change":"2024-04-02T13:22:22.023901+0000","last_active":"2024-04-02T13:22:23.032246+0000","last_peered":"2024-04-02T13:22:23.032246+0000","last_clean":"2024-04-02T13:22:23.032246+0000","last_became_active":"2024-04-02T13:22:22.023608+0000","last_became_peered":"2024-04-02T13:22:22.023608+0000","last_unstale":"2024-04-02T13:22:23.032246+0000","last_undegraded":"2024-04-02T13:22:23.032246+0000","last_fullsized":"2024-04-02T13:22:23.032246+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:17:59.479184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039013+0000","last_change":"2024-04-02T13:22:23.035009+0000","last_active":"2024-04-02T13:22:23.039013+0000","last_peered":"2024-04-02T13:22:23.039013+0000","last_clean":"2024-04-02T13:22:23.039013+0000","last_became_active":"2024-04-02T13:22:23.034677+0000","last_became_peered":"2024-04-02T13:22:23.034677+0000","last_unstale":"2024-04-02T13:22:23.039013+0000","last_undegraded":"2024-04-02T13:22:23.039013+0000","last_fullsized":"2024-04-02T13:22:23.039013+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:26.663472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.056050+0000","last_change":"2024-04-02T13:22:22.024810+0000","last_active":"2024-04-02T13:22:23.056050+0000","last_peered":"2024-04-02T13:22:23.056050+0000","last_clean":"2024-04-02T13:22:23.056050+0000","last_became_active":"2024-04-02T13:22:22.024398+0000","last_became_peered":"2024-04-02T13:22:22.024398+0000","last_unstale":"2024-04-02T13:22:23.056050+0000","last_undegraded":"2024-04-02T13:22:23.056050+0000","last_fullsized":"2024-04-02T13:22:23.056050+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:12.453015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042484+0000","last_change":"2024-04-02T13:22:23.040187+0000","last_active":"2024-04-02T13:22:23.042484+0000","last_peered":"2024-04-02T13:22:23.042484+0000","last_clean":"2024-04-02T13:22:23.042484+0000","last_became_active":"2024-04-02T13:22:23.039647+0000","last_became_peered":"2024-04-02T13:22:23.039647+0000","last_unstale":"2024-04-02T13:22:23.042484+0000","last_undegraded":"2024-04-02T13:22:23.042484+0000","last_fullsized":"2024-04-02T13:22:23.042484+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:59:33.138858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.024466+0000","last_change":"2024-04-02T13:22:22.024822+0000","last_active":"2024-04-02T13:22:23.024466+0000","last_peered":"2024-04-02T13:22:23.024466+0000","last_clean":"2024-04-02T13:22:23.024466+0000","last_became_active":"2024-04-02T13:22:22.024542+0000","last_became_peered":"2024-04-02T13:22:22.024542+0000","last_unstale":"2024-04-02T13:22:23.024466+0000","last_undegraded":"2024-04-02T13:22:23.024466+0000","last_fullsized":"2024-04-02T13:22:23.024466+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:41:21.508739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047685+0000","last_change":"2024-04-02T13:22:23.042906+0000","last_active":"2024-04-02T13:22:23.047685+0000","last_peered":"2024-04-02T13:22:23.047685+0000","last_clean":"2024-04-02T13:22:23.047685+0000","last_became_active":"2024-04-02T13:22:23.042644+0000","last_became_peered":"2024-04-02T13:22:23.042644+0000","last_unstale":"2024-04-02T13:22:23.047685+0000","last_undegraded":"2024-04-02T13:22:23.047685+0000","last_fullsized":"2024-04-02T13:22:23.047685+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:04.247519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.033755+0000","last_change":"2024-04-02T13:22:22.028715+0000","last_active":"2024-04-02T13:22:23.033755+0000","last_peered":"2024-04-02T13:22:23.033755+0000","last_clean":"2024-04-02T13:22:23.033755+0000","last_became_active":"2024-04-02T13:22:22.028465+0000","last_became_peered":"2024-04-02T13:22:22.028465+0000","last_unstale":"2024-04-02T13:22:23.033755+0000","last_undegraded":"2024-04-02T13:22:23.033755+0000","last_fullsized":"2024-04-02T13:22:23.033755+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:38:49.518806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039899+0000","last_change":"2024-04-02T13:22:23.032289+0000","last_active":"2024-04-02T13:22:23.039899+0000","last_peered":"2024-04-02T13:22:23.039899+0000","last_clean":"2024-04-02T13:22:23.039899+0000","last_became_active":"2024-04-02T13:22:23.031957+0000","last_became_peered":"2024-04-02T13:22:23.031957+0000","last_unstale":"2024-04-02T13:22:23.039899+0000","last_undegraded":"2024-04-02T13:22:23.039899+0000","last_fullsized":"2024-04-02T13:22:23.039899+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:48.881381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.019829+0000","last_change":"2024-04-02T13:22:22.022886+0000","last_active":"2024-04-02T13:22:23.019829+0000","last_peered":"2024-04-02T13:22:23.019829+0000","last_clean":"2024-04-02T13:22:23.019829+0000","last_became_active":"2024-04-02T13:22:22.022599+0000","last_became_peered":"2024-04-02T13:22:22.022599+0000","last_unstale":"2024-04-02T13:22:23.019829+0000","last_undegraded":"2024-04-02T13:22:23.019829+0000","last_fullsized":"2024-04-02T13:22:23.019829+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:05:47.082744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039783+0000","last_change":"2024-04-02T13:22:23.037812+0000","last_active":"2024-04-02T13:22:23.039783+0000","last_peered":"2024-04-02T13:22:23.039783+0000","last_clean":"2024-04-02T13:22:23.039783+0000","last_became_active":"2024-04-02T13:22:23.037498+0000","last_became_peered":"2024-04-02T13:22:23.037498+0000","last_unstale":"2024-04-02T13:22:23.039783+0000","last_undegraded":"2024-04-02T13:22:23.039783+0000","last_fullsized":"2024-04-02T13:22:23.039783+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:59.544978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.058418+0000","last_change":"2024-04-02T13:22:22.031817+0000","last_active":"2024-04-02T13:22:23.058418+0000","last_peered":"2024-04-02T13:22:23.058418+0000","last_clean":"2024-04-02T13:22:23.058418+0000","last_became_active":"2024-04-02T13:22:22.031562+0000","last_became_peered":"2024-04-02T13:22:22.031562+0000","last_unstale":"2024-04-02T13:22:23.058418+0000","last_undegraded":"2024-04-02T13:22:23.058418+0000","last_fullsized":"2024-04-02T13:22:23.058418+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:24:04.737397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.032641+0000","last_change":"2024-04-02T13:22:23.029419+0000","last_active":"2024-04-02T13:22:23.032641+0000","last_peered":"2024-04-02T13:22:23.032641+0000","last_clean":"2024-04-02T13:22:23.032641+0000","last_became_active":"2024-04-02T13:22:23.029100+0000","last_became_peered":"2024-04-02T13:22:23.029100+0000","last_unstale":"2024-04-02T13:22:23.032641+0000","last_undegraded":"2024-04-02T13:22:23.032641+0000","last_fullsized":"2024-04-02T13:22:23.032641+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:18:00.483711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.029591+0000","last_change":"2024-04-02T13:22:22.026307+0000","last_active":"2024-04-02T13:22:23.029591+0000","last_peered":"2024-04-02T13:22:23.029591+0000","last_clean":"2024-04-02T13:22:23.029591+0000","last_became_active":"2024-04-02T13:22:22.025780+0000","last_became_peered":"2024-04-02T13:22:22.025780+0000","last_unstale":"2024-04-02T13:22:23.029591+0000","last_undegraded":"2024-04-02T13:22:23.029591+0000","last_fullsized":"2024-04-02T13:22:23.029591+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:10:22.681686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.037765+0000","last_change":"2024-04-02T13:22:23.036100+0000","last_active":"2024-04-02T13:22:23.037765+0000","last_peered":"2024-04-02T13:22:23.037765+0000","last_clean":"2024-04-02T13:22:23.037765+0000","last_became_active":"2024-04-02T13:22:23.035392+0000","last_became_peered":"2024-04-02T13:22:23.035392+0000","last_unstale":"2024-04-02T13:22:23.037765+0000","last_undegraded":"2024-04-02T13:22:23.037765+0000","last_fullsized":"2024-04-02T13:22:23.037765+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:27:00.329601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.034353+0000","last_change":"2024-04-02T13:22:22.029527+0000","last_active":"2024-04-02T13:22:23.034353+0000","last_peered":"2024-04-02T13:22:23.034353+0000","last_clean":"2024-04-02T13:22:23.034353+0000","last_became_active":"2024-04-02T13:22:22.029176+0000","last_became_peered":"2024-04-02T13:22:22.029176+0000","last_unstale":"2024-04-02T13:22:23.034353+0000","last_undegraded":"2024-04-02T13:22:23.034353+0000","last_fullsized":"2024-04-02T13:22:23.034353+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:52.945501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045753+0000","last_change":"2024-04-02T13:22:23.039965+0000","last_active":"2024-04-02T13:22:23.045753+0000","last_peered":"2024-04-02T13:22:23.045753+0000","last_clean":"2024-04-02T13:22:23.045753+0000","last_became_active":"2024-04-02T13:22:23.039706+0000","last_became_peered":"2024-04-02T13:22:23.039706+0000","last_unstale":"2024-04-02T13:22:23.045753+0000","last_undegraded":"2024-04-02T13:22:23.045753+0000","last_fullsized":"2024-04-02T13:22:23.045753+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:49:33.986566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.024972+0000","last_change":"2024-04-02T13:22:22.024186+0000","last_active":"2024-04-02T13:22:23.024972+0000","last_peered":"2024-04-02T13:22:23.024972+0000","last_clean":"2024-04-02T13:22:23.024972+0000","last_became_active":"2024-04-02T13:22:22.023418+0000","last_became_peered":"2024-04-02T13:22:22.023418+0000","last_unstale":"2024-04-02T13:22:23.024972+0000","last_undegraded":"2024-04-02T13:22:23.024972+0000","last_fullsized":"2024-04-02T13:22:23.024972+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:50.957322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039267+0000","last_change":"2024-04-02T13:22:23.035490+0000","last_active":"2024-04-02T13:22:23.039267+0000","last_peered":"2024-04-02T13:22:23.039267+0000","last_clean":"2024-04-02T13:22:23.039267+0000","last_became_active":"2024-04-02T13:22:23.035227+0000","last_became_peered":"2024-04-02T13:22:23.035227+0000","last_unstale":"2024-04-02T13:22:23.039267+0000","last_undegraded":"2024-04-02T13:22:23.039267+0000","last_fullsized":"2024-04-02T13:22:23.039267+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:18:33.463457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.024631+0000","last_change":"2024-04-02T13:22:22.019009+0000","last_active":"2024-04-02T13:22:23.024631+0000","last_peered":"2024-04-02T13:22:23.024631+0000","last_clean":"2024-04-02T13:22:23.024631+0000","last_became_active":"2024-04-02T13:22:22.018749+0000","last_became_peered":"2024-04-02T13:22:22.018749+0000","last_unstale":"2024-04-02T13:22:23.024631+0000","last_undegraded":"2024-04-02T13:22:23.024631+0000","last_fullsized":"2024-04-02T13:22:23.024631+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:58.540451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045303+0000","last_change":"2024-04-02T13:22:23.038948+0000","last_active":"2024-04-02T13:22:23.045303+0000","last_peered":"2024-04-02T13:22:23.045303+0000","last_clean":"2024-04-02T13:22:23.045303+0000","last_became_active":"2024-04-02T13:22:23.038672+0000","last_became_peered":"2024-04-02T13:22:23.038672+0000","last_unstale":"2024-04-02T13:22:23.045303+0000","last_undegraded":"2024-04-02T13:22:23.045303+0000","last_fullsized":"2024-04-02T13:22:23.045303+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:57:23.743511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.025479+0000","last_change":"2024-04-02T13:22:22.022762+0000","last_active":"2024-04-02T13:22:23.025479+0000","last_peered":"2024-04-02T13:22:23.025479+0000","last_clean":"2024-04-02T13:22:23.025479+0000","last_became_active":"2024-04-02T13:22:22.022473+0000","last_became_peered":"2024-04-02T13:22:22.022473+0000","last_unstale":"2024-04-02T13:22:23.025479+0000","last_undegraded":"2024-04-02T13:22:23.025479+0000","last_fullsized":"2024-04-02T13:22:23.025479+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:29:41.604642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047857+0000","last_change":"2024-04-02T13:22:23.040149+0000","last_active":"2024-04-02T13:22:23.047857+0000","last_peered":"2024-04-02T13:22:23.047857+0000","last_clean":"2024-04-02T13:22:23.047857+0000","last_became_active":"2024-04-02T13:22:23.039874+0000","last_became_peered":"2024-04-02T13:22:23.039874+0000","last_unstale":"2024-04-02T13:22:23.047857+0000","last_undegraded":"2024-04-02T13:22:23.047857+0000","last_fullsized":"2024-04-02T13:22:23.047857+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:51.781413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.027835+0000","last_change":"2024-04-02T13:22:22.022345+0000","last_active":"2024-04-02T13:22:23.027835+0000","last_peered":"2024-04-02T13:22:23.027835+0000","last_clean":"2024-04-02T13:22:23.027835+0000","last_became_active":"2024-04-02T13:22:22.022093+0000","last_became_peered":"2024-04-02T13:22:22.022093+0000","last_unstale":"2024-04-02T13:22:23.027835+0000","last_undegraded":"2024-04-02T13:22:23.027835+0000","last_fullsized":"2024-04-02T13:22:23.027835+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:49:32.982039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039757+0000","last_change":"2024-04-02T13:22:23.036708+0000","last_active":"2024-04-02T13:22:23.039757+0000","last_peered":"2024-04-02T13:22:23.039757+0000","last_clean":"2024-04-02T13:22:23.039757+0000","last_became_active":"2024-04-02T13:22:23.036429+0000","last_became_peered":"2024-04-02T13:22:23.036429+0000","last_unstale":"2024-04-02T13:22:23.039757+0000","last_undegraded":"2024-04-02T13:22:23.039757+0000","last_fullsized":"2024-04-02T13:22:23.039757+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:18:05.259057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.031614+0000","last_change":"2024-04-02T13:22:22.028543+0000","last_active":"2024-04-02T13:22:23.031614+0000","last_peered":"2024-04-02T13:22:23.031614+0000","last_clean":"2024-04-02T13:22:23.031614+0000","last_became_active":"2024-04-02T13:22:22.028283+0000","last_became_peered":"2024-04-02T13:22:22.028283+0000","last_unstale":"2024-04-02T13:22:23.031614+0000","last_undegraded":"2024-04-02T13:22:23.031614+0000","last_fullsized":"2024-04-02T13:22:23.031614+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:47.876854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":228,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.025805+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:22:23.025805+0000","last_peered":"2024-04-02T13:22:23.025805+0000","last_clean":"2024-04-02T13:22:23.025805+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:22:23.025805+0000","last_undegraded":"2024-04-02T13:22:23.025805+0000","last_fullsized":"2024-04-02T13:22:23.025805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28200,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343640,"statfs":{"total":96636764160,"available":96607887360,"internally_reserved":0,"allocated":1208320,"data_stored":858945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":269105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":268453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":269035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":270087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":593920,"data_stored":261286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":614400,"data_stored":268613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28180,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343660,"statfs":{"total":96636764160,"available":96607907840,"internally_reserved":0,"allocated":1187840,"data_stored":851654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:22:26.062 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:22:26.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.195+0000 7f0ddb17c640 1 Processor -- start 2024-04-02T13:22:26.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.195+0000 7f0ddb17c640 1 -- start start 2024-04-02T13:22:26.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.196+0000 7f0ddb17c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0dd40d0a30 0x7f0dd406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.196+0000 7f0ddb17c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dd40d0000 con 0x7f0dd40d2040 2024-04-02T13:22:26.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.196+0000 7f0ddb17c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dd406c1d0 con 0x7f0dd40d0e10 2024-04-02T13:22:26.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.196+0000 7f0ddb17c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0dd406cb00 con 0x7f0dd40d0a30 2024-04-02T13:22:26.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.196+0000 7f0dd8f18640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0dd40d0a30 0x7f0dd406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.196+0000 7f0dd9719640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0dd40d2040 0x7f0dd40d24f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:58462/0 (socket says 172.21.15.44:58462) 2024-04-02T13:22:26.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.196+0000 7f0dd9719640 1 -- 172.21.15.44:0/3373900391 learned_addr learned my addr 172.21.15.44:0/3373900391 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:26.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.197+0000 7f0dd8f18640 1 -- 172.21.15.44:0/3373900391 >> v1:172.21.15.69:6789/0 conn(0x7f0dd40d0e10 legacy=0x7f0dd40cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:26.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.197+0000 7f0dd8f18640 1 -- 172.21.15.44:0/3373900391 >> v1:172.21.15.44:6789/0 conn(0x7f0dd40d2040 legacy=0x7f0dd40d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.197+0000 7f0dd37fe640 1 -- 172.21.15.44:0/3373900391 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1913197601 0 0) 0x7f0dd40d0000 con 0x7f0dd40d2040 2024-04-02T13:22:26.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.197+0000 7f0dd8f18640 1 -- 172.21.15.44:0/3373900391 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd406c4b0 con 0x7f0dd40d0a30 2024-04-02T13:22:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.197+0000 7f0dd8f18640 1 --2- 172.21.15.44:0/3373900391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0dd40d0a30 0x7f0dd406cf90 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009d90 tx=0x7f0dcc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b6c6d2f094cd6f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.197+0000 7f0dd37fe640 1 -- 172.21.15.44:0/3373900391 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc039070 con 0x7f0dd40d0a30 2024-04-02T13:22:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.198+0000 7f0dd37fe640 1 -- 172.21.15.44:0/3373900391 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dcc02cbd0 con 0x7f0dd40d0a30 2024-04-02T13:22:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.198+0000 7f0dd37fe640 1 -- 172.21.15.44:0/3373900391 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc034440 con 0x7f0dd40d0a30 2024-04-02T13:22:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.198+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3373900391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0dd40d0a30 msgr2=0x7f0dd406cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.198+0000 7f0ddb17c640 1 --2- 172.21.15.44:0/3373900391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0dd40d0a30 0x7f0dd406cf90 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009d90 tx=0x7f0dcc02be20 comp rx=0 tx=0).stop 2024-04-02T13:22:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.198+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3373900391 shutdown_connections 2024-04-02T13:22:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.198+0000 7f0ddb17c640 1 --2- 172.21.15.44:0/3373900391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0dd40d0a30 0x7f0dd406cf90 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.198+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3373900391 >> 172.21.15.44:0/3373900391 conn(0x7f0dd409d470 msgr2=0x7f0dd40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.198+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3373900391 shutdown_connections 2024-04-02T13:22:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.198+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3373900391 wait complete. 2024-04-02T13:22:26.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.199+0000 7f0ddb17c640 1 Processor -- start 2024-04-02T13:22:26.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.199+0000 7f0ddb17c640 1 -- start start 2024-04-02T13:22:26.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.200+0000 7f0ddb17c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd40d0a30 0x7f0dd40b90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.200+0000 7f0ddb17c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0dd40d0e10 0x7f0dd40ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.200+0000 7f0ddb17c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dd40d2040 0x7f0dd40b6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.200+0000 7f0ddb17c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd40d2c00 con 0x7f0dd40d0e10 2024-04-02T13:22:26.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.200+0000 7f0ddb17c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd40d1270 con 0x7f0dd40d2040 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.200+0000 7f0ddb17c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0dd415e6f0 con 0x7f0dd40d0a30 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.200+0000 7f0dd3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0dd40d0e10 0x7f0dd40ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.200+0000 7f0dd3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0dd40d0e10 0x7f0dd40ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43524/0 (socket says 172.21.15.44:43524) 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.200+0000 7f0dd3fff640 1 -- 172.21.15.44:0/3712328399 learned_addr learned my addr 172.21.15.44:0/3712328399 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.200+0000 7f0dd9719640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dd40d2040 0x7f0dd40b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.200+0000 7f0dd8f18640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd40d0a30 0x7f0dd40b90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.201+0000 7f0dd3fff640 1 -- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd40d0a30 msgr2=0x7f0dd40b90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.201+0000 7f0dd3fff640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd40d0a30 0x7f0dd40b90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.201+0000 7f0dd3fff640 1 -- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dd40d2040 msgr2=0x7f0dd40b6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.201+0000 7f0dd3fff640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dd40d2040 0x7f0dd40b6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.201+0000 7f0dd3fff640 1 -- 172.21.15.44:0/3712328399 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd40caac0 con 0x7f0dd40d0e10 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.201+0000 7f0dd8f18640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd40d0a30 0x7f0dd40b90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.201+0000 7f0dd3fff640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0dd40d0e10 0x7f0dd40ca580 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800b3c0 tx=0x7f0dc800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:26.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.201+0000 7f0dd17fa640 1 -- 172.21.15.44:0/3712328399 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8011830 con 0x7f0dd40d0e10 2024-04-02T13:22:26.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.201+0000 7f0dd17fa640 1 -- 172.21.15.44:0/3712328399 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dc8016070 con 0x7f0dd40d0e10 2024-04-02T13:22:26.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.201+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3712328399 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dd40b4ff0 con 0x7f0dd40d0e10 2024-04-02T13:22:26.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.201+0000 7f0dd17fa640 1 -- 172.21.15.44:0/3712328399 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8011c30 con 0x7f0dd40d0e10 2024-04-02T13:22:26.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.201+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3712328399 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dd40b55b0 con 0x7f0dd40d0e10 2024-04-02T13:22:26.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.202+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3712328399 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dd40684f0 con 0x7f0dd40d0e10 2024-04-02T13:22:26.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.203+0000 7f0dd17fa640 1 -- 172.21.15.44:0/3712328399 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0dc801b460 con 0x7f0dd40d0e10 2024-04-02T13:22:26.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.203+0000 7f0dd17fa640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0dc006c8f0 0x7f0dc006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.203+0000 7f0dd17fa640 1 -- 172.21.15.44:0/3712328399 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f0dc808b8e0 con 0x7f0dd40d0e10 2024-04-02T13:22:26.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.203+0000 7f0dd8f18640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0dc006c8f0 0x7f0dc006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.204+0000 7f0dd8f18640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0dc006c8f0 0x7f0dc006edb0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc036000 tx=0x7f0dcc030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:26.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.208+0000 7f0dd17fa640 1 -- 172.21.15.44:0/3712328399 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0dd40caac0 con 0x7f0dd40d0e10 2024-04-02T13:22:26.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.389+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3712328399 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0dd40b65d0 con 0x7f0dc006c8f0 2024-04-02T13:22:26.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.395+0000 7f0dd17fa640 1 -- 172.21.15.44:0/3712328399 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251891 (secure 0 0 0) 0x7f0dd40b65d0 con 0x7f0dc006c8f0 2024-04-02T13:22:26.400 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:26.401 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:22:26.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.398+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0dc006c8f0 msgr2=0x7f0dc006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.398+0000 7f0ddb17c640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0dc006c8f0 0x7f0dc006edb0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc036000 tx=0x7f0dcc030000 comp rx=0 tx=0).stop 2024-04-02T13:22:26.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.398+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3712328399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0dd40d0e10 msgr2=0x7f0dd40ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.398+0000 7f0ddb17c640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0dd40d0e10 0x7f0dd40ca580 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800b3c0 tx=0x7f0dc800b880 comp rx=0 tx=0).stop 2024-04-02T13:22:26.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.398+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3712328399 shutdown_connections 2024-04-02T13:22:26.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.398+0000 7f0ddb17c640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0dc006c8f0 0x7f0dc006edb0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.398+0000 7f0ddb17c640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dd40d2040 0x7f0dd40b6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.398+0000 7f0ddb17c640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0dd40d0e10 0x7f0dd40ca580 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.398+0000 7f0ddb17c640 1 --2- 172.21.15.44:0/3712328399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd40d0a30 0x7f0dd40b90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.398+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3712328399 >> 172.21.15.44:0/3712328399 conn(0x7f0dd409d470 msgr2=0x7f0dd415e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:26.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.398+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3712328399 shutdown_connections 2024-04-02T13:22:26.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.398+0000 7f0ddb17c640 1 -- 172.21.15.44:0/3712328399 wait complete. 2024-04-02T13:22:26.410 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-04-02T13:22:24.902312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222100,"kb_used_data":5916,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752620,"statfs":{"total":773094113280,"available":772866682880,"internally_reserved":0,"allocated":6057984,"data_stored":3317178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000993"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.044805+0000","last_change":"2024-04-02T13:22:23.038062+0000","last_active":"2024-04-02T13:22:23.044805+0000","last_peered":"2024-04-02T13:22:23.044805+0000","last_clean":"2024-04-02T13:22:23.044805+0000","last_became_active":"2024-04-02T13:22:23.037713+0000","last_became_peered":"2024-04-02T13:22:23.037713+0000","last_unstale":"2024-04-02T13:22:23.044805+0000","last_undegraded":"2024-04-02T13:22:23.044805+0000","last_fullsized":"2024-04-02T13:22:23.044805+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:17.264918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.041501+0000","last_change":"2024-04-02T13:22:23.039918+0000","last_active":"2024-04-02T13:22:23.041501+0000","last_peered":"2024-04-02T13:22:23.041501+0000","last_clean":"2024-04-02T13:22:23.041501+0000","last_became_active":"2024-04-02T13:22:23.038737+0000","last_became_peered":"2024-04-02T13:22:23.038737+0000","last_unstale":"2024-04-02T13:22:23.041501+0000","last_undegraded":"2024-04-02T13:22:23.041501+0000","last_fullsized":"2024-04-02T13:22:23.041501+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:35.277058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045422+0000","last_change":"2024-04-02T13:22:23.039151+0000","last_active":"2024-04-02T13:22:23.045422+0000","last_peered":"2024-04-02T13:22:23.045422+0000","last_clean":"2024-04-02T13:22:23.045422+0000","last_became_active":"2024-04-02T13:22:23.038711+0000","last_became_peered":"2024-04-02T13:22:23.038711+0000","last_unstale":"2024-04-02T13:22:23.045422+0000","last_undegraded":"2024-04-02T13:22:23.045422+0000","last_fullsized":"2024-04-02T13:22:23.045422+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:38.210075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.033592+0000","last_change":"2024-04-02T13:22:23.030782+0000","last_active":"2024-04-02T13:22:23.033592+0000","last_peered":"2024-04-02T13:22:23.033592+0000","last_clean":"2024-04-02T13:22:23.033592+0000","last_became_active":"2024-04-02T13:22:23.030502+0000","last_became_peered":"2024-04-02T13:22:23.030502+0000","last_unstale":"2024-04-02T13:22:23.033592+0000","last_undegraded":"2024-04-02T13:22:23.033592+0000","last_fullsized":"2024-04-02T13:22:23.033592+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:18:33.463457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.041602+0000","last_change":"2024-04-02T13:22:23.037131+0000","last_active":"2024-04-02T13:22:23.041602+0000","last_peered":"2024-04-02T13:22:23.041602+0000","last_clean":"2024-04-02T13:22:23.041602+0000","last_became_active":"2024-04-02T13:22:23.036685+0000","last_became_peered":"2024-04-02T13:22:23.036685+0000","last_unstale":"2024-04-02T13:22:23.041602+0000","last_undegraded":"2024-04-02T13:22:23.041602+0000","last_fullsized":"2024-04-02T13:22:23.041602+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:48.881381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.044840+0000","last_change":"2024-04-02T13:22:23.036926+0000","last_active":"2024-04-02T13:22:23.044840+0000","last_peered":"2024-04-02T13:22:23.044840+0000","last_clean":"2024-04-02T13:22:23.044840+0000","last_became_active":"2024-04-02T13:22:23.036606+0000","last_became_peered":"2024-04-02T13:22:23.036606+0000","last_unstale":"2024-04-02T13:22:23.044840+0000","last_undegraded":"2024-04-02T13:22:23.044840+0000","last_fullsized":"2024-04-02T13:22:23.044840+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:59:33.138858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045393+0000","last_change":"2024-04-02T13:22:23.039377+0000","last_active":"2024-04-02T13:22:23.045393+0000","last_peered":"2024-04-02T13:22:23.045393+0000","last_clean":"2024-04-02T13:22:23.045393+0000","last_became_active":"2024-04-02T13:22:23.039110+0000","last_became_peered":"2024-04-02T13:22:23.039110+0000","last_unstale":"2024-04-02T13:22:23.045393+0000","last_undegraded":"2024-04-02T13:22:23.045393+0000","last_fullsized":"2024-04-02T13:22:23.045393+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:54.364791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.050308+0000","last_change":"2024-04-02T13:22:23.046347+0000","last_active":"2024-04-02T13:22:23.050308+0000","last_peered":"2024-04-02T13:22:23.050308+0000","last_clean":"2024-04-02T13:22:23.050308+0000","last_became_active":"2024-04-02T13:22:23.046061+0000","last_became_peered":"2024-04-02T13:22:23.046061+0000","last_unstale":"2024-04-02T13:22:23.050308+0000","last_undegraded":"2024-04-02T13:22:23.050308+0000","last_fullsized":"2024-04-02T13:22:23.050308+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:49:32.798661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.050369+0000","last_change":"2024-04-02T13:22:23.042203+0000","last_active":"2024-04-02T13:22:23.050369+0000","last_peered":"2024-04-02T13:22:23.050369+0000","last_clean":"2024-04-02T13:22:23.050369+0000","last_became_active":"2024-04-02T13:22:23.041607+0000","last_became_peered":"2024-04-02T13:22:23.041607+0000","last_unstale":"2024-04-02T13:22:23.050369+0000","last_undegraded":"2024-04-02T13:22:23.050369+0000","last_fullsized":"2024-04-02T13:22:23.050369+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:56:23.553708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.044597+0000","last_change":"2024-04-02T13:22:23.034698+0000","last_active":"2024-04-02T13:22:23.044597+0000","last_peered":"2024-04-02T13:22:23.044597+0000","last_clean":"2024-04-02T13:22:23.044597+0000","last_became_active":"2024-04-02T13:22:23.034323+0000","last_became_peered":"2024-04-02T13:22:23.034323+0000","last_unstale":"2024-04-02T13:22:23.044597+0000","last_undegraded":"2024-04-02T13:22:23.044597+0000","last_fullsized":"2024-04-02T13:22:23.044597+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:18:05.259057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.043118+0000","last_change":"2024-04-02T13:22:23.035966+0000","last_active":"2024-04-02T13:22:23.043118+0000","last_peered":"2024-04-02T13:22:23.043118+0000","last_clean":"2024-04-02T13:22:23.043118+0000","last_became_active":"2024-04-02T13:22:23.035715+0000","last_became_peered":"2024-04-02T13:22:23.035715+0000","last_unstale":"2024-04-02T13:22:23.043118+0000","last_undegraded":"2024-04-02T13:22:23.043118+0000","last_fullsized":"2024-04-02T13:22:23.043118+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:31:21.359318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.050317+0000","last_change":"2024-04-02T13:22:23.042316+0000","last_active":"2024-04-02T13:22:23.050317+0000","last_peered":"2024-04-02T13:22:23.050317+0000","last_clean":"2024-04-02T13:22:23.050317+0000","last_became_active":"2024-04-02T13:22:23.042013+0000","last_became_peered":"2024-04-02T13:22:23.042013+0000","last_unstale":"2024-04-02T13:22:23.050317+0000","last_undegraded":"2024-04-02T13:22:23.050317+0000","last_fullsized":"2024-04-02T13:22:23.050317+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:22:35.034740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045513+0000","last_change":"2024-04-02T13:22:23.035938+0000","last_active":"2024-04-02T13:22:23.045513+0000","last_peered":"2024-04-02T13:22:23.045513+0000","last_clean":"2024-04-02T13:22:23.045513+0000","last_became_active":"2024-04-02T13:22:23.034813+0000","last_became_peered":"2024-04-02T13:22:23.034813+0000","last_unstale":"2024-04-02T13:22:23.045513+0000","last_undegraded":"2024-04-02T13:22:23.045513+0000","last_fullsized":"2024-04-02T13:22:23.045513+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:53.950028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045199+0000","last_change":"2024-04-02T13:22:23.038717+0000","last_active":"2024-04-02T13:22:23.045199+0000","last_peered":"2024-04-02T13:22:23.045199+0000","last_clean":"2024-04-02T13:22:23.045199+0000","last_became_active":"2024-04-02T13:22:23.038424+0000","last_became_peered":"2024-04-02T13:22:23.038424+0000","last_unstale":"2024-04-02T13:22:23.045199+0000","last_undegraded":"2024-04-02T13:22:23.045199+0000","last_fullsized":"2024-04-02T13:22:23.045199+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:30:33.169862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042349+0000","last_change":"2024-04-02T13:22:23.040984+0000","last_active":"2024-04-02T13:22:23.042349+0000","last_peered":"2024-04-02T13:22:23.042349+0000","last_clean":"2024-04-02T13:22:23.042349+0000","last_became_active":"2024-04-02T13:22:23.040645+0000","last_became_peered":"2024-04-02T13:22:23.040645+0000","last_unstale":"2024-04-02T13:22:23.042349+0000","last_undegraded":"2024-04-02T13:22:23.042349+0000","last_fullsized":"2024-04-02T13:22:23.042349+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:17:04.245119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.044750+0000","last_change":"2024-04-02T13:22:23.035200+0000","last_active":"2024-04-02T13:22:23.044750+0000","last_peered":"2024-04-02T13:22:23.044750+0000","last_clean":"2024-04-02T13:22:23.044750+0000","last_became_active":"2024-04-02T13:22:23.034896+0000","last_became_peered":"2024-04-02T13:22:23.034896+0000","last_unstale":"2024-04-02T13:22:23.044750+0000","last_undegraded":"2024-04-02T13:22:23.044750+0000","last_fullsized":"2024-04-02T13:22:23.044750+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:38:50.523334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047621+0000","last_change":"2024-04-02T13:22:23.043885+0000","last_active":"2024-04-02T13:22:23.047621+0000","last_peered":"2024-04-02T13:22:23.047621+0000","last_clean":"2024-04-02T13:22:23.047621+0000","last_became_active":"2024-04-02T13:22:23.043378+0000","last_became_peered":"2024-04-02T13:22:23.043378+0000","last_unstale":"2024-04-02T13:22:23.047621+0000","last_undegraded":"2024-04-02T13:22:23.047621+0000","last_fullsized":"2024-04-02T13:22:23.047621+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:41:22.513266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.041522+0000","last_change":"2024-04-02T13:22:23.040144+0000","last_active":"2024-04-02T13:22:23.041522+0000","last_peered":"2024-04-02T13:22:23.041522+0000","last_clean":"2024-04-02T13:22:23.041522+0000","last_became_active":"2024-04-02T13:22:23.039298+0000","last_became_peered":"2024-04-02T13:22:23.039298+0000","last_unstale":"2024-04-02T13:22:23.041522+0000","last_undegraded":"2024-04-02T13:22:23.041522+0000","last_fullsized":"2024-04-02T13:22:23.041522+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:10:58.898565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042331+0000","last_change":"2024-04-02T13:22:23.037200+0000","last_active":"2024-04-02T13:22:23.042331+0000","last_peered":"2024-04-02T13:22:23.042331+0000","last_clean":"2024-04-02T13:22:23.042331+0000","last_became_active":"2024-04-02T13:22:23.036883+0000","last_became_peered":"2024-04-02T13:22:23.036883+0000","last_unstale":"2024-04-02T13:22:23.042331+0000","last_undegraded":"2024-04-02T13:22:23.042331+0000","last_fullsized":"2024-04-02T13:22:23.042331+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:17.264918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042484+0000","last_change":"2024-04-02T13:22:23.040208+0000","last_active":"2024-04-02T13:22:23.042484+0000","last_peered":"2024-04-02T13:22:23.042484+0000","last_clean":"2024-04-02T13:22:23.042484+0000","last_became_active":"2024-04-02T13:22:23.039883+0000","last_became_peered":"2024-04-02T13:22:23.039883+0000","last_unstale":"2024-04-02T13:22:23.042484+0000","last_undegraded":"2024-04-02T13:22:23.042484+0000","last_fullsized":"2024-04-02T13:22:23.042484+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:17.264918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039730+0000","last_change":"2024-04-02T13:22:23.036840+0000","last_active":"2024-04-02T13:22:23.039730+0000","last_peered":"2024-04-02T13:22:23.039730+0000","last_clean":"2024-04-02T13:22:23.039730+0000","last_became_active":"2024-04-02T13:22:23.036568+0000","last_became_peered":"2024-04-02T13:22:23.036568+0000","last_unstale":"2024-04-02T13:22:23.039730+0000","last_undegraded":"2024-04-02T13:22:23.039730+0000","last_fullsized":"2024-04-02T13:22:23.039730+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:38:50.523334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047658+0000","last_change":"2024-04-02T13:22:23.041137+0000","last_active":"2024-04-02T13:22:23.047658+0000","last_peered":"2024-04-02T13:22:23.047658+0000","last_clean":"2024-04-02T13:22:23.047658+0000","last_became_active":"2024-04-02T13:22:23.040811+0000","last_became_peered":"2024-04-02T13:22:23.040811+0000","last_unstale":"2024-04-02T13:22:23.047658+0000","last_undegraded":"2024-04-02T13:22:23.047658+0000","last_fullsized":"2024-04-02T13:22:23.047658+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:04.281077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042889+0000","last_change":"2024-04-02T13:22:23.035328+0000","last_active":"2024-04-02T13:22:23.042889+0000","last_peered":"2024-04-02T13:22:23.042889+0000","last_clean":"2024-04-02T13:22:23.042889+0000","last_became_active":"2024-04-02T13:22:23.035041+0000","last_became_peered":"2024-04-02T13:22:23.035041+0000","last_unstale":"2024-04-02T13:22:23.042889+0000","last_undegraded":"2024-04-02T13:22:23.042889+0000","last_fullsized":"2024-04-02T13:22:23.042889+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:58:34.769523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045734+0000","last_change":"2024-04-02T13:22:23.040898+0000","last_active":"2024-04-02T13:22:23.045734+0000","last_peered":"2024-04-02T13:22:23.045734+0000","last_clean":"2024-04-02T13:22:23.045734+0000","last_became_active":"2024-04-02T13:22:23.040612+0000","last_became_peered":"2024-04-02T13:22:23.040612+0000","last_unstale":"2024-04-02T13:22:23.045734+0000","last_undegraded":"2024-04-02T13:22:23.045734+0000","last_fullsized":"2024-04-02T13:22:23.045734+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:10:23.686213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047648+0000","last_change":"2024-04-02T13:22:23.044186+0000","last_active":"2024-04-02T13:22:23.047648+0000","last_peered":"2024-04-02T13:22:23.047648+0000","last_clean":"2024-04-02T13:22:23.047648+0000","last_became_active":"2024-04-02T13:22:23.043772+0000","last_became_peered":"2024-04-02T13:22:23.043772+0000","last_unstale":"2024-04-02T13:22:23.047648+0000","last_undegraded":"2024-04-02T13:22:23.047648+0000","last_fullsized":"2024-04-02T13:22:23.047648+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:59.544978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042052+0000","last_change":"2024-04-02T13:22:23.036634+0000","last_active":"2024-04-02T13:22:23.042052+0000","last_peered":"2024-04-02T13:22:23.042052+0000","last_clean":"2024-04-02T13:22:23.042052+0000","last_became_active":"2024-04-02T13:22:23.036205+0000","last_became_peered":"2024-04-02T13:22:23.036205+0000","last_unstale":"2024-04-02T13:22:23.042052+0000","last_undegraded":"2024-04-02T13:22:23.042052+0000","last_fullsized":"2024-04-02T13:22:23.042052+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:37:25.803096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045050+0000","last_change":"2024-04-02T13:22:23.040147+0000","last_active":"2024-04-02T13:22:23.045050+0000","last_peered":"2024-04-02T13:22:23.045050+0000","last_clean":"2024-04-02T13:22:23.045050+0000","last_became_active":"2024-04-02T13:22:23.039316+0000","last_became_peered":"2024-04-02T13:22:23.039316+0000","last_unstale":"2024-04-02T13:22:23.045050+0000","last_undegraded":"2024-04-02T13:22:23.045050+0000","last_fullsized":"2024-04-02T13:22:23.045050+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:38.210075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042462+0000","last_change":"2024-04-02T13:22:23.040164+0000","last_active":"2024-04-02T13:22:23.042462+0000","last_peered":"2024-04-02T13:22:23.042462+0000","last_clean":"2024-04-02T13:22:23.042462+0000","last_became_active":"2024-04-02T13:22:23.039870+0000","last_became_peered":"2024-04-02T13:22:23.039870+0000","last_unstale":"2024-04-02T13:22:23.042462+0000","last_undegraded":"2024-04-02T13:22:23.042462+0000","last_fullsized":"2024-04-02T13:22:23.042462+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:53.950028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.053898+0000","last_change":"2024-04-02T13:22:23.045746+0000","last_active":"2024-04-02T13:22:23.053898+0000","last_peered":"2024-04-02T13:22:23.053898+0000","last_clean":"2024-04-02T13:22:23.053898+0000","last_became_active":"2024-04-02T13:22:23.045455+0000","last_became_peered":"2024-04-02T13:22:23.045455+0000","last_unstale":"2024-04-02T13:22:23.053898+0000","last_undegraded":"2024-04-02T13:22:23.053898+0000","last_fullsized":"2024-04-02T13:22:23.053898+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:35.277058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.035653+0000","last_change":"2024-04-02T13:22:23.033659+0000","last_active":"2024-04-02T13:22:23.035653+0000","last_peered":"2024-04-02T13:22:23.035653+0000","last_clean":"2024-04-02T13:22:23.035653+0000","last_became_active":"2024-04-02T13:22:23.033351+0000","last_became_peered":"2024-04-02T13:22:23.033351+0000","last_unstale":"2024-04-02T13:22:23.035653+0000","last_undegraded":"2024-04-02T13:22:23.035653+0000","last_fullsized":"2024-04-02T13:22:23.035653+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:04.247519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045370+0000","last_change":"2024-04-02T13:22:23.039319+0000","last_active":"2024-04-02T13:22:23.045370+0000","last_peered":"2024-04-02T13:22:23.045370+0000","last_clean":"2024-04-02T13:22:23.045370+0000","last_became_active":"2024-04-02T13:22:23.038317+0000","last_became_peered":"2024-04-02T13:22:23.038317+0000","last_unstale":"2024-04-02T13:22:23.045370+0000","last_undegraded":"2024-04-02T13:22:23.045370+0000","last_fullsized":"2024-04-02T13:22:23.045370+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:55:36.337222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039805+0000","last_change":"2024-04-02T13:22:23.038665+0000","last_active":"2024-04-02T13:22:23.039805+0000","last_peered":"2024-04-02T13:22:23.039805+0000","last_clean":"2024-04-02T13:22:23.039805+0000","last_became_active":"2024-04-02T13:22:23.038394+0000","last_became_peered":"2024-04-02T13:22:23.038394+0000","last_unstale":"2024-04-02T13:22:23.039805+0000","last_undegraded":"2024-04-02T13:22:23.039805+0000","last_fullsized":"2024-04-02T13:22:23.039805+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:51.961849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.037368+0000","last_change":"2024-04-02T13:22:23.035212+0000","last_active":"2024-04-02T13:22:23.037368+0000","last_peered":"2024-04-02T13:22:23.037368+0000","last_clean":"2024-04-02T13:22:23.037368+0000","last_became_active":"2024-04-02T13:22:23.034895+0000","last_became_peered":"2024-04-02T13:22:23.034895+0000","last_unstale":"2024-04-02T13:22:23.037368+0000","last_undegraded":"2024-04-02T13:22:23.037368+0000","last_fullsized":"2024-04-02T13:22:23.037368+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:59.544978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.054821+0000","last_change":"2024-04-02T13:22:22.023171+0000","last_active":"2024-04-02T13:22:23.054821+0000","last_peered":"2024-04-02T13:22:23.054821+0000","last_clean":"2024-04-02T13:22:23.054821+0000","last_became_active":"2024-04-02T13:22:22.022887+0000","last_became_peered":"2024-04-02T13:22:22.022887+0000","last_unstale":"2024-04-02T13:22:23.054821+0000","last_undegraded":"2024-04-02T13:22:23.054821+0000","last_fullsized":"2024-04-02T13:22:23.054821+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:32:17.085706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039365+0000","last_change":"2024-04-02T13:22:23.035768+0000","last_active":"2024-04-02T13:22:23.039365+0000","last_peered":"2024-04-02T13:22:23.039365+0000","last_clean":"2024-04-02T13:22:23.039365+0000","last_became_active":"2024-04-02T13:22:23.035005+0000","last_became_peered":"2024-04-02T13:22:23.035005+0000","last_unstale":"2024-04-02T13:22:23.039365+0000","last_undegraded":"2024-04-02T13:22:23.039365+0000","last_fullsized":"2024-04-02T13:22:23.039365+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:41:22.513266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.056444+0000","last_change":"2024-04-02T13:22:22.022403+0000","last_active":"2024-04-02T13:22:23.056444+0000","last_peered":"2024-04-02T13:22:23.056444+0000","last_clean":"2024-04-02T13:22:23.056444+0000","last_became_active":"2024-04-02T13:22:22.022136+0000","last_became_peered":"2024-04-02T13:22:22.022136+0000","last_unstale":"2024-04-02T13:22:23.056444+0000","last_undegraded":"2024-04-02T13:22:23.056444+0000","last_fullsized":"2024-04-02T13:22:23.056444+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:24:04.737397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042523+0000","last_change":"2024-04-02T13:22:23.040354+0000","last_active":"2024-04-02T13:22:23.042523+0000","last_peered":"2024-04-02T13:22:23.042523+0000","last_clean":"2024-04-02T13:22:23.042523+0000","last_became_active":"2024-04-02T13:22:23.039514+0000","last_became_peered":"2024-04-02T13:22:23.039514+0000","last_unstale":"2024-04-02T13:22:23.042523+0000","last_undegraded":"2024-04-02T13:22:23.042523+0000","last_fullsized":"2024-04-02T13:22:23.042523+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:30:33.169862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.055728+0000","last_change":"2024-04-02T13:22:22.023345+0000","last_active":"2024-04-02T13:22:23.055728+0000","last_peered":"2024-04-02T13:22:23.055728+0000","last_clean":"2024-04-02T13:22:23.055728+0000","last_became_active":"2024-04-02T13:22:22.022938+0000","last_became_peered":"2024-04-02T13:22:22.022938+0000","last_unstale":"2024-04-02T13:22:23.055728+0000","last_undegraded":"2024-04-02T13:22:23.055728+0000","last_fullsized":"2024-04-02T13:22:23.055728+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:29:41.604642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.037748+0000","last_change":"2024-04-02T13:22:23.035616+0000","last_active":"2024-04-02T13:22:23.037748+0000","last_peered":"2024-04-02T13:22:23.037748+0000","last_clean":"2024-04-02T13:22:23.037748+0000","last_became_active":"2024-04-02T13:22:23.035013+0000","last_became_peered":"2024-04-02T13:22:23.035013+0000","last_unstale":"2024-04-02T13:22:23.037748+0000","last_undegraded":"2024-04-02T13:22:23.037748+0000","last_fullsized":"2024-04-02T13:22:23.037748+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:35:42.066923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.058058+0000","last_change":"2024-04-02T13:22:22.023284+0000","last_active":"2024-04-02T13:22:23.058058+0000","last_peered":"2024-04-02T13:22:23.058058+0000","last_clean":"2024-04-02T13:22:23.058058+0000","last_became_active":"2024-04-02T13:22:22.023017+0000","last_became_peered":"2024-04-02T13:22:22.023017+0000","last_unstale":"2024-04-02T13:22:23.058058+0000","last_undegraded":"2024-04-02T13:22:23.058058+0000","last_fullsized":"2024-04-02T13:22:23.058058+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:51.321428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.041938+0000","last_change":"2024-04-02T13:22:23.038949+0000","last_active":"2024-04-02T13:22:23.041938+0000","last_peered":"2024-04-02T13:22:23.041938+0000","last_clean":"2024-04-02T13:22:23.041938+0000","last_became_active":"2024-04-02T13:22:23.038640+0000","last_became_peered":"2024-04-02T13:22:23.038640+0000","last_unstale":"2024-04-02T13:22:23.041938+0000","last_undegraded":"2024-04-02T13:22:23.041938+0000","last_fullsized":"2024-04-02T13:22:23.041938+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:38:50.523334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.060104+0000","last_change":"2024-04-02T13:22:22.031515+0000","last_active":"2024-04-02T13:22:23.060104+0000","last_peered":"2024-04-02T13:22:23.060104+0000","last_clean":"2024-04-02T13:22:23.060104+0000","last_became_active":"2024-04-02T13:22:22.031230+0000","last_became_peered":"2024-04-02T13:22:22.031230+0000","last_unstale":"2024-04-02T13:22:23.060104+0000","last_undegraded":"2024-04-02T13:22:23.060104+0000","last_fullsized":"2024-04-02T13:22:23.060104+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:51.321428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.035753+0000","last_change":"2024-04-02T13:22:23.034001+0000","last_active":"2024-04-02T13:22:23.035753+0000","last_peered":"2024-04-02T13:22:23.035753+0000","last_clean":"2024-04-02T13:22:23.035753+0000","last_became_active":"2024-04-02T13:22:23.033723+0000","last_became_peered":"2024-04-02T13:22:23.033723+0000","last_unstale":"2024-04-02T13:22:23.035753+0000","last_undegraded":"2024-04-02T13:22:23.035753+0000","last_fullsized":"2024-04-02T13:22:23.035753+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:41:22.513266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.053761+0000","last_change":"2024-04-02T13:22:22.019210+0000","last_active":"2024-04-02T13:22:23.053761+0000","last_peered":"2024-04-02T13:22:23.053761+0000","last_clean":"2024-04-02T13:22:23.053761+0000","last_became_active":"2024-04-02T13:22:22.018868+0000","last_became_peered":"2024-04-02T13:22:22.018868+0000","last_unstale":"2024-04-02T13:22:23.053761+0000","last_undegraded":"2024-04-02T13:22:23.053761+0000","last_fullsized":"2024-04-02T13:22:23.053761+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:12.453015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047676+0000","last_change":"2024-04-02T13:22:23.036999+0000","last_active":"2024-04-02T13:22:23.047676+0000","last_peered":"2024-04-02T13:22:23.047676+0000","last_clean":"2024-04-02T13:22:23.047676+0000","last_became_active":"2024-04-02T13:22:23.036602+0000","last_became_peered":"2024-04-02T13:22:23.036602+0000","last_unstale":"2024-04-02T13:22:23.047676+0000","last_undegraded":"2024-04-02T13:22:23.047676+0000","last_fullsized":"2024-04-02T13:22:23.047676+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:56:56.678078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.029459+0000","last_change":"2024-04-02T13:22:22.024170+0000","last_active":"2024-04-02T13:22:23.029459+0000","last_peered":"2024-04-02T13:22:23.029459+0000","last_clean":"2024-04-02T13:22:23.029459+0000","last_became_active":"2024-04-02T13:22:22.023880+0000","last_became_peered":"2024-04-02T13:22:22.023880+0000","last_unstale":"2024-04-02T13:22:23.029459+0000","last_undegraded":"2024-04-02T13:22:23.029459+0000","last_fullsized":"2024-04-02T13:22:23.029459+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:29:41.604642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.040937+0000","last_change":"2024-04-02T13:22:23.039340+0000","last_active":"2024-04-02T13:22:23.040937+0000","last_peered":"2024-04-02T13:22:23.040937+0000","last_clean":"2024-04-02T13:22:23.040937+0000","last_became_active":"2024-04-02T13:22:23.039090+0000","last_became_peered":"2024-04-02T13:22:23.039090+0000","last_unstale":"2024-04-02T13:22:23.040937+0000","last_undegraded":"2024-04-02T13:22:23.040937+0000","last_fullsized":"2024-04-02T13:22:23.040937+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:51.961849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.040333+0000","last_change":"2024-04-02T13:22:22.030883+0000","last_active":"2024-04-02T13:22:23.040333+0000","last_peered":"2024-04-02T13:22:23.040333+0000","last_clean":"2024-04-02T13:22:23.040333+0000","last_became_active":"2024-04-02T13:22:22.030643+0000","last_became_peered":"2024-04-02T13:22:22.030643+0000","last_unstale":"2024-04-02T13:22:23.040333+0000","last_undegraded":"2024-04-02T13:22:23.040333+0000","last_fullsized":"2024-04-02T13:22:23.040333+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:25.658945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.036158+0000","last_change":"2024-04-02T13:22:23.031013+0000","last_active":"2024-04-02T13:22:23.036158+0000","last_peered":"2024-04-02T13:22:23.036158+0000","last_clean":"2024-04-02T13:22:23.036158+0000","last_became_active":"2024-04-02T13:22:23.030716+0000","last_became_peered":"2024-04-02T13:22:23.030716+0000","last_unstale":"2024-04-02T13:22:23.036158+0000","last_undegraded":"2024-04-02T13:22:23.036158+0000","last_fullsized":"2024-04-02T13:22:23.036158+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:18:00.483711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.029075+0000","last_change":"2024-04-02T13:22:22.023831+0000","last_active":"2024-04-02T13:22:23.029075+0000","last_peered":"2024-04-02T13:22:23.029075+0000","last_clean":"2024-04-02T13:22:23.029075+0000","last_became_active":"2024-04-02T13:22:22.023496+0000","last_became_peered":"2024-04-02T13:22:22.023496+0000","last_unstale":"2024-04-02T13:22:23.029075+0000","last_undegraded":"2024-04-02T13:22:23.029075+0000","last_fullsized":"2024-04-02T13:22:23.029075+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:49:32.982039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039195+0000","last_change":"2024-04-02T13:22:23.035333+0000","last_active":"2024-04-02T13:22:23.039195+0000","last_peered":"2024-04-02T13:22:23.039195+0000","last_clean":"2024-04-02T13:22:23.039195+0000","last_became_active":"2024-04-02T13:22:23.034614+0000","last_became_peered":"2024-04-02T13:22:23.034614+0000","last_unstale":"2024-04-02T13:22:23.039195+0000","last_undegraded":"2024-04-02T13:22:23.039195+0000","last_fullsized":"2024-04-02T13:22:23.039195+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:10:23.686213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.025408+0000","last_change":"2024-04-02T13:22:22.026761+0000","last_active":"2024-04-02T13:22:23.025408+0000","last_peered":"2024-04-02T13:22:23.025408+0000","last_clean":"2024-04-02T13:22:23.025408+0000","last_became_active":"2024-04-02T13:22:22.026460+0000","last_became_peered":"2024-04-02T13:22:22.026460+0000","last_unstale":"2024-04-02T13:22:23.025408+0000","last_undegraded":"2024-04-02T13:22:23.025408+0000","last_fullsized":"2024-04-02T13:22:23.025408+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:24:04.737397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045792+0000","last_change":"2024-04-02T13:22:23.041432+0000","last_active":"2024-04-02T13:22:23.045792+0000","last_peered":"2024-04-02T13:22:23.045792+0000","last_clean":"2024-04-02T13:22:23.045792+0000","last_became_active":"2024-04-02T13:22:23.041174+0000","last_became_peered":"2024-04-02T13:22:23.041174+0000","last_unstale":"2024-04-02T13:22:23.045792+0000","last_undegraded":"2024-04-02T13:22:23.045792+0000","last_fullsized":"2024-04-02T13:22:23.045792+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:18:33.463457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.057937+0000","last_change":"2024-04-02T13:22:22.027465+0000","last_active":"2024-04-02T13:22:23.057937+0000","last_peered":"2024-04-02T13:22:23.057937+0000","last_clean":"2024-04-02T13:22:23.057937+0000","last_became_active":"2024-04-02T13:22:22.027213+0000","last_became_peered":"2024-04-02T13:22:22.027213+0000","last_unstale":"2024-04-02T13:22:23.057937+0000","last_undegraded":"2024-04-02T13:22:23.057937+0000","last_fullsized":"2024-04-02T13:22:23.057937+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:25.658945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042405+0000","last_change":"2024-04-02T13:22:23.041462+0000","last_active":"2024-04-02T13:22:23.042405+0000","last_peered":"2024-04-02T13:22:23.042405+0000","last_clean":"2024-04-02T13:22:23.042405+0000","last_became_active":"2024-04-02T13:22:23.041190+0000","last_became_peered":"2024-04-02T13:22:23.041190+0000","last_unstale":"2024-04-02T13:22:23.042405+0000","last_undegraded":"2024-04-02T13:22:23.042405+0000","last_fullsized":"2024-04-02T13:22:23.042405+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:53.950028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":18,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.061790+0000","last_change":"2024-04-02T13:22:22.025433+0000","last_active":"2024-04-02T13:22:23.061790+0000","last_peered":"2024-04-02T13:22:23.061790+0000","last_clean":"2024-04-02T13:22:23.061790+0000","last_became_active":"2024-04-02T13:22:22.025190+0000","last_became_peered":"2024-04-02T13:22:22.025190+0000","last_unstale":"2024-04-02T13:22:23.061790+0000","last_undegraded":"2024-04-02T13:22:23.061790+0000","last_fullsized":"2024-04-02T13:22:23.061790+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:10:57.894038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039768+0000","last_change":"2024-04-02T13:22:23.035548+0000","last_active":"2024-04-02T13:22:23.039768+0000","last_peered":"2024-04-02T13:22:23.039768+0000","last_clean":"2024-04-02T13:22:23.039768+0000","last_became_active":"2024-04-02T13:22:23.035057+0000","last_became_peered":"2024-04-02T13:22:23.035057+0000","last_unstale":"2024-04-02T13:22:23.039768+0000","last_undegraded":"2024-04-02T13:22:23.039768+0000","last_fullsized":"2024-04-02T13:22:23.039768+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:04.247519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.058099+0000","last_change":"2024-04-02T13:22:22.030861+0000","last_active":"2024-04-02T13:22:23.058099+0000","last_peered":"2024-04-02T13:22:23.058099+0000","last_clean":"2024-04-02T13:22:23.058099+0000","last_became_active":"2024-04-02T13:22:22.030416+0000","last_became_peered":"2024-04-02T13:22:22.030416+0000","last_unstale":"2024-04-02T13:22:23.058099+0000","last_undegraded":"2024-04-02T13:22:23.058099+0000","last_fullsized":"2024-04-02T13:22:23.058099+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:29:35.357053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.037365+0000","last_change":"2024-04-02T13:22:23.035360+0000","last_active":"2024-04-02T13:22:23.037365+0000","last_peered":"2024-04-02T13:22:23.037365+0000","last_clean":"2024-04-02T13:22:23.037365+0000","last_became_active":"2024-04-02T13:22:23.035070+0000","last_became_peered":"2024-04-02T13:22:23.035070+0000","last_unstale":"2024-04-02T13:22:23.037365+0000","last_undegraded":"2024-04-02T13:22:23.037365+0000","last_fullsized":"2024-04-02T13:22:23.037365+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:59:33.138858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.031945+0000","last_change":"2024-04-02T13:22:22.032136+0000","last_active":"2024-04-02T13:22:23.031945+0000","last_peered":"2024-04-02T13:22:23.031945+0000","last_clean":"2024-04-02T13:22:23.031945+0000","last_became_active":"2024-04-02T13:22:22.031828+0000","last_became_peered":"2024-04-02T13:22:22.031828+0000","last_unstale":"2024-04-02T13:22:23.031945+0000","last_undegraded":"2024-04-02T13:22:23.031945+0000","last_fullsized":"2024-04-02T13:22:23.031945+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:12.453015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.037740+0000","last_change":"2024-04-02T13:22:23.033476+0000","last_active":"2024-04-02T13:22:23.037740+0000","last_peered":"2024-04-02T13:22:23.037740+0000","last_clean":"2024-04-02T13:22:23.037740+0000","last_became_active":"2024-04-02T13:22:23.033170+0000","last_became_peered":"2024-04-02T13:22:23.033170+0000","last_unstale":"2024-04-02T13:22:23.037740+0000","last_undegraded":"2024-04-02T13:22:23.037740+0000","last_fullsized":"2024-04-02T13:22:23.037740+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:17:04.245119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.027168+0000","last_change":"2024-04-02T13:22:22.024652+0000","last_active":"2024-04-02T13:22:23.027168+0000","last_peered":"2024-04-02T13:22:23.027168+0000","last_clean":"2024-04-02T13:22:23.027168+0000","last_became_active":"2024-04-02T13:22:22.024371+0000","last_became_peered":"2024-04-02T13:22:22.024371+0000","last_unstale":"2024-04-02T13:22:23.027168+0000","last_undegraded":"2024-04-02T13:22:23.027168+0000","last_fullsized":"2024-04-02T13:22:23.027168+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:51.321428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047668+0000","last_change":"2024-04-02T13:22:23.044787+0000","last_active":"2024-04-02T13:22:23.047668+0000","last_peered":"2024-04-02T13:22:23.047668+0000","last_clean":"2024-04-02T13:22:23.047668+0000","last_became_active":"2024-04-02T13:22:23.044295+0000","last_became_peered":"2024-04-02T13:22:23.044295+0000","last_unstale":"2024-04-02T13:22:23.047668+0000","last_undegraded":"2024-04-02T13:22:23.047668+0000","last_fullsized":"2024-04-02T13:22:23.047668+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:51.961849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":20,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.060163+0000","last_change":"2024-04-02T13:22:22.022925+0000","last_active":"2024-04-02T13:22:23.060163+0000","last_peered":"2024-04-02T13:22:23.060163+0000","last_clean":"2024-04-02T13:22:23.060163+0000","last_became_active":"2024-04-02T13:22:22.022662+0000","last_became_peered":"2024-04-02T13:22:22.022662+0000","last_unstale":"2024-04-02T13:22:23.060163+0000","last_undegraded":"2024-04-02T13:22:23.060163+0000","last_fullsized":"2024-04-02T13:22:23.060163+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:34.272531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.037724+0000","last_change":"2024-04-02T13:22:23.035631+0000","last_active":"2024-04-02T13:22:23.037724+0000","last_peered":"2024-04-02T13:22:23.037724+0000","last_clean":"2024-04-02T13:22:23.037724+0000","last_became_active":"2024-04-02T13:22:23.035330+0000","last_became_peered":"2024-04-02T13:22:23.035330+0000","last_unstale":"2024-04-02T13:22:23.037724+0000","last_undegraded":"2024-04-02T13:22:23.037724+0000","last_fullsized":"2024-04-02T13:22:23.037724+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:49:36.074639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.058015+0000","last_change":"2024-04-02T13:22:22.031587+0000","last_active":"2024-04-02T13:22:23.058015+0000","last_peered":"2024-04-02T13:22:23.058015+0000","last_clean":"2024-04-02T13:22:23.058015+0000","last_became_active":"2024-04-02T13:22:22.031325+0000","last_became_peered":"2024-04-02T13:22:22.031325+0000","last_unstale":"2024-04-02T13:22:23.058015+0000","last_undegraded":"2024-04-02T13:22:23.058015+0000","last_fullsized":"2024-04-02T13:22:23.058015+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:18:04.254530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045019+0000","last_change":"2024-04-02T13:22:23.039729+0000","last_active":"2024-04-02T13:22:23.045019+0000","last_peered":"2024-04-02T13:22:23.045019+0000","last_clean":"2024-04-02T13:22:23.045019+0000","last_became_active":"2024-04-02T13:22:23.039281+0000","last_became_peered":"2024-04-02T13:22:23.039281+0000","last_unstale":"2024-04-02T13:22:23.045019+0000","last_undegraded":"2024-04-02T13:22:23.045019+0000","last_fullsized":"2024-04-02T13:22:23.045019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:57:23.743511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.056301+0000","last_change":"2024-04-02T13:22:22.023564+0000","last_active":"2024-04-02T13:22:23.056301+0000","last_peered":"2024-04-02T13:22:23.056301+0000","last_clean":"2024-04-02T13:22:23.056301+0000","last_became_active":"2024-04-02T13:22:22.023273+0000","last_became_peered":"2024-04-02T13:22:22.023273+0000","last_unstale":"2024-04-02T13:22:23.056301+0000","last_undegraded":"2024-04-02T13:22:23.056301+0000","last_fullsized":"2024-04-02T13:22:23.056301+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:52:08.715533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.041702+0000","last_change":"2024-04-02T13:22:23.038304+0000","last_active":"2024-04-02T13:22:23.041702+0000","last_peered":"2024-04-02T13:22:23.041702+0000","last_clean":"2024-04-02T13:22:23.041702+0000","last_became_active":"2024-04-02T13:22:23.038037+0000","last_became_peered":"2024-04-02T13:22:23.038037+0000","last_unstale":"2024-04-02T13:22:23.041702+0000","last_undegraded":"2024-04-02T13:22:23.041702+0000","last_fullsized":"2024-04-02T13:22:23.041702+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:18:05.259057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.055601+0000","last_change":"2024-04-02T13:22:22.021325+0000","last_active":"2024-04-02T13:22:23.055601+0000","last_peered":"2024-04-02T13:22:23.055601+0000","last_clean":"2024-04-02T13:22:23.055601+0000","last_became_active":"2024-04-02T13:22:22.020527+0000","last_became_peered":"2024-04-02T13:22:22.020527+0000","last_unstale":"2024-04-02T13:22:23.055601+0000","last_undegraded":"2024-04-02T13:22:23.055601+0000","last_fullsized":"2024-04-02T13:22:23.055601+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:47.876854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045554+0000","last_change":"2024-04-02T13:22:23.038942+0000","last_active":"2024-04-02T13:22:23.045554+0000","last_peered":"2024-04-02T13:22:23.045554+0000","last_clean":"2024-04-02T13:22:23.045554+0000","last_became_active":"2024-04-02T13:22:23.038606+0000","last_became_peered":"2024-04-02T13:22:23.038606+0000","last_unstale":"2024-04-02T13:22:23.045554+0000","last_undegraded":"2024-04-02T13:22:23.045554+0000","last_fullsized":"2024-04-02T13:22:23.045554+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:30:33.169862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.032246+0000","last_change":"2024-04-02T13:22:22.023901+0000","last_active":"2024-04-02T13:22:23.032246+0000","last_peered":"2024-04-02T13:22:23.032246+0000","last_clean":"2024-04-02T13:22:23.032246+0000","last_became_active":"2024-04-02T13:22:22.023608+0000","last_became_peered":"2024-04-02T13:22:22.023608+0000","last_unstale":"2024-04-02T13:22:23.032246+0000","last_undegraded":"2024-04-02T13:22:23.032246+0000","last_fullsized":"2024-04-02T13:22:23.032246+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:17:59.479184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039013+0000","last_change":"2024-04-02T13:22:23.035009+0000","last_active":"2024-04-02T13:22:23.039013+0000","last_peered":"2024-04-02T13:22:23.039013+0000","last_clean":"2024-04-02T13:22:23.039013+0000","last_became_active":"2024-04-02T13:22:23.034677+0000","last_became_peered":"2024-04-02T13:22:23.034677+0000","last_unstale":"2024-04-02T13:22:23.039013+0000","last_undegraded":"2024-04-02T13:22:23.039013+0000","last_fullsized":"2024-04-02T13:22:23.039013+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:26.663472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.056050+0000","last_change":"2024-04-02T13:22:22.024810+0000","last_active":"2024-04-02T13:22:23.056050+0000","last_peered":"2024-04-02T13:22:23.056050+0000","last_clean":"2024-04-02T13:22:23.056050+0000","last_became_active":"2024-04-02T13:22:22.024398+0000","last_became_peered":"2024-04-02T13:22:22.024398+0000","last_unstale":"2024-04-02T13:22:23.056050+0000","last_undegraded":"2024-04-02T13:22:23.056050+0000","last_fullsized":"2024-04-02T13:22:23.056050+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:12.453015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.042484+0000","last_change":"2024-04-02T13:22:23.040187+0000","last_active":"2024-04-02T13:22:23.042484+0000","last_peered":"2024-04-02T13:22:23.042484+0000","last_clean":"2024-04-02T13:22:23.042484+0000","last_became_active":"2024-04-02T13:22:23.039647+0000","last_became_peered":"2024-04-02T13:22:23.039647+0000","last_unstale":"2024-04-02T13:22:23.042484+0000","last_undegraded":"2024-04-02T13:22:23.042484+0000","last_fullsized":"2024-04-02T13:22:23.042484+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:59:33.138858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.024466+0000","last_change":"2024-04-02T13:22:22.024822+0000","last_active":"2024-04-02T13:22:23.024466+0000","last_peered":"2024-04-02T13:22:23.024466+0000","last_clean":"2024-04-02T13:22:23.024466+0000","last_became_active":"2024-04-02T13:22:22.024542+0000","last_became_peered":"2024-04-02T13:22:22.024542+0000","last_unstale":"2024-04-02T13:22:23.024466+0000","last_undegraded":"2024-04-02T13:22:23.024466+0000","last_fullsized":"2024-04-02T13:22:23.024466+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:41:21.508739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047685+0000","last_change":"2024-04-02T13:22:23.042906+0000","last_active":"2024-04-02T13:22:23.047685+0000","last_peered":"2024-04-02T13:22:23.047685+0000","last_clean":"2024-04-02T13:22:23.047685+0000","last_became_active":"2024-04-02T13:22:23.042644+0000","last_became_peered":"2024-04-02T13:22:23.042644+0000","last_unstale":"2024-04-02T13:22:23.047685+0000","last_undegraded":"2024-04-02T13:22:23.047685+0000","last_fullsized":"2024-04-02T13:22:23.047685+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:04.247519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.033755+0000","last_change":"2024-04-02T13:22:22.028715+0000","last_active":"2024-04-02T13:22:23.033755+0000","last_peered":"2024-04-02T13:22:23.033755+0000","last_clean":"2024-04-02T13:22:23.033755+0000","last_became_active":"2024-04-02T13:22:22.028465+0000","last_became_peered":"2024-04-02T13:22:22.028465+0000","last_unstale":"2024-04-02T13:22:23.033755+0000","last_undegraded":"2024-04-02T13:22:23.033755+0000","last_fullsized":"2024-04-02T13:22:23.033755+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:38:49.518806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039899+0000","last_change":"2024-04-02T13:22:23.032289+0000","last_active":"2024-04-02T13:22:23.039899+0000","last_peered":"2024-04-02T13:22:23.039899+0000","last_clean":"2024-04-02T13:22:23.039899+0000","last_became_active":"2024-04-02T13:22:23.031957+0000","last_became_peered":"2024-04-02T13:22:23.031957+0000","last_unstale":"2024-04-02T13:22:23.039899+0000","last_undegraded":"2024-04-02T13:22:23.039899+0000","last_fullsized":"2024-04-02T13:22:23.039899+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:48.881381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.019829+0000","last_change":"2024-04-02T13:22:22.022886+0000","last_active":"2024-04-02T13:22:23.019829+0000","last_peered":"2024-04-02T13:22:23.019829+0000","last_clean":"2024-04-02T13:22:23.019829+0000","last_became_active":"2024-04-02T13:22:22.022599+0000","last_became_peered":"2024-04-02T13:22:22.022599+0000","last_unstale":"2024-04-02T13:22:23.019829+0000","last_undegraded":"2024-04-02T13:22:23.019829+0000","last_fullsized":"2024-04-02T13:22:23.019829+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:05:47.082744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039783+0000","last_change":"2024-04-02T13:22:23.037812+0000","last_active":"2024-04-02T13:22:23.039783+0000","last_peered":"2024-04-02T13:22:23.039783+0000","last_clean":"2024-04-02T13:22:23.039783+0000","last_became_active":"2024-04-02T13:22:23.037498+0000","last_became_peered":"2024-04-02T13:22:23.037498+0000","last_unstale":"2024-04-02T13:22:23.039783+0000","last_undegraded":"2024-04-02T13:22:23.039783+0000","last_fullsized":"2024-04-02T13:22:23.039783+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:59.544978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.058418+0000","last_change":"2024-04-02T13:22:22.031817+0000","last_active":"2024-04-02T13:22:23.058418+0000","last_peered":"2024-04-02T13:22:23.058418+0000","last_clean":"2024-04-02T13:22:23.058418+0000","last_became_active":"2024-04-02T13:22:22.031562+0000","last_became_peered":"2024-04-02T13:22:22.031562+0000","last_unstale":"2024-04-02T13:22:23.058418+0000","last_undegraded":"2024-04-02T13:22:23.058418+0000","last_fullsized":"2024-04-02T13:22:23.058418+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:24:04.737397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.032641+0000","last_change":"2024-04-02T13:22:23.029419+0000","last_active":"2024-04-02T13:22:23.032641+0000","last_peered":"2024-04-02T13:22:23.032641+0000","last_clean":"2024-04-02T13:22:23.032641+0000","last_became_active":"2024-04-02T13:22:23.029100+0000","last_became_peered":"2024-04-02T13:22:23.029100+0000","last_unstale":"2024-04-02T13:22:23.032641+0000","last_undegraded":"2024-04-02T13:22:23.032641+0000","last_fullsized":"2024-04-02T13:22:23.032641+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:18:00.483711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.029591+0000","last_change":"2024-04-02T13:22:22.026307+0000","last_active":"2024-04-02T13:22:23.029591+0000","last_peered":"2024-04-02T13:22:23.029591+0000","last_clean":"2024-04-02T13:22:23.029591+0000","last_became_active":"2024-04-02T13:22:22.025780+0000","last_became_peered":"2024-04-02T13:22:22.025780+0000","last_unstale":"2024-04-02T13:22:23.029591+0000","last_undegraded":"2024-04-02T13:22:23.029591+0000","last_fullsized":"2024-04-02T13:22:23.029591+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:10:22.681686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.037765+0000","last_change":"2024-04-02T13:22:23.036100+0000","last_active":"2024-04-02T13:22:23.037765+0000","last_peered":"2024-04-02T13:22:23.037765+0000","last_clean":"2024-04-02T13:22:23.037765+0000","last_became_active":"2024-04-02T13:22:23.035392+0000","last_became_peered":"2024-04-02T13:22:23.035392+0000","last_unstale":"2024-04-02T13:22:23.037765+0000","last_undegraded":"2024-04-02T13:22:23.037765+0000","last_fullsized":"2024-04-02T13:22:23.037765+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:27:00.329601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.034353+0000","last_change":"2024-04-02T13:22:22.029527+0000","last_active":"2024-04-02T13:22:23.034353+0000","last_peered":"2024-04-02T13:22:23.034353+0000","last_clean":"2024-04-02T13:22:23.034353+0000","last_became_active":"2024-04-02T13:22:22.029176+0000","last_became_peered":"2024-04-02T13:22:22.029176+0000","last_unstale":"2024-04-02T13:22:23.034353+0000","last_undegraded":"2024-04-02T13:22:23.034353+0000","last_fullsized":"2024-04-02T13:22:23.034353+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:52.945501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045753+0000","last_change":"2024-04-02T13:22:23.039965+0000","last_active":"2024-04-02T13:22:23.045753+0000","last_peered":"2024-04-02T13:22:23.045753+0000","last_clean":"2024-04-02T13:22:23.045753+0000","last_became_active":"2024-04-02T13:22:23.039706+0000","last_became_peered":"2024-04-02T13:22:23.039706+0000","last_unstale":"2024-04-02T13:22:23.045753+0000","last_undegraded":"2024-04-02T13:22:23.045753+0000","last_fullsized":"2024-04-02T13:22:23.045753+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:49:33.986566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.024972+0000","last_change":"2024-04-02T13:22:22.024186+0000","last_active":"2024-04-02T13:22:23.024972+0000","last_peered":"2024-04-02T13:22:23.024972+0000","last_clean":"2024-04-02T13:22:23.024972+0000","last_became_active":"2024-04-02T13:22:22.023418+0000","last_became_peered":"2024-04-02T13:22:22.023418+0000","last_unstale":"2024-04-02T13:22:23.024972+0000","last_undegraded":"2024-04-02T13:22:23.024972+0000","last_fullsized":"2024-04-02T13:22:23.024972+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:50.957322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039267+0000","last_change":"2024-04-02T13:22:23.035490+0000","last_active":"2024-04-02T13:22:23.039267+0000","last_peered":"2024-04-02T13:22:23.039267+0000","last_clean":"2024-04-02T13:22:23.039267+0000","last_became_active":"2024-04-02T13:22:23.035227+0000","last_became_peered":"2024-04-02T13:22:23.035227+0000","last_unstale":"2024-04-02T13:22:23.039267+0000","last_undegraded":"2024-04-02T13:22:23.039267+0000","last_fullsized":"2024-04-02T13:22:23.039267+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:18:33.463457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.024631+0000","last_change":"2024-04-02T13:22:22.019009+0000","last_active":"2024-04-02T13:22:23.024631+0000","last_peered":"2024-04-02T13:22:23.024631+0000","last_clean":"2024-04-02T13:22:23.024631+0000","last_became_active":"2024-04-02T13:22:22.018749+0000","last_became_peered":"2024-04-02T13:22:22.018749+0000","last_unstale":"2024-04-02T13:22:23.024631+0000","last_undegraded":"2024-04-02T13:22:23.024631+0000","last_fullsized":"2024-04-02T13:22:23.024631+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:58.540451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.045303+0000","last_change":"2024-04-02T13:22:23.038948+0000","last_active":"2024-04-02T13:22:23.045303+0000","last_peered":"2024-04-02T13:22:23.045303+0000","last_clean":"2024-04-02T13:22:23.045303+0000","last_became_active":"2024-04-02T13:22:23.038672+0000","last_became_peered":"2024-04-02T13:22:23.038672+0000","last_unstale":"2024-04-02T13:22:23.045303+0000","last_undegraded":"2024-04-02T13:22:23.045303+0000","last_fullsized":"2024-04-02T13:22:23.045303+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:57:23.743511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.025479+0000","last_change":"2024-04-02T13:22:22.022762+0000","last_active":"2024-04-02T13:22:23.025479+0000","last_peered":"2024-04-02T13:22:23.025479+0000","last_clean":"2024-04-02T13:22:23.025479+0000","last_became_active":"2024-04-02T13:22:22.022473+0000","last_became_peered":"2024-04-02T13:22:22.022473+0000","last_unstale":"2024-04-02T13:22:23.025479+0000","last_undegraded":"2024-04-02T13:22:23.025479+0000","last_fullsized":"2024-04-02T13:22:23.025479+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:29:41.604642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.047857+0000","last_change":"2024-04-02T13:22:23.040149+0000","last_active":"2024-04-02T13:22:23.047857+0000","last_peered":"2024-04-02T13:22:23.047857+0000","last_clean":"2024-04-02T13:22:23.047857+0000","last_became_active":"2024-04-02T13:22:23.039874+0000","last_became_peered":"2024-04-02T13:22:23.039874+0000","last_unstale":"2024-04-02T13:22:23.047857+0000","last_undegraded":"2024-04-02T13:22:23.047857+0000","last_fullsized":"2024-04-02T13:22:23.047857+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:51.781413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.027835+0000","last_change":"2024-04-02T13:22:22.022345+0000","last_active":"2024-04-02T13:22:23.027835+0000","last_peered":"2024-04-02T13:22:23.027835+0000","last_clean":"2024-04-02T13:22:23.027835+0000","last_became_active":"2024-04-02T13:22:22.022093+0000","last_became_peered":"2024-04-02T13:22:22.022093+0000","last_unstale":"2024-04-02T13:22:23.027835+0000","last_undegraded":"2024-04-02T13:22:23.027835+0000","last_fullsized":"2024-04-02T13:22:23.027835+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:49:32.982039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.039757+0000","last_change":"2024-04-02T13:22:23.036708+0000","last_active":"2024-04-02T13:22:23.039757+0000","last_peered":"2024-04-02T13:22:23.039757+0000","last_clean":"2024-04-02T13:22:23.039757+0000","last_became_active":"2024-04-02T13:22:23.036429+0000","last_became_peered":"2024-04-02T13:22:23.036429+0000","last_unstale":"2024-04-02T13:22:23.039757+0000","last_undegraded":"2024-04-02T13:22:23.039757+0000","last_fullsized":"2024-04-02T13:22:23.039757+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:22.005730+0000","last_clean_scrub_stamp":"2024-04-02T13:22:22.005730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:18:05.259057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.031614+0000","last_change":"2024-04-02T13:22:22.028543+0000","last_active":"2024-04-02T13:22:23.031614+0000","last_peered":"2024-04-02T13:22:23.031614+0000","last_clean":"2024-04-02T13:22:23.031614+0000","last_became_active":"2024-04-02T13:22:22.028283+0000","last_became_peered":"2024-04-02T13:22:22.028283+0000","last_unstale":"2024-04-02T13:22:23.031614+0000","last_undegraded":"2024-04-02T13:22:23.031614+0000","last_fullsized":"2024-04-02T13:22:23.031614+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:22:21.001203+0000","last_clean_scrub_stamp":"2024-04-02T13:22:21.001203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:47.876854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":228,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-02T13:22:23.025805+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:22:23.025805+0000","last_peered":"2024-04-02T13:22:23.025805+0000","last_clean":"2024-04-02T13:22:23.025805+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:22:23.025805+0000","last_undegraded":"2024-04-02T13:22:23.025805+0000","last_fullsized":"2024-04-02T13:22:23.025805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28200,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343640,"statfs":{"total":96636764160,"available":96607887360,"internally_reserved":0,"allocated":1208320,"data_stored":858945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":269105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":268453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":269035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":270087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":593920,"data_stored":261286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":614400,"data_stored":268613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28180,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343660,"statfs":{"total":96636764160,"available":96607907840,"internally_reserved":0,"allocated":1187840,"data_stored":851654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:22:26.415 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T13:22:26.415 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T13:22:26.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.557+0000 7fbdfd622640 1 Processor -- start 2024-04-02T13:22:26.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.557+0000 7fbdfd622640 1 -- start start 2024-04-02T13:22:26.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.558+0000 7fbdfd622640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbdf815e4d0 0x7fbdf80cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.558+0000 7fbdfd622640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbdf80cfd20 0x7fbdf80d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.558+0000 7fbdfd622640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d1060 0x7fbdf80d1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.558+0000 7fbdfd622640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fbdf80b3a90 con 0x7fbdf80cfd20 2024-04-02T13:22:26.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.558+0000 7fbdfd622640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbdf80b4610 con 0x7fbdf815e4d0 2024-04-02T13:22:26.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.558+0000 7fbdfd622640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbdf815e8b0 con 0x7fbdf80d1060 2024-04-02T13:22:26.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.558+0000 7fbdf67fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbdf80cfd20 0x7fbdf80d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.558+0000 7fbdf67fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbdf80cfd20 0x7fbdf80d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43526/0 (socket says 172.21.15.44:43526) 2024-04-02T13:22:26.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.558+0000 7fbdf67fc640 1 -- 172.21.15.44:0/1763885747 learned_addr learned my addr 172.21.15.44:0/1763885747 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:26.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.559+0000 7fbdf77fe640 1 --2- 172.21.15.44:0/1763885747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d1060 0x7fbdf80d1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.559+0000 7fbdf6ffd640 1 --2- 172.21.15.44:0/1763885747 >> v2:172.21.15.69:3300/0 conn(0x7fbdf815e4d0 0x7fbdf80cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.559+0000 7fbdf67fc640 1 -- 172.21.15.44:0/1763885747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d1060 msgr2=0x7fbdf80d1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:26.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.559+0000 7fbdf67fc640 1 --2- 172.21.15.44:0/1763885747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d1060 0x7fbdf80d1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.559+0000 7fbdf67fc640 1 -- 172.21.15.44:0/1763885747 >> v2:172.21.15.69:3300/0 conn(0x7fbdf815e4d0 msgr2=0x7fbdf80cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:26.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.559+0000 7fbdf67fc640 1 --2- 172.21.15.44:0/1763885747 >> v2:172.21.15.69:3300/0 conn(0x7fbdf815e4d0 0x7fbdf80cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.559+0000 7fbdf67fc640 1 -- 172.21.15.44:0/1763885747 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf80d2fb0 con 0x7fbdf80cfd20 2024-04-02T13:22:26.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.559+0000 7fbdf67fc640 1 --2- 172.21.15.44:0/1763885747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdf80cfd20 0x7fbdf80d0b20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fbdf80b6d00 tx=0x7fbde002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b2121470bc2ae1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:26.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.560+0000 7fbdf5ffb640 1 -- 172.21.15.44:0/1763885747 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0035020 con 0x7fbdf80cfd20 2024-04-02T13:22:26.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.560+0000 7fbdf5ffb640 1 -- 172.21.15.44:0/1763885747 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbde0031870 con 0x7fbdf80cfd20 2024-04-02T13:22:26.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.560+0000 7fbdf5ffb640 1 -- 172.21.15.44:0/1763885747 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0031c40 con 0x7fbdf80cfd20 2024-04-02T13:22:26.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.560+0000 7fbdfd622640 1 -- 172.21.15.44:0/1763885747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdf80cfd20 msgr2=0x7fbdf80d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.560+0000 7fbdfd622640 1 --2- 172.21.15.44:0/1763885747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdf80cfd20 0x7fbdf80d0b20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fbdf80b6d00 tx=0x7fbde002b910 comp rx=0 tx=0).stop 2024-04-02T13:22:26.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.560+0000 7fbdfd622640 1 -- 172.21.15.44:0/1763885747 shutdown_connections 2024-04-02T13:22:26.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.560+0000 7fbdfd622640 1 --2- 172.21.15.44:0/1763885747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d1060 0x7fbdf80d1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.560+0000 7fbdfd622640 1 --2- 172.21.15.44:0/1763885747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdf80cfd20 0x7fbdf80d0b20 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.560+0000 7fbdfd622640 1 --2- 172.21.15.44:0/1763885747 >> v2:172.21.15.69:3300/0 conn(0x7fbdf815e4d0 0x7fbdf80cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.560+0000 7fbdfd622640 1 -- 172.21.15.44:0/1763885747 >> 172.21.15.44:0/1763885747 conn(0x7fbdf809d410 msgr2=0x7fbdf80d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:26.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.560+0000 7fbdfd622640 1 -- 172.21.15.44:0/1763885747 shutdown_connections 2024-04-02T13:22:26.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.560+0000 7fbdfd622640 1 -- 172.21.15.44:0/1763885747 wait complete. 2024-04-02T13:22:26.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.561+0000 7fbdfd622640 1 Processor -- start 2024-04-02T13:22:26.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.561+0000 7fbdfd622640 1 -- start start 2024-04-02T13:22:26.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.562+0000 7fbdfd622640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdf80cfd20 0x7fbdf814f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.562+0000 7fbdfd622640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdf80d1060 0x7fbdf815ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.562+0000 7fbdfd622640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf815e4d0 0x7fbdf80b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.562+0000 7fbdfd622640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf80b42a0 con 0x7fbdf80d1060 2024-04-02T13:22:26.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.562+0000 7fbdfd622640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf80b3fc0 con 0x7fbdf80cfd20 2024-04-02T13:22:26.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.562+0000 7fbdfd622640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbdf80b0d00 con 0x7fbdf815e4d0 2024-04-02T13:22:26.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.562+0000 7fbdf67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdf80d1060 0x7fbdf815ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.562+0000 7fbdf67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdf80d1060 0x7fbdf815ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43540/0 (socket says 172.21.15.44:43540) 2024-04-02T13:22:26.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.562+0000 7fbdf67fc640 1 -- 172.21.15.44:0/3481304304 learned_addr learned my addr 172.21.15.44:0/3481304304 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:26.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdf6ffd640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdf80cfd20 0x7fbdf814f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdf77fe640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf815e4d0 0x7fbdf80b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdf67fc640 1 -- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf815e4d0 msgr2=0x7fbdf80b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdf67fc640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf815e4d0 0x7fbdf80b5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdf67fc640 1 -- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdf80cfd20 msgr2=0x7fbdf814f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdf67fc640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdf80cfd20 0x7fbdf814f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdf67fc640 1 -- 172.21.15.44:0/3481304304 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf80b8070 con 0x7fbdf80d1060 2024-04-02T13:22:26.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdf6ffd640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdf80cfd20 0x7fbdf814f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:26.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdf67fc640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdf80d1060 0x7fbdf815ea30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbde0000c00 tx=0x7fbde00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdef7fe640 1 -- 172.21.15.44:0/3481304304 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0040070 con 0x7fbdf80d1060 2024-04-02T13:22:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdef7fe640 1 -- 172.21.15.44:0/3481304304 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbde003e070 con 0x7fbdf80d1060 2024-04-02T13:22:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdfd622640 1 -- 172.21.15.44:0/3481304304 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf80b8350 con 0x7fbdf80d1060 2024-04-02T13:22:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdef7fe640 1 -- 172.21.15.44:0/3481304304 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0035040 con 0x7fbdf80d1060 2024-04-02T13:22:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.563+0000 7fbdfd622640 1 -- 172.21.15.44:0/3481304304 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf80b8910 con 0x7fbdf80d1060 2024-04-02T13:22:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.564+0000 7fbdfd622640 1 -- 172.21.15.44:0/3481304304 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdf8068c00 con 0x7fbdf80d1060 2024-04-02T13:22:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.564+0000 7fbdef7fe640 1 -- 172.21.15.44:0/3481304304 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbde0038020 con 0x7fbdf80d1060 2024-04-02T13:22:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.565+0000 7fbdef7fe640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbdd806c8f0 0x7fbdd806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.565+0000 7fbdef7fe640 1 -- 172.21.15.44:0/3481304304 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fbde00adc60 con 0x7fbdf80d1060 2024-04-02T13:22:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.565+0000 7fbdf6ffd640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbdd806c8f0 0x7fbdd806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.565+0000 7fbdf6ffd640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbdd806c8f0 0x7fbdd806edb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0006fd0 tx=0x7fbdf0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:26.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.569+0000 7fbdef7fe640 1 -- 172.21.15.44:0/3481304304 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbde002f320 con 0x7fbdf80d1060 2024-04-02T13:22:26.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.823+0000 7fbdfd622640 1 -- 172.21.15.44:0/3481304304 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fbdf80b2a80 con 0x7fbdf80d1060 2024-04-02T13:22:26.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.825+0000 7fbdef7fe640 1 -- 172.21.15.44:0/3481304304 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) ==== 190+0+147 (secure 0 0 0) 0x7fbde007bf00 con 0x7fbdf80d1060 2024-04-02T13:22:26.830 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-04-02T13:22:26.830 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A== 2024-04-02T13:22:26.830 INFO:teuthology.orchestra.run.smithi044.stdout: caps mds = "allow" 2024-04-02T13:22:26.830 INFO:teuthology.orchestra.run.smithi044.stdout: caps mon = "allow r" 2024-04-02T13:22:26.830 INFO:teuthology.orchestra.run.smithi044.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-02T13:22:26.831 INFO:teuthology.orchestra.run.smithi044.stderr:updated caps for client.0 2024-04-02T13:22:26.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.827+0000 7fbdfd622640 1 -- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbdd806c8f0 msgr2=0x7fbdd806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.827+0000 7fbdfd622640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbdd806c8f0 0x7fbdd806edb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0006fd0 tx=0x7fbdf0008040 comp rx=0 tx=0).stop 2024-04-02T13:22:26.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.827+0000 7fbdfd622640 1 -- 172.21.15.44:0/3481304304 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdf80d1060 msgr2=0x7fbdf815ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.827+0000 7fbdfd622640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdf80d1060 0x7fbdf815ea30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbde0000c00 tx=0x7fbde00046c0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.827+0000 7fbdfd622640 1 -- 172.21.15.44:0/3481304304 shutdown_connections 2024-04-02T13:22:26.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.827+0000 7fbdfd622640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbdd806c8f0 0x7fbdd806edb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.827+0000 7fbdfd622640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf815e4d0 0x7fbdf80b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.827+0000 7fbdfd622640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdf80d1060 0x7fbdf815ea30 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.827+0000 7fbdfd622640 1 --2- 172.21.15.44:0/3481304304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdf80cfd20 0x7fbdf814f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.827+0000 7fbdfd622640 1 -- 172.21.15.44:0/3481304304 >> 172.21.15.44:0/3481304304 conn(0x7fbdf809d410 msgr2=0x7fbdf806c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:26.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.827+0000 7fbdfd622640 1 -- 172.21.15.44:0/3481304304 shutdown_connections 2024-04-02T13:22:26.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.827+0000 7fbdfd622640 1 -- 172.21.15.44:0/3481304304 wait complete. 2024-04-02T13:22:26.848 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:22:26.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.955+0000 7f9d9412b640 1 Processor -- start 2024-04-02T13:22:26.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.955+0000 7f9d9412b640 1 -- start start 2024-04-02T13:22:26.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.956+0000 7f9d9412b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d8c15e550 0x7f9d8c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.956+0000 7f9d9412b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9d8c15f040 0x7f9d8c0d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.956+0000 7f9d9412b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d8c15fb50 con 0x7f9d8c06d530 2024-04-02T13:22:26.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.956+0000 7f9d9412b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f9d8c06d970 con 0x7f9d8c15f040 2024-04-02T13:22:26.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.956+0000 7f9d9412b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9d8c06ca90 con 0x7f9d8c15e550 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.956+0000 7f9d926c8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9d8c15f040 0x7f9d8c0d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.956+0000 7f9d926c8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9d8c15f040 0x7f9d8c0d0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43554/0 (socket says 172.21.15.44:43554) 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.956+0000 7f9d926c8640 1 -- 172.21.15.44:0/354637057 learned_addr learned my addr 172.21.15.44:0/354637057 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.956+0000 7f9d91ec7640 1 --2- 172.21.15.44:0/354637057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d8c15e550 0x7f9d8c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.956+0000 7f9d926c8640 1 -- 172.21.15.44:0/354637057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d8c15e550 msgr2=0x7f9d8c06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.956+0000 7f9d926c8640 1 --2- 172.21.15.44:0/354637057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d8c15e550 0x7f9d8c06cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.956+0000 7f9d926c8640 1 -- 172.21.15.44:0/354637057 >> v1:172.21.15.69:6789/0 conn(0x7f9d8c06d530 legacy=0x7f9d8c15e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.956+0000 7f9d926c8640 1 -- 172.21.15.44:0/354637057 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8c15f590 con 0x7f9d8c15f040 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.957+0000 7f9d926c8640 1 --2- 172.21.15.44:0/354637057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d8c15f040 0x7f9d8c0d0ab0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009e50 tx=0x7f9d8402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c6e00f5824b4cdf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.957+0000 7f9d90ec5640 1 -- 172.21.15.44:0/354637057 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8402cb90 con 0x7f9d8c15f040 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.957+0000 7f9d90ec5640 1 -- 172.21.15.44:0/354637057 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d8402cd30 con 0x7f9d8c15f040 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.957+0000 7f9d90ec5640 1 -- 172.21.15.44:0/354637057 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d84034610 con 0x7f9d8c15f040 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.958+0000 7f9d9412b640 1 -- 172.21.15.44:0/354637057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d8c15f040 msgr2=0x7f9d8c0d0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.958+0000 7f9d9412b640 1 --2- 172.21.15.44:0/354637057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d8c15f040 0x7f9d8c0d0ab0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009e50 tx=0x7f9d8402bff0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.958+0000 7f9d9412b640 1 -- 172.21.15.44:0/354637057 shutdown_connections 2024-04-02T13:22:26.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.958+0000 7f9d9412b640 1 --2- 172.21.15.44:0/354637057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d8c15f040 0x7f9d8c0d0ab0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.958+0000 7f9d9412b640 1 --2- 172.21.15.44:0/354637057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d8c15e550 0x7f9d8c06cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.958+0000 7f9d9412b640 1 -- 172.21.15.44:0/354637057 >> 172.21.15.44:0/354637057 conn(0x7f9d8c09aa20 msgr2=0x7f9d8c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:26.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.958+0000 7f9d9412b640 1 -- 172.21.15.44:0/354637057 shutdown_connections 2024-04-02T13:22:26.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.958+0000 7f9d9412b640 1 -- 172.21.15.44:0/354637057 wait complete. 2024-04-02T13:22:26.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.959+0000 7f9d9412b640 1 Processor -- start 2024-04-02T13:22:26.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.959+0000 7f9d9412b640 1 -- start start 2024-04-02T13:22:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d9412b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d8c06d530 0x7f9d8c0c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d9412b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d8c15e550 0x7f9d8c09c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d9412b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d8c15f040 0x7f9d8c09ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d9412b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8c0d0590 con 0x7f9d8c06d530 2024-04-02T13:22:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d9412b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8c0d02b0 con 0x7f9d8c15f040 2024-04-02T13:22:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d9412b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9d8c0cc4b0 con 0x7f9d8c15e550 2024-04-02T13:22:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d91ec7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d8c06d530 0x7f9d8c0c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d91ec7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d8c06d530 0x7f9d8c0c9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43570/0 (socket says 172.21.15.44:43570) 2024-04-02T13:22:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d91ec7640 1 -- 172.21.15.44:0/163933568 learned_addr learned my addr 172.21.15.44:0/163933568 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d91ec7640 1 -- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d8c15e550 msgr2=0x7f9d8c09c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d926c8640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d8c15f040 0x7f9d8c09ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d916c6640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d8c15e550 0x7f9d8c09c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d91ec7640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d8c15e550 0x7f9d8c09c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d91ec7640 1 -- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d8c15f040 msgr2=0x7f9d8c09ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d91ec7640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d8c15f040 0x7f9d8c09ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d91ec7640 1 -- 172.21.15.44:0/163933568 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8c0b4b80 con 0x7f9d8c06d530 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d91ec7640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d8c06d530 0x7f9d8c0c9940 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9d8800d5f0 tx=0x7f9d8800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d827fc640 1 -- 172.21.15.44:0/163933568 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d880125f0 con 0x7f9d8c06d530 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d827fc640 1 -- 172.21.15.44:0/163933568 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d88019070 con 0x7f9d8c06d530 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d9412b640 1 -- 172.21.15.44:0/163933568 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d8c0b4e60 con 0x7f9d8c06d530 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d827fc640 1 -- 172.21.15.44:0/163933568 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d880129d0 con 0x7f9d8c06d530 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.960+0000 7f9d9412b640 1 -- 172.21.15.44:0/163933568 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d8c0b5420 con 0x7f9d8c06d530 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.961+0000 7f9d9412b640 1 -- 172.21.15.44:0/163933568 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d8c0684f0 con 0x7f9d8c06d530 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.962+0000 7f9d827fc640 1 -- 172.21.15.44:0/163933568 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9d88012b70 con 0x7f9d8c06d530 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.962+0000 7f9d827fc640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9d7006c8f0 0x7f9d7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.962+0000 7f9d827fc640 1 -- 172.21.15.44:0/163933568 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f9d8808c160 con 0x7f9d8c06d530 2024-04-02T13:22:26.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.963+0000 7f9d916c6640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9d7006c8f0 0x7f9d7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:26.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.963+0000 7f9d916c6640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9d7006c8f0 0x7f9d7006edb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0703b0 tx=0x7f9d7c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:26.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:26.967+0000 7f9d827fc640 1 -- 172.21.15.44:0/163933568 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9d8805a360 con 0x7f9d8c06d530 2024-04-02T13:22:27.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.190+0000 7f9d9412b640 1 -- 172.21.15.44:0/163933568 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d8c0b3530 con 0x7f9d8c06d530 2024-04-02T13:22:27.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.190+0000 7f9d827fc640 1 -- 172.21.15.44:0/163933568 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5181 (secure 0 0 0) 0x7f9d88059c60 con 0x7f9d8c06d530 2024-04-02T13:22:27.192 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:27.192 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/4131284806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":4131284806},{"type":"v1","addr":"172.21.15.44:6833","nonce":4131284806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/4262427145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":4262427145},{"type":"v1","addr":"172.21.15.44:6835","nonce":4262427145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4112565894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4112565894},{"type":"v1","addr":"172.21.15.69:6836","nonce":4112565894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:22:22.422387+0000","modified":"2024-04-02T13:22:24.398291+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6837/1745145720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1745145720},{"type":"v1","addr":"172.21.15.69:6837","nonce":1745145720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":3}]} 2024-04-02T13:22:27.193 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-02T13:22:27.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.192+0000 7f9d9412b640 1 -- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9d7006c8f0 msgr2=0x7f9d7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.192+0000 7f9d9412b640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9d7006c8f0 0x7f9d7006edb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0703b0 tx=0x7f9d7c009040 comp rx=0 tx=0).stop 2024-04-02T13:22:27.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.192+0000 7f9d9412b640 1 -- 172.21.15.44:0/163933568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d8c06d530 msgr2=0x7f9d8c0c9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.192+0000 7f9d9412b640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d8c06d530 0x7f9d8c0c9940 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9d8800d5f0 tx=0x7f9d8800dab0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.192+0000 7f9d9412b640 1 -- 172.21.15.44:0/163933568 shutdown_connections 2024-04-02T13:22:27.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.192+0000 7f9d9412b640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9d7006c8f0 0x7f9d7006edb0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.192+0000 7f9d9412b640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d8c15f040 0x7f9d8c09ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.192+0000 7f9d9412b640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d8c15e550 0x7f9d8c09c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.192+0000 7f9d9412b640 1 --2- 172.21.15.44:0/163933568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d8c06d530 0x7f9d8c0c9940 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.192+0000 7f9d9412b640 1 -- 172.21.15.44:0/163933568 >> 172.21.15.44:0/163933568 conn(0x7f9d8c09aa20 msgr2=0x7f9d8c0ccfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:27.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.192+0000 7f9d9412b640 1 -- 172.21.15.44:0/163933568 shutdown_connections 2024-04-02T13:22:27.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.192+0000 7f9d9412b640 1 -- 172.21.15.44:0/163933568 wait complete. 2024-04-02T13:22:27.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:22:22.422387+0000', 'modified': '2024-04-02T13:22:24.398291+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5991}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5991': {'gid': 5991, 'name': 'd', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.69:6837/1745145720', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1745145720}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1745145720}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5991, 'qdb_cluster': [5991]} 2024-04-02T13:22:27.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:22:27.607 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-02T13:22:27.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.712+0000 7fef158eb640 1 Processor -- start 2024-04-02T13:22:27.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.712+0000 7fef158eb640 1 -- start start 2024-04-02T13:22:27.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.712+0000 7fef158eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef1015e0f0 0x7fef100d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.712+0000 7fef158eb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fef100d0630 0x7fef1006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.712+0000 7fef158eb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef100aed70 con 0x7fef1015e4d0 2024-04-02T13:22:27.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.712+0000 7fef158eb640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fef1006dfe0 con 0x7fef100d0630 2024-04-02T13:22:27.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.712+0000 7fef158eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fef1006e2c0 con 0x7fef1015e0f0 2024-04-02T13:22:27.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.713+0000 7fef07fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fef100d0630 0x7fef1006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.713+0000 7fef07fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fef100d0630 0x7fef1006b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43580/0 (socket says 172.21.15.44:43580) 2024-04-02T13:22:27.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.713+0000 7fef07fff640 1 -- 172.21.15.44:0/2010315436 learned_addr learned my addr 172.21.15.44:0/2010315436 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:27.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.713+0000 7fef07fff640 1 -- 172.21.15.44:0/2010315436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef1015e0f0 msgr2=0x7fef100d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:27.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.713+0000 7fef07fff640 1 --2- 172.21.15.44:0/2010315436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef1015e0f0 0x7fef100d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.713+0000 7fef07fff640 1 -- 172.21.15.44:0/2010315436 >> v1:172.21.15.69:6789/0 conn(0x7fef1015e4d0 legacy=0x7fef10161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:27.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.713+0000 7fef07fff640 1 -- 172.21.15.44:0/2010315436 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1006e5a0 con 0x7fef100d0630 2024-04-02T13:22:27.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.714+0000 7fef07fff640 1 --2- 172.21.15.44:0/2010315436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef100d0630 0x7fef1006b4d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7feef80098a0 tx=0x7feef802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bcc5bd87c14b106 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:27.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.714+0000 7fef0e7fc640 1 -- 172.21.15.44:0/2010315436 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8034070 con 0x7fef100d0630 2024-04-02T13:22:27.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.714+0000 7fef0e7fc640 1 -- 172.21.15.44:0/2010315436 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feef8030840 con 0x7fef100d0630 2024-04-02T13:22:27.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.714+0000 7fef0e7fc640 1 -- 172.21.15.44:0/2010315436 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8030c10 con 0x7fef100d0630 2024-04-02T13:22:27.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.714+0000 7fef158eb640 1 -- 172.21.15.44:0/2010315436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef100d0630 msgr2=0x7fef1006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.714+0000 7fef158eb640 1 --2- 172.21.15.44:0/2010315436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef100d0630 0x7fef1006b4d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7feef80098a0 tx=0x7feef802b980 comp rx=0 tx=0).stop 2024-04-02T13:22:27.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.714+0000 7fef158eb640 1 -- 172.21.15.44:0/2010315436 shutdown_connections 2024-04-02T13:22:27.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.714+0000 7fef158eb640 1 --2- 172.21.15.44:0/2010315436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef100d0630 0x7fef1006b4d0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.714+0000 7fef158eb640 1 --2- 172.21.15.44:0/2010315436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef1015e0f0 0x7fef100d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.714+0000 7fef158eb640 1 -- 172.21.15.44:0/2010315436 >> 172.21.15.44:0/2010315436 conn(0x7fef100b2980 msgr2=0x7fef100b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:27.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.714+0000 7fef158eb640 1 -- 172.21.15.44:0/2010315436 shutdown_connections 2024-04-02T13:22:27.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.715+0000 7fef158eb640 1 -- 172.21.15.44:0/2010315436 wait complete. 2024-04-02T13:22:27.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.716+0000 7fef158eb640 1 Processor -- start 2024-04-02T13:22:27.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.716+0000 7fef158eb640 1 -- start start 2024-04-02T13:22:27.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.716+0000 7fef158eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef100d0630 0x7fef101f78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.716+0000 7fef158eb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef1015e0f0 0x7fef100c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.716+0000 7fef158eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef1015e4d0 0x7fef1015b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.716+0000 7fef158eb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fef1006eda0 con 0x7fef1015e0f0 2024-04-02T13:22:27.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.716+0000 7fef158eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fef100d0af0 con 0x7fef100d0630 2024-04-02T13:22:27.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.716+0000 7fef158eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fef10072f60 con 0x7fef1015e4d0 2024-04-02T13:22:27.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.716+0000 7fef07fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef1015e0f0 0x7fef100c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.716+0000 7fef07fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef1015e0f0 0x7fef100c6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43588/0 (socket says 172.21.15.44:43588) 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.716+0000 7fef07fff640 1 -- 172.21.15.44:0/3099230027 learned_addr learned my addr 172.21.15.44:0/3099230027 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef0effd640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef100d0630 0x7fef101f78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef0f7fe640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef1015e4d0 0x7fef1015b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef07fff640 1 -- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef1015e4d0 msgr2=0x7fef1015b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef07fff640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef1015e4d0 0x7fef1015b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef07fff640 1 -- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef100d0630 msgr2=0x7fef101f78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef07fff640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef100d0630 0x7fef101f78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef07fff640 1 -- 172.21.15.44:0/3099230027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef101f8720 con 0x7fef1015e0f0 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef0effd640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef100d0630 0x7fef101f78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef07fff640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef1015e0f0 0x7fef100c6be0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7feef80098a0 tx=0x7feef8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef0cff9640 1 -- 172.21.15.44:0/3099230027 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8048070 con 0x7fef1015e0f0 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef0cff9640 1 -- 172.21.15.44:0/3099230027 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feef8030640 con 0x7fef1015e0f0 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef158eb640 1 -- 172.21.15.44:0/3099230027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef101f7e80 con 0x7fef1015e0f0 2024-04-02T13:22:27.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef0cff9640 1 -- 172.21.15.44:0/3099230027 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8034040 con 0x7fef1015e0f0 2024-04-02T13:22:27.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.717+0000 7fef158eb640 1 -- 172.21.15.44:0/3099230027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef101f8440 con 0x7fef1015e0f0 2024-04-02T13:22:27.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.719+0000 7fef0cff9640 1 -- 172.21.15.44:0/3099230027 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feef804b070 con 0x7fef1015e0f0 2024-04-02T13:22:27.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.719+0000 7fef0cff9640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feef006c920 0x7feef006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.720+0000 7fef0cff9640 1 -- 172.21.15.44:0/3099230027 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7feef80adcb0 con 0x7fef1015e0f0 2024-04-02T13:22:27.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.720+0000 7fef0effd640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feef006c920 0x7feef006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.720+0000 7fef185af740 1 Processor -- start 2024-04-02T13:22:27.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.720+0000 7fef0effd640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feef006c920 0x7feef006ede0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fef00009930 tx=0x7fef00008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:27.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.720+0000 7fef185af740 1 -- start start 2024-04-02T13:22:27.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.721+0000 7fef185af740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371eb90 0x5646536f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.721+0000 7fef185af740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371ef70 0x564653721920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.721+0000 7fef185af740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5646536f2070 0x564653721e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.721+0000 7fef185af740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56465365b060 con 0x56465371eb90 2024-04-02T13:22:27.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.721+0000 7fef185af740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564653603650 con 0x5646536f2070 2024-04-02T13:22:27.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.721+0000 7fef185af740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564653657010 con 0x56465371ef70 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.721+0000 7fef0effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371eb90 0x5646536f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.721+0000 7fef0effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371eb90 0x5646536f1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43592/0 (socket says 172.21.15.44:43592) 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.721+0000 7fef0effd640 1 -- 172.21.15.44:0/666551759 learned_addr learned my addr 172.21.15.44:0/666551759 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.721+0000 7fef0f7fe640 1 --2- 172.21.15.44:0/666551759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5646536f2070 0x564653721e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.721+0000 7fef07fff640 1 --2- 172.21.15.44:0/666551759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371ef70 0x564653721920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.722+0000 7fef0effd640 1 -- 172.21.15.44:0/666551759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371ef70 msgr2=0x564653721920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.722+0000 7fef0effd640 1 --2- 172.21.15.44:0/666551759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371ef70 0x564653721920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.722+0000 7fef0effd640 1 -- 172.21.15.44:0/666551759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5646536f2070 msgr2=0x564653721e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.722+0000 7fef0effd640 1 --2- 172.21.15.44:0/666551759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5646536f2070 0x564653721e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.722+0000 7fef0effd640 1 -- 172.21.15.44:0/666551759 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56465336f2d0 con 0x56465371eb90 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.722+0000 7fef0effd640 1 --2- 172.21.15.44:0/666551759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371eb90 0x5646536f1b30 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fef0000d940 tx=0x7fef00010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb8a970a218495a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.722+0000 7fef057fa640 1 -- 172.21.15.44:0/666551759 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef00024070 con 0x56465371eb90 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.722+0000 7fef057fa640 1 -- 172.21.15.44:0/666551759 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef00014bc0 con 0x56465371eb90 2024-04-02T13:22:27.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.722+0000 7fef057fa640 1 -- 172.21.15.44:0/666551759 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0001f400 con 0x56465371eb90 2024-04-02T13:22:27.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.722+0000 7fef185af740 1 -- 172.21.15.44:0/666551759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371eb90 msgr2=0x5646536f1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.722+0000 7fef185af740 1 --2- 172.21.15.44:0/666551759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371eb90 0x5646536f1b30 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fef0000d940 tx=0x7fef00010000 comp rx=0 tx=0).stop 2024-04-02T13:22:27.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.723+0000 7fef185af740 1 -- 172.21.15.44:0/666551759 shutdown_connections 2024-04-02T13:22:27.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.723+0000 7fef185af740 1 --2- 172.21.15.44:0/666551759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5646536f2070 0x564653721e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.723+0000 7fef185af740 1 --2- 172.21.15.44:0/666551759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371ef70 0x564653721920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.723+0000 7fef185af740 1 --2- 172.21.15.44:0/666551759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371eb90 0x5646536f1b30 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.723+0000 7fef185af740 1 -- 172.21.15.44:0/666551759 >> 172.21.15.44:0/666551759 conn(0x5646536f3da0 msgr2=0x56465373c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:27.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.723+0000 7fef185af740 1 -- 172.21.15.44:0/666551759 shutdown_connections 2024-04-02T13:22:27.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.723+0000 7fef185af740 1 -- 172.21.15.44:0/666551759 wait complete. 2024-04-02T13:22:27.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.723+0000 7fef185af740 1 -- start start 2024-04-02T13:22:27.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.723+0000 7fef185af740 1 Processor -- start 2024-04-02T13:22:27.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef185af740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371eb90 0x564653722ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef185af740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371ef70 0x56465374d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef185af740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 0x564653727610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef185af740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5646536f6440 con 0x564653724190 2024-04-02T13:22:27.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef185af740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564653616a10 con 0x56465371eb90 2024-04-02T13:22:27.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef185af740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5646536f8c80 con 0x56465371ef70 2024-04-02T13:22:27.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef185af740 10 client.0 _ll_register_callbacks cb 0x5646536530b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:22:27.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef0f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 0x564653727610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef0f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 0x564653727610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43608/0 (socket says 172.21.15.44:43608) 2024-04-02T13:22:27.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef0f7fe640 1 -- 172.21.15.44:0/1757529061 learned_addr learned my addr 172.21.15.44:0/1757529061 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:27.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef0effd640 1 --2- 172.21.15.44:0/1757529061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371eb90 0x564653722ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef07fff640 1 --2- 172.21.15.44:0/1757529061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371ef70 0x56465374d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef0f7fe640 1 -- 172.21.15.44:0/1757529061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371ef70 msgr2=0x56465374d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef0f7fe640 1 --2- 172.21.15.44:0/1757529061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371ef70 0x56465374d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef0f7fe640 1 -- 172.21.15.44:0/1757529061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371eb90 msgr2=0x564653722ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef0f7fe640 1 --2- 172.21.15.44:0/1757529061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371eb90 0x564653722ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.724+0000 7fef0f7fe640 1 -- 172.21.15.44:0/1757529061 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646534396b0 con 0x564653724190 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7fef0f7fe640 1 --2- 172.21.15.44:0/1757529061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 0x564653727610 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fef0800bba0 tx=0x7fef08007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c3af05eb225d71 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7feeee7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7feeee7fc640 1 -- 172.21.15.44:0/1757529061 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0801c060 con 0x564653724190 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7feeee7fc640 1 -- 172.21.15.44:0/1757529061 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fef08017410 con 0x564653724190 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7fef185af740 1 -- 172.21.15.44:0/1757529061 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef00014bc0 con 0x564653724190 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7feeee7fc640 1 -- 172.21.15.44:0/1757529061 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fef08022050 con 0x564653724190 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7fef185af740 10 client.6126 fetch_fsmap learned FSMap version 25 2024-04-02T13:22:27.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7fef185af740 1 -- 172.21.15.44:0/1757529061 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x564653726250 con 0x564653724190 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7feeee7fc640 1 -- 172.21.15.44:0/1757529061 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef00014bc0 con 0x564653724190 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7feeee7fc640 1 -- 172.21.15.44:0/1757529061 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7fef08025020 con 0x564653724190 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7fef185af740 10 client.6126 fetch_fsmap finished waiting for FSMap version 25 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7fef185af740 10 client.6126 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7fef185af740 20 client.6126 populate_metadata read hostname 'smithi044' 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7fef185af740 1 --2- 172.21.15.44:0/1757529061 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x564653726b60 0x56465375add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7fef185af740 4 client.6126 mds_command: new command op to 5991 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7fef185af740 1 -- 172.21.15.44:0/1757529061 --> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564653602cf0 con 0x564653726b60 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.725+0000 7fef0effd640 1 --2- 172.21.15.44:0/1757529061 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x564653726b60 0x56465375add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.726+0000 7feeee7fc640 10 client.6126 ms_handle_connect on v2:172.21.15.69:6835/1745145720 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.726+0000 7fef0effd640 1 --2- 172.21.15.44:0/1757529061 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x564653726b60 0x56465375add0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=356a52bcd9135bb9 server_cookie=20f4fb45e365d7b1 in_seq=0 out_seq=0 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7feeee7fc640 1 -- 172.21.15.44:0/1757529061 <== mds.0 v2:172.21.15.69:6835/1745145720 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564653602cf0 con 0x564653726b60 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7feeee7fc640 10 client.6126 handle_command_reply: tid=0 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7fef185af740 1 client.6126 shutdown 2024-04-02T13:22:27.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7fef185af740 1 -- 172.21.15.44:0/1757529061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 msgr2=0x564653727610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7fef185af740 1 --2- 172.21.15.44:0/1757529061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 0x564653727610 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fef0800bba0 tx=0x7fef08007680 comp rx=0 tx=0).stop 2024-04-02T13:22:27.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7fef185af740 1 -- 172.21.15.44:0/1757529061 shutdown_connections 2024-04-02T13:22:27.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7fef185af740 1 --2- 172.21.15.44:0/1757529061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 0x564653727610 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7fef185af740 1 --2- 172.21.15.44:0/1757529061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371ef70 0x56465374d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7fef185af740 1 --2- 172.21.15.44:0/1757529061 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x564653726b60 0x56465375add0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7fef185af740 1 --2- 172.21.15.44:0/1757529061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371eb90 0x564653722ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7feeee7fc640 0 client.6126 ms_handle_reset on v2:172.21.15.69:6835/1745145720 2024-04-02T13:22:27.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7fef185af740 1 -- 172.21.15.44:0/1757529061 >> 172.21.15.44:0/1757529061 conn(0x5646536f2070 msgr2=0x5646536f5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:27.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7fef185af740 1 -- 172.21.15.44:0/1757529061 shutdown_connections 2024-04-02T13:22:27.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7fef185af740 1 -- 172.21.15.44:0/1757529061 wait complete. 2024-04-02T13:22:27.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.727+0000 7fef185af740 20 client.6126 trim_cache size 0 max 16384 2024-04-02T13:22:27.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.745+0000 7fef185af740 1 Processor -- start 2024-04-02T13:22:27.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.745+0000 7fef185af740 1 -- start start 2024-04-02T13:22:27.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef185af740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371eb90 0x564653725510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef185af740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371ef70 0x56465374c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef185af740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 0x56465374cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef185af740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fef101fffe0 con 0x564653724190 2024-04-02T13:22:27.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef185af740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fef100b5a50 con 0x56465371ef70 2024-04-02T13:22:27.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef185af740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fef100b60a0 con 0x56465371eb90 2024-04-02T13:22:27.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef0f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 0x56465374cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef0f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 0x56465374cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43622/0 (socket says 172.21.15.44:43622) 2024-04-02T13:22:27.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef0f7fe640 1 -- 172.21.15.44:0/1772886699 learned_addr learned my addr 172.21.15.44:0/1772886699 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:27.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef0f7fe640 1 -- 172.21.15.44:0/1772886699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371eb90 msgr2=0x564653725510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:27.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef0f7fe640 1 --2- 172.21.15.44:0/1772886699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371eb90 0x564653725510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef0f7fe640 1 -- 172.21.15.44:0/1772886699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371ef70 msgr2=0x56465374c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:27.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef0f7fe640 1 --2- 172.21.15.44:0/1772886699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371ef70 0x56465374c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.746+0000 7fef0f7fe640 1 -- 172.21.15.44:0/1772886699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646536f8f60 con 0x564653724190 2024-04-02T13:22:27.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.747+0000 7fef0f7fe640 1 --2- 172.21.15.44:0/1772886699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 0x56465374cee0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fef0800b520 tx=0x7fef08013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1166283a4fa538 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:27.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.747+0000 7fef04ff9640 1 -- 172.21.15.44:0/1772886699 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08017410 con 0x564653724190 2024-04-02T13:22:27.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.747+0000 7fef04ff9640 1 -- 172.21.15.44:0/1772886699 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef080175f0 con 0x564653724190 2024-04-02T13:22:27.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.747+0000 7fef04ff9640 1 -- 172.21.15.44:0/1772886699 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08017c20 con 0x564653724190 2024-04-02T13:22:27.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.747+0000 7fef185af740 1 -- 172.21.15.44:0/1772886699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 msgr2=0x56465374cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.747+0000 7fef185af740 1 --2- 172.21.15.44:0/1772886699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 0x56465374cee0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fef0800b520 tx=0x7fef08013000 comp rx=0 tx=0).stop 2024-04-02T13:22:27.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.747+0000 7fef185af740 1 -- 172.21.15.44:0/1772886699 shutdown_connections 2024-04-02T13:22:27.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.747+0000 7fef185af740 1 --2- 172.21.15.44:0/1772886699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564653724190 0x56465374cee0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.747+0000 7fef185af740 1 --2- 172.21.15.44:0/1772886699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371ef70 0x56465374c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.747+0000 7fef185af740 1 --2- 172.21.15.44:0/1772886699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56465371eb90 0x564653725510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.747+0000 7fef185af740 1 -- 172.21.15.44:0/1772886699 >> 172.21.15.44:0/1772886699 conn(0x5646536f2070 msgr2=0x5646536ece50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:27.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.747+0000 7fef185af740 1 -- 172.21.15.44:0/1772886699 shutdown_connections 2024-04-02T13:22:27.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.747+0000 7fef185af740 1 -- 172.21.15.44:0/1772886699 wait complete. 2024-04-02T13:22:27.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.748+0000 7fef185af740 1 -- start start 2024-04-02T13:22:27.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.748+0000 7fef185af740 1 Processor -- start 2024-04-02T13:22:27.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef185af740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371eb90 0x56465379c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef185af740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371ef70 0x56465374c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef185af740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564653724190 0x5646536f0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef185af740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5646537269e0 con 0x56465371ef70 2024-04-02T13:22:27.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef185af740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564653726590 con 0x56465371eb90 2024-04-02T13:22:27.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef185af740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564653726f70 con 0x564653724190 2024-04-02T13:22:27.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef185af740 10 client.0 _ll_register_callbacks cb 0x7fef08009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:22:27.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef07fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371ef70 0x56465374c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef07fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371ef70 0x56465374c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43636/0 (socket says 172.21.15.44:43636) 2024-04-02T13:22:27.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef07fff640 1 -- 172.21.15.44:0/2317358335 learned_addr learned my addr 172.21.15.44:0/2317358335 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:27.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef0effd640 1 --2- 172.21.15.44:0/2317358335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371eb90 0x56465379c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef0f7fe640 1 --2- 172.21.15.44:0/2317358335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564653724190 0x5646536f0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef07fff640 1 -- 172.21.15.44:0/2317358335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564653724190 msgr2=0x5646536f0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:27.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef07fff640 1 --2- 172.21.15.44:0/2317358335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564653724190 0x5646536f0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef07fff640 1 -- 172.21.15.44:0/2317358335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371eb90 msgr2=0x56465379c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:27.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef07fff640 1 --2- 172.21.15.44:0/2317358335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371eb90 0x56465379c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef07fff640 1 -- 172.21.15.44:0/2317358335 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646537214e0 con 0x56465371ef70 2024-04-02T13:22:27.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7fef07fff640 1 --2- 172.21.15.44:0/2317358335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371ef70 0x56465374c560 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7feef802bf90 tx=0x7feef807b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f8f1f453c60b126 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:27.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7feeee7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:22:27.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.749+0000 7feeee7fc640 1 -- 172.21.15.44:0/2317358335 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef80844f0 con 0x56465371ef70 2024-04-02T13:22:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7feeee7fc640 1 -- 172.21.15.44:0/2317358335 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7feef8077f80 con 0x56465371ef70 2024-04-02T13:22:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7fef185af740 1 -- 172.21.15.44:0/2317358335 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef080175f0 con 0x56465371ef70 2024-04-02T13:22:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7feeee7fc640 1 -- 172.21.15.44:0/2317358335 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7feef809c070 con 0x56465371ef70 2024-04-02T13:22:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7fef185af740 10 client.6132 fetch_fsmap learned FSMap version 25 2024-04-02T13:22:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7fef185af740 1 -- 172.21.15.44:0/2317358335 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x56465374cb00 con 0x56465371ef70 2024-04-02T13:22:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7feeee7fc640 1 -- 172.21.15.44:0/2317358335 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef080175f0 con 0x56465371ef70 2024-04-02T13:22:27.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7feeee7fc640 1 -- 172.21.15.44:0/2317358335 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7feef8042070 con 0x56465371ef70 2024-04-02T13:22:27.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7fef185af740 10 client.6132 fetch_fsmap finished waiting for FSMap version 25 2024-04-02T13:22:27.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7fef185af740 10 client.6132 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-02T13:22:27.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7fef185af740 20 client.6132 populate_metadata read hostname 'smithi044' 2024-04-02T13:22:27.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7fef185af740 1 --2- 172.21.15.44:0/2317358335 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x56465371dc40 0x5646537b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7fef185af740 4 client.6132 mds_command: new command op to 5991 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:22:27.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7fef185af740 1 -- 172.21.15.44:0/2317358335 --> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] -- command(tid 0: {"prefix": "status"}) -- 0x56465372a0d0 con 0x56465371dc40 2024-04-02T13:22:27.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7fef0effd640 1 --2- 172.21.15.44:0/2317358335 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x56465371dc40 0x5646537b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7feeee7fc640 10 client.6132 ms_handle_connect on v2:172.21.15.69:6835/1745145720 2024-04-02T13:22:27.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.750+0000 7fef0effd640 1 --2- 172.21.15.44:0/2317358335 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x56465371dc40 0x5646537b6eb0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ac8ed198f6c8a2 server_cookie=30c8e3f666b9be18 in_seq=0 out_seq=0 2024-04-02T13:22:27.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7feeee7fc640 1 -- 172.21.15.44:0/2317358335 <== mds.0 v2:172.21.15.69:6835/1745145720 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x56465372a0d0 con 0x56465371dc40 2024-04-02T13:22:27.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7feeee7fc640 10 client.6132 handle_command_reply: tid=0 2024-04-02T13:22:27.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef185af740 1 client.6132 shutdown 2024-04-02T13:22:27.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef185af740 1 -- 172.21.15.44:0/2317358335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371ef70 msgr2=0x56465374c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef185af740 1 --2- 172.21.15.44:0/2317358335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371ef70 0x56465374c560 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7feef802bf90 tx=0x7feef807b3a0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef185af740 1 -- 172.21.15.44:0/2317358335 shutdown_connections 2024-04-02T13:22:27.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef185af740 1 --2- 172.21.15.44:0/2317358335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564653724190 0x5646536f0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef185af740 1 --2- 172.21.15.44:0/2317358335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56465371ef70 0x56465374c560 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef185af740 1 --2- 172.21.15.44:0/2317358335 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x56465371dc40 0x5646537b6eb0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef185af740 1 --2- 172.21.15.44:0/2317358335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56465371eb90 0x56465379c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7feeee7fc640 0 client.6132 ms_handle_reset on v2:172.21.15.69:6835/1745145720 2024-04-02T13:22:27.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef185af740 1 -- 172.21.15.44:0/2317358335 >> 172.21.15.44:0/2317358335 conn(0x5646536f2070 msgr2=0x56465375d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:27.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef185af740 1 -- 172.21.15.44:0/2317358335 shutdown_connections 2024-04-02T13:22:27.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef185af740 1 -- 172.21.15.44:0/2317358335 wait complete. 2024-04-02T13:22:27.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef185af740 20 client.6132 trim_cache size 0 max 16384 2024-04-02T13:22:27.772 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:22:27.772 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:22:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:22:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 5991, 2024-04-02T13:22:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:22:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:22:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:22:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 4.7078651279999999, 2024-04-02T13:22:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 25, 2024-04-02T13:22:27.774 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 52, 2024-04-02T13:22:27.774 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 52, 2024-04-02T13:22:27.774 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 9.7392196640000002 2024-04-02T13:22:27.774 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:22:27.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef158eb640 1 -- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feef006c920 msgr2=0x7feef006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef158eb640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feef006c920 0x7feef006ede0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fef00009930 tx=0x7fef00008040 comp rx=0 tx=0).stop 2024-04-02T13:22:27.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef158eb640 1 -- 172.21.15.44:0/3099230027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef1015e0f0 msgr2=0x7fef100c6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef158eb640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef1015e0f0 0x7fef100c6be0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7feef80098a0 tx=0x7feef8004730 comp rx=0 tx=0).stop 2024-04-02T13:22:27.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef158eb640 1 -- 172.21.15.44:0/3099230027 shutdown_connections 2024-04-02T13:22:27.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef158eb640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feef006c920 0x7feef006ede0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef158eb640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef1015e4d0 0x7fef1015b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef158eb640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef1015e0f0 0x7fef100c6be0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef158eb640 1 --2- 172.21.15.44:0/3099230027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef100d0630 0x7fef101f78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.751+0000 7fef158eb640 1 -- 172.21.15.44:0/3099230027 >> 172.21.15.44:0/3099230027 conn(0x7fef100b2980 msgr2=0x7fef100b1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:27.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.752+0000 7fef158eb640 1 -- 172.21.15.44:0/3099230027 shutdown_connections 2024-04-02T13:22:27.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.752+0000 7fef158eb640 1 -- 172.21.15.44:0/3099230027 wait complete. 2024-04-02T13:22:27.778 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:22:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.878+0000 7f957dbe2640 1 Processor -- start 2024-04-02T13:22:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.878+0000 7f957dbe2640 1 -- start start 2024-04-02T13:22:27.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.878+0000 7f957dbe2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f95780d0a30 0x7f957806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.878+0000 7f957dbe2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f95780d2040 0x7f95780d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.878+0000 7f957dbe2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95780d0000 con 0x7f95780d0e10 2024-04-02T13:22:27.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.878+0000 7f957dbe2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f957806cb00 con 0x7f95780d2040 2024-04-02T13:22:27.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.878+0000 7f957dbe2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f957806c1d0 con 0x7f95780d0a30 2024-04-02T13:22:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.879+0000 7f9577fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f95780d2040 0x7f95780d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.879+0000 7f9577fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f95780d2040 0x7f95780d24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43646/0 (socket says 172.21.15.44:43646) 2024-04-02T13:22:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.879+0000 7f9577fff640 1 -- 172.21.15.44:0/4276947773 learned_addr learned my addr 172.21.15.44:0/4276947773 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.879+0000 7f95777fe640 1 --2- 172.21.15.44:0/4276947773 >> v2:172.21.15.69:3300/0 conn(0x7f95780d0a30 0x7f957806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.879+0000 7f9577fff640 1 -- 172.21.15.44:0/4276947773 >> v2:172.21.15.69:3300/0 conn(0x7f95780d0a30 msgr2=0x7f957806cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.879+0000 7f9577fff640 1 --2- 172.21.15.44:0/4276947773 >> v2:172.21.15.69:3300/0 conn(0x7f95780d0a30 0x7f957806cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.879+0000 7f9577fff640 1 -- 172.21.15.44:0/4276947773 >> v1:172.21.15.69:6789/0 conn(0x7f95780d0e10 legacy=0x7f95780cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.879+0000 7f9577fff640 1 -- 172.21.15.44:0/4276947773 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957806c4b0 con 0x7f95780d2040 2024-04-02T13:22:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.880+0000 7f9577fff640 1 --2- 172.21.15.44:0/4276947773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95780d2040 0x7f95780d24f0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f957806de20 tx=0x7f957002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ed5d6fcf6e5bfa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.880+0000 7f95767fc640 1 -- 172.21.15.44:0/4276947773 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957002ca80 con 0x7f95780d2040 2024-04-02T13:22:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.880+0000 7f95767fc640 1 -- 172.21.15.44:0/4276947773 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f957002cc20 con 0x7f95780d2040 2024-04-02T13:22:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.880+0000 7f95767fc640 1 -- 172.21.15.44:0/4276947773 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95700343f0 con 0x7f95780d2040 2024-04-02T13:22:27.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.880+0000 7f957dbe2640 1 -- 172.21.15.44:0/4276947773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95780d2040 msgr2=0x7f95780d24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.880+0000 7f957dbe2640 1 --2- 172.21.15.44:0/4276947773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95780d2040 0x7f95780d24f0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f957806de20 tx=0x7f957002bee0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.880+0000 7f957dbe2640 1 -- 172.21.15.44:0/4276947773 shutdown_connections 2024-04-02T13:22:27.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.880+0000 7f957dbe2640 1 --2- 172.21.15.44:0/4276947773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95780d2040 0x7f95780d24f0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.880+0000 7f957dbe2640 1 --2- 172.21.15.44:0/4276947773 >> v2:172.21.15.69:3300/0 conn(0x7f95780d0a30 0x7f957806cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.880+0000 7f957dbe2640 1 -- 172.21.15.44:0/4276947773 >> 172.21.15.44:0/4276947773 conn(0x7f957809d470 msgr2=0x7f95780d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:27.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.880+0000 7f957dbe2640 1 -- 172.21.15.44:0/4276947773 shutdown_connections 2024-04-02T13:22:27.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.880+0000 7f957dbe2640 1 -- 172.21.15.44:0/4276947773 wait complete. 2024-04-02T13:22:27.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.881+0000 7f957dbe2640 1 Processor -- start 2024-04-02T13:22:27.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.882+0000 7f957dbe2640 1 -- start start 2024-04-02T13:22:27.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.882+0000 7f957dbe2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95780d0a30 0x7f95781514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.882+0000 7f957dbe2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95780d0e10 0x7f95780b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.882+0000 7f957dbe2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95780d2040 0x7f95780b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.882+0000 7f957dbe2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f957815e6f0 con 0x7f95780d2040 2024-04-02T13:22:27.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.882+0000 7f957dbe2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f95780d1270 con 0x7f95780d0a30 2024-04-02T13:22:27.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.882+0000 7f957dbe2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f957815e870 con 0x7f95780d0e10 2024-04-02T13:22:27.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f9577fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95780d2040 0x7f95780b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f9577fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95780d2040 0x7f95780b5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43654/0 (socket says 172.21.15.44:43654) 2024-04-02T13:22:27.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f9577fff640 1 -- 172.21.15.44:0/395421710 learned_addr learned my addr 172.21.15.44:0/395421710 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:27.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f95777fe640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95780d0a30 0x7f95781514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f9577fff640 1 -- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95780d0e10 msgr2=0x7f95780b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f9576ffd640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95780d0e10 0x7f95780b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f9577fff640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95780d0e10 0x7f95780b58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f9577fff640 1 -- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95780d0a30 msgr2=0x7f95781514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:27.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f9577fff640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95780d0a30 0x7f95781514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:27.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f9577fff640 1 -- 172.21.15.44:0/395421710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95781519f0 con 0x7f95780d2040 2024-04-02T13:22:27.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f95777fe640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95780d0a30 0x7f95781514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:27.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f9576ffd640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95780d0e10 0x7f95780b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:27.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f9577fff640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95780d2040 0x7f95780b5de0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9570036040 tx=0x7f9570004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:27.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f955ffff640 1 -- 172.21.15.44:0/395421710 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9570043020 con 0x7f95780d2040 2024-04-02T13:22:27.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f955ffff640 1 -- 172.21.15.44:0/395421710 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f957003e450 con 0x7f95780d2040 2024-04-02T13:22:27.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f957dbe2640 1 -- 172.21.15.44:0/395421710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95780b47d0 con 0x7f95780d2040 2024-04-02T13:22:27.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f955ffff640 1 -- 172.21.15.44:0/395421710 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9570009510 con 0x7f95780d2040 2024-04-02T13:22:27.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.883+0000 7f957dbe2640 1 -- 172.21.15.44:0/395421710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95780b4d90 con 0x7f95780d2040 2024-04-02T13:22:27.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.884+0000 7f957dbe2640 1 -- 172.21.15.44:0/395421710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95780684f0 con 0x7f95780d2040 2024-04-02T13:22:27.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.885+0000 7f955ffff640 1 -- 172.21.15.44:0/395421710 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9570046020 con 0x7f95780d2040 2024-04-02T13:22:27.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.885+0000 7f955ffff640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f956406c8f0 0x7f956406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:27.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.885+0000 7f955ffff640 1 -- 172.21.15.44:0/395421710 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f95700ae490 con 0x7f95780d2040 2024-04-02T13:22:27.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.885+0000 7f95777fe640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f956406c8f0 0x7f956406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:27.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.886+0000 7f95777fe640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f956406c8f0 0x7f956406edb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f95780ce5c0 tx=0x7f9568008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:27.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:27.890+0000 7f955ffff640 1 -- 172.21.15.44:0/395421710 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f957004d030 con 0x7f95780d2040 2024-04-02T13:22:28.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.113+0000 7f957dbe2640 1 -- 172.21.15.44:0/395421710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9578161830 con 0x7f95780d2040 2024-04-02T13:22:28.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.113+0000 7f955ffff640 1 -- 172.21.15.44:0/395421710 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f957007c730 con 0x7f95780d2040 2024-04-02T13:22:28.115 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:22:28.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.115+0000 7f957dbe2640 1 -- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f956406c8f0 msgr2=0x7f956406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:28.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.115+0000 7f957dbe2640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f956406c8f0 0x7f956406edb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f95780ce5c0 tx=0x7f9568008040 comp rx=0 tx=0).stop 2024-04-02T13:22:28.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.115+0000 7f957dbe2640 1 -- 172.21.15.44:0/395421710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95780d2040 msgr2=0x7f95780b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:28.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.115+0000 7f957dbe2640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95780d2040 0x7f95780b5de0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9570036040 tx=0x7f9570004a40 comp rx=0 tx=0).stop 2024-04-02T13:22:28.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.115+0000 7f957dbe2640 1 -- 172.21.15.44:0/395421710 shutdown_connections 2024-04-02T13:22:28.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.115+0000 7f957dbe2640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f956406c8f0 0x7f956406edb0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:28.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.115+0000 7f957dbe2640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95780d2040 0x7f95780b5de0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:28.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.115+0000 7f957dbe2640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95780d0e10 0x7f95780b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:28.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.115+0000 7f957dbe2640 1 --2- 172.21.15.44:0/395421710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95780d0a30 0x7f95781514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:28.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.115+0000 7f957dbe2640 1 -- 172.21.15.44:0/395421710 >> 172.21.15.44:0/395421710 conn(0x7f957809d470 msgr2=0x7f9578160a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:28.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.115+0000 7f957dbe2640 1 -- 172.21.15.44:0/395421710 shutdown_connections 2024-04-02T13:22:28.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28.115+0000 7f957dbe2640 1 -- 172.21.15.44:0/395421710 wait complete. 2024-04-02T13:22:28.128 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:22:28.128 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:22:28.128 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:22:28.128 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:22:28.128 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:22:28.128 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:22:28.128 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:22:28.129 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:22:28.129 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:22:28.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:28.129 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:22:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:22:28.145 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:22:28.145 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:22:28.146 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:22:28.146 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:22:28.146 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:22:28.146 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:22:28.146 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:22:28.146 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:22:28.146 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:22:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:22:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:22:28.147 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:22:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:22:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:22:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:22:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42631sec preferred_lft 42631sec 2024-04-02T13:22:28.148 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:22:28.148 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:22:28.148 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:22:28.148 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:22:28.148 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:22:28.148 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:22:28.148 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:22:28.149 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:22:28.149 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:22:28.149 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:22:28.149 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:22:28.149 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:22:28.149 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:22:28.149 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:22:28.150 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:22:28.150 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:22:28.150 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:22:28.150 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:22:28.150 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:22:28.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:28.151 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:22:28.211 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:22:28.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:28.213 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:22:28.268 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:22:28.268 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:22:28.268 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:22:28.269 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:22:28.269 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:22:28.269 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:22:28.269 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:22:28.269 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:22:28.270 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:22:28.270 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:22:28.270 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:22:28.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:22:28.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:22:28.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:28.404 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:22:28.418 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:22:28.418 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:22:28.418 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:22:28.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:28.419 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:22:28.485 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:22:28.486 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:22:28.486 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:22:28.540 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:22:28.540 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:22:28.540 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:22:28.594 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:22:28.661 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:22:28.717 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:22:28.717 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:22:28.717 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:22:28.717 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:22:28.717 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:22:28.717 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:22:28.717 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:22:28.717 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:22:28.717 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:22:28.718 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:22:28.719 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:22:28.719 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:22:28.719 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:22:28.719 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:22:28.719 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:22:28.719 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:22:28.719 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:22:28.719 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:22:28.719 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:22:28.719 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:22:28.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:28.719 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:22:28.771 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:22:28.772 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:22:28.813 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:22:28.839 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:22:28.895 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:22:28.895 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:22:28.895 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:22:28.896 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:22:28.897 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:22:28.898 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:22:28.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:28.898 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:22:28.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:22:28.897+0000 7f42c6a0c480 -1 init, newargv = 0x5653e5966740 newargc=15 2024-04-02T13:22:28.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[57218]: starting ceph client 2024-04-02T13:22:28.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[57218]: starting fuse 2024-04-02T13:22:28.913 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:22:28.913 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:22:28.913 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:22:28.913 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:22:28.914 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:22:28.915 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:22:28.915 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:22:28.915 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:22:28.915 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:22:28.915 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:22:28.915 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:22:28.915 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:22:28.915 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:22:28.915 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:22:28.915 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:22:29.034 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.57218.asok 2024-04-02T13:22:29.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:22:29.044 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.57218.asok 2024-04-02T13:22:29.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:29.044 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.57218.asok status 2024-04-02T13:22:29.182 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:22:29.182 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:22:29.182 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:22:29.182 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:22:29.182 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:22:29.182 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:22:29.182 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:22:29.182 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "57218", 2024-04-02T13:22:29.182 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:22:29.182 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:29.182 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 6144, 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 6144 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 459113346 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:22:29.183 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:22:29.184 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 459113346 2024-04-02T13:22:29.184 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:29.184 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.6144 192.168.0.1:0/459113346", 2024-04-02T13:22:29.184 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/459113346", 2024-04-02T13:22:29.184 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:22:29.184 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 25, 2024-04-02T13:22:29.184 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 52, 2024-04-02T13:22:29.184 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:22:29.184 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:22:29.184 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:22:29.184 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:22:29.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:29.190 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:22:29.190 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:22:29.204 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:22:29.205 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:22:29.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:29.205 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:22:29.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:29.273 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_a) 2024-04-02T13:22:29.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:29.334 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parent_a/alpha) 2024-04-02T13:22:29.392 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:22:29.392 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:22:29.392 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:22:29.392 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:22:29.392 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/parent_a/alpha").st_ino) 2024-04-02T13:22:29.392 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:22:29.498 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627777 2024-04-02T13:22:29.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:22:29.506 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:22:29.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.646+0000 7f0ed3732640 1 Processor -- start 2024-04-02T13:22:29.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.646+0000 7f0ed3732640 1 -- start start 2024-04-02T13:22:29.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.647+0000 7f0ed3732640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0ecc15c1c0 0x7f0ecc0affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:29.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.647+0000 7f0ed3732640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ecc0b1640 con 0x7f0ecc0b3d70 2024-04-02T13:22:29.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.647+0000 7f0ed3732640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ecc0aef20 con 0x7f0ecc0b0c10 2024-04-02T13:22:29.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.647+0000 7f0ed3732640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0ecc0af200 con 0x7f0ecc15c1c0 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.647+0000 7f0ed0ccd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0ecc0b3d70 0x7f0ecc0b0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:58476/0 (socket says 172.21.15.44:58476) 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.647+0000 7f0ed0ccd640 1 -- 172.21.15.44:0/3129010720 learned_addr learned my addr 172.21.15.44:0/3129010720 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.647+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2105712521 0 0) 0x7f0ec0001110 con 0x7f0ecc0b3d70 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.647+0000 7f0ed14ce640 1 --2- 172.21.15.44:0/3129010720 >> v2:172.21.15.69:3300/0 conn(0x7f0ecc15c1c0 0x7f0ecc0affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.647+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0eb4003520 con 0x7f0ecc0b3d70 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2168591826 0 0) 0x7f0eb4003520 con 0x7f0ecc0b3d70 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 >> v2:172.21.15.69:3300/0 conn(0x7f0ecc15c1c0 msgr2=0x7f0ecc0affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ecbfff640 1 --2- 172.21.15.44:0/3129010720 >> v2:172.21.15.69:3300/0 conn(0x7f0ecc15c1c0 0x7f0ecc0affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 >> v1:172.21.15.69:6789/0 conn(0x7f0ecc0b0c10 legacy=0x7f0ecc165e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ecc0af4e0 con 0x7f0ecc0b3d70 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f0ec0001840 con 0x7f0ecc0b3d70 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 >> v1:172.21.15.44:6789/0 conn(0x7f0ecc0b3d70 legacy=0x7f0ecc0b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ecbfff640 1 --2- 172.21.15.44:0/3129010720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eb40039b0 0x7f0eb4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ecbfff640 1 --2- 172.21.15.44:0/3129010720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0eb4004800 0x7f0eb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ecbfff640 1 --2- 172.21.15.44:0/3129010720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb40051c0 0x7f0eb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0ecc0af200 con 0x7f0eb40039b0 2024-04-02T13:22:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb400c060 con 0x7f0eb4004800 2024-04-02T13:22:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb400c260 con 0x7f0eb40051c0 2024-04-02T13:22:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ed14ce640 1 --2- 172.21.15.44:0/3129010720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eb40039b0 0x7f0eb4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ed0ccd640 1 --2- 172.21.15.44:0/3129010720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0eb4004800 0x7f0eb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ed1ccf640 1 --2- 172.21.15.44:0/3129010720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb40051c0 0x7f0eb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ed14ce640 1 -- 172.21.15.44:0/3129010720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb40051c0 msgr2=0x7f0eb400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ed14ce640 1 --2- 172.21.15.44:0/3129010720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb40051c0 0x7f0eb400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ed14ce640 1 -- 172.21.15.44:0/3129010720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0eb4004800 msgr2=0x7f0eb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ed14ce640 1 --2- 172.21.15.44:0/3129010720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0eb4004800 0x7f0eb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ed14ce640 1 -- 172.21.15.44:0/3129010720 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb4003520 con 0x7f0eb40039b0 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.648+0000 7f0ed1ccf640 1 -- 172.21.15.44:0/3129010720 reap_dead start 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.649+0000 7f0ed14ce640 1 --2- 172.21.15.44:0/3129010720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eb40039b0 0x7f0eb4003dd0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009fc0 tx=0x7f0eb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbbb7325f01f2cdd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.649+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc039070 con 0x7f0eb40039b0 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.649+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ebc02cc20 con 0x7f0eb40039b0 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.649+0000 7f0ecbfff640 1 -- 172.21.15.44:0/3129010720 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc009c10 con 0x7f0eb40039b0 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.649+0000 7f0ed3732640 1 -- 172.21.15.44:0/3129010720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eb40039b0 msgr2=0x7f0eb4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.649+0000 7f0ed3732640 1 --2- 172.21.15.44:0/3129010720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eb40039b0 0x7f0eb4003dd0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009fc0 tx=0x7f0eb4005830 comp rx=0 tx=0).stop 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.650+0000 7f0ed3732640 1 -- 172.21.15.44:0/3129010720 shutdown_connections 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.650+0000 7f0ed3732640 1 --2- 172.21.15.44:0/3129010720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eb40039b0 0x7f0eb4003dd0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.650+0000 7f0ed3732640 1 -- 172.21.15.44:0/3129010720 >> 172.21.15.44:0/3129010720 conn(0x7f0ecc09aa20 msgr2=0x7f0ecc09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.650+0000 7f0ed3732640 1 -- 172.21.15.44:0/3129010720 shutdown_connections 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.650+0000 7f0ed3732640 1 -- 172.21.15.44:0/3129010720 wait complete. 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.651+0000 7f0ed3732640 1 Processor -- start 2024-04-02T13:22:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.651+0000 7f0ed3732640 1 -- start start 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.651+0000 7f0ed3732640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eb40039b0 0x7f0ecc153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.651+0000 7f0ed3732640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ecc0c64f0 0x7f0ecc0cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.651+0000 7f0ed3732640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ecc0ce020 0x7f0ecc1f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.651+0000 7f0ed3732640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0ecc0cc990 con 0x7f0eb40039b0 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.651+0000 7f0ed3732640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0ecc0afaa0 con 0x7f0ecc0c64f0 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.651+0000 7f0ed3732640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0ecc0aebf0 con 0x7f0ecc0ce020 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed14ce640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eb40039b0 0x7f0ecc153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed14ce640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eb40039b0 0x7f0ecc153680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43674/0 (socket says 172.21.15.44:43674) 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed14ce640 1 -- 172.21.15.44:0/317028166 learned_addr learned my addr 172.21.15.44:0/317028166 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed1ccf640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ecc0ce020 0x7f0ecc1f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed0ccd640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ecc0c64f0 0x7f0ecc0cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed14ce640 1 -- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ecc0ce020 msgr2=0x7f0ecc1f5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed14ce640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ecc0ce020 0x7f0ecc1f5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed14ce640 1 -- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ecc0c64f0 msgr2=0x7f0ecc0cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed14ce640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ecc0c64f0 0x7f0ecc0cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed14ce640 1 -- 172.21.15.44:0/317028166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ecc0c7810 con 0x7f0eb40039b0 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed1ccf640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ecc0ce020 0x7f0ecc1f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed14ce640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eb40039b0 0x7f0ecc153680 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc02c690 tx=0x7f0ebc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ec9ffb640 1 -- 172.21.15.44:0/317028166 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc039050 con 0x7f0eb40039b0 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ec9ffb640 1 -- 172.21.15.44:0/317028166 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ebc03a040 con 0x7f0eb40039b0 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed3732640 1 -- 172.21.15.44:0/317028166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ecc0b4d00 con 0x7f0eb40039b0 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ec9ffb640 1 -- 172.21.15.44:0/317028166 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc009c10 con 0x7f0eb40039b0 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.652+0000 7f0ed3732640 1 -- 172.21.15.44:0/317028166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ecc0b52c0 con 0x7f0eb40039b0 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.653+0000 7f0ed3732640 1 -- 172.21.15.44:0/317028166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ecc0b54c0 con 0x7f0eb40039b0 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.654+0000 7f0ec9ffb640 1 -- 172.21.15.44:0/317028166 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ebc047030 con 0x7f0eb40039b0 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.654+0000 7f0ec9ffb640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0eb806c8f0 0x7f0eb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.654+0000 7f0ec9ffb640 1 -- 172.21.15.44:0/317028166 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f0ebc0ae0b0 con 0x7f0eb40039b0 2024-04-02T13:22:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.655+0000 7f0ed0ccd640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0eb806c8f0 0x7f0eb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:29.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.655+0000 7f0ed0ccd640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0eb806c8f0 0x7f0eb806edb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00012b0 tx=0x7f0ec0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:29.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.659+0000 7f0ec9ffb640 1 -- 172.21.15.44:0/317028166 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0ecc0b54c0 con 0x7f0eb40039b0 2024-04-02T13:22:29.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.882+0000 7f0ed3732640 1 -- 172.21.15.44:0/317028166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ecc0684f0 con 0x7f0eb40039b0 2024-04-02T13:22:29.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.882+0000 7f0ec9ffb640 1 -- 172.21.15.44:0/317028166 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5181 (secure 0 0 0) 0x7f0ebc04e060 con 0x7f0eb40039b0 2024-04-02T13:22:29.884 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:29.884 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/4131284806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":4131284806},{"type":"v1","addr":"172.21.15.44:6833","nonce":4131284806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/4262427145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":4262427145},{"type":"v1","addr":"172.21.15.44:6835","nonce":4262427145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4112565894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4112565894},{"type":"v1","addr":"172.21.15.69:6836","nonce":4112565894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:22:22.422387+0000","modified":"2024-04-02T13:22:24.398291+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6837/1745145720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1745145720},{"type":"v1","addr":"172.21.15.69:6837","nonce":1745145720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":3}]} 2024-04-02T13:22:29.884 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-02T13:22:29.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.884+0000 7f0ed3732640 1 -- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0eb806c8f0 msgr2=0x7f0eb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:29.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.884+0000 7f0ed3732640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0eb806c8f0 0x7f0eb806edb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00012b0 tx=0x7f0ec0009040 comp rx=0 tx=0).stop 2024-04-02T13:22:29.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.884+0000 7f0ed3732640 1 -- 172.21.15.44:0/317028166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eb40039b0 msgr2=0x7f0ecc153680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:29.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.884+0000 7f0ed3732640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eb40039b0 0x7f0ecc153680 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc02c690 tx=0x7f0ebc009750 comp rx=0 tx=0).stop 2024-04-02T13:22:29.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.884+0000 7f0ed3732640 1 -- 172.21.15.44:0/317028166 shutdown_connections 2024-04-02T13:22:29.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.884+0000 7f0ed3732640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0eb806c8f0 0x7f0eb806edb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:29.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.884+0000 7f0ed3732640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ecc0ce020 0x7f0ecc1f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:29.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.884+0000 7f0ed3732640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ecc0c64f0 0x7f0ecc0cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:29.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.884+0000 7f0ed3732640 1 --2- 172.21.15.44:0/317028166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eb40039b0 0x7f0ecc153680 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:29.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.884+0000 7f0ed3732640 1 -- 172.21.15.44:0/317028166 >> 172.21.15.44:0/317028166 conn(0x7f0ecc09aa20 msgr2=0x7f0ecc0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:29.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.884+0000 7f0ed3732640 1 -- 172.21.15.44:0/317028166 shutdown_connections 2024-04-02T13:22:29.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:29.884+0000 7f0ed3732640 1 -- 172.21.15.44:0/317028166 wait complete. 2024-04-02T13:22:29.897 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-04-02T13:22:30.038 INFO:teuthology.orchestra.run.smithi069.stdout:{"message":"","return_code":0} 2024-04-02T13:22:30.039 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-02T13:22:30.039 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-04-02T13:22:30.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.079+0000 7f1d86f6ff40 1 Processor -- start 2024-04-02T13:22:30.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.079+0000 7f1d86f6ff40 1 -- start start 2024-04-02T13:22:30.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d86f6ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e3516854e0 0x55e351739d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d86f6ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e351751bb0 0x55e351734860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d86f6ff40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e351757b80 0x55e351683cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d86f6ff40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e35162a3e0 con 0x55e351757b80 2024-04-02T13:22:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d86f6ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e35162e430 con 0x55e351751bb0 2024-04-02T13:22:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d86f6ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55e3516b0a60 con 0x55e3516854e0 2024-04-02T13:22:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d85271640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e351757b80 0x55e351683cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d85271640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e351757b80 0x55e351683cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43686/0 (socket says 172.21.15.44:43686) 2024-04-02T13:22:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d85271640 1 -- 172.21.15.44:0/2644486877 learned_addr learned my addr 172.21.15.44:0/2644486877 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d85271640 1 -- 172.21.15.44:0/2644486877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e3516854e0 msgr2=0x55e351739d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d85271640 1 --2- 172.21.15.44:0/2644486877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e3516854e0 0x55e351739d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d85271640 1 -- 172.21.15.44:0/2644486877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e351751bb0 msgr2=0x55e351734860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d85271640 1 --2- 172.21.15.44:0/2644486877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e351751bb0 0x55e351734860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d85271640 1 -- 172.21.15.44:0/2644486877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e351751370 con 0x55e351757b80 2024-04-02T13:22:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d85271640 1 --2- 172.21.15.44:0/2644486877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e351757b80 0x55e351683cc0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1d74044620 tx=0x7f1d7404c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dcf7e8f8bc00eec server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d8326d640 1 -- 172.21.15.44:0/2644486877 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d74063070 con 0x55e351757b80 2024-04-02T13:22:30.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d8326d640 1 -- 172.21.15.44:0/2644486877 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d740525e0 con 0x55e351757b80 2024-04-02T13:22:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.080+0000 7f1d8326d640 1 -- 172.21.15.44:0/2644486877 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d740529d0 con 0x55e351757b80 2024-04-02T13:22:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.081+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/2644486877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e351757b80 msgr2=0x55e351683cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.081+0000 7f1d86f6ff40 1 --2- 172.21.15.44:0/2644486877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e351757b80 0x55e351683cc0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1d74044620 tx=0x7f1d7404c000 comp rx=0 tx=0).stop 2024-04-02T13:22:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.081+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/2644486877 shutdown_connections 2024-04-02T13:22:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.081+0000 7f1d86f6ff40 1 --2- 172.21.15.44:0/2644486877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e351757b80 0x55e351683cc0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.081+0000 7f1d86f6ff40 1 --2- 172.21.15.44:0/2644486877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e351751bb0 0x55e351734860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.081+0000 7f1d86f6ff40 1 --2- 172.21.15.44:0/2644486877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e3516854e0 0x55e351739d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.081+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/2644486877 >> 172.21.15.44:0/2644486877 conn(0x55e3516afc80 msgr2=0x55e3516ada20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.081+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/2644486877 shutdown_connections 2024-04-02T13:22:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.081+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/2644486877 wait complete. 2024-04-02T13:22:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.082+0000 7f1d86f6ff40 1 Processor -- start 2024-04-02T13:22:30.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.082+0000 7f1d86f6ff40 1 -- start start 2024-04-02T13:22:30.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.082+0000 7f1d86f6ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e3516854e0 0x55e351738fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.082+0000 7f1d86f6ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e351751bb0 0x55e35173b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.082+0000 7f1d86f6ff40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e351757b80 0x55e35174c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.082+0000 7f1d86f6ff40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e3516b2240 con 0x55e351757b80 2024-04-02T13:22:30.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.082+0000 7f1d86f6ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e3516ae2f0 con 0x55e351751bb0 2024-04-02T13:22:30.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.082+0000 7f1d86f6ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55e3516ae540 con 0x55e3516854e0 2024-04-02T13:22:30.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.082+0000 7f1d85271640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e351757b80 0x55e35174c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.082+0000 7f1d85271640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e351757b80 0x55e35174c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43688/0 (socket says 172.21.15.44:43688) 2024-04-02T13:22:30.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.082+0000 7f1d85271640 1 -- 172.21.15.44:0/254736116 learned_addr learned my addr 172.21.15.44:0/254736116 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d8426f640 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e351751bb0 0x55e35173b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d84a70640 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e3516854e0 0x55e351738fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d85271640 1 -- 172.21.15.44:0/254736116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e3516854e0 msgr2=0x55e351738fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d85271640 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e3516854e0 0x55e351738fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d85271640 1 -- 172.21.15.44:0/254736116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e351751bb0 msgr2=0x55e35173b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d85271640 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e351751bb0 0x55e35173b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d85271640 1 -- 172.21.15.44:0/254736116 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3516cc3d0 con 0x55e351757b80 2024-04-02T13:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d8426f640 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e351751bb0 0x55e35173b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d85271640 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e351757b80 0x55e35174c7a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f1d7401cbd0 tx=0x7f1d74003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d8226b640 1 -- 172.21.15.44:0/254736116 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7406d020 con 0x55e351757b80 2024-04-02T13:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d8226b640 1 -- 172.21.15.44:0/254736116 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d7405e670 con 0x55e351757b80 2024-04-02T13:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d8226b640 1 -- 172.21.15.44:0/254736116 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d74063040 con 0x55e351757b80 2024-04-02T13:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/254736116 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e3516cc6b0 con 0x55e351757b80 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.083+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/254736116 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e35173b0f0 con 0x55e351757b80 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.085+0000 7f1d8226b640 1 -- 172.21.15.44:0/254736116 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1d7405ea40 con 0x55e351757b80 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.085+0000 7f1d8226b640 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d7006b300 0x7f1d7006d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.085+0000 7f1d8226b640 1 -- 172.21.15.44:0/254736116 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f1d740cdb30 con 0x55e351757b80 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.085+0000 7f1d84a70640 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d7006b300 0x7f1d7006d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.085+0000 7f1d86f6ff40 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55e3516841b0 0x55e3517f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.085+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/254736116 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:1 8.7 8:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52) -- 0x55e3517f3ea0 con 0x55e3516841b0 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.085+0000 7f1d8426f640 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55e3516841b0 0x55e3517f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d84a70640 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d7006b300 0x7f1d7006d7c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0011e0 tx=0x7f1d7c0013c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d8426f640 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55e3516841b0 0x55e3517f3960 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d8426f640 1 -- 172.21.15.44:0/254736116 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv1 ondisk = 0) ==== 164+0+95 (crc 0 0 0) 0x7f1d7800c500 con 0x55e3516841b0 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/254736116 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55e3516841b0 msgr2=0x55e3517f3960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d86f6ff40 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55e3516841b0 0x55e3517f3960 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/254736116 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d7006b300 msgr2=0x7f1d7006d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d86f6ff40 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d7006b300 0x7f1d7006d7c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0011e0 tx=0x7f1d7c0013c0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/254736116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e351757b80 msgr2=0x55e35174c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d86f6ff40 1 --2- 172.21.15.44:0/254736116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e351757b80 0x55e35174c7a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f1d7401cbd0 tx=0x7f1d74003330 comp rx=0 tx=0).stop 2024-04-02T13:22:30.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d85271640 1 -- 172.21.15.44:0/254736116 reap_dead start 2024-04-02T13:22:30.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/254736116 shutdown_connections 2024-04-02T13:22:30.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/254736116 >> 172.21.15.44:0/254736116 conn(0x55e3516afc80 msgr2=0x55e3516ad9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/254736116 shutdown_connections 2024-04-02T13:22:30.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.086+0000 7f1d86f6ff40 1 -- 172.21.15.44:0/254736116 wait complete. 2024-04-02T13:22:30.093 INFO:teuthology.orchestra.run.smithi044.stdout:Yalphaparent_a 2024-04-02T13:22:30.093 DEBUG:teuthology.orchestra.run.smithi044:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-04-02T13:22:30.225 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:22:30.225 INFO:teuthology.orchestra.run.smithi044.stdout: "ino": 1099511627777, 2024-04-02T13:22:30.225 INFO:teuthology.orchestra.run.smithi044.stdout: "ancestors": [ 2024-04-02T13:22:30.225 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:30.225 INFO:teuthology.orchestra.run.smithi044.stdout: "dirino": 1099511627776, 2024-04-02T13:22:30.226 INFO:teuthology.orchestra.run.smithi044.stdout: "dname": "alpha", 2024-04-02T13:22:30.226 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 2 2024-04-02T13:22:30.226 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:22:30.226 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:22:30.226 INFO:teuthology.orchestra.run.smithi044.stdout: "dirino": 1, 2024-04-02T13:22:30.226 INFO:teuthology.orchestra.run.smithi044.stdout: "dname": "parent_a", 2024-04-02T13:22:30.226 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 4 2024-04-02T13:22:30.226 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:30.226 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-02T13:22:30.226 INFO:teuthology.orchestra.run.smithi044.stdout: "pool": 8, 2024-04-02T13:22:30.226 INFO:teuthology.orchestra.run.smithi044.stdout: "old_pools": [] 2024-04-02T13:22:30.226 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:22:30.226 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:30.233 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'oh i'"'"'m sorry did i overwrite your xattr?' 2024-04-02T13:22:30.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.274+0000 7f8694d7ff40 1 Processor -- start 2024-04-02T13:22:30.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.274+0000 7f8694d7ff40 1 -- start start 2024-04-02T13:22:30.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8694d7ff40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576b4a2eaf0 0x5576b4a9ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8694d7ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5576b4a2f6d0 0x5576b4ab4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8694d7ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576b4a9e3b0 0x5576b4ab4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8694d7ff40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5576b498e3e0 con 0x5576b4a2eaf0 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8694d7ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5576b4992430 con 0x5576b4a9e3b0 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8694d7ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5576b4a2de70 con 0x5576b4a2f6d0 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8692880640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576b4a2eaf0 0x5576b4a9ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8692880640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576b4a2eaf0 0x5576b4a9ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43704/0 (socket says 172.21.15.44:43704) 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8692880640 1 -- 172.21.15.44:0/3244067387 learned_addr learned my addr 172.21.15.44:0/3244067387 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8693081640 1 --2- 172.21.15.44:0/3244067387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576b4a9e3b0 0x5576b4ab4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f869207f640 1 --2- 172.21.15.44:0/3244067387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5576b4a2f6d0 0x5576b4ab4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8692880640 1 -- 172.21.15.44:0/3244067387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5576b4a2f6d0 msgr2=0x5576b4ab4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8692880640 1 --2- 172.21.15.44:0/3244067387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5576b4a2f6d0 0x5576b4ab4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8692880640 1 -- 172.21.15.44:0/3244067387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576b4a9e3b0 msgr2=0x5576b4ab4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8692880640 1 --2- 172.21.15.44:0/3244067387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576b4a9e3b0 0x5576b4ab4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8692880640 1 -- 172.21.15.44:0/3244067387 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576b4ab9200 con 0x5576b4a2eaf0 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.275+0000 7f8692880640 1 --2- 172.21.15.44:0/3244067387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576b4a2eaf0 0x5576b4a9ffa0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f867c00b320 tx=0x7f867c00b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f114bb3d4d017f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.276+0000 7f869107d640 1 -- 172.21.15.44:0/3244067387 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c008070 con 0x5576b4a2eaf0 2024-04-02T13:22:30.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.276+0000 7f869107d640 1 -- 172.21.15.44:0/3244067387 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f867c00d5c0 con 0x5576b4a2eaf0 2024-04-02T13:22:30.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.276+0000 7f869107d640 1 -- 172.21.15.44:0/3244067387 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c00d9b0 con 0x5576b4a2eaf0 2024-04-02T13:22:30.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.276+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3244067387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576b4a2eaf0 msgr2=0x5576b4a9ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.276+0000 7f8694d7ff40 1 --2- 172.21.15.44:0/3244067387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576b4a2eaf0 0x5576b4a9ffa0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f867c00b320 tx=0x7f867c00b7e0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.276+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3244067387 shutdown_connections 2024-04-02T13:22:30.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.276+0000 7f8694d7ff40 1 --2- 172.21.15.44:0/3244067387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576b4a9e3b0 0x5576b4ab4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.276+0000 7f8694d7ff40 1 --2- 172.21.15.44:0/3244067387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5576b4a2f6d0 0x5576b4ab4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.276+0000 7f8694d7ff40 1 --2- 172.21.15.44:0/3244067387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576b4a2eaf0 0x5576b4a9ffa0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.276+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3244067387 >> 172.21.15.44:0/3244067387 conn(0x5576b49e62d0 msgr2=0x5576b4aa09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.276+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3244067387 shutdown_connections 2024-04-02T13:22:30.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.276+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3244067387 wait complete. 2024-04-02T13:22:30.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.277+0000 7f8694d7ff40 1 Processor -- start 2024-04-02T13:22:30.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.277+0000 7f8694d7ff40 1 -- start start 2024-04-02T13:22:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.277+0000 7f8694d7ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576b4a2eaf0 0x5576b4ab35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.277+0000 7f8694d7ff40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576b4a2f6d0 0x5576b4a98fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.277+0000 7f8694d7ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5576b4a9e3b0 0x5576b4aa0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.277+0000 7f8694d7ff40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5576b4aa1980 con 0x5576b4a2f6d0 2024-04-02T13:22:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.277+0000 7f8694d7ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5576b4ab54c0 con 0x5576b4a2eaf0 2024-04-02T13:22:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.277+0000 7f8694d7ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5576b4ab94a0 con 0x5576b4a9e3b0 2024-04-02T13:22:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.277+0000 7f869207f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576b4a2f6d0 0x5576b4a98fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.277+0000 7f869207f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576b4a2f6d0 0x5576b4a98fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43706/0 (socket says 172.21.15.44:43706) 2024-04-02T13:22:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.277+0000 7f869207f640 1 -- 172.21.15.44:0/3199056142 learned_addr learned my addr 172.21.15.44:0/3199056142 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.278+0000 7f8693081640 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5576b4a9e3b0 0x5576b4aa0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.278+0000 7f869207f640 1 -- 172.21.15.44:0/3199056142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5576b4a9e3b0 msgr2=0x5576b4aa0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.278+0000 7f8692880640 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576b4a2eaf0 0x5576b4ab35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.278+0000 7f869207f640 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5576b4a9e3b0 0x5576b4aa0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.278+0000 7f869207f640 1 -- 172.21.15.44:0/3199056142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576b4a2eaf0 msgr2=0x5576b4ab35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.278+0000 7f869207f640 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576b4a2eaf0 0x5576b4ab35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.278+0000 7f869207f640 1 -- 172.21.15.44:0/3199056142 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576b4b4fd40 con 0x5576b4a2f6d0 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.278+0000 7f869207f640 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576b4a2f6d0 0x5576b4a98fa0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f8684055040 tx=0x7f8684002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.278+0000 7f868bfff640 1 -- 172.21.15.44:0/3199056142 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8684062020 con 0x5576b4a2f6d0 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.278+0000 7f868bfff640 1 -- 172.21.15.44:0/3199056142 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8684053bc0 con 0x5576b4a2f6d0 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.278+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3199056142 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5576b4b50c10 con 0x5576b4a2f6d0 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.278+0000 7f868bfff640 1 -- 172.21.15.44:0/3199056142 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8684057040 con 0x5576b4a2f6d0 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.278+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3199056142 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576b4ab42f0 con 0x5576b4a2f6d0 2024-04-02T13:22:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.279+0000 7f868bfff640 1 -- 172.21.15.44:0/3199056142 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8684065020 con 0x5576b4a2f6d0 2024-04-02T13:22:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.280+0000 7f868bfff640 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f867406b2d0 0x7f867406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.280+0000 7f868bfff640 1 -- 172.21.15.44:0/3199056142 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f86840cc960 con 0x5576b4a2f6d0 2024-04-02T13:22:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.280+0000 7f8694d7ff40 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5576b4a994e0 0x5576b4b623f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.280+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3199056142 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:1 8.7 8:e2e07930:::10000000001.00000000:head [setxattr parent (40) in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52) -- 0x5576b4b50350 con 0x5576b4a994e0 2024-04-02T13:22:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.280+0000 7f8692880640 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f867406b2d0 0x7f867406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.280+0000 7f8693081640 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5576b4a994e0 0x5576b4b623f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.281+0000 7f8693081640 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5576b4a994e0 0x5576b4b623f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.281+0000 7f8692880640 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f867406b2d0 0x7f867406d790 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f867c00b210 tx=0x7f867c009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.282+0000 7f8693081640 1 -- 172.21.15.44:0/3199056142 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (40)] v52'2 uv2 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f868c0026b0 con 0x5576b4a994e0 2024-04-02T13:22:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.282+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3199056142 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5576b4a994e0 msgr2=0x5576b4b623f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.282+0000 7f8694d7ff40 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5576b4a994e0 0x5576b4b623f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.282+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3199056142 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f867406b2d0 msgr2=0x7f867406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.282+0000 7f8694d7ff40 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f867406b2d0 0x7f867406d790 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f867c00b210 tx=0x7f867c009000 comp rx=0 tx=0).stop 2024-04-02T13:22:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.282+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3199056142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576b4a2f6d0 msgr2=0x5576b4a98fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.282+0000 7f8694d7ff40 1 --2- 172.21.15.44:0/3199056142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576b4a2f6d0 0x5576b4a98fa0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f8684055040 tx=0x7f8684002fb0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.282+0000 7f8693081640 1 -- 172.21.15.44:0/3199056142 reap_dead start 2024-04-02T13:22:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.282+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3199056142 shutdown_connections 2024-04-02T13:22:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.282+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3199056142 >> 172.21.15.44:0/3199056142 conn(0x5576b49e62d0 msgr2=0x5576b4ab4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.282+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3199056142 shutdown_connections 2024-04-02T13:22:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.282+0000 7f8694d7ff40 1 -- 172.21.15.44:0/3199056142 wait complete. 2024-04-02T13:22:30.288 DEBUG:teuthology.orchestra.run.smithi044:> sudo daemon-helper kill ceph --cluster ceph -w 2024-04-02T13:22:30.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:30.331 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub start / repair,recursive 2024-04-02T13:22:30.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.435+0000 7f4e5d8d9640 1 Processor -- start 2024-04-02T13:22:30.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.435+0000 7f4e5d8d9640 1 -- start start 2024-04-02T13:22:30.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.435+0000 7f4e5d8d9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4e5815e1d0 0x7f4e580d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.435+0000 7f4e5d8d9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4e580d2dd0 0x7f4e580b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.435+0000 7f4e5d8d9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e580d1640 con 0x7f4e580cec60 2024-04-02T13:22:30.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.435+0000 7f4e5d8d9640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4e580cfe00 con 0x7f4e5815e1d0 2024-04-02T13:22:30.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.435+0000 7f4e5d8d9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4e580cf1f0 con 0x7f4e580d2dd0 2024-04-02T13:22:30.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.436+0000 7f4e56ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4e5815e1d0 0x7f4e580d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.436+0000 7f4e4ffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4e580cec60 0x7f4e580d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:58500/0 (socket says 172.21.15.44:58500) 2024-04-02T13:22:30.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.436+0000 7f4e4ffff640 1 -- 172.21.15.44:0/3067189490 learned_addr learned my addr 172.21.15.44:0/3067189490 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:30.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.436+0000 7f4e577fe640 1 --2- 172.21.15.44:0/3067189490 >> v2:172.21.15.69:3300/0 conn(0x7f4e580d2dd0 0x7f4e580b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.436+0000 7f4e56ffd640 1 -- 172.21.15.44:0/3067189490 >> v2:172.21.15.69:3300/0 conn(0x7f4e580d2dd0 msgr2=0x7f4e580b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.436+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/3067189490 >> v2:172.21.15.69:3300/0 conn(0x7f4e580d2dd0 0x7f4e580b4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.436+0000 7f4e56ffd640 1 -- 172.21.15.44:0/3067189490 >> v1:172.21.15.44:6789/0 conn(0x7f4e580cec60 legacy=0x7f4e580d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.436+0000 7f4e56ffd640 1 -- 172.21.15.44:0/3067189490 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e580cf4d0 con 0x7f4e5815e1d0 2024-04-02T13:22:30.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.436+0000 7f4e567fc640 1 -- 172.21.15.44:0/3067189490 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3037334553 0 0) 0x7f4e580d1640 con 0x7f4e580cec60 2024-04-02T13:22:30.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.437+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/3067189490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e5815e1d0 0x7f4e580d1100 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f4e580d4080 tx=0x7f4e4802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aaf2c853a6cc40 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.437+0000 7f4e567fc640 1 -- 172.21.15.44:0/3067189490 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e480315a0 con 0x7f4e5815e1d0 2024-04-02T13:22:30.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.437+0000 7f4e567fc640 1 -- 172.21.15.44:0/3067189490 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e48037070 con 0x7f4e5815e1d0 2024-04-02T13:22:30.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.437+0000 7f4e567fc640 1 -- 172.21.15.44:0/3067189490 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e48031960 con 0x7f4e5815e1d0 2024-04-02T13:22:30.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.438+0000 7f4e5d8d9640 1 -- 172.21.15.44:0/3067189490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e5815e1d0 msgr2=0x7f4e580d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.438+0000 7f4e5d8d9640 1 --2- 172.21.15.44:0/3067189490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e5815e1d0 0x7f4e580d1100 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f4e580d4080 tx=0x7f4e4802b840 comp rx=0 tx=0).stop 2024-04-02T13:22:30.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.438+0000 7f4e5d8d9640 1 -- 172.21.15.44:0/3067189490 shutdown_connections 2024-04-02T13:22:30.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.438+0000 7f4e5d8d9640 1 --2- 172.21.15.44:0/3067189490 >> v2:172.21.15.69:3300/0 conn(0x7f4e580d2dd0 0x7f4e580b4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.438+0000 7f4e5d8d9640 1 --2- 172.21.15.44:0/3067189490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e5815e1d0 0x7f4e580d1100 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.438+0000 7f4e5d8d9640 1 -- 172.21.15.44:0/3067189490 >> 172.21.15.44:0/3067189490 conn(0x7f4e5809d3b0 msgr2=0x7f4e580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.438+0000 7f4e5d8d9640 1 -- 172.21.15.44:0/3067189490 shutdown_connections 2024-04-02T13:22:30.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.438+0000 7f4e5d8d9640 1 -- 172.21.15.44:0/3067189490 wait complete. 2024-04-02T13:22:30.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.439+0000 7f4e5d8d9640 1 Processor -- start 2024-04-02T13:22:30.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.439+0000 7f4e5d8d9640 1 -- start start 2024-04-02T13:22:30.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e5d8d9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e580cec60 0x7f4e581f41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e5d8d9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e580d2dd0 0x7f4e58150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e5d8d9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e5815e1d0 0x7f4e581505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e5d8d9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5815e5b0 con 0x7f4e580cec60 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e5d8d9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e580cfa90 con 0x7f4e580d2dd0 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e5d8d9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4e5815e730 con 0x7f4e5815e1d0 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e580cec60 0x7f4e581f41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e580cec60 0x7f4e581f41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43718/0 (socket says 172.21.15.44:43718) 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e56ffd640 1 -- 172.21.15.44:0/624998728 learned_addr learned my addr 172.21.15.44:0/624998728 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e4ffff640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e580d2dd0 0x7f4e58150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e577fe640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e5815e1d0 0x7f4e581505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e56ffd640 1 -- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e5815e1d0 msgr2=0x7f4e581505c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e5815e1d0 0x7f4e581505c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e56ffd640 1 -- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e580d2dd0 msgr2=0x7f4e58150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e580d2dd0 0x7f4e58150080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.440+0000 7f4e56ffd640 1 -- 172.21.15.44:0/624998728 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e580a1d00 con 0x7f4e580cec60 2024-04-02T13:22:30.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.441+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e580cec60 0x7f4e581f41d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4e4802be50 tx=0x7f4e480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.441+0000 7f4e54ff9640 1 -- 172.21.15.44:0/624998728 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e48032630 con 0x7f4e580cec60 2024-04-02T13:22:30.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.441+0000 7f4e54ff9640 1 -- 172.21.15.44:0/624998728 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e480327d0 con 0x7f4e580cec60 2024-04-02T13:22:30.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.441+0000 7f4e5d8d9640 1 -- 172.21.15.44:0/624998728 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e580a1460 con 0x7f4e580cec60 2024-04-02T13:22:30.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.441+0000 7f4e54ff9640 1 -- 172.21.15.44:0/624998728 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e48032bc0 con 0x7f4e580cec60 2024-04-02T13:22:30.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.441+0000 7f4e5d8d9640 1 -- 172.21.15.44:0/624998728 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e580a1a20 con 0x7f4e580cec60 2024-04-02T13:22:30.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.443+0000 7f4e54ff9640 1 -- 172.21.15.44:0/624998728 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4e4804c020 con 0x7f4e580cec60 2024-04-02T13:22:30.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.443+0000 7f4e54ff9640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e3806c8f0 0x7f4e3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.443+0000 7f4e54ff9640 1 -- 172.21.15.44:0/624998728 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f4e4802f020 con 0x7f4e580cec60 2024-04-02T13:22:30.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.443+0000 7f4e4ffff640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e3806c8f0 0x7f4e3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.444+0000 7f4e605a2740 1 Processor -- start 2024-04-02T13:22:30.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.444+0000 7f4e605a2740 1 -- start start 2024-04-02T13:22:30.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.444+0000 7f4e4ffff640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e3806c8f0 0x7f4e3806edb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4e40007fd0 tx=0x7f4e40007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e605a2740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb7c310 0x55a85fb7c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e605a2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4c330 0x55a85fb79580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e605a2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb4fe90 0x55a85fb79ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e605a2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a85fab6770 con 0x55a85fb4c330 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e605a2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a85fa5e960 con 0x55a85fb4fe90 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e605a2740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a85fab28c0 con 0x55a85fb7c310 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e4ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4c330 0x55a85fb79580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e4ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4c330 0x55a85fb79580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43732/0 (socket says 172.21.15.44:43732) 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e4ffff640 1 -- 172.21.15.44:0/779077305 learned_addr learned my addr 172.21.15.44:0/779077305 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/779077305 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb7c310 0x55a85fb7c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e577fe640 1 --2- 172.21.15.44:0/779077305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb4fe90 0x55a85fb79ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e4ffff640 1 -- 172.21.15.44:0/779077305 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb7c310 msgr2=0x55a85fb7c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e4ffff640 1 --2- 172.21.15.44:0/779077305 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb7c310 0x55a85fb7c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e4ffff640 1 -- 172.21.15.44:0/779077305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb4fe90 msgr2=0x55a85fb79ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e4ffff640 1 --2- 172.21.15.44:0/779077305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb4fe90 0x55a85fb79ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e4ffff640 1 -- 172.21.15.44:0/779077305 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a85f8927e0 con 0x55a85fb4c330 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e577fe640 1 --2- 172.21.15.44:0/779077305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb4fe90 0x55a85fb79ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:22:30.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e4ffff640 1 --2- 172.21.15.44:0/779077305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4c330 0x55a85fb79580 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4e400077e0 tx=0x7f4e40013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd499e5d6369347 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e4d7fa640 1 -- 172.21.15.44:0/779077305 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e40026070 con 0x55a85fb4c330 2024-04-02T13:22:30.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e4d7fa640 1 -- 172.21.15.44:0/779077305 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e40001cf0 con 0x55a85fb4c330 2024-04-02T13:22:30.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.445+0000 7f4e4d7fa640 1 -- 172.21.15.44:0/779077305 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e40013d70 con 0x55a85fb4c330 2024-04-02T13:22:30.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.446+0000 7f4e605a2740 1 -- 172.21.15.44:0/779077305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4c330 msgr2=0x55a85fb79580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.446+0000 7f4e605a2740 1 --2- 172.21.15.44:0/779077305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4c330 0x55a85fb79580 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4e400077e0 tx=0x7f4e40013000 comp rx=0 tx=0).stop 2024-04-02T13:22:30.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.446+0000 7f4e605a2740 1 -- 172.21.15.44:0/779077305 shutdown_connections 2024-04-02T13:22:30.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.446+0000 7f4e605a2740 1 --2- 172.21.15.44:0/779077305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb4fe90 0x55a85fb79ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.446+0000 7f4e605a2740 1 --2- 172.21.15.44:0/779077305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4c330 0x55a85fb79580 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.446+0000 7f4e605a2740 1 --2- 172.21.15.44:0/779077305 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb7c310 0x55a85fb7c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.446+0000 7f4e605a2740 1 -- 172.21.15.44:0/779077305 >> 172.21.15.44:0/779077305 conn(0x55a85fb4e5a0 msgr2=0x55a85fb97120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.446+0000 7f4e605a2740 1 -- 172.21.15.44:0/779077305 shutdown_connections 2024-04-02T13:22:30.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.446+0000 7f4e605a2740 1 -- 172.21.15.44:0/779077305 wait complete. 2024-04-02T13:22:30.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.447+0000 7f4e605a2740 1 -- start start 2024-04-02T13:22:30.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.447+0000 7f4e605a2740 1 Processor -- start 2024-04-02T13:22:30.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e605a2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 0x55a85fbaaf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e605a2740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb7c310 0x55a85fb822f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e605a2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb82830 0x55a85fb83940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e605a2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a85fb51040 con 0x55a85fb4fe90 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e605a2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a85fa71d20 con 0x55a85fb82830 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e605a2740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a85fb50430 con 0x55a85fb7c310 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e605a2740 10 client.0 _ll_register_callbacks cb 0x55a85fa3ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 0x55a85fbaaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 0x55a85fbaaf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43738/0 (socket says 172.21.15.44:43738) 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e56ffd640 1 -- 172.21.15.44:0/1032700539 learned_addr learned my addr 172.21.15.44:0/1032700539 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e577fe640 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb82830 0x55a85fb83940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e4ffff640 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb7c310 0x55a85fb822f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e56ffd640 1 -- 172.21.15.44:0/1032700539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb7c310 msgr2=0x55a85fb822f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb7c310 0x55a85fb822f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e56ffd640 1 -- 172.21.15.44:0/1032700539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb82830 msgr2=0x55a85fb83940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb82830 0x55a85fb83940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e56ffd640 1 -- 172.21.15.44:0/1032700539 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a85f8f2f70 con 0x55a85fb4fe90 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e577fe640 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb82830 0x55a85fb83940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 0x55a85fbaaf40 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f4e48000c00 tx=0x7f4e4807b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a34478ef3bb8aed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e367fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e367fc640 1 -- 172.21.15.44:0/1032700539 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e48083fe0 con 0x55a85fb4fe90 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e367fc640 1 -- 172.21.15.44:0/1032700539 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f4e4807b710 con 0x55a85fb4fe90 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.448+0000 7f4e605a2740 1 -- 172.21.15.44:0/1032700539 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e40001cf0 con 0x55a85fb4fe90 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e367fc640 1 -- 172.21.15.44:0/1032700539 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4e4807b910 con 0x55a85fb4fe90 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e605a2740 10 client.6189 fetch_fsmap learned FSMap version 25 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e605a2740 1 -- 172.21.15.44:0/1032700539 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55a85f78a170 con 0x55a85fb4fe90 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e367fc640 1 -- 172.21.15.44:0/1032700539 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e40001cf0 con 0x55a85fb4fe90 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e367fc640 1 -- 172.21.15.44:0/1032700539 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f4e48031070 con 0x55a85fb4fe90 2024-04-02T13:22:30.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e605a2740 10 client.6189 fetch_fsmap finished waiting for FSMap version 25 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e605a2740 10 client.6189 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e605a2740 20 client.6189 populate_metadata read hostname 'smithi044' 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e605a2740 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55a85fba8000 0x55a85fb7f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e605a2740 4 client.6189 mds_command: new command op to 5991 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e605a2740 1 -- 172.21.15.44:0/1032700539 --> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a85fac3cb0 con 0x55a85fba8000 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e4ffff640 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55a85fba8000 0x55a85fb7f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e4ffff640 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55a85fba8000 0x55a85fb7f530 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98b339d2da68182b server_cookie=fe03200a06503794 in_seq=0 out_seq=0 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.449+0000 7f4e367fc640 10 client.6189 ms_handle_connect on v2:172.21.15.69:6835/1745145720 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e367fc640 1 -- 172.21.15.44:0/1032700539 <== mds.0 v2:172.21.15.69:6835/1745145720 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a85fac3cb0 con 0x55a85fba8000 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e367fc640 10 client.6189 handle_command_reply: tid=0 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e605a2740 1 client.6189 shutdown 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e605a2740 1 -- 172.21.15.44:0/1032700539 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 msgr2=0x55a85fbaaf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e605a2740 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 0x55a85fbaaf40 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f4e48000c00 tx=0x7f4e4807b4b0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e605a2740 1 -- 172.21.15.44:0/1032700539 shutdown_connections 2024-04-02T13:22:30.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e605a2740 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55a85fba8000 0x55a85fb7f530 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e605a2740 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb82830 0x55a85fb83940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e605a2740 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb7c310 0x55a85fb822f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e367fc640 0 client.6189 ms_handle_reset on v2:172.21.15.69:6835/1745145720 2024-04-02T13:22:30.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e605a2740 1 --2- 172.21.15.44:0/1032700539 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 0x55a85fbaaf40 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e605a2740 1 -- 172.21.15.44:0/1032700539 >> 172.21.15.44:0/1032700539 conn(0x55a85fb4c330 msgr2=0x55a85fb9a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e605a2740 1 -- 172.21.15.44:0/1032700539 shutdown_connections 2024-04-02T13:22:30.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e605a2740 1 -- 172.21.15.44:0/1032700539 wait complete. 2024-04-02T13:22:30.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.450+0000 7f4e605a2740 20 client.6189 trim_cache size 0 max 16384 2024-04-02T13:22:30.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.456+0000 7f40e571c640 1 Processor -- start 2024-04-02T13:22:30.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.456+0000 7f40e571c640 1 -- start start 2024-04-02T13:22:30.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.456+0000 7f40e571c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f40e00d0a10 0x7f40e00d0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.456+0000 7f40e571c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e00cf8c0 con 0x7f40e00d22b0 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.456+0000 7f40e571c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e00cfba0 con 0x7f40e0161920 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.456+0000 7f40e571c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f40e00d13b0 con 0x7f40e00d0a10 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.457+0000 7f40d67fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f40e00d0a10 0x7f40e00d0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.457+0000 7f40d67fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f40e00d0a10 0x7f40e00d0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43742/0 (socket says 172.21.15.44:43742) 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.457+0000 7f40d67fc640 1 -- 172.21.15.44:0/3648445283 learned_addr learned my addr 172.21.15.44:0/3648445283 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.457+0000 7f40d67fc640 1 -- 172.21.15.44:0/3648445283 >> v1:172.21.15.69:6789/0 conn(0x7f40e0161920 legacy=0x7f40e00cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.457+0000 7f40d67fc640 1 -- 172.21.15.44:0/3648445283 >> v1:172.21.15.44:6789/0 conn(0x7f40e00d22b0 legacy=0x7f40e00d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.457+0000 7f40d67fc640 1 -- 172.21.15.44:0/3648445283 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e00cfe80 con 0x7f40e00d0a10 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.458+0000 7f40d67fc640 1 --2- 172.21.15.44:0/3648445283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40e00d0a10 0x7f40e00d0e70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f40c8009870 tx=0x7f40c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0a0c4f9535f385 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.458+0000 7f40de7fc640 1 -- 172.21.15.44:0/3648445283 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c8034070 con 0x7f40e00d0a10 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.458+0000 7f40de7fc640 1 -- 172.21.15.44:0/3648445283 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c8030860 con 0x7f40e00d0a10 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.458+0000 7f40de7fc640 1 -- 172.21.15.44:0/3648445283 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c8030c30 con 0x7f40e00d0a10 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.458+0000 7f40e571c640 1 -- 172.21.15.44:0/3648445283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40e00d0a10 msgr2=0x7f40e00d0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.458+0000 7f40e571c640 1 --2- 172.21.15.44:0/3648445283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40e00d0a10 0x7f40e00d0e70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f40c8009870 tx=0x7f40c802b9a0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.458+0000 7f40e571c640 1 -- 172.21.15.44:0/3648445283 shutdown_connections 2024-04-02T13:22:30.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.458+0000 7f40e571c640 1 --2- 172.21.15.44:0/3648445283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40e00d0a10 0x7f40e00d0e70 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.458+0000 7f40e571c640 1 -- 172.21.15.44:0/3648445283 >> 172.21.15.44:0/3648445283 conn(0x7f40e009d470 msgr2=0x7f40e00d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.458+0000 7f40e571c640 1 -- 172.21.15.44:0/3648445283 shutdown_connections 2024-04-02T13:22:30.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.458+0000 7f40e571c640 1 -- 172.21.15.44:0/3648445283 wait complete. 2024-04-02T13:22:30.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.460+0000 7f40e571c640 1 Processor -- start 2024-04-02T13:22:30.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.460+0000 7f40e571c640 1 -- start start 2024-04-02T13:22:30.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.460+0000 7f40e571c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e00d0a10 0x7f40e0153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.460+0000 7f40e571c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40e00d22b0 0x7f40e00cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.460+0000 7f40e571c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40e0161920 0x7f40e00cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.460+0000 7f40e571c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f40e00d0160 con 0x7f40e0161920 2024-04-02T13:22:30.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.460+0000 7f40e571c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f40e00d1e20 con 0x7f40e00d22b0 2024-04-02T13:22:30.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.460+0000 7f40e571c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f40e0161d00 con 0x7f40e00d0a10 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40e0161920 0x7f40e00cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40e0161920 0x7f40e00cbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43744/0 (socket says 172.21.15.44:43744) 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40df7fe640 1 -- 172.21.15.44:0/559431608 learned_addr learned my addr 172.21.15.44:0/559431608 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40d67fc640 1 --2- 172.21.15.44:0/559431608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40e00d22b0 0x7f40e00cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40deffd640 1 --2- 172.21.15.44:0/559431608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e00d0a10 0x7f40e0153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40df7fe640 1 -- 172.21.15.44:0/559431608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e00d0a10 msgr2=0x7f40e0153160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40df7fe640 1 --2- 172.21.15.44:0/559431608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e00d0a10 0x7f40e0153160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40df7fe640 1 -- 172.21.15.44:0/559431608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40e00d22b0 msgr2=0x7f40e00cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40df7fe640 1 --2- 172.21.15.44:0/559431608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40e00d22b0 0x7f40e00cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40df7fe640 1 -- 172.21.15.44:0/559431608 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e00cdd60 con 0x7f40e0161920 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40deffd640 1 --2- 172.21.15.44:0/559431608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e00d0a10 0x7f40e0153160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40df7fe640 1 --2- 172.21.15.44:0/559431608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40e0161920 0x7f40e00cbbc0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f40d800c7d0 tx=0x7f40d800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40d7fff640 1 -- 172.21.15.44:0/559431608 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d8003b70 con 0x7f40e0161920 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40d7fff640 1 -- 172.21.15.44:0/559431608 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d8003d30 con 0x7f40e0161920 2024-04-02T13:22:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40e571c640 1 -- 172.21.15.44:0/559431608 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e00ce040 con 0x7f40e0161920 2024-04-02T13:22:30.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40d7fff640 1 -- 172.21.15.44:0/559431608 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d800aa90 con 0x7f40e0161920 2024-04-02T13:22:30.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.461+0000 7f40e571c640 1 -- 172.21.15.44:0/559431608 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e015e300 con 0x7f40e0161920 2024-04-02T13:22:30.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.462+0000 7f40e571c640 1 -- 172.21.15.44:0/559431608 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f40e00ca650 con 0x7f40e0161920 2024-04-02T13:22:30.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.463+0000 7f40d7fff640 1 -- 172.21.15.44:0/559431608 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f40d800ac30 con 0x7f40e0161920 2024-04-02T13:22:30.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.464+0000 7f40d7fff640 1 --2- 172.21.15.44:0/559431608 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f40c006c920 0x7f40c006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.464+0000 7f40d7fff640 1 -- 172.21.15.44:0/559431608 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f40d808c4b0 con 0x7f40e0161920 2024-04-02T13:22:30.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.464+0000 7f40deffd640 1 --2- 172.21.15.44:0/559431608 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f40c006c920 0x7f40c006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.464+0000 7f40d7fff640 1 -- 172.21.15.44:0/559431608 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+501 (secure 0 0 0) 0x7f40d808c960 con 0x7f40e0161920 2024-04-02T13:22:30.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.464+0000 7f40e571c640 1 -- 172.21.15.44:0/559431608 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7f40e00684f0 con 0x7f40e0161920 2024-04-02T13:22:30.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.464+0000 7f40deffd640 1 --2- 172.21.15.44:0/559431608 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f40c006c920 0x7f40c006ede0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f40d0006fd0 tx=0x7f40d0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.470+0000 7f4e605a2740 1 Processor -- start 2024-04-02T13:22:30.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.470+0000 7f4e605a2740 1 -- start start 2024-04-02T13:22:30.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e605a2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 0x55a85fb76f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e605a2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb7c310 0x55a85fb9a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e605a2740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb82830 0x55a85fb9ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e605a2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4e58200c70 con 0x55a85fb4fe90 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e605a2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581ffcb0 con 0x55a85fb7c310 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e605a2740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4e5815c720 con 0x55a85fb82830 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 0x55a85fb76f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 0x55a85fb76f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43750/0 (socket says 172.21.15.44:43750) 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e56ffd640 1 -- 172.21.15.44:0/3702375351 learned_addr learned my addr 172.21.15.44:0/3702375351 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e4ffff640 1 --2- 172.21.15.44:0/3702375351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb7c310 0x55a85fb9a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e577fe640 1 --2- 172.21.15.44:0/3702375351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb82830 0x55a85fb9ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e56ffd640 1 -- 172.21.15.44:0/3702375351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb82830 msgr2=0x55a85fb9ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/3702375351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb82830 0x55a85fb9ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e56ffd640 1 -- 172.21.15.44:0/3702375351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb7c310 msgr2=0x55a85fb9a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/3702375351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb7c310 0x55a85fb9a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e56ffd640 1 -- 172.21.15.44:0/3702375351 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a85fb81570 con 0x55a85fb4fe90 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/3702375351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 0x55a85fb76f60 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4e4802b880 tx=0x7f4e48050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74663aba759c290d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e4cff9640 1 -- 172.21.15.44:0/3702375351 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e48083fe0 con 0x55a85fb4fe90 2024-04-02T13:22:30.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e4cff9640 1 -- 172.21.15.44:0/3702375351 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e48083da0 con 0x55a85fb4fe90 2024-04-02T13:22:30.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.471+0000 7f4e4cff9640 1 -- 172.21.15.44:0/3702375351 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4807aff0 con 0x55a85fb4fe90 2024-04-02T13:22:30.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.472+0000 7f4e605a2740 1 -- 172.21.15.44:0/3702375351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 msgr2=0x55a85fb76f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.472+0000 7f4e605a2740 1 --2- 172.21.15.44:0/3702375351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 0x55a85fb76f60 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4e4802b880 tx=0x7f4e48050000 comp rx=0 tx=0).stop 2024-04-02T13:22:30.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.472+0000 7f4e605a2740 1 -- 172.21.15.44:0/3702375351 shutdown_connections 2024-04-02T13:22:30.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.472+0000 7f4e605a2740 1 --2- 172.21.15.44:0/3702375351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb82830 0x55a85fb9ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.472+0000 7f4e605a2740 1 --2- 172.21.15.44:0/3702375351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb7c310 0x55a85fb9a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.472+0000 7f4e605a2740 1 --2- 172.21.15.44:0/3702375351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb4fe90 0x55a85fb76f60 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.472+0000 7f4e605a2740 1 -- 172.21.15.44:0/3702375351 >> 172.21.15.44:0/3702375351 conn(0x55a85fb4c330 msgr2=0x55a85fba7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.472+0000 7f4e605a2740 1 -- 172.21.15.44:0/3702375351 shutdown_connections 2024-04-02T13:22:30.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.472+0000 7f4e605a2740 1 -- 172.21.15.44:0/3702375351 wait complete. 2024-04-02T13:22:30.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.473+0000 7f4e605a2740 1 -- start start 2024-04-02T13:22:30.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.473+0000 7f4e605a2740 1 Processor -- start 2024-04-02T13:22:30.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.473+0000 7f4e605a2740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb4fe90 0x55a85fbf7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.473+0000 7f4e605a2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb7c310 0x55a85fb83200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.473+0000 7f4e605a2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb82830 0x55a85fb9a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.473+0000 7f4e605a2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a85fb778b0 con 0x55a85fb82830 2024-04-02T13:22:30.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e605a2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5815cd70 con 0x55a85fb7c310 2024-04-02T13:22:30.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e605a2740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a85fb7fdc0 con 0x55a85fb4fe90 2024-04-02T13:22:30.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e605a2740 10 client.0 _ll_register_callbacks cb 0x7f4e48009150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:22:30.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb82830 0x55a85fb9a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb82830 0x55a85fb9a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43758/0 (socket says 172.21.15.44:43758) 2024-04-02T13:22:30.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e577fe640 1 -- 172.21.15.44:0/3486808625 learned_addr learned my addr 172.21.15.44:0/3486808625 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/3486808625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb4fe90 0x55a85fbf7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e4ffff640 1 --2- 172.21.15.44:0/3486808625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb7c310 0x55a85fb83200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e577fe640 1 -- 172.21.15.44:0/3486808625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb4fe90 msgr2=0x55a85fbf7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e577fe640 1 --2- 172.21.15.44:0/3486808625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb4fe90 0x55a85fbf7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e577fe640 1 -- 172.21.15.44:0/3486808625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb7c310 msgr2=0x55a85fb83200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e577fe640 1 --2- 172.21.15.44:0/3486808625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb7c310 0x55a85fb83200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e577fe640 1 -- 172.21.15.44:0/3486808625 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a85fb95ca0 con 0x55a85fb82830 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e577fe640 1 --2- 172.21.15.44:0/3486808625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb82830 0x55a85fb9a630 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4e500045b0 tx=0x7f4e50014a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e402e64b673073 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e367fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e367fc640 1 -- 172.21.15.44:0/3486808625 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e50025060 con 0x55a85fb82830 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e367fc640 1 -- 172.21.15.44:0/3486808625 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f4e50020410 con 0x55a85fb82830 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.474+0000 7f4e605a2740 1 -- 172.21.15.44:0/3486808625 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e48083da0 con 0x55a85fb82830 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e367fc640 1 -- 172.21.15.44:0/3486808625 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4e50020610 con 0x55a85fb82830 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e605a2740 10 client.6204 fetch_fsmap learned FSMap version 25 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e605a2740 1 -- 172.21.15.44:0/3486808625 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55a85fbfb060 con 0x55a85fb82830 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e367fc640 1 -- 172.21.15.44:0/3486808625 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e48083da0 con 0x55a85fb82830 2024-04-02T13:22:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e367fc640 1 -- 172.21.15.44:0/3486808625 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f4e500207b0 con 0x55a85fb82830 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e605a2740 10 client.6204 fetch_fsmap finished waiting for FSMap version 25 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e605a2740 10 client.6204 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e605a2740 20 client.6204 populate_metadata read hostname 'smithi044' 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e605a2740 1 --2- 172.21.15.44:0/3486808625 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55a85fb9abc0 0x55a85fb75c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e605a2740 4 client.6204 mds_command: new command op to 5991 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e605a2740 1 -- 172.21.15.44:0/3486808625 --> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) -- 0x55a85fabe000 con 0x55a85fb9abc0 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/3486808625 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55a85fb9abc0 0x55a85fb75c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e367fc640 10 client.6204 ms_handle_connect on v2:172.21.15.69:6835/1745145720 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.475+0000 7f4e56ffd640 1 --2- 172.21.15.44:0/3486808625 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55a85fb9abc0 0x55a85fb75c60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d94e3f75db23e19 server_cookie=17ac532b865fa702 in_seq=0 out_seq=0 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.476+0000 7f4e367fc640 1 -- 172.21.15.44:0/3486808625 <== mds.0 v2:172.21.15.69:6835/1745145720 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a85fabe000 con 0x55a85fb9abc0 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.476+0000 7f4e367fc640 10 client.6204 handle_command_reply: tid=0 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.476+0000 7f4e605a2740 1 client.6204 shutdown 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.476+0000 7f4e605a2740 1 -- 172.21.15.44:0/3486808625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb82830 msgr2=0x55a85fb9a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.476+0000 7f4e605a2740 1 --2- 172.21.15.44:0/3486808625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb82830 0x55a85fb9a630 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4e500045b0 tx=0x7f4e50014a00 comp rx=0 tx=0).stop 2024-04-02T13:22:30.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.476+0000 7f4e605a2740 1 -- 172.21.15.44:0/3486808625 shutdown_connections 2024-04-02T13:22:30.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.476+0000 7f4e605a2740 1 --2- 172.21.15.44:0/3486808625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a85fb82830 0x55a85fb9a630 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.476+0000 7f4e605a2740 1 --2- 172.21.15.44:0/3486808625 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55a85fb9abc0 0x55a85fb75c60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.476+0000 7f4e605a2740 1 --2- 172.21.15.44:0/3486808625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a85fb7c310 0x55a85fb83200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.476+0000 7f4e605a2740 1 --2- 172.21.15.44:0/3486808625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a85fb4fe90 0x55a85fbf7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.476+0000 7f4e367fc640 0 client.6204 ms_handle_reset on v2:172.21.15.69:6835/1745145720 2024-04-02T13:22:30.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.476+0000 7f4e605a2740 1 -- 172.21.15.44:0/3486808625 >> 172.21.15.44:0/3486808625 conn(0x55a85fb4c330 msgr2=0x55a85fbb1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e605a2740 1 -- 172.21.15.44:0/3486808625 shutdown_connections 2024-04-02T13:22:30.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e605a2740 1 -- 172.21.15.44:0/3486808625 wait complete. 2024-04-02T13:22:30.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e605a2740 20 client.6204 trim_cache size 0 max 16384 2024-04-02T13:22:30.484 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:22:30.485 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-02T13:22:30.485 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "44755161-7470-444e-8fef-adae67206339", 2024-04-02T13:22:30.485 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-02T13:22:30.485 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:22:30.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e5d8d9640 1 -- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e3806c8f0 msgr2=0x7f4e3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e5d8d9640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e3806c8f0 0x7f4e3806edb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4e40007fd0 tx=0x7f4e40007de0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e5d8d9640 1 -- 172.21.15.44:0/624998728 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e580cec60 msgr2=0x7f4e581f41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e5d8d9640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e580cec60 0x7f4e581f41d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4e4802be50 tx=0x7f4e480046c0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e5d8d9640 1 -- 172.21.15.44:0/624998728 shutdown_connections 2024-04-02T13:22:30.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e5d8d9640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e3806c8f0 0x7f4e3806edb0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e5d8d9640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e5815e1d0 0x7f4e581505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e5d8d9640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e580d2dd0 0x7f4e58150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e5d8d9640 1 --2- 172.21.15.44:0/624998728 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e580cec60 0x7f4e581f41d0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e5d8d9640 1 -- 172.21.15.44:0/624998728 >> 172.21.15.44:0/624998728 conn(0x7f4e5809d3b0 msgr2=0x7f4e580cd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e5d8d9640 1 -- 172.21.15.44:0/624998728 shutdown_connections 2024-04-02T13:22:30.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.477+0000 7f4e5d8d9640 1 -- 172.21.15.44:0/624998728 wait complete. 2024-04-02T13:22:30.487 INFO:tasks.ceph.mds.d.smithi069.stderr:2024-04-02T13:22:30.476+0000 7fbf49514640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={6144=0-4194304@1} caps={6144=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6144 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x55c98c586680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(8)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-02T13:22:30.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:22:30.490 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub status 2024-04-02T13:22:30.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.591+0000 7f8f7feeb640 1 Processor -- start 2024-04-02T13:22:30.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.591+0000 7f8f7feeb640 1 -- start start 2024-04-02T13:22:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7feeb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f7815e260 0x7f8f780d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7feeb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8f7815e640 0x7f8f780d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7feeb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f780cf120 con 0x7f8f780cec60 2024-04-02T13:22:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7feeb640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f8f780b5a50 con 0x7f8f7815e640 2024-04-02T13:22:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7feeb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8f780d01c0 con 0x7f8f7815e260 2024-04-02T13:22:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7d486640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8f7815e640 0x7f8f780d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7e488640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f8f780cec60 0x7f8f78161f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:58530/0 (socket says 172.21.15.44:58530) 2024-04-02T13:22:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7e488640 1 -- 172.21.15.44:0/3478025031 learned_addr learned my addr 172.21.15.44:0/3478025031 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:22:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/3478025031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f7815e260 0x7f8f780d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7d486640 1 -- 172.21.15.44:0/3478025031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f7815e260 msgr2=0x7f8f780d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7d486640 1 --2- 172.21.15.44:0/3478025031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f7815e260 0x7f8f780d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7d486640 1 -- 172.21.15.44:0/3478025031 >> v1:172.21.15.44:6789/0 conn(0x7f8f780cec60 legacy=0x7f8f78161f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7d486640 1 -- 172.21.15.44:0/3478025031 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f780cf400 con 0x7f8f7815e640 2024-04-02T13:22:30.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7cc85640 1 -- 172.21.15.44:0/3478025031 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4159418877 0 0) 0x7f8f780cf120 con 0x7f8f780cec60 2024-04-02T13:22:30.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.592+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/3478025031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f7815e260 0x7f8f780d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:30.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.593+0000 7f8f7d486640 1 --2- 172.21.15.44:0/3478025031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7815e640 0x7f8f780d0ad0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8f680097a0 tx=0x7f8f6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c3ef6b1a2ab1b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.593+0000 7f8f7cc85640 1 -- 172.21.15.44:0/3478025031 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f68035020 con 0x7f8f7815e640 2024-04-02T13:22:30.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.593+0000 7f8f7cc85640 1 -- 172.21.15.44:0/3478025031 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f68031830 con 0x7f8f7815e640 2024-04-02T13:22:30.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.593+0000 7f8f7cc85640 1 -- 172.21.15.44:0/3478025031 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f68031c00 con 0x7f8f7815e640 2024-04-02T13:22:30.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.594+0000 7f8f7feeb640 1 -- 172.21.15.44:0/3478025031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7815e640 msgr2=0x7f8f780d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.594+0000 7f8f7feeb640 1 --2- 172.21.15.44:0/3478025031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7815e640 0x7f8f780d0ad0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8f680097a0 tx=0x7f8f6802b8d0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.594+0000 7f8f7feeb640 1 -- 172.21.15.44:0/3478025031 shutdown_connections 2024-04-02T13:22:30.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.594+0000 7f8f7feeb640 1 --2- 172.21.15.44:0/3478025031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7815e640 0x7f8f780d0ad0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.594+0000 7f8f7feeb640 1 --2- 172.21.15.44:0/3478025031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f7815e260 0x7f8f780d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.594+0000 7f8f7feeb640 1 -- 172.21.15.44:0/3478025031 >> 172.21.15.44:0/3478025031 conn(0x7f8f7809d3c0 msgr2=0x7f8f780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.594+0000 7f8f7feeb640 1 -- 172.21.15.44:0/3478025031 shutdown_connections 2024-04-02T13:22:30.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.594+0000 7f8f7feeb640 1 -- 172.21.15.44:0/3478025031 wait complete. 2024-04-02T13:22:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.595+0000 7f8f7feeb640 1 Processor -- start 2024-04-02T13:22:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.595+0000 7f8f7feeb640 1 -- start start 2024-04-02T13:22:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7feeb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f780cec60 0x7f8f7815c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7feeb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f7815e260 0x7f8f78160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7feeb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f7815e640 0x7f8f78161130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7feeb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8f780cfca0 con 0x7f8f780cec60 2024-04-02T13:22:30.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7feeb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8f780cf9c0 con 0x7f8f7815e640 2024-04-02T13:22:30.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7feeb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8f780d2ea0 con 0x7f8f7815e260 2024-04-02T13:22:30.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7dc87640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f780cec60 0x7f8f7815c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7dc87640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f780cec60 0x7f8f7815c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43786/0 (socket says 172.21.15.44:43786) 2024-04-02T13:22:30.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7dc87640 1 -- 172.21.15.44:0/2478424442 learned_addr learned my addr 172.21.15.44:0/2478424442 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7e488640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f7815e640 0x7f8f78161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7d486640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f7815e260 0x7f8f78160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7dc87640 1 -- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f7815e260 msgr2=0x7f8f78160bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f7815e260 0x7f8f78160bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7dc87640 1 -- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f7815e640 msgr2=0x7f8f78161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f7815e640 0x7f8f78161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7dc87640 1 -- 172.21.15.44:0/2478424442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f781f3a50 con 0x7f8f780cec60 2024-04-02T13:22:30.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7d486640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f7815e260 0x7f8f78160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:30.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7e488640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f7815e640 0x7f8f78161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:30.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.596+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f780cec60 0x7f8f7815c670 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009fc0 tx=0x7f8f7000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.597+0000 7f8f6e7fc640 1 -- 172.21.15.44:0/2478424442 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7001a070 con 0x7f8f780cec60 2024-04-02T13:22:30.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.597+0000 7f8f6e7fc640 1 -- 172.21.15.44:0/2478424442 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f70003bb0 con 0x7f8f780cec60 2024-04-02T13:22:30.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.597+0000 7f8f7feeb640 1 -- 172.21.15.44:0/2478424442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f780cc9f0 con 0x7f8f780cec60 2024-04-02T13:22:30.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.597+0000 7f8f6e7fc640 1 -- 172.21.15.44:0/2478424442 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70004030 con 0x7f8f780cec60 2024-04-02T13:22:30.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.597+0000 7f8f7feeb640 1 -- 172.21.15.44:0/2478424442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f781f3fa0 con 0x7f8f780cec60 2024-04-02T13:22:30.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.598+0000 7f8f6e7fc640 1 -- 172.21.15.44:0/2478424442 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8f7001d070 con 0x7f8f780cec60 2024-04-02T13:22:30.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.598+0000 7f8f6e7fc640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8f5c06c8f0 0x7f8f5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.598+0000 7f8f6e7fc640 1 -- 172.21.15.44:0/2478424442 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f8f7008be70 con 0x7f8f780cec60 2024-04-02T13:22:30.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.599+0000 7f8f7d486640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8f5c06c8f0 0x7f8f5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.599+0000 7f8f7d486640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8f5c06c8f0 0x7f8f5c06edb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8f68036000 tx=0x7f8f6803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.599+0000 7f8f82ba9740 1 Processor -- start 2024-04-02T13:22:30.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.600+0000 7f8f82ba9740 1 -- start start 2024-04-02T13:22:30.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.600+0000 7f8f82ba9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e33b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.600+0000 7f8f82ba9740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 0x564f9e36b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.600+0000 7f8f82ba9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e33be80 0x564f9e36bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.600+0000 7f8f82ba9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564f9e2a5830 con 0x564f9e3689a0 2024-04-02T13:22:30.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.600+0000 7f8f82ba9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564f9e24d820 con 0x564f9e33be80 2024-04-02T13:22:30.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.600+0000 7f8f82ba9740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564f9e2a17e0 con 0x564f9e368d80 2024-04-02T13:22:30.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.600+0000 7f8f7dc87640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e33b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.600+0000 7f8f7dc87640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e33b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43800/0 (socket says 172.21.15.44:43800) 2024-04-02T13:22:30.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.600+0000 7f8f7dc87640 1 -- 172.21.15.44:0/882628513 learned_addr learned my addr 172.21.15.44:0/882628513 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.600+0000 7f8f7d486640 1 --2- 172.21.15.44:0/882628513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 0x564f9e36b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.600+0000 7f8f7e488640 1 --2- 172.21.15.44:0/882628513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e33be80 0x564f9e36bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.601+0000 7f8f7dc87640 1 -- 172.21.15.44:0/882628513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 msgr2=0x564f9e36b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.601+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/882628513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 0x564f9e36b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.601+0000 7f8f7dc87640 1 -- 172.21.15.44:0/882628513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e33be80 msgr2=0x564f9e36bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.601+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/882628513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e33be80 0x564f9e36bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.601+0000 7f8f7dc87640 1 -- 172.21.15.44:0/882628513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f9e0837d0 con 0x564f9e3689a0 2024-04-02T13:22:30.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.601+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/882628513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e33b940 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8f7008a210 tx=0x7f8f70059c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47ab198f45c6e23 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.601+0000 7f8f5b7fe640 1 -- 172.21.15.44:0/882628513 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f700561c0 con 0x564f9e3689a0 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.601+0000 7f8f5b7fe640 1 -- 172.21.15.44:0/882628513 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f70059db0 con 0x564f9e3689a0 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.601+0000 7f8f5b7fe640 1 -- 172.21.15.44:0/882628513 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70056360 con 0x564f9e3689a0 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.602+0000 7f8f82ba9740 1 -- 172.21.15.44:0/882628513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 msgr2=0x564f9e33b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.602+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/882628513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e33b940 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8f7008a210 tx=0x7f8f70059c90 comp rx=0 tx=0).stop 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.602+0000 7f8f82ba9740 1 -- 172.21.15.44:0/882628513 shutdown_connections 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.602+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/882628513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e33be80 0x564f9e36bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.602+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/882628513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 0x564f9e36b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.602+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/882628513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e33b940 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.602+0000 7f8f82ba9740 1 -- 172.21.15.44:0/882628513 >> 172.21.15.44:0/882628513 conn(0x564f9e33dbb0 msgr2=0x564f9e386730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.602+0000 7f8f82ba9740 1 -- 172.21.15.44:0/882628513 shutdown_connections 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.602+0000 7f8f82ba9740 1 -- 172.21.15.44:0/882628513 wait complete. 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.602+0000 7f8f82ba9740 1 -- start start 2024-04-02T13:22:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.602+0000 7f8f82ba9740 1 Processor -- start 2024-04-02T13:22:30.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.603+0000 7f8f82ba9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e3663b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.603+0000 7f8f82ba9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e368d80 0x564f9e3966c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.603+0000 7f8f82ba9740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e37e130 0x564f9e36f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.603+0000 7f8f82ba9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564f9e340150 con 0x564f9e3689a0 2024-04-02T13:22:30.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.603+0000 7f8f82ba9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564f9e260d30 con 0x564f9e368d80 2024-04-02T13:22:30.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.603+0000 7f8f82ba9740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564f9e33f540 con 0x564f9e37e130 2024-04-02T13:22:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.603+0000 7f8f82ba9740 10 client.0 _ll_register_callbacks cb 0x564f9e260720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:22:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.603+0000 7f8f7dc87640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e3663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.603+0000 7f8f7dc87640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e3663b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43806/0 (socket says 172.21.15.44:43806) 2024-04-02T13:22:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.603+0000 7f8f7dc87640 1 -- 172.21.15.44:0/3028502010 learned_addr learned my addr 172.21.15.44:0/3028502010 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.603+0000 7f8f7e488640 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e37e130 0x564f9e36f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.603+0000 7f8f7d486640 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e368d80 0x564f9e3966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f7dc87640 1 -- 172.21.15.44:0/3028502010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e37e130 msgr2=0x564f9e36f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e37e130 0x564f9e36f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f7dc87640 1 -- 172.21.15.44:0/3028502010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e368d80 msgr2=0x564f9e3966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e368d80 0x564f9e3966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f7dc87640 1 -- 172.21.15.44:0/3028502010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f9dfb93f0 con 0x564f9e3689a0 2024-04-02T13:22:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f7e488640 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e37e130 0x564f9e36f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:22:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e3663b0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8f7005e010 tx=0x7f8f70000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69bb33aedb913249 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:22:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f58ff9640 1 -- 172.21.15.44:0/3028502010 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f700623f0 con 0x564f9e3689a0 2024-04-02T13:22:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f58ff9640 1 -- 172.21.15.44:0/3028502010 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f8f70077cb0 con 0x564f9e3689a0 2024-04-02T13:22:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f82ba9740 1 -- 172.21.15.44:0/3028502010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f70059db0 con 0x564f9e3689a0 2024-04-02T13:22:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f58ff9640 1 -- 172.21.15.44:0/3028502010 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8f70079100 con 0x564f9e3689a0 2024-04-02T13:22:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f82ba9740 10 client.6219 fetch_fsmap learned FSMap version 25 2024-04-02T13:22:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f82ba9740 1 -- 172.21.15.44:0/3028502010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x564f9df790f0 con 0x564f9e3689a0 2024-04-02T13:22:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f58ff9640 1 -- 172.21.15.44:0/3028502010 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f70059db0 con 0x564f9e3689a0 2024-04-02T13:22:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f58ff9640 1 -- 172.21.15.44:0/3028502010 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f8f70058fa0 con 0x564f9e3689a0 2024-04-02T13:22:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f82ba9740 10 client.6219 fetch_fsmap finished waiting for FSMap version 25 2024-04-02T13:22:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f82ba9740 10 client.6219 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-02T13:22:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f82ba9740 20 client.6219 populate_metadata read hostname 'smithi044' 2024-04-02T13:22:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x564f9e36d920 0x564f9e3a3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f82ba9740 4 client.6219 mds_command: new command op to 5991 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:22:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.604+0000 7f8f82ba9740 1 -- 172.21.15.44:0/3028502010 --> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f9e2b2d70 con 0x564f9e36d920 2024-04-02T13:22:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.605+0000 7f8f7d486640 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x564f9e36d920 0x564f9e3a3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.605+0000 7f8f7d486640 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x564f9e36d920 0x564f9e3a3f40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e77aebbad02c52fc server_cookie=9f4b46fe911971a7 in_seq=0 out_seq=0 2024-04-02T13:22:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.605+0000 7f8f58ff9640 10 client.6219 ms_handle_connect on v2:172.21.15.69:6835/1745145720 2024-04-02T13:22:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.605+0000 7f8f58ff9640 1 -- 172.21.15.44:0/3028502010 <== mds.0 v2:172.21.15.69:6835/1745145720 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564f9e2b2d70 con 0x564f9e36d920 2024-04-02T13:22:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f58ff9640 10 client.6219 handle_command_reply: tid=0 2024-04-02T13:22:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f82ba9740 1 client.6219 shutdown 2024-04-02T13:22:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f82ba9740 1 -- 172.21.15.44:0/3028502010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 msgr2=0x564f9e3663b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e3663b0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8f7005e010 tx=0x7f8f70000f40 comp rx=0 tx=0).stop 2024-04-02T13:22:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f82ba9740 1 -- 172.21.15.44:0/3028502010 shutdown_connections 2024-04-02T13:22:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e37e130 0x564f9e36f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x564f9e36d920 0x564f9e3a3f40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e368d80 0x564f9e3966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/3028502010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e3663b0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f58ff9640 0 client.6219 ms_handle_reset on v2:172.21.15.69:6835/1745145720 2024-04-02T13:22:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f82ba9740 1 -- 172.21.15.44:0/3028502010 >> 172.21.15.44:0/3028502010 conn(0x564f9e33be80 msgr2=0x564f9e36b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f82ba9740 1 -- 172.21.15.44:0/3028502010 shutdown_connections 2024-04-02T13:22:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f82ba9740 1 -- 172.21.15.44:0/3028502010 wait complete. 2024-04-02T13:22:30.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.606+0000 7f8f82ba9740 20 client.6219 trim_cache size 0 max 16384 2024-04-02T13:22:30.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.624+0000 7f8f82ba9740 1 Processor -- start 2024-04-02T13:22:30.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.624+0000 7f8f82ba9740 1 -- start start 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.624+0000 7f8f82ba9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e37ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.624+0000 7f8f82ba9740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 0x564f9e36efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.624+0000 7f8f82ba9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e37e130 0x564f9e36f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.624+0000 7f8f82ba9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564f9e380660 con 0x564f9e3689a0 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.624+0000 7f8f82ba9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8f78200c20 con 0x564f9e37e130 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.624+0000 7f8f82ba9740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8f781ffc00 con 0x564f9e368d80 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.624+0000 7f8f7dc87640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e37ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.624+0000 7f8f7dc87640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e37ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43818/0 (socket says 172.21.15.44:43818) 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.624+0000 7f8f7dc87640 1 -- 172.21.15.44:0/3699464061 learned_addr learned my addr 172.21.15.44:0/3699464061 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.625+0000 7f8f7e488640 1 --2- 172.21.15.44:0/3699464061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e37e130 0x564f9e36f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.625+0000 7f8f7d486640 1 --2- 172.21.15.44:0/3699464061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 0x564f9e36efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.625+0000 7f8f7dc87640 1 -- 172.21.15.44:0/3699464061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 msgr2=0x564f9e36efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.625+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/3699464061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 0x564f9e36efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.625+0000 7f8f7dc87640 1 -- 172.21.15.44:0/3699464061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e37e130 msgr2=0x564f9e36f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.625+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/3699464061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e37e130 0x564f9e36f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.625+0000 7f8f7dc87640 1 -- 172.21.15.44:0/3699464061 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f9e2aaab0 con 0x564f9e3689a0 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.625+0000 7f8f7e488640 1 --2- 172.21.15.44:0/3699464061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e37e130 0x564f9e36f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.625+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/3699464061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e37ea60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009f90 tx=0x7f8f70085040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9526780bcd471a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.625+0000 7f8f5affd640 1 -- 172.21.15.44:0/3699464061 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f700621f0 con 0x564f9e3689a0 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.625+0000 7f8f5affd640 1 -- 172.21.15.44:0/3699464061 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f700623b0 con 0x564f9e3689a0 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.625+0000 7f8f5affd640 1 -- 172.21.15.44:0/3699464061 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f700602b0 con 0x564f9e3689a0 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.626+0000 7f8f82ba9740 1 -- 172.21.15.44:0/3699464061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 msgr2=0x564f9e37ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.626+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/3699464061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e37ea60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009f90 tx=0x7f8f70085040 comp rx=0 tx=0).stop 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.626+0000 7f8f82ba9740 1 -- 172.21.15.44:0/3699464061 shutdown_connections 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.626+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/3699464061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e37e130 0x564f9e36f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.626+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/3699464061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 0x564f9e36efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.626+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/3699464061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e37ea60 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.626+0000 7f8f82ba9740 1 -- 172.21.15.44:0/3699464061 >> 172.21.15.44:0/3699464061 conn(0x564f9e33be80 msgr2=0x564f9e3962c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.626+0000 7f8f82ba9740 1 -- 172.21.15.44:0/3699464061 shutdown_connections 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.626+0000 7f8f82ba9740 1 -- 172.21.15.44:0/3699464061 wait complete. 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.626+0000 7f8f82ba9740 1 -- start start 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.626+0000 7f8f82ba9740 1 Processor -- start 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f82ba9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e3a89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f82ba9740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 0x564f9e340d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f82ba9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e37e130 0x564f9e3412b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f82ba9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8f781ff5c0 con 0x564f9e3689a0 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f82ba9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8f780b7ae0 con 0x564f9e37e130 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f82ba9740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564f9e36dd00 con 0x564f9e368d80 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f82ba9740 10 client.0 _ll_register_callbacks cb 0x7f8f68009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f7dc87640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e3a89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f7dc87640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e3a89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43830/0 (socket says 172.21.15.44:43830) 2024-04-02T13:22:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f7dc87640 1 -- 172.21.15.44:0/1060932287 learned_addr learned my addr 172.21.15.44:0/1060932287 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:22:30.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f7d486640 1 --2- 172.21.15.44:0/1060932287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 0x564f9e340d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f7e488640 1 --2- 172.21.15.44:0/1060932287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e37e130 0x564f9e3412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f7dc87640 1 -- 172.21.15.44:0/1060932287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 msgr2=0x564f9e340d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/1060932287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 0x564f9e340d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f7dc87640 1 -- 172.21.15.44:0/1060932287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e37e130 msgr2=0x564f9e3412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:22:30.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/1060932287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e37e130 0x564f9e3412b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.627+0000 7f8f7dc87640 1 -- 172.21.15.44:0/1060932287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f9e365420 con 0x564f9e3689a0 2024-04-02T13:22:30.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f7dc87640 1 --2- 172.21.15.44:0/1060932287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e3a89d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8f70059f30 tx=0x7f8f70012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2ae9c5b808a954 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:22:30.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f58ff9640 1 -- 172.21.15.44:0/1060932287 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f700621f0 con 0x564f9e3689a0 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f58ff9640 1 -- 172.21.15.44:0/1060932287 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f8f70077cb0 con 0x564f9e3689a0 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f82ba9740 1 -- 172.21.15.44:0/1060932287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f700623b0 con 0x564f9e3689a0 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f58ff9640 1 -- 172.21.15.44:0/1060932287 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8f700558a0 con 0x564f9e3689a0 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f82ba9740 10 client.6225 fetch_fsmap learned FSMap version 25 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f82ba9740 1 -- 172.21.15.44:0/1060932287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x564f9e33cd50 con 0x564f9e3689a0 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f58ff9640 1 -- 172.21.15.44:0/1060932287 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f700623b0 con 0x564f9e3689a0 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f58ff9640 1 -- 172.21.15.44:0/1060932287 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f8f70055a40 con 0x564f9e3689a0 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f82ba9740 10 client.6225 fetch_fsmap finished waiting for FSMap version 25 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f82ba9740 10 client.6225 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f82ba9740 20 client.6225 populate_metadata read hostname 'smithi044' 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/1060932287 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x564f9e387d90 0x564f9e388170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f82ba9740 4 client.6225 mds_command: new command op to 5991 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f82ba9740 1 -- 172.21.15.44:0/1060932287 --> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f9e381200 con 0x564f9e387d90 2024-04-02T13:22:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.628+0000 7f8f7d486640 1 --2- 172.21.15.44:0/1060932287 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x564f9e387d90 0x564f9e388170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:22:30.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f7d486640 1 --2- 172.21.15.44:0/1060932287 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x564f9e387d90 0x564f9e388170 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be2472dc1d52d978 server_cookie=85ab6917da55bbf2 in_seq=0 out_seq=0 2024-04-02T13:22:30.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f58ff9640 10 client.6225 ms_handle_connect on v2:172.21.15.69:6835/1745145720 2024-04-02T13:22:30.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f58ff9640 1 -- 172.21.15.44:0/1060932287 <== mds.0 v2:172.21.15.69:6835/1745145720 1 ==== command_reply(tid 0: 0 ) ==== 8+0+260 (crc 0 0 0) 0x564f9e381200 con 0x564f9e387d90 2024-04-02T13:22:30.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f58ff9640 10 client.6225 handle_command_reply: tid=0 2024-04-02T13:22:30.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f82ba9740 1 client.6225 shutdown 2024-04-02T13:22:30.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f82ba9740 1 -- 172.21.15.44:0/1060932287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 msgr2=0x564f9e3a89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/1060932287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e3a89d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8f70059f30 tx=0x7f8f70012000 comp rx=0 tx=0).stop 2024-04-02T13:22:30.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f82ba9740 1 -- 172.21.15.44:0/1060932287 shutdown_connections 2024-04-02T13:22:30.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/1060932287 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x564f9e387d90 0x564f9e388170 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/1060932287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f9e37e130 0x564f9e3412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/1060932287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f9e368d80 0x564f9e340d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f58ff9640 0 client.6225 ms_handle_reset on v2:172.21.15.69:6835/1745145720 2024-04-02T13:22:30.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f82ba9740 1 --2- 172.21.15.44:0/1060932287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f9e3689a0 0x564f9e3a89d0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f82ba9740 1 -- 172.21.15.44:0/1060932287 >> 172.21.15.44:0/1060932287 conn(0x564f9e33be80 msgr2=0x564f9e3963e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f82ba9740 1 -- 172.21.15.44:0/1060932287 shutdown_connections 2024-04-02T13:22:30.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f82ba9740 1 -- 172.21.15.44:0/1060932287 wait complete. 2024-04-02T13:22:30.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.629+0000 7f8f82ba9740 20 client.6225 trim_cache size 0 max 16384 2024-04-02T13:22:30.639 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:22:30.639 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T13:22:30.639 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-02T13:22:30.639 INFO:teuthology.orchestra.run.smithi044.stdout: "44755161-7470-444e-8fef-adae67206339": { 2024-04-02T13:22:30.639 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-02T13:22:30.639 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "44755161-7470-444e-8fef-adae67206339", 2024-04-02T13:22:30.639 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,repair" 2024-04-02T13:22:30.640 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:30.640 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:22:30.640 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:22:30.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.630+0000 7f8f7feeb640 1 -- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8f5c06c8f0 msgr2=0x7f8f5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.630+0000 7f8f7feeb640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8f5c06c8f0 0x7f8f5c06edb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8f68036000 tx=0x7f8f6803c040 comp rx=0 tx=0).stop 2024-04-02T13:22:30.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.630+0000 7f8f7feeb640 1 -- 172.21.15.44:0/2478424442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f780cec60 msgr2=0x7f8f7815c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:22:30.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.630+0000 7f8f7feeb640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f780cec60 0x7f8f7815c670 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009fc0 tx=0x7f8f7000d560 comp rx=0 tx=0).stop 2024-04-02T13:22:30.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.630+0000 7f8f7feeb640 1 -- 172.21.15.44:0/2478424442 shutdown_connections 2024-04-02T13:22:30.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.630+0000 7f8f7feeb640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8f5c06c8f0 0x7f8f5c06edb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.630+0000 7f8f7feeb640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f7815e640 0x7f8f78161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.630+0000 7f8f7feeb640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f7815e260 0x7f8f78160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.630+0000 7f8f7feeb640 1 --2- 172.21.15.44:0/2478424442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f780cec60 0x7f8f7815c670 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:22:30.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.630+0000 7f8f7feeb640 1 -- 172.21.15.44:0/2478424442 >> 172.21.15.44:0/2478424442 conn(0x7f8f7809d3c0 msgr2=0x7f8f780b47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:22:30.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.630+0000 7f8f7feeb640 1 -- 172.21.15.44:0/2478424442 shutdown_connections 2024-04-02T13:22:30.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:30.630+0000 7f8f7feeb640 1 -- 172.21.15.44:0/2478424442 wait complete. 2024-04-02T13:22:30.643 INFO:tasks.cephfs.filesystem:scrub status for tag:44755161-7470-444e-8fef-adae67206339 - {'path': '/', 'tag': '44755161-7470-444e-8fef-adae67206339', 'options': 'recursive,repair'} 2024-04-02T13:22:31.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:31.066+0000 7f40d7fff640 1 -- 172.21.15.44:0/559431608 <== mon.0 v2:172.21.15.44:3300/0 7 ==== log(6 entries from seq 1 at 2024-04-02T13:22:30.476601+0000) ==== 1078+0+0 (secure 0 0 0) 0x7f40d808a960 con 0x7f40e0161920 2024-04-02T13:22:31.068 INFO:teuthology.orchestra.run.smithi044.stdout: cluster: 2024-04-02T13:22:31.068 INFO:teuthology.orchestra.run.smithi044.stdout: id: 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:22:31.068 INFO:teuthology.orchestra.run.smithi044.stdout: health: HEALTH_OK 2024-04-02T13:22:31.068 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:31.068 INFO:teuthology.orchestra.run.smithi044.stdout: services: 2024-04-02T13:22:31.068 INFO:teuthology.orchestra.run.smithi044.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: mgr: x(active, since 2m), standbys: y 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: mds: 1/1 daemons up, 3 standby 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: data: 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: volumes: 1/1 healthy 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: pools: 3 pools, 97 pgs 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: objects: 24 objects, 579 KiB 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: usage: 217 MiB used, 720 GiB / 720 GiB avail 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: pgs: 97 active+clean 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: io: 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: client: 2.0 KiB/s wr, 0 op/s rd, 6 op/s wr 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:31.069 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:22:31.070 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:22:30.476601+0000 mds.d [INF] scrub queued for path: / 2024-04-02T13:22:31.070 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:22:30.476604+0000 mds.d [INF] scrub summary: idle+waiting paths [/] 2024-04-02T13:22:31.070 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:22:30.476609+0000 mds.d [INF] scrub summary: active paths [/] 2024-04-02T13:22:31.070 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:22:30.477334+0000 mds.d [WRN] bad backtrace on inode 0x10000000001(/parent_a/alpha), rewriting it 2024-04-02T13:22:31.070 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:22:30.477340+0000 mds.d [INF] Scrub repaired inode 0x10000000001 (/parent_a/alpha) 2024-04-02T13:22:31.070 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:22:30.477608+0000 mds.d [INF] scrub summary: idle+waiting paths [/] 2024-04-02T13:22:34.072 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:22:33.015121+0000 mds.d [INF] scrub summary: idle 2024-04-02T13:22:34.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:22:34.069+0000 7f40d7fff640 1 -- 172.21.15.44:0/559431608 <== mon.0 v2:172.21.15.44:3300/0 8 ==== log(1 entries from seq 7 at 2024-04-02T13:22:33.015121+0000) ==== 189+0+0 (secure 0 0 0) 0x7f40d8058180 con 0x7f40e0161920 2024-04-02T13:22:52.035 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:22:52.038 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:23:00.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:00.644 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub status 2024-04-02T13:23:00.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.757+0000 7f94cb10e640 1 Processor -- start 2024-04-02T13:23:00.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.757+0000 7f94cb10e640 1 -- start start 2024-04-02T13:23:00.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.757+0000 7f94cb10e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f94c40d0a30 0x7f94c40b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94cb10e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94c40b3990 0x7f94c4143a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94cb10e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f94c4143f60 0x7f94c4071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94cb10e640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f94c40b3df0 con 0x7f94c40d0a30 2024-04-02T13:23:00.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94cb10e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f94c4161e10 con 0x7f94c4143f60 2024-04-02T13:23:00.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94cb10e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f94c40d0e10 con 0x7f94c40b3990 2024-04-02T13:23:00.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94c8eaa640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f94c40d0a30 0x7f94c40b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94c8eaa640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f94c40d0a30 0x7f94c40b3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45974/0 (socket says 172.21.15.44:45974) 2024-04-02T13:23:00.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94c8eaa640 1 -- 172.21.15.44:0/2706678395 learned_addr learned my addr 172.21.15.44:0/2706678395 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:00.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94c3fff640 1 --2- 172.21.15.44:0/2706678395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94c40b3990 0x7f94c4143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94c96ab640 1 --2- 172.21.15.44:0/2706678395 >> v2:172.21.15.69:3300/0 conn(0x7f94c4143f60 0x7f94c4071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94c8eaa640 1 -- 172.21.15.44:0/2706678395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94c40b3990 msgr2=0x7f94c4143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:00.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/2706678395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94c40b3990 0x7f94c4143a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94c8eaa640 1 -- 172.21.15.44:0/2706678395 >> v2:172.21.15.69:3300/0 conn(0x7f94c4143f60 msgr2=0x7f94c4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:00.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/2706678395 >> v2:172.21.15.69:3300/0 conn(0x7f94c4143f60 0x7f94c4071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94c8eaa640 1 -- 172.21.15.44:0/2706678395 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c4144750 con 0x7f94c40d0a30 2024-04-02T13:23:00.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.758+0000 7f94c3fff640 1 --2- 172.21.15.44:0/2706678395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94c40b3990 0x7f94c4143a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:00.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.759+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/2706678395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94c40d0a30 0x7f94c40b3450 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f94c40d0fc0 tx=0x7f94b802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf415739b8b0b75 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:00.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.759+0000 7f94c37fe640 1 -- 172.21.15.44:0/2706678395 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b80317c0 con 0x7f94c40d0a30 2024-04-02T13:23:00.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.759+0000 7f94c37fe640 1 -- 172.21.15.44:0/2706678395 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94b8037070 con 0x7f94c40d0a30 2024-04-02T13:23:00.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.759+0000 7f94c37fe640 1 -- 172.21.15.44:0/2706678395 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8031b80 con 0x7f94c40d0a30 2024-04-02T13:23:00.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.760+0000 7f94cb10e640 1 -- 172.21.15.44:0/2706678395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94c40d0a30 msgr2=0x7f94c40b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:00.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.760+0000 7f94cb10e640 1 --2- 172.21.15.44:0/2706678395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94c40d0a30 0x7f94c40b3450 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f94c40d0fc0 tx=0x7f94b802b860 comp rx=0 tx=0).stop 2024-04-02T13:23:00.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.760+0000 7f94cb10e640 1 -- 172.21.15.44:0/2706678395 shutdown_connections 2024-04-02T13:23:00.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.760+0000 7f94cb10e640 1 --2- 172.21.15.44:0/2706678395 >> v2:172.21.15.69:3300/0 conn(0x7f94c4143f60 0x7f94c4071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.760+0000 7f94cb10e640 1 --2- 172.21.15.44:0/2706678395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94c40b3990 0x7f94c4143a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.760+0000 7f94cb10e640 1 --2- 172.21.15.44:0/2706678395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94c40d0a30 0x7f94c40b3450 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.760+0000 7f94cb10e640 1 -- 172.21.15.44:0/2706678395 >> 172.21.15.44:0/2706678395 conn(0x7f94c40b5d90 msgr2=0x7f94c406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:00.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.760+0000 7f94cb10e640 1 -- 172.21.15.44:0/2706678395 shutdown_connections 2024-04-02T13:23:00.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.760+0000 7f94cb10e640 1 -- 172.21.15.44:0/2706678395 wait complete. 2024-04-02T13:23:00.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.761+0000 7f94cb10e640 1 Processor -- start 2024-04-02T13:23:00.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.761+0000 7f94cb10e640 1 -- start start 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94cb10e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94c40b3990 0x7f94c4140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94cb10e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94c40d0a30 0x7f94c41f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94cb10e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94c4143f60 0x7f94c4143590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94cb10e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f94c4162620 con 0x7f94c40d0a30 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94cb10e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f94c4162340 con 0x7f94c40b3990 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94cb10e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f94c4144390 con 0x7f94c4143f60 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94c3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94c40d0a30 0x7f94c41f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94c3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94c40d0a30 0x7f94c41f74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45988/0 (socket says 172.21.15.44:45988) 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94c3fff640 1 -- 172.21.15.44:0/3002677307 learned_addr learned my addr 172.21.15.44:0/3002677307 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/3002677307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94c40b3990 0x7f94c4140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94c3fff640 1 -- 172.21.15.44:0/3002677307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94c4143f60 msgr2=0x7f94c4143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94c3fff640 1 --2- 172.21.15.44:0/3002677307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94c4143f60 0x7f94c4143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94c3fff640 1 -- 172.21.15.44:0/3002677307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94c40b3990 msgr2=0x7f94c4140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94c3fff640 1 --2- 172.21.15.44:0/3002677307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94c40b3990 0x7f94c4140f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94c3fff640 1 -- 172.21.15.44:0/3002677307 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c41f7c00 con 0x7f94c40d0a30 2024-04-02T13:23:00.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.762+0000 7f94c3fff640 1 --2- 172.21.15.44:0/3002677307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94c40d0a30 0x7f94c41f74b0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f94b000e810 tx=0x7f94b000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:00.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.763+0000 7f94c17fa640 1 -- 172.21.15.44:0/3002677307 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b000f040 con 0x7f94c40d0a30 2024-04-02T13:23:00.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.763+0000 7f94c17fa640 1 -- 172.21.15.44:0/3002677307 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94b0004030 con 0x7f94c40d0a30 2024-04-02T13:23:00.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.763+0000 7f94cb10e640 1 -- 172.21.15.44:0/3002677307 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c41f7ee0 con 0x7f94c40d0a30 2024-04-02T13:23:00.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.763+0000 7f94c17fa640 1 -- 172.21.15.44:0/3002677307 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b0004400 con 0x7f94c40d0a30 2024-04-02T13:23:00.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.763+0000 7f94cb10e640 1 -- 172.21.15.44:0/3002677307 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c4143070 con 0x7f94c40d0a30 2024-04-02T13:23:00.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.765+0000 7f94c17fa640 1 -- 172.21.15.44:0/3002677307 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f94b000c6d0 con 0x7f94c40d0a30 2024-04-02T13:23:00.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.765+0000 7f94c17fa640 1 --2- 172.21.15.44:0/3002677307 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f94a806c8f0 0x7f94a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.765+0000 7f94c17fa640 1 -- 172.21.15.44:0/3002677307 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f94b008b930 con 0x7f94c40d0a30 2024-04-02T13:23:00.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.765+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/3002677307 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f94a806c8f0 0x7f94a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.766+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/3002677307 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f94a806c8f0 0x7f94a806edb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f94b8036000 tx=0x7f94b8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:00.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.766+0000 7f94cddd3740 1 Processor -- start 2024-04-02T13:23:00.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.766+0000 7f94cddd3740 1 -- start start 2024-04-02T13:23:00.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.766+0000 7f94cddd3740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 0x55bcd8b83940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.766+0000 7f94cddd3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb0d80 0x55bcd8bb3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.766+0000 7f94cddd3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8b83e80 0x55bcd8bb3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.766+0000 7f94cddd3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bcd8aed830 con 0x55bcd8b83e80 2024-04-02T13:23:00.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.766+0000 7f94cddd3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bcd8a95820 con 0x55bcd8bb0d80 2024-04-02T13:23:00.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.766+0000 7f94cddd3740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55bcd8ae97e0 con 0x55bcd8bb09a0 2024-04-02T13:23:00.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94c96ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8b83e80 0x55bcd8bb3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94c96ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8b83e80 0x55bcd8bb3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45990/0 (socket says 172.21.15.44:45990) 2024-04-02T13:23:00.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94c96ab640 1 -- 172.21.15.44:0/4276596054 learned_addr learned my addr 172.21.15.44:0/4276596054 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:00.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94c3fff640 1 --2- 172.21.15.44:0/4276596054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb0d80 0x55bcd8bb3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/4276596054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 0x55bcd8b83940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94c96ab640 1 -- 172.21.15.44:0/4276596054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 msgr2=0x55bcd8b83940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:00.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94c96ab640 1 --2- 172.21.15.44:0/4276596054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 0x55bcd8b83940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94c96ab640 1 -- 172.21.15.44:0/4276596054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb0d80 msgr2=0x55bcd8bb3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:00.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94c96ab640 1 --2- 172.21.15.44:0/4276596054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb0d80 0x55bcd8bb3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94c96ab640 1 -- 172.21.15.44:0/4276596054 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcd88cb7d0 con 0x55bcd8b83e80 2024-04-02T13:23:00.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94c96ab640 1 --2- 172.21.15.44:0/4276596054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8b83e80 0x55bcd8bb3c70 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009ee0 tx=0x7f94bc007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ed5bd2b299706e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:00.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94ae7fc640 1 -- 172.21.15.44:0/4276596054 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc019070 con 0x55bcd8b83e80 2024-04-02T13:23:00.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94ae7fc640 1 -- 172.21.15.44:0/4276596054 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94bc012710 con 0x55bcd8b83e80 2024-04-02T13:23:00.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.767+0000 7f94ae7fc640 1 -- 172.21.15.44:0/4276596054 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc012b00 con 0x55bcd8b83e80 2024-04-02T13:23:00.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.768+0000 7f94cddd3740 1 -- 172.21.15.44:0/4276596054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8b83e80 msgr2=0x55bcd8bb3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:00.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.768+0000 7f94cddd3740 1 --2- 172.21.15.44:0/4276596054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8b83e80 0x55bcd8bb3c70 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009ee0 tx=0x7f94bc007700 comp rx=0 tx=0).stop 2024-04-02T13:23:00.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.768+0000 7f94cddd3740 1 -- 172.21.15.44:0/4276596054 shutdown_connections 2024-04-02T13:23:00.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.768+0000 7f94cddd3740 1 --2- 172.21.15.44:0/4276596054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8b83e80 0x55bcd8bb3c70 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.768+0000 7f94cddd3740 1 --2- 172.21.15.44:0/4276596054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb0d80 0x55bcd8bb3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.768+0000 7f94cddd3740 1 --2- 172.21.15.44:0/4276596054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 0x55bcd8b83940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.768+0000 7f94cddd3740 1 -- 172.21.15.44:0/4276596054 >> 172.21.15.44:0/4276596054 conn(0x55bcd8b85bb0 msgr2=0x55bcd8bce730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:00.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.768+0000 7f94cddd3740 1 -- 172.21.15.44:0/4276596054 shutdown_connections 2024-04-02T13:23:00.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.768+0000 7f94cddd3740 1 -- 172.21.15.44:0/4276596054 wait complete. 2024-04-02T13:23:00.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.768+0000 7f94cddd3740 1 -- start start 2024-04-02T13:23:00.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.769+0000 7f94cddd3740 1 Processor -- start 2024-04-02T13:23:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.769+0000 7f94cddd3740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 0x55bcd8bb6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.769+0000 7f94cddd3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb0d80 0x55bcd8be28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.769+0000 7f94cddd3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb1d80 0x55bcd8ba8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.769+0000 7f94cddd3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bcd8b88150 con 0x55bcd8bb0d80 2024-04-02T13:23:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.769+0000 7f94cddd3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bcd8aa8d30 con 0x55bcd8bb1d80 2024-04-02T13:23:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.769+0000 7f94cddd3740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55bcd8b87540 con 0x55bcd8bb09a0 2024-04-02T13:23:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94cddd3740 10 client.0 _ll_register_callbacks cb 0x55bcd8aa8720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:23:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94c3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb0d80 0x55bcd8be28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94c3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb0d80 0x55bcd8be28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45996/0 (socket says 172.21.15.44:45996) 2024-04-02T13:23:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94c3fff640 1 -- 172.21.15.44:0/2032803489 learned_addr learned my addr 172.21.15.44:0/2032803489 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/2032803489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 0x55bcd8bb6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94c96ab640 1 --2- 172.21.15.44:0/2032803489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb1d80 0x55bcd8ba8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94c3fff640 1 -- 172.21.15.44:0/2032803489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 msgr2=0x55bcd8bb6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94c3fff640 1 --2- 172.21.15.44:0/2032803489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 0x55bcd8bb6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94c3fff640 1 -- 172.21.15.44:0/2032803489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb1d80 msgr2=0x55bcd8ba8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94c3fff640 1 --2- 172.21.15.44:0/2032803489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb1d80 0x55bcd8ba8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94c3fff640 1 -- 172.21.15.44:0/2032803489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcd88013f0 con 0x55bcd8bb0d80 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94c3fff640 1 --2- 172.21.15.44:0/2032803489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb0d80 0x55bcd8be28d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f94b0059260 tx=0x7f94b005da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a383495b727ffd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94a37fe640 1 -- 172.21.15.44:0/2032803489 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b0020040 con 0x55bcd8bb0d80 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94a37fe640 1 -- 172.21.15.44:0/2032803489 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f94b00552b0 con 0x55bcd8bb0d80 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94cddd3740 1 -- 172.21.15.44:0/2032803489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94bc012710 con 0x55bcd8bb0d80 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94a37fe640 1 -- 172.21.15.44:0/2032803489 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f94b0058ec0 con 0x55bcd8bb0d80 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94cddd3740 10 client.6237 fetch_fsmap learned FSMap version 25 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.770+0000 7f94cddd3740 1 -- 172.21.15.44:0/2032803489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55bcd87c10f0 con 0x55bcd8bb0d80 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.771+0000 7f94a37fe640 1 -- 172.21.15.44:0/2032803489 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94bc012710 con 0x55bcd8bb0d80 2024-04-02T13:23:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.771+0000 7f94a37fe640 1 -- 172.21.15.44:0/2032803489 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f94b0012050 con 0x55bcd8bb0d80 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.771+0000 7f94cddd3740 10 client.6237 fetch_fsmap finished waiting for FSMap version 25 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.771+0000 7f94cddd3740 10 client.6237 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.771+0000 7f94cddd3740 20 client.6237 populate_metadata read hostname 'smithi044' 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.771+0000 7f94cddd3740 1 --2- 172.21.15.44:0/2032803489 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55bcd8bb6440 0x55bcd8baba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.771+0000 7f94cddd3740 4 client.6237 mds_command: new command op to 5991 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.771+0000 7f94cddd3740 1 -- 172.21.15.44:0/2032803489 --> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcd8afad70 con 0x55bcd8bb6440 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.771+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/2032803489 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55bcd8bb6440 0x55bcd8baba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.771+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/2032803489 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55bcd8bb6440 0x55bcd8baba20 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=894ace5f773cde7a server_cookie=20379a4844061528 in_seq=0 out_seq=0 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.771+0000 7f94a37fe640 10 client.6237 ms_handle_connect on v2:172.21.15.69:6835/1745145720 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94a37fe640 1 -- 172.21.15.44:0/2032803489 <== mds.0 v2:172.21.15.69:6835/1745145720 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bcd8afad70 con 0x55bcd8bb6440 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94a37fe640 10 client.6237 handle_command_reply: tid=0 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94cddd3740 1 client.6237 shutdown 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94cddd3740 1 -- 172.21.15.44:0/2032803489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb0d80 msgr2=0x55bcd8be28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94cddd3740 1 --2- 172.21.15.44:0/2032803489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb0d80 0x55bcd8be28d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f94b0059260 tx=0x7f94b005da50 comp rx=0 tx=0).stop 2024-04-02T13:23:00.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94cddd3740 1 -- 172.21.15.44:0/2032803489 shutdown_connections 2024-04-02T13:23:00.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94cddd3740 1 --2- 172.21.15.44:0/2032803489 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55bcd8bb6440 0x55bcd8baba20 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94cddd3740 1 --2- 172.21.15.44:0/2032803489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb1d80 0x55bcd8ba8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94cddd3740 1 --2- 172.21.15.44:0/2032803489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb0d80 0x55bcd8be28d0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94cddd3740 1 --2- 172.21.15.44:0/2032803489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 0x55bcd8bb6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94a37fe640 0 client.6237 ms_handle_reset on v2:172.21.15.69:6835/1745145720 2024-04-02T13:23:00.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94cddd3740 1 -- 172.21.15.44:0/2032803489 >> 172.21.15.44:0/2032803489 conn(0x55bcd8b83e80 msgr2=0x55bcd8bd1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:00.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94cddd3740 1 -- 172.21.15.44:0/2032803489 shutdown_connections 2024-04-02T13:23:00.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.772+0000 7f94cddd3740 1 -- 172.21.15.44:0/2032803489 wait complete. 2024-04-02T13:23:00.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.773+0000 7f94cddd3740 20 client.6237 trim_cache size 0 max 16384 2024-04-02T13:23:00.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.791+0000 7f94cddd3740 1 Processor -- start 2024-04-02T13:23:00.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.791+0000 7f94cddd3740 1 -- start start 2024-04-02T13:23:00.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94cddd3740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 0x55bcd8bb6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94cddd3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb0d80 0x55bcd8bf23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94cddd3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 0x55bcd8bd1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94cddd3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f94c4200f90 con 0x55bcd8bb1d80 2024-04-02T13:23:00.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94cddd3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f94c4200000 con 0x55bcd8bb0d80 2024-04-02T13:23:00.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94cddd3740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f94c415e010 con 0x55bcd8bb09a0 2024-04-02T13:23:00.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94c96ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 0x55bcd8bd1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94c96ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 0x55bcd8bd1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46008/0 (socket says 172.21.15.44:46008) 2024-04-02T13:23:00.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94c96ab640 1 -- 172.21.15.44:0/2047355386 learned_addr learned my addr 172.21.15.44:0/2047355386 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:00.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/2047355386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 0x55bcd8bb6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94c3fff640 1 --2- 172.21.15.44:0/2047355386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb0d80 0x55bcd8bf23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94c96ab640 1 -- 172.21.15.44:0/2047355386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 msgr2=0x55bcd8bb6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:00.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94c96ab640 1 --2- 172.21.15.44:0/2047355386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 0x55bcd8bb6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94c96ab640 1 -- 172.21.15.44:0/2047355386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb0d80 msgr2=0x55bcd8bf23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:00.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94c96ab640 1 --2- 172.21.15.44:0/2047355386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb0d80 0x55bcd8bf23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94c96ab640 1 -- 172.21.15.44:0/2047355386 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcd8bb05d0 con 0x55bcd8bb1d80 2024-04-02T13:23:00.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94c3fff640 1 --2- 172.21.15.44:0/2047355386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb0d80 0x55bcd8bf23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:00.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/2047355386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 0x55bcd8bb6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:00.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94c96ab640 1 --2- 172.21.15.44:0/2047355386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 0x55bcd8bd1c40 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009fd0 tx=0x7f94bc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f027c251c40acb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:00.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94adffb640 1 -- 172.21.15.44:0/2047355386 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc019040 con 0x55bcd8bb1d80 2024-04-02T13:23:00.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94adffb640 1 -- 172.21.15.44:0/2047355386 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94bc003b90 con 0x55bcd8bb1d80 2024-04-02T13:23:00.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.792+0000 7f94adffb640 1 -- 172.21.15.44:0/2047355386 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc004040 con 0x55bcd8bb1d80 2024-04-02T13:23:00.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.793+0000 7f94cddd3740 1 -- 172.21.15.44:0/2047355386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 msgr2=0x55bcd8bd1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:00.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.793+0000 7f94cddd3740 1 --2- 172.21.15.44:0/2047355386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 0x55bcd8bd1c40 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009fd0 tx=0x7f94bc012000 comp rx=0 tx=0).stop 2024-04-02T13:23:00.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.793+0000 7f94cddd3740 1 -- 172.21.15.44:0/2047355386 shutdown_connections 2024-04-02T13:23:00.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.793+0000 7f94cddd3740 1 --2- 172.21.15.44:0/2047355386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 0x55bcd8bd1c40 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.793+0000 7f94cddd3740 1 --2- 172.21.15.44:0/2047355386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb0d80 0x55bcd8bf23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.793+0000 7f94cddd3740 1 --2- 172.21.15.44:0/2047355386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb09a0 0x55bcd8bb6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.793+0000 7f94cddd3740 1 -- 172.21.15.44:0/2047355386 >> 172.21.15.44:0/2047355386 conn(0x55bcd8b83e80 msgr2=0x55bcd8b7f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:00.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.793+0000 7f94cddd3740 1 -- 172.21.15.44:0/2047355386 shutdown_connections 2024-04-02T13:23:00.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.793+0000 7f94cddd3740 1 -- 172.21.15.44:0/2047355386 wait complete. 2024-04-02T13:23:00.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.793+0000 7f94cddd3740 1 -- start start 2024-04-02T13:23:00.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.794+0000 7f94cddd3740 1 Processor -- start 2024-04-02T13:23:00.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.794+0000 7f94cddd3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb09a0 0x55bcd8be4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.794+0000 7f94cddd3740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb0d80 0x55bcd8be5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.794+0000 7f94cddd3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 0x55bcd8be6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.794+0000 7f94cddd3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f94c41ff9b0 con 0x55bcd8bb1d80 2024-04-02T13:23:00.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.794+0000 7f94cddd3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bcd8be0e70 con 0x55bcd8bb09a0 2024-04-02T13:23:00.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.794+0000 7f94cddd3740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55bcd8be71e0 con 0x55bcd8bb0d80 2024-04-02T13:23:00.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.794+0000 7f94cddd3740 10 client.0 _ll_register_callbacks cb 0x7f94b0059560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:23:00.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.794+0000 7f94c96ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 0x55bcd8be6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94c96ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 0x55bcd8be6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46014/0 (socket says 172.21.15.44:46014) 2024-04-02T13:23:00.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94c96ab640 1 -- 172.21.15.44:0/3237958901 learned_addr learned my addr 172.21.15.44:0/3237958901 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:00.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94c3fff640 1 --2- 172.21.15.44:0/3237958901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb0d80 0x55bcd8be5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/3237958901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb09a0 0x55bcd8be4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94c96ab640 1 -- 172.21.15.44:0/3237958901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb0d80 msgr2=0x55bcd8be5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:00.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94c96ab640 1 --2- 172.21.15.44:0/3237958901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb0d80 0x55bcd8be5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94c96ab640 1 -- 172.21.15.44:0/3237958901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb09a0 msgr2=0x55bcd8be4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94c96ab640 1 --2- 172.21.15.44:0/3237958901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb09a0 0x55bcd8be4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94c96ab640 1 -- 172.21.15.44:0/3237958901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcd8be1460 con 0x55bcd8bb1d80 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94c96ab640 1 --2- 172.21.15.44:0/3237958901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 0x55bcd8be6140 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009b50 tx=0x7f94bc009450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d569084e7322d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94a37fe640 1 -- 172.21.15.44:0/3237958901 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc034060 con 0x55bcd8bb1d80 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94a37fe640 1 -- 172.21.15.44:0/3237958901 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f94bc012510 con 0x55bcd8bb1d80 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94cddd3740 1 -- 172.21.15.44:0/3237958901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94bc003b90 con 0x55bcd8bb1d80 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94a37fe640 1 -- 172.21.15.44:0/3237958901 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f94bc03a050 con 0x55bcd8bb1d80 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94cddd3740 10 client.6243 fetch_fsmap learned FSMap version 25 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94cddd3740 1 -- 172.21.15.44:0/3237958901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55bcd8bb0320 con 0x55bcd8bb1d80 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.795+0000 7f94a37fe640 1 -- 172.21.15.44:0/3237958901 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94bc003b90 con 0x55bcd8bb1d80 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.796+0000 7f94a37fe640 1 -- 172.21.15.44:0/3237958901 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f94bc03d020 con 0x55bcd8bb1d80 2024-04-02T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.796+0000 7f94cddd3740 10 client.6243 fetch_fsmap finished waiting for FSMap version 25 2024-04-02T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.796+0000 7f94cddd3740 10 client.6243 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-02T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.796+0000 7f94cddd3740 20 client.6243 populate_metadata read hostname 'smithi044' 2024-04-02T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.796+0000 7f94cddd3740 1 --2- 172.21.15.44:0/3237958901 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55bcd8c47c20 0x55bcd8be6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.796+0000 7f94cddd3740 4 client.6243 mds_command: new command op to 5991 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.796+0000 7f94cddd3740 1 -- 172.21.15.44:0/3237958901 --> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bcd8af50c0 con 0x55bcd8c47c20 2024-04-02T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.796+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/3237958901 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55bcd8c47c20 0x55bcd8be6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.796+0000 7f94a37fe640 10 client.6243 ms_handle_connect on v2:172.21.15.69:6835/1745145720 2024-04-02T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.796+0000 7f94c8eaa640 1 --2- 172.21.15.44:0/3237958901 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55bcd8c47c20 0x55bcd8be6680 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3791815d858043d server_cookie=58a9957f6886d6e6 in_seq=0 out_seq=0 2024-04-02T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.796+0000 7f94a37fe640 1 -- 172.21.15.44:0/3237958901 <== mds.0 v2:172.21.15.69:6835/1745145720 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bcd8af50c0 con 0x55bcd8c47c20 2024-04-02T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.796+0000 7f94a37fe640 10 client.6243 handle_command_reply: tid=0 2024-04-02T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.796+0000 7f94cddd3740 1 client.6243 shutdown 2024-04-02T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cddd3740 1 -- 172.21.15.44:0/3237958901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 msgr2=0x55bcd8be6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cddd3740 1 --2- 172.21.15.44:0/3237958901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 0x55bcd8be6140 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009b50 tx=0x7f94bc009450 comp rx=0 tx=0).stop 2024-04-02T13:23:00.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cddd3740 1 -- 172.21.15.44:0/3237958901 shutdown_connections 2024-04-02T13:23:00.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cddd3740 1 --2- 172.21.15.44:0/3237958901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcd8bb1d80 0x55bcd8be6140 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cddd3740 1 --2- 172.21.15.44:0/3237958901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bcd8bb0d80 0x55bcd8be5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cddd3740 1 --2- 172.21.15.44:0/3237958901 >> [v2:172.21.15.69:6835/1745145720,v1:172.21.15.69:6837/1745145720] conn(0x55bcd8c47c20 0x55bcd8be6680 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cddd3740 1 --2- 172.21.15.44:0/3237958901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcd8bb09a0 0x55bcd8be4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94a37fe640 0 client.6243 ms_handle_reset on v2:172.21.15.69:6835/1745145720 2024-04-02T13:23:00.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cddd3740 1 -- 172.21.15.44:0/3237958901 >> 172.21.15.44:0/3237958901 conn(0x55bcd8b83e80 msgr2=0x55bcd8bf23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:00.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cddd3740 1 -- 172.21.15.44:0/3237958901 shutdown_connections 2024-04-02T13:23:00.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cddd3740 1 -- 172.21.15.44:0/3237958901 wait complete. 2024-04-02T13:23:00.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cddd3740 20 client.6243 trim_cache size 0 max 16384 2024-04-02T13:23:00.807 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:23:00.807 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-02T13:23:00.807 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-02T13:23:00.807 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:23:00.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cb10e640 1 -- 172.21.15.44:0/3002677307 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f94a806c8f0 msgr2=0x7f94a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:00.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cb10e640 1 --2- 172.21.15.44:0/3002677307 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f94a806c8f0 0x7f94a806edb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f94b8036000 tx=0x7f94b8008030 comp rx=0 tx=0).stop 2024-04-02T13:23:00.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cb10e640 1 -- 172.21.15.44:0/3002677307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94c40d0a30 msgr2=0x7f94c41f74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:00.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cb10e640 1 --2- 172.21.15.44:0/3002677307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94c40d0a30 0x7f94c41f74b0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f94b000e810 tx=0x7f94b000ecd0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cb10e640 1 -- 172.21.15.44:0/3002677307 shutdown_connections 2024-04-02T13:23:00.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cb10e640 1 --2- 172.21.15.44:0/3002677307 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f94a806c8f0 0x7f94a806edb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cb10e640 1 --2- 172.21.15.44:0/3002677307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94c4143f60 0x7f94c4143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cb10e640 1 --2- 172.21.15.44:0/3002677307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94c40d0a30 0x7f94c41f74b0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cb10e640 1 --2- 172.21.15.44:0/3002677307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94c40b3990 0x7f94c4140f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:00.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cb10e640 1 -- 172.21.15.44:0/3002677307 >> 172.21.15.44:0/3002677307 conn(0x7f94c40b5d90 msgr2=0x7f94c415f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:00.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cb10e640 1 -- 172.21.15.44:0/3002677307 shutdown_connections 2024-04-02T13:23:00.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:00.797+0000 7f94cb10e640 1 -- 172.21.15.44:0/3002677307 wait complete. 2024-04-02T13:23:00.811 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T13:23:00.912 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:23:01.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.012+0000 7ff00b495640 1 Processor -- start 2024-04-02T13:23:01.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.012+0000 7ff00b495640 1 -- start start 2024-04-02T13:23:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff00b495640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff0041617e0 0x7ff0040ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff00b495640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0040cffd0 0x7ff0041629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff00b495640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0040ccc90 0x7ff0040cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff00b495640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff004162f00 con 0x7ff0040cffd0 2024-04-02T13:23:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff00b495640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff0040cef30 con 0x7ff0041617e0 2024-04-02T13:23:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff00b495640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0040cf210 con 0x7ff0040ccc90 2024-04-02T13:23:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff008a30640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0040cffd0 0x7ff0041629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff008a30640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0040cffd0 0x7ff0041629c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46020/0 (socket says 172.21.15.44:46020) 2024-04-02T13:23:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff008a30640 1 -- 172.21.15.44:0/4106225941 learned_addr learned my addr 172.21.15.44:0/4106225941 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff009a32640 1 --2- 172.21.15.44:0/4106225941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0040ccc90 0x7ff0040cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff009231640 1 --2- 172.21.15.44:0/4106225941 >> v2:172.21.15.69:3300/0 conn(0x7ff0041617e0 0x7ff0040ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff008a30640 1 -- 172.21.15.44:0/4106225941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0040ccc90 msgr2=0x7ff0040cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:01.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff008a30640 1 --2- 172.21.15.44:0/4106225941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0040ccc90 0x7ff0040cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff008a30640 1 -- 172.21.15.44:0/4106225941 >> v2:172.21.15.69:3300/0 conn(0x7ff0041617e0 msgr2=0x7ff0040ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:01.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff008a30640 1 --2- 172.21.15.44:0/4106225941 >> v2:172.21.15.69:3300/0 conn(0x7ff0041617e0 0x7ff0040ce8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.013+0000 7ff008a30640 1 -- 172.21.15.44:0/4106225941 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0040cf4f0 con 0x7ff0040cffd0 2024-04-02T13:23:01.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.014+0000 7ff008a30640 1 --2- 172.21.15.44:0/4106225941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0040cffd0 0x7ff0041629c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff004071260 tx=0x7feff402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a8df91233c650 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:01.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.014+0000 7feffbfff640 1 -- 172.21.15.44:0/4106225941 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4035020 con 0x7ff0040cffd0 2024-04-02T13:23:01.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.014+0000 7feffbfff640 1 -- 172.21.15.44:0/4106225941 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff4031870 con 0x7ff0040cffd0 2024-04-02T13:23:01.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.014+0000 7feffbfff640 1 -- 172.21.15.44:0/4106225941 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4031c40 con 0x7ff0040cffd0 2024-04-02T13:23:01.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.014+0000 7ff00b495640 1 -- 172.21.15.44:0/4106225941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0040cffd0 msgr2=0x7ff0041629c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.014+0000 7ff00b495640 1 --2- 172.21.15.44:0/4106225941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0040cffd0 0x7ff0041629c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff004071260 tx=0x7feff402b910 comp rx=0 tx=0).stop 2024-04-02T13:23:01.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.014+0000 7ff00b495640 1 -- 172.21.15.44:0/4106225941 shutdown_connections 2024-04-02T13:23:01.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.014+0000 7ff00b495640 1 --2- 172.21.15.44:0/4106225941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0040ccc90 0x7ff0040cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.014+0000 7ff00b495640 1 --2- 172.21.15.44:0/4106225941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0040cffd0 0x7ff0041629c0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.014+0000 7ff00b495640 1 --2- 172.21.15.44:0/4106225941 >> v2:172.21.15.69:3300/0 conn(0x7ff0041617e0 0x7ff0040ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.014+0000 7ff00b495640 1 -- 172.21.15.44:0/4106225941 >> 172.21.15.44:0/4106225941 conn(0x7ff0040b20e0 msgr2=0x7ff0040b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:01.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.015+0000 7ff00b495640 1 -- 172.21.15.44:0/4106225941 shutdown_connections 2024-04-02T13:23:01.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.015+0000 7ff00b495640 1 -- 172.21.15.44:0/4106225941 wait complete. 2024-04-02T13:23:01.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.016+0000 7ff00b495640 1 Processor -- start 2024-04-02T13:23:01.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.016+0000 7ff00b495640 1 -- start start 2024-04-02T13:23:01.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.016+0000 7ff00b495640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0040ccc90 0x7ff0040c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.016+0000 7ff00b495640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0040cffd0 0x7ff00415e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.016+0000 7ff00b495640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0041617e0 0x7ff0040c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.016+0000 7ff00b495640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff0040cd680 con 0x7ff0040ccc90 2024-04-02T13:23:01.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.016+0000 7ff00b495640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0040cfab0 con 0x7ff0040cffd0 2024-04-02T13:23:01.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.016+0000 7ff00b495640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff004161bc0 con 0x7ff0041617e0 2024-04-02T13:23:01.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.016+0000 7ff009231640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0040ccc90 0x7ff0040c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff009231640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0040ccc90 0x7ff0040c75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46032/0 (socket says 172.21.15.44:46032) 2024-04-02T13:23:01.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff009231640 1 -- 172.21.15.44:0/3910199796 learned_addr learned my addr 172.21.15.44:0/3910199796 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:01.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff008a30640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0040cffd0 0x7ff00415e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff009a32640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0041617e0 0x7ff0040c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff009231640 1 -- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0041617e0 msgr2=0x7ff0040c89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff009231640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0041617e0 0x7ff0040c89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff009231640 1 -- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0040cffd0 msgr2=0x7ff00415e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff009231640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0040cffd0 0x7ff00415e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff009231640 1 -- 172.21.15.44:0/3910199796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0040c85d0 con 0x7ff0040ccc90 2024-04-02T13:23:01.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff009a32640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0041617e0 0x7ff0040c89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:01.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff008a30640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0040cffd0 0x7ff00415e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff009231640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0040ccc90 0x7ff0040c75c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff00000c7f0 tx=0x7ff00000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7feff9ffb640 1 -- 172.21.15.44:0/3910199796 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff000015070 con 0x7ff0040ccc90 2024-04-02T13:23:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7feff9ffb640 1 -- 172.21.15.44:0/3910199796 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff000004240 con 0x7ff0040ccc90 2024-04-02T13:23:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff00b495640 1 -- 172.21.15.44:0/3910199796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0040c7d30 con 0x7ff0040ccc90 2024-04-02T13:23:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7feff9ffb640 1 -- 172.21.15.44:0/3910199796 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff000007620 con 0x7ff0040ccc90 2024-04-02T13:23:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.017+0000 7ff00b495640 1 -- 172.21.15.44:0/3910199796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0040c82f0 con 0x7ff0040ccc90 2024-04-02T13:23:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.018+0000 7ff00b495640 1 -- 172.21.15.44:0/3910199796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0040684f0 con 0x7ff0040ccc90 2024-04-02T13:23:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.019+0000 7feff9ffb640 1 -- 172.21.15.44:0/3910199796 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0000079d0 con 0x7ff0040ccc90 2024-04-02T13:23:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.019+0000 7feff9ffb640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fefe806c920 0x7fefe806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.019+0000 7feff9ffb640 1 -- 172.21.15.44:0/3910199796 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7ff00008bd50 con 0x7ff0040ccc90 2024-04-02T13:23:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.019+0000 7ff008a30640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fefe806c920 0x7fefe806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.020+0000 7ff008a30640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fefe806c920 0x7fefe806ede0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7feff4036000 tx=0x7feff403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.024+0000 7feff9ffb640 1 -- 172.21.15.44:0/3910199796 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff00008f030 con 0x7ff0040ccc90 2024-04-02T13:23:01.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.248+0000 7ff00b495640 1 -- 172.21.15.44:0/3910199796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0040728f0 con 0x7ff0040ccc90 2024-04-02T13:23:01.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.249+0000 7feff9ffb640 1 -- 172.21.15.44:0/3910199796 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5181 (secure 0 0 0) 0x7ff000059ff0 con 0x7ff0040ccc90 2024-04-02T13:23:01.252 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:01.252 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/4131284806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":4131284806},{"type":"v1","addr":"172.21.15.44:6833","nonce":4131284806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/4262427145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":4262427145},{"type":"v1","addr":"172.21.15.44:6835","nonce":4262427145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4112565894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4112565894},{"type":"v1","addr":"172.21.15.69:6836","nonce":4112565894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:22:22.422387+0000","modified":"2024-04-02T13:22:24.398291+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6837/1745145720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1745145720},{"type":"v1","addr":"172.21.15.69:6837","nonce":1745145720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":3}]} 2024-04-02T13:23:01.252 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-02T13:23:01.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.251+0000 7ff00b495640 1 -- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fefe806c920 msgr2=0x7fefe806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.251+0000 7ff00b495640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fefe806c920 0x7fefe806ede0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7feff4036000 tx=0x7feff403c040 comp rx=0 tx=0).stop 2024-04-02T13:23:01.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.251+0000 7ff00b495640 1 -- 172.21.15.44:0/3910199796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0040ccc90 msgr2=0x7ff0040c75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.251+0000 7ff00b495640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0040ccc90 0x7ff0040c75c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff00000c7f0 tx=0x7ff00000ccb0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.251+0000 7ff00b495640 1 -- 172.21.15.44:0/3910199796 shutdown_connections 2024-04-02T13:23:01.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.251+0000 7ff00b495640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fefe806c920 0x7fefe806ede0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.251+0000 7ff00b495640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0041617e0 0x7ff0040c89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.251+0000 7ff00b495640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0040cffd0 0x7ff00415e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.251+0000 7ff00b495640 1 --2- 172.21.15.44:0/3910199796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0040ccc90 0x7ff0040c75c0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.251+0000 7ff00b495640 1 -- 172.21.15.44:0/3910199796 >> 172.21.15.44:0/3910199796 conn(0x7ff0040b20e0 msgr2=0x7ff00415fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:01.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.251+0000 7ff00b495640 1 -- 172.21.15.44:0/3910199796 shutdown_connections 2024-04-02T13:23:01.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.251+0000 7ff00b495640 1 -- 172.21.15.44:0/3910199796 wait complete. 2024-04-02T13:23:01.265 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-04-02T13:23:01.402 INFO:teuthology.orchestra.run.smithi069.stdout:{"message":"","return_code":0} 2024-04-02T13:23:01.402 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-02T13:23:01.403 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-04-02T13:23:01.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.442+0000 7f234b2d7f40 1 Processor -- start 2024-04-02T13:23:01.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.442+0000 7f234b2d7f40 1 -- start start 2024-04-02T13:23:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f234b2d7f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563590a17830 0x563590951310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f234b2d7f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563590951850 0x563590a255c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f234b2d7f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563590a25b00 0x563590a25f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f234b2d7f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5635908f63e0 con 0x563590a17830 2024-04-02T13:23:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f234b2d7f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5635908fa430 con 0x563590951850 2024-04-02T13:23:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f234b2d7f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x563590999650 con 0x563590a25b00 2024-04-02T13:23:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f2348dd8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563590a17830 0x563590951310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f2348dd8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563590a17830 0x563590951310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46046/0 (socket says 172.21.15.44:46046) 2024-04-02T13:23:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f2348dd8640 1 -- 172.21.15.44:0/887415839 learned_addr learned my addr 172.21.15.44:0/887415839 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f23485d7640 1 --2- 172.21.15.44:0/887415839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563590951850 0x563590a255c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f23495d9640 1 --2- 172.21.15.44:0/887415839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563590a25b00 0x563590a25f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f2348dd8640 1 -- 172.21.15.44:0/887415839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563590a25b00 msgr2=0x563590a25f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f2348dd8640 1 --2- 172.21.15.44:0/887415839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563590a25b00 0x563590a25f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f2348dd8640 1 -- 172.21.15.44:0/887415839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563590951850 msgr2=0x563590a255c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f2348dd8640 1 --2- 172.21.15.44:0/887415839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563590951850 0x563590a255c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.443+0000 7f2348dd8640 1 -- 172.21.15.44:0/887415839 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563590a225c0 con 0x563590a17830 2024-04-02T13:23:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.444+0000 7f2348dd8640 1 --2- 172.21.15.44:0/887415839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563590a17830 0x563590951310 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f23400061a0 tx=0x7f2340006c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee4b698978e8b93 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.444+0000 7f23475d5640 1 -- 172.21.15.44:0/887415839 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2340018070 con 0x563590a17830 2024-04-02T13:23:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.444+0000 7f23475d5640 1 -- 172.21.15.44:0/887415839 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2340003a30 con 0x563590a17830 2024-04-02T13:23:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.444+0000 7f23475d5640 1 -- 172.21.15.44:0/887415839 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2340003e20 con 0x563590a17830 2024-04-02T13:23:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.444+0000 7f234b2d7f40 1 -- 172.21.15.44:0/887415839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563590a17830 msgr2=0x563590951310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.444+0000 7f234b2d7f40 1 --2- 172.21.15.44:0/887415839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563590a17830 0x563590951310 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f23400061a0 tx=0x7f2340006c30 comp rx=0 tx=0).stop 2024-04-02T13:23:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.444+0000 7f234b2d7f40 1 -- 172.21.15.44:0/887415839 shutdown_connections 2024-04-02T13:23:01.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.444+0000 7f234b2d7f40 1 --2- 172.21.15.44:0/887415839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563590a25b00 0x563590a25f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.444+0000 7f234b2d7f40 1 --2- 172.21.15.44:0/887415839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563590951850 0x563590a255c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.444+0000 7f234b2d7f40 1 --2- 172.21.15.44:0/887415839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563590a17830 0x563590951310 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.444+0000 7f234b2d7f40 1 -- 172.21.15.44:0/887415839 >> 172.21.15.44:0/887415839 conn(0x56359097c500 msgr2=0x56359097daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:01.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.444+0000 7f234b2d7f40 1 -- 172.21.15.44:0/887415839 shutdown_connections 2024-04-02T13:23:01.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.444+0000 7f234b2d7f40 1 -- 172.21.15.44:0/887415839 wait complete. 2024-04-02T13:23:01.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.445+0000 7f234b2d7f40 1 Processor -- start 2024-04-02T13:23:01.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.445+0000 7f234b2d7f40 1 -- start start 2024-04-02T13:23:01.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f234b2d7f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563590951850 0x563590a0ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f234b2d7f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563590a17830 0x563590abea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f234b2d7f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563590a25b00 0x563590a13730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f234b2d7f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56359094e1f0 con 0x563590a25b00 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f234b2d7f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563590951c90 con 0x563590a17830 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f234b2d7f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x563590a250e0 con 0x563590951850 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f23495d9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563590a25b00 0x563590a13730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f2348dd8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563590951850 0x563590a0ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f23495d9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563590a25b00 0x563590a13730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46062/0 (socket says 172.21.15.44:46062) 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f23495d9640 1 -- 172.21.15.44:0/917571159 learned_addr learned my addr 172.21.15.44:0/917571159 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f23485d7640 1 --2- 172.21.15.44:0/917571159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563590a17830 0x563590abea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f23495d9640 1 -- 172.21.15.44:0/917571159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563590951850 msgr2=0x563590a0ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f23495d9640 1 --2- 172.21.15.44:0/917571159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563590951850 0x563590a0ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f23495d9640 1 -- 172.21.15.44:0/917571159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563590a17830 msgr2=0x563590abea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f23495d9640 1 --2- 172.21.15.44:0/917571159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563590a17830 0x563590abea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f23495d9640 1 -- 172.21.15.44:0/917571159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563590abdf90 con 0x563590a25b00 2024-04-02T13:23:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f23495d9640 1 --2- 172.21.15.44:0/917571159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563590a25b00 0x563590a13730 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f233800b340 tx=0x7f2334006330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f23465d3640 1 -- 172.21.15.44:0/917571159 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2338001c20 con 0x563590a25b00 2024-04-02T13:23:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f23465d3640 1 -- 172.21.15.44:0/917571159 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2338064070 con 0x563590a25b00 2024-04-02T13:23:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f234b2d7f40 1 -- 172.21.15.44:0/917571159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563590ac1f40 con 0x563590a25b00 2024-04-02T13:23:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.446+0000 7f23465d3640 1 -- 172.21.15.44:0/917571159 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23380014e0 con 0x563590a25b00 2024-04-02T13:23:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.447+0000 7f234b2d7f40 1 -- 172.21.15.44:0/917571159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563590abe240 con 0x563590a25b00 2024-04-02T13:23:01.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.448+0000 7f23465d3640 1 -- 172.21.15.44:0/917571159 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f233805a050 con 0x563590a25b00 2024-04-02T13:23:01.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.448+0000 7f23465d3640 1 --2- 172.21.15.44:0/917571159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f232806b300 0x7f232806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.448+0000 7f23465d3640 1 -- 172.21.15.44:0/917571159 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f23380cbcc0 con 0x563590a25b00 2024-04-02T13:23:01.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.448+0000 7f2348dd8640 1 --2- 172.21.15.44:0/917571159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f232806b300 0x7f232806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.448+0000 7f234b2d7f40 1 --2- 172.21.15.44:0/917571159 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x563590a18d90 0x563590a1fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.448+0000 7f234b2d7f40 1 -- 172.21.15.44:0/917571159 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:1 8.7 8:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52) -- 0x563590abef80 con 0x563590a18d90 2024-04-02T13:23:01.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f23485d7640 1 --2- 172.21.15.44:0/917571159 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x563590a18d90 0x563590a1fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f2348dd8640 1 --2- 172.21.15.44:0/917571159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f232806b300 0x7f232806d7c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2340016040 tx=0x7f2340008070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f23485d7640 1 --2- 172.21.15.44:0/917571159 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x563590a18d90 0x563590a1fd70 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f23485d7640 1 -- 172.21.15.44:0/917571159 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv3 ondisk = 0) ==== 164+0+95 (crc 0 0 0) 0x7f2330009b20 con 0x563590a18d90 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f234b2d7f40 1 -- 172.21.15.44:0/917571159 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x563590a18d90 msgr2=0x563590a1fd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f234b2d7f40 1 --2- 172.21.15.44:0/917571159 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x563590a18d90 0x563590a1fd70 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f234b2d7f40 1 -- 172.21.15.44:0/917571159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f232806b300 msgr2=0x7f232806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f234b2d7f40 1 --2- 172.21.15.44:0/917571159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f232806b300 0x7f232806d7c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2340016040 tx=0x7f2340008070 comp rx=0 tx=0).stop 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f234b2d7f40 1 -- 172.21.15.44:0/917571159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563590a25b00 msgr2=0x563590a13730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f234b2d7f40 1 --2- 172.21.15.44:0/917571159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563590a25b00 0x563590a13730 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f233800b340 tx=0x7f2334006330 comp rx=0 tx=0).stop 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f23495d9640 1 -- 172.21.15.44:0/917571159 reap_dead start 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f234b2d7f40 1 -- 172.21.15.44:0/917571159 shutdown_connections 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f234b2d7f40 1 -- 172.21.15.44:0/917571159 >> 172.21.15.44:0/917571159 conn(0x56359097c500 msgr2=0x56359097da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f234b2d7f40 1 -- 172.21.15.44:0/917571159 shutdown_connections 2024-04-02T13:23:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.449+0000 7f234b2d7f40 1 -- 172.21.15.44:0/917571159 wait complete. 2024-04-02T13:23:01.458 INFO:teuthology.orchestra.run.smithi044.stdout:Yalphaparent_a 2024-04-02T13:23:01.459 DEBUG:teuthology.orchestra.run.smithi044:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-04-02T13:23:01.543 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: "ino": 1099511627777, 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: "ancestors": [ 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: "dirino": 1099511627776, 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: "dname": "alpha", 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 4 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: "dirino": 1, 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: "dname": "parent_a", 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 7 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-02T13:23:01.544 INFO:teuthology.orchestra.run.smithi044.stdout: "pool": 8, 2024-04-02T13:23:01.545 INFO:teuthology.orchestra.run.smithi044.stdout: "old_pools": [] 2024-04-02T13:23:01.545 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:23:01.545 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:01.546 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:23:01.546 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:23:01.634 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:23:01.678 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:23:01.679 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:23:01.679 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:23:01.679 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:23:01.694 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:23:01.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.832+0000 7f87dfbd6640 1 Processor -- start 2024-04-02T13:23:01.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.832+0000 7f87dfbd6640 1 -- start start 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.832+0000 7f87dfbd6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f87d815e550 0x7f87d806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dfbd6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f87d806d530 0x7f87d815e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dfbd6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87d815ee70 0x7f87d80ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dfbd6640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f87d806d970 con 0x7f87d815e550 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dfbd6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f87d815fb50 con 0x7f87d806d530 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dfbd6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f87d815f300 con 0x7f87d815ee70 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dd972640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f87d815e550 0x7f87d806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dd972640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f87d815e550 0x7f87d806cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46074/0 (socket says 172.21.15.44:46074) 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dd972640 1 -- 172.21.15.44:0/1230510099 learned_addr learned my addr 172.21.15.44:0/1230510099 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dd171640 1 --2- 172.21.15.44:0/1230510099 >> v2:172.21.15.69:3300/0 conn(0x7f87d806d530 0x7f87d815e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87de173640 1 --2- 172.21.15.44:0/1230510099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87d815ee70 0x7f87d80ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dd972640 1 -- 172.21.15.44:0/1230510099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87d815ee70 msgr2=0x7f87d80ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dd972640 1 --2- 172.21.15.44:0/1230510099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87d815ee70 0x7f87d80ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dd972640 1 -- 172.21.15.44:0/1230510099 >> v2:172.21.15.69:3300/0 conn(0x7f87d806d530 msgr2=0x7f87d815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:01.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dd972640 1 --2- 172.21.15.44:0/1230510099 >> v2:172.21.15.69:3300/0 conn(0x7f87d806d530 0x7f87d815e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.833+0000 7f87dd972640 1 -- 172.21.15.44:0/1230510099 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d806ca90 con 0x7f87d815e550 2024-04-02T13:23:01.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.834+0000 7f87dd972640 1 --2- 172.21.15.44:0/1230510099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87d815e550 0x7f87d806cff0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f87d80b4a20 tx=0x7f87d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b09e924832cc817 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:01.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.834+0000 7f87dc970640 1 -- 172.21.15.44:0/1230510099 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4034070 con 0x7f87d815e550 2024-04-02T13:23:01.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.834+0000 7f87dc970640 1 -- 172.21.15.44:0/1230510099 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87d4030860 con 0x7f87d815e550 2024-04-02T13:23:01.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.834+0000 7f87dc970640 1 -- 172.21.15.44:0/1230510099 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4030c30 con 0x7f87d815e550 2024-04-02T13:23:01.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.834+0000 7f87dfbd6640 1 -- 172.21.15.44:0/1230510099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87d815e550 msgr2=0x7f87d806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.834+0000 7f87dfbd6640 1 --2- 172.21.15.44:0/1230510099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87d815e550 0x7f87d806cff0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f87d80b4a20 tx=0x7f87d402b9a0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.834+0000 7f87dfbd6640 1 -- 172.21.15.44:0/1230510099 shutdown_connections 2024-04-02T13:23:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.834+0000 7f87dfbd6640 1 --2- 172.21.15.44:0/1230510099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87d815ee70 0x7f87d80ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.834+0000 7f87dfbd6640 1 --2- 172.21.15.44:0/1230510099 >> v2:172.21.15.69:3300/0 conn(0x7f87d806d530 0x7f87d815e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.834+0000 7f87dfbd6640 1 --2- 172.21.15.44:0/1230510099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87d815e550 0x7f87d806cff0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.834+0000 7f87dfbd6640 1 -- 172.21.15.44:0/1230510099 >> 172.21.15.44:0/1230510099 conn(0x7f87d809aa20 msgr2=0x7f87d809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.835+0000 7f87dfbd6640 1 -- 172.21.15.44:0/1230510099 shutdown_connections 2024-04-02T13:23:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.835+0000 7f87dfbd6640 1 -- 172.21.15.44:0/1230510099 wait complete. 2024-04-02T13:23:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.836+0000 7f87dfbd6640 1 Processor -- start 2024-04-02T13:23:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.836+0000 7f87dfbd6640 1 -- start start 2024-04-02T13:23:01.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.836+0000 7f87dfbd6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87d806d530 0x7f87d80b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.836+0000 7f87dfbd6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87d815e550 0x7f87d814f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.836+0000 7f87dfbd6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87d815ee70 0x7f87d814ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.836+0000 7f87dfbd6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f87d80d02b0 con 0x7f87d815e550 2024-04-02T13:23:01.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.836+0000 7f87dfbd6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f87d815f870 con 0x7f87d806d530 2024-04-02T13:23:01.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.836+0000 7f87dfbd6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f87d80d0590 con 0x7f87d815ee70 2024-04-02T13:23:01.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87dd171640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87d815e550 0x7f87d814f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87dd171640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87d815e550 0x7f87d814f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46082/0 (socket says 172.21.15.44:46082) 2024-04-02T13:23:01.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87dd171640 1 -- 172.21.15.44:0/2708868296 learned_addr learned my addr 172.21.15.44:0/2708868296 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:01.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87dd972640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87d806d530 0x7f87d80b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87de173640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87d815ee70 0x7f87d814ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87dd171640 1 -- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87d815ee70 msgr2=0x7f87d814ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87dd171640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87d815ee70 0x7f87d814ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87dd171640 1 -- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87d806d530 msgr2=0x7f87d80b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87dd171640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87d806d530 0x7f87d80b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87dd171640 1 -- 172.21.15.44:0/2708868296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d8150750 con 0x7f87d815e550 2024-04-02T13:23:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87dd171640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87d815e550 0x7f87d814f9f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f87c800d5f0 tx=0x7f87c800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87ce7fc640 1 -- 172.21.15.44:0/2708868296 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c80125f0 con 0x7f87d815e550 2024-04-02T13:23:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87ce7fc640 1 -- 172.21.15.44:0/2708868296 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c8019070 con 0x7f87d815e550 2024-04-02T13:23:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87dfbd6640 1 -- 172.21.15.44:0/2708868296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d81509f0 con 0x7f87d815e550 2024-04-02T13:23:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87ce7fc640 1 -- 172.21.15.44:0/2708868296 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c80129d0 con 0x7f87d815e550 2024-04-02T13:23:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.837+0000 7f87dfbd6640 1 -- 172.21.15.44:0/2708868296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d8150fb0 con 0x7f87d815e550 2024-04-02T13:23:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.838+0000 7f87dfbd6640 1 -- 172.21.15.44:0/2708868296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87d809b690 con 0x7f87d815e550 2024-04-02T13:23:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.839+0000 7f87ce7fc640 1 -- 172.21.15.44:0/2708868296 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f87c8012b70 con 0x7f87d815e550 2024-04-02T13:23:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.839+0000 7f87ce7fc640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f87bc06c8f0 0x7f87bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:01.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.839+0000 7f87ce7fc640 1 -- 172.21.15.44:0/2708868296 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f87c808bf90 con 0x7f87d815e550 2024-04-02T13:23:01.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.839+0000 7f87dd972640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f87bc06c8f0 0x7f87bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:01.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.840+0000 7f87dd972640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f87bc06c8f0 0x7f87bc06edb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f87d4036000 tx=0x7f87d403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:01.844+0000 7f87ce7fc640 1 -- 172.21.15.44:0/2708868296 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f87d809b690 con 0x7f87d815e550 2024-04-02T13:23:02.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.029+0000 7f87dfbd6640 1 -- 172.21.15.44:0/2708868296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87d80684f0 con 0x7f87d815e550 2024-04-02T13:23:02.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.030+0000 7f87ce7fc640 1 -- 172.21.15.44:0/2708868296 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+16894 (secure 0 0 0) 0x7f87c8059be0 con 0x7f87d815e550 2024-04-02T13:23:02.033 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:02.033 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":52,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:22:24.017854+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:22:20.662776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-02T13:22:21.332577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:23:02.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.032+0000 7f87dfbd6640 1 -- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f87bc06c8f0 msgr2=0x7f87bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.032+0000 7f87dfbd6640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f87bc06c8f0 0x7f87bc06edb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f87d4036000 tx=0x7f87d403b040 comp rx=0 tx=0).stop 2024-04-02T13:23:02.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.032+0000 7f87dfbd6640 1 -- 172.21.15.44:0/2708868296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87d815e550 msgr2=0x7f87d814f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.032+0000 7f87dfbd6640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87d815e550 0x7f87d814f9f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f87c800d5f0 tx=0x7f87c800dab0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.032+0000 7f87dfbd6640 1 -- 172.21.15.44:0/2708868296 shutdown_connections 2024-04-02T13:23:02.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.032+0000 7f87dfbd6640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f87bc06c8f0 0x7f87bc06edb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.032+0000 7f87dfbd6640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87d815ee70 0x7f87d814ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.032+0000 7f87dfbd6640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87d815e550 0x7f87d814f9f0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.032+0000 7f87dfbd6640 1 --2- 172.21.15.44:0/2708868296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87d806d530 0x7f87d80b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.032+0000 7f87dfbd6640 1 -- 172.21.15.44:0/2708868296 >> 172.21.15.44:0/2708868296 conn(0x7f87d809aa20 msgr2=0x7f87d806ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:02.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.032+0000 7f87dfbd6640 1 -- 172.21.15.44:0/2708868296 shutdown_connections 2024-04-02T13:23:02.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.032+0000 7f87dfbd6640 1 -- 172.21.15.44:0/2708868296 wait complete. 2024-04-02T13:23:02.046 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:22:20.662776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:22:21.332577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:23:02.046 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:23:02.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.148+0000 7fbd8f111640 1 Processor -- start 2024-04-02T13:23:02.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.148+0000 7fbd8f111640 1 -- start start 2024-04-02T13:23:02.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.149+0000 7fbd8f111640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbd8815a940 0x7fbd880ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.149+0000 7fbd8f111640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd88071200 0x7fbd8806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.149+0000 7fbd8f111640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd880ce9f0 con 0x7fbd880cc640 2024-04-02T13:23:02.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.149+0000 7fbd8f111640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fbd8806ef80 con 0x7fbd8815a940 2024-04-02T13:23:02.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.149+0000 7fbd8f111640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbd880ce020 con 0x7fbd88071200 2024-04-02T13:23:02.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.149+0000 7fbd8cead640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbd8815a940 0x7fbd880ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.149+0000 7fbd8d6ae640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbd880cc640 0x7fbd880acd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41218/0 (socket says 172.21.15.44:41218) 2024-04-02T13:23:02.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.149+0000 7fbd8d6ae640 1 -- 172.21.15.44:0/666626502 learned_addr learned my addr 172.21.15.44:0/666626502 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:02.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.150+0000 7fbd8cead640 1 -- 172.21.15.44:0/666626502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd88071200 msgr2=0x7fbd8806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:02.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.150+0000 7fbd87fff640 1 --2- 172.21.15.44:0/666626502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd88071200 0x7fbd8806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.150+0000 7fbd8cead640 1 --2- 172.21.15.44:0/666626502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd88071200 0x7fbd8806e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.150+0000 7fbd877fe640 1 -- 172.21.15.44:0/666626502 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 819471404 0 0) 0x7fbd880ce9f0 con 0x7fbd880cc640 2024-04-02T13:23:02.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.150+0000 7fbd8cead640 1 -- 172.21.15.44:0/666626502 >> v1:172.21.15.44:6789/0 conn(0x7fbd880cc640 legacy=0x7fbd880acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.150+0000 7fbd8cead640 1 -- 172.21.15.44:0/666626502 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8806e9c0 con 0x7fbd8815a940 2024-04-02T13:23:02.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.150+0000 7fbd87fff640 1 --2- 172.21.15.44:0/666626502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd88071200 0x7fbd8806e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:02.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.150+0000 7fbd8cead640 1 --2- 172.21.15.44:0/666626502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd8815a940 0x7fbd880ce4b0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fbd800097a0 tx=0x7fbd8002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9d66e8cf2bbbae server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:02.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.150+0000 7fbd877fe640 1 -- 172.21.15.44:0/666626502 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd800315a0 con 0x7fbd8815a940 2024-04-02T13:23:02.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.151+0000 7fbd877fe640 1 -- 172.21.15.44:0/666626502 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd80037070 con 0x7fbd8815a940 2024-04-02T13:23:02.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.151+0000 7fbd877fe640 1 -- 172.21.15.44:0/666626502 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80031960 con 0x7fbd8815a940 2024-04-02T13:23:02.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.151+0000 7fbd8f111640 1 -- 172.21.15.44:0/666626502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd8815a940 msgr2=0x7fbd880ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.151+0000 7fbd8f111640 1 --2- 172.21.15.44:0/666626502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd8815a940 0x7fbd880ce4b0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fbd800097a0 tx=0x7fbd8002b840 comp rx=0 tx=0).stop 2024-04-02T13:23:02.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.151+0000 7fbd8f111640 1 -- 172.21.15.44:0/666626502 shutdown_connections 2024-04-02T13:23:02.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.151+0000 7fbd8f111640 1 --2- 172.21.15.44:0/666626502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd88071200 0x7fbd8806e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.151+0000 7fbd8f111640 1 --2- 172.21.15.44:0/666626502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd8815a940 0x7fbd880ce4b0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.151+0000 7fbd8f111640 1 -- 172.21.15.44:0/666626502 >> 172.21.15.44:0/666626502 conn(0x7fbd8806b920 msgr2=0x7fbd8806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:02.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.151+0000 7fbd8f111640 1 -- 172.21.15.44:0/666626502 shutdown_connections 2024-04-02T13:23:02.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.151+0000 7fbd8f111640 1 -- 172.21.15.44:0/666626502 wait complete. 2024-04-02T13:23:02.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.152+0000 7fbd8f111640 1 Processor -- start 2024-04-02T13:23:02.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.152+0000 7fbd8f111640 1 -- start start 2024-04-02T13:23:02.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.153+0000 7fbd8f111640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd88071200 0x7fbd8813e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.153+0000 7fbd8f111640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd880cc640 0x7fbd88158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.153+0000 7fbd8f111640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd8815a940 0x7fbd8809bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.153+0000 7fbd8f111640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8806e600 con 0x7fbd8815a940 2024-04-02T13:23:02.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.153+0000 7fbd8f111640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbd880cdcb0 con 0x7fbd88071200 2024-04-02T13:23:02.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.153+0000 7fbd8f111640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8806e780 con 0x7fbd880cc640 2024-04-02T13:23:02.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.153+0000 7fbd8d6ae640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd8815a940 0x7fbd8809bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.153+0000 7fbd8d6ae640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd8815a940 0x7fbd8809bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46092/0 (socket says 172.21.15.44:46092) 2024-04-02T13:23:02.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.153+0000 7fbd8d6ae640 1 -- 172.21.15.44:0/58003681 learned_addr learned my addr 172.21.15.44:0/58003681 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:02.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.153+0000 7fbd87fff640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd880cc640 0x7fbd88158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.154+0000 7fbd8cead640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd88071200 0x7fbd8813e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.154+0000 7fbd8d6ae640 1 -- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd880cc640 msgr2=0x7fbd88158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.154+0000 7fbd8d6ae640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd880cc640 0x7fbd88158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.154+0000 7fbd8d6ae640 1 -- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd88071200 msgr2=0x7fbd8813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.154+0000 7fbd8d6ae640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd88071200 0x7fbd8813e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.154+0000 7fbd8d6ae640 1 -- 172.21.15.44:0/58003681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8809ae40 con 0x7fbd8815a940 2024-04-02T13:23:02.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.154+0000 7fbd8cead640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd88071200 0x7fbd8813e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:02.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.154+0000 7fbd8d6ae640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd8815a940 0x7fbd8809bee0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbd7000c450 tx=0x7fbd7000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:02.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.154+0000 7fbd857fa640 1 -- 172.21.15.44:0/58003681 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70018070 con 0x7fbd8815a940 2024-04-02T13:23:02.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.154+0000 7fbd857fa640 1 -- 172.21.15.44:0/58003681 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd70007500 con 0x7fbd8815a940 2024-04-02T13:23:02.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.154+0000 7fbd8f111640 1 -- 172.21.15.44:0/58003681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd8809b120 con 0x7fbd8815a940 2024-04-02T13:23:02.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.154+0000 7fbd857fa640 1 -- 172.21.15.44:0/58003681 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70004a20 con 0x7fbd8815a940 2024-04-02T13:23:02.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.154+0000 7fbd8f111640 1 -- 172.21.15.44:0/58003681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd8809b6e0 con 0x7fbd8815a940 2024-04-02T13:23:02.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.155+0000 7fbd8f111640 1 -- 172.21.15.44:0/58003681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd880684f0 con 0x7fbd8815a940 2024-04-02T13:23:02.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.155+0000 7fbd857fa640 1 -- 172.21.15.44:0/58003681 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbd7001b070 con 0x7fbd8815a940 2024-04-02T13:23:02.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.156+0000 7fbd857fa640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd6806c920 0x7fbd6806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.156+0000 7fbd857fa640 1 -- 172.21.15.44:0/58003681 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fbd7008af70 con 0x7fbd8815a940 2024-04-02T13:23:02.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.156+0000 7fbd8cead640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd6806c920 0x7fbd6806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.157+0000 7fbd8cead640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd6806c920 0x7fbd6806ede0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbd80036000 tx=0x7fbd80008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:02.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.160+0000 7fbd857fa640 1 -- 172.21.15.44:0/58003681 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbd70059200 con 0x7fbd8815a940 2024-04-02T13:23:02.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.341+0000 7fbd8f111640 1 -- 172.21.15.44:0/58003681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbd88153ec0 con 0x7fbd8815a940 2024-04-02T13:23:02.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.342+0000 7fbd857fa640 1 -- 172.21.15.44:0/58003681 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) ==== 93+0+10 (secure 0 0 0) 0x7fbd70058b00 con 0x7fbd8815a940 2024-04-02T13:23:02.345 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:23:02.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.344+0000 7fbd8f111640 1 -- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd6806c920 msgr2=0x7fbd6806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.344+0000 7fbd8f111640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd6806c920 0x7fbd6806ede0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbd80036000 tx=0x7fbd80008030 comp rx=0 tx=0).stop 2024-04-02T13:23:02.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.344+0000 7fbd8f111640 1 -- 172.21.15.44:0/58003681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd8815a940 msgr2=0x7fbd8809bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.344+0000 7fbd8f111640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd8815a940 0x7fbd8809bee0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbd7000c450 tx=0x7fbd7000c910 comp rx=0 tx=0).stop 2024-04-02T13:23:02.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.344+0000 7fbd8f111640 1 -- 172.21.15.44:0/58003681 shutdown_connections 2024-04-02T13:23:02.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.344+0000 7fbd8f111640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd6806c920 0x7fbd6806ede0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.344+0000 7fbd8f111640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd8815a940 0x7fbd8809bee0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.344+0000 7fbd8f111640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd880cc640 0x7fbd88158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.344+0000 7fbd8f111640 1 --2- 172.21.15.44:0/58003681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd88071200 0x7fbd8813e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.344+0000 7fbd8f111640 1 -- 172.21.15.44:0/58003681 >> 172.21.15.44:0/58003681 conn(0x7fbd8806b920 msgr2=0x7fbd880ae840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:02.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.344+0000 7fbd8f111640 1 -- 172.21.15.44:0/58003681 shutdown_connections 2024-04-02T13:23:02.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.344+0000 7fbd8f111640 1 -- 172.21.15.44:0/58003681 wait complete. 2024-04-02T13:23:02.357 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:23:02.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.459+0000 7fdec79c7640 1 Processor -- start 2024-04-02T13:23:02.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.459+0000 7fdec79c7640 1 -- start start 2024-04-02T13:23:02.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.460+0000 7fdec79c7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdec00c6b40 0x7fdec0166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.460+0000 7fdec79c7640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdec00cd760 con 0x7fdec00b4050 2024-04-02T13:23:02.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.460+0000 7fdec79c7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdec00cda40 con 0x7fdec015c260 2024-04-02T13:23:02.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.460+0000 7fdec79c7640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fdec00cdd20 con 0x7fdec00c6b40 2024-04-02T13:23:02.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.460+0000 7fdec5f64640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdec00c6b40 0x7fdec0166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.460+0000 7fdec4f62640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fdec00b4050 0x7fdec00ccaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41234/0 (socket says 172.21.15.44:41234) 2024-04-02T13:23:02.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.460+0000 7fdec4f62640 1 -- 172.21.15.44:0/572198567 learned_addr learned my addr 172.21.15.44:0/572198567 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:02.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.460+0000 7fdec5f64640 1 -- 172.21.15.44:0/572198567 >> v1:172.21.15.69:6789/0 conn(0x7fdec015c260 legacy=0x7fdec00b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.461+0000 7fdec5f64640 1 -- 172.21.15.44:0/572198567 >> v1:172.21.15.44:6789/0 conn(0x7fdec00b4050 legacy=0x7fdec00ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.461+0000 7fdebffff640 1 -- 172.21.15.44:0/572198567 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1078581813 0 0) 0x7fdec00cd760 con 0x7fdec00b4050 2024-04-02T13:23:02.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.461+0000 7fdec5f64640 1 -- 172.21.15.44:0/572198567 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec00ce000 con 0x7fdec00c6b40 2024-04-02T13:23:02.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.461+0000 7fdec5f64640 1 --2- 172.21.15.44:0/572198567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdec00c6b40 0x7fdec0166cf0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0009d90 tx=0x7fdeb002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d691d73aa76fa7b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:02.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.461+0000 7fdebffff640 1 -- 172.21.15.44:0/572198567 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0039070 con 0x7fdec00c6b40 2024-04-02T13:23:02.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.462+0000 7fdebffff640 1 -- 172.21.15.44:0/572198567 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb002cd00 con 0x7fdec00c6b40 2024-04-02T13:23:02.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.462+0000 7fdebffff640 1 -- 172.21.15.44:0/572198567 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0034660 con 0x7fdec00c6b40 2024-04-02T13:23:02.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.462+0000 7fdec79c7640 1 -- 172.21.15.44:0/572198567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdec00c6b40 msgr2=0x7fdec0166cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.462+0000 7fdec79c7640 1 --2- 172.21.15.44:0/572198567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdec00c6b40 0x7fdec0166cf0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0009d90 tx=0x7fdeb002bf50 comp rx=0 tx=0).stop 2024-04-02T13:23:02.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.462+0000 7fdec79c7640 1 -- 172.21.15.44:0/572198567 shutdown_connections 2024-04-02T13:23:02.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.462+0000 7fdec79c7640 1 --2- 172.21.15.44:0/572198567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdec00c6b40 0x7fdec0166cf0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.462+0000 7fdec79c7640 1 -- 172.21.15.44:0/572198567 >> 172.21.15.44:0/572198567 conn(0x7fdec009aa20 msgr2=0x7fdec009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:02.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.462+0000 7fdec79c7640 1 -- 172.21.15.44:0/572198567 shutdown_connections 2024-04-02T13:23:02.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.462+0000 7fdec79c7640 1 -- 172.21.15.44:0/572198567 wait complete. 2024-04-02T13:23:02.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.463+0000 7fdec79c7640 1 Processor -- start 2024-04-02T13:23:02.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.463+0000 7fdec79c7640 1 -- start start 2024-04-02T13:23:02.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec79c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec00b4050 0x7fdec00b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec79c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec00c6b40 0x7fdec015d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec79c7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdec015c260 0x7fdec0151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec79c7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdec00cd380 con 0x7fdec015c260 2024-04-02T13:23:02.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec79c7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdec00cd200 con 0x7fdec00b4050 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec79c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdec00cd500 con 0x7fdec00c6b40 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec5f64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdec015c260 0x7fdec0151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec5f64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdec015c260 0x7fdec0151020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46112/0 (socket says 172.21.15.44:46112) 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec5f64640 1 -- 172.21.15.44:0/2193693389 learned_addr learned my addr 172.21.15.44:0/2193693389 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec5763640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec00b4050 0x7fdec00b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec4f62640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec00c6b40 0x7fdec015d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec5f64640 1 -- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec00c6b40 msgr2=0x7fdec015d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec5f64640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec00c6b40 0x7fdec015d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec5f64640 1 -- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec00b4050 msgr2=0x7fdec00b53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec5f64640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec00b4050 0x7fdec00b53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec5f64640 1 -- 172.21.15.44:0/2193693389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec00b7180 con 0x7fdec015c260 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec5763640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec00b4050 0x7fdec00b53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.464+0000 7fdec4f62640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec00c6b40 0x7fdec015d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:02.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.465+0000 7fdec5f64640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdec015c260 0x7fdec0151020 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0037040 tx=0x7fdeb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:02.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.465+0000 7fdebdffb640 1 -- 172.21.15.44:0/2193693389 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0039050 con 0x7fdec015c260 2024-04-02T13:23:02.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.465+0000 7fdebdffb640 1 -- 172.21.15.44:0/2193693389 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb003a040 con 0x7fdec015c260 2024-04-02T13:23:02.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.465+0000 7fdec79c7640 1 -- 172.21.15.44:0/2193693389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec00b7460 con 0x7fdec015c260 2024-04-02T13:23:02.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.465+0000 7fdebdffb640 1 -- 172.21.15.44:0/2193693389 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb003e610 con 0x7fdec015c260 2024-04-02T13:23:02.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.465+0000 7fdec79c7640 1 -- 172.21.15.44:0/2193693389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec00b7a20 con 0x7fdec015c260 2024-04-02T13:23:02.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.466+0000 7fdec79c7640 1 -- 172.21.15.44:0/2193693389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdec00684f0 con 0x7fdec015c260 2024-04-02T13:23:02.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.466+0000 7fdebdffb640 1 -- 172.21.15.44:0/2193693389 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdeb0047020 con 0x7fdec015c260 2024-04-02T13:23:02.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.467+0000 7fdebdffb640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdeac06c8f0 0x7fdeac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.467+0000 7fdebdffb640 1 -- 172.21.15.44:0/2193693389 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fdeb00ae090 con 0x7fdec015c260 2024-04-02T13:23:02.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.467+0000 7fdec5763640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdeac06c8f0 0x7fdeac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.468+0000 7fdec5763640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdeac06c8f0 0x7fdeac06edb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8000f60 tx=0x7fdeb8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:02.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.471+0000 7fdebdffb640 1 -- 172.21.15.44:0/2193693389 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fdeb004e020 con 0x7fdec015c260 2024-04-02T13:23:02.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.654+0000 7fdec79c7640 1 -- 172.21.15.44:0/2193693389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdec00cee60 con 0x7fdec015c260 2024-04-02T13:23:02.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.654+0000 7fdebdffb640 1 -- 172.21.15.44:0/2193693389 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) ==== 104+0+11 (secure 0 0 0) 0x7fdeb007c3f0 con 0x7fdec015c260 2024-04-02T13:23:02.657 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:23:02.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.656+0000 7fdec79c7640 1 -- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdeac06c8f0 msgr2=0x7fdeac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.656+0000 7fdec79c7640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdeac06c8f0 0x7fdeac06edb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8000f60 tx=0x7fdeb8009040 comp rx=0 tx=0).stop 2024-04-02T13:23:02.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.656+0000 7fdec79c7640 1 -- 172.21.15.44:0/2193693389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdec015c260 msgr2=0x7fdec0151020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.656+0000 7fdec79c7640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdec015c260 0x7fdec0151020 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0037040 tx=0x7fdeb0004a40 comp rx=0 tx=0).stop 2024-04-02T13:23:02.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.656+0000 7fdec79c7640 1 -- 172.21.15.44:0/2193693389 shutdown_connections 2024-04-02T13:23:02.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.656+0000 7fdec79c7640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdeac06c8f0 0x7fdeac06edb0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.656+0000 7fdec79c7640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdec015c260 0x7fdec0151020 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.656+0000 7fdec79c7640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec00c6b40 0x7fdec015d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.656+0000 7fdec79c7640 1 --2- 172.21.15.44:0/2193693389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec00b4050 0x7fdec00b53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.656+0000 7fdec79c7640 1 -- 172.21.15.44:0/2193693389 >> 172.21.15.44:0/2193693389 conn(0x7fdec009aa20 msgr2=0x7fdec00d08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:02.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.656+0000 7fdec79c7640 1 -- 172.21.15.44:0/2193693389 shutdown_connections 2024-04-02T13:23:02.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.656+0000 7fdec79c7640 1 -- 172.21.15.44:0/2193693389 wait complete. 2024-04-02T13:23:02.669 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:23:02.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.771+0000 7fe2e40c7640 1 Processor -- start 2024-04-02T13:23:02.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.771+0000 7fe2e40c7640 1 -- start start 2024-04-02T13:23:02.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.772+0000 7fe2e40c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc15e640 0x7fe2dc0d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.772+0000 7fe2e40c7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe2dc0cec60 0x7fe2dc0d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.772+0000 7fe2e40c7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2dc0cf120 con 0x7fe2dc15e260 2024-04-02T13:23:02.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.772+0000 7fe2e40c7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe2dc0b5a50 con 0x7fe2dc0cec60 2024-04-02T13:23:02.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.772+0000 7fe2e40c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe2dc0d01c0 con 0x7fe2dc15e640 2024-04-02T13:23:02.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.772+0000 7fe2e2664640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe2dc0cec60 0x7fe2dc0d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.772+0000 7fe2e1e63640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe2dc15e260 0x7fe2dc0d2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:47142/0 (socket says 172.21.15.44:47142) 2024-04-02T13:23:02.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.772+0000 7fe2e1e63640 1 -- 172.21.15.44:0/1036404541 learned_addr learned my addr 172.21.15.44:0/1036404541 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:02.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.772+0000 7fe2e1662640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc15e640 0x7fe2dc0d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.773+0000 7fe2e2664640 1 -- 172.21.15.44:0/1036404541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc15e640 msgr2=0x7fe2dc0d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:02.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.773+0000 7fe2e2664640 1 --2- 172.21.15.44:0/1036404541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc15e640 0x7fe2dc0d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.773+0000 7fe2e2664640 1 -- 172.21.15.44:0/1036404541 >> v1:172.21.15.69:6789/0 conn(0x7fe2dc15e260 legacy=0x7fe2dc0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.773+0000 7fe2e2664640 1 -- 172.21.15.44:0/1036404541 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2dc0cf400 con 0x7fe2dc0cec60 2024-04-02T13:23:02.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.773+0000 7fe2e2664640 1 --2- 172.21.15.44:0/1036404541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0cec60 0x7fe2dc0d1010 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009d90 tx=0x7fe2cc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f38626018a3d8cdd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:02.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.774+0000 7fe2e0e61640 1 -- 172.21.15.44:0/1036404541 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc039070 con 0x7fe2dc0cec60 2024-04-02T13:23:02.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.774+0000 7fe2e0e61640 1 -- 172.21.15.44:0/1036404541 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2cc02cc50 con 0x7fe2dc0cec60 2024-04-02T13:23:02.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.774+0000 7fe2e0e61640 1 -- 172.21.15.44:0/1036404541 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc034440 con 0x7fe2dc0cec60 2024-04-02T13:23:02.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.774+0000 7fe2e40c7640 1 -- 172.21.15.44:0/1036404541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0cec60 msgr2=0x7fe2dc0d1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.774+0000 7fe2e40c7640 1 --2- 172.21.15.44:0/1036404541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0cec60 0x7fe2dc0d1010 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009d90 tx=0x7fe2cc02bea0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.774+0000 7fe2e40c7640 1 -- 172.21.15.44:0/1036404541 shutdown_connections 2024-04-02T13:23:02.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.774+0000 7fe2e40c7640 1 --2- 172.21.15.44:0/1036404541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0cec60 0x7fe2dc0d1010 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.774+0000 7fe2e40c7640 1 --2- 172.21.15.44:0/1036404541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc15e640 0x7fe2dc0d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.774+0000 7fe2e40c7640 1 -- 172.21.15.44:0/1036404541 >> 172.21.15.44:0/1036404541 conn(0x7fe2dc09d3c0 msgr2=0x7fe2dc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:02.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.774+0000 7fe2e40c7640 1 -- 172.21.15.44:0/1036404541 shutdown_connections 2024-04-02T13:23:02.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.774+0000 7fe2e40c7640 1 -- 172.21.15.44:0/1036404541 wait complete. 2024-04-02T13:23:02.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.775+0000 7fe2e40c7640 1 Processor -- start 2024-04-02T13:23:02.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.775+0000 7fe2e40c7640 1 -- start start 2024-04-02T13:23:02.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.776+0000 7fe2e40c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc0cec60 0x7fe2dc0cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.776+0000 7fe2e40c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc15e260 0x7fe2dc0b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.776+0000 7fe2e40c7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2dc15e640 0x7fe2dc156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.776+0000 7fe2e40c7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc0cfca0 con 0x7fe2dc15e640 2024-04-02T13:23:02.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.776+0000 7fe2e40c7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc0cf9c0 con 0x7fe2dc15e260 2024-04-02T13:23:02.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.776+0000 7fe2e40c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe2dc0d3070 con 0x7fe2dc0cec60 2024-04-02T13:23:02.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.776+0000 7fe2e2664640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2dc15e640 0x7fe2dc156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.776+0000 7fe2e2664640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2dc15e640 0x7fe2dc156f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46128/0 (socket says 172.21.15.44:46128) 2024-04-02T13:23:02.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.776+0000 7fe2e2664640 1 -- 172.21.15.44:0/3472975053 learned_addr learned my addr 172.21.15.44:0/3472975053 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:02.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.776+0000 7fe2e1e63640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc0cec60 0x7fe2dc0cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.776+0000 7fe2e1662640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc15e260 0x7fe2dc0b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.777+0000 7fe2e2664640 1 -- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc0cec60 msgr2=0x7fe2dc0cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.777+0000 7fe2e2664640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc0cec60 0x7fe2dc0cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.777+0000 7fe2e2664640 1 -- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc15e260 msgr2=0x7fe2dc0b8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.777+0000 7fe2e2664640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc15e260 0x7fe2dc0b8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.777+0000 7fe2e2664640 1 -- 172.21.15.44:0/3472975053 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2dc15f960 con 0x7fe2dc15e640 2024-04-02T13:23:02.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.777+0000 7fe2e2664640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2dc15e640 0x7fe2dc156f70 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009eb0 tx=0x7fe2cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:02.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.777+0000 7fe2da7fc640 1 -- 172.21.15.44:0/3472975053 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc039050 con 0x7fe2dc15e640 2024-04-02T13:23:02.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.777+0000 7fe2da7fc640 1 -- 172.21.15.44:0/3472975053 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2cc045070 con 0x7fe2dc15e640 2024-04-02T13:23:02.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.777+0000 7fe2e40c7640 1 -- 172.21.15.44:0/3472975053 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2dc15fc40 con 0x7fe2dc15e640 2024-04-02T13:23:02.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.777+0000 7fe2da7fc640 1 -- 172.21.15.44:0/3472975053 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc03e610 con 0x7fe2dc15e640 2024-04-02T13:23:02.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.777+0000 7fe2e40c7640 1 -- 172.21.15.44:0/3472975053 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2dc160200 con 0x7fe2dc15e640 2024-04-02T13:23:02.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.778+0000 7fe2e40c7640 1 -- 172.21.15.44:0/3472975053 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2dc06cb70 con 0x7fe2dc15e640 2024-04-02T13:23:02.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.778+0000 7fe2da7fc640 1 -- 172.21.15.44:0/3472975053 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2cc03e7b0 con 0x7fe2dc15e640 2024-04-02T13:23:02.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.779+0000 7fe2da7fc640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe2c806c8f0 0x7fe2c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:02.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.779+0000 7fe2da7fc640 1 -- 172.21.15.44:0/3472975053 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fe2cc0ae640 con 0x7fe2dc15e640 2024-04-02T13:23:02.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.779+0000 7fe2e1e63640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe2c806c8f0 0x7fe2c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:02.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.779+0000 7fe2e1e63640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe2c806c8f0 0x7fe2c806edb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4007fd0 tx=0x7fe2d4007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:02.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.783+0000 7fe2da7fc640 1 -- 172.21.15.44:0/3472975053 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe2dc06cb70 con 0x7fe2dc15e640 2024-04-02T13:23:02.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.969+0000 7fe2e40c7640 1 -- 172.21.15.44:0/3472975053 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe2dc0684f0 con 0x7fe2dc15e640 2024-04-02T13:23:02.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.969+0000 7fe2da7fc640 1 -- 172.21.15.44:0/3472975053 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) ==== 100+0+11 (secure 0 0 0) 0x7fe2cc07c320 con 0x7fe2dc15e640 2024-04-02T13:23:02.972 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:23:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.971+0000 7fe2e40c7640 1 -- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe2c806c8f0 msgr2=0x7fe2c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.971+0000 7fe2e40c7640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe2c806c8f0 0x7fe2c806edb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4007fd0 tx=0x7fe2d4007e80 comp rx=0 tx=0).stop 2024-04-02T13:23:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.971+0000 7fe2e40c7640 1 -- 172.21.15.44:0/3472975053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2dc15e640 msgr2=0x7fe2dc156f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.971+0000 7fe2e40c7640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2dc15e640 0x7fe2dc156f70 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009eb0 tx=0x7fe2cc004a40 comp rx=0 tx=0).stop 2024-04-02T13:23:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.971+0000 7fe2e40c7640 1 -- 172.21.15.44:0/3472975053 shutdown_connections 2024-04-02T13:23:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.971+0000 7fe2e40c7640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe2c806c8f0 0x7fe2c806edb0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.971+0000 7fe2e40c7640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2dc15e640 0x7fe2dc156f70 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.971+0000 7fe2e40c7640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc15e260 0x7fe2dc0b8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.971+0000 7fe2e40c7640 1 --2- 172.21.15.44:0/3472975053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc0cec60 0x7fe2dc0cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.971+0000 7fe2e40c7640 1 -- 172.21.15.44:0/3472975053 >> 172.21.15.44:0/3472975053 conn(0x7fe2dc09d3c0 msgr2=0x7fe2dc06d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.971+0000 7fe2e40c7640 1 -- 172.21.15.44:0/3472975053 shutdown_connections 2024-04-02T13:23:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:02.971+0000 7fe2e40c7640 1 -- 172.21.15.44:0/3472975053 wait complete. 2024-04-02T13:23:02.985 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:23:03.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.086+0000 7f9b69952640 1 Processor -- start 2024-04-02T13:23:03.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.086+0000 7f9b69952640 1 -- start start 2024-04-02T13:23:03.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.087+0000 7f9b69952640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640d0a30 0x7f9b6406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.087+0000 7f9b69952640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b640d0000 con 0x7f9b640d0e10 2024-04-02T13:23:03.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.087+0000 7f9b69952640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b6406c1d0 con 0x7f9b640d2040 2024-04-02T13:23:03.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.087+0000 7f9b69952640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9b6406cb00 con 0x7f9b640d0a30 2024-04-02T13:23:03.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.087+0000 7f9b627fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f9b640d0e10 0x7f9b640cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41236/0 (socket says 172.21.15.44:41236) 2024-04-02T13:23:03.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.087+0000 7f9b627fc640 1 -- 172.21.15.44:0/1020493678 learned_addr learned my addr 172.21.15.44:0/1020493678 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:03.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460528933 0 0) 0x7f9b640d0000 con 0x7f9b640d0e10 2024-04-02T13:23:03.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b62ffd640 1 --2- 172.21.15.44:0/1020493678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640d0a30 0x7f9b6406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:03.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b4c003520 con 0x7f9b640d0e10 2024-04-02T13:23:03.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3694482368 0 0) 0x7f9b4c003520 con 0x7f9b640d0e10 2024-04-02T13:23:03.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640d0a30 msgr2=0x7f9b6406cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:03.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 --2- 172.21.15.44:0/1020493678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640d0a30 0x7f9b6406cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 >> v1:172.21.15.69:6789/0 conn(0x7f9b640d2040 legacy=0x7f9b640d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:03.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6406c4b0 con 0x7f9b640d0e10 2024-04-02T13:23:03.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f9b58001670 con 0x7f9b640d0e10 2024-04-02T13:23:03.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 >> v1:172.21.15.44:6789/0 conn(0x7f9b640d0e10 legacy=0x7f9b640cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:03.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 --2- 172.21.15.44:0/1020493678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640d0a30 0x7f9b4c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 --2- 172.21.15.44:0/1020493678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b4c003f60 0x7f9b4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 --2- 172.21.15.44:0/1020493678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b4c004d40 0x7f9b4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6406cb00 con 0x7f9b4c003f60 2024-04-02T13:23:03.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c00bb50 con 0x7f9b4c004d40 2024-04-02T13:23:03.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9b4c00bd50 con 0x7f9b640d0a30 2024-04-02T13:23:03.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b627fc640 1 --2- 172.21.15.44:0/1020493678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b4c003f60 0x7f9b4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:03.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b637fe640 1 --2- 172.21.15.44:0/1020493678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b4c004d40 0x7f9b4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:03.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b62ffd640 1 --2- 172.21.15.44:0/1020493678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640d0a30 0x7f9b4c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:03.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b627fc640 1 -- 172.21.15.44:0/1020493678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640d0a30 msgr2=0x7f9b4c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:03.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b627fc640 1 --2- 172.21.15.44:0/1020493678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640d0a30 0x7f9b4c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b627fc640 1 -- 172.21.15.44:0/1020493678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b4c004d40 msgr2=0x7f9b4c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:03.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b627fc640 1 --2- 172.21.15.44:0/1020493678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b4c004d40 0x7f9b4c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.088+0000 7f9b627fc640 1 -- 172.21.15.44:0/1020493678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4c003520 con 0x7f9b4c003f60 2024-04-02T13:23:03.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.089+0000 7f9b627fc640 1 --2- 172.21.15.44:0/1020493678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b4c003f60 0x7f9b4c004800 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800c210 tx=0x7f9b4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5561bc5fb1d206e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:03.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.089+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b58007d30 con 0x7f9b4c003f60 2024-04-02T13:23:03.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.089+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b58030070 con 0x7f9b4c003f60 2024-04-02T13:23:03.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.089+0000 7f9b61ffb640 1 -- 172.21.15.44:0/1020493678 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5802cd60 con 0x7f9b4c003f60 2024-04-02T13:23:03.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.090+0000 7f9b69952640 1 -- 172.21.15.44:0/1020493678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b4c003f60 msgr2=0x7f9b4c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:03.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.090+0000 7f9b69952640 1 --2- 172.21.15.44:0/1020493678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b4c003f60 0x7f9b4c004800 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800c210 tx=0x7f9b4c005830 comp rx=0 tx=0).stop 2024-04-02T13:23:03.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.090+0000 7f9b637fe640 1 -- 172.21.15.44:0/1020493678 reap_dead start 2024-04-02T13:23:03.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.090+0000 7f9b69952640 1 -- 172.21.15.44:0/1020493678 shutdown_connections 2024-04-02T13:23:03.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.090+0000 7f9b69952640 1 -- 172.21.15.44:0/1020493678 >> 172.21.15.44:0/1020493678 conn(0x7f9b6409d470 msgr2=0x7f9b640d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:03.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.090+0000 7f9b69952640 1 -- 172.21.15.44:0/1020493678 shutdown_connections 2024-04-02T13:23:03.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.090+0000 7f9b69952640 1 -- 172.21.15.44:0/1020493678 wait complete. 2024-04-02T13:23:03.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.091+0000 7f9b69952640 1 Processor -- start 2024-04-02T13:23:03.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.091+0000 7f9b69952640 1 -- start start 2024-04-02T13:23:03.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b69952640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b4c003f60 0x7f9b64158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b69952640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b640b3b70 0x7f9b640cc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b69952640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640b3ff0 0x7f9b640cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b69952640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6415e6f0 con 0x7f9b4c003f60 2024-04-02T13:23:03.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b69952640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9b640d1270 con 0x7f9b640b3b70 2024-04-02T13:23:03.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b69952640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9b6415e870 con 0x7f9b640b3ff0 2024-04-02T13:23:03.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b62ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b4c003f60 0x7f9b64158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b62ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b4c003f60 0x7f9b64158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46146/0 (socket says 172.21.15.44:46146) 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b62ffd640 1 -- 172.21.15.44:0/1160318279 learned_addr learned my addr 172.21.15.44:0/1160318279 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b637fe640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640b3ff0 0x7f9b640cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b627fc640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b640b3b70 0x7f9b640cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b62ffd640 1 -- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640b3ff0 msgr2=0x7f9b640cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b62ffd640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640b3ff0 0x7f9b640cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b62ffd640 1 -- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b640b3b70 msgr2=0x7f9b640cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b62ffd640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b640b3b70 0x7f9b640cc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b62ffd640 1 -- 172.21.15.44:0/1160318279 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b64159dc0 con 0x7f9b4c003f60 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b637fe640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640b3ff0 0x7f9b640cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.092+0000 7f9b627fc640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b640b3b70 0x7f9b640cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.093+0000 7f9b62ffd640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b4c003f60 0x7f9b64158e90 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c011a00 tx=0x7f9b5c011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.093+0000 7f9b4bfff640 1 -- 172.21.15.44:0/1160318279 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c00fa90 con 0x7f9b4c003f60 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.093+0000 7f9b4bfff640 1 -- 172.21.15.44:0/1160318279 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b5c00fc30 con 0x7f9b4c003f60 2024-04-02T13:23:03.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.093+0000 7f9b4bfff640 1 -- 172.21.15.44:0/1160318279 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c014440 con 0x7f9b4c003f60 2024-04-02T13:23:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.093+0000 7f9b69952640 1 -- 172.21.15.44:0/1160318279 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b640b81d0 con 0x7f9b4c003f60 2024-04-02T13:23:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.093+0000 7f9b69952640 1 -- 172.21.15.44:0/1160318279 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b640b8790 con 0x7f9b4c003f60 2024-04-02T13:23:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.094+0000 7f9b69952640 1 -- 172.21.15.44:0/1160318279 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b640684f0 con 0x7f9b4c003f60 2024-04-02T13:23:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.094+0000 7f9b4bfff640 1 -- 172.21.15.44:0/1160318279 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9b5c012040 con 0x7f9b4c003f60 2024-04-02T13:23:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.095+0000 7f9b4bfff640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9b5006c8f0 0x7f9b5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.095+0000 7f9b4bfff640 1 -- 172.21.15.44:0/1160318279 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f9b5c08c380 con 0x7f9b4c003f60 2024-04-02T13:23:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.095+0000 7f9b627fc640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9b5006c8f0 0x7f9b5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:03.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.095+0000 7f9b627fc640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9b5006c8f0 0x7f9b5006edb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800c210 tx=0x7f9b58009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:03.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.099+0000 7f9b4bfff640 1 -- 172.21.15.44:0/1160318279 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9b64159dc0 con 0x7f9b4c003f60 2024-04-02T13:23:03.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.297+0000 7f9b69952640 1 -- 172.21.15.44:0/1160318279 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9b640d17c0 con 0x7f9b4c003f60 2024-04-02T13:23:03.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.297+0000 7f9b4bfff640 1 -- 172.21.15.44:0/1160318279 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v52) ==== 80+0+0 (secure 0 0 0) 0x7f9b5c019030 con 0x7f9b4c003f60 2024-04-02T13:23:03.300 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:23:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.299+0000 7f9b69952640 1 -- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9b5006c8f0 msgr2=0x7f9b5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.299+0000 7f9b69952640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9b5006c8f0 0x7f9b5006edb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800c210 tx=0x7f9b58009030 comp rx=0 tx=0).stop 2024-04-02T13:23:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.299+0000 7f9b69952640 1 -- 172.21.15.44:0/1160318279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b4c003f60 msgr2=0x7f9b64158e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.299+0000 7f9b69952640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b4c003f60 0x7f9b64158e90 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c011a00 tx=0x7f9b5c011ec0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.299+0000 7f9b69952640 1 -- 172.21.15.44:0/1160318279 shutdown_connections 2024-04-02T13:23:03.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.299+0000 7f9b69952640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9b5006c8f0 0x7f9b5006edb0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.299+0000 7f9b69952640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b640b3ff0 0x7f9b640cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.299+0000 7f9b69952640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b640b3b70 0x7f9b640cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.299+0000 7f9b69952640 1 --2- 172.21.15.44:0/1160318279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b4c003f60 0x7f9b64158e90 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.299+0000 7f9b69952640 1 -- 172.21.15.44:0/1160318279 >> 172.21.15.44:0/1160318279 conn(0x7f9b6409d470 msgr2=0x7f9b64161c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:03.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.299+0000 7f9b69952640 1 -- 172.21.15.44:0/1160318279 shutdown_connections 2024-04-02T13:23:03.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.299+0000 7f9b69952640 1 -- 172.21.15.44:0/1160318279 wait complete. 2024-04-02T13:23:03.313 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:23:03.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:03.313 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:23:03.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[57218]: fuse finished with error 0 and tester_r 0 2024-04-02T13:23:03.339 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:23:03.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:03.339 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:23:03.507 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:23:03.507 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:23:03.507 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:23:03.525 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:23:03.525 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:23:03.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:03.525 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-02T13:23:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:23:03.583 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-02T13:23:03.583 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:23:03.583 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:23:03.583 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:23:03.598 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:23:03.598 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:23:03.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:03.599 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-02T13:23:03.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:23:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-02T13:23:03.653 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:23:03.654 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:23:03.654 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:23:03.671 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:23:03.671 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:23:03.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:03.672 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-02T13:23:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:23:03.729 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-02T13:23:03.730 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:23:03.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.828+0000 7ff10359d640 1 Processor -- start 2024-04-02T13:23:03.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.829+0000 7ff10359d640 1 -- start start 2024-04-02T13:23:03.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.829+0000 7ff10359d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0fc0d0a30 0x7ff0fc06cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.829+0000 7ff10359d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff0fc0d0e10 0x7ff0fc0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.829+0000 7ff10359d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0cfe30 0x7ff0fc0d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.829+0000 7ff10359d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff0fc06cb00 con 0x7ff0fc0d0a30 2024-04-02T13:23:03.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.829+0000 7ff10359d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff0fc06c1d0 con 0x7ff0fc0d0e10 2024-04-02T13:23:03.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.829+0000 7ff10359d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0fc06c4b0 con 0x7ff0fc0cfe30 2024-04-02T13:23:03.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.830+0000 7ff101339640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0fc0d0a30 0x7ff0fc06cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:03.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.830+0000 7ff101339640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0fc0d0a30 0x7ff0fc06cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46148/0 (socket says 172.21.15.44:46148) 2024-04-02T13:23:03.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.830+0000 7ff101339640 1 -- 172.21.15.44:0/847479997 learned_addr learned my addr 172.21.15.44:0/847479997 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:03.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.830+0000 7ff101339640 1 -- 172.21.15.44:0/847479997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0cfe30 msgr2=0x7ff0fc0d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:03.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.830+0000 7ff101339640 1 --2- 172.21.15.44:0/847479997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0cfe30 0x7ff0fc0d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.830+0000 7ff101339640 1 -- 172.21.15.44:0/847479997 >> v2:172.21.15.69:3300/0 conn(0x7ff0fc0d0e10 msgr2=0x7ff0fc0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:03.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.830+0000 7ff101339640 1 --2- 172.21.15.44:0/847479997 >> v2:172.21.15.69:3300/0 conn(0x7ff0fc0d0e10 0x7ff0fc0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.830+0000 7ff101339640 1 -- 172.21.15.44:0/847479997 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc06c790 con 0x7ff0fc0d0a30 2024-04-02T13:23:03.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.831+0000 7ff101339640 1 --2- 172.21.15.44:0/847479997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc0d0a30 0x7ff0fc06cf90 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc0b6d30 tx=0x7ff0f802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15cddb96a51194e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:03.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.831+0000 7ff0f3fff640 1 -- 172.21.15.44:0/847479997 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f802e070 con 0x7ff0fc0d0a30 2024-04-02T13:23:03.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.831+0000 7ff0f3fff640 1 -- 172.21.15.44:0/847479997 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f8037070 con 0x7ff0fc0d0a30 2024-04-02T13:23:03.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.831+0000 7ff0f3fff640 1 -- 172.21.15.44:0/847479997 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8031a00 con 0x7ff0fc0d0a30 2024-04-02T13:23:03.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.831+0000 7ff10359d640 1 -- 172.21.15.44:0/847479997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc0d0a30 msgr2=0x7ff0fc06cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:03.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.831+0000 7ff10359d640 1 --2- 172.21.15.44:0/847479997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc0d0a30 0x7ff0fc06cf90 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc0b6d30 tx=0x7ff0f802b880 comp rx=0 tx=0).stop 2024-04-02T13:23:03.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.831+0000 7ff10359d640 1 -- 172.21.15.44:0/847479997 shutdown_connections 2024-04-02T13:23:03.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.831+0000 7ff10359d640 1 --2- 172.21.15.44:0/847479997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0cfe30 0x7ff0fc0d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.831+0000 7ff10359d640 1 --2- 172.21.15.44:0/847479997 >> v2:172.21.15.69:3300/0 conn(0x7ff0fc0d0e10 0x7ff0fc0cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.831+0000 7ff10359d640 1 --2- 172.21.15.44:0/847479997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc0d0a30 0x7ff0fc06cf90 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.831+0000 7ff10359d640 1 -- 172.21.15.44:0/847479997 >> 172.21.15.44:0/847479997 conn(0x7ff0fc09d470 msgr2=0x7ff0fc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.831+0000 7ff10359d640 1 -- 172.21.15.44:0/847479997 shutdown_connections 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.831+0000 7ff10359d640 1 -- 172.21.15.44:0/847479997 wait complete. 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.833+0000 7ff10359d640 1 Processor -- start 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.833+0000 7ff10359d640 1 -- start start 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.833+0000 7ff10359d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0fc0cfe30 0x7ff0fc0c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.833+0000 7ff10359d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0d0a30 0x7ff0fc152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.833+0000 7ff10359d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc0d0e10 0x7ff0fc0b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.833+0000 7ff10359d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc15e870 con 0x7ff0fc0d0e10 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.833+0000 7ff10359d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc15e6f0 con 0x7ff0fc0cfe30 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.833+0000 7ff10359d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0fc0d4da0 con 0x7ff0fc0d0a30 2024-04-02T13:23:03.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.833+0000 7ff101b3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc0d0e10 0x7ff0fc0b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.833+0000 7ff101b3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc0d0e10 0x7ff0fc0b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46152/0 (socket says 172.21.15.44:46152) 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.833+0000 7ff101b3a640 1 -- 172.21.15.44:0/1939873197 learned_addr learned my addr 172.21.15.44:0/1939873197 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff100b38640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0d0a30 0x7ff0fc152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff101339640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0fc0cfe30 0x7ff0fc0c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff101b3a640 1 -- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0d0a30 msgr2=0x7ff0fc152790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff101b3a640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0d0a30 0x7ff0fc152790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff101b3a640 1 -- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0fc0cfe30 msgr2=0x7ff0fc0c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff101b3a640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0fc0cfe30 0x7ff0fc0c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff101b3a640 1 -- 172.21.15.44:0/1939873197 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc0b6a70 con 0x7ff0fc0d0e10 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff100b38640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0d0a30 0x7ff0fc152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff101339640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0fc0cfe30 0x7ff0fc0c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff101b3a640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc0d0e10 0x7ff0fc0b51a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400b970 tx=0x7ff0f400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff0f1ffb640 1 -- 172.21.15.44:0/1939873197 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4019070 con 0x7ff0fc0d0e10 2024-04-02T13:23:03.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff0f1ffb640 1 -- 172.21.15.44:0/1939873197 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f400cb60 con 0x7ff0fc0d0e10 2024-04-02T13:23:03.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff10359d640 1 -- 172.21.15.44:0/1939873197 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0fc0b59d0 con 0x7ff0fc0d0e10 2024-04-02T13:23:03.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff0f1ffb640 1 -- 172.21.15.44:0/1939873197 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4014440 con 0x7ff0fc0d0e10 2024-04-02T13:23:03.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.834+0000 7ff10359d640 1 -- 172.21.15.44:0/1939873197 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0fc0b5f90 con 0x7ff0fc0d0e10 2024-04-02T13:23:03.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.835+0000 7ff10359d640 1 -- 172.21.15.44:0/1939873197 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0fc0684f0 con 0x7ff0fc0d0e10 2024-04-02T13:23:03.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.836+0000 7ff0f1ffb640 1 -- 172.21.15.44:0/1939873197 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0f401c070 con 0x7ff0fc0d0e10 2024-04-02T13:23:03.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.836+0000 7ff0f1ffb640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0e006c8f0 0x7ff0e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:03.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.836+0000 7ff0f1ffb640 1 -- 172.21.15.44:0/1939873197 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7ff0f408bae0 con 0x7ff0fc0d0e10 2024-04-02T13:23:03.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.837+0000 7ff101339640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0e006c8f0 0x7ff0e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:03.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.837+0000 7ff101339640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0e006c8f0 0x7ff0e006edb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8036000 tx=0x7ff0f8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:03.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:03.841+0000 7ff0f1ffb640 1 -- 172.21.15.44:0/1939873197 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff0f4059cf0 con 0x7ff0fc0d0e10 2024-04-02T13:23:04.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.068+0000 7ff10359d640 1 -- 172.21.15.44:0/1939873197 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0fc0caf50 con 0x7ff0fc0d0e10 2024-04-02T13:23:04.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.068+0000 7ff0f1ffb640 1 -- 172.21.15.44:0/1939873197 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5181 (secure 0 0 0) 0x7ff0f40595f0 con 0x7ff0fc0d0e10 2024-04-02T13:23:04.071 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:04.071 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/4131284806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":4131284806},{"type":"v1","addr":"172.21.15.44:6833","nonce":4131284806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/4262427145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":4262427145},{"type":"v1","addr":"172.21.15.44:6835","nonce":4262427145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4112565894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4112565894},{"type":"v1","addr":"172.21.15.69:6836","nonce":4112565894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:22:22.422387+0000","modified":"2024-04-02T13:22:24.398291+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6837/1745145720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1745145720},{"type":"v1","addr":"172.21.15.69:6837","nonce":1745145720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":3}]} 2024-04-02T13:23:04.071 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-02T13:23:04.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.070+0000 7ff10359d640 1 -- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0e006c8f0 msgr2=0x7ff0e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.070+0000 7ff10359d640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0e006c8f0 0x7ff0e006edb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8036000 tx=0x7ff0f8008030 comp rx=0 tx=0).stop 2024-04-02T13:23:04.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.070+0000 7ff10359d640 1 -- 172.21.15.44:0/1939873197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc0d0e10 msgr2=0x7ff0fc0b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.070+0000 7ff10359d640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc0d0e10 0x7ff0fc0b51a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400b970 tx=0x7ff0f400be30 comp rx=0 tx=0).stop 2024-04-02T13:23:04.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.071+0000 7ff10359d640 1 -- 172.21.15.44:0/1939873197 shutdown_connections 2024-04-02T13:23:04.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.071+0000 7ff10359d640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0e006c8f0 0x7ff0e006edb0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.071+0000 7ff10359d640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc0d0e10 0x7ff0fc0b51a0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.071+0000 7ff10359d640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0d0a30 0x7ff0fc152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.071+0000 7ff10359d640 1 --2- 172.21.15.44:0/1939873197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0fc0cfe30 0x7ff0fc0c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.071+0000 7ff10359d640 1 -- 172.21.15.44:0/1939873197 >> 172.21.15.44:0/1939873197 conn(0x7ff0fc09d470 msgr2=0x7ff0fc15d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:04.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.071+0000 7ff10359d640 1 -- 172.21.15.44:0/1939873197 shutdown_connections 2024-04-02T13:23:04.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.071+0000 7ff10359d640 1 -- 172.21.15.44:0/1939873197 wait complete. 2024-04-02T13:23:04.084 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:23:04.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.183+0000 7f9c3828f640 1 Processor -- start 2024-04-02T13:23:04.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.183+0000 7f9c3828f640 1 -- start start 2024-04-02T13:23:04.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c3828f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300cffd0 0x7f9c300ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c3828f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c300cd3a0 0x7f9c301629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c3828f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c300cef30 con 0x7f9c301617e0 2024-04-02T13:23:04.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c3828f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9c30162f00 con 0x7f9c300cd3a0 2024-04-02T13:23:04.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c3828f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9c300aea90 con 0x7f9c300cffd0 2024-04-02T13:23:04.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c3602b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f9c301617e0 0x7f9c300ccc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41246/0 (socket says 172.21.15.44:41246) 2024-04-02T13:23:04.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c3602b640 1 -- 172.21.15.44:0/3065268059 learned_addr learned my addr 172.21.15.44:0/3065268059 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:04.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 459879698 0 0) 0x7f9c300cef30 con 0x7f9c301617e0 2024-04-02T13:23:04.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c1c003520 con 0x7f9c301617e0 2024-04-02T13:23:04.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c3682c640 1 --2- 172.21.15.44:0/3065268059 >> v2:172.21.15.69:3300/0 conn(0x7f9c300cd3a0 0x7f9c301629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c3582a640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300cffd0 0x7f9c300ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3081668435 0 0) 0x7f9c1c003520 con 0x7f9c301617e0 2024-04-02T13:23:04.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300cffd0 msgr2=0x7f9c300ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:04.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c35029640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300cffd0 0x7f9c300ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 >> v2:172.21.15.69:3300/0 conn(0x7f9c300cd3a0 msgr2=0x7f9c301629c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:04.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.184+0000 7f9c35029640 1 --2- 172.21.15.44:0/3065268059 >> v2:172.21.15.69:3300/0 conn(0x7f9c300cd3a0 0x7f9c301629c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c300cf210 con 0x7f9c301617e0 2024-04-02T13:23:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f9c2c001670 con 0x7f9c301617e0 2024-04-02T13:23:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 >> v1:172.21.15.44:6789/0 conn(0x7f9c301617e0 legacy=0x7f9c300ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c35029640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c0039b0 0x7f9c1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c35029640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c004800 0x7f9c1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c35029640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300cffd0 0x7f9c1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9c30162f00 con 0x7f9c1c004800 2024-04-02T13:23:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9c300aea90 con 0x7f9c1c0039b0 2024-04-02T13:23:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9c1c00b9c0 con 0x7f9c300cffd0 2024-04-02T13:23:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c3582a640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c004800 0x7f9c1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c3682c640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300cffd0 0x7f9c1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c3602b640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c0039b0 0x7f9c1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c3582a640 1 -- 172.21.15.44:0/3065268059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300cffd0 msgr2=0x7f9c1c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c3582a640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300cffd0 0x7f9c1c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c3582a640 1 -- 172.21.15.44:0/3065268059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c0039b0 msgr2=0x7f9c1c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c3582a640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c0039b0 0x7f9c1c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c3582a640 1 -- 172.21.15.44:0/3065268059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c1c003520 con 0x7f9c1c004800 2024-04-02T13:23:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c3682c640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300cffd0 0x7f9c1c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:23:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.185+0000 7f9c3602b640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c0039b0 0x7f9c1c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:23:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.186+0000 7f9c3582a640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c004800 0x7f9c1c004c80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009ac0 tx=0x7f9c1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a1f063bed51e59 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.186+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20039070 con 0x7f9c1c004800 2024-04-02T13:23:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.186+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c2002c680 con 0x7f9c1c004800 2024-04-02T13:23:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.186+0000 7f9c35029640 1 -- 172.21.15.44:0/3065268059 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20009690 con 0x7f9c1c004800 2024-04-02T13:23:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.186+0000 7f9c3828f640 1 -- 172.21.15.44:0/3065268059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c004800 msgr2=0x7f9c1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.186+0000 7f9c3828f640 1 --2- 172.21.15.44:0/3065268059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c004800 0x7f9c1c004c80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009ac0 tx=0x7f9c1c005830 comp rx=0 tx=0).stop 2024-04-02T13:23:04.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.186+0000 7f9c3682c640 1 -- 172.21.15.44:0/3065268059 reap_dead start 2024-04-02T13:23:04.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.187+0000 7f9c3828f640 1 -- 172.21.15.44:0/3065268059 shutdown_connections 2024-04-02T13:23:04.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.187+0000 7f9c3828f640 1 -- 172.21.15.44:0/3065268059 >> 172.21.15.44:0/3065268059 conn(0x7f9c300b20e0 msgr2=0x7f9c300b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:04.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.187+0000 7f9c3828f640 1 -- 172.21.15.44:0/3065268059 shutdown_connections 2024-04-02T13:23:04.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.187+0000 7f9c3828f640 1 -- 172.21.15.44:0/3065268059 wait complete. 2024-04-02T13:23:04.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.188+0000 7f9c3828f640 1 Processor -- start 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.188+0000 7f9c3828f640 1 -- start start 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.188+0000 7f9c3828f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c004800 0x7f9c3014e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.188+0000 7f9c3828f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c300cb0a0 0x7f9c300cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.188+0000 7f9c3828f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300b51f0 0x7f9c300b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.188+0000 7f9c3828f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9c300cfab0 con 0x7f9c300cb0a0 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.188+0000 7f9c3828f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9c300cf7d0 con 0x7f9c1c004800 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.188+0000 7f9c3828f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9c30161bc0 con 0x7f9c300b51f0 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c3582a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c300cb0a0 0x7f9c300cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c3582a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c300cb0a0 0x7f9c300cb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46164/0 (socket says 172.21.15.44:46164) 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c3582a640 1 -- 172.21.15.44:0/2673947780 learned_addr learned my addr 172.21.15.44:0/2673947780 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c3682c640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300b51f0 0x7f9c300b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c3602b640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c004800 0x7f9c3014e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c3582a640 1 -- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300b51f0 msgr2=0x7f9c300b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c3582a640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300b51f0 0x7f9c300b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c3582a640 1 -- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c004800 msgr2=0x7f9c3014e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c3582a640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c004800 0x7f9c3014e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c3582a640 1 -- 172.21.15.44:0/2673947780 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c301f7730 con 0x7f9c300cb0a0 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c3582a640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c300cb0a0 0x7f9c300cb520 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9c20037040 tx=0x7f9c20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c26ffd640 1 -- 172.21.15.44:0/2673947780 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20042020 con 0x7f9c300cb0a0 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c26ffd640 1 -- 172.21.15.44:0/2673947780 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c2003a040 con 0x7f9c300cb0a0 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c3828f640 1 -- 172.21.15.44:0/2673947780 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c301f7a10 con 0x7f9c300cb0a0 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c26ffd640 1 -- 172.21.15.44:0/2673947780 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20039070 con 0x7f9c300cb0a0 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.189+0000 7f9c3828f640 1 -- 172.21.15.44:0/2673947780 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c301f7fd0 con 0x7f9c300cb0a0 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.190+0000 7f9c3828f640 1 -- 172.21.15.44:0/2673947780 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c300cba60 con 0x7f9c300cb0a0 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.191+0000 7f9c26ffd640 1 -- 172.21.15.44:0/2673947780 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9c20045020 con 0x7f9c300cb0a0 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.191+0000 7f9c26ffd640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c1406c920 0x7f9c1406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.191+0000 7f9c26ffd640 1 -- 172.21.15.44:0/2673947780 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f9c200ad370 con 0x7f9c300cb0a0 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.191+0000 7f9c3602b640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c1406c920 0x7f9c1406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.192+0000 7f9c3602b640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c1406c920 0x7f9c1406ede0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9c300ca5c0 tx=0x7f9c2c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:04.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.196+0000 7f9c26ffd640 1 -- 172.21.15.44:0/2673947780 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9c2004c030 con 0x7f9c300cb0a0 2024-04-02T13:23:04.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.381+0000 7f9c3828f640 1 -- 172.21.15.44:0/2673947780 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c300684f0 con 0x7f9c300cb0a0 2024-04-02T13:23:04.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.382+0000 7f9c26ffd640 1 -- 172.21.15.44:0/2673947780 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+16894 (secure 0 0 0) 0x7f9c2007b610 con 0x7f9c300cb0a0 2024-04-02T13:23:04.385 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:04.385 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":52,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:22:24.017854+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:22:20.662776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-02T13:22:21.332577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:23:04.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.384+0000 7f9c3828f640 1 -- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c1406c920 msgr2=0x7f9c1406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.384+0000 7f9c3828f640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c1406c920 0x7f9c1406ede0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9c300ca5c0 tx=0x7f9c2c009040 comp rx=0 tx=0).stop 2024-04-02T13:23:04.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.384+0000 7f9c3828f640 1 -- 172.21.15.44:0/2673947780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c300cb0a0 msgr2=0x7f9c300cb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.384+0000 7f9c3828f640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c300cb0a0 0x7f9c300cb520 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9c20037040 tx=0x7f9c20004730 comp rx=0 tx=0).stop 2024-04-02T13:23:04.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.384+0000 7f9c3828f640 1 -- 172.21.15.44:0/2673947780 shutdown_connections 2024-04-02T13:23:04.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.384+0000 7f9c3828f640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c1406c920 0x7f9c1406ede0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.384+0000 7f9c3828f640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c300b51f0 0x7f9c300b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.384+0000 7f9c3828f640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c300cb0a0 0x7f9c300cb520 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.384+0000 7f9c3828f640 1 --2- 172.21.15.44:0/2673947780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c004800 0x7f9c3014e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.384+0000 7f9c3828f640 1 -- 172.21.15.44:0/2673947780 >> 172.21.15.44:0/2673947780 conn(0x7f9c300b20e0 msgr2=0x7f9c3015f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:04.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.384+0000 7f9c3828f640 1 -- 172.21.15.44:0/2673947780 shutdown_connections 2024-04-02T13:23:04.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.384+0000 7f9c3828f640 1 -- 172.21.15.44:0/2673947780 wait complete. 2024-04-02T13:23:04.398 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:22:20.662776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:22:21.332577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:23:04.398 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:23:04.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.500+0000 7f768550e640 1 Processor -- start 2024-04-02T13:23:04.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.500+0000 7f768550e640 1 -- start start 2024-04-02T13:23:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.500+0000 7f768550e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f768006cff0 0x7f768006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.500+0000 7f768550e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f768015e5d0 0x7f76800d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.500+0000 7f768550e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76800b4180 0x7f76800ce860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.500+0000 7f768550e640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f76800d1bf0 con 0x7f768006cff0 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.500+0000 7f768550e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f768006d910 con 0x7f768015e5d0 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.500+0000 7f768550e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f76800d1630 con 0x7f76800b4180 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.501+0000 7f767effd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f768006cff0 0x7f768006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.501+0000 7f767effd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f768006cff0 0x7f768006d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46170/0 (socket says 172.21.15.44:46170) 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.501+0000 7f767effd640 1 -- 172.21.15.44:0/2040860743 learned_addr learned my addr 172.21.15.44:0/2040860743 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.501+0000 7f767e7fc640 1 --2- 172.21.15.44:0/2040860743 >> v2:172.21.15.69:3300/0 conn(0x7f768015e5d0 0x7f76800d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.501+0000 7f767f7fe640 1 --2- 172.21.15.44:0/2040860743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76800b4180 0x7f76800ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.501+0000 7f767effd640 1 -- 172.21.15.44:0/2040860743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76800b4180 msgr2=0x7f76800ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.501+0000 7f767effd640 1 --2- 172.21.15.44:0/2040860743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76800b4180 0x7f76800ce860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.501+0000 7f767effd640 1 -- 172.21.15.44:0/2040860743 >> v2:172.21.15.69:3300/0 conn(0x7f768015e5d0 msgr2=0x7f76800d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.501+0000 7f767effd640 1 --2- 172.21.15.44:0/2040860743 >> v2:172.21.15.69:3300/0 conn(0x7f768015e5d0 0x7f76800d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.501+0000 7f767effd640 1 -- 172.21.15.44:0/2040860743 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76800d1910 con 0x7f768006cff0 2024-04-02T13:23:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.501+0000 7f767f7fe640 1 --2- 172.21.15.44:0/2040860743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76800b4180 0x7f76800ce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.501+0000 7f767e7fc640 1 --2- 172.21.15.44:0/2040860743 >> v2:172.21.15.69:3300/0 conn(0x7f768015e5d0 0x7f76800d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.502+0000 7f767effd640 1 --2- 172.21.15.44:0/2040860743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f768006cff0 0x7f768006d3d0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f7680161e40 tx=0x7f767002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fdac92a6a30b12d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.502+0000 7f767dffb640 1 -- 172.21.15.44:0/2040860743 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76700317d0 con 0x7f768006cff0 2024-04-02T13:23:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.502+0000 7f767dffb640 1 -- 172.21.15.44:0/2040860743 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7670037070 con 0x7f768006cff0 2024-04-02T13:23:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.502+0000 7f767dffb640 1 -- 172.21.15.44:0/2040860743 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7670031b90 con 0x7f768006cff0 2024-04-02T13:23:04.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.502+0000 7f768550e640 1 -- 172.21.15.44:0/2040860743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f768006cff0 msgr2=0x7f768006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.502+0000 7f768550e640 1 --2- 172.21.15.44:0/2040860743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f768006cff0 0x7f768006d3d0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f7680161e40 tx=0x7f767002b870 comp rx=0 tx=0).stop 2024-04-02T13:23:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.502+0000 7f768550e640 1 -- 172.21.15.44:0/2040860743 shutdown_connections 2024-04-02T13:23:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.502+0000 7f768550e640 1 --2- 172.21.15.44:0/2040860743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76800b4180 0x7f76800ce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.502+0000 7f768550e640 1 --2- 172.21.15.44:0/2040860743 >> v2:172.21.15.69:3300/0 conn(0x7f768015e5d0 0x7f76800d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.502+0000 7f768550e640 1 --2- 172.21.15.44:0/2040860743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f768006cff0 0x7f768006d3d0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.502+0000 7f768550e640 1 -- 172.21.15.44:0/2040860743 >> 172.21.15.44:0/2040860743 conn(0x7f768009aa20 msgr2=0x7f768009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.503+0000 7f768550e640 1 -- 172.21.15.44:0/2040860743 shutdown_connections 2024-04-02T13:23:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.503+0000 7f768550e640 1 -- 172.21.15.44:0/2040860743 wait complete. 2024-04-02T13:23:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.504+0000 7f768550e640 1 Processor -- start 2024-04-02T13:23:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.504+0000 7f768550e640 1 -- start start 2024-04-02T13:23:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.504+0000 7f768550e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f768006cff0 0x7f768015c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.504+0000 7f768550e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76800b4180 0x7f76800b5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.504+0000 7f768550e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f768015e5d0 0x7f76800b70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.504+0000 7f768550e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f76800d1270 con 0x7f76800b4180 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.504+0000 7f768550e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f76800b1820 con 0x7f768006cff0 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.504+0000 7f768550e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f76800d13f0 con 0x7f768015e5d0 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f767e7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76800b4180 0x7f76800b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f767e7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76800b4180 0x7f76800b5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46174/0 (socket says 172.21.15.44:46174) 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f767e7fc640 1 -- 172.21.15.44:0/2153698334 learned_addr learned my addr 172.21.15.44:0/2153698334 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f767effd640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f768006cff0 0x7f768015c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f767f7fe640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f768015e5d0 0x7f76800b70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f767e7fc640 1 -- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f768015e5d0 msgr2=0x7f76800b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f767e7fc640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f768015e5d0 0x7f76800b70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f767e7fc640 1 -- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f768006cff0 msgr2=0x7f768015c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f767e7fc640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f768006cff0 0x7f768015c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f767e7fc640 1 -- 172.21.15.44:0/2153698334 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76800b49d0 con 0x7f76800b4180 2024-04-02T13:23:04.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f767f7fe640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f768015e5d0 0x7f76800b70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:04.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f767e7fc640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76800b4180 0x7f76800b5790 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f767400c7f0 tx=0x7f767400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:04.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f76677fe640 1 -- 172.21.15.44:0/2153698334 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7674015070 con 0x7f76800b4180 2024-04-02T13:23:04.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f76677fe640 1 -- 172.21.15.44:0/2153698334 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7674007a30 con 0x7f76800b4180 2024-04-02T13:23:04.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f768550e640 1 -- 172.21.15.44:0/2153698334 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76800b4cb0 con 0x7f76800b4180 2024-04-02T13:23:04.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f76677fe640 1 -- 172.21.15.44:0/2153698334 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7674007e00 con 0x7f76800b4180 2024-04-02T13:23:04.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.505+0000 7f768550e640 1 -- 172.21.15.44:0/2153698334 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76800b5270 con 0x7f76800b4180 2024-04-02T13:23:04.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.506+0000 7f768550e640 1 -- 172.21.15.44:0/2153698334 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76800684f0 con 0x7f76800b4180 2024-04-02T13:23:04.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.507+0000 7f76677fe640 1 -- 172.21.15.44:0/2153698334 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7674018070 con 0x7f76800b4180 2024-04-02T13:23:04.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.507+0000 7f76677fe640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f766c06c920 0x7f766c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.507+0000 7f76677fe640 1 -- 172.21.15.44:0/2153698334 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f767408bb40 con 0x7f76800b4180 2024-04-02T13:23:04.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.507+0000 7f767effd640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f766c06c920 0x7f766c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.508+0000 7f767effd640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f766c06c920 0x7f766c06ede0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7670036000 tx=0x7f767003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:04.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.512+0000 7f76677fe640 1 -- 172.21.15.44:0/2153698334 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7674059820 con 0x7f76800b4180 2024-04-02T13:23:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.694+0000 7f768550e640 1 -- 172.21.15.44:0/2153698334 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f76800cf850 con 0x7f76800b4180 2024-04-02T13:23:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.694+0000 7f76677fe640 1 -- 172.21.15.44:0/2153698334 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) ==== 93+0+10 (secure 0 0 0) 0x7f767405db20 con 0x7f76800b4180 2024-04-02T13:23:04.697 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:23:04.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.696+0000 7f768550e640 1 -- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f766c06c920 msgr2=0x7f766c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.696+0000 7f768550e640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f766c06c920 0x7f766c06ede0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7670036000 tx=0x7f767003d040 comp rx=0 tx=0).stop 2024-04-02T13:23:04.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.696+0000 7f768550e640 1 -- 172.21.15.44:0/2153698334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76800b4180 msgr2=0x7f76800b5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.696+0000 7f768550e640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76800b4180 0x7f76800b5790 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f767400c7f0 tx=0x7f767400ccb0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.696+0000 7f768550e640 1 -- 172.21.15.44:0/2153698334 shutdown_connections 2024-04-02T13:23:04.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.696+0000 7f768550e640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f766c06c920 0x7f766c06ede0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.696+0000 7f768550e640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f768015e5d0 0x7f76800b70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.696+0000 7f768550e640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76800b4180 0x7f76800b5790 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.696+0000 7f768550e640 1 --2- 172.21.15.44:0/2153698334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f768006cff0 0x7f768015c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.696+0000 7f768550e640 1 -- 172.21.15.44:0/2153698334 >> 172.21.15.44:0/2153698334 conn(0x7f768009aa20 msgr2=0x7f76800b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:04.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.696+0000 7f768550e640 1 -- 172.21.15.44:0/2153698334 shutdown_connections 2024-04-02T13:23:04.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.696+0000 7f768550e640 1 -- 172.21.15.44:0/2153698334 wait complete. 2024-04-02T13:23:04.709 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:23:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.811+0000 7fc688408640 1 Processor -- start 2024-04-02T13:23:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.811+0000 7fc688408640 1 -- start start 2024-04-02T13:23:04.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.812+0000 7fc688408640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc68015e1d0 0x7fc6800d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.812+0000 7fc688408640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc6800cec60 0x7fc6800d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.812+0000 7fc688408640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc68015e5b0 0x7fc6800d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.812+0000 7fc688408640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc6800cfe00 con 0x7fc68015e5b0 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.812+0000 7fc688408640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc6800d1640 con 0x7fc6800cec60 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.812+0000 7fc688408640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc6800cf1f0 con 0x7fc68015e1d0 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.812+0000 7fc6869a5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc68015e5b0 0x7fc6800d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.812+0000 7fc6869a5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc68015e5b0 0x7fc6800d2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60202/0 (socket says 172.21.15.44:60202) 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.813+0000 7fc6869a5640 1 -- 172.21.15.44:0/2483370820 learned_addr learned my addr 172.21.15.44:0/2483370820 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.813+0000 7fc6861a4640 1 --2- 172.21.15.44:0/2483370820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc68015e1d0 0x7fc6800d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.813+0000 7fc6869a5640 1 -- 172.21.15.44:0/2483370820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc68015e1d0 msgr2=0x7fc6800d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.813+0000 7fc6859a3640 1 --2- 172.21.15.44:0/2483370820 >> v2:172.21.15.69:3300/0 conn(0x7fc6800cec60 0x7fc6800d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.813+0000 7fc6869a5640 1 --2- 172.21.15.44:0/2483370820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc68015e1d0 0x7fc6800d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.813+0000 7fc6869a5640 1 -- 172.21.15.44:0/2483370820 >> v2:172.21.15.69:3300/0 conn(0x7fc6800cec60 msgr2=0x7fc6800d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.813+0000 7fc6869a5640 1 --2- 172.21.15.44:0/2483370820 >> v2:172.21.15.69:3300/0 conn(0x7fc6800cec60 0x7fc6800d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.813+0000 7fc6869a5640 1 -- 172.21.15.44:0/2483370820 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6800cf4d0 con 0x7fc68015e5b0 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.813+0000 7fc6861a4640 1 --2- 172.21.15.44:0/2483370820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc68015e1d0 0x7fc6800d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.814+0000 7fc6869a5640 1 --2- 172.21.15.44:0/2483370820 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc68015e5b0 0x7fc6800d2c00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc68009dae0 tx=0x7fc67802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9deb617c18ef5105 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:04.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.814+0000 7fc6851a2640 1 -- 172.21.15.44:0/2483370820 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67802c9e0 con 0x7fc68015e5b0 2024-04-02T13:23:04.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.814+0000 7fc6851a2640 1 -- 172.21.15.44:0/2483370820 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc67802cb80 con 0x7fc68015e5b0 2024-04-02T13:23:04.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.814+0000 7fc6851a2640 1 -- 172.21.15.44:0/2483370820 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6780343f0 con 0x7fc68015e5b0 2024-04-02T13:23:04.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.814+0000 7fc688408640 1 -- 172.21.15.44:0/2483370820 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc68015e5b0 msgr2=0x7fc6800d2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.814+0000 7fc688408640 1 --2- 172.21.15.44:0/2483370820 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc68015e5b0 0x7fc6800d2c00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc68009dae0 tx=0x7fc67802be40 comp rx=0 tx=0).stop 2024-04-02T13:23:04.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.814+0000 7fc688408640 1 -- 172.21.15.44:0/2483370820 shutdown_connections 2024-04-02T13:23:04.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.814+0000 7fc688408640 1 --2- 172.21.15.44:0/2483370820 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc68015e5b0 0x7fc6800d2c00 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.814+0000 7fc688408640 1 --2- 172.21.15.44:0/2483370820 >> v2:172.21.15.69:3300/0 conn(0x7fc6800cec60 0x7fc6800d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.814+0000 7fc688408640 1 --2- 172.21.15.44:0/2483370820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc68015e1d0 0x7fc6800d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.814+0000 7fc688408640 1 -- 172.21.15.44:0/2483370820 >> 172.21.15.44:0/2483370820 conn(0x7fc68009d3b0 msgr2=0x7fc6800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.814+0000 7fc688408640 1 -- 172.21.15.44:0/2483370820 shutdown_connections 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.814+0000 7fc688408640 1 -- 172.21.15.44:0/2483370820 wait complete. 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.815+0000 7fc688408640 1 Processor -- start 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.815+0000 7fc688408640 1 -- start start 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.816+0000 7fc688408640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6800cec60 0x7fc6800cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.816+0000 7fc688408640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc68015e1d0 0x7fc6801482a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.816+0000 7fc688408640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc68015e5b0 0x7fc6800b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.816+0000 7fc688408640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc6800b4ce0 con 0x7fc6800cec60 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.816+0000 7fc688408640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc6800cfa90 con 0x7fc68015e1d0 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.816+0000 7fc688408640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc6800b4e60 con 0x7fc68015e5b0 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.816+0000 7fc6861a4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6800cec60 0x7fc6800cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.816+0000 7fc6861a4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6800cec60 0x7fc6800cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60216/0 (socket says 172.21.15.44:60216) 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.816+0000 7fc6861a4640 1 -- 172.21.15.44:0/2762966662 learned_addr learned my addr 172.21.15.44:0/2762966662 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.816+0000 7fc6869a5640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc68015e5b0 0x7fc6800b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.816+0000 7fc6859a3640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc68015e1d0 0x7fc6801482a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.817+0000 7fc6861a4640 1 -- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc68015e5b0 msgr2=0x7fc6800b8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.817+0000 7fc6861a4640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc68015e5b0 0x7fc6800b8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.817+0000 7fc6861a4640 1 -- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc68015e1d0 msgr2=0x7fc6801482a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.817+0000 7fc6861a4640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc68015e1d0 0x7fc6801482a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.817+0000 7fc6861a4640 1 -- 172.21.15.44:0/2762966662 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6800cc980 con 0x7fc6800cec60 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.817+0000 7fc6861a4640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6800cec60 0x7fc6800cda20 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00c7f0 tx=0x7fc67c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.817+0000 7fc676ffd640 1 -- 172.21.15.44:0/2762966662 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c015070 con 0x7fc6800cec60 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.817+0000 7fc676ffd640 1 -- 172.21.15.44:0/2762966662 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc67c007a30 con 0x7fc6800cec60 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.817+0000 7fc688408640 1 -- 172.21.15.44:0/2762966662 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6800ccc60 con 0x7fc6800cec60 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.817+0000 7fc676ffd640 1 -- 172.21.15.44:0/2762966662 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c007e00 con 0x7fc6800cec60 2024-04-02T13:23:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.817+0000 7fc688408640 1 -- 172.21.15.44:0/2762966662 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6800cd220 con 0x7fc6800cec60 2024-04-02T13:23:04.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.818+0000 7fc688408640 1 -- 172.21.15.44:0/2762966662 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6800684f0 con 0x7fc6800cec60 2024-04-02T13:23:04.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.819+0000 7fc676ffd640 1 -- 172.21.15.44:0/2762966662 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc67c018070 con 0x7fc6800cec60 2024-04-02T13:23:04.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.819+0000 7fc676ffd640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc66406c8f0 0x7fc66406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:04.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.819+0000 7fc676ffd640 1 -- 172.21.15.44:0/2762966662 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fc67c08bb70 con 0x7fc6800cec60 2024-04-02T13:23:04.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.819+0000 7fc6859a3640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc66406c8f0 0x7fc66406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:04.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.820+0000 7fc6859a3640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc66406c8f0 0x7fc66406edb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc6800b6eb0 tx=0x7fc670008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:04.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:04.824+0000 7fc676ffd640 1 -- 172.21.15.44:0/2762966662 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc67c0597c0 con 0x7fc6800cec60 2024-04-02T13:23:05.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.007+0000 7fc688408640 1 -- 172.21.15.44:0/2762966662 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc6800d47f0 con 0x7fc6800cec60 2024-04-02T13:23:05.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.007+0000 7fc676ffd640 1 -- 172.21.15.44:0/2762966662 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) ==== 104+0+11 (secure 0 0 0) 0x7fc67c05dac0 con 0x7fc6800cec60 2024-04-02T13:23:05.010 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:23:05.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.009+0000 7fc688408640 1 -- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc66406c8f0 msgr2=0x7fc66406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.009+0000 7fc688408640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc66406c8f0 0x7fc66406edb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc6800b6eb0 tx=0x7fc670008040 comp rx=0 tx=0).stop 2024-04-02T13:23:05.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.009+0000 7fc688408640 1 -- 172.21.15.44:0/2762966662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6800cec60 msgr2=0x7fc6800cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.009+0000 7fc688408640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6800cec60 0x7fc6800cda20 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00c7f0 tx=0x7fc67c00cbf0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.009+0000 7fc688408640 1 -- 172.21.15.44:0/2762966662 shutdown_connections 2024-04-02T13:23:05.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.009+0000 7fc688408640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc66406c8f0 0x7fc66406edb0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.009+0000 7fc688408640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc68015e5b0 0x7fc6800b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.009+0000 7fc688408640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc68015e1d0 0x7fc6801482a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.009+0000 7fc688408640 1 --2- 172.21.15.44:0/2762966662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6800cec60 0x7fc6800cda20 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.009+0000 7fc688408640 1 -- 172.21.15.44:0/2762966662 >> 172.21.15.44:0/2762966662 conn(0x7fc68009d3b0 msgr2=0x7fc68015c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:05.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.009+0000 7fc688408640 1 -- 172.21.15.44:0/2762966662 shutdown_connections 2024-04-02T13:23:05.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.009+0000 7fc688408640 1 -- 172.21.15.44:0/2762966662 wait complete. 2024-04-02T13:23:05.022 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:23:05.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.124+0000 7fb788cd9640 1 Processor -- start 2024-04-02T13:23:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.124+0000 7fb788cd9640 1 -- start start 2024-04-02T13:23:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb788cd9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb78415e190 0x7fb78406ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb788cd9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb7840d0280 0x7fb7840d0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb788cd9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb78406b570 con 0x7fb7840b04e0 2024-04-02T13:23:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb788cd9640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fb78406b850 con 0x7fb7840d0280 2024-04-02T13:23:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb788cd9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb78406bb30 con 0x7fb78415e190 2024-04-02T13:23:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb779d9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb7840d0280 0x7fb7840d0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb782d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb7840b04e0 0x7fb784072230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40932/0 (socket says 172.21.15.44:40932) 2024-04-02T13:23:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb782d9d640 1 -- 172.21.15.44:0/661587800 learned_addr learned my addr 172.21.15.44:0/661587800 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb78259c640 1 --2- 172.21.15.44:0/661587800 >> v2:172.21.15.69:3300/0 conn(0x7fb78415e190 0x7fb78406ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb781d9b640 1 -- 172.21.15.44:0/661587800 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1622910143 0 0) 0x7fb78406b570 con 0x7fb7840b04e0 2024-04-02T13:23:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb781d9b640 1 -- 172.21.15.44:0/661587800 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb770003520 con 0x7fb7840b04e0 2024-04-02T13:23:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb779d9b640 1 -- 172.21.15.44:0/661587800 >> v2:172.21.15.69:3300/0 conn(0x7fb78415e190 msgr2=0x7fb78406ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb779d9b640 1 --2- 172.21.15.44:0/661587800 >> v2:172.21.15.69:3300/0 conn(0x7fb78415e190 0x7fb78406ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb779d9b640 1 -- 172.21.15.44:0/661587800 >> v1:172.21.15.44:6789/0 conn(0x7fb7840b04e0 legacy=0x7fb784072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.125+0000 7fb779d9b640 1 -- 172.21.15.44:0/661587800 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb78406be10 con 0x7fb7840d0280 2024-04-02T13:23:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.126+0000 7fb779d9b640 1 --2- 172.21.15.44:0/661587800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7840d0280 0x7fb7840d0ff0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb7840cda50 tx=0x7fb76c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997500a1bb277dc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:05.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.127+0000 7fb781d9b640 1 -- 172.21.15.44:0/661587800 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0315a0 con 0x7fb7840d0280 2024-04-02T13:23:05.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.127+0000 7fb781d9b640 1 -- 172.21.15.44:0/661587800 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb76c037070 con 0x7fb7840d0280 2024-04-02T13:23:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.127+0000 7fb781d9b640 1 -- 172.21.15.44:0/661587800 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c031960 con 0x7fb7840d0280 2024-04-02T13:23:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.127+0000 7fb788cd9640 1 -- 172.21.15.44:0/661587800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7840d0280 msgr2=0x7fb7840d0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.127+0000 7fb788cd9640 1 --2- 172.21.15.44:0/661587800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7840d0280 0x7fb7840d0ff0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb7840cda50 tx=0x7fb76c02b840 comp rx=0 tx=0).stop 2024-04-02T13:23:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.127+0000 7fb788cd9640 1 -- 172.21.15.44:0/661587800 shutdown_connections 2024-04-02T13:23:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.127+0000 7fb788cd9640 1 --2- 172.21.15.44:0/661587800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7840d0280 0x7fb7840d0ff0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.127+0000 7fb788cd9640 1 --2- 172.21.15.44:0/661587800 >> v2:172.21.15.69:3300/0 conn(0x7fb78415e190 0x7fb78406ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.127+0000 7fb788cd9640 1 -- 172.21.15.44:0/661587800 >> 172.21.15.44:0/661587800 conn(0x7fb78406c7c0 msgr2=0x7fb78406cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.127+0000 7fb788cd9640 1 -- 172.21.15.44:0/661587800 shutdown_connections 2024-04-02T13:23:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.127+0000 7fb788cd9640 1 -- 172.21.15.44:0/661587800 wait complete. 2024-04-02T13:23:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.129+0000 7fb788cd9640 1 Processor -- start 2024-04-02T13:23:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.129+0000 7fb788cd9640 1 -- start start 2024-04-02T13:23:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb788cd9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7840b04e0 0x7fb78415f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb788cd9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7840d0280 0x7fb78413f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb788cd9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb78415e190 0x7fb7841400e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb788cd9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb7840aff90 con 0x7fb7840b04e0 2024-04-02T13:23:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb788cd9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb78415e780 con 0x7fb78415e190 2024-04-02T13:23:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb788cd9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb7840b0110 con 0x7fb7840d0280 2024-04-02T13:23:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb78259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7840b04e0 0x7fb78415f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb78259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7840b04e0 0x7fb78415f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60236/0 (socket says 172.21.15.44:60236) 2024-04-02T13:23:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb78259c640 1 -- 172.21.15.44:0/2198374475 learned_addr learned my addr 172.21.15.44:0/2198374475 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb782d9d640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb78415e190 0x7fb7841400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb779d9b640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7840d0280 0x7fb78413f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb78259c640 1 -- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7840d0280 msgr2=0x7fb78413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb78259c640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7840d0280 0x7fb78413f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb78259c640 1 -- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb78415e190 msgr2=0x7fb7841400e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb78259c640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb78415e190 0x7fb7841400e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb78259c640 1 -- 172.21.15.44:0/2198374475 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb78413fcf0 con 0x7fb7840b04e0 2024-04-02T13:23:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb779d9b640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7840d0280 0x7fb78413f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb78259c640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7840b04e0 0x7fb78415f820 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb77400e9e0 tx=0x7fb77400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb77b7fe640 1 -- 172.21.15.44:0/2198374475 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774017070 con 0x7fb7840b04e0 2024-04-02T13:23:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb77b7fe640 1 -- 172.21.15.44:0/2198374475 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb77400c9d0 con 0x7fb7840b04e0 2024-04-02T13:23:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb77b7fe640 1 -- 172.21.15.44:0/2198374475 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77400cdc0 con 0x7fb7840b04e0 2024-04-02T13:23:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.130+0000 7fb788cd9640 1 -- 172.21.15.44:0/2198374475 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb78415ff90 con 0x7fb7840b04e0 2024-04-02T13:23:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.131+0000 7fb788cd9640 1 -- 172.21.15.44:0/2198374475 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb784160550 con 0x7fb7840b04e0 2024-04-02T13:23:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.131+0000 7fb788cd9640 1 -- 172.21.15.44:0/2198374475 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb78415da00 con 0x7fb7840b04e0 2024-04-02T13:23:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.132+0000 7fb77b7fe640 1 -- 172.21.15.44:0/2198374475 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb77401a070 con 0x7fb7840b04e0 2024-04-02T13:23:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.132+0000 7fb77b7fe640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb76406c8f0 0x7fb76406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.132+0000 7fb77b7fe640 1 -- 172.21.15.44:0/2198374475 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fb77408bcc0 con 0x7fb7840b04e0 2024-04-02T13:23:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.133+0000 7fb779d9b640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb76406c8f0 0x7fb76406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.133+0000 7fb779d9b640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb76406c8f0 0x7fb76406edb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb76c036000 tx=0x7fb76c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:05.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.137+0000 7fb77b7fe640 1 -- 172.21.15.44:0/2198374475 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb78413fcf0 con 0x7fb7840b04e0 2024-04-02T13:23:05.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.321+0000 7fb788cd9640 1 -- 172.21.15.44:0/2198374475 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb7840b62c0 con 0x7fb7840b04e0 2024-04-02T13:23:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.322+0000 7fb77b7fe640 1 -- 172.21.15.44:0/2198374475 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) ==== 100+0+11 (secure 0 0 0) 0x7fb7840b62c0 con 0x7fb7840b04e0 2024-04-02T13:23:05.325 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:23:05.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.324+0000 7fb788cd9640 1 -- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb76406c8f0 msgr2=0x7fb76406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.324+0000 7fb788cd9640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb76406c8f0 0x7fb76406edb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb76c036000 tx=0x7fb76c03d040 comp rx=0 tx=0).stop 2024-04-02T13:23:05.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.324+0000 7fb788cd9640 1 -- 172.21.15.44:0/2198374475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7840b04e0 msgr2=0x7fb78415f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.324+0000 7fb788cd9640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7840b04e0 0x7fb78415f820 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb77400e9e0 tx=0x7fb77400eea0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.324+0000 7fb788cd9640 1 -- 172.21.15.44:0/2198374475 shutdown_connections 2024-04-02T13:23:05.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.324+0000 7fb788cd9640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb76406c8f0 0x7fb76406edb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.324+0000 7fb788cd9640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb78415e190 0x7fb7841400e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.324+0000 7fb788cd9640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7840d0280 0x7fb78413f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.324+0000 7fb788cd9640 1 --2- 172.21.15.44:0/2198374475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7840b04e0 0x7fb78415f820 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.324+0000 7fb788cd9640 1 -- 172.21.15.44:0/2198374475 >> 172.21.15.44:0/2198374475 conn(0x7fb78406c7c0 msgr2=0x7fb7840d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:05.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.324+0000 7fb788cd9640 1 -- 172.21.15.44:0/2198374475 shutdown_connections 2024-04-02T13:23:05.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.324+0000 7fb788cd9640 1 -- 172.21.15.44:0/2198374475 wait complete. 2024-04-02T13:23:05.337 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:23:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.435+0000 7fe8f05f6640 1 Processor -- start 2024-04-02T13:23:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.435+0000 7fe8f05f6640 1 -- start start 2024-04-02T13:23:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8f05f6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe8e815e0f0 0x7fe8e80d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8f05f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e80d0630 0x7fe8e806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8f05f6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe8e815e4d0 0x7fe8e806ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8f05f6640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe8e806dfe0 con 0x7fe8e815e4d0 2024-04-02T13:23:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8f05f6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe8e806e2c0 con 0x7fe8e815e0f0 2024-04-02T13:23:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8f05f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e806e5a0 con 0x7fe8e80d0630 2024-04-02T13:23:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8eeb93640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe8e815e4d0 0x7fe8e806ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8eeb93640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe8e815e4d0 0x7fe8e806ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60252/0 (socket says 172.21.15.44:60252) 2024-04-02T13:23:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8eeb93640 1 -- 172.21.15.44:0/2570111093 learned_addr learned my addr 172.21.15.44:0/2570111093 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8ee392640 1 --2- 172.21.15.44:0/2570111093 >> v2:172.21.15.69:3300/0 conn(0x7fe8e815e0f0 0x7fe8e80d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8edb91640 1 --2- 172.21.15.44:0/2570111093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e80d0630 0x7fe8e806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8eeb93640 1 -- 172.21.15.44:0/2570111093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e80d0630 msgr2=0x7fe8e806b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8eeb93640 1 --2- 172.21.15.44:0/2570111093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e80d0630 0x7fe8e806b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8eeb93640 1 -- 172.21.15.44:0/2570111093 >> v2:172.21.15.69:3300/0 conn(0x7fe8e815e0f0 msgr2=0x7fe8e80d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8eeb93640 1 --2- 172.21.15.44:0/2570111093 >> v2:172.21.15.69:3300/0 conn(0x7fe8e815e0f0 0x7fe8e80d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.436+0000 7fe8eeb93640 1 -- 172.21.15.44:0/2570111093 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e806e880 con 0x7fe8e815e4d0 2024-04-02T13:23:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.437+0000 7fe8eeb93640 1 --2- 172.21.15.44:0/2570111093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e815e4d0 0x7fe8e806ba10 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe8e806e4d0 tx=0x7fe8e002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a16231cf65e20590 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.437+0000 7fe8ed390640 1 -- 172.21.15.44:0/2570111093 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e002caf0 con 0x7fe8e815e4d0 2024-04-02T13:23:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.437+0000 7fe8ed390640 1 -- 172.21.15.44:0/2570111093 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8e002cc90 con 0x7fe8e815e4d0 2024-04-02T13:23:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.437+0000 7fe8ed390640 1 -- 172.21.15.44:0/2570111093 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e0034610 con 0x7fe8e815e4d0 2024-04-02T13:23:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.438+0000 7fe8f05f6640 1 -- 172.21.15.44:0/2570111093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e815e4d0 msgr2=0x7fe8e806ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.438+0000 7fe8f05f6640 1 --2- 172.21.15.44:0/2570111093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e815e4d0 0x7fe8e806ba10 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe8e806e4d0 tx=0x7fe8e002bf50 comp rx=0 tx=0).stop 2024-04-02T13:23:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.438+0000 7fe8f05f6640 1 -- 172.21.15.44:0/2570111093 shutdown_connections 2024-04-02T13:23:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.438+0000 7fe8f05f6640 1 --2- 172.21.15.44:0/2570111093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e815e4d0 0x7fe8e806ba10 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.438+0000 7fe8f05f6640 1 --2- 172.21.15.44:0/2570111093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e80d0630 0x7fe8e806b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.438+0000 7fe8f05f6640 1 --2- 172.21.15.44:0/2570111093 >> v2:172.21.15.69:3300/0 conn(0x7fe8e815e0f0 0x7fe8e80d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.438+0000 7fe8f05f6640 1 -- 172.21.15.44:0/2570111093 >> 172.21.15.44:0/2570111093 conn(0x7fe8e80b2980 msgr2=0x7fe8e80b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.438+0000 7fe8f05f6640 1 -- 172.21.15.44:0/2570111093 shutdown_connections 2024-04-02T13:23:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.438+0000 7fe8f05f6640 1 -- 172.21.15.44:0/2570111093 wait complete. 2024-04-02T13:23:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.439+0000 7fe8f05f6640 1 Processor -- start 2024-04-02T13:23:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.439+0000 7fe8f05f6640 1 -- start start 2024-04-02T13:23:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8f05f6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e80d0630 0x7fe8e8159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8f05f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e815e0f0 0x7fe8e81572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8f05f6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e815e4d0 0x7fe8e8158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8f05f6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e8072f60 con 0x7fe8e815e4d0 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8f05f6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e806eda0 con 0x7fe8e80d0630 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8f05f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e80730e0 con 0x7fe8e815e0f0 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8eeb93640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e815e4d0 0x7fe8e8158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8eeb93640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e815e4d0 0x7fe8e8158040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60260/0 (socket says 172.21.15.44:60260) 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8eeb93640 1 -- 172.21.15.44:0/1865029785 learned_addr learned my addr 172.21.15.44:0/1865029785 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8edb91640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e815e0f0 0x7fe8e81572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8ee392640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e80d0630 0x7fe8e8159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8eeb93640 1 -- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e815e0f0 msgr2=0x7fe8e81572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8eeb93640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e815e0f0 0x7fe8e81572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8eeb93640 1 -- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e80d0630 msgr2=0x7fe8e8159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8eeb93640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e80d0630 0x7fe8e8159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8eeb93640 1 -- 172.21.15.44:0/1865029785 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e8158770 con 0x7fe8e815e4d0 2024-04-02T13:23:05.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8ee392640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e80d0630 0x7fe8e8159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:05.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8eeb93640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e815e4d0 0x7fe8e8158040 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009f40 tx=0x7fe8e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:05.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8deffd640 1 -- 172.21.15.44:0/1865029785 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e0043020 con 0x7fe8e815e4d0 2024-04-02T13:23:05.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8deffd640 1 -- 172.21.15.44:0/1865029785 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8e003e450 con 0x7fe8e815e4d0 2024-04-02T13:23:05.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8f05f6640 1 -- 172.21.15.44:0/1865029785 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e8158a50 con 0x7fe8e815e4d0 2024-04-02T13:23:05.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8deffd640 1 -- 172.21.15.44:0/1865029785 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e0009510 con 0x7fe8e815e4d0 2024-04-02T13:23:05.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.440+0000 7fe8f05f6640 1 -- 172.21.15.44:0/1865029785 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e8159010 con 0x7fe8e815e4d0 2024-04-02T13:23:05.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.442+0000 7fe8f05f6640 1 -- 172.21.15.44:0/1865029785 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8e80684f0 con 0x7fe8e815e4d0 2024-04-02T13:23:05.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.442+0000 7fe8deffd640 1 -- 172.21.15.44:0/1865029785 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe8e0046020 con 0x7fe8e815e4d0 2024-04-02T13:23:05.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.442+0000 7fe8deffd640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe8cc06c8f0 0x7fe8cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.443+0000 7fe8deffd640 1 -- 172.21.15.44:0/1865029785 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fe8e00ae7c0 con 0x7fe8e815e4d0 2024-04-02T13:23:05.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.443+0000 7fe8ee392640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe8cc06c8f0 0x7fe8cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.443+0000 7fe8ee392640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe8cc06c8f0 0x7fe8cc06edb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4006fd0 tx=0x7fe8e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:05.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.447+0000 7fe8deffd640 1 -- 172.21.15.44:0/1865029785 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe8e004d020 con 0x7fe8e815e4d0 2024-04-02T13:23:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.670+0000 7fe8f05f6640 1 -- 172.21.15.44:0/1865029785 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8e8157e00 con 0x7fe8e815e4d0 2024-04-02T13:23:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.670+0000 7fe8deffd640 1 -- 172.21.15.44:0/1865029785 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5181 (secure 0 0 0) 0x7fe8e007c9d0 con 0x7fe8e815e4d0 2024-04-02T13:23:05.673 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:05.673 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/4131284806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":4131284806},{"type":"v1","addr":"172.21.15.44:6833","nonce":4131284806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/4262427145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":4262427145},{"type":"v1","addr":"172.21.15.44:6835","nonce":4262427145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4112565894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4112565894},{"type":"v1","addr":"172.21.15.69:6836","nonce":4112565894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:22:22.422387+0000","modified":"2024-04-02T13:22:24.398291+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6837/1745145720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1745145720},{"type":"v1","addr":"172.21.15.69:6837","nonce":1745145720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":3}]} 2024-04-02T13:23:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-02T13:23:05.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.672+0000 7fe8f05f6640 1 -- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe8cc06c8f0 msgr2=0x7fe8cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.672+0000 7fe8f05f6640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe8cc06c8f0 0x7fe8cc06edb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4006fd0 tx=0x7fe8e4008040 comp rx=0 tx=0).stop 2024-04-02T13:23:05.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.672+0000 7fe8f05f6640 1 -- 172.21.15.44:0/1865029785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e815e4d0 msgr2=0x7fe8e8158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.672+0000 7fe8f05f6640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e815e4d0 0x7fe8e8158040 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009f40 tx=0x7fe8e0004a40 comp rx=0 tx=0).stop 2024-04-02T13:23:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.672+0000 7fe8f05f6640 1 -- 172.21.15.44:0/1865029785 shutdown_connections 2024-04-02T13:23:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.672+0000 7fe8f05f6640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe8cc06c8f0 0x7fe8cc06edb0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.672+0000 7fe8f05f6640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e815e4d0 0x7fe8e8158040 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.672+0000 7fe8f05f6640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e815e0f0 0x7fe8e81572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.672+0000 7fe8f05f6640 1 --2- 172.21.15.44:0/1865029785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e80d0630 0x7fe8e8159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.672+0000 7fe8f05f6640 1 -- 172.21.15.44:0/1865029785 >> 172.21.15.44:0/1865029785 conn(0x7fe8e80b2980 msgr2=0x7fe8e80b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.672+0000 7fe8f05f6640 1 -- 172.21.15.44:0/1865029785 shutdown_connections 2024-04-02T13:23:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.672+0000 7fe8f05f6640 1 -- 172.21.15.44:0/1865029785 wait complete. 2024-04-02T13:23:05.686 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:23:05.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.784+0000 7f2a7052b640 1 Processor -- start 2024-04-02T13:23:05.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.784+0000 7f2a7052b640 1 -- start start 2024-04-02T13:23:05.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.785+0000 7f2a7052b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2a680b3d70 0x7f2a680b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.785+0000 7f2a7052b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a680b0c10 0x7f2a68165e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.785+0000 7f2a7052b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a680b1640 con 0x7f2a6815c1c0 2024-04-02T13:23:05.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.785+0000 7f2a7052b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2a680aef20 con 0x7f2a680b3d70 2024-04-02T13:23:05.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.785+0000 7f2a7052b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2a680af200 con 0x7f2a680b0c10 2024-04-02T13:23:05.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.785+0000 7f2a6eac8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a680b0c10 0x7f2a68165e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.785+0000 7f2a6dac6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2a680b3d70 0x7f2a680b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.785+0000 7f2a6e2c7640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2a6815c1c0 0x7f2a680affc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:40504/0 (socket says 172.21.15.44:40504) 2024-04-02T13:23:05.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.785+0000 7f2a6e2c7640 1 -- 172.21.15.44:0/640081508 learned_addr learned my addr 172.21.15.44:0/640081508 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:05.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.785+0000 7f2a6eac8640 1 -- 172.21.15.44:0/640081508 >> v2:172.21.15.69:3300/0 conn(0x7f2a680b3d70 msgr2=0x7f2a680b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:05.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.785+0000 7f2a6eac8640 1 --2- 172.21.15.44:0/640081508 >> v2:172.21.15.69:3300/0 conn(0x7f2a680b3d70 0x7f2a680b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.785+0000 7f2a6eac8640 1 -- 172.21.15.44:0/640081508 >> v1:172.21.15.69:6789/0 conn(0x7f2a6815c1c0 legacy=0x7f2a680affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.785+0000 7f2a6eac8640 1 -- 172.21.15.44:0/640081508 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a680af4e0 con 0x7f2a680b0c10 2024-04-02T13:23:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.786+0000 7f2a6eac8640 1 --2- 172.21.15.44:0/640081508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a680b0c10 0x7f2a68165e60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2a60009d90 tx=0x7f2a6002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eefd9643d93602f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.786+0000 7f2a6d2c5640 1 -- 172.21.15.44:0/640081508 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a60039070 con 0x7f2a680b0c10 2024-04-02T13:23:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.786+0000 7f2a6d2c5640 1 -- 172.21.15.44:0/640081508 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a6002cc50 con 0x7f2a680b0c10 2024-04-02T13:23:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.786+0000 7f2a6d2c5640 1 -- 172.21.15.44:0/640081508 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a60034440 con 0x7f2a680b0c10 2024-04-02T13:23:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.787+0000 7f2a7052b640 1 -- 172.21.15.44:0/640081508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a680b0c10 msgr2=0x7f2a68165e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.787+0000 7f2a7052b640 1 --2- 172.21.15.44:0/640081508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a680b0c10 0x7f2a68165e60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2a60009d90 tx=0x7f2a6002bea0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.787+0000 7f2a7052b640 1 -- 172.21.15.44:0/640081508 shutdown_connections 2024-04-02T13:23:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.787+0000 7f2a7052b640 1 --2- 172.21.15.44:0/640081508 >> v2:172.21.15.69:3300/0 conn(0x7f2a680b3d70 0x7f2a680b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.787+0000 7f2a7052b640 1 --2- 172.21.15.44:0/640081508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a680b0c10 0x7f2a68165e60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.787+0000 7f2a7052b640 1 -- 172.21.15.44:0/640081508 >> 172.21.15.44:0/640081508 conn(0x7f2a6809aa20 msgr2=0x7f2a6809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.787+0000 7f2a7052b640 1 -- 172.21.15.44:0/640081508 shutdown_connections 2024-04-02T13:23:05.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.787+0000 7f2a7052b640 1 -- 172.21.15.44:0/640081508 wait complete. 2024-04-02T13:23:05.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.788+0000 7f2a7052b640 1 Processor -- start 2024-04-02T13:23:05.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.788+0000 7f2a7052b640 1 -- start start 2024-04-02T13:23:05.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a7052b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a680b0c10 0x7f2a68151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a7052b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a680b3d70 0x7f2a680c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a7052b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a6815c1c0 0x7f2a6815d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a7052b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2a680cc990 con 0x7f2a680b0c10 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a7052b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a680afaa0 con 0x7f2a680b3d70 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a7052b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2a680aebf0 con 0x7f2a6815c1c0 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a6e2c7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a680b0c10 0x7f2a68151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a6e2c7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a680b0c10 0x7f2a68151490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60272/0 (socket says 172.21.15.44:60272) 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a6e2c7640 1 -- 172.21.15.44:0/3756685296 learned_addr learned my addr 172.21.15.44:0/3756685296 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a6eac8640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a6815c1c0 0x7f2a6815d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a6dac6640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a680b3d70 0x7f2a680c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a6e2c7640 1 -- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a6815c1c0 msgr2=0x7f2a6815d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a6e2c7640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a6815c1c0 0x7f2a6815d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a6e2c7640 1 -- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a680b3d70 msgr2=0x7f2a680c7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a6e2c7640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a680b3d70 0x7f2a680c7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a6e2c7640 1 -- 172.21.15.44:0/3756685296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6815dff0 con 0x7f2a680b0c10 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.789+0000 7f2a6e2c7640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a680b0c10 0x7f2a68151490 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2a6400c400 tx=0x7f2a6400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.790+0000 7f2a5effd640 1 -- 172.21.15.44:0/3756685296 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a64017070 con 0x7f2a680b0c10 2024-04-02T13:23:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.790+0000 7f2a5effd640 1 -- 172.21.15.44:0/3756685296 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a64005ce0 con 0x7f2a680b0c10 2024-04-02T13:23:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.790+0000 7f2a7052b640 1 -- 172.21.15.44:0/3756685296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a6815e2d0 con 0x7f2a680b0c10 2024-04-02T13:23:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.790+0000 7f2a5effd640 1 -- 172.21.15.44:0/3756685296 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a64007310 con 0x7f2a680b0c10 2024-04-02T13:23:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.790+0000 7f2a7052b640 1 -- 172.21.15.44:0/3756685296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a6815e890 con 0x7f2a680b0c10 2024-04-02T13:23:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.791+0000 7f2a7052b640 1 -- 172.21.15.44:0/3756685296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a680684f0 con 0x7f2a680b0c10 2024-04-02T13:23:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.791+0000 7f2a5effd640 1 -- 172.21.15.44:0/3756685296 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a6401a070 con 0x7f2a680b0c10 2024-04-02T13:23:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.792+0000 7f2a5effd640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a4c06c8f0 0x7f2a4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.792+0000 7f2a5effd640 1 -- 172.21.15.44:0/3756685296 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f2a6408b020 con 0x7f2a680b0c10 2024-04-02T13:23:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.792+0000 7f2a6dac6640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a4c06c8f0 0x7f2a4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.792+0000 7f2a6dac6640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a4c06c8f0 0x7f2a4c06edb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2a6809ff40 tx=0x7f2a58009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:05.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.796+0000 7f2a5effd640 1 -- 172.21.15.44:0/3756685296 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2a680b1640 con 0x7f2a680b0c10 2024-04-02T13:23:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.981+0000 7f2a7052b640 1 -- 172.21.15.44:0/3756685296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a680ce020 con 0x7f2a680b0c10 2024-04-02T13:23:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.983+0000 7f2a5effd640 1 -- 172.21.15.44:0/3756685296 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+16894 (secure 0 0 0) 0x7f2a64058c70 con 0x7f2a680b0c10 2024-04-02T13:23:05.986 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:05.986 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":52,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:22:24.017854+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:22:20.662776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-02T13:22:21.332577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:23:05.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.985+0000 7f2a7052b640 1 -- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a4c06c8f0 msgr2=0x7f2a4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.985+0000 7f2a7052b640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a4c06c8f0 0x7f2a4c06edb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2a6809ff40 tx=0x7f2a58009190 comp rx=0 tx=0).stop 2024-04-02T13:23:05.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.985+0000 7f2a7052b640 1 -- 172.21.15.44:0/3756685296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a680b0c10 msgr2=0x7f2a68151490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:05.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.985+0000 7f2a7052b640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a680b0c10 0x7f2a68151490 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2a6400c400 tx=0x7f2a6400c8c0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.985+0000 7f2a7052b640 1 -- 172.21.15.44:0/3756685296 shutdown_connections 2024-04-02T13:23:05.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.985+0000 7f2a7052b640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a4c06c8f0 0x7f2a4c06edb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.985+0000 7f2a7052b640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a6815c1c0 0x7f2a6815d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.985+0000 7f2a7052b640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a680b3d70 0x7f2a680c7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.985+0000 7f2a7052b640 1 --2- 172.21.15.44:0/3756685296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a680b0c10 0x7f2a68151490 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:05.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.985+0000 7f2a7052b640 1 -- 172.21.15.44:0/3756685296 >> 172.21.15.44:0/3756685296 conn(0x7f2a6809aa20 msgr2=0x7f2a680ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:05.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.985+0000 7f2a7052b640 1 -- 172.21.15.44:0/3756685296 shutdown_connections 2024-04-02T13:23:05.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:05.985+0000 7f2a7052b640 1 -- 172.21.15.44:0/3756685296 wait complete. 2024-04-02T13:23:05.998 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T13:23:05.998 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:23:06.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.097+0000 7f4287093640 1 Processor -- start 2024-04-02T13:23:06.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.097+0000 7f4287093640 1 -- start start 2024-04-02T13:23:06.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.098+0000 7f4287093640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f428006ece0 0x7f4280165ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.098+0000 7f4287093640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f428006dc40 con 0x7f42800cece0 2024-04-02T13:23:06.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.098+0000 7f4287093640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f428006df20 con 0x7f42800d09f0 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.098+0000 7f4287093640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f428006d9e0 con 0x7f428006ece0 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.098+0000 7f4284e2f640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f42800cece0 0x7f42800cf0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40948/0 (socket says 172.21.15.44:40948) 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.098+0000 7f4284e2f640 1 -- 172.21.15.44:0/1635812650 learned_addr learned my addr 172.21.15.44:0/1635812650 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.098+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2988244291 0 0) 0x7f428006dc40 con 0x7f42800cece0 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.098+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f426c003520 con 0x7f42800cece0 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.098+0000 7f4285630640 1 --2- 172.21.15.44:0/1635812650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f428006ece0 0x7f4280165ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4212118244 0 0) 0x7f426c003520 con 0x7f42800cece0 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f428006ece0 msgr2=0x7f4280165ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427f7fe640 1 --2- 172.21.15.44:0/1635812650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f428006ece0 0x7f4280165ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 >> v1:172.21.15.69:6789/0 conn(0x7f42800d09f0 legacy=0x7f428006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428006e200 con 0x7f42800cece0 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f4278001670 con 0x7f42800cece0 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 >> v1:172.21.15.44:6789/0 conn(0x7f42800cece0 legacy=0x7f42800cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427f7fe640 1 --2- 172.21.15.44:0/1635812650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f426c0039e0 0x7f426c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427f7fe640 1 --2- 172.21.15.44:0/1635812650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f426c004800 0x7f426c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427f7fe640 1 --2- 172.21.15.44:0/1635812650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f428006ece0 0x7f426c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f428006d9e0 con 0x7f426c0039e0 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f426c00baa0 con 0x7f426c004800 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f426c00bca0 con 0x7f428006ece0 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f4284e2f640 1 --2- 172.21.15.44:0/1635812650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f426c0039e0 0x7f426c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f427ffff640 1 --2- 172.21.15.44:0/1635812650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f426c004800 0x7f426c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f4285630640 1 --2- 172.21.15.44:0/1635812650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f428006ece0 0x7f426c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f4284e2f640 1 -- 172.21.15.44:0/1635812650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f428006ece0 msgr2=0x7f426c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f4284e2f640 1 --2- 172.21.15.44:0/1635812650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f428006ece0 0x7f426c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f4284e2f640 1 -- 172.21.15.44:0/1635812650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f426c004800 msgr2=0x7f426c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f4284e2f640 1 --2- 172.21.15.44:0/1635812650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f426c004800 0x7f426c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.099+0000 7f4284e2f640 1 -- 172.21.15.44:0/1635812650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f426c003520 con 0x7f426c0039e0 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.100+0000 7f4284e2f640 1 --2- 172.21.15.44:0/1635812650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f426c0039e0 0x7f426c003e00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f427800c210 tx=0x7f426c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66eebe70ec78917e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:06.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.100+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278007cf0 con 0x7f426c0039e0 2024-04-02T13:23:06.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.100+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4278030070 con 0x7f426c0039e0 2024-04-02T13:23:06.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.100+0000 7f427f7fe640 1 -- 172.21.15.44:0/1635812650 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278005d30 con 0x7f426c0039e0 2024-04-02T13:23:06.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.101+0000 7f4287093640 1 -- 172.21.15.44:0/1635812650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f426c0039e0 msgr2=0x7f426c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.101+0000 7f4287093640 1 --2- 172.21.15.44:0/1635812650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f426c0039e0 0x7f426c003e00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f427800c210 tx=0x7f426c005830 comp rx=0 tx=0).stop 2024-04-02T13:23:06.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.101+0000 7f4285630640 1 -- 172.21.15.44:0/1635812650 reap_dead start 2024-04-02T13:23:06.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.101+0000 7f4287093640 1 -- 172.21.15.44:0/1635812650 shutdown_connections 2024-04-02T13:23:06.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.101+0000 7f4287093640 1 -- 172.21.15.44:0/1635812650 >> 172.21.15.44:0/1635812650 conn(0x7f42800cfcd0 msgr2=0x7f42800d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:06.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.101+0000 7f4287093640 1 -- 172.21.15.44:0/1635812650 shutdown_connections 2024-04-02T13:23:06.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.101+0000 7f4287093640 1 -- 172.21.15.44:0/1635812650 wait complete. 2024-04-02T13:23:06.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.102+0000 7f4287093640 1 Processor -- start 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.102+0000 7f4287093640 1 -- start start 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f4287093640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f426c0039e0 0x7f4280168a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f4287093640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4280148b10 0x7f428014b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f4287093640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4280148590 0x7f428014b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f4287093640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f428015b220 con 0x7f4280148b10 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f4287093640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f428006e7c0 con 0x7f426c0039e0 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f4287093640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f428015b3a0 con 0x7f4280148590 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f427ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4280148b10 0x7f428014b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f427ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4280148b10 0x7f428014b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60296/0 (socket says 172.21.15.44:60296) 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f427ffff640 1 -- 172.21.15.44:0/2024541236 learned_addr learned my addr 172.21.15.44:0/2024541236 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f427ffff640 1 -- 172.21.15.44:0/2024541236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4280148590 msgr2=0x7f428014b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f427ffff640 1 --2- 172.21.15.44:0/2024541236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4280148590 0x7f428014b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f427ffff640 1 -- 172.21.15.44:0/2024541236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f426c0039e0 msgr2=0x7f4280168a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f427ffff640 1 --2- 172.21.15.44:0/2024541236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f426c0039e0 0x7f4280168a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f427ffff640 1 -- 172.21.15.44:0/2024541236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4280169320 con 0x7f4280148b10 2024-04-02T13:23:06.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f427ffff640 1 --2- 172.21.15.44:0/2024541236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4280148b10 0x7f428014b1c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f427000fbf0 tx=0x7f427000d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:06.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f427d7fa640 1 -- 172.21.15.44:0/2024541236 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427000db50 con 0x7f4280148b10 2024-04-02T13:23:06.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f427d7fa640 1 -- 172.21.15.44:0/2024541236 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4270018070 con 0x7f4280148b10 2024-04-02T13:23:06.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f4287093640 1 -- 172.21.15.44:0/2024541236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4280169600 con 0x7f4280148b10 2024-04-02T13:23:06.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f427d7fa640 1 -- 172.21.15.44:0/2024541236 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4270003970 con 0x7f4280148b10 2024-04-02T13:23:06.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.103+0000 7f4287093640 1 -- 172.21.15.44:0/2024541236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4280169bc0 con 0x7f4280148b10 2024-04-02T13:23:06.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.104+0000 7f4287093640 1 -- 172.21.15.44:0/2024541236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4280074f20 con 0x7f4280148b10 2024-04-02T13:23:06.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.105+0000 7f427d7fa640 1 -- 172.21.15.44:0/2024541236 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4270003b10 con 0x7f4280148b10 2024-04-02T13:23:06.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.105+0000 7f427d7fa640 1 --2- 172.21.15.44:0/2024541236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f426006c920 0x7f426006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.106+0000 7f427d7fa640 1 -- 172.21.15.44:0/2024541236 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f427008be90 con 0x7f4280148b10 2024-04-02T13:23:06.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.106+0000 7f4284e2f640 1 --2- 172.21.15.44:0/2024541236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f426006c920 0x7f426006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.106+0000 7f4284e2f640 1 --2- 172.21.15.44:0/2024541236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f426006c920 0x7f426006ede0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4278034000 tx=0x7f4278039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:06.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.110+0000 7f427d7fa640 1 -- 172.21.15.44:0/2024541236 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f427005a0a0 con 0x7f4280148b10 2024-04-02T13:23:06.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.332+0000 7f4287093640 1 -- 172.21.15.44:0/2024541236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4280153a40 con 0x7f4280148b10 2024-04-02T13:23:06.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.333+0000 7f427d7fa640 1 -- 172.21.15.44:0/2024541236 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5181 (secure 0 0 0) 0x7f42700599a0 con 0x7f4280148b10 2024-04-02T13:23:06.336 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:06.336 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/4131284806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":4131284806},{"type":"v1","addr":"172.21.15.44:6833","nonce":4131284806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/4262427145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":4262427145},{"type":"v1","addr":"172.21.15.44:6835","nonce":4262427145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4112565894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4112565894},{"type":"v1","addr":"172.21.15.69:6836","nonce":4112565894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:22:22.422387+0000","modified":"2024-04-02T13:22:24.398291+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6837/1745145720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1745145720},{"type":"v1","addr":"172.21.15.69:6837","nonce":1745145720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":3}]} 2024-04-02T13:23:06.336 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-02T13:23:06.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.335+0000 7f4287093640 1 -- 172.21.15.44:0/2024541236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f426006c920 msgr2=0x7f426006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.335+0000 7f4287093640 1 --2- 172.21.15.44:0/2024541236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f426006c920 0x7f426006ede0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4278034000 tx=0x7f4278039040 comp rx=0 tx=0).stop 2024-04-02T13:23:06.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.335+0000 7f4287093640 1 -- 172.21.15.44:0/2024541236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4280148b10 msgr2=0x7f428014b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.335+0000 7f4287093640 1 --2- 172.21.15.44:0/2024541236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4280148b10 0x7f428014b1c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f427000fbf0 tx=0x7f427000d520 comp rx=0 tx=0).stop 2024-04-02T13:23:06.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.335+0000 7f4287093640 1 -- 172.21.15.44:0/2024541236 shutdown_connections 2024-04-02T13:23:06.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.335+0000 7f4287093640 1 --2- 172.21.15.44:0/2024541236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f426006c920 0x7f426006ede0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.335+0000 7f4287093640 1 --2- 172.21.15.44:0/2024541236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4280148590 0x7f428014b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.335+0000 7f4287093640 1 --2- 172.21.15.44:0/2024541236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4280148b10 0x7f428014b1c0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.335+0000 7f4287093640 1 --2- 172.21.15.44:0/2024541236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f426c0039e0 0x7f4280168a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.335+0000 7f4287093640 1 -- 172.21.15.44:0/2024541236 >> 172.21.15.44:0/2024541236 conn(0x7f42800cfcd0 msgr2=0x7f4280071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:06.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.335+0000 7f4287093640 1 -- 172.21.15.44:0/2024541236 shutdown_connections 2024-04-02T13:23:06.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.335+0000 7f4287093640 1 -- 172.21.15.44:0/2024541236 wait complete. 2024-04-02T13:23:06.349 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:23:06.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.447+0000 7fef0d8e4640 1 Processor -- start 2024-04-02T13:23:06.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.447+0000 7fef0d8e4640 1 -- start start 2024-04-02T13:23:06.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.448+0000 7fef0d8e4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef0815e2d0 0x7fef080d1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.448+0000 7fef0d8e4640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fef080cf400 0x7fef0806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.448+0000 7fef0d8e4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef080727b0 con 0x7fef0815e6b0 2024-04-02T13:23:06.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.448+0000 7fef0d8e4640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fef08072a90 con 0x7fef080cf400 2024-04-02T13:23:06.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.448+0000 7fef0d8e4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fef08072d70 con 0x7fef0815e2d0 2024-04-02T13:23:06.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.448+0000 7fef077fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fef080cf400 0x7fef0806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.448+0000 7fef077fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fef080cf400 0x7fef0806ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60308/0 (socket says 172.21.15.44:60308) 2024-04-02T13:23:06.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.448+0000 7fef077fe640 1 -- 172.21.15.44:0/466487437 learned_addr learned my addr 172.21.15.44:0/466487437 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:06.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.448+0000 7fef06ffd640 1 --2- 172.21.15.44:0/466487437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef0815e2d0 0x7fef080d1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.449+0000 7fef077fe640 1 -- 172.21.15.44:0/466487437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef0815e2d0 msgr2=0x7fef080d1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:06.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.449+0000 7fef077fe640 1 --2- 172.21.15.44:0/466487437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef0815e2d0 0x7fef080d1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.449+0000 7fef077fe640 1 -- 172.21.15.44:0/466487437 >> v1:172.21.15.69:6789/0 conn(0x7fef0815e6b0 legacy=0x7fef080cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.449+0000 7fef077fe640 1 -- 172.21.15.44:0/466487437 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef08073050 con 0x7fef080cf400 2024-04-02T13:23:06.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.449+0000 7fef077fe640 1 --2- 172.21.15.44:0/466487437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef080cf400 0x7fef0806ef20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7feef8009e50 tx=0x7feef802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e208bf49d089d71 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:06.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.450+0000 7fef05ffb640 1 -- 172.21.15.44:0/466487437 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef802cb00 con 0x7fef080cf400 2024-04-02T13:23:06.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.450+0000 7fef05ffb640 1 -- 172.21.15.44:0/466487437 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feef802cca0 con 0x7fef080cf400 2024-04-02T13:23:06.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.450+0000 7fef05ffb640 1 -- 172.21.15.44:0/466487437 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8034610 con 0x7fef080cf400 2024-04-02T13:23:06.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.450+0000 7fef0d8e4640 1 -- 172.21.15.44:0/466487437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef080cf400 msgr2=0x7fef0806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.450+0000 7fef0d8e4640 1 --2- 172.21.15.44:0/466487437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef080cf400 0x7fef0806ef20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7feef8009e50 tx=0x7feef802bf60 comp rx=0 tx=0).stop 2024-04-02T13:23:06.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.450+0000 7fef0d8e4640 1 -- 172.21.15.44:0/466487437 shutdown_connections 2024-04-02T13:23:06.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.450+0000 7fef0d8e4640 1 --2- 172.21.15.44:0/466487437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef080cf400 0x7fef0806ef20 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.450+0000 7fef0d8e4640 1 --2- 172.21.15.44:0/466487437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef0815e2d0 0x7fef080d1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.450+0000 7fef0d8e4640 1 -- 172.21.15.44:0/466487437 >> 172.21.15.44:0/466487437 conn(0x7fef0806c7c0 msgr2=0x7fef0806cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:06.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.450+0000 7fef0d8e4640 1 -- 172.21.15.44:0/466487437 shutdown_connections 2024-04-02T13:23:06.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.450+0000 7fef0d8e4640 1 -- 172.21.15.44:0/466487437 wait complete. 2024-04-02T13:23:06.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.451+0000 7fef0d8e4640 1 Processor -- start 2024-04-02T13:23:06.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.451+0000 7fef0d8e4640 1 -- start start 2024-04-02T13:23:06.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef0d8e4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef080cf400 0x7fef080b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef0d8e4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef0815e2d0 0x7fef080b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef0d8e4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef0815e6b0 0x7fef081461c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef0d8e4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fef080d16c0 con 0x7fef0815e2d0 2024-04-02T13:23:06.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef0d8e4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fef08072440 con 0x7fef080cf400 2024-04-02T13:23:06.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef0d8e4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fef080d1840 con 0x7fef0815e6b0 2024-04-02T13:23:06.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef067fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef0815e2d0 0x7fef080b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef067fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef0815e2d0 0x7fef080b1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60314/0 (socket says 172.21.15.44:60314) 2024-04-02T13:23:06.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef067fc640 1 -- 172.21.15.44:0/3437452017 learned_addr learned my addr 172.21.15.44:0/3437452017 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:06.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef06ffd640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef080cf400 0x7fef080b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef077fe640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef0815e6b0 0x7fef081461c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef067fc640 1 -- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef0815e6b0 msgr2=0x7fef081461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef067fc640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef0815e6b0 0x7fef081461c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef067fc640 1 -- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef080cf400 msgr2=0x7fef080b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.452+0000 7fef067fc640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef080cf400 0x7fef080b34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.453+0000 7fef067fc640 1 -- 172.21.15.44:0/3437452017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef080b2410 con 0x7fef0815e2d0 2024-04-02T13:23:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.453+0000 7fef067fc640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef0815e2d0 0x7fef080b1d30 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7feef000c3c0 tx=0x7feef000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.453+0000 7feefffff640 1 -- 172.21.15.44:0/3437452017 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef00052d0 con 0x7fef0815e2d0 2024-04-02T13:23:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.453+0000 7feefffff640 1 -- 172.21.15.44:0/3437452017 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feef00054b0 con 0x7fef0815e2d0 2024-04-02T13:23:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.453+0000 7fef0d8e4640 1 -- 172.21.15.44:0/3437452017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef080b26f0 con 0x7fef0815e2d0 2024-04-02T13:23:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.453+0000 7feefffff640 1 -- 172.21.15.44:0/3437452017 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef0006a40 con 0x7fef0815e2d0 2024-04-02T13:23:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.453+0000 7fef0d8e4640 1 -- 172.21.15.44:0/3437452017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef080b2cb0 con 0x7fef0815e2d0 2024-04-02T13:23:06.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.454+0000 7fef0d8e4640 1 -- 172.21.15.44:0/3437452017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef08068c00 con 0x7fef0815e2d0 2024-04-02T13:23:06.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.454+0000 7feefffff640 1 -- 172.21.15.44:0/3437452017 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feef0006be0 con 0x7fef0815e2d0 2024-04-02T13:23:06.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.455+0000 7feefffff640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feee806c8f0 0x7feee806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.455+0000 7feefffff640 1 -- 172.21.15.44:0/3437452017 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7feef008b040 con 0x7fef0815e2d0 2024-04-02T13:23:06.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.455+0000 7fef06ffd640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feee806c8f0 0x7feee806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.455+0000 7fef06ffd640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feee806c8f0 0x7feee806edb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fef00009b40 tx=0x7fef00009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:06.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.459+0000 7feefffff640 1 -- 172.21.15.44:0/3437452017 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7feef0058d20 con 0x7fef0815e2d0 2024-04-02T13:23:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.681+0000 7fef0d8e4640 1 -- 172.21.15.44:0/3437452017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef080d1210 con 0x7fef0815e2d0 2024-04-02T13:23:06.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.682+0000 7feefffff640 1 -- 172.21.15.44:0/3437452017 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5181 (secure 0 0 0) 0x7feef005f020 con 0x7fef0815e2d0 2024-04-02T13:23:06.685 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/4131284806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":4131284806},{"type":"v1","addr":"172.21.15.44:6833","nonce":4131284806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/4262427145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":4262427145},{"type":"v1","addr":"172.21.15.44:6835","nonce":4262427145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4112565894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4112565894},{"type":"v1","addr":"172.21.15.69:6836","nonce":4112565894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:22:22.422387+0000","modified":"2024-04-02T13:22:24.398291+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6837/1745145720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1745145720},{"type":"v1","addr":"172.21.15.69:6837","nonce":1745145720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":3}]} 2024-04-02T13:23:06.685 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-02T13:23:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.684+0000 7fef0d8e4640 1 -- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feee806c8f0 msgr2=0x7feee806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.684+0000 7fef0d8e4640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feee806c8f0 0x7feee806edb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fef00009b40 tx=0x7fef00009190 comp rx=0 tx=0).stop 2024-04-02T13:23:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.684+0000 7fef0d8e4640 1 -- 172.21.15.44:0/3437452017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef0815e2d0 msgr2=0x7fef080b1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.684+0000 7fef0d8e4640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef0815e2d0 0x7fef080b1d30 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7feef000c3c0 tx=0x7feef000c880 comp rx=0 tx=0).stop 2024-04-02T13:23:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.684+0000 7fef0d8e4640 1 -- 172.21.15.44:0/3437452017 shutdown_connections 2024-04-02T13:23:06.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.684+0000 7fef0d8e4640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feee806c8f0 0x7feee806edb0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.684+0000 7fef0d8e4640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef0815e6b0 0x7fef081461c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.684+0000 7fef0d8e4640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef0815e2d0 0x7fef080b1d30 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.684+0000 7fef0d8e4640 1 --2- 172.21.15.44:0/3437452017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef080cf400 0x7fef080b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.684+0000 7fef0d8e4640 1 -- 172.21.15.44:0/3437452017 >> 172.21.15.44:0/3437452017 conn(0x7fef0806c7c0 msgr2=0x7fef0806cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.684+0000 7fef0d8e4640 1 -- 172.21.15.44:0/3437452017 shutdown_connections 2024-04-02T13:23:06.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.684+0000 7fef0d8e4640 1 -- 172.21.15.44:0/3437452017 wait complete. 2024-04-02T13:23:06.698 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:23:06.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.796+0000 7fd9b9796640 1 Processor -- start 2024-04-02T13:23:06.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.796+0000 7fd9b9796640 1 -- start start 2024-04-02T13:23:06.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b9796640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd9b40d28f0 0x7fd9b40d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b9796640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9b40b47b0 0x7fd9b40b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b9796640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9b40d4d70 con 0x7fd9b40b5200 2024-04-02T13:23:06.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b9796640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd9b40d5050 con 0x7fd9b40d28f0 2024-04-02T13:23:06.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b9796640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd9b40d23a0 con 0x7fd9b40b47b0 2024-04-02T13:23:06.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b2ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd9b40d28f0 0x7fd9b40d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b2ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd9b40d28f0 0x7fd9b40d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60320/0 (socket says 172.21.15.44:60320) 2024-04-02T13:23:06.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b27fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd9b40b5200 0x7fd9b40b40a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40952/0 (socket says 172.21.15.44:40952) 2024-04-02T13:23:06.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b2ffd640 1 -- 172.21.15.44:0/1657755753 learned_addr learned my addr 172.21.15.44:0/1657755753 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:06.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b37fe640 1 --2- 172.21.15.44:0/1657755753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9b40b47b0 0x7fd9b40b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b2ffd640 1 -- 172.21.15.44:0/1657755753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9b40b47b0 msgr2=0x7fd9b40b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:06.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b2ffd640 1 --2- 172.21.15.44:0/1657755753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9b40b47b0 0x7fd9b40b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b2ffd640 1 -- 172.21.15.44:0/1657755753 >> v1:172.21.15.44:6789/0 conn(0x7fd9b40b5200 legacy=0x7fd9b40b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b2ffd640 1 -- 172.21.15.44:0/1657755753 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b40d2030 con 0x7fd9b40d28f0 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b1ffb640 1 -- 172.21.15.44:0/1657755753 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2335914082 0 0) 0x7fd9b40d4d70 con 0x7fd9b40b5200 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.797+0000 7fd9b37fe640 1 --2- 172.21.15.44:0/1657755753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9b40b47b0 0x7fd9b40b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.798+0000 7fd9b2ffd640 1 --2- 172.21.15.44:0/1657755753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9b40d28f0 0x7fd9b40d2cd0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac009870 tx=0x7fd9ac02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d31da3153144d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.798+0000 7fd9b1ffb640 1 -- 172.21.15.44:0/1657755753 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac0317c0 con 0x7fd9b40d28f0 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.798+0000 7fd9b1ffb640 1 -- 172.21.15.44:0/1657755753 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9ac037070 con 0x7fd9b40d28f0 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.798+0000 7fd9b1ffb640 1 -- 172.21.15.44:0/1657755753 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac031b80 con 0x7fd9b40d28f0 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.799+0000 7fd9b9796640 1 -- 172.21.15.44:0/1657755753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9b40d28f0 msgr2=0x7fd9b40d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.799+0000 7fd9b9796640 1 --2- 172.21.15.44:0/1657755753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9b40d28f0 0x7fd9b40d2cd0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac009870 tx=0x7fd9ac02b860 comp rx=0 tx=0).stop 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.799+0000 7fd9b9796640 1 -- 172.21.15.44:0/1657755753 shutdown_connections 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.799+0000 7fd9b9796640 1 --2- 172.21.15.44:0/1657755753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9b40b47b0 0x7fd9b40b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.799+0000 7fd9b9796640 1 --2- 172.21.15.44:0/1657755753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9b40d28f0 0x7fd9b40d2cd0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.799+0000 7fd9b9796640 1 -- 172.21.15.44:0/1657755753 >> 172.21.15.44:0/1657755753 conn(0x7fd9b40d4120 msgr2=0x7fd9b40d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.799+0000 7fd9b9796640 1 -- 172.21.15.44:0/1657755753 shutdown_connections 2024-04-02T13:23:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.799+0000 7fd9b9796640 1 -- 172.21.15.44:0/1657755753 wait complete. 2024-04-02T13:23:06.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.800+0000 7fd9b9796640 1 Processor -- start 2024-04-02T13:23:06.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.800+0000 7fd9b9796640 1 -- start start 2024-04-02T13:23:06.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b9796640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9b40b47b0 0x7fd9b40a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b9796640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9b40b5200 0x7fd9b415fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b9796640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b40d28f0 0x7fd9b409f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b9796640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b40d1260 con 0x7fd9b40b47b0 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b9796640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b40b5020 con 0x7fd9b40d28f0 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b9796640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd9b40d13e0 con 0x7fd9b40b5200 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9b40b47b0 0x7fd9b40a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9b40b47b0 0x7fd9b40a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60322/0 (socket says 172.21.15.44:60322) 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b2ffd640 1 -- 172.21.15.44:0/3290277998 learned_addr learned my addr 172.21.15.44:0/3290277998 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b27fc640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9b40b5200 0x7fd9b415fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b37fe640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b40d28f0 0x7fd9b409f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b2ffd640 1 -- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9b40b5200 msgr2=0x7fd9b415fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b2ffd640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9b40b5200 0x7fd9b415fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b2ffd640 1 -- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b40d28f0 msgr2=0x7fd9b409f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b2ffd640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b40d28f0 0x7fd9b409f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b2ffd640 1 -- 172.21.15.44:0/3290277998 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b415ffd0 con 0x7fd9b40b47b0 2024-04-02T13:23:06.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.801+0000 7fd9b27fc640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9b40b5200 0x7fd9b415fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:06.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.802+0000 7fd9b2ffd640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9b40b47b0 0x7fd9b40a1940 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac000c00 tx=0x7fd9ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:06.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.802+0000 7fd9ab7fe640 1 -- 172.21.15.44:0/3290277998 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac032660 con 0x7fd9b40b47b0 2024-04-02T13:23:06.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.802+0000 7fd9ab7fe640 1 -- 172.21.15.44:0/3290277998 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9ac032800 con 0x7fd9b40b47b0 2024-04-02T13:23:06.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.802+0000 7fd9b9796640 1 -- 172.21.15.44:0/3290277998 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b409e9e0 con 0x7fd9b40b47b0 2024-04-02T13:23:06.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.802+0000 7fd9ab7fe640 1 -- 172.21.15.44:0/3290277998 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac032bf0 con 0x7fd9b40b47b0 2024-04-02T13:23:06.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.802+0000 7fd9b9796640 1 -- 172.21.15.44:0/3290277998 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b409efa0 con 0x7fd9b40b47b0 2024-04-02T13:23:06.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.803+0000 7fd9b9796640 1 -- 172.21.15.44:0/3290277998 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b40684f0 con 0x7fd9b40b47b0 2024-04-02T13:23:06.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.803+0000 7fd9ab7fe640 1 -- 172.21.15.44:0/3290277998 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd9ac04c060 con 0x7fd9b40b47b0 2024-04-02T13:23:06.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.803+0000 7fd9ab7fe640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd99406c920 0x7fd99406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:06.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.804+0000 7fd9ab7fe640 1 -- 172.21.15.44:0/3290277998 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fd9ac0ad910 con 0x7fd9b40b47b0 2024-04-02T13:23:06.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.804+0000 7fd9b27fc640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd99406c920 0x7fd99406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:06.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.804+0000 7fd9b27fc640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd99406c920 0x7fd99406ede0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4160c20 tx=0x7fd99c007cc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.808+0000 7fd9ab7fe640 1 -- 172.21.15.44:0/3290277998 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd9ac040070 con 0x7fd9b40b47b0 2024-04-02T13:23:06.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.994+0000 7fd9b9796640 1 -- 172.21.15.44:0/3290277998 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9b40b8080 con 0x7fd9b40b47b0 2024-04-02T13:23:06.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.996+0000 7fd9ab7fe640 1 -- 172.21.15.44:0/3290277998 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+16894 (secure 0 0 0) 0x7fd9ac07bb10 con 0x7fd9b40b47b0 2024-04-02T13:23:06.999 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:06.999 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":52,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:22:24.017854+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:22:20.662776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-02T13:22:21.332577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:23:07.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.997+0000 7fd9b9796640 1 -- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd99406c920 msgr2=0x7fd99406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:07.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.997+0000 7fd9b9796640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd99406c920 0x7fd99406ede0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4160c20 tx=0x7fd99c007cc0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.997+0000 7fd9b9796640 1 -- 172.21.15.44:0/3290277998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9b40b47b0 msgr2=0x7fd9b40a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:07.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.997+0000 7fd9b9796640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9b40b47b0 0x7fd9b40a1940 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac000c00 tx=0x7fd9ac0046c0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.998+0000 7fd9b9796640 1 -- 172.21.15.44:0/3290277998 shutdown_connections 2024-04-02T13:23:07.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.998+0000 7fd9b9796640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd99406c920 0x7fd99406ede0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.998+0000 7fd9b9796640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b40d28f0 0x7fd9b409f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.998+0000 7fd9b9796640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9b40b5200 0x7fd9b415fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.998+0000 7fd9b9796640 1 --2- 172.21.15.44:0/3290277998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9b40b47b0 0x7fd9b40a1940 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.998+0000 7fd9b9796640 1 -- 172.21.15.44:0/3290277998 >> 172.21.15.44:0/3290277998 conn(0x7fd9b40d4120 msgr2=0x7fd9b4161840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:07.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.998+0000 7fd9b9796640 1 -- 172.21.15.44:0/3290277998 shutdown_connections 2024-04-02T13:23:07.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:06.998+0000 7fd9b9796640 1 -- 172.21.15.44:0/3290277998 wait complete. 2024-04-02T13:23:07.011 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:23:07.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.112+0000 7f6d95c7b640 1 Processor -- start 2024-04-02T13:23:07.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.112+0000 7f6d95c7b640 1 -- start start 2024-04-02T13:23:07.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d95c7b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900d0c50 0x7f6d900d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d95c7b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6d9006ce40 0x7f6d9006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d95c7b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d900cc600 con 0x7f6d9015e4a0 2024-04-02T13:23:07.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d95c7b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6d900af300 con 0x7f6d9006ce40 2024-04-02T13:23:07.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d95c7b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6d900cc8e0 con 0x7f6d900d0c50 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8f7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6d9015e4a0 0x7f6d900aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40954/0 (socket says 172.21.15.44:40954) 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8f7fe640 1 -- 172.21.15.44:0/2475829855 learned_addr learned my addr 172.21.15.44:0/2475829855 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8effd640 1 --2- 172.21.15.44:0/2475829855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900d0c50 0x7f6d900d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8ffff640 1 --2- 172.21.15.44:0/2475829855 >> v2:172.21.15.69:3300/0 conn(0x7f6d9006ce40 0x7f6d9006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1879005371 0 0) 0x7f6d900cc600 con 0x7f6d9015e4a0 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d7c003520 con 0x7f6d9015e4a0 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4190494621 0 0) 0x7f6d7c003520 con 0x7f6d9015e4a0 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900d0c50 msgr2=0x7f6d900d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 --2- 172.21.15.44:0/2475829855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900d0c50 0x7f6d900d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 >> v2:172.21.15.69:3300/0 conn(0x7f6d9006ce40 msgr2=0x7f6d9006d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 --2- 172.21.15.44:0/2475829855 >> v2:172.21.15.69:3300/0 conn(0x7f6d9006ce40 0x7f6d9006d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d900ccbc0 con 0x7f6d9015e4a0 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f6d80001670 con 0x7f6d9015e4a0 2024-04-02T13:23:07.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 >> v1:172.21.15.44:6789/0 conn(0x7f6d9015e4a0 legacy=0x7f6d900aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 --2- 172.21.15.44:0/2475829855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d7c0039b0 0x7f6d7c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 --2- 172.21.15.44:0/2475829855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d7c004800 0x7f6d7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 --2- 172.21.15.44:0/2475829855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900d0c50 0x7f6d7c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6d900af300 con 0x7f6d7c0039b0 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6d900cc8e0 con 0x7f6d7c004800 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6d7c00b9c0 con 0x7f6d900d0c50 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.113+0000 7f6d8f7fe640 1 --2- 172.21.15.44:0/2475829855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d7c0039b0 0x7f6d7c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.114+0000 7f6d8effd640 1 --2- 172.21.15.44:0/2475829855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d7c004800 0x7f6d7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.114+0000 7f6d8ffff640 1 --2- 172.21.15.44:0/2475829855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900d0c50 0x7f6d7c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.114+0000 7f6d8f7fe640 1 -- 172.21.15.44:0/2475829855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900d0c50 msgr2=0x7f6d7c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.114+0000 7f6d8f7fe640 1 --2- 172.21.15.44:0/2475829855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900d0c50 0x7f6d7c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.114+0000 7f6d8f7fe640 1 -- 172.21.15.44:0/2475829855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d7c004800 msgr2=0x7f6d7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.114+0000 7f6d8f7fe640 1 --2- 172.21.15.44:0/2475829855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d7c004800 0x7f6d7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.114+0000 7f6d8f7fe640 1 -- 172.21.15.44:0/2475829855 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7c003520 con 0x7f6d7c0039b0 2024-04-02T13:23:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.114+0000 7f6d8f7fe640 1 --2- 172.21.15.44:0/2475829855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d7c0039b0 0x7f6d7c003dd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f6d8000c240 tx=0x7f6d7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7204f4df7efa62c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:07.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.115+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d80007d30 con 0x7f6d7c0039b0 2024-04-02T13:23:07.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.115+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d80030070 con 0x7f6d7c0039b0 2024-04-02T13:23:07.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.115+0000 7f6d8e7fc640 1 -- 172.21.15.44:0/2475829855 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8002cd60 con 0x7f6d7c0039b0 2024-04-02T13:23:07.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.115+0000 7f6d95c7b640 1 -- 172.21.15.44:0/2475829855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d7c0039b0 msgr2=0x7f6d7c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:07.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.115+0000 7f6d95c7b640 1 --2- 172.21.15.44:0/2475829855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d7c0039b0 0x7f6d7c003dd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f6d8000c240 tx=0x7f6d7c005830 comp rx=0 tx=0).stop 2024-04-02T13:23:07.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.115+0000 7f6d8ffff640 1 -- 172.21.15.44:0/2475829855 reap_dead start 2024-04-02T13:23:07.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.115+0000 7f6d95c7b640 1 -- 172.21.15.44:0/2475829855 shutdown_connections 2024-04-02T13:23:07.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.115+0000 7f6d95c7b640 1 -- 172.21.15.44:0/2475829855 >> 172.21.15.44:0/2475829855 conn(0x7f6d9009aa20 msgr2=0x7f6d9009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:07.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.115+0000 7f6d95c7b640 1 -- 172.21.15.44:0/2475829855 shutdown_connections 2024-04-02T13:23:07.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.115+0000 7f6d95c7b640 1 -- 172.21.15.44:0/2475829855 wait complete. 2024-04-02T13:23:07.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.116+0000 7f6d95c7b640 1 Processor -- start 2024-04-02T13:23:07.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.117+0000 7f6d95c7b640 1 -- start start 2024-04-02T13:23:07.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.117+0000 7f6d95c7b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d7c0039b0 0x7f6d9014e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.117+0000 7f6d95c7b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900b4d90 0x7f6d900b73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.117+0000 7f6d95c7b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9015ea90 0x7f6d900b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.117+0000 7f6d95c7b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6d900cd460 con 0x7f6d9015ea90 2024-04-02T13:23:07.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.117+0000 7f6d95c7b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6d900cd180 con 0x7f6d7c0039b0 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.117+0000 7f6d95c7b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6d9015e880 con 0x7f6d900b4d90 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.117+0000 7f6d8ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9015ea90 0x7f6d900b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.117+0000 7f6d8ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9015ea90 0x7f6d900b2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60338/0 (socket says 172.21.15.44:60338) 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.117+0000 7f6d8ffff640 1 -- 172.21.15.44:0/4242780993 learned_addr learned my addr 172.21.15.44:0/4242780993 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d8f7fe640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d7c0039b0 0x7f6d9014e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d8effd640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900b4d90 0x7f6d900b73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d8ffff640 1 -- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900b4d90 msgr2=0x7f6d900b73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d8ffff640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900b4d90 0x7f6d900b73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d8ffff640 1 -- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d7c0039b0 msgr2=0x7f6d9014e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d8ffff640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d7c0039b0 0x7f6d9014e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d8ffff640 1 -- 172.21.15.44:0/4242780993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d900c6d40 con 0x7f6d9015ea90 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d8f7fe640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d7c0039b0 0x7f6d9014e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d8effd640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900b4d90 0x7f6d900b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d8ffff640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9015ea90 0x7f6d900b2db0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f6d88011c80 tx=0x7f6d8800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d87fff640 1 -- 172.21.15.44:0/4242780993 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d88015430 con 0x7f6d9015ea90 2024-04-02T13:23:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d87fff640 1 -- 172.21.15.44:0/4242780993 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d880155f0 con 0x7f6d9015ea90 2024-04-02T13:23:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d87fff640 1 -- 172.21.15.44:0/4242780993 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d880159e0 con 0x7f6d9015ea90 2024-04-02T13:23:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d95c7b640 1 -- 172.21.15.44:0/4242780993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d900c7020 con 0x7f6d9015ea90 2024-04-02T13:23:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.118+0000 7f6d95c7b640 1 -- 172.21.15.44:0/4242780993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d900c75e0 con 0x7f6d9015ea90 2024-04-02T13:23:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.119+0000 7f6d95c7b640 1 -- 172.21.15.44:0/4242780993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d900684f0 con 0x7f6d9015ea90 2024-04-02T13:23:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.119+0000 7f6d87fff640 1 -- 172.21.15.44:0/4242780993 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6d88015b80 con 0x7f6d9015ea90 2024-04-02T13:23:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.120+0000 7f6d87fff640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6d7006c8f0 0x7f6d7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.120+0000 7f6d87fff640 1 -- 172.21.15.44:0/4242780993 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f6d8808c680 con 0x7f6d9015ea90 2024-04-02T13:23:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.120+0000 7f6d8f7fe640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6d7006c8f0 0x7f6d7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.121+0000 7f6d8f7fe640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6d7006c8f0 0x7f6d7006edb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6d80034000 tx=0x7f6d80007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:07.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.124+0000 7f6d87fff640 1 -- 172.21.15.44:0/4242780993 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6d88024030 con 0x7f6d9015ea90 2024-04-02T13:23:07.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.348+0000 7f6d95c7b640 1 -- 172.21.15.44:0/4242780993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6d900cc370 con 0x7f6d9015ea90 2024-04-02T13:23:07.352 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:23:07.349+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:23:07.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.350+0000 7f6d87fff640 1 -- 172.21.15.44:0/4242780993 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7f6d88019030 con 0x7f6d9015ea90 2024-04-02T13:23:07.353 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:23:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.352+0000 7f6d95c7b640 1 -- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6d7006c8f0 msgr2=0x7f6d7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.352+0000 7f6d95c7b640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6d7006c8f0 0x7f6d7006edb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6d80034000 tx=0x7f6d80007000 comp rx=0 tx=0).stop 2024-04-02T13:23:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.352+0000 7f6d95c7b640 1 -- 172.21.15.44:0/4242780993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9015ea90 msgr2=0x7f6d900b2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.352+0000 7f6d95c7b640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9015ea90 0x7f6d900b2db0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f6d88011c80 tx=0x7f6d8800f580 comp rx=0 tx=0).stop 2024-04-02T13:23:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.352+0000 7f6d95c7b640 1 -- 172.21.15.44:0/4242780993 shutdown_connections 2024-04-02T13:23:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.352+0000 7f6d95c7b640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6d7006c8f0 0x7f6d7006edb0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.352+0000 7f6d95c7b640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9015ea90 0x7f6d900b2db0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.352+0000 7f6d95c7b640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d900b4d90 0x7f6d900b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.352+0000 7f6d95c7b640 1 --2- 172.21.15.44:0/4242780993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d7c0039b0 0x7f6d9014e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.352+0000 7f6d95c7b640 1 -- 172.21.15.44:0/4242780993 >> 172.21.15.44:0/4242780993 conn(0x7f6d9009aa20 msgr2=0x7f6d9009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.353+0000 7f6d95c7b640 1 -- 172.21.15.44:0/4242780993 shutdown_connections 2024-04-02T13:23:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.353+0000 7f6d95c7b640 1 -- 172.21.15.44:0/4242780993 wait complete. 2024-04-02T13:23:07.362 INFO:tasks.ceph.mds.d.smithi069.stderr: -4898> 2024-04-02T13:22:25.002+0000 7fbf4d51c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:23:07.362 INFO:tasks.ceph.mds.d.smithi069.stderr: -3169> 2024-04-02T13:22:30.476+0000 7fbf49514640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={6144=0-4194304@1} caps={6144=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6144 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x55c98c586680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(8)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-02T13:23:07.366 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T13:23:07.389 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-04-02T13:23:07.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.466+0000 7fbd397aa640 1 Processor -- start 2024-04-02T13:23:07.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.466+0000 7fbd397aa640 1 -- start start 2024-04-02T13:23:07.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.467+0000 7fbd397aa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd3415e1d0 0x7fbd340d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.467+0000 7fbd397aa640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbd340d2dd0 0x7fbd340b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.467+0000 7fbd397aa640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd340d1640 con 0x7fbd340cec60 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.467+0000 7fbd397aa640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fbd340cfe00 con 0x7fbd340d2dd0 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.467+0000 7fbd397aa640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbd340cf1f0 con 0x7fbd3415e1d0 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.467+0000 7fbd337fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbd340d2dd0 0x7fbd340b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.467+0000 7fbd337fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbd340d2dd0 0x7fbd340b4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60352/0 (socket says 172.21.15.44:60352) 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.467+0000 7fbd337fe640 1 -- 172.21.15.44:0/4101805752 learned_addr learned my addr 172.21.15.44:0/4101805752 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.467+0000 7fbd32ffd640 1 --2- 172.21.15.44:0/4101805752 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd3415e1d0 0x7fbd340d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.467+0000 7fbd337fe640 1 -- 172.21.15.44:0/4101805752 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd3415e1d0 msgr2=0x7fbd340d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.467+0000 7fbd337fe640 1 --2- 172.21.15.44:0/4101805752 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd3415e1d0 0x7fbd340d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.467+0000 7fbd337fe640 1 -- 172.21.15.44:0/4101805752 >> v1:172.21.15.69:6789/0 conn(0x7fbd340cec60 legacy=0x7fbd340d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.467+0000 7fbd337fe640 1 -- 172.21.15.44:0/4101805752 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd340cf4d0 con 0x7fbd340d2dd0 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.468+0000 7fbd337fe640 1 --2- 172.21.15.44:0/4101805752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd340d2dd0 0x7fbd340b4ce0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fbd24009e50 tx=0x7fbd2402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c70aa631febb2b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.468+0000 7fbd31ffb640 1 -- 172.21.15.44:0/4101805752 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2402ca70 con 0x7fbd340d2dd0 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.468+0000 7fbd31ffb640 1 -- 172.21.15.44:0/4101805752 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd2402cc10 con 0x7fbd340d2dd0 2024-04-02T13:23:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.468+0000 7fbd31ffb640 1 -- 172.21.15.44:0/4101805752 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd240343f0 con 0x7fbd340d2dd0 2024-04-02T13:23:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.469+0000 7fbd397aa640 1 -- 172.21.15.44:0/4101805752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd340d2dd0 msgr2=0x7fbd340b4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:07.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.469+0000 7fbd397aa640 1 --2- 172.21.15.44:0/4101805752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd340d2dd0 0x7fbd340b4ce0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fbd24009e50 tx=0x7fbd2402bed0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.469+0000 7fbd397aa640 1 -- 172.21.15.44:0/4101805752 shutdown_connections 2024-04-02T13:23:07.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.469+0000 7fbd397aa640 1 --2- 172.21.15.44:0/4101805752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd340d2dd0 0x7fbd340b4ce0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.469+0000 7fbd397aa640 1 --2- 172.21.15.44:0/4101805752 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd3415e1d0 0x7fbd340d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.469+0000 7fbd397aa640 1 -- 172.21.15.44:0/4101805752 >> 172.21.15.44:0/4101805752 conn(0x7fbd3409d3b0 msgr2=0x7fbd340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:07.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.469+0000 7fbd397aa640 1 -- 172.21.15.44:0/4101805752 shutdown_connections 2024-04-02T13:23:07.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.469+0000 7fbd397aa640 1 -- 172.21.15.44:0/4101805752 wait complete. 2024-04-02T13:23:07.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.470+0000 7fbd397aa640 1 Processor -- start 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.470+0000 7fbd397aa640 1 -- start start 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd397aa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd340cec60 0x7fbd341f4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd397aa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd340d2dd0 0x7fbd3415a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd397aa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd3415e1d0 0x7fbd340a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd397aa640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3415e5b0 con 0x7fbd340d2dd0 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd397aa640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbd340cfa90 con 0x7fbd340cec60 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd397aa640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbd3415e730 con 0x7fbd3415e1d0 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd327fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd340d2dd0 0x7fbd3415a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd327fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd340d2dd0 0x7fbd3415a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60356/0 (socket says 172.21.15.44:60356) 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd327fc640 1 -- 172.21.15.44:0/3863278563 learned_addr learned my addr 172.21.15.44:0/3863278563 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd32ffd640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd340cec60 0x7fbd341f4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd337fe640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd3415e1d0 0x7fbd340a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd327fc640 1 -- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd3415e1d0 msgr2=0x7fbd340a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd327fc640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd3415e1d0 0x7fbd340a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd327fc640 1 -- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd340cec60 msgr2=0x7fbd341f4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:07.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd327fc640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd340cec60 0x7fbd341f4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:07.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.471+0000 7fbd327fc640 1 -- 172.21.15.44:0/3863278563 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd341f4890 con 0x7fbd340d2dd0 2024-04-02T13:23:07.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.472+0000 7fbd327fc640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd340d2dd0 0x7fbd3415a550 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00c3c0 tx=0x7fbd1c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:07.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.472+0000 7fbd2b7fe640 1 -- 172.21.15.44:0/3863278563 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0052d0 con 0x7fbd340d2dd0 2024-04-02T13:23:07.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.472+0000 7fbd2b7fe640 1 -- 172.21.15.44:0/3863278563 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd1c005490 con 0x7fbd340d2dd0 2024-04-02T13:23:07.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.472+0000 7fbd2b7fe640 1 -- 172.21.15.44:0/3863278563 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c007260 con 0x7fbd340d2dd0 2024-04-02T13:23:07.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.472+0000 7fbd397aa640 1 -- 172.21.15.44:0/3863278563 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd3415aa90 con 0x7fbd340d2dd0 2024-04-02T13:23:07.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.472+0000 7fbd397aa640 1 -- 172.21.15.44:0/3863278563 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd3409de20 con 0x7fbd340d2dd0 2024-04-02T13:23:07.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.473+0000 7fbd397aa640 1 -- 172.21.15.44:0/3863278563 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd340a0b60 con 0x7fbd340d2dd0 2024-04-02T13:23:07.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.473+0000 7fbd2b7fe640 1 -- 172.21.15.44:0/3863278563 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbd1c007400 con 0x7fbd340d2dd0 2024-04-02T13:23:07.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.474+0000 7fbd2b7fe640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd1406c8f0 0x7fbd1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:07.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.474+0000 7fbd2b7fe640 1 -- 172.21.15.44:0/3863278563 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6187+0+0 (secure 0 0 0) 0x7fbd1c08b110 con 0x7fbd340d2dd0 2024-04-02T13:23:07.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.474+0000 7fbd32ffd640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd1406c8f0 0x7fbd1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:07.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.474+0000 7fbd32ffd640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd1406c8f0 0x7fbd1406edb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c00aac0 tx=0x7fbd2c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:07.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.478+0000 7fbd2b7fe640 1 -- 172.21.15.44:0/3863278563 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbd1c058d20 con 0x7fbd340d2dd0 2024-04-02T13:23:07.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:07.704+0000 7fbd397aa640 1 -- 172.21.15.44:0/3863278563 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fbd340d47f0 con 0x7fbd340d2dd0 2024-04-02T13:23:08.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.363+0000 7fbd2b7fe640 1 -- 172.21.15.44:0/3863278563 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7fbd1c05f030 con 0x7fbd340d2dd0 2024-04-02T13:23:08.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.365+0000 7fbd397aa640 1 -- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd1406c8f0 msgr2=0x7fbd1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:08.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.365+0000 7fbd397aa640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd1406c8f0 0x7fbd1406edb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c00aac0 tx=0x7fbd2c008040 comp rx=0 tx=0).stop 2024-04-02T13:23:08.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.365+0000 7fbd397aa640 1 -- 172.21.15.44:0/3863278563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd340d2dd0 msgr2=0x7fbd3415a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.365+0000 7fbd397aa640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd340d2dd0 0x7fbd3415a550 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00c3c0 tx=0x7fbd1c00c880 comp rx=0 tx=0).stop 2024-04-02T13:23:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.365+0000 7fbd397aa640 1 -- 172.21.15.44:0/3863278563 shutdown_connections 2024-04-02T13:23:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.365+0000 7fbd397aa640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd1406c8f0 0x7fbd1406edb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.365+0000 7fbd397aa640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd3415e1d0 0x7fbd340a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.365+0000 7fbd397aa640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd340d2dd0 0x7fbd3415a550 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.365+0000 7fbd397aa640 1 --2- 172.21.15.44:0/3863278563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd340cec60 0x7fbd341f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.365+0000 7fbd397aa640 1 -- 172.21.15.44:0/3863278563 >> 172.21.15.44:0/3863278563 conn(0x7fbd3409d3b0 msgr2=0x7fbd340d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.366+0000 7fbd397aa640 1 -- 172.21.15.44:0/3863278563 shutdown_connections 2024-04-02T13:23:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.366+0000 7fbd397aa640 1 -- 172.21.15.44:0/3863278563 wait complete. 2024-04-02T13:23:08.381 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T13:23:08.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.490+0000 7f774c4ba640 1 Processor -- start 2024-04-02T13:23:08.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.490+0000 7f774c4ba640 1 -- start start 2024-04-02T13:23:08.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.491+0000 7f774c4ba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f774415c260 0x7f77440b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:08.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.491+0000 7f774c4ba640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f77440b3e80 0x7f77440ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:08.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.491+0000 7f774c4ba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77440cd4a0 con 0x7f77440cd030 2024-04-02T13:23:08.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.491+0000 7f774c4ba640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f77440cd760 con 0x7f77440b3e80 2024-04-02T13:23:08.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.491+0000 7f774c4ba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f77440cda40 con 0x7f774415c260 2024-04-02T13:23:08.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.491+0000 7f7749a55640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f77440b3e80 0x7f77440ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:08.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.491+0000 7f7749a55640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f77440b3e80 0x7f77440ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60358/0 (socket says 172.21.15.44:60358) 2024-04-02T13:23:08.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.491+0000 7f7749a55640 1 -- 172.21.15.44:0/2312253736 learned_addr learned my addr 172.21.15.44:0/2312253736 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:08.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.491+0000 7f774a256640 1 --2- 172.21.15.44:0/2312253736 >> v2:172.21.15.69:3300/0 conn(0x7f774415c260 0x7f77440b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:08.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.491+0000 7f7749a55640 1 -- 172.21.15.44:0/2312253736 >> v2:172.21.15.69:3300/0 conn(0x7f774415c260 msgr2=0x7f77440b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:08.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.491+0000 7f7749a55640 1 --2- 172.21.15.44:0/2312253736 >> v2:172.21.15.69:3300/0 conn(0x7f774415c260 0x7f77440b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:08.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.491+0000 7f7749a55640 1 -- 172.21.15.44:0/2312253736 >> v1:172.21.15.69:6789/0 conn(0x7f77440cd030 legacy=0x7f77440ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:08.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.491+0000 7f7749a55640 1 -- 172.21.15.44:0/2312253736 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77440cdd20 con 0x7f77440b3e80 2024-04-02T13:23:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.492+0000 7f7749a55640 1 --2- 172.21.15.44:0/2312253736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77440b3e80 0x7f77440ccaf0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7744071d40 tx=0x7f773802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea38f720daf6fae server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.492+0000 7f7749254640 1 -- 172.21.15.44:0/2312253736 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7738034070 con 0x7f77440b3e80 2024-04-02T13:23:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.492+0000 7f7749254640 1 -- 172.21.15.44:0/2312253736 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7738030860 con 0x7f77440b3e80 2024-04-02T13:23:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.492+0000 7f7749254640 1 -- 172.21.15.44:0/2312253736 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7738030c30 con 0x7f77440b3e80 2024-04-02T13:23:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.493+0000 7f774c4ba640 1 -- 172.21.15.44:0/2312253736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77440b3e80 msgr2=0x7f77440ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.493+0000 7f774c4ba640 1 --2- 172.21.15.44:0/2312253736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77440b3e80 0x7f77440ccaf0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7744071d40 tx=0x7f773802b9a0 comp rx=0 tx=0).stop 2024-04-02T13:23:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.493+0000 7f774c4ba640 1 -- 172.21.15.44:0/2312253736 shutdown_connections 2024-04-02T13:23:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.493+0000 7f774c4ba640 1 --2- 172.21.15.44:0/2312253736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77440b3e80 0x7f77440ccaf0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.493+0000 7f774c4ba640 1 --2- 172.21.15.44:0/2312253736 >> v2:172.21.15.69:3300/0 conn(0x7f774415c260 0x7f77440b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.493+0000 7f774c4ba640 1 -- 172.21.15.44:0/2312253736 >> 172.21.15.44:0/2312253736 conn(0x7f774409aa20 msgr2=0x7f774409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.493+0000 7f774c4ba640 1 -- 172.21.15.44:0/2312253736 shutdown_connections 2024-04-02T13:23:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.493+0000 7f774c4ba640 1 -- 172.21.15.44:0/2312253736 wait complete. 2024-04-02T13:23:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.494+0000 7f774c4ba640 1 Processor -- start 2024-04-02T13:23:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.494+0000 7f774c4ba640 1 -- start start 2024-04-02T13:23:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774c4ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77440b3e80 0x7f774409e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774c4ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77440cd030 0x7f77440b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774c4ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f774415c260 0x7f774415e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774c4ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f77440c6b40 con 0x7f774415c260 2024-04-02T13:23:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774c4ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f77440cfe60 con 0x7f77440cd030 2024-04-02T13:23:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774c4ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f77440c6cc0 con 0x7f77440b3e80 2024-04-02T13:23:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774aa57640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f774415c260 0x7f774415e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774aa57640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f774415c260 0x7f774415e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60370/0 (socket says 172.21.15.44:60370) 2024-04-02T13:23:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774aa57640 1 -- 172.21.15.44:0/3502322665 learned_addr learned my addr 172.21.15.44:0/3502322665 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f7749a55640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77440cd030 0x7f77440b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774a256640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77440b3e80 0x7f774409e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774aa57640 1 -- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77440b3e80 msgr2=0x7f774409e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774aa57640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77440b3e80 0x7f774409e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774aa57640 1 -- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77440cd030 msgr2=0x7f77440b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774aa57640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77440cd030 0x7f77440b77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774aa57640 1 -- 172.21.15.44:0/3502322665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77440b7cf0 con 0x7f774415c260 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.495+0000 7f774aa57640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f774415c260 0x7f774415e9a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f773400c3f0 tx=0x7f773400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.496+0000 7f7742ffd640 1 -- 172.21.15.44:0/3502322665 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7734017070 con 0x7f774415c260 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.496+0000 7f7742ffd640 1 -- 172.21.15.44:0/3502322665 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77340052d0 con 0x7f774415c260 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.496+0000 7f774c4ba640 1 -- 172.21.15.44:0/3502322665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f774409ef00 con 0x7f774415c260 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.496+0000 7f7742ffd640 1 -- 172.21.15.44:0/3502322665 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7734003850 con 0x7f774415c260 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.496+0000 7f774c4ba640 1 -- 172.21.15.44:0/3502322665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f774409f4c0 con 0x7f774415c260 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.497+0000 7f774c4ba640 1 -- 172.21.15.44:0/3502322665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77440684f0 con 0x7f774415c260 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.497+0000 7f7742ffd640 1 -- 172.21.15.44:0/3502322665 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f773401a070 con 0x7f774415c260 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.498+0000 7f7742ffd640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f773006c8f0 0x7f773006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.498+0000 7f7742ffd640 1 -- 172.21.15.44:0/3502322665 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6245+0+0 (secure 0 0 0) 0x7f773408af80 con 0x7f774415c260 2024-04-02T13:23:08.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.498+0000 7f774a256640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f773006c8f0 0x7f773006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:08.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.498+0000 7f774a256640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f773006c8f0 0x7f773006edb0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f773c011340 tx=0x7f773c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:08.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.502+0000 7f7742ffd640 1 -- 172.21.15.44:0/3502322665 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7734058b40 con 0x7f774415c260 2024-04-02T13:23:08.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:08.686+0000 7f774c4ba640 1 -- 172.21.15.44:0/3502322665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f77440ccd90 con 0x7f774415c260 2024-04-02T13:23:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.370+0000 7f7742ffd640 1 -- 172.21.15.44:0/3502322665 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v55) ==== 181+0+0 (secure 0 0 0) 0x7f773405ce40 con 0x7f774415c260 2024-04-02T13:23:09.373 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' removed 2024-04-02T13:23:09.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.374+0000 7f774c4ba640 1 -- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f773006c8f0 msgr2=0x7f773006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:09.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.374+0000 7f774c4ba640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f773006c8f0 0x7f773006edb0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f773c011340 tx=0x7f773c00b040 comp rx=0 tx=0).stop 2024-04-02T13:23:09.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.374+0000 7f774c4ba640 1 -- 172.21.15.44:0/3502322665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f774415c260 msgr2=0x7f774415e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:09.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.374+0000 7f774c4ba640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f774415c260 0x7f774415e9a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f773400c3f0 tx=0x7f773400c8b0 comp rx=0 tx=0).stop 2024-04-02T13:23:09.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.374+0000 7f774c4ba640 1 -- 172.21.15.44:0/3502322665 shutdown_connections 2024-04-02T13:23:09.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.374+0000 7f774c4ba640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f773006c8f0 0x7f773006edb0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:09.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.374+0000 7f774c4ba640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f774415c260 0x7f774415e9a0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:09.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.374+0000 7f774c4ba640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77440cd030 0x7f77440b77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:09.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.374+0000 7f774c4ba640 1 --2- 172.21.15.44:0/3502322665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77440b3e80 0x7f774409e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:09.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.375+0000 7f774c4ba640 1 -- 172.21.15.44:0/3502322665 >> 172.21.15.44:0/3502322665 conn(0x7f774409aa20 msgr2=0x7f7744070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:09.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.376+0000 7f774c4ba640 1 -- 172.21.15.44:0/3502322665 shutdown_connections 2024-04-02T13:23:09.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.376+0000 7f774c4ba640 1 -- 172.21.15.44:0/3502322665 wait complete. 2024-04-02T13:23:09.397 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T13:23:09.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.505+0000 7f527801b640 1 Processor -- start 2024-04-02T13:23:09.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.505+0000 7f527801b640 1 -- start start 2024-04-02T13:23:09.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.506+0000 7f527801b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f527015e470 0x7f52700b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:09.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.506+0000 7f527801b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52700b1540 0x7f52700d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:09.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.506+0000 7f527801b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52700ced00 con 0x7f52700ce790 2024-04-02T13:23:09.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.506+0000 7f527801b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f52700cefe0 con 0x7f527015e470 2024-04-02T13:23:09.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.506+0000 7f527801b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f52700cf2c0 con 0x7f52700b1540 2024-04-02T13:23:09.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.506+0000 7f52755b6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52700b1540 0x7f52700d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:09.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.506+0000 7f5275db7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f527015e470 0x7f52700b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:09.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.506+0000 7f52755b6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52700b1540 0x7f52700d1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:51998/0 (socket says 172.21.15.44:51998) 2024-04-02T13:23:09.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.506+0000 7f52755b6640 1 -- 172.21.15.44:0/315835802 learned_addr learned my addr 172.21.15.44:0/315835802 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:09.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.506+0000 7f52765b8640 1 --1- 172.21.15.44:0/315835802 >> v1:172.21.15.69:6789/0 conn(0x7f52700ce790 0x7f5270161fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:40544/0 (socket says 172.21.15.44:40544) 2024-04-02T13:23:09.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.507+0000 7f5274db5640 1 -- 172.21.15.44:0/315835802 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1172080557 0 0) 0x7f52700ced00 con 0x7f52700ce790 2024-04-02T13:23:09.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.507+0000 7f5274db5640 1 -- 172.21.15.44:0/315835802 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f525c003520 con 0x7f52700ce790 2024-04-02T13:23:09.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.507+0000 7f5275db7640 1 -- 172.21.15.44:0/315835802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52700b1540 msgr2=0x7f52700d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:09.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.507+0000 7f5275db7640 1 --2- 172.21.15.44:0/315835802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52700b1540 0x7f52700d1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:09.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.507+0000 7f5275db7640 1 -- 172.21.15.44:0/315835802 >> v1:172.21.15.69:6789/0 conn(0x7f52700ce790 legacy=0x7f5270161fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:09.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.507+0000 7f5275db7640 1 -- 172.21.15.44:0/315835802 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52700cf5a0 con 0x7f527015e470 2024-04-02T13:23:09.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.507+0000 7f52755b6640 1 --2- 172.21.15.44:0/315835802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52700b1540 0x7f52700d1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:23:09.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.507+0000 7f5275db7640 1 --2- 172.21.15.44:0/315835802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f527015e470 0x7f52700b32c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f526c0097a0 tx=0x7f526c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ac3ee873fb40fd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:09.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.508+0000 7f5274db5640 1 -- 172.21.15.44:0/315835802 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c0315a0 con 0x7f527015e470 2024-04-02T13:23:09.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.508+0000 7f5274db5640 1 -- 172.21.15.44:0/315835802 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f526c037070 con 0x7f527015e470 2024-04-02T13:23:09.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.508+0000 7f5274db5640 1 -- 172.21.15.44:0/315835802 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c031940 con 0x7f527015e470 2024-04-02T13:23:09.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.508+0000 7f527801b640 1 -- 172.21.15.44:0/315835802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f527015e470 msgr2=0x7f52700b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:09.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.508+0000 7f527801b640 1 --2- 172.21.15.44:0/315835802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f527015e470 0x7f52700b32c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f526c0097a0 tx=0x7f526c02b840 comp rx=0 tx=0).stop 2024-04-02T13:23:09.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.508+0000 7f527801b640 1 -- 172.21.15.44:0/315835802 shutdown_connections 2024-04-02T13:23:09.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.508+0000 7f527801b640 1 --2- 172.21.15.44:0/315835802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52700b1540 0x7f52700d1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:09.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.508+0000 7f527801b640 1 --2- 172.21.15.44:0/315835802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f527015e470 0x7f52700b32c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:09.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.508+0000 7f527801b640 1 -- 172.21.15.44:0/315835802 >> 172.21.15.44:0/315835802 conn(0x7f527009aa20 msgr2=0x7f527009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:09.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.508+0000 7f527801b640 1 -- 172.21.15.44:0/315835802 shutdown_connections 2024-04-02T13:23:09.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.508+0000 7f527801b640 1 -- 172.21.15.44:0/315835802 wait complete. 2024-04-02T13:23:09.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.509+0000 7f527801b640 1 Processor -- start 2024-04-02T13:23:09.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.509+0000 7f527801b640 1 -- start start 2024-04-02T13:23:09.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.510+0000 7f527801b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52700b1540 0x7f52701f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:09.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.510+0000 7f527801b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52700ce790 0x7f52701f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:09.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.510+0000 7f527801b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f527015e470 0x7f527015aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.510+0000 7f527801b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f52700d20f0 con 0x7f52700b1540 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.510+0000 7f527801b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f527006cb60 con 0x7f527015e470 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.510+0000 7f527801b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f52700d2270 con 0x7f52700ce790 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.510+0000 7f5275db7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52700b1540 0x7f52701f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.510+0000 7f5275db7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52700b1540 0x7f52701f6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60376/0 (socket says 172.21.15.44:60376) 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.510+0000 7f5275db7640 1 -- 172.21.15.44:0/3825576308 learned_addr learned my addr 172.21.15.44:0/3825576308 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.510+0000 7f52755b6640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52700ce790 0x7f52701f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.510+0000 7f52765b8640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f527015e470 0x7f527015aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.511+0000 7f5275db7640 1 -- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52700ce790 msgr2=0x7f52701f5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.511+0000 7f5275db7640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52700ce790 0x7f52701f5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.511+0000 7f5275db7640 1 -- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f527015e470 msgr2=0x7f527015aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.511+0000 7f5275db7640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f527015e470 0x7f527015aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.511+0000 7f5275db7640 1 -- 172.21.15.44:0/3825576308 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52700c5b30 con 0x7f52700b1540 2024-04-02T13:23:09.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.511+0000 7f52765b8640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f527015e470 0x7f527015aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:09.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.511+0000 7f5275db7640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52700b1540 0x7f52701f6fe0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f526c0098c0 tx=0x7f526c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:09.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.511+0000 7f52667fc640 1 -- 172.21.15.44:0/3825576308 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c032430 con 0x7f52700b1540 2024-04-02T13:23:09.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.511+0000 7f52667fc640 1 -- 172.21.15.44:0/3825576308 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f526c0325d0 con 0x7f52700b1540 2024-04-02T13:23:09.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.511+0000 7f527801b640 1 -- 172.21.15.44:0/3825576308 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52700c5e10 con 0x7f52700b1540 2024-04-02T13:23:09.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.511+0000 7f52667fc640 1 -- 172.21.15.44:0/3825576308 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c032980 con 0x7f52700b1540 2024-04-02T13:23:09.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.511+0000 7f527801b640 1 -- 172.21.15.44:0/3825576308 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52700c63d0 con 0x7f52700b1540 2024-04-02T13:23:09.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.512+0000 7f527801b640 1 -- 172.21.15.44:0/3825576308 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52700684f0 con 0x7f52700b1540 2024-04-02T13:23:09.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.512+0000 7f52667fc640 1 -- 172.21.15.44:0/3825576308 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f526c032b20 con 0x7f52700b1540 2024-04-02T13:23:09.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.513+0000 7f52667fc640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f525406c8f0 0x7f525406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:09.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.513+0000 7f52667fc640 1 -- 172.21.15.44:0/3825576308 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5807+0+0 (secure 0 0 0) 0x7f526c02f030 con 0x7f52700b1540 2024-04-02T13:23:09.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.513+0000 7f52755b6640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f525406c8f0 0x7f525406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:09.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.514+0000 7f52755b6640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f525406c8f0 0x7f525406edb0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5270070630 tx=0x7f5260009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:09.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.517+0000 7f52667fc640 1 -- 172.21.15.44:0/3825576308 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f526c040070 con 0x7f52700b1540 2024-04-02T13:23:09.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:09.702+0000 7f527801b640 1 -- 172.21.15.44:0/3825576308 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f52700d2d10 con 0x7f52700b1540 2024-04-02T13:23:10.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.371+0000 7f52667fc640 1 -- 172.21.15.44:0/3825576308 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v56) ==== 169+0+0 (secure 0 0 0) 0x7f526c07bc00 con 0x7f52700b1540 2024-04-02T13:23:10.374 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' removed 2024-04-02T13:23:10.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.375+0000 7f527801b640 1 -- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f525406c8f0 msgr2=0x7f525406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.375+0000 7f527801b640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f525406c8f0 0x7f525406edb0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5270070630 tx=0x7f5260009240 comp rx=0 tx=0).stop 2024-04-02T13:23:10.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.375+0000 7f527801b640 1 -- 172.21.15.44:0/3825576308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52700b1540 msgr2=0x7f52701f6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.375+0000 7f527801b640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52700b1540 0x7f52701f6fe0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f526c0098c0 tx=0x7f526c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.375+0000 7f527801b640 1 -- 172.21.15.44:0/3825576308 shutdown_connections 2024-04-02T13:23:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.375+0000 7f527801b640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f525406c8f0 0x7f525406edb0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.375+0000 7f527801b640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f527015e470 0x7f527015aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.375+0000 7f527801b640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52700ce790 0x7f52701f5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.375+0000 7f527801b640 1 --2- 172.21.15.44:0/3825576308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52700b1540 0x7f52701f6fe0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.375+0000 7f527801b640 1 -- 172.21.15.44:0/3825576308 >> 172.21.15.44:0/3825576308 conn(0x7f527009aa20 msgr2=0x7f527015d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.375+0000 7f527801b640 1 -- 172.21.15.44:0/3825576308 shutdown_connections 2024-04-02T13:23:10.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.375+0000 7f527801b640 1 -- 172.21.15.44:0/3825576308 wait complete. 2024-04-02T13:23:10.393 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T13:23:10.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.500+0000 7faf27552640 1 Processor -- start 2024-04-02T13:23:10.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.500+0000 7faf27552640 1 -- start start 2024-04-02T13:23:10.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.501+0000 7faf27552640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2015e1d0 0x7faf200d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.501+0000 7faf27552640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf200d1640 con 0x7faf200cec60 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.501+0000 7faf27552640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf200cf1f0 con 0x7faf200d2dd0 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.501+0000 7faf27552640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faf200cfe00 con 0x7faf2015e1d0 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.501+0000 7faf24aed640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7faf200cec60 0x7faf200d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40958/0 (socket says 172.21.15.44:40958) 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.501+0000 7faf24aed640 1 -- 172.21.15.44:0/1219226602 learned_addr learned my addr 172.21.15.44:0/1219226602 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf252ee640 1 --2- 172.21.15.44:0/1219226602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2015e1d0 0x7faf200d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1463283992 0 0) 0x7faf200d1640 con 0x7faf200cec60 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf08003520 con 0x7faf200cec60 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 794793823 0 0) 0x7faf08003520 con 0x7faf200cec60 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2015e1d0 msgr2=0x7faf200d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 --2- 172.21.15.44:0/1219226602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2015e1d0 0x7faf200d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 >> v1:172.21.15.69:6789/0 conn(0x7faf200d2dd0 legacy=0x7faf200b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf200cf4d0 con 0x7faf200cec60 2024-04-02T13:23:10.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7faf14001670 con 0x7faf200cec60 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 >> v1:172.21.15.44:6789/0 conn(0x7faf200cec60 legacy=0x7faf200d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 --2- 172.21.15.44:0/1219226602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf080039e0 0x7faf08003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 --2- 172.21.15.44:0/1219226602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf08004800 0x7faf08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 --2- 172.21.15.44:0/1219226602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2015e1d0 0x7faf08005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faf200cfe00 con 0x7faf08004800 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faf0800baf0 con 0x7faf080039e0 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faf0800bcf0 con 0x7faf2015e1d0 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf24aed640 1 --2- 172.21.15.44:0/1219226602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf08004800 0x7faf08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf25aef640 1 --2- 172.21.15.44:0/1219226602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2015e1d0 0x7faf08005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf252ee640 1 --2- 172.21.15.44:0/1219226602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf080039e0 0x7faf08003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf24aed640 1 -- 172.21.15.44:0/1219226602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2015e1d0 msgr2=0x7faf08005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf24aed640 1 --2- 172.21.15.44:0/1219226602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2015e1d0 0x7faf08005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf24aed640 1 -- 172.21.15.44:0/1219226602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf080039e0 msgr2=0x7faf08003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.502+0000 7faf24aed640 1 --2- 172.21.15.44:0/1219226602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf080039e0 0x7faf08003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.503+0000 7faf24aed640 1 -- 172.21.15.44:0/1219226602 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf08003520 con 0x7faf08004800 2024-04-02T13:23:10.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.503+0000 7faf24aed640 1 --2- 172.21.15.44:0/1219226602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf08004800 0x7faf08004c80 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7faf1400c210 tx=0x7faf08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d827c9a4d8e6e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:10.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.503+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf14007d30 con 0x7faf08004800 2024-04-02T13:23:10.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.503+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf14030070 con 0x7faf08004800 2024-04-02T13:23:10.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.503+0000 7faf1ffff640 1 -- 172.21.15.44:0/1219226602 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1402cd60 con 0x7faf08004800 2024-04-02T13:23:10.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.504+0000 7faf27552640 1 -- 172.21.15.44:0/1219226602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf08004800 msgr2=0x7faf08004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.504+0000 7faf27552640 1 --2- 172.21.15.44:0/1219226602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf08004800 0x7faf08004c80 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7faf1400c210 tx=0x7faf08005830 comp rx=0 tx=0).stop 2024-04-02T13:23:10.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.504+0000 7faf25aef640 1 -- 172.21.15.44:0/1219226602 reap_dead start 2024-04-02T13:23:10.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.504+0000 7faf27552640 1 -- 172.21.15.44:0/1219226602 shutdown_connections 2024-04-02T13:23:10.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.504+0000 7faf27552640 1 -- 172.21.15.44:0/1219226602 >> 172.21.15.44:0/1219226602 conn(0x7faf2009d3b0 msgr2=0x7faf200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:10.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.504+0000 7faf27552640 1 -- 172.21.15.44:0/1219226602 shutdown_connections 2024-04-02T13:23:10.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.504+0000 7faf27552640 1 -- 172.21.15.44:0/1219226602 wait complete. 2024-04-02T13:23:10.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.505+0000 7faf27552640 1 Processor -- start 2024-04-02T13:23:10.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.505+0000 7faf27552640 1 -- start start 2024-04-02T13:23:10.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf27552640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf08004800 0x7faf20160e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:10.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf27552640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf2009de20 0x7faf2015af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:10.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf27552640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf200b5400 0x7faf200b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:10.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf27552640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faf200b7fd0 con 0x7faf2009de20 2024-04-02T13:23:10.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf27552640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faf200cfa90 con 0x7faf200b5400 2024-04-02T13:23:10.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf27552640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faf200b8150 con 0x7faf08004800 2024-04-02T13:23:10.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf24aed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf2009de20 0x7faf2015af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:10.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf24aed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf2009de20 0x7faf2015af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60406/0 (socket says 172.21.15.44:60406) 2024-04-02T13:23:10.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf24aed640 1 -- 172.21.15.44:0/3505804953 learned_addr learned my addr 172.21.15.44:0/3505804953 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:10.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf252ee640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf08004800 0x7faf20160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:10.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf25aef640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf200b5400 0x7faf200b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:10.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf24aed640 1 -- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf08004800 msgr2=0x7faf20160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf24aed640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf08004800 0x7faf20160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf24aed640 1 -- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf200b5400 msgr2=0x7faf200b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf24aed640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf200b5400 0x7faf200b58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.506+0000 7faf24aed640 1 -- 172.21.15.44:0/3505804953 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf20161370 con 0x7faf2009de20 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.507+0000 7faf24aed640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf2009de20 0x7faf2015af70 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7faf1400c210 tx=0x7faf1402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.507+0000 7faf1dffb640 1 -- 172.21.15.44:0/3505804953 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf14046070 con 0x7faf2009de20 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.507+0000 7faf1dffb640 1 -- 172.21.15.44:0/3505804953 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf14004bf0 con 0x7faf2009de20 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.507+0000 7faf27552640 1 -- 172.21.15.44:0/3505804953 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2009e2b0 con 0x7faf2009de20 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.507+0000 7faf1dffb640 1 -- 172.21.15.44:0/3505804953 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf14007d10 con 0x7faf2009de20 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.507+0000 7faf27552640 1 -- 172.21.15.44:0/3505804953 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf2009e870 con 0x7faf2009de20 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.508+0000 7faf27552640 1 -- 172.21.15.44:0/3505804953 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf200684f0 con 0x7faf2009de20 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.508+0000 7faf1dffb640 1 -- 172.21.15.44:0/3505804953 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faf14007070 con 0x7faf2009de20 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.509+0000 7faf1dffb640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf0c06c8f0 0x7faf0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.509+0000 7faf1dffb640 1 -- 172.21.15.44:0/3505804953 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5335+0+0 (secure 0 0 0) 0x7faf140acbd0 con 0x7faf2009de20 2024-04-02T13:23:10.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.509+0000 7faf252ee640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf0c06c8f0 0x7faf0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:10.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.510+0000 7faf252ee640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf0c06c8f0 0x7faf0c06edb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7faf18009bc0 tx=0x7faf18009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:10.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.513+0000 7faf1dffb640 1 -- 172.21.15.44:0/3505804953 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7faf1407ab20 con 0x7faf2009de20 2024-04-02T13:23:10.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.702+0000 7faf27552640 1 -- 172.21.15.44:0/3505804953 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7faf200d47f0 con 0x7faf2009de20 2024-04-02T13:23:10.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.705+0000 7faf1dffb640 1 -- 172.21.15.44:0/3505804953 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7faf1407ee20 con 0x7faf2009de20 2024-04-02T13:23:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.707+0000 7faf27552640 1 -- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf0c06c8f0 msgr2=0x7faf0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.707+0000 7faf27552640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf0c06c8f0 0x7faf0c06edb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7faf18009bc0 tx=0x7faf18009190 comp rx=0 tx=0).stop 2024-04-02T13:23:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.707+0000 7faf27552640 1 -- 172.21.15.44:0/3505804953 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf2009de20 msgr2=0x7faf2015af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.707+0000 7faf27552640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf2009de20 0x7faf2015af70 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7faf1400c210 tx=0x7faf1402e070 comp rx=0 tx=0).stop 2024-04-02T13:23:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.707+0000 7faf27552640 1 -- 172.21.15.44:0/3505804953 shutdown_connections 2024-04-02T13:23:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.707+0000 7faf27552640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf0c06c8f0 0x7faf0c06edb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.707+0000 7faf27552640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf200b5400 0x7faf200b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.707+0000 7faf27552640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf2009de20 0x7faf2015af70 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.707+0000 7faf27552640 1 --2- 172.21.15.44:0/3505804953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf08004800 0x7faf20160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.707+0000 7faf27552640 1 -- 172.21.15.44:0/3505804953 >> 172.21.15.44:0/3505804953 conn(0x7faf2009d3b0 msgr2=0x7faf200d20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.707+0000 7faf27552640 1 -- 172.21.15.44:0/3505804953 shutdown_connections 2024-04-02T13:23:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.707+0000 7faf27552640 1 -- 172.21.15.44:0/3505804953 wait complete. 2024-04-02T13:23:10.720 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2024-04-02T13:23:10.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.820+0000 7fc8123e2640 1 Processor -- start 2024-04-02T13:23:10.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.820+0000 7fc8123e2640 1 -- start start 2024-04-02T13:23:10.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc8123e2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc80c0cd610 0x7fc80c0b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:10.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc8123e2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc80c0cafc0 con 0x7fc80c0cbd80 2024-04-02T13:23:10.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc8123e2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc80c0cb2a0 con 0x7fc80c15c2b0 2024-04-02T13:23:10.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc8123e2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc80c070120 con 0x7fc80c0cd610 2024-04-02T13:23:10.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc80b7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc80c0cd610 0x7fc80c0b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:10.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc81097f640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc80c0cbd80 0x7fc80c0b3920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40962/0 (socket says 172.21.15.44:40962) 2024-04-02T13:23:10.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc81097f640 1 -- 172.21.15.44:0/1129075719 learned_addr learned my addr 172.21.15.44:0/1129075719 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:10.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc80b7fe640 1 -- 172.21.15.44:0/1129075719 >> v1:172.21.15.69:6789/0 conn(0x7fc80c15c2b0 legacy=0x7fc80c0ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc80b7fe640 1 -- 172.21.15.44:0/1129075719 >> v1:172.21.15.44:6789/0 conn(0x7fc80c0cbd80 legacy=0x7fc80c0b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc80b7fe640 1 -- 172.21.15.44:0/1129075719 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80c0cb580 con 0x7fc80c0cd610 2024-04-02T13:23:10.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc80affd640 1 -- 172.21.15.44:0/1129075719 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 392449120 0 0) 0x7fc80c0cafc0 con 0x7fc80c0cbd80 2024-04-02T13:23:10.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc80b7fe640 1 --2- 172.21.15.44:0/1129075719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc80c0cd610 0x7fc80c0b33e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc8000097a0 tx=0x7fc80002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85db8c268a308b7a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:10.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc80affd640 1 -- 172.21.15.44:0/1129075719 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc800035020 con 0x7fc80c0cd610 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc80affd640 1 -- 172.21.15.44:0/1129075719 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc800031830 con 0x7fc80c0cd610 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.821+0000 7fc80affd640 1 -- 172.21.15.44:0/1129075719 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc800031c00 con 0x7fc80c0cd610 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.822+0000 7fc8123e2640 1 -- 172.21.15.44:0/1129075719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc80c0cd610 msgr2=0x7fc80c0b33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.822+0000 7fc8123e2640 1 --2- 172.21.15.44:0/1129075719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc80c0cd610 0x7fc80c0b33e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc8000097a0 tx=0x7fc80002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.822+0000 7fc8123e2640 1 -- 172.21.15.44:0/1129075719 shutdown_connections 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.822+0000 7fc8123e2640 1 --2- 172.21.15.44:0/1129075719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc80c0cd610 0x7fc80c0b33e0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.822+0000 7fc8123e2640 1 -- 172.21.15.44:0/1129075719 >> 172.21.15.44:0/1129075719 conn(0x7fc80c0b20e0 msgr2=0x7fc80c0b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.822+0000 7fc8123e2640 1 -- 172.21.15.44:0/1129075719 shutdown_connections 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.822+0000 7fc8123e2640 1 -- 172.21.15.44:0/1129075719 wait complete. 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.823+0000 7fc8123e2640 1 Processor -- start 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.823+0000 7fc8123e2640 1 -- start start 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc8123e2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc80c0cbd80 0x7fc80c0c6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc8123e2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c0cd610 0x7fc80c1513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:10.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc8123e2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc80c15c2b0 0x7fc80c159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc8123e2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c06fcd0 con 0x7fc80c15c2b0 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc8123e2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c15c690 con 0x7fc80c0cbd80 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc8123e2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc80c06fe50 con 0x7fc80c0cd610 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc81097f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc80c15c2b0 0x7fc80c159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc81097f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc80c15c2b0 0x7fc80c159900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60432/0 (socket says 172.21.15.44:60432) 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc81097f640 1 -- 172.21.15.44:0/3992406803 learned_addr learned my addr 172.21.15.44:0/3992406803 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc80bfff640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc80c0cbd80 0x7fc80c0c6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc80b7fe640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c0cd610 0x7fc80c1513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc81097f640 1 -- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c0cd610 msgr2=0x7fc80c1513d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc81097f640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c0cd610 0x7fc80c1513d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc81097f640 1 -- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc80c0cbd80 msgr2=0x7fc80c0c6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc81097f640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc80c0cbd80 0x7fc80c0c6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc81097f640 1 -- 172.21.15.44:0/3992406803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80c154760 con 0x7fc80c15c2b0 2024-04-02T13:23:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc80b7fe640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c0cd610 0x7fc80c1513d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:10.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.824+0000 7fc81097f640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc80c15c2b0 0x7fc80c159900 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc80400c7d0 tx=0x7fc80400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:10.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.825+0000 7fc808ff9640 1 -- 172.21.15.44:0/3992406803 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804018070 con 0x7fc80c15c2b0 2024-04-02T13:23:10.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.825+0000 7fc808ff9640 1 -- 172.21.15.44:0/3992406803 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc80400a640 con 0x7fc80c15c2b0 2024-04-02T13:23:10.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.825+0000 7fc8123e2640 1 -- 172.21.15.44:0/3992406803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc80c0c73c0 con 0x7fc80c15c2b0 2024-04-02T13:23:10.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.825+0000 7fc808ff9640 1 -- 172.21.15.44:0/3992406803 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80400aa70 con 0x7fc80c15c2b0 2024-04-02T13:23:10.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.825+0000 7fc8123e2640 1 -- 172.21.15.44:0/3992406803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc80c1541a0 con 0x7fc80c15c2b0 2024-04-02T13:23:10.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.826+0000 7fc8123e2640 1 -- 172.21.15.44:0/3992406803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc80c0684f0 con 0x7fc80c15c2b0 2024-04-02T13:23:10.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.826+0000 7fc808ff9640 1 -- 172.21.15.44:0/3992406803 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc80401b070 con 0x7fc80c15c2b0 2024-04-02T13:23:10.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.827+0000 7fc808ff9640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc7f806c8f0 0x7fc7f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:10.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.827+0000 7fc808ff9640 1 -- 172.21.15.44:0/3992406803 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5335+0+0 (secure 0 0 0) 0x7fc80408af90 con 0x7fc80c15c2b0 2024-04-02T13:23:10.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.827+0000 7fc80bfff640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc7f806c8f0 0x7fc7f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:10.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.827+0000 7fc80bfff640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc7f806c8f0 0x7fc7f806edb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc000f60 tx=0x7fc7fc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:10.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:10.831+0000 7fc808ff9640 1 -- 172.21.15.44:0/3992406803 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc80c0c73c0 con 0x7fc80c15c2b0 2024-04-02T13:23:11.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.093+0000 7fc8123e2640 1 -- 172.21.15.44:0/3992406803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) -- 0x7fc80c0caa60 con 0x7fc80c15c2b0 2024-04-02T13:23:11.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.379+0000 7fc808ff9640 1 -- 172.21.15.44:0/3992406803 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v165) ==== 149+0+0 (secure 0 0 0) 0x7fc804058ee0 con 0x7fc80c15c2b0 2024-04-02T13:23:11.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.381+0000 7fc8123e2640 1 -- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc7f806c8f0 msgr2=0x7fc7f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:11.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.381+0000 7fc8123e2640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc7f806c8f0 0x7fc7f806edb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc000f60 tx=0x7fc7fc009040 comp rx=0 tx=0).stop 2024-04-02T13:23:11.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.381+0000 7fc8123e2640 1 -- 172.21.15.44:0/3992406803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc80c15c2b0 msgr2=0x7fc80c159900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:11.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.381+0000 7fc8123e2640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc80c15c2b0 0x7fc80c159900 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc80400c7d0 tx=0x7fc80400cc90 comp rx=0 tx=0).stop 2024-04-02T13:23:11.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.381+0000 7fc8123e2640 1 -- 172.21.15.44:0/3992406803 shutdown_connections 2024-04-02T13:23:11.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.381+0000 7fc8123e2640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc7f806c8f0 0x7fc7f806edb0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.381+0000 7fc8123e2640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc80c15c2b0 0x7fc80c159900 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.381+0000 7fc8123e2640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c0cd610 0x7fc80c1513d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.381+0000 7fc8123e2640 1 --2- 172.21.15.44:0/3992406803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc80c0cbd80 0x7fc80c0c6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.381+0000 7fc8123e2640 1 -- 172.21.15.44:0/3992406803 >> 172.21.15.44:0/3992406803 conn(0x7fc80c0b20e0 msgr2=0x7fc80c15a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:11.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.381+0000 7fc8123e2640 1 -- 172.21.15.44:0/3992406803 shutdown_connections 2024-04-02T13:23:11.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.381+0000 7fc8123e2640 1 -- 172.21.15.44:0/3992406803 wait complete. 2024-04-02T13:23:11.394 INFO:tasks.cephfs_test_runner:That the MDS can repair an inodes backtrace in the data pool ... ok 2024-04-02T13:23:11.395 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-04-02T13:23:11.395 INFO:tasks.cephfs_test_runner:test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-02T13:23:11.395 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:23:11.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.496+0000 7f2192967640 1 Processor -- start 2024-04-02T13:23:11.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.496+0000 7f2192967640 1 -- start start 2024-04-02T13:23:11.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.497+0000 7f2192967640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f218c159db0 0x7f218c15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:11.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.497+0000 7f2192967640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f218c071e50 0x7f218c0cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:11.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.497+0000 7f2192967640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f218c0726c0 con 0x7f218c0ccc20 2024-04-02T13:23:11.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.497+0000 7f2192967640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f218c073240 con 0x7f218c159db0 2024-04-02T13:23:11.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.497+0000 7f2192967640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f218c0cd050 con 0x7f218c071e50 2024-04-02T13:23:11.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.497+0000 7f218b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f218c071e50 0x7f218c0cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:11.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.497+0000 7f2190f04640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f218c0ccc20 0x7f218c06d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:40582/0 (socket says 172.21.15.44:40582) 2024-04-02T13:23:11.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.497+0000 7f218bfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f218c159db0 0x7f218c15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:11.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.497+0000 7f2190f04640 1 -- 172.21.15.44:0/3908282296 learned_addr learned my addr 172.21.15.44:0/3908282296 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:11.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.498+0000 7f218affd640 1 -- 172.21.15.44:0/3908282296 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2331275310 0 0) 0x7f218c0726c0 con 0x7f218c0ccc20 2024-04-02T13:23:11.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.498+0000 7f218affd640 1 -- 172.21.15.44:0/3908282296 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2178003520 con 0x7f218c0ccc20 2024-04-02T13:23:11.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.498+0000 7f218bfff640 1 -- 172.21.15.44:0/3908282296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f218c071e50 msgr2=0x7f218c0cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:11.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.498+0000 7f218bfff640 1 --2- 172.21.15.44:0/3908282296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f218c071e50 0x7f218c0cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.498+0000 7f218bfff640 1 -- 172.21.15.44:0/3908282296 >> v1:172.21.15.69:6789/0 conn(0x7f218c0ccc20 legacy=0x7f218c06d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:11.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.498+0000 7f218bfff640 1 -- 172.21.15.44:0/3908282296 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218c0729a0 con 0x7f218c159db0 2024-04-02T13:23:11.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.498+0000 7f218b7fe640 1 --2- 172.21.15.44:0/3908282296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f218c071e50 0x7f218c0cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:23:11.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.498+0000 7f218bfff640 1 --2- 172.21.15.44:0/3908282296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f218c159db0 0x7f218c15c6d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f217c0097a0 tx=0x7f217c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db4fb70eafbb08c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:11.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.499+0000 7f218affd640 1 -- 172.21.15.44:0/3908282296 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c0315a0 con 0x7f218c159db0 2024-04-02T13:23:11.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.499+0000 7f218affd640 1 -- 172.21.15.44:0/3908282296 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f217c037070 con 0x7f218c159db0 2024-04-02T13:23:11.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.499+0000 7f218affd640 1 -- 172.21.15.44:0/3908282296 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c031960 con 0x7f218c159db0 2024-04-02T13:23:11.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.499+0000 7f2192967640 1 -- 172.21.15.44:0/3908282296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f218c159db0 msgr2=0x7f218c15c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:11.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.499+0000 7f2192967640 1 --2- 172.21.15.44:0/3908282296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f218c159db0 0x7f218c15c6d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f217c0097a0 tx=0x7f217c02b840 comp rx=0 tx=0).stop 2024-04-02T13:23:11.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.499+0000 7f2192967640 1 -- 172.21.15.44:0/3908282296 shutdown_connections 2024-04-02T13:23:11.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.499+0000 7f2192967640 1 --2- 172.21.15.44:0/3908282296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f218c071e50 0x7f218c0cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.499+0000 7f2192967640 1 --2- 172.21.15.44:0/3908282296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f218c159db0 0x7f218c15c6d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.499+0000 7f2192967640 1 -- 172.21.15.44:0/3908282296 >> 172.21.15.44:0/3908282296 conn(0x7f218c0cfcd0 msgr2=0x7f218c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:11.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.499+0000 7f2192967640 1 -- 172.21.15.44:0/3908282296 shutdown_connections 2024-04-02T13:23:11.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.499+0000 7f2192967640 1 -- 172.21.15.44:0/3908282296 wait complete. 2024-04-02T13:23:11.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.500+0000 7f2192967640 1 Processor -- start 2024-04-02T13:23:11.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.500+0000 7f2192967640 1 -- start start 2024-04-02T13:23:11.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.501+0000 7f2192967640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f218c071e50 0x7f218c148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:11.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.501+0000 7f2192967640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f218c0ccc20 0x7f218c15bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:11.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.501+0000 7f2192967640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f218c159db0 0x7f218c15c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:11.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.501+0000 7f2192967640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f218c0722d0 con 0x7f218c159db0 2024-04-02T13:23:11.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.501+0000 7f2192967640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f218c072f60 con 0x7f218c071e50 2024-04-02T13:23:11.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.501+0000 7f2192967640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f218c072450 con 0x7f218c0ccc20 2024-04-02T13:23:11.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.501+0000 7f2190f04640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f218c159db0 0x7f218c15c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:11.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.501+0000 7f2190f04640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f218c159db0 0x7f218c15c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60444/0 (socket says 172.21.15.44:60444) 2024-04-02T13:23:11.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.501+0000 7f2190f04640 1 -- 172.21.15.44:0/3121113318 learned_addr learned my addr 172.21.15.44:0/3121113318 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:11.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.501+0000 7f218bfff640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f218c071e50 0x7f218c148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:11.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.501+0000 7f218b7fe640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f218c0ccc20 0x7f218c15bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:11.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.502+0000 7f2190f04640 1 -- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f218c0ccc20 msgr2=0x7f218c15bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:11.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.502+0000 7f2190f04640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f218c0ccc20 0x7f218c15bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.502+0000 7f2190f04640 1 -- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f218c071e50 msgr2=0x7f218c148990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:11.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.502+0000 7f2190f04640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f218c071e50 0x7f218c148990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.502+0000 7f2190f04640 1 -- 172.21.15.44:0/3121113318 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218c156240 con 0x7f218c159db0 2024-04-02T13:23:11.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.502+0000 7f2190f04640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f218c159db0 0x7f218c15c0e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f218400cb10 tx=0x7f218400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:11.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.502+0000 7f2188ff9640 1 -- 172.21.15.44:0/3121113318 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2184017070 con 0x7f218c159db0 2024-04-02T13:23:11.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.502+0000 7f2188ff9640 1 -- 172.21.15.44:0/3121113318 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f218400ac70 con 0x7f218c159db0 2024-04-02T13:23:11.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.502+0000 7f2192967640 1 -- 172.21.15.44:0/3121113318 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f218c156520 con 0x7f218c159db0 2024-04-02T13:23:11.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.502+0000 7f2188ff9640 1 -- 172.21.15.44:0/3121113318 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218400ae60 con 0x7f218c159db0 2024-04-02T13:23:11.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.502+0000 7f2192967640 1 -- 172.21.15.44:0/3121113318 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f218c158d10 con 0x7f218c159db0 2024-04-02T13:23:11.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.503+0000 7f2192967640 1 -- 172.21.15.44:0/3121113318 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f218c074f20 con 0x7f218c159db0 2024-04-02T13:23:11.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.503+0000 7f2188ff9640 1 -- 172.21.15.44:0/3121113318 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f218401a070 con 0x7f218c159db0 2024-04-02T13:23:11.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.504+0000 7f2188ff9640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f216c06c8f0 0x7f216c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:11.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.504+0000 7f2188ff9640 1 -- 172.21.15.44:0/3121113318 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5335+0+0 (secure 0 0 0) 0x7f218408b750 con 0x7f218c159db0 2024-04-02T13:23:11.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.504+0000 7f218bfff640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f216c06c8f0 0x7f216c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:11.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.505+0000 7f218bfff640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f216c06c8f0 0x7f216c06edb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f217c036000 tx=0x7f217c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.508+0000 7f2188ff9640 1 -- 172.21.15.44:0/3121113318 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f21840596b0 con 0x7f218c159db0 2024-04-02T13:23:11.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.693+0000 7f2192967640 1 -- 172.21.15.44:0/3121113318 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f218c0684f0 con 0x7f218c159db0 2024-04-02T13:23:11.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.693+0000 7f2188ff9640 1 -- 172.21.15.44:0/3121113318 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+13245 (secure 0 0 0) 0x7f218405d9b0 con 0x7f218c159db0 2024-04-02T13:23:11.695 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:11.695 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":56,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:23:10.371554+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6837/1745145720":"2024-04-03T13:23:07.350562+0000","172.21.15.69:6835/1745145720":"2024-04-03T13:23:07.350562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:23:11.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.696+0000 7f2192967640 1 -- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f216c06c8f0 msgr2=0x7f216c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:11.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.696+0000 7f2192967640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f216c06c8f0 0x7f216c06edb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f217c036000 tx=0x7f217c03d040 comp rx=0 tx=0).stop 2024-04-02T13:23:11.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.696+0000 7f2192967640 1 -- 172.21.15.44:0/3121113318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f218c159db0 msgr2=0x7f218c15c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:11.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.696+0000 7f2192967640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f218c159db0 0x7f218c15c0e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f218400cb10 tx=0x7f218400a060 comp rx=0 tx=0).stop 2024-04-02T13:23:11.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.696+0000 7f2192967640 1 -- 172.21.15.44:0/3121113318 shutdown_connections 2024-04-02T13:23:11.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.696+0000 7f2192967640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f216c06c8f0 0x7f216c06edb0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.696+0000 7f2192967640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f218c159db0 0x7f218c15c0e0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.696+0000 7f2192967640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f218c0ccc20 0x7f218c15bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.696+0000 7f2192967640 1 --2- 172.21.15.44:0/3121113318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f218c071e50 0x7f218c148990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.696+0000 7f2192967640 1 -- 172.21.15.44:0/3121113318 >> 172.21.15.44:0/3121113318 conn(0x7f218c0cfcd0 msgr2=0x7f218c0acf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:11.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.696+0000 7f2192967640 1 -- 172.21.15.44:0/3121113318 shutdown_connections 2024-04-02T13:23:11.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.696+0000 7f2192967640 1 -- 172.21.15.44:0/3121113318 wait complete. 2024-04-02T13:23:11.709 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:23:11.709 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:23:11.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.812+0000 7f99b6996640 1 Processor -- start 2024-04-02T13:23:11.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.812+0000 7f99b6996640 1 -- start start 2024-04-02T13:23:11.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.813+0000 7f99b6996640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99b006b4d0 0x7f99b006b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:11.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.813+0000 7f99b6996640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f99b00cd770 0x7f99b006eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.813+0000 7f99b6996640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99b006bdf0 con 0x7f99b00721f0 2024-04-02T13:23:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.813+0000 7f99b6996640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f99b006f3f0 con 0x7f99b00cd770 2024-04-02T13:23:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.813+0000 7f99b6996640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f99b006f6d0 con 0x7f99b006b4d0 2024-04-02T13:23:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.813+0000 7f99affff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99b006b4d0 0x7f99b006b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.813+0000 7f99af7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f99b00cd770 0x7f99b006eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.813+0000 7f99af7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f99b00cd770 0x7f99b006eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:44114/0 (socket says 172.21.15.44:44114) 2024-04-02T13:23:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.813+0000 7f99b4f33640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f99b00721f0 0x7f99b0072660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:40592/0 (socket says 172.21.15.44:40592) 2024-04-02T13:23:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.813+0000 7f99af7fe640 1 -- 172.21.15.44:0/881383745 learned_addr learned my addr 172.21.15.44:0/881383745 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.814+0000 7f99af7fe640 1 -- 172.21.15.44:0/881383745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99b006b4d0 msgr2=0x7f99b006b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.814+0000 7f99af7fe640 1 --2- 172.21.15.44:0/881383745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99b006b4d0 0x7f99b006b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.814+0000 7f99af7fe640 1 -- 172.21.15.44:0/881383745 >> v1:172.21.15.69:6789/0 conn(0x7f99b00721f0 legacy=0x7f99b0072660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.814+0000 7f99af7fe640 1 -- 172.21.15.44:0/881383745 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b006f9b0 con 0x7f99b00cd770 2024-04-02T13:23:11.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.814+0000 7f99af7fe640 1 --2- 172.21.15.44:0/881383745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99b00cd770 0x7f99b006eba0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f99a0009870 tx=0x7f99a002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d005e88427b42806 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:11.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.814+0000 7f99aeffd640 1 -- 172.21.15.44:0/881383745 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a0034070 con 0x7f99b00cd770 2024-04-02T13:23:11.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.815+0000 7f99aeffd640 1 -- 172.21.15.44:0/881383745 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99a0030840 con 0x7f99b00cd770 2024-04-02T13:23:11.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.815+0000 7f99aeffd640 1 -- 172.21.15.44:0/881383745 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a0030c10 con 0x7f99b00cd770 2024-04-02T13:23:11.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.815+0000 7f99b6996640 1 -- 172.21.15.44:0/881383745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99b00cd770 msgr2=0x7f99b006eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:11.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.815+0000 7f99b6996640 1 --2- 172.21.15.44:0/881383745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99b00cd770 0x7f99b006eba0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f99a0009870 tx=0x7f99a002b980 comp rx=0 tx=0).stop 2024-04-02T13:23:11.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.815+0000 7f99b6996640 1 -- 172.21.15.44:0/881383745 shutdown_connections 2024-04-02T13:23:11.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.815+0000 7f99b6996640 1 --2- 172.21.15.44:0/881383745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99b00cd770 0x7f99b006eba0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.815+0000 7f99b6996640 1 --2- 172.21.15.44:0/881383745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99b006b4d0 0x7f99b006b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.815+0000 7f99b6996640 1 -- 172.21.15.44:0/881383745 >> 172.21.15.44:0/881383745 conn(0x7f99b00b2a10 msgr2=0x7f99b00b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:11.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.815+0000 7f99b6996640 1 -- 172.21.15.44:0/881383745 shutdown_connections 2024-04-02T13:23:11.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.815+0000 7f99b6996640 1 -- 172.21.15.44:0/881383745 wait complete. 2024-04-02T13:23:11.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.816+0000 7f99b6996640 1 Processor -- start 2024-04-02T13:23:11.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.816+0000 7f99b6996640 1 -- start start 2024-04-02T13:23:11.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.817+0000 7f99b6996640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99b006b4d0 0x7f99b01f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:11.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.817+0000 7f99b6996640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99b00721f0 0x7f99b015f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:11.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.817+0000 7f99b6996640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99b00cd770 0x7f99b01476f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:11.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.817+0000 7f99b6996640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f99b006f0e0 con 0x7f99b006b4d0 2024-04-02T13:23:11.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.817+0000 7f99b6996640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f99b006ff70 con 0x7f99b00cd770 2024-04-02T13:23:11.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.817+0000 7f99b6996640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f99b015c470 con 0x7f99b00721f0 2024-04-02T13:23:11.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.817+0000 7f99affff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99b006b4d0 0x7f99b01f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:11.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.817+0000 7f99affff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99b006b4d0 0x7f99b01f5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60448/0 (socket says 172.21.15.44:60448) 2024-04-02T13:23:11.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.817+0000 7f99affff640 1 -- 172.21.15.44:0/2280921390 learned_addr learned my addr 172.21.15.44:0/2280921390 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:11.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.817+0000 7f99af7fe640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99b00721f0 0x7f99b015f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:11.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.817+0000 7f99b4f33640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99b00cd770 0x7f99b01476f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:11.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.818+0000 7f99affff640 1 -- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99b00721f0 msgr2=0x7f99b015f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:11.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.818+0000 7f99affff640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99b00721f0 0x7f99b015f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.818+0000 7f99affff640 1 -- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99b00cd770 msgr2=0x7f99b01476f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:11.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.818+0000 7f99affff640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99b00cd770 0x7f99b01476f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:11.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.818+0000 7f99affff640 1 -- 172.21.15.44:0/2280921390 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b015eb50 con 0x7f99b006b4d0 2024-04-02T13:23:11.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.818+0000 7f99affff640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99b006b4d0 0x7f99b01f5fc0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f99a400e9e0 tx=0x7f99a400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:11.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.818+0000 7f99acff9640 1 -- 172.21.15.44:0/2280921390 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a4017070 con 0x7f99b006b4d0 2024-04-02T13:23:11.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.818+0000 7f99acff9640 1 -- 172.21.15.44:0/2280921390 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99a400c9d0 con 0x7f99b006b4d0 2024-04-02T13:23:11.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.818+0000 7f99b6996640 1 -- 172.21.15.44:0/2280921390 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b015ee30 con 0x7f99b006b4d0 2024-04-02T13:23:11.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.818+0000 7f99acff9640 1 -- 172.21.15.44:0/2280921390 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a400cdc0 con 0x7f99b006b4d0 2024-04-02T13:23:11.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.818+0000 7f99b6996640 1 -- 172.21.15.44:0/2280921390 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b015f650 con 0x7f99b006b4d0 2024-04-02T13:23:11.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.819+0000 7f99b6996640 1 -- 172.21.15.44:0/2280921390 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99b00684f0 con 0x7f99b006b4d0 2024-04-02T13:23:11.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.819+0000 7f99acff9640 1 -- 172.21.15.44:0/2280921390 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f99a401a070 con 0x7f99b006b4d0 2024-04-02T13:23:11.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.820+0000 7f99acff9640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f999c06c8f0 0x7f999c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:11.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.820+0000 7f99acff9640 1 -- 172.21.15.44:0/2280921390 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5335+0+0 (secure 0 0 0) 0x7f99a408b9a0 con 0x7f99b006b4d0 2024-04-02T13:23:11.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.820+0000 7f99af7fe640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f999c06c8f0 0x7f999c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:11.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.821+0000 7f99af7fe640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f999c06c8f0 0x7f999c06edb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f99a0036000 tx=0x7f99a003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:11.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:11.824+0000 7f99acff9640 1 -- 172.21.15.44:0/2280921390 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f99a4059900 con 0x7f99b006b4d0 2024-04-02T13:23:12.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.008+0000 7f99b6996640 1 -- 172.21.15.44:0/2280921390 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f99b006c2b0 con 0x7f99b006b4d0 2024-04-02T13:23:12.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.009+0000 7f99acff9640 1 -- 172.21.15.44:0/2280921390 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7f99a405dc00 con 0x7f99b006b4d0 2024-04-02T13:23:12.011 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:23:12.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.011+0000 7f99b6996640 1 -- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f999c06c8f0 msgr2=0x7f999c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:12.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.011+0000 7f99b6996640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f999c06c8f0 0x7f999c06edb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f99a0036000 tx=0x7f99a003b040 comp rx=0 tx=0).stop 2024-04-02T13:23:12.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.011+0000 7f99b6996640 1 -- 172.21.15.44:0/2280921390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99b006b4d0 msgr2=0x7f99b01f5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:12.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.011+0000 7f99b6996640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99b006b4d0 0x7f99b01f5fc0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f99a400e9e0 tx=0x7f99a400eea0 comp rx=0 tx=0).stop 2024-04-02T13:23:12.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.011+0000 7f99b6996640 1 -- 172.21.15.44:0/2280921390 shutdown_connections 2024-04-02T13:23:12.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.011+0000 7f99b6996640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f999c06c8f0 0x7f999c06edb0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:12.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.011+0000 7f99b6996640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99b00cd770 0x7f99b01476f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:12.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.011+0000 7f99b6996640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99b00721f0 0x7f99b015f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:12.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.011+0000 7f99b6996640 1 --2- 172.21.15.44:0/2280921390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99b006b4d0 0x7f99b01f5fc0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:12.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.011+0000 7f99b6996640 1 -- 172.21.15.44:0/2280921390 >> 172.21.15.44:0/2280921390 conn(0x7f99b00b2a10 msgr2=0x7f99b00b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:12.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.011+0000 7f99b6996640 1 -- 172.21.15.44:0/2280921390 shutdown_connections 2024-04-02T13:23:12.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.011+0000 7f99b6996640 1 -- 172.21.15.44:0/2280921390 wait complete. 2024-04-02T13:23:12.023 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2024-04-02T13:23:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.126+0000 7f5173662640 1 Processor -- start 2024-04-02T13:23:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.126+0000 7f5173662640 1 -- start start 2024-04-02T13:23:12.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.127+0000 7f5173662640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c159db0 0x7f516c15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:12.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.127+0000 7f5173662640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f516c0726c0 con 0x7f516c0ccdf0 2024-04-02T13:23:12.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.127+0000 7f5173662640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f516c0729a0 con 0x7f516c071e50 2024-04-02T13:23:12.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.127+0000 7f5173662640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f516c073240 con 0x7f516c159db0 2024-04-02T13:23:12.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.127+0000 7f5171bff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f516c0ccdf0 0x7f516c165cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40972/0 (socket says 172.21.15.44:40972) 2024-04-02T13:23:12.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.127+0000 7f5171bff640 1 -- 172.21.15.44:0/2730107292 learned_addr learned my addr 172.21.15.44:0/2730107292 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:12.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.127+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3227419466 0 0) 0x7f516c0726c0 con 0x7f516c0ccdf0 2024-04-02T13:23:12.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.127+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5158003520 con 0x7f516c0ccdf0 2024-04-02T13:23:12.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.127+0000 7f51713fe640 1 --2- 172.21.15.44:0/2730107292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c159db0 0x7f516c15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:12.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1693939491 0 0) 0x7f5158003520 con 0x7f516c0ccdf0 2024-04-02T13:23:12.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c159db0 msgr2=0x7f516c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:12.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5163fff640 1 --2- 172.21.15.44:0/2730107292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c159db0 0x7f516c15c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:12.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 >> v1:172.21.15.69:6789/0 conn(0x7f516c071e50 legacy=0x7f516c0cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:12.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516c072c80 con 0x7f516c0ccdf0 2024-04-02T13:23:12.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f51640019f0 con 0x7f516c0ccdf0 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f51713fe640 1 --2- 172.21.15.44:0/2730107292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c159db0 0x7f516c15c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 >> v1:172.21.15.44:6789/0 conn(0x7f516c0ccdf0 legacy=0x7f516c165cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5163fff640 1 --2- 172.21.15.44:0/2730107292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c159db0 0x7f51580039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5163fff640 1 --2- 172.21.15.44:0/2730107292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5158003fd0 0x7f5158004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5163fff640 1 --2- 172.21.15.44:0/2730107292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5158004d40 0x7f51580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f516c073240 con 0x7f5158004d40 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f515800bbb0 con 0x7f5158003fd0 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f515800bdb0 con 0x7f516c159db0 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5171bff640 1 --2- 172.21.15.44:0/2730107292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5158004d40 0x7f51580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f51713fe640 1 --2- 172.21.15.44:0/2730107292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c159db0 0x7f51580039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5170bfd640 1 --2- 172.21.15.44:0/2730107292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5158003fd0 0x7f5158004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5171bff640 1 -- 172.21.15.44:0/2730107292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c159db0 msgr2=0x7f51580039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5171bff640 1 --2- 172.21.15.44:0/2730107292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c159db0 0x7f51580039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5171bff640 1 -- 172.21.15.44:0/2730107292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5158003fd0 msgr2=0x7f5158004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:12.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5171bff640 1 --2- 172.21.15.44:0/2730107292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5158003fd0 0x7f5158004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:12.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.128+0000 7f5171bff640 1 -- 172.21.15.44:0/2730107292 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5158003520 con 0x7f5158004d40 2024-04-02T13:23:12.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.129+0000 7f5171bff640 1 --2- 172.21.15.44:0/2730107292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5158004d40 0x7f51580051c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f516400a840 tx=0x7f5158005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd4c084ee03e4752 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:12.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.129+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516400a460 con 0x7f5158004d40 2024-04-02T13:23:12.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.129+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5164034650 con 0x7f5158004d40 2024-04-02T13:23:12.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.129+0000 7f5163fff640 1 -- 172.21.15.44:0/2730107292 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164034a80 con 0x7f5158004d40 2024-04-02T13:23:12.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.130+0000 7f5173662640 1 -- 172.21.15.44:0/2730107292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5158004d40 msgr2=0x7f51580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:12.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.130+0000 7f5173662640 1 --2- 172.21.15.44:0/2730107292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5158004d40 0x7f51580051c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f516400a840 tx=0x7f5158005830 comp rx=0 tx=0).stop 2024-04-02T13:23:12.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.130+0000 7f5171bff640 1 -- 172.21.15.44:0/2730107292 reap_dead start 2024-04-02T13:23:12.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.130+0000 7f5173662640 1 -- 172.21.15.44:0/2730107292 shutdown_connections 2024-04-02T13:23:12.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.130+0000 7f5173662640 1 -- 172.21.15.44:0/2730107292 >> 172.21.15.44:0/2730107292 conn(0x7f516c0cfcd0 msgr2=0x7f516c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:12.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.130+0000 7f5173662640 1 -- 172.21.15.44:0/2730107292 shutdown_connections 2024-04-02T13:23:12.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.130+0000 7f5173662640 1 -- 172.21.15.44:0/2730107292 wait complete. 2024-04-02T13:23:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.131+0000 7f5173662640 1 Processor -- start 2024-04-02T13:23:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.131+0000 7f5173662640 1 -- start start 2024-04-02T13:23:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5173662640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5158004d40 0x7f516c1480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5173662640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c156200 0x7f516c0b4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5173662640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f516c0b47c0 0x7f516c14a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5173662640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f516c072450 con 0x7f516c0b47c0 2024-04-02T13:23:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5173662640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f516c0722d0 con 0x7f5158004d40 2024-04-02T13:23:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5173662640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f516c06e650 con 0x7f516c156200 2024-04-02T13:23:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5171bff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f516c0b47c0 0x7f516c14a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5171bff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f516c0b47c0 0x7f516c14a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60460/0 (socket says 172.21.15.44:60460) 2024-04-02T13:23:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5171bff640 1 -- 172.21.15.44:0/4147662364 learned_addr learned my addr 172.21.15.44:0/4147662364 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f51713fe640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5158004d40 0x7f516c1480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5170bfd640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c156200 0x7f516c0b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5171bff640 1 -- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c156200 msgr2=0x7f516c0b4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5171bff640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c156200 0x7f516c0b4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5171bff640 1 -- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5158004d40 msgr2=0x7f516c1480f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5171bff640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5158004d40 0x7f516c1480f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5171bff640 1 -- 172.21.15.44:0/4147662364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516c0b31e0 con 0x7f516c0b47c0 2024-04-02T13:23:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5171bff640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f516c0b47c0 0x7f516c14a940 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f5164001cf0 tx=0x7f516c072e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.132+0000 7f5161ffb640 1 -- 172.21.15.44:0/4147662364 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164001250 con 0x7f516c0b47c0 2024-04-02T13:23:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.133+0000 7f5161ffb640 1 -- 172.21.15.44:0/4147662364 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5164044070 con 0x7f516c0b47c0 2024-04-02T13:23:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.133+0000 7f5173662640 1 -- 172.21.15.44:0/4147662364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f516c0b34c0 con 0x7f516c0b47c0 2024-04-02T13:23:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.133+0000 7f5161ffb640 1 -- 172.21.15.44:0/4147662364 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51640019f0 con 0x7f516c0b47c0 2024-04-02T13:23:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.133+0000 7f5173662640 1 -- 172.21.15.44:0/4147662364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f516c0b3a80 con 0x7f516c0b47c0 2024-04-02T13:23:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.134+0000 7f5173662640 1 -- 172.21.15.44:0/4147662364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f516c074f20 con 0x7f516c0b47c0 2024-04-02T13:23:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.134+0000 7f5161ffb640 1 -- 172.21.15.44:0/4147662364 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f516403c070 con 0x7f516c0b47c0 2024-04-02T13:23:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.135+0000 7f5161ffb640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f515006ca90 0x7f515006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.135+0000 7f5161ffb640 1 -- 172.21.15.44:0/4147662364 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5335+0+0 (secure 0 0 0) 0x7f51640ad320 con 0x7f516c0b47c0 2024-04-02T13:23:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.135+0000 7f51713fe640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f515006ca90 0x7f515006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.135+0000 7f51713fe640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f515006ca90 0x7f515006ef50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5168006fa0 tx=0x7f5168008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.139+0000 7f5161ffb640 1 -- 172.21.15.44:0/4147662364 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f516407b390 con 0x7f516c0b47c0 2024-04-02T13:23:12.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:12.395+0000 7f5173662640 1 -- 172.21.15.44:0/4147662364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) -- 0x7f516c0ae9c0 con 0x7f516c0b47c0 2024-04-02T13:23:13.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.386+0000 7f5161ffb640 1 -- 172.21.15.44:0/4147662364 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v167) ==== 172+0+0 (secure 0 0 0) 0x7f516407f690 con 0x7f516c0b47c0 2024-04-02T13:23:13.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.389+0000 7f5173662640 1 -- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f515006ca90 msgr2=0x7f515006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:13.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.389+0000 7f5173662640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f515006ca90 0x7f515006ef50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5168006fa0 tx=0x7f5168008040 comp rx=0 tx=0).stop 2024-04-02T13:23:13.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.389+0000 7f5173662640 1 -- 172.21.15.44:0/4147662364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f516c0b47c0 msgr2=0x7f516c14a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:13.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.389+0000 7f5173662640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f516c0b47c0 0x7f516c14a940 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f5164001cf0 tx=0x7f516c072e50 comp rx=0 tx=0).stop 2024-04-02T13:23:13.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.389+0000 7f5173662640 1 -- 172.21.15.44:0/4147662364 shutdown_connections 2024-04-02T13:23:13.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.389+0000 7f5173662640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f515006ca90 0x7f515006ef50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.389+0000 7f5173662640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f516c0b47c0 0x7f516c14a940 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.389+0000 7f5173662640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c156200 0x7f516c0b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.389+0000 7f5173662640 1 --2- 172.21.15.44:0/4147662364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5158004d40 0x7f516c1480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.389+0000 7f5173662640 1 -- 172.21.15.44:0/4147662364 >> 172.21.15.44:0/4147662364 conn(0x7f516c0cfcd0 msgr2=0x7f516c0b1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:13.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.389+0000 7f5173662640 1 -- 172.21.15.44:0/4147662364 shutdown_connections 2024-04-02T13:23:13.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.389+0000 7f5173662640 1 -- 172.21.15.44:0/4147662364 wait complete. 2024-04-02T13:23:13.404 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T13:23:13.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.513+0000 7f6545f8d640 1 Processor -- start 2024-04-02T13:23:13.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.513+0000 7f6545f8d640 1 -- start start 2024-04-02T13:23:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f6545f8d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f654015e470 0x7f65400b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f6545f8d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f65400b1540 0x7f65400d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f6545f8d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65400ce790 0x7f65400d20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f6545f8d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f65400d2630 con 0x7f654015e470 2024-04-02T13:23:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f6545f8d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f65400ced00 con 0x7f65400b1540 2024-04-02T13:23:13.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f6545f8d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f65400cefe0 con 0x7f65400ce790 2024-04-02T13:23:13.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f653f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f654015e470 0x7f65400b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:13.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f653f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f654015e470 0x7f65400b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60468/0 (socket says 172.21.15.44:60468) 2024-04-02T13:23:13.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f653f7fe640 1 -- 172.21.15.44:0/690205180 learned_addr learned my addr 172.21.15.44:0/690205180 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:13.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f653effd640 1 --2- 172.21.15.44:0/690205180 >> v2:172.21.15.69:3300/0 conn(0x7f65400b1540 0x7f65400d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:13.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f653ffff640 1 --2- 172.21.15.44:0/690205180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65400ce790 0x7f65400d20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:13.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f653f7fe640 1 -- 172.21.15.44:0/690205180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65400ce790 msgr2=0x7f65400d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:13.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f653f7fe640 1 --2- 172.21.15.44:0/690205180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65400ce790 0x7f65400d20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f653f7fe640 1 -- 172.21.15.44:0/690205180 >> v2:172.21.15.69:3300/0 conn(0x7f65400b1540 msgr2=0x7f65400d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:13.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f653f7fe640 1 --2- 172.21.15.44:0/690205180 >> v2:172.21.15.69:3300/0 conn(0x7f65400b1540 0x7f65400d1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f653f7fe640 1 -- 172.21.15.44:0/690205180 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65400cf2c0 con 0x7f654015e470 2024-04-02T13:23:13.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.514+0000 7f653ffff640 1 --2- 172.21.15.44:0/690205180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65400ce790 0x7f65400d20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:13.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.515+0000 7f653f7fe640 1 --2- 172.21.15.44:0/690205180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f654015e470 0x7f65400b32c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f6540163070 tx=0x7f653002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b3a6b5363e9db server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:13.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.515+0000 7f653e7fc640 1 -- 172.21.15.44:0/690205180 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653002e070 con 0x7f654015e470 2024-04-02T13:23:13.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.515+0000 7f653e7fc640 1 -- 172.21.15.44:0/690205180 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6530037070 con 0x7f654015e470 2024-04-02T13:23:13.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.515+0000 7f653e7fc640 1 -- 172.21.15.44:0/690205180 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6530031a00 con 0x7f654015e470 2024-04-02T13:23:13.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.516+0000 7f6545f8d640 1 -- 172.21.15.44:0/690205180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f654015e470 msgr2=0x7f65400b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:13.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.516+0000 7f6545f8d640 1 --2- 172.21.15.44:0/690205180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f654015e470 0x7f65400b32c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f6540163070 tx=0x7f653002b880 comp rx=0 tx=0).stop 2024-04-02T13:23:13.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.516+0000 7f6545f8d640 1 -- 172.21.15.44:0/690205180 shutdown_connections 2024-04-02T13:23:13.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.516+0000 7f6545f8d640 1 --2- 172.21.15.44:0/690205180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65400ce790 0x7f65400d20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.516+0000 7f6545f8d640 1 --2- 172.21.15.44:0/690205180 >> v2:172.21.15.69:3300/0 conn(0x7f65400b1540 0x7f65400d1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.516+0000 7f6545f8d640 1 --2- 172.21.15.44:0/690205180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f654015e470 0x7f65400b32c0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.516+0000 7f6545f8d640 1 -- 172.21.15.44:0/690205180 >> 172.21.15.44:0/690205180 conn(0x7f654009aa20 msgr2=0x7f654009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:13.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.516+0000 7f6545f8d640 1 -- 172.21.15.44:0/690205180 shutdown_connections 2024-04-02T13:23:13.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.516+0000 7f6545f8d640 1 -- 172.21.15.44:0/690205180 wait complete. 2024-04-02T13:23:13.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.517+0000 7f6545f8d640 1 Processor -- start 2024-04-02T13:23:13.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.517+0000 7f6545f8d640 1 -- start start 2024-04-02T13:23:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f6545f8d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65400b1540 0x7f65400c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f6545f8d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65400ce790 0x7f65400cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f6545f8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f654015e470 0x7f654014eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f6545f8d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f654006cb60 con 0x7f654015e470 2024-04-02T13:23:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f6545f8d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f654015e850 con 0x7f65400b1540 2024-04-02T13:23:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f6545f8d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6540161fd0 con 0x7f65400ce790 2024-04-02T13:23:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f653ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f654015e470 0x7f654014eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f653ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f654015e470 0x7f654014eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60474/0 (socket says 172.21.15.44:60474) 2024-04-02T13:23:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f653ffff640 1 -- 172.21.15.44:0/1891823510 learned_addr learned my addr 172.21.15.44:0/1891823510 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f653f7fe640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65400b1540 0x7f65400c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f653effd640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65400ce790 0x7f65400cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f653ffff640 1 -- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65400ce790 msgr2=0x7f65400cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:13.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f653ffff640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65400ce790 0x7f65400cb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f653ffff640 1 -- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65400b1540 msgr2=0x7f65400c8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:13.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f653ffff640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65400b1540 0x7f65400c8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.518+0000 7f653ffff640 1 -- 172.21.15.44:0/1891823510 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65400c96c0 con 0x7f654015e470 2024-04-02T13:23:13.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.519+0000 7f653ffff640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f654015e470 0x7f654014eec0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6538009fa0 tx=0x7f653800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:13.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.519+0000 7f6527fff640 1 -- 172.21.15.44:0/1891823510 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653801a070 con 0x7f654015e470 2024-04-02T13:23:13.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.519+0000 7f6527fff640 1 -- 172.21.15.44:0/1891823510 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6538003bb0 con 0x7f654015e470 2024-04-02T13:23:13.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.519+0000 7f6545f8d640 1 -- 172.21.15.44:0/1891823510 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65400c99a0 con 0x7f654015e470 2024-04-02T13:23:13.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.519+0000 7f6527fff640 1 -- 172.21.15.44:0/1891823510 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538004030 con 0x7f654015e470 2024-04-02T13:23:13.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.519+0000 7f6545f8d640 1 -- 172.21.15.44:0/1891823510 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65400c9f60 con 0x7f654015e470 2024-04-02T13:23:13.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.520+0000 7f6545f8d640 1 -- 172.21.15.44:0/1891823510 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65400684f0 con 0x7f654015e470 2024-04-02T13:23:13.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.520+0000 7f6527fff640 1 -- 172.21.15.44:0/1891823510 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f653801d070 con 0x7f654015e470 2024-04-02T13:23:13.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.521+0000 7f6527fff640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f652c06c920 0x7f652c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:13.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.521+0000 7f6527fff640 1 -- 172.21.15.44:0/1891823510 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5335+0+0 (secure 0 0 0) 0x7f653808bc90 con 0x7f654015e470 2024-04-02T13:23:13.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.521+0000 7f653f7fe640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f652c06c920 0x7f652c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:13.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.522+0000 7f653f7fe640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f652c06c920 0x7f652c06ede0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6530036000 tx=0x7f653003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:13.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.525+0000 7f6527fff640 1 -- 172.21.15.44:0/1891823510 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6538010070 con 0x7f654015e470 2024-04-02T13:23:13.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.710+0000 7f6545f8d640 1 -- 172.21.15.44:0/1891823510 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f65400d2d10 con 0x7f654015e470 2024-04-02T13:23:13.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.713+0000 7f6527fff640 1 -- 172.21.15.44:0/1891823510 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) ==== 122+0+0 (secure 0 0 0) 0x7f6538059be0 con 0x7f654015e470 2024-04-02T13:23:13.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.715+0000 7f6545f8d640 1 -- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f652c06c920 msgr2=0x7f652c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:13.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.715+0000 7f6545f8d640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f652c06c920 0x7f652c06ede0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6530036000 tx=0x7f653003d040 comp rx=0 tx=0).stop 2024-04-02T13:23:13.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.715+0000 7f6545f8d640 1 -- 172.21.15.44:0/1891823510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f654015e470 msgr2=0x7f654014eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:13.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.715+0000 7f6545f8d640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f654015e470 0x7f654014eec0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6538009fa0 tx=0x7f653800d560 comp rx=0 tx=0).stop 2024-04-02T13:23:13.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.716+0000 7f6545f8d640 1 -- 172.21.15.44:0/1891823510 shutdown_connections 2024-04-02T13:23:13.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.716+0000 7f6545f8d640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f652c06c920 0x7f652c06ede0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.716+0000 7f6545f8d640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f654015e470 0x7f654014eec0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.716+0000 7f6545f8d640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65400ce790 0x7f65400cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.716+0000 7f6545f8d640 1 --2- 172.21.15.44:0/1891823510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65400b1540 0x7f65400c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:13.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.716+0000 7f6545f8d640 1 -- 172.21.15.44:0/1891823510 >> 172.21.15.44:0/1891823510 conn(0x7f654009aa20 msgr2=0x7f65400afac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:13.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.716+0000 7f6545f8d640 1 -- 172.21.15.44:0/1891823510 shutdown_connections 2024-04-02T13:23:13.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:13.716+0000 7f6545f8d640 1 -- 172.21.15.44:0/1891823510 wait complete. 2024-04-02T13:23:13.728 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:23:13.728 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:23:13.786 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:23:13.836 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:23:13.836 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:23:13.836 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:23:13.851 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:23:13.851 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:23:13.851 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:23:13.869 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:23:13.869 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:23:13.869 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:23:13.907 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:23:13.907 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:23:13.907 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:23:13.926 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:23:13.926 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T13:23:13.926 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:23:14.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.046+0000 7f3a5b7af640 1 Processor -- start 2024-04-02T13:23:14.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.047+0000 7f3a5b7af640 1 -- start start 2024-04-02T13:23:14.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.047+0000 7f3a5b7af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a540d28f0 0x7f3a540d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:14.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.047+0000 7f3a5b7af640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3a540b5200 0x7f3a540b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:14.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.047+0000 7f3a5b7af640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a540d4d70 con 0x7f3a540d1260 2024-04-02T13:23:14.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.047+0000 7f3a5b7af640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f3a540d5050 con 0x7f3a540b5200 2024-04-02T13:23:14.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.048+0000 7f3a5b7af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3a540d23a0 con 0x7f3a540d28f0 2024-04-02T13:23:14.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.048+0000 7f3a58d4a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3a540b5200 0x7f3a540b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:14.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.048+0000 7f3a58d4a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3a540b5200 0x7f3a540b40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60484/0 (socket says 172.21.15.44:60484) 2024-04-02T13:23:14.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.048+0000 7f3a58d4a640 1 -- 172.21.15.44:0/3485252910 learned_addr learned my addr 172.21.15.44:0/3485252910 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:14.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.048+0000 7f3a5954b640 1 --2- 172.21.15.44:0/3485252910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a540d28f0 0x7f3a540d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:14.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.048+0000 7f3a58d4a640 1 -- 172.21.15.44:0/3485252910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a540d28f0 msgr2=0x7f3a540d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:14.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.048+0000 7f3a58d4a640 1 --2- 172.21.15.44:0/3485252910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a540d28f0 0x7f3a540d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:14.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.048+0000 7f3a58d4a640 1 -- 172.21.15.44:0/3485252910 >> v1:172.21.15.69:6789/0 conn(0x7f3a540d1260 legacy=0x7f3a540ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:14.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.048+0000 7f3a58d4a640 1 -- 172.21.15.44:0/3485252910 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a540d2030 con 0x7f3a540b5200 2024-04-02T13:23:14.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.048+0000 7f3a5954b640 1 --2- 172.21.15.44:0/3485252910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a540d28f0 0x7f3a540d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:14.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.049+0000 7f3a58d4a640 1 --2- 172.21.15.44:0/3485252910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a540b5200 0x7f3a540b40a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009870 tx=0x7f3a4402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c209aa630bcd49 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:14.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.049+0000 7f3a4bfff640 1 -- 172.21.15.44:0/3485252910 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a440317c0 con 0x7f3a540b5200 2024-04-02T13:23:14.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.049+0000 7f3a4bfff640 1 -- 172.21.15.44:0/3485252910 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a44037070 con 0x7f3a540b5200 2024-04-02T13:23:14.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.049+0000 7f3a4bfff640 1 -- 172.21.15.44:0/3485252910 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44031b80 con 0x7f3a540b5200 2024-04-02T13:23:14.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.049+0000 7f3a5b7af640 1 -- 172.21.15.44:0/3485252910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a540b5200 msgr2=0x7f3a540b40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:14.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.049+0000 7f3a5b7af640 1 --2- 172.21.15.44:0/3485252910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a540b5200 0x7f3a540b40a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009870 tx=0x7f3a4402b860 comp rx=0 tx=0).stop 2024-04-02T13:23:14.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.049+0000 7f3a5b7af640 1 -- 172.21.15.44:0/3485252910 shutdown_connections 2024-04-02T13:23:14.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.049+0000 7f3a5b7af640 1 --2- 172.21.15.44:0/3485252910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a540b5200 0x7f3a540b40a0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:14.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.049+0000 7f3a5b7af640 1 --2- 172.21.15.44:0/3485252910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a540d28f0 0x7f3a540d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:14.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.049+0000 7f3a5b7af640 1 -- 172.21.15.44:0/3485252910 >> 172.21.15.44:0/3485252910 conn(0x7f3a540d4120 msgr2=0x7f3a540d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:14.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.050+0000 7f3a5b7af640 1 -- 172.21.15.44:0/3485252910 shutdown_connections 2024-04-02T13:23:14.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.050+0000 7f3a5b7af640 1 -- 172.21.15.44:0/3485252910 wait complete. 2024-04-02T13:23:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.051+0000 7f3a5b7af640 1 Processor -- start 2024-04-02T13:23:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.051+0000 7f3a5b7af640 1 -- start start 2024-04-02T13:23:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.051+0000 7f3a5b7af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a540b5200 0x7f3a5415d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.051+0000 7f3a5b7af640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a540d1260 0x7f3a541f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.051+0000 7f3a5b7af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a540d28f0 0x7f3a5415b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.051+0000 7f3a5b7af640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3a540b8130 con 0x7f3a540d28f0 2024-04-02T13:23:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.051+0000 7f3a5b7af640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3a540b5020 con 0x7f3a540d1260 2024-04-02T13:23:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.051+0000 7f3a5b7af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3a540b82b0 con 0x7f3a540b5200 2024-04-02T13:23:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a59d4c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a540d28f0 0x7f3a5415b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a59d4c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a540d28f0 0x7f3a5415b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60494/0 (socket says 172.21.15.44:60494) 2024-04-02T13:23:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a59d4c640 1 -- 172.21.15.44:0/1031901337 learned_addr learned my addr 172.21.15.44:0/1031901337 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a5954b640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a540b5200 0x7f3a5415d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a58d4a640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a540d1260 0x7f3a541f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a59d4c640 1 -- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a540b5200 msgr2=0x7f3a5415d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a59d4c640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a540b5200 0x7f3a5415d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a59d4c640 1 -- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a540d1260 msgr2=0x7f3a541f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a59d4c640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a540d1260 0x7f3a541f3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a59d4c640 1 -- 172.21.15.44:0/1031901337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a541f4f00 con 0x7f3a540d28f0 2024-04-02T13:23:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a59d4c640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a540d28f0 0x7f3a5415b5c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c00c6b0 tx=0x7f3a4c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a49ffb640 1 -- 172.21.15.44:0/1031901337 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c003b70 con 0x7f3a540d28f0 2024-04-02T13:23:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a49ffb640 1 -- 172.21.15.44:0/1031901337 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a4c003d30 con 0x7f3a540d28f0 2024-04-02T13:23:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a5b7af640 1 -- 172.21.15.44:0/1031901337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a541f4660 con 0x7f3a540d28f0 2024-04-02T13:23:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a49ffb640 1 -- 172.21.15.44:0/1031901337 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c005310 con 0x7f3a540d28f0 2024-04-02T13:23:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.052+0000 7f3a5b7af640 1 -- 172.21.15.44:0/1031901337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a541f4c20 con 0x7f3a540d28f0 2024-04-02T13:23:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.053+0000 7f3a5b7af640 1 -- 172.21.15.44:0/1031901337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a540684f0 con 0x7f3a540d28f0 2024-04-02T13:23:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.054+0000 7f3a49ffb640 1 -- 172.21.15.44:0/1031901337 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3a4c0054b0 con 0x7f3a540d28f0 2024-04-02T13:23:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.054+0000 7f3a49ffb640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3a3806c8f0 0x7f3a3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.054+0000 7f3a49ffb640 1 -- 172.21.15.44:0/1031901337 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5335+0+0 (secure 0 0 0) 0x7f3a4c08b840 con 0x7f3a540d28f0 2024-04-02T13:23:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.054+0000 7f3a5954b640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3a3806c8f0 0x7f3a3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.055+0000 7f3a5954b640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3a3806c8f0 0x7f3a3806edb0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3a540a2000 tx=0x7f3a50008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:14.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.059+0000 7f3a49ffb640 1 -- 172.21.15.44:0/1031901337 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3a4c059de0 con 0x7f3a540d28f0 2024-04-02T13:23:14.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.280+0000 7f3a5b7af640 1 -- 172.21.15.44:0/1031901337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a540b8880 con 0x7f3a540d28f0 2024-04-02T13:23:14.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.280+0000 7f3a49ffb640 1 -- 172.21.15.44:0/1031901337 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+3869 (secure 0 0 0) 0x7f3a4c0596e0 con 0x7f3a540d28f0 2024-04-02T13:23:14.282 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:14.282 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/4131284806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":4131284806},{"type":"v1","addr":"172.21.15.44:6833","nonce":4131284806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/4262427145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":4262427145},{"type":"v1","addr":"172.21.15.44:6835","nonce":4262427145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4112565894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4112565894},{"type":"v1","addr":"172.21.15.69:6836","nonce":4112565894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5845,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1749143470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1749143470},{"type":"v1","addr":"172.21.15.69:6837","nonce":1749143470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[]} 2024-04-02T13:23:14.282 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 27 2024-04-02T13:23:14.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.282+0000 7f3a5b7af640 1 -- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3a3806c8f0 msgr2=0x7f3a3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:14.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.282+0000 7f3a5b7af640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3a3806c8f0 0x7f3a3806edb0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3a540a2000 tx=0x7f3a50008040 comp rx=0 tx=0).stop 2024-04-02T13:23:14.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.282+0000 7f3a5b7af640 1 -- 172.21.15.44:0/1031901337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a540d28f0 msgr2=0x7f3a5415b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:14.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.282+0000 7f3a5b7af640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a540d28f0 0x7f3a5415b5c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c00c6b0 tx=0x7f3a4c00cb70 comp rx=0 tx=0).stop 2024-04-02T13:23:14.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.282+0000 7f3a5b7af640 1 -- 172.21.15.44:0/1031901337 shutdown_connections 2024-04-02T13:23:14.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.282+0000 7f3a5b7af640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3a3806c8f0 0x7f3a3806edb0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:14.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.282+0000 7f3a5b7af640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a540d28f0 0x7f3a5415b5c0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:14.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.282+0000 7f3a5b7af640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a540d1260 0x7f3a541f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:14.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.282+0000 7f3a5b7af640 1 --2- 172.21.15.44:0/1031901337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a540b5200 0x7f3a5415d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:14.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.282+0000 7f3a5b7af640 1 -- 172.21.15.44:0/1031901337 >> 172.21.15.44:0/1031901337 conn(0x7f3a540d4120 msgr2=0x7f3a540b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:14.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.282+0000 7f3a5b7af640 1 -- 172.21.15.44:0/1031901337 shutdown_connections 2024-04-02T13:23:14.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:14.282+0000 7f3a5b7af640 1 -- 172.21.15.44:0/1031901337 wait complete. 2024-04-02T13:23:14.295 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T13:23:14.295 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T13:23:14.295 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T13:23:14.296 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:23:14.296 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T13:23:14.296 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T13:23:14.296 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T13:23:14.296 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:23:14.296 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T13:23:14.296 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T13:23:14.297 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T13:23:14.297 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:23:14.297 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T13:23:14.297 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T13:23:14.297 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T13:23:14.297 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:23:14.306 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T13:23:14.306 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T13:23:14.308 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T13:23:14.308 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T13:23:14.309 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T13:23:14.309 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T13:23:14.349 INFO:tasks.ceph.mds.c:Started 2024-04-02T13:23:14.349 DEBUG:teuthology.parallel:result is None 2024-04-02T13:23:14.351 INFO:tasks.ceph.mds.b:Started 2024-04-02T13:23:14.352 DEBUG:teuthology.parallel:result is None 2024-04-02T13:23:14.352 INFO:tasks.ceph.mds.d:Started 2024-04-02T13:23:14.352 DEBUG:teuthology.parallel:result is None 2024-04-02T13:23:14.423 INFO:tasks.ceph.mds.c.smithi044.stdout:starting mds.c at 2024-04-02T13:23:14.436 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-04-02T13:23:14.436 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-04-02T13:23:20.306 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T13:23:20.306 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T13:23:20.309 INFO:tasks.ceph.mds.a:Started 2024-04-02T13:23:20.309 DEBUG:teuthology.parallel:result is None 2024-04-02T13:23:20.310 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T13:23:20.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-04-02T13:23:20.398 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:23:20.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.424+0000 7fc9c7e55640 1 Processor -- start 2024-04-02T13:23:20.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.424+0000 7fc9c7e55640 1 -- start start 2024-04-02T13:23:20.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.425+0000 7fc9c7e55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c015c250 0x7fc9c00aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:20.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.425+0000 7fc9c7e55640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc9c00b3510 0x7fc9c00d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:20.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.425+0000 7fc9c7e55640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc9c00d08d0 0x7fc9c00b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:20.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.425+0000 7fc9c7e55640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc9c015c630 con 0x7fc9c00d08d0 2024-04-02T13:23:20.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.425+0000 7fc9c7e55640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc9c00af420 con 0x7fc9c00b3510 2024-04-02T13:23:20.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.425+0000 7fc9c7e55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc9c00af700 con 0x7fc9c015c250 2024-04-02T13:23:20.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.425+0000 7fc9c63f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc9c00d08d0 0x7fc9c00b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:20.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.425+0000 7fc9c63f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc9c00d08d0 0x7fc9c00b1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53266/0 (socket says 172.21.15.44:53266) 2024-04-02T13:23:20.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.425+0000 7fc9c53f0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc9c00b3510 0x7fc9c00d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:20.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.425+0000 7fc9c63f2640 1 -- 172.21.15.44:0/1880965142 learned_addr learned my addr 172.21.15.44:0/1880965142 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:20.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.425+0000 7fc9c5bf1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c015c250 0x7fc9c00aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:20.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.426+0000 7fc9c63f2640 1 -- 172.21.15.44:0/1880965142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c015c250 msgr2=0x7fc9c00aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:20.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.426+0000 7fc9c63f2640 1 --2- 172.21.15.44:0/1880965142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c015c250 0x7fc9c00aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:20.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.426+0000 7fc9c63f2640 1 -- 172.21.15.44:0/1880965142 >> v2:172.21.15.69:3300/0 conn(0x7fc9c00b3510 msgr2=0x7fc9c00d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:20.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.426+0000 7fc9c63f2640 1 --2- 172.21.15.44:0/1880965142 >> v2:172.21.15.69:3300/0 conn(0x7fc9c00b3510 0x7fc9c00d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:20.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.426+0000 7fc9c63f2640 1 -- 172.21.15.44:0/1880965142 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c00af9e0 con 0x7fc9c00d08d0 2024-04-02T13:23:20.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.426+0000 7fc9c63f2640 1 --2- 172.21.15.44:0/1880965142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9c00d08d0 0x7fc9c00b1e40 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00b4ad0 tx=0x7fc9b802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a83a2ac464c0aaf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:20.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.427+0000 7fc9c4bef640 1 -- 172.21.15.44:0/1880965142 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b802c9e0 con 0x7fc9c00d08d0 2024-04-02T13:23:20.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.427+0000 7fc9c4bef640 1 -- 172.21.15.44:0/1880965142 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9b802cb80 con 0x7fc9c00d08d0 2024-04-02T13:23:20.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.427+0000 7fc9c4bef640 1 -- 172.21.15.44:0/1880965142 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b80343f0 con 0x7fc9c00d08d0 2024-04-02T13:23:20.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.427+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1880965142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9c00d08d0 msgr2=0x7fc9c00b1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:20.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.427+0000 7fc9c7e55640 1 --2- 172.21.15.44:0/1880965142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9c00d08d0 0x7fc9c00b1e40 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00b4ad0 tx=0x7fc9b802be40 comp rx=0 tx=0).stop 2024-04-02T13:23:20.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.427+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1880965142 shutdown_connections 2024-04-02T13:23:20.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.427+0000 7fc9c7e55640 1 --2- 172.21.15.44:0/1880965142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9c00d08d0 0x7fc9c00b1e40 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:20.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.427+0000 7fc9c7e55640 1 --2- 172.21.15.44:0/1880965142 >> v2:172.21.15.69:3300/0 conn(0x7fc9c00b3510 0x7fc9c00d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.427+0000 7fc9c7e55640 1 --2- 172.21.15.44:0/1880965142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c015c250 0x7fc9c00aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.427+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1880965142 >> 172.21.15.44:0/1880965142 conn(0x7fc9c0073850 msgr2=0x7fc9c009aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.428+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1880965142 shutdown_connections 2024-04-02T13:23:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.428+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1880965142 wait complete. 2024-04-02T13:23:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.429+0000 7fc9c7e55640 1 Processor -- start 2024-04-02T13:23:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.429+0000 7fc9c7e55640 1 -- start start 2024-04-02T13:23:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.429+0000 7fc9c7e55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9c00b3510 0x7fc9c006cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.429+0000 7fc9c7e55640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9c00d08d0 0x7fc9c015d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.429+0000 7fc9c7e55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c015c250 0x7fc9c015dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c7e55640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c00b2380 con 0x7fc9c00d08d0 2024-04-02T13:23:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c7e55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c00af130 con 0x7fc9c00b3510 2024-04-02T13:23:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c7e55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc9c00b2500 con 0x7fc9c015c250 2024-04-02T13:23:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c53f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9c00d08d0 0x7fc9c015d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c53f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9c00d08d0 0x7fc9c015d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53268/0 (socket says 172.21.15.44:53268) 2024-04-02T13:23:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c53f0640 1 -- 172.21.15.44:0/1390121532 learned_addr learned my addr 172.21.15.44:0/1390121532 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c5bf1640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9c00b3510 0x7fc9c006cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c63f2640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c015c250 0x7fc9c015dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c53f0640 1 -- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c015c250 msgr2=0x7fc9c015dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c53f0640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c015c250 0x7fc9c015dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c53f0640 1 -- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9c00b3510 msgr2=0x7fc9c006cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c53f0640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9c00b3510 0x7fc9c006cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c53f0640 1 -- 172.21.15.44:0/1390121532 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c006d4c0 con 0x7fc9c00d08d0 2024-04-02T13:23:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9c53f0640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9c00d08d0 0x7fc9c015d960 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000e890 tx=0x7fc9b000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.430+0000 7fc9b67fc640 1 -- 172.21.15.44:0/1390121532 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0017070 con 0x7fc9c00d08d0 2024-04-02T13:23:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.431+0000 7fc9b67fc640 1 -- 172.21.15.44:0/1390121532 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9b000c940 con 0x7fc9c00d08d0 2024-04-02T13:23:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.431+0000 7fc9b67fc640 1 -- 172.21.15.44:0/1390121532 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b000cd10 con 0x7fc9c00d08d0 2024-04-02T13:23:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.431+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1390121532 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9c006d780 con 0x7fc9c00d08d0 2024-04-02T13:23:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.431+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1390121532 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9c006dd40 con 0x7fc9c00d08d0 2024-04-02T13:23:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.432+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1390121532 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9c00711c0 con 0x7fc9c00d08d0 2024-04-02T13:23:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.432+0000 7fc9b67fc640 1 -- 172.21.15.44:0/1390121532 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc9b001a070 con 0x7fc9c00d08d0 2024-04-02T13:23:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.432+0000 7fc9b67fc640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9a406c920 0x7fc9a406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.432+0000 7fc9b67fc640 1 -- 172.21.15.44:0/1390121532 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5335+0+0 (secure 0 0 0) 0x7fc9b008b8b0 con 0x7fc9c00d08d0 2024-04-02T13:23:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.433+0000 7fc9c5bf1640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9a406c920 0x7fc9a406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:20.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.433+0000 7fc9c5bf1640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9a406c920 0x7fc9a406ede0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009aa0 tx=0x7fc9bc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:20.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.437+0000 7fc9b67fc640 1 -- 172.21.15.44:0/1390121532 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc9c00711c0 con 0x7fc9c00d08d0 2024-04-02T13:23:20.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:20.633+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1390121532 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fc9c006df40 con 0x7fc9c00d08d0 2024-04-02T13:23:21.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.426+0000 7fc9b67fc640 1 -- 172.21.15.44:0/1390121532 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v61) ==== 97+0+0 (secure 0 0 0) 0x7fc9b0059800 con 0x7fc9c00d08d0 2024-04-02T13:23:21.428 INFO:teuthology.orchestra.run.smithi044.stderr: removed all blocklist entries 2024-04-02T13:23:21.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.429+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9a406c920 msgr2=0x7fc9a406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:21.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.429+0000 7fc9c7e55640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9a406c920 0x7fc9a406ede0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009aa0 tx=0x7fc9bc009190 comp rx=0 tx=0).stop 2024-04-02T13:23:21.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.429+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1390121532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9c00d08d0 msgr2=0x7fc9c015d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:21.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.429+0000 7fc9c7e55640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9c00d08d0 0x7fc9c015d960 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000e890 tx=0x7fc9b000ed50 comp rx=0 tx=0).stop 2024-04-02T13:23:21.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.429+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1390121532 shutdown_connections 2024-04-02T13:23:21.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.429+0000 7fc9c7e55640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9a406c920 0x7fc9a406ede0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.429+0000 7fc9c7e55640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c015c250 0x7fc9c015dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.429+0000 7fc9c7e55640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9c00d08d0 0x7fc9c015d960 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.429+0000 7fc9c7e55640 1 --2- 172.21.15.44:0/1390121532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9c00b3510 0x7fc9c006cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.429+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1390121532 >> 172.21.15.44:0/1390121532 conn(0x7fc9c0073850 msgr2=0x7fc9c0070ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:21.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.429+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1390121532 shutdown_connections 2024-04-02T13:23:21.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.429+0000 7fc9c7e55640 1 -- 172.21.15.44:0/1390121532 wait complete. 2024-04-02T13:23:21.444 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T13:23:21.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.549+0000 7f640623a640 1 Processor -- start 2024-04-02T13:23:21.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.549+0000 7f640623a640 1 -- start start 2024-04-02T13:23:21.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f640623a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f64001617e0 0x7f64000ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:21.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f640623a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64000cffd0 0x7f64001629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:21.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f640623a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64000cef30 con 0x7f64000ccc90 2024-04-02T13:23:21.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f640623a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f6400162f00 con 0x7f64001617e0 2024-04-02T13:23:21.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f640623a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f64000aea90 con 0x7f64000cffd0 2024-04-02T13:23:21.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f63ff7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f64001617e0 0x7f64000ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:21.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f63fffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f64000ccc90 0x7f64000cd0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:43998/0 (socket says 172.21.15.44:43998) 2024-04-02T13:23:21.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f63ff7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f64001617e0 0x7f64000ce8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53282/0 (socket says 172.21.15.44:53282) 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f63fffff640 1 -- 172.21.15.44:0/203564489 learned_addr learned my addr 172.21.15.44:0/203564489 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f63feffd640 1 --2- 172.21.15.44:0/203564489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64000cffd0 0x7f64001629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f63ff7fe640 1 -- 172.21.15.44:0/203564489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64000cffd0 msgr2=0x7f64001629c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f63ff7fe640 1 --2- 172.21.15.44:0/203564489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64000cffd0 0x7f64001629c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f63ff7fe640 1 -- 172.21.15.44:0/203564489 >> v1:172.21.15.44:6789/0 conn(0x7f64000ccc90 legacy=0x7f64000cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f63fe7fc640 1 -- 172.21.15.44:0/203564489 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3843966522 0 0) 0x7f64000cef30 con 0x7f64000ccc90 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f63ff7fe640 1 -- 172.21.15.44:0/203564489 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64000cf210 con 0x7f64001617e0 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.550+0000 7f63feffd640 1 --2- 172.21.15.44:0/203564489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64000cffd0 0x7f64001629c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.551+0000 7f63ff7fe640 1 --2- 172.21.15.44:0/203564489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64001617e0 0x7f64000ce8c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f63f00097a0 tx=0x7f63f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87d8b7a72acbaee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.551+0000 7f63fe7fc640 1 -- 172.21.15.44:0/203564489 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f00315a0 con 0x7f64001617e0 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.551+0000 7f63fe7fc640 1 -- 172.21.15.44:0/203564489 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63f0037070 con 0x7f64001617e0 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.551+0000 7f63fe7fc640 1 -- 172.21.15.44:0/203564489 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f0031960 con 0x7f64001617e0 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.552+0000 7f640623a640 1 -- 172.21.15.44:0/203564489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64001617e0 msgr2=0x7f64000ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:21.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.552+0000 7f640623a640 1 --2- 172.21.15.44:0/203564489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64001617e0 0x7f64000ce8c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f63f00097a0 tx=0x7f63f002b840 comp rx=0 tx=0).stop 2024-04-02T13:23:21.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.552+0000 7f640623a640 1 -- 172.21.15.44:0/203564489 shutdown_connections 2024-04-02T13:23:21.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.552+0000 7f640623a640 1 --2- 172.21.15.44:0/203564489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64000cffd0 0x7f64001629c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.552+0000 7f640623a640 1 --2- 172.21.15.44:0/203564489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64001617e0 0x7f64000ce8c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.552+0000 7f640623a640 1 -- 172.21.15.44:0/203564489 >> 172.21.15.44:0/203564489 conn(0x7f64000b20e0 msgr2=0x7f64000b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:21.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.552+0000 7f640623a640 1 -- 172.21.15.44:0/203564489 shutdown_connections 2024-04-02T13:23:21.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.552+0000 7f640623a640 1 -- 172.21.15.44:0/203564489 wait complete. 2024-04-02T13:23:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.553+0000 7f640623a640 1 Processor -- start 2024-04-02T13:23:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.553+0000 7f640623a640 1 -- start start 2024-04-02T13:23:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f640623a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64000ccc90 0x7f64001f4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f640623a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64000cffd0 0x7f64000d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f640623a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64001617e0 0x7f6400151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f640623a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f64000cfab0 con 0x7f64000cffd0 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f640623a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f64000cf7d0 con 0x7f64001617e0 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f640623a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6400161bc0 con 0x7f64000ccc90 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f63feffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64000cffd0 0x7f64000d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f63feffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64000cffd0 0x7f64000d06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53292/0 (socket says 172.21.15.44:53292) 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f63feffd640 1 -- 172.21.15.44:0/2951105181 learned_addr learned my addr 172.21.15.44:0/2951105181 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f63ff7fe640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64000ccc90 0x7f64001f4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f63fffff640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64001617e0 0x7f6400151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f63feffd640 1 -- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64000ccc90 msgr2=0x7f64001f4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f63feffd640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64000ccc90 0x7f64001f4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f63feffd640 1 -- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64001617e0 msgr2=0x7f6400151500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f63feffd640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64001617e0 0x7f6400151500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f63feffd640 1 -- 172.21.15.44:0/2951105181 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64000c75c0 con 0x7f64000cffd0 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f63ff7fe640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64000ccc90 0x7f64001f4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:21.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.554+0000 7f63feffd640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64000cffd0 0x7f64000d06f0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f63f400c7f0 tx=0x7f63f400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:21.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.555+0000 7f63e7fff640 1 -- 172.21.15.44:0/2951105181 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f4015070 con 0x7f64000cffd0 2024-04-02T13:23:21.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.555+0000 7f63e7fff640 1 -- 172.21.15.44:0/2951105181 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63f4007a30 con 0x7f64000cffd0 2024-04-02T13:23:21.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.555+0000 7f640623a640 1 -- 172.21.15.44:0/2951105181 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64001f78e0 con 0x7f64000cffd0 2024-04-02T13:23:21.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.555+0000 7f63e7fff640 1 -- 172.21.15.44:0/2951105181 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f4007e00 con 0x7f64000cffd0 2024-04-02T13:23:21.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.555+0000 7f640623a640 1 -- 172.21.15.44:0/2951105181 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64001f7ea0 con 0x7f64000cffd0 2024-04-02T13:23:21.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.556+0000 7f640623a640 1 -- 172.21.15.44:0/2951105181 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64000684f0 con 0x7f64000cffd0 2024-04-02T13:23:21.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.556+0000 7f63e7fff640 1 -- 172.21.15.44:0/2951105181 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f63f4018070 con 0x7f64000cffd0 2024-04-02T13:23:21.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.556+0000 7f63e7fff640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f63ec06c8f0 0x7f63ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:21.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.557+0000 7f63e7fff640 1 -- 172.21.15.44:0/2951105181 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5249+0+0 (secure 0 0 0) 0x7f63f408b830 con 0x7f64000cffd0 2024-04-02T13:23:21.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.557+0000 7f63ff7fe640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f63ec06c8f0 0x7f63ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:21.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.557+0000 7f63ff7fe640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f63ec06c8f0 0x7f63ec06edb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f63f0036000 tx=0x7f63f0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:21.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.561+0000 7f63e7fff640 1 -- 172.21.15.44:0/2951105181 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f63f4059eb0 con 0x7f64000cffd0 2024-04-02T13:23:21.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.817+0000 7f640623a640 1 -- 172.21.15.44:0/2951105181 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f64000ce190 con 0x7f64000cffd0 2024-04-02T13:23:21.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.818+0000 7f63e7fff640 1 -- 172.21.15.44:0/2951105181 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) ==== 80+0+5304 (secure 0 0 0) 0x7f63f40597b0 con 0x7f64000cffd0 2024-04-02T13:23:21.819 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "auth_dump": [ 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.a", 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmVhZrMhAAklqx5uKCFA5kiqYwyLBKpg==", 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.b", 2024-04-02T13:23:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmalQ+AxAA2SqH9CX/66OIXR7SDID97w==", 2024-04-02T13:23:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:23:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:23:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:23:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.c", 2024-04-02T13:23:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxm8BLKORAAjMIrmdQ/bXbPk2sAnYJWTw==", 2024-04-02T13:23:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:23:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:23:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:23:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.d", 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmNoaRChAAEqxWmBjBWG79Tkj6dU8FQg==", 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.0", 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDfBQxm4GP9KxAA/AmOunAfVaczAWRGVXvFVg==", 2024-04-02T13:23:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.1", 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDiBQxmDL/nKRAADF2C30+IGAXL9XwDQDKCUA==", 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.2", 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDlBQxm1gOCKBAAuYxN9jNHuRYEqc8WrsYErQ==", 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.3", 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDoBQxmLLFQJxAAFMCzphGU7TOt4nC0Nt+y+g==", 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:23:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.4", 2024-04-02T13:23:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDuBQxmtk8QLhAAR4gm0irR6C3D2SkhFjvDKQ==", 2024-04-02T13:23:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:23:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:23:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.5", 2024-04-02T13:23:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDxBQxm6Ee/LBAAX5h+9ML70ztw2QH/50OzOg==", 2024-04-02T13:23:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:23:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:23:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.6", 2024-04-02T13:23:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD0BQxm35hwKxAAvsgMmiXdLo+c7wlvb9RtVQ==", 2024-04-02T13:23:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:23:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:23:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.7", 2024-04-02T13:23:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD3BQxmYLBOKhAA1vuSOBDcFf2zT78eoTHtHQ==", 2024-04-02T13:23:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.0", 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A==", 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow r", 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.1", 2024-04-02T13:23:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxm3sYsGBAAeRCNbP5qiQ8YnRVsDoaaFQ==", 2024-04-02T13:23:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:23:21.833 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:23:21.833 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:23:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:23:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.2", 2024-04-02T13:23:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmKNS5FBAAKvsCNmuG+fI1D/vwu6MC5w==", 2024-04-02T13:23:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:23:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:23:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:23:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:23:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.3", 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmslsSHhAAPsAfALW7RimJDyXdOa2wbQ==", 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.admin", 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmq+VcFRAA9CgeH75Q0sXf27cZdJthJg==", 2024-04-02T13:23:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.x", 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmR5ICLxAAtQeg/gq0OAq6e0mroBZVxQ==", 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.y", 2024-04-02T13:23:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmxoquKhAAE5fMnJIuCy5hGJ1aEAx8Vg==", 2024-04-02T13:23:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:23:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:23:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:23:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:23:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-02T13:23:21.837 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:23:21.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.820+0000 7f640623a640 1 -- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f63ec06c8f0 msgr2=0x7f63ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:21.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.820+0000 7f640623a640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f63ec06c8f0 0x7f63ec06edb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f63f0036000 tx=0x7f63f0008030 comp rx=0 tx=0).stop 2024-04-02T13:23:21.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.820+0000 7f640623a640 1 -- 172.21.15.44:0/2951105181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64000cffd0 msgr2=0x7f64000d06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:21.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.820+0000 7f640623a640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64000cffd0 0x7f64000d06f0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f63f400c7f0 tx=0x7f63f400ccb0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.820+0000 7f640623a640 1 -- 172.21.15.44:0/2951105181 shutdown_connections 2024-04-02T13:23:21.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.820+0000 7f640623a640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f63ec06c8f0 0x7f63ec06edb0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.820+0000 7f640623a640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64001617e0 0x7f6400151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.820+0000 7f640623a640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64000cffd0 0x7f64000d06f0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.820+0000 7f640623a640 1 --2- 172.21.15.44:0/2951105181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64000ccc90 0x7f64001f4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.820+0000 7f640623a640 1 -- 172.21.15.44:0/2951105181 >> 172.21.15.44:0/2951105181 conn(0x7f64000b20e0 msgr2=0x7f64000b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:21.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.820+0000 7f640623a640 1 -- 172.21.15.44:0/2951105181 shutdown_connections 2024-04-02T13:23:21.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.820+0000 7f640623a640 1 -- 172.21.15.44:0/2951105181 wait complete. 2024-04-02T13:23:21.841 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:23:21.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.940+0000 7fe837d50640 1 Processor -- start 2024-04-02T13:23:21.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.940+0000 7fe837d50640 1 -- start start 2024-04-02T13:23:21.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.941+0000 7fe837d50640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe83015aa00 0x7fe8300b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:21.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.941+0000 7fe837d50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8300ce2e0 0x7fe830071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:21.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.941+0000 7fe837d50640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe83015ade0 con 0x7fe830072bf0 2024-04-02T13:23:21.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.941+0000 7fe837d50640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe8300af1b0 con 0x7fe83015aa00 2024-04-02T13:23:21.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.941+0000 7fe837d50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe8300af490 con 0x7fe8300ce2e0 2024-04-02T13:23:21.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.941+0000 7fe835aec640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe83015aa00 0x7fe8300b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:21.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.941+0000 7fe835aec640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe83015aa00 0x7fe8300b2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53296/0 (socket says 172.21.15.44:53296) 2024-04-02T13:23:21.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.941+0000 7fe835aec640 1 -- 172.21.15.44:0/3494085652 learned_addr learned my addr 172.21.15.44:0/3494085652 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:21.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.941+0000 7fe8362ed640 1 --2- 172.21.15.44:0/3494085652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8300ce2e0 0x7fe830071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:21.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.941+0000 7fe835aec640 1 -- 172.21.15.44:0/3494085652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8300ce2e0 msgr2=0x7fe830071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:21.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.941+0000 7fe835aec640 1 --2- 172.21.15.44:0/3494085652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8300ce2e0 0x7fe830071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.941+0000 7fe835aec640 1 -- 172.21.15.44:0/3494085652 >> v1:172.21.15.69:6789/0 conn(0x7fe830072bf0 legacy=0x7fe8300cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:21.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.941+0000 7fe835aec640 1 -- 172.21.15.44:0/3494085652 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8300723f0 con 0x7fe83015aa00 2024-04-02T13:23:21.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.942+0000 7fe835aec640 1 --2- 172.21.15.44:0/3494085652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe83015aa00 0x7fe8300b2d00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00aa10 tx=0x7fe82c02cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634f2b4bb80bd46f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:21.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.942+0000 7fe834aea640 1 -- 172.21.15.44:0/3494085652 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c03a070 con 0x7fe83015aa00 2024-04-02T13:23:21.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.942+0000 7fe834aea640 1 -- 172.21.15.44:0/3494085652 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe82c02db80 con 0x7fe83015aa00 2024-04-02T13:23:21.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.942+0000 7fe834aea640 1 -- 172.21.15.44:0/3494085652 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c00b040 con 0x7fe83015aa00 2024-04-02T13:23:21.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.943+0000 7fe837d50640 1 -- 172.21.15.44:0/3494085652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe83015aa00 msgr2=0x7fe8300b2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:21.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.943+0000 7fe837d50640 1 --2- 172.21.15.44:0/3494085652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe83015aa00 0x7fe8300b2d00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00aa10 tx=0x7fe82c02cdd0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.943+0000 7fe837d50640 1 -- 172.21.15.44:0/3494085652 shutdown_connections 2024-04-02T13:23:21.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.943+0000 7fe837d50640 1 --2- 172.21.15.44:0/3494085652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8300ce2e0 0x7fe830071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.943+0000 7fe837d50640 1 --2- 172.21.15.44:0/3494085652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe83015aa00 0x7fe8300b2d00 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.943+0000 7fe837d50640 1 -- 172.21.15.44:0/3494085652 >> 172.21.15.44:0/3494085652 conn(0x7fe8300cfcd0 msgr2=0x7fe8300d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:21.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.943+0000 7fe837d50640 1 -- 172.21.15.44:0/3494085652 shutdown_connections 2024-04-02T13:23:21.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.943+0000 7fe837d50640 1 -- 172.21.15.44:0/3494085652 wait complete. 2024-04-02T13:23:21.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.944+0000 7fe837d50640 1 Processor -- start 2024-04-02T13:23:21.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.944+0000 7fe837d50640 1 -- start start 2024-04-02T13:23:21.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.944+0000 7fe837d50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe830072bf0 0x7fe8301fb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:21.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe837d50640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8300ce2e0 0x7fe8301fbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:21.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe837d50640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe83015aa00 0x7fe83006dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:21.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe837d50640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe8300731d0 con 0x7fe83015aa00 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe837d50640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe830073050 con 0x7fe8300ce2e0 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe837d50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe8300cc750 con 0x7fe830072bf0 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe8362ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe83015aa00 0x7fe83006dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe8362ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe83015aa00 0x7fe83006dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53306/0 (socket says 172.21.15.44:53306) 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe8362ed640 1 -- 172.21.15.44:0/508126487 learned_addr learned my addr 172.21.15.44:0/508126487 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe835aec640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe830072bf0 0x7fe8301fb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe8352eb640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8300ce2e0 0x7fe8301fbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe8362ed640 1 -- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe830072bf0 msgr2=0x7fe8301fb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe8362ed640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe830072bf0 0x7fe8301fb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe8362ed640 1 -- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8300ce2e0 msgr2=0x7fe8301fbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe8362ed640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8300ce2e0 0x7fe8301fbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe8362ed640 1 -- 172.21.15.44:0/508126487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83006e9c0 con 0x7fe83015aa00 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe835aec640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe830072bf0 0x7fe8301fb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:21.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe8362ed640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe83015aa00 0x7fe83006dc40 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fe828007e70 tx=0x7fe82800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:21.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe8267fc640 1 -- 172.21.15.44:0/508126487 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe828014620 con 0x7fe83015aa00 2024-04-02T13:23:21.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe8267fc640 1 -- 172.21.15.44:0/508126487 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe828019070 con 0x7fe83015aa00 2024-04-02T13:23:21.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe837d50640 1 -- 172.21.15.44:0/508126487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe830145f50 con 0x7fe83015aa00 2024-04-02T13:23:21.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe8267fc640 1 -- 172.21.15.44:0/508126487 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe828014a20 con 0x7fe83015aa00 2024-04-02T13:23:21.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.945+0000 7fe837d50640 1 -- 172.21.15.44:0/508126487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe83014bdf0 con 0x7fe83015aa00 2024-04-02T13:23:21.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.946+0000 7fe837d50640 1 -- 172.21.15.44:0/508126487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe830074f20 con 0x7fe83015aa00 2024-04-02T13:23:21.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.947+0000 7fe8267fc640 1 -- 172.21.15.44:0/508126487 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe828014bc0 con 0x7fe83015aa00 2024-04-02T13:23:21.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.947+0000 7fe8267fc640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe81406c920 0x7fe81406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:21.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.947+0000 7fe8267fc640 1 -- 172.21.15.44:0/508126487 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5249+0+0 (secure 0 0 0) 0x7fe82808c210 con 0x7fe83015aa00 2024-04-02T13:23:21.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.947+0000 7fe835aec640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe81406c920 0x7fe81406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:21.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.948+0000 7fe835aec640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe81406c920 0x7fe81406ede0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe82c037000 tx=0x7fe82c031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:21.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:21.952+0000 7fe8267fc640 1 -- 172.21.15.44:0/508126487 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe828011260 con 0x7fe83015aa00 2024-04-02T13:23:22.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.137+0000 7fe837d50640 1 -- 172.21.15.44:0/508126487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe8300684f0 con 0x7fe83015aa00 2024-04-02T13:23:22.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.138+0000 7fe8267fc640 1 -- 172.21.15.44:0/508126487 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+13116 (secure 0 0 0) 0x7fe82805a890 con 0x7fe83015aa00 2024-04-02T13:23:22.139 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:22.140 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":61,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:23:21.422689+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:23:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.140+0000 7fe837d50640 1 -- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe81406c920 msgr2=0x7fe81406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.140+0000 7fe837d50640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe81406c920 0x7fe81406ede0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe82c037000 tx=0x7fe82c031000 comp rx=0 tx=0).stop 2024-04-02T13:23:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.140+0000 7fe837d50640 1 -- 172.21.15.44:0/508126487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe83015aa00 msgr2=0x7fe83006dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.140+0000 7fe837d50640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe83015aa00 0x7fe83006dc40 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fe828007e70 tx=0x7fe82800e4b0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.140+0000 7fe837d50640 1 -- 172.21.15.44:0/508126487 shutdown_connections 2024-04-02T13:23:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.140+0000 7fe837d50640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe81406c920 0x7fe81406ede0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.140+0000 7fe837d50640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe83015aa00 0x7fe83006dc40 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.140+0000 7fe837d50640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8300ce2e0 0x7fe8301fbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.140+0000 7fe837d50640 1 --2- 172.21.15.44:0/508126487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe830072bf0 0x7fe8301fb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.140+0000 7fe837d50640 1 -- 172.21.15.44:0/508126487 >> 172.21.15.44:0/508126487 conn(0x7fe8300cfcd0 msgr2=0x7fe8300af870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.140+0000 7fe837d50640 1 -- 172.21.15.44:0/508126487 shutdown_connections 2024-04-02T13:23:22.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.140+0000 7fe837d50640 1 -- 172.21.15.44:0/508126487 wait complete. 2024-04-02T13:23:22.152 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:23:22.153 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:23:22.173 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:23:22.198 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:23:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.255+0000 7f215f0b2640 1 Processor -- start 2024-04-02T13:23:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.256+0000 7f215f0b2640 1 -- start start 2024-04-02T13:23:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.256+0000 7f215f0b2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f215815bfa0 0x7f21580b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.256+0000 7f215f0b2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f215806bbe0 0x7f21580d0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.256+0000 7f215f0b2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21580afcf0 con 0x7f21580add60 2024-04-02T13:23:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.256+0000 7f215f0b2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f215815c380 con 0x7f215806bbe0 2024-04-02T13:23:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.256+0000 7f215f0b2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f21580d0a10 con 0x7f215815bfa0 2024-04-02T13:23:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.256+0000 7f215d64f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f215806bbe0 0x7f21580d0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.256+0000 7f215d64f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f215806bbe0 0x7f21580d0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53312/0 (socket says 172.21.15.44:53312) 2024-04-02T13:23:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.257+0000 7f215d64f640 1 -- 172.21.15.44:0/3142347380 learned_addr learned my addr 172.21.15.44:0/3142347380 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.257+0000 7f215ce4e640 1 --2- 172.21.15.44:0/3142347380 >> v2:172.21.15.69:3300/0 conn(0x7f215815bfa0 0x7f21580b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.257+0000 7f215d64f640 1 -- 172.21.15.44:0/3142347380 >> v2:172.21.15.69:3300/0 conn(0x7f215815bfa0 msgr2=0x7f21580b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.257+0000 7f215d64f640 1 --2- 172.21.15.44:0/3142347380 >> v2:172.21.15.69:3300/0 conn(0x7f215815bfa0 0x7f21580b3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.257+0000 7f215d64f640 1 -- 172.21.15.44:0/3142347380 >> v1:172.21.15.69:6789/0 conn(0x7f21580add60 legacy=0x7f215806b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.257+0000 7f215d64f640 1 -- 172.21.15.44:0/3142347380 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21580affd0 con 0x7f215806bbe0 2024-04-02T13:23:22.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.258+0000 7f215d64f640 1 --2- 172.21.15.44:0/3142347380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f215806bbe0 0x7f21580d0ea0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2158072330 tx=0x7f215002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae125c6336981d5c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.258+0000 7f21577fe640 1 -- 172.21.15.44:0/3142347380 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215002c9f0 con 0x7f215806bbe0 2024-04-02T13:23:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.258+0000 7f21577fe640 1 -- 172.21.15.44:0/3142347380 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f215002cb90 con 0x7f215806bbe0 2024-04-02T13:23:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.258+0000 7f21577fe640 1 -- 172.21.15.44:0/3142347380 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21500343f0 con 0x7f215806bbe0 2024-04-02T13:23:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.258+0000 7f215f0b2640 1 -- 172.21.15.44:0/3142347380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f215806bbe0 msgr2=0x7f21580d0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.258+0000 7f215f0b2640 1 --2- 172.21.15.44:0/3142347380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f215806bbe0 0x7f21580d0ea0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2158072330 tx=0x7f215002be50 comp rx=0 tx=0).stop 2024-04-02T13:23:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.258+0000 7f215f0b2640 1 -- 172.21.15.44:0/3142347380 shutdown_connections 2024-04-02T13:23:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.258+0000 7f215f0b2640 1 --2- 172.21.15.44:0/3142347380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f215806bbe0 0x7f21580d0ea0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.258+0000 7f215f0b2640 1 --2- 172.21.15.44:0/3142347380 >> v2:172.21.15.69:3300/0 conn(0x7f215815bfa0 0x7f21580b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.258+0000 7f215f0b2640 1 -- 172.21.15.44:0/3142347380 >> 172.21.15.44:0/3142347380 conn(0x7f215806c5e0 msgr2=0x7f215806c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.258+0000 7f215f0b2640 1 -- 172.21.15.44:0/3142347380 shutdown_connections 2024-04-02T13:23:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.258+0000 7f215f0b2640 1 -- 172.21.15.44:0/3142347380 wait complete. 2024-04-02T13:23:22.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.259+0000 7f215f0b2640 1 Processor -- start 2024-04-02T13:23:22.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.259+0000 7f215f0b2640 1 -- start start 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.260+0000 7f215f0b2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f215806bbe0 0x7f215814cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.260+0000 7f215f0b2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21580add60 0x7f21581f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.260+0000 7f215f0b2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f215815bfa0 0x7f21580cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.260+0000 7f215f0b2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f21580b0590 con 0x7f215806bbe0 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.260+0000 7f215f0b2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f21580d0790 con 0x7f21580add60 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.260+0000 7f215f0b2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f21580af8c0 con 0x7f215815bfa0 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.260+0000 7f215ce4e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f215806bbe0 0x7f215814cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.260+0000 7f215ce4e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f215806bbe0 0x7f215814cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53324/0 (socket says 172.21.15.44:53324) 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.260+0000 7f215ce4e640 1 -- 172.21.15.44:0/4155690106 learned_addr learned my addr 172.21.15.44:0/4155690106 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.261+0000 7f215ce4e640 1 -- 172.21.15.44:0/4155690106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f215815bfa0 msgr2=0x7f21580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.261+0000 7f215ce4e640 1 --2- 172.21.15.44:0/4155690106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f215815bfa0 0x7f21580cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.261+0000 7f215ce4e640 1 -- 172.21.15.44:0/4155690106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21580add60 msgr2=0x7f21581f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.261+0000 7f2157fff640 1 --2- 172.21.15.44:0/4155690106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21580add60 0x7f21581f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.261+0000 7f215ce4e640 1 --2- 172.21.15.44:0/4155690106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21580add60 0x7f21581f3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.261+0000 7f215ce4e640 1 -- 172.21.15.44:0/4155690106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215814d050 con 0x7f215806bbe0 2024-04-02T13:23:22.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.261+0000 7f215ce4e640 1 --2- 172.21.15.44:0/4155690106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f215806bbe0 0x7f215814cb10 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f214c00c7f0 tx=0x7f214c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:22.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.261+0000 7f21557fa640 1 -- 172.21.15.44:0/4155690106 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214c015070 con 0x7f215806bbe0 2024-04-02T13:23:22.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.261+0000 7f21557fa640 1 -- 172.21.15.44:0/4155690106 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f214c004220 con 0x7f215806bbe0 2024-04-02T13:23:22.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.261+0000 7f215f0b2640 1 -- 172.21.15.44:0/4155690106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21581f46e0 con 0x7f215806bbe0 2024-04-02T13:23:22.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.261+0000 7f21557fa640 1 -- 172.21.15.44:0/4155690106 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214c007620 con 0x7f215806bbe0 2024-04-02T13:23:22.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.261+0000 7f215f0b2640 1 -- 172.21.15.44:0/4155690106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21581f4ca0 con 0x7f215806bbe0 2024-04-02T13:23:22.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.262+0000 7f215f0b2640 1 -- 172.21.15.44:0/4155690106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2158068c00 con 0x7f215806bbe0 2024-04-02T13:23:22.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.262+0000 7f21557fa640 1 -- 172.21.15.44:0/4155690106 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f214c0079d0 con 0x7f215806bbe0 2024-04-02T13:23:22.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.263+0000 7f21557fa640 1 --2- 172.21.15.44:0/4155690106 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f213c06c920 0x7f213c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.263+0000 7f21557fa640 1 -- 172.21.15.44:0/4155690106 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5249+0+0 (secure 0 0 0) 0x7f214c08b910 con 0x7f215806bbe0 2024-04-02T13:23:22.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.263+0000 7f2157fff640 1 --2- 172.21.15.44:0/4155690106 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f213c06c920 0x7f213c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.264+0000 7f2157fff640 1 --2- 172.21.15.44:0/4155690106 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f213c06c920 0x7f213c06ede0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2158160b20 tx=0x7f2144009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:22.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.267+0000 7f21557fa640 1 -- 172.21.15.44:0/4155690106 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f214c059f00 con 0x7f215806bbe0 2024-04-02T13:23:22.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.452+0000 7f215f0b2640 1 -- 172.21.15.44:0/4155690106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f21581604b0 con 0x7f215806bbe0 2024-04-02T13:23:22.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.452+0000 7f21557fa640 1 -- 172.21.15.44:0/4155690106 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) ==== 93+0+10 (secure 0 0 0) 0x7f214c0600d0 con 0x7f215806bbe0 2024-04-02T13:23:22.454 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:23:22.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.454+0000 7f215f0b2640 1 -- 172.21.15.44:0/4155690106 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f213c06c920 msgr2=0x7f213c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.454+0000 7f215f0b2640 1 --2- 172.21.15.44:0/4155690106 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f213c06c920 0x7f213c06ede0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2158160b20 tx=0x7f2144009040 comp rx=0 tx=0).stop 2024-04-02T13:23:22.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.454+0000 7f215f0b2640 1 -- 172.21.15.44:0/4155690106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f215806bbe0 msgr2=0x7f215814cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.454+0000 7f215f0b2640 1 --2- 172.21.15.44:0/4155690106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f215806bbe0 0x7f215814cb10 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f214c00c7f0 tx=0x7f214c00ccb0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.454+0000 7f215f0b2640 1 -- 172.21.15.44:0/4155690106 shutdown_connections 2024-04-02T13:23:22.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.454+0000 7f215f0b2640 1 --2- 172.21.15.44:0/4155690106 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f213c06c920 0x7f213c06ede0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.454+0000 7f215f0b2640 1 --2- 172.21.15.44:0/4155690106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f215815bfa0 0x7f21580cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.454+0000 7f215f0b2640 1 --2- 172.21.15.44:0/4155690106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21580add60 0x7f21581f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.454+0000 7f215f0b2640 1 --2- 172.21.15.44:0/4155690106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f215806bbe0 0x7f215814cb10 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.454+0000 7f215f0b2640 1 -- 172.21.15.44:0/4155690106 >> 172.21.15.44:0/4155690106 conn(0x7f215806c5e0 msgr2=0x7f215815f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:22.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.455+0000 7f215f0b2640 1 -- 172.21.15.44:0/4155690106 shutdown_connections 2024-04-02T13:23:22.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.455+0000 7f215f0b2640 1 -- 172.21.15.44:0/4155690106 wait complete. 2024-04-02T13:23:22.467 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T13:23:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.608+0000 7fca8ef15640 1 Processor -- start 2024-04-02T13:23:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.608+0000 7fca8ef15640 1 -- start start 2024-04-02T13:23:22.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca8ef15640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fca88159db0 0x7fca8815c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca8ef15640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fca880ccdf0 0x7fca88165cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca8ef15640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca880726c0 con 0x7fca88071e50 2024-04-02T13:23:22.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca8ef15640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fca88073240 con 0x7fca880ccdf0 2024-04-02T13:23:22.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca8ef15640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fca880729a0 con 0x7fca88159db0 2024-04-02T13:23:22.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca8d4b2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fca880ccdf0 0x7fca88165cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca87fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fca88071e50 0x7fca880cc6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44002/0 (socket says 172.21.15.44:44002) 2024-04-02T13:23:22.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca8d4b2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fca880ccdf0 0x7fca88165cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53330/0 (socket says 172.21.15.44:53330) 2024-04-02T13:23:22.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca87fff640 1 -- 172.21.15.44:0/2190159821 learned_addr learned my addr 172.21.15.44:0/2190159821 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:22.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca8ccb1640 1 --2- 172.21.15.44:0/2190159821 >> v2:172.21.15.69:3300/0 conn(0x7fca88159db0 0x7fca8815c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca8d4b2640 1 -- 172.21.15.44:0/2190159821 >> v2:172.21.15.69:3300/0 conn(0x7fca88159db0 msgr2=0x7fca8815c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:22.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca8d4b2640 1 --2- 172.21.15.44:0/2190159821 >> v2:172.21.15.69:3300/0 conn(0x7fca88159db0 0x7fca8815c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca8d4b2640 1 -- 172.21.15.44:0/2190159821 >> v1:172.21.15.44:6789/0 conn(0x7fca88071e50 legacy=0x7fca880cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca8d4b2640 1 -- 172.21.15.44:0/2190159821 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca88072c80 con 0x7fca880ccdf0 2024-04-02T13:23:22.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca877fe640 1 -- 172.21.15.44:0/2190159821 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2335365862 0 0) 0x7fca880726c0 con 0x7fca88071e50 2024-04-02T13:23:22.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.609+0000 7fca8ccb1640 1 --2- 172.21.15.44:0/2190159821 >> v2:172.21.15.69:3300/0 conn(0x7fca88159db0 0x7fca8815c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:22.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.610+0000 7fca8d4b2640 1 --2- 172.21.15.44:0/2190159821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca880ccdf0 0x7fca88165cf0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fca880733c0 tx=0x7fca8002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e2354d54acd8a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:22.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.610+0000 7fca877fe640 1 -- 172.21.15.44:0/2190159821 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca80039070 con 0x7fca880ccdf0 2024-04-02T13:23:22.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.610+0000 7fca877fe640 1 -- 172.21.15.44:0/2190159821 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca8002cbd0 con 0x7fca880ccdf0 2024-04-02T13:23:22.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.610+0000 7fca877fe640 1 -- 172.21.15.44:0/2190159821 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca80034440 con 0x7fca880ccdf0 2024-04-02T13:23:22.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.611+0000 7fca8ef15640 1 -- 172.21.15.44:0/2190159821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca880ccdf0 msgr2=0x7fca88165cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.611+0000 7fca8ef15640 1 --2- 172.21.15.44:0/2190159821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca880ccdf0 0x7fca88165cf0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fca880733c0 tx=0x7fca8002be20 comp rx=0 tx=0).stop 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.611+0000 7fca8ef15640 1 -- 172.21.15.44:0/2190159821 shutdown_connections 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.611+0000 7fca8ef15640 1 --2- 172.21.15.44:0/2190159821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca880ccdf0 0x7fca88165cf0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.611+0000 7fca8ef15640 1 --2- 172.21.15.44:0/2190159821 >> v2:172.21.15.69:3300/0 conn(0x7fca88159db0 0x7fca8815c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.611+0000 7fca8ef15640 1 -- 172.21.15.44:0/2190159821 >> 172.21.15.44:0/2190159821 conn(0x7fca880cfcd0 msgr2=0x7fca880d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.611+0000 7fca8ef15640 1 -- 172.21.15.44:0/2190159821 shutdown_connections 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.611+0000 7fca8ef15640 1 -- 172.21.15.44:0/2190159821 wait complete. 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.612+0000 7fca8ef15640 1 Processor -- start 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.612+0000 7fca8ef15640 1 -- start start 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca8ef15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca88071e50 0x7fca881584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca8ef15640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca880ccdf0 0x7fca8815d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca8ef15640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca88159db0 0x7fca8815ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca8ef15640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fca88072480 con 0x7fca880ccdf0 2024-04-02T13:23:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca8ef15640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fca88072300 con 0x7fca88159db0 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca8ef15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fca8806e650 con 0x7fca88071e50 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca87fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca880ccdf0 0x7fca8815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca87fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca880ccdf0 0x7fca8815d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53342/0 (socket says 172.21.15.44:53342) 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca87fff640 1 -- 172.21.15.44:0/2015424858 learned_addr learned my addr 172.21.15.44:0/2015424858 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca8d4b2640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca88159db0 0x7fca8815ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca87fff640 1 -- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca88071e50 msgr2=0x7fca881584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca87fff640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca88071e50 0x7fca881584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca87fff640 1 -- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca88159db0 msgr2=0x7fca8815ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca87fff640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca88159db0 0x7fca8815ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca87fff640 1 -- 172.21.15.44:0/2015424858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca8815dec0 con 0x7fca880ccdf0 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca8d4b2640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca88159db0 0x7fca8815ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca87fff640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca880ccdf0 0x7fca8815d750 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fca7400c4d0 tx=0x7fca7400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca857fa640 1 -- 172.21.15.44:0/2015424858 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca74018070 con 0x7fca880ccdf0 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca857fa640 1 -- 172.21.15.44:0/2015424858 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca74007500 con 0x7fca880ccdf0 2024-04-02T13:23:22.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca8ef15640 1 -- 172.21.15.44:0/2015424858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca8815e1a0 con 0x7fca880ccdf0 2024-04-02T13:23:22.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca857fa640 1 -- 172.21.15.44:0/2015424858 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca74004a20 con 0x7fca880ccdf0 2024-04-02T13:23:22.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.613+0000 7fca8ef15640 1 -- 172.21.15.44:0/2015424858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca8815e760 con 0x7fca880ccdf0 2024-04-02T13:23:22.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.614+0000 7fca8ef15640 1 -- 172.21.15.44:0/2015424858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca88074f20 con 0x7fca880ccdf0 2024-04-02T13:23:22.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.615+0000 7fca857fa640 1 -- 172.21.15.44:0/2015424858 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fca7401b070 con 0x7fca880ccdf0 2024-04-02T13:23:22.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.615+0000 7fca857fa640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca6c06c920 0x7fca6c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.615+0000 7fca857fa640 1 -- 172.21.15.44:0/2015424858 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5249+0+0 (secure 0 0 0) 0x7fca7408acd0 con 0x7fca880ccdf0 2024-04-02T13:23:22.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.615+0000 7fca8ccb1640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca6c06c920 0x7fca6c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.616+0000 7fca8ccb1640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca6c06c920 0x7fca6c06ede0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fca7c006e80 tx=0x7fca7c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.620+0000 7fca857fa640 1 -- 172.21.15.44:0/2015424858 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fca740592c0 con 0x7fca880ccdf0 2024-04-02T13:23:22.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.804+0000 7fca8ef15640 1 -- 172.21.15.44:0/2015424858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fca880cf040 con 0x7fca880ccdf0 2024-04-02T13:23:22.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.805+0000 7fca857fa640 1 -- 172.21.15.44:0/2015424858 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v61) ==== 84+0+65 (secure 0 0 0) 0x7fca74058bc0 con 0x7fca880ccdf0 2024-04-02T13:23:22.807 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:22.807 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-04-02T13:23:22.807 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:22.807 INFO:teuthology.orchestra.run.smithi044.stdout: "poolnum": 1, 2024-04-02T13:23:22.807 INFO:teuthology.orchestra.run.smithi044.stdout: "poolname": ".mgr" 2024-04-02T13:23:22.807 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:22.807 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-04-02T13:23:22.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.807+0000 7fca8ef15640 1 -- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca6c06c920 msgr2=0x7fca6c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.807+0000 7fca8ef15640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca6c06c920 0x7fca6c06ede0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fca7c006e80 tx=0x7fca7c008040 comp rx=0 tx=0).stop 2024-04-02T13:23:22.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.807+0000 7fca8ef15640 1 -- 172.21.15.44:0/2015424858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca880ccdf0 msgr2=0x7fca8815d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.807+0000 7fca8ef15640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca880ccdf0 0x7fca8815d750 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fca7400c4d0 tx=0x7fca7400c990 comp rx=0 tx=0).stop 2024-04-02T13:23:22.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.807+0000 7fca8ef15640 1 -- 172.21.15.44:0/2015424858 shutdown_connections 2024-04-02T13:23:22.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.807+0000 7fca8ef15640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca6c06c920 0x7fca6c06ede0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.807+0000 7fca8ef15640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca88159db0 0x7fca8815ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.807+0000 7fca8ef15640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca880ccdf0 0x7fca8815d750 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.807+0000 7fca8ef15640 1 --2- 172.21.15.44:0/2015424858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca88071e50 0x7fca881584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.807+0000 7fca8ef15640 1 -- 172.21.15.44:0/2015424858 >> 172.21.15.44:0/2015424858 conn(0x7fca880cfcd0 msgr2=0x7fca880af870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:22.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.807+0000 7fca8ef15640 1 -- 172.21.15.44:0/2015424858 shutdown_connections 2024-04-02T13:23:22.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.807+0000 7fca8ef15640 1 -- 172.21.15.44:0/2015424858 wait complete. 2024-04-02T13:23:22.820 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T13:23:22.820 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T13:23:22.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.924+0000 7f90e7c63640 1 Processor -- start 2024-04-02T13:23:22.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.924+0000 7f90e7c63640 1 -- start start 2024-04-02T13:23:22.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.925+0000 7f90e7c63640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f90e006daa0 0x7f90e00b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.925+0000 7f90e7c63640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d2e00 0x7f90e00d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.925+0000 7f90e7c63640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90e006de80 con 0x7f90e00cfdc0 2024-04-02T13:23:22.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.925+0000 7f90e7c63640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f90e00cf930 con 0x7f90e006daa0 2024-04-02T13:23:22.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.925+0000 7f90e7c63640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f90e00cf000 con 0x7f90e00d2e00 2024-04-02T13:23:22.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.925+0000 7f90e59ff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f90e006daa0 0x7f90e00b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.925+0000 7f90e51fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f90e00cfdc0 0x7f90e00d26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44018/0 (socket says 172.21.15.44:44018) 2024-04-02T13:23:22.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.925+0000 7f90e51fe640 1 -- 172.21.15.44:0/822497275 learned_addr learned my addr 172.21.15.44:0/822497275 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:22.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.925+0000 7f90e6200640 1 --2- 172.21.15.44:0/822497275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d2e00 0x7f90e00d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.925+0000 7f90e59ff640 1 -- 172.21.15.44:0/822497275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d2e00 msgr2=0x7f90e00d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:22.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.925+0000 7f90e59ff640 1 --2- 172.21.15.44:0/822497275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d2e00 0x7f90e00d11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.925+0000 7f90e59ff640 1 -- 172.21.15.44:0/822497275 >> v1:172.21.15.44:6789/0 conn(0x7f90e00cfdc0 legacy=0x7f90e00d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.925+0000 7f90e59ff640 1 -- 172.21.15.44:0/822497275 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e00cf2e0 con 0x7f90e006daa0 2024-04-02T13:23:22.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.926+0000 7f90e59ff640 1 --2- 172.21.15.44:0/822497275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e006daa0 0x7f90e00b35b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009d90 tx=0x7f90d002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d256110bd79d79b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:22.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.926+0000 7f90e49fd640 1 -- 172.21.15.44:0/822497275 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d0039070 con 0x7f90e006daa0 2024-04-02T13:23:22.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.926+0000 7f90e49fd640 1 -- 172.21.15.44:0/822497275 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90d002cc50 con 0x7f90e006daa0 2024-04-02T13:23:22.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.926+0000 7f90e49fd640 1 -- 172.21.15.44:0/822497275 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d0034440 con 0x7f90e006daa0 2024-04-02T13:23:22.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.927+0000 7f90e7c63640 1 -- 172.21.15.44:0/822497275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e006daa0 msgr2=0x7f90e00b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.927+0000 7f90e7c63640 1 --2- 172.21.15.44:0/822497275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e006daa0 0x7f90e00b35b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009d90 tx=0x7f90d002bea0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.927+0000 7f90e7c63640 1 -- 172.21.15.44:0/822497275 shutdown_connections 2024-04-02T13:23:22.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.927+0000 7f90e7c63640 1 --2- 172.21.15.44:0/822497275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d2e00 0x7f90e00d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.927+0000 7f90e7c63640 1 --2- 172.21.15.44:0/822497275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e006daa0 0x7f90e00b35b0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.927+0000 7f90e7c63640 1 -- 172.21.15.44:0/822497275 >> 172.21.15.44:0/822497275 conn(0x7f90e009d430 msgr2=0x7f90e00d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:22.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.927+0000 7f90e7c63640 1 -- 172.21.15.44:0/822497275 shutdown_connections 2024-04-02T13:23:22.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.927+0000 7f90e7c63640 1 -- 172.21.15.44:0/822497275 wait complete. 2024-04-02T13:23:22.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.928+0000 7f90e7c63640 1 Processor -- start 2024-04-02T13:23:22.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.928+0000 7f90e7c63640 1 -- start start 2024-04-02T13:23:22.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e7c63640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e006daa0 0x7f90e01f4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e7c63640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90e00cfdc0 0x7f90e00b7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e7c63640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d2e00 0x7f90e00b1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e7c63640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f90e00ceda0 con 0x7f90e006daa0 2024-04-02T13:23:22.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e7c63640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f90e00cec20 con 0x7f90e00cfdc0 2024-04-02T13:23:22.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e7c63640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f90e015dd60 con 0x7f90e00d2e00 2024-04-02T13:23:22.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e59ff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e006daa0 0x7f90e01f4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e59ff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e006daa0 0x7f90e01f4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53354/0 (socket says 172.21.15.44:53354) 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e59ff640 1 -- 172.21.15.44:0/4271943430 learned_addr learned my addr 172.21.15.44:0/4271943430 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e51fe640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90e00cfdc0 0x7f90e00b7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e6200640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d2e00 0x7f90e00b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e59ff640 1 -- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d2e00 msgr2=0x7f90e00b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e59ff640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d2e00 0x7f90e00b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e59ff640 1 -- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90e00cfdc0 msgr2=0x7f90e00b7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e59ff640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90e00cfdc0 0x7f90e00b7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.929+0000 7f90e59ff640 1 -- 172.21.15.44:0/4271943430 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e00c8780 con 0x7f90e006daa0 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.930+0000 7f90e59ff640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e006daa0 0x7f90e01f4810 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009d90 tx=0x7f90d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.930+0000 7f90de7fc640 1 -- 172.21.15.44:0/4271943430 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d0039050 con 0x7f90e006daa0 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.930+0000 7f90de7fc640 1 -- 172.21.15.44:0/4271943430 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90d0045070 con 0x7f90e006daa0 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.930+0000 7f90e7c63640 1 -- 172.21.15.44:0/4271943430 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e00b8680 con 0x7f90e006daa0 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.930+0000 7f90de7fc640 1 -- 172.21.15.44:0/4271943430 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d003e610 con 0x7f90e006daa0 2024-04-02T13:23:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.930+0000 7f90e7c63640 1 -- 172.21.15.44:0/4271943430 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e00b8c40 con 0x7f90e006daa0 2024-04-02T13:23:22.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.931+0000 7f90e7c63640 1 -- 172.21.15.44:0/4271943430 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90e00684f0 con 0x7f90e006daa0 2024-04-02T13:23:22.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.931+0000 7f90de7fc640 1 -- 172.21.15.44:0/4271943430 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f90d003e7b0 con 0x7f90e006daa0 2024-04-02T13:23:22.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.932+0000 7f90de7fc640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f90cc06c8f0 0x7f90cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:22.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.932+0000 7f90de7fc640 1 -- 172.21.15.44:0/4271943430 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5249+0+0 (secure 0 0 0) 0x7f90d00ae420 con 0x7f90e006daa0 2024-04-02T13:23:22.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.932+0000 7f90e51fe640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f90cc06c8f0 0x7f90cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:22.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.932+0000 7f90e51fe640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f90cc06c8f0 0x7f90cc06edb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f90e00c7cf0 tx=0x7f90d4007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:22.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:22.936+0000 7f90de7fc640 1 -- 172.21.15.44:0/4271943430 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f90d007c340 con 0x7f90e006daa0 2024-04-02T13:23:23.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.122+0000 7f90e7c63640 1 -- 172.21.15.44:0/4271943430 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f90e00b69f0 con 0x7f90e006daa0 2024-04-02T13:23:23.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.432+0000 7f90de7fc640 1 -- 172.21.15.44:0/4271943430 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v62) ==== 138+0+0 (secure 0 0 0) 0x7f90d0080640 con 0x7f90e006daa0 2024-04-02T13:23:23.434 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' created 2024-04-02T13:23:23.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.434+0000 7f90e7c63640 1 -- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f90cc06c8f0 msgr2=0x7f90cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:23.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.434+0000 7f90e7c63640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f90cc06c8f0 0x7f90cc06edb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f90e00c7cf0 tx=0x7f90d4007e80 comp rx=0 tx=0).stop 2024-04-02T13:23:23.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.434+0000 7f90e7c63640 1 -- 172.21.15.44:0/4271943430 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e006daa0 msgr2=0x7f90e01f4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:23.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.434+0000 7f90e7c63640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e006daa0 0x7f90e01f4810 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009d90 tx=0x7f90d0004a40 comp rx=0 tx=0).stop 2024-04-02T13:23:23.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.434+0000 7f90e7c63640 1 -- 172.21.15.44:0/4271943430 shutdown_connections 2024-04-02T13:23:23.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.434+0000 7f90e7c63640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f90cc06c8f0 0x7f90cc06edb0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:23.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.434+0000 7f90e7c63640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90e00d2e00 0x7f90e00b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:23.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.434+0000 7f90e7c63640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90e00cfdc0 0x7f90e00b7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:23.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.434+0000 7f90e7c63640 1 --2- 172.21.15.44:0/4271943430 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90e006daa0 0x7f90e01f4810 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:23.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.434+0000 7f90e7c63640 1 -- 172.21.15.44:0/4271943430 >> 172.21.15.44:0/4271943430 conn(0x7f90e009d430 msgr2=0x7f90e015b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:23.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.434+0000 7f90e7c63640 1 -- 172.21.15.44:0/4271943430 shutdown_connections 2024-04-02T13:23:23.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.435+0000 7f90e7c63640 1 -- 172.21.15.44:0/4271943430 wait complete. 2024-04-02T13:23:23.452 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T13:23:23.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.560+0000 7fd49f3ba640 1 Processor -- start 2024-04-02T13:23:23.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.560+0000 7fd49f3ba640 1 -- start start 2024-04-02T13:23:23.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49f3ba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd49815e470 0x7fd4980b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:23.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49f3ba640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd4980ce790 0x7fd49806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:23.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49f3ba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4980ced00 con 0x7fd4980b1540 2024-04-02T13:23:23.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49f3ba640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd4980cefe0 con 0x7fd4980ce790 2024-04-02T13:23:23.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49f3ba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd4980cf2c0 con 0x7fd49815e470 2024-04-02T13:23:23.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49d957640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd4980ce790 0x7fd49806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:23.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49d957640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd4980ce790 0x7fd49806d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53358/0 (socket says 172.21.15.44:53358) 2024-04-02T13:23:23.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49d957640 1 -- 172.21.15.44:0/3935098195 learned_addr learned my addr 172.21.15.44:0/3935098195 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:23.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49d156640 1 --2- 172.21.15.44:0/3935098195 >> v2:172.21.15.69:3300/0 conn(0x7fd49815e470 0x7fd4980b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:23.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49d957640 1 -- 172.21.15.44:0/3935098195 >> v2:172.21.15.69:3300/0 conn(0x7fd49815e470 msgr2=0x7fd4980b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:23.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49d957640 1 --2- 172.21.15.44:0/3935098195 >> v2:172.21.15.69:3300/0 conn(0x7fd49815e470 0x7fd4980b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:23.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49d957640 1 -- 172.21.15.44:0/3935098195 >> v1:172.21.15.44:6789/0 conn(0x7fd4980b1540 legacy=0x7fd4980d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:23.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd48bfff640 1 -- 172.21.15.44:0/3935098195 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4262406332 0 0) 0x7fd4980ced00 con 0x7fd4980b1540 2024-04-02T13:23:23.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49d957640 1 -- 172.21.15.44:0/3935098195 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4980cf5a0 con 0x7fd4980ce790 2024-04-02T13:23:23.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.561+0000 7fd49d156640 1 --2- 172.21.15.44:0/3935098195 >> v2:172.21.15.69:3300/0 conn(0x7fd49815e470 0x7fd4980b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:23.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.562+0000 7fd49d957640 1 --2- 172.21.15.44:0/3935098195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4980ce790 0x7fd49806d0f0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd49806f1c0 tx=0x7fd49002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae89ab2ce66adff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:23.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.562+0000 7fd48bfff640 1 -- 172.21.15.44:0/3935098195 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490035020 con 0x7fd4980ce790 2024-04-02T13:23:23.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.562+0000 7fd48bfff640 1 -- 172.21.15.44:0/3935098195 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd490031870 con 0x7fd4980ce790 2024-04-02T13:23:23.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.562+0000 7fd48bfff640 1 -- 172.21.15.44:0/3935098195 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490031c40 con 0x7fd4980ce790 2024-04-02T13:23:23.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.563+0000 7fd49f3ba640 1 -- 172.21.15.44:0/3935098195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4980ce790 msgr2=0x7fd49806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:23.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.563+0000 7fd49f3ba640 1 --2- 172.21.15.44:0/3935098195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4980ce790 0x7fd49806d0f0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd49806f1c0 tx=0x7fd49002b910 comp rx=0 tx=0).stop 2024-04-02T13:23:23.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.563+0000 7fd49f3ba640 1 -- 172.21.15.44:0/3935098195 shutdown_connections 2024-04-02T13:23:23.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.563+0000 7fd49f3ba640 1 --2- 172.21.15.44:0/3935098195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4980ce790 0x7fd49806d0f0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:23.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.563+0000 7fd49f3ba640 1 --2- 172.21.15.44:0/3935098195 >> v2:172.21.15.69:3300/0 conn(0x7fd49815e470 0x7fd4980b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:23.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.563+0000 7fd49f3ba640 1 -- 172.21.15.44:0/3935098195 >> 172.21.15.44:0/3935098195 conn(0x7fd49809aa20 msgr2=0x7fd49809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:23.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.563+0000 7fd49f3ba640 1 -- 172.21.15.44:0/3935098195 shutdown_connections 2024-04-02T13:23:23.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.563+0000 7fd49f3ba640 1 -- 172.21.15.44:0/3935098195 wait complete. 2024-04-02T13:23:23.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.564+0000 7fd49f3ba640 1 Processor -- start 2024-04-02T13:23:23.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.564+0000 7fd49f3ba640 1 -- start start 2024-04-02T13:23:23.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49f3ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4980b1540 0x7fd4980b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:23.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49f3ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4980ce790 0x7fd4980c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:23.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49f3ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49815e470 0x7fd4980c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:23.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49f3ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd4980d22c0 con 0x7fd49815e470 2024-04-02T13:23:23.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49f3ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd49806cb60 con 0x7fd4980ce790 2024-04-02T13:23:23.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49f3ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd4980d2440 con 0x7fd4980b1540 2024-04-02T13:23:23.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49d957640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49815e470 0x7fd4980c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:23.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49d957640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49815e470 0x7fd4980c5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53366/0 (socket says 172.21.15.44:53366) 2024-04-02T13:23:23.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49d957640 1 -- 172.21.15.44:0/2501162664 learned_addr learned my addr 172.21.15.44:0/2501162664 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:23.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49c955640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4980ce790 0x7fd4980c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:23.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49d156640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4980b1540 0x7fd4980b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:23.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49d957640 1 -- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4980b1540 msgr2=0x7fd4980b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49d957640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4980b1540 0x7fd4980b6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49d957640 1 -- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4980ce790 msgr2=0x7fd4980c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49d957640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4980ce790 0x7fd4980c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49d957640 1 -- 172.21.15.44:0/2501162664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4980c6520 con 0x7fd49815e470 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49c955640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4980ce790 0x7fd4980c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.565+0000 7fd49d957640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49815e470 0x7fd4980c5ef0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd49002b950 tx=0x7fd4900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.566+0000 7fd489ffb640 1 -- 172.21.15.44:0/2501162664 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490040070 con 0x7fd49815e470 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.566+0000 7fd489ffb640 1 -- 172.21.15.44:0/2501162664 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd490030c50 con 0x7fd49815e470 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.566+0000 7fd489ffb640 1 -- 172.21.15.44:0/2501162664 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490035040 con 0x7fd49815e470 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.566+0000 7fd49f3ba640 1 -- 172.21.15.44:0/2501162664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4980c6800 con 0x7fd49815e470 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.566+0000 7fd49f3ba640 1 -- 172.21.15.44:0/2501162664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4980c70a0 con 0x7fd49815e470 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.567+0000 7fd49f3ba640 1 -- 172.21.15.44:0/2501162664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4980c69d0 con 0x7fd49815e470 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.568+0000 7fd489ffb640 1 -- 172.21.15.44:0/2501162664 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd490038020 con 0x7fd49815e470 2024-04-02T13:23:23.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.568+0000 7fd489ffb640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd47c06c8f0 0x7fd47c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:23.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.568+0000 7fd489ffb640 1 -- 172.21.15.44:0/2501162664 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5619+0+0 (secure 0 0 0) 0x7fd4900ad8d0 con 0x7fd49815e470 2024-04-02T13:23:23.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.568+0000 7fd49d156640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd47c06c8f0 0x7fd47c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:23.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.569+0000 7fd49d156640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd47c06c8f0 0x7fd47c06edb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd494009ed0 tx=0x7fd49400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:23.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.573+0000 7fd489ffb640 1 -- 172.21.15.44:0/2501162664 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd49007bd50 con 0x7fd49815e470 2024-04-02T13:23:23.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:23.759+0000 7fd49f3ba640 1 -- 172.21.15.44:0/2501162664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd4980684f0 con 0x7fd49815e470 2024-04-02T13:23:24.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.434+0000 7fd489ffb640 1 -- 172.21.15.44:0/2501162664 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v63) ==== 170+0+0 (secure 0 0 0) 0x7fd49007b650 con 0x7fd49815e470 2024-04-02T13:23:24.436 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' created 2024-04-02T13:23:24.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.437+0000 7fd49f3ba640 1 -- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd47c06c8f0 msgr2=0x7fd47c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:24.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.437+0000 7fd49f3ba640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd47c06c8f0 0x7fd47c06edb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd494009ed0 tx=0x7fd49400b040 comp rx=0 tx=0).stop 2024-04-02T13:23:24.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.437+0000 7fd49f3ba640 1 -- 172.21.15.44:0/2501162664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49815e470 msgr2=0x7fd4980c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:24.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.437+0000 7fd49f3ba640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49815e470 0x7fd4980c5ef0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd49002b950 tx=0x7fd4900046c0 comp rx=0 tx=0).stop 2024-04-02T13:23:24.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.437+0000 7fd49f3ba640 1 -- 172.21.15.44:0/2501162664 shutdown_connections 2024-04-02T13:23:24.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.437+0000 7fd49f3ba640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd47c06c8f0 0x7fd47c06edb0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:24.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.437+0000 7fd49f3ba640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd49815e470 0x7fd4980c5ef0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:24.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.437+0000 7fd49f3ba640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4980ce790 0x7fd4980c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:24.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.437+0000 7fd49f3ba640 1 --2- 172.21.15.44:0/2501162664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4980b1540 0x7fd4980b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:24.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.437+0000 7fd49f3ba640 1 -- 172.21.15.44:0/2501162664 >> 172.21.15.44:0/2501162664 conn(0x7fd49809aa20 msgr2=0x7fd498161f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:24.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.437+0000 7fd49f3ba640 1 -- 172.21.15.44:0/2501162664 shutdown_connections 2024-04-02T13:23:24.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.437+0000 7fd49f3ba640 1 -- 172.21.15.44:0/2501162664 wait complete. 2024-04-02T13:23:24.460 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T13:23:24.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.566+0000 7fb2a41c8640 1 Processor -- start 2024-04-02T13:23:24.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.566+0000 7fb2a41c8640 1 -- start start 2024-04-02T13:23:24.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.567+0000 7fb2a41c8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb29c0cec60 0x7fb29c0d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:24.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.567+0000 7fb2a41c8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb29c15e5b0 0x7fb29c0d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:24.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.567+0000 7fb2a41c8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb29c0cf1f0 con 0x7fb29c15e1d0 2024-04-02T13:23:24.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.567+0000 7fb2a41c8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fb29c0cfe00 con 0x7fb29c15e5b0 2024-04-02T13:23:24.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.567+0000 7fb2a41c8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb29c0cf4d0 con 0x7fb29c0cec60 2024-04-02T13:23:24.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.567+0000 7fb2a2765640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb29c15e5b0 0x7fb29c0d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:24.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.567+0000 7fb2a2765640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb29c15e5b0 0x7fb29c0d2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56270/0 (socket says 172.21.15.44:56270) 2024-04-02T13:23:24.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.567+0000 7fb2a2765640 1 -- 172.21.15.44:0/498751187 learned_addr learned my addr 172.21.15.44:0/498751187 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:24.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.568+0000 7fb2a1763640 1 --2- 172.21.15.44:0/498751187 >> v2:172.21.15.69:3300/0 conn(0x7fb29c0cec60 0x7fb29c0d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:24.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.568+0000 7fb2a2765640 1 -- 172.21.15.44:0/498751187 >> v2:172.21.15.69:3300/0 conn(0x7fb29c0cec60 msgr2=0x7fb29c0d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:24.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.568+0000 7fb2a2765640 1 --2- 172.21.15.44:0/498751187 >> v2:172.21.15.69:3300/0 conn(0x7fb29c0cec60 0x7fb29c0d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:24.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.568+0000 7fb2a2765640 1 -- 172.21.15.44:0/498751187 >> v1:172.21.15.69:6789/0 conn(0x7fb29c15e1d0 legacy=0x7fb29c0d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:24.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.568+0000 7fb2a2765640 1 -- 172.21.15.44:0/498751187 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29c0cf7b0 con 0x7fb29c15e5b0 2024-04-02T13:23:24.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.568+0000 7fb2a1763640 1 --2- 172.21.15.44:0/498751187 >> v2:172.21.15.69:3300/0 conn(0x7fb29c0cec60 0x7fb29c0d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:24.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.569+0000 7fb2a2765640 1 --2- 172.21.15.44:0/498751187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb29c15e5b0 0x7fb29c0d2c00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb29c161f20 tx=0x7fb28c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e00b5f8e977bf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:24.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.569+0000 7fb2a0f62640 1 -- 172.21.15.44:0/498751187 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c02ca80 con 0x7fb29c15e5b0 2024-04-02T13:23:24.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.569+0000 7fb2a0f62640 1 -- 172.21.15.44:0/498751187 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb28c02cc20 con 0x7fb29c15e5b0 2024-04-02T13:23:24.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.569+0000 7fb2a0f62640 1 -- 172.21.15.44:0/498751187 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c0343f0 con 0x7fb29c15e5b0 2024-04-02T13:23:24.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.569+0000 7fb2a41c8640 1 -- 172.21.15.44:0/498751187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb29c15e5b0 msgr2=0x7fb29c0d2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:24.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.569+0000 7fb2a41c8640 1 --2- 172.21.15.44:0/498751187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb29c15e5b0 0x7fb29c0d2c00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb29c161f20 tx=0x7fb28c02bee0 comp rx=0 tx=0).stop 2024-04-02T13:23:24.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.569+0000 7fb2a41c8640 1 -- 172.21.15.44:0/498751187 shutdown_connections 2024-04-02T13:23:24.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.569+0000 7fb2a41c8640 1 --2- 172.21.15.44:0/498751187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb29c15e5b0 0x7fb29c0d2c00 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:24.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.569+0000 7fb2a41c8640 1 --2- 172.21.15.44:0/498751187 >> v2:172.21.15.69:3300/0 conn(0x7fb29c0cec60 0x7fb29c0d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:24.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.569+0000 7fb2a41c8640 1 -- 172.21.15.44:0/498751187 >> 172.21.15.44:0/498751187 conn(0x7fb29c09d3b0 msgr2=0x7fb29c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:24.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.569+0000 7fb2a41c8640 1 -- 172.21.15.44:0/498751187 shutdown_connections 2024-04-02T13:23:24.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.569+0000 7fb2a41c8640 1 -- 172.21.15.44:0/498751187 wait complete. 2024-04-02T13:23:24.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.571+0000 7fb2a41c8640 1 Processor -- start 2024-04-02T13:23:24.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.571+0000 7fb2a41c8640 1 -- start start 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.571+0000 7fb2a41c8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb29c0cec60 0x7fb29c15a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.571+0000 7fb2a41c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb29c15e1d0 0x7fb29c153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.571+0000 7fb2a41c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb29c15e5b0 0x7fb29c0ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.571+0000 7fb2a41c8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c0b4e60 con 0x7fb29c15e1d0 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.571+0000 7fb2a41c8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c0b4ce0 con 0x7fb29c0cec60 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.571+0000 7fb2a41c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb29c0b4fe0 con 0x7fb29c15e5b0 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.571+0000 7fb2a1763640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb29c15e1d0 0x7fb29c153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.571+0000 7fb2a1763640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb29c15e1d0 0x7fb29c153820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56278/0 (socket says 172.21.15.44:56278) 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.571+0000 7fb2a1763640 1 -- 172.21.15.44:0/3848799759 learned_addr learned my addr 172.21.15.44:0/3848799759 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.572+0000 7fb2a2765640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb29c15e5b0 0x7fb29c0ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.572+0000 7fb2a1f64640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb29c0cec60 0x7fb29c15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.572+0000 7fb2a1763640 1 -- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb29c15e5b0 msgr2=0x7fb29c0ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.572+0000 7fb2a1763640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb29c15e5b0 0x7fb29c0ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.572+0000 7fb2a1763640 1 -- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb29c0cec60 msgr2=0x7fb29c15a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:24.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.572+0000 7fb2a1763640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb29c0cec60 0x7fb29c15a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:24.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.572+0000 7fb2a1763640 1 -- 172.21.15.44:0/3848799759 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29c1555e0 con 0x7fb29c15e1d0 2024-04-02T13:23:24.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.572+0000 7fb2a1763640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb29c15e1d0 0x7fb29c153820 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fb29000e730 tx=0x7fb29000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:24.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.572+0000 7fb29a7fc640 1 -- 172.21.15.44:0/3848799759 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29000f040 con 0x7fb29c15e1d0 2024-04-02T13:23:24.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.572+0000 7fb29a7fc640 1 -- 172.21.15.44:0/3848799759 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb290004230 con 0x7fb29c15e1d0 2024-04-02T13:23:24.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.572+0000 7fb2a41c8640 1 -- 172.21.15.44:0/3848799759 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb29c1558c0 con 0x7fb29c15e1d0 2024-04-02T13:23:24.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.572+0000 7fb29a7fc640 1 -- 172.21.15.44:0/3848799759 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29000ee40 con 0x7fb29c15e1d0 2024-04-02T13:23:24.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.572+0000 7fb2a41c8640 1 -- 172.21.15.44:0/3848799759 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb29c155e80 con 0x7fb29c15e1d0 2024-04-02T13:23:24.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.573+0000 7fb2a41c8640 1 -- 172.21.15.44:0/3848799759 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb29c0684f0 con 0x7fb29c15e1d0 2024-04-02T13:23:24.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.574+0000 7fb29a7fc640 1 -- 172.21.15.44:0/3848799759 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb29000c6d0 con 0x7fb29c15e1d0 2024-04-02T13:23:24.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.574+0000 7fb29a7fc640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb28806c8f0 0x7fb28806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:24.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.574+0000 7fb29a7fc640 1 -- 172.21.15.44:0/3848799759 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6001+0+0 (secure 0 0 0) 0x7fb29008bc30 con 0x7fb29c15e1d0 2024-04-02T13:23:24.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.574+0000 7fb2a1f64640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb28806c8f0 0x7fb28806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:24.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.575+0000 7fb2a1f64640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb28806c8f0 0x7fb28806edb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb294001780 tx=0x7fb294009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:24.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.578+0000 7fb29a7fc640 1 -- 172.21.15.44:0/3848799759 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb290017020 con 0x7fb29c15e1d0 2024-04-02T13:23:24.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:24.802+0000 7fb2a41c8640 1 -- 172.21.15.44:0/3848799759 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb29c06c3b0 con 0x7fb29c15e1d0 2024-04-02T13:23:25.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.455+0000 7fb29a7fc640 1 -- 172.21.15.44:0/3848799759 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v31) ==== 173+0+0 (secure 0 0 0) 0x7fb290059a50 con 0x7fb29c15e1d0 2024-04-02T13:23:25.459 INFO:teuthology.orchestra.run.smithi044.stderr:new fs with metadata pool 9 and data pool 10 2024-04-02T13:23:25.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.462+0000 7fb2a41c8640 1 -- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb28806c8f0 msgr2=0x7fb28806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:25.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.462+0000 7fb2a41c8640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb28806c8f0 0x7fb28806edb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb294001780 tx=0x7fb294009040 comp rx=0 tx=0).stop 2024-04-02T13:23:25.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.462+0000 7fb2a41c8640 1 -- 172.21.15.44:0/3848799759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb29c15e1d0 msgr2=0x7fb29c153820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:25.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.462+0000 7fb2a41c8640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb29c15e1d0 0x7fb29c153820 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fb29000e730 tx=0x7fb29000ebf0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.464+0000 7fb2a41c8640 1 -- 172.21.15.44:0/3848799759 shutdown_connections 2024-04-02T13:23:25.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.464+0000 7fb2a41c8640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb28806c8f0 0x7fb28806edb0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.464+0000 7fb2a41c8640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb29c15e5b0 0x7fb29c0ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.464+0000 7fb2a41c8640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb29c15e1d0 0x7fb29c153820 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.464+0000 7fb2a41c8640 1 --2- 172.21.15.44:0/3848799759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb29c0cec60 0x7fb29c15a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.464+0000 7fb2a41c8640 1 -- 172.21.15.44:0/3848799759 >> 172.21.15.44:0/3848799759 conn(0x7fb29c09d3b0 msgr2=0x7fb29c0b73a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:25.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.467+0000 7fb2a41c8640 1 -- 172.21.15.44:0/3848799759 shutdown_connections 2024-04-02T13:23:25.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.469+0000 7fb2a41c8640 1 -- 172.21.15.44:0/3848799759 wait complete. 2024-04-02T13:23:25.489 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:23:25.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.592+0000 7f79e0f11640 1 Processor -- start 2024-04-02T13:23:25.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.592+0000 7f79e0f11640 1 -- start start 2024-04-02T13:23:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79e0f11640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f79dc15c050 0x7f79dc0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79e0f11640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc0b1ed0 0x7f79dc0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79e0f11640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79dc0ccb10 con 0x7f79dc15c430 2024-04-02T13:23:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79e0f11640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f79dc0b2480 con 0x7f79dc15c050 2024-04-02T13:23:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79e0f11640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f79dc0b2760 con 0x7f79dc0b1ed0 2024-04-02T13:23:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79da59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f79dc15c050 0x7f79dc0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:25.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79dad9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc0b1ed0 0x7f79dc0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:25.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79d3fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f79dc15c430 0x7f79dc0b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:47144/0 (socket says 172.21.15.44:47144) 2024-04-02T13:23:25.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79d3fff640 1 -- 172.21.15.44:0/463532139 learned_addr learned my addr 172.21.15.44:0/463532139 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:25.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79dad9d640 1 -- 172.21.15.44:0/463532139 >> v2:172.21.15.69:3300/0 conn(0x7f79dc15c050 msgr2=0x7f79dc0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:25.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79dad9d640 1 --2- 172.21.15.44:0/463532139 >> v2:172.21.15.69:3300/0 conn(0x7f79dc15c050 0x7f79dc0ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79dad9d640 1 -- 172.21.15.44:0/463532139 >> v1:172.21.15.69:6789/0 conn(0x7f79dc15c430 legacy=0x7f79dc0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:25.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79dad9d640 1 -- 172.21.15.44:0/463532139 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc0b2a40 con 0x7f79dc0b1ed0 2024-04-02T13:23:25.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.593+0000 7f79da59c640 1 --2- 172.21.15.44:0/463532139 >> v2:172.21.15.69:3300/0 conn(0x7f79dc15c050 0x7f79dc0ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:23:25.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.594+0000 7f79dad9d640 1 --2- 172.21.15.44:0/463532139 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc0b1ed0 0x7f79dc0ced90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009d90 tx=0x7f79d402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcb7b6fb0fc0e95 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:25.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.594+0000 7f79d9d9b640 1 -- 172.21.15.44:0/463532139 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d4039070 con 0x7f79dc0b1ed0 2024-04-02T13:23:25.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.594+0000 7f79d9d9b640 1 -- 172.21.15.44:0/463532139 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d402cce0 con 0x7f79dc0b1ed0 2024-04-02T13:23:25.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.594+0000 7f79d9d9b640 1 -- 172.21.15.44:0/463532139 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d4034660 con 0x7f79dc0b1ed0 2024-04-02T13:23:25.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.595+0000 7f79e0f11640 1 -- 172.21.15.44:0/463532139 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc0b1ed0 msgr2=0x7f79dc0ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:25.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.595+0000 7f79e0f11640 1 --2- 172.21.15.44:0/463532139 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc0b1ed0 0x7f79dc0ced90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009d90 tx=0x7f79d402bf30 comp rx=0 tx=0).stop 2024-04-02T13:23:25.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.595+0000 7f79e0f11640 1 -- 172.21.15.44:0/463532139 shutdown_connections 2024-04-02T13:23:25.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.595+0000 7f79e0f11640 1 --2- 172.21.15.44:0/463532139 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc0b1ed0 0x7f79dc0ced90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.595+0000 7f79e0f11640 1 --2- 172.21.15.44:0/463532139 >> v2:172.21.15.69:3300/0 conn(0x7f79dc15c050 0x7f79dc0ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.595+0000 7f79e0f11640 1 -- 172.21.15.44:0/463532139 >> 172.21.15.44:0/463532139 conn(0x7f79dc09aa20 msgr2=0x7f79dc09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:25.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.595+0000 7f79e0f11640 1 -- 172.21.15.44:0/463532139 shutdown_connections 2024-04-02T13:23:25.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.595+0000 7f79e0f11640 1 -- 172.21.15.44:0/463532139 wait complete. 2024-04-02T13:23:25.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.596+0000 7f79e0f11640 1 Processor -- start 2024-04-02T13:23:25.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.596+0000 7f79e0f11640 1 -- start start 2024-04-02T13:23:25.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79e0f11640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc0b1ed0 0x7f79dc145d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:25.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79e0f11640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79dc15c050 0x7f79dc155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:25.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79e0f11640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15c430 0x7f79dc0cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79e0f11640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc0cd140 con 0x7f79dc15c050 2024-04-02T13:23:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79e0f11640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc0ccfc0 con 0x7f79dc15c430 2024-04-02T13:23:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79e0f11640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f79dc0cd2c0 con 0x7f79dc0b1ed0 2024-04-02T13:23:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79d3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79dc15c050 0x7f79dc155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79d3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79dc15c050 0x7f79dc155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56282/0 (socket says 172.21.15.44:56282) 2024-04-02T13:23:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79d3fff640 1 -- 172.21.15.44:0/1759028015 learned_addr learned my addr 172.21.15.44:0/1759028015 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79da59c640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc0b1ed0 0x7f79dc145d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79dad9d640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15c430 0x7f79dc0cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79d3fff640 1 -- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc0b1ed0 msgr2=0x7f79dc145d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79d3fff640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc0b1ed0 0x7f79dc145d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79d3fff640 1 -- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15c430 msgr2=0x7f79dc0cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79d3fff640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15c430 0x7f79dc0cb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.597+0000 7f79d3fff640 1 -- 172.21.15.44:0/1759028015 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc155b60 con 0x7f79dc15c050 2024-04-02T13:23:25.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.598+0000 7f79d3fff640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79dc15c050 0x7f79dc155330 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f79c400c3f0 tx=0x7f79c400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:25.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.598+0000 7f79d2ffd640 1 -- 172.21.15.44:0/1759028015 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c4017070 con 0x7f79dc15c050 2024-04-02T13:23:25.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.598+0000 7f79d2ffd640 1 -- 172.21.15.44:0/1759028015 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79c4005ce0 con 0x7f79dc15c050 2024-04-02T13:23:25.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.598+0000 7f79e0f11640 1 -- 172.21.15.44:0/1759028015 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79dc155e40 con 0x7f79dc15c050 2024-04-02T13:23:25.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.598+0000 7f79d2ffd640 1 -- 172.21.15.44:0/1759028015 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c4007310 con 0x7f79dc15c050 2024-04-02T13:23:25.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.598+0000 7f79e0f11640 1 -- 172.21.15.44:0/1759028015 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79dc156400 con 0x7f79dc15c050 2024-04-02T13:23:25.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.599+0000 7f79e0f11640 1 -- 172.21.15.44:0/1759028015 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79dc0684f0 con 0x7f79dc15c050 2024-04-02T13:23:25.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.599+0000 7f79d2ffd640 1 -- 172.21.15.44:0/1759028015 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f79c401a070 con 0x7f79dc15c050 2024-04-02T13:23:25.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.600+0000 7f79d2ffd640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f79bc06c8f0 0x7f79bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:25.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.600+0000 7f79d2ffd640 1 -- 172.21.15.44:0/1759028015 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f79c408afa0 con 0x7f79dc15c050 2024-04-02T13:23:25.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.600+0000 7f79da59c640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f79bc06c8f0 0x7f79bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:25.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.600+0000 7f79da59c640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f79bc06c8f0 0x7f79bc06edb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f79cc009560 tx=0x7f79cc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:25.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.604+0000 7f79d2ffd640 1 -- 172.21.15.44:0/1759028015 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f79c4058be0 con 0x7f79dc15c050 2024-04-02T13:23:25.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.792+0000 7f79e0f11640 1 -- 172.21.15.44:0/1759028015 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79dc156920 con 0x7f79dc15c050 2024-04-02T13:23:25.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.793+0000 7f79d2ffd640 1 -- 172.21.15.44:0/1759028015 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) ==== 74+0+16849 (secure 0 0 0) 0x7f79c405cee0 con 0x7f79dc15c050 2024-04-02T13:23:25.795 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:25.795 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":64,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:23:24.803585+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:23:23.126712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-02T13:23:23.763593+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:23:25.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.795+0000 7f79e0f11640 1 -- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f79bc06c8f0 msgr2=0x7f79bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:25.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.795+0000 7f79e0f11640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f79bc06c8f0 0x7f79bc06edb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f79cc009560 tx=0x7f79cc009190 comp rx=0 tx=0).stop 2024-04-02T13:23:25.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.795+0000 7f79e0f11640 1 -- 172.21.15.44:0/1759028015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79dc15c050 msgr2=0x7f79dc155330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:25.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.795+0000 7f79e0f11640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79dc15c050 0x7f79dc155330 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f79c400c3f0 tx=0x7f79c400c8b0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.795+0000 7f79e0f11640 1 -- 172.21.15.44:0/1759028015 shutdown_connections 2024-04-02T13:23:25.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.795+0000 7f79e0f11640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f79bc06c8f0 0x7f79bc06edb0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.795+0000 7f79e0f11640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15c430 0x7f79dc0cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.795+0000 7f79e0f11640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79dc15c050 0x7f79dc155330 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.795+0000 7f79e0f11640 1 --2- 172.21.15.44:0/1759028015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc0b1ed0 0x7f79dc145d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.795+0000 7f79e0f11640 1 -- 172.21.15.44:0/1759028015 >> 172.21.15.44:0/1759028015 conn(0x7f79dc09aa20 msgr2=0x7f79dc0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:25.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.795+0000 7f79e0f11640 1 -- 172.21.15.44:0/1759028015 shutdown_connections 2024-04-02T13:23:25.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.795+0000 7f79e0f11640 1 -- 172.21.15.44:0/1759028015 wait complete. 2024-04-02T13:23:25.808 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:23:25.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.908+0000 7fbcd451b640 1 Processor -- start 2024-04-02T13:23:25.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.908+0000 7fbcd451b640 1 -- start start 2024-04-02T13:23:25.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.909+0000 7fbcd451b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbccc0ce2e0 0x7fbccc165820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:25.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.909+0000 7fbcd451b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbccc0b3160 con 0x7fbccc0b2d00 2024-04-02T13:23:25.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.909+0000 7fbcd451b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbccc15ade0 con 0x7fbccc15aa00 2024-04-02T13:23:25.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.909+0000 7fbcd451b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fbccc0721a0 con 0x7fbccc0ce2e0 2024-04-02T13:23:25.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.909+0000 7fbcd2ab8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbccc0ce2e0 0x7fbccc165820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:25.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.909+0000 7fbcd2ab8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbccc0ce2e0 0x7fbccc165820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56284/0 (socket says 172.21.15.44:56284) 2024-04-02T13:23:25.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.909+0000 7fbcd2ab8640 1 -- 172.21.15.44:0/2867300571 learned_addr learned my addr 172.21.15.44:0/2867300571 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:25.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.909+0000 7fbcd2ab8640 1 -- 172.21.15.44:0/2867300571 >> v1:172.21.15.69:6789/0 conn(0x7fbccc15aa00 legacy=0x7fbccc072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:25.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.909+0000 7fbcd2ab8640 1 -- 172.21.15.44:0/2867300571 >> v1:172.21.15.44:6789/0 conn(0x7fbccc0b2d00 legacy=0x7fbccc0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:25.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.909+0000 7fbcd2ab8640 1 -- 172.21.15.44:0/2867300571 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbccc0af1b0 con 0x7fbccc0ce2e0 2024-04-02T13:23:25.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.910+0000 7fbcd2ab8640 1 --2- 172.21.15.44:0/2867300571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbccc0ce2e0 0x7fbccc165820 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009e50 tx=0x7fbcc402d090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca330e3a932e810 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:25.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.910+0000 7fbcd12b5640 1 -- 172.21.15.44:0/2867300571 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc402dc30 con 0x7fbccc0ce2e0 2024-04-02T13:23:25.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.910+0000 7fbcd12b5640 1 -- 172.21.15.44:0/2867300571 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc402ddd0 con 0x7fbccc0ce2e0 2024-04-02T13:23:25.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.910+0000 7fbcd12b5640 1 -- 172.21.15.44:0/2867300571 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4035610 con 0x7fbccc0ce2e0 2024-04-02T13:23:25.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.911+0000 7fbcd451b640 1 -- 172.21.15.44:0/2867300571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbccc0ce2e0 msgr2=0x7fbccc165820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:25.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.911+0000 7fbcd451b640 1 --2- 172.21.15.44:0/2867300571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbccc0ce2e0 0x7fbccc165820 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009e50 tx=0x7fbcc402d090 comp rx=0 tx=0).stop 2024-04-02T13:23:25.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.911+0000 7fbcd451b640 1 -- 172.21.15.44:0/2867300571 shutdown_connections 2024-04-02T13:23:25.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.911+0000 7fbcd451b640 1 --2- 172.21.15.44:0/2867300571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbccc0ce2e0 0x7fbccc165820 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.911+0000 7fbcd451b640 1 -- 172.21.15.44:0/2867300571 >> 172.21.15.44:0/2867300571 conn(0x7fbccc0cfcd0 msgr2=0x7fbccc0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:25.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.911+0000 7fbcd451b640 1 -- 172.21.15.44:0/2867300571 shutdown_connections 2024-04-02T13:23:25.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.911+0000 7fbcd451b640 1 -- 172.21.15.44:0/2867300571 wait complete. 2024-04-02T13:23:25.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.912+0000 7fbcd451b640 1 Processor -- start 2024-04-02T13:23:25.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.912+0000 7fbcd451b640 1 -- start start 2024-04-02T13:23:25.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd451b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbccc0b2d00 0x7fbccc14bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:25.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd451b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbccc0ce2e0 0x7fbccc152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:25.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd451b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbccc15aa00 0x7fbccc159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:25.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd451b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc0726d0 con 0x7fbccc15aa00 2024-04-02T13:23:25.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd451b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc0723f0 con 0x7fbccc0b2d00 2024-04-02T13:23:25.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd451b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbccc0cc750 con 0x7fbccc0ce2e0 2024-04-02T13:23:25.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd2ab8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbccc15aa00 0x7fbccc159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:25.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd2ab8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbccc15aa00 0x7fbccc159230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56292/0 (socket says 172.21.15.44:56292) 2024-04-02T13:23:25.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd2ab8640 1 -- 172.21.15.44:0/108441077 learned_addr learned my addr 172.21.15.44:0/108441077 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:25.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd22b7640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbccc0b2d00 0x7fbccc14bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:25.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd1ab6640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbccc0ce2e0 0x7fbccc152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:25.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd2ab8640 1 -- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbccc0ce2e0 msgr2=0x7fbccc152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:25.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd2ab8640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbccc0ce2e0 0x7fbccc152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd2ab8640 1 -- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbccc0b2d00 msgr2=0x7fbccc14bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:25.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd2ab8640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbccc0b2d00 0x7fbccc14bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:25.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd2ab8640 1 -- 172.21.15.44:0/108441077 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbccc151960 con 0x7fbccc15aa00 2024-04-02T13:23:25.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd22b7640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbccc0b2d00 0x7fbccc14bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:25.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd2ab8640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbccc15aa00 0x7fbccc159230 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4037040 tx=0x7fbcc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:25.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcc2ffd640 1 -- 172.21.15.44:0/108441077 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4009510 con 0x7fbccc15aa00 2024-04-02T13:23:25.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcc2ffd640 1 -- 172.21.15.44:0/108441077 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc4044070 con 0x7fbccc15aa00 2024-04-02T13:23:25.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcd451b640 1 -- 172.21.15.44:0/108441077 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbccc151c40 con 0x7fbccc15aa00 2024-04-02T13:23:25.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.913+0000 7fbcc2ffd640 1 -- 172.21.15.44:0/108441077 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc403f880 con 0x7fbccc15aa00 2024-04-02T13:23:25.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.914+0000 7fbcd451b640 1 -- 172.21.15.44:0/108441077 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbccc152200 con 0x7fbccc15aa00 2024-04-02T13:23:25.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.914+0000 7fbcd451b640 1 -- 172.21.15.44:0/108441077 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbccc074f20 con 0x7fbccc15aa00 2024-04-02T13:23:25.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.915+0000 7fbcc2ffd640 1 -- 172.21.15.44:0/108441077 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbcc402dc40 con 0x7fbccc15aa00 2024-04-02T13:23:25.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.915+0000 7fbcc2ffd640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcb006c920 0x7fbcb006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:25.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.915+0000 7fbcc2ffd640 1 -- 172.21.15.44:0/108441077 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7fbcc40ae3f0 con 0x7fbccc15aa00 2024-04-02T13:23:25.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.916+0000 7fbcd22b7640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcb006c920 0x7fbcb006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:25.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.916+0000 7fbcd22b7640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcb006c920 0x7fbcb006ede0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8001660 tx=0x7fbcc8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:25.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:25.920+0000 7fbcc2ffd640 1 -- 172.21.15.44:0/108441077 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbcc407c720 con 0x7fbccc15aa00 2024-04-02T13:23:26.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.104+0000 7fbcd451b640 1 -- 172.21.15.44:0/108441077 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbccc0d09a0 con 0x7fbccc15aa00 2024-04-02T13:23:26.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.105+0000 7fbcc2ffd640 1 -- 172.21.15.44:0/108441077 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) ==== 74+0+16849 (secure 0 0 0) 0x7fbcc4034e30 con 0x7fbccc15aa00 2024-04-02T13:23:26.107 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:26.107 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":64,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:23:24.803585+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:23:23.126712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-02T13:23:23.763593+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:23:26.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.107+0000 7fbcd451b640 1 -- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcb006c920 msgr2=0x7fbcb006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:26.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.107+0000 7fbcd451b640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcb006c920 0x7fbcb006ede0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8001660 tx=0x7fbcc8009040 comp rx=0 tx=0).stop 2024-04-02T13:23:26.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.107+0000 7fbcd451b640 1 -- 172.21.15.44:0/108441077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbccc15aa00 msgr2=0x7fbccc159230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:26.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.107+0000 7fbcd451b640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbccc15aa00 0x7fbccc159230 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4037040 tx=0x7fbcc4004a40 comp rx=0 tx=0).stop 2024-04-02T13:23:26.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.107+0000 7fbcd451b640 1 -- 172.21.15.44:0/108441077 shutdown_connections 2024-04-02T13:23:26.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.107+0000 7fbcd451b640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcb006c920 0x7fbcb006ede0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:26.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.107+0000 7fbcd451b640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbccc15aa00 0x7fbccc159230 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:26.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.107+0000 7fbcd451b640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbccc0ce2e0 0x7fbccc152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:26.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.107+0000 7fbcd451b640 1 --2- 172.21.15.44:0/108441077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbccc0b2d00 0x7fbccc14bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:26.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.107+0000 7fbcd451b640 1 -- 172.21.15.44:0/108441077 >> 172.21.15.44:0/108441077 conn(0x7fbccc0cfcd0 msgr2=0x7fbccc0cca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:26.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.107+0000 7fbcd451b640 1 -- 172.21.15.44:0/108441077 shutdown_connections 2024-04-02T13:23:26.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.107+0000 7fbcd451b640 1 -- 172.21.15.44:0/108441077 wait complete. 2024-04-02T13:23:26.120 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T13:23:26.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.222+0000 7f4384fc8640 1 Processor -- start 2024-04-02T13:23:26.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.222+0000 7f4384fc8640 1 -- start start 2024-04-02T13:23:26.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f4384fc8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f438015e550 0x7f438006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:26.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f4384fc8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f438015f040 0x7f43800d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:26.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f4384fc8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f438015fb50 con 0x7f438006d530 2024-04-02T13:23:26.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f4384fc8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f438006d970 con 0x7f438015f040 2024-04-02T13:23:26.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f4384fc8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f438006ca90 con 0x7f438015e550 2024-04-02T13:23:26.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f437ed9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f438015f040 0x7f43800d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:26.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f437ed9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f438015f040 0x7f43800d0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56302/0 (socket says 172.21.15.44:56302) 2024-04-02T13:23:26.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f437ed9d640 1 -- 172.21.15.44:0/3377530411 learned_addr learned my addr 172.21.15.44:0/3377530411 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:26.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f437e59c640 1 --2- 172.21.15.44:0/3377530411 >> v2:172.21.15.69:3300/0 conn(0x7f438015e550 0x7f438006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:26.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f437ed9d640 1 -- 172.21.15.44:0/3377530411 >> v2:172.21.15.69:3300/0 conn(0x7f438015e550 msgr2=0x7f438006cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:26.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f437ed9d640 1 --2- 172.21.15.44:0/3377530411 >> v2:172.21.15.69:3300/0 conn(0x7f438015e550 0x7f438006cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:26.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f437ed9d640 1 -- 172.21.15.44:0/3377530411 >> v1:172.21.15.69:6789/0 conn(0x7f438006d530 legacy=0x7f438015e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:26.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f437ed9d640 1 -- 172.21.15.44:0/3377530411 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f438015f590 con 0x7f438015f040 2024-04-02T13:23:26.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.223+0000 7f437e59c640 1 --2- 172.21.15.44:0/3377530411 >> v2:172.21.15.69:3300/0 conn(0x7f438015e550 0x7f438006cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:26.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.224+0000 7f437ed9d640 1 --2- 172.21.15.44:0/3377530411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f438015f040 0x7f43800d0ab0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f43800b49c0 tx=0x7f437802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5838d2a0943664fc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:26.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.224+0000 7f437dd9b640 1 -- 172.21.15.44:0/3377530411 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437802cbb0 con 0x7f438015f040 2024-04-02T13:23:26.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.224+0000 7f437dd9b640 1 -- 172.21.15.44:0/3377530411 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f437802cd50 con 0x7f438015f040 2024-04-02T13:23:26.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.224+0000 7f437dd9b640 1 -- 172.21.15.44:0/3377530411 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4378034610 con 0x7f438015f040 2024-04-02T13:23:26.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.225+0000 7f4384fc8640 1 -- 172.21.15.44:0/3377530411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f438015f040 msgr2=0x7f43800d0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:26.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.225+0000 7f4384fc8640 1 --2- 172.21.15.44:0/3377530411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f438015f040 0x7f43800d0ab0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f43800b49c0 tx=0x7f437802c010 comp rx=0 tx=0).stop 2024-04-02T13:23:26.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.225+0000 7f4384fc8640 1 -- 172.21.15.44:0/3377530411 shutdown_connections 2024-04-02T13:23:26.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.225+0000 7f4384fc8640 1 --2- 172.21.15.44:0/3377530411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f438015f040 0x7f43800d0ab0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.225+0000 7f4384fc8640 1 --2- 172.21.15.44:0/3377530411 >> v2:172.21.15.69:3300/0 conn(0x7f438015e550 0x7f438006cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.225+0000 7f4384fc8640 1 -- 172.21.15.44:0/3377530411 >> 172.21.15.44:0/3377530411 conn(0x7f438009aa20 msgr2=0x7f438009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.225+0000 7f4384fc8640 1 -- 172.21.15.44:0/3377530411 shutdown_connections 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.225+0000 7f4384fc8640 1 -- 172.21.15.44:0/3377530411 wait complete. 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.226+0000 7f4384fc8640 1 Processor -- start 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.226+0000 7f4384fc8640 1 -- start start 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f4384fc8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f438006d530 0x7f43801f1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f4384fc8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f438015e550 0x7f43801f46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f4384fc8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f438015f040 0x7f43801f4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f4384fc8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f43800d0590 con 0x7f438006d530 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f4384fc8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f43800d02b0 con 0x7f438015e550 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f4384fc8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f43800cc4b0 con 0x7f438015f040 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f437e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f438006d530 0x7f43801f1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f437e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f438006d530 0x7f43801f1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56318/0 (socket says 172.21.15.44:56318) 2024-04-02T13:23:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f437e59c640 1 -- 172.21.15.44:0/2395559787 learned_addr learned my addr 172.21.15.44:0/2395559787 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f4377fff640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f438015e550 0x7f43801f46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f437e59c640 1 -- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f438015f040 msgr2=0x7f43801f4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f437ed9d640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f438015f040 0x7f43801f4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f437e59c640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f438015f040 0x7f43801f4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f437e59c640 1 -- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f438015e550 msgr2=0x7f43801f46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f437e59c640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f438015e550 0x7f43801f46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f437e59c640 1 -- 172.21.15.44:0/2395559787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43801f4c30 con 0x7f438006d530 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f4377fff640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f438015e550 0x7f43801f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f437e59c640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f438006d530 0x7f43801f1190 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f437000d7d0 tx=0x7f437000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f4376ffd640 1 -- 172.21.15.44:0/2395559787 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4370004030 con 0x7f438006d530 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.227+0000 7f4376ffd640 1 -- 172.21.15.44:0/2395559787 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43700041d0 con 0x7f438006d530 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.228+0000 7f4384fc8640 1 -- 172.21.15.44:0/2395559787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43800c9f80 con 0x7f438006d530 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.228+0000 7f4376ffd640 1 -- 172.21.15.44:0/2395559787 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437000b6c0 con 0x7f438006d530 2024-04-02T13:23:26.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.228+0000 7f4384fc8640 1 -- 172.21.15.44:0/2395559787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43800ca500 con 0x7f438006d530 2024-04-02T13:23:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.228+0000 7f4384fc8640 1 -- 172.21.15.44:0/2395559787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43800684f0 con 0x7f438006d530 2024-04-02T13:23:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.229+0000 7f4376ffd640 1 -- 172.21.15.44:0/2395559787 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f437000b860 con 0x7f438006d530 2024-04-02T13:23:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.229+0000 7f4376ffd640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f436006c920 0x7f436006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.230+0000 7f4376ffd640 1 -- 172.21.15.44:0/2395559787 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f437008c070 con 0x7f438006d530 2024-04-02T13:23:26.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.230+0000 7f4377fff640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f436006c920 0x7f436006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:26.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.230+0000 7f4377fff640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f436006c920 0x7f436006ede0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f43800b7f10 tx=0x7f4368009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:26.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.234+0000 7f4376ffd640 1 -- 172.21.15.44:0/2395559787 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f43801f4c30 con 0x7f438006d530 2024-04-02T13:23:26.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:26.461+0000 7f4384fc8640 1 -- 172.21.15.44:0/2395559787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f438015ec40 con 0x7f438006d530 2024-04-02T13:23:27.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-04-02T13:23:27.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.463+0000 7f4376ffd640 1 -- 172.21.15.44:0/2395559787 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v34) ==== 118+0+0 (secure 0 0 0) 0x7f437005a270 con 0x7f438006d530 2024-04-02T13:23:27.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.466+0000 7f4384fc8640 1 -- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f436006c920 msgr2=0x7f436006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:27.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.466+0000 7f4384fc8640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f436006c920 0x7f436006ede0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f43800b7f10 tx=0x7f4368009040 comp rx=0 tx=0).stop 2024-04-02T13:23:27.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.466+0000 7f4384fc8640 1 -- 172.21.15.44:0/2395559787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f438006d530 msgr2=0x7f43801f1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:27.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.466+0000 7f4384fc8640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f438006d530 0x7f43801f1190 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f437000d7d0 tx=0x7f437000dc90 comp rx=0 tx=0).stop 2024-04-02T13:23:27.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.466+0000 7f4384fc8640 1 -- 172.21.15.44:0/2395559787 shutdown_connections 2024-04-02T13:23:27.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.466+0000 7f4384fc8640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f436006c920 0x7f436006ede0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:27.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.466+0000 7f4384fc8640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f438015f040 0x7f43801f4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:27.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.466+0000 7f4384fc8640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f438015e550 0x7f43801f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:27.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.466+0000 7f4384fc8640 1 --2- 172.21.15.44:0/2395559787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f438006d530 0x7f43801f1190 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:27.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.466+0000 7f4384fc8640 1 -- 172.21.15.44:0/2395559787 >> 172.21.15.44:0/2395559787 conn(0x7f438009aa20 msgr2=0x7f43800cd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:27.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.466+0000 7f4384fc8640 1 -- 172.21.15.44:0/2395559787 shutdown_connections 2024-04-02T13:23:27.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.466+0000 7f4384fc8640 1 -- 172.21.15.44:0/2395559787 wait complete. 2024-04-02T13:23:27.481 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T13:23:27.481 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T13:23:27.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.591+0000 7f532bdd6640 1 Processor -- start 2024-04-02T13:23:27.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.591+0000 7f532bdd6640 1 -- start start 2024-04-02T13:23:27.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.592+0000 7f532bdd6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532406d700 0x7f532415e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:27.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.592+0000 7f532bdd6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f532415ee70 0x7f53240d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:27.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.592+0000 7f532bdd6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f532415fb50 con 0x7f532415e550 2024-04-02T13:23:27.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.592+0000 7f532bdd6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f532406ca90 con 0x7f532415ee70 2024-04-02T13:23:27.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.592+0000 7f532bdd6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f532415f590 con 0x7f532406d700 2024-04-02T13:23:27.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.592+0000 7f532a373640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f532415ee70 0x7f53240d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:27.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.592+0000 7f5329371640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532406d700 0x7f532415e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:27.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.592+0000 7f5329b72640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f532415e550 0x7f532406cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:47170/0 (socket says 172.21.15.44:47170) 2024-04-02T13:23:27.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.592+0000 7f5329b72640 1 -- 172.21.15.44:0/3009484043 learned_addr learned my addr 172.21.15.44:0/3009484043 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:27.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.592+0000 7f532a373640 1 -- 172.21.15.44:0/3009484043 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532406d700 msgr2=0x7f532415e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:27.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.592+0000 7f532a373640 1 --2- 172.21.15.44:0/3009484043 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532406d700 0x7f532415e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:27.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.592+0000 7f532a373640 1 -- 172.21.15.44:0/3009484043 >> v1:172.21.15.69:6789/0 conn(0x7f532415e550 legacy=0x7f532406cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:27.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.592+0000 7f532a373640 1 -- 172.21.15.44:0/3009484043 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f532415f320 con 0x7f532415ee70 2024-04-02T13:23:27.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.593+0000 7f532a373640 1 --2- 172.21.15.44:0/3009484043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532415ee70 0x7f53240d0ab0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f531c009d90 tx=0x7f531c02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffaba4c43cf206b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:27.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.593+0000 7f5328b70640 1 -- 172.21.15.44:0/3009484043 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c039070 con 0x7f532415ee70 2024-04-02T13:23:27.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.593+0000 7f5328b70640 1 -- 172.21.15.44:0/3009484043 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f531c02cce0 con 0x7f532415ee70 2024-04-02T13:23:27.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.593+0000 7f5328b70640 1 -- 172.21.15.44:0/3009484043 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c034660 con 0x7f532415ee70 2024-04-02T13:23:27.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.594+0000 7f532bdd6640 1 -- 172.21.15.44:0/3009484043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532415ee70 msgr2=0x7f53240d0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:27.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.594+0000 7f532bdd6640 1 --2- 172.21.15.44:0/3009484043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532415ee70 0x7f53240d0ab0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f531c009d90 tx=0x7f531c02bf30 comp rx=0 tx=0).stop 2024-04-02T13:23:27.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.594+0000 7f532bdd6640 1 -- 172.21.15.44:0/3009484043 shutdown_connections 2024-04-02T13:23:27.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.594+0000 7f532bdd6640 1 --2- 172.21.15.44:0/3009484043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532415ee70 0x7f53240d0ab0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:27.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.594+0000 7f532bdd6640 1 --2- 172.21.15.44:0/3009484043 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532406d700 0x7f532415e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.594+0000 7f532bdd6640 1 -- 172.21.15.44:0/3009484043 >> 172.21.15.44:0/3009484043 conn(0x7f532409aa20 msgr2=0x7f532409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.594+0000 7f532bdd6640 1 -- 172.21.15.44:0/3009484043 shutdown_connections 2024-04-02T13:23:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.594+0000 7f532bdd6640 1 -- 172.21.15.44:0/3009484043 wait complete. 2024-04-02T13:23:27.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.595+0000 7f532bdd6640 1 Processor -- start 2024-04-02T13:23:27.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.595+0000 7f532bdd6640 1 -- start start 2024-04-02T13:23:27.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532bdd6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532406d700 0x7f53241f3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:27.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532bdd6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532415e550 0x7f53241f5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:27.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532bdd6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f532415ee70 0x7f532415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:27.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532bdd6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f53240d0590 con 0x7f532415ee70 2024-04-02T13:23:27.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532bdd6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f53240d02b0 con 0x7f532415e550 2024-04-02T13:23:27.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532bdd6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f53240cc4b0 con 0x7f532406d700 2024-04-02T13:23:27.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532a373640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f532415ee70 0x7f532415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:27.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532a373640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f532415ee70 0x7f532415b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56322/0 (socket says 172.21.15.44:56322) 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532a373640 1 -- 172.21.15.44:0/57583000 learned_addr learned my addr 172.21.15.44:0/57583000 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f5329b72640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532406d700 0x7f53241f3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f5329371640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532415e550 0x7f53241f5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532a373640 1 -- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532406d700 msgr2=0x7f53241f3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532a373640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532406d700 0x7f53241f3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532a373640 1 -- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532415e550 msgr2=0x7f53241f5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532a373640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532415e550 0x7f53241f5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.596+0000 7f532a373640 1 -- 172.21.15.44:0/57583000 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f532409d6e0 con 0x7f532415ee70 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.597+0000 7f532a373640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f532415ee70 0x7f532415b1d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f531c009d90 tx=0x7f531c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.597+0000 7f531a7fc640 1 -- 172.21.15.44:0/57583000 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c039050 con 0x7f532415ee70 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.597+0000 7f531a7fc640 1 -- 172.21.15.44:0/57583000 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f531c045070 con 0x7f532415ee70 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.597+0000 7f532bdd6640 1 -- 172.21.15.44:0/57583000 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53241f57c0 con 0x7f532415ee70 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.597+0000 7f531a7fc640 1 -- 172.21.15.44:0/57583000 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c009450 con 0x7f532415ee70 2024-04-02T13:23:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.597+0000 7f532bdd6640 1 -- 172.21.15.44:0/57583000 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53241f5d80 con 0x7f532415ee70 2024-04-02T13:23:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.598+0000 7f532bdd6640 1 -- 172.21.15.44:0/57583000 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53240684f0 con 0x7f532415ee70 2024-04-02T13:23:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.598+0000 7f531a7fc640 1 -- 172.21.15.44:0/57583000 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f531c03e850 con 0x7f532415ee70 2024-04-02T13:23:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.599+0000 7f531a7fc640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f530806c8f0 0x7f530806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.599+0000 7f531a7fc640 1 -- 172.21.15.44:0/57583000 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f531c0ae930 con 0x7f532415ee70 2024-04-02T13:23:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.599+0000 7f5329b72640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f530806c8f0 0x7f530806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.599+0000 7f5329b72640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f530806c8f0 0x7f530806edb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f532406ca20 tx=0x7f5320007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:27.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.603+0000 7f531a7fc640 1 -- 172.21.15.44:0/57583000 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f531c07c6a0 con 0x7f532415ee70 2024-04-02T13:23:27.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:27.830+0000 7f532bdd6640 1 -- 172.21.15.44:0/57583000 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f53240cf0a0 con 0x7f532415ee70 2024-04-02T13:23:28.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.465+0000 7f531a7fc640 1 -- 172.21.15.44:0/57583000 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v35) ==== 122+0+0 (secure 0 0 0) 0x7f531c0809a0 con 0x7f532415ee70 2024-04-02T13:23:28.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.468+0000 7f532bdd6640 1 -- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f530806c8f0 msgr2=0x7f530806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:28.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.468+0000 7f532bdd6640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f530806c8f0 0x7f530806edb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f532406ca20 tx=0x7f5320007e80 comp rx=0 tx=0).stop 2024-04-02T13:23:28.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.468+0000 7f532bdd6640 1 -- 172.21.15.44:0/57583000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f532415ee70 msgr2=0x7f532415b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:28.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.468+0000 7f532bdd6640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f532415ee70 0x7f532415b1d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f531c009d90 tx=0x7f531c004a40 comp rx=0 tx=0).stop 2024-04-02T13:23:28.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.468+0000 7f532bdd6640 1 -- 172.21.15.44:0/57583000 shutdown_connections 2024-04-02T13:23:28.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.468+0000 7f532bdd6640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f530806c8f0 0x7f530806edb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.468+0000 7f532bdd6640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f532415ee70 0x7f532415b1d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.468+0000 7f532bdd6640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532415e550 0x7f53241f5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.468+0000 7f532bdd6640 1 --2- 172.21.15.44:0/57583000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532406d700 0x7f53241f3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.468+0000 7f532bdd6640 1 -- 172.21.15.44:0/57583000 >> 172.21.15.44:0/57583000 conn(0x7f532409aa20 msgr2=0x7f53240cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:28.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.468+0000 7f532bdd6640 1 -- 172.21.15.44:0/57583000 shutdown_connections 2024-04-02T13:23:28.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.468+0000 7f532bdd6640 1 -- 172.21.15.44:0/57583000 wait complete. 2024-04-02T13:23:28.484 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T13:23:28.484 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:23:28.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.610+0000 7fcc0259d640 1 Processor -- start 2024-04-02T13:23:28.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.610+0000 7fcc0259d640 1 -- start start 2024-04-02T13:23:28.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.610+0000 7fcc0259d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcbfc06d700 0x7fcbfc15e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.610+0000 7fcc0259d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbfc15ee70 0x7fcbfc0d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.610+0000 7fcc0259d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbfc15fb50 con 0x7fcbfc15e550 2024-04-02T13:23:28.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.610+0000 7fcc0259d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcbfc06ca90 con 0x7fcbfc06d700 2024-04-02T13:23:28.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.610+0000 7fcc0259d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcbfc15f590 con 0x7fcbfc15ee70 2024-04-02T13:23:28.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.610+0000 7fcbfbfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fcbfc15e550 0x7fcbfc06cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:58048/0 (socket says 172.21.15.44:58048) 2024-04-02T13:23:28.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.610+0000 7fcbfbfff640 1 -- 172.21.15.44:0/2202406070 learned_addr learned my addr 172.21.15.44:0/2202406070 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:28.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfb7fe640 1 --2- 172.21.15.44:0/2202406070 >> v2:172.21.15.69:3300/0 conn(0x7fcbfc06d700 0x7fcbfc15e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcc00b3a640 1 --2- 172.21.15.44:0/2202406070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbfc15ee70 0x7fcbfc0d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1803198348 0 0) 0x7fcbfc15fb50 con 0x7fcbfc15e550 2024-04-02T13:23:28.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbe4003520 con 0x7fcbfc15e550 2024-04-02T13:23:28.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3492090660 0 0) 0x7fcbe4003520 con 0x7fcbfc15e550 2024-04-02T13:23:28.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbfc15ee70 msgr2=0x7fcbfc0d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:28.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 --2- 172.21.15.44:0/2202406070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbfc15ee70 0x7fcbfc0d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 >> v2:172.21.15.69:3300/0 conn(0x7fcbfc06d700 msgr2=0x7fcbfc15e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 --2- 172.21.15.44:0/2202406070 >> v2:172.21.15.69:3300/0 conn(0x7fcbfc06d700 0x7fcbfc15e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc15f320 con 0x7fcbfc15e550 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fcbf0001670 con 0x7fcbfc15e550 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 >> v1:172.21.15.44:6789/0 conn(0x7fcbfc15e550 legacy=0x7fcbfc06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 --2- 172.21.15.44:0/2202406070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbe40039b0 0x7fcbe4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 --2- 172.21.15.44:0/2202406070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbfc15ee70 0x7fcbe4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 --2- 172.21.15.44:0/2202406070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbe4004d40 0x7fcbe40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc06ca90 con 0x7fcbe40039b0 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc15f590 con 0x7fcbe4004d40 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe400b9c0 con 0x7fcbfc15ee70 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfbfff640 1 --2- 172.21.15.44:0/2202406070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbe40039b0 0x7fcbe4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfb7fe640 1 --2- 172.21.15.44:0/2202406070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbfc15ee70 0x7fcbe4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcc00b3a640 1 --2- 172.21.15.44:0/2202406070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbe4004d40 0x7fcbe40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfbfff640 1 -- 172.21.15.44:0/2202406070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbfc15ee70 msgr2=0x7fcbe4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:28.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.611+0000 7fcbfbfff640 1 --2- 172.21.15.44:0/2202406070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbfc15ee70 0x7fcbe4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.612+0000 7fcbfbfff640 1 -- 172.21.15.44:0/2202406070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbe4004d40 msgr2=0x7fcbe40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:28.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.612+0000 7fcbfbfff640 1 --2- 172.21.15.44:0/2202406070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbe4004d40 0x7fcbe40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.612+0000 7fcbfbfff640 1 -- 172.21.15.44:0/2202406070 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe4003520 con 0x7fcbe40039b0 2024-04-02T13:23:28.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.612+0000 7fcbfbfff640 1 --2- 172.21.15.44:0/2202406070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbe40039b0 0x7fcbe4003dd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0001050 tx=0x7fcbe4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661d71549582b0ea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:28.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.612+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0007d70 con 0x7fcbe40039b0 2024-04-02T13:23:28.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.612+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbf0030070 con 0x7fcbe40039b0 2024-04-02T13:23:28.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.612+0000 7fcbfaffd640 1 -- 172.21.15.44:0/2202406070 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf002cd80 con 0x7fcbe40039b0 2024-04-02T13:23:28.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.613+0000 7fcc0259d640 1 -- 172.21.15.44:0/2202406070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbe40039b0 msgr2=0x7fcbe4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:28.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.613+0000 7fcc0259d640 1 --2- 172.21.15.44:0/2202406070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbe40039b0 0x7fcbe4003dd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0001050 tx=0x7fcbe4005830 comp rx=0 tx=0).stop 2024-04-02T13:23:28.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.613+0000 7fcc00b3a640 1 -- 172.21.15.44:0/2202406070 reap_dead start 2024-04-02T13:23:28.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.613+0000 7fcc0259d640 1 -- 172.21.15.44:0/2202406070 shutdown_connections 2024-04-02T13:23:28.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.613+0000 7fcc0259d640 1 -- 172.21.15.44:0/2202406070 >> 172.21.15.44:0/2202406070 conn(0x7fcbfc09aa20 msgr2=0x7fcbfc09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:28.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.613+0000 7fcc0259d640 1 -- 172.21.15.44:0/2202406070 shutdown_connections 2024-04-02T13:23:28.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.613+0000 7fcc0259d640 1 -- 172.21.15.44:0/2202406070 wait complete. 2024-04-02T13:23:28.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.614+0000 7fcc0259d640 1 Processor -- start 2024-04-02T13:23:28.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.614+0000 7fcc0259d640 1 -- start start 2024-04-02T13:23:28.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.615+0000 7fcc0259d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbe40039b0 0x7fcbfc15b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.615+0000 7fcc0259d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbfc15bf90 0x7fcbfc0b6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.615+0000 7fcc0259d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbfc070930 0x7fcbfc0b7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.615+0000 7fcc0259d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc0d0590 con 0x7fcbe40039b0 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.615+0000 7fcc0259d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc0d02b0 con 0x7fcbfc070930 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.615+0000 7fcc0259d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcbfc0cc4b0 con 0x7fcbfc15bf90 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.615+0000 7fcbfbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbe40039b0 0x7fcbfc15b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.615+0000 7fcbfbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbe40039b0 0x7fcbfc15b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56346/0 (socket says 172.21.15.44:56346) 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.615+0000 7fcbfbfff640 1 -- 172.21.15.44:0/3849508478 learned_addr learned my addr 172.21.15.44:0/3849508478 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.615+0000 7fcc00b3a640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbfc070930 0x7fcbfc0b7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.615+0000 7fcbfb7fe640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbfc15bf90 0x7fcbfc0b6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.616+0000 7fcbfbfff640 1 -- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbfc15bf90 msgr2=0x7fcbfc0b6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.616+0000 7fcbfbfff640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbfc15bf90 0x7fcbfc0b6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.616+0000 7fcbfbfff640 1 -- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbfc070930 msgr2=0x7fcbfc0b7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.616+0000 7fcbfbfff640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbfc070930 0x7fcbfc0b7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.616+0000 7fcbfbfff640 1 -- 172.21.15.44:0/3849508478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc09ccd0 con 0x7fcbe40039b0 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.616+0000 7fcbfbfff640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbe40039b0 0x7fcbfc15b950 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fcbf002c590 tx=0x7fcbf0000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:28.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.616+0000 7fcbf8ff9640 1 -- 172.21.15.44:0/3849508478 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0004bf0 con 0x7fcbe40039b0 2024-04-02T13:23:28.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.616+0000 7fcbf8ff9640 1 -- 172.21.15.44:0/3849508478 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbf0004d90 con 0x7fcbe40039b0 2024-04-02T13:23:28.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.616+0000 7fcc0259d640 1 -- 172.21.15.44:0/3849508478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbfc0b5850 con 0x7fcbe40039b0 2024-04-02T13:23:28.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.616+0000 7fcbf8ff9640 1 -- 172.21.15.44:0/3849508478 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0007270 con 0x7fcbe40039b0 2024-04-02T13:23:28.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.616+0000 7fcc0259d640 1 -- 172.21.15.44:0/3849508478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbfc0ca120 con 0x7fcbe40039b0 2024-04-02T13:23:28.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.617+0000 7fcc0259d640 1 -- 172.21.15.44:0/3849508478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbfc0b5d70 con 0x7fcbe40039b0 2024-04-02T13:23:28.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.617+0000 7fcbf8ff9640 1 -- 172.21.15.44:0/3849508478 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcbf0047020 con 0x7fcbe40039b0 2024-04-02T13:23:28.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.618+0000 7fcbf8ff9640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcbe806c8f0 0x7fcbe806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.618+0000 7fcbf8ff9640 1 -- 172.21.15.44:0/3849508478 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fcbf00acda0 con 0x7fcbe40039b0 2024-04-02T13:23:28.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.618+0000 7fcbfb7fe640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcbe806c8f0 0x7fcbe806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.618+0000 7fcbfb7fe640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcbe806c8f0 0x7fcbe806edb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0b32c0 tx=0x7fcbec009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:28.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.622+0000 7fcbf8ff9640 1 -- 172.21.15.44:0/3849508478 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fcbfc0b5d70 con 0x7fcbe40039b0 2024-04-02T13:23:28.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.846+0000 7fcc0259d640 1 -- 172.21.15.44:0/3849508478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbfc0d1c60 con 0x7fcbe40039b0 2024-04-02T13:23:28.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.847+0000 7fcbf8ff9640 1 -- 172.21.15.44:0/3849508478 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5179 (secure 0 0 0) 0x7fcbfc09ccd0 con 0x7fcbe40039b0 2024-04-02T13:23:28.849 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:28.849 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1535287484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1535287484},{"type":"v1","addr":"172.21.15.69:6836","nonce":1535287484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6429,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2367107002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2367107002},{"type":"v1","addr":"172.21.15.44:6833","nonce":2367107002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/553542317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":553542317},{"type":"v1","addr":"172.21.15.69:6837","nonce":553542317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:23:24.803884+0000","modified":"2024-04-02T13:23:27.831218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6450":{"gid":6450,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/3127613802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3127613802},{"type":"v1","addr":"172.21.15.44:6835","nonce":3127613802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6450,"qdb_cluster":[6450]},"id":4}]} 2024-04-02T13:23:28.849 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 35 2024-04-02T13:23:28.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.849+0000 7fcc0259d640 1 -- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcbe806c8f0 msgr2=0x7fcbe806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:28.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.849+0000 7fcc0259d640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcbe806c8f0 0x7fcbe806edb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0b32c0 tx=0x7fcbec009190 comp rx=0 tx=0).stop 2024-04-02T13:23:28.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.849+0000 7fcc0259d640 1 -- 172.21.15.44:0/3849508478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbe40039b0 msgr2=0x7fcbfc15b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:28.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.849+0000 7fcc0259d640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbe40039b0 0x7fcbfc15b950 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fcbf002c590 tx=0x7fcbf0000fe0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.849+0000 7fcc0259d640 1 -- 172.21.15.44:0/3849508478 shutdown_connections 2024-04-02T13:23:28.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.849+0000 7fcc0259d640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcbe806c8f0 0x7fcbe806edb0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.849+0000 7fcc0259d640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbfc070930 0x7fcbfc0b7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.849+0000 7fcc0259d640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbfc15bf90 0x7fcbfc0b6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.849+0000 7fcc0259d640 1 --2- 172.21.15.44:0/3849508478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbe40039b0 0x7fcbfc15b950 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.849+0000 7fcc0259d640 1 -- 172.21.15.44:0/3849508478 >> 172.21.15.44:0/3849508478 conn(0x7fcbfc09aa20 msgr2=0x7fcbfc0cdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:28.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.849+0000 7fcc0259d640 1 -- 172.21.15.44:0/3849508478 shutdown_connections 2024-04-02T13:23:28.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.849+0000 7fcc0259d640 1 -- 172.21.15.44:0/3849508478 wait complete. 2024-04-02T13:23:28.862 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:23:28.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.962+0000 7f5cd3c60640 1 Processor -- start 2024-04-02T13:23:28.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.962+0000 7f5cd3c60640 1 -- start start 2024-04-02T13:23:28.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd3c60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ccc0cece0 0x7f5ccc0cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd3c60640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5ccc0d09f0 0x7f5ccc06d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd3c60640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ccc06ece0 0x7f5ccc15eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd3c60640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f5ccc0cf600 con 0x7f5ccc0d09f0 2024-04-02T13:23:28.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd3c60640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5ccc06dc40 con 0x7f5ccc06ece0 2024-04-02T13:23:28.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd3c60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5ccc06df20 con 0x7f5ccc0cece0 2024-04-02T13:23:28.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd11fb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5ccc0d09f0 0x7f5ccc06d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd11fb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5ccc0d09f0 0x7f5ccc06d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56354/0 (socket says 172.21.15.44:56354) 2024-04-02T13:23:28.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd11fb640 1 -- 172.21.15.44:0/3893142904 learned_addr learned my addr 172.21.15.44:0/3893142904 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:28.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd21fd640 1 --2- 172.21.15.44:0/3893142904 >> v2:172.21.15.69:3300/0 conn(0x7f5ccc06ece0 0x7f5ccc15eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd19fc640 1 --2- 172.21.15.44:0/3893142904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ccc0cece0 0x7f5ccc0cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd11fb640 1 -- 172.21.15.44:0/3893142904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ccc0cece0 msgr2=0x7f5ccc0cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:28.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd11fb640 1 --2- 172.21.15.44:0/3893142904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ccc0cece0 0x7f5ccc0cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd11fb640 1 -- 172.21.15.44:0/3893142904 >> v2:172.21.15.69:3300/0 conn(0x7f5ccc06ece0 msgr2=0x7f5ccc15eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:28.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd11fb640 1 --2- 172.21.15.44:0/3893142904 >> v2:172.21.15.69:3300/0 conn(0x7f5ccc06ece0 0x7f5ccc15eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.963+0000 7f5cd11fb640 1 -- 172.21.15.44:0/3893142904 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ccc06e200 con 0x7f5ccc0d09f0 2024-04-02T13:23:28.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.964+0000 7f5cd11fb640 1 --2- 172.21.15.44:0/3893142904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ccc0d09f0 0x7f5ccc06d2d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0cf7e0 tx=0x7f5cc002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15816b87f1700506 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:28.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.964+0000 7f5cd09fa640 1 -- 172.21.15.44:0/3893142904 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0035020 con 0x7f5ccc0d09f0 2024-04-02T13:23:28.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.964+0000 7f5cd09fa640 1 -- 172.21.15.44:0/3893142904 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cc0031640 con 0x7f5ccc0d09f0 2024-04-02T13:23:28.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.964+0000 7f5cd09fa640 1 -- 172.21.15.44:0/3893142904 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc002d040 con 0x7f5ccc0d09f0 2024-04-02T13:23:28.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.965+0000 7f5cd3c60640 1 -- 172.21.15.44:0/3893142904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ccc0d09f0 msgr2=0x7f5ccc06d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:28.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.965+0000 7f5cd3c60640 1 --2- 172.21.15.44:0/3893142904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ccc0d09f0 0x7f5ccc06d2d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0cf7e0 tx=0x7f5cc002b8f0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.965+0000 7f5cd3c60640 1 -- 172.21.15.44:0/3893142904 shutdown_connections 2024-04-02T13:23:28.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.965+0000 7f5cd3c60640 1 --2- 172.21.15.44:0/3893142904 >> v2:172.21.15.69:3300/0 conn(0x7f5ccc06ece0 0x7f5ccc15eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.965+0000 7f5cd3c60640 1 --2- 172.21.15.44:0/3893142904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ccc0d09f0 0x7f5ccc06d2d0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.965+0000 7f5cd3c60640 1 --2- 172.21.15.44:0/3893142904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ccc0cece0 0x7f5ccc0cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.965+0000 7f5cd3c60640 1 -- 172.21.15.44:0/3893142904 >> 172.21.15.44:0/3893142904 conn(0x7f5ccc0cfcd0 msgr2=0x7f5ccc0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:28.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.965+0000 7f5cd3c60640 1 -- 172.21.15.44:0/3893142904 shutdown_connections 2024-04-02T13:23:28.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.965+0000 7f5cd3c60640 1 -- 172.21.15.44:0/3893142904 wait complete. 2024-04-02T13:23:28.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.966+0000 7f5cd3c60640 1 Processor -- start 2024-04-02T13:23:28.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.966+0000 7f5cd3c60640 1 -- start start 2024-04-02T13:23:28.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd3c60640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ccc06ece0 0x7f5ccc09d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd3c60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ccc0cece0 0x7f5ccc153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd3c60640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc0d09f0 0x7f5ccc152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd3c60640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc06d810 con 0x7f5ccc06ece0 2024-04-02T13:23:28.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd3c60640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc06e7c0 con 0x7f5ccc0d09f0 2024-04-02T13:23:28.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd3c60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5ccc06d990 con 0x7f5ccc0cece0 2024-04-02T13:23:28.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd19fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ccc06ece0 0x7f5ccc09d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd19fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ccc06ece0 0x7f5ccc09d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56362/0 (socket says 172.21.15.44:56362) 2024-04-02T13:23:28.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd19fc640 1 -- 172.21.15.44:0/2756047737 learned_addr learned my addr 172.21.15.44:0/2756047737 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:28.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd21fd640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc0d09f0 0x7f5ccc152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd11fb640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ccc0cece0 0x7f5ccc153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd19fc640 1 -- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ccc0cece0 msgr2=0x7f5ccc153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:28.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd19fc640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ccc0cece0 0x7f5ccc153e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd19fc640 1 -- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc0d09f0 msgr2=0x7f5ccc152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd19fc640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc0d09f0 0x7f5ccc152bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.967+0000 7f5cd19fc640 1 -- 172.21.15.44:0/2756047737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ccc154380 con 0x7f5ccc06ece0 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.968+0000 7f5cd19fc640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ccc06ece0 0x7f5ccc09d040 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400e9e0 tx=0x7f5cc400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.968+0000 7f5cca7fc640 1 -- 172.21.15.44:0/2756047737 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4017070 con 0x7f5ccc06ece0 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.968+0000 7f5cca7fc640 1 -- 172.21.15.44:0/2756047737 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cc400c9d0 con 0x7f5ccc06ece0 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.968+0000 7f5cd3c60640 1 -- 172.21.15.44:0/2756047737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ccc154630 con 0x7f5ccc06ece0 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.968+0000 7f5cca7fc640 1 -- 172.21.15.44:0/2756047737 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc400cdc0 con 0x7f5ccc06ece0 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.968+0000 7f5cd3c60640 1 -- 172.21.15.44:0/2756047737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ccc154bf0 con 0x7f5ccc06ece0 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.969+0000 7f5cd3c60640 1 -- 172.21.15.44:0/2756047737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ccc074f20 con 0x7f5ccc06ece0 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.969+0000 7f5cca7fc640 1 -- 172.21.15.44:0/2756047737 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5cc401a070 con 0x7f5ccc06ece0 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.970+0000 7f5cca7fc640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cb806c8f0 0x7f5cb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.970+0000 7f5cca7fc640 1 -- 172.21.15.44:0/2756047737 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f5cc408c090 con 0x7f5ccc06ece0 2024-04-02T13:23:28.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.970+0000 7f5cd11fb640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cb806c8f0 0x7f5cb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:28.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.970+0000 7f5cd11fb640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cb806c8f0 0x7f5cb806edb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0036000 tx=0x7f5cc003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:28.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:28.974+0000 7f5cca7fc640 1 -- 172.21.15.44:0/2756047737 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5cc405a330 con 0x7f5ccc06ece0 2024-04-02T13:23:29.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.159+0000 7f5cd3c60640 1 -- 172.21.15.44:0/2756047737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ccc0684f0 con 0x7f5ccc06ece0 2024-04-02T13:23:29.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.161+0000 7f5cca7fc640 1 -- 172.21.15.44:0/2756047737 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) ==== 74+0+16836 (secure 0 0 0) 0x7f5cc4059c30 con 0x7f5ccc06ece0 2024-04-02T13:23:29.163 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:29.163 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":65,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:23:26.455136+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:23:23.126712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-02T13:23:23.763593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:23:29.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.162+0000 7f5cd3c60640 1 -- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cb806c8f0 msgr2=0x7f5cb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.162+0000 7f5cd3c60640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cb806c8f0 0x7f5cb806edb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0036000 tx=0x7f5cc003c040 comp rx=0 tx=0).stop 2024-04-02T13:23:29.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.162+0000 7f5cd3c60640 1 -- 172.21.15.44:0/2756047737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ccc06ece0 msgr2=0x7f5ccc09d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.162+0000 7f5cd3c60640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ccc06ece0 0x7f5ccc09d040 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400e9e0 tx=0x7f5cc400eea0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.163+0000 7f5cd3c60640 1 -- 172.21.15.44:0/2756047737 shutdown_connections 2024-04-02T13:23:29.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.163+0000 7f5cd3c60640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5cb806c8f0 0x7f5cb806edb0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.163+0000 7f5cd3c60640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc0d09f0 0x7f5ccc152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.163+0000 7f5cd3c60640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ccc0cece0 0x7f5ccc153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.163+0000 7f5cd3c60640 1 --2- 172.21.15.44:0/2756047737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ccc06ece0 0x7f5ccc09d040 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.163+0000 7f5cd3c60640 1 -- 172.21.15.44:0/2756047737 >> 172.21.15.44:0/2756047737 conn(0x7f5ccc0cfcd0 msgr2=0x7f5ccc0b0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:29.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.163+0000 7f5cd3c60640 1 -- 172.21.15.44:0/2756047737 shutdown_connections 2024-04-02T13:23:29.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.163+0000 7f5cd3c60640 1 -- 172.21.15.44:0/2756047737 wait complete. 2024-04-02T13:23:29.175 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T13:23:29.176 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:23:29.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.276+0000 7fc0c6174640 1 Processor -- start 2024-04-02T13:23:29.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.276+0000 7fc0c6174640 1 -- start start 2024-04-02T13:23:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0c6174640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0c00d09f0 0x7fc0c006d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0c6174640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0c006ece0 0x7fc0c00acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0c6174640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0c006dc40 con 0x7fc0c00cece0 2024-04-02T13:23:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0c6174640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc0c006df20 con 0x7fc0c00d09f0 2024-04-02T13:23:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0c6174640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c006e200 con 0x7fc0c006ece0 2024-04-02T13:23:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0bf7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc0c00cece0 0x7fc0c00cf0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:58050/0 (socket says 172.21.15.44:58050) 2024-04-02T13:23:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0bf7fe640 1 -- 172.21.15.44:0/1954859546 learned_addr learned my addr 172.21.15.44:0/1954859546 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2532087338 0 0) 0x7fc0c006dc40 con 0x7fc0c00cece0 2024-04-02T13:23:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0beffd640 1 --2- 172.21.15.44:0/1954859546 >> v2:172.21.15.69:3300/0 conn(0x7fc0c00d09f0 0x7fc0c006d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0a8003520 con 0x7fc0c00cece0 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0bffff640 1 --2- 172.21.15.44:0/1954859546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0c006ece0 0x7fc0c00acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1379885808 0 0) 0x7fc0a8003520 con 0x7fc0c00cece0 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0c006ece0 msgr2=0x7fc0c00acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0be7fc640 1 --2- 172.21.15.44:0/1954859546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0c006ece0 0x7fc0c00acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 >> v2:172.21.15.69:3300/0 conn(0x7fc0c00d09f0 msgr2=0x7fc0c006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0be7fc640 1 --2- 172.21.15.44:0/1954859546 >> v2:172.21.15.69:3300/0 conn(0x7fc0c00d09f0 0x7fc0c006d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c006e4e0 con 0x7fc0c00cece0 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fc0b4001670 con 0x7fc0c00cece0 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 >> v1:172.21.15.44:6789/0 conn(0x7fc0c00cece0 legacy=0x7fc0c00cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.277+0000 7fc0be7fc640 1 --2- 172.21.15.44:0/1954859546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0a80039b0 0x7fc0a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.278+0000 7fc0be7fc640 1 --2- 172.21.15.44:0/1954859546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0c006ece0 0x7fc0a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.278+0000 7fc0be7fc640 1 --2- 172.21.15.44:0/1954859546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a8004d40 0x7fc0a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.278+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c006df20 con 0x7fc0a80039b0 2024-04-02T13:23:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.278+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c006e200 con 0x7fc0a8004d40 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.278+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc0a800b9c0 con 0x7fc0c006ece0 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.278+0000 7fc0bf7fe640 1 --2- 172.21.15.44:0/1954859546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0a80039b0 0x7fc0a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.278+0000 7fc0bffff640 1 --2- 172.21.15.44:0/1954859546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a8004d40 0x7fc0a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.278+0000 7fc0beffd640 1 --2- 172.21.15.44:0/1954859546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0c006ece0 0x7fc0a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.278+0000 7fc0bf7fe640 1 -- 172.21.15.44:0/1954859546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0c006ece0 msgr2=0x7fc0a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.278+0000 7fc0bf7fe640 1 --2- 172.21.15.44:0/1954859546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0c006ece0 0x7fc0a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.278+0000 7fc0bf7fe640 1 -- 172.21.15.44:0/1954859546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a8004d40 msgr2=0x7fc0a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.278+0000 7fc0bf7fe640 1 --2- 172.21.15.44:0/1954859546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a8004d40 0x7fc0a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.278+0000 7fc0bf7fe640 1 -- 172.21.15.44:0/1954859546 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0a8003520 con 0x7fc0a80039b0 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.279+0000 7fc0bf7fe640 1 --2- 172.21.15.44:0/1954859546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0a80039b0 0x7fc0a8003dd0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400c210 tx=0x7fc0a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6e30d0d0d77a64 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.279+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4007d50 con 0x7fc0a80039b0 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.279+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b4030070 con 0x7fc0a80039b0 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.279+0000 7fc0be7fc640 1 -- 172.21.15.44:0/1954859546 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b402cd20 con 0x7fc0a80039b0 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.279+0000 7fc0c6174640 1 -- 172.21.15.44:0/1954859546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0a80039b0 msgr2=0x7fc0a8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.279+0000 7fc0c6174640 1 --2- 172.21.15.44:0/1954859546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0a80039b0 0x7fc0a8003dd0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400c210 tx=0x7fc0a8005830 comp rx=0 tx=0).stop 2024-04-02T13:23:29.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.279+0000 7fc0bffff640 1 -- 172.21.15.44:0/1954859546 reap_dead start 2024-04-02T13:23:29.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.279+0000 7fc0c6174640 1 -- 172.21.15.44:0/1954859546 shutdown_connections 2024-04-02T13:23:29.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.279+0000 7fc0c6174640 1 -- 172.21.15.44:0/1954859546 >> 172.21.15.44:0/1954859546 conn(0x7fc0c00cfcd0 msgr2=0x7fc0c00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:29.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.279+0000 7fc0c6174640 1 -- 172.21.15.44:0/1954859546 shutdown_connections 2024-04-02T13:23:29.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.279+0000 7fc0c6174640 1 -- 172.21.15.44:0/1954859546 wait complete. 2024-04-02T13:23:29.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.280+0000 7fc0c6174640 1 Processor -- start 2024-04-02T13:23:29.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.281+0000 7fc0c6174640 1 -- start start 2024-04-02T13:23:29.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.281+0000 7fc0c6174640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80039b0 0x7fc0c015b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.281+0000 7fc0c6174640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c009ce80 0x7fc0c01486e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.281+0000 7fc0c6174640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c00b0d10 0x7fc0c00b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.281+0000 7fc0c6174640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c006d990 con 0x7fc0c00b0d10 2024-04-02T13:23:29.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.281+0000 7fc0c6174640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c006d810 con 0x7fc0c009ce80 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.281+0000 7fc0c6174640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c015b220 con 0x7fc0a80039b0 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.281+0000 7fc0bffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c00b0d10 0x7fc0c00b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0bffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c00b0d10 0x7fc0c00b1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56390/0 (socket says 172.21.15.44:56390) 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0bffff640 1 -- 172.21.15.44:0/3795218970 learned_addr learned my addr 172.21.15.44:0/3795218970 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0bffff640 1 -- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80039b0 msgr2=0x7fc0c015b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0bf7fe640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80039b0 0x7fc0c015b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0beffd640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c009ce80 0x7fc0c01486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0bffff640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80039b0 0x7fc0c015b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0bffff640 1 -- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c009ce80 msgr2=0x7fc0c01486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0bffff640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c009ce80 0x7fc0c01486e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0bffff640 1 -- 172.21.15.44:0/3795218970 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c009d2e0 con 0x7fc0c00b0d10 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0bffff640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c00b0d10 0x7fc0c00b1190 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fc0c00b5120 tx=0x7fc0b800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0a7fff640 1 -- 172.21.15.44:0/3795218970 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b800fe10 con 0x7fc0c00b0d10 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0a7fff640 1 -- 172.21.15.44:0/3795218970 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b8015430 con 0x7fc0c00b0d10 2024-04-02T13:23:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0a7fff640 1 -- 172.21.15.44:0/3795218970 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8015820 con 0x7fc0c00b0d10 2024-04-02T13:23:29.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0c6174640 1 -- 172.21.15.44:0/3795218970 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c015bc10 con 0x7fc0c00b0d10 2024-04-02T13:23:29.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.282+0000 7fc0c6174640 1 -- 172.21.15.44:0/3795218970 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c015c1d0 con 0x7fc0c00b0d10 2024-04-02T13:23:29.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.283+0000 7fc0c6174640 1 -- 172.21.15.44:0/3795218970 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c0074f20 con 0x7fc0c00b0d10 2024-04-02T13:23:29.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.284+0000 7fc0a7fff640 1 -- 172.21.15.44:0/3795218970 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc0b80159c0 con 0x7fc0c00b0d10 2024-04-02T13:23:29.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.284+0000 7fc0a7fff640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0ac06c8c0 0x7fc0ac06ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.284+0000 7fc0a7fff640 1 -- 172.21.15.44:0/3795218970 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fc0b808caa0 con 0x7fc0c00b0d10 2024-04-02T13:23:29.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.284+0000 7fc0bf7fe640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0ac06c8c0 0x7fc0ac06ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.285+0000 7fc0bf7fe640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0ac06c8c0 0x7fc0ac06ed80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4034000 tx=0x7fc0b4007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:29.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.289+0000 7fc0a7fff640 1 -- 172.21.15.44:0/3795218970 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc0b8019030 con 0x7fc0c00b0d10 2024-04-02T13:23:29.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.469+0000 7fc0c6174640 1 -- 172.21.15.44:0/3795218970 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc0c0073450 con 0x7fc0ac06c8c0 2024-04-02T13:23:29.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.476+0000 7fc0a7fff640 1 -- 172.21.15.44:0/3795218970 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252036 (secure 0 0 0) 0x7fc0c0073450 con 0x7fc0ac06c8c0 2024-04-02T13:23:29.481 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:29.482 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:23:29.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.479+0000 7fc0c6174640 1 -- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0ac06c8c0 msgr2=0x7fc0ac06ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.479+0000 7fc0c6174640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0ac06c8c0 0x7fc0ac06ed80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4034000 tx=0x7fc0b4007000 comp rx=0 tx=0).stop 2024-04-02T13:23:29.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.479+0000 7fc0c6174640 1 -- 172.21.15.44:0/3795218970 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c00b0d10 msgr2=0x7fc0c00b1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.479+0000 7fc0c6174640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c00b0d10 0x7fc0c00b1190 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fc0c00b5120 tx=0x7fc0b800f580 comp rx=0 tx=0).stop 2024-04-02T13:23:29.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.479+0000 7fc0c6174640 1 -- 172.21.15.44:0/3795218970 shutdown_connections 2024-04-02T13:23:29.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.479+0000 7fc0c6174640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc0ac06c8c0 0x7fc0ac06ed80 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.479+0000 7fc0c6174640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c00b0d10 0x7fc0c00b1190 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.479+0000 7fc0c6174640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c009ce80 0x7fc0c01486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.479+0000 7fc0c6174640 1 --2- 172.21.15.44:0/3795218970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80039b0 0x7fc0c015b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.479+0000 7fc0c6174640 1 -- 172.21.15.44:0/3795218970 >> 172.21.15.44:0/3795218970 conn(0x7fc0c00cfcd0 msgr2=0x7fc0c00cf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:29.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.479+0000 7fc0c6174640 1 -- 172.21.15.44:0/3795218970 shutdown_connections 2024-04-02T13:23:29.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.479+0000 7fc0c6174640 1 -- 172.21.15.44:0/3795218970 wait complete. 2024-04-02T13:23:29.491 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-04-02T13:23:28.918876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":231904,"kb_used_data":7368,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754742816,"statfs":{"total":773094113280,"available":772856643584,"internally_reserved":0,"allocated":7544832,"data_stored":4025292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.480723"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053223+0000","last_change":"2024-04-02T13:23:25.469431+0000","last_active":"2024-04-02T13:23:27.053223+0000","last_peered":"2024-04-02T13:23:27.053223+0000","last_clean":"2024-04-02T13:23:27.053223+0000","last_became_active":"2024-04-02T13:23:25.469157+0000","last_became_peered":"2024-04-02T13:23:25.469157+0000","last_unstale":"2024-04-02T13:23:27.053223+0000","last_undegraded":"2024-04-02T13:23:27.053223+0000","last_fullsized":"2024-04-02T13:23:27.053223+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:08.239549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052716+0000","last_change":"2024-04-02T13:23:25.461364+0000","last_active":"2024-04-02T13:23:27.052716+0000","last_peered":"2024-04-02T13:23:27.052716+0000","last_clean":"2024-04-02T13:23:27.052716+0000","last_became_active":"2024-04-02T13:23:25.461071+0000","last_became_peered":"2024-04-02T13:23:25.461071+0000","last_unstale":"2024-04-02T13:23:27.052716+0000","last_undegraded":"2024-04-02T13:23:27.052716+0000","last_fullsized":"2024-04-02T13:23:27.052716+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:29:14.631137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462494+0000","last_change":"2024-04-02T13:23:25.467141+0000","last_active":"2024-04-02T13:23:26.462494+0000","last_peered":"2024-04-02T13:23:26.462494+0000","last_clean":"2024-04-02T13:23:26.462494+0000","last_became_active":"2024-04-02T13:23:25.466843+0000","last_became_peered":"2024-04-02T13:23:25.466843+0000","last_unstale":"2024-04-02T13:23:26.462494+0000","last_undegraded":"2024-04-02T13:23:26.462494+0000","last_fullsized":"2024-04-02T13:23:26.462494+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:12.317864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463849+0000","last_change":"2024-04-02T13:23:25.464706+0000","last_active":"2024-04-02T13:23:26.463849+0000","last_peered":"2024-04-02T13:23:26.463849+0000","last_clean":"2024-04-02T13:23:26.463849+0000","last_became_active":"2024-04-02T13:23:25.464429+0000","last_became_peered":"2024-04-02T13:23:25.464429+0000","last_unstale":"2024-04-02T13:23:26.463849+0000","last_undegraded":"2024-04-02T13:23:26.463849+0000","last_fullsized":"2024-04-02T13:23:26.463849+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:33:53.740788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052710+0000","last_change":"2024-04-02T13:23:25.466279+0000","last_active":"2024-04-02T13:23:27.052710+0000","last_peered":"2024-04-02T13:23:27.052710+0000","last_clean":"2024-04-02T13:23:27.052710+0000","last_became_active":"2024-04-02T13:23:25.463166+0000","last_became_peered":"2024-04-02T13:23:25.463166+0000","last_unstale":"2024-04-02T13:23:27.052710+0000","last_undegraded":"2024-04-02T13:23:27.052710+0000","last_fullsized":"2024-04-02T13:23:27.052710+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:55.928767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053460+0000","last_change":"2024-04-02T13:23:25.461164+0000","last_active":"2024-04-02T13:23:27.053460+0000","last_peered":"2024-04-02T13:23:27.053460+0000","last_clean":"2024-04-02T13:23:27.053460+0000","last_became_active":"2024-04-02T13:23:25.460895+0000","last_became_peered":"2024-04-02T13:23:25.460895+0000","last_unstale":"2024-04-02T13:23:27.053460+0000","last_undegraded":"2024-04-02T13:23:27.053460+0000","last_fullsized":"2024-04-02T13:23:27.053460+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:59:08.284683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464835+0000","last_change":"2024-04-02T13:23:25.462561+0000","last_active":"2024-04-02T13:23:26.464835+0000","last_peered":"2024-04-02T13:23:26.464835+0000","last_clean":"2024-04-02T13:23:26.464835+0000","last_became_active":"2024-04-02T13:23:25.462285+0000","last_became_peered":"2024-04-02T13:23:25.462285+0000","last_unstale":"2024-04-02T13:23:26.464835+0000","last_undegraded":"2024-04-02T13:23:26.464835+0000","last_fullsized":"2024-04-02T13:23:26.464835+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:12:01.325498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462155+0000","last_change":"2024-04-02T13:23:25.459669+0000","last_active":"2024-04-02T13:23:26.462155+0000","last_peered":"2024-04-02T13:23:26.462155+0000","last_clean":"2024-04-02T13:23:26.462155+0000","last_became_active":"2024-04-02T13:23:25.459387+0000","last_became_peered":"2024-04-02T13:23:25.459387+0000","last_unstale":"2024-04-02T13:23:26.462155+0000","last_undegraded":"2024-04-02T13:23:26.462155+0000","last_fullsized":"2024-04-02T13:23:26.462155+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:42:26.126912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462200+0000","last_change":"2024-04-02T13:23:25.463425+0000","last_active":"2024-04-02T13:23:26.462200+0000","last_peered":"2024-04-02T13:23:26.462200+0000","last_clean":"2024-04-02T13:23:26.462200+0000","last_became_active":"2024-04-02T13:23:25.463167+0000","last_became_peered":"2024-04-02T13:23:25.463167+0000","last_unstale":"2024-04-02T13:23:26.462200+0000","last_undegraded":"2024-04-02T13:23:26.462200+0000","last_fullsized":"2024-04-02T13:23:26.462200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:41:31.047808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462795+0000","last_change":"2024-04-02T13:23:25.468862+0000","last_active":"2024-04-02T13:23:26.462795+0000","last_peered":"2024-04-02T13:23:26.462795+0000","last_clean":"2024-04-02T13:23:26.462795+0000","last_became_active":"2024-04-02T13:23:25.468576+0000","last_became_peered":"2024-04-02T13:23:25.468576+0000","last_unstale":"2024-04-02T13:23:26.462795+0000","last_undegraded":"2024-04-02T13:23:26.462795+0000","last_fullsized":"2024-04-02T13:23:26.462795+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:41.526796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464034+0000","last_change":"2024-04-02T13:23:25.470138+0000","last_active":"2024-04-02T13:23:26.464034+0000","last_peered":"2024-04-02T13:23:26.464034+0000","last_clean":"2024-04-02T13:23:26.464034+0000","last_became_active":"2024-04-02T13:23:25.468618+0000","last_became_peered":"2024-04-02T13:23:25.468618+0000","last_unstale":"2024-04-02T13:23:26.464034+0000","last_undegraded":"2024-04-02T13:23:26.464034+0000","last_fullsized":"2024-04-02T13:23:26.464034+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:56:05.702719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463248+0000","last_change":"2024-04-02T13:23:25.460450+0000","last_active":"2024-04-02T13:23:26.463248+0000","last_peered":"2024-04-02T13:23:26.463248+0000","last_clean":"2024-04-02T13:23:26.463248+0000","last_became_active":"2024-04-02T13:23:25.459878+0000","last_became_peered":"2024-04-02T13:23:25.459878+0000","last_unstale":"2024-04-02T13:23:26.463248+0000","last_undegraded":"2024-04-02T13:23:26.463248+0000","last_fullsized":"2024-04-02T13:23:26.463248+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:50:07.988009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462453+0000","last_change":"2024-04-02T13:23:25.465881+0000","last_active":"2024-04-02T13:23:26.462453+0000","last_peered":"2024-04-02T13:23:26.462453+0000","last_clean":"2024-04-02T13:23:26.462453+0000","last_became_active":"2024-04-02T13:23:25.465078+0000","last_became_peered":"2024-04-02T13:23:25.465078+0000","last_unstale":"2024-04-02T13:23:26.462453+0000","last_undegraded":"2024-04-02T13:23:26.462453+0000","last_fullsized":"2024-04-02T13:23:26.462453+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:24.343720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462168+0000","last_change":"2024-04-02T13:23:25.463214+0000","last_active":"2024-04-02T13:23:26.462168+0000","last_peered":"2024-04-02T13:23:26.462168+0000","last_clean":"2024-04-02T13:23:26.462168+0000","last_became_active":"2024-04-02T13:23:25.462714+0000","last_became_peered":"2024-04-02T13:23:25.462714+0000","last_unstale":"2024-04-02T13:23:26.462168+0000","last_undegraded":"2024-04-02T13:23:26.462168+0000","last_fullsized":"2024-04-02T13:23:26.462168+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:48:09.589536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462446+0000","last_change":"2024-04-02T13:23:25.469693+0000","last_active":"2024-04-02T13:23:26.462446+0000","last_peered":"2024-04-02T13:23:26.462446+0000","last_clean":"2024-04-02T13:23:26.462446+0000","last_became_active":"2024-04-02T13:23:25.469403+0000","last_became_peered":"2024-04-02T13:23:25.469403+0000","last_unstale":"2024-04-02T13:23:26.462446+0000","last_undegraded":"2024-04-02T13:23:26.462446+0000","last_fullsized":"2024-04-02T13:23:26.462446+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:19.829312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464833+0000","last_change":"2024-04-02T13:23:25.469379+0000","last_active":"2024-04-02T13:23:26.464833+0000","last_peered":"2024-04-02T13:23:26.464833+0000","last_clean":"2024-04-02T13:23:26.464833+0000","last_became_active":"2024-04-02T13:23:25.468644+0000","last_became_peered":"2024-04-02T13:23:25.468644+0000","last_unstale":"2024-04-02T13:23:26.464833+0000","last_undegraded":"2024-04-02T13:23:26.464833+0000","last_fullsized":"2024-04-02T13:23:26.464833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:18:06.672051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053379+0000","last_change":"2024-04-02T13:23:25.476808+0000","last_active":"2024-04-02T13:23:27.053379+0000","last_peered":"2024-04-02T13:23:27.053379+0000","last_clean":"2024-04-02T13:23:27.053379+0000","last_became_active":"2024-04-02T13:23:25.476528+0000","last_became_peered":"2024-04-02T13:23:25.476528+0000","last_unstale":"2024-04-02T13:23:27.053379+0000","last_undegraded":"2024-04-02T13:23:27.053379+0000","last_fullsized":"2024-04-02T13:23:27.053379+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:15:15.333981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464448+0000","last_change":"2024-04-02T13:23:25.465204+0000","last_active":"2024-04-02T13:23:26.464448+0000","last_peered":"2024-04-02T13:23:26.464448+0000","last_clean":"2024-04-02T13:23:26.464448+0000","last_became_active":"2024-04-02T13:23:25.464913+0000","last_became_peered":"2024-04-02T13:23:25.464913+0000","last_unstale":"2024-04-02T13:23:26.464448+0000","last_undegraded":"2024-04-02T13:23:26.464448+0000","last_fullsized":"2024-04-02T13:23:26.464448+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:23.669394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053012+0000","last_change":"2024-04-02T13:23:25.465160+0000","last_active":"2024-04-02T13:23:27.053012+0000","last_peered":"2024-04-02T13:23:27.053012+0000","last_clean":"2024-04-02T13:23:27.053012+0000","last_became_active":"2024-04-02T13:23:25.464760+0000","last_became_peered":"2024-04-02T13:23:25.464760+0000","last_unstale":"2024-04-02T13:23:27.053012+0000","last_undegraded":"2024-04-02T13:23:27.053012+0000","last_fullsized":"2024-04-02T13:23:27.053012+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:29:14.631137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053362+0000","last_change":"2024-04-02T13:23:25.463632+0000","last_active":"2024-04-02T13:23:27.053362+0000","last_peered":"2024-04-02T13:23:27.053362+0000","last_clean":"2024-04-02T13:23:27.053362+0000","last_became_active":"2024-04-02T13:23:25.463340+0000","last_became_peered":"2024-04-02T13:23:25.463340+0000","last_unstale":"2024-04-02T13:23:27.053362+0000","last_undegraded":"2024-04-02T13:23:27.053362+0000","last_fullsized":"2024-04-02T13:23:27.053362+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:38.002816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462586+0000","last_change":"2024-04-02T13:23:25.466118+0000","last_active":"2024-04-02T13:23:26.462586+0000","last_peered":"2024-04-02T13:23:26.462586+0000","last_clean":"2024-04-02T13:23:26.462586+0000","last_became_active":"2024-04-02T13:23:25.465634+0000","last_became_peered":"2024-04-02T13:23:25.465634+0000","last_unstale":"2024-04-02T13:23:26.462586+0000","last_undegraded":"2024-04-02T13:23:26.462586+0000","last_fullsized":"2024-04-02T13:23:26.462586+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:32.179030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053035+0000","last_change":"2024-04-02T13:23:25.459181+0000","last_active":"2024-04-02T13:23:27.053035+0000","last_peered":"2024-04-02T13:23:27.053035+0000","last_clean":"2024-04-02T13:23:27.053035+0000","last_became_active":"2024-04-02T13:23:25.458914+0000","last_became_peered":"2024-04-02T13:23:25.458914+0000","last_unstale":"2024-04-02T13:23:27.053035+0000","last_undegraded":"2024-04-02T13:23:27.053035+0000","last_fullsized":"2024-04-02T13:23:27.053035+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:47:07.091547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052380+0000","last_change":"2024-04-02T13:23:25.458589+0000","last_active":"2024-04-02T13:23:27.052380+0000","last_peered":"2024-04-02T13:23:27.052380+0000","last_clean":"2024-04-02T13:23:27.052380+0000","last_became_active":"2024-04-02T13:23:25.458288+0000","last_became_peered":"2024-04-02T13:23:25.458288+0000","last_unstale":"2024-04-02T13:23:27.052380+0000","last_undegraded":"2024-04-02T13:23:27.052380+0000","last_fullsized":"2024-04-02T13:23:27.052380+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:30:05.004986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462096+0000","last_change":"2024-04-02T13:23:25.466717+0000","last_active":"2024-04-02T13:23:26.462096+0000","last_peered":"2024-04-02T13:23:26.462096+0000","last_clean":"2024-04-02T13:23:26.462096+0000","last_became_active":"2024-04-02T13:23:25.466413+0000","last_became_peered":"2024-04-02T13:23:25.466413+0000","last_unstale":"2024-04-02T13:23:26.462096+0000","last_undegraded":"2024-04-02T13:23:26.462096+0000","last_fullsized":"2024-04-02T13:23:26.462096+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:50:07.988009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463354+0000","last_change":"2024-04-02T13:23:25.462058+0000","last_active":"2024-04-02T13:23:26.463354+0000","last_peered":"2024-04-02T13:23:26.463354+0000","last_clean":"2024-04-02T13:23:26.463354+0000","last_became_active":"2024-04-02T13:23:25.461817+0000","last_became_peered":"2024-04-02T13:23:25.461817+0000","last_unstale":"2024-04-02T13:23:26.463354+0000","last_undegraded":"2024-04-02T13:23:26.463354+0000","last_fullsized":"2024-04-02T13:23:26.463354+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:48:09.589536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462480+0000","last_change":"2024-04-02T13:23:25.464229+0000","last_active":"2024-04-02T13:23:26.462480+0000","last_peered":"2024-04-02T13:23:26.462480+0000","last_clean":"2024-04-02T13:23:26.462480+0000","last_became_active":"2024-04-02T13:23:25.463967+0000","last_became_peered":"2024-04-02T13:23:25.463967+0000","last_unstale":"2024-04-02T13:23:26.462480+0000","last_undegraded":"2024-04-02T13:23:26.462480+0000","last_fullsized":"2024-04-02T13:23:26.462480+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:48:45.430677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464184+0000","last_change":"2024-04-02T13:23:25.463865+0000","last_active":"2024-04-02T13:23:26.464184+0000","last_peered":"2024-04-02T13:23:26.464184+0000","last_clean":"2024-04-02T13:23:26.464184+0000","last_became_active":"2024-04-02T13:23:25.463599+0000","last_became_peered":"2024-04-02T13:23:25.463599+0000","last_unstale":"2024-04-02T13:23:26.464184+0000","last_undegraded":"2024-04-02T13:23:26.464184+0000","last_fullsized":"2024-04-02T13:23:26.464184+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:21:55.421252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052964+0000","last_change":"2024-04-02T13:23:25.469447+0000","last_active":"2024-04-02T13:23:27.052964+0000","last_peered":"2024-04-02T13:23:27.052964+0000","last_clean":"2024-04-02T13:23:27.052964+0000","last_became_active":"2024-04-02T13:23:25.469026+0000","last_became_peered":"2024-04-02T13:23:25.469026+0000","last_unstale":"2024-04-02T13:23:27.052964+0000","last_undegraded":"2024-04-02T13:23:27.052964+0000","last_fullsized":"2024-04-02T13:23:27.052964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:04:33.058654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462966+0000","last_change":"2024-04-02T13:23:25.465032+0000","last_active":"2024-04-02T13:23:26.462966+0000","last_peered":"2024-04-02T13:23:26.462966+0000","last_clean":"2024-04-02T13:23:26.462966+0000","last_became_active":"2024-04-02T13:23:25.463803+0000","last_became_peered":"2024-04-02T13:23:25.463803+0000","last_unstale":"2024-04-02T13:23:26.462966+0000","last_undegraded":"2024-04-02T13:23:26.462966+0000","last_fullsized":"2024-04-02T13:23:26.462966+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:07:35.829689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463021+0000","last_change":"2024-04-02T13:23:25.460164+0000","last_active":"2024-04-02T13:23:26.463021+0000","last_peered":"2024-04-02T13:23:26.463021+0000","last_clean":"2024-04-02T13:23:26.463021+0000","last_became_active":"2024-04-02T13:23:25.459883+0000","last_became_peered":"2024-04-02T13:23:25.459883+0000","last_unstale":"2024-04-02T13:23:26.463021+0000","last_undegraded":"2024-04-02T13:23:26.463021+0000","last_fullsized":"2024-04-02T13:23:26.463021+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:54.435494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463647+0000","last_change":"2024-04-02T13:23:25.470481+0000","last_active":"2024-04-02T13:23:26.463647+0000","last_peered":"2024-04-02T13:23:26.463647+0000","last_clean":"2024-04-02T13:23:26.463647+0000","last_became_active":"2024-04-02T13:23:25.469773+0000","last_became_peered":"2024-04-02T13:23:25.469773+0000","last_unstale":"2024-04-02T13:23:26.463647+0000","last_undegraded":"2024-04-02T13:23:26.463647+0000","last_fullsized":"2024-04-02T13:23:26.463647+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:10:22.691623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052259+0000","last_change":"2024-04-02T13:23:25.462340+0000","last_active":"2024-04-02T13:23:27.052259+0000","last_peered":"2024-04-02T13:23:27.052259+0000","last_clean":"2024-04-02T13:23:27.052259+0000","last_became_active":"2024-04-02T13:23:25.462074+0000","last_became_peered":"2024-04-02T13:23:25.462074+0000","last_unstale":"2024-04-02T13:23:27.052259+0000","last_undegraded":"2024-04-02T13:23:27.052259+0000","last_fullsized":"2024-04-02T13:23:27.052259+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:00:37.296114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052934+0000","last_change":"2024-04-02T13:23:25.466488+0000","last_active":"2024-04-02T13:23:27.052934+0000","last_peered":"2024-04-02T13:23:27.052934+0000","last_clean":"2024-04-02T13:23:27.052934+0000","last_became_active":"2024-04-02T13:23:25.466218+0000","last_became_peered":"2024-04-02T13:23:25.466218+0000","last_unstale":"2024-04-02T13:23:27.052934+0000","last_undegraded":"2024-04-02T13:23:27.052934+0000","last_fullsized":"2024-04-02T13:23:27.052934+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:02:00.562229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"64'4","reported_seq":18,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053520+0000","last_change":"2024-04-02T13:23:24.456383+0000","last_active":"2024-04-02T13:23:27.053520+0000","last_peered":"2024-04-02T13:23:27.053520+0000","last_clean":"2024-04-02T13:23:27.053520+0000","last_became_active":"2024-04-02T13:23:24.456093+0000","last_became_peered":"2024-04-02T13:23:24.456093+0000","last_unstale":"2024-04-02T13:23:27.053520+0000","last_undegraded":"2024-04-02T13:23:27.053520+0000","last_fullsized":"2024-04-02T13:23:27.053520+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:33:52.736716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464365+0000","last_change":"2024-04-02T13:23:25.471025+0000","last_active":"2024-04-02T13:23:26.464365+0000","last_peered":"2024-04-02T13:23:26.464365+0000","last_clean":"2024-04-02T13:23:26.464365+0000","last_became_active":"2024-04-02T13:23:25.470669+0000","last_became_peered":"2024-04-02T13:23:25.470669+0000","last_unstale":"2024-04-02T13:23:26.464365+0000","last_undegraded":"2024-04-02T13:23:26.464365+0000","last_fullsized":"2024-04-02T13:23:26.464365+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:47.944508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053764+0000","last_change":"2024-04-02T13:23:24.459158+0000","last_active":"2024-04-02T13:23:27.053764+0000","last_peered":"2024-04-02T13:23:27.053764+0000","last_clean":"2024-04-02T13:23:27.053764+0000","last_became_active":"2024-04-02T13:23:24.458790+0000","last_became_peered":"2024-04-02T13:23:24.458790+0000","last_unstale":"2024-04-02T13:23:27.053764+0000","last_undegraded":"2024-04-02T13:23:27.053764+0000","last_fullsized":"2024-04-02T13:23:27.053764+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:42:25.122839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053831+0000","last_change":"2024-04-02T13:23:25.463933+0000","last_active":"2024-04-02T13:23:27.053831+0000","last_peered":"2024-04-02T13:23:27.053831+0000","last_clean":"2024-04-02T13:23:27.053831+0000","last_became_active":"2024-04-02T13:23:25.463612+0000","last_became_peered":"2024-04-02T13:23:25.463612+0000","last_unstale":"2024-04-02T13:23:27.053831+0000","last_undegraded":"2024-04-02T13:23:27.053831+0000","last_fullsized":"2024-04-02T13:23:27.053831+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:41.526796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052947+0000","last_change":"2024-04-02T13:23:24.456996+0000","last_active":"2024-04-02T13:23:27.052947+0000","last_peered":"2024-04-02T13:23:27.052947+0000","last_clean":"2024-04-02T13:23:27.052947+0000","last_became_active":"2024-04-02T13:23:24.456720+0000","last_became_peered":"2024-04-02T13:23:24.456720+0000","last_unstale":"2024-04-02T13:23:27.052947+0000","last_undegraded":"2024-04-02T13:23:27.052947+0000","last_fullsized":"2024-04-02T13:23:27.052947+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:48:44.426604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462339+0000","last_change":"2024-04-02T13:23:25.471124+0000","last_active":"2024-04-02T13:23:26.462339+0000","last_peered":"2024-04-02T13:23:26.462339+0000","last_clean":"2024-04-02T13:23:26.462339+0000","last_became_active":"2024-04-02T13:23:25.470234+0000","last_became_peered":"2024-04-02T13:23:25.470234+0000","last_unstale":"2024-04-02T13:23:26.462339+0000","last_undegraded":"2024-04-02T13:23:26.462339+0000","last_fullsized":"2024-04-02T13:23:26.462339+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:00:37.296114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053413+0000","last_change":"2024-04-02T13:23:24.455277+0000","last_active":"2024-04-02T13:23:27.053413+0000","last_peered":"2024-04-02T13:23:27.053413+0000","last_clean":"2024-04-02T13:23:27.053413+0000","last_became_active":"2024-04-02T13:23:24.452815+0000","last_became_peered":"2024-04-02T13:23:24.452815+0000","last_unstale":"2024-04-02T13:23:27.053413+0000","last_undegraded":"2024-04-02T13:23:27.053413+0000","last_fullsized":"2024-04-02T13:23:27.053413+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:31.174958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464833+0000","last_change":"2024-04-02T13:23:25.466125+0000","last_active":"2024-04-02T13:23:26.464833+0000","last_peered":"2024-04-02T13:23:26.464833+0000","last_clean":"2024-04-02T13:23:26.464833+0000","last_became_active":"2024-04-02T13:23:25.465847+0000","last_became_peered":"2024-04-02T13:23:25.465847+0000","last_unstale":"2024-04-02T13:23:26.464833+0000","last_undegraded":"2024-04-02T13:23:26.464833+0000","last_fullsized":"2024-04-02T13:23:26.464833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:37.703991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053263+0000","last_change":"2024-04-02T13:23:24.452751+0000","last_active":"2024-04-02T13:23:27.053263+0000","last_peered":"2024-04-02T13:23:27.053263+0000","last_clean":"2024-04-02T13:23:27.053263+0000","last_became_active":"2024-04-02T13:23:24.452492+0000","last_became_peered":"2024-04-02T13:23:24.452492+0000","last_unstale":"2024-04-02T13:23:27.053263+0000","last_undegraded":"2024-04-02T13:23:27.053263+0000","last_fullsized":"2024-04-02T13:23:27.053263+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:40.522723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052600+0000","last_change":"2024-04-02T13:23:25.476798+0000","last_active":"2024-04-02T13:23:27.052600+0000","last_peered":"2024-04-02T13:23:27.052600+0000","last_clean":"2024-04-02T13:23:27.052600+0000","last_became_active":"2024-04-02T13:23:25.476538+0000","last_became_peered":"2024-04-02T13:23:25.476538+0000","last_unstale":"2024-04-02T13:23:27.052600+0000","last_undegraded":"2024-04-02T13:23:27.052600+0000","last_fullsized":"2024-04-02T13:23:27.052600+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:00.139996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053061+0000","last_change":"2024-04-02T13:23:24.451654+0000","last_active":"2024-04-02T13:23:27.053061+0000","last_peered":"2024-04-02T13:23:27.053061+0000","last_clean":"2024-04-02T13:23:27.053061+0000","last_became_active":"2024-04-02T13:23:24.451084+0000","last_became_peered":"2024-04-02T13:23:24.451084+0000","last_unstale":"2024-04-02T13:23:27.053061+0000","last_undegraded":"2024-04-02T13:23:27.053061+0000","last_fullsized":"2024-04-02T13:23:27.053061+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:00:36.292042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052648+0000","last_change":"2024-04-02T13:23:25.468308+0000","last_active":"2024-04-02T13:23:27.052648+0000","last_peered":"2024-04-02T13:23:27.052648+0000","last_clean":"2024-04-02T13:23:27.052648+0000","last_became_active":"2024-04-02T13:23:25.468029+0000","last_became_peered":"2024-04-02T13:23:25.468029+0000","last_unstale":"2024-04-02T13:23:27.052648+0000","last_undegraded":"2024-04-02T13:23:27.052648+0000","last_fullsized":"2024-04-02T13:23:27.052648+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:47.944508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052515+0000","last_change":"2024-04-02T13:23:24.458224+0000","last_active":"2024-04-02T13:23:27.052515+0000","last_peered":"2024-04-02T13:23:27.052515+0000","last_clean":"2024-04-02T13:23:27.052515+0000","last_became_active":"2024-04-02T13:23:24.457941+0000","last_became_peered":"2024-04-02T13:23:24.457941+0000","last_unstale":"2024-04-02T13:23:27.052515+0000","last_undegraded":"2024-04-02T13:23:27.052515+0000","last_fullsized":"2024-04-02T13:23:27.052515+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:59:07.280610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053263+0000","last_change":"2024-04-02T13:23:25.462821+0000","last_active":"2024-04-02T13:23:27.053263+0000","last_peered":"2024-04-02T13:23:27.053263+0000","last_clean":"2024-04-02T13:23:27.053263+0000","last_became_active":"2024-04-02T13:23:25.462554+0000","last_became_peered":"2024-04-02T13:23:25.462554+0000","last_unstale":"2024-04-02T13:23:27.053263+0000","last_undegraded":"2024-04-02T13:23:27.053263+0000","last_fullsized":"2024-04-02T13:23:27.053263+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:13:19.036189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052390+0000","last_change":"2024-04-02T13:23:24.457917+0000","last_active":"2024-04-02T13:23:27.052390+0000","last_peered":"2024-04-02T13:23:27.052390+0000","last_clean":"2024-04-02T13:23:27.052390+0000","last_became_active":"2024-04-02T13:23:24.457121+0000","last_became_peered":"2024-04-02T13:23:24.457121+0000","last_unstale":"2024-04-02T13:23:27.052390+0000","last_undegraded":"2024-04-02T13:23:27.052390+0000","last_fullsized":"2024-04-02T13:23:27.052390+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:23:36.457600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053104+0000","last_change":"2024-04-02T13:23:25.468788+0000","last_active":"2024-04-02T13:23:27.053104+0000","last_peered":"2024-04-02T13:23:27.053104+0000","last_clean":"2024-04-02T13:23:27.053104+0000","last_became_active":"2024-04-02T13:23:25.468321+0000","last_became_peered":"2024-04-02T13:23:25.468321+0000","last_unstale":"2024-04-02T13:23:27.053104+0000","last_undegraded":"2024-04-02T13:23:27.053104+0000","last_fullsized":"2024-04-02T13:23:27.053104+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:03:49.303465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462281+0000","last_change":"2024-04-02T13:23:24.444565+0000","last_active":"2024-04-02T13:23:26.462281+0000","last_peered":"2024-04-02T13:23:26.462281+0000","last_clean":"2024-04-02T13:23:26.462281+0000","last_became_active":"2024-04-02T13:23:24.444311+0000","last_became_peered":"2024-04-02T13:23:24.444311+0000","last_unstale":"2024-04-02T13:23:26.462281+0000","last_undegraded":"2024-04-02T13:23:26.462281+0000","last_fullsized":"2024-04-02T13:23:26.462281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:07.235476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052820+0000","last_change":"2024-04-02T13:23:25.461614+0000","last_active":"2024-04-02T13:23:27.052820+0000","last_peered":"2024-04-02T13:23:27.052820+0000","last_clean":"2024-04-02T13:23:27.052820+0000","last_became_active":"2024-04-02T13:23:25.461353+0000","last_became_peered":"2024-04-02T13:23:25.461353+0000","last_unstale":"2024-04-02T13:23:27.052820+0000","last_undegraded":"2024-04-02T13:23:27.052820+0000","last_fullsized":"2024-04-02T13:23:27.052820+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:55:29.376845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464431+0000","last_change":"2024-04-02T13:23:24.457899+0000","last_active":"2024-04-02T13:23:26.464431+0000","last_peered":"2024-04-02T13:23:26.464431+0000","last_clean":"2024-04-02T13:23:26.464431+0000","last_became_active":"2024-04-02T13:23:24.457540+0000","last_became_peered":"2024-04-02T13:23:24.457540+0000","last_unstale":"2024-04-02T13:23:26.464431+0000","last_undegraded":"2024-04-02T13:23:26.464431+0000","last_fullsized":"2024-04-02T13:23:26.464431+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:34:05.703937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052955+0000","last_change":"2024-04-02T13:23:25.466861+0000","last_active":"2024-04-02T13:23:27.052955+0000","last_peered":"2024-04-02T13:23:27.052955+0000","last_clean":"2024-04-02T13:23:27.052955+0000","last_became_active":"2024-04-02T13:23:25.466425+0000","last_became_peered":"2024-04-02T13:23:25.466425+0000","last_unstale":"2024-04-02T13:23:27.052955+0000","last_undegraded":"2024-04-02T13:23:27.052955+0000","last_fullsized":"2024-04-02T13:23:27.052955+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:29.261268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"64'3","reported_seq":17,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462550+0000","last_change":"2024-04-02T13:23:24.444242+0000","last_active":"2024-04-02T13:23:26.462550+0000","last_peered":"2024-04-02T13:23:26.462550+0000","last_clean":"2024-04-02T13:23:26.462550+0000","last_became_active":"2024-04-02T13:23:24.443907+0000","last_became_peered":"2024-04-02T13:23:24.443907+0000","last_unstale":"2024-04-02T13:23:26.462550+0000","last_undegraded":"2024-04-02T13:23:26.462550+0000","last_fullsized":"2024-04-02T13:23:26.462550+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:06:37.459794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464256+0000","last_change":"2024-04-02T13:23:25.467436+0000","last_active":"2024-04-02T13:23:26.464256+0000","last_peered":"2024-04-02T13:23:26.464256+0000","last_clean":"2024-04-02T13:23:26.464256+0000","last_became_active":"2024-04-02T13:23:25.467129+0000","last_became_peered":"2024-04-02T13:23:25.467129+0000","last_unstale":"2024-04-02T13:23:26.464256+0000","last_undegraded":"2024-04-02T13:23:26.464256+0000","last_fullsized":"2024-04-02T13:23:26.464256+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:46.130146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"64'6","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053015+0000","last_change":"2024-04-02T13:23:24.455586+0000","last_active":"2024-04-02T13:23:27.053015+0000","last_peered":"2024-04-02T13:23:27.053015+0000","last_clean":"2024-04-02T13:23:27.053015+0000","last_became_active":"2024-04-02T13:23:24.455252+0000","last_became_peered":"2024-04-02T13:23:24.455252+0000","last_unstale":"2024-04-02T13:23:27.053015+0000","last_undegraded":"2024-04-02T13:23:27.053015+0000","last_fullsized":"2024-04-02T13:23:27.053015+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:33.668366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463948+0000","last_change":"2024-04-02T13:23:25.468250+0000","last_active":"2024-04-02T13:23:26.463948+0000","last_peered":"2024-04-02T13:23:26.463948+0000","last_clean":"2024-04-02T13:23:26.463948+0000","last_became_active":"2024-04-02T13:23:25.467952+0000","last_became_peered":"2024-04-02T13:23:25.467952+0000","last_unstale":"2024-04-02T13:23:26.463948+0000","last_undegraded":"2024-04-02T13:23:26.463948+0000","last_fullsized":"2024-04-02T13:23:26.463948+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:34.672439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463021+0000","last_change":"2024-04-02T13:23:24.454154+0000","last_active":"2024-04-02T13:23:26.463021+0000","last_peered":"2024-04-02T13:23:26.463021+0000","last_clean":"2024-04-02T13:23:26.463021+0000","last_became_active":"2024-04-02T13:23:24.453881+0000","last_became_peered":"2024-04-02T13:23:24.453881+0000","last_unstale":"2024-04-02T13:23:26.463021+0000","last_undegraded":"2024-04-02T13:23:26.463021+0000","last_fullsized":"2024-04-02T13:23:26.463021+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:28.257195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462141+0000","last_change":"2024-04-02T13:23:25.461190+0000","last_active":"2024-04-02T13:23:26.462141+0000","last_peered":"2024-04-02T13:23:26.462141+0000","last_clean":"2024-04-02T13:23:26.462141+0000","last_became_active":"2024-04-02T13:23:25.460911+0000","last_became_peered":"2024-04-02T13:23:25.460911+0000","last_unstale":"2024-04-02T13:23:26.462141+0000","last_undegraded":"2024-04-02T13:23:26.462141+0000","last_fullsized":"2024-04-02T13:23:26.462141+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:11:21.268539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463039+0000","last_change":"2024-04-02T13:23:24.452152+0000","last_active":"2024-04-02T13:23:26.463039+0000","last_peered":"2024-04-02T13:23:26.463039+0000","last_clean":"2024-04-02T13:23:26.463039+0000","last_became_active":"2024-04-02T13:23:24.451877+0000","last_became_peered":"2024-04-02T13:23:24.451877+0000","last_unstale":"2024-04-02T13:23:26.463039+0000","last_undegraded":"2024-04-02T13:23:26.463039+0000","last_fullsized":"2024-04-02T13:23:26.463039+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:46.940435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052645+0000","last_change":"2024-04-02T13:23:25.462012+0000","last_active":"2024-04-02T13:23:27.052645+0000","last_peered":"2024-04-02T13:23:27.052645+0000","last_clean":"2024-04-02T13:23:27.052645+0000","last_became_active":"2024-04-02T13:23:25.461738+0000","last_became_peered":"2024-04-02T13:23:25.461738+0000","last_unstale":"2024-04-02T13:23:27.052645+0000","last_undegraded":"2024-04-02T13:23:27.052645+0000","last_fullsized":"2024-04-02T13:23:27.052645+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:46.130146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462570+0000","last_change":"2024-04-02T13:23:24.450883+0000","last_active":"2024-04-02T13:23:26.462570+0000","last_peered":"2024-04-02T13:23:26.462570+0000","last_clean":"2024-04-02T13:23:26.462570+0000","last_became_active":"2024-04-02T13:23:24.450640+0000","last_became_peered":"2024-04-02T13:23:24.450640+0000","last_unstale":"2024-04-02T13:23:26.462570+0000","last_undegraded":"2024-04-02T13:23:26.462570+0000","last_fullsized":"2024-04-02T13:23:26.462570+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:59.135923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052931+0000","last_change":"2024-04-02T13:23:25.465196+0000","last_active":"2024-04-02T13:23:27.052931+0000","last_peered":"2024-04-02T13:23:27.052931+0000","last_clean":"2024-04-02T13:23:27.052931+0000","last_became_active":"2024-04-02T13:23:25.464890+0000","last_became_peered":"2024-04-02T13:23:25.464890+0000","last_unstale":"2024-04-02T13:23:27.052931+0000","last_undegraded":"2024-04-02T13:23:27.052931+0000","last_fullsized":"2024-04-02T13:23:27.052931+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:06:38.463867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"64'8","reported_seq":22,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464021+0000","last_change":"2024-04-02T13:23:24.459111+0000","last_active":"2024-04-02T13:23:26.464021+0000","last_peered":"2024-04-02T13:23:26.464021+0000","last_clean":"2024-04-02T13:23:26.464021+0000","last_became_active":"2024-04-02T13:23:24.458776+0000","last_became_peered":"2024-04-02T13:23:24.458776+0000","last_unstale":"2024-04-02T13:23:26.464021+0000","last_undegraded":"2024-04-02T13:23:26.464021+0000","last_fullsized":"2024-04-02T13:23:26.464021+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:27:53.204273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462501+0000","last_change":"2024-04-02T13:23:25.469323+0000","last_active":"2024-04-02T13:23:26.462501+0000","last_peered":"2024-04-02T13:23:26.462501+0000","last_clean":"2024-04-02T13:23:26.462501+0000","last_became_active":"2024-04-02T13:23:25.469044+0000","last_became_peered":"2024-04-02T13:23:25.469044+0000","last_unstale":"2024-04-02T13:23:26.462501+0000","last_undegraded":"2024-04-02T13:23:26.462501+0000","last_fullsized":"2024-04-02T13:23:26.462501+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:59:08.284683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463310+0000","last_change":"2024-04-02T13:23:24.447016+0000","last_active":"2024-04-02T13:23:26.463310+0000","last_peered":"2024-04-02T13:23:26.463310+0000","last_clean":"2024-04-02T13:23:26.463310+0000","last_became_active":"2024-04-02T13:23:24.446709+0000","last_became_peered":"2024-04-02T13:23:24.446709+0000","last_unstale":"2024-04-02T13:23:26.463310+0000","last_undegraded":"2024-04-02T13:23:26.463310+0000","last_fullsized":"2024-04-02T13:23:26.463310+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:54.924694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053068+0000","last_change":"2024-04-02T13:23:25.463056+0000","last_active":"2024-04-02T13:23:27.053068+0000","last_peered":"2024-04-02T13:23:27.053068+0000","last_clean":"2024-04-02T13:23:27.053068+0000","last_became_active":"2024-04-02T13:23:25.462734+0000","last_became_peered":"2024-04-02T13:23:25.462734+0000","last_unstale":"2024-04-02T13:23:27.053068+0000","last_undegraded":"2024-04-02T13:23:27.053068+0000","last_fullsized":"2024-04-02T13:23:27.053068+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:06:38.463867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462857+0000","last_change":"2024-04-02T13:23:24.456013+0000","last_active":"2024-04-02T13:23:26.462857+0000","last_peered":"2024-04-02T13:23:26.462857+0000","last_clean":"2024-04-02T13:23:26.462857+0000","last_became_active":"2024-04-02T13:23:24.455233+0000","last_became_peered":"2024-04-02T13:23:24.455233+0000","last_unstale":"2024-04-02T13:23:26.462857+0000","last_undegraded":"2024-04-02T13:23:26.462857+0000","last_fullsized":"2024-04-02T13:23:26.462857+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:03:48.299392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464376+0000","last_change":"2024-04-02T13:23:25.470692+0000","last_active":"2024-04-02T13:23:26.464376+0000","last_peered":"2024-04-02T13:23:26.464376+0000","last_clean":"2024-04-02T13:23:26.464376+0000","last_became_active":"2024-04-02T13:23:25.470429+0000","last_became_peered":"2024-04-02T13:23:25.470429+0000","last_unstale":"2024-04-02T13:23:26.464376+0000","last_undegraded":"2024-04-02T13:23:26.464376+0000","last_fullsized":"2024-04-02T13:23:26.464376+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:55.928767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462658+0000","last_change":"2024-04-02T13:23:24.453950+0000","last_active":"2024-04-02T13:23:26.462658+0000","last_peered":"2024-04-02T13:23:26.462658+0000","last_clean":"2024-04-02T13:23:26.462658+0000","last_became_active":"2024-04-02T13:23:24.453635+0000","last_became_peered":"2024-04-02T13:23:24.453635+0000","last_unstale":"2024-04-02T13:23:26.462658+0000","last_undegraded":"2024-04-02T13:23:26.462658+0000","last_fullsized":"2024-04-02T13:23:26.462658+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:04:32.054581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052919+0000","last_change":"2024-04-02T13:23:25.461765+0000","last_active":"2024-04-02T13:23:27.052919+0000","last_peered":"2024-04-02T13:23:27.052919+0000","last_clean":"2024-04-02T13:23:27.052919+0000","last_became_active":"2024-04-02T13:23:25.461499+0000","last_became_peered":"2024-04-02T13:23:25.461499+0000","last_unstale":"2024-04-02T13:23:27.052919+0000","last_undegraded":"2024-04-02T13:23:27.052919+0000","last_fullsized":"2024-04-02T13:23:27.052919+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:07:17.900627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053334+0000","last_change":"2024-04-02T13:23:24.453957+0000","last_active":"2024-04-02T13:23:27.053334+0000","last_peered":"2024-04-02T13:23:27.053334+0000","last_clean":"2024-04-02T13:23:27.053334+0000","last_became_active":"2024-04-02T13:23:24.453710+0000","last_became_peered":"2024-04-02T13:23:24.453710+0000","last_unstale":"2024-04-02T13:23:27.053334+0000","last_undegraded":"2024-04-02T13:23:27.053334+0000","last_fullsized":"2024-04-02T13:23:27.053334+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:07:34.825616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052431+0000","last_change":"2024-04-02T13:23:25.462648+0000","last_active":"2024-04-02T13:23:27.052431+0000","last_peered":"2024-04-02T13:23:27.052431+0000","last_clean":"2024-04-02T13:23:27.052431+0000","last_became_active":"2024-04-02T13:23:25.462359+0000","last_became_peered":"2024-04-02T13:23:25.462359+0000","last_unstale":"2024-04-02T13:23:27.052431+0000","last_undegraded":"2024-04-02T13:23:27.052431+0000","last_fullsized":"2024-04-02T13:23:27.052431+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:19.829312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052226+0000","last_change":"2024-04-02T13:23:24.457209+0000","last_active":"2024-04-02T13:23:27.052226+0000","last_peered":"2024-04-02T13:23:27.052226+0000","last_clean":"2024-04-02T13:23:27.052226+0000","last_became_active":"2024-04-02T13:23:24.456904+0000","last_became_peered":"2024-04-02T13:23:24.456904+0000","last_unstale":"2024-04-02T13:23:27.052226+0000","last_undegraded":"2024-04-02T13:23:27.052226+0000","last_fullsized":"2024-04-02T13:23:27.052226+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:11:20.264466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462811+0000","last_change":"2024-04-02T13:23:25.462538+0000","last_active":"2024-04-02T13:23:26.462811+0000","last_peered":"2024-04-02T13:23:26.462811+0000","last_clean":"2024-04-02T13:23:26.462811+0000","last_became_active":"2024-04-02T13:23:25.462334+0000","last_became_peered":"2024-04-02T13:23:25.462334+0000","last_unstale":"2024-04-02T13:23:26.462811+0000","last_undegraded":"2024-04-02T13:23:26.462811+0000","last_fullsized":"2024-04-02T13:23:26.462811+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:11.330576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463073+0000","last_change":"2024-04-02T13:23:24.453376+0000","last_active":"2024-04-02T13:23:26.463073+0000","last_peered":"2024-04-02T13:23:26.463073+0000","last_clean":"2024-04-02T13:23:26.463073+0000","last_became_active":"2024-04-02T13:23:24.452980+0000","last_became_peered":"2024-04-02T13:23:24.452980+0000","last_unstale":"2024-04-02T13:23:26.463073+0000","last_undegraded":"2024-04-02T13:23:26.463073+0000","last_fullsized":"2024-04-02T13:23:26.463073+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:08.196023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052935+0000","last_change":"2024-04-02T13:23:25.468632+0000","last_active":"2024-04-02T13:23:27.052935+0000","last_peered":"2024-04-02T13:23:27.052935+0000","last_clean":"2024-04-02T13:23:27.052935+0000","last_became_active":"2024-04-02T13:23:25.468312+0000","last_became_peered":"2024-04-02T13:23:25.468312+0000","last_unstale":"2024-04-02T13:23:27.052935+0000","last_undegraded":"2024-04-02T13:23:27.052935+0000","last_fullsized":"2024-04-02T13:23:27.052935+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:00.139996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052135+0000","last_change":"2024-04-02T13:23:24.458596+0000","last_active":"2024-04-02T13:23:27.052135+0000","last_peered":"2024-04-02T13:23:27.052135+0000","last_clean":"2024-04-02T13:23:27.052135+0000","last_became_active":"2024-04-02T13:23:24.458305+0000","last_became_peered":"2024-04-02T13:23:24.458305+0000","last_unstale":"2024-04-02T13:23:27.052135+0000","last_undegraded":"2024-04-02T13:23:27.052135+0000","last_fullsized":"2024-04-02T13:23:27.052135+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:37:49.847371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052658+0000","last_change":"2024-04-02T13:23:25.467538+0000","last_active":"2024-04-02T13:23:27.052658+0000","last_peered":"2024-04-02T13:23:27.052658+0000","last_clean":"2024-04-02T13:23:27.052658+0000","last_became_active":"2024-04-02T13:23:25.467134+0000","last_became_peered":"2024-04-02T13:23:25.467134+0000","last_unstale":"2024-04-02T13:23:27.052658+0000","last_undegraded":"2024-04-02T13:23:27.052658+0000","last_fullsized":"2024-04-02T13:23:27.052658+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:12.317864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463634+0000","last_change":"2024-04-02T13:23:24.455671+0000","last_active":"2024-04-02T13:23:26.463634+0000","last_peered":"2024-04-02T13:23:26.463634+0000","last_clean":"2024-04-02T13:23:26.463634+0000","last_became_active":"2024-04-02T13:23:24.455338+0000","last_became_peered":"2024-04-02T13:23:24.455338+0000","last_unstale":"2024-04-02T13:23:26.463634+0000","last_undegraded":"2024-04-02T13:23:26.463634+0000","last_fullsized":"2024-04-02T13:23:26.463634+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:07:16.896554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464837+0000","last_change":"2024-04-02T13:23:25.457248+0000","last_active":"2024-04-02T13:23:26.464837+0000","last_peered":"2024-04-02T13:23:26.464837+0000","last_clean":"2024-04-02T13:23:26.464837+0000","last_became_active":"2024-04-02T13:23:25.456953+0000","last_became_peered":"2024-04-02T13:23:25.456953+0000","last_unstale":"2024-04-02T13:23:26.464837+0000","last_undegraded":"2024-04-02T13:23:26.464837+0000","last_fullsized":"2024-04-02T13:23:26.464837+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:36:44.493855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"65'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462814+0000","last_change":"2024-04-02T13:23:24.450342+0000","last_active":"2024-04-02T13:23:26.462814+0000","last_peered":"2024-04-02T13:23:26.462814+0000","last_clean":"2024-04-02T13:23:26.462814+0000","last_became_active":"2024-04-02T13:23:24.449824+0000","last_became_peered":"2024-04-02T13:23:24.449824+0000","last_unstale":"2024-04-02T13:23:26.462814+0000","last_undegraded":"2024-04-02T13:23:26.462814+0000","last_fullsized":"2024-04-02T13:23:26.462814+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:22.665321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462640+0000","last_change":"2024-04-02T13:23:25.457764+0000","last_active":"2024-04-02T13:23:26.462640+0000","last_peered":"2024-04-02T13:23:26.462640+0000","last_clean":"2024-04-02T13:23:26.462640+0000","last_became_active":"2024-04-02T13:23:25.457484+0000","last_became_peered":"2024-04-02T13:23:25.457484+0000","last_unstale":"2024-04-02T13:23:26.462640+0000","last_undegraded":"2024-04-02T13:23:26.462640+0000","last_fullsized":"2024-04-02T13:23:26.462640+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:01:09.000108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053550+0000","last_change":"2024-04-02T13:23:24.456306+0000","last_active":"2024-04-02T13:23:27.053550+0000","last_peered":"2024-04-02T13:23:27.053550+0000","last_clean":"2024-04-02T13:23:27.053550+0000","last_became_active":"2024-04-02T13:23:24.456047+0000","last_became_peered":"2024-04-02T13:23:24.456047+0000","last_unstale":"2024-04-02T13:23:27.053550+0000","last_undegraded":"2024-04-02T13:23:27.053550+0000","last_fullsized":"2024-04-02T13:23:27.053550+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:23.339647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052939+0000","last_change":"2024-04-02T13:23:25.461396+0000","last_active":"2024-04-02T13:23:27.052939+0000","last_peered":"2024-04-02T13:23:27.052939+0000","last_clean":"2024-04-02T13:23:27.052939+0000","last_became_active":"2024-04-02T13:23:25.461030+0000","last_became_peered":"2024-04-02T13:23:25.461030+0000","last_unstale":"2024-04-02T13:23:27.052939+0000","last_undegraded":"2024-04-02T13:23:27.052939+0000","last_fullsized":"2024-04-02T13:23:27.052939+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:09.200095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053121+0000","last_change":"2024-04-02T13:23:24.454332+0000","last_active":"2024-04-02T13:23:27.053121+0000","last_peered":"2024-04-02T13:23:27.053121+0000","last_clean":"2024-04-02T13:23:27.053121+0000","last_became_active":"2024-04-02T13:23:24.454066+0000","last_became_peered":"2024-04-02T13:23:24.454066+0000","last_unstale":"2024-04-02T13:23:27.053121+0000","last_undegraded":"2024-04-02T13:23:27.053121+0000","last_fullsized":"2024-04-02T13:23:27.053121+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:56:04.698646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052523+0000","last_change":"2024-04-02T13:23:25.466702+0000","last_active":"2024-04-02T13:23:27.052523+0000","last_peered":"2024-04-02T13:23:27.052523+0000","last_clean":"2024-04-02T13:23:27.052523+0000","last_became_active":"2024-04-02T13:23:25.465630+0000","last_became_peered":"2024-04-02T13:23:25.465630+0000","last_unstale":"2024-04-02T13:23:27.052523+0000","last_undegraded":"2024-04-02T13:23:27.052523+0000","last_fullsized":"2024-04-02T13:23:27.052523+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:21:55.421252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463131+0000","last_change":"2024-04-02T13:23:24.448017+0000","last_active":"2024-04-02T13:23:26.463131+0000","last_peered":"2024-04-02T13:23:26.463131+0000","last_clean":"2024-04-02T13:23:26.463131+0000","last_became_active":"2024-04-02T13:23:24.447718+0000","last_became_peered":"2024-04-02T13:23:24.447718+0000","last_unstale":"2024-04-02T13:23:26.463131+0000","last_undegraded":"2024-04-02T13:23:26.463131+0000","last_fullsized":"2024-04-02T13:23:26.463131+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:29:13.627065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463798+0000","last_change":"2024-04-02T13:23:25.462615+0000","last_active":"2024-04-02T13:23:26.463798+0000","last_peered":"2024-04-02T13:23:26.463798+0000","last_clean":"2024-04-02T13:23:26.463798+0000","last_became_active":"2024-04-02T13:23:25.462337+0000","last_became_peered":"2024-04-02T13:23:25.462337+0000","last_unstale":"2024-04-02T13:23:26.463798+0000","last_undegraded":"2024-04-02T13:23:26.463798+0000","last_fullsized":"2024-04-02T13:23:26.463798+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:24:01.044977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053087+0000","last_change":"2024-04-02T13:23:24.451981+0000","last_active":"2024-04-02T13:23:27.053087+0000","last_peered":"2024-04-02T13:23:27.053087+0000","last_clean":"2024-04-02T13:23:27.053087+0000","last_became_active":"2024-04-02T13:23:24.451689+0000","last_became_peered":"2024-04-02T13:23:24.451689+0000","last_unstale":"2024-04-02T13:23:27.053087+0000","last_undegraded":"2024-04-02T13:23:27.053087+0000","last_fullsized":"2024-04-02T13:23:27.053087+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:18.825240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052603+0000","last_change":"2024-04-02T13:23:25.476796+0000","last_active":"2024-04-02T13:23:27.052603+0000","last_peered":"2024-04-02T13:23:27.052603+0000","last_clean":"2024-04-02T13:23:27.052603+0000","last_became_active":"2024-04-02T13:23:25.476510+0000","last_became_peered":"2024-04-02T13:23:25.476510+0000","last_unstale":"2024-04-02T13:23:27.052603+0000","last_undegraded":"2024-04-02T13:23:27.052603+0000","last_fullsized":"2024-04-02T13:23:27.052603+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:29.261268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052614+0000","last_change":"2024-04-02T13:23:24.457619+0000","last_active":"2024-04-02T13:23:27.052614+0000","last_peered":"2024-04-02T13:23:27.052614+0000","last_clean":"2024-04-02T13:23:27.052614+0000","last_became_active":"2024-04-02T13:23:24.457344+0000","last_became_peered":"2024-04-02T13:23:24.457344+0000","last_unstale":"2024-04-02T13:23:27.052614+0000","last_undegraded":"2024-04-02T13:23:27.052614+0000","last_fullsized":"2024-04-02T13:23:27.052614+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:11.313791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464832+0000","last_change":"2024-04-02T13:23:25.463480+0000","last_active":"2024-04-02T13:23:26.464832+0000","last_peered":"2024-04-02T13:23:26.464832+0000","last_clean":"2024-04-02T13:23:26.464832+0000","last_became_active":"2024-04-02T13:23:25.463195+0000","last_became_peered":"2024-04-02T13:23:25.463195+0000","last_unstale":"2024-04-02T13:23:26.464832+0000","last_undegraded":"2024-04-02T13:23:26.464832+0000","last_fullsized":"2024-04-02T13:23:26.464832+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:13:54.388782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052359+0000","last_change":"2024-04-02T13:23:24.454546+0000","last_active":"2024-04-02T13:23:27.052359+0000","last_peered":"2024-04-02T13:23:27.052359+0000","last_clean":"2024-04-02T13:23:27.052359+0000","last_became_active":"2024-04-02T13:23:24.454287+0000","last_became_peered":"2024-04-02T13:23:24.454287+0000","last_unstale":"2024-04-02T13:23:27.052359+0000","last_undegraded":"2024-04-02T13:23:27.052359+0000","last_fullsized":"2024-04-02T13:23:27.052359+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:24:00.040904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052748+0000","last_change":"2024-04-02T13:23:25.466900+0000","last_active":"2024-04-02T13:23:27.052748+0000","last_peered":"2024-04-02T13:23:27.052748+0000","last_clean":"2024-04-02T13:23:27.052748+0000","last_became_active":"2024-04-02T13:23:25.466583+0000","last_became_peered":"2024-04-02T13:23:25.466583+0000","last_unstale":"2024-04-02T13:23:27.052748+0000","last_undegraded":"2024-04-02T13:23:27.052748+0000","last_fullsized":"2024-04-02T13:23:27.052748+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:23.669394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052662+0000","last_change":"2024-04-02T13:23:24.451554+0000","last_active":"2024-04-02T13:23:27.052662+0000","last_peered":"2024-04-02T13:23:27.052662+0000","last_clean":"2024-04-02T13:23:27.052662+0000","last_became_active":"2024-04-02T13:23:24.451290+0000","last_became_peered":"2024-04-02T13:23:24.451290+0000","last_unstale":"2024-04-02T13:23:27.052662+0000","last_undegraded":"2024-04-02T13:23:27.052662+0000","last_fullsized":"2024-04-02T13:23:27.052662+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:41:30.043735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":255,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462040+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:23:26.462040+0000","last_peered":"2024-04-02T13:23:26.462040+0000","last_clean":"2024-04-02T13:23:26.462040+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:23:26.462040+0000","last_undegraded":"2024-04-02T13:23:26.462040+0000","last_fullsized":"2024-04-02T13:23:26.462040+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":1396736,"data_stored":946111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":790528,"data_stored":355001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":790528,"data_stored":354989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":356579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":354967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":354967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":815104,"data_stored":357343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1380352,"data_stored":945335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:23:29.496 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:23:29.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.631+0000 7f29d31bc640 1 Processor -- start 2024-04-02T13:23:29.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.631+0000 7f29d31bc640 1 -- start start 2024-04-02T13:23:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.632+0000 7f29d31bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29cc15e140 0x7f29cc0d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.632+0000 7f29d31bc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f29cc0d1150 0x7f29cc0b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.632+0000 7f29d31bc640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29cc0b70b0 con 0x7f29cc15e520 2024-04-02T13:23:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.632+0000 7f29d31bc640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f29cc0b9f50 con 0x7f29cc0d1150 2024-04-02T13:23:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.632+0000 7f29d31bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f29cc0b7390 con 0x7f29cc15e140 2024-04-02T13:23:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.632+0000 7f29d1759640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f29cc0d1150 0x7f29cc0b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.632+0000 7f29d1759640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f29cc0d1150 0x7f29cc0b4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56400/0 (socket says 172.21.15.44:56400) 2024-04-02T13:23:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.632+0000 7f29d1759640 1 -- 172.21.15.44:0/1354064481 learned_addr learned my addr 172.21.15.44:0/1354064481 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.632+0000 7f29d0f58640 1 --2- 172.21.15.44:0/1354064481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29cc15e140 0x7f29cc0d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.633+0000 7f29d1759640 1 -- 172.21.15.44:0/1354064481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29cc15e140 msgr2=0x7f29cc0d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.633+0000 7f29d1759640 1 --2- 172.21.15.44:0/1354064481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29cc15e140 0x7f29cc0d2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.633+0000 7f29d1759640 1 -- 172.21.15.44:0/1354064481 >> v1:172.21.15.69:6789/0 conn(0x7f29cc15e520 legacy=0x7f29cc0d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.633+0000 7f29d1759640 1 -- 172.21.15.44:0/1354064481 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29cc0b7670 con 0x7f29cc0d1150 2024-04-02T13:23:29.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.633+0000 7f29d1759640 1 --2- 172.21.15.44:0/1354064481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29cc0d1150 0x7f29cc0b4cc0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009870 tx=0x7f29bc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a078348865a3791a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.634+0000 7f29cb7fe640 1 -- 172.21.15.44:0/1354064481 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc034070 con 0x7f29cc0d1150 2024-04-02T13:23:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.634+0000 7f29cb7fe640 1 -- 172.21.15.44:0/1354064481 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29bc030840 con 0x7f29cc0d1150 2024-04-02T13:23:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.634+0000 7f29cb7fe640 1 -- 172.21.15.44:0/1354064481 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc030c10 con 0x7f29cc0d1150 2024-04-02T13:23:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.634+0000 7f29d31bc640 1 -- 172.21.15.44:0/1354064481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29cc0d1150 msgr2=0x7f29cc0b4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.634+0000 7f29d31bc640 1 --2- 172.21.15.44:0/1354064481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29cc0d1150 0x7f29cc0b4cc0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009870 tx=0x7f29bc02b980 comp rx=0 tx=0).stop 2024-04-02T13:23:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.634+0000 7f29d31bc640 1 -- 172.21.15.44:0/1354064481 shutdown_connections 2024-04-02T13:23:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.634+0000 7f29d31bc640 1 --2- 172.21.15.44:0/1354064481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29cc0d1150 0x7f29cc0b4cc0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.634+0000 7f29d31bc640 1 --2- 172.21.15.44:0/1354064481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29cc15e140 0x7f29cc0d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.634+0000 7f29d31bc640 1 -- 172.21.15.44:0/1354064481 >> 172.21.15.44:0/1354064481 conn(0x7f29cc09d390 msgr2=0x7f29cc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.634+0000 7f29d31bc640 1 -- 172.21.15.44:0/1354064481 shutdown_connections 2024-04-02T13:23:29.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.634+0000 7f29d31bc640 1 -- 172.21.15.44:0/1354064481 wait complete. 2024-04-02T13:23:29.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.635+0000 7f29d31bc640 1 Processor -- start 2024-04-02T13:23:29.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.635+0000 7f29d31bc640 1 -- start start 2024-04-02T13:23:29.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.636+0000 7f29d31bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29cc0d1150 0x7f29cc0c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.636+0000 7f29d31bc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29cc15e140 0x7f29cc0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.636+0000 7f29d31bc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29cc15e520 0x7f29cc14f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.636+0000 7f29d31bc640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc0d2b70 con 0x7f29cc15e520 2024-04-02T13:23:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.636+0000 7f29d31bc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc0d4ee0 con 0x7f29cc15e140 2024-04-02T13:23:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.636+0000 7f29d31bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f29cc0d2cf0 con 0x7f29cc0d1150 2024-04-02T13:23:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.636+0000 7f29d1759640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29cc15e520 0x7f29cc14f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.636+0000 7f29d1759640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29cc15e520 0x7f29cc14f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56416/0 (socket says 172.21.15.44:56416) 2024-04-02T13:23:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.636+0000 7f29d1759640 1 -- 172.21.15.44:0/1057937046 learned_addr learned my addr 172.21.15.44:0/1057937046 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.636+0000 7f29cbfff640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29cc15e140 0x7f29cc0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.636+0000 7f29d0f58640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29cc0d1150 0x7f29cc0c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.636+0000 7f29d1759640 1 -- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29cc0d1150 msgr2=0x7f29cc0c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.636+0000 7f29d1759640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29cc0d1150 0x7f29cc0c7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.637+0000 7f29d1759640 1 -- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29cc15e140 msgr2=0x7f29cc0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.637+0000 7f29d1759640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29cc15e140 0x7f29cc0cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.637+0000 7f29d1759640 1 -- 172.21.15.44:0/1057937046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29cc14f9e0 con 0x7f29cc15e520 2024-04-02T13:23:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.637+0000 7f29d1759640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29cc15e520 0x7f29cc14f4a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009870 tx=0x7f29bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.637+0000 7f29c97fa640 1 -- 172.21.15.44:0/1057937046 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc048070 con 0x7f29cc15e520 2024-04-02T13:23:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.637+0000 7f29c97fa640 1 -- 172.21.15.44:0/1057937046 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29bc030640 con 0x7f29cc15e520 2024-04-02T13:23:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.637+0000 7f29d31bc640 1 -- 172.21.15.44:0/1057937046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29cc0c8220 con 0x7f29cc15e520 2024-04-02T13:23:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.637+0000 7f29c97fa640 1 -- 172.21.15.44:0/1057937046 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc034040 con 0x7f29cc15e520 2024-04-02T13:23:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.637+0000 7f29d31bc640 1 -- 172.21.15.44:0/1057937046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29cc0c87e0 con 0x7f29cc15e520 2024-04-02T13:23:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.638+0000 7f29d31bc640 1 -- 172.21.15.44:0/1057937046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29cc0684f0 con 0x7f29cc15e520 2024-04-02T13:23:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.639+0000 7f29c97fa640 1 -- 172.21.15.44:0/1057937046 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f29bc04b070 con 0x7f29cc15e520 2024-04-02T13:23:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.639+0000 7f29c97fa640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29b806c8f0 0x7f29b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.639+0000 7f29c97fa640 1 -- 172.21.15.44:0/1057937046 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f29bc0ad9e0 con 0x7f29cc15e520 2024-04-02T13:23:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.639+0000 7f29d0f58640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29b806c8f0 0x7f29b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.640+0000 7f29d0f58640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29b806c8f0 0x7f29b806edb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f29c40113e0 tx=0x7f29c400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:29.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.643+0000 7f29c97fa640 1 -- 172.21.15.44:0/1057937046 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f29bc07b6c0 con 0x7f29cc15e520 2024-04-02T13:23:29.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.823+0000 7f29d31bc640 1 -- 172.21.15.44:0/1057937046 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f29cc0c7600 con 0x7f29b806c8f0 2024-04-02T13:23:29.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.830+0000 7f29c97fa640 1 -- 172.21.15.44:0/1057937046 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252036 (secure 0 0 0) 0x7f29cc0c7600 con 0x7f29b806c8f0 2024-04-02T13:23:29.834 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:29.835 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:23:29.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.833+0000 7f29d31bc640 1 -- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29b806c8f0 msgr2=0x7f29b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.833+0000 7f29d31bc640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29b806c8f0 0x7f29b806edb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f29c40113e0 tx=0x7f29c400b040 comp rx=0 tx=0).stop 2024-04-02T13:23:29.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.833+0000 7f29d31bc640 1 -- 172.21.15.44:0/1057937046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29cc15e520 msgr2=0x7f29cc14f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.833+0000 7f29d31bc640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29cc15e520 0x7f29cc14f4a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009870 tx=0x7f29bc004730 comp rx=0 tx=0).stop 2024-04-02T13:23:29.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.833+0000 7f29d31bc640 1 -- 172.21.15.44:0/1057937046 shutdown_connections 2024-04-02T13:23:29.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.833+0000 7f29d31bc640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29b806c8f0 0x7f29b806edb0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.833+0000 7f29d31bc640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29cc15e520 0x7f29cc14f4a0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.833+0000 7f29d31bc640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29cc15e140 0x7f29cc0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.833+0000 7f29d31bc640 1 --2- 172.21.15.44:0/1057937046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29cc0d1150 0x7f29cc0c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.833+0000 7f29d31bc640 1 -- 172.21.15.44:0/1057937046 >> 172.21.15.44:0/1057937046 conn(0x7f29cc09d390 msgr2=0x7f29cc0ceb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:29.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.833+0000 7f29d31bc640 1 -- 172.21.15.44:0/1057937046 shutdown_connections 2024-04-02T13:23:29.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.833+0000 7f29d31bc640 1 -- 172.21.15.44:0/1057937046 wait complete. 2024-04-02T13:23:29.845 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-04-02T13:23:28.918876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":231904,"kb_used_data":7368,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754742816,"statfs":{"total":773094113280,"available":772856643584,"internally_reserved":0,"allocated":7544832,"data_stored":4025292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.480723"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053223+0000","last_change":"2024-04-02T13:23:25.469431+0000","last_active":"2024-04-02T13:23:27.053223+0000","last_peered":"2024-04-02T13:23:27.053223+0000","last_clean":"2024-04-02T13:23:27.053223+0000","last_became_active":"2024-04-02T13:23:25.469157+0000","last_became_peered":"2024-04-02T13:23:25.469157+0000","last_unstale":"2024-04-02T13:23:27.053223+0000","last_undegraded":"2024-04-02T13:23:27.053223+0000","last_fullsized":"2024-04-02T13:23:27.053223+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:08.239549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052716+0000","last_change":"2024-04-02T13:23:25.461364+0000","last_active":"2024-04-02T13:23:27.052716+0000","last_peered":"2024-04-02T13:23:27.052716+0000","last_clean":"2024-04-02T13:23:27.052716+0000","last_became_active":"2024-04-02T13:23:25.461071+0000","last_became_peered":"2024-04-02T13:23:25.461071+0000","last_unstale":"2024-04-02T13:23:27.052716+0000","last_undegraded":"2024-04-02T13:23:27.052716+0000","last_fullsized":"2024-04-02T13:23:27.052716+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:29:14.631137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462494+0000","last_change":"2024-04-02T13:23:25.467141+0000","last_active":"2024-04-02T13:23:26.462494+0000","last_peered":"2024-04-02T13:23:26.462494+0000","last_clean":"2024-04-02T13:23:26.462494+0000","last_became_active":"2024-04-02T13:23:25.466843+0000","last_became_peered":"2024-04-02T13:23:25.466843+0000","last_unstale":"2024-04-02T13:23:26.462494+0000","last_undegraded":"2024-04-02T13:23:26.462494+0000","last_fullsized":"2024-04-02T13:23:26.462494+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:12.317864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463849+0000","last_change":"2024-04-02T13:23:25.464706+0000","last_active":"2024-04-02T13:23:26.463849+0000","last_peered":"2024-04-02T13:23:26.463849+0000","last_clean":"2024-04-02T13:23:26.463849+0000","last_became_active":"2024-04-02T13:23:25.464429+0000","last_became_peered":"2024-04-02T13:23:25.464429+0000","last_unstale":"2024-04-02T13:23:26.463849+0000","last_undegraded":"2024-04-02T13:23:26.463849+0000","last_fullsized":"2024-04-02T13:23:26.463849+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:33:53.740788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052710+0000","last_change":"2024-04-02T13:23:25.466279+0000","last_active":"2024-04-02T13:23:27.052710+0000","last_peered":"2024-04-02T13:23:27.052710+0000","last_clean":"2024-04-02T13:23:27.052710+0000","last_became_active":"2024-04-02T13:23:25.463166+0000","last_became_peered":"2024-04-02T13:23:25.463166+0000","last_unstale":"2024-04-02T13:23:27.052710+0000","last_undegraded":"2024-04-02T13:23:27.052710+0000","last_fullsized":"2024-04-02T13:23:27.052710+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:55.928767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053460+0000","last_change":"2024-04-02T13:23:25.461164+0000","last_active":"2024-04-02T13:23:27.053460+0000","last_peered":"2024-04-02T13:23:27.053460+0000","last_clean":"2024-04-02T13:23:27.053460+0000","last_became_active":"2024-04-02T13:23:25.460895+0000","last_became_peered":"2024-04-02T13:23:25.460895+0000","last_unstale":"2024-04-02T13:23:27.053460+0000","last_undegraded":"2024-04-02T13:23:27.053460+0000","last_fullsized":"2024-04-02T13:23:27.053460+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:59:08.284683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464835+0000","last_change":"2024-04-02T13:23:25.462561+0000","last_active":"2024-04-02T13:23:26.464835+0000","last_peered":"2024-04-02T13:23:26.464835+0000","last_clean":"2024-04-02T13:23:26.464835+0000","last_became_active":"2024-04-02T13:23:25.462285+0000","last_became_peered":"2024-04-02T13:23:25.462285+0000","last_unstale":"2024-04-02T13:23:26.464835+0000","last_undegraded":"2024-04-02T13:23:26.464835+0000","last_fullsized":"2024-04-02T13:23:26.464835+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:12:01.325498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462155+0000","last_change":"2024-04-02T13:23:25.459669+0000","last_active":"2024-04-02T13:23:26.462155+0000","last_peered":"2024-04-02T13:23:26.462155+0000","last_clean":"2024-04-02T13:23:26.462155+0000","last_became_active":"2024-04-02T13:23:25.459387+0000","last_became_peered":"2024-04-02T13:23:25.459387+0000","last_unstale":"2024-04-02T13:23:26.462155+0000","last_undegraded":"2024-04-02T13:23:26.462155+0000","last_fullsized":"2024-04-02T13:23:26.462155+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:42:26.126912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462200+0000","last_change":"2024-04-02T13:23:25.463425+0000","last_active":"2024-04-02T13:23:26.462200+0000","last_peered":"2024-04-02T13:23:26.462200+0000","last_clean":"2024-04-02T13:23:26.462200+0000","last_became_active":"2024-04-02T13:23:25.463167+0000","last_became_peered":"2024-04-02T13:23:25.463167+0000","last_unstale":"2024-04-02T13:23:26.462200+0000","last_undegraded":"2024-04-02T13:23:26.462200+0000","last_fullsized":"2024-04-02T13:23:26.462200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:41:31.047808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462795+0000","last_change":"2024-04-02T13:23:25.468862+0000","last_active":"2024-04-02T13:23:26.462795+0000","last_peered":"2024-04-02T13:23:26.462795+0000","last_clean":"2024-04-02T13:23:26.462795+0000","last_became_active":"2024-04-02T13:23:25.468576+0000","last_became_peered":"2024-04-02T13:23:25.468576+0000","last_unstale":"2024-04-02T13:23:26.462795+0000","last_undegraded":"2024-04-02T13:23:26.462795+0000","last_fullsized":"2024-04-02T13:23:26.462795+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:41.526796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464034+0000","last_change":"2024-04-02T13:23:25.470138+0000","last_active":"2024-04-02T13:23:26.464034+0000","last_peered":"2024-04-02T13:23:26.464034+0000","last_clean":"2024-04-02T13:23:26.464034+0000","last_became_active":"2024-04-02T13:23:25.468618+0000","last_became_peered":"2024-04-02T13:23:25.468618+0000","last_unstale":"2024-04-02T13:23:26.464034+0000","last_undegraded":"2024-04-02T13:23:26.464034+0000","last_fullsized":"2024-04-02T13:23:26.464034+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:56:05.702719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463248+0000","last_change":"2024-04-02T13:23:25.460450+0000","last_active":"2024-04-02T13:23:26.463248+0000","last_peered":"2024-04-02T13:23:26.463248+0000","last_clean":"2024-04-02T13:23:26.463248+0000","last_became_active":"2024-04-02T13:23:25.459878+0000","last_became_peered":"2024-04-02T13:23:25.459878+0000","last_unstale":"2024-04-02T13:23:26.463248+0000","last_undegraded":"2024-04-02T13:23:26.463248+0000","last_fullsized":"2024-04-02T13:23:26.463248+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:50:07.988009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462453+0000","last_change":"2024-04-02T13:23:25.465881+0000","last_active":"2024-04-02T13:23:26.462453+0000","last_peered":"2024-04-02T13:23:26.462453+0000","last_clean":"2024-04-02T13:23:26.462453+0000","last_became_active":"2024-04-02T13:23:25.465078+0000","last_became_peered":"2024-04-02T13:23:25.465078+0000","last_unstale":"2024-04-02T13:23:26.462453+0000","last_undegraded":"2024-04-02T13:23:26.462453+0000","last_fullsized":"2024-04-02T13:23:26.462453+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:24.343720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462168+0000","last_change":"2024-04-02T13:23:25.463214+0000","last_active":"2024-04-02T13:23:26.462168+0000","last_peered":"2024-04-02T13:23:26.462168+0000","last_clean":"2024-04-02T13:23:26.462168+0000","last_became_active":"2024-04-02T13:23:25.462714+0000","last_became_peered":"2024-04-02T13:23:25.462714+0000","last_unstale":"2024-04-02T13:23:26.462168+0000","last_undegraded":"2024-04-02T13:23:26.462168+0000","last_fullsized":"2024-04-02T13:23:26.462168+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:48:09.589536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462446+0000","last_change":"2024-04-02T13:23:25.469693+0000","last_active":"2024-04-02T13:23:26.462446+0000","last_peered":"2024-04-02T13:23:26.462446+0000","last_clean":"2024-04-02T13:23:26.462446+0000","last_became_active":"2024-04-02T13:23:25.469403+0000","last_became_peered":"2024-04-02T13:23:25.469403+0000","last_unstale":"2024-04-02T13:23:26.462446+0000","last_undegraded":"2024-04-02T13:23:26.462446+0000","last_fullsized":"2024-04-02T13:23:26.462446+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:19.829312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464833+0000","last_change":"2024-04-02T13:23:25.469379+0000","last_active":"2024-04-02T13:23:26.464833+0000","last_peered":"2024-04-02T13:23:26.464833+0000","last_clean":"2024-04-02T13:23:26.464833+0000","last_became_active":"2024-04-02T13:23:25.468644+0000","last_became_peered":"2024-04-02T13:23:25.468644+0000","last_unstale":"2024-04-02T13:23:26.464833+0000","last_undegraded":"2024-04-02T13:23:26.464833+0000","last_fullsized":"2024-04-02T13:23:26.464833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:18:06.672051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053379+0000","last_change":"2024-04-02T13:23:25.476808+0000","last_active":"2024-04-02T13:23:27.053379+0000","last_peered":"2024-04-02T13:23:27.053379+0000","last_clean":"2024-04-02T13:23:27.053379+0000","last_became_active":"2024-04-02T13:23:25.476528+0000","last_became_peered":"2024-04-02T13:23:25.476528+0000","last_unstale":"2024-04-02T13:23:27.053379+0000","last_undegraded":"2024-04-02T13:23:27.053379+0000","last_fullsized":"2024-04-02T13:23:27.053379+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:15:15.333981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464448+0000","last_change":"2024-04-02T13:23:25.465204+0000","last_active":"2024-04-02T13:23:26.464448+0000","last_peered":"2024-04-02T13:23:26.464448+0000","last_clean":"2024-04-02T13:23:26.464448+0000","last_became_active":"2024-04-02T13:23:25.464913+0000","last_became_peered":"2024-04-02T13:23:25.464913+0000","last_unstale":"2024-04-02T13:23:26.464448+0000","last_undegraded":"2024-04-02T13:23:26.464448+0000","last_fullsized":"2024-04-02T13:23:26.464448+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:23.669394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053012+0000","last_change":"2024-04-02T13:23:25.465160+0000","last_active":"2024-04-02T13:23:27.053012+0000","last_peered":"2024-04-02T13:23:27.053012+0000","last_clean":"2024-04-02T13:23:27.053012+0000","last_became_active":"2024-04-02T13:23:25.464760+0000","last_became_peered":"2024-04-02T13:23:25.464760+0000","last_unstale":"2024-04-02T13:23:27.053012+0000","last_undegraded":"2024-04-02T13:23:27.053012+0000","last_fullsized":"2024-04-02T13:23:27.053012+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:29:14.631137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053362+0000","last_change":"2024-04-02T13:23:25.463632+0000","last_active":"2024-04-02T13:23:27.053362+0000","last_peered":"2024-04-02T13:23:27.053362+0000","last_clean":"2024-04-02T13:23:27.053362+0000","last_became_active":"2024-04-02T13:23:25.463340+0000","last_became_peered":"2024-04-02T13:23:25.463340+0000","last_unstale":"2024-04-02T13:23:27.053362+0000","last_undegraded":"2024-04-02T13:23:27.053362+0000","last_fullsized":"2024-04-02T13:23:27.053362+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:38.002816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462586+0000","last_change":"2024-04-02T13:23:25.466118+0000","last_active":"2024-04-02T13:23:26.462586+0000","last_peered":"2024-04-02T13:23:26.462586+0000","last_clean":"2024-04-02T13:23:26.462586+0000","last_became_active":"2024-04-02T13:23:25.465634+0000","last_became_peered":"2024-04-02T13:23:25.465634+0000","last_unstale":"2024-04-02T13:23:26.462586+0000","last_undegraded":"2024-04-02T13:23:26.462586+0000","last_fullsized":"2024-04-02T13:23:26.462586+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:32.179030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053035+0000","last_change":"2024-04-02T13:23:25.459181+0000","last_active":"2024-04-02T13:23:27.053035+0000","last_peered":"2024-04-02T13:23:27.053035+0000","last_clean":"2024-04-02T13:23:27.053035+0000","last_became_active":"2024-04-02T13:23:25.458914+0000","last_became_peered":"2024-04-02T13:23:25.458914+0000","last_unstale":"2024-04-02T13:23:27.053035+0000","last_undegraded":"2024-04-02T13:23:27.053035+0000","last_fullsized":"2024-04-02T13:23:27.053035+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:47:07.091547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052380+0000","last_change":"2024-04-02T13:23:25.458589+0000","last_active":"2024-04-02T13:23:27.052380+0000","last_peered":"2024-04-02T13:23:27.052380+0000","last_clean":"2024-04-02T13:23:27.052380+0000","last_became_active":"2024-04-02T13:23:25.458288+0000","last_became_peered":"2024-04-02T13:23:25.458288+0000","last_unstale":"2024-04-02T13:23:27.052380+0000","last_undegraded":"2024-04-02T13:23:27.052380+0000","last_fullsized":"2024-04-02T13:23:27.052380+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:30:05.004986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462096+0000","last_change":"2024-04-02T13:23:25.466717+0000","last_active":"2024-04-02T13:23:26.462096+0000","last_peered":"2024-04-02T13:23:26.462096+0000","last_clean":"2024-04-02T13:23:26.462096+0000","last_became_active":"2024-04-02T13:23:25.466413+0000","last_became_peered":"2024-04-02T13:23:25.466413+0000","last_unstale":"2024-04-02T13:23:26.462096+0000","last_undegraded":"2024-04-02T13:23:26.462096+0000","last_fullsized":"2024-04-02T13:23:26.462096+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:50:07.988009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463354+0000","last_change":"2024-04-02T13:23:25.462058+0000","last_active":"2024-04-02T13:23:26.463354+0000","last_peered":"2024-04-02T13:23:26.463354+0000","last_clean":"2024-04-02T13:23:26.463354+0000","last_became_active":"2024-04-02T13:23:25.461817+0000","last_became_peered":"2024-04-02T13:23:25.461817+0000","last_unstale":"2024-04-02T13:23:26.463354+0000","last_undegraded":"2024-04-02T13:23:26.463354+0000","last_fullsized":"2024-04-02T13:23:26.463354+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:48:09.589536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462480+0000","last_change":"2024-04-02T13:23:25.464229+0000","last_active":"2024-04-02T13:23:26.462480+0000","last_peered":"2024-04-02T13:23:26.462480+0000","last_clean":"2024-04-02T13:23:26.462480+0000","last_became_active":"2024-04-02T13:23:25.463967+0000","last_became_peered":"2024-04-02T13:23:25.463967+0000","last_unstale":"2024-04-02T13:23:26.462480+0000","last_undegraded":"2024-04-02T13:23:26.462480+0000","last_fullsized":"2024-04-02T13:23:26.462480+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:48:45.430677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464184+0000","last_change":"2024-04-02T13:23:25.463865+0000","last_active":"2024-04-02T13:23:26.464184+0000","last_peered":"2024-04-02T13:23:26.464184+0000","last_clean":"2024-04-02T13:23:26.464184+0000","last_became_active":"2024-04-02T13:23:25.463599+0000","last_became_peered":"2024-04-02T13:23:25.463599+0000","last_unstale":"2024-04-02T13:23:26.464184+0000","last_undegraded":"2024-04-02T13:23:26.464184+0000","last_fullsized":"2024-04-02T13:23:26.464184+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:21:55.421252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052964+0000","last_change":"2024-04-02T13:23:25.469447+0000","last_active":"2024-04-02T13:23:27.052964+0000","last_peered":"2024-04-02T13:23:27.052964+0000","last_clean":"2024-04-02T13:23:27.052964+0000","last_became_active":"2024-04-02T13:23:25.469026+0000","last_became_peered":"2024-04-02T13:23:25.469026+0000","last_unstale":"2024-04-02T13:23:27.052964+0000","last_undegraded":"2024-04-02T13:23:27.052964+0000","last_fullsized":"2024-04-02T13:23:27.052964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:04:33.058654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462966+0000","last_change":"2024-04-02T13:23:25.465032+0000","last_active":"2024-04-02T13:23:26.462966+0000","last_peered":"2024-04-02T13:23:26.462966+0000","last_clean":"2024-04-02T13:23:26.462966+0000","last_became_active":"2024-04-02T13:23:25.463803+0000","last_became_peered":"2024-04-02T13:23:25.463803+0000","last_unstale":"2024-04-02T13:23:26.462966+0000","last_undegraded":"2024-04-02T13:23:26.462966+0000","last_fullsized":"2024-04-02T13:23:26.462966+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:07:35.829689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463021+0000","last_change":"2024-04-02T13:23:25.460164+0000","last_active":"2024-04-02T13:23:26.463021+0000","last_peered":"2024-04-02T13:23:26.463021+0000","last_clean":"2024-04-02T13:23:26.463021+0000","last_became_active":"2024-04-02T13:23:25.459883+0000","last_became_peered":"2024-04-02T13:23:25.459883+0000","last_unstale":"2024-04-02T13:23:26.463021+0000","last_undegraded":"2024-04-02T13:23:26.463021+0000","last_fullsized":"2024-04-02T13:23:26.463021+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:54.435494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463647+0000","last_change":"2024-04-02T13:23:25.470481+0000","last_active":"2024-04-02T13:23:26.463647+0000","last_peered":"2024-04-02T13:23:26.463647+0000","last_clean":"2024-04-02T13:23:26.463647+0000","last_became_active":"2024-04-02T13:23:25.469773+0000","last_became_peered":"2024-04-02T13:23:25.469773+0000","last_unstale":"2024-04-02T13:23:26.463647+0000","last_undegraded":"2024-04-02T13:23:26.463647+0000","last_fullsized":"2024-04-02T13:23:26.463647+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:10:22.691623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052259+0000","last_change":"2024-04-02T13:23:25.462340+0000","last_active":"2024-04-02T13:23:27.052259+0000","last_peered":"2024-04-02T13:23:27.052259+0000","last_clean":"2024-04-02T13:23:27.052259+0000","last_became_active":"2024-04-02T13:23:25.462074+0000","last_became_peered":"2024-04-02T13:23:25.462074+0000","last_unstale":"2024-04-02T13:23:27.052259+0000","last_undegraded":"2024-04-02T13:23:27.052259+0000","last_fullsized":"2024-04-02T13:23:27.052259+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:00:37.296114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052934+0000","last_change":"2024-04-02T13:23:25.466488+0000","last_active":"2024-04-02T13:23:27.052934+0000","last_peered":"2024-04-02T13:23:27.052934+0000","last_clean":"2024-04-02T13:23:27.052934+0000","last_became_active":"2024-04-02T13:23:25.466218+0000","last_became_peered":"2024-04-02T13:23:25.466218+0000","last_unstale":"2024-04-02T13:23:27.052934+0000","last_undegraded":"2024-04-02T13:23:27.052934+0000","last_fullsized":"2024-04-02T13:23:27.052934+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:02:00.562229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"64'4","reported_seq":18,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053520+0000","last_change":"2024-04-02T13:23:24.456383+0000","last_active":"2024-04-02T13:23:27.053520+0000","last_peered":"2024-04-02T13:23:27.053520+0000","last_clean":"2024-04-02T13:23:27.053520+0000","last_became_active":"2024-04-02T13:23:24.456093+0000","last_became_peered":"2024-04-02T13:23:24.456093+0000","last_unstale":"2024-04-02T13:23:27.053520+0000","last_undegraded":"2024-04-02T13:23:27.053520+0000","last_fullsized":"2024-04-02T13:23:27.053520+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:33:52.736716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464365+0000","last_change":"2024-04-02T13:23:25.471025+0000","last_active":"2024-04-02T13:23:26.464365+0000","last_peered":"2024-04-02T13:23:26.464365+0000","last_clean":"2024-04-02T13:23:26.464365+0000","last_became_active":"2024-04-02T13:23:25.470669+0000","last_became_peered":"2024-04-02T13:23:25.470669+0000","last_unstale":"2024-04-02T13:23:26.464365+0000","last_undegraded":"2024-04-02T13:23:26.464365+0000","last_fullsized":"2024-04-02T13:23:26.464365+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:47.944508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053764+0000","last_change":"2024-04-02T13:23:24.459158+0000","last_active":"2024-04-02T13:23:27.053764+0000","last_peered":"2024-04-02T13:23:27.053764+0000","last_clean":"2024-04-02T13:23:27.053764+0000","last_became_active":"2024-04-02T13:23:24.458790+0000","last_became_peered":"2024-04-02T13:23:24.458790+0000","last_unstale":"2024-04-02T13:23:27.053764+0000","last_undegraded":"2024-04-02T13:23:27.053764+0000","last_fullsized":"2024-04-02T13:23:27.053764+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:42:25.122839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053831+0000","last_change":"2024-04-02T13:23:25.463933+0000","last_active":"2024-04-02T13:23:27.053831+0000","last_peered":"2024-04-02T13:23:27.053831+0000","last_clean":"2024-04-02T13:23:27.053831+0000","last_became_active":"2024-04-02T13:23:25.463612+0000","last_became_peered":"2024-04-02T13:23:25.463612+0000","last_unstale":"2024-04-02T13:23:27.053831+0000","last_undegraded":"2024-04-02T13:23:27.053831+0000","last_fullsized":"2024-04-02T13:23:27.053831+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:41.526796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052947+0000","last_change":"2024-04-02T13:23:24.456996+0000","last_active":"2024-04-02T13:23:27.052947+0000","last_peered":"2024-04-02T13:23:27.052947+0000","last_clean":"2024-04-02T13:23:27.052947+0000","last_became_active":"2024-04-02T13:23:24.456720+0000","last_became_peered":"2024-04-02T13:23:24.456720+0000","last_unstale":"2024-04-02T13:23:27.052947+0000","last_undegraded":"2024-04-02T13:23:27.052947+0000","last_fullsized":"2024-04-02T13:23:27.052947+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:48:44.426604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462339+0000","last_change":"2024-04-02T13:23:25.471124+0000","last_active":"2024-04-02T13:23:26.462339+0000","last_peered":"2024-04-02T13:23:26.462339+0000","last_clean":"2024-04-02T13:23:26.462339+0000","last_became_active":"2024-04-02T13:23:25.470234+0000","last_became_peered":"2024-04-02T13:23:25.470234+0000","last_unstale":"2024-04-02T13:23:26.462339+0000","last_undegraded":"2024-04-02T13:23:26.462339+0000","last_fullsized":"2024-04-02T13:23:26.462339+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:00:37.296114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053413+0000","last_change":"2024-04-02T13:23:24.455277+0000","last_active":"2024-04-02T13:23:27.053413+0000","last_peered":"2024-04-02T13:23:27.053413+0000","last_clean":"2024-04-02T13:23:27.053413+0000","last_became_active":"2024-04-02T13:23:24.452815+0000","last_became_peered":"2024-04-02T13:23:24.452815+0000","last_unstale":"2024-04-02T13:23:27.053413+0000","last_undegraded":"2024-04-02T13:23:27.053413+0000","last_fullsized":"2024-04-02T13:23:27.053413+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:31.174958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464833+0000","last_change":"2024-04-02T13:23:25.466125+0000","last_active":"2024-04-02T13:23:26.464833+0000","last_peered":"2024-04-02T13:23:26.464833+0000","last_clean":"2024-04-02T13:23:26.464833+0000","last_became_active":"2024-04-02T13:23:25.465847+0000","last_became_peered":"2024-04-02T13:23:25.465847+0000","last_unstale":"2024-04-02T13:23:26.464833+0000","last_undegraded":"2024-04-02T13:23:26.464833+0000","last_fullsized":"2024-04-02T13:23:26.464833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:37.703991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053263+0000","last_change":"2024-04-02T13:23:24.452751+0000","last_active":"2024-04-02T13:23:27.053263+0000","last_peered":"2024-04-02T13:23:27.053263+0000","last_clean":"2024-04-02T13:23:27.053263+0000","last_became_active":"2024-04-02T13:23:24.452492+0000","last_became_peered":"2024-04-02T13:23:24.452492+0000","last_unstale":"2024-04-02T13:23:27.053263+0000","last_undegraded":"2024-04-02T13:23:27.053263+0000","last_fullsized":"2024-04-02T13:23:27.053263+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:12:40.522723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052600+0000","last_change":"2024-04-02T13:23:25.476798+0000","last_active":"2024-04-02T13:23:27.052600+0000","last_peered":"2024-04-02T13:23:27.052600+0000","last_clean":"2024-04-02T13:23:27.052600+0000","last_became_active":"2024-04-02T13:23:25.476538+0000","last_became_peered":"2024-04-02T13:23:25.476538+0000","last_unstale":"2024-04-02T13:23:27.052600+0000","last_undegraded":"2024-04-02T13:23:27.052600+0000","last_fullsized":"2024-04-02T13:23:27.052600+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:00.139996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053061+0000","last_change":"2024-04-02T13:23:24.451654+0000","last_active":"2024-04-02T13:23:27.053061+0000","last_peered":"2024-04-02T13:23:27.053061+0000","last_clean":"2024-04-02T13:23:27.053061+0000","last_became_active":"2024-04-02T13:23:24.451084+0000","last_became_peered":"2024-04-02T13:23:24.451084+0000","last_unstale":"2024-04-02T13:23:27.053061+0000","last_undegraded":"2024-04-02T13:23:27.053061+0000","last_fullsized":"2024-04-02T13:23:27.053061+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:00:36.292042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052648+0000","last_change":"2024-04-02T13:23:25.468308+0000","last_active":"2024-04-02T13:23:27.052648+0000","last_peered":"2024-04-02T13:23:27.052648+0000","last_clean":"2024-04-02T13:23:27.052648+0000","last_became_active":"2024-04-02T13:23:25.468029+0000","last_became_peered":"2024-04-02T13:23:25.468029+0000","last_unstale":"2024-04-02T13:23:27.052648+0000","last_undegraded":"2024-04-02T13:23:27.052648+0000","last_fullsized":"2024-04-02T13:23:27.052648+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:47.944508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052515+0000","last_change":"2024-04-02T13:23:24.458224+0000","last_active":"2024-04-02T13:23:27.052515+0000","last_peered":"2024-04-02T13:23:27.052515+0000","last_clean":"2024-04-02T13:23:27.052515+0000","last_became_active":"2024-04-02T13:23:24.457941+0000","last_became_peered":"2024-04-02T13:23:24.457941+0000","last_unstale":"2024-04-02T13:23:27.052515+0000","last_undegraded":"2024-04-02T13:23:27.052515+0000","last_fullsized":"2024-04-02T13:23:27.052515+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:59:07.280610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053263+0000","last_change":"2024-04-02T13:23:25.462821+0000","last_active":"2024-04-02T13:23:27.053263+0000","last_peered":"2024-04-02T13:23:27.053263+0000","last_clean":"2024-04-02T13:23:27.053263+0000","last_became_active":"2024-04-02T13:23:25.462554+0000","last_became_peered":"2024-04-02T13:23:25.462554+0000","last_unstale":"2024-04-02T13:23:27.053263+0000","last_undegraded":"2024-04-02T13:23:27.053263+0000","last_fullsized":"2024-04-02T13:23:27.053263+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:13:19.036189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052390+0000","last_change":"2024-04-02T13:23:24.457917+0000","last_active":"2024-04-02T13:23:27.052390+0000","last_peered":"2024-04-02T13:23:27.052390+0000","last_clean":"2024-04-02T13:23:27.052390+0000","last_became_active":"2024-04-02T13:23:24.457121+0000","last_became_peered":"2024-04-02T13:23:24.457121+0000","last_unstale":"2024-04-02T13:23:27.052390+0000","last_undegraded":"2024-04-02T13:23:27.052390+0000","last_fullsized":"2024-04-02T13:23:27.052390+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:23:36.457600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053104+0000","last_change":"2024-04-02T13:23:25.468788+0000","last_active":"2024-04-02T13:23:27.053104+0000","last_peered":"2024-04-02T13:23:27.053104+0000","last_clean":"2024-04-02T13:23:27.053104+0000","last_became_active":"2024-04-02T13:23:25.468321+0000","last_became_peered":"2024-04-02T13:23:25.468321+0000","last_unstale":"2024-04-02T13:23:27.053104+0000","last_undegraded":"2024-04-02T13:23:27.053104+0000","last_fullsized":"2024-04-02T13:23:27.053104+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:03:49.303465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462281+0000","last_change":"2024-04-02T13:23:24.444565+0000","last_active":"2024-04-02T13:23:26.462281+0000","last_peered":"2024-04-02T13:23:26.462281+0000","last_clean":"2024-04-02T13:23:26.462281+0000","last_became_active":"2024-04-02T13:23:24.444311+0000","last_became_peered":"2024-04-02T13:23:24.444311+0000","last_unstale":"2024-04-02T13:23:26.462281+0000","last_undegraded":"2024-04-02T13:23:26.462281+0000","last_fullsized":"2024-04-02T13:23:26.462281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:07.235476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052820+0000","last_change":"2024-04-02T13:23:25.461614+0000","last_active":"2024-04-02T13:23:27.052820+0000","last_peered":"2024-04-02T13:23:27.052820+0000","last_clean":"2024-04-02T13:23:27.052820+0000","last_became_active":"2024-04-02T13:23:25.461353+0000","last_became_peered":"2024-04-02T13:23:25.461353+0000","last_unstale":"2024-04-02T13:23:27.052820+0000","last_undegraded":"2024-04-02T13:23:27.052820+0000","last_fullsized":"2024-04-02T13:23:27.052820+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:55:29.376845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464431+0000","last_change":"2024-04-02T13:23:24.457899+0000","last_active":"2024-04-02T13:23:26.464431+0000","last_peered":"2024-04-02T13:23:26.464431+0000","last_clean":"2024-04-02T13:23:26.464431+0000","last_became_active":"2024-04-02T13:23:24.457540+0000","last_became_peered":"2024-04-02T13:23:24.457540+0000","last_unstale":"2024-04-02T13:23:26.464431+0000","last_undegraded":"2024-04-02T13:23:26.464431+0000","last_fullsized":"2024-04-02T13:23:26.464431+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:34:05.703937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052955+0000","last_change":"2024-04-02T13:23:25.466861+0000","last_active":"2024-04-02T13:23:27.052955+0000","last_peered":"2024-04-02T13:23:27.052955+0000","last_clean":"2024-04-02T13:23:27.052955+0000","last_became_active":"2024-04-02T13:23:25.466425+0000","last_became_peered":"2024-04-02T13:23:25.466425+0000","last_unstale":"2024-04-02T13:23:27.052955+0000","last_undegraded":"2024-04-02T13:23:27.052955+0000","last_fullsized":"2024-04-02T13:23:27.052955+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:29.261268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"64'3","reported_seq":17,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462550+0000","last_change":"2024-04-02T13:23:24.444242+0000","last_active":"2024-04-02T13:23:26.462550+0000","last_peered":"2024-04-02T13:23:26.462550+0000","last_clean":"2024-04-02T13:23:26.462550+0000","last_became_active":"2024-04-02T13:23:24.443907+0000","last_became_peered":"2024-04-02T13:23:24.443907+0000","last_unstale":"2024-04-02T13:23:26.462550+0000","last_undegraded":"2024-04-02T13:23:26.462550+0000","last_fullsized":"2024-04-02T13:23:26.462550+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:06:37.459794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464256+0000","last_change":"2024-04-02T13:23:25.467436+0000","last_active":"2024-04-02T13:23:26.464256+0000","last_peered":"2024-04-02T13:23:26.464256+0000","last_clean":"2024-04-02T13:23:26.464256+0000","last_became_active":"2024-04-02T13:23:25.467129+0000","last_became_peered":"2024-04-02T13:23:25.467129+0000","last_unstale":"2024-04-02T13:23:26.464256+0000","last_undegraded":"2024-04-02T13:23:26.464256+0000","last_fullsized":"2024-04-02T13:23:26.464256+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:46.130146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"64'6","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053015+0000","last_change":"2024-04-02T13:23:24.455586+0000","last_active":"2024-04-02T13:23:27.053015+0000","last_peered":"2024-04-02T13:23:27.053015+0000","last_clean":"2024-04-02T13:23:27.053015+0000","last_became_active":"2024-04-02T13:23:24.455252+0000","last_became_peered":"2024-04-02T13:23:24.455252+0000","last_unstale":"2024-04-02T13:23:27.053015+0000","last_undegraded":"2024-04-02T13:23:27.053015+0000","last_fullsized":"2024-04-02T13:23:27.053015+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:33.668366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463948+0000","last_change":"2024-04-02T13:23:25.468250+0000","last_active":"2024-04-02T13:23:26.463948+0000","last_peered":"2024-04-02T13:23:26.463948+0000","last_clean":"2024-04-02T13:23:26.463948+0000","last_became_active":"2024-04-02T13:23:25.467952+0000","last_became_peered":"2024-04-02T13:23:25.467952+0000","last_unstale":"2024-04-02T13:23:26.463948+0000","last_undegraded":"2024-04-02T13:23:26.463948+0000","last_fullsized":"2024-04-02T13:23:26.463948+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:34.672439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463021+0000","last_change":"2024-04-02T13:23:24.454154+0000","last_active":"2024-04-02T13:23:26.463021+0000","last_peered":"2024-04-02T13:23:26.463021+0000","last_clean":"2024-04-02T13:23:26.463021+0000","last_became_active":"2024-04-02T13:23:24.453881+0000","last_became_peered":"2024-04-02T13:23:24.453881+0000","last_unstale":"2024-04-02T13:23:26.463021+0000","last_undegraded":"2024-04-02T13:23:26.463021+0000","last_fullsized":"2024-04-02T13:23:26.463021+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:28.257195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462141+0000","last_change":"2024-04-02T13:23:25.461190+0000","last_active":"2024-04-02T13:23:26.462141+0000","last_peered":"2024-04-02T13:23:26.462141+0000","last_clean":"2024-04-02T13:23:26.462141+0000","last_became_active":"2024-04-02T13:23:25.460911+0000","last_became_peered":"2024-04-02T13:23:25.460911+0000","last_unstale":"2024-04-02T13:23:26.462141+0000","last_undegraded":"2024-04-02T13:23:26.462141+0000","last_fullsized":"2024-04-02T13:23:26.462141+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:11:21.268539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463039+0000","last_change":"2024-04-02T13:23:24.452152+0000","last_active":"2024-04-02T13:23:26.463039+0000","last_peered":"2024-04-02T13:23:26.463039+0000","last_clean":"2024-04-02T13:23:26.463039+0000","last_became_active":"2024-04-02T13:23:24.451877+0000","last_became_peered":"2024-04-02T13:23:24.451877+0000","last_unstale":"2024-04-02T13:23:26.463039+0000","last_undegraded":"2024-04-02T13:23:26.463039+0000","last_fullsized":"2024-04-02T13:23:26.463039+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:46.940435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052645+0000","last_change":"2024-04-02T13:23:25.462012+0000","last_active":"2024-04-02T13:23:27.052645+0000","last_peered":"2024-04-02T13:23:27.052645+0000","last_clean":"2024-04-02T13:23:27.052645+0000","last_became_active":"2024-04-02T13:23:25.461738+0000","last_became_peered":"2024-04-02T13:23:25.461738+0000","last_unstale":"2024-04-02T13:23:27.052645+0000","last_undegraded":"2024-04-02T13:23:27.052645+0000","last_fullsized":"2024-04-02T13:23:27.052645+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:46.130146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462570+0000","last_change":"2024-04-02T13:23:24.450883+0000","last_active":"2024-04-02T13:23:26.462570+0000","last_peered":"2024-04-02T13:23:26.462570+0000","last_clean":"2024-04-02T13:23:26.462570+0000","last_became_active":"2024-04-02T13:23:24.450640+0000","last_became_peered":"2024-04-02T13:23:24.450640+0000","last_unstale":"2024-04-02T13:23:26.462570+0000","last_undegraded":"2024-04-02T13:23:26.462570+0000","last_fullsized":"2024-04-02T13:23:26.462570+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:59.135923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052931+0000","last_change":"2024-04-02T13:23:25.465196+0000","last_active":"2024-04-02T13:23:27.052931+0000","last_peered":"2024-04-02T13:23:27.052931+0000","last_clean":"2024-04-02T13:23:27.052931+0000","last_became_active":"2024-04-02T13:23:25.464890+0000","last_became_peered":"2024-04-02T13:23:25.464890+0000","last_unstale":"2024-04-02T13:23:27.052931+0000","last_undegraded":"2024-04-02T13:23:27.052931+0000","last_fullsized":"2024-04-02T13:23:27.052931+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:06:38.463867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"64'8","reported_seq":22,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464021+0000","last_change":"2024-04-02T13:23:24.459111+0000","last_active":"2024-04-02T13:23:26.464021+0000","last_peered":"2024-04-02T13:23:26.464021+0000","last_clean":"2024-04-02T13:23:26.464021+0000","last_became_active":"2024-04-02T13:23:24.458776+0000","last_became_peered":"2024-04-02T13:23:24.458776+0000","last_unstale":"2024-04-02T13:23:26.464021+0000","last_undegraded":"2024-04-02T13:23:26.464021+0000","last_fullsized":"2024-04-02T13:23:26.464021+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:27:53.204273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462501+0000","last_change":"2024-04-02T13:23:25.469323+0000","last_active":"2024-04-02T13:23:26.462501+0000","last_peered":"2024-04-02T13:23:26.462501+0000","last_clean":"2024-04-02T13:23:26.462501+0000","last_became_active":"2024-04-02T13:23:25.469044+0000","last_became_peered":"2024-04-02T13:23:25.469044+0000","last_unstale":"2024-04-02T13:23:26.462501+0000","last_undegraded":"2024-04-02T13:23:26.462501+0000","last_fullsized":"2024-04-02T13:23:26.462501+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:59:08.284683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463310+0000","last_change":"2024-04-02T13:23:24.447016+0000","last_active":"2024-04-02T13:23:26.463310+0000","last_peered":"2024-04-02T13:23:26.463310+0000","last_clean":"2024-04-02T13:23:26.463310+0000","last_became_active":"2024-04-02T13:23:24.446709+0000","last_became_peered":"2024-04-02T13:23:24.446709+0000","last_unstale":"2024-04-02T13:23:26.463310+0000","last_undegraded":"2024-04-02T13:23:26.463310+0000","last_fullsized":"2024-04-02T13:23:26.463310+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:54.924694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053068+0000","last_change":"2024-04-02T13:23:25.463056+0000","last_active":"2024-04-02T13:23:27.053068+0000","last_peered":"2024-04-02T13:23:27.053068+0000","last_clean":"2024-04-02T13:23:27.053068+0000","last_became_active":"2024-04-02T13:23:25.462734+0000","last_became_peered":"2024-04-02T13:23:25.462734+0000","last_unstale":"2024-04-02T13:23:27.053068+0000","last_undegraded":"2024-04-02T13:23:27.053068+0000","last_fullsized":"2024-04-02T13:23:27.053068+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:06:38.463867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462857+0000","last_change":"2024-04-02T13:23:24.456013+0000","last_active":"2024-04-02T13:23:26.462857+0000","last_peered":"2024-04-02T13:23:26.462857+0000","last_clean":"2024-04-02T13:23:26.462857+0000","last_became_active":"2024-04-02T13:23:24.455233+0000","last_became_peered":"2024-04-02T13:23:24.455233+0000","last_unstale":"2024-04-02T13:23:26.462857+0000","last_undegraded":"2024-04-02T13:23:26.462857+0000","last_fullsized":"2024-04-02T13:23:26.462857+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:03:48.299392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464376+0000","last_change":"2024-04-02T13:23:25.470692+0000","last_active":"2024-04-02T13:23:26.464376+0000","last_peered":"2024-04-02T13:23:26.464376+0000","last_clean":"2024-04-02T13:23:26.464376+0000","last_became_active":"2024-04-02T13:23:25.470429+0000","last_became_peered":"2024-04-02T13:23:25.470429+0000","last_unstale":"2024-04-02T13:23:26.464376+0000","last_undegraded":"2024-04-02T13:23:26.464376+0000","last_fullsized":"2024-04-02T13:23:26.464376+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:55.928767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462658+0000","last_change":"2024-04-02T13:23:24.453950+0000","last_active":"2024-04-02T13:23:26.462658+0000","last_peered":"2024-04-02T13:23:26.462658+0000","last_clean":"2024-04-02T13:23:26.462658+0000","last_became_active":"2024-04-02T13:23:24.453635+0000","last_became_peered":"2024-04-02T13:23:24.453635+0000","last_unstale":"2024-04-02T13:23:26.462658+0000","last_undegraded":"2024-04-02T13:23:26.462658+0000","last_fullsized":"2024-04-02T13:23:26.462658+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:04:32.054581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052919+0000","last_change":"2024-04-02T13:23:25.461765+0000","last_active":"2024-04-02T13:23:27.052919+0000","last_peered":"2024-04-02T13:23:27.052919+0000","last_clean":"2024-04-02T13:23:27.052919+0000","last_became_active":"2024-04-02T13:23:25.461499+0000","last_became_peered":"2024-04-02T13:23:25.461499+0000","last_unstale":"2024-04-02T13:23:27.052919+0000","last_undegraded":"2024-04-02T13:23:27.052919+0000","last_fullsized":"2024-04-02T13:23:27.052919+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:07:17.900627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053334+0000","last_change":"2024-04-02T13:23:24.453957+0000","last_active":"2024-04-02T13:23:27.053334+0000","last_peered":"2024-04-02T13:23:27.053334+0000","last_clean":"2024-04-02T13:23:27.053334+0000","last_became_active":"2024-04-02T13:23:24.453710+0000","last_became_peered":"2024-04-02T13:23:24.453710+0000","last_unstale":"2024-04-02T13:23:27.053334+0000","last_undegraded":"2024-04-02T13:23:27.053334+0000","last_fullsized":"2024-04-02T13:23:27.053334+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:07:34.825616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052431+0000","last_change":"2024-04-02T13:23:25.462648+0000","last_active":"2024-04-02T13:23:27.052431+0000","last_peered":"2024-04-02T13:23:27.052431+0000","last_clean":"2024-04-02T13:23:27.052431+0000","last_became_active":"2024-04-02T13:23:25.462359+0000","last_became_peered":"2024-04-02T13:23:25.462359+0000","last_unstale":"2024-04-02T13:23:27.052431+0000","last_undegraded":"2024-04-02T13:23:27.052431+0000","last_fullsized":"2024-04-02T13:23:27.052431+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:19.829312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052226+0000","last_change":"2024-04-02T13:23:24.457209+0000","last_active":"2024-04-02T13:23:27.052226+0000","last_peered":"2024-04-02T13:23:27.052226+0000","last_clean":"2024-04-02T13:23:27.052226+0000","last_became_active":"2024-04-02T13:23:24.456904+0000","last_became_peered":"2024-04-02T13:23:24.456904+0000","last_unstale":"2024-04-02T13:23:27.052226+0000","last_undegraded":"2024-04-02T13:23:27.052226+0000","last_fullsized":"2024-04-02T13:23:27.052226+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:11:20.264466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462811+0000","last_change":"2024-04-02T13:23:25.462538+0000","last_active":"2024-04-02T13:23:26.462811+0000","last_peered":"2024-04-02T13:23:26.462811+0000","last_clean":"2024-04-02T13:23:26.462811+0000","last_became_active":"2024-04-02T13:23:25.462334+0000","last_became_peered":"2024-04-02T13:23:25.462334+0000","last_unstale":"2024-04-02T13:23:26.462811+0000","last_undegraded":"2024-04-02T13:23:26.462811+0000","last_fullsized":"2024-04-02T13:23:26.462811+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:11.330576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463073+0000","last_change":"2024-04-02T13:23:24.453376+0000","last_active":"2024-04-02T13:23:26.463073+0000","last_peered":"2024-04-02T13:23:26.463073+0000","last_clean":"2024-04-02T13:23:26.463073+0000","last_became_active":"2024-04-02T13:23:24.452980+0000","last_became_peered":"2024-04-02T13:23:24.452980+0000","last_unstale":"2024-04-02T13:23:26.463073+0000","last_undegraded":"2024-04-02T13:23:26.463073+0000","last_fullsized":"2024-04-02T13:23:26.463073+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:08.196023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052935+0000","last_change":"2024-04-02T13:23:25.468632+0000","last_active":"2024-04-02T13:23:27.052935+0000","last_peered":"2024-04-02T13:23:27.052935+0000","last_clean":"2024-04-02T13:23:27.052935+0000","last_became_active":"2024-04-02T13:23:25.468312+0000","last_became_peered":"2024-04-02T13:23:25.468312+0000","last_unstale":"2024-04-02T13:23:27.052935+0000","last_undegraded":"2024-04-02T13:23:27.052935+0000","last_fullsized":"2024-04-02T13:23:27.052935+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:00.139996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052135+0000","last_change":"2024-04-02T13:23:24.458596+0000","last_active":"2024-04-02T13:23:27.052135+0000","last_peered":"2024-04-02T13:23:27.052135+0000","last_clean":"2024-04-02T13:23:27.052135+0000","last_became_active":"2024-04-02T13:23:24.458305+0000","last_became_peered":"2024-04-02T13:23:24.458305+0000","last_unstale":"2024-04-02T13:23:27.052135+0000","last_undegraded":"2024-04-02T13:23:27.052135+0000","last_fullsized":"2024-04-02T13:23:27.052135+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:37:49.847371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052658+0000","last_change":"2024-04-02T13:23:25.467538+0000","last_active":"2024-04-02T13:23:27.052658+0000","last_peered":"2024-04-02T13:23:27.052658+0000","last_clean":"2024-04-02T13:23:27.052658+0000","last_became_active":"2024-04-02T13:23:25.467134+0000","last_became_peered":"2024-04-02T13:23:25.467134+0000","last_unstale":"2024-04-02T13:23:27.052658+0000","last_undegraded":"2024-04-02T13:23:27.052658+0000","last_fullsized":"2024-04-02T13:23:27.052658+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:12.317864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463634+0000","last_change":"2024-04-02T13:23:24.455671+0000","last_active":"2024-04-02T13:23:26.463634+0000","last_peered":"2024-04-02T13:23:26.463634+0000","last_clean":"2024-04-02T13:23:26.463634+0000","last_became_active":"2024-04-02T13:23:24.455338+0000","last_became_peered":"2024-04-02T13:23:24.455338+0000","last_unstale":"2024-04-02T13:23:26.463634+0000","last_undegraded":"2024-04-02T13:23:26.463634+0000","last_fullsized":"2024-04-02T13:23:26.463634+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:07:16.896554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464837+0000","last_change":"2024-04-02T13:23:25.457248+0000","last_active":"2024-04-02T13:23:26.464837+0000","last_peered":"2024-04-02T13:23:26.464837+0000","last_clean":"2024-04-02T13:23:26.464837+0000","last_became_active":"2024-04-02T13:23:25.456953+0000","last_became_peered":"2024-04-02T13:23:25.456953+0000","last_unstale":"2024-04-02T13:23:26.464837+0000","last_undegraded":"2024-04-02T13:23:26.464837+0000","last_fullsized":"2024-04-02T13:23:26.464837+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:36:44.493855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"65'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462814+0000","last_change":"2024-04-02T13:23:24.450342+0000","last_active":"2024-04-02T13:23:26.462814+0000","last_peered":"2024-04-02T13:23:26.462814+0000","last_clean":"2024-04-02T13:23:26.462814+0000","last_became_active":"2024-04-02T13:23:24.449824+0000","last_became_peered":"2024-04-02T13:23:24.449824+0000","last_unstale":"2024-04-02T13:23:26.462814+0000","last_undegraded":"2024-04-02T13:23:26.462814+0000","last_fullsized":"2024-04-02T13:23:26.462814+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:22.665321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462640+0000","last_change":"2024-04-02T13:23:25.457764+0000","last_active":"2024-04-02T13:23:26.462640+0000","last_peered":"2024-04-02T13:23:26.462640+0000","last_clean":"2024-04-02T13:23:26.462640+0000","last_became_active":"2024-04-02T13:23:25.457484+0000","last_became_peered":"2024-04-02T13:23:25.457484+0000","last_unstale":"2024-04-02T13:23:26.462640+0000","last_undegraded":"2024-04-02T13:23:26.462640+0000","last_fullsized":"2024-04-02T13:23:26.462640+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:01:09.000108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053550+0000","last_change":"2024-04-02T13:23:24.456306+0000","last_active":"2024-04-02T13:23:27.053550+0000","last_peered":"2024-04-02T13:23:27.053550+0000","last_clean":"2024-04-02T13:23:27.053550+0000","last_became_active":"2024-04-02T13:23:24.456047+0000","last_became_peered":"2024-04-02T13:23:24.456047+0000","last_unstale":"2024-04-02T13:23:27.053550+0000","last_undegraded":"2024-04-02T13:23:27.053550+0000","last_fullsized":"2024-04-02T13:23:27.053550+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:23.339647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052939+0000","last_change":"2024-04-02T13:23:25.461396+0000","last_active":"2024-04-02T13:23:27.052939+0000","last_peered":"2024-04-02T13:23:27.052939+0000","last_clean":"2024-04-02T13:23:27.052939+0000","last_became_active":"2024-04-02T13:23:25.461030+0000","last_became_peered":"2024-04-02T13:23:25.461030+0000","last_unstale":"2024-04-02T13:23:27.052939+0000","last_undegraded":"2024-04-02T13:23:27.052939+0000","last_fullsized":"2024-04-02T13:23:27.052939+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:09.200095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053121+0000","last_change":"2024-04-02T13:23:24.454332+0000","last_active":"2024-04-02T13:23:27.053121+0000","last_peered":"2024-04-02T13:23:27.053121+0000","last_clean":"2024-04-02T13:23:27.053121+0000","last_became_active":"2024-04-02T13:23:24.454066+0000","last_became_peered":"2024-04-02T13:23:24.454066+0000","last_unstale":"2024-04-02T13:23:27.053121+0000","last_undegraded":"2024-04-02T13:23:27.053121+0000","last_fullsized":"2024-04-02T13:23:27.053121+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:56:04.698646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052523+0000","last_change":"2024-04-02T13:23:25.466702+0000","last_active":"2024-04-02T13:23:27.052523+0000","last_peered":"2024-04-02T13:23:27.052523+0000","last_clean":"2024-04-02T13:23:27.052523+0000","last_became_active":"2024-04-02T13:23:25.465630+0000","last_became_peered":"2024-04-02T13:23:25.465630+0000","last_unstale":"2024-04-02T13:23:27.052523+0000","last_undegraded":"2024-04-02T13:23:27.052523+0000","last_fullsized":"2024-04-02T13:23:27.052523+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:21:55.421252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463131+0000","last_change":"2024-04-02T13:23:24.448017+0000","last_active":"2024-04-02T13:23:26.463131+0000","last_peered":"2024-04-02T13:23:26.463131+0000","last_clean":"2024-04-02T13:23:26.463131+0000","last_became_active":"2024-04-02T13:23:24.447718+0000","last_became_peered":"2024-04-02T13:23:24.447718+0000","last_unstale":"2024-04-02T13:23:26.463131+0000","last_undegraded":"2024-04-02T13:23:26.463131+0000","last_fullsized":"2024-04-02T13:23:26.463131+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:29:13.627065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.463798+0000","last_change":"2024-04-02T13:23:25.462615+0000","last_active":"2024-04-02T13:23:26.463798+0000","last_peered":"2024-04-02T13:23:26.463798+0000","last_clean":"2024-04-02T13:23:26.463798+0000","last_became_active":"2024-04-02T13:23:25.462337+0000","last_became_peered":"2024-04-02T13:23:25.462337+0000","last_unstale":"2024-04-02T13:23:26.463798+0000","last_undegraded":"2024-04-02T13:23:26.463798+0000","last_fullsized":"2024-04-02T13:23:26.463798+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:24:01.044977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.053087+0000","last_change":"2024-04-02T13:23:24.451981+0000","last_active":"2024-04-02T13:23:27.053087+0000","last_peered":"2024-04-02T13:23:27.053087+0000","last_clean":"2024-04-02T13:23:27.053087+0000","last_became_active":"2024-04-02T13:23:24.451689+0000","last_became_peered":"2024-04-02T13:23:24.451689+0000","last_unstale":"2024-04-02T13:23:27.053087+0000","last_undegraded":"2024-04-02T13:23:27.053087+0000","last_fullsized":"2024-04-02T13:23:27.053087+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:18.825240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052603+0000","last_change":"2024-04-02T13:23:25.476796+0000","last_active":"2024-04-02T13:23:27.052603+0000","last_peered":"2024-04-02T13:23:27.052603+0000","last_clean":"2024-04-02T13:23:27.052603+0000","last_became_active":"2024-04-02T13:23:25.476510+0000","last_became_peered":"2024-04-02T13:23:25.476510+0000","last_unstale":"2024-04-02T13:23:27.052603+0000","last_undegraded":"2024-04-02T13:23:27.052603+0000","last_fullsized":"2024-04-02T13:23:27.052603+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:29.261268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052614+0000","last_change":"2024-04-02T13:23:24.457619+0000","last_active":"2024-04-02T13:23:27.052614+0000","last_peered":"2024-04-02T13:23:27.052614+0000","last_clean":"2024-04-02T13:23:27.052614+0000","last_became_active":"2024-04-02T13:23:24.457344+0000","last_became_peered":"2024-04-02T13:23:24.457344+0000","last_unstale":"2024-04-02T13:23:27.052614+0000","last_undegraded":"2024-04-02T13:23:27.052614+0000","last_fullsized":"2024-04-02T13:23:27.052614+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:11.313791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.464832+0000","last_change":"2024-04-02T13:23:25.463480+0000","last_active":"2024-04-02T13:23:26.464832+0000","last_peered":"2024-04-02T13:23:26.464832+0000","last_clean":"2024-04-02T13:23:26.464832+0000","last_became_active":"2024-04-02T13:23:25.463195+0000","last_became_peered":"2024-04-02T13:23:25.463195+0000","last_unstale":"2024-04-02T13:23:26.464832+0000","last_undegraded":"2024-04-02T13:23:26.464832+0000","last_fullsized":"2024-04-02T13:23:26.464832+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:13:54.388782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052359+0000","last_change":"2024-04-02T13:23:24.454546+0000","last_active":"2024-04-02T13:23:27.052359+0000","last_peered":"2024-04-02T13:23:27.052359+0000","last_clean":"2024-04-02T13:23:27.052359+0000","last_became_active":"2024-04-02T13:23:24.454287+0000","last_became_peered":"2024-04-02T13:23:24.454287+0000","last_unstale":"2024-04-02T13:23:27.052359+0000","last_undegraded":"2024-04-02T13:23:27.052359+0000","last_fullsized":"2024-04-02T13:23:27.052359+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:24:00.040904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052748+0000","last_change":"2024-04-02T13:23:25.466900+0000","last_active":"2024-04-02T13:23:27.052748+0000","last_peered":"2024-04-02T13:23:27.052748+0000","last_clean":"2024-04-02T13:23:27.052748+0000","last_became_active":"2024-04-02T13:23:25.466583+0000","last_became_peered":"2024-04-02T13:23:25.466583+0000","last_unstale":"2024-04-02T13:23:27.052748+0000","last_undegraded":"2024-04-02T13:23:27.052748+0000","last_fullsized":"2024-04-02T13:23:27.052748+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:24.432663+0000","last_clean_scrub_stamp":"2024-04-02T13:23:24.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:23.669394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:27.052662+0000","last_change":"2024-04-02T13:23:24.451554+0000","last_active":"2024-04-02T13:23:27.052662+0000","last_peered":"2024-04-02T13:23:27.052662+0000","last_clean":"2024-04-02T13:23:27.052662+0000","last_became_active":"2024-04-02T13:23:24.451290+0000","last_became_peered":"2024-04-02T13:23:24.451290+0000","last_unstale":"2024-04-02T13:23:27.052662+0000","last_undegraded":"2024-04-02T13:23:27.052662+0000","last_fullsized":"2024-04-02T13:23:27.052662+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:23:23.428590+0000","last_clean_scrub_stamp":"2024-04-02T13:23:23.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:41:30.043735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":255,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T13:23:26.462040+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:23:26.462040+0000","last_peered":"2024-04-02T13:23:26.462040+0000","last_clean":"2024-04-02T13:23:26.462040+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:23:26.462040+0000","last_undegraded":"2024-04-02T13:23:26.462040+0000","last_fullsized":"2024-04-02T13:23:26.462040+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":1396736,"data_stored":946111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":790528,"data_stored":355001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":790528,"data_stored":354989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":356579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":354967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":354967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":815104,"data_stored":357343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1380352,"data_stored":945335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:23:29.849 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T13:23:29.849 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T13:23:29.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.992+0000 7fb5e8732640 1 Processor -- start 2024-04-02T13:23:29.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.992+0000 7fb5e8732640 1 -- start start 2024-04-02T13:23:29.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.993+0000 7fb5e8732640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb5e015e4a0 0x7fb5e00aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.993+0000 7fb5e8732640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb5e00d0c50 0x7fb5e00d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.993+0000 7fb5e8732640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e006ce40 0x7fb5e006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.993+0000 7fb5e8732640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fb5e00cc600 con 0x7fb5e015e4a0 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.993+0000 7fb5e8732640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb5e00cc8e0 con 0x7fb5e00d0c50 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.993+0000 7fb5e8732640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb5e00ccbc0 con 0x7fb5e006ce40 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.993+0000 7fb5e64ce640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb5e015e4a0 0x7fb5e00aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.993+0000 7fb5e64ce640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb5e015e4a0 0x7fb5e00aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56418/0 (socket says 172.21.15.44:56418) 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.993+0000 7fb5e64ce640 1 -- 172.21.15.44:0/2490018907 learned_addr learned my addr 172.21.15.44:0/2490018907 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.994+0000 7fb5e6ccf640 1 --2- 172.21.15.44:0/2490018907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e006ce40 0x7fb5e006d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.994+0000 7fb5e5ccd640 1 --2- 172.21.15.44:0/2490018907 >> v2:172.21.15.69:3300/0 conn(0x7fb5e00d0c50 0x7fb5e00d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.994+0000 7fb5e64ce640 1 -- 172.21.15.44:0/2490018907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e006ce40 msgr2=0x7fb5e006d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.994+0000 7fb5e64ce640 1 --2- 172.21.15.44:0/2490018907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e006ce40 0x7fb5e006d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.994+0000 7fb5e64ce640 1 -- 172.21.15.44:0/2490018907 >> v2:172.21.15.69:3300/0 conn(0x7fb5e00d0c50 msgr2=0x7fb5e00d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.994+0000 7fb5e64ce640 1 --2- 172.21.15.44:0/2490018907 >> v2:172.21.15.69:3300/0 conn(0x7fb5e00d0c50 0x7fb5e00d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.994+0000 7fb5e64ce640 1 -- 172.21.15.44:0/2490018907 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e00ccea0 con 0x7fb5e015e4a0 2024-04-02T13:23:29.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.994+0000 7fb5e64ce640 1 --2- 172.21.15.44:0/2490018907 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e015e4a0 0x7fb5e00aebf0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb5e00b8060 tx=0x7fb5dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3077ae21c0b470d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:29.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.995+0000 7fb5e54cc640 1 -- 172.21.15.44:0/2490018907 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc035020 con 0x7fb5e015e4a0 2024-04-02T13:23:29.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.995+0000 7fb5e54cc640 1 -- 172.21.15.44:0/2490018907 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5dc031870 con 0x7fb5e015e4a0 2024-04-02T13:23:29.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.995+0000 7fb5e54cc640 1 -- 172.21.15.44:0/2490018907 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc031c40 con 0x7fb5e015e4a0 2024-04-02T13:23:29.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.995+0000 7fb5e8732640 1 -- 172.21.15.44:0/2490018907 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e015e4a0 msgr2=0x7fb5e00aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:29.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.995+0000 7fb5e8732640 1 --2- 172.21.15.44:0/2490018907 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e015e4a0 0x7fb5e00aebf0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb5e00b8060 tx=0x7fb5dc02b910 comp rx=0 tx=0).stop 2024-04-02T13:23:29.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.995+0000 7fb5e8732640 1 -- 172.21.15.44:0/2490018907 shutdown_connections 2024-04-02T13:23:29.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.995+0000 7fb5e8732640 1 --2- 172.21.15.44:0/2490018907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e006ce40 0x7fb5e006d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.995+0000 7fb5e8732640 1 --2- 172.21.15.44:0/2490018907 >> v2:172.21.15.69:3300/0 conn(0x7fb5e00d0c50 0x7fb5e00d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.995+0000 7fb5e8732640 1 --2- 172.21.15.44:0/2490018907 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e015e4a0 0x7fb5e00aebf0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:29.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.995+0000 7fb5e8732640 1 -- 172.21.15.44:0/2490018907 >> 172.21.15.44:0/2490018907 conn(0x7fb5e009aa20 msgr2=0x7fb5e009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:29.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.995+0000 7fb5e8732640 1 -- 172.21.15.44:0/2490018907 shutdown_connections 2024-04-02T13:23:29.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.995+0000 7fb5e8732640 1 -- 172.21.15.44:0/2490018907 wait complete. 2024-04-02T13:23:30.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.996+0000 7fb5e8732640 1 Processor -- start 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.996+0000 7fb5e8732640 1 -- start start 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.997+0000 7fb5e8732640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e006ce40 0x7fb5e014cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.997+0000 7fb5e8732640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e00d0c50 0x7fb5e014f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.997+0000 7fb5e8732640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e015e4a0 0x7fb5e014fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.997+0000 7fb5e8732640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e015e880 con 0x7fb5e00d0c50 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.997+0000 7fb5e8732640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e00cd460 con 0x7fb5e015e4a0 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.997+0000 7fb5e8732640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb5e00af130 con 0x7fb5e006ce40 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.997+0000 7fb5e5ccd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e00d0c50 0x7fb5e014f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.997+0000 7fb5e5ccd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e00d0c50 0x7fb5e014f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56432/0 (socket says 172.21.15.44:56432) 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.997+0000 7fb5e5ccd640 1 -- 172.21.15.44:0/491008201 learned_addr learned my addr 172.21.15.44:0/491008201 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.997+0000 7fb5e6ccf640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e015e4a0 0x7fb5e014fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.997+0000 7fb5e64ce640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e006ce40 0x7fb5e014cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.998+0000 7fb5e5ccd640 1 -- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e006ce40 msgr2=0x7fb5e014cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.998+0000 7fb5e5ccd640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e006ce40 0x7fb5e014cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.998+0000 7fb5e5ccd640 1 -- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e015e4a0 msgr2=0x7fb5e014fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:30.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.998+0000 7fb5e5ccd640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e015e4a0 0x7fb5e014fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.998+0000 7fb5e5ccd640 1 -- 172.21.15.44:0/491008201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e014d050 con 0x7fb5e00d0c50 2024-04-02T13:23:30.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.998+0000 7fb5e5ccd640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e00d0c50 0x7fb5e014f570 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fb5d000d800 tx=0x7fb5d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:30.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.998+0000 7fb5d6ffd640 1 -- 172.21.15.44:0/491008201 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0004020 con 0x7fb5e00d0c50 2024-04-02T13:23:30.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.998+0000 7fb5d6ffd640 1 -- 172.21.15.44:0/491008201 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5d00041c0 con 0x7fb5e00d0c50 2024-04-02T13:23:30.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.998+0000 7fb5d6ffd640 1 -- 172.21.15.44:0/491008201 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0004390 con 0x7fb5e00d0c50 2024-04-02T13:23:30.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.998+0000 7fb5e8732640 1 -- 172.21.15.44:0/491008201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5e01615c0 con 0x7fb5e00d0c50 2024-04-02T13:23:30.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.998+0000 7fb5e8732640 1 -- 172.21.15.44:0/491008201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5e00c6eb0 con 0x7fb5e00d0c50 2024-04-02T13:23:30.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.999+0000 7fb5e8732640 1 -- 172.21.15.44:0/491008201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5e00684f0 con 0x7fb5e00d0c50 2024-04-02T13:23:30.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:29.999+0000 7fb5d6ffd640 1 -- 172.21.15.44:0/491008201 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb5d001b020 con 0x7fb5e00d0c50 2024-04-02T13:23:30.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.000+0000 7fb5d6ffd640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb5c406c8f0 0x7fb5c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:30.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.000+0000 7fb5d6ffd640 1 -- 172.21.15.44:0/491008201 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fb5d008b890 con 0x7fb5e00d0c50 2024-04-02T13:23:30.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.000+0000 7fb5e64ce640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb5c406c8f0 0x7fb5c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:30.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.000+0000 7fb5e64ce640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb5c406c8f0 0x7fb5c406edb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc036000 tx=0x7fb5dc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:30.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.004+0000 7fb5d6ffd640 1 -- 172.21.15.44:0/491008201 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb5d0059570 con 0x7fb5e00d0c50 2024-04-02T13:23:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.260+0000 7fb5e8732640 1 -- 172.21.15.44:0/491008201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fb5e0070e00 con 0x7fb5e00d0c50 2024-04-02T13:23:30.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.262+0000 7fb5d6ffd640 1 -- 172.21.15.44:0/491008201 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7fb5d005d870 con 0x7fb5e00d0c50 2024-04-02T13:23:30.264 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-04-02T13:23:30.264 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A== 2024-04-02T13:23:30.264 INFO:teuthology.orchestra.run.smithi044.stdout: caps mds = "allow" 2024-04-02T13:23:30.264 INFO:teuthology.orchestra.run.smithi044.stdout: caps mon = "allow r" 2024-04-02T13:23:30.264 INFO:teuthology.orchestra.run.smithi044.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-02T13:23:30.264 INFO:teuthology.orchestra.run.smithi044.stderr:updated caps for client.0 2024-04-02T13:23:30.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.265+0000 7fb5e8732640 1 -- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb5c406c8f0 msgr2=0x7fb5c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:30.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.265+0000 7fb5e8732640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb5c406c8f0 0x7fb5c406edb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc036000 tx=0x7fb5dc03c040 comp rx=0 tx=0).stop 2024-04-02T13:23:30.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.265+0000 7fb5e8732640 1 -- 172.21.15.44:0/491008201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e00d0c50 msgr2=0x7fb5e014f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:30.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.265+0000 7fb5e8732640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e00d0c50 0x7fb5e014f570 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fb5d000d800 tx=0x7fb5d000dcc0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.265+0000 7fb5e8732640 1 -- 172.21.15.44:0/491008201 shutdown_connections 2024-04-02T13:23:30.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.265+0000 7fb5e8732640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb5c406c8f0 0x7fb5c406edb0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.265+0000 7fb5e8732640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e015e4a0 0x7fb5e014fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.265+0000 7fb5e8732640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e00d0c50 0x7fb5e014f570 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.265+0000 7fb5e8732640 1 --2- 172.21.15.44:0/491008201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e006ce40 0x7fb5e014cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.265+0000 7fb5e8732640 1 -- 172.21.15.44:0/491008201 >> 172.21.15.44:0/491008201 conn(0x7fb5e009aa20 msgr2=0x7fb5e00b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:30.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.265+0000 7fb5e8732640 1 -- 172.21.15.44:0/491008201 shutdown_connections 2024-04-02T13:23:30.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.265+0000 7fb5e8732640 1 -- 172.21.15.44:0/491008201 wait complete. 2024-04-02T13:23:30.277 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:23:30.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.377+0000 7f0a6b900640 1 Processor -- start 2024-04-02T13:23:30.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.377+0000 7f0a6b900640 1 -- start start 2024-04-02T13:23:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.378+0000 7f0a6b900640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a6415e0f0 0x7f0a640d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.378+0000 7f0a6b900640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a6406dfe0 con 0x7f0a640d0630 2024-04-02T13:23:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.378+0000 7f0a6b900640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a6406e2c0 con 0x7f0a6406bbe0 2024-04-02T13:23:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.378+0000 7f0a6b900640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0a6406e5a0 con 0x7f0a6415e0f0 2024-04-02T13:23:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.378+0000 7f0a68e9b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0a640d0630 0x7f0a6406b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:58056/0 (socket says 172.21.15.44:58056) 2024-04-02T13:23:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.378+0000 7f0a68e9b640 1 -- 172.21.15.44:0/3488815314 learned_addr learned my addr 172.21.15.44:0/3488815314 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.378+0000 7f0a6969c640 1 --2- 172.21.15.44:0/3488815314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a6415e0f0 0x7f0a640d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.378+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1373062278 0 0) 0x7f0a6406dfe0 con 0x7f0a640d0630 2024-04-02T13:23:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.378+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a50003520 con 0x7f0a640d0630 2024-04-02T13:23:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3485724154 0 0) 0x7f0a50003520 con 0x7f0a640d0630 2024-04-02T13:23:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a6415e0f0 msgr2=0x7f0a640d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a5bfff640 1 --2- 172.21.15.44:0/3488815314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a6415e0f0 0x7f0a640d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 >> v1:172.21.15.69:6789/0 conn(0x7f0a6406bbe0 legacy=0x7f0a640add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6406e880 con 0x7f0a640d0630 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f0a54001670 con 0x7f0a640d0630 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 >> v1:172.21.15.44:6789/0 conn(0x7f0a640d0630 legacy=0x7f0a6406b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a5bfff640 1 --2- 172.21.15.44:0/3488815314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a500039e0 0x7f0a50003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a5bfff640 1 --2- 172.21.15.44:0/3488815314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a50004800 0x7f0a50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a5bfff640 1 --2- 172.21.15.44:0/3488815314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a6415e0f0 0x7f0a500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6406e5a0 con 0x7f0a50004800 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5000bb00 con 0x7f0a500039e0 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0a5000bd00 con 0x7f0a6415e0f0 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a68e9b640 1 --2- 172.21.15.44:0/3488815314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a50004800 0x7f0a50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a6969c640 1 --2- 172.21.15.44:0/3488815314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a500039e0 0x7f0a50003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a69e9d640 1 --2- 172.21.15.44:0/3488815314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a6415e0f0 0x7f0a500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a68e9b640 1 -- 172.21.15.44:0/3488815314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a6415e0f0 msgr2=0x7f0a500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a68e9b640 1 --2- 172.21.15.44:0/3488815314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a6415e0f0 0x7f0a500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a68e9b640 1 -- 172.21.15.44:0/3488815314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a500039e0 msgr2=0x7f0a50003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a68e9b640 1 --2- 172.21.15.44:0/3488815314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a500039e0 0x7f0a50003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a68e9b640 1 -- 172.21.15.44:0/3488815314 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a50003520 con 0x7f0a50004800 2024-04-02T13:23:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.379+0000 7f0a69e9d640 1 --2- 172.21.15.44:0/3488815314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a6415e0f0 0x7f0a500051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.380+0000 7f0a68e9b640 1 --2- 172.21.15.44:0/3488815314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a50004800 0x7f0a50004c80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400c210 tx=0x7f0a50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d530466053fb2490 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.380+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a54007cf0 con 0x7f0a50004800 2024-04-02T13:23:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.380+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a54030070 con 0x7f0a50004800 2024-04-02T13:23:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.380+0000 7f0a5bfff640 1 -- 172.21.15.44:0/3488815314 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a54005d30 con 0x7f0a50004800 2024-04-02T13:23:30.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.381+0000 7f0a6b900640 1 -- 172.21.15.44:0/3488815314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a50004800 msgr2=0x7f0a50004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.381+0000 7f0a6b900640 1 --2- 172.21.15.44:0/3488815314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a50004800 0x7f0a50004c80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400c210 tx=0x7f0a50005830 comp rx=0 tx=0).stop 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.381+0000 7f0a69e9d640 1 -- 172.21.15.44:0/3488815314 reap_dead start 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.381+0000 7f0a6b900640 1 -- 172.21.15.44:0/3488815314 shutdown_connections 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.381+0000 7f0a6b900640 1 -- 172.21.15.44:0/3488815314 >> 172.21.15.44:0/3488815314 conn(0x7f0a640b2980 msgr2=0x7f0a640b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.381+0000 7f0a6b900640 1 -- 172.21.15.44:0/3488815314 shutdown_connections 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.381+0000 7f0a6b900640 1 -- 172.21.15.44:0/3488815314 wait complete. 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.382+0000 7f0a6b900640 1 Processor -- start 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.382+0000 7f0a6b900640 1 -- start start 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a6b900640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a50004800 0x7f0a640b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a6b900640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a640b6600 0x7f0a640ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a6b900640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a640ca780 0x7f0a64148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a6b900640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6415e4d0 con 0x7f0a640ca780 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a6b900640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6406eda0 con 0x7f0a50004800 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a6b900640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0a6415e650 con 0x7f0a640b6600 2024-04-02T13:23:30.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a69e9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a640ca780 0x7f0a64148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a69e9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a640ca780 0x7f0a64148eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56442/0 (socket says 172.21.15.44:56442) 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a69e9d640 1 -- 172.21.15.44:0/3909730414 learned_addr learned my addr 172.21.15.44:0/3909730414 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a68e9b640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a640b6600 0x7f0a640ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a6969c640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a50004800 0x7f0a640b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a69e9d640 1 -- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a640b6600 msgr2=0x7f0a640ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a69e9d640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a640b6600 0x7f0a640ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a69e9d640 1 -- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a50004800 msgr2=0x7f0a640b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a69e9d640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a50004800 0x7f0a640b3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a69e9d640 1 -- 172.21.15.44:0/3909730414 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a640b51c0 con 0x7f0a640ca780 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.383+0000 7f0a69e9d640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a640ca780 0x7f0a64148eb0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0a60000f80 tx=0x7f0a6000ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.384+0000 7f0a59ffb640 1 -- 172.21.15.44:0/3909730414 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a60003b80 con 0x7f0a640ca780 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.384+0000 7f0a59ffb640 1 -- 172.21.15.44:0/3909730414 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a60003d40 con 0x7f0a640ca780 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.384+0000 7f0a6b900640 1 -- 172.21.15.44:0/3909730414 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a640b4140 con 0x7f0a640ca780 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.384+0000 7f0a59ffb640 1 -- 172.21.15.44:0/3909730414 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a60006b50 con 0x7f0a640ca780 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.384+0000 7f0a6b900640 1 -- 172.21.15.44:0/3909730414 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a640b4700 con 0x7f0a640ca780 2024-04-02T13:23:30.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.385+0000 7f0a6b900640 1 -- 172.21.15.44:0/3909730414 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a640684f0 con 0x7f0a640ca780 2024-04-02T13:23:30.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.385+0000 7f0a59ffb640 1 -- 172.21.15.44:0/3909730414 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0a6001a020 con 0x7f0a640ca780 2024-04-02T13:23:30.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.386+0000 7f0a59ffb640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a4806c8f0 0x7f0a4806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:30.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.386+0000 7f0a59ffb640 1 -- 172.21.15.44:0/3909730414 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f0a6008b260 con 0x7f0a640ca780 2024-04-02T13:23:30.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.386+0000 7f0a6969c640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a4806c8f0 0x7f0a4806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:30.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.386+0000 7f0a6969c640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a4806c8f0 0x7f0a4806edb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009bc0 tx=0x7f0a5c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:30.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.390+0000 7f0a59ffb640 1 -- 172.21.15.44:0/3909730414 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0a60059500 con 0x7f0a640ca780 2024-04-02T13:23:30.428 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:23:30.427+0000 7fb85d32a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:23:30.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.615+0000 7f0a6b900640 1 -- 172.21.15.44:0/3909730414 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a640727f0 con 0x7f0a640ca780 2024-04-02T13:23:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.616+0000 7f0a59ffb640 1 -- 172.21.15.44:0/3909730414 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5179 (secure 0 0 0) 0x7f0a60058e00 con 0x7f0a640ca780 2024-04-02T13:23:30.618 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:30.618 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1535287484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1535287484},{"type":"v1","addr":"172.21.15.69:6836","nonce":1535287484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6429,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2367107002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2367107002},{"type":"v1","addr":"172.21.15.44:6833","nonce":2367107002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/553542317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":553542317},{"type":"v1","addr":"172.21.15.69:6837","nonce":553542317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:23:24.803884+0000","modified":"2024-04-02T13:23:27.831218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6450":{"gid":6450,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/3127613802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3127613802},{"type":"v1","addr":"172.21.15.44:6835","nonce":3127613802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6450,"qdb_cluster":[6450]},"id":4}]} 2024-04-02T13:23:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 35 2024-04-02T13:23:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.618+0000 7f0a6b900640 1 -- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a4806c8f0 msgr2=0x7f0a4806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.618+0000 7f0a6b900640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a4806c8f0 0x7f0a4806edb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009bc0 tx=0x7f0a5c009190 comp rx=0 tx=0).stop 2024-04-02T13:23:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.618+0000 7f0a6b900640 1 -- 172.21.15.44:0/3909730414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a640ca780 msgr2=0x7f0a64148eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.618+0000 7f0a6b900640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a640ca780 0x7f0a64148eb0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0a60000f80 tx=0x7f0a6000ead0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.618+0000 7f0a6b900640 1 -- 172.21.15.44:0/3909730414 shutdown_connections 2024-04-02T13:23:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.618+0000 7f0a6b900640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a4806c8f0 0x7f0a4806edb0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.618+0000 7f0a6b900640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a640ca780 0x7f0a64148eb0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.618+0000 7f0a6b900640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a640b6600 0x7f0a640ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.618+0000 7f0a6b900640 1 --2- 172.21.15.44:0/3909730414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a50004800 0x7f0a640b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.618+0000 7f0a6b900640 1 -- 172.21.15.44:0/3909730414 >> 172.21.15.44:0/3909730414 conn(0x7f0a640b2980 msgr2=0x7f0a640b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.618+0000 7f0a6b900640 1 -- 172.21.15.44:0/3909730414 shutdown_connections 2024-04-02T13:23:30.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:30.618+0000 7f0a6b900640 1 -- 172.21.15.44:0/3909730414 wait complete. 2024-04-02T13:23:31.034 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:23:24.803884+0000', 'modified': '2024-04-02T13:23:27.831218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6450}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6450': {'gid': 6450, 'name': 'a', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.44:6835/3127613802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6834', 'nonce': 3127613802}, {'type': 'v1', 'addr': '172.21.15.44:6835', 'nonce': 3127613802}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6450, 'qdb_cluster': [6450]} 2024-04-02T13:23:31.034 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:23:31.034 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T13:23:31.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.138+0000 7fbb7ff15640 1 Processor -- start 2024-04-02T13:23:31.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.138+0000 7fbb7ff15640 1 -- start start 2024-04-02T13:23:31.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.139+0000 7fbb7ff15640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbb7815c1c0 0x7fbb780affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.139+0000 7fbb7ff15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb780b0c10 0x7fbb78165e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.139+0000 7fbb7ff15640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb780b1640 con 0x7fbb780b3d70 2024-04-02T13:23:31.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.139+0000 7fbb7ff15640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbb780aef20 con 0x7fbb7815c1c0 2024-04-02T13:23:31.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.139+0000 7fbb7ff15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbb780af200 con 0x7fbb780b0c10 2024-04-02T13:23:31.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.139+0000 7fbb7d4b0640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbb780b3d70 0x7fbb780b0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:58058/0 (socket says 172.21.15.44:58058) 2024-04-02T13:23:31.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.139+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/3890184771 learned_addr learned my addr 172.21.15.44:0/3890184771 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:31.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.139+0000 7fbb7e4b2640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb780b0c10 0x7fbb78165e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.139+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/3890184771 >> v2:172.21.15.69:3300/0 conn(0x7fbb7815c1c0 0x7fbb780affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.139+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3220927083 0 0) 0x7fbb6c001110 con 0x7fbb780b3d70 2024-04-02T13:23:31.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb60003520 con 0x7fbb780b3d70 2024-04-02T13:23:31.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2457686647 0 0) 0x7fbb60003520 con 0x7fbb780b3d70 2024-04-02T13:23:31.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb780b0c10 msgr2=0x7fbb78165e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:31.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb780b0c10 0x7fbb78165e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 >> v2:172.21.15.69:3300/0 conn(0x7fbb7815c1c0 msgr2=0x7fbb780affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:31.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 --2- 172.21.15.44:0/3890184771 >> v2:172.21.15.69:3300/0 conn(0x7fbb7815c1c0 0x7fbb780affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb780af4e0 con 0x7fbb780b3d70 2024-04-02T13:23:31.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fbb6c001860 con 0x7fbb780b3d70 2024-04-02T13:23:31.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/3890184771 >> v2:172.21.15.69:3300/0 conn(0x7fbb7815c1c0 0x7fbb780affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:23:31.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 >> v1:172.21.15.44:6789/0 conn(0x7fbb780b3d70 legacy=0x7fbb780b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb600039b0 0x7fbb60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb60004800 0x7fbb60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb600051c0 0x7fbb6000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbb780aef20 con 0x7fbb600051c0 2024-04-02T13:23:31.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbb780af200 con 0x7fbb600039b0 2024-04-02T13:23:31.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbb6000bf00 con 0x7fbb60004800 2024-04-02T13:23:31.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7e4b2640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb780b0c10 0x7fbb78165e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:23:31.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb600039b0 0x7fbb60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7e4b2640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb600051c0 0x7fbb6000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb60004800 0x7fbb60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7e4b2640 1 -- 172.21.15.44:0/3890184771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb60004800 msgr2=0x7fbb60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:31.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7e4b2640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb60004800 0x7fbb60004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7e4b2640 1 -- 172.21.15.44:0/3890184771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb600039b0 msgr2=0x7fbb60003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:31.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7e4b2640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb600039b0 0x7fbb60003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7e4b2640 1 -- 172.21.15.44:0/3890184771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb60003520 con 0x7fbb600051c0 2024-04-02T13:23:31.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.140+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb600039b0 0x7fbb60003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:23:31.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.141+0000 7fbb7e4b2640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb600051c0 0x7fbb6000b9c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009b30 tx=0x7fbb60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0b8189de9f6b16 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:31.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.141+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb700097d0 con 0x7fbb600051c0 2024-04-02T13:23:31.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.141+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb70004030 con 0x7fbb600051c0 2024-04-02T13:23:31.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.141+0000 7fbb7ccaf640 1 -- 172.21.15.44:0/3890184771 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70004400 con 0x7fbb600051c0 2024-04-02T13:23:31.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.142+0000 7fbb7ff15640 1 -- 172.21.15.44:0/3890184771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb600051c0 msgr2=0x7fbb6000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.142+0000 7fbb7ff15640 1 --2- 172.21.15.44:0/3890184771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb600051c0 0x7fbb6000b9c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009b30 tx=0x7fbb60005830 comp rx=0 tx=0).stop 2024-04-02T13:23:31.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.142+0000 7fbb7e4b2640 1 -- 172.21.15.44:0/3890184771 reap_dead start 2024-04-02T13:23:31.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.142+0000 7fbb7ff15640 1 -- 172.21.15.44:0/3890184771 shutdown_connections 2024-04-02T13:23:31.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.142+0000 7fbb7ff15640 1 -- 172.21.15.44:0/3890184771 >> 172.21.15.44:0/3890184771 conn(0x7fbb7809aa20 msgr2=0x7fbb7809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:31.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.142+0000 7fbb7ff15640 1 -- 172.21.15.44:0/3890184771 shutdown_connections 2024-04-02T13:23:31.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.142+0000 7fbb7ff15640 1 -- 172.21.15.44:0/3890184771 wait complete. 2024-04-02T13:23:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.143+0000 7fbb7ff15640 1 Processor -- start 2024-04-02T13:23:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.143+0000 7fbb7ff15640 1 -- start start 2024-04-02T13:23:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7ff15640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb78157fb0 0x7fbb780b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7ff15640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb780c64a0 0x7fbb780b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7ff15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb780ca770 0x7fbb780cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7ff15640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbb780cc990 con 0x7fbb780c64a0 2024-04-02T13:23:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7ff15640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbb780afaa0 con 0x7fbb78157fb0 2024-04-02T13:23:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7ff15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbb780aebf0 con 0x7fbb780ca770 2024-04-02T13:23:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7d4b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb780c64a0 0x7fbb780b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7d4b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb780c64a0 0x7fbb780b7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56468/0 (socket says 172.21.15.44:56468) 2024-04-02T13:23:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/1868870070 learned_addr learned my addr 172.21.15.44:0/1868870070 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb78157fb0 0x7fbb780b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7e4b2640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb780ca770 0x7fbb780cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb780ca770 msgr2=0x7fbb780cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb780ca770 0x7fbb780cabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb78157fb0 msgr2=0x7fbb780b6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb78157fb0 0x7fbb780b6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/1868870070 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb780ca290 con 0x7fbb780c64a0 2024-04-02T13:23:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.144+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb780c64a0 0x7fbb780b7380 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c007340 tx=0x7fbb6c007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.145+0000 7fbb767fc640 1 -- 172.21.15.44:0/1868870070 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c00f9c0 con 0x7fbb780c64a0 2024-04-02T13:23:31.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.145+0000 7fbb767fc640 1 -- 172.21.15.44:0/1868870070 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb6c018070 con 0x7fbb780c64a0 2024-04-02T13:23:31.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.145+0000 7fbb7ff15640 1 -- 172.21.15.44:0/1868870070 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb780c69f0 con 0x7fbb780c64a0 2024-04-02T13:23:31.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.145+0000 7fbb767fc640 1 -- 172.21.15.44:0/1868870070 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c00fda0 con 0x7fbb780c64a0 2024-04-02T13:23:31.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.145+0000 7fbb7ff15640 1 -- 172.21.15.44:0/1868870070 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb780c6fb0 con 0x7fbb780c64a0 2024-04-02T13:23:31.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.146+0000 7fbb767fc640 1 -- 172.21.15.44:0/1868870070 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbb6c01d460 con 0x7fbb780c64a0 2024-04-02T13:23:31.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.146+0000 7fbb767fc640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbb6406c920 0x7fbb6406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.147+0000 7fbb767fc640 1 -- 172.21.15.44:0/1868870070 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fbb6c08bd50 con 0x7fbb780c64a0 2024-04-02T13:23:31.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.147+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbb6406c920 0x7fbb6406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.147+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbb6406c920 0x7fbb6406ede0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbb68007740 tx=0x7fbb6800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:31.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.148+0000 7fbb82bda740 1 Processor -- start 2024-04-02T13:23:31.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.148+0000 7fbb82bda740 1 -- start start 2024-04-02T13:23:31.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.148+0000 7fbb82bda740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 0x558eff3b6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.148+0000 7fbb82bda740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3e6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.148+0000 7fbb82bda740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3b7070 0x558eff3e6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.148+0000 7fbb82bda740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558eff320060 con 0x558eff3e3f70 2024-04-02T13:23:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.148+0000 7fbb82bda740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558eff2c8650 con 0x558eff3b7070 2024-04-02T13:23:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.148+0000 7fbb82bda740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x558eff31c010 con 0x558eff3e3b90 2024-04-02T13:23:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.148+0000 7fbb7d4b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3e6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.148+0000 7fbb7d4b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3e6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56476/0 (socket says 172.21.15.44:56476) 2024-04-02T13:23:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.148+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/3014946112 learned_addr learned my addr 172.21.15.44:0/3014946112 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.149+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/3014946112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 0x558eff3b6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.149+0000 7fbb7e4b2640 1 --2- 172.21.15.44:0/3014946112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3b7070 0x558eff3e6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.149+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/3014946112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 msgr2=0x558eff3b6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.149+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/3014946112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 0x558eff3b6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.149+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/3014946112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3b7070 msgr2=0x558eff3e6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.149+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/3014946112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3b7070 0x558eff3e6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.149+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/3014946112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eff0342d0 con 0x558eff3e3f70 2024-04-02T13:23:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.149+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/3014946112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3e6920 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c08a0f0 tx=0x7fbb6c05a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ae9279816b4216 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.149+0000 7fbb5f7fe640 1 -- 172.21.15.44:0/3014946112 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c05dd30 con 0x558eff3e3f70 2024-04-02T13:23:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.149+0000 7fbb5f7fe640 1 -- 172.21.15.44:0/3014946112 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb6c062560 con 0x558eff3e3f70 2024-04-02T13:23:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.149+0000 7fbb5f7fe640 1 -- 172.21.15.44:0/3014946112 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c055ff0 con 0x558eff3e3f70 2024-04-02T13:23:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.150+0000 7fbb82bda740 1 -- 172.21.15.44:0/3014946112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 msgr2=0x558eff3e6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.150+0000 7fbb82bda740 1 --2- 172.21.15.44:0/3014946112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3e6920 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c08a0f0 tx=0x7fbb6c05a080 comp rx=0 tx=0).stop 2024-04-02T13:23:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.150+0000 7fbb82bda740 1 -- 172.21.15.44:0/3014946112 shutdown_connections 2024-04-02T13:23:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.150+0000 7fbb82bda740 1 --2- 172.21.15.44:0/3014946112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3b7070 0x558eff3e6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.150+0000 7fbb82bda740 1 --2- 172.21.15.44:0/3014946112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3e6920 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.150+0000 7fbb82bda740 1 --2- 172.21.15.44:0/3014946112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 0x558eff3b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.150+0000 7fbb82bda740 1 -- 172.21.15.44:0/3014946112 >> 172.21.15.44:0/3014946112 conn(0x558eff3b8da0 msgr2=0x558eff401920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:31.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.150+0000 7fbb82bda740 1 -- 172.21.15.44:0/3014946112 shutdown_connections 2024-04-02T13:23:31.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.150+0000 7fbb82bda740 1 -- 172.21.15.44:0/3014946112 wait complete. 2024-04-02T13:23:31.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.150+0000 7fbb82bda740 1 -- start start 2024-04-02T13:23:31.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.151+0000 7fbb82bda740 1 Processor -- start 2024-04-02T13:23:31.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.151+0000 7fbb82bda740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e3b90 0x558eff3d54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.151+0000 7fbb82bda740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3f70 0x558eff3e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.151+0000 7fbb82bda740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e8390 0x558eff3e87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.151+0000 7fbb82bda740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558eff3bb440 con 0x558eff3e8390 2024-04-02T13:23:31.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.151+0000 7fbb82bda740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558eff2dba10 con 0x558eff3e3b90 2024-04-02T13:23:31.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.151+0000 7fbb82bda740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x558eff3bdc80 con 0x558eff3e3f70 2024-04-02T13:23:31.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb82bda740 10 client.0 _ll_register_callbacks cb 0x558eff3180b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:23:31.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb7e4b2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e8390 0x558eff3e87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb7e4b2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e8390 0x558eff3e87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56480/0 (socket says 172.21.15.44:56480) 2024-04-02T13:23:31.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb7e4b2640 1 -- 172.21.15.44:0/2869041271 learned_addr learned my addr 172.21.15.44:0/2869041271 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:31.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/2869041271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e3b90 0x558eff3d54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/2869041271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3f70 0x558eff3e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb7e4b2640 1 -- 172.21.15.44:0/2869041271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3f70 msgr2=0x558eff3e5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:31.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb7e4b2640 1 --2- 172.21.15.44:0/2869041271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3f70 0x558eff3e5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb7e4b2640 1 -- 172.21.15.44:0/2869041271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e3b90 msgr2=0x558eff3d54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:31.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb7e4b2640 1 --2- 172.21.15.44:0/2869041271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e3b90 0x558eff3d54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb7e4b2640 1 -- 172.21.15.44:0/2869041271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eff0fe6b0 con 0x558eff3e8390 2024-04-02T13:23:31.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb7e4b2640 1 --2- 172.21.15.44:0/2869041271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e8390 0x558eff3e87f0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009b30 tx=0x7fbb700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53e2f1a9617ea71 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:31.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb5cff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:23:31.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb5cff9640 1 -- 172.21.15.44:0/2869041271 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68009de0 con 0x558eff3e8390 2024-04-02T13:23:31.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb5cff9640 1 -- 172.21.15.44:0/2869041271 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fbb70005b90 con 0x558eff3e8390 2024-04-02T13:23:31.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb82bda740 1 -- 172.21.15.44:0/2869041271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb6c062560 con 0x558eff3e8390 2024-04-02T13:23:31.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb5cff9640 1 -- 172.21.15.44:0/2869041271 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbb7004f050 con 0x558eff3e8390 2024-04-02T13:23:31.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb82bda740 10 client.6579 fetch_fsmap learned FSMap version 35 2024-04-02T13:23:31.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.152+0000 7fbb82bda740 1 -- 172.21.15.44:0/2869041271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x558eff400550 con 0x558eff3e8390 2024-04-02T13:23:31.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.153+0000 7fbb5cff9640 1 -- 172.21.15.44:0/2869041271 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb6c062560 con 0x558eff3e8390 2024-04-02T13:23:31.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.153+0000 7fbb5cff9640 1 -- 172.21.15.44:0/2869041271 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7fbb70052020 con 0x558eff3e8390 2024-04-02T13:23:31.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.153+0000 7fbb82bda740 10 client.6579 fetch_fsmap finished waiting for FSMap version 35 2024-04-02T13:23:31.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.153+0000 7fbb82bda740 10 client.6579 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:23:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.153+0000 7fbb82bda740 20 client.6579 populate_metadata read hostname 'smithi044' 2024-04-02T13:23:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.153+0000 7fbb82bda740 1 --2- 172.21.15.44:0/2869041271 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x558eff415df0 0x558eff4015a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.153+0000 7fbb82bda740 4 client.6579 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:23:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.153+0000 7fbb82bda740 1 -- 172.21.15.44:0/2869041271 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558eff2c7cf0 con 0x558eff415df0 2024-04-02T13:23:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.153+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/2869041271 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x558eff415df0 0x558eff4015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.153+0000 7fbb5cff9640 10 client.6579 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:23:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.153+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/2869041271 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x558eff415df0 0x558eff4015a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a69b896940492b70 server_cookie=48507b241648552e in_seq=0 out_seq=0 2024-04-02T13:23:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb5cff9640 1 -- 172.21.15.44:0/2869041271 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558eff2c7cf0 con 0x558eff415df0 2024-04-02T13:23:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb5cff9640 10 client.6579 handle_command_reply: tid=0 2024-04-02T13:23:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb82bda740 1 client.6579 shutdown 2024-04-02T13:23:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb82bda740 1 -- 172.21.15.44:0/2869041271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e8390 msgr2=0x558eff3e87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb82bda740 1 --2- 172.21.15.44:0/2869041271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e8390 0x558eff3e87f0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009b30 tx=0x7fbb700046c0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb82bda740 1 -- 172.21.15.44:0/2869041271 shutdown_connections 2024-04-02T13:23:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb82bda740 1 --2- 172.21.15.44:0/2869041271 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x558eff415df0 0x558eff4015a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb82bda740 1 --2- 172.21.15.44:0/2869041271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e8390 0x558eff3e87f0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb82bda740 1 --2- 172.21.15.44:0/2869041271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3f70 0x558eff3e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb5cff9640 0 client.6579 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:23:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb82bda740 1 --2- 172.21.15.44:0/2869041271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e3b90 0x558eff3d54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb82bda740 1 -- 172.21.15.44:0/2869041271 >> 172.21.15.44:0/2869041271 conn(0x558eff3b7070 msgr2=0x558eff3baa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb82bda740 1 -- 172.21.15.44:0/2869041271 shutdown_connections 2024-04-02T13:23:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb82bda740 1 -- 172.21.15.44:0/2869041271 wait complete. 2024-04-02T13:23:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.154+0000 7fbb82bda740 20 client.6579 trim_cache size 0 max 16384 2024-04-02T13:23:31.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.173+0000 7fbb82bda740 1 Processor -- start 2024-04-02T13:23:31.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.173+0000 7fbb82bda740 1 -- start start 2024-04-02T13:23:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.173+0000 7fbb82bda740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 0x558eff414140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.173+0000 7fbb82bda740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3d4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.173+0000 7fbb82bda740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e8390 0x558eff3e56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.173+0000 7fbb82bda740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbb781feaf0 con 0x558eff3e3f70 2024-04-02T13:23:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.173+0000 7fbb82bda740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbb780b6030 con 0x558eff3e8390 2024-04-02T13:23:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.173+0000 7fbb82bda740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbb781fe4a0 con 0x558eff3e3b90 2024-04-02T13:23:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.173+0000 7fbb7d4b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3d4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.173+0000 7fbb7d4b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3d4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56482/0 (socket says 172.21.15.44:56482) 2024-04-02T13:23:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.173+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/1354466928 learned_addr learned my addr 172.21.15.44:0/1354466928 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.174+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/1354466928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 0x558eff414140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.174+0000 7fbb7e4b2640 1 --2- 172.21.15.44:0/1354466928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e8390 0x558eff3e56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.174+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/1354466928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 msgr2=0x558eff414140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.174+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/1354466928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 0x558eff414140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.174+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/1354466928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e8390 msgr2=0x558eff3e56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.174+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/1354466928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e8390 0x558eff3e56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.174+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/1354466928 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eff3e6c70 con 0x558eff3e3f70 2024-04-02T13:23:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.174+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/1354466928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3d4610 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c08a1d0 tx=0x7fbb6c060000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ef845a15f7fbb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.174+0000 7fbb5effd640 1 -- 172.21.15.44:0/1354466928 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c05dd30 con 0x558eff3e3f70 2024-04-02T13:23:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.174+0000 7fbb5effd640 1 -- 172.21.15.44:0/1354466928 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb6c062560 con 0x558eff3e3f70 2024-04-02T13:23:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.174+0000 7fbb5effd640 1 -- 172.21.15.44:0/1354466928 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c055ff0 con 0x558eff3e3f70 2024-04-02T13:23:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.174+0000 7fbb82bda740 1 -- 172.21.15.44:0/1354466928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 msgr2=0x558eff3d4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.174+0000 7fbb82bda740 1 --2- 172.21.15.44:0/1354466928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3d4610 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c08a1d0 tx=0x7fbb6c060000 comp rx=0 tx=0).stop 2024-04-02T13:23:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.175+0000 7fbb82bda740 1 -- 172.21.15.44:0/1354466928 shutdown_connections 2024-04-02T13:23:31.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.175+0000 7fbb82bda740 1 --2- 172.21.15.44:0/1354466928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e8390 0x558eff3e56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.175+0000 7fbb82bda740 1 --2- 172.21.15.44:0/1354466928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3d4610 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.175+0000 7fbb82bda740 1 --2- 172.21.15.44:0/1354466928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 0x558eff414140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.175+0000 7fbb82bda740 1 -- 172.21.15.44:0/1354466928 >> 172.21.15.44:0/1354466928 conn(0x558eff3b7070 msgr2=0x558eff3e91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:31.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.175+0000 7fbb82bda740 1 -- 172.21.15.44:0/1354466928 shutdown_connections 2024-04-02T13:23:31.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.175+0000 7fbb82bda740 1 -- 172.21.15.44:0/1354466928 wait complete. 2024-04-02T13:23:31.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.175+0000 7fbb82bda740 1 -- start start 2024-04-02T13:23:31.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.175+0000 7fbb82bda740 1 Processor -- start 2024-04-02T13:23:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb82bda740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 0x558eff47d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb82bda740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3debf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb82bda740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e8390 0x558eff47fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb82bda740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558eff415c70 con 0x558eff3e3f70 2024-04-02T13:23:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb82bda740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558eff401dd0 con 0x558eff3e8390 2024-04-02T13:23:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb82bda740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x558eff416290 con 0x558eff3e3b90 2024-04-02T13:23:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb82bda740 10 client.0 _ll_register_callbacks cb 0x7fbb680095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:23:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb7d4b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3debf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb7d4b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3debf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56498/0 (socket says 172.21.15.44:56498) 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/2051882748 learned_addr learned my addr 172.21.15.44:0/2051882748 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/2051882748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 0x558eff47d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb7e4b2640 1 --2- 172.21.15.44:0/2051882748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e8390 0x558eff47fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/2051882748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 msgr2=0x558eff47d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/2051882748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 0x558eff47d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/2051882748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e8390 msgr2=0x558eff47fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/2051882748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e8390 0x558eff47fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.176+0000 7fbb7d4b0640 1 -- 172.21.15.44:0/2051882748 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eff3e5440 con 0x558eff3e3f70 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb7d4b0640 1 --2- 172.21.15.44:0/2051882748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3debf0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c08a1d0 tx=0x7fbb6c004800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d97542cde09930 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb5cff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb5cff9640 1 -- 172.21.15.44:0/2051882748 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c055c20 con 0x558eff3e3f70 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb5cff9640 1 -- 172.21.15.44:0/2051882748 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fbb6c058dd0 con 0x558eff3e3f70 2024-04-02T13:23:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb82bda740 1 -- 172.21.15.44:0/2051882748 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb6c062560 con 0x558eff3e3f70 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb5cff9640 1 -- 172.21.15.44:0/2051882748 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbb6c07f050 con 0x558eff3e3f70 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb82bda740 10 client.6585 fetch_fsmap learned FSMap version 35 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb82bda740 1 -- 172.21.15.44:0/2051882748 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x558eff480920 con 0x558eff3e3f70 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb5cff9640 1 -- 172.21.15.44:0/2051882748 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb6c062560 con 0x558eff3e3f70 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb5cff9640 1 -- 172.21.15.44:0/2051882748 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7fbb6c082020 con 0x558eff3e3f70 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb82bda740 10 client.6585 fetch_fsmap finished waiting for FSMap version 35 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb82bda740 10 client.6585 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb82bda740 20 client.6585 populate_metadata read hostname 'smithi044' 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb82bda740 1 --2- 172.21.15.44:0/2051882748 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x558eff465850 0x558eff465c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb82bda740 4 client.6585 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb82bda740 1 -- 172.21.15.44:0/2051882748 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "status"}) -- 0x558eff402510 con 0x558eff465850 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.177+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/2051882748 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x558eff465850 0x558eff465c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb7dcb1640 1 --2- 172.21.15.44:0/2051882748 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x558eff465850 0x558eff465c30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5932142a68f5bc0 server_cookie=ddc40c42ecf54855 in_seq=0 out_seq=0 2024-04-02T13:23:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb5cff9640 10 client.6585 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:23:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb5cff9640 1 -- 172.21.15.44:0/2051882748 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x558eff402510 con 0x558eff465850 2024-04-02T13:23:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb5cff9640 10 client.6585 handle_command_reply: tid=0 2024-04-02T13:23:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb82bda740 1 client.6585 shutdown 2024-04-02T13:23:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb82bda740 1 -- 172.21.15.44:0/2051882748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 msgr2=0x558eff3debf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb82bda740 1 --2- 172.21.15.44:0/2051882748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3debf0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c08a1d0 tx=0x7fbb6c004800 comp rx=0 tx=0).stop 2024-04-02T13:23:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb82bda740 1 -- 172.21.15.44:0/2051882748 shutdown_connections 2024-04-02T13:23:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb82bda740 1 --2- 172.21.15.44:0/2051882748 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x558eff465850 0x558eff465c30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb82bda740 1 --2- 172.21.15.44:0/2051882748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eff3e8390 0x558eff47fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb82bda740 1 --2- 172.21.15.44:0/2051882748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eff3e3f70 0x558eff3debf0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb5cff9640 0 client.6585 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:23:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb82bda740 1 --2- 172.21.15.44:0/2051882748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eff3e3b90 0x558eff47d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb82bda740 1 -- 172.21.15.44:0/2051882748 >> 172.21.15.44:0/2051882748 conn(0x558eff3b7070 msgr2=0x558eff425ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb82bda740 1 -- 172.21.15.44:0/2051882748 shutdown_connections 2024-04-02T13:23:31.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb82bda740 1 -- 172.21.15.44:0/2051882748 wait complete. 2024-04-02T13:23:31.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.178+0000 7fbb82bda740 20 client.6585 trim_cache size 0 max 16384 2024-04-02T13:23:31.187 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:23:31.187 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:23:31.187 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:23:31.187 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 6450, 2024-04-02T13:23:31.187 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:23:31.187 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:23:31.187 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:23:31.187 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 5.7227480030000004, 2024-04-02T13:23:31.188 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 35, 2024-04-02T13:23:31.188 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 65, 2024-04-02T13:23:31.188 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 65, 2024-04-02T13:23:31.188 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 10.780362735000001 2024-04-02T13:23:31.188 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:23:31.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.179+0000 7fbb7ff15640 1 -- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbb6406c920 msgr2=0x7fbb6406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.179+0000 7fbb7ff15640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbb6406c920 0x7fbb6406ede0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbb68007740 tx=0x7fbb6800b040 comp rx=0 tx=0).stop 2024-04-02T13:23:31.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.179+0000 7fbb7ff15640 1 -- 172.21.15.44:0/1868870070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb780c64a0 msgr2=0x7fbb780b7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.179+0000 7fbb7ff15640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb780c64a0 0x7fbb780b7380 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c007340 tx=0x7fbb6c007800 comp rx=0 tx=0).stop 2024-04-02T13:23:31.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.179+0000 7fbb7ff15640 1 -- 172.21.15.44:0/1868870070 shutdown_connections 2024-04-02T13:23:31.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.179+0000 7fbb7ff15640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbb6406c920 0x7fbb6406ede0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.179+0000 7fbb7ff15640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb780ca770 0x7fbb780cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.179+0000 7fbb7ff15640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb780c64a0 0x7fbb780b7380 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.179+0000 7fbb7ff15640 1 --2- 172.21.15.44:0/1868870070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb78157fb0 0x7fbb780b6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.179+0000 7fbb7ff15640 1 -- 172.21.15.44:0/1868870070 >> 172.21.15.44:0/1868870070 conn(0x7fbb7809aa20 msgr2=0x7fbb780b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:31.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.179+0000 7fbb7ff15640 1 -- 172.21.15.44:0/1868870070 shutdown_connections 2024-04-02T13:23:31.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.179+0000 7fbb7ff15640 1 -- 172.21.15.44:0/1868870070 wait complete. 2024-04-02T13:23:31.191 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:23:31.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.290+0000 7f1ef689a640 1 Processor -- start 2024-04-02T13:23:31.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.290+0000 7f1ef689a640 1 -- start start 2024-04-02T13:23:31.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.291+0000 7f1ef689a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1ef00d0a30 0x7f1ef006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.291+0000 7f1ef689a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1ef00d2040 0x7f1ef00d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.291+0000 7f1ef689a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ef00d0000 con 0x7f1ef00d0e10 2024-04-02T13:23:31.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.291+0000 7f1ef689a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f1ef006cb00 con 0x7f1ef00d2040 2024-04-02T13:23:31.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.291+0000 7f1ef689a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1ef006c1d0 con 0x7f1ef00d0a30 2024-04-02T13:23:31.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.291+0000 7f1ef4e37640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1ef00d2040 0x7f1ef00d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.291+0000 7f1ef4e37640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1ef00d2040 0x7f1ef00d24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56514/0 (socket says 172.21.15.44:56514) 2024-04-02T13:23:31.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.291+0000 7f1ef4e37640 1 -- 172.21.15.44:0/29447847 learned_addr learned my addr 172.21.15.44:0/29447847 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:31.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.291+0000 7f1eef7fe640 1 --1- 172.21.15.44:0/29447847 >> v1:172.21.15.44:6789/0 conn(0x7f1ef00d0e10 0x7f1ef00cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:58060/0 (socket says 172.21.15.44:58060) 2024-04-02T13:23:31.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.291+0000 7f1eeffff640 1 --2- 172.21.15.44:0/29447847 >> v2:172.21.15.69:3300/0 conn(0x7f1ef00d0a30 0x7f1ef006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.292+0000 7f1ef4e37640 1 -- 172.21.15.44:0/29447847 >> v2:172.21.15.69:3300/0 conn(0x7f1ef00d0a30 msgr2=0x7f1ef006cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:31.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.292+0000 7f1ef4e37640 1 --2- 172.21.15.44:0/29447847 >> v2:172.21.15.69:3300/0 conn(0x7f1ef00d0a30 0x7f1ef006cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.292+0000 7f1ef4e37640 1 -- 172.21.15.44:0/29447847 >> v1:172.21.15.44:6789/0 conn(0x7f1ef00d0e10 legacy=0x7f1ef00cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.292+0000 7f1eeeffd640 1 -- 172.21.15.44:0/29447847 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 361590565 0 0) 0x7f1ef00d0000 con 0x7f1ef00d0e10 2024-04-02T13:23:31.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.292+0000 7f1ef4e37640 1 -- 172.21.15.44:0/29447847 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ef006c4b0 con 0x7f1ef00d2040 2024-04-02T13:23:31.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.292+0000 7f1ef4e37640 1 --2- 172.21.15.44:0/29447847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ef00d2040 0x7f1ef00d24f0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f1ef006de20 tx=0x7f1ee802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9de854878895be4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:31.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.293+0000 7f1eeeffd640 1 -- 172.21.15.44:0/29447847 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee802ca80 con 0x7f1ef00d2040 2024-04-02T13:23:31.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.293+0000 7f1eeeffd640 1 -- 172.21.15.44:0/29447847 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ee802cc20 con 0x7f1ef00d2040 2024-04-02T13:23:31.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.293+0000 7f1eeeffd640 1 -- 172.21.15.44:0/29447847 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee80343f0 con 0x7f1ef00d2040 2024-04-02T13:23:31.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.293+0000 7f1ef689a640 1 -- 172.21.15.44:0/29447847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ef00d2040 msgr2=0x7f1ef00d24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.293+0000 7f1ef689a640 1 --2- 172.21.15.44:0/29447847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ef00d2040 0x7f1ef00d24f0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f1ef006de20 tx=0x7f1ee802bee0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.293+0000 7f1ef689a640 1 -- 172.21.15.44:0/29447847 shutdown_connections 2024-04-02T13:23:31.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.293+0000 7f1ef689a640 1 --2- 172.21.15.44:0/29447847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ef00d2040 0x7f1ef00d24f0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.293+0000 7f1ef689a640 1 --2- 172.21.15.44:0/29447847 >> v2:172.21.15.69:3300/0 conn(0x7f1ef00d0a30 0x7f1ef006cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.293+0000 7f1ef689a640 1 -- 172.21.15.44:0/29447847 >> 172.21.15.44:0/29447847 conn(0x7f1ef009d470 msgr2=0x7f1ef00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:31.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.293+0000 7f1ef689a640 1 -- 172.21.15.44:0/29447847 shutdown_connections 2024-04-02T13:23:31.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.293+0000 7f1ef689a640 1 -- 172.21.15.44:0/29447847 wait complete. 2024-04-02T13:23:31.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.294+0000 7f1ef689a640 1 Processor -- start 2024-04-02T13:23:31.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.294+0000 7f1ef689a640 1 -- start start 2024-04-02T13:23:31.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.295+0000 7f1ef689a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ef00d0a30 0x7f1ef0161210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.295+0000 7f1ef689a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ef00d0e10 0x7f1ef01f8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.295+0000 7f1ef689a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ef00d2040 0x7f1ef00b5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.295+0000 7f1ef689a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef015e6f0 con 0x7f1ef00d0e10 2024-04-02T13:23:31.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.295+0000 7f1ef689a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef00d1270 con 0x7f1ef00d0a30 2024-04-02T13:23:31.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.295+0000 7f1ef689a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1ef015e870 con 0x7f1ef00d2040 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.295+0000 7f1eef7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ef00d0e10 0x7f1ef01f8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.295+0000 7f1eef7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ef00d0e10 0x7f1ef01f8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56522/0 (socket says 172.21.15.44:56522) 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.295+0000 7f1eef7fe640 1 -- 172.21.15.44:0/353861760 learned_addr learned my addr 172.21.15.44:0/353861760 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.295+0000 7f1eeffff640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ef00d0a30 0x7f1ef0161210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.295+0000 7f1ef4e37640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ef00d2040 0x7f1ef00b5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.296+0000 7f1eef7fe640 1 -- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ef00d2040 msgr2=0x7f1ef00b5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.296+0000 7f1eef7fe640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ef00d2040 0x7f1ef00b5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.296+0000 7f1eef7fe640 1 -- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ef00d0a30 msgr2=0x7f1ef0161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.296+0000 7f1eef7fe640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ef00d0a30 0x7f1ef0161210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.296+0000 7f1eef7fe640 1 -- 172.21.15.44:0/353861760 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ef00ccf50 con 0x7f1ef00d0e10 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.296+0000 7f1ef4e37640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ef00d2040 0x7f1ef00b5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.296+0000 7f1eef7fe640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ef00d0e10 0x7f1ef01f8360 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800c440 tx=0x7f1ed800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.296+0000 7f1eecff9640 1 -- 172.21.15.44:0/353861760 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8018070 con 0x7f1ef00d0e10 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.296+0000 7f1eecff9640 1 -- 172.21.15.44:0/353861760 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed80074e0 con 0x7f1ef00d0e10 2024-04-02T13:23:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.296+0000 7f1ef689a640 1 -- 172.21.15.44:0/353861760 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ef00cd230 con 0x7f1ef00d0e10 2024-04-02T13:23:31.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.296+0000 7f1eecff9640 1 -- 172.21.15.44:0/353861760 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8004a20 con 0x7f1ef00d0e10 2024-04-02T13:23:31.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.296+0000 7f1ef689a640 1 -- 172.21.15.44:0/353861760 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ef0161750 con 0x7f1ef00d0e10 2024-04-02T13:23:31.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.297+0000 7f1ef689a640 1 -- 172.21.15.44:0/353861760 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ef00684f0 con 0x7f1ef00d0e10 2024-04-02T13:23:31.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.297+0000 7f1eecff9640 1 -- 172.21.15.44:0/353861760 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1ed801b070 con 0x7f1ef00d0e10 2024-04-02T13:23:31.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.298+0000 7f1eecff9640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1ed006c920 0x7f1ed006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:31.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.298+0000 7f1eecff9640 1 -- 172.21.15.44:0/353861760 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f1ed808af60 con 0x7f1ef00d0e10 2024-04-02T13:23:31.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.298+0000 7f1eeffff640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1ed006c920 0x7f1ed006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:31.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.298+0000 7f1eeffff640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1ed006c920 0x7f1ed006ede0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00d4950 tx=0x7f1ee0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:31.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.302+0000 7f1eecff9640 1 -- 172.21.15.44:0/353861760 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1ed8059200 con 0x7f1ef00d0e10 2024-04-02T13:23:31.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.526+0000 7f1ef689a640 1 -- 172.21.15.44:0/353861760 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1ef00d17c0 con 0x7f1ef00d0e10 2024-04-02T13:23:31.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.526+0000 7f1eecff9640 1 -- 172.21.15.44:0/353861760 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) ==== 53+0+73 (secure 0 0 0) 0x7f1ed8058b00 con 0x7f1ef00d0e10 2024-04-02T13:23:31.528 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:23:31.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.528+0000 7f1ef689a640 1 -- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1ed006c920 msgr2=0x7f1ed006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.528+0000 7f1ef689a640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1ed006c920 0x7f1ed006ede0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00d4950 tx=0x7f1ee0008040 comp rx=0 tx=0).stop 2024-04-02T13:23:31.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.528+0000 7f1ef689a640 1 -- 172.21.15.44:0/353861760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ef00d0e10 msgr2=0x7f1ef01f8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:31.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.528+0000 7f1ef689a640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ef00d0e10 0x7f1ef01f8360 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800c440 tx=0x7f1ed800c900 comp rx=0 tx=0).stop 2024-04-02T13:23:31.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.528+0000 7f1ef689a640 1 -- 172.21.15.44:0/353861760 shutdown_connections 2024-04-02T13:23:31.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.528+0000 7f1ef689a640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1ed006c920 0x7f1ed006ede0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.528+0000 7f1ef689a640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ef00d2040 0x7f1ef00b5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.528+0000 7f1ef689a640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ef00d0e10 0x7f1ef01f8360 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.528+0000 7f1ef689a640 1 --2- 172.21.15.44:0/353861760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ef00d0a30 0x7f1ef0161210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:31.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.528+0000 7f1ef689a640 1 -- 172.21.15.44:0/353861760 >> 172.21.15.44:0/353861760 conn(0x7f1ef009d470 msgr2=0x7f1ef006cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:31.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.528+0000 7f1ef689a640 1 -- 172.21.15.44:0/353861760 shutdown_connections 2024-04-02T13:23:31.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31.528+0000 7f1ef689a640 1 -- 172.21.15.44:0/353861760 wait complete. 2024-04-02T13:23:31.541 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:23:31.541 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:23:31.541 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:23:31.541 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:23:31.541 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:23:31.541 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:23:31.541 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:23:31.541 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:23:31.541 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:23:31.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:31.542 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:23:31.559 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:23:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:23:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:23:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:23:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:23:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:23:31.560 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:23:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:23:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:23:31.560 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:23:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:23:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:23:31.560 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:23:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42567sec preferred_lft 42567sec 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:23:31.561 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:23:31.562 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:23:31.562 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:23:31.562 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:23:31.562 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:23:31.562 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:23:31.562 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:23:31.562 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:23:31.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:31.562 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:23:31.625 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:23:31.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:31.627 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:23:31.682 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:23:31.682 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:23:31.682 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:23:31.682 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:23:31.682 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:23:31.682 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:23:31.683 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:23:31.683 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:23:31.683 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:23:31.683 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:23:31.683 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:23:31.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:23:31.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:23:31.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:31.817 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:23:31.831 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:23:31.832 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:23:31.832 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:23:31.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:31.832 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:23:31.898 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:23:31.898 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:23:31.898 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:23:31.953 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:23:31.953 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:23:31.953 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:23:32.008 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:23:32.075 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:23:32.129 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:23:32.130 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:23:32.130 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:23:32.130 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:23:32.130 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:23:32.130 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:23:32.130 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:23:32.130 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:23:32.130 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:23:32.130 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:23:32.130 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:23:32.130 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:23:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:23:32.132 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:23:32.132 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:23:32.132 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:23:32.132 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:23:32.132 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:23:32.132 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:23:32.132 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:23:32.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:32.132 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:23:32.184 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:23:32.184 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:23:32.227 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:23:32.253 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:23:32.308 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:23:32.308 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:23:32.309 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:23:32.309 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:23:32.309 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:23:32.309 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:23:32.309 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:23:32.309 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:23:32.309 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:23:32.309 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:23:32.309 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:23:32.309 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:23:32.309 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:23:32.309 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:23:32.310 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:23:32.310 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:23:32.310 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:23:32.310 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:23:32.310 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:23:32.310 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:23:32.310 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:23:32.310 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:23:32.310 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:23:32.310 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:23:32.310 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:23:32.310 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:23:32.310 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:23:32.311 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:23:32.311 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:23:32.311 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:23:32.311 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:23:32.311 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:23:32.311 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:23:32.311 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:23:32.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:32.312 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:23:32.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:23:32.310+0000 7fa287635480 -1 init, newargv = 0x556b0bf5efd0 newargc=15 2024-04-02T13:23:32.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[60355]: starting ceph client 2024-04-02T13:23:32.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[60355]: starting fuse 2024-04-02T13:23:32.330 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:23:32.331 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:23:32.331 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:23:32.331 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:23:32.331 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:23:32.331 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:23:32.331 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:23:32.331 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:23:32.331 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:23:32.331 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:23:32.332 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:23:32.333 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:23:32.333 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:23:32.333 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:23:32.333 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:23:32.333 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:23:32.451 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.60355.asok 2024-04-02T13:23:32.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:23:32.462 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.60355.asok 2024-04-02T13:23:32.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:32.462 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.60355.asok status 2024-04-02T13:23:32.600 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:23:32.600 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:23:32.600 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:23:32.600 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:23:32.600 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:23:32.600 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:23:32.600 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:23:32.600 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "60355", 2024-04-02T13:23:32.600 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:23:32.600 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:32.600 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 6603, 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 6603 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3828152852 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:23:32.601 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:23:32.602 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:23:32.602 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3828152852 2024-04-02T13:23:32.602 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:32.602 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.6603 192.168.0.1:0/3828152852", 2024-04-02T13:23:32.602 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/3828152852", 2024-04-02T13:23:32.602 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:23:32.602 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 35, 2024-04-02T13:23:32.602 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 65, 2024-04-02T13:23:32.602 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:23:32.602 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:23:32.602 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:23:32.602 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:23:32.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:32.608 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:23:32.608 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:23:32.623 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:23:32.623 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:23:32.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:32.623 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:23:32.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:32.694 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_test) 2024-04-02T13:23:32.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:32.752 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir_test/file) 2024-04-02T13:23:32.809 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:23:32.809 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:23:32.809 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:23:32.809 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:23:32.809 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_test/file").st_ino) 2024-04-02T13:23:32.809 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:23:32.913 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627777 2024-04-02T13:23:32.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:23:32.922 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:23:33.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.061+0000 7f47720dc640 1 Processor -- start 2024-04-02T13:23:33.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.061+0000 7f47720dc640 1 -- start start 2024-04-02T13:23:33.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f47720dc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f476c0d0c50 0x7f476c0d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f47720dc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f476c06ce40 0x7f476c06d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f47720dc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f476c0cc600 con 0x7f476c15e4a0 2024-04-02T13:23:33.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f47720dc640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f476c0af300 con 0x7f476c06ce40 2024-04-02T13:23:33.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f47720dc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f476c0cc8e0 con 0x7f476c0d0c50 2024-04-02T13:23:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f476bfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f476c06ce40 0x7f476c06d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f476b7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f476c15e4a0 0x7f476c0aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:58072/0 (socket says 172.21.15.44:58072) 2024-04-02T13:23:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f476b7fe640 1 -- 172.21.15.44:0/551400989 learned_addr learned my addr 172.21.15.44:0/551400989 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:23:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f476affd640 1 --2- 172.21.15.44:0/551400989 >> v2:172.21.15.69:3300/0 conn(0x7f476c0d0c50 0x7f476c0d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f476bfff640 1 -- 172.21.15.44:0/551400989 >> v2:172.21.15.69:3300/0 conn(0x7f476c0d0c50 msgr2=0x7f476c0d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f476bfff640 1 --2- 172.21.15.44:0/551400989 >> v2:172.21.15.69:3300/0 conn(0x7f476c0d0c50 0x7f476c0d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f476bfff640 1 -- 172.21.15.44:0/551400989 >> v1:172.21.15.44:6789/0 conn(0x7f476c15e4a0 legacy=0x7f476c0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f476a7fc640 1 -- 172.21.15.44:0/551400989 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1301270610 0 0) 0x7f476c0cc600 con 0x7f476c15e4a0 2024-04-02T13:23:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.062+0000 7f476bfff640 1 -- 172.21.15.44:0/551400989 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476c0ccbc0 con 0x7f476c06ce40 2024-04-02T13:23:33.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.063+0000 7f476bfff640 1 --2- 172.21.15.44:0/551400989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f476c06ce40 0x7f476c06d2f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f476c0b8060 tx=0x7f476402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7a15c2d46ecae2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.063+0000 7f476a7fc640 1 -- 172.21.15.44:0/551400989 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764039070 con 0x7f476c06ce40 2024-04-02T13:23:33.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.063+0000 7f476a7fc640 1 -- 172.21.15.44:0/551400989 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f476402cc80 con 0x7f476c06ce40 2024-04-02T13:23:33.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.063+0000 7f476a7fc640 1 -- 172.21.15.44:0/551400989 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764034660 con 0x7f476c06ce40 2024-04-02T13:23:33.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.064+0000 7f47720dc640 1 -- 172.21.15.44:0/551400989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f476c06ce40 msgr2=0x7f476c06d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.064+0000 7f47720dc640 1 --2- 172.21.15.44:0/551400989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f476c06ce40 0x7f476c06d2f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f476c0b8060 tx=0x7f476402bed0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.064+0000 7f47720dc640 1 -- 172.21.15.44:0/551400989 shutdown_connections 2024-04-02T13:23:33.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.064+0000 7f47720dc640 1 --2- 172.21.15.44:0/551400989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f476c06ce40 0x7f476c06d2f0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.064+0000 7f47720dc640 1 --2- 172.21.15.44:0/551400989 >> v2:172.21.15.69:3300/0 conn(0x7f476c0d0c50 0x7f476c0d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.064+0000 7f47720dc640 1 -- 172.21.15.44:0/551400989 >> 172.21.15.44:0/551400989 conn(0x7f476c09aa20 msgr2=0x7f476c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:33.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.064+0000 7f47720dc640 1 -- 172.21.15.44:0/551400989 shutdown_connections 2024-04-02T13:23:33.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.064+0000 7f47720dc640 1 -- 172.21.15.44:0/551400989 wait complete. 2024-04-02T13:23:33.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.065+0000 7f47720dc640 1 Processor -- start 2024-04-02T13:23:33.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.065+0000 7f47720dc640 1 -- start start 2024-04-02T13:23:33.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f47720dc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f476c06ce40 0x7f476c157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f47720dc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f476c0d0c50 0x7f476c0b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f47720dc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f476c15e4a0 0x7f476c0b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f47720dc640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f476c0cd460 con 0x7f476c15e4a0 2024-04-02T13:23:33.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f47720dc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f476c0cd180 con 0x7f476c0d0c50 2024-04-02T13:23:33.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f47720dc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f476c15e880 con 0x7f476c06ce40 2024-04-02T13:23:33.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f476bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f476c15e4a0 0x7f476c0b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f476bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f476c15e4a0 0x7f476c0b32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56534/0 (socket says 172.21.15.44:56534) 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f476bfff640 1 -- 172.21.15.44:0/2569248699 learned_addr learned my addr 172.21.15.44:0/2569248699 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f476affd640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f476c0d0c50 0x7f476c0b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f476b7fe640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f476c06ce40 0x7f476c157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f476bfff640 1 -- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f476c06ce40 msgr2=0x7f476c157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f476bfff640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f476c06ce40 0x7f476c157430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f476bfff640 1 -- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f476c0d0c50 msgr2=0x7f476c0b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f476bfff640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f476c0d0c50 0x7f476c0b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f476bfff640 1 -- 172.21.15.44:0/2569248699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476c0c6ea0 con 0x7f476c15e4a0 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f476affd640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f476c0d0c50 0x7f476c0b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.066+0000 7f476bfff640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f476c15e4a0 0x7f476c0b32f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4764009e80 tx=0x7f4764004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.067+0000 7f4753fff640 1 -- 172.21.15.44:0/2569248699 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764039050 con 0x7f476c15e4a0 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.067+0000 7f4753fff640 1 -- 172.21.15.44:0/2569248699 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4764045070 con 0x7f476c15e4a0 2024-04-02T13:23:33.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.067+0000 7f47720dc640 1 -- 172.21.15.44:0/2569248699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f476c0c6880 con 0x7f476c15e4a0 2024-04-02T13:23:33.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.067+0000 7f4753fff640 1 -- 172.21.15.44:0/2569248699 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476403e830 con 0x7f476c15e4a0 2024-04-02T13:23:33.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.067+0000 7f47720dc640 1 -- 172.21.15.44:0/2569248699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f476c0c7420 con 0x7f476c15e4a0 2024-04-02T13:23:33.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.068+0000 7f47720dc640 1 -- 172.21.15.44:0/2569248699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f476c0684f0 con 0x7f476c15e4a0 2024-04-02T13:23:33.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.068+0000 7f4753fff640 1 -- 172.21.15.44:0/2569248699 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f476403e9d0 con 0x7f476c15e4a0 2024-04-02T13:23:33.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.068+0000 7f4753fff640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f475806c8f0 0x7f475806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.068+0000 7f4753fff640 1 -- 172.21.15.44:0/2569248699 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f47640aea60 con 0x7f476c15e4a0 2024-04-02T13:23:33.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.069+0000 7f476b7fe640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f475806c8f0 0x7f475806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.069+0000 7f476b7fe640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f475806c8f0 0x7f475806edb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f475c000f60 tx=0x7f475c007c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.073+0000 7f4753fff640 1 -- 172.21.15.44:0/2569248699 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f476407ce20 con 0x7f476c15e4a0 2024-04-02T13:23:33.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.298+0000 7f47720dc640 1 -- 172.21.15.44:0/2569248699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f476c0d07a0 con 0x7f476c15e4a0 2024-04-02T13:23:33.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.299+0000 7f4753fff640 1 -- 172.21.15.44:0/2569248699 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5179 (secure 0 0 0) 0x7f476407c720 con 0x7f476c15e4a0 2024-04-02T13:23:33.300 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:33.301 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1535287484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1535287484},{"type":"v1","addr":"172.21.15.69:6836","nonce":1535287484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6429,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2367107002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2367107002},{"type":"v1","addr":"172.21.15.44:6833","nonce":2367107002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/553542317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":553542317},{"type":"v1","addr":"172.21.15.69:6837","nonce":553542317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:23:24.803884+0000","modified":"2024-04-02T13:23:27.831218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6450":{"gid":6450,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6835/3127613802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3127613802},{"type":"v1","addr":"172.21.15.44:6835","nonce":3127613802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6450,"qdb_cluster":[6450]},"id":4}]} 2024-04-02T13:23:33.301 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 35 2024-04-02T13:23:33.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.300+0000 7f47720dc640 1 -- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f475806c8f0 msgr2=0x7f475806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.300+0000 7f47720dc640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f475806c8f0 0x7f475806edb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f475c000f60 tx=0x7f475c007c00 comp rx=0 tx=0).stop 2024-04-02T13:23:33.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.300+0000 7f47720dc640 1 -- 172.21.15.44:0/2569248699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f476c15e4a0 msgr2=0x7f476c0b32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.300+0000 7f47720dc640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f476c15e4a0 0x7f476c0b32f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4764009e80 tx=0x7f4764004a40 comp rx=0 tx=0).stop 2024-04-02T13:23:33.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.300+0000 7f47720dc640 1 -- 172.21.15.44:0/2569248699 shutdown_connections 2024-04-02T13:23:33.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.300+0000 7f47720dc640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f475806c8f0 0x7f475806edb0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.300+0000 7f47720dc640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f476c15e4a0 0x7f476c0b32f0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.300+0000 7f47720dc640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f476c0d0c50 0x7f476c0b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.300+0000 7f47720dc640 1 --2- 172.21.15.44:0/2569248699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f476c06ce40 0x7f476c157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.300+0000 7f47720dc640 1 -- 172.21.15.44:0/2569248699 >> 172.21.15.44:0/2569248699 conn(0x7f476c09aa20 msgr2=0x7f476c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:33.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.301+0000 7f47720dc640 1 -- 172.21.15.44:0/2569248699 shutdown_connections 2024-04-02T13:23:33.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.301+0000 7f47720dc640 1 -- 172.21.15.44:0/2569248699 wait complete. 2024-04-02T13:23:33.313 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-04-02T13:23:33.444 INFO:teuthology.orchestra.run.smithi044.stdout:{"message":"","return_code":0} 2024-04-02T13:23:33.444 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-02T13:23:33.445 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-04-02T13:23:33.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.485+0000 7fe084c44f40 1 Processor -- start 2024-04-02T13:23:33.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.486+0000 7fe084c44f40 1 -- start start 2024-04-02T13:23:33.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.486+0000 7fe084c44f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d15268d5b0 0x55d152670ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.486+0000 7fe084c44f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d15268f220 0x55d152718410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.486+0000 7fe084c44f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d152719430 0x55d152715410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.486+0000 7fe084c44f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d152719a20 con 0x55d15268d5b0 2024-04-02T13:23:33.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.486+0000 7fe084c44f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d1525ec3e0 con 0x55d152719430 2024-04-02T13:23:33.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.486+0000 7fe084c44f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55d15268d9c0 con 0x55d15268f220 2024-04-02T13:23:33.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.486+0000 7fe081eaa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d15268d5b0 0x55d152670ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.486+0000 7fe081eaa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d15268d5b0 0x55d152670ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56548/0 (socket says 172.21.15.44:56548) 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.486+0000 7fe081eaa640 1 -- 172.21.15.44:0/2845328636 learned_addr learned my addr 172.21.15.44:0/2845328636 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.486+0000 7fe0826ab640 1 --2- 172.21.15.44:0/2845328636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d152719430 0x55d152715410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.486+0000 7fe07bfff640 1 --2- 172.21.15.44:0/2845328636 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d15268f220 0x55d152718410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.487+0000 7fe081eaa640 1 -- 172.21.15.44:0/2845328636 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d15268f220 msgr2=0x55d152718410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.487+0000 7fe081eaa640 1 --2- 172.21.15.44:0/2845328636 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d15268f220 0x55d152718410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.487+0000 7fe081eaa640 1 -- 172.21.15.44:0/2845328636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d152719430 msgr2=0x55d152715410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.487+0000 7fe081eaa640 1 --2- 172.21.15.44:0/2845328636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d152719430 0x55d152715410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.487+0000 7fe081eaa640 1 -- 172.21.15.44:0/2845328636 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1527151d0 con 0x55d15268d5b0 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.487+0000 7fe081eaa640 1 --2- 172.21.15.44:0/2845328636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d15268d5b0 0x55d152670ff0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe07400a7a0 tx=0x7fe07400a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89166ffc1bf9083 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.487+0000 7fe080ea8640 1 -- 172.21.15.44:0/2845328636 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe074018070 con 0x55d15268d5b0 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.487+0000 7fe080ea8640 1 -- 172.21.15.44:0/2845328636 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe074013410 con 0x55d15268d5b0 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.487+0000 7fe080ea8640 1 -- 172.21.15.44:0/2845328636 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0740137c0 con 0x55d15268d5b0 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.488+0000 7fe084c44f40 1 -- 172.21.15.44:0/2845328636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d15268d5b0 msgr2=0x55d152670ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.488+0000 7fe084c44f40 1 --2- 172.21.15.44:0/2845328636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d15268d5b0 0x55d152670ff0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe07400a7a0 tx=0x7fe07400a890 comp rx=0 tx=0).stop 2024-04-02T13:23:33.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.488+0000 7fe084c44f40 1 -- 172.21.15.44:0/2845328636 shutdown_connections 2024-04-02T13:23:33.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.488+0000 7fe084c44f40 1 --2- 172.21.15.44:0/2845328636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d152719430 0x55d152715410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.488+0000 7fe084c44f40 1 --2- 172.21.15.44:0/2845328636 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d15268f220 0x55d152718410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.488+0000 7fe084c44f40 1 --2- 172.21.15.44:0/2845328636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d15268d5b0 0x55d152670ff0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.488+0000 7fe084c44f40 1 -- 172.21.15.44:0/2845328636 >> 172.21.15.44:0/2845328636 conn(0x55d152672500 msgr2=0x55d152684c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:33.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.488+0000 7fe084c44f40 1 -- 172.21.15.44:0/2845328636 shutdown_connections 2024-04-02T13:23:33.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.488+0000 7fe084c44f40 1 -- 172.21.15.44:0/2845328636 wait complete. 2024-04-02T13:23:33.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.488+0000 7fe084c44f40 1 Processor -- start 2024-04-02T13:23:33.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.488+0000 7fe084c44f40 1 -- start start 2024-04-02T13:23:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.489+0000 7fe084c44f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d15268d5b0 0x55d152685b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.489+0000 7fe084c44f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d15268f220 0x55d152689fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.489+0000 7fe084c44f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d152719430 0x55d1527ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.489+0000 7fe084c44f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d152714f30 con 0x55d15268f220 2024-04-02T13:23:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.489+0000 7fe084c44f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d15271c6d0 con 0x55d15268d5b0 2024-04-02T13:23:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.489+0000 7fe084c44f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55d15271c930 con 0x55d152719430 2024-04-02T13:23:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.489+0000 7fe07bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d15268f220 0x55d152689fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.489+0000 7fe07bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d15268f220 0x55d152689fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56560/0 (socket says 172.21.15.44:56560) 2024-04-02T13:23:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.489+0000 7fe07bfff640 1 -- 172.21.15.44:0/1478049768 learned_addr learned my addr 172.21.15.44:0/1478049768 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.489+0000 7fe081eaa640 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d15268d5b0 0x55d152685b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.489+0000 7fe0826ab640 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d152719430 0x55d1527ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.490+0000 7fe07bfff640 1 -- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d152719430 msgr2=0x55d1527ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.490+0000 7fe07bfff640 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d152719430 0x55d1527ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.490+0000 7fe07bfff640 1 -- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d15268d5b0 msgr2=0x55d152685b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.490+0000 7fe07bfff640 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d15268d5b0 0x55d152685b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.490+0000 7fe07bfff640 1 -- 172.21.15.44:0/1478049768 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d152718950 con 0x55d15268f220 2024-04-02T13:23:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.490+0000 7fe07bfff640 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d15268f220 0x55d152689fb0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe06c055040 tx=0x7fe06c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.490+0000 7fe07affd640 1 -- 172.21.15.44:0/1478049768 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c061070 con 0x55d15268f220 2024-04-02T13:23:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.490+0000 7fe07affd640 1 -- 172.21.15.44:0/1478049768 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe06c052d20 con 0x55d15268f220 2024-04-02T13:23:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.490+0000 7fe084c44f40 1 -- 172.21.15.44:0/1478049768 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d152711c40 con 0x55d15268f220 2024-04-02T13:23:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.490+0000 7fe07affd640 1 -- 172.21.15.44:0/1478049768 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c057040 con 0x55d15268f220 2024-04-02T13:23:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.490+0000 7fe084c44f40 1 -- 172.21.15.44:0/1478049768 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1526731d0 con 0x55d15268f220 2024-04-02T13:23:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.491+0000 7fe07affd640 1 -- 172.21.15.44:0/1478049768 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe06c064070 con 0x55d15268f220 2024-04-02T13:23:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.492+0000 7fe07affd640 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe06406b2d0 0x7fe06406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.492+0000 7fe07affd640 1 -- 172.21.15.44:0/1478049768 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fe06c0cc530 con 0x55d15268f220 2024-04-02T13:23:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.492+0000 7fe084c44f40 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55d1527147e0 0x55d152686770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.492+0000 7fe081eaa640 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe06406b2d0 0x7fe06406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.492+0000 7fe084c44f40 1 -- 172.21.15.44:0/1478049768 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:1 10.7 10:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65) -- 0x55d152712060 con 0x55d1527147e0 2024-04-02T13:23:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.492+0000 7fe0826ab640 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55d1527147e0 0x55d152686770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.492+0000 7fe081eaa640 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe06406b2d0 0x7fe06406d790 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe074016040 tx=0x7fe07400c000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.492+0000 7fe0826ab640 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55d1527147e0 0x55d152686770 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.493+0000 7fe0826ab640 1 -- 172.21.15.44:0/1478049768 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (94) out=94b] v0'0 uv1 ondisk = 0) ==== 164+0+94 (crc 0 0 0) 0x7fe07c002720 con 0x55d1527147e0 2024-04-02T13:23:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.493+0000 7fe084c44f40 1 -- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55d1527147e0 msgr2=0x55d152686770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.493+0000 7fe084c44f40 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55d1527147e0 0x55d152686770 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.493+0000 7fe084c44f40 1 -- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe06406b2d0 msgr2=0x7fe06406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.493+0000 7fe084c44f40 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe06406b2d0 0x7fe06406d790 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe074016040 tx=0x7fe07400c000 comp rx=0 tx=0).stop 2024-04-02T13:23:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.493+0000 7fe084c44f40 1 -- 172.21.15.44:0/1478049768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d15268f220 msgr2=0x55d152689fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.493+0000 7fe084c44f40 1 --2- 172.21.15.44:0/1478049768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d15268f220 0x55d152689fb0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe06c055040 tx=0x7fe06c002fb0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.493+0000 7fe0826ab640 1 -- 172.21.15.44:0/1478049768 reap_dead start 2024-04-02T13:23:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.493+0000 7fe084c44f40 1 -- 172.21.15.44:0/1478049768 shutdown_connections 2024-04-02T13:23:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.493+0000 7fe084c44f40 1 -- 172.21.15.44:0/1478049768 >> 172.21.15.44:0/1478049768 conn(0x55d152672500 msgr2=0x55d152712bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.493+0000 7fe084c44f40 1 -- 172.21.15.44:0/1478049768 shutdown_connections 2024-04-02T13:23:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.493+0000 7fe084c44f40 1 -- 172.21.15.44:0/1478049768 wait complete. 2024-04-02T13:23:33.500 INFO:teuthology.orchestra.run.smithi044.stdout:Xfiledir_test 2024-04-02T13:23:33.500 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:33.500 DEBUG:teuthology.orchestra.run.smithi044:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-04-02T13:23:33.587 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:23:33.587 INFO:teuthology.orchestra.run.smithi044.stdout: "ino": 1099511627777, 2024-04-02T13:23:33.587 INFO:teuthology.orchestra.run.smithi044.stdout: "ancestors": [ 2024-04-02T13:23:33.587 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:33.587 INFO:teuthology.orchestra.run.smithi044.stdout: "dirino": 1099511627776, 2024-04-02T13:23:33.587 INFO:teuthology.orchestra.run.smithi044.stdout: "dname": "file", 2024-04-02T13:23:33.587 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 2 2024-04-02T13:23:33.587 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:23:33.587 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:23:33.587 INFO:teuthology.orchestra.run.smithi044.stdout: "dirino": 1, 2024-04-02T13:23:33.588 INFO:teuthology.orchestra.run.smithi044.stdout: "dname": "dir_test", 2024-04-02T13:23:33.588 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 4 2024-04-02T13:23:33.588 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:33.588 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-02T13:23:33.588 INFO:teuthology.orchestra.run.smithi044.stdout: "pool": 10, 2024-04-02T13:23:33.588 INFO:teuthology.orchestra.run.smithi044.stdout: "old_pools": [] 2024-04-02T13:23:33.588 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:23:33.588 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:23:33.589 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'The backtrace is corrupted' 2024-04-02T13:23:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.674+0000 7fd350441f40 1 Processor -- start 2024-04-02T13:23:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.674+0000 7fd350441f40 1 -- start start 2024-04-02T13:23:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.674+0000 7fd350441f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5585f2ba6c70 0x5585f2bab020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.674+0000 7fd350441f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5585f2bd8b30 0x5585f2c82bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.674+0000 7fd350441f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585f2c7fee0 0x5585f2bd7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.674+0000 7fd350441f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5585f2b4f3e0 con 0x5585f2bd8b30 2024-04-02T13:23:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.674+0000 7fd350441f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5585f2c63590 con 0x5585f2c7fee0 2024-04-02T13:23:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.674+0000 7fd350441f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5585f2c62140 con 0x5585f2ba6c70 2024-04-02T13:23:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.674+0000 7fd34cf4f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5585f2bd8b30 0x5585f2c82bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd34cf4f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5585f2bd8b30 0x5585f2c82bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56570/0 (socket says 172.21.15.44:56570) 2024-04-02T13:23:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd34cf4f640 1 -- 172.21.15.44:0/1589467192 learned_addr learned my addr 172.21.15.44:0/1589467192 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd34d750640 1 --2- 172.21.15.44:0/1589467192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5585f2ba6c70 0x5585f2bab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd34df51640 1 --2- 172.21.15.44:0/1589467192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585f2c7fee0 0x5585f2bd7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd34cf4f640 1 -- 172.21.15.44:0/1589467192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5585f2ba6c70 msgr2=0x5585f2bab020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd34cf4f640 1 --2- 172.21.15.44:0/1589467192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5585f2ba6c70 0x5585f2bab020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd34cf4f640 1 -- 172.21.15.44:0/1589467192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585f2c7fee0 msgr2=0x5585f2bd7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd34cf4f640 1 --2- 172.21.15.44:0/1589467192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585f2c7fee0 0x5585f2bd7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd34cf4f640 1 -- 172.21.15.44:0/1589467192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585f2c83110 con 0x5585f2bd8b30 2024-04-02T13:23:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd34df51640 1 --2- 172.21.15.44:0/1589467192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585f2c7fee0 0x5585f2bd7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd34cf4f640 1 --2- 172.21.15.44:0/1589467192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5585f2bd8b30 0x5585f2c82bd0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd33800b230 tx=0x7fd33800b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d14c3e6e8423208 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd3477fe640 1 -- 172.21.15.44:0/1589467192 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33800e5c0 con 0x5585f2bd8b30 2024-04-02T13:23:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd3477fe640 1 -- 172.21.15.44:0/1589467192 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd338006070 con 0x5585f2bd8b30 2024-04-02T13:23:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.675+0000 7fd3477fe640 1 -- 172.21.15.44:0/1589467192 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33800e9c0 con 0x5585f2bd8b30 2024-04-02T13:23:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.676+0000 7fd350441f40 1 -- 172.21.15.44:0/1589467192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5585f2bd8b30 msgr2=0x5585f2c82bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.676+0000 7fd350441f40 1 --2- 172.21.15.44:0/1589467192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5585f2bd8b30 0x5585f2c82bd0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd33800b230 tx=0x7fd33800b6f0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.676+0000 7fd350441f40 1 -- 172.21.15.44:0/1589467192 shutdown_connections 2024-04-02T13:23:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.676+0000 7fd350441f40 1 --2- 172.21.15.44:0/1589467192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585f2c7fee0 0x5585f2bd7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.676+0000 7fd350441f40 1 --2- 172.21.15.44:0/1589467192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5585f2bd8b30 0x5585f2c82bd0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.676+0000 7fd350441f40 1 --2- 172.21.15.44:0/1589467192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5585f2ba6c70 0x5585f2bab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.676+0000 7fd350441f40 1 -- 172.21.15.44:0/1589467192 >> 172.21.15.44:0/1589467192 conn(0x5585f2ba81c0 msgr2=0x5585f2bd20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.676+0000 7fd350441f40 1 -- 172.21.15.44:0/1589467192 shutdown_connections 2024-04-02T13:23:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.676+0000 7fd350441f40 1 -- 172.21.15.44:0/1589467192 wait complete. 2024-04-02T13:23:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.676+0000 7fd350441f40 1 Processor -- start 2024-04-02T13:23:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.676+0000 7fd350441f40 1 -- start start 2024-04-02T13:23:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd350441f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5585f2ba6c70 0x5585f2c5e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd350441f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585f2bd8b30 0x5585f2c60b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd350441f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5585f2c7fee0 0x5585f2c61050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd350441f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5585f2ba8a50 con 0x5585f2ba6c70 2024-04-02T13:23:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd350441f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5585f2ba8790 con 0x5585f2bd8b30 2024-04-02T13:23:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd350441f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5585f2c826b0 con 0x5585f2c7fee0 2024-04-02T13:23:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd34d750640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5585f2ba6c70 0x5585f2c5e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd34d750640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5585f2ba6c70 0x5585f2c5e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56586/0 (socket says 172.21.15.44:56586) 2024-04-02T13:23:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd34d750640 1 -- 172.21.15.44:0/3245804681 learned_addr learned my addr 172.21.15.44:0/3245804681 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd34cf4f640 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585f2bd8b30 0x5585f2c60b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd34df51640 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5585f2c7fee0 0x5585f2c61050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd34d750640 1 -- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5585f2c7fee0 msgr2=0x5585f2c61050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd34d750640 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5585f2c7fee0 0x5585f2c61050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd34d750640 1 -- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585f2bd8b30 msgr2=0x5585f2c60b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd34d750640 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585f2bd8b30 0x5585f2c60b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd34d750640 1 -- 172.21.15.44:0/3245804681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585f2bec1a0 con 0x5585f2ba6c70 2024-04-02T13:23:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.677+0000 7fd34df51640 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5585f2c7fee0 0x5585f2c61050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.678+0000 7fd34d750640 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5585f2ba6c70 0x5585f2c5e6a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd348008840 tx=0x7fd348008930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.678+0000 7fd3467fc640 1 -- 172.21.15.44:0/3245804681 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3480147e0 con 0x5585f2ba6c70 2024-04-02T13:23:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.678+0000 7fd3467fc640 1 -- 172.21.15.44:0/3245804681 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd34801a070 con 0x5585f2ba6c70 2024-04-02T13:23:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.678+0000 7fd350441f40 1 -- 172.21.15.44:0/3245804681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5585f2c5f260 con 0x5585f2ba6c70 2024-04-02T13:23:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.678+0000 7fd3467fc640 1 -- 172.21.15.44:0/3245804681 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348014bc0 con 0x5585f2ba6c70 2024-04-02T13:23:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.678+0000 7fd350441f40 1 -- 172.21.15.44:0/3245804681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585f2c5eea0 con 0x5585f2ba6c70 2024-04-02T13:23:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.679+0000 7fd3467fc640 1 -- 172.21.15.44:0/3245804681 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd348015470 con 0x5585f2ba6c70 2024-04-02T13:23:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.679+0000 7fd3467fc640 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd33006b2d0 0x7fd33006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.680+0000 7fd3467fc640 1 -- 172.21.15.44:0/3245804681 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fd34808b2f0 con 0x5585f2ba6c70 2024-04-02T13:23:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.680+0000 7fd34cf4f640 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd33006b2d0 0x7fd33006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.680+0000 7fd350441f40 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x5585f2c5c990 0x5585f2d25080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.680+0000 7fd350441f40 1 -- 172.21.15.44:0/3245804681 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:1 10.7 10:e2e07930:::10000000001.00000000:head [setxattr parent (26) in=32b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65) -- 0x5585f2c79390 con 0x5585f2c5c990 2024-04-02T13:23:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.680+0000 7fd34df51640 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x5585f2c5c990 0x5585f2d25080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.680+0000 7fd34cf4f640 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd33006b2d0 0x7fd33006d790 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd33800b230 tx=0x7fd338009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.680+0000 7fd34df51640 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x5585f2c5c990 0x5585f2d25080 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.682+0000 7fd34df51640 1 -- 172.21.15.44:0/3245804681 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (26)] v65'2 uv2 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd34005a040 con 0x5585f2c5c990 2024-04-02T13:23:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.682+0000 7fd350441f40 1 -- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x5585f2c5c990 msgr2=0x5585f2d25080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.682+0000 7fd350441f40 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x5585f2c5c990 0x5585f2d25080 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.682+0000 7fd350441f40 1 -- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd33006b2d0 msgr2=0x7fd33006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.682+0000 7fd350441f40 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd33006b2d0 0x7fd33006d790 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd33800b230 tx=0x7fd338009000 comp rx=0 tx=0).stop 2024-04-02T13:23:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.682+0000 7fd350441f40 1 -- 172.21.15.44:0/3245804681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5585f2ba6c70 msgr2=0x5585f2c5e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.682+0000 7fd350441f40 1 --2- 172.21.15.44:0/3245804681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5585f2ba6c70 0x5585f2c5e6a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd348008840 tx=0x7fd348008930 comp rx=0 tx=0).stop 2024-04-02T13:23:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.682+0000 7fd34df51640 1 -- 172.21.15.44:0/3245804681 reap_dead start 2024-04-02T13:23:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.682+0000 7fd350441f40 1 -- 172.21.15.44:0/3245804681 shutdown_connections 2024-04-02T13:23:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.682+0000 7fd350441f40 1 -- 172.21.15.44:0/3245804681 >> 172.21.15.44:0/3245804681 conn(0x5585f2ba81c0 msgr2=0x5585f2c79980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.682+0000 7fd350441f40 1 -- 172.21.15.44:0/3245804681 shutdown_connections 2024-04-02T13:23:33.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.682+0000 7fd350441f40 1 -- 172.21.15.44:0/3245804681 wait complete. 2024-04-02T13:23:33.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:33.691 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub start / recursive 2024-04-02T13:23:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.831+0000 7f63ccff2640 1 Processor -- start 2024-04-02T13:23:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.831+0000 7f63ccff2640 1 -- start start 2024-04-02T13:23:33.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.832+0000 7f63ccff2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f63c815e1d0 0x7f63c80d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.832+0000 7f63ccff2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c80cec60 0x7f63c80d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.832+0000 7f63ccff2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c80d1640 con 0x7f63c815e5b0 2024-04-02T13:23:33.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.832+0000 7f63ccff2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f63c80cfe00 con 0x7f63c815e1d0 2024-04-02T13:23:33.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.832+0000 7f63ccff2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f63c80cf1f0 con 0x7f63c80cec60 2024-04-02T13:23:33.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.833+0000 7f63c5d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c80cec60 0x7f63c80d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.833+0000 7f63c659c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f63c815e1d0 0x7f63c80d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.833+0000 7f63c5d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c80cec60 0x7f63c80d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:45100/0 (socket says 172.21.15.44:45100) 2024-04-02T13:23:33.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.833+0000 7f63c6d9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f63c815e5b0 0x7f63c80b4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:47206/0 (socket says 172.21.15.44:47206) 2024-04-02T13:23:33.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.833+0000 7f63c5d9b640 1 -- 172.21.15.44:0/1333906403 learned_addr learned my addr 172.21.15.44:0/1333906403 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:33.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.833+0000 7f63c559a640 1 -- 172.21.15.44:0/1333906403 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2435248428 0 0) 0x7f63c80d1640 con 0x7f63c815e5b0 2024-04-02T13:23:33.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.833+0000 7f63c559a640 1 -- 172.21.15.44:0/1333906403 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63b0003520 con 0x7f63c815e5b0 2024-04-02T13:23:33.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.833+0000 7f63c5d9b640 1 -- 172.21.15.44:0/1333906403 >> v2:172.21.15.69:3300/0 conn(0x7f63c815e1d0 msgr2=0x7f63c80d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.833+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/1333906403 >> v2:172.21.15.69:3300/0 conn(0x7f63c815e1d0 0x7f63c80d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.833+0000 7f63c5d9b640 1 -- 172.21.15.44:0/1333906403 >> v1:172.21.15.69:6789/0 conn(0x7f63c815e5b0 legacy=0x7f63c80b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.833+0000 7f63c5d9b640 1 -- 172.21.15.44:0/1333906403 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c80cf4d0 con 0x7f63c80cec60 2024-04-02T13:23:33.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.834+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/1333906403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c80cec60 0x7f63c80d26c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009870 tx=0x7f63bc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=932ca9b4e5c30fd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.834+0000 7f63c559a640 1 -- 172.21.15.44:0/1333906403 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc035020 con 0x7f63c80cec60 2024-04-02T13:23:33.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.834+0000 7f63c559a640 1 -- 172.21.15.44:0/1333906403 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63bc031640 con 0x7f63c80cec60 2024-04-02T13:23:33.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.834+0000 7f63c559a640 1 -- 172.21.15.44:0/1333906403 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc02d040 con 0x7f63c80cec60 2024-04-02T13:23:33.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.835+0000 7f63ccff2640 1 -- 172.21.15.44:0/1333906403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c80cec60 msgr2=0x7f63c80d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.835+0000 7f63ccff2640 1 --2- 172.21.15.44:0/1333906403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c80cec60 0x7f63c80d26c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009870 tx=0x7f63bc02b8f0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.835+0000 7f63ccff2640 1 -- 172.21.15.44:0/1333906403 shutdown_connections 2024-04-02T13:23:33.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.835+0000 7f63ccff2640 1 --2- 172.21.15.44:0/1333906403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c80cec60 0x7f63c80d26c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.835+0000 7f63ccff2640 1 --2- 172.21.15.44:0/1333906403 >> v2:172.21.15.69:3300/0 conn(0x7f63c815e1d0 0x7f63c80d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.835+0000 7f63ccff2640 1 -- 172.21.15.44:0/1333906403 >> 172.21.15.44:0/1333906403 conn(0x7f63c809d3b0 msgr2=0x7f63c80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:33.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.835+0000 7f63ccff2640 1 -- 172.21.15.44:0/1333906403 shutdown_connections 2024-04-02T13:23:33.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.835+0000 7f63ccff2640 1 -- 172.21.15.44:0/1333906403 wait complete. 2024-04-02T13:23:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.836+0000 7f63ccff2640 1 Processor -- start 2024-04-02T13:23:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.836+0000 7f63ccff2640 1 -- start start 2024-04-02T13:23:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63ccff2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63c80cec60 0x7f63c815b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63ccff2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c815e1d0 0x7f63c8155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63ccff2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c815e5b0 0x7f63c8156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63ccff2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f63c80d2c00 con 0x7f63c80cec60 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63ccff2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f63c80cfa90 con 0x7f63c815e1d0 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63ccff2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f63c80d2d80 con 0x7f63c815e5b0 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63c659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63c80cec60 0x7f63c815b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63c659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63c80cec60 0x7f63c815b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56590/0 (socket says 172.21.15.44:56590) 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63c659c640 1 -- 172.21.15.44:0/2373343735 learned_addr learned my addr 172.21.15.44:0/2373343735 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63c6d9d640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c815e5b0 0x7f63c8156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c815e1d0 0x7f63c8155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63c659c640 1 -- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c815e5b0 msgr2=0x7f63c8156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63c659c640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c815e5b0 0x7f63c8156af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63c659c640 1 -- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c815e1d0 msgr2=0x7f63c8155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63c659c640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c815e1d0 0x7f63c8155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63c659c640 1 -- 172.21.15.44:0/2373343735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c81f7820 con 0x7f63c80cec60 2024-04-02T13:23:33.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.837+0000 7f63c659c640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63c80cec60 0x7f63c815b250 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f63c000e9b0 tx=0x7f63c000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.838+0000 7f63b6ffd640 1 -- 172.21.15.44:0/2373343735 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0017070 con 0x7f63c80cec60 2024-04-02T13:23:33.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.838+0000 7f63b6ffd640 1 -- 172.21.15.44:0/2373343735 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63c000c9a0 con 0x7f63c80cec60 2024-04-02T13:23:33.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.838+0000 7f63ccff2640 1 -- 172.21.15.44:0/2373343735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c81f7b00 con 0x7f63c80cec60 2024-04-02T13:23:33.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.838+0000 7f63b6ffd640 1 -- 172.21.15.44:0/2373343735 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c000cd90 con 0x7f63c80cec60 2024-04-02T13:23:33.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.838+0000 7f63ccff2640 1 -- 172.21.15.44:0/2373343735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c81f80c0 con 0x7f63c80cec60 2024-04-02T13:23:33.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.839+0000 7f63b6ffd640 1 -- 172.21.15.44:0/2373343735 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f63c001a070 con 0x7f63c80cec60 2024-04-02T13:23:33.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.839+0000 7f63b6ffd640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f63a806c920 0x7f63a806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.840+0000 7f63b6ffd640 1 -- 172.21.15.44:0/2373343735 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f63c008bfc0 con 0x7f63c80cec60 2024-04-02T13:23:33.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.840+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f63a806c920 0x7f63a806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.840+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f63a806c920 0x7f63a806ede0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f63bc036000 tx=0x7f63bc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.840+0000 7f63cfbd6740 1 Processor -- start 2024-04-02T13:23:33.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.841+0000 7f63cfbd6740 1 -- start start 2024-04-02T13:23:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.841+0000 7f63cfbd6740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16420 0x5614cdbe93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.841+0000 7f63cfbd6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 0x5614cdc191b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.841+0000 7f63cfbd6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdbe9900 0x5614cdc196f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.841+0000 7f63cfbd6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5614cdb52430 con 0x5614cdc16800 2024-04-02T13:23:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.841+0000 7f63cfbd6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5614cdaf0a40 con 0x5614cdbe9900 2024-04-02T13:23:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.841+0000 7f63cfbd6740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5614cdb4e3e0 con 0x5614cdc16420 2024-04-02T13:23:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.841+0000 7f63c5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 0x5614cdc191b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.842+0000 7f63c5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 0x5614cdc191b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56592/0 (socket says 172.21.15.44:56592) 2024-04-02T13:23:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.842+0000 7f63c5d9b640 1 -- 172.21.15.44:0/1253907005 learned_addr learned my addr 172.21.15.44:0/1253907005 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.842+0000 7f63c6d9d640 1 --2- 172.21.15.44:0/1253907005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdbe9900 0x5614cdc196f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.842+0000 7f63c659c640 1 --2- 172.21.15.44:0/1253907005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16420 0x5614cdbe93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.842+0000 7f63c5d9b640 1 -- 172.21.15.44:0/1253907005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16420 msgr2=0x5614cdbe93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.842+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/1253907005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16420 0x5614cdbe93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.842+0000 7f63c5d9b640 1 -- 172.21.15.44:0/1253907005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdbe9900 msgr2=0x5614cdc196f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.842+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/1253907005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdbe9900 0x5614cdc196f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.842+0000 7f63c5d9b640 1 -- 172.21.15.44:0/1253907005 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614cdb1a960 con 0x5614cdc16800 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.842+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/1253907005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 0x5614cdc191b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f63bc02bf00 tx=0x7f63bc030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7ae4a33285d4b86 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.842+0000 7f63a7fff640 1 -- 172.21.15.44:0/1253907005 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc035040 con 0x5614cdc16800 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.842+0000 7f63a7fff640 1 -- 172.21.15.44:0/1253907005 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63bc0036d0 con 0x5614cdc16800 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.842+0000 7f63a7fff640 1 -- 172.21.15.44:0/1253907005 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc003a80 con 0x5614cdc16800 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.843+0000 7f63cfbd6740 1 -- 172.21.15.44:0/1253907005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 msgr2=0x5614cdc191b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.843+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/1253907005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 0x5614cdc191b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f63bc02bf00 tx=0x7f63bc030000 comp rx=0 tx=0).stop 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.843+0000 7f63cfbd6740 1 -- 172.21.15.44:0/1253907005 shutdown_connections 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.843+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/1253907005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdbe9900 0x5614cdc196f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.843+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/1253907005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 0x5614cdc191b0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.843+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/1253907005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16420 0x5614cdbe93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.843+0000 7f63cfbd6740 1 -- 172.21.15.44:0/1253907005 >> 172.21.15.44:0/1253907005 conn(0x5614cdbeb630 msgr2=0x5614cdc341b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.843+0000 7f63cfbd6740 1 -- 172.21.15.44:0/1253907005 shutdown_connections 2024-04-02T13:23:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.843+0000 7f63cfbd6740 1 -- 172.21.15.44:0/1253907005 wait complete. 2024-04-02T13:23:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.844+0000 7f63cfbd6740 1 -- start start 2024-04-02T13:23:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.844+0000 7f63cfbd6740 1 Processor -- start 2024-04-02T13:23:33.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.844+0000 7f63cfbd6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc16420 0x5614cdc1b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.844+0000 7f63cfbd6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 0x5614cdc44840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.844+0000 7f63cfbd6740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc05eb0 0x5614cdc03450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63cfbd6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5614cdbedcd0 con 0x5614cdc16800 2024-04-02T13:23:33.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63cfbd6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5614cdaee710 con 0x5614cdc16420 2024-04-02T13:23:33.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63cfbd6740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5614cdbf0510 con 0x5614cdc05eb0 2024-04-02T13:23:33.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63cfbd6740 10 client.0 _ll_register_callbacks cb 0x5614cd865b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:23:33.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63c5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 0x5614cdc44840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63c5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 0x5614cdc44840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56606/0 (socket says 172.21.15.44:56606) 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63c5d9b640 1 -- 172.21.15.44:0/2633667268 learned_addr learned my addr 172.21.15.44:0/2633667268 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63c6d9d640 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc05eb0 0x5614cdc03450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63c659c640 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc16420 0x5614cdc1b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63c5d9b640 1 -- 172.21.15.44:0/2633667268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc05eb0 msgr2=0x5614cdc03450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc05eb0 0x5614cdc03450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63c5d9b640 1 -- 172.21.15.44:0/2633667268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc16420 msgr2=0x5614cdc1b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc16420 0x5614cdc1b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63c5d9b640 1 -- 172.21.15.44:0/2633667268 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614cd9327b0 con 0x5614cdc16800 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63c6d9d640 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc05eb0 0x5614cdc03450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63c659c640 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc16420 0x5614cdc1b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 0x5614cdc44840 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009630 tx=0x7f63bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96ba184ccc6dce5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63a57fa640 1 -- 172.21.15.44:0/2633667268 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc05c060 con 0x5614cdc16800 2024-04-02T13:23:33.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63a57fa640 1 -- 172.21.15.44:0/2633667268 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f63bc003bd0 con 0x5614cdc16800 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63cfbd6740 1 -- 172.21.15.44:0/2633667268 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63bc0036d0 con 0x5614cdc16800 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63a57fa640 1 -- 172.21.15.44:0/2633667268 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f63bc062050 con 0x5614cdc16800 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63cfbd6740 10 client.6636 fetch_fsmap learned FSMap version 35 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.845+0000 7f63cfbd6740 1 -- 172.21.15.44:0/2633667268 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x5614cd992f40 con 0x5614cdc16800 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.846+0000 7f63a57fa640 1 -- 172.21.15.44:0/2633667268 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63bc0036d0 con 0x5614cdc16800 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.846+0000 7f63a57fa640 1 -- 172.21.15.44:0/2633667268 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f63bc031440 con 0x5614cdc16800 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.846+0000 7f63cfbd6740 10 client.6636 fetch_fsmap finished waiting for FSMap version 35 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.846+0000 7f63cfbd6740 10 client.6636 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.846+0000 7f63cfbd6740 20 client.6636 populate_metadata read hostname 'smithi044' 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.846+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614cdc20420 0x5614cdc33f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.846+0000 7f63cfbd6740 4 client.6636 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.846+0000 7f63cfbd6740 1 -- 172.21.15.44:0/2633667268 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614cdb5f970 con 0x5614cdc20420 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.846+0000 7f63c659c640 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614cdc20420 0x5614cdc33f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.846+0000 7f63c659c640 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614cdc20420 0x5614cdc33f30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c773ff81834a7d03 server_cookie=844ebd5b09aaa090 in_seq=0 out_seq=0 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.846+0000 7f63a57fa640 10 client.6636 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63a57fa640 1 -- 172.21.15.44:0/2633667268 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5614cdb5f970 con 0x5614cdc20420 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63a57fa640 10 client.6636 handle_command_reply: tid=0 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63cfbd6740 1 client.6636 shutdown 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63cfbd6740 1 -- 172.21.15.44:0/2633667268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 msgr2=0x5614cdc44840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 0x5614cdc44840 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009630 tx=0x7f63bc004730 comp rx=0 tx=0).stop 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63cfbd6740 1 -- 172.21.15.44:0/2633667268 shutdown_connections 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614cdc20420 0x5614cdc33f30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc05eb0 0x5614cdc03450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16800 0x5614cdc44840 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/2633667268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc16420 0x5614cdc1b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63a57fa640 0 client.6636 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63cfbd6740 1 -- 172.21.15.44:0/2633667268 >> 172.21.15.44:0/2633667268 conn(0x5614cdbe9900 msgr2=0x5614cdbed2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63cfbd6740 1 -- 172.21.15.44:0/2633667268 shutdown_connections 2024-04-02T13:23:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63cfbd6740 1 -- 172.21.15.44:0/2633667268 wait complete. 2024-04-02T13:23:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.847+0000 7f63cfbd6740 20 client.6636 trim_cache size 0 max 16384 2024-04-02T13:23:33.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.865+0000 7f63cfbd6740 1 Processor -- start 2024-04-02T13:23:33.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.865+0000 7f63cfbd6740 1 -- start start 2024-04-02T13:23:33.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.865+0000 7f63cfbd6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc05eb0 0x5614cdc376b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.865+0000 7f63cfbd6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc16420 0x5614cdc33e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.865+0000 7f63cfbd6740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16800 0x5614cdc343c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.865+0000 7f63cfbd6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5614cdc21450 con 0x5614cdc05eb0 2024-04-02T13:23:33.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.865+0000 7f63cfbd6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f63c8200ca0 con 0x5614cdc16420 2024-04-02T13:23:33.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.865+0000 7f63cfbd6740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f63c81ffcf0 con 0x5614cdc16800 2024-04-02T13:23:33.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63c659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc05eb0 0x5614cdc376b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63c659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc05eb0 0x5614cdc376b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56608/0 (socket says 172.21.15.44:56608) 2024-04-02T13:23:33.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63c659c640 1 -- 172.21.15.44:0/2526458201 learned_addr learned my addr 172.21.15.44:0/2526458201 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:33.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/2526458201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc16420 0x5614cdc33e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63c6d9d640 1 --2- 172.21.15.44:0/2526458201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16800 0x5614cdc343c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63c659c640 1 -- 172.21.15.44:0/2526458201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16800 msgr2=0x5614cdc343c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63c659c640 1 --2- 172.21.15.44:0/2526458201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16800 0x5614cdc343c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63c659c640 1 -- 172.21.15.44:0/2526458201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc16420 msgr2=0x5614cdc33e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63c659c640 1 --2- 172.21.15.44:0/2526458201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc16420 0x5614cdc33e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63c659c640 1 -- 172.21.15.44:0/2526458201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614cd8260a0 con 0x5614cdc05eb0 2024-04-02T13:23:33.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63c6d9d640 1 --2- 172.21.15.44:0/2526458201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16800 0x5614cdc343c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:23:33.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63c659c640 1 --2- 172.21.15.44:0/2526458201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc05eb0 0x5614cdc376b0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f63c00596e0 tx=0x7f63c008a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390b38513b10b9df server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63a77fe640 1 -- 172.21.15.44:0/2526458201 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0021040 con 0x5614cdc05eb0 2024-04-02T13:23:33.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63a77fe640 1 -- 172.21.15.44:0/2526458201 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63c0057f10 con 0x5614cdc05eb0 2024-04-02T13:23:33.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.866+0000 7f63a77fe640 1 -- 172.21.15.44:0/2526458201 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0059060 con 0x5614cdc05eb0 2024-04-02T13:23:33.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.867+0000 7f63cfbd6740 1 -- 172.21.15.44:0/2526458201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc05eb0 msgr2=0x5614cdc376b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.867+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/2526458201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc05eb0 0x5614cdc376b0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f63c00596e0 tx=0x7f63c008a230 comp rx=0 tx=0).stop 2024-04-02T13:23:33.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.867+0000 7f63cfbd6740 1 -- 172.21.15.44:0/2526458201 shutdown_connections 2024-04-02T13:23:33.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.867+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/2526458201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16800 0x5614cdc343c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.867+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/2526458201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc16420 0x5614cdc33e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.867+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/2526458201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc05eb0 0x5614cdc376b0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.867+0000 7f63cfbd6740 1 -- 172.21.15.44:0/2526458201 >> 172.21.15.44:0/2526458201 conn(0x5614cdbe9900 msgr2=0x5614cdc1b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:33.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.867+0000 7f63cfbd6740 1 -- 172.21.15.44:0/2526458201 shutdown_connections 2024-04-02T13:23:33.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.867+0000 7f63cfbd6740 1 -- 172.21.15.44:0/2526458201 wait complete. 2024-04-02T13:23:33.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.867+0000 7f63cfbd6740 1 -- start start 2024-04-02T13:23:33.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.867+0000 7f63cfbd6740 1 Processor -- start 2024-04-02T13:23:33.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.868+0000 7f63cfbd6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc05eb0 0x5614cdcb61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.868+0000 7f63cfbd6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16420 0x5614cdc147f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.868+0000 7f63cfbd6740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16800 0x5614cdc14d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.868+0000 7f63cfbd6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f63c81ff700 con 0x5614cdc16420 2024-04-02T13:23:33.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.868+0000 7f63cfbd6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f63c815f450 con 0x5614cdc05eb0 2024-04-02T13:23:33.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.868+0000 7f63cfbd6740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5614cdc21450 con 0x5614cdc16800 2024-04-02T13:23:33.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.868+0000 7f63cfbd6740 10 client.0 _ll_register_callbacks cb 0x7f63b8004be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:23:33.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.868+0000 7f63c5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16420 0x5614cdc147f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.868+0000 7f63c5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16420 0x5614cdc147f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56620/0 (socket says 172.21.15.44:56620) 2024-04-02T13:23:33.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.868+0000 7f63c5d9b640 1 -- 172.21.15.44:0/1529165248 learned_addr learned my addr 172.21.15.44:0/1529165248 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:33.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.868+0000 7f63c659c640 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc05eb0 0x5614cdcb61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.868+0000 7f63c6d9d640 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16800 0x5614cdc14d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63c5d9b640 1 -- 172.21.15.44:0/1529165248 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16800 msgr2=0x5614cdc14d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16800 0x5614cdc14d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63c5d9b640 1 -- 172.21.15.44:0/1529165248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc05eb0 msgr2=0x5614cdcb61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc05eb0 0x5614cdcb61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63c5d9b640 1 -- 172.21.15.44:0/1529165248 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614cdc15450 con 0x5614cdc16420 2024-04-02T13:23:33.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63c6d9d640 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16800 0x5614cdc14d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:33.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63c5d9b640 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16420 0x5614cdc147f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0040b0 tx=0x7f63bc008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3456bba584aa7f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:23:33.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63a57fa640 1 -- 172.21.15.44:0/1529165248 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc06b060 con 0x5614cdc16420 2024-04-02T13:23:33.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63a57fa640 1 -- 172.21.15.44:0/1529165248 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f63bc003dd0 con 0x5614cdc16420 2024-04-02T13:23:33.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63cfbd6740 1 -- 172.21.15.44:0/1529165248 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63c0057f10 con 0x5614cdc16420 2024-04-02T13:23:33.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63a57fa640 1 -- 172.21.15.44:0/1529165248 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f63bc071050 con 0x5614cdc16420 2024-04-02T13:23:33.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63cfbd6740 10 client.6642 fetch_fsmap learned FSMap version 35 2024-04-02T13:23:33.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63cfbd6740 1 -- 172.21.15.44:0/1529165248 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x5614cdc55580 con 0x5614cdc16420 2024-04-02T13:23:33.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63a57fa640 1 -- 172.21.15.44:0/1529165248 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63c0057f10 con 0x5614cdc16420 2024-04-02T13:23:33.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63a57fa640 1 -- 172.21.15.44:0/1529165248 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f63bc074020 con 0x5614cdc16420 2024-04-02T13:23:33.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63cfbd6740 10 client.6642 fetch_fsmap finished waiting for FSMap version 35 2024-04-02T13:23:33.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63cfbd6740 10 client.6642 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-04-02T13:23:33.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63cfbd6740 20 client.6642 populate_metadata read hostname 'smithi044' 2024-04-02T13:23:33.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614cdbef620 0x5614cdc54500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63cfbd6740 4 client.6642 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}] 2024-04-02T13:23:33.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.869+0000 7f63cfbd6740 1 -- 172.21.15.44:0/1529165248 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}) -- 0x5614cdc21980 con 0x5614cdbef620 2024-04-02T13:23:33.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.870+0000 7f63c659c640 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614cdbef620 0x5614cdc54500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.870+0000 7f63c659c640 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614cdbef620 0x5614cdc54500 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4326c3cabf668383 server_cookie=c0806c58dc93f107 in_seq=0 out_seq=0 2024-04-02T13:23:33.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.870+0000 7f63a57fa640 10 client.6642 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:23:33.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.870+0000 7f63a57fa640 1 -- 172.21.15.44:0/1529165248 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5614cdc21980 con 0x5614cdbef620 2024-04-02T13:23:33.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.870+0000 7f63a57fa640 10 client.6642 handle_command_reply: tid=0 2024-04-02T13:23:33.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.870+0000 7f63cfbd6740 1 client.6642 shutdown 2024-04-02T13:23:33.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63cfbd6740 1 -- 172.21.15.44:0/1529165248 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16420 msgr2=0x5614cdc147f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16420 0x5614cdc147f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0040b0 tx=0x7f63bc008030 comp rx=0 tx=0).stop 2024-04-02T13:23:33.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63cfbd6740 1 -- 172.21.15.44:0/1529165248 shutdown_connections 2024-04-02T13:23:33.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614cdbef620 0x5614cdc54500 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614cdc16800 0x5614cdc14d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63a57fa640 0 client.6642 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:23:33.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614cdc16420 0x5614cdc147f0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63cfbd6740 1 --2- 172.21.15.44:0/1529165248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614cdc05eb0 0x5614cdcb61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63cfbd6740 1 -- 172.21.15.44:0/1529165248 >> 172.21.15.44:0/1529165248 conn(0x5614cdbe9900 msgr2=0x5614cdc1bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:33.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63cfbd6740 1 -- 172.21.15.44:0/1529165248 shutdown_connections 2024-04-02T13:23:33.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63cfbd6740 1 -- 172.21.15.44:0/1529165248 wait complete. 2024-04-02T13:23:33.881 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:23:33.871+0000 7fb859322640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={6603=0-4194304@1} caps={6603=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6603 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x55dcba8b1600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-02T13:23:33.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63cfbd6740 20 client.6642 trim_cache size 0 max 16384 2024-04-02T13:23:33.882 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:23:33.882 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-02T13:23:33.882 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "c82feae0-27e5-44d9-b782-d5babebbc651", 2024-04-02T13:23:33.882 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-02T13:23:33.882 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:23:33.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63ccff2640 1 -- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f63a806c920 msgr2=0x7f63a806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63ccff2640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f63a806c920 0x7f63a806ede0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f63bc036000 tx=0x7f63bc03c040 comp rx=0 tx=0).stop 2024-04-02T13:23:33.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63ccff2640 1 -- 172.21.15.44:0/2373343735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63c80cec60 msgr2=0x7f63c815b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.871+0000 7f63ccff2640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63c80cec60 0x7f63c815b250 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f63c000e9b0 tx=0x7f63c000ee70 comp rx=0 tx=0).stop 2024-04-02T13:23:33.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.872+0000 7f63ccff2640 1 -- 172.21.15.44:0/2373343735 shutdown_connections 2024-04-02T13:23:33.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.872+0000 7f63ccff2640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f63a806c920 0x7f63a806ede0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.872+0000 7f63ccff2640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c815e5b0 0x7f63c8156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.872+0000 7f63ccff2640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c815e1d0 0x7f63c8155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.872+0000 7f63ccff2640 1 --2- 172.21.15.44:0/2373343735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63c80cec60 0x7f63c815b250 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.872+0000 7f63ccff2640 1 -- 172.21.15.44:0/2373343735 >> 172.21.15.44:0/2373343735 conn(0x7f63c809d3b0 msgr2=0x7f63c80b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:33.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.872+0000 7f63ccff2640 1 -- 172.21.15.44:0/2373343735 shutdown_connections 2024-04-02T13:23:33.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.872+0000 7f63ccff2640 1 -- 172.21.15.44:0/2373343735 wait complete. 2024-04-02T13:23:33.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:23:33.885 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-04-02T13:23:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.987+0000 7f65e6bba640 1 Processor -- start 2024-04-02T13:23:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.987+0000 7f65e6bba640 1 -- start start 2024-04-02T13:23:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.988+0000 7f65e6bba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65e015e1d0 0x7f65e00d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.988+0000 7f65e6bba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f65e00cec60 0x7f65e00d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.988+0000 7f65e6bba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65e00d1640 con 0x7f65e015e5b0 2024-04-02T13:23:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.988+0000 7f65e6bba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f65e00cfe00 con 0x7f65e00cec60 2024-04-02T13:23:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.988+0000 7f65e6bba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f65e00cf1f0 con 0x7f65e015e1d0 2024-04-02T13:23:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.989+0000 7f65dffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f65e00cec60 0x7f65e00d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.989+0000 7f65dffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f65e00cec60 0x7f65e00d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:59610/0 (socket says 172.21.15.44:59610) 2024-04-02T13:23:33.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.989+0000 7f65e5157640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f65e015e5b0 0x7f65e00b4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:47212/0 (socket says 172.21.15.44:47212) 2024-04-02T13:23:33.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.989+0000 7f65dffff640 1 -- 172.21.15.44:0/2257902708 learned_addr learned my addr 172.21.15.44:0/2257902708 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:33.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.989+0000 7f65e4956640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65e015e1d0 0x7f65e00d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.989+0000 7f65dffff640 1 -- 172.21.15.44:0/2257902708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65e015e1d0 msgr2=0x7f65e00d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:33.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.989+0000 7f65dffff640 1 --2- 172.21.15.44:0/2257902708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65e015e1d0 0x7f65e00d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.989+0000 7f65dffff640 1 -- 172.21.15.44:0/2257902708 >> v1:172.21.15.69:6789/0 conn(0x7f65e015e5b0 legacy=0x7f65e00b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.989+0000 7f65dffff640 1 -- 172.21.15.44:0/2257902708 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e00cf4d0 con 0x7f65e00cec60 2024-04-02T13:23:33.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.989+0000 7f65df7fe640 1 -- 172.21.15.44:0/2257902708 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2559755811 0 0) 0x7f65e00d1640 con 0x7f65e015e5b0 2024-04-02T13:23:33.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.989+0000 7f65e4956640 1 --2- 172.21.15.44:0/2257902708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65e015e1d0 0x7f65e00d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:23:33.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.990+0000 7f65dffff640 1 --2- 172.21.15.44:0/2257902708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e00cec60 0x7f65e00d26c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f65d4009870 tx=0x7f65d402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a765ec3d8b7661b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.990+0000 7f65df7fe640 1 -- 172.21.15.44:0/2257902708 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d4035020 con 0x7f65e00cec60 2024-04-02T13:23:33.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.990+0000 7f65df7fe640 1 -- 172.21.15.44:0/2257902708 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65d4031640 con 0x7f65e00cec60 2024-04-02T13:23:33.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.990+0000 7f65df7fe640 1 -- 172.21.15.44:0/2257902708 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d402d040 con 0x7f65e00cec60 2024-04-02T13:23:33.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.990+0000 7f65e6bba640 1 -- 172.21.15.44:0/2257902708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e00cec60 msgr2=0x7f65e00d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.990+0000 7f65e6bba640 1 --2- 172.21.15.44:0/2257902708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e00cec60 0x7f65e00d26c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f65d4009870 tx=0x7f65d402b8f0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.990+0000 7f65e6bba640 1 -- 172.21.15.44:0/2257902708 shutdown_connections 2024-04-02T13:23:33.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.990+0000 7f65e6bba640 1 --2- 172.21.15.44:0/2257902708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e00cec60 0x7f65e00d26c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.990+0000 7f65e6bba640 1 --2- 172.21.15.44:0/2257902708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65e015e1d0 0x7f65e00d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.990+0000 7f65e6bba640 1 -- 172.21.15.44:0/2257902708 >> 172.21.15.44:0/2257902708 conn(0x7f65e009d3b0 msgr2=0x7f65e00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:33.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.990+0000 7f65e6bba640 1 -- 172.21.15.44:0/2257902708 shutdown_connections 2024-04-02T13:23:33.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.991+0000 7f65e6bba640 1 -- 172.21.15.44:0/2257902708 wait complete. 2024-04-02T13:23:33.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.992+0000 7f65e6bba640 1 Processor -- start 2024-04-02T13:23:33.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.992+0000 7f65e6bba640 1 -- start start 2024-04-02T13:23:33.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.992+0000 7f65e6bba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65e00cec60 0x7f65e0150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.992+0000 7f65e6bba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e015e1d0 0x7f65e015ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.992+0000 7f65e6bba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e015e5b0 0x7f65e00cc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.992+0000 7f65e6bba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f65e00d2c00 con 0x7f65e015e5b0 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.992+0000 7f65e6bba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f65e00cfa90 con 0x7f65e015e1d0 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.992+0000 7f65e6bba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f65e00d2d80 con 0x7f65e00cec60 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65e5157640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e015e5b0 0x7f65e00cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65e5157640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e015e5b0 0x7f65e00cc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56624/0 (socket says 172.21.15.44:56624) 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65e5157640 1 -- 172.21.15.44:0/3976537326 learned_addr learned my addr 172.21.15.44:0/3976537326 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65dffff640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e015e1d0 0x7f65e015ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65e4956640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65e00cec60 0x7f65e0150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65e5157640 1 -- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65e00cec60 msgr2=0x7f65e0150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65e5157640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65e00cec60 0x7f65e0150420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65e5157640 1 -- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e015e1d0 msgr2=0x7f65e015ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65e5157640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e015e1d0 0x7f65e015ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:33.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65e5157640 1 -- 172.21.15.44:0/3976537326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e015f420 con 0x7f65e015e5b0 2024-04-02T13:23:33.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65e5157640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e015e5b0 0x7f65e00cc200 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f65d000c440 tx=0x7f65d000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:33.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65dd7fa640 1 -- 172.21.15.44:0/3976537326 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d00074e0 con 0x7f65e015e5b0 2024-04-02T13:23:33.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65dd7fa640 1 -- 172.21.15.44:0/3976537326 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65d00047e0 con 0x7f65e015e5b0 2024-04-02T13:23:33.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65e6bba640 1 -- 172.21.15.44:0/3976537326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e015f700 con 0x7f65e015e5b0 2024-04-02T13:23:33.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65dd7fa640 1 -- 172.21.15.44:0/3976537326 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d0004bd0 con 0x7f65e015e5b0 2024-04-02T13:23:33.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.993+0000 7f65e6bba640 1 -- 172.21.15.44:0/3976537326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e015ffa0 con 0x7f65e015e5b0 2024-04-02T13:23:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.995+0000 7f65dd7fa640 1 -- 172.21.15.44:0/3976537326 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f65d0011070 con 0x7f65e015e5b0 2024-04-02T13:23:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.995+0000 7f65dd7fa640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f65cc06c8f0 0x7f65cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.995+0000 7f65dd7fa640 1 -- 172.21.15.44:0/3976537326 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f65d008b790 con 0x7f65e015e5b0 2024-04-02T13:23:34.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.996+0000 7f65e4956640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f65cc06c8f0 0x7f65cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.996+0000 7f65e4956640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f65cc06c8f0 0x7f65cc06edb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f65d8009bd0 tx=0x7f65d800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.996+0000 7f65e97a7740 1 Processor -- start 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.996+0000 7f65e97a7740 1 -- start start 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e97a7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587836719a0 0x558783644940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e97a7740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558783671d80 0x558783674730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e97a7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558783644e80 0x558783674c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e97a7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5587835ae830 con 0x5587836719a0 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e97a7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558783556820 con 0x558783644e80 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e97a7740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5587835aa7e0 con 0x558783671d80 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e4956640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587836719a0 0x558783644940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e4956640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587836719a0 0x558783644940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56636/0 (socket says 172.21.15.44:56636) 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e4956640 1 -- 172.21.15.44:0/1232604056 learned_addr learned my addr 172.21.15.44:0/1232604056 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e5157640 1 --2- 172.21.15.44:0/1232604056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558783644e80 0x558783674c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65dffff640 1 --2- 172.21.15.44:0/1232604056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558783671d80 0x558783674730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e4956640 1 -- 172.21.15.44:0/1232604056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558783671d80 msgr2=0x558783674730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e4956640 1 --2- 172.21.15.44:0/1232604056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558783671d80 0x558783674730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e4956640 1 -- 172.21.15.44:0/1232604056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558783644e80 msgr2=0x558783674c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e4956640 1 --2- 172.21.15.44:0/1232604056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558783644e80 0x558783674c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.997+0000 7f65e4956640 1 -- 172.21.15.44:0/1232604056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55878338c7d0 con 0x5587836719a0 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.998+0000 7f65e4956640 1 --2- 172.21.15.44:0/1232604056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587836719a0 0x558783644940 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f65d8003da0 tx=0x7f65d8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2030c2c841f0e26 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.998+0000 7f65c67fc640 1 -- 172.21.15.44:0/1232604056 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d801b070 con 0x5587836719a0 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.998+0000 7f65c67fc640 1 -- 172.21.15.44:0/1232604056 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65d8027070 con 0x5587836719a0 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.998+0000 7f65c67fc640 1 -- 172.21.15.44:0/1232604056 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d8017a50 con 0x5587836719a0 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.998+0000 7f65e97a7740 1 -- 172.21.15.44:0/1232604056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587836719a0 msgr2=0x558783644940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.998+0000 7f65e97a7740 1 --2- 172.21.15.44:0/1232604056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587836719a0 0x558783644940 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f65d8003da0 tx=0x7f65d8014000 comp rx=0 tx=0).stop 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.998+0000 7f65e97a7740 1 -- 172.21.15.44:0/1232604056 shutdown_connections 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.998+0000 7f65e97a7740 1 --2- 172.21.15.44:0/1232604056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558783644e80 0x558783674c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.998+0000 7f65e97a7740 1 --2- 172.21.15.44:0/1232604056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558783671d80 0x558783674730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.998+0000 7f65e97a7740 1 --2- 172.21.15.44:0/1232604056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587836719a0 0x558783644940 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.998+0000 7f65e97a7740 1 -- 172.21.15.44:0/1232604056 >> 172.21.15.44:0/1232604056 conn(0x558783646bb0 msgr2=0x55878368f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:34.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.999+0000 7f65e97a7740 1 -- 172.21.15.44:0/1232604056 shutdown_connections 2024-04-02T13:23:34.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.999+0000 7f65e97a7740 1 -- 172.21.15.44:0/1232604056 wait complete. 2024-04-02T13:23:34.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.999+0000 7f65e97a7740 1 -- start start 2024-04-02T13:23:34.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:33.999+0000 7f65e97a7740 1 Processor -- start 2024-04-02T13:23:34.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.000+0000 7f65e97a7740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5587836719a0 0x55878365b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.000+0000 7f65e97a7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 0x55878366d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.000+0000 7f65e97a7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55878366f980 0x55878366deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.000+0000 7f65e97a7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558783649150 con 0x558783671d80 2024-04-02T13:23:34.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.000+0000 7f65e97a7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558783569d30 con 0x55878366f980 2024-04-02T13:23:34.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.000+0000 7f65e97a7740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x558783648540 con 0x5587836719a0 2024-04-02T13:23:34.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.000+0000 7f65e97a7740 10 client.0 _ll_register_callbacks cb 0x558783569720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:23:34.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.000+0000 7f65dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 0x55878366d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.000+0000 7f65dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 0x55878366d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56652/0 (socket says 172.21.15.44:56652) 2024-04-02T13:23:34.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.000+0000 7f65dffff640 1 -- 172.21.15.44:0/2302825922 learned_addr learned my addr 172.21.15.44:0/2302825922 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:34.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.000+0000 7f65e5157640 1 --2- 172.21.15.44:0/2302825922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55878366f980 0x55878366deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.000+0000 7f65e4956640 1 --2- 172.21.15.44:0/2302825922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5587836719a0 0x55878365b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65dffff640 1 -- 172.21.15.44:0/2302825922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5587836719a0 msgr2=0x55878365b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65dffff640 1 --2- 172.21.15.44:0/2302825922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5587836719a0 0x55878365b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65dffff640 1 -- 172.21.15.44:0/2302825922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55878366f980 msgr2=0x55878366deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65dffff640 1 --2- 172.21.15.44:0/2302825922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55878366f980 0x55878366deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65dffff640 1 -- 172.21.15.44:0/2302825922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587832c23f0 con 0x558783671d80 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65dffff640 1 --2- 172.21.15.44:0/2302825922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 0x55878366d970 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f65d4009870 tx=0x7f65d402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1671d924c0fded server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65bf7fe640 1 -- 172.21.15.44:0/2302825922 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d404b060 con 0x558783671d80 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65bf7fe640 1 -- 172.21.15.44:0/2302825922 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f65d4003d50 con 0x558783671d80 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65e97a7740 1 -- 172.21.15.44:0/2302825922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65d8027070 con 0x558783671d80 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65bf7fe640 1 -- 172.21.15.44:0/2302825922 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f65d4051050 con 0x558783671d80 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65e97a7740 10 client.6651 fetch_fsmap learned FSMap version 35 2024-04-02T13:23:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65e97a7740 1 -- 172.21.15.44:0/2302825922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x5587832820f0 con 0x558783671d80 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65bf7fe640 1 -- 172.21.15.44:0/2302825922 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65d8027070 con 0x558783671d80 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65bf7fe640 1 -- 172.21.15.44:0/2302825922 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f65d4054020 con 0x558783671d80 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65e97a7740 10 client.6651 fetch_fsmap finished waiting for FSMap version 35 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65e97a7740 10 client.6651 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65e97a7740 20 client.6651 populate_metadata read hostname 'smithi044' 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65e97a7740 1 --2- 172.21.15.44:0/2302825922 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5587836a2e30 0x5587836a3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65e97a7740 4 client.6651 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.001+0000 7f65e97a7740 1 -- 172.21.15.44:0/2302825922 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587835bbd70 con 0x5587836a2e30 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.002+0000 7f65e4956640 1 --2- 172.21.15.44:0/2302825922 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5587836a2e30 0x5587836a3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.002+0000 7f65e4956640 1 --2- 172.21.15.44:0/2302825922 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5587836a2e30 0x5587836a3230 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8e005ef73cf06bd server_cookie=4df5ba1447415c58 in_seq=0 out_seq=0 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.002+0000 7f65bf7fe640 10 client.6651 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.002+0000 7f65bf7fe640 1 -- 172.21.15.44:0/2302825922 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5587835bbd70 con 0x5587836a2e30 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.002+0000 7f65bf7fe640 10 client.6651 handle_command_reply: tid=0 2024-04-02T13:23:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.002+0000 7f65e97a7740 1 client.6651 shutdown 2024-04-02T13:23:34.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.003+0000 7f65e97a7740 1 -- 172.21.15.44:0/2302825922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 msgr2=0x55878366d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:34.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.003+0000 7f65e97a7740 1 --2- 172.21.15.44:0/2302825922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 0x55878366d970 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f65d4009870 tx=0x7f65d402e000 comp rx=0 tx=0).stop 2024-04-02T13:23:34.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.003+0000 7f65e97a7740 1 -- 172.21.15.44:0/2302825922 shutdown_connections 2024-04-02T13:23:34.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.003+0000 7f65e97a7740 1 --2- 172.21.15.44:0/2302825922 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5587836a2e30 0x5587836a3230 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.003+0000 7f65e97a7740 1 --2- 172.21.15.44:0/2302825922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55878366f980 0x55878366deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.003+0000 7f65e97a7740 1 --2- 172.21.15.44:0/2302825922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 0x55878366d970 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.003+0000 7f65bf7fe640 0 client.6651 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:23:34.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.003+0000 7f65e97a7740 1 --2- 172.21.15.44:0/2302825922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5587836719a0 0x55878365b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.003+0000 7f65e97a7740 1 -- 172.21.15.44:0/2302825922 >> 172.21.15.44:0/2302825922 conn(0x558783644e80 msgr2=0x558783692c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:34.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.003+0000 7f65e97a7740 1 -- 172.21.15.44:0/2302825922 shutdown_connections 2024-04-02T13:23:34.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.003+0000 7f65e97a7740 1 -- 172.21.15.44:0/2302825922 wait complete. 2024-04-02T13:23:34.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.003+0000 7f65e97a7740 20 client.6651 trim_cache size 0 max 16384 2024-04-02T13:23:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.020+0000 7f65e97a7740 1 Processor -- start 2024-04-02T13:23:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.020+0000 7f65e97a7740 1 -- start start 2024-04-02T13:23:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e97a7740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55878366f980 0x55878365b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e97a7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587836719a0 0x55878366db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e97a7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 0x5587836b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e97a7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f65e0200cc0 con 0x558783671d80 2024-04-02T13:23:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e97a7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f65e01ffce0 con 0x5587836719a0 2024-04-02T13:23:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e97a7740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f65e0161370 con 0x55878366f980 2024-04-02T13:23:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e5157640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 0x5587836b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e5157640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 0x5587836b1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56662/0 (socket says 172.21.15.44:56662) 2024-04-02T13:23:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e5157640 1 -- 172.21.15.44:0/2568028028 learned_addr learned my addr 172.21.15.44:0/2568028028 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65dffff640 1 --2- 172.21.15.44:0/2568028028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587836719a0 0x55878366db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e4956640 1 --2- 172.21.15.44:0/2568028028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55878366f980 0x55878365b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e5157640 1 -- 172.21.15.44:0/2568028028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55878366f980 msgr2=0x55878365b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e5157640 1 --2- 172.21.15.44:0/2568028028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55878366f980 0x55878365b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e5157640 1 -- 172.21.15.44:0/2568028028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587836719a0 msgr2=0x55878366db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e5157640 1 --2- 172.21.15.44:0/2568028028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587836719a0 0x55878366db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.021+0000 7f65e5157640 1 -- 172.21.15.44:0/2568028028 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55878364aec0 con 0x558783671d80 2024-04-02T13:23:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.022+0000 7f65e5157640 1 --2- 172.21.15.44:0/2568028028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 0x5587836b1e10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f65d00592f0 tx=0x7f65d008b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f506f287625f4f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.022+0000 7f65c5ffb640 1 -- 172.21.15.44:0/2568028028 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d0077270 con 0x558783671d80 2024-04-02T13:23:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.022+0000 7f65c5ffb640 1 -- 172.21.15.44:0/2568028028 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65d0076070 con 0x558783671d80 2024-04-02T13:23:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.022+0000 7f65c5ffb640 1 -- 172.21.15.44:0/2568028028 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d0061e90 con 0x558783671d80 2024-04-02T13:23:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.022+0000 7f65e97a7740 1 -- 172.21.15.44:0/2568028028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 msgr2=0x5587836b1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.022+0000 7f65e97a7740 1 --2- 172.21.15.44:0/2568028028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 0x5587836b1e10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f65d00592f0 tx=0x7f65d008b040 comp rx=0 tx=0).stop 2024-04-02T13:23:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.022+0000 7f65e97a7740 1 -- 172.21.15.44:0/2568028028 shutdown_connections 2024-04-02T13:23:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.022+0000 7f65e97a7740 1 --2- 172.21.15.44:0/2568028028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558783671d80 0x5587836b1e10 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.022+0000 7f65e97a7740 1 --2- 172.21.15.44:0/2568028028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587836719a0 0x55878366db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.022+0000 7f65e97a7740 1 --2- 172.21.15.44:0/2568028028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55878366f980 0x55878365b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.022+0000 7f65e97a7740 1 -- 172.21.15.44:0/2568028028 >> 172.21.15.44:0/2568028028 conn(0x558783644e80 msgr2=0x5587836400d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.022+0000 7f65e97a7740 1 -- 172.21.15.44:0/2568028028 shutdown_connections 2024-04-02T13:23:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.022+0000 7f65e97a7740 1 -- 172.21.15.44:0/2568028028 wait complete. 2024-04-02T13:23:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.023+0000 7f65e97a7740 1 -- start start 2024-04-02T13:23:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.023+0000 7f65e97a7740 1 Processor -- start 2024-04-02T13:23:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e97a7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55878366f980 0x558783649f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e97a7740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5587836719a0 0x55878368f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e97a7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558783671d80 0x55878368f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e97a7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558783677cd0 con 0x55878366f980 2024-04-02T13:23:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e97a7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f65e01ff690 con 0x558783671d80 2024-04-02T13:23:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e97a7740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x558783676810 con 0x5587836719a0 2024-04-02T13:23:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e97a7740 10 client.0 _ll_register_callbacks cb 0x7f65d4004440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e4956640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55878366f980 0x558783649f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e4956640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55878366f980 0x558783649f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56674/0 (socket says 172.21.15.44:56674) 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e4956640 1 -- 172.21.15.44:0/3751457166 learned_addr learned my addr 172.21.15.44:0/3751457166 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65dffff640 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5587836719a0 0x55878368f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e5157640 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558783671d80 0x55878368f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e4956640 1 -- 172.21.15.44:0/3751457166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5587836719a0 msgr2=0x55878368f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e4956640 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5587836719a0 0x55878368f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e4956640 1 -- 172.21.15.44:0/3751457166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558783671d80 msgr2=0x55878368f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e4956640 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558783671d80 0x55878368f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e4956640 1 -- 172.21.15.44:0/3751457166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55878365ba10 con 0x55878366f980 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65dffff640 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5587836719a0 0x55878368f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65e4956640 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55878366f980 0x558783649f50 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f65d8009fa0 tx=0x7f65d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d2c33af2d6064e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.024+0000 7f65bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:23:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65bf7fe640 1 -- 172.21.15.44:0/3751457166 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d803e060 con 0x55878366f980 2024-04-02T13:23:34.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65bf7fe640 1 -- 172.21.15.44:0/3751457166 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f65d8017a60 con 0x55878366f980 2024-04-02T13:23:34.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65e97a7740 1 -- 172.21.15.44:0/3751457166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65d0076070 con 0x55878366f980 2024-04-02T13:23:34.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65bf7fe640 1 -- 172.21.15.44:0/3751457166 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f65d8044050 con 0x55878366f980 2024-04-02T13:23:34.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65e97a7740 10 client.6657 fetch_fsmap learned FSMap version 35 2024-04-02T13:23:34.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65e97a7740 1 -- 172.21.15.44:0/3751457166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x55878370a340 con 0x55878366f980 2024-04-02T13:23:34.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65bf7fe640 1 -- 172.21.15.44:0/3751457166 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65d0076070 con 0x55878366f980 2024-04-02T13:23:34.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65bf7fe640 1 -- 172.21.15.44:0/3751457166 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f65d8047020 con 0x55878366f980 2024-04-02T13:23:34.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65e97a7740 10 client.6657 fetch_fsmap finished waiting for FSMap version 35 2024-04-02T13:23:34.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65e97a7740 10 client.6657 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-04-02T13:23:34.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65e97a7740 20 client.6657 populate_metadata read hostname 'smithi044' 2024-04-02T13:23:34.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65e97a7740 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5587837033e0 0x5587836b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:23:34.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65e97a7740 4 client.6657 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:23:34.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65e97a7740 1 -- 172.21.15.44:0/3751457166 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5587835b60c0 con 0x5587837033e0 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65dffff640 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5587837033e0 0x5587836b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65bf7fe640 10 client.6657 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.025+0000 7f65dffff640 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5587837033e0 0x5587836b1e10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f4ea7dfcca58f1c server_cookie=2ef3aecbc4e39e6e in_seq=0 out_seq=0 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65bf7fe640 1 -- 172.21.15.44:0/3751457166 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+253 (crc 0 0 0) 0x5587835b60c0 con 0x5587837033e0 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65bf7fe640 10 client.6657 handle_command_reply: tid=0 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e97a7740 1 client.6657 shutdown 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e97a7740 1 -- 172.21.15.44:0/3751457166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55878366f980 msgr2=0x558783649f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e97a7740 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55878366f980 0x558783649f50 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f65d8009fa0 tx=0x7f65d8004a40 comp rx=0 tx=0).stop 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e97a7740 1 -- 172.21.15.44:0/3751457166 shutdown_connections 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e97a7740 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5587837033e0 0x5587836b1e10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e97a7740 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558783671d80 0x55878368f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e97a7740 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5587836719a0 0x55878368f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65bf7fe640 0 client.6657 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:23:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e97a7740 1 --2- 172.21.15.44:0/3751457166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55878366f980 0x558783649f50 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e97a7740 1 -- 172.21.15.44:0/3751457166 >> 172.21.15.44:0/3751457166 conn(0x558783644e80 msgr2=0x558783649840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:34.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e97a7740 1 -- 172.21.15.44:0/3751457166 shutdown_connections 2024-04-02T13:23:34.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e97a7740 1 -- 172.21.15.44:0/3751457166 wait complete. 2024-04-02T13:23:34.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e97a7740 20 client.6657 trim_cache size 0 max 16384 2024-04-02T13:23:34.035 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:23:34.035 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T13:23:34.035 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-02T13:23:34.035 INFO:teuthology.orchestra.run.smithi044.stdout: "c82feae0-27e5-44d9-b782-d5babebbc651": { 2024-04-02T13:23:34.035 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-02T13:23:34.035 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "c82feae0-27e5-44d9-b782-d5babebbc651", 2024-04-02T13:23:34.036 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive" 2024-04-02T13:23:34.036 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:34.036 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:23:34.036 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:23:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e6bba640 1 -- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f65cc06c8f0 msgr2=0x7f65cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e6bba640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f65cc06c8f0 0x7f65cc06edb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f65d8009bd0 tx=0x7f65d800b040 comp rx=0 tx=0).stop 2024-04-02T13:23:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e6bba640 1 -- 172.21.15.44:0/3976537326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e015e5b0 msgr2=0x7f65e00cc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:23:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.026+0000 7f65e6bba640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e015e5b0 0x7f65e00cc200 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f65d000c440 tx=0x7f65d000c900 comp rx=0 tx=0).stop 2024-04-02T13:23:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.027+0000 7f65e6bba640 1 -- 172.21.15.44:0/3976537326 shutdown_connections 2024-04-02T13:23:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.027+0000 7f65e6bba640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f65cc06c8f0 0x7f65cc06edb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.027+0000 7f65e6bba640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e015e5b0 0x7f65e00cc200 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.027+0000 7f65e6bba640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e015e1d0 0x7f65e015ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.027+0000 7f65e6bba640 1 --2- 172.21.15.44:0/3976537326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65e00cec60 0x7f65e0150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:23:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.027+0000 7f65e6bba640 1 -- 172.21.15.44:0/3976537326 >> 172.21.15.44:0/3976537326 conn(0x7f65e009d3b0 msgr2=0x7f65e00b6400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:23:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.027+0000 7f65e6bba640 1 -- 172.21.15.44:0/3976537326 shutdown_connections 2024-04-02T13:23:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:23:34.027+0000 7f65e6bba640 1 -- 172.21.15.44:0/3976537326 wait complete. 2024-04-02T13:23:34.039 INFO:tasks.cephfs.filesystem:scrub status for tag:c82feae0-27e5-44d9-b782-d5babebbc651 - {'path': '/', 'tag': 'c82feae0-27e5-44d9-b782-d5babebbc651', 'options': 'recursive'} 2024-04-02T13:23:37.492 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:23:37.490+0000 7fe12a6e8640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-02T13:23:52.270 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:23:52.273 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:24:04.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:24:04.040 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-04-02T13:24:04.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.152+0000 7ff9edb96640 1 Processor -- start 2024-04-02T13:24:04.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.152+0000 7ff9edb96640 1 -- start start 2024-04-02T13:24:04.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9edb96640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff9e80b3b60 0x7ff9e8143a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9edb96640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9e80d0e10 con 0x7ff9e80d0a30 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9edb96640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9e8144750 con 0x7ff9e8143f60 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9edb96640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff9e8161e10 con 0x7ff9e80b3b60 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e77fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff9e80d0a30 0x7ff9e80b3450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:34688/0 (socket says 172.21.15.44:34688) 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e77fe640 1 -- 172.21.15.44:0/3858046773 learned_addr learned my addr 172.21.15.44:0/3858046773 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e6ffd640 1 --2- 172.21.15.44:0/3858046773 >> v2:172.21.15.69:3300/0 conn(0x7ff9e80b3b60 0x7ff9e8143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3641870458 0 0) 0x7ff9dc001110 con 0x7ff9e80d0a30 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9d0003520 con 0x7ff9e80d0a30 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2752308812 0 0) 0x7ff9e8144750 con 0x7ff9e8143f60 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9dc001110 con 0x7ff9e8143f60 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3215947168 0 0) 0x7ff9d0003520 con 0x7ff9e80d0a30 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 >> v2:172.21.15.69:3300/0 conn(0x7ff9e80b3b60 msgr2=0x7ff9e8143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 --2- 172.21.15.44:0/3858046773 >> v2:172.21.15.69:3300/0 conn(0x7ff9e80b3b60 0x7ff9e8143a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 >> v1:172.21.15.69:6789/0 conn(0x7ff9e8143f60 legacy=0x7ff9e815e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e8162060 con 0x7ff9e80d0a30 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7ff9dc001840 con 0x7ff9e80d0a30 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 >> v1:172.21.15.44:6789/0 conn(0x7ff9e80d0a30 legacy=0x7ff9e80b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 --2- 172.21.15.44:0/3858046773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d0003b40 0x7ff9d0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 --2- 172.21.15.44:0/3858046773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d0004800 0x7ff9d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.153+0000 7ff9e67fc640 1 --2- 172.21.15.44:0/3858046773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9d00051c0 0x7ff9d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.154+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e8161e10 con 0x7ff9d00051c0 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.154+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d000c0f0 con 0x7ff9d0003b40 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.154+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d000c2f0 con 0x7ff9d0004800 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.154+0000 7ff9e7fff640 1 --2- 172.21.15.44:0/3858046773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9d00051c0 0x7ff9d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.154+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/3858046773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d0003b40 0x7ff9d0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.154+0000 7ff9e6ffd640 1 --2- 172.21.15.44:0/3858046773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d0004800 0x7ff9d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.154+0000 7ff9e7fff640 1 -- 172.21.15.44:0/3858046773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d0004800 msgr2=0x7ff9d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.154+0000 7ff9e7fff640 1 --2- 172.21.15.44:0/3858046773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d0004800 0x7ff9d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.154+0000 7ff9e7fff640 1 -- 172.21.15.44:0/3858046773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d0003b40 msgr2=0x7ff9d0003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.154+0000 7ff9e7fff640 1 --2- 172.21.15.44:0/3858046773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d0003b40 0x7ff9d0003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.154+0000 7ff9e7fff640 1 -- 172.21.15.44:0/3858046773 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d0003520 con 0x7ff9d00051c0 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.155+0000 7ff9e7fff640 1 -- 172.21.15.44:0/3858046773 reap_dead start 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.155+0000 7ff9e7fff640 1 --2- 172.21.15.44:0/3858046773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9d00051c0 0x7ff9d000b9c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0001fd0 tx=0x7ff9d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a37b7eab751b7d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.155+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0001bf0 con 0x7ff9d00051c0 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.155+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9e0003b70 con 0x7ff9d00051c0 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.155+0000 7ff9e67fc640 1 -- 172.21.15.44:0/3858046773 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e00050d0 con 0x7ff9d00051c0 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.155+0000 7ff9edb96640 1 -- 172.21.15.44:0/3858046773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9d00051c0 msgr2=0x7ff9d000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.155+0000 7ff9edb96640 1 --2- 172.21.15.44:0/3858046773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9d00051c0 0x7ff9d000b9c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0001fd0 tx=0x7ff9d0005830 comp rx=0 tx=0).stop 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.155+0000 7ff9edb96640 1 -- 172.21.15.44:0/3858046773 shutdown_connections 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.155+0000 7ff9edb96640 1 --2- 172.21.15.44:0/3858046773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9d00051c0 0x7ff9d000b9c0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.155+0000 7ff9edb96640 1 -- 172.21.15.44:0/3858046773 >> 172.21.15.44:0/3858046773 conn(0x7ff9e80b5d90 msgr2=0x7ff9e806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.155+0000 7ff9edb96640 1 -- 172.21.15.44:0/3858046773 shutdown_connections 2024-04-02T13:24:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.155+0000 7ff9edb96640 1 -- 172.21.15.44:0/3858046773 wait complete. 2024-04-02T13:24:04.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.157+0000 7ff9edb96640 1 Processor -- start 2024-04-02T13:24:04.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.157+0000 7ff9edb96640 1 -- start start 2024-04-02T13:24:04.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9edb96640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d00051c0 0x7ff9e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9edb96640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9e8140b30 0x7ff9e8140fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9edb96640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e81f7580 0x7ff9e81f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9edb96640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e81443f0 con 0x7ff9e8140b30 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9edb96640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e8162620 con 0x7ff9e81f7580 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9edb96640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff9e8162b20 con 0x7ff9d00051c0 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9e6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9e8140b30 0x7ff9e8140fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9e6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9e8140b30 0x7ff9e8140fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56288/0 (socket says 172.21.15.44:56288) 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9e6ffd640 1 -- 172.21.15.44:0/3054884151 learned_addr learned my addr 172.21.15.44:0/3054884151 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9e7fff640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e81f7580 0x7ff9e81f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d00051c0 0x7ff9e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9e6ffd640 1 -- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d00051c0 msgr2=0x7ff9e806e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9e6ffd640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d00051c0 0x7ff9e806e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9e6ffd640 1 -- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e81f7580 msgr2=0x7ff9e81f79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9e6ffd640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e81f7580 0x7ff9e81f79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9e6ffd640 1 -- 172.21.15.44:0/3054884151 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e8141c80 con 0x7ff9e8140b30 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d00051c0 0x7ff9e806e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9e6ffd640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9e8140b30 0x7ff9e8140fb0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8011990 tx=0x7ff9d8011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9cffff640 1 -- 172.21.15.44:0/3054884151 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8019070 con 0x7ff9e8140b30 2024-04-02T13:24:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9cffff640 1 -- 172.21.15.44:0/3054884151 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9d8004240 con 0x7ff9e8140b30 2024-04-02T13:24:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9edb96640 1 -- 172.21.15.44:0/3054884151 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9e8141f60 con 0x7ff9e8140b30 2024-04-02T13:24:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.158+0000 7ff9cffff640 1 -- 172.21.15.44:0/3054884151 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d800f860 con 0x7ff9e8140b30 2024-04-02T13:24:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.159+0000 7ff9edb96640 1 -- 172.21.15.44:0/3054884151 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9e81f82d0 con 0x7ff9e8140b30 2024-04-02T13:24:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.160+0000 7ff9cffff640 1 -- 172.21.15.44:0/3054884151 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff9d801c070 con 0x7ff9e8140b30 2024-04-02T13:24:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.161+0000 7ff9cffff640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff9d406c920 0x7ff9d406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.161+0000 7ff9cffff640 1 -- 172.21.15.44:0/3054884151 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7ff9d808c0d0 con 0x7ff9e8140b30 2024-04-02T13:24:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.161+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff9d406c920 0x7ff9d406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.161+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff9d406c920 0x7ff9d406ede0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc001900 tx=0x7ff9dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.161+0000 7ff9f077e740 1 Processor -- start 2024-04-02T13:24:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.162+0000 7ff9f077e740 1 -- start start 2024-04-02T13:24:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.162+0000 7ff9f077e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac7719a0 0x5614ac744940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.162+0000 7ff9f077e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac771d80 0x5614ac774730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.162+0000 7ff9f077e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac744e80 0x5614ac774c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.162+0000 7ff9f077e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5614ac6ae830 con 0x5614ac744e80 2024-04-02T13:24:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.162+0000 7ff9f077e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5614ac656820 con 0x5614ac7719a0 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.162+0000 7ff9f077e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5614ac6aa7e0 con 0x5614ac771d80 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9e7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac744e80 0x5614ac774c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9e7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac744e80 0x5614ac774c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56290/0 (socket says 172.21.15.44:56290) 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9e7fff640 1 -- 172.21.15.44:0/4237910904 learned_addr learned my addr 172.21.15.44:0/4237910904 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/4237910904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac7719a0 0x5614ac744940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9e6ffd640 1 --2- 172.21.15.44:0/4237910904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac771d80 0x5614ac774730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9e7fff640 1 -- 172.21.15.44:0/4237910904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac771d80 msgr2=0x5614ac774730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9e7fff640 1 --2- 172.21.15.44:0/4237910904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac771d80 0x5614ac774730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9e7fff640 1 -- 172.21.15.44:0/4237910904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac7719a0 msgr2=0x5614ac744940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9e7fff640 1 --2- 172.21.15.44:0/4237910904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac7719a0 0x5614ac744940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9e7fff640 1 -- 172.21.15.44:0/4237910904 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614ac48c7d0 con 0x5614ac744e80 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9e7fff640 1 --2- 172.21.15.44:0/4237910904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac744e80 0x5614ac774c70 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff9e002c1e0 tx=0x7ff9e0009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ea3f7f323818c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9cd7fa640 1 -- 172.21.15.44:0/4237910904 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0003b70 con 0x5614ac744e80 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9cd7fa640 1 -- 172.21.15.44:0/4237910904 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9e00050d0 con 0x5614ac744e80 2024-04-02T13:24:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.163+0000 7ff9cd7fa640 1 -- 172.21.15.44:0/4237910904 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e00054a0 con 0x5614ac744e80 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.164+0000 7ff9f077e740 1 -- 172.21.15.44:0/4237910904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac744e80 msgr2=0x5614ac774c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.164+0000 7ff9f077e740 1 --2- 172.21.15.44:0/4237910904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac744e80 0x5614ac774c70 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff9e002c1e0 tx=0x7ff9e0009030 comp rx=0 tx=0).stop 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.164+0000 7ff9f077e740 1 -- 172.21.15.44:0/4237910904 shutdown_connections 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.164+0000 7ff9f077e740 1 --2- 172.21.15.44:0/4237910904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac744e80 0x5614ac774c70 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.164+0000 7ff9f077e740 1 --2- 172.21.15.44:0/4237910904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac771d80 0x5614ac774730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.164+0000 7ff9f077e740 1 --2- 172.21.15.44:0/4237910904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac7719a0 0x5614ac744940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.164+0000 7ff9f077e740 1 -- 172.21.15.44:0/4237910904 >> 172.21.15.44:0/4237910904 conn(0x5614ac746bb0 msgr2=0x5614ac78f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.164+0000 7ff9f077e740 1 -- 172.21.15.44:0/4237910904 shutdown_connections 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.164+0000 7ff9f077e740 1 -- 172.21.15.44:0/4237910904 wait complete. 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.165+0000 7ff9f077e740 1 -- start start 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.165+0000 7ff9f077e740 1 Processor -- start 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.165+0000 7ff9f077e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac7719a0 0x5614ac786ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.165+0000 7ff9f077e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac771d80 0x5614ac79f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.165+0000 7ff9f077e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac770f50 0x5614ac788250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.165+0000 7ff9f077e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5614ac749150 con 0x5614ac7719a0 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.165+0000 7ff9f077e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5614ac669d30 con 0x5614ac771d80 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.165+0000 7ff9f077e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5614ac748540 con 0x5614ac770f50 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.165+0000 7ff9f077e740 10 client.0 _ll_register_callbacks cb 0x5614ac669720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.165+0000 7ff9e77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac7719a0 0x5614ac786ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9e77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac7719a0 0x5614ac786ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56304/0 (socket says 172.21.15.44:56304) 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9e77fe640 1 -- 172.21.15.44:0/2112051602 learned_addr learned my addr 172.21.15.44:0/2112051602 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9e7fff640 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac770f50 0x5614ac788250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9e6ffd640 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac771d80 0x5614ac79f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9e77fe640 1 -- 172.21.15.44:0/2112051602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac770f50 msgr2=0x5614ac788250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac770f50 0x5614ac788250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9e77fe640 1 -- 172.21.15.44:0/2112051602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac771d80 msgr2=0x5614ac79f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac771d80 0x5614ac79f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9e77fe640 1 -- 172.21.15.44:0/2112051602 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614ac3c23f0 con 0x5614ac7719a0 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9e7fff640 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac770f50 0x5614ac788250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac7719a0 0x5614ac786ab0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc015a60 tx=0x7ff9e8162230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02e1b86e56d8077 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9c67fc640 1 -- 172.21.15.44:0/2112051602 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc026070 con 0x5614ac7719a0 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9c67fc640 1 -- 172.21.15.44:0/2112051602 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7ff9dc014c20 con 0x5614ac7719a0 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9f077e740 1 -- 172.21.15.44:0/2112051602 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9e00050d0 con 0x5614ac7719a0 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9c67fc640 1 -- 172.21.15.44:0/2112051602 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff9dc032050 con 0x5614ac7719a0 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9f077e740 10 client.6669 fetch_fsmap learned FSMap version 36 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.166+0000 7ff9f077e740 1 -- 172.21.15.44:0/2112051602 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x5614ac3820f0 con 0x5614ac7719a0 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.167+0000 7ff9c67fc640 1 -- 172.21.15.44:0/2112051602 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9e00050d0 con 0x5614ac7719a0 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.167+0000 7ff9c67fc640 1 -- 172.21.15.44:0/2112051602 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7ff9dc035020 con 0x5614ac7719a0 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.167+0000 7ff9f077e740 10 client.6669 fetch_fsmap finished waiting for FSMap version 36 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.167+0000 7ff9f077e740 10 client.6669 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.167+0000 7ff9f077e740 20 client.6669 populate_metadata read hostname 'smithi044' 2024-04-02T13:24:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.167+0000 7ff9f077e740 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614ac776a70 0x5614ac776ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.167+0000 7ff9f077e740 4 client.6669 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.167+0000 7ff9f077e740 1 -- 172.21.15.44:0/2112051602 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614ac6bbd70 con 0x5614ac776a70 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.167+0000 7ff9e6ffd640 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614ac776a70 0x5614ac776ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.167+0000 7ff9e6ffd640 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614ac776a70 0x5614ac776ef0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ec4ce52fa7a7afb server_cookie=be060aa485341128 in_seq=0 out_seq=0 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.167+0000 7ff9c67fc640 10 client.6669 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9c67fc640 1 -- 172.21.15.44:0/2112051602 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5614ac6bbd70 con 0x5614ac776a70 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9c67fc640 10 client.6669 handle_command_reply: tid=0 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9f077e740 1 client.6669 shutdown 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9f077e740 1 -- 172.21.15.44:0/2112051602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac7719a0 msgr2=0x5614ac786ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9f077e740 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac7719a0 0x5614ac786ab0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc015a60 tx=0x7ff9e8162230 comp rx=0 tx=0).stop 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9f077e740 1 -- 172.21.15.44:0/2112051602 shutdown_connections 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9f077e740 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614ac776a70 0x5614ac776ef0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9f077e740 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac770f50 0x5614ac788250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9f077e740 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac771d80 0x5614ac79f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9f077e740 1 --2- 172.21.15.44:0/2112051602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac7719a0 0x5614ac786ab0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9c67fc640 0 client.6669 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:04.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9f077e740 1 -- 172.21.15.44:0/2112051602 >> 172.21.15.44:0/2112051602 conn(0x5614ac744e80 msgr2=0x5614ac774720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9f077e740 1 -- 172.21.15.44:0/2112051602 shutdown_connections 2024-04-02T13:24:04.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9f077e740 1 -- 172.21.15.44:0/2112051602 wait complete. 2024-04-02T13:24:04.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.168+0000 7ff9f077e740 20 client.6669 trim_cache size 0 max 16384 2024-04-02T13:24:04.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.186+0000 7ff9f077e740 1 Processor -- start 2024-04-02T13:24:04.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.186+0000 7ff9f077e740 1 -- start start 2024-04-02T13:24:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.186+0000 7ff9f077e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac770f50 0x5614ac770490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9f077e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac7719a0 0x5614ac777880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9f077e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac771d80 0x5614ac777dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9f077e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e8201400 con 0x5614ac770f50 2024-04-02T13:24:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9f077e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e8200460 con 0x5614ac771d80 2024-04-02T13:24:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9f077e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff9e81ff7f0 con 0x5614ac7719a0 2024-04-02T13:24:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9e77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac770f50 0x5614ac770490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9e77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac770f50 0x5614ac770490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56306/0 (socket says 172.21.15.44:56306) 2024-04-02T13:24:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9e77fe640 1 -- 172.21.15.44:0/3528898846 learned_addr learned my addr 172.21.15.44:0/3528898846 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9e77fe640 1 -- 172.21.15.44:0/3528898846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac7719a0 msgr2=0x5614ac777880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/3528898846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac7719a0 0x5614ac777880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9e77fe640 1 -- 172.21.15.44:0/3528898846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac771d80 msgr2=0x5614ac777dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/3528898846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac771d80 0x5614ac777dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9e77fe640 1 -- 172.21.15.44:0/3528898846 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614ac79fa00 con 0x5614ac770f50 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/3528898846 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac770f50 0x5614ac770490 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc007510 tx=0x7ff9dc023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51235f660e0c1f4f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9ccff9640 1 -- 172.21.15.44:0/3528898846 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc03b070 con 0x5614ac770f50 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9ccff9640 1 -- 172.21.15.44:0/3528898846 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9dc0226e0 con 0x5614ac770f50 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.187+0000 7ff9ccff9640 1 -- 172.21.15.44:0/3528898846 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc026040 con 0x5614ac770f50 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.188+0000 7ff9f077e740 1 -- 172.21.15.44:0/3528898846 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac770f50 msgr2=0x5614ac770490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.188+0000 7ff9f077e740 1 --2- 172.21.15.44:0/3528898846 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac770f50 0x5614ac770490 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc007510 tx=0x7ff9dc023000 comp rx=0 tx=0).stop 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.188+0000 7ff9f077e740 1 -- 172.21.15.44:0/3528898846 shutdown_connections 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.188+0000 7ff9f077e740 1 --2- 172.21.15.44:0/3528898846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac771d80 0x5614ac777dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.188+0000 7ff9f077e740 1 --2- 172.21.15.44:0/3528898846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac7719a0 0x5614ac777880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.188+0000 7ff9f077e740 1 --2- 172.21.15.44:0/3528898846 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac770f50 0x5614ac770490 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.188+0000 7ff9f077e740 1 -- 172.21.15.44:0/3528898846 >> 172.21.15.44:0/3528898846 conn(0x5614ac744e80 msgr2=0x5614ac7868c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.188+0000 7ff9f077e740 1 -- 172.21.15.44:0/3528898846 shutdown_connections 2024-04-02T13:24:04.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.188+0000 7ff9f077e740 1 -- 172.21.15.44:0/3528898846 wait complete. 2024-04-02T13:24:04.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.189+0000 7ff9f077e740 1 -- start start 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.189+0000 7ff9f077e740 1 Processor -- start 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.189+0000 7ff9f077e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac770f50 0x5614ac7439b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.189+0000 7ff9f077e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac7719a0 0x5614ac743ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.189+0000 7ff9f077e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac771d80 0x5614ac749130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.189+0000 7ff9f077e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5614ac793840 con 0x5614ac771d80 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.189+0000 7ff9f077e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e81ffe40 con 0x5614ac770f50 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.189+0000 7ff9f077e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5614ac776ee0 con 0x5614ac7719a0 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.189+0000 7ff9f077e740 10 client.0 _ll_register_callbacks cb 0x7ff9dc0077a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.189+0000 7ff9e7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac771d80 0x5614ac749130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.189+0000 7ff9e7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac771d80 0x5614ac749130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56320/0 (socket says 172.21.15.44:56320) 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9e7fff640 1 -- 172.21.15.44:0/2558578830 learned_addr learned my addr 172.21.15.44:0/2558578830 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/2558578830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac770f50 0x5614ac7439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9e6ffd640 1 --2- 172.21.15.44:0/2558578830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac7719a0 0x5614ac743ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9e7fff640 1 -- 172.21.15.44:0/2558578830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac7719a0 msgr2=0x5614ac743ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9e7fff640 1 --2- 172.21.15.44:0/2558578830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac7719a0 0x5614ac743ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9e7fff640 1 -- 172.21.15.44:0/2558578830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac770f50 msgr2=0x5614ac7439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9e7fff640 1 --2- 172.21.15.44:0/2558578830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac770f50 0x5614ac7439b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9e7fff640 1 -- 172.21.15.44:0/2558578830 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614ac74bbb0 con 0x5614ac771d80 2024-04-02T13:24:04.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9e7fff640 1 --2- 172.21.15.44:0/2558578830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac771d80 0x5614ac749130 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0035000 tx=0x7ff9e0001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f332b7c340cafbd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:24:04.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9c67fc640 1 -- 172.21.15.44:0/2558578830 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0005510 con 0x5614ac771d80 2024-04-02T13:24:04.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9c67fc640 1 -- 172.21.15.44:0/2558578830 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7ff9e00056b0 con 0x5614ac771d80 2024-04-02T13:24:04.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9f077e740 1 -- 172.21.15.44:0/2558578830 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9dc0226e0 con 0x5614ac771d80 2024-04-02T13:24:04.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9c67fc640 1 -- 172.21.15.44:0/2558578830 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff9e0052050 con 0x5614ac771d80 2024-04-02T13:24:04.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9f077e740 10 client.6675 fetch_fsmap learned FSMap version 36 2024-04-02T13:24:04.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9f077e740 1 -- 172.21.15.44:0/2558578830 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x5614ac770bf0 con 0x5614ac771d80 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.190+0000 7ff9c67fc640 1 -- 172.21.15.44:0/2558578830 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9dc0226e0 con 0x5614ac771d80 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9c67fc640 1 -- 172.21.15.44:0/2558578830 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7ff9e0055020 con 0x5614ac771d80 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 10 client.6675 fetch_fsmap finished waiting for FSMap version 36 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 10 client.6675 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 20 client.6675 populate_metadata read hostname 'smithi044' 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 1 --2- 172.21.15.44:0/2558578830 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614ac749670 0x5614ac749a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 4 client.6675 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 1 -- 172.21.15.44:0/2558578830 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5614ac6b60c0 con 0x5614ac749670 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/2558578830 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614ac749670 0x5614ac749a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9c67fc640 10 client.6675 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9e77fe640 1 --2- 172.21.15.44:0/2558578830 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614ac749670 0x5614ac749a80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbbcb4d25517d0b4 server_cookie=50f411044b81e4 in_seq=0 out_seq=0 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9c67fc640 1 -- 172.21.15.44:0/2558578830 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5614ac6b60c0 con 0x5614ac749670 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9c67fc640 10 client.6675 handle_command_reply: tid=0 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 1 client.6675 shutdown 2024-04-02T13:24:04.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 1 -- 172.21.15.44:0/2558578830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac771d80 msgr2=0x5614ac749130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 1 --2- 172.21.15.44:0/2558578830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac771d80 0x5614ac749130 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0035000 tx=0x7ff9e0001650 comp rx=0 tx=0).stop 2024-04-02T13:24:04.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 1 -- 172.21.15.44:0/2558578830 shutdown_connections 2024-04-02T13:24:04.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 1 --2- 172.21.15.44:0/2558578830 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5614ac749670 0x5614ac749a80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 1 --2- 172.21.15.44:0/2558578830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5614ac771d80 0x5614ac749130 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 1 --2- 172.21.15.44:0/2558578830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5614ac7719a0 0x5614ac743ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9c67fc640 0 client.6675 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:04.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 1 --2- 172.21.15.44:0/2558578830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5614ac770f50 0x5614ac7439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.191+0000 7ff9f077e740 1 -- 172.21.15.44:0/2558578830 >> 172.21.15.44:0/2558578830 conn(0x5614ac744e80 msgr2=0x5614ac7a8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9f077e740 1 -- 172.21.15.44:0/2558578830 shutdown_connections 2024-04-02T13:24:04.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9f077e740 1 -- 172.21.15.44:0/2558578830 wait complete. 2024-04-02T13:24:04.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9f077e740 20 client.6675 trim_cache size 0 max 16384 2024-04-02T13:24:04.200 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:24:04.201 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-02T13:24:04.201 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-02T13:24:04.201 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:24:04.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9edb96640 1 -- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff9d406c920 msgr2=0x7ff9d406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9edb96640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff9d406c920 0x7ff9d406ede0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc001900 tx=0x7ff9dc009040 comp rx=0 tx=0).stop 2024-04-02T13:24:04.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9edb96640 1 -- 172.21.15.44:0/3054884151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9e8140b30 msgr2=0x7ff9e8140fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9edb96640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9e8140b30 0x7ff9e8140fb0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8011990 tx=0x7ff9d8011e50 comp rx=0 tx=0).stop 2024-04-02T13:24:04.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9edb96640 1 -- 172.21.15.44:0/3054884151 shutdown_connections 2024-04-02T13:24:04.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9edb96640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff9d406c920 0x7ff9d406ede0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9edb96640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e81f7580 0x7ff9e81f79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9edb96640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9e8140b30 0x7ff9e8140fb0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9edb96640 1 --2- 172.21.15.44:0/3054884151 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d00051c0 0x7ff9e806e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9edb96640 1 -- 172.21.15.44:0/3054884151 >> 172.21.15.44:0/3054884151 conn(0x7ff9e80b5d90 msgr2=0x7ff9e806f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9edb96640 1 -- 172.21.15.44:0/3054884151 shutdown_connections 2024-04-02T13:24:04.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.192+0000 7ff9edb96640 1 -- 172.21.15.44:0/3054884151 wait complete. 2024-04-02T13:24:04.205 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T13:24:04.206 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:24:04.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.304+0000 7f112e1e2640 1 Processor -- start 2024-04-02T13:24:04.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.304+0000 7f112e1e2640 1 -- start start 2024-04-02T13:24:04.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.304+0000 7f112e1e2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f112806d700 0x7f112815e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.304+0000 7f112e1e2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f112815ee70 0x7f11280d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.304+0000 7f112e1e2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f112815fb50 con 0x7f112815e550 2024-04-02T13:24:04.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.304+0000 7f112e1e2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f112806ca90 con 0x7f112815ee70 2024-04-02T13:24:04.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.304+0000 7f112e1e2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f112815f590 con 0x7f112806d700 2024-04-02T13:24:04.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.304+0000 7f1127fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f112815ee70 0x7f11280d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.305+0000 7f11277fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f112815e550 0x7f112806cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:34704/0 (socket says 172.21.15.44:34704) 2024-04-02T13:24:04.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.305+0000 7f11277fe640 1 -- 172.21.15.44:0/3718204975 learned_addr learned my addr 172.21.15.44:0/3718204975 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:04.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.305+0000 7f1126ffd640 1 --2- 172.21.15.44:0/3718204975 >> v2:172.21.15.69:3300/0 conn(0x7f112806d700 0x7f112815e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.305+0000 7f1127fff640 1 -- 172.21.15.44:0/3718204975 >> v2:172.21.15.69:3300/0 conn(0x7f112806d700 msgr2=0x7f112815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.305+0000 7f1127fff640 1 --2- 172.21.15.44:0/3718204975 >> v2:172.21.15.69:3300/0 conn(0x7f112806d700 0x7f112815e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.305+0000 7f1127fff640 1 -- 172.21.15.44:0/3718204975 >> v1:172.21.15.44:6789/0 conn(0x7f112815e550 legacy=0x7f112806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.305+0000 7f1127fff640 1 -- 172.21.15.44:0/3718204975 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112815f320 con 0x7f112815ee70 2024-04-02T13:24:04.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.305+0000 7f11267fc640 1 -- 172.21.15.44:0/3718204975 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 679389464 0 0) 0x7f112815fb50 con 0x7f112815e550 2024-04-02T13:24:04.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.306+0000 7f1127fff640 1 --2- 172.21.15.44:0/3718204975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f112815ee70 0x7f11280d0ab0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f11280b4a20 tx=0x7f112002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d22a8e288ac034 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.306+0000 7f11267fc640 1 -- 172.21.15.44:0/3718204975 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1120039070 con 0x7f112815ee70 2024-04-02T13:24:04.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.306+0000 7f11267fc640 1 -- 172.21.15.44:0/3718204975 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f112002cd00 con 0x7f112815ee70 2024-04-02T13:24:04.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.306+0000 7f11267fc640 1 -- 172.21.15.44:0/3718204975 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1120034660 con 0x7f112815ee70 2024-04-02T13:24:04.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.306+0000 7f112e1e2640 1 -- 172.21.15.44:0/3718204975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f112815ee70 msgr2=0x7f11280d0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.306+0000 7f112e1e2640 1 --2- 172.21.15.44:0/3718204975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f112815ee70 0x7f11280d0ab0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f11280b4a20 tx=0x7f112002bf50 comp rx=0 tx=0).stop 2024-04-02T13:24:04.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.306+0000 7f112e1e2640 1 -- 172.21.15.44:0/3718204975 shutdown_connections 2024-04-02T13:24:04.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.306+0000 7f112e1e2640 1 --2- 172.21.15.44:0/3718204975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f112815ee70 0x7f11280d0ab0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.306+0000 7f112e1e2640 1 --2- 172.21.15.44:0/3718204975 >> v2:172.21.15.69:3300/0 conn(0x7f112806d700 0x7f112815e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.306+0000 7f112e1e2640 1 -- 172.21.15.44:0/3718204975 >> 172.21.15.44:0/3718204975 conn(0x7f112809aa20 msgr2=0x7f112809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.306+0000 7f112e1e2640 1 -- 172.21.15.44:0/3718204975 shutdown_connections 2024-04-02T13:24:04.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.306+0000 7f112e1e2640 1 -- 172.21.15.44:0/3718204975 wait complete. 2024-04-02T13:24:04.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.307+0000 7f112e1e2640 1 Processor -- start 2024-04-02T13:24:04.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.308+0000 7f112e1e2640 1 -- start start 2024-04-02T13:24:04.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.308+0000 7f112e1e2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f112806d700 0x7f1128151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.308+0000 7f112e1e2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112815e550 0x7f11280b6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.308+0000 7f112e1e2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f112815ee70 0x7f11280cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.308+0000 7f112e1e2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f11280d0590 con 0x7f112815ee70 2024-04-02T13:24:04.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.308+0000 7f112e1e2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f11280d02b0 con 0x7f112815e550 2024-04-02T13:24:04.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.308+0000 7f112e1e2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f11280cc4b0 con 0x7f112806d700 2024-04-02T13:24:04.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.308+0000 7f1127fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f112815ee70 0x7f11280cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.308+0000 7f1127fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f112815ee70 0x7f11280cb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56342/0 (socket says 172.21.15.44:56342) 2024-04-02T13:24:04.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.308+0000 7f1127fff640 1 -- 172.21.15.44:0/165802683 learned_addr learned my addr 172.21.15.44:0/165802683 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:04.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.309+0000 7f11277fe640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f112806d700 0x7f1128151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.309+0000 7f1126ffd640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112815e550 0x7f11280b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.309+0000 7f1127fff640 1 -- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f112806d700 msgr2=0x7f1128151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.309+0000 7f1127fff640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f112806d700 0x7f1128151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.309+0000 7f1127fff640 1 -- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112815e550 msgr2=0x7f11280b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.309+0000 7f1127fff640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112815e550 0x7f11280b6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.309+0000 7f1127fff640 1 -- 172.21.15.44:0/165802683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11281f74a0 con 0x7f112815ee70 2024-04-02T13:24:04.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.309+0000 7f1127fff640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f112815ee70 0x7f11280cb070 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1120037040 tx=0x7f1120004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.309+0000 7f110ffff640 1 -- 172.21.15.44:0/165802683 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1120044020 con 0x7f112815ee70 2024-04-02T13:24:04.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.309+0000 7f110ffff640 1 -- 172.21.15.44:0/165802683 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f112003a040 con 0x7f112815ee70 2024-04-02T13:24:04.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.309+0000 7f110ffff640 1 -- 172.21.15.44:0/165802683 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1120039050 con 0x7f112815ee70 2024-04-02T13:24:04.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.309+0000 7f112e1e2640 1 -- 172.21.15.44:0/165802683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11281f7780 con 0x7f112815ee70 2024-04-02T13:24:04.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.309+0000 7f112e1e2640 1 -- 172.21.15.44:0/165802683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11281f7d40 con 0x7f112815ee70 2024-04-02T13:24:04.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.310+0000 7f112e1e2640 1 -- 172.21.15.44:0/165802683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11280684f0 con 0x7f112815ee70 2024-04-02T13:24:04.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.311+0000 7f110ffff640 1 -- 172.21.15.44:0/165802683 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1120003b90 con 0x7f112815ee70 2024-04-02T13:24:04.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.311+0000 7f110ffff640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f111406c8f0 0x7f111406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.311+0000 7f110ffff640 1 -- 172.21.15.44:0/165802683 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f11200ae4b0 con 0x7f112815ee70 2024-04-02T13:24:04.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.311+0000 7f11277fe640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f111406c8f0 0x7f111406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.312+0000 7f11277fe640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f111406c8f0 0x7f111406edb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1118007fd0 tx=0x7f1118007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.316+0000 7f110ffff640 1 -- 172.21.15.44:0/165802683 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f112004e020 con 0x7f112815ee70 2024-04-02T13:24:04.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.539+0000 7f112e1e2640 1 -- 172.21.15.44:0/165802683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f112809fc70 con 0x7f112815ee70 2024-04-02T13:24:04.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.539+0000 7f110ffff640 1 -- 172.21.15.44:0/165802683 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5179 (secure 0 0 0) 0x7f112007c760 con 0x7f112815ee70 2024-04-02T13:24:04.542 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:04.543 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1535287484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1535287484},{"type":"v1","addr":"172.21.15.69:6836","nonce":1535287484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6429,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2367107002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2367107002},{"type":"v1","addr":"172.21.15.44:6833","nonce":2367107002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/553542317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":553542317},{"type":"v1","addr":"172.21.15.69:6837","nonce":553542317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:23:24.803884+0000","modified":"2024-04-02T13:23:37.482039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6450":{"gid":6450,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":6,"addr":"172.21.15.44:6835/3127613802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3127613802},{"type":"v1","addr":"172.21.15.44:6835","nonce":3127613802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6450,"qdb_cluster":[6450]},"id":4}]} 2024-04-02T13:24:04.543 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 36 2024-04-02T13:24:04.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.541+0000 7f112e1e2640 1 -- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f111406c8f0 msgr2=0x7f111406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.541+0000 7f112e1e2640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f111406c8f0 0x7f111406edb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1118007fd0 tx=0x7f1118007de0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.541+0000 7f112e1e2640 1 -- 172.21.15.44:0/165802683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f112815ee70 msgr2=0x7f11280cb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.541+0000 7f112e1e2640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f112815ee70 0x7f11280cb070 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1120037040 tx=0x7f1120004a40 comp rx=0 tx=0).stop 2024-04-02T13:24:04.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.541+0000 7f112e1e2640 1 -- 172.21.15.44:0/165802683 shutdown_connections 2024-04-02T13:24:04.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.541+0000 7f112e1e2640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f111406c8f0 0x7f111406edb0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.541+0000 7f112e1e2640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f112815ee70 0x7f11280cb070 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.541+0000 7f112e1e2640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112815e550 0x7f11280b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.541+0000 7f112e1e2640 1 --2- 172.21.15.44:0/165802683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f112806d700 0x7f1128151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.541+0000 7f112e1e2640 1 -- 172.21.15.44:0/165802683 >> 172.21.15.44:0/165802683 conn(0x7f112809aa20 msgr2=0x7f112806cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.541+0000 7f112e1e2640 1 -- 172.21.15.44:0/165802683 shutdown_connections 2024-04-02T13:24:04.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.541+0000 7f112e1e2640 1 -- 172.21.15.44:0/165802683 wait complete. 2024-04-02T13:24:04.555 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-04-02T13:24:04.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.655+0000 7f172259d640 1 Processor -- start 2024-04-02T13:24:04.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.655+0000 7f172259d640 1 -- start start 2024-04-02T13:24:04.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.656+0000 7f172259d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f171c0d09a0 0x7f171c0d0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.656+0000 7f172259d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f171c0d4850 0x7f171c161f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.656+0000 7f172259d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f171c0d4c90 0x7f171c1624b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.656+0000 7f172259d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f171c0cfcb0 con 0x7f171c0d4c90 2024-04-02T13:24:04.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.656+0000 7f172259d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f171c0cee00 con 0x7f171c0d09a0 2024-04-02T13:24:04.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.656+0000 7f172259d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f171c0d12c0 con 0x7f171c0d4850 2024-04-02T13:24:04.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.656+0000 7f1720b3a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f171c0d4c90 0x7f171c1624b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.656+0000 7f1720b3a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f171c0d4c90 0x7f171c1624b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39238/0 (socket says 172.21.15.44:39238) 2024-04-02T13:24:04.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.656+0000 7f1720b3a640 1 -- 172.21.15.44:0/3262981063 learned_addr learned my addr 172.21.15.44:0/3262981063 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:04.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.657+0000 7f171b7fe640 1 --2- 172.21.15.44:0/3262981063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f171c0d4850 0x7f171c161f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.657+0000 7f171bfff640 1 --2- 172.21.15.44:0/3262981063 >> v2:172.21.15.69:3300/0 conn(0x7f171c0d09a0 0x7f171c0d0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.657+0000 7f1720b3a640 1 -- 172.21.15.44:0/3262981063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f171c0d4850 msgr2=0x7f171c161f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.657+0000 7f1720b3a640 1 --2- 172.21.15.44:0/3262981063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f171c0d4850 0x7f171c161f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.657+0000 7f1720b3a640 1 -- 172.21.15.44:0/3262981063 >> v2:172.21.15.69:3300/0 conn(0x7f171c0d09a0 msgr2=0x7f171c0d0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.657+0000 7f1720b3a640 1 --2- 172.21.15.44:0/3262981063 >> v2:172.21.15.69:3300/0 conn(0x7f171c0d09a0 0x7f171c0d0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.657+0000 7f1720b3a640 1 -- 172.21.15.44:0/3262981063 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171c0cf0a0 con 0x7f171c0d4c90 2024-04-02T13:24:04.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.657+0000 7f1720b3a640 1 --2- 172.21.15.44:0/3262981063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f171c0d4c90 0x7f171c1624b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f171c09d9e0 tx=0x7f171402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2765f2b58eabb4ad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.658+0000 7f171affd640 1 -- 172.21.15.44:0/3262981063 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171402ca70 con 0x7f171c0d4c90 2024-04-02T13:24:04.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.658+0000 7f171affd640 1 -- 172.21.15.44:0/3262981063 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f171402cc10 con 0x7f171c0d4c90 2024-04-02T13:24:04.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.658+0000 7f171affd640 1 -- 172.21.15.44:0/3262981063 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17140343f0 con 0x7f171c0d4c90 2024-04-02T13:24:04.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.658+0000 7f172259d640 1 -- 172.21.15.44:0/3262981063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f171c0d4c90 msgr2=0x7f171c1624b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.658+0000 7f172259d640 1 --2- 172.21.15.44:0/3262981063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f171c0d4c90 0x7f171c1624b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f171c09d9e0 tx=0x7f171402bed0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.658+0000 7f172259d640 1 -- 172.21.15.44:0/3262981063 shutdown_connections 2024-04-02T13:24:04.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.658+0000 7f172259d640 1 --2- 172.21.15.44:0/3262981063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f171c0d4c90 0x7f171c1624b0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.658+0000 7f172259d640 1 --2- 172.21.15.44:0/3262981063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f171c0d4850 0x7f171c161f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.658+0000 7f172259d640 1 --2- 172.21.15.44:0/3262981063 >> v2:172.21.15.69:3300/0 conn(0x7f171c0d09a0 0x7f171c0d0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.658+0000 7f172259d640 1 -- 172.21.15.44:0/3262981063 >> 172.21.15.44:0/3262981063 conn(0x7f171c09d470 msgr2=0x7f171c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.658+0000 7f172259d640 1 -- 172.21.15.44:0/3262981063 shutdown_connections 2024-04-02T13:24:04.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.658+0000 7f172259d640 1 -- 172.21.15.44:0/3262981063 wait complete. 2024-04-02T13:24:04.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.659+0000 7f172259d640 1 Processor -- start 2024-04-02T13:24:04.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.660+0000 7f172259d640 1 -- start start 2024-04-02T13:24:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.660+0000 7f172259d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f171c0d09a0 0x7f171c0b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.660+0000 7f172259d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f171c0d4850 0x7f171c161070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.660+0000 7f172259d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f171c0d4c90 0x7f171c1615b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.660+0000 7f172259d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f171c0cf940 con 0x7f171c0d09a0 2024-04-02T13:24:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.660+0000 7f172259d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f171c0cf660 con 0x7f171c0d4c90 2024-04-02T13:24:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.660+0000 7f172259d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f171c0d0140 con 0x7f171c0d4850 2024-04-02T13:24:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.660+0000 7f171bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f171c0d09a0 0x7f171c0b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.660+0000 7f171bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f171c0d09a0 0x7f171c0b3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39248/0 (socket says 172.21.15.44:39248) 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.660+0000 7f171bfff640 1 -- 172.21.15.44:0/1141740496 learned_addr learned my addr 172.21.15.44:0/1141740496 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.661+0000 7f171b7fe640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f171c0d4850 0x7f171c161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.661+0000 7f171bfff640 1 -- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f171c0d4850 msgr2=0x7f171c161070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.661+0000 7f171bfff640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f171c0d4850 0x7f171c161070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.661+0000 7f171bfff640 1 -- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f171c0d4c90 msgr2=0x7f171c1615b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.661+0000 7f1720b3a640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f171c0d4c90 0x7f171c1615b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.661+0000 7f171bfff640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f171c0d4c90 0x7f171c1615b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.661+0000 7f171bfff640 1 -- 172.21.15.44:0/1141740496 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171c0b4410 con 0x7f171c0d09a0 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.661+0000 7f171bfff640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f171c0d09a0 0x7f171c0b3aa0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f170c00d800 tx=0x7f170c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.661+0000 7f1718ff9640 1 -- 172.21.15.44:0/1141740496 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170c018030 con 0x7f171c0d09a0 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.661+0000 7f1718ff9640 1 -- 172.21.15.44:0/1141740496 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f170c0096a0 con 0x7f171c0d09a0 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.661+0000 7f172259d640 1 -- 172.21.15.44:0/1141740496 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f171c0b46f0 con 0x7f171c0d09a0 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.661+0000 7f1718ff9640 1 -- 172.21.15.44:0/1141740496 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170c009870 con 0x7f171c0d09a0 2024-04-02T13:24:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.661+0000 7f172259d640 1 -- 172.21.15.44:0/1141740496 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f171c0b4cb0 con 0x7f171c0d09a0 2024-04-02T13:24:04.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.663+0000 7f1718ff9640 1 -- 172.21.15.44:0/1141740496 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f170c01b070 con 0x7f171c0d09a0 2024-04-02T13:24:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.663+0000 7f1718ff9640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f16fc06c8f0 0x7f16fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.663+0000 7f1718ff9640 1 -- 172.21.15.44:0/1141740496 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f170c08bf70 con 0x7f171c0d09a0 2024-04-02T13:24:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.663+0000 7f171b7fe640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f16fc06c8f0 0x7f16fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.664+0000 7f171b7fe640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f16fc06c8f0 0x7f16fc06edb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1704006fd0 tx=0x7f1704008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.664+0000 7f1725183740 1 Processor -- start 2024-04-02T13:24:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.664+0000 7f1725183740 1 -- start start 2024-04-02T13:24:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1725183740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fa8dc0 0x5645d1fa91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1725183740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d1face10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1725183740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d1ff6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1725183740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5645d1f705d0 con 0x5645d1ff64d0 2024-04-02T13:24:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1725183740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5645d1f126b0 con 0x5645d1fa8dc0 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1725183740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5645d1eb0890 con 0x5645d1faebe0 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1720b3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d1ff6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1720b3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d1ff6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39262/0 (socket says 172.21.15.44:39262) 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1720b3a640 1 -- 172.21.15.44:0/2076506143 learned_addr learned my addr 172.21.15.44:0/2076506143 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f171bfff640 1 --2- 172.21.15.44:0/2076506143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fa8dc0 0x5645d1fa91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f171b7fe640 1 --2- 172.21.15.44:0/2076506143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d1face10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1720b3a640 1 -- 172.21.15.44:0/2076506143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 msgr2=0x5645d1face10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1720b3a640 1 --2- 172.21.15.44:0/2076506143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d1face10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1720b3a640 1 -- 172.21.15.44:0/2076506143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fa8dc0 msgr2=0x5645d1fa91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1720b3a640 1 --2- 172.21.15.44:0/2076506143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fa8dc0 0x5645d1fa91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1720b3a640 1 -- 172.21.15.44:0/2076506143 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645d1edf1d0 con 0x5645d1ff64d0 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f171b7fe640 1 --2- 172.21.15.44:0/2076506143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d1face10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1720b3a640 1 --2- 172.21.15.44:0/2076506143 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d1ff6950 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f1714009f40 tx=0x7f171402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873c70c60530417f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1711ffb640 1 -- 172.21.15.44:0/2076506143 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1714045070 con 0x5645d1ff64d0 2024-04-02T13:24:04.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1711ffb640 1 -- 172.21.15.44:0/2076506143 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f171402ca70 con 0x5645d1ff64d0 2024-04-02T13:24:04.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.665+0000 7f1711ffb640 1 -- 172.21.15.44:0/2076506143 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1714009510 con 0x5645d1ff64d0 2024-04-02T13:24:04.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.666+0000 7f1725183740 1 -- 172.21.15.44:0/2076506143 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 msgr2=0x5645d1ff6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.666+0000 7f1725183740 1 --2- 172.21.15.44:0/2076506143 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d1ff6950 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f1714009f40 tx=0x7f171402e000 comp rx=0 tx=0).stop 2024-04-02T13:24:04.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.666+0000 7f1725183740 1 -- 172.21.15.44:0/2076506143 shutdown_connections 2024-04-02T13:24:04.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.666+0000 7f1725183740 1 --2- 172.21.15.44:0/2076506143 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d1ff6950 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.666+0000 7f1725183740 1 --2- 172.21.15.44:0/2076506143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d1face10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.666+0000 7f1725183740 1 --2- 172.21.15.44:0/2076506143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fa8dc0 0x5645d1fa91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.666+0000 7f1725183740 1 -- 172.21.15.44:0/2076506143 >> 172.21.15.44:0/2076506143 conn(0x5645d1faae40 msgr2=0x5645d1fac700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.666+0000 7f1725183740 1 -- 172.21.15.44:0/2076506143 shutdown_connections 2024-04-02T13:24:04.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.666+0000 7f1725183740 1 -- 172.21.15.44:0/2076506143 wait complete. 2024-04-02T13:24:04.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.667+0000 7f1725183740 1 -- start start 2024-04-02T13:24:04.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.667+0000 7f1725183740 1 Processor -- start 2024-04-02T13:24:04.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f1725183740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d1febe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f1725183740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d2007e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f1725183740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 0x5645d20083c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f1725183740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5645d1eae200 con 0x5645d1ff64d0 2024-04-02T13:24:04.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f1725183740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5645d1f0e660 con 0x5645d1fed7f0 2024-04-02T13:24:04.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f1725183740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5645d1fd5530 con 0x5645d1faebe0 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f1725183740 10 client.0 _ll_register_callbacks cb 0x5645d1ed1040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f171b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d2007e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f171b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d2007e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39268/0 (socket says 172.21.15.44:39268) 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f171b7fe640 1 -- 172.21.15.44:0/1525293419 learned_addr learned my addr 172.21.15.44:0/1525293419 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f1720b3a640 1 --2- 172.21.15.44:0/1525293419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 0x5645d20083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f171bfff640 1 --2- 172.21.15.44:0/1525293419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d1febe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f171b7fe640 1 -- 172.21.15.44:0/1525293419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 msgr2=0x5645d1febe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f171b7fe640 1 --2- 172.21.15.44:0/1525293419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d1febe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f171b7fe640 1 -- 172.21.15.44:0/1525293419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 msgr2=0x5645d20083c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f171b7fe640 1 --2- 172.21.15.44:0/1525293419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 0x5645d20083c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f171b7fe640 1 -- 172.21.15.44:0/1525293419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645d1cf27c0 con 0x5645d1ff64d0 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f171b7fe640 1 --2- 172.21.15.44:0/1525293419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d2007e80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f170400ed60 tx=0x7f1704004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e31eab145bbe01 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f16faffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f16faffd640 1 -- 172.21.15.44:0/1525293419 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170400f040 con 0x5645d1ff64d0 2024-04-02T13:24:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f16faffd640 1 -- 172.21.15.44:0/1525293419 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f1704015dd0 con 0x5645d1ff64d0 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.668+0000 7f1725183740 1 -- 172.21.15.44:0/1525293419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f171402ca70 con 0x5645d1ff64d0 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f16faffd640 1 -- 172.21.15.44:0/1525293419 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f170402d050 con 0x5645d1ff64d0 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f1725183740 10 client.6696 fetch_fsmap learned FSMap version 36 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f1725183740 1 -- 172.21.15.44:0/1525293419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x5645d1d52f50 con 0x5645d1ff64d0 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f16faffd640 1 -- 172.21.15.44:0/1525293419 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f171402ca70 con 0x5645d1ff64d0 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f16faffd640 1 -- 172.21.15.44:0/1525293419 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f1704030020 con 0x5645d1ff64d0 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f1725183740 10 client.6696 fetch_fsmap finished waiting for FSMap version 36 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f1725183740 10 client.6696 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f1725183740 20 client.6696 populate_metadata read hostname 'smithi044' 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f1725183740 1 --2- 172.21.15.44:0/1525293419 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5645d1fda340 0x5645d1fda720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f1725183740 4 client.6696 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f1725183740 1 -- 172.21.15.44:0/1525293419 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645d1f25ee0 con 0x5645d1fda340 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f171bfff640 1 --2- 172.21.15.44:0/1525293419 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5645d1fda340 0x5645d1fda720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f171bfff640 1 --2- 172.21.15.44:0/1525293419 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5645d1fda340 0x5645d1fda720 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b3272fe4d6eed9f server_cookie=315a500b7b11bbbf in_seq=0 out_seq=0 2024-04-02T13:24:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.669+0000 7f16faffd640 10 client.6696 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f16faffd640 1 -- 172.21.15.44:0/1525293419 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5645d1f25ee0 con 0x5645d1fda340 2024-04-02T13:24:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f16faffd640 10 client.6696 handle_command_reply: tid=0 2024-04-02T13:24:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f1725183740 1 client.6696 shutdown 2024-04-02T13:24:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f1725183740 1 -- 172.21.15.44:0/1525293419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 msgr2=0x5645d2007e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f1725183740 1 --2- 172.21.15.44:0/1525293419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d2007e80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f170400ed60 tx=0x7f1704004730 comp rx=0 tx=0).stop 2024-04-02T13:24:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f1725183740 1 -- 172.21.15.44:0/1525293419 shutdown_connections 2024-04-02T13:24:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f1725183740 1 --2- 172.21.15.44:0/1525293419 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5645d1fda340 0x5645d1fda720 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f1725183740 1 --2- 172.21.15.44:0/1525293419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 0x5645d20083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f1725183740 1 --2- 172.21.15.44:0/1525293419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d2007e80 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f1725183740 1 --2- 172.21.15.44:0/1525293419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d1febe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f16faffd640 0 client.6696 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:04.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f1725183740 1 -- 172.21.15.44:0/1525293419 >> 172.21.15.44:0/1525293419 conn(0x5645d1fa8dc0 msgr2=0x5645d1ff4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f1725183740 1 -- 172.21.15.44:0/1525293419 shutdown_connections 2024-04-02T13:24:04.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f1725183740 1 -- 172.21.15.44:0/1525293419 wait complete. 2024-04-02T13:24:04.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.670+0000 7f1725183740 20 client.6696 trim_cache size 0 max 16384 2024-04-02T13:24:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.684+0000 7f1725183740 1 Processor -- start 2024-04-02T13:24:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.684+0000 7f1725183740 1 -- start start 2024-04-02T13:24:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.684+0000 7f1725183740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d200e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.684+0000 7f1725183740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 0x5645d1fe96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.684+0000 7f1725183740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d2069b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.684+0000 7f1725183740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f171c200050 con 0x5645d1ff64d0 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.684+0000 7f1725183740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f171c0c8cd0 con 0x5645d1fed7f0 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.684+0000 7f1725183740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f171c0c9320 con 0x5645d1faebe0 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.684+0000 7f1720b3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d2069b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.684+0000 7f1720b3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d2069b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39272/0 (socket says 172.21.15.44:39272) 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.684+0000 7f1720b3a640 1 -- 172.21.15.44:0/1042153609 learned_addr learned my addr 172.21.15.44:0/1042153609 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.685+0000 7f171bfff640 1 --2- 172.21.15.44:0/1042153609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d200e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.685+0000 7f171b7fe640 1 --2- 172.21.15.44:0/1042153609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 0x5645d1fe96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.685+0000 7f1720b3a640 1 -- 172.21.15.44:0/1042153609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 msgr2=0x5645d200e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.685+0000 7f1720b3a640 1 --2- 172.21.15.44:0/1042153609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d200e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.685+0000 7f1720b3a640 1 -- 172.21.15.44:0/1042153609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 msgr2=0x5645d1fe96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.685+0000 7f1720b3a640 1 --2- 172.21.15.44:0/1042153609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 0x5645d1fe96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.685+0000 7f1720b3a640 1 -- 172.21.15.44:0/1042153609 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645d1fdaa90 con 0x5645d1ff64d0 2024-04-02T13:24:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.685+0000 7f1720b3a640 1 --2- 172.21.15.44:0/1042153609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d2069b40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f17140072a0 tx=0x7f1714009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25889e6c428fb904 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.685+0000 7f17117fa640 1 -- 172.21.15.44:0/1042153609 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1714045040 con 0x5645d1ff64d0 2024-04-02T13:24:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.685+0000 7f17117fa640 1 -- 172.21.15.44:0/1042153609 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f171402ca70 con 0x5645d1ff64d0 2024-04-02T13:24:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.685+0000 7f17117fa640 1 -- 172.21.15.44:0/1042153609 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171402cc20 con 0x5645d1ff64d0 2024-04-02T13:24:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.685+0000 7f1725183740 1 -- 172.21.15.44:0/1042153609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 msgr2=0x5645d2069b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.685+0000 7f1725183740 1 --2- 172.21.15.44:0/1042153609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d2069b40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f17140072a0 tx=0x7f1714009000 comp rx=0 tx=0).stop 2024-04-02T13:24:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.686+0000 7f1725183740 1 -- 172.21.15.44:0/1042153609 shutdown_connections 2024-04-02T13:24:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.686+0000 7f1725183740 1 --2- 172.21.15.44:0/1042153609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d2069b40 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.686+0000 7f1725183740 1 --2- 172.21.15.44:0/1042153609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 0x5645d1fe96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.686+0000 7f1725183740 1 --2- 172.21.15.44:0/1042153609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d200e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.686+0000 7f1725183740 1 -- 172.21.15.44:0/1042153609 >> 172.21.15.44:0/1042153609 conn(0x5645d1fa8dc0 msgr2=0x5645d1fa3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.686+0000 7f1725183740 1 -- 172.21.15.44:0/1042153609 shutdown_connections 2024-04-02T13:24:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.686+0000 7f1725183740 1 -- 172.21.15.44:0/1042153609 wait complete. 2024-04-02T13:24:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.686+0000 7f1725183740 1 -- start start 2024-04-02T13:24:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.686+0000 7f1725183740 1 Processor -- start 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1725183740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d1ff6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1725183740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 0x5645d1fd49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1725183740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d1fd4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1725183740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5645d1feb120 con 0x5645d1ff64d0 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1725183740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5645d2007960 con 0x5645d1fed7f0 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1725183740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5645d1fd8d20 con 0x5645d1faebe0 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1725183740 10 client.0 _ll_register_callbacks cb 0x7f1704009480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1720b3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d1fd4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1720b3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d1fd4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39280/0 (socket says 172.21.15.44:39280) 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1720b3a640 1 -- 172.21.15.44:0/2134493601 learned_addr learned my addr 172.21.15.44:0/2134493601 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f171bfff640 1 --2- 172.21.15.44:0/2134493601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d1ff6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f171b7fe640 1 --2- 172.21.15.44:0/2134493601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 0x5645d1fd49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1720b3a640 1 -- 172.21.15.44:0/2134493601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 msgr2=0x5645d1ff6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1720b3a640 1 --2- 172.21.15.44:0/2134493601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d1ff6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1720b3a640 1 -- 172.21.15.44:0/2134493601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 msgr2=0x5645d1fd49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1720b3a640 1 --2- 172.21.15.44:0/2134493601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 0x5645d1fd49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.687+0000 7f1720b3a640 1 -- 172.21.15.44:0/2134493601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645d1be60b0 con 0x5645d1ff64d0 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f1720b3a640 1 --2- 172.21.15.44:0/2134493601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d1fd4ee0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f1714036040 tx=0x7f1714004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546a7c5be46c63c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f16faffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f16faffd640 1 -- 172.21.15.44:0/2134493601 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1714045040 con 0x5645d1ff64d0 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f16faffd640 1 -- 172.21.15.44:0/2134493601 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f171402c870 con 0x5645d1ff64d0 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f1725183740 1 -- 172.21.15.44:0/2134493601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f171402ca70 con 0x5645d1ff64d0 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f16faffd640 1 -- 172.21.15.44:0/2134493601 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f171405d050 con 0x5645d1ff64d0 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f1725183740 10 client.6702 fetch_fsmap learned FSMap version 36 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f1725183740 1 -- 172.21.15.44:0/2134493601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x5645d1fe9f70 con 0x5645d1ff64d0 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f16faffd640 1 -- 172.21.15.44:0/2134493601 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f171402ca70 con 0x5645d1ff64d0 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f16faffd640 1 -- 172.21.15.44:0/2134493601 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f1714060020 con 0x5645d1ff64d0 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f1725183740 10 client.6702 fetch_fsmap finished waiting for FSMap version 36 2024-04-02T13:24:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f1725183740 10 client.6702 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f1725183740 20 client.6702 populate_metadata read hostname 'smithi044' 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f1725183740 1 --2- 172.21.15.44:0/2134493601 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5645d206eb30 0x5645d206ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f1725183740 4 client.6702 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f1725183740 1 -- 172.21.15.44:0/2134493601 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x5645d1eb34d0 con 0x5645d206eb30 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.688+0000 7f171bfff640 1 --2- 172.21.15.44:0/2134493601 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5645d206eb30 0x5645d206ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f16faffd640 10 client.6702 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f171bfff640 1 --2- 172.21.15.44:0/2134493601 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5645d206eb30 0x5645d206ef10 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cd40c1eb7817850 server_cookie=91904e60ebd4abe5 in_seq=0 out_seq=0 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f16faffd640 1 -- 172.21.15.44:0/2134493601 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+141 (crc 0 0 0) 0x5645d1eb34d0 con 0x5645d206eb30 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f16faffd640 10 client.6702 handle_command_reply: tid=0 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f1725183740 1 client.6702 shutdown 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f1725183740 1 -- 172.21.15.44:0/2134493601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 msgr2=0x5645d1fd4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f1725183740 1 --2- 172.21.15.44:0/2134493601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d1fd4ee0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f1714036040 tx=0x7f1714004ab0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f1725183740 1 -- 172.21.15.44:0/2134493601 shutdown_connections 2024-04-02T13:24:04.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f1725183740 1 --2- 172.21.15.44:0/2134493601 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x5645d206eb30 0x5645d206ef10 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f16faffd640 0 client.6702 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:04.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f1725183740 1 --2- 172.21.15.44:0/2134493601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645d1ff64d0 0x5645d1fd4ee0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f1725183740 1 --2- 172.21.15.44:0/2134493601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645d1fed7f0 0x5645d1fd49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f1725183740 1 --2- 172.21.15.44:0/2134493601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5645d1faebe0 0x5645d1ff6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f1725183740 1 -- 172.21.15.44:0/2134493601 >> 172.21.15.44:0/2134493601 conn(0x5645d1fa8dc0 msgr2=0x5645d1fec510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f1725183740 1 -- 172.21.15.44:0/2134493601 shutdown_connections 2024-04-02T13:24:04.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f1725183740 1 -- 172.21.15.44:0/2134493601 wait complete. 2024-04-02T13:24:04.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.689+0000 7f1725183740 20 client.6702 trim_cache size 0 max 16384 2024-04-02T13:24:04.698 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:04.699 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-04-02T13:24:04.699 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:04.699 INFO:teuthology.orchestra.run.smithi044.stdout: "damage_type": "backtrace", 2024-04-02T13:24:04.699 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 1858961328, 2024-04-02T13:24:04.699 INFO:teuthology.orchestra.run.smithi044.stdout: "ino": 1099511627777, 2024-04-02T13:24:04.699 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/dir_test/file" 2024-04-02T13:24:04.699 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:04.699 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-04-02T13:24:04.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.690+0000 7f172259d640 1 -- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f16fc06c8f0 msgr2=0x7f16fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.690+0000 7f172259d640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f16fc06c8f0 0x7f16fc06edb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1704006fd0 tx=0x7f1704008040 comp rx=0 tx=0).stop 2024-04-02T13:24:04.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.690+0000 7f172259d640 1 -- 172.21.15.44:0/1141740496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f171c0d09a0 msgr2=0x7f171c0b3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.690+0000 7f172259d640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f171c0d09a0 0x7f171c0b3aa0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f170c00d800 tx=0x7f170c00dcc0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.690+0000 7f172259d640 1 -- 172.21.15.44:0/1141740496 shutdown_connections 2024-04-02T13:24:04.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.690+0000 7f172259d640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f16fc06c8f0 0x7f16fc06edb0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.690+0000 7f172259d640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f171c0d4c90 0x7f171c1615b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.690+0000 7f172259d640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f171c0d4850 0x7f171c161070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.690+0000 7f172259d640 1 --2- 172.21.15.44:0/1141740496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f171c0d09a0 0x7f171c0b3aa0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.690+0000 7f172259d640 1 -- 172.21.15.44:0/1141740496 >> 172.21.15.44:0/1141740496 conn(0x7f171c09d470 msgr2=0x7f171c0b1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.690+0000 7f172259d640 1 -- 172.21.15.44:0/1141740496 shutdown_connections 2024-04-02T13:24:04.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.690+0000 7f172259d640 1 -- 172.21.15.44:0/1141740496 wait complete. 2024-04-02T13:24:04.703 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-02T13:24:04.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.801+0000 7f0aeb760640 1 Processor -- start 2024-04-02T13:24:04.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.801+0000 7f0aeb760640 1 -- start start 2024-04-02T13:24:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0aeb760640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0ae40ccd10 0x7f0ae40cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0aeb760640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae4160170 con 0x7f0ae415f740 2024-04-02T13:24:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0aeb760640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae4160450 con 0x7f0ae415fc70 2024-04-02T13:24:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0aeb760640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae4160730 con 0x7f0ae40ccd10 2024-04-02T13:24:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0ae94fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0ae415f740 0x7f0ae40cc600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56574/0 (socket says 172.21.15.44:56574) 2024-04-02T13:24:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0ae94fc640 1 -- 172.21.15.44:0/1231022596 learned_addr learned my addr 172.21.15.44:0/1231022596 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0ae9cfd640 1 --2- 172.21.15.44:0/1231022596 >> v2:172.21.15.69:3300/0 conn(0x7f0ae40ccd10 0x7f0ae40cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 632185656 0 0) 0x7f0ae4160170 con 0x7f0ae415f740 2024-04-02T13:24:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ad0003520 con 0x7f0ae415f740 2024-04-02T13:24:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 961627772 0 0) 0x7f0ad0003520 con 0x7f0ae415f740 2024-04-02T13:24:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 >> v2:172.21.15.69:3300/0 conn(0x7f0ae40ccd10 msgr2=0x7f0ae40cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0adbfff640 1 --2- 172.21.15.44:0/1231022596 >> v2:172.21.15.69:3300/0 conn(0x7f0ae40ccd10 0x7f0ae40cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 >> v1:172.21.15.69:6789/0 conn(0x7f0ae415fc70 legacy=0x7f0ae40ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.802+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae4160a10 con 0x7f0ae415f740 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f0ae0001670 con 0x7f0ae415f740 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 >> v1:172.21.15.44:6789/0 conn(0x7f0ae415f740 legacy=0x7f0ae40cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0adbfff640 1 --2- 172.21.15.44:0/1231022596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad00039b0 0x7f0ad0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0adbfff640 1 --2- 172.21.15.44:0/1231022596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ad0004800 0x7f0ad0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0adbfff640 1 --2- 172.21.15.44:0/1231022596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ad00051c0 0x7f0ad000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae4160730 con 0x7f0ad00039b0 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad000c060 con 0x7f0ad0004800 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0ad000c260 con 0x7f0ad00051c0 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0ae94fc640 1 --2- 172.21.15.44:0/1231022596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad00039b0 0x7f0ad0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0ae8cfb640 1 --2- 172.21.15.44:0/1231022596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ad0004800 0x7f0ad0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0ae9cfd640 1 --2- 172.21.15.44:0/1231022596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ad00051c0 0x7f0ad000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0ae94fc640 1 -- 172.21.15.44:0/1231022596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ad00051c0 msgr2=0x7f0ad000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0ae94fc640 1 --2- 172.21.15.44:0/1231022596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ad00051c0 0x7f0ad000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0ae94fc640 1 -- 172.21.15.44:0/1231022596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ad0004800 msgr2=0x7f0ad0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:04.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0ae94fc640 1 --2- 172.21.15.44:0/1231022596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ad0004800 0x7f0ad0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0ae94fc640 1 -- 172.21.15.44:0/1231022596 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad0003520 con 0x7f0ad00039b0 2024-04-02T13:24:04.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.803+0000 7f0ae9cfd640 1 -- 172.21.15.44:0/1231022596 reap_dead start 2024-04-02T13:24:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.804+0000 7f0ae94fc640 1 --2- 172.21.15.44:0/1231022596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad00039b0 0x7f0ad0003dd0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0ae000c240 tx=0x7f0ad0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eab3aa93c847514 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.804+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0007cf0 con 0x7f0ad00039b0 2024-04-02T13:24:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.804+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae0030070 con 0x7f0ad00039b0 2024-04-02T13:24:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.804+0000 7f0adbfff640 1 -- 172.21.15.44:0/1231022596 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0005d70 con 0x7f0ad00039b0 2024-04-02T13:24:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.804+0000 7f0aeb760640 1 -- 172.21.15.44:0/1231022596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad00039b0 msgr2=0x7f0ad0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.804+0000 7f0aeb760640 1 --2- 172.21.15.44:0/1231022596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad00039b0 0x7f0ad0003dd0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0ae000c240 tx=0x7f0ad0005830 comp rx=0 tx=0).stop 2024-04-02T13:24:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.804+0000 7f0aeb760640 1 -- 172.21.15.44:0/1231022596 shutdown_connections 2024-04-02T13:24:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.804+0000 7f0aeb760640 1 --2- 172.21.15.44:0/1231022596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad00039b0 0x7f0ad0003dd0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.804+0000 7f0aeb760640 1 -- 172.21.15.44:0/1231022596 >> 172.21.15.44:0/1231022596 conn(0x7f0ae40cfcd0 msgr2=0x7f0ae40d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.805+0000 7f0aeb760640 1 -- 172.21.15.44:0/1231022596 shutdown_connections 2024-04-02T13:24:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.805+0000 7f0aeb760640 1 -- 172.21.15.44:0/1231022596 wait complete. 2024-04-02T13:24:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.806+0000 7f0aeb760640 1 Processor -- start 2024-04-02T13:24:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.806+0000 7f0aeb760640 1 -- start start 2024-04-02T13:24:04.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.806+0000 7f0aeb760640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad00039b0 0x7f0ae41510f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.806+0000 7f0aeb760640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ae4156e00 0x7f0ae415a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.806+0000 7f0aeb760640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ae415a6d0 0x7f0ae4154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.806+0000 7f0aeb760640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae40cf230 con 0x7f0ad00039b0 2024-04-02T13:24:04.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.806+0000 7f0aeb760640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae40cf0b0 con 0x7f0ae415a6d0 2024-04-02T13:24:04.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.806+0000 7f0aeb760640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae40cf3b0 con 0x7f0ae4156e00 2024-04-02T13:24:04.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.806+0000 7f0ae94fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad00039b0 0x7f0ae41510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.806+0000 7f0ae94fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad00039b0 0x7f0ae41510f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39298/0 (socket says 172.21.15.44:39298) 2024-04-02T13:24:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.806+0000 7f0ae94fc640 1 -- 172.21.15.44:0/1554548388 learned_addr learned my addr 172.21.15.44:0/1554548388 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.807+0000 7f0ae9cfd640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ae415a6d0 0x7f0ae4154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.807+0000 7f0ae8cfb640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ae4156e00 0x7f0ae415a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.807+0000 7f0ae94fc640 1 -- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ae4156e00 msgr2=0x7f0ae415a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.807+0000 7f0ae94fc640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ae4156e00 0x7f0ae415a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.807+0000 7f0ae94fc640 1 -- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ae415a6d0 msgr2=0x7f0ae4154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.807+0000 7f0ae94fc640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ae415a6d0 0x7f0ae4154620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.807+0000 7f0ae94fc640 1 -- 172.21.15.44:0/1554548388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae4151630 con 0x7f0ad00039b0 2024-04-02T13:24:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.807+0000 7f0ae94fc640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad00039b0 0x7f0ae41510f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0ae002c4c0 tx=0x7f0ae4160340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.807+0000 7f0ad9ffb640 1 -- 172.21.15.44:0/1554548388 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0007cf0 con 0x7f0ad00039b0 2024-04-02T13:24:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.807+0000 7f0ad9ffb640 1 -- 172.21.15.44:0/1554548388 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae00049f0 con 0x7f0ad00039b0 2024-04-02T13:24:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.807+0000 7f0aeb760640 1 -- 172.21.15.44:0/1554548388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae41590f0 con 0x7f0ad00039b0 2024-04-02T13:24:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.807+0000 7f0ad9ffb640 1 -- 172.21.15.44:0/1554548388 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0001c30 con 0x7f0ad00039b0 2024-04-02T13:24:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.807+0000 7f0aeb760640 1 -- 172.21.15.44:0/1554548388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae41596b0 con 0x7f0ad00039b0 2024-04-02T13:24:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.808+0000 7f0aeb760640 1 -- 172.21.15.44:0/1554548388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae40684f0 con 0x7f0ad00039b0 2024-04-02T13:24:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.809+0000 7f0ad9ffb640 1 -- 172.21.15.44:0/1554548388 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ae0007070 con 0x7f0ad00039b0 2024-04-02T13:24:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.809+0000 7f0ad9ffb640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ac806c8f0 0x7f0ac806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.810+0000 7f0ad9ffb640 1 -- 172.21.15.44:0/1554548388 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f0ae00acf40 con 0x7f0ad00039b0 2024-04-02T13:24:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.810+0000 7f0ae8cfb640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ac806c8f0 0x7f0ac806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.810+0000 7f0ae8cfb640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ac806c8f0 0x7f0ac806edb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4000c00 tx=0x7f0ad4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:04.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:04.814+0000 7f0ad9ffb640 1 -- 172.21.15.44:0/1554548388 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0ae007b1e0 con 0x7f0ad00039b0 2024-04-02T13:24:05.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.039+0000 7f0aeb760640 1 -- 172.21.15.44:0/1554548388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0ae40cee90 con 0x7f0ad00039b0 2024-04-02T13:24:05.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.039+0000 7f0ad9ffb640 1 -- 172.21.15.44:0/1554548388 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+164 (secure 0 0 0) 0x7f0ae007aae0 con 0x7f0ad00039b0 2024-04-02T13:24:05.043 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:05.043 INFO:teuthology.orchestra.run.smithi044.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-04-02T13:24:05.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.041+0000 7f0aeb760640 1 -- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ac806c8f0 msgr2=0x7f0ac806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.041+0000 7f0aeb760640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ac806c8f0 0x7f0ac806edb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4000c00 tx=0x7f0ad4009040 comp rx=0 tx=0).stop 2024-04-02T13:24:05.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.041+0000 7f0aeb760640 1 -- 172.21.15.44:0/1554548388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad00039b0 msgr2=0x7f0ae41510f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.041+0000 7f0aeb760640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad00039b0 0x7f0ae41510f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0ae002c4c0 tx=0x7f0ae4160340 comp rx=0 tx=0).stop 2024-04-02T13:24:05.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.042+0000 7f0aeb760640 1 -- 172.21.15.44:0/1554548388 shutdown_connections 2024-04-02T13:24:05.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.042+0000 7f0aeb760640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ac806c8f0 0x7f0ac806edb0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.042+0000 7f0aeb760640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ae415a6d0 0x7f0ae4154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.042+0000 7f0aeb760640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ae4156e00 0x7f0ae415a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.042+0000 7f0aeb760640 1 --2- 172.21.15.44:0/1554548388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad00039b0 0x7f0ae41510f0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.042+0000 7f0aeb760640 1 -- 172.21.15.44:0/1554548388 >> 172.21.15.44:0/1554548388 conn(0x7f0ae40cfcd0 msgr2=0x7f0ae4071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:05.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.042+0000 7f0aeb760640 1 -- 172.21.15.44:0/1554548388 shutdown_connections 2024-04-02T13:24:05.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.042+0000 7f0aeb760640 1 -- 172.21.15.44:0/1554548388 wait complete. 2024-04-02T13:24:05.055 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-02T13:24:05.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:24:05.055 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub start / repair,recursive,force 2024-04-02T13:24:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.161+0000 7f8574d7c640 1 Processor -- start 2024-04-02T13:24:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.161+0000 7f8574d7c640 1 -- start start 2024-04-02T13:24:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.162+0000 7f8574d7c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f85700b4180 0x7f8570168f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.162+0000 7f8574d7c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85700d1630 con 0x7f857006cff0 2024-04-02T13:24:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.162+0000 7f8574d7c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85700d1910 con 0x7f857015e5d0 2024-04-02T13:24:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.162+0000 7f8574d7c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f85700d1bf0 con 0x7f85700b4180 2024-04-02T13:24:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.162+0000 7f856e59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f857006cff0 0x7f857006d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56590/0 (socket says 172.21.15.44:56590) 2024-04-02T13:24:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.162+0000 7f856e59c640 1 -- 172.21.15.44:0/2574830889 learned_addr learned my addr 172.21.15.44:0/2574830889 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.162+0000 7f856ed9d640 1 --2- 172.21.15.44:0/2574830889 >> v2:172.21.15.69:3300/0 conn(0x7f85700b4180 0x7f8570168f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.162+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3692131350 0 0) 0x7f85700d1630 con 0x7f857006cff0 2024-04-02T13:24:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.162+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f855c003520 con 0x7f857006cff0 2024-04-02T13:24:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 152711649 0 0) 0x7f855c003520 con 0x7f857006cff0 2024-04-02T13:24:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 >> v2:172.21.15.69:3300/0 conn(0x7f85700b4180 msgr2=0x7f8570168f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856dd9b640 1 --2- 172.21.15.44:0/2574830889 >> v2:172.21.15.69:3300/0 conn(0x7f85700b4180 0x7f8570168f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 >> v1:172.21.15.69:6789/0 conn(0x7f857015e5d0 legacy=0x7f85700d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85700b1540 con 0x7f857006cff0 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f8560001670 con 0x7f857006cff0 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 >> v1:172.21.15.44:6789/0 conn(0x7f857006cff0 legacy=0x7f857006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856dd9b640 1 --2- 172.21.15.44:0/2574830889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c0039b0 0x7f855c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856dd9b640 1 --2- 172.21.15.44:0/2574830889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f855c004800 0x7f855c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856dd9b640 1 --2- 172.21.15.44:0/2574830889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c0051c0 0x7f855c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f85700d1bf0 con 0x7f855c0039b0 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f855c00c160 con 0x7f855c0051c0 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f855c00c360 con 0x7f855c004800 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856e59c640 1 --2- 172.21.15.44:0/2574830889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c0039b0 0x7f855c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856ed9d640 1 --2- 172.21.15.44:0/2574830889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c0051c0 0x7f855c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f8567fff640 1 --2- 172.21.15.44:0/2574830889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f855c004800 0x7f855c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856e59c640 1 -- 172.21.15.44:0/2574830889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f855c004800 msgr2=0x7f855c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856e59c640 1 --2- 172.21.15.44:0/2574830889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f855c004800 0x7f855c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856e59c640 1 -- 172.21.15.44:0/2574830889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c0051c0 msgr2=0x7f855c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856e59c640 1 --2- 172.21.15.44:0/2574830889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c0051c0 0x7f855c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856ed9d640 1 -- 172.21.15.44:0/2574830889 reap_dead start 2024-04-02T13:24:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.163+0000 7f856e59c640 1 -- 172.21.15.44:0/2574830889 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855c003520 con 0x7f855c0039b0 2024-04-02T13:24:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.164+0000 7f856e59c640 1 --2- 172.21.15.44:0/2574830889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c0039b0 0x7f855c003dd0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f856000c240 tx=0x7f855c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91110c4f283c5c2e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.164+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8560007d10 con 0x7f855c0039b0 2024-04-02T13:24:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.164+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8560030070 con 0x7f855c0039b0 2024-04-02T13:24:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.164+0000 7f856dd9b640 1 -- 172.21.15.44:0/2574830889 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8560005d90 con 0x7f855c0039b0 2024-04-02T13:24:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.165+0000 7f8574d7c640 1 -- 172.21.15.44:0/2574830889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c0039b0 msgr2=0x7f855c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.165+0000 7f8574d7c640 1 --2- 172.21.15.44:0/2574830889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c0039b0 0x7f855c003dd0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f856000c240 tx=0x7f855c005830 comp rx=0 tx=0).stop 2024-04-02T13:24:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.165+0000 7f8574d7c640 1 -- 172.21.15.44:0/2574830889 shutdown_connections 2024-04-02T13:24:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.165+0000 7f8574d7c640 1 --2- 172.21.15.44:0/2574830889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c0039b0 0x7f855c003dd0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.165+0000 7f8574d7c640 1 -- 172.21.15.44:0/2574830889 >> 172.21.15.44:0/2574830889 conn(0x7f857009aa20 msgr2=0x7f857009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.165+0000 7f8574d7c640 1 -- 172.21.15.44:0/2574830889 shutdown_connections 2024-04-02T13:24:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.165+0000 7f8574d7c640 1 -- 172.21.15.44:0/2574830889 wait complete. 2024-04-02T13:24:05.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.166+0000 7f8574d7c640 1 Processor -- start 2024-04-02T13:24:05.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.166+0000 7f8574d7c640 1 -- start start 2024-04-02T13:24:05.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f8574d7c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c0039b0 0x7f8570153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f8574d7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85701544e0 0x7f85701569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f8574d7c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85701f71f0 0x7f8570156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f8574d7c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f857006d550 con 0x7f85701f71f0 2024-04-02T13:24:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f8574d7c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f857006d3d0 con 0x7f855c0039b0 2024-04-02T13:24:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f8574d7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f857006d6d0 con 0x7f85701544e0 2024-04-02T13:24:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f856ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85701f71f0 0x7f8570156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f856ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85701f71f0 0x7f8570156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39324/0 (socket says 172.21.15.44:39324) 2024-04-02T13:24:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f856ed9d640 1 -- 172.21.15.44:0/4132865821 learned_addr learned my addr 172.21.15.44:0/4132865821 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f8567fff640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85701544e0 0x7f85701569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f856e59c640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c0039b0 0x7f8570153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f856ed9d640 1 -- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85701544e0 msgr2=0x7f85701569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f856ed9d640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85701544e0 0x7f85701569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f856ed9d640 1 -- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c0039b0 msgr2=0x7f8570153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f856ed9d640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c0039b0 0x7f8570153520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f856ed9d640 1 -- 172.21.15.44:0/4132865821 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8570154970 con 0x7f85701f71f0 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f856ed9d640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85701f71f0 0x7f8570156ef0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8570073280 tx=0x7f856800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f8566ffd640 1 -- 172.21.15.44:0/4132865821 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8568011e90 con 0x7f85701f71f0 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.167+0000 7f8566ffd640 1 -- 172.21.15.44:0/4132865821 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f856801c070 con 0x7f85701f71f0 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.168+0000 7f8566ffd640 1 -- 172.21.15.44:0/4132865821 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8568003bd0 con 0x7f85701f71f0 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.168+0000 7f8574d7c640 1 -- 172.21.15.44:0/4132865821 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8570154c50 con 0x7f85701f71f0 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.168+0000 7f8574d7c640 1 -- 172.21.15.44:0/4132865821 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8570155210 con 0x7f85701f71f0 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.169+0000 7f8566ffd640 1 -- 172.21.15.44:0/4132865821 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8568004030 con 0x7f85701f71f0 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.170+0000 7f8566ffd640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f855006cac0 0x7f855006ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.170+0000 7f8566ffd640 1 -- 172.21.15.44:0/4132865821 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f856808c690 con 0x7f85701f71f0 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.170+0000 7f856e59c640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f855006cac0 0x7f855006ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.170+0000 7f856e59c640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f855006cac0 0x7f855006ef80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8560034000 tx=0x7f8560009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.170+0000 7f8577a36740 1 Processor -- start 2024-04-02T13:24:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.171+0000 7f8577a36740 1 -- start start 2024-04-02T13:24:05.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.171+0000 7f8577a36740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7160 0x56512c49a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.171+0000 7f8577a36740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c7540 0x56512c4c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.171+0000 7f8577a36740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c49a640 0x56512c4ca430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.171+0000 7f8577a36740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56512c403570 con 0x56512c4c7160 2024-04-02T13:24:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.171+0000 7f8577a36740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56512c3ab8d0 con 0x56512c49a640 2024-04-02T13:24:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.171+0000 7f8577a36740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x56512c3ff520 con 0x56512c4c7540 2024-04-02T13:24:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.171+0000 7f856e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7160 0x56512c49a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.172+0000 7f856e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7160 0x56512c49a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39330/0 (socket says 172.21.15.44:39330) 2024-04-02T13:24:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.172+0000 7f856e59c640 1 -- 172.21.15.44:0/1787109262 learned_addr learned my addr 172.21.15.44:0/1787109262 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.172+0000 7f8567fff640 1 --2- 172.21.15.44:0/1787109262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c7540 0x56512c4c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.172+0000 7f856ed9d640 1 --2- 172.21.15.44:0/1787109262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c49a640 0x56512c4ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.172+0000 7f856e59c640 1 -- 172.21.15.44:0/1787109262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c7540 msgr2=0x56512c4c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.172+0000 7f856e59c640 1 --2- 172.21.15.44:0/1787109262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c7540 0x56512c4c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.172+0000 7f856e59c640 1 -- 172.21.15.44:0/1787109262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c49a640 msgr2=0x56512c4ca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.172+0000 7f856e59c640 1 --2- 172.21.15.44:0/1787109262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c49a640 0x56512c4ca430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.172+0000 7f856e59c640 1 -- 172.21.15.44:0/1787109262 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56512c1e3940 con 0x56512c4c7160 2024-04-02T13:24:05.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.172+0000 7f856e59c640 1 --2- 172.21.15.44:0/1787109262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7160 0x56512c49a100 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8560001830 tx=0x7f8560007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a96f8c19124b0b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.172+0000 7f854ffff640 1 -- 172.21.15.44:0/1787109262 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856002cd00 con 0x56512c4c7160 2024-04-02T13:24:05.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.172+0000 7f854ffff640 1 -- 172.21.15.44:0/1787109262 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8560030040 con 0x56512c4c7160 2024-04-02T13:24:05.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.172+0000 7f854ffff640 1 -- 172.21.15.44:0/1787109262 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8560007070 con 0x56512c4c7160 2024-04-02T13:24:05.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.173+0000 7f8577a36740 1 -- 172.21.15.44:0/1787109262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7160 msgr2=0x56512c49a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.173+0000 7f8577a36740 1 --2- 172.21.15.44:0/1787109262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7160 0x56512c49a100 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8560001830 tx=0x7f8560007000 comp rx=0 tx=0).stop 2024-04-02T13:24:05.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.173+0000 7f8577a36740 1 -- 172.21.15.44:0/1787109262 shutdown_connections 2024-04-02T13:24:05.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.173+0000 7f8577a36740 1 --2- 172.21.15.44:0/1787109262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c49a640 0x56512c4ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.173+0000 7f8577a36740 1 --2- 172.21.15.44:0/1787109262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c7540 0x56512c4c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.173+0000 7f8577a36740 1 --2- 172.21.15.44:0/1787109262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7160 0x56512c49a100 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.173+0000 7f8577a36740 1 -- 172.21.15.44:0/1787109262 >> 172.21.15.44:0/1787109262 conn(0x56512c49c370 msgr2=0x56512c4e4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.173+0000 7f8577a36740 1 -- 172.21.15.44:0/1787109262 shutdown_connections 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.173+0000 7f8577a36740 1 -- 172.21.15.44:0/1787109262 wait complete. 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.173+0000 7f8577a36740 1 -- start start 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.174+0000 7f8577a36740 1 Processor -- start 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.174+0000 7f8577a36740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7160 0x56512c4c1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.174+0000 7f8577a36740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 0x56512c4bfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.174+0000 7f8577a36740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c01f0 0x56512c4f8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.174+0000 7f8577a36740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56512c49e910 con 0x56512c4c7540 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.174+0000 7f8577a36740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56512c3bec90 con 0x56512c4c7160 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.174+0000 7f8577a36740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x56512c49dd00 con 0x56512c4c01f0 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f8577a36740 10 client.0 _ll_register_callbacks cb 0x56512c116d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f8567fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 0x56512c4bfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f8567fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 0x56512c4bfcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39342/0 (socket says 172.21.15.44:39342) 2024-04-02T13:24:05.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f8567fff640 1 -- 172.21.15.44:0/2528889891 learned_addr learned my addr 172.21.15.44:0/2528889891 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f856ed9d640 1 --2- 172.21.15.44:0/2528889891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c01f0 0x56512c4f8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f856e59c640 1 --2- 172.21.15.44:0/2528889891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7160 0x56512c4c1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f8567fff640 1 -- 172.21.15.44:0/2528889891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c01f0 msgr2=0x56512c4f8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f8567fff640 1 --2- 172.21.15.44:0/2528889891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c01f0 0x56512c4f8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f8567fff640 1 -- 172.21.15.44:0/2528889891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7160 msgr2=0x56512c4c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f8567fff640 1 --2- 172.21.15.44:0/2528889891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7160 0x56512c4c1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f8567fff640 1 -- 172.21.15.44:0/2528889891 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56512c2440d0 con 0x56512c4c7540 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f8567fff640 1 --2- 172.21.15.44:0/2528889891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 0x56512c4bfcb0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8558003bc0 tx=0x7f8558001660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba96b5430e8b5bc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f854d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f854d7fa640 1 -- 172.21.15.44:0/2528889891 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8558015680 con 0x56512c4c7540 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f854d7fa640 1 -- 172.21.15.44:0/2528889891 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f8558010040 con 0x56512c4c7540 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f8577a36740 1 -- 172.21.15.44:0/2528889891 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8560030040 con 0x56512c4c7540 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f854d7fa640 1 -- 172.21.15.44:0/2528889891 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8558015820 con 0x56512c4c7540 2024-04-02T13:24:05.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f8577a36740 10 client.6720 fetch_fsmap learned FSMap version 36 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.175+0000 7f8577a36740 1 -- 172.21.15.44:0/2528889891 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x56512c0d7230 con 0x56512c4c7540 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.176+0000 7f854d7fa640 1 -- 172.21.15.44:0/2528889891 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8560030040 con 0x56512c4c7540 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.176+0000 7f854d7fa640 1 -- 172.21.15.44:0/2528889891 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f85580159c0 con 0x56512c4c7540 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.176+0000 7f8577a36740 10 client.6720 fetch_fsmap finished waiting for FSMap version 36 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.176+0000 7f8577a36740 10 client.6720 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.176+0000 7f8577a36740 20 client.6720 populate_metadata read hostname 'smithi044' 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.176+0000 7f8577a36740 1 --2- 172.21.15.44:0/2528889891 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x56512c4f8d40 0x56512c4c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.176+0000 7f8577a36740 4 client.6720 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.176+0000 7f8577a36740 1 -- 172.21.15.44:0/2528889891 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56512c410ab0 con 0x56512c4f8d40 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.176+0000 7f856e59c640 1 --2- 172.21.15.44:0/2528889891 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x56512c4f8d40 0x56512c4c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.176+0000 7f854d7fa640 10 client.6720 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.176+0000 7f856e59c640 1 --2- 172.21.15.44:0/2528889891 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x56512c4f8d40 0x56512c4c86d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4e6fbac9d915fd3 server_cookie=bfd294d7b50c6064 in_seq=0 out_seq=0 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f854d7fa640 1 -- 172.21.15.44:0/2528889891 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56512c410ab0 con 0x56512c4f8d40 2024-04-02T13:24:05.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f854d7fa640 10 client.6720 handle_command_reply: tid=0 2024-04-02T13:24:05.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f8577a36740 1 client.6720 shutdown 2024-04-02T13:24:05.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f8577a36740 1 -- 172.21.15.44:0/2528889891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 msgr2=0x56512c4bfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f8577a36740 1 --2- 172.21.15.44:0/2528889891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 0x56512c4bfcb0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8558003bc0 tx=0x7f8558001660 comp rx=0 tx=0).stop 2024-04-02T13:24:05.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f8577a36740 1 -- 172.21.15.44:0/2528889891 shutdown_connections 2024-04-02T13:24:05.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f8577a36740 1 --2- 172.21.15.44:0/2528889891 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x56512c4f8d40 0x56512c4c86d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f8577a36740 1 --2- 172.21.15.44:0/2528889891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c01f0 0x56512c4f8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f854d7fa640 0 client.6720 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:05.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f8577a36740 1 --2- 172.21.15.44:0/2528889891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 0x56512c4bfcb0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f8577a36740 1 --2- 172.21.15.44:0/2528889891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7160 0x56512c4c1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f8577a36740 1 -- 172.21.15.44:0/2528889891 >> 172.21.15.44:0/2528889891 conn(0x56512c49a640 msgr2=0x56512c49ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:05.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f8577a36740 1 -- 172.21.15.44:0/2528889891 shutdown_connections 2024-04-02T13:24:05.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f8577a36740 1 -- 172.21.15.44:0/2528889891 wait complete. 2024-04-02T13:24:05.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.177+0000 7f8577a36740 20 client.6720 trim_cache size 0 max 16384 2024-04-02T13:24:05.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.195+0000 7f8577a36740 1 Processor -- start 2024-04-02T13:24:05.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.195+0000 7f8577a36740 1 -- start start 2024-04-02T13:24:05.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f8577a36740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c01f0 0x56512c4c7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f8577a36740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c7160 0x56512c4e7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f8577a36740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7540 0x56512c4cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f8577a36740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8570201590 con 0x56512c4c01f0 2024-04-02T13:24:05.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f8577a36740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f85702005b0 con 0x56512c4c7540 2024-04-02T13:24:05.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f8577a36740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f85701ff910 con 0x56512c4c7160 2024-04-02T13:24:05.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f856e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c01f0 0x56512c4c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f856e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c01f0 0x56512c4c7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39356/0 (socket says 172.21.15.44:39356) 2024-04-02T13:24:05.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f856e59c640 1 -- 172.21.15.44:0/187243689 learned_addr learned my addr 172.21.15.44:0/187243689 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:05.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f856e59c640 1 -- 172.21.15.44:0/187243689 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c7160 msgr2=0x56512c4e7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f856e59c640 1 --2- 172.21.15.44:0/187243689 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c7160 0x56512c4e7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f856e59c640 1 -- 172.21.15.44:0/187243689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7540 msgr2=0x56512c4cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f856e59c640 1 --2- 172.21.15.44:0/187243689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7540 0x56512c4cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f856e59c640 1 -- 172.21.15.44:0/187243689 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56512c4caee0 con 0x56512c4c01f0 2024-04-02T13:24:05.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f856e59c640 1 --2- 172.21.15.44:0/187243689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c01f0 0x56512c4c7920 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f856000c240 tx=0x7f856003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420fdc19b637f63b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f854f7fe640 1 -- 172.21.15.44:0/187243689 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8560007320 con 0x56512c4c01f0 2024-04-02T13:24:05.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f854f7fe640 1 -- 172.21.15.44:0/187243689 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8560030070 con 0x56512c4c01f0 2024-04-02T13:24:05.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.196+0000 7f854f7fe640 1 -- 172.21.15.44:0/187243689 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85600076c0 con 0x56512c4c01f0 2024-04-02T13:24:05.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.197+0000 7f8577a36740 1 -- 172.21.15.44:0/187243689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c01f0 msgr2=0x56512c4c7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.197+0000 7f8577a36740 1 --2- 172.21.15.44:0/187243689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c01f0 0x56512c4c7920 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f856000c240 tx=0x7f856003e000 comp rx=0 tx=0).stop 2024-04-02T13:24:05.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.197+0000 7f8577a36740 1 -- 172.21.15.44:0/187243689 shutdown_connections 2024-04-02T13:24:05.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.197+0000 7f8577a36740 1 --2- 172.21.15.44:0/187243689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7540 0x56512c4cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.197+0000 7f8577a36740 1 --2- 172.21.15.44:0/187243689 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c7160 0x56512c4e7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.197+0000 7f8577a36740 1 --2- 172.21.15.44:0/187243689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c01f0 0x56512c4c7920 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.197+0000 7f8577a36740 1 -- 172.21.15.44:0/187243689 >> 172.21.15.44:0/187243689 conn(0x56512c49a640 msgr2=0x56512c4ca240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:05.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.197+0000 7f8577a36740 1 -- 172.21.15.44:0/187243689 shutdown_connections 2024-04-02T13:24:05.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.197+0000 7f8577a36740 1 -- 172.21.15.44:0/187243689 wait complete. 2024-04-02T13:24:05.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.197+0000 7f8577a36740 1 -- start start 2024-04-02T13:24:05.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.197+0000 7f8577a36740 1 Processor -- start 2024-04-02T13:24:05.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.198+0000 7f8577a36740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c01f0 0x56512c4c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.198+0000 7f8577a36740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7160 0x56512c4e6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.198+0000 7f8577a36740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 0x56512c4e6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.198+0000 7f8577a36740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56512c4994b0 con 0x56512c4c7540 2024-04-02T13:24:05.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.198+0000 7f8577a36740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f85701fff60 con 0x56512c4c7160 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.198+0000 7f8577a36740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x56512c4c0e50 con 0x56512c4c01f0 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.198+0000 7f8577a36740 10 client.0 _ll_register_callbacks cb 0x7f8558003820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.198+0000 7f856ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 0x56512c4e6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.198+0000 7f856ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 0x56512c4e6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39372/0 (socket says 172.21.15.44:39372) 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.198+0000 7f856ed9d640 1 -- 172.21.15.44:0/810368182 learned_addr learned my addr 172.21.15.44:0/810368182 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f8567fff640 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7160 0x56512c4e6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f856e59c640 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c01f0 0x56512c4c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f856ed9d640 1 -- 172.21.15.44:0/810368182 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c01f0 msgr2=0x56512c4c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f856ed9d640 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c01f0 0x56512c4c5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f856ed9d640 1 -- 172.21.15.44:0/810368182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7160 msgr2=0x56512c4e6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f856ed9d640 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7160 0x56512c4e6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f856ed9d640 1 -- 172.21.15.44:0/810368182 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56512c4e7000 con 0x56512c4c7540 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f8567fff640 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7160 0x56512c4e6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:05.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f856ed9d640 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 0x56512c4e6680 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f856805a300 tx=0x7f856805e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee33ddcc65818916 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f854d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f854d7fa640 1 -- 172.21.15.44:0/810368182 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85680599c0 con 0x56512c4c7540 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f854d7fa640 1 -- 172.21.15.44:0/810368182 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f85680560c0 con 0x56512c4c7540 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f8577a36740 1 -- 172.21.15.44:0/810368182 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8560030070 con 0x56512c4c7540 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f854d7fa640 1 -- 172.21.15.44:0/810368182 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8568058730 con 0x56512c4c7540 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f8577a36740 10 client.6726 fetch_fsmap learned FSMap version 36 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f8577a36740 1 -- 172.21.15.44:0/810368182 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x56512c55fd90 con 0x56512c4c7540 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f854d7fa640 1 -- 172.21.15.44:0/810368182 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8560030070 con 0x56512c4c7540 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f854d7fa640 1 -- 172.21.15.44:0/810368182 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f85680562c0 con 0x56512c4c7540 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f8577a36740 10 client.6726 fetch_fsmap finished waiting for FSMap version 36 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f8577a36740 10 client.6726 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f8577a36740 20 client.6726 populate_metadata read hostname 'smithi044' 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f8577a36740 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x56512c55ed10 0x56512c55f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f8577a36740 4 client.6726 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive", "force"]}] 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.199+0000 7f8577a36740 1 -- 172.21.15.44:0/810368182 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive", "force"]}) -- 0x56512c4c1a60 con 0x56512c55ed10 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.200+0000 7f856e59c640 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x56512c55ed10 0x56512c55f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.200+0000 7f854d7fa640 10 client.6726 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.200+0000 7f856e59c640 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x56512c55ed10 0x56512c55f120 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e73e4617b614d0 server_cookie=4fb930384f363235 in_seq=0 out_seq=0 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f854d7fa640 1 -- 172.21.15.44:0/810368182 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56512c4c1a60 con 0x56512c55ed10 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f854d7fa640 10 client.6726 handle_command_reply: tid=0 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8577a36740 1 client.6726 shutdown 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8577a36740 1 -- 172.21.15.44:0/810368182 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 msgr2=0x56512c4e6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8577a36740 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 0x56512c4e6680 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f856805a300 tx=0x7f856805e730 comp rx=0 tx=0).stop 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8577a36740 1 -- 172.21.15.44:0/810368182 shutdown_connections 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8577a36740 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x56512c55ed10 0x56512c55f120 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8577a36740 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56512c4c7540 0x56512c4e6680 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8577a36740 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56512c4c7160 0x56512c4e6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f854d7fa640 0 client.6726 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:05.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8577a36740 1 --2- 172.21.15.44:0/810368182 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56512c4c01f0 0x56512c4c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8577a36740 1 -- 172.21.15.44:0/810368182 >> 172.21.15.44:0/810368182 conn(0x56512c49a640 msgr2=0x56512c4cdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:05.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8577a36740 1 -- 172.21.15.44:0/810368182 shutdown_connections 2024-04-02T13:24:05.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8577a36740 1 -- 172.21.15.44:0/810368182 wait complete. 2024-04-02T13:24:05.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8577a36740 20 client.6726 trim_cache size 0 max 16384 2024-04-02T13:24:05.210 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:24:05.201+0000 7fb859322640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v4 ap=1 DIRTYPARENT s=0 n(v0 rc2024-04-02T13:23:32.807273+0000 1=1+0) caps={6603=pAsLsXsFs/-@2},l=6603(-1) | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 scrubqueue=0 0x55dcba8b1600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v4>,<0x1/dir_test v7>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-02T13:24:05.210 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:24:05.211 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-02T13:24:05.211 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "c6358bd4-edb4-483c-9b82-f098c3b18271", 2024-04-02T13:24:05.211 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-02T13:24:05.211 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:24:05.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8574d7c640 1 -- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f855006cac0 msgr2=0x7f855006ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8574d7c640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f855006cac0 0x7f855006ef80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8560034000 tx=0x7f8560009030 comp rx=0 tx=0).stop 2024-04-02T13:24:05.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8574d7c640 1 -- 172.21.15.44:0/4132865821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85701f71f0 msgr2=0x7f8570156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.201+0000 7f8574d7c640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85701f71f0 0x7f8570156ef0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8570073280 tx=0x7f856800f580 comp rx=0 tx=0).stop 2024-04-02T13:24:05.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.202+0000 7f8574d7c640 1 -- 172.21.15.44:0/4132865821 shutdown_connections 2024-04-02T13:24:05.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.202+0000 7f8574d7c640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f855006cac0 0x7f855006ef80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.202+0000 7f8574d7c640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85701f71f0 0x7f8570156ef0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.202+0000 7f8574d7c640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85701544e0 0x7f85701569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.202+0000 7f8574d7c640 1 --2- 172.21.15.44:0/4132865821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c0039b0 0x7f8570153520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.202+0000 7f8574d7c640 1 -- 172.21.15.44:0/4132865821 >> 172.21.15.44:0/4132865821 conn(0x7f857009aa20 msgr2=0x7f85700b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:05.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.202+0000 7f8574d7c640 1 -- 172.21.15.44:0/4132865821 shutdown_connections 2024-04-02T13:24:05.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.202+0000 7f8574d7c640 1 -- 172.21.15.44:0/4132865821 wait complete. 2024-04-02T13:24:05.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:24:05.215 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-04-02T13:24:05.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.317+0000 7f3fd1eeb640 1 Processor -- start 2024-04-02T13:24:05.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.317+0000 7f3fd1eeb640 1 -- start start 2024-04-02T13:24:05.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fd1eeb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3fcc0b4180 0x7f3fcc168f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fd1eeb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fcc0d1630 con 0x7f3fcc06cff0 2024-04-02T13:24:05.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fd1eeb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fcc0d1910 con 0x7f3fcc15e5d0 2024-04-02T13:24:05.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fd1eeb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3fcc0d1bf0 con 0x7f3fcc0b4180 2024-04-02T13:24:05.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fcb7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f3fcc06cff0 0x7f3fcc06d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56600/0 (socket says 172.21.15.44:56600) 2024-04-02T13:24:05.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/552508897 learned_addr learned my addr 172.21.15.44:0/552508897 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:05.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2319282300 0 0) 0x7f3fcc0d1630 con 0x7f3fcc06cff0 2024-04-02T13:24:05.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fb4003520 con 0x7f3fcc06cff0 2024-04-02T13:24:05.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/552508897 >> v2:172.21.15.69:3300/0 conn(0x7f3fcc0b4180 0x7f3fcc168f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1791693453 0 0) 0x7f3fb4003520 con 0x7f3fcc06cff0 2024-04-02T13:24:05.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 >> v2:172.21.15.69:3300/0 conn(0x7f3fcc0b4180 msgr2=0x7f3fcc168f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fca7fc640 1 --2- 172.21.15.44:0/552508897 >> v2:172.21.15.69:3300/0 conn(0x7f3fcc0b4180 0x7f3fcc168f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 >> v1:172.21.15.69:6789/0 conn(0x7f3fcc15e5d0 legacy=0x7f3fcc0d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fcc0b1540 con 0x7f3fcc06cff0 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f3fbc001670 con 0x7f3fcc06cff0 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 >> v1:172.21.15.44:6789/0 conn(0x7f3fcc06cff0 legacy=0x7f3fcc06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fca7fc640 1 --2- 172.21.15.44:0/552508897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fb40039b0 0x7f3fb4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.318+0000 7f3fca7fc640 1 --2- 172.21.15.44:0/552508897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fb4004800 0x7f3fb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.319+0000 7f3fca7fc640 1 --2- 172.21.15.44:0/552508897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fb40051c0 0x7f3fb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.319+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc0d1bf0 con 0x7f3fb40039b0 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.319+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb400c160 con 0x7f3fb4004800 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.319+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3fb400c360 con 0x7f3fb40051c0 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.319+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/552508897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fb40039b0 0x7f3fb4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.319+0000 7f3fcaffd640 1 --2- 172.21.15.44:0/552508897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fb4004800 0x7f3fb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.319+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/552508897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fb40051c0 0x7f3fb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.319+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/552508897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fb40051c0 msgr2=0x7f3fb400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.319+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/552508897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fb40051c0 0x7f3fb400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.319+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/552508897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fb4004800 msgr2=0x7f3fb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.319+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/552508897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fb4004800 0x7f3fb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.319+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/552508897 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb4003520 con 0x7f3fb40039b0 2024-04-02T13:24:05.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.319+0000 7f3fcbfff640 1 -- 172.21.15.44:0/552508897 reap_dead start 2024-04-02T13:24:05.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.320+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/552508897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fb40039b0 0x7f3fb4003dd0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00c240 tx=0x7f3fb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64426fe4fa783648 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.320+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc007d10 con 0x7f3fb40039b0 2024-04-02T13:24:05.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.320+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fbc030070 con 0x7f3fb40039b0 2024-04-02T13:24:05.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.320+0000 7f3fca7fc640 1 -- 172.21.15.44:0/552508897 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc005d90 con 0x7f3fb40039b0 2024-04-02T13:24:05.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.320+0000 7f3fd1eeb640 1 -- 172.21.15.44:0/552508897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fb40039b0 msgr2=0x7f3fb4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.320+0000 7f3fd1eeb640 1 --2- 172.21.15.44:0/552508897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fb40039b0 0x7f3fb4003dd0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00c240 tx=0x7f3fb4005830 comp rx=0 tx=0).stop 2024-04-02T13:24:05.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.320+0000 7f3fd1eeb640 1 -- 172.21.15.44:0/552508897 shutdown_connections 2024-04-02T13:24:05.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.320+0000 7f3fd1eeb640 1 --2- 172.21.15.44:0/552508897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fb40039b0 0x7f3fb4003dd0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.320+0000 7f3fd1eeb640 1 -- 172.21.15.44:0/552508897 >> 172.21.15.44:0/552508897 conn(0x7f3fcc09aa20 msgr2=0x7f3fcc09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:05.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.320+0000 7f3fd1eeb640 1 -- 172.21.15.44:0/552508897 shutdown_connections 2024-04-02T13:24:05.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.320+0000 7f3fd1eeb640 1 -- 172.21.15.44:0/552508897 wait complete. 2024-04-02T13:24:05.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.322+0000 7f3fd1eeb640 1 Processor -- start 2024-04-02T13:24:05.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.322+0000 7f3fd1eeb640 1 -- start start 2024-04-02T13:24:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.322+0000 7f3fd1eeb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fb40039b0 0x7f3fcc1f76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.322+0000 7f3fd1eeb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fcc0b4700 0x7f3fcc0c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.322+0000 7f3fd1eeb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fcc0b4b80 0x7f3fcc0c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.322+0000 7f3fd1eeb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc06d550 con 0x7f3fcc0b4b80 2024-04-02T13:24:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.322+0000 7f3fd1eeb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc06d3d0 con 0x7f3fcc0b4700 2024-04-02T13:24:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.322+0000 7f3fd1eeb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3fcc06d6d0 con 0x7f3fb40039b0 2024-04-02T13:24:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fcbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fcc0b4b80 0x7f3fcc0c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fcbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fcc0b4b80 0x7f3fcc0c7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39390/0 (socket says 172.21.15.44:39390) 2024-04-02T13:24:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fcbfff640 1 -- 172.21.15.44:0/714810403 learned_addr learned my addr 172.21.15.44:0/714810403 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fcaffd640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fcc0b4700 0x7f3fcc0c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fb40039b0 0x7f3fcc1f76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fcbfff640 1 -- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fb40039b0 msgr2=0x7f3fcc1f76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fb40039b0 0x7f3fcc1f76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fcbfff640 1 -- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fcc0b4700 msgr2=0x7f3fcc0c7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fcc0b4700 0x7f3fcc0c7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fcbfff640 1 -- 172.21.15.44:0/714810403 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fcc1605c0 con 0x7f3fcc0b4b80 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fb40039b0 0x7f3fcc1f76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fcc0b4b80 0x7f3fcc0c7a60 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc073280 tx=0x7f3fc400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fb3fff640 1 -- 172.21.15.44:0/714810403 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc400fdb0 con 0x7f3fcc0b4b80 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fb3fff640 1 -- 172.21.15.44:0/714810403 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fc401c070 con 0x7f3fcc0b4b80 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fd1eeb640 1 -- 172.21.15.44:0/714810403 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fcc160880 con 0x7f3fcc0b4b80 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fb3fff640 1 -- 172.21.15.44:0/714810403 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4003bd0 con 0x7f3fcc0b4b80 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.323+0000 7f3fd1eeb640 1 -- 172.21.15.44:0/714810403 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fcc160e40 con 0x7f3fcc0b4b80 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.325+0000 7f3fb3fff640 1 -- 172.21.15.44:0/714810403 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3fc4004030 con 0x7f3fcc0b4b80 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.325+0000 7f3fb3fff640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fb806c8f0 0x7f3fb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.326+0000 7f3fb3fff640 1 -- 172.21.15.44:0/714810403 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f3fc408c620 con 0x7f3fcc0b4b80 2024-04-02T13:24:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.326+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fb806c8f0 0x7f3fb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.326+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fb806c8f0 0x7f3fb806edb0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc034000 tx=0x7f3fbc009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.326+0000 7f3fd4bad740 1 Processor -- start 2024-04-02T13:24:05.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.326+0000 7f3fd4bad740 1 -- start start 2024-04-02T13:24:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.327+0000 7f3fd4bad740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a170239a0 0x564a16ff6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.327+0000 7f3fd4bad740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a17023d80 0x564a17026730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.327+0000 7f3fd4bad740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a16ff6e80 0x564a17026c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.327+0000 7f3fd4bad740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564a16f60830 con 0x564a17023d80 2024-04-02T13:24:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.327+0000 7f3fd4bad740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564a16f08820 con 0x564a170239a0 2024-04-02T13:24:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.327+0000 7f3fd4bad740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564a16f5c7e0 con 0x564a16ff6e80 2024-04-02T13:24:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.327+0000 7f3fcaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a17023d80 0x564a17026730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.327+0000 7f3fcaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a17023d80 0x564a17026730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39400/0 (socket says 172.21.15.44:39400) 2024-04-02T13:24:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.327+0000 7f3fcaffd640 1 -- 172.21.15.44:0/1831169276 learned_addr learned my addr 172.21.15.44:0/1831169276 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.327+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/1831169276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a16ff6e80 0x564a17026c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.327+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/1831169276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a170239a0 0x564a16ff6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.328+0000 7f3fcaffd640 1 -- 172.21.15.44:0/1831169276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a16ff6e80 msgr2=0x564a17026c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.328+0000 7f3fcaffd640 1 --2- 172.21.15.44:0/1831169276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a16ff6e80 0x564a17026c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.328+0000 7f3fcaffd640 1 -- 172.21.15.44:0/1831169276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a170239a0 msgr2=0x564a16ff6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.328+0000 7f3fcaffd640 1 --2- 172.21.15.44:0/1831169276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a170239a0 0x564a16ff6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.328+0000 7f3fcaffd640 1 -- 172.21.15.44:0/1831169276 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a16d3e7d0 con 0x564a17023d80 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.328+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/1831169276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a16ff6e80 0x564a17026c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.328+0000 7f3fcaffd640 1 --2- 172.21.15.44:0/1831169276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a17023d80 0x564a17026730 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0005540 tx=0x7f3fc0007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43aa5725ba0d4c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.328+0000 7f3fb17fa640 1 -- 172.21.15.44:0/1831169276 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc001b070 con 0x564a17023d80 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.328+0000 7f3fb17fa640 1 -- 172.21.15.44:0/1831169276 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fc0001720 con 0x564a17023d80 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.328+0000 7f3fb17fa640 1 -- 172.21.15.44:0/1831169276 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0001af0 con 0x564a17023d80 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.329+0000 7f3fd4bad740 1 -- 172.21.15.44:0/1831169276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a17023d80 msgr2=0x564a17026730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.329+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/1831169276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a17023d80 0x564a17026730 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0005540 tx=0x7f3fc0007c80 comp rx=0 tx=0).stop 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.329+0000 7f3fd4bad740 1 -- 172.21.15.44:0/1831169276 shutdown_connections 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.329+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/1831169276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a16ff6e80 0x564a17026c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.329+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/1831169276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a17023d80 0x564a17026730 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.329+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/1831169276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a170239a0 0x564a16ff6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.329+0000 7f3fd4bad740 1 -- 172.21.15.44:0/1831169276 >> 172.21.15.44:0/1831169276 conn(0x564a16ff8bb0 msgr2=0x564a17041730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.329+0000 7f3fd4bad740 1 -- 172.21.15.44:0/1831169276 shutdown_connections 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.329+0000 7f3fd4bad740 1 -- 172.21.15.44:0/1831169276 wait complete. 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.329+0000 7f3fd4bad740 1 -- start start 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.329+0000 7f3fd4bad740 1 Processor -- start 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fd4bad740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 0x564a17014cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fd4bad740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a17023d80 0x564a17028770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fd4bad740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a1703ec10 0x564a1703f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fd4bad740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564a16ffb150 con 0x564a170239a0 2024-04-02T13:24:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fd4bad740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564a16f1bd30 con 0x564a17023d80 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fd4bad740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564a16ffa540 con 0x564a1703ec10 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fd4bad740 10 client.0 _ll_register_callbacks cb 0x564a16f1b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fcb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 0x564a17014cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fcb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 0x564a17014cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39410/0 (socket says 172.21.15.44:39410) 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/3287605745 learned_addr learned my addr 172.21.15.44:0/3287605745 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fcaffd640 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a17023d80 0x564a17028770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a1703ec10 0x564a1703f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/3287605745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a1703ec10 msgr2=0x564a1703f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a1703ec10 0x564a1703f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/3287605745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a17023d80 msgr2=0x564a17028770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.330+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a17023d80 0x564a17028770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/3287605745 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a16c743f0 con 0x564a170239a0 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a1703ec10 0x564a1703f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 0x564a17014cb0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00c330 tx=0x7f3fbc02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c86864715d64b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3faa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:24:05.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3faa7fc640 1 -- 172.21.15.44:0/3287605745 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc004bf0 con 0x564a170239a0 2024-04-02T13:24:05.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3faa7fc640 1 -- 172.21.15.44:0/3287605745 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f3fbc004d90 con 0x564a170239a0 2024-04-02T13:24:05.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3fd4bad740 1 -- 172.21.15.44:0/3287605745 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fc0001720 con 0x564a170239a0 2024-04-02T13:24:05.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3faa7fc640 1 -- 172.21.15.44:0/3287605745 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3fbc007070 con 0x564a170239a0 2024-04-02T13:24:05.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3fd4bad740 10 client.6738 fetch_fsmap learned FSMap version 36 2024-04-02T13:24:05.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3fd4bad740 1 -- 172.21.15.44:0/3287605745 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x564a16c340f0 con 0x564a170239a0 2024-04-02T13:24:05.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3faa7fc640 1 -- 172.21.15.44:0/3287605745 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fc0001720 con 0x564a170239a0 2024-04-02T13:24:05.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3faa7fc640 1 -- 172.21.15.44:0/3287605745 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f3fbc055050 con 0x564a170239a0 2024-04-02T13:24:05.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3fd4bad740 10 client.6738 fetch_fsmap finished waiting for FSMap version 36 2024-04-02T13:24:05.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3fd4bad740 10 client.6738 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-04-02T13:24:05.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3fd4bad740 20 client.6738 populate_metadata read hostname 'smithi044' 2024-04-02T13:24:05.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x564a1703d360 0x564a1703c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3fd4bad740 4 client.6738 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3fd4bad740 1 -- 172.21.15.44:0/3287605745 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a16f6dd70 con 0x564a1703d360 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.331+0000 7f3fcaffd640 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x564a1703d360 0x564a1703c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3faa7fc640 10 client.6738 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3fcaffd640 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x564a1703d360 0x564a1703c7b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dafa08b4bebe946d server_cookie=266ffa2c6963c117 in_seq=0 out_seq=0 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3faa7fc640 1 -- 172.21.15.44:0/3287605745 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a16f6dd70 con 0x564a1703d360 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3faa7fc640 10 client.6738 handle_command_reply: tid=0 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3fd4bad740 1 client.6738 shutdown 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3fd4bad740 1 -- 172.21.15.44:0/3287605745 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 msgr2=0x564a17014cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 0x564a17014cb0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00c330 tx=0x7f3fbc02e000 comp rx=0 tx=0).stop 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3fd4bad740 1 -- 172.21.15.44:0/3287605745 shutdown_connections 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x564a1703d360 0x564a1703c7b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a1703ec10 0x564a1703f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a17023d80 0x564a17028770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/3287605745 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 0x564a17014cb0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3faa7fc640 0 client.6738 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:05.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.332+0000 7f3fd4bad740 1 -- 172.21.15.44:0/3287605745 >> 172.21.15.44:0/3287605745 conn(0x564a16ff6e80 msgr2=0x564a17026720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:05.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.333+0000 7f3fd4bad740 1 -- 172.21.15.44:0/3287605745 shutdown_connections 2024-04-02T13:24:05.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.333+0000 7f3fd4bad740 1 -- 172.21.15.44:0/3287605745 wait complete. 2024-04-02T13:24:05.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.333+0000 7f3fd4bad740 20 client.6738 trim_cache size 0 max 16384 2024-04-02T13:24:05.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.350+0000 7f3fd4bad740 1 Processor -- start 2024-04-02T13:24:05.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.350+0000 7f3fd4bad740 1 -- start start 2024-04-02T13:24:05.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fd4bad740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 0x564a17051270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fd4bad740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a17023d80 0x564a170422a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fd4bad740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a1703ec10 0x564a17014ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fd4bad740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc2014d0 con 0x564a170239a0 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fd4bad740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc200510 con 0x564a1703ec10 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fd4bad740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3fcc1ff870 con 0x564a17023d80 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fcb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 0x564a17051270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fcb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 0x564a17051270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39422/0 (socket says 172.21.15.44:39422) 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/2006074607 learned_addr learned my addr 172.21.15.44:0/2006074607 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/2006074607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a1703ec10 0x564a17014ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/2006074607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a17023d80 msgr2=0x564a170422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/2006074607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a17023d80 0x564a170422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/2006074607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a1703ec10 msgr2=0x564a17014ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/2006074607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a1703ec10 0x564a17014ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/2006074607 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a1703fc40 con 0x564a170239a0 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.351+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/2006074607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a1703ec10 0x564a17014ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.352+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/2006074607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 0x564a17051270 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc02c4e0 tx=0x7f3fbc03c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce8a049454712c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.352+0000 7f3fb0ff9640 1 -- 172.21.15.44:0/2006074607 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc007530 con 0x564a170239a0 2024-04-02T13:24:05.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.352+0000 7f3fb0ff9640 1 -- 172.21.15.44:0/2006074607 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fbc030040 con 0x564a170239a0 2024-04-02T13:24:05.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.352+0000 7f3fb0ff9640 1 -- 172.21.15.44:0/2006074607 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc036480 con 0x564a170239a0 2024-04-02T13:24:05.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.352+0000 7f3fd4bad740 1 -- 172.21.15.44:0/2006074607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 msgr2=0x564a17051270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.352+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/2006074607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 0x564a17051270 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc02c4e0 tx=0x7f3fbc03c000 comp rx=0 tx=0).stop 2024-04-02T13:24:05.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.352+0000 7f3fd4bad740 1 -- 172.21.15.44:0/2006074607 shutdown_connections 2024-04-02T13:24:05.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.352+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/2006074607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a1703ec10 0x564a17014ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.352+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/2006074607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a17023d80 0x564a170422a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.352+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/2006074607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a170239a0 0x564a17051270 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.352+0000 7f3fd4bad740 1 -- 172.21.15.44:0/2006074607 >> 172.21.15.44:0/2006074607 conn(0x564a16ff6e80 msgr2=0x564a16ffd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:05.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.352+0000 7f3fd4bad740 1 -- 172.21.15.44:0/2006074607 shutdown_connections 2024-04-02T13:24:05.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.352+0000 7f3fd4bad740 1 -- 172.21.15.44:0/2006074607 wait complete. 2024-04-02T13:24:05.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.353+0000 7f3fd4bad740 1 -- start start 2024-04-02T13:24:05.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.353+0000 7f3fd4bad740 1 Processor -- start 2024-04-02T13:24:05.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fd4bad740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a170239a0 0x564a17028720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fd4bad740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a17023d80 0x564a1703c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fd4bad740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a1703ec10 0x564a1703ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fd4bad740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564a17055bd0 con 0x564a1703ec10 2024-04-02T13:24:05.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fd4bad740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc1ffec0 con 0x564a17023d80 2024-04-02T13:24:05.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fd4bad740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564a16ffbad0 con 0x564a170239a0 2024-04-02T13:24:05.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fd4bad740 10 client.0 _ll_register_callbacks cb 0x7f3fb400bfe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:24:05.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fcbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a1703ec10 0x564a1703ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fcbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a1703ec10 0x564a1703ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39436/0 (socket says 172.21.15.44:39436) 2024-04-02T13:24:05.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fcbfff640 1 -- 172.21.15.44:0/2654027216 learned_addr learned my addr 172.21.15.44:0/2654027216 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:05.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/2654027216 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a170239a0 0x564a17028720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fcaffd640 1 --2- 172.21.15.44:0/2654027216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a17023d80 0x564a1703c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fcbfff640 1 -- 172.21.15.44:0/2654027216 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a170239a0 msgr2=0x564a17028720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/2654027216 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a170239a0 0x564a17028720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fcbfff640 1 -- 172.21.15.44:0/2654027216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a17023d80 msgr2=0x564a1703c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:05.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/2654027216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a17023d80 0x564a1703c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fcbfff640 1 -- 172.21.15.44:0/2654027216 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a16ffb640 con 0x564a1703ec10 2024-04-02T13:24:05.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fcbfff640 1 --2- 172.21.15.44:0/2654027216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a1703ec10 0x564a1703ccf0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40093e0 tx=0x7f3fc405e630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aedae6768f5cae1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:05.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3faa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:24:05.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3faa7fc640 1 -- 172.21.15.44:0/2654027216 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc405e170 con 0x564a1703ec10 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3faa7fc640 1 -- 172.21.15.44:0/2654027216 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f3fc405e310 con 0x564a1703ec10 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.354+0000 7f3fd4bad740 1 -- 172.21.15.44:0/2654027216 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fbc030040 con 0x564a1703ec10 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3faa7fc640 1 -- 172.21.15.44:0/2654027216 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3fc4077050 con 0x564a1703ec10 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3fd4bad740 10 client.6744 fetch_fsmap learned FSMap version 36 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3fd4bad740 1 -- 172.21.15.44:0/2654027216 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x564a16ffb850 con 0x564a1703ec10 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3faa7fc640 1 -- 172.21.15.44:0/2654027216 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fbc030040 con 0x564a1703ec10 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3faa7fc640 1 -- 172.21.15.44:0/2654027216 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f3fc406ebd0 con 0x564a1703ec10 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3fd4bad740 10 client.6744 fetch_fsmap finished waiting for FSMap version 36 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3fd4bad740 10 client.6744 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3fd4bad740 20 client.6744 populate_metadata read hostname 'smithi044' 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/2654027216 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x564a170647c0 0x564a17064ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3fd4bad740 4 client.6744 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3fd4bad740 1 -- 172.21.15.44:0/2654027216 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564a1703fe10 con 0x564a170647c0 2024-04-02T13:24:05.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/2654027216 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x564a170647c0 0x564a17064ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3faa7fc640 10 client.6744 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.355+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/2654027216 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x564a170647c0 0x564a17064ba0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4f1851e44f291a5 server_cookie=549346304f7e78df in_seq=0 out_seq=0 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3faa7fc640 1 -- 172.21.15.44:0/2654027216 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x564a1703fe10 con 0x564a170647c0 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3faa7fc640 10 client.6744 handle_command_reply: tid=0 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd4bad740 1 client.6744 shutdown 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd4bad740 1 -- 172.21.15.44:0/2654027216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a1703ec10 msgr2=0x564a1703ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/2654027216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a1703ec10 0x564a1703ccf0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40093e0 tx=0x7f3fc405e630 comp rx=0 tx=0).stop 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd4bad740 1 -- 172.21.15.44:0/2654027216 shutdown_connections 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/2654027216 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x564a170647c0 0x564a17064ba0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/2654027216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564a1703ec10 0x564a1703ccf0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/2654027216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564a17023d80 0x564a1703c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3faa7fc640 0 client.6744 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd4bad740 1 --2- 172.21.15.44:0/2654027216 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564a170239a0 0x564a17028720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd4bad740 1 -- 172.21.15.44:0/2654027216 >> 172.21.15.44:0/2654027216 conn(0x564a16ff6e80 msgr2=0x564a16ff20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:05.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd4bad740 1 -- 172.21.15.44:0/2654027216 shutdown_connections 2024-04-02T13:24:05.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd4bad740 1 -- 172.21.15.44:0/2654027216 wait complete. 2024-04-02T13:24:05.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd4bad740 20 client.6744 trim_cache size 0 max 16384 2024-04-02T13:24:05.365 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:24:05.365 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T13:24:05.365 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-02T13:24:05.365 INFO:teuthology.orchestra.run.smithi044.stdout: "c6358bd4-edb4-483c-9b82-f098c3b18271": { 2024-04-02T13:24:05.365 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-02T13:24:05.365 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "c6358bd4-edb4-483c-9b82-f098c3b18271", 2024-04-02T13:24:05.365 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,repair,force" 2024-04-02T13:24:05.365 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:05.366 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:05.366 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:24:05.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd1eeb640 1 -- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fb806c8f0 msgr2=0x7f3fb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd1eeb640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fb806c8f0 0x7f3fb806edb0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc034000 tx=0x7f3fbc009030 comp rx=0 tx=0).stop 2024-04-02T13:24:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd1eeb640 1 -- 172.21.15.44:0/714810403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fcc0b4b80 msgr2=0x7f3fcc0c7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd1eeb640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fcc0b4b80 0x7f3fcc0c7a60 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc073280 tx=0x7f3fc400f580 comp rx=0 tx=0).stop 2024-04-02T13:24:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd1eeb640 1 -- 172.21.15.44:0/714810403 shutdown_connections 2024-04-02T13:24:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd1eeb640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fb806c8f0 0x7f3fb806edb0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd1eeb640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fcc0b4b80 0x7f3fcc0c7a60 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd1eeb640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fcc0b4700 0x7f3fcc0c7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd1eeb640 1 --2- 172.21.15.44:0/714810403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fb40039b0 0x7f3fcc1f76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd1eeb640 1 -- 172.21.15.44:0/714810403 >> 172.21.15.44:0/714810403 conn(0x7f3fcc09aa20 msgr2=0x7f3fcc0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.356+0000 7f3fd1eeb640 1 -- 172.21.15.44:0/714810403 shutdown_connections 2024-04-02T13:24:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:05.357+0000 7f3fd1eeb640 1 -- 172.21.15.44:0/714810403 wait complete. 2024-04-02T13:24:05.370 INFO:tasks.cephfs.filesystem:scrub status for tag:c6358bd4-edb4-483c-9b82-f098c3b18271 - {'path': '/', 'tag': 'c6358bd4-edb4-483c-9b82-f098c3b18271', 'options': 'recursive,repair,force'} 2024-04-02T13:24:22.401 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:24:22.405 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:24:35.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:24:35.371 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-04-02T13:24:35.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.485+0000 7f5bad44e640 1 Processor -- start 2024-04-02T13:24:35.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.485+0000 7f5bad44e640 1 -- start start 2024-04-02T13:24:35.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.486+0000 7f5bad44e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5ba815e420 0x7f5ba80ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.486+0000 7f5bad44e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ba80721f0 0x7f5ba80cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.486+0000 7f5bad44e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ba806c220 con 0x7f5ba80cfc40 2024-04-02T13:24:35.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.486+0000 7f5bad44e640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f5ba80d00b0 con 0x7f5ba815e420 2024-04-02T13:24:35.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.486+0000 7f5bad44e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5ba815f7e0 con 0x7f5ba80721f0 2024-04-02T13:24:35.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.486+0000 7f5ba6ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5ba815e420 0x7f5ba80ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.486+0000 7f5ba77fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f5ba80cfc40 0x7f5ba8161eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48042/0 (socket says 172.21.15.44:48042) 2024-04-02T13:24:35.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.486+0000 7f5ba77fe640 1 -- 172.21.15.44:0/2265905485 learned_addr learned my addr 172.21.15.44:0/2265905485 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:35.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.486+0000 7f5ba67fc640 1 --2- 172.21.15.44:0/2265905485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ba80721f0 0x7f5ba80cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.486+0000 7f5ba6ffd640 1 -- 172.21.15.44:0/2265905485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ba80721f0 msgr2=0x7f5ba80cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:35.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.486+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/2265905485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ba80721f0 0x7f5ba80cf700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.486+0000 7f5ba6ffd640 1 -- 172.21.15.44:0/2265905485 >> v1:172.21.15.44:6789/0 conn(0x7f5ba80cfc40 legacy=0x7f5ba8161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.486+0000 7f5ba6ffd640 1 -- 172.21.15.44:0/2265905485 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba815ec60 con 0x7f5ba815e420 2024-04-02T13:24:35.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.487+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/2265905485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba815e420 0x7f5ba80ceca0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5ba00097a0 tx=0x7f5ba002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df29abf4d5220d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:35.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.487+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/2265905485 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0038070 con 0x7f5ba815e420 2024-04-02T13:24:35.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.487+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/2265905485 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba0031700 con 0x7f5ba815e420 2024-04-02T13:24:35.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.487+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/2265905485 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0031ad0 con 0x7f5ba815e420 2024-04-02T13:24:35.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.488+0000 7f5bad44e640 1 -- 172.21.15.44:0/2265905485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba815e420 msgr2=0x7f5ba80ceca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.488+0000 7f5bad44e640 1 --2- 172.21.15.44:0/2265905485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba815e420 0x7f5ba80ceca0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5ba00097a0 tx=0x7f5ba002b7a0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.488+0000 7f5bad44e640 1 -- 172.21.15.44:0/2265905485 shutdown_connections 2024-04-02T13:24:35.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.488+0000 7f5bad44e640 1 --2- 172.21.15.44:0/2265905485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ba80721f0 0x7f5ba80cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.488+0000 7f5bad44e640 1 --2- 172.21.15.44:0/2265905485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba815e420 0x7f5ba80ceca0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.488+0000 7f5bad44e640 1 -- 172.21.15.44:0/2265905485 >> 172.21.15.44:0/2265905485 conn(0x7f5ba80b2a10 msgr2=0x7f5ba80b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:35.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.488+0000 7f5bad44e640 1 -- 172.21.15.44:0/2265905485 shutdown_connections 2024-04-02T13:24:35.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.488+0000 7f5bad44e640 1 -- 172.21.15.44:0/2265905485 wait complete. 2024-04-02T13:24:35.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.489+0000 7f5bad44e640 1 Processor -- start 2024-04-02T13:24:35.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.489+0000 7f5bad44e640 1 -- start start 2024-04-02T13:24:35.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.490+0000 7f5bad44e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ba80721f0 0x7f5ba81463c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.490+0000 7f5bad44e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ba80cfc40 0x7f5ba8159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.490+0000 7f5bad44e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba815e420 0x7f5ba815a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.490+0000 7f5bad44e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba815f500 con 0x7f5ba815e420 2024-04-02T13:24:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.490+0000 7f5bad44e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba815f220 con 0x7f5ba80721f0 2024-04-02T13:24:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.490+0000 7f5bad44e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5ba80cf1e0 con 0x7f5ba80cfc40 2024-04-02T13:24:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.490+0000 7f5ba77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba815e420 0x7f5ba815a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.490+0000 7f5ba77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba815e420 0x7f5ba815a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47400/0 (socket says 172.21.15.44:47400) 2024-04-02T13:24:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.490+0000 7f5ba77fe640 1 -- 172.21.15.44:0/1123748347 learned_addr learned my addr 172.21.15.44:0/1123748347 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.490+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ba80721f0 0x7f5ba81463c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.490+0000 7f5ba67fc640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ba80cfc40 0x7f5ba8159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.491+0000 7f5ba77fe640 1 -- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ba80cfc40 msgr2=0x7f5ba8159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.491+0000 7f5ba77fe640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ba80cfc40 0x7f5ba8159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.491+0000 7f5ba77fe640 1 -- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ba80721f0 msgr2=0x7f5ba81463c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.491+0000 7f5ba77fe640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ba80721f0 0x7f5ba81463c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.491+0000 7f5ba77fe640 1 -- 172.21.15.44:0/1123748347 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba815c2b0 con 0x7f5ba815e420 2024-04-02T13:24:35.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.491+0000 7f5ba67fc640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ba80cfc40 0x7f5ba8159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:35.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.491+0000 7f5ba77fe640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba815e420 0x7f5ba815a7b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800cbf0 tx=0x7f5b9800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:35.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.491+0000 7f5b9f7fe640 1 -- 172.21.15.44:0/1123748347 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b98017070 con 0x7f5ba815e420 2024-04-02T13:24:35.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.491+0000 7f5b9f7fe640 1 -- 172.21.15.44:0/1123748347 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b9800ad40 con 0x7f5ba815e420 2024-04-02T13:24:35.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.491+0000 7f5bad44e640 1 -- 172.21.15.44:0/1123748347 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ba815c590 con 0x7f5ba815e420 2024-04-02T13:24:35.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.491+0000 7f5b9f7fe640 1 -- 172.21.15.44:0/1123748347 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b98012600 con 0x7f5ba815e420 2024-04-02T13:24:35.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.491+0000 7f5bad44e640 1 -- 172.21.15.44:0/1123748347 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ba815cb50 con 0x7f5ba815e420 2024-04-02T13:24:35.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.492+0000 7f5b9f7fe640 1 -- 172.21.15.44:0/1123748347 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5b9801a070 con 0x7f5ba815e420 2024-04-02T13:24:35.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.493+0000 7f5b9f7fe640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b8806c920 0x7f5b8806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.493+0000 7f5b9f7fe640 1 -- 172.21.15.44:0/1123748347 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f5b9808ba70 con 0x7f5ba815e420 2024-04-02T13:24:35.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.493+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b8806c920 0x7f5b8806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.494+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b8806c920 0x7f5b8806ede0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0035000 tx=0x7f5ba0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:35.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.494+0000 7f5bb003f740 1 Processor -- start 2024-04-02T13:24:35.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.494+0000 7f5bb003f740 1 -- start start 2024-04-02T13:24:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5bb003f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 0x55ebf9be7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5bb003f740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c14d80 0x55ebf9c17730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5bb003f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9be7e80 0x55ebf9c17c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5bb003f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ebf9b51830 con 0x55ebf9c149a0 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5bb003f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ebf9af9820 con 0x55ebf9be7e80 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5bb003f740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ebf9b4d7e0 con 0x55ebf9c14d80 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5ba6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 0x55ebf9be7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5ba6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 0x55ebf9be7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47412/0 (socket says 172.21.15.44:47412) 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5ba6ffd640 1 -- 172.21.15.44:0/1825049621 learned_addr learned my addr 172.21.15.44:0/1825049621 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5ba67fc640 1 --2- 172.21.15.44:0/1825049621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c14d80 0x55ebf9c17730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5ba77fe640 1 --2- 172.21.15.44:0/1825049621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9be7e80 0x55ebf9c17c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5ba6ffd640 1 -- 172.21.15.44:0/1825049621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c14d80 msgr2=0x55ebf9c17730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/1825049621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c14d80 0x55ebf9c17730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5ba6ffd640 1 -- 172.21.15.44:0/1825049621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9be7e80 msgr2=0x55ebf9c17c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/1825049621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9be7e80 0x55ebf9c17c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.495+0000 7f5ba6ffd640 1 -- 172.21.15.44:0/1825049621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebf992f7d0 con 0x55ebf9c149a0 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.496+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/1825049621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 0x55ebf9be7940 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0009890 tx=0x7f5ba0030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c766045ad7baeeae server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.496+0000 7f5b9cff9640 1 -- 172.21.15.44:0/1825049621 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba004d070 con 0x55ebf9c149a0 2024-04-02T13:24:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.496+0000 7f5b9cff9640 1 -- 172.21.15.44:0/1825049621 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba003d6d0 con 0x55ebf9c149a0 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.496+0000 7f5b9cff9640 1 -- 172.21.15.44:0/1825049621 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0038040 con 0x55ebf9c149a0 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.496+0000 7f5bb003f740 1 -- 172.21.15.44:0/1825049621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 msgr2=0x55ebf9be7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.496+0000 7f5bb003f740 1 --2- 172.21.15.44:0/1825049621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 0x55ebf9be7940 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0009890 tx=0x7f5ba0030000 comp rx=0 tx=0).stop 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.496+0000 7f5bb003f740 1 -- 172.21.15.44:0/1825049621 shutdown_connections 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.496+0000 7f5bb003f740 1 --2- 172.21.15.44:0/1825049621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9be7e80 0x55ebf9c17c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.496+0000 7f5bb003f740 1 --2- 172.21.15.44:0/1825049621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c14d80 0x55ebf9c17730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.496+0000 7f5bb003f740 1 --2- 172.21.15.44:0/1825049621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 0x55ebf9be7940 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.496+0000 7f5bb003f740 1 -- 172.21.15.44:0/1825049621 >> 172.21.15.44:0/1825049621 conn(0x55ebf9be9bb0 msgr2=0x55ebf9c32730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.496+0000 7f5bb003f740 1 -- 172.21.15.44:0/1825049621 shutdown_connections 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.496+0000 7f5bb003f740 1 -- 172.21.15.44:0/1825049621 wait complete. 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.497+0000 7f5bb003f740 1 -- start start 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.497+0000 7f5bb003f740 1 Processor -- start 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5bb003f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c149a0 0x55ebf9c01eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5bb003f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 0x55ebf9c1a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5bb003f740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c423f0 0x55ebf9c42870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5bb003f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ebf9bec150 con 0x55ebf9c14d80 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5bb003f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ebf9b0cd30 con 0x55ebf9c149a0 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5bb003f740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ebf9beb540 con 0x55ebf9c423f0 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5bb003f740 10 client.0 _ll_register_callbacks cb 0x55ebf9b0c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5ba67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 0x55ebf9c1a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5ba67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 0x55ebf9c1a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47428/0 (socket says 172.21.15.44:47428) 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5ba67fc640 1 -- 172.21.15.44:0/1354659128 learned_addr learned my addr 172.21.15.44:0/1354659128 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c149a0 0x55ebf9c01eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5ba77fe640 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c423f0 0x55ebf9c42870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5ba67fc640 1 -- 172.21.15.44:0/1354659128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c423f0 msgr2=0x55ebf9c42870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5ba67fc640 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c423f0 0x55ebf9c42870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5ba67fc640 1 -- 172.21.15.44:0/1354659128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c149a0 msgr2=0x55ebf9c01eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5ba67fc640 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c149a0 0x55ebf9c01eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5ba67fc640 1 -- 172.21.15.44:0/1354659128 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebf98653f0 con 0x55ebf9c14d80 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5ba77fe640 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c423f0 0x55ebf9c42870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c149a0 0x55ebf9c01eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5ba67fc640 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 0x55ebf9c1a180 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f5b90012380 tx=0x7f5b90012840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471cf06fdd67de45 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.498+0000 7f5b85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5b85ffb640 1 -- 172.21.15.44:0/1354659128 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b90019070 con 0x55ebf9c14d80 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5b85ffb640 1 -- 172.21.15.44:0/1354659128 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f5b90004300 con 0x55ebf9c14d80 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5bb003f740 1 -- 172.21.15.44:0/1354659128 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ba003d6d0 con 0x55ebf9c14d80 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5b85ffb640 1 -- 172.21.15.44:0/1354659128 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5b90024050 con 0x55ebf9c14d80 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5bb003f740 10 client.6759 fetch_fsmap learned FSMap version 37 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5bb003f740 1 -- 172.21.15.44:0/1354659128 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x55ebf98250f0 con 0x55ebf9c14d80 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5b85ffb640 1 -- 172.21.15.44:0/1354659128 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba003d6d0 con 0x55ebf9c14d80 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5b85ffb640 1 -- 172.21.15.44:0/1354659128 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 37) ==== 3494+0+0 (secure 0 0 0) 0x7f5b90027020 con 0x55ebf9c14d80 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5bb003f740 10 client.6759 fetch_fsmap finished waiting for FSMap version 37 2024-04-02T13:24:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5bb003f740 10 client.6759 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5bb003f740 20 client.6759 populate_metadata read hostname 'smithi044' 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5bb003f740 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55ebf9c44350 0x55ebf9c44790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5bb003f740 4 client.6759 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5bb003f740 1 -- 172.21.15.44:0/1354659128 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebf9b5ed70 con 0x55ebf9c44350 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.499+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55ebf9c44350 0x55ebf9c44790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5b85ffb640 10 client.6759 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55ebf9c44350 0x55ebf9c44790 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57590356cc71dc58 server_cookie=8201c033f9de7bee in_seq=0 out_seq=0 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5b85ffb640 1 -- 172.21.15.44:0/1354659128 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ebf9b5ed70 con 0x55ebf9c44350 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5b85ffb640 10 client.6759 handle_command_reply: tid=0 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5bb003f740 1 client.6759 shutdown 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5bb003f740 1 -- 172.21.15.44:0/1354659128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 msgr2=0x55ebf9c1a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5bb003f740 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 0x55ebf9c1a180 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f5b90012380 tx=0x7f5b90012840 comp rx=0 tx=0).stop 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5bb003f740 1 -- 172.21.15.44:0/1354659128 shutdown_connections 2024-04-02T13:24:35.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5bb003f740 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55ebf9c44350 0x55ebf9c44790 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5bb003f740 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c423f0 0x55ebf9c42870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5bb003f740 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 0x55ebf9c1a180 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5bb003f740 1 --2- 172.21.15.44:0/1354659128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c149a0 0x55ebf9c01eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5b85ffb640 0 client.6759 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.500+0000 7f5bb003f740 1 -- 172.21.15.44:0/1354659128 >> 172.21.15.44:0/1354659128 conn(0x55ebf9be7e80 msgr2=0x55ebf9bec680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.501+0000 7f5bb003f740 1 -- 172.21.15.44:0/1354659128 shutdown_connections 2024-04-02T13:24:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.501+0000 7f5bb003f740 1 -- 172.21.15.44:0/1354659128 wait complete. 2024-04-02T13:24:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.501+0000 7f5bb003f740 20 client.6759 trim_cache size 0 max 16384 2024-04-02T13:24:35.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.519+0000 7f5bb003f740 1 Processor -- start 2024-04-02T13:24:35.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.519+0000 7f5bb003f740 1 -- start start 2024-04-02T13:24:35.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.519+0000 7f5bb003f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 0x55ebf9c2dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.519+0000 7f5bb003f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c14d80 0x55ebf9beb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.519+0000 7f5bb003f740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c423f0 0x55ebf9c19730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.519+0000 7f5bb003f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ebf9bee070 con 0x55ebf9c149a0 2024-04-02T13:24:35.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.519+0000 7f5bb003f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba8201140 con 0x55ebf9c14d80 2024-04-02T13:24:35.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.519+0000 7f5bb003f740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5ba8155140 con 0x55ebf9c423f0 2024-04-02T13:24:35.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.519+0000 7f5ba6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 0x55ebf9c2dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.519+0000 7f5ba6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 0x55ebf9c2dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47430/0 (socket says 172.21.15.44:47430) 2024-04-02T13:24:35.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.519+0000 7f5ba6ffd640 1 -- 172.21.15.44:0/785274897 learned_addr learned my addr 172.21.15.44:0/785274897 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:35.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5ba67fc640 1 --2- 172.21.15.44:0/785274897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c14d80 0x55ebf9beb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5ba6ffd640 1 -- 172.21.15.44:0/785274897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c423f0 msgr2=0x55ebf9c19730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:35.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5ba77fe640 1 --2- 172.21.15.44:0/785274897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c423f0 0x55ebf9c19730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/785274897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c423f0 0x55ebf9c19730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5ba6ffd640 1 -- 172.21.15.44:0/785274897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c14d80 msgr2=0x55ebf9beb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:35.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/785274897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c14d80 0x55ebf9beb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5ba6ffd640 1 -- 172.21.15.44:0/785274897 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebf9be38d0 con 0x55ebf9c149a0 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5ba77fe640 1 --2- 172.21.15.44:0/785274897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c423f0 0x55ebf9c19730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/785274897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 0x55ebf9c2dc30 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0036040 tx=0x7f5ba003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed9a8db4c5341988 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5b87fff640 1 -- 172.21.15.44:0/785274897 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0065070 con 0x55ebf9c149a0 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5b87fff640 1 -- 172.21.15.44:0/785274897 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba003d8e0 con 0x55ebf9c149a0 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5b87fff640 1 -- 172.21.15.44:0/785274897 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0041040 con 0x55ebf9c149a0 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5bb003f740 1 -- 172.21.15.44:0/785274897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 msgr2=0x55ebf9c2dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5bb003f740 1 --2- 172.21.15.44:0/785274897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 0x55ebf9c2dc30 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0036040 tx=0x7f5ba003e000 comp rx=0 tx=0).stop 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5bb003f740 1 -- 172.21.15.44:0/785274897 shutdown_connections 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5bb003f740 1 --2- 172.21.15.44:0/785274897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c423f0 0x55ebf9c19730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5bb003f740 1 --2- 172.21.15.44:0/785274897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c14d80 0x55ebf9beb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5bb003f740 1 --2- 172.21.15.44:0/785274897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c149a0 0x55ebf9c2dc30 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.520+0000 7f5bb003f740 1 -- 172.21.15.44:0/785274897 >> 172.21.15.44:0/785274897 conn(0x55ebf9be7e80 msgr2=0x55ebf9c01cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.521+0000 7f5bb003f740 1 -- 172.21.15.44:0/785274897 shutdown_connections 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.521+0000 7f5bb003f740 1 -- 172.21.15.44:0/785274897 wait complete. 2024-04-02T13:24:35.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.521+0000 7f5bb003f740 1 -- start start 2024-04-02T13:24:35.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.521+0000 7f5bb003f740 1 Processor -- start 2024-04-02T13:24:35.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5bb003f740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c149a0 0x55ebf9cabcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5bb003f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 0x55ebf9c44940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5bb003f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c423f0 0x55ebf9c32750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5bb003f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba8154a70 con 0x55ebf9c14d80 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5bb003f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba815dac0 con 0x55ebf9c423f0 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5bb003f740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ebf9cac750 con 0x55ebf9c149a0 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5bb003f740 10 client.0 _ll_register_callbacks cb 0x7f5b9000bc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5ba67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 0x55ebf9c44940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5ba67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 0x55ebf9c44940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47434/0 (socket says 172.21.15.44:47434) 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5ba67fc640 1 -- 172.21.15.44:0/2106428114 learned_addr learned my addr 172.21.15.44:0/2106428114 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/2106428114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c149a0 0x55ebf9cabcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5ba77fe640 1 --2- 172.21.15.44:0/2106428114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c423f0 0x55ebf9c32750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5ba67fc640 1 -- 172.21.15.44:0/2106428114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c149a0 msgr2=0x55ebf9cabcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5ba67fc640 1 --2- 172.21.15.44:0/2106428114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c149a0 0x55ebf9cabcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5ba67fc640 1 -- 172.21.15.44:0/2106428114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c423f0 msgr2=0x55ebf9c32750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5ba67fc640 1 --2- 172.21.15.44:0/2106428114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c423f0 0x55ebf9c32750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.522+0000 7f5ba67fc640 1 -- 172.21.15.44:0/2106428114 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebf9cac2c0 con 0x55ebf9c14d80 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5ba67fc640 1 --2- 172.21.15.44:0/2106428114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 0x55ebf9c44940 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5b90003d70 tx=0x7f5b900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f97a3281a550065 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5b85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5b85ffb640 1 -- 172.21.15.44:0/2106428114 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b90019040 con 0x55ebf9c14d80 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5b85ffb640 1 -- 172.21.15.44:0/2106428114 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f5b90004300 con 0x55ebf9c14d80 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5bb003f740 1 -- 172.21.15.44:0/2106428114 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ba003d8e0 con 0x55ebf9c14d80 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5b85ffb640 1 -- 172.21.15.44:0/2106428114 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5b90034050 con 0x55ebf9c14d80 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5bb003f740 10 client.6765 fetch_fsmap learned FSMap version 37 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5bb003f740 1 -- 172.21.15.44:0/2106428114 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x55ebf9cac4d0 con 0x55ebf9c14d80 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5b85ffb640 1 -- 172.21.15.44:0/2106428114 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba003d8e0 con 0x55ebf9c14d80 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5b85ffb640 1 -- 172.21.15.44:0/2106428114 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 37) ==== 3494+0+0 (secure 0 0 0) 0x7f5b90037020 con 0x55ebf9c14d80 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5bb003f740 10 client.6765 fetch_fsmap finished waiting for FSMap version 37 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5bb003f740 10 client.6765 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5bb003f740 20 client.6765 populate_metadata read hostname 'smithi044' 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5bb003f740 1 --2- 172.21.15.44:0/2106428114 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55ebf9be6cd0 0x55ebf9c32c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5bb003f740 4 client.6765 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5bb003f740 1 -- 172.21.15.44:0/2106428114 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ebf9b590c0 con 0x55ebf9be6cd0 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.523+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/2106428114 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55ebf9be6cd0 0x55ebf9c32c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5ba6ffd640 1 --2- 172.21.15.44:0/2106428114 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55ebf9be6cd0 0x55ebf9c32c90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbd1a5a752620ae3 server_cookie=3a899743d42e4500 in_seq=0 out_seq=0 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5b85ffb640 10 client.6765 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5b85ffb640 1 -- 172.21.15.44:0/2106428114 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ebf9b590c0 con 0x55ebf9be6cd0 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5b85ffb640 10 client.6765 handle_command_reply: tid=0 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5bb003f740 1 client.6765 shutdown 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5bb003f740 1 -- 172.21.15.44:0/2106428114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 msgr2=0x55ebf9c44940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5bb003f740 1 --2- 172.21.15.44:0/2106428114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 0x55ebf9c44940 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5b90003d70 tx=0x7f5b900046c0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5bb003f740 1 -- 172.21.15.44:0/2106428114 shutdown_connections 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5bb003f740 1 --2- 172.21.15.44:0/2106428114 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55ebf9be6cd0 0x55ebf9c32c90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5bb003f740 1 --2- 172.21.15.44:0/2106428114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebf9c423f0 0x55ebf9c32750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5b85ffb640 0 client.6765 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5bb003f740 1 --2- 172.21.15.44:0/2106428114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebf9c14d80 0x55ebf9c44940 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5bb003f740 1 --2- 172.21.15.44:0/2106428114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebf9c149a0 0x55ebf9cabcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5bb003f740 1 -- 172.21.15.44:0/2106428114 >> 172.21.15.44:0/2106428114 conn(0x55ebf9be7e80 msgr2=0x55ebf9beb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:35.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5bb003f740 1 -- 172.21.15.44:0/2106428114 shutdown_connections 2024-04-02T13:24:35.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5bb003f740 1 -- 172.21.15.44:0/2106428114 wait complete. 2024-04-02T13:24:35.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.524+0000 7f5bb003f740 20 client.6765 trim_cache size 0 max 16384 2024-04-02T13:24:35.533 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:24:35.533 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-02T13:24:35.533 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-02T13:24:35.534 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:24:35.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.525+0000 7f5bad44e640 1 -- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b8806c920 msgr2=0x7f5b8806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.525+0000 7f5bad44e640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b8806c920 0x7f5b8806ede0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0035000 tx=0x7f5ba0008030 comp rx=0 tx=0).stop 2024-04-02T13:24:35.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.525+0000 7f5bad44e640 1 -- 172.21.15.44:0/1123748347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba815e420 msgr2=0x7f5ba815a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.525+0000 7f5bad44e640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba815e420 0x7f5ba815a7b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800cbf0 tx=0x7f5b9800a430 comp rx=0 tx=0).stop 2024-04-02T13:24:35.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.525+0000 7f5bad44e640 1 -- 172.21.15.44:0/1123748347 shutdown_connections 2024-04-02T13:24:35.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.525+0000 7f5bad44e640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b8806c920 0x7f5b8806ede0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.525+0000 7f5bad44e640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba815e420 0x7f5ba815a7b0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.525+0000 7f5bad44e640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ba80cfc40 0x7f5ba8159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.525+0000 7f5bad44e640 1 --2- 172.21.15.44:0/1123748347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ba80721f0 0x7f5ba81463c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.525+0000 7f5bad44e640 1 -- 172.21.15.44:0/1123748347 >> 172.21.15.44:0/1123748347 conn(0x7f5ba80b2a10 msgr2=0x7f5ba80ce170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:35.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.525+0000 7f5bad44e640 1 -- 172.21.15.44:0/1123748347 shutdown_connections 2024-04-02T13:24:35.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.525+0000 7f5bad44e640 1 -- 172.21.15.44:0/1123748347 wait complete. 2024-04-02T13:24:35.537 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T13:24:35.538 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:24:35.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.637+0000 7febaf10e640 1 Processor -- start 2024-04-02T13:24:35.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.637+0000 7febaf10e640 1 -- start start 2024-04-02T13:24:35.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.638+0000 7febaf10e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7feba815e4a0 0x7feba80aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.638+0000 7febaf10e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feba80cc600 con 0x7feba80d0c50 2024-04-02T13:24:35.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.638+0000 7febaf10e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feba80cc8e0 con 0x7feba806d550 2024-04-02T13:24:35.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.638+0000 7febaf10e640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7feba80ccbc0 con 0x7feba815e4a0 2024-04-02T13:24:35.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.638+0000 7febaceaa640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7feba815e4a0 0x7feba80aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.638+0000 7feba7fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7feba80d0c50 0x7feba806ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48058/0 (socket says 172.21.15.44:48058) 2024-04-02T13:24:35.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.638+0000 7feba7fff640 1 -- 172.21.15.44:0/2175420127 learned_addr learned my addr 172.21.15.44:0/2175420127 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:35.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.638+0000 7febaceaa640 1 -- 172.21.15.44:0/2175420127 >> v1:172.21.15.69:6789/0 conn(0x7feba806d550 legacy=0x7feba80cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:35.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.638+0000 7febaceaa640 1 -- 172.21.15.44:0/2175420127 >> v1:172.21.15.44:6789/0 conn(0x7feba80d0c50 legacy=0x7feba806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.638+0000 7feba77fe640 1 -- 172.21.15.44:0/2175420127 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 512923399 0 0) 0x7feba80cc600 con 0x7feba80d0c50 2024-04-02T13:24:35.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.638+0000 7febaceaa640 1 -- 172.21.15.44:0/2175420127 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba80ccea0 con 0x7feba815e4a0 2024-04-02T13:24:35.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.639+0000 7febaceaa640 1 --2- 172.21.15.44:0/2175420127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feba815e4a0 0x7feba80aebf0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0097d0 tx=0x7feb9c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aae5d58902bf429 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:35.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.639+0000 7feba77fe640 1 -- 172.21.15.44:0/2175420127 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c035020 con 0x7feba815e4a0 2024-04-02T13:24:35.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.639+0000 7feba77fe640 1 -- 172.21.15.44:0/2175420127 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb9c031830 con 0x7feba815e4a0 2024-04-02T13:24:35.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.639+0000 7feba77fe640 1 -- 172.21.15.44:0/2175420127 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c031c00 con 0x7feba815e4a0 2024-04-02T13:24:35.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.640+0000 7febaf10e640 1 -- 172.21.15.44:0/2175420127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feba815e4a0 msgr2=0x7feba80aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.640+0000 7febaf10e640 1 --2- 172.21.15.44:0/2175420127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feba815e4a0 0x7feba80aebf0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0097d0 tx=0x7feb9c02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.640+0000 7febaf10e640 1 -- 172.21.15.44:0/2175420127 shutdown_connections 2024-04-02T13:24:35.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.640+0000 7febaf10e640 1 --2- 172.21.15.44:0/2175420127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feba815e4a0 0x7feba80aebf0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.640+0000 7febaf10e640 1 -- 172.21.15.44:0/2175420127 >> 172.21.15.44:0/2175420127 conn(0x7feba809aa20 msgr2=0x7feba809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:35.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.640+0000 7febaf10e640 1 -- 172.21.15.44:0/2175420127 shutdown_connections 2024-04-02T13:24:35.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.640+0000 7febaf10e640 1 -- 172.21.15.44:0/2175420127 wait complete. 2024-04-02T13:24:35.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.641+0000 7febaf10e640 1 Processor -- start 2024-04-02T13:24:35.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.641+0000 7febaf10e640 1 -- start start 2024-04-02T13:24:35.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febaf10e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba806d550 0x7feba8160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febaf10e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feba80d0c50 0x7feba80c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febaf10e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feba815e4a0 0x7feba814cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febaf10e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feba815e880 con 0x7feba815e4a0 2024-04-02T13:24:35.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febaf10e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7feba80cd460 con 0x7feba80d0c50 2024-04-02T13:24:35.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febaf10e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7feba80af130 con 0x7feba806d550 2024-04-02T13:24:35.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febad6ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feba815e4a0 0x7feba814cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febad6ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feba815e4a0 0x7feba814cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47460/0 (socket says 172.21.15.44:47460) 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febad6ab640 1 -- 172.21.15.44:0/1464922821 learned_addr learned my addr 172.21.15.44:0/1464922821 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7feba7fff640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feba80d0c50 0x7feba80c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febaceaa640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba806d550 0x7feba8160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febad6ab640 1 -- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba806d550 msgr2=0x7feba8160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febad6ab640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba806d550 0x7feba8160eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febad6ab640 1 -- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feba80d0c50 msgr2=0x7feba80c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febad6ab640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feba80d0c50 0x7feba80c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febad6ab640 1 -- 172.21.15.44:0/1464922821 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba81613f0 con 0x7feba815e4a0 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7feba7fff640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feba80d0c50 0x7feba80c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febad6ab640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feba815e4a0 0x7feba814cb10 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7feba000c700 tx=0x7feba000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7feba57fa640 1 -- 172.21.15.44:0/1464922821 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba0016020 con 0x7feba815e4a0 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7feba57fa640 1 -- 172.21.15.44:0/1464922821 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feba000aa50 con 0x7feba815e4a0 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7febaf10e640 1 -- 172.21.15.44:0/1464922821 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba80cbc00 con 0x7feba815e4a0 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.642+0000 7feba57fa640 1 -- 172.21.15.44:0/1464922821 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba000ae40 con 0x7feba815e4a0 2024-04-02T13:24:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.643+0000 7febaf10e640 1 -- 172.21.15.44:0/1464922821 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba80cbea0 con 0x7feba815e4a0 2024-04-02T13:24:35.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.643+0000 7febaf10e640 1 -- 172.21.15.44:0/1464922821 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feba80684f0 con 0x7feba815e4a0 2024-04-02T13:24:35.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.644+0000 7feba57fa640 1 -- 172.21.15.44:0/1464922821 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feba0019020 con 0x7feba815e4a0 2024-04-02T13:24:35.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.644+0000 7feba57fa640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feb8c06c8f0 0x7feb8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.644+0000 7feba57fa640 1 -- 172.21.15.44:0/1464922821 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7feba008bc60 con 0x7feba815e4a0 2024-04-02T13:24:35.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.645+0000 7febaceaa640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feb8c06c8f0 0x7feb8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.645+0000 7febaceaa640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feb8c06c8f0 0x7feb8c06edb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7feb9c036000 tx=0x7feb9c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:35.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.649+0000 7feba57fa640 1 -- 172.21.15.44:0/1464922821 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7feba0020030 con 0x7feba815e4a0 2024-04-02T13:24:35.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.873+0000 7febaf10e640 1 -- 172.21.15.44:0/1464922821 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feba80676a0 con 0x7feba815e4a0 2024-04-02T13:24:35.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.873+0000 7feba57fa640 1 -- 172.21.15.44:0/1464922821 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5180 (secure 0 0 0) 0x7feba0059f00 con 0x7feba815e4a0 2024-04-02T13:24:35.876 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:35.876 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":37,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1535287484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1535287484},{"type":"v1","addr":"172.21.15.69:6836","nonce":1535287484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6429,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2367107002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2367107002},{"type":"v1","addr":"172.21.15.44:6833","nonce":2367107002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/553542317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":553542317},{"type":"v1","addr":"172.21.15.69:6837","nonce":553542317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:23:24.803884+0000","modified":"2024-04-02T13:24:05.482501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6450":{"gid":6450,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":13,"addr":"172.21.15.44:6835/3127613802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3127613802},{"type":"v1","addr":"172.21.15.44:6835","nonce":3127613802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6450,"qdb_cluster":[6450]},"id":4}]} 2024-04-02T13:24:35.876 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 37 2024-04-02T13:24:35.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.875+0000 7febaf10e640 1 -- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feb8c06c8f0 msgr2=0x7feb8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.875+0000 7febaf10e640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feb8c06c8f0 0x7feb8c06edb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7feb9c036000 tx=0x7feb9c03c040 comp rx=0 tx=0).stop 2024-04-02T13:24:35.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.875+0000 7febaf10e640 1 -- 172.21.15.44:0/1464922821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feba815e4a0 msgr2=0x7feba814cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.875+0000 7febaf10e640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feba815e4a0 0x7feba814cb10 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7feba000c700 tx=0x7feba000cbc0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.875+0000 7febaf10e640 1 -- 172.21.15.44:0/1464922821 shutdown_connections 2024-04-02T13:24:35.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.875+0000 7febaf10e640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feb8c06c8f0 0x7feb8c06edb0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.875+0000 7febaf10e640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feba815e4a0 0x7feba814cb10 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.875+0000 7febaf10e640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feba80d0c50 0x7feba80c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.875+0000 7febaf10e640 1 --2- 172.21.15.44:0/1464922821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba806d550 0x7feba8160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.875+0000 7febaf10e640 1 -- 172.21.15.44:0/1464922821 >> 172.21.15.44:0/1464922821 conn(0x7feba809aa20 msgr2=0x7feba80b0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:35.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.876+0000 7febaf10e640 1 -- 172.21.15.44:0/1464922821 shutdown_connections 2024-04-02T13:24:35.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.876+0000 7febaf10e640 1 -- 172.21.15.44:0/1464922821 wait complete. 2024-04-02T13:24:35.888 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-04-02T13:24:35.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.989+0000 7f31071c8640 1 Processor -- start 2024-04-02T13:24:35.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.990+0000 7f31071c8640 1 -- start start 2024-04-02T13:24:35.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.990+0000 7f31071c8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f310006d700 0x7f310015e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.990+0000 7f31071c8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f310015ee70 0x7f31000d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.990+0000 7f31071c8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f310015fb50 con 0x7f310015e550 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.990+0000 7f31071c8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f310006ca90 con 0x7f310006d700 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.990+0000 7f31071c8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f310015f590 con 0x7f310015ee70 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.990+0000 7f30fffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f310006d700 0x7f310015e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.990+0000 7f30fffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f310006d700 0x7f310015e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47462/0 (socket says 172.21.15.44:47462) 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.990+0000 7f30fffff640 1 -- 172.21.15.44:0/3713625822 learned_addr learned my addr 172.21.15.44:0/3713625822 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.991+0000 7f3105765640 1 --2- 172.21.15.44:0/3713625822 >> v2:172.21.15.69:3300/0 conn(0x7f310015ee70 0x7f31000d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.991+0000 7f30fffff640 1 -- 172.21.15.44:0/3713625822 >> v2:172.21.15.69:3300/0 conn(0x7f310015ee70 msgr2=0x7f31000d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.991+0000 7f30fffff640 1 --2- 172.21.15.44:0/3713625822 >> v2:172.21.15.69:3300/0 conn(0x7f310015ee70 0x7f31000d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.991+0000 7f30fffff640 1 -- 172.21.15.44:0/3713625822 >> v1:172.21.15.69:6789/0 conn(0x7f310015e550 legacy=0x7f310006cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.991+0000 7f30fffff640 1 -- 172.21.15.44:0/3713625822 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310015f320 con 0x7f310006d700 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.991+0000 7f30fffff640 1 --2- 172.21.15.44:0/3713625822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f310006d700 0x7f310015e930 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f310009b870 tx=0x7f30ec02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ff9da12790f66 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.992+0000 7f30ff7fe640 1 -- 172.21.15.44:0/3713625822 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec02f850 con 0x7f310006d700 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.992+0000 7f30ff7fe640 1 -- 172.21.15.44:0/3713625822 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30ec02f9f0 con 0x7f310006d700 2024-04-02T13:24:35.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.992+0000 7f30ff7fe640 1 -- 172.21.15.44:0/3713625822 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec02fdc0 con 0x7f310006d700 2024-04-02T13:24:35.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.992+0000 7f31071c8640 1 -- 172.21.15.44:0/3713625822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f310006d700 msgr2=0x7f310015e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:35.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.992+0000 7f31071c8640 1 --2- 172.21.15.44:0/3713625822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f310006d700 0x7f310015e930 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f310009b870 tx=0x7f30ec02ba30 comp rx=0 tx=0).stop 2024-04-02T13:24:35.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.992+0000 7f31071c8640 1 -- 172.21.15.44:0/3713625822 shutdown_connections 2024-04-02T13:24:35.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.992+0000 7f31071c8640 1 --2- 172.21.15.44:0/3713625822 >> v2:172.21.15.69:3300/0 conn(0x7f310015ee70 0x7f31000d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.992+0000 7f31071c8640 1 --2- 172.21.15.44:0/3713625822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f310006d700 0x7f310015e930 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:35.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.992+0000 7f31071c8640 1 -- 172.21.15.44:0/3713625822 >> 172.21.15.44:0/3713625822 conn(0x7f310009aa20 msgr2=0x7f310009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:35.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.992+0000 7f31071c8640 1 -- 172.21.15.44:0/3713625822 shutdown_connections 2024-04-02T13:24:35.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.992+0000 7f31071c8640 1 -- 172.21.15.44:0/3713625822 wait complete. 2024-04-02T13:24:35.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.993+0000 7f31071c8640 1 Processor -- start 2024-04-02T13:24:35.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.993+0000 7f31071c8640 1 -- start start 2024-04-02T13:24:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.994+0000 7f31071c8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f310006d700 0x7f31000b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.994+0000 7f31071c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f310015e550 0x7f31001565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.994+0000 7f31071c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f310015ee70 0x7f31000b5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.994+0000 7f31071c8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f31000d0590 con 0x7f310015e550 2024-04-02T13:24:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.994+0000 7f31071c8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31000d02b0 con 0x7f310006d700 2024-04-02T13:24:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.994+0000 7f31071c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f31000cc4b0 con 0x7f310015ee70 2024-04-02T13:24:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.994+0000 7f30fffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f310015e550 0x7f31001565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.994+0000 7f30fffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f310015e550 0x7f31001565f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47470/0 (socket says 172.21.15.44:47470) 2024-04-02T13:24:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.994+0000 7f30fffff640 1 -- 172.21.15.44:0/1549804219 learned_addr learned my addr 172.21.15.44:0/1549804219 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.994+0000 7f3104f64640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f310006d700 0x7f31000b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.994+0000 7f3105765640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f310015ee70 0x7f31000b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.995+0000 7f30fffff640 1 -- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f310015ee70 msgr2=0x7f31000b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.995+0000 7f30fffff640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f310015ee70 0x7f31000b5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.995+0000 7f30fffff640 1 -- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f310006d700 msgr2=0x7f31000b6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.995+0000 7f30fffff640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f310006d700 0x7f31000b6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.995+0000 7f30fffff640 1 -- 172.21.15.44:0/1549804219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3100156b30 con 0x7f310015e550 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.995+0000 7f30fffff640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f310015e550 0x7f31001565f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f30ec02ba70 tx=0x7f30ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.995+0000 7f30fd7fa640 1 -- 172.21.15.44:0/1549804219 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec037c90 con 0x7f310015e550 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.995+0000 7f30fd7fa640 1 -- 172.21.15.44:0/1549804219 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30ec046070 con 0x7f310015e550 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.995+0000 7f30fd7fa640 1 -- 172.21.15.44:0/1549804219 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec041640 con 0x7f310015e550 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.995+0000 7f31071c8640 1 -- 172.21.15.44:0/1549804219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31000b7600 con 0x7f310015e550 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.995+0000 7f31071c8640 1 -- 172.21.15.44:0/1549804219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31000c8fd0 con 0x7f310015e550 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.996+0000 7f30fd7fa640 1 -- 172.21.15.44:0/1549804219 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f30ec040070 con 0x7f310015e550 2024-04-02T13:24:36.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.997+0000 7f30fd7fa640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f30e406c8f0 0x7f30e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.997+0000 7f30fd7fa640 1 -- 172.21.15.44:0/1549804219 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f30ec0ae3f0 con 0x7f310015e550 2024-04-02T13:24:36.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.997+0000 7f3104f64640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f30e406c8f0 0x7f30e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.997+0000 7f3104f64640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f30e406c8f0 0x7f30e406edb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f30f40016e0 tx=0x7f30f4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.998+0000 7f3109da9740 1 Processor -- start 2024-04-02T13:24:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.998+0000 7f3109da9740 1 -- start start 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.998+0000 7f3109da9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf54dc0 0x55b64bf551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.998+0000 7f3109da9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bf5abe0 0x55b64bf58e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.998+0000 7f3109da9740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bfa24d0 0x55b64bfa2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f3109da9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b64bf1c5d0 con 0x55b64bf5abe0 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f3109da9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b64bebe6b0 con 0x55b64bf54dc0 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f3109da9740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55b64be5c890 con 0x55b64bfa24d0 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f30fffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bf5abe0 0x55b64bf58e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f30fffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bf5abe0 0x55b64bf58e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47478/0 (socket says 172.21.15.44:47478) 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f30fffff640 1 -- 172.21.15.44:0/561819758 learned_addr learned my addr 172.21.15.44:0/561819758 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f3104f64640 1 --2- 172.21.15.44:0/561819758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf54dc0 0x55b64bf551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f3105765640 1 --2- 172.21.15.44:0/561819758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bfa24d0 0x55b64bfa2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f30fffff640 1 -- 172.21.15.44:0/561819758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bfa24d0 msgr2=0x55b64bfa2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f30fffff640 1 --2- 172.21.15.44:0/561819758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bfa24d0 0x55b64bfa2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f30fffff640 1 -- 172.21.15.44:0/561819758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf54dc0 msgr2=0x55b64bf551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f30fffff640 1 --2- 172.21.15.44:0/561819758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf54dc0 0x55b64bf551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f30fffff640 1 -- 172.21.15.44:0/561819758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b64be8b1d0 con 0x55b64bf5abe0 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f3104f64640 1 --2- 172.21.15.44:0/561819758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf54dc0 0x55b64bf551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f30fffff640 1 --2- 172.21.15.44:0/561819758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bf5abe0 0x55b64bf58e10 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0845b0 tx=0x7f30ec0781a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa15bedd6b21e24 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f30ea7fc640 1 -- 172.21.15.44:0/561819758 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec0802b0 con 0x55b64bf5abe0 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f30ea7fc640 1 -- 172.21.15.44:0/561819758 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30ec084ae0 con 0x55b64bf5abe0 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:35.999+0000 7f30ea7fc640 1 -- 172.21.15.44:0/561819758 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec07a200 con 0x55b64bf5abe0 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.000+0000 7f3109da9740 1 -- 172.21.15.44:0/561819758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bf5abe0 msgr2=0x55b64bf58e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.000+0000 7f3109da9740 1 --2- 172.21.15.44:0/561819758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bf5abe0 0x55b64bf58e10 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0845b0 tx=0x7f30ec0781a0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.000+0000 7f3109da9740 1 -- 172.21.15.44:0/561819758 shutdown_connections 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.000+0000 7f3109da9740 1 --2- 172.21.15.44:0/561819758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bfa24d0 0x55b64bfa2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.000+0000 7f3109da9740 1 --2- 172.21.15.44:0/561819758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bf5abe0 0x55b64bf58e10 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.000+0000 7f3109da9740 1 --2- 172.21.15.44:0/561819758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf54dc0 0x55b64bf551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.000+0000 7f3109da9740 1 -- 172.21.15.44:0/561819758 >> 172.21.15.44:0/561819758 conn(0x55b64bf56e40 msgr2=0x55b64bf58700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:36.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.000+0000 7f3109da9740 1 -- 172.21.15.44:0/561819758 shutdown_connections 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.000+0000 7f3109da9740 1 -- 172.21.15.44:0/561819758 wait complete. 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.000+0000 7f3109da9740 1 -- start start 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.001+0000 7f3109da9740 1 Processor -- start 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.001+0000 7f3109da9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 0x55b64bf86ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.001+0000 7f3109da9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf87220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.001+0000 7f3109da9740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfafff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.001+0000 7f3109da9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b64be5a200 con 0x55b64bfa24d0 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.001+0000 7f3109da9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b64beba660 con 0x55b64bf5abe0 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.001+0000 7f3109da9740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55b64bf1c5d0 con 0x55b64bf9af40 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.001+0000 7f3109da9740 10 client.0 _ll_register_callbacks cb 0x55b64be7d040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.001+0000 7f30fffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf87220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.001+0000 7f30fffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf87220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47488/0 (socket says 172.21.15.44:47488) 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f30fffff640 1 -- 172.21.15.44:0/3180635862 learned_addr learned my addr 172.21.15.44:0/3180635862 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f3104f64640 1 --2- 172.21.15.44:0/3180635862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 0x55b64bf86ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f3105765640 1 --2- 172.21.15.44:0/3180635862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfafff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f30fffff640 1 -- 172.21.15.44:0/3180635862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 msgr2=0x55b64bfafff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:36.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f30fffff640 1 --2- 172.21.15.44:0/3180635862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfafff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f30fffff640 1 -- 172.21.15.44:0/3180635862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 msgr2=0x55b64bf86ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:36.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f30fffff640 1 --2- 172.21.15.44:0/3180635862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 0x55b64bf86ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f30fffff640 1 -- 172.21.15.44:0/3180635862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b64bc9e7c0 con 0x55b64bfa24d0 2024-04-02T13:24:36.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f30fffff640 1 --2- 172.21.15.44:0/3180635862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf87220 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0ac750 tx=0x7f30ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1096a407e1b8186 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f30df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:24:36.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f30df7fe640 1 -- 172.21.15.44:0/3180635862 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec099060 con 0x55b64bfa24d0 2024-04-02T13:24:36.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f30df7fe640 1 -- 172.21.15.44:0/3180635862 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f30ec0ae0b0 con 0x55b64bfa24d0 2024-04-02T13:24:36.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f3109da9740 1 -- 172.21.15.44:0/3180635862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30ec084ae0 con 0x55b64bfa24d0 2024-04-02T13:24:36.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f30df7fe640 1 -- 172.21.15.44:0/3180635862 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f30ec0a0050 con 0x55b64bfa24d0 2024-04-02T13:24:36.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f3109da9740 10 client.6786 fetch_fsmap learned FSMap version 37 2024-04-02T13:24:36.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f3109da9740 1 -- 172.21.15.44:0/3180635862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x55b64bcfef50 con 0x55b64bfa24d0 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.002+0000 7f30df7fe640 1 -- 172.21.15.44:0/3180635862 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30ec084ae0 con 0x55b64bfa24d0 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.003+0000 7f30df7fe640 1 -- 172.21.15.44:0/3180635862 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 37) ==== 3494+0+0 (secure 0 0 0) 0x7f30ec09b300 con 0x55b64bfa24d0 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.003+0000 7f3109da9740 10 client.6786 fetch_fsmap finished waiting for FSMap version 37 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.003+0000 7f3109da9740 10 client.6786 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.003+0000 7f3109da9740 20 client.6786 populate_metadata read hostname 'smithi044' 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.003+0000 7f3109da9740 1 --2- 172.21.15.44:0/3180635862 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55b64bfb1960 0x55b64bfbd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.003+0000 7f3109da9740 4 client.6786 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.003+0000 7f3109da9740 1 -- 172.21.15.44:0/3180635862 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b64bed1ee0 con 0x55b64bfb1960 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.003+0000 7f3104f64640 1 --2- 172.21.15.44:0/3180635862 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55b64bfb1960 0x55b64bfbd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.003+0000 7f30df7fe640 10 client.6786 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.003+0000 7f3104f64640 1 --2- 172.21.15.44:0/3180635862 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55b64bfb1960 0x55b64bfbd920 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61156c6d06aeca56 server_cookie=7ed1c6b001514934 in_seq=0 out_seq=0 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f30df7fe640 1 -- 172.21.15.44:0/3180635862 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b64bed1ee0 con 0x55b64bfb1960 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f30df7fe640 10 client.6786 handle_command_reply: tid=0 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f3109da9740 1 client.6786 shutdown 2024-04-02T13:24:36.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f3109da9740 1 -- 172.21.15.44:0/3180635862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 msgr2=0x55b64bf87220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f3109da9740 1 --2- 172.21.15.44:0/3180635862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf87220 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0ac750 tx=0x7f30ec0046c0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f3109da9740 1 -- 172.21.15.44:0/3180635862 shutdown_connections 2024-04-02T13:24:36.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f3109da9740 1 --2- 172.21.15.44:0/3180635862 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55b64bfb1960 0x55b64bfbd920 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f3109da9740 1 --2- 172.21.15.44:0/3180635862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfafff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f3109da9740 1 --2- 172.21.15.44:0/3180635862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf87220 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f30df7fe640 0 client.6786 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:36.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f3109da9740 1 --2- 172.21.15.44:0/3180635862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 0x55b64bf86ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f3109da9740 1 -- 172.21.15.44:0/3180635862 >> 172.21.15.44:0/3180635862 conn(0x55b64bf54dc0 msgr2=0x55b64bfa0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:36.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f3109da9740 1 -- 172.21.15.44:0/3180635862 shutdown_connections 2024-04-02T13:24:36.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f3109da9740 1 -- 172.21.15.44:0/3180635862 wait complete. 2024-04-02T13:24:36.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.004+0000 7f3109da9740 20 client.6786 trim_cache size 0 max 16384 2024-04-02T13:24:36.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.018+0000 7f3109da9740 1 Processor -- start 2024-04-02T13:24:36.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.018+0000 7f3109da9740 1 -- start start 2024-04-02T13:24:36.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.018+0000 7f3109da9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 0x55b64bf9ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.018+0000 7f3109da9740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfa04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.018+0000 7f3109da9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf85c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.018+0000 7f3109da9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f31002002f0 con 0x55b64bfa24d0 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.018+0000 7f3109da9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31000cab80 con 0x55b64bf5abe0 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.018+0000 7f3109da9740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f31001ffc80 con 0x55b64bf9af40 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.018+0000 7f3105765640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf85c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.018+0000 7f3105765640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf85c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47504/0 (socket says 172.21.15.44:47504) 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.018+0000 7f3105765640 1 -- 172.21.15.44:0/4251864513 learned_addr learned my addr 172.21.15.44:0/4251864513 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.019+0000 7f3105765640 1 -- 172.21.15.44:0/4251864513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 msgr2=0x55b64bfa04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.019+0000 7f3104f64640 1 --2- 172.21.15.44:0/4251864513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 0x55b64bf9ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.019+0000 7f30fffff640 1 --2- 172.21.15.44:0/4251864513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfa04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.019+0000 7f3105765640 1 --2- 172.21.15.44:0/4251864513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfa04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.019+0000 7f3105765640 1 -- 172.21.15.44:0/4251864513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 msgr2=0x55b64bf9ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.019+0000 7f3105765640 1 --2- 172.21.15.44:0/4251864513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 0x55b64bf9ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.019+0000 7f3105765640 1 -- 172.21.15.44:0/4251864513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b64bb920b0 con 0x55b64bfa24d0 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.019+0000 7f3104f64640 1 --2- 172.21.15.44:0/4251864513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 0x55b64bf9ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.019+0000 7f30fffff640 1 --2- 172.21.15.44:0/4251864513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfa04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.019+0000 7f3105765640 1 --2- 172.21.15.44:0/4251864513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf85c20 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f30f8015680 tx=0x7f30f8015b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d813b6048c2997 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.019+0000 7f30e9ffb640 1 -- 172.21.15.44:0/4251864513 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8007a90 con 0x55b64bfa24d0 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.019+0000 7f30e9ffb640 1 -- 172.21.15.44:0/4251864513 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f8007c30 con 0x55b64bfa24d0 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.019+0000 7f30e9ffb640 1 -- 172.21.15.44:0/4251864513 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8015e80 con 0x55b64bfa24d0 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.020+0000 7f3109da9740 1 -- 172.21.15.44:0/4251864513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 msgr2=0x55b64bf85c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.020+0000 7f3109da9740 1 --2- 172.21.15.44:0/4251864513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf85c20 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f30f8015680 tx=0x7f30f8015b40 comp rx=0 tx=0).stop 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.020+0000 7f3109da9740 1 -- 172.21.15.44:0/4251864513 shutdown_connections 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.020+0000 7f3109da9740 1 --2- 172.21.15.44:0/4251864513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf85c20 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.020+0000 7f3109da9740 1 --2- 172.21.15.44:0/4251864513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfa04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.020+0000 7f3109da9740 1 --2- 172.21.15.44:0/4251864513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 0x55b64bf9ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.020+0000 7f3109da9740 1 -- 172.21.15.44:0/4251864513 >> 172.21.15.44:0/4251864513 conn(0x55b64bf54dc0 msgr2=0x55b64bf55480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.020+0000 7f3109da9740 1 -- 172.21.15.44:0/4251864513 shutdown_connections 2024-04-02T13:24:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.020+0000 7f3109da9740 1 -- 172.21.15.44:0/4251864513 wait complete. 2024-04-02T13:24:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.020+0000 7f3109da9740 1 -- start start 2024-04-02T13:24:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.020+0000 7f3109da9740 1 Processor -- start 2024-04-02T13:24:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3109da9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 0x55b64bfb8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3109da9740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfb89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3109da9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf585d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3109da9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b64be5a200 con 0x55b64bfa24d0 2024-04-02T13:24:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3109da9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b64beba660 con 0x55b64bf5abe0 2024-04-02T13:24:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3109da9740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55b64bfb1d50 con 0x55b64bf9af40 2024-04-02T13:24:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3109da9740 10 client.0 _ll_register_callbacks cb 0x7f30f80097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:24:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3105765640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3105765640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47520/0 (socket says 172.21.15.44:47520) 2024-04-02T13:24:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3105765640 1 -- 172.21.15.44:0/2020595743 learned_addr learned my addr 172.21.15.44:0/2020595743 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3104f64640 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 0x55b64bfb8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f30fffff640 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfb89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3105765640 1 -- 172.21.15.44:0/2020595743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 msgr2=0x55b64bfb89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3105765640 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfb89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3105765640 1 -- 172.21.15.44:0/2020595743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 msgr2=0x55b64bfb8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3105765640 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 0x55b64bfb8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f3105765640 1 -- 172.21.15.44:0/2020595743 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b64bf86820 con 0x55b64bfa24d0 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.021+0000 7f30fffff640 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfb89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f3105765640 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf585d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f30f801d000 tx=0x7f30f8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0016372f6645df server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f30df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f30df7fe640 1 -- 172.21.15.44:0/2020595743 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f802f060 con 0x55b64bfa24d0 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f30df7fe640 1 -- 172.21.15.44:0/2020595743 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f30f802a420 con 0x55b64bfa24d0 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f3109da9740 1 -- 172.21.15.44:0/2020595743 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30f8007c30 con 0x55b64bfa24d0 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f30df7fe640 1 -- 172.21.15.44:0/2020595743 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f30f8035050 con 0x55b64bfa24d0 2024-04-02T13:24:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f3109da9740 10 client.6792 fetch_fsmap learned FSMap version 37 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f3109da9740 1 -- 172.21.15.44:0/2020595743 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x55b64bfafb50 con 0x55b64bfa24d0 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f30df7fe640 1 -- 172.21.15.44:0/2020595743 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f8007c30 con 0x55b64bfa24d0 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f30df7fe640 1 -- 172.21.15.44:0/2020595743 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 37) ==== 3494+0+0 (secure 0 0 0) 0x7f30f8038020 con 0x55b64bfa24d0 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f3109da9740 10 client.6792 fetch_fsmap finished waiting for FSMap version 37 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f3109da9740 10 client.6792 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f3109da9740 20 client.6792 populate_metadata read hostname 'smithi044' 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f3109da9740 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55b64c019e70 0x55b64c01a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f3109da9740 4 client.6792 mds_command: new command op to 6450 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f3109da9740 1 -- 172.21.15.44:0/2020595743 --> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55b64bf575a0 con 0x55b64c019e70 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.022+0000 7f3104f64640 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55b64c019e70 0x55b64c01a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f3104f64640 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55b64c019e70 0x55b64c01a250 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8f28cbe7425eba6 server_cookie=114bb3d23ce705be in_seq=0 out_seq=0 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f30df7fe640 10 client.6792 ms_handle_connect on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f30df7fe640 1 -- 172.21.15.44:0/2020595743 <== mds.0 v2:172.21.15.44:6834/3127613802 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b64bf575a0 con 0x55b64c019e70 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f30df7fe640 10 client.6792 handle_command_reply: tid=0 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f3109da9740 1 client.6792 shutdown 2024-04-02T13:24:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f3109da9740 1 -- 172.21.15.44:0/2020595743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 msgr2=0x55b64bf585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f3109da9740 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf585d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f30f801d000 tx=0x7f30f8018000 comp rx=0 tx=0).stop 2024-04-02T13:24:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f3109da9740 1 -- 172.21.15.44:0/2020595743 shutdown_connections 2024-04-02T13:24:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f3109da9740 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.44:6834/3127613802,v1:172.21.15.44:6835/3127613802] conn(0x55b64c019e70 0x55b64c01a250 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f3109da9740 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b64bfa24d0 0x55b64bf585d0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f3109da9740 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b64bf9af40 0x55b64bfb89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f30df7fe640 0 client.6792 ms_handle_reset on v2:172.21.15.44:6834/3127613802 2024-04-02T13:24:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f3109da9740 1 --2- 172.21.15.44:0/2020595743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b64bf5abe0 0x55b64bfb8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f3109da9740 1 -- 172.21.15.44:0/2020595743 >> 172.21.15.44:0/2020595743 conn(0x55b64bf54dc0 msgr2=0x55b64bfa07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f3109da9740 1 -- 172.21.15.44:0/2020595743 shutdown_connections 2024-04-02T13:24:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f3109da9740 1 -- 172.21.15.44:0/2020595743 wait complete. 2024-04-02T13:24:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.023+0000 7f3109da9740 20 client.6792 trim_cache size 0 max 16384 2024-04-02T13:24:36.049 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:36.050 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-02T13:24:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.024+0000 7f31071c8640 1 -- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f30e406c8f0 msgr2=0x7f30e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.024+0000 7f31071c8640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f30e406c8f0 0x7f30e406edb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f30f40016e0 tx=0x7f30f4009040 comp rx=0 tx=0).stop 2024-04-02T13:24:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.024+0000 7f31071c8640 1 -- 172.21.15.44:0/1549804219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f310015e550 msgr2=0x7f31001565f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.024+0000 7f31071c8640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f310015e550 0x7f31001565f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f30ec02ba70 tx=0x7f30ec004730 comp rx=0 tx=0).stop 2024-04-02T13:24:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.024+0000 7f31071c8640 1 -- 172.21.15.44:0/1549804219 shutdown_connections 2024-04-02T13:24:36.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.024+0000 7f31071c8640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f30e406c8f0 0x7f30e406edb0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.024+0000 7f31071c8640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f310015ee70 0x7f31000b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.024+0000 7f31071c8640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f310015e550 0x7f31001565f0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.024+0000 7f31071c8640 1 --2- 172.21.15.44:0/1549804219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f310006d700 0x7f31000b6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.024+0000 7f31071c8640 1 -- 172.21.15.44:0/1549804219 >> 172.21.15.44:0/1549804219 conn(0x7f310009aa20 msgr2=0x7f31000d09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:36.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.024+0000 7f31071c8640 1 -- 172.21.15.44:0/1549804219 shutdown_connections 2024-04-02T13:24:36.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.024+0000 7f31071c8640 1 -- 172.21.15.44:0/1549804219 wait complete. 2024-04-02T13:24:36.052 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-02T13:24:36.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.150+0000 7fe3821ba640 1 Processor -- start 2024-04-02T13:24:36.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.150+0000 7fe3821ba640 1 -- start start 2024-04-02T13:24:36.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.151+0000 7fe3821ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe37c0d2470 0x7fe37c0d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.151+0000 7fe3821ba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe37c0b56f0 0x7fe37c0d2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.151+0000 7fe3821ba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe37c0d0c80 con 0x7fe37c1621c0 2024-04-02T13:24:36.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.151+0000 7fe3821ba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe37c0d32d0 con 0x7fe37c0b56f0 2024-04-02T13:24:36.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.151+0000 7fe3821ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe37c0d15b0 con 0x7fe37c0d2470 2024-04-02T13:24:36.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.151+0000 7fe37bfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe37c1621c0 0x7fe37c162650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48066/0 (socket says 172.21.15.44:48066) 2024-04-02T13:24:36.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.151+0000 7fe37bfff640 1 -- 172.21.15.44:0/237578288 learned_addr learned my addr 172.21.15.44:0/237578288 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:36.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.151+0000 7fe37b7fe640 1 --2- 172.21.15.44:0/237578288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe37c0d2470 0x7fe37c0d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.152+0000 7fe37affd640 1 --2- 172.21.15.44:0/237578288 >> v2:172.21.15.69:3300/0 conn(0x7fe37c0b56f0 0x7fe37c0d2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.152+0000 7fe37a7fc640 1 -- 172.21.15.44:0/237578288 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3459903696 0 0) 0x7fe37c0d0c80 con 0x7fe37c1621c0 2024-04-02T13:24:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.152+0000 7fe37a7fc640 1 -- 172.21.15.44:0/237578288 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe364003520 con 0x7fe37c1621c0 2024-04-02T13:24:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.152+0000 7fe37affd640 1 -- 172.21.15.44:0/237578288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe37c0d2470 msgr2=0x7fe37c0d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.152+0000 7fe37affd640 1 --2- 172.21.15.44:0/237578288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe37c0d2470 0x7fe37c0d2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.152+0000 7fe37affd640 1 -- 172.21.15.44:0/237578288 >> v1:172.21.15.44:6789/0 conn(0x7fe37c1621c0 legacy=0x7fe37c162650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.152+0000 7fe37affd640 1 -- 172.21.15.44:0/237578288 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37c0d0f60 con 0x7fe37c0b56f0 2024-04-02T13:24:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.152+0000 7fe37b7fe640 1 --2- 172.21.15.44:0/237578288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe37c0d2470 0x7fe37c0d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:24:36.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.153+0000 7fe37affd640 1 --2- 172.21.15.44:0/237578288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe37c0b56f0 0x7fe37c0d2d90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe3700097a0 tx=0x7fe37002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=851e0258d5e24b2b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.153+0000 7fe37a7fc640 1 -- 172.21.15.44:0/237578288 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3700315a0 con 0x7fe37c0b56f0 2024-04-02T13:24:36.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.153+0000 7fe37a7fc640 1 -- 172.21.15.44:0/237578288 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe370037070 con 0x7fe37c0b56f0 2024-04-02T13:24:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.153+0000 7fe37a7fc640 1 -- 172.21.15.44:0/237578288 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe370031940 con 0x7fe37c0b56f0 2024-04-02T13:24:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.153+0000 7fe3821ba640 1 -- 172.21.15.44:0/237578288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe37c0b56f0 msgr2=0x7fe37c0d2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.153+0000 7fe3821ba640 1 --2- 172.21.15.44:0/237578288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe37c0b56f0 0x7fe37c0d2d90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe3700097a0 tx=0x7fe37002b840 comp rx=0 tx=0).stop 2024-04-02T13:24:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.153+0000 7fe3821ba640 1 -- 172.21.15.44:0/237578288 shutdown_connections 2024-04-02T13:24:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.153+0000 7fe3821ba640 1 --2- 172.21.15.44:0/237578288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe37c0b56f0 0x7fe37c0d2d90 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.153+0000 7fe3821ba640 1 --2- 172.21.15.44:0/237578288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe37c0d2470 0x7fe37c0d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.153+0000 7fe3821ba640 1 -- 172.21.15.44:0/237578288 >> 172.21.15.44:0/237578288 conn(0x7fe37c09d4b0 msgr2=0x7fe37c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.153+0000 7fe3821ba640 1 -- 172.21.15.44:0/237578288 shutdown_connections 2024-04-02T13:24:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.153+0000 7fe3821ba640 1 -- 172.21.15.44:0/237578288 wait complete. 2024-04-02T13:24:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.155+0000 7fe3821ba640 1 Processor -- start 2024-04-02T13:24:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.155+0000 7fe3821ba640 1 -- start start 2024-04-02T13:24:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.155+0000 7fe3821ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe37c0b56f0 0x7fe37c1591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.155+0000 7fe3821ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe37c0d2470 0x7fe37c0b7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.155+0000 7fe3821ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe37c1621c0 0x7fe37c0b8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.155+0000 7fe3821ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c0cfe10 con 0x7fe37c0b56f0 2024-04-02T13:24:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.155+0000 7fe3821ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c0d1a40 con 0x7fe37c0d2470 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.155+0000 7fe3821ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe37c0cff90 con 0x7fe37c1621c0 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe37b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe37c0b56f0 0x7fe37c1591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe37b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe37c0b56f0 0x7fe37c1591e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47528/0 (socket says 172.21.15.44:47528) 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe37b7fe640 1 -- 172.21.15.44:0/2207431105 learned_addr learned my addr 172.21.15.44:0/2207431105 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe37affd640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe37c0d2470 0x7fe37c0b7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe37bfff640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe37c1621c0 0x7fe37c0b8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe37b7fe640 1 -- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe37c1621c0 msgr2=0x7fe37c0b8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe37b7fe640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe37c1621c0 0x7fe37c0b8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe37b7fe640 1 -- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe37c0d2470 msgr2=0x7fe37c0b7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe37b7fe640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe37c0d2470 0x7fe37c0b7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe37b7fe640 1 -- 172.21.15.44:0/2207431105 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37c0b6f00 con 0x7fe37c0b56f0 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe37b7fe640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe37c0b56f0 0x7fe37c1591e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fe36c00ebe0 tx=0x7fe36c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe363fff640 1 -- 172.21.15.44:0/2207431105 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c017070 con 0x7fe37c0b56f0 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe363fff640 1 -- 172.21.15.44:0/2207431105 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c00cc30 con 0x7fe37c0b56f0 2024-04-02T13:24:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe3821ba640 1 -- 172.21.15.44:0/2207431105 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe37c0b9240 con 0x7fe37c0b56f0 2024-04-02T13:24:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe363fff640 1 -- 172.21.15.44:0/2207431105 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c013470 con 0x7fe37c0b56f0 2024-04-02T13:24:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.156+0000 7fe3821ba640 1 -- 172.21.15.44:0/2207431105 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe37c0b7480 con 0x7fe37c0b56f0 2024-04-02T13:24:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.157+0000 7fe3821ba640 1 -- 172.21.15.44:0/2207431105 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe37c0684f0 con 0x7fe37c0b56f0 2024-04-02T13:24:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.158+0000 7fe363fff640 1 -- 172.21.15.44:0/2207431105 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe36c01a070 con 0x7fe37c0b56f0 2024-04-02T13:24:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.158+0000 7fe363fff640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe36806c8f0 0x7fe36806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.158+0000 7fe363fff640 1 -- 172.21.15.44:0/2207431105 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7fe36c08bc60 con 0x7fe37c0b56f0 2024-04-02T13:24:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.159+0000 7fe37affd640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe36806c8f0 0x7fe36806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.159+0000 7fe37affd640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe36806c8f0 0x7fe36806edb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe370036000 tx=0x7fe370008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.163+0000 7fe363fff640 1 -- 172.21.15.44:0/2207431105 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe36c059f00 con 0x7fe37c0b56f0 2024-04-02T13:24:36.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.393+0000 7fe3821ba640 1 -- 172.21.15.44:0/2207431105 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe37c08e260 con 0x7fe37c0b56f0 2024-04-02T13:24:36.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.394+0000 7fe363fff640 1 -- 172.21.15.44:0/2207431105 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fe36c059800 con 0x7fe37c0b56f0 2024-04-02T13:24:36.396 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:36.396 INFO:teuthology.orchestra.run.smithi044.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-02T13:24:36.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.396+0000 7fe3821ba640 1 -- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe36806c8f0 msgr2=0x7fe36806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.396+0000 7fe3821ba640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe36806c8f0 0x7fe36806edb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe370036000 tx=0x7fe370008030 comp rx=0 tx=0).stop 2024-04-02T13:24:36.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.396+0000 7fe3821ba640 1 -- 172.21.15.44:0/2207431105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe37c0b56f0 msgr2=0x7fe37c1591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.396+0000 7fe3821ba640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe37c0b56f0 0x7fe37c1591e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fe36c00ebe0 tx=0x7fe36c00c520 comp rx=0 tx=0).stop 2024-04-02T13:24:36.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.396+0000 7fe3821ba640 1 -- 172.21.15.44:0/2207431105 shutdown_connections 2024-04-02T13:24:36.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.396+0000 7fe3821ba640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe36806c8f0 0x7fe36806edb0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.396+0000 7fe3821ba640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe37c1621c0 0x7fe37c0b8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.396+0000 7fe3821ba640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe37c0d2470 0x7fe37c0b7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.396+0000 7fe3821ba640 1 --2- 172.21.15.44:0/2207431105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe37c0b56f0 0x7fe37c1591e0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.396+0000 7fe3821ba640 1 -- 172.21.15.44:0/2207431105 >> 172.21.15.44:0/2207431105 conn(0x7fe37c09d4b0 msgr2=0x7fe37c06ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:36.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.396+0000 7fe3821ba640 1 -- 172.21.15.44:0/2207431105 shutdown_connections 2024-04-02T13:24:36.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.396+0000 7fe3821ba640 1 -- 172.21.15.44:0/2207431105 wait complete. 2024-04-02T13:24:36.408 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-02T13:24:36.408 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:24:36.408 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:24:36.450 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:24:36.508 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:24:36.508 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:24:36.508 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:24:36.508 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:24:36.523 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:24:36.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.663+0000 7f1312d96640 1 Processor -- start 2024-04-02T13:24:36.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.663+0000 7f1312d96640 1 -- start start 2024-04-02T13:24:36.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.664+0000 7f1312d96640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f130c0d2750 0x7f130c0d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.664+0000 7f1312d96640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130c06d4e0 con 0x7f130c0d0990 2024-04-02T13:24:36.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.664+0000 7f1312d96640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130c0cee00 con 0x7f130c0b34c0 2024-04-02T13:24:36.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.664+0000 7f1312d96640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f130c0d1330 con 0x7f130c0d2750 2024-04-02T13:24:36.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.664+0000 7f130bfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f130c0d0990 0x7f130c0d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48076/0 (socket says 172.21.15.44:48076) 2024-04-02T13:24:36.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.664+0000 7f130bfff640 1 -- 172.21.15.44:0/2930771296 learned_addr learned my addr 172.21.15.44:0/2930771296 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:36.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3824608391 0 0) 0x7f130c06d4e0 con 0x7f130c0d0990 2024-04-02T13:24:36.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12fc003520 con 0x7f130c0d0990 2024-04-02T13:24:36.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f1311333640 1 --2- 172.21.15.44:0/2930771296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f130c0d2750 0x7f130c0d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1681711789 0 0) 0x7f12fc003520 con 0x7f130c0d0990 2024-04-02T13:24:36.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f130c0d2750 msgr2=0x7f130c0d0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:36.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 --2- 172.21.15.44:0/2930771296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f130c0d2750 0x7f130c0d0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 >> v1:172.21.15.69:6789/0 conn(0x7f130c0b34c0 legacy=0x7f130c0cfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130c06d200 con 0x7f130c0d0990 2024-04-02T13:24:36.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f12f8001670 con 0x7f130c0d0990 2024-04-02T13:24:36.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 >> v1:172.21.15.44:6789/0 conn(0x7f130c0d0990 legacy=0x7f130c0d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 --2- 172.21.15.44:0/2930771296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc0039e0 0x7f12fc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 --2- 172.21.15.44:0/2930771296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc004800 0x7f12fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 --2- 172.21.15.44:0/2930771296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f130c0d2750 0x7f12fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f12fc00ba70 con 0x7f12fc004800 2024-04-02T13:24:36.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f12fc00bbf0 con 0x7f12fc0039e0 2024-04-02T13:24:36.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f12fc00bdf0 con 0x7f130c0d2750 2024-04-02T13:24:36.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130bfff640 1 --2- 172.21.15.44:0/2930771296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc004800 0x7f12fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f1310b32640 1 --2- 172.21.15.44:0/2930771296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc0039e0 0x7f12fc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f1311333640 1 --2- 172.21.15.44:0/2930771296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f130c0d2750 0x7f12fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130bfff640 1 -- 172.21.15.44:0/2930771296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f130c0d2750 msgr2=0x7f12fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:36.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130bfff640 1 --2- 172.21.15.44:0/2930771296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f130c0d2750 0x7f12fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130bfff640 1 -- 172.21.15.44:0/2930771296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc0039e0 msgr2=0x7f12fc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:36.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130bfff640 1 --2- 172.21.15.44:0/2930771296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc0039e0 0x7f12fc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f130bfff640 1 -- 172.21.15.44:0/2930771296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12fc003520 con 0x7f12fc004800 2024-04-02T13:24:36.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.665+0000 7f1311333640 1 --2- 172.21.15.44:0/2930771296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f130c0d2750 0x7f12fc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:36.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.666+0000 7f130bfff640 1 --2- 172.21.15.44:0/2930771296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc004800 0x7f12fc004c80 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f12f8001eb0 tx=0x7f12fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400e6a196476be87 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.666+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f8007d30 con 0x7f12fc004800 2024-04-02T13:24:36.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.666+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12f8030070 con 0x7f12fc004800 2024-04-02T13:24:36.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.666+0000 7f130b7fe640 1 -- 172.21.15.44:0/2930771296 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f8005d30 con 0x7f12fc004800 2024-04-02T13:24:36.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.667+0000 7f1312d96640 1 -- 172.21.15.44:0/2930771296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc004800 msgr2=0x7f12fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.667+0000 7f1312d96640 1 --2- 172.21.15.44:0/2930771296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12fc004800 0x7f12fc004c80 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f12f8001eb0 tx=0x7f12fc005830 comp rx=0 tx=0).stop 2024-04-02T13:24:36.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.667+0000 7f1311333640 1 -- 172.21.15.44:0/2930771296 reap_dead start 2024-04-02T13:24:36.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.667+0000 7f1312d96640 1 -- 172.21.15.44:0/2930771296 shutdown_connections 2024-04-02T13:24:36.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.667+0000 7f1312d96640 1 -- 172.21.15.44:0/2930771296 >> 172.21.15.44:0/2930771296 conn(0x7f130c09d470 msgr2=0x7f130c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:36.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.667+0000 7f1312d96640 1 -- 172.21.15.44:0/2930771296 shutdown_connections 2024-04-02T13:24:36.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.667+0000 7f1312d96640 1 -- 172.21.15.44:0/2930771296 wait complete. 2024-04-02T13:24:36.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.668+0000 7f1312d96640 1 Processor -- start 2024-04-02T13:24:36.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.668+0000 7f1312d96640 1 -- start start 2024-04-02T13:24:36.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f1312d96640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc004800 0x7f130c1f91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f1312d96640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f130c0b6150 0x7f130c0c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f1312d96640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f130c0b65b0 0x7f130c0ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f1312d96640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f130c0cf380 con 0x7f130c0b6150 2024-04-02T13:24:36.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f1312d96640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f130c0cf6f0 con 0x7f12fc004800 2024-04-02T13:24:36.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f1312d96640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f130c0b38a0 con 0x7f130c0b65b0 2024-04-02T13:24:36.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f130bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f130c0b6150 0x7f130c0c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f130bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f130c0b6150 0x7f130c0c9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47544/0 (socket says 172.21.15.44:47544) 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f130bfff640 1 -- 172.21.15.44:0/1799552385 learned_addr learned my addr 172.21.15.44:0/1799552385 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f130bfff640 1 -- 172.21.15.44:0/1799552385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f130c0b65b0 msgr2=0x7f130c0ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f130bfff640 1 --2- 172.21.15.44:0/1799552385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f130c0b65b0 0x7f130c0ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f130bfff640 1 -- 172.21.15.44:0/1799552385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc004800 msgr2=0x7f130c1f91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f1310b32640 1 --2- 172.21.15.44:0/1799552385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc004800 0x7f130c1f91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f130bfff640 1 --2- 172.21.15.44:0/1799552385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc004800 0x7f130c1f91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f130bfff640 1 -- 172.21.15.44:0/1799552385 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130c0ca630 con 0x7f130c0b6150 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f130bfff640 1 --2- 172.21.15.44:0/1799552385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f130c0b6150 0x7f130c0c9bb0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f12f802c510 tx=0x7f12f802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f13097fa640 1 -- 172.21.15.44:0/1799552385 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f8004bf0 con 0x7f130c0b6150 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.669+0000 7f13097fa640 1 -- 172.21.15.44:0/1799552385 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12f8004d90 con 0x7f130c0b6150 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.670+0000 7f1312d96640 1 -- 172.21.15.44:0/1799552385 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f130c0c8df0 con 0x7f130c0b6150 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.670+0000 7f13097fa640 1 -- 172.21.15.44:0/1799552385 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f8007290 con 0x7f130c0b6150 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.670+0000 7f1312d96640 1 -- 172.21.15.44:0/1799552385 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f130c0c93b0 con 0x7f130c0b6150 2024-04-02T13:24:36.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.670+0000 7f1312d96640 1 -- 172.21.15.44:0/1799552385 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f130c0684f0 con 0x7f130c0b6150 2024-04-02T13:24:36.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.671+0000 7f13097fa640 1 -- 172.21.15.44:0/1799552385 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f12f8048020 con 0x7f130c0b6150 2024-04-02T13:24:36.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.671+0000 7f13097fa640 1 --2- 172.21.15.44:0/1799552385 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f12f006c920 0x7f12f006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.671+0000 7f13097fa640 1 -- 172.21.15.44:0/1799552385 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f12f80acec0 con 0x7f130c0b6150 2024-04-02T13:24:36.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.672+0000 7f1310b32640 1 --2- 172.21.15.44:0/1799552385 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f12f006c920 0x7f12f006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.672+0000 7f1310b32640 1 --2- 172.21.15.44:0/1799552385 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f12f006c920 0x7f12f006ede0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1300005510 tx=0x7f1300009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.676+0000 7f13097fa640 1 -- 172.21.15.44:0/1799552385 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f12f807b160 con 0x7f130c0b6150 2024-04-02T13:24:36.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.864+0000 7f1312d96640 1 -- 172.21.15.44:0/1799552385 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f130c0d1760 con 0x7f130c0b6150 2024-04-02T13:24:36.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.865+0000 7f13097fa640 1 -- 172.21.15.44:0/1799552385 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+17076 (secure 0 0 0) 0x7f12f807aa60 con 0x7f130c0b6150 2024-04-02T13:24:36.867 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:36.867 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":67,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:24:09.591968+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:23:23.126712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-02T13:23:23.763593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.18","mappings":[{"from":7,"to":2}]},{"pgid":"10.4","mappings":[{"from":4,"to":3}]},{"pgid":"10.10","mappings":[{"from":4,"to":3}]},{"pgid":"10.37","mappings":[{"from":4,"to":5}]},{"pgid":"10.39","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:24:36.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.867+0000 7f1312d96640 1 -- 172.21.15.44:0/1799552385 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f12f006c920 msgr2=0x7f12f006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.867+0000 7f1312d96640 1 --2- 172.21.15.44:0/1799552385 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f12f006c920 0x7f12f006ede0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1300005510 tx=0x7f1300009040 comp rx=0 tx=0).stop 2024-04-02T13:24:36.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.867+0000 7f1312d96640 1 -- 172.21.15.44:0/1799552385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f130c0b6150 msgr2=0x7f130c0c9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.867+0000 7f1312d96640 1 --2- 172.21.15.44:0/1799552385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f130c0b6150 0x7f130c0c9bb0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f12f802c510 tx=0x7f12f802e070 comp rx=0 tx=0).stop 2024-04-02T13:24:36.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.867+0000 7f1312d96640 1 -- 172.21.15.44:0/1799552385 shutdown_connections 2024-04-02T13:24:36.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.867+0000 7f1312d96640 1 --2- 172.21.15.44:0/1799552385 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f12f006c920 0x7f12f006ede0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.867+0000 7f1312d96640 1 --2- 172.21.15.44:0/1799552385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f130c0b65b0 0x7f130c0ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.867+0000 7f1312d96640 1 --2- 172.21.15.44:0/1799552385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f130c0b6150 0x7f130c0c9bb0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.867+0000 7f1312d96640 1 --2- 172.21.15.44:0/1799552385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12fc004800 0x7f130c1f91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.867+0000 7f1312d96640 1 -- 172.21.15.44:0/1799552385 >> 172.21.15.44:0/1799552385 conn(0x7f130c09d470 msgr2=0x7f130c0b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:36.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.867+0000 7f1312d96640 1 -- 172.21.15.44:0/1799552385 shutdown_connections 2024-04-02T13:24:36.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.867+0000 7f1312d96640 1 -- 172.21.15.44:0/1799552385 wait complete. 2024-04-02T13:24:36.880 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:23:23.126712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:23:23.763593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:24:36.880 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:24:36.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.983+0000 7f491aa88640 1 Processor -- start 2024-04-02T13:24:36.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.983+0000 7f491aa88640 1 -- start start 2024-04-02T13:24:36.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.984+0000 7f491aa88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f491406cff0 0x7f491406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.984+0000 7f491aa88640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f49140b4180 0x7f49140b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.984+0000 7f491aa88640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f491406d910 con 0x7f491415e5d0 2024-04-02T13:24:36.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.984+0000 7f491aa88640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f49140d1bf0 con 0x7f49140b4180 2024-04-02T13:24:36.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.984+0000 7f491aa88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f49140d1630 con 0x7f491406cff0 2024-04-02T13:24:36.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.984+0000 7f4919025640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f49140b4180 0x7f49140b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.984+0000 7f4919025640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f49140b4180 0x7f49140b1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47546/0 (socket says 172.21.15.44:47546) 2024-04-02T13:24:36.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.984+0000 7f4919025640 1 -- 172.21.15.44:0/3952431529 learned_addr learned my addr 172.21.15.44:0/3952431529 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:36.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.984+0000 7f4918824640 1 --2- 172.21.15.44:0/3952431529 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f491406cff0 0x7f491406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.984+0000 7f4919025640 1 -- 172.21.15.44:0/3952431529 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f491406cff0 msgr2=0x7f491406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:36.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.984+0000 7f4919025640 1 --2- 172.21.15.44:0/3952431529 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f491406cff0 0x7f491406d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.984+0000 7f4919025640 1 -- 172.21.15.44:0/3952431529 >> v1:172.21.15.69:6789/0 conn(0x7f491415e5d0 legacy=0x7f49140d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.984+0000 7f4919025640 1 -- 172.21.15.44:0/3952431529 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49140d1910 con 0x7f49140b4180 2024-04-02T13:24:36.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.985+0000 7f4919025640 1 --2- 172.21.15.44:0/3952431529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49140b4180 0x7f49140b1d40 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009870 tx=0x7f48fc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289d39d5b20ba52c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.985+0000 7f49137fe640 1 -- 172.21.15.44:0/3952431529 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48fc035020 con 0x7f49140b4180 2024-04-02T13:24:36.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.985+0000 7f49137fe640 1 -- 172.21.15.44:0/3952431529 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48fc031640 con 0x7f49140b4180 2024-04-02T13:24:36.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.985+0000 7f49137fe640 1 -- 172.21.15.44:0/3952431529 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48fc02d040 con 0x7f49140b4180 2024-04-02T13:24:36.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.986+0000 7f491aa88640 1 -- 172.21.15.44:0/3952431529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49140b4180 msgr2=0x7f49140b1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.986+0000 7f491aa88640 1 --2- 172.21.15.44:0/3952431529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49140b4180 0x7f49140b1d40 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009870 tx=0x7f48fc02b8f0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.986+0000 7f491aa88640 1 -- 172.21.15.44:0/3952431529 shutdown_connections 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.986+0000 7f491aa88640 1 --2- 172.21.15.44:0/3952431529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49140b4180 0x7f49140b1d40 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.986+0000 7f491aa88640 1 --2- 172.21.15.44:0/3952431529 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f491406cff0 0x7f491406d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.986+0000 7f491aa88640 1 -- 172.21.15.44:0/3952431529 >> 172.21.15.44:0/3952431529 conn(0x7f491409aa20 msgr2=0x7f491409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.986+0000 7f491aa88640 1 -- 172.21.15.44:0/3952431529 shutdown_connections 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.986+0000 7f491aa88640 1 -- 172.21.15.44:0/3952431529 wait complete. 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.987+0000 7f491aa88640 1 Processor -- start 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.987+0000 7f491aa88640 1 -- start start 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f491aa88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491406cff0 0x7f491414e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f491aa88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140b4180 0x7f491414ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f491aa88640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491415e5d0 0x7f49140c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f491aa88640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f49140b2280 con 0x7f491415e5d0 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f491aa88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f49140b1820 con 0x7f491406cff0 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f491aa88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f49140b2400 con 0x7f49140b4180 2024-04-02T13:24:36.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f4919025640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491415e5d0 0x7f49140c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f4919025640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491415e5d0 0x7f49140c7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47560/0 (socket says 172.21.15.44:47560) 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f4919025640 1 -- 172.21.15.44:0/3986994409 learned_addr learned my addr 172.21.15.44:0/3986994409 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f4918824640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491406cff0 0x7f491414e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f4913fff640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140b4180 0x7f491414ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f4919025640 1 -- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140b4180 msgr2=0x7f491414ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f4919025640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140b4180 0x7f491414ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f4919025640 1 -- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491406cff0 msgr2=0x7f491414e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f4919025640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491406cff0 0x7f491414e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.988+0000 7f4919025640 1 -- 172.21.15.44:0/3986994409 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f491415f680 con 0x7f491415e5d0 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.989+0000 7f4919025640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491415e5d0 0x7f49140c7b20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009870 tx=0x7f48fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.989+0000 7f49117fa640 1 -- 172.21.15.44:0/3986994409 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48fc040070 con 0x7f491415e5d0 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.989+0000 7f49117fa640 1 -- 172.21.15.44:0/3986994409 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48fc0399f0 con 0x7f491415e5d0 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.989+0000 7f491aa88640 1 -- 172.21.15.44:0/3986994409 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4914160230 con 0x7f491415e5d0 2024-04-02T13:24:36.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.989+0000 7f49117fa640 1 -- 172.21.15.44:0/3986994409 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48fc035040 con 0x7f491415e5d0 2024-04-02T13:24:36.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.989+0000 7f491aa88640 1 -- 172.21.15.44:0/3986994409 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f491414e8f0 con 0x7f491415e5d0 2024-04-02T13:24:36.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.990+0000 7f491aa88640 1 -- 172.21.15.44:0/3986994409 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49140684f0 con 0x7f491415e5d0 2024-04-02T13:24:36.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.990+0000 7f49117fa640 1 -- 172.21.15.44:0/3986994409 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f48fc04b060 con 0x7f491415e5d0 2024-04-02T13:24:36.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.991+0000 7f49117fa640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48f406c8f0 0x7f48f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:36.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.991+0000 7f49117fa640 1 -- 172.21.15.44:0/3986994409 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f48fc0ad4e0 con 0x7f491415e5d0 2024-04-02T13:24:36.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.991+0000 7f4918824640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48f406c8f0 0x7f48f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:36.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.991+0000 7f4918824640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48f406c8f0 0x7f48f406edb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f490c006fd0 tx=0x7f490c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:36.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:36.995+0000 7f49117fa640 1 -- 172.21.15.44:0/3986994409 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f48fc07b160 con 0x7f491415e5d0 2024-04-02T13:24:37.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.176+0000 7f491aa88640 1 -- 172.21.15.44:0/3986994409 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f491406cdd0 con 0x7f491415e5d0 2024-04-02T13:24:37.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.177+0000 7f49117fa640 1 -- 172.21.15.44:0/3986994409 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) ==== 93+0+10 (secure 0 0 0) 0x7f48fc07f460 con 0x7f491415e5d0 2024-04-02T13:24:37.179 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:24:37.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.179+0000 7f491aa88640 1 -- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48f406c8f0 msgr2=0x7f48f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.179+0000 7f491aa88640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48f406c8f0 0x7f48f406edb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f490c006fd0 tx=0x7f490c008040 comp rx=0 tx=0).stop 2024-04-02T13:24:37.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.179+0000 7f491aa88640 1 -- 172.21.15.44:0/3986994409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491415e5d0 msgr2=0x7f49140c7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.179+0000 7f491aa88640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491415e5d0 0x7f49140c7b20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009870 tx=0x7f48fc0046c0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.179+0000 7f491aa88640 1 -- 172.21.15.44:0/3986994409 shutdown_connections 2024-04-02T13:24:37.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.179+0000 7f491aa88640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f48f406c8f0 0x7f48f406edb0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.179+0000 7f491aa88640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491415e5d0 0x7f49140c7b20 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.179+0000 7f491aa88640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140b4180 0x7f491414ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.179+0000 7f491aa88640 1 --2- 172.21.15.44:0/3986994409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491406cff0 0x7f491414e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.179+0000 7f491aa88640 1 -- 172.21.15.44:0/3986994409 >> 172.21.15.44:0/3986994409 conn(0x7f491409aa20 msgr2=0x7f49140d01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.179+0000 7f491aa88640 1 -- 172.21.15.44:0/3986994409 shutdown_connections 2024-04-02T13:24:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.179+0000 7f491aa88640 1 -- 172.21.15.44:0/3986994409 wait complete. 2024-04-02T13:24:37.192 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:24:37.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.294+0000 7f1ca6faa640 1 Processor -- start 2024-04-02T13:24:37.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.294+0000 7f1ca6faa640 1 -- start start 2024-04-02T13:24:37.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.295+0000 7f1ca6faa640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1ca015e090 0x7f1ca006df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.295+0000 7f1ca6faa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ca006bbe0 0x7f1ca00d1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.295+0000 7f1ca6faa640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ca006e590 con 0x7f1ca0072890 2024-04-02T13:24:37.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.295+0000 7f1ca6faa640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f1ca00d0cf0 con 0x7f1ca015e090 2024-04-02T13:24:37.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.295+0000 7f1ca6faa640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca006f860 con 0x7f1ca006bbe0 2024-04-02T13:24:37.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.295+0000 7f1ca4d46640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1ca015e090 0x7f1ca006df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.295+0000 7f1c9ffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f1ca0072890 0x7f1ca006b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48080/0 (socket says 172.21.15.44:48080) 2024-04-02T13:24:37.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.295+0000 7f1ca4d46640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1ca015e090 0x7f1ca006df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47576/0 (socket says 172.21.15.44:47576) 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.295+0000 7f1c9ffff640 1 -- 172.21.15.44:0/706392993 learned_addr learned my addr 172.21.15.44:0/706392993 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.295+0000 7f1ca5547640 1 --2- 172.21.15.44:0/706392993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ca006bbe0 0x7f1ca00d1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.296+0000 7f1ca4d46640 1 -- 172.21.15.44:0/706392993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ca006bbe0 msgr2=0x7f1ca00d1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.296+0000 7f1ca4d46640 1 --2- 172.21.15.44:0/706392993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ca006bbe0 0x7f1ca00d1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.296+0000 7f1ca4d46640 1 -- 172.21.15.44:0/706392993 >> v1:172.21.15.44:6789/0 conn(0x7f1ca0072890 legacy=0x7f1ca006b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.296+0000 7f1ca4d46640 1 -- 172.21.15.44:0/706392993 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca006fb40 con 0x7f1ca015e090 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.296+0000 7f1c9f7fe640 1 -- 172.21.15.44:0/706392993 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 395396717 0 0) 0x7f1ca006e590 con 0x7f1ca0072890 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.296+0000 7f1ca4d46640 1 --2- 172.21.15.44:0/706392993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ca015e090 0x7f1ca006df20 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f1c980097a0 tx=0x7f1c9802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66fa207fc2f3691 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.296+0000 7f1c9f7fe640 1 -- 172.21.15.44:0/706392993 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c980315a0 con 0x7f1ca015e090 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.296+0000 7f1c9f7fe640 1 -- 172.21.15.44:0/706392993 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c98037070 con 0x7f1ca015e090 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.296+0000 7f1c9f7fe640 1 -- 172.21.15.44:0/706392993 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c98031960 con 0x7f1ca015e090 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.297+0000 7f1ca6faa640 1 -- 172.21.15.44:0/706392993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ca015e090 msgr2=0x7f1ca006df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.297+0000 7f1ca6faa640 1 --2- 172.21.15.44:0/706392993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ca015e090 0x7f1ca006df20 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f1c980097a0 tx=0x7f1c9802b840 comp rx=0 tx=0).stop 2024-04-02T13:24:37.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.297+0000 7f1ca6faa640 1 -- 172.21.15.44:0/706392993 shutdown_connections 2024-04-02T13:24:37.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.297+0000 7f1ca6faa640 1 --2- 172.21.15.44:0/706392993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ca006bbe0 0x7f1ca00d1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.297+0000 7f1ca6faa640 1 --2- 172.21.15.44:0/706392993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ca015e090 0x7f1ca006df20 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.297+0000 7f1ca6faa640 1 -- 172.21.15.44:0/706392993 >> 172.21.15.44:0/706392993 conn(0x7f1ca006c5e0 msgr2=0x7f1ca006c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:37.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.297+0000 7f1ca6faa640 1 -- 172.21.15.44:0/706392993 shutdown_connections 2024-04-02T13:24:37.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.297+0000 7f1ca6faa640 1 -- 172.21.15.44:0/706392993 wait complete. 2024-04-02T13:24:37.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.298+0000 7f1ca6faa640 1 Processor -- start 2024-04-02T13:24:37.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.298+0000 7f1ca6faa640 1 -- start start 2024-04-02T13:24:37.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.299+0000 7f1ca6faa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ca006bbe0 0x7f1ca014d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.299+0000 7f1ca6faa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ca0072890 0x7f1ca00c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.299+0000 7f1ca6faa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ca015e090 0x7f1ca014e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.299+0000 7f1ca6faa640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca015e5f0 con 0x7f1ca015e090 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.299+0000 7f1ca6faa640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca015e470 con 0x7f1ca0072890 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.299+0000 7f1ca6faa640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca00b3dd0 con 0x7f1ca006bbe0 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.299+0000 7f1ca5547640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ca015e090 0x7f1ca014e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.299+0000 7f1ca5547640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ca015e090 0x7f1ca014e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47588/0 (socket says 172.21.15.44:47588) 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.299+0000 7f1ca5547640 1 -- 172.21.15.44:0/1577963277 learned_addr learned my addr 172.21.15.44:0/1577963277 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.299+0000 7f1c9ffff640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ca0072890 0x7f1ca00c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.299+0000 7f1ca4d46640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ca006bbe0 0x7f1ca014d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.300+0000 7f1ca5547640 1 -- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ca006bbe0 msgr2=0x7f1ca014d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.300+0000 7f1ca5547640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ca006bbe0 0x7f1ca014d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.300+0000 7f1ca5547640 1 -- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ca0072890 msgr2=0x7f1ca00c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.300+0000 7f1ca5547640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ca0072890 0x7f1ca00c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.300+0000 7f1ca5547640 1 -- 172.21.15.44:0/1577963277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca014eae0 con 0x7f1ca015e090 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.300+0000 7f1ca5547640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ca015e090 0x7f1ca014e090 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800e7c0 tx=0x7f1c8800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:37.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.300+0000 7f1c9d7fa640 1 -- 172.21.15.44:0/1577963277 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c88017070 con 0x7f1ca015e090 2024-04-02T13:24:37.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.300+0000 7f1c9d7fa640 1 -- 172.21.15.44:0/1577963277 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c8800c730 con 0x7f1ca015e090 2024-04-02T13:24:37.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.300+0000 7f1ca6faa640 1 -- 172.21.15.44:0/1577963277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca014edc0 con 0x7f1ca015e090 2024-04-02T13:24:37.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.300+0000 7f1c9d7fa640 1 -- 172.21.15.44:0/1577963277 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8800cb20 con 0x7f1ca015e090 2024-04-02T13:24:37.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.300+0000 7f1ca6faa640 1 -- 172.21.15.44:0/1577963277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca014f380 con 0x7f1ca015e090 2024-04-02T13:24:37.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.301+0000 7f1ca6faa640 1 -- 172.21.15.44:0/1577963277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ca0068c00 con 0x7f1ca015e090 2024-04-02T13:24:37.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.301+0000 7f1c9d7fa640 1 -- 172.21.15.44:0/1577963277 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1c8801a070 con 0x7f1ca015e090 2024-04-02T13:24:37.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.302+0000 7f1c9d7fa640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1c8006c8f0 0x7f1c8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.302+0000 7f1c9d7fa640 1 -- 172.21.15.44:0/1577963277 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f1c8808bb20 con 0x7f1ca015e090 2024-04-02T13:24:37.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.302+0000 7f1ca4d46640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1c8006c8f0 0x7f1c8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.303+0000 7f1ca4d46640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1c8006c8f0 0x7f1c8006edb0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1c98036000 tx=0x7f1c9803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:37.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.306+0000 7f1c9d7fa640 1 -- 172.21.15.44:0/1577963277 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1c880596e0 con 0x7f1ca015e090 2024-04-02T13:24:37.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.493+0000 7f1ca6faa640 1 -- 172.21.15.44:0/1577963277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1ca00754a0 con 0x7f1ca015e090 2024-04-02T13:24:37.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.493+0000 7f1c9d7fa640 1 -- 172.21.15.44:0/1577963277 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) ==== 104+0+11 (secure 0 0 0) 0x7f1c8805d9e0 con 0x7f1ca015e090 2024-04-02T13:24:37.496 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:24:37.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.495+0000 7f1ca6faa640 1 -- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1c8006c8f0 msgr2=0x7f1c8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.495+0000 7f1ca6faa640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1c8006c8f0 0x7f1c8006edb0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1c98036000 tx=0x7f1c9803d040 comp rx=0 tx=0).stop 2024-04-02T13:24:37.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.495+0000 7f1ca6faa640 1 -- 172.21.15.44:0/1577963277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ca015e090 msgr2=0x7f1ca014e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.495+0000 7f1ca6faa640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ca015e090 0x7f1ca014e090 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800e7c0 tx=0x7f1c8800ec80 comp rx=0 tx=0).stop 2024-04-02T13:24:37.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.496+0000 7f1ca6faa640 1 -- 172.21.15.44:0/1577963277 shutdown_connections 2024-04-02T13:24:37.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.496+0000 7f1ca6faa640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1c8006c8f0 0x7f1c8006edb0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.496+0000 7f1ca6faa640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ca015e090 0x7f1ca014e090 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.496+0000 7f1ca6faa640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ca0072890 0x7f1ca00c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.496+0000 7f1ca6faa640 1 --2- 172.21.15.44:0/1577963277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ca006bbe0 0x7f1ca014d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.496+0000 7f1ca6faa640 1 -- 172.21.15.44:0/1577963277 >> 172.21.15.44:0/1577963277 conn(0x7f1ca006c5e0 msgr2=0x7f1ca006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:37.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.496+0000 7f1ca6faa640 1 -- 172.21.15.44:0/1577963277 shutdown_connections 2024-04-02T13:24:37.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.496+0000 7f1ca6faa640 1 -- 172.21.15.44:0/1577963277 wait complete. 2024-04-02T13:24:37.508 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:24:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.610+0000 7f993708f640 1 Processor -- start 2024-04-02T13:24:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.610+0000 7f993708f640 1 -- start start 2024-04-02T13:24:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.611+0000 7f993708f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99300d0a30 0x7f99300b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.611+0000 7f993708f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99300d0e10 con 0x7f99300b3990 2024-04-02T13:24:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.611+0000 7f993708f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9930144750 con 0x7f9930144130 2024-04-02T13:24:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.611+0000 7f993708f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f99300b3df0 con 0x7f99300d0a30 2024-04-02T13:24:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.611+0000 7f992ffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f99300b3990 0x7f9930143a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48090/0 (socket says 172.21.15.44:48090) 2024-04-02T13:24:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.611+0000 7f992ffff640 1 -- 172.21.15.44:0/2347864228 learned_addr learned my addr 172.21.15.44:0/2347864228 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.611+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3332551897 0 0) 0x7f9918001110 con 0x7f99300b3990 2024-04-02T13:24:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.611+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f991c003520 con 0x7f99300b3990 2024-04-02T13:24:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.611+0000 7f9934e2b640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99300d0a30 0x7f99300b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2160260718 0 0) 0x7f991c003520 con 0x7f99300b3990 2024-04-02T13:24:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99300d0a30 msgr2=0x7f99300b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992f7fe640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99300d0a30 0x7f99300b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 >> v1:172.21.15.69:6789/0 conn(0x7f9930144130 legacy=0x7f993015e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9930162060 con 0x7f99300b3990 2024-04-02T13:24:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f9918001840 con 0x7f99300b3990 2024-04-02T13:24:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 >> v1:172.21.15.44:6789/0 conn(0x7f99300b3990 legacy=0x7f9930143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992f7fe640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99300d0a30 0x7f991c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992f7fe640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f991c003f60 0x7f991c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992f7fe640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f991c004d40 0x7f991c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f991c00bb50 con 0x7f991c003f60 2024-04-02T13:24:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f991c00bcd0 con 0x7f991c004d40 2024-04-02T13:24:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f991c00bed0 con 0x7f99300d0a30 2024-04-02T13:24:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992ffff640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f991c003f60 0x7f991c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f993562c640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f991c004d40 0x7f991c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f9934e2b640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99300d0a30 0x7f991c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992ffff640 1 -- 172.21.15.44:0/2347864228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99300d0a30 msgr2=0x7f991c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992ffff640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99300d0a30 0x7f991c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992ffff640 1 -- 172.21.15.44:0/2347864228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f991c004d40 msgr2=0x7f991c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992ffff640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f991c004d40 0x7f991c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f992ffff640 1 -- 172.21.15.44:0/2347864228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f991c003520 con 0x7f991c003f60 2024-04-02T13:24:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f993562c640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f991c004d40 0x7f991c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.612+0000 7f9934e2b640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99300d0a30 0x7f991c0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.613+0000 7f992ffff640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f991c003f60 0x7f991c004800 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f991800c4c0 tx=0x7f991c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2affc29150df67 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.613+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991800c0e0 con 0x7f991c003f60 2024-04-02T13:24:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.613+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9918030070 con 0x7f991c003f60 2024-04-02T13:24:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.613+0000 7f992f7fe640 1 -- 172.21.15.44:0/2347864228 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99180058d0 con 0x7f991c003f60 2024-04-02T13:24:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.614+0000 7f993708f640 1 -- 172.21.15.44:0/2347864228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f991c003f60 msgr2=0x7f991c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.614+0000 7f993708f640 1 --2- 172.21.15.44:0/2347864228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f991c003f60 0x7f991c004800 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f991800c4c0 tx=0x7f991c005830 comp rx=0 tx=0).stop 2024-04-02T13:24:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.614+0000 7f993562c640 1 -- 172.21.15.44:0/2347864228 reap_dead start 2024-04-02T13:24:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.614+0000 7f993708f640 1 -- 172.21.15.44:0/2347864228 shutdown_connections 2024-04-02T13:24:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.614+0000 7f993708f640 1 -- 172.21.15.44:0/2347864228 >> 172.21.15.44:0/2347864228 conn(0x7f99300b5d90 msgr2=0x7f993006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.614+0000 7f993708f640 1 -- 172.21.15.44:0/2347864228 shutdown_connections 2024-04-02T13:24:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.614+0000 7f993708f640 1 -- 172.21.15.44:0/2347864228 wait complete. 2024-04-02T13:24:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.615+0000 7f993708f640 1 Processor -- start 2024-04-02T13:24:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.615+0000 7f993708f640 1 -- start start 2024-04-02T13:24:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993708f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f991c003f60 0x7f99301e81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993708f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99300b0690 0x7f993012fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993708f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99301302b0 0x7f993012ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993708f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9930162620 con 0x7f99301302b0 2024-04-02T13:24:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993708f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9930161e10 con 0x7f99300b0690 2024-04-02T13:24:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993708f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9930162b20 con 0x7f991c003f60 2024-04-02T13:24:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993562c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99301302b0 0x7f993012ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993562c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99301302b0 0x7f993012ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47598/0 (socket says 172.21.15.44:47598) 2024-04-02T13:24:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993562c640 1 -- 172.21.15.44:0/2435360545 learned_addr learned my addr 172.21.15.44:0/2435360545 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f992ffff640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99300b0690 0x7f993012fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f9934e2b640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f991c003f60 0x7f99301e81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993562c640 1 -- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f991c003f60 msgr2=0x7f99301e81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993562c640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f991c003f60 0x7f99301e81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993562c640 1 -- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99300b0690 msgr2=0x7f993012fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993562c640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99300b0690 0x7f993012fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993562c640 1 -- 172.21.15.44:0/2435360545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9930141120 con 0x7f99301302b0 2024-04-02T13:24:37.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993562c640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99301302b0 0x7f993012ee20 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f9930144510 tx=0x7f992000e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:37.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f992d7fa640 1 -- 172.21.15.44:0/2435360545 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9920019070 con 0x7f99301302b0 2024-04-02T13:24:37.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f992d7fa640 1 -- 172.21.15.44:0/2435360545 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99200052e0 con 0x7f99301302b0 2024-04-02T13:24:37.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f992d7fa640 1 -- 172.21.15.44:0/2435360545 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9920003830 con 0x7f99301302b0 2024-04-02T13:24:37.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993708f640 1 -- 172.21.15.44:0/2435360545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f993012dd80 con 0x7f99301302b0 2024-04-02T13:24:37.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.616+0000 7f993708f640 1 -- 172.21.15.44:0/2435360545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f993012e340 con 0x7f99301302b0 2024-04-02T13:24:37.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.617+0000 7f993708f640 1 -- 172.21.15.44:0/2435360545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99300684f0 con 0x7f99301302b0 2024-04-02T13:24:37.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.618+0000 7f992d7fa640 1 -- 172.21.15.44:0/2435360545 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f99200039d0 con 0x7f99301302b0 2024-04-02T13:24:37.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.618+0000 7f992d7fa640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f991006c920 0x7f991006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.618+0000 7f992d7fa640 1 -- 172.21.15.44:0/2435360545 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f992008b050 con 0x7f99301302b0 2024-04-02T13:24:37.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.619+0000 7f9934e2b640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f991006c920 0x7f991006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.619+0000 7f9934e2b640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f991006c920 0x7f991006ede0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9928007520 tx=0x7f992800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:37.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.623+0000 7f992d7fa640 1 -- 172.21.15.44:0/2435360545 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9920059250 con 0x7f99301302b0 2024-04-02T13:24:37.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.810+0000 7f993708f640 1 -- 172.21.15.44:0/2435360545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f99300b0a70 con 0x7f99301302b0 2024-04-02T13:24:37.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.810+0000 7f992d7fa640 1 -- 172.21.15.44:0/2435360545 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) ==== 100+0+11 (secure 0 0 0) 0x7f9920058b50 con 0x7f99301302b0 2024-04-02T13:24:37.813 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:24:37.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.812+0000 7f993708f640 1 -- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f991006c920 msgr2=0x7f991006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.812+0000 7f993708f640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f991006c920 0x7f991006ede0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9928007520 tx=0x7f992800b040 comp rx=0 tx=0).stop 2024-04-02T13:24:37.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.812+0000 7f993708f640 1 -- 172.21.15.44:0/2435360545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99301302b0 msgr2=0x7f993012ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.812+0000 7f993708f640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99301302b0 0x7f993012ee20 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f9930144510 tx=0x7f992000e820 comp rx=0 tx=0).stop 2024-04-02T13:24:37.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.812+0000 7f993708f640 1 -- 172.21.15.44:0/2435360545 shutdown_connections 2024-04-02T13:24:37.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.812+0000 7f993708f640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f991006c920 0x7f991006ede0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.812+0000 7f993708f640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99301302b0 0x7f993012ee20 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.812+0000 7f993708f640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99300b0690 0x7f993012fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.812+0000 7f993708f640 1 --2- 172.21.15.44:0/2435360545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f991c003f60 0x7f99301e81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.812+0000 7f993708f640 1 -- 172.21.15.44:0/2435360545 >> 172.21.15.44:0/2435360545 conn(0x7f99300b5d90 msgr2=0x7f99300adf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:37.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.813+0000 7f993708f640 1 -- 172.21.15.44:0/2435360545 shutdown_connections 2024-04-02T13:24:37.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.813+0000 7f993708f640 1 -- 172.21.15.44:0/2435360545 wait complete. 2024-04-02T13:24:37.826 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:24:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.926+0000 7f20aafe9640 1 Processor -- start 2024-04-02T13:24:37.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.926+0000 7f20aafe9640 1 -- start start 2024-04-02T13:24:37.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.927+0000 7f20aafe9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f20a40d0a30 0x7f20a406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.927+0000 7f20aafe9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20a40d0000 con 0x7f20a40d0e10 2024-04-02T13:24:37.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.927+0000 7f20aafe9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20a406c1d0 con 0x7f20a40d2040 2024-04-02T13:24:37.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.927+0000 7f20aafe9640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f20a406cb00 con 0x7f20a40d0a30 2024-04-02T13:24:37.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.927+0000 7f20a8d85640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f20a40d0a30 0x7f20a406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.927+0000 7f20a3fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f20a40d0e10 0x7f20a40cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48104/0 (socket says 172.21.15.44:48104) 2024-04-02T13:24:37.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.927+0000 7f20a3fff640 1 -- 172.21.15.44:0/2569829701 learned_addr learned my addr 172.21.15.44:0/2569829701 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:37.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.928+0000 7f20a8d85640 1 -- 172.21.15.44:0/2569829701 >> v1:172.21.15.69:6789/0 conn(0x7f20a40d2040 legacy=0x7f20a40d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.928+0000 7f20a8d85640 1 -- 172.21.15.44:0/2569829701 >> v1:172.21.15.44:6789/0 conn(0x7f20a40d0e10 legacy=0x7f20a40cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.928+0000 7f20a8d85640 1 -- 172.21.15.44:0/2569829701 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a406c4b0 con 0x7f20a40d0a30 2024-04-02T13:24:37.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.928+0000 7f20a37fe640 1 -- 172.21.15.44:0/2569829701 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2606965389 0 0) 0x7f20a40d0000 con 0x7f20a40d0e10 2024-04-02T13:24:37.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.928+0000 7f20a8d85640 1 --2- 172.21.15.44:0/2569829701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20a40d0a30 0x7f20a406cf90 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f209c009d90 tx=0x7f209c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6009eae2d6495f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:37.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.929+0000 7f20a37fe640 1 -- 172.21.15.44:0/2569829701 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209c039070 con 0x7f20a40d0a30 2024-04-02T13:24:37.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.929+0000 7f20a37fe640 1 -- 172.21.15.44:0/2569829701 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f209c02cc60 con 0x7f20a40d0a30 2024-04-02T13:24:37.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.929+0000 7f20a37fe640 1 -- 172.21.15.44:0/2569829701 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209c034660 con 0x7f20a40d0a30 2024-04-02T13:24:37.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.929+0000 7f20aafe9640 1 -- 172.21.15.44:0/2569829701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20a40d0a30 msgr2=0x7f20a406cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.929+0000 7f20aafe9640 1 --2- 172.21.15.44:0/2569829701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20a40d0a30 0x7f20a406cf90 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f209c009d90 tx=0x7f209c02beb0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.929+0000 7f20aafe9640 1 -- 172.21.15.44:0/2569829701 shutdown_connections 2024-04-02T13:24:37.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.929+0000 7f20aafe9640 1 --2- 172.21.15.44:0/2569829701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20a40d0a30 0x7f20a406cf90 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.929+0000 7f20aafe9640 1 -- 172.21.15.44:0/2569829701 >> 172.21.15.44:0/2569829701 conn(0x7f20a409d470 msgr2=0x7f20a40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:37.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.929+0000 7f20aafe9640 1 -- 172.21.15.44:0/2569829701 shutdown_connections 2024-04-02T13:24:37.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.929+0000 7f20aafe9640 1 -- 172.21.15.44:0/2569829701 wait complete. 2024-04-02T13:24:37.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.930+0000 7f20aafe9640 1 Processor -- start 2024-04-02T13:24:37.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.930+0000 7f20aafe9640 1 -- start start 2024-04-02T13:24:37.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.931+0000 7f20aafe9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20a40d0a30 0x7f20a40cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.931+0000 7f20aafe9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20a40d0e10 0x7f20a40ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.931+0000 7f20aafe9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20a40d2040 0x7f20a4159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.931+0000 7f20aafe9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f20a40d2c00 con 0x7f20a40d0e10 2024-04-02T13:24:37.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.931+0000 7f20aafe9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f20a40d1270 con 0x7f20a40d2040 2024-04-02T13:24:37.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.931+0000 7f20aafe9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f20a415e6f0 con 0x7f20a40d0a30 2024-04-02T13:24:37.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.931+0000 7f20a3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20a40d0e10 0x7f20a40ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.931+0000 7f20a3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20a40d0e10 0x7f20a40ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47618/0 (socket says 172.21.15.44:47618) 2024-04-02T13:24:37.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.931+0000 7f20a3fff640 1 -- 172.21.15.44:0/4173967200 learned_addr learned my addr 172.21.15.44:0/4173967200 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:37.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.931+0000 7f20a8d85640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20a40d0a30 0x7f20a40cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.931+0000 7f20a9586640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20a40d2040 0x7f20a4159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.932+0000 7f20a3fff640 1 -- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20a40d0a30 msgr2=0x7f20a40cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.932+0000 7f20a3fff640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20a40d0a30 0x7f20a40cbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.932+0000 7f20a3fff640 1 -- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20a40d2040 msgr2=0x7f20a4159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.932+0000 7f20a3fff640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20a40d2040 0x7f20a4159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.932+0000 7f20a3fff640 1 -- 172.21.15.44:0/4173967200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a40cd550 con 0x7f20a40d0e10 2024-04-02T13:24:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.932+0000 7f20a3fff640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20a40d0e10 0x7f20a40ccdb0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f209800c3c0 tx=0x7f209800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.932+0000 7f20a17fa640 1 -- 172.21.15.44:0/4173967200 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20980074e0 con 0x7f20a40d0e10 2024-04-02T13:24:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.932+0000 7f20a17fa640 1 -- 172.21.15.44:0/4173967200 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20980047e0 con 0x7f20a40d0e10 2024-04-02T13:24:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.932+0000 7f20a17fa640 1 -- 172.21.15.44:0/4173967200 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2098004bd0 con 0x7f20a40d0e10 2024-04-02T13:24:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.932+0000 7f20aafe9640 1 -- 172.21.15.44:0/4173967200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20a41e7dc0 con 0x7f20a40d0e10 2024-04-02T13:24:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.932+0000 7f20aafe9640 1 -- 172.21.15.44:0/4173967200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20a41e8380 con 0x7f20a40d0e10 2024-04-02T13:24:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.933+0000 7f20aafe9640 1 -- 172.21.15.44:0/4173967200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20a40684f0 con 0x7f20a40d0e10 2024-04-02T13:24:37.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.934+0000 7f20a17fa640 1 -- 172.21.15.44:0/4173967200 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2098011070 con 0x7f20a40d0e10 2024-04-02T13:24:37.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.934+0000 7f20a17fa640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f209006c8f0 0x7f209006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:37.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.934+0000 7f20a17fa640 1 -- 172.21.15.44:0/4173967200 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f209808b810 con 0x7f20a40d0e10 2024-04-02T13:24:37.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.934+0000 7f20a8d85640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f209006c8f0 0x7f209006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:37.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.935+0000 7f20a8d85640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f209006c8f0 0x7f209006edb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f209c036000 tx=0x7f209c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:37.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:37.938+0000 7f20a17fa640 1 -- 172.21.15.44:0/4173967200 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f20980593d0 con 0x7f20a40d0e10 2024-04-02T13:24:38.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.134+0000 7f20aafe9640 1 -- 172.21.15.44:0/4173967200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f20a40d17c0 con 0x7f20a40d0e10 2024-04-02T13:24:38.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.135+0000 7f20a17fa640 1 -- 172.21.15.44:0/4173967200 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v67) ==== 80+0+0 (secure 0 0 0) 0x7f209805d6d0 con 0x7f20a40d0e10 2024-04-02T13:24:38.137 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:24:38.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.137+0000 7f20aafe9640 1 -- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f209006c8f0 msgr2=0x7f209006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:38.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.137+0000 7f20aafe9640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f209006c8f0 0x7f209006edb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f209c036000 tx=0x7f209c030000 comp rx=0 tx=0).stop 2024-04-02T13:24:38.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.137+0000 7f20aafe9640 1 -- 172.21.15.44:0/4173967200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20a40d0e10 msgr2=0x7f20a40ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:38.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.137+0000 7f20aafe9640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20a40d0e10 0x7f20a40ccdb0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f209800c3c0 tx=0x7f209800c880 comp rx=0 tx=0).stop 2024-04-02T13:24:38.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.137+0000 7f20aafe9640 1 -- 172.21.15.44:0/4173967200 shutdown_connections 2024-04-02T13:24:38.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.137+0000 7f20aafe9640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f209006c8f0 0x7f209006edb0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.137+0000 7f20aafe9640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20a40d2040 0x7f20a4159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.137+0000 7f20aafe9640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20a40d0e10 0x7f20a40ccdb0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.137+0000 7f20aafe9640 1 --2- 172.21.15.44:0/4173967200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20a40d0a30 0x7f20a40cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.137+0000 7f20aafe9640 1 -- 172.21.15.44:0/4173967200 >> 172.21.15.44:0/4173967200 conn(0x7f20a409d470 msgr2=0x7f20a40d45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:38.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.137+0000 7f20aafe9640 1 -- 172.21.15.44:0/4173967200 shutdown_connections 2024-04-02T13:24:38.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.137+0000 7f20aafe9640 1 -- 172.21.15.44:0/4173967200 wait complete. 2024-04-02T13:24:38.150 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:24:38.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:24:38.151 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:24:38.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[60355]: fuse finished with error 0 and tester_r 0 2024-04-02T13:24:38.178 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:24:38.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:24:38.178 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:24:38.346 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:24:38.346 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:24:38.347 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:24:38.364 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:24:38.364 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:24:38.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:24:38.365 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-02T13:24:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:24:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-02T13:24:38.422 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:24:38.422 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:24:38.422 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:24:38.438 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:24:38.438 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:24:38.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:24:38.438 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-02T13:24:38.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:24:38.493 INFO:teuthology.orchestra.run.smithi044.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-02T13:24:38.493 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:24:38.493 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:24:38.493 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:24:38.511 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:24:38.511 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:24:38.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:24:38.511 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-02T13:24:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:24:38.567 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-02T13:24:38.567 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:24:38.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.667+0000 7fe546469640 1 Processor -- start 2024-04-02T13:24:38.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.667+0000 7fe546469640 1 -- start start 2024-04-02T13:24:38.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.668+0000 7fe546469640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe54006cf90 0x7fe5400cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:38.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.668+0000 7fe546469640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5400cfe30 0x7fe5400d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:38.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.668+0000 7fe546469640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe54006c1d0 con 0x7fe5400d0a30 2024-04-02T13:24:38.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.668+0000 7fe546469640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe54006cb00 con 0x7fe54006cf90 2024-04-02T13:24:38.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.668+0000 7fe546469640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe54006d3f0 con 0x7fe5400cfe30 2024-04-02T13:24:38.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.668+0000 7fe53ffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe5400d0a30 0x7fe5400d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48112/0 (socket says 172.21.15.44:48112) 2024-04-02T13:24:38.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.668+0000 7fe53ffff640 1 -- 172.21.15.44:0/718437649 learned_addr learned my addr 172.21.15.44:0/718437649 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:38.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.668+0000 7fe53f7fe640 1 --2- 172.21.15.44:0/718437649 >> v2:172.21.15.69:3300/0 conn(0x7fe54006cf90 0x7fe5400cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:38.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.668+0000 7fe544a06640 1 --2- 172.21.15.44:0/718437649 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5400cfe30 0x7fe5400d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:38.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.668+0000 7fe53effd640 1 -- 172.21.15.44:0/718437649 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2152001085 0 0) 0x7fe54006c1d0 con 0x7fe5400d0a30 2024-04-02T13:24:38.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.668+0000 7fe53effd640 1 -- 172.21.15.44:0/718437649 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe528003520 con 0x7fe5400d0a30 2024-04-02T13:24:38.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.669+0000 7fe544a06640 1 -- 172.21.15.44:0/718437649 >> v2:172.21.15.69:3300/0 conn(0x7fe54006cf90 msgr2=0x7fe5400cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:38.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.669+0000 7fe544a06640 1 --2- 172.21.15.44:0/718437649 >> v2:172.21.15.69:3300/0 conn(0x7fe54006cf90 0x7fe5400cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.669+0000 7fe544a06640 1 -- 172.21.15.44:0/718437649 >> v1:172.21.15.44:6789/0 conn(0x7fe5400d0a30 legacy=0x7fe5400d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:38.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.669+0000 7fe544a06640 1 -- 172.21.15.44:0/718437649 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54006c4b0 con 0x7fe5400cfe30 2024-04-02T13:24:38.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.669+0000 7fe53f7fe640 1 --2- 172.21.15.44:0/718437649 >> v2:172.21.15.69:3300/0 conn(0x7fe54006cf90 0x7fe5400cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:24:38.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.669+0000 7fe544a06640 1 --2- 172.21.15.44:0/718437649 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5400cfe30 0x7fe5400d2040 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe538009d90 tx=0x7fe53802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ecc3c62a9f1e7ed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:38.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.669+0000 7fe53effd640 1 -- 172.21.15.44:0/718437649 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe538039070 con 0x7fe5400cfe30 2024-04-02T13:24:38.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.670+0000 7fe53effd640 1 -- 172.21.15.44:0/718437649 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe53802cbc0 con 0x7fe5400cfe30 2024-04-02T13:24:38.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.670+0000 7fe53effd640 1 -- 172.21.15.44:0/718437649 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe538034440 con 0x7fe5400cfe30 2024-04-02T13:24:38.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.670+0000 7fe546469640 1 -- 172.21.15.44:0/718437649 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5400cfe30 msgr2=0x7fe5400d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:38.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.670+0000 7fe546469640 1 --2- 172.21.15.44:0/718437649 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5400cfe30 0x7fe5400d2040 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe538009d90 tx=0x7fe53802be10 comp rx=0 tx=0).stop 2024-04-02T13:24:38.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.670+0000 7fe546469640 1 -- 172.21.15.44:0/718437649 shutdown_connections 2024-04-02T13:24:38.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.670+0000 7fe546469640 1 --2- 172.21.15.44:0/718437649 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5400cfe30 0x7fe5400d2040 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.670+0000 7fe546469640 1 --2- 172.21.15.44:0/718437649 >> v2:172.21.15.69:3300/0 conn(0x7fe54006cf90 0x7fe5400cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.670+0000 7fe546469640 1 -- 172.21.15.44:0/718437649 >> 172.21.15.44:0/718437649 conn(0x7fe54009d470 msgr2=0x7fe5400d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:38.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.670+0000 7fe546469640 1 -- 172.21.15.44:0/718437649 shutdown_connections 2024-04-02T13:24:38.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.670+0000 7fe546469640 1 -- 172.21.15.44:0/718437649 wait complete. 2024-04-02T13:24:38.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.671+0000 7fe546469640 1 Processor -- start 2024-04-02T13:24:38.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.671+0000 7fe546469640 1 -- start start 2024-04-02T13:24:38.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe546469640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe54006cf90 0x7fe5400b9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:38.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe546469640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5400cfe30 0x7fe5400b6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:38.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe546469640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5400d0a30 0x7fe5400b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:38.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe546469640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe54015e870 con 0x7fe5400cfe30 2024-04-02T13:24:38.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe546469640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe54015e6f0 con 0x7fe54006cf90 2024-04-02T13:24:38.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe546469640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe5400d4da0 con 0x7fe5400d0a30 2024-04-02T13:24:38.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe53f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5400cfe30 0x7fe5400b6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:38.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe53f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5400cfe30 0x7fe5400b6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47620/0 (socket says 172.21.15.44:47620) 2024-04-02T13:24:38.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe53f7fe640 1 -- 172.21.15.44:0/916584684 learned_addr learned my addr 172.21.15.44:0/916584684 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:38.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe544a06640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5400d0a30 0x7fe5400b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:38.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe53ffff640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe54006cf90 0x7fe5400b9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:38.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe53f7fe640 1 -- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5400d0a30 msgr2=0x7fe5400b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:38.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe53f7fe640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5400d0a30 0x7fe5400b2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe53f7fe640 1 -- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe54006cf90 msgr2=0x7fe5400b9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:38.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe53f7fe640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe54006cf90 0x7fe5400b9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.672+0000 7fe53f7fe640 1 -- 172.21.15.44:0/916584684 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54014b2e0 con 0x7fe5400cfe30 2024-04-02T13:24:38.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.673+0000 7fe53f7fe640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5400cfe30 0x7fe5400b6d30 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe53400e910 tx=0x7fe53400edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:38.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.673+0000 7fe53cff9640 1 -- 172.21.15.44:0/916584684 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534017070 con 0x7fe5400cfe30 2024-04-02T13:24:38.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.673+0000 7fe53cff9640 1 -- 172.21.15.44:0/916584684 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe53400ca00 con 0x7fe5400cfe30 2024-04-02T13:24:38.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.673+0000 7fe546469640 1 -- 172.21.15.44:0/916584684 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54014b5c0 con 0x7fe5400cfe30 2024-04-02T13:24:38.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.673+0000 7fe53cff9640 1 -- 172.21.15.44:0/916584684 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53400cdb0 con 0x7fe5400cfe30 2024-04-02T13:24:38.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.673+0000 7fe546469640 1 -- 172.21.15.44:0/916584684 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe54014bb80 con 0x7fe5400cfe30 2024-04-02T13:24:38.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.674+0000 7fe546469640 1 -- 172.21.15.44:0/916584684 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5400684f0 con 0x7fe5400cfe30 2024-04-02T13:24:38.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.674+0000 7fe53cff9640 1 -- 172.21.15.44:0/916584684 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe53401a070 con 0x7fe5400cfe30 2024-04-02T13:24:38.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.675+0000 7fe53cff9640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe52c06c8f0 0x7fe52c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:38.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.675+0000 7fe53cff9640 1 -- 172.21.15.44:0/916584684 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7fe53408be50 con 0x7fe5400cfe30 2024-04-02T13:24:38.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.675+0000 7fe53ffff640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe52c06c8f0 0x7fe52c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:38.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.675+0000 7fe53ffff640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe52c06c8f0 0x7fe52c06edb0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe530005880 tx=0x7fe530009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:38.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.679+0000 7fe53cff9640 1 -- 172.21.15.44:0/916584684 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe534059aa0 con 0x7fe5400cfe30 2024-04-02T13:24:38.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.900+0000 7fe546469640 1 -- 172.21.15.44:0/916584684 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5400d17c0 con 0x7fe5400cfe30 2024-04-02T13:24:38.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.901+0000 7fe53cff9640 1 -- 172.21.15.44:0/916584684 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5180 (secure 0 0 0) 0x7fe53405dda0 con 0x7fe5400cfe30 2024-04-02T13:24:38.903 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:38.903 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":37,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1535287484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1535287484},{"type":"v1","addr":"172.21.15.69:6836","nonce":1535287484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6429,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2367107002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2367107002},{"type":"v1","addr":"172.21.15.44:6833","nonce":2367107002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/553542317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":553542317},{"type":"v1","addr":"172.21.15.69:6837","nonce":553542317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:23:24.803884+0000","modified":"2024-04-02T13:24:05.482501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6450":{"gid":6450,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":13,"addr":"172.21.15.44:6835/3127613802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3127613802},{"type":"v1","addr":"172.21.15.44:6835","nonce":3127613802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6450,"qdb_cluster":[6450]},"id":4}]} 2024-04-02T13:24:38.903 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 37 2024-04-02T13:24:38.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.903+0000 7fe546469640 1 -- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe52c06c8f0 msgr2=0x7fe52c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:38.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.903+0000 7fe546469640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe52c06c8f0 0x7fe52c06edb0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe530005880 tx=0x7fe530009040 comp rx=0 tx=0).stop 2024-04-02T13:24:38.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.903+0000 7fe546469640 1 -- 172.21.15.44:0/916584684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5400cfe30 msgr2=0x7fe5400b6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:38.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.903+0000 7fe546469640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5400cfe30 0x7fe5400b6d30 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe53400e910 tx=0x7fe53400edd0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.903+0000 7fe546469640 1 -- 172.21.15.44:0/916584684 shutdown_connections 2024-04-02T13:24:38.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.903+0000 7fe546469640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe52c06c8f0 0x7fe52c06edb0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.903+0000 7fe546469640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5400d0a30 0x7fe5400b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.903+0000 7fe546469640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5400cfe30 0x7fe5400b6d30 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.903+0000 7fe546469640 1 --2- 172.21.15.44:0/916584684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe54006cf90 0x7fe5400b9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:38.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.903+0000 7fe546469640 1 -- 172.21.15.44:0/916584684 >> 172.21.15.44:0/916584684 conn(0x7fe54009d470 msgr2=0x7fe54015a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:38.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.903+0000 7fe546469640 1 -- 172.21.15.44:0/916584684 shutdown_connections 2024-04-02T13:24:38.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:38.903+0000 7fe546469640 1 -- 172.21.15.44:0/916584684 wait complete. 2024-04-02T13:24:38.916 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:24:39.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.016+0000 7f5f73c1b640 1 Processor -- start 2024-04-02T13:24:39.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.016+0000 7f5f73c1b640 1 -- start start 2024-04-02T13:24:39.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.016+0000 7f5f73c1b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5f6c15e4a0 0x7f5f6c0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.016+0000 7f5f73c1b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f6c06d550 0x7f5f6c0d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.016+0000 7f5f73c1b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f6c0cc600 con 0x7f5f6c0d0c50 2024-04-02T13:24:39.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.016+0000 7f5f73c1b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f5f6c0cc8e0 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.016+0000 7f5f73c1b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6c0ccbc0 con 0x7f5f6c06d550 2024-04-02T13:24:39.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.017+0000 7f5f719b7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5f6c15e4a0 0x7f5f6c0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.017+0000 7f5f719b7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5f6c15e4a0 0x7f5f6c0aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47622/0 (socket says 172.21.15.44:47622) 2024-04-02T13:24:39.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.017+0000 7f5f719b7640 1 -- 172.21.15.44:0/3705692754 learned_addr learned my addr 172.21.15.44:0/3705692754 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:39.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.017+0000 7f5f721b8640 1 --2- 172.21.15.44:0/3705692754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f6c06d550 0x7f5f6c0d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.017+0000 7f5f719b7640 1 -- 172.21.15.44:0/3705692754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f6c06d550 msgr2=0x7f5f6c0d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:39.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.017+0000 7f5f719b7640 1 --2- 172.21.15.44:0/3705692754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f6c06d550 0x7f5f6c0d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.017+0000 7f5f719b7640 1 -- 172.21.15.44:0/3705692754 >> v1:172.21.15.69:6789/0 conn(0x7f5f6c0d0c50 legacy=0x7f5f6c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:39.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.017+0000 7f5f719b7640 1 -- 172.21.15.44:0/3705692754 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c0ccea0 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.018+0000 7f5f719b7640 1 --2- 172.21.15.44:0/3705692754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f6c15e4a0 0x7f5f6c0aebf0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009870 tx=0x7f5f6402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a19cd06975fcac50 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:39.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.018+0000 7f5f709b5640 1 -- 172.21.15.44:0/3705692754 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64035020 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.018+0000 7f5f709b5640 1 -- 172.21.15.44:0/3705692754 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f64031640 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.018+0000 7f5f709b5640 1 -- 172.21.15.44:0/3705692754 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6402d040 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.018+0000 7f5f73c1b640 1 -- 172.21.15.44:0/3705692754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f6c15e4a0 msgr2=0x7f5f6c0aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.018+0000 7f5f73c1b640 1 --2- 172.21.15.44:0/3705692754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f6c15e4a0 0x7f5f6c0aebf0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009870 tx=0x7f5f6402b8f0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.018+0000 7f5f73c1b640 1 -- 172.21.15.44:0/3705692754 shutdown_connections 2024-04-02T13:24:39.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.018+0000 7f5f73c1b640 1 --2- 172.21.15.44:0/3705692754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f6c06d550 0x7f5f6c0d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.018+0000 7f5f73c1b640 1 --2- 172.21.15.44:0/3705692754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f6c15e4a0 0x7f5f6c0aebf0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.018+0000 7f5f73c1b640 1 -- 172.21.15.44:0/3705692754 >> 172.21.15.44:0/3705692754 conn(0x7f5f6c09aa20 msgr2=0x7f5f6c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:39.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.019+0000 7f5f73c1b640 1 -- 172.21.15.44:0/3705692754 shutdown_connections 2024-04-02T13:24:39.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.019+0000 7f5f73c1b640 1 -- 172.21.15.44:0/3705692754 wait complete. 2024-04-02T13:24:39.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.020+0000 7f5f73c1b640 1 Processor -- start 2024-04-02T13:24:39.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.020+0000 7f5f73c1b640 1 -- start start 2024-04-02T13:24:39.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.020+0000 7f5f73c1b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f6c06d550 0x7f5f6c0cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.020+0000 7f5f73c1b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f6c0d0c50 0x7f5f6c155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.020+0000 7f5f73c1b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f6c15e4a0 0x7f5f6c160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.020+0000 7f5f73c1b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c15e880 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.020+0000 7f5f73c1b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c0cd460 con 0x7f5f6c06d550 2024-04-02T13:24:39.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.020+0000 7f5f73c1b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6c0af130 con 0x7f5f6c0d0c50 2024-04-02T13:24:39.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.020+0000 7f5f721b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f6c15e4a0 0x7f5f6c160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f721b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f6c15e4a0 0x7f5f6c160f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47624/0 (socket says 172.21.15.44:47624) 2024-04-02T13:24:39.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f721b8640 1 -- 172.21.15.44:0/589926945 learned_addr learned my addr 172.21.15.44:0/589926945 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:39.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f719b7640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f6c06d550 0x7f5f6c0cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f711b6640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f6c0d0c50 0x7f5f6c155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f721b8640 1 -- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f6c0d0c50 msgr2=0x7f5f6c155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f721b8640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f6c0d0c50 0x7f5f6c155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f721b8640 1 -- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f6c06d550 msgr2=0x7f5f6c0cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f721b8640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f6c06d550 0x7f5f6c0cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f721b8640 1 -- 172.21.15.44:0/589926945 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c0caa50 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f721b8640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f6c15e4a0 0x7f5f6c160f90 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00cd70 tx=0x7f5f5c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:39.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f6a7fc640 1 -- 172.21.15.44:0/589926945 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c003b90 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f6a7fc640 1 -- 172.21.15.44:0/589926945 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f5c003d30 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f73c1b640 1 -- 172.21.15.44:0/589926945 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f6c0cad30 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f6a7fc640 1 -- 172.21.15.44:0/589926945 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c004240 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.021+0000 7f5f73c1b640 1 -- 172.21.15.44:0/589926945 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f6c0cb2f0 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.022+0000 7f5f73c1b640 1 -- 172.21.15.44:0/589926945 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f6c0b18e0 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.022+0000 7f5f6a7fc640 1 -- 172.21.15.44:0/589926945 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5f5c0043e0 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.023+0000 7f5f6a7fc640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f5806c8f0 0x7f5f5806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.023+0000 7f5f6a7fc640 1 -- 172.21.15.44:0/589926945 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f5f5c08c6d0 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.023+0000 7f5f719b7640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f5806c8f0 0x7f5f5806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.024+0000 7f5f719b7640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f5806c8f0 0x7f5f5806edb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5f64036000 tx=0x7f5f6403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:39.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.027+0000 7f5f6a7fc640 1 -- 172.21.15.44:0/589926945 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5f6c0b18e0 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.212+0000 7f5f73c1b640 1 -- 172.21.15.44:0/589926945 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f6c0cb4f0 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.213+0000 7f5f6a7fc640 1 -- 172.21.15.44:0/589926945 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+17076 (secure 0 0 0) 0x7f5f6c0cb4f0 con 0x7f5f6c15e4a0 2024-04-02T13:24:39.216 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:39.216 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":67,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:24:09.591968+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:23:23.126712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-02T13:23:23.763593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.18","mappings":[{"from":7,"to":2}]},{"pgid":"10.4","mappings":[{"from":4,"to":3}]},{"pgid":"10.10","mappings":[{"from":4,"to":3}]},{"pgid":"10.37","mappings":[{"from":4,"to":5}]},{"pgid":"10.39","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:24:39.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.215+0000 7f5f73c1b640 1 -- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f5806c8f0 msgr2=0x7f5f5806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.215+0000 7f5f73c1b640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f5806c8f0 0x7f5f5806edb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5f64036000 tx=0x7f5f6403c040 comp rx=0 tx=0).stop 2024-04-02T13:24:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.215+0000 7f5f73c1b640 1 -- 172.21.15.44:0/589926945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f6c15e4a0 msgr2=0x7f5f6c160f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.215+0000 7f5f73c1b640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f6c15e4a0 0x7f5f6c160f90 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00cd70 tx=0x7f5f5c00a660 comp rx=0 tx=0).stop 2024-04-02T13:24:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.215+0000 7f5f73c1b640 1 -- 172.21.15.44:0/589926945 shutdown_connections 2024-04-02T13:24:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.215+0000 7f5f73c1b640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f5806c8f0 0x7f5f5806edb0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.215+0000 7f5f73c1b640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f6c15e4a0 0x7f5f6c160f90 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.215+0000 7f5f73c1b640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f6c0d0c50 0x7f5f6c155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.215+0000 7f5f73c1b640 1 --2- 172.21.15.44:0/589926945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f6c06d550 0x7f5f6c0cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.215+0000 7f5f73c1b640 1 -- 172.21.15.44:0/589926945 >> 172.21.15.44:0/589926945 conn(0x7f5f6c09aa20 msgr2=0x7f5f6c0d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.215+0000 7f5f73c1b640 1 -- 172.21.15.44:0/589926945 shutdown_connections 2024-04-02T13:24:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.215+0000 7f5f73c1b640 1 -- 172.21.15.44:0/589926945 wait complete. 2024-04-02T13:24:39.228 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:23:23.126712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:23:23.763593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:24:39.229 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:24:39.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.331+0000 7f1d32196640 1 Processor -- start 2024-04-02T13:24:39.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.331+0000 7f1d32196640 1 -- start start 2024-04-02T13:24:39.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.331+0000 7f1d32196640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d2c0d0a30 0x7f1d2c0b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.331+0000 7f1d32196640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d2c0d0e10 con 0x7f1d2c0b3990 2024-04-02T13:24:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.331+0000 7f1d32196640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d2c144750 con 0x7f1d2c144130 2024-04-02T13:24:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.331+0000 7f1d32196640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1d2c161e10 con 0x7f1d2c0d0a30 2024-04-02T13:24:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.332+0000 7f1d2affd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f1d2c0b3990 0x7f1d2c143a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48116/0 (socket says 172.21.15.44:48116) 2024-04-02T13:24:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.332+0000 7f1d2affd640 1 -- 172.21.15.44:0/3327951330 learned_addr learned my addr 172.21.15.44:0/3327951330 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.332+0000 7f1d2b7fe640 1 --2- 172.21.15.44:0/3327951330 >> v2:172.21.15.69:3300/0 conn(0x7f1d2c0d0a30 0x7f1d2c0b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.332+0000 7f1d2a7fc640 1 -- 172.21.15.44:0/3327951330 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 827825916 0 0) 0x7f1d20001110 con 0x7f1d2c0b3990 2024-04-02T13:24:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.332+0000 7f1d2a7fc640 1 -- 172.21.15.44:0/3327951330 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d14003520 con 0x7f1d2c0b3990 2024-04-02T13:24:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.332+0000 7f1d2a7fc640 1 -- 172.21.15.44:0/3327951330 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4037809080 0 0) 0x7f1d2c144750 con 0x7f1d2c144130 2024-04-02T13:24:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.332+0000 7f1d2a7fc640 1 -- 172.21.15.44:0/3327951330 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d20001110 con 0x7f1d2c144130 2024-04-02T13:24:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.332+0000 7f1d2b7fe640 1 -- 172.21.15.44:0/3327951330 >> v1:172.21.15.69:6789/0 conn(0x7f1d2c144130 legacy=0x7f1d2c15e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.332+0000 7f1d2b7fe640 1 -- 172.21.15.44:0/3327951330 >> v1:172.21.15.44:6789/0 conn(0x7f1d2c0b3990 legacy=0x7f1d2c143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.332+0000 7f1d2a7fc640 1 -- 172.21.15.44:0/3327951330 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4195924020 0 0) 0x7f1d14003520 con 0x7f1d2c0b3990 2024-04-02T13:24:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.332+0000 7f1d2b7fe640 1 -- 172.21.15.44:0/3327951330 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2c162060 con 0x7f1d2c0d0a30 2024-04-02T13:24:39.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.333+0000 7f1d2b7fe640 1 --2- 172.21.15.44:0/3327951330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d2c0d0a30 0x7f1d2c0b3450 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0097a0 tx=0x7f1d1c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bafcb0995b73d5b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:39.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.333+0000 7f1d2a7fc640 1 -- 172.21.15.44:0/3327951330 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c037070 con 0x7f1d2c0d0a30 2024-04-02T13:24:39.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.333+0000 7f1d2a7fc640 1 -- 172.21.15.44:0/3327951330 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d1c030510 con 0x7f1d2c0d0a30 2024-04-02T13:24:39.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.333+0000 7f1d2a7fc640 1 -- 172.21.15.44:0/3327951330 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20001840 con 0x7f1d2c0d0a30 2024-04-02T13:24:39.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.334+0000 7f1d32196640 1 -- 172.21.15.44:0/3327951330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d2c0d0a30 msgr2=0x7f1d2c0b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.334+0000 7f1d32196640 1 --2- 172.21.15.44:0/3327951330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d2c0d0a30 0x7f1d2c0b3450 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0097a0 tx=0x7f1d1c02b790 comp rx=0 tx=0).stop 2024-04-02T13:24:39.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.334+0000 7f1d32196640 1 -- 172.21.15.44:0/3327951330 shutdown_connections 2024-04-02T13:24:39.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.334+0000 7f1d32196640 1 --2- 172.21.15.44:0/3327951330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d2c0d0a30 0x7f1d2c0b3450 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.334+0000 7f1d32196640 1 -- 172.21.15.44:0/3327951330 >> 172.21.15.44:0/3327951330 conn(0x7f1d2c0b5d90 msgr2=0x7f1d2c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:39.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.334+0000 7f1d32196640 1 -- 172.21.15.44:0/3327951330 shutdown_connections 2024-04-02T13:24:39.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.334+0000 7f1d32196640 1 -- 172.21.15.44:0/3327951330 wait complete. 2024-04-02T13:24:39.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.335+0000 7f1d32196640 1 Processor -- start 2024-04-02T13:24:39.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.335+0000 7f1d32196640 1 -- start start 2024-04-02T13:24:39.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d32196640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d2c0b3990 0x7f1d2c1f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d32196640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d2c0d0a30 0x7f1d2c1e7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d32196640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d2c144130 0x7f1d2c1e8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d32196640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2c162b20 con 0x7f1d2c144130 2024-04-02T13:24:39.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d32196640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2c162620 con 0x7f1d2c0d0a30 2024-04-02T13:24:39.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d32196640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1d2c162ca0 con 0x7f1d2c0b3990 2024-04-02T13:24:39.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d2bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d2c144130 0x7f1d2c1e8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d2bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d2c144130 0x7f1d2c1e8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47638/0 (socket says 172.21.15.44:47638) 2024-04-02T13:24:39.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d2bfff640 1 -- 172.21.15.44:0/2866344687 learned_addr learned my addr 172.21.15.44:0/2866344687 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:39.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d2affd640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d2c0d0a30 0x7f1d2c1e7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d2b7fe640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d2c0b3990 0x7f1d2c1f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d2bfff640 1 -- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d2c0b3990 msgr2=0x7f1d2c1f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d2bfff640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d2c0b3990 0x7f1d2c1f3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d2bfff640 1 -- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d2c0d0a30 msgr2=0x7f1d2c1e7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d2bfff640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d2c0d0a30 0x7f1d2c1e7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d2bfff640 1 -- 172.21.15.44:0/2866344687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2c0ae4c0 con 0x7f1d2c144130 2024-04-02T13:24:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.336+0000 7f1d2b7fe640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d2c0b3990 0x7f1d2c1f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.337+0000 7f1d2bfff640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d2c144130 0x7f1d2c1e8140 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400c7c0 tx=0x7f1d2400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.337+0000 7f1d13fff640 1 -- 172.21.15.44:0/2866344687 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d24003b70 con 0x7f1d2c144130 2024-04-02T13:24:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.337+0000 7f1d13fff640 1 -- 172.21.15.44:0/2866344687 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d24003d10 con 0x7f1d2c144130 2024-04-02T13:24:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.337+0000 7f1d13fff640 1 -- 172.21.15.44:0/2866344687 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2400aa50 con 0x7f1d2c144130 2024-04-02T13:24:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.337+0000 7f1d32196640 1 -- 172.21.15.44:0/2866344687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d2c0ae7a0 con 0x7f1d2c144130 2024-04-02T13:24:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.337+0000 7f1d32196640 1 -- 172.21.15.44:0/2866344687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d2c0aed60 con 0x7f1d2c144130 2024-04-02T13:24:39.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.338+0000 7f1d32196640 1 -- 172.21.15.44:0/2866344687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d2c0684f0 con 0x7f1d2c144130 2024-04-02T13:24:39.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.338+0000 7f1d13fff640 1 -- 172.21.15.44:0/2866344687 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1d2400abf0 con 0x7f1d2c144130 2024-04-02T13:24:39.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.339+0000 7f1d13fff640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d1806c8f0 0x7f1d1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.339+0000 7f1d13fff640 1 -- 172.21.15.44:0/2866344687 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f1d2408bcf0 con 0x7f1d2c144130 2024-04-02T13:24:39.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.339+0000 7f1d2b7fe640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d1806c8f0 0x7f1d1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.339+0000 7f1d2b7fe640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d1806c8f0 0x7f1d1806edb0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c034000 tx=0x7f1d1c02f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:39.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.343+0000 7f1d13fff640 1 -- 172.21.15.44:0/2866344687 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1d24059f70 con 0x7f1d2c144130 2024-04-02T13:24:39.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.525+0000 7f1d32196640 1 -- 172.21.15.44:0/2866344687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1d2c06bcb0 con 0x7f1d2c144130 2024-04-02T13:24:39.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.525+0000 7f1d13fff640 1 -- 172.21.15.44:0/2866344687 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) ==== 93+0+10 (secure 0 0 0) 0x7f1d24059870 con 0x7f1d2c144130 2024-04-02T13:24:39.527 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:24:39.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.527+0000 7f1d32196640 1 -- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d1806c8f0 msgr2=0x7f1d1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.527+0000 7f1d32196640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d1806c8f0 0x7f1d1806edb0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c034000 tx=0x7f1d1c02f000 comp rx=0 tx=0).stop 2024-04-02T13:24:39.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.527+0000 7f1d32196640 1 -- 172.21.15.44:0/2866344687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d2c144130 msgr2=0x7f1d2c1e8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.527+0000 7f1d32196640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d2c144130 0x7f1d2c1e8140 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400c7c0 tx=0x7f1d2400cc80 comp rx=0 tx=0).stop 2024-04-02T13:24:39.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.527+0000 7f1d32196640 1 -- 172.21.15.44:0/2866344687 shutdown_connections 2024-04-02T13:24:39.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.527+0000 7f1d32196640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d1806c8f0 0x7f1d1806edb0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.527+0000 7f1d32196640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d2c144130 0x7f1d2c1e8140 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.527+0000 7f1d32196640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d2c0d0a30 0x7f1d2c1e7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.527+0000 7f1d32196640 1 --2- 172.21.15.44:0/2866344687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d2c0b3990 0x7f1d2c1f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.527+0000 7f1d32196640 1 -- 172.21.15.44:0/2866344687 >> 172.21.15.44:0/2866344687 conn(0x7f1d2c0b5d90 msgr2=0x7f1d2c06f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:39.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.527+0000 7f1d32196640 1 -- 172.21.15.44:0/2866344687 shutdown_connections 2024-04-02T13:24:39.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.527+0000 7f1d32196640 1 -- 172.21.15.44:0/2866344687 wait complete. 2024-04-02T13:24:39.540 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:24:39.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.643+0000 7f9a9600f640 1 Processor -- start 2024-04-02T13:24:39.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.643+0000 7f9a9600f640 1 -- start start 2024-04-02T13:24:39.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a9600f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a9013ef60 0x7f9a9006f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a9600f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9a9006d940 0x7f9a90071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a9600f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9a9006dda0 0x7f9a90071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a9600f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f9a9006e300 con 0x7f9a9006d940 2024-04-02T13:24:39.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a9600f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9a9006e5e0 con 0x7f9a9006dda0 2024-04-02T13:24:39.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a9600f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9a9006e8c0 con 0x7f9a9013ef60 2024-04-02T13:24:39.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a8effd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9a9006d940 0x7f9a90071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a8effd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9a9006d940 0x7f9a90071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47644/0 (socket says 172.21.15.44:47644) 2024-04-02T13:24:39.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a8effd640 1 -- 172.21.15.44:0/2249972177 learned_addr learned my addr 172.21.15.44:0/2249972177 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:39.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a8effd640 1 -- 172.21.15.44:0/2249972177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a9013ef60 msgr2=0x7f9a9006f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a8effd640 1 --2- 172.21.15.44:0/2249972177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a9013ef60 0x7f9a9006f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a8effd640 1 -- 172.21.15.44:0/2249972177 >> v2:172.21.15.69:3300/0 conn(0x7f9a9006dda0 msgr2=0x7f9a90071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a8ffff640 1 --2- 172.21.15.44:0/2249972177 >> v2:172.21.15.69:3300/0 conn(0x7f9a9006dda0 0x7f9a90071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a8effd640 1 --2- 172.21.15.44:0/2249972177 >> v2:172.21.15.69:3300/0 conn(0x7f9a9006dda0 0x7f9a90071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.644+0000 7f9a8effd640 1 -- 172.21.15.44:0/2249972177 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9006eba0 con 0x7f9a9006d940 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.645+0000 7f9a8effd640 1 --2- 172.21.15.44:0/2249972177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a9006d940 0x7f9a90071920 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f9a90156bb0 tx=0x7f9a8402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b359555492136c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.645+0000 7f9a8e7fc640 1 -- 172.21.15.44:0/2249972177 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8402e070 con 0x7f9a9006d940 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.645+0000 7f9a8e7fc640 1 -- 172.21.15.44:0/2249972177 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a84037070 con 0x7f9a9006d940 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.645+0000 7f9a8e7fc640 1 -- 172.21.15.44:0/2249972177 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84031a00 con 0x7f9a9006d940 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.646+0000 7f9a9600f640 1 -- 172.21.15.44:0/2249972177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a9006d940 msgr2=0x7f9a90071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.646+0000 7f9a9600f640 1 --2- 172.21.15.44:0/2249972177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a9006d940 0x7f9a90071920 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f9a90156bb0 tx=0x7f9a8402b880 comp rx=0 tx=0).stop 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.646+0000 7f9a9600f640 1 -- 172.21.15.44:0/2249972177 shutdown_connections 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.646+0000 7f9a9600f640 1 --2- 172.21.15.44:0/2249972177 >> v2:172.21.15.69:3300/0 conn(0x7f9a9006dda0 0x7f9a90071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.646+0000 7f9a9600f640 1 --2- 172.21.15.44:0/2249972177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a9006d940 0x7f9a90071920 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.646+0000 7f9a9600f640 1 --2- 172.21.15.44:0/2249972177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a9013ef60 0x7f9a9006f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.646+0000 7f9a9600f640 1 -- 172.21.15.44:0/2249972177 >> 172.21.15.44:0/2249972177 conn(0x7f9a900b5d90 msgr2=0x7f9a9006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.646+0000 7f9a9600f640 1 -- 172.21.15.44:0/2249972177 shutdown_connections 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.646+0000 7f9a9600f640 1 -- 172.21.15.44:0/2249972177 wait complete. 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.647+0000 7f9a9600f640 1 Processor -- start 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.647+0000 7f9a9600f640 1 -- start start 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.648+0000 7f9a9600f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a9006d940 0x7f9a901f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.648+0000 7f9a9600f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a9006dda0 0x7f9a900a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.648+0000 7f9a9600f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a9013ef60 0x7f9a9006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.648+0000 7f9a9600f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9a90072720 con 0x7f9a9013ef60 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.648+0000 7f9a9600f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9a900725a0 con 0x7f9a9006dda0 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.648+0000 7f9a9600f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9a9013f6f0 con 0x7f9a9006d940 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.648+0000 7f9a8ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a9013ef60 0x7f9a9006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.648+0000 7f9a8ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a9013ef60 0x7f9a9006b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47646/0 (socket says 172.21.15.44:47646) 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.648+0000 7f9a8ffff640 1 -- 172.21.15.44:0/2716918083 learned_addr learned my addr 172.21.15.44:0/2716918083 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.648+0000 7f9a8f7fe640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a9006d940 0x7f9a901f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.648+0000 7f9a8effd640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a9006dda0 0x7f9a900a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.649+0000 7f9a8ffff640 1 -- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a9006d940 msgr2=0x7f9a901f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.649+0000 7f9a8ffff640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a9006d940 0x7f9a901f2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.649+0000 7f9a8ffff640 1 -- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a9006dda0 msgr2=0x7f9a900a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.649+0000 7f9a8ffff640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a9006dda0 0x7f9a900a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.649+0000 7f9a8ffff640 1 -- 172.21.15.44:0/2716918083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a900aa830 con 0x7f9a9013ef60 2024-04-02T13:24:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.649+0000 7f9a8effd640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a9006dda0 0x7f9a900a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.649+0000 7f9a8ffff640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a9013ef60 0x7f9a9006b4d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800cd90 tx=0x7f9a8800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.649+0000 7f9a77fff640 1 -- 172.21.15.44:0/2716918083 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88003b90 con 0x7f9a9013ef60 2024-04-02T13:24:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.649+0000 7f9a77fff640 1 -- 172.21.15.44:0/2716918083 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a88003d50 con 0x7f9a9013ef60 2024-04-02T13:24:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.649+0000 7f9a9600f640 1 -- 172.21.15.44:0/2716918083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a900aab10 con 0x7f9a9013ef60 2024-04-02T13:24:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.649+0000 7f9a77fff640 1 -- 172.21.15.44:0/2716918083 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88004260 con 0x7f9a9013ef60 2024-04-02T13:24:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.649+0000 7f9a9600f640 1 -- 172.21.15.44:0/2716918083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a900ab0d0 con 0x7f9a9013ef60 2024-04-02T13:24:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.650+0000 7f9a9600f640 1 -- 172.21.15.44:0/2716918083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a900684f0 con 0x7f9a9013ef60 2024-04-02T13:24:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.650+0000 7f9a77fff640 1 -- 172.21.15.44:0/2716918083 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a88004400 con 0x7f9a9013ef60 2024-04-02T13:24:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.651+0000 7f9a77fff640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9a7c06c920 0x7f9a7c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.651+0000 7f9a77fff640 1 -- 172.21.15.44:0/2716918083 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f9a8808c820 con 0x7f9a9013ef60 2024-04-02T13:24:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.651+0000 7f9a8f7fe640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9a7c06c920 0x7f9a7c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.651+0000 7f9a8f7fe640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9a7c06c920 0x7f9a7c06ede0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9a800098e0 tx=0x7f9a80009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.655+0000 7f9a77fff640 1 -- 172.21.15.44:0/2716918083 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9a8805a990 con 0x7f9a9013ef60 2024-04-02T13:24:39.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.840+0000 7f9a9600f640 1 -- 172.21.15.44:0/2716918083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9a9015bbb0 con 0x7f9a9013ef60 2024-04-02T13:24:39.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.840+0000 7f9a77fff640 1 -- 172.21.15.44:0/2716918083 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) ==== 104+0+11 (secure 0 0 0) 0x7f9a8805a290 con 0x7f9a9013ef60 2024-04-02T13:24:39.843 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:24:39.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.842+0000 7f9a9600f640 1 -- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9a7c06c920 msgr2=0x7f9a7c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.842+0000 7f9a9600f640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9a7c06c920 0x7f9a7c06ede0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9a800098e0 tx=0x7f9a80009060 comp rx=0 tx=0).stop 2024-04-02T13:24:39.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.842+0000 7f9a9600f640 1 -- 172.21.15.44:0/2716918083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a9013ef60 msgr2=0x7f9a9006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.842+0000 7f9a9600f640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a9013ef60 0x7f9a9006b4d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800cd90 tx=0x7f9a8800a660 comp rx=0 tx=0).stop 2024-04-02T13:24:39.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.842+0000 7f9a9600f640 1 -- 172.21.15.44:0/2716918083 shutdown_connections 2024-04-02T13:24:39.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.842+0000 7f9a9600f640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9a7c06c920 0x7f9a7c06ede0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.842+0000 7f9a9600f640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a9013ef60 0x7f9a9006b4d0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.842+0000 7f9a9600f640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a9006dda0 0x7f9a900a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.842+0000 7f9a9600f640 1 --2- 172.21.15.44:0/2716918083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a9006d940 0x7f9a901f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.842+0000 7f9a9600f640 1 -- 172.21.15.44:0/2716918083 >> 172.21.15.44:0/2716918083 conn(0x7f9a900b5d90 msgr2=0x7f9a900cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:39.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.842+0000 7f9a9600f640 1 -- 172.21.15.44:0/2716918083 shutdown_connections 2024-04-02T13:24:39.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.842+0000 7f9a9600f640 1 -- 172.21.15.44:0/2716918083 wait complete. 2024-04-02T13:24:39.855 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:24:39.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.957+0000 7f1c483c7640 1 Processor -- start 2024-04-02T13:24:39.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.957+0000 7f1c483c7640 1 -- start start 2024-04-02T13:24:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.958+0000 7f1c483c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c400cec60 0x7f1c400d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.958+0000 7f1c483c7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1c4015e5b0 0x7f1c400d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.958+0000 7f1c483c7640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c400cf1f0 con 0x7f1c4015e1d0 2024-04-02T13:24:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.958+0000 7f1c483c7640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f1c400cfe00 con 0x7f1c4015e5b0 2024-04-02T13:24:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.958+0000 7f1c483c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1c400cf4d0 con 0x7f1c400cec60 2024-04-02T13:24:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.959+0000 7f1c46964640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1c4015e5b0 0x7f1c400d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.959+0000 7f1c46163640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f1c4015e1d0 0x7f1c400d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48120/0 (socket says 172.21.15.44:48120) 2024-04-02T13:24:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.959+0000 7f1c46163640 1 -- 172.21.15.44:0/2746078467 learned_addr learned my addr 172.21.15.44:0/2746078467 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.959+0000 7f1c46964640 1 -- 172.21.15.44:0/2746078467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c400cec60 msgr2=0x7f1c400d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.959+0000 7f1c45962640 1 --2- 172.21.15.44:0/2746078467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c400cec60 0x7f1c400d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.959+0000 7f1c46964640 1 --2- 172.21.15.44:0/2746078467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c400cec60 0x7f1c400d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.959+0000 7f1c46964640 1 -- 172.21.15.44:0/2746078467 >> v1:172.21.15.44:6789/0 conn(0x7f1c4015e1d0 legacy=0x7f1c400d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.959+0000 7f1c45161640 1 -- 172.21.15.44:0/2746078467 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 207531443 0 0) 0x7f1c400cf1f0 con 0x7f1c4015e1d0 2024-04-02T13:24:39.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.959+0000 7f1c46964640 1 -- 172.21.15.44:0/2746078467 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c400cf7b0 con 0x7f1c4015e5b0 2024-04-02T13:24:39.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.959+0000 7f1c45962640 1 --2- 172.21.15.44:0/2746078467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c400cec60 0x7f1c400d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:39.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.960+0000 7f1c46964640 1 --2- 172.21.15.44:0/2746078467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4015e5b0 0x7f1c400d2c00 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009d90 tx=0x7f1c3802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453fb1ad1c5630a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:39.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.960+0000 7f1c45161640 1 -- 172.21.15.44:0/2746078467 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c38039070 con 0x7f1c4015e5b0 2024-04-02T13:24:39.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.960+0000 7f1c45161640 1 -- 172.21.15.44:0/2746078467 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c3802cbc0 con 0x7f1c4015e5b0 2024-04-02T13:24:39.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.960+0000 7f1c45161640 1 -- 172.21.15.44:0/2746078467 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c38034440 con 0x7f1c4015e5b0 2024-04-02T13:24:39.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.960+0000 7f1c483c7640 1 -- 172.21.15.44:0/2746078467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4015e5b0 msgr2=0x7f1c400d2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.960+0000 7f1c483c7640 1 --2- 172.21.15.44:0/2746078467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4015e5b0 0x7f1c400d2c00 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009d90 tx=0x7f1c3802be10 comp rx=0 tx=0).stop 2024-04-02T13:24:39.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.960+0000 7f1c483c7640 1 -- 172.21.15.44:0/2746078467 shutdown_connections 2024-04-02T13:24:39.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.960+0000 7f1c483c7640 1 --2- 172.21.15.44:0/2746078467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4015e5b0 0x7f1c400d2c00 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.960+0000 7f1c483c7640 1 --2- 172.21.15.44:0/2746078467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c400cec60 0x7f1c400d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.960+0000 7f1c483c7640 1 -- 172.21.15.44:0/2746078467 >> 172.21.15.44:0/2746078467 conn(0x7f1c4009d3b0 msgr2=0x7f1c400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:39.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.960+0000 7f1c483c7640 1 -- 172.21.15.44:0/2746078467 shutdown_connections 2024-04-02T13:24:39.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.960+0000 7f1c483c7640 1 -- 172.21.15.44:0/2746078467 wait complete. 2024-04-02T13:24:39.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.962+0000 7f1c483c7640 1 Processor -- start 2024-04-02T13:24:39.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.962+0000 7f1c483c7640 1 -- start start 2024-04-02T13:24:39.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.962+0000 7f1c483c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c400cec60 0x7f1c40152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.962+0000 7f1c483c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4015e1d0 0x7f1c40153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.962+0000 7f1c483c7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4015e5b0 0x7f1c4015ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.962+0000 7f1c483c7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1c400b4e60 con 0x7f1c4015e5b0 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.962+0000 7f1c483c7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1c400b4ce0 con 0x7f1c4015e1d0 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.962+0000 7f1c483c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1c400b4fe0 con 0x7f1c400cec60 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.962+0000 7f1c46964640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4015e5b0 0x7f1c4015ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.962+0000 7f1c46964640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4015e5b0 0x7f1c4015ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47658/0 (socket says 172.21.15.44:47658) 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.962+0000 7f1c46964640 1 -- 172.21.15.44:0/1113941382 learned_addr learned my addr 172.21.15.44:0/1113941382 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.963+0000 7f1c46163640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c400cec60 0x7f1c40152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.963+0000 7f1c45962640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4015e1d0 0x7f1c40153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.963+0000 7f1c46964640 1 -- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c400cec60 msgr2=0x7f1c40152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.963+0000 7f1c46964640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c400cec60 0x7f1c40152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.963+0000 7f1c46964640 1 -- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4015e1d0 msgr2=0x7f1c40153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.963+0000 7f1c46964640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4015e1d0 0x7f1c40153040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.963+0000 7f1c46964640 1 -- 172.21.15.44:0/1113941382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c40153a20 con 0x7f1c4015e5b0 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.963+0000 7f1c46964640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4015e5b0 0x7f1c4015ab90 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009d90 tx=0x7f1c38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:39.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.963+0000 7f1c36ffd640 1 -- 172.21.15.44:0/1113941382 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c38039050 con 0x7f1c4015e5b0 2024-04-02T13:24:39.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.963+0000 7f1c36ffd640 1 -- 172.21.15.44:0/1113941382 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c38045070 con 0x7f1c4015e5b0 2024-04-02T13:24:39.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.963+0000 7f1c483c7640 1 -- 172.21.15.44:0/1113941382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c40153d00 con 0x7f1c4015e5b0 2024-04-02T13:24:39.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.963+0000 7f1c36ffd640 1 -- 172.21.15.44:0/1113941382 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3803e830 con 0x7f1c4015e5b0 2024-04-02T13:24:39.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.963+0000 7f1c483c7640 1 -- 172.21.15.44:0/1113941382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c401542c0 con 0x7f1c4015e5b0 2024-04-02T13:24:39.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.964+0000 7f1c483c7640 1 -- 172.21.15.44:0/1113941382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c400684f0 con 0x7f1c4015e5b0 2024-04-02T13:24:39.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.965+0000 7f1c36ffd640 1 -- 172.21.15.44:0/1113941382 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1c3803e9d0 con 0x7f1c4015e5b0 2024-04-02T13:24:39.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.965+0000 7f1c36ffd640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1c2406c8f0 0x7f1c2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:39.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.965+0000 7f1c36ffd640 1 -- 172.21.15.44:0/1113941382 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f1c380aeb80 con 0x7f1c4015e5b0 2024-04-02T13:24:39.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.965+0000 7f1c46163640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1c2406c8f0 0x7f1c2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:39.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.966+0000 7f1c46163640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1c2406c8f0 0x7f1c2406edb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c000f60 tx=0x7f1c3c007b50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:39.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:39.969+0000 7f1c36ffd640 1 -- 172.21.15.44:0/1113941382 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1c3807ceb0 con 0x7f1c4015e5b0 2024-04-02T13:24:40.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.152+0000 7f1c483c7640 1 -- 172.21.15.44:0/1113941382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1c400d47f0 con 0x7f1c4015e5b0 2024-04-02T13:24:40.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.152+0000 7f1c36ffd640 1 -- 172.21.15.44:0/1113941382 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) ==== 100+0+11 (secure 0 0 0) 0x7f1c3807c7b0 con 0x7f1c4015e5b0 2024-04-02T13:24:40.155 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:24:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.154+0000 7f1c483c7640 1 -- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1c2406c8f0 msgr2=0x7f1c2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.154+0000 7f1c483c7640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1c2406c8f0 0x7f1c2406edb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c000f60 tx=0x7f1c3c007b50 comp rx=0 tx=0).stop 2024-04-02T13:24:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.155+0000 7f1c483c7640 1 -- 172.21.15.44:0/1113941382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4015e5b0 msgr2=0x7f1c4015ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.155+0000 7f1c483c7640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4015e5b0 0x7f1c4015ab90 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009d90 tx=0x7f1c38004a40 comp rx=0 tx=0).stop 2024-04-02T13:24:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.155+0000 7f1c483c7640 1 -- 172.21.15.44:0/1113941382 shutdown_connections 2024-04-02T13:24:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.155+0000 7f1c483c7640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1c2406c8f0 0x7f1c2406edb0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.155+0000 7f1c483c7640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4015e5b0 0x7f1c4015ab90 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.155+0000 7f1c483c7640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4015e1d0 0x7f1c40153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.155+0000 7f1c483c7640 1 --2- 172.21.15.44:0/1113941382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c400cec60 0x7f1c40152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.155+0000 7f1c483c7640 1 -- 172.21.15.44:0/1113941382 >> 172.21.15.44:0/1113941382 conn(0x7f1c4009d3b0 msgr2=0x7f1c400d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.155+0000 7f1c483c7640 1 -- 172.21.15.44:0/1113941382 shutdown_connections 2024-04-02T13:24:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.155+0000 7f1c483c7640 1 -- 172.21.15.44:0/1113941382 wait complete. 2024-04-02T13:24:40.167 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:24:40.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.267+0000 7f1006c75640 1 Processor -- start 2024-04-02T13:24:40.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.267+0000 7f1006c75640 1 -- start start 2024-04-02T13:24:40.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.268+0000 7f1006c75640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10000d0a30 0x7f100006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.268+0000 7f1006c75640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10000d0e10 0x7f10000cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.268+0000 7f1006c75640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f100006c1d0 con 0x7f10000cfe30 2024-04-02T13:24:40.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.268+0000 7f1006c75640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f100006cb00 con 0x7f10000d0e10 2024-04-02T13:24:40.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.268+0000 7f1006c75640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f100006c4b0 con 0x7f10000d0a30 2024-04-02T13:24:40.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.269+0000 7f1004a11640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10000d0a30 0x7f100006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.269+0000 7f0ffffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10000d0e10 0x7f10000cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.269+0000 7f0ffffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10000d0e10 0x7f10000cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42108/0 (socket says 172.21.15.44:42108) 2024-04-02T13:24:40.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.269+0000 7f1004a11640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10000d0a30 0x7f100006cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:33194/0 (socket says 172.21.15.44:33194) 2024-04-02T13:24:40.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.269+0000 7f0ffffff640 1 -- 172.21.15.44:0/3719949907 learned_addr learned my addr 172.21.15.44:0/3719949907 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:40.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.269+0000 7f1004a11640 1 -- 172.21.15.44:0/3719949907 >> v2:172.21.15.69:3300/0 conn(0x7f10000d0e10 msgr2=0x7f10000cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:40.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.269+0000 7f1004a11640 1 --2- 172.21.15.44:0/3719949907 >> v2:172.21.15.69:3300/0 conn(0x7f10000d0e10 0x7f10000cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.269+0000 7f1004a11640 1 -- 172.21.15.44:0/3719949907 >> v1:172.21.15.69:6789/0 conn(0x7f10000cfe30 legacy=0x7f10000d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.269+0000 7f1004a11640 1 -- 172.21.15.44:0/3719949907 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f100006c790 con 0x7f10000d0a30 2024-04-02T13:24:40.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.270+0000 7f1004a11640 1 --2- 172.21.15.44:0/3719949907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10000d0a30 0x7f100006cf90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009d90 tx=0x7f0ff802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7854e92a3c066df server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:40.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.270+0000 7f0fff7fe640 1 -- 172.21.15.44:0/3719949907 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8039070 con 0x7f10000d0a30 2024-04-02T13:24:40.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.270+0000 7f0fff7fe640 1 -- 172.21.15.44:0/3719949907 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ff802cbc0 con 0x7f10000d0a30 2024-04-02T13:24:40.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.270+0000 7f0fff7fe640 1 -- 172.21.15.44:0/3719949907 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8034440 con 0x7f10000d0a30 2024-04-02T13:24:40.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.270+0000 7f1006c75640 1 -- 172.21.15.44:0/3719949907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10000d0a30 msgr2=0x7f100006cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.270+0000 7f1006c75640 1 --2- 172.21.15.44:0/3719949907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10000d0a30 0x7f100006cf90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009d90 tx=0x7f0ff802be10 comp rx=0 tx=0).stop 2024-04-02T13:24:40.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.271+0000 7f1006c75640 1 -- 172.21.15.44:0/3719949907 shutdown_connections 2024-04-02T13:24:40.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.271+0000 7f1006c75640 1 --2- 172.21.15.44:0/3719949907 >> v2:172.21.15.69:3300/0 conn(0x7f10000d0e10 0x7f10000cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.271+0000 7f1006c75640 1 --2- 172.21.15.44:0/3719949907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10000d0a30 0x7f100006cf90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.271+0000 7f1006c75640 1 -- 172.21.15.44:0/3719949907 >> 172.21.15.44:0/3719949907 conn(0x7f100009d470 msgr2=0x7f10000d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:40.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.271+0000 7f1006c75640 1 -- 172.21.15.44:0/3719949907 shutdown_connections 2024-04-02T13:24:40.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.271+0000 7f1006c75640 1 -- 172.21.15.44:0/3719949907 wait complete. 2024-04-02T13:24:40.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.272+0000 7f1006c75640 1 Processor -- start 2024-04-02T13:24:40.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.272+0000 7f1006c75640 1 -- start start 2024-04-02T13:24:40.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f1006c75640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10000cfe30 0x7f10000b64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f1006c75640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10000d0a30 0x7f10001f8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f1006c75640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10000d0e10 0x7f10000b90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f1006c75640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f100015e870 con 0x7f10000d0a30 2024-04-02T13:24:40.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f1006c75640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f100015e6f0 con 0x7f10000cfe30 2024-04-02T13:24:40.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f1006c75640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f10000d4da0 con 0x7f10000d0e10 2024-04-02T13:24:40.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f0ffffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10000d0a30 0x7f10001f8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f0ffffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10000d0a30 0x7f10001f8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47666/0 (socket says 172.21.15.44:47666) 2024-04-02T13:24:40.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f0ffffff640 1 -- 172.21.15.44:0/1586736232 learned_addr learned my addr 172.21.15.44:0/1586736232 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:40.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f1004a11640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10000cfe30 0x7f10000b64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f1005212640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10000d0e10 0x7f10000b90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f0ffffff640 1 -- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10000d0e10 msgr2=0x7f10000b90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f0ffffff640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10000d0e10 0x7f10000b90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f0ffffff640 1 -- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10000cfe30 msgr2=0x7f10000b64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f0ffffff640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10000cfe30 0x7f10000b64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f0ffffff640 1 -- 172.21.15.44:0/1586736232 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1000159480 con 0x7f10000d0a30 2024-04-02T13:24:40.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f0ffffff640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10000d0a30 0x7f10001f8770 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009fd0 tx=0x7f0fe800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:40.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f0ffd7fa640 1 -- 172.21.15.44:0/1586736232 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8017070 con 0x7f10000d0a30 2024-04-02T13:24:40.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f0ffd7fa640 1 -- 172.21.15.44:0/1586736232 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fe800c940 con 0x7f10000d0a30 2024-04-02T13:24:40.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f1006c75640 1 -- 172.21.15.44:0/1586736232 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10000b3cd0 con 0x7f10000d0a30 2024-04-02T13:24:40.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f0ffd7fa640 1 -- 172.21.15.44:0/1586736232 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe800cd10 con 0x7f10000d0a30 2024-04-02T13:24:40.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.273+0000 7f1006c75640 1 -- 172.21.15.44:0/1586736232 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10001591a0 con 0x7f10000d0a30 2024-04-02T13:24:40.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.274+0000 7f1006c75640 1 -- 172.21.15.44:0/1586736232 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1000073730 con 0x7f10000d0a30 2024-04-02T13:24:40.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.275+0000 7f0ffd7fa640 1 -- 172.21.15.44:0/1586736232 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0fe801a070 con 0x7f10000d0a30 2024-04-02T13:24:40.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.275+0000 7f0ffd7fa640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0fe006c920 0x7f0fe006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.275+0000 7f0ffd7fa640 1 -- 172.21.15.44:0/1586736232 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f0fe808bb40 con 0x7f10000d0a30 2024-04-02T13:24:40.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.275+0000 7f1004a11640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0fe006c920 0x7f0fe006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.276+0000 7f1004a11640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0fe006c920 0x7f0fe006ede0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8036000 tx=0x7f0ff8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:40.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.280+0000 7f0ffd7fa640 1 -- 172.21.15.44:0/1586736232 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1000073730 con 0x7f10000d0a30 2024-04-02T13:24:40.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.502+0000 7f1006c75640 1 -- 172.21.15.44:0/1586736232 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10000684f0 con 0x7f10000d0a30 2024-04-02T13:24:40.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.503+0000 7f0ffd7fa640 1 -- 172.21.15.44:0/1586736232 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5180 (secure 0 0 0) 0x7f0fe8059790 con 0x7f10000d0a30 2024-04-02T13:24:40.506 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:40.506 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":37,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1535287484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1535287484},{"type":"v1","addr":"172.21.15.69:6836","nonce":1535287484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6429,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2367107002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2367107002},{"type":"v1","addr":"172.21.15.44:6833","nonce":2367107002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/553542317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":553542317},{"type":"v1","addr":"172.21.15.69:6837","nonce":553542317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:23:24.803884+0000","modified":"2024-04-02T13:24:05.482501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6450":{"gid":6450,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":13,"addr":"172.21.15.44:6835/3127613802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3127613802},{"type":"v1","addr":"172.21.15.44:6835","nonce":3127613802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6450,"qdb_cluster":[6450]},"id":4}]} 2024-04-02T13:24:40.506 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 37 2024-04-02T13:24:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.505+0000 7f1006c75640 1 -- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0fe006c920 msgr2=0x7f0fe006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.505+0000 7f1006c75640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0fe006c920 0x7f0fe006ede0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8036000 tx=0x7f0ff8030000 comp rx=0 tx=0).stop 2024-04-02T13:24:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.505+0000 7f1006c75640 1 -- 172.21.15.44:0/1586736232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10000d0a30 msgr2=0x7f10001f8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.505+0000 7f1006c75640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10000d0a30 0x7f10001f8770 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009fd0 tx=0x7f0fe800ed40 comp rx=0 tx=0).stop 2024-04-02T13:24:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.505+0000 7f1006c75640 1 -- 172.21.15.44:0/1586736232 shutdown_connections 2024-04-02T13:24:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.505+0000 7f1006c75640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0fe006c920 0x7f0fe006ede0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.505+0000 7f1006c75640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10000d0e10 0x7f10000b90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.505+0000 7f1006c75640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10000d0a30 0x7f10001f8770 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.505+0000 7f1006c75640 1 --2- 172.21.15.44:0/1586736232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10000cfe30 0x7f10000b64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.505+0000 7f1006c75640 1 -- 172.21.15.44:0/1586736232 >> 172.21.15.44:0/1586736232 conn(0x7f100009d470 msgr2=0x7f10000cee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.505+0000 7f1006c75640 1 -- 172.21.15.44:0/1586736232 shutdown_connections 2024-04-02T13:24:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.505+0000 7f1006c75640 1 -- 172.21.15.44:0/1586736232 wait complete. 2024-04-02T13:24:40.519 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:24:40.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.620+0000 7fe4f5dbc640 1 Processor -- start 2024-04-02T13:24:40.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.620+0000 7fe4f5dbc640 1 -- start start 2024-04-02T13:24:40.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4f5dbc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe4f006ece0 0x7fe4f0165ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4f5dbc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4f006dc40 con 0x7fe4f00cece0 2024-04-02T13:24:40.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4f5dbc640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4f006df20 con 0x7fe4f00d09f0 2024-04-02T13:24:40.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4f5dbc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe4f006d9e0 con 0x7fe4f006ece0 2024-04-02T13:24:40.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4ef7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe4f00cece0 0x7fe4f00cf0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48130/0 (socket says 172.21.15.44:48130) 2024-04-02T13:24:40.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4ef7fe640 1 -- 172.21.15.44:0/3136757214 learned_addr learned my addr 172.21.15.44:0/3136757214 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:40.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4effff640 1 --2- 172.21.15.44:0/3136757214 >> v2:172.21.15.69:3300/0 conn(0x7fe4f006ece0 0x7fe4f0165ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2121958941 0 0) 0x7fe4f006dc40 con 0x7fe4f00cece0 2024-04-02T13:24:40.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4dc003520 con 0x7fe4f00cece0 2024-04-02T13:24:40.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1580777008 0 0) 0x7fe4dc003520 con 0x7fe4f00cece0 2024-04-02T13:24:40.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 >> v2:172.21.15.69:3300/0 conn(0x7fe4f006ece0 msgr2=0x7fe4f0165ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:40.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 --2- 172.21.15.44:0/3136757214 >> v2:172.21.15.69:3300/0 conn(0x7fe4f006ece0 0x7fe4f0165ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 >> v1:172.21.15.69:6789/0 conn(0x7fe4f00d09f0 legacy=0x7fe4f006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f006e200 con 0x7fe4f00cece0 2024-04-02T13:24:40.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fe4e0001670 con 0x7fe4f00cece0 2024-04-02T13:24:40.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4effff640 1 --2- 172.21.15.44:0/3136757214 >> v2:172.21.15.69:3300/0 conn(0x7fe4f006ece0 0x7fe4f0165ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:24:40.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 >> v1:172.21.15.44:6789/0 conn(0x7fe4f00cece0 legacy=0x7fe4f00cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 --2- 172.21.15.44:0/3136757214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0039b0 0x7fe4dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 --2- 172.21.15.44:0/3136757214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe4dc0043b0 0x7fe4dc004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 --2- 172.21.15.44:0/3136757214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4dc005080 0x7fe4dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f006d9e0 con 0x7fe4dc0043b0 2024-04-02T13:24:40.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc00bfe0 con 0x7fe4dc0039b0 2024-04-02T13:24:40.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.621+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe4dc00c1e0 con 0x7fe4dc005080 2024-04-02T13:24:40.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.622+0000 7fe4e7fff640 1 --2- 172.21.15.44:0/3136757214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe4dc0043b0 0x7fe4dc004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.622+0000 7fe4ef7fe640 1 --2- 172.21.15.44:0/3136757214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0039b0 0x7fe4dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.622+0000 7fe4effff640 1 --2- 172.21.15.44:0/3136757214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4dc005080 0x7fe4dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.622+0000 7fe4e7fff640 1 -- 172.21.15.44:0/3136757214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4dc005080 msgr2=0x7fe4dc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:40.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.622+0000 7fe4e7fff640 1 --2- 172.21.15.44:0/3136757214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4dc005080 0x7fe4dc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.622+0000 7fe4e7fff640 1 -- 172.21.15.44:0/3136757214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0039b0 msgr2=0x7fe4dc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:40.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.622+0000 7fe4e7fff640 1 --2- 172.21.15.44:0/3136757214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0039b0 0x7fe4dc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.622+0000 7fe4effff640 1 -- 172.21.15.44:0/3136757214 reap_dead start 2024-04-02T13:24:40.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.622+0000 7fe4e7fff640 1 -- 172.21.15.44:0/3136757214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4dc003520 con 0x7fe4dc0043b0 2024-04-02T13:24:40.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.623+0000 7fe4e7fff640 1 --2- 172.21.15.44:0/3136757214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe4dc0043b0 0x7fe4dc004880 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8001860 tx=0x7fe4dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5766d61ce382fc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:40.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.623+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8038070 con 0x7fe4dc0043b0 2024-04-02T13:24:40.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.623+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4d8033410 con 0x7fe4dc0043b0 2024-04-02T13:24:40.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.623+0000 7fe4eeffd640 1 -- 172.21.15.44:0/3136757214 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8001430 con 0x7fe4dc0043b0 2024-04-02T13:24:40.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.623+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/3136757214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe4dc0043b0 msgr2=0x7fe4dc004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.623+0000 7fe4f5dbc640 1 --2- 172.21.15.44:0/3136757214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe4dc0043b0 0x7fe4dc004880 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8001860 tx=0x7fe4dc005830 comp rx=0 tx=0).stop 2024-04-02T13:24:40.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.623+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/3136757214 shutdown_connections 2024-04-02T13:24:40.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.623+0000 7fe4f5dbc640 1 --2- 172.21.15.44:0/3136757214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe4dc0043b0 0x7fe4dc004880 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.623+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/3136757214 >> 172.21.15.44:0/3136757214 conn(0x7fe4f00cfcd0 msgr2=0x7fe4f00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:40.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.623+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/3136757214 shutdown_connections 2024-04-02T13:24:40.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.623+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/3136757214 wait complete. 2024-04-02T13:24:40.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.625+0000 7fe4f5dbc640 1 Processor -- start 2024-04-02T13:24:40.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.625+0000 7fe4f5dbc640 1 -- start start 2024-04-02T13:24:40.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.625+0000 7fe4f5dbc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4dc0043b0 0x7fe4f00c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.625+0000 7fe4f5dbc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4f0168450 0x7fe4f00acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.625+0000 7fe4f5dbc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe4f00ad290 0x7fe4f0152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.625+0000 7fe4f5dbc640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f015b220 con 0x7fe4f00ad290 2024-04-02T13:24:40.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.625+0000 7fe4f5dbc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f006e7c0 con 0x7fe4f0168450 2024-04-02T13:24:40.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.625+0000 7fe4f5dbc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe4f015b3a0 con 0x7fe4dc0043b0 2024-04-02T13:24:40.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.625+0000 7fe4effff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe4f00ad290 0x7fe4f0152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.625+0000 7fe4effff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe4f00ad290 0x7fe4f0152050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47684/0 (socket says 172.21.15.44:47684) 2024-04-02T13:24:40.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.625+0000 7fe4effff640 1 -- 172.21.15.44:0/1618322804 learned_addr learned my addr 172.21.15.44:0/1618322804 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:40.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.625+0000 7fe4e7fff640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4f0168450 0x7fe4f00acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.626+0000 7fe4ef7fe640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4dc0043b0 0x7fe4f00c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.626+0000 7fe4effff640 1 -- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4dc0043b0 msgr2=0x7fe4f00c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.626+0000 7fe4effff640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4dc0043b0 0x7fe4f00c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.626+0000 7fe4effff640 1 -- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4f0168450 msgr2=0x7fe4f00acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.626+0000 7fe4effff640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4f0168450 0x7fe4f00acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.626+0000 7fe4effff640 1 -- 172.21.15.44:0/1618322804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f0168c80 con 0x7fe4f00ad290 2024-04-02T13:24:40.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.626+0000 7fe4effff640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe4f00ad290 0x7fe4f0152050 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0070e80 tx=0x7fe4e800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:40.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.626+0000 7fe4ecff9640 1 -- 172.21.15.44:0/1618322804 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8003990 con 0x7fe4f00ad290 2024-04-02T13:24:40.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.626+0000 7fe4ecff9640 1 -- 172.21.15.44:0/1618322804 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4e8003b30 con 0x7fe4f00ad290 2024-04-02T13:24:40.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.626+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/1618322804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4f0168f60 con 0x7fe4f00ad290 2024-04-02T13:24:40.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.626+0000 7fe4ecff9640 1 -- 172.21.15.44:0/1618322804 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8004020 con 0x7fe4f00ad290 2024-04-02T13:24:40.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.626+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/1618322804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4f01688d0 con 0x7fe4f00ad290 2024-04-02T13:24:40.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.627+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/1618322804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4f0074f20 con 0x7fe4f00ad290 2024-04-02T13:24:40.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.628+0000 7fe4ecff9640 1 -- 172.21.15.44:0/1618322804 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe4e80041c0 con 0x7fe4f00ad290 2024-04-02T13:24:40.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.628+0000 7fe4ecff9640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe4d006c920 0x7fe4d006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.628+0000 7fe4ecff9640 1 -- 172.21.15.44:0/1618322804 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7fe4e808c900 con 0x7fe4f00ad290 2024-04-02T13:24:40.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.628+0000 7fe4ef7fe640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe4d006c920 0x7fe4d006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.629+0000 7fe4ef7fe640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe4d006c920 0x7fe4d006ede0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00adba0 tx=0x7fe4e00078f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:40.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.633+0000 7fe4ecff9640 1 -- 172.21.15.44:0/1618322804 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe4e8019030 con 0x7fe4f00ad290 2024-04-02T13:24:40.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.819+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/1618322804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe4f0169480 con 0x7fe4f00ad290 2024-04-02T13:24:40.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.820+0000 7fe4ecff9640 1 -- 172.21.15.44:0/1618322804 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+17076 (secure 0 0 0) 0x7fe4e8010040 con 0x7fe4f00ad290 2024-04-02T13:24:40.822 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:40.822 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":67,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:24:09.591968+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:23:23.126712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-02T13:23:23.763593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.18","mappings":[{"from":7,"to":2}]},{"pgid":"10.4","mappings":[{"from":4,"to":3}]},{"pgid":"10.10","mappings":[{"from":4,"to":3}]},{"pgid":"10.37","mappings":[{"from":4,"to":5}]},{"pgid":"10.39","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:24:40.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.822+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe4d006c920 msgr2=0x7fe4d006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.822+0000 7fe4f5dbc640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe4d006c920 0x7fe4d006ede0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00adba0 tx=0x7fe4e00078f0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.822+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/1618322804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe4f00ad290 msgr2=0x7fe4f0152050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.822+0000 7fe4f5dbc640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe4f00ad290 0x7fe4f0152050 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0070e80 tx=0x7fe4e800f580 comp rx=0 tx=0).stop 2024-04-02T13:24:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.822+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/1618322804 shutdown_connections 2024-04-02T13:24:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.822+0000 7fe4f5dbc640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe4d006c920 0x7fe4d006ede0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.822+0000 7fe4f5dbc640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe4f00ad290 0x7fe4f0152050 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.822+0000 7fe4f5dbc640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4f0168450 0x7fe4f00acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.822+0000 7fe4f5dbc640 1 --2- 172.21.15.44:0/1618322804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4dc0043b0 0x7fe4f00c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.822+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/1618322804 >> 172.21.15.44:0/1618322804 conn(0x7fe4f00cfcd0 msgr2=0x7fe4f006d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.822+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/1618322804 shutdown_connections 2024-04-02T13:24:40.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.822+0000 7fe4f5dbc640 1 -- 172.21.15.44:0/1618322804 wait complete. 2024-04-02T13:24:40.835 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T13:24:40.835 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:24:40.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.934+0000 7f5929024640 1 Processor -- start 2024-04-02T13:24:40.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.934+0000 7f5929024640 1 -- start start 2024-04-02T13:24:40.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.935+0000 7f5929024640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f59240d0a30 0x7f592406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.935+0000 7f5929024640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f59240d2040 0x7f59240d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.935+0000 7f5929024640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59240d0000 con 0x7f59240d0e10 2024-04-02T13:24:40.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.935+0000 7f5929024640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f592406cb00 con 0x7f59240d0a30 2024-04-02T13:24:40.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.935+0000 7f5929024640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f592406c1d0 con 0x7f59240d2040 2024-04-02T13:24:40.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.935+0000 7f5922d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f59240d0a30 0x7f592406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.935+0000 7f5922d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f59240d0a30 0x7f592406cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47700/0 (socket says 172.21.15.44:47700) 2024-04-02T13:24:40.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.935+0000 7f5922d9d640 1 -- 172.21.15.44:0/4101618144 learned_addr learned my addr 172.21.15.44:0/4101618144 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:40.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.935+0000 7f592359e640 1 --2- 172.21.15.44:0/4101618144 >> v2:172.21.15.69:3300/0 conn(0x7f59240d2040 0x7f59240d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.935+0000 7f5922d9d640 1 -- 172.21.15.44:0/4101618144 >> v2:172.21.15.69:3300/0 conn(0x7f59240d2040 msgr2=0x7f59240d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:40.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.935+0000 7f5922d9d640 1 --2- 172.21.15.44:0/4101618144 >> v2:172.21.15.69:3300/0 conn(0x7f59240d2040 0x7f59240d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.935+0000 7f5922d9d640 1 -- 172.21.15.44:0/4101618144 >> v1:172.21.15.69:6789/0 conn(0x7f59240d0e10 legacy=0x7f59240cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.935+0000 7f5922d9d640 1 -- 172.21.15.44:0/4101618144 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f592406c4b0 con 0x7f59240d0a30 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.936+0000 7f5922d9d640 1 --2- 172.21.15.44:0/4101618144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59240d0a30 0x7f592406cf90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f592408c000 tx=0x7f591c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06133a8198ca58 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.936+0000 7f5921d9b640 1 -- 172.21.15.44:0/4101618144 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c02e070 con 0x7f59240d0a30 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.936+0000 7f5921d9b640 1 -- 172.21.15.44:0/4101618144 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f591c037070 con 0x7f59240d0a30 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.936+0000 7f5921d9b640 1 -- 172.21.15.44:0/4101618144 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c031a00 con 0x7f59240d0a30 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.937+0000 7f5929024640 1 -- 172.21.15.44:0/4101618144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59240d0a30 msgr2=0x7f592406cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.937+0000 7f5929024640 1 --2- 172.21.15.44:0/4101618144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59240d0a30 0x7f592406cf90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f592408c000 tx=0x7f591c02b880 comp rx=0 tx=0).stop 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.937+0000 7f5929024640 1 -- 172.21.15.44:0/4101618144 shutdown_connections 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.937+0000 7f5929024640 1 --2- 172.21.15.44:0/4101618144 >> v2:172.21.15.69:3300/0 conn(0x7f59240d2040 0x7f59240d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.937+0000 7f5929024640 1 --2- 172.21.15.44:0/4101618144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59240d0a30 0x7f592406cf90 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.937+0000 7f5929024640 1 -- 172.21.15.44:0/4101618144 >> 172.21.15.44:0/4101618144 conn(0x7f592409d470 msgr2=0x7f59240d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.937+0000 7f5929024640 1 -- 172.21.15.44:0/4101618144 shutdown_connections 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.937+0000 7f5929024640 1 -- 172.21.15.44:0/4101618144 wait complete. 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.938+0000 7f5929024640 1 Processor -- start 2024-04-02T13:24:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.938+0000 7f5929024640 1 -- start start 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f5929024640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59240d0a30 0x7f59241551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f5929024640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59240d0e10 0x7f59240cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f5929024640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59240d2040 0x7f59240b9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f5929024640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f592415e6f0 con 0x7f59240d0a30 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f5929024640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f59240d1270 con 0x7f59240d2040 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f5929024640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f592415e870 con 0x7f59240d0e10 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f5922d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59240d0a30 0x7f59241551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f5922d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59240d0a30 0x7f59241551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47716/0 (socket says 172.21.15.44:47716) 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f5922d9d640 1 -- 172.21.15.44:0/4094082337 learned_addr learned my addr 172.21.15.44:0/4094082337 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f592259c640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59240d0e10 0x7f59240cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f592359e640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59240d2040 0x7f59240b9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f5922d9d640 1 -- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59240d0e10 msgr2=0x7f59240cdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f5922d9d640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59240d0e10 0x7f59240cdb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f5922d9d640 1 -- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59240d2040 msgr2=0x7f59240b9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.939+0000 7f5922d9d640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59240d2040 0x7f59240b9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.940+0000 7f5922d9d640 1 -- 172.21.15.44:0/4094082337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59240b8190 con 0x7f59240d0a30 2024-04-02T13:24:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.940+0000 7f592359e640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59240d2040 0x7f59240b9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.940+0000 7f5922d9d640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59240d0a30 0x7f59241551a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f591c02b8c0 tx=0x7f591c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.940+0000 7f591b7fe640 1 -- 172.21.15.44:0/4094082337 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c032670 con 0x7f59240d0a30 2024-04-02T13:24:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.940+0000 7f591b7fe640 1 -- 172.21.15.44:0/4094082337 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f591c032830 con 0x7f59240d0a30 2024-04-02T13:24:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.940+0000 7f5929024640 1 -- 172.21.15.44:0/4094082337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59240b8450 con 0x7f59240d0a30 2024-04-02T13:24:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.940+0000 7f591b7fe640 1 -- 172.21.15.44:0/4094082337 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c032c20 con 0x7f59240d0a30 2024-04-02T13:24:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.940+0000 7f5929024640 1 -- 172.21.15.44:0/4094082337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59240b8a10 con 0x7f59240d0a30 2024-04-02T13:24:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.941+0000 7f5929024640 1 -- 172.21.15.44:0/4094082337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f592406df70 con 0x7f59240d0a30 2024-04-02T13:24:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.941+0000 7f591b7fe640 1 -- 172.21.15.44:0/4094082337 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f591c04c060 con 0x7f59240d0a30 2024-04-02T13:24:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.942+0000 7f591b7fe640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f590406c920 0x7f590406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:40.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.942+0000 7f591b7fe640 1 -- 172.21.15.44:0/4094082337 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f591c0ad770 con 0x7f59240d0a30 2024-04-02T13:24:40.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.942+0000 7f592259c640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f590406c920 0x7f590406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:40.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.943+0000 7f592259c640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f590406c920 0x7f590406ede0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f590c006fd0 tx=0x7f590c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:40.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:40.946+0000 7f591b7fe640 1 -- 172.21.15.44:0/4094082337 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f592406df70 con 0x7f59240d0a30 2024-04-02T13:24:41.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.169+0000 7f5929024640 1 -- 172.21.15.44:0/4094082337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59240d3ab0 con 0x7f59240d0a30 2024-04-02T13:24:41.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.169+0000 7f591b7fe640 1 -- 172.21.15.44:0/4094082337 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5180 (secure 0 0 0) 0x7f591c040070 con 0x7f59240d0a30 2024-04-02T13:24:41.171 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:41.171 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":37,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1535287484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1535287484},{"type":"v1","addr":"172.21.15.69:6836","nonce":1535287484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6429,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2367107002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2367107002},{"type":"v1","addr":"172.21.15.44:6833","nonce":2367107002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/553542317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":553542317},{"type":"v1","addr":"172.21.15.69:6837","nonce":553542317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:23:24.803884+0000","modified":"2024-04-02T13:24:05.482501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6450":{"gid":6450,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":13,"addr":"172.21.15.44:6835/3127613802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3127613802},{"type":"v1","addr":"172.21.15.44:6835","nonce":3127613802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6450,"qdb_cluster":[6450]},"id":4}]} 2024-04-02T13:24:41.171 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 37 2024-04-02T13:24:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.171+0000 7f5929024640 1 -- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f590406c920 msgr2=0x7f590406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.171+0000 7f5929024640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f590406c920 0x7f590406ede0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f590c006fd0 tx=0x7f590c008040 comp rx=0 tx=0).stop 2024-04-02T13:24:41.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.171+0000 7f5929024640 1 -- 172.21.15.44:0/4094082337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59240d0a30 msgr2=0x7f59241551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.171+0000 7f5929024640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59240d0a30 0x7f59241551a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f591c02b8c0 tx=0x7f591c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.171+0000 7f5929024640 1 -- 172.21.15.44:0/4094082337 shutdown_connections 2024-04-02T13:24:41.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.171+0000 7f5929024640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f590406c920 0x7f590406ede0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.171+0000 7f5929024640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59240d2040 0x7f59240b9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.171+0000 7f5929024640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59240d0e10 0x7f59240cdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.171+0000 7f5929024640 1 --2- 172.21.15.44:0/4094082337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59240d0a30 0x7f59241551a0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.171+0000 7f5929024640 1 -- 172.21.15.44:0/4094082337 >> 172.21.15.44:0/4094082337 conn(0x7f592409d470 msgr2=0x7f59241566b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:41.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.171+0000 7f5929024640 1 -- 172.21.15.44:0/4094082337 shutdown_connections 2024-04-02T13:24:41.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.171+0000 7f5929024640 1 -- 172.21.15.44:0/4094082337 wait complete. 2024-04-02T13:24:41.184 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:24:41.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.283+0000 7f3e19ad3640 1 Processor -- start 2024-04-02T13:24:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.283+0000 7f3e19ad3640 1 -- start start 2024-04-02T13:24:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.284+0000 7f3e19ad3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e140d0a30 0x7f3e1406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.284+0000 7f3e19ad3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e140d0000 con 0x7f3e140d0e10 2024-04-02T13:24:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.284+0000 7f3e19ad3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1406c1d0 con 0x7f3e140d2040 2024-04-02T13:24:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.284+0000 7f3e19ad3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3e1406cb00 con 0x7f3e140d0a30 2024-04-02T13:24:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.284+0000 7f3e12ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f3e140d0e10 0x7f3e140cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48138/0 (socket says 172.21.15.44:48138) 2024-04-02T13:24:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.284+0000 7f3e12ffd640 1 -- 172.21.15.44:0/1766648864 learned_addr learned my addr 172.21.15.44:0/1766648864 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e137fe640 1 --2- 172.21.15.44:0/1766648864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e140d0a30 0x7f3e1406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3692815788 0 0) 0x7f3e140d0000 con 0x7f3e140d0e10 2024-04-02T13:24:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dfc003520 con 0x7f3e140d0e10 2024-04-02T13:24:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3658124658 0 0) 0x7f3e1406c1d0 con 0x7f3e140d2040 2024-04-02T13:24:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e140d0000 con 0x7f3e140d2040 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3318251510 0 0) 0x7f3dfc003520 con 0x7f3e140d0e10 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e140d0a30 msgr2=0x7f3e1406cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 --2- 172.21.15.44:0/1766648864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e140d0a30 0x7f3e1406cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 >> v1:172.21.15.69:6789/0 conn(0x7f3e140d2040 legacy=0x7f3e140d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1406c4b0 con 0x7f3e140d0e10 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f3e08001670 con 0x7f3e140d0e10 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 >> v1:172.21.15.44:6789/0 conn(0x7f3e140d0e10 legacy=0x7f3e140cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 --2- 172.21.15.44:0/1766648864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dfc003b70 0x7f3dfc003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 --2- 172.21.15.44:0/1766648864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3dfc004800 0x7f3dfc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 --2- 172.21.15.44:0/1766648864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e140d0a30 0x7f3dfc0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1406cb00 con 0x7f3dfc004800 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc00bad0 con 0x7f3dfc003b70 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3dfc00bcd0 con 0x7f3e140d0a30 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e12ffd640 1 --2- 172.21.15.44:0/1766648864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3dfc004800 0x7f3dfc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e137fe640 1 --2- 172.21.15.44:0/1766648864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dfc003b70 0x7f3dfc003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e13fff640 1 --2- 172.21.15.44:0/1766648864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e140d0a30 0x7f3dfc0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e12ffd640 1 -- 172.21.15.44:0/1766648864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e140d0a30 msgr2=0x7f3dfc0051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e12ffd640 1 --2- 172.21.15.44:0/1766648864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e140d0a30 0x7f3dfc0051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e12ffd640 1 -- 172.21.15.44:0/1766648864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dfc003b70 msgr2=0x7f3dfc003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e12ffd640 1 --2- 172.21.15.44:0/1766648864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dfc003b70 0x7f3dfc003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e12ffd640 1 -- 172.21.15.44:0/1766648864 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dfc003520 con 0x7f3dfc004800 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.285+0000 7f3e13fff640 1 --2- 172.21.15.44:0/1766648864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e140d0a30 0x7f3dfc0051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.286+0000 7f3e12ffd640 1 --2- 172.21.15.44:0/1766648864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3dfc004800 0x7f3dfc004c80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800c190 tx=0x7f3dfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40a65c2bd16a72d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.286+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08007ce0 con 0x7f3dfc004800 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.286+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e08030070 con 0x7f3dfc004800 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.286+0000 7f3e127fc640 1 -- 172.21.15.44:0/1766648864 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08005d50 con 0x7f3dfc004800 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.287+0000 7f3e19ad3640 1 -- 172.21.15.44:0/1766648864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3dfc004800 msgr2=0x7f3dfc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.287+0000 7f3e19ad3640 1 --2- 172.21.15.44:0/1766648864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3dfc004800 0x7f3dfc004c80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800c190 tx=0x7f3dfc005830 comp rx=0 tx=0).stop 2024-04-02T13:24:41.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.287+0000 7f3e13fff640 1 -- 172.21.15.44:0/1766648864 reap_dead start 2024-04-02T13:24:41.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.287+0000 7f3e19ad3640 1 -- 172.21.15.44:0/1766648864 shutdown_connections 2024-04-02T13:24:41.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.287+0000 7f3e19ad3640 1 -- 172.21.15.44:0/1766648864 >> 172.21.15.44:0/1766648864 conn(0x7f3e1409d470 msgr2=0x7f3e140d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:41.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.287+0000 7f3e19ad3640 1 -- 172.21.15.44:0/1766648864 shutdown_connections 2024-04-02T13:24:41.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.287+0000 7f3e19ad3640 1 -- 172.21.15.44:0/1766648864 wait complete. 2024-04-02T13:24:41.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.288+0000 7f3e19ad3640 1 Processor -- start 2024-04-02T13:24:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.288+0000 7f3e19ad3640 1 -- start start 2024-04-02T13:24:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e19ad3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3dfc004800 0x7f3e140b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e19ad3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e140ccb80 0x7f3e1415fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e19ad3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e14156150 0x7f3e141565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e19ad3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1415e6f0 con 0x7f3e140ccb80 2024-04-02T13:24:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e19ad3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3e140d1270 con 0x7f3e14156150 2024-04-02T13:24:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e19ad3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3e1415e870 con 0x7f3dfc004800 2024-04-02T13:24:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e12ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e140ccb80 0x7f3e1415fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e12ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e140ccb80 0x7f3e1415fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47742/0 (socket says 172.21.15.44:47742) 2024-04-02T13:24:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e12ffd640 1 -- 172.21.15.44:0/3386579131 learned_addr learned my addr 172.21.15.44:0/3386579131 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e13fff640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e14156150 0x7f3e141565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e137fe640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3dfc004800 0x7f3e140b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e12ffd640 1 -- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3dfc004800 msgr2=0x7f3e140b50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e12ffd640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3dfc004800 0x7f3e140b50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e12ffd640 1 -- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e14156150 msgr2=0x7f3e141565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e12ffd640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e14156150 0x7f3e141565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e12ffd640 1 -- 172.21.15.44:0/3386579131 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e140cd3d0 con 0x7f3e140ccb80 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e13fff640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e14156150 0x7f3e141565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.289+0000 7f3e12ffd640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e140ccb80 0x7f3e1415fcb0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f3e080018f0 tx=0x7f3e08001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.290+0000 7f3dfbfff640 1 -- 172.21.15.44:0/3386579131 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08046070 con 0x7f3e140ccb80 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.290+0000 7f3dfbfff640 1 -- 172.21.15.44:0/3386579131 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e080049e0 con 0x7f3e140ccb80 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.290+0000 7f3e19ad3640 1 -- 172.21.15.44:0/3386579131 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e140cd6b0 con 0x7f3e140ccb80 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.290+0000 7f3dfbfff640 1 -- 172.21.15.44:0/3386579131 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08004dd0 con 0x7f3e140ccb80 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.290+0000 7f3e19ad3640 1 -- 172.21.15.44:0/3386579131 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e140cdc70 con 0x7f3e140ccb80 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.291+0000 7f3e19ad3640 1 -- 172.21.15.44:0/3386579131 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e140684f0 con 0x7f3e140ccb80 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.291+0000 7f3dfbfff640 1 -- 172.21.15.44:0/3386579131 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3e08007070 con 0x7f3e140ccb80 2024-04-02T13:24:41.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.291+0000 7f3dfbfff640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e0006c920 0x7f3e0006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.292+0000 7f3dfbfff640 1 -- 172.21.15.44:0/3386579131 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f3e080ace40 con 0x7f3e140ccb80 2024-04-02T13:24:41.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.292+0000 7f3e137fe640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e0006c920 0x7f3e0006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.292+0000 7f3e137fe640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e0006c920 0x7f3e0006ede0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3e04009a20 tx=0x7f3e04008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:41.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.296+0000 7f3dfbfff640 1 -- 172.21.15.44:0/3386579131 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3e0807aa90 con 0x7f3e140ccb80 2024-04-02T13:24:41.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.519+0000 7f3e19ad3640 1 -- 172.21.15.44:0/3386579131 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e140d17c0 con 0x7f3e140ccb80 2024-04-02T13:24:41.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.519+0000 7f3dfbfff640 1 -- 172.21.15.44:0/3386579131 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5180 (secure 0 0 0) 0x7f3e0807ed90 con 0x7f3e140ccb80 2024-04-02T13:24:41.521 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:41.521 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":37,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1535287484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1535287484},{"type":"v1","addr":"172.21.15.69:6836","nonce":1535287484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6429,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2367107002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2367107002},{"type":"v1","addr":"172.21.15.44:6833","nonce":2367107002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/553542317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":553542317},{"type":"v1","addr":"172.21.15.69:6837","nonce":553542317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:23:24.803884+0000","modified":"2024-04-02T13:24:05.482501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6450":{"gid":6450,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":13,"addr":"172.21.15.44:6835/3127613802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3127613802},{"type":"v1","addr":"172.21.15.44:6835","nonce":3127613802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6450,"qdb_cluster":[6450]},"id":4}]} 2024-04-02T13:24:41.522 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 37 2024-04-02T13:24:41.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.521+0000 7f3e19ad3640 1 -- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e0006c920 msgr2=0x7f3e0006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.521+0000 7f3e19ad3640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e0006c920 0x7f3e0006ede0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3e04009a20 tx=0x7f3e04008040 comp rx=0 tx=0).stop 2024-04-02T13:24:41.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.521+0000 7f3e19ad3640 1 -- 172.21.15.44:0/3386579131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e140ccb80 msgr2=0x7f3e1415fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.521+0000 7f3e19ad3640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e140ccb80 0x7f3e1415fcb0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f3e080018f0 tx=0x7f3e08001a30 comp rx=0 tx=0).stop 2024-04-02T13:24:41.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.521+0000 7f3e19ad3640 1 -- 172.21.15.44:0/3386579131 shutdown_connections 2024-04-02T13:24:41.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.521+0000 7f3e19ad3640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3e0006c920 0x7f3e0006ede0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.521+0000 7f3e19ad3640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e14156150 0x7f3e141565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.521+0000 7f3e19ad3640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e140ccb80 0x7f3e1415fcb0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.521+0000 7f3e19ad3640 1 --2- 172.21.15.44:0/3386579131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3dfc004800 0x7f3e140b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.521+0000 7f3e19ad3640 1 -- 172.21.15.44:0/3386579131 >> 172.21.15.44:0/3386579131 conn(0x7f3e1409d470 msgr2=0x7f3e140b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:41.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.521+0000 7f3e19ad3640 1 -- 172.21.15.44:0/3386579131 shutdown_connections 2024-04-02T13:24:41.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.521+0000 7f3e19ad3640 1 -- 172.21.15.44:0/3386579131 wait complete. 2024-04-02T13:24:41.534 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:24:41.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.634+0000 7f73e664a640 1 Processor -- start 2024-04-02T13:24:41.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.634+0000 7f73e664a640 1 -- start start 2024-04-02T13:24:41.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73e664a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f73e015c160 0x7f73e006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73e664a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f73e00d0890 0x7f73e00cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73e664a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73e006bab0 con 0x7f73e0160600 2024-04-02T13:24:41.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73e664a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f73e006bd90 con 0x7f73e015c160 2024-04-02T13:24:41.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73e664a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f73e00ccec0 con 0x7f73e00d0890 2024-04-02T13:24:41.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73dffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f73e015c160 0x7f73e006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73df7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f73e0160600 0x7f73e00d0180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48144/0 (socket says 172.21.15.44:48144) 2024-04-02T13:24:41.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73df7fe640 1 -- 172.21.15.44:0/3928625469 learned_addr learned my addr 172.21.15.44:0/3928625469 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:41.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73e4be7640 1 --2- 172.21.15.44:0/3928625469 >> v2:172.21.15.69:3300/0 conn(0x7f73e00d0890 0x7f73e00cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73dffff640 1 -- 172.21.15.44:0/3928625469 >> v2:172.21.15.69:3300/0 conn(0x7f73e00d0890 msgr2=0x7f73e00cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:41.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73dffff640 1 --2- 172.21.15.44:0/3928625469 >> v2:172.21.15.69:3300/0 conn(0x7f73e00d0890 0x7f73e00cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73dffff640 1 -- 172.21.15.44:0/3928625469 >> v1:172.21.15.44:6789/0 conn(0x7f73e0160600 legacy=0x7f73e00d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73deffd640 1 -- 172.21.15.44:0/3928625469 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4057542178 0 0) 0x7f73e006bab0 con 0x7f73e0160600 2024-04-02T13:24:41.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.635+0000 7f73dffff640 1 -- 172.21.15.44:0/3928625469 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e00cd1a0 con 0x7f73e015c160 2024-04-02T13:24:41.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.636+0000 7f73dffff640 1 --2- 172.21.15.44:0/3928625469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e015c160 0x7f73e006b4d0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f73e00cfa20 tx=0x7f73d002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599903a8b31bec5b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:41.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.636+0000 7f73deffd640 1 -- 172.21.15.44:0/3928625469 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d00315a0 con 0x7f73e015c160 2024-04-02T13:24:41.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.636+0000 7f73deffd640 1 -- 172.21.15.44:0/3928625469 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d0037070 con 0x7f73e015c160 2024-04-02T13:24:41.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.636+0000 7f73deffd640 1 -- 172.21.15.44:0/3928625469 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d0031960 con 0x7f73e015c160 2024-04-02T13:24:41.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.636+0000 7f73e664a640 1 -- 172.21.15.44:0/3928625469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e015c160 msgr2=0x7f73e006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.636+0000 7f73e664a640 1 --2- 172.21.15.44:0/3928625469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e015c160 0x7f73e006b4d0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f73e00cfa20 tx=0x7f73d002b840 comp rx=0 tx=0).stop 2024-04-02T13:24:41.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.637+0000 7f73e664a640 1 -- 172.21.15.44:0/3928625469 shutdown_connections 2024-04-02T13:24:41.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.637+0000 7f73e664a640 1 --2- 172.21.15.44:0/3928625469 >> v2:172.21.15.69:3300/0 conn(0x7f73e00d0890 0x7f73e00cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.637+0000 7f73e664a640 1 --2- 172.21.15.44:0/3928625469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e015c160 0x7f73e006b4d0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.637+0000 7f73e664a640 1 -- 172.21.15.44:0/3928625469 >> 172.21.15.44:0/3928625469 conn(0x7f73e00b2980 msgr2=0x7f73e00b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:41.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.637+0000 7f73e664a640 1 -- 172.21.15.44:0/3928625469 shutdown_connections 2024-04-02T13:24:41.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.637+0000 7f73e664a640 1 -- 172.21.15.44:0/3928625469 wait complete. 2024-04-02T13:24:41.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.638+0000 7f73e664a640 1 Processor -- start 2024-04-02T13:24:41.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.638+0000 7f73e664a640 1 -- start start 2024-04-02T13:24:41.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73e664a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e00d0890 0x7f73e00c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73e664a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e015c160 0x7f73e00caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73e664a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e0160600 0x7f73e01f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73e664a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f73e00add60 con 0x7f73e015c160 2024-04-02T13:24:41.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73e664a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f73e0160a60 con 0x7f73e0160600 2024-04-02T13:24:41.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73e664a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f73e00adee0 con 0x7f73e00d0890 2024-04-02T13:24:41.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e015c160 0x7f73e00caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e015c160 0x7f73e00caeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47766/0 (socket says 172.21.15.44:47766) 2024-04-02T13:24:41.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73df7fe640 1 -- 172.21.15.44:0/2716432029 learned_addr learned my addr 172.21.15.44:0/2716432029 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:41.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73dffff640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e00d0890 0x7f73e00c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73e4be7640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e0160600 0x7f73e01f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73df7fe640 1 -- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e00d0890 msgr2=0x7f73e00c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73df7fe640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e00d0890 0x7f73e00c7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73df7fe640 1 -- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e0160600 msgr2=0x7f73e01f58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73df7fe640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e0160600 0x7f73e01f58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73df7fe640 1 -- 172.21.15.44:0/2716432029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e015b450 con 0x7f73e015c160 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73dffff640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e00d0890 0x7f73e00c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73e4be7640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e0160600 0x7f73e01f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73df7fe640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e015c160 0x7f73e00caeb0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f73d400c430 tx=0x7f73d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73dcff9640 1 -- 172.21.15.44:0/2716432029 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d40074e0 con 0x7f73e015c160 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73dcff9640 1 -- 172.21.15.44:0/2716432029 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d40047e0 con 0x7f73e015c160 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73e664a640 1 -- 172.21.15.44:0/2716432029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73e00c83c0 con 0x7f73e015c160 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.639+0000 7f73dcff9640 1 -- 172.21.15.44:0/2716432029 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4004bd0 con 0x7f73e015c160 2024-04-02T13:24:41.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.640+0000 7f73e664a640 1 -- 172.21.15.44:0/2716432029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73e00ca0f0 con 0x7f73e015c160 2024-04-02T13:24:41.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.640+0000 7f73e664a640 1 -- 172.21.15.44:0/2716432029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73e0067b30 con 0x7f73e015c160 2024-04-02T13:24:41.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.641+0000 7f73dcff9640 1 -- 172.21.15.44:0/2716432029 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f73d4011070 con 0x7f73e015c160 2024-04-02T13:24:41.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.641+0000 7f73dcff9640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73cc06c8f0 0x7f73cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.641+0000 7f73dcff9640 1 -- 172.21.15.44:0/2716432029 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f73d408b910 con 0x7f73e015c160 2024-04-02T13:24:41.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.641+0000 7f73dffff640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73cc06c8f0 0x7f73cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.642+0000 7f73dffff640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73cc06c8f0 0x7f73cc06edb0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f73d0036000 tx=0x7f73d0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:41.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.646+0000 7f73dcff9640 1 -- 172.21.15.44:0/2716432029 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f73e0067b30 con 0x7f73e015c160 2024-04-02T13:24:41.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.831+0000 7f73e664a640 1 -- 172.21.15.44:0/2716432029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73e00684f0 con 0x7f73e015c160 2024-04-02T13:24:41.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.832+0000 7f73dcff9640 1 -- 172.21.15.44:0/2716432029 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+17076 (secure 0 0 0) 0x7f73d400e280 con 0x7f73e015c160 2024-04-02T13:24:41.834 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:41.835 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":67,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:24:09.591968+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:23:23.126712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-02T13:23:23.763593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.18","mappings":[{"from":7,"to":2}]},{"pgid":"10.4","mappings":[{"from":4,"to":3}]},{"pgid":"10.10","mappings":[{"from":4,"to":3}]},{"pgid":"10.37","mappings":[{"from":4,"to":5}]},{"pgid":"10.39","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:24:41.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.834+0000 7f73e664a640 1 -- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73cc06c8f0 msgr2=0x7f73cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.834+0000 7f73e664a640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73cc06c8f0 0x7f73cc06edb0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f73d0036000 tx=0x7f73d0008030 comp rx=0 tx=0).stop 2024-04-02T13:24:41.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.834+0000 7f73e664a640 1 -- 172.21.15.44:0/2716432029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e015c160 msgr2=0x7f73e00caeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.834+0000 7f73e664a640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e015c160 0x7f73e00caeb0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f73d400c430 tx=0x7f73d400c8f0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.834+0000 7f73e664a640 1 -- 172.21.15.44:0/2716432029 shutdown_connections 2024-04-02T13:24:41.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.834+0000 7f73e664a640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73cc06c8f0 0x7f73cc06edb0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.834+0000 7f73e664a640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e0160600 0x7f73e01f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.834+0000 7f73e664a640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e015c160 0x7f73e00caeb0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.834+0000 7f73e664a640 1 --2- 172.21.15.44:0/2716432029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e00d0890 0x7f73e00c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.834+0000 7f73e664a640 1 -- 172.21.15.44:0/2716432029 >> 172.21.15.44:0/2716432029 conn(0x7f73e00b2980 msgr2=0x7f73e00c6890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:41.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.834+0000 7f73e664a640 1 -- 172.21.15.44:0/2716432029 shutdown_connections 2024-04-02T13:24:41.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.834+0000 7f73e664a640 1 -- 172.21.15.44:0/2716432029 wait complete. 2024-04-02T13:24:41.847 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:24:41.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.947+0000 7f8db481a640 1 Processor -- start 2024-04-02T13:24:41.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.947+0000 7f8db481a640 1 -- start start 2024-04-02T13:24:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.948+0000 7f8db481a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8db00d0c50 0x7f8db00d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.948+0000 7f8db481a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8db006ce40 0x7f8db006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.948+0000 7f8db481a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8db00cc600 con 0x7f8db015e4a0 2024-04-02T13:24:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.948+0000 7f8db481a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8db00af300 con 0x7f8db006ce40 2024-04-02T13:24:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.948+0000 7f8db481a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8db00cc8e0 con 0x7f8db00d0c50 2024-04-02T13:24:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.948+0000 7f8dae59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f8db015e4a0 0x7f8db00aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48148/0 (socket says 172.21.15.44:48148) 2024-04-02T13:24:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.948+0000 7f8dae59c640 1 -- 172.21.15.44:0/3996431781 learned_addr learned my addr 172.21.15.44:0/3996431781 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.948+0000 7f8daed9d640 1 --2- 172.21.15.44:0/3996431781 >> v2:172.21.15.69:3300/0 conn(0x7f8db006ce40 0x7f8db006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.948+0000 7f8dadd9b640 1 --2- 172.21.15.44:0/3996431781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8db00d0c50 0x7f8db00d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.948+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2023462893 0 0) 0x7f8db00cc600 con 0x7f8db015e4a0 2024-04-02T13:24:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d9c003520 con 0x7f8db015e4a0 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2997539692 0 0) 0x7f8d9c003520 con 0x7f8db015e4a0 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8db00d0c50 msgr2=0x7f8db00d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 --2- 172.21.15.44:0/3996431781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8db00d0c50 0x7f8db00d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 >> v2:172.21.15.69:3300/0 conn(0x7f8db006ce40 msgr2=0x7f8db006d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 --2- 172.21.15.44:0/3996431781 >> v2:172.21.15.69:3300/0 conn(0x7f8db006ce40 0x7f8db006d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db00ccbc0 con 0x7f8db015e4a0 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f8da8001670 con 0x7f8db015e4a0 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 >> v1:172.21.15.44:6789/0 conn(0x7f8db015e4a0 legacy=0x7f8db00aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 --2- 172.21.15.44:0/3996431781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d9c0039b0 0x7f8d9c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 --2- 172.21.15.44:0/3996431781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d9c004800 0x7f8d9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 --2- 172.21.15.44:0/3996431781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8db00d0c50 0x7f8d9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8db00af300 con 0x7f8d9c004800 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8db00cc8e0 con 0x7f8d9c0039b0 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8d9c00b9c0 con 0x7f8db00d0c50 2024-04-02T13:24:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dadd9b640 1 --2- 172.21.15.44:0/3996431781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d9c004800 0x7f8d9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dae59c640 1 --2- 172.21.15.44:0/3996431781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d9c0039b0 0x7f8d9c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8daed9d640 1 --2- 172.21.15.44:0/3996431781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8db00d0c50 0x7f8d9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dadd9b640 1 -- 172.21.15.44:0/3996431781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8db00d0c50 msgr2=0x7f8d9c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:41.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dadd9b640 1 --2- 172.21.15.44:0/3996431781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8db00d0c50 0x7f8d9c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dadd9b640 1 -- 172.21.15.44:0/3996431781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d9c0039b0 msgr2=0x7f8d9c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:41.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dadd9b640 1 --2- 172.21.15.44:0/3996431781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d9c0039b0 0x7f8d9c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.949+0000 7f8dadd9b640 1 -- 172.21.15.44:0/3996431781 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9c003520 con 0x7f8d9c004800 2024-04-02T13:24:41.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.950+0000 7f8dadd9b640 1 --2- 172.21.15.44:0/3996431781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d9c004800 0x7f8d9c004c80 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009a90 tx=0x7f8d9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8793d475d6a1795 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:41.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.950+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d98039070 con 0x7f8d9c004800 2024-04-02T13:24:41.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.950+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d9802c680 con 0x7f8d9c004800 2024-04-02T13:24:41.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.950+0000 7f8dad59a640 1 -- 172.21.15.44:0/3996431781 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d98009690 con 0x7f8d9c004800 2024-04-02T13:24:41.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.951+0000 7f8db481a640 1 -- 172.21.15.44:0/3996431781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d9c004800 msgr2=0x7f8d9c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.951+0000 7f8db481a640 1 --2- 172.21.15.44:0/3996431781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d9c004800 0x7f8d9c004c80 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009a90 tx=0x7f8d9c005830 comp rx=0 tx=0).stop 2024-04-02T13:24:41.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.951+0000 7f8daed9d640 1 -- 172.21.15.44:0/3996431781 reap_dead start 2024-04-02T13:24:41.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.951+0000 7f8db481a640 1 -- 172.21.15.44:0/3996431781 shutdown_connections 2024-04-02T13:24:41.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.951+0000 7f8db481a640 1 -- 172.21.15.44:0/3996431781 >> 172.21.15.44:0/3996431781 conn(0x7f8db009aa20 msgr2=0x7f8db009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:41.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.951+0000 7f8db481a640 1 -- 172.21.15.44:0/3996431781 shutdown_connections 2024-04-02T13:24:41.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.951+0000 7f8db481a640 1 -- 172.21.15.44:0/3996431781 wait complete. 2024-04-02T13:24:41.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.952+0000 7f8db481a640 1 Processor -- start 2024-04-02T13:24:41.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.952+0000 7f8db481a640 1 -- start start 2024-04-02T13:24:41.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8db481a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d9c004800 0x7f8db01f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8db481a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8db015f670 0x7f8db00b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8db481a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8db00c8280 0x7f8db00c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8db481a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8db00cd460 con 0x7f8db00c8280 2024-04-02T13:24:41.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8db481a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8db00cd180 con 0x7f8db015f670 2024-04-02T13:24:41.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8db481a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8db015e880 con 0x7f8d9c004800 2024-04-02T13:24:41.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8daed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8db00c8280 0x7f8db00c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8daed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8db00c8280 0x7f8db00c8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47796/0 (socket says 172.21.15.44:47796) 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8daed9d640 1 -- 172.21.15.44:0/479718681 learned_addr learned my addr 172.21.15.44:0/479718681 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8dae59c640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d9c004800 0x7f8db01f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8dadd9b640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8db015f670 0x7f8db00b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8daed9d640 1 -- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d9c004800 msgr2=0x7f8db01f74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8daed9d640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d9c004800 0x7f8db01f74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8daed9d640 1 -- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8db015f670 msgr2=0x7f8db00b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8daed9d640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8db015f670 0x7f8db00b3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.953+0000 7f8daed9d640 1 -- 172.21.15.44:0/479718681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db01f7dc0 con 0x7f8db00c8280 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.954+0000 7f8daed9d640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8db00c8280 0x7f8db00c8700 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f8db00b1b20 tx=0x7f8da00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.954+0000 7f8da6ffd640 1 -- 172.21.15.44:0/479718681 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0003990 con 0x7f8db00c8280 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.954+0000 7f8da6ffd640 1 -- 172.21.15.44:0/479718681 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8da001b070 con 0x7f8db00c8280 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.954+0000 7f8db481a640 1 -- 172.21.15.44:0/479718681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db01f80a0 con 0x7f8db00c8280 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.954+0000 7f8da6ffd640 1 -- 172.21.15.44:0/479718681 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0003b70 con 0x7f8db00c8280 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.954+0000 7f8db481a640 1 -- 172.21.15.44:0/479718681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db015fde0 con 0x7f8db00c8280 2024-04-02T13:24:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.955+0000 7f8db481a640 1 -- 172.21.15.44:0/479718681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8db00684f0 con 0x7f8db00c8280 2024-04-02T13:24:41.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.955+0000 7f8da6ffd640 1 -- 172.21.15.44:0/479718681 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8da0005d10 con 0x7f8db00c8280 2024-04-02T13:24:41.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.956+0000 7f8da6ffd640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8d9006c8f0 0x7f8d9006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:41.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.956+0000 7f8da6ffd640 1 -- 172.21.15.44:0/479718681 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6246+0+0 (secure 0 0 0) 0x7f8da008c360 con 0x7f8db00c8280 2024-04-02T13:24:41.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.956+0000 7f8dae59c640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8d9006c8f0 0x7f8d9006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:41.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.956+0000 7f8dae59c640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8d9006c8f0 0x7f8d9006edb0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8da80015d0 tx=0x7f8da80078f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:41.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:41.960+0000 7f8da6ffd640 1 -- 172.21.15.44:0/479718681 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8da0025020 con 0x7f8db00c8280 2024-04-02T13:24:42.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.185+0000 7f8db481a640 1 -- 172.21.15.44:0/479718681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f8db00d1250 con 0x7f8db00c8280 2024-04-02T13:24:42.188 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:24:42.186+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:24:42.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.187+0000 7f8da6ffd640 1 -- 172.21.15.44:0/479718681 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v38) ==== 161+0+0 (secure 0 0 0) 0x7f8da005a4d0 con 0x7f8db00c8280 2024-04-02T13:24:42.189 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:24:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.189+0000 7f8db481a640 1 -- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8d9006c8f0 msgr2=0x7f8d9006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.189+0000 7f8db481a640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8d9006c8f0 0x7f8d9006edb0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8da80015d0 tx=0x7f8da80078f0 comp rx=0 tx=0).stop 2024-04-02T13:24:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.189+0000 7f8db481a640 1 -- 172.21.15.44:0/479718681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8db00c8280 msgr2=0x7f8db00c8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.189+0000 7f8db481a640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8db00c8280 0x7f8db00c8700 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f8db00b1b20 tx=0x7f8da00115a0 comp rx=0 tx=0).stop 2024-04-02T13:24:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.189+0000 7f8db481a640 1 -- 172.21.15.44:0/479718681 shutdown_connections 2024-04-02T13:24:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.189+0000 7f8db481a640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8d9006c8f0 0x7f8d9006edb0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.189+0000 7f8db481a640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8db00c8280 0x7f8db00c8700 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.189+0000 7f8db481a640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8db015f670 0x7f8db00b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.189+0000 7f8db481a640 1 --2- 172.21.15.44:0/479718681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d9c004800 0x7f8db01f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.190+0000 7f8db481a640 1 -- 172.21.15.44:0/479718681 >> 172.21.15.44:0/479718681 conn(0x7f8db009aa20 msgr2=0x7f8db009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.190+0000 7f8db481a640 1 -- 172.21.15.44:0/479718681 shutdown_connections 2024-04-02T13:24:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.190+0000 7f8db481a640 1 -- 172.21.15.44:0/479718681 wait complete. 2024-04-02T13:24:42.195 INFO:tasks.ceph.mds.a.smithi044.stderr: -6461> 2024-04-02T13:23:30.427+0000 7fb85d32a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:24:42.196 INFO:tasks.ceph.mds.a.smithi044.stderr: -4818> 2024-04-02T13:23:33.871+0000 7fb859322640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={6603=0-4194304@1} caps={6603=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6603 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x55dcba8b1600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-02T13:24:42.198 INFO:tasks.ceph.mds.a.smithi044.stderr: -2890> 2024-04-02T13:24:05.201+0000 7fb859322640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v4 ap=1 DIRTYPARENT s=0 n(v0 rc2024-04-02T13:23:32.807273+0000 1=1+0) caps={6603=pAsLsXsFs/-@2},l=6603(-1) | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 scrubqueue=0 0x55dcba8b1600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v4>,<0x1/dir_test v7>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-02T13:24:42.203 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T13:24:42.224 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:24:42.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.305+0000 7fbff7655640 1 Processor -- start 2024-04-02T13:24:42.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.305+0000 7fbff7655640 1 -- start start 2024-04-02T13:24:42.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbff7655640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbff015a940 0x7fbff00ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:42.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbff7655640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbff00cc640 0x7fbff00cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:42.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbff7655640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff00ccf90 con 0x7fbff0071200 2024-04-02T13:24:42.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbff7655640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fbff006ef80 con 0x7fbff015a940 2024-04-02T13:24:42.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbff7655640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbff00ce020 con 0x7fbff00cc640 2024-04-02T13:24:42.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbff53f1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbff015a940 0x7fbff00ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:42.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbff4bf0640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbff0071200 0x7fbff006e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48164/0 (socket says 172.21.15.44:48164) 2024-04-02T13:24:42.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbff4bf0640 1 -- 172.21.15.44:0/2485025226 learned_addr learned my addr 172.21.15.44:0/2485025226 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:42.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbff5bf2640 1 --2- 172.21.15.44:0/2485025226 >> v2:172.21.15.69:3300/0 conn(0x7fbff00cc640 0x7fbff00cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:42.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbff53f1640 1 -- 172.21.15.44:0/2485025226 >> v2:172.21.15.69:3300/0 conn(0x7fbff00cc640 msgr2=0x7fbff00cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:42.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbff53f1640 1 --2- 172.21.15.44:0/2485025226 >> v2:172.21.15.69:3300/0 conn(0x7fbff00cc640 0x7fbff00cca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:42.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbff53f1640 1 -- 172.21.15.44:0/2485025226 >> v1:172.21.15.44:6789/0 conn(0x7fbff0071200 legacy=0x7fbff006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:42.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbfe7fff640 1 -- 172.21.15.44:0/2485025226 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2510540809 0 0) 0x7fbff00ccf90 con 0x7fbff0071200 2024-04-02T13:24:42.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.306+0000 7fbff53f1640 1 -- 172.21.15.44:0/2485025226 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff00ce9f0 con 0x7fbff015a940 2024-04-02T13:24:42.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.307+0000 7fbff53f1640 1 --2- 172.21.15.44:0/2485025226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff015a940 0x7fbff00ce4b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fbff00735b0 tx=0x7fbfec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464f1e594f679f71 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:42.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.307+0000 7fbfe7fff640 1 -- 172.21.15.44:0/2485025226 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec0315a0 con 0x7fbff015a940 2024-04-02T13:24:42.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.307+0000 7fbfe7fff640 1 -- 172.21.15.44:0/2485025226 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfec037070 con 0x7fbff015a940 2024-04-02T13:24:42.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.307+0000 7fbfe7fff640 1 -- 172.21.15.44:0/2485025226 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec031960 con 0x7fbff015a940 2024-04-02T13:24:42.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.308+0000 7fbff7655640 1 -- 172.21.15.44:0/2485025226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff015a940 msgr2=0x7fbff00ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:42.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.308+0000 7fbff7655640 1 --2- 172.21.15.44:0/2485025226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff015a940 0x7fbff00ce4b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fbff00735b0 tx=0x7fbfec02b840 comp rx=0 tx=0).stop 2024-04-02T13:24:42.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.308+0000 7fbff7655640 1 -- 172.21.15.44:0/2485025226 shutdown_connections 2024-04-02T13:24:42.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.308+0000 7fbff7655640 1 --2- 172.21.15.44:0/2485025226 >> v2:172.21.15.69:3300/0 conn(0x7fbff00cc640 0x7fbff00cca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:42.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.308+0000 7fbff7655640 1 --2- 172.21.15.44:0/2485025226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff015a940 0x7fbff00ce4b0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:42.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.308+0000 7fbff7655640 1 -- 172.21.15.44:0/2485025226 >> 172.21.15.44:0/2485025226 conn(0x7fbff006b920 msgr2=0x7fbff006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:42.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.308+0000 7fbff7655640 1 -- 172.21.15.44:0/2485025226 shutdown_connections 2024-04-02T13:24:42.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.308+0000 7fbff7655640 1 -- 172.21.15.44:0/2485025226 wait complete. 2024-04-02T13:24:42.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.309+0000 7fbff7655640 1 Processor -- start 2024-04-02T13:24:42.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.309+0000 7fbff7655640 1 -- start start 2024-04-02T13:24:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff7655640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbff0071200 0x7fbff009ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:42.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff7655640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbff00cc640 0x7fbff014d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:42.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff7655640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff015a940 0x7fbff015d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:42.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff7655640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbff00cdcb0 con 0x7fbff015a940 2024-04-02T13:24:42.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff7655640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbff006eca0 con 0x7fbff0071200 2024-04-02T13:24:42.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff7655640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbff00d00a0 con 0x7fbff00cc640 2024-04-02T13:24:42.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff5bf2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff015a940 0x7fbff015d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:42.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff5bf2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff015a940 0x7fbff015d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47840/0 (socket says 172.21.15.44:47840) 2024-04-02T13:24:42.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff5bf2640 1 -- 172.21.15.44:0/2096031557 learned_addr learned my addr 172.21.15.44:0/2096031557 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:42.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff53f1640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbff0071200 0x7fbff009ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff4bf0640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbff00cc640 0x7fbff014d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff5bf2640 1 -- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbff00cc640 msgr2=0x7fbff014d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff5bf2640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbff00cc640 0x7fbff014d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff5bf2640 1 -- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbff0071200 msgr2=0x7fbff009ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff5bf2640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbff0071200 0x7fbff009ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff5bf2640 1 -- 172.21.15.44:0/2096031557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff014daf0 con 0x7fbff015a940 2024-04-02T13:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff53f1640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbff0071200 0x7fbff009ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff5bf2640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff015a940 0x7fbff015d250 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8007e40 tx=0x7fbfe800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbfe5ffb640 1 -- 172.21.15.44:0/2096031557 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8014620 con 0x7fbff015a940 2024-04-02T13:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbfe5ffb640 1 -- 172.21.15.44:0/2096031557 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe8019070 con 0x7fbff015a940 2024-04-02T13:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbff7655640 1 -- 172.21.15.44:0/2096031557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff014ddd0 con 0x7fbff015a940 2024-04-02T13:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.310+0000 7fbfe5ffb640 1 -- 172.21.15.44:0/2096031557 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8014a20 con 0x7fbff015a940 2024-04-02T13:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.311+0000 7fbff7655640 1 -- 172.21.15.44:0/2096031557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff014e390 con 0x7fbff015a940 2024-04-02T13:24:42.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.312+0000 7fbff7655640 1 -- 172.21.15.44:0/2096031557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff015b720 con 0x7fbff015a940 2024-04-02T13:24:42.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.312+0000 7fbfe5ffb640 1 -- 172.21.15.44:0/2096031557 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbfe8014bc0 con 0x7fbff015a940 2024-04-02T13:24:42.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.312+0000 7fbfe5ffb640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbfd406c8f0 0x7fbfd406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:42.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.313+0000 7fbfe5ffb640 1 -- 172.21.15.44:0/2096031557 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6332+0+0 (secure 0 0 0) 0x7fbfe808c7c0 con 0x7fbff015a940 2024-04-02T13:24:42.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.313+0000 7fbff53f1640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbfd406c8f0 0x7fbfd406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:42.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.313+0000 7fbff53f1640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbfd406c8f0 0x7fbfd406edb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbfec036000 tx=0x7fbfec008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:42.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.317+0000 7fbfe5ffb640 1 -- 172.21.15.44:0/2096031557 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbfe805a8e0 con 0x7fbff015a940 2024-04-02T13:24:42.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:42.546+0000 7fbff7655640 1 -- 172.21.15.44:0/2096031557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fbff00afa10 con 0x7fbff015a940 2024-04-02T13:24:43.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.189+0000 7fbfe5ffb640 1 -- 172.21.15.44:0/2096031557 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) ==== 104+0+0 (secure 0 0 0) 0x7fbfe805a1e0 con 0x7fbff015a940 2024-04-02T13:24:43.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.192+0000 7fbff7655640 1 -- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbfd406c8f0 msgr2=0x7fbfd406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:43.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.192+0000 7fbff7655640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbfd406c8f0 0x7fbfd406edb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbfec036000 tx=0x7fbfec008030 comp rx=0 tx=0).stop 2024-04-02T13:24:43.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.192+0000 7fbff7655640 1 -- 172.21.15.44:0/2096031557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff015a940 msgr2=0x7fbff015d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:43.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.192+0000 7fbff7655640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff015a940 0x7fbff015d250 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8007e40 tx=0x7fbfe800e4b0 comp rx=0 tx=0).stop 2024-04-02T13:24:43.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.192+0000 7fbff7655640 1 -- 172.21.15.44:0/2096031557 shutdown_connections 2024-04-02T13:24:43.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.192+0000 7fbff7655640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbfd406c8f0 0x7fbfd406edb0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:43.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.192+0000 7fbff7655640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbff015a940 0x7fbff015d250 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:43.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.192+0000 7fbff7655640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbff00cc640 0x7fbff014d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:43.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.192+0000 7fbff7655640 1 --2- 172.21.15.44:0/2096031557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbff0071200 0x7fbff009ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:43.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.192+0000 7fbff7655640 1 -- 172.21.15.44:0/2096031557 >> 172.21.15.44:0/2096031557 conn(0x7fbff006b920 msgr2=0x7fbff00cca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:43.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.192+0000 7fbff7655640 1 -- 172.21.15.44:0/2096031557 shutdown_connections 2024-04-02T13:24:43.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.192+0000 7fbff7655640 1 -- 172.21.15.44:0/2096031557 wait complete. 2024-04-02T13:24:43.208 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T13:24:43.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.360+0000 7eff41f9d640 1 Processor -- start 2024-04-02T13:24:43.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.360+0000 7eff41f9d640 1 -- start start 2024-04-02T13:24:43.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.361+0000 7eff41f9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7eff3c15e000 0x7eff3c0d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:43.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.361+0000 7eff41f9d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3c0b4720 con 0x7eff3c15e3e0 2024-04-02T13:24:43.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.361+0000 7eff41f9d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3c0b4a00 con 0x7eff3c162470 2024-04-02T13:24:43.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.361+0000 7eff41f9d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7eff3c0b57c0 con 0x7eff3c15e000 2024-04-02T13:24:43.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.361+0000 7eff40f9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7eff3c15e000 0x7eff3c0d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:43.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.361+0000 7eff40f9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7eff3c15e000 0x7eff3c0d47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47850/0 (socket says 172.21.15.44:47850) 2024-04-02T13:24:43.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.361+0000 7eff3bfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7eff3c15e3e0 0x7eff3c161d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48174/0 (socket says 172.21.15.44:48174) 2024-04-02T13:24:43.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.361+0000 7eff40f9b640 1 -- 172.21.15.44:0/727070975 learned_addr learned my addr 172.21.15.44:0/727070975 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:43.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.361+0000 7eff40f9b640 1 -- 172.21.15.44:0/727070975 >> v1:172.21.15.69:6789/0 conn(0x7eff3c162470 legacy=0x7eff3c0cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:43.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.361+0000 7eff40f9b640 1 -- 172.21.15.44:0/727070975 >> v1:172.21.15.44:6789/0 conn(0x7eff3c15e3e0 legacy=0x7eff3c161d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:43.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.362+0000 7eff3b7fe640 1 -- 172.21.15.44:0/727070975 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 96858093 0 0) 0x7eff3c0b4720 con 0x7eff3c15e3e0 2024-04-02T13:24:43.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.362+0000 7eff40f9b640 1 -- 172.21.15.44:0/727070975 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3c0b4ce0 con 0x7eff3c15e000 2024-04-02T13:24:43.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.362+0000 7eff40f9b640 1 --2- 172.21.15.44:0/727070975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c15e000 0x7eff3c0d47a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7eff34009e50 tx=0x7eff3402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c81d5834ded521 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:43.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.362+0000 7eff3b7fe640 1 -- 172.21.15.44:0/727070975 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3402cbb0 con 0x7eff3c15e000 2024-04-02T13:24:43.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.362+0000 7eff3b7fe640 1 -- 172.21.15.44:0/727070975 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff3402cd50 con 0x7eff3c15e000 2024-04-02T13:24:43.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.362+0000 7eff3b7fe640 1 -- 172.21.15.44:0/727070975 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff34034610 con 0x7eff3c15e000 2024-04-02T13:24:43.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.363+0000 7eff41f9d640 1 -- 172.21.15.44:0/727070975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c15e000 msgr2=0x7eff3c0d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:43.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.363+0000 7eff41f9d640 1 --2- 172.21.15.44:0/727070975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c15e000 0x7eff3c0d47a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7eff34009e50 tx=0x7eff3402c010 comp rx=0 tx=0).stop 2024-04-02T13:24:43.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.363+0000 7eff41f9d640 1 -- 172.21.15.44:0/727070975 shutdown_connections 2024-04-02T13:24:43.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.363+0000 7eff41f9d640 1 --2- 172.21.15.44:0/727070975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c15e000 0x7eff3c0d47a0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:43.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.363+0000 7eff41f9d640 1 -- 172.21.15.44:0/727070975 >> 172.21.15.44:0/727070975 conn(0x7eff3c09d1f0 msgr2=0x7eff3c0d3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:43.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.363+0000 7eff41f9d640 1 -- 172.21.15.44:0/727070975 shutdown_connections 2024-04-02T13:24:43.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.363+0000 7eff41f9d640 1 -- 172.21.15.44:0/727070975 wait complete. 2024-04-02T13:24:43.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.364+0000 7eff41f9d640 1 Processor -- start 2024-04-02T13:24:43.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.364+0000 7eff41f9d640 1 -- start start 2024-04-02T13:24:43.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.365+0000 7eff41f9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff3c15e000 0x7eff3c15afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.365+0000 7eff41f9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c15e3e0 0x7eff3c15b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.365+0000 7eff41f9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3c162470 0x7eff3c0cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.365+0000 7eff41f9d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c0d4ce0 con 0x7eff3c15e3e0 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.365+0000 7eff41f9d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c0b52a0 con 0x7eff3c15e000 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.365+0000 7eff41f9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7eff3c0b41b0 con 0x7eff3c162470 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.365+0000 7eff3bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c15e3e0 0x7eff3c15b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.365+0000 7eff3bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c15e3e0 0x7eff3c15b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47866/0 (socket says 172.21.15.44:47866) 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.365+0000 7eff3bfff640 1 -- 172.21.15.44:0/3891930378 learned_addr learned my addr 172.21.15.44:0/3891930378 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff40f9b640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff3c15e000 0x7eff3c15afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff4179c640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3c162470 0x7eff3c0cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff3bfff640 1 -- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3c162470 msgr2=0x7eff3c0cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff3bfff640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3c162470 0x7eff3c0cfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff3bfff640 1 -- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff3c15e000 msgr2=0x7eff3c15afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:43.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff3bfff640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff3c15e000 0x7eff3c15afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:43.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff3bfff640 1 -- 172.21.15.44:0/3891930378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3c15ea40 con 0x7eff3c15e3e0 2024-04-02T13:24:43.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff4179c640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3c162470 0x7eff3c0cfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:43.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff40f9b640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff3c15e000 0x7eff3c15afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:43.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff3bfff640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c15e3e0 0x7eff3c15b510 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7eff3000c460 tx=0x7eff3000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:43.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff397fa640 1 -- 172.21.15.44:0/3891930378 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30018070 con 0x7eff3c15e3e0 2024-04-02T13:24:43.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff397fa640 1 -- 172.21.15.44:0/3891930378 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff30007520 con 0x7eff3c15e3e0 2024-04-02T13:24:43.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff41f9d640 1 -- 172.21.15.44:0/3891930378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff3c0caff0 con 0x7eff3c15e3e0 2024-04-02T13:24:43.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff397fa640 1 -- 172.21.15.44:0/3891930378 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30004a20 con 0x7eff3c15e3e0 2024-04-02T13:24:43.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.366+0000 7eff41f9d640 1 -- 172.21.15.44:0/3891930378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff3c0cb290 con 0x7eff3c15e3e0 2024-04-02T13:24:43.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.367+0000 7eff41f9d640 1 -- 172.21.15.44:0/3891930378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff3c04f860 con 0x7eff3c15e3e0 2024-04-02T13:24:43.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.368+0000 7eff397fa640 1 -- 172.21.15.44:0/3891930378 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7eff3001b070 con 0x7eff3c15e3e0 2024-04-02T13:24:43.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.368+0000 7eff397fa640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff2806c8f0 0x7eff2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:43.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.368+0000 7eff397fa640 1 -- 172.21.15.44:0/3891930378 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6332+0+0 (secure 0 0 0) 0x7eff3008b150 con 0x7eff3c15e3e0 2024-04-02T13:24:43.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.368+0000 7eff40f9b640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff2806c8f0 0x7eff2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:43.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.369+0000 7eff40f9b640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff2806c8f0 0x7eff2806edb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7eff34035000 tx=0x7eff34030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:43.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.373+0000 7eff397fa640 1 -- 172.21.15.44:0/3891930378 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7eff3c15ea40 con 0x7eff3c15e3e0 2024-04-02T13:24:43.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:43.558+0000 7eff41f9d640 1 -- 172.21.15.44:0/3891930378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7eff3c0d2bf0 con 0x7eff3c15e3e0 2024-04-02T13:24:44.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.190+0000 7eff397fa640 1 -- 172.21.15.44:0/3891930378 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v69) ==== 181+0+0 (secure 0 0 0) 0x7eff300593a0 con 0x7eff3c15e3e0 2024-04-02T13:24:44.193 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' removed 2024-04-02T13:24:44.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.194+0000 7eff41f9d640 1 -- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff2806c8f0 msgr2=0x7eff2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:44.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.194+0000 7eff41f9d640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff2806c8f0 0x7eff2806edb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7eff34035000 tx=0x7eff34030000 comp rx=0 tx=0).stop 2024-04-02T13:24:44.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.194+0000 7eff41f9d640 1 -- 172.21.15.44:0/3891930378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c15e3e0 msgr2=0x7eff3c15b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:44.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.194+0000 7eff41f9d640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c15e3e0 0x7eff3c15b510 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7eff3000c460 tx=0x7eff3000c920 comp rx=0 tx=0).stop 2024-04-02T13:24:44.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.194+0000 7eff41f9d640 1 -- 172.21.15.44:0/3891930378 shutdown_connections 2024-04-02T13:24:44.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.194+0000 7eff41f9d640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff2806c8f0 0x7eff2806edb0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:44.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.194+0000 7eff41f9d640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3c162470 0x7eff3c0cfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:44.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.194+0000 7eff41f9d640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c15e3e0 0x7eff3c15b510 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:44.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.194+0000 7eff41f9d640 1 --2- 172.21.15.44:0/3891930378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff3c15e000 0x7eff3c15afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:44.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.194+0000 7eff41f9d640 1 -- 172.21.15.44:0/3891930378 >> 172.21.15.44:0/3891930378 conn(0x7eff3c09d1f0 msgr2=0x7eff3c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:44.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.194+0000 7eff41f9d640 1 -- 172.21.15.44:0/3891930378 shutdown_connections 2024-04-02T13:24:44.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.194+0000 7eff41f9d640 1 -- 172.21.15.44:0/3891930378 wait complete. 2024-04-02T13:24:44.210 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T13:24:44.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.325+0000 7f71a1982640 1 Processor -- start 2024-04-02T13:24:44.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.325+0000 7f71a1982640 1 -- start start 2024-04-02T13:24:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f71a1982640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f719c06daa0 0x7f719c0b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:44.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f71a1982640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f719c06de80 con 0x7f719c0cfdc0 2024-04-02T13:24:44.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f71a1982640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f719c0cf000 con 0x7f719c0d2e00 2024-04-02T13:24:44.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f71a1982640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f719c0cf930 con 0x7f719c06daa0 2024-04-02T13:24:44.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f7193fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f719c0cfdc0 0x7f719c0d26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48184/0 (socket says 172.21.15.44:48184) 2024-04-02T13:24:44.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f7193fff640 1 -- 172.21.15.44:0/4249748597 learned_addr learned my addr 172.21.15.44:0/4249748597 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:44.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 903743469 0 0) 0x7f719c06de80 con 0x7f719c0cfdc0 2024-04-02T13:24:44.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7188003520 con 0x7f719c0cfdc0 2024-04-02T13:24:44.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f719affd640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f719c06daa0 0x7f719c0b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:44.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2048295400 0 0) 0x7f7188003520 con 0x7f719c0cfdc0 2024-04-02T13:24:44.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f719c06daa0 msgr2=0x7f719c0b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:44.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f719a7fc640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f719c06daa0 0x7f719c0b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:44.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 >> v1:172.21.15.69:6789/0 conn(0x7f719c0d2e00 legacy=0x7f719c15dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:44.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719c0cf2e0 con 0x7f719c0cfdc0 2024-04-02T13:24:44.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.326+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f7184001670 con 0x7f719c0cfdc0 2024-04-02T13:24:44.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 >> v1:172.21.15.44:6789/0 conn(0x7f719c0cfdc0 legacy=0x7f719c0d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:44.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f719a7fc640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71880039e0 0x7f7188003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:44.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f719a7fc640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7188004800 0x7f7188004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:44.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f719a7fc640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71880051c0 0x7f718800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:44.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f719c0cf930 con 0x7f7188004800 2024-04-02T13:24:44.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f718800c090 con 0x7f71880051c0 2024-04-02T13:24:44.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f718800c290 con 0x7f71880039e0 2024-04-02T13:24:44.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f719affd640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f719c06daa0 0x7f719c0b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:44.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f7193fff640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7188004800 0x7f7188004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:44.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f719b7fe640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71880051c0 0x7f718800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:44.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f719affd640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71880039e0 0x7f7188003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:44.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f7193fff640 1 -- 172.21.15.44:0/4249748597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71880039e0 msgr2=0x7f7188003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:44.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f7193fff640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71880039e0 0x7f7188003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:44.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f7193fff640 1 -- 172.21.15.44:0/4249748597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71880051c0 msgr2=0x7f718800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:44.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f7193fff640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71880051c0 0x7f718800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:44.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f7193fff640 1 -- 172.21.15.44:0/4249748597 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7188003520 con 0x7f7188004800 2024-04-02T13:24:44.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.327+0000 7f719affd640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71880039e0 0x7f7188003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:44.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.328+0000 7f7193fff640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7188004800 0x7f7188004c80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f719c0d01c0 tx=0x7f7188005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa1cba75dfdeeaa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:44.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.328+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7184007d30 con 0x7f7188004800 2024-04-02T13:24:44.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.328+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7184030070 con 0x7f7188004800 2024-04-02T13:24:44.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.328+0000 7f719a7fc640 1 -- 172.21.15.44:0/4249748597 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7184005d30 con 0x7f7188004800 2024-04-02T13:24:44.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.328+0000 7f71a1982640 1 -- 172.21.15.44:0/4249748597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7188004800 msgr2=0x7f7188004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:44.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.328+0000 7f71a1982640 1 --2- 172.21.15.44:0/4249748597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7188004800 0x7f7188004c80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f719c0d01c0 tx=0x7f7188005830 comp rx=0 tx=0).stop 2024-04-02T13:24:44.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.328+0000 7f719b7fe640 1 -- 172.21.15.44:0/4249748597 reap_dead start 2024-04-02T13:24:44.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.328+0000 7f71a1982640 1 -- 172.21.15.44:0/4249748597 shutdown_connections 2024-04-02T13:24:44.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.328+0000 7f71a1982640 1 -- 172.21.15.44:0/4249748597 >> 172.21.15.44:0/4249748597 conn(0x7f719c09d430 msgr2=0x7f719c0d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:44.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.329+0000 7f71a1982640 1 -- 172.21.15.44:0/4249748597 shutdown_connections 2024-04-02T13:24:44.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.329+0000 7f71a1982640 1 -- 172.21.15.44:0/4249748597 wait complete. 2024-04-02T13:24:44.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.330+0000 7f71a1982640 1 Processor -- start 2024-04-02T13:24:44.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.330+0000 7f71a1982640 1 -- start start 2024-04-02T13:24:44.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.330+0000 7f71a1982640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7188004800 0x7f719c15f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:44.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.330+0000 7f71a1982640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f719c0cca50 0x7f719c0cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:44.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.330+0000 7f71a1982640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f719c0b1d70 0x7f719c0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:44.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.330+0000 7f71a1982640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f719c0ceda0 con 0x7f719c0cca50 2024-04-02T13:24:44.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.330+0000 7f71a1982640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f719c0cec20 con 0x7f7188004800 2024-04-02T13:24:44.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.330+0000 7f71a1982640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f719c0b35b0 con 0x7f719c0b1d70 2024-04-02T13:24:44.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f7193fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f719c0cca50 0x7f719c0cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:44.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f7193fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f719c0cca50 0x7f719c0cced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47872/0 (socket says 172.21.15.44:47872) 2024-04-02T13:24:44.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f7193fff640 1 -- 172.21.15.44:0/1315939364 learned_addr learned my addr 172.21.15.44:0/1315939364 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:44.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f719affd640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7188004800 0x7f719c15f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:44.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f719b7fe640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f719c0b1d70 0x7f719c0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f7193fff640 1 -- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f719c0b1d70 msgr2=0x7f719c0b21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f7193fff640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f719c0b1d70 0x7f719c0b21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f7193fff640 1 -- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7188004800 msgr2=0x7f719c15f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f7193fff640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7188004800 0x7f719c15f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f7193fff640 1 -- 172.21.15.44:0/1315939364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719c0cdad0 con 0x7f719c0cca50 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f7193fff640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f719c0cca50 0x7f719c0cced0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f718402c540 tx=0x7f7184001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f7198ff9640 1 -- 172.21.15.44:0/1315939364 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7184045070 con 0x7f719c0cca50 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f7198ff9640 1 -- 172.21.15.44:0/1315939364 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7184004bf0 con 0x7f719c0cca50 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f71a1982640 1 -- 172.21.15.44:0/1315939364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f719c0b6dd0 con 0x7f719c0cca50 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f7198ff9640 1 -- 172.21.15.44:0/1315939364 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7184007d10 con 0x7f719c0cca50 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.331+0000 7f71a1982640 1 -- 172.21.15.44:0/1315939364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f719c0cd510 con 0x7f719c0cca50 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.332+0000 7f71a1982640 1 -- 172.21.15.44:0/1315939364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f719c0b2730 con 0x7f719c0cca50 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.333+0000 7f7198ff9640 1 -- 172.21.15.44:0/1315939364 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7184007070 con 0x7f719c0cca50 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.333+0000 7f7198ff9640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f717c06c920 0x7f717c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:44.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.333+0000 7f7198ff9640 1 -- 172.21.15.44:0/1315939364 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5865+0+0 (secure 0 0 0) 0x7f71840acb80 con 0x7f719c0cca50 2024-04-02T13:24:44.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.333+0000 7f719affd640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f717c06c920 0x7f717c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:44.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.334+0000 7f719affd640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f717c06c920 0x7f717c06ede0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f718c006fd0 tx=0x7f718c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:44.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.338+0000 7f7198ff9640 1 -- 172.21.15.44:0/1315939364 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f718407a9a0 con 0x7f719c0cca50 2024-04-02T13:24:44.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:44.521+0000 7f71a1982640 1 -- 172.21.15.44:0/1315939364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f719c0684f0 con 0x7f719c0cca50 2024-04-02T13:24:45.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.195+0000 7f7198ff9640 1 -- 172.21.15.44:0/1315939364 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v70) ==== 169+0+0 (secure 0 0 0) 0x7f718407eca0 con 0x7f719c0cca50 2024-04-02T13:24:45.197 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' removed 2024-04-02T13:24:45.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.198+0000 7f71a1982640 1 -- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f717c06c920 msgr2=0x7f717c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:45.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.198+0000 7f71a1982640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f717c06c920 0x7f717c06ede0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f718c006fd0 tx=0x7f718c008040 comp rx=0 tx=0).stop 2024-04-02T13:24:45.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.198+0000 7f71a1982640 1 -- 172.21.15.44:0/1315939364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f719c0cca50 msgr2=0x7f719c0cced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:45.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.198+0000 7f71a1982640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f719c0cca50 0x7f719c0cced0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f718402c540 tx=0x7f7184001a40 comp rx=0 tx=0).stop 2024-04-02T13:24:45.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.198+0000 7f71a1982640 1 -- 172.21.15.44:0/1315939364 shutdown_connections 2024-04-02T13:24:45.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.198+0000 7f71a1982640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f717c06c920 0x7f717c06ede0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.198+0000 7f71a1982640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f719c0b1d70 0x7f719c0b21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.198+0000 7f71a1982640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f719c0cca50 0x7f719c0cced0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.198+0000 7f71a1982640 1 --2- 172.21.15.44:0/1315939364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7188004800 0x7f719c15f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.198+0000 7f71a1982640 1 -- 172.21.15.44:0/1315939364 >> 172.21.15.44:0/1315939364 conn(0x7f719c09d430 msgr2=0x7f719c0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:45.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.198+0000 7f71a1982640 1 -- 172.21.15.44:0/1315939364 shutdown_connections 2024-04-02T13:24:45.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.198+0000 7f71a1982640 1 -- 172.21.15.44:0/1315939364 wait complete. 2024-04-02T13:24:45.214 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T13:24:45.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.330+0000 7f7fafafb640 1 Processor -- start 2024-04-02T13:24:45.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.330+0000 7f7fafafb640 1 -- start start 2024-04-02T13:24:45.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.331+0000 7f7fafafb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7fa815c260 0x7f7fa80b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:45.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.331+0000 7f7fafafb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa80cd760 con 0x7f7fa80cd200 2024-04-02T13:24:45.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.331+0000 7f7fafafb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa80cda40 con 0x7f7fa80b3e80 2024-04-02T13:24:45.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.331+0000 7f7fafafb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7fa80cdd20 con 0x7f7fa815c260 2024-04-02T13:24:45.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.331+0000 7f7fae098640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7fa80cd200 0x7f7fa80c5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33476/0 (socket says 172.21.15.44:33476) 2024-04-02T13:24:45.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.331+0000 7f7fae098640 1 -- 172.21.15.44:0/1937156495 learned_addr learned my addr 172.21.15.44:0/1937156495 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:45.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.331+0000 7f7fad897640 1 --2- 172.21.15.44:0/1937156495 >> v2:172.21.15.69:3300/0 conn(0x7f7fa815c260 0x7f7fa80b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:45.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.331+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2686485328 0 0) 0x7f7fa80cd760 con 0x7f7fa80cd200 2024-04-02T13:24:45.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f94003520 con 0x7f7fa80cd200 2024-04-02T13:24:45.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3456520224 0 0) 0x7f7f94003520 con 0x7f7fa80cd200 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 >> v2:172.21.15.69:3300/0 conn(0x7f7fa815c260 msgr2=0x7f7fa80b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 --2- 172.21.15.44:0/1937156495 >> v2:172.21.15.69:3300/0 conn(0x7f7fa815c260 0x7f7fa80b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 >> v1:172.21.15.69:6789/0 conn(0x7f7fa80b3e80 legacy=0x7f7fa80ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa80ce000 con 0x7f7fa80cd200 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f7fa0001a10 con 0x7f7fa80cd200 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 >> v1:172.21.15.44:6789/0 conn(0x7f7fa80cd200 legacy=0x7f7fa80c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 --2- 172.21.15.44:0/1937156495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f940039b0 0x7f7f94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 --2- 172.21.15.44:0/1937156495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f940043b0 0x7f7f94004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 --2- 172.21.15.44:0/1937156495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f94005080 0x7f7f9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa80cdd20 con 0x7f7f940043b0 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9400bfe0 con 0x7f7f940039b0 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7f9400c1e0 con 0x7f7f94005080 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fad096640 1 --2- 172.21.15.44:0/1937156495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f940043b0 0x7f7f94004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:45.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fad897640 1 --2- 172.21.15.44:0/1937156495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f940039b0 0x7f7f94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:45.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fae098640 1 --2- 172.21.15.44:0/1937156495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f94005080 0x7f7f9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:45.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fad096640 1 -- 172.21.15.44:0/1937156495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f94005080 msgr2=0x7f7f9400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:45.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fad096640 1 --2- 172.21.15.44:0/1937156495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f94005080 0x7f7f9400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fad096640 1 -- 172.21.15.44:0/1937156495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f940039b0 msgr2=0x7f7f94003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:45.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fad096640 1 --2- 172.21.15.44:0/1937156495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f940039b0 0x7f7f94003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fad096640 1 -- 172.21.15.44:0/1937156495 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f94003520 con 0x7f7f940043b0 2024-04-02T13:24:45.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fae098640 1 --2- 172.21.15.44:0/1937156495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f94005080 0x7f7f9400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:45.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.332+0000 7f7fae098640 1 -- 172.21.15.44:0/1937156495 reap_dead start 2024-04-02T13:24:45.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.333+0000 7f7fad096640 1 --2- 172.21.15.44:0/1937156495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f940043b0 0x7f7f94004880 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4001860 tx=0x7f7f94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d03961a390360f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:45.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.333+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4001430 con 0x7f7f940043b0 2024-04-02T13:24:45.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.333+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fa4033610 con 0x7f7f940043b0 2024-04-02T13:24:45.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.333+0000 7f7fac895640 1 -- 172.21.15.44:0/1937156495 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4033a00 con 0x7f7f940043b0 2024-04-02T13:24:45.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.334+0000 7f7fafafb640 1 -- 172.21.15.44:0/1937156495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f940043b0 msgr2=0x7f7f94004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:45.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.334+0000 7f7fafafb640 1 --2- 172.21.15.44:0/1937156495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f940043b0 0x7f7f94004880 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4001860 tx=0x7f7f94005830 comp rx=0 tx=0).stop 2024-04-02T13:24:45.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.334+0000 7f7fafafb640 1 -- 172.21.15.44:0/1937156495 shutdown_connections 2024-04-02T13:24:45.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.334+0000 7f7fafafb640 1 --2- 172.21.15.44:0/1937156495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f940043b0 0x7f7f94004880 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.334+0000 7f7fafafb640 1 -- 172.21.15.44:0/1937156495 >> 172.21.15.44:0/1937156495 conn(0x7f7fa809aa20 msgr2=0x7f7fa809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:45.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.334+0000 7f7fafafb640 1 -- 172.21.15.44:0/1937156495 shutdown_connections 2024-04-02T13:24:45.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.334+0000 7f7fafafb640 1 -- 172.21.15.44:0/1937156495 wait complete. 2024-04-02T13:24:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.335+0000 7f7fafafb640 1 Processor -- start 2024-04-02T13:24:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.335+0000 7f7fafafb640 1 -- start start 2024-04-02T13:24:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fafafb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f940043b0 0x7f7fa815eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fafafb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fa81e5bd0 0x7f7fa81f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fafafb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fa81e6050 0x7f7fa81f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fafafb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa80b3ac0 con 0x7f7fa81e6050 2024-04-02T13:24:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fafafb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa80b3940 con 0x7f7f940043b0 2024-04-02T13:24:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fafafb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7fa80b3c40 con 0x7f7fa81e5bd0 2024-04-02T13:24:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fae098640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fa81e6050 0x7f7fa81f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fae098640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fa81e6050 0x7f7fa81f5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35368/0 (socket says 172.21.15.44:35368) 2024-04-02T13:24:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fae098640 1 -- 172.21.15.44:0/2700329730 learned_addr learned my addr 172.21.15.44:0/2700329730 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fad897640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f940043b0 0x7f7fa815eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fad096640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fa81e5bd0 0x7f7fa81f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fae098640 1 -- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fa81e5bd0 msgr2=0x7f7fa81f4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fae098640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fa81e5bd0 0x7f7fa81f4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fae098640 1 -- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f940043b0 msgr2=0x7f7fa815eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fae098640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f940043b0 0x7f7fa815eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fae098640 1 -- 172.21.15.44:0/2700329730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa81e4b30 con 0x7f7fa81e6050 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7fae098640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fa81e6050 0x7f7fa81f5440 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80ce830 tx=0x7f7fa000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7f9e7fc640 1 -- 172.21.15.44:0/2700329730 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa000eb90 con 0x7f7fa81e6050 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7f9e7fc640 1 -- 172.21.15.44:0/2700329730 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fa0018070 con 0x7f7fa81e6050 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.336+0000 7f7f9e7fc640 1 -- 172.21.15.44:0/2700329730 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0013420 con 0x7f7fa81e6050 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.337+0000 7f7fafafb640 1 -- 172.21.15.44:0/2700329730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa81e4e10 con 0x7f7fa81e6050 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.337+0000 7f7fafafb640 1 -- 172.21.15.44:0/2700329730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa81e53d0 con 0x7f7fa81e6050 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.338+0000 7f7fafafb640 1 -- 172.21.15.44:0/2700329730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fa80684f0 con 0x7f7fa81e6050 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.338+0000 7f7f9e7fc640 1 -- 172.21.15.44:0/2700329730 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7fa00135c0 con 0x7f7fa81e6050 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.338+0000 7f7f9e7fc640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f8c06c920 0x7f7f8c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:45.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.338+0000 7f7f9e7fc640 1 -- 172.21.15.44:0/2700329730 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5335+0+0 (secure 0 0 0) 0x7f7fa008bd50 con 0x7f7fa81e6050 2024-04-02T13:24:45.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.339+0000 7f7fad897640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f8c06c920 0x7f7f8c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:45.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.339+0000 7f7fad897640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f8c06c920 0x7f7f8c06ede0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8158580 tx=0x7f7f98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:45.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.343+0000 7f7f9e7fc640 1 -- 172.21.15.44:0/2700329730 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7fa008f030 con 0x7f7fa81e6050 2024-04-02T13:24:45.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.527+0000 7f7fafafb640 1 -- 172.21.15.44:0/2700329730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f7fa806f510 con 0x7f7fa81e6050 2024-04-02T13:24:45.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.530+0000 7f7f9e7fc640 1 -- 172.21.15.44:0/2700329730 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) ==== 104+0+0 (secure 0 0 0) 0x7f7fa005a2f0 con 0x7f7fa81e6050 2024-04-02T13:24:45.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.533+0000 7f7fafafb640 1 -- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f8c06c920 msgr2=0x7f7f8c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:45.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.533+0000 7f7fafafb640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f8c06c920 0x7f7f8c06ede0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8158580 tx=0x7f7f98008040 comp rx=0 tx=0).stop 2024-04-02T13:24:45.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.533+0000 7f7fafafb640 1 -- 172.21.15.44:0/2700329730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fa81e6050 msgr2=0x7f7fa81f5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:45.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.533+0000 7f7fafafb640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fa81e6050 0x7f7fa81f5440 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80ce830 tx=0x7f7fa000e530 comp rx=0 tx=0).stop 2024-04-02T13:24:45.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.533+0000 7f7fafafb640 1 -- 172.21.15.44:0/2700329730 shutdown_connections 2024-04-02T13:24:45.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.533+0000 7f7fafafb640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f8c06c920 0x7f7f8c06ede0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.533+0000 7f7fafafb640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fa81e6050 0x7f7fa81f5440 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.533+0000 7f7fafafb640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fa81e5bd0 0x7f7fa81f4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.533+0000 7f7fafafb640 1 --2- 172.21.15.44:0/2700329730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f940043b0 0x7f7fa815eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.533+0000 7f7fafafb640 1 -- 172.21.15.44:0/2700329730 >> 172.21.15.44:0/2700329730 conn(0x7f7fa809aa20 msgr2=0x7f7fa806fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:45.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.533+0000 7f7fafafb640 1 -- 172.21.15.44:0/2700329730 shutdown_connections 2024-04-02T13:24:45.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.533+0000 7f7fafafb640 1 -- 172.21.15.44:0/2700329730 wait complete. 2024-04-02T13:24:45.545 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2024-04-02T13:24:45.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.644+0000 7fee14877640 1 Processor -- start 2024-04-02T13:24:45.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.645+0000 7fee14877640 1 -- start start 2024-04-02T13:24:45.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.645+0000 7fee14877640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fee100d0a30 0x7fee1006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:45.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.645+0000 7fee14877640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee100d0000 con 0x7fee100d0e10 2024-04-02T13:24:45.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.645+0000 7fee14877640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee1006c1d0 con 0x7fee100d2040 2024-04-02T13:24:45.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.645+0000 7fee14877640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fee1006cb00 con 0x7fee100d0a30 2024-04-02T13:24:45.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.646+0000 7fee0e59c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fee100d0a30 0x7fee1006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:45.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.646+0000 7fee0dd9b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fee100d0e10 0x7fee100cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33480/0 (socket says 172.21.15.44:33480) 2024-04-02T13:24:45.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.646+0000 7fee0dd9b640 1 -- 172.21.15.44:0/2410985355 learned_addr learned my addr 172.21.15.44:0/2410985355 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:45.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.646+0000 7fee0e59c640 1 -- 172.21.15.44:0/2410985355 >> v1:172.21.15.69:6789/0 conn(0x7fee100d2040 legacy=0x7fee100d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:45.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.646+0000 7fee0e59c640 1 -- 172.21.15.44:0/2410985355 >> v1:172.21.15.44:6789/0 conn(0x7fee100d0e10 legacy=0x7fee100cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:45.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.646+0000 7fee0d59a640 1 -- 172.21.15.44:0/2410985355 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3366554673 0 0) 0x7fee100d0000 con 0x7fee100d0e10 2024-04-02T13:24:45.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.646+0000 7fee0e59c640 1 -- 172.21.15.44:0/2410985355 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee1006c4b0 con 0x7fee100d0a30 2024-04-02T13:24:45.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.647+0000 7fee0e59c640 1 --2- 172.21.15.44:0/2410985355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee100d0a30 0x7fee1006cf90 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fee080097a0 tx=0x7fee0802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d52adf2e922ff6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:45.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.647+0000 7fee0d59a640 1 -- 172.21.15.44:0/2410985355 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee080315a0 con 0x7fee100d0a30 2024-04-02T13:24:45.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.647+0000 7fee0d59a640 1 -- 172.21.15.44:0/2410985355 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee08037070 con 0x7fee100d0a30 2024-04-02T13:24:45.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.647+0000 7fee0d59a640 1 -- 172.21.15.44:0/2410985355 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee08031960 con 0x7fee100d0a30 2024-04-02T13:24:45.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.647+0000 7fee14877640 1 -- 172.21.15.44:0/2410985355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee100d0a30 msgr2=0x7fee1006cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:45.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.647+0000 7fee14877640 1 --2- 172.21.15.44:0/2410985355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee100d0a30 0x7fee1006cf90 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fee080097a0 tx=0x7fee0802b840 comp rx=0 tx=0).stop 2024-04-02T13:24:45.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.647+0000 7fee14877640 1 -- 172.21.15.44:0/2410985355 shutdown_connections 2024-04-02T13:24:45.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.647+0000 7fee14877640 1 --2- 172.21.15.44:0/2410985355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee100d0a30 0x7fee1006cf90 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.647+0000 7fee14877640 1 -- 172.21.15.44:0/2410985355 >> 172.21.15.44:0/2410985355 conn(0x7fee1009d470 msgr2=0x7fee100d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:45.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.647+0000 7fee14877640 1 -- 172.21.15.44:0/2410985355 shutdown_connections 2024-04-02T13:24:45.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.648+0000 7fee14877640 1 -- 172.21.15.44:0/2410985355 wait complete. 2024-04-02T13:24:45.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.649+0000 7fee14877640 1 Processor -- start 2024-04-02T13:24:45.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.649+0000 7fee14877640 1 -- start start 2024-04-02T13:24:45.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.649+0000 7fee14877640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee100d0a30 0x7fee10146760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:45.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.649+0000 7fee14877640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee100d0e10 0x7fee100c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:45.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee14877640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee100d2040 0x7fee100b6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:45.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee14877640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fee100d2c00 con 0x7fee100d2040 2024-04-02T13:24:45.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee14877640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fee100d1270 con 0x7fee100d0e10 2024-04-02T13:24:45.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee14877640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fee1015e6f0 con 0x7fee100d0a30 2024-04-02T13:24:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee0ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee100d2040 0x7fee100b6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee0ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee100d2040 0x7fee100b6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35386/0 (socket says 172.21.15.44:35386) 2024-04-02T13:24:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee0ed9d640 1 -- 172.21.15.44:0/3566160783 learned_addr learned my addr 172.21.15.44:0/3566160783 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee0e59c640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee100d0a30 0x7fee10146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee0dd9b640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee100d0e10 0x7fee100c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee0ed9d640 1 -- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee100d0a30 msgr2=0x7fee10146760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee0ed9d640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee100d0a30 0x7fee10146760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee0ed9d640 1 -- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee100d0e10 msgr2=0x7fee100c9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee0ed9d640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee100d0e10 0x7fee100c9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee0ed9d640 1 -- 172.21.15.44:0/3566160783 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee100b6970 con 0x7fee100d2040 2024-04-02T13:24:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee0ed9d640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee100d2040 0x7fee100b6430 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fee0000b970 tx=0x7fee0000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee06ffd640 1 -- 172.21.15.44:0/3566160783 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00019070 con 0x7fee100d2040 2024-04-02T13:24:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee06ffd640 1 -- 172.21.15.44:0/3566160783 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee0000cb40 con 0x7fee100d2040 2024-04-02T13:24:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee14877640 1 -- 172.21.15.44:0/3566160783 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee100ca0f0 con 0x7fee100d2040 2024-04-02T13:24:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee06ffd640 1 -- 172.21.15.44:0/3566160783 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00014440 con 0x7fee100d2040 2024-04-02T13:24:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.650+0000 7fee14877640 1 -- 172.21.15.44:0/3566160783 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee1015eb90 con 0x7fee100d2040 2024-04-02T13:24:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.651+0000 7fee14877640 1 -- 172.21.15.44:0/3566160783 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee1015e980 con 0x7fee100d2040 2024-04-02T13:24:45.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.652+0000 7fee06ffd640 1 -- 172.21.15.44:0/3566160783 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fee0001c070 con 0x7fee100d2040 2024-04-02T13:24:45.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.652+0000 7fee06ffd640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fedf006c8f0 0x7fedf006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:45.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.652+0000 7fee06ffd640 1 -- 172.21.15.44:0/3566160783 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5335+0+0 (secure 0 0 0) 0x7fee0008b5d0 con 0x7fee100d2040 2024-04-02T13:24:45.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.653+0000 7fee0e59c640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fedf006c8f0 0x7fedf006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:45.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.653+0000 7fee0e59c640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fedf006c8f0 0x7fedf006edb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fee08036000 tx=0x7fee0803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:45.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.657+0000 7fee06ffd640 1 -- 172.21.15.44:0/3566160783 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fee00059480 con 0x7fee100d2040 2024-04-02T13:24:45.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:45.911+0000 7fee14877640 1 -- 172.21.15.44:0/3566160783 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) -- 0x7fee100684f0 con 0x7fee100d2040 2024-04-02T13:24:46.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.197+0000 7fee06ffd640 1 -- 172.21.15.44:0/3566160783 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v230) ==== 169+0+0 (secure 0 0 0) 0x7fee0005d780 con 0x7fee100d2040 2024-04-02T13:24:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.199+0000 7fee14877640 1 -- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fedf006c8f0 msgr2=0x7fedf006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.199+0000 7fee14877640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fedf006c8f0 0x7fedf006edb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fee08036000 tx=0x7fee0803d040 comp rx=0 tx=0).stop 2024-04-02T13:24:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.199+0000 7fee14877640 1 -- 172.21.15.44:0/3566160783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee100d2040 msgr2=0x7fee100b6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.199+0000 7fee14877640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee100d2040 0x7fee100b6430 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fee0000b970 tx=0x7fee0000be30 comp rx=0 tx=0).stop 2024-04-02T13:24:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.199+0000 7fee14877640 1 -- 172.21.15.44:0/3566160783 shutdown_connections 2024-04-02T13:24:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.199+0000 7fee14877640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fedf006c8f0 0x7fedf006edb0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.199+0000 7fee14877640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee100d2040 0x7fee100b6430 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.199+0000 7fee14877640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee100d0e10 0x7fee100c9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.199+0000 7fee14877640 1 --2- 172.21.15.44:0/3566160783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee100d0a30 0x7fee10146760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.199+0000 7fee14877640 1 -- 172.21.15.44:0/3566160783 >> 172.21.15.44:0/3566160783 conn(0x7fee1009d470 msgr2=0x7fee100d2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:46.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.199+0000 7fee14877640 1 -- 172.21.15.44:0/3566160783 shutdown_connections 2024-04-02T13:24:46.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.199+0000 7fee14877640 1 -- 172.21.15.44:0/3566160783 wait complete. 2024-04-02T13:24:46.213 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-04-02T13:24:46.213 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_dentry_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-04-02T13:24:46.213 INFO:tasks.cephfs_test_runner:test_health_status_after_dentry_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-02T13:24:46.213 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:24:46.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.313+0000 7fb177567640 1 Processor -- start 2024-04-02T13:24:46.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.313+0000 7fb177567640 1 -- start start 2024-04-02T13:24:46.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.314+0000 7fb177567640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb17006cf90 0x7fb1700cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.314+0000 7fb177567640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1700cfe30 0x7fb1700d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.314+0000 7fb177567640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb17006c1d0 con 0x7fb1700d0a30 2024-04-02T13:24:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.314+0000 7fb177567640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb17006cb00 con 0x7fb17006cf90 2024-04-02T13:24:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.314+0000 7fb177567640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb17006d3f0 con 0x7fb1700cfe30 2024-04-02T13:24:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.315+0000 7fb174b02640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb17006cf90 0x7fb1700cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.315+0000 7fb175b04640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1700cfe30 0x7fb1700d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.315+0000 7fb175303640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb1700d0a30 0x7fb1700d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:43050/0 (socket says 172.21.15.44:43050) 2024-04-02T13:24:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.315+0000 7fb175303640 1 -- 172.21.15.44:0/3081505646 learned_addr learned my addr 172.21.15.44:0/3081505646 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.315+0000 7fb175b04640 1 -- 172.21.15.44:0/3081505646 >> v2:172.21.15.69:3300/0 conn(0x7fb17006cf90 msgr2=0x7fb1700cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.315+0000 7fb175b04640 1 --2- 172.21.15.44:0/3081505646 >> v2:172.21.15.69:3300/0 conn(0x7fb17006cf90 0x7fb1700cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.315+0000 7fb175b04640 1 -- 172.21.15.44:0/3081505646 >> v1:172.21.15.69:6789/0 conn(0x7fb1700d0a30 legacy=0x7fb1700d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.315+0000 7fb175b04640 1 -- 172.21.15.44:0/3081505646 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17006c4b0 con 0x7fb1700cfe30 2024-04-02T13:24:46.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.316+0000 7fb175b04640 1 --2- 172.21.15.44:0/3081505646 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1700cfe30 0x7fb1700d2040 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb160009d90 tx=0x7fb16002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8379a9c5c30d75fe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:46.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.316+0000 7fb16ffff640 1 -- 172.21.15.44:0/3081505646 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160039070 con 0x7fb1700cfe30 2024-04-02T13:24:46.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.316+0000 7fb16ffff640 1 -- 172.21.15.44:0/3081505646 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb16002cbc0 con 0x7fb1700cfe30 2024-04-02T13:24:46.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.316+0000 7fb16ffff640 1 -- 172.21.15.44:0/3081505646 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160034440 con 0x7fb1700cfe30 2024-04-02T13:24:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.316+0000 7fb177567640 1 -- 172.21.15.44:0/3081505646 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1700cfe30 msgr2=0x7fb1700d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.316+0000 7fb177567640 1 --2- 172.21.15.44:0/3081505646 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1700cfe30 0x7fb1700d2040 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb160009d90 tx=0x7fb16002be10 comp rx=0 tx=0).stop 2024-04-02T13:24:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.317+0000 7fb177567640 1 -- 172.21.15.44:0/3081505646 shutdown_connections 2024-04-02T13:24:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.317+0000 7fb177567640 1 --2- 172.21.15.44:0/3081505646 >> v2:172.21.15.69:3300/0 conn(0x7fb17006cf90 0x7fb1700cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.317+0000 7fb177567640 1 --2- 172.21.15.44:0/3081505646 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1700cfe30 0x7fb1700d2040 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.317+0000 7fb177567640 1 -- 172.21.15.44:0/3081505646 >> 172.21.15.44:0/3081505646 conn(0x7fb17009d470 msgr2=0x7fb1700d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:46.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.317+0000 7fb177567640 1 -- 172.21.15.44:0/3081505646 shutdown_connections 2024-04-02T13:24:46.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.317+0000 7fb177567640 1 -- 172.21.15.44:0/3081505646 wait complete. 2024-04-02T13:24:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.318+0000 7fb177567640 1 Processor -- start 2024-04-02T13:24:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.318+0000 7fb177567640 1 -- start start 2024-04-02T13:24:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.318+0000 7fb177567640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb17006cf90 0x7fb17014e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.318+0000 7fb177567640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1700cfe30 0x7fb1700b8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.318+0000 7fb177567640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1700d0a30 0x7fb1700b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb177567640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb17015e870 con 0x7fb1700cfe30 2024-04-02T13:24:46.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb177567640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb17015e6f0 con 0x7fb17006cf90 2024-04-02T13:24:46.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb177567640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb1700d4da0 con 0x7fb1700d0a30 2024-04-02T13:24:46.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb174b02640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1700cfe30 0x7fb1700b8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb174b02640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1700cfe30 0x7fb1700b8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35398/0 (socket says 172.21.15.44:35398) 2024-04-02T13:24:46.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb174b02640 1 -- 172.21.15.44:0/2582148834 learned_addr learned my addr 172.21.15.44:0/2582148834 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:46.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb175303640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb17006cf90 0x7fb17014e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb175b04640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1700d0a30 0x7fb1700b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb174b02640 1 -- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1700d0a30 msgr2=0x7fb1700b8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb174b02640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1700d0a30 0x7fb1700b8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb174b02640 1 -- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb17006cf90 msgr2=0x7fb17014e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb174b02640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb17006cf90 0x7fb17014e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb174b02640 1 -- 172.21.15.44:0/2582148834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17014eac0 con 0x7fb1700cfe30 2024-04-02T13:24:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb175303640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb17006cf90 0x7fb17014e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb174b02640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1700cfe30 0x7fb1700b8750 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fb16400e870 tx=0x7fb16400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb16dffb640 1 -- 172.21.15.44:0/2582148834 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb164017070 con 0x7fb1700cfe30 2024-04-02T13:24:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb16dffb640 1 -- 172.21.15.44:0/2582148834 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb16400c940 con 0x7fb1700cfe30 2024-04-02T13:24:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb16dffb640 1 -- 172.21.15.44:0/2582148834 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16400cd10 con 0x7fb1700cfe30 2024-04-02T13:24:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.319+0000 7fb177567640 1 -- 172.21.15.44:0/2582148834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1700c7c30 con 0x7fb1700cfe30 2024-04-02T13:24:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.320+0000 7fb177567640 1 -- 172.21.15.44:0/2582148834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1700b80c0 con 0x7fb1700cfe30 2024-04-02T13:24:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.320+0000 7fb177567640 1 -- 172.21.15.44:0/2582148834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1700684f0 con 0x7fb1700cfe30 2024-04-02T13:24:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.321+0000 7fb16dffb640 1 -- 172.21.15.44:0/2582148834 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb16401a070 con 0x7fb1700cfe30 2024-04-02T13:24:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.321+0000 7fb16dffb640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb15c06c8f0 0x7fb15c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.321+0000 7fb16dffb640 1 -- 172.21.15.44:0/2582148834 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5335+0+0 (secure 0 0 0) 0x7fb16408b830 con 0x7fb1700cfe30 2024-04-02T13:24:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.322+0000 7fb175303640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb15c06c8f0 0x7fb15c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.322+0000 7fb175303640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb15c06c8f0 0x7fb15c06edb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb168000f60 tx=0x7fb168009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.326+0000 7fb16dffb640 1 -- 172.21.15.44:0/2582148834 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb17014eac0 con 0x7fb1700cfe30 2024-04-02T13:24:46.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.509+0000 7fb177567640 1 -- 172.21.15.44:0/2582148834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb1700d17c0 con 0x7fb1700cfe30 2024-04-02T13:24:46.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.510+0000 7fb16dffb640 1 -- 172.21.15.44:0/2582148834 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+13246 (secure 0 0 0) 0x7fb164059dd0 con 0x7fb1700cfe30 2024-04-02T13:24:46.512 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:46.513 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":70,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:24:45.191652+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6835/3127613802":"2024-04-03T13:24:42.186599+0000","172.21.15.44:6834/3127613802":"2024-04-03T13:24:42.186599+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:24:46.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.512+0000 7fb177567640 1 -- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb15c06c8f0 msgr2=0x7fb15c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.512+0000 7fb177567640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb15c06c8f0 0x7fb15c06edb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb168000f60 tx=0x7fb168009040 comp rx=0 tx=0).stop 2024-04-02T13:24:46.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.512+0000 7fb177567640 1 -- 172.21.15.44:0/2582148834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1700cfe30 msgr2=0x7fb1700b8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.512+0000 7fb177567640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1700cfe30 0x7fb1700b8750 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fb16400e870 tx=0x7fb16400ed30 comp rx=0 tx=0).stop 2024-04-02T13:24:46.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.512+0000 7fb177567640 1 -- 172.21.15.44:0/2582148834 shutdown_connections 2024-04-02T13:24:46.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.512+0000 7fb177567640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb15c06c8f0 0x7fb15c06edb0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.512+0000 7fb177567640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1700d0a30 0x7fb1700b8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.512+0000 7fb177567640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1700cfe30 0x7fb1700b8750 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.512+0000 7fb177567640 1 --2- 172.21.15.44:0/2582148834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb17006cf90 0x7fb17014e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.512+0000 7fb177567640 1 -- 172.21.15.44:0/2582148834 >> 172.21.15.44:0/2582148834 conn(0x7fb17009d470 msgr2=0x7fb1700d3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:46.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.512+0000 7fb177567640 1 -- 172.21.15.44:0/2582148834 shutdown_connections 2024-04-02T13:24:46.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.512+0000 7fb177567640 1 -- 172.21.15.44:0/2582148834 wait complete. 2024-04-02T13:24:46.525 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:24:46.525 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:24:46.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.628+0000 7f158f87b640 1 Processor -- start 2024-04-02T13:24:46.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.629+0000 7f158f87b640 1 -- start start 2024-04-02T13:24:46.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.629+0000 7f158f87b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15880721f0 0x7f15880cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.629+0000 7f158f87b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f15880cfc40 0x7f15880cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.629+0000 7f158f87b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f158806c220 con 0x7f158815e420 2024-04-02T13:24:46.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.629+0000 7f158f87b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f15880d00b0 con 0x7f15880cfc40 2024-04-02T13:24:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.629+0000 7f158f87b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f15880cf3b0 con 0x7f15880721f0 2024-04-02T13:24:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.630+0000 7f158d617640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f158815e420 0x7f15880ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33490/0 (socket says 172.21.15.44:33490) 2024-04-02T13:24:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.630+0000 7f158d617640 1 -- 172.21.15.44:0/2109797748 learned_addr learned my addr 172.21.15.44:0/2109797748 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.630+0000 7f158de18640 1 --2- 172.21.15.44:0/2109797748 >> v2:172.21.15.69:3300/0 conn(0x7f15880cfc40 0x7f15880cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.630+0000 7f158ce16640 1 --2- 172.21.15.44:0/2109797748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15880721f0 0x7f15880cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.630+0000 7f157ffff640 1 -- 172.21.15.44:0/2109797748 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3138141042 0 0) 0x7f1584001110 con 0x7f158815e420 2024-04-02T13:24:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.630+0000 7f157ffff640 1 -- 172.21.15.44:0/2109797748 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1574003520 con 0x7f158815e420 2024-04-02T13:24:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.630+0000 7f158ce16640 1 -- 172.21.15.44:0/2109797748 >> v2:172.21.15.69:3300/0 conn(0x7f15880cfc40 msgr2=0x7f15880cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.630+0000 7f158ce16640 1 --2- 172.21.15.44:0/2109797748 >> v2:172.21.15.69:3300/0 conn(0x7f15880cfc40 0x7f15880cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.630+0000 7f158ce16640 1 -- 172.21.15.44:0/2109797748 >> v1:172.21.15.44:6789/0 conn(0x7f158815e420 legacy=0x7f15880ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.630+0000 7f158ce16640 1 -- 172.21.15.44:0/2109797748 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158815f7e0 con 0x7f15880721f0 2024-04-02T13:24:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.630+0000 7f158de18640 1 --2- 172.21.15.44:0/2109797748 >> v2:172.21.15.69:3300/0 conn(0x7f15880cfc40 0x7f15880cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:24:46.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.631+0000 7f158ce16640 1 --2- 172.21.15.44:0/2109797748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15880721f0 0x7f15880cf700 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f15780097a0 tx=0x7f157802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd4682994b3a22ce server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:46.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.631+0000 7f157ffff640 1 -- 172.21.15.44:0/2109797748 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578038070 con 0x7f15880721f0 2024-04-02T13:24:46.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.631+0000 7f157ffff640 1 -- 172.21.15.44:0/2109797748 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1578031700 con 0x7f15880721f0 2024-04-02T13:24:46.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.631+0000 7f157ffff640 1 -- 172.21.15.44:0/2109797748 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578031ab0 con 0x7f15880721f0 2024-04-02T13:24:46.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.632+0000 7f158f87b640 1 -- 172.21.15.44:0/2109797748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15880721f0 msgr2=0x7f15880cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.632+0000 7f158f87b640 1 --2- 172.21.15.44:0/2109797748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15880721f0 0x7f15880cf700 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f15780097a0 tx=0x7f157802b7a0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.632+0000 7f158f87b640 1 -- 172.21.15.44:0/2109797748 shutdown_connections 2024-04-02T13:24:46.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.632+0000 7f158f87b640 1 --2- 172.21.15.44:0/2109797748 >> v2:172.21.15.69:3300/0 conn(0x7f15880cfc40 0x7f15880cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.632+0000 7f158f87b640 1 --2- 172.21.15.44:0/2109797748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15880721f0 0x7f15880cf700 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.632+0000 7f158f87b640 1 -- 172.21.15.44:0/2109797748 >> 172.21.15.44:0/2109797748 conn(0x7f15880b2a10 msgr2=0x7f15880b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:46.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.632+0000 7f158f87b640 1 -- 172.21.15.44:0/2109797748 shutdown_connections 2024-04-02T13:24:46.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.632+0000 7f158f87b640 1 -- 172.21.15.44:0/2109797748 wait complete. 2024-04-02T13:24:46.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.633+0000 7f158f87b640 1 Processor -- start 2024-04-02T13:24:46.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.633+0000 7f158f87b640 1 -- start start 2024-04-02T13:24:46.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158f87b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15880721f0 0x7f158815a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158f87b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15880cfc40 0x7f158814f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158f87b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158815e420 0x7f15880b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158f87b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f158815f220 con 0x7f15880cfc40 2024-04-02T13:24:46.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158f87b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f158815ef40 con 0x7f158815e420 2024-04-02T13:24:46.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158f87b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f158815f500 con 0x7f15880721f0 2024-04-02T13:24:46.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158ce16640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15880cfc40 0x7f158814f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158ce16640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15880cfc40 0x7f158814f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35414/0 (socket says 172.21.15.44:35414) 2024-04-02T13:24:46.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158ce16640 1 -- 172.21.15.44:0/3506164073 learned_addr learned my addr 172.21.15.44:0/3506164073 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:46.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158d617640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15880721f0 0x7f158815a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158de18640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158815e420 0x7f15880b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158ce16640 1 -- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15880721f0 msgr2=0x7f158815a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158ce16640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15880721f0 0x7f158815a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158ce16640 1 -- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158815e420 msgr2=0x7f15880b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158ce16640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158815e420 0x7f15880b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158ce16640 1 -- 172.21.15.44:0/3506164073 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158814fcb0 con 0x7f15880cfc40 2024-04-02T13:24:46.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158d617640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15880721f0 0x7f158815a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:46.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158ce16640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15880cfc40 0x7f158814f770 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f157802b7e0 tx=0x7f15780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:46.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f157dffb640 1 -- 172.21.15.44:0/3506164073 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578042070 con 0x7f15880cfc40 2024-04-02T13:24:46.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f158f87b640 1 -- 172.21.15.44:0/3506164073 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f158815ae50 con 0x7f15880cfc40 2024-04-02T13:24:46.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f157dffb640 1 -- 172.21.15.44:0/3506164073 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f157803d890 con 0x7f15880cfc40 2024-04-02T13:24:46.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.634+0000 7f157dffb640 1 -- 172.21.15.44:0/3506164073 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578038040 con 0x7f15880cfc40 2024-04-02T13:24:46.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.635+0000 7f158f87b640 1 -- 172.21.15.44:0/3506164073 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f158815b410 con 0x7f15880cfc40 2024-04-02T13:24:46.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.636+0000 7f158f87b640 1 -- 172.21.15.44:0/3506164073 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1588067b70 con 0x7f15880cfc40 2024-04-02T13:24:46.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.636+0000 7f157dffb640 1 -- 172.21.15.44:0/3506164073 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1578045070 con 0x7f15880cfc40 2024-04-02T13:24:46.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.637+0000 7f157dffb640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f156c06c8f0 0x7f156c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.637+0000 7f157dffb640 1 -- 172.21.15.44:0/3506164073 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5335+0+0 (secure 0 0 0) 0x7f15780ad730 con 0x7f15880cfc40 2024-04-02T13:24:46.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.637+0000 7f158d617640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f156c06c8f0 0x7f156c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.637+0000 7f158d617640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f156c06c8f0 0x7f156c06edb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1584007fd0 tx=0x7f1584009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:46.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.641+0000 7f157dffb640 1 -- 172.21.15.44:0/3506164073 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1588067b70 con 0x7f15880cfc40 2024-04-02T13:24:46.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.825+0000 7f158f87b640 1 -- 172.21.15.44:0/3506164073 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f15880684f0 con 0x7f15880cfc40 2024-04-02T13:24:46.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.826+0000 7f157dffb640 1 -- 172.21.15.44:0/3506164073 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) ==== 93+0+10 (secure 0 0 0) 0x7f1578030070 con 0x7f15880cfc40 2024-04-02T13:24:46.828 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:24:46.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.827+0000 7f158f87b640 1 -- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f156c06c8f0 msgr2=0x7f156c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.827+0000 7f158f87b640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f156c06c8f0 0x7f156c06edb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1584007fd0 tx=0x7f1584009040 comp rx=0 tx=0).stop 2024-04-02T13:24:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.827+0000 7f158f87b640 1 -- 172.21.15.44:0/3506164073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15880cfc40 msgr2=0x7f158814f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.827+0000 7f158f87b640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15880cfc40 0x7f158814f770 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f157802b7e0 tx=0x7f15780046c0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.828+0000 7f158f87b640 1 -- 172.21.15.44:0/3506164073 shutdown_connections 2024-04-02T13:24:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.828+0000 7f158f87b640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f156c06c8f0 0x7f156c06edb0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.828+0000 7f158f87b640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158815e420 0x7f15880b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.828+0000 7f158f87b640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15880cfc40 0x7f158814f770 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.828+0000 7f158f87b640 1 --2- 172.21.15.44:0/3506164073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15880721f0 0x7f158815a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.828+0000 7f158f87b640 1 -- 172.21.15.44:0/3506164073 >> 172.21.15.44:0/3506164073 conn(0x7f15880b2a10 msgr2=0x7f15880b0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:46.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.828+0000 7f158f87b640 1 -- 172.21.15.44:0/3506164073 shutdown_connections 2024-04-02T13:24:46.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.828+0000 7f158f87b640 1 -- 172.21.15.44:0/3506164073 wait complete. 2024-04-02T13:24:46.840 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair' 2024-04-02T13:24:46.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.940+0000 7f4c2543c640 1 Processor -- start 2024-04-02T13:24:46.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.940+0000 7f4c2543c640 1 -- start start 2024-04-02T13:24:46.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c2543c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c200cece0 0x7f4c200cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c2543c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4c200d09f0 0x7f4c2006d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c2543c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4c2006ece0 0x7f4c2015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c2543c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4c200cf600 con 0x7f4c2006ece0 2024-04-02T13:24:46.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c2543c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4c2006dc40 con 0x7f4c200d09f0 2024-04-02T13:24:46.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c2543c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4c2006df20 con 0x7f4c200cece0 2024-04-02T13:24:46.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c1f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4c2006ece0 0x7f4c2015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c1f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4c2006ece0 0x7f4c2015eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35416/0 (socket says 172.21.15.44:35416) 2024-04-02T13:24:46.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c1f7fe640 1 -- 172.21.15.44:0/3937226721 learned_addr learned my addr 172.21.15.44:0/3937226721 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:46.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c1e7fc640 1 --2- 172.21.15.44:0/3937226721 >> v2:172.21.15.69:3300/0 conn(0x7f4c200d09f0 0x7f4c2006d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c1effd640 1 --2- 172.21.15.44:0/3937226721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c200cece0 0x7f4c200cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c1f7fe640 1 -- 172.21.15.44:0/3937226721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c200cece0 msgr2=0x7f4c200cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:46.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c1f7fe640 1 --2- 172.21.15.44:0/3937226721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c200cece0 0x7f4c200cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c1f7fe640 1 -- 172.21.15.44:0/3937226721 >> v2:172.21.15.69:3300/0 conn(0x7f4c200d09f0 msgr2=0x7f4c2006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:46.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c1f7fe640 1 --2- 172.21.15.44:0/3937226721 >> v2:172.21.15.69:3300/0 conn(0x7f4c200d09f0 0x7f4c2006d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.941+0000 7f4c1f7fe640 1 -- 172.21.15.44:0/3937226721 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2006e200 con 0x7f4c2006ece0 2024-04-02T13:24:46.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.942+0000 7f4c1f7fe640 1 --2- 172.21.15.44:0/3937226721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c2006ece0 0x7f4c2015eb60 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f4c200cf7e0 tx=0x7f4c1002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dee828013bb9621 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:46.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.942+0000 7f4c1dffb640 1 -- 172.21.15.44:0/3937226721 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1002ca50 con 0x7f4c2006ece0 2024-04-02T13:24:46.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.942+0000 7f4c1dffb640 1 -- 172.21.15.44:0/3937226721 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1002cbf0 con 0x7f4c2006ece0 2024-04-02T13:24:46.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.942+0000 7f4c1dffb640 1 -- 172.21.15.44:0/3937226721 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c100343f0 con 0x7f4c2006ece0 2024-04-02T13:24:46.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.943+0000 7f4c2543c640 1 -- 172.21.15.44:0/3937226721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c2006ece0 msgr2=0x7f4c2015eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.943+0000 7f4c2543c640 1 --2- 172.21.15.44:0/3937226721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c2006ece0 0x7f4c2015eb60 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f4c200cf7e0 tx=0x7f4c1002beb0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.943+0000 7f4c2543c640 1 -- 172.21.15.44:0/3937226721 shutdown_connections 2024-04-02T13:24:46.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.943+0000 7f4c2543c640 1 --2- 172.21.15.44:0/3937226721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c2006ece0 0x7f4c2015eb60 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.943+0000 7f4c2543c640 1 --2- 172.21.15.44:0/3937226721 >> v2:172.21.15.69:3300/0 conn(0x7f4c200d09f0 0x7f4c2006d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.943+0000 7f4c2543c640 1 --2- 172.21.15.44:0/3937226721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c200cece0 0x7f4c200cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.943+0000 7f4c2543c640 1 -- 172.21.15.44:0/3937226721 >> 172.21.15.44:0/3937226721 conn(0x7f4c200cfcd0 msgr2=0x7f4c200d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:46.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.943+0000 7f4c2543c640 1 -- 172.21.15.44:0/3937226721 shutdown_connections 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.943+0000 7f4c2543c640 1 -- 172.21.15.44:0/3937226721 wait complete. 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.944+0000 7f4c2543c640 1 Processor -- start 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.944+0000 7f4c2543c640 1 -- start start 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c2543c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c2006ece0 0x7f4c20156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c2543c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c200cece0 0x7f4c201529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c2543c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c200d09f0 0x7f4c201f8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c2543c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2006d810 con 0x7f4c2006ece0 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c2543c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2006e7c0 con 0x7f4c200d09f0 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c2543c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4c2006d990 con 0x7f4c200cece0 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c1effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c2006ece0 0x7f4c20156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c1effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c2006ece0 0x7f4c20156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35432/0 (socket says 172.21.15.44:35432) 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c1effd640 1 -- 172.21.15.44:0/2118308031 learned_addr learned my addr 172.21.15.44:0/2118308031 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c1e7fc640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c200cece0 0x7f4c201529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c1f7fe640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c200d09f0 0x7f4c201f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c1effd640 1 -- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c200cece0 msgr2=0x7f4c201529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c1effd640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c200cece0 0x7f4c201529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c1effd640 1 -- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c200d09f0 msgr2=0x7f4c201f8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:46.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c1effd640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c200d09f0 0x7f4c201f8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:46.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.945+0000 7f4c1effd640 1 -- 172.21.15.44:0/2118308031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c201571d0 con 0x7f4c2006ece0 2024-04-02T13:24:46.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.946+0000 7f4c1effd640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c2006ece0 0x7f4c20156c90 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800d800 tx=0x7f4c1800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:46.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.946+0000 7f4c177fe640 1 -- 172.21.15.44:0/2118308031 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c18018020 con 0x7f4c2006ece0 2024-04-02T13:24:46.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.946+0000 7f4c177fe640 1 -- 172.21.15.44:0/2118308031 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c180096a0 con 0x7f4c2006ece0 2024-04-02T13:24:46.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.946+0000 7f4c2543c640 1 -- 172.21.15.44:0/2118308031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c201577c0 con 0x7f4c2006ece0 2024-04-02T13:24:46.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.946+0000 7f4c177fe640 1 -- 172.21.15.44:0/2118308031 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c18009a70 con 0x7f4c2006ece0 2024-04-02T13:24:46.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.946+0000 7f4c2543c640 1 -- 172.21.15.44:0/2118308031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c201f7c00 con 0x7f4c2006ece0 2024-04-02T13:24:46.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.947+0000 7f4c2543c640 1 -- 172.21.15.44:0/2118308031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c20074f20 con 0x7f4c2006ece0 2024-04-02T13:24:46.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.947+0000 7f4c177fe640 1 -- 172.21.15.44:0/2118308031 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4c1801b020 con 0x7f4c2006ece0 2024-04-02T13:24:46.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.948+0000 7f4c177fe640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4c0006c8f0 0x7f4c0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:46.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.948+0000 7f4c177fe640 1 -- 172.21.15.44:0/2118308031 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5335+0+0 (secure 0 0 0) 0x7f4c1808bb40 con 0x7f4c2006ece0 2024-04-02T13:24:46.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.948+0000 7f4c1e7fc640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4c0006c8f0 0x7f4c0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:46.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.949+0000 7f4c1e7fc640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4c0006c8f0 0x7f4c0006edb0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4c08006fd0 tx=0x7f4c08008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:46.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:46.952+0000 7f4c177fe640 1 -- 172.21.15.44:0/2118308031 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4c18022030 con 0x7f4c2006ece0 2024-04-02T13:24:47.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:47.209+0000 7f4c2543c640 1 -- 172.21.15.44:0/2118308031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]} v 0) -- 0x7f4c200684f0 con 0x7f4c2006ece0 2024-04-02T13:24:48.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.204+0000 7f4c177fe640 1 -- 172.21.15.44:0/2118308031 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]}]=0 v232) ==== 169+0+0 (secure 0 0 0) 0x7f4c1805a170 con 0x7f4c2006ece0 2024-04-02T13:24:48.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.206+0000 7f4c2543c640 1 -- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4c0006c8f0 msgr2=0x7f4c0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:48.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.206+0000 7f4c2543c640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4c0006c8f0 0x7f4c0006edb0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4c08006fd0 tx=0x7f4c08008040 comp rx=0 tx=0).stop 2024-04-02T13:24:48.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.206+0000 7f4c2543c640 1 -- 172.21.15.44:0/2118308031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c2006ece0 msgr2=0x7f4c20156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:48.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.206+0000 7f4c2543c640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c2006ece0 0x7f4c20156c90 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800d800 tx=0x7f4c1800dcc0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.206+0000 7f4c2543c640 1 -- 172.21.15.44:0/2118308031 shutdown_connections 2024-04-02T13:24:48.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.206+0000 7f4c2543c640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4c0006c8f0 0x7f4c0006edb0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.206+0000 7f4c2543c640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c200d09f0 0x7f4c201f8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.206+0000 7f4c2543c640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c200cece0 0x7f4c201529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.206+0000 7f4c2543c640 1 --2- 172.21.15.44:0/2118308031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c2006ece0 0x7f4c20156c90 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.206+0000 7f4c2543c640 1 -- 172.21.15.44:0/2118308031 >> 172.21.15.44:0/2118308031 conn(0x7f4c200cfcd0 msgr2=0x7f4c200ad9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:48.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.207+0000 7f4c2543c640 1 -- 172.21.15.44:0/2118308031 shutdown_connections 2024-04-02T13:24:48.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.207+0000 7f4c2543c640 1 -- 172.21.15.44:0/2118308031 wait complete. 2024-04-02T13:24:48.223 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T13:24:48.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.331+0000 7fcc68847640 1 Processor -- start 2024-04-02T13:24:48.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.331+0000 7fcc68847640 1 -- start start 2024-04-02T13:24:48.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc68847640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcc640d2550 0x7fcc6415fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:48.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc68847640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc640d2930 0x7fcc6415f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:48.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc68847640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcc640cf870 0x7fcc640cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:48.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc68847640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fcc640d1e80 con 0x7fcc640cf870 2024-04-02T13:24:48.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc68847640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcc640d2160 con 0x7fcc640d2550 2024-04-02T13:24:48.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc68847640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcc640ced90 con 0x7fcc640d2930 2024-04-02T13:24:48.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc62d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcc640cf870 0x7fcc640cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:48.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc62d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcc640cf870 0x7fcc640cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35446/0 (socket says 172.21.15.44:35446) 2024-04-02T13:24:48.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc62d9d640 1 -- 172.21.15.44:0/1236655036 learned_addr learned my addr 172.21.15.44:0/1236655036 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:48.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc6259c640 1 --2- 172.21.15.44:0/1236655036 >> v2:172.21.15.69:3300/0 conn(0x7fcc640d2550 0x7fcc6415fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:48.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc59d9b640 1 --2- 172.21.15.44:0/1236655036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc640d2930 0x7fcc6415f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:48.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc62d9d640 1 -- 172.21.15.44:0/1236655036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc640d2930 msgr2=0x7fcc6415f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:48.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc62d9d640 1 --2- 172.21.15.44:0/1236655036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc640d2930 0x7fcc6415f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc62d9d640 1 -- 172.21.15.44:0/1236655036 >> v2:172.21.15.69:3300/0 conn(0x7fcc640d2550 msgr2=0x7fcc6415fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:48.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc62d9d640 1 --2- 172.21.15.44:0/1236655036 >> v2:172.21.15.69:3300/0 conn(0x7fcc640d2550 0x7fcc6415fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.332+0000 7fcc62d9d640 1 -- 172.21.15.44:0/1236655036 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc640cf070 con 0x7fcc640cf870 2024-04-02T13:24:48.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.333+0000 7fcc62d9d640 1 --2- 172.21.15.44:0/1236655036 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc640cf870 0x7fcc640cfd00 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fcc6408c610 tx=0x7fcc5c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9266fa1903faa79 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:48.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.333+0000 7fcc61d9b640 1 -- 172.21.15.44:0/1236655036 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c02ca70 con 0x7fcc640cf870 2024-04-02T13:24:48.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.333+0000 7fcc61d9b640 1 -- 172.21.15.44:0/1236655036 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc5c02cc10 con 0x7fcc640cf870 2024-04-02T13:24:48.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.333+0000 7fcc61d9b640 1 -- 172.21.15.44:0/1236655036 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c0343f0 con 0x7fcc640cf870 2024-04-02T13:24:48.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.333+0000 7fcc68847640 1 -- 172.21.15.44:0/1236655036 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc640cf870 msgr2=0x7fcc640cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:48.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.333+0000 7fcc68847640 1 --2- 172.21.15.44:0/1236655036 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc640cf870 0x7fcc640cfd00 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fcc6408c610 tx=0x7fcc5c02bed0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.334+0000 7fcc68847640 1 -- 172.21.15.44:0/1236655036 shutdown_connections 2024-04-02T13:24:48.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.334+0000 7fcc68847640 1 --2- 172.21.15.44:0/1236655036 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc640cf870 0x7fcc640cfd00 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.334+0000 7fcc68847640 1 --2- 172.21.15.44:0/1236655036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc640d2930 0x7fcc6415f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.334+0000 7fcc68847640 1 --2- 172.21.15.44:0/1236655036 >> v2:172.21.15.69:3300/0 conn(0x7fcc640d2550 0x7fcc6415fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.334+0000 7fcc68847640 1 -- 172.21.15.44:0/1236655036 >> 172.21.15.44:0/1236655036 conn(0x7fcc6409d3c0 msgr2=0x7fcc640d4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:48.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.334+0000 7fcc68847640 1 -- 172.21.15.44:0/1236655036 shutdown_connections 2024-04-02T13:24:48.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.334+0000 7fcc68847640 1 -- 172.21.15.44:0/1236655036 wait complete. 2024-04-02T13:24:48.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.335+0000 7fcc68847640 1 Processor -- start 2024-04-02T13:24:48.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.335+0000 7fcc68847640 1 -- start start 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc68847640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc640cf870 0x7fcc6414c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc68847640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc640d2550 0x7fcc641e7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc68847640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc640d2930 0x7fcc641600f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc68847640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcc640d2d70 con 0x7fcc640d2550 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc68847640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcc640d1b70 con 0x7fcc640d2930 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc68847640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcc641f4730 con 0x7fcc640cf870 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc59d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc640d2550 0x7fcc641e7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc59d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc640d2550 0x7fcc641e7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35448/0 (socket says 172.21.15.44:35448) 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc59d9b640 1 -- 172.21.15.44:0/3656789601 learned_addr learned my addr 172.21.15.44:0/3656789601 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc62d9d640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc640d2930 0x7fcc641600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc59d9b640 1 -- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc640cf870 msgr2=0x7fcc6414c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc59d9b640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc640cf870 0x7fcc6414c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc59d9b640 1 -- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc640d2930 msgr2=0x7fcc641600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc59d9b640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc640d2930 0x7fcc641600f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc59d9b640 1 -- 172.21.15.44:0/3656789601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc641f4a10 con 0x7fcc640d2550 2024-04-02T13:24:48.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc62d9d640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc640d2930 0x7fcc641600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:48.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.336+0000 7fcc59d9b640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc640d2550 0x7fcc641e7bb0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00d800 tx=0x7fcc4c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:48.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.337+0000 7fcc5b7fe640 1 -- 172.21.15.44:0/3656789601 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c00b6c0 con 0x7fcc640d2550 2024-04-02T13:24:48.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.337+0000 7fcc5b7fe640 1 -- 172.21.15.44:0/3656789601 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc4c019070 con 0x7fcc640d2550 2024-04-02T13:24:48.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.337+0000 7fcc68847640 1 -- 172.21.15.44:0/3656789601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc641f4cf0 con 0x7fcc640d2550 2024-04-02T13:24:48.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.337+0000 7fcc5b7fe640 1 -- 172.21.15.44:0/3656789601 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c00ba80 con 0x7fcc640d2550 2024-04-02T13:24:48.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.337+0000 7fcc68847640 1 -- 172.21.15.44:0/3656789601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc6414ca00 con 0x7fcc640d2550 2024-04-02T13:24:48.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.338+0000 7fcc68847640 1 -- 172.21.15.44:0/3656789601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc640684f0 con 0x7fcc640d2550 2024-04-02T13:24:48.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.338+0000 7fcc5b7fe640 1 -- 172.21.15.44:0/3656789601 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcc4c00bc20 con 0x7fcc640d2550 2024-04-02T13:24:48.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.338+0000 7fcc5b7fe640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcc4406c8f0 0x7fcc4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:48.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.338+0000 7fcc5b7fe640 1 -- 172.21.15.44:0/3656789601 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5335+0+0 (secure 0 0 0) 0x7fcc4c08be40 con 0x7fcc640d2550 2024-04-02T13:24:48.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.339+0000 7fcc6259c640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcc4406c8f0 0x7fcc4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:48.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.339+0000 7fcc6259c640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcc4406c8f0 0x7fcc4406edb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcc5400b7f0 tx=0x7fcc54008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:48.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.343+0000 7fcc5b7fe640 1 -- 172.21.15.44:0/3656789601 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fcc4c01c360 con 0x7fcc640d2550 2024-04-02T13:24:48.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.526+0000 7fcc68847640 1 -- 172.21.15.44:0/3656789601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fcc640b6800 con 0x7fcc640d2550 2024-04-02T13:24:48.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.531+0000 7fcc5b7fe640 1 -- 172.21.15.44:0/3656789601 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7fcc4c05a470 con 0x7fcc640d2550 2024-04-02T13:24:48.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.533+0000 7fcc68847640 1 -- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcc4406c8f0 msgr2=0x7fcc4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:48.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.533+0000 7fcc68847640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcc4406c8f0 0x7fcc4406edb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcc5400b7f0 tx=0x7fcc54008040 comp rx=0 tx=0).stop 2024-04-02T13:24:48.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.533+0000 7fcc68847640 1 -- 172.21.15.44:0/3656789601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc640d2550 msgr2=0x7fcc641e7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:48.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.533+0000 7fcc68847640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc640d2550 0x7fcc641e7bb0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00d800 tx=0x7fcc4c00dcc0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.533+0000 7fcc68847640 1 -- 172.21.15.44:0/3656789601 shutdown_connections 2024-04-02T13:24:48.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.533+0000 7fcc68847640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcc4406c8f0 0x7fcc4406edb0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.533+0000 7fcc68847640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc640d2930 0x7fcc641600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.533+0000 7fcc68847640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc640d2550 0x7fcc641e7bb0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.533+0000 7fcc68847640 1 --2- 172.21.15.44:0/3656789601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc640cf870 0x7fcc6414c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.533+0000 7fcc68847640 1 -- 172.21.15.44:0/3656789601 >> 172.21.15.44:0/3656789601 conn(0x7fcc6409d3c0 msgr2=0x7fcc640b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:48.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.533+0000 7fcc68847640 1 -- 172.21.15.44:0/3656789601 shutdown_connections 2024-04-02T13:24:48.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.533+0000 7fcc68847640 1 -- 172.21.15.44:0/3656789601 wait complete. 2024-04-02T13:24:48.547 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:24:48.547 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:24:48.593 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:24:48.638 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:24:48.638 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:24:48.638 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:24:48.654 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:24:48.654 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:24:48.654 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:24:48.696 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:24:48.697 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:24:48.697 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:24:48.711 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:24:48.712 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:24:48.712 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:24:48.755 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:24:48.755 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T13:24:48.755 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:24:48.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.854+0000 7f059a767640 1 Processor -- start 2024-04-02T13:24:48.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.854+0000 7f059a767640 1 -- start start 2024-04-02T13:24:48.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.855+0000 7f059a767640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f05940cd3a0 0x7f05940ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:48.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.855+0000 7f059a767640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05940cef30 con 0x7f05941617e0 2024-04-02T13:24:48.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.855+0000 7f059a767640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05940cf210 con 0x7f05940cffd0 2024-04-02T13:24:48.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.855+0000 7f059a767640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f05940cf4f0 con 0x7f05940cd3a0 2024-04-02T13:24:48.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.855+0000 7f0598d04640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f05940cd3a0 0x7f05940ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:48.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.855+0000 7f0593fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f05941617e0 0x7f05940ccc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33506/0 (socket says 172.21.15.44:33506) 2024-04-02T13:24:48.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.855+0000 7f0593fff640 1 -- 172.21.15.44:0/4191843623 learned_addr learned my addr 172.21.15.44:0/4191843623 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:48.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.855+0000 7f0598d04640 1 -- 172.21.15.44:0/4191843623 >> v1:172.21.15.69:6789/0 conn(0x7f05940cffd0 legacy=0x7f05940ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:48.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.855+0000 7f0598d04640 1 -- 172.21.15.44:0/4191843623 >> v1:172.21.15.44:6789/0 conn(0x7f05941617e0 legacy=0x7f05940ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:48.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.855+0000 7f0592ffd640 1 -- 172.21.15.44:0/4191843623 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3639050659 0 0) 0x7f05940cef30 con 0x7f05941617e0 2024-04-02T13:24:48.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.855+0000 7f0598d04640 1 -- 172.21.15.44:0/4191843623 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05940cf7d0 con 0x7f05940cd3a0 2024-04-02T13:24:48.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.856+0000 7f0598d04640 1 --2- 172.21.15.44:0/4191843623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05940cd3a0 0x7f05940ce8c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f058c009d90 tx=0x7f058c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee96f25e35af458 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:48.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.856+0000 7f0592ffd640 1 -- 172.21.15.44:0/4191843623 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c039070 con 0x7f05940cd3a0 2024-04-02T13:24:48.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.856+0000 7f0592ffd640 1 -- 172.21.15.44:0/4191843623 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f058c02cd00 con 0x7f05940cd3a0 2024-04-02T13:24:48.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.856+0000 7f0592ffd640 1 -- 172.21.15.44:0/4191843623 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c034660 con 0x7f05940cd3a0 2024-04-02T13:24:48.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.856+0000 7f059a767640 1 -- 172.21.15.44:0/4191843623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05940cd3a0 msgr2=0x7f05940ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:48.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.857+0000 7f059a767640 1 --2- 172.21.15.44:0/4191843623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05940cd3a0 0x7f05940ce8c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f058c009d90 tx=0x7f058c02bf50 comp rx=0 tx=0).stop 2024-04-02T13:24:48.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.857+0000 7f059a767640 1 -- 172.21.15.44:0/4191843623 shutdown_connections 2024-04-02T13:24:48.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.857+0000 7f059a767640 1 --2- 172.21.15.44:0/4191843623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05940cd3a0 0x7f05940ce8c0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.857+0000 7f059a767640 1 -- 172.21.15.44:0/4191843623 >> 172.21.15.44:0/4191843623 conn(0x7f05940b20e0 msgr2=0x7f05940b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:48.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.857+0000 7f059a767640 1 -- 172.21.15.44:0/4191843623 shutdown_connections 2024-04-02T13:24:48.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.857+0000 7f059a767640 1 -- 172.21.15.44:0/4191843623 wait complete. 2024-04-02T13:24:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.858+0000 7f059a767640 1 Processor -- start 2024-04-02T13:24:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.858+0000 7f059a767640 1 -- start start 2024-04-02T13:24:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f059a767640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05940cd3a0 0x7f05941f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f059a767640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05940cffd0 0x7f05940b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f059a767640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05941617e0 0x7f05940b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f059a767640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0594161d40 con 0x7f05940cffd0 2024-04-02T13:24:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f059a767640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0594161bc0 con 0x7f05941617e0 2024-04-02T13:24:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f059a767640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f05941629c0 con 0x7f05940cd3a0 2024-04-02T13:24:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f05937fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05940cffd0 0x7f05940b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f05937fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05940cffd0 0x7f05940b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35478/0 (socket says 172.21.15.44:35478) 2024-04-02T13:24:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f05937fe640 1 -- 172.21.15.44:0/3487500777 learned_addr learned my addr 172.21.15.44:0/3487500777 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f0598d04640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05941617e0 0x7f05940b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f0593fff640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05940cd3a0 0x7f05941f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f05937fe640 1 -- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05940cd3a0 msgr2=0x7f05941f41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f05937fe640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05940cd3a0 0x7f05941f41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f05937fe640 1 -- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05941617e0 msgr2=0x7f05940b08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f05937fe640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05941617e0 0x7f05940b08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.859+0000 7f05937fe640 1 -- 172.21.15.44:0/3487500777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05940c8610 con 0x7f05940cffd0 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.860+0000 7f05937fe640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05940cffd0 0x7f05940b31d0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f058400c400 tx=0x7f058400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.860+0000 7f0590ff9640 1 -- 172.21.15.44:0/3487500777 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0584017070 con 0x7f05940cffd0 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.860+0000 7f0590ff9640 1 -- 172.21.15.44:0/3487500777 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05840052d0 con 0x7f05940cffd0 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.860+0000 7f059a767640 1 -- 172.21.15.44:0/3487500777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05940c88d0 con 0x7f05940cffd0 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.860+0000 7f0590ff9640 1 -- 172.21.15.44:0/3487500777 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0584003830 con 0x7f05940cffd0 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.860+0000 7f059a767640 1 -- 172.21.15.44:0/3487500777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05940c9170 con 0x7f05940cffd0 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.861+0000 7f059a767640 1 -- 172.21.15.44:0/3487500777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05940c9370 con 0x7f05940cffd0 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.861+0000 7f0590ff9640 1 -- 172.21.15.44:0/3487500777 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f058401a070 con 0x7f05940cffd0 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.861+0000 7f0590ff9640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f058006c8f0 0x7f058006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.862+0000 7f0590ff9640 1 -- 172.21.15.44:0/3487500777 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5335+0+0 (secure 0 0 0) 0x7f058408acb0 con 0x7f05940cffd0 2024-04-02T13:24:48.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.862+0000 7f0593fff640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f058006c8f0 0x7f058006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:48.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.862+0000 7f0593fff640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f058006c8f0 0x7f058006edb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0588007c30 tx=0x7f0588007b80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:48.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:48.866+0000 7f0590ff9640 1 -- 172.21.15.44:0/3487500777 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f05940c9370 con 0x7f05940cffd0 2024-04-02T13:24:49.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.089+0000 7f059a767640 1 -- 172.21.15.44:0/3487500777 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05940684f0 con 0x7f05940cffd0 2024-04-02T13:24:49.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.090+0000 7f0590ff9640 1 -- 172.21.15.44:0/3487500777 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+3866 (secure 0 0 0) 0x7f0584058c00 con 0x7f05940cffd0 2024-04-02T13:24:49.093 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:49.093 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":39,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1535287484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1535287484},{"type":"v1","addr":"172.21.15.69:6836","nonce":1535287484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6429,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2367107002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2367107002},{"type":"v1","addr":"172.21.15.44:6833","nonce":2367107002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/553542317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":553542317},{"type":"v1","addr":"172.21.15.69:6837","nonce":553542317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3827834671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3827834671},{"type":"v1","addr":"172.21.15.44:6835","nonce":3827834671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[]} 2024-04-02T13:24:49.093 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 39 2024-04-02T13:24:49.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.091+0000 7f059a767640 1 -- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f058006c8f0 msgr2=0x7f058006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:49.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.091+0000 7f059a767640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f058006c8f0 0x7f058006edb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0588007c30 tx=0x7f0588007b80 comp rx=0 tx=0).stop 2024-04-02T13:24:49.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.091+0000 7f059a767640 1 -- 172.21.15.44:0/3487500777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05940cffd0 msgr2=0x7f05940b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:49.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.091+0000 7f059a767640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05940cffd0 0x7f05940b31d0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f058400c400 tx=0x7f058400c8c0 comp rx=0 tx=0).stop 2024-04-02T13:24:49.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.091+0000 7f059a767640 1 -- 172.21.15.44:0/3487500777 shutdown_connections 2024-04-02T13:24:49.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.091+0000 7f059a767640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f058006c8f0 0x7f058006edb0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:49.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.091+0000 7f059a767640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05941617e0 0x7f05940b08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:49.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.091+0000 7f059a767640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05940cffd0 0x7f05940b31d0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:49.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.091+0000 7f059a767640 1 --2- 172.21.15.44:0/3487500777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05940cd3a0 0x7f05941f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:49.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.091+0000 7f059a767640 1 -- 172.21.15.44:0/3487500777 >> 172.21.15.44:0/3487500777 conn(0x7f05940b20e0 msgr2=0x7f059406ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:49.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.092+0000 7f059a767640 1 -- 172.21.15.44:0/3487500777 shutdown_connections 2024-04-02T13:24:49.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.092+0000 7f059a767640 1 -- 172.21.15.44:0/3487500777 wait complete. 2024-04-02T13:24:49.105 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T13:24:49.106 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T13:24:49.106 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T13:24:49.106 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:24:49.106 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T13:24:49.106 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T13:24:49.106 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T13:24:49.107 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:24:49.107 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T13:24:49.107 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T13:24:49.107 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T13:24:49.107 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:24:49.107 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T13:24:49.107 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T13:24:49.107 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T13:24:49.107 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:24:49.115 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T13:24:49.116 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T13:24:49.116 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T13:24:49.116 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T13:24:49.119 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T13:24:49.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T13:24:49.120 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T13:24:49.120 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T13:24:49.159 INFO:tasks.ceph.mds.a:Started 2024-04-02T13:24:49.160 DEBUG:teuthology.parallel:result is None 2024-04-02T13:24:49.161 INFO:tasks.ceph.mds.c:Started 2024-04-02T13:24:49.161 DEBUG:teuthology.parallel:result is None 2024-04-02T13:24:49.162 INFO:tasks.ceph.mds.b:Started 2024-04-02T13:24:49.162 DEBUG:teuthology.parallel:result is None 2024-04-02T13:24:49.162 INFO:tasks.ceph.mds.d:Started 2024-04-02T13:24:49.163 DEBUG:teuthology.parallel:result is None 2024-04-02T13:24:49.163 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T13:24:49.236 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:24:49.237 INFO:tasks.ceph.mds.c.smithi044.stdout:starting mds.c at 2024-04-02T13:24:49.245 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-04-02T13:24:49.245 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-04-02T13:24:49.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.268+0000 7f0a3daba640 1 Processor -- start 2024-04-02T13:24:49.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.268+0000 7f0a3daba640 1 -- start start 2024-04-02T13:24:49.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.269+0000 7f0a3daba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a380d0a30 0x7f0a3806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:49.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.269+0000 7f0a3daba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0a380d0e10 0x7f0a380cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:49.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.269+0000 7f0a3daba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a3806c1d0 con 0x7f0a380cfe30 2024-04-02T13:24:49.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.269+0000 7f0a3daba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0a3806cb00 con 0x7f0a380d0e10 2024-04-02T13:24:49.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.269+0000 7f0a3daba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0a3806c4b0 con 0x7f0a380d0a30 2024-04-02T13:24:49.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.269+0000 7f0a37fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0a380cfe30 0x7f0a380d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33524/0 (socket says 172.21.15.44:33524) 2024-04-02T13:24:49.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.269+0000 7f0a37fff640 1 -- 172.21.15.44:0/1736617369 learned_addr learned my addr 172.21.15.44:0/1736617369 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:49.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.269+0000 7f0a377fe640 1 --2- 172.21.15.44:0/1736617369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a380d0a30 0x7f0a3806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:49.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.269+0000 7f0a36ffd640 1 --2- 172.21.15.44:0/1736617369 >> v2:172.21.15.69:3300/0 conn(0x7f0a380d0e10 0x7f0a380cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:49.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.270+0000 7f0a367fc640 1 -- 172.21.15.44:0/1736617369 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 249987381 0 0) 0x7f0a3806c1d0 con 0x7f0a380cfe30 2024-04-02T13:24:49.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.270+0000 7f0a367fc640 1 -- 172.21.15.44:0/1736617369 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a20003520 con 0x7f0a380cfe30 2024-04-02T13:24:49.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.270+0000 7f0a377fe640 1 -- 172.21.15.44:0/1736617369 >> v2:172.21.15.69:3300/0 conn(0x7f0a380d0e10 msgr2=0x7f0a380cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:49.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.270+0000 7f0a377fe640 1 --2- 172.21.15.44:0/1736617369 >> v2:172.21.15.69:3300/0 conn(0x7f0a380d0e10 0x7f0a380cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:49.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.270+0000 7f0a377fe640 1 -- 172.21.15.44:0/1736617369 >> v1:172.21.15.44:6789/0 conn(0x7f0a380cfe30 legacy=0x7f0a380d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:49.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.270+0000 7f0a377fe640 1 -- 172.21.15.44:0/1736617369 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3806c790 con 0x7f0a380d0a30 2024-04-02T13:24:49.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.270+0000 7f0a377fe640 1 --2- 172.21.15.44:0/1736617369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a380d0a30 0x7f0a3806cf90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0a280097a0 tx=0x7f0a2802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d62f301d8f466d37 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:49.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.271+0000 7f0a367fc640 1 -- 172.21.15.44:0/1736617369 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a280315a0 con 0x7f0a380d0a30 2024-04-02T13:24:49.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.271+0000 7f0a367fc640 1 -- 172.21.15.44:0/1736617369 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a28037070 con 0x7f0a380d0a30 2024-04-02T13:24:49.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.271+0000 7f0a3daba640 1 -- 172.21.15.44:0/1736617369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a380d0a30 msgr2=0x7f0a3806cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:49.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.271+0000 7f0a3daba640 1 --2- 172.21.15.44:0/1736617369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a380d0a30 0x7f0a3806cf90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0a280097a0 tx=0x7f0a2802b840 comp rx=0 tx=0).stop 2024-04-02T13:24:49.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.271+0000 7f0a3daba640 1 -- 172.21.15.44:0/1736617369 shutdown_connections 2024-04-02T13:24:49.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.271+0000 7f0a3daba640 1 --2- 172.21.15.44:0/1736617369 >> v2:172.21.15.69:3300/0 conn(0x7f0a380d0e10 0x7f0a380cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:49.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.271+0000 7f0a3daba640 1 --2- 172.21.15.44:0/1736617369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a380d0a30 0x7f0a3806cf90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:49.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.271+0000 7f0a3daba640 1 -- 172.21.15.44:0/1736617369 >> 172.21.15.44:0/1736617369 conn(0x7f0a3809d470 msgr2=0x7f0a380d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:49.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.271+0000 7f0a3daba640 1 -- 172.21.15.44:0/1736617369 shutdown_connections 2024-04-02T13:24:49.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.271+0000 7f0a3daba640 1 -- 172.21.15.44:0/1736617369 wait complete. 2024-04-02T13:24:49.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.273+0000 7f0a3daba640 1 Processor -- start 2024-04-02T13:24:49.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.273+0000 7f0a3daba640 1 -- start start 2024-04-02T13:24:49.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a3daba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a380cfe30 0x7f0a381595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:49.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a3daba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a380d0a30 0x7f0a38158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:49.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a3daba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a380d0e10 0x7f0a3815f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:49.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a3daba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3815e870 con 0x7f0a380cfe30 2024-04-02T13:24:49.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a3daba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3815e6f0 con 0x7f0a380d0a30 2024-04-02T13:24:49.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a3daba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0a380d4da0 con 0x7f0a380d0e10 2024-04-02T13:24:49.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a377fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a380cfe30 0x7f0a381595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:49.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a37fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a380d0e10 0x7f0a3815f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:49.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a36ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a380d0a30 0x7f0a38158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:49.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a37fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a380d0e10 0x7f0a3815f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:35384/0 (socket says 172.21.15.44:35384) 2024-04-02T13:24:49.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a37fff640 1 -- 172.21.15.44:0/212824175 learned_addr learned my addr 172.21.15.44:0/212824175 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:49.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a377fe640 1 -- 172.21.15.44:0/212824175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a380d0e10 msgr2=0x7f0a3815f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:49.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a377fe640 1 --2- 172.21.15.44:0/212824175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a380d0e10 0x7f0a3815f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:49.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a377fe640 1 -- 172.21.15.44:0/212824175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a380d0a30 msgr2=0x7f0a38158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:49.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a377fe640 1 --2- 172.21.15.44:0/212824175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a380d0a30 0x7f0a38158f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:49.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a377fe640 1 -- 172.21.15.44:0/212824175 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a380b9620 con 0x7f0a380cfe30 2024-04-02T13:24:49.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a37fff640 1 --2- 172.21.15.44:0/212824175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a380d0e10 0x7f0a3815f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:24:49.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.274+0000 7f0a377fe640 1 --2- 172.21.15.44:0/212824175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a380cfe30 0x7f0a381595b0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f0a28033000 tx=0x7f0a280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:49.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.275+0000 7f0a1ffff640 1 -- 172.21.15.44:0/212824175 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a28006e10 con 0x7f0a380cfe30 2024-04-02T13:24:49.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.275+0000 7f0a1ffff640 1 -- 172.21.15.44:0/212824175 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a280317c0 con 0x7f0a380cfe30 2024-04-02T13:24:49.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.275+0000 7f0a3daba640 1 -- 172.21.15.44:0/212824175 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a380b9900 con 0x7f0a380cfe30 2024-04-02T13:24:49.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.275+0000 7f0a1ffff640 1 -- 172.21.15.44:0/212824175 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a28031bb0 con 0x7f0a380cfe30 2024-04-02T13:24:49.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.275+0000 7f0a3daba640 1 -- 172.21.15.44:0/212824175 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a380b4ff0 con 0x7f0a380cfe30 2024-04-02T13:24:49.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.276+0000 7f0a1ffff640 1 -- 172.21.15.44:0/212824175 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0a28031d50 con 0x7f0a380cfe30 2024-04-02T13:24:49.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.276+0000 7f0a3daba640 1 -- 172.21.15.44:0/212824175 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a380684f0 con 0x7f0a380cfe30 2024-04-02T13:24:49.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.277+0000 7f0a1ffff640 1 --2- 172.21.15.44:0/212824175 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a2406c8f0 0x7f0a2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:49.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.277+0000 7f0a1ffff640 1 -- 172.21.15.44:0/212824175 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5335+0+0 (secure 0 0 0) 0x7f0a2802f020 con 0x7f0a380cfe30 2024-04-02T13:24:49.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.277+0000 7f0a36ffd640 1 --2- 172.21.15.44:0/212824175 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a2406c8f0 0x7f0a2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:49.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.277+0000 7f0a36ffd640 1 --2- 172.21.15.44:0/212824175 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a2406c8f0 0x7f0a2406edb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0a38160d10 tx=0x7f0a2c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:49.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.281+0000 7f0a1ffff640 1 -- 172.21.15.44:0/212824175 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0a2807bde0 con 0x7f0a380cfe30 2024-04-02T13:24:49.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:49.482+0000 7f0a3daba640 1 -- 172.21.15.44:0/212824175 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f0a3815de20 con 0x7f0a380cfe30 2024-04-02T13:24:50.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.544+0000 7f0a1ffff640 1 -- 172.21.15.44:0/212824175 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v75) ==== 97+0+0 (secure 0 0 0) 0x7f0a280800e0 con 0x7f0a380cfe30 2024-04-02T13:24:50.547 INFO:teuthology.orchestra.run.smithi044.stderr: removed all blocklist entries 2024-04-02T13:24:50.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.547+0000 7f0a3daba640 1 -- 172.21.15.44:0/212824175 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a2406c8f0 msgr2=0x7f0a2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:50.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.547+0000 7f0a3daba640 1 --2- 172.21.15.44:0/212824175 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a2406c8f0 0x7f0a2406edb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0a38160d10 tx=0x7f0a2c009190 comp rx=0 tx=0).stop 2024-04-02T13:24:50.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.547+0000 7f0a3daba640 1 -- 172.21.15.44:0/212824175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a380cfe30 msgr2=0x7f0a381595b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:50.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.547+0000 7f0a3daba640 1 --2- 172.21.15.44:0/212824175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a380cfe30 0x7f0a381595b0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f0a28033000 tx=0x7f0a280046c0 comp rx=0 tx=0).stop 2024-04-02T13:24:50.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.547+0000 7f0a3daba640 1 -- 172.21.15.44:0/212824175 shutdown_connections 2024-04-02T13:24:50.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.547+0000 7f0a3daba640 1 --2- 172.21.15.44:0/212824175 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a2406c8f0 0x7f0a2406edb0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:50.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.547+0000 7f0a3daba640 1 --2- 172.21.15.44:0/212824175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a380d0e10 0x7f0a3815f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:50.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.547+0000 7f0a3daba640 1 --2- 172.21.15.44:0/212824175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a380d0a30 0x7f0a38158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:50.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.547+0000 7f0a3daba640 1 --2- 172.21.15.44:0/212824175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a380cfe30 0x7f0a381595b0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:50.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.547+0000 7f0a3daba640 1 -- 172.21.15.44:0/212824175 >> 172.21.15.44:0/212824175 conn(0x7f0a3809d470 msgr2=0x7f0a380cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:50.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.547+0000 7f0a3daba640 1 -- 172.21.15.44:0/212824175 shutdown_connections 2024-04-02T13:24:50.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.547+0000 7f0a3daba640 1 -- 172.21.15.44:0/212824175 wait complete. 2024-04-02T13:24:50.563 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T13:24:50.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.668+0000 7fdb08e47640 1 Processor -- start 2024-04-02T13:24:50.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.668+0000 7fdb08e47640 1 -- start start 2024-04-02T13:24:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.669+0000 7fdb08e47640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdb0415e0f0 0x7fdb040d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.669+0000 7fdb08e47640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb0406dfe0 con 0x7fdb040d0630 2024-04-02T13:24:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.669+0000 7fdb08e47640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb0406e2c0 con 0x7fdb0406bbe0 2024-04-02T13:24:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.669+0000 7fdb08e47640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fdb0406e5a0 con 0x7fdb0415e0f0 2024-04-02T13:24:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.670+0000 7fdb0259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdb0415e0f0 0x7fdb040d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.670+0000 7fdb01d9b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fdb040d0630 0x7fdb0406b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33526/0 (socket says 172.21.15.44:33526) 2024-04-02T13:24:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.670+0000 7fdb01d9b640 1 -- 172.21.15.44:0/654336416 learned_addr learned my addr 172.21.15.44:0/654336416 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:50.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.670+0000 7fdb0259c640 1 -- 172.21.15.44:0/654336416 >> v1:172.21.15.69:6789/0 conn(0x7fdb0406bbe0 legacy=0x7fdb040add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:50.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.670+0000 7fdb0259c640 1 -- 172.21.15.44:0/654336416 >> v1:172.21.15.44:6789/0 conn(0x7fdb040d0630 legacy=0x7fdb0406b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:50.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.670+0000 7fdb0159a640 1 -- 172.21.15.44:0/654336416 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1615548871 0 0) 0x7fdb0406dfe0 con 0x7fdb040d0630 2024-04-02T13:24:50.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.670+0000 7fdb0259c640 1 -- 172.21.15.44:0/654336416 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0406e880 con 0x7fdb0415e0f0 2024-04-02T13:24:50.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.671+0000 7fdb0259c640 1 --2- 172.21.15.44:0/654336416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb0415e0f0 0x7fdb040d00f0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009d90 tx=0x7fdaf402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b532f53f7450e8e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:50.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.671+0000 7fdb0159a640 1 -- 172.21.15.44:0/654336416 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4039070 con 0x7fdb0415e0f0 2024-04-02T13:24:50.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.671+0000 7fdb0159a640 1 -- 172.21.15.44:0/654336416 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaf402cd00 con 0x7fdb0415e0f0 2024-04-02T13:24:50.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.671+0000 7fdb0159a640 1 -- 172.21.15.44:0/654336416 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4034660 con 0x7fdb0415e0f0 2024-04-02T13:24:50.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.671+0000 7fdb08e47640 1 -- 172.21.15.44:0/654336416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb0415e0f0 msgr2=0x7fdb040d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:50.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.671+0000 7fdb08e47640 1 --2- 172.21.15.44:0/654336416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb0415e0f0 0x7fdb040d00f0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009d90 tx=0x7fdaf402bf50 comp rx=0 tx=0).stop 2024-04-02T13:24:50.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.671+0000 7fdb08e47640 1 -- 172.21.15.44:0/654336416 shutdown_connections 2024-04-02T13:24:50.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.671+0000 7fdb08e47640 1 --2- 172.21.15.44:0/654336416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb0415e0f0 0x7fdb040d00f0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:50.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.671+0000 7fdb08e47640 1 -- 172.21.15.44:0/654336416 >> 172.21.15.44:0/654336416 conn(0x7fdb040b2980 msgr2=0x7fdb040b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:50.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.671+0000 7fdb08e47640 1 -- 172.21.15.44:0/654336416 shutdown_connections 2024-04-02T13:24:50.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.671+0000 7fdb08e47640 1 -- 172.21.15.44:0/654336416 wait complete. 2024-04-02T13:24:50.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.673+0000 7fdb08e47640 1 Processor -- start 2024-04-02T13:24:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.673+0000 7fdb08e47640 1 -- start start 2024-04-02T13:24:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.673+0000 7fdb08e47640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb0406bbe0 0x7fdb041500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.673+0000 7fdb08e47640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb040d0630 0x7fdb041e9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.673+0000 7fdb08e47640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb0415e0f0 0x7fdb040cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.673+0000 7fdb08e47640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0415e4d0 con 0x7fdb0415e0f0 2024-04-02T13:24:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.673+0000 7fdb08e47640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0406eda0 con 0x7fdb0406bbe0 2024-04-02T13:24:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.673+0000 7fdb08e47640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdb0415e650 con 0x7fdb040d0630 2024-04-02T13:24:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdb02d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb0415e0f0 0x7fdb040cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdb02d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb0415e0f0 0x7fdb040cd910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35554/0 (socket says 172.21.15.44:35554) 2024-04-02T13:24:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdb02d9d640 1 -- 172.21.15.44:0/1315879432 learned_addr learned my addr 172.21.15.44:0/1315879432 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdb01d9b640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb040d0630 0x7fdb041e9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdb0259c640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb0406bbe0 0x7fdb041500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdb02d9d640 1 -- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb040d0630 msgr2=0x7fdb041e9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:50.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdb02d9d640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb040d0630 0x7fdb041e9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:50.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdb02d9d640 1 -- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb0406bbe0 msgr2=0x7fdb041500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:50.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdb02d9d640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb0406bbe0 0x7fdb041500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:50.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdb02d9d640 1 -- 172.21.15.44:0/1315879432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb041505e0 con 0x7fdb0415e0f0 2024-04-02T13:24:50.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdb02d9d640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb0415e0f0 0x7fdb040cd910 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fdafc00b490 tx=0x7fdafc00b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:50.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdafaffd640 1 -- 172.21.15.44:0/1315879432 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdafc015020 con 0x7fdb0415e0f0 2024-04-02T13:24:50.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdafaffd640 1 -- 172.21.15.44:0/1315879432 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdafc011850 con 0x7fdb0415e0f0 2024-04-02T13:24:50.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdb08e47640 1 -- 172.21.15.44:0/1315879432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb041e9fa0 con 0x7fdb0415e0f0 2024-04-02T13:24:50.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdafaffd640 1 -- 172.21.15.44:0/1315879432 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdafc011c40 con 0x7fdb0415e0f0 2024-04-02T13:24:50.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.674+0000 7fdb08e47640 1 -- 172.21.15.44:0/1315879432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb040c5860 con 0x7fdb0415e0f0 2024-04-02T13:24:50.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.675+0000 7fdb08e47640 1 -- 172.21.15.44:0/1315879432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb040c5a60 con 0x7fdb0415e0f0 2024-04-02T13:24:50.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.676+0000 7fdafaffd640 1 -- 172.21.15.44:0/1315879432 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdafc018020 con 0x7fdb0415e0f0 2024-04-02T13:24:50.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.676+0000 7fdafaffd640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdae406c8f0 0x7fdae406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:50.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.676+0000 7fdafaffd640 1 -- 172.21.15.44:0/1315879432 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5249+0+0 (secure 0 0 0) 0x7fdafc08b740 con 0x7fdb0415e0f0 2024-04-02T13:24:50.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.676+0000 7fdb0259c640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdae406c8f0 0x7fdae406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:50.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.677+0000 7fdb0259c640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdae406c8f0 0x7fdae406edb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4036000 tx=0x7fdaf4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:50.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.681+0000 7fdafaffd640 1 -- 172.21.15.44:0/1315879432 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fdb040c5a60 con 0x7fdb0415e0f0 2024-04-02T13:24:50.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.937+0000 7fdb08e47640 1 -- 172.21.15.44:0/1315879432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fdb040684f0 con 0x7fdb0415e0f0 2024-04-02T13:24:50.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.937+0000 7fdafaffd640 1 -- 172.21.15.44:0/1315879432 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) ==== 80+0+5304 (secure 0 0 0) 0x7fdafc01f020 con 0x7fdb0415e0f0 2024-04-02T13:24:50.940 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:50.940 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:24:50.941 INFO:teuthology.orchestra.run.smithi044.stdout: "auth_dump": [ 2024-04-02T13:24:50.941 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.941 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.a", 2024-04-02T13:24:50.941 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmVhZrMhAAklqx5uKCFA5kiqYwyLBKpg==", 2024-04-02T13:24:50.941 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.941 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:24:50.941 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:24:50.941 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:24:50.941 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.941 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.941 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.941 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.941 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.b", 2024-04-02T13:24:50.942 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmalQ+AxAA2SqH9CX/66OIXR7SDID97w==", 2024-04-02T13:24:50.942 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.942 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:24:50.942 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:24:50.942 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:24:50.942 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.942 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.942 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.942 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.942 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.c", 2024-04-02T13:24:50.942 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxm8BLKORAAjMIrmdQ/bXbPk2sAnYJWTw==", 2024-04-02T13:24:50.942 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.942 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.d", 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmNoaRChAAEqxWmBjBWG79Tkj6dU8FQg==", 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.943 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.0", 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDfBQxm4GP9KxAA/AmOunAfVaczAWRGVXvFVg==", 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.1", 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDiBQxmDL/nKRAADF2C30+IGAXL9XwDQDKCUA==", 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.944 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.2", 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDlBQxm1gOCKBAAuYxN9jNHuRYEqc8WrsYErQ==", 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.945 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.3", 2024-04-02T13:24:50.946 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDoBQxmLLFQJxAAFMCzphGU7TOt4nC0Nt+y+g==", 2024-04-02T13:24:50.946 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.946 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:24:50.946 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:24:50.946 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.946 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.946 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.946 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.946 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.4", 2024-04-02T13:24:50.946 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDuBQxmtk8QLhAAR4gm0irR6C3D2SkhFjvDKQ==", 2024-04-02T13:24:50.946 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.946 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:24:50.946 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.5", 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDxBQxm6Ee/LBAAX5h+9ML70ztw2QH/50OzOg==", 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.947 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.6", 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD0BQxm35hwKxAAvsgMmiXdLo+c7wlvb9RtVQ==", 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.7", 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD3BQxmYLBOKhAA1vuSOBDcFf2zT78eoTHtHQ==", 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.948 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.0", 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A==", 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow r", 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.1", 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxm3sYsGBAAeRCNbP5qiQ8YnRVsDoaaFQ==", 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:24:50.949 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.2", 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmKNS5FBAAKvsCNmuG+fI1D/vwu6MC5w==", 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.950 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.3", 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmslsSHhAAPsAfALW7RimJDyXdOa2wbQ==", 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.admin", 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmq+VcFRAA9CgeH75Q0sXf27cZdJthJg==", 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:24:50.951 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.x", 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmR5ICLxAAtQeg/gq0OAq6e0mroBZVxQ==", 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.y", 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmxoquKhAAE5fMnJIuCy5hGJ1aEAx8Vg==", 2024-04-02T13:24:50.952 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:24:50.953 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:24:50.953 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:24:50.953 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:24:50.953 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.953 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:50.953 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-02T13:24:50.953 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:24:50.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.939+0000 7fdb08e47640 1 -- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdae406c8f0 msgr2=0x7fdae406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:50.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.939+0000 7fdb08e47640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdae406c8f0 0x7fdae406edb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4036000 tx=0x7fdaf4030000 comp rx=0 tx=0).stop 2024-04-02T13:24:50.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.939+0000 7fdb08e47640 1 -- 172.21.15.44:0/1315879432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb0415e0f0 msgr2=0x7fdb040cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:50.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.939+0000 7fdb08e47640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb0415e0f0 0x7fdb040cd910 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fdafc00b490 tx=0x7fdafc00b950 comp rx=0 tx=0).stop 2024-04-02T13:24:50.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.939+0000 7fdb08e47640 1 -- 172.21.15.44:0/1315879432 shutdown_connections 2024-04-02T13:24:50.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.939+0000 7fdb08e47640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdae406c8f0 0x7fdae406edb0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:50.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.939+0000 7fdb08e47640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb0415e0f0 0x7fdb040cd910 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:50.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.939+0000 7fdb08e47640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb040d0630 0x7fdb041e9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:50.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.939+0000 7fdb08e47640 1 --2- 172.21.15.44:0/1315879432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb0406bbe0 0x7fdb041500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:50.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.939+0000 7fdb08e47640 1 -- 172.21.15.44:0/1315879432 >> 172.21.15.44:0/1315879432 conn(0x7fdb040b2980 msgr2=0x7fdb040b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:50.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.939+0000 7fdb08e47640 1 -- 172.21.15.44:0/1315879432 shutdown_connections 2024-04-02T13:24:50.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:50.939+0000 7fdb08e47640 1 -- 172.21.15.44:0/1315879432 wait complete. 2024-04-02T13:24:50.956 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:24:51.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.054+0000 7f01a5047640 1 Processor -- start 2024-04-02T13:24:51.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.054+0000 7f01a5047640 1 -- start start 2024-04-02T13:24:51.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.055+0000 7f01a5047640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01a00d2040 0x7f01a00d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.055+0000 7f01a5047640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01a00d0000 con 0x7f01a00d0a30 2024-04-02T13:24:51.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.055+0000 7f01a5047640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01a006c1d0 con 0x7f01a006cf90 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.055+0000 7f01a5047640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f01a006cb00 con 0x7f01a00d2040 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.055+0000 7f019ed9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f01a00d0a30 0x7f01a00d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33534/0 (socket says 172.21.15.44:33534) 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.055+0000 7f019ed9d640 1 -- 172.21.15.44:0/4127289955 learned_addr learned my addr 172.21.15.44:0/4127289955 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.055+0000 7f019f59e640 1 --2- 172.21.15.44:0/4127289955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01a00d2040 0x7f01a00d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.055+0000 7f019dd9b640 1 -- 172.21.15.44:0/4127289955 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3109815843 0 0) 0x7f01a00d0000 con 0x7f01a00d0a30 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.055+0000 7f019dd9b640 1 -- 172.21.15.44:0/4127289955 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f018c003520 con 0x7f01a00d0a30 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.055+0000 7f019dd9b640 1 -- 172.21.15.44:0/4127289955 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 993067123 0 0) 0x7f01a006c1d0 con 0x7f01a006cf90 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.056+0000 7f019dd9b640 1 -- 172.21.15.44:0/4127289955 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01a00d0000 con 0x7f01a006cf90 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.056+0000 7f019f59e640 1 -- 172.21.15.44:0/4127289955 >> v1:172.21.15.69:6789/0 conn(0x7f01a006cf90 legacy=0x7f01a00cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.056+0000 7f019f59e640 1 -- 172.21.15.44:0/4127289955 >> v1:172.21.15.44:6789/0 conn(0x7f01a00d0a30 legacy=0x7f01a00d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.056+0000 7f019f59e640 1 -- 172.21.15.44:0/4127289955 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a006c4b0 con 0x7f01a00d2040 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.056+0000 7f019f59e640 1 --2- 172.21.15.44:0/4127289955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01a00d2040 0x7f01a00d24f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0190009dc0 tx=0x7f019002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90b561eea590ce4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.057+0000 7f019dd9b640 1 -- 172.21.15.44:0/4127289955 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190039070 con 0x7f01a00d2040 2024-04-02T13:24:51.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.057+0000 7f019dd9b640 1 -- 172.21.15.44:0/4127289955 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f019002cc40 con 0x7f01a00d2040 2024-04-02T13:24:51.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.057+0000 7f019dd9b640 1 -- 172.21.15.44:0/4127289955 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190034440 con 0x7f01a00d2040 2024-04-02T13:24:51.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.057+0000 7f01a5047640 1 -- 172.21.15.44:0/4127289955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01a00d2040 msgr2=0x7f01a00d24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.057+0000 7f01a5047640 1 --2- 172.21.15.44:0/4127289955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01a00d2040 0x7f01a00d24f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0190009dc0 tx=0x7f019002be90 comp rx=0 tx=0).stop 2024-04-02T13:24:51.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.057+0000 7f01a5047640 1 -- 172.21.15.44:0/4127289955 shutdown_connections 2024-04-02T13:24:51.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.057+0000 7f01a5047640 1 --2- 172.21.15.44:0/4127289955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01a00d2040 0x7f01a00d24f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.057+0000 7f01a5047640 1 -- 172.21.15.44:0/4127289955 >> 172.21.15.44:0/4127289955 conn(0x7f01a009d470 msgr2=0x7f01a00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.057+0000 7f01a5047640 1 -- 172.21.15.44:0/4127289955 shutdown_connections 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.057+0000 7f01a5047640 1 -- 172.21.15.44:0/4127289955 wait complete. 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.059+0000 7f01a5047640 1 Processor -- start 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.059+0000 7f01a5047640 1 -- start start 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.059+0000 7f01a5047640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01a006cf90 0x7f01a015a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.059+0000 7f01a5047640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a00d0a30 0x7f01a00cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.059+0000 7f01a5047640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01a00d2040 0x7f01a00c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.059+0000 7f01a5047640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f01a015e6f0 con 0x7f01a00d2040 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.059+0000 7f01a5047640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f01a006d3f0 con 0x7f01a00d0a30 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.059+0000 7f01a5047640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f01a015e870 con 0x7f01a006cf90 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.059+0000 7f019f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01a00d2040 0x7f01a00c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.059+0000 7f019f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01a00d2040 0x7f01a00c7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35568/0 (socket says 172.21.15.44:35568) 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.059+0000 7f019f59e640 1 -- 172.21.15.44:0/4141294761 learned_addr learned my addr 172.21.15.44:0/4141294761 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:51.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.060+0000 7f019e59c640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a00d0a30 0x7f01a00cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.060+0000 7f019f59e640 1 -- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01a006cf90 msgr2=0x7f01a015a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.060+0000 7f019f59e640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01a006cf90 0x7f01a015a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.060+0000 7f019f59e640 1 -- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a00d0a30 msgr2=0x7f01a00cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.060+0000 7f019f59e640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a00d0a30 0x7f01a00cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.060+0000 7f019f59e640 1 -- 172.21.15.44:0/4141294761 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a00cc300 con 0x7f01a00d2040 2024-04-02T13:24:51.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.060+0000 7f019e59c640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a00d0a30 0x7f01a00cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:51.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.060+0000 7f019f59e640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01a00d2040 0x7f01a00c7c70 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f0190009ee0 tx=0x7f0190004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:51.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.060+0000 7f01977fe640 1 -- 172.21.15.44:0/4141294761 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190039050 con 0x7f01a00d2040 2024-04-02T13:24:51.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.060+0000 7f01977fe640 1 -- 172.21.15.44:0/4141294761 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0190045070 con 0x7f01a00d2040 2024-04-02T13:24:51.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.060+0000 7f01977fe640 1 -- 172.21.15.44:0/4141294761 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190009450 con 0x7f01a00d2040 2024-04-02T13:24:51.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.060+0000 7f01a5047640 1 -- 172.21.15.44:0/4141294761 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a00cc5e0 con 0x7f01a00d2040 2024-04-02T13:24:51.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.060+0000 7f01a5047640 1 -- 172.21.15.44:0/4141294761 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a00ccba0 con 0x7f01a00d2040 2024-04-02T13:24:51.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.061+0000 7f01a5047640 1 -- 172.21.15.44:0/4141294761 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01a00684f0 con 0x7f01a00d2040 2024-04-02T13:24:51.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.062+0000 7f01977fe640 1 -- 172.21.15.44:0/4141294761 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0190040690 con 0x7f01a00d2040 2024-04-02T13:24:51.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.062+0000 7f01977fe640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f018006c920 0x7f018006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.062+0000 7f01977fe640 1 -- 172.21.15.44:0/4141294761 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5249+0+0 (secure 0 0 0) 0x7f01900add10 con 0x7f01a00d2040 2024-04-02T13:24:51.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.062+0000 7f019ed9d640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f018006c920 0x7f018006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.063+0000 7f019ed9d640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f018006c920 0x7f018006ede0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0198005880 tx=0x7f0198007950 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:51.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.066+0000 7f01977fe640 1 -- 172.21.15.44:0/4141294761 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f019004e020 con 0x7f01a00d2040 2024-04-02T13:24:51.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.251+0000 7f01a5047640 1 -- 172.21.15.44:0/4141294761 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01a00cf9d0 con 0x7f01a00d2040 2024-04-02T13:24:51.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.252+0000 7f01977fe640 1 -- 172.21.15.44:0/4141294761 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+13117 (secure 0 0 0) 0x7f019007c310 con 0x7f01a00d2040 2024-04-02T13:24:51.255 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:51.255 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":75,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:24:50.543607+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:24:51.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.253+0000 7f01a5047640 1 -- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f018006c920 msgr2=0x7f018006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.253+0000 7f01a5047640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f018006c920 0x7f018006ede0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0198005880 tx=0x7f0198007950 comp rx=0 tx=0).stop 2024-04-02T13:24:51.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.253+0000 7f01a5047640 1 -- 172.21.15.44:0/4141294761 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01a00d2040 msgr2=0x7f01a00c7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.253+0000 7f01a5047640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01a00d2040 0x7f01a00c7c70 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f0190009ee0 tx=0x7f0190004a40 comp rx=0 tx=0).stop 2024-04-02T13:24:51.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.254+0000 7f01a5047640 1 -- 172.21.15.44:0/4141294761 shutdown_connections 2024-04-02T13:24:51.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.254+0000 7f01a5047640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f018006c920 0x7f018006ede0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.254+0000 7f01a5047640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01a00d2040 0x7f01a00c7c70 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.254+0000 7f01a5047640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a00d0a30 0x7f01a00cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.254+0000 7f01a5047640 1 --2- 172.21.15.44:0/4141294761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01a006cf90 0x7f01a015a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.254+0000 7f01a5047640 1 -- 172.21.15.44:0/4141294761 >> 172.21.15.44:0/4141294761 conn(0x7f01a009d470 msgr2=0x7f01a015d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:51.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.254+0000 7f01a5047640 1 -- 172.21.15.44:0/4141294761 shutdown_connections 2024-04-02T13:24:51.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.254+0000 7f01a5047640 1 -- 172.21.15.44:0/4141294761 wait complete. 2024-04-02T13:24:51.268 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:24:51.268 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:24:51.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.369+0000 7f4fdceeb640 1 Processor -- start 2024-04-02T13:24:51.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.369+0000 7f4fdceeb640 1 -- start start 2024-04-02T13:24:51.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.370+0000 7f4fdceeb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80ce9a0 0x7f4fd80d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.370+0000 7f4fdceeb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4fd80d0670 0x7f4fd8164c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.370+0000 7f4fdceeb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fd80cdc20 con 0x7f4fd81554c0 2024-04-02T13:24:51.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.370+0000 7f4fdceeb640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4fd80cdec0 con 0x7f4fd80d0670 2024-04-02T13:24:51.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.370+0000 7f4fdceeb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd80ce1a0 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.370+0000 7f4fd6d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4fd80d0670 0x7f4fd8164c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.370+0000 7f4fd6d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4fd80d0670 0x7f4fd8164c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35580/0 (socket says 172.21.15.44:35580) 2024-04-02T13:24:51.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.370+0000 7f4fd6d9d640 1 -- 172.21.15.44:0/4275378178 learned_addr learned my addr 172.21.15.44:0/4275378178 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:51.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.370+0000 7f4fd5d9b640 1 --2- 172.21.15.44:0/4275378178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80ce9a0 0x7f4fd80d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.370+0000 7f4fd6d9d640 1 -- 172.21.15.44:0/4275378178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80ce9a0 msgr2=0x7f4fd80d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:51.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.370+0000 7f4fd6d9d640 1 --2- 172.21.15.44:0/4275378178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80ce9a0 0x7f4fd80d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.370+0000 7f4fd6d9d640 1 -- 172.21.15.44:0/4275378178 >> v1:172.21.15.69:6789/0 conn(0x7f4fd81554c0 legacy=0x7f4fd8071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.371+0000 7f4fd6d9d640 1 -- 172.21.15.44:0/4275378178 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd80ce480 con 0x7f4fd80d0670 2024-04-02T13:24:51.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.371+0000 7f4fd6d9d640 1 --2- 172.21.15.44:0/4275378178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd80d0670 0x7f4fd8164c10 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000bb10 tx=0x7f4fd002e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf7145384d50629 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:51.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.371+0000 7f4fd559a640 1 -- 172.21.15.44:0/4275378178 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd003b070 con 0x7f4fd80d0670 2024-04-02T13:24:51.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.371+0000 7f4fd559a640 1 -- 172.21.15.44:0/4275378178 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fd0036440 con 0x7f4fd80d0670 2024-04-02T13:24:51.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.371+0000 7f4fd559a640 1 -- 172.21.15.44:0/4275378178 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0036810 con 0x7f4fd80d0670 2024-04-02T13:24:51.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.372+0000 7f4fdceeb640 1 -- 172.21.15.44:0/4275378178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd80d0670 msgr2=0x7f4fd8164c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.372+0000 7f4fdceeb640 1 --2- 172.21.15.44:0/4275378178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd80d0670 0x7f4fd8164c10 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000bb10 tx=0x7f4fd002e150 comp rx=0 tx=0).stop 2024-04-02T13:24:51.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.372+0000 7f4fdceeb640 1 -- 172.21.15.44:0/4275378178 shutdown_connections 2024-04-02T13:24:51.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.372+0000 7f4fdceeb640 1 --2- 172.21.15.44:0/4275378178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd80d0670 0x7f4fd8164c10 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.372+0000 7f4fdceeb640 1 --2- 172.21.15.44:0/4275378178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80ce9a0 0x7f4fd80d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.372+0000 7f4fdceeb640 1 -- 172.21.15.44:0/4275378178 >> 172.21.15.44:0/4275378178 conn(0x7f4fd806b920 msgr2=0x7f4fd806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:51.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.372+0000 7f4fdceeb640 1 -- 172.21.15.44:0/4275378178 shutdown_connections 2024-04-02T13:24:51.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.372+0000 7f4fdceeb640 1 -- 172.21.15.44:0/4275378178 wait complete. 2024-04-02T13:24:51.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.373+0000 7f4fdceeb640 1 Processor -- start 2024-04-02T13:24:51.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.373+0000 7f4fdceeb640 1 -- start start 2024-04-02T13:24:51.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.374+0000 7f4fdceeb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd80ce9a0 0x7f4fd8152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.374+0000 7f4fdceeb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80d0670 0x7f4fd80b4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.374+0000 7f4fdceeb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fd81554c0 0x7f4fd80afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.374+0000 7f4fdceeb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd815c290 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.374+0000 7f4fdceeb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd8071830 con 0x7f4fd81554c0 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.374+0000 7f4fdceeb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd80b0150 con 0x7f4fd80d0670 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.374+0000 7f4fd659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd80ce9a0 0x7f4fd8152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.374+0000 7f4fd659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd80ce9a0 0x7f4fd8152ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35592/0 (socket says 172.21.15.44:35592) 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.374+0000 7f4fd659c640 1 -- 172.21.15.44:0/915876344 learned_addr learned my addr 172.21.15.44:0/915876344 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.374+0000 7f4fd5d9b640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80d0670 0x7f4fd80b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.374+0000 7f4fd6d9d640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fd81554c0 0x7f4fd80afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.375+0000 7f4fd659c640 1 -- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80d0670 msgr2=0x7f4fd80b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.375+0000 7f4fd659c640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80d0670 0x7f4fd80b4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.375+0000 7f4fd659c640 1 -- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fd81554c0 msgr2=0x7f4fd80afc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.375+0000 7f4fd659c640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fd81554c0 0x7f4fd80afc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.375+0000 7f4fd659c640 1 -- 172.21.15.44:0/915876344 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd80b2320 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.375+0000 7f4fd659c640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd80ce9a0 0x7f4fd8152ec0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f4fc800c730 tx=0x7f4fc800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:51.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.375+0000 7f4fceffd640 1 -- 172.21.15.44:0/915876344 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8016020 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.375+0000 7f4fceffd640 1 -- 172.21.15.44:0/915876344 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fc8006d50 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.375+0000 7f4fdceeb640 1 -- 172.21.15.44:0/915876344 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd8158580 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.375+0000 7f4fceffd640 1 -- 172.21.15.44:0/915876344 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8003bb0 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.375+0000 7f4fdceeb640 1 -- 172.21.15.44:0/915876344 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd80b2600 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.376+0000 7f4fdceeb640 1 -- 172.21.15.44:0/915876344 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd80684f0 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.376+0000 7f4fceffd640 1 -- 172.21.15.44:0/915876344 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4fc800a850 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.377+0000 7f4fceffd640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4fb806c920 0x7f4fb806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.377+0000 7f4fceffd640 1 -- 172.21.15.44:0/915876344 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5249+0+0 (secure 0 0 0) 0x7f4fc808b3e0 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.377+0000 7f4fd5d9b640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4fb806c920 0x7f4fb806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.378+0000 7f4fd5d9b640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4fb806c920 0x7f4fb806ede0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4fd815bb70 tx=0x7f4fc0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:51.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.381+0000 7f4fceffd640 1 -- 172.21.15.44:0/915876344 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4fc8020020 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.564+0000 7f4fdceeb640 1 -- 172.21.15.44:0/915876344 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4fd80af930 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.565+0000 7f4fceffd640 1 -- 172.21.15.44:0/915876344 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) ==== 93+0+10 (secure 0 0 0) 0x7f4fc8059410 con 0x7f4fd80ce9a0 2024-04-02T13:24:51.568 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:24:51.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.567+0000 7f4fdceeb640 1 -- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4fb806c920 msgr2=0x7f4fb806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.567+0000 7f4fdceeb640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4fb806c920 0x7f4fb806ede0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4fd815bb70 tx=0x7f4fc0008040 comp rx=0 tx=0).stop 2024-04-02T13:24:51.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.567+0000 7f4fdceeb640 1 -- 172.21.15.44:0/915876344 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd80ce9a0 msgr2=0x7f4fd8152ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.567+0000 7f4fdceeb640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd80ce9a0 0x7f4fd8152ec0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f4fc800c730 tx=0x7f4fc800cbf0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.567+0000 7f4fdceeb640 1 -- 172.21.15.44:0/915876344 shutdown_connections 2024-04-02T13:24:51.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.567+0000 7f4fdceeb640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4fb806c920 0x7f4fb806ede0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.567+0000 7f4fdceeb640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fd81554c0 0x7f4fd80afc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.567+0000 7f4fdceeb640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80d0670 0x7f4fd80b4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.567+0000 7f4fdceeb640 1 --2- 172.21.15.44:0/915876344 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd80ce9a0 0x7f4fd8152ec0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.567+0000 7f4fdceeb640 1 -- 172.21.15.44:0/915876344 >> 172.21.15.44:0/915876344 conn(0x7f4fd806b920 msgr2=0x7f4fd8071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:51.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.567+0000 7f4fdceeb640 1 -- 172.21.15.44:0/915876344 shutdown_connections 2024-04-02T13:24:51.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.567+0000 7f4fdceeb640 1 -- 172.21.15.44:0/915876344 wait complete. 2024-04-02T13:24:51.581 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T13:24:51.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.680+0000 7f2ed1b2b640 1 Processor -- start 2024-04-02T13:24:51.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.680+0000 7f2ed1b2b640 1 -- start start 2024-04-02T13:24:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2ed1b2b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2ecc15e4d0 0x7f2ecc06b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2ed1b2b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ecc06dfe0 con 0x7f2ecc072f60 2024-04-02T13:24:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2ed1b2b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ecc06e2c0 con 0x7f2ecc15e0f0 2024-04-02T13:24:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2ed1b2b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2ecc06e5a0 con 0x7f2ecc15e4d0 2024-04-02T13:24:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2ecbfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f2ecc072f60 0x7f2ecc0add60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33550/0 (socket says 172.21.15.44:33550) 2024-04-02T13:24:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2ecbfff640 1 -- 172.21.15.44:0/2330976731 learned_addr learned my addr 172.21.15.44:0/2330976731 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2ecaffd640 1 --2- 172.21.15.44:0/2330976731 >> v2:172.21.15.69:3300/0 conn(0x7f2ecc15e4d0 0x7f2ecc06b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4059798938 0 0) 0x7f2ecc06dfe0 con 0x7f2ecc072f60 2024-04-02T13:24:51.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2eb4003520 con 0x7f2ecc072f60 2024-04-02T13:24:51.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3773137668 0 0) 0x7f2eb4003520 con 0x7f2ecc072f60 2024-04-02T13:24:51.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 >> v2:172.21.15.69:3300/0 conn(0x7f2ecc15e4d0 msgr2=0x7f2ecc06b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:51.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2eca7fc640 1 --2- 172.21.15.44:0/2330976731 >> v2:172.21.15.69:3300/0 conn(0x7f2ecc15e4d0 0x7f2ecc06b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 >> v1:172.21.15.69:6789/0 conn(0x7f2ecc15e0f0 legacy=0x7f2ecc0d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ecc06e880 con 0x7f2ecc072f60 2024-04-02T13:24:51.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.681+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f2ec40019f0 con 0x7f2ecc072f60 2024-04-02T13:24:51.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 >> v1:172.21.15.44:6789/0 conn(0x7f2ecc072f60 legacy=0x7f2ecc0add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2eca7fc640 1 --2- 172.21.15.44:0/2330976731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb4003a00 0x7f2eb4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2eca7fc640 1 --2- 172.21.15.44:0/2330976731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb4004800 0x7f2eb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2eca7fc640 1 --2- 172.21.15.44:0/2330976731 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb40051c0 0x7f2eb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ecc06e5a0 con 0x7f2eb40051c0 2024-04-02T13:24:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb400c0a0 con 0x7f2eb4003a00 2024-04-02T13:24:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb400c2a0 con 0x7f2eb4004800 2024-04-02T13:24:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2ecbfff640 1 --2- 172.21.15.44:0/2330976731 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb40051c0 0x7f2eb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2ecb7fe640 1 --2- 172.21.15.44:0/2330976731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb4003a00 0x7f2eb4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2ecaffd640 1 --2- 172.21.15.44:0/2330976731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb4004800 0x7f2eb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2ecbfff640 1 -- 172.21.15.44:0/2330976731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb4004800 msgr2=0x7f2eb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2ecbfff640 1 --2- 172.21.15.44:0/2330976731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb4004800 0x7f2eb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2ecbfff640 1 -- 172.21.15.44:0/2330976731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb4003a00 msgr2=0x7f2eb4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2ecbfff640 1 --2- 172.21.15.44:0/2330976731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb4003a00 0x7f2eb4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.682+0000 7f2ecbfff640 1 -- 172.21.15.44:0/2330976731 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb4003520 con 0x7f2eb40051c0 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.683+0000 7f2ecbfff640 1 -- 172.21.15.44:0/2330976731 reap_dead start 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.683+0000 7f2ecbfff640 1 --2- 172.21.15.44:0/2330976731 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb40051c0 0x7f2eb400b9c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f2ec400a770 tx=0x7f2eb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f3499a82ff1557 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.683+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec400a390 con 0x7f2eb40051c0 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.683+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ec4034650 con 0x7f2eb40051c0 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.683+0000 7f2eca7fc640 1 -- 172.21.15.44:0/2330976731 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec4034a40 con 0x7f2eb40051c0 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.683+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/2330976731 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb40051c0 msgr2=0x7f2eb400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.683+0000 7f2ed1b2b640 1 --2- 172.21.15.44:0/2330976731 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb40051c0 0x7f2eb400b9c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f2ec400a770 tx=0x7f2eb4005830 comp rx=0 tx=0).stop 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.683+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/2330976731 shutdown_connections 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.683+0000 7f2ed1b2b640 1 --2- 172.21.15.44:0/2330976731 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb40051c0 0x7f2eb400b9c0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.683+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/2330976731 >> 172.21.15.44:0/2330976731 conn(0x7f2ecc0b2980 msgr2=0x7f2ecc0b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:51.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.683+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/2330976731 shutdown_connections 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.684+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/2330976731 wait complete. 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.685+0000 7f2ed1b2b640 1 Processor -- start 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.685+0000 7f2ed1b2b640 1 -- start start 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.685+0000 7f2ed1b2b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb40051c0 0x7f2ecc157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.685+0000 7f2ed1b2b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ecc155f90 0x7f2ecc0c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.685+0000 7f2ed1b2b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ecc1583f0 0x7f2ecc0c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.685+0000 7f2ed1b2b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ecc0d00f0 con 0x7f2ecc155f90 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.685+0000 7f2ed1b2b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2ecc06eda0 con 0x7f2eb40051c0 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.685+0000 7f2ed1b2b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2ecc0d0270 con 0x7f2ecc1583f0 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ecaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ecc155f90 0x7f2ecc0c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ecaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ecc155f90 0x7f2ecc0c5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35614/0 (socket says 172.21.15.44:35614) 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ecaffd640 1 -- 172.21.15.44:0/1121631711 learned_addr learned my addr 172.21.15.44:0/1121631711 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ecb7fe640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb40051c0 0x7f2ecc157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ecbfff640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ecc1583f0 0x7f2ecc0c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ecaffd640 1 -- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ecc1583f0 msgr2=0x7f2ecc0c63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ecaffd640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ecc1583f0 0x7f2ecc0c63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ecaffd640 1 -- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb40051c0 msgr2=0x7f2ecc157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ecaffd640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb40051c0 0x7f2ecc157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ecaffd640 1 -- 172.21.15.44:0/1121631711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ecc158920 con 0x7f2ecc155f90 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ecb7fe640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb40051c0 0x7f2ecc157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ecaffd640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ecc155f90 0x7f2ecc0c5e90 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00119c0 tx=0x7f2ec0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2eb3fff640 1 -- 172.21.15.44:0/1121631711 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0019070 con 0x7f2ecc155f90 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2eb3fff640 1 -- 172.21.15.44:0/1121631711 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ec0004240 con 0x7f2ecc155f90 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2eb3fff640 1 -- 172.21.15.44:0/1121631711 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec000f880 con 0x7f2ecc155f90 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/1121631711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ecc158c00 con 0x7f2ecc155f90 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.686+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/1121631711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ecc1591c0 con 0x7f2ecc155f90 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.687+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/1121631711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ecc0684f0 con 0x7f2ecc155f90 2024-04-02T13:24:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.688+0000 7f2eb3fff640 1 -- 172.21.15.44:0/1121631711 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2ec001c070 con 0x7f2ecc155f90 2024-04-02T13:24:51.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.688+0000 7f2eb3fff640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2eb806c8f0 0x7f2eb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:51.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.688+0000 7f2eb3fff640 1 -- 172.21.15.44:0/1121631711 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5249+0+0 (secure 0 0 0) 0x7f2ec008bcc0 con 0x7f2ecc155f90 2024-04-02T13:24:51.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.688+0000 7f2ecb7fe640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2eb806c8f0 0x7f2eb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:51.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.689+0000 7f2ecb7fe640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2eb806c8f0 0x7f2eb806edb0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc000c00 tx=0x7f2ebc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:51.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.693+0000 7f2eb3fff640 1 -- 172.21.15.44:0/1121631711 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2ec005a340 con 0x7f2ecc155f90 2024-04-02T13:24:51.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.877+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/1121631711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f2ecc155c50 con 0x7f2ecc155f90 2024-04-02T13:24:51.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.877+0000 7f2eb3fff640 1 -- 172.21.15.44:0/1121631711 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v75) ==== 84+0+65 (secure 0 0 0) 0x7f2ec0059c40 con 0x7f2ecc155f90 2024-04-02T13:24:51.881 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:51.881 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-04-02T13:24:51.881 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:24:51.881 INFO:teuthology.orchestra.run.smithi044.stdout: "poolnum": 1, 2024-04-02T13:24:51.881 INFO:teuthology.orchestra.run.smithi044.stdout: "poolname": ".mgr" 2024-04-02T13:24:51.881 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:24:51.881 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-04-02T13:24:51.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.879+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2eb806c8f0 msgr2=0x7f2eb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.879+0000 7f2ed1b2b640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2eb806c8f0 0x7f2eb806edb0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc000c00 tx=0x7f2ebc009040 comp rx=0 tx=0).stop 2024-04-02T13:24:51.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.879+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/1121631711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ecc155f90 msgr2=0x7f2ecc0c5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:51.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.879+0000 7f2ed1b2b640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ecc155f90 0x7f2ecc0c5e90 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00119c0 tx=0x7f2ec0011e80 comp rx=0 tx=0).stop 2024-04-02T13:24:51.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.879+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/1121631711 shutdown_connections 2024-04-02T13:24:51.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.879+0000 7f2ed1b2b640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2eb806c8f0 0x7f2eb806edb0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.879+0000 7f2ed1b2b640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ecc1583f0 0x7f2ecc0c63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.879+0000 7f2ed1b2b640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ecc155f90 0x7f2ecc0c5e90 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.879+0000 7f2ed1b2b640 1 --2- 172.21.15.44:0/1121631711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb40051c0 0x7f2ecc157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:51.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.879+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/1121631711 >> 172.21.15.44:0/1121631711 conn(0x7f2ecc0b2980 msgr2=0x7f2ecc0add60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:51.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.879+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/1121631711 shutdown_connections 2024-04-02T13:24:51.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.879+0000 7f2ed1b2b640 1 -- 172.21.15.44:0/1121631711 wait complete. 2024-04-02T13:24:51.893 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T13:24:51.893 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T13:24:52.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.997+0000 7fd0035cd640 1 Processor -- start 2024-04-02T13:24:52.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.998+0000 7fd0035cd640 1 -- start start 2024-04-02T13:24:52.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.998+0000 7fd0035cd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcffc06cff0 0x7fcffc06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.998+0000 7fd0035cd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcffc15e5d0 0x7fcffc0d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.998+0000 7fd0035cd640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcffc06d910 con 0x7fcffc0b4180 2024-04-02T13:24:52.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.998+0000 7fd0035cd640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcffc0d1bf0 con 0x7fcffc15e5d0 2024-04-02T13:24:52.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.998+0000 7fd0035cd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcffc0d1630 con 0x7fcffc06cff0 2024-04-02T13:24:52.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.999+0000 7fd001369640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcffc06cff0 0x7fcffc06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.999+0000 7fd000b68640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcffc15e5d0 0x7fcffc0d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.999+0000 7fd000b68640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcffc15e5d0 0x7fcffc0d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:46806/0 (socket says 172.21.15.44:46806) 2024-04-02T13:24:52.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.999+0000 7fd001369640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcffc06cff0 0x7fcffc06d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:35446/0 (socket says 172.21.15.44:35446) 2024-04-02T13:24:52.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.999+0000 7fd000b68640 1 -- 172.21.15.44:0/2375820233 learned_addr learned my addr 172.21.15.44:0/2375820233 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:52.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.999+0000 7fd000b68640 1 -- 172.21.15.44:0/2375820233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcffc06cff0 msgr2=0x7fcffc06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:52.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.999+0000 7fd000b68640 1 --2- 172.21.15.44:0/2375820233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcffc06cff0 0x7fcffc06d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.999+0000 7fd000b68640 1 -- 172.21.15.44:0/2375820233 >> v1:172.21.15.69:6789/0 conn(0x7fcffc0b4180 legacy=0x7fcffc0ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:52.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:51.999+0000 7fd000b68640 1 -- 172.21.15.44:0/2375820233 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcffc0d1910 con 0x7fcffc15e5d0 2024-04-02T13:24:52.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.000+0000 7fd000b68640 1 --2- 172.21.15.44:0/2375820233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcffc15e5d0 0x7fcffc0d0d30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009870 tx=0x7fcfec02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff021d5d4ba72d3d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:52.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.000+0000 7fcff3fff640 1 -- 172.21.15.44:0/2375820233 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec035020 con 0x7fcffc15e5d0 2024-04-02T13:24:52.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.000+0000 7fcff3fff640 1 -- 172.21.15.44:0/2375820233 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfec031640 con 0x7fcffc15e5d0 2024-04-02T13:24:52.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.000+0000 7fcff3fff640 1 -- 172.21.15.44:0/2375820233 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec02d040 con 0x7fcffc15e5d0 2024-04-02T13:24:52.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.000+0000 7fd0035cd640 1 -- 172.21.15.44:0/2375820233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcffc15e5d0 msgr2=0x7fcffc0d0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:52.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.000+0000 7fd0035cd640 1 --2- 172.21.15.44:0/2375820233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcffc15e5d0 0x7fcffc0d0d30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009870 tx=0x7fcfec02b8f0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.000+0000 7fd0035cd640 1 -- 172.21.15.44:0/2375820233 shutdown_connections 2024-04-02T13:24:52.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.000+0000 7fd0035cd640 1 --2- 172.21.15.44:0/2375820233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcffc15e5d0 0x7fcffc0d0d30 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.000+0000 7fd0035cd640 1 --2- 172.21.15.44:0/2375820233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcffc06cff0 0x7fcffc06d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.001+0000 7fd0035cd640 1 -- 172.21.15.44:0/2375820233 >> 172.21.15.44:0/2375820233 conn(0x7fcffc09aa20 msgr2=0x7fcffc09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:52.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.001+0000 7fd0035cd640 1 -- 172.21.15.44:0/2375820233 shutdown_connections 2024-04-02T13:24:52.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.001+0000 7fd0035cd640 1 -- 172.21.15.44:0/2375820233 wait complete. 2024-04-02T13:24:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.002+0000 7fd0035cd640 1 Processor -- start 2024-04-02T13:24:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.002+0000 7fd0035cd640 1 -- start start 2024-04-02T13:24:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.002+0000 7fd0035cd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcffc06cff0 0x7fcffc0b6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.002+0000 7fd0035cd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcffc0b4180 0x7fcffc0b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.002+0000 7fd0035cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcffc15e5d0 0x7fcffc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.002+0000 7fd0035cd640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc0d1270 con 0x7fcffc15e5d0 2024-04-02T13:24:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.002+0000 7fd0035cd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc0b1820 con 0x7fcffc06cff0 2024-04-02T13:24:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.002+0000 7fd0035cd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcffc0d13f0 con 0x7fcffc0b4180 2024-04-02T13:24:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fd001b6a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcffc15e5d0 0x7fcffc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fd001b6a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcffc15e5d0 0x7fcffc0b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35630/0 (socket says 172.21.15.44:35630) 2024-04-02T13:24:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fd001b6a640 1 -- 172.21.15.44:0/642879469 learned_addr learned my addr 172.21.15.44:0/642879469 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fd000b68640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcffc0b4180 0x7fcffc0b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fd001369640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcffc06cff0 0x7fcffc0b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fd001b6a640 1 -- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcffc0b4180 msgr2=0x7fcffc0b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fd001b6a640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcffc0b4180 0x7fcffc0b49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fd001b6a640 1 -- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcffc06cff0 msgr2=0x7fcffc0b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fd001b6a640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcffc06cff0 0x7fcffc0b6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fd001b6a640 1 -- 172.21.15.44:0/642879469 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcffc0b7170 con 0x7fcffc15e5d0 2024-04-02T13:24:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fd001b6a640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcffc15e5d0 0x7fcffc0b4f00 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fcff800c760 tx=0x7fcff800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fcff1ffb640 1 -- 172.21.15.44:0/642879469 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff8003b70 con 0x7fcffc15e5d0 2024-04-02T13:24:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fcff1ffb640 1 -- 172.21.15.44:0/642879469 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcff8003d10 con 0x7fcffc15e5d0 2024-04-02T13:24:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fd0035cd640 1 -- 172.21.15.44:0/642879469 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcffc0b7450 con 0x7fcffc15e5d0 2024-04-02T13:24:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.003+0000 7fcff1ffb640 1 -- 172.21.15.44:0/642879469 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff800aa90 con 0x7fcffc15e5d0 2024-04-02T13:24:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.004+0000 7fd0035cd640 1 -- 172.21.15.44:0/642879469 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcffc0b7a10 con 0x7fcffc15e5d0 2024-04-02T13:24:52.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.005+0000 7fd0035cd640 1 -- 172.21.15.44:0/642879469 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcffc0684f0 con 0x7fcffc15e5d0 2024-04-02T13:24:52.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.005+0000 7fcff1ffb640 1 -- 172.21.15.44:0/642879469 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcff800ac30 con 0x7fcffc15e5d0 2024-04-02T13:24:52.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.005+0000 7fcff1ffb640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcfe006c8f0 0x7fcfe006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.005+0000 7fcff1ffb640 1 -- 172.21.15.44:0/642879469 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5249+0+0 (secure 0 0 0) 0x7fcff808b7d0 con 0x7fcffc15e5d0 2024-04-02T13:24:52.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.005+0000 7fd001369640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcfe006c8f0 0x7fcfe006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.006+0000 7fd001369640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcfe006c8f0 0x7fcfe006edb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009ad0 tx=0x7fcff4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.010+0000 7fcff1ffb640 1 -- 172.21.15.44:0/642879469 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fcff8059e50 con 0x7fcffc15e5d0 2024-04-02T13:24:52.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.197+0000 7fd0035cd640 1 -- 172.21.15.44:0/642879469 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fcffc06cdd0 con 0x7fcffc15e5d0 2024-04-02T13:24:52.463 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:24:52.466 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:24:52.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.547+0000 7fcff1ffb640 1 -- 172.21.15.44:0/642879469 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v76) ==== 138+0+0 (secure 0 0 0) 0x7fcff8059750 con 0x7fcffc15e5d0 2024-04-02T13:24:52.550 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' created 2024-04-02T13:24:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.549+0000 7fd0035cd640 1 -- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcfe006c8f0 msgr2=0x7fcfe006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.549+0000 7fd0035cd640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcfe006c8f0 0x7fcfe006edb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009ad0 tx=0x7fcff4008040 comp rx=0 tx=0).stop 2024-04-02T13:24:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.549+0000 7fd0035cd640 1 -- 172.21.15.44:0/642879469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcffc15e5d0 msgr2=0x7fcffc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.549+0000 7fd0035cd640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcffc15e5d0 0x7fcffc0b4f00 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fcff800c760 tx=0x7fcff800cc20 comp rx=0 tx=0).stop 2024-04-02T13:24:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.550+0000 7fd0035cd640 1 -- 172.21.15.44:0/642879469 shutdown_connections 2024-04-02T13:24:52.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.550+0000 7fd0035cd640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcfe006c8f0 0x7fcfe006edb0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.550+0000 7fd0035cd640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcffc15e5d0 0x7fcffc0b4f00 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.550+0000 7fd0035cd640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcffc0b4180 0x7fcffc0b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.550+0000 7fd0035cd640 1 --2- 172.21.15.44:0/642879469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcffc06cff0 0x7fcffc0b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.550+0000 7fd0035cd640 1 -- 172.21.15.44:0/642879469 >> 172.21.15.44:0/642879469 conn(0x7fcffc09aa20 msgr2=0x7fcffc06f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:52.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.550+0000 7fd0035cd640 1 -- 172.21.15.44:0/642879469 shutdown_connections 2024-04-02T13:24:52.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.550+0000 7fd0035cd640 1 -- 172.21.15.44:0/642879469 wait complete. 2024-04-02T13:24:52.567 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T13:24:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.714+0000 7f1a5cd96640 1 Processor -- start 2024-04-02T13:24:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.714+0000 7f1a5cd96640 1 -- start start 2024-04-02T13:24:52.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a5cd96640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580d0a30 0x7f1a580b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a5cd96640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1a58144130 0x7f1a5815e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a5cd96640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a580d0e10 con 0x7f1a580b3990 2024-04-02T13:24:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a5cd96640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1a580b3df0 con 0x7f1a58144130 2024-04-02T13:24:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a5cd96640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1a58161e10 con 0x7f1a580d0a30 2024-04-02T13:24:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a4ffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f1a580b3990 0x7f1a58143a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33566/0 (socket says 172.21.15.44:33566) 2024-04-02T13:24:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a4ffff640 1 -- 172.21.15.44:0/2651755590 learned_addr learned my addr 172.21.15.44:0/2651755590 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2254330209 0 0) 0x7f1a40001110 con 0x7f1a580b3990 2024-04-02T13:24:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a5659c640 1 --2- 172.21.15.44:0/2651755590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580d0a30 0x7f1a580b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a56d9d640 1 --2- 172.21.15.44:0/2651755590 >> v2:172.21.15.69:3300/0 conn(0x7f1a58144130 0x7f1a5815e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a44003520 con 0x7f1a580b3990 2024-04-02T13:24:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1841595843 0 0) 0x7f1a44003520 con 0x7f1a580b3990 2024-04-02T13:24:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580d0a30 msgr2=0x7f1a580b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 --2- 172.21.15.44:0/2651755590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580d0a30 0x7f1a580b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 >> v2:172.21.15.69:3300/0 conn(0x7f1a58144130 msgr2=0x7f1a5815e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 --2- 172.21.15.44:0/2651755590 >> v2:172.21.15.69:3300/0 conn(0x7f1a58144130 0x7f1a5815e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a58144750 con 0x7f1a580b3990 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f1a40001840 con 0x7f1a580b3990 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 >> v1:172.21.15.44:6789/0 conn(0x7f1a580b3990 legacy=0x7f1a58143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 --2- 172.21.15.44:0/2651755590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580d0a30 0x7f1a440039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 --2- 172.21.15.44:0/2651755590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a44003f30 0x7f1a44004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 --2- 172.21.15.44:0/2651755590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a44004d40 0x7f1a440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1a58161e10 con 0x7f1a44004d40 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4400b9c0 con 0x7f1a44003f30 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.715+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1a4400bb90 con 0x7f1a580d0a30 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.716+0000 7f1a56d9d640 1 --2- 172.21.15.44:0/2651755590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a44004d40 0x7f1a440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.716+0000 7f1a5659c640 1 --2- 172.21.15.44:0/2651755590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580d0a30 0x7f1a440039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.716+0000 7f1a4ffff640 1 --2- 172.21.15.44:0/2651755590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a44003f30 0x7f1a44004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.716+0000 7f1a56d9d640 1 -- 172.21.15.44:0/2651755590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580d0a30 msgr2=0x7f1a440039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.716+0000 7f1a56d9d640 1 --2- 172.21.15.44:0/2651755590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580d0a30 0x7f1a440039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.716+0000 7f1a56d9d640 1 -- 172.21.15.44:0/2651755590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a44003f30 msgr2=0x7f1a44004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.716+0000 7f1a56d9d640 1 --2- 172.21.15.44:0/2651755590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a44003f30 0x7f1a44004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.716+0000 7f1a56d9d640 1 -- 172.21.15.44:0/2651755590 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a44003520 con 0x7f1a44004d40 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.717+0000 7f1a56d9d640 1 --2- 172.21.15.44:0/2651755590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a44004d40 0x7f1a440051c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000e170 tx=0x7f1a44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2beb7012ac688d32 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.717+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50039070 con 0x7f1a44004d40 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.717+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5002cc20 con 0x7f1a44004d40 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.717+0000 7f1a55d9b640 1 -- 172.21.15.44:0/2651755590 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50009c10 con 0x7f1a44004d40 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.717+0000 7f1a5cd96640 1 -- 172.21.15.44:0/2651755590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a44004d40 msgr2=0x7f1a440051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.717+0000 7f1a5cd96640 1 --2- 172.21.15.44:0/2651755590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a44004d40 0x7f1a440051c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000e170 tx=0x7f1a44005830 comp rx=0 tx=0).stop 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.717+0000 7f1a56d9d640 1 -- 172.21.15.44:0/2651755590 reap_dead start 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.717+0000 7f1a5cd96640 1 -- 172.21.15.44:0/2651755590 shutdown_connections 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.717+0000 7f1a5cd96640 1 -- 172.21.15.44:0/2651755590 >> 172.21.15.44:0/2651755590 conn(0x7f1a580b5d90 msgr2=0x7f1a5806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.717+0000 7f1a5cd96640 1 -- 172.21.15.44:0/2651755590 shutdown_connections 2024-04-02T13:24:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.717+0000 7f1a5cd96640 1 -- 172.21.15.44:0/2651755590 wait complete. 2024-04-02T13:24:52.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.719+0000 7f1a5cd96640 1 Processor -- start 2024-04-02T13:24:52.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.719+0000 7f1a5cd96640 1 -- start start 2024-04-02T13:24:52.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.719+0000 7f1a5cd96640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a44004d40 0x7f1a58158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.719+0000 7f1a5cd96640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a5812ec60 0x7f1a581316b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.719+0000 7f1a5cd96640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a580b27d0 0x7f1a5815d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.719+0000 7f1a5cd96640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1a58162620 con 0x7f1a580b27d0 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.719+0000 7f1a5cd96640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1a58162340 con 0x7f1a5812ec60 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.719+0000 7f1a5cd96640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1a58162b20 con 0x7f1a44004d40 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.719+0000 7f1a56d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a580b27d0 0x7f1a5815d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a56d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a580b27d0 0x7f1a5815d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35650/0 (socket says 172.21.15.44:35650) 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a56d9d640 1 -- 172.21.15.44:0/3024467026 learned_addr learned my addr 172.21.15.44:0/3024467026 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a4ffff640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a5812ec60 0x7f1a581316b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a5659c640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a44004d40 0x7f1a58158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a56d9d640 1 -- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a44004d40 msgr2=0x7f1a58158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a56d9d640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a44004d40 0x7f1a58158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a56d9d640 1 -- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a5812ec60 msgr2=0x7f1a581316b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a56d9d640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a5812ec60 0x7f1a581316b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a56d9d640 1 -- 172.21.15.44:0/3024467026 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a5815a5a0 con 0x7f1a580b27d0 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a56d9d640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a580b27d0 0x7f1a5815d230 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000e170 tx=0x7f1a50009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:52.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a4effd640 1 -- 172.21.15.44:0/3024467026 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50039070 con 0x7f1a580b27d0 2024-04-02T13:24:52.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a4effd640 1 -- 172.21.15.44:0/3024467026 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5003a040 con 0x7f1a580b27d0 2024-04-02T13:24:52.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a5cd96640 1 -- 172.21.15.44:0/3024467026 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a58131bf0 con 0x7f1a580b27d0 2024-04-02T13:24:52.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a4effd640 1 -- 172.21.15.44:0/3024467026 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50009c10 con 0x7f1a580b27d0 2024-04-02T13:24:52.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.720+0000 7f1a5cd96640 1 -- 172.21.15.44:0/3024467026 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a5815a840 con 0x7f1a580b27d0 2024-04-02T13:24:52.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.721+0000 7f1a5cd96640 1 -- 172.21.15.44:0/3024467026 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a580684f0 con 0x7f1a580b27d0 2024-04-02T13:24:52.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.722+0000 7f1a4effd640 1 -- 172.21.15.44:0/3024467026 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1a50046070 con 0x7f1a580b27d0 2024-04-02T13:24:52.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.722+0000 7f1a4effd640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1a3806c920 0x7f1a3806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:52.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.722+0000 7f1a4effd640 1 -- 172.21.15.44:0/3024467026 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5619+0+0 (secure 0 0 0) 0x7f1a500ad640 con 0x7f1a580b27d0 2024-04-02T13:24:52.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.722+0000 7f1a5659c640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1a3806c920 0x7f1a3806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:52.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.723+0000 7f1a5659c640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1a3806c920 0x7f1a3806ede0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009bc0 tx=0x7f1a48009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:52.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.727+0000 7f1a4effd640 1 -- 172.21.15.44:0/3024467026 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1a5007baf0 con 0x7f1a580b27d0 2024-04-02T13:24:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:52.911+0000 7f1a5cd96640 1 -- 172.21.15.44:0/3024467026 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f1a5806c200 con 0x7f1a580b27d0 2024-04-02T13:24:53.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.553+0000 7f1a4effd640 1 -- 172.21.15.44:0/3024467026 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v77) ==== 170+0+0 (secure 0 0 0) 0x7f1a5007b3f0 con 0x7f1a580b27d0 2024-04-02T13:24:53.556 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' created 2024-04-02T13:24:53.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.556+0000 7f1a5cd96640 1 -- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1a3806c920 msgr2=0x7f1a3806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:53.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.556+0000 7f1a5cd96640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1a3806c920 0x7f1a3806ede0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009bc0 tx=0x7f1a48009190 comp rx=0 tx=0).stop 2024-04-02T13:24:53.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.556+0000 7f1a5cd96640 1 -- 172.21.15.44:0/3024467026 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a580b27d0 msgr2=0x7f1a5815d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:53.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.556+0000 7f1a5cd96640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a580b27d0 0x7f1a5815d230 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000e170 tx=0x7f1a50009750 comp rx=0 tx=0).stop 2024-04-02T13:24:53.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.556+0000 7f1a5cd96640 1 -- 172.21.15.44:0/3024467026 shutdown_connections 2024-04-02T13:24:53.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.556+0000 7f1a5cd96640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1a3806c920 0x7f1a3806ede0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:53.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.556+0000 7f1a5cd96640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a580b27d0 0x7f1a5815d230 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:53.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.556+0000 7f1a5cd96640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a5812ec60 0x7f1a581316b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:53.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.556+0000 7f1a5cd96640 1 --2- 172.21.15.44:0/3024467026 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a44004d40 0x7f1a58158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:53.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.556+0000 7f1a5cd96640 1 -- 172.21.15.44:0/3024467026 >> 172.21.15.44:0/3024467026 conn(0x7f1a580b5d90 msgr2=0x7f1a5813f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:53.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.556+0000 7f1a5cd96640 1 -- 172.21.15.44:0/3024467026 shutdown_connections 2024-04-02T13:24:53.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.556+0000 7f1a5cd96640 1 -- 172.21.15.44:0/3024467026 wait complete. 2024-04-02T13:24:53.581 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T13:24:53.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.688+0000 7fd28dbd4640 1 Processor -- start 2024-04-02T13:24:53.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.688+0000 7fd28dbd4640 1 -- start start 2024-04-02T13:24:53.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.689+0000 7fd28dbd4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd28815c220 0x7fd2880b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:53.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.689+0000 7fd28dbd4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2880d03d0 0x7fd2880d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:53.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.689+0000 7fd28dbd4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2880cca50 con 0x7fd2880b1440 2024-04-02T13:24:53.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.689+0000 7fd28dbd4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd2880ccd60 con 0x7fd28815c220 2024-04-02T13:24:53.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.689+0000 7fd28dbd4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd2880cef40 con 0x7fd2880d03d0 2024-04-02T13:24:53.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.689+0000 7fd286ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd2880b1440 0x7fd2880ce780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33576/0 (socket says 172.21.15.44:33576) 2024-04-02T13:24:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.689+0000 7fd286ffd640 1 -- 172.21.15.44:0/2227036024 learned_addr learned my addr 172.21.15.44:0/2227036024 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.689+0000 7fd2877fe640 1 --2- 172.21.15.44:0/2227036024 >> v2:172.21.15.69:3300/0 conn(0x7fd28815c220 0x7fd2880b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.689+0000 7fd287fff640 1 --2- 172.21.15.44:0/2227036024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2880d03d0 0x7fd2880d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.689+0000 7fd2867fc640 1 -- 172.21.15.44:0/2227036024 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1382615394 0 0) 0x7fd27c001110 con 0x7fd2880b1440 2024-04-02T13:24:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.689+0000 7fd2867fc640 1 -- 172.21.15.44:0/2227036024 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd270003520 con 0x7fd2880b1440 2024-04-02T13:24:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.690+0000 7fd2877fe640 1 -- 172.21.15.44:0/2227036024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2880d03d0 msgr2=0x7fd2880d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.690+0000 7fd2877fe640 1 --2- 172.21.15.44:0/2227036024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2880d03d0 0x7fd2880d0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.690+0000 7fd2877fe640 1 -- 172.21.15.44:0/2227036024 >> v1:172.21.15.44:6789/0 conn(0x7fd2880b1440 legacy=0x7fd2880ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.690+0000 7fd2867fc640 1 -- 172.21.15.44:0/2227036024 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2786136490 0 0) 0x7fd270003520 con 0x7fd2880b1440 2024-04-02T13:24:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.690+0000 7fd2877fe640 1 -- 172.21.15.44:0/2227036024 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2880cf220 con 0x7fd28815c220 2024-04-02T13:24:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.690+0000 7fd287fff640 1 --2- 172.21.15.44:0/2227036024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2880d03d0 0x7fd2880d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:24:53.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.690+0000 7fd2877fe640 1 --2- 172.21.15.44:0/2227036024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd28815c220 0x7fd2880b1db0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd2780097a0 tx=0x7fd27802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72030f7705ce1a97 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:53.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.691+0000 7fd2867fc640 1 -- 172.21.15.44:0/2227036024 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278037070 con 0x7fd28815c220 2024-04-02T13:24:53.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.691+0000 7fd2867fc640 1 -- 172.21.15.44:0/2227036024 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd278030510 con 0x7fd28815c220 2024-04-02T13:24:53.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.691+0000 7fd2867fc640 1 -- 172.21.15.44:0/2227036024 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c001840 con 0x7fd28815c220 2024-04-02T13:24:53.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.691+0000 7fd28dbd4640 1 -- 172.21.15.44:0/2227036024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd28815c220 msgr2=0x7fd2880b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:53.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.691+0000 7fd28dbd4640 1 --2- 172.21.15.44:0/2227036024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd28815c220 0x7fd2880b1db0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd2780097a0 tx=0x7fd27802b790 comp rx=0 tx=0).stop 2024-04-02T13:24:53.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.691+0000 7fd28dbd4640 1 -- 172.21.15.44:0/2227036024 shutdown_connections 2024-04-02T13:24:53.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.691+0000 7fd28dbd4640 1 --2- 172.21.15.44:0/2227036024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2880d03d0 0x7fd2880d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:53.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.691+0000 7fd28dbd4640 1 --2- 172.21.15.44:0/2227036024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd28815c220 0x7fd2880b1db0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:53.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.691+0000 7fd28dbd4640 1 -- 172.21.15.44:0/2227036024 >> 172.21.15.44:0/2227036024 conn(0x7fd28809aa20 msgr2=0x7fd28809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:53.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.691+0000 7fd28dbd4640 1 -- 172.21.15.44:0/2227036024 shutdown_connections 2024-04-02T13:24:53.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.691+0000 7fd28dbd4640 1 -- 172.21.15.44:0/2227036024 wait complete. 2024-04-02T13:24:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.692+0000 7fd28dbd4640 1 Processor -- start 2024-04-02T13:24:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.693+0000 7fd28dbd4640 1 -- start start 2024-04-02T13:24:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.693+0000 7fd28dbd4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2880b1440 0x7fd2880c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.693+0000 7fd28dbd4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2880d03d0 0x7fd2880b0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.693+0000 7fd28dbd4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd28815c220 0x7fd2881568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.693+0000 7fd28dbd4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd2880cd1f0 con 0x7fd2880b1440 2024-04-02T13:24:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.693+0000 7fd28dbd4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd28815c600 con 0x7fd28815c220 2024-04-02T13:24:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.693+0000 7fd28dbd4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd2880cd370 con 0x7fd2880d03d0 2024-04-02T13:24:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.693+0000 7fd2877fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2880b1440 0x7fd2880c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.693+0000 7fd2877fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2880b1440 0x7fd2880c7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35666/0 (socket says 172.21.15.44:35666) 2024-04-02T13:24:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.693+0000 7fd2877fe640 1 -- 172.21.15.44:0/402974844 learned_addr learned my addr 172.21.15.44:0/402974844 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd287fff640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd28815c220 0x7fd2881568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd286ffd640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2880d03d0 0x7fd2880b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd2877fe640 1 -- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2880d03d0 msgr2=0x7fd2880b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd2877fe640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2880d03d0 0x7fd2880b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd2877fe640 1 -- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd28815c220 msgr2=0x7fd2881568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd2877fe640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd28815c220 0x7fd2881568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd2877fe640 1 -- 172.21.15.44:0/402974844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2880b0bd0 con 0x7fd2880b1440 2024-04-02T13:24:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd286ffd640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2880d03d0 0x7fd2880b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd2877fe640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2880b1440 0x7fd2880c7e50 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd27802b7d0 tx=0x7fd2780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd26ffff640 1 -- 172.21.15.44:0/402974844 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278041020 con 0x7fd2880b1440 2024-04-02T13:24:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd26ffff640 1 -- 172.21.15.44:0/402974844 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27803c870 con 0x7fd2880b1440 2024-04-02T13:24:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd26ffff640 1 -- 172.21.15.44:0/402974844 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278037040 con 0x7fd2880b1440 2024-04-02T13:24:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd28dbd4640 1 -- 172.21.15.44:0/402974844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2880af060 con 0x7fd2880b1440 2024-04-02T13:24:53.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.694+0000 7fd28dbd4640 1 -- 172.21.15.44:0/402974844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2880af620 con 0x7fd2880b1440 2024-04-02T13:24:53.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.695+0000 7fd28dbd4640 1 -- 172.21.15.44:0/402974844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2880684f0 con 0x7fd2880b1440 2024-04-02T13:24:53.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.695+0000 7fd26ffff640 1 -- 172.21.15.44:0/402974844 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd278044020 con 0x7fd2880b1440 2024-04-02T13:24:53.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.696+0000 7fd26ffff640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd27406c8f0 0x7fd27406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:53.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.696+0000 7fd26ffff640 1 -- 172.21.15.44:0/402974844 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6001+0+0 (secure 0 0 0) 0x7fd2780ad6b0 con 0x7fd2880b1440 2024-04-02T13:24:53.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.696+0000 7fd286ffd640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd27406c8f0 0x7fd27406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:53.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.697+0000 7fd286ffd640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd27406c8f0 0x7fd27406edb0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd2880c71c0 tx=0x7fd27c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:53.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.701+0000 7fd26ffff640 1 -- 172.21.15.44:0/402974844 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd27804b030 con 0x7fd2880b1440 2024-04-02T13:24:53.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:53.923+0000 7fd28dbd4640 1 -- 172.21.15.44:0/402974844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd2880b0e00 con 0x7fd2880b1440 2024-04-02T13:24:54.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.554+0000 7fd26ffff640 1 -- 172.21.15.44:0/402974844 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v41) ==== 174+0+0 (secure 0 0 0) 0x7fd27807b9b0 con 0x7fd2880b1440 2024-04-02T13:24:54.557 INFO:teuthology.orchestra.run.smithi044.stderr:new fs with metadata pool 11 and data pool 12 2024-04-02T13:24:54.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.557+0000 7fd28dbd4640 1 -- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd27406c8f0 msgr2=0x7fd27406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:54.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.557+0000 7fd28dbd4640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd27406c8f0 0x7fd27406edb0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd2880c71c0 tx=0x7fd27c009040 comp rx=0 tx=0).stop 2024-04-02T13:24:54.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.557+0000 7fd28dbd4640 1 -- 172.21.15.44:0/402974844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2880b1440 msgr2=0x7fd2880c7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:54.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.557+0000 7fd28dbd4640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2880b1440 0x7fd2880c7e50 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd27802b7d0 tx=0x7fd2780046c0 comp rx=0 tx=0).stop 2024-04-02T13:24:54.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.557+0000 7fd28dbd4640 1 -- 172.21.15.44:0/402974844 shutdown_connections 2024-04-02T13:24:54.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.557+0000 7fd28dbd4640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd27406c8f0 0x7fd27406edb0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:54.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.557+0000 7fd28dbd4640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd28815c220 0x7fd2881568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:54.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.557+0000 7fd28dbd4640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2880d03d0 0x7fd2880b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:54.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.557+0000 7fd28dbd4640 1 --2- 172.21.15.44:0/402974844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2880b1440 0x7fd2880c7e50 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:54.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.557+0000 7fd28dbd4640 1 -- 172.21.15.44:0/402974844 >> 172.21.15.44:0/402974844 conn(0x7fd28809aa20 msgr2=0x7fd28806f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:54.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.558+0000 7fd28dbd4640 1 -- 172.21.15.44:0/402974844 shutdown_connections 2024-04-02T13:24:54.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.558+0000 7fd28dbd4640 1 -- 172.21.15.44:0/402974844 wait complete. 2024-04-02T13:24:54.591 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:24:54.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.695+0000 7f71f01cd640 1 Processor -- start 2024-04-02T13:24:54.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.695+0000 7f71f01cd640 1 -- start start 2024-04-02T13:24:54.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.696+0000 7f71f01cd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f71e806ece0 0x7f71e8165ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:54.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.696+0000 7f71f01cd640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e806dc40 con 0x7f71e80d09f0 2024-04-02T13:24:54.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.696+0000 7f71f01cd640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e806df20 con 0x7f71e80cece0 2024-04-02T13:24:54.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.696+0000 7f71f01cd640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f71e806d9e0 con 0x7f71e806ece0 2024-04-02T13:24:54.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.696+0000 7f71ed768640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f71e80d09f0 0x7f71e806d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:45210/0 (socket says 172.21.15.44:45210) 2024-04-02T13:24:54.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.696+0000 7f71ed768640 1 -- 172.21.15.44:0/2997243058 learned_addr learned my addr 172.21.15.44:0/2997243058 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:54.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.696+0000 7f71ee76a640 1 --2- 172.21.15.44:0/2997243058 >> v2:172.21.15.69:3300/0 conn(0x7f71e806ece0 0x7f71e8165ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:54.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.696+0000 7f71ecf67640 1 -- 172.21.15.44:0/2997243058 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2212118042 0 0) 0x7f71e806dc40 con 0x7f71e80d09f0 2024-04-02T13:24:54.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.696+0000 7f71ecf67640 1 -- 172.21.15.44:0/2997243058 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71d4003520 con 0x7f71e80d09f0 2024-04-02T13:24:54.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.696+0000 7f71ee76a640 1 -- 172.21.15.44:0/2997243058 >> v1:172.21.15.69:6789/0 conn(0x7f71e80cece0 legacy=0x7f71e80cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:54.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.696+0000 7f71ecf67640 1 -- 172.21.15.44:0/2997243058 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 82041773 0 0) 0x7f71e806df20 con 0x7f71e80cece0 2024-04-02T13:24:54.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.697+0000 7f71ee76a640 1 -- 172.21.15.44:0/2997243058 >> v1:172.21.15.44:6789/0 conn(0x7f71e80d09f0 legacy=0x7f71e806d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:54.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.697+0000 7f71ee76a640 1 -- 172.21.15.44:0/2997243058 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e806e200 con 0x7f71e806ece0 2024-04-02T13:24:54.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.697+0000 7f71ee76a640 1 --2- 172.21.15.44:0/2997243058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71e806ece0 0x7f71e8165ef0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f71e0009d90 tx=0x7f71e002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=386a55b8b89b2fa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:54.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.697+0000 7f71ecf67640 1 -- 172.21.15.44:0/2997243058 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e0039070 con 0x7f71e806ece0 2024-04-02T13:24:54.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.698+0000 7f71ecf67640 1 -- 172.21.15.44:0/2997243058 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71e002cd00 con 0x7f71e806ece0 2024-04-02T13:24:54.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.698+0000 7f71ecf67640 1 -- 172.21.15.44:0/2997243058 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e0034640 con 0x7f71e806ece0 2024-04-02T13:24:54.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.698+0000 7f71f01cd640 1 -- 172.21.15.44:0/2997243058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71e806ece0 msgr2=0x7f71e8165ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:54.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.698+0000 7f71f01cd640 1 --2- 172.21.15.44:0/2997243058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71e806ece0 0x7f71e8165ef0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f71e0009d90 tx=0x7f71e002bf50 comp rx=0 tx=0).stop 2024-04-02T13:24:54.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.698+0000 7f71f01cd640 1 -- 172.21.15.44:0/2997243058 shutdown_connections 2024-04-02T13:24:54.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.698+0000 7f71f01cd640 1 --2- 172.21.15.44:0/2997243058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71e806ece0 0x7f71e8165ef0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:54.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.698+0000 7f71f01cd640 1 -- 172.21.15.44:0/2997243058 >> 172.21.15.44:0/2997243058 conn(0x7f71e80cfcd0 msgr2=0x7f71e80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:54.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.698+0000 7f71f01cd640 1 -- 172.21.15.44:0/2997243058 shutdown_connections 2024-04-02T13:24:54.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.698+0000 7f71f01cd640 1 -- 172.21.15.44:0/2997243058 wait complete. 2024-04-02T13:24:54.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.699+0000 7f71f01cd640 1 Processor -- start 2024-04-02T13:24:54.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.699+0000 7f71f01cd640 1 -- start start 2024-04-02T13:24:54.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.700+0000 7f71f01cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71e806ece0 0x7f71e8158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:54.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.700+0000 7f71f01cd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71e80cece0 0x7f71e8153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:54.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.700+0000 7f71f01cd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71e80d09f0 0x7f71e81f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:54.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.700+0000 7f71f01cd640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f71e815b220 con 0x7f71e806ece0 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.700+0000 7f71f01cd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f71e806e7c0 con 0x7f71e80cece0 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.700+0000 7f71f01cd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f71e815b3a0 con 0x7f71e80d09f0 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.700+0000 7f71edf69640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71e806ece0 0x7f71e8158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.700+0000 7f71edf69640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71e806ece0 0x7f71e8158050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41750/0 (socket says 172.21.15.44:41750) 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.700+0000 7f71edf69640 1 -- 172.21.15.44:0/1104870407 learned_addr learned my addr 172.21.15.44:0/1104870407 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.700+0000 7f71ed768640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71e80cece0 0x7f71e8153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.700+0000 7f71ee76a640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71e80d09f0 0x7f71e81f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.700+0000 7f71edf69640 1 -- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71e80d09f0 msgr2=0x7f71e81f0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.701+0000 7f71edf69640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71e80d09f0 0x7f71e81f0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.701+0000 7f71edf69640 1 -- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71e80cece0 msgr2=0x7f71e8153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.701+0000 7f71edf69640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71e80cece0 0x7f71e8153e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.701+0000 7f71edf69640 1 -- 172.21.15.44:0/1104870407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e815cd70 con 0x7f71e806ece0 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.701+0000 7f71ed768640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71e80cece0 0x7f71e8153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:24:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.701+0000 7f71edf69640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71e806ece0 0x7f71e8158050 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f71e400c4e0 tx=0x7f71e400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.701+0000 7f71de7fc640 1 -- 172.21.15.44:0/1104870407 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e4017070 con 0x7f71e806ece0 2024-04-02T13:24:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.701+0000 7f71de7fc640 1 -- 172.21.15.44:0/1104870407 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71e40035f0 con 0x7f71e806ece0 2024-04-02T13:24:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.701+0000 7f71de7fc640 1 -- 172.21.15.44:0/1104870407 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e40039e0 con 0x7f71e806ece0 2024-04-02T13:24:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.701+0000 7f71f01cd640 1 -- 172.21.15.44:0/1104870407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71e815d050 con 0x7f71e806ece0 2024-04-02T13:24:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.701+0000 7f71f01cd640 1 -- 172.21.15.44:0/1104870407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71e815d610 con 0x7f71e806ece0 2024-04-02T13:24:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.702+0000 7f71f01cd640 1 -- 172.21.15.44:0/1104870407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71e80684f0 con 0x7f71e806ece0 2024-04-02T13:24:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.702+0000 7f71de7fc640 1 -- 172.21.15.44:0/1104870407 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f71e401a070 con 0x7f71e806ece0 2024-04-02T13:24:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.703+0000 7f71de7fc640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f71cc06c8f0 0x7f71cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.703+0000 7f71de7fc640 1 -- 172.21.15.44:0/1104870407 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f71e408b060 con 0x7f71e806ece0 2024-04-02T13:24:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.703+0000 7f71ed768640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f71cc06c8f0 0x7f71cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.704+0000 7f71ed768640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f71cc06c8f0 0x7f71cc06edb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f71e80afd80 tx=0x7f71d8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.707+0000 7f71de7fc640 1 -- 172.21.15.44:0/1104870407 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f71e4058c10 con 0x7f71e806ece0 2024-04-02T13:24:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.894+0000 7f71f01cd640 1 -- 172.21.15.44:0/1104870407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f71e80afdb0 con 0x7f71e806ece0 2024-04-02T13:24:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.895+0000 7f71de7fc640 1 -- 172.21.15.44:0/1104870407 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16910 (secure 0 0 0) 0x7f71e405cf10 con 0x7f71e806ece0 2024-04-02T13:24:54.898 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:54.898 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":78,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:24:53.925344+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:24:52.202720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-02T13:24:52.916529+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:24:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.897+0000 7f71f01cd640 1 -- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f71cc06c8f0 msgr2=0x7f71cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.897+0000 7f71f01cd640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f71cc06c8f0 0x7f71cc06edb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f71e80afd80 tx=0x7f71d8009040 comp rx=0 tx=0).stop 2024-04-02T13:24:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.897+0000 7f71f01cd640 1 -- 172.21.15.44:0/1104870407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71e806ece0 msgr2=0x7f71e8158050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.897+0000 7f71f01cd640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71e806ece0 0x7f71e8158050 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f71e400c4e0 tx=0x7f71e400c9a0 comp rx=0 tx=0).stop 2024-04-02T13:24:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.897+0000 7f71f01cd640 1 -- 172.21.15.44:0/1104870407 shutdown_connections 2024-04-02T13:24:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.897+0000 7f71f01cd640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f71cc06c8f0 0x7f71cc06edb0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.897+0000 7f71f01cd640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71e80d09f0 0x7f71e81f0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.897+0000 7f71f01cd640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71e80cece0 0x7f71e8153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.897+0000 7f71f01cd640 1 --2- 172.21.15.44:0/1104870407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71e806ece0 0x7f71e8158050 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.897+0000 7f71f01cd640 1 -- 172.21.15.44:0/1104870407 >> 172.21.15.44:0/1104870407 conn(0x7f71e80cfcd0 msgr2=0x7f71e8072500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.897+0000 7f71f01cd640 1 -- 172.21.15.44:0/1104870407 shutdown_connections 2024-04-02T13:24:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:54.897+0000 7f71f01cd640 1 -- 172.21.15.44:0/1104870407 wait complete. 2024-04-02T13:24:54.911 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:24:55.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.009+0000 7f004da7b640 1 Processor -- start 2024-04-02T13:24:55.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.009+0000 7f004da7b640 1 -- start start 2024-04-02T13:24:55.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.010+0000 7f004da7b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f004806cf90 0x7f00480cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:55.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.010+0000 7f004da7b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004806c1d0 con 0x7f00480d0a30 2024-04-02T13:24:55.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.010+0000 7f004da7b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004806c4b0 con 0x7f00480cfe30 2024-04-02T13:24:55.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.010+0000 7f004da7b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f004806cb00 con 0x7f004806cf90 2024-04-02T13:24:55.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.010+0000 7f0046ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f004806cf90 0x7f00480cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:55.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.010+0000 7f00477fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f00480d0a30 0x7f00480d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:45214/0 (socket says 172.21.15.44:45214) 2024-04-02T13:24:55.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.010+0000 7f00477fe640 1 -- 172.21.15.44:0/3128182384 learned_addr learned my addr 172.21.15.44:0/3128182384 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:55.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.011+0000 7f0046ffd640 1 -- 172.21.15.44:0/3128182384 >> v1:172.21.15.69:6789/0 conn(0x7f00480cfe30 legacy=0x7f00480d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:55.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.011+0000 7f0046ffd640 1 -- 172.21.15.44:0/3128182384 >> v1:172.21.15.44:6789/0 conn(0x7f00480d0a30 legacy=0x7f00480d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:55.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.011+0000 7f00467fc640 1 -- 172.21.15.44:0/3128182384 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1955614187 0 0) 0x7f004806c1d0 con 0x7f00480d0a30 2024-04-02T13:24:55.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.011+0000 7f0046ffd640 1 -- 172.21.15.44:0/3128182384 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004806c790 con 0x7f004806cf90 2024-04-02T13:24:55.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.011+0000 7f0046ffd640 1 --2- 172.21.15.44:0/3128182384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004806cf90 0x7f00480cf8f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f003c0097a0 tx=0x7f003c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d743b19bdd825da server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:55.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.011+0000 7f00467fc640 1 -- 172.21.15.44:0/3128182384 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c0315a0 con 0x7f004806cf90 2024-04-02T13:24:55.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.012+0000 7f00467fc640 1 -- 172.21.15.44:0/3128182384 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f003c037070 con 0x7f004806cf90 2024-04-02T13:24:55.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.012+0000 7f00467fc640 1 -- 172.21.15.44:0/3128182384 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c031960 con 0x7f004806cf90 2024-04-02T13:24:55.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.012+0000 7f004da7b640 1 -- 172.21.15.44:0/3128182384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004806cf90 msgr2=0x7f00480cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:55.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.012+0000 7f004da7b640 1 --2- 172.21.15.44:0/3128182384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004806cf90 0x7f00480cf8f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f003c0097a0 tx=0x7f003c02b840 comp rx=0 tx=0).stop 2024-04-02T13:24:55.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.012+0000 7f004da7b640 1 -- 172.21.15.44:0/3128182384 shutdown_connections 2024-04-02T13:24:55.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.012+0000 7f004da7b640 1 --2- 172.21.15.44:0/3128182384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004806cf90 0x7f00480cf8f0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.012+0000 7f004da7b640 1 -- 172.21.15.44:0/3128182384 >> 172.21.15.44:0/3128182384 conn(0x7f004809d470 msgr2=0x7f00480d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:55.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.012+0000 7f004da7b640 1 -- 172.21.15.44:0/3128182384 shutdown_connections 2024-04-02T13:24:55.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.012+0000 7f004da7b640 1 -- 172.21.15.44:0/3128182384 wait complete. 2024-04-02T13:24:55.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.013+0000 7f004da7b640 1 Processor -- start 2024-04-02T13:24:55.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.013+0000 7f004da7b640 1 -- start start 2024-04-02T13:24:55.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f004da7b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004806cf90 0x7f00481562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:55.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f004da7b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00480cfe30 0x7f00480b6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:55.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f004da7b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00480d0a30 0x7f0048155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:55.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f004da7b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f00480d4da0 con 0x7f004806cf90 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f004da7b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f004806d3f0 con 0x7f00480d0a30 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f004da7b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f00480d4f20 con 0x7f00480cfe30 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f00477fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004806cf90 0x7f00481562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f00477fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004806cf90 0x7f00481562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41766/0 (socket says 172.21.15.44:41766) 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f00477fe640 1 -- 172.21.15.44:0/267846769 learned_addr learned my addr 172.21.15.44:0/267846769 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f0047fff640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00480d0a30 0x7f0048155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f0046ffd640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00480cfe30 0x7f00480b6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f00477fe640 1 -- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00480cfe30 msgr2=0x7f00480b6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f00477fe640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00480cfe30 0x7f00480b6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f00477fe640 1 -- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00480d0a30 msgr2=0x7f0048155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.014+0000 7f00477fe640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00480d0a30 0x7f0048155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.015+0000 7f00477fe640 1 -- 172.21.15.44:0/267846769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00481e7f70 con 0x7f004806cf90 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.015+0000 7f0047fff640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00480d0a30 0x7f0048155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:55.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.015+0000 7f00477fe640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004806cf90 0x7f00481562f0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f003800c500 tx=0x7f003800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:55.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.015+0000 7f002ffff640 1 -- 172.21.15.44:0/267846769 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0038018070 con 0x7f004806cf90 2024-04-02T13:24:55.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.015+0000 7f002ffff640 1 -- 172.21.15.44:0/267846769 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0038007580 con 0x7f004806cf90 2024-04-02T13:24:55.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.015+0000 7f004da7b640 1 -- 172.21.15.44:0/267846769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00481e8250 con 0x7f004806cf90 2024-04-02T13:24:55.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.015+0000 7f002ffff640 1 -- 172.21.15.44:0/267846769 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0038004a20 con 0x7f004806cf90 2024-04-02T13:24:55.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.015+0000 7f004da7b640 1 -- 172.21.15.44:0/267846769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00481e8810 con 0x7f004806cf90 2024-04-02T13:24:55.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.016+0000 7f004da7b640 1 -- 172.21.15.44:0/267846769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00480684f0 con 0x7f004806cf90 2024-04-02T13:24:55.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.016+0000 7f002ffff640 1 -- 172.21.15.44:0/267846769 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f003801b070 con 0x7f004806cf90 2024-04-02T13:24:55.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.017+0000 7f002ffff640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f003406c920 0x7f003406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:55.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.017+0000 7f002ffff640 1 -- 172.21.15.44:0/267846769 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f003808af70 con 0x7f004806cf90 2024-04-02T13:24:55.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.017+0000 7f0046ffd640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f003406c920 0x7f003406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:55.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.018+0000 7f0046ffd640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f003406c920 0x7f003406ede0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f003c036000 tx=0x7f003c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:55.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.021+0000 7f002ffff640 1 -- 172.21.15.44:0/267846769 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f00380592a0 con 0x7f004806cf90 2024-04-02T13:24:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.209+0000 7f004da7b640 1 -- 172.21.15.44:0/267846769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f00480d17c0 con 0x7f004806cf90 2024-04-02T13:24:55.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.211+0000 7f002ffff640 1 -- 172.21.15.44:0/267846769 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16910 (secure 0 0 0) 0x7f0038058ba0 con 0x7f004806cf90 2024-04-02T13:24:55.214 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:55.214 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":78,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:24:53.925344+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:24:52.202720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-02T13:24:52.916529+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:24:55.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.213+0000 7f004da7b640 1 -- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f003406c920 msgr2=0x7f003406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:55.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.213+0000 7f004da7b640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f003406c920 0x7f003406ede0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f003c036000 tx=0x7f003c008030 comp rx=0 tx=0).stop 2024-04-02T13:24:55.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.213+0000 7f004da7b640 1 -- 172.21.15.44:0/267846769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004806cf90 msgr2=0x7f00481562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:55.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.213+0000 7f004da7b640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004806cf90 0x7f00481562f0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f003800c500 tx=0x7f003800c9c0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.213+0000 7f004da7b640 1 -- 172.21.15.44:0/267846769 shutdown_connections 2024-04-02T13:24:55.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.213+0000 7f004da7b640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f003406c920 0x7f003406ede0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.213+0000 7f004da7b640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00480d0a30 0x7f0048155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.213+0000 7f004da7b640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00480cfe30 0x7f00480b6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.213+0000 7f004da7b640 1 --2- 172.21.15.44:0/267846769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f004806cf90 0x7f00481562f0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.213+0000 7f004da7b640 1 -- 172.21.15.44:0/267846769 >> 172.21.15.44:0/267846769 conn(0x7f004809d470 msgr2=0x7f004815b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.213+0000 7f004da7b640 1 -- 172.21.15.44:0/267846769 shutdown_connections 2024-04-02T13:24:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.213+0000 7f004da7b640 1 -- 172.21.15.44:0/267846769 wait complete. 2024-04-02T13:24:55.227 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T13:24:55.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.328+0000 7f38613c8640 1 Processor -- start 2024-04-02T13:24:55.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.328+0000 7f38613c8640 1 -- start start 2024-04-02T13:24:55.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.329+0000 7f38613c8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f385c158870 0x7f385c158c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:55.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.329+0000 7f38613c8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f385c0720d0 0x7f385c15f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:55.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.329+0000 7f38613c8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385c15b8f0 con 0x7f385c06ef00 2024-04-02T13:24:55.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.329+0000 7f38613c8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f385c15bbd0 con 0x7f385c158870 2024-04-02T13:24:55.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.329+0000 7f38613c8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f385c15beb0 con 0x7f385c0720d0 2024-04-02T13:24:55.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.329+0000 7f385affd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f385c158870 0x7f385c158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:55.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.329+0000 7f385affd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f385c158870 0x7f385c158c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41772/0 (socket says 172.21.15.44:41772) 2024-04-02T13:24:55.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.329+0000 7f385a7fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f385c06ef00 0x7f385c0719c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:45218/0 (socket says 172.21.15.44:45218) 2024-04-02T13:24:55.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.329+0000 7f385affd640 1 -- 172.21.15.44:0/4115725238 learned_addr learned my addr 172.21.15.44:0/4115725238 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:55.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.330+0000 7f385b7fe640 1 --2- 172.21.15.44:0/4115725238 >> v2:172.21.15.69:3300/0 conn(0x7f385c0720d0 0x7f385c15f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:55.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.330+0000 7f385affd640 1 -- 172.21.15.44:0/4115725238 >> v2:172.21.15.69:3300/0 conn(0x7f385c0720d0 msgr2=0x7f385c15f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:55.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.330+0000 7f385affd640 1 --2- 172.21.15.44:0/4115725238 >> v2:172.21.15.69:3300/0 conn(0x7f385c0720d0 0x7f385c15f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.330+0000 7f385affd640 1 -- 172.21.15.44:0/4115725238 >> v1:172.21.15.44:6789/0 conn(0x7f385c06ef00 legacy=0x7f385c0719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:55.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.330+0000 7f385affd640 1 -- 172.21.15.44:0/4115725238 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385c15c190 con 0x7f385c158870 2024-04-02T13:24:55.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.330+0000 7f3859ffb640 1 -- 172.21.15.44:0/4115725238 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 591288479 0 0) 0x7f385c15b8f0 con 0x7f385c06ef00 2024-04-02T13:24:55.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.330+0000 7f385b7fe640 1 --2- 172.21.15.44:0/4115725238 >> v2:172.21.15.69:3300/0 conn(0x7f385c0720d0 0x7f385c15f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:55.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.331+0000 7f385affd640 1 --2- 172.21.15.44:0/4115725238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c158870 0x7f385c158c50 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f385c0b5ad0 tx=0x7f384c02e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c602b83ca1139296 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:55.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.331+0000 7f3859ffb640 1 -- 172.21.15.44:0/4115725238 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c02ea90 con 0x7f385c158870 2024-04-02T13:24:55.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.331+0000 7f3859ffb640 1 -- 172.21.15.44:0/4115725238 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f384c02ec30 con 0x7f385c158870 2024-04-02T13:24:55.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.331+0000 7f3859ffb640 1 -- 172.21.15.44:0/4115725238 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c009600 con 0x7f385c158870 2024-04-02T13:24:55.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.331+0000 7f38613c8640 1 -- 172.21.15.44:0/4115725238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c158870 msgr2=0x7f385c158c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:55.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.331+0000 7f38613c8640 1 --2- 172.21.15.44:0/4115725238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c158870 0x7f385c158c50 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f385c0b5ad0 tx=0x7f384c02e0f0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.331+0000 7f38613c8640 1 -- 172.21.15.44:0/4115725238 shutdown_connections 2024-04-02T13:24:55.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.331+0000 7f38613c8640 1 --2- 172.21.15.44:0/4115725238 >> v2:172.21.15.69:3300/0 conn(0x7f385c0720d0 0x7f385c15f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.331+0000 7f38613c8640 1 --2- 172.21.15.44:0/4115725238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c158870 0x7f385c158c50 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.331+0000 7f38613c8640 1 -- 172.21.15.44:0/4115725238 >> 172.21.15.44:0/4115725238 conn(0x7f385c0b22c0 msgr2=0x7f385c0b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:55.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.331+0000 7f38613c8640 1 -- 172.21.15.44:0/4115725238 shutdown_connections 2024-04-02T13:24:55.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.331+0000 7f38613c8640 1 -- 172.21.15.44:0/4115725238 wait complete. 2024-04-02T13:24:55.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.333+0000 7f38613c8640 1 Processor -- start 2024-04-02T13:24:55.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.333+0000 7f38613c8640 1 -- start start 2024-04-02T13:24:55.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.333+0000 7f38613c8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c06ef00 0x7f385c0cf350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:55.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.333+0000 7f38613c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c0720d0 0x7f385c1536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:55.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.333+0000 7f38613c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f385c158870 0x7f385c1573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:55.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.333+0000 7f38613c8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f385c06b650 con 0x7f385c0720d0 2024-04-02T13:24:55.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.333+0000 7f38613c8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f385c06b4d0 con 0x7f385c06ef00 2024-04-02T13:24:55.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.333+0000 7f38613c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f385c06b7d0 con 0x7f385c158870 2024-04-02T13:24:55.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f385a7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c0720d0 0x7f385c1536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:55.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f385a7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c0720d0 0x7f385c1536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41782/0 (socket says 172.21.15.44:41782) 2024-04-02T13:24:55.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f385a7fc640 1 -- 172.21.15.44:0/983452904 learned_addr learned my addr 172.21.15.44:0/983452904 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:55.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f385affd640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c06ef00 0x7f385c0cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:55.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f385b7fe640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f385c158870 0x7f385c1573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:55.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f385a7fc640 1 -- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f385c158870 msgr2=0x7f385c1573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:55.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f385a7fc640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f385c158870 0x7f385c1573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f385a7fc640 1 -- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c06ef00 msgr2=0x7f385c0cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f385a7fc640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c06ef00 0x7f385c0cf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f385a7fc640 1 -- 172.21.15.44:0/983452904 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385c0ce2b0 con 0x7f385c0720d0 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f385affd640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c06ef00 0x7f385c0cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f385a7fc640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c0720d0 0x7f385c1536e0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f384400c420 tx=0x7f384400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f38537fe640 1 -- 172.21.15.44:0/983452904 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38440074e0 con 0x7f385c0720d0 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f38537fe640 1 -- 172.21.15.44:0/983452904 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38440047e0 con 0x7f385c0720d0 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f38613c8640 1 -- 172.21.15.44:0/983452904 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f385c0ce590 con 0x7f385c0720d0 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f38537fe640 1 -- 172.21.15.44:0/983452904 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3844004c10 con 0x7f385c0720d0 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.334+0000 7f38613c8640 1 -- 172.21.15.44:0/983452904 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f385c0ceb50 con 0x7f385c0720d0 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.335+0000 7f38613c8640 1 -- 172.21.15.44:0/983452904 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f385c0684f0 con 0x7f385c0720d0 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.336+0000 7f38537fe640 1 -- 172.21.15.44:0/983452904 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3844011070 con 0x7f385c0720d0 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.336+0000 7f38537fe640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f383c06c8f0 0x7f383c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:55.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.336+0000 7f38537fe640 1 -- 172.21.15.44:0/983452904 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f384408b8b0 con 0x7f385c0720d0 2024-04-02T13:24:55.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.336+0000 7f385affd640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f383c06c8f0 0x7f383c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:55.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.337+0000 7f385affd640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f383c06c8f0 0x7f383c06edb0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f384c037000 tx=0x7f384c032000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:55.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.340+0000 7f38537fe640 1 -- 172.21.15.44:0/983452904 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f384400e280 con 0x7f385c0720d0 2024-04-02T13:24:55.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:55.565+0000 7f38613c8640 1 -- 172.21.15.44:0/983452904 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f385c074780 con 0x7f385c0720d0 2024-04-02T13:24:56.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.576+0000 7f38537fe640 1 -- 172.21.15.44:0/983452904 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v43) ==== 118+0+0 (secure 0 0 0) 0x7f3844059be0 con 0x7f385c0720d0 2024-04-02T13:24:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.579+0000 7f38613c8640 1 -- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f383c06c8f0 msgr2=0x7f383c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.579+0000 7f38613c8640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f383c06c8f0 0x7f383c06edb0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f384c037000 tx=0x7f384c032000 comp rx=0 tx=0).stop 2024-04-02T13:24:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.579+0000 7f38613c8640 1 -- 172.21.15.44:0/983452904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c0720d0 msgr2=0x7f385c1536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.579+0000 7f38613c8640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c0720d0 0x7f385c1536e0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f384400c420 tx=0x7f384400c8e0 comp rx=0 tx=0).stop 2024-04-02T13:24:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.579+0000 7f38613c8640 1 -- 172.21.15.44:0/983452904 shutdown_connections 2024-04-02T13:24:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.579+0000 7f38613c8640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f383c06c8f0 0x7f383c06edb0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.579+0000 7f38613c8640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f385c158870 0x7f385c1573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.579+0000 7f38613c8640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c0720d0 0x7f385c1536e0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.579+0000 7f38613c8640 1 --2- 172.21.15.44:0/983452904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c06ef00 0x7f385c0cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.579+0000 7f38613c8640 1 -- 172.21.15.44:0/983452904 >> 172.21.15.44:0/983452904 conn(0x7f385c0b22c0 msgr2=0x7f385c15f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.579+0000 7f38613c8640 1 -- 172.21.15.44:0/983452904 shutdown_connections 2024-04-02T13:24:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.579+0000 7f38613c8640 1 -- 172.21.15.44:0/983452904 wait complete. 2024-04-02T13:24:56.595 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T13:24:56.596 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T13:24:56.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.703+0000 7f74009db640 1 Processor -- start 2024-04-02T13:24:56.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.703+0000 7f74009db640 1 -- start start 2024-04-02T13:24:56.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.704+0000 7f74009db640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f73fc161780 0x7f73fc06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:56.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.704+0000 7f74009db640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0ceed0 0x7f73fc0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:56.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.704+0000 7f74009db640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73fc0cc6c0 con 0x7f73fc0cf5b0 2024-04-02T13:24:56.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.704+0000 7f74009db640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f73fc0cc9a0 con 0x7f73fc161780 2024-04-02T13:24:56.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.704+0000 7f74009db640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f73fc0ccc80 con 0x7f73fc0ceed0 2024-04-02T13:24:56.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.704+0000 7f73f9d9b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f73fc0cf5b0 0x7f73fc0ce7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:45230/0 (socket says 172.21.15.44:45230) 2024-04-02T13:24:56.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.704+0000 7f73f9d9b640 1 -- 172.21.15.44:0/210589062 learned_addr learned my addr 172.21.15.44:0/210589062 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:56.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73fa59c640 1 --2- 172.21.15.44:0/210589062 >> v2:172.21.15.69:3300/0 conn(0x7f73fc161780 0x7f73fc06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73fad9d640 1 --2- 172.21.15.44:0/210589062 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0ceed0 0x7f73fc0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 997114940 0 0) 0x7f73fc0cc6c0 con 0x7f73fc0cf5b0 2024-04-02T13:24:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73e8003520 con 0x7f73fc0cf5b0 2024-04-02T13:24:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2635563097 0 0) 0x7f73e8003520 con 0x7f73fc0cf5b0 2024-04-02T13:24:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0ceed0 msgr2=0x7f73fc0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 --2- 172.21.15.44:0/210589062 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0ceed0 0x7f73fc0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 >> v2:172.21.15.69:3300/0 conn(0x7f73fc161780 msgr2=0x7f73fc06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 --2- 172.21.15.44:0/210589062 >> v2:172.21.15.69:3300/0 conn(0x7f73fc161780 0x7f73fc06cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73fc0ccf60 con 0x7f73fc0cf5b0 2024-04-02T13:24:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f73e4001670 con 0x7f73fc0cf5b0 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 >> v1:172.21.15.44:6789/0 conn(0x7f73fc0cf5b0 legacy=0x7f73fc0ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 --2- 172.21.15.44:0/210589062 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0ceed0 0x7f73e80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 --2- 172.21.15.44:0/210589062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e8003f30 0x7f73e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 --2- 172.21.15.44:0/210589062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e8004d40 0x7f73e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f73fc0cc9a0 con 0x7f73e8004d40 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f73fc0ccc80 con 0x7f73e8003f30 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f73e800ba10 con 0x7f73fc0ceed0 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73fa59c640 1 --2- 172.21.15.44:0/210589062 >> v2:172.21.15.69:3300/0 conn(0x7f73fc161780 0x7f73fc06cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73fad9d640 1 --2- 172.21.15.44:0/210589062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e8004d40 0x7f73e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73f9d9b640 1 --2- 172.21.15.44:0/210589062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e8003f30 0x7f73e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73fa59c640 1 --2- 172.21.15.44:0/210589062 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0ceed0 0x7f73e80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73fad9d640 1 -- 172.21.15.44:0/210589062 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0ceed0 msgr2=0x7f73e80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73fad9d640 1 --2- 172.21.15.44:0/210589062 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0ceed0 0x7f73e80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73fad9d640 1 -- 172.21.15.44:0/210589062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e8003f30 msgr2=0x7f73e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73fad9d640 1 --2- 172.21.15.44:0/210589062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e8003f30 0x7f73e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.705+0000 7f73fad9d640 1 -- 172.21.15.44:0/210589062 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e8003520 con 0x7f73e8004d40 2024-04-02T13:24:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.706+0000 7f73fa59c640 1 --2- 172.21.15.44:0/210589062 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0ceed0 0x7f73e80039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.706+0000 7f73fad9d640 1 --2- 172.21.15.44:0/210589062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e8004d40 0x7f73e80051c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f73f4009ac0 tx=0x7f73e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511f1ba2de23bfed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.706+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4039070 con 0x7f73e8004d40 2024-04-02T13:24:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.706+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73f402c680 con 0x7f73e8004d40 2024-04-02T13:24:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.706+0000 7f73f959a640 1 -- 172.21.15.44:0/210589062 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4009690 con 0x7f73e8004d40 2024-04-02T13:24:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.707+0000 7f74009db640 1 -- 172.21.15.44:0/210589062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e8004d40 msgr2=0x7f73e80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.707+0000 7f74009db640 1 --2- 172.21.15.44:0/210589062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e8004d40 0x7f73e80051c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f73f4009ac0 tx=0x7f73e8005830 comp rx=0 tx=0).stop 2024-04-02T13:24:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.707+0000 7f73fad9d640 1 -- 172.21.15.44:0/210589062 reap_dead start 2024-04-02T13:24:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.707+0000 7f74009db640 1 -- 172.21.15.44:0/210589062 shutdown_connections 2024-04-02T13:24:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.707+0000 7f74009db640 1 -- 172.21.15.44:0/210589062 >> 172.21.15.44:0/210589062 conn(0x7f73fc073850 msgr2=0x7f73fc09aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:56.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.707+0000 7f74009db640 1 -- 172.21.15.44:0/210589062 shutdown_connections 2024-04-02T13:24:56.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.707+0000 7f74009db640 1 -- 172.21.15.44:0/210589062 wait complete. 2024-04-02T13:24:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.708+0000 7f74009db640 1 Processor -- start 2024-04-02T13:24:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.708+0000 7f74009db640 1 -- start start 2024-04-02T13:24:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f74009db640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e8004d40 0x7f73fc15cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f74009db640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73fc15d0c0 0x7f73fc0d0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f74009db640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0d06c0 0x7f73fc0d0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f74009db640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f73fc06d530 con 0x7f73e8004d40 2024-04-02T13:24:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f74009db640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f73fc161b60 con 0x7f73fc15d0c0 2024-04-02T13:24:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f74009db640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f73fc06d6b0 con 0x7f73fc0d06c0 2024-04-02T13:24:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f73fa59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e8004d40 0x7f73fc15cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f73fa59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e8004d40 0x7f73fc15cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41798/0 (socket says 172.21.15.44:41798) 2024-04-02T13:24:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f73fa59c640 1 -- 172.21.15.44:0/2668918303 learned_addr learned my addr 172.21.15.44:0/2668918303 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f73fad9d640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0d06c0 0x7f73fc0d0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f73f9d9b640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73fc15d0c0 0x7f73fc0d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f73fa59c640 1 -- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0d06c0 msgr2=0x7f73fc0d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f73fa59c640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0d06c0 0x7f73fc0d0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f73fa59c640 1 -- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73fc15d0c0 msgr2=0x7f73fc0d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f73fa59c640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73fc15d0c0 0x7f73fc0d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f73fa59c640 1 -- 172.21.15.44:0/2668918303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73fc154f70 con 0x7f73e8004d40 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f73f9d9b640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73fc15d0c0 0x7f73fc0d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.709+0000 7f73fad9d640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0d06c0 0x7f73fc0d0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.710+0000 7f73fa59c640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e8004d40 0x7f73fc15cb80 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f73ec011530 tx=0x7f73ec0119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.710+0000 7f73f2ffd640 1 -- 172.21.15.44:0/2668918303 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec01a020 con 0x7f73e8004d40 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.710+0000 7f73f2ffd640 1 -- 172.21.15.44:0/2668918303 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73ec016680 con 0x7f73e8004d40 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.710+0000 7f74009db640 1 -- 172.21.15.44:0/2668918303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73fc155250 con 0x7f73e8004d40 2024-04-02T13:24:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.710+0000 7f73f2ffd640 1 -- 172.21.15.44:0/2668918303 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec016a30 con 0x7f73e8004d40 2024-04-02T13:24:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.710+0000 7f74009db640 1 -- 172.21.15.44:0/2668918303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73fc155810 con 0x7f73e8004d40 2024-04-02T13:24:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.711+0000 7f74009db640 1 -- 172.21.15.44:0/2668918303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73fc068c00 con 0x7f73e8004d40 2024-04-02T13:24:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.711+0000 7f73f2ffd640 1 -- 172.21.15.44:0/2668918303 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f73ec01d020 con 0x7f73e8004d40 2024-04-02T13:24:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.712+0000 7f73f2ffd640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73dc06c920 0x7f73dc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.712+0000 7f73f2ffd640 1 -- 172.21.15.44:0/2668918303 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f73ec08ced0 con 0x7f73e8004d40 2024-04-02T13:24:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.712+0000 7f73f9d9b640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73dc06c920 0x7f73dc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.713+0000 7f73f9d9b640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73dc06c920 0x7f73dc06ede0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f73e4000c00 tx=0x7f73e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.717+0000 7f73f2ffd640 1 -- 172.21.15.44:0/2668918303 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f73ec024020 con 0x7f73e8004d40 2024-04-02T13:24:56.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:56.940+0000 7f74009db640 1 -- 172.21.15.44:0/2668918303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f73fc15d4a0 con 0x7f73e8004d40 2024-04-02T13:24:57.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.583+0000 7f73f2ffd640 1 -- 172.21.15.44:0/2668918303 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7f73ec05b170 con 0x7f73e8004d40 2024-04-02T13:24:57.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.586+0000 7f74009db640 1 -- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73dc06c920 msgr2=0x7f73dc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:57.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.586+0000 7f74009db640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73dc06c920 0x7f73dc06ede0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f73e4000c00 tx=0x7f73e4009040 comp rx=0 tx=0).stop 2024-04-02T13:24:57.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.586+0000 7f74009db640 1 -- 172.21.15.44:0/2668918303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e8004d40 msgr2=0x7f73fc15cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:57.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.586+0000 7f74009db640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e8004d40 0x7f73fc15cb80 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f73ec011530 tx=0x7f73ec0119f0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.586+0000 7f74009db640 1 -- 172.21.15.44:0/2668918303 shutdown_connections 2024-04-02T13:24:57.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.586+0000 7f74009db640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73dc06c920 0x7f73dc06ede0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.586+0000 7f74009db640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73fc0d06c0 0x7f73fc0d0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.586+0000 7f74009db640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73fc15d0c0 0x7f73fc0d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.586+0000 7f74009db640 1 --2- 172.21.15.44:0/2668918303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73e8004d40 0x7f73fc15cb80 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.586+0000 7f74009db640 1 -- 172.21.15.44:0/2668918303 >> 172.21.15.44:0/2668918303 conn(0x7f73fc073850 msgr2=0x7f73fc0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:57.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.586+0000 7f74009db640 1 -- 172.21.15.44:0/2668918303 shutdown_connections 2024-04-02T13:24:57.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.586+0000 7f74009db640 1 -- 172.21.15.44:0/2668918303 wait complete. 2024-04-02T13:24:57.602 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T13:24:57.603 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:24:57.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.707+0000 7f512cb75640 1 Processor -- start 2024-04-02T13:24:57.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.707+0000 7f512cb75640 1 -- start start 2024-04-02T13:24:57.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f512cb75640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f512815e4a0 0x7f51280aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f512cb75640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f51280d0c50 0x7f51280d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f512cb75640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51280cc600 con 0x7f512806ce40 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f512cb75640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f51280cc8e0 con 0x7f51280d0c50 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f512cb75640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f51280ccbc0 con 0x7f512815e4a0 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f5125d9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f51280d0c50 0x7f51280d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f5126d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f512806ce40 0x7f51280cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:45246/0 (socket says 172.21.15.44:45246) 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f5126d9d640 1 -- 172.21.15.44:0/4201615769 learned_addr learned my addr 172.21.15.44:0/4201615769 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f512659c640 1 --2- 172.21.15.44:0/4201615769 >> v2:172.21.15.69:3300/0 conn(0x7f512815e4a0 0x7f51280aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f5125d9b640 1 -- 172.21.15.44:0/4201615769 >> v2:172.21.15.69:3300/0 conn(0x7f512815e4a0 msgr2=0x7f51280aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f5125d9b640 1 --2- 172.21.15.44:0/4201615769 >> v2:172.21.15.69:3300/0 conn(0x7f512815e4a0 0x7f51280aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f5125d9b640 1 -- 172.21.15.44:0/4201615769 >> v1:172.21.15.44:6789/0 conn(0x7f512806ce40 legacy=0x7f51280cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f5125d9b640 1 -- 172.21.15.44:0/4201615769 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51280ccea0 con 0x7f51280d0c50 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f512559a640 1 -- 172.21.15.44:0/4201615769 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2096128496 0 0) 0x7f51280cc600 con 0x7f512806ce40 2024-04-02T13:24:57.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.708+0000 7f512659c640 1 --2- 172.21.15.44:0/4201615769 >> v2:172.21.15.69:3300/0 conn(0x7f512815e4a0 0x7f51280aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.709+0000 7f5125d9b640 1 --2- 172.21.15.44:0/4201615769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51280d0c50 0x7f51280d10b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f51280d1f80 tx=0x7f511002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf70354f01ed954 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.709+0000 7f512559a640 1 -- 172.21.15.44:0/4201615769 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110035020 con 0x7f51280d0c50 2024-04-02T13:24:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.709+0000 7f512559a640 1 -- 172.21.15.44:0/4201615769 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5110031830 con 0x7f51280d0c50 2024-04-02T13:24:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.709+0000 7f512559a640 1 -- 172.21.15.44:0/4201615769 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110031c00 con 0x7f51280d0c50 2024-04-02T13:24:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.710+0000 7f512cb75640 1 -- 172.21.15.44:0/4201615769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51280d0c50 msgr2=0x7f51280d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.710+0000 7f512cb75640 1 --2- 172.21.15.44:0/4201615769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51280d0c50 0x7f51280d10b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f51280d1f80 tx=0x7f511002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.710+0000 7f512cb75640 1 -- 172.21.15.44:0/4201615769 shutdown_connections 2024-04-02T13:24:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.710+0000 7f512cb75640 1 --2- 172.21.15.44:0/4201615769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51280d0c50 0x7f51280d10b0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.710+0000 7f512cb75640 1 --2- 172.21.15.44:0/4201615769 >> v2:172.21.15.69:3300/0 conn(0x7f512815e4a0 0x7f51280aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.710+0000 7f512cb75640 1 -- 172.21.15.44:0/4201615769 >> 172.21.15.44:0/4201615769 conn(0x7f512809aa20 msgr2=0x7f512809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.710+0000 7f512cb75640 1 -- 172.21.15.44:0/4201615769 shutdown_connections 2024-04-02T13:24:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.710+0000 7f512cb75640 1 -- 172.21.15.44:0/4201615769 wait complete. 2024-04-02T13:24:57.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.711+0000 7f512cb75640 1 Processor -- start 2024-04-02T13:24:57.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.711+0000 7f512cb75640 1 -- start start 2024-04-02T13:24:57.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.712+0000 7f512cb75640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f512806ce40 0x7f512815ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:57.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.712+0000 7f512cb75640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51280d0c50 0x7f51281534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:57.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.712+0000 7f512cb75640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512815e4a0 0x7f51280c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:57.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.712+0000 7f512cb75640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f512815e880 con 0x7f512806ce40 2024-04-02T13:24:57.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.712+0000 7f512cb75640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f51280cd460 con 0x7f512815e4a0 2024-04-02T13:24:57.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.712+0000 7f512cb75640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f51280af130 con 0x7f51280d0c50 2024-04-02T13:24:57.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.712+0000 7f512659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f512806ce40 0x7f512815ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:57.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.712+0000 7f512659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f512806ce40 0x7f512815ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41814/0 (socket says 172.21.15.44:41814) 2024-04-02T13:24:57.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.712+0000 7f512659c640 1 -- 172.21.15.44:0/1830850101 learned_addr learned my addr 172.21.15.44:0/1830850101 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:57.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.712+0000 7f5126d9d640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512815e4a0 0x7f51280c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:57.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.712+0000 7f5125d9b640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51280d0c50 0x7f51281534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:57.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.713+0000 7f512659c640 1 -- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51280d0c50 msgr2=0x7f51281534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:57.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.713+0000 7f512659c640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51280d0c50 0x7f51281534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.713+0000 7f512659c640 1 -- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512815e4a0 msgr2=0x7f51280c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:57.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.713+0000 7f512659c640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512815e4a0 0x7f51280c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.713+0000 7f512659c640 1 -- 172.21.15.44:0/1830850101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5128153a10 con 0x7f512806ce40 2024-04-02T13:24:57.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.713+0000 7f5125d9b640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51280d0c50 0x7f51281534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:57.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.713+0000 7f5126d9d640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512815e4a0 0x7f51280c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:57.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.713+0000 7f512659c640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f512806ce40 0x7f512815ea00 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f511800ce00 tx=0x7f511800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:57.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.713+0000 7f511effd640 1 -- 172.21.15.44:0/1830850101 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511800f890 con 0x7f512806ce40 2024-04-02T13:24:57.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.713+0000 7f511effd640 1 -- 172.21.15.44:0/1830850101 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f511800fa30 con 0x7f512806ce40 2024-04-02T13:24:57.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.713+0000 7f512cb75640 1 -- 172.21.15.44:0/1830850101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f512815c4a0 con 0x7f512806ce40 2024-04-02T13:24:57.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.713+0000 7f511effd640 1 -- 172.21.15.44:0/1830850101 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511800fe40 con 0x7f512806ce40 2024-04-02T13:24:57.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.713+0000 7f512cb75640 1 -- 172.21.15.44:0/1830850101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51280b4f90 con 0x7f512806ce40 2024-04-02T13:24:57.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.714+0000 7f512cb75640 1 -- 172.21.15.44:0/1830850101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51280684f0 con 0x7f512806ce40 2024-04-02T13:24:57.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.714+0000 7f511effd640 1 -- 172.21.15.44:0/1830850101 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5118018400 con 0x7f512806ce40 2024-04-02T13:24:57.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.715+0000 7f511effd640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f510806c8f0 0x7f510806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:57.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.715+0000 7f511effd640 1 -- 172.21.15.44:0/1830850101 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f511808caa0 con 0x7f512806ce40 2024-04-02T13:24:57.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.715+0000 7f5125d9b640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f510806c8f0 0x7f510806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:57.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.715+0000 7f5125d9b640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f510806c8f0 0x7f510806edb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5110036000 tx=0x7f5110008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:57.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.719+0000 7f511effd640 1 -- 172.21.15.44:0/1830850101 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5118014030 con 0x7f512806ce40 2024-04-02T13:24:57.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.947+0000 7f512cb75640 1 -- 172.21.15.44:0/1830850101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51280b0680 con 0x7f512806ce40 2024-04-02T13:24:57.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.948+0000 7f511effd640 1 -- 172.21.15.44:0/1830850101 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5180 (secure 0 0 0) 0x7f511800ae20 con 0x7f512806ce40 2024-04-02T13:24:57.951 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:57.952 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3662927745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3662927745},{"type":"v1","addr":"172.21.15.44:6835","nonce":3662927745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6394,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1053037366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1053037366},{"type":"v1","addr":"172.21.15.69:6836","nonce":1053037366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6987,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2392159648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2392159648},{"type":"v1","addr":"172.21.15.44:6833","nonce":2392159648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:24:53.925714+0000","modified":"2024-04-02T13:24:56.942238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6837/698185867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":698185867},{"type":"v1","addr":"172.21.15.69:6837","nonce":698185867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7002,"qdb_cluster":[7002]},"id":5}]} 2024-04-02T13:24:57.952 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 44 2024-04-02T13:24:57.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.950+0000 7f512cb75640 1 -- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f510806c8f0 msgr2=0x7f510806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:57.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.950+0000 7f512cb75640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f510806c8f0 0x7f510806edb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5110036000 tx=0x7f5110008030 comp rx=0 tx=0).stop 2024-04-02T13:24:57.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.950+0000 7f512cb75640 1 -- 172.21.15.44:0/1830850101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f512806ce40 msgr2=0x7f512815ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:57.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.950+0000 7f512cb75640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f512806ce40 0x7f512815ea00 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f511800ce00 tx=0x7f511800a650 comp rx=0 tx=0).stop 2024-04-02T13:24:57.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.950+0000 7f512cb75640 1 -- 172.21.15.44:0/1830850101 shutdown_connections 2024-04-02T13:24:57.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.950+0000 7f512cb75640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f510806c8f0 0x7f510806edb0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.950+0000 7f512cb75640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f512815e4a0 0x7f51280c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.950+0000 7f512cb75640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51280d0c50 0x7f51281534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.950+0000 7f512cb75640 1 --2- 172.21.15.44:0/1830850101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f512806ce40 0x7f512815ea00 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:57.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.950+0000 7f512cb75640 1 -- 172.21.15.44:0/1830850101 >> 172.21.15.44:0/1830850101 conn(0x7f512809aa20 msgr2=0x7f51280b0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:57.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.950+0000 7f512cb75640 1 -- 172.21.15.44:0/1830850101 shutdown_connections 2024-04-02T13:24:57.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:57.950+0000 7f512cb75640 1 -- 172.21.15.44:0/1830850101 wait complete. 2024-04-02T13:24:57.964 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:24:58.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.063+0000 7fe9a343c640 1 Processor -- start 2024-04-02T13:24:58.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.063+0000 7fe9a343c640 1 -- start start 2024-04-02T13:24:58.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.063+0000 7fe9a343c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe99c15e0f0 0x7fe99c0d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.063+0000 7fe9a343c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe99c06dfe0 con 0x7fe99c0d0630 2024-04-02T13:24:58.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.063+0000 7fe9a343c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe99c06e2c0 con 0x7fe99c06bbe0 2024-04-02T13:24:58.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.063+0000 7fe9a343c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe99c06e5a0 con 0x7fe99c15e0f0 2024-04-02T13:24:58.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe9a09d7640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe99c0d0630 0x7fe99c06b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:45248/0 (socket says 172.21.15.44:45248) 2024-04-02T13:24:58.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe9a09d7640 1 -- 172.21.15.44:0/808525978 learned_addr learned my addr 172.21.15.44:0/808525978 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:58.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2250286773 0 0) 0x7fe99c06dfe0 con 0x7fe99c0d0630 2024-04-02T13:24:58.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe9a11d8640 1 --2- 172.21.15.44:0/808525978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe99c15e0f0 0x7fe99c0d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe988003520 con 0x7fe99c0d0630 2024-04-02T13:24:58.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1773239605 0 0) 0x7fe988003520 con 0x7fe99c0d0630 2024-04-02T13:24:58.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe99c15e0f0 msgr2=0x7fe99c0d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:58.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 --2- 172.21.15.44:0/808525978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe99c15e0f0 0x7fe99c0d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 >> v1:172.21.15.69:6789/0 conn(0x7fe99c06bbe0 legacy=0x7fe99c0add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe99c06e880 con 0x7fe99c0d0630 2024-04-02T13:24:58.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fe98c001670 con 0x7fe99c0d0630 2024-04-02T13:24:58.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 >> v1:172.21.15.44:6789/0 conn(0x7fe99c0d0630 legacy=0x7fe99c06b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 --2- 172.21.15.44:0/808525978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe99c15e0f0 0x7fe9880039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 --2- 172.21.15.44:0/808525978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe988003f60 0x7fe988004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 --2- 172.21.15.44:0/808525978 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe988004d40 0x7fe9880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c06e5a0 con 0x7fe988003f60 2024-04-02T13:24:58.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe98800bb30 con 0x7fe988004d40 2024-04-02T13:24:58.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe98800bd30 con 0x7fe99c15e0f0 2024-04-02T13:24:58.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe9a09d7640 1 --2- 172.21.15.44:0/808525978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe988003f60 0x7fe988004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe9a11d8640 1 --2- 172.21.15.44:0/808525978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe99c15e0f0 0x7fe9880039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.064+0000 7fe9a19d9640 1 --2- 172.21.15.44:0/808525978 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe988004d40 0x7fe9880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.065+0000 7fe9a09d7640 1 -- 172.21.15.44:0/808525978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe99c15e0f0 msgr2=0x7fe9880039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:58.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.065+0000 7fe9a09d7640 1 --2- 172.21.15.44:0/808525978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe99c15e0f0 0x7fe9880039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.065+0000 7fe9a09d7640 1 -- 172.21.15.44:0/808525978 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe988004d40 msgr2=0x7fe9880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:58.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.065+0000 7fe9a09d7640 1 --2- 172.21.15.44:0/808525978 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe988004d40 0x7fe9880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.065+0000 7fe9a09d7640 1 -- 172.21.15.44:0/808525978 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe988003520 con 0x7fe988003f60 2024-04-02T13:24:58.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.065+0000 7fe9a19d9640 1 --2- 172.21.15.44:0/808525978 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe988004d40 0x7fe9880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:58.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.065+0000 7fe9a09d7640 1 --2- 172.21.15.44:0/808525978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe988003f60 0x7fe988004800 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fe98c00c240 tx=0x7fe988005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d36f1dbefb8853 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:58.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.065+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c007cf0 con 0x7fe988003f60 2024-04-02T13:24:58.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.066+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe98c030070 con 0x7fe988003f60 2024-04-02T13:24:58.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.066+0000 7fe993fff640 1 -- 172.21.15.44:0/808525978 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c005d70 con 0x7fe988003f60 2024-04-02T13:24:58.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.066+0000 7fe9a343c640 1 -- 172.21.15.44:0/808525978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe988003f60 msgr2=0x7fe988004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.066+0000 7fe9a343c640 1 --2- 172.21.15.44:0/808525978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe988003f60 0x7fe988004800 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fe98c00c240 tx=0x7fe988005830 comp rx=0 tx=0).stop 2024-04-02T13:24:58.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.066+0000 7fe9a19d9640 1 -- 172.21.15.44:0/808525978 reap_dead start 2024-04-02T13:24:58.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.066+0000 7fe9a343c640 1 -- 172.21.15.44:0/808525978 shutdown_connections 2024-04-02T13:24:58.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.066+0000 7fe9a343c640 1 -- 172.21.15.44:0/808525978 >> 172.21.15.44:0/808525978 conn(0x7fe99c0b2980 msgr2=0x7fe99c0b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:58.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.066+0000 7fe9a343c640 1 -- 172.21.15.44:0/808525978 shutdown_connections 2024-04-02T13:24:58.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.066+0000 7fe9a343c640 1 -- 172.21.15.44:0/808525978 wait complete. 2024-04-02T13:24:58.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.067+0000 7fe9a343c640 1 Processor -- start 2024-04-02T13:24:58.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.067+0000 7fe9a343c640 1 -- start start 2024-04-02T13:24:58.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a343c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe988003f60 0x7fe99c156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a343c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c159da0 0x7fe99c0ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a343c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe99c0c7e80 0x7fe99c0c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a343c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c15e4d0 con 0x7fe988003f60 2024-04-02T13:24:58.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a343c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c06eda0 con 0x7fe99c159da0 2024-04-02T13:24:58.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a343c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe99c15e650 con 0x7fe99c0c7e80 2024-04-02T13:24:58.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a11d8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe988003f60 0x7fe99c156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a11d8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe988003f60 0x7fe99c156dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41824/0 (socket says 172.21.15.44:41824) 2024-04-02T13:24:58.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a11d8640 1 -- 172.21.15.44:0/3432824989 learned_addr learned my addr 172.21.15.44:0/3432824989 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:58.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a09d7640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c159da0 0x7fe99c0ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a19d9640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe99c0c7e80 0x7fe99c0c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a11d8640 1 -- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe99c0c7e80 msgr2=0x7fe99c0c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a11d8640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe99c0c7e80 0x7fe99c0c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a11d8640 1 -- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c159da0 msgr2=0x7fe99c0ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a11d8640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c159da0 0x7fe99c0ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.068+0000 7fe9a11d8640 1 -- 172.21.15.44:0/3432824989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe99c0b3860 con 0x7fe988003f60 2024-04-02T13:24:58.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.069+0000 7fe9a11d8640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe988003f60 0x7fe99c156dd0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fe9980119d0 tx=0x7fe998011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:58.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.069+0000 7fe991ffb640 1 -- 172.21.15.44:0/3432824989 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99800fb50 con 0x7fe988003f60 2024-04-02T13:24:58.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.069+0000 7fe991ffb640 1 -- 172.21.15.44:0/3432824989 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe99800fcf0 con 0x7fe988003f60 2024-04-02T13:24:58.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.069+0000 7fe991ffb640 1 -- 172.21.15.44:0/3432824989 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99800fea0 con 0x7fe988003f60 2024-04-02T13:24:58.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.069+0000 7fe9a343c640 1 -- 172.21.15.44:0/3432824989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe99c0b3b40 con 0x7fe988003f60 2024-04-02T13:24:58.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.069+0000 7fe9a343c640 1 -- 172.21.15.44:0/3432824989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe99c0b4100 con 0x7fe988003f60 2024-04-02T13:24:58.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.070+0000 7fe9a343c640 1 -- 172.21.15.44:0/3432824989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe99c0684f0 con 0x7fe988003f60 2024-04-02T13:24:58.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.070+0000 7fe991ffb640 1 -- 172.21.15.44:0/3432824989 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe998012040 con 0x7fe988003f60 2024-04-02T13:24:58.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.071+0000 7fe991ffb640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe98006c920 0x7fe98006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.071+0000 7fe991ffb640 1 -- 172.21.15.44:0/3432824989 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fe99808c370 con 0x7fe988003f60 2024-04-02T13:24:58.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.071+0000 7fe9a09d7640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe98006c920 0x7fe98006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.071+0000 7fe9a09d7640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe98006c920 0x7fe98006ede0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe98c034000 tx=0x7fe98c007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:58.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.075+0000 7fe991ffb640 1 -- 172.21.15.44:0/3432824989 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe998019030 con 0x7fe988003f60 2024-04-02T13:24:58.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.261+0000 7fe9a343c640 1 -- 172.21.15.44:0/3432824989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe99c0727f0 con 0x7fe988003f60 2024-04-02T13:24:58.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.262+0000 7fe991ffb640 1 -- 172.21.15.44:0/3432824989 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+16897 (secure 0 0 0) 0x7fe99805a6a0 con 0x7fe988003f60 2024-04-02T13:24:58.266 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:58.266 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":79,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:24:55.564387+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:24:52.202720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-02T13:24:52.916529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:24:58.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.265+0000 7fe9a343c640 1 -- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe98006c920 msgr2=0x7fe98006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.265+0000 7fe9a343c640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe98006c920 0x7fe98006ede0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe98c034000 tx=0x7fe98c007000 comp rx=0 tx=0).stop 2024-04-02T13:24:58.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.265+0000 7fe9a343c640 1 -- 172.21.15.44:0/3432824989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe988003f60 msgr2=0x7fe99c156dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.265+0000 7fe9a343c640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe988003f60 0x7fe99c156dd0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fe9980119d0 tx=0x7fe998011e90 comp rx=0 tx=0).stop 2024-04-02T13:24:58.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.265+0000 7fe9a343c640 1 -- 172.21.15.44:0/3432824989 shutdown_connections 2024-04-02T13:24:58.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.265+0000 7fe9a343c640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe98006c920 0x7fe98006ede0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.265+0000 7fe9a343c640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe99c0c7e80 0x7fe99c0c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.265+0000 7fe9a343c640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c159da0 0x7fe99c0ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.265+0000 7fe9a343c640 1 --2- 172.21.15.44:0/3432824989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe988003f60 0x7fe99c156dd0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.265+0000 7fe9a343c640 1 -- 172.21.15.44:0/3432824989 >> 172.21.15.44:0/3432824989 conn(0x7fe99c0b2980 msgr2=0x7fe99c0af540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:58.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.265+0000 7fe9a343c640 1 -- 172.21.15.44:0/3432824989 shutdown_connections 2024-04-02T13:24:58.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.265+0000 7fe9a343c640 1 -- 172.21.15.44:0/3432824989 wait complete. 2024-04-02T13:24:58.279 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T13:24:58.279 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:24:58.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.377+0000 7f0cc3996640 1 Processor -- start 2024-04-02T13:24:58.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.377+0000 7f0cc3996640 1 -- start start 2024-04-02T13:24:58.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cc3996640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cbc0cece0 0x7f0cbc0cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cc3996640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0cbc06ece0 0x7f0cbc0acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cc3996640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cbc0cf600 con 0x7f0cbc0d09f0 2024-04-02T13:24:58.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cc3996640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0cbc06d9e0 con 0x7f0cbc06ece0 2024-04-02T13:24:58.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cc3996640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0cbc06dc40 con 0x7f0cbc0cece0 2024-04-02T13:24:58.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cc0f31640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0cbc0d09f0 0x7f0cbc06d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:45254/0 (socket says 172.21.15.44:45254) 2024-04-02T13:24:58.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cc0f31640 1 -- 172.21.15.44:0/2528925557 learned_addr learned my addr 172.21.15.44:0/2528925557 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:24:58.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cc1732640 1 --2- 172.21.15.44:0/2528925557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cbc0cece0 0x7f0cbc0cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1189576269 0 0) 0x7f0cbc0cf600 con 0x7f0cbc0d09f0 2024-04-02T13:24:58.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cc1f33640 1 --2- 172.21.15.44:0/2528925557 >> v2:172.21.15.69:3300/0 conn(0x7f0cbc06ece0 0x7f0cbc0acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ca8003520 con 0x7f0cbc0d09f0 2024-04-02T13:24:58.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3094806385 0 0) 0x7f0ca8003520 con 0x7f0cbc0d09f0 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cbc0cece0 msgr2=0x7f0cbc0cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cb3fff640 1 --2- 172.21.15.44:0/2528925557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cbc0cece0 0x7f0cbc0cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 >> v2:172.21.15.69:3300/0 conn(0x7f0cbc06ece0 msgr2=0x7f0cbc0acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cb3fff640 1 --2- 172.21.15.44:0/2528925557 >> v2:172.21.15.69:3300/0 conn(0x7f0cbc06ece0 0x7f0cbc0acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cbc06df20 con 0x7f0cbc0d09f0 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f0cac001670 con 0x7f0cbc0d09f0 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 >> v1:172.21.15.44:6789/0 conn(0x7f0cbc0d09f0 legacy=0x7f0cbc06d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.378+0000 7f0cb3fff640 1 --2- 172.21.15.44:0/2528925557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cbc0cece0 0x7f0ca80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.379+0000 7f0cb3fff640 1 --2- 172.21.15.44:0/2528925557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ca8003f30 0x7f0ca8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.379+0000 7f0cb3fff640 1 --2- 172.21.15.44:0/2528925557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ca8004d40 0x7f0ca80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.379+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc06d9e0 con 0x7f0ca8004d40 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.379+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc06dc40 con 0x7f0ca8003f30 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.379+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca800dbd0 con 0x7f0cbc0cece0 2024-04-02T13:24:58.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.379+0000 7f0cc1f33640 1 --2- 172.21.15.44:0/2528925557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ca8004d40 0x7f0ca80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.379+0000 7f0cc1732640 1 --2- 172.21.15.44:0/2528925557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cbc0cece0 0x7f0ca80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.379+0000 7f0cc0f31640 1 --2- 172.21.15.44:0/2528925557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ca8003f30 0x7f0ca8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.379+0000 7f0cc1f33640 1 -- 172.21.15.44:0/2528925557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cbc0cece0 msgr2=0x7f0ca80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:58.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.379+0000 7f0cc1f33640 1 --2- 172.21.15.44:0/2528925557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cbc0cece0 0x7f0ca80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.379+0000 7f0cc1f33640 1 -- 172.21.15.44:0/2528925557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ca8003f30 msgr2=0x7f0ca8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:58.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.379+0000 7f0cc1f33640 1 --2- 172.21.15.44:0/2528925557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ca8003f30 0x7f0ca8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.379+0000 7f0cc1f33640 1 -- 172.21.15.44:0/2528925557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca8003520 con 0x7f0ca8004d40 2024-04-02T13:24:58.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.380+0000 7f0cc1f33640 1 --2- 172.21.15.44:0/2528925557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ca8004d40 0x7f0ca80051c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4009f90 tx=0x7f0ca8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc3cfa221dbf7c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:58.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.380+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb4039070 con 0x7f0ca8004d40 2024-04-02T13:24:58.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.380+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cb402cc00 con 0x7f0ca8004d40 2024-04-02T13:24:58.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.380+0000 7f0cb3fff640 1 -- 172.21.15.44:0/2528925557 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb4009c10 con 0x7f0ca8004d40 2024-04-02T13:24:58.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.380+0000 7f0cc3996640 1 -- 172.21.15.44:0/2528925557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ca8004d40 msgr2=0x7f0ca80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.380+0000 7f0cc3996640 1 --2- 172.21.15.44:0/2528925557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ca8004d40 0x7f0ca80051c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4009f90 tx=0x7f0ca8005830 comp rx=0 tx=0).stop 2024-04-02T13:24:58.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.380+0000 7f0cc1f33640 1 -- 172.21.15.44:0/2528925557 reap_dead start 2024-04-02T13:24:58.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.380+0000 7f0cc3996640 1 -- 172.21.15.44:0/2528925557 shutdown_connections 2024-04-02T13:24:58.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.380+0000 7f0cc3996640 1 -- 172.21.15.44:0/2528925557 >> 172.21.15.44:0/2528925557 conn(0x7f0cbc0cfcd0 msgr2=0x7f0cbc0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.380+0000 7f0cc3996640 1 -- 172.21.15.44:0/2528925557 shutdown_connections 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.381+0000 7f0cc3996640 1 -- 172.21.15.44:0/2528925557 wait complete. 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.382+0000 7f0cc3996640 1 Processor -- start 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.382+0000 7f0cc3996640 1 -- start start 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.382+0000 7f0cc3996640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cbc149880 0x7f0cbc1e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.382+0000 7f0cc3996640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0cbc1e57a0 0x7f0cbc15c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.382+0000 7f0cc3996640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cbc15c600 0x7f0cbc15ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.382+0000 7f0cc3996640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc06e7c0 con 0x7f0cbc1e57a0 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.382+0000 7f0cc3996640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc06e4e0 con 0x7f0cbc149880 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.382+0000 7f0cc3996640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0cbc15b220 con 0x7f0cbc15c600 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.382+0000 7f0cc0f31640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0cbc1e57a0 0x7f0cbc15c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.382+0000 7f0cc0f31640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0cbc1e57a0 0x7f0cbc15c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41842/0 (socket says 172.21.15.44:41842) 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.382+0000 7f0cc0f31640 1 -- 172.21.15.44:0/1703015153 learned_addr learned my addr 172.21.15.44:0/1703015153 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cc1f33640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cbc15c600 0x7f0cbc15ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cc1732640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cbc149880 0x7f0cbc1e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cc0f31640 1 -- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cbc15c600 msgr2=0x7f0cbc15ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cc0f31640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cbc15c600 0x7f0cbc15ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cc0f31640 1 -- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cbc149880 msgr2=0x7f0cbc1e5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cc0f31640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cbc149880 0x7f0cbc1e5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cc0f31640 1 -- 172.21.15.44:0/1703015153 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cbc149c60 con 0x7f0cbc1e57a0 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cc1732640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cbc149880 0x7f0cbc1e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cc0f31640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0cbc1e57a0 0x7f0cbc15c0c0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f0cac004840 tx=0x7f0cac007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cb1ffb640 1 -- 172.21.15.44:0/1703015153 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac00f960 con 0x7f0cbc1e57a0 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cb1ffb640 1 -- 172.21.15.44:0/1703015153 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cac018070 con 0x7f0cbc1e57a0 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cc3996640 1 -- 172.21.15.44:0/1703015153 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cbc09b7a0 con 0x7f0cbc1e57a0 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cb1ffb640 1 -- 172.21.15.44:0/1703015153 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac00fd60 con 0x7f0cbc1e57a0 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.383+0000 7f0cc3996640 1 -- 172.21.15.44:0/1703015153 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cbc09bd60 con 0x7f0cbc1e57a0 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.384+0000 7f0cc3996640 1 -- 172.21.15.44:0/1703015153 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cbc074f20 con 0x7f0cbc1e57a0 2024-04-02T13:24:58.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.385+0000 7f0cb1ffb640 1 -- 172.21.15.44:0/1703015153 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0cac01d460 con 0x7f0cbc1e57a0 2024-04-02T13:24:58.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.385+0000 7f0cb1ffb640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ca006c920 0x7f0ca006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.385+0000 7f0cb1ffb640 1 -- 172.21.15.44:0/1703015153 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f0cac08be70 con 0x7f0cbc1e57a0 2024-04-02T13:24:58.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.386+0000 7f0cc1732640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ca006c920 0x7f0ca006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.386+0000 7f0cc1732640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ca006c920 0x7f0ca006ede0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009bc0 tx=0x7f0cb8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:58.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.390+0000 7f0cb1ffb640 1 -- 172.21.15.44:0/1703015153 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0cac05a070 con 0x7f0cbc1e57a0 2024-04-02T13:24:58.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.570+0000 7f0cc3996640 1 -- 172.21.15.44:0/1703015153 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0cbc0684f0 con 0x7f0ca006c920 2024-04-02T13:24:58.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.576+0000 7f0cb1ffb640 1 -- 172.21.15.44:0/1703015153 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246754 (secure 0 0 0) 0x7f0cbc0684f0 con 0x7f0ca006c920 2024-04-02T13:24:58.582 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:58.584 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:24:58.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.579+0000 7f0cc3996640 1 -- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ca006c920 msgr2=0x7f0ca006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.579+0000 7f0cc3996640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ca006c920 0x7f0ca006ede0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009bc0 tx=0x7f0cb8008040 comp rx=0 tx=0).stop 2024-04-02T13:24:58.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.580+0000 7f0cc3996640 1 -- 172.21.15.44:0/1703015153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0cbc1e57a0 msgr2=0x7f0cbc15c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.580+0000 7f0cc3996640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0cbc1e57a0 0x7f0cbc15c0c0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f0cac004840 tx=0x7f0cac007720 comp rx=0 tx=0).stop 2024-04-02T13:24:58.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.580+0000 7f0cc3996640 1 -- 172.21.15.44:0/1703015153 shutdown_connections 2024-04-02T13:24:58.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.580+0000 7f0cc3996640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ca006c920 0x7f0ca006ede0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.580+0000 7f0cc3996640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cbc15c600 0x7f0cbc15ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.580+0000 7f0cc3996640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0cbc1e57a0 0x7f0cbc15c0c0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.580+0000 7f0cc3996640 1 --2- 172.21.15.44:0/1703015153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cbc149880 0x7f0cbc1e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.580+0000 7f0cc3996640 1 -- 172.21.15.44:0/1703015153 >> 172.21.15.44:0/1703015153 conn(0x7f0cbc0cfcd0 msgr2=0x7f0cbc159de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:58.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.580+0000 7f0cc3996640 1 -- 172.21.15.44:0/1703015153 shutdown_connections 2024-04-02T13:24:58.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.580+0000 7f0cc3996640 1 -- 172.21.15.44:0/1703015153 wait complete. 2024-04-02T13:24:58.594 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-04-02T13:24:56.941220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237144,"kb_used_data":8484,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754737576,"statfs":{"total":773094113280,"available":772851277824,"internally_reserved":0,"allocated":8687616,"data_stored":4582735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.391544"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567106+0000","last_change":"2024-04-02T13:24:53.567106+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567106+0000","last_undegraded":"2024-04-02T13:24:53.567106+0000","last_fullsized":"2024-04-02T13:24:53.567106+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.569705+0000","last_change":"2024-04-02T13:24:53.569705+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.569705+0000","last_undegraded":"2024-04-02T13:24:53.569705+0000","last_fullsized":"2024-04-02T13:24:53.569705+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560847+0000","last_change":"2024-04-02T13:24:53.560847+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560847+0000","last_undegraded":"2024-04-02T13:24:53.560847+0000","last_fullsized":"2024-04-02T13:24:53.560847+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.562054+0000","last_change":"2024-04-02T13:24:53.562054+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.562054+0000","last_undegraded":"2024-04-02T13:24:53.562054+0000","last_fullsized":"2024-04-02T13:24:53.562054+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565779+0000","last_change":"2024-04-02T13:24:53.565779+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565779+0000","last_undegraded":"2024-04-02T13:24:53.565779+0000","last_fullsized":"2024-04-02T13:24:53.565779+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.564580+0000","last_change":"2024-04-02T13:24:53.564580+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.564580+0000","last_undegraded":"2024-04-02T13:24:53.564580+0000","last_fullsized":"2024-04-02T13:24:53.564580+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561157+0000","last_change":"2024-04-02T13:24:53.561157+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561157+0000","last_undegraded":"2024-04-02T13:24:53.561157+0000","last_fullsized":"2024-04-02T13:24:53.561157+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567092+0000","last_change":"2024-04-02T13:24:53.567092+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567092+0000","last_undegraded":"2024-04-02T13:24:53.567092+0000","last_fullsized":"2024-04-02T13:24:53.567092+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566497+0000","last_change":"2024-04-02T13:24:53.566497+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566497+0000","last_undegraded":"2024-04-02T13:24:53.566497+0000","last_fullsized":"2024-04-02T13:24:53.566497+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.562553+0000","last_change":"2024-04-02T13:24:53.562553+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.562553+0000","last_undegraded":"2024-04-02T13:24:53.562553+0000","last_fullsized":"2024-04-02T13:24:53.562553+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.570289+0000","last_change":"2024-04-02T13:24:53.570289+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.570289+0000","last_undegraded":"2024-04-02T13:24:53.570289+0000","last_fullsized":"2024-04-02T13:24:53.570289+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561617+0000","last_change":"2024-04-02T13:24:53.561617+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561617+0000","last_undegraded":"2024-04-02T13:24:53.561617+0000","last_fullsized":"2024-04-02T13:24:53.561617+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565353+0000","last_change":"2024-04-02T13:24:53.565353+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565353+0000","last_undegraded":"2024-04-02T13:24:53.565353+0000","last_fullsized":"2024-04-02T13:24:53.565353+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.564097+0000","last_change":"2024-04-02T13:24:53.564097+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.564097+0000","last_undegraded":"2024-04-02T13:24:53.564097+0000","last_fullsized":"2024-04-02T13:24:53.564097+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567769+0000","last_change":"2024-04-02T13:24:53.567769+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567769+0000","last_undegraded":"2024-04-02T13:24:53.567769+0000","last_fullsized":"2024-04-02T13:24:53.567769+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566403+0000","last_change":"2024-04-02T13:24:53.566403+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566403+0000","last_undegraded":"2024-04-02T13:24:53.566403+0000","last_fullsized":"2024-04-02T13:24:53.566403+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565036+0000","last_change":"2024-04-02T13:24:53.565036+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565036+0000","last_undegraded":"2024-04-02T13:24:53.565036+0000","last_fullsized":"2024-04-02T13:24:53.565036+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560762+0000","last_change":"2024-04-02T13:24:53.560762+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560762+0000","last_undegraded":"2024-04-02T13:24:53.560762+0000","last_fullsized":"2024-04-02T13:24:53.560762+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561514+0000","last_change":"2024-04-02T13:24:53.561514+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561514+0000","last_undegraded":"2024-04-02T13:24:53.561514+0000","last_fullsized":"2024-04-02T13:24:53.561514+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567531+0000","last_change":"2024-04-02T13:24:53.567531+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567531+0000","last_undegraded":"2024-04-02T13:24:53.567531+0000","last_fullsized":"2024-04-02T13:24:53.567531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567537+0000","last_change":"2024-04-02T13:24:53.567537+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567537+0000","last_undegraded":"2024-04-02T13:24:53.567537+0000","last_fullsized":"2024-04-02T13:24:53.567537+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565239+0000","last_change":"2024-04-02T13:24:53.565239+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565239+0000","last_undegraded":"2024-04-02T13:24:53.565239+0000","last_fullsized":"2024-04-02T13:24:53.565239+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566185+0000","last_change":"2024-04-02T13:24:53.566185+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566185+0000","last_undegraded":"2024-04-02T13:24:53.566185+0000","last_fullsized":"2024-04-02T13:24:53.566185+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567230+0000","last_change":"2024-04-02T13:24:53.567230+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567230+0000","last_undegraded":"2024-04-02T13:24:53.567230+0000","last_fullsized":"2024-04-02T13:24:53.567230+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.564364+0000","last_change":"2024-04-02T13:24:53.564364+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.564364+0000","last_undegraded":"2024-04-02T13:24:53.564364+0000","last_fullsized":"2024-04-02T13:24:53.564364+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.562181+0000","last_change":"2024-04-02T13:24:53.562181+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.562181+0000","last_undegraded":"2024-04-02T13:24:53.562181+0000","last_fullsized":"2024-04-02T13:24:53.562181+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566599+0000","last_change":"2024-04-02T13:24:53.566599+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566599+0000","last_undegraded":"2024-04-02T13:24:53.566599+0000","last_fullsized":"2024-04-02T13:24:53.566599+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566535+0000","last_change":"2024-04-02T13:24:53.566535+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566535+0000","last_undegraded":"2024-04-02T13:24:53.566535+0000","last_fullsized":"2024-04-02T13:24:53.566535+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567572+0000","last_change":"2024-04-02T13:24:53.567572+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567572+0000","last_undegraded":"2024-04-02T13:24:53.567572+0000","last_fullsized":"2024-04-02T13:24:53.567572+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566012+0000","last_change":"2024-04-02T13:24:53.566012+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566012+0000","last_undegraded":"2024-04-02T13:24:53.566012+0000","last_fullsized":"2024-04-02T13:24:53.566012+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560614+0000","last_change":"2024-04-02T13:24:53.560614+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560614+0000","last_undegraded":"2024-04-02T13:24:53.560614+0000","last_fullsized":"2024-04-02T13:24:53.560614+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565897+0000","last_change":"2024-04-02T13:24:53.565897+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565897+0000","last_undegraded":"2024-04-02T13:24:53.565897+0000","last_fullsized":"2024-04-02T13:24:53.565897+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560034+0000","last_change":"2024-04-02T13:24:53.560034+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560034+0000","last_undegraded":"2024-04-02T13:24:53.560034+0000","last_fullsized":"2024-04-02T13:24:53.560034+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579157+0000","last_change":"2024-04-02T13:24:53.573023+0000","last_active":"2024-04-02T13:24:53.579157+0000","last_peered":"2024-04-02T13:24:53.579157+0000","last_clean":"2024-04-02T13:24:53.579157+0000","last_became_active":"2024-04-02T13:24:53.571627+0000","last_became_peered":"2024-04-02T13:24:53.571627+0000","last_unstale":"2024-04-02T13:24:53.579157+0000","last_undegraded":"2024-04-02T13:24:53.579157+0000","last_fullsized":"2024-04-02T13:24:53.579157+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:37:37.314207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567025+0000","last_change":"2024-04-02T13:24:53.567025+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567025+0000","last_undegraded":"2024-04-02T13:24:53.567025+0000","last_fullsized":"2024-04-02T13:24:53.567025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.574869+0000","last_change":"2024-04-02T13:24:53.573555+0000","last_active":"2024-04-02T13:24:53.574869+0000","last_peered":"2024-04-02T13:24:53.574869+0000","last_clean":"2024-04-02T13:24:53.574869+0000","last_became_active":"2024-04-02T13:24:53.573158+0000","last_became_peered":"2024-04-02T13:24:53.573158+0000","last_unstale":"2024-04-02T13:24:53.574869+0000","last_undegraded":"2024-04-02T13:24:53.574869+0000","last_fullsized":"2024-04-02T13:24:53.574869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:51.244520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.568084+0000","last_change":"2024-04-02T13:24:53.568084+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.568084+0000","last_undegraded":"2024-04-02T13:24:53.568084+0000","last_fullsized":"2024-04-02T13:24:53.568084+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579904+0000","last_change":"2024-04-02T13:24:53.578788+0000","last_active":"2024-04-02T13:24:53.579904+0000","last_peered":"2024-04-02T13:24:53.579904+0000","last_clean":"2024-04-02T13:24:53.579904+0000","last_became_active":"2024-04-02T13:24:53.578523+0000","last_became_peered":"2024-04-02T13:24:53.578523+0000","last_unstale":"2024-04-02T13:24:53.579904+0000","last_undegraded":"2024-04-02T13:24:53.579904+0000","last_fullsized":"2024-04-02T13:24:53.579904+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:23:42.359046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567122+0000","last_change":"2024-04-02T13:24:53.567122+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567122+0000","last_undegraded":"2024-04-02T13:24:53.567122+0000","last_fullsized":"2024-04-02T13:24:53.567122+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.581991+0000","last_change":"2024-04-02T13:24:53.575848+0000","last_active":"2024-04-02T13:24:53.581991+0000","last_peered":"2024-04-02T13:24:53.581991+0000","last_clean":"2024-04-02T13:24:53.581991+0000","last_became_active":"2024-04-02T13:24:53.574537+0000","last_became_peered":"2024-04-02T13:24:53.574537+0000","last_unstale":"2024-04-02T13:24:53.581991+0000","last_undegraded":"2024-04-02T13:24:53.581991+0000","last_fullsized":"2024-04-02T13:24:53.581991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:06.116927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.564902+0000","last_change":"2024-04-02T13:24:53.564902+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.564902+0000","last_undegraded":"2024-04-02T13:24:53.564902+0000","last_fullsized":"2024-04-02T13:24:53.564902+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.582044+0000","last_change":"2024-04-02T13:24:53.574172+0000","last_active":"2024-04-02T13:24:53.582044+0000","last_peered":"2024-04-02T13:24:53.582044+0000","last_clean":"2024-04-02T13:24:53.582044+0000","last_became_active":"2024-04-02T13:24:53.573883+0000","last_became_peered":"2024-04-02T13:24:53.573883+0000","last_unstale":"2024-04-02T13:24:53.582044+0000","last_undegraded":"2024-04-02T13:24:53.582044+0000","last_fullsized":"2024-04-02T13:24:53.582044+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:14:09.640907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.564305+0000","last_change":"2024-04-02T13:24:53.564305+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.564305+0000","last_undegraded":"2024-04-02T13:24:53.564305+0000","last_fullsized":"2024-04-02T13:24:53.564305+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.573289+0000","last_change":"2024-04-02T13:24:53.571030+0000","last_active":"2024-04-02T13:24:53.573289+0000","last_peered":"2024-04-02T13:24:53.573289+0000","last_clean":"2024-04-02T13:24:53.573289+0000","last_became_active":"2024-04-02T13:24:53.570763+0000","last_became_peered":"2024-04-02T13:24:53.570763+0000","last_unstale":"2024-04-02T13:24:53.573289+0000","last_undegraded":"2024-04-02T13:24:53.573289+0000","last_fullsized":"2024-04-02T13:24:53.573289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:09:54.529069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567111+0000","last_change":"2024-04-02T13:24:53.567111+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567111+0000","last_undegraded":"2024-04-02T13:24:53.567111+0000","last_fullsized":"2024-04-02T13:24:53.567111+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579543+0000","last_change":"2024-04-02T13:24:53.578249+0000","last_active":"2024-04-02T13:24:53.579543+0000","last_peered":"2024-04-02T13:24:53.579543+0000","last_clean":"2024-04-02T13:24:53.579543+0000","last_became_active":"2024-04-02T13:24:53.577992+0000","last_became_peered":"2024-04-02T13:24:53.577992+0000","last_unstale":"2024-04-02T13:24:53.579543+0000","last_undegraded":"2024-04-02T13:24:53.579543+0000","last_fullsized":"2024-04-02T13:24:53.579543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:46:33.336625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561459+0000","last_change":"2024-04-02T13:24:53.561459+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561459+0000","last_undegraded":"2024-04-02T13:24:53.561459+0000","last_fullsized":"2024-04-02T13:24:53.561459+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579095+0000","last_change":"2024-04-02T13:24:53.572838+0000","last_active":"2024-04-02T13:24:53.579095+0000","last_peered":"2024-04-02T13:24:53.579095+0000","last_clean":"2024-04-02T13:24:53.579095+0000","last_became_active":"2024-04-02T13:24:53.571893+0000","last_became_peered":"2024-04-02T13:24:53.571893+0000","last_unstale":"2024-04-02T13:24:53.579095+0000","last_undegraded":"2024-04-02T13:24:53.579095+0000","last_fullsized":"2024-04-02T13:24:53.579095+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:08:46.014739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.563866+0000","last_change":"2024-04-02T13:24:53.563866+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.563866+0000","last_undegraded":"2024-04-02T13:24:53.563866+0000","last_fullsized":"2024-04-02T13:24:53.563866+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.582125+0000","last_change":"2024-04-02T13:24:53.579943+0000","last_active":"2024-04-02T13:24:53.582125+0000","last_peered":"2024-04-02T13:24:53.582125+0000","last_clean":"2024-04-02T13:24:53.582125+0000","last_became_active":"2024-04-02T13:24:53.579632+0000","last_became_peered":"2024-04-02T13:24:53.579632+0000","last_unstale":"2024-04-02T13:24:53.582125+0000","last_undegraded":"2024-04-02T13:24:53.582125+0000","last_fullsized":"2024-04-02T13:24:53.582125+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:06.592645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560270+0000","last_change":"2024-04-02T13:24:53.560270+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560270+0000","last_undegraded":"2024-04-02T13:24:53.560270+0000","last_fullsized":"2024-04-02T13:24:53.560270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.573320+0000","last_change":"2024-04-02T13:24:53.570841+0000","last_active":"2024-04-02T13:24:53.573320+0000","last_peered":"2024-04-02T13:24:53.573320+0000","last_clean":"2024-04-02T13:24:53.573320+0000","last_became_active":"2024-04-02T13:24:53.570567+0000","last_became_peered":"2024-04-02T13:24:53.570567+0000","last_unstale":"2024-04-02T13:24:53.573320+0000","last_undegraded":"2024-04-02T13:24:53.573320+0000","last_fullsized":"2024-04-02T13:24:53.573320+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:30:36.461673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.563298+0000","last_change":"2024-04-02T13:24:53.563298+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.563298+0000","last_undegraded":"2024-04-02T13:24:53.563298+0000","last_fullsized":"2024-04-02T13:24:53.563298+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.581606+0000","last_change":"2024-04-02T13:24:53.577054+0000","last_active":"2024-04-02T13:24:53.581606+0000","last_peered":"2024-04-02T13:24:53.581606+0000","last_clean":"2024-04-02T13:24:53.581606+0000","last_became_active":"2024-04-02T13:24:53.575955+0000","last_became_peered":"2024-04-02T13:24:53.575955+0000","last_unstale":"2024-04-02T13:24:53.581606+0000","last_undegraded":"2024-04-02T13:24:53.581606+0000","last_fullsized":"2024-04-02T13:24:53.581606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:50:57.375379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560085+0000","last_change":"2024-04-02T13:24:53.560085+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560085+0000","last_undegraded":"2024-04-02T13:24:53.560085+0000","last_fullsized":"2024-04-02T13:24:53.560085+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.581965+0000","last_change":"2024-04-02T13:24:53.575781+0000","last_active":"2024-04-02T13:24:53.581965+0000","last_peered":"2024-04-02T13:24:53.581965+0000","last_clean":"2024-04-02T13:24:53.581965+0000","last_became_active":"2024-04-02T13:24:53.574619+0000","last_became_peered":"2024-04-02T13:24:53.574619+0000","last_unstale":"2024-04-02T13:24:53.581965+0000","last_undegraded":"2024-04-02T13:24:53.581965+0000","last_fullsized":"2024-04-02T13:24:53.581965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:14:47.150301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561204+0000","last_change":"2024-04-02T13:24:53.561204+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561204+0000","last_undegraded":"2024-04-02T13:24:53.561204+0000","last_fullsized":"2024-04-02T13:24:53.561204+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.574834+0000","last_change":"2024-04-02T13:24:53.572426+0000","last_active":"2024-04-02T13:24:53.574834+0000","last_peered":"2024-04-02T13:24:53.574834+0000","last_clean":"2024-04-02T13:24:53.574834+0000","last_became_active":"2024-04-02T13:24:53.572153+0000","last_became_peered":"2024-04-02T13:24:53.572153+0000","last_unstale":"2024-04-02T13:24:53.574834+0000","last_undegraded":"2024-04-02T13:24:53.574834+0000","last_fullsized":"2024-04-02T13:24:53.574834+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:18:20.845843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561018+0000","last_change":"2024-04-02T13:24:53.561018+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561018+0000","last_undegraded":"2024-04-02T13:24:53.561018+0000","last_fullsized":"2024-04-02T13:24:53.561018+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.582163+0000","last_change":"2024-04-02T13:24:53.580278+0000","last_active":"2024-04-02T13:24:53.582163+0000","last_peered":"2024-04-02T13:24:53.582163+0000","last_clean":"2024-04-02T13:24:53.582163+0000","last_became_active":"2024-04-02T13:24:53.580002+0000","last_became_peered":"2024-04-02T13:24:53.580002+0000","last_unstale":"2024-04-02T13:24:53.582163+0000","last_undegraded":"2024-04-02T13:24:53.582163+0000","last_fullsized":"2024-04-02T13:24:53.582163+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:47.224016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.559858+0000","last_change":"2024-04-02T13:24:53.559858+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.559858+0000","last_undegraded":"2024-04-02T13:24:53.559858+0000","last_fullsized":"2024-04-02T13:24:53.559858+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.581549+0000","last_change":"2024-04-02T13:24:53.576553+0000","last_active":"2024-04-02T13:24:53.581549+0000","last_peered":"2024-04-02T13:24:53.581549+0000","last_clean":"2024-04-02T13:24:53.581549+0000","last_became_active":"2024-04-02T13:24:53.576219+0000","last_became_peered":"2024-04-02T13:24:53.576219+0000","last_unstale":"2024-04-02T13:24:53.581549+0000","last_undegraded":"2024-04-02T13:24:53.581549+0000","last_fullsized":"2024-04-02T13:24:53.581549+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:28.254107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561070+0000","last_change":"2024-04-02T13:24:53.561070+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561070+0000","last_undegraded":"2024-04-02T13:24:53.561070+0000","last_fullsized":"2024-04-02T13:24:53.561070+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.578243+0000","last_change":"2024-04-02T13:24:53.575318+0000","last_active":"2024-04-02T13:24:53.578243+0000","last_peered":"2024-04-02T13:24:53.578243+0000","last_clean":"2024-04-02T13:24:53.578243+0000","last_became_active":"2024-04-02T13:24:53.575019+0000","last_became_peered":"2024-04-02T13:24:53.575019+0000","last_unstale":"2024-04-02T13:24:53.578243+0000","last_undegraded":"2024-04-02T13:24:53.578243+0000","last_fullsized":"2024-04-02T13:24:53.578243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:39:30.962000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.562727+0000","last_change":"2024-04-02T13:24:53.562727+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.562727+0000","last_undegraded":"2024-04-02T13:24:53.562727+0000","last_fullsized":"2024-04-02T13:24:53.562727+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.578673+0000","last_change":"2024-04-02T13:24:53.577390+0000","last_active":"2024-04-02T13:24:53.578673+0000","last_peered":"2024-04-02T13:24:53.578673+0000","last_clean":"2024-04-02T13:24:53.578673+0000","last_became_active":"2024-04-02T13:24:53.577079+0000","last_became_peered":"2024-04-02T13:24:53.577079+0000","last_unstale":"2024-04-02T13:24:53.578673+0000","last_undegraded":"2024-04-02T13:24:53.578673+0000","last_fullsized":"2024-04-02T13:24:53.578673+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:46:33.336625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566607+0000","last_change":"2024-04-02T13:24:53.566607+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566607+0000","last_undegraded":"2024-04-02T13:24:53.566607+0000","last_fullsized":"2024-04-02T13:24:53.566607+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.582111+0000","last_change":"2024-04-02T13:24:53.577294+0000","last_active":"2024-04-02T13:24:53.582111+0000","last_peered":"2024-04-02T13:24:53.582111+0000","last_clean":"2024-04-02T13:24:53.582111+0000","last_became_active":"2024-04-02T13:24:53.577018+0000","last_became_peered":"2024-04-02T13:24:53.577018+0000","last_unstale":"2024-04-02T13:24:53.582111+0000","last_undegraded":"2024-04-02T13:24:53.582111+0000","last_fullsized":"2024-04-02T13:24:53.582111+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:09:03.943800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565690+0000","last_change":"2024-04-02T13:24:53.565690+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565690+0000","last_undegraded":"2024-04-02T13:24:53.565690+0000","last_fullsized":"2024-04-02T13:24:53.565690+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579186+0000","last_change":"2024-04-02T13:24:53.573420+0000","last_active":"2024-04-02T13:24:53.579186+0000","last_peered":"2024-04-02T13:24:53.579186+0000","last_clean":"2024-04-02T13:24:53.579186+0000","last_became_active":"2024-04-02T13:24:53.571909+0000","last_became_peered":"2024-04-02T13:24:53.571909+0000","last_unstale":"2024-04-02T13:24:53.579186+0000","last_undegraded":"2024-04-02T13:24:53.579186+0000","last_fullsized":"2024-04-02T13:24:53.579186+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:03:28.676340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.562556+0000","last_change":"2024-04-02T13:24:53.562556+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.562556+0000","last_undegraded":"2024-04-02T13:24:53.562556+0000","last_fullsized":"2024-04-02T13:24:53.562556+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.576658+0000","last_change":"2024-04-02T13:24:53.570594+0000","last_active":"2024-04-02T13:24:53.576658+0000","last_peered":"2024-04-02T13:24:53.576658+0000","last_clean":"2024-04-02T13:24:53.576658+0000","last_became_active":"2024-04-02T13:24:53.569678+0000","last_became_peered":"2024-04-02T13:24:53.569678+0000","last_unstale":"2024-04-02T13:24:53.576658+0000","last_undegraded":"2024-04-02T13:24:53.576658+0000","last_fullsized":"2024-04-02T13:24:53.576658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:06.116927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.564134+0000","last_change":"2024-04-02T13:24:53.564134+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.564134+0000","last_undegraded":"2024-04-02T13:24:53.564134+0000","last_fullsized":"2024-04-02T13:24:53.564134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579285+0000","last_change":"2024-04-02T13:24:53.571104+0000","last_active":"2024-04-02T13:24:53.579285+0000","last_peered":"2024-04-02T13:24:53.579285+0000","last_clean":"2024-04-02T13:24:53.579285+0000","last_became_active":"2024-04-02T13:24:53.570758+0000","last_became_peered":"2024-04-02T13:24:53.570758+0000","last_unstale":"2024-04-02T13:24:53.579285+0000","last_undegraded":"2024-04-02T13:24:53.579285+0000","last_fullsized":"2024-04-02T13:24:53.579285+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:48:22.399865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.569084+0000","last_change":"2024-04-02T13:24:53.569084+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.569084+0000","last_undegraded":"2024-04-02T13:24:53.569084+0000","last_fullsized":"2024-04-02T13:24:53.569084+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579375+0000","last_change":"2024-04-02T13:24:53.573510+0000","last_active":"2024-04-02T13:24:53.579375+0000","last_peered":"2024-04-02T13:24:53.579375+0000","last_clean":"2024-04-02T13:24:53.579375+0000","last_became_active":"2024-04-02T13:24:53.572671+0000","last_became_peered":"2024-04-02T13:24:53.572671+0000","last_unstale":"2024-04-02T13:24:53.579375+0000","last_undegraded":"2024-04-02T13:24:53.579375+0000","last_fullsized":"2024-04-02T13:24:53.579375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:11:32.316427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560133+0000","last_change":"2024-04-02T13:24:53.560133+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560133+0000","last_undegraded":"2024-04-02T13:24:53.560133+0000","last_fullsized":"2024-04-02T13:24:53.560133+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.581839+0000","last_change":"2024-04-02T13:24:53.574067+0000","last_active":"2024-04-02T13:24:53.581839+0000","last_peered":"2024-04-02T13:24:53.581839+0000","last_clean":"2024-04-02T13:24:53.581839+0000","last_became_active":"2024-04-02T13:24:53.573799+0000","last_became_peered":"2024-04-02T13:24:53.573799+0000","last_unstale":"2024-04-02T13:24:53.581839+0000","last_undegraded":"2024-04-02T13:24:53.581839+0000","last_fullsized":"2024-04-02T13:24:53.581839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:26:00.810278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.568001+0000","last_change":"2024-04-02T13:24:53.568001+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.568001+0000","last_undegraded":"2024-04-02T13:24:53.568001+0000","last_fullsized":"2024-04-02T13:24:53.568001+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579545+0000","last_change":"2024-04-02T13:24:53.575531+0000","last_active":"2024-04-02T13:24:53.579545+0000","last_peered":"2024-04-02T13:24:53.579545+0000","last_clean":"2024-04-02T13:24:53.579545+0000","last_became_active":"2024-04-02T13:24:53.575222+0000","last_became_peered":"2024-04-02T13:24:53.575222+0000","last_unstale":"2024-04-02T13:24:53.579545+0000","last_undegraded":"2024-04-02T13:24:53.579545+0000","last_fullsized":"2024-04-02T13:24:53.579545+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:48:43.076818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560146+0000","last_change":"2024-04-02T13:24:53.560146+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560146+0000","last_undegraded":"2024-04-02T13:24:53.560146+0000","last_fullsized":"2024-04-02T13:24:53.560146+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.578577+0000","last_change":"2024-04-02T13:24:53.577485+0000","last_active":"2024-04-02T13:24:53.578577+0000","last_peered":"2024-04-02T13:24:53.578577+0000","last_clean":"2024-04-02T13:24:53.578577+0000","last_became_active":"2024-04-02T13:24:53.575330+0000","last_became_peered":"2024-04-02T13:24:53.575330+0000","last_unstale":"2024-04-02T13:24:53.578577+0000","last_undegraded":"2024-04-02T13:24:53.578577+0000","last_fullsized":"2024-04-02T13:24:53.578577+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:51.244520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567636+0000","last_change":"2024-04-02T13:24:53.567636+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567636+0000","last_undegraded":"2024-04-02T13:24:53.567636+0000","last_fullsized":"2024-04-02T13:24:53.567636+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.573910+0000","last_change":"2024-04-02T13:24:53.572963+0000","last_active":"2024-04-02T13:24:53.573910+0000","last_peered":"2024-04-02T13:24:53.573910+0000","last_clean":"2024-04-02T13:24:53.573910+0000","last_became_active":"2024-04-02T13:24:53.572536+0000","last_became_peered":"2024-04-02T13:24:53.572536+0000","last_unstale":"2024-04-02T13:24:53.573910+0000","last_undegraded":"2024-04-02T13:24:53.573910+0000","last_fullsized":"2024-04-02T13:24:53.573910+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:39:30.962000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567123+0000","last_change":"2024-04-02T13:24:53.567123+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567123+0000","last_undegraded":"2024-04-02T13:24:53.567123+0000","last_fullsized":"2024-04-02T13:24:53.567123+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.582138+0000","last_change":"2024-04-02T13:24:53.580007+0000","last_active":"2024-04-02T13:24:53.582138+0000","last_peered":"2024-04-02T13:24:53.582138+0000","last_clean":"2024-04-02T13:24:53.582138+0000","last_became_active":"2024-04-02T13:24:53.579686+0000","last_became_peered":"2024-04-02T13:24:53.579686+0000","last_unstale":"2024-04-02T13:24:53.582138+0000","last_undegraded":"2024-04-02T13:24:53.582138+0000","last_fullsized":"2024-04-02T13:24:53.582138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:49.667468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560541+0000","last_change":"2024-04-02T13:24:53.560541+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560541+0000","last_undegraded":"2024-04-02T13:24:53.560541+0000","last_fullsized":"2024-04-02T13:24:53.560541+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579133+0000","last_change":"2024-04-02T13:24:53.572298+0000","last_active":"2024-04-02T13:24:53.579133+0000","last_peered":"2024-04-02T13:24:53.579133+0000","last_clean":"2024-04-02T13:24:53.579133+0000","last_became_active":"2024-04-02T13:24:53.570380+0000","last_became_peered":"2024-04-02T13:24:53.570380+0000","last_unstale":"2024-04-02T13:24:53.579133+0000","last_undegraded":"2024-04-02T13:24:53.579133+0000","last_fullsized":"2024-04-02T13:24:53.579133+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:56:57.490956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.559526+0000","last_change":"2024-04-02T13:24:53.559526+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.559526+0000","last_undegraded":"2024-04-02T13:24:53.559526+0000","last_fullsized":"2024-04-02T13:24:53.559526+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579521+0000","last_change":"2024-04-02T13:24:53.572484+0000","last_active":"2024-04-02T13:24:53.579521+0000","last_peered":"2024-04-02T13:24:53.579521+0000","last_clean":"2024-04-02T13:24:53.579521+0000","last_became_active":"2024-04-02T13:24:53.572124+0000","last_became_peered":"2024-04-02T13:24:53.572124+0000","last_unstale":"2024-04-02T13:24:53.579521+0000","last_undegraded":"2024-04-02T13:24:53.579521+0000","last_fullsized":"2024-04-02T13:24:53.579521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:14:51.783505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.559822+0000","last_change":"2024-04-02T13:24:53.559822+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.559822+0000","last_undegraded":"2024-04-02T13:24:53.559822+0000","last_fullsized":"2024-04-02T13:24:53.559822+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579207+0000","last_change":"2024-04-02T13:24:53.574193+0000","last_active":"2024-04-02T13:24:53.579207+0000","last_peered":"2024-04-02T13:24:53.579207+0000","last_clean":"2024-04-02T13:24:53.579207+0000","last_became_active":"2024-04-02T13:24:53.570565+0000","last_became_peered":"2024-04-02T13:24:53.570565+0000","last_unstale":"2024-04-02T13:24:53.579207+0000","last_undegraded":"2024-04-02T13:24:53.579207+0000","last_fullsized":"2024-04-02T13:24:53.579207+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:48:35.205658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560048+0000","last_change":"2024-04-02T13:24:53.560048+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560048+0000","last_undegraded":"2024-04-02T13:24:53.560048+0000","last_fullsized":"2024-04-02T13:24:53.560048+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.574610+0000","last_change":"2024-04-02T13:24:53.567583+0000","last_active":"2024-04-02T13:24:53.574610+0000","last_peered":"2024-04-02T13:24:53.574610+0000","last_clean":"2024-04-02T13:24:53.574610+0000","last_became_active":"2024-04-02T13:24:53.567289+0000","last_became_peered":"2024-04-02T13:24:53.567289+0000","last_unstale":"2024-04-02T13:24:53.574610+0000","last_undegraded":"2024-04-02T13:24:53.574610+0000","last_fullsized":"2024-04-02T13:24:53.574610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:14:47.150301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565970+0000","last_change":"2024-04-02T13:24:53.565970+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565970+0000","last_undegraded":"2024-04-02T13:24:53.565970+0000","last_fullsized":"2024-04-02T13:24:53.565970+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579322+0000","last_change":"2024-04-02T13:24:53.572298+0000","last_active":"2024-04-02T13:24:53.579322+0000","last_peered":"2024-04-02T13:24:53.579322+0000","last_clean":"2024-04-02T13:24:53.579322+0000","last_became_active":"2024-04-02T13:24:53.572035+0000","last_became_peered":"2024-04-02T13:24:53.572035+0000","last_unstale":"2024-04-02T13:24:53.579322+0000","last_undegraded":"2024-04-02T13:24:53.579322+0000","last_fullsized":"2024-04-02T13:24:53.579322+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:07:25.222536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":278,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.568085+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:24:53.568085+0000","last_peered":"2024-04-02T13:24:53.568085+0000","last_clean":"2024-04-02T13:24:53.568085+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:24:53.568085+0000","last_undegraded":"2024-04-02T13:24:53.568085+0000","last_fullsized":"2024-04-02T13:24:53.568085+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1536000,"data_stored":1018118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":427750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":421083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":427750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":942080,"data_stored":427750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":421083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":421083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1536000,"data_stored":1018118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:24:58.598 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:24:58.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.732+0000 7f367c4ba640 1 Processor -- start 2024-04-02T13:24:58.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.732+0000 7f367c4ba640 1 -- start start 2024-04-02T13:24:58.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.733+0000 7f367c4ba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36740d0a30 0x7f367406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.733+0000 7f367c4ba640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f36740d2040 0x7f36740d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.733+0000 7f367c4ba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36740d0000 con 0x7f36740d0e10 2024-04-02T13:24:58.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.733+0000 7f367c4ba640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f367406cb00 con 0x7f36740d2040 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.733+0000 7f367c4ba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f367406c1d0 con 0x7f36740d0a30 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.733+0000 7f367aa57640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f36740d2040 0x7f36740d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.734+0000 7f367aa57640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f36740d2040 0x7f36740d24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41856/0 (socket says 172.21.15.44:41856) 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.734+0000 7f367aa57640 1 -- 172.21.15.44:0/1947757732 learned_addr learned my addr 172.21.15.44:0/1947757732 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.734+0000 7f367a256640 1 --2- 172.21.15.44:0/1947757732 >> v2:172.21.15.69:3300/0 conn(0x7f36740d0a30 0x7f367406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.734+0000 7f367aa57640 1 -- 172.21.15.44:0/1947757732 >> v2:172.21.15.69:3300/0 conn(0x7f36740d0a30 msgr2=0x7f367406cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.734+0000 7f367aa57640 1 --2- 172.21.15.44:0/1947757732 >> v2:172.21.15.69:3300/0 conn(0x7f36740d0a30 0x7f367406cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.734+0000 7f367aa57640 1 -- 172.21.15.44:0/1947757732 >> v1:172.21.15.69:6789/0 conn(0x7f36740d0e10 legacy=0x7f36740cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.734+0000 7f367aa57640 1 -- 172.21.15.44:0/1947757732 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f367406c4b0 con 0x7f36740d2040 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.734+0000 7f367a256640 1 --2- 172.21.15.44:0/1947757732 >> v2:172.21.15.69:3300/0 conn(0x7f36740d0a30 0x7f367406cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.735+0000 7f367aa57640 1 --2- 172.21.15.44:0/1947757732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36740d2040 0x7f36740d24f0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f367406de20 tx=0x7f366c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=facac5d441dcc050 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.735+0000 7f3679254640 1 -- 172.21.15.44:0/1947757732 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c02c9f0 con 0x7f36740d2040 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.735+0000 7f3679254640 1 -- 172.21.15.44:0/1947757732 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f366c02cb90 con 0x7f36740d2040 2024-04-02T13:24:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.735+0000 7f3679254640 1 -- 172.21.15.44:0/1947757732 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c0343f0 con 0x7f36740d2040 2024-04-02T13:24:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.735+0000 7f367c4ba640 1 -- 172.21.15.44:0/1947757732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36740d2040 msgr2=0x7f36740d24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.735+0000 7f367c4ba640 1 --2- 172.21.15.44:0/1947757732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36740d2040 0x7f36740d24f0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f367406de20 tx=0x7f366c02be50 comp rx=0 tx=0).stop 2024-04-02T13:24:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.735+0000 7f367c4ba640 1 -- 172.21.15.44:0/1947757732 shutdown_connections 2024-04-02T13:24:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.735+0000 7f367c4ba640 1 --2- 172.21.15.44:0/1947757732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36740d2040 0x7f36740d24f0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.735+0000 7f367c4ba640 1 --2- 172.21.15.44:0/1947757732 >> v2:172.21.15.69:3300/0 conn(0x7f36740d0a30 0x7f367406cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.735+0000 7f367c4ba640 1 -- 172.21.15.44:0/1947757732 >> 172.21.15.44:0/1947757732 conn(0x7f367409d470 msgr2=0x7f36740d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.735+0000 7f367c4ba640 1 -- 172.21.15.44:0/1947757732 shutdown_connections 2024-04-02T13:24:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.735+0000 7f367c4ba640 1 -- 172.21.15.44:0/1947757732 wait complete. 2024-04-02T13:24:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.736+0000 7f367c4ba640 1 Processor -- start 2024-04-02T13:24:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.736+0000 7f367c4ba640 1 -- start start 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.737+0000 7f367c4ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36740d0a30 0x7f36740b5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.737+0000 7f367c4ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36740d0e10 0x7f36741575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.737+0000 7f367c4ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36740d2040 0x7f36740c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.737+0000 7f367c4ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f367415e6f0 con 0x7f36740d0a30 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.737+0000 7f367c4ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f36740d1270 con 0x7f36740d0e10 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.737+0000 7f367c4ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f367415e870 con 0x7f36740d2040 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.737+0000 7f367a256640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36740d0a30 0x7f36740b5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.737+0000 7f367a256640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36740d0a30 0x7f36740b5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41864/0 (socket says 172.21.15.44:41864) 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.737+0000 7f367a256640 1 -- 172.21.15.44:0/3962707094 learned_addr learned my addr 172.21.15.44:0/3962707094 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.737+0000 7f367aa57640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36740d2040 0x7f36740c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.737+0000 7f3679a55640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36740d0e10 0x7f36741575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.738+0000 7f367a256640 1 -- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36740d2040 msgr2=0x7f36740c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.738+0000 7f367a256640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36740d2040 0x7f36740c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.738+0000 7f367a256640 1 -- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36740d0e10 msgr2=0x7f36741575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.738+0000 7f367a256640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36740d0e10 0x7f36741575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.738+0000 7f367a256640 1 -- 172.21.15.44:0/3962707094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3674157b30 con 0x7f36740d0a30 2024-04-02T13:24:58.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.738+0000 7f367aa57640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36740d2040 0x7f36740c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:24:58.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.738+0000 7f367a256640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36740d0a30 0x7f36740b5df0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f367000c820 tx=0x7f367000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:58.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.738+0000 7f366affd640 1 -- 172.21.15.44:0/3962707094 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670015070 con 0x7f36740d0a30 2024-04-02T13:24:58.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.738+0000 7f366affd640 1 -- 172.21.15.44:0/3962707094 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3670007a30 con 0x7f36740d0a30 2024-04-02T13:24:58.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.738+0000 7f367c4ba640 1 -- 172.21.15.44:0/3962707094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36740b4d50 con 0x7f36740d0a30 2024-04-02T13:24:58.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.738+0000 7f366affd640 1 -- 172.21.15.44:0/3962707094 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670007de0 con 0x7f36740d0a30 2024-04-02T13:24:58.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.738+0000 7f367c4ba640 1 -- 172.21.15.44:0/3962707094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36740b5310 con 0x7f36740d0a30 2024-04-02T13:24:58.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.739+0000 7f367c4ba640 1 -- 172.21.15.44:0/3962707094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36740684f0 con 0x7f36740d0a30 2024-04-02T13:24:58.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.740+0000 7f366affd640 1 -- 172.21.15.44:0/3962707094 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3670018070 con 0x7f36740d0a30 2024-04-02T13:24:58.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.740+0000 7f366affd640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f365806c8f0 0x7f365806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:58.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.740+0000 7f366affd640 1 -- 172.21.15.44:0/3962707094 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f367008bb80 con 0x7f36740d0a30 2024-04-02T13:24:58.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.740+0000 7f3679a55640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f365806c8f0 0x7f365806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:58.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.741+0000 7f3679a55640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f365806c8f0 0x7f365806edb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3664006fd0 tx=0x7f3664008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:58.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.745+0000 7f366affd640 1 -- 172.21.15.44:0/3962707094 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3674157b30 con 0x7f36740d0a30 2024-04-02T13:24:58.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.927+0000 7f367c4ba640 1 -- 172.21.15.44:0/3962707094 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f36740cf450 con 0x7f365806c8f0 2024-04-02T13:24:58.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.933+0000 7f366affd640 1 -- 172.21.15.44:0/3962707094 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246754 (secure 0 0 0) 0x7f36740cf450 con 0x7f365806c8f0 2024-04-02T13:24:58.939 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:58.940 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:24:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.937+0000 7f367c4ba640 1 -- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f365806c8f0 msgr2=0x7f365806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.937+0000 7f367c4ba640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f365806c8f0 0x7f365806edb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3664006fd0 tx=0x7f3664008040 comp rx=0 tx=0).stop 2024-04-02T13:24:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.937+0000 7f367c4ba640 1 -- 172.21.15.44:0/3962707094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36740d0a30 msgr2=0x7f36740b5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.937+0000 7f367c4ba640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36740d0a30 0x7f36740b5df0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f367000c820 tx=0x7f367000cce0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.937+0000 7f367c4ba640 1 -- 172.21.15.44:0/3962707094 shutdown_connections 2024-04-02T13:24:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.937+0000 7f367c4ba640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f365806c8f0 0x7f365806edb0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.937+0000 7f367c4ba640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36740d2040 0x7f36740c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.937+0000 7f367c4ba640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36740d0e10 0x7f36741575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.937+0000 7f367c4ba640 1 --2- 172.21.15.44:0/3962707094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36740d0a30 0x7f36740b5df0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.937+0000 7f367c4ba640 1 -- 172.21.15.44:0/3962707094 >> 172.21.15.44:0/3962707094 conn(0x7f367409d470 msgr2=0x7f36740ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.937+0000 7f367c4ba640 1 -- 172.21.15.44:0/3962707094 shutdown_connections 2024-04-02T13:24:58.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:58.937+0000 7f367c4ba640 1 -- 172.21.15.44:0/3962707094 wait complete. 2024-04-02T13:24:58.951 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-04-02T13:24:56.941220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237144,"kb_used_data":8484,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754737576,"statfs":{"total":773094113280,"available":772851277824,"internally_reserved":0,"allocated":8687616,"data_stored":4582735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.391544"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567106+0000","last_change":"2024-04-02T13:24:53.567106+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567106+0000","last_undegraded":"2024-04-02T13:24:53.567106+0000","last_fullsized":"2024-04-02T13:24:53.567106+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.569705+0000","last_change":"2024-04-02T13:24:53.569705+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.569705+0000","last_undegraded":"2024-04-02T13:24:53.569705+0000","last_fullsized":"2024-04-02T13:24:53.569705+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560847+0000","last_change":"2024-04-02T13:24:53.560847+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560847+0000","last_undegraded":"2024-04-02T13:24:53.560847+0000","last_fullsized":"2024-04-02T13:24:53.560847+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.562054+0000","last_change":"2024-04-02T13:24:53.562054+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.562054+0000","last_undegraded":"2024-04-02T13:24:53.562054+0000","last_fullsized":"2024-04-02T13:24:53.562054+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565779+0000","last_change":"2024-04-02T13:24:53.565779+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565779+0000","last_undegraded":"2024-04-02T13:24:53.565779+0000","last_fullsized":"2024-04-02T13:24:53.565779+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.564580+0000","last_change":"2024-04-02T13:24:53.564580+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.564580+0000","last_undegraded":"2024-04-02T13:24:53.564580+0000","last_fullsized":"2024-04-02T13:24:53.564580+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561157+0000","last_change":"2024-04-02T13:24:53.561157+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561157+0000","last_undegraded":"2024-04-02T13:24:53.561157+0000","last_fullsized":"2024-04-02T13:24:53.561157+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567092+0000","last_change":"2024-04-02T13:24:53.567092+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567092+0000","last_undegraded":"2024-04-02T13:24:53.567092+0000","last_fullsized":"2024-04-02T13:24:53.567092+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566497+0000","last_change":"2024-04-02T13:24:53.566497+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566497+0000","last_undegraded":"2024-04-02T13:24:53.566497+0000","last_fullsized":"2024-04-02T13:24:53.566497+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.562553+0000","last_change":"2024-04-02T13:24:53.562553+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.562553+0000","last_undegraded":"2024-04-02T13:24:53.562553+0000","last_fullsized":"2024-04-02T13:24:53.562553+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.570289+0000","last_change":"2024-04-02T13:24:53.570289+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.570289+0000","last_undegraded":"2024-04-02T13:24:53.570289+0000","last_fullsized":"2024-04-02T13:24:53.570289+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561617+0000","last_change":"2024-04-02T13:24:53.561617+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561617+0000","last_undegraded":"2024-04-02T13:24:53.561617+0000","last_fullsized":"2024-04-02T13:24:53.561617+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565353+0000","last_change":"2024-04-02T13:24:53.565353+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565353+0000","last_undegraded":"2024-04-02T13:24:53.565353+0000","last_fullsized":"2024-04-02T13:24:53.565353+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.564097+0000","last_change":"2024-04-02T13:24:53.564097+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.564097+0000","last_undegraded":"2024-04-02T13:24:53.564097+0000","last_fullsized":"2024-04-02T13:24:53.564097+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567769+0000","last_change":"2024-04-02T13:24:53.567769+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567769+0000","last_undegraded":"2024-04-02T13:24:53.567769+0000","last_fullsized":"2024-04-02T13:24:53.567769+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566403+0000","last_change":"2024-04-02T13:24:53.566403+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566403+0000","last_undegraded":"2024-04-02T13:24:53.566403+0000","last_fullsized":"2024-04-02T13:24:53.566403+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565036+0000","last_change":"2024-04-02T13:24:53.565036+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565036+0000","last_undegraded":"2024-04-02T13:24:53.565036+0000","last_fullsized":"2024-04-02T13:24:53.565036+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560762+0000","last_change":"2024-04-02T13:24:53.560762+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560762+0000","last_undegraded":"2024-04-02T13:24:53.560762+0000","last_fullsized":"2024-04-02T13:24:53.560762+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561514+0000","last_change":"2024-04-02T13:24:53.561514+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561514+0000","last_undegraded":"2024-04-02T13:24:53.561514+0000","last_fullsized":"2024-04-02T13:24:53.561514+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567531+0000","last_change":"2024-04-02T13:24:53.567531+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567531+0000","last_undegraded":"2024-04-02T13:24:53.567531+0000","last_fullsized":"2024-04-02T13:24:53.567531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567537+0000","last_change":"2024-04-02T13:24:53.567537+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567537+0000","last_undegraded":"2024-04-02T13:24:53.567537+0000","last_fullsized":"2024-04-02T13:24:53.567537+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565239+0000","last_change":"2024-04-02T13:24:53.565239+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565239+0000","last_undegraded":"2024-04-02T13:24:53.565239+0000","last_fullsized":"2024-04-02T13:24:53.565239+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566185+0000","last_change":"2024-04-02T13:24:53.566185+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566185+0000","last_undegraded":"2024-04-02T13:24:53.566185+0000","last_fullsized":"2024-04-02T13:24:53.566185+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567230+0000","last_change":"2024-04-02T13:24:53.567230+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567230+0000","last_undegraded":"2024-04-02T13:24:53.567230+0000","last_fullsized":"2024-04-02T13:24:53.567230+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.564364+0000","last_change":"2024-04-02T13:24:53.564364+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.564364+0000","last_undegraded":"2024-04-02T13:24:53.564364+0000","last_fullsized":"2024-04-02T13:24:53.564364+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.562181+0000","last_change":"2024-04-02T13:24:53.562181+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.562181+0000","last_undegraded":"2024-04-02T13:24:53.562181+0000","last_fullsized":"2024-04-02T13:24:53.562181+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566599+0000","last_change":"2024-04-02T13:24:53.566599+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566599+0000","last_undegraded":"2024-04-02T13:24:53.566599+0000","last_fullsized":"2024-04-02T13:24:53.566599+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566535+0000","last_change":"2024-04-02T13:24:53.566535+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566535+0000","last_undegraded":"2024-04-02T13:24:53.566535+0000","last_fullsized":"2024-04-02T13:24:53.566535+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567572+0000","last_change":"2024-04-02T13:24:53.567572+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567572+0000","last_undegraded":"2024-04-02T13:24:53.567572+0000","last_fullsized":"2024-04-02T13:24:53.567572+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566012+0000","last_change":"2024-04-02T13:24:53.566012+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566012+0000","last_undegraded":"2024-04-02T13:24:53.566012+0000","last_fullsized":"2024-04-02T13:24:53.566012+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560614+0000","last_change":"2024-04-02T13:24:53.560614+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560614+0000","last_undegraded":"2024-04-02T13:24:53.560614+0000","last_fullsized":"2024-04-02T13:24:53.560614+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565897+0000","last_change":"2024-04-02T13:24:53.565897+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565897+0000","last_undegraded":"2024-04-02T13:24:53.565897+0000","last_fullsized":"2024-04-02T13:24:53.565897+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560034+0000","last_change":"2024-04-02T13:24:53.560034+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560034+0000","last_undegraded":"2024-04-02T13:24:53.560034+0000","last_fullsized":"2024-04-02T13:24:53.560034+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579157+0000","last_change":"2024-04-02T13:24:53.573023+0000","last_active":"2024-04-02T13:24:53.579157+0000","last_peered":"2024-04-02T13:24:53.579157+0000","last_clean":"2024-04-02T13:24:53.579157+0000","last_became_active":"2024-04-02T13:24:53.571627+0000","last_became_peered":"2024-04-02T13:24:53.571627+0000","last_unstale":"2024-04-02T13:24:53.579157+0000","last_undegraded":"2024-04-02T13:24:53.579157+0000","last_fullsized":"2024-04-02T13:24:53.579157+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:37:37.314207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567025+0000","last_change":"2024-04-02T13:24:53.567025+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567025+0000","last_undegraded":"2024-04-02T13:24:53.567025+0000","last_fullsized":"2024-04-02T13:24:53.567025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.574869+0000","last_change":"2024-04-02T13:24:53.573555+0000","last_active":"2024-04-02T13:24:53.574869+0000","last_peered":"2024-04-02T13:24:53.574869+0000","last_clean":"2024-04-02T13:24:53.574869+0000","last_became_active":"2024-04-02T13:24:53.573158+0000","last_became_peered":"2024-04-02T13:24:53.573158+0000","last_unstale":"2024-04-02T13:24:53.574869+0000","last_undegraded":"2024-04-02T13:24:53.574869+0000","last_fullsized":"2024-04-02T13:24:53.574869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:51.244520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.568084+0000","last_change":"2024-04-02T13:24:53.568084+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.568084+0000","last_undegraded":"2024-04-02T13:24:53.568084+0000","last_fullsized":"2024-04-02T13:24:53.568084+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579904+0000","last_change":"2024-04-02T13:24:53.578788+0000","last_active":"2024-04-02T13:24:53.579904+0000","last_peered":"2024-04-02T13:24:53.579904+0000","last_clean":"2024-04-02T13:24:53.579904+0000","last_became_active":"2024-04-02T13:24:53.578523+0000","last_became_peered":"2024-04-02T13:24:53.578523+0000","last_unstale":"2024-04-02T13:24:53.579904+0000","last_undegraded":"2024-04-02T13:24:53.579904+0000","last_fullsized":"2024-04-02T13:24:53.579904+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:23:42.359046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567122+0000","last_change":"2024-04-02T13:24:53.567122+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567122+0000","last_undegraded":"2024-04-02T13:24:53.567122+0000","last_fullsized":"2024-04-02T13:24:53.567122+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.581991+0000","last_change":"2024-04-02T13:24:53.575848+0000","last_active":"2024-04-02T13:24:53.581991+0000","last_peered":"2024-04-02T13:24:53.581991+0000","last_clean":"2024-04-02T13:24:53.581991+0000","last_became_active":"2024-04-02T13:24:53.574537+0000","last_became_peered":"2024-04-02T13:24:53.574537+0000","last_unstale":"2024-04-02T13:24:53.581991+0000","last_undegraded":"2024-04-02T13:24:53.581991+0000","last_fullsized":"2024-04-02T13:24:53.581991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:06.116927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.564902+0000","last_change":"2024-04-02T13:24:53.564902+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.564902+0000","last_undegraded":"2024-04-02T13:24:53.564902+0000","last_fullsized":"2024-04-02T13:24:53.564902+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.582044+0000","last_change":"2024-04-02T13:24:53.574172+0000","last_active":"2024-04-02T13:24:53.582044+0000","last_peered":"2024-04-02T13:24:53.582044+0000","last_clean":"2024-04-02T13:24:53.582044+0000","last_became_active":"2024-04-02T13:24:53.573883+0000","last_became_peered":"2024-04-02T13:24:53.573883+0000","last_unstale":"2024-04-02T13:24:53.582044+0000","last_undegraded":"2024-04-02T13:24:53.582044+0000","last_fullsized":"2024-04-02T13:24:53.582044+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:14:09.640907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.564305+0000","last_change":"2024-04-02T13:24:53.564305+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.564305+0000","last_undegraded":"2024-04-02T13:24:53.564305+0000","last_fullsized":"2024-04-02T13:24:53.564305+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.573289+0000","last_change":"2024-04-02T13:24:53.571030+0000","last_active":"2024-04-02T13:24:53.573289+0000","last_peered":"2024-04-02T13:24:53.573289+0000","last_clean":"2024-04-02T13:24:53.573289+0000","last_became_active":"2024-04-02T13:24:53.570763+0000","last_became_peered":"2024-04-02T13:24:53.570763+0000","last_unstale":"2024-04-02T13:24:53.573289+0000","last_undegraded":"2024-04-02T13:24:53.573289+0000","last_fullsized":"2024-04-02T13:24:53.573289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:09:54.529069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567111+0000","last_change":"2024-04-02T13:24:53.567111+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567111+0000","last_undegraded":"2024-04-02T13:24:53.567111+0000","last_fullsized":"2024-04-02T13:24:53.567111+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579543+0000","last_change":"2024-04-02T13:24:53.578249+0000","last_active":"2024-04-02T13:24:53.579543+0000","last_peered":"2024-04-02T13:24:53.579543+0000","last_clean":"2024-04-02T13:24:53.579543+0000","last_became_active":"2024-04-02T13:24:53.577992+0000","last_became_peered":"2024-04-02T13:24:53.577992+0000","last_unstale":"2024-04-02T13:24:53.579543+0000","last_undegraded":"2024-04-02T13:24:53.579543+0000","last_fullsized":"2024-04-02T13:24:53.579543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:46:33.336625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561459+0000","last_change":"2024-04-02T13:24:53.561459+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561459+0000","last_undegraded":"2024-04-02T13:24:53.561459+0000","last_fullsized":"2024-04-02T13:24:53.561459+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579095+0000","last_change":"2024-04-02T13:24:53.572838+0000","last_active":"2024-04-02T13:24:53.579095+0000","last_peered":"2024-04-02T13:24:53.579095+0000","last_clean":"2024-04-02T13:24:53.579095+0000","last_became_active":"2024-04-02T13:24:53.571893+0000","last_became_peered":"2024-04-02T13:24:53.571893+0000","last_unstale":"2024-04-02T13:24:53.579095+0000","last_undegraded":"2024-04-02T13:24:53.579095+0000","last_fullsized":"2024-04-02T13:24:53.579095+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:08:46.014739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.563866+0000","last_change":"2024-04-02T13:24:53.563866+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.563866+0000","last_undegraded":"2024-04-02T13:24:53.563866+0000","last_fullsized":"2024-04-02T13:24:53.563866+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.582125+0000","last_change":"2024-04-02T13:24:53.579943+0000","last_active":"2024-04-02T13:24:53.582125+0000","last_peered":"2024-04-02T13:24:53.582125+0000","last_clean":"2024-04-02T13:24:53.582125+0000","last_became_active":"2024-04-02T13:24:53.579632+0000","last_became_peered":"2024-04-02T13:24:53.579632+0000","last_unstale":"2024-04-02T13:24:53.582125+0000","last_undegraded":"2024-04-02T13:24:53.582125+0000","last_fullsized":"2024-04-02T13:24:53.582125+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:06.592645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560270+0000","last_change":"2024-04-02T13:24:53.560270+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560270+0000","last_undegraded":"2024-04-02T13:24:53.560270+0000","last_fullsized":"2024-04-02T13:24:53.560270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.573320+0000","last_change":"2024-04-02T13:24:53.570841+0000","last_active":"2024-04-02T13:24:53.573320+0000","last_peered":"2024-04-02T13:24:53.573320+0000","last_clean":"2024-04-02T13:24:53.573320+0000","last_became_active":"2024-04-02T13:24:53.570567+0000","last_became_peered":"2024-04-02T13:24:53.570567+0000","last_unstale":"2024-04-02T13:24:53.573320+0000","last_undegraded":"2024-04-02T13:24:53.573320+0000","last_fullsized":"2024-04-02T13:24:53.573320+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:30:36.461673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.563298+0000","last_change":"2024-04-02T13:24:53.563298+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.563298+0000","last_undegraded":"2024-04-02T13:24:53.563298+0000","last_fullsized":"2024-04-02T13:24:53.563298+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.581606+0000","last_change":"2024-04-02T13:24:53.577054+0000","last_active":"2024-04-02T13:24:53.581606+0000","last_peered":"2024-04-02T13:24:53.581606+0000","last_clean":"2024-04-02T13:24:53.581606+0000","last_became_active":"2024-04-02T13:24:53.575955+0000","last_became_peered":"2024-04-02T13:24:53.575955+0000","last_unstale":"2024-04-02T13:24:53.581606+0000","last_undegraded":"2024-04-02T13:24:53.581606+0000","last_fullsized":"2024-04-02T13:24:53.581606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:50:57.375379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560085+0000","last_change":"2024-04-02T13:24:53.560085+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560085+0000","last_undegraded":"2024-04-02T13:24:53.560085+0000","last_fullsized":"2024-04-02T13:24:53.560085+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.581965+0000","last_change":"2024-04-02T13:24:53.575781+0000","last_active":"2024-04-02T13:24:53.581965+0000","last_peered":"2024-04-02T13:24:53.581965+0000","last_clean":"2024-04-02T13:24:53.581965+0000","last_became_active":"2024-04-02T13:24:53.574619+0000","last_became_peered":"2024-04-02T13:24:53.574619+0000","last_unstale":"2024-04-02T13:24:53.581965+0000","last_undegraded":"2024-04-02T13:24:53.581965+0000","last_fullsized":"2024-04-02T13:24:53.581965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:14:47.150301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561204+0000","last_change":"2024-04-02T13:24:53.561204+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561204+0000","last_undegraded":"2024-04-02T13:24:53.561204+0000","last_fullsized":"2024-04-02T13:24:53.561204+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.574834+0000","last_change":"2024-04-02T13:24:53.572426+0000","last_active":"2024-04-02T13:24:53.574834+0000","last_peered":"2024-04-02T13:24:53.574834+0000","last_clean":"2024-04-02T13:24:53.574834+0000","last_became_active":"2024-04-02T13:24:53.572153+0000","last_became_peered":"2024-04-02T13:24:53.572153+0000","last_unstale":"2024-04-02T13:24:53.574834+0000","last_undegraded":"2024-04-02T13:24:53.574834+0000","last_fullsized":"2024-04-02T13:24:53.574834+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:18:20.845843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561018+0000","last_change":"2024-04-02T13:24:53.561018+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561018+0000","last_undegraded":"2024-04-02T13:24:53.561018+0000","last_fullsized":"2024-04-02T13:24:53.561018+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.582163+0000","last_change":"2024-04-02T13:24:53.580278+0000","last_active":"2024-04-02T13:24:53.582163+0000","last_peered":"2024-04-02T13:24:53.582163+0000","last_clean":"2024-04-02T13:24:53.582163+0000","last_became_active":"2024-04-02T13:24:53.580002+0000","last_became_peered":"2024-04-02T13:24:53.580002+0000","last_unstale":"2024-04-02T13:24:53.582163+0000","last_undegraded":"2024-04-02T13:24:53.582163+0000","last_fullsized":"2024-04-02T13:24:53.582163+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:51:47.224016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.559858+0000","last_change":"2024-04-02T13:24:53.559858+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.559858+0000","last_undegraded":"2024-04-02T13:24:53.559858+0000","last_fullsized":"2024-04-02T13:24:53.559858+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.581549+0000","last_change":"2024-04-02T13:24:53.576553+0000","last_active":"2024-04-02T13:24:53.581549+0000","last_peered":"2024-04-02T13:24:53.581549+0000","last_clean":"2024-04-02T13:24:53.581549+0000","last_became_active":"2024-04-02T13:24:53.576219+0000","last_became_peered":"2024-04-02T13:24:53.576219+0000","last_unstale":"2024-04-02T13:24:53.581549+0000","last_undegraded":"2024-04-02T13:24:53.581549+0000","last_fullsized":"2024-04-02T13:24:53.581549+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:28.254107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.561070+0000","last_change":"2024-04-02T13:24:53.561070+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.561070+0000","last_undegraded":"2024-04-02T13:24:53.561070+0000","last_fullsized":"2024-04-02T13:24:53.561070+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.578243+0000","last_change":"2024-04-02T13:24:53.575318+0000","last_active":"2024-04-02T13:24:53.578243+0000","last_peered":"2024-04-02T13:24:53.578243+0000","last_clean":"2024-04-02T13:24:53.578243+0000","last_became_active":"2024-04-02T13:24:53.575019+0000","last_became_peered":"2024-04-02T13:24:53.575019+0000","last_unstale":"2024-04-02T13:24:53.578243+0000","last_undegraded":"2024-04-02T13:24:53.578243+0000","last_fullsized":"2024-04-02T13:24:53.578243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:39:30.962000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.562727+0000","last_change":"2024-04-02T13:24:53.562727+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.562727+0000","last_undegraded":"2024-04-02T13:24:53.562727+0000","last_fullsized":"2024-04-02T13:24:53.562727+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.578673+0000","last_change":"2024-04-02T13:24:53.577390+0000","last_active":"2024-04-02T13:24:53.578673+0000","last_peered":"2024-04-02T13:24:53.578673+0000","last_clean":"2024-04-02T13:24:53.578673+0000","last_became_active":"2024-04-02T13:24:53.577079+0000","last_became_peered":"2024-04-02T13:24:53.577079+0000","last_unstale":"2024-04-02T13:24:53.578673+0000","last_undegraded":"2024-04-02T13:24:53.578673+0000","last_fullsized":"2024-04-02T13:24:53.578673+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:46:33.336625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.566607+0000","last_change":"2024-04-02T13:24:53.566607+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.566607+0000","last_undegraded":"2024-04-02T13:24:53.566607+0000","last_fullsized":"2024-04-02T13:24:53.566607+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.582111+0000","last_change":"2024-04-02T13:24:53.577294+0000","last_active":"2024-04-02T13:24:53.582111+0000","last_peered":"2024-04-02T13:24:53.582111+0000","last_clean":"2024-04-02T13:24:53.582111+0000","last_became_active":"2024-04-02T13:24:53.577018+0000","last_became_peered":"2024-04-02T13:24:53.577018+0000","last_unstale":"2024-04-02T13:24:53.582111+0000","last_undegraded":"2024-04-02T13:24:53.582111+0000","last_fullsized":"2024-04-02T13:24:53.582111+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:09:03.943800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565690+0000","last_change":"2024-04-02T13:24:53.565690+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565690+0000","last_undegraded":"2024-04-02T13:24:53.565690+0000","last_fullsized":"2024-04-02T13:24:53.565690+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579186+0000","last_change":"2024-04-02T13:24:53.573420+0000","last_active":"2024-04-02T13:24:53.579186+0000","last_peered":"2024-04-02T13:24:53.579186+0000","last_clean":"2024-04-02T13:24:53.579186+0000","last_became_active":"2024-04-02T13:24:53.571909+0000","last_became_peered":"2024-04-02T13:24:53.571909+0000","last_unstale":"2024-04-02T13:24:53.579186+0000","last_undegraded":"2024-04-02T13:24:53.579186+0000","last_fullsized":"2024-04-02T13:24:53.579186+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:03:28.676340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.562556+0000","last_change":"2024-04-02T13:24:53.562556+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.562556+0000","last_undegraded":"2024-04-02T13:24:53.562556+0000","last_fullsized":"2024-04-02T13:24:53.562556+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.576658+0000","last_change":"2024-04-02T13:24:53.570594+0000","last_active":"2024-04-02T13:24:53.576658+0000","last_peered":"2024-04-02T13:24:53.576658+0000","last_clean":"2024-04-02T13:24:53.576658+0000","last_became_active":"2024-04-02T13:24:53.569678+0000","last_became_peered":"2024-04-02T13:24:53.569678+0000","last_unstale":"2024-04-02T13:24:53.576658+0000","last_undegraded":"2024-04-02T13:24:53.576658+0000","last_fullsized":"2024-04-02T13:24:53.576658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:06.116927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.564134+0000","last_change":"2024-04-02T13:24:53.564134+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.564134+0000","last_undegraded":"2024-04-02T13:24:53.564134+0000","last_fullsized":"2024-04-02T13:24:53.564134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579285+0000","last_change":"2024-04-02T13:24:53.571104+0000","last_active":"2024-04-02T13:24:53.579285+0000","last_peered":"2024-04-02T13:24:53.579285+0000","last_clean":"2024-04-02T13:24:53.579285+0000","last_became_active":"2024-04-02T13:24:53.570758+0000","last_became_peered":"2024-04-02T13:24:53.570758+0000","last_unstale":"2024-04-02T13:24:53.579285+0000","last_undegraded":"2024-04-02T13:24:53.579285+0000","last_fullsized":"2024-04-02T13:24:53.579285+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:48:22.399865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.569084+0000","last_change":"2024-04-02T13:24:53.569084+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.569084+0000","last_undegraded":"2024-04-02T13:24:53.569084+0000","last_fullsized":"2024-04-02T13:24:53.569084+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579375+0000","last_change":"2024-04-02T13:24:53.573510+0000","last_active":"2024-04-02T13:24:53.579375+0000","last_peered":"2024-04-02T13:24:53.579375+0000","last_clean":"2024-04-02T13:24:53.579375+0000","last_became_active":"2024-04-02T13:24:53.572671+0000","last_became_peered":"2024-04-02T13:24:53.572671+0000","last_unstale":"2024-04-02T13:24:53.579375+0000","last_undegraded":"2024-04-02T13:24:53.579375+0000","last_fullsized":"2024-04-02T13:24:53.579375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:11:32.316427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560133+0000","last_change":"2024-04-02T13:24:53.560133+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560133+0000","last_undegraded":"2024-04-02T13:24:53.560133+0000","last_fullsized":"2024-04-02T13:24:53.560133+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.581839+0000","last_change":"2024-04-02T13:24:53.574067+0000","last_active":"2024-04-02T13:24:53.581839+0000","last_peered":"2024-04-02T13:24:53.581839+0000","last_clean":"2024-04-02T13:24:53.581839+0000","last_became_active":"2024-04-02T13:24:53.573799+0000","last_became_peered":"2024-04-02T13:24:53.573799+0000","last_unstale":"2024-04-02T13:24:53.581839+0000","last_undegraded":"2024-04-02T13:24:53.581839+0000","last_fullsized":"2024-04-02T13:24:53.581839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:26:00.810278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.568001+0000","last_change":"2024-04-02T13:24:53.568001+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.568001+0000","last_undegraded":"2024-04-02T13:24:53.568001+0000","last_fullsized":"2024-04-02T13:24:53.568001+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579545+0000","last_change":"2024-04-02T13:24:53.575531+0000","last_active":"2024-04-02T13:24:53.579545+0000","last_peered":"2024-04-02T13:24:53.579545+0000","last_clean":"2024-04-02T13:24:53.579545+0000","last_became_active":"2024-04-02T13:24:53.575222+0000","last_became_peered":"2024-04-02T13:24:53.575222+0000","last_unstale":"2024-04-02T13:24:53.579545+0000","last_undegraded":"2024-04-02T13:24:53.579545+0000","last_fullsized":"2024-04-02T13:24:53.579545+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:48:43.076818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560146+0000","last_change":"2024-04-02T13:24:53.560146+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560146+0000","last_undegraded":"2024-04-02T13:24:53.560146+0000","last_fullsized":"2024-04-02T13:24:53.560146+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.578577+0000","last_change":"2024-04-02T13:24:53.577485+0000","last_active":"2024-04-02T13:24:53.578577+0000","last_peered":"2024-04-02T13:24:53.578577+0000","last_clean":"2024-04-02T13:24:53.578577+0000","last_became_active":"2024-04-02T13:24:53.575330+0000","last_became_peered":"2024-04-02T13:24:53.575330+0000","last_unstale":"2024-04-02T13:24:53.578577+0000","last_undegraded":"2024-04-02T13:24:53.578577+0000","last_fullsized":"2024-04-02T13:24:53.578577+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:51.244520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567636+0000","last_change":"2024-04-02T13:24:53.567636+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567636+0000","last_undegraded":"2024-04-02T13:24:53.567636+0000","last_fullsized":"2024-04-02T13:24:53.567636+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.573910+0000","last_change":"2024-04-02T13:24:53.572963+0000","last_active":"2024-04-02T13:24:53.573910+0000","last_peered":"2024-04-02T13:24:53.573910+0000","last_clean":"2024-04-02T13:24:53.573910+0000","last_became_active":"2024-04-02T13:24:53.572536+0000","last_became_peered":"2024-04-02T13:24:53.572536+0000","last_unstale":"2024-04-02T13:24:53.573910+0000","last_undegraded":"2024-04-02T13:24:53.573910+0000","last_fullsized":"2024-04-02T13:24:53.573910+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:39:30.962000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.567123+0000","last_change":"2024-04-02T13:24:53.567123+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.567123+0000","last_undegraded":"2024-04-02T13:24:53.567123+0000","last_fullsized":"2024-04-02T13:24:53.567123+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.582138+0000","last_change":"2024-04-02T13:24:53.580007+0000","last_active":"2024-04-02T13:24:53.582138+0000","last_peered":"2024-04-02T13:24:53.582138+0000","last_clean":"2024-04-02T13:24:53.582138+0000","last_became_active":"2024-04-02T13:24:53.579686+0000","last_became_peered":"2024-04-02T13:24:53.579686+0000","last_unstale":"2024-04-02T13:24:53.582138+0000","last_undegraded":"2024-04-02T13:24:53.582138+0000","last_fullsized":"2024-04-02T13:24:53.582138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:49.667468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560541+0000","last_change":"2024-04-02T13:24:53.560541+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560541+0000","last_undegraded":"2024-04-02T13:24:53.560541+0000","last_fullsized":"2024-04-02T13:24:53.560541+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579133+0000","last_change":"2024-04-02T13:24:53.572298+0000","last_active":"2024-04-02T13:24:53.579133+0000","last_peered":"2024-04-02T13:24:53.579133+0000","last_clean":"2024-04-02T13:24:53.579133+0000","last_became_active":"2024-04-02T13:24:53.570380+0000","last_became_peered":"2024-04-02T13:24:53.570380+0000","last_unstale":"2024-04-02T13:24:53.579133+0000","last_undegraded":"2024-04-02T13:24:53.579133+0000","last_fullsized":"2024-04-02T13:24:53.579133+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:56:57.490956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.559526+0000","last_change":"2024-04-02T13:24:53.559526+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.559526+0000","last_undegraded":"2024-04-02T13:24:53.559526+0000","last_fullsized":"2024-04-02T13:24:53.559526+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579521+0000","last_change":"2024-04-02T13:24:53.572484+0000","last_active":"2024-04-02T13:24:53.579521+0000","last_peered":"2024-04-02T13:24:53.579521+0000","last_clean":"2024-04-02T13:24:53.579521+0000","last_became_active":"2024-04-02T13:24:53.572124+0000","last_became_peered":"2024-04-02T13:24:53.572124+0000","last_unstale":"2024-04-02T13:24:53.579521+0000","last_undegraded":"2024-04-02T13:24:53.579521+0000","last_fullsized":"2024-04-02T13:24:53.579521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:14:51.783505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.559822+0000","last_change":"2024-04-02T13:24:53.559822+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.559822+0000","last_undegraded":"2024-04-02T13:24:53.559822+0000","last_fullsized":"2024-04-02T13:24:53.559822+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579207+0000","last_change":"2024-04-02T13:24:53.574193+0000","last_active":"2024-04-02T13:24:53.579207+0000","last_peered":"2024-04-02T13:24:53.579207+0000","last_clean":"2024-04-02T13:24:53.579207+0000","last_became_active":"2024-04-02T13:24:53.570565+0000","last_became_peered":"2024-04-02T13:24:53.570565+0000","last_unstale":"2024-04-02T13:24:53.579207+0000","last_undegraded":"2024-04-02T13:24:53.579207+0000","last_fullsized":"2024-04-02T13:24:53.579207+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:48:35.205658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.560048+0000","last_change":"2024-04-02T13:24:53.560048+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.560048+0000","last_undegraded":"2024-04-02T13:24:53.560048+0000","last_fullsized":"2024-04-02T13:24:53.560048+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.574610+0000","last_change":"2024-04-02T13:24:53.567583+0000","last_active":"2024-04-02T13:24:53.574610+0000","last_peered":"2024-04-02T13:24:53.574610+0000","last_clean":"2024-04-02T13:24:53.574610+0000","last_became_active":"2024-04-02T13:24:53.567289+0000","last_became_peered":"2024-04-02T13:24:53.567289+0000","last_unstale":"2024-04-02T13:24:53.574610+0000","last_undegraded":"2024-04-02T13:24:53.574610+0000","last_fullsized":"2024-04-02T13:24:53.574610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:14:47.150301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-02T13:24:53.565970+0000","last_change":"2024-04-02T13:24:53.565970+0000","last_active":"2024-04-02T13:24:53.549098+0000","last_peered":"2024-04-02T13:24:53.549098+0000","last_clean":"2024-04-02T13:24:53.549098+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T13:24:53.565970+0000","last_undegraded":"2024-04-02T13:24:53.565970+0000","last_fullsized":"2024-04-02T13:24:53.565970+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.579322+0000","last_change":"2024-04-02T13:24:53.572298+0000","last_active":"2024-04-02T13:24:53.579322+0000","last_peered":"2024-04-02T13:24:53.579322+0000","last_clean":"2024-04-02T13:24:53.579322+0000","last_became_active":"2024-04-02T13:24:53.572035+0000","last_became_peered":"2024-04-02T13:24:53.572035+0000","last_unstale":"2024-04-02T13:24:53.579322+0000","last_undegraded":"2024-04-02T13:24:53.579322+0000","last_fullsized":"2024-04-02T13:24:53.579322+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:07:25.222536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":278,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-02T13:24:53.568085+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:24:53.568085+0000","last_peered":"2024-04-02T13:24:53.568085+0000","last_clean":"2024-04-02T13:24:53.568085+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:24:53.568085+0000","last_undegraded":"2024-04-02T13:24:53.568085+0000","last_fullsized":"2024-04-02T13:24:53.568085+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1536000,"data_stored":1018118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":427750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":421083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":427750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":942080,"data_stored":427750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":421083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":421083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1536000,"data_stored":1018118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:24:58.956 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:24:59.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.089+0000 7fbd7fd8d640 1 Processor -- start 2024-04-02T13:24:59.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.089+0000 7fbd7fd8d640 1 -- start start 2024-04-02T13:24:59.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7fd8d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbd7815e550 0x7fbd7806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7fd8d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd7815f040 0x7fbd780d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7fd8d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd7815fb50 con 0x7fbd7806d530 2024-04-02T13:24:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7fd8d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fbd7806d970 con 0x7fbd7815e550 2024-04-02T13:24:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7fd8d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbd7806ca90 con 0x7fbd7815f040 2024-04-02T13:24:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7db29640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbd7815e550 0x7fbd7806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7db29640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbd7815e550 0x7fbd7806cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41868/0 (socket says 172.21.15.44:41868) 2024-04-02T13:24:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7db29640 1 -- 172.21.15.44:0/3909095610 learned_addr learned my addr 172.21.15.44:0/3909095610 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7e32a640 1 --2- 172.21.15.44:0/3909095610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd7815f040 0x7fbd780d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7db29640 1 -- 172.21.15.44:0/3909095610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd7815f040 msgr2=0x7fbd780d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7db29640 1 --2- 172.21.15.44:0/3909095610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd7815f040 0x7fbd780d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7db29640 1 -- 172.21.15.44:0/3909095610 >> v1:172.21.15.69:6789/0 conn(0x7fbd7806d530 legacy=0x7fbd7815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7db29640 1 -- 172.21.15.44:0/3909095610 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd7815f590 con 0x7fbd7815e550 2024-04-02T13:24:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.090+0000 7fbd7e32a640 1 --2- 172.21.15.44:0/3909095610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd7815f040 0x7fbd780d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.091+0000 7fbd7db29640 1 --2- 172.21.15.44:0/3909095610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd7815e550 0x7fbd7806cff0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fbd74009870 tx=0x7fbd7402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edfa00afb8a1c778 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.091+0000 7fbd7cb27640 1 -- 172.21.15.44:0/3909095610 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd74034070 con 0x7fbd7815e550 2024-04-02T13:24:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.091+0000 7fbd7cb27640 1 -- 172.21.15.44:0/3909095610 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd74030840 con 0x7fbd7815e550 2024-04-02T13:24:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.091+0000 7fbd7cb27640 1 -- 172.21.15.44:0/3909095610 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd74030c10 con 0x7fbd7815e550 2024-04-02T13:24:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.092+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3909095610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd7815e550 msgr2=0x7fbd7806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.092+0000 7fbd7fd8d640 1 --2- 172.21.15.44:0/3909095610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd7815e550 0x7fbd7806cff0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fbd74009870 tx=0x7fbd7402b980 comp rx=0 tx=0).stop 2024-04-02T13:24:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.092+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3909095610 shutdown_connections 2024-04-02T13:24:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.092+0000 7fbd7fd8d640 1 --2- 172.21.15.44:0/3909095610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd7815f040 0x7fbd780d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.092+0000 7fbd7fd8d640 1 --2- 172.21.15.44:0/3909095610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd7815e550 0x7fbd7806cff0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.092+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3909095610 >> 172.21.15.44:0/3909095610 conn(0x7fbd7809aa20 msgr2=0x7fbd7809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.092+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3909095610 shutdown_connections 2024-04-02T13:24:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.092+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3909095610 wait complete. 2024-04-02T13:24:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.093+0000 7fbd7fd8d640 1 Processor -- start 2024-04-02T13:24:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.093+0000 7fbd7fd8d640 1 -- start start 2024-04-02T13:24:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7fd8d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd7806d530 0x7fbd781f5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7fd8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd7815e550 0x7fbd781503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7fd8d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd7815f040 0x7fbd780b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7fd8d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd780d0590 con 0x7fbd7815e550 2024-04-02T13:24:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7fd8d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbd780d02b0 con 0x7fbd7815f040 2024-04-02T13:24:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7fd8d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbd780cc4b0 con 0x7fbd7806d530 2024-04-02T13:24:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7d328640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd7815e550 0x7fbd781503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7d328640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd7815e550 0x7fbd781503e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41884/0 (socket says 172.21.15.44:41884) 2024-04-02T13:24:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7d328640 1 -- 172.21.15.44:0/3475539050 learned_addr learned my addr 172.21.15.44:0/3475539050 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7e32a640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd7815f040 0x7fbd780b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7db29640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd7806d530 0x7fbd781f5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7d328640 1 -- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd7806d530 msgr2=0x7fbd781f5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7d328640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd7806d530 0x7fbd781f5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7d328640 1 -- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd7815f040 msgr2=0x7fbd780b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7d328640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd7815f040 0x7fbd780b1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7d328640 1 -- 172.21.15.44:0/3475539050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd780b1ea0 con 0x7fbd7815e550 2024-04-02T13:24:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7d328640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd7815e550 0x7fbd781503e0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800b3f0 tx=0x7fbd6800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd6e7fc640 1 -- 172.21.15.44:0/3475539050 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd68015020 con 0x7fbd7815e550 2024-04-02T13:24:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd6e7fc640 1 -- 172.21.15.44:0/3475539050 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd68011830 con 0x7fbd7815e550 2024-04-02T13:24:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3475539050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd780b2180 con 0x7fbd7815e550 2024-04-02T13:24:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd6e7fc640 1 -- 172.21.15.44:0/3475539050 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd68011c20 con 0x7fbd7815e550 2024-04-02T13:24:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.094+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3475539050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd780b2740 con 0x7fbd7815e550 2024-04-02T13:24:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.095+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3475539050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd780aebc0 con 0x7fbd7815e550 2024-04-02T13:24:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.096+0000 7fbd6e7fc640 1 -- 172.21.15.44:0/3475539050 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbd68018020 con 0x7fbd7815e550 2024-04-02T13:24:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.096+0000 7fbd6e7fc640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd5c06c920 0x7fbd5c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.096+0000 7fbd6e7fc640 1 -- 172.21.15.44:0/3475539050 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fbd6808bcb0 con 0x7fbd7815e550 2024-04-02T13:24:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.096+0000 7fbd7db29640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd5c06c920 0x7fbd5c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.097+0000 7fbd7db29640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd5c06c920 0x7fbd5c06ede0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fbd74036000 tx=0x7fbd7403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:59.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.101+0000 7fbd6e7fc640 1 -- 172.21.15.44:0/3475539050 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbd6801f020 con 0x7fbd7815e550 2024-04-02T13:24:59.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.280+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3475539050 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbd780684f0 con 0x7fbd5c06c920 2024-04-02T13:24:59.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.287+0000 7fbd6e7fc640 1 -- 172.21.15.44:0/3475539050 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252097 (secure 0 0 0) 0x7fbd780684f0 con 0x7fbd5c06c920 2024-04-02T13:24:59.293 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:59.294 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:24:59.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.290+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd5c06c920 msgr2=0x7fbd5c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.290+0000 7fbd7fd8d640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd5c06c920 0x7fbd5c06ede0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fbd74036000 tx=0x7fbd7403b040 comp rx=0 tx=0).stop 2024-04-02T13:24:59.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.290+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3475539050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd7815e550 msgr2=0x7fbd781503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.290+0000 7fbd7fd8d640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd7815e550 0x7fbd781503e0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800b3f0 tx=0x7fbd6800b8b0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.290+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3475539050 shutdown_connections 2024-04-02T13:24:59.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.290+0000 7fbd7fd8d640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbd5c06c920 0x7fbd5c06ede0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.290+0000 7fbd7fd8d640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd7815f040 0x7fbd780b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.290+0000 7fbd7fd8d640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd7815e550 0x7fbd781503e0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.290+0000 7fbd7fd8d640 1 --2- 172.21.15.44:0/3475539050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd7806d530 0x7fbd781f5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.290+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3475539050 >> 172.21.15.44:0/3475539050 conn(0x7fbd7809aa20 msgr2=0x7fbd7806f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:59.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.290+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3475539050 shutdown_connections 2024-04-02T13:24:59.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.290+0000 7fbd7fd8d640 1 -- 172.21.15.44:0/3475539050 wait complete. 2024-04-02T13:24:59.303 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-04-02T13:24:58.941614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241600,"kb_used_data":8776,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754733120,"statfs":{"total":773094113280,"available":772846714880,"internally_reserved":0,"allocated":8986624,"data_stored":4718428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001328"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064800+0000","last_change":"2024-04-02T13:24:54.582190+0000","last_active":"2024-04-02T13:24:56.064800+0000","last_peered":"2024-04-02T13:24:56.064800+0000","last_clean":"2024-04-02T13:24:56.064800+0000","last_became_active":"2024-04-02T13:24:54.581814+0000","last_became_peered":"2024-04-02T13:24:54.581814+0000","last_unstale":"2024-04-02T13:24:56.064800+0000","last_undegraded":"2024-04-02T13:24:56.064800+0000","last_fullsized":"2024-04-02T13:24:56.064800+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:14:01.535902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064308+0000","last_change":"2024-04-02T13:24:54.584115+0000","last_active":"2024-04-02T13:24:56.064308+0000","last_peered":"2024-04-02T13:24:56.064308+0000","last_clean":"2024-04-02T13:24:56.064308+0000","last_became_active":"2024-04-02T13:24:54.583792+0000","last_became_peered":"2024-04-02T13:24:54.583792+0000","last_unstale":"2024-04-02T13:24:56.064308+0000","last_undegraded":"2024-04-02T13:24:56.064308+0000","last_fullsized":"2024-04-02T13:24:56.064308+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:30:34.688997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574787+0000","last_change":"2024-04-02T13:24:54.574235+0000","last_active":"2024-04-02T13:24:55.574787+0000","last_peered":"2024-04-02T13:24:55.574787+0000","last_clean":"2024-04-02T13:24:55.574787+0000","last_became_active":"2024-04-02T13:24:54.573908+0000","last_became_peered":"2024-04-02T13:24:54.573908+0000","last_unstale":"2024-04-02T13:24:55.574787+0000","last_undegraded":"2024-04-02T13:24:55.574787+0000","last_fullsized":"2024-04-02T13:24:55.574787+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:18:21.848167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574407+0000","last_change":"2024-04-02T13:24:54.581284+0000","last_active":"2024-04-02T13:24:55.574407+0000","last_peered":"2024-04-02T13:24:55.574407+0000","last_clean":"2024-04-02T13:24:55.574407+0000","last_became_active":"2024-04-02T13:24:54.580976+0000","last_became_peered":"2024-04-02T13:24:54.580976+0000","last_unstale":"2024-04-02T13:24:55.574407+0000","last_undegraded":"2024-04-02T13:24:55.574407+0000","last_fullsized":"2024-04-02T13:24:55.574407+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:25.789065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064080+0000","last_change":"2024-04-02T13:24:54.580261+0000","last_active":"2024-04-02T13:24:56.064080+0000","last_peered":"2024-04-02T13:24:56.064080+0000","last_clean":"2024-04-02T13:24:56.064080+0000","last_became_active":"2024-04-02T13:24:54.579973+0000","last_became_peered":"2024-04-02T13:24:54.579973+0000","last_unstale":"2024-04-02T13:24:56.064080+0000","last_undegraded":"2024-04-02T13:24:56.064080+0000","last_fullsized":"2024-04-02T13:24:56.064080+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:43:00.164244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056185+0000","last_change":"2024-04-02T13:24:54.585758+0000","last_active":"2024-04-02T13:24:56.056185+0000","last_peered":"2024-04-02T13:24:56.056185+0000","last_clean":"2024-04-02T13:24:56.056185+0000","last_became_active":"2024-04-02T13:24:54.585490+0000","last_became_peered":"2024-04-02T13:24:54.585490+0000","last_unstale":"2024-04-02T13:24:56.056185+0000","last_undegraded":"2024-04-02T13:24:56.056185+0000","last_fullsized":"2024-04-02T13:24:56.056185+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:16.381648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064052+0000","last_change":"2024-04-02T13:24:54.577884+0000","last_active":"2024-04-02T13:24:56.064052+0000","last_peered":"2024-04-02T13:24:56.064052+0000","last_clean":"2024-04-02T13:24:56.064052+0000","last_became_active":"2024-04-02T13:24:54.577328+0000","last_became_peered":"2024-04-02T13:24:54.577328+0000","last_unstale":"2024-04-02T13:24:56.064052+0000","last_undegraded":"2024-04-02T13:24:56.064052+0000","last_fullsized":"2024-04-02T13:24:56.064052+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:48:44.079142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063748+0000","last_change":"2024-04-02T13:24:54.579426+0000","last_active":"2024-04-02T13:24:56.063748+0000","last_peered":"2024-04-02T13:24:56.063748+0000","last_clean":"2024-04-02T13:24:56.063748+0000","last_became_active":"2024-04-02T13:24:54.579147+0000","last_became_peered":"2024-04-02T13:24:54.579147+0000","last_unstale":"2024-04-02T13:24:56.063748+0000","last_undegraded":"2024-04-02T13:24:56.063748+0000","last_fullsized":"2024-04-02T13:24:56.063748+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:49:38.705972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574247+0000","last_change":"2024-04-02T13:24:54.587257+0000","last_active":"2024-04-02T13:24:55.574247+0000","last_peered":"2024-04-02T13:24:55.574247+0000","last_clean":"2024-04-02T13:24:55.574247+0000","last_became_active":"2024-04-02T13:24:54.586571+0000","last_became_peered":"2024-04-02T13:24:54.586571+0000","last_unstale":"2024-04-02T13:24:55.574247+0000","last_undegraded":"2024-04-02T13:24:55.574247+0000","last_fullsized":"2024-04-02T13:24:55.574247+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:43:27.497564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573950+0000","last_change":"2024-04-02T13:24:54.570536+0000","last_active":"2024-04-02T13:24:55.573950+0000","last_peered":"2024-04-02T13:24:55.573950+0000","last_clean":"2024-04-02T13:24:55.573950+0000","last_became_active":"2024-04-02T13:24:54.570264+0000","last_became_peered":"2024-04-02T13:24:54.570264+0000","last_unstale":"2024-04-02T13:24:55.573950+0000","last_undegraded":"2024-04-02T13:24:55.573950+0000","last_fullsized":"2024-04-02T13:24:55.573950+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:58:28.798683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064238+0000","last_change":"2024-04-02T13:24:54.584742+0000","last_active":"2024-04-02T13:24:56.064238+0000","last_peered":"2024-04-02T13:24:56.064238+0000","last_clean":"2024-04-02T13:24:56.064238+0000","last_became_active":"2024-04-02T13:24:54.584094+0000","last_became_peered":"2024-04-02T13:24:54.584094+0000","last_unstale":"2024-04-02T13:24:56.064238+0000","last_undegraded":"2024-04-02T13:24:56.064238+0000","last_fullsized":"2024-04-02T13:24:56.064238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:16.445368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574563+0000","last_change":"2024-04-02T13:24:54.574509+0000","last_active":"2024-04-02T13:24:55.574563+0000","last_peered":"2024-04-02T13:24:55.574563+0000","last_clean":"2024-04-02T13:24:55.574563+0000","last_became_active":"2024-04-02T13:24:54.574199+0000","last_became_peered":"2024-04-02T13:24:54.574199+0000","last_unstale":"2024-04-02T13:24:55.574563+0000","last_undegraded":"2024-04-02T13:24:55.574563+0000","last_fullsized":"2024-04-02T13:24:55.574563+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:48:23.402189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064197+0000","last_change":"2024-04-02T13:24:54.586211+0000","last_active":"2024-04-02T13:24:56.064197+0000","last_peered":"2024-04-02T13:24:56.064197+0000","last_clean":"2024-04-02T13:24:56.064197+0000","last_became_active":"2024-04-02T13:24:54.585863+0000","last_became_peered":"2024-04-02T13:24:54.585863+0000","last_unstale":"2024-04-02T13:24:56.064197+0000","last_undegraded":"2024-04-02T13:24:56.064197+0000","last_fullsized":"2024-04-02T13:24:56.064197+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:15:41.434299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056212+0000","last_change":"2024-04-02T13:24:54.584568+0000","last_active":"2024-04-02T13:24:56.056212+0000","last_peered":"2024-04-02T13:24:56.056212+0000","last_clean":"2024-04-02T13:24:56.056212+0000","last_became_active":"2024-04-02T13:24:54.583120+0000","last_became_peered":"2024-04-02T13:24:54.583120+0000","last_unstale":"2024-04-02T13:24:56.056212+0000","last_undegraded":"2024-04-02T13:24:56.056212+0000","last_fullsized":"2024-04-02T13:24:56.056212+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:08.440154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063782+0000","last_change":"2024-04-02T13:24:54.576524+0000","last_active":"2024-04-02T13:24:56.063782+0000","last_peered":"2024-04-02T13:24:56.063782+0000","last_clean":"2024-04-02T13:24:56.063782+0000","last_became_active":"2024-04-02T13:24:54.576151+0000","last_became_peered":"2024-04-02T13:24:54.576151+0000","last_unstale":"2024-04-02T13:24:56.063782+0000","last_undegraded":"2024-04-02T13:24:56.063782+0000","last_fullsized":"2024-04-02T13:24:56.063782+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:24:50.774098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064257+0000","last_change":"2024-04-02T13:24:54.568847+0000","last_active":"2024-04-02T13:24:56.064257+0000","last_peered":"2024-04-02T13:24:56.064257+0000","last_clean":"2024-04-02T13:24:56.064257+0000","last_became_active":"2024-04-02T13:24:54.568551+0000","last_became_peered":"2024-04-02T13:24:54.568551+0000","last_unstale":"2024-04-02T13:24:56.064257+0000","last_undegraded":"2024-04-02T13:24:56.064257+0000","last_fullsized":"2024-04-02T13:24:56.064257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:24:50.774098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064735+0000","last_change":"2024-04-02T13:24:54.580503+0000","last_active":"2024-04-02T13:24:56.064735+0000","last_peered":"2024-04-02T13:24:56.064735+0000","last_clean":"2024-04-02T13:24:56.064735+0000","last_became_active":"2024-04-02T13:24:54.580207+0000","last_became_peered":"2024-04-02T13:24:54.580207+0000","last_unstale":"2024-04-02T13:24:56.064735+0000","last_undegraded":"2024-04-02T13:24:56.064735+0000","last_fullsized":"2024-04-02T13:24:56.064735+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:18:08.897243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574697+0000","last_change":"2024-04-02T13:24:54.582497+0000","last_active":"2024-04-02T13:24:55.574697+0000","last_peered":"2024-04-02T13:24:55.574697+0000","last_clean":"2024-04-02T13:24:55.574697+0000","last_became_active":"2024-04-02T13:24:54.582221+0000","last_became_peered":"2024-04-02T13:24:54.582221+0000","last_unstale":"2024-04-02T13:24:55.574697+0000","last_undegraded":"2024-04-02T13:24:55.574697+0000","last_fullsized":"2024-04-02T13:24:55.574697+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:30:37.463997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056042+0000","last_change":"2024-04-02T13:24:54.586251+0000","last_active":"2024-04-02T13:24:56.056042+0000","last_peered":"2024-04-02T13:24:56.056042+0000","last_clean":"2024-04-02T13:24:56.056042+0000","last_became_active":"2024-04-02T13:24:54.585975+0000","last_became_peered":"2024-04-02T13:24:54.585975+0000","last_unstale":"2024-04-02T13:24:56.056042+0000","last_undegraded":"2024-04-02T13:24:56.056042+0000","last_fullsized":"2024-04-02T13:24:56.056042+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:03.634266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064056+0000","last_change":"2024-04-02T13:24:54.583263+0000","last_active":"2024-04-02T13:24:56.064056+0000","last_peered":"2024-04-02T13:24:56.064056+0000","last_clean":"2024-04-02T13:24:56.064056+0000","last_became_active":"2024-04-02T13:24:54.582968+0000","last_became_peered":"2024-04-02T13:24:54.582968+0000","last_unstale":"2024-04-02T13:24:56.064056+0000","last_undegraded":"2024-04-02T13:24:56.064056+0000","last_fullsized":"2024-04-02T13:24:56.064056+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:00:37.401118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574383+0000","last_change":"2024-04-02T13:24:54.580613+0000","last_active":"2024-04-02T13:24:55.574383+0000","last_peered":"2024-04-02T13:24:55.574383+0000","last_clean":"2024-04-02T13:24:55.574383+0000","last_became_active":"2024-04-02T13:24:54.580356+0000","last_became_peered":"2024-04-02T13:24:54.580356+0000","last_unstale":"2024-04-02T13:24:55.574383+0000","last_undegraded":"2024-04-02T13:24:55.574383+0000","last_fullsized":"2024-04-02T13:24:55.574383+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:50:28.795387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064138+0000","last_change":"2024-04-02T13:24:54.586237+0000","last_active":"2024-04-02T13:24:56.064138+0000","last_peered":"2024-04-02T13:24:56.064138+0000","last_clean":"2024-04-02T13:24:56.064138+0000","last_became_active":"2024-04-02T13:24:54.585910+0000","last_became_peered":"2024-04-02T13:24:54.585910+0000","last_unstale":"2024-04-02T13:24:56.064138+0000","last_undegraded":"2024-04-02T13:24:56.064138+0000","last_fullsized":"2024-04-02T13:24:56.064138+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:02:06.412550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574181+0000","last_change":"2024-04-02T13:24:54.573990+0000","last_active":"2024-04-02T13:24:55.574181+0000","last_peered":"2024-04-02T13:24:55.574181+0000","last_clean":"2024-04-02T13:24:55.574181+0000","last_became_active":"2024-04-02T13:24:54.573671+0000","last_became_peered":"2024-04-02T13:24:54.573671+0000","last_unstale":"2024-04-02T13:24:55.574181+0000","last_undegraded":"2024-04-02T13:24:55.574181+0000","last_fullsized":"2024-04-02T13:24:55.574181+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:29:58.020569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063647+0000","last_change":"2024-04-02T13:24:54.589275+0000","last_active":"2024-04-02T13:24:56.063647+0000","last_peered":"2024-04-02T13:24:56.063647+0000","last_clean":"2024-04-02T13:24:56.063647+0000","last_became_active":"2024-04-02T13:24:54.589010+0000","last_became_peered":"2024-04-02T13:24:54.589010+0000","last_unstale":"2024-04-02T13:24:56.063647+0000","last_undegraded":"2024-04-02T13:24:56.063647+0000","last_fullsized":"2024-04-02T13:24:56.063647+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:10:41.126700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064560+0000","last_change":"2024-04-02T13:24:54.590378+0000","last_active":"2024-04-02T13:24:56.064560+0000","last_peered":"2024-04-02T13:24:56.064560+0000","last_clean":"2024-04-02T13:24:56.064560+0000","last_became_active":"2024-04-02T13:24:54.589691+0000","last_became_peered":"2024-04-02T13:24:54.589691+0000","last_unstale":"2024-04-02T13:24:56.064560+0000","last_undegraded":"2024-04-02T13:24:56.064560+0000","last_fullsized":"2024-04-02T13:24:56.064560+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:16.381648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574149+0000","last_change":"2024-04-02T13:24:54.575310+0000","last_active":"2024-04-02T13:24:55.574149+0000","last_peered":"2024-04-02T13:24:55.574149+0000","last_clean":"2024-04-02T13:24:55.574149+0000","last_became_active":"2024-04-02T13:24:54.575029+0000","last_became_peered":"2024-04-02T13:24:54.575029+0000","last_unstale":"2024-04-02T13:24:55.574149+0000","last_undegraded":"2024-04-02T13:24:55.574149+0000","last_fullsized":"2024-04-02T13:24:55.574149+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:12.654707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064538+0000","last_change":"2024-04-02T13:24:54.572607+0000","last_active":"2024-04-02T13:24:56.064538+0000","last_peered":"2024-04-02T13:24:56.064538+0000","last_clean":"2024-04-02T13:24:56.064538+0000","last_became_active":"2024-04-02T13:24:54.571815+0000","last_became_peered":"2024-04-02T13:24:54.571815+0000","last_unstale":"2024-04-02T13:24:56.064538+0000","last_undegraded":"2024-04-02T13:24:56.064538+0000","last_fullsized":"2024-04-02T13:24:56.064538+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:16.445368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064414+0000","last_change":"2024-04-02T13:24:54.566761+0000","last_active":"2024-04-02T13:24:56.064414+0000","last_peered":"2024-04-02T13:24:56.064414+0000","last_clean":"2024-04-02T13:24:56.064414+0000","last_became_active":"2024-04-02T13:24:54.566488+0000","last_became_peered":"2024-04-02T13:24:54.566488+0000","last_unstale":"2024-04-02T13:24:56.064414+0000","last_undegraded":"2024-04-02T13:24:56.064414+0000","last_fullsized":"2024-04-02T13:24:56.064414+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:14:10.643231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064315+0000","last_change":"2024-04-02T13:24:54.588347+0000","last_active":"2024-04-02T13:24:56.064315+0000","last_peered":"2024-04-02T13:24:56.064315+0000","last_clean":"2024-04-02T13:24:56.064315+0000","last_became_active":"2024-04-02T13:24:54.588078+0000","last_became_peered":"2024-04-02T13:24:54.588078+0000","last_unstale":"2024-04-02T13:24:56.064315+0000","last_undegraded":"2024-04-02T13:24:56.064315+0000","last_fullsized":"2024-04-02T13:24:56.064315+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:14:01.535902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064151+0000","last_change":"2024-04-02T13:24:54.571000+0000","last_active":"2024-04-02T13:24:56.064151+0000","last_peered":"2024-04-02T13:24:56.064151+0000","last_clean":"2024-04-02T13:24:56.064151+0000","last_became_active":"2024-04-02T13:24:54.570722+0000","last_became_peered":"2024-04-02T13:24:54.570722+0000","last_unstale":"2024-04-02T13:24:56.064151+0000","last_undegraded":"2024-04-02T13:24:56.064151+0000","last_fullsized":"2024-04-02T13:24:56.064151+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:42:33.303173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574577+0000","last_change":"2024-04-02T13:24:54.582308+0000","last_active":"2024-04-02T13:24:55.574577+0000","last_peered":"2024-04-02T13:24:55.574577+0000","last_clean":"2024-04-02T13:24:55.574577+0000","last_became_active":"2024-04-02T13:24:54.582030+0000","last_became_peered":"2024-04-02T13:24:54.582030+0000","last_unstale":"2024-04-02T13:24:55.574577+0000","last_undegraded":"2024-04-02T13:24:55.574577+0000","last_fullsized":"2024-04-02T13:24:55.574577+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:53:36.691313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064449+0000","last_change":"2024-04-02T13:24:54.577125+0000","last_active":"2024-04-02T13:24:56.064449+0000","last_peered":"2024-04-02T13:24:56.064449+0000","last_clean":"2024-04-02T13:24:56.064449+0000","last_became_active":"2024-04-02T13:24:54.576839+0000","last_became_peered":"2024-04-02T13:24:54.576839+0000","last_unstale":"2024-04-02T13:24:56.064449+0000","last_undegraded":"2024-04-02T13:24:56.064449+0000","last_fullsized":"2024-04-02T13:24:56.064449+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:59.553628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574571+0000","last_change":"2024-04-02T13:24:54.576466+0000","last_active":"2024-04-02T13:24:55.574571+0000","last_peered":"2024-04-02T13:24:55.574571+0000","last_clean":"2024-04-02T13:24:55.574571+0000","last_became_active":"2024-04-02T13:24:54.576195+0000","last_became_peered":"2024-04-02T13:24:54.576195+0000","last_unstale":"2024-04-02T13:24:55.574571+0000","last_undegraded":"2024-04-02T13:24:55.574571+0000","last_fullsized":"2024-04-02T13:24:55.574571+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:43:11.976047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"78'4","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064703+0000","last_change":"2024-04-02T13:24:53.573023+0000","last_active":"2024-04-02T13:24:56.064703+0000","last_peered":"2024-04-02T13:24:56.064703+0000","last_clean":"2024-04-02T13:24:56.064703+0000","last_became_active":"2024-04-02T13:24:53.571627+0000","last_became_peered":"2024-04-02T13:24:53.571627+0000","last_unstale":"2024-04-02T13:24:56.064703+0000","last_undegraded":"2024-04-02T13:24:56.064703+0000","last_fullsized":"2024-04-02T13:24:56.064703+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:05:17.417576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063953+0000","last_change":"2024-04-02T13:24:54.572282+0000","last_active":"2024-04-02T13:24:56.063953+0000","last_peered":"2024-04-02T13:24:56.063953+0000","last_clean":"2024-04-02T13:24:56.063953+0000","last_became_active":"2024-04-02T13:24:54.572000+0000","last_became_peered":"2024-04-02T13:24:54.572000+0000","last_unstale":"2024-04-02T13:24:56.063953+0000","last_undegraded":"2024-04-02T13:24:56.063953+0000","last_fullsized":"2024-04-02T13:24:56.063953+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:59.553628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064680+0000","last_change":"2024-04-02T13:24:53.573555+0000","last_active":"2024-04-02T13:24:56.064680+0000","last_peered":"2024-04-02T13:24:56.064680+0000","last_clean":"2024-04-02T13:24:56.064680+0000","last_became_active":"2024-04-02T13:24:53.573158+0000","last_became_peered":"2024-04-02T13:24:53.573158+0000","last_unstale":"2024-04-02T13:24:56.064680+0000","last_undegraded":"2024-04-02T13:24:56.064680+0000","last_fullsized":"2024-04-02T13:24:56.064680+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:58:27.796359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574630+0000","last_change":"2024-04-02T13:24:54.576685+0000","last_active":"2024-04-02T13:24:55.574630+0000","last_peered":"2024-04-02T13:24:55.574630+0000","last_clean":"2024-04-02T13:24:55.574630+0000","last_became_active":"2024-04-02T13:24:54.576321+0000","last_became_peered":"2024-04-02T13:24:54.576321+0000","last_unstale":"2024-04-02T13:24:55.574630+0000","last_undegraded":"2024-04-02T13:24:55.574630+0000","last_fullsized":"2024-04-02T13:24:55.574630+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:27.408601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064612+0000","last_change":"2024-04-02T13:24:53.578788+0000","last_active":"2024-04-02T13:24:56.064612+0000","last_peered":"2024-04-02T13:24:56.064612+0000","last_clean":"2024-04-02T13:24:56.064612+0000","last_became_active":"2024-04-02T13:24:53.578523+0000","last_became_peered":"2024-04-02T13:24:53.578523+0000","last_unstale":"2024-04-02T13:24:56.064612+0000","last_undegraded":"2024-04-02T13:24:56.064612+0000","last_fullsized":"2024-04-02T13:24:56.064612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:22:06.708202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574544+0000","last_change":"2024-04-02T13:24:54.580439+0000","last_active":"2024-04-02T13:24:55.574544+0000","last_peered":"2024-04-02T13:24:55.574544+0000","last_clean":"2024-04-02T13:24:55.574544+0000","last_became_active":"2024-04-02T13:24:54.579623+0000","last_became_peered":"2024-04-02T13:24:54.579623+0000","last_unstale":"2024-04-02T13:24:55.574544+0000","last_undegraded":"2024-04-02T13:24:55.574544+0000","last_fullsized":"2024-04-02T13:24:55.574544+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:48:59.714798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064200+0000","last_change":"2024-04-02T13:24:53.575848+0000","last_active":"2024-04-02T13:24:56.064200+0000","last_peered":"2024-04-02T13:24:56.064200+0000","last_clean":"2024-04-02T13:24:56.064200+0000","last_became_active":"2024-04-02T13:24:53.574537+0000","last_became_peered":"2024-04-02T13:24:53.574537+0000","last_unstale":"2024-04-02T13:24:56.064200+0000","last_undegraded":"2024-04-02T13:24:56.064200+0000","last_fullsized":"2024-04-02T13:24:56.064200+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:02:30.957187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064736+0000","last_change":"2024-04-02T13:24:54.583494+0000","last_active":"2024-04-02T13:24:56.064736+0000","last_peered":"2024-04-02T13:24:56.064736+0000","last_clean":"2024-04-02T13:24:56.064736+0000","last_became_active":"2024-04-02T13:24:54.580371+0000","last_became_peered":"2024-04-02T13:24:54.580371+0000","last_unstale":"2024-04-02T13:24:56.064736+0000","last_undegraded":"2024-04-02T13:24:56.064736+0000","last_fullsized":"2024-04-02T13:24:56.064736+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:03.634266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574609+0000","last_change":"2024-04-02T13:24:53.574172+0000","last_active":"2024-04-02T13:24:55.574609+0000","last_peered":"2024-04-02T13:24:55.574609+0000","last_clean":"2024-04-02T13:24:55.574609+0000","last_became_active":"2024-04-02T13:24:53.573883+0000","last_became_peered":"2024-04-02T13:24:53.573883+0000","last_unstale":"2024-04-02T13:24:55.574609+0000","last_undegraded":"2024-04-02T13:24:55.574609+0000","last_fullsized":"2024-04-02T13:24:55.574609+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:08.877638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064443+0000","last_change":"2024-04-02T13:24:54.585871+0000","last_active":"2024-04-02T13:24:56.064443+0000","last_peered":"2024-04-02T13:24:56.064443+0000","last_clean":"2024-04-02T13:24:56.064443+0000","last_became_active":"2024-04-02T13:24:54.585586+0000","last_became_peered":"2024-04-02T13:24:54.585586+0000","last_unstale":"2024-04-02T13:24:56.064443+0000","last_undegraded":"2024-04-02T13:24:56.064443+0000","last_fullsized":"2024-04-02T13:24:56.064443+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:04:44.744456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573845+0000","last_change":"2024-04-02T13:24:53.571030+0000","last_active":"2024-04-02T13:24:55.573845+0000","last_peered":"2024-04-02T13:24:55.573845+0000","last_clean":"2024-04-02T13:24:55.573845+0000","last_became_active":"2024-04-02T13:24:53.570763+0000","last_became_peered":"2024-04-02T13:24:53.570763+0000","last_unstale":"2024-04-02T13:24:55.573845+0000","last_undegraded":"2024-04-02T13:24:55.573845+0000","last_fullsized":"2024-04-02T13:24:55.573845+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:07.392500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574899+0000","last_change":"2024-04-02T13:24:54.586818+0000","last_active":"2024-04-02T13:24:55.574899+0000","last_peered":"2024-04-02T13:24:55.574899+0000","last_clean":"2024-04-02T13:24:55.574899+0000","last_became_active":"2024-04-02T13:24:54.586528+0000","last_became_peered":"2024-04-02T13:24:54.586528+0000","last_unstale":"2024-04-02T13:24:55.574899+0000","last_undegraded":"2024-04-02T13:24:55.574899+0000","last_fullsized":"2024-04-02T13:24:55.574899+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:13:07.461166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064057+0000","last_change":"2024-04-02T13:24:53.578249+0000","last_active":"2024-04-02T13:24:56.064057+0000","last_peered":"2024-04-02T13:24:56.064057+0000","last_clean":"2024-04-02T13:24:56.064057+0000","last_became_active":"2024-04-02T13:24:53.577992+0000","last_became_peered":"2024-04-02T13:24:53.577992+0000","last_unstale":"2024-04-02T13:24:56.064057+0000","last_undegraded":"2024-04-02T13:24:56.064057+0000","last_fullsized":"2024-04-02T13:24:56.064057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:23:42.359046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064672+0000","last_change":"2024-04-02T13:24:54.578694+0000","last_active":"2024-04-02T13:24:56.064672+0000","last_peered":"2024-04-02T13:24:56.064672+0000","last_clean":"2024-04-02T13:24:56.064672+0000","last_became_active":"2024-04-02T13:24:54.578394+0000","last_became_peered":"2024-04-02T13:24:54.578394+0000","last_unstale":"2024-04-02T13:24:56.064672+0000","last_undegraded":"2024-04-02T13:24:56.064672+0000","last_fullsized":"2024-04-02T13:24:56.064672+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:07.594969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063737+0000","last_change":"2024-04-02T13:24:53.572838+0000","last_active":"2024-04-02T13:24:56.063737+0000","last_peered":"2024-04-02T13:24:56.063737+0000","last_clean":"2024-04-02T13:24:56.063737+0000","last_became_active":"2024-04-02T13:24:53.571893+0000","last_became_peered":"2024-04-02T13:24:53.571893+0000","last_unstale":"2024-04-02T13:24:56.063737+0000","last_undegraded":"2024-04-02T13:24:56.063737+0000","last_fullsized":"2024-04-02T13:24:56.063737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:06:01.172766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063725+0000","last_change":"2024-04-02T13:24:54.579310+0000","last_active":"2024-04-02T13:24:56.063725+0000","last_peered":"2024-04-02T13:24:56.063725+0000","last_clean":"2024-04-02T13:24:56.063725+0000","last_became_active":"2024-04-02T13:24:54.578356+0000","last_became_peered":"2024-04-02T13:24:54.578356+0000","last_unstale":"2024-04-02T13:24:56.063725+0000","last_undegraded":"2024-04-02T13:24:56.063725+0000","last_fullsized":"2024-04-02T13:24:56.063725+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:42:33.303173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064892+0000","last_change":"2024-04-02T13:24:53.579943+0000","last_active":"2024-04-02T13:24:56.064892+0000","last_peered":"2024-04-02T13:24:56.064892+0000","last_clean":"2024-04-02T13:24:56.064892+0000","last_became_active":"2024-04-02T13:24:53.579632+0000","last_became_peered":"2024-04-02T13:24:53.579632+0000","last_unstale":"2024-04-02T13:24:56.064892+0000","last_undegraded":"2024-04-02T13:24:56.064892+0000","last_fullsized":"2024-04-02T13:24:56.064892+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:48:22.399865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573933+0000","last_change":"2024-04-02T13:24:54.580573+0000","last_active":"2024-04-02T13:24:55.573933+0000","last_peered":"2024-04-02T13:24:55.573933+0000","last_clean":"2024-04-02T13:24:55.573933+0000","last_became_active":"2024-04-02T13:24:54.580323+0000","last_became_peered":"2024-04-02T13:24:54.580323+0000","last_unstale":"2024-04-02T13:24:55.573933+0000","last_undegraded":"2024-04-02T13:24:55.573933+0000","last_fullsized":"2024-04-02T13:24:55.573933+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:43:43.130201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574105+0000","last_change":"2024-04-02T13:24:53.570841+0000","last_active":"2024-04-02T13:24:55.574105+0000","last_peered":"2024-04-02T13:24:55.574105+0000","last_clean":"2024-04-02T13:24:55.574105+0000","last_became_active":"2024-04-02T13:24:53.570567+0000","last_became_peered":"2024-04-02T13:24:53.570567+0000","last_unstale":"2024-04-02T13:24:55.574105+0000","last_undegraded":"2024-04-02T13:24:55.574105+0000","last_fullsized":"2024-04-02T13:24:55.574105+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:51.244520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056346+0000","last_change":"2024-04-02T13:24:54.572495+0000","last_active":"2024-04-02T13:24:56.056346+0000","last_peered":"2024-04-02T13:24:56.056346+0000","last_clean":"2024-04-02T13:24:56.056346+0000","last_became_active":"2024-04-02T13:24:54.572181+0000","last_became_peered":"2024-04-02T13:24:54.572181+0000","last_unstale":"2024-04-02T13:24:56.056346+0000","last_undegraded":"2024-04-02T13:24:56.056346+0000","last_fullsized":"2024-04-02T13:24:56.056346+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:34:15.079374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"78'3","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064694+0000","last_change":"2024-04-02T13:24:53.577054+0000","last_active":"2024-04-02T13:24:56.064694+0000","last_peered":"2024-04-02T13:24:56.064694+0000","last_clean":"2024-04-02T13:24:56.064694+0000","last_became_active":"2024-04-02T13:24:53.575955+0000","last_became_peered":"2024-04-02T13:24:53.575955+0000","last_unstale":"2024-04-02T13:24:56.064694+0000","last_undegraded":"2024-04-02T13:24:56.064694+0000","last_fullsized":"2024-04-02T13:24:56.064694+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:16:43.448093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574776+0000","last_change":"2024-04-02T13:24:54.572847+0000","last_active":"2024-04-02T13:24:55.574776+0000","last_peered":"2024-04-02T13:24:55.574776+0000","last_clean":"2024-04-02T13:24:55.574776+0000","last_became_active":"2024-04-02T13:24:54.572588+0000","last_became_peered":"2024-04-02T13:24:54.572588+0000","last_unstale":"2024-04-02T13:24:55.574776+0000","last_undegraded":"2024-04-02T13:24:55.574776+0000","last_fullsized":"2024-04-02T13:24:55.574776+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:09:55.531393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"78'6","reported_seq":20,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064310+0000","last_change":"2024-04-02T13:24:53.575781+0000","last_active":"2024-04-02T13:24:56.064310+0000","last_peered":"2024-04-02T13:24:56.064310+0000","last_clean":"2024-04-02T13:24:56.064310+0000","last_became_active":"2024-04-02T13:24:53.574619+0000","last_became_peered":"2024-04-02T13:24:53.574619+0000","last_unstale":"2024-04-02T13:24:56.064310+0000","last_undegraded":"2024-04-02T13:24:56.064310+0000","last_fullsized":"2024-04-02T13:24:56.064310+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:29:57.018246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574715+0000","last_change":"2024-04-02T13:24:54.568437+0000","last_active":"2024-04-02T13:24:55.574715+0000","last_peered":"2024-04-02T13:24:55.574715+0000","last_clean":"2024-04-02T13:24:55.574715+0000","last_became_active":"2024-04-02T13:24:54.568101+0000","last_became_peered":"2024-04-02T13:24:54.568101+0000","last_unstale":"2024-04-02T13:24:55.574715+0000","last_undegraded":"2024-04-02T13:24:55.574715+0000","last_fullsized":"2024-04-02T13:24:55.574715+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:07:26.224860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573889+0000","last_change":"2024-04-02T13:24:53.572426+0000","last_active":"2024-04-02T13:24:55.573889+0000","last_peered":"2024-04-02T13:24:55.573889+0000","last_clean":"2024-04-02T13:24:55.573889+0000","last_became_active":"2024-04-02T13:24:53.572153+0000","last_became_peered":"2024-04-02T13:24:53.572153+0000","last_unstale":"2024-04-02T13:24:55.573889+0000","last_undegraded":"2024-04-02T13:24:55.573889+0000","last_fullsized":"2024-04-02T13:24:55.573889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:57:16.242492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064499+0000","last_change":"2024-04-02T13:24:54.578234+0000","last_active":"2024-04-02T13:24:56.064499+0000","last_peered":"2024-04-02T13:24:56.064499+0000","last_clean":"2024-04-02T13:24:56.064499+0000","last_became_active":"2024-04-02T13:24:54.577914+0000","last_became_peered":"2024-04-02T13:24:54.577914+0000","last_unstale":"2024-04-02T13:24:56.064499+0000","last_undegraded":"2024-04-02T13:24:56.064499+0000","last_fullsized":"2024-04-02T13:24:56.064499+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:52.246844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064325+0000","last_change":"2024-04-02T13:24:53.580278+0000","last_active":"2024-04-02T13:24:56.064325+0000","last_peered":"2024-04-02T13:24:56.064325+0000","last_clean":"2024-04-02T13:24:56.064325+0000","last_became_active":"2024-04-02T13:24:53.580002+0000","last_became_peered":"2024-04-02T13:24:53.580002+0000","last_unstale":"2024-04-02T13:24:56.064325+0000","last_undegraded":"2024-04-02T13:24:56.064325+0000","last_fullsized":"2024-04-02T13:24:56.064325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:24.786741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064483+0000","last_change":"2024-04-02T13:24:54.579646+0000","last_active":"2024-04-02T13:24:56.064483+0000","last_peered":"2024-04-02T13:24:56.064483+0000","last_clean":"2024-04-02T13:24:56.064483+0000","last_became_active":"2024-04-02T13:24:54.579360+0000","last_became_peered":"2024-04-02T13:24:54.579360+0000","last_unstale":"2024-04-02T13:24:56.064483+0000","last_undegraded":"2024-04-02T13:24:56.064483+0000","last_fullsized":"2024-04-02T13:24:56.064483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:39:31.964323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063999+0000","last_change":"2024-04-02T13:24:53.576553+0000","last_active":"2024-04-02T13:24:56.063999+0000","last_peered":"2024-04-02T13:24:56.063999+0000","last_clean":"2024-04-02T13:24:56.063999+0000","last_became_active":"2024-04-02T13:24:53.576219+0000","last_became_peered":"2024-04-02T13:24:53.576219+0000","last_unstale":"2024-04-02T13:24:56.063999+0000","last_undegraded":"2024-04-02T13:24:56.063999+0000","last_fullsized":"2024-04-02T13:24:56.063999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:12:36.353661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064755+0000","last_change":"2024-04-02T13:24:54.584841+0000","last_active":"2024-04-02T13:24:56.064755+0000","last_peered":"2024-04-02T13:24:56.064755+0000","last_clean":"2024-04-02T13:24:56.064755+0000","last_became_active":"2024-04-02T13:24:54.584565+0000","last_became_peered":"2024-04-02T13:24:54.584565+0000","last_unstale":"2024-04-02T13:24:56.064755+0000","last_undegraded":"2024-04-02T13:24:56.064755+0000","last_fullsized":"2024-04-02T13:24:56.064755+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:23:43.361370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"78'8","reported_seq":22,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063758+0000","last_change":"2024-04-02T13:24:53.575318+0000","last_active":"2024-04-02T13:24:56.063758+0000","last_peered":"2024-04-02T13:24:56.063758+0000","last_clean":"2024-04-02T13:24:56.063758+0000","last_became_active":"2024-04-02T13:24:53.575019+0000","last_became_peered":"2024-04-02T13:24:53.575019+0000","last_unstale":"2024-04-02T13:24:56.063758+0000","last_undegraded":"2024-04-02T13:24:56.063758+0000","last_fullsized":"2024-04-02T13:24:56.063758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:06.592645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.055822+0000","last_change":"2024-04-02T13:24:54.587605+0000","last_active":"2024-04-02T13:24:56.055822+0000","last_peered":"2024-04-02T13:24:56.055822+0000","last_clean":"2024-04-02T13:24:56.055822+0000","last_became_active":"2024-04-02T13:24:54.587310+0000","last_became_peered":"2024-04-02T13:24:54.587310+0000","last_unstale":"2024-04-02T13:24:56.055822+0000","last_undegraded":"2024-04-02T13:24:56.055822+0000","last_fullsized":"2024-04-02T13:24:56.055822+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:04:44.744456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064915+0000","last_change":"2024-04-02T13:24:53.577390+0000","last_active":"2024-04-02T13:24:56.064915+0000","last_peered":"2024-04-02T13:24:56.064915+0000","last_clean":"2024-04-02T13:24:56.064915+0000","last_became_active":"2024-04-02T13:24:53.577079+0000","last_became_peered":"2024-04-02T13:24:53.577079+0000","last_unstale":"2024-04-02T13:24:56.064915+0000","last_undegraded":"2024-04-02T13:24:56.064915+0000","last_fullsized":"2024-04-02T13:24:56.064915+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:07:25.222536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064186+0000","last_change":"2024-04-02T13:24:54.579775+0000","last_active":"2024-04-02T13:24:56.064186+0000","last_peered":"2024-04-02T13:24:56.064186+0000","last_clean":"2024-04-02T13:24:56.064186+0000","last_became_active":"2024-04-02T13:24:54.579459+0000","last_became_peered":"2024-04-02T13:24:54.579459+0000","last_unstale":"2024-04-02T13:24:56.064186+0000","last_undegraded":"2024-04-02T13:24:56.064186+0000","last_fullsized":"2024-04-02T13:24:56.064186+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:03:48.945748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574863+0000","last_change":"2024-04-02T13:24:53.577294+0000","last_active":"2024-04-02T13:24:55.574863+0000","last_peered":"2024-04-02T13:24:55.574863+0000","last_clean":"2024-04-02T13:24:55.574863+0000","last_became_active":"2024-04-02T13:24:53.577018+0000","last_became_peered":"2024-04-02T13:24:53.577018+0000","last_unstale":"2024-04-02T13:24:55.574863+0000","last_undegraded":"2024-04-02T13:24:55.574863+0000","last_fullsized":"2024-04-02T13:24:55.574863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:18:07.894919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064584+0000","last_change":"2024-04-02T13:24:54.581345+0000","last_active":"2024-04-02T13:24:56.064584+0000","last_peered":"2024-04-02T13:24:56.064584+0000","last_clean":"2024-04-02T13:24:56.064584+0000","last_became_active":"2024-04-02T13:24:54.581091+0000","last_became_peered":"2024-04-02T13:24:54.581091+0000","last_unstale":"2024-04-02T13:24:56.064584+0000","last_undegraded":"2024-04-02T13:24:56.064584+0000","last_fullsized":"2024-04-02T13:24:56.064584+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:09:04.946124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064588+0000","last_change":"2024-04-02T13:24:53.573420+0000","last_active":"2024-04-02T13:24:56.064588+0000","last_peered":"2024-04-02T13:24:56.064588+0000","last_clean":"2024-04-02T13:24:56.064588+0000","last_became_active":"2024-04-02T13:24:53.571909+0000","last_became_peered":"2024-04-02T13:24:53.571909+0000","last_unstale":"2024-04-02T13:24:56.064588+0000","last_undegraded":"2024-04-02T13:24:56.064588+0000","last_fullsized":"2024-04-02T13:24:56.064588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:08:06.577978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056130+0000","last_change":"2024-04-02T13:24:54.586519+0000","last_active":"2024-04-02T13:24:56.056130+0000","last_peered":"2024-04-02T13:24:56.056130+0000","last_clean":"2024-04-02T13:24:56.056130+0000","last_became_active":"2024-04-02T13:24:54.584751+0000","last_became_peered":"2024-04-02T13:24:54.584751+0000","last_unstale":"2024-04-02T13:24:56.056130+0000","last_undegraded":"2024-04-02T13:24:56.056130+0000","last_fullsized":"2024-04-02T13:24:56.056130+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:25.219021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574737+0000","last_change":"2024-04-02T13:24:53.570594+0000","last_active":"2024-04-02T13:24:55.574737+0000","last_peered":"2024-04-02T13:24:55.574737+0000","last_clean":"2024-04-02T13:24:55.574737+0000","last_became_active":"2024-04-02T13:24:53.569678+0000","last_became_peered":"2024-04-02T13:24:53.569678+0000","last_unstale":"2024-04-02T13:24:55.574737+0000","last_undegraded":"2024-04-02T13:24:55.574737+0000","last_fullsized":"2024-04-02T13:24:55.574737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:02:30.957187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064778+0000","last_change":"2024-04-02T13:24:54.575816+0000","last_active":"2024-04-02T13:24:56.064778+0000","last_peered":"2024-04-02T13:24:56.064778+0000","last_clean":"2024-04-02T13:24:56.064778+0000","last_became_active":"2024-04-02T13:24:54.572762+0000","last_became_peered":"2024-04-02T13:24:54.572762+0000","last_unstale":"2024-04-02T13:24:56.064778+0000","last_undegraded":"2024-04-02T13:24:56.064778+0000","last_fullsized":"2024-04-02T13:24:56.064778+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:34:15.079374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056135+0000","last_change":"2024-04-02T13:24:53.571104+0000","last_active":"2024-04-02T13:24:56.056135+0000","last_peered":"2024-04-02T13:24:56.056135+0000","last_clean":"2024-04-02T13:24:56.056135+0000","last_became_active":"2024-04-02T13:24:53.570758+0000","last_became_peered":"2024-04-02T13:24:53.570758+0000","last_unstale":"2024-04-02T13:24:56.056135+0000","last_undegraded":"2024-04-02T13:24:56.056135+0000","last_fullsized":"2024-04-02T13:24:56.056135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:12:29.693987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064481+0000","last_change":"2024-04-02T13:24:54.585045+0000","last_active":"2024-04-02T13:24:56.064481+0000","last_peered":"2024-04-02T13:24:56.064481+0000","last_clean":"2024-04-02T13:24:56.064481+0000","last_became_active":"2024-04-02T13:24:54.584797+0000","last_became_peered":"2024-04-02T13:24:54.584797+0000","last_unstale":"2024-04-02T13:24:56.064481+0000","last_undegraded":"2024-04-02T13:24:56.064481+0000","last_fullsized":"2024-04-02T13:24:56.064481+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:25.219021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.055901+0000","last_change":"2024-04-02T13:24:53.573510+0000","last_active":"2024-04-02T13:24:56.055901+0000","last_peered":"2024-04-02T13:24:56.055901+0000","last_clean":"2024-04-02T13:24:56.055901+0000","last_became_active":"2024-04-02T13:24:53.572671+0000","last_became_peered":"2024-04-02T13:24:53.572671+0000","last_unstale":"2024-04-02T13:24:56.055901+0000","last_undegraded":"2024-04-02T13:24:56.055901+0000","last_fullsized":"2024-04-02T13:24:56.055901+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:36.056546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064931+0000","last_change":"2024-04-02T13:24:54.578914+0000","last_active":"2024-04-02T13:24:56.064931+0000","last_peered":"2024-04-02T13:24:56.064931+0000","last_clean":"2024-04-02T13:24:56.064931+0000","last_became_active":"2024-04-02T13:24:54.578531+0000","last_became_peered":"2024-04-02T13:24:54.578531+0000","last_unstale":"2024-04-02T13:24:56.064931+0000","last_undegraded":"2024-04-02T13:24:56.064931+0000","last_fullsized":"2024-04-02T13:24:56.064931+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:50.669792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.055891+0000","last_change":"2024-04-02T13:24:53.574067+0000","last_active":"2024-04-02T13:24:56.055891+0000","last_peered":"2024-04-02T13:24:56.055891+0000","last_clean":"2024-04-02T13:24:56.055891+0000","last_became_active":"2024-04-02T13:24:53.573799+0000","last_became_peered":"2024-04-02T13:24:53.573799+0000","last_unstale":"2024-04-02T13:24:56.055891+0000","last_undegraded":"2024-04-02T13:24:56.055891+0000","last_fullsized":"2024-04-02T13:24:56.055891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:30:06.264873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574142+0000","last_change":"2024-04-02T13:24:54.579259+0000","last_active":"2024-04-02T13:24:55.574142+0000","last_peered":"2024-04-02T13:24:55.574142+0000","last_clean":"2024-04-02T13:24:55.574142+0000","last_became_active":"2024-04-02T13:24:54.578957+0000","last_became_peered":"2024-04-02T13:24:54.578957+0000","last_unstale":"2024-04-02T13:24:55.574142+0000","last_undegraded":"2024-04-02T13:24:55.574142+0000","last_fullsized":"2024-04-02T13:24:55.574142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:02:31.959511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573976+0000","last_change":"2024-04-02T13:24:53.575531+0000","last_active":"2024-04-02T13:24:55.573976+0000","last_peered":"2024-04-02T13:24:55.573976+0000","last_clean":"2024-04-02T13:24:55.573976+0000","last_became_active":"2024-04-02T13:24:53.575222+0000","last_became_peered":"2024-04-02T13:24:53.575222+0000","last_unstale":"2024-04-02T13:24:55.573976+0000","last_undegraded":"2024-04-02T13:24:55.573976+0000","last_fullsized":"2024-04-02T13:24:55.573976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:39:30.962000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064095+0000","last_change":"2024-04-02T13:24:54.575971+0000","last_active":"2024-04-02T13:24:56.064095+0000","last_peered":"2024-04-02T13:24:56.064095+0000","last_clean":"2024-04-02T13:24:56.064095+0000","last_became_active":"2024-04-02T13:24:54.575707+0000","last_became_peered":"2024-04-02T13:24:54.575707+0000","last_unstale":"2024-04-02T13:24:56.064095+0000","last_undegraded":"2024-04-02T13:24:56.064095+0000","last_fullsized":"2024-04-02T13:24:56.064095+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:08.394824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"79'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064634+0000","last_change":"2024-04-02T13:24:53.577485+0000","last_active":"2024-04-02T13:24:56.064634+0000","last_peered":"2024-04-02T13:24:56.064634+0000","last_clean":"2024-04-02T13:24:56.064634+0000","last_became_active":"2024-04-02T13:24:53.575330+0000","last_became_peered":"2024-04-02T13:24:53.575330+0000","last_unstale":"2024-04-02T13:24:56.064634+0000","last_undegraded":"2024-04-02T13:24:56.064634+0000","last_fullsized":"2024-04-02T13:24:56.064634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:49.667468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574799+0000","last_change":"2024-04-02T13:24:54.576490+0000","last_active":"2024-04-02T13:24:55.574799+0000","last_peered":"2024-04-02T13:24:55.574799+0000","last_clean":"2024-04-02T13:24:55.574799+0000","last_became_active":"2024-04-02T13:24:54.576224+0000","last_became_peered":"2024-04-02T13:24:54.576224+0000","last_unstale":"2024-04-02T13:24:55.574799+0000","last_undegraded":"2024-04-02T13:24:55.574799+0000","last_fullsized":"2024-04-02T13:24:55.574799+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:15:49.793401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064732+0000","last_change":"2024-04-02T13:24:53.572963+0000","last_active":"2024-04-02T13:24:56.064732+0000","last_peered":"2024-04-02T13:24:56.064732+0000","last_clean":"2024-04-02T13:24:56.064732+0000","last_became_active":"2024-04-02T13:24:53.572536+0000","last_became_peered":"2024-04-02T13:24:53.572536+0000","last_unstale":"2024-04-02T13:24:56.064732+0000","last_undegraded":"2024-04-02T13:24:56.064732+0000","last_fullsized":"2024-04-02T13:24:56.064732+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:43:42.127877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064578+0000","last_change":"2024-04-02T13:24:54.574356+0000","last_active":"2024-04-02T13:24:56.064578+0000","last_peered":"2024-04-02T13:24:56.064578+0000","last_clean":"2024-04-02T13:24:56.064578+0000","last_became_active":"2024-04-02T13:24:54.573985+0000","last_became_peered":"2024-04-02T13:24:54.573985+0000","last_unstale":"2024-04-02T13:24:56.064578+0000","last_undegraded":"2024-04-02T13:24:56.064578+0000","last_fullsized":"2024-04-02T13:24:56.064578+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:08.440154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064559+0000","last_change":"2024-04-02T13:24:53.580007+0000","last_active":"2024-04-02T13:24:56.064559+0000","last_peered":"2024-04-02T13:24:56.064559+0000","last_clean":"2024-04-02T13:24:56.064559+0000","last_became_active":"2024-04-02T13:24:53.579686+0000","last_became_peered":"2024-04-02T13:24:53.579686+0000","last_unstale":"2024-04-02T13:24:56.064559+0000","last_undegraded":"2024-04-02T13:24:56.064559+0000","last_fullsized":"2024-04-02T13:24:56.064559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:11.652383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064232+0000","last_change":"2024-04-02T13:24:54.585002+0000","last_active":"2024-04-02T13:24:56.064232+0000","last_peered":"2024-04-02T13:24:56.064232+0000","last_clean":"2024-04-02T13:24:56.064232+0000","last_became_active":"2024-04-02T13:24:54.584620+0000","last_became_peered":"2024-04-02T13:24:54.584620+0000","last_unstale":"2024-04-02T13:24:56.064232+0000","last_undegraded":"2024-04-02T13:24:56.064232+0000","last_fullsized":"2024-04-02T13:24:56.064232+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:57:17.244816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064180+0000","last_change":"2024-04-02T13:24:53.572298+0000","last_active":"2024-04-02T13:24:56.064180+0000","last_peered":"2024-04-02T13:24:56.064180+0000","last_clean":"2024-04-02T13:24:56.064180+0000","last_became_active":"2024-04-02T13:24:53.570380+0000","last_became_peered":"2024-04-02T13:24:53.570380+0000","last_unstale":"2024-04-02T13:24:56.064180+0000","last_undegraded":"2024-04-02T13:24:56.064180+0000","last_fullsized":"2024-04-02T13:24:56.064180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:43:26.495241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064671+0000","last_change":"2024-04-02T13:24:54.573352+0000","last_active":"2024-04-02T13:24:56.064671+0000","last_peered":"2024-04-02T13:24:56.064671+0000","last_clean":"2024-04-02T13:24:56.064671+0000","last_became_active":"2024-04-02T13:24:54.573039+0000","last_became_peered":"2024-04-02T13:24:54.573039+0000","last_unstale":"2024-04-02T13:24:56.064671+0000","last_undegraded":"2024-04-02T13:24:56.064671+0000","last_fullsized":"2024-04-02T13:24:56.064671+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:46:34.338949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064247+0000","last_change":"2024-04-02T13:24:53.572484+0000","last_active":"2024-04-02T13:24:56.064247+0000","last_peered":"2024-04-02T13:24:56.064247+0000","last_clean":"2024-04-02T13:24:56.064247+0000","last_became_active":"2024-04-02T13:24:53.572124+0000","last_became_peered":"2024-04-02T13:24:53.572124+0000","last_unstale":"2024-04-02T13:24:56.064247+0000","last_undegraded":"2024-04-02T13:24:56.064247+0000","last_fullsized":"2024-04-02T13:24:56.064247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:39:18.965555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574852+0000","last_change":"2024-04-02T13:24:54.573678+0000","last_active":"2024-04-02T13:24:55.574852+0000","last_peered":"2024-04-02T13:24:55.574852+0000","last_clean":"2024-04-02T13:24:55.574852+0000","last_became_active":"2024-04-02T13:24:54.573380+0000","last_became_peered":"2024-04-02T13:24:54.573380+0000","last_unstale":"2024-04-02T13:24:55.574852+0000","last_undegraded":"2024-04-02T13:24:55.574852+0000","last_fullsized":"2024-04-02T13:24:55.574852+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:48:44.079142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064587+0000","last_change":"2024-04-02T13:24:53.574193+0000","last_active":"2024-04-02T13:24:56.064587+0000","last_peered":"2024-04-02T13:24:56.064587+0000","last_clean":"2024-04-02T13:24:56.064587+0000","last_became_active":"2024-04-02T13:24:53.570565+0000","last_became_peered":"2024-04-02T13:24:53.570565+0000","last_unstale":"2024-04-02T13:24:56.064587+0000","last_undegraded":"2024-04-02T13:24:56.064587+0000","last_fullsized":"2024-04-02T13:24:56.064587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:50:27.793063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574928+0000","last_change":"2024-04-02T13:24:54.577947+0000","last_active":"2024-04-02T13:24:55.574928+0000","last_peered":"2024-04-02T13:24:55.574928+0000","last_clean":"2024-04-02T13:24:55.574928+0000","last_became_active":"2024-04-02T13:24:54.577637+0000","last_became_peered":"2024-04-02T13:24:54.577637+0000","last_unstale":"2024-04-02T13:24:55.574928+0000","last_undegraded":"2024-04-02T13:24:55.574928+0000","last_fullsized":"2024-04-02T13:24:55.574928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:08.394824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574508+0000","last_change":"2024-04-02T13:24:53.567583+0000","last_active":"2024-04-02T13:24:55.574508+0000","last_peered":"2024-04-02T13:24:55.574508+0000","last_clean":"2024-04-02T13:24:55.574508+0000","last_became_active":"2024-04-02T13:24:53.567289+0000","last_became_peered":"2024-04-02T13:24:53.567289+0000","last_unstale":"2024-04-02T13:24:55.574508+0000","last_undegraded":"2024-04-02T13:24:55.574508+0000","last_fullsized":"2024-04-02T13:24:55.574508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:29:00.076440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064704+0000","last_change":"2024-04-02T13:24:54.578014+0000","last_active":"2024-04-02T13:24:56.064704+0000","last_peered":"2024-04-02T13:24:56.064704+0000","last_clean":"2024-04-02T13:24:56.064704+0000","last_became_active":"2024-04-02T13:24:54.577445+0000","last_became_peered":"2024-04-02T13:24:54.577445+0000","last_unstale":"2024-04-02T13:24:56.064704+0000","last_undegraded":"2024-04-02T13:24:56.064704+0000","last_fullsized":"2024-04-02T13:24:56.064704+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:30:34.688997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056408+0000","last_change":"2024-04-02T13:24:53.572298+0000","last_active":"2024-04-02T13:24:56.056408+0000","last_peered":"2024-04-02T13:24:56.056408+0000","last_clean":"2024-04-02T13:24:56.056408+0000","last_became_active":"2024-04-02T13:24:53.572035+0000","last_became_peered":"2024-04-02T13:24:53.572035+0000","last_unstale":"2024-04-02T13:24:56.056408+0000","last_undegraded":"2024-04-02T13:24:56.056408+0000","last_fullsized":"2024-04-02T13:24:56.056408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:35.859707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":282,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064569+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:24:56.064569+0000","last_peered":"2024-04-02T13:24:56.064569+0000","last_clean":"2024-04-02T13:24:56.064569+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:24:56.064569+0000","last_undegraded":"2024-04-02T13:24:56.064569+0000","last_fullsized":"2024-04-02T13:24:56.064569+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574908,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":1031999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":978944,"data_stored":443221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574908,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":441643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574908,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":987136,"data_stored":443325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":442191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":978944,"data_stored":442351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":974848,"data_stored":441721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574908,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1560576,"data_stored":1031977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:24:59.308 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-02T13:24:59.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.441+0000 7fd330396640 1 Processor -- start 2024-04-02T13:24:59.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.441+0000 7fd330396640 1 -- start start 2024-04-02T13:24:59.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.442+0000 7fd330396640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd3280d15f0 0x7fd3280b2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.442+0000 7fd330396640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd3280cefc0 0x7fd3280d2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.442+0000 7fd330396640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3280b1dc0 con 0x7fd3280d1e00 2024-04-02T13:24:59.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.442+0000 7fd330396640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd3280b26d0 con 0x7fd3280cefc0 2024-04-02T13:24:59.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.442+0000 7fd330396640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd3280cf600 con 0x7fd3280d15f0 2024-04-02T13:24:59.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.442+0000 7fd32e933640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd3280cefc0 0x7fd3280d2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.442+0000 7fd32e933640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd3280cefc0 0x7fd3280d2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41892/0 (socket says 172.21.15.44:41892) 2024-04-02T13:24:59.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.442+0000 7fd32e933640 1 -- 172.21.15.44:0/1493309327 learned_addr learned my addr 172.21.15.44:0/1493309327 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:59.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.442+0000 7fd32e132640 1 --2- 172.21.15.44:0/1493309327 >> v2:172.21.15.69:3300/0 conn(0x7fd3280d15f0 0x7fd3280b2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.443+0000 7fd32e933640 1 -- 172.21.15.44:0/1493309327 >> v2:172.21.15.69:3300/0 conn(0x7fd3280d15f0 msgr2=0x7fd3280b2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:59.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.443+0000 7fd32e933640 1 --2- 172.21.15.44:0/1493309327 >> v2:172.21.15.69:3300/0 conn(0x7fd3280d15f0 0x7fd3280b2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.443+0000 7fd32e933640 1 -- 172.21.15.44:0/1493309327 >> v1:172.21.15.69:6789/0 conn(0x7fd3280d1e00 legacy=0x7fd3280ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.443+0000 7fd32e933640 1 -- 172.21.15.44:0/1493309327 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3280b2080 con 0x7fd3280cefc0 2024-04-02T13:24:59.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.443+0000 7fd32e933640 1 --2- 172.21.15.44:0/1493309327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3280cefc0 0x7fd3280d2280 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd3280cf930 tx=0x7fd31802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e089608f34c085 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:59.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.443+0000 7fd32d130640 1 -- 172.21.15.44:0/1493309327 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31802cb20 con 0x7fd3280cefc0 2024-04-02T13:24:59.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.443+0000 7fd32d130640 1 -- 172.21.15.44:0/1493309327 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd31802ccc0 con 0x7fd3280cefc0 2024-04-02T13:24:59.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.443+0000 7fd32d130640 1 -- 172.21.15.44:0/1493309327 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd318034610 con 0x7fd3280cefc0 2024-04-02T13:24:59.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.444+0000 7fd330396640 1 -- 172.21.15.44:0/1493309327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3280cefc0 msgr2=0x7fd3280d2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.444+0000 7fd330396640 1 --2- 172.21.15.44:0/1493309327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3280cefc0 0x7fd3280d2280 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd3280cf930 tx=0x7fd31802bf80 comp rx=0 tx=0).stop 2024-04-02T13:24:59.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.444+0000 7fd330396640 1 -- 172.21.15.44:0/1493309327 shutdown_connections 2024-04-02T13:24:59.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.444+0000 7fd330396640 1 --2- 172.21.15.44:0/1493309327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3280cefc0 0x7fd3280d2280 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.444+0000 7fd330396640 1 --2- 172.21.15.44:0/1493309327 >> v2:172.21.15.69:3300/0 conn(0x7fd3280d15f0 0x7fd3280b2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.444+0000 7fd330396640 1 -- 172.21.15.44:0/1493309327 >> 172.21.15.44:0/1493309327 conn(0x7fd32809aa20 msgr2=0x7fd32809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:59.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.444+0000 7fd330396640 1 -- 172.21.15.44:0/1493309327 shutdown_connections 2024-04-02T13:24:59.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.444+0000 7fd330396640 1 -- 172.21.15.44:0/1493309327 wait complete. 2024-04-02T13:24:59.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.445+0000 7fd330396640 1 Processor -- start 2024-04-02T13:24:59.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.445+0000 7fd330396640 1 -- start start 2024-04-02T13:24:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd330396640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3280cefc0 0x7fd3280cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd330396640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3280d15f0 0x7fd3281f7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd330396640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3280d1e00 0x7fd32815d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd330396640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd3280b30a0 con 0x7fd3280d15f0 2024-04-02T13:24:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd330396640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd3280d19d0 con 0x7fd3280cefc0 2024-04-02T13:24:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd330396640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd3280b3220 con 0x7fd3280d1e00 2024-04-02T13:24:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd32d931640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3280d15f0 0x7fd3281f7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd32d931640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3280d15f0 0x7fd3281f7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41904/0 (socket says 172.21.15.44:41904) 2024-04-02T13:24:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd32d931640 1 -- 172.21.15.44:0/79658153 learned_addr learned my addr 172.21.15.44:0/79658153 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd32e132640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3280cefc0 0x7fd3280cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd32e933640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3280d1e00 0x7fd32815d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd32d931640 1 -- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3280d1e00 msgr2=0x7fd32815d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd32d931640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3280d1e00 0x7fd32815d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd32d931640 1 -- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3280cefc0 msgr2=0x7fd3280cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd32d931640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3280cefc0 0x7fd3280cb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.446+0000 7fd32d931640 1 -- 172.21.15.44:0/79658153 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3280cbf20 con 0x7fd3280d15f0 2024-04-02T13:24:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.447+0000 7fd32d931640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3280d15f0 0x7fd3281f7e40 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00c430 tx=0x7fd31c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.447+0000 7fd326ffd640 1 -- 172.21.15.44:0/79658153 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c0052d0 con 0x7fd3280d15f0 2024-04-02T13:24:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.447+0000 7fd326ffd640 1 -- 172.21.15.44:0/79658153 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd31c005490 con 0x7fd3280d15f0 2024-04-02T13:24:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.447+0000 7fd330396640 1 -- 172.21.15.44:0/79658153 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3280c95e0 con 0x7fd3280d15f0 2024-04-02T13:24:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.447+0000 7fd326ffd640 1 -- 172.21.15.44:0/79658153 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c007260 con 0x7fd3280d15f0 2024-04-02T13:24:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.447+0000 7fd330396640 1 -- 172.21.15.44:0/79658153 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3280c6ff0 con 0x7fd3280d15f0 2024-04-02T13:24:59.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.448+0000 7fd330396640 1 -- 172.21.15.44:0/79658153 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3280684f0 con 0x7fd3280d15f0 2024-04-02T13:24:59.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.448+0000 7fd326ffd640 1 -- 172.21.15.44:0/79658153 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd31c007400 con 0x7fd3280d15f0 2024-04-02T13:24:59.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.448+0000 7fd326ffd640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd31406c8f0 0x7fd31406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.449+0000 7fd326ffd640 1 -- 172.21.15.44:0/79658153 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fd31c08b0d0 con 0x7fd3280d15f0 2024-04-02T13:24:59.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.449+0000 7fd32e132640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd31406c8f0 0x7fd31406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.449+0000 7fd32e132640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd31406c8f0 0x7fd31406edb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd32806f3e0 tx=0x7fd320009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:59.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.453+0000 7fd326ffd640 1 -- 172.21.15.44:0/79658153 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd31c058db0 con 0x7fd3280d15f0 2024-04-02T13:24:59.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.709+0000 7fd330396640 1 -- 172.21.15.44:0/79658153 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fd3280b18d0 con 0x7fd3280d15f0 2024-04-02T13:24:59.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.710+0000 7fd326ffd640 1 -- 172.21.15.44:0/79658153 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1013 (secure 0 0 0) 0x7fd31c05d0b0 con 0x7fd3280d15f0 2024-04-02T13:24:59.714 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:24:59.714 INFO:teuthology.orchestra.run.smithi044.stdout:{"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":297,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":79,"num_osds":8,"num_up_osds":8,"osd_up_since":1712064013,"num_in_osds":8,"osd_in_since":1712064005,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":247398400,"bytes_avail":772846714880,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":44,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"d","status":"up:active","gid":7002}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-04-02T13:24:30.935521+0000","services":{}},"progress_events":{}} 2024-04-02T13:24:59.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.712+0000 7fd330396640 1 -- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd31406c8f0 msgr2=0x7fd31406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.712+0000 7fd330396640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd31406c8f0 0x7fd31406edb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd32806f3e0 tx=0x7fd320009190 comp rx=0 tx=0).stop 2024-04-02T13:24:59.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.712+0000 7fd330396640 1 -- 172.21.15.44:0/79658153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3280d15f0 msgr2=0x7fd3281f7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.712+0000 7fd330396640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3280d15f0 0x7fd3281f7e40 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00c430 tx=0x7fd31c00c8f0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.712+0000 7fd330396640 1 -- 172.21.15.44:0/79658153 shutdown_connections 2024-04-02T13:24:59.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.712+0000 7fd330396640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd31406c8f0 0x7fd31406edb0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.712+0000 7fd330396640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3280d1e00 0x7fd32815d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.712+0000 7fd330396640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3280d15f0 0x7fd3281f7e40 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.712+0000 7fd330396640 1 --2- 172.21.15.44:0/79658153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3280cefc0 0x7fd3280cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.712+0000 7fd330396640 1 -- 172.21.15.44:0/79658153 >> 172.21.15.44:0/79658153 conn(0x7fd32809aa20 msgr2=0x7fd3280ce8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:59.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.712+0000 7fd330396640 1 -- 172.21.15.44:0/79658153 shutdown_connections 2024-04-02T13:24:59.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.712+0000 7fd330396640 1 -- 172.21.15.44:0/79658153 wait complete. 2024-04-02T13:24:59.726 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-02T13:24:59.726 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:24:59.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.824+0000 7f46301e9640 1 Processor -- start 2024-04-02T13:24:59.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.824+0000 7f46301e9640 1 -- start start 2024-04-02T13:24:59.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.825+0000 7f46301e9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f462815e4a0 0x7f46280aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.825+0000 7f46301e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46280d0c50 0x7f46280d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.825+0000 7f46301e9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f462806ce40 0x7f462806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.825+0000 7f46301e9640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f46280cc600 con 0x7f462815e4a0 2024-04-02T13:24:59.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.825+0000 7f46301e9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f46280cc8e0 con 0x7f462806ce40 2024-04-02T13:24:59.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.825+0000 7f46301e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f46280ccbc0 con 0x7f46280d0c50 2024-04-02T13:24:59.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.825+0000 7f462df85640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f462815e4a0 0x7f46280aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.825+0000 7f462df85640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f462815e4a0 0x7f46280aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41920/0 (socket says 172.21.15.44:41920) 2024-04-02T13:24:59.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.825+0000 7f462df85640 1 -- 172.21.15.44:0/1477139032 learned_addr learned my addr 172.21.15.44:0/1477139032 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:59.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.825+0000 7f462df85640 1 -- 172.21.15.44:0/1477139032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46280d0c50 msgr2=0x7f46280d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:59.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.825+0000 7f462e786640 1 --2- 172.21.15.44:0/1477139032 >> v2:172.21.15.69:3300/0 conn(0x7f462806ce40 0x7f462806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.825+0000 7f462d784640 1 --2- 172.21.15.44:0/1477139032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46280d0c50 0x7f46280d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.826+0000 7f462df85640 1 --2- 172.21.15.44:0/1477139032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46280d0c50 0x7f46280d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.826+0000 7f462df85640 1 -- 172.21.15.44:0/1477139032 >> v2:172.21.15.69:3300/0 conn(0x7f462806ce40 msgr2=0x7f462806d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:24:59.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.826+0000 7f462df85640 1 --2- 172.21.15.44:0/1477139032 >> v2:172.21.15.69:3300/0 conn(0x7f462806ce40 0x7f462806d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.826+0000 7f462df85640 1 -- 172.21.15.44:0/1477139032 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46280ccea0 con 0x7f462815e4a0 2024-04-02T13:24:59.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.826+0000 7f462df85640 1 --2- 172.21.15.44:0/1477139032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f462815e4a0 0x7f46280aebf0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f46280cfa10 tx=0x7f462002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12b7821150cf32c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:59.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.826+0000 7f462cf83640 1 -- 172.21.15.44:0/1477139032 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620035020 con 0x7f462815e4a0 2024-04-02T13:24:59.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.826+0000 7f462cf83640 1 -- 172.21.15.44:0/1477139032 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4620031870 con 0x7f462815e4a0 2024-04-02T13:24:59.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.826+0000 7f462cf83640 1 -- 172.21.15.44:0/1477139032 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620031c40 con 0x7f462815e4a0 2024-04-02T13:24:59.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.827+0000 7f46301e9640 1 -- 172.21.15.44:0/1477139032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f462815e4a0 msgr2=0x7f46280aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.827+0000 7f46301e9640 1 --2- 172.21.15.44:0/1477139032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f462815e4a0 0x7f46280aebf0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f46280cfa10 tx=0x7f462002b910 comp rx=0 tx=0).stop 2024-04-02T13:24:59.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.827+0000 7f46301e9640 1 -- 172.21.15.44:0/1477139032 shutdown_connections 2024-04-02T13:24:59.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.827+0000 7f46301e9640 1 --2- 172.21.15.44:0/1477139032 >> v2:172.21.15.69:3300/0 conn(0x7f462806ce40 0x7f462806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.827+0000 7f46301e9640 1 --2- 172.21.15.44:0/1477139032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46280d0c50 0x7f46280d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.827+0000 7f46301e9640 1 --2- 172.21.15.44:0/1477139032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f462815e4a0 0x7f46280aebf0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.827+0000 7f46301e9640 1 -- 172.21.15.44:0/1477139032 >> 172.21.15.44:0/1477139032 conn(0x7f462809aa20 msgr2=0x7f462809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:24:59.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.827+0000 7f46301e9640 1 -- 172.21.15.44:0/1477139032 shutdown_connections 2024-04-02T13:24:59.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.827+0000 7f46301e9640 1 -- 172.21.15.44:0/1477139032 wait complete. 2024-04-02T13:24:59.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.828+0000 7f46301e9640 1 Processor -- start 2024-04-02T13:24:59.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.828+0000 7f46301e9640 1 -- start start 2024-04-02T13:24:59.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f46301e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f462806ce40 0x7f46281f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f46301e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46280d0c50 0x7f46280cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f46301e9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f462815e4a0 0x7f4628161380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f46301e9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f462815e880 con 0x7f462815e4a0 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f46301e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f46280cd460 con 0x7f46280d0c50 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f46301e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f46280af130 con 0x7f462806ce40 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f462e786640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f462815e4a0 0x7f4628161380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f462e786640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f462815e4a0 0x7f4628161380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41936/0 (socket says 172.21.15.44:41936) 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f462e786640 1 -- 172.21.15.44:0/1337334680 learned_addr learned my addr 172.21.15.44:0/1337334680 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f462d784640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46280d0c50 0x7f46280cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f462df85640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f462806ce40 0x7f46281f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f462e786640 1 -- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f462806ce40 msgr2=0x7f46281f40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f462e786640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f462806ce40 0x7f46281f40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f462e786640 1 -- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46280d0c50 msgr2=0x7f46280cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f462e786640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46280d0c50 0x7f46280cac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:24:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f462e786640 1 -- 172.21.15.44:0/1337334680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46280c9b70 con 0x7f462815e4a0 2024-04-02T13:24:59.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f462df85640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f462806ce40 0x7f46281f40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:24:59.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.829+0000 7f462e786640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f462815e4a0 0x7f4628161380 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f461800cd70 tx=0x7f461800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:59.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.830+0000 7f46267fc640 1 -- 172.21.15.44:0/1337334680 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618003b90 con 0x7f462815e4a0 2024-04-02T13:24:59.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.830+0000 7f46267fc640 1 -- 172.21.15.44:0/1337334680 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4618003d30 con 0x7f462815e4a0 2024-04-02T13:24:59.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.830+0000 7f46301e9640 1 -- 172.21.15.44:0/1337334680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46280c9e50 con 0x7f462815e4a0 2024-04-02T13:24:59.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.830+0000 7f46267fc640 1 -- 172.21.15.44:0/1337334680 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618004240 con 0x7f462815e4a0 2024-04-02T13:24:59.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.830+0000 7f46301e9640 1 -- 172.21.15.44:0/1337334680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46280ca410 con 0x7f462815e4a0 2024-04-02T13:24:59.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.831+0000 7f46301e9640 1 -- 172.21.15.44:0/1337334680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46280684f0 con 0x7f462815e4a0 2024-04-02T13:24:59.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.831+0000 7f46267fc640 1 -- 172.21.15.44:0/1337334680 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f46180043e0 con 0x7f462815e4a0 2024-04-02T13:24:59.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.832+0000 7f46267fc640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f461406c8f0 0x7f461406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:24:59.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.832+0000 7f46267fc640 1 -- 172.21.15.44:0/1337334680 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f461808c780 con 0x7f462815e4a0 2024-04-02T13:24:59.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.832+0000 7f462df85640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f461406c8f0 0x7f461406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:24:59.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.832+0000 7f462df85640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f461406c8f0 0x7f461406edb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4620036000 tx=0x7f4620008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:24:59.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:24:59.836+0000 7f46267fc640 1 -- 172.21.15.44:0/1337334680 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f461801d340 con 0x7f462815e4a0 2024-04-02T13:25:00.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.018+0000 7f46301e9640 1 -- 172.21.15.44:0/1337334680 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f46280aec50 con 0x7f461406c8f0 2024-04-02T13:25:00.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.025+0000 7f46267fc640 1 -- 172.21.15.44:0/1337334680 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252097 (secure 0 0 0) 0x7f46280aec50 con 0x7f461406c8f0 2024-04-02T13:25:00.031 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:25:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.027+0000 7f46301e9640 1 -- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f461406c8f0 msgr2=0x7f461406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.027+0000 7f46301e9640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f461406c8f0 0x7f461406edb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4620036000 tx=0x7f4620008030 comp rx=0 tx=0).stop 2024-04-02T13:25:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.027+0000 7f46301e9640 1 -- 172.21.15.44:0/1337334680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f462815e4a0 msgr2=0x7f4628161380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.027+0000 7f46301e9640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f462815e4a0 0x7f4628161380 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f461800cd70 tx=0x7f461800a660 comp rx=0 tx=0).stop 2024-04-02T13:25:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.027+0000 7f46301e9640 1 -- 172.21.15.44:0/1337334680 shutdown_connections 2024-04-02T13:25:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.027+0000 7f46301e9640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f461406c8f0 0x7f461406edb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.027+0000 7f46301e9640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f462815e4a0 0x7f4628161380 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.027+0000 7f46301e9640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46280d0c50 0x7f46280cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.027+0000 7f46301e9640 1 --2- 172.21.15.44:0/1337334680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f462806ce40 0x7f46281f40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.027+0000 7f46301e9640 1 -- 172.21.15.44:0/1337334680 >> 172.21.15.44:0/1337334680 conn(0x7f462809aa20 msgr2=0x7f462809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.027+0000 7f46301e9640 1 -- 172.21.15.44:0/1337334680 shutdown_connections 2024-04-02T13:25:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:00.027+0000 7f46301e9640 1 -- 172.21.15.44:0/1337334680 wait complete. 2024-04-02T13:25:00.041 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-04-02T13:24:58.941614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241600,"kb_used_data":8776,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754733120,"statfs":{"total":773094113280,"available":772846714880,"internally_reserved":0,"allocated":8986624,"data_stored":4718428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001328"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064800+0000","last_change":"2024-04-02T13:24:54.582190+0000","last_active":"2024-04-02T13:24:56.064800+0000","last_peered":"2024-04-02T13:24:56.064800+0000","last_clean":"2024-04-02T13:24:56.064800+0000","last_became_active":"2024-04-02T13:24:54.581814+0000","last_became_peered":"2024-04-02T13:24:54.581814+0000","last_unstale":"2024-04-02T13:24:56.064800+0000","last_undegraded":"2024-04-02T13:24:56.064800+0000","last_fullsized":"2024-04-02T13:24:56.064800+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:14:01.535902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064308+0000","last_change":"2024-04-02T13:24:54.584115+0000","last_active":"2024-04-02T13:24:56.064308+0000","last_peered":"2024-04-02T13:24:56.064308+0000","last_clean":"2024-04-02T13:24:56.064308+0000","last_became_active":"2024-04-02T13:24:54.583792+0000","last_became_peered":"2024-04-02T13:24:54.583792+0000","last_unstale":"2024-04-02T13:24:56.064308+0000","last_undegraded":"2024-04-02T13:24:56.064308+0000","last_fullsized":"2024-04-02T13:24:56.064308+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:30:34.688997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574787+0000","last_change":"2024-04-02T13:24:54.574235+0000","last_active":"2024-04-02T13:24:55.574787+0000","last_peered":"2024-04-02T13:24:55.574787+0000","last_clean":"2024-04-02T13:24:55.574787+0000","last_became_active":"2024-04-02T13:24:54.573908+0000","last_became_peered":"2024-04-02T13:24:54.573908+0000","last_unstale":"2024-04-02T13:24:55.574787+0000","last_undegraded":"2024-04-02T13:24:55.574787+0000","last_fullsized":"2024-04-02T13:24:55.574787+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:18:21.848167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574407+0000","last_change":"2024-04-02T13:24:54.581284+0000","last_active":"2024-04-02T13:24:55.574407+0000","last_peered":"2024-04-02T13:24:55.574407+0000","last_clean":"2024-04-02T13:24:55.574407+0000","last_became_active":"2024-04-02T13:24:54.580976+0000","last_became_peered":"2024-04-02T13:24:54.580976+0000","last_unstale":"2024-04-02T13:24:55.574407+0000","last_undegraded":"2024-04-02T13:24:55.574407+0000","last_fullsized":"2024-04-02T13:24:55.574407+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:25.789065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064080+0000","last_change":"2024-04-02T13:24:54.580261+0000","last_active":"2024-04-02T13:24:56.064080+0000","last_peered":"2024-04-02T13:24:56.064080+0000","last_clean":"2024-04-02T13:24:56.064080+0000","last_became_active":"2024-04-02T13:24:54.579973+0000","last_became_peered":"2024-04-02T13:24:54.579973+0000","last_unstale":"2024-04-02T13:24:56.064080+0000","last_undegraded":"2024-04-02T13:24:56.064080+0000","last_fullsized":"2024-04-02T13:24:56.064080+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:43:00.164244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056185+0000","last_change":"2024-04-02T13:24:54.585758+0000","last_active":"2024-04-02T13:24:56.056185+0000","last_peered":"2024-04-02T13:24:56.056185+0000","last_clean":"2024-04-02T13:24:56.056185+0000","last_became_active":"2024-04-02T13:24:54.585490+0000","last_became_peered":"2024-04-02T13:24:54.585490+0000","last_unstale":"2024-04-02T13:24:56.056185+0000","last_undegraded":"2024-04-02T13:24:56.056185+0000","last_fullsized":"2024-04-02T13:24:56.056185+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:16.381648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064052+0000","last_change":"2024-04-02T13:24:54.577884+0000","last_active":"2024-04-02T13:24:56.064052+0000","last_peered":"2024-04-02T13:24:56.064052+0000","last_clean":"2024-04-02T13:24:56.064052+0000","last_became_active":"2024-04-02T13:24:54.577328+0000","last_became_peered":"2024-04-02T13:24:54.577328+0000","last_unstale":"2024-04-02T13:24:56.064052+0000","last_undegraded":"2024-04-02T13:24:56.064052+0000","last_fullsized":"2024-04-02T13:24:56.064052+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:48:44.079142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063748+0000","last_change":"2024-04-02T13:24:54.579426+0000","last_active":"2024-04-02T13:24:56.063748+0000","last_peered":"2024-04-02T13:24:56.063748+0000","last_clean":"2024-04-02T13:24:56.063748+0000","last_became_active":"2024-04-02T13:24:54.579147+0000","last_became_peered":"2024-04-02T13:24:54.579147+0000","last_unstale":"2024-04-02T13:24:56.063748+0000","last_undegraded":"2024-04-02T13:24:56.063748+0000","last_fullsized":"2024-04-02T13:24:56.063748+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:49:38.705972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574247+0000","last_change":"2024-04-02T13:24:54.587257+0000","last_active":"2024-04-02T13:24:55.574247+0000","last_peered":"2024-04-02T13:24:55.574247+0000","last_clean":"2024-04-02T13:24:55.574247+0000","last_became_active":"2024-04-02T13:24:54.586571+0000","last_became_peered":"2024-04-02T13:24:54.586571+0000","last_unstale":"2024-04-02T13:24:55.574247+0000","last_undegraded":"2024-04-02T13:24:55.574247+0000","last_fullsized":"2024-04-02T13:24:55.574247+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:43:27.497564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573950+0000","last_change":"2024-04-02T13:24:54.570536+0000","last_active":"2024-04-02T13:24:55.573950+0000","last_peered":"2024-04-02T13:24:55.573950+0000","last_clean":"2024-04-02T13:24:55.573950+0000","last_became_active":"2024-04-02T13:24:54.570264+0000","last_became_peered":"2024-04-02T13:24:54.570264+0000","last_unstale":"2024-04-02T13:24:55.573950+0000","last_undegraded":"2024-04-02T13:24:55.573950+0000","last_fullsized":"2024-04-02T13:24:55.573950+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:58:28.798683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064238+0000","last_change":"2024-04-02T13:24:54.584742+0000","last_active":"2024-04-02T13:24:56.064238+0000","last_peered":"2024-04-02T13:24:56.064238+0000","last_clean":"2024-04-02T13:24:56.064238+0000","last_became_active":"2024-04-02T13:24:54.584094+0000","last_became_peered":"2024-04-02T13:24:54.584094+0000","last_unstale":"2024-04-02T13:24:56.064238+0000","last_undegraded":"2024-04-02T13:24:56.064238+0000","last_fullsized":"2024-04-02T13:24:56.064238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:16.445368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574563+0000","last_change":"2024-04-02T13:24:54.574509+0000","last_active":"2024-04-02T13:24:55.574563+0000","last_peered":"2024-04-02T13:24:55.574563+0000","last_clean":"2024-04-02T13:24:55.574563+0000","last_became_active":"2024-04-02T13:24:54.574199+0000","last_became_peered":"2024-04-02T13:24:54.574199+0000","last_unstale":"2024-04-02T13:24:55.574563+0000","last_undegraded":"2024-04-02T13:24:55.574563+0000","last_fullsized":"2024-04-02T13:24:55.574563+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:48:23.402189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064197+0000","last_change":"2024-04-02T13:24:54.586211+0000","last_active":"2024-04-02T13:24:56.064197+0000","last_peered":"2024-04-02T13:24:56.064197+0000","last_clean":"2024-04-02T13:24:56.064197+0000","last_became_active":"2024-04-02T13:24:54.585863+0000","last_became_peered":"2024-04-02T13:24:54.585863+0000","last_unstale":"2024-04-02T13:24:56.064197+0000","last_undegraded":"2024-04-02T13:24:56.064197+0000","last_fullsized":"2024-04-02T13:24:56.064197+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:15:41.434299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056212+0000","last_change":"2024-04-02T13:24:54.584568+0000","last_active":"2024-04-02T13:24:56.056212+0000","last_peered":"2024-04-02T13:24:56.056212+0000","last_clean":"2024-04-02T13:24:56.056212+0000","last_became_active":"2024-04-02T13:24:54.583120+0000","last_became_peered":"2024-04-02T13:24:54.583120+0000","last_unstale":"2024-04-02T13:24:56.056212+0000","last_undegraded":"2024-04-02T13:24:56.056212+0000","last_fullsized":"2024-04-02T13:24:56.056212+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:08.440154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063782+0000","last_change":"2024-04-02T13:24:54.576524+0000","last_active":"2024-04-02T13:24:56.063782+0000","last_peered":"2024-04-02T13:24:56.063782+0000","last_clean":"2024-04-02T13:24:56.063782+0000","last_became_active":"2024-04-02T13:24:54.576151+0000","last_became_peered":"2024-04-02T13:24:54.576151+0000","last_unstale":"2024-04-02T13:24:56.063782+0000","last_undegraded":"2024-04-02T13:24:56.063782+0000","last_fullsized":"2024-04-02T13:24:56.063782+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:24:50.774098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064257+0000","last_change":"2024-04-02T13:24:54.568847+0000","last_active":"2024-04-02T13:24:56.064257+0000","last_peered":"2024-04-02T13:24:56.064257+0000","last_clean":"2024-04-02T13:24:56.064257+0000","last_became_active":"2024-04-02T13:24:54.568551+0000","last_became_peered":"2024-04-02T13:24:54.568551+0000","last_unstale":"2024-04-02T13:24:56.064257+0000","last_undegraded":"2024-04-02T13:24:56.064257+0000","last_fullsized":"2024-04-02T13:24:56.064257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:24:50.774098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064735+0000","last_change":"2024-04-02T13:24:54.580503+0000","last_active":"2024-04-02T13:24:56.064735+0000","last_peered":"2024-04-02T13:24:56.064735+0000","last_clean":"2024-04-02T13:24:56.064735+0000","last_became_active":"2024-04-02T13:24:54.580207+0000","last_became_peered":"2024-04-02T13:24:54.580207+0000","last_unstale":"2024-04-02T13:24:56.064735+0000","last_undegraded":"2024-04-02T13:24:56.064735+0000","last_fullsized":"2024-04-02T13:24:56.064735+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:18:08.897243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574697+0000","last_change":"2024-04-02T13:24:54.582497+0000","last_active":"2024-04-02T13:24:55.574697+0000","last_peered":"2024-04-02T13:24:55.574697+0000","last_clean":"2024-04-02T13:24:55.574697+0000","last_became_active":"2024-04-02T13:24:54.582221+0000","last_became_peered":"2024-04-02T13:24:54.582221+0000","last_unstale":"2024-04-02T13:24:55.574697+0000","last_undegraded":"2024-04-02T13:24:55.574697+0000","last_fullsized":"2024-04-02T13:24:55.574697+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:30:37.463997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056042+0000","last_change":"2024-04-02T13:24:54.586251+0000","last_active":"2024-04-02T13:24:56.056042+0000","last_peered":"2024-04-02T13:24:56.056042+0000","last_clean":"2024-04-02T13:24:56.056042+0000","last_became_active":"2024-04-02T13:24:54.585975+0000","last_became_peered":"2024-04-02T13:24:54.585975+0000","last_unstale":"2024-04-02T13:24:56.056042+0000","last_undegraded":"2024-04-02T13:24:56.056042+0000","last_fullsized":"2024-04-02T13:24:56.056042+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:03.634266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064056+0000","last_change":"2024-04-02T13:24:54.583263+0000","last_active":"2024-04-02T13:24:56.064056+0000","last_peered":"2024-04-02T13:24:56.064056+0000","last_clean":"2024-04-02T13:24:56.064056+0000","last_became_active":"2024-04-02T13:24:54.582968+0000","last_became_peered":"2024-04-02T13:24:54.582968+0000","last_unstale":"2024-04-02T13:24:56.064056+0000","last_undegraded":"2024-04-02T13:24:56.064056+0000","last_fullsized":"2024-04-02T13:24:56.064056+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:00:37.401118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574383+0000","last_change":"2024-04-02T13:24:54.580613+0000","last_active":"2024-04-02T13:24:55.574383+0000","last_peered":"2024-04-02T13:24:55.574383+0000","last_clean":"2024-04-02T13:24:55.574383+0000","last_became_active":"2024-04-02T13:24:54.580356+0000","last_became_peered":"2024-04-02T13:24:54.580356+0000","last_unstale":"2024-04-02T13:24:55.574383+0000","last_undegraded":"2024-04-02T13:24:55.574383+0000","last_fullsized":"2024-04-02T13:24:55.574383+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:50:28.795387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064138+0000","last_change":"2024-04-02T13:24:54.586237+0000","last_active":"2024-04-02T13:24:56.064138+0000","last_peered":"2024-04-02T13:24:56.064138+0000","last_clean":"2024-04-02T13:24:56.064138+0000","last_became_active":"2024-04-02T13:24:54.585910+0000","last_became_peered":"2024-04-02T13:24:54.585910+0000","last_unstale":"2024-04-02T13:24:56.064138+0000","last_undegraded":"2024-04-02T13:24:56.064138+0000","last_fullsized":"2024-04-02T13:24:56.064138+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:02:06.412550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574181+0000","last_change":"2024-04-02T13:24:54.573990+0000","last_active":"2024-04-02T13:24:55.574181+0000","last_peered":"2024-04-02T13:24:55.574181+0000","last_clean":"2024-04-02T13:24:55.574181+0000","last_became_active":"2024-04-02T13:24:54.573671+0000","last_became_peered":"2024-04-02T13:24:54.573671+0000","last_unstale":"2024-04-02T13:24:55.574181+0000","last_undegraded":"2024-04-02T13:24:55.574181+0000","last_fullsized":"2024-04-02T13:24:55.574181+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:29:58.020569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063647+0000","last_change":"2024-04-02T13:24:54.589275+0000","last_active":"2024-04-02T13:24:56.063647+0000","last_peered":"2024-04-02T13:24:56.063647+0000","last_clean":"2024-04-02T13:24:56.063647+0000","last_became_active":"2024-04-02T13:24:54.589010+0000","last_became_peered":"2024-04-02T13:24:54.589010+0000","last_unstale":"2024-04-02T13:24:56.063647+0000","last_undegraded":"2024-04-02T13:24:56.063647+0000","last_fullsized":"2024-04-02T13:24:56.063647+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:10:41.126700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064560+0000","last_change":"2024-04-02T13:24:54.590378+0000","last_active":"2024-04-02T13:24:56.064560+0000","last_peered":"2024-04-02T13:24:56.064560+0000","last_clean":"2024-04-02T13:24:56.064560+0000","last_became_active":"2024-04-02T13:24:54.589691+0000","last_became_peered":"2024-04-02T13:24:54.589691+0000","last_unstale":"2024-04-02T13:24:56.064560+0000","last_undegraded":"2024-04-02T13:24:56.064560+0000","last_fullsized":"2024-04-02T13:24:56.064560+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:16.381648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574149+0000","last_change":"2024-04-02T13:24:54.575310+0000","last_active":"2024-04-02T13:24:55.574149+0000","last_peered":"2024-04-02T13:24:55.574149+0000","last_clean":"2024-04-02T13:24:55.574149+0000","last_became_active":"2024-04-02T13:24:54.575029+0000","last_became_peered":"2024-04-02T13:24:54.575029+0000","last_unstale":"2024-04-02T13:24:55.574149+0000","last_undegraded":"2024-04-02T13:24:55.574149+0000","last_fullsized":"2024-04-02T13:24:55.574149+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:12.654707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064538+0000","last_change":"2024-04-02T13:24:54.572607+0000","last_active":"2024-04-02T13:24:56.064538+0000","last_peered":"2024-04-02T13:24:56.064538+0000","last_clean":"2024-04-02T13:24:56.064538+0000","last_became_active":"2024-04-02T13:24:54.571815+0000","last_became_peered":"2024-04-02T13:24:54.571815+0000","last_unstale":"2024-04-02T13:24:56.064538+0000","last_undegraded":"2024-04-02T13:24:56.064538+0000","last_fullsized":"2024-04-02T13:24:56.064538+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:16.445368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064414+0000","last_change":"2024-04-02T13:24:54.566761+0000","last_active":"2024-04-02T13:24:56.064414+0000","last_peered":"2024-04-02T13:24:56.064414+0000","last_clean":"2024-04-02T13:24:56.064414+0000","last_became_active":"2024-04-02T13:24:54.566488+0000","last_became_peered":"2024-04-02T13:24:54.566488+0000","last_unstale":"2024-04-02T13:24:56.064414+0000","last_undegraded":"2024-04-02T13:24:56.064414+0000","last_fullsized":"2024-04-02T13:24:56.064414+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:14:10.643231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064315+0000","last_change":"2024-04-02T13:24:54.588347+0000","last_active":"2024-04-02T13:24:56.064315+0000","last_peered":"2024-04-02T13:24:56.064315+0000","last_clean":"2024-04-02T13:24:56.064315+0000","last_became_active":"2024-04-02T13:24:54.588078+0000","last_became_peered":"2024-04-02T13:24:54.588078+0000","last_unstale":"2024-04-02T13:24:56.064315+0000","last_undegraded":"2024-04-02T13:24:56.064315+0000","last_fullsized":"2024-04-02T13:24:56.064315+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:14:01.535902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064151+0000","last_change":"2024-04-02T13:24:54.571000+0000","last_active":"2024-04-02T13:24:56.064151+0000","last_peered":"2024-04-02T13:24:56.064151+0000","last_clean":"2024-04-02T13:24:56.064151+0000","last_became_active":"2024-04-02T13:24:54.570722+0000","last_became_peered":"2024-04-02T13:24:54.570722+0000","last_unstale":"2024-04-02T13:24:56.064151+0000","last_undegraded":"2024-04-02T13:24:56.064151+0000","last_fullsized":"2024-04-02T13:24:56.064151+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:42:33.303173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574577+0000","last_change":"2024-04-02T13:24:54.582308+0000","last_active":"2024-04-02T13:24:55.574577+0000","last_peered":"2024-04-02T13:24:55.574577+0000","last_clean":"2024-04-02T13:24:55.574577+0000","last_became_active":"2024-04-02T13:24:54.582030+0000","last_became_peered":"2024-04-02T13:24:54.582030+0000","last_unstale":"2024-04-02T13:24:55.574577+0000","last_undegraded":"2024-04-02T13:24:55.574577+0000","last_fullsized":"2024-04-02T13:24:55.574577+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:53:36.691313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064449+0000","last_change":"2024-04-02T13:24:54.577125+0000","last_active":"2024-04-02T13:24:56.064449+0000","last_peered":"2024-04-02T13:24:56.064449+0000","last_clean":"2024-04-02T13:24:56.064449+0000","last_became_active":"2024-04-02T13:24:54.576839+0000","last_became_peered":"2024-04-02T13:24:54.576839+0000","last_unstale":"2024-04-02T13:24:56.064449+0000","last_undegraded":"2024-04-02T13:24:56.064449+0000","last_fullsized":"2024-04-02T13:24:56.064449+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:59.553628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574571+0000","last_change":"2024-04-02T13:24:54.576466+0000","last_active":"2024-04-02T13:24:55.574571+0000","last_peered":"2024-04-02T13:24:55.574571+0000","last_clean":"2024-04-02T13:24:55.574571+0000","last_became_active":"2024-04-02T13:24:54.576195+0000","last_became_peered":"2024-04-02T13:24:54.576195+0000","last_unstale":"2024-04-02T13:24:55.574571+0000","last_undegraded":"2024-04-02T13:24:55.574571+0000","last_fullsized":"2024-04-02T13:24:55.574571+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:43:11.976047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"78'4","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064703+0000","last_change":"2024-04-02T13:24:53.573023+0000","last_active":"2024-04-02T13:24:56.064703+0000","last_peered":"2024-04-02T13:24:56.064703+0000","last_clean":"2024-04-02T13:24:56.064703+0000","last_became_active":"2024-04-02T13:24:53.571627+0000","last_became_peered":"2024-04-02T13:24:53.571627+0000","last_unstale":"2024-04-02T13:24:56.064703+0000","last_undegraded":"2024-04-02T13:24:56.064703+0000","last_fullsized":"2024-04-02T13:24:56.064703+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:05:17.417576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063953+0000","last_change":"2024-04-02T13:24:54.572282+0000","last_active":"2024-04-02T13:24:56.063953+0000","last_peered":"2024-04-02T13:24:56.063953+0000","last_clean":"2024-04-02T13:24:56.063953+0000","last_became_active":"2024-04-02T13:24:54.572000+0000","last_became_peered":"2024-04-02T13:24:54.572000+0000","last_unstale":"2024-04-02T13:24:56.063953+0000","last_undegraded":"2024-04-02T13:24:56.063953+0000","last_fullsized":"2024-04-02T13:24:56.063953+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:59.553628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064680+0000","last_change":"2024-04-02T13:24:53.573555+0000","last_active":"2024-04-02T13:24:56.064680+0000","last_peered":"2024-04-02T13:24:56.064680+0000","last_clean":"2024-04-02T13:24:56.064680+0000","last_became_active":"2024-04-02T13:24:53.573158+0000","last_became_peered":"2024-04-02T13:24:53.573158+0000","last_unstale":"2024-04-02T13:24:56.064680+0000","last_undegraded":"2024-04-02T13:24:56.064680+0000","last_fullsized":"2024-04-02T13:24:56.064680+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:58:27.796359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574630+0000","last_change":"2024-04-02T13:24:54.576685+0000","last_active":"2024-04-02T13:24:55.574630+0000","last_peered":"2024-04-02T13:24:55.574630+0000","last_clean":"2024-04-02T13:24:55.574630+0000","last_became_active":"2024-04-02T13:24:54.576321+0000","last_became_peered":"2024-04-02T13:24:54.576321+0000","last_unstale":"2024-04-02T13:24:55.574630+0000","last_undegraded":"2024-04-02T13:24:55.574630+0000","last_fullsized":"2024-04-02T13:24:55.574630+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:27.408601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064612+0000","last_change":"2024-04-02T13:24:53.578788+0000","last_active":"2024-04-02T13:24:56.064612+0000","last_peered":"2024-04-02T13:24:56.064612+0000","last_clean":"2024-04-02T13:24:56.064612+0000","last_became_active":"2024-04-02T13:24:53.578523+0000","last_became_peered":"2024-04-02T13:24:53.578523+0000","last_unstale":"2024-04-02T13:24:56.064612+0000","last_undegraded":"2024-04-02T13:24:56.064612+0000","last_fullsized":"2024-04-02T13:24:56.064612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:22:06.708202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574544+0000","last_change":"2024-04-02T13:24:54.580439+0000","last_active":"2024-04-02T13:24:55.574544+0000","last_peered":"2024-04-02T13:24:55.574544+0000","last_clean":"2024-04-02T13:24:55.574544+0000","last_became_active":"2024-04-02T13:24:54.579623+0000","last_became_peered":"2024-04-02T13:24:54.579623+0000","last_unstale":"2024-04-02T13:24:55.574544+0000","last_undegraded":"2024-04-02T13:24:55.574544+0000","last_fullsized":"2024-04-02T13:24:55.574544+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:48:59.714798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064200+0000","last_change":"2024-04-02T13:24:53.575848+0000","last_active":"2024-04-02T13:24:56.064200+0000","last_peered":"2024-04-02T13:24:56.064200+0000","last_clean":"2024-04-02T13:24:56.064200+0000","last_became_active":"2024-04-02T13:24:53.574537+0000","last_became_peered":"2024-04-02T13:24:53.574537+0000","last_unstale":"2024-04-02T13:24:56.064200+0000","last_undegraded":"2024-04-02T13:24:56.064200+0000","last_fullsized":"2024-04-02T13:24:56.064200+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:02:30.957187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064736+0000","last_change":"2024-04-02T13:24:54.583494+0000","last_active":"2024-04-02T13:24:56.064736+0000","last_peered":"2024-04-02T13:24:56.064736+0000","last_clean":"2024-04-02T13:24:56.064736+0000","last_became_active":"2024-04-02T13:24:54.580371+0000","last_became_peered":"2024-04-02T13:24:54.580371+0000","last_unstale":"2024-04-02T13:24:56.064736+0000","last_undegraded":"2024-04-02T13:24:56.064736+0000","last_fullsized":"2024-04-02T13:24:56.064736+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:03.634266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574609+0000","last_change":"2024-04-02T13:24:53.574172+0000","last_active":"2024-04-02T13:24:55.574609+0000","last_peered":"2024-04-02T13:24:55.574609+0000","last_clean":"2024-04-02T13:24:55.574609+0000","last_became_active":"2024-04-02T13:24:53.573883+0000","last_became_peered":"2024-04-02T13:24:53.573883+0000","last_unstale":"2024-04-02T13:24:55.574609+0000","last_undegraded":"2024-04-02T13:24:55.574609+0000","last_fullsized":"2024-04-02T13:24:55.574609+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:08.877638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064443+0000","last_change":"2024-04-02T13:24:54.585871+0000","last_active":"2024-04-02T13:24:56.064443+0000","last_peered":"2024-04-02T13:24:56.064443+0000","last_clean":"2024-04-02T13:24:56.064443+0000","last_became_active":"2024-04-02T13:24:54.585586+0000","last_became_peered":"2024-04-02T13:24:54.585586+0000","last_unstale":"2024-04-02T13:24:56.064443+0000","last_undegraded":"2024-04-02T13:24:56.064443+0000","last_fullsized":"2024-04-02T13:24:56.064443+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:04:44.744456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573845+0000","last_change":"2024-04-02T13:24:53.571030+0000","last_active":"2024-04-02T13:24:55.573845+0000","last_peered":"2024-04-02T13:24:55.573845+0000","last_clean":"2024-04-02T13:24:55.573845+0000","last_became_active":"2024-04-02T13:24:53.570763+0000","last_became_peered":"2024-04-02T13:24:53.570763+0000","last_unstale":"2024-04-02T13:24:55.573845+0000","last_undegraded":"2024-04-02T13:24:55.573845+0000","last_fullsized":"2024-04-02T13:24:55.573845+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:07.392500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574899+0000","last_change":"2024-04-02T13:24:54.586818+0000","last_active":"2024-04-02T13:24:55.574899+0000","last_peered":"2024-04-02T13:24:55.574899+0000","last_clean":"2024-04-02T13:24:55.574899+0000","last_became_active":"2024-04-02T13:24:54.586528+0000","last_became_peered":"2024-04-02T13:24:54.586528+0000","last_unstale":"2024-04-02T13:24:55.574899+0000","last_undegraded":"2024-04-02T13:24:55.574899+0000","last_fullsized":"2024-04-02T13:24:55.574899+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:13:07.461166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064057+0000","last_change":"2024-04-02T13:24:53.578249+0000","last_active":"2024-04-02T13:24:56.064057+0000","last_peered":"2024-04-02T13:24:56.064057+0000","last_clean":"2024-04-02T13:24:56.064057+0000","last_became_active":"2024-04-02T13:24:53.577992+0000","last_became_peered":"2024-04-02T13:24:53.577992+0000","last_unstale":"2024-04-02T13:24:56.064057+0000","last_undegraded":"2024-04-02T13:24:56.064057+0000","last_fullsized":"2024-04-02T13:24:56.064057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:23:42.359046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064672+0000","last_change":"2024-04-02T13:24:54.578694+0000","last_active":"2024-04-02T13:24:56.064672+0000","last_peered":"2024-04-02T13:24:56.064672+0000","last_clean":"2024-04-02T13:24:56.064672+0000","last_became_active":"2024-04-02T13:24:54.578394+0000","last_became_peered":"2024-04-02T13:24:54.578394+0000","last_unstale":"2024-04-02T13:24:56.064672+0000","last_undegraded":"2024-04-02T13:24:56.064672+0000","last_fullsized":"2024-04-02T13:24:56.064672+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:07.594969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063737+0000","last_change":"2024-04-02T13:24:53.572838+0000","last_active":"2024-04-02T13:24:56.063737+0000","last_peered":"2024-04-02T13:24:56.063737+0000","last_clean":"2024-04-02T13:24:56.063737+0000","last_became_active":"2024-04-02T13:24:53.571893+0000","last_became_peered":"2024-04-02T13:24:53.571893+0000","last_unstale":"2024-04-02T13:24:56.063737+0000","last_undegraded":"2024-04-02T13:24:56.063737+0000","last_fullsized":"2024-04-02T13:24:56.063737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:06:01.172766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063725+0000","last_change":"2024-04-02T13:24:54.579310+0000","last_active":"2024-04-02T13:24:56.063725+0000","last_peered":"2024-04-02T13:24:56.063725+0000","last_clean":"2024-04-02T13:24:56.063725+0000","last_became_active":"2024-04-02T13:24:54.578356+0000","last_became_peered":"2024-04-02T13:24:54.578356+0000","last_unstale":"2024-04-02T13:24:56.063725+0000","last_undegraded":"2024-04-02T13:24:56.063725+0000","last_fullsized":"2024-04-02T13:24:56.063725+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:42:33.303173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064892+0000","last_change":"2024-04-02T13:24:53.579943+0000","last_active":"2024-04-02T13:24:56.064892+0000","last_peered":"2024-04-02T13:24:56.064892+0000","last_clean":"2024-04-02T13:24:56.064892+0000","last_became_active":"2024-04-02T13:24:53.579632+0000","last_became_peered":"2024-04-02T13:24:53.579632+0000","last_unstale":"2024-04-02T13:24:56.064892+0000","last_undegraded":"2024-04-02T13:24:56.064892+0000","last_fullsized":"2024-04-02T13:24:56.064892+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:48:22.399865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573933+0000","last_change":"2024-04-02T13:24:54.580573+0000","last_active":"2024-04-02T13:24:55.573933+0000","last_peered":"2024-04-02T13:24:55.573933+0000","last_clean":"2024-04-02T13:24:55.573933+0000","last_became_active":"2024-04-02T13:24:54.580323+0000","last_became_peered":"2024-04-02T13:24:54.580323+0000","last_unstale":"2024-04-02T13:24:55.573933+0000","last_undegraded":"2024-04-02T13:24:55.573933+0000","last_fullsized":"2024-04-02T13:24:55.573933+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:43:43.130201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574105+0000","last_change":"2024-04-02T13:24:53.570841+0000","last_active":"2024-04-02T13:24:55.574105+0000","last_peered":"2024-04-02T13:24:55.574105+0000","last_clean":"2024-04-02T13:24:55.574105+0000","last_became_active":"2024-04-02T13:24:53.570567+0000","last_became_peered":"2024-04-02T13:24:53.570567+0000","last_unstale":"2024-04-02T13:24:55.574105+0000","last_undegraded":"2024-04-02T13:24:55.574105+0000","last_fullsized":"2024-04-02T13:24:55.574105+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:51.244520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056346+0000","last_change":"2024-04-02T13:24:54.572495+0000","last_active":"2024-04-02T13:24:56.056346+0000","last_peered":"2024-04-02T13:24:56.056346+0000","last_clean":"2024-04-02T13:24:56.056346+0000","last_became_active":"2024-04-02T13:24:54.572181+0000","last_became_peered":"2024-04-02T13:24:54.572181+0000","last_unstale":"2024-04-02T13:24:56.056346+0000","last_undegraded":"2024-04-02T13:24:56.056346+0000","last_fullsized":"2024-04-02T13:24:56.056346+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:34:15.079374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"78'3","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064694+0000","last_change":"2024-04-02T13:24:53.577054+0000","last_active":"2024-04-02T13:24:56.064694+0000","last_peered":"2024-04-02T13:24:56.064694+0000","last_clean":"2024-04-02T13:24:56.064694+0000","last_became_active":"2024-04-02T13:24:53.575955+0000","last_became_peered":"2024-04-02T13:24:53.575955+0000","last_unstale":"2024-04-02T13:24:56.064694+0000","last_undegraded":"2024-04-02T13:24:56.064694+0000","last_fullsized":"2024-04-02T13:24:56.064694+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:16:43.448093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574776+0000","last_change":"2024-04-02T13:24:54.572847+0000","last_active":"2024-04-02T13:24:55.574776+0000","last_peered":"2024-04-02T13:24:55.574776+0000","last_clean":"2024-04-02T13:24:55.574776+0000","last_became_active":"2024-04-02T13:24:54.572588+0000","last_became_peered":"2024-04-02T13:24:54.572588+0000","last_unstale":"2024-04-02T13:24:55.574776+0000","last_undegraded":"2024-04-02T13:24:55.574776+0000","last_fullsized":"2024-04-02T13:24:55.574776+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:09:55.531393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"78'6","reported_seq":20,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064310+0000","last_change":"2024-04-02T13:24:53.575781+0000","last_active":"2024-04-02T13:24:56.064310+0000","last_peered":"2024-04-02T13:24:56.064310+0000","last_clean":"2024-04-02T13:24:56.064310+0000","last_became_active":"2024-04-02T13:24:53.574619+0000","last_became_peered":"2024-04-02T13:24:53.574619+0000","last_unstale":"2024-04-02T13:24:56.064310+0000","last_undegraded":"2024-04-02T13:24:56.064310+0000","last_fullsized":"2024-04-02T13:24:56.064310+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:29:57.018246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574715+0000","last_change":"2024-04-02T13:24:54.568437+0000","last_active":"2024-04-02T13:24:55.574715+0000","last_peered":"2024-04-02T13:24:55.574715+0000","last_clean":"2024-04-02T13:24:55.574715+0000","last_became_active":"2024-04-02T13:24:54.568101+0000","last_became_peered":"2024-04-02T13:24:54.568101+0000","last_unstale":"2024-04-02T13:24:55.574715+0000","last_undegraded":"2024-04-02T13:24:55.574715+0000","last_fullsized":"2024-04-02T13:24:55.574715+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:07:26.224860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573889+0000","last_change":"2024-04-02T13:24:53.572426+0000","last_active":"2024-04-02T13:24:55.573889+0000","last_peered":"2024-04-02T13:24:55.573889+0000","last_clean":"2024-04-02T13:24:55.573889+0000","last_became_active":"2024-04-02T13:24:53.572153+0000","last_became_peered":"2024-04-02T13:24:53.572153+0000","last_unstale":"2024-04-02T13:24:55.573889+0000","last_undegraded":"2024-04-02T13:24:55.573889+0000","last_fullsized":"2024-04-02T13:24:55.573889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:57:16.242492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064499+0000","last_change":"2024-04-02T13:24:54.578234+0000","last_active":"2024-04-02T13:24:56.064499+0000","last_peered":"2024-04-02T13:24:56.064499+0000","last_clean":"2024-04-02T13:24:56.064499+0000","last_became_active":"2024-04-02T13:24:54.577914+0000","last_became_peered":"2024-04-02T13:24:54.577914+0000","last_unstale":"2024-04-02T13:24:56.064499+0000","last_undegraded":"2024-04-02T13:24:56.064499+0000","last_fullsized":"2024-04-02T13:24:56.064499+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:52.246844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064325+0000","last_change":"2024-04-02T13:24:53.580278+0000","last_active":"2024-04-02T13:24:56.064325+0000","last_peered":"2024-04-02T13:24:56.064325+0000","last_clean":"2024-04-02T13:24:56.064325+0000","last_became_active":"2024-04-02T13:24:53.580002+0000","last_became_peered":"2024-04-02T13:24:53.580002+0000","last_unstale":"2024-04-02T13:24:56.064325+0000","last_undegraded":"2024-04-02T13:24:56.064325+0000","last_fullsized":"2024-04-02T13:24:56.064325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:24.786741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064483+0000","last_change":"2024-04-02T13:24:54.579646+0000","last_active":"2024-04-02T13:24:56.064483+0000","last_peered":"2024-04-02T13:24:56.064483+0000","last_clean":"2024-04-02T13:24:56.064483+0000","last_became_active":"2024-04-02T13:24:54.579360+0000","last_became_peered":"2024-04-02T13:24:54.579360+0000","last_unstale":"2024-04-02T13:24:56.064483+0000","last_undegraded":"2024-04-02T13:24:56.064483+0000","last_fullsized":"2024-04-02T13:24:56.064483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:39:31.964323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063999+0000","last_change":"2024-04-02T13:24:53.576553+0000","last_active":"2024-04-02T13:24:56.063999+0000","last_peered":"2024-04-02T13:24:56.063999+0000","last_clean":"2024-04-02T13:24:56.063999+0000","last_became_active":"2024-04-02T13:24:53.576219+0000","last_became_peered":"2024-04-02T13:24:53.576219+0000","last_unstale":"2024-04-02T13:24:56.063999+0000","last_undegraded":"2024-04-02T13:24:56.063999+0000","last_fullsized":"2024-04-02T13:24:56.063999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:12:36.353661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064755+0000","last_change":"2024-04-02T13:24:54.584841+0000","last_active":"2024-04-02T13:24:56.064755+0000","last_peered":"2024-04-02T13:24:56.064755+0000","last_clean":"2024-04-02T13:24:56.064755+0000","last_became_active":"2024-04-02T13:24:54.584565+0000","last_became_peered":"2024-04-02T13:24:54.584565+0000","last_unstale":"2024-04-02T13:24:56.064755+0000","last_undegraded":"2024-04-02T13:24:56.064755+0000","last_fullsized":"2024-04-02T13:24:56.064755+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:23:43.361370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"78'8","reported_seq":22,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063758+0000","last_change":"2024-04-02T13:24:53.575318+0000","last_active":"2024-04-02T13:24:56.063758+0000","last_peered":"2024-04-02T13:24:56.063758+0000","last_clean":"2024-04-02T13:24:56.063758+0000","last_became_active":"2024-04-02T13:24:53.575019+0000","last_became_peered":"2024-04-02T13:24:53.575019+0000","last_unstale":"2024-04-02T13:24:56.063758+0000","last_undegraded":"2024-04-02T13:24:56.063758+0000","last_fullsized":"2024-04-02T13:24:56.063758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:06.592645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.055822+0000","last_change":"2024-04-02T13:24:54.587605+0000","last_active":"2024-04-02T13:24:56.055822+0000","last_peered":"2024-04-02T13:24:56.055822+0000","last_clean":"2024-04-02T13:24:56.055822+0000","last_became_active":"2024-04-02T13:24:54.587310+0000","last_became_peered":"2024-04-02T13:24:54.587310+0000","last_unstale":"2024-04-02T13:24:56.055822+0000","last_undegraded":"2024-04-02T13:24:56.055822+0000","last_fullsized":"2024-04-02T13:24:56.055822+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:04:44.744456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064915+0000","last_change":"2024-04-02T13:24:53.577390+0000","last_active":"2024-04-02T13:24:56.064915+0000","last_peered":"2024-04-02T13:24:56.064915+0000","last_clean":"2024-04-02T13:24:56.064915+0000","last_became_active":"2024-04-02T13:24:53.577079+0000","last_became_peered":"2024-04-02T13:24:53.577079+0000","last_unstale":"2024-04-02T13:24:56.064915+0000","last_undegraded":"2024-04-02T13:24:56.064915+0000","last_fullsized":"2024-04-02T13:24:56.064915+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:07:25.222536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064186+0000","last_change":"2024-04-02T13:24:54.579775+0000","last_active":"2024-04-02T13:24:56.064186+0000","last_peered":"2024-04-02T13:24:56.064186+0000","last_clean":"2024-04-02T13:24:56.064186+0000","last_became_active":"2024-04-02T13:24:54.579459+0000","last_became_peered":"2024-04-02T13:24:54.579459+0000","last_unstale":"2024-04-02T13:24:56.064186+0000","last_undegraded":"2024-04-02T13:24:56.064186+0000","last_fullsized":"2024-04-02T13:24:56.064186+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:03:48.945748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574863+0000","last_change":"2024-04-02T13:24:53.577294+0000","last_active":"2024-04-02T13:24:55.574863+0000","last_peered":"2024-04-02T13:24:55.574863+0000","last_clean":"2024-04-02T13:24:55.574863+0000","last_became_active":"2024-04-02T13:24:53.577018+0000","last_became_peered":"2024-04-02T13:24:53.577018+0000","last_unstale":"2024-04-02T13:24:55.574863+0000","last_undegraded":"2024-04-02T13:24:55.574863+0000","last_fullsized":"2024-04-02T13:24:55.574863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:18:07.894919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064584+0000","last_change":"2024-04-02T13:24:54.581345+0000","last_active":"2024-04-02T13:24:56.064584+0000","last_peered":"2024-04-02T13:24:56.064584+0000","last_clean":"2024-04-02T13:24:56.064584+0000","last_became_active":"2024-04-02T13:24:54.581091+0000","last_became_peered":"2024-04-02T13:24:54.581091+0000","last_unstale":"2024-04-02T13:24:56.064584+0000","last_undegraded":"2024-04-02T13:24:56.064584+0000","last_fullsized":"2024-04-02T13:24:56.064584+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:09:04.946124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064588+0000","last_change":"2024-04-02T13:24:53.573420+0000","last_active":"2024-04-02T13:24:56.064588+0000","last_peered":"2024-04-02T13:24:56.064588+0000","last_clean":"2024-04-02T13:24:56.064588+0000","last_became_active":"2024-04-02T13:24:53.571909+0000","last_became_peered":"2024-04-02T13:24:53.571909+0000","last_unstale":"2024-04-02T13:24:56.064588+0000","last_undegraded":"2024-04-02T13:24:56.064588+0000","last_fullsized":"2024-04-02T13:24:56.064588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:08:06.577978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056130+0000","last_change":"2024-04-02T13:24:54.586519+0000","last_active":"2024-04-02T13:24:56.056130+0000","last_peered":"2024-04-02T13:24:56.056130+0000","last_clean":"2024-04-02T13:24:56.056130+0000","last_became_active":"2024-04-02T13:24:54.584751+0000","last_became_peered":"2024-04-02T13:24:54.584751+0000","last_unstale":"2024-04-02T13:24:56.056130+0000","last_undegraded":"2024-04-02T13:24:56.056130+0000","last_fullsized":"2024-04-02T13:24:56.056130+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:25.219021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574737+0000","last_change":"2024-04-02T13:24:53.570594+0000","last_active":"2024-04-02T13:24:55.574737+0000","last_peered":"2024-04-02T13:24:55.574737+0000","last_clean":"2024-04-02T13:24:55.574737+0000","last_became_active":"2024-04-02T13:24:53.569678+0000","last_became_peered":"2024-04-02T13:24:53.569678+0000","last_unstale":"2024-04-02T13:24:55.574737+0000","last_undegraded":"2024-04-02T13:24:55.574737+0000","last_fullsized":"2024-04-02T13:24:55.574737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:02:30.957187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064778+0000","last_change":"2024-04-02T13:24:54.575816+0000","last_active":"2024-04-02T13:24:56.064778+0000","last_peered":"2024-04-02T13:24:56.064778+0000","last_clean":"2024-04-02T13:24:56.064778+0000","last_became_active":"2024-04-02T13:24:54.572762+0000","last_became_peered":"2024-04-02T13:24:54.572762+0000","last_unstale":"2024-04-02T13:24:56.064778+0000","last_undegraded":"2024-04-02T13:24:56.064778+0000","last_fullsized":"2024-04-02T13:24:56.064778+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:34:15.079374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056135+0000","last_change":"2024-04-02T13:24:53.571104+0000","last_active":"2024-04-02T13:24:56.056135+0000","last_peered":"2024-04-02T13:24:56.056135+0000","last_clean":"2024-04-02T13:24:56.056135+0000","last_became_active":"2024-04-02T13:24:53.570758+0000","last_became_peered":"2024-04-02T13:24:53.570758+0000","last_unstale":"2024-04-02T13:24:56.056135+0000","last_undegraded":"2024-04-02T13:24:56.056135+0000","last_fullsized":"2024-04-02T13:24:56.056135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:12:29.693987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064481+0000","last_change":"2024-04-02T13:24:54.585045+0000","last_active":"2024-04-02T13:24:56.064481+0000","last_peered":"2024-04-02T13:24:56.064481+0000","last_clean":"2024-04-02T13:24:56.064481+0000","last_became_active":"2024-04-02T13:24:54.584797+0000","last_became_peered":"2024-04-02T13:24:54.584797+0000","last_unstale":"2024-04-02T13:24:56.064481+0000","last_undegraded":"2024-04-02T13:24:56.064481+0000","last_fullsized":"2024-04-02T13:24:56.064481+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:25.219021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.055901+0000","last_change":"2024-04-02T13:24:53.573510+0000","last_active":"2024-04-02T13:24:56.055901+0000","last_peered":"2024-04-02T13:24:56.055901+0000","last_clean":"2024-04-02T13:24:56.055901+0000","last_became_active":"2024-04-02T13:24:53.572671+0000","last_became_peered":"2024-04-02T13:24:53.572671+0000","last_unstale":"2024-04-02T13:24:56.055901+0000","last_undegraded":"2024-04-02T13:24:56.055901+0000","last_fullsized":"2024-04-02T13:24:56.055901+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:36.056546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064931+0000","last_change":"2024-04-02T13:24:54.578914+0000","last_active":"2024-04-02T13:24:56.064931+0000","last_peered":"2024-04-02T13:24:56.064931+0000","last_clean":"2024-04-02T13:24:56.064931+0000","last_became_active":"2024-04-02T13:24:54.578531+0000","last_became_peered":"2024-04-02T13:24:54.578531+0000","last_unstale":"2024-04-02T13:24:56.064931+0000","last_undegraded":"2024-04-02T13:24:56.064931+0000","last_fullsized":"2024-04-02T13:24:56.064931+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:50.669792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.055891+0000","last_change":"2024-04-02T13:24:53.574067+0000","last_active":"2024-04-02T13:24:56.055891+0000","last_peered":"2024-04-02T13:24:56.055891+0000","last_clean":"2024-04-02T13:24:56.055891+0000","last_became_active":"2024-04-02T13:24:53.573799+0000","last_became_peered":"2024-04-02T13:24:53.573799+0000","last_unstale":"2024-04-02T13:24:56.055891+0000","last_undegraded":"2024-04-02T13:24:56.055891+0000","last_fullsized":"2024-04-02T13:24:56.055891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:30:06.264873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574142+0000","last_change":"2024-04-02T13:24:54.579259+0000","last_active":"2024-04-02T13:24:55.574142+0000","last_peered":"2024-04-02T13:24:55.574142+0000","last_clean":"2024-04-02T13:24:55.574142+0000","last_became_active":"2024-04-02T13:24:54.578957+0000","last_became_peered":"2024-04-02T13:24:54.578957+0000","last_unstale":"2024-04-02T13:24:55.574142+0000","last_undegraded":"2024-04-02T13:24:55.574142+0000","last_fullsized":"2024-04-02T13:24:55.574142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:02:31.959511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573976+0000","last_change":"2024-04-02T13:24:53.575531+0000","last_active":"2024-04-02T13:24:55.573976+0000","last_peered":"2024-04-02T13:24:55.573976+0000","last_clean":"2024-04-02T13:24:55.573976+0000","last_became_active":"2024-04-02T13:24:53.575222+0000","last_became_peered":"2024-04-02T13:24:53.575222+0000","last_unstale":"2024-04-02T13:24:55.573976+0000","last_undegraded":"2024-04-02T13:24:55.573976+0000","last_fullsized":"2024-04-02T13:24:55.573976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:39:30.962000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064095+0000","last_change":"2024-04-02T13:24:54.575971+0000","last_active":"2024-04-02T13:24:56.064095+0000","last_peered":"2024-04-02T13:24:56.064095+0000","last_clean":"2024-04-02T13:24:56.064095+0000","last_became_active":"2024-04-02T13:24:54.575707+0000","last_became_peered":"2024-04-02T13:24:54.575707+0000","last_unstale":"2024-04-02T13:24:56.064095+0000","last_undegraded":"2024-04-02T13:24:56.064095+0000","last_fullsized":"2024-04-02T13:24:56.064095+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:08.394824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"79'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064634+0000","last_change":"2024-04-02T13:24:53.577485+0000","last_active":"2024-04-02T13:24:56.064634+0000","last_peered":"2024-04-02T13:24:56.064634+0000","last_clean":"2024-04-02T13:24:56.064634+0000","last_became_active":"2024-04-02T13:24:53.575330+0000","last_became_peered":"2024-04-02T13:24:53.575330+0000","last_unstale":"2024-04-02T13:24:56.064634+0000","last_undegraded":"2024-04-02T13:24:56.064634+0000","last_fullsized":"2024-04-02T13:24:56.064634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:49.667468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574799+0000","last_change":"2024-04-02T13:24:54.576490+0000","last_active":"2024-04-02T13:24:55.574799+0000","last_peered":"2024-04-02T13:24:55.574799+0000","last_clean":"2024-04-02T13:24:55.574799+0000","last_became_active":"2024-04-02T13:24:54.576224+0000","last_became_peered":"2024-04-02T13:24:54.576224+0000","last_unstale":"2024-04-02T13:24:55.574799+0000","last_undegraded":"2024-04-02T13:24:55.574799+0000","last_fullsized":"2024-04-02T13:24:55.574799+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:15:49.793401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064732+0000","last_change":"2024-04-02T13:24:53.572963+0000","last_active":"2024-04-02T13:24:56.064732+0000","last_peered":"2024-04-02T13:24:56.064732+0000","last_clean":"2024-04-02T13:24:56.064732+0000","last_became_active":"2024-04-02T13:24:53.572536+0000","last_became_peered":"2024-04-02T13:24:53.572536+0000","last_unstale":"2024-04-02T13:24:56.064732+0000","last_undegraded":"2024-04-02T13:24:56.064732+0000","last_fullsized":"2024-04-02T13:24:56.064732+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:43:42.127877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064578+0000","last_change":"2024-04-02T13:24:54.574356+0000","last_active":"2024-04-02T13:24:56.064578+0000","last_peered":"2024-04-02T13:24:56.064578+0000","last_clean":"2024-04-02T13:24:56.064578+0000","last_became_active":"2024-04-02T13:24:54.573985+0000","last_became_peered":"2024-04-02T13:24:54.573985+0000","last_unstale":"2024-04-02T13:24:56.064578+0000","last_undegraded":"2024-04-02T13:24:56.064578+0000","last_fullsized":"2024-04-02T13:24:56.064578+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:08.440154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064559+0000","last_change":"2024-04-02T13:24:53.580007+0000","last_active":"2024-04-02T13:24:56.064559+0000","last_peered":"2024-04-02T13:24:56.064559+0000","last_clean":"2024-04-02T13:24:56.064559+0000","last_became_active":"2024-04-02T13:24:53.579686+0000","last_became_peered":"2024-04-02T13:24:53.579686+0000","last_unstale":"2024-04-02T13:24:56.064559+0000","last_undegraded":"2024-04-02T13:24:56.064559+0000","last_fullsized":"2024-04-02T13:24:56.064559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:11.652383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064232+0000","last_change":"2024-04-02T13:24:54.585002+0000","last_active":"2024-04-02T13:24:56.064232+0000","last_peered":"2024-04-02T13:24:56.064232+0000","last_clean":"2024-04-02T13:24:56.064232+0000","last_became_active":"2024-04-02T13:24:54.584620+0000","last_became_peered":"2024-04-02T13:24:54.584620+0000","last_unstale":"2024-04-02T13:24:56.064232+0000","last_undegraded":"2024-04-02T13:24:56.064232+0000","last_fullsized":"2024-04-02T13:24:56.064232+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:57:17.244816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064180+0000","last_change":"2024-04-02T13:24:53.572298+0000","last_active":"2024-04-02T13:24:56.064180+0000","last_peered":"2024-04-02T13:24:56.064180+0000","last_clean":"2024-04-02T13:24:56.064180+0000","last_became_active":"2024-04-02T13:24:53.570380+0000","last_became_peered":"2024-04-02T13:24:53.570380+0000","last_unstale":"2024-04-02T13:24:56.064180+0000","last_undegraded":"2024-04-02T13:24:56.064180+0000","last_fullsized":"2024-04-02T13:24:56.064180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:43:26.495241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064671+0000","last_change":"2024-04-02T13:24:54.573352+0000","last_active":"2024-04-02T13:24:56.064671+0000","last_peered":"2024-04-02T13:24:56.064671+0000","last_clean":"2024-04-02T13:24:56.064671+0000","last_became_active":"2024-04-02T13:24:54.573039+0000","last_became_peered":"2024-04-02T13:24:54.573039+0000","last_unstale":"2024-04-02T13:24:56.064671+0000","last_undegraded":"2024-04-02T13:24:56.064671+0000","last_fullsized":"2024-04-02T13:24:56.064671+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:46:34.338949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064247+0000","last_change":"2024-04-02T13:24:53.572484+0000","last_active":"2024-04-02T13:24:56.064247+0000","last_peered":"2024-04-02T13:24:56.064247+0000","last_clean":"2024-04-02T13:24:56.064247+0000","last_became_active":"2024-04-02T13:24:53.572124+0000","last_became_peered":"2024-04-02T13:24:53.572124+0000","last_unstale":"2024-04-02T13:24:56.064247+0000","last_undegraded":"2024-04-02T13:24:56.064247+0000","last_fullsized":"2024-04-02T13:24:56.064247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:39:18.965555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574852+0000","last_change":"2024-04-02T13:24:54.573678+0000","last_active":"2024-04-02T13:24:55.574852+0000","last_peered":"2024-04-02T13:24:55.574852+0000","last_clean":"2024-04-02T13:24:55.574852+0000","last_became_active":"2024-04-02T13:24:54.573380+0000","last_became_peered":"2024-04-02T13:24:54.573380+0000","last_unstale":"2024-04-02T13:24:55.574852+0000","last_undegraded":"2024-04-02T13:24:55.574852+0000","last_fullsized":"2024-04-02T13:24:55.574852+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:48:44.079142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064587+0000","last_change":"2024-04-02T13:24:53.574193+0000","last_active":"2024-04-02T13:24:56.064587+0000","last_peered":"2024-04-02T13:24:56.064587+0000","last_clean":"2024-04-02T13:24:56.064587+0000","last_became_active":"2024-04-02T13:24:53.570565+0000","last_became_peered":"2024-04-02T13:24:53.570565+0000","last_unstale":"2024-04-02T13:24:56.064587+0000","last_undegraded":"2024-04-02T13:24:56.064587+0000","last_fullsized":"2024-04-02T13:24:56.064587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:50:27.793063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574928+0000","last_change":"2024-04-02T13:24:54.577947+0000","last_active":"2024-04-02T13:24:55.574928+0000","last_peered":"2024-04-02T13:24:55.574928+0000","last_clean":"2024-04-02T13:24:55.574928+0000","last_became_active":"2024-04-02T13:24:54.577637+0000","last_became_peered":"2024-04-02T13:24:54.577637+0000","last_unstale":"2024-04-02T13:24:55.574928+0000","last_undegraded":"2024-04-02T13:24:55.574928+0000","last_fullsized":"2024-04-02T13:24:55.574928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:08.394824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574508+0000","last_change":"2024-04-02T13:24:53.567583+0000","last_active":"2024-04-02T13:24:55.574508+0000","last_peered":"2024-04-02T13:24:55.574508+0000","last_clean":"2024-04-02T13:24:55.574508+0000","last_became_active":"2024-04-02T13:24:53.567289+0000","last_became_peered":"2024-04-02T13:24:53.567289+0000","last_unstale":"2024-04-02T13:24:55.574508+0000","last_undegraded":"2024-04-02T13:24:55.574508+0000","last_fullsized":"2024-04-02T13:24:55.574508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:29:00.076440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064704+0000","last_change":"2024-04-02T13:24:54.578014+0000","last_active":"2024-04-02T13:24:56.064704+0000","last_peered":"2024-04-02T13:24:56.064704+0000","last_clean":"2024-04-02T13:24:56.064704+0000","last_became_active":"2024-04-02T13:24:54.577445+0000","last_became_peered":"2024-04-02T13:24:54.577445+0000","last_unstale":"2024-04-02T13:24:56.064704+0000","last_undegraded":"2024-04-02T13:24:56.064704+0000","last_fullsized":"2024-04-02T13:24:56.064704+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:30:34.688997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056408+0000","last_change":"2024-04-02T13:24:53.572298+0000","last_active":"2024-04-02T13:24:56.056408+0000","last_peered":"2024-04-02T13:24:56.056408+0000","last_clean":"2024-04-02T13:24:56.056408+0000","last_became_active":"2024-04-02T13:24:53.572035+0000","last_became_peered":"2024-04-02T13:24:53.572035+0000","last_unstale":"2024-04-02T13:24:56.056408+0000","last_undegraded":"2024-04-02T13:24:56.056408+0000","last_fullsized":"2024-04-02T13:24:56.056408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:35.859707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":282,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064569+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:24:56.064569+0000","last_peered":"2024-04-02T13:24:56.064569+0000","last_clean":"2024-04-02T13:24:56.064569+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:24:56.064569+0000","last_undegraded":"2024-04-02T13:24:56.064569+0000","last_fullsized":"2024-04-02T13:24:56.064569+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574908,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":1031999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":978944,"data_stored":443221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574908,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":441643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574908,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":987136,"data_stored":443325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":442191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":978944,"data_stored":442351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":974848,"data_stored":441721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574908,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1560576,"data_stored":1031977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:25:00.220 INFO:tasks.ceph.mds.d.smithi069.stderr:2024-04-02T13:25:00.217+0000 7f1f76d60640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:25:03.046 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:25:03.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.157+0000 7f9c35847640 1 Processor -- start 2024-04-02T13:25:03.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.157+0000 7f9c35847640 1 -- start start 2024-04-02T13:25:03.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c35847640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c30159db0 0x7f9c3015c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c35847640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c300ccdf0 0x7f9c30165cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c35847640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c300726c0 con 0x7f9c30071e50 2024-04-02T13:25:03.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c35847640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9c30073240 con 0x7f9c300ccdf0 2024-04-02T13:25:03.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c35847640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9c300729a0 con 0x7f9c30159db0 2024-04-02T13:25:03.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c27fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f9c30071e50 0x7f9c300cc6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:45262/0 (socket says 172.21.15.44:45262) 2024-04-02T13:25:03.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c27fff640 1 -- 172.21.15.44:0/3452085260 learned_addr learned my addr 172.21.15.44:0/3452085260 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:03.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c2effd640 1 --2- 172.21.15.44:0/3452085260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c30159db0 0x7f9c3015c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c2f7fe640 1 --2- 172.21.15.44:0/3452085260 >> v2:172.21.15.69:3300/0 conn(0x7f9c300ccdf0 0x7f9c30165cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c2e7fc640 1 -- 172.21.15.44:0/3452085260 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 651810634 0 0) 0x7f9c300726c0 con 0x7f9c30071e50 2024-04-02T13:25:03.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c2e7fc640 1 -- 172.21.15.44:0/3452085260 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c1c003520 con 0x7f9c30071e50 2024-04-02T13:25:03.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c2f7fe640 1 -- 172.21.15.44:0/3452085260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c30159db0 msgr2=0x7f9c3015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:03.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c2f7fe640 1 --2- 172.21.15.44:0/3452085260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c30159db0 0x7f9c3015c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c2f7fe640 1 -- 172.21.15.44:0/3452085260 >> v1:172.21.15.44:6789/0 conn(0x7f9c30071e50 legacy=0x7f9c300cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c2f7fe640 1 -- 172.21.15.44:0/3452085260 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c30072c80 con 0x7f9c300ccdf0 2024-04-02T13:25:03.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.158+0000 7f9c2effd640 1 --2- 172.21.15.44:0/3452085260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c30159db0 0x7f9c3015c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:25:03.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.159+0000 7f9c2f7fe640 1 --2- 172.21.15.44:0/3452085260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c300ccdf0 0x7f9c30165cf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9c300733c0 tx=0x7f9c2802be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32c4fff87beedeb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:03.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.159+0000 7f9c2e7fc640 1 -- 172.21.15.44:0/3452085260 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c28039070 con 0x7f9c300ccdf0 2024-04-02T13:25:03.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.160+0000 7f9c2e7fc640 1 -- 172.21.15.44:0/3452085260 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c2802cbb0 con 0x7f9c300ccdf0 2024-04-02T13:25:03.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.160+0000 7f9c2e7fc640 1 -- 172.21.15.44:0/3452085260 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c28034440 con 0x7f9c300ccdf0 2024-04-02T13:25:03.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.160+0000 7f9c35847640 1 -- 172.21.15.44:0/3452085260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c300ccdf0 msgr2=0x7f9c30165cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.160+0000 7f9c35847640 1 --2- 172.21.15.44:0/3452085260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c300ccdf0 0x7f9c30165cf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9c300733c0 tx=0x7f9c2802be00 comp rx=0 tx=0).stop 2024-04-02T13:25:03.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.160+0000 7f9c35847640 1 -- 172.21.15.44:0/3452085260 shutdown_connections 2024-04-02T13:25:03.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.160+0000 7f9c35847640 1 --2- 172.21.15.44:0/3452085260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c300ccdf0 0x7f9c30165cf0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.160+0000 7f9c35847640 1 --2- 172.21.15.44:0/3452085260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c30159db0 0x7f9c3015c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.160+0000 7f9c35847640 1 -- 172.21.15.44:0/3452085260 >> 172.21.15.44:0/3452085260 conn(0x7f9c300cfcd0 msgr2=0x7f9c300d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:03.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.160+0000 7f9c35847640 1 -- 172.21.15.44:0/3452085260 shutdown_connections 2024-04-02T13:25:03.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.160+0000 7f9c35847640 1 -- 172.21.15.44:0/3452085260 wait complete. 2024-04-02T13:25:03.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.162+0000 7f9c35847640 1 Processor -- start 2024-04-02T13:25:03.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.162+0000 7f9c35847640 1 -- start start 2024-04-02T13:25:03.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c35847640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c30071e50 0x7f9c300b4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c35847640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c300ccdf0 0x7f9c301f4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c35847640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c30159db0 0x7f9c3015ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c35847640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9c30072450 con 0x7f9c30071e50 2024-04-02T13:25:03.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c35847640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9c300722d0 con 0x7f9c300ccdf0 2024-04-02T13:25:03.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c35847640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3006e650 con 0x7f9c30159db0 2024-04-02T13:25:03.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c2effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c30071e50 0x7f9c300b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c2effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c30071e50 0x7f9c300b4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41944/0 (socket says 172.21.15.44:41944) 2024-04-02T13:25:03.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c2effd640 1 -- 172.21.15.44:0/128336168 learned_addr learned my addr 172.21.15.44:0/128336168 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:03.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c27fff640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c300ccdf0 0x7f9c301f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c2f7fe640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c30159db0 0x7f9c3015ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c2effd640 1 -- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c30159db0 msgr2=0x7f9c3015ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c2effd640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c30159db0 0x7f9c3015ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c2effd640 1 -- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c300ccdf0 msgr2=0x7f9c301f4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c2effd640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c300ccdf0 0x7f9c301f4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c2effd640 1 -- 172.21.15.44:0/128336168 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c301f4e60 con 0x7f9c30071e50 2024-04-02T13:25:03.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c27fff640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c300ccdf0 0x7f9c301f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:03.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c2effd640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c30071e50 0x7f9c300b4950 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000e990 tx=0x7f9c2000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:03.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.163+0000 7f9c2cff9640 1 -- 172.21.15.44:0/128336168 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20017070 con 0x7f9c30071e50 2024-04-02T13:25:03.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.164+0000 7f9c2cff9640 1 -- 172.21.15.44:0/128336168 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c2000c9e0 con 0x7f9c30071e50 2024-04-02T13:25:03.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.164+0000 7f9c35847640 1 -- 172.21.15.44:0/128336168 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c300b38b0 con 0x7f9c30071e50 2024-04-02T13:25:03.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.164+0000 7f9c2cff9640 1 -- 172.21.15.44:0/128336168 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2000cdb0 con 0x7f9c30071e50 2024-04-02T13:25:03.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.164+0000 7f9c35847640 1 -- 172.21.15.44:0/128336168 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c300b3e70 con 0x7f9c30071e50 2024-04-02T13:25:03.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.165+0000 7f9c35847640 1 -- 172.21.15.44:0/128336168 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c30074f20 con 0x7f9c30071e50 2024-04-02T13:25:03.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.165+0000 7f9c2cff9640 1 -- 172.21.15.44:0/128336168 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9c2001a070 con 0x7f9c30071e50 2024-04-02T13:25:03.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.165+0000 7f9c2cff9640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c1006c8f0 0x7f9c1006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.166+0000 7f9c2cff9640 1 -- 172.21.15.44:0/128336168 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f9c2008bbc0 con 0x7f9c30071e50 2024-04-02T13:25:03.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.166+0000 7f9c27fff640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c1006c8f0 0x7f9c1006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.166+0000 7f9c27fff640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c1006c8f0 0x7f9c1006edb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9c18007fd0 tx=0x7f9c18009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:03.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.170+0000 7f9c2cff9640 1 -- 172.21.15.44:0/128336168 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9c301f4e60 con 0x7f9c30071e50 2024-04-02T13:25:03.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.351+0000 7f9c35847640 1 -- 172.21.15.44:0/128336168 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9c30155db0 con 0x7f9c1006c8f0 2024-04-02T13:25:03.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.357+0000 7f9c2cff9640 1 -- 172.21.15.44:0/128336168 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252097 (secure 0 0 0) 0x7f9c30155db0 con 0x7f9c1006c8f0 2024-04-02T13:25:03.364 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:03.365 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:25:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.360+0000 7f9c35847640 1 -- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c1006c8f0 msgr2=0x7f9c1006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.360+0000 7f9c35847640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c1006c8f0 0x7f9c1006edb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9c18007fd0 tx=0x7f9c18009040 comp rx=0 tx=0).stop 2024-04-02T13:25:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.360+0000 7f9c35847640 1 -- 172.21.15.44:0/128336168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c30071e50 msgr2=0x7f9c300b4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.360+0000 7f9c35847640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c30071e50 0x7f9c300b4950 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000e990 tx=0x7f9c2000ee50 comp rx=0 tx=0).stop 2024-04-02T13:25:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.360+0000 7f9c35847640 1 -- 172.21.15.44:0/128336168 shutdown_connections 2024-04-02T13:25:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.360+0000 7f9c35847640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c1006c8f0 0x7f9c1006edb0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.360+0000 7f9c35847640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c30159db0 0x7f9c3015ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.360+0000 7f9c35847640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c300ccdf0 0x7f9c301f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.360+0000 7f9c35847640 1 --2- 172.21.15.44:0/128336168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c30071e50 0x7f9c300b4950 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.360+0000 7f9c35847640 1 -- 172.21.15.44:0/128336168 >> 172.21.15.44:0/128336168 conn(0x7f9c300cfcd0 msgr2=0x7f9c300b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.360+0000 7f9c35847640 1 -- 172.21.15.44:0/128336168 shutdown_connections 2024-04-02T13:25:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.360+0000 7f9c35847640 1 -- 172.21.15.44:0/128336168 wait complete. 2024-04-02T13:25:03.374 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-04-02T13:25:02.942666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241600,"kb_used_data":8776,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754733120,"statfs":{"total":773094113280,"available":772846714880,"internally_reserved":0,"allocated":8986624,"data_stored":4718428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.385216"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064800+0000","last_change":"2024-04-02T13:24:54.582190+0000","last_active":"2024-04-02T13:24:56.064800+0000","last_peered":"2024-04-02T13:24:56.064800+0000","last_clean":"2024-04-02T13:24:56.064800+0000","last_became_active":"2024-04-02T13:24:54.581814+0000","last_became_peered":"2024-04-02T13:24:54.581814+0000","last_unstale":"2024-04-02T13:24:56.064800+0000","last_undegraded":"2024-04-02T13:24:56.064800+0000","last_fullsized":"2024-04-02T13:24:56.064800+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:14:01.535902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064308+0000","last_change":"2024-04-02T13:24:54.584115+0000","last_active":"2024-04-02T13:24:56.064308+0000","last_peered":"2024-04-02T13:24:56.064308+0000","last_clean":"2024-04-02T13:24:56.064308+0000","last_became_active":"2024-04-02T13:24:54.583792+0000","last_became_peered":"2024-04-02T13:24:54.583792+0000","last_unstale":"2024-04-02T13:24:56.064308+0000","last_undegraded":"2024-04-02T13:24:56.064308+0000","last_fullsized":"2024-04-02T13:24:56.064308+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:30:34.688997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574787+0000","last_change":"2024-04-02T13:24:54.574235+0000","last_active":"2024-04-02T13:24:55.574787+0000","last_peered":"2024-04-02T13:24:55.574787+0000","last_clean":"2024-04-02T13:24:55.574787+0000","last_became_active":"2024-04-02T13:24:54.573908+0000","last_became_peered":"2024-04-02T13:24:54.573908+0000","last_unstale":"2024-04-02T13:24:55.574787+0000","last_undegraded":"2024-04-02T13:24:55.574787+0000","last_fullsized":"2024-04-02T13:24:55.574787+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:18:21.848167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574407+0000","last_change":"2024-04-02T13:24:54.581284+0000","last_active":"2024-04-02T13:24:55.574407+0000","last_peered":"2024-04-02T13:24:55.574407+0000","last_clean":"2024-04-02T13:24:55.574407+0000","last_became_active":"2024-04-02T13:24:54.580976+0000","last_became_peered":"2024-04-02T13:24:54.580976+0000","last_unstale":"2024-04-02T13:24:55.574407+0000","last_undegraded":"2024-04-02T13:24:55.574407+0000","last_fullsized":"2024-04-02T13:24:55.574407+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:25.789065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064080+0000","last_change":"2024-04-02T13:24:54.580261+0000","last_active":"2024-04-02T13:24:56.064080+0000","last_peered":"2024-04-02T13:24:56.064080+0000","last_clean":"2024-04-02T13:24:56.064080+0000","last_became_active":"2024-04-02T13:24:54.579973+0000","last_became_peered":"2024-04-02T13:24:54.579973+0000","last_unstale":"2024-04-02T13:24:56.064080+0000","last_undegraded":"2024-04-02T13:24:56.064080+0000","last_fullsized":"2024-04-02T13:24:56.064080+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:43:00.164244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056185+0000","last_change":"2024-04-02T13:24:54.585758+0000","last_active":"2024-04-02T13:24:56.056185+0000","last_peered":"2024-04-02T13:24:56.056185+0000","last_clean":"2024-04-02T13:24:56.056185+0000","last_became_active":"2024-04-02T13:24:54.585490+0000","last_became_peered":"2024-04-02T13:24:54.585490+0000","last_unstale":"2024-04-02T13:24:56.056185+0000","last_undegraded":"2024-04-02T13:24:56.056185+0000","last_fullsized":"2024-04-02T13:24:56.056185+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:16.381648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064052+0000","last_change":"2024-04-02T13:24:54.577884+0000","last_active":"2024-04-02T13:24:56.064052+0000","last_peered":"2024-04-02T13:24:56.064052+0000","last_clean":"2024-04-02T13:24:56.064052+0000","last_became_active":"2024-04-02T13:24:54.577328+0000","last_became_peered":"2024-04-02T13:24:54.577328+0000","last_unstale":"2024-04-02T13:24:56.064052+0000","last_undegraded":"2024-04-02T13:24:56.064052+0000","last_fullsized":"2024-04-02T13:24:56.064052+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:48:44.079142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063748+0000","last_change":"2024-04-02T13:24:54.579426+0000","last_active":"2024-04-02T13:24:56.063748+0000","last_peered":"2024-04-02T13:24:56.063748+0000","last_clean":"2024-04-02T13:24:56.063748+0000","last_became_active":"2024-04-02T13:24:54.579147+0000","last_became_peered":"2024-04-02T13:24:54.579147+0000","last_unstale":"2024-04-02T13:24:56.063748+0000","last_undegraded":"2024-04-02T13:24:56.063748+0000","last_fullsized":"2024-04-02T13:24:56.063748+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:49:38.705972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574247+0000","last_change":"2024-04-02T13:24:54.587257+0000","last_active":"2024-04-02T13:24:55.574247+0000","last_peered":"2024-04-02T13:24:55.574247+0000","last_clean":"2024-04-02T13:24:55.574247+0000","last_became_active":"2024-04-02T13:24:54.586571+0000","last_became_peered":"2024-04-02T13:24:54.586571+0000","last_unstale":"2024-04-02T13:24:55.574247+0000","last_undegraded":"2024-04-02T13:24:55.574247+0000","last_fullsized":"2024-04-02T13:24:55.574247+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:43:27.497564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573950+0000","last_change":"2024-04-02T13:24:54.570536+0000","last_active":"2024-04-02T13:24:55.573950+0000","last_peered":"2024-04-02T13:24:55.573950+0000","last_clean":"2024-04-02T13:24:55.573950+0000","last_became_active":"2024-04-02T13:24:54.570264+0000","last_became_peered":"2024-04-02T13:24:54.570264+0000","last_unstale":"2024-04-02T13:24:55.573950+0000","last_undegraded":"2024-04-02T13:24:55.573950+0000","last_fullsized":"2024-04-02T13:24:55.573950+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:58:28.798683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064238+0000","last_change":"2024-04-02T13:24:54.584742+0000","last_active":"2024-04-02T13:24:56.064238+0000","last_peered":"2024-04-02T13:24:56.064238+0000","last_clean":"2024-04-02T13:24:56.064238+0000","last_became_active":"2024-04-02T13:24:54.584094+0000","last_became_peered":"2024-04-02T13:24:54.584094+0000","last_unstale":"2024-04-02T13:24:56.064238+0000","last_undegraded":"2024-04-02T13:24:56.064238+0000","last_fullsized":"2024-04-02T13:24:56.064238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:16.445368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574563+0000","last_change":"2024-04-02T13:24:54.574509+0000","last_active":"2024-04-02T13:24:55.574563+0000","last_peered":"2024-04-02T13:24:55.574563+0000","last_clean":"2024-04-02T13:24:55.574563+0000","last_became_active":"2024-04-02T13:24:54.574199+0000","last_became_peered":"2024-04-02T13:24:54.574199+0000","last_unstale":"2024-04-02T13:24:55.574563+0000","last_undegraded":"2024-04-02T13:24:55.574563+0000","last_fullsized":"2024-04-02T13:24:55.574563+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:48:23.402189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064197+0000","last_change":"2024-04-02T13:24:54.586211+0000","last_active":"2024-04-02T13:24:56.064197+0000","last_peered":"2024-04-02T13:24:56.064197+0000","last_clean":"2024-04-02T13:24:56.064197+0000","last_became_active":"2024-04-02T13:24:54.585863+0000","last_became_peered":"2024-04-02T13:24:54.585863+0000","last_unstale":"2024-04-02T13:24:56.064197+0000","last_undegraded":"2024-04-02T13:24:56.064197+0000","last_fullsized":"2024-04-02T13:24:56.064197+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:15:41.434299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056212+0000","last_change":"2024-04-02T13:24:54.584568+0000","last_active":"2024-04-02T13:24:56.056212+0000","last_peered":"2024-04-02T13:24:56.056212+0000","last_clean":"2024-04-02T13:24:56.056212+0000","last_became_active":"2024-04-02T13:24:54.583120+0000","last_became_peered":"2024-04-02T13:24:54.583120+0000","last_unstale":"2024-04-02T13:24:56.056212+0000","last_undegraded":"2024-04-02T13:24:56.056212+0000","last_fullsized":"2024-04-02T13:24:56.056212+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:08.440154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063782+0000","last_change":"2024-04-02T13:24:54.576524+0000","last_active":"2024-04-02T13:24:56.063782+0000","last_peered":"2024-04-02T13:24:56.063782+0000","last_clean":"2024-04-02T13:24:56.063782+0000","last_became_active":"2024-04-02T13:24:54.576151+0000","last_became_peered":"2024-04-02T13:24:54.576151+0000","last_unstale":"2024-04-02T13:24:56.063782+0000","last_undegraded":"2024-04-02T13:24:56.063782+0000","last_fullsized":"2024-04-02T13:24:56.063782+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:24:50.774098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064257+0000","last_change":"2024-04-02T13:24:54.568847+0000","last_active":"2024-04-02T13:24:56.064257+0000","last_peered":"2024-04-02T13:24:56.064257+0000","last_clean":"2024-04-02T13:24:56.064257+0000","last_became_active":"2024-04-02T13:24:54.568551+0000","last_became_peered":"2024-04-02T13:24:54.568551+0000","last_unstale":"2024-04-02T13:24:56.064257+0000","last_undegraded":"2024-04-02T13:24:56.064257+0000","last_fullsized":"2024-04-02T13:24:56.064257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:24:50.774098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064735+0000","last_change":"2024-04-02T13:24:54.580503+0000","last_active":"2024-04-02T13:24:56.064735+0000","last_peered":"2024-04-02T13:24:56.064735+0000","last_clean":"2024-04-02T13:24:56.064735+0000","last_became_active":"2024-04-02T13:24:54.580207+0000","last_became_peered":"2024-04-02T13:24:54.580207+0000","last_unstale":"2024-04-02T13:24:56.064735+0000","last_undegraded":"2024-04-02T13:24:56.064735+0000","last_fullsized":"2024-04-02T13:24:56.064735+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:18:08.897243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574697+0000","last_change":"2024-04-02T13:24:54.582497+0000","last_active":"2024-04-02T13:24:55.574697+0000","last_peered":"2024-04-02T13:24:55.574697+0000","last_clean":"2024-04-02T13:24:55.574697+0000","last_became_active":"2024-04-02T13:24:54.582221+0000","last_became_peered":"2024-04-02T13:24:54.582221+0000","last_unstale":"2024-04-02T13:24:55.574697+0000","last_undegraded":"2024-04-02T13:24:55.574697+0000","last_fullsized":"2024-04-02T13:24:55.574697+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:30:37.463997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056042+0000","last_change":"2024-04-02T13:24:54.586251+0000","last_active":"2024-04-02T13:24:56.056042+0000","last_peered":"2024-04-02T13:24:56.056042+0000","last_clean":"2024-04-02T13:24:56.056042+0000","last_became_active":"2024-04-02T13:24:54.585975+0000","last_became_peered":"2024-04-02T13:24:54.585975+0000","last_unstale":"2024-04-02T13:24:56.056042+0000","last_undegraded":"2024-04-02T13:24:56.056042+0000","last_fullsized":"2024-04-02T13:24:56.056042+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:03.634266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064056+0000","last_change":"2024-04-02T13:24:54.583263+0000","last_active":"2024-04-02T13:24:56.064056+0000","last_peered":"2024-04-02T13:24:56.064056+0000","last_clean":"2024-04-02T13:24:56.064056+0000","last_became_active":"2024-04-02T13:24:54.582968+0000","last_became_peered":"2024-04-02T13:24:54.582968+0000","last_unstale":"2024-04-02T13:24:56.064056+0000","last_undegraded":"2024-04-02T13:24:56.064056+0000","last_fullsized":"2024-04-02T13:24:56.064056+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:00:37.401118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574383+0000","last_change":"2024-04-02T13:24:54.580613+0000","last_active":"2024-04-02T13:24:55.574383+0000","last_peered":"2024-04-02T13:24:55.574383+0000","last_clean":"2024-04-02T13:24:55.574383+0000","last_became_active":"2024-04-02T13:24:54.580356+0000","last_became_peered":"2024-04-02T13:24:54.580356+0000","last_unstale":"2024-04-02T13:24:55.574383+0000","last_undegraded":"2024-04-02T13:24:55.574383+0000","last_fullsized":"2024-04-02T13:24:55.574383+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:50:28.795387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064138+0000","last_change":"2024-04-02T13:24:54.586237+0000","last_active":"2024-04-02T13:24:56.064138+0000","last_peered":"2024-04-02T13:24:56.064138+0000","last_clean":"2024-04-02T13:24:56.064138+0000","last_became_active":"2024-04-02T13:24:54.585910+0000","last_became_peered":"2024-04-02T13:24:54.585910+0000","last_unstale":"2024-04-02T13:24:56.064138+0000","last_undegraded":"2024-04-02T13:24:56.064138+0000","last_fullsized":"2024-04-02T13:24:56.064138+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:02:06.412550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574181+0000","last_change":"2024-04-02T13:24:54.573990+0000","last_active":"2024-04-02T13:24:55.574181+0000","last_peered":"2024-04-02T13:24:55.574181+0000","last_clean":"2024-04-02T13:24:55.574181+0000","last_became_active":"2024-04-02T13:24:54.573671+0000","last_became_peered":"2024-04-02T13:24:54.573671+0000","last_unstale":"2024-04-02T13:24:55.574181+0000","last_undegraded":"2024-04-02T13:24:55.574181+0000","last_fullsized":"2024-04-02T13:24:55.574181+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:29:58.020569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063647+0000","last_change":"2024-04-02T13:24:54.589275+0000","last_active":"2024-04-02T13:24:56.063647+0000","last_peered":"2024-04-02T13:24:56.063647+0000","last_clean":"2024-04-02T13:24:56.063647+0000","last_became_active":"2024-04-02T13:24:54.589010+0000","last_became_peered":"2024-04-02T13:24:54.589010+0000","last_unstale":"2024-04-02T13:24:56.063647+0000","last_undegraded":"2024-04-02T13:24:56.063647+0000","last_fullsized":"2024-04-02T13:24:56.063647+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:10:41.126700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064560+0000","last_change":"2024-04-02T13:24:54.590378+0000","last_active":"2024-04-02T13:24:56.064560+0000","last_peered":"2024-04-02T13:24:56.064560+0000","last_clean":"2024-04-02T13:24:56.064560+0000","last_became_active":"2024-04-02T13:24:54.589691+0000","last_became_peered":"2024-04-02T13:24:54.589691+0000","last_unstale":"2024-04-02T13:24:56.064560+0000","last_undegraded":"2024-04-02T13:24:56.064560+0000","last_fullsized":"2024-04-02T13:24:56.064560+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:16.381648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574149+0000","last_change":"2024-04-02T13:24:54.575310+0000","last_active":"2024-04-02T13:24:55.574149+0000","last_peered":"2024-04-02T13:24:55.574149+0000","last_clean":"2024-04-02T13:24:55.574149+0000","last_became_active":"2024-04-02T13:24:54.575029+0000","last_became_peered":"2024-04-02T13:24:54.575029+0000","last_unstale":"2024-04-02T13:24:55.574149+0000","last_undegraded":"2024-04-02T13:24:55.574149+0000","last_fullsized":"2024-04-02T13:24:55.574149+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:12.654707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064538+0000","last_change":"2024-04-02T13:24:54.572607+0000","last_active":"2024-04-02T13:24:56.064538+0000","last_peered":"2024-04-02T13:24:56.064538+0000","last_clean":"2024-04-02T13:24:56.064538+0000","last_became_active":"2024-04-02T13:24:54.571815+0000","last_became_peered":"2024-04-02T13:24:54.571815+0000","last_unstale":"2024-04-02T13:24:56.064538+0000","last_undegraded":"2024-04-02T13:24:56.064538+0000","last_fullsized":"2024-04-02T13:24:56.064538+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:16.445368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064414+0000","last_change":"2024-04-02T13:24:54.566761+0000","last_active":"2024-04-02T13:24:56.064414+0000","last_peered":"2024-04-02T13:24:56.064414+0000","last_clean":"2024-04-02T13:24:56.064414+0000","last_became_active":"2024-04-02T13:24:54.566488+0000","last_became_peered":"2024-04-02T13:24:54.566488+0000","last_unstale":"2024-04-02T13:24:56.064414+0000","last_undegraded":"2024-04-02T13:24:56.064414+0000","last_fullsized":"2024-04-02T13:24:56.064414+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:14:10.643231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064315+0000","last_change":"2024-04-02T13:24:54.588347+0000","last_active":"2024-04-02T13:24:56.064315+0000","last_peered":"2024-04-02T13:24:56.064315+0000","last_clean":"2024-04-02T13:24:56.064315+0000","last_became_active":"2024-04-02T13:24:54.588078+0000","last_became_peered":"2024-04-02T13:24:54.588078+0000","last_unstale":"2024-04-02T13:24:56.064315+0000","last_undegraded":"2024-04-02T13:24:56.064315+0000","last_fullsized":"2024-04-02T13:24:56.064315+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:14:01.535902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064151+0000","last_change":"2024-04-02T13:24:54.571000+0000","last_active":"2024-04-02T13:24:56.064151+0000","last_peered":"2024-04-02T13:24:56.064151+0000","last_clean":"2024-04-02T13:24:56.064151+0000","last_became_active":"2024-04-02T13:24:54.570722+0000","last_became_peered":"2024-04-02T13:24:54.570722+0000","last_unstale":"2024-04-02T13:24:56.064151+0000","last_undegraded":"2024-04-02T13:24:56.064151+0000","last_fullsized":"2024-04-02T13:24:56.064151+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:42:33.303173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574577+0000","last_change":"2024-04-02T13:24:54.582308+0000","last_active":"2024-04-02T13:24:55.574577+0000","last_peered":"2024-04-02T13:24:55.574577+0000","last_clean":"2024-04-02T13:24:55.574577+0000","last_became_active":"2024-04-02T13:24:54.582030+0000","last_became_peered":"2024-04-02T13:24:54.582030+0000","last_unstale":"2024-04-02T13:24:55.574577+0000","last_undegraded":"2024-04-02T13:24:55.574577+0000","last_fullsized":"2024-04-02T13:24:55.574577+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:53:36.691313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064449+0000","last_change":"2024-04-02T13:24:54.577125+0000","last_active":"2024-04-02T13:24:56.064449+0000","last_peered":"2024-04-02T13:24:56.064449+0000","last_clean":"2024-04-02T13:24:56.064449+0000","last_became_active":"2024-04-02T13:24:54.576839+0000","last_became_peered":"2024-04-02T13:24:54.576839+0000","last_unstale":"2024-04-02T13:24:56.064449+0000","last_undegraded":"2024-04-02T13:24:56.064449+0000","last_fullsized":"2024-04-02T13:24:56.064449+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:59.553628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574571+0000","last_change":"2024-04-02T13:24:54.576466+0000","last_active":"2024-04-02T13:24:55.574571+0000","last_peered":"2024-04-02T13:24:55.574571+0000","last_clean":"2024-04-02T13:24:55.574571+0000","last_became_active":"2024-04-02T13:24:54.576195+0000","last_became_peered":"2024-04-02T13:24:54.576195+0000","last_unstale":"2024-04-02T13:24:55.574571+0000","last_undegraded":"2024-04-02T13:24:55.574571+0000","last_fullsized":"2024-04-02T13:24:55.574571+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:43:11.976047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"78'4","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064703+0000","last_change":"2024-04-02T13:24:53.573023+0000","last_active":"2024-04-02T13:24:56.064703+0000","last_peered":"2024-04-02T13:24:56.064703+0000","last_clean":"2024-04-02T13:24:56.064703+0000","last_became_active":"2024-04-02T13:24:53.571627+0000","last_became_peered":"2024-04-02T13:24:53.571627+0000","last_unstale":"2024-04-02T13:24:56.064703+0000","last_undegraded":"2024-04-02T13:24:56.064703+0000","last_fullsized":"2024-04-02T13:24:56.064703+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:05:17.417576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063953+0000","last_change":"2024-04-02T13:24:54.572282+0000","last_active":"2024-04-02T13:24:56.063953+0000","last_peered":"2024-04-02T13:24:56.063953+0000","last_clean":"2024-04-02T13:24:56.063953+0000","last_became_active":"2024-04-02T13:24:54.572000+0000","last_became_peered":"2024-04-02T13:24:54.572000+0000","last_unstale":"2024-04-02T13:24:56.063953+0000","last_undegraded":"2024-04-02T13:24:56.063953+0000","last_fullsized":"2024-04-02T13:24:56.063953+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:59.553628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064680+0000","last_change":"2024-04-02T13:24:53.573555+0000","last_active":"2024-04-02T13:24:56.064680+0000","last_peered":"2024-04-02T13:24:56.064680+0000","last_clean":"2024-04-02T13:24:56.064680+0000","last_became_active":"2024-04-02T13:24:53.573158+0000","last_became_peered":"2024-04-02T13:24:53.573158+0000","last_unstale":"2024-04-02T13:24:56.064680+0000","last_undegraded":"2024-04-02T13:24:56.064680+0000","last_fullsized":"2024-04-02T13:24:56.064680+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:58:27.796359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574630+0000","last_change":"2024-04-02T13:24:54.576685+0000","last_active":"2024-04-02T13:24:55.574630+0000","last_peered":"2024-04-02T13:24:55.574630+0000","last_clean":"2024-04-02T13:24:55.574630+0000","last_became_active":"2024-04-02T13:24:54.576321+0000","last_became_peered":"2024-04-02T13:24:54.576321+0000","last_unstale":"2024-04-02T13:24:55.574630+0000","last_undegraded":"2024-04-02T13:24:55.574630+0000","last_fullsized":"2024-04-02T13:24:55.574630+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:27.408601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064612+0000","last_change":"2024-04-02T13:24:53.578788+0000","last_active":"2024-04-02T13:24:56.064612+0000","last_peered":"2024-04-02T13:24:56.064612+0000","last_clean":"2024-04-02T13:24:56.064612+0000","last_became_active":"2024-04-02T13:24:53.578523+0000","last_became_peered":"2024-04-02T13:24:53.578523+0000","last_unstale":"2024-04-02T13:24:56.064612+0000","last_undegraded":"2024-04-02T13:24:56.064612+0000","last_fullsized":"2024-04-02T13:24:56.064612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:22:06.708202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574544+0000","last_change":"2024-04-02T13:24:54.580439+0000","last_active":"2024-04-02T13:24:55.574544+0000","last_peered":"2024-04-02T13:24:55.574544+0000","last_clean":"2024-04-02T13:24:55.574544+0000","last_became_active":"2024-04-02T13:24:54.579623+0000","last_became_peered":"2024-04-02T13:24:54.579623+0000","last_unstale":"2024-04-02T13:24:55.574544+0000","last_undegraded":"2024-04-02T13:24:55.574544+0000","last_fullsized":"2024-04-02T13:24:55.574544+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:48:59.714798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064200+0000","last_change":"2024-04-02T13:24:53.575848+0000","last_active":"2024-04-02T13:24:56.064200+0000","last_peered":"2024-04-02T13:24:56.064200+0000","last_clean":"2024-04-02T13:24:56.064200+0000","last_became_active":"2024-04-02T13:24:53.574537+0000","last_became_peered":"2024-04-02T13:24:53.574537+0000","last_unstale":"2024-04-02T13:24:56.064200+0000","last_undegraded":"2024-04-02T13:24:56.064200+0000","last_fullsized":"2024-04-02T13:24:56.064200+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:02:30.957187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064736+0000","last_change":"2024-04-02T13:24:54.583494+0000","last_active":"2024-04-02T13:24:56.064736+0000","last_peered":"2024-04-02T13:24:56.064736+0000","last_clean":"2024-04-02T13:24:56.064736+0000","last_became_active":"2024-04-02T13:24:54.580371+0000","last_became_peered":"2024-04-02T13:24:54.580371+0000","last_unstale":"2024-04-02T13:24:56.064736+0000","last_undegraded":"2024-04-02T13:24:56.064736+0000","last_fullsized":"2024-04-02T13:24:56.064736+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:03.634266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574609+0000","last_change":"2024-04-02T13:24:53.574172+0000","last_active":"2024-04-02T13:24:55.574609+0000","last_peered":"2024-04-02T13:24:55.574609+0000","last_clean":"2024-04-02T13:24:55.574609+0000","last_became_active":"2024-04-02T13:24:53.573883+0000","last_became_peered":"2024-04-02T13:24:53.573883+0000","last_unstale":"2024-04-02T13:24:55.574609+0000","last_undegraded":"2024-04-02T13:24:55.574609+0000","last_fullsized":"2024-04-02T13:24:55.574609+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:08.877638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064443+0000","last_change":"2024-04-02T13:24:54.585871+0000","last_active":"2024-04-02T13:24:56.064443+0000","last_peered":"2024-04-02T13:24:56.064443+0000","last_clean":"2024-04-02T13:24:56.064443+0000","last_became_active":"2024-04-02T13:24:54.585586+0000","last_became_peered":"2024-04-02T13:24:54.585586+0000","last_unstale":"2024-04-02T13:24:56.064443+0000","last_undegraded":"2024-04-02T13:24:56.064443+0000","last_fullsized":"2024-04-02T13:24:56.064443+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:04:44.744456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573845+0000","last_change":"2024-04-02T13:24:53.571030+0000","last_active":"2024-04-02T13:24:55.573845+0000","last_peered":"2024-04-02T13:24:55.573845+0000","last_clean":"2024-04-02T13:24:55.573845+0000","last_became_active":"2024-04-02T13:24:53.570763+0000","last_became_peered":"2024-04-02T13:24:53.570763+0000","last_unstale":"2024-04-02T13:24:55.573845+0000","last_undegraded":"2024-04-02T13:24:55.573845+0000","last_fullsized":"2024-04-02T13:24:55.573845+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:07.392500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574899+0000","last_change":"2024-04-02T13:24:54.586818+0000","last_active":"2024-04-02T13:24:55.574899+0000","last_peered":"2024-04-02T13:24:55.574899+0000","last_clean":"2024-04-02T13:24:55.574899+0000","last_became_active":"2024-04-02T13:24:54.586528+0000","last_became_peered":"2024-04-02T13:24:54.586528+0000","last_unstale":"2024-04-02T13:24:55.574899+0000","last_undegraded":"2024-04-02T13:24:55.574899+0000","last_fullsized":"2024-04-02T13:24:55.574899+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:13:07.461166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064057+0000","last_change":"2024-04-02T13:24:53.578249+0000","last_active":"2024-04-02T13:24:56.064057+0000","last_peered":"2024-04-02T13:24:56.064057+0000","last_clean":"2024-04-02T13:24:56.064057+0000","last_became_active":"2024-04-02T13:24:53.577992+0000","last_became_peered":"2024-04-02T13:24:53.577992+0000","last_unstale":"2024-04-02T13:24:56.064057+0000","last_undegraded":"2024-04-02T13:24:56.064057+0000","last_fullsized":"2024-04-02T13:24:56.064057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:23:42.359046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064672+0000","last_change":"2024-04-02T13:24:54.578694+0000","last_active":"2024-04-02T13:24:56.064672+0000","last_peered":"2024-04-02T13:24:56.064672+0000","last_clean":"2024-04-02T13:24:56.064672+0000","last_became_active":"2024-04-02T13:24:54.578394+0000","last_became_peered":"2024-04-02T13:24:54.578394+0000","last_unstale":"2024-04-02T13:24:56.064672+0000","last_undegraded":"2024-04-02T13:24:56.064672+0000","last_fullsized":"2024-04-02T13:24:56.064672+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:07.594969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063737+0000","last_change":"2024-04-02T13:24:53.572838+0000","last_active":"2024-04-02T13:24:56.063737+0000","last_peered":"2024-04-02T13:24:56.063737+0000","last_clean":"2024-04-02T13:24:56.063737+0000","last_became_active":"2024-04-02T13:24:53.571893+0000","last_became_peered":"2024-04-02T13:24:53.571893+0000","last_unstale":"2024-04-02T13:24:56.063737+0000","last_undegraded":"2024-04-02T13:24:56.063737+0000","last_fullsized":"2024-04-02T13:24:56.063737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:06:01.172766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063725+0000","last_change":"2024-04-02T13:24:54.579310+0000","last_active":"2024-04-02T13:24:56.063725+0000","last_peered":"2024-04-02T13:24:56.063725+0000","last_clean":"2024-04-02T13:24:56.063725+0000","last_became_active":"2024-04-02T13:24:54.578356+0000","last_became_peered":"2024-04-02T13:24:54.578356+0000","last_unstale":"2024-04-02T13:24:56.063725+0000","last_undegraded":"2024-04-02T13:24:56.063725+0000","last_fullsized":"2024-04-02T13:24:56.063725+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:42:33.303173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064892+0000","last_change":"2024-04-02T13:24:53.579943+0000","last_active":"2024-04-02T13:24:56.064892+0000","last_peered":"2024-04-02T13:24:56.064892+0000","last_clean":"2024-04-02T13:24:56.064892+0000","last_became_active":"2024-04-02T13:24:53.579632+0000","last_became_peered":"2024-04-02T13:24:53.579632+0000","last_unstale":"2024-04-02T13:24:56.064892+0000","last_undegraded":"2024-04-02T13:24:56.064892+0000","last_fullsized":"2024-04-02T13:24:56.064892+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:48:22.399865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573933+0000","last_change":"2024-04-02T13:24:54.580573+0000","last_active":"2024-04-02T13:24:55.573933+0000","last_peered":"2024-04-02T13:24:55.573933+0000","last_clean":"2024-04-02T13:24:55.573933+0000","last_became_active":"2024-04-02T13:24:54.580323+0000","last_became_peered":"2024-04-02T13:24:54.580323+0000","last_unstale":"2024-04-02T13:24:55.573933+0000","last_undegraded":"2024-04-02T13:24:55.573933+0000","last_fullsized":"2024-04-02T13:24:55.573933+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:43:43.130201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574105+0000","last_change":"2024-04-02T13:24:53.570841+0000","last_active":"2024-04-02T13:24:55.574105+0000","last_peered":"2024-04-02T13:24:55.574105+0000","last_clean":"2024-04-02T13:24:55.574105+0000","last_became_active":"2024-04-02T13:24:53.570567+0000","last_became_peered":"2024-04-02T13:24:53.570567+0000","last_unstale":"2024-04-02T13:24:55.574105+0000","last_undegraded":"2024-04-02T13:24:55.574105+0000","last_fullsized":"2024-04-02T13:24:55.574105+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:51.244520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056346+0000","last_change":"2024-04-02T13:24:54.572495+0000","last_active":"2024-04-02T13:24:56.056346+0000","last_peered":"2024-04-02T13:24:56.056346+0000","last_clean":"2024-04-02T13:24:56.056346+0000","last_became_active":"2024-04-02T13:24:54.572181+0000","last_became_peered":"2024-04-02T13:24:54.572181+0000","last_unstale":"2024-04-02T13:24:56.056346+0000","last_undegraded":"2024-04-02T13:24:56.056346+0000","last_fullsized":"2024-04-02T13:24:56.056346+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:34:15.079374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"78'3","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064694+0000","last_change":"2024-04-02T13:24:53.577054+0000","last_active":"2024-04-02T13:24:56.064694+0000","last_peered":"2024-04-02T13:24:56.064694+0000","last_clean":"2024-04-02T13:24:56.064694+0000","last_became_active":"2024-04-02T13:24:53.575955+0000","last_became_peered":"2024-04-02T13:24:53.575955+0000","last_unstale":"2024-04-02T13:24:56.064694+0000","last_undegraded":"2024-04-02T13:24:56.064694+0000","last_fullsized":"2024-04-02T13:24:56.064694+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:16:43.448093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574776+0000","last_change":"2024-04-02T13:24:54.572847+0000","last_active":"2024-04-02T13:24:55.574776+0000","last_peered":"2024-04-02T13:24:55.574776+0000","last_clean":"2024-04-02T13:24:55.574776+0000","last_became_active":"2024-04-02T13:24:54.572588+0000","last_became_peered":"2024-04-02T13:24:54.572588+0000","last_unstale":"2024-04-02T13:24:55.574776+0000","last_undegraded":"2024-04-02T13:24:55.574776+0000","last_fullsized":"2024-04-02T13:24:55.574776+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:09:55.531393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"78'6","reported_seq":20,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064310+0000","last_change":"2024-04-02T13:24:53.575781+0000","last_active":"2024-04-02T13:24:56.064310+0000","last_peered":"2024-04-02T13:24:56.064310+0000","last_clean":"2024-04-02T13:24:56.064310+0000","last_became_active":"2024-04-02T13:24:53.574619+0000","last_became_peered":"2024-04-02T13:24:53.574619+0000","last_unstale":"2024-04-02T13:24:56.064310+0000","last_undegraded":"2024-04-02T13:24:56.064310+0000","last_fullsized":"2024-04-02T13:24:56.064310+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:29:57.018246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574715+0000","last_change":"2024-04-02T13:24:54.568437+0000","last_active":"2024-04-02T13:24:55.574715+0000","last_peered":"2024-04-02T13:24:55.574715+0000","last_clean":"2024-04-02T13:24:55.574715+0000","last_became_active":"2024-04-02T13:24:54.568101+0000","last_became_peered":"2024-04-02T13:24:54.568101+0000","last_unstale":"2024-04-02T13:24:55.574715+0000","last_undegraded":"2024-04-02T13:24:55.574715+0000","last_fullsized":"2024-04-02T13:24:55.574715+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:07:26.224860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573889+0000","last_change":"2024-04-02T13:24:53.572426+0000","last_active":"2024-04-02T13:24:55.573889+0000","last_peered":"2024-04-02T13:24:55.573889+0000","last_clean":"2024-04-02T13:24:55.573889+0000","last_became_active":"2024-04-02T13:24:53.572153+0000","last_became_peered":"2024-04-02T13:24:53.572153+0000","last_unstale":"2024-04-02T13:24:55.573889+0000","last_undegraded":"2024-04-02T13:24:55.573889+0000","last_fullsized":"2024-04-02T13:24:55.573889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:57:16.242492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064499+0000","last_change":"2024-04-02T13:24:54.578234+0000","last_active":"2024-04-02T13:24:56.064499+0000","last_peered":"2024-04-02T13:24:56.064499+0000","last_clean":"2024-04-02T13:24:56.064499+0000","last_became_active":"2024-04-02T13:24:54.577914+0000","last_became_peered":"2024-04-02T13:24:54.577914+0000","last_unstale":"2024-04-02T13:24:56.064499+0000","last_undegraded":"2024-04-02T13:24:56.064499+0000","last_fullsized":"2024-04-02T13:24:56.064499+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:52.246844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064325+0000","last_change":"2024-04-02T13:24:53.580278+0000","last_active":"2024-04-02T13:24:56.064325+0000","last_peered":"2024-04-02T13:24:56.064325+0000","last_clean":"2024-04-02T13:24:56.064325+0000","last_became_active":"2024-04-02T13:24:53.580002+0000","last_became_peered":"2024-04-02T13:24:53.580002+0000","last_unstale":"2024-04-02T13:24:56.064325+0000","last_undegraded":"2024-04-02T13:24:56.064325+0000","last_fullsized":"2024-04-02T13:24:56.064325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:24.786741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064483+0000","last_change":"2024-04-02T13:24:54.579646+0000","last_active":"2024-04-02T13:24:56.064483+0000","last_peered":"2024-04-02T13:24:56.064483+0000","last_clean":"2024-04-02T13:24:56.064483+0000","last_became_active":"2024-04-02T13:24:54.579360+0000","last_became_peered":"2024-04-02T13:24:54.579360+0000","last_unstale":"2024-04-02T13:24:56.064483+0000","last_undegraded":"2024-04-02T13:24:56.064483+0000","last_fullsized":"2024-04-02T13:24:56.064483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:39:31.964323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063999+0000","last_change":"2024-04-02T13:24:53.576553+0000","last_active":"2024-04-02T13:24:56.063999+0000","last_peered":"2024-04-02T13:24:56.063999+0000","last_clean":"2024-04-02T13:24:56.063999+0000","last_became_active":"2024-04-02T13:24:53.576219+0000","last_became_peered":"2024-04-02T13:24:53.576219+0000","last_unstale":"2024-04-02T13:24:56.063999+0000","last_undegraded":"2024-04-02T13:24:56.063999+0000","last_fullsized":"2024-04-02T13:24:56.063999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:12:36.353661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064755+0000","last_change":"2024-04-02T13:24:54.584841+0000","last_active":"2024-04-02T13:24:56.064755+0000","last_peered":"2024-04-02T13:24:56.064755+0000","last_clean":"2024-04-02T13:24:56.064755+0000","last_became_active":"2024-04-02T13:24:54.584565+0000","last_became_peered":"2024-04-02T13:24:54.584565+0000","last_unstale":"2024-04-02T13:24:56.064755+0000","last_undegraded":"2024-04-02T13:24:56.064755+0000","last_fullsized":"2024-04-02T13:24:56.064755+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:23:43.361370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"78'8","reported_seq":22,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.063758+0000","last_change":"2024-04-02T13:24:53.575318+0000","last_active":"2024-04-02T13:24:56.063758+0000","last_peered":"2024-04-02T13:24:56.063758+0000","last_clean":"2024-04-02T13:24:56.063758+0000","last_became_active":"2024-04-02T13:24:53.575019+0000","last_became_peered":"2024-04-02T13:24:53.575019+0000","last_unstale":"2024-04-02T13:24:56.063758+0000","last_undegraded":"2024-04-02T13:24:56.063758+0000","last_fullsized":"2024-04-02T13:24:56.063758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:06.592645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.055822+0000","last_change":"2024-04-02T13:24:54.587605+0000","last_active":"2024-04-02T13:24:56.055822+0000","last_peered":"2024-04-02T13:24:56.055822+0000","last_clean":"2024-04-02T13:24:56.055822+0000","last_became_active":"2024-04-02T13:24:54.587310+0000","last_became_peered":"2024-04-02T13:24:54.587310+0000","last_unstale":"2024-04-02T13:24:56.055822+0000","last_undegraded":"2024-04-02T13:24:56.055822+0000","last_fullsized":"2024-04-02T13:24:56.055822+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:04:44.744456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064915+0000","last_change":"2024-04-02T13:24:53.577390+0000","last_active":"2024-04-02T13:24:56.064915+0000","last_peered":"2024-04-02T13:24:56.064915+0000","last_clean":"2024-04-02T13:24:56.064915+0000","last_became_active":"2024-04-02T13:24:53.577079+0000","last_became_peered":"2024-04-02T13:24:53.577079+0000","last_unstale":"2024-04-02T13:24:56.064915+0000","last_undegraded":"2024-04-02T13:24:56.064915+0000","last_fullsized":"2024-04-02T13:24:56.064915+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:07:25.222536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064186+0000","last_change":"2024-04-02T13:24:54.579775+0000","last_active":"2024-04-02T13:24:56.064186+0000","last_peered":"2024-04-02T13:24:56.064186+0000","last_clean":"2024-04-02T13:24:56.064186+0000","last_became_active":"2024-04-02T13:24:54.579459+0000","last_became_peered":"2024-04-02T13:24:54.579459+0000","last_unstale":"2024-04-02T13:24:56.064186+0000","last_undegraded":"2024-04-02T13:24:56.064186+0000","last_fullsized":"2024-04-02T13:24:56.064186+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:03:48.945748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574863+0000","last_change":"2024-04-02T13:24:53.577294+0000","last_active":"2024-04-02T13:24:55.574863+0000","last_peered":"2024-04-02T13:24:55.574863+0000","last_clean":"2024-04-02T13:24:55.574863+0000","last_became_active":"2024-04-02T13:24:53.577018+0000","last_became_peered":"2024-04-02T13:24:53.577018+0000","last_unstale":"2024-04-02T13:24:55.574863+0000","last_undegraded":"2024-04-02T13:24:55.574863+0000","last_fullsized":"2024-04-02T13:24:55.574863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:18:07.894919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064584+0000","last_change":"2024-04-02T13:24:54.581345+0000","last_active":"2024-04-02T13:24:56.064584+0000","last_peered":"2024-04-02T13:24:56.064584+0000","last_clean":"2024-04-02T13:24:56.064584+0000","last_became_active":"2024-04-02T13:24:54.581091+0000","last_became_peered":"2024-04-02T13:24:54.581091+0000","last_unstale":"2024-04-02T13:24:56.064584+0000","last_undegraded":"2024-04-02T13:24:56.064584+0000","last_fullsized":"2024-04-02T13:24:56.064584+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:09:04.946124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064588+0000","last_change":"2024-04-02T13:24:53.573420+0000","last_active":"2024-04-02T13:24:56.064588+0000","last_peered":"2024-04-02T13:24:56.064588+0000","last_clean":"2024-04-02T13:24:56.064588+0000","last_became_active":"2024-04-02T13:24:53.571909+0000","last_became_peered":"2024-04-02T13:24:53.571909+0000","last_unstale":"2024-04-02T13:24:56.064588+0000","last_undegraded":"2024-04-02T13:24:56.064588+0000","last_fullsized":"2024-04-02T13:24:56.064588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:08:06.577978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056130+0000","last_change":"2024-04-02T13:24:54.586519+0000","last_active":"2024-04-02T13:24:56.056130+0000","last_peered":"2024-04-02T13:24:56.056130+0000","last_clean":"2024-04-02T13:24:56.056130+0000","last_became_active":"2024-04-02T13:24:54.584751+0000","last_became_peered":"2024-04-02T13:24:54.584751+0000","last_unstale":"2024-04-02T13:24:56.056130+0000","last_undegraded":"2024-04-02T13:24:56.056130+0000","last_fullsized":"2024-04-02T13:24:56.056130+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:25.219021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574737+0000","last_change":"2024-04-02T13:24:53.570594+0000","last_active":"2024-04-02T13:24:55.574737+0000","last_peered":"2024-04-02T13:24:55.574737+0000","last_clean":"2024-04-02T13:24:55.574737+0000","last_became_active":"2024-04-02T13:24:53.569678+0000","last_became_peered":"2024-04-02T13:24:53.569678+0000","last_unstale":"2024-04-02T13:24:55.574737+0000","last_undegraded":"2024-04-02T13:24:55.574737+0000","last_fullsized":"2024-04-02T13:24:55.574737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:02:30.957187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064778+0000","last_change":"2024-04-02T13:24:54.575816+0000","last_active":"2024-04-02T13:24:56.064778+0000","last_peered":"2024-04-02T13:24:56.064778+0000","last_clean":"2024-04-02T13:24:56.064778+0000","last_became_active":"2024-04-02T13:24:54.572762+0000","last_became_peered":"2024-04-02T13:24:54.572762+0000","last_unstale":"2024-04-02T13:24:56.064778+0000","last_undegraded":"2024-04-02T13:24:56.064778+0000","last_fullsized":"2024-04-02T13:24:56.064778+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:34:15.079374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056135+0000","last_change":"2024-04-02T13:24:53.571104+0000","last_active":"2024-04-02T13:24:56.056135+0000","last_peered":"2024-04-02T13:24:56.056135+0000","last_clean":"2024-04-02T13:24:56.056135+0000","last_became_active":"2024-04-02T13:24:53.570758+0000","last_became_peered":"2024-04-02T13:24:53.570758+0000","last_unstale":"2024-04-02T13:24:56.056135+0000","last_undegraded":"2024-04-02T13:24:56.056135+0000","last_fullsized":"2024-04-02T13:24:56.056135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:12:29.693987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064481+0000","last_change":"2024-04-02T13:24:54.585045+0000","last_active":"2024-04-02T13:24:56.064481+0000","last_peered":"2024-04-02T13:24:56.064481+0000","last_clean":"2024-04-02T13:24:56.064481+0000","last_became_active":"2024-04-02T13:24:54.584797+0000","last_became_peered":"2024-04-02T13:24:54.584797+0000","last_unstale":"2024-04-02T13:24:56.064481+0000","last_undegraded":"2024-04-02T13:24:56.064481+0000","last_fullsized":"2024-04-02T13:24:56.064481+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:25.219021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.055901+0000","last_change":"2024-04-02T13:24:53.573510+0000","last_active":"2024-04-02T13:24:56.055901+0000","last_peered":"2024-04-02T13:24:56.055901+0000","last_clean":"2024-04-02T13:24:56.055901+0000","last_became_active":"2024-04-02T13:24:53.572671+0000","last_became_peered":"2024-04-02T13:24:53.572671+0000","last_unstale":"2024-04-02T13:24:56.055901+0000","last_undegraded":"2024-04-02T13:24:56.055901+0000","last_fullsized":"2024-04-02T13:24:56.055901+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:36.056546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064931+0000","last_change":"2024-04-02T13:24:54.578914+0000","last_active":"2024-04-02T13:24:56.064931+0000","last_peered":"2024-04-02T13:24:56.064931+0000","last_clean":"2024-04-02T13:24:56.064931+0000","last_became_active":"2024-04-02T13:24:54.578531+0000","last_became_peered":"2024-04-02T13:24:54.578531+0000","last_unstale":"2024-04-02T13:24:56.064931+0000","last_undegraded":"2024-04-02T13:24:56.064931+0000","last_fullsized":"2024-04-02T13:24:56.064931+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:50.669792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.055891+0000","last_change":"2024-04-02T13:24:53.574067+0000","last_active":"2024-04-02T13:24:56.055891+0000","last_peered":"2024-04-02T13:24:56.055891+0000","last_clean":"2024-04-02T13:24:56.055891+0000","last_became_active":"2024-04-02T13:24:53.573799+0000","last_became_peered":"2024-04-02T13:24:53.573799+0000","last_unstale":"2024-04-02T13:24:56.055891+0000","last_undegraded":"2024-04-02T13:24:56.055891+0000","last_fullsized":"2024-04-02T13:24:56.055891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:30:06.264873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574142+0000","last_change":"2024-04-02T13:24:54.579259+0000","last_active":"2024-04-02T13:24:55.574142+0000","last_peered":"2024-04-02T13:24:55.574142+0000","last_clean":"2024-04-02T13:24:55.574142+0000","last_became_active":"2024-04-02T13:24:54.578957+0000","last_became_peered":"2024-04-02T13:24:54.578957+0000","last_unstale":"2024-04-02T13:24:55.574142+0000","last_undegraded":"2024-04-02T13:24:55.574142+0000","last_fullsized":"2024-04-02T13:24:55.574142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:02:31.959511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.573976+0000","last_change":"2024-04-02T13:24:53.575531+0000","last_active":"2024-04-02T13:24:55.573976+0000","last_peered":"2024-04-02T13:24:55.573976+0000","last_clean":"2024-04-02T13:24:55.573976+0000","last_became_active":"2024-04-02T13:24:53.575222+0000","last_became_peered":"2024-04-02T13:24:53.575222+0000","last_unstale":"2024-04-02T13:24:55.573976+0000","last_undegraded":"2024-04-02T13:24:55.573976+0000","last_fullsized":"2024-04-02T13:24:55.573976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:39:30.962000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064095+0000","last_change":"2024-04-02T13:24:54.575971+0000","last_active":"2024-04-02T13:24:56.064095+0000","last_peered":"2024-04-02T13:24:56.064095+0000","last_clean":"2024-04-02T13:24:56.064095+0000","last_became_active":"2024-04-02T13:24:54.575707+0000","last_became_peered":"2024-04-02T13:24:54.575707+0000","last_unstale":"2024-04-02T13:24:56.064095+0000","last_undegraded":"2024-04-02T13:24:56.064095+0000","last_fullsized":"2024-04-02T13:24:56.064095+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:08.394824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"79'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064634+0000","last_change":"2024-04-02T13:24:53.577485+0000","last_active":"2024-04-02T13:24:56.064634+0000","last_peered":"2024-04-02T13:24:56.064634+0000","last_clean":"2024-04-02T13:24:56.064634+0000","last_became_active":"2024-04-02T13:24:53.575330+0000","last_became_peered":"2024-04-02T13:24:53.575330+0000","last_unstale":"2024-04-02T13:24:56.064634+0000","last_undegraded":"2024-04-02T13:24:56.064634+0000","last_fullsized":"2024-04-02T13:24:56.064634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:49.667468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574799+0000","last_change":"2024-04-02T13:24:54.576490+0000","last_active":"2024-04-02T13:24:55.574799+0000","last_peered":"2024-04-02T13:24:55.574799+0000","last_clean":"2024-04-02T13:24:55.574799+0000","last_became_active":"2024-04-02T13:24:54.576224+0000","last_became_peered":"2024-04-02T13:24:54.576224+0000","last_unstale":"2024-04-02T13:24:55.574799+0000","last_undegraded":"2024-04-02T13:24:55.574799+0000","last_fullsized":"2024-04-02T13:24:55.574799+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:15:49.793401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064732+0000","last_change":"2024-04-02T13:24:53.572963+0000","last_active":"2024-04-02T13:24:56.064732+0000","last_peered":"2024-04-02T13:24:56.064732+0000","last_clean":"2024-04-02T13:24:56.064732+0000","last_became_active":"2024-04-02T13:24:53.572536+0000","last_became_peered":"2024-04-02T13:24:53.572536+0000","last_unstale":"2024-04-02T13:24:56.064732+0000","last_undegraded":"2024-04-02T13:24:56.064732+0000","last_fullsized":"2024-04-02T13:24:56.064732+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:43:42.127877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064578+0000","last_change":"2024-04-02T13:24:54.574356+0000","last_active":"2024-04-02T13:24:56.064578+0000","last_peered":"2024-04-02T13:24:56.064578+0000","last_clean":"2024-04-02T13:24:56.064578+0000","last_became_active":"2024-04-02T13:24:54.573985+0000","last_became_peered":"2024-04-02T13:24:54.573985+0000","last_unstale":"2024-04-02T13:24:56.064578+0000","last_undegraded":"2024-04-02T13:24:56.064578+0000","last_fullsized":"2024-04-02T13:24:56.064578+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:08.440154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064559+0000","last_change":"2024-04-02T13:24:53.580007+0000","last_active":"2024-04-02T13:24:56.064559+0000","last_peered":"2024-04-02T13:24:56.064559+0000","last_clean":"2024-04-02T13:24:56.064559+0000","last_became_active":"2024-04-02T13:24:53.579686+0000","last_became_peered":"2024-04-02T13:24:53.579686+0000","last_unstale":"2024-04-02T13:24:56.064559+0000","last_undegraded":"2024-04-02T13:24:56.064559+0000","last_fullsized":"2024-04-02T13:24:56.064559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:11.652383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064232+0000","last_change":"2024-04-02T13:24:54.585002+0000","last_active":"2024-04-02T13:24:56.064232+0000","last_peered":"2024-04-02T13:24:56.064232+0000","last_clean":"2024-04-02T13:24:56.064232+0000","last_became_active":"2024-04-02T13:24:54.584620+0000","last_became_peered":"2024-04-02T13:24:54.584620+0000","last_unstale":"2024-04-02T13:24:56.064232+0000","last_undegraded":"2024-04-02T13:24:56.064232+0000","last_fullsized":"2024-04-02T13:24:56.064232+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:57:17.244816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064180+0000","last_change":"2024-04-02T13:24:53.572298+0000","last_active":"2024-04-02T13:24:56.064180+0000","last_peered":"2024-04-02T13:24:56.064180+0000","last_clean":"2024-04-02T13:24:56.064180+0000","last_became_active":"2024-04-02T13:24:53.570380+0000","last_became_peered":"2024-04-02T13:24:53.570380+0000","last_unstale":"2024-04-02T13:24:56.064180+0000","last_undegraded":"2024-04-02T13:24:56.064180+0000","last_fullsized":"2024-04-02T13:24:56.064180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:43:26.495241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064671+0000","last_change":"2024-04-02T13:24:54.573352+0000","last_active":"2024-04-02T13:24:56.064671+0000","last_peered":"2024-04-02T13:24:56.064671+0000","last_clean":"2024-04-02T13:24:56.064671+0000","last_became_active":"2024-04-02T13:24:54.573039+0000","last_became_peered":"2024-04-02T13:24:54.573039+0000","last_unstale":"2024-04-02T13:24:56.064671+0000","last_undegraded":"2024-04-02T13:24:56.064671+0000","last_fullsized":"2024-04-02T13:24:56.064671+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:46:34.338949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064247+0000","last_change":"2024-04-02T13:24:53.572484+0000","last_active":"2024-04-02T13:24:56.064247+0000","last_peered":"2024-04-02T13:24:56.064247+0000","last_clean":"2024-04-02T13:24:56.064247+0000","last_became_active":"2024-04-02T13:24:53.572124+0000","last_became_peered":"2024-04-02T13:24:53.572124+0000","last_unstale":"2024-04-02T13:24:56.064247+0000","last_undegraded":"2024-04-02T13:24:56.064247+0000","last_fullsized":"2024-04-02T13:24:56.064247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:39:18.965555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574852+0000","last_change":"2024-04-02T13:24:54.573678+0000","last_active":"2024-04-02T13:24:55.574852+0000","last_peered":"2024-04-02T13:24:55.574852+0000","last_clean":"2024-04-02T13:24:55.574852+0000","last_became_active":"2024-04-02T13:24:54.573380+0000","last_became_peered":"2024-04-02T13:24:54.573380+0000","last_unstale":"2024-04-02T13:24:55.574852+0000","last_undegraded":"2024-04-02T13:24:55.574852+0000","last_fullsized":"2024-04-02T13:24:55.574852+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:48:44.079142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064587+0000","last_change":"2024-04-02T13:24:53.574193+0000","last_active":"2024-04-02T13:24:56.064587+0000","last_peered":"2024-04-02T13:24:56.064587+0000","last_clean":"2024-04-02T13:24:56.064587+0000","last_became_active":"2024-04-02T13:24:53.570565+0000","last_became_peered":"2024-04-02T13:24:53.570565+0000","last_unstale":"2024-04-02T13:24:56.064587+0000","last_undegraded":"2024-04-02T13:24:56.064587+0000","last_fullsized":"2024-04-02T13:24:56.064587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:50:27.793063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574928+0000","last_change":"2024-04-02T13:24:54.577947+0000","last_active":"2024-04-02T13:24:55.574928+0000","last_peered":"2024-04-02T13:24:55.574928+0000","last_clean":"2024-04-02T13:24:55.574928+0000","last_became_active":"2024-04-02T13:24:54.577637+0000","last_became_peered":"2024-04-02T13:24:54.577637+0000","last_unstale":"2024-04-02T13:24:55.574928+0000","last_undegraded":"2024-04-02T13:24:55.574928+0000","last_fullsized":"2024-04-02T13:24:55.574928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:08.394824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:55.574508+0000","last_change":"2024-04-02T13:24:53.567583+0000","last_active":"2024-04-02T13:24:55.574508+0000","last_peered":"2024-04-02T13:24:55.574508+0000","last_clean":"2024-04-02T13:24:55.574508+0000","last_became_active":"2024-04-02T13:24:53.567289+0000","last_became_peered":"2024-04-02T13:24:53.567289+0000","last_unstale":"2024-04-02T13:24:55.574508+0000","last_undegraded":"2024-04-02T13:24:55.574508+0000","last_fullsized":"2024-04-02T13:24:55.574508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:29:00.076440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064704+0000","last_change":"2024-04-02T13:24:54.578014+0000","last_active":"2024-04-02T13:24:56.064704+0000","last_peered":"2024-04-02T13:24:56.064704+0000","last_clean":"2024-04-02T13:24:56.064704+0000","last_became_active":"2024-04-02T13:24:54.577445+0000","last_became_peered":"2024-04-02T13:24:54.577445+0000","last_unstale":"2024-04-02T13:24:56.064704+0000","last_undegraded":"2024-04-02T13:24:56.064704+0000","last_fullsized":"2024-04-02T13:24:56.064704+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:53.549098+0000","last_clean_scrub_stamp":"2024-04-02T13:24:53.549098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:30:34.688997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.056408+0000","last_change":"2024-04-02T13:24:53.572298+0000","last_active":"2024-04-02T13:24:56.056408+0000","last_peered":"2024-04-02T13:24:56.056408+0000","last_clean":"2024-04-02T13:24:56.056408+0000","last_became_active":"2024-04-02T13:24:53.572035+0000","last_became_peered":"2024-04-02T13:24:53.572035+0000","last_unstale":"2024-04-02T13:24:56.056408+0000","last_undegraded":"2024-04-02T13:24:56.056408+0000","last_fullsized":"2024-04-02T13:24:56.056408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:24:52.546774+0000","last_clean_scrub_stamp":"2024-04-02T13:24:52.546774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:35.859707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":282,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-02T13:24:56.064569+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:24:56.064569+0000","last_peered":"2024-04-02T13:24:56.064569+0000","last_clean":"2024-04-02T13:24:56.064569+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:24:56.064569+0000","last_undegraded":"2024-04-02T13:24:56.064569+0000","last_fullsized":"2024-04-02T13:24:56.064569+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574908,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":1031999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":978944,"data_stored":443221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574908,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":441643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574908,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":987136,"data_stored":443325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":442191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":978944,"data_stored":442351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":974848,"data_stored":441721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574908,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1560576,"data_stored":1031977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:25:03.378 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T13:25:03.378 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T13:25:03.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.521+0000 7f2a5b332640 1 Processor -- start 2024-04-02T13:25:03.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.521+0000 7f2a5b332640 1 -- start start 2024-04-02T13:25:03.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a5b332640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2a5415e5e0 0x7f2a54071930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a5b332640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2a54071e70 0x7f2a54072300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a5b332640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a540ce5f0 0x7f2a540cea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a5b332640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2a5415e9c0 con 0x7f2a5415e5e0 2024-04-02T13:25:03.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a5b332640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2a540ce160 con 0x7f2a54071e70 2024-04-02T13:25:03.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a5b332640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2a540cd870 con 0x7f2a540ce5f0 2024-04-02T13:25:03.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a590ce640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2a5415e5e0 0x7f2a54071930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a590ce640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2a5415e5e0 0x7f2a54071930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41954/0 (socket says 172.21.15.44:41954) 2024-04-02T13:25:03.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a590ce640 1 -- 172.21.15.44:0/4059647938 learned_addr learned my addr 172.21.15.44:0/4059647938 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:03.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a590ce640 1 -- 172.21.15.44:0/4059647938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a540ce5f0 msgr2=0x7f2a540cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:03.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a598cf640 1 --2- 172.21.15.44:0/4059647938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a540ce5f0 0x7f2a540cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a588cd640 1 --2- 172.21.15.44:0/4059647938 >> v2:172.21.15.69:3300/0 conn(0x7f2a54071e70 0x7f2a54072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a590ce640 1 --2- 172.21.15.44:0/4059647938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a540ce5f0 0x7f2a540cea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a590ce640 1 -- 172.21.15.44:0/4059647938 >> v2:172.21.15.69:3300/0 conn(0x7f2a54071e70 msgr2=0x7f2a54072300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:03.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a590ce640 1 --2- 172.21.15.44:0/4059647938 >> v2:172.21.15.69:3300/0 conn(0x7f2a54071e70 0x7f2a54072300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.522+0000 7f2a590ce640 1 -- 172.21.15.44:0/4059647938 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a54072840 con 0x7f2a5415e5e0 2024-04-02T13:25:03.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.523+0000 7f2a590ce640 1 --2- 172.21.15.44:0/4059647938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a5415e5e0 0x7f2a54071930 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2a540cc6f0 tx=0x7f2a5002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaeaab3d7d79bb61 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:03.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.523+0000 7f2a4bfff640 1 -- 172.21.15.44:0/4059647938 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5003b070 con 0x7f2a5415e5e0 2024-04-02T13:25:03.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.523+0000 7f2a4bfff640 1 -- 172.21.15.44:0/4059647938 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a50036440 con 0x7f2a5415e5e0 2024-04-02T13:25:03.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.523+0000 7f2a4bfff640 1 -- 172.21.15.44:0/4059647938 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50036810 con 0x7f2a5415e5e0 2024-04-02T13:25:03.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.523+0000 7f2a5b332640 1 -- 172.21.15.44:0/4059647938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a5415e5e0 msgr2=0x7f2a54071930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.523+0000 7f2a5b332640 1 --2- 172.21.15.44:0/4059647938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a5415e5e0 0x7f2a54071930 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2a540cc6f0 tx=0x7f2a5002e0e0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.524+0000 7f2a5b332640 1 -- 172.21.15.44:0/4059647938 shutdown_connections 2024-04-02T13:25:03.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.524+0000 7f2a5b332640 1 --2- 172.21.15.44:0/4059647938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a540ce5f0 0x7f2a540cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.524+0000 7f2a5b332640 1 --2- 172.21.15.44:0/4059647938 >> v2:172.21.15.69:3300/0 conn(0x7f2a54071e70 0x7f2a54072300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.524+0000 7f2a5b332640 1 --2- 172.21.15.44:0/4059647938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a5415e5e0 0x7f2a54071930 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.524+0000 7f2a5b332640 1 -- 172.21.15.44:0/4059647938 >> 172.21.15.44:0/4059647938 conn(0x7f2a540cfca0 msgr2=0x7f2a540d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:03.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.524+0000 7f2a5b332640 1 -- 172.21.15.44:0/4059647938 shutdown_connections 2024-04-02T13:25:03.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.524+0000 7f2a5b332640 1 -- 172.21.15.44:0/4059647938 wait complete. 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.525+0000 7f2a5b332640 1 Processor -- start 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.525+0000 7f2a5b332640 1 -- start start 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.525+0000 7f2a5b332640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a54071e70 0x7f2a5409bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.525+0000 7f2a5b332640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a540ce5f0 0x7f2a54155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.525+0000 7f2a5b332640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a5415e5e0 0x7f2a540b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.525+0000 7f2a5b332640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2a540cddf0 con 0x7f2a54071e70 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.525+0000 7f2a5b332640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a540cdb10 con 0x7f2a540ce5f0 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.525+0000 7f2a5b332640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2a5406ec50 con 0x7f2a5415e5e0 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a590ce640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a54071e70 0x7f2a5409bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a590ce640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a54071e70 0x7f2a5409bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41968/0 (socket says 172.21.15.44:41968) 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a590ce640 1 -- 172.21.15.44:0/361788693 learned_addr learned my addr 172.21.15.44:0/361788693 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a590ce640 1 -- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a5415e5e0 msgr2=0x7f2a540b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a598cf640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a5415e5e0 0x7f2a540b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a588cd640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a540ce5f0 0x7f2a54155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a590ce640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a5415e5e0 0x7f2a540b4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a590ce640 1 -- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a540ce5f0 msgr2=0x7f2a54155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a590ce640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a540ce5f0 0x7f2a54155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a590ce640 1 -- 172.21.15.44:0/361788693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5409b5d0 con 0x7f2a54071e70 2024-04-02T13:25:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a590ce640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a54071e70 0x7f2a5409bd90 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f2a5002e120 tx=0x7f2a500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a49ffb640 1 -- 172.21.15.44:0/361788693 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5003b040 con 0x7f2a54071e70 2024-04-02T13:25:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a49ffb640 1 -- 172.21.15.44:0/361788693 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a50046070 con 0x7f2a54071e70 2024-04-02T13:25:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a5b332640 1 -- 172.21.15.44:0/361788693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a540b2f90 con 0x7f2a54071e70 2024-04-02T13:25:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a49ffb640 1 -- 172.21.15.44:0/361788693 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50034bb0 con 0x7f2a54071e70 2024-04-02T13:25:03.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.526+0000 7f2a5b332640 1 -- 172.21.15.44:0/361788693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a5409b870 con 0x7f2a54071e70 2024-04-02T13:25:03.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.527+0000 7f2a5b332640 1 -- 172.21.15.44:0/361788693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a54074f20 con 0x7f2a54071e70 2024-04-02T13:25:03.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.528+0000 7f2a49ffb640 1 -- 172.21.15.44:0/361788693 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a5004b460 con 0x7f2a54071e70 2024-04-02T13:25:03.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.528+0000 7f2a49ffb640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a3806c920 0x7f2a3806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.528+0000 7f2a49ffb640 1 -- 172.21.15.44:0/361788693 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f2a500b0b30 con 0x7f2a54071e70 2024-04-02T13:25:03.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.528+0000 7f2a588cd640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a3806c920 0x7f2a3806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.529+0000 7f2a588cd640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a3806c920 0x7f2a3806ede0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2a540cae50 tx=0x7f2a44008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:03.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.533+0000 7f2a49ffb640 1 -- 172.21.15.44:0/361788693 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2a5007edd0 con 0x7f2a54071e70 2024-04-02T13:25:03.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.791+0000 7f2a5b332640 1 -- 172.21.15.44:0/361788693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f2a540684f0 con 0x7f2a54071e70 2024-04-02T13:25:03.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.793+0000 7f2a49ffb640 1 -- 172.21.15.44:0/361788693 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) ==== 190+0+147 (secure 0 0 0) 0x7f2a5007e6d0 con 0x7f2a54071e70 2024-04-02T13:25:03.796 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-04-02T13:25:03.797 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A== 2024-04-02T13:25:03.797 INFO:teuthology.orchestra.run.smithi044.stdout: caps mds = "allow" 2024-04-02T13:25:03.797 INFO:teuthology.orchestra.run.smithi044.stdout: caps mon = "allow r" 2024-04-02T13:25:03.797 INFO:teuthology.orchestra.run.smithi044.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-02T13:25:03.797 INFO:teuthology.orchestra.run.smithi044.stderr:updated caps for client.0 2024-04-02T13:25:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.795+0000 7f2a5b332640 1 -- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a3806c920 msgr2=0x7f2a3806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.795+0000 7f2a5b332640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a3806c920 0x7f2a3806ede0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2a540cae50 tx=0x7f2a44008040 comp rx=0 tx=0).stop 2024-04-02T13:25:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.795+0000 7f2a5b332640 1 -- 172.21.15.44:0/361788693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a54071e70 msgr2=0x7f2a5409bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.795+0000 7f2a5b332640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a54071e70 0x7f2a5409bd90 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f2a5002e120 tx=0x7f2a500046c0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.795+0000 7f2a5b332640 1 -- 172.21.15.44:0/361788693 shutdown_connections 2024-04-02T13:25:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.795+0000 7f2a5b332640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a3806c920 0x7f2a3806ede0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.795+0000 7f2a5b332640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a5415e5e0 0x7f2a540b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.795+0000 7f2a5b332640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a540ce5f0 0x7f2a54155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.795+0000 7f2a5b332640 1 --2- 172.21.15.44:0/361788693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a54071e70 0x7f2a5409bd90 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.795+0000 7f2a5b332640 1 -- 172.21.15.44:0/361788693 >> 172.21.15.44:0/361788693 conn(0x7f2a540cfca0 msgr2=0x7f2a5406ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.796+0000 7f2a5b332640 1 -- 172.21.15.44:0/361788693 shutdown_connections 2024-04-02T13:25:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.796+0000 7f2a5b332640 1 -- 172.21.15.44:0/361788693 wait complete. 2024-04-02T13:25:03.809 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:25:03.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.908+0000 7fd9e7f32640 1 Processor -- start 2024-04-02T13:25:03.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.908+0000 7fd9e7f32640 1 -- start start 2024-04-02T13:25:03.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.909+0000 7fd9e7f32640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd9e015e4a0 0x7fd9e00aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.909+0000 7fd9e7f32640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9e00d0c50 0x7fd9e00d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.909+0000 7fd9e7f32640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9e00cc600 con 0x7fd9e006ce40 2024-04-02T13:25:03.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.909+0000 7fd9e7f32640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd9e00cc8e0 con 0x7fd9e015e4a0 2024-04-02T13:25:03.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.909+0000 7fd9e7f32640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e00ccbc0 con 0x7fd9e00d0c50 2024-04-02T13:25:03.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.909+0000 7fd9e5cce640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd9e015e4a0 0x7fd9e00aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.909+0000 7fd9e64cf640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd9e006ce40 0x7fd9e00cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:45278/0 (socket says 172.21.15.44:45278) 2024-04-02T13:25:03.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.909+0000 7fd9e64cf640 1 -- 172.21.15.44:0/2766055218 learned_addr learned my addr 172.21.15.44:0/2766055218 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:03.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.909+0000 7fd9e54cd640 1 --2- 172.21.15.44:0/2766055218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9e00d0c50 0x7fd9e00d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.909+0000 7fd9e5cce640 1 -- 172.21.15.44:0/2766055218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9e00d0c50 msgr2=0x7fd9e00d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:03.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.909+0000 7fd9e5cce640 1 --2- 172.21.15.44:0/2766055218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9e00d0c50 0x7fd9e00d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.909+0000 7fd9e5cce640 1 -- 172.21.15.44:0/2766055218 >> v1:172.21.15.44:6789/0 conn(0x7fd9e006ce40 legacy=0x7fd9e00cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.909+0000 7fd9e5cce640 1 -- 172.21.15.44:0/2766055218 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e00ccea0 con 0x7fd9e015e4a0 2024-04-02T13:25:03.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.910+0000 7fd9e5cce640 1 --2- 172.21.15.44:0/2766055218 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9e015e4a0 0x7fd9e00aebf0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0097a0 tx=0x7fd9dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd5bb892ac052ec server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:03.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.910+0000 7fd9e4ccc640 1 -- 172.21.15.44:0/2766055218 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc035020 con 0x7fd9e015e4a0 2024-04-02T13:25:03.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.910+0000 7fd9e4ccc640 1 -- 172.21.15.44:0/2766055218 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9dc031830 con 0x7fd9e015e4a0 2024-04-02T13:25:03.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.910+0000 7fd9e4ccc640 1 -- 172.21.15.44:0/2766055218 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc031c00 con 0x7fd9e015e4a0 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.910+0000 7fd9e7f32640 1 -- 172.21.15.44:0/2766055218 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9e015e4a0 msgr2=0x7fd9e00aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.910+0000 7fd9e7f32640 1 --2- 172.21.15.44:0/2766055218 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9e015e4a0 0x7fd9e00aebf0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0097a0 tx=0x7fd9dc02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.911+0000 7fd9e7f32640 1 -- 172.21.15.44:0/2766055218 shutdown_connections 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.911+0000 7fd9e7f32640 1 --2- 172.21.15.44:0/2766055218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9e00d0c50 0x7fd9e00d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.911+0000 7fd9e7f32640 1 --2- 172.21.15.44:0/2766055218 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9e015e4a0 0x7fd9e00aebf0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.911+0000 7fd9e7f32640 1 -- 172.21.15.44:0/2766055218 >> 172.21.15.44:0/2766055218 conn(0x7fd9e009aa20 msgr2=0x7fd9e009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.911+0000 7fd9e7f32640 1 -- 172.21.15.44:0/2766055218 shutdown_connections 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.911+0000 7fd9e7f32640 1 -- 172.21.15.44:0/2766055218 wait complete. 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.912+0000 7fd9e7f32640 1 Processor -- start 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.912+0000 7fd9e7f32640 1 -- start start 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.912+0000 7fd9e7f32640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9e006ce40 0x7fd9e00c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.912+0000 7fd9e7f32640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9e00d0c50 0x7fd9e00ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e7f32640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9e015e4a0 0x7fd9e00b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e7f32640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e015e880 con 0x7fd9e006ce40 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e7f32640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e00cd460 con 0x7fd9e015e4a0 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e7f32640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e00af130 con 0x7fd9e00d0c50 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e5cce640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9e006ce40 0x7fd9e00c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e5cce640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9e006ce40 0x7fd9e00c9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41982/0 (socket says 172.21.15.44:41982) 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e5cce640 1 -- 172.21.15.44:0/1347907551 learned_addr learned my addr 172.21.15.44:0/1347907551 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e5cce640 1 -- 172.21.15.44:0/1347907551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9e00d0c50 msgr2=0x7fd9e00ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e5cce640 1 --2- 172.21.15.44:0/1347907551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9e00d0c50 0x7fd9e00ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e64cf640 1 --2- 172.21.15.44:0/1347907551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9e015e4a0 0x7fd9e00b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e5cce640 1 -- 172.21.15.44:0/1347907551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9e015e4a0 msgr2=0x7fd9e00b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e5cce640 1 --2- 172.21.15.44:0/1347907551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9e015e4a0 0x7fd9e00b2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e5cce640 1 -- 172.21.15.44:0/1347907551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e015f800 con 0x7fd9e006ce40 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e5cce640 1 --2- 172.21.15.44:0/1347907551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9e006ce40 0x7fd9e00c9080 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc034040 tx=0x7fd9dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9d67fc640 1 -- 172.21.15.44:0/1347907551 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc040070 con 0x7fd9e006ce40 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9d67fc640 1 -- 172.21.15.44:0/1347907551 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9dc0399a0 con 0x7fd9e006ce40 2024-04-02T13:25:03.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e7f32640 1 -- 172.21.15.44:0/1347907551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9e015fae0 con 0x7fd9e006ce40 2024-04-02T13:25:03.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9d67fc640 1 -- 172.21.15.44:0/1347907551 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc035040 con 0x7fd9e006ce40 2024-04-02T13:25:03.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.913+0000 7fd9e7f32640 1 -- 172.21.15.44:0/1347907551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9e01600a0 con 0x7fd9e006ce40 2024-04-02T13:25:03.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.914+0000 7fd9e7f32640 1 -- 172.21.15.44:0/1347907551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9e00684f0 con 0x7fd9e006ce40 2024-04-02T13:25:03.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.915+0000 7fd9d67fc640 1 -- 172.21.15.44:0/1347907551 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd9dc038020 con 0x7fd9e006ce40 2024-04-02T13:25:03.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.915+0000 7fd9d67fc640 1 --2- 172.21.15.44:0/1347907551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd9c406c8f0 0x7fd9c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:03.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.915+0000 7fd9d67fc640 1 -- 172.21.15.44:0/1347907551 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fd9dc0ad770 con 0x7fd9e006ce40 2024-04-02T13:25:03.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.915+0000 7fd9e54cd640 1 --2- 172.21.15.44:0/1347907551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd9c406c8f0 0x7fd9c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:03.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.916+0000 7fd9e54cd640 1 --2- 172.21.15.44:0/1347907551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd9c406c8f0 0x7fd9c406edb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd9e015f330 tx=0x7fd9d0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:03.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:03.920+0000 7fd9d67fc640 1 -- 172.21.15.44:0/1347907551 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd9dc07b980 con 0x7fd9e006ce40 2024-04-02T13:25:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.144+0000 7fd9e7f32640 1 -- 172.21.15.44:0/1347907551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9e00cc370 con 0x7fd9e006ce40 2024-04-02T13:25:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.144+0000 7fd9d67fc640 1 -- 172.21.15.44:0/1347907551 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5180 (secure 0 0 0) 0x7fd9dc07b280 con 0x7fd9e006ce40 2024-04-02T13:25:04.148 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:04.148 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3662927745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3662927745},{"type":"v1","addr":"172.21.15.44:6835","nonce":3662927745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6394,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1053037366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1053037366},{"type":"v1","addr":"172.21.15.69:6836","nonce":1053037366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6987,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2392159648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2392159648},{"type":"v1","addr":"172.21.15.44:6833","nonce":2392159648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:24:53.925714+0000","modified":"2024-04-02T13:24:56.942238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6837/698185867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":698185867},{"type":"v1","addr":"172.21.15.69:6837","nonce":698185867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7002,"qdb_cluster":[7002]},"id":5}]} 2024-04-02T13:25:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 44 2024-04-02T13:25:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.146+0000 7fd9e7f32640 1 -- 172.21.15.44:0/1347907551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd9c406c8f0 msgr2=0x7fd9c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.146+0000 7fd9e7f32640 1 --2- 172.21.15.44:0/1347907551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd9c406c8f0 0x7fd9c406edb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd9e015f330 tx=0x7fd9d0008040 comp rx=0 tx=0).stop 2024-04-02T13:25:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.146+0000 7fd9e7f32640 1 -- 172.21.15.44:0/1347907551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9e006ce40 msgr2=0x7fd9e00c9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.146+0000 7fd9e7f32640 1 --2- 172.21.15.44:0/1347907551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9e006ce40 0x7fd9e00c9080 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc034040 tx=0x7fd9dc0046c0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.147+0000 7fd9e7f32640 1 -- 172.21.15.44:0/1347907551 shutdown_connections 2024-04-02T13:25:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.147+0000 7fd9e7f32640 1 --2- 172.21.15.44:0/1347907551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd9c406c8f0 0x7fd9c406edb0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.147+0000 7fd9e7f32640 1 --2- 172.21.15.44:0/1347907551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9e015e4a0 0x7fd9e00b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.147+0000 7fd9e7f32640 1 --2- 172.21.15.44:0/1347907551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9e00d0c50 0x7fd9e00ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.147+0000 7fd9e7f32640 1 --2- 172.21.15.44:0/1347907551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9e006ce40 0x7fd9e00c9080 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.147+0000 7fd9e7f32640 1 -- 172.21.15.44:0/1347907551 >> 172.21.15.44:0/1347907551 conn(0x7fd9e009aa20 msgr2=0x7fd9e009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.147+0000 7fd9e7f32640 1 -- 172.21.15.44:0/1347907551 shutdown_connections 2024-04-02T13:25:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.147+0000 7fd9e7f32640 1 -- 172.21.15.44:0/1347907551 wait complete. 2024-04-02T13:25:04.562 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:24:53.925714+0000', 'modified': '2024-04-02T13:24:56.942238+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7002}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7002': {'gid': 7002, 'name': 'd', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.69:6837/698185867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 698185867}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 698185867}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7002, 'qdb_cluster': [7002]} 2024-04-02T13:25:04.562 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:25:04.563 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-02T13:25:04.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.670+0000 7fc33a822640 1 Processor -- start 2024-04-02T13:25:04.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.670+0000 7fc33a822640 1 -- start start 2024-04-02T13:25:04.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc33a822640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc3340d0c50 0x7fc3340d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc33a822640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3340cc600 con 0x7fc33406ce40 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc33a822640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3340cc8e0 con 0x7fc33415e4a0 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc33a822640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc3340af300 con 0x7fc3340d0c50 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc338dbf640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc33406ce40 0x7fc3340cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48416/0 (socket says 172.21.15.44:48416) 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc338dbf640 1 -- 172.21.15.44:0/3221828359 learned_addr learned my addr 172.21.15.44:0/3221828359 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc3337fe640 1 --2- 172.21.15.44:0/3221828359 >> v2:172.21.15.69:3300/0 conn(0x7fc3340d0c50 0x7fc3340d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1209844827 0 0) 0x7fc3340cc600 con 0x7fc33406ce40 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc320003520 con 0x7fc33406ce40 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 961631277 0 0) 0x7fc3340cc8e0 con 0x7fc33415e4a0 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3340cc600 con 0x7fc33415e4a0 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2366798915 0 0) 0x7fc320003520 con 0x7fc33406ce40 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 >> v2:172.21.15.69:3300/0 conn(0x7fc3340d0c50 msgr2=0x7fc3340d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc332ffd640 1 --2- 172.21.15.44:0/3221828359 >> v2:172.21.15.69:3300/0 conn(0x7fc3340d0c50 0x7fc3340d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 >> v1:172.21.15.69:6789/0 conn(0x7fc33415e4a0 legacy=0x7fc3340aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3340ccbc0 con 0x7fc33406ce40 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fc32c001970 con 0x7fc33406ce40 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 >> v1:172.21.15.44:6789/0 conn(0x7fc33406ce40 legacy=0x7fc3340cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc332ffd640 1 --2- 172.21.15.44:0/3221828359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc320003b90 0x7fc320003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.671+0000 7fc332ffd640 1 --2- 172.21.15.44:0/3221828359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc320004800 0x7fc320004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc332ffd640 1 --2- 172.21.15.44:0/3221828359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3200051c0 0x7fc32000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3340af300 con 0x7fc320004800 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc32000c090 con 0x7fc320003b90 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc32000c290 con 0x7fc3200051c0 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc3337fe640 1 --2- 172.21.15.44:0/3221828359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc320004800 0x7fc320004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc333fff640 1 --2- 172.21.15.44:0/3221828359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc320003b90 0x7fc320003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc338dbf640 1 --2- 172.21.15.44:0/3221828359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3200051c0 0x7fc32000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc3337fe640 1 -- 172.21.15.44:0/3221828359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3200051c0 msgr2=0x7fc32000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc3337fe640 1 --2- 172.21.15.44:0/3221828359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3200051c0 0x7fc32000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc3337fe640 1 -- 172.21.15.44:0/3221828359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc320003b90 msgr2=0x7fc320003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc3337fe640 1 --2- 172.21.15.44:0/3221828359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc320003b90 0x7fc320003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc3337fe640 1 -- 172.21.15.44:0/3221828359 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc320003520 con 0x7fc320004800 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc338dbf640 1 -- 172.21.15.44:0/3221828359 reap_dead start 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.672+0000 7fc333fff640 1 --2- 172.21.15.44:0/3221828359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc320003b90 0x7fc320003f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.673+0000 7fc3337fe640 1 --2- 172.21.15.44:0/3221828359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc320004800 0x7fc320004c80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fc324009a90 tx=0x7fc320005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83d08041cc1bb0a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.673+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324039070 con 0x7fc320004800 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.673+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc32402c680 con 0x7fc320004800 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.673+0000 7fc332ffd640 1 -- 172.21.15.44:0/3221828359 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324009690 con 0x7fc320004800 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.673+0000 7fc33a822640 1 -- 172.21.15.44:0/3221828359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc320004800 msgr2=0x7fc320004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.673+0000 7fc33a822640 1 --2- 172.21.15.44:0/3221828359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc320004800 0x7fc320004c80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fc324009a90 tx=0x7fc320005830 comp rx=0 tx=0).stop 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.673+0000 7fc33a822640 1 -- 172.21.15.44:0/3221828359 shutdown_connections 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.673+0000 7fc33a822640 1 --2- 172.21.15.44:0/3221828359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc320004800 0x7fc320004c80 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.673+0000 7fc33a822640 1 -- 172.21.15.44:0/3221828359 >> 172.21.15.44:0/3221828359 conn(0x7fc33409aa20 msgr2=0x7fc33409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:04.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.673+0000 7fc33a822640 1 -- 172.21.15.44:0/3221828359 shutdown_connections 2024-04-02T13:25:04.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.673+0000 7fc33a822640 1 -- 172.21.15.44:0/3221828359 wait complete. 2024-04-02T13:25:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.674+0000 7fc33a822640 1 Processor -- start 2024-04-02T13:25:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.675+0000 7fc33a822640 1 -- start start 2024-04-02T13:25:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.675+0000 7fc33a822640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc320004800 0x7fc3341514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.675+0000 7fc33a822640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3340cb870 0x7fc3340b5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.675+0000 7fc33a822640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3340b72e0 0x7fc3340b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.675+0000 7fc33a822640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3340cd460 con 0x7fc320004800 2024-04-02T13:25:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.675+0000 7fc33a822640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc3340cd180 con 0x7fc3340b72e0 2024-04-02T13:25:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.675+0000 7fc33a822640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc33415e880 con 0x7fc3340cb870 2024-04-02T13:25:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.675+0000 7fc333fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc320004800 0x7fc3341514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc333fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc320004800 0x7fc3341514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59822/0 (socket says 172.21.15.44:59822) 2024-04-02T13:25:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc333fff640 1 -- 172.21.15.44:0/2511707672 learned_addr learned my addr 172.21.15.44:0/2511707672 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc3337fe640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3340cb870 0x7fc3340b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc338dbf640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3340b72e0 0x7fc3340b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc333fff640 1 -- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3340cb870 msgr2=0x7fc3340b5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc333fff640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3340cb870 0x7fc3340b5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc333fff640 1 -- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3340b72e0 msgr2=0x7fc3340b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc333fff640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3340b72e0 0x7fc3340b0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc333fff640 1 -- 172.21.15.44:0/2511707672 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3340b6240 con 0x7fc320004800 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc3337fe640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3340cb870 0x7fc3340b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc338dbf640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3340b72e0 0x7fc3340b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc333fff640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc320004800 0x7fc3341514b0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00e7e0 tx=0x7fc31c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc330ff9640 1 -- 172.21.15.44:0/2511707672 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c0050d0 con 0x7fc320004800 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc330ff9640 1 -- 172.21.15.44:0/2511707672 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc31c005270 con 0x7fc320004800 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc33a822640 1 -- 172.21.15.44:0/2511707672 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3340b6520 con 0x7fc320004800 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc330ff9640 1 -- 172.21.15.44:0/2511707672 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c005440 con 0x7fc320004800 2024-04-02T13:25:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.676+0000 7fc33a822640 1 -- 172.21.15.44:0/2511707672 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3340b6ae0 con 0x7fc320004800 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.678+0000 7fc330ff9640 1 -- 172.21.15.44:0/2511707672 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc31c01a020 con 0x7fc320004800 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.678+0000 7fc330ff9640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc31406c920 0x7fc31406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.678+0000 7fc330ff9640 1 -- 172.21.15.44:0/2511707672 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fc31c08b490 con 0x7fc320004800 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.679+0000 7fc3337fe640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc31406c920 0x7fc31406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.679+0000 7fc3337fe640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc31406c920 0x7fc31406ede0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc324036000 tx=0x7fc324030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.679+0000 7fc33d409740 1 Processor -- start 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.679+0000 7fc33d409740 1 -- start start 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc33d409740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28b90 0x55d166efbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc33d409740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28f70 0x55d166f2b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc33d409740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166efc070 0x55d166f2be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc33d409740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d166e65060 con 0x55d166efc070 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc33d409740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d166e0d650 con 0x55d166f28f70 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc33d409740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55d166e61010 con 0x55d166f28b90 2024-04-02T13:25:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc338dbf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166efc070 0x55d166f2be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc338dbf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166efc070 0x55d166f2be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59826/0 (socket says 172.21.15.44:59826) 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc338dbf640 1 -- 172.21.15.44:0/1875352254 learned_addr learned my addr 172.21.15.44:0/1875352254 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc333fff640 1 --2- 172.21.15.44:0/1875352254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28b90 0x55d166efbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc3337fe640 1 --2- 172.21.15.44:0/1875352254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28f70 0x55d166f2b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc338dbf640 1 -- 172.21.15.44:0/1875352254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28b90 msgr2=0x55d166efbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc338dbf640 1 --2- 172.21.15.44:0/1875352254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28b90 0x55d166efbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc338dbf640 1 -- 172.21.15.44:0/1875352254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28f70 msgr2=0x55d166f2b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc338dbf640 1 --2- 172.21.15.44:0/1875352254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28f70 0x55d166f2b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.680+0000 7fc338dbf640 1 -- 172.21.15.44:0/1875352254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d166b792d0 con 0x55d166efc070 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.681+0000 7fc338dbf640 1 --2- 172.21.15.44:0/1875352254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166efc070 0x55d166f2be60 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fc33406cdd0 tx=0x7fc32c005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e188dddcbb1bb0b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.681+0000 7fc329ffb640 1 -- 172.21.15.44:0/1875352254 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c019070 con 0x55d166efc070 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.681+0000 7fc329ffb640 1 -- 172.21.15.44:0/1875352254 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc32c007950 con 0x55d166efc070 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.681+0000 7fc329ffb640 1 -- 172.21.15.44:0/1875352254 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c010040 con 0x55d166efc070 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.681+0000 7fc33d409740 1 -- 172.21.15.44:0/1875352254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166efc070 msgr2=0x55d166f2be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.681+0000 7fc33d409740 1 --2- 172.21.15.44:0/1875352254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166efc070 0x55d166f2be60 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fc33406cdd0 tx=0x7fc32c005f40 comp rx=0 tx=0).stop 2024-04-02T13:25:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.681+0000 7fc33d409740 1 -- 172.21.15.44:0/1875352254 shutdown_connections 2024-04-02T13:25:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.681+0000 7fc33d409740 1 --2- 172.21.15.44:0/1875352254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166efc070 0x55d166f2be60 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.681+0000 7fc33d409740 1 --2- 172.21.15.44:0/1875352254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28f70 0x55d166f2b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.681+0000 7fc33d409740 1 --2- 172.21.15.44:0/1875352254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28b90 0x55d166efbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.681+0000 7fc33d409740 1 -- 172.21.15.44:0/1875352254 >> 172.21.15.44:0/1875352254 conn(0x55d166efdda0 msgr2=0x55d166f46920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.681+0000 7fc33d409740 1 -- 172.21.15.44:0/1875352254 shutdown_connections 2024-04-02T13:25:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.681+0000 7fc33d409740 1 -- 172.21.15.44:0/1875352254 wait complete. 2024-04-02T13:25:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.682+0000 7fc33d409740 1 -- start start 2024-04-02T13:25:04.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.682+0000 7fc33d409740 1 Processor -- start 2024-04-02T13:25:04.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.683+0000 7fc33d409740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28b90 0x55d166f34000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.683+0000 7fc33d409740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 0x55d166f312a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.683+0000 7fc33d409740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f2f7f0 0x55d166f317e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.683+0000 7fc33d409740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d166f00440 con 0x55d166f28f70 2024-04-02T13:25:04.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.683+0000 7fc33d409740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d166e20a10 con 0x55d166f2f7f0 2024-04-02T13:25:04.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.683+0000 7fc33d409740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55d166f02c80 con 0x55d166f28b90 2024-04-02T13:25:04.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.683+0000 7fc33d409740 10 client.0 _ll_register_callbacks cb 0x55d166e5d0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:04.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.683+0000 7fc3337fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 0x55d166f312a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.683+0000 7fc3337fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 0x55d166f312a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59836/0 (socket says 172.21.15.44:59836) 2024-04-02T13:25:04.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.683+0000 7fc3337fe640 1 -- 172.21.15.44:0/3964331754 learned_addr learned my addr 172.21.15.44:0/3964331754 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:04.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.683+0000 7fc333fff640 1 --2- 172.21.15.44:0/3964331754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28b90 0x55d166f34000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.683+0000 7fc338dbf640 1 --2- 172.21.15.44:0/3964331754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f2f7f0 0x55d166f317e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc3337fe640 1 -- 172.21.15.44:0/3964331754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28b90 msgr2=0x55d166f34000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc3337fe640 1 --2- 172.21.15.44:0/3964331754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28b90 0x55d166f34000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc3337fe640 1 -- 172.21.15.44:0/3964331754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f2f7f0 msgr2=0x55d166f317e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc3337fe640 1 --2- 172.21.15.44:0/3964331754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f2f7f0 0x55d166f317e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc3337fe640 1 -- 172.21.15.44:0/3964331754 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d166c436b0 con 0x55d166f28f70 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc3337fe640 1 --2- 172.21.15.44:0/3964331754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 0x55d166f312a0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc324009bb0 tx=0x7fc3240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79a2ae9368951a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc312ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc312ffd640 1 -- 172.21.15.44:0/3964331754 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324039040 con 0x55d166f28f70 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc312ffd640 1 -- 172.21.15.44:0/3964331754 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fc32403fd50 con 0x55d166f28f70 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc33d409740 1 -- 172.21.15.44:0/3964331754 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc32c007950 con 0x55d166f28f70 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc312ffd640 1 -- 172.21.15.44:0/3964331754 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fc324059050 con 0x55d166f28f70 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc33d409740 10 client.7161 fetch_fsmap learned FSMap version 44 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc33d409740 1 -- 172.21.15.44:0/3964331754 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x55d166f34540 con 0x55d166f28f70 2024-04-02T13:25:04.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc312ffd640 1 -- 172.21.15.44:0/3964331754 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc32c007950 con 0x55d166f28f70 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc312ffd640 1 -- 172.21.15.44:0/3964331754 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 44) ==== 3494+0+0 (secure 0 0 0) 0x7fc32405c020 con 0x55d166f28f70 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc33d409740 10 client.7161 fetch_fsmap finished waiting for FSMap version 44 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc33d409740 10 client.7161 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc33d409740 20 client.7161 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc33d409740 1 --2- 172.21.15.44:0/3964331754 >> [v2:172.21.15.69:6835/698185867,v1:172.21.15.69:6837/698185867] conn(0x55d166f5a970 0x55d166f64d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc33d409740 4 client.7161 mds_command: new command op to 7002 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.684+0000 7fc33d409740 1 -- 172.21.15.44:0/3964331754 --> [v2:172.21.15.69:6835/698185867,v1:172.21.15.69:6837/698185867] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d166e0ccf0 con 0x55d166f5a970 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.685+0000 7fc333fff640 1 --2- 172.21.15.44:0/3964331754 >> [v2:172.21.15.69:6835/698185867,v1:172.21.15.69:6837/698185867] conn(0x55d166f5a970 0x55d166f64d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.685+0000 7fc312ffd640 10 client.7161 ms_handle_connect on v2:172.21.15.69:6835/698185867 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.685+0000 7fc333fff640 1 --2- 172.21.15.44:0/3964331754 >> [v2:172.21.15.69:6835/698185867,v1:172.21.15.69:6837/698185867] conn(0x55d166f5a970 0x55d166f64d50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=234dc1877e82ef2e server_cookie=53345f1fd9d23b32 in_seq=0 out_seq=0 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc312ffd640 1 -- 172.21.15.44:0/3964331754 <== mds.0 v2:172.21.15.69:6835/698185867 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d166e0ccf0 con 0x55d166f5a970 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc312ffd640 10 client.7161 handle_command_reply: tid=0 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc33d409740 1 client.7161 shutdown 2024-04-02T13:25:04.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc33d409740 1 -- 172.21.15.44:0/3964331754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 msgr2=0x55d166f312a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc33d409740 1 --2- 172.21.15.44:0/3964331754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 0x55d166f312a0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc324009bb0 tx=0x7fc3240046c0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc33d409740 1 -- 172.21.15.44:0/3964331754 shutdown_connections 2024-04-02T13:25:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc33d409740 1 --2- 172.21.15.44:0/3964331754 >> [v2:172.21.15.69:6835/698185867,v1:172.21.15.69:6837/698185867] conn(0x55d166f5a970 0x55d166f64d50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc33d409740 1 --2- 172.21.15.44:0/3964331754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f2f7f0 0x55d166f317e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc33d409740 1 --2- 172.21.15.44:0/3964331754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 0x55d166f312a0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc312ffd640 0 client.7161 ms_handle_reset on v2:172.21.15.69:6835/698185867 2024-04-02T13:25:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc33d409740 1 --2- 172.21.15.44:0/3964331754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28b90 0x55d166f34000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc33d409740 1 -- 172.21.15.44:0/3964331754 >> 172.21.15.44:0/3964331754 conn(0x55d166efc070 msgr2=0x55d166effb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc33d409740 1 -- 172.21.15.44:0/3964331754 shutdown_connections 2024-04-02T13:25:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc33d409740 1 -- 172.21.15.44:0/3964331754 wait complete. 2024-04-02T13:25:04.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.686+0000 7fc33d409740 20 client.7161 trim_cache size 0 max 16384 2024-04-02T13:25:04.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.705+0000 7fc33d409740 1 Processor -- start 2024-04-02T13:25:04.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.705+0000 7fc33d409740 1 -- start start 2024-04-02T13:25:04.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.705+0000 7fc33d409740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28b90 0x55d166f00050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.705+0000 7fc33d409740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 0x55d166f1c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.705+0000 7fc33d409740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f2f7f0 0x55d166f6a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.705+0000 7fc33d409740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3342005c0 con 0x55d166f28f70 2024-04-02T13:25:04.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.705+0000 7fc33d409740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc3341ffa70 con 0x55d166f28b90 2024-04-02T13:25:04.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.705+0000 7fc33d409740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc3341fffa0 con 0x55d166f2f7f0 2024-04-02T13:25:04.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc3337fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 0x55d166f1c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc3337fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 0x55d166f1c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59842/0 (socket says 172.21.15.44:59842) 2024-04-02T13:25:04.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc3337fe640 1 -- 172.21.15.44:0/4203726440 learned_addr learned my addr 172.21.15.44:0/4203726440 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc333fff640 1 --2- 172.21.15.44:0/4203726440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28b90 0x55d166f00050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc338dbf640 1 --2- 172.21.15.44:0/4203726440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f2f7f0 0x55d166f6a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc3337fe640 1 -- 172.21.15.44:0/4203726440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f2f7f0 msgr2=0x55d166f6a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc3337fe640 1 --2- 172.21.15.44:0/4203726440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f2f7f0 0x55d166f6a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc3337fe640 1 -- 172.21.15.44:0/4203726440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28b90 msgr2=0x55d166f00050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc3337fe640 1 --2- 172.21.15.44:0/4203726440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28b90 0x55d166f00050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc3337fe640 1 -- 172.21.15.44:0/4203726440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d166f308b0 con 0x55d166f28f70 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc3337fe640 1 --2- 172.21.15.44:0/4203726440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 0x55d166f1c7a0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc324009b80 tx=0x7fc324043070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2eda9c9d6c079f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc3297fa640 1 -- 172.21.15.44:0/4203726440 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324062070 con 0x55d166f28f70 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc3297fa640 1 -- 172.21.15.44:0/4203726440 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc324051070 con 0x55d166f28f70 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.706+0000 7fc3297fa640 1 -- 172.21.15.44:0/4203726440 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324039040 con 0x55d166f28f70 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.707+0000 7fc33d409740 1 -- 172.21.15.44:0/4203726440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 msgr2=0x55d166f1c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.707+0000 7fc33d409740 1 --2- 172.21.15.44:0/4203726440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 0x55d166f1c7a0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc324009b80 tx=0x7fc324043070 comp rx=0 tx=0).stop 2024-04-02T13:25:04.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.707+0000 7fc33d409740 1 -- 172.21.15.44:0/4203726440 shutdown_connections 2024-04-02T13:25:04.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.707+0000 7fc33d409740 1 --2- 172.21.15.44:0/4203726440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f2f7f0 0x55d166f6a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.707+0000 7fc33d409740 1 --2- 172.21.15.44:0/4203726440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f28f70 0x55d166f1c7a0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.707+0000 7fc33d409740 1 --2- 172.21.15.44:0/4203726440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28b90 0x55d166f00050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.707+0000 7fc33d409740 1 -- 172.21.15.44:0/4203726440 >> 172.21.15.44:0/4203726440 conn(0x55d166efc070 msgr2=0x55d166f2b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:04.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.707+0000 7fc33d409740 1 -- 172.21.15.44:0/4203726440 shutdown_connections 2024-04-02T13:25:04.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.707+0000 7fc33d409740 1 -- 172.21.15.44:0/4203726440 wait complete. 2024-04-02T13:25:04.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.707+0000 7fc33d409740 1 -- start start 2024-04-02T13:25:04.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.707+0000 7fc33d409740 1 Processor -- start 2024-04-02T13:25:04.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.708+0000 7fc33d409740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28b90 0x55d166fc0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.708+0000 7fc33d409740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28f70 0x55d166f2b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.708+0000 7fc33d409740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f2f7f0 0x55d166f2b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.708+0000 7fc33d409740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d166f5a7f0 con 0x55d166f2f7f0 2024-04-02T13:25:04.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.708+0000 7fc33d409740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d166f31f00 con 0x55d166f28b90 2024-04-02T13:25:04.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.708+0000 7fc33d409740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55d166f5adc0 con 0x55d166f28f70 2024-04-02T13:25:04.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.708+0000 7fc33d409740 10 client.0 _ll_register_callbacks cb 0x7fc3200036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:04.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.708+0000 7fc338dbf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f2f7f0 0x55d166f2b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.708+0000 7fc338dbf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f2f7f0 0x55d166f2b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59844/0 (socket says 172.21.15.44:59844) 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.708+0000 7fc338dbf640 1 -- 172.21.15.44:0/609422299 learned_addr learned my addr 172.21.15.44:0/609422299 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.708+0000 7fc3337fe640 1 --2- 172.21.15.44:0/609422299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28f70 0x55d166f2b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.708+0000 7fc333fff640 1 --2- 172.21.15.44:0/609422299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28b90 0x55d166fc0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc338dbf640 1 -- 172.21.15.44:0/609422299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28f70 msgr2=0x55d166f2b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc338dbf640 1 --2- 172.21.15.44:0/609422299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28f70 0x55d166f2b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc338dbf640 1 -- 172.21.15.44:0/609422299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28b90 msgr2=0x55d166fc0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc338dbf640 1 --2- 172.21.15.44:0/609422299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28b90 0x55d166fc0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc338dbf640 1 -- 172.21.15.44:0/609422299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d166f6ab30 con 0x55d166f2f7f0 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc338dbf640 1 --2- 172.21.15.44:0/609422299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f2f7f0 0x55d166f2b960 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc334161f60 tx=0x7fc3340cc7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a490254554cfb13 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc312ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc312ffd640 1 -- 172.21.15.44:0/609422299 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c019030 con 0x55d166f2f7f0 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc312ffd640 1 -- 172.21.15.44:0/609422299 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fc32c022650 con 0x55d166f2f7f0 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc33d409740 1 -- 172.21.15.44:0/609422299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc324051070 con 0x55d166f2f7f0 2024-04-02T13:25:04.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc312ffd640 1 -- 172.21.15.44:0/609422299 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fc32c032050 con 0x55d166f2f7f0 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc33d409740 10 client.7167 fetch_fsmap learned FSMap version 44 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc33d409740 1 -- 172.21.15.44:0/609422299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x55d166f68ea0 con 0x55d166f2f7f0 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc312ffd640 1 -- 172.21.15.44:0/609422299 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc324051070 con 0x55d166f2f7f0 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc312ffd640 1 -- 172.21.15.44:0/609422299 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 44) ==== 3494+0+0 (secure 0 0 0) 0x7fc32c035020 con 0x55d166f2f7f0 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc33d409740 10 client.7167 fetch_fsmap finished waiting for FSMap version 44 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc33d409740 10 client.7167 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc33d409740 20 client.7167 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc33d409740 1 --2- 172.21.15.44:0/609422299 >> [v2:172.21.15.69:6835/698185867,v1:172.21.15.69:6837/698185867] conn(0x55d166f15710 0x55d166f15af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc33d409740 4 client.7167 mds_command: new command op to 7002 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.709+0000 7fc33d409740 1 -- 172.21.15.44:0/609422299 --> [v2:172.21.15.69:6835/698185867,v1:172.21.15.69:6837/698185867] -- command(tid 0: {"prefix": "status"}) -- 0x55d166f35950 con 0x55d166f15710 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc333fff640 1 --2- 172.21.15.44:0/609422299 >> [v2:172.21.15.69:6835/698185867,v1:172.21.15.69:6837/698185867] conn(0x55d166f15710 0x55d166f15af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc333fff640 1 --2- 172.21.15.44:0/609422299 >> [v2:172.21.15.69:6835/698185867,v1:172.21.15.69:6837/698185867] conn(0x55d166f15710 0x55d166f15af0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a7b55f224a4078b server_cookie=b77c286212909bba in_seq=0 out_seq=0 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc312ffd640 10 client.7167 ms_handle_connect on v2:172.21.15.69:6835/698185867 2024-04-02T13:25:04.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc312ffd640 1 -- 172.21.15.44:0/609422299 <== mds.0 v2:172.21.15.69:6835/698185867 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x55d166f35950 con 0x55d166f15710 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc312ffd640 10 client.7167 handle_command_reply: tid=0 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc33d409740 1 client.7167 shutdown 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc33d409740 1 -- 172.21.15.44:0/609422299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f2f7f0 msgr2=0x55d166f2b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc33d409740 1 --2- 172.21.15.44:0/609422299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f2f7f0 0x55d166f2b960 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc334161f60 tx=0x7fc3340cc7d0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc33d409740 1 -- 172.21.15.44:0/609422299 shutdown_connections 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc33d409740 1 --2- 172.21.15.44:0/609422299 >> [v2:172.21.15.69:6835/698185867,v1:172.21.15.69:6837/698185867] conn(0x55d166f15710 0x55d166f15af0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc33d409740 1 --2- 172.21.15.44:0/609422299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d166f2f7f0 0x55d166f2b960 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc33d409740 1 --2- 172.21.15.44:0/609422299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d166f28f70 0x55d166f2b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc33d409740 1 --2- 172.21.15.44:0/609422299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d166f28b90 0x55d166fc0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc33d409740 1 -- 172.21.15.44:0/609422299 >> 172.21.15.44:0/609422299 conn(0x55d166efc070 msgr2=0x55d166f49770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc312ffd640 0 client.7167 ms_handle_reset on v2:172.21.15.69:6835/698185867 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.710+0000 7fc33d409740 1 -- 172.21.15.44:0/609422299 shutdown_connections 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33d409740 1 -- 172.21.15.44:0/609422299 wait complete. 2024-04-02T13:25:04.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33d409740 20 client.7167 trim_cache size 0 max 16384 2024-04-02T13:25:04.720 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:25:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:25:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:25:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 7002, 2024-04-02T13:25:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:25:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:25:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:25:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 10.15538684, 2024-04-02T13:25:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 44, 2024-04-02T13:25:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 79, 2024-04-02T13:25:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 79, 2024-04-02T13:25:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 15.467013396 2024-04-02T13:25:04.721 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:25:04.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33a822640 1 -- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc31406c920 msgr2=0x7fc31406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33a822640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc31406c920 0x7fc31406ede0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc324036000 tx=0x7fc324030000 comp rx=0 tx=0).stop 2024-04-02T13:25:04.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33a822640 1 -- 172.21.15.44:0/2511707672 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc320004800 msgr2=0x7fc3341514b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33a822640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc320004800 0x7fc3341514b0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00e7e0 tx=0x7fc31c00eca0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33a822640 1 -- 172.21.15.44:0/2511707672 shutdown_connections 2024-04-02T13:25:04.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33a822640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc31406c920 0x7fc31406ede0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33a822640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3340b72e0 0x7fc3340b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33a822640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3340cb870 0x7fc3340b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33a822640 1 --2- 172.21.15.44:0/2511707672 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc320004800 0x7fc3341514b0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33a822640 1 -- 172.21.15.44:0/2511707672 >> 172.21.15.44:0/2511707672 conn(0x7fc33409aa20 msgr2=0x7fc33409afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:04.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33a822640 1 -- 172.21.15.44:0/2511707672 shutdown_connections 2024-04-02T13:25:04.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.711+0000 7fc33a822640 1 -- 172.21.15.44:0/2511707672 wait complete. 2024-04-02T13:25:04.725 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:25:04.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.823+0000 7f679f9a3640 1 Processor -- start 2024-04-02T13:25:04.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.823+0000 7f679f9a3640 1 -- start start 2024-04-02T13:25:04.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.824+0000 7f679f9a3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f679815e140 0x7f67980d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.824+0000 7f679f9a3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67980d1150 0x7f67980b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.824+0000 7f679f9a3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67980b70b0 con 0x7f679815e520 2024-04-02T13:25:04.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.824+0000 7f679f9a3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f67980b9f50 con 0x7f679815e140 2024-04-02T13:25:04.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.824+0000 7f679f9a3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f67980b7390 con 0x7f67980d1150 2024-04-02T13:25:04.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.824+0000 7f679cf3e640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f679815e520 0x7f67980d0a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48418/0 (socket says 172.21.15.44:48418) 2024-04-02T13:25:04.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.824+0000 7f679cf3e640 1 -- 172.21.15.44:0/3831265975 learned_addr learned my addr 172.21.15.44:0/3831265975 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:04.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f679d73f640 1 --2- 172.21.15.44:0/3831265975 >> v2:172.21.15.69:3300/0 conn(0x7f679815e140 0x7f67980d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f679df40640 1 --2- 172.21.15.44:0/3831265975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67980d1150 0x7f67980b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1455769085 0 0) 0x7f67980b70b0 con 0x7f679815e520 2024-04-02T13:25:04.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6784003520 con 0x7f679815e520 2024-04-02T13:25:04.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1146661293 0 0) 0x7f6784003520 con 0x7f679815e520 2024-04-02T13:25:04.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67980d1150 msgr2=0x7f67980b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 --2- 172.21.15.44:0/3831265975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67980d1150 0x7f67980b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 >> v2:172.21.15.69:3300/0 conn(0x7f679815e140 msgr2=0x7f67980d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 --2- 172.21.15.44:0/3831265975 >> v2:172.21.15.69:3300/0 conn(0x7f679815e140 0x7f67980d2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67980b7670 con 0x7f679815e520 2024-04-02T13:25:04.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f6788001670 con 0x7f679815e520 2024-04-02T13:25:04.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 >> v1:172.21.15.44:6789/0 conn(0x7f679815e520 legacy=0x7f67980d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 --2- 172.21.15.44:0/3831265975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67840039b0 0x7f6784003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 --2- 172.21.15.44:0/3831265975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6784004800 0x7f6784004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 --2- 172.21.15.44:0/3831265975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67980d1150 0x7f67840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f67980b9f50 con 0x7f6784004800 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f67980b7390 con 0x7f67840039b0 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f678400b9c0 con 0x7f67980d1150 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f679cf3e640 1 --2- 172.21.15.44:0/3831265975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6784004800 0x7f6784004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f679d73f640 1 --2- 172.21.15.44:0/3831265975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67840039b0 0x7f6784003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f679df40640 1 --2- 172.21.15.44:0/3831265975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67980d1150 0x7f67840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f679cf3e640 1 -- 172.21.15.44:0/3831265975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67980d1150 msgr2=0x7f67840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f679cf3e640 1 --2- 172.21.15.44:0/3831265975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67980d1150 0x7f67840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f679cf3e640 1 -- 172.21.15.44:0/3831265975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67840039b0 msgr2=0x7f6784003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f679cf3e640 1 --2- 172.21.15.44:0/3831265975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67840039b0 0x7f6784003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.825+0000 7f679cf3e640 1 -- 172.21.15.44:0/3831265975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6784003520 con 0x7f6784004800 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.826+0000 7f679cf3e640 1 --2- 172.21.15.44:0/3831265975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6784004800 0x7f6784004c80 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f678800c240 tx=0x7f6784005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d1f3836c4cb78f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:04.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.826+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6788007d30 con 0x7f6784004800 2024-04-02T13:25:04.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.826+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6788030070 con 0x7f6784004800 2024-04-02T13:25:04.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.826+0000 7f678ffff640 1 -- 172.21.15.44:0/3831265975 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6788005d30 con 0x7f6784004800 2024-04-02T13:25:04.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.827+0000 7f679f9a3640 1 -- 172.21.15.44:0/3831265975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6784004800 msgr2=0x7f6784004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.827+0000 7f679f9a3640 1 --2- 172.21.15.44:0/3831265975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6784004800 0x7f6784004c80 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f678800c240 tx=0x7f6784005830 comp rx=0 tx=0).stop 2024-04-02T13:25:04.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.827+0000 7f679df40640 1 -- 172.21.15.44:0/3831265975 reap_dead start 2024-04-02T13:25:04.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.827+0000 7f679f9a3640 1 -- 172.21.15.44:0/3831265975 shutdown_connections 2024-04-02T13:25:04.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.827+0000 7f679f9a3640 1 -- 172.21.15.44:0/3831265975 >> 172.21.15.44:0/3831265975 conn(0x7f679809d390 msgr2=0x7f67980d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:04.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.827+0000 7f679f9a3640 1 -- 172.21.15.44:0/3831265975 shutdown_connections 2024-04-02T13:25:04.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.827+0000 7f679f9a3640 1 -- 172.21.15.44:0/3831265975 wait complete. 2024-04-02T13:25:04.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.828+0000 7f679f9a3640 1 Processor -- start 2024-04-02T13:25:04.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.828+0000 7f679f9a3640 1 -- start start 2024-04-02T13:25:04.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679f9a3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6784004800 0x7f679815a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679f9a3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f679809dd00 0x7f6798150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679f9a3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980cd530 0x7f67980cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679f9a3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f67980d2b70 con 0x7f679809dd00 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679f9a3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f67980d4ee0 con 0x7f67980cd530 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679f9a3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f67980d2cf0 con 0x7f6784004800 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679cf3e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f679809dd00 0x7f6798150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679cf3e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f679809dd00 0x7f6798150260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59858/0 (socket says 172.21.15.44:59858) 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679cf3e640 1 -- 172.21.15.44:0/1648655813 learned_addr learned my addr 172.21.15.44:0/1648655813 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679df40640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980cd530 0x7f67980cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679cf3e640 1 -- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6784004800 msgr2=0x7f679815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679d73f640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6784004800 0x7f679815a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679cf3e640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6784004800 0x7f679815a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679cf3e640 1 -- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980cd530 msgr2=0x7f67980cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:04.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679cf3e640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980cd530 0x7f67980cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:04.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679cf3e640 1 -- 172.21.15.44:0/1648655813 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f679809e2c0 con 0x7f679809dd00 2024-04-02T13:25:04.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.829+0000 7f679cf3e640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f679809dd00 0x7f6798150260 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f678800c330 tx=0x7f67980b7840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:04.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.830+0000 7f678dffb640 1 -- 172.21.15.44:0/1648655813 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6788045070 con 0x7f679809dd00 2024-04-02T13:25:04.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.830+0000 7f678dffb640 1 -- 172.21.15.44:0/1648655813 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6788004bf0 con 0x7f679809dd00 2024-04-02T13:25:04.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.830+0000 7f679f9a3640 1 -- 172.21.15.44:0/1648655813 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f679809e5a0 con 0x7f679809dd00 2024-04-02T13:25:04.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.830+0000 7f678dffb640 1 -- 172.21.15.44:0/1648655813 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6788001c70 con 0x7f679809dd00 2024-04-02T13:25:04.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.830+0000 7f679f9a3640 1 -- 172.21.15.44:0/1648655813 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f679809eb60 con 0x7f679809dd00 2024-04-02T13:25:04.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.831+0000 7f679f9a3640 1 -- 172.21.15.44:0/1648655813 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67980684f0 con 0x7f679809dd00 2024-04-02T13:25:04.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.831+0000 7f678dffb640 1 -- 172.21.15.44:0/1648655813 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6788007070 con 0x7f679809dd00 2024-04-02T13:25:04.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.831+0000 7f678dffb640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f677c06c920 0x7f677c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:04.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.832+0000 7f678dffb640 1 -- 172.21.15.44:0/1648655813 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f67880accd0 con 0x7f679809dd00 2024-04-02T13:25:04.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.832+0000 7f679d73f640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f677c06c920 0x7f677c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:04.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.832+0000 7f679d73f640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f677c06c920 0x7f677c06ede0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6790009a00 tx=0x7f6790008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:04.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:04.836+0000 7f678dffb640 1 -- 172.21.15.44:0/1648655813 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f678807b0e0 con 0x7f679809dd00 2024-04-02T13:25:05.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.058+0000 7f679f9a3640 1 -- 172.21.15.44:0/1648655813 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f67980cb100 con 0x7f679809dd00 2024-04-02T13:25:05.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.058+0000 7f678dffb640 1 -- 172.21.15.44:0/1648655813 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v44) ==== 53+0+73 (secure 0 0 0) 0x7f678807a9e0 con 0x7f679809dd00 2024-04-02T13:25:05.062 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:25:05.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.060+0000 7f679f9a3640 1 -- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f677c06c920 msgr2=0x7f677c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:05.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.060+0000 7f679f9a3640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f677c06c920 0x7f677c06ede0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6790009a00 tx=0x7f6790008040 comp rx=0 tx=0).stop 2024-04-02T13:25:05.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.060+0000 7f679f9a3640 1 -- 172.21.15.44:0/1648655813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f679809dd00 msgr2=0x7f6798150260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:05.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.060+0000 7f679f9a3640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f679809dd00 0x7f6798150260 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f678800c330 tx=0x7f67980b7840 comp rx=0 tx=0).stop 2024-04-02T13:25:05.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.060+0000 7f679f9a3640 1 -- 172.21.15.44:0/1648655813 shutdown_connections 2024-04-02T13:25:05.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.060+0000 7f679f9a3640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f677c06c920 0x7f677c06ede0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:05.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.060+0000 7f679f9a3640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980cd530 0x7f67980cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:05.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.060+0000 7f679f9a3640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f679809dd00 0x7f6798150260 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:05.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.060+0000 7f679f9a3640 1 --2- 172.21.15.44:0/1648655813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6784004800 0x7f679815a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:05.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.060+0000 7f679f9a3640 1 -- 172.21.15.44:0/1648655813 >> 172.21.15.44:0/1648655813 conn(0x7f679809d390 msgr2=0x7f679815e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:05.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.060+0000 7f679f9a3640 1 -- 172.21.15.44:0/1648655813 shutdown_connections 2024-04-02T13:25:05.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05.060+0000 7f679f9a3640 1 -- 172.21.15.44:0/1648655813 wait complete. 2024-04-02T13:25:05.074 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:25:05.074 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:25:05.074 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:25:05.074 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:25:05.074 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:05.074 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:25:05.074 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:25:05.074 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:25:05.074 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:25:05.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:05.075 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:25:05.089 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:25:05.090 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:25:05.090 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:25:05.090 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:05.090 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:25:05.090 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:05.090 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:25:05.090 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:25:05.090 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:25:05.090 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:25:05.090 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:25:05.090 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:25:05.090 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42474sec preferred_lft 42474sec 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:05.091 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:25:05.092 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:25:05.092 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:25:05.092 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:05.092 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:25:05.092 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:25:05.092 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:25:05.092 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:05.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:05.092 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:25:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:25:05.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:05.156 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:25:05.212 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:25:05.212 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:25:05.212 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:25:05.212 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:25:05.212 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:25:05.212 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:25:05.212 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:25:05.213 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:25:05.213 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:25:05.213 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:25:05.213 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:25:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:25:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:25:05.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:05.347 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:25:05.361 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:25:05.361 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:25:05.361 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:25:05.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:05.362 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:25:05.430 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:25:05.430 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:25:05.430 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:05.484 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:25:05.485 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:25:05.485 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:05.539 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:25:05.607 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:25:05.662 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:25:05.662 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:25:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:25:05.664 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:25:05.665 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:25:05.665 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:25:05.665 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:25:05.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:05.665 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:25:05.717 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:25:05.717 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:25:05.760 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:25:05.786 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:25:05.843 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:25:05.843 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:25:05.843 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:25:05.843 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:25:05.843 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:25:05.843 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:25:05.843 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:25:05.843 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:25:05.843 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:25:05.843 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:25:05.844 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:25:05.844 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:25:05.844 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:25:05.844 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:25:05.844 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:25:05.844 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:25:05.844 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:25:05.844 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:25:05.844 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:25:05.844 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:25:05.844 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:25:05.844 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:25:05.844 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:25:05.845 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:25:05.845 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:25:05.845 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:25:05.845 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:25:05.845 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:25:05.845 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:25:05.845 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:25:05.845 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:25:05.845 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:25:05.845 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:25:05.845 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:25:05.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:05.846 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:25:05.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:25:05.842+0000 7fd7cbc6a480 -1 init, newargv = 0x557b75099620 newargc=15 2024-04-02T13:25:05.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[64068]: starting ceph client 2024-04-02T13:25:05.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[64068]: starting fuse 2024-04-02T13:25:05.861 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:25:05.861 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:25:05.861 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:25:05.861 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:25:05.861 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:25:05.861 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:25:05.861 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:25:05.861 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:25:05.861 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:25:05.862 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:25:05.862 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:25:05.862 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:25:05.862 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:25:05.862 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:25:05.862 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:25:05.862 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:25:05.862 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:25:05.862 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:25:05.862 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:25:05.862 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:25:05.862 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:25:05.862 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:25:05.863 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:25:05.863 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:25:05.863 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:25:05.863 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:25:05.863 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:25:05.863 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:25:05.863 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:25:05.981 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.64068.asok 2024-04-02T13:25:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:25:05.991 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.64068.asok 2024-04-02T13:25:05.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:05.991 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.64068.asok status 2024-04-02T13:25:06.129 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:25:06.129 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:25:06.129 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:25:06.129 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:25:06.129 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:25:06.129 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:25:06.129 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:25:06.129 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "64068", 2024-04-02T13:25:06.129 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:25:06.130 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:25:06.130 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:25:06.130 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:25:06.130 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 7179, 2024-04-02T13:25:06.130 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:25:06.130 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:25:06.130 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:25:06.130 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 7179 2024-04-02T13:25:06.130 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:25:06.130 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:25:06.131 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:25:06.131 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:25:06.131 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2502170950 2024-04-02T13:25:06.131 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:25:06.131 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:25:06.131 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:25:06.131 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:25:06.131 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:25:06.131 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2502170950 2024-04-02T13:25:06.131 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:25:06.131 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.7179 192.168.0.1:0/2502170950", 2024-04-02T13:25:06.131 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/2502170950", 2024-04-02T13:25:06.132 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:25:06.132 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 44, 2024-04-02T13:25:06.132 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 79, 2024-04-02T13:25:06.132 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:25:06.132 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:25:06.132 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:25:06.132 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:25:06.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:06.137 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:25:06.137 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:06.152 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:25:06.152 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:06.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:06.153 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:06.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:06.221 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir/) 2024-04-02T13:25:06.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:06.283 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_undamaged) 2024-04-02T13:25:06.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:06.342 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_to_be_damaged) 2024-04-02T13:25:06.398 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:25:06.399 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:25:06.399 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:25:06.399 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:25:06.399 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir").st_ino) 2024-04-02T13:25:06.399 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:25:06.504 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627776 2024-04-02T13:25:06.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:25:06.513 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:25:06.513 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:25:06.568 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:25:06.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.706+0000 7f129a71c640 1 Processor -- start 2024-04-02T13:25:06.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.706+0000 7f129a71c640 1 -- start start 2024-04-02T13:25:06.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.707+0000 7f129a71c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f129415e4a0 0x7f12940aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:06.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.707+0000 7f129a71c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f12940d0c50 0x7f12940d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:06.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.707+0000 7f129a71c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f129406ce40 0x7f129406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.707+0000 7f129a71c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f129406d7e0 con 0x7f12940d0c50 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.707+0000 7f129a71c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f12940cc600 con 0x7f129406ce40 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.707+0000 7f129a71c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f12940cc8e0 con 0x7f129415e4a0 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.707+0000 7f12937fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f12940d0c50 0x7f12940d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.707+0000 7f12937fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f12940d0c50 0x7f12940d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59868/0 (socket says 172.21.15.44:59868) 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.707+0000 7f12937fe640 1 -- 172.21.15.44:0/3934594558 learned_addr learned my addr 172.21.15.44:0/3934594558 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.708+0000 7f12937fe640 1 -- 172.21.15.44:0/3934594558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f129415e4a0 msgr2=0x7f12940aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.708+0000 7f12937fe640 1 --2- 172.21.15.44:0/3934594558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f129415e4a0 0x7f12940aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.708+0000 7f12937fe640 1 -- 172.21.15.44:0/3934594558 >> v2:172.21.15.69:3300/0 conn(0x7f129406ce40 msgr2=0x7f129406d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.708+0000 7f12937fe640 1 --2- 172.21.15.44:0/3934594558 >> v2:172.21.15.69:3300/0 conn(0x7f129406ce40 0x7f129406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.708+0000 7f12937fe640 1 -- 172.21.15.44:0/3934594558 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12940ccbc0 con 0x7f12940d0c50 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.708+0000 7f12937fe640 1 --2- 172.21.15.44:0/3934594558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12940d0c50 0x7f12940d10b0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f12940750f0 tx=0x7f128802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27dd5ae0cb7dd295 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.709+0000 7f1292ffd640 1 -- 172.21.15.44:0/3934594558 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288035020 con 0x7f12940d0c50 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.709+0000 7f1292ffd640 1 -- 172.21.15.44:0/3934594558 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1288031870 con 0x7f12940d0c50 2024-04-02T13:25:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.709+0000 7f1292ffd640 1 -- 172.21.15.44:0/3934594558 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288031c40 con 0x7f12940d0c50 2024-04-02T13:25:06.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.709+0000 7f129a71c640 1 -- 172.21.15.44:0/3934594558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12940d0c50 msgr2=0x7f12940d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:06.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.709+0000 7f129a71c640 1 --2- 172.21.15.44:0/3934594558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12940d0c50 0x7f12940d10b0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f12940750f0 tx=0x7f128802b910 comp rx=0 tx=0).stop 2024-04-02T13:25:06.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.709+0000 7f129a71c640 1 -- 172.21.15.44:0/3934594558 shutdown_connections 2024-04-02T13:25:06.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.709+0000 7f129a71c640 1 --2- 172.21.15.44:0/3934594558 >> v2:172.21.15.69:3300/0 conn(0x7f129406ce40 0x7f129406d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:06.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.709+0000 7f129a71c640 1 --2- 172.21.15.44:0/3934594558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12940d0c50 0x7f12940d10b0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.709+0000 7f129a71c640 1 --2- 172.21.15.44:0/3934594558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f129415e4a0 0x7f12940aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.709+0000 7f129a71c640 1 -- 172.21.15.44:0/3934594558 >> 172.21.15.44:0/3934594558 conn(0x7f129409aa20 msgr2=0x7f129409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.709+0000 7f129a71c640 1 -- 172.21.15.44:0/3934594558 shutdown_connections 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.709+0000 7f129a71c640 1 -- 172.21.15.44:0/3934594558 wait complete. 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.710+0000 7f129a71c640 1 Processor -- start 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.711+0000 7f129a71c640 1 -- start start 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.711+0000 7f129a71c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129406ce40 0x7f12941586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.711+0000 7f129a71c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12940d0c50 0x7f12940ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.711+0000 7f129a71c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f129415e4a0 0x7f12940ca700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.711+0000 7f129a71c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f12940cd460 con 0x7f129406ce40 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.711+0000 7f129a71c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f12940cd180 con 0x7f12940d0c50 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.711+0000 7f129a71c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f129415e880 con 0x7f129415e4a0 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.711+0000 7f1293fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129406ce40 0x7f12941586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.711+0000 7f1293fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129406ce40 0x7f12941586d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59880/0 (socket says 172.21.15.44:59880) 2024-04-02T13:25:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.711+0000 7f1293fff640 1 -- 172.21.15.44:0/939735379 learned_addr learned my addr 172.21.15.44:0/939735379 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.712+0000 7f1293fff640 1 -- 172.21.15.44:0/939735379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f129415e4a0 msgr2=0x7f12940ca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.712+0000 7f1293fff640 1 --2- 172.21.15.44:0/939735379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f129415e4a0 0x7f12940ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.712+0000 7f1293fff640 1 -- 172.21.15.44:0/939735379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12940d0c50 msgr2=0x7f12940ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.712+0000 7f1293fff640 1 --2- 172.21.15.44:0/939735379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12940d0c50 0x7f12940ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.712+0000 7f1293fff640 1 -- 172.21.15.44:0/939735379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1294158c10 con 0x7f129406ce40 2024-04-02T13:25:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.712+0000 7f1293fff640 1 --2- 172.21.15.44:0/939735379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129406ce40 0x7f12941586d0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f128400b4d0 tx=0x7f128400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.712+0000 7f1290ff9640 1 -- 172.21.15.44:0/939735379 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1284014070 con 0x7f129406ce40 2024-04-02T13:25:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.712+0000 7f1290ff9640 1 -- 172.21.15.44:0/939735379 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1284010870 con 0x7f129406ce40 2024-04-02T13:25:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.712+0000 7f129a71c640 1 -- 172.21.15.44:0/939735379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f129415ebe0 con 0x7f129406ce40 2024-04-02T13:25:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.712+0000 7f1290ff9640 1 -- 172.21.15.44:0/939735379 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1284010c60 con 0x7f129406ce40 2024-04-02T13:25:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.712+0000 7f129a71c640 1 -- 172.21.15.44:0/939735379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f129415f1a0 con 0x7f129406ce40 2024-04-02T13:25:06.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.713+0000 7f129a71c640 1 -- 172.21.15.44:0/939735379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12940684f0 con 0x7f129406ce40 2024-04-02T13:25:06.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.713+0000 7f1290ff9640 1 -- 172.21.15.44:0/939735379 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1284017070 con 0x7f129406ce40 2024-04-02T13:25:06.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.714+0000 7f1290ff9640 1 --2- 172.21.15.44:0/939735379 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f128006c8f0 0x7f128006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:06.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.714+0000 7f1290ff9640 1 -- 172.21.15.44:0/939735379 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f128408ba10 con 0x7f129406ce40 2024-04-02T13:25:06.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.714+0000 7f12937fe640 1 --2- 172.21.15.44:0/939735379 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f128006c8f0 0x7f128006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:06.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.715+0000 7f12937fe640 1 --2- 172.21.15.44:0/939735379 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f128006c8f0 0x7f128006edb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1288036000 tx=0x7f1288030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:06.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.718+0000 7f1290ff9640 1 -- 172.21.15.44:0/939735379 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1284059d40 con 0x7f129406ce40 2024-04-02T13:25:06.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.905+0000 7f129a71c640 1 -- 172.21.15.44:0/939735379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f12940ce550 con 0x7f129406ce40 2024-04-02T13:25:06.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.907+0000 7f1290ff9640 1 -- 172.21.15.44:0/939735379 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+16897 (secure 0 0 0) 0x7f1284059640 con 0x7f129406ce40 2024-04-02T13:25:06.910 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:06.910 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":79,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:24:55.564387+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:24:52.202720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-02T13:24:52.916529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:25:06.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.909+0000 7f129a71c640 1 -- 172.21.15.44:0/939735379 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f128006c8f0 msgr2=0x7f128006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:06.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.909+0000 7f129a71c640 1 --2- 172.21.15.44:0/939735379 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f128006c8f0 0x7f128006edb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1288036000 tx=0x7f1288030000 comp rx=0 tx=0).stop 2024-04-02T13:25:06.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.909+0000 7f129a71c640 1 -- 172.21.15.44:0/939735379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129406ce40 msgr2=0x7f12941586d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:06.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.909+0000 7f129a71c640 1 --2- 172.21.15.44:0/939735379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129406ce40 0x7f12941586d0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f128400b4d0 tx=0x7f128400b990 comp rx=0 tx=0).stop 2024-04-02T13:25:06.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.909+0000 7f129a71c640 1 -- 172.21.15.44:0/939735379 shutdown_connections 2024-04-02T13:25:06.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.909+0000 7f129a71c640 1 --2- 172.21.15.44:0/939735379 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f128006c8f0 0x7f128006edb0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:06.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.909+0000 7f129a71c640 1 --2- 172.21.15.44:0/939735379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f129415e4a0 0x7f12940ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:06.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.909+0000 7f129a71c640 1 --2- 172.21.15.44:0/939735379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12940d0c50 0x7f12940ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:06.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.909+0000 7f129a71c640 1 --2- 172.21.15.44:0/939735379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129406ce40 0x7f12941586d0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:06.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.909+0000 7f129a71c640 1 -- 172.21.15.44:0/939735379 >> 172.21.15.44:0/939735379 conn(0x7f129409aa20 msgr2=0x7f12940b0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:06.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.909+0000 7f129a71c640 1 -- 172.21.15.44:0/939735379 shutdown_connections 2024-04-02T13:25:06.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:06.909+0000 7f129a71c640 1 -- 172.21.15.44:0/939735379 wait complete. 2024-04-02T13:25:06.923 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:24:52.202720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:24:52.916529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:25:06.923 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:25:07.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.025+0000 7fe56c51c640 1 Processor -- start 2024-04-02T13:25:07.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.025+0000 7fe56c51c640 1 -- start start 2024-04-02T13:25:07.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.025+0000 7fe56c51c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe56406cff0 0x7fe56406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.025+0000 7fe56c51c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe56415e5d0 0x7fe5640d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe56c51c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe56406d910 con 0x7fe5640b4180 2024-04-02T13:25:07.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe56c51c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe5640d1bf0 con 0x7fe56406cff0 2024-04-02T13:25:07.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe56c51c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe5640d1630 con 0x7fe56415e5d0 2024-04-02T13:25:07.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe56aab9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe5640b4180 0x7fe5640ce860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48430/0 (socket says 172.21.15.44:48430) 2024-04-02T13:25:07.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe56aab9640 1 -- 172.21.15.44:0/2060689003 learned_addr learned my addr 172.21.15.44:0/2060689003 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:07.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2011411932 0 0) 0x7fe56406d910 con 0x7fe5640b4180 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe550003520 con 0x7fe5640b4180 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe56a2b8640 1 --2- 172.21.15.44:0/2060689003 >> v2:172.21.15.69:3300/0 conn(0x7fe56406cff0 0x7fe56406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe569ab7640 1 --2- 172.21.15.44:0/2060689003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe56415e5d0 0x7fe5640d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4250808459 0 0) 0x7fe550003520 con 0x7fe5640b4180 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe56415e5d0 msgr2=0x7fe5640d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 --2- 172.21.15.44:0/2060689003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe56415e5d0 0x7fe5640d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 >> v2:172.21.15.69:3300/0 conn(0x7fe56406cff0 msgr2=0x7fe56406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 --2- 172.21.15.44:0/2060689003 >> v2:172.21.15.69:3300/0 conn(0x7fe56406cff0 0x7fe56406d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5640d1910 con 0x7fe5640b4180 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fe55c0019f0 con 0x7fe5640b4180 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 >> v1:172.21.15.44:6789/0 conn(0x7fe5640b4180 legacy=0x7fe5640ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 --2- 172.21.15.44:0/2060689003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe56415e5d0 0x7fe5500039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 --2- 172.21.15.44:0/2060689003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe550003f30 0x7fe550004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 --2- 172.21.15.44:0/2060689003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe550004d40 0x7fe5500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe5640d1bf0 con 0x7fe550004d40 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe5640d1630 con 0x7fe550003f30 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe55000ba10 con 0x7fe56415e5d0 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.026+0000 7fe56aab9640 1 --2- 172.21.15.44:0/2060689003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe550004d40 0x7fe5500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.027+0000 7fe569ab7640 1 --2- 172.21.15.44:0/2060689003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe550003f30 0x7fe550004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.027+0000 7fe56a2b8640 1 --2- 172.21.15.44:0/2060689003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe56415e5d0 0x7fe5500039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.027+0000 7fe56aab9640 1 -- 172.21.15.44:0/2060689003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe56415e5d0 msgr2=0x7fe5500039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.027+0000 7fe56aab9640 1 --2- 172.21.15.44:0/2060689003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe56415e5d0 0x7fe5500039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.027+0000 7fe56aab9640 1 -- 172.21.15.44:0/2060689003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe550003f30 msgr2=0x7fe550004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.027+0000 7fe56aab9640 1 --2- 172.21.15.44:0/2060689003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe550003f30 0x7fe550004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.027+0000 7fe56aab9640 1 -- 172.21.15.44:0/2060689003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe550003520 con 0x7fe550004d40 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.027+0000 7fe56a2b8640 1 --2- 172.21.15.44:0/2060689003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe56415e5d0 0x7fe5500039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.027+0000 7fe56aab9640 1 --2- 172.21.15.44:0/2060689003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe550004d40 0x7fe5500051c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00a870 tx=0x7fe550005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2778bfa0b2f731ae server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:07.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.028+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c00a4c0 con 0x7fe550004d40 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.028+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe55c034630 con 0x7fe550004d40 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.028+0000 7fe5692b6640 1 -- 172.21.15.44:0/2060689003 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c034a20 con 0x7fe550004d40 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.028+0000 7fe56c51c640 1 -- 172.21.15.44:0/2060689003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe550004d40 msgr2=0x7fe5500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.028+0000 7fe56c51c640 1 --2- 172.21.15.44:0/2060689003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe550004d40 0x7fe5500051c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00a870 tx=0x7fe550005830 comp rx=0 tx=0).stop 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.028+0000 7fe56aab9640 1 -- 172.21.15.44:0/2060689003 reap_dead start 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.028+0000 7fe56c51c640 1 -- 172.21.15.44:0/2060689003 shutdown_connections 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.028+0000 7fe56c51c640 1 -- 172.21.15.44:0/2060689003 >> 172.21.15.44:0/2060689003 conn(0x7fe56409aa20 msgr2=0x7fe56409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.028+0000 7fe56c51c640 1 -- 172.21.15.44:0/2060689003 shutdown_connections 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.028+0000 7fe56c51c640 1 -- 172.21.15.44:0/2060689003 wait complete. 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.029+0000 7fe56c51c640 1 Processor -- start 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.029+0000 7fe56c51c640 1 -- start start 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56c51c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe550004d40 0x7fe564156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56c51c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe564153500 0x7fe56415ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56c51c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5640c8600 0x7fe5640c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56c51c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe5640b2d50 con 0x7fe550004d40 2024-04-02T13:25:07.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56c51c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe5640b1820 con 0x7fe564153500 2024-04-02T13:25:07.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56c51c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe5640b2ed0 con 0x7fe5640c8600 2024-04-02T13:25:07.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56a2b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe550004d40 0x7fe564156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56a2b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe550004d40 0x7fe564156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59898/0 (socket says 172.21.15.44:59898) 2024-04-02T13:25:07.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56a2b8640 1 -- 172.21.15.44:0/1693850989 learned_addr learned my addr 172.21.15.44:0/1693850989 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:07.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56aab9640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5640c8600 0x7fe5640c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe569ab7640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe564153500 0x7fe56415ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56a2b8640 1 -- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5640c8600 msgr2=0x7fe5640c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56a2b8640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5640c8600 0x7fe5640c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56a2b8640 1 -- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe564153500 msgr2=0x7fe56415ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56a2b8640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe564153500 0x7fe56415ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.030+0000 7fe56a2b8640 1 -- 172.21.15.44:0/1693850989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe564156870 con 0x7fe550004d40 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.031+0000 7fe56a2b8640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe550004d40 0x7fe564156280 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe5600116d0 tx=0x7fe560011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.031+0000 7fe55affd640 1 -- 172.21.15.44:0/1693850989 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56000f9c0 con 0x7fe550004d40 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.031+0000 7fe55affd640 1 -- 172.21.15.44:0/1693850989 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe56000fb60 con 0x7fe550004d40 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.031+0000 7fe56c51c640 1 -- 172.21.15.44:0/1693850989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe564156b50 con 0x7fe550004d40 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.031+0000 7fe55affd640 1 -- 172.21.15.44:0/1693850989 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe560015460 con 0x7fe550004d40 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.031+0000 7fe56c51c640 1 -- 172.21.15.44:0/1693850989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5640c5f90 con 0x7fe550004d40 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.032+0000 7fe56c51c640 1 -- 172.21.15.44:0/1693850989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5640684f0 con 0x7fe550004d40 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.032+0000 7fe55affd640 1 -- 172.21.15.44:0/1693850989 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe560015600 con 0x7fe550004d40 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.033+0000 7fe55affd640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe54806c8f0 0x7fe54806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.033+0000 7fe55affd640 1 -- 172.21.15.44:0/1693850989 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fe56008c700 con 0x7fe550004d40 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.033+0000 7fe569ab7640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe54806c8f0 0x7fe54806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.033+0000 7fe569ab7640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe54806c8f0 0x7fe54806edb0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe5640d0200 tx=0x7fe554008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.037+0000 7fe55affd640 1 -- 172.21.15.44:0/1693850989 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe560019030 con 0x7fe550004d40 2024-04-02T13:25:07.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.220+0000 7fe56c51c640 1 -- 172.21.15.44:0/1693850989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe5640c8140 con 0x7fe550004d40 2024-04-02T13:25:07.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.220+0000 7fe55affd640 1 -- 172.21.15.44:0/1693850989 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) ==== 93+0+10 (secure 0 0 0) 0x7fe56005a9a0 con 0x7fe550004d40 2024-04-02T13:25:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:25:07.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.222+0000 7fe56c51c640 1 -- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe54806c8f0 msgr2=0x7fe54806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.222+0000 7fe56c51c640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe54806c8f0 0x7fe54806edb0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe5640d0200 tx=0x7fe554008040 comp rx=0 tx=0).stop 2024-04-02T13:25:07.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.222+0000 7fe56c51c640 1 -- 172.21.15.44:0/1693850989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe550004d40 msgr2=0x7fe564156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.222+0000 7fe56c51c640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe550004d40 0x7fe564156280 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe5600116d0 tx=0x7fe560011b90 comp rx=0 tx=0).stop 2024-04-02T13:25:07.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.222+0000 7fe56c51c640 1 -- 172.21.15.44:0/1693850989 shutdown_connections 2024-04-02T13:25:07.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.222+0000 7fe56c51c640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe54806c8f0 0x7fe54806edb0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.222+0000 7fe56c51c640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5640c8600 0x7fe5640c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.222+0000 7fe56c51c640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe564153500 0x7fe56415ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.222+0000 7fe56c51c640 1 --2- 172.21.15.44:0/1693850989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe550004d40 0x7fe564156280 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.222+0000 7fe56c51c640 1 -- 172.21.15.44:0/1693850989 >> 172.21.15.44:0/1693850989 conn(0x7fe56409aa20 msgr2=0x7fe5640aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:07.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.222+0000 7fe56c51c640 1 -- 172.21.15.44:0/1693850989 shutdown_connections 2024-04-02T13:25:07.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.222+0000 7fe56c51c640 1 -- 172.21.15.44:0/1693850989 wait complete. 2024-04-02T13:25:07.362 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:25:07.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.465+0000 7f9fa1c33640 1 Processor -- start 2024-04-02T13:25:07.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.465+0000 7f9fa1c33640 1 -- start start 2024-04-02T13:25:07.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.465+0000 7f9fa1c33640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9f9c0cec60 0x7f9f9c0d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.465+0000 7f9fa1c33640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f9c0cf1f0 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.465+0000 7f9fa1c33640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f9c0cf4d0 con 0x7f9f9c15e5b0 2024-04-02T13:25:07.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.465+0000 7f9fa1c33640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f9f9c0cfe00 con 0x7f9f9c0cec60 2024-04-02T13:25:07.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.466+0000 7f9f9affd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9f9c0cec60 0x7f9f9c0d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.466+0000 7f9f9b7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f9f9c15e1d0 0x7f9f9c0d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48444/0 (socket says 172.21.15.44:48444) 2024-04-02T13:25:07.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.466+0000 7f9f9b7fe640 1 -- 172.21.15.44:0/3222205673 learned_addr learned my addr 172.21.15.44:0/3222205673 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:07.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.466+0000 7f9f9affd640 1 --2- 172.21.15.44:0/3222205673 >> v2:172.21.15.44:3300/0 conn(0x7f9f9c0cec60 0x7f9f9c0d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59908/0 (socket says 172.21.15.44:59908) 2024-04-02T13:25:07.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.466+0000 7f9f9affd640 1 -- 172.21.15.44:0/3222205673 >> v1:172.21.15.69:6789/0 conn(0x7f9f9c15e5b0 legacy=0x7f9f9c0b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:07.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.466+0000 7f9f9affd640 1 -- 172.21.15.44:0/3222205673 >> v1:172.21.15.44:6789/0 conn(0x7f9f9c15e1d0 legacy=0x7f9f9c0d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.466+0000 7f9f9a7fc640 1 -- 172.21.15.44:0/3222205673 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2412926241 0 0) 0x7f9f9c0cf1f0 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.466+0000 7f9f9affd640 1 -- 172.21.15.44:0/3222205673 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f9c0cf7b0 con 0x7f9f9c0cec60 2024-04-02T13:25:07.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.467+0000 7f9f9affd640 1 --2- 172.21.15.44:0/3222205673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c0cec60 0x7f9f9c0d26c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9f900097a0 tx=0x7f9f9002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b915cc63a7e70bf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:07.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.467+0000 7f9f9a7fc640 1 -- 172.21.15.44:0/3222205673 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f900315a0 con 0x7f9f9c0cec60 2024-04-02T13:25:07.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.467+0000 7f9f9a7fc640 1 -- 172.21.15.44:0/3222205673 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f90037070 con 0x7f9f9c0cec60 2024-04-02T13:25:07.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.467+0000 7f9f9a7fc640 1 -- 172.21.15.44:0/3222205673 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90031960 con 0x7f9f9c0cec60 2024-04-02T13:25:07.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.467+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3222205673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c0cec60 msgr2=0x7f9f9c0d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.467+0000 7f9fa1c33640 1 --2- 172.21.15.44:0/3222205673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c0cec60 0x7f9f9c0d26c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9f900097a0 tx=0x7f9f9002b840 comp rx=0 tx=0).stop 2024-04-02T13:25:07.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.467+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3222205673 shutdown_connections 2024-04-02T13:25:07.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.467+0000 7f9fa1c33640 1 --2- 172.21.15.44:0/3222205673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c0cec60 0x7f9f9c0d26c0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.467+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3222205673 >> 172.21.15.44:0/3222205673 conn(0x7f9f9c09d3b0 msgr2=0x7f9f9c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.467+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3222205673 shutdown_connections 2024-04-02T13:25:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.467+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3222205673 wait complete. 2024-04-02T13:25:07.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.468+0000 7f9fa1c33640 1 Processor -- start 2024-04-02T13:25:07.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.469+0000 7f9fa1c33640 1 -- start start 2024-04-02T13:25:07.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.469+0000 7f9fa1c33640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f9c0cec60 0x7f9f9c1e7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.469+0000 7f9fa1c33640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c15e1d0 0x7f9f9c1612a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.469+0000 7f9fa1c33640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f9c15e5b0 0x7f9f9c15f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.469+0000 7f9fa1c33640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c0b8150 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.469+0000 7f9fa1c33640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c0b7fd0 con 0x7f9f9c15e5b0 2024-04-02T13:25:07.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.469+0000 7f9fa1c33640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9f9c0b82d0 con 0x7f9f9c0cec60 2024-04-02T13:25:07.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.469+0000 7f9f9affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c15e1d0 0x7f9f9c1612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.469+0000 7f9f9affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c15e1d0 0x7f9f9c1612a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59914/0 (socket says 172.21.15.44:59914) 2024-04-02T13:25:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.469+0000 7f9f9affd640 1 -- 172.21.15.44:0/3085057030 learned_addr learned my addr 172.21.15.44:0/3085057030 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.470+0000 7f9f9affd640 1 -- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f9c0cec60 msgr2=0x7f9f9c1e7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.470+0000 7f9f9affd640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f9c0cec60 0x7f9f9c1e7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.470+0000 7f9f9bfff640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f9c15e5b0 0x7f9f9c15f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.470+0000 7f9f9affd640 1 -- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f9c15e5b0 msgr2=0x7f9f9c15f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.470+0000 7f9f9affd640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f9c15e5b0 0x7f9f9c15f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.470+0000 7f9f9affd640 1 -- 172.21.15.44:0/3085057030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f9c1e8300 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.470+0000 7f9f9bfff640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f9c15e5b0 0x7f9f9c15f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.470+0000 7f9f9affd640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c15e1d0 0x7f9f9c1612a0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f9f90034040 tx=0x7f9f900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.470+0000 7f9f83fff640 1 -- 172.21.15.44:0/3085057030 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90032840 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.470+0000 7f9f83fff640 1 -- 172.21.15.44:0/3085057030 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f900329e0 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.470+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3085057030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f9c14bf60 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.470+0000 7f9f83fff640 1 -- 172.21.15.44:0/3085057030 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90032dd0 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.470+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3085057030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f9c14c800 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.471+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3085057030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f9c0684f0 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.472+0000 7f9f83fff640 1 -- 172.21.15.44:0/3085057030 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9f9003b3f0 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.472+0000 7f9f83fff640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9f8806c8f0 0x7f9f8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.472+0000 7f9f83fff640 1 -- 172.21.15.44:0/3085057030 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f9f9002f030 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.472+0000 7f9f9b7fe640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9f8806c8f0 0x7f9f8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.473+0000 7f9f9b7fe640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9f8806c8f0 0x7f9f8806edb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c007fd0 tx=0x7f9f8c007960 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:07.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.477+0000 7f9f83fff640 1 -- 172.21.15.44:0/3085057030 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9f90040070 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.662+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3085057030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9f9c0b6900 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.663+0000 7f9f83fff640 1 -- 172.21.15.44:0/3085057030 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) ==== 104+0+11 (secure 0 0 0) 0x7f9f9007bf50 con 0x7f9f9c15e1d0 2024-04-02T13:25:07.666 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:25:07.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.665+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9f8806c8f0 msgr2=0x7f9f8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.665+0000 7f9fa1c33640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9f8806c8f0 0x7f9f8806edb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c007fd0 tx=0x7f9f8c007960 comp rx=0 tx=0).stop 2024-04-02T13:25:07.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.665+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3085057030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c15e1d0 msgr2=0x7f9f9c1612a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.665+0000 7f9fa1c33640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c15e1d0 0x7f9f9c1612a0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f9f90034040 tx=0x7f9f900046c0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.665+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3085057030 shutdown_connections 2024-04-02T13:25:07.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.665+0000 7f9fa1c33640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9f8806c8f0 0x7f9f8806edb0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.665+0000 7f9fa1c33640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f9c15e5b0 0x7f9f9c15f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.665+0000 7f9fa1c33640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c15e1d0 0x7f9f9c1612a0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.665+0000 7f9fa1c33640 1 --2- 172.21.15.44:0/3085057030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f9c0cec60 0x7f9f9c1e7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.665+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3085057030 >> 172.21.15.44:0/3085057030 conn(0x7f9f9c09d3b0 msgr2=0x7f9f9c0b9710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:07.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.665+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3085057030 shutdown_connections 2024-04-02T13:25:07.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.665+0000 7f9fa1c33640 1 -- 172.21.15.44:0/3085057030 wait complete. 2024-04-02T13:25:07.678 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:25:07.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.780+0000 7fd5a1f9d640 1 Processor -- start 2024-04-02T13:25:07.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.780+0000 7fd5a1f9d640 1 -- start start 2024-04-02T13:25:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd5a1f9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd59c15e240 0x7fd59c0d0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd5a1f9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd59c0d0e60 0x7fd59c0cea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd5a1f9d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd59c0c7a80 con 0x7fd59c0cef40 2024-04-02T13:25:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd5a1f9d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd59c0c7dd0 con 0x7fd59c0d0e60 2024-04-02T13:25:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd5a1f9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd59c0d12a0 con 0x7fd59c15e240 2024-04-02T13:25:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd5a179c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd59c0cef40 0x7fd59c0d1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48460/0 (socket says 172.21.15.44:48460) 2024-04-02T13:25:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd5a179c640 1 -- 172.21.15.44:0/1844699600 learned_addr learned my addr 172.21.15.44:0/1844699600 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:07.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd5a0f9b640 1 --2- 172.21.15.44:0/1844699600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd59c15e240 0x7fd59c0d0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd59bfff640 1 --2- 172.21.15.44:0/1844699600 >> v2:172.21.15.69:3300/0 conn(0x7fd59c0d0e60 0x7fd59c0cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd59b7fe640 1 -- 172.21.15.44:0/1844699600 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 45793938 0 0) 0x7fd59c0c7a80 con 0x7fd59c0cef40 2024-04-02T13:25:07.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd59b7fe640 1 -- 172.21.15.44:0/1844699600 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd58c003520 con 0x7fd59c0cef40 2024-04-02T13:25:07.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd5a0f9b640 1 -- 172.21.15.44:0/1844699600 >> v2:172.21.15.69:3300/0 conn(0x7fd59c0d0e60 msgr2=0x7fd59c0cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:07.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd5a0f9b640 1 --2- 172.21.15.44:0/1844699600 >> v2:172.21.15.69:3300/0 conn(0x7fd59c0d0e60 0x7fd59c0cea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd5a0f9b640 1 -- 172.21.15.44:0/1844699600 >> v1:172.21.15.44:6789/0 conn(0x7fd59c0cef40 legacy=0x7fd59c0d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd5a0f9b640 1 -- 172.21.15.44:0/1844699600 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd59c0cf830 con 0x7fd59c15e240 2024-04-02T13:25:07.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.781+0000 7fd59bfff640 1 --2- 172.21.15.44:0/1844699600 >> v2:172.21.15.69:3300/0 conn(0x7fd59c0d0e60 0x7fd59c0cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:25:07.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.782+0000 7fd5a0f9b640 1 --2- 172.21.15.44:0/1844699600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd59c15e240 0x7fd59c0d0920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd5900097a0 tx=0x7fd59002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4cd8b4491b1a9a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:07.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.782+0000 7fd59b7fe640 1 -- 172.21.15.44:0/1844699600 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5900315a0 con 0x7fd59c15e240 2024-04-02T13:25:07.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.782+0000 7fd59b7fe640 1 -- 172.21.15.44:0/1844699600 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd590037070 con 0x7fd59c15e240 2024-04-02T13:25:07.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.782+0000 7fd59b7fe640 1 -- 172.21.15.44:0/1844699600 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590031940 con 0x7fd59c15e240 2024-04-02T13:25:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.783+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/1844699600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd59c15e240 msgr2=0x7fd59c0d0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.783+0000 7fd5a1f9d640 1 --2- 172.21.15.44:0/1844699600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd59c15e240 0x7fd59c0d0920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd5900097a0 tx=0x7fd59002b840 comp rx=0 tx=0).stop 2024-04-02T13:25:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.783+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/1844699600 shutdown_connections 2024-04-02T13:25:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.783+0000 7fd5a1f9d640 1 --2- 172.21.15.44:0/1844699600 >> v2:172.21.15.69:3300/0 conn(0x7fd59c0d0e60 0x7fd59c0cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.783+0000 7fd5a1f9d640 1 --2- 172.21.15.44:0/1844699600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd59c15e240 0x7fd59c0d0920 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.783+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/1844699600 >> 172.21.15.44:0/1844699600 conn(0x7fd59c09d1f0 msgr2=0x7fd59c0d3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.783+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/1844699600 shutdown_connections 2024-04-02T13:25:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.783+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/1844699600 wait complete. 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.784+0000 7fd5a1f9d640 1 Processor -- start 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.784+0000 7fd5a1f9d640 1 -- start start 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a1f9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd59c0cef40 0x7fd59c152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a1f9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd59c0d0e60 0x7fd59c0cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a1f9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd59c15e240 0x7fd59c15fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a1f9d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c15e620 con 0x7fd59c0cef40 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a1f9d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c0cfdf0 con 0x7fd59c15e240 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a1f9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd59c0cf3d0 con 0x7fd59c0d0e60 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a0f9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd59c0cef40 0x7fd59c152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a0f9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd59c0cef40 0x7fd59c152d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59922/0 (socket says 172.21.15.44:59922) 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a0f9b640 1 -- 172.21.15.44:0/2217436660 learned_addr learned my addr 172.21.15.44:0/2217436660 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a179c640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd59c15e240 0x7fd59c15fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd59bfff640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd59c0d0e60 0x7fd59c0cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a0f9b640 1 -- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd59c0d0e60 msgr2=0x7fd59c0cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a0f9b640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd59c0d0e60 0x7fd59c0cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a0f9b640 1 -- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd59c15e240 msgr2=0x7fd59c15fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a0f9b640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd59c15e240 0x7fd59c15fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a0f9b640 1 -- 172.21.15.44:0/2217436660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd59c0b8150 con 0x7fd59c0cef40 2024-04-02T13:25:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd59bfff640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd59c0d0e60 0x7fd59c0cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.785+0000 7fd5a179c640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd59c15e240 0x7fd59c15fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.786+0000 7fd5a0f9b640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd59c0cef40 0x7fd59c152d40 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fd5900098c0 tx=0x7fd5900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.786+0000 7fd5997fa640 1 -- 172.21.15.44:0/2217436660 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590032430 con 0x7fd59c0cef40 2024-04-02T13:25:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.786+0000 7fd5997fa640 1 -- 172.21.15.44:0/2217436660 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5900325d0 con 0x7fd59c0cef40 2024-04-02T13:25:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.786+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/2217436660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd59c0cbe70 con 0x7fd59c0cef40 2024-04-02T13:25:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.786+0000 7fd5997fa640 1 -- 172.21.15.44:0/2217436660 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590032980 con 0x7fd59c0cef40 2024-04-02T13:25:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.786+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/2217436660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd59c153ac0 con 0x7fd59c0cef40 2024-04-02T13:25:07.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.787+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/2217436660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd59c0c9b50 con 0x7fd59c0cef40 2024-04-02T13:25:07.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.787+0000 7fd5997fa640 1 -- 172.21.15.44:0/2217436660 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd590032b20 con 0x7fd59c0cef40 2024-04-02T13:25:07.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.788+0000 7fd5997fa640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd58006c8f0 0x7fd58006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:07.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.788+0000 7fd5997fa640 1 -- 172.21.15.44:0/2217436660 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fd59002f030 con 0x7fd59c0cef40 2024-04-02T13:25:07.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.788+0000 7fd59bfff640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd58006c8f0 0x7fd58006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:07.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.788+0000 7fd59bfff640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd58006c8f0 0x7fd58006edb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd59c160970 tx=0x7fd588008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:07.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.792+0000 7fd5997fa640 1 -- 172.21.15.44:0/2217436660 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd59c0c9b50 con 0x7fd59c0cef40 2024-04-02T13:25:07.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.977+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/2217436660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd59c0d18f0 con 0x7fd59c0cef40 2024-04-02T13:25:07.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.978+0000 7fd5997fa640 1 -- 172.21.15.44:0/2217436660 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) ==== 100+0+11 (secure 0 0 0) 0x7fd590040070 con 0x7fd59c0cef40 2024-04-02T13:25:07.981 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:25:07.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.980+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd58006c8f0 msgr2=0x7fd58006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.980+0000 7fd5a1f9d640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd58006c8f0 0x7fd58006edb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd59c160970 tx=0x7fd588008040 comp rx=0 tx=0).stop 2024-04-02T13:25:07.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.980+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/2217436660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd59c0cef40 msgr2=0x7fd59c152d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:07.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.980+0000 7fd5a1f9d640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd59c0cef40 0x7fd59c152d40 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fd5900098c0 tx=0x7fd5900046c0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.980+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/2217436660 shutdown_connections 2024-04-02T13:25:07.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.980+0000 7fd5a1f9d640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd58006c8f0 0x7fd58006edb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.980+0000 7fd5a1f9d640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd59c15e240 0x7fd59c15fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.980+0000 7fd5a1f9d640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd59c0d0e60 0x7fd59c0cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.980+0000 7fd5a1f9d640 1 --2- 172.21.15.44:0/2217436660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd59c0cef40 0x7fd59c152d40 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:07.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.980+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/2217436660 >> 172.21.15.44:0/2217436660 conn(0x7fd59c09d1f0 msgr2=0x7fd59c0c9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:07.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.980+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/2217436660 shutdown_connections 2024-04-02T13:25:07.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:07.980+0000 7fd5a1f9d640 1 -- 172.21.15.44:0/2217436660 wait complete. 2024-04-02T13:25:07.994 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:25:08.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.093+0000 7f5f862d5640 1 Processor -- start 2024-04-02T13:25:08.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.093+0000 7f5f862d5640 1 -- start start 2024-04-02T13:25:08.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.094+0000 7f5f862d5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f800b2d00 0x7f5f800cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.094+0000 7f5f862d5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5f800ce110 0x7f5f80071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.094+0000 7f5f862d5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f800b3160 con 0x7f5f8015aa00 2024-04-02T13:25:08.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.094+0000 7f5f862d5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5f8015ade0 con 0x7f5f800ce110 2024-04-02T13:25:08.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.094+0000 7f5f862d5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5f800af1b0 con 0x7f5f800b2d00 2024-04-02T13:25:08.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.094+0000 7f5f84872640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5f800ce110 0x7f5f80071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.094+0000 7f5f7f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f800b2d00 0x7f5f800cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.094+0000 7f5f7ffff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5f8015aa00 0x7f5f80072bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:58228/0 (socket says 172.21.15.44:58228) 2024-04-02T13:25:08.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.094+0000 7f5f7ffff640 1 -- 172.21.15.44:0/2370673343 learned_addr learned my addr 172.21.15.44:0/2370673343 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:08.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.094+0000 7f5f84872640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5f800ce110 0x7f5f80071190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:60454/0 (socket says 172.21.15.44:60454) 2024-04-02T13:25:08.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.095+0000 7f5f84872640 1 -- 172.21.15.44:0/2370673343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f800b2d00 msgr2=0x7f5f800cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:08.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.095+0000 7f5f84872640 1 --2- 172.21.15.44:0/2370673343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f800b2d00 0x7f5f800cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.095+0000 7f5f84872640 1 -- 172.21.15.44:0/2370673343 >> v1:172.21.15.69:6789/0 conn(0x7f5f8015aa00 legacy=0x7f5f80072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.095+0000 7f5f84872640 1 -- 172.21.15.44:0/2370673343 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f800af490 con 0x7f5f800ce110 2024-04-02T13:25:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.095+0000 7f5f84872640 1 --2- 172.21.15.44:0/2370673343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f800ce110 0x7f5f80071190 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009d90 tx=0x7f5f7802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c0024d26ae423c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.096+0000 7f5f7effd640 1 -- 172.21.15.44:0/2370673343 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78039070 con 0x7f5f800ce110 2024-04-02T13:25:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.096+0000 7f5f7effd640 1 -- 172.21.15.44:0/2370673343 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f7802cbc0 con 0x7f5f800ce110 2024-04-02T13:25:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.096+0000 7f5f7effd640 1 -- 172.21.15.44:0/2370673343 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78034440 con 0x7f5f800ce110 2024-04-02T13:25:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.096+0000 7f5f862d5640 1 -- 172.21.15.44:0/2370673343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f800ce110 msgr2=0x7f5f80071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.096+0000 7f5f862d5640 1 --2- 172.21.15.44:0/2370673343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f800ce110 0x7f5f80071190 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009d90 tx=0x7f5f7802be10 comp rx=0 tx=0).stop 2024-04-02T13:25:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.096+0000 7f5f862d5640 1 -- 172.21.15.44:0/2370673343 shutdown_connections 2024-04-02T13:25:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.096+0000 7f5f862d5640 1 --2- 172.21.15.44:0/2370673343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f800ce110 0x7f5f80071190 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.096+0000 7f5f862d5640 1 --2- 172.21.15.44:0/2370673343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f800b2d00 0x7f5f800cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.096+0000 7f5f862d5640 1 -- 172.21.15.44:0/2370673343 >> 172.21.15.44:0/2370673343 conn(0x7f5f800cfcd0 msgr2=0x7f5f800d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.096+0000 7f5f862d5640 1 -- 172.21.15.44:0/2370673343 shutdown_connections 2024-04-02T13:25:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.096+0000 7f5f862d5640 1 -- 172.21.15.44:0/2370673343 wait complete. 2024-04-02T13:25:08.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.097+0000 7f5f862d5640 1 Processor -- start 2024-04-02T13:25:08.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.098+0000 7f5f862d5640 1 -- start start 2024-04-02T13:25:08.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.098+0000 7f5f862d5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f800b2d00 0x7f5f80148540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.098+0000 7f5f862d5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f800ce110 0x7f5f801557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.098+0000 7f5f862d5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f8015aa00 0x7f5f80150680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.098+0000 7f5f862d5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5f800ce5a0 con 0x7f5f800b2d00 2024-04-02T13:25:08.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.098+0000 7f5f862d5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5f800726d0 con 0x7f5f800ce110 2024-04-02T13:25:08.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.098+0000 7f5f862d5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5f800cc750 con 0x7f5f8015aa00 2024-04-02T13:25:08.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.098+0000 7f5f7ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f800b2d00 0x7f5f80148540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.098+0000 7f5f7ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f800b2d00 0x7f5f80148540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59932/0 (socket says 172.21.15.44:59932) 2024-04-02T13:25:08.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.098+0000 7f5f7ffff640 1 -- 172.21.15.44:0/3914028805 learned_addr learned my addr 172.21.15.44:0/3914028805 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:08.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.099+0000 7f5f7f7fe640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f800ce110 0x7f5f801557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.099+0000 7f5f84872640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f8015aa00 0x7f5f80150680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.099+0000 7f5f7ffff640 1 -- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f8015aa00 msgr2=0x7f5f80150680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.099+0000 7f5f7ffff640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f8015aa00 0x7f5f80150680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.099+0000 7f5f7ffff640 1 -- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f800ce110 msgr2=0x7f5f801557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.099+0000 7f5f7ffff640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f800ce110 0x7f5f801557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.099+0000 7f5f7ffff640 1 -- 172.21.15.44:0/3914028805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f80150bc0 con 0x7f5f800b2d00 2024-04-02T13:25:08.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.099+0000 7f5f7ffff640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f800b2d00 0x7f5f80148540 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000c3c0 tx=0x7f5f7000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:08.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.099+0000 7f5f7cff9640 1 -- 172.21.15.44:0/3914028805 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70017070 con 0x7f5f800b2d00 2024-04-02T13:25:08.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.099+0000 7f5f7cff9640 1 -- 172.21.15.44:0/3914028805 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f70005ce0 con 0x7f5f800b2d00 2024-04-02T13:25:08.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.099+0000 7f5f7cff9640 1 -- 172.21.15.44:0/3914028805 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f700072f0 con 0x7f5f800b2d00 2024-04-02T13:25:08.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.099+0000 7f5f862d5640 1 -- 172.21.15.44:0/3914028805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f8014ea20 con 0x7f5f800b2d00 2024-04-02T13:25:08.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.099+0000 7f5f862d5640 1 -- 172.21.15.44:0/3914028805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f8014efa0 con 0x7f5f800b2d00 2024-04-02T13:25:08.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.100+0000 7f5f862d5640 1 -- 172.21.15.44:0/3914028805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f800684f0 con 0x7f5f800b2d00 2024-04-02T13:25:08.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.101+0000 7f5f7cff9640 1 -- 172.21.15.44:0/3914028805 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5f7001a070 con 0x7f5f800b2d00 2024-04-02T13:25:08.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.101+0000 7f5f7cff9640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f6c06c8f0 0x7f5f6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.101+0000 7f5f7cff9640 1 -- 172.21.15.44:0/3914028805 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f5f7008b100 con 0x7f5f800b2d00 2024-04-02T13:25:08.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.101+0000 7f5f7f7fe640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f6c06c8f0 0x7f5f6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.102+0000 7f5f7f7fe640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f6c06c8f0 0x7f5f6c06edb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5f8014e3c0 tx=0x7f5f74008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:08.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.105+0000 7f5f7cff9640 1 -- 172.21.15.44:0/3914028805 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5f80150bc0 con 0x7f5f800b2d00 2024-04-02T13:25:08.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.304+0000 7f5f862d5640 1 -- 172.21.15.44:0/3914028805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5f8015a000 con 0x7f5f800b2d00 2024-04-02T13:25:08.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.305+0000 7f5f7cff9640 1 -- 172.21.15.44:0/3914028805 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v79) ==== 80+0+0 (secure 0 0 0) 0x7f5f70059320 con 0x7f5f800b2d00 2024-04-02T13:25:08.308 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:25:08.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.307+0000 7f5f862d5640 1 -- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f6c06c8f0 msgr2=0x7f5f6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.307+0000 7f5f862d5640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f6c06c8f0 0x7f5f6c06edb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5f8014e3c0 tx=0x7f5f74008040 comp rx=0 tx=0).stop 2024-04-02T13:25:08.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.307+0000 7f5f862d5640 1 -- 172.21.15.44:0/3914028805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f800b2d00 msgr2=0x7f5f80148540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.307+0000 7f5f862d5640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f800b2d00 0x7f5f80148540 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000c3c0 tx=0x7f5f7000c880 comp rx=0 tx=0).stop 2024-04-02T13:25:08.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.307+0000 7f5f862d5640 1 -- 172.21.15.44:0/3914028805 shutdown_connections 2024-04-02T13:25:08.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.307+0000 7f5f862d5640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5f6c06c8f0 0x7f5f6c06edb0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.307+0000 7f5f862d5640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f8015aa00 0x7f5f80150680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.307+0000 7f5f862d5640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f800ce110 0x7f5f801557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.307+0000 7f5f862d5640 1 --2- 172.21.15.44:0/3914028805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f800b2d00 0x7f5f80148540 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.307+0000 7f5f862d5640 1 -- 172.21.15.44:0/3914028805 >> 172.21.15.44:0/3914028805 conn(0x7f5f800cfcd0 msgr2=0x7f5f800b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:08.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.307+0000 7f5f862d5640 1 -- 172.21.15.44:0/3914028805 shutdown_connections 2024-04-02T13:25:08.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.307+0000 7f5f862d5640 1 -- 172.21.15.44:0/3914028805 wait complete. 2024-04-02T13:25:08.322 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:25:08.322 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:25:08.338 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:25:08.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.476+0000 7fb4288c5640 1 Processor -- start 2024-04-02T13:25:08.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.476+0000 7fb4288c5640 1 -- start start 2024-04-02T13:25:08.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.477+0000 7fb4288c5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb4240cece0 0x7fb4240cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.477+0000 7fb4288c5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4240d09f0 0x7fb42406d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.477+0000 7fb4288c5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4240cf600 con 0x7fb42406ece0 2024-04-02T13:25:08.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.477+0000 7fb4288c5640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fb42406dc40 con 0x7fb4240cece0 2024-04-02T13:25:08.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.477+0000 7fb4288c5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb42406df20 con 0x7fb4240d09f0 2024-04-02T13:25:08.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.477+0000 7fb42259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb4240cece0 0x7fb4240cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.477+0000 7fb42259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb4240cece0 0x7fb4240cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59946/0 (socket says 172.21.15.44:59946) 2024-04-02T13:25:08.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.477+0000 7fb42259c640 1 -- 172.21.15.44:0/3089844127 learned_addr learned my addr 172.21.15.44:0/3089844127 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:08.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.477+0000 7fb421d9b640 1 --2- 172.21.15.44:0/3089844127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4240d09f0 0x7fb42406d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.477+0000 7fb42259c640 1 -- 172.21.15.44:0/3089844127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4240d09f0 msgr2=0x7fb42406d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:08.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.477+0000 7fb42259c640 1 --2- 172.21.15.44:0/3089844127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4240d09f0 0x7fb42406d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.477+0000 7fb42259c640 1 -- 172.21.15.44:0/3089844127 >> v1:172.21.15.69:6789/0 conn(0x7fb42406ece0 legacy=0x7fb42415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.477+0000 7fb42259c640 1 -- 172.21.15.44:0/3089844127 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42406e200 con 0x7fb4240cece0 2024-04-02T13:25:08.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.478+0000 7fb42259c640 1 --2- 172.21.15.44:0/3089844127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4240cece0 0x7fb4240cf0c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb4140098a0 tx=0x7fb41402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a078c4e6331741 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:08.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.478+0000 7fb42159a640 1 -- 172.21.15.44:0/3089844127 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414034070 con 0x7fb4240cece0 2024-04-02T13:25:08.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.478+0000 7fb42159a640 1 -- 172.21.15.44:0/3089844127 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb414030840 con 0x7fb4240cece0 2024-04-02T13:25:08.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.478+0000 7fb42159a640 1 -- 172.21.15.44:0/3089844127 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414030c10 con 0x7fb4240cece0 2024-04-02T13:25:08.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.478+0000 7fb4288c5640 1 -- 172.21.15.44:0/3089844127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4240cece0 msgr2=0x7fb4240cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.478+0000 7fb4288c5640 1 --2- 172.21.15.44:0/3089844127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4240cece0 0x7fb4240cf0c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb4140098a0 tx=0x7fb41402b980 comp rx=0 tx=0).stop 2024-04-02T13:25:08.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.479+0000 7fb4288c5640 1 -- 172.21.15.44:0/3089844127 shutdown_connections 2024-04-02T13:25:08.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.479+0000 7fb4288c5640 1 --2- 172.21.15.44:0/3089844127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4240d09f0 0x7fb42406d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.479+0000 7fb4288c5640 1 --2- 172.21.15.44:0/3089844127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4240cece0 0x7fb4240cf0c0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.479+0000 7fb4288c5640 1 -- 172.21.15.44:0/3089844127 >> 172.21.15.44:0/3089844127 conn(0x7fb4240cfcd0 msgr2=0x7fb4240d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:08.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.479+0000 7fb4288c5640 1 -- 172.21.15.44:0/3089844127 shutdown_connections 2024-04-02T13:25:08.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.479+0000 7fb4288c5640 1 -- 172.21.15.44:0/3089844127 wait complete. 2024-04-02T13:25:08.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.480+0000 7fb4288c5640 1 Processor -- start 2024-04-02T13:25:08.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.480+0000 7fb4288c5640 1 -- start start 2024-04-02T13:25:08.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb4288c5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb42406ece0 0x7fb4241412b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb4288c5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4240cece0 0x7fb42409cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb4288c5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4240d09f0 0x7fb42409d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb4288c5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb42406d810 con 0x7fb42406ece0 2024-04-02T13:25:08.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb4288c5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb42406e7c0 con 0x7fb4240d09f0 2024-04-02T13:25:08.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb4288c5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb42406d990 con 0x7fb4240cece0 2024-04-02T13:25:08.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb42259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb42406ece0 0x7fb4241412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb42259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb42406ece0 0x7fb4241412b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59948/0 (socket says 172.21.15.44:59948) 2024-04-02T13:25:08.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb42259c640 1 -- 172.21.15.44:0/1995271013 learned_addr learned my addr 172.21.15.44:0/1995271013 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:08.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb421d9b640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4240cece0 0x7fb42409cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb422d9d640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4240d09f0 0x7fb42409d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb42259c640 1 -- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4240cece0 msgr2=0x7fb42409cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb42259c640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4240cece0 0x7fb42409cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb42259c640 1 -- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4240d09f0 msgr2=0x7fb42409d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb42259c640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4240d09f0 0x7fb42409d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb42259c640 1 -- 172.21.15.44:0/1995271013 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4241417f0 con 0x7fb42406ece0 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb422d9d640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4240d09f0 0x7fb42409d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb42259c640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb42406ece0 0x7fb4241412b0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fb414000c00 tx=0x7fb414004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb412ffd640 1 -- 172.21.15.44:0/1995271013 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414048070 con 0x7fb42406ece0 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb412ffd640 1 -- 172.21.15.44:0/1995271013 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb414030640 con 0x7fb42406ece0 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb4288c5640 1 -- 172.21.15.44:0/1995271013 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb424152510 con 0x7fb42406ece0 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb412ffd640 1 -- 172.21.15.44:0/1995271013 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414034040 con 0x7fb42406ece0 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.481+0000 7fb4288c5640 1 -- 172.21.15.44:0/1995271013 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb424152ad0 con 0x7fb42406ece0 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.483+0000 7fb4288c5640 1 -- 172.21.15.44:0/1995271013 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb424074f20 con 0x7fb42406ece0 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.483+0000 7fb412ffd640 1 -- 172.21.15.44:0/1995271013 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb41404b070 con 0x7fb42406ece0 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.483+0000 7fb412ffd640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb40406c920 0x7fb40406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.483+0000 7fb412ffd640 1 -- 172.21.15.44:0/1995271013 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fb4140adb80 con 0x7fb42406ece0 2024-04-02T13:25:08.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.484+0000 7fb421d9b640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb40406c920 0x7fb40406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.484+0000 7fb421d9b640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb40406c920 0x7fb40406ede0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb418006fd0 tx=0x7fb418008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:08.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.488+0000 7fb412ffd640 1 -- 172.21.15.44:0/1995271013 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb41407bf40 con 0x7fb42406ece0 2024-04-02T13:25:08.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.672+0000 7fb4288c5640 1 -- 172.21.15.44:0/1995271013 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb4240684f0 con 0x7fb42406ece0 2024-04-02T13:25:08.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.673+0000 7fb412ffd640 1 -- 172.21.15.44:0/1995271013 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+16897 (secure 0 0 0) 0x7fb414030810 con 0x7fb42406ece0 2024-04-02T13:25:08.677 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:08.677 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":79,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:24:55.564387+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:24:52.202720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-02T13:24:52.916529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:25:08.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.675+0000 7fb4288c5640 1 -- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb40406c920 msgr2=0x7fb40406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.675+0000 7fb4288c5640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb40406c920 0x7fb40406ede0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb418006fd0 tx=0x7fb418008040 comp rx=0 tx=0).stop 2024-04-02T13:25:08.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.675+0000 7fb4288c5640 1 -- 172.21.15.44:0/1995271013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb42406ece0 msgr2=0x7fb4241412b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.675+0000 7fb4288c5640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb42406ece0 0x7fb4241412b0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fb414000c00 tx=0x7fb414004730 comp rx=0 tx=0).stop 2024-04-02T13:25:08.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.675+0000 7fb4288c5640 1 -- 172.21.15.44:0/1995271013 shutdown_connections 2024-04-02T13:25:08.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.675+0000 7fb4288c5640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb40406c920 0x7fb40406ede0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.675+0000 7fb4288c5640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4240d09f0 0x7fb42409d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.675+0000 7fb4288c5640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4240cece0 0x7fb42409cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.675+0000 7fb4288c5640 1 --2- 172.21.15.44:0/1995271013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb42406ece0 0x7fb4241412b0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.675+0000 7fb4288c5640 1 -- 172.21.15.44:0/1995271013 >> 172.21.15.44:0/1995271013 conn(0x7fb4240cfcd0 msgr2=0x7fb4240acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:08.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.675+0000 7fb4288c5640 1 -- 172.21.15.44:0/1995271013 shutdown_connections 2024-04-02T13:25:08.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.675+0000 7fb4288c5640 1 -- 172.21.15.44:0/1995271013 wait complete. 2024-04-02T13:25:08.689 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:24:52.202720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:24:52.916529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:25:08.690 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:25:08.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.791+0000 7f865be88640 1 Processor -- start 2024-04-02T13:25:08.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.791+0000 7f865be88640 1 -- start start 2024-04-02T13:25:08.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.792+0000 7f865be88640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f86540ce9a0 0x7f86540d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.792+0000 7f865be88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86540d0670 0x7f8654164c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.792+0000 7f865be88640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86540cdc20 con 0x7f86541554c0 2024-04-02T13:25:08.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.792+0000 7f865be88640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f86540d0ae0 con 0x7f86540ce9a0 2024-04-02T13:25:08.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.792+0000 7f865be88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f86540cdec0 con 0x7f86540d0670 2024-04-02T13:25:08.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.792+0000 7f8659423640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f86540ce9a0 0x7f86540d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.792+0000 7f8659423640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f86540ce9a0 0x7f86540d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59952/0 (socket says 172.21.15.44:59952) 2024-04-02T13:25:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.792+0000 7f8659423640 1 -- 172.21.15.44:0/2256155726 learned_addr learned my addr 172.21.15.44:0/2256155726 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.792+0000 7f865a425640 1 --2- 172.21.15.44:0/2256155726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86540d0670 0x7f8654164c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.792+0000 7f8659423640 1 -- 172.21.15.44:0/2256155726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86540d0670 msgr2=0x7f8654164c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.792+0000 7f8659423640 1 --2- 172.21.15.44:0/2256155726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86540d0670 0x7f8654164c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.793+0000 7f8659423640 1 -- 172.21.15.44:0/2256155726 >> v1:172.21.15.69:6789/0 conn(0x7f86541554c0 legacy=0x7f8654071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.793+0000 7f8659423640 1 -- 172.21.15.44:0/2256155726 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86540ce1a0 con 0x7f86540ce9a0 2024-04-02T13:25:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.793+0000 7f8659423640 1 --2- 172.21.15.44:0/2256155726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86540ce9a0 0x7f86540d0130 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f8648009870 tx=0x7f864802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41cd470b10daf8c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.793+0000 7f8658c22640 1 -- 172.21.15.44:0/2256155726 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648034070 con 0x7f86540ce9a0 2024-04-02T13:25:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.793+0000 7f8658c22640 1 -- 172.21.15.44:0/2256155726 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8648030840 con 0x7f86540ce9a0 2024-04-02T13:25:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.793+0000 7f8658c22640 1 -- 172.21.15.44:0/2256155726 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648030c10 con 0x7f86540ce9a0 2024-04-02T13:25:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.794+0000 7f865be88640 1 -- 172.21.15.44:0/2256155726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86540ce9a0 msgr2=0x7f86540d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.794+0000 7f865be88640 1 --2- 172.21.15.44:0/2256155726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86540ce9a0 0x7f86540d0130 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f8648009870 tx=0x7f864802b980 comp rx=0 tx=0).stop 2024-04-02T13:25:08.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.794+0000 7f865be88640 1 -- 172.21.15.44:0/2256155726 shutdown_connections 2024-04-02T13:25:08.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.794+0000 7f865be88640 1 --2- 172.21.15.44:0/2256155726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86540ce9a0 0x7f86540d0130 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.794+0000 7f865be88640 1 --2- 172.21.15.44:0/2256155726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86540d0670 0x7f8654164c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.794+0000 7f865be88640 1 -- 172.21.15.44:0/2256155726 >> 172.21.15.44:0/2256155726 conn(0x7f865406b920 msgr2=0x7f865406bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:08.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.794+0000 7f865be88640 1 -- 172.21.15.44:0/2256155726 shutdown_connections 2024-04-02T13:25:08.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.794+0000 7f865be88640 1 -- 172.21.15.44:0/2256155726 wait complete. 2024-04-02T13:25:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.795+0000 7f865be88640 1 Processor -- start 2024-04-02T13:25:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.795+0000 7f865be88640 1 -- start start 2024-04-02T13:25:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f865be88640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86540ce9a0 0x7f865414d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f865be88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86540d0670 0x7f8654151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f865be88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86541554c0 0x7f865415d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f865be88640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f865415c290 con 0x7f86540ce9a0 2024-04-02T13:25:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f865be88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f86541558a0 con 0x7f86541554c0 2024-04-02T13:25:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f865be88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8654159bf0 con 0x7f86540d0670 2024-04-02T13:25:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f8659c24640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86540ce9a0 0x7f865414d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f8659c24640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86540ce9a0 0x7f865414d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59958/0 (socket says 172.21.15.44:59958) 2024-04-02T13:25:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f8659c24640 1 -- 172.21.15.44:0/2837166173 learned_addr learned my addr 172.21.15.44:0/2837166173 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f8659423640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86540d0670 0x7f8654151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f865a425640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86541554c0 0x7f865415d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f8659c24640 1 -- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86540d0670 msgr2=0x7f8654151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f8659c24640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86540d0670 0x7f8654151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f8659c24640 1 -- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86541554c0 msgr2=0x7f865415d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f8659c24640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86541554c0 0x7f865415d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.796+0000 7f8659c24640 1 -- 172.21.15.44:0/2837166173 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865414d5f0 con 0x7f86540ce9a0 2024-04-02T13:25:08.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.797+0000 7f8659c24640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86540ce9a0 0x7f865414d0b0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f864400c790 tx=0x7f864400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:08.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.797+0000 7f86527fc640 1 -- 172.21.15.44:0/2837166173 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644016020 con 0x7f86540ce9a0 2024-04-02T13:25:08.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.797+0000 7f86527fc640 1 -- 172.21.15.44:0/2837166173 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8644006d50 con 0x7f86540ce9a0 2024-04-02T13:25:08.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.797+0000 7f865be88640 1 -- 172.21.15.44:0/2837166173 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86541520e0 con 0x7f86540ce9a0 2024-04-02T13:25:08.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.797+0000 7f86527fc640 1 -- 172.21.15.44:0/2837166173 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644003bb0 con 0x7f86540ce9a0 2024-04-02T13:25:08.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.797+0000 7f865be88640 1 -- 172.21.15.44:0/2837166173 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8654159e40 con 0x7f86540ce9a0 2024-04-02T13:25:08.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.798+0000 7f865be88640 1 -- 172.21.15.44:0/2837166173 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f865415b710 con 0x7f86540ce9a0 2024-04-02T13:25:08.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.798+0000 7f86527fc640 1 -- 172.21.15.44:0/2837166173 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f864400a850 con 0x7f86540ce9a0 2024-04-02T13:25:08.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.799+0000 7f86527fc640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f864006c8f0 0x7f864006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:08.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.799+0000 7f86527fc640 1 -- 172.21.15.44:0/2837166173 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f864408b820 con 0x7f86540ce9a0 2024-04-02T13:25:08.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.799+0000 7f8659423640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f864006c8f0 0x7f864006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:08.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.799+0000 7f8659423640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f864006c8f0 0x7f864006edb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8648036000 tx=0x7f864803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:08.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.803+0000 7f86527fc640 1 -- 172.21.15.44:0/2837166173 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f865415b710 con 0x7f86540ce9a0 2024-04-02T13:25:08.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.988+0000 7f865be88640 1 -- 172.21.15.44:0/2837166173 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f86540684f0 con 0x7f86540ce9a0 2024-04-02T13:25:08.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.988+0000 7f86527fc640 1 -- 172.21.15.44:0/2837166173 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) ==== 93+0+10 (secure 0 0 0) 0x7f8644020020 con 0x7f86540ce9a0 2024-04-02T13:25:08.991 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:25:08.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.990+0000 7f865be88640 1 -- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f864006c8f0 msgr2=0x7f864006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.990+0000 7f865be88640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f864006c8f0 0x7f864006edb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8648036000 tx=0x7f864803b040 comp rx=0 tx=0).stop 2024-04-02T13:25:08.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.990+0000 7f865be88640 1 -- 172.21.15.44:0/2837166173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86540ce9a0 msgr2=0x7f865414d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:08.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.990+0000 7f865be88640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86540ce9a0 0x7f865414d0b0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f864400c790 tx=0x7f864400cc50 comp rx=0 tx=0).stop 2024-04-02T13:25:08.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.990+0000 7f865be88640 1 -- 172.21.15.44:0/2837166173 shutdown_connections 2024-04-02T13:25:08.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.990+0000 7f865be88640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f864006c8f0 0x7f864006edb0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.990+0000 7f865be88640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86541554c0 0x7f865415d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.990+0000 7f865be88640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86540d0670 0x7f8654151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.990+0000 7f865be88640 1 --2- 172.21.15.44:0/2837166173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86540ce9a0 0x7f865414d0b0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:08.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.990+0000 7f865be88640 1 -- 172.21.15.44:0/2837166173 >> 172.21.15.44:0/2837166173 conn(0x7f865406b920 msgr2=0x7f86540adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:08.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.990+0000 7f865be88640 1 -- 172.21.15.44:0/2837166173 shutdown_connections 2024-04-02T13:25:08.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:08.990+0000 7f865be88640 1 -- 172.21.15.44:0/2837166173 wait complete. 2024-04-02T13:25:09.004 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:25:09.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.104+0000 7f419feb7640 1 Processor -- start 2024-04-02T13:25:09.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.105+0000 7f419feb7640 1 -- start start 2024-04-02T13:25:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.105+0000 7f419feb7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f41980cfdc0 0x7f41980b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.105+0000 7f419feb7640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419806de80 con 0x7f41980d2e00 2024-04-02T13:25:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.105+0000 7f419feb7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41980cf000 con 0x7f419806daa0 2024-04-02T13:25:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.105+0000 7f419feb7640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f41980cf930 con 0x7f41980cfdc0 2024-04-02T13:25:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.105+0000 7f419e454640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f41980cfdc0 0x7f41980b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.105+0000 7f419d452640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f41980d2e00 0x7f419815dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48462/0 (socket says 172.21.15.44:48462) 2024-04-02T13:25:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.105+0000 7f419d452640 1 -- 172.21.15.44:0/2695376371 learned_addr learned my addr 172.21.15.44:0/2695376371 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.106+0000 7f419e454640 1 -- 172.21.15.44:0/2695376371 >> v1:172.21.15.69:6789/0 conn(0x7f419806daa0 legacy=0x7f41980d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.106+0000 7f419e454640 1 -- 172.21.15.44:0/2695376371 >> v1:172.21.15.44:6789/0 conn(0x7f41980d2e00 legacy=0x7f419815dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.106+0000 7f419e454640 1 -- 172.21.15.44:0/2695376371 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41980cf2e0 con 0x7f41980cfdc0 2024-04-02T13:25:09.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.106+0000 7f419cc51640 1 -- 172.21.15.44:0/2695376371 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2191312188 0 0) 0x7f419806de80 con 0x7f41980d2e00 2024-04-02T13:25:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.107+0000 7f419e454640 1 --2- 172.21.15.44:0/2695376371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41980cfdc0 0x7f41980b35b0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f4190009d90 tx=0x7f419002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35a84d68a26de41 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.107+0000 7f419cc51640 1 -- 172.21.15.44:0/2695376371 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190039070 con 0x7f41980cfdc0 2024-04-02T13:25:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.107+0000 7f419cc51640 1 -- 172.21.15.44:0/2695376371 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f419002cc60 con 0x7f41980cfdc0 2024-04-02T13:25:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.107+0000 7f419cc51640 1 -- 172.21.15.44:0/2695376371 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190034660 con 0x7f41980cfdc0 2024-04-02T13:25:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.107+0000 7f419feb7640 1 -- 172.21.15.44:0/2695376371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41980cfdc0 msgr2=0x7f41980b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.107+0000 7f419feb7640 1 --2- 172.21.15.44:0/2695376371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41980cfdc0 0x7f41980b35b0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f4190009d90 tx=0x7f419002beb0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.107+0000 7f419feb7640 1 -- 172.21.15.44:0/2695376371 shutdown_connections 2024-04-02T13:25:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.107+0000 7f419feb7640 1 --2- 172.21.15.44:0/2695376371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41980cfdc0 0x7f41980b35b0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.107+0000 7f419feb7640 1 -- 172.21.15.44:0/2695376371 >> 172.21.15.44:0/2695376371 conn(0x7f419809d430 msgr2=0x7f41980d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.107+0000 7f419feb7640 1 -- 172.21.15.44:0/2695376371 shutdown_connections 2024-04-02T13:25:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.107+0000 7f419feb7640 1 -- 172.21.15.44:0/2695376371 wait complete. 2024-04-02T13:25:09.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.108+0000 7f419feb7640 1 Processor -- start 2024-04-02T13:25:09.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.109+0000 7f419feb7640 1 -- start start 2024-04-02T13:25:09.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.109+0000 7f419feb7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f419806daa0 0x7f4198158400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.109+0000 7f419feb7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41980cfdc0 0x7f41981576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.109+0000 7f419feb7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41980d2e00 0x7f41980b6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.109+0000 7f419feb7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f41980ceda0 con 0x7f41980d2e00 2024-04-02T13:25:09.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.109+0000 7f419feb7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f41980cec20 con 0x7f41980cfdc0 2024-04-02T13:25:09.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.109+0000 7f419feb7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f41980d11c0 con 0x7f419806daa0 2024-04-02T13:25:09.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.109+0000 7f419e454640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41980d2e00 0x7f41980b6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.109+0000 7f419e454640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41980d2e00 0x7f41980b6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59964/0 (socket says 172.21.15.44:59964) 2024-04-02T13:25:09.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.109+0000 7f419e454640 1 -- 172.21.15.44:0/1578666285 learned_addr learned my addr 172.21.15.44:0/1578666285 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.110+0000 7f419dc53640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f419806daa0 0x7f4198158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.110+0000 7f419d452640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41980cfdc0 0x7f41981576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.110+0000 7f419e454640 1 -- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f419806daa0 msgr2=0x7f4198158400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.110+0000 7f419e454640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f419806daa0 0x7f4198158400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.110+0000 7f419e454640 1 -- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41980cfdc0 msgr2=0x7f41981576c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.110+0000 7f419e454640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41980cfdc0 0x7f41981576c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.110+0000 7f419e454640 1 -- 172.21.15.44:0/1578666285 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4198158940 con 0x7f41980d2e00 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.110+0000 7f419e454640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41980d2e00 0x7f41980b6d30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f4190009eb0 tx=0x7f4190004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.110+0000 7f418e7fc640 1 -- 172.21.15.44:0/1578666285 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190044030 con 0x7f41980d2e00 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.110+0000 7f418e7fc640 1 -- 172.21.15.44:0/1578666285 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f419002ca60 con 0x7f41980d2e00 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.110+0000 7f419feb7640 1 -- 172.21.15.44:0/1578666285 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4198157c00 con 0x7f41980d2e00 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.110+0000 7f418e7fc640 1 -- 172.21.15.44:0/1578666285 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190039040 con 0x7f41980d2e00 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.110+0000 7f419feb7640 1 -- 172.21.15.44:0/1578666285 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41980c9180 con 0x7f41980d2e00 2024-04-02T13:25:09.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.111+0000 7f419feb7640 1 -- 172.21.15.44:0/1578666285 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41980684f0 con 0x7f41980d2e00 2024-04-02T13:25:09.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.112+0000 7f418e7fc640 1 -- 172.21.15.44:0/1578666285 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4190047070 con 0x7f41980d2e00 2024-04-02T13:25:09.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.112+0000 7f418e7fc640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f417c06c920 0x7f417c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.112+0000 7f418e7fc640 1 -- 172.21.15.44:0/1578666285 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f41900ae680 con 0x7f41980d2e00 2024-04-02T13:25:09.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.112+0000 7f419dc53640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f417c06c920 0x7f417c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.113+0000 7f419dc53640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f417c06c920 0x7f417c06ede0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4194004ea0 tx=0x7f4194009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:09.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.116+0000 7f418e7fc640 1 -- 172.21.15.44:0/1578666285 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f419004e070 con 0x7f41980d2e00 2024-04-02T13:25:09.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.304+0000 7f419feb7640 1 -- 172.21.15.44:0/1578666285 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f41980ce440 con 0x7f41980d2e00 2024-04-02T13:25:09.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.305+0000 7f418e7fc640 1 -- 172.21.15.44:0/1578666285 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) ==== 104+0+11 (secure 0 0 0) 0x7f4198157c00 con 0x7f41980d2e00 2024-04-02T13:25:09.308 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:25:09.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.306+0000 7f419feb7640 1 -- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f417c06c920 msgr2=0x7f417c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.306+0000 7f419feb7640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f417c06c920 0x7f417c06ede0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4194004ea0 tx=0x7f4194009040 comp rx=0 tx=0).stop 2024-04-02T13:25:09.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.306+0000 7f419feb7640 1 -- 172.21.15.44:0/1578666285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41980d2e00 msgr2=0x7f41980b6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.306+0000 7f419feb7640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41980d2e00 0x7f41980b6d30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f4190009eb0 tx=0x7f4190004a40 comp rx=0 tx=0).stop 2024-04-02T13:25:09.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.307+0000 7f419feb7640 1 -- 172.21.15.44:0/1578666285 shutdown_connections 2024-04-02T13:25:09.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.307+0000 7f419feb7640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f417c06c920 0x7f417c06ede0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.307+0000 7f419feb7640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41980d2e00 0x7f41980b6d30 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.307+0000 7f419feb7640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41980cfdc0 0x7f41981576c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.307+0000 7f419feb7640 1 --2- 172.21.15.44:0/1578666285 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f419806daa0 0x7f4198158400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.307+0000 7f419feb7640 1 -- 172.21.15.44:0/1578666285 >> 172.21.15.44:0/1578666285 conn(0x7f419809d430 msgr2=0x7f41980b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:09.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.307+0000 7f419feb7640 1 -- 172.21.15.44:0/1578666285 shutdown_connections 2024-04-02T13:25:09.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.307+0000 7f419feb7640 1 -- 172.21.15.44:0/1578666285 wait complete. 2024-04-02T13:25:09.320 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:25:09.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.422+0000 7f3b6478f640 1 Processor -- start 2024-04-02T13:25:09.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.422+0000 7f3b6478f640 1 -- start start 2024-04-02T13:25:09.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.422+0000 7f3b6478f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3b5c06cff0 0x7f3b5c06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.422+0000 7f3b6478f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b5c0b4180 0x7f3b5c0b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.422+0000 7f3b6478f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b5c06d910 con 0x7f3b5c15e5d0 2024-04-02T13:25:09.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.422+0000 7f3b6478f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f3b5c0d1bf0 con 0x7f3b5c06cff0 2024-04-02T13:25:09.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.422+0000 7f3b6478f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3b5c0d1630 con 0x7f3b5c0b4180 2024-04-02T13:25:09.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.423+0000 7f3b6252b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3b5c06cff0 0x7f3b5c06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.423+0000 7f3b6252b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3b5c06cff0 0x7f3b5c06d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59968/0 (socket says 172.21.15.44:59968) 2024-04-02T13:25:09.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.423+0000 7f3b6252b640 1 -- 172.21.15.44:0/2649574107 learned_addr learned my addr 172.21.15.44:0/2649574107 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:09.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.423+0000 7f3b62d2c640 1 --2- 172.21.15.44:0/2649574107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b5c0b4180 0x7f3b5c0b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.423+0000 7f3b6252b640 1 -- 172.21.15.44:0/2649574107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b5c0b4180 msgr2=0x7f3b5c0b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:09.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.423+0000 7f3b6252b640 1 --2- 172.21.15.44:0/2649574107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b5c0b4180 0x7f3b5c0b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.423+0000 7f3b6252b640 1 -- 172.21.15.44:0/2649574107 >> v1:172.21.15.69:6789/0 conn(0x7f3b5c15e5d0 legacy=0x7f3b5c0d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.423+0000 7f3b6252b640 1 -- 172.21.15.44:0/2649574107 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b5c0d1910 con 0x7f3b5c06cff0 2024-04-02T13:25:09.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.423+0000 7f3b62d2c640 1 --2- 172.21.15.44:0/2649574107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b5c0b4180 0x7f3b5c0b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:09.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.424+0000 7f3b6252b640 1 --2- 172.21.15.44:0/2649574107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b5c06cff0 0x7f3b5c06d3d0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009870 tx=0x7f3b5802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4656828c19d0af37 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:09.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.424+0000 7f3b61529640 1 -- 172.21.15.44:0/2649574107 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b58035020 con 0x7f3b5c06cff0 2024-04-02T13:25:09.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.424+0000 7f3b61529640 1 -- 172.21.15.44:0/2649574107 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b58031640 con 0x7f3b5c06cff0 2024-04-02T13:25:09.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.424+0000 7f3b61529640 1 -- 172.21.15.44:0/2649574107 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5802d040 con 0x7f3b5c06cff0 2024-04-02T13:25:09.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.424+0000 7f3b6478f640 1 -- 172.21.15.44:0/2649574107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b5c06cff0 msgr2=0x7f3b5c06d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.424+0000 7f3b6478f640 1 --2- 172.21.15.44:0/2649574107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b5c06cff0 0x7f3b5c06d3d0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009870 tx=0x7f3b5802b8f0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.424+0000 7f3b6478f640 1 -- 172.21.15.44:0/2649574107 shutdown_connections 2024-04-02T13:25:09.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.424+0000 7f3b6478f640 1 --2- 172.21.15.44:0/2649574107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b5c0b4180 0x7f3b5c0b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.424+0000 7f3b6478f640 1 --2- 172.21.15.44:0/2649574107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b5c06cff0 0x7f3b5c06d3d0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.424+0000 7f3b6478f640 1 -- 172.21.15.44:0/2649574107 >> 172.21.15.44:0/2649574107 conn(0x7f3b5c09aa20 msgr2=0x7f3b5c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:09.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.425+0000 7f3b6478f640 1 -- 172.21.15.44:0/2649574107 shutdown_connections 2024-04-02T13:25:09.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.425+0000 7f3b6478f640 1 -- 172.21.15.44:0/2649574107 wait complete. 2024-04-02T13:25:09.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.426+0000 7f3b6478f640 1 Processor -- start 2024-04-02T13:25:09.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.426+0000 7f3b6478f640 1 -- start start 2024-04-02T13:25:09.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.426+0000 7f3b6478f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b5c06cff0 0x7f3b5c152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.426+0000 7f3b6478f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b5c0b4180 0x7f3b5c0ce860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.426+0000 7f3b6478f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b5c15e5d0 0x7f3b5c0ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.426+0000 7f3b6478f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5c0b2280 con 0x7f3b5c0b4180 2024-04-02T13:25:09.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.426+0000 7f3b6478f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5c0b1820 con 0x7f3b5c15e5d0 2024-04-02T13:25:09.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.426+0000 7f3b6478f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3b5c0b2400 con 0x7f3b5c06cff0 2024-04-02T13:25:09.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.426+0000 7f3b61d2a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b5c0b4180 0x7f3b5c0ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.426+0000 7f3b61d2a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b5c0b4180 0x7f3b5c0ce860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59974/0 (socket says 172.21.15.44:59974) 2024-04-02T13:25:09.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.426+0000 7f3b61d2a640 1 -- 172.21.15.44:0/2621815405 learned_addr learned my addr 172.21.15.44:0/2621815405 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:09.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.427+0000 7f3b61d2a640 1 -- 172.21.15.44:0/2621815405 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b5c06cff0 msgr2=0x7f3b5c152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.427+0000 7f3b61d2a640 1 --2- 172.21.15.44:0/2621815405 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b5c06cff0 0x7f3b5c152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.427+0000 7f3b62d2c640 1 --2- 172.21.15.44:0/2621815405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b5c15e5d0 0x7f3b5c0ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.427+0000 7f3b61d2a640 1 -- 172.21.15.44:0/2621815405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b5c15e5d0 msgr2=0x7f3b5c0ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.427+0000 7f3b61d2a640 1 --2- 172.21.15.44:0/2621815405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b5c15e5d0 0x7f3b5c0ceda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.427+0000 7f3b61d2a640 1 -- 172.21.15.44:0/2621815405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b5c0c8900 con 0x7f3b5c0b4180 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.427+0000 7f3b61d2a640 1 --2- 172.21.15.44:0/2621815405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b5c0b4180 0x7f3b5c0ce860 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00c3c0 tx=0x7f3b4c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.427+0000 7f3b52ffd640 1 -- 172.21.15.44:0/2621815405 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c017070 con 0x7f3b5c0b4180 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.427+0000 7f3b52ffd640 1 -- 172.21.15.44:0/2621815405 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b4c0052f0 con 0x7f3b5c0b4180 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.427+0000 7f3b6478f640 1 -- 172.21.15.44:0/2621815405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b5c0b6e70 con 0x7f3b5c0b4180 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.427+0000 7f3b52ffd640 1 -- 172.21.15.44:0/2621815405 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c003830 con 0x7f3b5c0b4180 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.427+0000 7f3b6478f640 1 -- 172.21.15.44:0/2621815405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b5c0c5f20 con 0x7f3b5c0b4180 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.428+0000 7f3b6478f640 1 -- 172.21.15.44:0/2621815405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b5c0684f0 con 0x7f3b5c0b4180 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.429+0000 7f3b52ffd640 1 -- 172.21.15.44:0/2621815405 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3b4c01a070 con 0x7f3b5c0b4180 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.429+0000 7f3b52ffd640 1 --2- 172.21.15.44:0/2621815405 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b4006c8f0 0x7f3b4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.429+0000 7f3b52ffd640 1 -- 172.21.15.44:0/2621815405 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f3b4c08af50 con 0x7f3b5c0b4180 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.429+0000 7f3b6252b640 1 --2- 172.21.15.44:0/2621815405 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b4006c8f0 0x7f3b4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.430+0000 7f3b6252b640 1 --2- 172.21.15.44:0/2621815405 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b4006c8f0 0x7f3b4006edb0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3b58036000 tx=0x7f3b58030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:09.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.434+0000 7f3b52ffd640 1 -- 172.21.15.44:0/2621815405 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3b4c059280 con 0x7f3b5c0b4180 2024-04-02T13:25:09.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.620+0000 7f3b6478f640 1 -- 172.21.15.44:0/2621815405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3b5c06cdd0 con 0x7f3b5c0b4180 2024-04-02T13:25:09.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.620+0000 7f3b52ffd640 1 -- 172.21.15.44:0/2621815405 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) ==== 100+0+11 (secure 0 0 0) 0x7f3b4c058b80 con 0x7f3b5c0b4180 2024-04-02T13:25:09.624 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:25:09.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.623+0000 7f3b6478f640 1 -- 172.21.15.44:0/2621815405 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b4006c8f0 msgr2=0x7f3b4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.623+0000 7f3b6478f640 1 --2- 172.21.15.44:0/2621815405 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b4006c8f0 0x7f3b4006edb0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3b58036000 tx=0x7f3b58030000 comp rx=0 tx=0).stop 2024-04-02T13:25:09.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.623+0000 7f3b6478f640 1 -- 172.21.15.44:0/2621815405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b5c0b4180 msgr2=0x7f3b5c0ce860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.623+0000 7f3b6478f640 1 --2- 172.21.15.44:0/2621815405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b5c0b4180 0x7f3b5c0ce860 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00c3c0 tx=0x7f3b4c00c880 comp rx=0 tx=0).stop 2024-04-02T13:25:09.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.623+0000 7f3b6478f640 1 -- 172.21.15.44:0/2621815405 shutdown_connections 2024-04-02T13:25:09.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.623+0000 7f3b6478f640 1 --2- 172.21.15.44:0/2621815405 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b4006c8f0 0x7f3b4006edb0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.623+0000 7f3b6478f640 1 --2- 172.21.15.44:0/2621815405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b5c15e5d0 0x7f3b5c0ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.623+0000 7f3b6478f640 1 --2- 172.21.15.44:0/2621815405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b5c0b4180 0x7f3b5c0ce860 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.623+0000 7f3b6478f640 1 --2- 172.21.15.44:0/2621815405 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b5c06cff0 0x7f3b5c152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.623+0000 7f3b6478f640 1 -- 172.21.15.44:0/2621815405 >> 172.21.15.44:0/2621815405 conn(0x7f3b5c09aa20 msgr2=0x7f3b5c070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:09.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.623+0000 7f3b6478f640 1 -- 172.21.15.44:0/2621815405 shutdown_connections 2024-04-02T13:25:09.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.623+0000 7f3b6478f640 1 -- 172.21.15.44:0/2621815405 wait complete. 2024-04-02T13:25:09.636 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:25:09.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.735+0000 7f2a834b1640 1 Processor -- start 2024-04-02T13:25:09.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.735+0000 7f2a834b1640 1 -- start start 2024-04-02T13:25:09.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.736+0000 7f2a834b1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2a7c15e4d0 0x7f2a7c06b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.736+0000 7f2a834b1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2a7c072f60 0x7f2a7c06ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.736+0000 7f2a834b1640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a7c06dfe0 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.736+0000 7f2a834b1640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2a7c0aed70 con 0x7f2a7c15e4d0 2024-04-02T13:25:09.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.736+0000 7f2a834b1640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2a7c06e2c0 con 0x7f2a7c072f60 2024-04-02T13:25:09.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.736+0000 7f2a80a4c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2a7c15e4d0 0x7f2a7c06b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.736+0000 7f2a80a4c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2a7c15e4d0 0x7f2a7c06b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59986/0 (socket says 172.21.15.44:59986) 2024-04-02T13:25:09.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.736+0000 7f2a80a4c640 1 -- 172.21.15.44:0/2605357433 learned_addr learned my addr 172.21.15.44:0/2605357433 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:09.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.736+0000 7f2a81a4e640 1 --2- 172.21.15.44:0/2605357433 >> v2:172.21.15.69:3300/0 conn(0x7f2a7c072f60 0x7f2a7c06ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.736+0000 7f2a80a4c640 1 -- 172.21.15.44:0/2605357433 >> v2:172.21.15.69:3300/0 conn(0x7f2a7c072f60 msgr2=0x7f2a7c06ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:09.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.737+0000 7f2a80a4c640 1 --2- 172.21.15.44:0/2605357433 >> v2:172.21.15.69:3300/0 conn(0x7f2a7c072f60 0x7f2a7c06ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.737+0000 7f2a80a4c640 1 -- 172.21.15.44:0/2605357433 >> v1:172.21.15.69:6789/0 conn(0x7f2a7c15e0f0 legacy=0x7f2a7c0d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:09.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.737+0000 7f2a80a4c640 1 -- 172.21.15.44:0/2605357433 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7c06e5a0 con 0x7f2a7c15e4d0 2024-04-02T13:25:09.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.737+0000 7f2a80a4c640 1 --2- 172.21.15.44:0/2605357433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c15e4d0 0x7f2a7c06b4d0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c0cb390 tx=0x7f2a6c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee58f8d66d355ef server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:09.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.737+0000 7f2a73fff640 1 -- 172.21.15.44:0/2605357433 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c02f850 con 0x7f2a7c15e4d0 2024-04-02T13:25:09.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.738+0000 7f2a73fff640 1 -- 172.21.15.44:0/2605357433 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a6c02f9f0 con 0x7f2a7c15e4d0 2024-04-02T13:25:09.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.738+0000 7f2a73fff640 1 -- 172.21.15.44:0/2605357433 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c02fdc0 con 0x7f2a7c15e4d0 2024-04-02T13:25:09.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.738+0000 7f2a834b1640 1 -- 172.21.15.44:0/2605357433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c15e4d0 msgr2=0x7f2a7c06b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.738+0000 7f2a834b1640 1 --2- 172.21.15.44:0/2605357433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c15e4d0 0x7f2a7c06b4d0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c0cb390 tx=0x7f2a6c02ba30 comp rx=0 tx=0).stop 2024-04-02T13:25:09.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.738+0000 7f2a834b1640 1 -- 172.21.15.44:0/2605357433 shutdown_connections 2024-04-02T13:25:09.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.738+0000 7f2a834b1640 1 --2- 172.21.15.44:0/2605357433 >> v2:172.21.15.69:3300/0 conn(0x7f2a7c072f60 0x7f2a7c06ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.738+0000 7f2a834b1640 1 --2- 172.21.15.44:0/2605357433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c15e4d0 0x7f2a7c06b4d0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.738+0000 7f2a834b1640 1 -- 172.21.15.44:0/2605357433 >> 172.21.15.44:0/2605357433 conn(0x7f2a7c0b2980 msgr2=0x7f2a7c0b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:09.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.738+0000 7f2a834b1640 1 -- 172.21.15.44:0/2605357433 shutdown_connections 2024-04-02T13:25:09.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.738+0000 7f2a834b1640 1 -- 172.21.15.44:0/2605357433 wait complete. 2024-04-02T13:25:09.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.739+0000 7f2a834b1640 1 Processor -- start 2024-04-02T13:25:09.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.739+0000 7f2a834b1640 1 -- start start 2024-04-02T13:25:09.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a834b1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a7c072f60 0x7f2a7c0c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a834b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c15e0f0 0x7f2a7c160a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a834b1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a7c15e4d0 0x7f2a7c15c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a834b1640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c0d0800 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a834b1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c0d0af0 con 0x7f2a7c15e4d0 2024-04-02T13:25:09.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a834b1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2a7c06eda0 con 0x7f2a7c072f60 2024-04-02T13:25:09.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a80a4c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c15e0f0 0x7f2a7c160a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a80a4c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c15e0f0 0x7f2a7c160a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59988/0 (socket says 172.21.15.44:59988) 2024-04-02T13:25:09.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a80a4c640 1 -- 172.21.15.44:0/2231528587 learned_addr learned my addr 172.21.15.44:0/2231528587 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:09.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a81a4e640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a7c15e4d0 0x7f2a7c15c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a8124d640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a7c072f60 0x7f2a7c0c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a80a4c640 1 -- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a7c072f60 msgr2=0x7f2a7c0c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a80a4c640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a7c072f60 0x7f2a7c0c5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a80a4c640 1 -- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a7c15e4d0 msgr2=0x7f2a7c15c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a80a4c640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a7c15e4d0 0x7f2a7c15c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.740+0000 7f2a80a4c640 1 -- 172.21.15.44:0/2231528587 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7c0c7090 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.741+0000 7f2a80a4c640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c15e0f0 0x7f2a7c160a90 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c02ba70 tx=0x7f2a6c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:09.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.741+0000 7f2a71ffb640 1 -- 172.21.15.44:0/2231528587 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c047070 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.741+0000 7f2a71ffb640 1 -- 172.21.15.44:0/2231528587 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a6c037d30 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.741+0000 7f2a71ffb640 1 -- 172.21.15.44:0/2231528587 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c042630 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.741+0000 7f2a834b1640 1 -- 172.21.15.44:0/2231528587 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a7c0c7370 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.741+0000 7f2a834b1640 1 -- 172.21.15.44:0/2231528587 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a7c0c5500 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.742+0000 7f2a834b1640 1 -- 172.21.15.44:0/2231528587 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a7c0cf710 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.742+0000 7f2a71ffb640 1 -- 172.21.15.44:0/2231528587 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a6c04a070 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.743+0000 7f2a71ffb640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a6006c8f0 0x7f2a6006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:09.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.743+0000 7f2a71ffb640 1 -- 172.21.15.44:0/2231528587 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f2a6c0adda0 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.743+0000 7f2a8124d640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a6006c8f0 0x7f2a6006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:09.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.744+0000 7f2a8124d640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a6006c8f0 0x7f2a6006edb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2a74001f10 tx=0x7f2a74009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:09.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.748+0000 7f2a71ffb640 1 -- 172.21.15.44:0/2231528587 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2a7c0cf710 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.943+0000 7f2a834b1640 1 -- 172.21.15.44:0/2231528587 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2a7c0684f0 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.944+0000 7f2a71ffb640 1 -- 172.21.15.44:0/2231528587 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v79) ==== 80+0+0 (secure 0 0 0) 0x7f2a6c033030 con 0x7f2a7c15e0f0 2024-04-02T13:25:09.947 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:25:09.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.946+0000 7f2a834b1640 1 -- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a6006c8f0 msgr2=0x7f2a6006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.946+0000 7f2a834b1640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a6006c8f0 0x7f2a6006edb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2a74001f10 tx=0x7f2a74009040 comp rx=0 tx=0).stop 2024-04-02T13:25:09.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.946+0000 7f2a834b1640 1 -- 172.21.15.44:0/2231528587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c15e0f0 msgr2=0x7f2a7c160a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:09.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.946+0000 7f2a834b1640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c15e0f0 0x7f2a7c160a90 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c02ba70 tx=0x7f2a6c004730 comp rx=0 tx=0).stop 2024-04-02T13:25:09.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.946+0000 7f2a834b1640 1 -- 172.21.15.44:0/2231528587 shutdown_connections 2024-04-02T13:25:09.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.946+0000 7f2a834b1640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a6006c8f0 0x7f2a6006edb0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.946+0000 7f2a834b1640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a7c15e4d0 0x7f2a7c15c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.946+0000 7f2a834b1640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c15e0f0 0x7f2a7c160a90 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.946+0000 7f2a834b1640 1 --2- 172.21.15.44:0/2231528587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a7c072f60 0x7f2a7c0c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:09.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.946+0000 7f2a834b1640 1 -- 172.21.15.44:0/2231528587 >> 172.21.15.44:0/2231528587 conn(0x7f2a7c0b2980 msgr2=0x7f2a7c0b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:09.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.946+0000 7f2a834b1640 1 -- 172.21.15.44:0/2231528587 shutdown_connections 2024-04-02T13:25:09.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:09.946+0000 7f2a834b1640 1 -- 172.21.15.44:0/2231528587 wait complete. 2024-04-02T13:25:09.959 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:25:09.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:09.960 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:09.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[64068]: fuse finished with error 0 and tester_r 0 2024-04-02T13:25:09.987 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:25:10.040 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:25:10.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:10.040 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:25:10.055 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:25:10.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.193+0000 7f09f011c640 1 Processor -- start 2024-04-02T13:25:10.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.193+0000 7f09f011c640 1 -- start start 2024-04-02T13:25:10.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.194+0000 7f09f011c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f09e80ca920 0x7f09e8161e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.194+0000 7f09f011c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09e80cdcb0 0x7f09e81623d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.194+0000 7f09f011c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09e80cd510 con 0x7f09e815e210 2024-04-02T13:25:10.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.194+0000 7f09f011c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f09e80716c0 con 0x7f09e80ca920 2024-04-02T13:25:10.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.194+0000 7f09f011c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f09e80719a0 con 0x7f09e80cdcb0 2024-04-02T13:25:10.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.194+0000 7f09ed6b7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f09e80ca920 0x7f09e8161e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.194+0000 7f09edeb8640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f09e815e210 0x7f09e80ccc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48470/0 (socket says 172.21.15.44:48470) 2024-04-02T13:25:10.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.194+0000 7f09edeb8640 1 -- 172.21.15.44:0/3012149926 learned_addr learned my addr 172.21.15.44:0/3012149926 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:10.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.194+0000 7f09ed6b7640 1 -- 172.21.15.44:0/3012149926 >> v2:172.21.15.69:3300/0 conn(0x7f09e80cdcb0 msgr2=0x7f09e81623d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:10.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.194+0000 7f09ed6b7640 1 --2- 172.21.15.44:0/3012149926 >> v2:172.21.15.69:3300/0 conn(0x7f09e80cdcb0 0x7f09e81623d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.194+0000 7f09ed6b7640 1 -- 172.21.15.44:0/3012149926 >> v1:172.21.15.44:6789/0 conn(0x7f09e815e210 legacy=0x7f09e80ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.194+0000 7f09ed6b7640 1 -- 172.21.15.44:0/3012149926 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e8071c80 con 0x7f09e80ca920 2024-04-02T13:25:10.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.194+0000 7f09eceb6640 1 -- 172.21.15.44:0/3012149926 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1076788639 0 0) 0x7f09e80cd510 con 0x7f09e815e210 2024-04-02T13:25:10.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.195+0000 7f09ed6b7640 1 --2- 172.21.15.44:0/3012149926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80ca920 0x7f09e8161e90 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f09e80cbe70 tx=0x7f09d802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538f90569e5d58d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:10.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.195+0000 7f09eceb6640 1 -- 172.21.15.44:0/3012149926 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d8034070 con 0x7f09e80ca920 2024-04-02T13:25:10.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.195+0000 7f09eceb6640 1 -- 172.21.15.44:0/3012149926 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09d8030610 con 0x7f09e80ca920 2024-04-02T13:25:10.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.195+0000 7f09eceb6640 1 -- 172.21.15.44:0/3012149926 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d80309e0 con 0x7f09e80ca920 2024-04-02T13:25:10.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.195+0000 7f09f011c640 1 -- 172.21.15.44:0/3012149926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80ca920 msgr2=0x7f09e8161e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.195+0000 7f09f011c640 1 --2- 172.21.15.44:0/3012149926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80ca920 0x7f09e8161e90 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f09e80cbe70 tx=0x7f09d802b960 comp rx=0 tx=0).stop 2024-04-02T13:25:10.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.195+0000 7f09f011c640 1 -- 172.21.15.44:0/3012149926 shutdown_connections 2024-04-02T13:25:10.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.195+0000 7f09f011c640 1 --2- 172.21.15.44:0/3012149926 >> v2:172.21.15.69:3300/0 conn(0x7f09e80cdcb0 0x7f09e81623d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.195+0000 7f09f011c640 1 --2- 172.21.15.44:0/3012149926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80ca920 0x7f09e8161e90 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.195+0000 7f09f011c640 1 -- 172.21.15.44:0/3012149926 >> 172.21.15.44:0/3012149926 conn(0x7f09e80b1c30 msgr2=0x7f09e80b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:10.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.196+0000 7f09f011c640 1 -- 172.21.15.44:0/3012149926 shutdown_connections 2024-04-02T13:25:10.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.196+0000 7f09f011c640 1 -- 172.21.15.44:0/3012149926 wait complete. 2024-04-02T13:25:10.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.197+0000 7f09f011c640 1 Processor -- start 2024-04-02T13:25:10.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.197+0000 7f09f011c640 1 -- start start 2024-04-02T13:25:10.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.197+0000 7f09f011c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80ca920 0x7f09e80c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.197+0000 7f09f011c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09e80cdcb0 0x7f09e81f5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.197+0000 7f09f011c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815e210 0x7f09e815e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.197+0000 7f09f011c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09e80cf180 con 0x7f09e80ca920 2024-04-02T13:25:10.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.197+0000 7f09f011c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09e80cf300 con 0x7f09e80cdcb0 2024-04-02T13:25:10.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.197+0000 7f09f011c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f09e80cf480 con 0x7f09e815e210 2024-04-02T13:25:10.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09edeb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80ca920 0x7f09e80c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09edeb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80ca920 0x7f09e80c9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60010/0 (socket says 172.21.15.44:60010) 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09edeb8640 1 -- 172.21.15.44:0/50072771 learned_addr learned my addr 172.21.15.44:0/50072771 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09ed6b7640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09e80cdcb0 0x7f09e81f5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09ee6b9640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815e210 0x7f09e815e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09edeb8640 1 -- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815e210 msgr2=0x7f09e815e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09edeb8640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815e210 0x7f09e815e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09edeb8640 1 -- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09e80cdcb0 msgr2=0x7f09e81f5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09edeb8640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09e80cdcb0 0x7f09e81f5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09edeb8640 1 -- 172.21.15.44:0/50072771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e815f3c0 con 0x7f09e80ca920 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09edeb8640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80ca920 0x7f09e80c9a20 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f09e400c460 tx=0x7f09e400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09de7fc640 1 -- 172.21.15.44:0/50072771 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e4018070 con 0x7f09e80ca920 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09de7fc640 1 -- 172.21.15.44:0/50072771 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e4007500 con 0x7f09e80ca920 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09f011c640 1 -- 172.21.15.44:0/50072771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e815f6a0 con 0x7f09e80ca920 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09de7fc640 1 -- 172.21.15.44:0/50072771 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e4004a20 con 0x7f09e80ca920 2024-04-02T13:25:10.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.198+0000 7f09f011c640 1 -- 172.21.15.44:0/50072771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e81598e0 con 0x7f09e80ca920 2024-04-02T13:25:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.199+0000 7f09f011c640 1 -- 172.21.15.44:0/50072771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09e80cc9a0 con 0x7f09e80ca920 2024-04-02T13:25:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.200+0000 7f09de7fc640 1 -- 172.21.15.44:0/50072771 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f09e401b070 con 0x7f09e80ca920 2024-04-02T13:25:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.200+0000 7f09de7fc640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09cc06c8f0 0x7f09cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.200+0000 7f09de7fc640 1 -- 172.21.15.44:0/50072771 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f09e408af90 con 0x7f09e80ca920 2024-04-02T13:25:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.200+0000 7f09ed6b7640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09cc06c8f0 0x7f09cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.201+0000 7f09ed6b7640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09cc06c8f0 0x7f09cc06edb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f09d8036000 tx=0x7f09d8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.205+0000 7f09de7fc640 1 -- 172.21.15.44:0/50072771 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f09e80cc9a0 con 0x7f09e80ca920 2024-04-02T13:25:10.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.426+0000 7f09f011c640 1 -- 172.21.15.44:0/50072771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09e80684f0 con 0x7f09e80ca920 2024-04-02T13:25:10.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.427+0000 7f09de7fc640 1 -- 172.21.15.44:0/50072771 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5180 (secure 0 0 0) 0x7f09e4059230 con 0x7f09e80ca920 2024-04-02T13:25:10.430 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:10.430 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3662927745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3662927745},{"type":"v1","addr":"172.21.15.44:6835","nonce":3662927745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6394,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1053037366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1053037366},{"type":"v1","addr":"172.21.15.69:6836","nonce":1053037366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6987,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2392159648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2392159648},{"type":"v1","addr":"172.21.15.44:6833","nonce":2392159648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:24:53.925714+0000","modified":"2024-04-02T13:24:56.942238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6837/698185867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":698185867},{"type":"v1","addr":"172.21.15.69:6837","nonce":698185867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7002,"qdb_cluster":[7002]},"id":5}]} 2024-04-02T13:25:10.431 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 44 2024-04-02T13:25:10.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.429+0000 7f09f011c640 1 -- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09cc06c8f0 msgr2=0x7f09cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.429+0000 7f09f011c640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09cc06c8f0 0x7f09cc06edb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f09d8036000 tx=0x7f09d8008030 comp rx=0 tx=0).stop 2024-04-02T13:25:10.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.429+0000 7f09f011c640 1 -- 172.21.15.44:0/50072771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80ca920 msgr2=0x7f09e80c9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.429+0000 7f09f011c640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80ca920 0x7f09e80c9a20 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f09e400c460 tx=0x7f09e400c920 comp rx=0 tx=0).stop 2024-04-02T13:25:10.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.429+0000 7f09f011c640 1 -- 172.21.15.44:0/50072771 shutdown_connections 2024-04-02T13:25:10.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.429+0000 7f09f011c640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09cc06c8f0 0x7f09cc06edb0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.429+0000 7f09f011c640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09e815e210 0x7f09e815e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.429+0000 7f09f011c640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09e80cdcb0 0x7f09e81f5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.429+0000 7f09f011c640 1 --2- 172.21.15.44:0/50072771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09e80ca920 0x7f09e80c9a20 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.429+0000 7f09f011c640 1 -- 172.21.15.44:0/50072771 >> 172.21.15.44:0/50072771 conn(0x7f09e80b1c30 msgr2=0x7f09e80ccbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:10.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.429+0000 7f09f011c640 1 -- 172.21.15.44:0/50072771 shutdown_connections 2024-04-02T13:25:10.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.429+0000 7f09f011c640 1 -- 172.21.15.44:0/50072771 wait complete. 2024-04-02T13:25:10.443 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-04-02T13:25:10.581 INFO:teuthology.orchestra.run.smithi069.stdout:{"message":"","return_code":0} 2024-04-02T13:25:10.581 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-02T13:25:10.581 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:25:10.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.682+0000 7f094edeb640 1 Processor -- start 2024-04-02T13:25:10.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.682+0000 7f094edeb640 1 -- start start 2024-04-02T13:25:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f094edeb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f094815e550 0x7f094806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f094edeb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f094815f040 0x7f09480d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f094edeb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f094815fb50 con 0x7f094806d530 2024-04-02T13:25:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f094edeb640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f094806d970 con 0x7f094815f040 2024-04-02T13:25:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f094edeb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f094806ca90 con 0x7f094815e550 2024-04-02T13:25:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f094d388640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f094815f040 0x7f09480d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f094d388640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f094815f040 0x7f09480d0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60014/0 (socket says 172.21.15.44:60014) 2024-04-02T13:25:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f094d388640 1 -- 172.21.15.44:0/1463442255 learned_addr learned my addr 172.21.15.44:0/1463442255 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f093ffff640 1 --1- 172.21.15.44:0/1463442255 >> v1:172.21.15.44:6789/0 conn(0x7f094806d530 0x7f094815e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48478/0 (socket says 172.21.15.44:48478) 2024-04-02T13:25:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f094cb87640 1 --2- 172.21.15.44:0/1463442255 >> v2:172.21.15.69:3300/0 conn(0x7f094815e550 0x7f094806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f094d388640 1 -- 172.21.15.44:0/1463442255 >> v2:172.21.15.69:3300/0 conn(0x7f094815e550 msgr2=0x7f094806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f094d388640 1 --2- 172.21.15.44:0/1463442255 >> v2:172.21.15.69:3300/0 conn(0x7f094815e550 0x7f094806cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f094d388640 1 -- 172.21.15.44:0/1463442255 >> v1:172.21.15.44:6789/0 conn(0x7f094806d530 legacy=0x7f094815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f093f7fe640 1 -- 172.21.15.44:0/1463442255 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 816193418 0 0) 0x7f094815fb50 con 0x7f094806d530 2024-04-02T13:25:10.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.683+0000 7f094d388640 1 -- 172.21.15.44:0/1463442255 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094815f590 con 0x7f094815f040 2024-04-02T13:25:10.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.684+0000 7f094d388640 1 --2- 172.21.15.44:0/1463442255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094815f040 0x7f09480d0ab0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f09480b49c0 tx=0x7f094002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3006f61e71f574e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:10.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.684+0000 7f093f7fe640 1 -- 172.21.15.44:0/1463442255 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094002cbb0 con 0x7f094815f040 2024-04-02T13:25:10.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.684+0000 7f093f7fe640 1 -- 172.21.15.44:0/1463442255 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f094002cd50 con 0x7f094815f040 2024-04-02T13:25:10.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.684+0000 7f093f7fe640 1 -- 172.21.15.44:0/1463442255 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940034610 con 0x7f094815f040 2024-04-02T13:25:10.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.684+0000 7f094edeb640 1 -- 172.21.15.44:0/1463442255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094815f040 msgr2=0x7f09480d0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.684+0000 7f094edeb640 1 --2- 172.21.15.44:0/1463442255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094815f040 0x7f09480d0ab0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f09480b49c0 tx=0x7f094002c010 comp rx=0 tx=0).stop 2024-04-02T13:25:10.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.685+0000 7f094edeb640 1 -- 172.21.15.44:0/1463442255 shutdown_connections 2024-04-02T13:25:10.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.685+0000 7f094edeb640 1 --2- 172.21.15.44:0/1463442255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094815f040 0x7f09480d0ab0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.685+0000 7f094edeb640 1 --2- 172.21.15.44:0/1463442255 >> v2:172.21.15.69:3300/0 conn(0x7f094815e550 0x7f094806cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.685+0000 7f094edeb640 1 -- 172.21.15.44:0/1463442255 >> 172.21.15.44:0/1463442255 conn(0x7f094809aa20 msgr2=0x7f094809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:10.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.685+0000 7f094edeb640 1 -- 172.21.15.44:0/1463442255 shutdown_connections 2024-04-02T13:25:10.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.685+0000 7f094edeb640 1 -- 172.21.15.44:0/1463442255 wait complete. 2024-04-02T13:25:10.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.686+0000 7f094edeb640 1 Processor -- start 2024-04-02T13:25:10.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.686+0000 7f094edeb640 1 -- start start 2024-04-02T13:25:10.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094edeb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f094806d530 0x7f09480cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094edeb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094815e550 0x7f09480b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094edeb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094815f040 0x7f0948154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094edeb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09480d0590 con 0x7f094815f040 2024-04-02T13:25:10.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094edeb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09480d02b0 con 0x7f094806d530 2024-04-02T13:25:10.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094edeb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f09480cc4b0 con 0x7f094815e550 2024-04-02T13:25:10.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094d388640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094815f040 0x7f0948154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094d388640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094815f040 0x7f0948154a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60030/0 (socket says 172.21.15.44:60030) 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094d388640 1 -- 172.21.15.44:0/2869650219 learned_addr learned my addr 172.21.15.44:0/2869650219 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094cb87640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f094806d530 0x7f09480cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f093ffff640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094815e550 0x7f09480b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094d388640 1 -- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094815e550 msgr2=0x7f09480b7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094d388640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094815e550 0x7f09480b7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094d388640 1 -- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f094806d530 msgr2=0x7f09480cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094d388640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f094806d530 0x7f09480cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094d388640 1 -- 172.21.15.44:0/2869650219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09481f3d20 con 0x7f094815f040 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094d388640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094815f040 0x7f0948154a40 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f0940009f40 tx=0x7f0940004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f093d7fa640 1 -- 172.21.15.44:0/2869650219 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940043020 con 0x7f094815f040 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f093d7fa640 1 -- 172.21.15.44:0/2869650219 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f094003e450 con 0x7f094815f040 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094edeb640 1 -- 172.21.15.44:0/2869650219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09481f4000 con 0x7f094815f040 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f093d7fa640 1 -- 172.21.15.44:0/2869650219 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940009510 con 0x7f094815f040 2024-04-02T13:25:10.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.687+0000 7f094edeb640 1 -- 172.21.15.44:0/2869650219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09481f45c0 con 0x7f094815f040 2024-04-02T13:25:10.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.689+0000 7f094edeb640 1 -- 172.21.15.44:0/2869650219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09480684f0 con 0x7f094815f040 2024-04-02T13:25:10.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.689+0000 7f093d7fa640 1 -- 172.21.15.44:0/2869650219 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0940046020 con 0x7f094815f040 2024-04-02T13:25:10.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.689+0000 7f093d7fa640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f093406c8f0 0x7f093406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.689+0000 7f093d7fa640 1 -- 172.21.15.44:0/2869650219 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f09400ae300 con 0x7f094815f040 2024-04-02T13:25:10.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.690+0000 7f094cb87640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f093406c8f0 0x7f093406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.690+0000 7f094cb87640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f093406c8f0 0x7f093406edb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f09440079a0 tx=0x7f0944008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:10.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.694+0000 7f093d7fa640 1 -- 172.21.15.44:0/2869650219 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f094004d030 con 0x7f094815f040 2024-04-02T13:25:10.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.919+0000 7f094edeb640 1 -- 172.21.15.44:0/2869650219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0948067d70 con 0x7f094815f040 2024-04-02T13:25:10.922 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:25:10.919+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:25:10.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.923+0000 7f093d7fa640 1 -- 172.21.15.44:0/2869650219 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v45) ==== 161+0+0 (secure 0 0 0) 0x7f094007bfe0 con 0x7f094815f040 2024-04-02T13:25:10.926 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:25:10.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.925+0000 7f094edeb640 1 -- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f093406c8f0 msgr2=0x7f093406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.925+0000 7f094edeb640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f093406c8f0 0x7f093406edb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f09440079a0 tx=0x7f0944008040 comp rx=0 tx=0).stop 2024-04-02T13:25:10.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.925+0000 7f094edeb640 1 -- 172.21.15.44:0/2869650219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094815f040 msgr2=0x7f0948154a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.925+0000 7f094edeb640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094815f040 0x7f0948154a40 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f0940009f40 tx=0x7f0940004a40 comp rx=0 tx=0).stop 2024-04-02T13:25:10.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.925+0000 7f094edeb640 1 -- 172.21.15.44:0/2869650219 shutdown_connections 2024-04-02T13:25:10.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.925+0000 7f094edeb640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f093406c8f0 0x7f093406edb0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.925+0000 7f094edeb640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094815f040 0x7f0948154a40 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.925+0000 7f094edeb640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094815e550 0x7f09480b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.925+0000 7f094edeb640 1 --2- 172.21.15.44:0/2869650219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f094806d530 0x7f09480cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.925+0000 7f094edeb640 1 -- 172.21.15.44:0/2869650219 >> 172.21.15.44:0/2869650219 conn(0x7f094809aa20 msgr2=0x7f094815e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:10.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.925+0000 7f094edeb640 1 -- 172.21.15.44:0/2869650219 shutdown_connections 2024-04-02T13:25:10.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.925+0000 7f094edeb640 1 -- 172.21.15.44:0/2869650219 wait complete. 2024-04-02T13:25:10.931 INFO:tasks.ceph.mds.d.smithi069.stderr: -3434> 2024-04-02T13:25:00.217+0000 7f1f76d60640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:25:10.939 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-04-02T13:25:10.959 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-04-02T13:25:10.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.979+0000 7f7fdce3df40 1 Processor -- start 2024-04-02T13:25:10.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.979+0000 7f7fdce3df40 1 -- start start 2024-04-02T13:25:10.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fdce3df40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b401693ce0 0x55b401716a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fdce3df40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b40171c3c0 0x55b4016ac440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fdce3df40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40171c7a0 0x55b401665a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fdce3df40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b4016940f0 con 0x55b401693ce0 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fdce3df40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b40160b3e0 con 0x55b40171c7a0 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fdce3df40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55b40173d6b0 con 0x55b40171c3c0 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fda17a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b401693ce0 0x55b401716a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fda17a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b401693ce0 0x55b401716a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60052/0 (socket says 172.21.15.44:60052) 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fda17a640 1 -- 172.21.15.44:0/219294444 learned_addr learned my addr 172.21.15.44:0/219294444 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fd9979640 1 --2- 172.21.15.44:0/219294444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b40171c3c0 0x55b4016ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fda97b640 1 --2- 172.21.15.44:0/219294444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40171c7a0 0x55b401665a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fda17a640 1 -- 172.21.15.44:0/219294444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b40171c3c0 msgr2=0x55b4016ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fda17a640 1 --2- 172.21.15.44:0/219294444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b40171c3c0 0x55b4016ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fda17a640 1 -- 172.21.15.44:0/219294444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40171c7a0 msgr2=0x55b401665a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fda17a640 1 --2- 172.21.15.44:0/219294444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40171c7a0 0x55b401665a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fda17a640 1 -- 172.21.15.44:0/219294444 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b40171d2a0 con 0x55b401693ce0 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fda97b640 1 --2- 172.21.15.44:0/219294444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40171c7a0 0x55b401665a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fda17a640 1 --2- 172.21.15.44:0/219294444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b401693ce0 0x55b401716a30 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f7fd401cfc0 tx=0x7f7fd404c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17fff394e9cf662 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:10.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fd8977640 1 -- 172.21.15.44:0/219294444 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4057040 con 0x55b401693ce0 2024-04-02T13:25:10.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fd8977640 1 -- 172.21.15.44:0/219294444 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fd4062070 con 0x55b401693ce0 2024-04-02T13:25:10.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.980+0000 7f7fd8977640 1 -- 172.21.15.44:0/219294444 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4052c50 con 0x55b401693ce0 2024-04-02T13:25:10.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.981+0000 7f7fdce3df40 1 -- 172.21.15.44:0/219294444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b401693ce0 msgr2=0x55b401716a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.981+0000 7f7fdce3df40 1 --2- 172.21.15.44:0/219294444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b401693ce0 0x55b401716a30 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f7fd401cfc0 tx=0x7f7fd404c000 comp rx=0 tx=0).stop 2024-04-02T13:25:10.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.981+0000 7f7fdce3df40 1 -- 172.21.15.44:0/219294444 shutdown_connections 2024-04-02T13:25:10.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.981+0000 7f7fdce3df40 1 --2- 172.21.15.44:0/219294444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40171c7a0 0x55b401665a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.981+0000 7f7fdce3df40 1 --2- 172.21.15.44:0/219294444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b40171c3c0 0x55b4016ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.981+0000 7f7fdce3df40 1 --2- 172.21.15.44:0/219294444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b401693ce0 0x55b401716a30 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.981+0000 7f7fdce3df40 1 -- 172.21.15.44:0/219294444 >> 172.21.15.44:0/219294444 conn(0x55b4016632d0 msgr2=0x55b4016636b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:10.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.981+0000 7f7fdce3df40 1 -- 172.21.15.44:0/219294444 shutdown_connections 2024-04-02T13:25:10.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.981+0000 7f7fdce3df40 1 -- 172.21.15.44:0/219294444 wait complete. 2024-04-02T13:25:10.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.981+0000 7f7fdce3df40 1 Processor -- start 2024-04-02T13:25:10.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.982+0000 7f7fdce3df40 1 -- start start 2024-04-02T13:25:10.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.982+0000 7f7fdce3df40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b401693ce0 0x55b401731080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.982+0000 7f7fdce3df40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b40171c3c0 0x55b401665c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.982+0000 7f7fdce3df40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40171c7a0 0x55b40172eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.982+0000 7f7fdce3df40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b401717f50 con 0x55b401693ce0 2024-04-02T13:25:10.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.982+0000 7f7fdce3df40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b401718880 con 0x55b40171c7a0 2024-04-02T13:25:10.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.982+0000 7f7fdce3df40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55b401718230 con 0x55b40171c3c0 2024-04-02T13:25:10.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.982+0000 7f7fda17a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b401693ce0 0x55b401731080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.982+0000 7f7fda17a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b401693ce0 0x55b401731080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60056/0 (socket says 172.21.15.44:60056) 2024-04-02T13:25:10.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.982+0000 7f7fda17a640 1 -- 172.21.15.44:0/1042609663 learned_addr learned my addr 172.21.15.44:0/1042609663 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:10.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.982+0000 7f7fda97b640 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40171c7a0 0x55b40172eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.982+0000 7f7fd9979640 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b40171c3c0 0x55b401665c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.983+0000 7f7fda17a640 1 -- 172.21.15.44:0/1042609663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b40171c3c0 msgr2=0x55b401665c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.983+0000 7f7fda17a640 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b40171c3c0 0x55b401665c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.983+0000 7f7fda17a640 1 -- 172.21.15.44:0/1042609663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40171c7a0 msgr2=0x55b40172eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.983+0000 7f7fda17a640 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40171c7a0 0x55b40172eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.983+0000 7f7fda17a640 1 -- 172.21.15.44:0/1042609663 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b401716a30 con 0x55b401693ce0 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.983+0000 7f7fda17a640 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b401693ce0 0x55b401731080 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f7fd401bce0 tx=0x7f7fd4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.983+0000 7f7fd37fe640 1 -- 172.21.15.44:0/1042609663 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4057040 con 0x55b401693ce0 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.983+0000 7f7fd37fe640 1 -- 172.21.15.44:0/1042609663 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fd4052bf0 con 0x55b401693ce0 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.983+0000 7f7fdce3df40 1 -- 172.21.15.44:0/1042609663 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b4017159b0 con 0x55b401693ce0 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.983+0000 7f7fd37fe640 1 -- 172.21.15.44:0/1042609663 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd405d470 con 0x55b401693ce0 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.983+0000 7f7fdce3df40 1 -- 172.21.15.44:0/1042609663 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4016661c0 con 0x55b401693ce0 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.984+0000 7f7fd37fe640 1 -- 172.21.15.44:0/1042609663 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7fd406f020 con 0x55b401693ce0 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.985+0000 7f7fd37fe640 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7fbc06b2d0 0x7f7fbc06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.985+0000 7f7fd37fe640 1 -- 172.21.15.44:0/1042609663 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f7fd40ce100 con 0x55b401693ce0 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.985+0000 7f7fd9979640 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7fbc06b2d0 0x7f7fbc06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.985+0000 7f7fdce3df40 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55b40173ca00 0x55b4017d6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:10.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.985+0000 7f7fdce3df40 1 -- 172.21.15.44:0/1042609663 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:1 11.10 11:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80) -- 0x55b4016ab380 con 0x55b40173ca00 2024-04-02T13:25:10.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.985+0000 7f7fda97b640 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55b40173ca00 0x55b4017d6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:10.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.985+0000 7f7fd9979640 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7fbc06b2d0 0x7f7fbc06d790 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x55b4016946a0 tx=0x7f7fc40059c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:10.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.985+0000 7f7fda97b640 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55b40173ca00 0x55b4017d6360 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:10.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.988+0000 7f7fda97b640 1 -- 172.21.15.44:0/1042609663 <== osd.0 v2:172.21.15.44:6800/2303805525 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v80'11 uv11 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7fcc00f6a0 con 0x55b40173ca00 2024-04-02T13:25:10.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.988+0000 7f7fdce3df40 1 -- 172.21.15.44:0/1042609663 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55b40173ca00 msgr2=0x55b4017d6360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.988+0000 7f7fdce3df40 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55b40173ca00 0x55b4017d6360 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.989+0000 7f7fdce3df40 1 -- 172.21.15.44:0/1042609663 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7fbc06b2d0 msgr2=0x7f7fbc06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.989+0000 7f7fdce3df40 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7fbc06b2d0 0x7f7fbc06d790 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x55b4016946a0 tx=0x7f7fc40059c0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.989+0000 7f7fdce3df40 1 -- 172.21.15.44:0/1042609663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b401693ce0 msgr2=0x55b401731080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:10.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.989+0000 7f7fdce3df40 1 --2- 172.21.15.44:0/1042609663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b401693ce0 0x55b401731080 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f7fd401bce0 tx=0x7f7fd4002fb0 comp rx=0 tx=0).stop 2024-04-02T13:25:10.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.989+0000 7f7fda97b640 1 -- 172.21.15.44:0/1042609663 reap_dead start 2024-04-02T13:25:10.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.989+0000 7f7fdce3df40 1 -- 172.21.15.44:0/1042609663 shutdown_connections 2024-04-02T13:25:10.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.989+0000 7f7fdce3df40 1 -- 172.21.15.44:0/1042609663 >> 172.21.15.44:0/1042609663 conn(0x55b4016632d0 msgr2=0x55b4016636b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:10.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.989+0000 7f7fdce3df40 1 -- 172.21.15.44:0/1042609663 shutdown_connections 2024-04-02T13:25:10.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:10.989+0000 7f7fdce3df40 1 -- 172.21.15.44:0/1042609663 wait complete. 2024-04-02T13:25:10.994 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-02T13:25:11.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.136+0000 7fd3b62d3640 1 Processor -- start 2024-04-02T13:25:11.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.136+0000 7fd3b62d3640 1 -- start start 2024-04-02T13:25:11.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.137+0000 7fd3b62d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3b015e4a0 0x7fd3b0070680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:11.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.137+0000 7fd3b62d3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd3b00d03b0 0x7fd3b00cb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:11.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.137+0000 7fd3b62d3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3b015e880 con 0x7fd3b00cb990 2024-04-02T13:25:11.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.137+0000 7fd3b62d3640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd3b015f8a0 con 0x7fd3b00d03b0 2024-04-02T13:25:11.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.137+0000 7fd3b62d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b00cbdc0 con 0x7fd3b015e4a0 2024-04-02T13:25:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.137+0000 7fd3af7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd3b00d03b0 0x7fd3b00cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.137+0000 7fd3af7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd3b00d03b0 0x7fd3b00cb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60068/0 (socket says 172.21.15.44:60068) 2024-04-02T13:25:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.137+0000 7fd3af7fe640 1 -- 172.21.15.44:0/716804189 learned_addr learned my addr 172.21.15.44:0/716804189 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.137+0000 7fd3affff640 1 --2- 172.21.15.44:0/716804189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3b015e4a0 0x7fd3b0070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.137+0000 7fd3af7fe640 1 -- 172.21.15.44:0/716804189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3b015e4a0 msgr2=0x7fd3b0070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.137+0000 7fd3af7fe640 1 --2- 172.21.15.44:0/716804189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3b015e4a0 0x7fd3b0070680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.137+0000 7fd3af7fe640 1 -- 172.21.15.44:0/716804189 >> v1:172.21.15.69:6789/0 conn(0x7fd3b00cb990 legacy=0x7fd3b0072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.137+0000 7fd3af7fe640 1 -- 172.21.15.44:0/716804189 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b00cebf0 con 0x7fd3b00d03b0 2024-04-02T13:25:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.138+0000 7fd3af7fe640 1 --2- 172.21.15.44:0/716804189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b00d03b0 0x7fd3b00cb450 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009870 tx=0x7fd3a002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917ff9ce0bd50056 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.138+0000 7fd3aeffd640 1 -- 172.21.15.44:0/716804189 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0035020 con 0x7fd3b00d03b0 2024-04-02T13:25:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.138+0000 7fd3aeffd640 1 -- 172.21.15.44:0/716804189 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a0031640 con 0x7fd3b00d03b0 2024-04-02T13:25:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.138+0000 7fd3aeffd640 1 -- 172.21.15.44:0/716804189 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a002d040 con 0x7fd3b00d03b0 2024-04-02T13:25:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.139+0000 7fd3b62d3640 1 -- 172.21.15.44:0/716804189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b00d03b0 msgr2=0x7fd3b00cb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.139+0000 7fd3b62d3640 1 --2- 172.21.15.44:0/716804189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b00d03b0 0x7fd3b00cb450 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009870 tx=0x7fd3a002b8f0 comp rx=0 tx=0).stop 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.139+0000 7fd3b62d3640 1 -- 172.21.15.44:0/716804189 shutdown_connections 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.139+0000 7fd3b62d3640 1 --2- 172.21.15.44:0/716804189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b00d03b0 0x7fd3b00cb450 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.139+0000 7fd3b62d3640 1 --2- 172.21.15.44:0/716804189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3b015e4a0 0x7fd3b0070680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.139+0000 7fd3b62d3640 1 -- 172.21.15.44:0/716804189 >> 172.21.15.44:0/716804189 conn(0x7fd3b00d0e00 msgr2=0x7fd3b00d1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.139+0000 7fd3b62d3640 1 -- 172.21.15.44:0/716804189 shutdown_connections 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.139+0000 7fd3b62d3640 1 -- 172.21.15.44:0/716804189 wait complete. 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.140+0000 7fd3b62d3640 1 Processor -- start 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.140+0000 7fd3b62d3640 1 -- start start 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3b62d3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b00cb990 0x7fd3b01563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3b62d3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3b00d03b0 0x7fd3b00b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3b62d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3b015e4a0 0x7fd3b00b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3b62d3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b015f0c0 con 0x7fd3b00cb990 2024-04-02T13:25:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3b62d3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b015ede0 con 0x7fd3b00d03b0 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3b62d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b015f3a0 con 0x7fd3b015e4a0 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3affff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b00cb990 0x7fd3b01563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3affff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b00cb990 0x7fd3b01563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60080/0 (socket says 172.21.15.44:60080) 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3affff640 1 -- 172.21.15.44:0/3487602073 learned_addr learned my addr 172.21.15.44:0/3487602073 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3b4870640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3b015e4a0 0x7fd3b00b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3af7fe640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3b00d03b0 0x7fd3b00b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3affff640 1 -- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3b015e4a0 msgr2=0x7fd3b00b3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3affff640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3b015e4a0 0x7fd3b00b3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3affff640 1 -- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3b00d03b0 msgr2=0x7fd3b00b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3affff640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3b00d03b0 0x7fd3b00b3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3affff640 1 -- 172.21.15.44:0/3487602073 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b0159df0 con 0x7fd3b00cb990 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.141+0000 7fd3affff640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b00cb990 0x7fd3b01563d0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400d5f0 tx=0x7fd3a400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.142+0000 7fd3acff9640 1 -- 172.21.15.44:0/3487602073 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40125f0 con 0x7fd3b00cb990 2024-04-02T13:25:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.142+0000 7fd3acff9640 1 -- 172.21.15.44:0/3487602073 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a4019070 con 0x7fd3b00cb990 2024-04-02T13:25:11.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.142+0000 7fd3b62d3640 1 -- 172.21.15.44:0/3487602073 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b00b4590 con 0x7fd3b00cb990 2024-04-02T13:25:11.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.142+0000 7fd3acff9640 1 -- 172.21.15.44:0/3487602073 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40129d0 con 0x7fd3b00cb990 2024-04-02T13:25:11.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.142+0000 7fd3b62d3640 1 -- 172.21.15.44:0/3487602073 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b00b50f0 con 0x7fd3b00cb990 2024-04-02T13:25:11.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.143+0000 7fd3b62d3640 1 -- 172.21.15.44:0/3487602073 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3b00684f0 con 0x7fd3b00cb990 2024-04-02T13:25:11.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.143+0000 7fd3acff9640 1 -- 172.21.15.44:0/3487602073 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd3a4012b70 con 0x7fd3b00cb990 2024-04-02T13:25:11.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.143+0000 7fd3acff9640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd39c06c8f0 0x7fd39c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:11.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.144+0000 7fd3acff9640 1 -- 172.21.15.44:0/3487602073 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fd3a408bfe0 con 0x7fd3b00cb990 2024-04-02T13:25:11.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.144+0000 7fd3af7fe640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd39c06c8f0 0x7fd39c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:11.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.144+0000 7fd3af7fe640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd39c06c8f0 0x7fd39c06edb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0036000 tx=0x7fd3a003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:11.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.148+0000 7fd3acff9640 1 -- 172.21.15.44:0/3487602073 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd3a4059be0 con 0x7fd3b00cb990 2024-04-02T13:25:11.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.371+0000 7fd3b62d3640 1 -- 172.21.15.44:0/3487602073 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fd3b00cb770 con 0x7fd3b00cb990 2024-04-02T13:25:11.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.930+0000 7fd3acff9640 1 -- 172.21.15.44:0/3487602073 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v46) ==== 162+0+0 (secure 0 0 0) 0x7fd3a405dee0 con 0x7fd3b00cb990 2024-04-02T13:25:11.933 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-02T13:25:11.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.932+0000 7fd3b62d3640 1 -- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd39c06c8f0 msgr2=0x7fd39c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:11.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.932+0000 7fd3b62d3640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd39c06c8f0 0x7fd39c06edb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0036000 tx=0x7fd3a003c040 comp rx=0 tx=0).stop 2024-04-02T13:25:11.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.932+0000 7fd3b62d3640 1 -- 172.21.15.44:0/3487602073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b00cb990 msgr2=0x7fd3b01563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:11.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.932+0000 7fd3b62d3640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b00cb990 0x7fd3b01563d0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400d5f0 tx=0x7fd3a400dab0 comp rx=0 tx=0).stop 2024-04-02T13:25:11.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.932+0000 7fd3b62d3640 1 -- 172.21.15.44:0/3487602073 shutdown_connections 2024-04-02T13:25:11.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.932+0000 7fd3b62d3640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd39c06c8f0 0x7fd39c06edb0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:11.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.932+0000 7fd3b62d3640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3b015e4a0 0x7fd3b00b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:11.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.932+0000 7fd3b62d3640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3b00d03b0 0x7fd3b00b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:11.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.932+0000 7fd3b62d3640 1 --2- 172.21.15.44:0/3487602073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b00cb990 0x7fd3b01563d0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:11.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.932+0000 7fd3b62d3640 1 -- 172.21.15.44:0/3487602073 >> 172.21.15.44:0/3487602073 conn(0x7fd3b00d0e00 msgr2=0x7fd3b00b1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:11.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.933+0000 7fd3b62d3640 1 -- 172.21.15.44:0/3487602073 shutdown_connections 2024-04-02T13:25:11.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:11.933+0000 7fd3b62d3640 1 -- 172.21.15.44:0/3487602073 wait complete. 2024-04-02T13:25:11.951 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:25:12.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.055+0000 7ff588b82640 1 Processor -- start 2024-04-02T13:25:12.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.055+0000 7ff588b82640 1 -- start start 2024-04-02T13:25:12.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.056+0000 7ff588b82640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5840cffd0 0x7ff5840ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:12.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.056+0000 7ff588b82640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff5840cd3a0 0x7ff5841629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:12.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.056+0000 7ff588b82640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5840cef30 con 0x7ff5841617e0 2024-04-02T13:25:12.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.056+0000 7ff588b82640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff584162f00 con 0x7ff5840cd3a0 2024-04-02T13:25:12.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.056+0000 7ff588b82640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff5840aea90 con 0x7ff5840cffd0 2024-04-02T13:25:12.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.056+0000 7ff582d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff5840cd3a0 0x7ff5841629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:12.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.056+0000 7ff582d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff5840cd3a0 0x7ff5841629c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60094/0 (socket says 172.21.15.44:60094) 2024-04-02T13:25:12.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.056+0000 7ff582d9d640 1 -- 172.21.15.44:0/163408394 learned_addr learned my addr 172.21.15.44:0/163408394 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:12.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.056+0000 7ff581d9b640 1 --2- 172.21.15.44:0/163408394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5840cffd0 0x7ff5840ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:12.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.056+0000 7ff582d9d640 1 -- 172.21.15.44:0/163408394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5840cffd0 msgr2=0x7ff5840ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:12.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.056+0000 7ff582d9d640 1 --2- 172.21.15.44:0/163408394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5840cffd0 0x7ff5840ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:12.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.056+0000 7ff582d9d640 1 -- 172.21.15.44:0/163408394 >> v1:172.21.15.69:6789/0 conn(0x7ff5841617e0 legacy=0x7ff5840ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:12.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.056+0000 7ff582d9d640 1 -- 172.21.15.44:0/163408394 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5840cf210 con 0x7ff5840cd3a0 2024-04-02T13:25:12.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.057+0000 7ff582d9d640 1 --2- 172.21.15.44:0/163408394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5840cd3a0 0x7ff5841629c0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009e50 tx=0x7ff57c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d94ba53885e561 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:12.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.057+0000 7ff58159a640 1 -- 172.21.15.44:0/163408394 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57c02ca70 con 0x7ff5840cd3a0 2024-04-02T13:25:12.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.057+0000 7ff58159a640 1 -- 172.21.15.44:0/163408394 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff57c02cc10 con 0x7ff5840cd3a0 2024-04-02T13:25:12.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.057+0000 7ff58159a640 1 -- 172.21.15.44:0/163408394 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57c0343f0 con 0x7ff5840cd3a0 2024-04-02T13:25:12.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.057+0000 7ff588b82640 1 -- 172.21.15.44:0/163408394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5840cd3a0 msgr2=0x7ff5841629c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:12.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.057+0000 7ff588b82640 1 --2- 172.21.15.44:0/163408394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5840cd3a0 0x7ff5841629c0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009e50 tx=0x7ff57c02bed0 comp rx=0 tx=0).stop 2024-04-02T13:25:12.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.058+0000 7ff588b82640 1 -- 172.21.15.44:0/163408394 shutdown_connections 2024-04-02T13:25:12.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.058+0000 7ff588b82640 1 --2- 172.21.15.44:0/163408394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5840cd3a0 0x7ff5841629c0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:12.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.058+0000 7ff588b82640 1 --2- 172.21.15.44:0/163408394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5840cffd0 0x7ff5840ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:12.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.058+0000 7ff588b82640 1 -- 172.21.15.44:0/163408394 >> 172.21.15.44:0/163408394 conn(0x7ff5840b20e0 msgr2=0x7ff5840b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:12.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.058+0000 7ff588b82640 1 -- 172.21.15.44:0/163408394 shutdown_connections 2024-04-02T13:25:12.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.058+0000 7ff588b82640 1 -- 172.21.15.44:0/163408394 wait complete. 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.059+0000 7ff588b82640 1 Processor -- start 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.059+0000 7ff588b82640 1 -- start start 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff588b82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5840cd3a0 0x7ff5841484c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff588b82640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5840cffd0 0x7ff58414b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff588b82640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5841617e0 0x7ff5840b5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff588b82640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff5840cfab0 con 0x7ff5840cd3a0 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff588b82640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff5840cf7d0 con 0x7ff5841617e0 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff588b82640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff584161bc0 con 0x7ff5840cffd0 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff58259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5840cd3a0 0x7ff5841484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff58259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5840cd3a0 0x7ff5841484c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60104/0 (socket says 172.21.15.44:60104) 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff58259c640 1 -- 172.21.15.44:0/1378018167 learned_addr learned my addr 172.21.15.44:0/1378018167 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff58259c640 1 -- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5840cffd0 msgr2=0x7ff58414b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff582d9d640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5841617e0 0x7ff5840b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:12.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff581d9b640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5840cffd0 0x7ff58414b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:12.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff58259c640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5840cffd0 0x7ff58414b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:12.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff58259c640 1 -- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5841617e0 msgr2=0x7ff5840b5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:12.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff58259c640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5841617e0 0x7ff5840b5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:12.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff58259c640 1 -- 172.21.15.44:0/1378018167 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff584148a00 con 0x7ff5840cd3a0 2024-04-02T13:25:12.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff58259c640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5840cd3a0 0x7ff5841484c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7ff57400c3c0 tx=0x7ff57400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:12.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff572ffd640 1 -- 172.21.15.44:0/1378018167 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5740052d0 con 0x7ff5840cd3a0 2024-04-02T13:25:12.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff572ffd640 1 -- 172.21.15.44:0/1378018167 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff574005490 con 0x7ff5840cd3a0 2024-04-02T13:25:12.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff588b82640 1 -- 172.21.15.44:0/1378018167 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5840c77d0 con 0x7ff5840cd3a0 2024-04-02T13:25:12.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff572ffd640 1 -- 172.21.15.44:0/1378018167 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff574007260 con 0x7ff5840cd3a0 2024-04-02T13:25:12.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.060+0000 7ff588b82640 1 -- 172.21.15.44:0/1378018167 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5840b4d40 con 0x7ff5840cd3a0 2024-04-02T13:25:12.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.061+0000 7ff588b82640 1 -- 172.21.15.44:0/1378018167 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5840684f0 con 0x7ff5840cd3a0 2024-04-02T13:25:12.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.062+0000 7ff572ffd640 1 -- 172.21.15.44:0/1378018167 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff574007400 con 0x7ff5840cd3a0 2024-04-02T13:25:12.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.062+0000 7ff572ffd640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff56406c8f0 0x7ff56406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:12.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.062+0000 7ff572ffd640 1 -- 172.21.15.44:0/1378018167 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7ff57408b1a0 con 0x7ff5840cd3a0 2024-04-02T13:25:12.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.062+0000 7ff581d9b640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff56406c8f0 0x7ff56406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:12.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.063+0000 7ff581d9b640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff56406c8f0 0x7ff56406edb0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff5841607c0 tx=0x7ff578008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:12.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.067+0000 7ff572ffd640 1 -- 172.21.15.44:0/1378018167 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff584148a00 con 0x7ff5840cd3a0 2024-04-02T13:25:12.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.289+0000 7ff588b82640 1 -- 172.21.15.44:0/1378018167 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5840c7cf0 con 0x7ff5840cd3a0 2024-04-02T13:25:12.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.290+0000 7ff572ffd640 1 -- 172.21.15.44:0/1378018167 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+5177 (secure 0 0 0) 0x7ff574059480 con 0x7ff5840cd3a0 2024-04-02T13:25:12.293 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:12.293 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":47,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3662927745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3662927745},{"type":"v1","addr":"172.21.15.44:6835","nonce":3662927745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6394,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1053037366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1053037366},{"type":"v1","addr":"172.21.15.69:6836","nonce":1053037366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3893800047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3893800047},{"type":"v1","addr":"172.21.15.69:6837","nonce":3893800047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:24:53.925714+0000","modified":"2024-04-02T13:25:11.933410+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":47,"state":"up:replay","state_seq":1,"addr":"172.21.15.44:6833/2392159648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2392159648},{"type":"v1","addr":"172.21.15.44:6833","nonce":2392159648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-04-02T13:25:12.294 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 47 2024-04-02T13:25:12.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.292+0000 7ff588b82640 1 -- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff56406c8f0 msgr2=0x7ff56406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:12.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.292+0000 7ff588b82640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff56406c8f0 0x7ff56406edb0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff5841607c0 tx=0x7ff578008040 comp rx=0 tx=0).stop 2024-04-02T13:25:12.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.292+0000 7ff588b82640 1 -- 172.21.15.44:0/1378018167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5840cd3a0 msgr2=0x7ff5841484c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:12.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.292+0000 7ff588b82640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5840cd3a0 0x7ff5841484c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7ff57400c3c0 tx=0x7ff57400c880 comp rx=0 tx=0).stop 2024-04-02T13:25:12.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.292+0000 7ff588b82640 1 -- 172.21.15.44:0/1378018167 shutdown_connections 2024-04-02T13:25:12.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.292+0000 7ff588b82640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff56406c8f0 0x7ff56406edb0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:12.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.292+0000 7ff588b82640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5841617e0 0x7ff5840b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:12.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.292+0000 7ff588b82640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5840cffd0 0x7ff58414b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:12.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.292+0000 7ff588b82640 1 --2- 172.21.15.44:0/1378018167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5840cd3a0 0x7ff5841484c0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:12.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.292+0000 7ff588b82640 1 -- 172.21.15.44:0/1378018167 >> 172.21.15.44:0/1378018167 conn(0x7ff5840b20e0 msgr2=0x7ff5840d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:12.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.292+0000 7ff588b82640 1 -- 172.21.15.44:0/1378018167 shutdown_connections 2024-04-02T13:25:12.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:12.292+0000 7ff588b82640 1 -- 172.21.15.44:0/1378018167 wait complete. 2024-04-02T13:25:12.709 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:24:53.925714+0000', 'modified': '2024-04-02T13:25:11.933410+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 80, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6987}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6987': {'gid': 6987, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.44:6833/2392159648', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6832', 'nonce': 2392159648}, {'type': 'v1', 'addr': '172.21.15.44:6833', 'nonce': 2392159648}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-02T13:25:12.709 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6987:up:replay 2024-04-02T13:25:13.709 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:25:13.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.820+0000 7f4131b9d640 1 Processor -- start 2024-04-02T13:25:13.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.820+0000 7f4131b9d640 1 -- start start 2024-04-02T13:25:13.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.821+0000 7f4131b9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f412c0d0c50 0x7f412c0d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:13.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.821+0000 7f4131b9d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f412c0cc600 con 0x7f412c15e4a0 2024-04-02T13:25:13.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.821+0000 7f4131b9d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f412c0cc8e0 con 0x7f412c06ce40 2024-04-02T13:25:13.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.821+0000 7f4131b9d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f412c0af300 con 0x7f412c0d0c50 2024-04-02T13:25:13.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.821+0000 7f412b7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f412c15e4a0 0x7f412c0aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48488/0 (socket says 172.21.15.44:48488) 2024-04-02T13:25:13.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.821+0000 7f412affd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f412c0d0c50 0x7f412c0d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:13.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.821+0000 7f412b7fe640 1 -- 172.21.15.44:0/1007404029 learned_addr learned my addr 172.21.15.44:0/1007404029 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:13.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.821+0000 7f412a7fc640 1 -- 172.21.15.44:0/1007404029 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344960410 0 0) 0x7f412c0cc600 con 0x7f412c15e4a0 2024-04-02T13:25:13.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.821+0000 7f412a7fc640 1 -- 172.21.15.44:0/1007404029 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4114003520 con 0x7f412c15e4a0 2024-04-02T13:25:13.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.821+0000 7f412affd640 1 -- 172.21.15.44:0/1007404029 >> v1:172.21.15.69:6789/0 conn(0x7f412c06ce40 legacy=0x7f412c0cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:13.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.822+0000 7f412affd640 1 -- 172.21.15.44:0/1007404029 >> v1:172.21.15.44:6789/0 conn(0x7f412c15e4a0 legacy=0x7f412c0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:13.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.822+0000 7f412affd640 1 -- 172.21.15.44:0/1007404029 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412c0ccbc0 con 0x7f412c0d0c50 2024-04-02T13:25:13.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.822+0000 7f412affd640 1 --2- 172.21.15.44:0/1007404029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f412c0d0c50 0x7f412c0d10b0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f41200097a0 tx=0x7f412002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72fe9017817e807 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:13.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.822+0000 7f412a7fc640 1 -- 172.21.15.44:0/1007404029 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4120035020 con 0x7f412c0d0c50 2024-04-02T13:25:13.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.822+0000 7f412a7fc640 1 -- 172.21.15.44:0/1007404029 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4120031830 con 0x7f412c0d0c50 2024-04-02T13:25:13.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.822+0000 7f412a7fc640 1 -- 172.21.15.44:0/1007404029 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4120031c00 con 0x7f412c0d0c50 2024-04-02T13:25:13.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.823+0000 7f4131b9d640 1 -- 172.21.15.44:0/1007404029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f412c0d0c50 msgr2=0x7f412c0d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:13.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.823+0000 7f4131b9d640 1 --2- 172.21.15.44:0/1007404029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f412c0d0c50 0x7f412c0d10b0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f41200097a0 tx=0x7f412002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:25:13.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.823+0000 7f4131b9d640 1 -- 172.21.15.44:0/1007404029 shutdown_connections 2024-04-02T13:25:13.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.823+0000 7f4131b9d640 1 --2- 172.21.15.44:0/1007404029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f412c0d0c50 0x7f412c0d10b0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:13.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.823+0000 7f4131b9d640 1 -- 172.21.15.44:0/1007404029 >> 172.21.15.44:0/1007404029 conn(0x7f412c09aa20 msgr2=0x7f412c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:13.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.823+0000 7f4131b9d640 1 -- 172.21.15.44:0/1007404029 shutdown_connections 2024-04-02T13:25:13.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.823+0000 7f4131b9d640 1 -- 172.21.15.44:0/1007404029 wait complete. 2024-04-02T13:25:13.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.824+0000 7f4131b9d640 1 Processor -- start 2024-04-02T13:25:13.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.824+0000 7f4131b9d640 1 -- start start 2024-04-02T13:25:13.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f4131b9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f412c06ce40 0x7f412c14c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:13.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f4131b9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f412c0d0c50 0x7f412c0ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:13.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f4131b9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f412c15e4a0 0x7f412c0cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:13.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f4131b9d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f412c0cd460 con 0x7f412c06ce40 2024-04-02T13:25:13.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f4131b9d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f412c0cd180 con 0x7f412c0d0c50 2024-04-02T13:25:13.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f4131b9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f412c15e880 con 0x7f412c15e4a0 2024-04-02T13:25:13.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f412b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f412c06ce40 0x7f412c14c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:13.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f412b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f412c06ce40 0x7f412c14c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60134/0 (socket says 172.21.15.44:60134) 2024-04-02T13:25:13.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f412b7fe640 1 -- 172.21.15.44:0/2702838868 learned_addr learned my addr 172.21.15.44:0/2702838868 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:13.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f412affd640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f412c0d0c50 0x7f412c0ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:13.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f412bfff640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f412c15e4a0 0x7f412c0cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:13.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f412b7fe640 1 -- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f412c15e4a0 msgr2=0x7f412c0cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:13.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f412b7fe640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f412c15e4a0 0x7f412c0cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:13.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f412b7fe640 1 -- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f412c0d0c50 msgr2=0x7f412c0ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:13.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.825+0000 7f412b7fe640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f412c0d0c50 0x7f412c0ce8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:13.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.826+0000 7f412b7fe640 1 -- 172.21.15.44:0/2702838868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412c15ac80 con 0x7f412c06ce40 2024-04-02T13:25:13.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.826+0000 7f412affd640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f412c0d0c50 0x7f412c0ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:13.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.826+0000 7f412b7fe640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f412c06ce40 0x7f412c14c250 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f411c00c460 tx=0x7f411c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:13.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.826+0000 7f4113fff640 1 -- 172.21.15.44:0/2702838868 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c017070 con 0x7f412c06ce40 2024-04-02T13:25:13.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.826+0000 7f4113fff640 1 -- 172.21.15.44:0/2702838868 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411c005ce0 con 0x7f412c06ce40 2024-04-02T13:25:13.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.826+0000 7f4131b9d640 1 -- 172.21.15.44:0/2702838868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f412c15af60 con 0x7f412c06ce40 2024-04-02T13:25:13.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.826+0000 7f4113fff640 1 -- 172.21.15.44:0/2702838868 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c007290 con 0x7f412c06ce40 2024-04-02T13:25:13.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.826+0000 7f4131b9d640 1 -- 172.21.15.44:0/2702838868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f412c15b520 con 0x7f412c06ce40 2024-04-02T13:25:13.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.827+0000 7f4131b9d640 1 -- 172.21.15.44:0/2702838868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f412c0684f0 con 0x7f412c06ce40 2024-04-02T13:25:13.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.827+0000 7f4113fff640 1 -- 172.21.15.44:0/2702838868 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f411c01a070 con 0x7f412c06ce40 2024-04-02T13:25:13.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.828+0000 7f4113fff640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f411806c8f0 0x7f411806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:13.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.828+0000 7f4113fff640 1 -- 172.21.15.44:0/2702838868 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f411c08b0a0 con 0x7f412c06ce40 2024-04-02T13:25:13.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.828+0000 7f412affd640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f411806c8f0 0x7f411806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:13.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.829+0000 7f412affd640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f411806c8f0 0x7f411806edb0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4120036000 tx=0x7f4120008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:13.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:13.832+0000 7f4113fff640 1 -- 172.21.15.44:0/2702838868 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f411c0592f0 con 0x7f412c06ce40 2024-04-02T13:25:14.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.056+0000 7f4131b9d640 1 -- 172.21.15.44:0/2702838868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f412c0b7c90 con 0x7f412c06ce40 2024-04-02T13:25:14.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.056+0000 7f4113fff640 1 -- 172.21.15.44:0/2702838868 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5177 (secure 0 0 0) 0x7f411c058bf0 con 0x7f412c06ce40 2024-04-02T13:25:14.060 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:14.060 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":49,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3662927745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3662927745},{"type":"v1","addr":"172.21.15.44:6835","nonce":3662927745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6394,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1053037366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1053037366},{"type":"v1","addr":"172.21.15.69:6836","nonce":1053037366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3893800047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3893800047},{"type":"v1","addr":"172.21.15.69:6837","nonce":3893800047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:24:53.925714+0000","modified":"2024-04-02T13:25:12.941591+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":47,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.44:6833/2392159648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2392159648},{"type":"v1","addr":"172.21.15.44:6833","nonce":2392159648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-04-02T13:25:14.060 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 49 2024-04-02T13:25:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.058+0000 7f4131b9d640 1 -- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f411806c8f0 msgr2=0x7f411806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.058+0000 7f4131b9d640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f411806c8f0 0x7f411806edb0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4120036000 tx=0x7f4120008030 comp rx=0 tx=0).stop 2024-04-02T13:25:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.058+0000 7f4131b9d640 1 -- 172.21.15.44:0/2702838868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f412c06ce40 msgr2=0x7f412c14c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.058+0000 7f4131b9d640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f412c06ce40 0x7f412c14c250 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f411c00c460 tx=0x7f411c00c920 comp rx=0 tx=0).stop 2024-04-02T13:25:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.058+0000 7f4131b9d640 1 -- 172.21.15.44:0/2702838868 shutdown_connections 2024-04-02T13:25:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.058+0000 7f4131b9d640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f411806c8f0 0x7f411806edb0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.058+0000 7f4131b9d640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f412c15e4a0 0x7f412c0cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.058+0000 7f4131b9d640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f412c0d0c50 0x7f412c0ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.058+0000 7f4131b9d640 1 --2- 172.21.15.44:0/2702838868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f412c06ce40 0x7f412c14c250 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.058+0000 7f4131b9d640 1 -- 172.21.15.44:0/2702838868 >> 172.21.15.44:0/2702838868 conn(0x7f412c09aa20 msgr2=0x7f412c15d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.058+0000 7f4131b9d640 1 -- 172.21.15.44:0/2702838868 shutdown_connections 2024-04-02T13:25:14.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:14.058+0000 7f4131b9d640 1 -- 172.21.15.44:0/2702838868 wait complete. 2024-04-02T13:25:14.476 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:24:53.925714+0000', 'modified': '2024-04-02T13:25:12.941591+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 80, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6987}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6987': {'gid': 6987, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.44:6833/2392159648', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6832', 'nonce': 2392159648}, {'type': 'v1', 'addr': '172.21.15.44:6833', 'nonce': 2392159648}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-02T13:25:14.476 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6987:up:rejoin 2024-04-02T13:25:15.219 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:25:15.217+0000 7f40dbbaf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:25:15.477 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:25:15.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.588+0000 7f7bf1655640 1 Processor -- start 2024-04-02T13:25:15.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.588+0000 7f7bf1655640 1 -- start start 2024-04-02T13:25:15.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7bf1655640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7bec159db0 0x7f7bec15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:15.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7bf1655640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bec0ccdf0 0x7f7bec165cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:15.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7bf1655640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bec0726c0 con 0x7f7bec071e50 2024-04-02T13:25:15.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7bf1655640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7bec073240 con 0x7f7bec159db0 2024-04-02T13:25:15.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7bf1655640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7bec0729a0 con 0x7f7bec0ccdf0 2024-04-02T13:25:15.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7beaffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7bec159db0 0x7f7bec15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:15.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7bea7fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7bec071e50 0x7f7bec0cc6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44610/0 (socket says 172.21.15.44:44610) 2024-04-02T13:25:15.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7beaffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7bec159db0 0x7f7bec15c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58100/0 (socket says 172.21.15.44:58100) 2024-04-02T13:25:15.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7bea7fc640 1 -- 172.21.15.44:0/3406368425 learned_addr learned my addr 172.21.15.44:0/3406368425 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:15.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7beaffd640 1 -- 172.21.15.44:0/3406368425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bec0ccdf0 msgr2=0x7f7bec165cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:15.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7beb7fe640 1 --2- 172.21.15.44:0/3406368425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bec0ccdf0 0x7f7bec165cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:15.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7be9ffb640 1 -- 172.21.15.44:0/3406368425 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4144768039 0 0) 0x7f7bec0726c0 con 0x7f7bec071e50 2024-04-02T13:25:15.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7beaffd640 1 --2- 172.21.15.44:0/3406368425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bec0ccdf0 0x7f7bec165cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:15.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7beaffd640 1 -- 172.21.15.44:0/3406368425 >> v1:172.21.15.44:6789/0 conn(0x7f7bec071e50 legacy=0x7f7bec0cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:15.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7beaffd640 1 -- 172.21.15.44:0/3406368425 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bec072c80 con 0x7f7bec159db0 2024-04-02T13:25:15.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.589+0000 7f7beb7fe640 1 --2- 172.21.15.44:0/3406368425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bec0ccdf0 0x7f7bec165cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:15.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.590+0000 7f7beaffd640 1 --2- 172.21.15.44:0/3406368425 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bec159db0 0x7f7bec15c6d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc009d90 tx=0x7f7bdc02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125cc547c7c5334d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:15.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.590+0000 7f7be9ffb640 1 -- 172.21.15.44:0/3406368425 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc039070 con 0x7f7bec159db0 2024-04-02T13:25:15.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.590+0000 7f7be9ffb640 1 -- 172.21.15.44:0/3406368425 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bdc02cbc0 con 0x7f7bec159db0 2024-04-02T13:25:15.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.590+0000 7f7be9ffb640 1 -- 172.21.15.44:0/3406368425 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc034440 con 0x7f7bec159db0 2024-04-02T13:25:15.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.591+0000 7f7bf1655640 1 -- 172.21.15.44:0/3406368425 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bec159db0 msgr2=0x7f7bec15c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:15.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.591+0000 7f7bf1655640 1 --2- 172.21.15.44:0/3406368425 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bec159db0 0x7f7bec15c6d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc009d90 tx=0x7f7bdc02be10 comp rx=0 tx=0).stop 2024-04-02T13:25:15.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.591+0000 7f7bf1655640 1 -- 172.21.15.44:0/3406368425 shutdown_connections 2024-04-02T13:25:15.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.591+0000 7f7bf1655640 1 --2- 172.21.15.44:0/3406368425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bec0ccdf0 0x7f7bec165cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:15.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.591+0000 7f7bf1655640 1 --2- 172.21.15.44:0/3406368425 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bec159db0 0x7f7bec15c6d0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:15.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.591+0000 7f7bf1655640 1 -- 172.21.15.44:0/3406368425 >> 172.21.15.44:0/3406368425 conn(0x7f7bec0cfcd0 msgr2=0x7f7bec0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:15.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.591+0000 7f7bf1655640 1 -- 172.21.15.44:0/3406368425 shutdown_connections 2024-04-02T13:25:15.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.591+0000 7f7bf1655640 1 -- 172.21.15.44:0/3406368425 wait complete. 2024-04-02T13:25:15.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.592+0000 7f7bf1655640 1 Processor -- start 2024-04-02T13:25:15.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.592+0000 7f7bf1655640 1 -- start start 2024-04-02T13:25:15.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.593+0000 7f7bf1655640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bec071e50 0x7f7bec09aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:15.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.593+0000 7f7bf1655640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bec0ccdf0 0x7f7bec15f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:15.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.593+0000 7f7bf1655640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bec159db0 0x7f7bec09c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:15.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.593+0000 7f7bf1655640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec072480 con 0x7f7bec071e50 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.593+0000 7f7bf1655640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec072300 con 0x7f7bec159db0 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.593+0000 7f7bf1655640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7bec06e650 con 0x7f7bec0ccdf0 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.593+0000 7f7beaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bec071e50 0x7f7bec09aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.593+0000 7f7beaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bec071e50 0x7f7bec09aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58114/0 (socket says 172.21.15.44:58114) 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.593+0000 7f7beaffd640 1 -- 172.21.15.44:0/2715904508 learned_addr learned my addr 172.21.15.44:0/2715904508 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.593+0000 7f7bea7fc640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bec0ccdf0 0x7f7bec15f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.593+0000 7f7beb7fe640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bec159db0 0x7f7bec09c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.594+0000 7f7beaffd640 1 -- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bec0ccdf0 msgr2=0x7f7bec15f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.594+0000 7f7beaffd640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bec0ccdf0 0x7f7bec15f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.594+0000 7f7beaffd640 1 -- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bec159db0 msgr2=0x7f7bec09c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.594+0000 7f7beaffd640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bec159db0 0x7f7bec09c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.594+0000 7f7beaffd640 1 -- 172.21.15.44:0/2715904508 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bec09b040 con 0x7f7bec071e50 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.594+0000 7f7bea7fc640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bec0ccdf0 0x7f7bec15f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.594+0000 7f7beaffd640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bec071e50 0x7f7bec09aa20 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc037040 tx=0x7f7bdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:15.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.594+0000 7f7bd37fe640 1 -- 172.21.15.44:0/2715904508 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc039050 con 0x7f7bec071e50 2024-04-02T13:25:15.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.594+0000 7f7bd37fe640 1 -- 172.21.15.44:0/2715904508 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bdc045070 con 0x7f7bec071e50 2024-04-02T13:25:15.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.594+0000 7f7bd37fe640 1 -- 172.21.15.44:0/2715904508 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc03ea50 con 0x7f7bec071e50 2024-04-02T13:25:15.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.594+0000 7f7bf1655640 1 -- 172.21.15.44:0/2715904508 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bec09b320 con 0x7f7bec071e50 2024-04-02T13:25:15.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.594+0000 7f7bf1655640 1 -- 172.21.15.44:0/2715904508 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bec09b8e0 con 0x7f7bec071e50 2024-04-02T13:25:15.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.595+0000 7f7bf1655640 1 -- 172.21.15.44:0/2715904508 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bec0684f0 con 0x7f7bec071e50 2024-04-02T13:25:15.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.595+0000 7f7bd37fe640 1 -- 172.21.15.44:0/2715904508 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7bdc02ca40 con 0x7f7bec071e50 2024-04-02T13:25:15.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.596+0000 7f7bd37fe640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7bd806c8f0 0x7f7bd806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:15.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.596+0000 7f7bd37fe640 1 -- 172.21.15.44:0/2715904508 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f7bdc0aead0 con 0x7f7bec071e50 2024-04-02T13:25:15.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.596+0000 7f7bea7fc640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7bd806c8f0 0x7f7bd806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:15.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.597+0000 7f7bea7fc640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7bd806c8f0 0x7f7bd806edb0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7bec09dbd0 tx=0x7f7be0007c10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:15.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.600+0000 7f7bd37fe640 1 -- 172.21.15.44:0/2715904508 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7bdc07ce40 con 0x7f7bec071e50 2024-04-02T13:25:15.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.823+0000 7f7bf1655640 1 -- 172.21.15.44:0/2715904508 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bec155e40 con 0x7f7bec071e50 2024-04-02T13:25:15.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.824+0000 7f7bd37fe640 1 -- 172.21.15.44:0/2715904508 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5184 (secure 0 0 0) 0x7f7bdc07c740 con 0x7f7bec071e50 2024-04-02T13:25:15.827 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:15.827 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":50,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3662927745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3662927745},{"type":"v1","addr":"172.21.15.44:6835","nonce":3662927745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6394,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1053037366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1053037366},{"type":"v1","addr":"172.21.15.69:6836","nonce":1053037366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3893800047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3893800047},{"type":"v1","addr":"172.21.15.69:6837","nonce":3893800047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:24:53.925714+0000","modified":"2024-04-02T13:25:14.947302+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.44:6833/2392159648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2392159648},{"type":"v1","addr":"172.21.15.44:6833","nonce":2392159648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":5}]} 2024-04-02T13:25:15.827 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 50 2024-04-02T13:25:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.826+0000 7f7bf1655640 1 -- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7bd806c8f0 msgr2=0x7f7bd806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.826+0000 7f7bf1655640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7bd806c8f0 0x7f7bd806edb0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7bec09dbd0 tx=0x7f7be0007c10 comp rx=0 tx=0).stop 2024-04-02T13:25:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.826+0000 7f7bf1655640 1 -- 172.21.15.44:0/2715904508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bec071e50 msgr2=0x7f7bec09aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.826+0000 7f7bf1655640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bec071e50 0x7f7bec09aa20 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc037040 tx=0x7f7bdc004a40 comp rx=0 tx=0).stop 2024-04-02T13:25:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.826+0000 7f7bf1655640 1 -- 172.21.15.44:0/2715904508 shutdown_connections 2024-04-02T13:25:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.826+0000 7f7bf1655640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7bd806c8f0 0x7f7bd806edb0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.826+0000 7f7bf1655640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bec159db0 0x7f7bec09c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.826+0000 7f7bf1655640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bec0ccdf0 0x7f7bec15f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.826+0000 7f7bf1655640 1 --2- 172.21.15.44:0/2715904508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bec071e50 0x7f7bec09aa20 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.826+0000 7f7bf1655640 1 -- 172.21.15.44:0/2715904508 >> 172.21.15.44:0/2715904508 conn(0x7f7bec0cfcd0 msgr2=0x7f7bec0cc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.826+0000 7f7bf1655640 1 -- 172.21.15.44:0/2715904508 shutdown_connections 2024-04-02T13:25:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:15.826+0000 7f7bf1655640 1 -- 172.21.15.44:0/2715904508 wait complete. 2024-04-02T13:25:16.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:24:53.925714+0000', 'modified': '2024-04-02T13:25:14.947302+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 80, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6987}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6987': {'gid': 6987, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.44:6833/2392159648', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6832', 'nonce': 2392159648}, {'type': 'v1', 'addr': '172.21.15.44:6833', 'nonce': 2392159648}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6987, 'qdb_cluster': [6987]} 2024-04-02T13:25:16.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:25:16.243 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T13:25:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.347+0000 7f5bcd00f640 1 Processor -- start 2024-04-02T13:25:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.347+0000 7f5bcd00f640 1 -- start start 2024-04-02T13:25:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.348+0000 7f5bcd00f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5bc815e4d0 0x7f5bc806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.348+0000 7f5bcd00f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bc806dfe0 con 0x7f5bc8072f60 2024-04-02T13:25:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.348+0000 7f5bcd00f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bc806e2c0 con 0x7f5bc815e0f0 2024-04-02T13:25:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.348+0000 7f5bcd00f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f5bc806e5a0 con 0x7f5bc815e4d0 2024-04-02T13:25:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.348+0000 7f5bc659c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5bc815e4d0 0x7f5bc806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.348+0000 7f5bc759e640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f5bc8072f60 0x7f5bc80add60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44626/0 (socket says 172.21.15.44:44626) 2024-04-02T13:25:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.348+0000 7f5bc759e640 1 -- 172.21.15.44:0/1666917210 learned_addr learned my addr 172.21.15.44:0/1666917210 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.348+0000 7f5bc659c640 1 -- 172.21.15.44:0/1666917210 >> v1:172.21.15.69:6789/0 conn(0x7f5bc815e0f0 legacy=0x7f5bc80d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.348+0000 7f5bc659c640 1 -- 172.21.15.44:0/1666917210 >> v1:172.21.15.44:6789/0 conn(0x7f5bc8072f60 legacy=0x7f5bc80add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.348+0000 7f5bc659c640 1 -- 172.21.15.44:0/1666917210 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc806e880 con 0x7f5bc815e4d0 2024-04-02T13:25:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.349+0000 7f5bc659c640 1 --2- 172.21.15.44:0/1666917210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc815e4d0 0x7f5bc806b4d0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc0097a0 tx=0x7f5bbc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99df3554b2d252f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.349+0000 7f5bc5d9b640 1 -- 172.21.15.44:0/1666917210 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc034070 con 0x7f5bc815e4d0 2024-04-02T13:25:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.349+0000 7f5bc5d9b640 1 -- 172.21.15.44:0/1666917210 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bbc030610 con 0x7f5bc815e4d0 2024-04-02T13:25:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.349+0000 7f5bc5d9b640 1 -- 172.21.15.44:0/1666917210 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc0309e0 con 0x7f5bc815e4d0 2024-04-02T13:25:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.350+0000 7f5bcd00f640 1 -- 172.21.15.44:0/1666917210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc815e4d0 msgr2=0x7f5bc806b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.350+0000 7f5bcd00f640 1 --2- 172.21.15.44:0/1666917210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc815e4d0 0x7f5bc806b4d0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc0097a0 tx=0x7f5bbc02b960 comp rx=0 tx=0).stop 2024-04-02T13:25:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.350+0000 7f5bcd00f640 1 -- 172.21.15.44:0/1666917210 shutdown_connections 2024-04-02T13:25:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.350+0000 7f5bcd00f640 1 --2- 172.21.15.44:0/1666917210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc815e4d0 0x7f5bc806b4d0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.350+0000 7f5bcd00f640 1 -- 172.21.15.44:0/1666917210 >> 172.21.15.44:0/1666917210 conn(0x7f5bc80b2980 msgr2=0x7f5bc80b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.350+0000 7f5bcd00f640 1 -- 172.21.15.44:0/1666917210 shutdown_connections 2024-04-02T13:25:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.350+0000 7f5bcd00f640 1 -- 172.21.15.44:0/1666917210 wait complete. 2024-04-02T13:25:16.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.351+0000 7f5bcd00f640 1 Processor -- start 2024-04-02T13:25:16.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.351+0000 7f5bcd00f640 1 -- start start 2024-04-02T13:25:16.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bcd00f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc8072f60 0x7f5bc8158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bcd00f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc815e0f0 0x7f5bc80c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bcd00f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bc815e4d0 0x7f5bc80c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bcd00f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc806eda0 con 0x7f5bc8072f60 2024-04-02T13:25:16.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bcd00f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc80d0800 con 0x7f5bc815e0f0 2024-04-02T13:25:16.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bcd00f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc806ba10 con 0x7f5bc815e4d0 2024-04-02T13:25:16.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bc6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc8072f60 0x7f5bc8158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bc6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc8072f60 0x7f5bc8158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58132/0 (socket says 172.21.15.44:58132) 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bc6d9d640 1 -- 172.21.15.44:0/4248673224 learned_addr learned my addr 172.21.15.44:0/4248673224 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bc659c640 1 --2- 172.21.15.44:0/4248673224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc815e0f0 0x7f5bc80c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bc6d9d640 1 -- 172.21.15.44:0/4248673224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bc815e4d0 msgr2=0x7f5bc80c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bc6d9d640 1 --2- 172.21.15.44:0/4248673224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bc815e4d0 0x7f5bc80c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bc6d9d640 1 -- 172.21.15.44:0/4248673224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc815e0f0 msgr2=0x7f5bc80c48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bc6d9d640 1 --2- 172.21.15.44:0/4248673224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc815e0f0 0x7f5bc80c48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bc6d9d640 1 -- 172.21.15.44:0/4248673224 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc80c4de0 con 0x7f5bc8072f60 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.352+0000 7f5bc6d9d640 1 --2- 172.21.15.44:0/4248673224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc8072f60 0x7f5bc8158480 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000c430 tx=0x7f5bc000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.353+0000 7f5baf7fe640 1 -- 172.21.15.44:0/4248673224 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0011790 con 0x7f5bc8072f60 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.353+0000 7f5baf7fe640 1 -- 172.21.15.44:0/4248673224 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bc0017070 con 0x7f5bc8072f60 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.353+0000 7f5baf7fe640 1 -- 172.21.15.44:0/4248673224 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0011bb0 con 0x7f5bc8072f60 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.353+0000 7f5bcd00f640 1 -- 172.21.15.44:0/4248673224 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc80c5080 con 0x7f5bc8072f60 2024-04-02T13:25:16.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.353+0000 7f5bcd00f640 1 -- 172.21.15.44:0/4248673224 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc80c5920 con 0x7f5bc8072f60 2024-04-02T13:25:16.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.354+0000 7f5baf7fe640 1 -- 172.21.15.44:0/4248673224 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5bc001c460 con 0x7f5bc8072f60 2024-04-02T13:25:16.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.354+0000 7f5baf7fe640 1 --2- 172.21.15.44:0/4248673224 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5bb406c8f0 0x7f5bb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.355+0000 7f5baf7fe640 1 -- 172.21.15.44:0/4248673224 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f5bc008bcf0 con 0x7f5bc8072f60 2024-04-02T13:25:16.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.355+0000 7f5bc659c640 1 --2- 172.21.15.44:0/4248673224 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5bb406c8f0 0x7f5bb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.355+0000 7f5bc659c640 1 --2- 172.21.15.44:0/4248673224 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5bb406c8f0 0x7f5bb406edb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc036000 tx=0x7f5bbc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:16.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.355+0000 7f5bcfbf1740 1 Processor -- start 2024-04-02T13:25:16.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.356+0000 7f5bcfbf1740 1 -- start start 2024-04-02T13:25:16.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.356+0000 7f5bcfbf1740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14b90 0x5568d7ae7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.356+0000 7f5bcfbf1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 0x5568d7b17920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.356+0000 7f5bcfbf1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7ae8070 0x5568d7b17e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.356+0000 7f5bcfbf1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5568d7a51060 con 0x5568d7b14f70 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.356+0000 7f5bcfbf1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5568d79f9650 con 0x5568d7ae8070 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.356+0000 7f5bcfbf1740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5568d7a4d010 con 0x5568d7b14b90 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bc659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 0x5568d7b17920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bc6d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14b90 0x5568d7ae7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bc659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 0x5568d7b17920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58148/0 (socket says 172.21.15.44:58148) 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bc659c640 1 -- 172.21.15.44:0/3760426554 learned_addr learned my addr 172.21.15.44:0/3760426554 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bc759e640 1 --2- 172.21.15.44:0/3760426554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7ae8070 0x5568d7b17e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bc659c640 1 -- 172.21.15.44:0/3760426554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14b90 msgr2=0x5568d7ae7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bc659c640 1 --2- 172.21.15.44:0/3760426554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14b90 0x5568d7ae7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bc659c640 1 -- 172.21.15.44:0/3760426554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7ae8070 msgr2=0x5568d7b17e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bc659c640 1 --2- 172.21.15.44:0/3760426554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7ae8070 0x5568d7b17e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bc659c640 1 -- 172.21.15.44:0/3760426554 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568d77652d0 con 0x5568d7b14f70 2024-04-02T13:25:16.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bc759e640 1 --2- 172.21.15.44:0/3760426554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7ae8070 0x5568d7b17e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bc659c640 1 --2- 172.21.15.44:0/3760426554 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 0x5568d7b17920 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009890 tx=0x7f5bbc030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ff6d77e8da0aa27 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bacff9640 1 -- 172.21.15.44:0/3760426554 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc056070 con 0x5568d7b14f70 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bacff9640 1 -- 172.21.15.44:0/3760426554 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bbc0464e0 con 0x5568d7b14f70 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.357+0000 7f5bacff9640 1 -- 172.21.15.44:0/3760426554 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc034040 con 0x5568d7b14f70 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.358+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/3760426554 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 msgr2=0x5568d7b17920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.358+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/3760426554 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 0x5568d7b17920 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009890 tx=0x7f5bbc030000 comp rx=0 tx=0).stop 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.358+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/3760426554 shutdown_connections 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.358+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/3760426554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7ae8070 0x5568d7b17e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.358+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/3760426554 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 0x5568d7b17920 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.358+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/3760426554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14b90 0x5568d7ae7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.358+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/3760426554 >> 172.21.15.44:0/3760426554 conn(0x5568d7ae9da0 msgr2=0x5568d7b32920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.358+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/3760426554 shutdown_connections 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.358+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/3760426554 wait complete. 2024-04-02T13:25:16.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.358+0000 7f5bcfbf1740 1 -- start start 2024-04-02T13:25:16.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.359+0000 7f5bcfbf1740 1 Processor -- start 2024-04-02T13:25:16.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.359+0000 7f5bcfbf1740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14b90 0x5568d7b33120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.359+0000 7f5bcfbf1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 0x5568d7b0d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.359+0000 7f5bcfbf1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b0ce20 0x5568d7b0ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.359+0000 7f5bcfbf1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5568d7aec440 con 0x5568d7b14f70 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.359+0000 7f5bcfbf1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5568d7a0ca10 con 0x5568d7b0ce20 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.359+0000 7f5bcfbf1740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5568d7aeec80 con 0x5568d7b14b90 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.359+0000 7f5bc659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 0x5568d7b0d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.359+0000 7f5bcfbf1740 10 client.0 _ll_register_callbacks cb 0x5568d7a490b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.359+0000 7f5bc659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 0x5568d7b0d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58158/0 (socket says 172.21.15.44:58158) 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.359+0000 7f5bc659c640 1 -- 172.21.15.44:0/3296117742 learned_addr learned my addr 172.21.15.44:0/3296117742 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.359+0000 7f5bc759e640 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b0ce20 0x5568d7b0ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bc6d9d640 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14b90 0x5568d7b33120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bc659c640 1 -- 172.21.15.44:0/3296117742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14b90 msgr2=0x5568d7b33120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bc659c640 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14b90 0x5568d7b33120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bc659c640 1 -- 172.21.15.44:0/3296117742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b0ce20 msgr2=0x5568d7b0ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bc659c640 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b0ce20 0x5568d7b0ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bc659c640 1 -- 172.21.15.44:0/3296117742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568d782f6b0 con 0x5568d7b14f70 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bc6d9d640 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14b90 0x5568d7b33120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bc659c640 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 0x5568d7b0d870 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009560 tx=0x7f5bbc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b257a3eee63b7436 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5ba5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/3296117742 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc061060 con 0x5568d7b14f70 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/3296117742 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f5bbc03a660 con 0x5568d7b14f70 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/3296117742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bbc0464e0 con 0x5568d7b14f70 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/3296117742 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f5bbc067050 con 0x5568d7b14f70 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bcfbf1740 10 client.7320 fetch_fsmap learned FSMap version 50 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/3296117742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x5568d7b16920 con 0x5568d7b14f70 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/3296117742 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bbc0464e0 con 0x5568d7b14f70 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/3296117742 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7f5bbc06a020 con 0x5568d7b14f70 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bcfbf1740 10 client.7320 fetch_fsmap finished waiting for FSMap version 50 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bcfbf1740 10 client.7320 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bcfbf1740 20 client.7320 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.360+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5568d7b17490 0x5568d7b10780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.361+0000 7f5bcfbf1740 4 client.7320 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.361+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/3296117742 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568d79f8cf0 con 0x5568d7b17490 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.361+0000 7f5bc6d9d640 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5568d7b17490 0x5568d7b10780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.361+0000 7f5ba5ffb640 10 client.7320 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.361+0000 7f5bc6d9d640 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5568d7b17490 0x5568d7b10780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e98b3484be20bd7e server_cookie=52876f58a921d455 in_seq=0 out_seq=0 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/3296117742 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5568d79f8cf0 con 0x5568d7b17490 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5ba5ffb640 10 client.7320 handle_command_reply: tid=0 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5bcfbf1740 1 client.7320 shutdown 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/3296117742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 msgr2=0x5568d7b0d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 0x5568d7b0d870 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009560 tx=0x7f5bbc004730 comp rx=0 tx=0).stop 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/3296117742 shutdown_connections 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5568d7b17490 0x5568d7b10780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b0ce20 0x5568d7b0ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14f70 0x5568d7b0d870 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5ba5ffb640 0 client.7320 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/3296117742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14b90 0x5568d7b33120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/3296117742 >> 172.21.15.44:0/3296117742 conn(0x5568d7ae8070 msgr2=0x5568d7aeba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/3296117742 shutdown_connections 2024-04-02T13:25:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/3296117742 wait complete. 2024-04-02T13:25:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.362+0000 7f5bcfbf1740 20 client.7320 trim_cache size 0 max 16384 2024-04-02T13:25:16.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.380+0000 7f5bcfbf1740 1 Processor -- start 2024-04-02T13:25:16.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.380+0000 7f5bcfbf1740 1 -- start start 2024-04-02T13:25:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bcfbf1740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b0ce20 0x5568d7b42ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bcfbf1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14b90 0x5568d7b42ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bcfbf1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b14f70 0x5568d7b1a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bcfbf1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc81fff30 con 0x5568d7b14b90 2024-04-02T13:25:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bcfbf1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc80c6d70 con 0x5568d7b14f70 2024-04-02T13:25:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bcfbf1740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc80c73c0 con 0x5568d7b0ce20 2024-04-02T13:25:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bc659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14b90 0x5568d7b42ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bc659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14b90 0x5568d7b42ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58164/0 (socket says 172.21.15.44:58164) 2024-04-02T13:25:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bc659c640 1 -- 172.21.15.44:0/364770371 learned_addr learned my addr 172.21.15.44:0/364770371 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bc6d9d640 1 --2- 172.21.15.44:0/364770371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b0ce20 0x5568d7b42ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bc759e640 1 --2- 172.21.15.44:0/364770371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b14f70 0x5568d7b1a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bc659c640 1 -- 172.21.15.44:0/364770371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b0ce20 msgr2=0x5568d7b42ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bc659c640 1 --2- 172.21.15.44:0/364770371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b0ce20 0x5568d7b42ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bc659c640 1 -- 172.21.15.44:0/364770371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b14f70 msgr2=0x5568d7b1a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bc659c640 1 --2- 172.21.15.44:0/364770371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b14f70 0x5568d7b1a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bc659c640 1 -- 172.21.15.44:0/364770371 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568d7aeec80 con 0x5568d7b14b90 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bc6d9d640 1 --2- 172.21.15.44:0/364770371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b0ce20 0x5568d7b42ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bc759e640 1 --2- 172.21.15.44:0/364770371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b14f70 0x5568d7b1a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5bc659c640 1 --2- 172.21.15.44:0/364770371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14b90 0x5568d7b42ff0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc046be0 tx=0x7f5bbc03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d3bca466bcffc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5ba7fff640 1 -- 172.21.15.44:0/364770371 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc056030 con 0x5568d7b14b90 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5ba7fff640 1 -- 172.21.15.44:0/364770371 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bbc058050 con 0x5568d7b14b90 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.381+0000 7f5ba7fff640 1 -- 172.21.15.44:0/364770371 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc034040 con 0x5568d7b14b90 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.382+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/364770371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14b90 msgr2=0x5568d7b42ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.382+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/364770371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14b90 0x5568d7b42ff0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc046be0 tx=0x7f5bbc03d000 comp rx=0 tx=0).stop 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.382+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/364770371 shutdown_connections 2024-04-02T13:25:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.382+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/364770371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b14f70 0x5568d7b1a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.382+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/364770371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b14b90 0x5568d7b42ff0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.382+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/364770371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b0ce20 0x5568d7b42ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.382+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/364770371 >> 172.21.15.44:0/364770371 conn(0x5568d7ae8070 msgr2=0x5568d7b15960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.382+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/364770371 shutdown_connections 2024-04-02T13:25:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.382+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/364770371 wait complete. 2024-04-02T13:25:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.383+0000 7f5bcfbf1740 1 -- start start 2024-04-02T13:25:16.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.383+0000 7f5bcfbf1740 1 Processor -- start 2024-04-02T13:25:16.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.383+0000 7f5bcfbf1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b0ce20 0x5568d7bac340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.383+0000 7f5bcfbf1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b14b90 0x5568d7b0d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.383+0000 7f5bcfbf1740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14f70 0x5568d7b0dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.383+0000 7f5bcfbf1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5568d7b17310 con 0x5568d7b0ce20 2024-04-02T13:25:16.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.383+0000 7f5bcfbf1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5568d7b44600 con 0x5568d7b14b90 2024-04-02T13:25:16.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.383+0000 7f5bcfbf1740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5568d7b17960 con 0x5568d7b14f70 2024-04-02T13:25:16.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.383+0000 7f5bcfbf1740 10 client.0 _ll_register_callbacks cb 0x7f5bbc054070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:16.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.383+0000 7f5bc6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b0ce20 0x5568d7bac340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.383+0000 7f5bc6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b0ce20 0x5568d7bac340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58180/0 (socket says 172.21.15.44:58180) 2024-04-02T13:25:16.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.383+0000 7f5bc6d9d640 1 -- 172.21.15.44:0/371568091 learned_addr learned my addr 172.21.15.44:0/371568091 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:16.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5bc759e640 1 --2- 172.21.15.44:0/371568091 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14f70 0x5568d7b0dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5bc659c640 1 --2- 172.21.15.44:0/371568091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b14b90 0x5568d7b0d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5bc6d9d640 1 -- 172.21.15.44:0/371568091 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14f70 msgr2=0x5568d7b0dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5bc6d9d640 1 --2- 172.21.15.44:0/371568091 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14f70 0x5568d7b0dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5bc6d9d640 1 -- 172.21.15.44:0/371568091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b14b90 msgr2=0x5568d7b0d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5bc6d9d640 1 --2- 172.21.15.44:0/371568091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b14b90 0x5568d7b0d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5bc6d9d640 1 -- 172.21.15.44:0/371568091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568d7bac880 con 0x5568d7b0ce20 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5bc6d9d640 1 --2- 172.21.15.44:0/371568091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b0ce20 0x5568d7bac340 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0055c00 tx=0x7f5bc0060040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45dd85008d1901e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5ba5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/371568091 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0057b60 con 0x5568d7b0ce20 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/371568091 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f5bc0055e10 con 0x5568d7b0ce20 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/371568091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bbc058050 con 0x5568d7b0ce20 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/371568091 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f5bc0061ef0 con 0x5568d7b0ce20 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5bcfbf1740 10 client.7326 fetch_fsmap learned FSMap version 50 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/371568091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x5568d7b0ece0 con 0x5568d7b0ce20 2024-04-02T13:25:16.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.384+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/371568091 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bbc058050 con 0x5568d7b0ce20 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/371568091 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7f5bc007f070 con 0x5568d7b0ce20 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 10 client.7326 fetch_fsmap finished waiting for FSMap version 50 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 10 client.7326 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 20 client.7326 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/371568091 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5568d7bb05f0 0x5568d7b33a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 4 client.7326 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/371568091 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "status"}) -- 0x5568d7a5e5a0 con 0x5568d7bb05f0 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bc659c640 1 --2- 172.21.15.44:0/371568091 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5568d7bb05f0 0x5568d7b33a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5ba5ffb640 10 client.7326 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bc659c640 1 --2- 172.21.15.44:0/371568091 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5568d7bb05f0 0x5568d7b33a50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ddcad2e2e694ba3 server_cookie=fbabb2461bdf702b in_seq=0 out_seq=0 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5ba5ffb640 1 -- 172.21.15.44:0/371568091 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5568d7a5e5a0 con 0x5568d7bb05f0 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5ba5ffb640 10 client.7326 handle_command_reply: tid=0 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 1 client.7326 shutdown 2024-04-02T13:25:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/371568091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b0ce20 msgr2=0x5568d7bac340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/371568091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b0ce20 0x5568d7bac340 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0055c00 tx=0x7f5bc0060040 comp rx=0 tx=0).stop 2024-04-02T13:25:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/371568091 shutdown_connections 2024-04-02T13:25:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/371568091 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5568d7bb05f0 0x5568d7b33a50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/371568091 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5568d7b14f70 0x5568d7b0dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/371568091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5568d7b14b90 0x5568d7b0d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 1 --2- 172.21.15.44:0/371568091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568d7b0ce20 0x5568d7bac340 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/371568091 >> 172.21.15.44:0/371568091 conn(0x5568d7ae8070 msgr2=0x5568d7b15960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5ba5ffb640 0 client.7326 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/371568091 shutdown_connections 2024-04-02T13:25:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 1 -- 172.21.15.44:0/371568091 wait complete. 2024-04-02T13:25:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.385+0000 7f5bcfbf1740 20 client.7326 trim_cache size 0 max 16384 2024-04-02T13:25:16.398 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:25:16.398 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:25:16.398 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:25:16.398 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 6987, 2024-04-02T13:25:16.398 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:25:16.398 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:25:16.398 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:25:16.398 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 4.4499645399999999, 2024-04-02T13:25:16.398 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 50, 2024-04-02T13:25:16.398 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 80, 2024-04-02T13:25:16.398 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 80, 2024-04-02T13:25:16.398 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 27.150881378000001 2024-04-02T13:25:16.398 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:25:16.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.386+0000 7f5bcd00f640 1 -- 172.21.15.44:0/4248673224 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5bb406c8f0 msgr2=0x7f5bb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.386+0000 7f5bcd00f640 1 --2- 172.21.15.44:0/4248673224 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5bb406c8f0 0x7f5bb406edb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc036000 tx=0x7f5bbc008030 comp rx=0 tx=0).stop 2024-04-02T13:25:16.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.386+0000 7f5bcd00f640 1 -- 172.21.15.44:0/4248673224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc8072f60 msgr2=0x7f5bc8158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.386+0000 7f5bcd00f640 1 --2- 172.21.15.44:0/4248673224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc8072f60 0x7f5bc8158480 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000c430 tx=0x7f5bc000c8f0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.386+0000 7f5bcd00f640 1 -- 172.21.15.44:0/4248673224 shutdown_connections 2024-04-02T13:25:16.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.386+0000 7f5bcd00f640 1 --2- 172.21.15.44:0/4248673224 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5bb406c8f0 0x7f5bb406edb0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.386+0000 7f5bcd00f640 1 --2- 172.21.15.44:0/4248673224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bc815e4d0 0x7f5bc80c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.386+0000 7f5bcd00f640 1 --2- 172.21.15.44:0/4248673224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc815e0f0 0x7f5bc80c48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.386+0000 7f5bcd00f640 1 --2- 172.21.15.44:0/4248673224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc8072f60 0x7f5bc8158480 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.386+0000 7f5bcd00f640 1 -- 172.21.15.44:0/4248673224 >> 172.21.15.44:0/4248673224 conn(0x7f5bc80b2980 msgr2=0x7f5bc80d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:16.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.386+0000 7f5bcd00f640 1 -- 172.21.15.44:0/4248673224 shutdown_connections 2024-04-02T13:25:16.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.386+0000 7f5bcd00f640 1 -- 172.21.15.44:0/4248673224 wait complete. 2024-04-02T13:25:16.401 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:25:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.500+0000 7fd0352d6640 1 Processor -- start 2024-04-02T13:25:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.500+0000 7fd0352d6640 1 -- start start 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd0352d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03015e0f0 0x7fd0300d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd0352d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03006dfe0 con 0x7fd03006bbe0 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd0352d6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03006e2c0 con 0x7fd0300d0630 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd0352d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd03006e5a0 con 0x7fd03015e0f0 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd02f7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd03006bbe0 0x7fd0300add60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44638/0 (socket says 172.21.15.44:44638) 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd02f7fe640 1 -- 172.21.15.44:0/2786299782 learned_addr learned my addr 172.21.15.44:0/2786299782 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd02effd640 1 --2- 172.21.15.44:0/2786299782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03015e0f0 0x7fd0300d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd02e7fc640 1 -- 172.21.15.44:0/2786299782 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1771146449 0 0) 0x7fd03006dfe0 con 0x7fd03006bbe0 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd02e7fc640 1 -- 172.21.15.44:0/2786299782 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd01c003520 con 0x7fd03006bbe0 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd02e7fc640 1 -- 172.21.15.44:0/2786299782 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 880199257 0 0) 0x7fd03006e2c0 con 0x7fd0300d0630 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd02e7fc640 1 -- 172.21.15.44:0/2786299782 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd03006dfe0 con 0x7fd0300d0630 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd02effd640 1 -- 172.21.15.44:0/2786299782 >> v1:172.21.15.69:6789/0 conn(0x7fd0300d0630 legacy=0x7fd03006b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd02effd640 1 -- 172.21.15.44:0/2786299782 >> v1:172.21.15.44:6789/0 conn(0x7fd03006bbe0 legacy=0x7fd0300add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.501+0000 7fd02effd640 1 -- 172.21.15.44:0/2786299782 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03006e880 con 0x7fd03015e0f0 2024-04-02T13:25:16.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.502+0000 7fd02effd640 1 --2- 172.21.15.44:0/2786299782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03015e0f0 0x7fd0300d00f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd0200097d0 tx=0x7fd02002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cd4b82d0fbea002 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:16.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.502+0000 7fd02e7fc640 1 -- 172.21.15.44:0/2786299782 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd020034070 con 0x7fd03015e0f0 2024-04-02T13:25:16.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.502+0000 7fd02e7fc640 1 -- 172.21.15.44:0/2786299782 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd020030880 con 0x7fd03015e0f0 2024-04-02T13:25:16.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.502+0000 7fd02e7fc640 1 -- 172.21.15.44:0/2786299782 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd020030c50 con 0x7fd03015e0f0 2024-04-02T13:25:16.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.503+0000 7fd0352d6640 1 -- 172.21.15.44:0/2786299782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03015e0f0 msgr2=0x7fd0300d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.503+0000 7fd0352d6640 1 --2- 172.21.15.44:0/2786299782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03015e0f0 0x7fd0300d00f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd0200097d0 tx=0x7fd02002b9c0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.503+0000 7fd0352d6640 1 -- 172.21.15.44:0/2786299782 shutdown_connections 2024-04-02T13:25:16.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.503+0000 7fd0352d6640 1 --2- 172.21.15.44:0/2786299782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03015e0f0 0x7fd0300d00f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.503+0000 7fd0352d6640 1 -- 172.21.15.44:0/2786299782 >> 172.21.15.44:0/2786299782 conn(0x7fd0300b2980 msgr2=0x7fd0300b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:16.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.503+0000 7fd0352d6640 1 -- 172.21.15.44:0/2786299782 shutdown_connections 2024-04-02T13:25:16.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.503+0000 7fd0352d6640 1 -- 172.21.15.44:0/2786299782 wait complete. 2024-04-02T13:25:16.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.504+0000 7fd0352d6640 1 Processor -- start 2024-04-02T13:25:16.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.504+0000 7fd0352d6640 1 -- start start 2024-04-02T13:25:16.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd0352d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd03006bbe0 0x7fd0301502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd0352d6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0300d0630 0x7fd0300c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd0352d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03015e0f0 0x7fd0300ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd0352d6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd03015e4d0 con 0x7fd03006bbe0 2024-04-02T13:25:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd0352d6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd03006eda0 con 0x7fd0300d0630 2024-04-02T13:25:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd0352d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd03015e650 con 0x7fd03015e0f0 2024-04-02T13:25:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd02effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd03006bbe0 0x7fd0301502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd02effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd03006bbe0 0x7fd0301502b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58182/0 (socket says 172.21.15.44:58182) 2024-04-02T13:25:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd02effd640 1 -- 172.21.15.44:0/2096865118 learned_addr learned my addr 172.21.15.44:0/2096865118 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd027fff640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0300d0630 0x7fd0300c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd02f7fe640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03015e0f0 0x7fd0300ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd02effd640 1 -- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03015e0f0 msgr2=0x7fd0300ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd02effd640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03015e0f0 0x7fd0300ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd02effd640 1 -- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0300d0630 msgr2=0x7fd0300c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.505+0000 7fd02effd640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0300d0630 0x7fd0300c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.506+0000 7fd02effd640 1 -- 172.21.15.44:0/2096865118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0300ce450 con 0x7fd03006bbe0 2024-04-02T13:25:16.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.506+0000 7fd02f7fe640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03015e0f0 0x7fd0300ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:16.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.506+0000 7fd02effd640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd03006bbe0 0x7fd0301502b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fd0200098f0 tx=0x7fd0200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:16.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.506+0000 7fd0277fe640 1 -- 172.21.15.44:0/2096865118 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd020034040 con 0x7fd03006bbe0 2024-04-02T13:25:16.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.506+0000 7fd0277fe640 1 -- 172.21.15.44:0/2096865118 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd02003ada0 con 0x7fd03006bbe0 2024-04-02T13:25:16.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.506+0000 7fd0352d6640 1 -- 172.21.15.44:0/2096865118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0300cbf90 con 0x7fd03006bbe0 2024-04-02T13:25:16.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.506+0000 7fd0277fe640 1 -- 172.21.15.44:0/2096865118 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd020006e10 con 0x7fd03006bbe0 2024-04-02T13:25:16.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.506+0000 7fd0352d6640 1 -- 172.21.15.44:0/2096865118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0300cdfc0 con 0x7fd03006bbe0 2024-04-02T13:25:16.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.507+0000 7fd0352d6640 1 -- 172.21.15.44:0/2096865118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0300684f0 con 0x7fd03006bbe0 2024-04-02T13:25:16.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.507+0000 7fd0277fe640 1 -- 172.21.15.44:0/2096865118 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd02004b060 con 0x7fd03006bbe0 2024-04-02T13:25:16.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.508+0000 7fd0277fe640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd01006c920 0x7fd01006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:16.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.508+0000 7fd0277fe640 1 -- 172.21.15.44:0/2096865118 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fd0200add50 con 0x7fd03006bbe0 2024-04-02T13:25:16.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.508+0000 7fd027fff640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd01006c920 0x7fd01006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:16.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.509+0000 7fd027fff640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd01006c920 0x7fd01006ede0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd018007fd0 tx=0x7fd018009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.512+0000 7fd0277fe640 1 -- 172.21.15.44:0/2096865118 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd0200b1030 con 0x7fd03006bbe0 2024-04-02T13:25:16.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.738+0000 7fd0352d6640 1 -- 172.21.15.44:0/2096865118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd0300b6cc0 con 0x7fd03006bbe0 2024-04-02T13:25:16.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.739+0000 7fd0277fe640 1 -- 172.21.15.44:0/2096865118 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) ==== 53+0+73 (secure 0 0 0) 0x7fd02003f070 con 0x7fd03006bbe0 2024-04-02T13:25:16.742 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:25:16.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.741+0000 7fd0352d6640 1 -- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd01006c920 msgr2=0x7fd01006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.741+0000 7fd0352d6640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd01006c920 0x7fd01006ede0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd018007fd0 tx=0x7fd018009040 comp rx=0 tx=0).stop 2024-04-02T13:25:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.741+0000 7fd0352d6640 1 -- 172.21.15.44:0/2096865118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd03006bbe0 msgr2=0x7fd0301502b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.741+0000 7fd0352d6640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd03006bbe0 0x7fd0301502b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fd0200098f0 tx=0x7fd0200046c0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.741+0000 7fd0352d6640 1 -- 172.21.15.44:0/2096865118 shutdown_connections 2024-04-02T13:25:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.741+0000 7fd0352d6640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd01006c920 0x7fd01006ede0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.741+0000 7fd0352d6640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd03015e0f0 0x7fd0300ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.741+0000 7fd0352d6640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0300d0630 0x7fd0300c9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.741+0000 7fd0352d6640 1 --2- 172.21.15.44:0/2096865118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd03006bbe0 0x7fd0301502b0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.741+0000 7fd0352d6640 1 -- 172.21.15.44:0/2096865118 >> 172.21.15.44:0/2096865118 conn(0x7fd0300b2980 msgr2=0x7fd0300b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.741+0000 7fd0352d6640 1 -- 172.21.15.44:0/2096865118 shutdown_connections 2024-04-02T13:25:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16.741+0000 7fd0352d6640 1 -- 172.21.15.44:0/2096865118 wait complete. 2024-04-02T13:25:16.755 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:25:16.755 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:25:16.755 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:25:16.755 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:25:16.755 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:16.755 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:25:16.755 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:25:16.755 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:25:16.755 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:25:16.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:16.756 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:25:16.770 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:25:16.771 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:25:16.771 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:25:16.771 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:16.771 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:25:16.771 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:16.771 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:25:16.771 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:25:16.771 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:25:16.771 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:25:16.771 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:25:16.771 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:25:16.771 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42462sec preferred_lft 42462sec 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:25:16.772 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:16.773 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:25:16.773 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:25:16.773 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:25:16.773 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:16.773 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:25:16.773 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:25:16.773 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:25:16.773 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:25:16.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:16.773 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:25:16.837 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:25:16.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:16.838 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:25:16.893 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:25:16.893 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:25:16.893 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:25:16.893 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:25:16.893 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:25:16.893 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:25:16.894 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:25:16.894 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:25:16.894 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:25:16.894 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:25:16.894 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:25:16.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:25:17.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:25:17.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:17.029 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:25:17.043 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:25:17.043 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:25:17.043 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:25:17.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:17.044 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:25:17.112 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:25:17.112 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:25:17.112 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:17.166 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:25:17.166 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:25:17.167 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:17.220 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:25:17.291 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:25:17.349 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:25:17.349 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:25:17.349 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:25:17.349 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:25:17.349 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:25:17.349 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:25:17.349 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:25:17.349 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:25:17.350 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:25:17.350 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:25:17.350 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:25:17.351 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:25:17.352 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:25:17.352 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:25:17.352 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:25:17.352 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:25:17.352 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:25:17.352 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:25:17.352 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:25:17.352 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:25:17.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:17.352 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:25:17.406 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:25:17.406 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:25:17.450 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:25:17.476 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:25:17.533 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:25:17.533 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:25:17.533 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:25:17.533 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:25:17.533 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:25:17.533 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:25:17.533 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:25:17.533 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:25:17.534 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:25:17.534 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:25:17.534 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:25:17.534 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:25:17.534 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:25:17.534 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:25:17.534 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:25:17.534 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:25:17.534 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:25:17.534 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:25:17.534 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:25:17.534 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:25:17.535 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:25:17.535 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:25:17.535 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:25:17.535 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:25:17.535 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:25:17.535 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:25:17.535 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:25:17.535 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:25:17.535 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:25:17.535 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:25:17.535 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:25:17.536 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:25:17.536 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:25:17.536 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:25:17.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:17.536 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:25:17.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:25:17.532+0000 7f8652445480 -1 init, newargv = 0x5631c9783040 newargc=15 2024-04-02T13:25:17.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[65412]: starting ceph client 2024-04-02T13:25:17.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[65412]: starting fuse 2024-04-02T13:25:17.551 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:25:17.551 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:25:17.552 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:25:17.553 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:25:17.553 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:25:17.553 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:25:17.553 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:25:17.553 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:25:17.553 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:25:17.553 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:25:17.553 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:25:17.553 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:25:17.553 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:25:17.553 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:25:17.553 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:25:17.672 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.65412.asok 2024-04-02T13:25:17.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:25:17.683 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.65412.asok 2024-04-02T13:25:17.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:17.684 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.65412.asok status 2024-04-02T13:25:17.821 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:25:17.821 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:25:17.821 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:25:17.822 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:25:17.822 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:25:17.822 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:25:17.822 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:25:17.822 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "65412", 2024-04-02T13:25:17.822 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:25:17.822 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:25:17.822 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:25:17.822 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:25:17.822 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 7341, 2024-04-02T13:25:17.822 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:25:17.822 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 7341 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 47930660 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 47930660 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:25:17.823 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.7341 192.168.0.1:0/47930660", 2024-04-02T13:25:17.824 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/47930660", 2024-04-02T13:25:17.824 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:25:17.824 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 50, 2024-04-02T13:25:17.824 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 80, 2024-04-02T13:25:17.824 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:25:17.824 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:25:17.824 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:25:17.824 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:25:17.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:17.830 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:25:17.830 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:17.845 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:25:17.845 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:17.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:17.845 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:17.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:17.911 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls subdir/) 2024-04-02T13:25:17.968 INFO:teuthology.orchestra.run.smithi044.stdout:file_undamaged 2024-04-02T13:25:17.969 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:25:17.969 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:25:17.969 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:25:17.969 DEBUG:teuthology.orchestra.run.smithi044:> import json 2024-04-02T13:25:17.969 DEBUG:teuthology.orchestra.run.smithi044:> import sys 2024-04-02T13:25:17.969 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:25:17.969 DEBUG:teuthology.orchestra.run.smithi044:> try: 2024-04-02T13:25:17.969 DEBUG:teuthology.orchestra.run.smithi044:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-04-02T13:25:17.969 DEBUG:teuthology.orchestra.run.smithi044:> except OSError as e: 2024-04-02T13:25:17.970 DEBUG:teuthology.orchestra.run.smithi044:> sys.exit(e.errno) 2024-04-02T13:25:17.970 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:25:17.970 DEBUG:teuthology.orchestra.run.smithi044:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-04-02T13:25:17.970 DEBUG:teuthology.orchestra.run.smithi044:> print(json.dumps( 2024-04-02T13:25:17.970 DEBUG:teuthology.orchestra.run.smithi044:> dict([(a, getattr(s, a)) for a in attrs]), 2024-04-02T13:25:17.970 DEBUG:teuthology.orchestra.run.smithi044:> indent=2)) 2024-04-02T13:25:17.970 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:25:18.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18 stdin-killer INFO: command exited with status 2: exiting normally with same code! 2024-04-02T13:25:18.084 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-02T13:25:18.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:18.085 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-04-02T13:25:18.141 INFO:teuthology.orchestra.run.smithi044.stdout:2 2024-04-02T13:25:18.141 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:25:18.141 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:25:18.196 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:25:18.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.335+0000 7fc7724c7640 1 Processor -- start 2024-04-02T13:25:18.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.335+0000 7fc7724c7640 1 -- start start 2024-04-02T13:25:18.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.336+0000 7fc7724c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc76c0b3d70 0x7fc76c0b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.336+0000 7fc7724c7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc76c0b0c10 0x7fc76c165e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.336+0000 7fc7724c7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c0b1640 con 0x7fc76c15c1c0 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.336+0000 7fc7724c7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc76c0aef20 con 0x7fc76c0b0c10 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.336+0000 7fc7724c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc76c0af200 con 0x7fc76c0b3d70 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.336+0000 7fc76b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc76c0b3d70 0x7fc76c0b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.336+0000 7fc770a64640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc76c0b0c10 0x7fc76c165e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.336+0000 7fc76b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc76c0b3d70 0x7fc76c0b06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:43254/0 (socket says 172.21.15.44:43254) 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.336+0000 7fc76b7fe640 1 -- 172.21.15.44:0/3467566000 learned_addr learned my addr 172.21.15.44:0/3467566000 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.336+0000 7fc76bfff640 1 --1- 172.21.15.44:0/3467566000 >> v1:172.21.15.69:6789/0 conn(0x7fc76c15c1c0 0x7fc76c0affc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:43516/0 (socket says 172.21.15.44:43516) 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.337+0000 7fc76b7fe640 1 -- 172.21.15.44:0/3467566000 >> v2:172.21.15.69:3300/0 conn(0x7fc76c0b0c10 msgr2=0x7fc76c165e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.337+0000 7fc76b7fe640 1 --2- 172.21.15.44:0/3467566000 >> v2:172.21.15.69:3300/0 conn(0x7fc76c0b0c10 0x7fc76c165e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.337+0000 7fc76b7fe640 1 -- 172.21.15.44:0/3467566000 >> v1:172.21.15.69:6789/0 conn(0x7fc76c15c1c0 legacy=0x7fc76c0affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.337+0000 7fc76b7fe640 1 -- 172.21.15.44:0/3467566000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c0af4e0 con 0x7fc76c0b3d70 2024-04-02T13:25:18.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.337+0000 7fc770a64640 1 --2- 172.21.15.44:0/3467566000 >> v2:172.21.15.69:3300/0 conn(0x7fc76c0b0c10 0x7fc76c165e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:25:18.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.337+0000 7fc76b7fe640 1 --2- 172.21.15.44:0/3467566000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc76c0b3d70 0x7fc76c0b06d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009870 tx=0x7fc75c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3d32c95cadf0f21 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:18.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.337+0000 7fc76affd640 1 -- 172.21.15.44:0/3467566000 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c034070 con 0x7fc76c0b3d70 2024-04-02T13:25:18.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.338+0000 7fc76affd640 1 -- 172.21.15.44:0/3467566000 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc75c030840 con 0x7fc76c0b3d70 2024-04-02T13:25:18.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.338+0000 7fc76affd640 1 -- 172.21.15.44:0/3467566000 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c030c10 con 0x7fc76c0b3d70 2024-04-02T13:25:18.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.338+0000 7fc7724c7640 1 -- 172.21.15.44:0/3467566000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc76c0b3d70 msgr2=0x7fc76c0b06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.338+0000 7fc7724c7640 1 --2- 172.21.15.44:0/3467566000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc76c0b3d70 0x7fc76c0b06d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009870 tx=0x7fc75c02b980 comp rx=0 tx=0).stop 2024-04-02T13:25:18.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.338+0000 7fc7724c7640 1 -- 172.21.15.44:0/3467566000 shutdown_connections 2024-04-02T13:25:18.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.338+0000 7fc7724c7640 1 --2- 172.21.15.44:0/3467566000 >> v2:172.21.15.69:3300/0 conn(0x7fc76c0b0c10 0x7fc76c165e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.338+0000 7fc7724c7640 1 --2- 172.21.15.44:0/3467566000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc76c0b3d70 0x7fc76c0b06d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.338+0000 7fc7724c7640 1 -- 172.21.15.44:0/3467566000 >> 172.21.15.44:0/3467566000 conn(0x7fc76c09aa20 msgr2=0x7fc76c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:18.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.338+0000 7fc7724c7640 1 -- 172.21.15.44:0/3467566000 shutdown_connections 2024-04-02T13:25:18.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.338+0000 7fc7724c7640 1 -- 172.21.15.44:0/3467566000 wait complete. 2024-04-02T13:25:18.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.339+0000 7fc7724c7640 1 Processor -- start 2024-04-02T13:25:18.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.339+0000 7fc7724c7640 1 -- start start 2024-04-02T13:25:18.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.340+0000 7fc7724c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc76c0b0c10 0x7fc76c0c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.340+0000 7fc7724c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc76c0b3d70 0x7fc76c0c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.340+0000 7fc7724c7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc76c15c1c0 0x7fc76c14fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.340+0000 7fc7724c7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c0b1080 con 0x7fc76c15c1c0 2024-04-02T13:25:18.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.340+0000 7fc7724c7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c0afaa0 con 0x7fc76c0b3d70 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.340+0000 7fc7724c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc76c0cc990 con 0x7fc76c0b0c10 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.340+0000 7fc770a64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc76c15c1c0 0x7fc76c14fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.340+0000 7fc770a64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc76c15c1c0 0x7fc76c14fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58198/0 (socket says 172.21.15.44:58198) 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.340+0000 7fc770a64640 1 -- 172.21.15.44:0/2424317176 learned_addr learned my addr 172.21.15.44:0/2424317176 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.340+0000 7fc76bfff640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc76c0b0c10 0x7fc76c0c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.340+0000 7fc76b7fe640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc76c0b3d70 0x7fc76c0c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.341+0000 7fc770a64640 1 -- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc76c0b0c10 msgr2=0x7fc76c0c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.341+0000 7fc770a64640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc76c0b0c10 0x7fc76c0c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.341+0000 7fc770a64640 1 -- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc76c0b3d70 msgr2=0x7fc76c0c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.341+0000 7fc770a64640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc76c0b3d70 0x7fc76c0c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.341+0000 7fc770a64640 1 -- 172.21.15.44:0/2424317176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c06d620 con 0x7fc76c15c1c0 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.341+0000 7fc770a64640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc76c15c1c0 0x7fc76c14fd90 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fc76400efc0 tx=0x7fc76400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.341+0000 7fc768ff9640 1 -- 172.21.15.44:0/2424317176 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc764018030 con 0x7fc76c15c1c0 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.341+0000 7fc768ff9640 1 -- 172.21.15.44:0/2424317176 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc76400cdf0 con 0x7fc76c15c1c0 2024-04-02T13:25:18.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.341+0000 7fc7724c7640 1 -- 172.21.15.44:0/2424317176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc76c06d900 con 0x7fc76c15c1c0 2024-04-02T13:25:18.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.341+0000 7fc768ff9640 1 -- 172.21.15.44:0/2424317176 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc764003bb0 con 0x7fc76c15c1c0 2024-04-02T13:25:18.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.341+0000 7fc7724c7640 1 -- 172.21.15.44:0/2424317176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76c06dec0 con 0x7fc76c15c1c0 2024-04-02T13:25:18.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.342+0000 7fc7724c7640 1 -- 172.21.15.44:0/2424317176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc76c0684f0 con 0x7fc76c15c1c0 2024-04-02T13:25:18.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.342+0000 7fc768ff9640 1 -- 172.21.15.44:0/2424317176 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc76401b070 con 0x7fc76c15c1c0 2024-04-02T13:25:18.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.343+0000 7fc768ff9640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc75806c8f0 0x7fc75806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.343+0000 7fc768ff9640 1 -- 172.21.15.44:0/2424317176 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fc76408c640 con 0x7fc76c15c1c0 2024-04-02T13:25:18.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.343+0000 7fc76bfff640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc75806c8f0 0x7fc75806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.344+0000 7fc76bfff640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc75806c8f0 0x7fc75806edb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc760007fd0 tx=0x7fc760007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:18.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.347+0000 7fc768ff9640 1 -- 172.21.15.44:0/2424317176 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc764010040 con 0x7fc76c15c1c0 2024-04-02T13:25:18.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.535+0000 7fc7724c7640 1 -- 172.21.15.44:0/2424317176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc76c0b07e0 con 0x7fc76c15c1c0 2024-04-02T13:25:18.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.536+0000 7fc768ff9640 1 -- 172.21.15.44:0/2424317176 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+17024 (secure 0 0 0) 0x7fc76405a1b0 con 0x7fc76c15c1c0 2024-04-02T13:25:18.540 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:18.540 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":80,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:25:10.921159+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:24:52.202720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-02T13:24:52.916529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6837/698185867":"2024-04-03T13:25:10.921135+0000","172.21.15.69:6835/698185867":"2024-04-03T13:25:10.921135+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:25:18.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.538+0000 7fc7724c7640 1 -- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc75806c8f0 msgr2=0x7fc75806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.538+0000 7fc7724c7640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc75806c8f0 0x7fc75806edb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc760007fd0 tx=0x7fc760007e80 comp rx=0 tx=0).stop 2024-04-02T13:25:18.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.538+0000 7fc7724c7640 1 -- 172.21.15.44:0/2424317176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc76c15c1c0 msgr2=0x7fc76c14fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.538+0000 7fc7724c7640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc76c15c1c0 0x7fc76c14fd90 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fc76400efc0 tx=0x7fc76400c490 comp rx=0 tx=0).stop 2024-04-02T13:25:18.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.538+0000 7fc7724c7640 1 -- 172.21.15.44:0/2424317176 shutdown_connections 2024-04-02T13:25:18.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.538+0000 7fc7724c7640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc75806c8f0 0x7fc75806edb0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.538+0000 7fc7724c7640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc76c15c1c0 0x7fc76c14fd90 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.538+0000 7fc7724c7640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc76c0b3d70 0x7fc76c0c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.538+0000 7fc7724c7640 1 --2- 172.21.15.44:0/2424317176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc76c0b0c10 0x7fc76c0c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.538+0000 7fc7724c7640 1 -- 172.21.15.44:0/2424317176 >> 172.21.15.44:0/2424317176 conn(0x7fc76c09aa20 msgr2=0x7fc76c0c7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:18.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.538+0000 7fc7724c7640 1 -- 172.21.15.44:0/2424317176 shutdown_connections 2024-04-02T13:25:18.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.538+0000 7fc7724c7640 1 -- 172.21.15.44:0/2424317176 wait complete. 2024-04-02T13:25:18.552 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:24:52.202720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:24:52.916529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:25:18.552 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:25:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.656+0000 7f96c42d6640 1 Processor -- start 2024-04-02T13:25:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.656+0000 7f96c42d6640 1 -- start start 2024-04-02T13:25:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.656+0000 7f96c42d6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f96bc0d07b0 0x7f96bc0ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.656+0000 7f96c42d6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f96bc0cf400 0x7f96bc15c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.656+0000 7f96c42d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96bc160b10 con 0x7f96bc1565d0 2024-04-02T13:25:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.657+0000 7f96c42d6640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f96bc0cfad0 con 0x7f96bc0d07b0 2024-04-02T13:25:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.657+0000 7f96c42d6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f96bc0cf870 con 0x7f96bc0cf400 2024-04-02T13:25:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.657+0000 7f96c1871640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f96bc0d07b0 0x7f96bc0ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.657+0000 7f96c2072640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f96bc1565d0 0x7f96bc0d00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44640/0 (socket says 172.21.15.44:44640) 2024-04-02T13:25:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.657+0000 7f96c2072640 1 -- 172.21.15.44:0/3932785611 learned_addr learned my addr 172.21.15.44:0/3932785611 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.657+0000 7f96c2873640 1 --2- 172.21.15.44:0/3932785611 >> v2:172.21.15.69:3300/0 conn(0x7f96bc0cf400 0x7f96bc15c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.657+0000 7f96c1871640 1 -- 172.21.15.44:0/3932785611 >> v2:172.21.15.69:3300/0 conn(0x7f96bc0cf400 msgr2=0x7f96bc15c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.657+0000 7f96c1871640 1 --2- 172.21.15.44:0/3932785611 >> v2:172.21.15.69:3300/0 conn(0x7f96bc0cf400 0x7f96bc15c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.657+0000 7f96c1871640 1 -- 172.21.15.44:0/3932785611 >> v1:172.21.15.44:6789/0 conn(0x7f96bc1565d0 legacy=0x7f96bc0d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.657+0000 7f96c1871640 1 -- 172.21.15.44:0/3932785611 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96bc15fd50 con 0x7f96bc0d07b0 2024-04-02T13:25:18.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.657+0000 7f96c1070640 1 -- 172.21.15.44:0/3932785611 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2664878949 0 0) 0x7f96bc160b10 con 0x7f96bc1565d0 2024-04-02T13:25:18.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.658+0000 7f96c1871640 1 --2- 172.21.15.44:0/3932785611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96bc0d07b0 0x7f96bc0ceec0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f96bc0cec60 tx=0x7f96ac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d978445769e792 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.658+0000 7f96c1070640 1 -- 172.21.15.44:0/3932785611 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac0315a0 con 0x7f96bc0d07b0 2024-04-02T13:25:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.658+0000 7f96c1070640 1 -- 172.21.15.44:0/3932785611 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96ac037070 con 0x7f96bc0d07b0 2024-04-02T13:25:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.658+0000 7f96c1070640 1 -- 172.21.15.44:0/3932785611 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac031960 con 0x7f96bc0d07b0 2024-04-02T13:25:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.658+0000 7f96c42d6640 1 -- 172.21.15.44:0/3932785611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96bc0d07b0 msgr2=0x7f96bc0ceec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.658+0000 7f96c42d6640 1 --2- 172.21.15.44:0/3932785611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96bc0d07b0 0x7f96bc0ceec0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f96bc0cec60 tx=0x7f96ac02b840 comp rx=0 tx=0).stop 2024-04-02T13:25:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.658+0000 7f96c42d6640 1 -- 172.21.15.44:0/3932785611 shutdown_connections 2024-04-02T13:25:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.658+0000 7f96c42d6640 1 --2- 172.21.15.44:0/3932785611 >> v2:172.21.15.69:3300/0 conn(0x7f96bc0cf400 0x7f96bc15c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.658+0000 7f96c42d6640 1 --2- 172.21.15.44:0/3932785611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96bc0d07b0 0x7f96bc0ceec0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.658+0000 7f96c42d6640 1 -- 172.21.15.44:0/3932785611 >> 172.21.15.44:0/3932785611 conn(0x7f96bc06b920 msgr2=0x7f96bc06bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.658+0000 7f96c42d6640 1 -- 172.21.15.44:0/3932785611 shutdown_connections 2024-04-02T13:25:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.659+0000 7f96c42d6640 1 -- 172.21.15.44:0/3932785611 wait complete. 2024-04-02T13:25:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.660+0000 7f96c42d6640 1 Processor -- start 2024-04-02T13:25:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.660+0000 7f96c42d6640 1 -- start start 2024-04-02T13:25:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.660+0000 7f96c42d6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc0cf400 0x7f96bc1e6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.660+0000 7f96c42d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96bc0d07b0 0x7f96bc14d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.660+0000 7f96c42d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96bc1565d0 0x7f96bc09b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.660+0000 7f96c42d6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1605f0 con 0x7f96bc0d07b0 2024-04-02T13:25:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.660+0000 7f96c42d6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc160310 con 0x7f96bc0cf400 2024-04-02T13:25:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.660+0000 7f96c42d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f96bc1569b0 con 0x7f96bc1565d0 2024-04-02T13:25:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.660+0000 7f96c1871640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96bc0d07b0 0x7f96bc14d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.660+0000 7f96c1871640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96bc0d07b0 0x7f96bc14d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58224/0 (socket says 172.21.15.44:58224) 2024-04-02T13:25:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.660+0000 7f96c1871640 1 -- 172.21.15.44:0/3128517850 learned_addr learned my addr 172.21.15.44:0/3128517850 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96c2072640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc0cf400 0x7f96bc1e6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96c2873640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96bc1565d0 0x7f96bc09b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96c1871640 1 -- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96bc1565d0 msgr2=0x7f96bc09b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96c1871640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96bc1565d0 0x7f96bc09b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96c1871640 1 -- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc0cf400 msgr2=0x7f96bc1e6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96c1871640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc0cf400 0x7f96bc1e6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96c1871640 1 -- 172.21.15.44:0/3128517850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96bc09b8c0 con 0x7f96bc0d07b0 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96c2873640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96bc1565d0 0x7f96bc09b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96c1871640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96bc0d07b0 0x7f96bc14d540 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009890 tx=0x7f96ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96b2ffd640 1 -- 172.21.15.44:0/3128517850 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac032630 con 0x7f96bc0d07b0 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96b2ffd640 1 -- 172.21.15.44:0/3128517850 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96ac0327d0 con 0x7f96bc0d07b0 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96c42d6640 1 -- 172.21.15.44:0/3128517850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96bc09d130 con 0x7f96bc0d07b0 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96b2ffd640 1 -- 172.21.15.44:0/3128517850 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac032bc0 con 0x7f96bc0d07b0 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.661+0000 7f96c42d6640 1 -- 172.21.15.44:0/3128517850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96bc09d6f0 con 0x7f96bc0d07b0 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.662+0000 7f96c42d6640 1 -- 172.21.15.44:0/3128517850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96bc0684f0 con 0x7f96bc0d07b0 2024-04-02T13:25:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.662+0000 7f96b2ffd640 1 -- 172.21.15.44:0/3128517850 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f96ac04c020 con 0x7f96bc0d07b0 2024-04-02T13:25:18.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.663+0000 7f96b2ffd640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f96a006c8f0 0x7f96a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.663+0000 7f96b2ffd640 1 -- 172.21.15.44:0/3128517850 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f96ac02f020 con 0x7f96bc0d07b0 2024-04-02T13:25:18.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.663+0000 7f96c2072640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f96a006c8f0 0x7f96a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.664+0000 7f96c2072640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f96a006c8f0 0x7f96a006edb0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f96b8000f60 tx=0x7f96b8007d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:18.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.667+0000 7f96b2ffd640 1 -- 172.21.15.44:0/3128517850 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f96bc09b8c0 con 0x7f96bc0d07b0 2024-04-02T13:25:18.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.854+0000 7f96c42d6640 1 -- 172.21.15.44:0/3128517850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f96bc09baf0 con 0x7f96bc0d07b0 2024-04-02T13:25:18.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.855+0000 7f96b2ffd640 1 -- 172.21.15.44:0/3128517850 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) ==== 93+0+10 (secure 0 0 0) 0x7f96ac040070 con 0x7f96bc0d07b0 2024-04-02T13:25:18.858 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:25:18.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.857+0000 7f96c42d6640 1 -- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f96a006c8f0 msgr2=0x7f96a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.857+0000 7f96c42d6640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f96a006c8f0 0x7f96a006edb0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f96b8000f60 tx=0x7f96b8007d50 comp rx=0 tx=0).stop 2024-04-02T13:25:18.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.857+0000 7f96c42d6640 1 -- 172.21.15.44:0/3128517850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96bc0d07b0 msgr2=0x7f96bc14d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.857+0000 7f96c42d6640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96bc0d07b0 0x7f96bc14d540 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009890 tx=0x7f96ac0046c0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.857+0000 7f96c42d6640 1 -- 172.21.15.44:0/3128517850 shutdown_connections 2024-04-02T13:25:18.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.857+0000 7f96c42d6640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f96a006c8f0 0x7f96a006edb0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.857+0000 7f96c42d6640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96bc1565d0 0x7f96bc09b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.857+0000 7f96c42d6640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96bc0d07b0 0x7f96bc14d540 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.857+0000 7f96c42d6640 1 --2- 172.21.15.44:0/3128517850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc0cf400 0x7f96bc1e6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.857+0000 7f96c42d6640 1 -- 172.21.15.44:0/3128517850 >> 172.21.15.44:0/3128517850 conn(0x7f96bc06b920 msgr2=0x7f96bc06bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:18.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.857+0000 7f96c42d6640 1 -- 172.21.15.44:0/3128517850 shutdown_connections 2024-04-02T13:25:18.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.857+0000 7f96c42d6640 1 -- 172.21.15.44:0/3128517850 wait complete. 2024-04-02T13:25:18.870 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:25:18.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.973+0000 7ff036d32640 1 Processor -- start 2024-04-02T13:25:18.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.973+0000 7ff036d32640 1 -- start start 2024-04-02T13:25:18.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.974+0000 7ff036d32640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff03015e1d0 0x7ff0300d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.974+0000 7ff036d32640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0300cec60 0x7ff0300d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.974+0000 7ff036d32640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff03015e5b0 0x7ff0300d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.974+0000 7ff036d32640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff0300cfe00 con 0x7ff0300cec60 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.974+0000 7ff036d32640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff0300d1640 con 0x7ff03015e5b0 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.974+0000 7ff036d32640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0300cf1f0 con 0x7ff03015e1d0 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.974+0000 7ff02ffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0300cec60 0x7ff0300d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.974+0000 7ff02ffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0300cec60 0x7ff0300d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58240/0 (socket says 172.21.15.44:58240) 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.974+0000 7ff02ffff640 1 -- 172.21.15.44:0/3515040989 learned_addr learned my addr 172.21.15.44:0/3515040989 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.975+0000 7ff0352cf640 1 --2- 172.21.15.44:0/3515040989 >> v2:172.21.15.69:3300/0 conn(0x7ff03015e5b0 0x7ff0300d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.975+0000 7ff034ace640 1 --2- 172.21.15.44:0/3515040989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff03015e1d0 0x7ff0300d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.975+0000 7ff02ffff640 1 -- 172.21.15.44:0/3515040989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff03015e1d0 msgr2=0x7ff0300d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.975+0000 7ff02ffff640 1 --2- 172.21.15.44:0/3515040989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff03015e1d0 0x7ff0300d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.975+0000 7ff02ffff640 1 -- 172.21.15.44:0/3515040989 >> v2:172.21.15.69:3300/0 conn(0x7ff03015e5b0 msgr2=0x7ff0300d2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.975+0000 7ff02ffff640 1 --2- 172.21.15.44:0/3515040989 >> v2:172.21.15.69:3300/0 conn(0x7ff03015e5b0 0x7ff0300d2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.975+0000 7ff02ffff640 1 -- 172.21.15.44:0/3515040989 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0300cf4d0 con 0x7ff0300cec60 2024-04-02T13:25:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.975+0000 7ff0352cf640 1 --2- 172.21.15.44:0/3515040989 >> v2:172.21.15.69:3300/0 conn(0x7ff03015e5b0 0x7ff0300d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.975+0000 7ff02ffff640 1 --2- 172.21.15.44:0/3515040989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0300cec60 0x7ff0300d26c0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7ff0300cb160 tx=0x7ff02402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec018000fd14df8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.976+0000 7ff02f7fe640 1 -- 172.21.15.44:0/3515040989 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02402e070 con 0x7ff0300cec60 2024-04-02T13:25:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.976+0000 7ff02f7fe640 1 -- 172.21.15.44:0/3515040989 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff024037070 con 0x7ff0300cec60 2024-04-02T13:25:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.976+0000 7ff02f7fe640 1 -- 172.21.15.44:0/3515040989 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024031a00 con 0x7ff0300cec60 2024-04-02T13:25:18.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.976+0000 7ff036d32640 1 -- 172.21.15.44:0/3515040989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0300cec60 msgr2=0x7ff0300d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.976+0000 7ff036d32640 1 --2- 172.21.15.44:0/3515040989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0300cec60 0x7ff0300d26c0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7ff0300cb160 tx=0x7ff02402b880 comp rx=0 tx=0).stop 2024-04-02T13:25:18.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.976+0000 7ff036d32640 1 -- 172.21.15.44:0/3515040989 shutdown_connections 2024-04-02T13:25:18.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.976+0000 7ff036d32640 1 --2- 172.21.15.44:0/3515040989 >> v2:172.21.15.69:3300/0 conn(0x7ff03015e5b0 0x7ff0300d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.976+0000 7ff036d32640 1 --2- 172.21.15.44:0/3515040989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0300cec60 0x7ff0300d26c0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.976+0000 7ff036d32640 1 --2- 172.21.15.44:0/3515040989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff03015e1d0 0x7ff0300d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.976+0000 7ff036d32640 1 -- 172.21.15.44:0/3515040989 >> 172.21.15.44:0/3515040989 conn(0x7ff03009d3b0 msgr2=0x7ff0300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:18.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.976+0000 7ff036d32640 1 -- 172.21.15.44:0/3515040989 shutdown_connections 2024-04-02T13:25:18.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.976+0000 7ff036d32640 1 -- 172.21.15.44:0/3515040989 wait complete. 2024-04-02T13:25:18.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.977+0000 7ff036d32640 1 Processor -- start 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.977+0000 7ff036d32640 1 -- start start 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.978+0000 7ff036d32640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0300cec60 0x7ff030153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.978+0000 7ff036d32640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff03015e1d0 0x7ff0300cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.978+0000 7ff036d32640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff03015e5b0 0x7ff0300cda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.978+0000 7ff036d32640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff0300b4ce0 con 0x7ff0300cec60 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.978+0000 7ff036d32640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0300cfa90 con 0x7ff03015e1d0 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.978+0000 7ff036d32640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0300b4e60 con 0x7ff03015e5b0 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.978+0000 7ff034ace640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0300cec60 0x7ff030153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.978+0000 7ff034ace640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0300cec60 0x7ff030153ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58242/0 (socket says 172.21.15.44:58242) 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.978+0000 7ff034ace640 1 -- 172.21.15.44:0/4228395635 learned_addr learned my addr 172.21.15.44:0/4228395635 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.978+0000 7ff02ffff640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff03015e1d0 0x7ff0300cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.978+0000 7ff0352cf640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff03015e5b0 0x7ff0300cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.979+0000 7ff034ace640 1 -- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff03015e5b0 msgr2=0x7ff0300cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.979+0000 7ff034ace640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff03015e5b0 0x7ff0300cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.979+0000 7ff034ace640 1 -- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff03015e1d0 msgr2=0x7ff0300cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:18.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.979+0000 7ff034ace640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff03015e1d0 0x7ff0300cd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:18.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.979+0000 7ff034ace640 1 -- 172.21.15.44:0/4228395635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0301f8210 con 0x7ff0300cec60 2024-04-02T13:25:18.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.979+0000 7ff0352cf640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff03015e5b0 0x7ff0300cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:18.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.979+0000 7ff034ace640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0300cec60 0x7ff030153ca0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7ff02800ce10 tx=0x7ff02800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:18.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.979+0000 7ff02d7fa640 1 -- 172.21.15.44:0/4228395635 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff028003b90 con 0x7ff0300cec60 2024-04-02T13:25:18.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.979+0000 7ff02d7fa640 1 -- 172.21.15.44:0/4228395635 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff028003d30 con 0x7ff0300cec60 2024-04-02T13:25:18.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.979+0000 7ff036d32640 1 -- 172.21.15.44:0/4228395635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0301f4120 con 0x7ff0300cec60 2024-04-02T13:25:18.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.979+0000 7ff02d7fa640 1 -- 172.21.15.44:0/4228395635 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff028004240 con 0x7ff0300cec60 2024-04-02T13:25:18.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.979+0000 7ff036d32640 1 -- 172.21.15.44:0/4228395635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0301f46e0 con 0x7ff0300cec60 2024-04-02T13:25:18.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.980+0000 7ff036d32640 1 -- 172.21.15.44:0/4228395635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0300684f0 con 0x7ff0300cec60 2024-04-02T13:25:18.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.981+0000 7ff02d7fa640 1 -- 172.21.15.44:0/4228395635 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0280043e0 con 0x7ff0300cec60 2024-04-02T13:25:18.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.981+0000 7ff02d7fa640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff01c06c8f0 0x7ff01c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:18.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.981+0000 7ff02d7fa640 1 -- 172.21.15.44:0/4228395635 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7ff02808c830 con 0x7ff0300cec60 2024-04-02T13:25:18.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.981+0000 7ff02ffff640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff01c06c8f0 0x7ff01c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:18.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.982+0000 7ff02ffff640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff01c06c8f0 0x7ff01c06edb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff024036000 tx=0x7ff02403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:18.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:18.985+0000 7ff02d7fa640 1 -- 172.21.15.44:0/4228395635 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff02805aa80 con 0x7ff0300cec60 2024-04-02T13:25:19.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.168+0000 7ff036d32640 1 -- 172.21.15.44:0/4228395635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff0300d47f0 con 0x7ff0300cec60 2024-04-02T13:25:19.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.169+0000 7ff02d7fa640 1 -- 172.21.15.44:0/4228395635 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) ==== 104+0+11 (secure 0 0 0) 0x7ff02805a380 con 0x7ff0300cec60 2024-04-02T13:25:19.172 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:25:19.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.170+0000 7ff036d32640 1 -- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff01c06c8f0 msgr2=0x7ff01c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.170+0000 7ff036d32640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff01c06c8f0 0x7ff01c06edb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff024036000 tx=0x7ff02403d040 comp rx=0 tx=0).stop 2024-04-02T13:25:19.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.170+0000 7ff036d32640 1 -- 172.21.15.44:0/4228395635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0300cec60 msgr2=0x7ff030153ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.170+0000 7ff036d32640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0300cec60 0x7ff030153ca0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7ff02800ce10 tx=0x7ff02800a660 comp rx=0 tx=0).stop 2024-04-02T13:25:19.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.170+0000 7ff036d32640 1 -- 172.21.15.44:0/4228395635 shutdown_connections 2024-04-02T13:25:19.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.170+0000 7ff036d32640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff01c06c8f0 0x7ff01c06edb0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.170+0000 7ff036d32640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff03015e5b0 0x7ff0300cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.170+0000 7ff036d32640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff03015e1d0 0x7ff0300cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.170+0000 7ff036d32640 1 --2- 172.21.15.44:0/4228395635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0300cec60 0x7ff030153ca0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.170+0000 7ff036d32640 1 -- 172.21.15.44:0/4228395635 >> 172.21.15.44:0/4228395635 conn(0x7ff03009d3b0 msgr2=0x7ff0300b8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:19.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.171+0000 7ff036d32640 1 -- 172.21.15.44:0/4228395635 shutdown_connections 2024-04-02T13:25:19.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.171+0000 7ff036d32640 1 -- 172.21.15.44:0/4228395635 wait complete. 2024-04-02T13:25:19.184 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:25:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.285+0000 7f85d5275640 1 Processor -- start 2024-04-02T13:25:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.285+0000 7f85d5275640 1 -- start start 2024-04-02T13:25:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.286+0000 7f85d5275640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85d015c1a0 0x7f85d006f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.286+0000 7f85d5275640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f85d00d21c0 0x7f85d00cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.286+0000 7f85d5275640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85d015c580 con 0x7f85d00cfd60 2024-04-02T13:25:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.286+0000 7f85d5275640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f85d0070ad0 con 0x7f85d00d21c0 2024-04-02T13:25:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.286+0000 7f85d5275640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f85d00d01c0 con 0x7f85d015c1a0 2024-04-02T13:25:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.286+0000 7f85ce7fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f85d00cfd60 0x7f85d00d1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44652/0 (socket says 172.21.15.44:44652) 2024-04-02T13:25:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.286+0000 7f85ce7fc640 1 -- 172.21.15.44:0/2016273294 learned_addr learned my addr 172.21.15.44:0/2016273294 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.286+0000 7f85ceffd640 1 --2- 172.21.15.44:0/2016273294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85d015c1a0 0x7f85d006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.286+0000 7f85cf7fe640 1 --2- 172.21.15.44:0/2016273294 >> v2:172.21.15.69:3300/0 conn(0x7f85d00d21c0 0x7f85d00cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.286+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3274869626 0 0) 0x7f85d015c580 con 0x7f85d00cfd60 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.286+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85bc003520 con 0x7f85d00cfd60 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3401557899 0 0) 0x7f85bc003520 con 0x7f85d00cfd60 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85d015c1a0 msgr2=0x7f85d006f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 --2- 172.21.15.44:0/2016273294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85d015c1a0 0x7f85d006f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 >> v2:172.21.15.69:3300/0 conn(0x7f85d00d21c0 msgr2=0x7f85d00cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 --2- 172.21.15.44:0/2016273294 >> v2:172.21.15.69:3300/0 conn(0x7f85d00d21c0 0x7f85d00cc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85d006fa30 con 0x7f85d00cfd60 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f85b8001670 con 0x7f85d00cfd60 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cf7fe640 1 --2- 172.21.15.44:0/2016273294 >> v2:172.21.15.69:3300/0 conn(0x7f85d00d21c0 0x7f85d00cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 >> v1:172.21.15.44:6789/0 conn(0x7f85d00cfd60 legacy=0x7f85d00d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 --2- 172.21.15.44:0/2016273294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85bc0039b0 0x7f85bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 --2- 172.21.15.44:0/2016273294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85bc004800 0x7f85bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 --2- 172.21.15.44:0/2016273294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85d015c1a0 0x7f85bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f85d0070ad0 con 0x7f85bc0039b0 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f85d00d01c0 con 0x7f85bc004800 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f85bc00b9c0 con 0x7f85d015c1a0 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85ceffd640 1 --2- 172.21.15.44:0/2016273294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85bc0039b0 0x7f85bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85ce7fc640 1 --2- 172.21.15.44:0/2016273294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85bc004800 0x7f85bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cf7fe640 1 --2- 172.21.15.44:0/2016273294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85d015c1a0 0x7f85bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85ceffd640 1 -- 172.21.15.44:0/2016273294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85d015c1a0 msgr2=0x7f85bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85ceffd640 1 --2- 172.21.15.44:0/2016273294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85d015c1a0 0x7f85bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85ceffd640 1 -- 172.21.15.44:0/2016273294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85bc004800 msgr2=0x7f85bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85ceffd640 1 --2- 172.21.15.44:0/2016273294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85bc004800 0x7f85bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85ceffd640 1 -- 172.21.15.44:0/2016273294 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85bc003520 con 0x7f85bc0039b0 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.287+0000 7f85cf7fe640 1 --2- 172.21.15.44:0/2016273294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85d015c1a0 0x7f85bc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.288+0000 7f85ceffd640 1 --2- 172.21.15.44:0/2016273294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85bc0039b0 0x7f85bc003dd0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009ac0 tx=0x7f85bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e58a959445ef72 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.288+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c8039070 con 0x7f85bc0039b0 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.288+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85c802c6b0 con 0x7f85bc0039b0 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.288+0000 7f85cdffb640 1 -- 172.21.15.44:0/2016273294 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c8009690 con 0x7f85bc0039b0 2024-04-02T13:25:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.289+0000 7f85d5275640 1 -- 172.21.15.44:0/2016273294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85bc0039b0 msgr2=0x7f85bc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.289+0000 7f85d5275640 1 --2- 172.21.15.44:0/2016273294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85bc0039b0 0x7f85bc003dd0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009ac0 tx=0x7f85bc005830 comp rx=0 tx=0).stop 2024-04-02T13:25:19.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.289+0000 7f85cf7fe640 1 -- 172.21.15.44:0/2016273294 reap_dead start 2024-04-02T13:25:19.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.289+0000 7f85d5275640 1 -- 172.21.15.44:0/2016273294 shutdown_connections 2024-04-02T13:25:19.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.289+0000 7f85d5275640 1 -- 172.21.15.44:0/2016273294 >> 172.21.15.44:0/2016273294 conn(0x7f85d009aa20 msgr2=0x7f85d009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:19.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.289+0000 7f85d5275640 1 -- 172.21.15.44:0/2016273294 shutdown_connections 2024-04-02T13:25:19.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.289+0000 7f85d5275640 1 -- 172.21.15.44:0/2016273294 wait complete. 2024-04-02T13:25:19.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.290+0000 7f85d5275640 1 Processor -- start 2024-04-02T13:25:19.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.290+0000 7f85d5275640 1 -- start start 2024-04-02T13:25:19.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85d5275640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85bc0039b0 0x7f85d006dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85d5275640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d00c4330 0x7f85d00cf040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85d5275640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85d0159e30 0x7f85d015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85d5275640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f85d00702d0 con 0x7f85bc0039b0 2024-04-02T13:25:19.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85d5275640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f85d006fff0 con 0x7f85d00c4330 2024-04-02T13:25:19.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85d5275640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f85d00705b0 con 0x7f85d0159e30 2024-04-02T13:25:19.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85ceffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85bc0039b0 0x7f85d006dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85ceffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85bc0039b0 0x7f85d006dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58254/0 (socket says 172.21.15.44:58254) 2024-04-02T13:25:19.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85ceffd640 1 -- 172.21.15.44:0/885791960 learned_addr learned my addr 172.21.15.44:0/885791960 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:19.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85ceffd640 1 -- 172.21.15.44:0/885791960 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85d0159e30 msgr2=0x7f85d015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85ceffd640 1 --2- 172.21.15.44:0/885791960 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85d0159e30 0x7f85d015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85ceffd640 1 -- 172.21.15.44:0/885791960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d00c4330 msgr2=0x7f85d00cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85ceffd640 1 --2- 172.21.15.44:0/885791960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d00c4330 0x7f85d00cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85ceffd640 1 -- 172.21.15.44:0/885791960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85d00c4830 con 0x7f85bc0039b0 2024-04-02T13:25:19.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85ceffd640 1 --2- 172.21.15.44:0/885791960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85bc0039b0 0x7f85d006dcc0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f85c8000c00 tx=0x7f85c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:19.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85c77fe640 1 -- 172.21.15.44:0/885791960 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c8042020 con 0x7f85bc0039b0 2024-04-02T13:25:19.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85c77fe640 1 -- 172.21.15.44:0/885791960 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85c803a040 con 0x7f85bc0039b0 2024-04-02T13:25:19.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85d5275640 1 -- 172.21.15.44:0/885791960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85d015e560 con 0x7f85bc0039b0 2024-04-02T13:25:19.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.291+0000 7f85c77fe640 1 -- 172.21.15.44:0/885791960 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c8039070 con 0x7f85bc0039b0 2024-04-02T13:25:19.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.292+0000 7f85d5275640 1 -- 172.21.15.44:0/885791960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85d015eb20 con 0x7f85bc0039b0 2024-04-02T13:25:19.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.293+0000 7f85d5275640 1 -- 172.21.15.44:0/885791960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85d0068c00 con 0x7f85bc0039b0 2024-04-02T13:25:19.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.293+0000 7f85c77fe640 1 -- 172.21.15.44:0/885791960 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f85c8045020 con 0x7f85bc0039b0 2024-04-02T13:25:19.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.293+0000 7f85c77fe640 1 --2- 172.21.15.44:0/885791960 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f85b006c920 0x7f85b006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.294+0000 7f85c77fe640 1 -- 172.21.15.44:0/885791960 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f85c80acdd0 con 0x7f85bc0039b0 2024-04-02T13:25:19.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.294+0000 7f85ce7fc640 1 --2- 172.21.15.44:0/885791960 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f85b006c920 0x7f85b006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.294+0000 7f85ce7fc640 1 --2- 172.21.15.44:0/885791960 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f85b006c920 0x7f85b006ede0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f85b8000f60 tx=0x7f85b8000ed0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:19.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.298+0000 7f85c77fe640 1 -- 172.21.15.44:0/885791960 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f85c804c020 con 0x7f85bc0039b0 2024-04-02T13:25:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.481+0000 7f85d5275640 1 -- 172.21.15.44:0/885791960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f85d00b2d00 con 0x7f85bc0039b0 2024-04-02T13:25:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.482+0000 7f85c77fe640 1 -- 172.21.15.44:0/885791960 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) ==== 100+0+11 (secure 0 0 0) 0x7f85c807b0b0 con 0x7f85bc0039b0 2024-04-02T13:25:19.485 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:25:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.483+0000 7f85d5275640 1 -- 172.21.15.44:0/885791960 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f85b006c920 msgr2=0x7f85b006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.483+0000 7f85d5275640 1 --2- 172.21.15.44:0/885791960 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f85b006c920 0x7f85b006ede0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f85b8000f60 tx=0x7f85b8000ed0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.483+0000 7f85d5275640 1 -- 172.21.15.44:0/885791960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85bc0039b0 msgr2=0x7f85d006dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.483+0000 7f85d5275640 1 --2- 172.21.15.44:0/885791960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85bc0039b0 0x7f85d006dcc0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f85c8000c00 tx=0x7f85c8004730 comp rx=0 tx=0).stop 2024-04-02T13:25:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.483+0000 7f85d5275640 1 -- 172.21.15.44:0/885791960 shutdown_connections 2024-04-02T13:25:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.484+0000 7f85d5275640 1 --2- 172.21.15.44:0/885791960 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f85b006c920 0x7f85b006ede0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.484+0000 7f85d5275640 1 --2- 172.21.15.44:0/885791960 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85d0159e30 0x7f85d015a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.484+0000 7f85d5275640 1 --2- 172.21.15.44:0/885791960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d00c4330 0x7f85d00cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.484+0000 7f85d5275640 1 --2- 172.21.15.44:0/885791960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85bc0039b0 0x7f85d006dcc0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.484+0000 7f85d5275640 1 -- 172.21.15.44:0/885791960 >> 172.21.15.44:0/885791960 conn(0x7f85d009aa20 msgr2=0x7f85d00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.484+0000 7f85d5275640 1 -- 172.21.15.44:0/885791960 shutdown_connections 2024-04-02T13:25:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.484+0000 7f85d5275640 1 -- 172.21.15.44:0/885791960 wait complete. 2024-04-02T13:25:19.497 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:25:19.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.596+0000 7f7da1647640 1 Processor -- start 2024-04-02T13:25:19.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.596+0000 7f7da1647640 1 -- start start 2024-04-02T13:25:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.597+0000 7f7da1647640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7d9c0d0a30 0x7f7d9c06cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.597+0000 7f7da1647640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d9c0d0000 con 0x7f7d9c0d2040 2024-04-02T13:25:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.597+0000 7f7da1647640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d9c06c1d0 con 0x7f7d9c0d0e10 2024-04-02T13:25:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.597+0000 7f7da1647640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7d9c06cb00 con 0x7f7d9c0d0a30 2024-04-02T13:25:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.597+0000 7f7d9b7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7d9c0d2040 0x7f7d9c0d24f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44666/0 (socket says 172.21.15.44:44666) 2024-04-02T13:25:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.597+0000 7f7d9b7fe640 1 -- 172.21.15.44:0/897252656 learned_addr learned my addr 172.21.15.44:0/897252656 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.597+0000 7f7d9affd640 1 --2- 172.21.15.44:0/897252656 >> v2:172.21.15.69:3300/0 conn(0x7f7d9c0d0a30 0x7f7d9c06cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.597+0000 7f7d99ffb640 1 -- 172.21.15.44:0/897252656 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3254670278 0 0) 0x7f7d9c0d0000 con 0x7f7d9c0d2040 2024-04-02T13:25:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.598+0000 7f7d99ffb640 1 -- 172.21.15.44:0/897252656 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d88003520 con 0x7f7d9c0d2040 2024-04-02T13:25:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.598+0000 7f7d99ffb640 1 -- 172.21.15.44:0/897252656 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 882123028 0 0) 0x7f7d9c06c1d0 con 0x7f7d9c0d0e10 2024-04-02T13:25:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.598+0000 7f7d99ffb640 1 -- 172.21.15.44:0/897252656 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d9c0d0000 con 0x7f7d9c0d0e10 2024-04-02T13:25:19.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.598+0000 7f7d9affd640 1 -- 172.21.15.44:0/897252656 >> v1:172.21.15.69:6789/0 conn(0x7f7d9c0d0e10 legacy=0x7f7d9c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.598+0000 7f7d9affd640 1 -- 172.21.15.44:0/897252656 >> v1:172.21.15.44:6789/0 conn(0x7f7d9c0d2040 legacy=0x7f7d9c0d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.598+0000 7f7d99ffb640 1 -- 172.21.15.44:0/897252656 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2498511117 0 0) 0x7f7d88003520 con 0x7f7d9c0d2040 2024-04-02T13:25:19.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.598+0000 7f7d9affd640 1 -- 172.21.15.44:0/897252656 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d9c06c4b0 con 0x7f7d9c0d0a30 2024-04-02T13:25:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.599+0000 7f7d9affd640 1 --2- 172.21.15.44:0/897252656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9c0d0a30 0x7f7d9c06cf90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7d940097a0 tx=0x7f7d9402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d87ad39d6dd104c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.599+0000 7f7d99ffb640 1 -- 172.21.15.44:0/897252656 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d94037070 con 0x7f7d9c0d0a30 2024-04-02T13:25:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.599+0000 7f7d99ffb640 1 -- 172.21.15.44:0/897252656 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d94030510 con 0x7f7d9c0d0a30 2024-04-02T13:25:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.599+0000 7f7d99ffb640 1 -- 172.21.15.44:0/897252656 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c0019f0 con 0x7f7d9c0d0a30 2024-04-02T13:25:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.599+0000 7f7da1647640 1 -- 172.21.15.44:0/897252656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9c0d0a30 msgr2=0x7f7d9c06cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.599+0000 7f7da1647640 1 --2- 172.21.15.44:0/897252656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9c0d0a30 0x7f7d9c06cf90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7d940097a0 tx=0x7f7d9402b790 comp rx=0 tx=0).stop 2024-04-02T13:25:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.599+0000 7f7da1647640 1 -- 172.21.15.44:0/897252656 shutdown_connections 2024-04-02T13:25:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.599+0000 7f7da1647640 1 --2- 172.21.15.44:0/897252656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9c0d0a30 0x7f7d9c06cf90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.599+0000 7f7da1647640 1 -- 172.21.15.44:0/897252656 >> 172.21.15.44:0/897252656 conn(0x7f7d9c09d470 msgr2=0x7f7d9c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:19.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.599+0000 7f7da1647640 1 -- 172.21.15.44:0/897252656 shutdown_connections 2024-04-02T13:25:19.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.599+0000 7f7da1647640 1 -- 172.21.15.44:0/897252656 wait complete. 2024-04-02T13:25:19.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.601+0000 7f7da1647640 1 Processor -- start 2024-04-02T13:25:19.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.601+0000 7f7da1647640 1 -- start start 2024-04-02T13:25:19.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.601+0000 7f7da1647640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d9c0d0a30 0x7f7d9c0b8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.601+0000 7f7da1647640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9c0d0e10 0x7f7d9c156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.601+0000 7f7da1647640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d9c0d2040 0x7f7d9c1f7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.601+0000 7f7da1647640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c0d2c00 con 0x7f7d9c0d2040 2024-04-02T13:25:19.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.601+0000 7f7da1647640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c0d1270 con 0x7f7d9c0d0e10 2024-04-02T13:25:19.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.601+0000 7f7da1647640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7d9c15e6f0 con 0x7f7d9c0d0a30 2024-04-02T13:25:19.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.601+0000 7f7d9b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d9c0d2040 0x7f7d9c1f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.601+0000 7f7d9b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d9c0d2040 0x7f7d9c1f7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58256/0 (socket says 172.21.15.44:58256) 2024-04-02T13:25:19.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.601+0000 7f7d9b7fe640 1 -- 172.21.15.44:0/3720122283 learned_addr learned my addr 172.21.15.44:0/3720122283 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7d9a7fc640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9c0d0e10 0x7f7d9c156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7d9affd640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d9c0d0a30 0x7f7d9c0b8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7d9b7fe640 1 -- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d9c0d0a30 msgr2=0x7f7d9c0b8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7d9b7fe640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d9c0d0a30 0x7f7d9c0b8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7d9b7fe640 1 -- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9c0d0e10 msgr2=0x7f7d9c156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7d9b7fe640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9c0d0e10 0x7f7d9c156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7d9b7fe640 1 -- 172.21.15.44:0/3720122283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d9c1f7b70 con 0x7f7d9c0d2040 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7d9a7fc640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9c0d0e10 0x7f7d9c156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7d9b7fe640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d9c0d2040 0x7f7d9c1f7420 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c001fb0 tx=0x7f7d8c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7d937fe640 1 -- 172.21.15.44:0/3720122283 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c017070 con 0x7f7d9c0d2040 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7d937fe640 1 -- 172.21.15.44:0/3720122283 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d8c012400 con 0x7f7d9c0d2040 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7da1647640 1 -- 172.21.15.44:0/3720122283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d9c1f7e50 con 0x7f7d9c0d2040 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7d937fe640 1 -- 172.21.15.44:0/3720122283 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c012830 con 0x7f7d9c0d2040 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.602+0000 7f7da1647640 1 -- 172.21.15.44:0/3720122283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d9c0b42d0 con 0x7f7d9c0d2040 2024-04-02T13:25:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.603+0000 7f7da1647640 1 -- 172.21.15.44:0/3720122283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d9c0b3ba0 con 0x7f7d9c0d2040 2024-04-02T13:25:19.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.604+0000 7f7d937fe640 1 -- 172.21.15.44:0/3720122283 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7d8c01a070 con 0x7f7d9c0d2040 2024-04-02T13:25:19.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.604+0000 7f7d937fe640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7d7c06c8f0 0x7f7d7c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.604+0000 7f7d937fe640 1 -- 172.21.15.44:0/3720122283 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f7d8c08bcc0 con 0x7f7d9c0d2040 2024-04-02T13:25:19.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.604+0000 7f7d9affd640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7d7c06c8f0 0x7f7d7c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.605+0000 7f7d9affd640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7d7c06c8f0 0x7f7d7c06edb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7d94034000 tx=0x7f7d9402f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:19.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.609+0000 7f7d937fe640 1 -- 172.21.15.44:0/3720122283 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7d8c059fa0 con 0x7f7d9c0d2040 2024-04-02T13:25:19.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.805+0000 7f7da1647640 1 -- 172.21.15.44:0/3720122283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7d9c0684f0 con 0x7f7d9c0d2040 2024-04-02T13:25:19.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.805+0000 7f7d937fe640 1 -- 172.21.15.44:0/3720122283 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v80) ==== 80+0+120 (secure 0 0 0) 0x7f7d8c0598a0 con 0x7f7d9c0d2040 2024-04-02T13:25:19.809 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.69:6837/698185867 2024-04-03T13:25:10.921135+0000 2024-04-02T13:25:19.809 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.69:6835/698185867 2024-04-03T13:25:10.921135+0000 2024-04-02T13:25:19.809 INFO:teuthology.orchestra.run.smithi044.stderr:listed 2 entries 2024-04-02T13:25:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.807+0000 7f7da1647640 1 -- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7d7c06c8f0 msgr2=0x7f7d7c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.807+0000 7f7da1647640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7d7c06c8f0 0x7f7d7c06edb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7d94034000 tx=0x7f7d9402f000 comp rx=0 tx=0).stop 2024-04-02T13:25:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.807+0000 7f7da1647640 1 -- 172.21.15.44:0/3720122283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d9c0d2040 msgr2=0x7f7d9c1f7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.807+0000 7f7da1647640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d9c0d2040 0x7f7d9c1f7420 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c001fb0 tx=0x7f7d8c00a430 comp rx=0 tx=0).stop 2024-04-02T13:25:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.807+0000 7f7da1647640 1 -- 172.21.15.44:0/3720122283 shutdown_connections 2024-04-02T13:25:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.807+0000 7f7da1647640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7d7c06c8f0 0x7f7d7c06edb0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.807+0000 7f7da1647640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d9c0d2040 0x7f7d9c1f7420 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.807+0000 7f7da1647640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9c0d0e10 0x7f7d9c156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.807+0000 7f7da1647640 1 --2- 172.21.15.44:0/3720122283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d9c0d0a30 0x7f7d9c0b8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.807+0000 7f7da1647640 1 -- 172.21.15.44:0/3720122283 >> 172.21.15.44:0/3720122283 conn(0x7f7d9c09d470 msgr2=0x7f7d9c15c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.807+0000 7f7da1647640 1 -- 172.21.15.44:0/3720122283 shutdown_connections 2024-04-02T13:25:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.807+0000 7f7da1647640 1 -- 172.21.15.44:0/3720122283 wait complete. 2024-04-02T13:25:19.821 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:25:19.821 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:25:19.836 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:25:19.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.974+0000 7f5548093640 1 Processor -- start 2024-04-02T13:25:19.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.975+0000 7f5548093640 1 -- start start 2024-04-02T13:25:19.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.975+0000 7f5548093640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55400d0a30 0x7f554006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.975+0000 7f5548093640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f55400d2040 0x7f55400d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.975+0000 7f5548093640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55400d0000 con 0x7f55400d0e10 2024-04-02T13:25:19.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.975+0000 7f5548093640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f554006cb00 con 0x7f55400d2040 2024-04-02T13:25:19.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.975+0000 7f5548093640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f554006c1d0 con 0x7f55400d0a30 2024-04-02T13:25:19.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.975+0000 7f5546630640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f55400d2040 0x7f55400d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.975+0000 7f554562e640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f55400d0e10 0x7f55400cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44682/0 (socket says 172.21.15.44:44682) 2024-04-02T13:25:19.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.975+0000 7f5546630640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f55400d2040 0x7f55400d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58258/0 (socket says 172.21.15.44:58258) 2024-04-02T13:25:19.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.975+0000 7f554562e640 1 -- 172.21.15.44:0/968046760 learned_addr learned my addr 172.21.15.44:0/968046760 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:19.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.976+0000 7f5545e2f640 1 --2- 172.21.15.44:0/968046760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55400d0a30 0x7f554006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.976+0000 7f5546630640 1 -- 172.21.15.44:0/968046760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55400d0a30 msgr2=0x7f554006cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:19.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.976+0000 7f5546630640 1 --2- 172.21.15.44:0/968046760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55400d0a30 0x7f554006cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.976+0000 7f5546630640 1 -- 172.21.15.44:0/968046760 >> v1:172.21.15.44:6789/0 conn(0x7f55400d0e10 legacy=0x7f55400cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.976+0000 7f5546630640 1 -- 172.21.15.44:0/968046760 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554006c4b0 con 0x7f55400d2040 2024-04-02T13:25:19.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.976+0000 7f5544e2d640 1 -- 172.21.15.44:0/968046760 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2125566562 0 0) 0x7f55400d0000 con 0x7f55400d0e10 2024-04-02T13:25:19.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.976+0000 7f5546630640 1 --2- 172.21.15.44:0/968046760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55400d2040 0x7f55400d24a0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f5530009d90 tx=0x7f553002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ef1271798f462 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:19.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.977+0000 7f5544e2d640 1 -- 172.21.15.44:0/968046760 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530039070 con 0x7f55400d2040 2024-04-02T13:25:19.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.977+0000 7f5544e2d640 1 -- 172.21.15.44:0/968046760 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f553002cbc0 con 0x7f55400d2040 2024-04-02T13:25:19.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.977+0000 7f5544e2d640 1 -- 172.21.15.44:0/968046760 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530034440 con 0x7f55400d2040 2024-04-02T13:25:19.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.977+0000 7f5548093640 1 -- 172.21.15.44:0/968046760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55400d2040 msgr2=0x7f55400d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.977+0000 7f5548093640 1 --2- 172.21.15.44:0/968046760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55400d2040 0x7f55400d24a0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f5530009d90 tx=0x7f553002be10 comp rx=0 tx=0).stop 2024-04-02T13:25:19.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.977+0000 7f5548093640 1 -- 172.21.15.44:0/968046760 shutdown_connections 2024-04-02T13:25:19.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.977+0000 7f5548093640 1 --2- 172.21.15.44:0/968046760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55400d2040 0x7f55400d24a0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.977+0000 7f5548093640 1 --2- 172.21.15.44:0/968046760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55400d0a30 0x7f554006cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.977+0000 7f5548093640 1 -- 172.21.15.44:0/968046760 >> 172.21.15.44:0/968046760 conn(0x7f554009d470 msgr2=0x7f55400d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:19.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.977+0000 7f5548093640 1 -- 172.21.15.44:0/968046760 shutdown_connections 2024-04-02T13:25:19.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.977+0000 7f5548093640 1 -- 172.21.15.44:0/968046760 wait complete. 2024-04-02T13:25:19.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.978+0000 7f5548093640 1 Processor -- start 2024-04-02T13:25:19.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.978+0000 7f5548093640 1 -- start start 2024-04-02T13:25:19.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5548093640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55400d0a30 0x7f5540159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5548093640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55400d0e10 0x7f554014e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5548093640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55400d2040 0x7f55400b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5548093640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f554015e6f0 con 0x7f55400d2040 2024-04-02T13:25:19.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5548093640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f55400d1270 con 0x7f55400d0e10 2024-04-02T13:25:19.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5548093640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f554015e870 con 0x7f55400d0a30 2024-04-02T13:25:19.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5546630640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55400d2040 0x7f55400b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5546630640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55400d2040 0x7f55400b87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58270/0 (socket says 172.21.15.44:58270) 2024-04-02T13:25:19.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5546630640 1 -- 172.21.15.44:0/3699775112 learned_addr learned my addr 172.21.15.44:0/3699775112 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:19.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5545e2f640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55400d0a30 0x7f5540159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f554562e640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55400d0e10 0x7f554014e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5546630640 1 -- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55400d0a30 msgr2=0x7f5540159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5546630640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55400d0a30 0x7f5540159ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5546630640 1 -- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55400d0e10 msgr2=0x7f554014e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:19.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5546630640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55400d0e10 0x7f554014e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:19.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.979+0000 7f5546630640 1 -- 172.21.15.44:0/3699775112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55400c8c40 con 0x7f55400d2040 2024-04-02T13:25:19.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.980+0000 7f5546630640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55400d2040 0x7f55400b87d0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f5530037040 tx=0x7f5530004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:19.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.980+0000 7f553e7fc640 1 -- 172.21.15.44:0/3699775112 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530044020 con 0x7f55400d2040 2024-04-02T13:25:19.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.980+0000 7f553e7fc640 1 -- 172.21.15.44:0/3699775112 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f553003a040 con 0x7f55400d2040 2024-04-02T13:25:19.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.980+0000 7f5548093640 1 -- 172.21.15.44:0/3699775112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55400c8960 con 0x7f55400d2040 2024-04-02T13:25:19.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.980+0000 7f553e7fc640 1 -- 172.21.15.44:0/3699775112 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530039050 con 0x7f55400d2040 2024-04-02T13:25:19.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.980+0000 7f5548093640 1 -- 172.21.15.44:0/3699775112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55401592b0 con 0x7f55400d2040 2024-04-02T13:25:19.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.981+0000 7f5548093640 1 -- 172.21.15.44:0/3699775112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5540073730 con 0x7f55400d2040 2024-04-02T13:25:19.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.981+0000 7f553e7fc640 1 -- 172.21.15.44:0/3699775112 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5530003bd0 con 0x7f55400d2040 2024-04-02T13:25:19.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.982+0000 7f553e7fc640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f552c06c8f0 0x7f552c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:19.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.982+0000 7f553e7fc640 1 -- 172.21.15.44:0/3699775112 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f55300ae430 con 0x7f55400d2040 2024-04-02T13:25:19.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.982+0000 7f5545e2f640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f552c06c8f0 0x7f552c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:19.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.982+0000 7f5545e2f640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f552c06c8f0 0x7f552c06edb0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f553800b7c0 tx=0x7f5538008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:19.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:19.986+0000 7f553e7fc640 1 -- 172.21.15.44:0/3699775112 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5540073730 con 0x7f55400d2040 2024-04-02T13:25:20.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.174+0000 7f5548093640 1 -- 172.21.15.44:0/3699775112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55400d17c0 con 0x7f55400d2040 2024-04-02T13:25:20.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.175+0000 7f553e7fc640 1 -- 172.21.15.44:0/3699775112 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+17024 (secure 0 0 0) 0x7f553004e060 con 0x7f55400d2040 2024-04-02T13:25:20.179 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:20.179 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":80,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:25:10.921159+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:24:52.202720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-02T13:24:52.916529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6837/698185867":"2024-04-03T13:25:10.921135+0000","172.21.15.69:6835/698185867":"2024-04-03T13:25:10.921135+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:25:20.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.177+0000 7f5548093640 1 -- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f552c06c8f0 msgr2=0x7f552c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.177+0000 7f5548093640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f552c06c8f0 0x7f552c06edb0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f553800b7c0 tx=0x7f5538008040 comp rx=0 tx=0).stop 2024-04-02T13:25:20.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.177+0000 7f5548093640 1 -- 172.21.15.44:0/3699775112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55400d2040 msgr2=0x7f55400b87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.177+0000 7f5548093640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55400d2040 0x7f55400b87d0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f5530037040 tx=0x7f5530004a40 comp rx=0 tx=0).stop 2024-04-02T13:25:20.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.177+0000 7f5548093640 1 -- 172.21.15.44:0/3699775112 shutdown_connections 2024-04-02T13:25:20.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.177+0000 7f5548093640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f552c06c8f0 0x7f552c06edb0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.177+0000 7f5548093640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55400d2040 0x7f55400b87d0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.177+0000 7f5548093640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55400d0e10 0x7f554014e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.177+0000 7f5548093640 1 --2- 172.21.15.44:0/3699775112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55400d0a30 0x7f5540159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.177+0000 7f5548093640 1 -- 172.21.15.44:0/3699775112 >> 172.21.15.44:0/3699775112 conn(0x7f554009d470 msgr2=0x7f55400b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:20.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.177+0000 7f5548093640 1 -- 172.21.15.44:0/3699775112 shutdown_connections 2024-04-02T13:25:20.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.177+0000 7f5548093640 1 -- 172.21.15.44:0/3699775112 wait complete. 2024-04-02T13:25:20.191 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:24:52.202720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:24:52.916529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:25:20.192 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:25:20.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.293+0000 7f83643ba640 1 Processor -- start 2024-04-02T13:25:20.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.293+0000 7f83643ba640 1 -- start start 2024-04-02T13:25:20.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.294+0000 7f83643ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f835c0d2550 0x7f835c15fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.294+0000 7f83643ba640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f835c0d2930 0x7f835c15f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.294+0000 7f83643ba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f835c0d1e80 con 0x7f835c0cf870 2024-04-02T13:25:20.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.294+0000 7f83643ba640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f835c0d2160 con 0x7f835c0d2930 2024-04-02T13:25:20.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.294+0000 7f83643ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f835c0ced90 con 0x7f835c0d2550 2024-04-02T13:25:20.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.294+0000 7f8361955640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f835c0d2930 0x7f835c15f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.294+0000 7f8361955640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f835c0d2930 0x7f835c15f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58286/0 (socket says 172.21.15.44:58286) 2024-04-02T13:25:20.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.294+0000 7f8361955640 1 -- 172.21.15.44:0/1294820964 learned_addr learned my addr 172.21.15.44:0/1294820964 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:20.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.295+0000 7f8362156640 1 --2- 172.21.15.44:0/1294820964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f835c0d2550 0x7f835c15fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.295+0000 7f8361955640 1 -- 172.21.15.44:0/1294820964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f835c0d2550 msgr2=0x7f835c15fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:20.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.295+0000 7f8361955640 1 --2- 172.21.15.44:0/1294820964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f835c0d2550 0x7f835c15fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.295+0000 7f8361955640 1 -- 172.21.15.44:0/1294820964 >> v1:172.21.15.69:6789/0 conn(0x7f835c0cf870 legacy=0x7f835c0cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.295+0000 7f8361955640 1 -- 172.21.15.44:0/1294820964 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f835c0cf070 con 0x7f835c0d2930 2024-04-02T13:25:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.295+0000 7f8361955640 1 --2- 172.21.15.44:0/1294820964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f835c0d2930 0x7f835c15f0a0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f834c009870 tx=0x7f834c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966e217b56fa460c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.296+0000 7f8361154640 1 -- 172.21.15.44:0/1294820964 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c034070 con 0x7f835c0d2930 2024-04-02T13:25:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.296+0000 7f8361154640 1 -- 172.21.15.44:0/1294820964 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f834c030840 con 0x7f835c0d2930 2024-04-02T13:25:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.296+0000 7f8361154640 1 -- 172.21.15.44:0/1294820964 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c030c10 con 0x7f835c0d2930 2024-04-02T13:25:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.296+0000 7f83643ba640 1 -- 172.21.15.44:0/1294820964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f835c0d2930 msgr2=0x7f835c15f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.296+0000 7f83643ba640 1 --2- 172.21.15.44:0/1294820964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f835c0d2930 0x7f835c15f0a0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f834c009870 tx=0x7f834c02b980 comp rx=0 tx=0).stop 2024-04-02T13:25:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.296+0000 7f83643ba640 1 -- 172.21.15.44:0/1294820964 shutdown_connections 2024-04-02T13:25:20.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.296+0000 7f83643ba640 1 --2- 172.21.15.44:0/1294820964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f835c0d2930 0x7f835c15f0a0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.296+0000 7f83643ba640 1 --2- 172.21.15.44:0/1294820964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f835c0d2550 0x7f835c15fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.296+0000 7f83643ba640 1 -- 172.21.15.44:0/1294820964 >> 172.21.15.44:0/1294820964 conn(0x7f835c09d3c0 msgr2=0x7f835c0d4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:20.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.296+0000 7f83643ba640 1 -- 172.21.15.44:0/1294820964 shutdown_connections 2024-04-02T13:25:20.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.296+0000 7f83643ba640 1 -- 172.21.15.44:0/1294820964 wait complete. 2024-04-02T13:25:20.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.297+0000 7f83643ba640 1 Processor -- start 2024-04-02T13:25:20.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.297+0000 7f83643ba640 1 -- start start 2024-04-02T13:25:20.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f83643ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f835c0cf870 0x7f835c155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f83643ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f835c0d2550 0x7f835c0cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f83643ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f835c0d2930 0x7f835c0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f83643ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f835c0d2d70 con 0x7f835c0d2550 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f83643ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f835c0d1b70 con 0x7f835c0d2930 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f83643ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f835c0c98a0 con 0x7f835c0cf870 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f8361955640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f835c0d2550 0x7f835c0cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f8361955640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f835c0d2550 0x7f835c0cb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58302/0 (socket says 172.21.15.44:58302) 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f8361955640 1 -- 172.21.15.44:0/2479208859 learned_addr learned my addr 172.21.15.44:0/2479208859 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f8362957640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f835c0d2930 0x7f835c0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f8362156640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f835c0cf870 0x7f835c155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f8361955640 1 -- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f835c0cf870 msgr2=0x7f835c155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f8361955640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f835c0cf870 0x7f835c155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f8361955640 1 -- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f835c0d2930 msgr2=0x7f835c0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f8361955640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f835c0d2930 0x7f835c0cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.298+0000 7f8361955640 1 -- 172.21.15.44:0/2479208859 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f835c0c8c90 con 0x7f835c0d2550 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.299+0000 7f8361955640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f835c0d2550 0x7f835c0cb070 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f834c000c00 tx=0x7f834c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:20.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.299+0000 7f8352ffd640 1 -- 172.21.15.44:0/2479208859 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c034040 con 0x7f835c0d2550 2024-04-02T13:25:20.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.299+0000 7f8352ffd640 1 -- 172.21.15.44:0/2479208859 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f834c039d90 con 0x7f835c0d2550 2024-04-02T13:25:20.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.299+0000 7f83643ba640 1 -- 172.21.15.44:0/2479208859 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f835c0c8f70 con 0x7f835c0d2550 2024-04-02T13:25:20.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.299+0000 7f8352ffd640 1 -- 172.21.15.44:0/2479208859 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c043610 con 0x7f835c0d2550 2024-04-02T13:25:20.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.299+0000 7f83643ba640 1 -- 172.21.15.44:0/2479208859 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f835c0c9530 con 0x7f835c0d2550 2024-04-02T13:25:20.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.300+0000 7f83643ba640 1 -- 172.21.15.44:0/2479208859 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f835c0684f0 con 0x7f835c0d2550 2024-04-02T13:25:20.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.300+0000 7f8352ffd640 1 -- 172.21.15.44:0/2479208859 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f834c04a060 con 0x7f835c0d2550 2024-04-02T13:25:20.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.301+0000 7f8352ffd640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f834006c8f0 0x7f834006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.301+0000 7f8352ffd640 1 -- 172.21.15.44:0/2479208859 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f834c0adbc0 con 0x7f835c0d2550 2024-04-02T13:25:20.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.301+0000 7f8362156640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f834006c8f0 0x7f834006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.301+0000 7f8362156640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f834006c8f0 0x7f834006edb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f83540079a0 tx=0x7f8354008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:20.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.305+0000 7f8352ffd640 1 -- 172.21.15.44:0/2479208859 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f834c03e070 con 0x7f835c0d2550 2024-04-02T13:25:20.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.488+0000 7f83643ba640 1 -- 172.21.15.44:0/2479208859 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f835c0ca8b0 con 0x7f835c0d2550 2024-04-02T13:25:20.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.489+0000 7f8352ffd640 1 -- 172.21.15.44:0/2479208859 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) ==== 93+0+10 (secure 0 0 0) 0x7f834c07b850 con 0x7f835c0d2550 2024-04-02T13:25:20.492 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:25:20.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.491+0000 7f83643ba640 1 -- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f834006c8f0 msgr2=0x7f834006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.491+0000 7f83643ba640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f834006c8f0 0x7f834006edb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f83540079a0 tx=0x7f8354008040 comp rx=0 tx=0).stop 2024-04-02T13:25:20.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.491+0000 7f83643ba640 1 -- 172.21.15.44:0/2479208859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f835c0d2550 msgr2=0x7f835c0cb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.491+0000 7f83643ba640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f835c0d2550 0x7f835c0cb070 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f834c000c00 tx=0x7f834c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.491+0000 7f83643ba640 1 -- 172.21.15.44:0/2479208859 shutdown_connections 2024-04-02T13:25:20.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.491+0000 7f83643ba640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f834006c8f0 0x7f834006edb0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.491+0000 7f83643ba640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f835c0d2930 0x7f835c0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.491+0000 7f83643ba640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f835c0d2550 0x7f835c0cb070 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.491+0000 7f83643ba640 1 --2- 172.21.15.44:0/2479208859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f835c0cf870 0x7f835c155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.491+0000 7f83643ba640 1 -- 172.21.15.44:0/2479208859 >> 172.21.15.44:0/2479208859 conn(0x7f835c09d3c0 msgr2=0x7f835c0b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:20.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.491+0000 7f83643ba640 1 -- 172.21.15.44:0/2479208859 shutdown_connections 2024-04-02T13:25:20.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.491+0000 7f83643ba640 1 -- 172.21.15.44:0/2479208859 wait complete. 2024-04-02T13:25:20.505 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:25:20.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.607+0000 7f8708352640 1 Processor -- start 2024-04-02T13:25:20.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.607+0000 7f8708352640 1 -- start start 2024-04-02T13:25:20.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f8708352640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87000d0a30 0x7f87000b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f8708352640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87000d0e10 con 0x7f87000b3990 2024-04-02T13:25:20.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f8708352640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8700144750 con 0x7f8700144130 2024-04-02T13:25:20.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f8708352640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f87000b3df0 con 0x7f87000d0a30 2024-04-02T13:25:20.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87058ed640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f87000b3990 0x7f8700143a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44698/0 (socket says 172.21.15.44:44698) 2024-04-02T13:25:20.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87058ed640 1 -- 172.21.15.44:0/4110056982 learned_addr learned my addr 172.21.15.44:0/4110056982 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:20.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87060ee640 1 --2- 172.21.15.44:0/4110056982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87000d0a30 0x7f87000b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3845053986 0 0) 0x7f86f0001110 con 0x7f87000b3990 2024-04-02T13:25:20.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86ec003520 con 0x7f87000b3990 2024-04-02T13:25:20.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2786472098 0 0) 0x7f8700144750 con 0x7f8700144130 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86f0001110 con 0x7f8700144130 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4258623959 0 0) 0x7f86ec003520 con 0x7f87000b3990 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87000d0a30 msgr2=0x7f87000b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87050ec640 1 --2- 172.21.15.44:0/4110056982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87000d0a30 0x7f87000b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 >> v1:172.21.15.69:6789/0 conn(0x7f8700144130 legacy=0x7f870015e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8700162060 con 0x7f87000b3990 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f86f0001840 con 0x7f87000b3990 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87060ee640 1 --2- 172.21.15.44:0/4110056982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87000d0a30 0x7f87000b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.608+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 >> v1:172.21.15.44:6789/0 conn(0x7f87000b3990 legacy=0x7f8700143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87050ec640 1 --2- 172.21.15.44:0/4110056982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87000d0a30 0x7f86ec003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87050ec640 1 --2- 172.21.15.44:0/4110056982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86ec0040b0 0x7f86ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87050ec640 1 --2- 172.21.15.44:0/4110056982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86ec004d40 0x7f86ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f86ec00bad0 con 0x7f86ec0040b0 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f86ec00bc80 con 0x7f86ec004d40 2024-04-02T13:25:20.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f86ec00be80 con 0x7f87000d0a30 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87058ed640 1 --2- 172.21.15.44:0/4110056982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86ec0040b0 0x7f86ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87060ee640 1 --2- 172.21.15.44:0/4110056982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87000d0a30 0x7f86ec003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87068ef640 1 --2- 172.21.15.44:0/4110056982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86ec004d40 0x7f86ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87058ed640 1 -- 172.21.15.44:0/4110056982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87000d0a30 msgr2=0x7f86ec003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87058ed640 1 --2- 172.21.15.44:0/4110056982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87000d0a30 0x7f86ec003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87058ed640 1 -- 172.21.15.44:0/4110056982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86ec004d40 msgr2=0x7f86ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87058ed640 1 --2- 172.21.15.44:0/4110056982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86ec004d40 0x7f86ec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.609+0000 7f87058ed640 1 -- 172.21.15.44:0/4110056982 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86ec003520 con 0x7f86ec0040b0 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.610+0000 7f87058ed640 1 --2- 172.21.15.44:0/4110056982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86ec0040b0 0x7f86ec004800 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f86f000c4c0 tx=0x7f86ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c5010242767b6bf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.610+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f000c0e0 con 0x7f86ec0040b0 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.610+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86f0030070 con 0x7f86ec0040b0 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.610+0000 7f87050ec640 1 -- 172.21.15.44:0/4110056982 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f00058d0 con 0x7f86ec0040b0 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.610+0000 7f8708352640 1 -- 172.21.15.44:0/4110056982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86ec0040b0 msgr2=0x7f86ec004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.610+0000 7f8708352640 1 --2- 172.21.15.44:0/4110056982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86ec0040b0 0x7f86ec004800 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f86f000c4c0 tx=0x7f86ec005830 comp rx=0 tx=0).stop 2024-04-02T13:25:20.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.610+0000 7f87068ef640 1 -- 172.21.15.44:0/4110056982 reap_dead start 2024-04-02T13:25:20.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.610+0000 7f8708352640 1 -- 172.21.15.44:0/4110056982 shutdown_connections 2024-04-02T13:25:20.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.610+0000 7f8708352640 1 -- 172.21.15.44:0/4110056982 >> 172.21.15.44:0/4110056982 conn(0x7f87000b5d90 msgr2=0x7f870006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:20.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.610+0000 7f8708352640 1 -- 172.21.15.44:0/4110056982 shutdown_connections 2024-04-02T13:25:20.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.611+0000 7f8708352640 1 -- 172.21.15.44:0/4110056982 wait complete. 2024-04-02T13:25:20.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.612+0000 7f8708352640 1 Processor -- start 2024-04-02T13:25:20.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.612+0000 7f8708352640 1 -- start start 2024-04-02T13:25:20.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.612+0000 7f8708352640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86ec0040b0 0x7f8700147700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.612+0000 7f8708352640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8700140f60 0x7f870015ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.612+0000 7f8708352640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f870015f290 0x7f87000b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.612+0000 7f8708352640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8700162620 con 0x7f86ec0040b0 2024-04-02T13:25:20.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.612+0000 7f8708352640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8700161e10 con 0x7f870015f290 2024-04-02T13:25:20.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.612+0000 7f8708352640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8700162b20 con 0x7f8700140f60 2024-04-02T13:25:20.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.612+0000 7f87060ee640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86ec0040b0 0x7f8700147700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.612+0000 7f87060ee640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86ec0040b0 0x7f8700147700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58318/0 (socket says 172.21.15.44:58318) 2024-04-02T13:25:20.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.612+0000 7f87060ee640 1 -- 172.21.15.44:0/2221294230 learned_addr learned my addr 172.21.15.44:0/2221294230 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:20.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f87058ed640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8700140f60 0x7f870015ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f87068ef640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f870015f290 0x7f87000b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f87060ee640 1 -- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8700140f60 msgr2=0x7f870015ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f87060ee640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8700140f60 0x7f870015ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f87060ee640 1 -- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f870015f290 msgr2=0x7f87000b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f87060ee640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f870015f290 0x7f87000b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f87060ee640 1 -- 172.21.15.44:0/2221294230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87001413e0 con 0x7f86ec0040b0 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f87058ed640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8700140f60 0x7f870015ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f87060ee640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86ec0040b0 0x7f8700147700 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0119d0 tx=0x7f86fc011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f86f6ffd640 1 -- 172.21.15.44:0/2221294230 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc00fa90 con 0x7f86ec0040b0 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f86f6ffd640 1 -- 172.21.15.44:0/2221294230 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86fc00fc30 con 0x7f86ec0040b0 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f8708352640 1 -- 172.21.15.44:0/2221294230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87001454b0 con 0x7f86ec0040b0 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f86f6ffd640 1 -- 172.21.15.44:0/2221294230 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc014440 con 0x7f86ec0040b0 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.613+0000 7f8708352640 1 -- 172.21.15.44:0/2221294230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8700145a70 con 0x7f86ec0040b0 2024-04-02T13:25:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.614+0000 7f8708352640 1 -- 172.21.15.44:0/2221294230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87000684f0 con 0x7f86ec0040b0 2024-04-02T13:25:20.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.614+0000 7f86f6ffd640 1 -- 172.21.15.44:0/2221294230 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f86fc012040 con 0x7f86ec0040b0 2024-04-02T13:25:20.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.615+0000 7f86f6ffd640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f86e406c920 0x7f86e406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.615+0000 7f86f6ffd640 1 -- 172.21.15.44:0/2221294230 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f86fc08c140 con 0x7f86ec0040b0 2024-04-02T13:25:20.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.615+0000 7f87058ed640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f86e406c920 0x7f86e406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.615+0000 7f87058ed640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f86e406c920 0x7f86e406ede0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f86f0034000 tx=0x7f86f0007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:20.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.619+0000 7f86f6ffd640 1 -- 172.21.15.44:0/2221294230 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f86fc019030 con 0x7f86ec0040b0 2024-04-02T13:25:20.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.803+0000 7f8708352640 1 -- 172.21.15.44:0/2221294230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f870006bcb0 con 0x7f86ec0040b0 2024-04-02T13:25:20.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.804+0000 7f86f6ffd640 1 -- 172.21.15.44:0/2221294230 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) ==== 104+0+11 (secure 0 0 0) 0x7f86fc05a420 con 0x7f86ec0040b0 2024-04-02T13:25:20.807 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:25:20.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.806+0000 7f8708352640 1 -- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f86e406c920 msgr2=0x7f86e406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.806+0000 7f8708352640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f86e406c920 0x7f86e406ede0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f86f0034000 tx=0x7f86f0007000 comp rx=0 tx=0).stop 2024-04-02T13:25:20.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.806+0000 7f8708352640 1 -- 172.21.15.44:0/2221294230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86ec0040b0 msgr2=0x7f8700147700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.806+0000 7f8708352640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86ec0040b0 0x7f8700147700 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0119d0 tx=0x7f86fc011e90 comp rx=0 tx=0).stop 2024-04-02T13:25:20.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.806+0000 7f8708352640 1 -- 172.21.15.44:0/2221294230 shutdown_connections 2024-04-02T13:25:20.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.806+0000 7f8708352640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f86e406c920 0x7f86e406ede0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.806+0000 7f8708352640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f870015f290 0x7f87000b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.806+0000 7f8708352640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8700140f60 0x7f870015ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.806+0000 7f8708352640 1 --2- 172.21.15.44:0/2221294230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86ec0040b0 0x7f8700147700 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.806+0000 7f8708352640 1 -- 172.21.15.44:0/2221294230 >> 172.21.15.44:0/2221294230 conn(0x7f87000b5d90 msgr2=0x7f8700160f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:20.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.806+0000 7f8708352640 1 -- 172.21.15.44:0/2221294230 shutdown_connections 2024-04-02T13:25:20.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.806+0000 7f8708352640 1 -- 172.21.15.44:0/2221294230 wait complete. 2024-04-02T13:25:20.820 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:25:20.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.921+0000 7f3a9744e640 1 Processor -- start 2024-04-02T13:25:20.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.921+0000 7f3a9744e640 1 -- start start 2024-04-02T13:25:20.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.922+0000 7f3a9744e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3a901565d0 0x7f3a900d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.922+0000 7f3a9744e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3a900d05e0 0x7f3a900ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.922+0000 7f3a9744e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a90160b10 con 0x7f3a900cf400 2024-04-02T13:25:20.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.922+0000 7f3a9744e640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f3a900cfad0 con 0x7f3a901565d0 2024-04-02T13:25:20.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.922+0000 7f3a9744e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3a900cf870 con 0x7f3a900d05e0 2024-04-02T13:25:20.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.922+0000 7f3a951ea640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3a901565d0 0x7f3a900d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.922+0000 7f3a951ea640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3a901565d0 0x7f3a900d00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58326/0 (socket says 172.21.15.44:58326) 2024-04-02T13:25:20.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.922+0000 7f3a951ea640 1 -- 172.21.15.44:0/2174373334 learned_addr learned my addr 172.21.15.44:0/2174373334 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:20.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.922+0000 7f3a949e9640 1 --2- 172.21.15.44:0/2174373334 >> v2:172.21.15.69:3300/0 conn(0x7f3a900d05e0 0x7f3a900ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.923+0000 7f3a951ea640 1 -- 172.21.15.44:0/2174373334 >> v2:172.21.15.69:3300/0 conn(0x7f3a900d05e0 msgr2=0x7f3a900ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:20.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.923+0000 7f3a951ea640 1 --2- 172.21.15.44:0/2174373334 >> v2:172.21.15.69:3300/0 conn(0x7f3a900d05e0 0x7f3a900ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.923+0000 7f3a951ea640 1 -- 172.21.15.44:0/2174373334 >> v1:172.21.15.69:6789/0 conn(0x7f3a900cf400 legacy=0x7f3a900cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:20.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.923+0000 7f3a951ea640 1 -- 172.21.15.44:0/2174373334 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9015fd50 con 0x7f3a901565d0 2024-04-02T13:25:20.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.923+0000 7f3a951ea640 1 --2- 172.21.15.44:0/2174373334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a901565d0 0x7f3a900d00a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f3a900b2a30 tx=0x7f3a8802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e5e074f85112a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:20.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.923+0000 7f3a8ffff640 1 -- 172.21.15.44:0/2174373334 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8802e070 con 0x7f3a901565d0 2024-04-02T13:25:20.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.923+0000 7f3a8ffff640 1 -- 172.21.15.44:0/2174373334 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a88037070 con 0x7f3a901565d0 2024-04-02T13:25:20.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.923+0000 7f3a8ffff640 1 -- 172.21.15.44:0/2174373334 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88031a00 con 0x7f3a901565d0 2024-04-02T13:25:20.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.924+0000 7f3a9744e640 1 -- 172.21.15.44:0/2174373334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a901565d0 msgr2=0x7f3a900d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.924+0000 7f3a9744e640 1 --2- 172.21.15.44:0/2174373334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a901565d0 0x7f3a900d00a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f3a900b2a30 tx=0x7f3a8802b880 comp rx=0 tx=0).stop 2024-04-02T13:25:20.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.924+0000 7f3a9744e640 1 -- 172.21.15.44:0/2174373334 shutdown_connections 2024-04-02T13:25:20.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.924+0000 7f3a9744e640 1 --2- 172.21.15.44:0/2174373334 >> v2:172.21.15.69:3300/0 conn(0x7f3a900d05e0 0x7f3a900ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.924+0000 7f3a9744e640 1 --2- 172.21.15.44:0/2174373334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a901565d0 0x7f3a900d00a0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.924+0000 7f3a9744e640 1 -- 172.21.15.44:0/2174373334 >> 172.21.15.44:0/2174373334 conn(0x7f3a9006b920 msgr2=0x7f3a9006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:20.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.924+0000 7f3a9744e640 1 -- 172.21.15.44:0/2174373334 shutdown_connections 2024-04-02T13:25:20.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.924+0000 7f3a9744e640 1 -- 172.21.15.44:0/2174373334 wait complete. 2024-04-02T13:25:20.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.925+0000 7f3a9744e640 1 Processor -- start 2024-04-02T13:25:20.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.925+0000 7f3a9744e640 1 -- start start 2024-04-02T13:25:20.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a9744e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a900cf400 0x7f3a901e6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a9744e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a900d05e0 0x7f3a9009d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a9744e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a901565d0 0x7f3a9014eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a9744e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3a901605f0 con 0x7f3a900cf400 2024-04-02T13:25:20.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a9744e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3a90160310 con 0x7f3a901565d0 2024-04-02T13:25:20.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a9744e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3a901569b0 con 0x7f3a900d05e0 2024-04-02T13:25:20.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a951ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a900cf400 0x7f3a901e6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a951ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a900cf400 0x7f3a901e6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58328/0 (socket says 172.21.15.44:58328) 2024-04-02T13:25:20.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a951ea640 1 -- 172.21.15.44:0/66323080 learned_addr learned my addr 172.21.15.44:0/66323080 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:20.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a959eb640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a901565d0 0x7f3a9014eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a949e9640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a900d05e0 0x7f3a9009d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a951ea640 1 -- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a900d05e0 msgr2=0x7f3a9009d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a951ea640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a900d05e0 0x7f3a9009d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a951ea640 1 -- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a901565d0 msgr2=0x7f3a9014eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:20.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a951ea640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a901565d0 0x7f3a9014eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:20.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.926+0000 7f3a951ea640 1 -- 172.21.15.44:0/66323080 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a90155400 con 0x7f3a900cf400 2024-04-02T13:25:20.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.927+0000 7f3a951ea640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a900cf400 0x7f3a901e6fa0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f3a8802b8c0 tx=0x7f3a880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:20.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.927+0000 7f3a8dffb640 1 -- 172.21.15.44:0/66323080 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88032670 con 0x7f3a900cf400 2024-04-02T13:25:20.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.927+0000 7f3a8dffb640 1 -- 172.21.15.44:0/66323080 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a88032830 con 0x7f3a900cf400 2024-04-02T13:25:20.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.927+0000 7f3a9744e640 1 -- 172.21.15.44:0/66323080 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a901556e0 con 0x7f3a900cf400 2024-04-02T13:25:20.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.927+0000 7f3a8dffb640 1 -- 172.21.15.44:0/66323080 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88032c20 con 0x7f3a900cf400 2024-04-02T13:25:20.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.927+0000 7f3a9744e640 1 -- 172.21.15.44:0/66323080 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a9014f9a0 con 0x7f3a900cf400 2024-04-02T13:25:20.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.928+0000 7f3a9744e640 1 -- 172.21.15.44:0/66323080 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a900684f0 con 0x7f3a900cf400 2024-04-02T13:25:20.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.929+0000 7f3a8dffb640 1 -- 172.21.15.44:0/66323080 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3a8804c060 con 0x7f3a900cf400 2024-04-02T13:25:20.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.929+0000 7f3a8dffb640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3a7c06c8f0 0x7f3a7c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:20.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.929+0000 7f3a8dffb640 1 -- 172.21.15.44:0/66323080 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f3a880ad660 con 0x7f3a900cf400 2024-04-02T13:25:20.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.929+0000 7f3a949e9640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3a7c06c8f0 0x7f3a7c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:20.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.930+0000 7f3a949e9640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3a7c06c8f0 0x7f3a7c06edb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3a9015fb80 tx=0x7f3a84008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:20.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:20.933+0000 7f3a8dffb640 1 -- 172.21.15.44:0/66323080 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3a88040070 con 0x7f3a900cf400 2024-04-02T13:25:21.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.119+0000 7f3a9744e640 1 -- 172.21.15.44:0/66323080 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3a901558e0 con 0x7f3a900cf400 2024-04-02T13:25:21.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.119+0000 7f3a8dffb640 1 -- 172.21.15.44:0/66323080 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) ==== 100+0+11 (secure 0 0 0) 0x7f3a8807b270 con 0x7f3a900cf400 2024-04-02T13:25:21.122 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:25:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.121+0000 7f3a9744e640 1 -- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3a7c06c8f0 msgr2=0x7f3a7c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.121+0000 7f3a9744e640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3a7c06c8f0 0x7f3a7c06edb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3a9015fb80 tx=0x7f3a84008040 comp rx=0 tx=0).stop 2024-04-02T13:25:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.121+0000 7f3a9744e640 1 -- 172.21.15.44:0/66323080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a900cf400 msgr2=0x7f3a901e6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.121+0000 7f3a9744e640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a900cf400 0x7f3a901e6fa0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f3a8802b8c0 tx=0x7f3a880046c0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.121+0000 7f3a9744e640 1 -- 172.21.15.44:0/66323080 shutdown_connections 2024-04-02T13:25:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.121+0000 7f3a9744e640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3a7c06c8f0 0x7f3a7c06edb0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.121+0000 7f3a9744e640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a901565d0 0x7f3a9014eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.121+0000 7f3a9744e640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a900d05e0 0x7f3a9009d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.121+0000 7f3a9744e640 1 --2- 172.21.15.44:0/66323080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a900cf400 0x7f3a901e6fa0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.121+0000 7f3a9744e640 1 -- 172.21.15.44:0/66323080 >> 172.21.15.44:0/66323080 conn(0x7f3a9006b920 msgr2=0x7f3a900b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.121+0000 7f3a9744e640 1 -- 172.21.15.44:0/66323080 shutdown_connections 2024-04-02T13:25:21.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.121+0000 7f3a9744e640 1 -- 172.21.15.44:0/66323080 wait complete. 2024-04-02T13:25:21.135 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:25:21.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.233+0000 7efc323e9640 1 Processor -- start 2024-04-02T13:25:21.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.234+0000 7efc323e9640 1 -- start start 2024-04-02T13:25:21.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.234+0000 7efc323e9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efc2c0d0a30 0x7efc2c06cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.234+0000 7efc323e9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7efc2c0d2040 0x7efc2c0d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.234+0000 7efc323e9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc2c0d0000 con 0x7efc2c0d0e10 2024-04-02T13:25:21.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.234+0000 7efc323e9640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7efc2c06cb00 con 0x7efc2c0d2040 2024-04-02T13:25:21.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.234+0000 7efc323e9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7efc2c06c1d0 con 0x7efc2c0d0a30 2024-04-02T13:25:21.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.235+0000 7efc30986640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7efc2c0d2040 0x7efc2c0d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.235+0000 7efc2b7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7efc2c0d0e10 0x7efc2c0cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44714/0 (socket says 172.21.15.44:44714) 2024-04-02T13:25:21.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.235+0000 7efc2b7fe640 1 -- 172.21.15.44:0/546362079 learned_addr learned my addr 172.21.15.44:0/546362079 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:21.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.235+0000 7efc2bfff640 1 --2- 172.21.15.44:0/546362079 >> v2:172.21.15.69:3300/0 conn(0x7efc2c0d0a30 0x7efc2c06cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.235+0000 7efc30986640 1 -- 172.21.15.44:0/546362079 >> v2:172.21.15.69:3300/0 conn(0x7efc2c0d0a30 msgr2=0x7efc2c06cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.235+0000 7efc30986640 1 --2- 172.21.15.44:0/546362079 >> v2:172.21.15.69:3300/0 conn(0x7efc2c0d0a30 0x7efc2c06cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.235+0000 7efc30986640 1 -- 172.21.15.44:0/546362079 >> v1:172.21.15.44:6789/0 conn(0x7efc2c0d0e10 legacy=0x7efc2c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.235+0000 7efc30986640 1 -- 172.21.15.44:0/546362079 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc2c06c4b0 con 0x7efc2c0d2040 2024-04-02T13:25:21.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.235+0000 7efc2affd640 1 -- 172.21.15.44:0/546362079 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1403510415 0 0) 0x7efc2c0d0000 con 0x7efc2c0d0e10 2024-04-02T13:25:21.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.235+0000 7efc2bfff640 1 --2- 172.21.15.44:0/546362079 >> v2:172.21.15.69:3300/0 conn(0x7efc2c0d0a30 0x7efc2c06cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:21.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.236+0000 7efc30986640 1 --2- 172.21.15.44:0/546362079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc2c0d2040 0x7efc2c0d24f0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7efc2c06de20 tx=0x7efc2402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56413fec5293def0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.236+0000 7efc2affd640 1 -- 172.21.15.44:0/546362079 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc24039070 con 0x7efc2c0d2040 2024-04-02T13:25:21.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.236+0000 7efc2affd640 1 -- 172.21.15.44:0/546362079 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc2402cbf0 con 0x7efc2c0d2040 2024-04-02T13:25:21.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.236+0000 7efc2affd640 1 -- 172.21.15.44:0/546362079 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc24034440 con 0x7efc2c0d2040 2024-04-02T13:25:21.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.236+0000 7efc323e9640 1 -- 172.21.15.44:0/546362079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc2c0d2040 msgr2=0x7efc2c0d24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.236+0000 7efc323e9640 1 --2- 172.21.15.44:0/546362079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc2c0d2040 0x7efc2c0d24f0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7efc2c06de20 tx=0x7efc2402be40 comp rx=0 tx=0).stop 2024-04-02T13:25:21.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.236+0000 7efc323e9640 1 -- 172.21.15.44:0/546362079 shutdown_connections 2024-04-02T13:25:21.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.236+0000 7efc323e9640 1 --2- 172.21.15.44:0/546362079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc2c0d2040 0x7efc2c0d24f0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.236+0000 7efc323e9640 1 --2- 172.21.15.44:0/546362079 >> v2:172.21.15.69:3300/0 conn(0x7efc2c0d0a30 0x7efc2c06cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.236+0000 7efc323e9640 1 -- 172.21.15.44:0/546362079 >> 172.21.15.44:0/546362079 conn(0x7efc2c09d470 msgr2=0x7efc2c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.236+0000 7efc323e9640 1 -- 172.21.15.44:0/546362079 shutdown_connections 2024-04-02T13:25:21.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.236+0000 7efc323e9640 1 -- 172.21.15.44:0/546362079 wait complete. 2024-04-02T13:25:21.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.237+0000 7efc323e9640 1 Processor -- start 2024-04-02T13:25:21.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.238+0000 7efc323e9640 1 -- start start 2024-04-02T13:25:21.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.238+0000 7efc323e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc2c0d0a30 0x7efc2c0cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.238+0000 7efc323e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc2c0d0e10 0x7efc2c0cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.238+0000 7efc323e9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc2c0d2040 0x7efc2c1e92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.238+0000 7efc323e9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efc2c15e6f0 con 0x7efc2c0d2040 2024-04-02T13:25:21.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.238+0000 7efc323e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efc2c0d1270 con 0x7efc2c0d0e10 2024-04-02T13:25:21.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.238+0000 7efc323e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7efc2c15e870 con 0x7efc2c0d0a30 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.238+0000 7efc30986640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc2c0d2040 0x7efc2c1e92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.238+0000 7efc30986640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc2c0d2040 0x7efc2c1e92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58336/0 (socket says 172.21.15.44:58336) 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.238+0000 7efc30986640 1 -- 172.21.15.44:0/1522831643 learned_addr learned my addr 172.21.15.44:0/1522831643 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.239+0000 7efc30986640 1 -- 172.21.15.44:0/1522831643 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc2c0d0a30 msgr2=0x7efc2c0cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.239+0000 7efc30986640 1 --2- 172.21.15.44:0/1522831643 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc2c0d0a30 0x7efc2c0cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.239+0000 7efc2b7fe640 1 --2- 172.21.15.44:0/1522831643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc2c0d0e10 0x7efc2c0cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.239+0000 7efc30986640 1 -- 172.21.15.44:0/1522831643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc2c0d0e10 msgr2=0x7efc2c0cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.239+0000 7efc30986640 1 --2- 172.21.15.44:0/1522831643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc2c0d0e10 0x7efc2c0cb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.239+0000 7efc30986640 1 -- 172.21.15.44:0/1522831643 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc2c0b49e0 con 0x7efc2c0d2040 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.239+0000 7efc30986640 1 --2- 172.21.15.44:0/1522831643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc2c0d2040 0x7efc2c1e92d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7efc24037040 tx=0x7efc24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.239+0000 7efc28ff9640 1 -- 172.21.15.44:0/1522831643 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc24044020 con 0x7efc2c0d2040 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.239+0000 7efc28ff9640 1 -- 172.21.15.44:0/1522831643 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc2403a040 con 0x7efc2c0d2040 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.239+0000 7efc323e9640 1 -- 172.21.15.44:0/1522831643 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc2c0b4cc0 con 0x7efc2c0d2040 2024-04-02T13:25:21.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.239+0000 7efc28ff9640 1 -- 172.21.15.44:0/1522831643 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc24039050 con 0x7efc2c0d2040 2024-04-02T13:25:21.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.239+0000 7efc323e9640 1 -- 172.21.15.44:0/1522831643 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc2c0b5280 con 0x7efc2c0d2040 2024-04-02T13:25:21.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.240+0000 7efc323e9640 1 -- 172.21.15.44:0/1522831643 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc2c0684f0 con 0x7efc2c0d2040 2024-04-02T13:25:21.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.241+0000 7efc28ff9640 1 -- 172.21.15.44:0/1522831643 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efc24003b90 con 0x7efc2c0d2040 2024-04-02T13:25:21.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.241+0000 7efc28ff9640 1 --2- 172.21.15.44:0/1522831643 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efc1806c8f0 0x7efc1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.241+0000 7efc28ff9640 1 -- 172.21.15.44:0/1522831643 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7efc240ae540 con 0x7efc2c0d2040 2024-04-02T13:25:21.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.241+0000 7efc2bfff640 1 --2- 172.21.15.44:0/1522831643 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efc1806c8f0 0x7efc1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.242+0000 7efc2bfff640 1 --2- 172.21.15.44:0/1522831643 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efc1806c8f0 0x7efc1806edb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7efc1c006fd0 tx=0x7efc1c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.245+0000 7efc28ff9640 1 -- 172.21.15.44:0/1522831643 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7efc2404e020 con 0x7efc2c0d2040 2024-04-02T13:25:21.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.440+0000 7efc323e9640 1 -- 172.21.15.44:0/1522831643 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efc2c0d17c0 con 0x7efc2c0d2040 2024-04-02T13:25:21.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.441+0000 7efc28ff9640 1 -- 172.21.15.44:0/1522831643 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v80) ==== 80+0+120 (secure 0 0 0) 0x7efc2407c820 con 0x7efc2c0d2040 2024-04-02T13:25:21.444 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.69:6837/698185867 2024-04-03T13:25:10.921135+0000 2024-04-02T13:25:21.444 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.69:6835/698185867 2024-04-03T13:25:10.921135+0000 2024-04-02T13:25:21.444 INFO:teuthology.orchestra.run.smithi044.stderr:listed 2 entries 2024-04-02T13:25:21.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.443+0000 7efc323e9640 1 -- 172.21.15.44:0/1522831643 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efc1806c8f0 msgr2=0x7efc1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.443+0000 7efc323e9640 1 --2- 172.21.15.44:0/1522831643 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efc1806c8f0 0x7efc1806edb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7efc1c006fd0 tx=0x7efc1c008040 comp rx=0 tx=0).stop 2024-04-02T13:25:21.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.443+0000 7efc323e9640 1 -- 172.21.15.44:0/1522831643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc2c0d2040 msgr2=0x7efc2c1e92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.443+0000 7efc323e9640 1 --2- 172.21.15.44:0/1522831643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc2c0d2040 0x7efc2c1e92d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7efc24037040 tx=0x7efc24004a40 comp rx=0 tx=0).stop 2024-04-02T13:25:21.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.443+0000 7efc323e9640 1 -- 172.21.15.44:0/1522831643 shutdown_connections 2024-04-02T13:25:21.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.443+0000 7efc323e9640 1 --2- 172.21.15.44:0/1522831643 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efc1806c8f0 0x7efc1806edb0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.443+0000 7efc323e9640 1 --2- 172.21.15.44:0/1522831643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc2c0d2040 0x7efc2c1e92d0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.443+0000 7efc323e9640 1 --2- 172.21.15.44:0/1522831643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc2c0d0e10 0x7efc2c0cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.443+0000 7efc323e9640 1 --2- 172.21.15.44:0/1522831643 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc2c0d0a30 0x7efc2c0cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.443+0000 7efc323e9640 1 -- 172.21.15.44:0/1522831643 >> 172.21.15.44:0/1522831643 conn(0x7efc2c09d470 msgr2=0x7efc2c15c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.443+0000 7efc323e9640 1 -- 172.21.15.44:0/1522831643 shutdown_connections 2024-04-02T13:25:21.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.443+0000 7efc323e9640 1 -- 172.21.15.44:0/1522831643 wait complete. 2024-04-02T13:25:21.456 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:25:21.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:21.457 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:25:21.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[65412]: fuse finished with error 0 and tester_r 0 2024-04-02T13:25:21.484 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:25:21.528 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:25:21.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:21.529 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:25:21.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:21.544 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub start /subdir recursive 2024-04-02T13:25:21.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.686+0000 7f6c4d700640 1 Processor -- start 2024-04-02T13:25:21.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.686+0000 7f6c4d700640 1 -- start start 2024-04-02T13:25:21.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.687+0000 7f6c4d700640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c480d1070 0x7f6c4815e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.687+0000 7f6c4d700640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c480b3aa0 con 0x7f6c480d2990 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.687+0000 7f6c4d700640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c480d2f50 con 0x7f6c480d0c90 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.687+0000 7f6c4d700640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6c480b3d40 con 0x7f6c480d1070 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.687+0000 7f6c477fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6c480d2990 0x7f6c480b4540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44720/0 (socket says 172.21.15.44:44720) 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.687+0000 7f6c477fe640 1 -- 172.21.15.44:0/828749320 learned_addr learned my addr 172.21.15.44:0/828749320 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.687+0000 7f6c467fc640 1 --2- 172.21.15.44:0/828749320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c480d1070 0x7f6c4815e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.687+0000 7f6c45ffb640 1 -- 172.21.15.44:0/828749320 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 836584932 0 0) 0x7f6c480b3aa0 con 0x7f6c480d2990 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.687+0000 7f6c45ffb640 1 -- 172.21.15.44:0/828749320 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c30003520 con 0x7f6c480d2990 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.687+0000 7f6c45ffb640 1 -- 172.21.15.44:0/828749320 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2629519325 0 0) 0x7f6c480d2f50 con 0x7f6c480d0c90 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.687+0000 7f6c45ffb640 1 -- 172.21.15.44:0/828749320 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c480b3aa0 con 0x7f6c480d0c90 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.688+0000 7f6c45ffb640 1 -- 172.21.15.44:0/828749320 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1565285200 0 0) 0x7f6c30003520 con 0x7f6c480d2990 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.688+0000 7f6c467fc640 1 -- 172.21.15.44:0/828749320 >> v1:172.21.15.69:6789/0 conn(0x7f6c480d0c90 legacy=0x7f6c480d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.688+0000 7f6c467fc640 1 -- 172.21.15.44:0/828749320 >> v1:172.21.15.44:6789/0 conn(0x7f6c480d2990 legacy=0x7f6c480b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.688+0000 7f6c467fc640 1 -- 172.21.15.44:0/828749320 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c480b4020 con 0x7f6c480d1070 2024-04-02T13:25:21.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.688+0000 7f6c467fc640 1 --2- 172.21.15.44:0/828749320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c480d1070 0x7f6c4815e1b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c0097d0 tx=0x7f6c3c02b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b31612ce9003e014 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.689+0000 7f6c45ffb640 1 -- 172.21.15.44:0/828749320 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c037070 con 0x7f6c480d1070 2024-04-02T13:25:21.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.689+0000 7f6c45ffb640 1 -- 172.21.15.44:0/828749320 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c3c030510 con 0x7f6c480d1070 2024-04-02T13:25:21.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.689+0000 7f6c45ffb640 1 -- 172.21.15.44:0/828749320 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c40001670 con 0x7f6c480d1070 2024-04-02T13:25:21.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.689+0000 7f6c4d700640 1 -- 172.21.15.44:0/828749320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c480d1070 msgr2=0x7f6c4815e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.689+0000 7f6c4d700640 1 --2- 172.21.15.44:0/828749320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c480d1070 0x7f6c4815e1b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c0097d0 tx=0x7f6c3c02b770 comp rx=0 tx=0).stop 2024-04-02T13:25:21.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.689+0000 7f6c4d700640 1 -- 172.21.15.44:0/828749320 shutdown_connections 2024-04-02T13:25:21.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.689+0000 7f6c4d700640 1 --2- 172.21.15.44:0/828749320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c480d1070 0x7f6c4815e1b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.689+0000 7f6c4d700640 1 -- 172.21.15.44:0/828749320 >> 172.21.15.44:0/828749320 conn(0x7f6c4809d3d0 msgr2=0x7f6c480d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.689+0000 7f6c4d700640 1 -- 172.21.15.44:0/828749320 shutdown_connections 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.689+0000 7f6c4d700640 1 -- 172.21.15.44:0/828749320 wait complete. 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.690+0000 7f6c4d700640 1 Processor -- start 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.690+0000 7f6c4d700640 1 -- start start 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.691+0000 7f6c4d700640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c480d0c90 0x7f6c480b5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.691+0000 7f6c4d700640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c480d1070 0x7f6c48159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.691+0000 7f6c4d700640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c480d2990 0x7f6c480ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.691+0000 7f6c4d700640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4815e870 con 0x7f6c480d2990 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.691+0000 7f6c4d700640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4815e6f0 con 0x7f6c480d1070 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.691+0000 7f6c4d700640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6c4815e9f0 con 0x7f6c480d0c90 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.691+0000 7f6c477fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c480d2990 0x7f6c480ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.691+0000 7f6c477fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c480d2990 0x7f6c480ca5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58352/0 (socket says 172.21.15.44:58352) 2024-04-02T13:25:21.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.691+0000 7f6c477fe640 1 -- 172.21.15.44:0/1712918255 learned_addr learned my addr 172.21.15.44:0/1712918255 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.691+0000 7f6c467fc640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c480d1070 0x7f6c48159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.691+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c480d0c90 0x7f6c480b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.692+0000 7f6c477fe640 1 -- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c480d0c90 msgr2=0x7f6c480b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.692+0000 7f6c477fe640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c480d0c90 0x7f6c480b5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.692+0000 7f6c477fe640 1 -- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c480d1070 msgr2=0x7f6c48159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.692+0000 7f6c477fe640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c480d1070 0x7f6c48159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.692+0000 7f6c477fe640 1 -- 172.21.15.44:0/1712918255 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c480b5e90 con 0x7f6c480d2990 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.692+0000 7f6c467fc640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c480d1070 0x7f6c48159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.692+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c480d0c90 0x7f6c480b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.692+0000 7f6c477fe640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c480d2990 0x7f6c480ca5d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000c780 tx=0x7f6c4000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.692+0000 7f6c2f7fe640 1 -- 172.21.15.44:0/1712918255 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c40017070 con 0x7f6c480d2990 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.692+0000 7f6c2f7fe640 1 -- 172.21.15.44:0/1712918255 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c40005ad0 con 0x7f6c480d2990 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.692+0000 7f6c4d700640 1 -- 172.21.15.44:0/1712918255 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c480b48b0 con 0x7f6c480d2990 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.692+0000 7f6c2f7fe640 1 -- 172.21.15.44:0/1712918255 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c40007070 con 0x7f6c480d2990 2024-04-02T13:25:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.692+0000 7f6c4d700640 1 -- 172.21.15.44:0/1712918255 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c480b4e70 con 0x7f6c480d2990 2024-04-02T13:25:21.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.694+0000 7f6c2f7fe640 1 -- 172.21.15.44:0/1712918255 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6c4001a070 con 0x7f6c480d2990 2024-04-02T13:25:21.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.694+0000 7f6c2f7fe640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c3406c920 0x7f6c3406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.694+0000 7f6c2f7fe640 1 -- 172.21.15.44:0/1712918255 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f6c4008b250 con 0x7f6c480d2990 2024-04-02T13:25:21.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.694+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c3406c920 0x7f6c3406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.695+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c3406c920 0x7f6c3406ede0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6c38001560 tx=0x7f6c38009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.695+0000 7f6c503bf740 1 Processor -- start 2024-04-02T13:25:21.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.695+0000 7f6c503bf740 1 -- start start 2024-04-02T13:25:21.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c503bf740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff5234fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c503bf740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 0x55ff5237fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c503bf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff52350330 0x55ff52380120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c503bf740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ff522b9af0 con 0x55ff5237ce50 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c503bf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ff52257ab0 con 0x55ff52350330 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c503bf740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ff522b5aa0 con 0x55ff5237d230 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c46ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff5234fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c46ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff5234fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58364/0 (socket says 172.21.15.44:58364) 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c46ffd640 1 -- 172.21.15.44:0/2484878385 learned_addr learned my addr 172.21.15.44:0/2484878385 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c477fe640 1 --2- 172.21.15.44:0/2484878385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff52350330 0x55ff52380120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c467fc640 1 --2- 172.21.15.44:0/2484878385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 0x55ff5237fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c46ffd640 1 -- 172.21.15.44:0/2484878385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 msgr2=0x55ff5237fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/2484878385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 0x55ff5237fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c46ffd640 1 -- 172.21.15.44:0/2484878385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff52350330 msgr2=0x55ff52380120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/2484878385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff52350330 0x55ff52380120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.696+0000 7f6c46ffd640 1 -- 172.21.15.44:0/2484878385 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff52099a20 con 0x55ff5237ce50 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.697+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/2484878385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff5234fdf0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f6c38003b80 tx=0x7f6c38011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f2b3399fde81d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.697+0000 7f6c2cff9640 1 -- 172.21.15.44:0/2484878385 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c380052e0 con 0x55ff5237ce50 2024-04-02T13:25:21.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.697+0000 7f6c2cff9640 1 -- 172.21.15.44:0/2484878385 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c38005480 con 0x55ff5237ce50 2024-04-02T13:25:21.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.697+0000 7f6c2cff9640 1 -- 172.21.15.44:0/2484878385 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38021450 con 0x55ff5237ce50 2024-04-02T13:25:21.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.697+0000 7f6c503bf740 1 -- 172.21.15.44:0/2484878385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 msgr2=0x55ff5234fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.697+0000 7f6c503bf740 1 --2- 172.21.15.44:0/2484878385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff5234fdf0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f6c38003b80 tx=0x7f6c38011000 comp rx=0 tx=0).stop 2024-04-02T13:25:21.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.697+0000 7f6c503bf740 1 -- 172.21.15.44:0/2484878385 shutdown_connections 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.697+0000 7f6c503bf740 1 --2- 172.21.15.44:0/2484878385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff52350330 0x55ff52380120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.697+0000 7f6c503bf740 1 --2- 172.21.15.44:0/2484878385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 0x55ff5237fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.697+0000 7f6c503bf740 1 --2- 172.21.15.44:0/2484878385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff5234fdf0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.697+0000 7f6c503bf740 1 -- 172.21.15.44:0/2484878385 >> 172.21.15.44:0/2484878385 conn(0x55ff52352060 msgr2=0x55ff5239abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.698+0000 7f6c503bf740 1 -- 172.21.15.44:0/2484878385 shutdown_connections 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.698+0000 7f6c503bf740 1 -- 172.21.15.44:0/2484878385 wait complete. 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.698+0000 7f6c503bf740 1 -- start start 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.698+0000 7f6c503bf740 1 Processor -- start 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c503bf740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff5237ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c503bf740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 0x55ff5237efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c503bf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5236a5f0 0x55ff52378360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c503bf740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ff52354600 con 0x55ff5237ce50 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c503bf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ff52255440 con 0x55ff5236a5f0 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c503bf740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ff523539f0 con 0x55ff5237d230 2024-04-02T13:25:21.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c503bf740 10 client.0 _ll_register_callbacks cb 0x55ff52270a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c46ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff5237ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c46ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff5237ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58370/0 (socket says 172.21.15.44:58370) 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c46ffd640 1 -- 172.21.15.44:0/3106513038 learned_addr learned my addr 172.21.15.44:0/3106513038 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c477fe640 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5236a5f0 0x55ff52378360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c467fc640 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 0x55ff5237efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c46ffd640 1 -- 172.21.15.44:0/3106513038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 msgr2=0x55ff5237efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 0x55ff5237efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c46ffd640 1 -- 172.21.15.44:0/3106513038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5236a5f0 msgr2=0x55ff52378360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5236a5f0 0x55ff52378360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c46ffd640 1 -- 172.21.15.44:0/3106513038 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff51f8d310 con 0x55ff5237ce50 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c467fc640 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 0x55ff5237efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff5237ea60 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800eda0 tx=0x7f6c38013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c358c888664ae82d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.699+0000 7f6c25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:21.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c25ffb640 1 -- 172.21.15.44:0/3106513038 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38021660 con 0x55ff5237ce50 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c25ffb640 1 -- 172.21.15.44:0/3106513038 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f6c38021800 con 0x55ff5237ce50 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c503bf740 1 -- 172.21.15.44:0/3106513038 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c38005480 con 0x55ff5237ce50 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c25ffb640 1 -- 172.21.15.44:0/3106513038 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f6c38033050 con 0x55ff5237ce50 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c503bf740 10 client.7419 fetch_fsmap learned FSMap version 50 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c503bf740 1 -- 172.21.15.44:0/3106513038 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x55ff520fa1b0 con 0x55ff5237ce50 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c25ffb640 1 -- 172.21.15.44:0/3106513038 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c38005480 con 0x55ff5237ce50 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c25ffb640 1 -- 172.21.15.44:0/3106513038 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7f6c38036020 con 0x55ff5237ce50 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c503bf740 10 client.7419 fetch_fsmap finished waiting for FSMap version 50 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c503bf740 10 client.7419 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c503bf740 20 client.7419 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ff523ae9e0 0x55ff523aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c503bf740 4 client.7419 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c503bf740 1 -- 172.21.15.44:0/3106513038 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff522c7030 con 0x55ff523ae9e0 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.700+0000 7f6c467fc640 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ff523ae9e0 0x55ff523aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c25ffb640 10 client.7419 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:21.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c467fc640 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ff523ae9e0 0x55ff523aedc0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e266bef4057d6d8 server_cookie=ae7544016fb52d13 in_seq=0 out_seq=0 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c25ffb640 1 -- 172.21.15.44:0/3106513038 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ff522c7030 con 0x55ff523ae9e0 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c25ffb640 10 client.7419 handle_command_reply: tid=0 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c503bf740 1 client.7419 shutdown 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c503bf740 1 -- 172.21.15.44:0/3106513038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 msgr2=0x55ff5237ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff5237ea60 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800eda0 tx=0x7f6c38013000 comp rx=0 tx=0).stop 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c503bf740 1 -- 172.21.15.44:0/3106513038 shutdown_connections 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ff523ae9e0 0x55ff523aedc0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5236a5f0 0x55ff52378360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 0x55ff5237efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c25ffb640 0 client.7419 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3106513038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff5237ea60 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c503bf740 1 -- 172.21.15.44:0/3106513038 >> 172.21.15.44:0/3106513038 conn(0x55ff52350330 msgr2=0x55ff5239e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c503bf740 1 -- 172.21.15.44:0/3106513038 shutdown_connections 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c503bf740 1 -- 172.21.15.44:0/3106513038 wait complete. 2024-04-02T13:25:21.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.701+0000 7f6c503bf740 20 client.7419 trim_cache size 0 max 16384 2024-04-02T13:25:21.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.719+0000 7f6c503bf740 1 Processor -- start 2024-04-02T13:25:21.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.719+0000 7f6c503bf740 1 -- start start 2024-04-02T13:25:21.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c503bf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5236a5f0 0x55ff523696c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c503bf740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff52369c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c503bf740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 0x55ff52375fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c503bf740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6c48200e60 con 0x55ff5237ce50 2024-04-02T13:25:21.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c503bf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6c481ffe90 con 0x55ff5236a5f0 2024-04-02T13:25:21.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c503bf740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6c4814a3b0 con 0x55ff5237d230 2024-04-02T13:25:21.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c467fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff52369c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c467fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff52369c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58378/0 (socket says 172.21.15.44:58378) 2024-04-02T13:25:21.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c467fc640 1 -- 172.21.15.44:0/3536866205 learned_addr learned my addr 172.21.15.44:0/3536866205 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:21.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c477fe640 1 --2- 172.21.15.44:0/3536866205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 0x55ff52375fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/3536866205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5236a5f0 0x55ff523696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c467fc640 1 -- 172.21.15.44:0/3536866205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 msgr2=0x55ff52375fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c467fc640 1 --2- 172.21.15.44:0/3536866205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 0x55ff52375fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c467fc640 1 -- 172.21.15.44:0/3536866205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5236a5f0 msgr2=0x55ff523696c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c467fc640 1 --2- 172.21.15.44:0/3536866205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5236a5f0 0x55ff523696c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c467fc640 1 -- 172.21.15.44:0/3536866205 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff51f66490 con 0x55ff5237ce50 2024-04-02T13:25:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c467fc640 1 --2- 172.21.15.44:0/3536866205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff52369c00 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c000c00 tx=0x7f6c3c008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8d397a4f3cace0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c27fff640 1 -- 172.21.15.44:0/3536866205 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c037040 con 0x55ff5237ce50 2024-04-02T13:25:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.720+0000 7f6c27fff640 1 -- 172.21.15.44:0/3536866205 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c3c045bc0 con 0x55ff5237ce50 2024-04-02T13:25:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.721+0000 7f6c27fff640 1 -- 172.21.15.44:0/3536866205 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c030980 con 0x55ff5237ce50 2024-04-02T13:25:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.721+0000 7f6c503bf740 1 -- 172.21.15.44:0/3536866205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 msgr2=0x55ff52369c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.721+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3536866205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff52369c00 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c000c00 tx=0x7f6c3c008030 comp rx=0 tx=0).stop 2024-04-02T13:25:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.721+0000 7f6c503bf740 1 -- 172.21.15.44:0/3536866205 shutdown_connections 2024-04-02T13:25:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.721+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3536866205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237d230 0x55ff52375fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.721+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3536866205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5237ce50 0x55ff52369c00 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.721+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3536866205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5236a5f0 0x55ff523696c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.721+0000 7f6c503bf740 1 -- 172.21.15.44:0/3536866205 >> 172.21.15.44:0/3536866205 conn(0x55ff52350330 msgr2=0x55ff52356e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.721+0000 7f6c503bf740 1 -- 172.21.15.44:0/3536866205 shutdown_connections 2024-04-02T13:25:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.721+0000 7f6c503bf740 1 -- 172.21.15.44:0/3536866205 wait complete. 2024-04-02T13:25:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.722+0000 7f6c503bf740 1 -- start start 2024-04-02T13:25:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.722+0000 7f6c503bf740 1 Processor -- start 2024-04-02T13:25:21.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.722+0000 7f6c503bf740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5236a5f0 0x55ff523b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.722+0000 7f6c503bf740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237ce50 0x55ff52382400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.722+0000 7f6c503bf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5237d230 0x55ff5237d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.722+0000 7f6c503bf740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ff52377ed0 con 0x55ff5236a5f0 2024-04-02T13:25:21.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.722+0000 7f6c503bf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4814aa00 con 0x55ff5237d230 2024-04-02T13:25:21.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.722+0000 7f6c503bf740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ff5237f5b0 con 0x55ff5237ce50 2024-04-02T13:25:21.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.722+0000 7f6c503bf740 10 client.0 _ll_register_callbacks cb 0x7f6c40055040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:21.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.722+0000 7f6c46ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5236a5f0 0x55ff523b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.722+0000 7f6c46ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5236a5f0 0x55ff523b0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58382/0 (socket says 172.21.15.44:58382) 2024-04-02T13:25:21.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.722+0000 7f6c46ffd640 1 -- 172.21.15.44:0/3937832645 learned_addr learned my addr 172.21.15.44:0/3937832645 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:21.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.722+0000 7f6c477fe640 1 --2- 172.21.15.44:0/3937832645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5237d230 0x55ff5237d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c467fc640 1 --2- 172.21.15.44:0/3937832645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237ce50 0x55ff52382400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c46ffd640 1 -- 172.21.15.44:0/3937832645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237ce50 msgr2=0x55ff52382400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/3937832645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237ce50 0x55ff52382400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c46ffd640 1 -- 172.21.15.44:0/3937832645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5237d230 msgr2=0x55ff5237d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/3937832645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5237d230 0x55ff5237d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c46ffd640 1 -- 172.21.15.44:0/3937832645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff523b4fa0 con 0x55ff5236a5f0 2024-04-02T13:25:21.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c46ffd640 1 --2- 172.21.15.44:0/3937832645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5236a5f0 0x55ff523b0120 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800eda0 tx=0x7f6c38001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c238a20e9f4f9b67 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:21.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c25ffb640 1 -- 172.21.15.44:0/3937832645 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38005480 con 0x55ff5236a5f0 2024-04-02T13:25:21.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c25ffb640 1 -- 172.21.15.44:0/3937832645 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f6c38038c80 con 0x55ff5236a5f0 2024-04-02T13:25:21.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c503bf740 1 -- 172.21.15.44:0/3937832645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c3c045bc0 con 0x55ff5236a5f0 2024-04-02T13:25:21.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c25ffb640 1 -- 172.21.15.44:0/3937832645 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f6c3802b460 con 0x55ff5236a5f0 2024-04-02T13:25:21.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c503bf740 10 client.7425 fetch_fsmap learned FSMap version 50 2024-04-02T13:25:21.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c503bf740 1 -- 172.21.15.44:0/3937832645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x55ff52381c60 con 0x55ff5236a5f0 2024-04-02T13:25:21.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c25ffb640 1 -- 172.21.15.44:0/3937832645 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c3c045bc0 con 0x55ff5236a5f0 2024-04-02T13:25:21.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c25ffb640 1 -- 172.21.15.44:0/3937832645 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7f6c38036040 con 0x55ff5236a5f0 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c503bf740 10 client.7425 fetch_fsmap finished waiting for FSMap version 50 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c503bf740 10 client.7425 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c503bf740 20 client.7425 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3937832645 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ff52417810 0x55ff5237db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c503bf740 4 client.7425 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["recursive"]}] 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.723+0000 7f6c503bf740 1 -- 172.21.15.44:0/3937832645 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["recursive"]}) -- 0x55ff5237e600 con 0x55ff52417810 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.724+0000 7f6c467fc640 1 --2- 172.21.15.44:0/3937832645 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ff52417810 0x55ff5237db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.724+0000 7f6c25ffb640 10 client.7425 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.724+0000 7f6c467fc640 1 --2- 172.21.15.44:0/3937832645 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ff52417810 0x55ff5237db50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eca21fc4dd22c2cf server_cookie=5edb24e80f3d0c9b in_seq=0 out_seq=0 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.724+0000 7f6c25ffb640 1 -- 172.21.15.44:0/3937832645 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ff5237e600 con 0x55ff52417810 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.724+0000 7f6c25ffb640 10 client.7425 handle_command_reply: tid=0 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.724+0000 7f6c503bf740 1 client.7425 shutdown 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.724+0000 7f6c503bf740 1 -- 172.21.15.44:0/3937832645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5236a5f0 msgr2=0x55ff523b0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.724+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3937832645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5236a5f0 0x55ff523b0120 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800eda0 tx=0x7f6c38001250 comp rx=0 tx=0).stop 2024-04-02T13:25:21.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c503bf740 1 -- 172.21.15.44:0/3937832645 shutdown_connections 2024-04-02T13:25:21.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3937832645 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ff52417810 0x55ff5237db50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3937832645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ff5237d230 0x55ff5237d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3937832645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ff5237ce50 0x55ff52382400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c503bf740 1 --2- 172.21.15.44:0/3937832645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ff5236a5f0 0x55ff523b0120 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c503bf740 1 -- 172.21.15.44:0/3937832645 >> 172.21.15.44:0/3937832645 conn(0x55ff52350330 msgr2=0x55ff523696c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c25ffb640 0 client.7425 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:21.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c503bf740 1 -- 172.21.15.44:0/3937832645 shutdown_connections 2024-04-02T13:25:21.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c503bf740 1 -- 172.21.15.44:0/3937832645 wait complete. 2024-04-02T13:25:21.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c503bf740 20 client.7425 trim_cache size 0 max 16384 2024-04-02T13:25:21.734 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:25:21.735 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-02T13:25:21.735 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "f6de806e-8ce9-4003-8ece-6d64e6938f15", 2024-04-02T13:25:21.735 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-02T13:25:21.735 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:25:21.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c4d700640 1 -- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c3406c920 msgr2=0x7f6c3406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c4d700640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c3406c920 0x7f6c3406ede0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6c38001560 tx=0x7f6c38009040 comp rx=0 tx=0).stop 2024-04-02T13:25:21.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c4d700640 1 -- 172.21.15.44:0/1712918255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c480d2990 msgr2=0x7f6c480ca5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c4d700640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c480d2990 0x7f6c480ca5d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000c780 tx=0x7f6c4000cc40 comp rx=0 tx=0).stop 2024-04-02T13:25:21.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c4d700640 1 -- 172.21.15.44:0/1712918255 shutdown_connections 2024-04-02T13:25:21.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c4d700640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c3406c920 0x7f6c3406ede0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c4d700640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c480d2990 0x7f6c480ca5d0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c4d700640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c480d1070 0x7f6c48159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c4d700640 1 --2- 172.21.15.44:0/1712918255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c480d0c90 0x7f6c480b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c4d700640 1 -- 172.21.15.44:0/1712918255 >> 172.21.15.44:0/1712918255 conn(0x7f6c4809d3d0 msgr2=0x7f6c480b17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c4d700640 1 -- 172.21.15.44:0/1712918255 shutdown_connections 2024-04-02T13:25:21.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.725+0000 7f6c4d700640 1 -- 172.21.15.44:0/1712918255 wait complete. 2024-04-02T13:25:21.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:21.739 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-04-02T13:25:21.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.840+0000 7fba5d550640 1 Processor -- start 2024-04-02T13:25:21.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.840+0000 7fba5d550640 1 -- start start 2024-04-02T13:25:21.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.841+0000 7fba5d550640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fba58072030 0x7fba5813f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.841+0000 7fba5d550640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba5806e050 con 0x7fba5806f3a0 2024-04-02T13:25:21.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.841+0000 7fba5d550640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba5806e300 con 0x7fba5813ef60 2024-04-02T13:25:21.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.841+0000 7fba5d550640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fba5806e5e0 con 0x7fba58072030 2024-04-02T13:25:21.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.841+0000 7fba577fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fba58072030 0x7fba5813f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.841+0000 7fba4ffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fba5806f3a0 0x7fba58071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44722/0 (socket says 172.21.15.44:44722) 2024-04-02T13:25:21.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.841+0000 7fba4ffff640 1 -- 172.21.15.44:0/1307408899 learned_addr learned my addr 172.21.15.44:0/1307408899 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:21.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.841+0000 7fba577fe640 1 -- 172.21.15.44:0/1307408899 >> v1:172.21.15.69:6789/0 conn(0x7fba5813ef60 legacy=0x7fba5806d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.841+0000 7fba577fe640 1 -- 172.21.15.44:0/1307408899 >> v1:172.21.15.44:6789/0 conn(0x7fba5806f3a0 legacy=0x7fba58071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.841+0000 7fba577fe640 1 -- 172.21.15.44:0/1307408899 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba5806e8c0 con 0x7fba58072030 2024-04-02T13:25:21.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.841+0000 7fba567fc640 1 -- 172.21.15.44:0/1307408899 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1459529674 0 0) 0x7fba5806e050 con 0x7fba5806f3a0 2024-04-02T13:25:21.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.842+0000 7fba577fe640 1 --2- 172.21.15.44:0/1307408899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba58072030 0x7fba5813f6f0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fba50009d90 tx=0x7fba5002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13720e917c6d0028 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.842+0000 7fba567fc640 1 -- 172.21.15.44:0/1307408899 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba50039070 con 0x7fba58072030 2024-04-02T13:25:21.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.842+0000 7fba567fc640 1 -- 172.21.15.44:0/1307408899 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba5002cc60 con 0x7fba58072030 2024-04-02T13:25:21.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.842+0000 7fba567fc640 1 -- 172.21.15.44:0/1307408899 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba50034660 con 0x7fba58072030 2024-04-02T13:25:21.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.843+0000 7fba5d550640 1 -- 172.21.15.44:0/1307408899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba58072030 msgr2=0x7fba5813f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.843+0000 7fba5d550640 1 --2- 172.21.15.44:0/1307408899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba58072030 0x7fba5813f6f0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fba50009d90 tx=0x7fba5002beb0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.843+0000 7fba5d550640 1 -- 172.21.15.44:0/1307408899 shutdown_connections 2024-04-02T13:25:21.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.843+0000 7fba5d550640 1 --2- 172.21.15.44:0/1307408899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba58072030 0x7fba5813f6f0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.843+0000 7fba5d550640 1 -- 172.21.15.44:0/1307408899 >> 172.21.15.44:0/1307408899 conn(0x7fba580b5d90 msgr2=0x7fba5806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.843+0000 7fba5d550640 1 -- 172.21.15.44:0/1307408899 shutdown_connections 2024-04-02T13:25:21.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.843+0000 7fba5d550640 1 -- 172.21.15.44:0/1307408899 wait complete. 2024-04-02T13:25:21.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.844+0000 7fba5d550640 1 Processor -- start 2024-04-02T13:25:21.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.844+0000 7fba5d550640 1 -- start start 2024-04-02T13:25:21.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.844+0000 7fba5d550640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba5806f3a0 0x7fba580a9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba5d550640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba58072030 0x7fba58139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba5d550640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5813ef60 0x7fba581f50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba5d550640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fba580725a0 con 0x7fba5813ef60 2024-04-02T13:25:21.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba5d550640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fba5813f340 con 0x7fba5806f3a0 2024-04-02T13:25:21.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba5d550640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fba58072720 con 0x7fba58072030 2024-04-02T13:25:21.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5813ef60 0x7fba581f50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5813ef60 0x7fba581f50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58400/0 (socket says 172.21.15.44:58400) 2024-04-02T13:25:21.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba577fe640 1 -- 172.21.15.44:0/1698818794 learned_addr learned my addr 172.21.15.44:0/1698818794 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:21.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba56ffd640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba5806f3a0 0x7fba580a9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba4ffff640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba58072030 0x7fba58139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba577fe640 1 -- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba58072030 msgr2=0x7fba58139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba577fe640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba58072030 0x7fba58139ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba577fe640 1 -- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba5806f3a0 msgr2=0x7fba580a9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba577fe640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba5806f3a0 0x7fba580a9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba577fe640 1 -- 172.21.15.44:0/1698818794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba580aa2f0 con 0x7fba5813ef60 2024-04-02T13:25:21.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba577fe640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5813ef60 0x7fba581f50f0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fba50037040 tx=0x7fba50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba4f7fe640 1 -- 172.21.15.44:0/1698818794 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba50039050 con 0x7fba5813ef60 2024-04-02T13:25:21.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba4f7fe640 1 -- 172.21.15.44:0/1698818794 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba5003a040 con 0x7fba5813ef60 2024-04-02T13:25:21.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba5d550640 1 -- 172.21.15.44:0/1698818794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba58158ac0 con 0x7fba5813ef60 2024-04-02T13:25:21.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba4f7fe640 1 -- 172.21.15.44:0/1698818794 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5003e610 con 0x7fba5813ef60 2024-04-02T13:25:21.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.845+0000 7fba5d550640 1 -- 172.21.15.44:0/1698818794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba58159060 con 0x7fba5813ef60 2024-04-02T13:25:21.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.847+0000 7fba4f7fe640 1 -- 172.21.15.44:0/1698818794 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fba50047020 con 0x7fba5813ef60 2024-04-02T13:25:21.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.847+0000 7fba4f7fe640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fba3806c8f0 0x7fba3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.847+0000 7fba4f7fe640 1 -- 172.21.15.44:0/1698818794 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fba500ae140 con 0x7fba5813ef60 2024-04-02T13:25:21.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.848+0000 7fba56ffd640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fba3806c8f0 0x7fba3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.848+0000 7fba56ffd640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fba3806c8f0 0x7fba3806edb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fba48000f60 tx=0x7fba48009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.848+0000 7fba60211740 1 Processor -- start 2024-04-02T13:25:21.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.849+0000 7fba60211740 1 -- start start 2024-04-02T13:25:21.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.849+0000 7fba60211740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676bb9a0 0x564e6768e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.849+0000 7fba60211740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bbd80 0x564e676be730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.849+0000 7fba60211740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e6768ee80 0x564e676bec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.849+0000 7fba60211740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564e675f8830 con 0x564e676bbd80 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.849+0000 7fba60211740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564e675a0820 con 0x564e676bb9a0 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.849+0000 7fba60211740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564e675f47e0 con 0x564e6768ee80 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.849+0000 7fba4ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bbd80 0x564e676be730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.849+0000 7fba4ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bbd80 0x564e676be730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58402/0 (socket says 172.21.15.44:58402) 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.849+0000 7fba4ffff640 1 -- 172.21.15.44:0/1325400200 learned_addr learned my addr 172.21.15.44:0/1325400200 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.849+0000 7fba56ffd640 1 --2- 172.21.15.44:0/1325400200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676bb9a0 0x564e6768e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.849+0000 7fba577fe640 1 --2- 172.21.15.44:0/1325400200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e6768ee80 0x564e676bec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.850+0000 7fba4ffff640 1 -- 172.21.15.44:0/1325400200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e6768ee80 msgr2=0x564e676bec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.850+0000 7fba4ffff640 1 --2- 172.21.15.44:0/1325400200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e6768ee80 0x564e676bec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.850+0000 7fba4ffff640 1 -- 172.21.15.44:0/1325400200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676bb9a0 msgr2=0x564e6768e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.850+0000 7fba4ffff640 1 --2- 172.21.15.44:0/1325400200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676bb9a0 0x564e6768e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.850+0000 7fba4ffff640 1 -- 172.21.15.44:0/1325400200 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e673d67d0 con 0x564e676bbd80 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.850+0000 7fba4ffff640 1 --2- 172.21.15.44:0/1325400200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bbd80 0x564e676be730 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fba400073c0 tx=0x7fba40005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9adfa6033a5fe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.850+0000 7fba4cff9640 1 -- 172.21.15.44:0/1325400200 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba40019070 con 0x564e676bbd80 2024-04-02T13:25:21.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.850+0000 7fba4cff9640 1 -- 172.21.15.44:0/1325400200 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba40005e10 con 0x564e676bbd80 2024-04-02T13:25:21.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.850+0000 7fba4cff9640 1 -- 172.21.15.44:0/1325400200 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba40001ad0 con 0x564e676bbd80 2024-04-02T13:25:21.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.851+0000 7fba60211740 1 -- 172.21.15.44:0/1325400200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bbd80 msgr2=0x564e676be730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.851+0000 7fba60211740 1 --2- 172.21.15.44:0/1325400200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bbd80 0x564e676be730 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fba400073c0 tx=0x7fba40005bc0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.851+0000 7fba60211740 1 -- 172.21.15.44:0/1325400200 shutdown_connections 2024-04-02T13:25:21.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.851+0000 7fba60211740 1 --2- 172.21.15.44:0/1325400200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e6768ee80 0x564e676bec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.851+0000 7fba60211740 1 --2- 172.21.15.44:0/1325400200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bbd80 0x564e676be730 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.851+0000 7fba60211740 1 --2- 172.21.15.44:0/1325400200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676bb9a0 0x564e6768e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.851+0000 7fba60211740 1 -- 172.21.15.44:0/1325400200 >> 172.21.15.44:0/1325400200 conn(0x564e67690bb0 msgr2=0x564e676d9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.851+0000 7fba60211740 1 -- 172.21.15.44:0/1325400200 shutdown_connections 2024-04-02T13:25:21.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.851+0000 7fba60211740 1 -- 172.21.15.44:0/1325400200 wait complete. 2024-04-02T13:25:21.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.851+0000 7fba60211740 1 -- start start 2024-04-02T13:25:21.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.852+0000 7fba60211740 1 Processor -- start 2024-04-02T13:25:21.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.852+0000 7fba60211740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e676d29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.852+0000 7fba60211740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 0x564e676acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.852+0000 7fba60211740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 0x564e676d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.852+0000 7fba60211740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564e67693150 con 0x564e676bb9a0 2024-04-02T13:25:21.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.852+0000 7fba60211740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564e675b3d30 con 0x564e676d12a0 2024-04-02T13:25:21.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.852+0000 7fba60211740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564e67692540 con 0x564e676bbd80 2024-04-02T13:25:21.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.852+0000 7fba60211740 10 client.0 _ll_register_callbacks cb 0x564e675b3720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:21.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.852+0000 7fba56ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e676d29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.852+0000 7fba56ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e676d29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58414/0 (socket says 172.21.15.44:58414) 2024-04-02T13:25:21.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.852+0000 7fba56ffd640 1 -- 172.21.15.44:0/3440930794 learned_addr learned my addr 172.21.15.44:0/3440930794 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:21.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba577fe640 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 0x564e676d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba4ffff640 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 0x564e676acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba56ffd640 1 -- 172.21.15.44:0/3440930794 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 msgr2=0x564e676acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba56ffd640 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 0x564e676acbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba56ffd640 1 -- 172.21.15.44:0/3440930794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 msgr2=0x564e676d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba56ffd640 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 0x564e676d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba56ffd640 1 -- 172.21.15.44:0/3440930794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e6730c3f0 con 0x564e676bb9a0 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba577fe640 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 0x564e676d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba56ffd640 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e676d29b0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fba48003b70 tx=0x7fba48001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4249057c8728e01 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba35ffb640 1 -- 172.21.15.44:0/3440930794 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba48028060 con 0x564e676bb9a0 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba35ffb640 1 -- 172.21.15.44:0/3440930794 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fba48010040 con 0x564e676bb9a0 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba60211740 1 -- 172.21.15.44:0/3440930794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba40005e10 con 0x564e676bb9a0 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba35ffb640 1 -- 172.21.15.44:0/3440930794 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fba4800ed00 con 0x564e676bb9a0 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba60211740 10 client.7440 fetch_fsmap learned FSMap version 50 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba60211740 1 -- 172.21.15.44:0/3440930794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x564e672cc0f0 con 0x564e676bb9a0 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba35ffb640 1 -- 172.21.15.44:0/3440930794 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba40005e10 con 0x564e676bb9a0 2024-04-02T13:25:21.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba35ffb640 1 -- 172.21.15.44:0/3440930794 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7fba48013420 con 0x564e676bb9a0 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.853+0000 7fba60211740 10 client.7440 fetch_fsmap finished waiting for FSMap version 50 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.854+0000 7fba60211740 10 client.7440 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.854+0000 7fba60211740 20 client.7440 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.854+0000 7fba60211740 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x564e676c1150 0x564e676f7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.854+0000 7fba60211740 4 client.7440 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.854+0000 7fba60211740 1 -- 172.21.15.44:0/3440930794 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e67605d70 con 0x564e676c1150 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.854+0000 7fba4ffff640 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x564e676c1150 0x564e676f7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.854+0000 7fba4ffff640 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x564e676c1150 0x564e676f7090 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15d03a07dbe70f2 server_cookie=3fddd1db7882c0c2 in_seq=0 out_seq=0 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.854+0000 7fba35ffb640 10 client.7440 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.854+0000 7fba35ffb640 1 -- 172.21.15.44:0/3440930794 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e67605d70 con 0x564e676c1150 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.854+0000 7fba35ffb640 10 client.7440 handle_command_reply: tid=0 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.855+0000 7fba60211740 1 client.7440 shutdown 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.855+0000 7fba60211740 1 -- 172.21.15.44:0/3440930794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 msgr2=0x564e676d29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.855+0000 7fba60211740 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e676d29b0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fba48003b70 tx=0x7fba48001080 comp rx=0 tx=0).stop 2024-04-02T13:25:21.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.855+0000 7fba60211740 1 -- 172.21.15.44:0/3440930794 shutdown_connections 2024-04-02T13:25:21.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.855+0000 7fba60211740 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x564e676c1150 0x564e676f7090 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.855+0000 7fba60211740 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 0x564e676d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.855+0000 7fba60211740 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 0x564e676acbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.855+0000 7fba60211740 1 --2- 172.21.15.44:0/3440930794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e676d29b0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.855+0000 7fba35ffb640 0 client.7440 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:21.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.855+0000 7fba60211740 1 -- 172.21.15.44:0/3440930794 >> 172.21.15.44:0/3440930794 conn(0x564e6768ee80 msgr2=0x564e676be720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.855+0000 7fba60211740 1 -- 172.21.15.44:0/3440930794 shutdown_connections 2024-04-02T13:25:21.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.855+0000 7fba60211740 1 -- 172.21.15.44:0/3440930794 wait complete. 2024-04-02T13:25:21.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.855+0000 7fba60211740 20 client.7440 trim_cache size 0 max 16384 2024-04-02T13:25:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.872+0000 7fba60211740 1 Processor -- start 2024-04-02T13:25:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.872+0000 7fba60211740 1 -- start start 2024-04-02T13:25:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba60211740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e676be720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba60211740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 0x564e676bec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba60211740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 0x564e676d00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba60211740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fba581febf0 con 0x564e676bb9a0 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba60211740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fba581fdc10 con 0x564e676d12a0 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba60211740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fba58157b00 con 0x564e676bbd80 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba56ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e676be720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba56ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e676be720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58422/0 (socket says 172.21.15.44:58422) 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba56ffd640 1 -- 172.21.15.44:0/2494294434 learned_addr learned my addr 172.21.15.44:0/2494294434 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba56ffd640 1 -- 172.21.15.44:0/2494294434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 msgr2=0x564e676bec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba56ffd640 1 --2- 172.21.15.44:0/2494294434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 0x564e676bec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba56ffd640 1 -- 172.21.15.44:0/2494294434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 msgr2=0x564e676d00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba56ffd640 1 --2- 172.21.15.44:0/2494294434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 0x564e676d00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.873+0000 7fba56ffd640 1 -- 172.21.15.44:0/2494294434 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e6768ac50 con 0x564e676bb9a0 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.874+0000 7fba56ffd640 1 --2- 172.21.15.44:0/2494294434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e676be720 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fba48001430 tx=0x7fba4801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71267a9fe5e10e8f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.874+0000 7fba37fff640 1 -- 172.21.15.44:0/2494294434 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba480052a0 con 0x564e676bb9a0 2024-04-02T13:25:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.874+0000 7fba37fff640 1 -- 172.21.15.44:0/2494294434 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba48019070 con 0x564e676bb9a0 2024-04-02T13:25:21.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.874+0000 7fba37fff640 1 -- 172.21.15.44:0/2494294434 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba48013420 con 0x564e676bb9a0 2024-04-02T13:25:21.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.874+0000 7fba60211740 1 -- 172.21.15.44:0/2494294434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 msgr2=0x564e676be720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.874+0000 7fba60211740 1 --2- 172.21.15.44:0/2494294434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e676be720 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fba48001430 tx=0x7fba4801d000 comp rx=0 tx=0).stop 2024-04-02T13:25:21.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.874+0000 7fba60211740 1 -- 172.21.15.44:0/2494294434 shutdown_connections 2024-04-02T13:25:21.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.874+0000 7fba60211740 1 --2- 172.21.15.44:0/2494294434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 0x564e676d00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.874+0000 7fba60211740 1 --2- 172.21.15.44:0/2494294434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 0x564e676bec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.874+0000 7fba60211740 1 --2- 172.21.15.44:0/2494294434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e676be720 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.874+0000 7fba60211740 1 -- 172.21.15.44:0/2494294434 >> 172.21.15.44:0/2494294434 conn(0x564e6768ee80 msgr2=0x564e676e8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.874+0000 7fba60211740 1 -- 172.21.15.44:0/2494294434 shutdown_connections 2024-04-02T13:25:21.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.874+0000 7fba60211740 1 -- 172.21.15.44:0/2494294434 wait complete. 2024-04-02T13:25:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.875+0000 7fba60211740 1 -- start start 2024-04-02T13:25:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.875+0000 7fba60211740 1 Processor -- start 2024-04-02T13:25:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba60211740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e67752730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba60211740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 0x564e676fbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba60211740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 0x564e677532d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba60211740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564e676d3b00 con 0x564e676bb9a0 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba60211740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fba581fd5f0 con 0x564e676d12a0 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba60211740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564e676d3630 con 0x564e676bbd80 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba60211740 10 client.0 _ll_register_callbacks cb 0x7fba48003820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba56ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e67752730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba56ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e67752730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58426/0 (socket says 172.21.15.44:58426) 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba56ffd640 1 -- 172.21.15.44:0/2925222529 learned_addr learned my addr 172.21.15.44:0/2925222529 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba4ffff640 1 --2- 172.21.15.44:0/2925222529 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 0x564e676fbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba577fe640 1 --2- 172.21.15.44:0/2925222529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 0x564e677532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba56ffd640 1 -- 172.21.15.44:0/2925222529 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 msgr2=0x564e676fbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba56ffd640 1 --2- 172.21.15.44:0/2925222529 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 0x564e676fbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba56ffd640 1 -- 172.21.15.44:0/2925222529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 msgr2=0x564e677532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba56ffd640 1 --2- 172.21.15.44:0/2925222529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 0x564e677532d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba56ffd640 1 -- 172.21.15.44:0/2925222529 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e676d9e70 con 0x564e676bb9a0 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba56ffd640 1 --2- 172.21.15.44:0/2925222529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e67752730 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fba48003b70 tx=0x7fba48020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35aa35cc1cbbab50 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba35ffb640 1 -- 172.21.15.44:0/2925222529 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4803a070 con 0x564e676bb9a0 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba35ffb640 1 -- 172.21.15.44:0/2925222529 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fba48013db0 con 0x564e676bb9a0 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba60211740 1 -- 172.21.15.44:0/2925222529 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba48019070 con 0x564e676bb9a0 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba35ffb640 1 -- 172.21.15.44:0/2925222529 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fba48045050 con 0x564e676bb9a0 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba60211740 10 client.7446 fetch_fsmap learned FSMap version 50 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.876+0000 7fba60211740 1 -- 172.21.15.44:0/2925222529 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x564e676d9960 con 0x564e676bb9a0 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba35ffb640 1 -- 172.21.15.44:0/2925222529 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba48019070 con 0x564e676bb9a0 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba35ffb640 1 -- 172.21.15.44:0/2925222529 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7fba48048020 con 0x564e676bb9a0 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 10 client.7446 fetch_fsmap finished waiting for FSMap version 50 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 10 client.7446 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 20 client.7446 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 1 --2- 172.21.15.44:0/2925222529 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x564e676bf010 0x564e676cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 4 client.7446 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:25:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 1 -- 172.21.15.44:0/2925222529 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564e676000c0 con 0x564e676bf010 2024-04-02T13:25:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba4ffff640 1 --2- 172.21.15.44:0/2925222529 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x564e676bf010 0x564e676cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba4ffff640 1 --2- 172.21.15.44:0/2925222529 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x564e676bf010 0x564e676cfc20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=755c31ae9a146891 server_cookie=90e88ca9e3360753 in_seq=0 out_seq=0 2024-04-02T13:25:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba35ffb640 10 client.7446 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba35ffb640 1 -- 172.21.15.44:0/2925222529 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564e676000c0 con 0x564e676bf010 2024-04-02T13:25:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba35ffb640 10 client.7446 handle_command_reply: tid=0 2024-04-02T13:25:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 1 client.7446 shutdown 2024-04-02T13:25:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 1 -- 172.21.15.44:0/2925222529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 msgr2=0x564e67752730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 1 --2- 172.21.15.44:0/2925222529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e67752730 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fba48003b70 tx=0x7fba48020000 comp rx=0 tx=0).stop 2024-04-02T13:25:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 1 -- 172.21.15.44:0/2925222529 shutdown_connections 2024-04-02T13:25:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 1 --2- 172.21.15.44:0/2925222529 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x564e676bf010 0x564e676cfc20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 1 --2- 172.21.15.44:0/2925222529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564e676d12a0 0x564e677532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 1 --2- 172.21.15.44:0/2925222529 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564e676bbd80 0x564e676fbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 1 --2- 172.21.15.44:0/2925222529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564e676bb9a0 0x564e67752730 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba35ffb640 0 client.7446 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 1 -- 172.21.15.44:0/2925222529 >> 172.21.15.44:0/2925222529 conn(0x564e6768ee80 msgr2=0x564e676ac9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.877+0000 7fba60211740 1 -- 172.21.15.44:0/2925222529 shutdown_connections 2024-04-02T13:25:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba60211740 1 -- 172.21.15.44:0/2925222529 wait complete. 2024-04-02T13:25:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba60211740 20 client.7446 trim_cache size 0 max 16384 2024-04-02T13:25:21.887 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:25:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T13:25:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-02T13:25:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: "f6de806e-8ce9-4003-8ece-6d64e6938f15": { 2024-04-02T13:25:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/subdir", 2024-04-02T13:25:21.888 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "f6de806e-8ce9-4003-8ece-6d64e6938f15", 2024-04-02T13:25:21.888 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive" 2024-04-02T13:25:21.888 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:25:21.888 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:25:21.888 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:25:21.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba5d550640 1 -- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fba3806c8f0 msgr2=0x7fba3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba5d550640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fba3806c8f0 0x7fba3806edb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fba48000f60 tx=0x7fba48009040 comp rx=0 tx=0).stop 2024-04-02T13:25:21.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba5d550640 1 -- 172.21.15.44:0/1698818794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5813ef60 msgr2=0x7fba581f50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:21.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba5d550640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5813ef60 0x7fba581f50f0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fba50037040 tx=0x7fba50004a40 comp rx=0 tx=0).stop 2024-04-02T13:25:21.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba5d550640 1 -- 172.21.15.44:0/1698818794 shutdown_connections 2024-04-02T13:25:21.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba5d550640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fba3806c8f0 0x7fba3806edb0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba5d550640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5813ef60 0x7fba581f50f0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba5d550640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba58072030 0x7fba58139ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba5d550640 1 --2- 172.21.15.44:0/1698818794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba5806f3a0 0x7fba580a9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:21.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba5d550640 1 -- 172.21.15.44:0/1698818794 >> 172.21.15.44:0/1698818794 conn(0x7fba580b5d90 msgr2=0x7fba5806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:21.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba5d550640 1 -- 172.21.15.44:0/1698818794 shutdown_connections 2024-04-02T13:25:21.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:21.878+0000 7fba5d550640 1 -- 172.21.15.44:0/1698818794 wait complete. 2024-04-02T13:25:21.892 INFO:tasks.cephfs.filesystem:scrub status for tag:f6de806e-8ce9-4003-8ece-6d64e6938f15 - {'path': '/subdir', 'tag': 'f6de806e-8ce9-4003-8ece-6d64e6938f15', 'options': 'recursive'} 2024-04-02T13:25:21.969 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:25:21.967+0000 7fe12a6e8640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-02T13:25:22.586 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:25:22.590 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:25:51.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:51.893 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-04-02T13:25:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.007+0000 7f0aa3c55640 1 Processor -- start 2024-04-02T13:25:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.008+0000 7f0aa3c55640 1 -- start start 2024-04-02T13:25:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.008+0000 7f0aa3c55640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0a9c15e470 0x7f0a9c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.008+0000 7f0aa3c55640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0a9c0ce790 0x7f0a9c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.008+0000 7f0aa3c55640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a9c0ced00 con 0x7f0a9c0b1540 2024-04-02T13:25:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.008+0000 7f0aa3c55640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0a9c0cefe0 con 0x7f0a9c0ce790 2024-04-02T13:25:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.008+0000 7f0aa3c55640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0a9c0cf2c0 con 0x7f0a9c15e470 2024-04-02T13:25:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.009+0000 7f0aa21f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0a9c0ce790 0x7f0a9c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.009+0000 7f0aa11f0640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0a9c0b1540 0x7f0a9c0d1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53010/0 (socket says 172.21.15.44:53010) 2024-04-02T13:25:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.009+0000 7f0aa11f0640 1 -- 172.21.15.44:0/308436135 learned_addr learned my addr 172.21.15.44:0/308436135 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.009+0000 7f0aa19f1640 1 --2- 172.21.15.44:0/308436135 >> v2:172.21.15.69:3300/0 conn(0x7f0a9c15e470 0x7f0a9c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.009+0000 7f0aa21f2640 1 -- 172.21.15.44:0/308436135 >> v2:172.21.15.69:3300/0 conn(0x7f0a9c15e470 msgr2=0x7f0a9c0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.009+0000 7f0aa21f2640 1 --2- 172.21.15.44:0/308436135 >> v2:172.21.15.69:3300/0 conn(0x7f0a9c15e470 0x7f0a9c0b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.009+0000 7f0aa21f2640 1 -- 172.21.15.44:0/308436135 >> v1:172.21.15.44:6789/0 conn(0x7f0a9c0b1540 legacy=0x7f0a9c0d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.009+0000 7f0aa21f2640 1 -- 172.21.15.44:0/308436135 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9c0cf5a0 con 0x7f0a9c0ce790 2024-04-02T13:25:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.010+0000 7f0aa21f2640 1 --2- 172.21.15.44:0/308436135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a9c0ce790 0x7f0a9c06d0f0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c06f1c0 tx=0x7f0a9402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7fcf56e40f4949 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.010+0000 7f0aa09ef640 1 -- 172.21.15.44:0/308436135 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94039070 con 0x7f0a9c0ce790 2024-04-02T13:25:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.010+0000 7f0aa09ef640 1 -- 172.21.15.44:0/308436135 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a9402cbf0 con 0x7f0a9c0ce790 2024-04-02T13:25:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.010+0000 7f0aa09ef640 1 -- 172.21.15.44:0/308436135 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94034440 con 0x7f0a9c0ce790 2024-04-02T13:25:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.010+0000 7f0aa3c55640 1 -- 172.21.15.44:0/308436135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a9c0ce790 msgr2=0x7f0a9c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.010+0000 7f0aa3c55640 1 --2- 172.21.15.44:0/308436135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a9c0ce790 0x7f0a9c06d0f0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c06f1c0 tx=0x7f0a9402be40 comp rx=0 tx=0).stop 2024-04-02T13:25:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.010+0000 7f0aa3c55640 1 -- 172.21.15.44:0/308436135 shutdown_connections 2024-04-02T13:25:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.010+0000 7f0aa3c55640 1 --2- 172.21.15.44:0/308436135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a9c0ce790 0x7f0a9c06d0f0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.010+0000 7f0aa3c55640 1 --2- 172.21.15.44:0/308436135 >> v2:172.21.15.69:3300/0 conn(0x7f0a9c15e470 0x7f0a9c0b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.010+0000 7f0aa3c55640 1 -- 172.21.15.44:0/308436135 >> 172.21.15.44:0/308436135 conn(0x7f0a9c09aa20 msgr2=0x7f0a9c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.011+0000 7f0aa3c55640 1 -- 172.21.15.44:0/308436135 shutdown_connections 2024-04-02T13:25:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.011+0000 7f0aa3c55640 1 -- 172.21.15.44:0/308436135 wait complete. 2024-04-02T13:25:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.012+0000 7f0aa3c55640 1 Processor -- start 2024-04-02T13:25:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.012+0000 7f0aa3c55640 1 -- start start 2024-04-02T13:25:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.012+0000 7f0aa3c55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c0b1540 0x7f0a9c0b7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.012+0000 7f0aa3c55640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a9c0ce790 0x7f0a9c161f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.012+0000 7f0aa3c55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9c15e470 0x7f0a9c162480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.012+0000 7f0aa3c55640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c0d22c0 con 0x7f0a9c0ce790 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.012+0000 7f0aa3c55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c06cb60 con 0x7f0a9c15e470 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.012+0000 7f0aa3c55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0a9c0d2440 con 0x7f0a9c0b1540 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0aa11f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a9c0ce790 0x7f0a9c161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0aa11f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a9c0ce790 0x7f0a9c161f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60916/0 (socket says 172.21.15.44:60916) 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0aa11f0640 1 -- 172.21.15.44:0/1400277718 learned_addr learned my addr 172.21.15.44:0/1400277718 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0aa21f2640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9c15e470 0x7f0a9c162480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0aa11f0640 1 -- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c0b1540 msgr2=0x7f0a9c0b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0aa11f0640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c0b1540 0x7f0a9c0b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0aa11f0640 1 -- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9c15e470 msgr2=0x7f0a9c162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0aa11f0640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9c15e470 0x7f0a9c162480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0aa11f0640 1 -- 172.21.15.44:0/1400277718 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9c1617c0 con 0x7f0a9c0ce790 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0aa21f2640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9c15e470 0x7f0a9c162480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0aa11f0640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a9c0ce790 0x7f0a9c161f40 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00c400 tx=0x7f0a8c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0a927fc640 1 -- 172.21.15.44:0/1400277718 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c017070 con 0x7f0a9c0ce790 2024-04-02T13:25:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0a927fc640 1 -- 172.21.15.44:0/1400277718 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a8c005ce0 con 0x7f0a9c0ce790 2024-04-02T13:25:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0aa3c55640 1 -- 172.21.15.44:0/1400277718 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a9c160c40 con 0x7f0a9c0ce790 2024-04-02T13:25:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0a927fc640 1 -- 172.21.15.44:0/1400277718 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c0072f0 con 0x7f0a9c0ce790 2024-04-02T13:25:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.013+0000 7f0aa3c55640 1 -- 172.21.15.44:0/1400277718 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a9c161200 con 0x7f0a9c0ce790 2024-04-02T13:25:52.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.015+0000 7f0a927fc640 1 -- 172.21.15.44:0/1400277718 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0a8c01a070 con 0x7f0a9c0ce790 2024-04-02T13:25:52.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.015+0000 7f0a927fc640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a8006c8f0 0x7f0a8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.015+0000 7f0a927fc640 1 -- 172.21.15.44:0/1400277718 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f0a8c08b0b0 con 0x7f0a9c0ce790 2024-04-02T13:25:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.015+0000 7f0aa19f1640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a8006c8f0 0x7f0a8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.016+0000 7f0aa19f1640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a8006c8f0 0x7f0a8006edb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0a98006fd0 tx=0x7f0a98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.016+0000 7f0aa6841740 1 Processor -- start 2024-04-02T13:25:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.016+0000 7f0aa6841740 1 -- start start 2024-04-02T13:25:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.017+0000 7f0aa6841740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9bef0 0x55a663bafd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.017+0000 7f0aa6841740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663b9e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.017+0000 7f0aa6841740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663b9ec70 0x55a663b9f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.017+0000 7f0aa6841740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a663ad8830 con 0x55a663b9c2d0 2024-04-02T13:25:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.017+0000 7f0aa6841740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a663a80820 con 0x55a663b9ec70 2024-04-02T13:25:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.017+0000 7f0aa6841740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a663ad47e0 con 0x55a663b9bef0 2024-04-02T13:25:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.017+0000 7f0aa11f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663b9e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.017+0000 7f0aa11f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663b9e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60926/0 (socket says 172.21.15.44:60926) 2024-04-02T13:25:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.017+0000 7f0aa11f0640 1 -- 172.21.15.44:0/980245222 learned_addr learned my addr 172.21.15.44:0/980245222 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.017+0000 7f0aa19f1640 1 --2- 172.21.15.44:0/980245222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9bef0 0x55a663bafd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.017+0000 7f0aa21f2640 1 --2- 172.21.15.44:0/980245222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663b9ec70 0x55a663b9f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.018+0000 7f0aa11f0640 1 -- 172.21.15.44:0/980245222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9bef0 msgr2=0x55a663bafd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.018+0000 7f0aa11f0640 1 --2- 172.21.15.44:0/980245222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9bef0 0x55a663bafd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.018+0000 7f0aa11f0640 1 -- 172.21.15.44:0/980245222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663b9ec70 msgr2=0x55a663b9f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.018+0000 7f0aa11f0640 1 --2- 172.21.15.44:0/980245222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663b9ec70 0x55a663b9f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.018+0000 7f0aa11f0640 1 -- 172.21.15.44:0/980245222 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6638b67d0 con 0x55a663b9c2d0 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.018+0000 7f0aa11f0640 1 --2- 172.21.15.44:0/980245222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663b9e730 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00cfd0 tx=0x7f0a8c058900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3acb2972f6afcf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.018+0000 7f0a7f7fe640 1 -- 172.21.15.44:0/980245222 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c056d30 con 0x55a663b9c2d0 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.018+0000 7f0a7f7fe640 1 -- 172.21.15.44:0/980245222 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a8c056f90 con 0x55a663b9c2d0 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.018+0000 7f0a7f7fe640 1 -- 172.21.15.44:0/980245222 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c055300 con 0x55a663b9c2d0 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.019+0000 7f0aa6841740 1 -- 172.21.15.44:0/980245222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 msgr2=0x55a663b9e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.019+0000 7f0aa6841740 1 --2- 172.21.15.44:0/980245222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663b9e730 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00cfd0 tx=0x7f0a8c058900 comp rx=0 tx=0).stop 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.019+0000 7f0aa6841740 1 -- 172.21.15.44:0/980245222 shutdown_connections 2024-04-02T13:25:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.019+0000 7f0aa6841740 1 --2- 172.21.15.44:0/980245222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663b9ec70 0x55a663b9f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.019+0000 7f0aa6841740 1 --2- 172.21.15.44:0/980245222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663b9e730 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.019+0000 7f0aa6841740 1 --2- 172.21.15.44:0/980245222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9bef0 0x55a663bafd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.019+0000 7f0aa6841740 1 -- 172.21.15.44:0/980245222 >> 172.21.15.44:0/980245222 conn(0x55a663b70bb0 msgr2=0x55a663bb9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.019+0000 7f0aa6841740 1 -- 172.21.15.44:0/980245222 shutdown_connections 2024-04-02T13:25:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.019+0000 7f0aa6841740 1 -- 172.21.15.44:0/980245222 wait complete. 2024-04-02T13:25:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.019+0000 7f0aa6841740 1 -- start start 2024-04-02T13:25:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.019+0000 7f0aa6841740 1 Processor -- start 2024-04-02T13:25:52.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.020+0000 7f0aa6841740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9c2d0 0x55a663bd28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.020+0000 7f0aa6841740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9ec70 0x55a663bac030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.020+0000 7f0aa6841740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663badc90 0x55a663bbca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.020+0000 7f0aa6841740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a663b74600 con 0x55a663b9ec70 2024-04-02T13:25:52.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.020+0000 7f0aa6841740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a663a93d30 con 0x55a663badc90 2024-04-02T13:25:52.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.020+0000 7f0aa6841740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a663b748e0 con 0x55a663b9c2d0 2024-04-02T13:25:52.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.020+0000 7f0aa6841740 10 client.0 _ll_register_callbacks cb 0x55a663a93720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:52.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.020+0000 7f0aa11f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9ec70 0x55a663bac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.020+0000 7f0aa11f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9ec70 0x55a663bac030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60942/0 (socket says 172.21.15.44:60942) 2024-04-02T13:25:52.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.020+0000 7f0aa11f0640 1 -- 172.21.15.44:0/1143826900 learned_addr learned my addr 172.21.15.44:0/1143826900 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa19f1640 1 --2- 172.21.15.44:0/1143826900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9c2d0 0x55a663bd28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa21f2640 1 --2- 172.21.15.44:0/1143826900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663badc90 0x55a663bbca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa11f0640 1 -- 172.21.15.44:0/1143826900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9c2d0 msgr2=0x55a663bd28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa11f0640 1 --2- 172.21.15.44:0/1143826900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9c2d0 0x55a663bd28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa11f0640 1 -- 172.21.15.44:0/1143826900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663badc90 msgr2=0x55a663bbca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa11f0640 1 --2- 172.21.15.44:0/1143826900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663badc90 0x55a663bbca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa11f0640 1 -- 172.21.15.44:0/1143826900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6637ec3f0 con 0x55a663b9ec70 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa11f0640 1 --2- 172.21.15.44:0/1143826900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9ec70 0x55a663bac030 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c001eb0 tx=0x7f0a8c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80aadff021fedae6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0a7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0a7cff9640 1 -- 172.21.15.44:0/1143826900 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c007690 con 0x55a663b9ec70 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0a7cff9640 1 -- 172.21.15.44:0/1143826900 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f0a8c072d00 con 0x55a663b9ec70 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa6841740 1 -- 172.21.15.44:0/1143826900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a8c056f90 con 0x55a663b9ec70 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0a7cff9640 1 -- 172.21.15.44:0/1143826900 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f0a8c0580e0 con 0x55a663b9ec70 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa6841740 10 client.7461 fetch_fsmap learned FSMap version 51 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa6841740 1 -- 172.21.15.44:0/1143826900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55a6637ac0f0 con 0x55a663b9ec70 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0a7cff9640 1 -- 172.21.15.44:0/1143826900 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a8c056f90 con 0x55a663b9ec70 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0a7cff9640 1 -- 172.21.15.44:0/1143826900 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f0a8c058280 con 0x55a663b9ec70 2024-04-02T13:25:52.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa6841740 10 client.7461 fetch_fsmap finished waiting for FSMap version 51 2024-04-02T13:25:52.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa6841740 10 client.7461 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-02T13:25:52.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa6841740 20 client.7461 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:52.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa6841740 1 --2- 172.21.15.44:0/1143826900 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55a663bd5930 0x55a663bd5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa6841740 4 client.7461 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:25:52.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.021+0000 7f0aa6841740 1 -- 172.21.15.44:0/1143826900 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a663ae5d70 con 0x55a663bd5930 2024-04-02T13:25:52.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.022+0000 7f0aa19f1640 1 --2- 172.21.15.44:0/1143826900 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55a663bd5930 0x55a663bd5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.022+0000 7f0aa19f1640 1 --2- 172.21.15.44:0/1143826900 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55a663bd5930 0x55a663bd5d10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6175d40cf7935d53 server_cookie=a128aab6e51f6ec0 in_seq=0 out_seq=0 2024-04-02T13:25:52.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.022+0000 7f0a7cff9640 10 client.7461 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:52.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.022+0000 7f0a7cff9640 1 -- 172.21.15.44:0/1143826900 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a663ae5d70 con 0x55a663bd5930 2024-04-02T13:25:52.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.022+0000 7f0a7cff9640 10 client.7461 handle_command_reply: tid=0 2024-04-02T13:25:52.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.022+0000 7f0aa6841740 1 client.7461 shutdown 2024-04-02T13:25:52.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.023+0000 7f0aa6841740 1 -- 172.21.15.44:0/1143826900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9ec70 msgr2=0x55a663bac030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.023+0000 7f0aa6841740 1 --2- 172.21.15.44:0/1143826900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9ec70 0x55a663bac030 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c001eb0 tx=0x7f0a8c000bc0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.023+0000 7f0aa6841740 1 -- 172.21.15.44:0/1143826900 shutdown_connections 2024-04-02T13:25:52.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.023+0000 7f0aa6841740 1 --2- 172.21.15.44:0/1143826900 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55a663bd5930 0x55a663bd5d10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.023+0000 7f0aa6841740 1 --2- 172.21.15.44:0/1143826900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663badc90 0x55a663bbca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.023+0000 7f0aa6841740 1 --2- 172.21.15.44:0/1143826900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9ec70 0x55a663bac030 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.023+0000 7f0a7cff9640 0 client.7461 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:52.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.023+0000 7f0aa6841740 1 --2- 172.21.15.44:0/1143826900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9c2d0 0x55a663bd28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.023+0000 7f0aa6841740 1 -- 172.21.15.44:0/1143826900 >> 172.21.15.44:0/1143826900 conn(0x55a663b9bef0 msgr2=0x55a663bb9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.023+0000 7f0aa6841740 1 -- 172.21.15.44:0/1143826900 shutdown_connections 2024-04-02T13:25:52.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.023+0000 7f0aa6841740 1 -- 172.21.15.44:0/1143826900 wait complete. 2024-04-02T13:25:52.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.023+0000 7f0aa6841740 20 client.7461 trim_cache size 0 max 16384 2024-04-02T13:25:52.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.041+0000 7f0aa6841740 1 Processor -- start 2024-04-02T13:25:52.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.041+0000 7f0aa6841740 1 -- start start 2024-04-02T13:25:52.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa6841740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663b75670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa6841740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663b9ec70 0x55a663b73550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa6841740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663badc90 0x55a663b73a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa6841740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c0ba7d0 con 0x55a663b9c2d0 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa6841740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c0b97f0 con 0x55a663b9ec70 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa6841740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0a9c0b5f40 con 0x55a663badc90 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa19f1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663b75670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa19f1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663b75670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60952/0 (socket says 172.21.15.44:60952) 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa19f1640 1 -- 172.21.15.44:0/1029204711 learned_addr learned my addr 172.21.15.44:0/1029204711 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa19f1640 1 -- 172.21.15.44:0/1029204711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663badc90 msgr2=0x55a663b73a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa19f1640 1 --2- 172.21.15.44:0/1029204711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663badc90 0x55a663b73a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa19f1640 1 -- 172.21.15.44:0/1029204711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663b9ec70 msgr2=0x55a663b73550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa19f1640 1 --2- 172.21.15.44:0/1029204711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663b9ec70 0x55a663b73550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa19f1640 1 -- 172.21.15.44:0/1029204711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a663bd1d80 con 0x55a663b9c2d0 2024-04-02T13:25:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0aa19f1640 1 --2- 172.21.15.44:0/1029204711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663b75670 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800bf80 tx=0x7f0a98010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f17cc120a7371c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0a7effd640 1 -- 172.21.15.44:0/1029204711 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a98023910 con 0x55a663b9c2d0 2024-04-02T13:25:52.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0a7effd640 1 -- 172.21.15.44:0/1029204711 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a9802d070 con 0x55a663b9c2d0 2024-04-02T13:25:52.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.042+0000 7f0a7effd640 1 -- 172.21.15.44:0/1029204711 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a98023cb0 con 0x55a663b9c2d0 2024-04-02T13:25:52.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.043+0000 7f0aa6841740 1 -- 172.21.15.44:0/1029204711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 msgr2=0x55a663b75670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.043+0000 7f0aa6841740 1 --2- 172.21.15.44:0/1029204711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663b75670 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800bf80 tx=0x7f0a98010000 comp rx=0 tx=0).stop 2024-04-02T13:25:52.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.043+0000 7f0aa6841740 1 -- 172.21.15.44:0/1029204711 shutdown_connections 2024-04-02T13:25:52.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.043+0000 7f0aa6841740 1 --2- 172.21.15.44:0/1029204711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663badc90 0x55a663b73a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.043+0000 7f0aa6841740 1 --2- 172.21.15.44:0/1029204711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663b9ec70 0x55a663b73550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.043+0000 7f0aa6841740 1 --2- 172.21.15.44:0/1029204711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663b75670 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.043+0000 7f0aa6841740 1 -- 172.21.15.44:0/1029204711 >> 172.21.15.44:0/1029204711 conn(0x55a663b9bef0 msgr2=0x55a663b6a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.043+0000 7f0aa6841740 1 -- 172.21.15.44:0/1029204711 shutdown_connections 2024-04-02T13:25:52.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.043+0000 7f0aa6841740 1 -- 172.21.15.44:0/1029204711 wait complete. 2024-04-02T13:25:52.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.044+0000 7f0aa6841740 1 -- start start 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.044+0000 7f0aa6841740 1 Processor -- start 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa6841740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663c3e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa6841740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9ec70 0x55a663b9b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa6841740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663badc90 0x55a663bd7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa6841740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c0b6590 con 0x55a663b9c2d0 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa6841740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a663bd1b40 con 0x55a663badc90 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa6841740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a663bd8c50 con 0x55a663b9ec70 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa6841740 10 client.0 _ll_register_callbacks cb 0x7f0a9800b580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa19f1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663c3e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa19f1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663c3e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60960/0 (socket says 172.21.15.44:60960) 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa19f1640 1 -- 172.21.15.44:0/3610621277 learned_addr learned my addr 172.21.15.44:0/3610621277 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa11f0640 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9ec70 0x55a663b9b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa21f2640 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663badc90 0x55a663bd7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa19f1640 1 -- 172.21.15.44:0/3610621277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9ec70 msgr2=0x55a663b9b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa19f1640 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9ec70 0x55a663b9b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa19f1640 1 -- 172.21.15.44:0/3610621277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663badc90 msgr2=0x55a663bd7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa19f1640 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663badc90 0x55a663bd7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa19f1640 1 -- 172.21.15.44:0/3610621277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a663b9ac00 con 0x55a663b9c2d0 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa21f2640 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663badc90 0x55a663bd7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa19f1640 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663c3e3e0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f0a98008010 tx=0x7f0a8c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c209bb0b11aadcc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0a7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0a7cff9640 1 -- 172.21.15.44:0/3610621277 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a98037070 con 0x55a663b9c2d0 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0a7cff9640 1 -- 172.21.15.44:0/3610621277 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f0a980273f0 con 0x55a663b9c2d0 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.045+0000 7f0aa6841740 1 -- 172.21.15.44:0/3610621277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a9802d070 con 0x55a663b9c2d0 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0a7cff9640 1 -- 172.21.15.44:0/3610621277 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f0a980275f0 con 0x55a663b9c2d0 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0aa6841740 10 client.7467 fetch_fsmap learned FSMap version 51 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0aa6841740 1 -- 172.21.15.44:0/3610621277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55a663bd89f0 con 0x55a663b9c2d0 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0a7cff9640 1 -- 172.21.15.44:0/3610621277 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a9802d070 con 0x55a663b9c2d0 2024-04-02T13:25:52.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0a7cff9640 1 -- 172.21.15.44:0/3610621277 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f0a98020040 con 0x55a663b9c2d0 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0aa6841740 10 client.7467 fetch_fsmap finished waiting for FSMap version 51 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0aa6841740 10 client.7467 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0aa6841740 20 client.7467 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0aa6841740 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55a663bd8510 0x55a663b73550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0aa6841740 4 client.7467 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0aa6841740 1 -- 172.21.15.44:0/3610621277 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a663b702a0 con 0x55a663bd8510 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0aa11f0640 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55a663bd8510 0x55a663b73550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0a7cff9640 10 client.7467 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.046+0000 7f0aa11f0640 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55a663bd8510 0x55a663b73550 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46159625ce13e5d3 server_cookie=a34547e225d221fe in_seq=0 out_seq=0 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0a7cff9640 1 -- 172.21.15.44:0/3610621277 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a663b702a0 con 0x55a663bd8510 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0a7cff9640 10 client.7467 handle_command_reply: tid=0 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0aa6841740 1 client.7467 shutdown 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0aa6841740 1 -- 172.21.15.44:0/3610621277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 msgr2=0x55a663c3e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0aa6841740 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663c3e3e0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f0a98008010 tx=0x7f0a8c013000 comp rx=0 tx=0).stop 2024-04-02T13:25:52.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0aa6841740 1 -- 172.21.15.44:0/3610621277 shutdown_connections 2024-04-02T13:25:52.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0aa6841740 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55a663bd8510 0x55a663b73550 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0aa6841740 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a663badc90 0x55a663bd7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0aa6841740 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a663b9ec70 0x55a663b9b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0a7cff9640 0 client.7467 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:52.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0aa6841740 1 --2- 172.21.15.44:0/3610621277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a663b9c2d0 0x55a663c3e3e0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0aa6841740 1 -- 172.21.15.44:0/3610621277 >> 172.21.15.44:0/3610621277 conn(0x55a663b9bef0 msgr2=0x55a663ba1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0aa6841740 1 -- 172.21.15.44:0/3610621277 shutdown_connections 2024-04-02T13:25:52.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0aa6841740 1 -- 172.21.15.44:0/3610621277 wait complete. 2024-04-02T13:25:52.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.047+0000 7f0aa6841740 20 client.7467 trim_cache size 0 max 16384 2024-04-02T13:25:52.055 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:25:52.056 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-02T13:25:52.056 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-02T13:25:52.056 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:25:52.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.048+0000 7f0aa3c55640 1 -- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a8006c8f0 msgr2=0x7f0a8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.048+0000 7f0aa3c55640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a8006c8f0 0x7f0a8006edb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0a98006fd0 tx=0x7f0a98008040 comp rx=0 tx=0).stop 2024-04-02T13:25:52.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.048+0000 7f0aa3c55640 1 -- 172.21.15.44:0/1400277718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a9c0ce790 msgr2=0x7f0a9c161f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.048+0000 7f0aa3c55640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a9c0ce790 0x7f0a9c161f40 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00c400 tx=0x7f0a8c00c8c0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.048+0000 7f0aa3c55640 1 -- 172.21.15.44:0/1400277718 shutdown_connections 2024-04-02T13:25:52.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.048+0000 7f0aa3c55640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a8006c8f0 0x7f0a8006edb0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.048+0000 7f0aa3c55640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9c15e470 0x7f0a9c162480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.048+0000 7f0aa3c55640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a9c0ce790 0x7f0a9c161f40 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.048+0000 7f0aa3c55640 1 --2- 172.21.15.44:0/1400277718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c0b1540 0x7f0a9c0b7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.048+0000 7f0aa3c55640 1 -- 172.21.15.44:0/1400277718 >> 172.21.15.44:0/1400277718 conn(0x7f0a9c09aa20 msgr2=0x7f0a9c0d19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.048+0000 7f0aa3c55640 1 -- 172.21.15.44:0/1400277718 shutdown_connections 2024-04-02T13:25:52.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.048+0000 7f0aa3c55640 1 -- 172.21.15.44:0/1400277718 wait complete. 2024-04-02T13:25:52.060 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T13:25:52.061 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:25:52.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.160+0000 7f6eb6260640 1 Processor -- start 2024-04-02T13:25:52.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.160+0000 7f6eb6260640 1 -- start start 2024-04-02T13:25:52.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.161+0000 7f6eb6260640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6eb0159db0 0x7f6eb015c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.161+0000 7f6eb6260640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6eb00ccdf0 0x7f6eb0165cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.161+0000 7f6eb6260640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6eb00726c0 con 0x7f6eb0071e50 2024-04-02T13:25:52.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.161+0000 7f6eb6260640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f6eb0073240 con 0x7f6eb00ccdf0 2024-04-02T13:25:52.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.161+0000 7f6eb6260640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6eb00729a0 con 0x7f6eb0159db0 2024-04-02T13:25:52.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.161+0000 7f6eaeffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6eb0071e50 0x7f6eb00cc6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53024/0 (socket says 172.21.15.44:53024) 2024-04-02T13:25:52.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.161+0000 7f6eaffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6eb00ccdf0 0x7f6eb0165cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.161+0000 7f6eaeffd640 1 -- 172.21.15.44:0/1117985357 learned_addr learned my addr 172.21.15.44:0/1117985357 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:52.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.161+0000 7f6eaf7fe640 1 --2- 172.21.15.44:0/1117985357 >> v2:172.21.15.69:3300/0 conn(0x7f6eb0159db0 0x7f6eb015c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.161+0000 7f6eaffff640 1 -- 172.21.15.44:0/1117985357 >> v2:172.21.15.69:3300/0 conn(0x7f6eb0159db0 msgr2=0x7f6eb015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.161+0000 7f6eaffff640 1 --2- 172.21.15.44:0/1117985357 >> v2:172.21.15.69:3300/0 conn(0x7f6eb0159db0 0x7f6eb015c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.161+0000 7f6eaffff640 1 -- 172.21.15.44:0/1117985357 >> v1:172.21.15.44:6789/0 conn(0x7f6eb0071e50 legacy=0x7f6eb00cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.161+0000 7f6eaffff640 1 -- 172.21.15.44:0/1117985357 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb0072c80 con 0x7f6eb00ccdf0 2024-04-02T13:25:52.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.162+0000 7f6eaffff640 1 --2- 172.21.15.44:0/1117985357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb00ccdf0 0x7f6eb0165cf0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f6eb00733c0 tx=0x7f6ea802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2edb044b9f5bc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.162+0000 7f6eae7fc640 1 -- 172.21.15.44:0/1117985357 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8039070 con 0x7f6eb00ccdf0 2024-04-02T13:25:52.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.162+0000 7f6eae7fc640 1 -- 172.21.15.44:0/1117985357 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ea802cbd0 con 0x7f6eb00ccdf0 2024-04-02T13:25:52.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.162+0000 7f6eae7fc640 1 -- 172.21.15.44:0/1117985357 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8034440 con 0x7f6eb00ccdf0 2024-04-02T13:25:52.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.163+0000 7f6eb6260640 1 -- 172.21.15.44:0/1117985357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb00ccdf0 msgr2=0x7f6eb0165cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.163+0000 7f6eb6260640 1 --2- 172.21.15.44:0/1117985357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb00ccdf0 0x7f6eb0165cf0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f6eb00733c0 tx=0x7f6ea802be20 comp rx=0 tx=0).stop 2024-04-02T13:25:52.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.163+0000 7f6eb6260640 1 -- 172.21.15.44:0/1117985357 shutdown_connections 2024-04-02T13:25:52.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.163+0000 7f6eb6260640 1 --2- 172.21.15.44:0/1117985357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb00ccdf0 0x7f6eb0165cf0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.163+0000 7f6eb6260640 1 --2- 172.21.15.44:0/1117985357 >> v2:172.21.15.69:3300/0 conn(0x7f6eb0159db0 0x7f6eb015c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.163+0000 7f6eb6260640 1 -- 172.21.15.44:0/1117985357 >> 172.21.15.44:0/1117985357 conn(0x7f6eb00cfcd0 msgr2=0x7f6eb00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.163+0000 7f6eb6260640 1 -- 172.21.15.44:0/1117985357 shutdown_connections 2024-04-02T13:25:52.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.163+0000 7f6eb6260640 1 -- 172.21.15.44:0/1117985357 wait complete. 2024-04-02T13:25:52.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.164+0000 7f6eb6260640 1 Processor -- start 2024-04-02T13:25:52.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.164+0000 7f6eb6260640 1 -- start start 2024-04-02T13:25:52.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eb6260640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb0071e50 0x7f6eb0155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eb6260640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb00ccdf0 0x7f6eb0157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eb6260640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb0159db0 0x7f6eb015adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eb6260640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb0072480 con 0x7f6eb0159db0 2024-04-02T13:25:52.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eb6260640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb0072300 con 0x7f6eb00ccdf0 2024-04-02T13:25:52.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eb6260640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6eb006e650 con 0x7f6eb0071e50 2024-04-02T13:25:52.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eaffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb0159db0 0x7f6eb015adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eaffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb0159db0 0x7f6eb015adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60988/0 (socket says 172.21.15.44:60988) 2024-04-02T13:25:52.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eaffff640 1 -- 172.21.15.44:0/2281583397 learned_addr learned my addr 172.21.15.44:0/2281583397 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eaf7fe640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb0071e50 0x7f6eb0155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eaeffd640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb00ccdf0 0x7f6eb0157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eaffff640 1 -- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb0071e50 msgr2=0x7f6eb0155110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eaffff640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb0071e50 0x7f6eb0155110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eaffff640 1 -- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb00ccdf0 msgr2=0x7f6eb0157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eaffff640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb00ccdf0 0x7f6eb0157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.165+0000 7f6eaffff640 1 -- 172.21.15.44:0/2281583397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb01585e0 con 0x7f6eb0159db0 2024-04-02T13:25:52.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.166+0000 7f6eaffff640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb0159db0 0x7f6eb015adc0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f6ea802c430 tx=0x7f6ea8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.166+0000 7f6e97fff640 1 -- 172.21.15.44:0/2281583397 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8039050 con 0x7f6eb0159db0 2024-04-02T13:25:52.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.166+0000 7f6e97fff640 1 -- 172.21.15.44:0/2281583397 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ea8045070 con 0x7f6eb0159db0 2024-04-02T13:25:52.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.166+0000 7f6eb6260640 1 -- 172.21.15.44:0/2281583397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb01588c0 con 0x7f6eb0159db0 2024-04-02T13:25:52.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.166+0000 7f6e97fff640 1 -- 172.21.15.44:0/2281583397 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea803e830 con 0x7f6eb0159db0 2024-04-02T13:25:52.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.166+0000 7f6eb6260640 1 -- 172.21.15.44:0/2281583397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb0158e80 con 0x7f6eb0159db0 2024-04-02T13:25:52.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.167+0000 7f6eb6260640 1 -- 172.21.15.44:0/2281583397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb0074f20 con 0x7f6eb0159db0 2024-04-02T13:25:52.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.167+0000 7f6e97fff640 1 -- 172.21.15.44:0/2281583397 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ea803e9d0 con 0x7f6eb0159db0 2024-04-02T13:25:52.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.168+0000 7f6e97fff640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6e9c06c8f0 0x7f6e9c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.168+0000 7f6e97fff640 1 -- 172.21.15.44:0/2281583397 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f6ea80ae950 con 0x7f6eb0159db0 2024-04-02T13:25:52.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.168+0000 7f6eaf7fe640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6e9c06c8f0 0x7f6e9c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.168+0000 7f6eaf7fe640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6e9c06c8f0 0x7f6e9c06edb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00079a0 tx=0x7f6ea0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.172+0000 7f6e97fff640 1 -- 172.21.15.44:0/2281583397 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6ea807c650 con 0x7f6eb0159db0 2024-04-02T13:25:52.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.395+0000 7f6eb6260640 1 -- 172.21.15.44:0/2281583397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6eb00684f0 con 0x7f6eb0159db0 2024-04-02T13:25:52.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.396+0000 7f6e97fff640 1 -- 172.21.15.44:0/2281583397 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5185 (secure 0 0 0) 0x7f6ea8080950 con 0x7f6eb0159db0 2024-04-02T13:25:52.398 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:52.398 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":51,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3662927745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3662927745},{"type":"v1","addr":"172.21.15.44:6835","nonce":3662927745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6394,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1053037366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1053037366},{"type":"v1","addr":"172.21.15.69:6836","nonce":1053037366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3893800047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3893800047},{"type":"v1","addr":"172.21.15.69:6837","nonce":3893800047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:24:53.925714+0000","modified":"2024-04-02T13:25:21.947562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":11,"addr":"172.21.15.44:6833/2392159648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2392159648},{"type":"v1","addr":"172.21.15.44:6833","nonce":2392159648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":5}]} 2024-04-02T13:25:52.399 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 51 2024-04-02T13:25:52.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.398+0000 7f6eb6260640 1 -- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6e9c06c8f0 msgr2=0x7f6e9c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.398+0000 7f6eb6260640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6e9c06c8f0 0x7f6e9c06edb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00079a0 tx=0x7f6ea0008040 comp rx=0 tx=0).stop 2024-04-02T13:25:52.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.398+0000 7f6eb6260640 1 -- 172.21.15.44:0/2281583397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb0159db0 msgr2=0x7f6eb015adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.398+0000 7f6eb6260640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb0159db0 0x7f6eb015adc0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f6ea802c430 tx=0x7f6ea8004a40 comp rx=0 tx=0).stop 2024-04-02T13:25:52.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.398+0000 7f6eb6260640 1 -- 172.21.15.44:0/2281583397 shutdown_connections 2024-04-02T13:25:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.398+0000 7f6eb6260640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6e9c06c8f0 0x7f6e9c06edb0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.398+0000 7f6eb6260640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb0159db0 0x7f6eb015adc0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.398+0000 7f6eb6260640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb00ccdf0 0x7f6eb0157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.398+0000 7f6eb6260640 1 --2- 172.21.15.44:0/2281583397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb0071e50 0x7f6eb0155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.398+0000 7f6eb6260640 1 -- 172.21.15.44:0/2281583397 >> 172.21.15.44:0/2281583397 conn(0x7f6eb00cfcd0 msgr2=0x7f6eb0156200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.398+0000 7f6eb6260640 1 -- 172.21.15.44:0/2281583397 shutdown_connections 2024-04-02T13:25:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.398+0000 7f6eb6260640 1 -- 172.21.15.44:0/2281583397 wait complete. 2024-04-02T13:25:52.411 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-04-02T13:25:52.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.512+0000 7f677eb22640 1 Processor -- start 2024-04-02T13:25:52.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.512+0000 7f677eb22640 1 -- start start 2024-04-02T13:25:52.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677eb22640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f677815e430 0x7f67780acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677eb22640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f677815e810 0x7f67780cae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677eb22640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f67780ca380 0x7f67780cb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677eb22640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f6778072eb0 con 0x7f67780ca380 2024-04-02T13:25:52.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677eb22640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f67780cfaa0 con 0x7f677815e430 2024-04-02T13:25:52.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677eb22640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f67780cf7c0 con 0x7f677815e810 2024-04-02T13:25:52.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677d0bf640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f67780ca380 0x7f67780cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677d0bf640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f67780ca380 0x7f67780cb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60996/0 (socket says 172.21.15.44:60996) 2024-04-02T13:25:52.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677d0bf640 1 -- 172.21.15.44:0/992283825 learned_addr learned my addr 172.21.15.44:0/992283825 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677c8be640 1 --2- 172.21.15.44:0/992283825 >> v2:172.21.15.69:3300/0 conn(0x7f677815e430 0x7f67780acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f6777fff640 1 --2- 172.21.15.44:0/992283825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f677815e810 0x7f67780cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677d0bf640 1 -- 172.21.15.44:0/992283825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f677815e810 msgr2=0x7f67780cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677d0bf640 1 --2- 172.21.15.44:0/992283825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f677815e810 0x7f67780cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677d0bf640 1 -- 172.21.15.44:0/992283825 >> v2:172.21.15.69:3300/0 conn(0x7f677815e430 msgr2=0x7f67780acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677d0bf640 1 --2- 172.21.15.44:0/992283825 >> v2:172.21.15.69:3300/0 conn(0x7f677815e430 0x7f67780acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677d0bf640 1 -- 172.21.15.44:0/992283825 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f677806e950 con 0x7f67780ca380 2024-04-02T13:25:52.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.513+0000 7f677c8be640 1 --2- 172.21.15.44:0/992283825 >> v2:172.21.15.69:3300/0 conn(0x7f677815e430 0x7f67780acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:52.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.514+0000 7f677d0bf640 1 --2- 172.21.15.44:0/992283825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67780ca380 0x7f67780cb360 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f6778073a00 tx=0x7f677002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45670106454dab85 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.514+0000 7f67777fe640 1 -- 172.21.15.44:0/992283825 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677002c9e0 con 0x7f67780ca380 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.514+0000 7f67777fe640 1 -- 172.21.15.44:0/992283825 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f677002cb80 con 0x7f67780ca380 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.514+0000 7f67777fe640 1 -- 172.21.15.44:0/992283825 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67700343f0 con 0x7f67780ca380 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.514+0000 7f677eb22640 1 -- 172.21.15.44:0/992283825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67780ca380 msgr2=0x7f67780cb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.514+0000 7f677eb22640 1 --2- 172.21.15.44:0/992283825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67780ca380 0x7f67780cb360 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f6778073a00 tx=0x7f677002be40 comp rx=0 tx=0).stop 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.514+0000 7f677eb22640 1 -- 172.21.15.44:0/992283825 shutdown_connections 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.514+0000 7f677eb22640 1 --2- 172.21.15.44:0/992283825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67780ca380 0x7f67780cb360 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.514+0000 7f677eb22640 1 --2- 172.21.15.44:0/992283825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f677815e810 0x7f67780cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.514+0000 7f677eb22640 1 --2- 172.21.15.44:0/992283825 >> v2:172.21.15.69:3300/0 conn(0x7f677815e430 0x7f67780acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.515+0000 7f677eb22640 1 -- 172.21.15.44:0/992283825 >> 172.21.15.44:0/992283825 conn(0x7f67780cfca0 msgr2=0x7f67780d0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.515+0000 7f677eb22640 1 -- 172.21.15.44:0/992283825 shutdown_connections 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.515+0000 7f677eb22640 1 -- 172.21.15.44:0/992283825 wait complete. 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.516+0000 7f677eb22640 1 Processor -- start 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.516+0000 7f677eb22640 1 -- start start 2024-04-02T13:25:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.516+0000 7f677eb22640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67780ca380 0x7f677814ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.516+0000 7f677eb22640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f677815e430 0x7f67780add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.516+0000 7f677eb22640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f677815e810 0x7f67780afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.516+0000 7f677eb22640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f67780ca8d0 con 0x7f677815e430 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f677eb22640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f677806ef10 con 0x7f677815e810 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f677eb22640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f67780caa50 con 0x7f67780ca380 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f6777fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f677815e430 0x7f67780add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f6777fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f677815e430 0x7f67780add00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32776/0 (socket says 172.21.15.44:32776) 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f6777fff640 1 -- 172.21.15.44:0/1545265788 learned_addr learned my addr 172.21.15.44:0/1545265788 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f6777fff640 1 -- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67780ca380 msgr2=0x7f677814ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f6777fff640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67780ca380 0x7f677814ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f6777fff640 1 -- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f677815e810 msgr2=0x7f67780afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f677d0bf640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f677815e810 0x7f67780afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f6777fff640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f677815e810 0x7f67780afc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f6777fff640 1 -- 172.21.15.44:0/1545265788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67780ae760 con 0x7f677815e430 2024-04-02T13:25:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f677d0bf640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f677815e810 0x7f67780afc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f6777fff640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f677815e430 0x7f67780add00 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f676400d800 tx=0x7f676400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f67757fa640 1 -- 172.21.15.44:0/1545265788 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676400b6c0 con 0x7f677815e430 2024-04-02T13:25:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f67757fa640 1 -- 172.21.15.44:0/1545265788 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6764019070 con 0x7f677815e430 2024-04-02T13:25:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f677eb22640 1 -- 172.21.15.44:0/1545265788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67780aea40 con 0x7f677815e430 2024-04-02T13:25:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f67757fa640 1 -- 172.21.15.44:0/1545265788 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676400ba80 con 0x7f677815e430 2024-04-02T13:25:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.517+0000 7f677eb22640 1 -- 172.21.15.44:0/1545265788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67780af000 con 0x7f677815e430 2024-04-02T13:25:52.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.519+0000 7f67757fa640 1 -- 172.21.15.44:0/1545265788 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f676400bc20 con 0x7f677815e430 2024-04-02T13:25:52.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.519+0000 7f67757fa640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f675c06c8f0 0x7f675c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.519+0000 7f67757fa640 1 -- 172.21.15.44:0/1545265788 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f676408c260 con 0x7f677815e430 2024-04-02T13:25:52.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.520+0000 7f677c8be640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f675c06c8f0 0x7f675c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.520+0000 7f677c8be640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f675c06c8f0 0x7f675c06edb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f676c0096f0 tx=0x7f676c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.520+0000 7f67817df740 1 Processor -- start 2024-04-02T13:25:52.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.520+0000 7f67817df740 1 -- start start 2024-04-02T13:25:52.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f67817df740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7f7dc0 0x55944a7f81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f67817df740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a7fdbe0 0x55944a7fbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f67817df740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a8454d0 0x55944a845950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f67817df740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55944a7bf5d0 con 0x55944a7fdbe0 2024-04-02T13:25:52.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f67817df740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55944a7616b0 con 0x55944a8454d0 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f67817df740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55944a6ff890 con 0x55944a7f7dc0 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f6777fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a7fdbe0 0x55944a7fbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f6777fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a7fdbe0 0x55944a7fbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32778/0 (socket says 172.21.15.44:32778) 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f6777fff640 1 -- 172.21.15.44:0/2120960186 learned_addr learned my addr 172.21.15.44:0/2120960186 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f677d0bf640 1 --2- 172.21.15.44:0/2120960186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a8454d0 0x55944a845950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f677c8be640 1 --2- 172.21.15.44:0/2120960186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7f7dc0 0x55944a7f81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f6777fff640 1 -- 172.21.15.44:0/2120960186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7f7dc0 msgr2=0x55944a7f81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f6777fff640 1 --2- 172.21.15.44:0/2120960186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7f7dc0 0x55944a7f81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f6777fff640 1 -- 172.21.15.44:0/2120960186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a8454d0 msgr2=0x55944a845950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f6777fff640 1 --2- 172.21.15.44:0/2120960186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a8454d0 0x55944a845950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.521+0000 7f6777fff640 1 -- 172.21.15.44:0/2120960186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55944a72e1d0 con 0x55944a7fdbe0 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.522+0000 7f6777fff640 1 --2- 172.21.15.44:0/2120960186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a7fdbe0 0x55944a7fbe10 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f676408a590 tx=0x7f67640560d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ab1fb374b0ae53 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.522+0000 7f67627fc640 1 -- 172.21.15.44:0/2120960186 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764089070 con 0x55944a7fdbe0 2024-04-02T13:25:52.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.522+0000 7f67627fc640 1 -- 172.21.15.44:0/2120960186 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6764062970 con 0x55944a7fdbe0 2024-04-02T13:25:52.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.522+0000 7f67627fc640 1 -- 172.21.15.44:0/2120960186 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764056370 con 0x55944a7fdbe0 2024-04-02T13:25:52.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.522+0000 7f67817df740 1 -- 172.21.15.44:0/2120960186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a7fdbe0 msgr2=0x55944a7fbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.522+0000 7f67817df740 1 --2- 172.21.15.44:0/2120960186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a7fdbe0 0x55944a7fbe10 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f676408a590 tx=0x7f67640560d0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.522+0000 7f67817df740 1 -- 172.21.15.44:0/2120960186 shutdown_connections 2024-04-02T13:25:52.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.522+0000 7f67817df740 1 --2- 172.21.15.44:0/2120960186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a8454d0 0x55944a845950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.522+0000 7f67817df740 1 --2- 172.21.15.44:0/2120960186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a7fdbe0 0x55944a7fbe10 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.522+0000 7f67817df740 1 --2- 172.21.15.44:0/2120960186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7f7dc0 0x55944a7f81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.522+0000 7f67817df740 1 -- 172.21.15.44:0/2120960186 >> 172.21.15.44:0/2120960186 conn(0x55944a7f9e40 msgr2=0x55944a7fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.522+0000 7f67817df740 1 -- 172.21.15.44:0/2120960186 shutdown_connections 2024-04-02T13:25:52.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.522+0000 7f67817df740 1 -- 172.21.15.44:0/2120960186 wait complete. 2024-04-02T13:25:52.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.523+0000 7f67817df740 1 -- start start 2024-04-02T13:25:52.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.523+0000 7f67817df740 1 Processor -- start 2024-04-02T13:25:52.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f67817df740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a7fdbe0 0x55944a82bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f67817df740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a8454d0 0x55944a857090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f67817df740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 0x55944a827910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f67817df740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55944a6fd200 con 0x55944a83ddd0 2024-04-02T13:25:52.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f67817df740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55944a75d660 con 0x55944a7fdbe0 2024-04-02T13:25:52.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f67817df740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55944a824530 con 0x55944a8454d0 2024-04-02T13:25:52.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f67817df740 10 client.0 _ll_register_callbacks cb 0x55944a720040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:52.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f677d0bf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 0x55944a827910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f677d0bf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 0x55944a827910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32784/0 (socket says 172.21.15.44:32784) 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f677d0bf640 1 -- 172.21.15.44:0/3089266127 learned_addr learned my addr 172.21.15.44:0/3089266127 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f677c8be640 1 --2- 172.21.15.44:0/3089266127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a7fdbe0 0x55944a82bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f6777fff640 1 --2- 172.21.15.44:0/3089266127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a8454d0 0x55944a857090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f677d0bf640 1 -- 172.21.15.44:0/3089266127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a8454d0 msgr2=0x55944a857090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f677d0bf640 1 --2- 172.21.15.44:0/3089266127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a8454d0 0x55944a857090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f677d0bf640 1 -- 172.21.15.44:0/3089266127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a7fdbe0 msgr2=0x55944a82bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f677d0bf640 1 --2- 172.21.15.44:0/3089266127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a7fdbe0 0x55944a82bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.524+0000 7f677d0bf640 1 -- 172.21.15.44:0/3089266127 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55944a5417c0 con 0x55944a83ddd0 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f677d0bf640 1 --2- 172.21.15.44:0/3089266127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 0x55944a827910 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f677002c450 tx=0x7f6770004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ce2ccf488710d84 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67577fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67577fe640 1 -- 172.21.15.44:0/3089266127 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6770009510 con 0x55944a83ddd0 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67577fe640 1 -- 172.21.15.44:0/3089266127 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f6770034b30 con 0x55944a83ddd0 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67817df740 1 -- 172.21.15.44:0/3089266127 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6764062970 con 0x55944a83ddd0 2024-04-02T13:25:52.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67577fe640 1 -- 172.21.15.44:0/3089266127 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f6770050050 con 0x55944a83ddd0 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67817df740 10 client.7488 fetch_fsmap learned FSMap version 51 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67817df740 1 -- 172.21.15.44:0/3089266127 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55944a5a1f50 con 0x55944a83ddd0 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67577fe640 1 -- 172.21.15.44:0/3089266127 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6764062970 con 0x55944a83ddd0 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67577fe640 1 -- 172.21.15.44:0/3089266127 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f6770053020 con 0x55944a83ddd0 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67817df740 10 client.7488 fetch_fsmap finished waiting for FSMap version 51 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67817df740 10 client.7488 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67817df740 20 client.7488 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67817df740 1 --2- 172.21.15.44:0/3089266127 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55944a829320 0x55944a829700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67817df740 4 client.7488 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f67817df740 1 -- 172.21.15.44:0/3089266127 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55944a774ee0 con 0x55944a829320 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.525+0000 7f677c8be640 1 --2- 172.21.15.44:0/3089266127 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55944a829320 0x55944a829700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.526+0000 7f67577fe640 10 client.7488 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.526+0000 7f677c8be640 1 --2- 172.21.15.44:0/3089266127 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55944a829320 0x55944a829700 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7201e007ea64a63 server_cookie=219625960aa3451c in_seq=0 out_seq=0 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.526+0000 7f67577fe640 1 -- 172.21.15.44:0/3089266127 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55944a774ee0 con 0x55944a829320 2024-04-02T13:25:52.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.526+0000 7f67577fe640 10 client.7488 handle_command_reply: tid=0 2024-04-02T13:25:52.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.526+0000 7f67817df740 1 client.7488 shutdown 2024-04-02T13:25:52.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.526+0000 7f67817df740 1 -- 172.21.15.44:0/3089266127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 msgr2=0x55944a827910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.526+0000 7f67817df740 1 --2- 172.21.15.44:0/3089266127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 0x55944a827910 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f677002c450 tx=0x7f6770004ab0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.526+0000 7f67817df740 1 -- 172.21.15.44:0/3089266127 shutdown_connections 2024-04-02T13:25:52.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.527+0000 7f67817df740 1 --2- 172.21.15.44:0/3089266127 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55944a829320 0x55944a829700 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.527+0000 7f67817df740 1 --2- 172.21.15.44:0/3089266127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 0x55944a827910 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.527+0000 7f67817df740 1 --2- 172.21.15.44:0/3089266127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a8454d0 0x55944a857090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.527+0000 7f67577fe640 0 client.7488 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:52.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.527+0000 7f67817df740 1 --2- 172.21.15.44:0/3089266127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a7fdbe0 0x55944a82bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.527+0000 7f67817df740 1 -- 172.21.15.44:0/3089266127 >> 172.21.15.44:0/3089266127 conn(0x55944a7f7dc0 msgr2=0x55944a843510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.527+0000 7f67817df740 1 -- 172.21.15.44:0/3089266127 shutdown_connections 2024-04-02T13:25:52.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.527+0000 7f67817df740 1 -- 172.21.15.44:0/3089266127 wait complete. 2024-04-02T13:25:52.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.527+0000 7f67817df740 20 client.7488 trim_cache size 0 max 16384 2024-04-02T13:25:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.540+0000 7f67817df740 1 Processor -- start 2024-04-02T13:25:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.540+0000 7f67817df740 1 -- start start 2024-04-02T13:25:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f67817df740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7fdbe0 0x55944a843f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f67817df740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a83ddd0 0x55944a844480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f67817df740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a8454d0 0x55944a855060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f67817df740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f67781fecf0 con 0x55944a8454d0 2024-04-02T13:25:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f67817df740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f67781fdd60 con 0x55944a83ddd0 2024-04-02T13:25:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f67817df740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f677814df30 con 0x55944a7fdbe0 2024-04-02T13:25:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f677d0bf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a8454d0 0x55944a855060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f677d0bf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a8454d0 0x55944a855060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32796/0 (socket says 172.21.15.44:32796) 2024-04-02T13:25:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f677d0bf640 1 -- 172.21.15.44:0/2407915329 learned_addr learned my addr 172.21.15.44:0/2407915329 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f6777fff640 1 --2- 172.21.15.44:0/2407915329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a83ddd0 0x55944a844480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f677c8be640 1 --2- 172.21.15.44:0/2407915329 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7fdbe0 0x55944a843f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f677d0bf640 1 -- 172.21.15.44:0/2407915329 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7fdbe0 msgr2=0x55944a843f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f677d0bf640 1 --2- 172.21.15.44:0/2407915329 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7fdbe0 0x55944a843f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f677d0bf640 1 -- 172.21.15.44:0/2407915329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a83ddd0 msgr2=0x55944a844480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f677d0bf640 1 --2- 172.21.15.44:0/2407915329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a83ddd0 0x55944a844480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f677d0bf640 1 -- 172.21.15.44:0/2407915329 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55944a4350b0 con 0x55944a8454d0 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.541+0000 7f677d0bf640 1 --2- 172.21.15.44:0/2407915329 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a8454d0 0x55944a855060 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f6770034600 tx=0x7f6770040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f150300649c4ff0b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.542+0000 7f6761ffb640 1 -- 172.21.15.44:0/2407915329 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6770034b30 con 0x55944a8454d0 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.542+0000 7f6761ffb640 1 -- 172.21.15.44:0/2407915329 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f677002cb80 con 0x55944a8454d0 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.542+0000 7f6761ffb640 1 -- 172.21.15.44:0/2407915329 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6770003b90 con 0x55944a8454d0 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.542+0000 7f67817df740 1 -- 172.21.15.44:0/2407915329 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a8454d0 msgr2=0x55944a855060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.542+0000 7f67817df740 1 --2- 172.21.15.44:0/2407915329 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a8454d0 0x55944a855060 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f6770034600 tx=0x7f6770040000 comp rx=0 tx=0).stop 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.542+0000 7f67817df740 1 -- 172.21.15.44:0/2407915329 shutdown_connections 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.542+0000 7f67817df740 1 --2- 172.21.15.44:0/2407915329 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a8454d0 0x55944a855060 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.542+0000 7f67817df740 1 --2- 172.21.15.44:0/2407915329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a83ddd0 0x55944a844480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.542+0000 7f67817df740 1 --2- 172.21.15.44:0/2407915329 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7fdbe0 0x55944a843f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.542+0000 7f67817df740 1 -- 172.21.15.44:0/2407915329 >> 172.21.15.44:0/2407915329 conn(0x55944a7f7dc0 msgr2=0x55944a829700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.542+0000 7f67817df740 1 -- 172.21.15.44:0/2407915329 shutdown_connections 2024-04-02T13:25:52.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.542+0000 7f67817df740 1 -- 172.21.15.44:0/2407915329 wait complete. 2024-04-02T13:25:52.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.543+0000 7f67817df740 1 -- start start 2024-04-02T13:25:52.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.543+0000 7f67817df740 1 Processor -- start 2024-04-02T13:25:52.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f67817df740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7fdbe0 0x55944a8c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f67817df740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 0x55944a843f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f67817df740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a8454d0 0x55944a844480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f67817df740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55944a83d640 con 0x55944a83ddd0 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f67817df740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f677814e580 con 0x55944a8454d0 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f67817df740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55944a82ad20 con 0x55944a7fdbe0 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f67817df740 10 client.0 _ll_register_callbacks cb 0x7f6770009340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f6777fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 0x55944a843f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f6777fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 0x55944a843f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32804/0 (socket says 172.21.15.44:32804) 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f6777fff640 1 -- 172.21.15.44:0/974469030 learned_addr learned my addr 172.21.15.44:0/974469030 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f677c8be640 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7fdbe0 0x55944a8c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f677d0bf640 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a8454d0 0x55944a844480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f6777fff640 1 -- 172.21.15.44:0/974469030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7fdbe0 msgr2=0x55944a8c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f6777fff640 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7fdbe0 0x55944a8c1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f6777fff640 1 -- 172.21.15.44:0/974469030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a8454d0 msgr2=0x55944a844480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f6777fff640 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a8454d0 0x55944a844480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f6777fff640 1 -- 172.21.15.44:0/974469030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55944a83cf00 con 0x55944a83ddd0 2024-04-02T13:25:52.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f677d0bf640 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a8454d0 0x55944a844480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f677c8be640 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7fdbe0 0x55944a8c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f6777fff640 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 0x55944a843f40 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f6764062440 tx=0x7f6764012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce3279bf6f0a327 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f67577fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f67577fe640 1 -- 172.21.15.44:0/974469030 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764089040 con 0x55944a83ddd0 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f67577fe640 1 -- 172.21.15.44:0/974469030 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f6764058000 con 0x55944a83ddd0 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f67817df740 1 -- 172.21.15.44:0/974469030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f677002cb80 con 0x55944a83ddd0 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.544+0000 7f67577fe640 1 -- 172.21.15.44:0/974469030 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f6764055fa0 con 0x55944a83ddd0 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67817df740 10 client.7494 fetch_fsmap learned FSMap version 51 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67817df740 1 -- 172.21.15.44:0/974469030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55944a8c4c10 con 0x55944a83ddd0 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67577fe640 1 -- 172.21.15.44:0/974469030 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f677002cb80 con 0x55944a83ddd0 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67577fe640 1 -- 172.21.15.44:0/974469030 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f6764013070 con 0x55944a83ddd0 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67817df740 10 client.7494 fetch_fsmap finished waiting for FSMap version 51 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67817df740 10 client.7494 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67817df740 20 client.7494 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67817df740 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55944a8c44f0 0x55944a851600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67817df740 4 client.7494 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67817df740 1 -- 172.21.15.44:0/974469030 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55944a7fa5a0 con 0x55944a8c44f0 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f677c8be640 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55944a8c44f0 0x55944a851600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67577fe640 10 client.7494 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f677c8be640 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55944a8c44f0 0x55944a851600 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ac977fc96a3e5bc server_cookie=31d91f3383bb15d1 in_seq=0 out_seq=0 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67577fe640 1 -- 172.21.15.44:0/974469030 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+237 (crc 0 0 0) 0x55944a7fa5a0 con 0x55944a8c44f0 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67577fe640 10 client.7494 handle_command_reply: tid=0 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.545+0000 7f67817df740 1 client.7494 shutdown 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f67817df740 1 -- 172.21.15.44:0/974469030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 msgr2=0x55944a843f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f67817df740 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 0x55944a843f40 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f6764062440 tx=0x7f6764012000 comp rx=0 tx=0).stop 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f67817df740 1 -- 172.21.15.44:0/974469030 shutdown_connections 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f67817df740 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55944a8c44f0 0x55944a851600 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f67817df740 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55944a8454d0 0x55944a844480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f67817df740 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55944a83ddd0 0x55944a843f40 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f67817df740 1 --2- 172.21.15.44:0/974469030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55944a7fdbe0 0x55944a8c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f67577fe640 0 client.7494 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:52.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f67817df740 1 -- 172.21.15.44:0/974469030 >> 172.21.15.44:0/974469030 conn(0x55944a7f7dc0 msgr2=0x55944a7f85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f67817df740 1 -- 172.21.15.44:0/974469030 shutdown_connections 2024-04-02T13:25:52.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f67817df740 1 -- 172.21.15.44:0/974469030 wait complete. 2024-04-02T13:25:52.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f67817df740 20 client.7494 trim_cache size 0 max 16384 2024-04-02T13:25:52.554 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:52.554 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-04-02T13:25:52.554 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:25:52.554 INFO:teuthology.orchestra.run.smithi044.stdout: "damage_type": "dentry", 2024-04-02T13:25:52.554 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 3968587995, 2024-04-02T13:25:52.555 INFO:teuthology.orchestra.run.smithi044.stdout: "ino": 1099511627776, 2024-04-02T13:25:52.555 INFO:teuthology.orchestra.run.smithi044.stdout: "frag": "*", 2024-04-02T13:25:52.555 INFO:teuthology.orchestra.run.smithi044.stdout: "dname": "file_to_be_damaged", 2024-04-02T13:25:52.555 INFO:teuthology.orchestra.run.smithi044.stdout: "snap_id": "head", 2024-04-02T13:25:52.555 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/subdir/file_to_be_damaged" 2024-04-02T13:25:52.555 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:25:52.555 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-04-02T13:25:52.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f677eb22640 1 -- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f675c06c8f0 msgr2=0x7f675c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f677eb22640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f675c06c8f0 0x7f675c06edb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f676c0096f0 tx=0x7f676c008040 comp rx=0 tx=0).stop 2024-04-02T13:25:52.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f677eb22640 1 -- 172.21.15.44:0/1545265788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f677815e430 msgr2=0x7f67780add00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f677eb22640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f677815e430 0x7f67780add00 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f676400d800 tx=0x7f676400dcc0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f677eb22640 1 -- 172.21.15.44:0/1545265788 shutdown_connections 2024-04-02T13:25:52.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f677eb22640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f675c06c8f0 0x7f675c06edb0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f677eb22640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f677815e810 0x7f67780afc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f677eb22640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f677815e430 0x7f67780add00 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f677eb22640 1 --2- 172.21.15.44:0/1545265788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67780ca380 0x7f677814ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f677eb22640 1 -- 172.21.15.44:0/1545265788 >> 172.21.15.44:0/1545265788 conn(0x7f67780cfca0 msgr2=0x7f67780719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f677eb22640 1 -- 172.21.15.44:0/1545265788 shutdown_connections 2024-04-02T13:25:52.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.546+0000 7f677eb22640 1 -- 172.21.15.44:0/1545265788 wait complete. 2024-04-02T13:25:52.559 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-02T13:25:52.659 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:25:52.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.657+0000 7f56cdf04640 1 Processor -- start 2024-04-02T13:25:52.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.657+0000 7f56cdf04640 1 -- start start 2024-04-02T13:25:52.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56cdf04640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f56c80d15f0 0x7f56c80b2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56cdf04640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56c80d1e00 0x7f56c80d2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56cdf04640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f56c80ce8b0 0x7f56c80ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56cdf04640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f56c80b26d0 con 0x7f56c80d15f0 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56cdf04640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f56c80b1dc0 con 0x7f56c80ce8b0 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56cdf04640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f56c80cf600 con 0x7f56c80d1e00 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56c77fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f56c80d15f0 0x7f56c80b2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56c77fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f56c80d15f0 0x7f56c80b2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32820/0 (socket says 172.21.15.44:32820) 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56c77fe640 1 -- 172.21.15.44:0/2922178695 learned_addr learned my addr 172.21.15.44:0/2922178695 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56c7fff640 1 --2- 172.21.15.44:0/2922178695 >> v2:172.21.15.69:3300/0 conn(0x7f56c80ce8b0 0x7f56c80ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56c6ffd640 1 --2- 172.21.15.44:0/2922178695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56c80d1e00 0x7f56c80d2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56c77fe640 1 -- 172.21.15.44:0/2922178695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56c80d1e00 msgr2=0x7f56c80d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56c77fe640 1 --2- 172.21.15.44:0/2922178695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56c80d1e00 0x7f56c80d2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56c77fe640 1 -- 172.21.15.44:0/2922178695 >> v2:172.21.15.69:3300/0 conn(0x7f56c80ce8b0 msgr2=0x7f56c80ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56c77fe640 1 --2- 172.21.15.44:0/2922178695 >> v2:172.21.15.69:3300/0 conn(0x7f56c80ce8b0 0x7f56c80ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56c77fe640 1 -- 172.21.15.44:0/2922178695 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c80b2080 con 0x7f56c80d15f0 2024-04-02T13:25:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56c7fff640 1 --2- 172.21.15.44:0/2922178695 >> v2:172.21.15.69:3300/0 conn(0x7f56c80ce8b0 0x7f56c80ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.658+0000 7f56c6ffd640 1 --2- 172.21.15.44:0/2922178695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56c80d1e00 0x7f56c80d2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:25:52.663 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:25:52.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.659+0000 7f56c77fe640 1 --2- 172.21.15.44:0/2922178695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56c80d15f0 0x7f56c80b2b60 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f56c809b780 tx=0x7f56b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b6ebbdc1884ac server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.659+0000 7f56c67fc640 1 -- 172.21.15.44:0/2922178695 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8035020 con 0x7f56c80d15f0 2024-04-02T13:25:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.659+0000 7f56c67fc640 1 -- 172.21.15.44:0/2922178695 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b8031870 con 0x7f56c80d15f0 2024-04-02T13:25:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.659+0000 7f56c67fc640 1 -- 172.21.15.44:0/2922178695 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8031c40 con 0x7f56c80d15f0 2024-04-02T13:25:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.660+0000 7f56cdf04640 1 -- 172.21.15.44:0/2922178695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56c80d15f0 msgr2=0x7f56c80b2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.660+0000 7f56cdf04640 1 --2- 172.21.15.44:0/2922178695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56c80d15f0 0x7f56c80b2b60 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f56c809b780 tx=0x7f56b802b910 comp rx=0 tx=0).stop 2024-04-02T13:25:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.660+0000 7f56cdf04640 1 -- 172.21.15.44:0/2922178695 shutdown_connections 2024-04-02T13:25:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.660+0000 7f56cdf04640 1 --2- 172.21.15.44:0/2922178695 >> v2:172.21.15.69:3300/0 conn(0x7f56c80ce8b0 0x7f56c80ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.660+0000 7f56cdf04640 1 --2- 172.21.15.44:0/2922178695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56c80d1e00 0x7f56c80d2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.660+0000 7f56cdf04640 1 --2- 172.21.15.44:0/2922178695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56c80d15f0 0x7f56c80b2b60 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.660+0000 7f56cdf04640 1 -- 172.21.15.44:0/2922178695 >> 172.21.15.44:0/2922178695 conn(0x7f56c809aa20 msgr2=0x7f56c809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.660+0000 7f56cdf04640 1 -- 172.21.15.44:0/2922178695 shutdown_connections 2024-04-02T13:25:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.660+0000 7f56cdf04640 1 -- 172.21.15.44:0/2922178695 wait complete. 2024-04-02T13:25:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.661+0000 7f56cdf04640 1 Processor -- start 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.661+0000 7f56cdf04640 1 -- start start 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.662+0000 7f56cdf04640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56c80ce8b0 0x7f56c8159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.662+0000 7f56cdf04640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56c80d15f0 0x7f56c80b5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.662+0000 7f56cdf04640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56c80d1e00 0x7f56c80afad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.662+0000 7f56cdf04640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f56c80b30a0 con 0x7f56c80d1e00 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.662+0000 7f56cdf04640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f56c80d19d0 con 0x7f56c80ce8b0 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.662+0000 7f56cdf04640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f56c80b3220 con 0x7f56c80d15f0 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.662+0000 7f56c7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56c80d1e00 0x7f56c80afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.662+0000 7f56c7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56c80d1e00 0x7f56c80afad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32828/0 (socket says 172.21.15.44:32828) 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.662+0000 7f56c7fff640 1 -- 172.21.15.44:0/75759332 learned_addr learned my addr 172.21.15.44:0/75759332 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.662+0000 7f56c6ffd640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56c80d15f0 0x7f56c80b5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.662+0000 7f56c77fe640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56c80ce8b0 0x7f56c8159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.663+0000 7f56c7fff640 1 -- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56c80d15f0 msgr2=0x7f56c80b5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.663+0000 7f56c7fff640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56c80d15f0 0x7f56c80b5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.663+0000 7f56c7fff640 1 -- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56c80ce8b0 msgr2=0x7f56c8159e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.663+0000 7f56c7fff640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56c80ce8b0 0x7f56c8159e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.663+0000 7f56c7fff640 1 -- 172.21.15.44:0/75759332 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c80b4a90 con 0x7f56c80d1e00 2024-04-02T13:25:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.663+0000 7f56c7fff640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56c80d1e00 0x7f56c80afad0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f56c0007db0 tx=0x7f56c000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.663+0000 7f56affff640 1 -- 172.21.15.44:0/75759332 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c001a070 con 0x7f56c80d1e00 2024-04-02T13:25:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.663+0000 7f56affff640 1 -- 172.21.15.44:0/75759332 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56c0003990 con 0x7f56c80d1e00 2024-04-02T13:25:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.663+0000 7f56affff640 1 -- 172.21.15.44:0/75759332 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c0003d60 con 0x7f56c80d1e00 2024-04-02T13:25:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.663+0000 7f56cdf04640 1 -- 172.21.15.44:0/75759332 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c80c7ab0 con 0x7f56c80d1e00 2024-04-02T13:25:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.663+0000 7f56cdf04640 1 -- 172.21.15.44:0/75759332 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c80c8070 con 0x7f56c80d1e00 2024-04-02T13:25:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.664+0000 7f56cdf04640 1 -- 172.21.15.44:0/75759332 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c80711e0 con 0x7f56c80d1e00 2024-04-02T13:25:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.664+0000 7f56affff640 1 -- 172.21.15.44:0/75759332 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f56c001d070 con 0x7f56c80d1e00 2024-04-02T13:25:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.665+0000 7f56affff640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56b406c8f0 0x7f56b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.665+0000 7f56affff640 1 -- 172.21.15.44:0/75759332 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f56c008be80 con 0x7f56c80d1e00 2024-04-02T13:25:52.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.665+0000 7f56c77fe640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56b406c8f0 0x7f56b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:52.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.666+0000 7f56c77fe640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56b406c8f0 0x7f56b406edb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f56b8036000 tx=0x7f56b803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.670+0000 7f56affff640 1 -- 172.21.15.44:0/75759332 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f56c80711e0 con 0x7f56c80d1e00 2024-04-02T13:25:52.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.895+0000 7f56cdf04640 1 -- 172.21.15.44:0/75759332 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f56c80b5c70 con 0x7f56c80d1e00 2024-04-02T13:25:52.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.895+0000 7f56affff640 1 -- 172.21.15.44:0/75759332 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+164 (secure 0 0 0) 0x7f56c0010070 con 0x7f56c80d1e00 2024-04-02T13:25:52.897 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:25:52.898 INFO:teuthology.orchestra.run.smithi044.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-04-02T13:25:52.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.897+0000 7f56cdf04640 1 -- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56b406c8f0 msgr2=0x7f56b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.897+0000 7f56cdf04640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56b406c8f0 0x7f56b406edb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f56b8036000 tx=0x7f56b803c040 comp rx=0 tx=0).stop 2024-04-02T13:25:52.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.897+0000 7f56cdf04640 1 -- 172.21.15.44:0/75759332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56c80d1e00 msgr2=0x7f56c80afad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:52.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.897+0000 7f56cdf04640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56c80d1e00 0x7f56c80afad0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f56c0007db0 tx=0x7f56c000d4a0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.897+0000 7f56cdf04640 1 -- 172.21.15.44:0/75759332 shutdown_connections 2024-04-02T13:25:52.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.897+0000 7f56cdf04640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56b406c8f0 0x7f56b406edb0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.897+0000 7f56cdf04640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56c80d1e00 0x7f56c80afad0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.897+0000 7f56cdf04640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56c80d15f0 0x7f56c80b5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.897+0000 7f56cdf04640 1 --2- 172.21.15.44:0/75759332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56c80ce8b0 0x7f56c8159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:52.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.897+0000 7f56cdf04640 1 -- 172.21.15.44:0/75759332 >> 172.21.15.44:0/75759332 conn(0x7f56c809aa20 msgr2=0x7f56c80b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:52.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.897+0000 7f56cdf04640 1 -- 172.21.15.44:0/75759332 shutdown_connections 2024-04-02T13:25:52.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:52.897+0000 7f56cdf04640 1 -- 172.21.15.44:0/75759332 wait complete. 2024-04-02T13:25:52.910 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-02T13:25:52.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:52.910 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub start /subdir repair,recursive 2024-04-02T13:25:53.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.015+0000 7f05560eb640 1 Processor -- start 2024-04-02T13:25:53.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.015+0000 7f05560eb640 1 -- start start 2024-04-02T13:25:53.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f05560eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05501554c0 0x7f0550071120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f05560eb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f05500d0840 0x7f0550164c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f05560eb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05500cdc20 con 0x7f05500ce9a0 2024-04-02T13:25:53.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f05560eb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f05500cdec0 con 0x7f05500d0840 2024-04-02T13:25:53.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f05560eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f05500ce1a0 con 0x7f05501554c0 2024-04-02T13:25:53.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f054ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f05500d0840 0x7f0550164c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f054effd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f05500ce9a0 0x7f05500d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:54826/0 (socket says 172.21.15.44:54826) 2024-04-02T13:25:53.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f054f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05501554c0 0x7f0550071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f054effd640 1 -- 172.21.15.44:0/1232771314 learned_addr learned my addr 172.21.15.44:0/1232771314 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:53.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f054f7fe640 1 -- 172.21.15.44:0/1232771314 >> v2:172.21.15.69:3300/0 conn(0x7f05500d0840 msgr2=0x7f0550164c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f054f7fe640 1 --2- 172.21.15.44:0/1232771314 >> v2:172.21.15.69:3300/0 conn(0x7f05500d0840 0x7f0550164c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f054f7fe640 1 -- 172.21.15.44:0/1232771314 >> v1:172.21.15.69:6789/0 conn(0x7f05500ce9a0 legacy=0x7f05500d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f054f7fe640 1 -- 172.21.15.44:0/1232771314 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05500ce480 con 0x7f05501554c0 2024-04-02T13:25:53.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.016+0000 7f054ffff640 1 --2- 172.21.15.44:0/1232771314 >> v2:172.21.15.69:3300/0 conn(0x7f05500d0840 0x7f0550164c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:25:53.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.017+0000 7f054f7fe640 1 --2- 172.21.15.44:0/1232771314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05501554c0 0x7f0550071120 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f05400097a0 tx=0x7f054002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2658083c21a9bff9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.017+0000 7f054e7fc640 1 -- 172.21.15.44:0/1232771314 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0540035020 con 0x7f05501554c0 2024-04-02T13:25:53.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.017+0000 7f054e7fc640 1 -- 172.21.15.44:0/1232771314 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0540031830 con 0x7f05501554c0 2024-04-02T13:25:53.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.017+0000 7f054e7fc640 1 -- 172.21.15.44:0/1232771314 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0540031c00 con 0x7f05501554c0 2024-04-02T13:25:53.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.018+0000 7f05560eb640 1 -- 172.21.15.44:0/1232771314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05501554c0 msgr2=0x7f0550071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.018+0000 7f05560eb640 1 --2- 172.21.15.44:0/1232771314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05501554c0 0x7f0550071120 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f05400097a0 tx=0x7f054002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.018+0000 7f05560eb640 1 -- 172.21.15.44:0/1232771314 shutdown_connections 2024-04-02T13:25:53.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.018+0000 7f05560eb640 1 --2- 172.21.15.44:0/1232771314 >> v2:172.21.15.69:3300/0 conn(0x7f05500d0840 0x7f0550164c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.018+0000 7f05560eb640 1 --2- 172.21.15.44:0/1232771314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05501554c0 0x7f0550071120 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.018+0000 7f05560eb640 1 -- 172.21.15.44:0/1232771314 >> 172.21.15.44:0/1232771314 conn(0x7f055006b920 msgr2=0x7f055006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:53.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.018+0000 7f05560eb640 1 -- 172.21.15.44:0/1232771314 shutdown_connections 2024-04-02T13:25:53.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.018+0000 7f05560eb640 1 -- 172.21.15.44:0/1232771314 wait complete. 2024-04-02T13:25:53.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.019+0000 7f05560eb640 1 Processor -- start 2024-04-02T13:25:53.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.019+0000 7f05560eb640 1 -- start start 2024-04-02T13:25:53.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f05560eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500ce9a0 0x7f0550141580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f05560eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05500d0840 0x7f05500b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f05560eb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05501554c0 0x7f055015a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f05560eb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f055015c290 con 0x7f05501554c0 2024-04-02T13:25:53.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f05560eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0550071be0 con 0x7f05500d0840 2024-04-02T13:25:53.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f05560eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f05501421a0 con 0x7f05500ce9a0 2024-04-02T13:25:53.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f054ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05501554c0 0x7f055015a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f054ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05501554c0 0x7f055015a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32832/0 (socket says 172.21.15.44:32832) 2024-04-02T13:25:53.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f054ffff640 1 -- 172.21.15.44:0/2902777824 learned_addr learned my addr 172.21.15.44:0/2902777824 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:53.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f054effd640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05500d0840 0x7f05500b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f054f7fe640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500ce9a0 0x7f0550141580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f054ffff640 1 -- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500ce9a0 msgr2=0x7f0550141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f054ffff640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500ce9a0 0x7f0550141580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f054ffff640 1 -- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05500d0840 msgr2=0x7f05500b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f054ffff640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05500d0840 0x7f05500b19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f054ffff640 1 -- 172.21.15.44:0/2902777824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05500b2450 con 0x7f05501554c0 2024-04-02T13:25:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.020+0000 7f054f7fe640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500ce9a0 0x7f0550141580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:25:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.021+0000 7f054ffff640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05501554c0 0x7f055015a520 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f054800e580 tx=0x7f054800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.021+0000 7f0537fff640 1 -- 172.21.15.44:0/2902777824 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0548003b90 con 0x7f05501554c0 2024-04-02T13:25:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.021+0000 7f0537fff640 1 -- 172.21.15.44:0/2902777824 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0548003d50 con 0x7f05501554c0 2024-04-02T13:25:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.021+0000 7f05560eb640 1 -- 172.21.15.44:0/2902777824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0550142410 con 0x7f05501554c0 2024-04-02T13:25:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.021+0000 7f0537fff640 1 -- 172.21.15.44:0/2902777824 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0548004260 con 0x7f05501554c0 2024-04-02T13:25:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.021+0000 7f05560eb640 1 -- 172.21.15.44:0/2902777824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f055014ee70 con 0x7f05501554c0 2024-04-02T13:25:53.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.022+0000 7f0537fff640 1 -- 172.21.15.44:0/2902777824 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0548004400 con 0x7f05501554c0 2024-04-02T13:25:53.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.023+0000 7f0537fff640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f053c06c920 0x7f053c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.023+0000 7f0537fff640 1 -- 172.21.15.44:0/2902777824 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f054808ea50 con 0x7f05501554c0 2024-04-02T13:25:53.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.023+0000 7f054f7fe640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f053c06c920 0x7f053c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.023+0000 7f054f7fe640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f053c06c920 0x7f053c06ede0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0540036000 tx=0x7f054003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.024+0000 7f0558dab740 1 Processor -- start 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.024+0000 7f0558dab740 1 -- start start 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.024+0000 7f0558dab740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 0x55eea67157b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.024+0000 7f0558dab740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717ed0 0x55eea66cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.024+0000 7f0558dab740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea66f7430 0x55eea66cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.024+0000 7f0558dab740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eea65dad40 con 0x55eea6717af0 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.024+0000 7f0558dab740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55eea6633680 con 0x55eea6717ed0 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.024+0000 7f0558dab740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55eea65db9f0 con 0x55eea66f7430 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f054f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 0x55eea67157b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f054effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717ed0 0x55eea66cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f054f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 0x55eea67157b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32844/0 (socket says 172.21.15.44:32844) 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f054f7fe640 1 -- 172.21.15.44:0/1761637349 learned_addr learned my addr 172.21.15.44:0/1761637349 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f054ffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea66f7430 0x55eea66cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f054f7fe640 1 -- 172.21.15.44:0/1761637349 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea66f7430 msgr2=0x55eea66cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f054f7fe640 1 --2- 172.21.15.44:0/1761637349 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea66f7430 0x55eea66cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f054f7fe640 1 -- 172.21.15.44:0/1761637349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717ed0 msgr2=0x55eea66cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f054f7fe640 1 --2- 172.21.15.44:0/1761637349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717ed0 0x55eea66cf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f054f7fe640 1 -- 172.21.15.44:0/1761637349 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eea640fa50 con 0x55eea6717af0 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f054f7fe640 1 --2- 172.21.15.44:0/1761637349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 0x55eea67157b0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f0540009890 tx=0x7f0540030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c8059c36b0b6916 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f05357fa640 1 -- 172.21.15.44:0/1761637349 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0540035040 con 0x55eea6717af0 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f05357fa640 1 -- 172.21.15.44:0/1761637349 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05400449c0 con 0x55eea6717af0 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.025+0000 7f05357fa640 1 -- 172.21.15.44:0/1761637349 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0540044db0 con 0x55eea6717af0 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.026+0000 7f0558dab740 1 -- 172.21.15.44:0/1761637349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 msgr2=0x55eea67157b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.026+0000 7f0558dab740 1 --2- 172.21.15.44:0/1761637349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 0x55eea67157b0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f0540009890 tx=0x7f0540030000 comp rx=0 tx=0).stop 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.026+0000 7f0558dab740 1 -- 172.21.15.44:0/1761637349 shutdown_connections 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.026+0000 7f0558dab740 1 --2- 172.21.15.44:0/1761637349 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea66f7430 0x55eea66cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.026+0000 7f0558dab740 1 --2- 172.21.15.44:0/1761637349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717ed0 0x55eea66cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.026+0000 7f0558dab740 1 --2- 172.21.15.44:0/1761637349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 0x55eea67157b0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.026+0000 7f0558dab740 1 -- 172.21.15.44:0/1761637349 >> 172.21.15.44:0/1761637349 conn(0x55eea66cc280 msgr2=0x55eea6714e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:53.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.026+0000 7f0558dab740 1 -- 172.21.15.44:0/1761637349 shutdown_connections 2024-04-02T13:25:53.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.026+0000 7f0558dab740 1 -- 172.21.15.44:0/1761637349 wait complete. 2024-04-02T13:25:53.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.027+0000 7f0558dab740 1 -- start start 2024-04-02T13:25:53.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.027+0000 7f0558dab740 1 Processor -- start 2024-04-02T13:25:53.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f0558dab740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea6717af0 0x55eea66f3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f0558dab740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717ed0 0x55eea66f8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f0558dab740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea66f2080 0x55eea66f86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f0558dab740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eea65eed80 con 0x55eea66f2080 2024-04-02T13:25:53.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f0558dab740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55eea662f630 con 0x55eea6717ed0 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f0558dab740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55eea66ca8b0 con 0x55eea6717af0 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f0558dab740 10 client.0 _ll_register_callbacks cb 0x55eea65f3570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f054ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea66f2080 0x55eea66f86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f054ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea66f2080 0x55eea66f86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32858/0 (socket says 172.21.15.44:32858) 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f054ffff640 1 -- 172.21.15.44:0/318508900 learned_addr learned my addr 172.21.15.44:0/318508900 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f054effd640 1 --2- 172.21.15.44:0/318508900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717ed0 0x55eea66f8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f054f7fe640 1 --2- 172.21.15.44:0/318508900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea6717af0 0x55eea66f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f054ffff640 1 -- 172.21.15.44:0/318508900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea6717af0 msgr2=0x55eea66f3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f054ffff640 1 --2- 172.21.15.44:0/318508900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea6717af0 0x55eea66f3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f054ffff640 1 -- 172.21.15.44:0/318508900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717ed0 msgr2=0x55eea66f8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f054ffff640 1 --2- 172.21.15.44:0/318508900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717ed0 0x55eea66f8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f054ffff640 1 -- 172.21.15.44:0/318508900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eea63073e0 con 0x55eea66f2080 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f054ffff640 1 --2- 172.21.15.44:0/318508900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea66f2080 0x55eea66f86d0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f054800ef40 tx=0x7f054805bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4734e822a816111c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.028+0000 7f052e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f052e7fc640 1 -- 172.21.15.44:0/318508900 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05480650f0 con 0x55eea66f2080 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f052e7fc640 1 -- 172.21.15.44:0/318508900 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f0548064c60 con 0x55eea66f2080 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f0558dab740 1 -- 172.21.15.44:0/318508900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05400449c0 con 0x55eea66f2080 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f052e7fc640 1 -- 172.21.15.44:0/318508900 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f0548064e60 con 0x55eea66f2080 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f0558dab740 10 client.7509 fetch_fsmap learned FSMap version 51 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f0558dab740 1 -- 172.21.15.44:0/318508900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55eea64701e0 con 0x55eea66f2080 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f052e7fc640 1 -- 172.21.15.44:0/318508900 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05400449c0 con 0x55eea66f2080 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f052e7fc640 1 -- 172.21.15.44:0/318508900 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f054805ba10 con 0x55eea66f2080 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f0558dab740 10 client.7509 fetch_fsmap finished waiting for FSMap version 51 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f0558dab740 10 client.7509 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f0558dab740 20 client.7509 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f0558dab740 1 --2- 172.21.15.44:0/318508900 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55eea6710660 0x55eea6710a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f0558dab740 4 client.7509 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f0558dab740 1 -- 172.21.15.44:0/318508900 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eea6640bc0 con 0x55eea6710660 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.030+0000 7f054f7fe640 1 --2- 172.21.15.44:0/318508900 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55eea6710660 0x55eea6710a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.031+0000 7f052e7fc640 10 client.7509 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.031+0000 7f054f7fe640 1 --2- 172.21.15.44:0/318508900 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55eea6710660 0x55eea6710a40 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddc258958d85d152 server_cookie=63498a2b54f8e7d in_seq=0 out_seq=0 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.031+0000 7f052e7fc640 1 -- 172.21.15.44:0/318508900 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eea6640bc0 con 0x55eea6710660 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.031+0000 7f052e7fc640 10 client.7509 handle_command_reply: tid=0 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.031+0000 7f0558dab740 1 client.7509 shutdown 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.032+0000 7f0558dab740 1 -- 172.21.15.44:0/318508900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea66f2080 msgr2=0x55eea66f86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.032+0000 7f0558dab740 1 --2- 172.21.15.44:0/318508900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea66f2080 0x55eea66f86d0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f054800ef40 tx=0x7f054805bfe0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.032+0000 7f0558dab740 1 -- 172.21.15.44:0/318508900 shutdown_connections 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.032+0000 7f0558dab740 1 --2- 172.21.15.44:0/318508900 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55eea6710660 0x55eea6710a40 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.032+0000 7f0558dab740 1 --2- 172.21.15.44:0/318508900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea66f2080 0x55eea66f86d0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.032+0000 7f0558dab740 1 --2- 172.21.15.44:0/318508900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717ed0 0x55eea66f8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.032+0000 7f052e7fc640 0 client.7509 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.032+0000 7f0558dab740 1 --2- 172.21.15.44:0/318508900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea6717af0 0x55eea66f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.032+0000 7f0558dab740 1 -- 172.21.15.44:0/318508900 >> 172.21.15.44:0/318508900 conn(0x55eea66f7430 msgr2=0x55eea66c8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:53.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.032+0000 7f0558dab740 1 -- 172.21.15.44:0/318508900 shutdown_connections 2024-04-02T13:25:53.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.032+0000 7f0558dab740 1 -- 172.21.15.44:0/318508900 wait complete. 2024-04-02T13:25:53.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.032+0000 7f0558dab740 20 client.7509 trim_cache size 0 max 16384 2024-04-02T13:25:53.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.051+0000 7f0558dab740 1 Processor -- start 2024-04-02T13:25:53.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.051+0000 7f0558dab740 1 -- start start 2024-04-02T13:25:53.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.051+0000 7f0558dab740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea66f2080 0x55eea673ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f0558dab740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717af0 0x55eea673cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f0558dab740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717ed0 0x55eea66f0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f0558dab740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f05501fcca0 con 0x55eea6717ed0 2024-04-02T13:25:53.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f0558dab740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f05501fbd20 con 0x55eea6717af0 2024-04-02T13:25:53.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f0558dab740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f05500cd310 con 0x55eea66f2080 2024-04-02T13:25:53.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f054ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717ed0 0x55eea66f0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f054ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717ed0 0x55eea66f0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32860/0 (socket says 172.21.15.44:32860) 2024-04-02T13:25:53.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f054ffff640 1 -- 172.21.15.44:0/2638095645 learned_addr learned my addr 172.21.15.44:0/2638095645 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:53.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f054ffff640 1 -- 172.21.15.44:0/2638095645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea66f2080 msgr2=0x55eea673ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f054ffff640 1 --2- 172.21.15.44:0/2638095645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea66f2080 0x55eea673ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f054ffff640 1 -- 172.21.15.44:0/2638095645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717af0 msgr2=0x55eea673cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f054effd640 1 --2- 172.21.15.44:0/2638095645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717af0 0x55eea673cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f054ffff640 1 --2- 172.21.15.44:0/2638095645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717af0 0x55eea673cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f054ffff640 1 -- 172.21.15.44:0/2638095645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eea62e04a0 con 0x55eea6717ed0 2024-04-02T13:25:53.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f054ffff640 1 --2- 172.21.15.44:0/2638095645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717ed0 0x55eea66f0ed0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f0548058f40 tx=0x7f054805f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d76d47af3150e7d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f0534ff9640 1 -- 172.21.15.44:0/2638095645 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05480650f0 con 0x55eea6717ed0 2024-04-02T13:25:53.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f0534ff9640 1 -- 172.21.15.44:0/2638095645 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05480046f0 con 0x55eea6717ed0 2024-04-02T13:25:53.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.052+0000 7f0534ff9640 1 -- 172.21.15.44:0/2638095645 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054807a1e0 con 0x55eea6717ed0 2024-04-02T13:25:53.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.053+0000 7f0558dab740 1 -- 172.21.15.44:0/2638095645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717ed0 msgr2=0x55eea66f0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.053+0000 7f0558dab740 1 --2- 172.21.15.44:0/2638095645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717ed0 0x55eea66f0ed0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f0548058f40 tx=0x7f054805f000 comp rx=0 tx=0).stop 2024-04-02T13:25:53.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.053+0000 7f0558dab740 1 -- 172.21.15.44:0/2638095645 shutdown_connections 2024-04-02T13:25:53.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.053+0000 7f0558dab740 1 --2- 172.21.15.44:0/2638095645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717ed0 0x55eea66f0ed0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.053+0000 7f0558dab740 1 --2- 172.21.15.44:0/2638095645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea6717af0 0x55eea673cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.053+0000 7f0558dab740 1 --2- 172.21.15.44:0/2638095645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea66f2080 0x55eea673ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.053+0000 7f0558dab740 1 -- 172.21.15.44:0/2638095645 >> 172.21.15.44:0/2638095645 conn(0x55eea66f7430 msgr2=0x55eea66f24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:53.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.053+0000 7f0558dab740 1 -- 172.21.15.44:0/2638095645 shutdown_connections 2024-04-02T13:25:53.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.053+0000 7f0558dab740 1 -- 172.21.15.44:0/2638095645 wait complete. 2024-04-02T13:25:53.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.054+0000 7f0558dab740 1 -- start start 2024-04-02T13:25:53.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.054+0000 7f0558dab740 1 Processor -- start 2024-04-02T13:25:53.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.054+0000 7f0558dab740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea66f2080 0x55eea66fc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.054+0000 7f0558dab740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 0x55eea672b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.054+0000 7f0558dab740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea6717ed0 0x55eea672b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.054+0000 7f0558dab740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f05501fb6d0 con 0x55eea6717af0 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.054+0000 7f0558dab740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55eea66f65c0 con 0x55eea66f2080 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.054+0000 7f0558dab740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55eea66ca3e0 con 0x55eea6717ed0 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.054+0000 7f0558dab740 10 client.0 _ll_register_callbacks cb 0x7f0548058950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.054+0000 7f054effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 0x55eea672b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.054+0000 7f054effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 0x55eea672b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32862/0 (socket says 172.21.15.44:32862) 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.054+0000 7f054effd640 1 -- 172.21.15.44:0/1046810593 learned_addr learned my addr 172.21.15.44:0/1046810593 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f054f7fe640 1 --2- 172.21.15.44:0/1046810593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea66f2080 0x55eea66fc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f054effd640 1 -- 172.21.15.44:0/1046810593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea6717ed0 msgr2=0x55eea672b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f054effd640 1 --2- 172.21.15.44:0/1046810593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea6717ed0 0x55eea672b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f054effd640 1 -- 172.21.15.44:0/1046810593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea66f2080 msgr2=0x55eea66fc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f054effd640 1 --2- 172.21.15.44:0/1046810593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea66f2080 0x55eea66fc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f054effd640 1 -- 172.21.15.44:0/1046810593 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eea6728e80 con 0x55eea6717af0 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f054f7fe640 1 --2- 172.21.15.44:0/1046810593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea66f2080 0x55eea66fc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:53.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f054effd640 1 --2- 172.21.15.44:0/1046810593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 0x55eea672b0c0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f0544000f60 tx=0x7f0544016790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b61da517d023464 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f052e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f052e7fc640 1 -- 172.21.15.44:0/1046810593 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544017430 con 0x55eea6717af0 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f052e7fc640 1 -- 172.21.15.44:0/1046810593 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f054401ca70 con 0x55eea6717af0 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f0558dab740 1 -- 172.21.15.44:0/1046810593 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05480046f0 con 0x55eea6717af0 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f052e7fc640 1 -- 172.21.15.44:0/1046810593 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f0544026050 con 0x55eea6717af0 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f0558dab740 10 client.7515 fetch_fsmap learned FSMap version 51 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f0558dab740 1 -- 172.21.15.44:0/1046810593 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55eea66fce60 con 0x55eea6717af0 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f052e7fc640 1 -- 172.21.15.44:0/1046810593 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05480046f0 con 0x55eea6717af0 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f052e7fc640 1 -- 172.21.15.44:0/1046810593 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f0544029020 con 0x55eea6717af0 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f0558dab740 10 client.7515 fetch_fsmap finished waiting for FSMap version 51 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f0558dab740 10 client.7515 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f0558dab740 20 client.7515 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.055+0000 7f0558dab740 1 --2- 172.21.15.44:0/1046810593 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55eea66f5de0 0x55eea66ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.056+0000 7f0558dab740 4 client.7515 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair", "recursive"]}] 2024-04-02T13:25:53.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.056+0000 7f0558dab740 1 -- 172.21.15.44:0/1046810593 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair", "recursive"]}) -- 0x55eea66ce330 con 0x55eea66f5de0 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.056+0000 7f054f7fe640 1 --2- 172.21.15.44:0/1046810593 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55eea66f5de0 0x55eea66ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.056+0000 7f052e7fc640 10 client.7515 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.056+0000 7f054f7fe640 1 --2- 172.21.15.44:0/1046810593 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55eea66f5de0 0x55eea66ce990 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eba2392120242a60 server_cookie=1a584037c65f94dd in_seq=0 out_seq=0 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.056+0000 7f052e7fc640 1 -- 172.21.15.44:0/1046810593 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55eea66ce330 con 0x55eea66f5de0 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.056+0000 7f052e7fc640 10 client.7515 handle_command_reply: tid=0 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.056+0000 7f0558dab740 1 client.7515 shutdown 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f0558dab740 1 -- 172.21.15.44:0/1046810593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 msgr2=0x55eea672b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f0558dab740 1 --2- 172.21.15.44:0/1046810593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 0x55eea672b0c0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f0544000f60 tx=0x7f0544016790 comp rx=0 tx=0).stop 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f0558dab740 1 -- 172.21.15.44:0/1046810593 shutdown_connections 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f0558dab740 1 --2- 172.21.15.44:0/1046810593 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55eea66f5de0 0x55eea66ce990 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f0558dab740 1 --2- 172.21.15.44:0/1046810593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eea6717ed0 0x55eea672b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f0558dab740 1 --2- 172.21.15.44:0/1046810593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eea6717af0 0x55eea672b0c0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f052e7fc640 0 client.7515 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:53.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f0558dab740 1 --2- 172.21.15.44:0/1046810593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eea66f2080 0x55eea66fc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f0558dab740 1 -- 172.21.15.44:0/1046810593 >> 172.21.15.44:0/1046810593 conn(0x55eea66f7430 msgr2=0x55eea66cfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:53.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f0558dab740 1 -- 172.21.15.44:0/1046810593 shutdown_connections 2024-04-02T13:25:53.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f0558dab740 1 -- 172.21.15.44:0/1046810593 wait complete. 2024-04-02T13:25:53.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f0558dab740 20 client.7515 trim_cache size 0 max 16384 2024-04-02T13:25:53.065 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:25:53.065 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-02T13:25:53.065 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "0b3c1c70-4e39-46fc-9e70-ed1fb017c327", 2024-04-02T13:25:53.065 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-02T13:25:53.065 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:25:53.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f05560eb640 1 -- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f053c06c920 msgr2=0x7f053c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f05560eb640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f053c06c920 0x7f053c06ede0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0540036000 tx=0x7f054003c040 comp rx=0 tx=0).stop 2024-04-02T13:25:53.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f05560eb640 1 -- 172.21.15.44:0/2902777824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05501554c0 msgr2=0x7f055015a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f05560eb640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05501554c0 0x7f055015a520 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f054800e580 tx=0x7f054800ea40 comp rx=0 tx=0).stop 2024-04-02T13:25:53.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f05560eb640 1 -- 172.21.15.44:0/2902777824 shutdown_connections 2024-04-02T13:25:53.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f05560eb640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f053c06c920 0x7f053c06ede0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f05560eb640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05501554c0 0x7f055015a520 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f05560eb640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05500d0840 0x7f05500b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f05560eb640 1 --2- 172.21.15.44:0/2902777824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500ce9a0 0x7f0550141580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f05560eb640 1 -- 172.21.15.44:0/2902777824 >> 172.21.15.44:0/2902777824 conn(0x7f055006b920 msgr2=0x7f055006e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:53.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.057+0000 7f05560eb640 1 -- 172.21.15.44:0/2902777824 shutdown_connections 2024-04-02T13:25:53.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.058+0000 7f05560eb640 1 -- 172.21.15.44:0/2902777824 wait complete. 2024-04-02T13:25:53.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:25:53.070 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-04-02T13:25:53.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.171+0000 7f2ae071b640 1 Processor -- start 2024-04-02T13:25:53.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.171+0000 7f2ae071b640 1 -- start start 2024-04-02T13:25:53.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.172+0000 7f2ae071b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2ad81554c0 0x7f2ad8071120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.172+0000 7f2ae071b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ad80cdc20 con 0x7f2ad80ce9a0 2024-04-02T13:25:53.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.172+0000 7f2ae071b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ad80cdec0 con 0x7f2ad80d0840 2024-04-02T13:25:53.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.172+0000 7f2ae071b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2ad80ce1a0 con 0x7f2ad81554c0 2024-04-02T13:25:53.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.172+0000 7f2ade4b7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2ad81554c0 0x7f2ad8071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.172+0000 7f2addcb6640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f2ad80ce9a0 0x7f2ad80d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53038/0 (socket says 172.21.15.44:53038) 2024-04-02T13:25:53.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.172+0000 7f2addcb6640 1 -- 172.21.15.44:0/183549106 learned_addr learned my addr 172.21.15.44:0/183549106 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:25:53.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.172+0000 7f2ade4b7640 1 -- 172.21.15.44:0/183549106 >> v1:172.21.15.69:6789/0 conn(0x7f2ad80d0840 legacy=0x7f2ad8164c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.172+0000 7f2ade4b7640 1 -- 172.21.15.44:0/183549106 >> v1:172.21.15.44:6789/0 conn(0x7f2ad80ce9a0 legacy=0x7f2ad80d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.172+0000 7f2ade4b7640 1 -- 172.21.15.44:0/183549106 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad80ce480 con 0x7f2ad81554c0 2024-04-02T13:25:53.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.173+0000 7f2ade4b7640 1 --2- 172.21.15.44:0/183549106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ad81554c0 0x7f2ad8071120 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f2ad00097a0 tx=0x7f2ad002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85131a973e2348b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.173+0000 7f2add4b5640 1 -- 172.21.15.44:0/183549106 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0037070 con 0x7f2ad81554c0 2024-04-02T13:25:53.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.173+0000 7f2add4b5640 1 -- 172.21.15.44:0/183549106 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad0030710 con 0x7f2ad81554c0 2024-04-02T13:25:53.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.173+0000 7f2add4b5640 1 -- 172.21.15.44:0/183549106 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0030ae0 con 0x7f2ad81554c0 2024-04-02T13:25:53.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.174+0000 7f2ae071b640 1 -- 172.21.15.44:0/183549106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ad81554c0 msgr2=0x7f2ad8071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.174+0000 7f2ae071b640 1 --2- 172.21.15.44:0/183549106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ad81554c0 0x7f2ad8071120 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f2ad00097a0 tx=0x7f2ad002b790 comp rx=0 tx=0).stop 2024-04-02T13:25:53.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.174+0000 7f2ae071b640 1 -- 172.21.15.44:0/183549106 shutdown_connections 2024-04-02T13:25:53.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.174+0000 7f2ae071b640 1 --2- 172.21.15.44:0/183549106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ad81554c0 0x7f2ad8071120 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.174+0000 7f2ae071b640 1 -- 172.21.15.44:0/183549106 >> 172.21.15.44:0/183549106 conn(0x7f2ad806b920 msgr2=0x7f2ad806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:53.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.174+0000 7f2ae071b640 1 -- 172.21.15.44:0/183549106 shutdown_connections 2024-04-02T13:25:53.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.174+0000 7f2ae071b640 1 -- 172.21.15.44:0/183549106 wait complete. 2024-04-02T13:25:53.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.175+0000 7f2ae071b640 1 Processor -- start 2024-04-02T13:25:53.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.175+0000 7f2ae071b640 1 -- start start 2024-04-02T13:25:53.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2ae071b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ad80ce9a0 0x7f2ad814a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2ae071b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ad80d0840 0x7f2ad8156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2ae071b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ad81554c0 0x7f2ad809b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2ae071b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad815c290 con 0x7f2ad81554c0 2024-04-02T13:25:53.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2ae071b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad8071be0 con 0x7f2ad80ce9a0 2024-04-02T13:25:53.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2ae071b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2ad814ad00 con 0x7f2ad80d0840 2024-04-02T13:25:53.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2adecb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ad81554c0 0x7f2ad809b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2adecb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ad81554c0 0x7f2ad809b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32880/0 (socket says 172.21.15.44:32880) 2024-04-02T13:25:53.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2adecb8640 1 -- 172.21.15.44:0/1764559579 learned_addr learned my addr 172.21.15.44:0/1764559579 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:53.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2ade4b7640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ad80ce9a0 0x7f2ad814a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2addcb6640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ad80d0840 0x7f2ad8156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2adecb8640 1 -- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ad80d0840 msgr2=0x7f2ad8156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2adecb8640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ad80d0840 0x7f2ad8156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2adecb8640 1 -- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ad80ce9a0 msgr2=0x7f2ad814a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2adecb8640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ad80ce9a0 0x7f2ad814a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.176+0000 7f2adecb8640 1 -- 172.21.15.44:0/1764559579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad8157470 con 0x7f2ad81554c0 2024-04-02T13:25:53.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.177+0000 7f2adecb8640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ad81554c0 0x7f2ad809b170 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800c760 tx=0x7f2ac800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.177+0000 7f2ad6ffd640 1 -- 172.21.15.44:0/1764559579 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8003b70 con 0x7f2ad81554c0 2024-04-02T13:25:53.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.177+0000 7f2ad6ffd640 1 -- 172.21.15.44:0/1764559579 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ac8003d10 con 0x7f2ad81554c0 2024-04-02T13:25:53.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.177+0000 7f2ae071b640 1 -- 172.21.15.44:0/1764559579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad8157190 con 0x7f2ad81554c0 2024-04-02T13:25:53.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.177+0000 7f2ad6ffd640 1 -- 172.21.15.44:0/1764559579 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac800aa90 con 0x7f2ad81554c0 2024-04-02T13:25:53.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.177+0000 7f2ae071b640 1 -- 172.21.15.44:0/1764559579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad809c550 con 0x7f2ad81554c0 2024-04-02T13:25:53.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.178+0000 7f2ad6ffd640 1 -- 172.21.15.44:0/1764559579 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2ac800ac30 con 0x7f2ad81554c0 2024-04-02T13:25:53.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.179+0000 7f2ad6ffd640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2ac406c8f0 0x7f2ac406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.179+0000 7f2ad6ffd640 1 -- 172.21.15.44:0/1764559579 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f2ac808bab0 con 0x7f2ad81554c0 2024-04-02T13:25:53.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.179+0000 7f2ade4b7640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2ac406c8f0 0x7f2ac406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.179+0000 7f2ade4b7640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2ac406c8f0 0x7f2ac406edb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0034000 tx=0x7f2ad002f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.180+0000 7f2ae33d6740 1 Processor -- start 2024-04-02T13:25:53.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.180+0000 7f2ae33d6740 1 -- start start 2024-04-02T13:25:53.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.180+0000 7f2ae33d6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a19a0 0x55ad39274940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.180+0000 7f2ae33d6740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a1d80 0x55ad392a4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2ae33d6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad39274e80 0x55ad392a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2ae33d6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ad391de830 con 0x55ad39274e80 2024-04-02T13:25:53.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2ae33d6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ad39186820 con 0x55ad392a19a0 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2ae33d6740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ad391da7e0 con 0x55ad392a1d80 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2adecb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad39274e80 0x55ad392a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2adecb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad39274e80 0x55ad392a4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32894/0 (socket says 172.21.15.44:32894) 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2adecb8640 1 -- 172.21.15.44:0/3799970864 learned_addr learned my addr 172.21.15.44:0/3799970864 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2ade4b7640 1 --2- 172.21.15.44:0/3799970864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a19a0 0x55ad39274940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2addcb6640 1 --2- 172.21.15.44:0/3799970864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a1d80 0x55ad392a4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2adecb8640 1 -- 172.21.15.44:0/3799970864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a1d80 msgr2=0x55ad392a4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2adecb8640 1 --2- 172.21.15.44:0/3799970864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a1d80 0x55ad392a4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2adecb8640 1 -- 172.21.15.44:0/3799970864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a19a0 msgr2=0x55ad39274940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2adecb8640 1 --2- 172.21.15.44:0/3799970864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a19a0 0x55ad39274940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2adecb8640 1 -- 172.21.15.44:0/3799970864 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad38fbc7d0 con 0x55ad39274e80 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2adecb8640 1 --2- 172.21.15.44:0/3799970864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad39274e80 0x55ad392a4c70 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8055b10 tx=0x7f2ac8059010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e49e677f28793d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2abffff640 1 -- 172.21.15.44:0/3799970864 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8088070 con 0x55ad39274e80 2024-04-02T13:25:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2abffff640 1 -- 172.21.15.44:0/3799970864 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ac80621e0 con 0x55ad39274e80 2024-04-02T13:25:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.181+0000 7f2abffff640 1 -- 172.21.15.44:0/3799970864 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8055d20 con 0x55ad39274e80 2024-04-02T13:25:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.182+0000 7f2ae33d6740 1 -- 172.21.15.44:0/3799970864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad39274e80 msgr2=0x55ad392a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.182+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/3799970864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad39274e80 0x55ad392a4c70 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8055b10 tx=0x7f2ac8059010 comp rx=0 tx=0).stop 2024-04-02T13:25:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.182+0000 7f2ae33d6740 1 -- 172.21.15.44:0/3799970864 shutdown_connections 2024-04-02T13:25:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.182+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/3799970864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad39274e80 0x55ad392a4c70 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.182+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/3799970864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a1d80 0x55ad392a4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.182+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/3799970864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a19a0 0x55ad39274940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.182+0000 7f2ae33d6740 1 -- 172.21.15.44:0/3799970864 >> 172.21.15.44:0/3799970864 conn(0x55ad39276bb0 msgr2=0x55ad392bf730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.182+0000 7f2ae33d6740 1 -- 172.21.15.44:0/3799970864 shutdown_connections 2024-04-02T13:25:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.182+0000 7f2ae33d6740 1 -- 172.21.15.44:0/3799970864 wait complete. 2024-04-02T13:25:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.183+0000 7f2ae33d6740 1 -- start start 2024-04-02T13:25:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.183+0000 7f2ae33d6740 1 Processor -- start 2024-04-02T13:25:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.183+0000 7f2ae33d6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a19a0 0x55ad3929a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ae33d6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a1d80 0x55ad3929a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ae33d6740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a5ff0 0x55ad392a3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ae33d6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ad39279150 con 0x55ad392a19a0 2024-04-02T13:25:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ae33d6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ad39199d30 con 0x55ad392a1d80 2024-04-02T13:25:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ae33d6740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ad39278540 con 0x55ad392a5ff0 2024-04-02T13:25:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ae33d6740 10 client.0 _ll_register_callbacks cb 0x55ad39199720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ade4b7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a19a0 0x55ad3929a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ade4b7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a19a0 0x55ad3929a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32900/0 (socket says 172.21.15.44:32900) 2024-04-02T13:25:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ade4b7640 1 -- 172.21.15.44:0/1578876235 learned_addr learned my addr 172.21.15.44:0/1578876235 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2addcb6640 1 --2- 172.21.15.44:0/1578876235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a1d80 0x55ad3929a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2adecb8640 1 --2- 172.21.15.44:0/1578876235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a5ff0 0x55ad392a3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ade4b7640 1 -- 172.21.15.44:0/1578876235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a5ff0 msgr2=0x55ad392a3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ade4b7640 1 --2- 172.21.15.44:0/1578876235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a5ff0 0x55ad392a3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ade4b7640 1 -- 172.21.15.44:0/1578876235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a1d80 msgr2=0x55ad3929a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ade4b7640 1 --2- 172.21.15.44:0/1578876235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a1d80 0x55ad3929a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ade4b7640 1 -- 172.21.15.44:0/1578876235 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad38ef23f0 con 0x55ad392a19a0 2024-04-02T13:25:53.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ade4b7640 1 --2- 172.21.15.44:0/1578876235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a19a0 0x55ad3929a480 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f2ad002bda0 tx=0x7f2ad0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6890e38c5e8c470 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2abd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:53.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2abd7fa640 1 -- 172.21.15.44:0/1578876235 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0037040 con 0x55ad392a19a0 2024-04-02T13:25:53.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2abd7fa640 1 -- 172.21.15.44:0/1578876235 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f2ad0053cc0 con 0x55ad392a19a0 2024-04-02T13:25:53.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ae33d6740 1 -- 172.21.15.44:0/1578876235 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ac80621e0 con 0x55ad392a19a0 2024-04-02T13:25:53.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2abd7fa640 1 -- 172.21.15.44:0/1578876235 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f2ad003c810 con 0x55ad392a19a0 2024-04-02T13:25:53.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ae33d6740 10 client.7530 fetch_fsmap learned FSMap version 51 2024-04-02T13:25:53.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.184+0000 7f2ae33d6740 1 -- 172.21.15.44:0/1578876235 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55ad38eb20f0 con 0x55ad392a19a0 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.185+0000 7f2abd7fa640 1 -- 172.21.15.44:0/1578876235 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ac80621e0 con 0x55ad392a19a0 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.185+0000 7f2abd7fa640 1 -- 172.21.15.44:0/1578876235 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f2ad005a050 con 0x55ad392a19a0 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.185+0000 7f2ae33d6740 10 client.7530 fetch_fsmap finished waiting for FSMap version 51 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.185+0000 7f2ae33d6740 10 client.7530 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.185+0000 7f2ae33d6740 20 client.7530 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.185+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/1578876235 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ad3928e9d0 0x55ad392a41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.185+0000 7f2ae33d6740 4 client.7530 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.185+0000 7f2ae33d6740 1 -- 172.21.15.44:0/1578876235 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad391ebd70 con 0x55ad3928e9d0 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.185+0000 7f2addcb6640 1 --2- 172.21.15.44:0/1578876235 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ad3928e9d0 0x55ad392a41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.185+0000 7f2addcb6640 1 --2- 172.21.15.44:0/1578876235 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ad3928e9d0 0x55ad392a41d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20c77bf20e174ee0 server_cookie=c3fe3cdbc0a501e in_seq=0 out_seq=0 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.185+0000 7f2abd7fa640 10 client.7530 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2abd7fa640 1 -- 172.21.15.44:0/1578876235 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ad391ebd70 con 0x55ad3928e9d0 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2abd7fa640 10 client.7530 handle_command_reply: tid=0 2024-04-02T13:25:53.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2ae33d6740 1 client.7530 shutdown 2024-04-02T13:25:53.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2ae33d6740 1 -- 172.21.15.44:0/1578876235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a19a0 msgr2=0x55ad3929a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/1578876235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a19a0 0x55ad3929a480 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f2ad002bda0 tx=0x7f2ad0004730 comp rx=0 tx=0).stop 2024-04-02T13:25:53.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2ae33d6740 1 -- 172.21.15.44:0/1578876235 shutdown_connections 2024-04-02T13:25:53.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/1578876235 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ad3928e9d0 0x55ad392a41d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/1578876235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a5ff0 0x55ad392a3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/1578876235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a1d80 0x55ad3929a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2abd7fa640 0 client.7530 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:53.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/1578876235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a19a0 0x55ad3929a480 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2ae33d6740 1 -- 172.21.15.44:0/1578876235 >> 172.21.15.44:0/1578876235 conn(0x55ad39274e80 msgr2=0x55ad392c2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:53.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2ae33d6740 1 -- 172.21.15.44:0/1578876235 shutdown_connections 2024-04-02T13:25:53.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2ae33d6740 1 -- 172.21.15.44:0/1578876235 wait complete. 2024-04-02T13:25:53.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.186+0000 7f2ae33d6740 20 client.7530 trim_cache size 0 max 16384 2024-04-02T13:25:53.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.204+0000 7f2ae33d6740 1 Processor -- start 2024-04-02T13:25:53.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.204+0000 7f2ae33d6740 1 -- start start 2024-04-02T13:25:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2ae33d6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a19a0 0x55ad392a0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2ae33d6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 0x55ad392a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2ae33d6740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a5ff0 0x55ad392c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2ae33d6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad81fef60 con 0x55ad392a1d80 2024-04-02T13:25:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2ae33d6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad81fdf90 con 0x55ad392a19a0 2024-04-02T13:25:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2ae33d6740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2ad81f5fe0 con 0x55ad392a5ff0 2024-04-02T13:25:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2addcb6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 0x55ad392a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2addcb6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 0x55ad392a3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32902/0 (socket says 172.21.15.44:32902) 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2addcb6640 1 -- 172.21.15.44:0/1973915788 learned_addr learned my addr 172.21.15.44:0/1973915788 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2ade4b7640 1 --2- 172.21.15.44:0/1973915788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a19a0 0x55ad392a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2adecb8640 1 --2- 172.21.15.44:0/1973915788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a5ff0 0x55ad392c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2addcb6640 1 -- 172.21.15.44:0/1973915788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a5ff0 msgr2=0x55ad392c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2addcb6640 1 --2- 172.21.15.44:0/1973915788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a5ff0 0x55ad392c2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2addcb6640 1 -- 172.21.15.44:0/1973915788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a19a0 msgr2=0x55ad392a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2addcb6640 1 --2- 172.21.15.44:0/1973915788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a19a0 0x55ad392a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2addcb6640 1 -- 172.21.15.44:0/1973915788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad3928e400 con 0x55ad392a1d80 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2ade4b7640 1 --2- 172.21.15.44:0/1973915788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a19a0 0x55ad392a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.205+0000 7f2addcb6640 1 --2- 172.21.15.44:0/1973915788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 0x55ad392a3e60 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f2acc001bf0 tx=0x7f2acc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e45285263a1a91d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.206+0000 7f2abf7fe640 1 -- 172.21.15.44:0/1973915788 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc01ebc0 con 0x55ad392a1d80 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.206+0000 7f2abf7fe640 1 -- 172.21.15.44:0/1973915788 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2acc023070 con 0x55ad392a1d80 2024-04-02T13:25:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.206+0000 7f2abf7fe640 1 -- 172.21.15.44:0/1973915788 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0058b0 con 0x55ad392a1d80 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.206+0000 7f2ae33d6740 1 -- 172.21.15.44:0/1973915788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 msgr2=0x55ad392a3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.206+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/1973915788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 0x55ad392a3e60 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f2acc001bf0 tx=0x7f2acc0057f0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.206+0000 7f2ae33d6740 1 -- 172.21.15.44:0/1973915788 shutdown_connections 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.206+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/1973915788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a5ff0 0x55ad392c2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.206+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/1973915788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 0x55ad392a3e60 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.206+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/1973915788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a19a0 0x55ad392a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.206+0000 7f2ae33d6740 1 -- 172.21.15.44:0/1973915788 >> 172.21.15.44:0/1973915788 conn(0x55ad39274e80 msgr2=0x55ad39299be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.206+0000 7f2ae33d6740 1 -- 172.21.15.44:0/1973915788 shutdown_connections 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.206+0000 7f2ae33d6740 1 -- 172.21.15.44:0/1973915788 wait complete. 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.207+0000 7f2ae33d6740 1 -- start start 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.207+0000 7f2ae33d6740 1 Processor -- start 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2ae33d6740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a19a0 0x55ad39323930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2ae33d6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 0x55ad3927a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2ae33d6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a5ff0 0x55ad392d2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2ae33d6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad81fd940 con 0x55ad392a1d80 2024-04-02T13:25:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2ae33d6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ad392ceea0 con 0x55ad392a5ff0 2024-04-02T13:25:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2ae33d6740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ad392d8700 con 0x55ad392a19a0 2024-04-02T13:25:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2ae33d6740 10 client.0 _ll_register_callbacks cb 0x7f2ad0030d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:25:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2addcb6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 0x55ad3927a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2addcb6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 0x55ad3927a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32906/0 (socket says 172.21.15.44:32906) 2024-04-02T13:25:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2addcb6640 1 -- 172.21.15.44:0/2277238187 learned_addr learned my addr 172.21.15.44:0/2277238187 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:25:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2adecb8640 1 --2- 172.21.15.44:0/2277238187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a5ff0 0x55ad392d2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2addcb6640 1 -- 172.21.15.44:0/2277238187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a19a0 msgr2=0x55ad39323930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2addcb6640 1 --2- 172.21.15.44:0/2277238187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a19a0 0x55ad39323930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2addcb6640 1 -- 172.21.15.44:0/2277238187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a5ff0 msgr2=0x55ad392d2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:25:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2addcb6640 1 --2- 172.21.15.44:0/2277238187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a5ff0 0x55ad392d2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2addcb6640 1 -- 172.21.15.44:0/2277238187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad392cde00 con 0x55ad392a1d80 2024-04-02T13:25:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2adecb8640 1 --2- 172.21.15.44:0/2277238187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a5ff0 0x55ad392d2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:25:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2addcb6640 1 --2- 172.21.15.44:0/2277238187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 0x55ad3927a3a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f2acc005880 tx=0x7f2acc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a403c846d896ad4e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:25:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2abd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:25:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2abd7fa640 1 -- 172.21.15.44:0/2277238187 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc005cc0 con 0x55ad392a1d80 2024-04-02T13:25:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2abd7fa640 1 -- 172.21.15.44:0/2277238187 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f2acc0070d0 con 0x55ad392a1d80 2024-04-02T13:25:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.208+0000 7f2ae33d6740 1 -- 172.21.15.44:0/2277238187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2acc023070 con 0x55ad392a1d80 2024-04-02T13:25:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2abd7fa640 1 -- 172.21.15.44:0/2277238187 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f2acc035050 con 0x55ad392a1d80 2024-04-02T13:25:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2ae33d6740 10 client.7536 fetch_fsmap learned FSMap version 51 2024-04-02T13:25:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2ae33d6740 1 -- 172.21.15.44:0/2277238187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55ad392a40e0 con 0x55ad392a1d80 2024-04-02T13:25:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2abd7fa640 1 -- 172.21.15.44:0/2277238187 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2acc023070 con 0x55ad392a1d80 2024-04-02T13:25:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2abd7fa640 1 -- 172.21.15.44:0/2277238187 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f2acc038020 con 0x55ad392a1d80 2024-04-02T13:25:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2ae33d6740 10 client.7536 fetch_fsmap finished waiting for FSMap version 51 2024-04-02T13:25:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2ae33d6740 10 client.7536 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-02T13:25:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2ae33d6740 20 client.7536 populate_metadata read hostname 'smithi044' 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/2277238187 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ad392d1830 0x55ad392d1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2ae33d6740 4 client.7536 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2ae33d6740 1 -- 172.21.15.44:0/2277238187 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ad391e60c0 con 0x55ad392d1830 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2ade4b7640 1 --2- 172.21.15.44:0/2277238187 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ad392d1830 0x55ad392d1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2abd7fa640 10 client.7536 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2ade4b7640 1 --2- 172.21.15.44:0/2277238187 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ad392d1830 0x55ad392d1c60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3777e805e464dfee server_cookie=70ba005743f29bab in_seq=0 out_seq=0 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2abd7fa640 1 -- 172.21.15.44:0/2277238187 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x55ad391e60c0 con 0x55ad392d1830 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.209+0000 7f2abd7fa640 10 client.7536 handle_command_reply: tid=0 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae33d6740 1 client.7536 shutdown 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae33d6740 1 -- 172.21.15.44:0/2277238187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 msgr2=0x55ad3927a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/2277238187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 0x55ad3927a3a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f2acc005880 tx=0x7f2acc009040 comp rx=0 tx=0).stop 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae33d6740 1 -- 172.21.15.44:0/2277238187 shutdown_connections 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/2277238187 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x55ad392d1830 0x55ad392d1c60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/2277238187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ad392a5ff0 0x55ad392d2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/2277238187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad392a1d80 0x55ad3927a3a0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2abd7fa640 0 client.7536 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:25:53.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae33d6740 1 --2- 172.21.15.44:0/2277238187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ad392a19a0 0x55ad39323930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae33d6740 1 -- 172.21.15.44:0/2277238187 >> 172.21.15.44:0/2277238187 conn(0x55ad39274e80 msgr2=0x55ad3928d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:53.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae33d6740 1 -- 172.21.15.44:0/2277238187 shutdown_connections 2024-04-02T13:25:53.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae33d6740 1 -- 172.21.15.44:0/2277238187 wait complete. 2024-04-02T13:25:53.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae33d6740 20 client.7536 trim_cache size 0 max 16384 2024-04-02T13:25:53.219 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:25:53.219 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T13:25:53.219 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-02T13:25:53.220 INFO:teuthology.orchestra.run.smithi044.stdout: "0b3c1c70-4e39-46fc-9e70-ed1fb017c327": { 2024-04-02T13:25:53.220 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/subdir", 2024-04-02T13:25:53.220 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "0b3c1c70-4e39-46fc-9e70-ed1fb017c327", 2024-04-02T13:25:53.220 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,repair" 2024-04-02T13:25:53.220 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:25:53.220 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:25:53.220 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:25:53.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae071b640 1 -- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2ac406c8f0 msgr2=0x7f2ac406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae071b640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2ac406c8f0 0x7f2ac406edb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0034000 tx=0x7f2ad002f000 comp rx=0 tx=0).stop 2024-04-02T13:25:53.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae071b640 1 -- 172.21.15.44:0/1764559579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ad81554c0 msgr2=0x7f2ad809b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:25:53.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae071b640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ad81554c0 0x7f2ad809b170 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800c760 tx=0x7f2ac800cc20 comp rx=0 tx=0).stop 2024-04-02T13:25:53.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae071b640 1 -- 172.21.15.44:0/1764559579 shutdown_connections 2024-04-02T13:25:53.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae071b640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2ac406c8f0 0x7f2ac406edb0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae071b640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ad81554c0 0x7f2ad809b170 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae071b640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ad80d0840 0x7f2ad8156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae071b640 1 --2- 172.21.15.44:0/1764559579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ad80ce9a0 0x7f2ad814a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:25:53.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.210+0000 7f2ae071b640 1 -- 172.21.15.44:0/1764559579 >> 172.21.15.44:0/1764559579 conn(0x7f2ad806b920 msgr2=0x7f2ad806bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:25:53.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.211+0000 7f2ae071b640 1 -- 172.21.15.44:0/1764559579 shutdown_connections 2024-04-02T13:25:53.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:25:53.211+0000 7f2ae071b640 1 -- 172.21.15.44:0/1764559579 wait complete. 2024-04-02T13:25:53.223 INFO:tasks.cephfs.filesystem:scrub status for tag:0b3c1c70-4e39-46fc-9e70-ed1fb017c327 - {'path': '/subdir', 'tag': '0b3c1c70-4e39-46fc-9e70-ed1fb017c327', 'options': 'recursive,repair'} 2024-04-02T13:26:22.781 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:26:22.784 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:26:23.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:23.224 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-04-02T13:26:23.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.336+0000 7fde39aba640 1 Processor -- start 2024-04-02T13:26:23.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.336+0000 7fde39aba640 1 -- start start 2024-04-02T13:26:23.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.337+0000 7fde39aba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde3413ef60 0x7fde3406f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.337+0000 7fde39aba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fde3406d940 0x7fde34071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.337+0000 7fde39aba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde3406e300 con 0x7fde3406dda0 2024-04-02T13:26:23.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.337+0000 7fde39aba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fde3406e5e0 con 0x7fde3406d940 2024-04-02T13:26:23.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.337+0000 7fde39aba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fde3406e8c0 con 0x7fde3413ef60 2024-04-02T13:26:23.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.337+0000 7fde33fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fde3406dda0 0x7fde3413f6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48608/0 (socket says 172.21.15.44:48608) 2024-04-02T13:26:23.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.337+0000 7fde33fff640 1 -- 172.21.15.44:0/526964975 learned_addr learned my addr 172.21.15.44:0/526964975 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.337+0000 7fde32ffd640 1 --2- 172.21.15.44:0/526964975 >> v2:172.21.15.69:3300/0 conn(0x7fde3406d940 0x7fde34071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.337+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2281335824 0 0) 0x7fde3406e300 con 0x7fde3406dda0 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.337+0000 7fde337fe640 1 --2- 172.21.15.44:0/526964975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde3413ef60 0x7fde3406f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.337+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde1c003520 con 0x7fde3406dda0 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3690231427 0 0) 0x7fde1c003520 con 0x7fde3406dda0 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde3413ef60 msgr2=0x7fde3406f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 --2- 172.21.15.44:0/526964975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde3413ef60 0x7fde3406f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 >> v2:172.21.15.69:3300/0 conn(0x7fde3406d940 msgr2=0x7fde34071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 --2- 172.21.15.44:0/526964975 >> v2:172.21.15.69:3300/0 conn(0x7fde3406d940 0x7fde34071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3406eba0 con 0x7fde3406dda0 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fde2c0019f0 con 0x7fde3406dda0 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde32ffd640 1 --2- 172.21.15.44:0/526964975 >> v2:172.21.15.69:3300/0 conn(0x7fde3406d940 0x7fde34071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 >> v1:172.21.15.44:6789/0 conn(0x7fde3406dda0 legacy=0x7fde3413f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 --2- 172.21.15.44:0/526964975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c0039b0 0x7fde1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 --2- 172.21.15.44:0/526964975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde3413ef60 0x7fde1c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 --2- 172.21.15.44:0/526964975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c004d40 0x7fde1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fde3406e5e0 con 0x7fde1c004d40 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fde3406e8c0 con 0x7fde1c0039b0 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fde1c00b9f0 con 0x7fde3413ef60 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde33fff640 1 --2- 172.21.15.44:0/526964975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c004d40 0x7fde1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde337fe640 1 --2- 172.21.15.44:0/526964975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c0039b0 0x7fde1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde32ffd640 1 --2- 172.21.15.44:0/526964975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde3413ef60 0x7fde1c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde33fff640 1 -- 172.21.15.44:0/526964975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde3413ef60 msgr2=0x7fde1c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde33fff640 1 --2- 172.21.15.44:0/526964975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde3413ef60 0x7fde1c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde33fff640 1 -- 172.21.15.44:0/526964975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c0039b0 msgr2=0x7fde1c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde33fff640 1 --2- 172.21.15.44:0/526964975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c0039b0 0x7fde1c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde33fff640 1 -- 172.21.15.44:0/526964975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1c003520 con 0x7fde1c004d40 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.338+0000 7fde337fe640 1 --2- 172.21.15.44:0/526964975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c0039b0 0x7fde1c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.339+0000 7fde33fff640 1 --2- 172.21.15.44:0/526964975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c004d40 0x7fde1c0051c0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fde2c00a780 tx=0x7fde1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e8d2faf146e5c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.339+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c00a3d0 con 0x7fde1c004d40 2024-04-02T13:26:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.339+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde2c034650 con 0x7fde1c004d40 2024-04-02T13:26:23.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.339+0000 7fde327fc640 1 -- 172.21.15.44:0/526964975 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c034a40 con 0x7fde1c004d40 2024-04-02T13:26:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.340+0000 7fde39aba640 1 -- 172.21.15.44:0/526964975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c004d40 msgr2=0x7fde1c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.340+0000 7fde39aba640 1 --2- 172.21.15.44:0/526964975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c004d40 0x7fde1c0051c0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fde2c00a780 tx=0x7fde1c005830 comp rx=0 tx=0).stop 2024-04-02T13:26:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.340+0000 7fde33fff640 1 -- 172.21.15.44:0/526964975 reap_dead start 2024-04-02T13:26:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.340+0000 7fde39aba640 1 -- 172.21.15.44:0/526964975 shutdown_connections 2024-04-02T13:26:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.340+0000 7fde39aba640 1 -- 172.21.15.44:0/526964975 >> 172.21.15.44:0/526964975 conn(0x7fde340b5d90 msgr2=0x7fde3406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.340+0000 7fde39aba640 1 -- 172.21.15.44:0/526964975 shutdown_connections 2024-04-02T13:26:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.340+0000 7fde39aba640 1 -- 172.21.15.44:0/526964975 wait complete. 2024-04-02T13:26:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.341+0000 7fde39aba640 1 Processor -- start 2024-04-02T13:26:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.341+0000 7fde39aba640 1 -- start start 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde39aba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde340af870 0x7fde340ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde39aba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde3414cd90 0x7fde34158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde39aba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde34159200 0x7fde3414b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde39aba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fde34072720 con 0x7fde3414cd90 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde39aba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fde340725a0 con 0x7fde34159200 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde39aba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fde3406f3a0 con 0x7fde340af870 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde32ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde3414cd90 0x7fde34158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde32ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde3414cd90 0x7fde34158cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47142/0 (socket says 172.21.15.44:47142) 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde32ffd640 1 -- 172.21.15.44:0/449171956 learned_addr learned my addr 172.21.15.44:0/449171956 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde33fff640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde34159200 0x7fde3414b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde337fe640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde340af870 0x7fde340ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde32ffd640 1 -- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde340af870 msgr2=0x7fde340ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde32ffd640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde340af870 0x7fde340ac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde32ffd640 1 -- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde34159200 msgr2=0x7fde3414b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde32ffd640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde34159200 0x7fde3414b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde32ffd640 1 -- 172.21.15.44:0/449171956 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3414d300 con 0x7fde3414cd90 2024-04-02T13:26:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.342+0000 7fde33fff640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde34159200 0x7fde3414b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.343+0000 7fde32ffd640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde3414cd90 0x7fde34158cc0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fde28011d00 tx=0x7fde2800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.343+0000 7fde1bfff640 1 -- 172.21.15.44:0/449171956 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2800fd20 con 0x7fde3414cd90 2024-04-02T13:26:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.343+0000 7fde1bfff640 1 -- 172.21.15.44:0/449171956 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde28004020 con 0x7fde3414cd90 2024-04-02T13:26:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.343+0000 7fde1bfff640 1 -- 172.21.15.44:0/449171956 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde28004410 con 0x7fde3414cd90 2024-04-02T13:26:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.343+0000 7fde39aba640 1 -- 172.21.15.44:0/449171956 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3414d5e0 con 0x7fde3414cd90 2024-04-02T13:26:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.343+0000 7fde39aba640 1 -- 172.21.15.44:0/449171956 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde3414dba0 con 0x7fde3414cd90 2024-04-02T13:26:23.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.344+0000 7fde1bfff640 1 -- 172.21.15.44:0/449171956 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fde28012040 con 0x7fde3414cd90 2024-04-02T13:26:23.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.345+0000 7fde1bfff640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fde2006c8f0 0x7fde2006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.345+0000 7fde1bfff640 1 -- 172.21.15.44:0/449171956 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fde2808c400 con 0x7fde3414cd90 2024-04-02T13:26:23.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.345+0000 7fde337fe640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fde2006c8f0 0x7fde2006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.346+0000 7fde337fe640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fde2006c8f0 0x7fde2006edb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fde24009c20 tx=0x7fde24008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.346+0000 7fde3c777740 1 Processor -- start 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.346+0000 7fde3c777740 1 -- start start 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde3c777740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a769a0 0x5578a0a49940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde3c777740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a76d80 0x5578a0a79730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde3c777740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a49e80 0x5578a0a79c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde3c777740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5578a09b3830 con 0x5578a0a49e80 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde3c777740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5578a095b820 con 0x5578a0a76d80 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde3c777740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5578a09af7e0 con 0x5578a0a769a0 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde33fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a49e80 0x5578a0a79c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde33fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a49e80 0x5578a0a79c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47156/0 (socket says 172.21.15.44:47156) 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde33fff640 1 -- 172.21.15.44:0/511131290 learned_addr learned my addr 172.21.15.44:0/511131290 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde32ffd640 1 --2- 172.21.15.44:0/511131290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a76d80 0x5578a0a79730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde337fe640 1 --2- 172.21.15.44:0/511131290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a769a0 0x5578a0a49940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde33fff640 1 -- 172.21.15.44:0/511131290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a769a0 msgr2=0x5578a0a49940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde33fff640 1 --2- 172.21.15.44:0/511131290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a769a0 0x5578a0a49940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde33fff640 1 -- 172.21.15.44:0/511131290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a76d80 msgr2=0x5578a0a79730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde33fff640 1 --2- 172.21.15.44:0/511131290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a76d80 0x5578a0a79730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde33fff640 1 -- 172.21.15.44:0/511131290 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578a07917d0 con 0x5578a0a49e80 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde32ffd640 1 --2- 172.21.15.44:0/511131290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a76d80 0x5578a0a79730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde33fff640 1 --2- 172.21.15.44:0/511131290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a49e80 0x5578a0a79c70 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fde2c02cba0 tx=0x7fde2c007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2509df32c2120563 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde197fa640 1 -- 172.21.15.44:0/511131290 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c045070 con 0x5578a0a49e80 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.347+0000 7fde197fa640 1 -- 172.21.15.44:0/511131290 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde2c03a040 con 0x5578a0a49e80 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.348+0000 7fde197fa640 1 -- 172.21.15.44:0/511131290 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24009690 con 0x5578a0a49e80 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.348+0000 7fde3c777740 1 -- 172.21.15.44:0/511131290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a49e80 msgr2=0x5578a0a79c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.348+0000 7fde3c777740 1 --2- 172.21.15.44:0/511131290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a49e80 0x5578a0a79c70 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fde2c02cba0 tx=0x7fde2c007000 comp rx=0 tx=0).stop 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.348+0000 7fde3c777740 1 -- 172.21.15.44:0/511131290 shutdown_connections 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.348+0000 7fde3c777740 1 --2- 172.21.15.44:0/511131290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a49e80 0x5578a0a79c70 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.348+0000 7fde3c777740 1 --2- 172.21.15.44:0/511131290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a76d80 0x5578a0a79730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.348+0000 7fde3c777740 1 --2- 172.21.15.44:0/511131290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a769a0 0x5578a0a49940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.348+0000 7fde3c777740 1 -- 172.21.15.44:0/511131290 >> 172.21.15.44:0/511131290 conn(0x5578a0a4bbb0 msgr2=0x5578a0a94730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.348+0000 7fde3c777740 1 -- 172.21.15.44:0/511131290 shutdown_connections 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.348+0000 7fde3c777740 1 -- 172.21.15.44:0/511131290 wait complete. 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.349+0000 7fde3c777740 1 -- start start 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.349+0000 7fde3c777740 1 Processor -- start 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde3c777740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 0x5578a0a7a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde3c777740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a76d80 0x5578a0a796d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde3c777740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a8e700 0x5578a0aa4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde3c777740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5578a0a4e150 con 0x5578a0a769a0 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde3c777740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5578a096ed30 con 0x5578a0a8e700 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde3c777740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5578a0a4d540 con 0x5578a0a76d80 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde3c777740 10 client.0 _ll_register_callbacks cb 0x5578a096e720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde337fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 0x5578a0a7a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde337fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 0x5578a0a7a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47170/0 (socket says 172.21.15.44:47170) 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde337fe640 1 -- 172.21.15.44:0/3715862116 learned_addr learned my addr 172.21.15.44:0/3715862116 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde32ffd640 1 --2- 172.21.15.44:0/3715862116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a76d80 0x5578a0a796d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde33fff640 1 --2- 172.21.15.44:0/3715862116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a8e700 0x5578a0aa4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde337fe640 1 -- 172.21.15.44:0/3715862116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a76d80 msgr2=0x5578a0a796d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde337fe640 1 --2- 172.21.15.44:0/3715862116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a76d80 0x5578a0a796d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde337fe640 1 -- 172.21.15.44:0/3715862116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a8e700 msgr2=0x5578a0aa4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde337fe640 1 --2- 172.21.15.44:0/3715862116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a8e700 0x5578a0aa4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde337fe640 1 -- 172.21.15.44:0/3715862116 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578a06c73f0 con 0x5578a0a769a0 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde337fe640 1 --2- 172.21.15.44:0/3715862116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 0x5578a0a7a720 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fde240036a0 tx=0x7fde24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f8bb7e41d95d43 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde127fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde127fc640 1 -- 172.21.15.44:0/3715862116 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24029070 con 0x5578a0a769a0 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.350+0000 7fde127fc640 1 -- 172.21.15.44:0/3715862116 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fde24016780 con 0x5578a0a769a0 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.351+0000 7fde3c777740 1 -- 172.21.15.44:0/3715862116 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde2c03a040 con 0x5578a0a769a0 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.351+0000 7fde127fc640 1 -- 172.21.15.44:0/3715862116 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fde24034050 con 0x5578a0a769a0 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.351+0000 7fde3c777740 10 client.7548 fetch_fsmap learned FSMap version 52 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.351+0000 7fde3c777740 1 -- 172.21.15.44:0/3715862116 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x5578a06870f0 con 0x5578a0a769a0 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.351+0000 7fde127fc640 1 -- 172.21.15.44:0/3715862116 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde2c03a040 con 0x5578a0a769a0 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.351+0000 7fde127fc640 1 -- 172.21.15.44:0/3715862116 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7fde24037020 con 0x5578a0a769a0 2024-04-02T13:26:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.351+0000 7fde3c777740 10 client.7548 fetch_fsmap finished waiting for FSMap version 52 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.351+0000 7fde3c777740 10 client.7548 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.351+0000 7fde3c777740 20 client.7548 populate_metadata read hostname 'smithi044' 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.351+0000 7fde3c777740 1 --2- 172.21.15.44:0/3715862116 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5578a0a8cd00 0x5578a0a8d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.351+0000 7fde3c777740 4 client.7548 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.351+0000 7fde3c777740 1 -- 172.21.15.44:0/3715862116 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578a09c0d70 con 0x5578a0a8cd00 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.351+0000 7fde32ffd640 1 --2- 172.21.15.44:0/3715862116 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5578a0a8cd00 0x5578a0a8d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde127fc640 10 client.7548 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde32ffd640 1 --2- 172.21.15.44:0/3715862116 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5578a0a8cd00 0x5578a0a8d130 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f49d6aafc31cb3ed server_cookie=69fd1cb1a8dad4c5 in_seq=0 out_seq=0 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde127fc640 1 -- 172.21.15.44:0/3715862116 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5578a09c0d70 con 0x5578a0a8cd00 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde127fc640 10 client.7548 handle_command_reply: tid=0 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde3c777740 1 client.7548 shutdown 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde3c777740 1 -- 172.21.15.44:0/3715862116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 msgr2=0x5578a0a7a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde3c777740 1 --2- 172.21.15.44:0/3715862116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 0x5578a0a7a720 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fde240036a0 tx=0x7fde24004730 comp rx=0 tx=0).stop 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde3c777740 1 -- 172.21.15.44:0/3715862116 shutdown_connections 2024-04-02T13:26:23.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde3c777740 1 --2- 172.21.15.44:0/3715862116 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5578a0a8cd00 0x5578a0a8d130 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde3c777740 1 --2- 172.21.15.44:0/3715862116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a8e700 0x5578a0aa4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde3c777740 1 --2- 172.21.15.44:0/3715862116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a76d80 0x5578a0a796d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde127fc640 0 client.7548 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:26:23.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde3c777740 1 --2- 172.21.15.44:0/3715862116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 0x5578a0a7a720 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde3c777740 1 -- 172.21.15.44:0/3715862116 >> 172.21.15.44:0/3715862116 conn(0x5578a0a49e80 msgr2=0x5578a0a97bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde3c777740 1 -- 172.21.15.44:0/3715862116 shutdown_connections 2024-04-02T13:26:23.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.352+0000 7fde3c777740 1 -- 172.21.15.44:0/3715862116 wait complete. 2024-04-02T13:26:23.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.353+0000 7fde3c777740 20 client.7548 trim_cache size 0 max 16384 2024-04-02T13:26:23.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.371+0000 7fde3c777740 1 Processor -- start 2024-04-02T13:26:23.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.371+0000 7fde3c777740 1 -- start start 2024-04-02T13:26:23.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.371+0000 7fde3c777740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 0x5578a0ab7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.371+0000 7fde3c777740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a76d80 0x5578a0ab77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.371+0000 7fde3c777740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a8e700 0x5578a0a75200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.371+0000 7fde3c777740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fde341ff800 con 0x5578a0a769a0 2024-04-02T13:26:23.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.371+0000 7fde3c777740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fde341fe8a0 con 0x5578a0a8e700 2024-04-02T13:26:23.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.371+0000 7fde3c777740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fde341fdc00 con 0x5578a0a76d80 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde337fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 0x5578a0ab7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde337fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 0x5578a0ab7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47184/0 (socket says 172.21.15.44:47184) 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde337fe640 1 -- 172.21.15.44:0/2760413872 learned_addr learned my addr 172.21.15.44:0/2760413872 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde33fff640 1 --2- 172.21.15.44:0/2760413872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a8e700 0x5578a0a75200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde32ffd640 1 --2- 172.21.15.44:0/2760413872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a76d80 0x5578a0ab77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde337fe640 1 -- 172.21.15.44:0/2760413872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a76d80 msgr2=0x5578a0ab77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde337fe640 1 --2- 172.21.15.44:0/2760413872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a76d80 0x5578a0ab77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde337fe640 1 -- 172.21.15.44:0/2760413872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a8e700 msgr2=0x5578a0a75200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde337fe640 1 --2- 172.21.15.44:0/2760413872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a8e700 0x5578a0a75200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde337fe640 1 -- 172.21.15.44:0/2760413872 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578a0a985d0 con 0x5578a0a769a0 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde337fe640 1 --2- 172.21.15.44:0/2760413872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 0x5578a0ab7260 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fde24008010 tx=0x7fde24026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4354c859d9090208 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde18ff9640 1 -- 172.21.15.44:0/2760413872 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24029040 con 0x5578a0a769a0 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde18ff9640 1 -- 172.21.15.44:0/2760413872 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde240169d0 con 0x5578a0a769a0 2024-04-02T13:26:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.372+0000 7fde18ff9640 1 -- 172.21.15.44:0/2760413872 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2402e440 con 0x5578a0a769a0 2024-04-02T13:26:23.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.373+0000 7fde3c777740 1 -- 172.21.15.44:0/2760413872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 msgr2=0x5578a0ab7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.373+0000 7fde3c777740 1 --2- 172.21.15.44:0/2760413872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 0x5578a0ab7260 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fde24008010 tx=0x7fde24026000 comp rx=0 tx=0).stop 2024-04-02T13:26:23.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.373+0000 7fde3c777740 1 -- 172.21.15.44:0/2760413872 shutdown_connections 2024-04-02T13:26:23.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.373+0000 7fde3c777740 1 --2- 172.21.15.44:0/2760413872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a8e700 0x5578a0a75200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.373+0000 7fde3c777740 1 --2- 172.21.15.44:0/2760413872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a76d80 0x5578a0ab77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.373+0000 7fde3c777740 1 --2- 172.21.15.44:0/2760413872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a769a0 0x5578a0ab7260 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.373+0000 7fde3c777740 1 -- 172.21.15.44:0/2760413872 >> 172.21.15.44:0/2760413872 conn(0x5578a0a49e80 msgr2=0x5578a0a8c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.373+0000 7fde3c777740 1 -- 172.21.15.44:0/2760413872 shutdown_connections 2024-04-02T13:26:23.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.373+0000 7fde3c777740 1 -- 172.21.15.44:0/2760413872 wait complete. 2024-04-02T13:26:23.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.374+0000 7fde3c777740 1 -- start start 2024-04-02T13:26:23.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.374+0000 7fde3c777740 1 Processor -- start 2024-04-02T13:26:23.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.374+0000 7fde3c777740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a769a0 0x5578a0ab7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.374+0000 7fde3c777740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a76d80 0x5578a0ab77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.374+0000 7fde3c777740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a8e700 0x5578a0a96b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.374+0000 7fde3c777740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5578a0a7a720 con 0x5578a0a8e700 2024-04-02T13:26:23.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.374+0000 7fde3c777740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fde341fe250 con 0x5578a0a76d80 2024-04-02T13:26:23.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.374+0000 7fde3c777740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5578a0a798c0 con 0x5578a0a769a0 2024-04-02T13:26:23.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.374+0000 7fde3c777740 10 client.0 _ll_register_callbacks cb 0x7fde240043a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:26:23.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde33fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a8e700 0x5578a0a96b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde33fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a8e700 0x5578a0a96b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47196/0 (socket says 172.21.15.44:47196) 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde33fff640 1 -- 172.21.15.44:0/3616132368 learned_addr learned my addr 172.21.15.44:0/3616132368 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde32ffd640 1 --2- 172.21.15.44:0/3616132368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a76d80 0x5578a0ab77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde337fe640 1 --2- 172.21.15.44:0/3616132368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a769a0 0x5578a0ab7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde33fff640 1 -- 172.21.15.44:0/3616132368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a769a0 msgr2=0x5578a0ab7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde33fff640 1 --2- 172.21.15.44:0/3616132368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a769a0 0x5578a0ab7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde33fff640 1 -- 172.21.15.44:0/3616132368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a76d80 msgr2=0x5578a0ab77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde33fff640 1 --2- 172.21.15.44:0/3616132368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a76d80 0x5578a0ab77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde33fff640 1 -- 172.21.15.44:0/3616132368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578a0aa38e0 con 0x5578a0a8e700 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde33fff640 1 --2- 172.21.15.44:0/3616132368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a8e700 0x5578a0a96b90 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fde2c001350 tx=0x7fde2c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c6382f260fb44d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde127fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde127fc640 1 -- 172.21.15.44:0/3616132368 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c04f030 con 0x5578a0a8e700 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde127fc640 1 -- 172.21.15.44:0/3616132368 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fde2c039040 con 0x5578a0a8e700 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde3c777740 1 -- 172.21.15.44:0/3616132368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde240169d0 con 0x5578a0a8e700 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde127fc640 1 -- 172.21.15.44:0/3616132368 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fde2c05b050 con 0x5578a0a8e700 2024-04-02T13:26:23.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde3c777740 10 client.7554 fetch_fsmap learned FSMap version 52 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.375+0000 7fde3c777740 1 -- 172.21.15.44:0/3616132368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x5578a0aadff0 con 0x5578a0a8e700 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde127fc640 1 -- 172.21.15.44:0/3616132368 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde240169d0 con 0x5578a0a8e700 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde127fc640 1 -- 172.21.15.44:0/3616132368 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7fde2c05e020 con 0x5578a0a8e700 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 10 client.7554 fetch_fsmap finished waiting for FSMap version 52 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 10 client.7554 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 20 client.7554 populate_metadata read hostname 'smithi044' 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 1 --2- 172.21.15.44:0/3616132368 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5578a0a451f0 0x5578a0a970d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 4 client.7554 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 1 -- 172.21.15.44:0/3616132368 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5578a09bb0c0 con 0x5578a0a451f0 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde337fe640 1 --2- 172.21.15.44:0/3616132368 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5578a0a451f0 0x5578a0a970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde337fe640 1 --2- 172.21.15.44:0/3616132368 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5578a0a451f0 0x5578a0a970d0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fbd995c924a02a1 server_cookie=fb9831129b1ebcb4 in_seq=0 out_seq=0 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde127fc640 10 client.7554 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde127fc640 1 -- 172.21.15.44:0/3616132368 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5578a09bb0c0 con 0x5578a0a451f0 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde127fc640 10 client.7554 handle_command_reply: tid=0 2024-04-02T13:26:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 1 client.7554 shutdown 2024-04-02T13:26:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 1 -- 172.21.15.44:0/3616132368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a8e700 msgr2=0x5578a0a96b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 1 --2- 172.21.15.44:0/3616132368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a8e700 0x5578a0a96b90 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fde2c001350 tx=0x7fde2c02e000 comp rx=0 tx=0).stop 2024-04-02T13:26:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 1 -- 172.21.15.44:0/3616132368 shutdown_connections 2024-04-02T13:26:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 1 --2- 172.21.15.44:0/3616132368 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x5578a0a451f0 0x5578a0a970d0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 1 --2- 172.21.15.44:0/3616132368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578a0a8e700 0x5578a0a96b90 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde127fc640 0 client.7554 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:26:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 1 --2- 172.21.15.44:0/3616132368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578a0a76d80 0x5578a0ab77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 1 --2- 172.21.15.44:0/3616132368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578a0a769a0 0x5578a0ab7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.376+0000 7fde3c777740 1 -- 172.21.15.44:0/3616132368 >> 172.21.15.44:0/3616132368 conn(0x5578a0a49e80 msgr2=0x5578a0a96140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde3c777740 1 -- 172.21.15.44:0/3616132368 shutdown_connections 2024-04-02T13:26:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde3c777740 1 -- 172.21.15.44:0/3616132368 wait complete. 2024-04-02T13:26:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde3c777740 20 client.7554 trim_cache size 0 max 16384 2024-04-02T13:26:23.386 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:26:23.386 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-02T13:26:23.386 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-02T13:26:23.386 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:26:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde39aba640 1 -- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fde2006c8f0 msgr2=0x7fde2006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde39aba640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fde2006c8f0 0x7fde2006edb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fde24009c20 tx=0x7fde24008040 comp rx=0 tx=0).stop 2024-04-02T13:26:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde39aba640 1 -- 172.21.15.44:0/449171956 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde3414cd90 msgr2=0x7fde34158cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde39aba640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde3414cd90 0x7fde34158cc0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fde28011d00 tx=0x7fde2800f610 comp rx=0 tx=0).stop 2024-04-02T13:26:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde39aba640 1 -- 172.21.15.44:0/449171956 shutdown_connections 2024-04-02T13:26:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde39aba640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fde2006c8f0 0x7fde2006edb0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde39aba640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde34159200 0x7fde3414b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde39aba640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde3414cd90 0x7fde34158cc0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde39aba640 1 --2- 172.21.15.44:0/449171956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde340af870 0x7fde340ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde39aba640 1 -- 172.21.15.44:0/449171956 >> 172.21.15.44:0/449171956 conn(0x7fde340b5d90 msgr2=0x7fde340b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde39aba640 1 -- 172.21.15.44:0/449171956 shutdown_connections 2024-04-02T13:26:23.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.377+0000 7fde39aba640 1 -- 172.21.15.44:0/449171956 wait complete. 2024-04-02T13:26:23.391 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T13:26:23.391 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:26:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.489+0000 7f31df255640 1 Processor -- start 2024-04-02T13:26:23.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.489+0000 7f31df255640 1 -- start start 2024-04-02T13:26:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.490+0000 7f31df255640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f31d80d0a30 0x7f31d806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.490+0000 7f31df255640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31d80d0e10 0x7f31d80cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.490+0000 7f31df255640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31d806c1d0 con 0x7f31d80cfe30 2024-04-02T13:26:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.490+0000 7f31df255640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f31d806cb00 con 0x7f31d80d0a30 2024-04-02T13:26:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.490+0000 7f31df255640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f31d806c4b0 con 0x7f31d80d0e10 2024-04-02T13:26:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.490+0000 7f31dcff1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f31d80d0a30 0x7f31d806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.490+0000 7f31dd7f2640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f31d80cfe30 0x7f31d80d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48612/0 (socket says 172.21.15.44:48612) 2024-04-02T13:26:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.490+0000 7f31dd7f2640 1 -- 172.21.15.44:0/2988059852 learned_addr learned my addr 172.21.15.44:0/2988059852 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.491+0000 7f31dcff1640 1 -- 172.21.15.44:0/2988059852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31d80d0e10 msgr2=0x7f31d80cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.491+0000 7f31dcff1640 1 --2- 172.21.15.44:0/2988059852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31d80d0e10 0x7f31d80cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.491+0000 7f31dcff1640 1 -- 172.21.15.44:0/2988059852 >> v1:172.21.15.44:6789/0 conn(0x7f31d80cfe30 legacy=0x7f31d80d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.491+0000 7f31dcff1640 1 -- 172.21.15.44:0/2988059852 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d806c790 con 0x7f31d80d0a30 2024-04-02T13:26:23.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.491+0000 7f31d77fe640 1 -- 172.21.15.44:0/2988059852 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346968179 0 0) 0x7f31d806c1d0 con 0x7f31d80cfe30 2024-04-02T13:26:23.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.491+0000 7f31dcff1640 1 --2- 172.21.15.44:0/2988059852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d80d0a30 0x7f31d806cf90 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009d90 tx=0x7f31d002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78359e542f86c57a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.491+0000 7f31d77fe640 1 -- 172.21.15.44:0/2988059852 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d0039070 con 0x7f31d80d0a30 2024-04-02T13:26:23.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.491+0000 7f31d77fe640 1 -- 172.21.15.44:0/2988059852 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31d002cbc0 con 0x7f31d80d0a30 2024-04-02T13:26:23.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.492+0000 7f31d77fe640 1 -- 172.21.15.44:0/2988059852 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d0034440 con 0x7f31d80d0a30 2024-04-02T13:26:23.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.492+0000 7f31df255640 1 -- 172.21.15.44:0/2988059852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d80d0a30 msgr2=0x7f31d806cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.492+0000 7f31df255640 1 --2- 172.21.15.44:0/2988059852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d80d0a30 0x7f31d806cf90 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009d90 tx=0x7f31d002be10 comp rx=0 tx=0).stop 2024-04-02T13:26:23.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.492+0000 7f31df255640 1 -- 172.21.15.44:0/2988059852 shutdown_connections 2024-04-02T13:26:23.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.492+0000 7f31df255640 1 --2- 172.21.15.44:0/2988059852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31d80d0e10 0x7f31d80cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.492+0000 7f31df255640 1 --2- 172.21.15.44:0/2988059852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d80d0a30 0x7f31d806cf90 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.492+0000 7f31df255640 1 -- 172.21.15.44:0/2988059852 >> 172.21.15.44:0/2988059852 conn(0x7f31d809d470 msgr2=0x7f31d80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.492+0000 7f31df255640 1 -- 172.21.15.44:0/2988059852 shutdown_connections 2024-04-02T13:26:23.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.492+0000 7f31df255640 1 -- 172.21.15.44:0/2988059852 wait complete. 2024-04-02T13:26:23.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.493+0000 7f31df255640 1 Processor -- start 2024-04-02T13:26:23.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.493+0000 7f31df255640 1 -- start start 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.494+0000 7f31df255640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31d80cfe30 0x7f31d80c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.494+0000 7f31df255640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d80d0a30 0x7f31d80b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.494+0000 7f31df255640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31d80d0e10 0x7f31d80b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.494+0000 7f31df255640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f31d815e870 con 0x7f31d80d0a30 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.494+0000 7f31df255640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31d815e6f0 con 0x7f31d80d0e10 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.494+0000 7f31df255640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f31d80d4da0 con 0x7f31d80cfe30 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.494+0000 7f31d7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d80d0a30 0x7f31d80b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.494+0000 7f31d7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d80d0a30 0x7f31d80b6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47202/0 (socket says 172.21.15.44:47202) 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.494+0000 7f31d7fff640 1 -- 172.21.15.44:0/2022258164 learned_addr learned my addr 172.21.15.44:0/2022258164 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.494+0000 7f31dcff1640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31d80cfe30 0x7f31d80c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.494+0000 7f31dd7f2640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31d80d0e10 0x7f31d80b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.495+0000 7f31d7fff640 1 -- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31d80cfe30 msgr2=0x7f31d80c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.495+0000 7f31d7fff640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31d80cfe30 0x7f31d80c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.495+0000 7f31d7fff640 1 -- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31d80d0e10 msgr2=0x7f31d80b72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.495+0000 7f31d7fff640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31d80d0e10 0x7f31d80b72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.495+0000 7f31d7fff640 1 -- 172.21.15.44:0/2022258164 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d80c8f00 con 0x7f31d80d0a30 2024-04-02T13:26:23.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.495+0000 7f31d7fff640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d80d0a30 0x7f31d80b6d60 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00b410 tx=0x7f31cc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.495+0000 7f31d57fa640 1 -- 172.21.15.44:0/2022258164 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc015020 con 0x7f31d80d0a30 2024-04-02T13:26:23.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.495+0000 7f31d57fa640 1 -- 172.21.15.44:0/2022258164 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31cc011830 con 0x7f31d80d0a30 2024-04-02T13:26:23.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.495+0000 7f31df255640 1 -- 172.21.15.44:0/2022258164 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d80c91e0 con 0x7f31d80d0a30 2024-04-02T13:26:23.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.495+0000 7f31d57fa640 1 -- 172.21.15.44:0/2022258164 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc011c20 con 0x7f31d80d0a30 2024-04-02T13:26:23.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.495+0000 7f31df255640 1 -- 172.21.15.44:0/2022258164 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d80c97a0 con 0x7f31d80d0a30 2024-04-02T13:26:23.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.496+0000 7f31df255640 1 -- 172.21.15.44:0/2022258164 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31d80684f0 con 0x7f31d80d0a30 2024-04-02T13:26:23.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.496+0000 7f31d57fa640 1 -- 172.21.15.44:0/2022258164 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f31cc018020 con 0x7f31d80d0a30 2024-04-02T13:26:23.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.497+0000 7f31d57fa640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f31c406c8f0 0x7f31c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.497+0000 7f31d57fa640 1 -- 172.21.15.44:0/2022258164 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f31cc08bb20 con 0x7f31d80d0a30 2024-04-02T13:26:23.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.497+0000 7f31dcff1640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f31c406c8f0 0x7f31c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.497+0000 7f31dcff1640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f31c406c8f0 0x7f31c406edb0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f31d0036000 tx=0x7f31d0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.501+0000 7f31d57fa640 1 -- 172.21.15.44:0/2022258164 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f31cc01f020 con 0x7f31d80d0a30 2024-04-02T13:26:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.725+0000 7f31df255640 1 -- 172.21.15.44:0/2022258164 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31d80b1de0 con 0x7f31d80d0a30 2024-04-02T13:26:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.725+0000 7f31d57fa640 1 -- 172.21.15.44:0/2022258164 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5185 (secure 0 0 0) 0x7f31cc059720 con 0x7f31d80d0a30 2024-04-02T13:26:23.729 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:23.729 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3662927745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3662927745},{"type":"v1","addr":"172.21.15.44:6835","nonce":3662927745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6394,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1053037366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1053037366},{"type":"v1","addr":"172.21.15.69:6836","nonce":1053037366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3893800047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3893800047},{"type":"v1","addr":"172.21.15.69:6837","nonce":3893800047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:24:53.925714+0000","modified":"2024-04-02T13:25:57.948654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.44:6833/2392159648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2392159648},{"type":"v1","addr":"172.21.15.44:6833","nonce":2392159648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":5}]} 2024-04-02T13:26:23.729 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 52 2024-04-02T13:26:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.727+0000 7f31df255640 1 -- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f31c406c8f0 msgr2=0x7f31c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.727+0000 7f31df255640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f31c406c8f0 0x7f31c406edb0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f31d0036000 tx=0x7f31d0030000 comp rx=0 tx=0).stop 2024-04-02T13:26:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.727+0000 7f31df255640 1 -- 172.21.15.44:0/2022258164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d80d0a30 msgr2=0x7f31d80b6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.727+0000 7f31df255640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d80d0a30 0x7f31d80b6d60 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00b410 tx=0x7f31cc00b8d0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.728+0000 7f31df255640 1 -- 172.21.15.44:0/2022258164 shutdown_connections 2024-04-02T13:26:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.728+0000 7f31df255640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f31c406c8f0 0x7f31c406edb0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.728+0000 7f31df255640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31d80d0e10 0x7f31d80b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.728+0000 7f31df255640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d80d0a30 0x7f31d80b6d60 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.728+0000 7f31df255640 1 --2- 172.21.15.44:0/2022258164 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31d80cfe30 0x7f31d80c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.728+0000 7f31df255640 1 -- 172.21.15.44:0/2022258164 >> 172.21.15.44:0/2022258164 conn(0x7f31d809d470 msgr2=0x7f31d80d3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.728+0000 7f31df255640 1 -- 172.21.15.44:0/2022258164 shutdown_connections 2024-04-02T13:26:23.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.728+0000 7f31df255640 1 -- 172.21.15.44:0/2022258164 wait complete. 2024-04-02T13:26:23.741 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-04-02T13:26:23.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.842+0000 7f5c4e63c640 1 Processor -- start 2024-04-02T13:26:23.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.842+0000 7f5c4e63c640 1 -- start start 2024-04-02T13:26:23.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c4e63c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c4815e520 0x7f5c48162950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c4e63c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5c480ad890 0x7f5c4806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c4e63c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c480d03e0 con 0x7f5c480adcf0 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c4e63c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5c480b4610 con 0x7f5c480ad890 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c4e63c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5c48162e90 con 0x7f5c4815e520 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c4cbd9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f5c480adcf0 0x7f5c4806e380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48620/0 (socket says 172.21.15.44:48620) 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c4cbd9640 1 -- 172.21.15.44:0/204024056 learned_addr learned my addr 172.21.15.44:0/204024056 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c477fe640 1 --2- 172.21.15.44:0/204024056 >> v2:172.21.15.69:3300/0 conn(0x7f5c480ad890 0x7f5c4806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c47fff640 1 --2- 172.21.15.44:0/204024056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c4815e520 0x7f5c48162950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 272396641 0 0) 0x7f5c480d03e0 con 0x7f5c480adcf0 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c30003520 con 0x7f5c480adcf0 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2974577745 0 0) 0x7f5c30003520 con 0x7f5c480adcf0 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c4815e520 msgr2=0x7f5c48162950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 --2- 172.21.15.44:0/204024056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c4815e520 0x7f5c48162950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 >> v2:172.21.15.69:3300/0 conn(0x7f5c480ad890 msgr2=0x7f5c4806b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 --2- 172.21.15.44:0/204024056 >> v2:172.21.15.69:3300/0 conn(0x7f5c480ad890 0x7f5c4806b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c480b3a90 con 0x7f5c480adcf0 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f5c400019f0 con 0x7f5c480adcf0 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c477fe640 1 --2- 172.21.15.44:0/204024056 >> v2:172.21.15.69:3300/0 conn(0x7f5c480ad890 0x7f5c4806b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 >> v1:172.21.15.44:6789/0 conn(0x7f5c480adcf0 legacy=0x7f5c4806e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 --2- 172.21.15.44:0/204024056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c4815e520 0x7f5c300039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 --2- 172.21.15.44:0/204024056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c30003f30 0x7f5c30004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 --2- 172.21.15.44:0/204024056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c30004d40 0x7f5c300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5c48162e90 con 0x7f5c30003f30 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3000b9c0 con 0x7f5c30004d40 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5c3000bb40 con 0x7f5c4815e520 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.843+0000 7f5c477fe640 1 --2- 172.21.15.44:0/204024056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c30003f30 0x7f5c30004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.844+0000 7f5c47fff640 1 --2- 172.21.15.44:0/204024056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c4815e520 0x7f5c300039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.844+0000 7f5c4cbd9640 1 --2- 172.21.15.44:0/204024056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c30004d40 0x7f5c300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.844+0000 7f5c477fe640 1 -- 172.21.15.44:0/204024056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c4815e520 msgr2=0x7f5c300039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.844+0000 7f5c477fe640 1 --2- 172.21.15.44:0/204024056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c4815e520 0x7f5c300039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.844+0000 7f5c477fe640 1 -- 172.21.15.44:0/204024056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c30004d40 msgr2=0x7f5c300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.844+0000 7f5c477fe640 1 --2- 172.21.15.44:0/204024056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c30004d40 0x7f5c300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.844+0000 7f5c477fe640 1 -- 172.21.15.44:0/204024056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c30003520 con 0x7f5c30003f30 2024-04-02T13:26:23.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.845+0000 7f5c477fe640 1 --2- 172.21.15.44:0/204024056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c30003f30 0x7f5c30004800 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c014f40 tx=0x7f5c30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f9e0a2202fdbd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.845+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c040070 con 0x7f5c30003f30 2024-04-02T13:26:23.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.845+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c3c03b600 con 0x7f5c30003f30 2024-04-02T13:26:23.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.845+0000 7f5c46ffd640 1 -- 172.21.15.44:0/204024056 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c014b90 con 0x7f5c30003f30 2024-04-02T13:26:23.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.845+0000 7f5c4e63c640 1 -- 172.21.15.44:0/204024056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c30003f30 msgr2=0x7f5c30004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.845+0000 7f5c4e63c640 1 --2- 172.21.15.44:0/204024056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c30003f30 0x7f5c30004800 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c014f40 tx=0x7f5c30005830 comp rx=0 tx=0).stop 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.845+0000 7f5c4cbd9640 1 -- 172.21.15.44:0/204024056 reap_dead start 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.845+0000 7f5c4e63c640 1 -- 172.21.15.44:0/204024056 shutdown_connections 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.845+0000 7f5c4e63c640 1 -- 172.21.15.44:0/204024056 >> 172.21.15.44:0/204024056 conn(0x7f5c480b22c0 msgr2=0x7f5c480b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.845+0000 7f5c4e63c640 1 -- 172.21.15.44:0/204024056 shutdown_connections 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.845+0000 7f5c4e63c640 1 -- 172.21.15.44:0/204024056 wait complete. 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.846+0000 7f5c4e63c640 1 Processor -- start 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.847+0000 7f5c4e63c640 1 -- start start 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.847+0000 7f5c4e63c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c30003f30 0x7f5c481520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.847+0000 7f5c4e63c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c48151b90 0x7f5c48151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.847+0000 7f5c4e63c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c480ae0d0 0x7f5c4815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.847+0000 7f5c4e63c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5c480b4050 con 0x7f5c480ae0d0 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.847+0000 7f5c4e63c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4806def0 con 0x7f5c48151b90 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.847+0000 7f5c4e63c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5c480b4330 con 0x7f5c30003f30 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.847+0000 7f5c4cbd9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c480ae0d0 0x7f5c4815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.847+0000 7f5c4cbd9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c480ae0d0 0x7f5c4815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47216/0 (socket says 172.21.15.44:47216) 2024-04-02T13:26:23.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.847+0000 7f5c4cbd9640 1 -- 172.21.15.44:0/1603200297 learned_addr learned my addr 172.21.15.44:0/1603200297 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.847+0000 7f5c477fe640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c48151b90 0x7f5c48151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.847+0000 7f5c47fff640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c30003f30 0x7f5c481520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.848+0000 7f5c4cbd9640 1 -- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c30003f30 msgr2=0x7f5c481520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.848+0000 7f5c4cbd9640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c30003f30 0x7f5c481520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.848+0000 7f5c4cbd9640 1 -- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c48151b90 msgr2=0x7f5c48151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.848+0000 7f5c4cbd9640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c48151b90 0x7f5c48151490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.848+0000 7f5c4cbd9640 1 -- 172.21.15.44:0/1603200297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c480af7a0 con 0x7f5c480ae0d0 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.848+0000 7f5c4cbd9640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c480ae0d0 0x7f5c4815ed90 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f5c48071950 tx=0x7f5c40007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.848+0000 7f5c44ff9640 1 -- 172.21.15.44:0/1603200297 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4000ab20 con 0x7f5c480ae0d0 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.848+0000 7f5c44ff9640 1 -- 172.21.15.44:0/1603200297 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c40018070 con 0x7f5c480ae0d0 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.848+0000 7f5c4e63c640 1 -- 172.21.15.44:0/1603200297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c480afa80 con 0x7f5c480ae0d0 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.848+0000 7f5c44ff9640 1 -- 172.21.15.44:0/1603200297 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c400133f0 con 0x7f5c480ae0d0 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.848+0000 7f5c4e63c640 1 -- 172.21.15.44:0/1603200297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c480ce7c0 con 0x7f5c480ae0d0 2024-04-02T13:26:23.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.849+0000 7f5c44ff9640 1 -- 172.21.15.44:0/1603200297 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5c40013590 con 0x7f5c480ae0d0 2024-04-02T13:26:23.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.850+0000 7f5c44ff9640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5c3406c920 0x7f5c3406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.850+0000 7f5c44ff9640 1 -- 172.21.15.44:0/1603200297 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f5c4008c0e0 con 0x7f5c480ae0d0 2024-04-02T13:26:23.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.850+0000 7f5c47fff640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5c3406c920 0x7f5c3406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.850+0000 7f5c47fff640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5c3406c920 0x7f5c3406ede0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5c38009ae0 tx=0x7f5c38009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.851+0000 7f5c51226740 1 Processor -- start 2024-04-02T13:26:23.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.851+0000 7f5c51226740 1 -- start start 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.851+0000 7f5c51226740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914f9dc0 0x563e914fa1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.851+0000 7f5c51226740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e914ffbe0 0x563e914fde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.851+0000 7f5c51226740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e915474d0 0x563e91547950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c51226740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563e914c15d0 con 0x563e914f9dc0 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c51226740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563e914636b0 con 0x563e915474d0 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c51226740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x563e91401890 con 0x563e914ffbe0 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c47fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914f9dc0 0x563e914fa1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c47fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914f9dc0 0x563e914fa1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47218/0 (socket says 172.21.15.44:47218) 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c47fff640 1 -- 172.21.15.44:0/1658008688 learned_addr learned my addr 172.21.15.44:0/1658008688 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c4cbd9640 1 --2- 172.21.15.44:0/1658008688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e915474d0 0x563e91547950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c477fe640 1 --2- 172.21.15.44:0/1658008688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e914ffbe0 0x563e914fde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c47fff640 1 -- 172.21.15.44:0/1658008688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e914ffbe0 msgr2=0x563e914fde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c47fff640 1 --2- 172.21.15.44:0/1658008688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e914ffbe0 0x563e914fde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c47fff640 1 -- 172.21.15.44:0/1658008688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e915474d0 msgr2=0x563e91547950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c47fff640 1 --2- 172.21.15.44:0/1658008688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e915474d0 0x563e91547950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c47fff640 1 -- 172.21.15.44:0/1658008688 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e914301d0 con 0x563e914f9dc0 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c4cbd9640 1 --2- 172.21.15.44:0/1658008688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e915474d0 0x563e91547950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c47fff640 1 --2- 172.21.15.44:0/1658008688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914f9dc0 0x563e914fa1a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f5c38006a70 tx=0x7f5c38017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a303b40784349512 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c2dffb640 1 -- 172.21.15.44:0/1658008688 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3801b070 con 0x563e914f9dc0 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c2dffb640 1 -- 172.21.15.44:0/1658008688 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c380044d0 con 0x563e914f9dc0 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.852+0000 7f5c2dffb640 1 -- 172.21.15.44:0/1658008688 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c38022430 con 0x563e914f9dc0 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.853+0000 7f5c51226740 1 -- 172.21.15.44:0/1658008688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914f9dc0 msgr2=0x563e914fa1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.853+0000 7f5c51226740 1 --2- 172.21.15.44:0/1658008688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914f9dc0 0x563e914fa1a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f5c38006a70 tx=0x7f5c38017000 comp rx=0 tx=0).stop 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.853+0000 7f5c51226740 1 -- 172.21.15.44:0/1658008688 shutdown_connections 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.853+0000 7f5c51226740 1 --2- 172.21.15.44:0/1658008688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e915474d0 0x563e91547950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.853+0000 7f5c51226740 1 --2- 172.21.15.44:0/1658008688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e914ffbe0 0x563e914fde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.853+0000 7f5c51226740 1 --2- 172.21.15.44:0/1658008688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914f9dc0 0x563e914fa1a0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.853+0000 7f5c51226740 1 -- 172.21.15.44:0/1658008688 >> 172.21.15.44:0/1658008688 conn(0x563e914fbe40 msgr2=0x563e914fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.853+0000 7f5c51226740 1 -- 172.21.15.44:0/1658008688 shutdown_connections 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.853+0000 7f5c51226740 1 -- 172.21.15.44:0/1658008688 wait complete. 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.853+0000 7f5c51226740 1 -- start start 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.854+0000 7f5c51226740 1 Processor -- start 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.854+0000 7f5c51226740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e914ffbe0 0x563e9153bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.854+0000 7f5c51226740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e915474d0 0x563e9153d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.854+0000 7f5c51226740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e9152a310 0x563e9150f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.854+0000 7f5c51226740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563e913ff200 con 0x563e915474d0 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.854+0000 7f5c51226740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563e9145f660 con 0x563e914ffbe0 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.854+0000 7f5c51226740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x563e91526530 con 0x563e9152a310 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.854+0000 7f5c51226740 10 client.0 _ll_register_callbacks cb 0x563e91422040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c477fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e915474d0 0x563e9153d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c477fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e915474d0 0x563e9153d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47230/0 (socket says 172.21.15.44:47230) 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c477fe640 1 -- 172.21.15.44:0/268383857 learned_addr learned my addr 172.21.15.44:0/268383857 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c47fff640 1 --2- 172.21.15.44:0/268383857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e914ffbe0 0x563e9153bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c4cbd9640 1 --2- 172.21.15.44:0/268383857 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e9152a310 0x563e9150f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c477fe640 1 -- 172.21.15.44:0/268383857 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e9152a310 msgr2=0x563e9150f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c477fe640 1 --2- 172.21.15.44:0/268383857 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e9152a310 0x563e9150f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c477fe640 1 -- 172.21.15.44:0/268383857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e914ffbe0 msgr2=0x563e9153bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c477fe640 1 --2- 172.21.15.44:0/268383857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e914ffbe0 0x563e9153bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c477fe640 1 -- 172.21.15.44:0/268383857 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e912437c0 con 0x563e915474d0 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c477fe640 1 --2- 172.21.15.44:0/268383857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e915474d0 0x563e9153d510 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c017090 tx=0x7f5c3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1eb6882448a758 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c26ffd640 1 -- 172.21.15.44:0/268383857 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c040040 con 0x563e915474d0 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c26ffd640 1 -- 172.21.15.44:0/268383857 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f5c3c004020 con 0x563e915474d0 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c51226740 1 -- 172.21.15.44:0/268383857 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c380044d0 con 0x563e915474d0 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c26ffd640 1 -- 172.21.15.44:0/268383857 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f5c3c004220 con 0x563e915474d0 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c51226740 10 client.7575 fetch_fsmap learned FSMap version 52 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.855+0000 7f5c51226740 1 -- 172.21.15.44:0/268383857 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x563e912a3f50 con 0x563e915474d0 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.856+0000 7f5c26ffd640 1 -- 172.21.15.44:0/268383857 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c380044d0 con 0x563e915474d0 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.856+0000 7f5c26ffd640 1 -- 172.21.15.44:0/268383857 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7f5c3c03b9a0 con 0x563e915474d0 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.856+0000 7f5c51226740 10 client.7575 fetch_fsmap finished waiting for FSMap version 52 2024-04-02T13:26:23.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.856+0000 7f5c51226740 10 client.7575 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.856+0000 7f5c51226740 20 client.7575 populate_metadata read hostname 'smithi044' 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.856+0000 7f5c51226740 1 --2- 172.21.15.44:0/268383857 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x563e9152ca80 0x563e91528340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.856+0000 7f5c51226740 4 client.7575 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.856+0000 7f5c51226740 1 -- 172.21.15.44:0/268383857 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e91476ee0 con 0x563e9152ca80 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.856+0000 7f5c47fff640 1 --2- 172.21.15.44:0/268383857 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x563e9152ca80 0x563e91528340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.856+0000 7f5c26ffd640 10 client.7575 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.856+0000 7f5c47fff640 1 --2- 172.21.15.44:0/268383857 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x563e9152ca80 0x563e91528340 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=202cfe2b53121dbb server_cookie=b9cbfed11860f12 in_seq=0 out_seq=0 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c26ffd640 1 -- 172.21.15.44:0/268383857 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563e91476ee0 con 0x563e9152ca80 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c26ffd640 10 client.7575 handle_command_reply: tid=0 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c51226740 1 client.7575 shutdown 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c51226740 1 -- 172.21.15.44:0/268383857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e915474d0 msgr2=0x563e9153d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c51226740 1 --2- 172.21.15.44:0/268383857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e915474d0 0x563e9153d510 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c017090 tx=0x7f5c3c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c51226740 1 -- 172.21.15.44:0/268383857 shutdown_connections 2024-04-02T13:26:23.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c51226740 1 --2- 172.21.15.44:0/268383857 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x563e9152ca80 0x563e91528340 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c51226740 1 --2- 172.21.15.44:0/268383857 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e9152a310 0x563e9150f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c51226740 1 --2- 172.21.15.44:0/268383857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e915474d0 0x563e9153d510 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c51226740 1 --2- 172.21.15.44:0/268383857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e914ffbe0 0x563e9153bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c26ffd640 0 client.7575 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:26:23.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c51226740 1 -- 172.21.15.44:0/268383857 >> 172.21.15.44:0/268383857 conn(0x563e914f9dc0 msgr2=0x563e91525ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c51226740 1 -- 172.21.15.44:0/268383857 shutdown_connections 2024-04-02T13:26:23.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c51226740 1 -- 172.21.15.44:0/268383857 wait complete. 2024-04-02T13:26:23.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.857+0000 7f5c51226740 20 client.7575 trim_cache size 0 max 16384 2024-04-02T13:26:23.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.870+0000 7f5c51226740 1 Processor -- start 2024-04-02T13:26:23.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c51226740 1 -- start start 2024-04-02T13:26:23.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c51226740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 0x563e915451b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c51226740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e9152a310 0x563e91526e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c51226740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e915474d0 0x563e91545f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c51226740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5c481fcf00 con 0x563e914ffbe0 2024-04-02T13:26:23.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c51226740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5c481fbf80 con 0x563e915474d0 2024-04-02T13:26:23.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c51226740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5c481fb430 con 0x563e9152a310 2024-04-02T13:26:23.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c47fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 0x563e915451b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c47fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 0x563e915451b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47242/0 (socket says 172.21.15.44:47242) 2024-04-02T13:26:23.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c47fff640 1 -- 172.21.15.44:0/3164563604 learned_addr learned my addr 172.21.15.44:0/3164563604 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:23.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c477fe640 1 --2- 172.21.15.44:0/3164563604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e9152a310 0x563e91526e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c4cbd9640 1 --2- 172.21.15.44:0/3164563604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e915474d0 0x563e91545f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c47fff640 1 -- 172.21.15.44:0/3164563604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e9152a310 msgr2=0x563e91526e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c47fff640 1 --2- 172.21.15.44:0/3164563604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e9152a310 0x563e91526e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c47fff640 1 -- 172.21.15.44:0/3164563604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e915474d0 msgr2=0x563e91545f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c47fff640 1 --2- 172.21.15.44:0/3164563604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e915474d0 0x563e91545f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.871+0000 7f5c47fff640 1 -- 172.21.15.44:0/3164563604 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e911370b0 con 0x563e914ffbe0 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c477fe640 1 --2- 172.21.15.44:0/3164563604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e9152a310 0x563e91526e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c4cbd9640 1 --2- 172.21.15.44:0/3164563604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e915474d0 0x563e91545f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c47fff640 1 --2- 172.21.15.44:0/3164563604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 0x563e915451b0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f5c38026000 tx=0x7f5c3801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc78969ad01c579 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c2d7fa640 1 -- 172.21.15.44:0/3164563604 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3803d070 con 0x563e914ffbe0 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c2d7fa640 1 -- 172.21.15.44:0/3164563604 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c38037030 con 0x563e914ffbe0 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c2d7fa640 1 -- 172.21.15.44:0/3164563604 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3801b040 con 0x563e914ffbe0 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c51226740 1 -- 172.21.15.44:0/3164563604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 msgr2=0x563e915451b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c51226740 1 --2- 172.21.15.44:0/3164563604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 0x563e915451b0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f5c38026000 tx=0x7f5c3801c000 comp rx=0 tx=0).stop 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c51226740 1 -- 172.21.15.44:0/3164563604 shutdown_connections 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c51226740 1 --2- 172.21.15.44:0/3164563604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e915474d0 0x563e91545f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c51226740 1 --2- 172.21.15.44:0/3164563604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e9152a310 0x563e91526e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c51226740 1 --2- 172.21.15.44:0/3164563604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 0x563e915451b0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c51226740 1 -- 172.21.15.44:0/3164563604 >> 172.21.15.44:0/3164563604 conn(0x563e914f9dc0 msgr2=0x563e9150f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c51226740 1 -- 172.21.15.44:0/3164563604 shutdown_connections 2024-04-02T13:26:23.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.872+0000 7f5c51226740 1 -- 172.21.15.44:0/3164563604 wait complete. 2024-04-02T13:26:23.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.873+0000 7f5c51226740 1 -- start start 2024-04-02T13:26:23.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.873+0000 7f5c51226740 1 Processor -- start 2024-04-02T13:26:23.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c51226740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 0x563e915451b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c51226740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e9152a310 0x563e915456f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c51226740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e915474d0 0x563e915bfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c51226740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563e9152c900 con 0x563e914ffbe0 2024-04-02T13:26:23.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c51226740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5c481fb930 con 0x563e9152a310 2024-04-02T13:26:23.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c51226740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x563e9152cf50 con 0x563e915474d0 2024-04-02T13:26:23.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c51226740 10 client.0 _ll_register_callbacks cb 0x7f5c300036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c47fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 0x563e915451b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c47fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 0x563e915451b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47254/0 (socket says 172.21.15.44:47254) 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c47fff640 1 -- 172.21.15.44:0/2781929776 learned_addr learned my addr 172.21.15.44:0/2781929776 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c477fe640 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e9152a310 0x563e915456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c4cbd9640 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e915474d0 0x563e915bfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c47fff640 1 -- 172.21.15.44:0/2781929776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e915474d0 msgr2=0x563e915bfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c47fff640 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e915474d0 0x563e915bfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c47fff640 1 -- 172.21.15.44:0/2781929776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e9152a310 msgr2=0x563e915456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c47fff640 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e9152a310 0x563e915456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c47fff640 1 -- 172.21.15.44:0/2781929776 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e91568300 con 0x563e914ffbe0 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c477fe640 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e9152a310 0x563e915456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c4cbd9640 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e915474d0 0x563e915bfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c47fff640 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 0x563e915451b0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f5c380098c0 tx=0x7f5c38017070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d4cbfec1e9db69 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.874+0000 7f5c26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c26ffd640 1 -- 172.21.15.44:0/2781929776 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3803d030 con 0x563e914ffbe0 2024-04-02T13:26:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c26ffd640 1 -- 172.21.15.44:0/2781929776 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f5c38044ca0 con 0x563e914ffbe0 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c51226740 1 -- 172.21.15.44:0/2781929776 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c38037030 con 0x563e914ffbe0 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c26ffd640 1 -- 172.21.15.44:0/2781929776 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f5c3801b040 con 0x563e914ffbe0 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c51226740 10 client.7581 fetch_fsmap learned FSMap version 52 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c51226740 1 -- 172.21.15.44:0/2781929776 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x563e91568c00 con 0x563e914ffbe0 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c26ffd640 1 -- 172.21.15.44:0/2781929776 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c38037030 con 0x563e914ffbe0 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c26ffd640 1 -- 172.21.15.44:0/2781929776 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7f5c380373f0 con 0x563e914ffbe0 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c51226740 10 client.7581 fetch_fsmap finished waiting for FSMap version 52 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c51226740 10 client.7581 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c51226740 20 client.7581 populate_metadata read hostname 'smithi044' 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c51226740 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x563e915c0090 0x563e915c0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c51226740 4 client.7581 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c51226740 1 -- 172.21.15.44:0/2781929776 --> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x563e914044d0 con 0x563e915c0090 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c477fe640 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x563e915c0090 0x563e915c0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c26ffd640 10 client.7581 ms_handle_connect on v2:172.21.15.44:6832/2392159648 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.875+0000 7f5c477fe640 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x563e915c0090 0x563e915c0470 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9e57e290be82e50 server_cookie=b7185b7e63e1c94 in_seq=0 out_seq=0 2024-04-02T13:26:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c26ffd640 1 -- 172.21.15.44:0/2781929776 <== mds.0 v2:172.21.15.44:6832/2392159648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563e914044d0 con 0x563e915c0090 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c26ffd640 10 client.7581 handle_command_reply: tid=0 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c51226740 1 client.7581 shutdown 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c51226740 1 -- 172.21.15.44:0/2781929776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 msgr2=0x563e915451b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c51226740 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 0x563e915451b0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f5c380098c0 tx=0x7f5c38017070 comp rx=0 tx=0).stop 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c51226740 1 -- 172.21.15.44:0/2781929776 shutdown_connections 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c51226740 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.44:6832/2392159648,v1:172.21.15.44:6833/2392159648] conn(0x563e915c0090 0x563e915c0470 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c51226740 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e915474d0 0x563e915bfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c51226740 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e9152a310 0x563e915456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c51226740 1 --2- 172.21.15.44:0/2781929776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e914ffbe0 0x563e915451b0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c26ffd640 0 client.7581 ms_handle_reset on v2:172.21.15.44:6832/2392159648 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c51226740 1 -- 172.21.15.44:0/2781929776 >> 172.21.15.44:0/2781929776 conn(0x563e914f9dc0 msgr2=0x563e9151ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c51226740 1 -- 172.21.15.44:0/2781929776 shutdown_connections 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c51226740 1 -- 172.21.15.44:0/2781929776 wait complete. 2024-04-02T13:26:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c51226740 20 client.7581 trim_cache size 0 max 16384 2024-04-02T13:26:23.886 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:23.886 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-02T13:26:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c4e63c640 1 -- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5c3406c920 msgr2=0x7f5c3406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c4e63c640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5c3406c920 0x7f5c3406ede0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5c38009ae0 tx=0x7f5c38009190 comp rx=0 tx=0).stop 2024-04-02T13:26:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c4e63c640 1 -- 172.21.15.44:0/1603200297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c480ae0d0 msgr2=0x7f5c4815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.876+0000 7f5c4e63c640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c480ae0d0 0x7f5c4815ed90 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f5c48071950 tx=0x7f5c40007b10 comp rx=0 tx=0).stop 2024-04-02T13:26:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.877+0000 7f5c4e63c640 1 -- 172.21.15.44:0/1603200297 shutdown_connections 2024-04-02T13:26:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.877+0000 7f5c4e63c640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5c3406c920 0x7f5c3406ede0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.877+0000 7f5c4e63c640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c480ae0d0 0x7f5c4815ed90 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.877+0000 7f5c4e63c640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c48151b90 0x7f5c48151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.877+0000 7f5c4e63c640 1 --2- 172.21.15.44:0/1603200297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c30003f30 0x7f5c481520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.877+0000 7f5c4e63c640 1 -- 172.21.15.44:0/1603200297 >> 172.21.15.44:0/1603200297 conn(0x7f5c480b22c0 msgr2=0x7f5c4806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.877+0000 7f5c4e63c640 1 -- 172.21.15.44:0/1603200297 shutdown_connections 2024-04-02T13:26:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.877+0000 7f5c4e63c640 1 -- 172.21.15.44:0/1603200297 wait complete. 2024-04-02T13:26:23.890 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-02T13:26:23.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.987+0000 7fa7e864e640 1 Processor -- start 2024-04-02T13:26:23.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.987+0000 7fa7e864e640 1 -- start start 2024-04-02T13:26:23.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.988+0000 7fa7e864e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7e00b56f0 0x7fa7e00cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.988+0000 7fa7e864e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa7e00d2f60 0x7fa7e00cf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.988+0000 7fa7e864e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e00d0c80 con 0x7fa7e00d2470 2024-04-02T13:26:23.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.988+0000 7fa7e864e640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa7e00d15b0 con 0x7fa7e00d2f60 2024-04-02T13:26:23.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.988+0000 7fa7e864e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e00d0f60 con 0x7fa7e00b56f0 2024-04-02T13:26:23.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.988+0000 7fa7e6beb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa7e00d2f60 0x7fa7e00cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.988+0000 7fa7e6beb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa7e00d2f60 0x7fa7e00cf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47258/0 (socket says 172.21.15.44:47258) 2024-04-02T13:26:23.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.989+0000 7fa7e6beb640 1 -- 172.21.15.44:0/2656091305 learned_addr learned my addr 172.21.15.44:0/2656091305 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:23.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.989+0000 7fa7e5be9640 1 --2- 172.21.15.44:0/2656091305 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7e00b56f0 0x7fa7e00cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.989+0000 7fa7e6beb640 1 -- 172.21.15.44:0/2656091305 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7e00b56f0 msgr2=0x7fa7e00cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:23.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.989+0000 7fa7e6beb640 1 --2- 172.21.15.44:0/2656091305 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7e00b56f0 0x7fa7e00cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.989+0000 7fa7e6beb640 1 -- 172.21.15.44:0/2656091305 >> v1:172.21.15.69:6789/0 conn(0x7fa7e00d2470 legacy=0x7fa7e00d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.989+0000 7fa7e6beb640 1 -- 172.21.15.44:0/2656091305 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e00d1240 con 0x7fa7e00d2f60 2024-04-02T13:26:23.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.989+0000 7fa7e5be9640 1 --2- 172.21.15.44:0/2656091305 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7e00b56f0 0x7fa7e00cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:23.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.989+0000 7fa7e6beb640 1 --2- 172.21.15.44:0/2656091305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7e00d2f60 0x7fa7e00cf340 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8009e50 tx=0x7fa7d802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f505b8cac150620 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.990+0000 7fa7e53e8640 1 -- 172.21.15.44:0/2656091305 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d802ca70 con 0x7fa7e00d2f60 2024-04-02T13:26:23.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.990+0000 7fa7e53e8640 1 -- 172.21.15.44:0/2656091305 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d802cc10 con 0x7fa7e00d2f60 2024-04-02T13:26:23.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.990+0000 7fa7e53e8640 1 -- 172.21.15.44:0/2656091305 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d80343f0 con 0x7fa7e00d2f60 2024-04-02T13:26:23.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.990+0000 7fa7e864e640 1 -- 172.21.15.44:0/2656091305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7e00d2f60 msgr2=0x7fa7e00cf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.990+0000 7fa7e864e640 1 --2- 172.21.15.44:0/2656091305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7e00d2f60 0x7fa7e00cf340 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8009e50 tx=0x7fa7d802bed0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.990+0000 7fa7e864e640 1 -- 172.21.15.44:0/2656091305 shutdown_connections 2024-04-02T13:26:23.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.990+0000 7fa7e864e640 1 --2- 172.21.15.44:0/2656091305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7e00d2f60 0x7fa7e00cf340 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.990+0000 7fa7e864e640 1 --2- 172.21.15.44:0/2656091305 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7e00b56f0 0x7fa7e00cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.990+0000 7fa7e864e640 1 -- 172.21.15.44:0/2656091305 >> 172.21.15.44:0/2656091305 conn(0x7fa7e009d4b0 msgr2=0x7fa7e00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:23.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.990+0000 7fa7e864e640 1 -- 172.21.15.44:0/2656091305 shutdown_connections 2024-04-02T13:26:23.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.990+0000 7fa7e864e640 1 -- 172.21.15.44:0/2656091305 wait complete. 2024-04-02T13:26:23.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.991+0000 7fa7e864e640 1 Processor -- start 2024-04-02T13:26:23.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.992+0000 7fa7e864e640 1 -- start start 2024-04-02T13:26:23.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.992+0000 7fa7e864e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7e00b56f0 0x7fa7e0149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.992+0000 7fa7e864e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7e00d2470 0x7fa7e0162400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.992+0000 7fa7e864e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7e00d2f60 0x7fa7e0147300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:23.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.992+0000 7fa7e864e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e015fba0 con 0x7fa7e00d2f60 2024-04-02T13:26:23.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.992+0000 7fa7e864e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e015fa20 con 0x7fa7e00b56f0 2024-04-02T13:26:23.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.992+0000 7fa7e864e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e00b2ee0 con 0x7fa7e00d2470 2024-04-02T13:26:23.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.992+0000 7fa7e6beb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7e00d2f60 0x7fa7e0147300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.992+0000 7fa7e6beb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7e00d2f60 0x7fa7e0147300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47270/0 (socket says 172.21.15.44:47270) 2024-04-02T13:26:23.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.992+0000 7fa7e6beb640 1 -- 172.21.15.44:0/2188057695 learned_addr learned my addr 172.21.15.44:0/2188057695 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:23.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7e63ea640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7e00b56f0 0x7fa7e0149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7e5be9640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7e00d2470 0x7fa7e0162400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:23.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7e6beb640 1 -- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7e00d2470 msgr2=0x7fa7e0162400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7e6beb640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7e00d2470 0x7fa7e0162400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7e6beb640 1 -- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7e00b56f0 msgr2=0x7fa7e0149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:23.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7e6beb640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7e00b56f0 0x7fa7e0149da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:23.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7e6beb640 1 -- 172.21.15.44:0/2188057695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e014a2e0 con 0x7fa7e00d2f60 2024-04-02T13:26:23.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7e5be9640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7e00d2470 0x7fa7e0162400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:23.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7e6beb640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7e00d2f60 0x7fa7e0147300 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8036040 tx=0x7fa7d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:23.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7d6ffd640 1 -- 172.21.15.44:0/2188057695 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8044020 con 0x7fa7e00d2f60 2024-04-02T13:26:23.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7d6ffd640 1 -- 172.21.15.44:0/2188057695 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d8040670 con 0x7fa7e00d2f60 2024-04-02T13:26:23.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7e864e640 1 -- 172.21.15.44:0/2188057695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e00cb3b0 con 0x7fa7e00d2f60 2024-04-02T13:26:23.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7d6ffd640 1 -- 172.21.15.44:0/2188057695 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8009510 con 0x7fa7e00d2f60 2024-04-02T13:26:23.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.993+0000 7fa7e864e640 1 -- 172.21.15.44:0/2188057695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e00cb970 con 0x7fa7e00d2f60 2024-04-02T13:26:23.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.994+0000 7fa7e864e640 1 -- 172.21.15.44:0/2188057695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e00684f0 con 0x7fa7e00d2f60 2024-04-02T13:26:23.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.995+0000 7fa7d6ffd640 1 -- 172.21.15.44:0/2188057695 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa7d8005d10 con 0x7fa7e00d2f60 2024-04-02T13:26:24.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.995+0000 7fa7d6ffd640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa7c406c8f0 0x7fa7c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:24.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.995+0000 7fa7d6ffd640 1 -- 172.21.15.44:0/2188057695 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fa7d80ae950 con 0x7fa7e00d2f60 2024-04-02T13:26:24.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.995+0000 7fa7e63ea640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa7c406c8f0 0x7fa7c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:24.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:23.996+0000 7fa7e63ea640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa7c406c8f0 0x7fa7c406edb0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc001780 tx=0x7fa7dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:24.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.000+0000 7fa7d6ffd640 1 -- 172.21.15.44:0/2188057695 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa7e014a2e0 con 0x7fa7e00d2f60 2024-04-02T13:26:24.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.224+0000 7fa7e864e640 1 -- 172.21.15.44:0/2188057695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa7e008d930 con 0x7fa7e00d2f60 2024-04-02T13:26:24.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.224+0000 7fa7d6ffd640 1 -- 172.21.15.44:0/2188057695 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fa7d807cd50 con 0x7fa7e00d2f60 2024-04-02T13:26:24.227 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:24.227 INFO:teuthology.orchestra.run.smithi044.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-02T13:26:24.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.226+0000 7fa7e864e640 1 -- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa7c406c8f0 msgr2=0x7fa7c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:24.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.226+0000 7fa7e864e640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa7c406c8f0 0x7fa7c406edb0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc001780 tx=0x7fa7dc009040 comp rx=0 tx=0).stop 2024-04-02T13:26:24.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.226+0000 7fa7e864e640 1 -- 172.21.15.44:0/2188057695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7e00d2f60 msgr2=0x7fa7e0147300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:24.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.226+0000 7fa7e864e640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7e00d2f60 0x7fa7e0147300 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8036040 tx=0x7fa7d8004a40 comp rx=0 tx=0).stop 2024-04-02T13:26:24.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.226+0000 7fa7e864e640 1 -- 172.21.15.44:0/2188057695 shutdown_connections 2024-04-02T13:26:24.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.226+0000 7fa7e864e640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa7c406c8f0 0x7fa7c406edb0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.226+0000 7fa7e864e640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7e00d2f60 0x7fa7e0147300 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.226+0000 7fa7e864e640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7e00d2470 0x7fa7e0162400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.226+0000 7fa7e864e640 1 --2- 172.21.15.44:0/2188057695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7e00b56f0 0x7fa7e0149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.226+0000 7fa7e864e640 1 -- 172.21.15.44:0/2188057695 >> 172.21.15.44:0/2188057695 conn(0x7fa7e009d4b0 msgr2=0x7fa7e006d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:24.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.226+0000 7fa7e864e640 1 -- 172.21.15.44:0/2188057695 shutdown_connections 2024-04-02T13:26:24.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.226+0000 7fa7e864e640 1 -- 172.21.15.44:0/2188057695 wait complete. 2024-04-02T13:26:24.240 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-02T13:26:24.240 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:26:24.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.337+0000 7fbcde4d3640 1 Processor -- start 2024-04-02T13:26:24.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.337+0000 7fbcde4d3640 1 -- start start 2024-04-02T13:26:24.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.338+0000 7fbcde4d3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbcd80d0a30 0x7fbcd806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:24.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.338+0000 7fbcde4d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd80d2040 0x7fbcd80d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:24.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.338+0000 7fbcde4d3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd80d0000 con 0x7fbcd80d0e10 2024-04-02T13:26:24.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.338+0000 7fbcde4d3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbcd806cb00 con 0x7fbcd80d0a30 2024-04-02T13:26:24.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.338+0000 7fbcde4d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd806c1d0 con 0x7fbcd80d2040 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.338+0000 7fbcd77fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbcd80d0e10 0x7fbcd80cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48626/0 (socket says 172.21.15.44:48626) 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.338+0000 7fbcd77fe640 1 -- 172.21.15.44:0/2553381398 learned_addr learned my addr 172.21.15.44:0/2553381398 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.338+0000 7fbcd7fff640 1 --2- 172.21.15.44:0/2553381398 >> v2:172.21.15.69:3300/0 conn(0x7fbcd80d0a30 0x7fbcd806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.338+0000 7fbcdca70640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd80d2040 0x7fbcd80d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1410930118 0 0) 0x7fbcd80d0000 con 0x7fbcd80d0e10 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcc4003520 con 0x7fbcd80d0e10 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 305370647 0 0) 0x7fbcc4003520 con 0x7fbcd80d0e10 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd80d2040 msgr2=0x7fbcd80d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd80d2040 0x7fbcd80d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 >> v2:172.21.15.69:3300/0 conn(0x7fbcd80d0a30 msgr2=0x7fbcd806cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 --2- 172.21.15.44:0/2553381398 >> v2:172.21.15.69:3300/0 conn(0x7fbcd80d0a30 0x7fbcd806cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd806c4b0 con 0x7fbcd80d0e10 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fbcc8001670 con 0x7fbcd80d0e10 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 >> v1:172.21.15.44:6789/0 conn(0x7fbcd80d0e10 legacy=0x7fbcd80cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbcc40039b0 0x7fbcc4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc4004800 0x7fbcc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:24.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd80d2040 0x7fbcc40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd806cb00 con 0x7fbcc40039b0 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd806c1d0 con 0x7fbcc4004800 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbcc400b9f0 con 0x7fbcd80d2040 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd7fff640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbcc40039b0 0x7fbcc4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcdca70640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd80d2040 0x7fbcc40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd77fe640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc4004800 0x7fbcc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd7fff640 1 -- 172.21.15.44:0/2553381398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd80d2040 msgr2=0x7fbcc40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd7fff640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd80d2040 0x7fbcc40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd7fff640 1 -- 172.21.15.44:0/2553381398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc4004800 msgr2=0x7fbcc4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd7fff640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc4004800 0x7fbcc4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd7fff640 1 -- 172.21.15.44:0/2553381398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcc4003520 con 0x7fbcc40039b0 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcdca70640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd80d2040 0x7fbcc40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.339+0000 7fbcd77fe640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc4004800 0x7fbcc4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:24.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.340+0000 7fbcd7fff640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbcc40039b0 0x7fbcc4003dd0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0009ac0 tx=0x7fbcc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1c9f8ec5efbe9b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:24.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.340+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0039070 con 0x7fbcc40039b0 2024-04-02T13:26:24.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.340+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc002c6d0 con 0x7fbcc40039b0 2024-04-02T13:26:24.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.340+0000 7fbcd6ffd640 1 -- 172.21.15.44:0/2553381398 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0009690 con 0x7fbcc40039b0 2024-04-02T13:26:24.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.341+0000 7fbcde4d3640 1 -- 172.21.15.44:0/2553381398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbcc40039b0 msgr2=0x7fbcc4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:24.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.341+0000 7fbcde4d3640 1 --2- 172.21.15.44:0/2553381398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbcc40039b0 0x7fbcc4003dd0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0009ac0 tx=0x7fbcc4005830 comp rx=0 tx=0).stop 2024-04-02T13:26:24.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.341+0000 7fbcdca70640 1 -- 172.21.15.44:0/2553381398 reap_dead start 2024-04-02T13:26:24.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.341+0000 7fbcde4d3640 1 -- 172.21.15.44:0/2553381398 shutdown_connections 2024-04-02T13:26:24.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.341+0000 7fbcde4d3640 1 -- 172.21.15.44:0/2553381398 >> 172.21.15.44:0/2553381398 conn(0x7fbcd809d470 msgr2=0x7fbcd80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:24.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.341+0000 7fbcde4d3640 1 -- 172.21.15.44:0/2553381398 shutdown_connections 2024-04-02T13:26:24.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.341+0000 7fbcde4d3640 1 -- 172.21.15.44:0/2553381398 wait complete. 2024-04-02T13:26:24.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.342+0000 7fbcde4d3640 1 Processor -- start 2024-04-02T13:26:24.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.342+0000 7fbcde4d3640 1 -- start start 2024-04-02T13:26:24.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcde4d3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbcc40039b0 0x7fbcd80c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:24.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcde4d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd81e8f40 0x7fbcd815fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:24.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcde4d3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd81e93c0 0x7fbcd81603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:24.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcde4d3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd815e6f0 con 0x7fbcc40039b0 2024-04-02T13:26:24.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcde4d3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd80d1270 con 0x7fbcd81e93c0 2024-04-02T13:26:24.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcde4d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd815e870 con 0x7fbcd81e8f40 2024-04-02T13:26:24.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcd7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbcc40039b0 0x7fbcd80c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:24.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcd7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbcc40039b0 0x7fbcd80c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47284/0 (socket says 172.21.15.44:47284) 2024-04-02T13:26:24.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcd7fff640 1 -- 172.21.15.44:0/1360124057 learned_addr learned my addr 172.21.15.44:0/1360124057 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcdca70640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd81e93c0 0x7fbcd81603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcd77fe640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd81e8f40 0x7fbcd815fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcd7fff640 1 -- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd81e8f40 msgr2=0x7fbcd815fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcd7fff640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd81e8f40 0x7fbcd815fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcd7fff640 1 -- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd81e93c0 msgr2=0x7fbcd81603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcd7fff640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd81e93c0 0x7fbcd81603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcd7fff640 1 -- 172.21.15.44:0/1360124057 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd80b3f40 con 0x7fbcc40039b0 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcd7fff640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbcc40039b0 0x7fbcd80c8cf0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0037040 tx=0x7fbcc0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcd4ff9640 1 -- 172.21.15.44:0/1360124057 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0042020 con 0x7fbcc40039b0 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcd4ff9640 1 -- 172.21.15.44:0/1360124057 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc003a040 con 0x7fbcc40039b0 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcde4d3640 1 -- 172.21.15.44:0/1360124057 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd80b4220 con 0x7fbcc40039b0 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.343+0000 7fbcd4ff9640 1 -- 172.21.15.44:0/1360124057 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0039070 con 0x7fbcc40039b0 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.344+0000 7fbcde4d3640 1 -- 172.21.15.44:0/1360124057 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd80b47e0 con 0x7fbcc40039b0 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.345+0000 7fbcde4d3640 1 -- 172.21.15.44:0/1360124057 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcd80684f0 con 0x7fbcc40039b0 2024-04-02T13:26:24.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.345+0000 7fbcd4ff9640 1 -- 172.21.15.44:0/1360124057 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbcc0045020 con 0x7fbcc40039b0 2024-04-02T13:26:24.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.345+0000 7fbcd4ff9640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcb806c8f0 0x7fbcb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:24.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.345+0000 7fbcd4ff9640 1 -- 172.21.15.44:0/1360124057 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fbcc00ad130 con 0x7fbcc40039b0 2024-04-02T13:26:24.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.345+0000 7fbcd77fe640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcb806c8f0 0x7fbcb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:24.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.346+0000 7fbcd77fe640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcb806c8f0 0x7fbcb806edb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80ba000 tx=0x7fbcc8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:24.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.350+0000 7fbcd4ff9640 1 -- 172.21.15.44:0/1360124057 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbcc004c070 con 0x7fbcc40039b0 2024-04-02T13:26:24.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.574+0000 7fbcde4d3640 1 -- 172.21.15.44:0/1360124057 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbcd80d14f0 con 0x7fbcc40039b0 2024-04-02T13:26:24.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.575+0000 7fbcd4ff9640 1 -- 172.21.15.44:0/1360124057 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v52) ==== 53+0+73 (secure 0 0 0) 0x7fbcc007b380 con 0x7fbcc40039b0 2024-04-02T13:26:24.578 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:26:24.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.577+0000 7fbcde4d3640 1 -- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcb806c8f0 msgr2=0x7fbcb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:24.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.577+0000 7fbcde4d3640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcb806c8f0 0x7fbcb806edb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80ba000 tx=0x7fbcc8009040 comp rx=0 tx=0).stop 2024-04-02T13:26:24.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.577+0000 7fbcde4d3640 1 -- 172.21.15.44:0/1360124057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbcc40039b0 msgr2=0x7fbcd80c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:24.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.577+0000 7fbcde4d3640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbcc40039b0 0x7fbcd80c8cf0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0037040 tx=0x7fbcc0004730 comp rx=0 tx=0).stop 2024-04-02T13:26:24.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.577+0000 7fbcde4d3640 1 -- 172.21.15.44:0/1360124057 shutdown_connections 2024-04-02T13:26:24.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.577+0000 7fbcde4d3640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbcb806c8f0 0x7fbcb806edb0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.577+0000 7fbcde4d3640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd81e93c0 0x7fbcd81603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.577+0000 7fbcde4d3640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd81e8f40 0x7fbcd815fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.577+0000 7fbcde4d3640 1 --2- 172.21.15.44:0/1360124057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbcc40039b0 0x7fbcd80c8cf0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:24.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.577+0000 7fbcde4d3640 1 -- 172.21.15.44:0/1360124057 >> 172.21.15.44:0/1360124057 conn(0x7fbcd809d470 msgr2=0x7fbcd80b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:24.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.577+0000 7fbcde4d3640 1 -- 172.21.15.44:0/1360124057 shutdown_connections 2024-04-02T13:26:24.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24.577+0000 7fbcde4d3640 1 -- 172.21.15.44:0/1360124057 wait complete. 2024-04-02T13:26:24.591 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:26:24.591 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:26:24.591 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:26:24.591 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:26:24.591 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:24.591 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:26:24.591 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:26:24.591 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:26:24.591 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:26:24.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:24.592 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:26:24.607 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:26:24.607 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:26:24.607 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:26:24.608 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:24.608 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:26:24.608 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:24.608 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:26:24.608 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:26:24.608 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:26:24.608 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:26:24.608 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:26:24.608 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:26:24.608 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:26:24.608 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:26:24.608 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:26:24.609 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:26:24.609 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42394sec preferred_lft 42394sec 2024-04-02T13:26:24.609 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:26:24.609 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:24.609 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:26:24.609 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:26:24.609 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:26:24.609 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:26:24.609 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:26:24.609 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:26:24.609 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:24.609 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:26:24.609 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:24.610 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:26:24.610 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:26:24.610 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:26:24.610 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:24.610 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:26:24.610 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:26:24.610 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:26:24.610 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:24.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:24.611 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:26:24.673 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:26:24.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:24.675 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:26:24.730 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:26:24.730 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:26:24.730 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:26:24.730 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:26:24.731 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:26:24.731 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:26:24.731 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:26:24.731 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:26:24.731 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:26:24.731 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:26:24.731 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:26:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:26:24.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:26:24.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:24.866 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:26:24.880 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:26:24.880 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:26:24.880 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:26:24.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:24.881 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:26:24.946 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:26:24.946 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:26:24.947 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:25.002 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:26:25.002 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:26:25.002 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:25.056 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:26:25.124 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:26:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:26:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:26:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:26:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:26:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:26:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:26:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:26:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:26:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:26:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:26:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:26:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:26:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:26:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:26:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:26:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:26:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:26:25.183 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:26:25.183 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:26:25.183 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:26:25.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:25.183 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:26:25.235 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:26:25.235 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:26:25.277 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:26:25.303 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:26:25.359 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:26:25.359 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:26:25.359 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:26:25.359 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:26:25.359 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:26:25.359 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:26:25.359 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:26:25.359 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:26:25.359 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:26:25.360 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:26:25.360 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:26:25.360 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:26:25.360 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:26:25.360 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:26:25.360 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:26:25.360 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:26:25.360 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:26:25.360 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:26:25.360 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:26:25.360 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:26:25.360 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:26:25.360 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:26:25.361 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:26:25.361 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:26:25.361 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:26:25.361 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:26:25.361 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:26:25.361 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:26:25.361 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:26:25.361 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:26:25.361 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:26:25.361 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:26:25.361 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:26:25.361 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:26:25.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:25.362 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:26:25.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:26:25.360+0000 7f2f35e4c480 -1 init, newargv = 0x558805d21020 newargc=15 2024-04-02T13:26:25.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[67119]: starting ceph client 2024-04-02T13:26:25.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[67119]: starting fuse 2024-04-02T13:26:25.377 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:26:25.377 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:26:25.377 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:26:25.377 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:26:25.378 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:26:25.378 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:26:25.378 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:26:25.378 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:26:25.378 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:26:25.378 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:26:25.378 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:26:25.378 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:26:25.379 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:26:25.379 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:26:25.379 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:26:25.379 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:26:25.379 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:26:25.379 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:26:25.379 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:26:25.379 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:26:25.380 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:26:25.380 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:26:25.380 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:26:25.380 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:26:25.380 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:26:25.380 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:26:25.381 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:26:25.381 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:26:25.381 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:26:25.498 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.67119.asok 2024-04-02T13:26:25.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:26:25.508 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.67119.asok 2024-04-02T13:26:25.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:25.508 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.67119.asok status 2024-04-02T13:26:25.646 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:26:25.646 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:26:25.646 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:26:25.646 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:26:25.646 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:26:25.646 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:26:25.646 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:26:25.646 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "67119", 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 7599, 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 7599 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 1673619137 2024-04-02T13:26:25.647 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:25.648 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:25.648 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:26:25.648 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:26:25.648 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:26:25.648 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 1673619137 2024-04-02T13:26:25.648 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:25.648 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.7599 192.168.0.1:0/1673619137", 2024-04-02T13:26:25.648 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/1673619137", 2024-04-02T13:26:25.648 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:26:25.648 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 52, 2024-04-02T13:26:25.648 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 80, 2024-04-02T13:26:25.648 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:26:25.649 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:26:25.649 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:26:25.649 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:26:25.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:25.654 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:26:25.654 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:25.669 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:26:25.669 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:25.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:25.670 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:25.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:25.737 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-04-02T13:26:25.794 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:26:25.794 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-04-02T13:26:25.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.873+0000 7fe433240f40 1 Processor -- start 2024-04-02T13:26:25.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.873+0000 7fe433240f40 1 -- start start 2024-04-02T13:26:25.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe433240f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cfab2fd380 0x55cfab3cf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:25.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe433240f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfab32eb30 0x55cfab3b3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:25.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe433240f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cfab3d5ee0 0x55cfab3d7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:25.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe433240f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cfab2a53e0 con 0x55cfab32eb30 2024-04-02T13:26:25.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe433240f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55cfab3b9590 con 0x55cfab2fd380 2024-04-02T13:26:25.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe433240f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55cfab3286c0 con 0x55cfab3d5ee0 2024-04-02T13:26:25.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe427fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfab32eb30 0x55cfab3b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:25.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe427fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfab32eb30 0x55cfab3b3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45800/0 (socket says 172.21.15.44:45800) 2024-04-02T13:26:25.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe427fff640 1 -- 172.21.15.44:0/3180906064 learned_addr learned my addr 172.21.15.44:0/3180906064 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:25.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe430581640 1 --2- 172.21.15.44:0/3180906064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cfab2fd380 0x55cfab3cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:25.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe427fff640 1 -- 172.21.15.44:0/3180906064 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cfab3d5ee0 msgr2=0x55cfab3d7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:25.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe427fff640 1 --2- 172.21.15.44:0/3180906064 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cfab3d5ee0 0x55cfab3d7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe427fff640 1 -- 172.21.15.44:0/3180906064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cfab2fd380 msgr2=0x55cfab3cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe427fff640 1 --2- 172.21.15.44:0/3180906064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cfab2fd380 0x55cfab3cf100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.874+0000 7fe427fff640 1 -- 172.21.15.44:0/3180906064 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfab3cf640 con 0x55cfab32eb30 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.875+0000 7fe427fff640 1 --2- 172.21.15.44:0/3180906064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfab32eb30 0x55cfab3b3970 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fe41800a4f0 tx=0x7fe41804e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32732ea5062aa1be server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.875+0000 7fe42f57f640 1 -- 172.21.15.44:0/3180906064 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418057070 con 0x55cfab32eb30 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.875+0000 7fe42f57f640 1 -- 172.21.15.44:0/3180906064 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe418061070 con 0x55cfab32eb30 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.875+0000 7fe42f57f640 1 -- 172.21.15.44:0/3180906064 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4180529c0 con 0x55cfab32eb30 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.875+0000 7fe433240f40 1 -- 172.21.15.44:0/3180906064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfab32eb30 msgr2=0x55cfab3b3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.875+0000 7fe433240f40 1 --2- 172.21.15.44:0/3180906064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfab32eb30 0x55cfab3b3970 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fe41800a4f0 tx=0x7fe41804e000 comp rx=0 tx=0).stop 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.875+0000 7fe433240f40 1 -- 172.21.15.44:0/3180906064 shutdown_connections 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.875+0000 7fe433240f40 1 --2- 172.21.15.44:0/3180906064 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cfab3d5ee0 0x55cfab3d7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.875+0000 7fe433240f40 1 --2- 172.21.15.44:0/3180906064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfab32eb30 0x55cfab3b3970 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.875+0000 7fe433240f40 1 --2- 172.21.15.44:0/3180906064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cfab2fd380 0x55cfab3cf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:25.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.875+0000 7fe433240f40 1 -- 172.21.15.44:0/3180906064 >> 172.21.15.44:0/3180906064 conn(0x55cfab2fe1c0 msgr2=0x55cfab3b7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.875+0000 7fe433240f40 1 -- 172.21.15.44:0/3180906064 shutdown_connections 2024-04-02T13:26:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.875+0000 7fe433240f40 1 -- 172.21.15.44:0/3180906064 wait complete. 2024-04-02T13:26:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.876+0000 7fe433240f40 1 Processor -- start 2024-04-02T13:26:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.876+0000 7fe433240f40 1 -- start start 2024-04-02T13:26:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe433240f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfab2fd380 0x55cfab3cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe433240f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cfab32eb30 0x55cfab344540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe433240f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cfab3d5ee0 0x55cfab3b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe433240f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cfab3d90f0 con 0x55cfab2fd380 2024-04-02T13:26:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe433240f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55cfab3b87b0 con 0x55cfab32eb30 2024-04-02T13:26:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe433240f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55cfab2fe950 con 0x55cfab3d5ee0 2024-04-02T13:26:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe430581640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfab2fd380 0x55cfab3cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe430581640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfab2fd380 0x55cfab3cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45806/0 (socket says 172.21.15.44:45806) 2024-04-02T13:26:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe430581640 1 -- 172.21.15.44:0/2253132202 learned_addr learned my addr 172.21.15.44:0/2253132202 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe427fff640 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cfab32eb30 0x55cfab344540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe430d82640 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cfab3d5ee0 0x55cfab3b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe430581640 1 -- 172.21.15.44:0/2253132202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cfab3d5ee0 msgr2=0x55cfab3b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe430581640 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cfab3d5ee0 0x55cfab3b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe430581640 1 -- 172.21.15.44:0/2253132202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cfab32eb30 msgr2=0x55cfab344540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe430581640 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cfab32eb30 0x55cfab344540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe430581640 1 -- 172.21.15.44:0/2253132202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfab46e840 con 0x55cfab2fd380 2024-04-02T13:26:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe430d82640 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cfab3d5ee0 0x55cfab3b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe430581640 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfab2fd380 0x55cfab3cde40 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fe42000cec0 tx=0x7fe42000a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe42e57d640 1 -- 172.21.15.44:0/2253132202 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420008040 con 0x55cfab2fd380 2024-04-02T13:26:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe42e57d640 1 -- 172.21.15.44:0/2253132202 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe420014020 con 0x55cfab2fd380 2024-04-02T13:26:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe433240f40 1 -- 172.21.15.44:0/2253132202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cfab344a80 con 0x55cfab2fd380 2024-04-02T13:26:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe42e57d640 1 -- 172.21.15.44:0/2253132202 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42000f400 con 0x55cfab2fd380 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.877+0000 7fe433240f40 1 -- 172.21.15.44:0/2253132202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfab3ce7e0 con 0x55cfab2fd380 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.879+0000 7fe42e57d640 1 -- 172.21.15.44:0/2253132202 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe42000f5c0 con 0x55cfab2fd380 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.879+0000 7fe42e57d640 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe41006b300 0x7fe41006d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.879+0000 7fe42e57d640 1 -- 172.21.15.44:0/2253132202 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fe42008a750 con 0x55cfab2fd380 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.879+0000 7fe427fff640 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe41006b300 0x7fe41006d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.879+0000 7fe433240f40 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55cfab3b45f0 0x55cfab3b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.879+0000 7fe433240f40 1 -- 172.21.15.44:0/2253132202 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:1 11.10 11:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80) -- 0x55cfab3b4fa0 con 0x55cfab3b45f0 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.879+0000 7fe430d82640 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55cfab3b45f0 0x55cfab3b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.880+0000 7fe427fff640 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe41006b300 0x7fe41006d7c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe418055040 tx=0x7fe418005a00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.880+0000 7fe430d82640 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55cfab3b45f0 0x55cfab3b4a60 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.881+0000 7fe430d82640 1 -- 172.21.15.44:0/2253132202 <== osd.0 v2:172.21.15.44:6800/2303805525 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v80'12 uv12 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe42800b520 con 0x55cfab3b45f0 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.881+0000 7fe433240f40 1 -- 172.21.15.44:0/2253132202 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55cfab3b45f0 msgr2=0x55cfab3b4a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.881+0000 7fe433240f40 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55cfab3b45f0 0x55cfab3b4a60 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.881+0000 7fe433240f40 1 -- 172.21.15.44:0/2253132202 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe41006b300 msgr2=0x7fe41006d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:25.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.881+0000 7fe433240f40 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe41006b300 0x7fe41006d7c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe418055040 tx=0x7fe418005a00 comp rx=0 tx=0).stop 2024-04-02T13:26:25.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.881+0000 7fe433240f40 1 -- 172.21.15.44:0/2253132202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfab2fd380 msgr2=0x55cfab3cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:25.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.881+0000 7fe433240f40 1 --2- 172.21.15.44:0/2253132202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfab2fd380 0x55cfab3cde40 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fe42000cec0 tx=0x7fe42000a5d0 comp rx=0 tx=0).stop 2024-04-02T13:26:25.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.881+0000 7fe430d82640 1 -- 172.21.15.44:0/2253132202 reap_dead start 2024-04-02T13:26:25.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.881+0000 7fe433240f40 1 -- 172.21.15.44:0/2253132202 shutdown_connections 2024-04-02T13:26:25.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.881+0000 7fe433240f40 1 -- 172.21.15.44:0/2253132202 >> 172.21.15.44:0/2253132202 conn(0x55cfab2fe1c0 msgr2=0x55cfab3bb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:25.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.881+0000 7fe433240f40 1 -- 172.21.15.44:0/2253132202 shutdown_connections 2024-04-02T13:26:25.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:25.881+0000 7fe433240f40 1 -- 172.21.15.44:0/2253132202 wait complete. 2024-04-02T13:26:25.897 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:26:25.897 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:26:25.969 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:26:26.017 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:26:26.017 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:26:26.017 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:26:26.017 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:26:26.033 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:26:26.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.170+0000 7f966790e640 1 Processor -- start 2024-04-02T13:26:26.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.171+0000 7f966790e640 1 -- start start 2024-04-02T13:26:26.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.171+0000 7f966790e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f96600d0c50 0x7f96600d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.171+0000 7f966790e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f966006ce40 0x7f966006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.171+0000 7f966790e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96600cc600 con 0x7f966015e4a0 2024-04-02T13:26:26.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.171+0000 7f966790e640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f96600af300 con 0x7f966006ce40 2024-04-02T13:26:26.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.171+0000 7f966790e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f96600cc8e0 con 0x7f96600d0c50 2024-04-02T13:26:26.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.172+0000 7f9665eab640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f966006ce40 0x7f966006d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.172+0000 7f9665eab640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f966006ce40 0x7f966006d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45810/0 (socket says 172.21.15.44:45810) 2024-04-02T13:26:26.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.172+0000 7f9665eab640 1 -- 172.21.15.44:0/3530362319 learned_addr learned my addr 172.21.15.44:0/3530362319 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:26.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.172+0000 7f9664ea9640 1 --2- 172.21.15.44:0/3530362319 >> v2:172.21.15.69:3300/0 conn(0x7f96600d0c50 0x7f96600d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.172+0000 7f9665eab640 1 -- 172.21.15.44:0/3530362319 >> v2:172.21.15.69:3300/0 conn(0x7f96600d0c50 msgr2=0x7f96600d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:26.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.172+0000 7f9665eab640 1 --2- 172.21.15.44:0/3530362319 >> v2:172.21.15.69:3300/0 conn(0x7f96600d0c50 0x7f96600d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.172+0000 7f9665eab640 1 -- 172.21.15.44:0/3530362319 >> v1:172.21.15.69:6789/0 conn(0x7f966015e4a0 legacy=0x7f96600aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:26.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.172+0000 7f9665eab640 1 -- 172.21.15.44:0/3530362319 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96600ccbc0 con 0x7f966006ce40 2024-04-02T13:26:26.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.173+0000 7f9665eab640 1 --2- 172.21.15.44:0/3530362319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f966006ce40 0x7f966006d2f0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f96600b8060 tx=0x7f965802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cdbc8678f15e7ca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:26.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.173+0000 7f9657fff640 1 -- 172.21.15.44:0/3530362319 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965802ca80 con 0x7f966006ce40 2024-04-02T13:26:26.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.173+0000 7f9657fff640 1 -- 172.21.15.44:0/3530362319 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f965802cc20 con 0x7f966006ce40 2024-04-02T13:26:26.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.173+0000 7f9657fff640 1 -- 172.21.15.44:0/3530362319 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96580343f0 con 0x7f966006ce40 2024-04-02T13:26:26.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.173+0000 7f966790e640 1 -- 172.21.15.44:0/3530362319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f966006ce40 msgr2=0x7f966006d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.173+0000 7f966790e640 1 --2- 172.21.15.44:0/3530362319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f966006ce40 0x7f966006d2f0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f96600b8060 tx=0x7f965802bee0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.173+0000 7f966790e640 1 -- 172.21.15.44:0/3530362319 shutdown_connections 2024-04-02T13:26:26.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.173+0000 7f966790e640 1 --2- 172.21.15.44:0/3530362319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f966006ce40 0x7f966006d2f0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.173+0000 7f966790e640 1 --2- 172.21.15.44:0/3530362319 >> v2:172.21.15.69:3300/0 conn(0x7f96600d0c50 0x7f96600d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.173+0000 7f966790e640 1 -- 172.21.15.44:0/3530362319 >> 172.21.15.44:0/3530362319 conn(0x7f966009aa20 msgr2=0x7f966009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:26.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.173+0000 7f966790e640 1 -- 172.21.15.44:0/3530362319 shutdown_connections 2024-04-02T13:26:26.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.173+0000 7f966790e640 1 -- 172.21.15.44:0/3530362319 wait complete. 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.175+0000 7f966790e640 1 Processor -- start 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.175+0000 7f966790e640 1 -- start start 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.175+0000 7f966790e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f966006ce40 0x7f96600c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.175+0000 7f966790e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96600d0c50 0x7f966006fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.175+0000 7f966790e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f966015e4a0 0x7f9660070130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.175+0000 7f966790e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f96600cd460 con 0x7f96600d0c50 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.175+0000 7f966790e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f96600cd180 con 0x7f966006ce40 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.175+0000 7f966790e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f966015e880 con 0x7f966015e4a0 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.175+0000 7f9664ea9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96600d0c50 0x7f966006fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.175+0000 7f9664ea9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96600d0c50 0x7f966006fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45826/0 (socket says 172.21.15.44:45826) 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.175+0000 7f9664ea9640 1 -- 172.21.15.44:0/2532140588 learned_addr learned my addr 172.21.15.44:0/2532140588 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.176+0000 7f9665eab640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f966015e4a0 0x7f9660070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.176+0000 7f96656aa640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f966006ce40 0x7f96600c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.176+0000 7f9664ea9640 1 -- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f966015e4a0 msgr2=0x7f9660070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.176+0000 7f9664ea9640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f966015e4a0 0x7f9660070130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.176+0000 7f9664ea9640 1 -- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f966006ce40 msgr2=0x7f96600c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.176+0000 7f9664ea9640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f966006ce40 0x7f96600c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.176+0000 7f9664ea9640 1 -- 172.21.15.44:0/2532140588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96600c9f60 con 0x7f96600d0c50 2024-04-02T13:26:26.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.176+0000 7f9664ea9640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96600d0c50 0x7f966006fbf0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f965000c7f0 tx=0x7f965000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:26.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.176+0000 7f9655ffb640 1 -- 172.21.15.44:0/2532140588 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650015070 con 0x7f96600d0c50 2024-04-02T13:26:26.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.176+0000 7f9655ffb640 1 -- 172.21.15.44:0/2532140588 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9650004220 con 0x7f96600d0c50 2024-04-02T13:26:26.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.176+0000 7f9655ffb640 1 -- 172.21.15.44:0/2532140588 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650007620 con 0x7f96600d0c50 2024-04-02T13:26:26.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.176+0000 7f966790e640 1 -- 172.21.15.44:0/2532140588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96600ca240 con 0x7f96600d0c50 2024-04-02T13:26:26.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.176+0000 7f966790e640 1 -- 172.21.15.44:0/2532140588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96600ca800 con 0x7f96600d0c50 2024-04-02T13:26:26.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.177+0000 7f966790e640 1 -- 172.21.15.44:0/2532140588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96600684f0 con 0x7f96600d0c50 2024-04-02T13:26:26.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.178+0000 7f9655ffb640 1 -- 172.21.15.44:0/2532140588 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f96500079d0 con 0x7f96600d0c50 2024-04-02T13:26:26.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.178+0000 7f9655ffb640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f964406c8f0 0x7f964406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.178+0000 7f9655ffb640 1 -- 172.21.15.44:0/2532140588 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f965008c080 con 0x7f96600d0c50 2024-04-02T13:26:26.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.178+0000 7f96656aa640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f964406c8f0 0x7f964406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.179+0000 7f96656aa640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f964406c8f0 0x7f964406edb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f965c006fd0 tx=0x7f965c009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:26.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.182+0000 7f9655ffb640 1 -- 172.21.15.44:0/2532140588 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9650059c80 con 0x7f96600d0c50 2024-04-02T13:26:26.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.367+0000 7f966790e640 1 -- 172.21.15.44:0/2532140588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f96600c9ca0 con 0x7f96600d0c50 2024-04-02T13:26:26.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.368+0000 7f9655ffb640 1 -- 172.21.15.44:0/2532140588 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+17024 (secure 0 0 0) 0x7f965005df80 con 0x7f96600d0c50 2024-04-02T13:26:26.372 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:26.372 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":80,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:25:10.921159+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:24:52.202720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-02T13:24:52.916529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6837/698185867":"2024-04-03T13:25:10.921135+0000","172.21.15.69:6835/698185867":"2024-04-03T13:25:10.921135+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:26:26.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.370+0000 7f966790e640 1 -- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f964406c8f0 msgr2=0x7f964406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.370+0000 7f966790e640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f964406c8f0 0x7f964406edb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f965c006fd0 tx=0x7f965c009060 comp rx=0 tx=0).stop 2024-04-02T13:26:26.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.370+0000 7f966790e640 1 -- 172.21.15.44:0/2532140588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96600d0c50 msgr2=0x7f966006fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.370+0000 7f966790e640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96600d0c50 0x7f966006fbf0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f965000c7f0 tx=0x7f965000ccb0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.370+0000 7f966790e640 1 -- 172.21.15.44:0/2532140588 shutdown_connections 2024-04-02T13:26:26.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.370+0000 7f966790e640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f964406c8f0 0x7f964406edb0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.370+0000 7f966790e640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f966015e4a0 0x7f9660070130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.370+0000 7f966790e640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96600d0c50 0x7f966006fbf0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.370+0000 7f966790e640 1 --2- 172.21.15.44:0/2532140588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f966006ce40 0x7f96600c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.370+0000 7f966790e640 1 -- 172.21.15.44:0/2532140588 >> 172.21.15.44:0/2532140588 conn(0x7f966009aa20 msgr2=0x7f966006d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:26.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.370+0000 7f966790e640 1 -- 172.21.15.44:0/2532140588 shutdown_connections 2024-04-02T13:26:26.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.370+0000 7f966790e640 1 -- 172.21.15.44:0/2532140588 wait complete. 2024-04-02T13:26:26.384 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:24:52.202720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:24:52.916529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:26:26.385 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:26:26.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.485+0000 7fda1c286640 1 Processor -- start 2024-04-02T13:26:26.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.485+0000 7fda1c286640 1 -- start start 2024-04-02T13:26:26.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda1c286640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fda140cd770 0x7fda1406eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda1c286640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fda140ceec0 0x7fda140cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda1c286640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1406f3f0 con 0x7fda1406b4d0 2024-04-02T13:26:26.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda1c286640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fda1406f6d0 con 0x7fda140ceec0 2024-04-02T13:26:26.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda1c286640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fda1406f9b0 con 0x7fda140cd770 2024-04-02T13:26:26.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda1a823640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fda140ceec0 0x7fda140cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda1a022640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fda1406b4d0 0x7fda1406b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:36586/0 (socket says 172.21.15.44:36586) 2024-04-02T13:26:26.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda1a022640 1 -- 172.21.15.44:0/997418635 learned_addr learned my addr 172.21.15.44:0/997418635 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:26.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda19821640 1 --2- 172.21.15.44:0/997418635 >> v2:172.21.15.69:3300/0 conn(0x7fda140cd770 0x7fda1406eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda1a823640 1 -- 172.21.15.44:0/997418635 >> v2:172.21.15.69:3300/0 conn(0x7fda140cd770 msgr2=0x7fda1406eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:26.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda1a823640 1 --2- 172.21.15.44:0/997418635 >> v2:172.21.15.69:3300/0 conn(0x7fda140cd770 0x7fda1406eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda1a823640 1 -- 172.21.15.44:0/997418635 >> v1:172.21.15.44:6789/0 conn(0x7fda1406b4d0 legacy=0x7fda1406b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda19020640 1 -- 172.21.15.44:0/997418635 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3642150618 0 0) 0x7fda1406f3f0 con 0x7fda1406b4d0 2024-04-02T13:26:26.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.486+0000 7fda1a823640 1 -- 172.21.15.44:0/997418635 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1406fc90 con 0x7fda140ceec0 2024-04-02T13:26:26.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.487+0000 7fda1a823640 1 --2- 172.21.15.44:0/997418635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda140ceec0 0x7fda140cf310 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fda14075360 tx=0x7fda0402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c0a5221c30436c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:26.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.487+0000 7fda19020640 1 -- 172.21.15.44:0/997418635 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda04039070 con 0x7fda140ceec0 2024-04-02T13:26:26.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.487+0000 7fda19020640 1 -- 172.21.15.44:0/997418635 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda0402cc80 con 0x7fda140ceec0 2024-04-02T13:26:26.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.487+0000 7fda19020640 1 -- 172.21.15.44:0/997418635 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda04034660 con 0x7fda140ceec0 2024-04-02T13:26:26.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.488+0000 7fda1c286640 1 -- 172.21.15.44:0/997418635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda140ceec0 msgr2=0x7fda140cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.488+0000 7fda1c286640 1 --2- 172.21.15.44:0/997418635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda140ceec0 0x7fda140cf310 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fda14075360 tx=0x7fda0402bed0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.488+0000 7fda1c286640 1 -- 172.21.15.44:0/997418635 shutdown_connections 2024-04-02T13:26:26.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.488+0000 7fda1c286640 1 --2- 172.21.15.44:0/997418635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda140ceec0 0x7fda140cf310 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.488+0000 7fda1c286640 1 --2- 172.21.15.44:0/997418635 >> v2:172.21.15.69:3300/0 conn(0x7fda140cd770 0x7fda1406eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.488+0000 7fda1c286640 1 -- 172.21.15.44:0/997418635 >> 172.21.15.44:0/997418635 conn(0x7fda140b2a10 msgr2=0x7fda140b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:26.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.488+0000 7fda1c286640 1 -- 172.21.15.44:0/997418635 shutdown_connections 2024-04-02T13:26:26.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.488+0000 7fda1c286640 1 -- 172.21.15.44:0/997418635 wait complete. 2024-04-02T13:26:26.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.489+0000 7fda1c286640 1 Processor -- start 2024-04-02T13:26:26.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.489+0000 7fda1c286640 1 -- start start 2024-04-02T13:26:26.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1c286640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda1406b4d0 0x7fda140af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1c286640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda140cd770 0x7fda1414c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1c286640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda140ceec0 0x7fda1414cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1c286640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fda1415c470 con 0x7fda1406b4d0 2024-04-02T13:26:26.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1c286640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fda1406f0e0 con 0x7fda140ceec0 2024-04-02T13:26:26.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1c286640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fda1415c5f0 con 0x7fda140cd770 2024-04-02T13:26:26.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1a022640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda1406b4d0 0x7fda140af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1a022640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda1406b4d0 0x7fda140af020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45848/0 (socket says 172.21.15.44:45848) 2024-04-02T13:26:26.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1a022640 1 -- 172.21.15.44:0/2758396876 learned_addr learned my addr 172.21.15.44:0/2758396876 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:26.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1a823640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda140ceec0 0x7fda1414cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda19821640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda140cd770 0x7fda1414c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1a022640 1 -- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda140cd770 msgr2=0x7fda1414c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1a022640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda140cd770 0x7fda1414c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1a022640 1 -- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda140ceec0 msgr2=0x7fda1414cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1a022640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda140ceec0 0x7fda1414cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1a022640 1 -- 172.21.15.44:0/2758396876 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1409e5a0 con 0x7fda1406b4d0 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1a823640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda140ceec0 0x7fda1414cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.490+0000 7fda1a022640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda1406b4d0 0x7fda140af020 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00c4d0 tx=0x7fda0c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.491+0000 7fda12ffd640 1 -- 172.21.15.44:0/2758396876 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0c018070 con 0x7fda1406b4d0 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.491+0000 7fda12ffd640 1 -- 172.21.15.44:0/2758396876 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda0c007500 con 0x7fda1406b4d0 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.491+0000 7fda12ffd640 1 -- 172.21.15.44:0/2758396876 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0c004a20 con 0x7fda1406b4d0 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.491+0000 7fda1c286640 1 -- 172.21.15.44:0/2758396876 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda1415dd50 con 0x7fda1406b4d0 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.491+0000 7fda1c286640 1 -- 172.21.15.44:0/2758396876 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda140af560 con 0x7fda1406b4d0 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.492+0000 7fda1c286640 1 -- 172.21.15.44:0/2758396876 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda1406f490 con 0x7fda1406b4d0 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.492+0000 7fda12ffd640 1 -- 172.21.15.44:0/2758396876 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fda0c01b070 con 0x7fda1406b4d0 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.492+0000 7fda12ffd640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fda0006c8f0 0x7fda0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.493+0000 7fda12ffd640 1 -- 172.21.15.44:0/2758396876 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fda0c08afe0 con 0x7fda1406b4d0 2024-04-02T13:26:26.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.493+0000 7fda19821640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fda0006c8f0 0x7fda0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.493+0000 7fda19821640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fda0006c8f0 0x7fda0006edb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fda1409b0e0 tx=0x7fda08008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:26.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.497+0000 7fda12ffd640 1 -- 172.21.15.44:0/2758396876 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fda1406f490 con 0x7fda1406b4d0 2024-04-02T13:26:26.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.682+0000 7fda1c286640 1 -- 172.21.15.44:0/2758396876 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fda140684f0 con 0x7fda1406b4d0 2024-04-02T13:26:26.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.682+0000 7fda12ffd640 1 -- 172.21.15.44:0/2758396876 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) ==== 93+0+10 (secure 0 0 0) 0x7fda0c0592c0 con 0x7fda1406b4d0 2024-04-02T13:26:26.686 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:26:26.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.684+0000 7fda1c286640 1 -- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fda0006c8f0 msgr2=0x7fda0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.684+0000 7fda1c286640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fda0006c8f0 0x7fda0006edb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fda1409b0e0 tx=0x7fda08008040 comp rx=0 tx=0).stop 2024-04-02T13:26:26.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.684+0000 7fda1c286640 1 -- 172.21.15.44:0/2758396876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda1406b4d0 msgr2=0x7fda140af020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.684+0000 7fda1c286640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda1406b4d0 0x7fda140af020 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00c4d0 tx=0x7fda0c00c990 comp rx=0 tx=0).stop 2024-04-02T13:26:26.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.685+0000 7fda1c286640 1 -- 172.21.15.44:0/2758396876 shutdown_connections 2024-04-02T13:26:26.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.685+0000 7fda1c286640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fda0006c8f0 0x7fda0006edb0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.685+0000 7fda1c286640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda140ceec0 0x7fda1414cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.685+0000 7fda1c286640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda140cd770 0x7fda1414c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.685+0000 7fda1c286640 1 --2- 172.21.15.44:0/2758396876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda1406b4d0 0x7fda140af020 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.685+0000 7fda1c286640 1 -- 172.21.15.44:0/2758396876 >> 172.21.15.44:0/2758396876 conn(0x7fda140b2a10 msgr2=0x7fda140b1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:26.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.685+0000 7fda1c286640 1 -- 172.21.15.44:0/2758396876 shutdown_connections 2024-04-02T13:26:26.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.685+0000 7fda1c286640 1 -- 172.21.15.44:0/2758396876 wait complete. 2024-04-02T13:26:26.698 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:26:26.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.800+0000 7ff101a81640 1 Processor -- start 2024-04-02T13:26:26.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.800+0000 7ff101a81640 1 -- start start 2024-04-02T13:26:26.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff101a81640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0fc15e260 0x7ff0fc0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff101a81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0cec60 0x7ff0fc168280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff101a81640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0fc0cf120 con 0x7ff0fc15e640 2024-04-02T13:26:26.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff101a81640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff0fc0b5a50 con 0x7ff0fc15e260 2024-04-02T13:26:26.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff101a81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0fc0d01c0 con 0x7ff0fc0cec60 2024-04-02T13:26:26.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff0fb7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0fc15e260 0x7ff0fc0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff0faffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff0fc15e640 0x7ff0fc0d0ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:36600/0 (socket says 172.21.15.44:36600) 2024-04-02T13:26:26.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff0faffd640 1 -- 172.21.15.44:0/334872456 learned_addr learned my addr 172.21.15.44:0/334872456 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:26.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff0fbfff640 1 --2- 172.21.15.44:0/334872456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0cec60 0x7ff0fc168280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff0fb7fe640 1 -- 172.21.15.44:0/334872456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0cec60 msgr2=0x7ff0fc168280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:26.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff0fb7fe640 1 --2- 172.21.15.44:0/334872456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0cec60 0x7ff0fc168280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff0fb7fe640 1 -- 172.21.15.44:0/334872456 >> v1:172.21.15.44:6789/0 conn(0x7ff0fc15e640 legacy=0x7ff0fc0d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff0fb7fe640 1 -- 172.21.15.44:0/334872456 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc0cf400 con 0x7ff0fc15e260 2024-04-02T13:26:26.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.801+0000 7ff0fa7fc640 1 -- 172.21.15.44:0/334872456 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618597942 0 0) 0x7ff0fc0cf120 con 0x7ff0fc15e640 2024-04-02T13:26:26.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.802+0000 7ff0fb7fe640 1 --2- 172.21.15.44:0/334872456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc15e260 0x7ff0fc0d2960 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec0097a0 tx=0x7ff0ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae67544d4a2f170 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:26.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.802+0000 7ff0fa7fc640 1 -- 172.21.15.44:0/334872456 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec035020 con 0x7ff0fc15e260 2024-04-02T13:26:26.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.802+0000 7ff0fa7fc640 1 -- 172.21.15.44:0/334872456 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0ec031830 con 0x7ff0fc15e260 2024-04-02T13:26:26.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.802+0000 7ff0fa7fc640 1 -- 172.21.15.44:0/334872456 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec031c00 con 0x7ff0fc15e260 2024-04-02T13:26:26.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.802+0000 7ff101a81640 1 -- 172.21.15.44:0/334872456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc15e260 msgr2=0x7ff0fc0d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.802+0000 7ff101a81640 1 --2- 172.21.15.44:0/334872456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc15e260 0x7ff0fc0d2960 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec0097a0 tx=0x7ff0ec02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.802+0000 7ff101a81640 1 -- 172.21.15.44:0/334872456 shutdown_connections 2024-04-02T13:26:26.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.802+0000 7ff101a81640 1 --2- 172.21.15.44:0/334872456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc0cec60 0x7ff0fc168280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.802+0000 7ff101a81640 1 --2- 172.21.15.44:0/334872456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc15e260 0x7ff0fc0d2960 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.802+0000 7ff101a81640 1 -- 172.21.15.44:0/334872456 >> 172.21.15.44:0/334872456 conn(0x7ff0fc09d3c0 msgr2=0x7ff0fc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:26.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.803+0000 7ff101a81640 1 -- 172.21.15.44:0/334872456 shutdown_connections 2024-04-02T13:26:26.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.803+0000 7ff101a81640 1 -- 172.21.15.44:0/334872456 wait complete. 2024-04-02T13:26:26.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.804+0000 7ff101a81640 1 Processor -- start 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.804+0000 7ff101a81640 1 -- start start 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.804+0000 7ff101a81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0fc0cec60 0x7ff0fc15ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.804+0000 7ff101a81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc15e260 0x7ff0fc0ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.804+0000 7ff101a81640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc15e640 0x7ff0fc14b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.804+0000 7ff101a81640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc0cfca0 con 0x7ff0fc15e640 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.804+0000 7ff101a81640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc0cf9c0 con 0x7ff0fc0cec60 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.804+0000 7ff101a81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0fc0d11e0 con 0x7ff0fc15e260 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0fbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc15e640 0x7ff0fc14b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0fbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc15e640 0x7ff0fc14b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45866/0 (socket says 172.21.15.44:45866) 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0fbfff640 1 -- 172.21.15.44:0/3170738478 learned_addr learned my addr 172.21.15.44:0/3170738478 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0fb7fe640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0fc0cec60 0x7ff0fc15ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0faffd640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc15e260 0x7ff0fc0ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0fbfff640 1 -- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc15e260 msgr2=0x7ff0fc0ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0fbfff640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc15e260 0x7ff0fc0ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0fbfff640 1 -- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0fc0cec60 msgr2=0x7ff0fc15ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:26.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0fbfff640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0fc0cec60 0x7ff0fc15ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:26.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0fbfff640 1 -- 172.21.15.44:0/3170738478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc15bac0 con 0x7ff0fc15e640 2024-04-02T13:26:26.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0faffd640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc15e260 0x7ff0fc0ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:26.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0fbfff640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc15e640 0x7ff0fc14b170 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400cdc0 tx=0x7ff0f400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:26.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0e3fff640 1 -- 172.21.15.44:0/3170738478 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4003b90 con 0x7ff0fc15e640 2024-04-02T13:26:26.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0e3fff640 1 -- 172.21.15.44:0/3170738478 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f4003d50 con 0x7ff0fc15e640 2024-04-02T13:26:26.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff0e3fff640 1 -- 172.21.15.44:0/3170738478 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4004260 con 0x7ff0fc15e640 2024-04-02T13:26:26.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff101a81640 1 -- 172.21.15.44:0/3170738478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0fc15f080 con 0x7ff0fc15e640 2024-04-02T13:26:26.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.805+0000 7ff101a81640 1 -- 172.21.15.44:0/3170738478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0fc0b84e0 con 0x7ff0fc15e640 2024-04-02T13:26:26.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.806+0000 7ff101a81640 1 -- 172.21.15.44:0/3170738478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0fc0b7ff0 con 0x7ff0fc15e640 2024-04-02T13:26:26.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.807+0000 7ff0e3fff640 1 -- 172.21.15.44:0/3170738478 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0f4004400 con 0x7ff0fc15e640 2024-04-02T13:26:26.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.807+0000 7ff0e3fff640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0e806c8f0 0x7ff0e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:26.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.807+0000 7ff0e3fff640 1 -- 172.21.15.44:0/3170738478 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7ff0f408c6e0 con 0x7ff0fc15e640 2024-04-02T13:26:26.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.808+0000 7ff0fb7fe640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0e806c8f0 0x7ff0e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:26.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.808+0000 7ff0fb7fe640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0e806c8f0 0x7ff0e806edb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec036000 tx=0x7ff0ec008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:26.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.812+0000 7ff0e3fff640 1 -- 172.21.15.44:0/3170738478 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff0fc15bac0 con 0x7ff0fc15e640 2024-04-02T13:26:26.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.994+0000 7ff101a81640 1 -- 172.21.15.44:0/3170738478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff0fc0684f0 con 0x7ff0fc15e640 2024-04-02T13:26:26.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.995+0000 7ff0e3fff640 1 -- 172.21.15.44:0/3170738478 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) ==== 104+0+11 (secure 0 0 0) 0x7ff0f405a930 con 0x7ff0fc15e640 2024-04-02T13:26:26.998 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:26:27.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.997+0000 7ff101a81640 1 -- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0e806c8f0 msgr2=0x7ff0e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.997+0000 7ff101a81640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0e806c8f0 0x7ff0e806edb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec036000 tx=0x7ff0ec008030 comp rx=0 tx=0).stop 2024-04-02T13:26:27.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.997+0000 7ff101a81640 1 -- 172.21.15.44:0/3170738478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc15e640 msgr2=0x7ff0fc14b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.997+0000 7ff101a81640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc15e640 0x7ff0fc14b170 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400cdc0 tx=0x7ff0f400a660 comp rx=0 tx=0).stop 2024-04-02T13:26:27.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.997+0000 7ff101a81640 1 -- 172.21.15.44:0/3170738478 shutdown_connections 2024-04-02T13:26:27.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.997+0000 7ff101a81640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0e806c8f0 0x7ff0e806edb0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.997+0000 7ff101a81640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0fc15e640 0x7ff0fc14b170 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.997+0000 7ff101a81640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0fc15e260 0x7ff0fc0ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.997+0000 7ff101a81640 1 --2- 172.21.15.44:0/3170738478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0fc0cec60 0x7ff0fc15ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.997+0000 7ff101a81640 1 -- 172.21.15.44:0/3170738478 >> 172.21.15.44:0/3170738478 conn(0x7ff0fc09d3c0 msgr2=0x7ff0fc0b2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:27.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.997+0000 7ff101a81640 1 -- 172.21.15.44:0/3170738478 shutdown_connections 2024-04-02T13:26:27.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:26.997+0000 7ff101a81640 1 -- 172.21.15.44:0/3170738478 wait complete. 2024-04-02T13:26:27.011 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:26:27.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.112+0000 7f7937e8f640 1 Processor -- start 2024-04-02T13:26:27.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.112+0000 7f7937e8f640 1 -- start start 2024-04-02T13:26:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f7937e8f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f793006cff0 0x7f793006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:27.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f7937e8f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79300b4180 0x7f79300b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f7937e8f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793006d910 con 0x7f793015e5d0 2024-04-02T13:26:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f7937e8f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f79300d1bf0 con 0x7f793006cff0 2024-04-02T13:26:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f7937e8f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f79300d1630 con 0x7f79300b4180 2024-04-02T13:26:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f7935c2b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f793006cff0 0x7f793006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f793542a640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f793015e5d0 0x7f79300d0d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:36608/0 (socket says 172.21.15.44:36608) 2024-04-02T13:26:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f793542a640 1 -- 172.21.15.44:0/2531446885 learned_addr learned my addr 172.21.15.44:0/2531446885 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f7935c2b640 1 -- 172.21.15.44:0/2531446885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79300b4180 msgr2=0x7f79300b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f793642c640 1 --2- 172.21.15.44:0/2531446885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79300b4180 0x7f79300b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f7934c29640 1 -- 172.21.15.44:0/2531446885 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3563295862 0 0) 0x7f793006d910 con 0x7f793015e5d0 2024-04-02T13:26:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f7935c2b640 1 --2- 172.21.15.44:0/2531446885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79300b4180 0x7f79300b1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f7935c2b640 1 -- 172.21.15.44:0/2531446885 >> v1:172.21.15.44:6789/0 conn(0x7f793015e5d0 legacy=0x7f79300d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.113+0000 7f7935c2b640 1 -- 172.21.15.44:0/2531446885 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79300d1910 con 0x7f793006cff0 2024-04-02T13:26:27.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.114+0000 7f7935c2b640 1 --2- 172.21.15.44:0/2531446885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f793006cff0 0x7f793006d3d0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f79280097a0 tx=0x7f792802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cfc4de87ccde4af server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:27.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.114+0000 7f7934c29640 1 -- 172.21.15.44:0/2531446885 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79280315a0 con 0x7f793006cff0 2024-04-02T13:26:27.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.114+0000 7f7934c29640 1 -- 172.21.15.44:0/2531446885 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7928037070 con 0x7f793006cff0 2024-04-02T13:26:27.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.114+0000 7f7934c29640 1 -- 172.21.15.44:0/2531446885 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928031960 con 0x7f793006cff0 2024-04-02T13:26:27.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.115+0000 7f7937e8f640 1 -- 172.21.15.44:0/2531446885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f793006cff0 msgr2=0x7f793006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.115+0000 7f7937e8f640 1 --2- 172.21.15.44:0/2531446885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f793006cff0 0x7f793006d3d0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f79280097a0 tx=0x7f792802b840 comp rx=0 tx=0).stop 2024-04-02T13:26:27.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.115+0000 7f7937e8f640 1 -- 172.21.15.44:0/2531446885 shutdown_connections 2024-04-02T13:26:27.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.115+0000 7f7937e8f640 1 --2- 172.21.15.44:0/2531446885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79300b4180 0x7f79300b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.115+0000 7f7937e8f640 1 --2- 172.21.15.44:0/2531446885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f793006cff0 0x7f793006d3d0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.115+0000 7f7937e8f640 1 -- 172.21.15.44:0/2531446885 >> 172.21.15.44:0/2531446885 conn(0x7f793009aa20 msgr2=0x7f793009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:27.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.115+0000 7f7937e8f640 1 -- 172.21.15.44:0/2531446885 shutdown_connections 2024-04-02T13:26:27.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.115+0000 7f7937e8f640 1 -- 172.21.15.44:0/2531446885 wait complete. 2024-04-02T13:26:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.116+0000 7f7937e8f640 1 Processor -- start 2024-04-02T13:26:27.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.116+0000 7f7937e8f640 1 -- start start 2024-04-02T13:26:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7937e8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f793006cff0 0x7f793014afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7937e8f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79300b4180 0x7f79300b7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7937e8f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f793015e5d0 0x7f79300c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7937e8f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f79300b2280 con 0x7f793006cff0 2024-04-02T13:26:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7937e8f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f79300b1820 con 0x7f793015e5d0 2024-04-02T13:26:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7937e8f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f79300b2400 con 0x7f79300b4180 2024-04-02T13:26:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7935c2b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f793006cff0 0x7f793014afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7935c2b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f793006cff0 0x7f793014afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45882/0 (socket says 172.21.15.44:45882) 2024-04-02T13:26:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7935c2b640 1 -- 172.21.15.44:0/3492680927 learned_addr learned my addr 172.21.15.44:0/3492680927 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f793542a640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79300b4180 0x7f79300b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f793642c640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f793015e5d0 0x7f79300c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7935c2b640 1 -- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79300b4180 msgr2=0x7f79300b7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7935c2b640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79300b4180 0x7f79300b7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7935c2b640 1 -- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f793015e5d0 msgr2=0x7f79300c6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7935c2b640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f793015e5d0 0x7f79300c6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7935c2b640 1 -- 172.21.15.44:0/3492680927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79300c7390 con 0x7f793006cff0 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.117+0000 7f7935c2b640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f793006cff0 0x7f793014afc0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7928000c00 tx=0x7f79280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.118+0000 7f792e7fc640 1 -- 172.21.15.44:0/3492680927 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928004230 con 0x7f793006cff0 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.118+0000 7f792e7fc640 1 -- 172.21.15.44:0/3492680927 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79280043d0 con 0x7f793006cff0 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.118+0000 7f792e7fc640 1 -- 172.21.15.44:0/3492680927 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928032880 con 0x7f793006cff0 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.118+0000 7f7937e8f640 1 -- 172.21.15.44:0/3492680927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79300c7670 con 0x7f793006cff0 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.118+0000 7f7937e8f640 1 -- 172.21.15.44:0/3492680927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79300c7c30 con 0x7f793006cff0 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.119+0000 7f7937e8f640 1 -- 172.21.15.44:0/3492680927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79300684f0 con 0x7f793006cff0 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.119+0000 7f792e7fc640 1 -- 172.21.15.44:0/3492680927 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7928032a20 con 0x7f793006cff0 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.119+0000 7f792e7fc640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f791c06c8f0 0x7f791c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.119+0000 7f792e7fc640 1 -- 172.21.15.44:0/3492680927 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f792802f030 con 0x7f793006cff0 2024-04-02T13:26:27.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.120+0000 7f793542a640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f791c06c8f0 0x7f791c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:27.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.120+0000 7f793542a640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f791c06c8f0 0x7f791c06edb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7930161a70 tx=0x7f7924009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:27.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.124+0000 7f792e7fc640 1 -- 172.21.15.44:0/3492680927 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7928041070 con 0x7f793006cff0 2024-04-02T13:26:27.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.306+0000 7f7937e8f640 1 -- 172.21.15.44:0/3492680927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f793015d070 con 0x7f793006cff0 2024-04-02T13:26:27.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.307+0000 7f792e7fc640 1 -- 172.21.15.44:0/3492680927 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) ==== 100+0+11 (secure 0 0 0) 0x7f792807c220 con 0x7f793006cff0 2024-04-02T13:26:27.311 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:26:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.308+0000 7f7937e8f640 1 -- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f791c06c8f0 msgr2=0x7f791c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.308+0000 7f7937e8f640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f791c06c8f0 0x7f791c06edb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7930161a70 tx=0x7f7924009040 comp rx=0 tx=0).stop 2024-04-02T13:26:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.308+0000 7f7937e8f640 1 -- 172.21.15.44:0/3492680927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f793006cff0 msgr2=0x7f793014afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.308+0000 7f7937e8f640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f793006cff0 0x7f793014afc0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7928000c00 tx=0x7f79280046c0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.308+0000 7f7937e8f640 1 -- 172.21.15.44:0/3492680927 shutdown_connections 2024-04-02T13:26:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.309+0000 7f7937e8f640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f791c06c8f0 0x7f791c06edb0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.309+0000 7f7937e8f640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f793015e5d0 0x7f79300c6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.309+0000 7f7937e8f640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79300b4180 0x7f79300b7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.309+0000 7f7937e8f640 1 --2- 172.21.15.44:0/3492680927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f793006cff0 0x7f793014afc0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.309+0000 7f7937e8f640 1 -- 172.21.15.44:0/3492680927 >> 172.21.15.44:0/3492680927 conn(0x7f793009aa20 msgr2=0x7f7930070570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:27.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.309+0000 7f7937e8f640 1 -- 172.21.15.44:0/3492680927 shutdown_connections 2024-04-02T13:26:27.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.309+0000 7f7937e8f640 1 -- 172.21.15.44:0/3492680927 wait complete. 2024-04-02T13:26:27.323 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:26:27.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.421+0000 7f9c99cd9640 1 Processor -- start 2024-04-02T13:26:27.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.421+0000 7f9c99cd9640 1 -- start start 2024-04-02T13:26:27.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.422+0000 7f9c99cd9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c940d0a30 0x7f9c9406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:27.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.422+0000 7f9c99cd9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c940d0e10 0x7f9c940cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:27.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.422+0000 7f9c99cd9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c9406c1d0 con 0x7f9c940cfe30 2024-04-02T13:26:27.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.422+0000 7f9c99cd9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9c9406cb00 con 0x7f9c940d0a30 2024-04-02T13:26:27.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.422+0000 7f9c99cd9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9c9406c4b0 con 0x7f9c940d0e10 2024-04-02T13:26:27.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.423+0000 7f9c92ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c940d0e10 0x7f9c940cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:27.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.423+0000 7f9c937fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c940d0a30 0x7f9c9406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:27.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.423+0000 7f9c93fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9c940cfe30 0x7f9c940d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:42100/0 (socket says 172.21.15.44:42100) 2024-04-02T13:26:27.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.423+0000 7f9c93fff640 1 -- 172.21.15.44:0/2426738540 learned_addr learned my addr 172.21.15.44:0/2426738540 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:27.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.423+0000 7f9c92ffd640 1 -- 172.21.15.44:0/2426738540 >> v2:172.21.15.69:3300/0 conn(0x7f9c940d0a30 msgr2=0x7f9c9406cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:27.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.423+0000 7f9c92ffd640 1 --2- 172.21.15.44:0/2426738540 >> v2:172.21.15.69:3300/0 conn(0x7f9c940d0a30 0x7f9c9406cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.423+0000 7f9c92ffd640 1 -- 172.21.15.44:0/2426738540 >> v1:172.21.15.69:6789/0 conn(0x7f9c940cfe30 legacy=0x7f9c940d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.423+0000 7f9c92ffd640 1 -- 172.21.15.44:0/2426738540 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c9406c790 con 0x7f9c940d0e10 2024-04-02T13:26:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.424+0000 7f9c92ffd640 1 --2- 172.21.15.44:0/2426738540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c940d0e10 0x7f9c940cf8f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9c840097a0 tx=0x7f9c8402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7a46ac6382d3a71 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.424+0000 7f9c927fc640 1 -- 172.21.15.44:0/2426738540 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c840315a0 con 0x7f9c940d0e10 2024-04-02T13:26:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.424+0000 7f9c927fc640 1 -- 172.21.15.44:0/2426738540 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c84037070 con 0x7f9c940d0e10 2024-04-02T13:26:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.424+0000 7f9c927fc640 1 -- 172.21.15.44:0/2426738540 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c84031960 con 0x7f9c940d0e10 2024-04-02T13:26:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.424+0000 7f9c99cd9640 1 -- 172.21.15.44:0/2426738540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c940d0e10 msgr2=0x7f9c940cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.424+0000 7f9c99cd9640 1 --2- 172.21.15.44:0/2426738540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c940d0e10 0x7f9c940cf8f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9c840097a0 tx=0x7f9c8402b840 comp rx=0 tx=0).stop 2024-04-02T13:26:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.424+0000 7f9c99cd9640 1 -- 172.21.15.44:0/2426738540 shutdown_connections 2024-04-02T13:26:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.424+0000 7f9c99cd9640 1 --2- 172.21.15.44:0/2426738540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c940d0e10 0x7f9c940cf8f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.424+0000 7f9c99cd9640 1 --2- 172.21.15.44:0/2426738540 >> v2:172.21.15.69:3300/0 conn(0x7f9c940d0a30 0x7f9c9406cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.424+0000 7f9c99cd9640 1 -- 172.21.15.44:0/2426738540 >> 172.21.15.44:0/2426738540 conn(0x7f9c9409d470 msgr2=0x7f9c940d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.425+0000 7f9c99cd9640 1 -- 172.21.15.44:0/2426738540 shutdown_connections 2024-04-02T13:26:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.425+0000 7f9c99cd9640 1 -- 172.21.15.44:0/2426738540 wait complete. 2024-04-02T13:26:27.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.426+0000 7f9c99cd9640 1 Processor -- start 2024-04-02T13:26:27.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.426+0000 7f9c99cd9640 1 -- start start 2024-04-02T13:26:27.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.426+0000 7f9c99cd9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c940cfe30 0x7f9c940ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:27.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.426+0000 7f9c99cd9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c940d0a30 0x7f9c940cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:27.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.426+0000 7f9c99cd9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c940d0e10 0x7f9c941535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:27.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.426+0000 7f9c99cd9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9415e870 con 0x7f9c940cfe30 2024-04-02T13:26:27.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.426+0000 7f9c99cd9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9415e6f0 con 0x7f9c940d0e10 2024-04-02T13:26:27.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.426+0000 7f9c99cd9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9c940d4da0 con 0x7f9c940d0a30 2024-04-02T13:26:27.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c937fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c940cfe30 0x7f9c940ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:27.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c937fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c940cfe30 0x7f9c940ca620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45888/0 (socket says 172.21.15.44:45888) 2024-04-02T13:26:27.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c937fe640 1 -- 172.21.15.44:0/762275000 learned_addr learned my addr 172.21.15.44:0/762275000 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:27.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c93fff640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c940d0e10 0x7f9c941535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:27.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c92ffd640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c940d0a30 0x7f9c940cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:27.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c937fe640 1 -- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c940d0a30 msgr2=0x7f9c940cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c937fe640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c940d0a30 0x7f9c940cd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c937fe640 1 -- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c940d0e10 msgr2=0x7f9c941535d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c937fe640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c940d0e10 0x7f9c941535d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c937fe640 1 -- 172.21.15.44:0/762275000 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c94153b10 con 0x7f9c940cfe30 2024-04-02T13:26:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c92ffd640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c940d0a30 0x7f9c940cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c937fe640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c940cfe30 0x7f9c940ca620 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00e870 tx=0x7f9c7c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c8bfff640 1 -- 172.21.15.44:0/762275000 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c017070 con 0x7f9c940cfe30 2024-04-02T13:26:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c8bfff640 1 -- 172.21.15.44:0/762275000 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c7c00c940 con 0x7f9c940cfe30 2024-04-02T13:26:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c99cd9640 1 -- 172.21.15.44:0/762275000 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c940cb390 con 0x7f9c940cfe30 2024-04-02T13:26:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c8bfff640 1 -- 172.21.15.44:0/762275000 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c00cd30 con 0x7f9c940cfe30 2024-04-02T13:26:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.427+0000 7f9c99cd9640 1 -- 172.21.15.44:0/762275000 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c940ca110 con 0x7f9c940cfe30 2024-04-02T13:26:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.428+0000 7f9c99cd9640 1 -- 172.21.15.44:0/762275000 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c940684f0 con 0x7f9c940cfe30 2024-04-02T13:26:27.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.429+0000 7f9c8bfff640 1 -- 172.21.15.44:0/762275000 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9c7c01a070 con 0x7f9c940cfe30 2024-04-02T13:26:27.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.429+0000 7f9c8bfff640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c7406c8f0 0x7f9c7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:27.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.429+0000 7f9c8bfff640 1 -- 172.21.15.44:0/762275000 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f9c7c08bc80 con 0x7f9c940cfe30 2024-04-02T13:26:27.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.430+0000 7f9c92ffd640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c7406c8f0 0x7f9c7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:27.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.430+0000 7f9c92ffd640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c7406c8f0 0x7f9c7406edb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9c84036000 tx=0x7f9c84008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:27.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.434+0000 7f9c8bfff640 1 -- 172.21.15.44:0/762275000 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9c94153b10 con 0x7f9c940cfe30 2024-04-02T13:26:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.630+0000 7f9c99cd9640 1 -- 172.21.15.44:0/762275000 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9c940d17c0 con 0x7f9c940cfe30 2024-04-02T13:26:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.630+0000 7f9c8bfff640 1 -- 172.21.15.44:0/762275000 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v80) ==== 80+0+120 (secure 0 0 0) 0x7f9c7c059e50 con 0x7f9c940cfe30 2024-04-02T13:26:27.633 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.69:6837/698185867 2024-04-03T13:25:10.921135+0000 2024-04-02T13:26:27.634 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.69:6835/698185867 2024-04-03T13:25:10.921135+0000 2024-04-02T13:26:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:listed 2 entries 2024-04-02T13:26:27.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.632+0000 7f9c99cd9640 1 -- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c7406c8f0 msgr2=0x7f9c7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.632+0000 7f9c99cd9640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c7406c8f0 0x7f9c7406edb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9c84036000 tx=0x7f9c84008030 comp rx=0 tx=0).stop 2024-04-02T13:26:27.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.632+0000 7f9c99cd9640 1 -- 172.21.15.44:0/762275000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c940cfe30 msgr2=0x7f9c940ca620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:27.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.632+0000 7f9c99cd9640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c940cfe30 0x7f9c940ca620 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00e870 tx=0x7f9c7c00ed30 comp rx=0 tx=0).stop 2024-04-02T13:26:27.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.632+0000 7f9c99cd9640 1 -- 172.21.15.44:0/762275000 shutdown_connections 2024-04-02T13:26:27.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.632+0000 7f9c99cd9640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c7406c8f0 0x7f9c7406edb0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.632+0000 7f9c99cd9640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c940d0e10 0x7f9c941535d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.632+0000 7f9c99cd9640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c940d0a30 0x7f9c940cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.632+0000 7f9c99cd9640 1 --2- 172.21.15.44:0/762275000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c940cfe30 0x7f9c940ca620 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:27.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.632+0000 7f9c99cd9640 1 -- 172.21.15.44:0/762275000 >> 172.21.15.44:0/762275000 conn(0x7f9c9409d470 msgr2=0x7f9c94161060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:27.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.632+0000 7f9c99cd9640 1 -- 172.21.15.44:0/762275000 shutdown_connections 2024-04-02T13:26:27.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:27.632+0000 7f9c99cd9640 1 -- 172.21.15.44:0/762275000 wait complete. 2024-04-02T13:26:27.646 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:26:27.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:27.646 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:27.671 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[67119]: fuse finished with error 0 and tester_r 0 2024-04-02T13:26:27.673 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:26:27.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:27.673 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:26:27.752 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:26:27.753 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:26:27.753 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:26:27.771 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:26:27.771 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:26:27.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:27.772 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-02T13:26:27.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:26:27.830 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-02T13:26:27.830 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:26:27.830 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:26:27.830 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:26:27.845 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:26:27.845 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:26:27.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:27.845 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-02T13:26:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:26:27.900 INFO:teuthology.orchestra.run.smithi044.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-02T13:26:27.900 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:26:27.900 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:26:27.900 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:26:27.918 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:26:27.919 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:26:27.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:27.919 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-02T13:26:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:26:27.976 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-02T13:26:27.977 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:26:28.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.074+0000 7f1749f1c640 1 Processor -- start 2024-04-02T13:26:28.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.074+0000 7f1749f1c640 1 -- start start 2024-04-02T13:26:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.075+0000 7f1749f1c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f174415c2b0 0x7f17440ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.075+0000 7f1749f1c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1744072790 con 0x7f17440cbd80 2024-04-02T13:26:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.075+0000 7f1749f1c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17440cafc0 con 0x7f17440cc9b0 2024-04-02T13:26:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.075+0000 7f1749f1c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1744070120 con 0x7f174415c2b0 2024-04-02T13:26:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.075+0000 7f1742ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f17440cbd80 0x7f1744071970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:36618/0 (socket says 172.21.15.44:36618) 2024-04-02T13:26:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.075+0000 7f1742ffd640 1 -- 172.21.15.44:0/1106538878 learned_addr learned my addr 172.21.15.44:0/1106538878 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.075+0000 7f17437fe640 1 --2- 172.21.15.44:0/1106538878 >> v2:172.21.15.69:3300/0 conn(0x7f174415c2b0 0x7f17440ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.075+0000 7f17427fc640 1 -- 172.21.15.44:0/1106538878 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1995742967 0 0) 0x7f1744072790 con 0x7f17440cbd80 2024-04-02T13:26:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.075+0000 7f17427fc640 1 -- 172.21.15.44:0/1106538878 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f172c003520 con 0x7f17440cbd80 2024-04-02T13:26:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.076+0000 7f17437fe640 1 -- 172.21.15.44:0/1106538878 >> v1:172.21.15.69:6789/0 conn(0x7f17440cc9b0 legacy=0x7f1744072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.076+0000 7f17437fe640 1 -- 172.21.15.44:0/1106538878 >> v1:172.21.15.44:6789/0 conn(0x7f17440cbd80 legacy=0x7f1744071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.076+0000 7f17427fc640 1 -- 172.21.15.44:0/1106538878 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 379677177 0 0) 0x7f172c003520 con 0x7f17440cbd80 2024-04-02T13:26:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.076+0000 7f17437fe640 1 -- 172.21.15.44:0/1106538878 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17440cb2a0 con 0x7f174415c2b0 2024-04-02T13:26:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.077+0000 7f17437fe640 1 --2- 172.21.15.44:0/1106538878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f174415c2b0 0x7f17440ccf00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f17340097a0 tx=0x7f173402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3ccdc51adf91fdc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.077+0000 7f17427fc640 1 -- 172.21.15.44:0/1106538878 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1734037070 con 0x7f174415c2b0 2024-04-02T13:26:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.077+0000 7f17427fc640 1 -- 172.21.15.44:0/1106538878 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1734030510 con 0x7f174415c2b0 2024-04-02T13:26:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.077+0000 7f17427fc640 1 -- 172.21.15.44:0/1106538878 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1738001670 con 0x7f174415c2b0 2024-04-02T13:26:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.077+0000 7f1749f1c640 1 -- 172.21.15.44:0/1106538878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f174415c2b0 msgr2=0x7f17440ccf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.077+0000 7f1749f1c640 1 --2- 172.21.15.44:0/1106538878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f174415c2b0 0x7f17440ccf00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f17340097a0 tx=0x7f173402b790 comp rx=0 tx=0).stop 2024-04-02T13:26:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.077+0000 7f1749f1c640 1 -- 172.21.15.44:0/1106538878 shutdown_connections 2024-04-02T13:26:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.077+0000 7f1749f1c640 1 --2- 172.21.15.44:0/1106538878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f174415c2b0 0x7f17440ccf00 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.077+0000 7f1749f1c640 1 -- 172.21.15.44:0/1106538878 >> 172.21.15.44:0/1106538878 conn(0x7f17440b20e0 msgr2=0x7f17440b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.077+0000 7f1749f1c640 1 -- 172.21.15.44:0/1106538878 shutdown_connections 2024-04-02T13:26:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.077+0000 7f1749f1c640 1 -- 172.21.15.44:0/1106538878 wait complete. 2024-04-02T13:26:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.078+0000 7f1749f1c640 1 Processor -- start 2024-04-02T13:26:28.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.079+0000 7f1749f1c640 1 -- start start 2024-04-02T13:26:28.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.079+0000 7f1749f1c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17440cbd80 0x7f17440adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.079+0000 7f1749f1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17440cc9b0 0x7f174414eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.079+0000 7f1749f1c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f174415c2b0 0x7f174406d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.079+0000 7f1749f1c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f174415c690 con 0x7f174415c2b0 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.079+0000 7f1749f1c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f17440cb860 con 0x7f17440cbd80 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.079+0000 7f1749f1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f174406fcd0 con 0x7f17440cc9b0 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.079+0000 7f1743fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f174415c2b0 0x7f174406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.079+0000 7f1743fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f174415c2b0 0x7f174406d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45898/0 (socket says 172.21.15.44:45898) 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.079+0000 7f1743fff640 1 -- 172.21.15.44:0/1930348639 learned_addr learned my addr 172.21.15.44:0/1930348639 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f17437fe640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17440cbd80 0x7f17440adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f1742ffd640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17440cc9b0 0x7f174414eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f1743fff640 1 -- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17440cc9b0 msgr2=0x7f174414eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f1743fff640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17440cc9b0 0x7f174414eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f1743fff640 1 -- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17440cbd80 msgr2=0x7f17440adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f1743fff640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17440cbd80 0x7f17440adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f1743fff640 1 -- 172.21.15.44:0/1930348639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17440ce960 con 0x7f174415c2b0 2024-04-02T13:26:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f1742ffd640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17440cc9b0 0x7f174414eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f1743fff640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f174415c2b0 0x7f174406d900 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f173c00c7a0 tx=0x7f173c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f172bfff640 1 -- 172.21.15.44:0/1930348639 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c003b70 con 0x7f174415c2b0 2024-04-02T13:26:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f172bfff640 1 -- 172.21.15.44:0/1930348639 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f173c003d30 con 0x7f174415c2b0 2024-04-02T13:26:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f1749f1c640 1 -- 172.21.15.44:0/1930348639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17440ae570 con 0x7f174415c2b0 2024-04-02T13:26:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f172bfff640 1 -- 172.21.15.44:0/1930348639 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c00aa90 con 0x7f174415c2b0 2024-04-02T13:26:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.080+0000 7f1749f1c640 1 -- 172.21.15.44:0/1930348639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17440aeb30 con 0x7f174415c2b0 2024-04-02T13:26:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.081+0000 7f1749f1c640 1 -- 172.21.15.44:0/1930348639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17440684f0 con 0x7f174415c2b0 2024-04-02T13:26:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.082+0000 7f172bfff640 1 -- 172.21.15.44:0/1930348639 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f173c00ac30 con 0x7f174415c2b0 2024-04-02T13:26:28.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.082+0000 7f172bfff640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f173006c8f0 0x7f173006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.082+0000 7f172bfff640 1 -- 172.21.15.44:0/1930348639 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f173c08bbf0 con 0x7f174415c2b0 2024-04-02T13:26:28.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.082+0000 7f17437fe640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f173006c8f0 0x7f173006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.083+0000 7f17437fe640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f173006c8f0 0x7f173006edb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1734034000 tx=0x7f173402f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:28.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.086+0000 7f172bfff640 1 -- 172.21.15.44:0/1930348639 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f17440ce960 con 0x7f174415c2b0 2024-04-02T13:26:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.308+0000 7f1749f1c640 1 -- 172.21.15.44:0/1930348639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f174415b4d0 con 0x7f174415c2b0 2024-04-02T13:26:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.308+0000 7f172bfff640 1 -- 172.21.15.44:0/1930348639 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5185 (secure 0 0 0) 0x7f173c059e40 con 0x7f174415c2b0 2024-04-02T13:26:28.312 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:28.312 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3662927745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3662927745},{"type":"v1","addr":"172.21.15.44:6835","nonce":3662927745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6394,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1053037366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1053037366},{"type":"v1","addr":"172.21.15.69:6836","nonce":1053037366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3893800047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3893800047},{"type":"v1","addr":"172.21.15.69:6837","nonce":3893800047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:24:53.925714+0000","modified":"2024-04-02T13:25:57.948654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.44:6833/2392159648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2392159648},{"type":"v1","addr":"172.21.15.44:6833","nonce":2392159648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":5}]} 2024-04-02T13:26:28.312 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 52 2024-04-02T13:26:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.310+0000 7f1749f1c640 1 -- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f173006c8f0 msgr2=0x7f173006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.310+0000 7f1749f1c640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f173006c8f0 0x7f173006edb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1734034000 tx=0x7f173402f000 comp rx=0 tx=0).stop 2024-04-02T13:26:28.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.310+0000 7f1749f1c640 1 -- 172.21.15.44:0/1930348639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f174415c2b0 msgr2=0x7f174406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.310+0000 7f1749f1c640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f174415c2b0 0x7f174406d900 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f173c00c7a0 tx=0x7f173c00cc60 comp rx=0 tx=0).stop 2024-04-02T13:26:28.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.310+0000 7f1749f1c640 1 -- 172.21.15.44:0/1930348639 shutdown_connections 2024-04-02T13:26:28.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.310+0000 7f1749f1c640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f173006c8f0 0x7f173006edb0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.310+0000 7f1749f1c640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f174415c2b0 0x7f174406d900 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.310+0000 7f1749f1c640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17440cc9b0 0x7f174414eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.310+0000 7f1749f1c640 1 --2- 172.21.15.44:0/1930348639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17440cbd80 0x7f17440adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.310+0000 7f1749f1c640 1 -- 172.21.15.44:0/1930348639 >> 172.21.15.44:0/1930348639 conn(0x7f17440b20e0 msgr2=0x7f174415e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:28.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.310+0000 7f1749f1c640 1 -- 172.21.15.44:0/1930348639 shutdown_connections 2024-04-02T13:26:28.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.310+0000 7f1749f1c640 1 -- 172.21.15.44:0/1930348639 wait complete. 2024-04-02T13:26:28.324 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:26:28.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.422+0000 7f291d79d640 1 Processor -- start 2024-04-02T13:26:28.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.423+0000 7f291d79d640 1 -- start start 2024-04-02T13:26:28.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.423+0000 7f291d79d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f291815e430 0x7f29180d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.423+0000 7f291d79d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f29180d0f70 0x7f29180d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.423+0000 7f291d79d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29180d1e00 con 0x7f29180b3f50 2024-04-02T13:26:28.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.423+0000 7f291d79d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f291815e810 con 0x7f291815e430 2024-04-02T13:26:28.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.423+0000 7f291d79d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f29180b37e0 con 0x7f29180d0f70 2024-04-02T13:26:28.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.423+0000 7f2917fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f291815e430 0x7f29180d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.423+0000 7f2917fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f291815e430 0x7f29180d2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45906/0 (socket says 172.21.15.44:45906) 2024-04-02T13:26:28.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.423+0000 7f2917fff640 1 -- 172.21.15.44:0/2347655379 learned_addr learned my addr 172.21.15.44:0/2347655379 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:28.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.424+0000 7f29177fe640 1 --2- 172.21.15.44:0/2347655379 >> v2:172.21.15.69:3300/0 conn(0x7f29180d0f70 0x7f29180d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.424+0000 7f2917fff640 1 -- 172.21.15.44:0/2347655379 >> v2:172.21.15.69:3300/0 conn(0x7f29180d0f70 msgr2=0x7f29180d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:28.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.424+0000 7f2917fff640 1 --2- 172.21.15.44:0/2347655379 >> v2:172.21.15.69:3300/0 conn(0x7f29180d0f70 0x7f29180d29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.424+0000 7f2917fff640 1 -- 172.21.15.44:0/2347655379 >> v1:172.21.15.69:6789/0 conn(0x7f29180b3f50 legacy=0x7f29180b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:28.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.424+0000 7f2917fff640 1 -- 172.21.15.44:0/2347655379 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29180d20e0 con 0x7f291815e430 2024-04-02T13:26:28.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.424+0000 7f29177fe640 1 --2- 172.21.15.44:0/2347655379 >> v2:172.21.15.69:3300/0 conn(0x7f29180d0f70 0x7f29180d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:28.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.425+0000 7f2917fff640 1 --2- 172.21.15.44:0/2347655379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f291815e430 0x7f29180d2460 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f291806bbd0 tx=0x7f290802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f96f37720394b92 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:28.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.425+0000 7f2916ffd640 1 -- 172.21.15.44:0/2347655379 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908035020 con 0x7f291815e430 2024-04-02T13:26:28.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.425+0000 7f2916ffd640 1 -- 172.21.15.44:0/2347655379 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2908031870 con 0x7f291815e430 2024-04-02T13:26:28.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.425+0000 7f2916ffd640 1 -- 172.21.15.44:0/2347655379 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908031c40 con 0x7f291815e430 2024-04-02T13:26:28.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.425+0000 7f291d79d640 1 -- 172.21.15.44:0/2347655379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f291815e430 msgr2=0x7f29180d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.425+0000 7f291d79d640 1 --2- 172.21.15.44:0/2347655379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f291815e430 0x7f29180d2460 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f291806bbd0 tx=0x7f290802b910 comp rx=0 tx=0).stop 2024-04-02T13:26:28.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.425+0000 7f291d79d640 1 -- 172.21.15.44:0/2347655379 shutdown_connections 2024-04-02T13:26:28.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.425+0000 7f291d79d640 1 --2- 172.21.15.44:0/2347655379 >> v2:172.21.15.69:3300/0 conn(0x7f29180d0f70 0x7f29180d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.425+0000 7f291d79d640 1 --2- 172.21.15.44:0/2347655379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f291815e430 0x7f29180d2460 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.425+0000 7f291d79d640 1 -- 172.21.15.44:0/2347655379 >> 172.21.15.44:0/2347655379 conn(0x7f29180d3e80 msgr2=0x7f29180d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:28.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.425+0000 7f291d79d640 1 -- 172.21.15.44:0/2347655379 shutdown_connections 2024-04-02T13:26:28.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.425+0000 7f291d79d640 1 -- 172.21.15.44:0/2347655379 wait complete. 2024-04-02T13:26:28.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.426+0000 7f291d79d640 1 Processor -- start 2024-04-02T13:26:28.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.427+0000 7f291d79d640 1 -- start start 2024-04-02T13:26:28.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.427+0000 7f291d79d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29180b3f50 0x7f29181536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.427+0000 7f291d79d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29180d0f70 0x7f29180cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.427+0000 7f291d79d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f291815e430 0x7f29180cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.427+0000 7f291d79d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f29180d4b20 con 0x7f291815e430 2024-04-02T13:26:28.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.427+0000 7f291d79d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f29180d49a0 con 0x7f29180b3f50 2024-04-02T13:26:28.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.427+0000 7f291d79d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f29180cfaa0 con 0x7f29180d0f70 2024-04-02T13:26:28.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.427+0000 7f291cf9c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f291815e430 0x7f29180cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.427+0000 7f291cf9c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f291815e430 0x7f29180cb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45914/0 (socket says 172.21.15.44:45914) 2024-04-02T13:26:28.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.427+0000 7f291cf9c640 1 -- 172.21.15.44:0/2211428325 learned_addr learned my addr 172.21.15.44:0/2211428325 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:28.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f2917fff640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29180b3f50 0x7f29181536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f29177fe640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29180d0f70 0x7f29180cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f291cf9c640 1 -- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29180d0f70 msgr2=0x7f29180cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f291cf9c640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29180d0f70 0x7f29180cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f291cf9c640 1 -- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29180b3f50 msgr2=0x7f29181536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f291cf9c640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29180b3f50 0x7f29181536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f291cf9c640 1 -- 172.21.15.44:0/2211428325 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29180cbea0 con 0x7f291815e430 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f2917fff640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29180b3f50 0x7f29181536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f291cf9c640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f291815e430 0x7f29180cb960 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f291000c7b0 tx=0x7f291000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f2914ff9640 1 -- 172.21.15.44:0/2211428325 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910004b60 con 0x7f291815e430 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f2914ff9640 1 -- 172.21.15.44:0/2211428325 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2910004d00 con 0x7f291815e430 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f291d79d640 1 -- 172.21.15.44:0/2211428325 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29180cc140 con 0x7f291815e430 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f2914ff9640 1 -- 172.21.15.44:0/2211428325 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291000aa90 con 0x7f291815e430 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.428+0000 7f291d79d640 1 -- 172.21.15.44:0/2211428325 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29180cc700 con 0x7f291815e430 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.429+0000 7f291d79d640 1 -- 172.21.15.44:0/2211428325 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29180cc900 con 0x7f291815e430 2024-04-02T13:26:28.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.430+0000 7f2914ff9640 1 -- 172.21.15.44:0/2211428325 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f291000ac30 con 0x7f291815e430 2024-04-02T13:26:28.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.430+0000 7f2914ff9640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f290406c920 0x7f290406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.430+0000 7f2914ff9640 1 -- 172.21.15.44:0/2211428325 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f291008c290 con 0x7f291815e430 2024-04-02T13:26:28.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.430+0000 7f2917fff640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f290406c920 0x7f290406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.431+0000 7f2917fff640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f290406c920 0x7f290406ede0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2908036000 tx=0x7f290803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:28.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.434+0000 7f2914ff9640 1 -- 172.21.15.44:0/2211428325 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f291005a4e0 con 0x7f291815e430 2024-04-02T13:26:28.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.620+0000 7f291d79d640 1 -- 172.21.15.44:0/2211428325 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f291815d020 con 0x7f291815e430 2024-04-02T13:26:28.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.621+0000 7f2914ff9640 1 -- 172.21.15.44:0/2211428325 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+17024 (secure 0 0 0) 0x7f2910059de0 con 0x7f291815e430 2024-04-02T13:26:28.625 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:28.625 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":80,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:25:10.921159+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:24:52.202720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-02T13:24:52.916529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6837/698185867":"2024-04-03T13:25:10.921135+0000","172.21.15.69:6835/698185867":"2024-04-03T13:25:10.921135+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:26:28.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.623+0000 7f291d79d640 1 -- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f290406c920 msgr2=0x7f290406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.623+0000 7f291d79d640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f290406c920 0x7f290406ede0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2908036000 tx=0x7f290803c040 comp rx=0 tx=0).stop 2024-04-02T13:26:28.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.623+0000 7f291d79d640 1 -- 172.21.15.44:0/2211428325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f291815e430 msgr2=0x7f29180cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.623+0000 7f291d79d640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f291815e430 0x7f29180cb960 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f291000c7b0 tx=0x7f291000cc70 comp rx=0 tx=0).stop 2024-04-02T13:26:28.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.624+0000 7f291d79d640 1 -- 172.21.15.44:0/2211428325 shutdown_connections 2024-04-02T13:26:28.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.624+0000 7f291d79d640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f290406c920 0x7f290406ede0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.624+0000 7f291d79d640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f291815e430 0x7f29180cb960 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.624+0000 7f291d79d640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29180d0f70 0x7f29180cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.624+0000 7f291d79d640 1 --2- 172.21.15.44:0/2211428325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29180b3f50 0x7f29181536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.624+0000 7f291d79d640 1 -- 172.21.15.44:0/2211428325 >> 172.21.15.44:0/2211428325 conn(0x7f29180d3e80 msgr2=0x7f291806d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:28.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.624+0000 7f291d79d640 1 -- 172.21.15.44:0/2211428325 shutdown_connections 2024-04-02T13:26:28.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.624+0000 7f291d79d640 1 -- 172.21.15.44:0/2211428325 wait complete. 2024-04-02T13:26:28.638 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:24:52.202720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:24:52.916529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:26:28.638 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:26:28.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.740+0000 7f0a98b3d640 1 Processor -- start 2024-04-02T13:26:28.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.740+0000 7f0a98b3d640 1 -- start start 2024-04-02T13:26:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a98b3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9406b4d0 0x7f0a9406b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a98b3d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0a940cd770 0x7f0a9406eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a98b3d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0a940721f0 0x7f0a94072660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a98b3d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0a9406bdf0 con 0x7f0a940721f0 2024-04-02T13:26:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a98b3d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0a9406f3f0 con 0x7f0a940cd770 2024-04-02T13:26:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a98b3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0a9406f6d0 con 0x7f0a9406b4d0 2024-04-02T13:26:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a92d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0a940721f0 0x7f0a94072660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a92d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0a940721f0 0x7f0a94072660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45924/0 (socket says 172.21.15.44:45924) 2024-04-02T13:26:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a92d9d640 1 -- 172.21.15.44:0/3267778394 learned_addr learned my addr 172.21.15.44:0/3267778394 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a9259c640 1 --2- 172.21.15.44:0/3267778394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9406b4d0 0x7f0a9406b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a91d9b640 1 --2- 172.21.15.44:0/3267778394 >> v2:172.21.15.69:3300/0 conn(0x7f0a940cd770 0x7f0a9406eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a92d9d640 1 -- 172.21.15.44:0/3267778394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9406b4d0 msgr2=0x7f0a9406b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a92d9d640 1 --2- 172.21.15.44:0/3267778394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9406b4d0 0x7f0a9406b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a92d9d640 1 -- 172.21.15.44:0/3267778394 >> v2:172.21.15.69:3300/0 conn(0x7f0a940cd770 msgr2=0x7f0a9406eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a92d9d640 1 --2- 172.21.15.44:0/3267778394 >> v2:172.21.15.69:3300/0 conn(0x7f0a940cd770 0x7f0a9406eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.741+0000 7f0a92d9d640 1 -- 172.21.15.44:0/3267778394 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9406f9b0 con 0x7f0a940721f0 2024-04-02T13:26:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.742+0000 7f0a92d9d640 1 --2- 172.21.15.44:0/3267778394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940721f0 0x7f0a94072660 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f0a940b3ad0 tx=0x7f0a8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fbab18e94d49712 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.742+0000 7f0a9159a640 1 -- 172.21.15.44:0/3267778394 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c02ca70 con 0x7f0a940721f0 2024-04-02T13:26:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.742+0000 7f0a9159a640 1 -- 172.21.15.44:0/3267778394 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a8c02cc10 con 0x7f0a940721f0 2024-04-02T13:26:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.742+0000 7f0a9159a640 1 -- 172.21.15.44:0/3267778394 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c0343f0 con 0x7f0a940721f0 2024-04-02T13:26:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.743+0000 7f0a98b3d640 1 -- 172.21.15.44:0/3267778394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940721f0 msgr2=0x7f0a94072660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.743+0000 7f0a98b3d640 1 --2- 172.21.15.44:0/3267778394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940721f0 0x7f0a94072660 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f0a940b3ad0 tx=0x7f0a8c02bed0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.743+0000 7f0a98b3d640 1 -- 172.21.15.44:0/3267778394 shutdown_connections 2024-04-02T13:26:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.743+0000 7f0a98b3d640 1 --2- 172.21.15.44:0/3267778394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940721f0 0x7f0a94072660 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.743+0000 7f0a98b3d640 1 --2- 172.21.15.44:0/3267778394 >> v2:172.21.15.69:3300/0 conn(0x7f0a940cd770 0x7f0a9406eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.743+0000 7f0a98b3d640 1 --2- 172.21.15.44:0/3267778394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9406b4d0 0x7f0a9406b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.743+0000 7f0a98b3d640 1 -- 172.21.15.44:0/3267778394 >> 172.21.15.44:0/3267778394 conn(0x7f0a940b2a10 msgr2=0x7f0a940b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.743+0000 7f0a98b3d640 1 -- 172.21.15.44:0/3267778394 shutdown_connections 2024-04-02T13:26:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.743+0000 7f0a98b3d640 1 -- 172.21.15.44:0/3267778394 wait complete. 2024-04-02T13:26:28.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.744+0000 7f0a98b3d640 1 Processor -- start 2024-04-02T13:26:28.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.744+0000 7f0a98b3d640 1 -- start start 2024-04-02T13:26:28.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a98b3d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9406b4d0 0x7f0a94147680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a98b3d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940721f0 0x7f0a940afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a98b3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a940cd770 0x7f0a94153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a98b3d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9406f0e0 con 0x7f0a940721f0 2024-04-02T13:26:28.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a98b3d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9406ff70 con 0x7f0a9406b4d0 2024-04-02T13:26:28.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a98b3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0a9415c470 con 0x7f0a940cd770 2024-04-02T13:26:28.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a91d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940721f0 0x7f0a940afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a91d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940721f0 0x7f0a940afae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45934/0 (socket says 172.21.15.44:45934) 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a91d9b640 1 -- 172.21.15.44:0/2658406402 learned_addr learned my addr 172.21.15.44:0/2658406402 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a9259c640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9406b4d0 0x7f0a94147680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a92d9d640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a940cd770 0x7f0a94153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a91d9b640 1 -- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a940cd770 msgr2=0x7f0a94153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a91d9b640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a940cd770 0x7f0a94153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a91d9b640 1 -- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9406b4d0 msgr2=0x7f0a94147680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a91d9b640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9406b4d0 0x7f0a94147680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.745+0000 7f0a91d9b640 1 -- 172.21.15.44:0/2658406402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9415e330 con 0x7f0a940721f0 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.746+0000 7f0a91d9b640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940721f0 0x7f0a940afae0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800d800 tx=0x7f0a8800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.746+0000 7f0a82ffd640 1 -- 172.21.15.44:0/2658406402 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8800b6c0 con 0x7f0a940721f0 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.746+0000 7f0a82ffd640 1 -- 172.21.15.44:0/2658406402 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a88019070 con 0x7f0a940721f0 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.746+0000 7f0a98b3d640 1 -- 172.21.15.44:0/2658406402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a9415e610 con 0x7f0a940721f0 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.746+0000 7f0a82ffd640 1 -- 172.21.15.44:0/2658406402 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8800ba80 con 0x7f0a940721f0 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.746+0000 7f0a98b3d640 1 -- 172.21.15.44:0/2658406402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a94147bc0 con 0x7f0a940721f0 2024-04-02T13:26:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.747+0000 7f0a98b3d640 1 -- 172.21.15.44:0/2658406402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a940684f0 con 0x7f0a940721f0 2024-04-02T13:26:28.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.747+0000 7f0a82ffd640 1 -- 172.21.15.44:0/2658406402 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0a8800bc20 con 0x7f0a940721f0 2024-04-02T13:26:28.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.748+0000 7f0a82ffd640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a7406c8f0 0x7f0a7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:28.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.748+0000 7f0a82ffd640 1 -- 172.21.15.44:0/2658406402 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f0a8808bfe0 con 0x7f0a940721f0 2024-04-02T13:26:28.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.748+0000 7f0a9259c640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a7406c8f0 0x7f0a7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:28.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.748+0000 7f0a9259c640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a7406c8f0 0x7f0a7406edb0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0a840079a0 tx=0x7f0a84008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:28.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.752+0000 7f0a82ffd640 1 -- 172.21.15.44:0/2658406402 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0a88059be0 con 0x7f0a940721f0 2024-04-02T13:26:28.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.936+0000 7f0a98b3d640 1 -- 172.21.15.44:0/2658406402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0a940cdde0 con 0x7f0a940721f0 2024-04-02T13:26:28.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.936+0000 7f0a82ffd640 1 -- 172.21.15.44:0/2658406402 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) ==== 93+0+10 (secure 0 0 0) 0x7f0a8805dee0 con 0x7f0a940721f0 2024-04-02T13:26:28.940 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:26:28.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.938+0000 7f0a98b3d640 1 -- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a7406c8f0 msgr2=0x7f0a7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.938+0000 7f0a98b3d640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a7406c8f0 0x7f0a7406edb0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0a840079a0 tx=0x7f0a84008040 comp rx=0 tx=0).stop 2024-04-02T13:26:28.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.938+0000 7f0a98b3d640 1 -- 172.21.15.44:0/2658406402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940721f0 msgr2=0x7f0a940afae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:28.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.938+0000 7f0a98b3d640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940721f0 0x7f0a940afae0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800d800 tx=0x7f0a8800dcc0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.938+0000 7f0a98b3d640 1 -- 172.21.15.44:0/2658406402 shutdown_connections 2024-04-02T13:26:28.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.938+0000 7f0a98b3d640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0a7406c8f0 0x7f0a7406edb0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.938+0000 7f0a98b3d640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a940cd770 0x7f0a94153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.938+0000 7f0a98b3d640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940721f0 0x7f0a940afae0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.938+0000 7f0a98b3d640 1 --2- 172.21.15.44:0/2658406402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9406b4d0 0x7f0a94147680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:28.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.938+0000 7f0a98b3d640 1 -- 172.21.15.44:0/2658406402 >> 172.21.15.44:0/2658406402 conn(0x7f0a940b2a10 msgr2=0x7f0a94157820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:28.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.938+0000 7f0a98b3d640 1 -- 172.21.15.44:0/2658406402 shutdown_connections 2024-04-02T13:26:28.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:28.939+0000 7f0a98b3d640 1 -- 172.21.15.44:0/2658406402 wait complete. 2024-04-02T13:26:28.952 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:26:29.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.054+0000 7f176d89a640 1 Processor -- start 2024-04-02T13:26:29.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.054+0000 7f176d89a640 1 -- start start 2024-04-02T13:26:29.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.054+0000 7f176d89a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f176806cff0 0x7f176806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.054+0000 7f176d89a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f176815e5d0 0x7f17680d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.054+0000 7f176d89a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f176806d910 con 0x7f17680b4180 2024-04-02T13:26:29.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.054+0000 7f176d89a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f17680d1bf0 con 0x7f176815e5d0 2024-04-02T13:26:29.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.054+0000 7f176d89a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f17680d1630 con 0x7f176806cff0 2024-04-02T13:26:29.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.055+0000 7f17667fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f176815e5d0 0x7f17680d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.055+0000 7f17667fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f176815e5d0 0x7f17680d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45942/0 (socket says 172.21.15.44:45942) 2024-04-02T13:26:29.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.055+0000 7f17667fc640 1 -- 172.21.15.44:0/3072213742 learned_addr learned my addr 172.21.15.44:0/3072213742 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:29.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.055+0000 7f1766ffd640 1 --2- 172.21.15.44:0/3072213742 >> v2:172.21.15.69:3300/0 conn(0x7f176806cff0 0x7f176806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.055+0000 7f17667fc640 1 -- 172.21.15.44:0/3072213742 >> v2:172.21.15.69:3300/0 conn(0x7f176806cff0 msgr2=0x7f176806d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:29.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.055+0000 7f17667fc640 1 --2- 172.21.15.44:0/3072213742 >> v2:172.21.15.69:3300/0 conn(0x7f176806cff0 0x7f176806d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.055+0000 7f17667fc640 1 -- 172.21.15.44:0/3072213742 >> v1:172.21.15.69:6789/0 conn(0x7f17680b4180 legacy=0x7f17680ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:29.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.055+0000 7f17667fc640 1 -- 172.21.15.44:0/3072213742 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17680d1910 con 0x7f176815e5d0 2024-04-02T13:26:29.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.056+0000 7f17667fc640 1 --2- 172.21.15.44:0/3072213742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f176815e5d0 0x7f17680d0d30 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f1768071d40 tx=0x7f175c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9bebf700d1d1637 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:29.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.056+0000 7f1765ffb640 1 -- 172.21.15.44:0/3072213742 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c02e070 con 0x7f176815e5d0 2024-04-02T13:26:29.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.056+0000 7f1765ffb640 1 -- 172.21.15.44:0/3072213742 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f175c037070 con 0x7f176815e5d0 2024-04-02T13:26:29.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.056+0000 7f1765ffb640 1 -- 172.21.15.44:0/3072213742 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c031a00 con 0x7f176815e5d0 2024-04-02T13:26:29.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.056+0000 7f176d89a640 1 -- 172.21.15.44:0/3072213742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f176815e5d0 msgr2=0x7f17680d0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.056+0000 7f176d89a640 1 --2- 172.21.15.44:0/3072213742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f176815e5d0 0x7f17680d0d30 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f1768071d40 tx=0x7f175c02b880 comp rx=0 tx=0).stop 2024-04-02T13:26:29.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.056+0000 7f176d89a640 1 -- 172.21.15.44:0/3072213742 shutdown_connections 2024-04-02T13:26:29.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.056+0000 7f176d89a640 1 --2- 172.21.15.44:0/3072213742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f176815e5d0 0x7f17680d0d30 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.056+0000 7f176d89a640 1 --2- 172.21.15.44:0/3072213742 >> v2:172.21.15.69:3300/0 conn(0x7f176806cff0 0x7f176806d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.056+0000 7f176d89a640 1 -- 172.21.15.44:0/3072213742 >> 172.21.15.44:0/3072213742 conn(0x7f176809aa20 msgr2=0x7f176809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:29.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.056+0000 7f176d89a640 1 -- 172.21.15.44:0/3072213742 shutdown_connections 2024-04-02T13:26:29.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.056+0000 7f176d89a640 1 -- 172.21.15.44:0/3072213742 wait complete. 2024-04-02T13:26:29.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.058+0000 7f176d89a640 1 Processor -- start 2024-04-02T13:26:29.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.058+0000 7f176d89a640 1 -- start start 2024-04-02T13:26:29.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.058+0000 7f176d89a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f176806cff0 0x7f176815b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.058+0000 7f176d89a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680b4180 0x7f17680c5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.058+0000 7f176d89a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f176815e5d0 0x7f17680c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.058+0000 7f176d89a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f17680d1270 con 0x7f17680b4180 2024-04-02T13:26:29.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.058+0000 7f176d89a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f17680b1820 con 0x7f176815e5d0 2024-04-02T13:26:29.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.058+0000 7f176d89a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f17680d13f0 con 0x7f176806cff0 2024-04-02T13:26:29.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f17667fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680b4180 0x7f17680c5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f17667fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680b4180 0x7f17680c5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45952/0 (socket says 172.21.15.44:45952) 2024-04-02T13:26:29.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f17667fc640 1 -- 172.21.15.44:0/2237331900 learned_addr learned my addr 172.21.15.44:0/2237331900 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:29.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f1766ffd640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f176806cff0 0x7f176815b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f17677fe640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f176815e5d0 0x7f17680c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f17667fc640 1 -- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f176806cff0 msgr2=0x7f176815b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f17667fc640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f176806cff0 0x7f176815b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f17667fc640 1 -- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f176815e5d0 msgr2=0x7f17680c6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f17667fc640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f176815e5d0 0x7f17680c6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f17667fc640 1 -- 172.21.15.44:0/2237331900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f176815bbe0 con 0x7f17680b4180 2024-04-02T13:26:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f17667fc640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680b4180 0x7f17680c5b10 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f175c02b8c0 tx=0x7f175c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f174ffff640 1 -- 172.21.15.44:0/2237331900 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c032880 con 0x7f17680b4180 2024-04-02T13:26:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f174ffff640 1 -- 172.21.15.44:0/2237331900 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f175c032a20 con 0x7f17680b4180 2024-04-02T13:26:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f174ffff640 1 -- 172.21.15.44:0/2237331900 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c032e10 con 0x7f17680b4180 2024-04-02T13:26:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f176d89a640 1 -- 172.21.15.44:0/2237331900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17681f55d0 con 0x7f17680b4180 2024-04-02T13:26:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.059+0000 7f176d89a640 1 -- 172.21.15.44:0/2237331900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17681f4af0 con 0x7f17680b4180 2024-04-02T13:26:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.060+0000 7f176d89a640 1 -- 172.21.15.44:0/2237331900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17680684f0 con 0x7f17680b4180 2024-04-02T13:26:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.061+0000 7f174ffff640 1 -- 172.21.15.44:0/2237331900 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f175c0453f0 con 0x7f17680b4180 2024-04-02T13:26:29.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.061+0000 7f174ffff640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f175406c8f0 0x7f175406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.061+0000 7f174ffff640 1 -- 172.21.15.44:0/2237331900 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f175c0adca0 con 0x7f17680b4180 2024-04-02T13:26:29.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.061+0000 7f1766ffd640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f175406c8f0 0x7f175406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.062+0000 7f1766ffd640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f175406c8f0 0x7f175406edb0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f175800aac0 tx=0x7f1758008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:29.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.065+0000 7f174ffff640 1 -- 172.21.15.44:0/2237331900 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f176815bbe0 con 0x7f17680b4180 2024-04-02T13:26:29.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.251+0000 7f176d89a640 1 -- 172.21.15.44:0/2237331900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f176806cdd0 con 0x7f17680b4180 2024-04-02T13:26:29.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.252+0000 7f174ffff640 1 -- 172.21.15.44:0/2237331900 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) ==== 104+0+11 (secure 0 0 0) 0x7f175c040070 con 0x7f17680b4180 2024-04-02T13:26:29.256 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:26:29.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.254+0000 7f176d89a640 1 -- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f175406c8f0 msgr2=0x7f175406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.254+0000 7f176d89a640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f175406c8f0 0x7f175406edb0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f175800aac0 tx=0x7f1758008040 comp rx=0 tx=0).stop 2024-04-02T13:26:29.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.254+0000 7f176d89a640 1 -- 172.21.15.44:0/2237331900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680b4180 msgr2=0x7f17680c5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.254+0000 7f176d89a640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680b4180 0x7f17680c5b10 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f175c02b8c0 tx=0x7f175c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.254+0000 7f176d89a640 1 -- 172.21.15.44:0/2237331900 shutdown_connections 2024-04-02T13:26:29.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.254+0000 7f176d89a640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f175406c8f0 0x7f175406edb0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.254+0000 7f176d89a640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f176815e5d0 0x7f17680c6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.254+0000 7f176d89a640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17680b4180 0x7f17680c5b10 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.254+0000 7f176d89a640 1 --2- 172.21.15.44:0/2237331900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f176806cff0 0x7f176815b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.254+0000 7f176d89a640 1 -- 172.21.15.44:0/2237331900 >> 172.21.15.44:0/2237331900 conn(0x7f176809aa20 msgr2=0x7f17680d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:29.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.254+0000 7f176d89a640 1 -- 172.21.15.44:0/2237331900 shutdown_connections 2024-04-02T13:26:29.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.254+0000 7f176d89a640 1 -- 172.21.15.44:0/2237331900 wait complete. 2024-04-02T13:26:29.268 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:26:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.369+0000 7f9977eeb640 1 Processor -- start 2024-04-02T13:26:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.369+0000 7f9977eeb640 1 -- start start 2024-04-02T13:26:29.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9977eeb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f997006cff0 0x7f997006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9977eeb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f997015e5d0 0x7f99700d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9977eeb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f99700b4180 0x7f99700ce860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9977eeb640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f99700d1bf0 con 0x7f997015e5d0 2024-04-02T13:26:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9977eeb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f997006d910 con 0x7f99700b4180 2024-04-02T13:26:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9977eeb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f99700d1630 con 0x7f997006cff0 2024-04-02T13:26:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9975486640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f997015e5d0 0x7f99700d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9975486640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f997015e5d0 0x7f99700d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45960/0 (socket says 172.21.15.44:45960) 2024-04-02T13:26:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9975486640 1 -- 172.21.15.44:0/2137266089 learned_addr learned my addr 172.21.15.44:0/2137266089 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9975c87640 1 --2- 172.21.15.44:0/2137266089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f997006cff0 0x7f997006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9976488640 1 --2- 172.21.15.44:0/2137266089 >> v2:172.21.15.69:3300/0 conn(0x7f99700b4180 0x7f99700ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9975486640 1 -- 172.21.15.44:0/2137266089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f997006cff0 msgr2=0x7f997006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9975486640 1 --2- 172.21.15.44:0/2137266089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f997006cff0 0x7f997006d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9975486640 1 -- 172.21.15.44:0/2137266089 >> v2:172.21.15.69:3300/0 conn(0x7f99700b4180 msgr2=0x7f99700ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:29.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9975486640 1 --2- 172.21.15.44:0/2137266089 >> v2:172.21.15.69:3300/0 conn(0x7f99700b4180 0x7f99700ce860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.370+0000 7f9975486640 1 -- 172.21.15.44:0/2137266089 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99700d1910 con 0x7f997015e5d0 2024-04-02T13:26:29.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.371+0000 7f9975486640 1 --2- 172.21.15.44:0/2137266089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f997015e5d0 0x7f99700d0d30 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f99700cfd60 tx=0x7f996002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e32e660e69bf452 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:29.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.371+0000 7f9974c85640 1 -- 172.21.15.44:0/2137266089 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996002e070 con 0x7f997015e5d0 2024-04-02T13:26:29.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.371+0000 7f9974c85640 1 -- 172.21.15.44:0/2137266089 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9960037070 con 0x7f997015e5d0 2024-04-02T13:26:29.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.371+0000 7f9974c85640 1 -- 172.21.15.44:0/2137266089 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9960031a00 con 0x7f997015e5d0 2024-04-02T13:26:29.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.371+0000 7f9977eeb640 1 -- 172.21.15.44:0/2137266089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f997015e5d0 msgr2=0x7f99700d0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.371+0000 7f9977eeb640 1 --2- 172.21.15.44:0/2137266089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f997015e5d0 0x7f99700d0d30 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f99700cfd60 tx=0x7f996002b880 comp rx=0 tx=0).stop 2024-04-02T13:26:29.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.372+0000 7f9977eeb640 1 -- 172.21.15.44:0/2137266089 shutdown_connections 2024-04-02T13:26:29.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.372+0000 7f9977eeb640 1 --2- 172.21.15.44:0/2137266089 >> v2:172.21.15.69:3300/0 conn(0x7f99700b4180 0x7f99700ce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.372+0000 7f9977eeb640 1 --2- 172.21.15.44:0/2137266089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f997015e5d0 0x7f99700d0d30 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.372+0000 7f9977eeb640 1 --2- 172.21.15.44:0/2137266089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f997006cff0 0x7f997006d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.372+0000 7f9977eeb640 1 -- 172.21.15.44:0/2137266089 >> 172.21.15.44:0/2137266089 conn(0x7f997009aa20 msgr2=0x7f997009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:29.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.372+0000 7f9977eeb640 1 -- 172.21.15.44:0/2137266089 shutdown_connections 2024-04-02T13:26:29.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.372+0000 7f9977eeb640 1 -- 172.21.15.44:0/2137266089 wait complete. 2024-04-02T13:26:29.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.373+0000 7f9977eeb640 1 Processor -- start 2024-04-02T13:26:29.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.373+0000 7f9977eeb640 1 -- start start 2024-04-02T13:26:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.373+0000 7f9977eeb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f997006cff0 0x7f99700c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.373+0000 7f9977eeb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99700b4180 0x7f99701f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.373+0000 7f9977eeb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f997015e5d0 0x7f99700b5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9977eeb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f99700d1270 con 0x7f997006cff0 2024-04-02T13:26:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9977eeb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f99700b1820 con 0x7f99700b4180 2024-04-02T13:26:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9977eeb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f99700d13f0 con 0x7f997015e5d0 2024-04-02T13:26:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9975c87640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f997006cff0 0x7f99700c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9975c87640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f997006cff0 0x7f99700c8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45972/0 (socket says 172.21.15.44:45972) 2024-04-02T13:26:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9975c87640 1 -- 172.21.15.44:0/37230817 learned_addr learned my addr 172.21.15.44:0/37230817 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9976488640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f997015e5d0 0x7f99700b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9975c87640 1 -- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f997015e5d0 msgr2=0x7f99700b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9975486640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99700b4180 0x7f99701f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9975c87640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f997015e5d0 0x7f99700b5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9975c87640 1 -- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99700b4180 msgr2=0x7f99701f36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9975c87640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99700b4180 0x7f99701f36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9975c87640 1 -- 172.21.15.44:0/37230817 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99700b59c0 con 0x7f997006cff0 2024-04-02T13:26:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9975c87640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f997006cff0 0x7f99700c8610 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f996c00e8a0 tx=0x7f996c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f99667fc640 1 -- 172.21.15.44:0/37230817 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c017070 con 0x7f997006cff0 2024-04-02T13:26:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f99667fc640 1 -- 172.21.15.44:0/37230817 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f996c00ee90 con 0x7f997006cff0 2024-04-02T13:26:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9977eeb640 1 -- 172.21.15.44:0/37230817 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99700b5c40 con 0x7f997006cff0 2024-04-02T13:26:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f99667fc640 1 -- 172.21.15.44:0/37230817 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c00cb80 con 0x7f997006cff0 2024-04-02T13:26:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.374+0000 7f9977eeb640 1 -- 172.21.15.44:0/37230817 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f997015fb30 con 0x7f997006cff0 2024-04-02T13:26:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.375+0000 7f9977eeb640 1 -- 172.21.15.44:0/37230817 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99700684f0 con 0x7f997006cff0 2024-04-02T13:26:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.376+0000 7f99667fc640 1 -- 172.21.15.44:0/37230817 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f996c01a070 con 0x7f997006cff0 2024-04-02T13:26:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.376+0000 7f99667fc640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f995406c8f0 0x7f995406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.376+0000 7f99667fc640 1 -- 172.21.15.44:0/37230817 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f996c08bac0 con 0x7f997006cff0 2024-04-02T13:26:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.376+0000 7f9975486640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f995406c8f0 0x7f995406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.377+0000 7f9975486640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f995406c8f0 0x7f995406edb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9960036000 tx=0x7f9960008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.381+0000 7f99667fc640 1 -- 172.21.15.44:0/37230817 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f99700b59c0 con 0x7f997006cff0 2024-04-02T13:26:29.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.566+0000 7f9977eeb640 1 -- 172.21.15.44:0/37230817 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f99700b6160 con 0x7f997006cff0 2024-04-02T13:26:29.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.567+0000 7f99667fc640 1 -- 172.21.15.44:0/37230817 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) ==== 100+0+11 (secure 0 0 0) 0x7f996c059da0 con 0x7f997006cff0 2024-04-02T13:26:29.570 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:26:29.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.568+0000 7f9977eeb640 1 -- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f995406c8f0 msgr2=0x7f995406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.568+0000 7f9977eeb640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f995406c8f0 0x7f995406edb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9960036000 tx=0x7f9960008030 comp rx=0 tx=0).stop 2024-04-02T13:26:29.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.568+0000 7f9977eeb640 1 -- 172.21.15.44:0/37230817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f997006cff0 msgr2=0x7f99700c8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.568+0000 7f9977eeb640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f997006cff0 0x7f99700c8610 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f996c00e8a0 tx=0x7f996c00ed60 comp rx=0 tx=0).stop 2024-04-02T13:26:29.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.568+0000 7f9977eeb640 1 -- 172.21.15.44:0/37230817 shutdown_connections 2024-04-02T13:26:29.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.568+0000 7f9977eeb640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f995406c8f0 0x7f995406edb0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.568+0000 7f9977eeb640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f997015e5d0 0x7f99700b5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.568+0000 7f9977eeb640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99700b4180 0x7f99701f36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.568+0000 7f9977eeb640 1 --2- 172.21.15.44:0/37230817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f997006cff0 0x7f99700c8610 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.568+0000 7f9977eeb640 1 -- 172.21.15.44:0/37230817 >> 172.21.15.44:0/37230817 conn(0x7f997009aa20 msgr2=0x7f997006f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:29.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.568+0000 7f9977eeb640 1 -- 172.21.15.44:0/37230817 shutdown_connections 2024-04-02T13:26:29.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.569+0000 7f9977eeb640 1 -- 172.21.15.44:0/37230817 wait complete. 2024-04-02T13:26:29.583 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:26:29.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.682+0000 7ff5e739d640 1 Processor -- start 2024-04-02T13:26:29.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.682+0000 7ff5e739d640 1 -- start start 2024-04-02T13:26:29.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.682+0000 7ff5e739d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff5e015c1c0 0x7ff5e00affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.682+0000 7ff5e739d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5e00b3d70 0x7ff5e00b0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.682+0000 7ff5e739d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff5e00b0a40 0x7ff5e0164e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.682+0000 7ff5e739d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff5e00b1640 con 0x7ff5e015c1c0 2024-04-02T13:26:29.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.683+0000 7ff5e739d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff5e00aef20 con 0x7ff5e00b0a40 2024-04-02T13:26:29.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.683+0000 7ff5e739d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e00af200 con 0x7ff5e00b3d70 2024-04-02T13:26:29.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.683+0000 7ff5e5139640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff5e015c1c0 0x7ff5e00affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.683+0000 7ff5e5139640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff5e015c1c0 0x7ff5e00affc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45986/0 (socket says 172.21.15.44:45986) 2024-04-02T13:26:29.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.683+0000 7ff5e5139640 1 -- 172.21.15.44:0/4116318787 learned_addr learned my addr 172.21.15.44:0/4116318787 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:29.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.683+0000 7ff5e593a640 1 --2- 172.21.15.44:0/4116318787 >> v2:172.21.15.69:3300/0 conn(0x7ff5e00b0a40 0x7ff5e0164e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.683+0000 7ff5e4938640 1 --2- 172.21.15.44:0/4116318787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5e00b3d70 0x7ff5e00b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.683+0000 7ff5e5139640 1 -- 172.21.15.44:0/4116318787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5e00b3d70 msgr2=0x7ff5e00b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:29.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.683+0000 7ff5e5139640 1 --2- 172.21.15.44:0/4116318787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5e00b3d70 0x7ff5e00b0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.683+0000 7ff5e5139640 1 -- 172.21.15.44:0/4116318787 >> v2:172.21.15.69:3300/0 conn(0x7ff5e00b0a40 msgr2=0x7ff5e0164e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:29.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.683+0000 7ff5e5139640 1 --2- 172.21.15.44:0/4116318787 >> v2:172.21.15.69:3300/0 conn(0x7ff5e00b0a40 0x7ff5e0164e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.683+0000 7ff5e5139640 1 -- 172.21.15.44:0/4116318787 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e00af4e0 con 0x7ff5e015c1c0 2024-04-02T13:26:29.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.684+0000 7ff5e5139640 1 --2- 172.21.15.44:0/4116318787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e015c1c0 0x7ff5e00affc0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff5e00d0310 tx=0x7ff5dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86526cf301c82e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:29.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.684+0000 7ff5d7fff640 1 -- 172.21.15.44:0/4116318787 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc035020 con 0x7ff5e015c1c0 2024-04-02T13:26:29.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.684+0000 7ff5d7fff640 1 -- 172.21.15.44:0/4116318787 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5dc031870 con 0x7ff5e015c1c0 2024-04-02T13:26:29.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.684+0000 7ff5d7fff640 1 -- 172.21.15.44:0/4116318787 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc031c40 con 0x7ff5e015c1c0 2024-04-02T13:26:29.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.684+0000 7ff5e739d640 1 -- 172.21.15.44:0/4116318787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e015c1c0 msgr2=0x7ff5e00affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.684+0000 7ff5e739d640 1 --2- 172.21.15.44:0/4116318787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e015c1c0 0x7ff5e00affc0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff5e00d0310 tx=0x7ff5dc02b910 comp rx=0 tx=0).stop 2024-04-02T13:26:29.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.684+0000 7ff5e739d640 1 -- 172.21.15.44:0/4116318787 shutdown_connections 2024-04-02T13:26:29.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.684+0000 7ff5e739d640 1 --2- 172.21.15.44:0/4116318787 >> v2:172.21.15.69:3300/0 conn(0x7ff5e00b0a40 0x7ff5e0164e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.684+0000 7ff5e739d640 1 --2- 172.21.15.44:0/4116318787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5e00b3d70 0x7ff5e00b0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.684+0000 7ff5e739d640 1 --2- 172.21.15.44:0/4116318787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e015c1c0 0x7ff5e00affc0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.684+0000 7ff5e739d640 1 -- 172.21.15.44:0/4116318787 >> 172.21.15.44:0/4116318787 conn(0x7ff5e009aa20 msgr2=0x7ff5e009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:29.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.684+0000 7ff5e739d640 1 -- 172.21.15.44:0/4116318787 shutdown_connections 2024-04-02T13:26:29.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.685+0000 7ff5e739d640 1 -- 172.21.15.44:0/4116318787 wait complete. 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.686+0000 7ff5e739d640 1 Processor -- start 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.686+0000 7ff5e739d640 1 -- start start 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.686+0000 7ff5e739d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e00b0a40 0x7ff5e00cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.686+0000 7ff5e739d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5e00b3d70 0x7ff5e00ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.686+0000 7ff5e739d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5e015c1c0 0x7ff5e006fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.686+0000 7ff5e739d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e00cc990 con 0x7ff5e00b0a40 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.686+0000 7ff5e739d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e00afaa0 con 0x7ff5e00b3d70 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.686+0000 7ff5e739d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e00aebf0 con 0x7ff5e015c1c0 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.686+0000 7ff5e5139640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e00b0a40 0x7ff5e00cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.686+0000 7ff5e5139640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e00b0a40 0x7ff5e00cb7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45994/0 (socket says 172.21.15.44:45994) 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.686+0000 7ff5e5139640 1 -- 172.21.15.44:0/2713905603 learned_addr learned my addr 172.21.15.44:0/2713905603 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5e4938640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5e00b3d70 0x7ff5e00ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5e593a640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5e015c1c0 0x7ff5e006fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5e5139640 1 -- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5e015c1c0 msgr2=0x7ff5e006fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5e5139640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5e015c1c0 0x7ff5e006fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5e5139640 1 -- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5e00b3d70 msgr2=0x7ff5e00ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5e5139640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5e00b3d70 0x7ff5e00ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5e5139640 1 -- 172.21.15.44:0/2713905603 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e015e810 con 0x7ff5e00b0a40 2024-04-02T13:26:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5e4938640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5e00b3d70 0x7ff5e00ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5e5139640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e00b0a40 0x7ff5e00cb7f0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc000c00 tx=0x7ff5dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5d5ffb640 1 -- 172.21.15.44:0/2713905603 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc035040 con 0x7ff5e00b0a40 2024-04-02T13:26:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5d5ffb640 1 -- 172.21.15.44:0/2713905603 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5dc030d00 con 0x7ff5e00b0a40 2024-04-02T13:26:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5d5ffb640 1 -- 172.21.15.44:0/2713905603 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc0456a0 con 0x7ff5e00b0a40 2024-04-02T13:26:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5e739d640 1 -- 172.21.15.44:0/2713905603 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e015eaf0 con 0x7ff5e00b0a40 2024-04-02T13:26:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.687+0000 7ff5e739d640 1 -- 172.21.15.44:0/2713905603 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e00cbd30 con 0x7ff5e00b0a40 2024-04-02T13:26:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.688+0000 7ff5e739d640 1 -- 172.21.15.44:0/2713905603 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e015f010 con 0x7ff5e00b0a40 2024-04-02T13:26:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.689+0000 7ff5d5ffb640 1 -- 172.21.15.44:0/2713905603 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff5dc038020 con 0x7ff5e00b0a40 2024-04-02T13:26:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.689+0000 7ff5d5ffb640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff5c406c8f0 0x7ff5c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.689+0000 7ff5d5ffb640 1 -- 172.21.15.44:0/2713905603 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7ff5dc0adfd0 con 0x7ff5e00b0a40 2024-04-02T13:26:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.689+0000 7ff5e4938640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff5c406c8f0 0x7ff5c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:29.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.690+0000 7ff5e4938640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff5c406c8f0 0x7ff5c406edb0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff5e01f5730 tx=0x7ff5d0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:29.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.694+0000 7ff5d5ffb640 1 -- 172.21.15.44:0/2713905603 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff5dc07c220 con 0x7ff5e00b0a40 2024-04-02T13:26:29.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.918+0000 7ff5e739d640 1 -- 172.21.15.44:0/2713905603 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5e00684f0 con 0x7ff5e00b0a40 2024-04-02T13:26:29.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.919+0000 7ff5d5ffb640 1 -- 172.21.15.44:0/2713905603 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5185 (secure 0 0 0) 0x7ff5dc07bb20 con 0x7ff5e00b0a40 2024-04-02T13:26:29.923 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:29.923 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3662927745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3662927745},{"type":"v1","addr":"172.21.15.44:6835","nonce":3662927745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6394,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1053037366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1053037366},{"type":"v1","addr":"172.21.15.69:6836","nonce":1053037366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3893800047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3893800047},{"type":"v1","addr":"172.21.15.69:6837","nonce":3893800047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:24:53.925714+0000","modified":"2024-04-02T13:25:57.948654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.44:6833/2392159648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2392159648},{"type":"v1","addr":"172.21.15.44:6833","nonce":2392159648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":5}]} 2024-04-02T13:26:29.923 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 52 2024-04-02T13:26:29.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.921+0000 7ff5e739d640 1 -- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff5c406c8f0 msgr2=0x7ff5c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.921+0000 7ff5e739d640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff5c406c8f0 0x7ff5c406edb0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff5e01f5730 tx=0x7ff5d0008040 comp rx=0 tx=0).stop 2024-04-02T13:26:29.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.921+0000 7ff5e739d640 1 -- 172.21.15.44:0/2713905603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e00b0a40 msgr2=0x7ff5e00cb7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:29.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.921+0000 7ff5e739d640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e00b0a40 0x7ff5e00cb7f0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc000c00 tx=0x7ff5dc0046c0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.921+0000 7ff5e739d640 1 -- 172.21.15.44:0/2713905603 shutdown_connections 2024-04-02T13:26:29.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.921+0000 7ff5e739d640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff5c406c8f0 0x7ff5c406edb0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.921+0000 7ff5e739d640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5e015c1c0 0x7ff5e006fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.921+0000 7ff5e739d640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5e00b3d70 0x7ff5e00ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.921+0000 7ff5e739d640 1 --2- 172.21.15.44:0/2713905603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e00b0a40 0x7ff5e00cb7f0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:29.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.921+0000 7ff5e739d640 1 -- 172.21.15.44:0/2713905603 >> 172.21.15.44:0/2713905603 conn(0x7ff5e009aa20 msgr2=0x7ff5e00b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:29.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.921+0000 7ff5e739d640 1 -- 172.21.15.44:0/2713905603 shutdown_connections 2024-04-02T13:26:29.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:29.921+0000 7ff5e739d640 1 -- 172.21.15.44:0/2713905603 wait complete. 2024-04-02T13:26:29.935 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:26:30.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.034+0000 7f4d2a5d6640 1 Processor -- start 2024-04-02T13:26:30.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.034+0000 7f4d2a5d6640 1 -- start start 2024-04-02T13:26:30.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.035+0000 7f4d2a5d6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4d240cece0 0x7f4d240cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.035+0000 7f4d2a5d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d240cf600 con 0x7f4d2406ece0 2024-04-02T13:26:30.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.035+0000 7f4d2a5d6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d2406dc40 con 0x7f4d240d09f0 2024-04-02T13:26:30.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.035+0000 7f4d2a5d6640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4d2406d9e0 con 0x7f4d240cece0 2024-04-02T13:26:30.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.035+0000 7f4d23fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4d240cece0 0x7f4d240cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.035+0000 7f4d28b73640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4d2406ece0 0x7f4d240acd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:36628/0 (socket says 172.21.15.44:36628) 2024-04-02T13:26:30.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.035+0000 7f4d28b73640 1 -- 172.21.15.44:0/2009345020 learned_addr learned my addr 172.21.15.44:0/2009345020 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:30.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.035+0000 7f4d23fff640 1 -- 172.21.15.44:0/2009345020 >> v1:172.21.15.69:6789/0 conn(0x7f4d240d09f0 legacy=0x7f4d2406d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.035+0000 7f4d23fff640 1 -- 172.21.15.44:0/2009345020 >> v1:172.21.15.44:6789/0 conn(0x7f4d2406ece0 legacy=0x7f4d240acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.035+0000 7f4d23fff640 1 -- 172.21.15.44:0/2009345020 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2406df20 con 0x7f4d240cece0 2024-04-02T13:26:30.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.035+0000 7f4d22ffd640 1 -- 172.21.15.44:0/2009345020 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2894588762 0 0) 0x7f4d240cf600 con 0x7f4d2406ece0 2024-04-02T13:26:30.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.036+0000 7f4d23fff640 1 --2- 172.21.15.44:0/2009345020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d240cece0 0x7f4d240cf0c0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f4d140097a0 tx=0x7f4d1402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b10da38628a19989 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:30.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.036+0000 7f4d22ffd640 1 -- 172.21.15.44:0/2009345020 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d14035020 con 0x7f4d240cece0 2024-04-02T13:26:30.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.036+0000 7f4d22ffd640 1 -- 172.21.15.44:0/2009345020 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d14031830 con 0x7f4d240cece0 2024-04-02T13:26:30.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.036+0000 7f4d22ffd640 1 -- 172.21.15.44:0/2009345020 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d14031c00 con 0x7f4d240cece0 2024-04-02T13:26:30.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.037+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2009345020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d240cece0 msgr2=0x7f4d240cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.037+0000 7f4d2a5d6640 1 --2- 172.21.15.44:0/2009345020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d240cece0 0x7f4d240cf0c0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f4d140097a0 tx=0x7f4d1402b8d0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.037+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2009345020 shutdown_connections 2024-04-02T13:26:30.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.037+0000 7f4d2a5d6640 1 --2- 172.21.15.44:0/2009345020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d240cece0 0x7f4d240cf0c0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.037+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2009345020 >> 172.21.15.44:0/2009345020 conn(0x7f4d240cfcd0 msgr2=0x7f4d240d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:30.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.037+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2009345020 shutdown_connections 2024-04-02T13:26:30.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.037+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2009345020 wait complete. 2024-04-02T13:26:30.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.038+0000 7f4d2a5d6640 1 Processor -- start 2024-04-02T13:26:30.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.038+0000 7f4d2a5d6640 1 -- start start 2024-04-02T13:26:30.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d2a5d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d2406ece0 0x7f4d241f4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d2a5d6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d240cece0 0x7f4d2415db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d2a5d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d240d09f0 0x7f4d2409c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d2a5d6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2406e7c0 con 0x7f4d240d09f0 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d2a5d6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2406e4e0 con 0x7f4d240cece0 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d2a5d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4d2415b220 con 0x7f4d2406ece0 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d28b73640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d240d09f0 0x7f4d2409c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d28b73640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d240d09f0 0x7f4d2409c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46010/0 (socket says 172.21.15.44:46010) 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d28b73640 1 -- 172.21.15.44:0/2203578611 learned_addr learned my addr 172.21.15.44:0/2203578611 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d237fe640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d240cece0 0x7f4d2415db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d23fff640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d2406ece0 0x7f4d241f4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d28b73640 1 -- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d2406ece0 msgr2=0x7f4d241f4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d28b73640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d2406ece0 0x7f4d241f4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d28b73640 1 -- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d240cece0 msgr2=0x7f4d2415db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d28b73640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d240cece0 0x7f4d2415db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d28b73640 1 -- 172.21.15.44:0/2203578611 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2415e060 con 0x7f4d240d09f0 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.039+0000 7f4d237fe640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d240cece0 0x7f4d2415db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:30.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.040+0000 7f4d28b73640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d240d09f0 0x7f4d2409c470 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00c7a0 tx=0x7f4d1c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:30.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.040+0000 7f4d20ff9640 1 -- 172.21.15.44:0/2203578611 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c018070 con 0x7f4d240d09f0 2024-04-02T13:26:30.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.040+0000 7f4d20ff9640 1 -- 172.21.15.44:0/2203578611 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d1c00a640 con 0x7f4d240d09f0 2024-04-02T13:26:30.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.040+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2203578611 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d2409b0f0 con 0x7f4d240d09f0 2024-04-02T13:26:30.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.040+0000 7f4d20ff9640 1 -- 172.21.15.44:0/2203578611 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c00aa70 con 0x7f4d240d09f0 2024-04-02T13:26:30.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.040+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2203578611 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d2409b6b0 con 0x7f4d240d09f0 2024-04-02T13:26:30.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.041+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2203578611 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d24074f20 con 0x7f4d240d09f0 2024-04-02T13:26:30.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.041+0000 7f4d20ff9640 1 -- 172.21.15.44:0/2203578611 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4d1c01b070 con 0x7f4d240d09f0 2024-04-02T13:26:30.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.042+0000 7f4d20ff9640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4d1006c8f0 0x7f4d1006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.042+0000 7f4d20ff9640 1 -- 172.21.15.44:0/2203578611 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f4d1c08b360 con 0x7f4d240d09f0 2024-04-02T13:26:30.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.042+0000 7f4d23fff640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4d1006c8f0 0x7f4d1006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.042+0000 7f4d23fff640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4d1006c8f0 0x7f4d1006edb0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4d14036000 tx=0x7f4d14008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:30.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.046+0000 7f4d20ff9640 1 -- 172.21.15.44:0/2203578611 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4d1c0595b0 con 0x7f4d240d09f0 2024-04-02T13:26:30.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.231+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2203578611 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d24073450 con 0x7f4d240d09f0 2024-04-02T13:26:30.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.233+0000 7f4d20ff9640 1 -- 172.21.15.44:0/2203578611 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+17024 (secure 0 0 0) 0x7f4d1c058eb0 con 0x7f4d240d09f0 2024-04-02T13:26:30.236 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:30.236 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":80,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:25:10.921159+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:24:52.202720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-02T13:24:52.916529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6837/698185867":"2024-04-03T13:25:10.921135+0000","172.21.15.69:6835/698185867":"2024-04-03T13:25:10.921135+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:26:30.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.234+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4d1006c8f0 msgr2=0x7f4d1006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.234+0000 7f4d2a5d6640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4d1006c8f0 0x7f4d1006edb0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4d14036000 tx=0x7f4d14008030 comp rx=0 tx=0).stop 2024-04-02T13:26:30.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.235+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2203578611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d240d09f0 msgr2=0x7f4d2409c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.235+0000 7f4d2a5d6640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d240d09f0 0x7f4d2409c470 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00c7a0 tx=0x7f4d1c00cc60 comp rx=0 tx=0).stop 2024-04-02T13:26:30.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.235+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2203578611 shutdown_connections 2024-04-02T13:26:30.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.235+0000 7f4d2a5d6640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4d1006c8f0 0x7f4d1006edb0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.235+0000 7f4d2a5d6640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d240d09f0 0x7f4d2409c470 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.235+0000 7f4d2a5d6640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d240cece0 0x7f4d2415db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.235+0000 7f4d2a5d6640 1 --2- 172.21.15.44:0/2203578611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d2406ece0 0x7f4d241f4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.235+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2203578611 >> 172.21.15.44:0/2203578611 conn(0x7f4d240cfcd0 msgr2=0x7f4d240ada50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:30.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.235+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2203578611 shutdown_connections 2024-04-02T13:26:30.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.235+0000 7f4d2a5d6640 1 -- 172.21.15.44:0/2203578611 wait complete. 2024-04-02T13:26:30.250 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T13:26:30.250 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:26:30.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.348+0000 7fe9969a8640 1 Processor -- start 2024-04-02T13:26:30.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.348+0000 7fe9969a8640 1 -- start start 2024-04-02T13:26:30.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.349+0000 7fe9969a8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe9900d0a30 0x7fe99006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.349+0000 7fe9969a8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9900d0000 con 0x7fe9900d0e10 2024-04-02T13:26:30.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.349+0000 7fe9969a8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe99006c1d0 con 0x7fe9900d2040 2024-04-02T13:26:30.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.349+0000 7fe9969a8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe99006cb00 con 0x7fe9900d0a30 2024-04-02T13:26:30.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.349+0000 7fe98ffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe9900d0a30 0x7fe99006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.349+0000 7fe98f7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe9900d0e10 0x7fe9900cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:36642/0 (socket says 172.21.15.44:36642) 2024-04-02T13:26:30.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.349+0000 7fe98f7fe640 1 -- 172.21.15.44:0/1960991136 learned_addr learned my addr 172.21.15.44:0/1960991136 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:30.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.349+0000 7fe98ffff640 1 -- 172.21.15.44:0/1960991136 >> v1:172.21.15.69:6789/0 conn(0x7fe9900d2040 legacy=0x7fe9900d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:30.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.349+0000 7fe98ffff640 1 -- 172.21.15.44:0/1960991136 >> v1:172.21.15.44:6789/0 conn(0x7fe9900d0e10 legacy=0x7fe9900cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.349+0000 7fe98ffff640 1 -- 172.21.15.44:0/1960991136 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe99006c4b0 con 0x7fe9900d0a30 2024-04-02T13:26:30.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.350+0000 7fe98ffff640 1 --2- 172.21.15.44:0/1960991136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9900d0a30 0x7fe99006cf90 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe9840097a0 tx=0x7fe98402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6374e36f102d3959 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:30.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.350+0000 7fe98effd640 1 -- 172.21.15.44:0/1960991136 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9840315a0 con 0x7fe9900d0a30 2024-04-02T13:26:30.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.350+0000 7fe98effd640 1 -- 172.21.15.44:0/1960991136 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe984037070 con 0x7fe9900d0a30 2024-04-02T13:26:30.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.350+0000 7fe98effd640 1 -- 172.21.15.44:0/1960991136 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe984031960 con 0x7fe9900d0a30 2024-04-02T13:26:30.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.351+0000 7fe9969a8640 1 -- 172.21.15.44:0/1960991136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9900d0a30 msgr2=0x7fe99006cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.351+0000 7fe9969a8640 1 --2- 172.21.15.44:0/1960991136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9900d0a30 0x7fe99006cf90 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe9840097a0 tx=0x7fe98402b840 comp rx=0 tx=0).stop 2024-04-02T13:26:30.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.351+0000 7fe9969a8640 1 -- 172.21.15.44:0/1960991136 shutdown_connections 2024-04-02T13:26:30.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.351+0000 7fe9969a8640 1 --2- 172.21.15.44:0/1960991136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9900d0a30 0x7fe99006cf90 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.351+0000 7fe9969a8640 1 -- 172.21.15.44:0/1960991136 >> 172.21.15.44:0/1960991136 conn(0x7fe99009d470 msgr2=0x7fe9900d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:30.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.351+0000 7fe9969a8640 1 -- 172.21.15.44:0/1960991136 shutdown_connections 2024-04-02T13:26:30.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.351+0000 7fe9969a8640 1 -- 172.21.15.44:0/1960991136 wait complete. 2024-04-02T13:26:30.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.352+0000 7fe9969a8640 1 Processor -- start 2024-04-02T13:26:30.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.352+0000 7fe9969a8640 1 -- start start 2024-04-02T13:26:30.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe9969a8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9900d0a30 0x7fe9900b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe9969a8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9900d0e10 0x7fe9900b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe9969a8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9900d2040 0x7fe9900b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe9969a8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe9900d2c00 con 0x7fe9900d0e10 2024-04-02T13:26:30.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe9969a8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe9900d1270 con 0x7fe9900d2040 2024-04-02T13:26:30.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe9969a8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe99015e6f0 con 0x7fe9900d0a30 2024-04-02T13:26:30.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe98f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9900d0e10 0x7fe9900b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe98f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9900d0e10 0x7fe9900b5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46032/0 (socket says 172.21.15.44:46032) 2024-04-02T13:26:30.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe98f7fe640 1 -- 172.21.15.44:0/2789667025 learned_addr learned my addr 172.21.15.44:0/2789667025 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:30.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe994f45640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9900d2040 0x7fe9900b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe98ffff640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9900d0a30 0x7fe9900b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe98f7fe640 1 -- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9900d0a30 msgr2=0x7fe9900b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe98f7fe640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9900d0a30 0x7fe9900b50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe98f7fe640 1 -- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9900d2040 msgr2=0x7fe9900b5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe98f7fe640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9900d2040 0x7fe9900b5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe98f7fe640 1 -- 172.21.15.44:0/2789667025 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9900b6380 con 0x7fe9900d0e10 2024-04-02T13:26:30.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe98f7fe640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9900d0e10 0x7fe9900b5620 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fe98000c440 tx=0x7fe98000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:30.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.353+0000 7fe98cff9640 1 -- 172.21.15.44:0/2789667025 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe980017070 con 0x7fe9900d0e10 2024-04-02T13:26:30.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.354+0000 7fe98cff9640 1 -- 172.21.15.44:0/2789667025 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe980005ce0 con 0x7fe9900d0e10 2024-04-02T13:26:30.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.354+0000 7fe9969a8640 1 -- 172.21.15.44:0/2789667025 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9900b7770 con 0x7fe9900d0e10 2024-04-02T13:26:30.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.354+0000 7fe98cff9640 1 -- 172.21.15.44:0/2789667025 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9800072f0 con 0x7fe9900d0e10 2024-04-02T13:26:30.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.354+0000 7fe9969a8640 1 -- 172.21.15.44:0/2789667025 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9900b7d30 con 0x7fe9900d0e10 2024-04-02T13:26:30.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.355+0000 7fe9969a8640 1 -- 172.21.15.44:0/2789667025 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9900684f0 con 0x7fe9900d0e10 2024-04-02T13:26:30.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.355+0000 7fe98cff9640 1 -- 172.21.15.44:0/2789667025 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe98001a070 con 0x7fe9900d0e10 2024-04-02T13:26:30.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.355+0000 7fe98cff9640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe97c06c8f0 0x7fe97c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.355+0000 7fe98cff9640 1 -- 172.21.15.44:0/2789667025 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fe98008afd0 con 0x7fe9900d0e10 2024-04-02T13:26:30.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.356+0000 7fe98ffff640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe97c06c8f0 0x7fe97c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.356+0000 7fe98ffff640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe97c06c8f0 0x7fe97c06edb0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe984036000 tx=0x7fe984008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:30.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.360+0000 7fe98cff9640 1 -- 172.21.15.44:0/2789667025 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe9800592b0 con 0x7fe9900d0e10 2024-04-02T13:26:30.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.589+0000 7fe9969a8640 1 -- 172.21.15.44:0/2789667025 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9900d17c0 con 0x7fe9900d0e10 2024-04-02T13:26:30.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.590+0000 7fe98cff9640 1 -- 172.21.15.44:0/2789667025 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5185 (secure 0 0 0) 0x7fe980058bb0 con 0x7fe9900d0e10 2024-04-02T13:26:30.593 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:30.593 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3662927745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3662927745},{"type":"v1","addr":"172.21.15.44:6835","nonce":3662927745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6394,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1053037366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1053037366},{"type":"v1","addr":"172.21.15.69:6836","nonce":1053037366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3893800047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3893800047},{"type":"v1","addr":"172.21.15.69:6837","nonce":3893800047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:24:53.925714+0000","modified":"2024-04-02T13:25:57.948654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.44:6833/2392159648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2392159648},{"type":"v1","addr":"172.21.15.44:6833","nonce":2392159648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":5}]} 2024-04-02T13:26:30.593 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 52 2024-04-02T13:26:30.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.592+0000 7fe9969a8640 1 -- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe97c06c8f0 msgr2=0x7fe97c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.592+0000 7fe9969a8640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe97c06c8f0 0x7fe97c06edb0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe984036000 tx=0x7fe984008030 comp rx=0 tx=0).stop 2024-04-02T13:26:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.592+0000 7fe9969a8640 1 -- 172.21.15.44:0/2789667025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9900d0e10 msgr2=0x7fe9900b5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.592+0000 7fe9969a8640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9900d0e10 0x7fe9900b5620 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fe98000c440 tx=0x7fe98000c900 comp rx=0 tx=0).stop 2024-04-02T13:26:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.592+0000 7fe9969a8640 1 -- 172.21.15.44:0/2789667025 shutdown_connections 2024-04-02T13:26:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.592+0000 7fe9969a8640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe97c06c8f0 0x7fe97c06edb0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.592+0000 7fe9969a8640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9900d2040 0x7fe9900b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.592+0000 7fe9969a8640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9900d0e10 0x7fe9900b5620 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.592+0000 7fe9969a8640 1 --2- 172.21.15.44:0/2789667025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9900d0a30 0x7fe9900b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.592+0000 7fe9969a8640 1 -- 172.21.15.44:0/2789667025 >> 172.21.15.44:0/2789667025 conn(0x7fe99009d470 msgr2=0x7fe9900b2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.592+0000 7fe9969a8640 1 -- 172.21.15.44:0/2789667025 shutdown_connections 2024-04-02T13:26:30.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.592+0000 7fe9969a8640 1 -- 172.21.15.44:0/2789667025 wait complete. 2024-04-02T13:26:30.606 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:26:30.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.705+0000 7feff88d9640 1 Processor -- start 2024-04-02T13:26:30.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.705+0000 7feff88d9640 1 -- start start 2024-04-02T13:26:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.706+0000 7feff88d9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feff40cece0 0x7feff40cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.706+0000 7feff88d9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7feff406ece0 0x7feff40acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.706+0000 7feff88d9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feff40cf600 con 0x7feff40d09f0 2024-04-02T13:26:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.706+0000 7feff88d9640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7feff406d9e0 con 0x7feff406ece0 2024-04-02T13:26:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.706+0000 7feff88d9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7feff406dc40 con 0x7feff40cece0 2024-04-02T13:26:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.706+0000 7feff2d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7feff406ece0 0x7feff40acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.706+0000 7feff2d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7feff406ece0 0x7feff40acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46034/0 (socket says 172.21.15.44:46034) 2024-04-02T13:26:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.706+0000 7feff2d9d640 1 -- 172.21.15.44:0/3315193701 learned_addr learned my addr 172.21.15.44:0/3315193701 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.706+0000 7feff259c640 1 --2- 172.21.15.44:0/3315193701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feff40cece0 0x7feff40cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.707+0000 7feff2d9d640 1 -- 172.21.15.44:0/3315193701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feff40cece0 msgr2=0x7feff40cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:30.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.707+0000 7feff2d9d640 1 --2- 172.21.15.44:0/3315193701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feff40cece0 0x7feff40cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.707+0000 7feff2d9d640 1 -- 172.21.15.44:0/3315193701 >> v1:172.21.15.69:6789/0 conn(0x7feff40d09f0 legacy=0x7feff406d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.707+0000 7feff2d9d640 1 -- 172.21.15.44:0/3315193701 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff406df20 con 0x7feff406ece0 2024-04-02T13:26:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.707+0000 7feff259c640 1 --2- 172.21.15.44:0/3315193701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feff40cece0 0x7feff40cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.707+0000 7feff2d9d640 1 --2- 172.21.15.44:0/3315193701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff406ece0 0x7feff40acd50 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fefe400d180 tx=0x7fefe402e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac233d974ab6d068 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.707+0000 7feff159a640 1 -- 172.21.15.44:0/3315193701 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe400b0d0 con 0x7feff406ece0 2024-04-02T13:26:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.708+0000 7feff159a640 1 -- 172.21.15.44:0/3315193701 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefe400b270 con 0x7feff406ece0 2024-04-02T13:26:30.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.708+0000 7feff159a640 1 -- 172.21.15.44:0/3315193701 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe4009600 con 0x7feff406ece0 2024-04-02T13:26:30.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.708+0000 7feff88d9640 1 -- 172.21.15.44:0/3315193701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff406ece0 msgr2=0x7feff40acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.708+0000 7feff88d9640 1 --2- 172.21.15.44:0/3315193701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff406ece0 0x7feff40acd50 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fefe400d180 tx=0x7fefe402e160 comp rx=0 tx=0).stop 2024-04-02T13:26:30.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.708+0000 7feff88d9640 1 -- 172.21.15.44:0/3315193701 shutdown_connections 2024-04-02T13:26:30.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.708+0000 7feff88d9640 1 --2- 172.21.15.44:0/3315193701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff406ece0 0x7feff40acd50 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.708+0000 7feff88d9640 1 --2- 172.21.15.44:0/3315193701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feff40cece0 0x7feff40cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.708+0000 7feff88d9640 1 -- 172.21.15.44:0/3315193701 >> 172.21.15.44:0/3315193701 conn(0x7feff40cfcd0 msgr2=0x7feff40d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:30.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.708+0000 7feff88d9640 1 -- 172.21.15.44:0/3315193701 shutdown_connections 2024-04-02T13:26:30.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.708+0000 7feff88d9640 1 -- 172.21.15.44:0/3315193701 wait complete. 2024-04-02T13:26:30.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.709+0000 7feff88d9640 1 Processor -- start 2024-04-02T13:26:30.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.709+0000 7feff88d9640 1 -- start start 2024-04-02T13:26:30.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.710+0000 7feff88d9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feff406ece0 0x7feff41443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.710+0000 7feff88d9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff40cece0 0x7feff415bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.710+0000 7feff88d9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feff40d09f0 0x7feff409cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.710+0000 7feff88d9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feff406e7c0 con 0x7feff40cece0 2024-04-02T13:26:30.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.710+0000 7feff88d9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7feff406e4e0 con 0x7feff406ece0 2024-04-02T13:26:30.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.710+0000 7feff88d9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7feff415b220 con 0x7feff40d09f0 2024-04-02T13:26:30.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.710+0000 7feff1d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff40cece0 0x7feff415bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.710+0000 7feff1d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff40cece0 0x7feff415bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46036/0 (socket says 172.21.15.44:46036) 2024-04-02T13:26:30.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.710+0000 7feff1d9b640 1 -- 172.21.15.44:0/630957322 learned_addr learned my addr 172.21.15.44:0/630957322 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:30.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.710+0000 7feff259c640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feff406ece0 0x7feff41443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.710+0000 7feff2d9d640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feff40d09f0 0x7feff409cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.711+0000 7feff1d9b640 1 -- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feff40d09f0 msgr2=0x7feff409cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.711+0000 7feff1d9b640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feff40d09f0 0x7feff409cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.711+0000 7feff1d9b640 1 -- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feff406ece0 msgr2=0x7feff41443e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.711+0000 7feff1d9b640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feff406ece0 0x7feff41443e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.711+0000 7feff1d9b640 1 -- 172.21.15.44:0/630957322 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff4144d50 con 0x7feff40cece0 2024-04-02T13:26:30.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.711+0000 7feff259c640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feff406ece0 0x7feff41443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:30.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.711+0000 7feff1d9b640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff40cece0 0x7feff415bb60 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fefe800c440 tx=0x7fefe800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:30.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.711+0000 7fefe2ffd640 1 -- 172.21.15.44:0/630957322 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe80052d0 con 0x7feff40cece0 2024-04-02T13:26:30.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.711+0000 7fefe2ffd640 1 -- 172.21.15.44:0/630957322 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefe8005470 con 0x7feff40cece0 2024-04-02T13:26:30.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.711+0000 7feff88d9640 1 -- 172.21.15.44:0/630957322 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feff4145030 con 0x7feff40cece0 2024-04-02T13:26:30.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.711+0000 7fefe2ffd640 1 -- 172.21.15.44:0/630957322 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe8007260 con 0x7feff40cece0 2024-04-02T13:26:30.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.711+0000 7feff88d9640 1 -- 172.21.15.44:0/630957322 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feff41455f0 con 0x7feff40cece0 2024-04-02T13:26:30.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.712+0000 7feff88d9640 1 -- 172.21.15.44:0/630957322 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feff40684f0 con 0x7feff40cece0 2024-04-02T13:26:30.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.712+0000 7fefe2ffd640 1 -- 172.21.15.44:0/630957322 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fefe80074c0 con 0x7feff40cece0 2024-04-02T13:26:30.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.713+0000 7fefe2ffd640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fefd406c8f0 0x7fefd406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:30.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.713+0000 7fefe2ffd640 1 -- 172.21.15.44:0/630957322 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fefe808b1b0 con 0x7feff40cece0 2024-04-02T13:26:30.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.713+0000 7feff259c640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fefd406c8f0 0x7fefd406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:30.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.714+0000 7feff259c640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fefd406c8f0 0x7fefd406edb0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fefec006fd0 tx=0x7fefec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:30.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.718+0000 7fefe2ffd640 1 -- 172.21.15.44:0/630957322 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fefe8059400 con 0x7feff40cece0 2024-04-02T13:26:30.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.939+0000 7feff88d9640 1 -- 172.21.15.44:0/630957322 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feff40ad9b0 con 0x7feff40cece0 2024-04-02T13:26:30.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.939+0000 7fefe2ffd640 1 -- 172.21.15.44:0/630957322 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5185 (secure 0 0 0) 0x7fefe8058d00 con 0x7feff40cece0 2024-04-02T13:26:30.943 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:30.943 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3662927745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3662927745},{"type":"v1","addr":"172.21.15.44:6835","nonce":3662927745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6394,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1053037366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1053037366},{"type":"v1","addr":"172.21.15.69:6836","nonce":1053037366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3893800047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3893800047},{"type":"v1","addr":"172.21.15.69:6837","nonce":3893800047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:24:53.925714+0000","modified":"2024-04-02T13:25:57.948654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.44:6833/2392159648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2392159648},{"type":"v1","addr":"172.21.15.44:6833","nonce":2392159648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":5}]} 2024-04-02T13:26:30.943 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 52 2024-04-02T13:26:30.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.941+0000 7feff88d9640 1 -- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fefd406c8f0 msgr2=0x7fefd406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.941+0000 7feff88d9640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fefd406c8f0 0x7fefd406edb0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fefec006fd0 tx=0x7fefec008040 comp rx=0 tx=0).stop 2024-04-02T13:26:30.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.941+0000 7feff88d9640 1 -- 172.21.15.44:0/630957322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff40cece0 msgr2=0x7feff415bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:30.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.941+0000 7feff88d9640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff40cece0 0x7feff415bb60 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fefe800c440 tx=0x7fefe800c900 comp rx=0 tx=0).stop 2024-04-02T13:26:30.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.941+0000 7feff88d9640 1 -- 172.21.15.44:0/630957322 shutdown_connections 2024-04-02T13:26:30.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.941+0000 7feff88d9640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fefd406c8f0 0x7fefd406edb0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.941+0000 7feff88d9640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feff40d09f0 0x7feff409cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.941+0000 7feff88d9640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff40cece0 0x7feff415bb60 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.941+0000 7feff88d9640 1 --2- 172.21.15.44:0/630957322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feff406ece0 0x7feff41443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:30.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.941+0000 7feff88d9640 1 -- 172.21.15.44:0/630957322 >> 172.21.15.44:0/630957322 conn(0x7feff40cfcd0 msgr2=0x7feff4071190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:30.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.941+0000 7feff88d9640 1 -- 172.21.15.44:0/630957322 shutdown_connections 2024-04-02T13:26:30.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:30.941+0000 7feff88d9640 1 -- 172.21.15.44:0/630957322 wait complete. 2024-04-02T13:26:30.955 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:26:31.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.054+0000 7fd1f875f640 1 Processor -- start 2024-04-02T13:26:31.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.054+0000 7fd1f875f640 1 -- start start 2024-04-02T13:26:31.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.055+0000 7fd1f875f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd1f015e0f0 0x7fd1f00d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.055+0000 7fd1f875f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1f006bbe0 0x7fd1f00add60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.055+0000 7fd1f875f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1f006dfe0 con 0x7fd1f00d0630 2024-04-02T13:26:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.055+0000 7fd1f875f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd1f006e2c0 con 0x7fd1f015e0f0 2024-04-02T13:26:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.055+0000 7fd1f875f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd1f006e5a0 con 0x7fd1f006bbe0 2024-04-02T13:26:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.055+0000 7fd1f64fb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd1f015e0f0 0x7fd1f00d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.055+0000 7fd1f64fb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd1f015e0f0 0x7fd1f00d00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46048/0 (socket says 172.21.15.44:46048) 2024-04-02T13:26:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.055+0000 7fd1f64fb640 1 -- 172.21.15.44:0/3988658225 learned_addr learned my addr 172.21.15.44:0/3988658225 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.055+0000 7fd1f6cfc640 1 --2- 172.21.15.44:0/3988658225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1f006bbe0 0x7fd1f00add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.055+0000 7fd1f64fb640 1 -- 172.21.15.44:0/3988658225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1f006bbe0 msgr2=0x7fd1f00add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.055+0000 7fd1f64fb640 1 --2- 172.21.15.44:0/3988658225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1f006bbe0 0x7fd1f00add60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.055+0000 7fd1f64fb640 1 -- 172.21.15.44:0/3988658225 >> v1:172.21.15.69:6789/0 conn(0x7fd1f00d0630 legacy=0x7fd1f006b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.055+0000 7fd1f64fb640 1 -- 172.21.15.44:0/3988658225 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f006e880 con 0x7fd1f015e0f0 2024-04-02T13:26:31.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.056+0000 7fd1f64fb640 1 --2- 172.21.15.44:0/3988658225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1f015e0f0 0x7fd1f00d00f0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec009870 tx=0x7fd1ec02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c1d119300a16d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:31.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.056+0000 7fd1f54f9640 1 -- 172.21.15.44:0/3988658225 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec02f840 con 0x7fd1f015e0f0 2024-04-02T13:26:31.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.056+0000 7fd1f54f9640 1 -- 172.21.15.44:0/3988658225 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1ec02f9e0 con 0x7fd1f015e0f0 2024-04-02T13:26:31.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.056+0000 7fd1f54f9640 1 -- 172.21.15.44:0/3988658225 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec02fdb0 con 0x7fd1f015e0f0 2024-04-02T13:26:31.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.057+0000 7fd1f875f640 1 -- 172.21.15.44:0/3988658225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1f015e0f0 msgr2=0x7fd1f00d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.057+0000 7fd1f875f640 1 --2- 172.21.15.44:0/3988658225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1f015e0f0 0x7fd1f00d00f0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec009870 tx=0x7fd1ec02ba20 comp rx=0 tx=0).stop 2024-04-02T13:26:31.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.057+0000 7fd1f875f640 1 -- 172.21.15.44:0/3988658225 shutdown_connections 2024-04-02T13:26:31.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.057+0000 7fd1f875f640 1 --2- 172.21.15.44:0/3988658225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1f006bbe0 0x7fd1f00add60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.057+0000 7fd1f875f640 1 --2- 172.21.15.44:0/3988658225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1f015e0f0 0x7fd1f00d00f0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.057+0000 7fd1f875f640 1 -- 172.21.15.44:0/3988658225 >> 172.21.15.44:0/3988658225 conn(0x7fd1f00b2980 msgr2=0x7fd1f00b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:31.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.057+0000 7fd1f875f640 1 -- 172.21.15.44:0/3988658225 shutdown_connections 2024-04-02T13:26:31.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.057+0000 7fd1f875f640 1 -- 172.21.15.44:0/3988658225 wait complete. 2024-04-02T13:26:31.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.058+0000 7fd1f875f640 1 Processor -- start 2024-04-02T13:26:31.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.058+0000 7fd1f875f640 1 -- start start 2024-04-02T13:26:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f875f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1f006bbe0 0x7fd1f01e8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f875f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1f00d0630 0x7fd1f01f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f875f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1f015e0f0 0x7fd1f00c8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f875f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f015e4d0 con 0x7fd1f00d0630 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f875f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f006eda0 con 0x7fd1f015e0f0 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f875f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd1f015e650 con 0x7fd1f006bbe0 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f5cfa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1f00d0630 0x7fd1f01f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f5cfa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1f00d0630 0x7fd1f01f38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46050/0 (socket says 172.21.15.44:46050) 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f5cfa640 1 -- 172.21.15.44:0/3270763457 learned_addr learned my addr 172.21.15.44:0/3270763457 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f6cfc640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1f015e0f0 0x7fd1f00c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f64fb640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1f006bbe0 0x7fd1f01e8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f5cfa640 1 -- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1f006bbe0 msgr2=0x7fd1f01e8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f5cfa640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1f006bbe0 0x7fd1f01e8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f5cfa640 1 -- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1f015e0f0 msgr2=0x7fd1f00c8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f5cfa640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1f015e0f0 0x7fd1f00c8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f5cfa640 1 -- 172.21.15.44:0/3270763457 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f01e94d0 con 0x7fd1f00d0630 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f6cfc640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1f015e0f0 0x7fd1f00c8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f64fb640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1f006bbe0 0x7fd1f01e8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f5cfa640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1f00d0630 0x7fd1f01f38c0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000c400 tx=0x7fd1e000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:31.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1e6ffd640 1 -- 172.21.15.44:0/3270763457 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e00052d0 con 0x7fd1f00d0630 2024-04-02T13:26:31.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1e6ffd640 1 -- 172.21.15.44:0/3270763457 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1e0005490 con 0x7fd1f00d0630 2024-04-02T13:26:31.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f875f640 1 -- 172.21.15.44:0/3270763457 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f01e7ef0 con 0x7fd1f00d0630 2024-04-02T13:26:31.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1e6ffd640 1 -- 172.21.15.44:0/3270763457 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0007260 con 0x7fd1f00d0630 2024-04-02T13:26:31.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.059+0000 7fd1f875f640 1 -- 172.21.15.44:0/3270763457 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f01e84b0 con 0x7fd1f00d0630 2024-04-02T13:26:31.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.060+0000 7fd1f875f640 1 -- 172.21.15.44:0/3270763457 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1f00684f0 con 0x7fd1f00d0630 2024-04-02T13:26:31.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.061+0000 7fd1e6ffd640 1 -- 172.21.15.44:0/3270763457 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd1e0007400 con 0x7fd1f00d0630 2024-04-02T13:26:31.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.061+0000 7fd1e6ffd640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd1d406c8f0 0x7fd1d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.061+0000 7fd1e6ffd640 1 -- 172.21.15.44:0/3270763457 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7fd1e008b220 con 0x7fd1f00d0630 2024-04-02T13:26:31.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.062+0000 7fd1f64fb640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd1d406c8f0 0x7fd1d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.062+0000 7fd1f64fb640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd1d406c8f0 0x7fd1d406edb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec036000 tx=0x7fd1ec039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:31.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.066+0000 7fd1e6ffd640 1 -- 172.21.15.44:0/3270763457 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd1e0059500 con 0x7fd1f00d0630 2024-04-02T13:26:31.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.254+0000 7fd1f875f640 1 -- 172.21.15.44:0/3270763457 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd1f00727f0 con 0x7fd1f00d0630 2024-04-02T13:26:31.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.255+0000 7fd1e6ffd640 1 -- 172.21.15.44:0/3270763457 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+17024 (secure 0 0 0) 0x7fd1e0058e00 con 0x7fd1f00d0630 2024-04-02T13:26:31.259 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:31.259 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":80,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:25:10.921159+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:24:52.202720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-02T13:24:52.916529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6837/698185867":"2024-04-03T13:25:10.921135+0000","172.21.15.69:6835/698185867":"2024-04-03T13:25:10.921135+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:26:31.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.257+0000 7fd1f875f640 1 -- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd1d406c8f0 msgr2=0x7fd1d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.257+0000 7fd1f875f640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd1d406c8f0 0x7fd1d406edb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec036000 tx=0x7fd1ec039040 comp rx=0 tx=0).stop 2024-04-02T13:26:31.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.257+0000 7fd1f875f640 1 -- 172.21.15.44:0/3270763457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1f00d0630 msgr2=0x7fd1f01f38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.257+0000 7fd1f875f640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1f00d0630 0x7fd1f01f38c0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000c400 tx=0x7fd1e000c8c0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.257+0000 7fd1f875f640 1 -- 172.21.15.44:0/3270763457 shutdown_connections 2024-04-02T13:26:31.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.257+0000 7fd1f875f640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd1d406c8f0 0x7fd1d406edb0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.257+0000 7fd1f875f640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1f015e0f0 0x7fd1f00c8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.257+0000 7fd1f875f640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1f00d0630 0x7fd1f01f38c0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.257+0000 7fd1f875f640 1 --2- 172.21.15.44:0/3270763457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1f006bbe0 0x7fd1f01e8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.257+0000 7fd1f875f640 1 -- 172.21.15.44:0/3270763457 >> 172.21.15.44:0/3270763457 conn(0x7fd1f00b2980 msgr2=0x7fd1f006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:31.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.257+0000 7fd1f875f640 1 -- 172.21.15.44:0/3270763457 shutdown_connections 2024-04-02T13:26:31.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.257+0000 7fd1f875f640 1 -- 172.21.15.44:0/3270763457 wait complete. 2024-04-02T13:26:31.272 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:26:31.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.371+0000 7f2a4e938640 1 Processor -- start 2024-04-02T13:26:31.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.371+0000 7f2a4e938640 1 -- start start 2024-04-02T13:26:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a4e938640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2a4815e0f0 0x7f2a480d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a4e938640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a4806bbe0 0x7f2a480add60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a4e938640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a4806dfe0 con 0x7f2a480d0630 2024-04-02T13:26:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a4e938640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2a4806e2c0 con 0x7f2a4815e0f0 2024-04-02T13:26:31.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a4e938640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2a4806e5a0 con 0x7f2a4806bbe0 2024-04-02T13:26:31.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a47fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2a4815e0f0 0x7f2a480d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a477fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f2a480d0630 0x7f2a4806b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:36658/0 (socket says 172.21.15.44:36658) 2024-04-02T13:26:31.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a477fe640 1 -- 172.21.15.44:0/3850277704 learned_addr learned my addr 172.21.15.44:0/3850277704 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:31.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a4ced5640 1 --2- 172.21.15.44:0/3850277704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a4806bbe0 0x7f2a480add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a47fff640 1 -- 172.21.15.44:0/3850277704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a4806bbe0 msgr2=0x7f2a480add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:31.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a47fff640 1 --2- 172.21.15.44:0/3850277704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a4806bbe0 0x7f2a480add60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a47fff640 1 -- 172.21.15.44:0/3850277704 >> v1:172.21.15.44:6789/0 conn(0x7f2a480d0630 legacy=0x7f2a4806b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a47fff640 1 -- 172.21.15.44:0/3850277704 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4806e880 con 0x7f2a4815e0f0 2024-04-02T13:26:31.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.372+0000 7f2a46ffd640 1 -- 172.21.15.44:0/3850277704 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1013666867 0 0) 0x7f2a4806dfe0 con 0x7f2a480d0630 2024-04-02T13:26:31.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.373+0000 7f2a47fff640 1 --2- 172.21.15.44:0/3850277704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a4815e0f0 0x7f2a480d00f0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f2a300097a0 tx=0x7f2a3002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2b2aa0fb4d83be server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:31.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.373+0000 7f2a46ffd640 1 -- 172.21.15.44:0/3850277704 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a30034070 con 0x7f2a4815e0f0 2024-04-02T13:26:31.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.373+0000 7f2a46ffd640 1 -- 172.21.15.44:0/3850277704 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a30030610 con 0x7f2a4815e0f0 2024-04-02T13:26:31.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.373+0000 7f2a46ffd640 1 -- 172.21.15.44:0/3850277704 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a300309e0 con 0x7f2a4815e0f0 2024-04-02T13:26:31.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.374+0000 7f2a4e938640 1 -- 172.21.15.44:0/3850277704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a4815e0f0 msgr2=0x7f2a480d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.374+0000 7f2a4e938640 1 --2- 172.21.15.44:0/3850277704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a4815e0f0 0x7f2a480d00f0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f2a300097a0 tx=0x7f2a3002b960 comp rx=0 tx=0).stop 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.374+0000 7f2a4e938640 1 -- 172.21.15.44:0/3850277704 shutdown_connections 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.374+0000 7f2a4e938640 1 --2- 172.21.15.44:0/3850277704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a4806bbe0 0x7f2a480add60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.374+0000 7f2a4e938640 1 --2- 172.21.15.44:0/3850277704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a4815e0f0 0x7f2a480d00f0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.374+0000 7f2a4e938640 1 -- 172.21.15.44:0/3850277704 >> 172.21.15.44:0/3850277704 conn(0x7f2a480b2980 msgr2=0x7f2a480b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.374+0000 7f2a4e938640 1 -- 172.21.15.44:0/3850277704 shutdown_connections 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.374+0000 7f2a4e938640 1 -- 172.21.15.44:0/3850277704 wait complete. 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.375+0000 7f2a4e938640 1 Processor -- start 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.375+0000 7f2a4e938640 1 -- start start 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a4e938640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a4806bbe0 0x7f2a48157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a4e938640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a480d0630 0x7f2a480c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a4e938640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a4815e0f0 0x7f2a480c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a4e938640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4815e4d0 con 0x7f2a480d0630 2024-04-02T13:26:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a4e938640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4806eda0 con 0x7f2a4815e0f0 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a4e938640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2a4815e650 con 0x7f2a4806bbe0 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a477fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a480d0630 0x7f2a480c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a477fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a480d0630 0x7f2a480c95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46060/0 (socket says 172.21.15.44:46060) 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a477fe640 1 -- 172.21.15.44:0/2876205217 learned_addr learned my addr 172.21.15.44:0/2876205217 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a477fe640 1 -- 172.21.15.44:0/2876205217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a4806bbe0 msgr2=0x7f2a48157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a477fe640 1 --2- 172.21.15.44:0/2876205217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a4806bbe0 0x7f2a48157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a477fe640 1 -- 172.21.15.44:0/2876205217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a4815e0f0 msgr2=0x7f2a480c9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a4ced5640 1 --2- 172.21.15.44:0/2876205217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a4815e0f0 0x7f2a480c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a477fe640 1 --2- 172.21.15.44:0/2876205217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a4815e0f0 0x7f2a480c9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a477fe640 1 -- 172.21.15.44:0/2876205217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a480c61d0 con 0x7f2a480d0630 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a477fe640 1 --2- 172.21.15.44:0/2876205217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a480d0630 0x7f2a480c95b0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f2a3800c4d0 tx=0x7f2a3800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a44ff9640 1 -- 172.21.15.44:0/2876205217 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38018070 con 0x7f2a480d0630 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a44ff9640 1 -- 172.21.15.44:0/2876205217 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a38007500 con 0x7f2a480d0630 2024-04-02T13:26:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a4e938640 1 -- 172.21.15.44:0/2876205217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a480c6450 con 0x7f2a480d0630 2024-04-02T13:26:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a44ff9640 1 -- 172.21.15.44:0/2876205217 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38004a20 con 0x7f2a480d0630 2024-04-02T13:26:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.376+0000 7f2a4e938640 1 -- 172.21.15.44:0/2876205217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a480c6a10 con 0x7f2a480d0630 2024-04-02T13:26:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.377+0000 7f2a4e938640 1 -- 172.21.15.44:0/2876205217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a480684f0 con 0x7f2a480d0630 2024-04-02T13:26:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.378+0000 7f2a44ff9640 1 -- 172.21.15.44:0/2876205217 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a3801b070 con 0x7f2a480d0630 2024-04-02T13:26:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.378+0000 7f2a44ff9640 1 --2- 172.21.15.44:0/2876205217 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a2806c8f0 0x7f2a2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.378+0000 7f2a44ff9640 1 -- 172.21.15.44:0/2876205217 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6187+0+0 (secure 0 0 0) 0x7f2a3808b030 con 0x7f2a480d0630 2024-04-02T13:26:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.379+0000 7f2a47fff640 1 --2- 172.21.15.44:0/2876205217 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a2806c8f0 0x7f2a2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.379+0000 7f2a47fff640 1 --2- 172.21.15.44:0/2876205217 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a2806c8f0 0x7f2a2806edb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2a30036000 tx=0x7f2a30030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.383+0000 7f2a44ff9640 1 -- 172.21.15.44:0/2876205217 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2a480c61d0 con 0x7f2a480d0630 2024-04-02T13:26:31.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.604+0000 7f2a4e938640 1 -- 172.21.15.44:0/2876205217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f2a480727f0 con 0x7f2a480d0630 2024-04-02T13:26:31.607 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:26:31.604+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:26:31.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.606+0000 7f2a44ff9640 1 -- 172.21.15.44:0/2876205217 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v53) ==== 161+0+0 (secure 0 0 0) 0x7f2a38059280 con 0x7f2a480d0630 2024-04-02T13:26:31.609 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:26:31.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.609+0000 7f2a4e938640 1 -- 172.21.15.44:0/2876205217 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a2806c8f0 msgr2=0x7f2a2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.609+0000 7f2a4e938640 1 --2- 172.21.15.44:0/2876205217 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a2806c8f0 0x7f2a2806edb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2a30036000 tx=0x7f2a30030000 comp rx=0 tx=0).stop 2024-04-02T13:26:31.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.609+0000 7f2a4e938640 1 -- 172.21.15.44:0/2876205217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a480d0630 msgr2=0x7f2a480c95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.609+0000 7f2a4e938640 1 --2- 172.21.15.44:0/2876205217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a480d0630 0x7f2a480c95b0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f2a3800c4d0 tx=0x7f2a3800c990 comp rx=0 tx=0).stop 2024-04-02T13:26:31.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.609+0000 7f2a4e938640 1 -- 172.21.15.44:0/2876205217 shutdown_connections 2024-04-02T13:26:31.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.609+0000 7f2a4e938640 1 --2- 172.21.15.44:0/2876205217 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2a2806c8f0 0x7f2a2806edb0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.609+0000 7f2a4e938640 1 --2- 172.21.15.44:0/2876205217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a4815e0f0 0x7f2a480c9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.609+0000 7f2a4e938640 1 --2- 172.21.15.44:0/2876205217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a480d0630 0x7f2a480c95b0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.609+0000 7f2a4e938640 1 --2- 172.21.15.44:0/2876205217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a4806bbe0 0x7f2a48157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.609+0000 7f2a4e938640 1 -- 172.21.15.44:0/2876205217 >> 172.21.15.44:0/2876205217 conn(0x7f2a480b2980 msgr2=0x7f2a480b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:31.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.609+0000 7f2a4e938640 1 -- 172.21.15.44:0/2876205217 shutdown_connections 2024-04-02T13:26:31.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.609+0000 7f2a4e938640 1 -- 172.21.15.44:0/2876205217 wait complete. 2024-04-02T13:26:31.614 INFO:tasks.ceph.mds.a.smithi044.stderr: -4372> 2024-04-02T13:25:15.217+0000 7f40dbbaf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:26:31.624 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T13:26:31.644 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:26:31.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.726+0000 7f302f02c640 1 Processor -- start 2024-04-02T13:26:31.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.726+0000 7f302f02c640 1 -- start start 2024-04-02T13:26:31.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.727+0000 7f302f02c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f302815e420 0x7f30280ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.727+0000 7f302f02c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f302806c220 con 0x7f30280721f0 2024-04-02T13:26:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.727+0000 7f302f02c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f302815f7e0 con 0x7f30280cfe10 2024-04-02T13:26:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.727+0000 7f302f02c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f302815ec60 con 0x7f302815e420 2024-04-02T13:26:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.727+0000 7f3027fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f30280721f0 0x7f30280cf700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:36668/0 (socket says 172.21.15.44:36668) 2024-04-02T13:26:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.727+0000 7f3027fff640 1 -- 172.21.15.44:0/2562799119 learned_addr learned my addr 172.21.15.44:0/2562799119 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.727+0000 7f302cdc8640 1 --2- 172.21.15.44:0/2562799119 >> v2:172.21.15.69:3300/0 conn(0x7f302815e420 0x7f30280ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.727+0000 7f30277fe640 1 -- 172.21.15.44:0/2562799119 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 997675515 0 0) 0x7f3014001110 con 0x7f30280721f0 2024-04-02T13:26:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.727+0000 7f30277fe640 1 -- 172.21.15.44:0/2562799119 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3018003520 con 0x7f30280721f0 2024-04-02T13:26:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.727+0000 7f302cdc8640 1 -- 172.21.15.44:0/2562799119 >> v1:172.21.15.69:6789/0 conn(0x7f30280cfe10 legacy=0x7f30280cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.727+0000 7f302cdc8640 1 -- 172.21.15.44:0/2562799119 >> v1:172.21.15.44:6789/0 conn(0x7f30280721f0 legacy=0x7f30280cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.727+0000 7f302cdc8640 1 -- 172.21.15.44:0/2562799119 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f302815ef40 con 0x7f302815e420 2024-04-02T13:26:31.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.728+0000 7f302cdc8640 1 --2- 172.21.15.44:0/2562799119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302815e420 0x7f30280ceca0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f301c0097a0 tx=0x7f301c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbb4a98834fdaf48 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:31.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.728+0000 7f30277fe640 1 -- 172.21.15.44:0/2562799119 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301c0315a0 con 0x7f302815e420 2024-04-02T13:26:31.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.728+0000 7f30277fe640 1 -- 172.21.15.44:0/2562799119 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f301c037070 con 0x7f302815e420 2024-04-02T13:26:31.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.728+0000 7f30277fe640 1 -- 172.21.15.44:0/2562799119 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301c031940 con 0x7f302815e420 2024-04-02T13:26:31.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.729+0000 7f302f02c640 1 -- 172.21.15.44:0/2562799119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302815e420 msgr2=0x7f30280ceca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.729+0000 7f302f02c640 1 --2- 172.21.15.44:0/2562799119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302815e420 0x7f30280ceca0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f301c0097a0 tx=0x7f301c02b840 comp rx=0 tx=0).stop 2024-04-02T13:26:31.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.729+0000 7f302f02c640 1 -- 172.21.15.44:0/2562799119 shutdown_connections 2024-04-02T13:26:31.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.729+0000 7f302f02c640 1 --2- 172.21.15.44:0/2562799119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302815e420 0x7f30280ceca0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.729+0000 7f302f02c640 1 -- 172.21.15.44:0/2562799119 >> 172.21.15.44:0/2562799119 conn(0x7f30280b2a10 msgr2=0x7f30280b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:31.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.729+0000 7f302f02c640 1 -- 172.21.15.44:0/2562799119 shutdown_connections 2024-04-02T13:26:31.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.729+0000 7f302f02c640 1 -- 172.21.15.44:0/2562799119 wait complete. 2024-04-02T13:26:31.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.730+0000 7f302f02c640 1 Processor -- start 2024-04-02T13:26:31.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.730+0000 7f302f02c640 1 -- start start 2024-04-02T13:26:31.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f302f02c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30280721f0 0x7f302814de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f302f02c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30280cfe10 0x7f302815b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f302f02c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302815e420 0x7f302815c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f302f02c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f30280cf1e0 con 0x7f30280cfe10 2024-04-02T13:26:31.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f302f02c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f302815f500 con 0x7f302815e420 2024-04-02T13:26:31.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f302f02c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f30280cf360 con 0x7f30280721f0 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f3027fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30280cfe10 0x7f302815b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f3027fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30280cfe10 0x7f302815b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46096/0 (socket says 172.21.15.44:46096) 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f3027fff640 1 -- 172.21.15.44:0/3172024645 learned_addr learned my addr 172.21.15.44:0/3172024645 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f302cdc8640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30280721f0 0x7f302814de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f302d5c9640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302815e420 0x7f302815c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f3027fff640 1 -- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30280721f0 msgr2=0x7f302814de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f3027fff640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30280721f0 0x7f302814de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f3027fff640 1 -- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302815e420 msgr2=0x7f302815c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f3027fff640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302815e420 0x7f302815c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f3027fff640 1 -- 172.21.15.44:0/3172024645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3028157980 con 0x7f30280cfe10 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f302cdc8640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30280721f0 0x7f302814de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f302d5c9640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302815e420 0x7f302815c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f3027fff640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30280cfe10 0x7f302815b280 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f301400c730 tx=0x7f301400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:31.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f30257fa640 1 -- 172.21.15.44:0/3172024645 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014017070 con 0x7f30280cfe10 2024-04-02T13:26:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f30257fa640 1 -- 172.21.15.44:0/3172024645 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3014005ab0 con 0x7f30280cfe10 2024-04-02T13:26:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f302f02c640 1 -- 172.21.15.44:0/3172024645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3028157c60 con 0x7f30280cfe10 2024-04-02T13:26:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f30257fa640 1 -- 172.21.15.44:0/3172024645 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014007070 con 0x7f30280cfe10 2024-04-02T13:26:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.732+0000 7f302f02c640 1 -- 172.21.15.44:0/3172024645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3028158220 con 0x7f30280cfe10 2024-04-02T13:26:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.733+0000 7f302f02c640 1 -- 172.21.15.44:0/3172024645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30280684f0 con 0x7f30280cfe10 2024-04-02T13:26:31.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.733+0000 7f30257fa640 1 -- 172.21.15.44:0/3172024645 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f301401a070 con 0x7f30280cfe10 2024-04-02T13:26:31.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.734+0000 7f30257fa640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f300c06c920 0x7f300c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:31.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.734+0000 7f30257fa640 1 -- 172.21.15.44:0/3172024645 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 6273+0+0 (secure 0 0 0) 0x7f301408b0a0 con 0x7f30280cfe10 2024-04-02T13:26:31.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.734+0000 7f302cdc8640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f300c06c920 0x7f300c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:31.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.734+0000 7f302cdc8640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f300c06c920 0x7f300c06ede0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f301c036000 tx=0x7f301c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:31.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.738+0000 7f30257fa640 1 -- 172.21.15.44:0/3172024645 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f302806c220 con 0x7f30280cfe10 2024-04-02T13:26:31.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:31.960+0000 7f302f02c640 1 -- 172.21.15.44:0/3172024645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f30280b48f0 con 0x7f30280cfe10 2024-04-02T13:26:32.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.607+0000 7f30257fa640 1 -- 172.21.15.44:0/3172024645 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v54) ==== 104+0+0 (secure 0 0 0) 0x7f3014059320 con 0x7f30280cfe10 2024-04-02T13:26:32.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.610+0000 7f302f02c640 1 -- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f300c06c920 msgr2=0x7f300c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:32.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.610+0000 7f302f02c640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f300c06c920 0x7f300c06ede0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f301c036000 tx=0x7f301c008030 comp rx=0 tx=0).stop 2024-04-02T13:26:32.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.610+0000 7f302f02c640 1 -- 172.21.15.44:0/3172024645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30280cfe10 msgr2=0x7f302815b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:32.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.610+0000 7f302f02c640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30280cfe10 0x7f302815b280 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f301400c730 tx=0x7f301400cbf0 comp rx=0 tx=0).stop 2024-04-02T13:26:32.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.611+0000 7f302f02c640 1 -- 172.21.15.44:0/3172024645 shutdown_connections 2024-04-02T13:26:32.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.611+0000 7f302f02c640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f300c06c920 0x7f300c06ede0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:32.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.611+0000 7f302f02c640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302815e420 0x7f302815c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:32.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.611+0000 7f302f02c640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30280cfe10 0x7f302815b280 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:32.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.611+0000 7f302f02c640 1 --2- 172.21.15.44:0/3172024645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30280721f0 0x7f302814de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:32.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.611+0000 7f302f02c640 1 -- 172.21.15.44:0/3172024645 >> 172.21.15.44:0/3172024645 conn(0x7f30280b2a10 msgr2=0x7f3028159a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:32.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.611+0000 7f302f02c640 1 -- 172.21.15.44:0/3172024645 shutdown_connections 2024-04-02T13:26:32.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.611+0000 7f302f02c640 1 -- 172.21.15.44:0/3172024645 wait complete. 2024-04-02T13:26:32.628 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T13:26:32.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.779+0000 7ffbd8049640 1 Processor -- start 2024-04-02T13:26:32.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.779+0000 7ffbd8049640 1 -- start start 2024-04-02T13:26:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd8049640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ffbd00b1c30 0x7ffbd00b2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd8049640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd00b39a0 con 0x7ffbd006d520 2024-04-02T13:26:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd8049640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd00b2550 con 0x7ffbd00cd810 2024-04-02T13:26:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd8049640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ffbd00ce800 con 0x7ffbd00b1c30 2024-04-02T13:26:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd5de5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ffbd00b1c30 0x7ffbd00b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd65e6640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ffbd006d520 0x7ffbd006d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:36674/0 (socket says 172.21.15.44:36674) 2024-04-02T13:26:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd65e6640 1 -- 172.21.15.44:0/986318183 learned_addr learned my addr 172.21.15.44:0/986318183 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd5de5640 1 -- 172.21.15.44:0/986318183 >> v1:172.21.15.69:6789/0 conn(0x7ffbd00cd810 legacy=0x7ffbd00b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd5de5640 1 -- 172.21.15.44:0/986318183 >> v1:172.21.15.44:6789/0 conn(0x7ffbd006d520 legacy=0x7ffbd006d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd5de5640 1 -- 172.21.15.44:0/986318183 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd00cea30 con 0x7ffbd00b1c30 2024-04-02T13:26:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd5de5640 1 --2- 172.21.15.44:0/986318183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd00b1c30 0x7ffbd00b2010 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0097a0 tx=0x7ffbcc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd029346b41f5ab7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd4de3640 1 -- 172.21.15.44:0/986318183 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc035020 con 0x7ffbd00b1c30 2024-04-02T13:26:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd4de3640 1 -- 172.21.15.44:0/986318183 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbcc031830 con 0x7ffbd00b1c30 2024-04-02T13:26:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.780+0000 7ffbd4de3640 1 -- 172.21.15.44:0/986318183 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc031c00 con 0x7ffbd00b1c30 2024-04-02T13:26:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.781+0000 7ffbd8049640 1 -- 172.21.15.44:0/986318183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd00b1c30 msgr2=0x7ffbd00b2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.781+0000 7ffbd8049640 1 --2- 172.21.15.44:0/986318183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd00b1c30 0x7ffbd00b2010 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0097a0 tx=0x7ffbcc02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:26:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.781+0000 7ffbd8049640 1 -- 172.21.15.44:0/986318183 shutdown_connections 2024-04-02T13:26:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.781+0000 7ffbd8049640 1 --2- 172.21.15.44:0/986318183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd00b1c30 0x7ffbd00b2010 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.781+0000 7ffbd8049640 1 -- 172.21.15.44:0/986318183 >> 172.21.15.44:0/986318183 conn(0x7ffbd006b4d0 msgr2=0x7ffbd006b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.781+0000 7ffbd8049640 1 -- 172.21.15.44:0/986318183 shutdown_connections 2024-04-02T13:26:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.781+0000 7ffbd8049640 1 -- 172.21.15.44:0/986318183 wait complete. 2024-04-02T13:26:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.782+0000 7ffbd8049640 1 Processor -- start 2024-04-02T13:26:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.782+0000 7ffbd8049640 1 -- start start 2024-04-02T13:26:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.783+0000 7ffbd8049640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd006d520 0x7ffbd014fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.783+0000 7ffbd8049640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd00b1c30 0x7ffbd00c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.783+0000 7ffbd8049640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd00cd810 0x7ffbd00ae080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.783+0000 7ffbd8049640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd00cf2d0 con 0x7ffbd00b1c30 2024-04-02T13:26:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.783+0000 7ffbd8049640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd00ceff0 con 0x7ffbd006d520 2024-04-02T13:26:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.783+0000 7ffbd8049640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ffbd00cf5b0 con 0x7ffbd00cd810 2024-04-02T13:26:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.783+0000 7ffbd55e4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd00b1c30 0x7ffbd00c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.783+0000 7ffbd55e4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd00b1c30 0x7ffbd00c9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46114/0 (socket says 172.21.15.44:46114) 2024-04-02T13:26:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.783+0000 7ffbd55e4640 1 -- 172.21.15.44:0/1448495189 learned_addr learned my addr 172.21.15.44:0/1448495189 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.783+0000 7ffbd5de5640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd006d520 0x7ffbd014fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.783+0000 7ffbd65e6640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd00cd810 0x7ffbd00ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:32.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.783+0000 7ffbd55e4640 1 -- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd00cd810 msgr2=0x7ffbd00ae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:32.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.783+0000 7ffbd55e4640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd00cd810 0x7ffbd00ae080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:32.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.784+0000 7ffbd55e4640 1 -- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd006d520 msgr2=0x7ffbd014fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:32.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.784+0000 7ffbd55e4640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd006d520 0x7ffbd014fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:32.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.784+0000 7ffbd55e4640 1 -- 172.21.15.44:0/1448495189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd00ad020 con 0x7ffbd00b1c30 2024-04-02T13:26:32.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.784+0000 7ffbd55e4640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd00b1c30 0x7ffbd00c9ac0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000b3f0 tx=0x7ffbc000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:32.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.784+0000 7ffbc67fc640 1 -- 172.21.15.44:0/1448495189 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0015020 con 0x7ffbd00b1c30 2024-04-02T13:26:32.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.784+0000 7ffbc67fc640 1 -- 172.21.15.44:0/1448495189 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbc0011830 con 0x7ffbd00b1c30 2024-04-02T13:26:32.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.784+0000 7ffbc67fc640 1 -- 172.21.15.44:0/1448495189 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0011c20 con 0x7ffbd00b1c30 2024-04-02T13:26:32.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.784+0000 7ffbd8049640 1 -- 172.21.15.44:0/1448495189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd00ad2c0 con 0x7ffbd00b1c30 2024-04-02T13:26:32.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.784+0000 7ffbd8049640 1 -- 172.21.15.44:0/1448495189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd00ad880 con 0x7ffbd00b1c30 2024-04-02T13:26:32.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.786+0000 7ffbd8049640 1 -- 172.21.15.44:0/1448495189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbd00684f0 con 0x7ffbd00b1c30 2024-04-02T13:26:32.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.786+0000 7ffbc67fc640 1 -- 172.21.15.44:0/1448495189 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffbc0018020 con 0x7ffbd00b1c30 2024-04-02T13:26:32.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.786+0000 7ffbc67fc640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ffbb406c8f0 0x7ffbb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:32.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.786+0000 7ffbc67fc640 1 -- 172.21.15.44:0/1448495189 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 6273+0+0 (secure 0 0 0) 0x7ffbc008b8a0 con 0x7ffbd00b1c30 2024-04-02T13:26:32.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.786+0000 7ffbd5de5640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ffbb406c8f0 0x7ffbb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:32.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.788+0000 7ffbd5de5640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ffbb406c8f0 0x7ffbb406edb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc036000 tx=0x7ffbcc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:32.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.790+0000 7ffbc67fc640 1 -- 172.21.15.44:0/1448495189 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ffbc001f030 con 0x7ffbd00b1c30 2024-04-02T13:26:32.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:32.975+0000 7ffbd8049640 1 -- 172.21.15.44:0/1448495189 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ffbd00b0780 con 0x7ffbd00b1c30 2024-04-02T13:26:33.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.614+0000 7ffbc67fc640 1 -- 172.21.15.44:0/1448495189 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v82) ==== 181+0+0 (secure 0 0 0) 0x7ffbc0059b20 con 0x7ffbd00b1c30 2024-04-02T13:26:33.616 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' removed 2024-04-02T13:26:33.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.617+0000 7ffbd8049640 1 -- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ffbb406c8f0 msgr2=0x7ffbb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:33.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.617+0000 7ffbd8049640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ffbb406c8f0 0x7ffbb406edb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc036000 tx=0x7ffbcc008030 comp rx=0 tx=0).stop 2024-04-02T13:26:33.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.617+0000 7ffbd8049640 1 -- 172.21.15.44:0/1448495189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd00b1c30 msgr2=0x7ffbd00c9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:33.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.617+0000 7ffbd8049640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd00b1c30 0x7ffbd00c9ac0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000b3f0 tx=0x7ffbc000b8b0 comp rx=0 tx=0).stop 2024-04-02T13:26:33.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.617+0000 7ffbd8049640 1 -- 172.21.15.44:0/1448495189 shutdown_connections 2024-04-02T13:26:33.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.617+0000 7ffbd8049640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ffbb406c8f0 0x7ffbb406edb0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:33.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.617+0000 7ffbd8049640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd00cd810 0x7ffbd00ae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:33.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.617+0000 7ffbd8049640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd00b1c30 0x7ffbd00c9ac0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:33.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.617+0000 7ffbd8049640 1 --2- 172.21.15.44:0/1448495189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd006d520 0x7ffbd014fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:33.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.617+0000 7ffbd8049640 1 -- 172.21.15.44:0/1448495189 >> 172.21.15.44:0/1448495189 conn(0x7ffbd006b4d0 msgr2=0x7ffbd006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:33.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.617+0000 7ffbd8049640 1 -- 172.21.15.44:0/1448495189 shutdown_connections 2024-04-02T13:26:33.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.617+0000 7ffbd8049640 1 -- 172.21.15.44:0/1448495189 wait complete. 2024-04-02T13:26:33.633 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T13:26:33.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.743+0000 7fbc400d3640 1 Processor -- start 2024-04-02T13:26:33.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.743+0000 7fbc400d3640 1 -- start start 2024-04-02T13:26:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.744+0000 7fbc400d3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbc3815e470 0x7fbc380b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.744+0000 7fbc400d3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbc380ce790 0x7fbc3806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.744+0000 7fbc400d3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc380ced00 con 0x7fbc380b1540 2024-04-02T13:26:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.744+0000 7fbc400d3640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fbc380cefe0 con 0x7fbc3815e470 2024-04-02T13:26:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.744+0000 7fbc400d3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbc380cf2c0 con 0x7fbc380ce790 2024-04-02T13:26:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.744+0000 7fbc3de6f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbc3815e470 0x7fbc380b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.744+0000 7fbc3de6f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbc3815e470 0x7fbc380b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46128/0 (socket says 172.21.15.44:46128) 2024-04-02T13:26:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.744+0000 7fbc3de6f640 1 -- 172.21.15.44:0/3778078219 learned_addr learned my addr 172.21.15.44:0/3778078219 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.744+0000 7fbc3e670640 1 --2- 172.21.15.44:0/3778078219 >> v2:172.21.15.69:3300/0 conn(0x7fbc380ce790 0x7fbc3806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.744+0000 7fbc3de6f640 1 -- 172.21.15.44:0/3778078219 >> v2:172.21.15.69:3300/0 conn(0x7fbc380ce790 msgr2=0x7fbc3806d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.744+0000 7fbc3de6f640 1 --2- 172.21.15.44:0/3778078219 >> v2:172.21.15.69:3300/0 conn(0x7fbc380ce790 0x7fbc3806d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.744+0000 7fbc3de6f640 1 -- 172.21.15.44:0/3778078219 >> v1:172.21.15.69:6789/0 conn(0x7fbc380b1540 legacy=0x7fbc380d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.744+0000 7fbc3de6f640 1 -- 172.21.15.44:0/3778078219 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc380cf5a0 con 0x7fbc3815e470 2024-04-02T13:26:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.745+0000 7fbc3de6f640 1 --2- 172.21.15.44:0/3778078219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc3815e470 0x7fbc380b32c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fbc380cc460 tx=0x7fbc3402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4097de5de2b16861 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.745+0000 7fbc3ce6d640 1 -- 172.21.15.44:0/3778078219 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3402e070 con 0x7fbc3815e470 2024-04-02T13:26:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.745+0000 7fbc3ce6d640 1 -- 172.21.15.44:0/3778078219 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc34037070 con 0x7fbc3815e470 2024-04-02T13:26:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.745+0000 7fbc3ce6d640 1 -- 172.21.15.44:0/3778078219 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc34031a00 con 0x7fbc3815e470 2024-04-02T13:26:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.746+0000 7fbc400d3640 1 -- 172.21.15.44:0/3778078219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc3815e470 msgr2=0x7fbc380b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.746+0000 7fbc400d3640 1 --2- 172.21.15.44:0/3778078219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc3815e470 0x7fbc380b32c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fbc380cc460 tx=0x7fbc3402b880 comp rx=0 tx=0).stop 2024-04-02T13:26:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.746+0000 7fbc400d3640 1 -- 172.21.15.44:0/3778078219 shutdown_connections 2024-04-02T13:26:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.746+0000 7fbc400d3640 1 --2- 172.21.15.44:0/3778078219 >> v2:172.21.15.69:3300/0 conn(0x7fbc380ce790 0x7fbc3806d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.746+0000 7fbc400d3640 1 --2- 172.21.15.44:0/3778078219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc3815e470 0x7fbc380b32c0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.746+0000 7fbc400d3640 1 -- 172.21.15.44:0/3778078219 >> 172.21.15.44:0/3778078219 conn(0x7fbc3809aa20 msgr2=0x7fbc3809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.746+0000 7fbc400d3640 1 -- 172.21.15.44:0/3778078219 shutdown_connections 2024-04-02T13:26:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.746+0000 7fbc400d3640 1 -- 172.21.15.44:0/3778078219 wait complete. 2024-04-02T13:26:33.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.747+0000 7fbc400d3640 1 Processor -- start 2024-04-02T13:26:33.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.747+0000 7fbc400d3640 1 -- start start 2024-04-02T13:26:33.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc400d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc380b1540 0x7fbc38156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:33.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc400d3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc380ce790 0x7fbc3815c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:33.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc400d3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc3815e470 0x7fbc380c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:33.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc400d3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbc380d22c0 con 0x7fbc3815e470 2024-04-02T13:26:33.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc400d3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3806cb60 con 0x7fbc380ce790 2024-04-02T13:26:33.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc400d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbc380d2440 con 0x7fbc380b1540 2024-04-02T13:26:33.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc3e670640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc3815e470 0x7fbc380c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:33.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc3e670640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc3815e470 0x7fbc380c59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46144/0 (socket says 172.21.15.44:46144) 2024-04-02T13:26:33.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc3e670640 1 -- 172.21.15.44:0/3556866551 learned_addr learned my addr 172.21.15.44:0/3556866551 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:33.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc3d66e640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc380ce790 0x7fbc3815c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:33.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc3de6f640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc380b1540 0x7fbc38156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:33.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc3e670640 1 -- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc380b1540 msgr2=0x7fbc38156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:33.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc3e670640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc380b1540 0x7fbc38156f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:33.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc3e670640 1 -- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc380ce790 msgr2=0x7fbc3815c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:33.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc3e670640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc380ce790 0x7fbc3815c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:33.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc3e670640 1 -- 172.21.15.44:0/3556866551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3815b750 con 0x7fbc3815e470 2024-04-02T13:26:33.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc3de6f640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc380b1540 0x7fbc38156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:33.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc3e670640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc3815e470 0x7fbc380c59b0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fbc3000cda0 tx=0x7fbc3000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:33.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.748+0000 7fbc2e7fc640 1 -- 172.21.15.44:0/3556866551 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc30003b90 con 0x7fbc3815e470 2024-04-02T13:26:33.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.749+0000 7fbc2e7fc640 1 -- 172.21.15.44:0/3556866551 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc30003d30 con 0x7fbc3815e470 2024-04-02T13:26:33.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.749+0000 7fbc400d3640 1 -- 172.21.15.44:0/3556866551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc3815ba30 con 0x7fbc3815e470 2024-04-02T13:26:33.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.749+0000 7fbc2e7fc640 1 -- 172.21.15.44:0/3556866551 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc30004240 con 0x7fbc3815e470 2024-04-02T13:26:33.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.749+0000 7fbc400d3640 1 -- 172.21.15.44:0/3556866551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc3815bff0 con 0x7fbc3815e470 2024-04-02T13:26:33.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.749+0000 7fbc400d3640 1 -- 172.21.15.44:0/3556866551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc380b7b40 con 0x7fbc3815e470 2024-04-02T13:26:33.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.750+0000 7fbc2e7fc640 1 -- 172.21.15.44:0/3556866551 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbc300043e0 con 0x7fbc3815e470 2024-04-02T13:26:33.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.750+0000 7fbc2e7fc640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbc1c06c8f0 0x7fbc1c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:33.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.750+0000 7fbc2e7fc640 1 -- 172.21.15.44:0/3556866551 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5835+0+0 (secure 0 0 0) 0x7fbc3008c780 con 0x7fbc3815e470 2024-04-02T13:26:33.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.751+0000 7fbc3de6f640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbc1c06c8f0 0x7fbc1c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:33.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.751+0000 7fbc3de6f640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbc1c06c8f0 0x7fbc1c06edb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbc34036000 tx=0x7fbc34008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:33.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.755+0000 7fbc2e7fc640 1 -- 172.21.15.44:0/3556866551 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbc3001d340 con 0x7fbc3815e470 2024-04-02T13:26:33.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:33.939+0000 7fbc400d3640 1 -- 172.21.15.44:0/3556866551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbc380684f0 con 0x7fbc3815e470 2024-04-02T13:26:34.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.615+0000 7fbc2e7fc640 1 -- 172.21.15.44:0/3556866551 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v83) ==== 169+0+0 (secure 0 0 0) 0x7fbc3005aa90 con 0x7fbc3815e470 2024-04-02T13:26:34.618 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' removed 2024-04-02T13:26:34.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.618+0000 7fbc400d3640 1 -- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbc1c06c8f0 msgr2=0x7fbc1c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:34.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.618+0000 7fbc400d3640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbc1c06c8f0 0x7fbc1c06edb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbc34036000 tx=0x7fbc34008030 comp rx=0 tx=0).stop 2024-04-02T13:26:34.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.618+0000 7fbc400d3640 1 -- 172.21.15.44:0/3556866551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc3815e470 msgr2=0x7fbc380c59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:34.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.618+0000 7fbc400d3640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc3815e470 0x7fbc380c59b0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fbc3000cda0 tx=0x7fbc3000a660 comp rx=0 tx=0).stop 2024-04-02T13:26:34.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.618+0000 7fbc400d3640 1 -- 172.21.15.44:0/3556866551 shutdown_connections 2024-04-02T13:26:34.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.618+0000 7fbc400d3640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbc1c06c8f0 0x7fbc1c06edb0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:34.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.618+0000 7fbc400d3640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc3815e470 0x7fbc380c59b0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:34.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.618+0000 7fbc400d3640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc380ce790 0x7fbc3815c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:34.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.618+0000 7fbc400d3640 1 --2- 172.21.15.44:0/3556866551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc380b1540 0x7fbc38156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:34.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.618+0000 7fbc400d3640 1 -- 172.21.15.44:0/3556866551 >> 172.21.15.44:0/3556866551 conn(0x7fbc3809aa20 msgr2=0x7fbc380aedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:34.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.618+0000 7fbc400d3640 1 -- 172.21.15.44:0/3556866551 shutdown_connections 2024-04-02T13:26:34.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.618+0000 7fbc400d3640 1 -- 172.21.15.44:0/3556866551 wait complete. 2024-04-02T13:26:34.634 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T13:26:34.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.740+0000 7efca5582640 1 Processor -- start 2024-04-02T13:26:34.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.740+0000 7efca5582640 1 -- start start 2024-04-02T13:26:34.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.741+0000 7efca5582640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efca00721f0 0x7efca00cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:34.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.741+0000 7efca5582640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efca006c220 con 0x7efca015e420 2024-04-02T13:26:34.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.741+0000 7efca5582640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efca015f7e0 con 0x7efca00cfc40 2024-04-02T13:26:34.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.741+0000 7efca5582640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7efca00d00b0 con 0x7efca00721f0 2024-04-02T13:26:34.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.741+0000 7efc9effd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7efca015e420 0x7efca00ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54044/0 (socket says 172.21.15.44:54044) 2024-04-02T13:26:34.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.741+0000 7efc9effd640 1 -- 172.21.15.44:0/3967561908 learned_addr learned my addr 172.21.15.44:0/3967561908 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:34.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.741+0000 7efc9e7fc640 1 --2- 172.21.15.44:0/3967561908 >> v2:172.21.15.69:3300/0 conn(0x7efca00721f0 0x7efca00cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:34.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.741+0000 7efc9dffb640 1 -- 172.21.15.44:0/3967561908 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4289739321 0 0) 0x7efc90001490 con 0x7efca015e420 2024-04-02T13:26:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.741+0000 7efc9dffb640 1 -- 172.21.15.44:0/3967561908 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc88003520 con 0x7efca015e420 2024-04-02T13:26:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.741+0000 7efc9dffb640 1 -- 172.21.15.44:0/3967561908 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1879720012 0 0) 0x7efca015f7e0 con 0x7efca00cfc40 2024-04-02T13:26:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.742+0000 7efc9dffb640 1 -- 172.21.15.44:0/3967561908 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc90001490 con 0x7efca00cfc40 2024-04-02T13:26:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.742+0000 7efc9e7fc640 1 -- 172.21.15.44:0/3967561908 >> v1:172.21.15.69:6789/0 conn(0x7efca00cfc40 legacy=0x7efca00cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.742+0000 7efc9e7fc640 1 -- 172.21.15.44:0/3967561908 >> v1:172.21.15.44:6789/0 conn(0x7efca015e420 legacy=0x7efca00ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.742+0000 7efc9dffb640 1 -- 172.21.15.44:0/3967561908 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3581014073 0 0) 0x7efc88003520 con 0x7efca015e420 2024-04-02T13:26:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.742+0000 7efc9e7fc640 1 -- 172.21.15.44:0/3967561908 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca015ec60 con 0x7efca00721f0 2024-04-02T13:26:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.742+0000 7efc9e7fc640 1 --2- 172.21.15.44:0/3967561908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efca00721f0 0x7efca00cf700 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efc940097a0 tx=0x7efc9402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3884fc6f639ec401 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.743+0000 7efc9dffb640 1 -- 172.21.15.44:0/3967561908 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc940315a0 con 0x7efca00721f0 2024-04-02T13:26:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.743+0000 7efc9dffb640 1 -- 172.21.15.44:0/3967561908 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc94037070 con 0x7efca00721f0 2024-04-02T13:26:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.743+0000 7efc9dffb640 1 -- 172.21.15.44:0/3967561908 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc94031940 con 0x7efca00721f0 2024-04-02T13:26:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.743+0000 7efca5582640 1 -- 172.21.15.44:0/3967561908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efca00721f0 msgr2=0x7efca00cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.743+0000 7efca5582640 1 --2- 172.21.15.44:0/3967561908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efca00721f0 0x7efca00cf700 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efc940097a0 tx=0x7efc9402b840 comp rx=0 tx=0).stop 2024-04-02T13:26:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.743+0000 7efca5582640 1 -- 172.21.15.44:0/3967561908 shutdown_connections 2024-04-02T13:26:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.743+0000 7efca5582640 1 --2- 172.21.15.44:0/3967561908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efca00721f0 0x7efca00cf700 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.743+0000 7efca5582640 1 -- 172.21.15.44:0/3967561908 >> 172.21.15.44:0/3967561908 conn(0x7efca00b2a10 msgr2=0x7efca00b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.743+0000 7efca5582640 1 -- 172.21.15.44:0/3967561908 shutdown_connections 2024-04-02T13:26:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.743+0000 7efca5582640 1 -- 172.21.15.44:0/3967561908 wait complete. 2024-04-02T13:26:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.744+0000 7efca5582640 1 Processor -- start 2024-04-02T13:26:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.745+0000 7efca5582640 1 -- start start 2024-04-02T13:26:34.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.745+0000 7efca5582640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efca00721f0 0x7efca0150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:34.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.745+0000 7efca5582640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efca00cfc40 0x7efca0151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:34.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.745+0000 7efca5582640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efca015e420 0x7efca00b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:34.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.745+0000 7efca5582640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efca015f220 con 0x7efca00721f0 2024-04-02T13:26:34.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.745+0000 7efca5582640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efca00cf3b0 con 0x7efca015e420 2024-04-02T13:26:34.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.745+0000 7efca5582640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7efca015f500 con 0x7efca00cfc40 2024-04-02T13:26:34.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.745+0000 7efc9effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efca00721f0 0x7efca0150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:34.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.745+0000 7efc9effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efca00721f0 0x7efca0150920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52744/0 (socket says 172.21.15.44:52744) 2024-04-02T13:26:34.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.745+0000 7efc9effd640 1 -- 172.21.15.44:0/1098270893 learned_addr learned my addr 172.21.15.44:0/1098270893 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efc9f7fe640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efca015e420 0x7efca00b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efc9e7fc640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efca00cfc40 0x7efca0151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efc9effd640 1 -- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efca00cfc40 msgr2=0x7efca0151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efc9effd640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efca00cfc40 0x7efca0151b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efc9effd640 1 -- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efca015e420 msgr2=0x7efca00b49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efc9effd640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efca015e420 0x7efca00b49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efc9effd640 1 -- 172.21.15.44:0/1098270893 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca0154780 con 0x7efca00721f0 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efc9f7fe640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efca015e420 0x7efca00b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efc9e7fc640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efca00cfc40 0x7efca0151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efc9effd640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efca00721f0 0x7efca0150920 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7efc9000cca0 tx=0x7efc9000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efc877fe640 1 -- 172.21.15.44:0/1098270893 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90017070 con 0x7efca00721f0 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efc877fe640 1 -- 172.21.15.44:0/1098270893 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc90012400 con 0x7efca00721f0 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efca5582640 1 -- 172.21.15.44:0/1098270893 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca0153b70 con 0x7efca00721f0 2024-04-02T13:26:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efc877fe640 1 -- 172.21.15.44:0/1098270893 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90012830 con 0x7efca00721f0 2024-04-02T13:26:34.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.746+0000 7efca5582640 1 -- 172.21.15.44:0/1098270893 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca0154130 con 0x7efca00721f0 2024-04-02T13:26:34.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.747+0000 7efca5582640 1 -- 172.21.15.44:0/1098270893 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca00684f0 con 0x7efca00721f0 2024-04-02T13:26:34.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.747+0000 7efc877fe640 1 -- 172.21.15.44:0/1098270893 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efc9001a070 con 0x7efca00721f0 2024-04-02T13:26:34.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.748+0000 7efc877fe640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efc8c06c8f0 0x7efc8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:34.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.748+0000 7efc877fe640 1 -- 172.21.15.44:0/1098270893 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5421+0+0 (secure 0 0 0) 0x7efc9008b950 con 0x7efca00721f0 2024-04-02T13:26:34.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.748+0000 7efc9e7fc640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efc8c06c8f0 0x7efc8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:34.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.749+0000 7efc9e7fc640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efc8c06c8f0 0x7efc8c06edb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7efc94036000 tx=0x7efc94008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:34.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.752+0000 7efc877fe640 1 -- 172.21.15.44:0/1098270893 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7efca006c220 con 0x7efca00721f0 2024-04-02T13:26:34.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.947+0000 7efca5582640 1 -- 172.21.15.44:0/1098270893 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7efca00b3770 con 0x7efca00721f0 2024-04-02T13:26:34.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.950+0000 7efc877fe640 1 -- 172.21.15.44:0/1098270893 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) ==== 104+0+0 (secure 0 0 0) 0x7efc90059f30 con 0x7efca00721f0 2024-04-02T13:26:34.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.952+0000 7efca5582640 1 -- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efc8c06c8f0 msgr2=0x7efc8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:34.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.952+0000 7efca5582640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efc8c06c8f0 0x7efc8c06edb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7efc94036000 tx=0x7efc94008030 comp rx=0 tx=0).stop 2024-04-02T13:26:34.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.952+0000 7efca5582640 1 -- 172.21.15.44:0/1098270893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efca00721f0 msgr2=0x7efca0150920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:34.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.952+0000 7efca5582640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efca00721f0 0x7efca0150920 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7efc9000cca0 tx=0x7efc9000a430 comp rx=0 tx=0).stop 2024-04-02T13:26:34.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.952+0000 7efca5582640 1 -- 172.21.15.44:0/1098270893 shutdown_connections 2024-04-02T13:26:34.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.952+0000 7efca5582640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efc8c06c8f0 0x7efc8c06edb0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:34.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.952+0000 7efca5582640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efca015e420 0x7efca00b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:34.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.952+0000 7efca5582640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efca00cfc40 0x7efca0151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:34.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.952+0000 7efca5582640 1 --2- 172.21.15.44:0/1098270893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efca00721f0 0x7efca0150920 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:34.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.952+0000 7efca5582640 1 -- 172.21.15.44:0/1098270893 >> 172.21.15.44:0/1098270893 conn(0x7efca00b2a10 msgr2=0x7efca015d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:34.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.952+0000 7efca5582640 1 -- 172.21.15.44:0/1098270893 shutdown_connections 2024-04-02T13:26:34.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:34.952+0000 7efca5582640 1 -- 172.21.15.44:0/1098270893 wait complete. 2024-04-02T13:26:34.965 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair' 2024-04-02T13:26:35.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.065+0000 7ff8e6414640 1 Processor -- start 2024-04-02T13:26:35.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.065+0000 7ff8e6414640 1 -- start start 2024-04-02T13:26:35.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.066+0000 7ff8e6414640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff8e015c1c0 0x7ff8e00affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:35.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.066+0000 7ff8e6414640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8e00b3d70 0x7ff8e00b0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:35.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.066+0000 7ff8e6414640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8e00b1640 con 0x7ff8e00b0a40 2024-04-02T13:26:35.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.066+0000 7ff8e6414640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff8e00aef20 con 0x7ff8e015c1c0 2024-04-02T13:26:35.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.066+0000 7ff8e6414640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff8e00af200 con 0x7ff8e00b3d70 2024-04-02T13:26:35.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.066+0000 7ff8dffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff8e015c1c0 0x7ff8e00affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:35.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.066+0000 7ff8dffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff8e015c1c0 0x7ff8e00affc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52760/0 (socket says 172.21.15.44:52760) 2024-04-02T13:26:35.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.066+0000 7ff8dffff640 1 -- 172.21.15.44:0/605957121 learned_addr learned my addr 172.21.15.44:0/605957121 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:35.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.066+0000 7ff8df7fe640 1 --2- 172.21.15.44:0/605957121 >> v2:172.21.15.69:3300/0 conn(0x7ff8e00b3d70 0x7ff8e00b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:35.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.066+0000 7ff8dffff640 1 -- 172.21.15.44:0/605957121 >> v2:172.21.15.69:3300/0 conn(0x7ff8e00b3d70 msgr2=0x7ff8e00b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:35.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.066+0000 7ff8dffff640 1 --2- 172.21.15.44:0/605957121 >> v2:172.21.15.69:3300/0 conn(0x7ff8e00b3d70 0x7ff8e00b0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.066+0000 7ff8dffff640 1 -- 172.21.15.44:0/605957121 >> v1:172.21.15.69:6789/0 conn(0x7ff8e00b0a40 legacy=0x7ff8e0164e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.066+0000 7ff8dffff640 1 -- 172.21.15.44:0/605957121 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e00af4e0 con 0x7ff8e015c1c0 2024-04-02T13:26:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.067+0000 7ff8dffff640 1 --2- 172.21.15.44:0/605957121 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e015c1c0 0x7ff8e00affc0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00d2fc0 tx=0x7ff8c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc3b3c36b8fc19d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.067+0000 7ff8deffd640 1 -- 172.21.15.44:0/605957121 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8035020 con 0x7ff8e015c1c0 2024-04-02T13:26:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.067+0000 7ff8deffd640 1 -- 172.21.15.44:0/605957121 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8c8031870 con 0x7ff8e015c1c0 2024-04-02T13:26:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.067+0000 7ff8deffd640 1 -- 172.21.15.44:0/605957121 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8031c40 con 0x7ff8e015c1c0 2024-04-02T13:26:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.068+0000 7ff8e6414640 1 -- 172.21.15.44:0/605957121 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e015c1c0 msgr2=0x7ff8e00affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.068+0000 7ff8e6414640 1 --2- 172.21.15.44:0/605957121 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e015c1c0 0x7ff8e00affc0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00d2fc0 tx=0x7ff8c802b910 comp rx=0 tx=0).stop 2024-04-02T13:26:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.068+0000 7ff8e6414640 1 -- 172.21.15.44:0/605957121 shutdown_connections 2024-04-02T13:26:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.068+0000 7ff8e6414640 1 --2- 172.21.15.44:0/605957121 >> v2:172.21.15.69:3300/0 conn(0x7ff8e00b3d70 0x7ff8e00b0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.068+0000 7ff8e6414640 1 --2- 172.21.15.44:0/605957121 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e015c1c0 0x7ff8e00affc0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.068+0000 7ff8e6414640 1 -- 172.21.15.44:0/605957121 >> 172.21.15.44:0/605957121 conn(0x7ff8e009aa20 msgr2=0x7ff8e009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:35.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.068+0000 7ff8e6414640 1 -- 172.21.15.44:0/605957121 shutdown_connections 2024-04-02T13:26:35.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.068+0000 7ff8e6414640 1 -- 172.21.15.44:0/605957121 wait complete. 2024-04-02T13:26:35.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.069+0000 7ff8e6414640 1 Processor -- start 2024-04-02T13:26:35.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.069+0000 7ff8e6414640 1 -- start start 2024-04-02T13:26:35.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8e6414640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e00b0a40 0x7ff8e0152870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:35.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8e6414640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e00b3d70 0x7ff8e00c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:35.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8e6414640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e015c1c0 0x7ff8e00c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:35.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8e6414640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e00cc990 con 0x7ff8e00b3d70 2024-04-02T13:26:35.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8e6414640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e00afaa0 con 0x7ff8e00b0a40 2024-04-02T13:26:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8e6414640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff8e00aebf0 con 0x7ff8e015c1c0 2024-04-02T13:26:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e00b3d70 0x7ff8e00c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e00b3d70 0x7ff8e00c5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52762/0 (socket says 172.21.15.44:52762) 2024-04-02T13:26:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8df7fe640 1 -- 172.21.15.44:0/1919795442 learned_addr learned my addr 172.21.15.44:0/1919795442 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8dffff640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e00b0a40 0x7ff8e0152870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8e49b1640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e015c1c0 0x7ff8e00c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8df7fe640 1 -- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e015c1c0 msgr2=0x7ff8e00c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8df7fe640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e015c1c0 0x7ff8e00c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8df7fe640 1 -- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e00b0a40 msgr2=0x7ff8e0152870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8df7fe640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e00b0a40 0x7ff8e0152870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8df7fe640 1 -- 172.21.15.44:0/1919795442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e0152db0 con 0x7ff8e00b3d70 2024-04-02T13:26:35.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.070+0000 7ff8df7fe640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e00b3d70 0x7ff8e00c5ae0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000d800 tx=0x7ff8d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:35.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.071+0000 7ff8dcff9640 1 -- 172.21.15.44:0/1919795442 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d000b6c0 con 0x7ff8e00b3d70 2024-04-02T13:26:35.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.071+0000 7ff8dcff9640 1 -- 172.21.15.44:0/1919795442 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d0019070 con 0x7ff8e00b3d70 2024-04-02T13:26:35.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.071+0000 7ff8dcff9640 1 -- 172.21.15.44:0/1919795442 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d000ba80 con 0x7ff8e00b3d70 2024-04-02T13:26:35.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.071+0000 7ff8e6414640 1 -- 172.21.15.44:0/1919795442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e00b57f0 con 0x7ff8e00b3d70 2024-04-02T13:26:35.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.071+0000 7ff8e6414640 1 -- 172.21.15.44:0/1919795442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e00c6930 con 0x7ff8e00b3d70 2024-04-02T13:26:35.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.072+0000 7ff8e6414640 1 -- 172.21.15.44:0/1919795442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e00c6b70 con 0x7ff8e00b3d70 2024-04-02T13:26:35.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.072+0000 7ff8dcff9640 1 -- 172.21.15.44:0/1919795442 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff8d000bc20 con 0x7ff8e00b3d70 2024-04-02T13:26:35.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.072+0000 7ff8dcff9640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8c006c8f0 0x7ff8c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.073+0000 7ff8dcff9640 1 -- 172.21.15.44:0/1919795442 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5421+0+0 (secure 0 0 0) 0x7ff8d008bee0 con 0x7ff8e00b3d70 2024-04-02T13:26:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.073+0000 7ff8dffff640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8c006c8f0 0x7ff8c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.073+0000 7ff8dffff640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8c006c8f0 0x7ff8c006edb0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8036000 tx=0x7ff8c803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.077+0000 7ff8dcff9640 1 -- 172.21.15.44:0/1919795442 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff8d0011340 con 0x7ff8e00b3d70 2024-04-02T13:26:35.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.330+0000 7ff8e6414640 1 -- 172.21.15.44:0/1919795442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]} v 0) -- 0x7ff8e00684f0 con 0x7ff8e00b3d70 2024-04-02T13:26:35.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.624+0000 7ff8dcff9640 1 -- 172.21.15.44:0/1919795442 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]}]=0 v310) ==== 166+0+0 (secure 0 0 0) 0x7ff8d005a430 con 0x7ff8e00b3d70 2024-04-02T13:26:35.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.626+0000 7ff8e6414640 1 -- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8c006c8f0 msgr2=0x7ff8c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:35.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.626+0000 7ff8e6414640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8c006c8f0 0x7ff8c006edb0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8036000 tx=0x7ff8c803c040 comp rx=0 tx=0).stop 2024-04-02T13:26:35.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.626+0000 7ff8e6414640 1 -- 172.21.15.44:0/1919795442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e00b3d70 msgr2=0x7ff8e00c5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:35.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.626+0000 7ff8e6414640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e00b3d70 0x7ff8e00c5ae0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000d800 tx=0x7ff8d000dcc0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.626+0000 7ff8e6414640 1 -- 172.21.15.44:0/1919795442 shutdown_connections 2024-04-02T13:26:35.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.626+0000 7ff8e6414640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8c006c8f0 0x7ff8c006edb0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.626+0000 7ff8e6414640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e015c1c0 0x7ff8e00c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.626+0000 7ff8e6414640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e00b3d70 0x7ff8e00c5ae0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.626+0000 7ff8e6414640 1 --2- 172.21.15.44:0/1919795442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e00b0a40 0x7ff8e0152870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.626+0000 7ff8e6414640 1 -- 172.21.15.44:0/1919795442 >> 172.21.15.44:0/1919795442 conn(0x7ff8e009aa20 msgr2=0x7ff8e015b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:35.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.626+0000 7ff8e6414640 1 -- 172.21.15.44:0/1919795442 shutdown_connections 2024-04-02T13:26:35.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.626+0000 7ff8e6414640 1 -- 172.21.15.44:0/1919795442 wait complete. 2024-04-02T13:26:35.639 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-04-02T13:26:35.640 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-04-02T13:26:35.640 INFO:tasks.cephfs_test_runner:test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-02T13:26:35.640 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:26:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.742+0000 7f79efc4a640 1 Processor -- start 2024-04-02T13:26:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.743+0000 7f79efc4a640 1 -- start start 2024-04-02T13:26:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.743+0000 7f79efc4a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f79e80cece0 0x7f79e80cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:35.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.743+0000 7f79ed9e6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f79e80cece0 0x7f79e80cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:35.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.743+0000 7f79ed1e5640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f79e80d09f0 0x7f79e806d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54048/0 (socket says 172.21.15.44:54048) 2024-04-02T13:26:35.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.744+0000 7f79ed1e5640 1 -- 172.21.15.44:0/3250077349 learned_addr learned my addr 172.21.15.44:0/3250077349 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:35.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.744+0000 7f79efc4a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79e80cf600 con 0x7f79e80d09f0 2024-04-02T13:26:35.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.744+0000 7f79efc4a640 1 -- 172.21.15.44:0/3250077349 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79e806dc40 con 0x7f79e806ece0 2024-04-02T13:26:35.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.744+0000 7f79efc4a640 1 -- 172.21.15.44:0/3250077349 --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f79e806d9e0 con 0x7f79e80cece0 2024-04-02T13:26:35.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.744+0000 7f79ed9e6640 1 -- 172.21.15.44:0/3250077349 >> v1:172.21.15.69:6789/0 conn(0x7f79e806ece0 legacy=0x7f79e80acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:35.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.744+0000 7f79ed9e6640 1 -- 172.21.15.44:0/3250077349 >> v1:172.21.15.44:6789/0 conn(0x7f79e80d09f0 legacy=0x7f79e806d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:35.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.744+0000 7f79ed9e6640 1 -- 172.21.15.44:0/3250077349 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e806df20 con 0x7f79e80cece0 2024-04-02T13:26:35.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.745+0000 7f79ed9e6640 1 --2- 172.21.15.44:0/3250077349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e80cece0 0x7f79e80cf0c0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f79e00097a0 tx=0x7f79e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613ca3e3a366763f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:35.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.745+0000 7f79ec9e4640 1 -- 172.21.15.44:0/3250077349 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e00315a0 con 0x7f79e80cece0 2024-04-02T13:26:35.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.745+0000 7f79ec9e4640 1 -- 172.21.15.44:0/3250077349 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e0037070 con 0x7f79e80cece0 2024-04-02T13:26:35.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.745+0000 7f79ec9e4640 1 -- 172.21.15.44:0/3250077349 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0031960 con 0x7f79e80cece0 2024-04-02T13:26:35.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.745+0000 7f79efc4a640 1 -- 172.21.15.44:0/3250077349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e80cece0 msgr2=0x7f79e80cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:35.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.745+0000 7f79efc4a640 1 --2- 172.21.15.44:0/3250077349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e80cece0 0x7f79e80cf0c0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f79e00097a0 tx=0x7f79e002b840 comp rx=0 tx=0).stop 2024-04-02T13:26:35.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.745+0000 7f79efc4a640 1 -- 172.21.15.44:0/3250077349 shutdown_connections 2024-04-02T13:26:35.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.745+0000 7f79efc4a640 1 --2- 172.21.15.44:0/3250077349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e80cece0 0x7f79e80cf0c0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.745+0000 7f79efc4a640 1 -- 172.21.15.44:0/3250077349 >> 172.21.15.44:0/3250077349 conn(0x7f79e80cfcd0 msgr2=0x7f79e80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:35.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.745+0000 7f79efc4a640 1 -- 172.21.15.44:0/3250077349 shutdown_connections 2024-04-02T13:26:35.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.745+0000 7f79efc4a640 1 -- 172.21.15.44:0/3250077349 wait complete. 2024-04-02T13:26:35.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.747+0000 7f79efc4a640 1 Processor -- start 2024-04-02T13:26:35.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.747+0000 7f79efc4a640 1 -- start start 2024-04-02T13:26:35.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.747+0000 7f79efc4a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e806ece0 0x7f79e815c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:35.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.747+0000 7f79efc4a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79e80cece0 0x7f79e814e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:35.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.747+0000 7f79efc4a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79e80d09f0 0x7f79e81e5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:35.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.747+0000 7f79efc4a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f79e806e7c0 con 0x7f79e806ece0 2024-04-02T13:26:35.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.747+0000 7f79efc4a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f79e806e4e0 con 0x7f79e80d09f0 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.747+0000 7f79efc4a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f79e815b220 con 0x7f79e80cece0 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79ed9e6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e806ece0 0x7f79e815c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79ed9e6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e806ece0 0x7f79e815c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52780/0 (socket says 172.21.15.44:52780) 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79ed9e6640 1 -- 172.21.15.44:0/2396258078 learned_addr learned my addr 172.21.15.44:0/2396258078 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79ee1e7640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79e80d09f0 0x7f79e81e5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79ed1e5640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79e80cece0 0x7f79e814e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79ed9e6640 1 -- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79e80cece0 msgr2=0x7f79e814e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79ed9e6640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79e80cece0 0x7f79e814e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79ed9e6640 1 -- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79e80d09f0 msgr2=0x7f79e81e5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79ed9e6640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79e80d09f0 0x7f79e81e5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79ed9e6640 1 -- 172.21.15.44:0/2396258078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e80afc20 con 0x7f79e806ece0 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79ed1e5640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79e80cece0 0x7f79e814e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79ed9e6640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e806ece0 0x7f79e815c370 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f79e00098c0 tx=0x7f79e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79de7fc640 1 -- 172.21.15.44:0/2396258078 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0032840 con 0x7f79e806ece0 2024-04-02T13:26:35.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79de7fc640 1 -- 172.21.15.44:0/2396258078 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e00329e0 con 0x7f79e806ece0 2024-04-02T13:26:35.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79efc4a640 1 -- 172.21.15.44:0/2396258078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e80aff00 con 0x7f79e806ece0 2024-04-02T13:26:35.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79de7fc640 1 -- 172.21.15.44:0/2396258078 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0032dd0 con 0x7f79e806ece0 2024-04-02T13:26:35.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.748+0000 7f79efc4a640 1 -- 172.21.15.44:0/2396258078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e80b04c0 con 0x7f79e806ece0 2024-04-02T13:26:35.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.749+0000 7f79efc4a640 1 -- 172.21.15.44:0/2396258078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79e8074f20 con 0x7f79e806ece0 2024-04-02T13:26:35.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.750+0000 7f79de7fc640 1 -- 172.21.15.44:0/2396258078 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f79e004d3f0 con 0x7f79e806ece0 2024-04-02T13:26:35.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.750+0000 7f79de7fc640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f79cc06c8f0 0x7f79cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:35.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.750+0000 7f79de7fc640 1 -- 172.21.15.44:0/2396258078 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5421+0+0 (secure 0 0 0) 0x7f79e002f030 con 0x7f79e806ece0 2024-04-02T13:26:35.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.750+0000 7f79ed1e5640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f79cc06c8f0 0x7f79cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:35.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.751+0000 7f79ed1e5640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f79cc06c8f0 0x7f79cc06edb0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f79e815bc00 tx=0x7f79d8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:35.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.755+0000 7f79de7fc640 1 -- 172.21.15.44:0/2396258078 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f79e0040070 con 0x7f79e806ece0 2024-04-02T13:26:35.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.938+0000 7f79efc4a640 1 -- 172.21.15.44:0/2396258078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79e80684f0 con 0x7f79e806ece0 2024-04-02T13:26:35.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.939+0000 7f79de7fc640 1 -- 172.21.15.44:0/2396258078 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+13374 (secure 0 0 0) 0x7f79e007bb50 con 0x7f79e806ece0 2024-04-02T13:26:35.942 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:35.942 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":83,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:26:34.614513+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6833/2392159648":"2024-04-03T13:26:31.606075+0000","172.21.15.69:6837/698185867":"2024-04-03T13:25:10.921135+0000","172.21.15.44:6832/2392159648":"2024-04-03T13:26:31.606075+0000","172.21.15.69:6835/698185867":"2024-04-03T13:25:10.921135+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:26:35.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.941+0000 7f79efc4a640 1 -- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f79cc06c8f0 msgr2=0x7f79cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:35.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.941+0000 7f79efc4a640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f79cc06c8f0 0x7f79cc06edb0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f79e815bc00 tx=0x7f79d8009040 comp rx=0 tx=0).stop 2024-04-02T13:26:35.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.941+0000 7f79efc4a640 1 -- 172.21.15.44:0/2396258078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e806ece0 msgr2=0x7f79e815c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:35.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.941+0000 7f79efc4a640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e806ece0 0x7f79e815c370 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f79e00098c0 tx=0x7f79e00046c0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.941+0000 7f79efc4a640 1 -- 172.21.15.44:0/2396258078 shutdown_connections 2024-04-02T13:26:35.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.941+0000 7f79efc4a640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f79cc06c8f0 0x7f79cc06edb0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.941+0000 7f79efc4a640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79e80d09f0 0x7f79e81e5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.941+0000 7f79efc4a640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79e80cece0 0x7f79e814e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.941+0000 7f79efc4a640 1 --2- 172.21.15.44:0/2396258078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e806ece0 0x7f79e815c370 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:35.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.941+0000 7f79efc4a640 1 -- 172.21.15.44:0/2396258078 >> 172.21.15.44:0/2396258078 conn(0x7f79e80cfcd0 msgr2=0x7f79e80ccf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:35.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.941+0000 7f79efc4a640 1 -- 172.21.15.44:0/2396258078 shutdown_connections 2024-04-02T13:26:35.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:35.941+0000 7f79efc4a640 1 -- 172.21.15.44:0/2396258078 wait complete. 2024-04-02T13:26:35.955 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:26:35.955 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:26:36.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.057+0000 7f273f159640 1 Processor -- start 2024-04-02T13:26:36.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.057+0000 7f273f159640 1 -- start start 2024-04-02T13:26:36.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.058+0000 7f273f159640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f273815e1d0 0x7f27380d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:36.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.058+0000 7f273f159640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f27380cec60 0x7f27380d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:36.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.058+0000 7f273f159640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f273815e5b0 0x7f27380d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:36.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.058+0000 7f273f159640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f27380cfe00 con 0x7f273815e5b0 2024-04-02T13:26:36.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.058+0000 7f273f159640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f27380d1640 con 0x7f27380cec60 2024-04-02T13:26:36.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.058+0000 7f273f159640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f27380cf1f0 con 0x7f273815e1d0 2024-04-02T13:26:36.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.058+0000 7f273d6f6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f273815e5b0 0x7f27380d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:36.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.058+0000 7f273d6f6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f273815e5b0 0x7f27380d2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52788/0 (socket says 172.21.15.44:52788) 2024-04-02T13:26:36.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.058+0000 7f273d6f6640 1 -- 172.21.15.44:0/3462882313 learned_addr learned my addr 172.21.15.44:0/3462882313 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.059+0000 7f2737fff640 1 --2- 172.21.15.44:0/3462882313 >> v2:172.21.15.69:3300/0 conn(0x7f27380cec60 0x7f27380d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.059+0000 7f273cef5640 1 --2- 172.21.15.44:0/3462882313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f273815e1d0 0x7f27380d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.059+0000 7f273d6f6640 1 -- 172.21.15.44:0/3462882313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f273815e1d0 msgr2=0x7f27380d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.059+0000 7f273d6f6640 1 --2- 172.21.15.44:0/3462882313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f273815e1d0 0x7f27380d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.059+0000 7f273d6f6640 1 -- 172.21.15.44:0/3462882313 >> v2:172.21.15.69:3300/0 conn(0x7f27380cec60 msgr2=0x7f27380d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.059+0000 7f273d6f6640 1 --2- 172.21.15.44:0/3462882313 >> v2:172.21.15.69:3300/0 conn(0x7f27380cec60 0x7f27380d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.059+0000 7f273d6f6640 1 -- 172.21.15.44:0/3462882313 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27380cf4d0 con 0x7f273815e5b0 2024-04-02T13:26:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.059+0000 7f273d6f6640 1 --2- 172.21.15.44:0/3462882313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f273815e5b0 0x7f27380d2c00 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f273809dae0 tx=0x7f273002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c0db9f88237409 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.060+0000 7f27377fe640 1 -- 172.21.15.44:0/3462882313 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273002c9e0 con 0x7f273815e5b0 2024-04-02T13:26:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.060+0000 7f27377fe640 1 -- 172.21.15.44:0/3462882313 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f273002cb80 con 0x7f273815e5b0 2024-04-02T13:26:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.060+0000 7f27377fe640 1 -- 172.21.15.44:0/3462882313 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27300343f0 con 0x7f273815e5b0 2024-04-02T13:26:36.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.060+0000 7f273f159640 1 -- 172.21.15.44:0/3462882313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f273815e5b0 msgr2=0x7f27380d2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:36.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.060+0000 7f273f159640 1 --2- 172.21.15.44:0/3462882313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f273815e5b0 0x7f27380d2c00 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f273809dae0 tx=0x7f273002be40 comp rx=0 tx=0).stop 2024-04-02T13:26:36.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.060+0000 7f273f159640 1 -- 172.21.15.44:0/3462882313 shutdown_connections 2024-04-02T13:26:36.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.060+0000 7f273f159640 1 --2- 172.21.15.44:0/3462882313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f273815e5b0 0x7f27380d2c00 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.060+0000 7f273f159640 1 --2- 172.21.15.44:0/3462882313 >> v2:172.21.15.69:3300/0 conn(0x7f27380cec60 0x7f27380d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.060+0000 7f273f159640 1 --2- 172.21.15.44:0/3462882313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f273815e1d0 0x7f27380d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.060+0000 7f273f159640 1 -- 172.21.15.44:0/3462882313 >> 172.21.15.44:0/3462882313 conn(0x7f273809d3b0 msgr2=0x7f27380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:36.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.060+0000 7f273f159640 1 -- 172.21.15.44:0/3462882313 shutdown_connections 2024-04-02T13:26:36.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.060+0000 7f273f159640 1 -- 172.21.15.44:0/3462882313 wait complete. 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.061+0000 7f273f159640 1 Processor -- start 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.061+0000 7f273f159640 1 -- start start 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.062+0000 7f273f159640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27380cec60 0x7f27381526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.062+0000 7f273f159640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f273815e1d0 0x7f273815fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.062+0000 7f273f159640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f273815e5b0 0x7f27380b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.062+0000 7f273f159640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f27380b4ce0 con 0x7f273815e1d0 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.062+0000 7f273f159640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f27380cfa90 con 0x7f273815e5b0 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.062+0000 7f273f159640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f27380b4e60 con 0x7f27380cec60 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.062+0000 7f2737fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f273815e1d0 0x7f273815fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.062+0000 7f2737fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f273815e1d0 0x7f273815fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52802/0 (socket says 172.21.15.44:52802) 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.062+0000 7f2737fff640 1 -- 172.21.15.44:0/3277464538 learned_addr learned my addr 172.21.15.44:0/3277464538 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f273d6f6640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f273815e5b0 0x7f27380b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f273cef5640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27380cec60 0x7f27381526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f2737fff640 1 -- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27380cec60 msgr2=0x7f27381526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f2737fff640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27380cec60 0x7f27381526f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f2737fff640 1 -- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f273815e5b0 msgr2=0x7f27380b8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:36.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f2737fff640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f273815e5b0 0x7f27380b8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f2737fff640 1 -- 172.21.15.44:0/3277464538 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27380cb590 con 0x7f273815e1d0 2024-04-02T13:26:36.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f273cef5640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27380cec60 0x7f27381526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:36.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f273d6f6640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f273815e5b0 0x7f27380b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:36.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f2737fff640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f273815e1d0 0x7f273815fcd0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f272400d800 tx=0x7f272400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:36.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f27357fa640 1 -- 172.21.15.44:0/3277464538 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272400b6c0 con 0x7f273815e1d0 2024-04-02T13:26:36.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f27357fa640 1 -- 172.21.15.44:0/3277464538 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2724019070 con 0x7f273815e1d0 2024-04-02T13:26:36.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f273f159640 1 -- 172.21.15.44:0/3277464538 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27380cb870 con 0x7f273815e1d0 2024-04-02T13:26:36.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f27357fa640 1 -- 172.21.15.44:0/3277464538 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272400ba80 con 0x7f273815e1d0 2024-04-02T13:26:36.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.063+0000 7f273f159640 1 -- 172.21.15.44:0/3277464538 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27380cbe30 con 0x7f273815e1d0 2024-04-02T13:26:36.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.064+0000 7f273f159640 1 -- 172.21.15.44:0/3277464538 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27380684f0 con 0x7f273815e1d0 2024-04-02T13:26:36.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.065+0000 7f27357fa640 1 -- 172.21.15.44:0/3277464538 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f272400bc20 con 0x7f273815e1d0 2024-04-02T13:26:36.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.065+0000 7f27357fa640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f271c06c8f0 0x7f271c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:36.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.065+0000 7f27357fa640 1 -- 172.21.15.44:0/3277464538 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5421+0+0 (secure 0 0 0) 0x7f272408bf90 con 0x7f273815e1d0 2024-04-02T13:26:36.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.065+0000 7f273cef5640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f271c06c8f0 0x7f271c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:36.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.066+0000 7f273cef5640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f271c06c8f0 0x7f271c06edb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f272c006fd0 tx=0x7f272c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:36.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.069+0000 7f27357fa640 1 -- 172.21.15.44:0/3277464538 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f272405a450 con 0x7f273815e1d0 2024-04-02T13:26:36.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.252+0000 7f273f159640 1 -- 172.21.15.44:0/3277464538 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f273815bb50 con 0x7f273815e1d0 2024-04-02T13:26:36.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.253+0000 7f27357fa640 1 -- 172.21.15.44:0/3277464538 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f2724059d50 con 0x7f273815e1d0 2024-04-02T13:26:36.255 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:26:36.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.255+0000 7f273f159640 1 -- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f271c06c8f0 msgr2=0x7f271c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:36.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.255+0000 7f273f159640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f271c06c8f0 0x7f271c06edb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f272c006fd0 tx=0x7f272c008040 comp rx=0 tx=0).stop 2024-04-02T13:26:36.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.255+0000 7f273f159640 1 -- 172.21.15.44:0/3277464538 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f273815e1d0 msgr2=0x7f273815fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:36.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.255+0000 7f273f159640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f273815e1d0 0x7f273815fcd0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f272400d800 tx=0x7f272400dcc0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.255+0000 7f273f159640 1 -- 172.21.15.44:0/3277464538 shutdown_connections 2024-04-02T13:26:36.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.255+0000 7f273f159640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f271c06c8f0 0x7f271c06edb0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.255+0000 7f273f159640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f273815e5b0 0x7f27380b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.255+0000 7f273f159640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f273815e1d0 0x7f273815fcd0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.255+0000 7f273f159640 1 --2- 172.21.15.44:0/3277464538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27380cec60 0x7f27381526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.255+0000 7f273f159640 1 -- 172.21.15.44:0/3277464538 >> 172.21.15.44:0/3277464538 conn(0x7f273809d3b0 msgr2=0x7f27380d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:36.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.255+0000 7f273f159640 1 -- 172.21.15.44:0/3277464538 shutdown_connections 2024-04-02T13:26:36.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.255+0000 7f273f159640 1 -- 172.21.15.44:0/3277464538 wait complete. 2024-04-02T13:26:36.268 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair' 2024-04-02T13:26:36.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.367+0000 7fc19d2d9640 1 Processor -- start 2024-04-02T13:26:36.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.367+0000 7fc19d2d9640 1 -- start start 2024-04-02T13:26:36.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.368+0000 7fc19d2d9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc198166f50 0x7fc198169340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:36.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.368+0000 7fc19d2d9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1980cf5e0 con 0x7fc19806c7f0 2024-04-02T13:26:36.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.368+0000 7fc19d2d9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc19815e810 con 0x7fc19815e430 2024-04-02T13:26:36.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.368+0000 7fc19d2d9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc1980d1410 con 0x7fc198166f50 2024-04-02T13:26:36.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.368+0000 7fc1967fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc19806c7f0 0x7fc1980ceed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54058/0 (socket says 172.21.15.44:54058) 2024-04-02T13:26:36.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.368+0000 7fc1967fc640 1 -- 172.21.15.44:0/2867664429 learned_addr learned my addr 172.21.15.44:0/2867664429 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:36.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.368+0000 7fc1977fe640 1 --2- 172.21.15.44:0/2867664429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc198166f50 0x7fc198169340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.368+0000 7fc195ffb640 1 -- 172.21.15.44:0/2867664429 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2809154294 0 0) 0x7fc1980cf5e0 con 0x7fc19806c7f0 2024-04-02T13:26:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.368+0000 7fc195ffb640 1 -- 172.21.15.44:0/2867664429 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc180003520 con 0x7fc19806c7f0 2024-04-02T13:26:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.369+0000 7fc195ffb640 1 -- 172.21.15.44:0/2867664429 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2992799832 0 0) 0x7fc19815e810 con 0x7fc19815e430 2024-04-02T13:26:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.369+0000 7fc195ffb640 1 -- 172.21.15.44:0/2867664429 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1980cf5e0 con 0x7fc19815e430 2024-04-02T13:26:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.369+0000 7fc195ffb640 1 -- 172.21.15.44:0/2867664429 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2681542675 0 0) 0x7fc180003520 con 0x7fc19806c7f0 2024-04-02T13:26:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.369+0000 7fc1977fe640 1 -- 172.21.15.44:0/2867664429 >> v1:172.21.15.69:6789/0 conn(0x7fc19815e430 legacy=0x7fc1980ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.369+0000 7fc1977fe640 1 -- 172.21.15.44:0/2867664429 >> v1:172.21.15.44:6789/0 conn(0x7fc19806c7f0 legacy=0x7fc1980ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.369+0000 7fc1977fe640 1 -- 172.21.15.44:0/2867664429 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19806d240 con 0x7fc198166f50 2024-04-02T13:26:36.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.370+0000 7fc1977fe640 1 --2- 172.21.15.44:0/2867664429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc198166f50 0x7fc198169340 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc188009dc0 tx=0x7fc18802bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5508745ae7fd9120 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:36.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.370+0000 7fc195ffb640 1 -- 172.21.15.44:0/2867664429 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc188039070 con 0x7fc198166f50 2024-04-02T13:26:36.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.370+0000 7fc195ffb640 1 -- 172.21.15.44:0/2867664429 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc18802c880 con 0x7fc198166f50 2024-04-02T13:26:36.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.370+0000 7fc195ffb640 1 -- 172.21.15.44:0/2867664429 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c001670 con 0x7fc198166f50 2024-04-02T13:26:36.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.370+0000 7fc19d2d9640 1 -- 172.21.15.44:0/2867664429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc198166f50 msgr2=0x7fc198169340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:36.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.370+0000 7fc19d2d9640 1 --2- 172.21.15.44:0/2867664429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc198166f50 0x7fc198169340 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc188009dc0 tx=0x7fc18802bd50 comp rx=0 tx=0).stop 2024-04-02T13:26:36.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.370+0000 7fc19d2d9640 1 -- 172.21.15.44:0/2867664429 shutdown_connections 2024-04-02T13:26:36.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.370+0000 7fc19d2d9640 1 --2- 172.21.15.44:0/2867664429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc198166f50 0x7fc198169340 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.370+0000 7fc19d2d9640 1 -- 172.21.15.44:0/2867664429 >> 172.21.15.44:0/2867664429 conn(0x7fc198073850 msgr2=0x7fc19809aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:36.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.370+0000 7fc19d2d9640 1 -- 172.21.15.44:0/2867664429 shutdown_connections 2024-04-02T13:26:36.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.370+0000 7fc19d2d9640 1 -- 172.21.15.44:0/2867664429 wait complete. 2024-04-02T13:26:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.371+0000 7fc19d2d9640 1 Processor -- start 2024-04-02T13:26:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.372+0000 7fc19d2d9640 1 -- start start 2024-04-02T13:26:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.372+0000 7fc19d2d9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc19806c7f0 0x7fc198159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.372+0000 7fc19d2d9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc19815e430 0x7fc198159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.372+0000 7fc19d2d9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198166f50 0x7fc19815a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.372+0000 7fc19d2d9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc1980d0c10 con 0x7fc19815e430 2024-04-02T13:26:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.372+0000 7fc19d2d9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc19806cff0 con 0x7fc198166f50 2024-04-02T13:26:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.372+0000 7fc19d2d9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc1980d0ef0 con 0x7fc19806c7f0 2024-04-02T13:26:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.372+0000 7fc1967fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc19815e430 0x7fc198159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.372+0000 7fc1967fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc19815e430 0x7fc198159be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52818/0 (socket says 172.21.15.44:52818) 2024-04-02T13:26:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.372+0000 7fc1967fc640 1 -- 172.21.15.44:0/1191778317 learned_addr learned my addr 172.21.15.44:0/1191778317 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc1977fe640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198166f50 0x7fc19815a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:36.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc196ffd640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc19806c7f0 0x7fc198159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:36.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc1967fc640 1 -- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc19806c7f0 msgr2=0x7fc198159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:36.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc1967fc640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc19806c7f0 0x7fc198159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc1967fc640 1 -- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198166f50 msgr2=0x7fc19815a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:36.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc1967fc640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198166f50 0x7fc19815a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:36.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc1967fc640 1 -- 172.21.15.44:0/1191778317 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1980c5de0 con 0x7fc19815e430 2024-04-02T13:26:36.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc196ffd640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc19806c7f0 0x7fc198159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:36.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc1967fc640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc19815e430 0x7fc198159be0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00c750 tx=0x7fc18c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:36.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc17f7fe640 1 -- 172.21.15.44:0/1191778317 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c005ad0 con 0x7fc19815e430 2024-04-02T13:26:36.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc17f7fe640 1 -- 172.21.15.44:0/1191778317 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc18c005c70 con 0x7fc19815e430 2024-04-02T13:26:36.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc19d2d9640 1 -- 172.21.15.44:0/1191778317 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1980c60c0 con 0x7fc19815e430 2024-04-02T13:26:36.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc17f7fe640 1 -- 172.21.15.44:0/1191778317 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c0072b0 con 0x7fc19815e430 2024-04-02T13:26:36.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.373+0000 7fc19d2d9640 1 -- 172.21.15.44:0/1191778317 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1980c6680 con 0x7fc19815e430 2024-04-02T13:26:36.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.374+0000 7fc19d2d9640 1 -- 172.21.15.44:0/1191778317 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc198068c00 con 0x7fc19815e430 2024-04-02T13:26:36.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.374+0000 7fc17f7fe640 1 -- 172.21.15.44:0/1191778317 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc18c007450 con 0x7fc19815e430 2024-04-02T13:26:36.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.375+0000 7fc17f7fe640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc18406c8f0 0x7fc18406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:36.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.375+0000 7fc17f7fe640 1 -- 172.21.15.44:0/1191778317 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5421+0+0 (secure 0 0 0) 0x7fc18c08b930 con 0x7fc19815e430 2024-04-02T13:26:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.375+0000 7fc196ffd640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc18406c8f0 0x7fc18406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.376+0000 7fc196ffd640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc18406c8f0 0x7fc18406edb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc1980712f0 tx=0x7fc190009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:36.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.379+0000 7fc17f7fe640 1 -- 172.21.15.44:0/1191778317 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc18c005e20 con 0x7fc19815e430 2024-04-02T13:26:36.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:36.639+0000 7fc19d2d9640 1 -- 172.21.15.44:0/1191778317 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]} v 0) -- 0x7fc19815be70 con 0x7fc19815e430 2024-04-02T13:26:37.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.630+0000 7fc17f7fe640 1 -- 172.21.15.44:0/1191778317 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]}]=0 v312) ==== 170+0+0 (secure 0 0 0) 0x7fc18c059ea0 con 0x7fc19815e430 2024-04-02T13:26:37.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.633+0000 7fc19d2d9640 1 -- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc18406c8f0 msgr2=0x7fc18406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:37.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.633+0000 7fc19d2d9640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc18406c8f0 0x7fc18406edb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc1980712f0 tx=0x7fc190009040 comp rx=0 tx=0).stop 2024-04-02T13:26:37.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.633+0000 7fc19d2d9640 1 -- 172.21.15.44:0/1191778317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc19815e430 msgr2=0x7fc198159be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.633+0000 7fc19d2d9640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc19815e430 0x7fc198159be0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00c750 tx=0x7fc18c00cc10 comp rx=0 tx=0).stop 2024-04-02T13:26:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.633+0000 7fc19d2d9640 1 -- 172.21.15.44:0/1191778317 shutdown_connections 2024-04-02T13:26:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.633+0000 7fc19d2d9640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc18406c8f0 0x7fc18406edb0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.633+0000 7fc19d2d9640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198166f50 0x7fc19815a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.633+0000 7fc19d2d9640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc19815e430 0x7fc198159be0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.633+0000 7fc19d2d9640 1 --2- 172.21.15.44:0/1191778317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc19806c7f0 0x7fc198159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.633+0000 7fc19d2d9640 1 -- 172.21.15.44:0/1191778317 >> 172.21.15.44:0/1191778317 conn(0x7fc198073850 msgr2=0x7fc1980af3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.633+0000 7fc19d2d9640 1 -- 172.21.15.44:0/1191778317 shutdown_connections 2024-04-02T13:26:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.633+0000 7fc19d2d9640 1 -- 172.21.15.44:0/1191778317 wait complete. 2024-04-02T13:26:37.648 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T13:26:37.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.756+0000 7f0fc277c640 1 Processor -- start 2024-04-02T13:26:37.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.756+0000 7f0fc277c640 1 -- start start 2024-04-02T13:26:37.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.757+0000 7f0fc277c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fbc0b4180 0x7f0fbc168f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:37.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.757+0000 7f0fc277c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fbc0d1630 con 0x7f0fbc15e5d0 2024-04-02T13:26:37.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.757+0000 7f0fc277c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fbc0d1910 con 0x7f0fbc06cff0 2024-04-02T13:26:37.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.757+0000 7f0fc277c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0fbc0d1bf0 con 0x7f0fbc0b4180 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.757+0000 7f0fbb7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0fbc15e5d0 0x7f0fbc0d0d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54070/0 (socket says 172.21.15.44:54070) 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.757+0000 7f0fbb7fe640 1 -- 172.21.15.44:0/2866821688 learned_addr learned my addr 172.21.15.44:0/2866821688 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.757+0000 7f0fc0d19640 1 --2- 172.21.15.44:0/2866821688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fbc0b4180 0x7f0fbc168f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.757+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1704499433 0 0) 0x7f0fbc0d1630 con 0x7f0fbc15e5d0 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.757+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fa4003520 con 0x7f0fbc15e5d0 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1892013926 0 0) 0x7f0fbc0d1910 con 0x7f0fbc06cff0 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fbc0d1630 con 0x7f0fbc06cff0 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1126016408 0 0) 0x7f0fa4003520 con 0x7f0fbc15e5d0 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fbc0b4180 msgr2=0x7f0fbc168f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 --2- 172.21.15.44:0/2866821688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fbc0b4180 0x7f0fbc168f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 >> v1:172.21.15.69:6789/0 conn(0x7f0fbc06cff0 legacy=0x7f0fbc06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fbc0b1540 con 0x7f0fbc15e5d0 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f0fac001670 con 0x7f0fbc15e5d0 2024-04-02T13:26:37.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 >> v1:172.21.15.44:6789/0 conn(0x7f0fbc15e5d0 legacy=0x7f0fbc0d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 --2- 172.21.15.44:0/2866821688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa4003b70 0x7f0fa4003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 --2- 172.21.15.44:0/2866821688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fa4004800 0x7f0fa4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 --2- 172.21.15.44:0/2866821688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fbc0b4180 0x7f0fa4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0fbc0d1bf0 con 0x7f0fa4004800 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa400baf0 con 0x7f0fa4003b70 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0fa400bcf0 con 0x7f0fbc0b4180 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbb7fe640 1 --2- 172.21.15.44:0/2866821688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fa4004800 0x7f0fa4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbbfff640 1 --2- 172.21.15.44:0/2866821688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa4003b70 0x7f0fa4003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fc0d19640 1 --2- 172.21.15.44:0/2866821688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fbc0b4180 0x7f0fa4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbb7fe640 1 -- 172.21.15.44:0/2866821688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fbc0b4180 msgr2=0x7f0fa4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbb7fe640 1 --2- 172.21.15.44:0/2866821688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fbc0b4180 0x7f0fa4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbb7fe640 1 -- 172.21.15.44:0/2866821688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa4003b70 msgr2=0x7f0fa4003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbb7fe640 1 --2- 172.21.15.44:0/2866821688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa4003b70 0x7f0fa4003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.758+0000 7f0fbb7fe640 1 -- 172.21.15.44:0/2866821688 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa4003520 con 0x7f0fa4004800 2024-04-02T13:26:37.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.759+0000 7f0fbb7fe640 1 --2- 172.21.15.44:0/2866821688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fa4004800 0x7f0fa4004c80 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f0fac007fd0 tx=0x7f0fa4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df4fd5da679ce8f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:37.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.759+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac007ca0 con 0x7f0fa4004800 2024-04-02T13:26:37.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.759+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fac030070 con 0x7f0fa4004800 2024-04-02T13:26:37.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.759+0000 7f0fbaffd640 1 -- 172.21.15.44:0/2866821688 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac005d10 con 0x7f0fa4004800 2024-04-02T13:26:37.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.760+0000 7f0fc277c640 1 -- 172.21.15.44:0/2866821688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fa4004800 msgr2=0x7f0fa4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:37.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.760+0000 7f0fc277c640 1 --2- 172.21.15.44:0/2866821688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fa4004800 0x7f0fa4004c80 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f0fac007fd0 tx=0x7f0fa4005830 comp rx=0 tx=0).stop 2024-04-02T13:26:37.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.760+0000 7f0fc0d19640 1 -- 172.21.15.44:0/2866821688 reap_dead start 2024-04-02T13:26:37.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.760+0000 7f0fc277c640 1 -- 172.21.15.44:0/2866821688 shutdown_connections 2024-04-02T13:26:37.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.760+0000 7f0fc277c640 1 -- 172.21.15.44:0/2866821688 >> 172.21.15.44:0/2866821688 conn(0x7f0fbc09aa20 msgr2=0x7f0fbc09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:37.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.760+0000 7f0fc277c640 1 -- 172.21.15.44:0/2866821688 shutdown_connections 2024-04-02T13:26:37.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.760+0000 7f0fc277c640 1 -- 172.21.15.44:0/2866821688 wait complete. 2024-04-02T13:26:37.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.761+0000 7f0fc277c640 1 Processor -- start 2024-04-02T13:26:37.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.761+0000 7f0fc277c640 1 -- start start 2024-04-02T13:26:37.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fc277c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa4004800 0x7f0fbc156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:37.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fc277c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fbc161460 0x7f0fbc1f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:37.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fc277c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fbc1f4690 0x7f0fbc1f4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:37.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fc277c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0fbc0b2ed0 con 0x7f0fbc161460 2024-04-02T13:26:37.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fc277c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0fbc0b2d50 con 0x7f0fa4004800 2024-04-02T13:26:37.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fc277c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0fbc0b3050 con 0x7f0fbc1f4690 2024-04-02T13:26:37.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fbb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fbc161460 0x7f0fbc1f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:37.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fbb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fbc161460 0x7f0fbc1f7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52836/0 (socket says 172.21.15.44:52836) 2024-04-02T13:26:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fbb7fe640 1 -- 172.21.15.44:0/1938896832 learned_addr learned my addr 172.21.15.44:0/1938896832 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fc0d19640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fbc1f4690 0x7f0fbc1f4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fbbfff640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa4004800 0x7f0fbc156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fbb7fe640 1 -- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fbc1f4690 msgr2=0x7f0fbc1f4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fbb7fe640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fbc1f4690 0x7f0fbc1f4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fbb7fe640 1 -- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa4004800 msgr2=0x7f0fbc156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fbb7fe640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa4004800 0x7f0fbc156940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.762+0000 7f0fbb7fe640 1 -- 172.21.15.44:0/1938896832 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fbc1f7c50 con 0x7f0fbc161460 2024-04-02T13:26:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.763+0000 7f0fbb7fe640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fbc161460 0x7f0fbc1f7550 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00c250 tx=0x7f0fac0019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.763+0000 7f0fb8ff9640 1 -- 172.21.15.44:0/1938896832 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac0049e0 con 0x7f0fbc161460 2024-04-02T13:26:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.763+0000 7f0fb8ff9640 1 -- 172.21.15.44:0/1938896832 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fac004b80 con 0x7f0fbc161460 2024-04-02T13:26:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.763+0000 7f0fc277c640 1 -- 172.21.15.44:0/1938896832 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fbc1f7f30 con 0x7f0fbc161460 2024-04-02T13:26:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.763+0000 7f0fb8ff9640 1 -- 172.21.15.44:0/1938896832 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac007070 con 0x7f0fbc161460 2024-04-02T13:26:37.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.763+0000 7f0fc277c640 1 -- 172.21.15.44:0/1938896832 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fbc1f4170 con 0x7f0fbc161460 2024-04-02T13:26:37.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.764+0000 7f0fc277c640 1 -- 172.21.15.44:0/1938896832 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fbc0684f0 con 0x7f0fbc161460 2024-04-02T13:26:37.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.764+0000 7f0fb8ff9640 1 -- 172.21.15.44:0/1938896832 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0fac048020 con 0x7f0fbc161460 2024-04-02T13:26:37.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.765+0000 7f0fb8ff9640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0fa806c920 0x7f0fa806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:37.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.765+0000 7f0fb8ff9640 1 -- 172.21.15.44:0/1938896832 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5421+0+0 (secure 0 0 0) 0x7f0fac0acc50 con 0x7f0fbc161460 2024-04-02T13:26:37.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.765+0000 7f0fbbfff640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0fa806c920 0x7f0fa806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:37.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.766+0000 7f0fbbfff640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0fa806c920 0x7f0fa806ede0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0cf270 tx=0x7f0fb0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:37.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.769+0000 7f0fb8ff9640 1 -- 172.21.15.44:0/1938896832 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0fac0b2020 con 0x7f0fbc161460 2024-04-02T13:26:37.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.952+0000 7f0fc277c640 1 -- 172.21.15.44:0/1938896832 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f0fbc06cdd0 con 0x7f0fbc161460 2024-04-02T13:26:37.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.957+0000 7f0fb8ff9640 1 -- 172.21.15.44:0/1938896832 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7f0fac07b1a0 con 0x7f0fbc161460 2024-04-02T13:26:37.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.959+0000 7f0fc277c640 1 -- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0fa806c920 msgr2=0x7f0fa806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:37.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.959+0000 7f0fc277c640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0fa806c920 0x7f0fa806ede0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0cf270 tx=0x7f0fb0009040 comp rx=0 tx=0).stop 2024-04-02T13:26:37.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.959+0000 7f0fc277c640 1 -- 172.21.15.44:0/1938896832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fbc161460 msgr2=0x7f0fbc1f7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:37.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.959+0000 7f0fc277c640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fbc161460 0x7f0fbc1f7550 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00c250 tx=0x7f0fac0019c0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.959+0000 7f0fc277c640 1 -- 172.21.15.44:0/1938896832 shutdown_connections 2024-04-02T13:26:37.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.959+0000 7f0fc277c640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0fa806c920 0x7f0fa806ede0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.959+0000 7f0fc277c640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fbc1f4690 0x7f0fbc1f4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.959+0000 7f0fc277c640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fbc161460 0x7f0fbc1f7550 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.959+0000 7f0fc277c640 1 --2- 172.21.15.44:0/1938896832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa4004800 0x7f0fbc156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:37.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.959+0000 7f0fc277c640 1 -- 172.21.15.44:0/1938896832 >> 172.21.15.44:0/1938896832 conn(0x7f0fbc09aa20 msgr2=0x7f0fbc06ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:37.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.959+0000 7f0fc277c640 1 -- 172.21.15.44:0/1938896832 shutdown_connections 2024-04-02T13:26:37.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:37.959+0000 7f0fc277c640 1 -- 172.21.15.44:0/1938896832 wait complete. 2024-04-02T13:26:37.972 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:26:37.972 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:26:38.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:26:38.059 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:26:38.059 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:26:38.059 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:26:38.074 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:26:38.074 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:26:38.074 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:26:38.119 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:26:38.119 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:26:38.119 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:26:38.133 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:26:38.134 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:26:38.134 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:26:38.176 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:26:38.176 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T13:26:38.177 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:26:38.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.275+0000 7fc038a0d640 1 Processor -- start 2024-04-02T13:26:38.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.275+0000 7fc038a0d640 1 -- start start 2024-04-02T13:26:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc038a0d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc0340cece0 0x7fc0340cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc038a0d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0340d09f0 0x7fc03406d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc038a0d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc03406ece0 0x7fc03415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc038a0d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc0340cf600 con 0x7fc0340cece0 2024-04-02T13:26:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc038a0d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc03406dc40 con 0x7fc03406ece0 2024-04-02T13:26:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc038a0d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc03406df20 con 0x7fc0340d09f0 2024-04-02T13:26:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc03259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc0340cece0 0x7fc0340cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc03259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc0340cece0 0x7fc0340cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52844/0 (socket says 172.21.15.44:52844) 2024-04-02T13:26:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc03259c640 1 -- 172.21.15.44:0/950506437 learned_addr learned my addr 172.21.15.44:0/950506437 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc032d9d640 1 --2- 172.21.15.44:0/950506437 >> v2:172.21.15.69:3300/0 conn(0x7fc03406ece0 0x7fc03415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc031d9b640 1 --2- 172.21.15.44:0/950506437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0340d09f0 0x7fc03406d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc03259c640 1 -- 172.21.15.44:0/950506437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0340d09f0 msgr2=0x7fc03406d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc03259c640 1 --2- 172.21.15.44:0/950506437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0340d09f0 0x7fc03406d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc03259c640 1 -- 172.21.15.44:0/950506437 >> v2:172.21.15.69:3300/0 conn(0x7fc03406ece0 msgr2=0x7fc03415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc03259c640 1 --2- 172.21.15.44:0/950506437 >> v2:172.21.15.69:3300/0 conn(0x7fc03406ece0 0x7fc03415eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.276+0000 7fc03259c640 1 -- 172.21.15.44:0/950506437 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03406e200 con 0x7fc0340cece0 2024-04-02T13:26:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.277+0000 7fc03259c640 1 --2- 172.21.15.44:0/950506437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0340cece0 0x7fc0340cf0c0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fc0340cf7e0 tx=0x7fc02c02e0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2362a8b1b91b8c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:38.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.277+0000 7fc03159a640 1 -- 172.21.15.44:0/950506437 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c03b070 con 0x7fc0340cece0 2024-04-02T13:26:38.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.277+0000 7fc03159a640 1 -- 172.21.15.44:0/950506437 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc02c036440 con 0x7fc0340cece0 2024-04-02T13:26:38.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.277+0000 7fc03159a640 1 -- 172.21.15.44:0/950506437 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c02ee60 con 0x7fc0340cece0 2024-04-02T13:26:38.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.278+0000 7fc038a0d640 1 -- 172.21.15.44:0/950506437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0340cece0 msgr2=0x7fc0340cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:38.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.278+0000 7fc038a0d640 1 --2- 172.21.15.44:0/950506437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0340cece0 0x7fc0340cf0c0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fc0340cf7e0 tx=0x7fc02c02e0b0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.278+0000 7fc038a0d640 1 -- 172.21.15.44:0/950506437 shutdown_connections 2024-04-02T13:26:38.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.278+0000 7fc038a0d640 1 --2- 172.21.15.44:0/950506437 >> v2:172.21.15.69:3300/0 conn(0x7fc03406ece0 0x7fc03415eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.278+0000 7fc038a0d640 1 --2- 172.21.15.44:0/950506437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0340d09f0 0x7fc03406d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.278+0000 7fc038a0d640 1 --2- 172.21.15.44:0/950506437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0340cece0 0x7fc0340cf0c0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.278+0000 7fc038a0d640 1 -- 172.21.15.44:0/950506437 >> 172.21.15.44:0/950506437 conn(0x7fc0340cfcd0 msgr2=0x7fc0340d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:38.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.278+0000 7fc038a0d640 1 -- 172.21.15.44:0/950506437 shutdown_connections 2024-04-02T13:26:38.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.278+0000 7fc038a0d640 1 -- 172.21.15.44:0/950506437 wait complete. 2024-04-02T13:26:38.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.279+0000 7fc038a0d640 1 Processor -- start 2024-04-02T13:26:38.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.279+0000 7fc038a0d640 1 -- start start 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.280+0000 7fc038a0d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03406ece0 0x7fc0341495e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.280+0000 7fc038a0d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0340cece0 0x7fc0340b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.280+0000 7fc038a0d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0340d09f0 0x7fc0340b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.280+0000 7fc038a0d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc03406d810 con 0x7fc0340d09f0 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.280+0000 7fc038a0d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc03406e7c0 con 0x7fc03406ece0 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.280+0000 7fc038a0d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc03406d990 con 0x7fc0340cece0 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.280+0000 7fc032d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0340d09f0 0x7fc0340b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.280+0000 7fc032d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0340d09f0 0x7fc0340b0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52846/0 (socket says 172.21.15.44:52846) 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.280+0000 7fc032d9d640 1 -- 172.21.15.44:0/2242415579 learned_addr learned my addr 172.21.15.44:0/2242415579 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.281+0000 7fc03259c640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03406ece0 0x7fc0341495e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.281+0000 7fc031d9b640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0340cece0 0x7fc0340b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.281+0000 7fc032d9d640 1 -- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0340cece0 msgr2=0x7fc0340b0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.281+0000 7fc032d9d640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0340cece0 0x7fc0340b0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.281+0000 7fc032d9d640 1 -- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03406ece0 msgr2=0x7fc0341495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.281+0000 7fc032d9d640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03406ece0 0x7fc0341495e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.281+0000 7fc032d9d640 1 -- 172.21.15.44:0/2242415579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03409d7d0 con 0x7fc0340d09f0 2024-04-02T13:26:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.281+0000 7fc032d9d640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0340d09f0 0x7fc0340b0f60 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fc024007e30 tx=0x7fc02400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.281+0000 7fc02affd640 1 -- 172.21.15.44:0/2242415579 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc024014420 con 0x7fc0340d09f0 2024-04-02T13:26:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.281+0000 7fc02affd640 1 -- 172.21.15.44:0/2242415579 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc024019070 con 0x7fc0340d09f0 2024-04-02T13:26:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.281+0000 7fc038a0d640 1 -- 172.21.15.44:0/2242415579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc03409c730 con 0x7fc0340d09f0 2024-04-02T13:26:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.281+0000 7fc02affd640 1 -- 172.21.15.44:0/2242415579 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc024014800 con 0x7fc0340d09f0 2024-04-02T13:26:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.281+0000 7fc038a0d640 1 -- 172.21.15.44:0/2242415579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc03409ccf0 con 0x7fc0340d09f0 2024-04-02T13:26:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.282+0000 7fc038a0d640 1 -- 172.21.15.44:0/2242415579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0340684f0 con 0x7fc0340d09f0 2024-04-02T13:26:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.282+0000 7fc02affd640 1 -- 172.21.15.44:0/2242415579 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc0240149a0 con 0x7fc0340d09f0 2024-04-02T13:26:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.283+0000 7fc02affd640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc01406c8f0 0x7fc01406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.283+0000 7fc02affd640 1 -- 172.21.15.44:0/2242415579 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5421+0+0 (secure 0 0 0) 0x7fc02408c250 con 0x7fc0340d09f0 2024-04-02T13:26:38.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.283+0000 7fc03259c640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc01406c8f0 0x7fc01406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:38.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.283+0000 7fc03259c640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc01406c8f0 0x7fc01406edb0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc02c038000 tx=0x7fc02c032000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:38.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.287+0000 7fc02affd640 1 -- 172.21.15.44:0/2242415579 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc02405a7a0 con 0x7fc0340d09f0 2024-04-02T13:26:38.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.509+0000 7fc038a0d640 1 -- 172.21.15.44:0/2242415579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc03406c260 con 0x7fc0340d09f0 2024-04-02T13:26:38.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.509+0000 7fc02affd640 1 -- 172.21.15.44:0/2242415579 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+3869 (secure 0 0 0) 0x7fc02405a0a0 con 0x7fc0340d09f0 2024-04-02T13:26:38.512 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:38.512 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":54,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3662927745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3662927745},{"type":"v1","addr":"172.21.15.44:6835","nonce":3662927745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6394,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1053037366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1053037366},{"type":"v1","addr":"172.21.15.69:6836","nonce":1053037366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3893800047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3893800047},{"type":"v1","addr":"172.21.15.69:6837","nonce":3893800047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":7728,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/2423996048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2423996048},{"type":"v1","addr":"172.21.15.44:6833","nonce":2423996048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[]} 2024-04-02T13:26:38.512 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 54 2024-04-02T13:26:38.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.511+0000 7fc038a0d640 1 -- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc01406c8f0 msgr2=0x7fc01406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:38.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.511+0000 7fc038a0d640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc01406c8f0 0x7fc01406edb0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc02c038000 tx=0x7fc02c032000 comp rx=0 tx=0).stop 2024-04-02T13:26:38.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.511+0000 7fc038a0d640 1 -- 172.21.15.44:0/2242415579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0340d09f0 msgr2=0x7fc0340b0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:38.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.511+0000 7fc038a0d640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0340d09f0 0x7fc0340b0f60 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fc024007e30 tx=0x7fc02400e4b0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.511+0000 7fc038a0d640 1 -- 172.21.15.44:0/2242415579 shutdown_connections 2024-04-02T13:26:38.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.511+0000 7fc038a0d640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc01406c8f0 0x7fc01406edb0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.511+0000 7fc038a0d640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0340d09f0 0x7fc0340b0f60 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.511+0000 7fc038a0d640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0340cece0 0x7fc0340b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.511+0000 7fc038a0d640 1 --2- 172.21.15.44:0/2242415579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03406ece0 0x7fc0341495e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.511+0000 7fc038a0d640 1 -- 172.21.15.44:0/2242415579 >> 172.21.15.44:0/2242415579 conn(0x7fc0340cfcd0 msgr2=0x7fc03406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:38.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.511+0000 7fc038a0d640 1 -- 172.21.15.44:0/2242415579 shutdown_connections 2024-04-02T13:26:38.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.511+0000 7fc038a0d640 1 -- 172.21.15.44:0/2242415579 wait complete. 2024-04-02T13:26:38.525 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T13:26:38.525 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T13:26:38.525 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T13:26:38.525 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:26:38.525 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T13:26:38.525 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T13:26:38.525 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T13:26:38.526 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:26:38.526 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T13:26:38.526 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T13:26:38.526 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T13:26:38.526 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:26:38.526 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T13:26:38.526 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T13:26:38.527 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T13:26:38.527 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:26:38.534 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T13:26:38.534 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T13:26:38.535 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T13:26:38.535 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T13:26:38.538 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T13:26:38.538 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T13:26:38.538 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T13:26:38.539 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T13:26:38.577 INFO:tasks.ceph.mds.a:Started 2024-04-02T13:26:38.577 DEBUG:teuthology.parallel:result is None 2024-04-02T13:26:38.578 INFO:tasks.ceph.mds.c:Started 2024-04-02T13:26:38.578 DEBUG:teuthology.parallel:result is None 2024-04-02T13:26:38.581 INFO:tasks.ceph.mds.b:Started 2024-04-02T13:26:38.581 DEBUG:teuthology.parallel:result is None 2024-04-02T13:26:38.582 INFO:tasks.ceph.mds.d:Started 2024-04-02T13:26:38.582 DEBUG:teuthology.parallel:result is None 2024-04-02T13:26:38.582 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T13:26:38.653 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:26:38.654 INFO:tasks.ceph.mds.c.smithi044.stdout:starting mds.c at 2024-04-02T13:26:38.666 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-04-02T13:26:38.666 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-04-02T13:26:38.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.688+0000 7f834e704640 1 Processor -- start 2024-04-02T13:26:38.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.688+0000 7f834e704640 1 -- start start 2024-04-02T13:26:38.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.689+0000 7f834e704640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f83480d0a30 0x7f834806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:38.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.689+0000 7f834e704640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83480d0e10 0x7f83480cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:38.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.689+0000 7f834e704640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f834806c1d0 con 0x7f83480cfe30 2024-04-02T13:26:38.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.689+0000 7f834e704640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f834806cb00 con 0x7f83480d0a30 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.689+0000 7f834e704640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f834806c4b0 con 0x7f83480d0e10 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.689+0000 7f834cca1640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f83480cfe30 0x7f83480d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54080/0 (socket says 172.21.15.44:54080) 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.689+0000 7f834cca1640 1 -- 172.21.15.44:0/1469377029 learned_addr learned my addr 172.21.15.44:0/1469377029 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.689+0000 7f8347fff640 1 --2- 172.21.15.44:0/1469377029 >> v2:172.21.15.69:3300/0 conn(0x7f83480d0a30 0x7f834806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.689+0000 7f83477fe640 1 --2- 172.21.15.44:0/1469377029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83480d0e10 0x7f83480cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.689+0000 7f8346ffd640 1 -- 172.21.15.44:0/1469377029 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3512186515 0 0) 0x7f834806c1d0 con 0x7f83480cfe30 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.689+0000 7f8346ffd640 1 -- 172.21.15.44:0/1469377029 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8330003520 con 0x7f83480cfe30 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.690+0000 7f8347fff640 1 -- 172.21.15.44:0/1469377029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83480d0e10 msgr2=0x7f83480cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.690+0000 7f8347fff640 1 --2- 172.21.15.44:0/1469377029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83480d0e10 0x7f83480cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.690+0000 7f8346ffd640 1 -- 172.21.15.44:0/1469377029 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 474811847 0 0) 0x7f8330003520 con 0x7f83480cfe30 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.690+0000 7f8347fff640 1 -- 172.21.15.44:0/1469377029 >> v1:172.21.15.44:6789/0 conn(0x7f83480cfe30 legacy=0x7f83480d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.690+0000 7f8347fff640 1 -- 172.21.15.44:0/1469377029 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834806c790 con 0x7f83480d0a30 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.690+0000 7f8347fff640 1 --2- 172.21.15.44:0/1469377029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83480d0a30 0x7f834806cf90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f833c0097a0 tx=0x7f833c02b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1edf5d143932c6bf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:38.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.691+0000 7f8346ffd640 1 -- 172.21.15.44:0/1469377029 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c037070 con 0x7f83480d0a30 2024-04-02T13:26:38.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.691+0000 7f8346ffd640 1 -- 172.21.15.44:0/1469377029 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833c030510 con 0x7f83480d0a30 2024-04-02T13:26:38.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.691+0000 7f8346ffd640 1 -- 172.21.15.44:0/1469377029 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83400019f0 con 0x7f83480d0a30 2024-04-02T13:26:38.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.691+0000 7f834e704640 1 -- 172.21.15.44:0/1469377029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83480d0a30 msgr2=0x7f834806cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:38.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.691+0000 7f834e704640 1 --2- 172.21.15.44:0/1469377029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83480d0a30 0x7f834806cf90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f833c0097a0 tx=0x7f833c02b710 comp rx=0 tx=0).stop 2024-04-02T13:26:38.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.691+0000 7f834e704640 1 -- 172.21.15.44:0/1469377029 shutdown_connections 2024-04-02T13:26:38.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.691+0000 7f834e704640 1 --2- 172.21.15.44:0/1469377029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83480d0e10 0x7f83480cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.691+0000 7f834e704640 1 --2- 172.21.15.44:0/1469377029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83480d0a30 0x7f834806cf90 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.691+0000 7f834e704640 1 -- 172.21.15.44:0/1469377029 >> 172.21.15.44:0/1469377029 conn(0x7f834809d470 msgr2=0x7f83480d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:38.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.691+0000 7f834e704640 1 -- 172.21.15.44:0/1469377029 shutdown_connections 2024-04-02T13:26:38.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.691+0000 7f834e704640 1 -- 172.21.15.44:0/1469377029 wait complete. 2024-04-02T13:26:38.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.692+0000 7f834e704640 1 Processor -- start 2024-04-02T13:26:38.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.692+0000 7f834e704640 1 -- start start 2024-04-02T13:26:38.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.693+0000 7f834e704640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83480cfe30 0x7f834815fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:38.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.693+0000 7f834e704640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83480d0a30 0x7f83480cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:38.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.693+0000 7f834e704640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83480d0e10 0x7f83480cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:38.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.693+0000 7f834e704640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f834815e870 con 0x7f83480d0a30 2024-04-02T13:26:38.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.693+0000 7f834e704640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f834815e6f0 con 0x7f83480cfe30 2024-04-02T13:26:38.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.693+0000 7f834e704640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f83480d4da0 con 0x7f83480d0e10 2024-04-02T13:26:38.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.693+0000 7f83477fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83480d0a30 0x7f83480cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:38.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.693+0000 7f83477fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83480d0a30 0x7f83480cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52884/0 (socket says 172.21.15.44:52884) 2024-04-02T13:26:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.693+0000 7f83477fe640 1 -- 172.21.15.44:0/1703802087 learned_addr learned my addr 172.21.15.44:0/1703802087 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.693+0000 7f8347fff640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83480cfe30 0x7f834815fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.693+0000 7f834cca1640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83480d0e10 0x7f83480cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.694+0000 7f83477fe640 1 -- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83480d0e10 msgr2=0x7f83480cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.694+0000 7f83477fe640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83480d0e10 0x7f83480cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.694+0000 7f83477fe640 1 -- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83480cfe30 msgr2=0x7f834815fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.694+0000 7f83477fe640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83480cfe30 0x7f834815fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.694+0000 7f83477fe640 1 -- 172.21.15.44:0/1703802087 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83480ce130 con 0x7f83480d0a30 2024-04-02T13:26:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.694+0000 7f83477fe640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83480d0a30 0x7f83480cd6b0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f833800e840 tx=0x7f833800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.694+0000 7f8344ff9640 1 -- 172.21.15.44:0/1703802087 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338017070 con 0x7f83480d0a30 2024-04-02T13:26:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.694+0000 7f8344ff9640 1 -- 172.21.15.44:0/1703802087 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833800c8e0 con 0x7f83480d0a30 2024-04-02T13:26:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.694+0000 7f834e704640 1 -- 172.21.15.44:0/1703802087 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83480b58e0 con 0x7f83480d0a30 2024-04-02T13:26:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.694+0000 7f8344ff9640 1 -- 172.21.15.44:0/1703802087 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833800ccb0 con 0x7f83480d0a30 2024-04-02T13:26:38.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.694+0000 7f834e704640 1 -- 172.21.15.44:0/1703802087 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83480b5ea0 con 0x7f83480d0a30 2024-04-02T13:26:38.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.695+0000 7f834e704640 1 -- 172.21.15.44:0/1703802087 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83480b6660 con 0x7f83480d0a30 2024-04-02T13:26:38.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.695+0000 7f8344ff9640 1 -- 172.21.15.44:0/1703802087 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f833801a070 con 0x7f83480d0a30 2024-04-02T13:26:38.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.696+0000 7f8344ff9640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f833406c8f0 0x7f833406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:38.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.696+0000 7f8344ff9640 1 -- 172.21.15.44:0/1703802087 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5421+0+0 (secure 0 0 0) 0x7f833808b810 con 0x7f83480d0a30 2024-04-02T13:26:38.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.696+0000 7f8347fff640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f833406c8f0 0x7f833406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:38.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.697+0000 7f8347fff640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f833406c8f0 0x7f833406edb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f833c034000 tx=0x7f833c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:38.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.700+0000 7f8344ff9640 1 -- 172.21.15.44:0/1703802087 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f83480b6660 con 0x7f83480d0a30 2024-04-02T13:26:38.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:38.896+0000 7f834e704640 1 -- 172.21.15.44:0/1703802087 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f83480d17c0 con 0x7f83480d0a30 2024-04-02T13:26:39.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.643+0000 7f8344ff9640 1 -- 172.21.15.44:0/1703802087 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v85) ==== 97+0+0 (secure 0 0 0) 0x7f8338059710 con 0x7f83480d0a30 2024-04-02T13:26:39.646 INFO:teuthology.orchestra.run.smithi044.stderr: removed all blocklist entries 2024-04-02T13:26:39.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.646+0000 7f834e704640 1 -- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f833406c8f0 msgr2=0x7f833406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:39.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.646+0000 7f834e704640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f833406c8f0 0x7f833406edb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f833c034000 tx=0x7f833c008030 comp rx=0 tx=0).stop 2024-04-02T13:26:39.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.646+0000 7f834e704640 1 -- 172.21.15.44:0/1703802087 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83480d0a30 msgr2=0x7f83480cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:39.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.646+0000 7f834e704640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83480d0a30 0x7f83480cd6b0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f833800e840 tx=0x7f833800ed00 comp rx=0 tx=0).stop 2024-04-02T13:26:39.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.646+0000 7f834e704640 1 -- 172.21.15.44:0/1703802087 shutdown_connections 2024-04-02T13:26:39.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.646+0000 7f834e704640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f833406c8f0 0x7f833406edb0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:39.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.646+0000 7f834e704640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83480d0e10 0x7f83480cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:39.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.646+0000 7f834e704640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83480d0a30 0x7f83480cd6b0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:39.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.646+0000 7f834e704640 1 --2- 172.21.15.44:0/1703802087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83480cfe30 0x7f834815fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:39.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.646+0000 7f834e704640 1 -- 172.21.15.44:0/1703802087 >> 172.21.15.44:0/1703802087 conn(0x7f834809d470 msgr2=0x7f83480b1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:39.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.646+0000 7f834e704640 1 -- 172.21.15.44:0/1703802087 shutdown_connections 2024-04-02T13:26:39.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.646+0000 7f834e704640 1 -- 172.21.15.44:0/1703802087 wait complete. 2024-04-02T13:26:39.663 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T13:26:39.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.768+0000 7f161cd9d640 1 Processor -- start 2024-04-02T13:26:39.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.768+0000 7f161cd9d640 1 -- start start 2024-04-02T13:26:39.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f161cd9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16180d28f0 0x7f16180d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:39.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f161cd9d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16180d4d70 con 0x7f16180b47b0 2024-04-02T13:26:39.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f161cd9d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16180d2030 con 0x7f16180b5200 2024-04-02T13:26:39.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f161cd9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f16180d5050 con 0x7f16180d28f0 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f1616d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f16180b47b0 0x7f16180b7120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54094/0 (socket says 172.21.15.44:54094) 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f1616d9d640 1 -- 172.21.15.44:0/1552675124 learned_addr learned my addr 172.21.15.44:0/1552675124 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f161659c640 1 --2- 172.21.15.44:0/1552675124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16180d28f0 0x7f16180d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 901813075 0 0) 0x7f16180d4d70 con 0x7f16180b47b0 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1604003520 con 0x7f16180b47b0 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2985261851 0 0) 0x7f16180d2030 con 0x7f16180b5200 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16180d4d70 con 0x7f16180b5200 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1230216482 0 0) 0x7f1604003520 con 0x7f16180b47b0 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16180d28f0 msgr2=0x7f16180d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f1615d9b640 1 --2- 172.21.15.44:0/1552675124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16180d28f0 0x7f16180d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 >> v1:172.21.15.69:6789/0 conn(0x7f16180b5200 legacy=0x7f16180b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16180b4cb0 con 0x7f16180b47b0 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.769+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f16100019f0 con 0x7f16180b47b0 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 >> v1:172.21.15.44:6789/0 conn(0x7f16180b47b0 legacy=0x7f16180b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f1615d9b640 1 --2- 172.21.15.44:0/1552675124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1604003b70 0x7f1604003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f1615d9b640 1 --2- 172.21.15.44:0/1552675124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1604004800 0x7f1604004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f1615d9b640 1 --2- 172.21.15.44:0/1552675124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16180d28f0 0x7f1604005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f16180d5050 con 0x7f1604004800 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f160400bb80 con 0x7f1604003b70 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f160400bd80 con 0x7f16180d28f0 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f160ffff640 1 --2- 172.21.15.44:0/1552675124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1604004800 0x7f1604004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f161659c640 1 --2- 172.21.15.44:0/1552675124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1604003b70 0x7f1604003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f1616d9d640 1 --2- 172.21.15.44:0/1552675124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16180d28f0 0x7f1604005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f160ffff640 1 -- 172.21.15.44:0/1552675124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16180d28f0 msgr2=0x7f1604005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f160ffff640 1 --2- 172.21.15.44:0/1552675124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16180d28f0 0x7f1604005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f160ffff640 1 -- 172.21.15.44:0/1552675124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1604003b70 msgr2=0x7f1604003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f160ffff640 1 --2- 172.21.15.44:0/1552675124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1604003b70 0x7f1604003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.770+0000 7f160ffff640 1 -- 172.21.15.44:0/1552675124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1604003520 con 0x7f1604004800 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.771+0000 7f160ffff640 1 --2- 172.21.15.44:0/1552675124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1604004800 0x7f1604004c80 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f16000018d0 tx=0x7f1604005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ac409b7cdca692 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.771+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1600037070 con 0x7f1604004800 2024-04-02T13:26:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.771+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16000052d0 con 0x7f1604004800 2024-04-02T13:26:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.771+0000 7f1615d9b640 1 -- 172.21.15.44:0/1552675124 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16000014f0 con 0x7f1604004800 2024-04-02T13:26:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.771+0000 7f161cd9d640 1 -- 172.21.15.44:0/1552675124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1604004800 msgr2=0x7f1604004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.771+0000 7f161cd9d640 1 --2- 172.21.15.44:0/1552675124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1604004800 0x7f1604004c80 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f16000018d0 tx=0x7f1604005830 comp rx=0 tx=0).stop 2024-04-02T13:26:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.771+0000 7f1616d9d640 1 -- 172.21.15.44:0/1552675124 reap_dead start 2024-04-02T13:26:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.771+0000 7f161cd9d640 1 -- 172.21.15.44:0/1552675124 shutdown_connections 2024-04-02T13:26:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.771+0000 7f161cd9d640 1 -- 172.21.15.44:0/1552675124 >> 172.21.15.44:0/1552675124 conn(0x7f16180d4120 msgr2=0x7f16180d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.772+0000 7f161cd9d640 1 -- 172.21.15.44:0/1552675124 shutdown_connections 2024-04-02T13:26:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.772+0000 7f161cd9d640 1 -- 172.21.15.44:0/1552675124 wait complete. 2024-04-02T13:26:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.773+0000 7f161cd9d640 1 Processor -- start 2024-04-02T13:26:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.773+0000 7f161cd9d640 1 -- start start 2024-04-02T13:26:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f161cd9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1604004800 0x7f161815c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f161cd9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f161806d0c0 0x7f161806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f161cd9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16180b23d0 0x7f16180b2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f161cd9d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f16180d23a0 con 0x7f161806d0c0 2024-04-02T13:26:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f161cd9d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f16180d1530 con 0x7f16180b23d0 2024-04-02T13:26:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f161cd9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f16180b5020 con 0x7f1604004800 2024-04-02T13:26:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f160ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f161806d0c0 0x7f161806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f160ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f161806d0c0 0x7f161806d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52900/0 (socket says 172.21.15.44:52900) 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f160ffff640 1 -- 172.21.15.44:0/3770463017 learned_addr learned my addr 172.21.15.44:0/3770463017 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f1616d9d640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16180b23d0 0x7f16180b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f161659c640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1604004800 0x7f161815c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f160ffff640 1 -- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1604004800 msgr2=0x7f161815c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f160ffff640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1604004800 0x7f161815c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f160ffff640 1 -- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16180b23d0 msgr2=0x7f16180b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f160ffff640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16180b23d0 0x7f16180b2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f160ffff640 1 -- 172.21.15.44:0/3770463017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16181f3ee0 con 0x7f161806d0c0 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f160ffff640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f161806d0c0 0x7f161806d540 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f1600005040 tx=0x7f16000012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f160effd640 1 -- 172.21.15.44:0/3770463017 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1600037070 con 0x7f161806d0c0 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f160effd640 1 -- 172.21.15.44:0/3770463017 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1600038040 con 0x7f161806d0c0 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f161cd9d640 1 -- 172.21.15.44:0/3770463017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16181f41c0 con 0x7f161806d0c0 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f160effd640 1 -- 172.21.15.44:0/3770463017 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16000014f0 con 0x7f161806d0c0 2024-04-02T13:26:39.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.774+0000 7f161cd9d640 1 -- 172.21.15.44:0/3770463017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16181f4780 con 0x7f161806d0c0 2024-04-02T13:26:39.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.775+0000 7f161cd9d640 1 -- 172.21.15.44:0/3770463017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16181f4b60 con 0x7f161806d0c0 2024-04-02T13:26:39.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.776+0000 7f160effd640 1 -- 172.21.15.44:0/3770463017 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f16000052d0 con 0x7f161806d0c0 2024-04-02T13:26:39.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.776+0000 7f160effd640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f15f806c8f0 0x7f15f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:39.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.776+0000 7f160effd640 1 -- 172.21.15.44:0/3770463017 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5249+0+0 (secure 0 0 0) 0x7f16000ace40 con 0x7f161806d0c0 2024-04-02T13:26:39.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.777+0000 7f161659c640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f15f806c8f0 0x7f15f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:39.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.777+0000 7f161659c640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f15f806c8f0 0x7f15f806edb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f16180b9ed0 tx=0x7f1608008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:39.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:39.781+0000 7f160effd640 1 -- 172.21.15.44:0/3770463017 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f16181f4b60 con 0x7f161806d0c0 2024-04-02T13:26:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.035+0000 7f161cd9d640 1 -- 172.21.15.44:0/3770463017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f16180684f0 con 0x7f161806d0c0 2024-04-02T13:26:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.036+0000 7f160effd640 1 -- 172.21.15.44:0/3770463017 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) ==== 80+0+5304 (secure 0 0 0) 0x7f160007ade0 con 0x7f161806d0c0 2024-04-02T13:26:40.038 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:40.038 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:26:40.038 INFO:teuthology.orchestra.run.smithi044.stdout: "auth_dump": [ 2024-04-02T13:26:40.038 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.039 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.a", 2024-04-02T13:26:40.039 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmVhZrMhAAklqx5uKCFA5kiqYwyLBKpg==", 2024-04-02T13:26:40.039 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.039 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:26:40.039 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:26:40.039 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:26:40.039 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.039 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.039 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.039 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.039 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.b", 2024-04-02T13:26:40.039 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmalQ+AxAA2SqH9CX/66OIXR7SDID97w==", 2024-04-02T13:26:40.039 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.c", 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxm8BLKORAAjMIrmdQ/bXbPk2sAnYJWTw==", 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.040 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.d", 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmNoaRChAAEqxWmBjBWG79Tkj6dU8FQg==", 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.0", 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDfBQxm4GP9KxAA/AmOunAfVaczAWRGVXvFVg==", 2024-04-02T13:26:40.041 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.042 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:26:40.042 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:26:40.042 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.042 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.042 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.042 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.042 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.1", 2024-04-02T13:26:40.042 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDiBQxmDL/nKRAADF2C30+IGAXL9XwDQDKCUA==", 2024-04-02T13:26:40.042 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.042 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:26:40.042 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:26:40.042 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.2", 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDlBQxm1gOCKBAAuYxN9jNHuRYEqc8WrsYErQ==", 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.3", 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDoBQxmLLFQJxAAFMCzphGU7TOt4nC0Nt+y+g==", 2024-04-02T13:26:40.043 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.4", 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDuBQxmtk8QLhAAR4gm0irR6C3D2SkhFjvDKQ==", 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.044 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.5", 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDxBQxm6Ee/LBAAX5h+9ML70ztw2QH/50OzOg==", 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.6", 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD0BQxm35hwKxAAvsgMmiXdLo+c7wlvb9RtVQ==", 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:26:40.045 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.7", 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD3BQxmYLBOKhAA1vuSOBDcFf2zT78eoTHtHQ==", 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.0", 2024-04-02T13:26:40.046 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A==", 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow r", 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.1", 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxm3sYsGBAAeRCNbP5qiQ8YnRVsDoaaFQ==", 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.2", 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmKNS5FBAAKvsCNmuG+fI1D/vwu6MC5w==", 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.3", 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmslsSHhAAPsAfALW7RimJDyXdOa2wbQ==", 2024-04-02T13:26:40.048 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.admin", 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmq+VcFRAA9CgeH75Q0sXf27cZdJthJg==", 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.049 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.x", 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmR5ICLxAAtQeg/gq0OAq6e0mroBZVxQ==", 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.y", 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmxoquKhAAE5fMnJIuCy5hGJ1aEAx8Vg==", 2024-04-02T13:26:40.050 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:26:40.051 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:26:40.051 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:26:40.051 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:26:40.051 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.051 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.051 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-02T13:26:40.051 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:26:40.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.037+0000 7f161cd9d640 1 -- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f15f806c8f0 msgr2=0x7f15f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.037+0000 7f161cd9d640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f15f806c8f0 0x7f15f806edb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f16180b9ed0 tx=0x7f1608008040 comp rx=0 tx=0).stop 2024-04-02T13:26:40.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.037+0000 7f161cd9d640 1 -- 172.21.15.44:0/3770463017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f161806d0c0 msgr2=0x7f161806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.037+0000 7f161cd9d640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f161806d0c0 0x7f161806d540 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f1600005040 tx=0x7f16000012d0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.037+0000 7f161cd9d640 1 -- 172.21.15.44:0/3770463017 shutdown_connections 2024-04-02T13:26:40.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.037+0000 7f161cd9d640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f15f806c8f0 0x7f15f806edb0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.037+0000 7f161cd9d640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16180b23d0 0x7f16180b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.037+0000 7f161cd9d640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f161806d0c0 0x7f161806d540 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.037+0000 7f161cd9d640 1 --2- 172.21.15.44:0/3770463017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1604004800 0x7f161815c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.037+0000 7f161cd9d640 1 -- 172.21.15.44:0/3770463017 >> 172.21.15.44:0/3770463017 conn(0x7f16180d4120 msgr2=0x7f16180b7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:40.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.038+0000 7f161cd9d640 1 -- 172.21.15.44:0/3770463017 shutdown_connections 2024-04-02T13:26:40.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.038+0000 7f161cd9d640 1 -- 172.21.15.44:0/3770463017 wait complete. 2024-04-02T13:26:40.054 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:26:40.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.153+0000 7f894f31c640 1 Processor -- start 2024-04-02T13:26:40.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.154+0000 7f894f31c640 1 -- start start 2024-04-02T13:26:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.154+0000 7f894f31c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89480d0a30 0x7f894806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.154+0000 7f894f31c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f89480d0e10 0x7f89480cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.154+0000 7f894f31c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f894806c1d0 con 0x7f89480cfe30 2024-04-02T13:26:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.154+0000 7f894f31c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f894806cb00 con 0x7f89480d0e10 2024-04-02T13:26:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.154+0000 7f894f31c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f894806c4b0 con 0x7f89480d0a30 2024-04-02T13:26:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.154+0000 7f894c8b7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f89480d0e10 0x7f89480cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.154+0000 7f894d8b9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f89480cfe30 0x7f89480d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54100/0 (socket says 172.21.15.44:54100) 2024-04-02T13:26:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.154+0000 7f894d8b9640 1 -- 172.21.15.44:0/516097003 learned_addr learned my addr 172.21.15.44:0/516097003 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.155+0000 7f894d0b8640 1 --2- 172.21.15.44:0/516097003 >> v2:172.21.15.69:3300/0 conn(0x7f89480d0a30 0x7f894806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.155+0000 7f894c8b7640 1 -- 172.21.15.44:0/516097003 >> v2:172.21.15.69:3300/0 conn(0x7f89480d0a30 msgr2=0x7f894806cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.155+0000 7f894c8b7640 1 --2- 172.21.15.44:0/516097003 >> v2:172.21.15.69:3300/0 conn(0x7f89480d0a30 0x7f894806cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.155+0000 7f894c8b7640 1 -- 172.21.15.44:0/516097003 >> v1:172.21.15.44:6789/0 conn(0x7f89480cfe30 legacy=0x7f89480d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.155+0000 7f893ffff640 1 -- 172.21.15.44:0/516097003 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 248281955 0 0) 0x7f894806c1d0 con 0x7f89480cfe30 2024-04-02T13:26:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.155+0000 7f894c8b7640 1 -- 172.21.15.44:0/516097003 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f894806c790 con 0x7f89480d0e10 2024-04-02T13:26:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.156+0000 7f894c8b7640 1 --2- 172.21.15.44:0/516097003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89480d0e10 0x7f89480cf8f0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f894806de20 tx=0x7f893802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cca9e1081aa9d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.156+0000 7f893ffff640 1 -- 172.21.15.44:0/516097003 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89380315a0 con 0x7f89480d0e10 2024-04-02T13:26:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.156+0000 7f893ffff640 1 -- 172.21.15.44:0/516097003 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8938037070 con 0x7f89480d0e10 2024-04-02T13:26:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.156+0000 7f893ffff640 1 -- 172.21.15.44:0/516097003 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8938031960 con 0x7f89480d0e10 2024-04-02T13:26:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.156+0000 7f894f31c640 1 -- 172.21.15.44:0/516097003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89480d0e10 msgr2=0x7f89480cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.156+0000 7f894f31c640 1 --2- 172.21.15.44:0/516097003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89480d0e10 0x7f89480cf8f0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f894806de20 tx=0x7f893802b840 comp rx=0 tx=0).stop 2024-04-02T13:26:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.156+0000 7f894f31c640 1 -- 172.21.15.44:0/516097003 shutdown_connections 2024-04-02T13:26:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.156+0000 7f894f31c640 1 --2- 172.21.15.44:0/516097003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89480d0e10 0x7f89480cf8f0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.156+0000 7f894f31c640 1 --2- 172.21.15.44:0/516097003 >> v2:172.21.15.69:3300/0 conn(0x7f89480d0a30 0x7f894806cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.156+0000 7f894f31c640 1 -- 172.21.15.44:0/516097003 >> 172.21.15.44:0/516097003 conn(0x7f894809d470 msgr2=0x7f89480d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.156+0000 7f894f31c640 1 -- 172.21.15.44:0/516097003 shutdown_connections 2024-04-02T13:26:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.156+0000 7f894f31c640 1 -- 172.21.15.44:0/516097003 wait complete. 2024-04-02T13:26:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.157+0000 7f894f31c640 1 Processor -- start 2024-04-02T13:26:40.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.157+0000 7f894f31c640 1 -- start start 2024-04-02T13:26:40.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.158+0000 7f894f31c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89480cfe30 0x7f89480b6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.158+0000 7f894f31c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89480d0a30 0x7f89481601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.158+0000 7f894f31c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89480d0e10 0x7f89480b6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.158+0000 7f894f31c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f894815e870 con 0x7f89480cfe30 2024-04-02T13:26:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.158+0000 7f894f31c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f894815e6f0 con 0x7f89480d0a30 2024-04-02T13:26:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.158+0000 7f894f31c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f89480d4da0 con 0x7f89480d0e10 2024-04-02T13:26:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.158+0000 7f894d0b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89480cfe30 0x7f89480b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.158+0000 7f894d0b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89480cfe30 0x7f89480b6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52926/0 (socket says 172.21.15.44:52926) 2024-04-02T13:26:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.158+0000 7f894d0b8640 1 -- 172.21.15.44:0/3034044027 learned_addr learned my addr 172.21.15.44:0/3034044027 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.158+0000 7f894c8b7640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89480d0a30 0x7f89481601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.159+0000 7f894d8b9640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89480d0e10 0x7f89480b6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.159+0000 7f894d0b8640 1 -- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89480d0e10 msgr2=0x7f89480b6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.159+0000 7f894d0b8640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89480d0e10 0x7f89480b6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.159+0000 7f894d0b8640 1 -- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89480d0a30 msgr2=0x7f89481601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.159+0000 7f894d0b8640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89480d0a30 0x7f89481601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.159+0000 7f894d0b8640 1 -- 172.21.15.44:0/3034044027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89480c90d0 con 0x7f89480cfe30 2024-04-02T13:26:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.159+0000 7f894c8b7640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89480d0a30 0x7f89481601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.159+0000 7f894d0b8640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89480cfe30 0x7f89480b6560 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f894000e870 tx=0x7f894000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.159+0000 7f893dffb640 1 -- 172.21.15.44:0/3034044027 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940017070 con 0x7f89480cfe30 2024-04-02T13:26:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.159+0000 7f893dffb640 1 -- 172.21.15.44:0/3034044027 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f894000c940 con 0x7f89480cfe30 2024-04-02T13:26:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.159+0000 7f894f31c640 1 -- 172.21.15.44:0/3034044027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89480c93b0 con 0x7f89480cfe30 2024-04-02T13:26:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.159+0000 7f893dffb640 1 -- 172.21.15.44:0/3034044027 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894000cd30 con 0x7f89480cfe30 2024-04-02T13:26:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.159+0000 7f894f31c640 1 -- 172.21.15.44:0/3034044027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89480c9c50 con 0x7f89480cfe30 2024-04-02T13:26:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.160+0000 7f894f31c640 1 -- 172.21.15.44:0/3034044027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89480684f0 con 0x7f89480cfe30 2024-04-02T13:26:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.160+0000 7f893dffb640 1 -- 172.21.15.44:0/3034044027 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f894001a070 con 0x7f89480cfe30 2024-04-02T13:26:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.161+0000 7f893dffb640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f892c06c920 0x7f892c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.161+0000 7f893dffb640 1 -- 172.21.15.44:0/3034044027 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5249+0+0 (secure 0 0 0) 0x7f894008bb90 con 0x7f89480cfe30 2024-04-02T13:26:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.161+0000 7f894c8b7640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f892c06c920 0x7f892c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.162+0000 7f894c8b7640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f892c06c920 0x7f892c06ede0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8938036000 tx=0x7f8938008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:40.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.165+0000 7f893dffb640 1 -- 172.21.15.44:0/3034044027 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f894005a180 con 0x7f89480cfe30 2024-04-02T13:26:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.350+0000 7f894f31c640 1 -- 172.21.15.44:0/3034044027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89480d17c0 con 0x7f89480cfe30 2024-04-02T13:26:40.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.351+0000 7f893dffb640 1 -- 172.21.15.44:0/3034044027 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+13117 (secure 0 0 0) 0x7f8940059a80 con 0x7f89480cfe30 2024-04-02T13:26:40.353 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:40.353 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":87,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:26:39.650868+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:26:40.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.353+0000 7f894f31c640 1 -- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f892c06c920 msgr2=0x7f892c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.353+0000 7f894f31c640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f892c06c920 0x7f892c06ede0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8938036000 tx=0x7f8938008030 comp rx=0 tx=0).stop 2024-04-02T13:26:40.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.353+0000 7f894f31c640 1 -- 172.21.15.44:0/3034044027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89480cfe30 msgr2=0x7f89480b6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.353+0000 7f894f31c640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89480cfe30 0x7f89480b6560 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f894000e870 tx=0x7f894000ed30 comp rx=0 tx=0).stop 2024-04-02T13:26:40.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.353+0000 7f894f31c640 1 -- 172.21.15.44:0/3034044027 shutdown_connections 2024-04-02T13:26:40.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.353+0000 7f894f31c640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f892c06c920 0x7f892c06ede0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.353+0000 7f894f31c640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89480d0e10 0x7f89480b6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.353+0000 7f894f31c640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89480d0a30 0x7f89481601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.353+0000 7f894f31c640 1 --2- 172.21.15.44:0/3034044027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89480cfe30 0x7f89480b6560 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.353+0000 7f894f31c640 1 -- 172.21.15.44:0/3034044027 >> 172.21.15.44:0/3034044027 conn(0x7f894809d470 msgr2=0x7f894815cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:40.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.353+0000 7f894f31c640 1 -- 172.21.15.44:0/3034044027 shutdown_connections 2024-04-02T13:26:40.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.353+0000 7f894f31c640 1 -- 172.21.15.44:0/3034044027 wait complete. 2024-04-02T13:26:40.366 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:26:40.367 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:26:40.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.469+0000 7fe9574c7640 1 Processor -- start 2024-04-02T13:26:40.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.470+0000 7fe9574c7640 1 -- start start 2024-04-02T13:26:40.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.470+0000 7fe9574c7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe9500721f0 0x7fe9500cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.470+0000 7fe9574c7640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe95006c220 con 0x7fe95015e420 2024-04-02T13:26:40.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.470+0000 7fe9574c7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe95015f7e0 con 0x7fe9500cfc40 2024-04-02T13:26:40.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.470+0000 7fe9574c7640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe9500d00b0 con 0x7fe9500721f0 2024-04-02T13:26:40.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.471+0000 7fe954a62640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe9500721f0 0x7fe9500cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.471+0000 7fe955263640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe95015e420 0x7fe9500ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54108/0 (socket says 172.21.15.44:54108) 2024-04-02T13:26:40.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.471+0000 7fe955263640 1 -- 172.21.15.44:0/2435615431 learned_addr learned my addr 172.21.15.44:0/2435615431 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:40.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.471+0000 7fe954a62640 1 -- 172.21.15.44:0/2435615431 >> v1:172.21.15.69:6789/0 conn(0x7fe9500cfc40 legacy=0x7fe9500cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:40.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.471+0000 7fe954a62640 1 -- 172.21.15.44:0/2435615431 >> v1:172.21.15.44:6789/0 conn(0x7fe95015e420 legacy=0x7fe9500ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.471+0000 7fe954a62640 1 -- 172.21.15.44:0/2435615431 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95015ec60 con 0x7fe9500721f0 2024-04-02T13:26:40.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.472+0000 7fe954a62640 1 --2- 172.21.15.44:0/2435615431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9500721f0 0x7fe9500cf700 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fe9440097a0 tx=0x7fe94402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d3bcf6d2e91898 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:40.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.472+0000 7fe94ffff640 1 -- 172.21.15.44:0/2435615431 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944038070 con 0x7fe9500721f0 2024-04-02T13:26:40.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.472+0000 7fe94ffff640 1 -- 172.21.15.44:0/2435615431 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe944031700 con 0x7fe9500721f0 2024-04-02T13:26:40.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.472+0000 7fe94ffff640 1 -- 172.21.15.44:0/2435615431 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944031ad0 con 0x7fe9500721f0 2024-04-02T13:26:40.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.472+0000 7fe9574c7640 1 -- 172.21.15.44:0/2435615431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9500721f0 msgr2=0x7fe9500cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.472+0000 7fe9574c7640 1 --2- 172.21.15.44:0/2435615431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9500721f0 0x7fe9500cf700 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fe9440097a0 tx=0x7fe94402b7a0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.472+0000 7fe9574c7640 1 -- 172.21.15.44:0/2435615431 shutdown_connections 2024-04-02T13:26:40.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.472+0000 7fe9574c7640 1 --2- 172.21.15.44:0/2435615431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9500721f0 0x7fe9500cf700 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.472+0000 7fe9574c7640 1 -- 172.21.15.44:0/2435615431 >> 172.21.15.44:0/2435615431 conn(0x7fe9500b2a10 msgr2=0x7fe9500b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:40.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.472+0000 7fe9574c7640 1 -- 172.21.15.44:0/2435615431 shutdown_connections 2024-04-02T13:26:40.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.472+0000 7fe9574c7640 1 -- 172.21.15.44:0/2435615431 wait complete. 2024-04-02T13:26:40.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.473+0000 7fe9574c7640 1 Processor -- start 2024-04-02T13:26:40.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.474+0000 7fe9574c7640 1 -- start start 2024-04-02T13:26:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.474+0000 7fe9574c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9500721f0 0x7fe9501539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.474+0000 7fe9574c7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9500cfc40 0x7fe9500b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.474+0000 7fe9574c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe95015e420 0x7fe9500add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.474+0000 7fe9574c7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe95015f220 con 0x7fe9500cfc40 2024-04-02T13:26:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.474+0000 7fe9574c7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe9500cf3b0 con 0x7fe9500721f0 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.474+0000 7fe9574c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe95015f500 con 0x7fe95015e420 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.474+0000 7fe954a62640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9500cfc40 0x7fe9500b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.474+0000 7fe954a62640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9500cfc40 0x7fe9500b4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52938/0 (socket says 172.21.15.44:52938) 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.474+0000 7fe954a62640 1 -- 172.21.15.44:0/2040791920 learned_addr learned my addr 172.21.15.44:0/2040791920 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.474+0000 7fe955263640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9500721f0 0x7fe9501539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.474+0000 7fe955a64640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe95015e420 0x7fe9500add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.475+0000 7fe954a62640 1 -- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe95015e420 msgr2=0x7fe9500add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.475+0000 7fe954a62640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe95015e420 0x7fe9500add60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.475+0000 7fe954a62640 1 -- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9500721f0 msgr2=0x7fe9501539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.475+0000 7fe954a62640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9500721f0 0x7fe9501539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.475+0000 7fe954a62640 1 -- 172.21.15.44:0/2040791920 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9500b5860 con 0x7fe9500cfc40 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.475+0000 7fe954a62640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9500cfc40 0x7fe9500b4e60 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fe9440098c0 tx=0x7fe9440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.475+0000 7fe94dffb640 1 -- 172.21.15.44:0/2040791920 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944042070 con 0x7fe9500cfc40 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.475+0000 7fe94dffb640 1 -- 172.21.15.44:0/2040791920 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe944033a30 con 0x7fe9500cfc40 2024-04-02T13:26:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.475+0000 7fe9574c7640 1 -- 172.21.15.44:0/2040791920 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe950154330 con 0x7fe9500cfc40 2024-04-02T13:26:40.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.475+0000 7fe94dffb640 1 -- 172.21.15.44:0/2040791920 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944038040 con 0x7fe9500cfc40 2024-04-02T13:26:40.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.475+0000 7fe9574c7640 1 -- 172.21.15.44:0/2040791920 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe950153f20 con 0x7fe9500cfc40 2024-04-02T13:26:40.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.476+0000 7fe9574c7640 1 -- 172.21.15.44:0/2040791920 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9500684f0 con 0x7fe9500cfc40 2024-04-02T13:26:40.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.477+0000 7fe94dffb640 1 -- 172.21.15.44:0/2040791920 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe944045070 con 0x7fe9500cfc40 2024-04-02T13:26:40.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.477+0000 7fe94dffb640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe93c06c8f0 0x7fe93c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.477+0000 7fe94dffb640 1 -- 172.21.15.44:0/2040791920 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5249+0+0 (secure 0 0 0) 0x7fe9440ad650 con 0x7fe9500cfc40 2024-04-02T13:26:40.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.477+0000 7fe955263640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe93c06c8f0 0x7fe93c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.478+0000 7fe955263640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe93c06c8f0 0x7fe93c06edb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe9501f7e50 tx=0x7fe948007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:40.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.482+0000 7fe94dffb640 1 -- 172.21.15.44:0/2040791920 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe944030070 con 0x7fe9500cfc40 2024-04-02T13:26:40.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.664+0000 7fe9574c7640 1 -- 172.21.15.44:0/2040791920 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe9500b3770 con 0x7fe9500cfc40 2024-04-02T13:26:40.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.664+0000 7fe94dffb640 1 -- 172.21.15.44:0/2040791920 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fe944033e20 con 0x7fe9500cfc40 2024-04-02T13:26:40.667 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:26:40.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.666+0000 7fe9574c7640 1 -- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe93c06c8f0 msgr2=0x7fe93c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.666+0000 7fe9574c7640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe93c06c8f0 0x7fe93c06edb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe9501f7e50 tx=0x7fe948007e80 comp rx=0 tx=0).stop 2024-04-02T13:26:40.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.666+0000 7fe9574c7640 1 -- 172.21.15.44:0/2040791920 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9500cfc40 msgr2=0x7fe9500b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.666+0000 7fe9574c7640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9500cfc40 0x7fe9500b4e60 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fe9440098c0 tx=0x7fe9440046c0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.666+0000 7fe9574c7640 1 -- 172.21.15.44:0/2040791920 shutdown_connections 2024-04-02T13:26:40.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.666+0000 7fe9574c7640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe93c06c8f0 0x7fe93c06edb0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.666+0000 7fe9574c7640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe95015e420 0x7fe9500add60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.666+0000 7fe9574c7640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9500cfc40 0x7fe9500b4e60 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.666+0000 7fe9574c7640 1 --2- 172.21.15.44:0/2040791920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9500721f0 0x7fe9501539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.666+0000 7fe9574c7640 1 -- 172.21.15.44:0/2040791920 >> 172.21.15.44:0/2040791920 conn(0x7fe9500b2a10 msgr2=0x7fe9500cf530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:40.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.666+0000 7fe9574c7640 1 -- 172.21.15.44:0/2040791920 shutdown_connections 2024-04-02T13:26:40.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.666+0000 7fe9574c7640 1 -- 172.21.15.44:0/2040791920 wait complete. 2024-04-02T13:26:40.680 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T13:26:40.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.779+0000 7f9dffdba640 1 Processor -- start 2024-04-02T13:26:40.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.779+0000 7f9dffdba640 1 -- start start 2024-04-02T13:26:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dffdba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9df80d0a30 0x7f9df806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dffdba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df80d0000 con 0x7f9df80d2040 2024-04-02T13:26:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dffdba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df806c1d0 con 0x7f9df80d0e10 2024-04-02T13:26:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dffdba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9df806cb00 con 0x7f9df80d0a30 2024-04-02T13:26:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfe357640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f9df80d2040 0x7f9df80d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54122/0 (socket says 172.21.15.44:54122) 2024-04-02T13:26:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfe357640 1 -- 172.21.15.44:0/3235948551 learned_addr learned my addr 172.21.15.44:0/3235948551 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfdb56640 1 --2- 172.21.15.44:0/3235948551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9df80d0a30 0x7f9df806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 208143639 0 0) 0x7f9df80d0000 con 0x7f9df80d2040 2024-04-02T13:26:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9de4003520 con 0x7f9df80d2040 2024-04-02T13:26:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3947043741 0 0) 0x7f9de4003520 con 0x7f9df80d2040 2024-04-02T13:26:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9df80d0a30 msgr2=0x7f9df806cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 --2- 172.21.15.44:0/3235948551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9df80d0a30 0x7f9df806cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 >> v1:172.21.15.69:6789/0 conn(0x7f9df80d0e10 legacy=0x7f9df80cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df806c4b0 con 0x7f9df80d2040 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f9df00019f0 con 0x7f9df80d2040 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 >> v1:172.21.15.44:6789/0 conn(0x7f9df80d2040 legacy=0x7f9df80d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 --2- 172.21.15.44:0/3235948551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9df80d0a30 0x7f9de40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 --2- 172.21.15.44:0/3235948551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9de4003f20 0x7f9de4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 --2- 172.21.15.44:0/3235948551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de4004d40 0x7f9de40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9df806cb00 con 0x7f9de4004d40 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9de400bad0 con 0x7f9de4003f20 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.780+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9de400bcd0 con 0x7f9df80d0a30 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.781+0000 7f9dfe357640 1 --2- 172.21.15.44:0/3235948551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de4004d40 0x7f9de40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.781+0000 7f9dfdb56640 1 --2- 172.21.15.44:0/3235948551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9df80d0a30 0x7f9de40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.781+0000 7f9dfd355640 1 --2- 172.21.15.44:0/3235948551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9de4003f20 0x7f9de4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.781+0000 7f9dfe357640 1 -- 172.21.15.44:0/3235948551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9df80d0a30 msgr2=0x7f9de40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.781+0000 7f9dfe357640 1 --2- 172.21.15.44:0/3235948551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9df80d0a30 0x7f9de40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.781+0000 7f9dfe357640 1 -- 172.21.15.44:0/3235948551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9de4003f20 msgr2=0x7f9de4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.781+0000 7f9dfe357640 1 --2- 172.21.15.44:0/3235948551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9de4003f20 0x7f9de4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.781+0000 7f9dfe357640 1 -- 172.21.15.44:0/3235948551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de4003520 con 0x7f9de4004d40 2024-04-02T13:26:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.782+0000 7f9dfe357640 1 --2- 172.21.15.44:0/3235948551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de4004d40 0x7f9de40051c0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f9df000a920 tx=0x7f9de4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c5579b44e29fd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.782+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df000a4c0 con 0x7f9de4004d40 2024-04-02T13:26:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.782+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df0034650 con 0x7f9de4004d40 2024-04-02T13:26:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.782+0000 7f9dfcb54640 1 -- 172.21.15.44:0/3235948551 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0034a80 con 0x7f9de4004d40 2024-04-02T13:26:40.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.782+0000 7f9dffdba640 1 -- 172.21.15.44:0/3235948551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de4004d40 msgr2=0x7f9de40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.782+0000 7f9dffdba640 1 --2- 172.21.15.44:0/3235948551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de4004d40 0x7f9de40051c0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f9df000a920 tx=0x7f9de4005830 comp rx=0 tx=0).stop 2024-04-02T13:26:40.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.782+0000 7f9dfe357640 1 -- 172.21.15.44:0/3235948551 reap_dead start 2024-04-02T13:26:40.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.782+0000 7f9dffdba640 1 -- 172.21.15.44:0/3235948551 shutdown_connections 2024-04-02T13:26:40.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.782+0000 7f9dffdba640 1 -- 172.21.15.44:0/3235948551 >> 172.21.15.44:0/3235948551 conn(0x7f9df809d470 msgr2=0x7f9df80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:40.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.782+0000 7f9dffdba640 1 -- 172.21.15.44:0/3235948551 shutdown_connections 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.782+0000 7f9dffdba640 1 -- 172.21.15.44:0/3235948551 wait complete. 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.784+0000 7f9dffdba640 1 Processor -- start 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.784+0000 7f9dffdba640 1 -- start start 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.784+0000 7f9dffdba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9de4004d40 0x7f9df8152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.784+0000 7f9dffdba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9df815ff50 0x7f9df81603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.784+0000 7f9dffdba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9df80b3aa0 0x7f9df80b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.784+0000 7f9dffdba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9df815e6f0 con 0x7f9df80b3aa0 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.784+0000 7f9dffdba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9df80d1270 con 0x7f9df815ff50 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.784+0000 7f9dffdba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9df815e870 con 0x7f9de4004d40 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.784+0000 7f9dfe357640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9df80b3aa0 0x7f9df80b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.784+0000 7f9dfe357640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9df80b3aa0 0x7f9df80b3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52948/0 (socket says 172.21.15.44:52948) 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.784+0000 7f9dfe357640 1 -- 172.21.15.44:0/2615828247 learned_addr learned my addr 172.21.15.44:0/2615828247 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.785+0000 7f9dfdb56640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9de4004d40 0x7f9df8152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.785+0000 7f9dfd355640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9df815ff50 0x7f9df81603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.785+0000 7f9dfe357640 1 -- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9de4004d40 msgr2=0x7f9df8152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.785+0000 7f9dfe357640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9de4004d40 0x7f9df8152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.785+0000 7f9dfe357640 1 -- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9df815ff50 msgr2=0x7f9df81603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.785+0000 7f9dfe357640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9df815ff50 0x7f9df81603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.785+0000 7f9dfe357640 1 -- 172.21.15.44:0/2615828247 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df80b4790 con 0x7f9df80b3aa0 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.785+0000 7f9dfe357640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9df80b3aa0 0x7f9df80b3f20 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f9df000aa40 tx=0x7f9df0001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.785+0000 7f9dee7fc640 1 -- 172.21.15.44:0/2615828247 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df000a4c0 con 0x7f9df80b3aa0 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.785+0000 7f9dee7fc640 1 -- 172.21.15.44:0/2615828247 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df0044070 con 0x7f9df80b3aa0 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.785+0000 7f9dee7fc640 1 -- 172.21.15.44:0/2615828247 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0034c00 con 0x7f9df80b3aa0 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.785+0000 7f9dffdba640 1 -- 172.21.15.44:0/2615828247 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df80b4a70 con 0x7f9df80b3aa0 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.785+0000 7f9dffdba640 1 -- 172.21.15.44:0/2615828247 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df80b5030 con 0x7f9df80b3aa0 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.786+0000 7f9dffdba640 1 -- 172.21.15.44:0/2615828247 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df80684f0 con 0x7f9df80b3aa0 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.787+0000 7f9dee7fc640 1 -- 172.21.15.44:0/2615828247 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9df0049460 con 0x7f9df80b3aa0 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.787+0000 7f9dee7fc640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9ddc06c920 0x7f9ddc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:40.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.787+0000 7f9dee7fc640 1 -- 172.21.15.44:0/2615828247 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5249+0+0 (secure 0 0 0) 0x7f9df00add50 con 0x7f9df80b3aa0 2024-04-02T13:26:40.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.787+0000 7f9dfdb56640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9ddc06c920 0x7f9ddc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:40.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.788+0000 7f9dfdb56640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9ddc06c920 0x7f9ddc06ede0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f9df815c120 tx=0x7f9df4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:40.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.791+0000 7f9dee7fc640 1 -- 172.21.15.44:0/2615828247 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9df00b1030 con 0x7f9df80b3aa0 2024-04-02T13:26:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.977+0000 7f9dffdba640 1 -- 172.21.15.44:0/2615828247 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f9df80d17c0 con 0x7f9df80b3aa0 2024-04-02T13:26:40.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.977+0000 7f9dee7fc640 1 -- 172.21.15.44:0/2615828247 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v87) ==== 84+0+65 (secure 0 0 0) 0x7f9df007c460 con 0x7f9df80b3aa0 2024-04-02T13:26:40.980 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:40.980 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-04-02T13:26:40.980 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:26:40.981 INFO:teuthology.orchestra.run.smithi044.stdout: "poolnum": 1, 2024-04-02T13:26:40.981 INFO:teuthology.orchestra.run.smithi044.stdout: "poolname": ".mgr" 2024-04-02T13:26:40.981 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:40.981 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-04-02T13:26:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.979+0000 7f9dffdba640 1 -- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9ddc06c920 msgr2=0x7f9ddc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.979+0000 7f9dffdba640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9ddc06c920 0x7f9ddc06ede0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f9df815c120 tx=0x7f9df4008040 comp rx=0 tx=0).stop 2024-04-02T13:26:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.979+0000 7f9dffdba640 1 -- 172.21.15.44:0/2615828247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9df80b3aa0 msgr2=0x7f9df80b3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.979+0000 7f9dffdba640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9df80b3aa0 0x7f9df80b3f20 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f9df000aa40 tx=0x7f9df0001dc0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.979+0000 7f9dffdba640 1 -- 172.21.15.44:0/2615828247 shutdown_connections 2024-04-02T13:26:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.979+0000 7f9dffdba640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9ddc06c920 0x7f9ddc06ede0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.979+0000 7f9dffdba640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9df80b3aa0 0x7f9df80b3f20 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.979+0000 7f9dffdba640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9df815ff50 0x7f9df81603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.979+0000 7f9dffdba640 1 --2- 172.21.15.44:0/2615828247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9de4004d40 0x7f9df8152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.979+0000 7f9dffdba640 1 -- 172.21.15.44:0/2615828247 >> 172.21.15.44:0/2615828247 conn(0x7f9df809d470 msgr2=0x7f9df80b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.979+0000 7f9dffdba640 1 -- 172.21.15.44:0/2615828247 shutdown_connections 2024-04-02T13:26:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:40.979+0000 7f9dffdba640 1 -- 172.21.15.44:0/2615828247 wait complete. 2024-04-02T13:26:40.992 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T13:26:40.993 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T13:26:41.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.095+0000 7f09d15a1640 1 Processor -- start 2024-04-02T13:26:41.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.095+0000 7f09d15a1640 1 -- start start 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.096+0000 7f09d15a1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f09cc0cc630 0x7f09cc0cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.096+0000 7f09d15a1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09cc0d08a0 con 0x7f09cc0d0490 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.096+0000 7f09d15a1640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09cc0ccdc0 con 0x7f09cc15c090 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.096+0000 7f09d15a1640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f09cc0cd0a0 con 0x7f09cc0cc630 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.096+0000 7f09cb7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f09cc0cc630 0x7f09cc0cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.096+0000 7f09d0da0640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f09cc0d0490 0x7f09cc0b1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54132/0 (socket says 172.21.15.44:54132) 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.096+0000 7f09d0da0640 1 -- 172.21.15.44:0/2575987587 learned_addr learned my addr 172.21.15.44:0/2575987587 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.097+0000 7f09cb7fe640 1 -- 172.21.15.44:0/2575987587 >> v1:172.21.15.69:6789/0 conn(0x7f09cc15c090 legacy=0x7f09cc0cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.097+0000 7f09cb7fe640 1 -- 172.21.15.44:0/2575987587 >> v1:172.21.15.44:6789/0 conn(0x7f09cc0d0490 legacy=0x7f09cc0b1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.097+0000 7f09cb7fe640 1 -- 172.21.15.44:0/2575987587 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09cc0cd380 con 0x7f09cc0cc630 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.097+0000 7f09caffd640 1 -- 172.21.15.44:0/2575987587 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1921715663 0 0) 0x7f09c4001490 con 0x7f09cc0d0490 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.097+0000 7f09cb7fe640 1 --2- 172.21.15.44:0/2575987587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09cc0cc630 0x7f09cc0cff50 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f09c00097a0 tx=0x7f09c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6afb6774129df983 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.097+0000 7f09caffd640 1 -- 172.21.15.44:0/2575987587 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0035020 con 0x7f09cc0cc630 2024-04-02T13:26:41.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.098+0000 7f09caffd640 1 -- 172.21.15.44:0/2575987587 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09c0031830 con 0x7f09cc0cc630 2024-04-02T13:26:41.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.098+0000 7f09caffd640 1 -- 172.21.15.44:0/2575987587 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0031c00 con 0x7f09cc0cc630 2024-04-02T13:26:41.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.098+0000 7f09d15a1640 1 -- 172.21.15.44:0/2575987587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09cc0cc630 msgr2=0x7f09cc0cff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:41.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.098+0000 7f09d15a1640 1 --2- 172.21.15.44:0/2575987587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09cc0cc630 0x7f09cc0cff50 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f09c00097a0 tx=0x7f09c002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:26:41.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.098+0000 7f09d15a1640 1 -- 172.21.15.44:0/2575987587 shutdown_connections 2024-04-02T13:26:41.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.098+0000 7f09d15a1640 1 --2- 172.21.15.44:0/2575987587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09cc0cc630 0x7f09cc0cff50 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:41.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.098+0000 7f09d15a1640 1 -- 172.21.15.44:0/2575987587 >> 172.21.15.44:0/2575987587 conn(0x7f09cc073520 msgr2=0x7f09cc074b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:41.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.098+0000 7f09d15a1640 1 -- 172.21.15.44:0/2575987587 shutdown_connections 2024-04-02T13:26:41.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.098+0000 7f09d15a1640 1 -- 172.21.15.44:0/2575987587 wait complete. 2024-04-02T13:26:41.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.099+0000 7f09d15a1640 1 Processor -- start 2024-04-02T13:26:41.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.099+0000 7f09d15a1640 1 -- start start 2024-04-02T13:26:41.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.100+0000 7f09d15a1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09cc0cc630 0x7f09cc15a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:41.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.100+0000 7f09d15a1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09cc0d0490 0x7f09cc0c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:41.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.100+0000 7f09d15a1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09cc15c090 0x7f09cc0cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:41.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.100+0000 7f09d15a1640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc0ccaa0 con 0x7f09cc0cc630 2024-04-02T13:26:41.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.100+0000 7f09d15a1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc0cd940 con 0x7f09cc0d0490 2024-04-02T13:26:41.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.100+0000 7f09d15a1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f09cc15c470 con 0x7f09cc15c090 2024-04-02T13:26:41.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.100+0000 7f09cbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09cc0cc630 0x7f09cc15a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:41.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.100+0000 7f09cbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09cc0cc630 0x7f09cc15a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52976/0 (socket says 172.21.15.44:52976) 2024-04-02T13:26:41.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.100+0000 7f09cbfff640 1 -- 172.21.15.44:0/2864491163 learned_addr learned my addr 172.21.15.44:0/2864491163 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:41.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.100+0000 7f09cb7fe640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09cc0d0490 0x7f09cc0c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:41.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.100+0000 7f09d0da0640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09cc15c090 0x7f09cc0cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:41.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.101+0000 7f09cbfff640 1 -- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09cc15c090 msgr2=0x7f09cc0cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.101+0000 7f09cbfff640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09cc15c090 0x7f09cc0cafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.101+0000 7f09cbfff640 1 -- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09cc0d0490 msgr2=0x7f09cc0c7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.101+0000 7f09cbfff640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09cc0d0490 0x7f09cc0c7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.101+0000 7f09cbfff640 1 -- 172.21.15.44:0/2864491163 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09cc15a6b0 con 0x7f09cc0cc630 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.101+0000 7f09d0da0640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09cc15c090 0x7f09cc0cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.101+0000 7f09cbfff640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09cc0cc630 0x7f09cc15a170 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00c430 tx=0x7f09bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.101+0000 7f09c8ff9640 1 -- 172.21.15.44:0/2864491163 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc011790 con 0x7f09cc0cc630 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.101+0000 7f09c8ff9640 1 -- 172.21.15.44:0/2864491163 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09bc017070 con 0x7f09cc0cc630 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.101+0000 7f09c8ff9640 1 -- 172.21.15.44:0/2864491163 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc011bb0 con 0x7f09cc0cc630 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.101+0000 7f09d15a1640 1 -- 172.21.15.44:0/2864491163 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09cc0c66b0 con 0x7f09cc0cc630 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.101+0000 7f09d15a1640 1 -- 172.21.15.44:0/2864491163 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09cc0c6c70 con 0x7f09cc0cc630 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.102+0000 7f09d15a1640 1 -- 172.21.15.44:0/2864491163 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09cc04f860 con 0x7f09cc0cc630 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.102+0000 7f09c8ff9640 1 -- 172.21.15.44:0/2864491163 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f09bc01c460 con 0x7f09cc0cc630 2024-04-02T13:26:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.103+0000 7f09c8ff9640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09b806c8f0 0x7f09b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:41.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.103+0000 7f09c8ff9640 1 -- 172.21.15.44:0/2864491163 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5249+0+0 (secure 0 0 0) 0x7f09bc08b7c0 con 0x7f09cc0cc630 2024-04-02T13:26:41.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.103+0000 7f09cb7fe640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09b806c8f0 0x7f09b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:41.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.103+0000 7f09cb7fe640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09b806c8f0 0x7f09b806edb0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f09c0036000 tx=0x7f09c003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:41.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.107+0000 7f09c8ff9640 1 -- 172.21.15.44:0/2864491163 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f09bc059da0 con 0x7f09cc0cc630 2024-04-02T13:26:41.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.296+0000 7f09d15a1640 1 -- 172.21.15.44:0/2864491163 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f09cc0ce210 con 0x7f09cc0cc630 2024-04-02T13:26:41.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.647+0000 7f09c8ff9640 1 -- 172.21.15.44:0/2864491163 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v88) ==== 138+0+0 (secure 0 0 0) 0x7f09bc0596a0 con 0x7f09cc0cc630 2024-04-02T13:26:41.649 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' created 2024-04-02T13:26:41.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.649+0000 7f09d15a1640 1 -- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09b806c8f0 msgr2=0x7f09b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:41.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.649+0000 7f09d15a1640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09b806c8f0 0x7f09b806edb0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f09c0036000 tx=0x7f09c003d040 comp rx=0 tx=0).stop 2024-04-02T13:26:41.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.649+0000 7f09d15a1640 1 -- 172.21.15.44:0/2864491163 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09cc0cc630 msgr2=0x7f09cc15a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:41.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.649+0000 7f09d15a1640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09cc0cc630 0x7f09cc15a170 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00c430 tx=0x7f09bc00c8f0 comp rx=0 tx=0).stop 2024-04-02T13:26:41.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.649+0000 7f09d15a1640 1 -- 172.21.15.44:0/2864491163 shutdown_connections 2024-04-02T13:26:41.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.649+0000 7f09d15a1640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f09b806c8f0 0x7f09b806edb0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:41.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.649+0000 7f09d15a1640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09cc15c090 0x7f09cc0cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:41.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.649+0000 7f09d15a1640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09cc0d0490 0x7f09cc0c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:41.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.649+0000 7f09d15a1640 1 --2- 172.21.15.44:0/2864491163 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09cc0cc630 0x7f09cc15a170 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:41.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.649+0000 7f09d15a1640 1 -- 172.21.15.44:0/2864491163 >> 172.21.15.44:0/2864491163 conn(0x7f09cc073520 msgr2=0x7f09cc067230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:41.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.649+0000 7f09d15a1640 1 -- 172.21.15.44:0/2864491163 shutdown_connections 2024-04-02T13:26:41.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.649+0000 7f09d15a1640 1 -- 172.21.15.44:0/2864491163 wait complete. 2024-04-02T13:26:41.667 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T13:26:41.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.774+0000 7fd426a55640 1 Processor -- start 2024-04-02T13:26:41.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.774+0000 7fd426a55640 1 -- start start 2024-04-02T13:26:41.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.775+0000 7fd426a55640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd420071200 0x7fd42006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:41.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.775+0000 7fd426a55640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42006e9c0 con 0x7fd4200cc640 2024-04-02T13:26:41.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.775+0000 7fd426a55640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42006eca0 con 0x7fd42015a940 2024-04-02T13:26:41.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.775+0000 7fd426a55640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd42006ef80 con 0x7fd420071200 2024-04-02T13:26:41.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.775+0000 7fd424ff2640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd4200cc640 0x7fd4200cca50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54144/0 (socket says 172.21.15.44:54144) 2024-04-02T13:26:41.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.775+0000 7fd424ff2640 1 -- 172.21.15.44:0/2783563467 learned_addr learned my addr 172.21.15.44:0/2783563467 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:41.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.776+0000 7fd41f7fe640 1 --2- 172.21.15.44:0/2783563467 >> v2:172.21.15.69:3300/0 conn(0x7fd420071200 0x7fd42006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:41.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.776+0000 7fd41effd640 1 -- 172.21.15.44:0/2783563467 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 946003086 0 0) 0x7fd42006e9c0 con 0x7fd4200cc640 2024-04-02T13:26:41.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.776+0000 7fd41effd640 1 -- 172.21.15.44:0/2783563467 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd408003520 con 0x7fd4200cc640 2024-04-02T13:26:41.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.776+0000 7fd41effd640 1 -- 172.21.15.44:0/2783563467 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4294531628 0 0) 0x7fd42006eca0 con 0x7fd42015a940 2024-04-02T13:26:41.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.776+0000 7fd41effd640 1 -- 172.21.15.44:0/2783563467 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd42006e9c0 con 0x7fd42015a940 2024-04-02T13:26:41.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.776+0000 7fd41f7fe640 1 -- 172.21.15.44:0/2783563467 >> v1:172.21.15.69:6789/0 conn(0x7fd42015a940 legacy=0x7fd4200ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:41.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.776+0000 7fd41f7fe640 1 -- 172.21.15.44:0/2783563467 >> v1:172.21.15.44:6789/0 conn(0x7fd4200cc640 legacy=0x7fd4200cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:41.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.776+0000 7fd41effd640 1 -- 172.21.15.44:0/2783563467 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2901920201 0 0) 0x7fd408003520 con 0x7fd4200cc640 2024-04-02T13:26:41.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.776+0000 7fd41f7fe640 1 -- 172.21.15.44:0/2783563467 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4200cdcb0 con 0x7fd420071200 2024-04-02T13:26:41.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.777+0000 7fd41f7fe640 1 --2- 172.21.15.44:0/2783563467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd420071200 0x7fd42006e0c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd4140097a0 tx=0x7fd41402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b4c9693753254f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:41.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.777+0000 7fd41effd640 1 -- 172.21.15.44:0/2783563467 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd414037070 con 0x7fd420071200 2024-04-02T13:26:41.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.777+0000 7fd41effd640 1 -- 172.21.15.44:0/2783563467 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd414030510 con 0x7fd420071200 2024-04-02T13:26:41.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.777+0000 7fd41effd640 1 -- 172.21.15.44:0/2783563467 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4180019f0 con 0x7fd420071200 2024-04-02T13:26:41.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.778+0000 7fd426a55640 1 -- 172.21.15.44:0/2783563467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd420071200 msgr2=0x7fd42006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:41.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.778+0000 7fd426a55640 1 --2- 172.21.15.44:0/2783563467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd420071200 0x7fd42006e0c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd4140097a0 tx=0x7fd41402b790 comp rx=0 tx=0).stop 2024-04-02T13:26:41.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.778+0000 7fd426a55640 1 -- 172.21.15.44:0/2783563467 shutdown_connections 2024-04-02T13:26:41.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.778+0000 7fd426a55640 1 --2- 172.21.15.44:0/2783563467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd420071200 0x7fd42006e0c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:41.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.778+0000 7fd426a55640 1 -- 172.21.15.44:0/2783563467 >> 172.21.15.44:0/2783563467 conn(0x7fd42006b920 msgr2=0x7fd42006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:41.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.778+0000 7fd426a55640 1 -- 172.21.15.44:0/2783563467 shutdown_connections 2024-04-02T13:26:41.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.778+0000 7fd426a55640 1 -- 172.21.15.44:0/2783563467 wait complete. 2024-04-02T13:26:41.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.779+0000 7fd426a55640 1 Processor -- start 2024-04-02T13:26:41.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.779+0000 7fd426a55640 1 -- start start 2024-04-02T13:26:41.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd426a55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd420071200 0x7fd420157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:41.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd426a55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4200cc640 0x7fd42015bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:41.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd426a55640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd42015a940 0x7fd420158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:41.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd426a55640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd4200d00a0 con 0x7fd42015a940 2024-04-02T13:26:41.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd426a55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd4200ce020 con 0x7fd420071200 2024-04-02T13:26:41.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd426a55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd4200d0220 con 0x7fd4200cc640 2024-04-02T13:26:41.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd424ff2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd42015a940 0x7fd420158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:41.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd424ff2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd42015a940 0x7fd420158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52982/0 (socket says 172.21.15.44:52982) 2024-04-02T13:26:41.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd424ff2640 1 -- 172.21.15.44:0/3718178360 learned_addr learned my addr 172.21.15.44:0/3718178360 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:41.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd41f7fe640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4200cc640 0x7fd42015bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:41.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd41ffff640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd420071200 0x7fd420157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:41.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd424ff2640 1 -- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4200cc640 msgr2=0x7fd42015bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:41.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd424ff2640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4200cc640 0x7fd42015bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:41.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd424ff2640 1 -- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd420071200 msgr2=0x7fd420157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:41.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd424ff2640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd420071200 0x7fd420157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:41.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd424ff2640 1 -- 172.21.15.44:0/3718178360 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42015aea0 con 0x7fd42015a940 2024-04-02T13:26:41.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd41ffff640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd420071200 0x7fd420157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:41.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd424ff2640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd42015a940 0x7fd420158480 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fd418001fb0 tx=0x7fd41800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:41.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd41cff9640 1 -- 172.21.15.44:0/3718178360 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd418017070 con 0x7fd42015a940 2024-04-02T13:26:41.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd41cff9640 1 -- 172.21.15.44:0/3718178360 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd418012400 con 0x7fd42015a940 2024-04-02T13:26:41.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd426a55640 1 -- 172.21.15.44:0/3718178360 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd42015b180 con 0x7fd42015a940 2024-04-02T13:26:41.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.780+0000 7fd41cff9640 1 -- 172.21.15.44:0/3718178360 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd418012830 con 0x7fd42015a940 2024-04-02T13:26:41.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.781+0000 7fd426a55640 1 -- 172.21.15.44:0/3718178360 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd42015c1a0 con 0x7fd42015a940 2024-04-02T13:26:41.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.782+0000 7fd426a55640 1 -- 172.21.15.44:0/3718178360 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4200684f0 con 0x7fd42015a940 2024-04-02T13:26:41.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.782+0000 7fd41cff9640 1 -- 172.21.15.44:0/3718178360 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd41801a070 con 0x7fd42015a940 2024-04-02T13:26:41.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.782+0000 7fd41cff9640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd40c06c8f0 0x7fd40c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:41.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.783+0000 7fd41cff9640 1 -- 172.21.15.44:0/3718178360 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5619+0+0 (secure 0 0 0) 0x7fd41808baf0 con 0x7fd42015a940 2024-04-02T13:26:41.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.783+0000 7fd41ffff640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd40c06c8f0 0x7fd40c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:41.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.783+0000 7fd41ffff640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd40c06c8f0 0x7fd40c06edb0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd4100011e0 tx=0x7fd410009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:41.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.787+0000 7fd41cff9640 1 -- 172.21.15.44:0/3718178360 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd418059f70 con 0x7fd42015a940 2024-04-02T13:26:41.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:41.975+0000 7fd426a55640 1 -- 172.21.15.44:0/3718178360 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd4200ccdd0 con 0x7fd42015a940 2024-04-02T13:26:42.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.649+0000 7fd41cff9640 1 -- 172.21.15.44:0/3718178360 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v89) ==== 170+0+0 (secure 0 0 0) 0x7fd418059870 con 0x7fd42015a940 2024-04-02T13:26:42.652 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' created 2024-04-02T13:26:42.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.652+0000 7fd426a55640 1 -- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd40c06c8f0 msgr2=0x7fd40c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:42.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.652+0000 7fd426a55640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd40c06c8f0 0x7fd40c06edb0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd4100011e0 tx=0x7fd410009040 comp rx=0 tx=0).stop 2024-04-02T13:26:42.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.652+0000 7fd426a55640 1 -- 172.21.15.44:0/3718178360 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd42015a940 msgr2=0x7fd420158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:42.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.652+0000 7fd426a55640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd42015a940 0x7fd420158480 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fd418001fb0 tx=0x7fd41800a430 comp rx=0 tx=0).stop 2024-04-02T13:26:42.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.652+0000 7fd426a55640 1 -- 172.21.15.44:0/3718178360 shutdown_connections 2024-04-02T13:26:42.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.652+0000 7fd426a55640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd40c06c8f0 0x7fd40c06edb0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:42.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.652+0000 7fd426a55640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd42015a940 0x7fd420158480 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:42.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.652+0000 7fd426a55640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4200cc640 0x7fd42015bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:42.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.652+0000 7fd426a55640 1 --2- 172.21.15.44:0/3718178360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd420071200 0x7fd420157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:42.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.652+0000 7fd426a55640 1 -- 172.21.15.44:0/3718178360 >> 172.21.15.44:0/3718178360 conn(0x7fd42006b920 msgr2=0x7fd42006bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:42.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.652+0000 7fd426a55640 1 -- 172.21.15.44:0/3718178360 shutdown_connections 2024-04-02T13:26:42.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.652+0000 7fd426a55640 1 -- 172.21.15.44:0/3718178360 wait complete. 2024-04-02T13:26:42.678 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T13:26:42.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.785+0000 7f27fdaeb640 1 Processor -- start 2024-04-02T13:26:42.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.785+0000 7f27fdaeb640 1 -- start start 2024-04-02T13:26:42.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.786+0000 7f27fdaeb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f27f815e640 0x7f27f80d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:42.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.786+0000 7f27fdaeb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27f80cf120 con 0x7f27f80cec60 2024-04-02T13:26:42.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.786+0000 7f27fdaeb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27f80cf400 con 0x7f27f815e260 2024-04-02T13:26:42.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.786+0000 7f27fdaeb640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f27f80b5a50 con 0x7f27f815e640 2024-04-02T13:26:42.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.786+0000 7f27f6ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f27f815e640 0x7f27f80d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:42.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.786+0000 7f27f7fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f27f80cec60 0x7f27f8168280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54158/0 (socket says 172.21.15.44:54158) 2024-04-02T13:26:42.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.786+0000 7f27f7fff640 1 -- 172.21.15.44:0/4174971357 learned_addr learned my addr 172.21.15.44:0/4174971357 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:42.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.787+0000 7f27f6ffd640 1 -- 172.21.15.44:0/4174971357 >> v1:172.21.15.69:6789/0 conn(0x7f27f815e260 legacy=0x7f27f80d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:42.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.787+0000 7f27f6ffd640 1 -- 172.21.15.44:0/4174971357 >> v1:172.21.15.44:6789/0 conn(0x7f27f80cec60 legacy=0x7f27f8168280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:42.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.787+0000 7f27f6ffd640 1 -- 172.21.15.44:0/4174971357 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f80cf6e0 con 0x7f27f815e640 2024-04-02T13:26:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.787+0000 7f27f6ffd640 1 --2- 172.21.15.44:0/4174971357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27f815e640 0x7f27f80d0ad0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0097a0 tx=0x7f27ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46780f564e8ebfbd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.787+0000 7f27f67fc640 1 -- 172.21.15.44:0/4174971357 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ec035020 con 0x7f27f815e640 2024-04-02T13:26:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.787+0000 7f27f67fc640 1 -- 172.21.15.44:0/4174971357 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27ec031830 con 0x7f27f815e640 2024-04-02T13:26:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.787+0000 7f27f67fc640 1 -- 172.21.15.44:0/4174971357 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ec031c00 con 0x7f27f815e640 2024-04-02T13:26:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.788+0000 7f27fdaeb640 1 -- 172.21.15.44:0/4174971357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27f815e640 msgr2=0x7f27f80d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.788+0000 7f27fdaeb640 1 --2- 172.21.15.44:0/4174971357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27f815e640 0x7f27f80d0ad0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0097a0 tx=0x7f27ec02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:26:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.788+0000 7f27fdaeb640 1 -- 172.21.15.44:0/4174971357 shutdown_connections 2024-04-02T13:26:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.788+0000 7f27fdaeb640 1 --2- 172.21.15.44:0/4174971357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27f815e640 0x7f27f80d0ad0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.788+0000 7f27fdaeb640 1 -- 172.21.15.44:0/4174971357 >> 172.21.15.44:0/4174971357 conn(0x7f27f809d3c0 msgr2=0x7f27f80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.788+0000 7f27fdaeb640 1 -- 172.21.15.44:0/4174971357 shutdown_connections 2024-04-02T13:26:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.788+0000 7f27fdaeb640 1 -- 172.21.15.44:0/4174971357 wait complete. 2024-04-02T13:26:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.789+0000 7f27fdaeb640 1 Processor -- start 2024-04-02T13:26:42.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.789+0000 7f27fdaeb640 1 -- start start 2024-04-02T13:26:42.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27fdaeb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27f80cec60 0x7f27f8155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:42.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27fdaeb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27f815e260 0x7f27f80b6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:42.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27fdaeb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27f815e640 0x7f27f815b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:42.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27fdaeb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f27f80cfca0 con 0x7f27f815e640 2024-04-02T13:26:42.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27fdaeb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f27f80d3070 con 0x7f27f80cec60 2024-04-02T13:26:42.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27fdaeb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f27f80d1010 con 0x7f27f815e260 2024-04-02T13:26:42.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27f7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27f815e640 0x7f27f815b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:42.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27f7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27f815e640 0x7f27f815b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52996/0 (socket says 172.21.15.44:52996) 2024-04-02T13:26:42.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27f7fff640 1 -- 172.21.15.44:0/106003124 learned_addr learned my addr 172.21.15.44:0/106003124 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:42.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27f6ffd640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27f815e260 0x7f27f80b6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:42.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27f77fe640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27f80cec60 0x7f27f8155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:42.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27f7fff640 1 -- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27f815e260 msgr2=0x7f27f80b6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:42.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27f7fff640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27f815e260 0x7f27f80b6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:42.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27f7fff640 1 -- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27f80cec60 msgr2=0x7f27f8155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:42.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27f7fff640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27f80cec60 0x7f27f8155cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:42.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.790+0000 7f27f7fff640 1 -- 172.21.15.44:0/106003124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f80b76e0 con 0x7f27f815e640 2024-04-02T13:26:42.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.791+0000 7f27f7fff640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27f815e640 0x7f27f815b8c0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f27f000cbf0 tx=0x7f27f000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:42.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.791+0000 7f27dffff640 1 -- 172.21.15.44:0/106003124 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f0017070 con 0x7f27f815e640 2024-04-02T13:26:42.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.791+0000 7f27dffff640 1 -- 172.21.15.44:0/106003124 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27f000ad40 con 0x7f27f815e640 2024-04-02T13:26:42.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.791+0000 7f27fdaeb640 1 -- 172.21.15.44:0/106003124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f80b79c0 con 0x7f27f815e640 2024-04-02T13:26:42.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.791+0000 7f27dffff640 1 -- 172.21.15.44:0/106003124 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f0012600 con 0x7f27f815e640 2024-04-02T13:26:42.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.791+0000 7f27fdaeb640 1 -- 172.21.15.44:0/106003124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f815ec30 con 0x7f27f815e640 2024-04-02T13:26:42.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.792+0000 7f27fdaeb640 1 -- 172.21.15.44:0/106003124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f815ea20 con 0x7f27f815e640 2024-04-02T13:26:42.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.792+0000 7f27dffff640 1 -- 172.21.15.44:0/106003124 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f27f001a070 con 0x7f27f815e640 2024-04-02T13:26:42.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.793+0000 7f27dffff640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27e406c8f0 0x7f27e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:42.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.793+0000 7f27dffff640 1 -- 172.21.15.44:0/106003124 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6001+0+0 (secure 0 0 0) 0x7f27f008ba70 con 0x7f27f815e640 2024-04-02T13:26:42.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.793+0000 7f27f77fe640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27e406c8f0 0x7f27e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:42.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.793+0000 7f27f77fe640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27e406c8f0 0x7f27e406edb0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f27f815db70 tx=0x7f27e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:42.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:42.798+0000 7f27dffff640 1 -- 172.21.15.44:0/106003124 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f27f0059600 con 0x7f27f815e640 2024-04-02T13:26:43.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.024+0000 7f27fdaeb640 1 -- 172.21.15.44:0/106003124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f27f80684f0 con 0x7f27f815e640 2024-04-02T13:26:43.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.694+0000 7f27dffff640 1 -- 172.21.15.44:0/106003124 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v57) ==== 174+0+0 (secure 0 0 0) 0x7f27f005d900 con 0x7f27f815e640 2024-04-02T13:26:43.696 INFO:teuthology.orchestra.run.smithi044.stderr:new fs with metadata pool 13 and data pool 14 2024-04-02T13:26:43.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.701+0000 7f27fdaeb640 1 -- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27e406c8f0 msgr2=0x7f27e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:43.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.701+0000 7f27fdaeb640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27e406c8f0 0x7f27e406edb0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f27f815db70 tx=0x7f27e8009040 comp rx=0 tx=0).stop 2024-04-02T13:26:43.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.701+0000 7f27fdaeb640 1 -- 172.21.15.44:0/106003124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27f815e640 msgr2=0x7f27f815b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:43.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.701+0000 7f27fdaeb640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27f815e640 0x7f27f815b8c0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f27f000cbf0 tx=0x7f27f000a430 comp rx=0 tx=0).stop 2024-04-02T13:26:43.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.702+0000 7f27fdaeb640 1 -- 172.21.15.44:0/106003124 shutdown_connections 2024-04-02T13:26:43.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.702+0000 7f27fdaeb640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27e406c8f0 0x7f27e406edb0 secure :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f27f815db70 tx=0x7f27e8009040 comp rx=0 tx=0).stop 2024-04-02T13:26:43.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.702+0000 7f27fdaeb640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27f815e640 0x7f27f815b8c0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:43.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.703+0000 7f27fdaeb640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27f815e260 0x7f27f80b6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:43.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.703+0000 7f27fdaeb640 1 --2- 172.21.15.44:0/106003124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27f80cec60 0x7f27f8155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:43.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.703+0000 7f27fdaeb640 1 -- 172.21.15.44:0/106003124 >> 172.21.15.44:0/106003124 conn(0x7f27f809d3c0 msgr2=0x7f27f806c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:43.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.708+0000 7f27fdaeb640 1 -- 172.21.15.44:0/106003124 shutdown_connections 2024-04-02T13:26:43.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.709+0000 7f27fdaeb640 1 -- 172.21.15.44:0/106003124 wait complete. 2024-04-02T13:26:43.727 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:26:43.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.829+0000 7f9fefdc8640 1 Processor -- start 2024-04-02T13:26:43.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.829+0000 7f9fefdc8640 1 -- start start 2024-04-02T13:26:43.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.830+0000 7f9fefdc8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9fe815e4a0 0x7f9fe80aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:43.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.830+0000 7f9fefdc8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fe80cc600 con 0x7f9fe806d550 2024-04-02T13:26:43.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.830+0000 7f9fefdc8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fe80cc8e0 con 0x7f9fe80d0c50 2024-04-02T13:26:43.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.830+0000 7f9fefdc8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9fe80ccbc0 con 0x7f9fe815e4a0 2024-04-02T13:26:43.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.830+0000 7f9fee365640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f9fe806d550 0x7f9fe80cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54164/0 (socket says 172.21.15.44:54164) 2024-04-02T13:26:43.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.830+0000 7f9fee365640 1 -- 172.21.15.44:0/2771553485 learned_addr learned my addr 172.21.15.44:0/2771553485 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:43.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.830+0000 7f9fedb64640 1 --2- 172.21.15.44:0/2771553485 >> v2:172.21.15.69:3300/0 conn(0x7f9fe815e4a0 0x7f9fe80aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:43.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.830+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845144804 0 0) 0x7f9fe80cc600 con 0x7f9fe806d550 2024-04-02T13:26:43.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.830+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fd0003520 con 0x7f9fe806d550 2024-04-02T13:26:43.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.830+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1841506790 0 0) 0x7f9fe80cc8e0 con 0x7f9fe80d0c50 2024-04-02T13:26:43.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fe80cc600 con 0x7f9fe80d0c50 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 581959612 0 0) 0x7f9fd0003520 con 0x7f9fe806d550 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 >> v2:172.21.15.69:3300/0 conn(0x7f9fe815e4a0 msgr2=0x7f9fe80aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 --2- 172.21.15.44:0/2771553485 >> v2:172.21.15.69:3300/0 conn(0x7f9fe815e4a0 0x7f9fe80aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 >> v1:172.21.15.69:6789/0 conn(0x7f9fe80d0c50 legacy=0x7f9fe806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe80ccea0 con 0x7f9fe806d550 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f9fd80019f0 con 0x7f9fe806d550 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fedb64640 1 --2- 172.21.15.44:0/2771553485 >> v2:172.21.15.69:3300/0 conn(0x7f9fe815e4a0 0x7f9fe80aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 >> v1:172.21.15.44:6789/0 conn(0x7f9fe806d550 legacy=0x7f9fe80cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 --2- 172.21.15.44:0/2771553485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd0003b40 0x7f9fd0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 --2- 172.21.15.44:0/2771553485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fd0004800 0x7f9fd0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 --2- 172.21.15.44:0/2771553485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9fd00051c0 0x7f9fd000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe80ccbc0 con 0x7f9fd0003b40 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd000c040 con 0x7f9fd0004800 2024-04-02T13:26:43.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9fd000c240 con 0x7f9fd00051c0 2024-04-02T13:26:43.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fedb64640 1 --2- 172.21.15.44:0/2771553485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd0003b40 0x7f9fd0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:43.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fed363640 1 --2- 172.21.15.44:0/2771553485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fd0004800 0x7f9fd0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:43.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fee365640 1 --2- 172.21.15.44:0/2771553485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9fd00051c0 0x7f9fd000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:43.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fedb64640 1 -- 172.21.15.44:0/2771553485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9fd00051c0 msgr2=0x7f9fd000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:43.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fedb64640 1 --2- 172.21.15.44:0/2771553485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9fd00051c0 0x7f9fd000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:43.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fedb64640 1 -- 172.21.15.44:0/2771553485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fd0004800 msgr2=0x7f9fd0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:43.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fedb64640 1 --2- 172.21.15.44:0/2771553485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fd0004800 0x7f9fd0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:43.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fedb64640 1 -- 172.21.15.44:0/2771553485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd0003520 con 0x7f9fd0003b40 2024-04-02T13:26:43.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.831+0000 7f9fee365640 1 -- 172.21.15.44:0/2771553485 reap_dead start 2024-04-02T13:26:43.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.832+0000 7f9fedb64640 1 --2- 172.21.15.44:0/2771553485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd0003b40 0x7f9fd0003f40 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0009c90 tx=0x7f9fd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420cc1ea9eec57a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:43.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.832+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0039070 con 0x7f9fd0003b40 2024-04-02T13:26:43.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.832+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fe002c910 con 0x7f9fd0003b40 2024-04-02T13:26:43.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.832+0000 7f9fecb62640 1 -- 172.21.15.44:0/2771553485 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0009860 con 0x7f9fd0003b40 2024-04-02T13:26:43.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.833+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2771553485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd0003b40 msgr2=0x7f9fd0003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:43.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.833+0000 7f9fefdc8640 1 --2- 172.21.15.44:0/2771553485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd0003b40 0x7f9fd0003f40 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0009c90 tx=0x7f9fd0005830 comp rx=0 tx=0).stop 2024-04-02T13:26:43.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.833+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2771553485 shutdown_connections 2024-04-02T13:26:43.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.833+0000 7f9fefdc8640 1 --2- 172.21.15.44:0/2771553485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd0003b40 0x7f9fd0003f40 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:43.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.833+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2771553485 >> 172.21.15.44:0/2771553485 conn(0x7f9fe809aa20 msgr2=0x7f9fe809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:43.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.833+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2771553485 shutdown_connections 2024-04-02T13:26:43.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.833+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2771553485 wait complete. 2024-04-02T13:26:43.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.834+0000 7f9fefdc8640 1 Processor -- start 2024-04-02T13:26:43.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.834+0000 7f9fefdc8640 1 -- start start 2024-04-02T13:26:43.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fefdc8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd0003b40 0x7f9fe80c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:43.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fefdc8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9fe815a2b0 0x7f9fe81512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:43.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fefdc8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fe8161260 0x7f9fe80b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:43.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fefdc8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe815e880 con 0x7f9fd0003b40 2024-04-02T13:26:43.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fefdc8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe80cd460 con 0x7f9fe8161260 2024-04-02T13:26:43.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fefdc8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9fe80d10b0 con 0x7f9fe815a2b0 2024-04-02T13:26:43.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fedb64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd0003b40 0x7f9fe80c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:43.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fedb64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd0003b40 0x7f9fe80c8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53028/0 (socket says 172.21.15.44:53028) 2024-04-02T13:26:43.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fedb64640 1 -- 172.21.15.44:0/2329544442 learned_addr learned my addr 172.21.15.44:0/2329544442 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:43.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fee365640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fe8161260 0x7f9fe80b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:43.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fed363640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9fe815a2b0 0x7f9fe81512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:43.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fedb64640 1 -- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9fe815a2b0 msgr2=0x7f9fe81512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:43.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fedb64640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9fe815a2b0 0x7f9fe81512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:43.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fedb64640 1 -- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fe8161260 msgr2=0x7f9fe80b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:43.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fedb64640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fe8161260 0x7f9fe80b01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:43.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fedb64640 1 -- 172.21.15.44:0/2329544442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe81601c0 con 0x7f9fd0003b40 2024-04-02T13:26:43.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fee365640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fe8161260 0x7f9fe80b01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:43.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fed363640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9fe815a2b0 0x7f9fe81512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:43.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fedb64640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd0003b40 0x7f9fe80c8910 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0037040 tx=0x7f9fe0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:43.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fe67fc640 1 -- 172.21.15.44:0/2329544442 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0039050 con 0x7f9fd0003b40 2024-04-02T13:26:43.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fe67fc640 1 -- 172.21.15.44:0/2329544442 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fe003a040 con 0x7f9fd0003b40 2024-04-02T13:26:43.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fe67fc640 1 -- 172.21.15.44:0/2329544442 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0009860 con 0x7f9fd0003b40 2024-04-02T13:26:43.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2329544442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe81604a0 con 0x7f9fd0003b40 2024-04-02T13:26:43.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.835+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2329544442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe8160a60 con 0x7f9fd0003b40 2024-04-02T13:26:43.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.836+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2329544442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe80684f0 con 0x7f9fd0003b40 2024-04-02T13:26:43.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.837+0000 7f9fe67fc640 1 -- 172.21.15.44:0/2329544442 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9fe0047030 con 0x7f9fd0003b40 2024-04-02T13:26:43.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.837+0000 7f9fe67fc640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9fd406c8f0 0x7f9fd406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:43.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.837+0000 7f9fe67fc640 1 -- 172.21.15.44:0/2329544442 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f9fe00ad2c0 con 0x7f9fd0003b40 2024-04-02T13:26:43.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.837+0000 7f9fed363640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9fd406c8f0 0x7f9fd406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:43.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.838+0000 7f9fed363640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9fd406c8f0 0x7f9fd406edb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc000c00 tx=0x7f9fdc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:43.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:43.842+0000 7f9fe67fc640 1 -- 172.21.15.44:0/2329544442 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9fe007b4d0 con 0x7f9fd0003b40 2024-04-02T13:26:44.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.030+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2329544442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9fe8068d00 con 0x7f9fd0003b40 2024-04-02T13:26:44.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.031+0000 7f9fe67fc640 1 -- 172.21.15.44:0/2329544442 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+16850 (secure 0 0 0) 0x7f9fe00ad0e0 con 0x7f9fd0003b40 2024-04-02T13:26:44.034 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:44.034 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:26:43.025566+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:26:41.300970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-02T13:26:41.979586+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:26:44.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.034+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9fd406c8f0 msgr2=0x7f9fd406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:44.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.034+0000 7f9fefdc8640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9fd406c8f0 0x7f9fd406edb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc000c00 tx=0x7f9fdc009040 comp rx=0 tx=0).stop 2024-04-02T13:26:44.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.034+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2329544442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd0003b40 msgr2=0x7f9fe80c8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:44.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.034+0000 7f9fefdc8640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd0003b40 0x7f9fe80c8910 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0037040 tx=0x7f9fe0004730 comp rx=0 tx=0).stop 2024-04-02T13:26:44.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.034+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2329544442 shutdown_connections 2024-04-02T13:26:44.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.034+0000 7f9fefdc8640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9fd406c8f0 0x7f9fd406edb0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.034+0000 7f9fefdc8640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fe8161260 0x7f9fe80b01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.034+0000 7f9fefdc8640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9fe815a2b0 0x7f9fe81512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.034+0000 7f9fefdc8640 1 --2- 172.21.15.44:0/2329544442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd0003b40 0x7f9fe80c8910 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.034+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2329544442 >> 172.21.15.44:0/2329544442 conn(0x7f9fe809aa20 msgr2=0x7f9fe80cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:44.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.034+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2329544442 shutdown_connections 2024-04-02T13:26:44.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.034+0000 7f9fefdc8640 1 -- 172.21.15.44:0/2329544442 wait complete. 2024-04-02T13:26:44.047 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:26:44.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.146+0000 7fc600115640 1 Processor -- start 2024-04-02T13:26:44.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.146+0000 7fc600115640 1 -- start start 2024-04-02T13:26:44.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc600115640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc5f815e0f0 0x7fc5f80d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:44.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc600115640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc5f80d0630 0x7fc5f806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:44.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc600115640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5f815e4d0 0x7fc5f806ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc600115640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc5f806dfe0 con 0x7fc5f815e0f0 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc600115640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc5f806e2c0 con 0x7fc5f80d0630 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc600115640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc5f806e5a0 con 0x7fc5f815e4d0 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc5fdeb1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc5f815e0f0 0x7fc5f80d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc5fdeb1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc5f815e0f0 0x7fc5f80d00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53042/0 (socket says 172.21.15.44:53042) 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc5fdeb1640 1 -- 172.21.15.44:0/1811668129 learned_addr learned my addr 172.21.15.44:0/1811668129 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc5fd6b0640 1 --2- 172.21.15.44:0/1811668129 >> v2:172.21.15.69:3300/0 conn(0x7fc5f80d0630 0x7fc5f806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc5fe6b2640 1 --2- 172.21.15.44:0/1811668129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5f815e4d0 0x7fc5f806ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc5fdeb1640 1 -- 172.21.15.44:0/1811668129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5f815e4d0 msgr2=0x7fc5f806ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc5fdeb1640 1 --2- 172.21.15.44:0/1811668129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5f815e4d0 0x7fc5f806ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc5fdeb1640 1 -- 172.21.15.44:0/1811668129 >> v2:172.21.15.69:3300/0 conn(0x7fc5f80d0630 msgr2=0x7fc5f806b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc5fdeb1640 1 --2- 172.21.15.44:0/1811668129 >> v2:172.21.15.69:3300/0 conn(0x7fc5f80d0630 0x7fc5f806b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.147+0000 7fc5fdeb1640 1 -- 172.21.15.44:0/1811668129 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f806e880 con 0x7fc5f815e0f0 2024-04-02T13:26:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.148+0000 7fc5fdeb1640 1 --2- 172.21.15.44:0/1811668129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5f815e0f0 0x7fc5f80d00f0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fc5f809ee50 tx=0x7fc5f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5794ebd685e308e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:44.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.148+0000 7fc5fceaf640 1 -- 172.21.15.44:0/1811668129 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0034070 con 0x7fc5f815e0f0 2024-04-02T13:26:44.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.148+0000 7fc5fceaf640 1 -- 172.21.15.44:0/1811668129 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5f0030860 con 0x7fc5f815e0f0 2024-04-02T13:26:44.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.148+0000 7fc5fceaf640 1 -- 172.21.15.44:0/1811668129 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0030c30 con 0x7fc5f815e0f0 2024-04-02T13:26:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.149+0000 7fc600115640 1 -- 172.21.15.44:0/1811668129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5f815e0f0 msgr2=0x7fc5f80d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.149+0000 7fc600115640 1 --2- 172.21.15.44:0/1811668129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5f815e0f0 0x7fc5f80d00f0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fc5f809ee50 tx=0x7fc5f002b9a0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.149+0000 7fc600115640 1 -- 172.21.15.44:0/1811668129 shutdown_connections 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.149+0000 7fc600115640 1 --2- 172.21.15.44:0/1811668129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5f815e4d0 0x7fc5f806ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.149+0000 7fc600115640 1 --2- 172.21.15.44:0/1811668129 >> v2:172.21.15.69:3300/0 conn(0x7fc5f80d0630 0x7fc5f806b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.149+0000 7fc600115640 1 --2- 172.21.15.44:0/1811668129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5f815e0f0 0x7fc5f80d00f0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.149+0000 7fc600115640 1 -- 172.21.15.44:0/1811668129 >> 172.21.15.44:0/1811668129 conn(0x7fc5f80b2980 msgr2=0x7fc5f80b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.149+0000 7fc600115640 1 -- 172.21.15.44:0/1811668129 shutdown_connections 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.149+0000 7fc600115640 1 -- 172.21.15.44:0/1811668129 wait complete. 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.150+0000 7fc600115640 1 Processor -- start 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.150+0000 7fc600115640 1 -- start start 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc600115640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5f80d0630 0x7fc5f80cc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc600115640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5f815e0f0 0x7fc5f8158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc600115640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5f815e4d0 0x7fc5f815b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc600115640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f8072f60 con 0x7fc5f80d0630 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc600115640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f806eda0 con 0x7fc5f815e0f0 2024-04-02T13:26:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc600115640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc5f80730e0 con 0x7fc5f815e4d0 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5fdeb1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5f80d0630 0x7fc5f80cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5fdeb1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5f80d0630 0x7fc5f80cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53056/0 (socket says 172.21.15.44:53056) 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5fdeb1640 1 -- 172.21.15.44:0/3795601765 learned_addr learned my addr 172.21.15.44:0/3795601765 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5fe6b2640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5f815e4d0 0x7fc5f815b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5fd6b0640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5f815e0f0 0x7fc5f8158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5fdeb1640 1 -- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5f815e4d0 msgr2=0x7fc5f815b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5fdeb1640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5f815e4d0 0x7fc5f815b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5fdeb1640 1 -- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5f815e0f0 msgr2=0x7fc5f8158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5fdeb1640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5f815e0f0 0x7fc5f8158d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5fdeb1640 1 -- 172.21.15.44:0/3795601765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f8159b80 con 0x7fc5f80d0630 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5fdeb1640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5f80d0630 0x7fc5f80cc170 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fc5f002b9e0 tx=0x7fc5f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5f67fc640 1 -- 172.21.15.44:0/3795601765 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0048070 con 0x7fc5f80d0630 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5f67fc640 1 -- 172.21.15.44:0/3795601765 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5f0030660 con 0x7fc5f80d0630 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc600115640 1 -- 172.21.15.44:0/3795601765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5f80c9120 con 0x7fc5f80d0630 2024-04-02T13:26:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.151+0000 7fc5f67fc640 1 -- 172.21.15.44:0/3795601765 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0034040 con 0x7fc5f80d0630 2024-04-02T13:26:44.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.152+0000 7fc600115640 1 -- 172.21.15.44:0/3795601765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5f815bf10 con 0x7fc5f80d0630 2024-04-02T13:26:44.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.152+0000 7fc600115640 1 -- 172.21.15.44:0/3795601765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5f80684f0 con 0x7fc5f80d0630 2024-04-02T13:26:44.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.153+0000 7fc5f67fc640 1 -- 172.21.15.44:0/3795601765 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc5f004b070 con 0x7fc5f80d0630 2024-04-02T13:26:44.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.153+0000 7fc5f67fc640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc5e406c8f0 0x7fc5e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:44.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.153+0000 7fc5f67fc640 1 -- 172.21.15.44:0/3795601765 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fc5f00add50 con 0x7fc5f80d0630 2024-04-02T13:26:44.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.154+0000 7fc5fd6b0640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc5e406c8f0 0x7fc5e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:44.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.154+0000 7fc5fd6b0640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc5e406c8f0 0x7fc5e406edb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc5f81f77d0 tx=0x7fc5ec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:44.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.158+0000 7fc5f67fc640 1 -- 172.21.15.44:0/3795601765 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc5f00b1030 con 0x7fc5f80d0630 2024-04-02T13:26:44.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.345+0000 7fc600115640 1 -- 172.21.15.44:0/3795601765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc5f80727f0 con 0x7fc5f80d0630 2024-04-02T13:26:44.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.346+0000 7fc5f67fc640 1 -- 172.21.15.44:0/3795601765 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+16850 (secure 0 0 0) 0x7fc5f007bff0 con 0x7fc5f80d0630 2024-04-02T13:26:44.349 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:44.349 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:26:43.025566+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:26:41.300970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-02T13:26:41.979586+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:26:44.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.348+0000 7fc600115640 1 -- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc5e406c8f0 msgr2=0x7fc5e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:44.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.348+0000 7fc600115640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc5e406c8f0 0x7fc5e406edb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc5f81f77d0 tx=0x7fc5ec008040 comp rx=0 tx=0).stop 2024-04-02T13:26:44.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.348+0000 7fc600115640 1 -- 172.21.15.44:0/3795601765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5f80d0630 msgr2=0x7fc5f80cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:44.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.348+0000 7fc600115640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5f80d0630 0x7fc5f80cc170 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fc5f002b9e0 tx=0x7fc5f0004730 comp rx=0 tx=0).stop 2024-04-02T13:26:44.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.348+0000 7fc600115640 1 -- 172.21.15.44:0/3795601765 shutdown_connections 2024-04-02T13:26:44.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.348+0000 7fc600115640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc5e406c8f0 0x7fc5e406edb0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.348+0000 7fc600115640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5f815e4d0 0x7fc5f815b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.348+0000 7fc600115640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5f815e0f0 0x7fc5f8158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.348+0000 7fc600115640 1 --2- 172.21.15.44:0/3795601765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5f80d0630 0x7fc5f80cc170 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.348+0000 7fc600115640 1 -- 172.21.15.44:0/3795601765 >> 172.21.15.44:0/3795601765 conn(0x7fc5f80b2980 msgr2=0x7fc5f80b0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:44.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.349+0000 7fc600115640 1 -- 172.21.15.44:0/3795601765 shutdown_connections 2024-04-02T13:26:44.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.349+0000 7fc600115640 1 -- 172.21.15.44:0/3795601765 wait complete. 2024-04-02T13:26:44.362 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T13:26:44.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.465+0000 7f84eaf38640 1 Processor -- start 2024-04-02T13:26:44.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.465+0000 7f84eaf38640 1 -- start start 2024-04-02T13:26:44.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.465+0000 7f84eaf38640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f84e415c050 0x7f84e40ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:44.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.465+0000 7f84eaf38640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84e40b1ed0 0x7f84e40ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:44.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.465+0000 7f84eaf38640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e40ccb10 con 0x7f84e415c430 2024-04-02T13:26:44.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.465+0000 7f84eaf38640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f84e40b2480 con 0x7f84e415c050 2024-04-02T13:26:44.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.465+0000 7f84eaf38640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f84e40b2760 con 0x7f84e40b1ed0 2024-04-02T13:26:44.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.466+0000 7f84e8cd4640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f84e415c050 0x7f84e40ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:44.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.466+0000 7f84e8cd4640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f84e415c050 0x7f84e40ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53070/0 (socket says 172.21.15.44:53070) 2024-04-02T13:26:44.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.466+0000 7f84e8cd4640 1 -- 172.21.15.44:0/1943108660 learned_addr learned my addr 172.21.15.44:0/1943108660 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:44.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.466+0000 7f84e94d5640 1 --2- 172.21.15.44:0/1943108660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84e40b1ed0 0x7f84e40ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:44.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.466+0000 7f84e8cd4640 1 -- 172.21.15.44:0/1943108660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84e40b1ed0 msgr2=0x7f84e40ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:44.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.466+0000 7f84e8cd4640 1 --2- 172.21.15.44:0/1943108660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84e40b1ed0 0x7f84e40ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.466+0000 7f84e8cd4640 1 -- 172.21.15.44:0/1943108660 >> v1:172.21.15.69:6789/0 conn(0x7f84e415c430 legacy=0x7f84e40b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:44.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.466+0000 7f84e8cd4640 1 -- 172.21.15.44:0/1943108660 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e40b2a40 con 0x7f84e415c050 2024-04-02T13:26:44.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.467+0000 7f84e8cd4640 1 --2- 172.21.15.44:0/1943108660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e415c050 0x7f84e40ce850 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009870 tx=0x7f84d802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e05c1315faad69 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:44.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.467+0000 7f84e37fe640 1 -- 172.21.15.44:0/1943108660 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d802f840 con 0x7f84e415c050 2024-04-02T13:26:44.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.467+0000 7f84e37fe640 1 -- 172.21.15.44:0/1943108660 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84d802f9e0 con 0x7f84e415c050 2024-04-02T13:26:44.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.467+0000 7f84e37fe640 1 -- 172.21.15.44:0/1943108660 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d802fdb0 con 0x7f84e415c050 2024-04-02T13:26:44.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.467+0000 7f84eaf38640 1 -- 172.21.15.44:0/1943108660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e415c050 msgr2=0x7f84e40ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:44.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.467+0000 7f84eaf38640 1 --2- 172.21.15.44:0/1943108660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e415c050 0x7f84e40ce850 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009870 tx=0x7f84d802ba20 comp rx=0 tx=0).stop 2024-04-02T13:26:44.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.467+0000 7f84eaf38640 1 -- 172.21.15.44:0/1943108660 shutdown_connections 2024-04-02T13:26:44.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.467+0000 7f84eaf38640 1 --2- 172.21.15.44:0/1943108660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84e40b1ed0 0x7f84e40ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.467+0000 7f84eaf38640 1 --2- 172.21.15.44:0/1943108660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e415c050 0x7f84e40ce850 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.467+0000 7f84eaf38640 1 -- 172.21.15.44:0/1943108660 >> 172.21.15.44:0/1943108660 conn(0x7f84e409aa20 msgr2=0x7f84e409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:44.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.467+0000 7f84eaf38640 1 -- 172.21.15.44:0/1943108660 shutdown_connections 2024-04-02T13:26:44.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.467+0000 7f84eaf38640 1 -- 172.21.15.44:0/1943108660 wait complete. 2024-04-02T13:26:44.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.469+0000 7f84eaf38640 1 Processor -- start 2024-04-02T13:26:44.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.469+0000 7f84eaf38640 1 -- start start 2024-04-02T13:26:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.469+0000 7f84eaf38640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84e40b1ed0 0x7f84e40b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.469+0000 7f84eaf38640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84e415c050 0x7f84e40cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.469+0000 7f84eaf38640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e415c430 0x7f84e40cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84eaf38640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f84e40cd140 con 0x7f84e415c430 2024-04-02T13:26:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84eaf38640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84e40ccfc0 con 0x7f84e415c050 2024-04-02T13:26:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84eaf38640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f84e40cd2c0 con 0x7f84e40b1ed0 2024-04-02T13:26:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e94d5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e415c430 0x7f84e40cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e94d5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e415c430 0x7f84e40cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53072/0 (socket says 172.21.15.44:53072) 2024-04-02T13:26:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e94d5640 1 -- 172.21.15.44:0/3118553489 learned_addr learned my addr 172.21.15.44:0/3118553489 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e3fff640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84e415c050 0x7f84e40cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e8cd4640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84e40b1ed0 0x7f84e40b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e94d5640 1 -- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84e40b1ed0 msgr2=0x7f84e40b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e94d5640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84e40b1ed0 0x7f84e40b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e94d5640 1 -- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84e415c050 msgr2=0x7f84e40cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e94d5640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84e415c050 0x7f84e40cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e94d5640 1 -- 172.21.15.44:0/3118553489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e40b4ca0 con 0x7f84e415c430 2024-04-02T13:26:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e94d5640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e415c430 0x7f84e40cb980 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00efc0 tx=0x7f84dc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e17fa640 1 -- 172.21.15.44:0/3118553489 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc018020 con 0x7f84e415c430 2024-04-02T13:26:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e17fa640 1 -- 172.21.15.44:0/3118553489 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84dc014480 con 0x7f84e415c430 2024-04-02T13:26:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84e17fa640 1 -- 172.21.15.44:0/3118553489 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc014870 con 0x7f84e415c430 2024-04-02T13:26:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84eaf38640 1 -- 172.21.15.44:0/3118553489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e40b4f80 con 0x7f84e415c430 2024-04-02T13:26:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.470+0000 7f84eaf38640 1 -- 172.21.15.44:0/3118553489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e40b5540 con 0x7f84e415c430 2024-04-02T13:26:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.471+0000 7f84eaf38640 1 -- 172.21.15.44:0/3118553489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84e40684f0 con 0x7f84e415c430 2024-04-02T13:26:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.472+0000 7f84e17fa640 1 -- 172.21.15.44:0/3118553489 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f84dc01b020 con 0x7f84e415c430 2024-04-02T13:26:44.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.472+0000 7f84e17fa640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f84c806c8f0 0x7f84c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:44.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.472+0000 7f84e17fa640 1 -- 172.21.15.44:0/3118553489 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f84dc08c8c0 con 0x7f84e415c430 2024-04-02T13:26:44.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.472+0000 7f84e8cd4640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f84c806c8f0 0x7f84c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:44.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.473+0000 7f84e8cd4640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f84c806c8f0 0x7f84c806edb0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f84d8036000 tx=0x7f84d8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:44.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.477+0000 7f84e17fa640 1 -- 172.21.15.44:0/3118553489 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f84dc022030 con 0x7f84e415c430 2024-04-02T13:26:44.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:44.701+0000 7f84eaf38640 1 -- 172.21.15.44:0/3118553489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f84e40b40f0 con 0x7f84e415c430 2024-04-02T13:26:45.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.700+0000 7f84e17fa640 1 -- 172.21.15.44:0/3118553489 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v60) ==== 118+0+0 (secure 0 0 0) 0x7f84dc010040 con 0x7f84e415c430 2024-04-02T13:26:45.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.703+0000 7f84eaf38640 1 -- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f84c806c8f0 msgr2=0x7f84c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:45.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.703+0000 7f84eaf38640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f84c806c8f0 0x7f84c806edb0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f84d8036000 tx=0x7f84d8008030 comp rx=0 tx=0).stop 2024-04-02T13:26:45.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.703+0000 7f84eaf38640 1 -- 172.21.15.44:0/3118553489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e415c430 msgr2=0x7f84e40cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:45.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.703+0000 7f84eaf38640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e415c430 0x7f84e40cb980 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00efc0 tx=0x7f84dc00c490 comp rx=0 tx=0).stop 2024-04-02T13:26:45.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.703+0000 7f84eaf38640 1 -- 172.21.15.44:0/3118553489 shutdown_connections 2024-04-02T13:26:45.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.703+0000 7f84eaf38640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f84c806c8f0 0x7f84c806edb0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:45.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.703+0000 7f84eaf38640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e415c430 0x7f84e40cb980 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:45.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.703+0000 7f84eaf38640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84e415c050 0x7f84e40cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:45.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.703+0000 7f84eaf38640 1 --2- 172.21.15.44:0/3118553489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84e40b1ed0 0x7f84e40b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:45.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.703+0000 7f84eaf38640 1 -- 172.21.15.44:0/3118553489 >> 172.21.15.44:0/3118553489 conn(0x7f84e409aa20 msgr2=0x7f84e40cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:45.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.703+0000 7f84eaf38640 1 -- 172.21.15.44:0/3118553489 shutdown_connections 2024-04-02T13:26:45.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.703+0000 7f84eaf38640 1 -- 172.21.15.44:0/3118553489 wait complete. 2024-04-02T13:26:45.718 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T13:26:45.719 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T13:26:45.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.827+0000 7fd11d7af640 1 Processor -- start 2024-04-02T13:26:45.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.827+0000 7fd11d7af640 1 -- start start 2024-04-02T13:26:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.828+0000 7fd11d7af640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd11806cf90 0x7fd1180cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.828+0000 7fd11d7af640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd1180cfe30 0x7fd1180d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.828+0000 7fd11d7af640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd11806c1d0 con 0x7fd1180d0a30 2024-04-02T13:26:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.828+0000 7fd11d7af640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd11806cb00 con 0x7fd11806cf90 2024-04-02T13:26:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.828+0000 7fd11d7af640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd11806d3f0 con 0x7fd1180cfe30 2024-04-02T13:26:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.828+0000 7fd116ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd1180d0a30 0x7fd1180d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42930/0 (socket says 172.21.15.44:42930) 2024-04-02T13:26:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.828+0000 7fd1167fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd11806cf90 0x7fd1180cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.828+0000 7fd116ffd640 1 -- 172.21.15.44:0/2511660432 learned_addr learned my addr 172.21.15.44:0/2511660432 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.829+0000 7fd1167fc640 1 -- 172.21.15.44:0/2511660432 >> v2:172.21.15.69:3300/0 conn(0x7fd1180cfe30 msgr2=0x7fd1180d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.829+0000 7fd1177fe640 1 --2- 172.21.15.44:0/2511660432 >> v2:172.21.15.69:3300/0 conn(0x7fd1180cfe30 0x7fd1180d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.829+0000 7fd1167fc640 1 --2- 172.21.15.44:0/2511660432 >> v2:172.21.15.69:3300/0 conn(0x7fd1180cfe30 0x7fd1180d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.829+0000 7fd1167fc640 1 -- 172.21.15.44:0/2511660432 >> v1:172.21.15.44:6789/0 conn(0x7fd1180d0a30 legacy=0x7fd1180d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.829+0000 7fd1167fc640 1 -- 172.21.15.44:0/2511660432 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd11806c4b0 con 0x7fd11806cf90 2024-04-02T13:26:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.829+0000 7fd115ffb640 1 -- 172.21.15.44:0/2511660432 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2172341049 0 0) 0x7fd11806c1d0 con 0x7fd1180d0a30 2024-04-02T13:26:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.829+0000 7fd1167fc640 1 --2- 172.21.15.44:0/2511660432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd11806cf90 0x7fd1180cf8f0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fd11808c030 tx=0x7fd10002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0fc117065cd02e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.830+0000 7fd115ffb640 1 -- 172.21.15.44:0/2511660432 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1000315a0 con 0x7fd11806cf90 2024-04-02T13:26:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.830+0000 7fd115ffb640 1 -- 172.21.15.44:0/2511660432 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd100037070 con 0x7fd11806cf90 2024-04-02T13:26:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.830+0000 7fd115ffb640 1 -- 172.21.15.44:0/2511660432 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd100031960 con 0x7fd11806cf90 2024-04-02T13:26:45.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.830+0000 7fd11d7af640 1 -- 172.21.15.44:0/2511660432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd11806cf90 msgr2=0x7fd1180cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:45.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.830+0000 7fd11d7af640 1 --2- 172.21.15.44:0/2511660432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd11806cf90 0x7fd1180cf8f0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fd11808c030 tx=0x7fd10002b840 comp rx=0 tx=0).stop 2024-04-02T13:26:45.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.830+0000 7fd11d7af640 1 -- 172.21.15.44:0/2511660432 shutdown_connections 2024-04-02T13:26:45.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.830+0000 7fd11d7af640 1 --2- 172.21.15.44:0/2511660432 >> v2:172.21.15.69:3300/0 conn(0x7fd1180cfe30 0x7fd1180d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.830+0000 7fd11d7af640 1 --2- 172.21.15.44:0/2511660432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd11806cf90 0x7fd1180cf8f0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.830+0000 7fd11d7af640 1 -- 172.21.15.44:0/2511660432 >> 172.21.15.44:0/2511660432 conn(0x7fd11809d470 msgr2=0x7fd1180d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.830+0000 7fd11d7af640 1 -- 172.21.15.44:0/2511660432 shutdown_connections 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.830+0000 7fd11d7af640 1 -- 172.21.15.44:0/2511660432 wait complete. 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.831+0000 7fd11d7af640 1 Processor -- start 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.832+0000 7fd11d7af640 1 -- start start 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.832+0000 7fd11d7af640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd11806cf90 0x7fd1180c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.832+0000 7fd11d7af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1180cfe30 0x7fd11815a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.832+0000 7fd11d7af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1180d0a30 0x7fd11815aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.832+0000 7fd11d7af640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd11815e870 con 0x7fd1180cfe30 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.832+0000 7fd11d7af640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd11815e6f0 con 0x7fd11806cf90 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.832+0000 7fd11d7af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd1180d4da0 con 0x7fd1180d0a30 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.832+0000 7fd1167fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1180cfe30 0x7fd11815a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:45.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.832+0000 7fd1167fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1180cfe30 0x7fd11815a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49244/0 (socket says 172.21.15.44:49244) 2024-04-02T13:26:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.832+0000 7fd1167fc640 1 -- 172.21.15.44:0/1404922029 learned_addr learned my addr 172.21.15.44:0/1404922029 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd116ffd640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd11806cf90 0x7fd1180c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd1177fe640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1180d0a30 0x7fd11815aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd1167fc640 1 -- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1180d0a30 msgr2=0x7fd11815aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd1167fc640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1180d0a30 0x7fd11815aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd1167fc640 1 -- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd11806cf90 msgr2=0x7fd1180c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd1167fc640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd11806cf90 0x7fd1180c9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd1167fc640 1 -- 172.21.15.44:0/1404922029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1180b64c0 con 0x7fd1180cfe30 2024-04-02T13:26:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd1177fe640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1180d0a30 0x7fd11815aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd116ffd640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd11806cf90 0x7fd1180c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd1167fc640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1180cfe30 0x7fd11815a660 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fd100034040 tx=0x7fd1000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd10f7fe640 1 -- 172.21.15.44:0/1404922029 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd100004230 con 0x7fd1180cfe30 2024-04-02T13:26:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd10f7fe640 1 -- 172.21.15.44:0/1404922029 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1000043d0 con 0x7fd1180cfe30 2024-04-02T13:26:45.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd11d7af640 1 -- 172.21.15.44:0/1404922029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1180b67a0 con 0x7fd1180cfe30 2024-04-02T13:26:45.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd10f7fe640 1 -- 172.21.15.44:0/1404922029 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd100003850 con 0x7fd1180cfe30 2024-04-02T13:26:45.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.833+0000 7fd11d7af640 1 -- 172.21.15.44:0/1404922029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1181e8000 con 0x7fd1180cfe30 2024-04-02T13:26:45.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.834+0000 7fd11d7af640 1 -- 172.21.15.44:0/1404922029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1180684f0 con 0x7fd1180cfe30 2024-04-02T13:26:45.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.834+0000 7fd10f7fe640 1 -- 172.21.15.44:0/1404922029 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd1000039f0 con 0x7fd1180cfe30 2024-04-02T13:26:45.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.835+0000 7fd10f7fe640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd0f806c8f0 0x7fd0f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:45.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.835+0000 7fd10f7fe640 1 -- 172.21.15.44:0/1404922029 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7fd10002f030 con 0x7fd1180cfe30 2024-04-02T13:26:45.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.835+0000 7fd116ffd640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd0f806c8f0 0x7fd0f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:45.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.836+0000 7fd116ffd640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd0f806c8f0 0x7fd0f806edb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd1080012e0 tx=0x7fd108005b90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:45.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:45.839+0000 7fd10f7fe640 1 -- 172.21.15.44:0/1404922029 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd100041070 con 0x7fd1180cfe30 2024-04-02T13:26:46.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.062+0000 7fd11d7af640 1 -- 172.21.15.44:0/1404922029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd1180d17c0 con 0x7fd1180cfe30 2024-04-02T13:26:46.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.701+0000 7fd10f7fe640 1 -- 172.21.15.44:0/1404922029 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v61) ==== 122+0+0 (secure 0 0 0) 0x7fd100044340 con 0x7fd1180cfe30 2024-04-02T13:26:46.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.704+0000 7fd11d7af640 1 -- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd0f806c8f0 msgr2=0x7fd0f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:46.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.704+0000 7fd11d7af640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd0f806c8f0 0x7fd0f806edb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd1080012e0 tx=0x7fd108005b90 comp rx=0 tx=0).stop 2024-04-02T13:26:46.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.704+0000 7fd11d7af640 1 -- 172.21.15.44:0/1404922029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1180cfe30 msgr2=0x7fd11815a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:46.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.704+0000 7fd11d7af640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1180cfe30 0x7fd11815a660 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fd100034040 tx=0x7fd1000046c0 comp rx=0 tx=0).stop 2024-04-02T13:26:46.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.704+0000 7fd11d7af640 1 -- 172.21.15.44:0/1404922029 shutdown_connections 2024-04-02T13:26:46.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.704+0000 7fd11d7af640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd0f806c8f0 0x7fd0f806edb0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:46.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.704+0000 7fd11d7af640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1180d0a30 0x7fd11815aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:46.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.704+0000 7fd11d7af640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1180cfe30 0x7fd11815a660 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:46.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.704+0000 7fd11d7af640 1 --2- 172.21.15.44:0/1404922029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd11806cf90 0x7fd1180c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:46.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.704+0000 7fd11d7af640 1 -- 172.21.15.44:0/1404922029 >> 172.21.15.44:0/1404922029 conn(0x7fd11809d470 msgr2=0x7fd1180d3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:46.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.704+0000 7fd11d7af640 1 -- 172.21.15.44:0/1404922029 shutdown_connections 2024-04-02T13:26:46.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.704+0000 7fd11d7af640 1 -- 172.21.15.44:0/1404922029 wait complete. 2024-04-02T13:26:46.720 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T13:26:46.720 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:26:46.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.825+0000 7f0ad6ef2640 1 Processor -- start 2024-04-02T13:26:46.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.825+0000 7f0ad6ef2640 1 -- start start 2024-04-02T13:26:46.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.826+0000 7f0ad6ef2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0ad006bbe0 0x7f0ad0169140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:46.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.826+0000 7f0ad6ef2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ad00aed70 con 0x7f0ad015e4d0 2024-04-02T13:26:46.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.826+0000 7f0ad6ef2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ad006dfe0 con 0x7f0ad015e0f0 2024-04-02T13:26:46.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.826+0000 7f0ad6ef2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0ad006e2c0 con 0x7f0ad006bbe0 2024-04-02T13:26:46.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.826+0000 7f0ad548f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0ad006bbe0 0x7f0ad0169140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:46.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.826+0000 7f0acffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0ad015e4d0 0x7f0ad006b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42946/0 (socket says 172.21.15.44:42946) 2024-04-02T13:26:46.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.826+0000 7f0acffff640 1 -- 172.21.15.44:0/3691756997 learned_addr learned my addr 172.21.15.44:0/3691756997 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:46.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.826+0000 7f0ad548f640 1 -- 172.21.15.44:0/3691756997 >> v1:172.21.15.69:6789/0 conn(0x7f0ad015e0f0 legacy=0x7f0ad00d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:46.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.826+0000 7f0ad548f640 1 -- 172.21.15.44:0/3691756997 >> v1:172.21.15.44:6789/0 conn(0x7f0ad015e4d0 legacy=0x7f0ad006b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:46.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.826+0000 7f0acf7fe640 1 -- 172.21.15.44:0/3691756997 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3640695818 0 0) 0x7f0ad00aed70 con 0x7f0ad015e4d0 2024-04-02T13:26:46.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.826+0000 7f0ad548f640 1 -- 172.21.15.44:0/3691756997 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad006e5a0 con 0x7f0ad006bbe0 2024-04-02T13:26:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.827+0000 7f0ad548f640 1 --2- 172.21.15.44:0/3691756997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad006bbe0 0x7f0ad0169140 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009d90 tx=0x7f0ac802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0f89280dafde52 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.827+0000 7f0acf7fe640 1 -- 172.21.15.44:0/3691756997 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8039070 con 0x7f0ad006bbe0 2024-04-02T13:26:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.827+0000 7f0acf7fe640 1 -- 172.21.15.44:0/3691756997 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ac802cd00 con 0x7f0ad006bbe0 2024-04-02T13:26:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.827+0000 7f0acf7fe640 1 -- 172.21.15.44:0/3691756997 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8034660 con 0x7f0ad006bbe0 2024-04-02T13:26:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.828+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3691756997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad006bbe0 msgr2=0x7f0ad0169140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.828+0000 7f0ad6ef2640 1 --2- 172.21.15.44:0/3691756997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad006bbe0 0x7f0ad0169140 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009d90 tx=0x7f0ac802bf50 comp rx=0 tx=0).stop 2024-04-02T13:26:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.828+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3691756997 shutdown_connections 2024-04-02T13:26:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.828+0000 7f0ad6ef2640 1 --2- 172.21.15.44:0/3691756997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad006bbe0 0x7f0ad0169140 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.828+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3691756997 >> 172.21.15.44:0/3691756997 conn(0x7f0ad00b2980 msgr2=0x7f0ad00b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.828+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3691756997 shutdown_connections 2024-04-02T13:26:46.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.828+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3691756997 wait complete. 2024-04-02T13:26:46.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.829+0000 7f0ad6ef2640 1 Processor -- start 2024-04-02T13:26:46.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.829+0000 7f0ad6ef2640 1 -- start start 2024-04-02T13:26:46.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad6ef2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad006bbe0 0x7f0ad00b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:46.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad6ef2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ad015e0f0 0x7f0ad00cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:46.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad6ef2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ad015e4d0 0x7f0ad00ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:46.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad6ef2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad00d0800 con 0x7f0ad006bbe0 2024-04-02T13:26:46.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad6ef2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad00d0af0 con 0x7f0ad015e4d0 2024-04-02T13:26:46.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad6ef2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0ad006eda0 con 0x7f0ad015e0f0 2024-04-02T13:26:46.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad4c8e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad006bbe0 0x7f0ad00b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:46.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad4c8e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad006bbe0 0x7f0ad00b64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49260/0 (socket says 172.21.15.44:49260) 2024-04-02T13:26:46.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad4c8e640 1 -- 172.21.15.44:0/3495888845 learned_addr learned my addr 172.21.15.44:0/3495888845 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:46.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad548f640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ad015e4d0 0x7f0ad00ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:46.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0acffff640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ad015e0f0 0x7f0ad00cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:46.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad4c8e640 1 -- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ad015e0f0 msgr2=0x7f0ad00cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:46.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad4c8e640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ad015e0f0 0x7f0ad00cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:46.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad4c8e640 1 -- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ad015e4d0 msgr2=0x7f0ad00ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:46.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad4c8e640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ad015e4d0 0x7f0ad00ccfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:46.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad4c8e640 1 -- 172.21.15.44:0/3495888845 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad00cc2e0 con 0x7f0ad006bbe0 2024-04-02T13:26:46.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad548f640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ad015e4d0 0x7f0ad00ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:46.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0acffff640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ad015e0f0 0x7f0ad00cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:46.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0ad4c8e640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad006bbe0 0x7f0ad00b64a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400b420 tx=0x7f0ac400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:46.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0acd7fa640 1 -- 172.21.15.44:0/3495888845 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4011830 con 0x7f0ad006bbe0 2024-04-02T13:26:46.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.830+0000 7f0acd7fa640 1 -- 172.21.15.44:0/3495888845 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ac4016070 con 0x7f0ad006bbe0 2024-04-02T13:26:46.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.831+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3495888845 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad00cc5c0 con 0x7f0ad006bbe0 2024-04-02T13:26:46.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.831+0000 7f0acd7fa640 1 -- 172.21.15.44:0/3495888845 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4011c10 con 0x7f0ad006bbe0 2024-04-02T13:26:46.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.831+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3495888845 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ad00cc8a0 con 0x7f0ad006bbe0 2024-04-02T13:26:46.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.831+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3495888845 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ad00684f0 con 0x7f0ad006bbe0 2024-04-02T13:26:46.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.832+0000 7f0acd7fa640 1 -- 172.21.15.44:0/3495888845 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ac401b460 con 0x7f0ad006bbe0 2024-04-02T13:26:46.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.832+0000 7f0acd7fa640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ab406c8f0 0x7f0ab406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:46.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.832+0000 7f0acd7fa640 1 -- 172.21.15.44:0/3495888845 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7f0ac408b810 con 0x7f0ad006bbe0 2024-04-02T13:26:46.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.833+0000 7f0acffff640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ab406c8f0 0x7f0ab406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:46.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.833+0000 7f0acffff640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ab406c8f0 0x7f0ab406edb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00b5b90 tx=0x7f0abc007c10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:46.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:46.837+0000 7f0acd7fa640 1 -- 172.21.15.44:0/3495888845 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0ac4020020 con 0x7f0ad006bbe0 2024-04-02T13:26:47.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.059+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3495888845 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ad006b720 con 0x7f0ad006bbe0 2024-04-02T13:26:47.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.060+0000 7f0acd7fa640 1 -- 172.21.15.44:0/3495888845 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5177 (secure 0 0 0) 0x7f0ac4059ab0 con 0x7f0ad006bbe0 2024-04-02T13:26:47.062 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:47.062 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":61,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:26:46.063220+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6833/719680934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":719680934},{"type":"v1","addr":"172.21.15.44:6833","nonce":719680934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":6}]} 2024-04-02T13:26:47.063 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 61 2024-04-02T13:26:47.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.062+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ab406c8f0 msgr2=0x7f0ab406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.062+0000 7f0ad6ef2640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ab406c8f0 0x7f0ab406edb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00b5b90 tx=0x7f0abc007c10 comp rx=0 tx=0).stop 2024-04-02T13:26:47.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.062+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3495888845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad006bbe0 msgr2=0x7f0ad00b64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.062+0000 7f0ad6ef2640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad006bbe0 0x7f0ad00b64a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400b420 tx=0x7f0ac400b8e0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.062+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3495888845 shutdown_connections 2024-04-02T13:26:47.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.062+0000 7f0ad6ef2640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ab406c8f0 0x7f0ab406edb0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.062+0000 7f0ad6ef2640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ad015e4d0 0x7f0ad00ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.062+0000 7f0ad6ef2640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ad015e0f0 0x7f0ad00cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.062+0000 7f0ad6ef2640 1 --2- 172.21.15.44:0/3495888845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad006bbe0 0x7f0ad00b64a0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.062+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3495888845 >> 172.21.15.44:0/3495888845 conn(0x7f0ad00b2980 msgr2=0x7f0ad00b1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:47.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.062+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3495888845 shutdown_connections 2024-04-02T13:26:47.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.062+0000 7f0ad6ef2640 1 -- 172.21.15.44:0/3495888845 wait complete. 2024-04-02T13:26:47.075 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:26:47.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.174+0000 7f295ebf2640 1 Processor -- start 2024-04-02T13:26:47.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.174+0000 7f295ebf2640 1 -- start start 2024-04-02T13:26:47.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295ebf2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f295815aa00 0x7f29580b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295ebf2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2958072bf0 0x7f29580cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295ebf2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f29580ce110 0x7f295815e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295ebf2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f295815ade0 con 0x7f295815aa00 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295ebf2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f29580af1b0 con 0x7f29580ce110 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295ebf2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f29580af490 con 0x7f2958072bf0 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295c98e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f295815aa00 0x7f29580b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295c98e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f295815aa00 0x7f29580b2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49266/0 (socket says 172.21.15.44:49266) 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295c98e640 1 -- 172.21.15.44:0/3218617097 learned_addr learned my addr 172.21.15.44:0/3218617097 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f2957fff640 1 --2- 172.21.15.44:0/3218617097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2958072bf0 0x7f29580cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295d18f640 1 --2- 172.21.15.44:0/3218617097 >> v2:172.21.15.69:3300/0 conn(0x7f29580ce110 0x7f295815e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295c98e640 1 -- 172.21.15.44:0/3218617097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2958072bf0 msgr2=0x7f29580cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295c98e640 1 --2- 172.21.15.44:0/3218617097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2958072bf0 0x7f29580cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295c98e640 1 -- 172.21.15.44:0/3218617097 >> v2:172.21.15.69:3300/0 conn(0x7f29580ce110 msgr2=0x7f295815e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.175+0000 7f295c98e640 1 --2- 172.21.15.44:0/3218617097 >> v2:172.21.15.69:3300/0 conn(0x7f29580ce110 0x7f295815e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.176+0000 7f295c98e640 1 -- 172.21.15.44:0/3218617097 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29580723f0 con 0x7f295815aa00 2024-04-02T13:26:47.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.176+0000 7f295c98e640 1 --2- 172.21.15.44:0/3218617097 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f295815aa00 0x7f29580b2d00 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f295806f650 tx=0x7f295002e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4976324ab822313 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:47.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.176+0000 7f29577fe640 1 -- 172.21.15.44:0/3218617097 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2950034070 con 0x7f295815aa00 2024-04-02T13:26:47.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.176+0000 7f29577fe640 1 -- 172.21.15.44:0/3218617097 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f295002ee10 con 0x7f295815aa00 2024-04-02T13:26:47.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.176+0000 7f29577fe640 1 -- 172.21.15.44:0/3218617097 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2950038620 con 0x7f295815aa00 2024-04-02T13:26:47.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.177+0000 7f295ebf2640 1 -- 172.21.15.44:0/3218617097 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f295815aa00 msgr2=0x7f29580b2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.177+0000 7f295ebf2640 1 --2- 172.21.15.44:0/3218617097 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f295815aa00 0x7f29580b2d00 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f295806f650 tx=0x7f295002e060 comp rx=0 tx=0).stop 2024-04-02T13:26:47.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.177+0000 7f295ebf2640 1 -- 172.21.15.44:0/3218617097 shutdown_connections 2024-04-02T13:26:47.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.177+0000 7f295ebf2640 1 --2- 172.21.15.44:0/3218617097 >> v2:172.21.15.69:3300/0 conn(0x7f29580ce110 0x7f295815e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.177+0000 7f295ebf2640 1 --2- 172.21.15.44:0/3218617097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2958072bf0 0x7f29580cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.177+0000 7f295ebf2640 1 --2- 172.21.15.44:0/3218617097 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f295815aa00 0x7f29580b2d00 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.177+0000 7f295ebf2640 1 -- 172.21.15.44:0/3218617097 >> 172.21.15.44:0/3218617097 conn(0x7f29580cfcd0 msgr2=0x7f29580d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:47.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.177+0000 7f295ebf2640 1 -- 172.21.15.44:0/3218617097 shutdown_connections 2024-04-02T13:26:47.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.177+0000 7f295ebf2640 1 -- 172.21.15.44:0/3218617097 wait complete. 2024-04-02T13:26:47.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.178+0000 7f295ebf2640 1 Processor -- start 2024-04-02T13:26:47.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.178+0000 7f295ebf2640 1 -- start start 2024-04-02T13:26:47.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.179+0000 7f295ebf2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2958072bf0 0x7f295814b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.179+0000 7f295ebf2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29580ce110 0x7f295809aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.179+0000 7f295ebf2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f295815aa00 0x7f29581effa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.179+0000 7f295ebf2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2958073050 con 0x7f295815aa00 2024-04-02T13:26:47.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.179+0000 7f295ebf2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f29580ce5a0 con 0x7f29580ce110 2024-04-02T13:26:47.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.179+0000 7f295ebf2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f29580731d0 con 0x7f2958072bf0 2024-04-02T13:26:47.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.179+0000 7f295d18f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f295815aa00 0x7f29581effa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.179+0000 7f295d18f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f295815aa00 0x7f29581effa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49270/0 (socket says 172.21.15.44:49270) 2024-04-02T13:26:47.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.179+0000 7f295d18f640 1 -- 172.21.15.44:0/1194612816 learned_addr learned my addr 172.21.15.44:0/1194612816 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:47.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.179+0000 7f2957fff640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29580ce110 0x7f295809aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.179+0000 7f295c98e640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2958072bf0 0x7f295814b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.180+0000 7f295d18f640 1 -- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2958072bf0 msgr2=0x7f295814b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.180+0000 7f295d18f640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2958072bf0 0x7f295814b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.180+0000 7f295d18f640 1 -- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29580ce110 msgr2=0x7f295809aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.180+0000 7f295d18f640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29580ce110 0x7f295809aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.180+0000 7f295d18f640 1 -- 172.21.15.44:0/1194612816 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295814c640 con 0x7f295815aa00 2024-04-02T13:26:47.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.180+0000 7f2957fff640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29580ce110 0x7f295809aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:47.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.180+0000 7f295c98e640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2958072bf0 0x7f295814b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:47.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.180+0000 7f295d18f640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f295815aa00 0x7f29581effa0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f294800d800 tx=0x7f294800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:47.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.180+0000 7f29557fa640 1 -- 172.21.15.44:0/1194612816 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294800b6c0 con 0x7f295815aa00 2024-04-02T13:26:47.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.180+0000 7f29557fa640 1 -- 172.21.15.44:0/1194612816 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2948019070 con 0x7f295815aa00 2024-04-02T13:26:47.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.180+0000 7f295ebf2640 1 -- 172.21.15.44:0/1194612816 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f295814c920 con 0x7f295815aa00 2024-04-02T13:26:47.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.180+0000 7f29557fa640 1 -- 172.21.15.44:0/1194612816 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294800bac0 con 0x7f295815aa00 2024-04-02T13:26:47.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.180+0000 7f295ebf2640 1 -- 172.21.15.44:0/1194612816 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f295814cee0 con 0x7f295815aa00 2024-04-02T13:26:47.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.181+0000 7f295ebf2640 1 -- 172.21.15.44:0/1194612816 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29580684f0 con 0x7f295815aa00 2024-04-02T13:26:47.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.181+0000 7f29557fa640 1 -- 172.21.15.44:0/1194612816 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f294800bc60 con 0x7f295815aa00 2024-04-02T13:26:47.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.182+0000 7f29557fa640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f294406c8f0 0x7f294406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.182+0000 7f29557fa640 1 -- 172.21.15.44:0/1194612816 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7f294808c150 con 0x7f295815aa00 2024-04-02T13:26:47.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.182+0000 7f295c98e640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f294406c8f0 0x7f294406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.183+0000 7f295c98e640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f294406c8f0 0x7f294406edb0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2950039000 tx=0x7f2950034000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:47.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.186+0000 7f29557fa640 1 -- 172.21.15.44:0/1194612816 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f294805a3f0 con 0x7f295815aa00 2024-04-02T13:26:47.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.371+0000 7f295ebf2640 1 -- 172.21.15.44:0/1194612816 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29580ccc10 con 0x7f295815aa00 2024-04-02T13:26:47.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.372+0000 7f29557fa640 1 -- 172.21.15.44:0/1194612816 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+16837 (secure 0 0 0) 0x7f2948059cf0 con 0x7f295815aa00 2024-04-02T13:26:47.376 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:47.376 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:26:44.676976+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:26:41.300970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-02T13:26:41.979586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:26:47.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.374+0000 7f295ebf2640 1 -- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f294406c8f0 msgr2=0x7f294406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.374+0000 7f295ebf2640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f294406c8f0 0x7f294406edb0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2950039000 tx=0x7f2950034000 comp rx=0 tx=0).stop 2024-04-02T13:26:47.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.374+0000 7f295ebf2640 1 -- 172.21.15.44:0/1194612816 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f295815aa00 msgr2=0x7f29581effa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.374+0000 7f295ebf2640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f295815aa00 0x7f29581effa0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f294800d800 tx=0x7f294800dcc0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.375+0000 7f295ebf2640 1 -- 172.21.15.44:0/1194612816 shutdown_connections 2024-04-02T13:26:47.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.375+0000 7f295ebf2640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f294406c8f0 0x7f294406edb0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.375+0000 7f295ebf2640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f295815aa00 0x7f29581effa0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.375+0000 7f295ebf2640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29580ce110 0x7f295809aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.375+0000 7f295ebf2640 1 --2- 172.21.15.44:0/1194612816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2958072bf0 0x7f295814b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.375+0000 7f295ebf2640 1 -- 172.21.15.44:0/1194612816 >> 172.21.15.44:0/1194612816 conn(0x7f29580cfcd0 msgr2=0x7f29580b1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:47.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.375+0000 7f295ebf2640 1 -- 172.21.15.44:0/1194612816 shutdown_connections 2024-04-02T13:26:47.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.375+0000 7f295ebf2640 1 -- 172.21.15.44:0/1194612816 wait complete. 2024-04-02T13:26:47.388 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T13:26:47.389 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:26:47.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.488+0000 7f7d078b1640 1 Processor -- start 2024-04-02T13:26:47.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.488+0000 7f7d078b1640 1 -- start start 2024-04-02T13:26:47.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.488+0000 7f7d078b1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d000cece0 0x7f7d000cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.488+0000 7f7d078b1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7d0006ece0 0x7f7d000acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.489+0000 7f7d078b1640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d000cf600 con 0x7f7d000d09f0 2024-04-02T13:26:47.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.489+0000 7f7d078b1640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7d0006d9e0 con 0x7f7d0006ece0 2024-04-02T13:26:47.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.489+0000 7f7d078b1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7d0006dc40 con 0x7f7d000cece0 2024-04-02T13:26:47.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.489+0000 7f7d05e4e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7d0006ece0 0x7f7d000acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.489+0000 7f7d05e4e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7d0006ece0 0x7f7d000acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49276/0 (socket says 172.21.15.44:49276) 2024-04-02T13:26:47.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.489+0000 7f7d05e4e640 1 -- 172.21.15.44:0/1185743601 learned_addr learned my addr 172.21.15.44:0/1185743601 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:47.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.489+0000 7f7d0564d640 1 --2- 172.21.15.44:0/1185743601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d000cece0 0x7f7d000cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.489+0000 7f7d05e4e640 1 -- 172.21.15.44:0/1185743601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d000cece0 msgr2=0x7f7d000cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:47.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.489+0000 7f7d05e4e640 1 --2- 172.21.15.44:0/1185743601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d000cece0 0x7f7d000cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.489+0000 7f7d05e4e640 1 -- 172.21.15.44:0/1185743601 >> v1:172.21.15.69:6789/0 conn(0x7f7d000d09f0 legacy=0x7f7d0006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:47.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.489+0000 7f7d05e4e640 1 -- 172.21.15.44:0/1185743601 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0006df20 con 0x7f7d0006ece0 2024-04-02T13:26:47.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.490+0000 7f7d05e4e640 1 --2- 172.21.15.44:0/1185743601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d0006ece0 0x7f7d000acd50 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800d180 tx=0x7f7cf802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5502343033594551 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:47.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.490+0000 7f7cf7fff640 1 -- 172.21.15.44:0/1185743601 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf802ea60 con 0x7f7d0006ece0 2024-04-02T13:26:47.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.490+0000 7f7cf7fff640 1 -- 172.21.15.44:0/1185743601 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cf802ec00 con 0x7f7d0006ece0 2024-04-02T13:26:47.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.490+0000 7f7cf7fff640 1 -- 172.21.15.44:0/1185743601 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8009600 con 0x7f7d0006ece0 2024-04-02T13:26:47.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.490+0000 7f7d078b1640 1 -- 172.21.15.44:0/1185743601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d0006ece0 msgr2=0x7f7d000acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.490+0000 7f7d078b1640 1 --2- 172.21.15.44:0/1185743601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d0006ece0 0x7f7d000acd50 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800d180 tx=0x7f7cf802e0c0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.490+0000 7f7d078b1640 1 -- 172.21.15.44:0/1185743601 shutdown_connections 2024-04-02T13:26:47.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.490+0000 7f7d078b1640 1 --2- 172.21.15.44:0/1185743601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d0006ece0 0x7f7d000acd50 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.490+0000 7f7d078b1640 1 --2- 172.21.15.44:0/1185743601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d000cece0 0x7f7d000cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.490+0000 7f7d078b1640 1 -- 172.21.15.44:0/1185743601 >> 172.21.15.44:0/1185743601 conn(0x7f7d000cfcd0 msgr2=0x7f7d000d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:47.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.491+0000 7f7d078b1640 1 -- 172.21.15.44:0/1185743601 shutdown_connections 2024-04-02T13:26:47.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.491+0000 7f7d078b1640 1 -- 172.21.15.44:0/1185743601 wait complete. 2024-04-02T13:26:47.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.492+0000 7f7d078b1640 1 Processor -- start 2024-04-02T13:26:47.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.492+0000 7f7d078b1640 1 -- start start 2024-04-02T13:26:47.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.492+0000 7f7d078b1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d0006ece0 0x7f7d0009d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.492+0000 7f7d078b1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d000cece0 0x7f7d00142f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.492+0000 7f7d078b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d000d09f0 0x7f7d000b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.492+0000 7f7d078b1640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0006e7c0 con 0x7f7d000d09f0 2024-04-02T13:26:47.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.492+0000 7f7d078b1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0006e4e0 con 0x7f7d000cece0 2024-04-02T13:26:47.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.492+0000 7f7d078b1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7d0015b220 con 0x7f7d0006ece0 2024-04-02T13:26:47.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7d05e4e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d000d09f0 0x7f7d000b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7d05e4e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d000d09f0 0x7f7d000b0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49288/0 (socket says 172.21.15.44:49288) 2024-04-02T13:26:47.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7d05e4e640 1 -- 172.21.15.44:0/4123364601 learned_addr learned my addr 172.21.15.44:0/4123364601 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:47.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7d0564d640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d0006ece0 0x7f7d0009d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7d04e4c640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d000cece0 0x7f7d00142f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7d05e4e640 1 -- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d0006ece0 msgr2=0x7f7d0009d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7d05e4e640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d0006ece0 0x7f7d0009d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7d05e4e640 1 -- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d000cece0 msgr2=0x7f7d00142f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7d05e4e640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d000cece0 0x7f7d00142f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7d05e4e640 1 -- 172.21.15.44:0/4123364601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d00143800 con 0x7f7d000d09f0 2024-04-02T13:26:47.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7d05e4e640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d000d09f0 0x7f7d000b0db0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800d2a0 tx=0x7f7cf8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:47.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7cf5ffb640 1 -- 172.21.15.44:0/4123364601 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf800b720 con 0x7f7d000d09f0 2024-04-02T13:26:47.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7cf5ffb640 1 -- 172.21.15.44:0/4123364601 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cf8045070 con 0x7f7d000d09f0 2024-04-02T13:26:47.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7d078b1640 1 -- 172.21.15.44:0/4123364601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d00143ae0 con 0x7f7d000d09f0 2024-04-02T13:26:47.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7cf5ffb640 1 -- 172.21.15.44:0/4123364601 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8036a60 con 0x7f7d000d09f0 2024-04-02T13:26:47.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.493+0000 7f7d078b1640 1 -- 172.21.15.44:0/4123364601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d001440a0 con 0x7f7d000d09f0 2024-04-02T13:26:47.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.495+0000 7f7d078b1640 1 -- 172.21.15.44:0/4123364601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d00074f20 con 0x7f7d000d09f0 2024-04-02T13:26:47.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.495+0000 7f7cf5ffb640 1 -- 172.21.15.44:0/4123364601 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7cf8036c00 con 0x7f7d000d09f0 2024-04-02T13:26:47.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.495+0000 7f7cf5ffb640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7ce406c8f0 0x7f7ce406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.495+0000 7f7cf5ffb640 1 -- 172.21.15.44:0/4123364601 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7f7cf80aebc0 con 0x7f7d000d09f0 2024-04-02T13:26:47.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.495+0000 7f7d0564d640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7ce406c8f0 0x7f7ce406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.496+0000 7f7d0564d640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7ce406c8f0 0x7f7ce406edb0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00a970 tx=0x7f7cfc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:47.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.500+0000 7f7cf5ffb640 1 -- 172.21.15.44:0/4123364601 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7cf807cef0 con 0x7f7d000d09f0 2024-04-02T13:26:47.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.681+0000 7f7d078b1640 1 -- 172.21.15.44:0/4123364601 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7d000723a0 con 0x7f7ce406c8f0 2024-04-02T13:26:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.688+0000 7f7cf5ffb640 1 -- 172.21.15.44:0/4123364601 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252004 (secure 0 0 0) 0x7f7d000723a0 con 0x7f7ce406c8f0 2024-04-02T13:26:47.693 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:26:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.691+0000 7f7d078b1640 1 -- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7ce406c8f0 msgr2=0x7f7ce406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.691+0000 7f7d078b1640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7ce406c8f0 0x7f7ce406edb0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00a970 tx=0x7f7cfc008040 comp rx=0 tx=0).stop 2024-04-02T13:26:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.691+0000 7f7d078b1640 1 -- 172.21.15.44:0/4123364601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d000d09f0 msgr2=0x7f7d000b0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.691+0000 7f7d078b1640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d000d09f0 0x7f7d000b0db0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800d2a0 tx=0x7f7cf8004a40 comp rx=0 tx=0).stop 2024-04-02T13:26:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.691+0000 7f7d078b1640 1 -- 172.21.15.44:0/4123364601 shutdown_connections 2024-04-02T13:26:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.691+0000 7f7d078b1640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7ce406c8f0 0x7f7ce406edb0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.691+0000 7f7d078b1640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d000d09f0 0x7f7d000b0db0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.691+0000 7f7d078b1640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d000cece0 0x7f7d00142f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.691+0000 7f7d078b1640 1 --2- 172.21.15.44:0/4123364601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d0006ece0 0x7f7d0009d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.691+0000 7f7d078b1640 1 -- 172.21.15.44:0/4123364601 >> 172.21.15.44:0/4123364601 conn(0x7f7d000cfcd0 msgr2=0x7f7d00071c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.691+0000 7f7d078b1640 1 -- 172.21.15.44:0/4123364601 shutdown_connections 2024-04-02T13:26:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.691+0000 7f7d078b1640 1 -- 172.21.15.44:0/4123364601 wait complete. 2024-04-02T13:26:47.704 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-04-02T13:26:46.969476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259276,"kb_used_data":9812,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715444,"statfs":{"total":773094113280,"available":772828614656,"internally_reserved":0,"allocated":10047488,"data_stored":5228868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.320745"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.713848+0000","last_change":"2024-04-02T13:26:43.710931+0000","last_active":"2024-04-02T13:26:43.713848+0000","last_peered":"2024-04-02T13:26:43.713848+0000","last_clean":"2024-04-02T13:26:43.713848+0000","last_became_active":"2024-04-02T13:26:43.710522+0000","last_became_peered":"2024-04-02T13:26:43.710522+0000","last_unstale":"2024-04-02T13:26:43.713848+0000","last_undegraded":"2024-04-02T13:26:43.713848+0000","last_fullsized":"2024-04-02T13:26:43.713848+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:59:22.715019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698342+0000","last_change":"2024-04-02T13:26:43.690717+0000","last_active":"2024-04-02T13:26:43.698342+0000","last_peered":"2024-04-02T13:26:43.698342+0000","last_clean":"2024-04-02T13:26:43.698342+0000","last_became_active":"2024-04-02T13:26:43.690358+0000","last_became_peered":"2024-04-02T13:26:43.690358+0000","last_unstale":"2024-04-02T13:26:43.698342+0000","last_undegraded":"2024-04-02T13:26:43.698342+0000","last_fullsized":"2024-04-02T13:26:43.698342+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:27.943906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700646+0000","last_change":"2024-04-02T13:26:43.692683+0000","last_active":"2024-04-02T13:26:43.700646+0000","last_peered":"2024-04-02T13:26:43.700646+0000","last_clean":"2024-04-02T13:26:43.700646+0000","last_became_active":"2024-04-02T13:26:43.690759+0000","last_became_peered":"2024-04-02T13:26:43.690759+0000","last_unstale":"2024-04-02T13:26:43.700646+0000","last_undegraded":"2024-04-02T13:26:43.700646+0000","last_fullsized":"2024-04-02T13:26:43.700646+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:36:28.322738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700488+0000","last_change":"2024-04-02T13:26:43.698501+0000","last_active":"2024-04-02T13:26:43.700488+0000","last_peered":"2024-04-02T13:26:43.700488+0000","last_clean":"2024-04-02T13:26:43.700488+0000","last_became_active":"2024-04-02T13:26:43.698175+0000","last_became_peered":"2024-04-02T13:26:43.698175+0000","last_unstale":"2024-04-02T13:26:43.700488+0000","last_undegraded":"2024-04-02T13:26:43.700488+0000","last_fullsized":"2024-04-02T13:26:43.700488+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:24:01.313811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700498+0000","last_change":"2024-04-02T13:26:43.695954+0000","last_active":"2024-04-02T13:26:43.700498+0000","last_peered":"2024-04-02T13:26:43.700498+0000","last_clean":"2024-04-02T13:26:43.700498+0000","last_became_active":"2024-04-02T13:26:43.694696+0000","last_became_peered":"2024-04-02T13:26:43.694696+0000","last_unstale":"2024-04-02T13:26:43.700498+0000","last_undegraded":"2024-04-02T13:26:43.700498+0000","last_fullsized":"2024-04-02T13:26:43.700498+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:42.324949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.705396+0000","last_change":"2024-04-02T13:26:43.701921+0000","last_active":"2024-04-02T13:26:43.705396+0000","last_peered":"2024-04-02T13:26:43.705396+0000","last_clean":"2024-04-02T13:26:43.705396+0000","last_became_active":"2024-04-02T13:26:43.701649+0000","last_became_peered":"2024-04-02T13:26:43.701649+0000","last_unstale":"2024-04-02T13:26:43.705396+0000","last_undegraded":"2024-04-02T13:26:43.705396+0000","last_fullsized":"2024-04-02T13:26:43.705396+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:59:22.715019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700732+0000","last_change":"2024-04-02T13:26:43.693246+0000","last_active":"2024-04-02T13:26:43.700732+0000","last_peered":"2024-04-02T13:26:43.700732+0000","last_clean":"2024-04-02T13:26:43.700732+0000","last_became_active":"2024-04-02T13:26:43.692983+0000","last_became_peered":"2024-04-02T13:26:43.692983+0000","last_unstale":"2024-04-02T13:26:43.700732+0000","last_undegraded":"2024-04-02T13:26:43.700732+0000","last_fullsized":"2024-04-02T13:26:43.700732+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:36:28.322738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.703138+0000","last_change":"2024-04-02T13:26:43.698936+0000","last_active":"2024-04-02T13:26:43.703138+0000","last_peered":"2024-04-02T13:26:43.703138+0000","last_clean":"2024-04-02T13:26:43.703138+0000","last_became_active":"2024-04-02T13:26:43.697031+0000","last_became_peered":"2024-04-02T13:26:43.697031+0000","last_unstale":"2024-04-02T13:26:43.703138+0000","last_undegraded":"2024-04-02T13:26:43.703138+0000","last_fullsized":"2024-04-02T13:26:43.703138+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:50.758013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.697726+0000","last_change":"2024-04-02T13:26:43.690869+0000","last_active":"2024-04-02T13:26:43.697726+0000","last_peered":"2024-04-02T13:26:43.697726+0000","last_clean":"2024-04-02T13:26:43.697726+0000","last_became_active":"2024-04-02T13:26:43.690577+0000","last_became_peered":"2024-04-02T13:26:43.690577+0000","last_unstale":"2024-04-02T13:26:43.697726+0000","last_undegraded":"2024-04-02T13:26:43.697726+0000","last_fullsized":"2024-04-02T13:26:43.697726+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:24:32.385207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.695115+0000","last_change":"2024-04-02T13:26:43.693119+0000","last_active":"2024-04-02T13:26:43.695115+0000","last_peered":"2024-04-02T13:26:43.695115+0000","last_clean":"2024-04-02T13:26:43.695115+0000","last_became_active":"2024-04-02T13:26:43.692837+0000","last_became_peered":"2024-04-02T13:26:43.692837+0000","last_unstale":"2024-04-02T13:26:43.695115+0000","last_undegraded":"2024-04-02T13:26:43.695115+0000","last_fullsized":"2024-04-02T13:26:43.695115+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:27:50.911137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711550+0000","last_change":"2024-04-02T13:26:43.707639+0000","last_active":"2024-04-02T13:26:43.711550+0000","last_peered":"2024-04-02T13:26:43.711550+0000","last_clean":"2024-04-02T13:26:43.711550+0000","last_became_active":"2024-04-02T13:26:43.707341+0000","last_became_peered":"2024-04-02T13:26:43.707341+0000","last_unstale":"2024-04-02T13:26:43.711550+0000","last_undegraded":"2024-04-02T13:26:43.711550+0000","last_fullsized":"2024-04-02T13:26:43.711550+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:24:01.313811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.687123+0000","last_change":"2024-04-02T13:26:43.685929+0000","last_active":"2024-04-02T13:26:43.687123+0000","last_peered":"2024-04-02T13:26:43.687123+0000","last_clean":"2024-04-02T13:26:43.687123+0000","last_became_active":"2024-04-02T13:26:43.685617+0000","last_became_peered":"2024-04-02T13:26:43.685617+0000","last_unstale":"2024-04-02T13:26:43.687123+0000","last_undegraded":"2024-04-02T13:26:43.687123+0000","last_fullsized":"2024-04-02T13:26:43.687123+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:12.500724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700375+0000","last_change":"2024-04-02T13:26:43.696587+0000","last_active":"2024-04-02T13:26:43.700375+0000","last_peered":"2024-04-02T13:26:43.700375+0000","last_clean":"2024-04-02T13:26:43.700375+0000","last_became_active":"2024-04-02T13:26:43.696290+0000","last_became_peered":"2024-04-02T13:26:43.696290+0000","last_unstale":"2024-04-02T13:26:43.700375+0000","last_undegraded":"2024-04-02T13:26:43.700375+0000","last_fullsized":"2024-04-02T13:26:43.700375+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:42.324949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.697799+0000","last_change":"2024-04-02T13:26:43.691912+0000","last_active":"2024-04-02T13:26:43.697799+0000","last_peered":"2024-04-02T13:26:43.697799+0000","last_clean":"2024-04-02T13:26:43.697799+0000","last_became_active":"2024-04-02T13:26:43.691582+0000","last_became_peered":"2024-04-02T13:26:43.691582+0000","last_unstale":"2024-04-02T13:26:43.697799+0000","last_undegraded":"2024-04-02T13:26:43.697799+0000","last_fullsized":"2024-04-02T13:26:43.697799+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:12.471858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.704575+0000","last_change":"2024-04-02T13:26:43.702959+0000","last_active":"2024-04-02T13:26:43.704575+0000","last_peered":"2024-04-02T13:26:43.704575+0000","last_clean":"2024-04-02T13:26:43.704575+0000","last_became_active":"2024-04-02T13:26:43.702241+0000","last_became_peered":"2024-04-02T13:26:43.702241+0000","last_unstale":"2024-04-02T13:26:43.704575+0000","last_undegraded":"2024-04-02T13:26:43.704575+0000","last_fullsized":"2024-04-02T13:26:43.704575+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:38.335594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.712728+0000","last_change":"2024-04-02T13:26:43.709761+0000","last_active":"2024-04-02T13:26:43.712728+0000","last_peered":"2024-04-02T13:26:43.712728+0000","last_clean":"2024-04-02T13:26:43.712728+0000","last_became_active":"2024-04-02T13:26:43.709174+0000","last_became_peered":"2024-04-02T13:26:43.709174+0000","last_unstale":"2024-04-02T13:26:43.712728+0000","last_undegraded":"2024-04-02T13:26:43.712728+0000","last_fullsized":"2024-04-02T13:26:43.712728+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:59.316306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711601+0000","last_change":"2024-04-02T13:26:43.709303+0000","last_active":"2024-04-02T13:26:43.711601+0000","last_peered":"2024-04-02T13:26:43.711601+0000","last_clean":"2024-04-02T13:26:43.711601+0000","last_became_active":"2024-04-02T13:26:43.709006+0000","last_became_peered":"2024-04-02T13:26:43.709006+0000","last_unstale":"2024-04-02T13:26:43.711601+0000","last_undegraded":"2024-04-02T13:26:43.711601+0000","last_fullsized":"2024-04-02T13:26:43.711601+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:13.010848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.699825+0000","last_change":"2024-04-02T13:26:43.694684+0000","last_active":"2024-04-02T13:26:43.699825+0000","last_peered":"2024-04-02T13:26:43.699825+0000","last_clean":"2024-04-02T13:26:43.699825+0000","last_became_active":"2024-04-02T13:26:43.694383+0000","last_became_peered":"2024-04-02T13:26:43.694383+0000","last_unstale":"2024-04-02T13:26:43.699825+0000","last_undegraded":"2024-04-02T13:26:43.699825+0000","last_fullsized":"2024-04-02T13:26:43.699825+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:29:52.365602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700608+0000","last_change":"2024-04-02T13:26:43.692321+0000","last_active":"2024-04-02T13:26:43.700608+0000","last_peered":"2024-04-02T13:26:43.700608+0000","last_clean":"2024-04-02T13:26:43.700608+0000","last_became_active":"2024-04-02T13:26:43.691809+0000","last_became_peered":"2024-04-02T13:26:43.691809+0000","last_unstale":"2024-04-02T13:26:43.700608+0000","last_undegraded":"2024-04-02T13:26:43.700608+0000","last_fullsized":"2024-04-02T13:26:43.700608+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:53:51.490836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700471+0000","last_change":"2024-04-02T13:26:43.698461+0000","last_active":"2024-04-02T13:26:43.700471+0000","last_peered":"2024-04-02T13:26:43.700471+0000","last_clean":"2024-04-02T13:26:43.700471+0000","last_became_active":"2024-04-02T13:26:43.697788+0000","last_became_peered":"2024-04-02T13:26:43.697788+0000","last_unstale":"2024-04-02T13:26:43.700471+0000","last_undegraded":"2024-04-02T13:26:43.700471+0000","last_fullsized":"2024-04-02T13:26:43.700471+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:34:50.588433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700522+0000","last_change":"2024-04-02T13:26:43.696188+0000","last_active":"2024-04-02T13:26:43.700522+0000","last_peered":"2024-04-02T13:26:43.700522+0000","last_clean":"2024-04-02T13:26:43.700522+0000","last_became_active":"2024-04-02T13:26:43.695288+0000","last_became_peered":"2024-04-02T13:26:43.695288+0000","last_unstale":"2024-04-02T13:26:43.700522+0000","last_undegraded":"2024-04-02T13:26:43.700522+0000","last_fullsized":"2024-04-02T13:26:43.700522+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:24.166578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.697635+0000","last_change":"2024-04-02T13:26:43.691316+0000","last_active":"2024-04-02T13:26:43.697635+0000","last_peered":"2024-04-02T13:26:43.697635+0000","last_clean":"2024-04-02T13:26:43.697635+0000","last_became_active":"2024-04-02T13:26:43.691046+0000","last_became_peered":"2024-04-02T13:26:43.691046+0000","last_unstale":"2024-04-02T13:26:43.697635+0000","last_undegraded":"2024-04-02T13:26:43.697635+0000","last_fullsized":"2024-04-02T13:26:43.697635+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:20:25.030316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.708650+0000","last_change":"2024-04-02T13:26:43.705006+0000","last_active":"2024-04-02T13:26:43.708650+0000","last_peered":"2024-04-02T13:26:43.708650+0000","last_clean":"2024-04-02T13:26:43.708650+0000","last_became_active":"2024-04-02T13:26:43.704714+0000","last_became_peered":"2024-04-02T13:26:43.704714+0000","last_unstale":"2024-04-02T13:26:43.708650+0000","last_undegraded":"2024-04-02T13:26:43.708650+0000","last_fullsized":"2024-04-02T13:26:43.708650+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:38.335594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.695809+0000","last_change":"2024-04-02T13:26:43.690039+0000","last_active":"2024-04-02T13:26:43.695809+0000","last_peered":"2024-04-02T13:26:43.695809+0000","last_clean":"2024-04-02T13:26:43.695809+0000","last_became_active":"2024-04-02T13:26:43.689756+0000","last_became_peered":"2024-04-02T13:26:43.689756+0000","last_unstale":"2024-04-02T13:26:43.695809+0000","last_undegraded":"2024-04-02T13:26:43.695809+0000","last_fullsized":"2024-04-02T13:26:43.695809+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:31:29.042976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698438+0000","last_change":"2024-04-02T13:26:43.693985+0000","last_active":"2024-04-02T13:26:43.698438+0000","last_peered":"2024-04-02T13:26:43.698438+0000","last_clean":"2024-04-02T13:26:43.698438+0000","last_became_active":"2024-04-02T13:26:43.693688+0000","last_became_peered":"2024-04-02T13:26:43.693688+0000","last_unstale":"2024-04-02T13:26:43.698438+0000","last_undegraded":"2024-04-02T13:26:43.698438+0000","last_fullsized":"2024-04-02T13:26:43.698438+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:45:15.159479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.703484+0000","last_change":"2024-04-02T13:26:43.700670+0000","last_active":"2024-04-02T13:26:43.703484+0000","last_peered":"2024-04-02T13:26:43.703484+0000","last_clean":"2024-04-02T13:26:43.703484+0000","last_became_active":"2024-04-02T13:26:43.700357+0000","last_became_peered":"2024-04-02T13:26:43.700357+0000","last_unstale":"2024-04-02T13:26:43.703484+0000","last_undegraded":"2024-04-02T13:26:43.703484+0000","last_fullsized":"2024-04-02T13:26:43.703484+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:34:50.588433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.695227+0000","last_change":"2024-04-02T13:26:43.693781+0000","last_active":"2024-04-02T13:26:43.695227+0000","last_peered":"2024-04-02T13:26:43.695227+0000","last_clean":"2024-04-02T13:26:43.695227+0000","last_became_active":"2024-04-02T13:26:43.693525+0000","last_became_peered":"2024-04-02T13:26:43.693525+0000","last_unstale":"2024-04-02T13:26:43.695227+0000","last_undegraded":"2024-04-02T13:26:43.695227+0000","last_fullsized":"2024-04-02T13:26:43.695227+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:19.794846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700418+0000","last_change":"2024-04-02T13:26:43.696105+0000","last_active":"2024-04-02T13:26:43.700418+0000","last_peered":"2024-04-02T13:26:43.700418+0000","last_clean":"2024-04-02T13:26:43.700418+0000","last_became_active":"2024-04-02T13:26:43.695842+0000","last_became_peered":"2024-04-02T13:26:43.695842+0000","last_unstale":"2024-04-02T13:26:43.700418+0000","last_undegraded":"2024-04-02T13:26:43.700418+0000","last_fullsized":"2024-04-02T13:26:43.700418+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:02:57.270282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700706+0000","last_change":"2024-04-02T13:26:43.692666+0000","last_active":"2024-04-02T13:26:43.700706+0000","last_peered":"2024-04-02T13:26:43.700706+0000","last_clean":"2024-04-02T13:26:43.700706+0000","last_became_active":"2024-04-02T13:26:43.692390+0000","last_became_peered":"2024-04-02T13:26:43.692390+0000","last_unstale":"2024-04-02T13:26:43.700706+0000","last_undegraded":"2024-04-02T13:26:43.700706+0000","last_fullsized":"2024-04-02T13:26:43.700706+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:20:10.846657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.697613+0000","last_change":"2024-04-02T13:26:43.690139+0000","last_active":"2024-04-02T13:26:43.697613+0000","last_peered":"2024-04-02T13:26:43.697613+0000","last_clean":"2024-04-02T13:26:43.697613+0000","last_became_active":"2024-04-02T13:26:43.689885+0000","last_became_peered":"2024-04-02T13:26:43.689885+0000","last_unstale":"2024-04-02T13:26:43.697613+0000","last_undegraded":"2024-04-02T13:26:43.697613+0000","last_fullsized":"2024-04-02T13:26:43.697613+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:37.222294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698523+0000","last_change":"2024-04-02T13:26:43.696971+0000","last_active":"2024-04-02T13:26:43.698523+0000","last_peered":"2024-04-02T13:26:43.698523+0000","last_clean":"2024-04-02T13:26:43.698523+0000","last_became_active":"2024-04-02T13:26:43.696696+0000","last_became_peered":"2024-04-02T13:26:43.696696+0000","last_unstale":"2024-04-02T13:26:43.698523+0000","last_undegraded":"2024-04-02T13:26:43.698523+0000","last_fullsized":"2024-04-02T13:26:43.698523+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:55:04.746672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.712620+0000","last_change":"2024-04-02T13:26:43.709356+0000","last_active":"2024-04-02T13:26:43.712620+0000","last_peered":"2024-04-02T13:26:43.712620+0000","last_clean":"2024-04-02T13:26:43.712620+0000","last_became_active":"2024-04-02T13:26:43.706242+0000","last_became_peered":"2024-04-02T13:26:43.706242+0000","last_unstale":"2024-04-02T13:26:43.712620+0000","last_undegraded":"2024-04-02T13:26:43.712620+0000","last_fullsized":"2024-04-02T13:26:43.712620+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:41:26.513359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.705285+0000","last_change":"2024-04-02T13:26:43.700549+0000","last_active":"2024-04-02T13:26:43.705285+0000","last_peered":"2024-04-02T13:26:43.705285+0000","last_clean":"2024-04-02T13:26:43.705285+0000","last_became_active":"2024-04-02T13:26:43.700203+0000","last_became_peered":"2024-04-02T13:26:43.700203+0000","last_unstale":"2024-04-02T13:26:43.705285+0000","last_undegraded":"2024-04-02T13:26:43.705285+0000","last_fullsized":"2024-04-02T13:26:43.705285+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:38:28.720291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"90'4","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.710933+0000","last_change":"2024-04-02T13:26:42.674033+0000","last_active":"2024-04-02T13:26:43.710933+0000","last_peered":"2024-04-02T13:26:43.710933+0000","last_clean":"2024-04-02T13:26:43.710933+0000","last_became_active":"2024-04-02T13:26:42.673780+0000","last_became_peered":"2024-04-02T13:26:42.673780+0000","last_unstale":"2024-04-02T13:26:43.710933+0000","last_undegraded":"2024-04-02T13:26:43.710933+0000","last_fullsized":"2024-04-02T13:26:43.710933+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:36.033369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.709474+0000","last_change":"2024-04-02T13:26:43.707026+0000","last_active":"2024-04-02T13:26:43.709474+0000","last_peered":"2024-04-02T13:26:43.709474+0000","last_clean":"2024-04-02T13:26:43.709474+0000","last_became_active":"2024-04-02T13:26:43.706718+0000","last_became_peered":"2024-04-02T13:26:43.706718+0000","last_unstale":"2024-04-02T13:26:43.709474+0000","last_undegraded":"2024-04-02T13:26:43.709474+0000","last_fullsized":"2024-04-02T13:26:43.709474+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:21:28.111681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"90'1","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.708517+0000","last_change":"2024-04-02T13:26:42.667887+0000","last_active":"2024-04-02T13:26:43.708517+0000","last_peered":"2024-04-02T13:26:43.708517+0000","last_clean":"2024-04-02T13:26:43.708517+0000","last_became_active":"2024-04-02T13:26:42.667619+0000","last_became_peered":"2024-04-02T13:26:42.667619+0000","last_unstale":"2024-04-02T13:26:43.708517+0000","last_undegraded":"2024-04-02T13:26:43.708517+0000","last_fullsized":"2024-04-02T13:26:43.708517+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:41.538507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.706226+0000","last_change":"2024-04-02T13:26:43.700743+0000","last_active":"2024-04-02T13:26:43.706226+0000","last_peered":"2024-04-02T13:26:43.706226+0000","last_clean":"2024-04-02T13:26:43.706226+0000","last_became_active":"2024-04-02T13:26:43.700374+0000","last_became_peered":"2024-04-02T13:26:43.700374+0000","last_unstale":"2024-04-02T13:26:43.706226+0000","last_undegraded":"2024-04-02T13:26:43.706226+0000","last_fullsized":"2024-04-02T13:26:43.706226+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:59:22.715019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"90'2","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711076+0000","last_change":"2024-04-02T13:26:42.670504+0000","last_active":"2024-04-02T13:26:43.711076+0000","last_peered":"2024-04-02T13:26:43.711076+0000","last_clean":"2024-04-02T13:26:43.711076+0000","last_became_active":"2024-04-02T13:26:42.670203+0000","last_became_peered":"2024-04-02T13:26:42.670203+0000","last_unstale":"2024-04-02T13:26:43.711076+0000","last_undegraded":"2024-04-02T13:26:43.711076+0000","last_fullsized":"2024-04-02T13:26:43.711076+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:00:12.693543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.708680+0000","last_change":"2024-04-02T13:26:43.699888+0000","last_active":"2024-04-02T13:26:43.708680+0000","last_peered":"2024-04-02T13:26:43.708680+0000","last_clean":"2024-04-02T13:26:43.708680+0000","last_became_active":"2024-04-02T13:26:43.699573+0000","last_became_peered":"2024-04-02T13:26:43.699573+0000","last_unstale":"2024-04-02T13:26:43.708680+0000","last_undegraded":"2024-04-02T13:26:43.708680+0000","last_fullsized":"2024-04-02T13:26:43.708680+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:36:28.322738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"90'2","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.709203+0000","last_change":"2024-04-02T13:26:42.669152+0000","last_active":"2024-04-02T13:26:43.709203+0000","last_peered":"2024-04-02T13:26:43.709203+0000","last_clean":"2024-04-02T13:26:43.709203+0000","last_became_active":"2024-04-02T13:26:42.668843+0000","last_became_peered":"2024-04-02T13:26:42.668843+0000","last_unstale":"2024-04-02T13:26:43.709203+0000","last_undegraded":"2024-04-02T13:26:43.709203+0000","last_fullsized":"2024-04-02T13:26:43.709203+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:41.318403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.703161+0000","last_change":"2024-04-02T13:26:43.699149+0000","last_active":"2024-04-02T13:26:43.703161+0000","last_peered":"2024-04-02T13:26:43.703161+0000","last_clean":"2024-04-02T13:26:43.703161+0000","last_became_active":"2024-04-02T13:26:43.698645+0000","last_became_peered":"2024-04-02T13:26:43.698645+0000","last_unstale":"2024-04-02T13:26:43.703161+0000","last_undegraded":"2024-04-02T13:26:43.703161+0000","last_fullsized":"2024-04-02T13:26:43.703161+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:01:34.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"90'1","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.708330+0000","last_change":"2024-04-02T13:26:42.670719+0000","last_active":"2024-04-02T13:26:43.708330+0000","last_peered":"2024-04-02T13:26:43.708330+0000","last_clean":"2024-04-02T13:26:43.708330+0000","last_became_active":"2024-04-02T13:26:42.670419+0000","last_became_peered":"2024-04-02T13:26:42.670419+0000","last_unstale":"2024-04-02T13:26:43.708330+0000","last_undegraded":"2024-04-02T13:26:43.708330+0000","last_fullsized":"2024-04-02T13:26:43.708330+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:56.532143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711344+0000","last_change":"2024-04-02T13:26:43.707801+0000","last_active":"2024-04-02T13:26:43.711344+0000","last_peered":"2024-04-02T13:26:43.711344+0000","last_clean":"2024-04-02T13:26:43.711344+0000","last_became_active":"2024-04-02T13:26:43.707517+0000","last_became_peered":"2024-04-02T13:26:43.707517+0000","last_unstale":"2024-04-02T13:26:43.711344+0000","last_undegraded":"2024-04-02T13:26:43.711344+0000","last_fullsized":"2024-04-02T13:26:43.711344+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:13.852279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"90'1","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.714872+0000","last_change":"2024-04-02T13:26:42.670860+0000","last_active":"2024-04-02T13:26:43.714872+0000","last_peered":"2024-04-02T13:26:43.714872+0000","last_clean":"2024-04-02T13:26:43.714872+0000","last_became_active":"2024-04-02T13:26:42.670561+0000","last_became_peered":"2024-04-02T13:26:42.670561+0000","last_unstale":"2024-04-02T13:26:43.714872+0000","last_undegraded":"2024-04-02T13:26:43.714872+0000","last_fullsized":"2024-04-02T13:26:43.714872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:24:31.378662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.705378+0000","last_change":"2024-04-02T13:26:43.701049+0000","last_active":"2024-04-02T13:26:43.705378+0000","last_peered":"2024-04-02T13:26:43.705378+0000","last_clean":"2024-04-02T13:26:43.705378+0000","last_became_active":"2024-04-02T13:26:43.700742+0000","last_became_peered":"2024-04-02T13:26:43.700742+0000","last_unstale":"2024-04-02T13:26:43.705378+0000","last_undegraded":"2024-04-02T13:26:43.705378+0000","last_fullsized":"2024-04-02T13:26:43.705378+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:29:52.365602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.678887+0000","last_change":"2024-04-02T13:26:42.672938+0000","last_active":"2024-04-02T13:26:43.678887+0000","last_peered":"2024-04-02T13:26:43.678887+0000","last_clean":"2024-04-02T13:26:43.678887+0000","last_became_active":"2024-04-02T13:26:42.672660+0000","last_became_peered":"2024-04-02T13:26:42.672660+0000","last_unstale":"2024-04-02T13:26:43.678887+0000","last_undegraded":"2024-04-02T13:26:43.678887+0000","last_fullsized":"2024-04-02T13:26:43.678887+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:00.996929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700395+0000","last_change":"2024-04-02T13:26:43.698259+0000","last_active":"2024-04-02T13:26:43.700395+0000","last_peered":"2024-04-02T13:26:43.700395+0000","last_clean":"2024-04-02T13:26:43.700395+0000","last_became_active":"2024-04-02T13:26:43.697993+0000","last_became_peered":"2024-04-02T13:26:43.697993+0000","last_unstale":"2024-04-02T13:26:43.700395+0000","last_undegraded":"2024-04-02T13:26:43.700395+0000","last_fullsized":"2024-04-02T13:26:43.700395+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:37.039915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.682310+0000","last_change":"2024-04-02T13:26:42.666442+0000","last_active":"2024-04-02T13:26:43.682310+0000","last_peered":"2024-04-02T13:26:43.682310+0000","last_clean":"2024-04-02T13:26:43.682310+0000","last_became_active":"2024-04-02T13:26:42.666115+0000","last_became_peered":"2024-04-02T13:26:42.666115+0000","last_unstale":"2024-04-02T13:26:43.682310+0000","last_undegraded":"2024-04-02T13:26:43.682310+0000","last_fullsized":"2024-04-02T13:26:43.682310+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:55:11.259335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.697807+0000","last_change":"2024-04-02T13:26:43.689573+0000","last_active":"2024-04-02T13:26:43.697807+0000","last_peered":"2024-04-02T13:26:43.697807+0000","last_clean":"2024-04-02T13:26:43.697807+0000","last_became_active":"2024-04-02T13:26:43.689237+0000","last_became_peered":"2024-04-02T13:26:43.689237+0000","last_unstale":"2024-04-02T13:26:43.697807+0000","last_undegraded":"2024-04-02T13:26:43.697807+0000","last_fullsized":"2024-04-02T13:26:43.697807+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:01:35.449213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.686392+0000","last_change":"2024-04-02T13:26:42.671965+0000","last_active":"2024-04-02T13:26:43.686392+0000","last_peered":"2024-04-02T13:26:43.686392+0000","last_clean":"2024-04-02T13:26:43.686392+0000","last_became_active":"2024-04-02T13:26:42.671703+0000","last_became_peered":"2024-04-02T13:26:42.671703+0000","last_unstale":"2024-04-02T13:26:43.686392+0000","last_undegraded":"2024-04-02T13:26:43.686392+0000","last_fullsized":"2024-04-02T13:26:43.686392+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:41:21.985036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700651+0000","last_change":"2024-04-02T13:26:43.693977+0000","last_active":"2024-04-02T13:26:43.700651+0000","last_peered":"2024-04-02T13:26:43.700651+0000","last_clean":"2024-04-02T13:26:43.700651+0000","last_became_active":"2024-04-02T13:26:43.693717+0000","last_became_peered":"2024-04-02T13:26:43.693717+0000","last_unstale":"2024-04-02T13:26:43.700651+0000","last_undegraded":"2024-04-02T13:26:43.700651+0000","last_fullsized":"2024-04-02T13:26:43.700651+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:38:28.720291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.679385+0000","last_change":"2024-04-02T13:26:42.673632+0000","last_active":"2024-04-02T13:26:43.679385+0000","last_peered":"2024-04-02T13:26:43.679385+0000","last_clean":"2024-04-02T13:26:43.679385+0000","last_became_active":"2024-04-02T13:26:42.673357+0000","last_became_peered":"2024-04-02T13:26:42.673357+0000","last_unstale":"2024-04-02T13:26:43.679385+0000","last_undegraded":"2024-04-02T13:26:43.679385+0000","last_fullsized":"2024-04-02T13:26:43.679385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:00:12.693543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.699099+0000","last_change":"2024-04-02T13:26:43.692518+0000","last_active":"2024-04-02T13:26:43.699099+0000","last_peered":"2024-04-02T13:26:43.699099+0000","last_clean":"2024-04-02T13:26:43.699099+0000","last_became_active":"2024-04-02T13:26:43.692236+0000","last_became_peered":"2024-04-02T13:26:43.692236+0000","last_unstale":"2024-04-02T13:26:43.699099+0000","last_undegraded":"2024-04-02T13:26:43.699099+0000","last_fullsized":"2024-04-02T13:26:43.699099+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:20:10.846657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"90'3","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711502+0000","last_change":"2024-04-02T13:26:42.667969+0000","last_active":"2024-04-02T13:26:43.711502+0000","last_peered":"2024-04-02T13:26:43.711502+0000","last_clean":"2024-04-02T13:26:43.711502+0000","last_became_active":"2024-04-02T13:26:42.667696+0000","last_became_peered":"2024-04-02T13:26:42.667696+0000","last_unstale":"2024-04-02T13:26:43.711502+0000","last_undegraded":"2024-04-02T13:26:43.711502+0000","last_fullsized":"2024-04-02T13:26:43.711502+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:51.726256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700518+0000","last_change":"2024-04-02T13:26:43.697631+0000","last_active":"2024-04-02T13:26:43.700518+0000","last_peered":"2024-04-02T13:26:43.700518+0000","last_clean":"2024-04-02T13:26:43.700518+0000","last_became_active":"2024-04-02T13:26:43.697278+0000","last_became_peered":"2024-04-02T13:26:43.697278+0000","last_unstale":"2024-04-02T13:26:43.700518+0000","last_undegraded":"2024-04-02T13:26:43.700518+0000","last_fullsized":"2024-04-02T13:26:43.700518+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:24.166578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"90'6","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.716507+0000","last_change":"2024-04-02T13:26:42.666535+0000","last_active":"2024-04-02T13:26:43.716507+0000","last_peered":"2024-04-02T13:26:43.716507+0000","last_clean":"2024-04-02T13:26:43.716507+0000","last_became_active":"2024-04-02T13:26:42.666228+0000","last_became_peered":"2024-04-02T13:26:42.666228+0000","last_unstale":"2024-04-02T13:26:43.716507+0000","last_undegraded":"2024-04-02T13:26:43.716507+0000","last_fullsized":"2024-04-02T13:26:43.716507+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:47:58.212185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.703179+0000","last_change":"2024-04-02T13:26:43.699774+0000","last_active":"2024-04-02T13:26:43.703179+0000","last_peered":"2024-04-02T13:26:43.703179+0000","last_clean":"2024-04-02T13:26:43.703179+0000","last_became_active":"2024-04-02T13:26:43.699383+0000","last_became_peered":"2024-04-02T13:26:43.699383+0000","last_unstale":"2024-04-02T13:26:43.703179+0000","last_undegraded":"2024-04-02T13:26:43.703179+0000","last_fullsized":"2024-04-02T13:26:43.703179+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:44.626723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"90'2","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711849+0000","last_change":"2024-04-02T13:26:42.665005+0000","last_active":"2024-04-02T13:26:43.711849+0000","last_peered":"2024-04-02T13:26:43.711849+0000","last_clean":"2024-04-02T13:26:43.711849+0000","last_became_active":"2024-04-02T13:26:42.664711+0000","last_became_peered":"2024-04-02T13:26:42.664711+0000","last_unstale":"2024-04-02T13:26:43.711849+0000","last_undegraded":"2024-04-02T13:26:43.711849+0000","last_fullsized":"2024-04-02T13:26:43.711849+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:00.996929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700308+0000","last_change":"2024-04-02T13:26:43.694268+0000","last_active":"2024-04-02T13:26:43.700308+0000","last_peered":"2024-04-02T13:26:43.700308+0000","last_clean":"2024-04-02T13:26:43.700308+0000","last_became_active":"2024-04-02T13:26:43.694007+0000","last_became_peered":"2024-04-02T13:26:43.694007+0000","last_unstale":"2024-04-02T13:26:43.700308+0000","last_undegraded":"2024-04-02T13:26:43.700308+0000","last_fullsized":"2024-04-02T13:26:43.700308+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:02:57.270282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.688082+0000","last_change":"2024-04-02T13:26:42.673421+0000","last_active":"2024-04-02T13:26:43.688082+0000","last_peered":"2024-04-02T13:26:43.688082+0000","last_clean":"2024-04-02T13:26:43.688082+0000","last_became_active":"2024-04-02T13:26:42.673080+0000","last_became_peered":"2024-04-02T13:26:42.673080+0000","last_unstale":"2024-04-02T13:26:43.688082+0000","last_undegraded":"2024-04-02T13:26:43.688082+0000","last_fullsized":"2024-04-02T13:26:43.688082+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:06:30.310861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.692603+0000","last_change":"2024-04-02T13:26:43.691164+0000","last_active":"2024-04-02T13:26:43.692603+0000","last_peered":"2024-04-02T13:26:43.692603+0000","last_clean":"2024-04-02T13:26:43.692603+0000","last_became_active":"2024-04-02T13:26:43.690867+0000","last_became_peered":"2024-04-02T13:26:43.690867+0000","last_unstale":"2024-04-02T13:26:43.692603+0000","last_undegraded":"2024-04-02T13:26:43.692603+0000","last_fullsized":"2024-04-02T13:26:43.692603+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:09:15.323395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.684959+0000","last_change":"2024-04-02T13:26:42.665887+0000","last_active":"2024-04-02T13:26:43.684959+0000","last_peered":"2024-04-02T13:26:43.684959+0000","last_clean":"2024-04-02T13:26:43.684959+0000","last_became_active":"2024-04-02T13:26:42.665587+0000","last_became_peered":"2024-04-02T13:26:42.665587+0000","last_unstale":"2024-04-02T13:26:43.684959+0000","last_undegraded":"2024-04-02T13:26:43.684959+0000","last_fullsized":"2024-04-02T13:26:43.684959+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:11.465313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.692645+0000","last_change":"2024-04-02T13:26:43.691227+0000","last_active":"2024-04-02T13:26:43.692645+0000","last_peered":"2024-04-02T13:26:43.692645+0000","last_clean":"2024-04-02T13:26:43.692645+0000","last_became_active":"2024-04-02T13:26:43.690921+0000","last_became_peered":"2024-04-02T13:26:43.690921+0000","last_unstale":"2024-04-02T13:26:43.692645+0000","last_undegraded":"2024-04-02T13:26:43.692645+0000","last_fullsized":"2024-04-02T13:26:43.692645+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:13:22.417286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"90'8","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.715645+0000","last_change":"2024-04-02T13:26:42.666944+0000","last_active":"2024-04-02T13:26:43.715645+0000","last_peered":"2024-04-02T13:26:43.715645+0000","last_clean":"2024-04-02T13:26:43.715645+0000","last_became_active":"2024-04-02T13:26:42.666670+0000","last_became_peered":"2024-04-02T13:26:42.666670+0000","last_unstale":"2024-04-02T13:26:43.715645+0000","last_undegraded":"2024-04-02T13:26:43.715645+0000","last_fullsized":"2024-04-02T13:26:43.715645+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:53:50.484291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711009+0000","last_change":"2024-04-02T13:26:43.705898+0000","last_active":"2024-04-02T13:26:43.711009+0000","last_peered":"2024-04-02T13:26:43.711009+0000","last_clean":"2024-04-02T13:26:43.711009+0000","last_became_active":"2024-04-02T13:26:43.705566+0000","last_became_peered":"2024-04-02T13:26:43.705566+0000","last_unstale":"2024-04-02T13:26:43.711009+0000","last_undegraded":"2024-04-02T13:26:43.711009+0000","last_fullsized":"2024-04-02T13:26:43.711009+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:27:11.031758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"90'1","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711206+0000","last_change":"2024-04-02T13:26:42.671054+0000","last_active":"2024-04-02T13:26:43.711206+0000","last_peered":"2024-04-02T13:26:43.711206+0000","last_clean":"2024-04-02T13:26:43.711206+0000","last_became_active":"2024-04-02T13:26:42.670739+0000","last_became_peered":"2024-04-02T13:26:42.670739+0000","last_unstale":"2024-04-02T13:26:43.711206+0000","last_undegraded":"2024-04-02T13:26:43.711206+0000","last_fullsized":"2024-04-02T13:26:43.711206+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:34:49.581887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698313+0000","last_change":"2024-04-02T13:26:43.692814+0000","last_active":"2024-04-02T13:26:43.698313+0000","last_peered":"2024-04-02T13:26:43.698313+0000","last_clean":"2024-04-02T13:26:43.698313+0000","last_became_active":"2024-04-02T13:26:43.692537+0000","last_became_peered":"2024-04-02T13:26:43.692537+0000","last_unstale":"2024-04-02T13:26:43.698313+0000","last_undegraded":"2024-04-02T13:26:43.698313+0000","last_fullsized":"2024-04-02T13:26:43.698313+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:50.758013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"90'2","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711718+0000","last_change":"2024-04-02T13:26:42.667987+0000","last_active":"2024-04-02T13:26:43.711718+0000","last_peered":"2024-04-02T13:26:43.711718+0000","last_clean":"2024-04-02T13:26:43.711718+0000","last_became_active":"2024-04-02T13:26:42.667714+0000","last_became_peered":"2024-04-02T13:26:42.667714+0000","last_unstale":"2024-04-02T13:26:43.711718+0000","last_undegraded":"2024-04-02T13:26:43.711718+0000","last_fullsized":"2024-04-02T13:26:43.711718+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:47:58.212185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700807+0000","last_change":"2024-04-02T13:26:43.696729+0000","last_active":"2024-04-02T13:26:43.700807+0000","last_peered":"2024-04-02T13:26:43.700807+0000","last_clean":"2024-04-02T13:26:43.700807+0000","last_became_active":"2024-04-02T13:26:43.696334+0000","last_became_peered":"2024-04-02T13:26:43.696334+0000","last_unstale":"2024-04-02T13:26:43.700807+0000","last_undegraded":"2024-04-02T13:26:43.700807+0000","last_fullsized":"2024-04-02T13:26:43.700807+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:37.039915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"90'1","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711419+0000","last_change":"2024-04-02T13:26:42.667417+0000","last_active":"2024-04-02T13:26:43.711419+0000","last_peered":"2024-04-02T13:26:43.711419+0000","last_clean":"2024-04-02T13:26:43.711419+0000","last_became_active":"2024-04-02T13:26:42.667090+0000","last_became_peered":"2024-04-02T13:26:42.667090+0000","last_unstale":"2024-04-02T13:26:43.711419+0000","last_undegraded":"2024-04-02T13:26:43.711419+0000","last_fullsized":"2024-04-02T13:26:43.711419+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:36:03.171364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.706074+0000","last_change":"2024-04-02T13:26:43.703003+0000","last_active":"2024-04-02T13:26:43.706074+0000","last_peered":"2024-04-02T13:26:43.706074+0000","last_clean":"2024-04-02T13:26:43.706074+0000","last_became_active":"2024-04-02T13:26:43.702738+0000","last_became_peered":"2024-04-02T13:26:43.702738+0000","last_unstale":"2024-04-02T13:26:43.706074+0000","last_undegraded":"2024-04-02T13:26:43.706074+0000","last_fullsized":"2024-04-02T13:26:43.706074+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:36.661090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"90'2","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711133+0000","last_change":"2024-04-02T13:26:42.673224+0000","last_active":"2024-04-02T13:26:43.711133+0000","last_peered":"2024-04-02T13:26:43.711133+0000","last_clean":"2024-04-02T13:26:43.711133+0000","last_became_active":"2024-04-02T13:26:42.672899+0000","last_became_peered":"2024-04-02T13:26:42.672899+0000","last_unstale":"2024-04-02T13:26:43.711133+0000","last_undegraded":"2024-04-02T13:26:43.711133+0000","last_fullsized":"2024-04-02T13:26:43.711133+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:44:43.333598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.690161+0000","last_change":"2024-04-02T13:26:43.688467+0000","last_active":"2024-04-02T13:26:43.690161+0000","last_peered":"2024-04-02T13:26:43.690161+0000","last_clean":"2024-04-02T13:26:43.690161+0000","last_became_active":"2024-04-02T13:26:43.688176+0000","last_became_peered":"2024-04-02T13:26:43.688176+0000","last_unstale":"2024-04-02T13:26:43.690161+0000","last_undegraded":"2024-04-02T13:26:43.690161+0000","last_fullsized":"2024-04-02T13:26:43.690161+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:33:52.326948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.683071+0000","last_change":"2024-04-02T13:26:42.669537+0000","last_active":"2024-04-02T13:26:43.683071+0000","last_peered":"2024-04-02T13:26:43.683071+0000","last_clean":"2024-04-02T13:26:43.683071+0000","last_became_active":"2024-04-02T13:26:42.669264+0000","last_became_peered":"2024-04-02T13:26:42.669264+0000","last_unstale":"2024-04-02T13:26:43.683071+0000","last_undegraded":"2024-04-02T13:26:43.683071+0000","last_fullsized":"2024-04-02T13:26:43.683071+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:09:13.881054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698481+0000","last_change":"2024-04-02T13:26:43.693855+0000","last_active":"2024-04-02T13:26:43.698481+0000","last_peered":"2024-04-02T13:26:43.698481+0000","last_clean":"2024-04-02T13:26:43.698481+0000","last_became_active":"2024-04-02T13:26:43.693548+0000","last_became_peered":"2024-04-02T13:26:43.693548+0000","last_unstale":"2024-04-02T13:26:43.698481+0000","last_undegraded":"2024-04-02T13:26:43.698481+0000","last_fullsized":"2024-04-02T13:26:43.698481+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:10:54.661524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.684724+0000","last_change":"2024-04-02T13:26:42.676118+0000","last_active":"2024-04-02T13:26:43.684724+0000","last_peered":"2024-04-02T13:26:43.684724+0000","last_clean":"2024-04-02T13:26:43.684724+0000","last_became_active":"2024-04-02T13:26:42.675807+0000","last_became_peered":"2024-04-02T13:26:42.675807+0000","last_unstale":"2024-04-02T13:26:43.684724+0000","last_undegraded":"2024-04-02T13:26:43.684724+0000","last_fullsized":"2024-04-02T13:26:43.684724+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:40.845157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.705383+0000","last_change":"2024-04-02T13:26:43.695647+0000","last_active":"2024-04-02T13:26:43.705383+0000","last_peered":"2024-04-02T13:26:43.705383+0000","last_clean":"2024-04-02T13:26:43.705383+0000","last_became_active":"2024-04-02T13:26:43.695366+0000","last_became_peered":"2024-04-02T13:26:43.695366+0000","last_unstale":"2024-04-02T13:26:43.705383+0000","last_undegraded":"2024-04-02T13:26:43.705383+0000","last_fullsized":"2024-04-02T13:26:43.705383+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:31:29.042976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.682028+0000","last_change":"2024-04-02T13:26:42.677202+0000","last_active":"2024-04-02T13:26:43.682028+0000","last_peered":"2024-04-02T13:26:43.682028+0000","last_clean":"2024-04-02T13:26:43.682028+0000","last_became_active":"2024-04-02T13:26:42.676939+0000","last_became_peered":"2024-04-02T13:26:42.676939+0000","last_unstale":"2024-04-02T13:26:43.682028+0000","last_undegraded":"2024-04-02T13:26:43.682028+0000","last_fullsized":"2024-04-02T13:26:43.682028+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:56:00.746697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.701494+0000","last_change":"2024-04-02T13:26:43.697219+0000","last_active":"2024-04-02T13:26:43.701494+0000","last_peered":"2024-04-02T13:26:43.701494+0000","last_clean":"2024-04-02T13:26:43.701494+0000","last_became_active":"2024-04-02T13:26:43.696965+0000","last_became_peered":"2024-04-02T13:26:43.696965+0000","last_unstale":"2024-04-02T13:26:43.701494+0000","last_undegraded":"2024-04-02T13:26:43.701494+0000","last_fullsized":"2024-04-02T13:26:43.701494+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:44.626723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"90'2","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711847+0000","last_change":"2024-04-02T13:26:42.666211+0000","last_active":"2024-04-02T13:26:43.711847+0000","last_peered":"2024-04-02T13:26:43.711847+0000","last_clean":"2024-04-02T13:26:43.711847+0000","last_became_active":"2024-04-02T13:26:42.665907+0000","last_became_peered":"2024-04-02T13:26:42.665907+0000","last_unstale":"2024-04-02T13:26:43.711847+0000","last_undegraded":"2024-04-02T13:26:43.711847+0000","last_fullsized":"2024-04-02T13:26:43.711847+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:31:28.036430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.708497+0000","last_change":"2024-04-02T13:26:43.706420+0000","last_active":"2024-04-02T13:26:43.708497+0000","last_peered":"2024-04-02T13:26:43.708497+0000","last_clean":"2024-04-02T13:26:43.708497+0000","last_became_active":"2024-04-02T13:26:43.705247+0000","last_became_peered":"2024-04-02T13:26:43.705247+0000","last_unstale":"2024-04-02T13:26:43.708497+0000","last_undegraded":"2024-04-02T13:26:43.708497+0000","last_fullsized":"2024-04-02T13:26:43.708497+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:27:30.825398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.687236+0000","last_change":"2024-04-02T13:26:42.667415+0000","last_active":"2024-04-02T13:26:43.687236+0000","last_peered":"2024-04-02T13:26:43.687236+0000","last_clean":"2024-04-02T13:26:43.687236+0000","last_became_active":"2024-04-02T13:26:42.667058+0000","last_became_peered":"2024-04-02T13:26:42.667058+0000","last_unstale":"2024-04-02T13:26:43.687236+0000","last_undegraded":"2024-04-02T13:26:43.687236+0000","last_fullsized":"2024-04-02T13:26:43.687236+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:27:17.217234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.705243+0000","last_change":"2024-04-02T13:26:43.696526+0000","last_active":"2024-04-02T13:26:43.705243+0000","last_peered":"2024-04-02T13:26:43.705243+0000","last_clean":"2024-04-02T13:26:43.705243+0000","last_became_active":"2024-04-02T13:26:43.696261+0000","last_became_peered":"2024-04-02T13:26:43.696261+0000","last_unstale":"2024-04-02T13:26:43.705243+0000","last_undegraded":"2024-04-02T13:26:43.705243+0000","last_fullsized":"2024-04-02T13:26:43.705243+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:20:10.846657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.686268+0000","last_change":"2024-04-02T13:26:42.667396+0000","last_active":"2024-04-02T13:26:43.686268+0000","last_peered":"2024-04-02T13:26:43.686268+0000","last_clean":"2024-04-02T13:26:43.686268+0000","last_became_active":"2024-04-02T13:26:42.667065+0000","last_became_peered":"2024-04-02T13:26:42.667065+0000","last_unstale":"2024-04-02T13:26:43.686268+0000","last_undegraded":"2024-04-02T13:26:43.686268+0000","last_fullsized":"2024-04-02T13:26:43.686268+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:37.321974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698459+0000","last_change":"2024-04-02T13:26:43.694139+0000","last_active":"2024-04-02T13:26:43.698459+0000","last_peered":"2024-04-02T13:26:43.698459+0000","last_clean":"2024-04-02T13:26:43.698459+0000","last_became_active":"2024-04-02T13:26:43.693873+0000","last_became_peered":"2024-04-02T13:26:43.693873+0000","last_unstale":"2024-04-02T13:26:43.698459+0000","last_undegraded":"2024-04-02T13:26:43.698459+0000","last_fullsized":"2024-04-02T13:26:43.698459+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:57:42.834961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.688813+0000","last_change":"2024-04-02T13:26:42.669705+0000","last_active":"2024-04-02T13:26:43.688813+0000","last_peered":"2024-04-02T13:26:43.688813+0000","last_clean":"2024-04-02T13:26:43.688813+0000","last_became_active":"2024-04-02T13:26:42.669449+0000","last_became_peered":"2024-04-02T13:26:42.669449+0000","last_unstale":"2024-04-02T13:26:43.688813+0000","last_undegraded":"2024-04-02T13:26:43.688813+0000","last_fullsized":"2024-04-02T13:26:43.688813+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:24:00.307265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.695907+0000","last_change":"2024-04-02T13:26:43.694479+0000","last_active":"2024-04-02T13:26:43.695907+0000","last_peered":"2024-04-02T13:26:43.695907+0000","last_clean":"2024-04-02T13:26:43.695907+0000","last_became_active":"2024-04-02T13:26:43.694210+0000","last_became_peered":"2024-04-02T13:26:43.694210+0000","last_unstale":"2024-04-02T13:26:43.695907+0000","last_undegraded":"2024-04-02T13:26:43.695907+0000","last_fullsized":"2024-04-02T13:26:43.695907+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:31:56.365732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.688891+0000","last_change":"2024-04-02T13:26:42.675125+0000","last_active":"2024-04-02T13:26:43.688891+0000","last_peered":"2024-04-02T13:26:43.688891+0000","last_clean":"2024-04-02T13:26:43.688891+0000","last_became_active":"2024-04-02T13:26:42.674431+0000","last_became_peered":"2024-04-02T13:26:42.674431+0000","last_unstale":"2024-04-02T13:26:43.688891+0000","last_undegraded":"2024-04-02T13:26:43.688891+0000","last_fullsized":"2024-04-02T13:26:43.688891+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:00.996929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700384+0000","last_change":"2024-04-02T13:26:43.693504+0000","last_active":"2024-04-02T13:26:43.700384+0000","last_peered":"2024-04-02T13:26:43.700384+0000","last_clean":"2024-04-02T13:26:43.700384+0000","last_became_active":"2024-04-02T13:26:43.693241+0000","last_became_peered":"2024-04-02T13:26:43.693241+0000","last_unstale":"2024-04-02T13:26:43.700384+0000","last_undegraded":"2024-04-02T13:26:43.700384+0000","last_fullsized":"2024-04-02T13:26:43.700384+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:38:28.720291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.685377+0000","last_change":"2024-04-02T13:26:42.673148+0000","last_active":"2024-04-02T13:26:43.685377+0000","last_peered":"2024-04-02T13:26:43.685377+0000","last_clean":"2024-04-02T13:26:43.685377+0000","last_became_active":"2024-04-02T13:26:42.672851+0000","last_became_peered":"2024-04-02T13:26:42.672851+0000","last_unstale":"2024-04-02T13:26:43.685377+0000","last_undegraded":"2024-04-02T13:26:43.685377+0000","last_fullsized":"2024-04-02T13:26:43.685377+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:05:35.742023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698389+0000","last_change":"2024-04-02T13:26:43.693436+0000","last_active":"2024-04-02T13:26:43.698389+0000","last_peered":"2024-04-02T13:26:43.698389+0000","last_clean":"2024-04-02T13:26:43.698389+0000","last_became_active":"2024-04-02T13:26:43.693135+0000","last_became_peered":"2024-04-02T13:26:43.693135+0000","last_unstale":"2024-04-02T13:26:43.698389+0000","last_undegraded":"2024-04-02T13:26:43.698389+0000","last_fullsized":"2024-04-02T13:26:43.698389+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:01:34.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.688605+0000","last_change":"2024-04-02T13:26:42.665866+0000","last_active":"2024-04-02T13:26:43.688605+0000","last_peered":"2024-04-02T13:26:43.688605+0000","last_clean":"2024-04-02T13:26:43.688605+0000","last_became_active":"2024-04-02T13:26:42.665533+0000","last_became_peered":"2024-04-02T13:26:42.665533+0000","last_unstale":"2024-04-02T13:26:43.688605+0000","last_undegraded":"2024-04-02T13:26:43.688605+0000","last_fullsized":"2024-04-02T13:26:43.688605+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:23.160032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.705361+0000","last_change":"2024-04-02T13:26:43.692181+0000","last_active":"2024-04-02T13:26:43.705361+0000","last_peered":"2024-04-02T13:26:43.705361+0000","last_clean":"2024-04-02T13:26:43.705361+0000","last_became_active":"2024-04-02T13:26:43.691862+0000","last_became_peered":"2024-04-02T13:26:43.691862+0000","last_unstale":"2024-04-02T13:26:43.705361+0000","last_undegraded":"2024-04-02T13:26:43.705361+0000","last_fullsized":"2024-04-02T13:26:43.705361+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:00:13.700089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.683053+0000","last_change":"2024-04-02T13:26:42.666239+0000","last_active":"2024-04-02T13:26:43.683053+0000","last_peered":"2024-04-02T13:26:43.683053+0000","last_clean":"2024-04-02T13:26:43.683053+0000","last_became_active":"2024-04-02T13:26:42.665748+0000","last_became_peered":"2024-04-02T13:26:42.665748+0000","last_unstale":"2024-04-02T13:26:43.683053+0000","last_undegraded":"2024-04-02T13:26:43.683053+0000","last_fullsized":"2024-04-02T13:26:43.683053+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:06:30.310861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700779+0000","last_change":"2024-04-02T13:26:43.694733+0000","last_active":"2024-04-02T13:26:43.700779+0000","last_peered":"2024-04-02T13:26:43.700779+0000","last_clean":"2024-04-02T13:26:43.700779+0000","last_became_active":"2024-04-02T13:26:43.694462+0000","last_became_peered":"2024-04-02T13:26:43.694462+0000","last_unstale":"2024-04-02T13:26:43.700779+0000","last_undegraded":"2024-04-02T13:26:43.700779+0000","last_fullsized":"2024-04-02T13:26:43.700779+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:29:52.365602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.684160+0000","last_change":"2024-04-02T13:26:42.671125+0000","last_active":"2024-04-02T13:26:43.684160+0000","last_peered":"2024-04-02T13:26:43.684160+0000","last_clean":"2024-04-02T13:26:43.684160+0000","last_became_active":"2024-04-02T13:26:42.670831+0000","last_became_peered":"2024-04-02T13:26:42.670831+0000","last_unstale":"2024-04-02T13:26:43.684160+0000","last_undegraded":"2024-04-02T13:26:43.684160+0000","last_fullsized":"2024-04-02T13:26:43.684160+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:20:24.023770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":304,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.683109+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:26:43.683109+0000","last_peered":"2024-04-02T13:26:43.683109+0000","last_clean":"2024-04-02T13:26:43.683109+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:26:43.683109+0000","last_undegraded":"2024-04-02T13:26:43.683109+0000","last_fullsized":"2024-04-02T13:26:43.683109+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574929,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32860,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338980,"statfs":{"total":96636764160,"available":96603115520,"internally_reserved":0,"allocated":1712128,"data_stored":1099740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1130496,"data_stored":511532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574929,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":502149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1110016,"data_stored":509316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574929,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1110016,"data_stored":509316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574928,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1097728,"data_stored":502149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1097728,"data_stored":502149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574929,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32836,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339004,"statfs":{"total":96636764160,"available":96603140096,"internally_reserved":0,"allocated":1691648,"data_stored":1092517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:26:47.708 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:26:47.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.843+0000 7fe74c5cd640 1 Processor -- start 2024-04-02T13:26:47.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.843+0000 7fe74c5cd640 1 -- start start 2024-04-02T13:26:47.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.844+0000 7fe74c5cd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7440cece0 0x7fe7440cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.844+0000 7fe74c5cd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe7440d09f0 0x7fe74406d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.844+0000 7fe74c5cd640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7440cf600 con 0x7fe74406ece0 2024-04-02T13:26:47.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.844+0000 7fe74c5cd640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe74406dc40 con 0x7fe7440d09f0 2024-04-02T13:26:47.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.844+0000 7fe74c5cd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe74406df20 con 0x7fe7440cece0 2024-04-02T13:26:47.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.844+0000 7fe749b68640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe7440d09f0 0x7fe74406d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.844+0000 7fe74a369640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7440cece0 0x7fe7440cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.844+0000 7fe74ab6a640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe74406ece0 0x7fe74415eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:46478/0 (socket says 172.21.15.44:46478) 2024-04-02T13:26:47.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.844+0000 7fe74ab6a640 1 -- 172.21.15.44:0/3673855734 learned_addr learned my addr 172.21.15.44:0/3673855734 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:47.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.845+0000 7fe749b68640 1 -- 172.21.15.44:0/3673855734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7440cece0 msgr2=0x7fe7440cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:47.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.845+0000 7fe749b68640 1 --2- 172.21.15.44:0/3673855734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7440cece0 0x7fe7440cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.845+0000 7fe749b68640 1 -- 172.21.15.44:0/3673855734 >> v1:172.21.15.69:6789/0 conn(0x7fe74406ece0 legacy=0x7fe74415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.845+0000 7fe749367640 1 -- 172.21.15.44:0/3673855734 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4238499939 0 0) 0x7fe7440cf600 con 0x7fe74406ece0 2024-04-02T13:26:47.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.845+0000 7fe749b68640 1 -- 172.21.15.44:0/3673855734 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74406e200 con 0x7fe7440d09f0 2024-04-02T13:26:47.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.845+0000 7fe74a369640 1 --2- 172.21.15.44:0/3673855734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7440cece0 0x7fe7440cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:26:47.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.846+0000 7fe749b68640 1 --2- 172.21.15.44:0/3673855734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7440d09f0 0x7fe74406d2d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe7340097d0 tx=0x7fe73402b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c95b6735fa436dc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:47.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.846+0000 7fe749367640 1 -- 172.21.15.44:0/3673855734 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734035020 con 0x7fe7440d09f0 2024-04-02T13:26:47.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.846+0000 7fe749367640 1 -- 172.21.15.44:0/3673855734 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe734031800 con 0x7fe7440d09f0 2024-04-02T13:26:47.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.846+0000 7fe749367640 1 -- 172.21.15.44:0/3673855734 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734031bd0 con 0x7fe7440d09f0 2024-04-02T13:26:47.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.846+0000 7fe74c5cd640 1 -- 172.21.15.44:0/3673855734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7440d09f0 msgr2=0x7fe74406d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.846+0000 7fe74c5cd640 1 --2- 172.21.15.44:0/3673855734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7440d09f0 0x7fe74406d2d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe7340097d0 tx=0x7fe73402b8a0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.846+0000 7fe74c5cd640 1 -- 172.21.15.44:0/3673855734 shutdown_connections 2024-04-02T13:26:47.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.846+0000 7fe74c5cd640 1 --2- 172.21.15.44:0/3673855734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7440d09f0 0x7fe74406d2d0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.846+0000 7fe74c5cd640 1 --2- 172.21.15.44:0/3673855734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7440cece0 0x7fe7440cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.846+0000 7fe74c5cd640 1 -- 172.21.15.44:0/3673855734 >> 172.21.15.44:0/3673855734 conn(0x7fe7440cfcd0 msgr2=0x7fe7440d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:47.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.846+0000 7fe74c5cd640 1 -- 172.21.15.44:0/3673855734 shutdown_connections 2024-04-02T13:26:47.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.846+0000 7fe74c5cd640 1 -- 172.21.15.44:0/3673855734 wait complete. 2024-04-02T13:26:47.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.847+0000 7fe74c5cd640 1 Processor -- start 2024-04-02T13:26:47.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.848+0000 7fe74c5cd640 1 -- start start 2024-04-02T13:26:47.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.848+0000 7fe74c5cd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe74406ece0 0x7fe7441f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.848+0000 7fe74c5cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7440cece0 0x7fe74414ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.848+0000 7fe74c5cd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7440d09f0 0x7fe7440c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.848+0000 7fe74c5cd640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe74406d810 con 0x7fe7440cece0 2024-04-02T13:26:47.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.848+0000 7fe74c5cd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe74406e7c0 con 0x7fe74406ece0 2024-04-02T13:26:47.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.848+0000 7fe74c5cd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe74406d990 con 0x7fe7440d09f0 2024-04-02T13:26:47.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.848+0000 7fe749b68640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7440cece0 0x7fe74414ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.848+0000 7fe749b68640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7440cece0 0x7fe74414ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49298/0 (socket says 172.21.15.44:49298) 2024-04-02T13:26:47.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.848+0000 7fe749b68640 1 -- 172.21.15.44:0/543443191 learned_addr learned my addr 172.21.15.44:0/543443191 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:47.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.848+0000 7fe74a369640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe74406ece0 0x7fe7441f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.848+0000 7fe74ab6a640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7440d09f0 0x7fe7440c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.849+0000 7fe749b68640 1 -- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7440d09f0 msgr2=0x7fe7440c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.849+0000 7fe749b68640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7440d09f0 0x7fe7440c4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.849+0000 7fe749b68640 1 -- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe74406ece0 msgr2=0x7fe7441f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:47.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.849+0000 7fe749b68640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe74406ece0 0x7fe7441f4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:47.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.849+0000 7fe749b68640 1 -- 172.21.15.44:0/543443191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7441f46e0 con 0x7fe7440cece0 2024-04-02T13:26:47.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.849+0000 7fe749b68640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7440cece0 0x7fe74414ea60 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fe734034040 tx=0x7fe7340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:47.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.849+0000 7fe73affd640 1 -- 172.21.15.44:0/543443191 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734041070 con 0x7fe7440cece0 2024-04-02T13:26:47.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.849+0000 7fe73affd640 1 -- 172.21.15.44:0/543443191 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe73403ac80 con 0x7fe7440cece0 2024-04-02T13:26:47.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.849+0000 7fe74c5cd640 1 -- 172.21.15.44:0/543443191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7441f49c0 con 0x7fe7440cece0 2024-04-02T13:26:47.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.849+0000 7fe73affd640 1 -- 172.21.15.44:0/543443191 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734035040 con 0x7fe7440cece0 2024-04-02T13:26:47.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.849+0000 7fe74c5cd640 1 -- 172.21.15.44:0/543443191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7441f4f80 con 0x7fe7440cece0 2024-04-02T13:26:47.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.850+0000 7fe74c5cd640 1 -- 172.21.15.44:0/543443191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe744074f20 con 0x7fe7440cece0 2024-04-02T13:26:47.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.851+0000 7fe73affd640 1 -- 172.21.15.44:0/543443191 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe734038020 con 0x7fe7440cece0 2024-04-02T13:26:47.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.851+0000 7fe73affd640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe72806c8f0 0x7fe72806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:47.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.851+0000 7fe73affd640 1 -- 172.21.15.44:0/543443191 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7fe7340ad7b0 con 0x7fe7440cece0 2024-04-02T13:26:47.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.851+0000 7fe74a369640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe72806c8f0 0x7fe72806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:47.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.852+0000 7fe74a369640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe72806c8f0 0x7fe72806edb0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe740009560 tx=0x7fe740009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:47.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:47.855+0000 7fe73affd640 1 -- 172.21.15.44:0/543443191 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe73407b410 con 0x7fe7440cece0 2024-04-02T13:26:48.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.040+0000 7fe74c5cd640 1 -- 172.21.15.44:0/543443191 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe7440684f0 con 0x7fe72806c8f0 2024-04-02T13:26:48.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.047+0000 7fe73affd640 1 -- 172.21.15.44:0/543443191 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252004 (secure 0 0 0) 0x7fe7440684f0 con 0x7fe72806c8f0 2024-04-02T13:26:48.051 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:48.053 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:26:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.049+0000 7fe74c5cd640 1 -- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe72806c8f0 msgr2=0x7fe72806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.049+0000 7fe74c5cd640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe72806c8f0 0x7fe72806edb0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe740009560 tx=0x7fe740009190 comp rx=0 tx=0).stop 2024-04-02T13:26:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.050+0000 7fe74c5cd640 1 -- 172.21.15.44:0/543443191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7440cece0 msgr2=0x7fe74414ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.050+0000 7fe74c5cd640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7440cece0 0x7fe74414ea60 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fe734034040 tx=0x7fe7340046c0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.050+0000 7fe74c5cd640 1 -- 172.21.15.44:0/543443191 shutdown_connections 2024-04-02T13:26:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.050+0000 7fe74c5cd640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe72806c8f0 0x7fe72806edb0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.050+0000 7fe74c5cd640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7440d09f0 0x7fe7440c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.050+0000 7fe74c5cd640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7440cece0 0x7fe74414ea60 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.050+0000 7fe74c5cd640 1 --2- 172.21.15.44:0/543443191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe74406ece0 0x7fe7441f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.050+0000 7fe74c5cd640 1 -- 172.21.15.44:0/543443191 >> 172.21.15.44:0/543443191 conn(0x7fe7440cfcd0 msgr2=0x7fe744071190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:48.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.050+0000 7fe74c5cd640 1 -- 172.21.15.44:0/543443191 shutdown_connections 2024-04-02T13:26:48.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.050+0000 7fe74c5cd640 1 -- 172.21.15.44:0/543443191 wait complete. 2024-04-02T13:26:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-04-02T13:26:46.969476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259276,"kb_used_data":9812,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715444,"statfs":{"total":773094113280,"available":772828614656,"internally_reserved":0,"allocated":10047488,"data_stored":5228868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.320745"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.713848+0000","last_change":"2024-04-02T13:26:43.710931+0000","last_active":"2024-04-02T13:26:43.713848+0000","last_peered":"2024-04-02T13:26:43.713848+0000","last_clean":"2024-04-02T13:26:43.713848+0000","last_became_active":"2024-04-02T13:26:43.710522+0000","last_became_peered":"2024-04-02T13:26:43.710522+0000","last_unstale":"2024-04-02T13:26:43.713848+0000","last_undegraded":"2024-04-02T13:26:43.713848+0000","last_fullsized":"2024-04-02T13:26:43.713848+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:59:22.715019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698342+0000","last_change":"2024-04-02T13:26:43.690717+0000","last_active":"2024-04-02T13:26:43.698342+0000","last_peered":"2024-04-02T13:26:43.698342+0000","last_clean":"2024-04-02T13:26:43.698342+0000","last_became_active":"2024-04-02T13:26:43.690358+0000","last_became_peered":"2024-04-02T13:26:43.690358+0000","last_unstale":"2024-04-02T13:26:43.698342+0000","last_undegraded":"2024-04-02T13:26:43.698342+0000","last_fullsized":"2024-04-02T13:26:43.698342+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:27.943906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700646+0000","last_change":"2024-04-02T13:26:43.692683+0000","last_active":"2024-04-02T13:26:43.700646+0000","last_peered":"2024-04-02T13:26:43.700646+0000","last_clean":"2024-04-02T13:26:43.700646+0000","last_became_active":"2024-04-02T13:26:43.690759+0000","last_became_peered":"2024-04-02T13:26:43.690759+0000","last_unstale":"2024-04-02T13:26:43.700646+0000","last_undegraded":"2024-04-02T13:26:43.700646+0000","last_fullsized":"2024-04-02T13:26:43.700646+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:36:28.322738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700488+0000","last_change":"2024-04-02T13:26:43.698501+0000","last_active":"2024-04-02T13:26:43.700488+0000","last_peered":"2024-04-02T13:26:43.700488+0000","last_clean":"2024-04-02T13:26:43.700488+0000","last_became_active":"2024-04-02T13:26:43.698175+0000","last_became_peered":"2024-04-02T13:26:43.698175+0000","last_unstale":"2024-04-02T13:26:43.700488+0000","last_undegraded":"2024-04-02T13:26:43.700488+0000","last_fullsized":"2024-04-02T13:26:43.700488+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:24:01.313811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700498+0000","last_change":"2024-04-02T13:26:43.695954+0000","last_active":"2024-04-02T13:26:43.700498+0000","last_peered":"2024-04-02T13:26:43.700498+0000","last_clean":"2024-04-02T13:26:43.700498+0000","last_became_active":"2024-04-02T13:26:43.694696+0000","last_became_peered":"2024-04-02T13:26:43.694696+0000","last_unstale":"2024-04-02T13:26:43.700498+0000","last_undegraded":"2024-04-02T13:26:43.700498+0000","last_fullsized":"2024-04-02T13:26:43.700498+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:42.324949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.705396+0000","last_change":"2024-04-02T13:26:43.701921+0000","last_active":"2024-04-02T13:26:43.705396+0000","last_peered":"2024-04-02T13:26:43.705396+0000","last_clean":"2024-04-02T13:26:43.705396+0000","last_became_active":"2024-04-02T13:26:43.701649+0000","last_became_peered":"2024-04-02T13:26:43.701649+0000","last_unstale":"2024-04-02T13:26:43.705396+0000","last_undegraded":"2024-04-02T13:26:43.705396+0000","last_fullsized":"2024-04-02T13:26:43.705396+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:59:22.715019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700732+0000","last_change":"2024-04-02T13:26:43.693246+0000","last_active":"2024-04-02T13:26:43.700732+0000","last_peered":"2024-04-02T13:26:43.700732+0000","last_clean":"2024-04-02T13:26:43.700732+0000","last_became_active":"2024-04-02T13:26:43.692983+0000","last_became_peered":"2024-04-02T13:26:43.692983+0000","last_unstale":"2024-04-02T13:26:43.700732+0000","last_undegraded":"2024-04-02T13:26:43.700732+0000","last_fullsized":"2024-04-02T13:26:43.700732+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:36:28.322738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.703138+0000","last_change":"2024-04-02T13:26:43.698936+0000","last_active":"2024-04-02T13:26:43.703138+0000","last_peered":"2024-04-02T13:26:43.703138+0000","last_clean":"2024-04-02T13:26:43.703138+0000","last_became_active":"2024-04-02T13:26:43.697031+0000","last_became_peered":"2024-04-02T13:26:43.697031+0000","last_unstale":"2024-04-02T13:26:43.703138+0000","last_undegraded":"2024-04-02T13:26:43.703138+0000","last_fullsized":"2024-04-02T13:26:43.703138+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:50.758013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.697726+0000","last_change":"2024-04-02T13:26:43.690869+0000","last_active":"2024-04-02T13:26:43.697726+0000","last_peered":"2024-04-02T13:26:43.697726+0000","last_clean":"2024-04-02T13:26:43.697726+0000","last_became_active":"2024-04-02T13:26:43.690577+0000","last_became_peered":"2024-04-02T13:26:43.690577+0000","last_unstale":"2024-04-02T13:26:43.697726+0000","last_undegraded":"2024-04-02T13:26:43.697726+0000","last_fullsized":"2024-04-02T13:26:43.697726+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:24:32.385207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.695115+0000","last_change":"2024-04-02T13:26:43.693119+0000","last_active":"2024-04-02T13:26:43.695115+0000","last_peered":"2024-04-02T13:26:43.695115+0000","last_clean":"2024-04-02T13:26:43.695115+0000","last_became_active":"2024-04-02T13:26:43.692837+0000","last_became_peered":"2024-04-02T13:26:43.692837+0000","last_unstale":"2024-04-02T13:26:43.695115+0000","last_undegraded":"2024-04-02T13:26:43.695115+0000","last_fullsized":"2024-04-02T13:26:43.695115+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:27:50.911137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711550+0000","last_change":"2024-04-02T13:26:43.707639+0000","last_active":"2024-04-02T13:26:43.711550+0000","last_peered":"2024-04-02T13:26:43.711550+0000","last_clean":"2024-04-02T13:26:43.711550+0000","last_became_active":"2024-04-02T13:26:43.707341+0000","last_became_peered":"2024-04-02T13:26:43.707341+0000","last_unstale":"2024-04-02T13:26:43.711550+0000","last_undegraded":"2024-04-02T13:26:43.711550+0000","last_fullsized":"2024-04-02T13:26:43.711550+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:24:01.313811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.687123+0000","last_change":"2024-04-02T13:26:43.685929+0000","last_active":"2024-04-02T13:26:43.687123+0000","last_peered":"2024-04-02T13:26:43.687123+0000","last_clean":"2024-04-02T13:26:43.687123+0000","last_became_active":"2024-04-02T13:26:43.685617+0000","last_became_peered":"2024-04-02T13:26:43.685617+0000","last_unstale":"2024-04-02T13:26:43.687123+0000","last_undegraded":"2024-04-02T13:26:43.687123+0000","last_fullsized":"2024-04-02T13:26:43.687123+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:12.500724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700375+0000","last_change":"2024-04-02T13:26:43.696587+0000","last_active":"2024-04-02T13:26:43.700375+0000","last_peered":"2024-04-02T13:26:43.700375+0000","last_clean":"2024-04-02T13:26:43.700375+0000","last_became_active":"2024-04-02T13:26:43.696290+0000","last_became_peered":"2024-04-02T13:26:43.696290+0000","last_unstale":"2024-04-02T13:26:43.700375+0000","last_undegraded":"2024-04-02T13:26:43.700375+0000","last_fullsized":"2024-04-02T13:26:43.700375+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:42.324949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.697799+0000","last_change":"2024-04-02T13:26:43.691912+0000","last_active":"2024-04-02T13:26:43.697799+0000","last_peered":"2024-04-02T13:26:43.697799+0000","last_clean":"2024-04-02T13:26:43.697799+0000","last_became_active":"2024-04-02T13:26:43.691582+0000","last_became_peered":"2024-04-02T13:26:43.691582+0000","last_unstale":"2024-04-02T13:26:43.697799+0000","last_undegraded":"2024-04-02T13:26:43.697799+0000","last_fullsized":"2024-04-02T13:26:43.697799+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:12.471858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.704575+0000","last_change":"2024-04-02T13:26:43.702959+0000","last_active":"2024-04-02T13:26:43.704575+0000","last_peered":"2024-04-02T13:26:43.704575+0000","last_clean":"2024-04-02T13:26:43.704575+0000","last_became_active":"2024-04-02T13:26:43.702241+0000","last_became_peered":"2024-04-02T13:26:43.702241+0000","last_unstale":"2024-04-02T13:26:43.704575+0000","last_undegraded":"2024-04-02T13:26:43.704575+0000","last_fullsized":"2024-04-02T13:26:43.704575+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:38.335594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.712728+0000","last_change":"2024-04-02T13:26:43.709761+0000","last_active":"2024-04-02T13:26:43.712728+0000","last_peered":"2024-04-02T13:26:43.712728+0000","last_clean":"2024-04-02T13:26:43.712728+0000","last_became_active":"2024-04-02T13:26:43.709174+0000","last_became_peered":"2024-04-02T13:26:43.709174+0000","last_unstale":"2024-04-02T13:26:43.712728+0000","last_undegraded":"2024-04-02T13:26:43.712728+0000","last_fullsized":"2024-04-02T13:26:43.712728+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:59.316306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711601+0000","last_change":"2024-04-02T13:26:43.709303+0000","last_active":"2024-04-02T13:26:43.711601+0000","last_peered":"2024-04-02T13:26:43.711601+0000","last_clean":"2024-04-02T13:26:43.711601+0000","last_became_active":"2024-04-02T13:26:43.709006+0000","last_became_peered":"2024-04-02T13:26:43.709006+0000","last_unstale":"2024-04-02T13:26:43.711601+0000","last_undegraded":"2024-04-02T13:26:43.711601+0000","last_fullsized":"2024-04-02T13:26:43.711601+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:36:13.010848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.699825+0000","last_change":"2024-04-02T13:26:43.694684+0000","last_active":"2024-04-02T13:26:43.699825+0000","last_peered":"2024-04-02T13:26:43.699825+0000","last_clean":"2024-04-02T13:26:43.699825+0000","last_became_active":"2024-04-02T13:26:43.694383+0000","last_became_peered":"2024-04-02T13:26:43.694383+0000","last_unstale":"2024-04-02T13:26:43.699825+0000","last_undegraded":"2024-04-02T13:26:43.699825+0000","last_fullsized":"2024-04-02T13:26:43.699825+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:29:52.365602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700608+0000","last_change":"2024-04-02T13:26:43.692321+0000","last_active":"2024-04-02T13:26:43.700608+0000","last_peered":"2024-04-02T13:26:43.700608+0000","last_clean":"2024-04-02T13:26:43.700608+0000","last_became_active":"2024-04-02T13:26:43.691809+0000","last_became_peered":"2024-04-02T13:26:43.691809+0000","last_unstale":"2024-04-02T13:26:43.700608+0000","last_undegraded":"2024-04-02T13:26:43.700608+0000","last_fullsized":"2024-04-02T13:26:43.700608+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:53:51.490836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700471+0000","last_change":"2024-04-02T13:26:43.698461+0000","last_active":"2024-04-02T13:26:43.700471+0000","last_peered":"2024-04-02T13:26:43.700471+0000","last_clean":"2024-04-02T13:26:43.700471+0000","last_became_active":"2024-04-02T13:26:43.697788+0000","last_became_peered":"2024-04-02T13:26:43.697788+0000","last_unstale":"2024-04-02T13:26:43.700471+0000","last_undegraded":"2024-04-02T13:26:43.700471+0000","last_fullsized":"2024-04-02T13:26:43.700471+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:34:50.588433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700522+0000","last_change":"2024-04-02T13:26:43.696188+0000","last_active":"2024-04-02T13:26:43.700522+0000","last_peered":"2024-04-02T13:26:43.700522+0000","last_clean":"2024-04-02T13:26:43.700522+0000","last_became_active":"2024-04-02T13:26:43.695288+0000","last_became_peered":"2024-04-02T13:26:43.695288+0000","last_unstale":"2024-04-02T13:26:43.700522+0000","last_undegraded":"2024-04-02T13:26:43.700522+0000","last_fullsized":"2024-04-02T13:26:43.700522+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:24.166578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.697635+0000","last_change":"2024-04-02T13:26:43.691316+0000","last_active":"2024-04-02T13:26:43.697635+0000","last_peered":"2024-04-02T13:26:43.697635+0000","last_clean":"2024-04-02T13:26:43.697635+0000","last_became_active":"2024-04-02T13:26:43.691046+0000","last_became_peered":"2024-04-02T13:26:43.691046+0000","last_unstale":"2024-04-02T13:26:43.697635+0000","last_undegraded":"2024-04-02T13:26:43.697635+0000","last_fullsized":"2024-04-02T13:26:43.697635+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:20:25.030316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.708650+0000","last_change":"2024-04-02T13:26:43.705006+0000","last_active":"2024-04-02T13:26:43.708650+0000","last_peered":"2024-04-02T13:26:43.708650+0000","last_clean":"2024-04-02T13:26:43.708650+0000","last_became_active":"2024-04-02T13:26:43.704714+0000","last_became_peered":"2024-04-02T13:26:43.704714+0000","last_unstale":"2024-04-02T13:26:43.708650+0000","last_undegraded":"2024-04-02T13:26:43.708650+0000","last_fullsized":"2024-04-02T13:26:43.708650+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:38.335594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.695809+0000","last_change":"2024-04-02T13:26:43.690039+0000","last_active":"2024-04-02T13:26:43.695809+0000","last_peered":"2024-04-02T13:26:43.695809+0000","last_clean":"2024-04-02T13:26:43.695809+0000","last_became_active":"2024-04-02T13:26:43.689756+0000","last_became_peered":"2024-04-02T13:26:43.689756+0000","last_unstale":"2024-04-02T13:26:43.695809+0000","last_undegraded":"2024-04-02T13:26:43.695809+0000","last_fullsized":"2024-04-02T13:26:43.695809+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:31:29.042976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698438+0000","last_change":"2024-04-02T13:26:43.693985+0000","last_active":"2024-04-02T13:26:43.698438+0000","last_peered":"2024-04-02T13:26:43.698438+0000","last_clean":"2024-04-02T13:26:43.698438+0000","last_became_active":"2024-04-02T13:26:43.693688+0000","last_became_peered":"2024-04-02T13:26:43.693688+0000","last_unstale":"2024-04-02T13:26:43.698438+0000","last_undegraded":"2024-04-02T13:26:43.698438+0000","last_fullsized":"2024-04-02T13:26:43.698438+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:45:15.159479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.703484+0000","last_change":"2024-04-02T13:26:43.700670+0000","last_active":"2024-04-02T13:26:43.703484+0000","last_peered":"2024-04-02T13:26:43.703484+0000","last_clean":"2024-04-02T13:26:43.703484+0000","last_became_active":"2024-04-02T13:26:43.700357+0000","last_became_peered":"2024-04-02T13:26:43.700357+0000","last_unstale":"2024-04-02T13:26:43.703484+0000","last_undegraded":"2024-04-02T13:26:43.703484+0000","last_fullsized":"2024-04-02T13:26:43.703484+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:34:50.588433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.695227+0000","last_change":"2024-04-02T13:26:43.693781+0000","last_active":"2024-04-02T13:26:43.695227+0000","last_peered":"2024-04-02T13:26:43.695227+0000","last_clean":"2024-04-02T13:26:43.695227+0000","last_became_active":"2024-04-02T13:26:43.693525+0000","last_became_peered":"2024-04-02T13:26:43.693525+0000","last_unstale":"2024-04-02T13:26:43.695227+0000","last_undegraded":"2024-04-02T13:26:43.695227+0000","last_fullsized":"2024-04-02T13:26:43.695227+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:19.794846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700418+0000","last_change":"2024-04-02T13:26:43.696105+0000","last_active":"2024-04-02T13:26:43.700418+0000","last_peered":"2024-04-02T13:26:43.700418+0000","last_clean":"2024-04-02T13:26:43.700418+0000","last_became_active":"2024-04-02T13:26:43.695842+0000","last_became_peered":"2024-04-02T13:26:43.695842+0000","last_unstale":"2024-04-02T13:26:43.700418+0000","last_undegraded":"2024-04-02T13:26:43.700418+0000","last_fullsized":"2024-04-02T13:26:43.700418+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:02:57.270282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700706+0000","last_change":"2024-04-02T13:26:43.692666+0000","last_active":"2024-04-02T13:26:43.700706+0000","last_peered":"2024-04-02T13:26:43.700706+0000","last_clean":"2024-04-02T13:26:43.700706+0000","last_became_active":"2024-04-02T13:26:43.692390+0000","last_became_peered":"2024-04-02T13:26:43.692390+0000","last_unstale":"2024-04-02T13:26:43.700706+0000","last_undegraded":"2024-04-02T13:26:43.700706+0000","last_fullsized":"2024-04-02T13:26:43.700706+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:20:10.846657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.697613+0000","last_change":"2024-04-02T13:26:43.690139+0000","last_active":"2024-04-02T13:26:43.697613+0000","last_peered":"2024-04-02T13:26:43.697613+0000","last_clean":"2024-04-02T13:26:43.697613+0000","last_became_active":"2024-04-02T13:26:43.689885+0000","last_became_peered":"2024-04-02T13:26:43.689885+0000","last_unstale":"2024-04-02T13:26:43.697613+0000","last_undegraded":"2024-04-02T13:26:43.697613+0000","last_fullsized":"2024-04-02T13:26:43.697613+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:37.222294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698523+0000","last_change":"2024-04-02T13:26:43.696971+0000","last_active":"2024-04-02T13:26:43.698523+0000","last_peered":"2024-04-02T13:26:43.698523+0000","last_clean":"2024-04-02T13:26:43.698523+0000","last_became_active":"2024-04-02T13:26:43.696696+0000","last_became_peered":"2024-04-02T13:26:43.696696+0000","last_unstale":"2024-04-02T13:26:43.698523+0000","last_undegraded":"2024-04-02T13:26:43.698523+0000","last_fullsized":"2024-04-02T13:26:43.698523+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:55:04.746672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.712620+0000","last_change":"2024-04-02T13:26:43.709356+0000","last_active":"2024-04-02T13:26:43.712620+0000","last_peered":"2024-04-02T13:26:43.712620+0000","last_clean":"2024-04-02T13:26:43.712620+0000","last_became_active":"2024-04-02T13:26:43.706242+0000","last_became_peered":"2024-04-02T13:26:43.706242+0000","last_unstale":"2024-04-02T13:26:43.712620+0000","last_undegraded":"2024-04-02T13:26:43.712620+0000","last_fullsized":"2024-04-02T13:26:43.712620+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:41:26.513359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.705285+0000","last_change":"2024-04-02T13:26:43.700549+0000","last_active":"2024-04-02T13:26:43.705285+0000","last_peered":"2024-04-02T13:26:43.705285+0000","last_clean":"2024-04-02T13:26:43.705285+0000","last_became_active":"2024-04-02T13:26:43.700203+0000","last_became_peered":"2024-04-02T13:26:43.700203+0000","last_unstale":"2024-04-02T13:26:43.705285+0000","last_undegraded":"2024-04-02T13:26:43.705285+0000","last_fullsized":"2024-04-02T13:26:43.705285+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:38:28.720291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"90'4","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.710933+0000","last_change":"2024-04-02T13:26:42.674033+0000","last_active":"2024-04-02T13:26:43.710933+0000","last_peered":"2024-04-02T13:26:43.710933+0000","last_clean":"2024-04-02T13:26:43.710933+0000","last_became_active":"2024-04-02T13:26:42.673780+0000","last_became_peered":"2024-04-02T13:26:42.673780+0000","last_unstale":"2024-04-02T13:26:43.710933+0000","last_undegraded":"2024-04-02T13:26:43.710933+0000","last_fullsized":"2024-04-02T13:26:43.710933+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:36.033369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.709474+0000","last_change":"2024-04-02T13:26:43.707026+0000","last_active":"2024-04-02T13:26:43.709474+0000","last_peered":"2024-04-02T13:26:43.709474+0000","last_clean":"2024-04-02T13:26:43.709474+0000","last_became_active":"2024-04-02T13:26:43.706718+0000","last_became_peered":"2024-04-02T13:26:43.706718+0000","last_unstale":"2024-04-02T13:26:43.709474+0000","last_undegraded":"2024-04-02T13:26:43.709474+0000","last_fullsized":"2024-04-02T13:26:43.709474+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:21:28.111681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"90'1","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.708517+0000","last_change":"2024-04-02T13:26:42.667887+0000","last_active":"2024-04-02T13:26:43.708517+0000","last_peered":"2024-04-02T13:26:43.708517+0000","last_clean":"2024-04-02T13:26:43.708517+0000","last_became_active":"2024-04-02T13:26:42.667619+0000","last_became_peered":"2024-04-02T13:26:42.667619+0000","last_unstale":"2024-04-02T13:26:43.708517+0000","last_undegraded":"2024-04-02T13:26:43.708517+0000","last_fullsized":"2024-04-02T13:26:43.708517+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:41.538507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.706226+0000","last_change":"2024-04-02T13:26:43.700743+0000","last_active":"2024-04-02T13:26:43.706226+0000","last_peered":"2024-04-02T13:26:43.706226+0000","last_clean":"2024-04-02T13:26:43.706226+0000","last_became_active":"2024-04-02T13:26:43.700374+0000","last_became_peered":"2024-04-02T13:26:43.700374+0000","last_unstale":"2024-04-02T13:26:43.706226+0000","last_undegraded":"2024-04-02T13:26:43.706226+0000","last_fullsized":"2024-04-02T13:26:43.706226+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:59:22.715019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"90'2","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711076+0000","last_change":"2024-04-02T13:26:42.670504+0000","last_active":"2024-04-02T13:26:43.711076+0000","last_peered":"2024-04-02T13:26:43.711076+0000","last_clean":"2024-04-02T13:26:43.711076+0000","last_became_active":"2024-04-02T13:26:42.670203+0000","last_became_peered":"2024-04-02T13:26:42.670203+0000","last_unstale":"2024-04-02T13:26:43.711076+0000","last_undegraded":"2024-04-02T13:26:43.711076+0000","last_fullsized":"2024-04-02T13:26:43.711076+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:00:12.693543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.708680+0000","last_change":"2024-04-02T13:26:43.699888+0000","last_active":"2024-04-02T13:26:43.708680+0000","last_peered":"2024-04-02T13:26:43.708680+0000","last_clean":"2024-04-02T13:26:43.708680+0000","last_became_active":"2024-04-02T13:26:43.699573+0000","last_became_peered":"2024-04-02T13:26:43.699573+0000","last_unstale":"2024-04-02T13:26:43.708680+0000","last_undegraded":"2024-04-02T13:26:43.708680+0000","last_fullsized":"2024-04-02T13:26:43.708680+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:36:28.322738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"90'2","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.709203+0000","last_change":"2024-04-02T13:26:42.669152+0000","last_active":"2024-04-02T13:26:43.709203+0000","last_peered":"2024-04-02T13:26:43.709203+0000","last_clean":"2024-04-02T13:26:43.709203+0000","last_became_active":"2024-04-02T13:26:42.668843+0000","last_became_peered":"2024-04-02T13:26:42.668843+0000","last_unstale":"2024-04-02T13:26:43.709203+0000","last_undegraded":"2024-04-02T13:26:43.709203+0000","last_fullsized":"2024-04-02T13:26:43.709203+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:41.318403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.703161+0000","last_change":"2024-04-02T13:26:43.699149+0000","last_active":"2024-04-02T13:26:43.703161+0000","last_peered":"2024-04-02T13:26:43.703161+0000","last_clean":"2024-04-02T13:26:43.703161+0000","last_became_active":"2024-04-02T13:26:43.698645+0000","last_became_peered":"2024-04-02T13:26:43.698645+0000","last_unstale":"2024-04-02T13:26:43.703161+0000","last_undegraded":"2024-04-02T13:26:43.703161+0000","last_fullsized":"2024-04-02T13:26:43.703161+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:01:34.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"90'1","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.708330+0000","last_change":"2024-04-02T13:26:42.670719+0000","last_active":"2024-04-02T13:26:43.708330+0000","last_peered":"2024-04-02T13:26:43.708330+0000","last_clean":"2024-04-02T13:26:43.708330+0000","last_became_active":"2024-04-02T13:26:42.670419+0000","last_became_peered":"2024-04-02T13:26:42.670419+0000","last_unstale":"2024-04-02T13:26:43.708330+0000","last_undegraded":"2024-04-02T13:26:43.708330+0000","last_fullsized":"2024-04-02T13:26:43.708330+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:56.532143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711344+0000","last_change":"2024-04-02T13:26:43.707801+0000","last_active":"2024-04-02T13:26:43.711344+0000","last_peered":"2024-04-02T13:26:43.711344+0000","last_clean":"2024-04-02T13:26:43.711344+0000","last_became_active":"2024-04-02T13:26:43.707517+0000","last_became_peered":"2024-04-02T13:26:43.707517+0000","last_unstale":"2024-04-02T13:26:43.711344+0000","last_undegraded":"2024-04-02T13:26:43.711344+0000","last_fullsized":"2024-04-02T13:26:43.711344+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:13.852279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"90'1","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.714872+0000","last_change":"2024-04-02T13:26:42.670860+0000","last_active":"2024-04-02T13:26:43.714872+0000","last_peered":"2024-04-02T13:26:43.714872+0000","last_clean":"2024-04-02T13:26:43.714872+0000","last_became_active":"2024-04-02T13:26:42.670561+0000","last_became_peered":"2024-04-02T13:26:42.670561+0000","last_unstale":"2024-04-02T13:26:43.714872+0000","last_undegraded":"2024-04-02T13:26:43.714872+0000","last_fullsized":"2024-04-02T13:26:43.714872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:24:31.378662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.705378+0000","last_change":"2024-04-02T13:26:43.701049+0000","last_active":"2024-04-02T13:26:43.705378+0000","last_peered":"2024-04-02T13:26:43.705378+0000","last_clean":"2024-04-02T13:26:43.705378+0000","last_became_active":"2024-04-02T13:26:43.700742+0000","last_became_peered":"2024-04-02T13:26:43.700742+0000","last_unstale":"2024-04-02T13:26:43.705378+0000","last_undegraded":"2024-04-02T13:26:43.705378+0000","last_fullsized":"2024-04-02T13:26:43.705378+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:29:52.365602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.678887+0000","last_change":"2024-04-02T13:26:42.672938+0000","last_active":"2024-04-02T13:26:43.678887+0000","last_peered":"2024-04-02T13:26:43.678887+0000","last_clean":"2024-04-02T13:26:43.678887+0000","last_became_active":"2024-04-02T13:26:42.672660+0000","last_became_peered":"2024-04-02T13:26:42.672660+0000","last_unstale":"2024-04-02T13:26:43.678887+0000","last_undegraded":"2024-04-02T13:26:43.678887+0000","last_fullsized":"2024-04-02T13:26:43.678887+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:00.996929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700395+0000","last_change":"2024-04-02T13:26:43.698259+0000","last_active":"2024-04-02T13:26:43.700395+0000","last_peered":"2024-04-02T13:26:43.700395+0000","last_clean":"2024-04-02T13:26:43.700395+0000","last_became_active":"2024-04-02T13:26:43.697993+0000","last_became_peered":"2024-04-02T13:26:43.697993+0000","last_unstale":"2024-04-02T13:26:43.700395+0000","last_undegraded":"2024-04-02T13:26:43.700395+0000","last_fullsized":"2024-04-02T13:26:43.700395+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:37.039915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.682310+0000","last_change":"2024-04-02T13:26:42.666442+0000","last_active":"2024-04-02T13:26:43.682310+0000","last_peered":"2024-04-02T13:26:43.682310+0000","last_clean":"2024-04-02T13:26:43.682310+0000","last_became_active":"2024-04-02T13:26:42.666115+0000","last_became_peered":"2024-04-02T13:26:42.666115+0000","last_unstale":"2024-04-02T13:26:43.682310+0000","last_undegraded":"2024-04-02T13:26:43.682310+0000","last_fullsized":"2024-04-02T13:26:43.682310+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:55:11.259335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.697807+0000","last_change":"2024-04-02T13:26:43.689573+0000","last_active":"2024-04-02T13:26:43.697807+0000","last_peered":"2024-04-02T13:26:43.697807+0000","last_clean":"2024-04-02T13:26:43.697807+0000","last_became_active":"2024-04-02T13:26:43.689237+0000","last_became_peered":"2024-04-02T13:26:43.689237+0000","last_unstale":"2024-04-02T13:26:43.697807+0000","last_undegraded":"2024-04-02T13:26:43.697807+0000","last_fullsized":"2024-04-02T13:26:43.697807+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:01:35.449213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.686392+0000","last_change":"2024-04-02T13:26:42.671965+0000","last_active":"2024-04-02T13:26:43.686392+0000","last_peered":"2024-04-02T13:26:43.686392+0000","last_clean":"2024-04-02T13:26:43.686392+0000","last_became_active":"2024-04-02T13:26:42.671703+0000","last_became_peered":"2024-04-02T13:26:42.671703+0000","last_unstale":"2024-04-02T13:26:43.686392+0000","last_undegraded":"2024-04-02T13:26:43.686392+0000","last_fullsized":"2024-04-02T13:26:43.686392+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:41:21.985036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700651+0000","last_change":"2024-04-02T13:26:43.693977+0000","last_active":"2024-04-02T13:26:43.700651+0000","last_peered":"2024-04-02T13:26:43.700651+0000","last_clean":"2024-04-02T13:26:43.700651+0000","last_became_active":"2024-04-02T13:26:43.693717+0000","last_became_peered":"2024-04-02T13:26:43.693717+0000","last_unstale":"2024-04-02T13:26:43.700651+0000","last_undegraded":"2024-04-02T13:26:43.700651+0000","last_fullsized":"2024-04-02T13:26:43.700651+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:38:28.720291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.679385+0000","last_change":"2024-04-02T13:26:42.673632+0000","last_active":"2024-04-02T13:26:43.679385+0000","last_peered":"2024-04-02T13:26:43.679385+0000","last_clean":"2024-04-02T13:26:43.679385+0000","last_became_active":"2024-04-02T13:26:42.673357+0000","last_became_peered":"2024-04-02T13:26:42.673357+0000","last_unstale":"2024-04-02T13:26:43.679385+0000","last_undegraded":"2024-04-02T13:26:43.679385+0000","last_fullsized":"2024-04-02T13:26:43.679385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:00:12.693543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.699099+0000","last_change":"2024-04-02T13:26:43.692518+0000","last_active":"2024-04-02T13:26:43.699099+0000","last_peered":"2024-04-02T13:26:43.699099+0000","last_clean":"2024-04-02T13:26:43.699099+0000","last_became_active":"2024-04-02T13:26:43.692236+0000","last_became_peered":"2024-04-02T13:26:43.692236+0000","last_unstale":"2024-04-02T13:26:43.699099+0000","last_undegraded":"2024-04-02T13:26:43.699099+0000","last_fullsized":"2024-04-02T13:26:43.699099+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:20:10.846657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"90'3","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711502+0000","last_change":"2024-04-02T13:26:42.667969+0000","last_active":"2024-04-02T13:26:43.711502+0000","last_peered":"2024-04-02T13:26:43.711502+0000","last_clean":"2024-04-02T13:26:43.711502+0000","last_became_active":"2024-04-02T13:26:42.667696+0000","last_became_peered":"2024-04-02T13:26:42.667696+0000","last_unstale":"2024-04-02T13:26:43.711502+0000","last_undegraded":"2024-04-02T13:26:43.711502+0000","last_fullsized":"2024-04-02T13:26:43.711502+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:10:51.726256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700518+0000","last_change":"2024-04-02T13:26:43.697631+0000","last_active":"2024-04-02T13:26:43.700518+0000","last_peered":"2024-04-02T13:26:43.700518+0000","last_clean":"2024-04-02T13:26:43.700518+0000","last_became_active":"2024-04-02T13:26:43.697278+0000","last_became_peered":"2024-04-02T13:26:43.697278+0000","last_unstale":"2024-04-02T13:26:43.700518+0000","last_undegraded":"2024-04-02T13:26:43.700518+0000","last_fullsized":"2024-04-02T13:26:43.700518+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:24.166578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"90'6","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.716507+0000","last_change":"2024-04-02T13:26:42.666535+0000","last_active":"2024-04-02T13:26:43.716507+0000","last_peered":"2024-04-02T13:26:43.716507+0000","last_clean":"2024-04-02T13:26:43.716507+0000","last_became_active":"2024-04-02T13:26:42.666228+0000","last_became_peered":"2024-04-02T13:26:42.666228+0000","last_unstale":"2024-04-02T13:26:43.716507+0000","last_undegraded":"2024-04-02T13:26:43.716507+0000","last_fullsized":"2024-04-02T13:26:43.716507+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:47:58.212185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.703179+0000","last_change":"2024-04-02T13:26:43.699774+0000","last_active":"2024-04-02T13:26:43.703179+0000","last_peered":"2024-04-02T13:26:43.703179+0000","last_clean":"2024-04-02T13:26:43.703179+0000","last_became_active":"2024-04-02T13:26:43.699383+0000","last_became_peered":"2024-04-02T13:26:43.699383+0000","last_unstale":"2024-04-02T13:26:43.703179+0000","last_undegraded":"2024-04-02T13:26:43.703179+0000","last_fullsized":"2024-04-02T13:26:43.703179+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:44.626723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"90'2","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711849+0000","last_change":"2024-04-02T13:26:42.665005+0000","last_active":"2024-04-02T13:26:43.711849+0000","last_peered":"2024-04-02T13:26:43.711849+0000","last_clean":"2024-04-02T13:26:43.711849+0000","last_became_active":"2024-04-02T13:26:42.664711+0000","last_became_peered":"2024-04-02T13:26:42.664711+0000","last_unstale":"2024-04-02T13:26:43.711849+0000","last_undegraded":"2024-04-02T13:26:43.711849+0000","last_fullsized":"2024-04-02T13:26:43.711849+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:00.996929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700308+0000","last_change":"2024-04-02T13:26:43.694268+0000","last_active":"2024-04-02T13:26:43.700308+0000","last_peered":"2024-04-02T13:26:43.700308+0000","last_clean":"2024-04-02T13:26:43.700308+0000","last_became_active":"2024-04-02T13:26:43.694007+0000","last_became_peered":"2024-04-02T13:26:43.694007+0000","last_unstale":"2024-04-02T13:26:43.700308+0000","last_undegraded":"2024-04-02T13:26:43.700308+0000","last_fullsized":"2024-04-02T13:26:43.700308+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:02:57.270282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.688082+0000","last_change":"2024-04-02T13:26:42.673421+0000","last_active":"2024-04-02T13:26:43.688082+0000","last_peered":"2024-04-02T13:26:43.688082+0000","last_clean":"2024-04-02T13:26:43.688082+0000","last_became_active":"2024-04-02T13:26:42.673080+0000","last_became_peered":"2024-04-02T13:26:42.673080+0000","last_unstale":"2024-04-02T13:26:43.688082+0000","last_undegraded":"2024-04-02T13:26:43.688082+0000","last_fullsized":"2024-04-02T13:26:43.688082+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:06:30.310861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.692603+0000","last_change":"2024-04-02T13:26:43.691164+0000","last_active":"2024-04-02T13:26:43.692603+0000","last_peered":"2024-04-02T13:26:43.692603+0000","last_clean":"2024-04-02T13:26:43.692603+0000","last_became_active":"2024-04-02T13:26:43.690867+0000","last_became_peered":"2024-04-02T13:26:43.690867+0000","last_unstale":"2024-04-02T13:26:43.692603+0000","last_undegraded":"2024-04-02T13:26:43.692603+0000","last_fullsized":"2024-04-02T13:26:43.692603+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:09:15.323395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.684959+0000","last_change":"2024-04-02T13:26:42.665887+0000","last_active":"2024-04-02T13:26:43.684959+0000","last_peered":"2024-04-02T13:26:43.684959+0000","last_clean":"2024-04-02T13:26:43.684959+0000","last_became_active":"2024-04-02T13:26:42.665587+0000","last_became_peered":"2024-04-02T13:26:42.665587+0000","last_unstale":"2024-04-02T13:26:43.684959+0000","last_undegraded":"2024-04-02T13:26:43.684959+0000","last_fullsized":"2024-04-02T13:26:43.684959+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:11.465313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.692645+0000","last_change":"2024-04-02T13:26:43.691227+0000","last_active":"2024-04-02T13:26:43.692645+0000","last_peered":"2024-04-02T13:26:43.692645+0000","last_clean":"2024-04-02T13:26:43.692645+0000","last_became_active":"2024-04-02T13:26:43.690921+0000","last_became_peered":"2024-04-02T13:26:43.690921+0000","last_unstale":"2024-04-02T13:26:43.692645+0000","last_undegraded":"2024-04-02T13:26:43.692645+0000","last_fullsized":"2024-04-02T13:26:43.692645+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:13:22.417286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"90'8","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.715645+0000","last_change":"2024-04-02T13:26:42.666944+0000","last_active":"2024-04-02T13:26:43.715645+0000","last_peered":"2024-04-02T13:26:43.715645+0000","last_clean":"2024-04-02T13:26:43.715645+0000","last_became_active":"2024-04-02T13:26:42.666670+0000","last_became_peered":"2024-04-02T13:26:42.666670+0000","last_unstale":"2024-04-02T13:26:43.715645+0000","last_undegraded":"2024-04-02T13:26:43.715645+0000","last_fullsized":"2024-04-02T13:26:43.715645+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:53:50.484291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711009+0000","last_change":"2024-04-02T13:26:43.705898+0000","last_active":"2024-04-02T13:26:43.711009+0000","last_peered":"2024-04-02T13:26:43.711009+0000","last_clean":"2024-04-02T13:26:43.711009+0000","last_became_active":"2024-04-02T13:26:43.705566+0000","last_became_peered":"2024-04-02T13:26:43.705566+0000","last_unstale":"2024-04-02T13:26:43.711009+0000","last_undegraded":"2024-04-02T13:26:43.711009+0000","last_fullsized":"2024-04-02T13:26:43.711009+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:27:11.031758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"90'1","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711206+0000","last_change":"2024-04-02T13:26:42.671054+0000","last_active":"2024-04-02T13:26:43.711206+0000","last_peered":"2024-04-02T13:26:43.711206+0000","last_clean":"2024-04-02T13:26:43.711206+0000","last_became_active":"2024-04-02T13:26:42.670739+0000","last_became_peered":"2024-04-02T13:26:42.670739+0000","last_unstale":"2024-04-02T13:26:43.711206+0000","last_undegraded":"2024-04-02T13:26:43.711206+0000","last_fullsized":"2024-04-02T13:26:43.711206+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:34:49.581887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698313+0000","last_change":"2024-04-02T13:26:43.692814+0000","last_active":"2024-04-02T13:26:43.698313+0000","last_peered":"2024-04-02T13:26:43.698313+0000","last_clean":"2024-04-02T13:26:43.698313+0000","last_became_active":"2024-04-02T13:26:43.692537+0000","last_became_peered":"2024-04-02T13:26:43.692537+0000","last_unstale":"2024-04-02T13:26:43.698313+0000","last_undegraded":"2024-04-02T13:26:43.698313+0000","last_fullsized":"2024-04-02T13:26:43.698313+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:50.758013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"90'2","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711718+0000","last_change":"2024-04-02T13:26:42.667987+0000","last_active":"2024-04-02T13:26:43.711718+0000","last_peered":"2024-04-02T13:26:43.711718+0000","last_clean":"2024-04-02T13:26:43.711718+0000","last_became_active":"2024-04-02T13:26:42.667714+0000","last_became_peered":"2024-04-02T13:26:42.667714+0000","last_unstale":"2024-04-02T13:26:43.711718+0000","last_undegraded":"2024-04-02T13:26:43.711718+0000","last_fullsized":"2024-04-02T13:26:43.711718+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:47:58.212185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700807+0000","last_change":"2024-04-02T13:26:43.696729+0000","last_active":"2024-04-02T13:26:43.700807+0000","last_peered":"2024-04-02T13:26:43.700807+0000","last_clean":"2024-04-02T13:26:43.700807+0000","last_became_active":"2024-04-02T13:26:43.696334+0000","last_became_peered":"2024-04-02T13:26:43.696334+0000","last_unstale":"2024-04-02T13:26:43.700807+0000","last_undegraded":"2024-04-02T13:26:43.700807+0000","last_fullsized":"2024-04-02T13:26:43.700807+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:37.039915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"90'1","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711419+0000","last_change":"2024-04-02T13:26:42.667417+0000","last_active":"2024-04-02T13:26:43.711419+0000","last_peered":"2024-04-02T13:26:43.711419+0000","last_clean":"2024-04-02T13:26:43.711419+0000","last_became_active":"2024-04-02T13:26:42.667090+0000","last_became_peered":"2024-04-02T13:26:42.667090+0000","last_unstale":"2024-04-02T13:26:43.711419+0000","last_undegraded":"2024-04-02T13:26:43.711419+0000","last_fullsized":"2024-04-02T13:26:43.711419+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:36:03.171364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.706074+0000","last_change":"2024-04-02T13:26:43.703003+0000","last_active":"2024-04-02T13:26:43.706074+0000","last_peered":"2024-04-02T13:26:43.706074+0000","last_clean":"2024-04-02T13:26:43.706074+0000","last_became_active":"2024-04-02T13:26:43.702738+0000","last_became_peered":"2024-04-02T13:26:43.702738+0000","last_unstale":"2024-04-02T13:26:43.706074+0000","last_undegraded":"2024-04-02T13:26:43.706074+0000","last_fullsized":"2024-04-02T13:26:43.706074+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:36.661090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"90'2","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711133+0000","last_change":"2024-04-02T13:26:42.673224+0000","last_active":"2024-04-02T13:26:43.711133+0000","last_peered":"2024-04-02T13:26:43.711133+0000","last_clean":"2024-04-02T13:26:43.711133+0000","last_became_active":"2024-04-02T13:26:42.672899+0000","last_became_peered":"2024-04-02T13:26:42.672899+0000","last_unstale":"2024-04-02T13:26:43.711133+0000","last_undegraded":"2024-04-02T13:26:43.711133+0000","last_fullsized":"2024-04-02T13:26:43.711133+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:44:43.333598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.690161+0000","last_change":"2024-04-02T13:26:43.688467+0000","last_active":"2024-04-02T13:26:43.690161+0000","last_peered":"2024-04-02T13:26:43.690161+0000","last_clean":"2024-04-02T13:26:43.690161+0000","last_became_active":"2024-04-02T13:26:43.688176+0000","last_became_peered":"2024-04-02T13:26:43.688176+0000","last_unstale":"2024-04-02T13:26:43.690161+0000","last_undegraded":"2024-04-02T13:26:43.690161+0000","last_fullsized":"2024-04-02T13:26:43.690161+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:33:52.326948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.683071+0000","last_change":"2024-04-02T13:26:42.669537+0000","last_active":"2024-04-02T13:26:43.683071+0000","last_peered":"2024-04-02T13:26:43.683071+0000","last_clean":"2024-04-02T13:26:43.683071+0000","last_became_active":"2024-04-02T13:26:42.669264+0000","last_became_peered":"2024-04-02T13:26:42.669264+0000","last_unstale":"2024-04-02T13:26:43.683071+0000","last_undegraded":"2024-04-02T13:26:43.683071+0000","last_fullsized":"2024-04-02T13:26:43.683071+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:09:13.881054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698481+0000","last_change":"2024-04-02T13:26:43.693855+0000","last_active":"2024-04-02T13:26:43.698481+0000","last_peered":"2024-04-02T13:26:43.698481+0000","last_clean":"2024-04-02T13:26:43.698481+0000","last_became_active":"2024-04-02T13:26:43.693548+0000","last_became_peered":"2024-04-02T13:26:43.693548+0000","last_unstale":"2024-04-02T13:26:43.698481+0000","last_undegraded":"2024-04-02T13:26:43.698481+0000","last_fullsized":"2024-04-02T13:26:43.698481+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:10:54.661524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.684724+0000","last_change":"2024-04-02T13:26:42.676118+0000","last_active":"2024-04-02T13:26:43.684724+0000","last_peered":"2024-04-02T13:26:43.684724+0000","last_clean":"2024-04-02T13:26:43.684724+0000","last_became_active":"2024-04-02T13:26:42.675807+0000","last_became_peered":"2024-04-02T13:26:42.675807+0000","last_unstale":"2024-04-02T13:26:43.684724+0000","last_undegraded":"2024-04-02T13:26:43.684724+0000","last_fullsized":"2024-04-02T13:26:43.684724+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:40.845157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.705383+0000","last_change":"2024-04-02T13:26:43.695647+0000","last_active":"2024-04-02T13:26:43.705383+0000","last_peered":"2024-04-02T13:26:43.705383+0000","last_clean":"2024-04-02T13:26:43.705383+0000","last_became_active":"2024-04-02T13:26:43.695366+0000","last_became_peered":"2024-04-02T13:26:43.695366+0000","last_unstale":"2024-04-02T13:26:43.705383+0000","last_undegraded":"2024-04-02T13:26:43.705383+0000","last_fullsized":"2024-04-02T13:26:43.705383+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:31:29.042976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.682028+0000","last_change":"2024-04-02T13:26:42.677202+0000","last_active":"2024-04-02T13:26:43.682028+0000","last_peered":"2024-04-02T13:26:43.682028+0000","last_clean":"2024-04-02T13:26:43.682028+0000","last_became_active":"2024-04-02T13:26:42.676939+0000","last_became_peered":"2024-04-02T13:26:42.676939+0000","last_unstale":"2024-04-02T13:26:43.682028+0000","last_undegraded":"2024-04-02T13:26:43.682028+0000","last_fullsized":"2024-04-02T13:26:43.682028+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:56:00.746697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.701494+0000","last_change":"2024-04-02T13:26:43.697219+0000","last_active":"2024-04-02T13:26:43.701494+0000","last_peered":"2024-04-02T13:26:43.701494+0000","last_clean":"2024-04-02T13:26:43.701494+0000","last_became_active":"2024-04-02T13:26:43.696965+0000","last_became_peered":"2024-04-02T13:26:43.696965+0000","last_unstale":"2024-04-02T13:26:43.701494+0000","last_undegraded":"2024-04-02T13:26:43.701494+0000","last_fullsized":"2024-04-02T13:26:43.701494+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:44.626723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"90'2","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.711847+0000","last_change":"2024-04-02T13:26:42.666211+0000","last_active":"2024-04-02T13:26:43.711847+0000","last_peered":"2024-04-02T13:26:43.711847+0000","last_clean":"2024-04-02T13:26:43.711847+0000","last_became_active":"2024-04-02T13:26:42.665907+0000","last_became_peered":"2024-04-02T13:26:42.665907+0000","last_unstale":"2024-04-02T13:26:43.711847+0000","last_undegraded":"2024-04-02T13:26:43.711847+0000","last_fullsized":"2024-04-02T13:26:43.711847+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:31:28.036430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.708497+0000","last_change":"2024-04-02T13:26:43.706420+0000","last_active":"2024-04-02T13:26:43.708497+0000","last_peered":"2024-04-02T13:26:43.708497+0000","last_clean":"2024-04-02T13:26:43.708497+0000","last_became_active":"2024-04-02T13:26:43.705247+0000","last_became_peered":"2024-04-02T13:26:43.705247+0000","last_unstale":"2024-04-02T13:26:43.708497+0000","last_undegraded":"2024-04-02T13:26:43.708497+0000","last_fullsized":"2024-04-02T13:26:43.708497+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:27:30.825398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.687236+0000","last_change":"2024-04-02T13:26:42.667415+0000","last_active":"2024-04-02T13:26:43.687236+0000","last_peered":"2024-04-02T13:26:43.687236+0000","last_clean":"2024-04-02T13:26:43.687236+0000","last_became_active":"2024-04-02T13:26:42.667058+0000","last_became_peered":"2024-04-02T13:26:42.667058+0000","last_unstale":"2024-04-02T13:26:43.687236+0000","last_undegraded":"2024-04-02T13:26:43.687236+0000","last_fullsized":"2024-04-02T13:26:43.687236+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:27:17.217234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.705243+0000","last_change":"2024-04-02T13:26:43.696526+0000","last_active":"2024-04-02T13:26:43.705243+0000","last_peered":"2024-04-02T13:26:43.705243+0000","last_clean":"2024-04-02T13:26:43.705243+0000","last_became_active":"2024-04-02T13:26:43.696261+0000","last_became_peered":"2024-04-02T13:26:43.696261+0000","last_unstale":"2024-04-02T13:26:43.705243+0000","last_undegraded":"2024-04-02T13:26:43.705243+0000","last_fullsized":"2024-04-02T13:26:43.705243+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:20:10.846657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.686268+0000","last_change":"2024-04-02T13:26:42.667396+0000","last_active":"2024-04-02T13:26:43.686268+0000","last_peered":"2024-04-02T13:26:43.686268+0000","last_clean":"2024-04-02T13:26:43.686268+0000","last_became_active":"2024-04-02T13:26:42.667065+0000","last_became_peered":"2024-04-02T13:26:42.667065+0000","last_unstale":"2024-04-02T13:26:43.686268+0000","last_undegraded":"2024-04-02T13:26:43.686268+0000","last_fullsized":"2024-04-02T13:26:43.686268+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:37.321974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698459+0000","last_change":"2024-04-02T13:26:43.694139+0000","last_active":"2024-04-02T13:26:43.698459+0000","last_peered":"2024-04-02T13:26:43.698459+0000","last_clean":"2024-04-02T13:26:43.698459+0000","last_became_active":"2024-04-02T13:26:43.693873+0000","last_became_peered":"2024-04-02T13:26:43.693873+0000","last_unstale":"2024-04-02T13:26:43.698459+0000","last_undegraded":"2024-04-02T13:26:43.698459+0000","last_fullsized":"2024-04-02T13:26:43.698459+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:57:42.834961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.688813+0000","last_change":"2024-04-02T13:26:42.669705+0000","last_active":"2024-04-02T13:26:43.688813+0000","last_peered":"2024-04-02T13:26:43.688813+0000","last_clean":"2024-04-02T13:26:43.688813+0000","last_became_active":"2024-04-02T13:26:42.669449+0000","last_became_peered":"2024-04-02T13:26:42.669449+0000","last_unstale":"2024-04-02T13:26:43.688813+0000","last_undegraded":"2024-04-02T13:26:43.688813+0000","last_fullsized":"2024-04-02T13:26:43.688813+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:24:00.307265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.695907+0000","last_change":"2024-04-02T13:26:43.694479+0000","last_active":"2024-04-02T13:26:43.695907+0000","last_peered":"2024-04-02T13:26:43.695907+0000","last_clean":"2024-04-02T13:26:43.695907+0000","last_became_active":"2024-04-02T13:26:43.694210+0000","last_became_peered":"2024-04-02T13:26:43.694210+0000","last_unstale":"2024-04-02T13:26:43.695907+0000","last_undegraded":"2024-04-02T13:26:43.695907+0000","last_fullsized":"2024-04-02T13:26:43.695907+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:31:56.365732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.688891+0000","last_change":"2024-04-02T13:26:42.675125+0000","last_active":"2024-04-02T13:26:43.688891+0000","last_peered":"2024-04-02T13:26:43.688891+0000","last_clean":"2024-04-02T13:26:43.688891+0000","last_became_active":"2024-04-02T13:26:42.674431+0000","last_became_peered":"2024-04-02T13:26:42.674431+0000","last_unstale":"2024-04-02T13:26:43.688891+0000","last_undegraded":"2024-04-02T13:26:43.688891+0000","last_fullsized":"2024-04-02T13:26:43.688891+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:00.996929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700384+0000","last_change":"2024-04-02T13:26:43.693504+0000","last_active":"2024-04-02T13:26:43.700384+0000","last_peered":"2024-04-02T13:26:43.700384+0000","last_clean":"2024-04-02T13:26:43.700384+0000","last_became_active":"2024-04-02T13:26:43.693241+0000","last_became_peered":"2024-04-02T13:26:43.693241+0000","last_unstale":"2024-04-02T13:26:43.700384+0000","last_undegraded":"2024-04-02T13:26:43.700384+0000","last_fullsized":"2024-04-02T13:26:43.700384+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:38:28.720291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.685377+0000","last_change":"2024-04-02T13:26:42.673148+0000","last_active":"2024-04-02T13:26:43.685377+0000","last_peered":"2024-04-02T13:26:43.685377+0000","last_clean":"2024-04-02T13:26:43.685377+0000","last_became_active":"2024-04-02T13:26:42.672851+0000","last_became_peered":"2024-04-02T13:26:42.672851+0000","last_unstale":"2024-04-02T13:26:43.685377+0000","last_undegraded":"2024-04-02T13:26:43.685377+0000","last_fullsized":"2024-04-02T13:26:43.685377+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:05:35.742023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.698389+0000","last_change":"2024-04-02T13:26:43.693436+0000","last_active":"2024-04-02T13:26:43.698389+0000","last_peered":"2024-04-02T13:26:43.698389+0000","last_clean":"2024-04-02T13:26:43.698389+0000","last_became_active":"2024-04-02T13:26:43.693135+0000","last_became_peered":"2024-04-02T13:26:43.693135+0000","last_unstale":"2024-04-02T13:26:43.698389+0000","last_undegraded":"2024-04-02T13:26:43.698389+0000","last_fullsized":"2024-04-02T13:26:43.698389+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:01:34.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.688605+0000","last_change":"2024-04-02T13:26:42.665866+0000","last_active":"2024-04-02T13:26:43.688605+0000","last_peered":"2024-04-02T13:26:43.688605+0000","last_clean":"2024-04-02T13:26:43.688605+0000","last_became_active":"2024-04-02T13:26:42.665533+0000","last_became_peered":"2024-04-02T13:26:42.665533+0000","last_unstale":"2024-04-02T13:26:43.688605+0000","last_undegraded":"2024-04-02T13:26:43.688605+0000","last_fullsized":"2024-04-02T13:26:43.688605+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:23.160032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.705361+0000","last_change":"2024-04-02T13:26:43.692181+0000","last_active":"2024-04-02T13:26:43.705361+0000","last_peered":"2024-04-02T13:26:43.705361+0000","last_clean":"2024-04-02T13:26:43.705361+0000","last_became_active":"2024-04-02T13:26:43.691862+0000","last_became_peered":"2024-04-02T13:26:43.691862+0000","last_unstale":"2024-04-02T13:26:43.705361+0000","last_undegraded":"2024-04-02T13:26:43.705361+0000","last_fullsized":"2024-04-02T13:26:43.705361+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:00:13.700089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.683053+0000","last_change":"2024-04-02T13:26:42.666239+0000","last_active":"2024-04-02T13:26:43.683053+0000","last_peered":"2024-04-02T13:26:43.683053+0000","last_clean":"2024-04-02T13:26:43.683053+0000","last_became_active":"2024-04-02T13:26:42.665748+0000","last_became_peered":"2024-04-02T13:26:42.665748+0000","last_unstale":"2024-04-02T13:26:43.683053+0000","last_undegraded":"2024-04-02T13:26:43.683053+0000","last_fullsized":"2024-04-02T13:26:43.683053+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:06:30.310861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.700779+0000","last_change":"2024-04-02T13:26:43.694733+0000","last_active":"2024-04-02T13:26:43.700779+0000","last_peered":"2024-04-02T13:26:43.700779+0000","last_clean":"2024-04-02T13:26:43.700779+0000","last_became_active":"2024-04-02T13:26:43.694462+0000","last_became_peered":"2024-04-02T13:26:43.694462+0000","last_unstale":"2024-04-02T13:26:43.700779+0000","last_undegraded":"2024-04-02T13:26:43.700779+0000","last_fullsized":"2024-04-02T13:26:43.700779+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:42.647634+0000","last_clean_scrub_stamp":"2024-04-02T13:26:42.647634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:29:52.365602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.684160+0000","last_change":"2024-04-02T13:26:42.671125+0000","last_active":"2024-04-02T13:26:43.684160+0000","last_peered":"2024-04-02T13:26:43.684160+0000","last_clean":"2024-04-02T13:26:43.684160+0000","last_became_active":"2024-04-02T13:26:42.670831+0000","last_became_peered":"2024-04-02T13:26:42.670831+0000","last_unstale":"2024-04-02T13:26:43.684160+0000","last_undegraded":"2024-04-02T13:26:43.684160+0000","last_fullsized":"2024-04-02T13:26:43.684160+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:26:41.641088+0000","last_clean_scrub_stamp":"2024-04-02T13:26:41.641088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:20:24.023770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":304,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T13:26:43.683109+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:26:43.683109+0000","last_peered":"2024-04-02T13:26:43.683109+0000","last_clean":"2024-04-02T13:26:43.683109+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:26:43.683109+0000","last_undegraded":"2024-04-02T13:26:43.683109+0000","last_fullsized":"2024-04-02T13:26:43.683109+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574929,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32860,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338980,"statfs":{"total":96636764160,"available":96603115520,"internally_reserved":0,"allocated":1712128,"data_stored":1099740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1130496,"data_stored":511532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574929,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":502149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1110016,"data_stored":509316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574929,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1110016,"data_stored":509316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574928,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1097728,"data_stored":502149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1097728,"data_stored":502149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574929,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32836,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339004,"statfs":{"total":96636764160,"available":96603140096,"internally_reserved":0,"allocated":1691648,"data_stored":1092517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:26:48.066 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T13:26:48.066 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T13:26:48.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.209+0000 7f45b9150640 1 Processor -- start 2024-04-02T13:26:48.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.210+0000 7f45b9150640 1 -- start start 2024-04-02T13:26:48.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.210+0000 7f45b9150640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f45b406cf90 0x7f45b40cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:48.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.210+0000 7f45b9150640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f45b40cfe30 0x7f45b40d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:48.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.210+0000 7f45b9150640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45b406c1d0 con 0x7f45b40d0a30 2024-04-02T13:26:48.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.210+0000 7f45b9150640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f45b406cb00 con 0x7f45b40cfe30 2024-04-02T13:26:48.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.210+0000 7f45b9150640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f45b406d3f0 con 0x7f45b406cf90 2024-04-02T13:26:48.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.210+0000 7f45b359e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f45b40cfe30 0x7f45b40d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:48.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.210+0000 7f45b2d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f45b40d0a30 0x7f45b40d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42956/0 (socket says 172.21.15.44:42956) 2024-04-02T13:26:48.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.210+0000 7f45b359e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f45b40cfe30 0x7f45b40d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49302/0 (socket says 172.21.15.44:49302) 2024-04-02T13:26:48.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.210+0000 7f45b2d9d640 1 -- 172.21.15.44:0/3855100170 learned_addr learned my addr 172.21.15.44:0/3855100170 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:48.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.211+0000 7f45abfff640 1 --2- 172.21.15.44:0/3855100170 >> v2:172.21.15.69:3300/0 conn(0x7f45b406cf90 0x7f45b40cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:48.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.211+0000 7f45b359e640 1 -- 172.21.15.44:0/3855100170 >> v2:172.21.15.69:3300/0 conn(0x7f45b406cf90 msgr2=0x7f45b40cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:48.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.211+0000 7f45b359e640 1 --2- 172.21.15.44:0/3855100170 >> v2:172.21.15.69:3300/0 conn(0x7f45b406cf90 0x7f45b40cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.211+0000 7f45b359e640 1 -- 172.21.15.44:0/3855100170 >> v1:172.21.15.44:6789/0 conn(0x7f45b40d0a30 legacy=0x7f45b40d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.211+0000 7f45b259c640 1 -- 172.21.15.44:0/3855100170 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3496984177 0 0) 0x7f45b406c1d0 con 0x7f45b40d0a30 2024-04-02T13:26:48.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.211+0000 7f45b359e640 1 -- 172.21.15.44:0/3855100170 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b406c4b0 con 0x7f45b40cfe30 2024-04-02T13:26:48.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.211+0000 7f45b359e640 1 --2- 172.21.15.44:0/3855100170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45b40cfe30 0x7f45b40d2040 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f45b40b6d30 tx=0x7f45ac02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea05b9da93e1fce server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:48.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.212+0000 7f45b259c640 1 -- 172.21.15.44:0/3855100170 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac039070 con 0x7f45b40cfe30 2024-04-02T13:26:48.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.212+0000 7f45b259c640 1 -- 172.21.15.44:0/3855100170 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45ac02cbf0 con 0x7f45b40cfe30 2024-04-02T13:26:48.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.212+0000 7f45b259c640 1 -- 172.21.15.44:0/3855100170 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac034440 con 0x7f45b40cfe30 2024-04-02T13:26:48.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.212+0000 7f45b9150640 1 -- 172.21.15.44:0/3855100170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45b40cfe30 msgr2=0x7f45b40d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.212+0000 7f45b9150640 1 --2- 172.21.15.44:0/3855100170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45b40cfe30 0x7f45b40d2040 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f45b40b6d30 tx=0x7f45ac02be40 comp rx=0 tx=0).stop 2024-04-02T13:26:48.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.212+0000 7f45b9150640 1 -- 172.21.15.44:0/3855100170 shutdown_connections 2024-04-02T13:26:48.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.212+0000 7f45b9150640 1 --2- 172.21.15.44:0/3855100170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45b40cfe30 0x7f45b40d2040 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.212+0000 7f45b9150640 1 --2- 172.21.15.44:0/3855100170 >> v2:172.21.15.69:3300/0 conn(0x7f45b406cf90 0x7f45b40cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.212+0000 7f45b9150640 1 -- 172.21.15.44:0/3855100170 >> 172.21.15.44:0/3855100170 conn(0x7f45b409d470 msgr2=0x7f45b40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:48.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.212+0000 7f45b9150640 1 -- 172.21.15.44:0/3855100170 shutdown_connections 2024-04-02T13:26:48.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.212+0000 7f45b9150640 1 -- 172.21.15.44:0/3855100170 wait complete. 2024-04-02T13:26:48.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.214+0000 7f45b9150640 1 Processor -- start 2024-04-02T13:26:48.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.214+0000 7f45b9150640 1 -- start start 2024-04-02T13:26:48.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.214+0000 7f45b9150640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45b406cf90 0x7f45b4153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:48.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.214+0000 7f45b9150640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45b40cfe30 0x7f45b40b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:48.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.214+0000 7f45b9150640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45b40d0a30 0x7f45b40b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:48.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.214+0000 7f45b9150640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f45b415e870 con 0x7f45b406cf90 2024-04-02T13:26:48.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.214+0000 7f45b9150640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f45b415e6f0 con 0x7f45b40d0a30 2024-04-02T13:26:48.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.214+0000 7f45b9150640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f45b40d4da0 con 0x7f45b40cfe30 2024-04-02T13:26:48.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.214+0000 7f45b2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45b406cf90 0x7f45b4153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:48.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.214+0000 7f45b2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45b406cf90 0x7f45b4153860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49306/0 (socket says 172.21.15.44:49306) 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.214+0000 7f45b2d9d640 1 -- 172.21.15.44:0/3298033930 learned_addr learned my addr 172.21.15.44:0/3298033930 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.215+0000 7f45b359e640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45b40d0a30 0x7f45b40b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.215+0000 7f45abfff640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45b40cfe30 0x7f45b40b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.215+0000 7f45b2d9d640 1 -- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45b40cfe30 msgr2=0x7f45b40b28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.215+0000 7f45b2d9d640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45b40cfe30 0x7f45b40b28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.215+0000 7f45b2d9d640 1 -- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45b40d0a30 msgr2=0x7f45b40b2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.215+0000 7f45b2d9d640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45b40d0a30 0x7f45b40b2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.215+0000 7f45b2d9d640 1 -- 172.21.15.44:0/3298033930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b40b5190 con 0x7f45b406cf90 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.215+0000 7f45b2d9d640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45b406cf90 0x7f45b4153860 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f45a400c430 tx=0x7f45a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.215+0000 7f45ab7fe640 1 -- 172.21.15.44:0/3298033930 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a40074e0 con 0x7f45b406cf90 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.215+0000 7f45ab7fe640 1 -- 172.21.15.44:0/3298033930 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45a40047e0 con 0x7f45b406cf90 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.215+0000 7f45b9150640 1 -- 172.21.15.44:0/3298033930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b40b5470 con 0x7f45b406cf90 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.215+0000 7f45ab7fe640 1 -- 172.21.15.44:0/3298033930 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a4004bd0 con 0x7f45b406cf90 2024-04-02T13:26:48.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.215+0000 7f45b9150640 1 -- 172.21.15.44:0/3298033930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b40b5a30 con 0x7f45b406cf90 2024-04-02T13:26:48.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.216+0000 7f45b9150640 1 -- 172.21.15.44:0/3298033930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45b40684f0 con 0x7f45b406cf90 2024-04-02T13:26:48.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.217+0000 7f45ab7fe640 1 -- 172.21.15.44:0/3298033930 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f45a4011070 con 0x7f45b406cf90 2024-04-02T13:26:48.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.217+0000 7f45ab7fe640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f459406c8f0 0x7f459406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:48.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.217+0000 7f45ab7fe640 1 -- 172.21.15.44:0/3298033930 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7f45a408ba10 con 0x7f45b406cf90 2024-04-02T13:26:48.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.217+0000 7f45abfff640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f459406c8f0 0x7f459406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:48.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.218+0000 7f45abfff640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f459406c8f0 0x7f459406edb0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f45b41f7a00 tx=0x7f459c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:48.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.221+0000 7f45ab7fe640 1 -- 172.21.15.44:0/3298033930 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f45a4059c30 con 0x7f45b406cf90 2024-04-02T13:26:48.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.478+0000 7f45b9150640 1 -- 172.21.15.44:0/3298033930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f45b40d17c0 con 0x7f45b406cf90 2024-04-02T13:26:48.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.480+0000 7f45ab7fe640 1 -- 172.21.15.44:0/3298033930 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7f45a4059530 con 0x7f45b406cf90 2024-04-02T13:26:48.483 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-04-02T13:26:48.483 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A== 2024-04-02T13:26:48.483 INFO:teuthology.orchestra.run.smithi044.stdout: caps mds = "allow" 2024-04-02T13:26:48.483 INFO:teuthology.orchestra.run.smithi044.stdout: caps mon = "allow r" 2024-04-02T13:26:48.483 INFO:teuthology.orchestra.run.smithi044.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-02T13:26:48.483 INFO:teuthology.orchestra.run.smithi044.stderr:updated caps for client.0 2024-04-02T13:26:48.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.482+0000 7f45b9150640 1 -- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f459406c8f0 msgr2=0x7f459406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.482+0000 7f45b9150640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f459406c8f0 0x7f459406edb0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f45b41f7a00 tx=0x7f459c008040 comp rx=0 tx=0).stop 2024-04-02T13:26:48.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.482+0000 7f45b9150640 1 -- 172.21.15.44:0/3298033930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45b406cf90 msgr2=0x7f45b4153860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.482+0000 7f45b9150640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45b406cf90 0x7f45b4153860 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f45a400c430 tx=0x7f45a400c8f0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.482+0000 7f45b9150640 1 -- 172.21.15.44:0/3298033930 shutdown_connections 2024-04-02T13:26:48.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.482+0000 7f45b9150640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f459406c8f0 0x7f459406edb0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.482+0000 7f45b9150640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45b40d0a30 0x7f45b40b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.482+0000 7f45b9150640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45b40cfe30 0x7f45b40b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.482+0000 7f45b9150640 1 --2- 172.21.15.44:0/3298033930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45b406cf90 0x7f45b4153860 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.482+0000 7f45b9150640 1 -- 172.21.15.44:0/3298033930 >> 172.21.15.44:0/3298033930 conn(0x7f45b409d470 msgr2=0x7f45b40d3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:48.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.483+0000 7f45b9150640 1 -- 172.21.15.44:0/3298033930 shutdown_connections 2024-04-02T13:26:48.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.483+0000 7f45b9150640 1 -- 172.21.15.44:0/3298033930 wait complete. 2024-04-02T13:26:48.496 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:26:48.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.595+0000 7ff6b267b640 1 Processor -- start 2024-04-02T13:26:48.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.595+0000 7ff6b267b640 1 -- start start 2024-04-02T13:26:48.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6b267b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff6ac15c160 0x7ff6ac06b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:48.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6b267b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6ac160600 0x7ff6ac0d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:48.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6b267b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6ac06bab0 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6b267b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff6ac0ce6d0 con 0x7ff6ac15c160 2024-04-02T13:26:48.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6b267b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff6ac06bd90 con 0x7ff6ac160600 2024-04-02T13:26:48.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6abfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff6ac15c160 0x7ff6ac06b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:48.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6b0c18640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff6ac0d06c0 0x7ff6ac0ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42958/0 (socket says 172.21.15.44:42958) 2024-04-02T13:26:48.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6b0c18640 1 -- 172.21.15.44:0/391069475 learned_addr learned my addr 172.21.15.44:0/391069475 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:48.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6ab7fe640 1 --2- 172.21.15.44:0/391069475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6ac160600 0x7ff6ac0d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:48.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6abfff640 1 -- 172.21.15.44:0/391069475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6ac160600 msgr2=0x7ff6ac0d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:48.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6abfff640 1 --2- 172.21.15.44:0/391069475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6ac160600 0x7ff6ac0d0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6abfff640 1 -- 172.21.15.44:0/391069475 >> v1:172.21.15.44:6789/0 conn(0x7ff6ac0d06c0 legacy=0x7ff6ac0ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6abfff640 1 -- 172.21.15.44:0/391069475 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ac0ccec0 con 0x7ff6ac15c160 2024-04-02T13:26:48.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.596+0000 7ff6aaffd640 1 -- 172.21.15.44:0/391069475 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 721958381 0 0) 0x7ff6ac06bab0 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.597+0000 7ff6abfff640 1 --2- 172.21.15.44:0/391069475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6ac15c160 0x7ff6ac06b4d0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7ff6940097a0 tx=0x7ff69402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0d4892e7fda5b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:48.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.597+0000 7ff6aaffd640 1 -- 172.21.15.44:0/391069475 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6940315a0 con 0x7ff6ac15c160 2024-04-02T13:26:48.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.597+0000 7ff6aaffd640 1 -- 172.21.15.44:0/391069475 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff694037070 con 0x7ff6ac15c160 2024-04-02T13:26:48.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.597+0000 7ff6aaffd640 1 -- 172.21.15.44:0/391069475 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff694031960 con 0x7ff6ac15c160 2024-04-02T13:26:48.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.598+0000 7ff6b267b640 1 -- 172.21.15.44:0/391069475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6ac15c160 msgr2=0x7ff6ac06b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.598+0000 7ff6b267b640 1 --2- 172.21.15.44:0/391069475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6ac15c160 0x7ff6ac06b4d0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7ff6940097a0 tx=0x7ff69402b840 comp rx=0 tx=0).stop 2024-04-02T13:26:48.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.598+0000 7ff6b267b640 1 -- 172.21.15.44:0/391069475 shutdown_connections 2024-04-02T13:26:48.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.598+0000 7ff6b267b640 1 --2- 172.21.15.44:0/391069475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6ac160600 0x7ff6ac0d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.598+0000 7ff6b267b640 1 --2- 172.21.15.44:0/391069475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6ac15c160 0x7ff6ac06b4d0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.598+0000 7ff6b267b640 1 -- 172.21.15.44:0/391069475 >> 172.21.15.44:0/391069475 conn(0x7ff6ac0b2980 msgr2=0x7ff6ac0b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:48.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.598+0000 7ff6b267b640 1 -- 172.21.15.44:0/391069475 shutdown_connections 2024-04-02T13:26:48.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.598+0000 7ff6b267b640 1 -- 172.21.15.44:0/391069475 wait complete. 2024-04-02T13:26:48.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.599+0000 7ff6b267b640 1 Processor -- start 2024-04-02T13:26:48.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.599+0000 7ff6b267b640 1 -- start start 2024-04-02T13:26:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6b267b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6ac0d06c0 0x7ff6ac0c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6b267b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6ac15c160 0x7ff6ac0c34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6b267b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6ac160600 0x7ff6ac0c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6b267b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac160a60 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6b267b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac0ccb70 con 0x7ff6ac15c160 2024-04-02T13:26:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6b267b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff6ac0add60 con 0x7ff6ac160600 2024-04-02T13:26:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6abfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6ac0d06c0 0x7ff6ac0c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6abfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6ac0d06c0 0x7ff6ac0c8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49328/0 (socket says 172.21.15.44:49328) 2024-04-02T13:26:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6abfff640 1 -- 172.21.15.44:0/4093110906 learned_addr learned my addr 172.21.15.44:0/4093110906 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6b0c18640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6ac160600 0x7ff6ac0c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6ab7fe640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6ac15c160 0x7ff6ac0c34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6abfff640 1 -- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6ac160600 msgr2=0x7ff6ac0c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6abfff640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6ac160600 0x7ff6ac0c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6abfff640 1 -- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6ac15c160 msgr2=0x7ff6ac0c34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6abfff640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6ac15c160 0x7ff6ac0c34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6abfff640 1 -- 172.21.15.44:0/4093110906 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ac0c91a0 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6b0c18640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6ac160600 0x7ff6ac0c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6ab7fe640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6ac15c160 0x7ff6ac0c34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6abfff640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6ac0d06c0 0x7ff6ac0c8b10 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7ff694034040 tx=0x7ff6940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.600+0000 7ff6a8ff9640 1 -- 172.21.15.44:0/4093110906 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff694032840 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.601+0000 7ff6a8ff9640 1 -- 172.21.15.44:0/4093110906 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6940329e0 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.601+0000 7ff6b267b640 1 -- 172.21.15.44:0/4093110906 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6ac0c5110 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.601+0000 7ff6a8ff9640 1 -- 172.21.15.44:0/4093110906 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff694032dd0 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.601+0000 7ff6b267b640 1 -- 172.21.15.44:0/4093110906 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6ac0c9540 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.602+0000 7ff6b267b640 1 -- 172.21.15.44:0/4093110906 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6ac0c9370 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.602+0000 7ff6a8ff9640 1 -- 172.21.15.44:0/4093110906 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff69404e430 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.602+0000 7ff6a8ff9640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff68c06c8f0 0x7ff68c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:48.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.602+0000 7ff6a8ff9640 1 -- 172.21.15.44:0/4093110906 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7ff69402f030 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.603+0000 7ff6ab7fe640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff68c06c8f0 0x7ff68c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:48.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.603+0000 7ff6ab7fe640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff68c06c8f0 0x7ff68c06edb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac1e8630 tx=0x7ff69c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:48.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.607+0000 7ff6a8ff9640 1 -- 172.21.15.44:0/4093110906 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff694040070 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.830+0000 7ff6b267b640 1 -- 172.21.15.44:0/4093110906 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6ac0684f0 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.831+0000 7ff6a8ff9640 1 -- 172.21.15.44:0/4093110906 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5177 (secure 0 0 0) 0x7ff69407c030 con 0x7ff6ac0d06c0 2024-04-02T13:26:48.833 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:48.834 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":61,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:26:46.063220+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6833/719680934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":719680934},{"type":"v1","addr":"172.21.15.44:6833","nonce":719680934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":6}]} 2024-04-02T13:26:48.834 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 61 2024-04-02T13:26:48.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.832+0000 7ff6b267b640 1 -- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff68c06c8f0 msgr2=0x7ff68c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.832+0000 7ff6b267b640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff68c06c8f0 0x7ff68c06edb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac1e8630 tx=0x7ff69c008040 comp rx=0 tx=0).stop 2024-04-02T13:26:48.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.832+0000 7ff6b267b640 1 -- 172.21.15.44:0/4093110906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6ac0d06c0 msgr2=0x7ff6ac0c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:48.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.832+0000 7ff6b267b640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6ac0d06c0 0x7ff6ac0c8b10 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7ff694034040 tx=0x7ff6940046c0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.833+0000 7ff6b267b640 1 -- 172.21.15.44:0/4093110906 shutdown_connections 2024-04-02T13:26:48.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.833+0000 7ff6b267b640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff68c06c8f0 0x7ff68c06edb0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.833+0000 7ff6b267b640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6ac160600 0x7ff6ac0c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.833+0000 7ff6b267b640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6ac15c160 0x7ff6ac0c34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.833+0000 7ff6b267b640 1 --2- 172.21.15.44:0/4093110906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6ac0d06c0 0x7ff6ac0c8b10 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:48.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.833+0000 7ff6b267b640 1 -- 172.21.15.44:0/4093110906 >> 172.21.15.44:0/4093110906 conn(0x7ff6ac0b2980 msgr2=0x7ff6ac0b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:48.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.833+0000 7ff6b267b640 1 -- 172.21.15.44:0/4093110906 shutdown_connections 2024-04-02T13:26:48.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:48.833+0000 7ff6b267b640 1 -- 172.21.15.44:0/4093110906 wait complete. 2024-04-02T13:26:49.248 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:26:43.025860+0000', 'modified': '2024-04-02T13:26:46.063220+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7803': {'gid': 7803, 'name': 'a', 'rank': 0, 'incarnation': 57, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.44:6833/719680934', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6832', 'nonce': 719680934}, {'type': 'v1', 'addr': '172.21.15.44:6833', 'nonce': 719680934}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7803, 'qdb_cluster': [7803]} 2024-04-02T13:26:49.248 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:26:49.249 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T13:26:49.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.350+0000 7eff6b414640 1 Processor -- start 2024-04-02T13:26:49.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.350+0000 7eff6b414640 1 -- start start 2024-04-02T13:26:49.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.351+0000 7eff6b414640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7eff6415e430 0x7eff640ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.351+0000 7eff6b414640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7eff640b1ef0 0x7eff640b2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.351+0000 7eff6b414640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff6415e810 con 0x7eff6406c7f0 2024-04-02T13:26:49.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.351+0000 7eff6b414640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7eff640d1410 con 0x7eff6415e430 2024-04-02T13:26:49.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.351+0000 7eff6b414640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7eff6406cff0 con 0x7eff640b1ef0 2024-04-02T13:26:49.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.351+0000 7eff691b0640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7eff6415e430 0x7eff640ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.351+0000 7eff691b0640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7eff6415e430 0x7eff640ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49336/0 (socket says 172.21.15.44:49336) 2024-04-02T13:26:49.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.351+0000 7eff691b0640 1 -- 172.21.15.44:0/2933862878 learned_addr learned my addr 172.21.15.44:0/2933862878 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:49.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.352+0000 7eff699b1640 1 --2- 172.21.15.44:0/2933862878 >> v2:172.21.15.69:3300/0 conn(0x7eff640b1ef0 0x7eff640b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.352+0000 7eff691b0640 1 -- 172.21.15.44:0/2933862878 >> v2:172.21.15.69:3300/0 conn(0x7eff640b1ef0 msgr2=0x7eff640b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:49.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.352+0000 7eff691b0640 1 --2- 172.21.15.44:0/2933862878 >> v2:172.21.15.69:3300/0 conn(0x7eff640b1ef0 0x7eff640b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.352+0000 7eff691b0640 1 -- 172.21.15.44:0/2933862878 >> v1:172.21.15.69:6789/0 conn(0x7eff6406c7f0 legacy=0x7eff640ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.352+0000 7eff691b0640 1 -- 172.21.15.44:0/2933862878 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6406d240 con 0x7eff6415e430 2024-04-02T13:26:49.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.352+0000 7eff691b0640 1 --2- 172.21.15.44:0/2933862878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff6415e430 0x7eff640ce7c0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7eff640cc520 tx=0x7eff5402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6edb527c46d4284 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:49.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.353+0000 7eff63fff640 1 -- 172.21.15.44:0/2933862878 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5402ca80 con 0x7eff6415e430 2024-04-02T13:26:49.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.353+0000 7eff63fff640 1 -- 172.21.15.44:0/2933862878 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff5402cc20 con 0x7eff6415e430 2024-04-02T13:26:49.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.353+0000 7eff63fff640 1 -- 172.21.15.44:0/2933862878 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff540343f0 con 0x7eff6415e430 2024-04-02T13:26:49.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.353+0000 7eff6b414640 1 -- 172.21.15.44:0/2933862878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff6415e430 msgr2=0x7eff640ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.353+0000 7eff6b414640 1 --2- 172.21.15.44:0/2933862878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff6415e430 0x7eff640ce7c0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7eff640cc520 tx=0x7eff5402bee0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.353+0000 7eff6b414640 1 -- 172.21.15.44:0/2933862878 shutdown_connections 2024-04-02T13:26:49.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.353+0000 7eff6b414640 1 --2- 172.21.15.44:0/2933862878 >> v2:172.21.15.69:3300/0 conn(0x7eff640b1ef0 0x7eff640b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.353+0000 7eff6b414640 1 --2- 172.21.15.44:0/2933862878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff6415e430 0x7eff640ce7c0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.353+0000 7eff6b414640 1 -- 172.21.15.44:0/2933862878 >> 172.21.15.44:0/2933862878 conn(0x7eff64073850 msgr2=0x7eff6409aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:49.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.353+0000 7eff6b414640 1 -- 172.21.15.44:0/2933862878 shutdown_connections 2024-04-02T13:26:49.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.353+0000 7eff6b414640 1 -- 172.21.15.44:0/2933862878 wait complete. 2024-04-02T13:26:49.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.354+0000 7eff6b414640 1 Processor -- start 2024-04-02T13:26:49.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.354+0000 7eff6b414640 1 -- start start 2024-04-02T13:26:49.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff6b414640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff6406c7f0 0x7eff64155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff6b414640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff640b1ef0 0x7eff64152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff6b414640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff6415e430 0x7eff641f77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff6b414640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7eff640d0ef0 con 0x7eff6406c7f0 2024-04-02T13:26:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff6b414640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7eff640d0c10 con 0x7eff640b1ef0 2024-04-02T13:26:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff6b414640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7eff6406d800 con 0x7eff6415e430 2024-04-02T13:26:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff691b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff6406c7f0 0x7eff64155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff691b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff6406c7f0 0x7eff64155880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49352/0 (socket says 172.21.15.44:49352) 2024-04-02T13:26:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff691b0640 1 -- 172.21.15.44:0/4293753419 learned_addr learned my addr 172.21.15.44:0/4293753419 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff689af640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff640b1ef0 0x7eff64152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff699b1640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff6415e430 0x7eff641f77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff691b0640 1 -- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff6415e430 msgr2=0x7eff641f77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff691b0640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff6415e430 0x7eff641f77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff691b0640 1 -- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff640b1ef0 msgr2=0x7eff64152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff691b0640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff640b1ef0 0x7eff64152d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff691b0640 1 -- 172.21.15.44:0/4293753419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff640c7a60 con 0x7eff6406c7f0 2024-04-02T13:26:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.355+0000 7eff689af640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff640b1ef0 0x7eff64152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.356+0000 7eff691b0640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff6406c7f0 0x7eff64155880 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7eff54000f80 tx=0x7eff54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.356+0000 7eff61ffb640 1 -- 172.21.15.44:0/4293753419 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff54043020 con 0x7eff6406c7f0 2024-04-02T13:26:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.356+0000 7eff61ffb640 1 -- 172.21.15.44:0/4293753419 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff5403e450 con 0x7eff6406c7f0 2024-04-02T13:26:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.356+0000 7eff6b414640 1 -- 172.21.15.44:0/4293753419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff64153840 con 0x7eff6406c7f0 2024-04-02T13:26:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.356+0000 7eff61ffb640 1 -- 172.21.15.44:0/4293753419 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff54009510 con 0x7eff6406c7f0 2024-04-02T13:26:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.356+0000 7eff6b414640 1 -- 172.21.15.44:0/4293753419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff640cb6a0 con 0x7eff6406c7f0 2024-04-02T13:26:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.357+0000 7eff61ffb640 1 -- 172.21.15.44:0/4293753419 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7eff54046020 con 0x7eff6406c7f0 2024-04-02T13:26:49.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.358+0000 7eff61ffb640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff5006c8f0 0x7eff5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.358+0000 7eff61ffb640 1 -- 172.21.15.44:0/4293753419 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7eff540ae560 con 0x7eff6406c7f0 2024-04-02T13:26:49.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.358+0000 7eff689af640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff5006c8f0 0x7eff5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.359+0000 7eff689af640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff5006c8f0 0x7eff5006edb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7eff58007fd0 tx=0x7eff58009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:49.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.359+0000 7eff6dffe740 1 Processor -- start 2024-04-02T13:26:49.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.359+0000 7eff6dffe740 1 -- start start 2024-04-02T13:26:49.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.359+0000 7eff6dffe740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92b90 0x560100e65b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.359+0000 7eff6dffe740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92f70 0x560100e95920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.359+0000 7eff6dffe740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e66070 0x560100e95e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.359+0000 7eff6dffe740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560100dcf060 con 0x560100e92b90 2024-04-02T13:26:49.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.359+0000 7eff6dffe740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560100d77650 con 0x560100e66070 2024-04-02T13:26:49.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.359+0000 7eff6dffe740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x560100dcb010 con 0x560100e92f70 2024-04-02T13:26:49.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff691b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92b90 0x560100e65b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff691b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92b90 0x560100e65b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49358/0 (socket says 172.21.15.44:49358) 2024-04-02T13:26:49.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff691b0640 1 -- 172.21.15.44:0/2234978402 learned_addr learned my addr 172.21.15.44:0/2234978402 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:49.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff699b1640 1 --2- 172.21.15.44:0/2234978402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e66070 0x560100e95e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff689af640 1 --2- 172.21.15.44:0/2234978402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92f70 0x560100e95920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff691b0640 1 -- 172.21.15.44:0/2234978402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92f70 msgr2=0x560100e95920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:49.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff691b0640 1 --2- 172.21.15.44:0/2234978402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92f70 0x560100e95920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff691b0640 1 -- 172.21.15.44:0/2234978402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e66070 msgr2=0x560100e95e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:49.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff691b0640 1 --2- 172.21.15.44:0/2234978402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e66070 0x560100e95e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff691b0640 1 -- 172.21.15.44:0/2234978402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560100ae32d0 con 0x560100e92b90 2024-04-02T13:26:49.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff699b1640 1 --2- 172.21.15.44:0/2234978402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e66070 0x560100e95e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:49.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff689af640 1 --2- 172.21.15.44:0/2234978402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92f70 0x560100e95920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:49.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff691b0640 1 --2- 172.21.15.44:0/2234978402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92b90 0x560100e65b30 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7eff540847b0 tx=0x7eff5403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e30ba1f89564900 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:49.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff4affd640 1 -- 172.21.15.44:0/2234978402 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5407c100 con 0x560100e92b90 2024-04-02T13:26:49.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff4affd640 1 -- 172.21.15.44:0/2234978402 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff540804b0 con 0x560100e92b90 2024-04-02T13:26:49.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.360+0000 7eff4affd640 1 -- 172.21.15.44:0/2234978402 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff54084ce0 con 0x560100e92b90 2024-04-02T13:26:49.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.361+0000 7eff6dffe740 1 -- 172.21.15.44:0/2234978402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92b90 msgr2=0x560100e65b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.361+0000 7eff6dffe740 1 --2- 172.21.15.44:0/2234978402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92b90 0x560100e65b30 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7eff540847b0 tx=0x7eff5403c000 comp rx=0 tx=0).stop 2024-04-02T13:26:49.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.361+0000 7eff6dffe740 1 -- 172.21.15.44:0/2234978402 shutdown_connections 2024-04-02T13:26:49.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.361+0000 7eff6dffe740 1 --2- 172.21.15.44:0/2234978402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e66070 0x560100e95e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.361+0000 7eff6dffe740 1 --2- 172.21.15.44:0/2234978402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92f70 0x560100e95920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.361+0000 7eff6dffe740 1 --2- 172.21.15.44:0/2234978402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92b90 0x560100e65b30 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.361+0000 7eff6dffe740 1 -- 172.21.15.44:0/2234978402 >> 172.21.15.44:0/2234978402 conn(0x560100e67da0 msgr2=0x560100eb0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:49.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.361+0000 7eff6dffe740 1 -- 172.21.15.44:0/2234978402 shutdown_connections 2024-04-02T13:26:49.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.361+0000 7eff6dffe740 1 -- 172.21.15.44:0/2234978402 wait complete. 2024-04-02T13:26:49.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.361+0000 7eff6dffe740 1 -- start start 2024-04-02T13:26:49.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.362+0000 7eff6dffe740 1 Processor -- start 2024-04-02T13:26:49.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.362+0000 7eff6dffe740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92b90 0x560100e8c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.362+0000 7eff6dffe740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92f70 0x560100e8e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.362+0000 7eff6dffe740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e94e30 0x560100eb0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.362+0000 7eff6dffe740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560100e6a440 con 0x560100e92f70 2024-04-02T13:26:49.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.362+0000 7eff6dffe740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560100d8aa10 con 0x560100e92b90 2024-04-02T13:26:49.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.362+0000 7eff6dffe740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x560100e6cc80 con 0x560100e94e30 2024-04-02T13:26:49.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.362+0000 7eff6dffe740 10 client.0 _ll_register_callbacks cb 0x560100dc70b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:26:49.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff689af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92f70 0x560100e8e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff689af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92f70 0x560100e8e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49374/0 (socket says 172.21.15.44:49374) 2024-04-02T13:26:49.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff689af640 1 -- 172.21.15.44:0/2738760058 learned_addr learned my addr 172.21.15.44:0/2738760058 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:49.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff691b0640 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92b90 0x560100e8c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff699b1640 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e94e30 0x560100eb0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff689af640 1 -- 172.21.15.44:0/2738760058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e94e30 msgr2=0x560100eb0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:49.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff689af640 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e94e30 0x560100eb0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff689af640 1 -- 172.21.15.44:0/2738760058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92b90 msgr2=0x560100e8c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:49.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff689af640 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92b90 0x560100e8c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff689af640 1 -- 172.21.15.44:0/2738760058 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560100bad6b0 con 0x560100e92f70 2024-04-02T13:26:49.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff699b1640 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e94e30 0x560100eb0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:49.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff689af640 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92f70 0x560100e8e520 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7eff58000f60 tx=0x7eff58000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78085198f42aa498 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:49.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff43fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:26:49.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff43fff640 1 -- 172.21.15.44:0/2738760058 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58027060 con 0x560100e92f70 2024-04-02T13:26:49.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff6dffe740 1 -- 172.21.15.44:0/2738760058 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff540804b0 con 0x560100e92f70 2024-04-02T13:26:49.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff43fff640 1 -- 172.21.15.44:0/2738760058 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7eff58010040 con 0x560100e92f70 2024-04-02T13:26:49.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff43fff640 1 -- 172.21.15.44:0/2738760058 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7eff58003a40 con 0x560100e92f70 2024-04-02T13:26:49.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff6dffe740 10 client.7962 fetch_fsmap learned FSMap version 61 2024-04-02T13:26:49.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.363+0000 7eff6dffe740 1 -- 172.21.15.44:0/2738760058 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) -- 0x560100e95860 con 0x560100e92f70 2024-04-02T13:26:49.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.364+0000 7eff43fff640 1 -- 172.21.15.44:0/2738760058 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff540804b0 con 0x560100e92f70 2024-04-02T13:26:49.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.364+0000 7eff43fff640 1 -- 172.21.15.44:0/2738760058 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 61) ==== 3494+0+0 (secure 0 0 0) 0x7eff58003be0 con 0x560100e92f70 2024-04-02T13:26:49.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.364+0000 7eff6dffe740 10 client.7962 fetch_fsmap finished waiting for FSMap version 61 2024-04-02T13:26:49.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.364+0000 7eff6dffe740 10 client.7962 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:26:49.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.364+0000 7eff6dffe740 20 client.7962 populate_metadata read hostname 'smithi044' 2024-04-02T13:26:49.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.364+0000 7eff6dffe740 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.44:6832/719680934,v1:172.21.15.44:6833/719680934] conn(0x560100ec4df0 0x560100e98890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.364+0000 7eff6dffe740 4 client.7962 mds_command: new command op to 7803 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:26:49.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.364+0000 7eff6dffe740 1 -- 172.21.15.44:0/2738760058 --> [v2:172.21.15.44:6832/719680934,v1:172.21.15.44:6833/719680934] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560100d76cf0 con 0x560100ec4df0 2024-04-02T13:26:49.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.364+0000 7eff691b0640 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.44:6832/719680934,v1:172.21.15.44:6833/719680934] conn(0x560100ec4df0 0x560100e98890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.364+0000 7eff43fff640 10 client.7962 ms_handle_connect on v2:172.21.15.44:6832/719680934 2024-04-02T13:26:49.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.364+0000 7eff691b0640 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.44:6832/719680934,v1:172.21.15.44:6833/719680934] conn(0x560100ec4df0 0x560100e98890 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24877f60e559dc29 server_cookie=101f2b639c36c773 in_seq=0 out_seq=0 2024-04-02T13:26:49.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff43fff640 1 -- 172.21.15.44:0/2738760058 <== mds.0 v2:172.21.15.44:6832/719680934 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560100d76cf0 con 0x560100ec4df0 2024-04-02T13:26:49.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff43fff640 10 client.7962 handle_command_reply: tid=0 2024-04-02T13:26:49.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff6dffe740 1 client.7962 shutdown 2024-04-02T13:26:49.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff6dffe740 1 -- 172.21.15.44:0/2738760058 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92f70 msgr2=0x560100e8e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff6dffe740 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92f70 0x560100e8e520 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7eff58000f60 tx=0x7eff58000ed0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff6dffe740 1 -- 172.21.15.44:0/2738760058 shutdown_connections 2024-04-02T13:26:49.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff6dffe740 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.44:6832/719680934,v1:172.21.15.44:6833/719680934] conn(0x560100ec4df0 0x560100e98890 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff6dffe740 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e94e30 0x560100eb0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff6dffe740 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e92f70 0x560100e8e520 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff6dffe740 1 --2- 172.21.15.44:0/2738760058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92b90 0x560100e8c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff43fff640 0 client.7962 ms_handle_reset on v2:172.21.15.44:6832/719680934 2024-04-02T13:26:49.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff6dffe740 1 -- 172.21.15.44:0/2738760058 >> 172.21.15.44:0/2738760058 conn(0x560100e66070 msgr2=0x560100eb3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:49.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff6dffe740 1 -- 172.21.15.44:0/2738760058 shutdown_connections 2024-04-02T13:26:49.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff6dffe740 1 -- 172.21.15.44:0/2738760058 wait complete. 2024-04-02T13:26:49.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.365+0000 7eff6dffe740 20 client.7962 trim_cache size 0 max 16384 2024-04-02T13:26:49.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.383+0000 7eff6dffe740 1 Processor -- start 2024-04-02T13:26:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.383+0000 7eff6dffe740 1 -- start start 2024-04-02T13:26:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff6dffe740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92b90 0x560100eb0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff6dffe740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92f70 0x560100e93580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff6dffe740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 0x560100e93ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff6dffe740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7eff641ffef0 con 0x560100e94e30 2024-04-02T13:26:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff6dffe740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7eff641ff250 con 0x560100e92f70 2024-04-02T13:26:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff6dffe740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7eff641ff8a0 con 0x560100e92b90 2024-04-02T13:26:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff699b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 0x560100e93ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff699b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 0x560100e93ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49382/0 (socket says 172.21.15.44:49382) 2024-04-02T13:26:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff699b1640 1 -- 172.21.15.44:0/3663388181 learned_addr learned my addr 172.21.15.44:0/3663388181 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff691b0640 1 --2- 172.21.15.44:0/3663388181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92b90 0x560100eb0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff689af640 1 --2- 172.21.15.44:0/3663388181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92f70 0x560100e93580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff699b1640 1 -- 172.21.15.44:0/3663388181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92b90 msgr2=0x560100eb0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff699b1640 1 --2- 172.21.15.44:0/3663388181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92b90 0x560100eb0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff699b1640 1 -- 172.21.15.44:0/3663388181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92f70 msgr2=0x560100e93580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff699b1640 1 --2- 172.21.15.44:0/3663388181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92f70 0x560100e93580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.384+0000 7eff699b1640 1 -- 172.21.15.44:0/3663388181 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560100eb1660 con 0x560100e94e30 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.385+0000 7eff699b1640 1 --2- 172.21.15.44:0/3663388181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 0x560100e93ac0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0118b0 tx=0x7eff5c011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34c2bda4e79485c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.385+0000 7eff4a7fc640 1 -- 172.21.15.44:0/3663388181 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c01c070 con 0x560100e94e30 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.385+0000 7eff4a7fc640 1 -- 172.21.15.44:0/3663388181 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff5c004020 con 0x560100e94e30 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.385+0000 7eff4a7fc640 1 -- 172.21.15.44:0/3663388181 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0043f0 con 0x560100e94e30 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.385+0000 7eff6dffe740 1 -- 172.21.15.44:0/3663388181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 msgr2=0x560100e93ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.385+0000 7eff6dffe740 1 --2- 172.21.15.44:0/3663388181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 0x560100e93ac0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0118b0 tx=0x7eff5c011d70 comp rx=0 tx=0).stop 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.385+0000 7eff6dffe740 1 -- 172.21.15.44:0/3663388181 shutdown_connections 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.385+0000 7eff6dffe740 1 --2- 172.21.15.44:0/3663388181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 0x560100e93ac0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.385+0000 7eff6dffe740 1 --2- 172.21.15.44:0/3663388181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92f70 0x560100e93580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.385+0000 7eff6dffe740 1 --2- 172.21.15.44:0/3663388181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92b90 0x560100eb0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.385+0000 7eff6dffe740 1 -- 172.21.15.44:0/3663388181 >> 172.21.15.44:0/3663388181 conn(0x560100e66070 msgr2=0x560100e971b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.385+0000 7eff6dffe740 1 -- 172.21.15.44:0/3663388181 shutdown_connections 2024-04-02T13:26:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.385+0000 7eff6dffe740 1 -- 172.21.15.44:0/3663388181 wait complete. 2024-04-02T13:26:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.386+0000 7eff6dffe740 1 -- start start 2024-04-02T13:26:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.386+0000 7eff6dffe740 1 Processor -- start 2024-04-02T13:26:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff6dffe740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92b90 0x560100e64b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff6dffe740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92f70 0x560100eb3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff6dffe740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 0x560100eb4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff6dffe740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560100ec4c70 con 0x560100e94e30 2024-04-02T13:26:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff6dffe740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560100e95c00 con 0x560100e92f70 2024-04-02T13:26:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff6dffe740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x560100ec52c0 con 0x560100e92b90 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff6dffe740 10 client.0 _ll_register_callbacks cb 0x7eff58001420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff699b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 0x560100eb4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff699b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 0x560100eb4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49386/0 (socket says 172.21.15.44:49386) 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff699b1640 1 -- 172.21.15.44:0/1209389215 learned_addr learned my addr 172.21.15.44:0/1209389215 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff689af640 1 --2- 172.21.15.44:0/1209389215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92f70 0x560100eb3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff691b0640 1 --2- 172.21.15.44:0/1209389215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92b90 0x560100e64b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff699b1640 1 -- 172.21.15.44:0/1209389215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92b90 msgr2=0x560100e64b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff699b1640 1 --2- 172.21.15.44:0/1209389215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92b90 0x560100e64b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff699b1640 1 -- 172.21.15.44:0/1209389215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92f70 msgr2=0x560100eb3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff699b1640 1 --2- 172.21.15.44:0/1209389215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92f70 0x560100eb3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff699b1640 1 -- 172.21.15.44:0/1209389215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560100f2ced0 con 0x560100e94e30 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff699b1640 1 --2- 172.21.15.44:0/1209389215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 0x560100eb4370 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7eff5c011880 tx=0x7eff5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed64b7fe2947ef10 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff43fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:26:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.387+0000 7eff43fff640 1 -- 172.21.15.44:0/1209389215 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c01c040 con 0x560100e94e30 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff43fff640 1 -- 172.21.15.44:0/1209389215 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7eff5c013c70 con 0x560100e94e30 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff6dffe740 1 -- 172.21.15.44:0/1209389215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff5c004020 con 0x560100e94e30 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff43fff640 1 -- 172.21.15.44:0/1209389215 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7eff5c031050 con 0x560100e94e30 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff6dffe740 10 client.7968 fetch_fsmap learned FSMap version 61 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff6dffe740 1 -- 172.21.15.44:0/1209389215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) -- 0x560100f2ca40 con 0x560100e94e30 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff43fff640 1 -- 172.21.15.44:0/1209389215 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff5c004020 con 0x560100e94e30 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff43fff640 1 -- 172.21.15.44:0/1209389215 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 61) ==== 3494+0+0 (secure 0 0 0) 0x7eff5c034020 con 0x560100e94e30 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff6dffe740 10 client.7968 fetch_fsmap finished waiting for FSMap version 61 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff6dffe740 10 client.7968 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff6dffe740 20 client.7968 populate_metadata read hostname 'smithi044' 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff6dffe740 1 --2- 172.21.15.44:0/1209389215 >> [v2:172.21.15.44:6832/719680934,v1:172.21.15.44:6833/719680934] conn(0x560100e6a720 0x560100e6ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff6dffe740 4 client.7968 mds_command: new command op to 7803 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff6dffe740 1 -- 172.21.15.44:0/1209389215 --> [v2:172.21.15.44:6832/719680934,v1:172.21.15.44:6833/719680934] -- command(tid 0: {"prefix": "status"}) -- 0x560100ddc5a0 con 0x560100e6a720 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff691b0640 1 --2- 172.21.15.44:0/1209389215 >> [v2:172.21.15.44:6832/719680934,v1:172.21.15.44:6833/719680934] conn(0x560100e6a720 0x560100e6ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff43fff640 10 client.7968 ms_handle_connect on v2:172.21.15.44:6832/719680934 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.388+0000 7eff691b0640 1 --2- 172.21.15.44:0/1209389215 >> [v2:172.21.15.44:6832/719680934,v1:172.21.15.44:6833/719680934] conn(0x560100e6a720 0x560100e6ab00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a9d23d21ac16f4d server_cookie=2c3d7aa33f878ccb in_seq=0 out_seq=0 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff43fff640 1 -- 172.21.15.44:0/1209389215 <== mds.0 v2:172.21.15.44:6832/719680934 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x560100ddc5a0 con 0x560100e6a720 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff43fff640 10 client.7968 handle_command_reply: tid=0 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff6dffe740 1 client.7968 shutdown 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff6dffe740 1 -- 172.21.15.44:0/1209389215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 msgr2=0x560100eb4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff6dffe740 1 --2- 172.21.15.44:0/1209389215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 0x560100eb4370 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7eff5c011880 tx=0x7eff5c004730 comp rx=0 tx=0).stop 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff6dffe740 1 -- 172.21.15.44:0/1209389215 shutdown_connections 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff6dffe740 1 --2- 172.21.15.44:0/1209389215 >> [v2:172.21.15.44:6832/719680934,v1:172.21.15.44:6833/719680934] conn(0x560100e6a720 0x560100e6ab00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff6dffe740 1 --2- 172.21.15.44:0/1209389215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560100e94e30 0x560100eb4370 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff6dffe740 1 --2- 172.21.15.44:0/1209389215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560100e92f70 0x560100eb3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff43fff640 0 client.7968 ms_handle_reset on v2:172.21.15.44:6832/719680934 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff6dffe740 1 --2- 172.21.15.44:0/1209389215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560100e92b90 0x560100e64b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff6dffe740 1 -- 172.21.15.44:0/1209389215 >> 172.21.15.44:0/1209389215 conn(0x560100e66070 msgr2=0x560100e97110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff6dffe740 1 -- 172.21.15.44:0/1209389215 shutdown_connections 2024-04-02T13:26:49.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff6dffe740 1 -- 172.21.15.44:0/1209389215 wait complete. 2024-04-02T13:26:49.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.389+0000 7eff6dffe740 20 client.7968 trim_cache size 0 max 16384 2024-04-02T13:26:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:26:49.402 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:26:49.402 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:26:49.402 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 7803, 2024-04-02T13:26:49.402 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:26:49.402 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:26:49.403 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:26:49.403 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 5.696567237, 2024-04-02T13:26:49.403 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 61, 2024-04-02T13:26:49.403 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 91, 2024-04-02T13:26:49.403 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 91, 2024-04-02T13:26:49.403 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 10.736170538 2024-04-02T13:26:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:26:49.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.390+0000 7eff6b414640 1 -- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff5006c8f0 msgr2=0x7eff5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.390+0000 7eff6b414640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff5006c8f0 0x7eff5006edb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7eff58007fd0 tx=0x7eff58009040 comp rx=0 tx=0).stop 2024-04-02T13:26:49.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.390+0000 7eff6b414640 1 -- 172.21.15.44:0/4293753419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff6406c7f0 msgr2=0x7eff64155880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.390+0000 7eff6b414640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff6406c7f0 0x7eff64155880 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7eff54000f80 tx=0x7eff54004a40 comp rx=0 tx=0).stop 2024-04-02T13:26:49.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.390+0000 7eff6b414640 1 -- 172.21.15.44:0/4293753419 shutdown_connections 2024-04-02T13:26:49.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.390+0000 7eff6b414640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7eff5006c8f0 0x7eff5006edb0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.390+0000 7eff6b414640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff6415e430 0x7eff641f77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.390+0000 7eff6b414640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff640b1ef0 0x7eff64152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.390+0000 7eff6b414640 1 --2- 172.21.15.44:0/4293753419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff6406c7f0 0x7eff64155880 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.390+0000 7eff6b414640 1 -- 172.21.15.44:0/4293753419 >> 172.21.15.44:0/4293753419 conn(0x7eff64073850 msgr2=0x7eff640b0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:49.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.390+0000 7eff6b414640 1 -- 172.21.15.44:0/4293753419 shutdown_connections 2024-04-02T13:26:49.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.390+0000 7eff6b414640 1 -- 172.21.15.44:0/4293753419 wait complete. 2024-04-02T13:26:49.406 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:26:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.505+0000 7fe135e81640 1 Processor -- start 2024-04-02T13:26:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.505+0000 7fe135e81640 1 -- start start 2024-04-02T13:26:49.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe135e81640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe1300d28f0 0x7fe1300d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe135e81640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1300d4d70 con 0x7fe1300b47b0 2024-04-02T13:26:49.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe135e81640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1300d2030 con 0x7fe1300b5200 2024-04-02T13:26:49.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe135e81640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe1300d5050 con 0x7fe1300d28f0 2024-04-02T13:26:49.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12ffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe1300b47b0 0x7fe1300b7120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42966/0 (socket says 172.21.15.44:42966) 2024-04-02T13:26:49.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12ffff640 1 -- 172.21.15.44:0/2027887452 learned_addr learned my addr 172.21.15.44:0/2027887452 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:49.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12f7fe640 1 --2- 172.21.15.44:0/2027887452 >> v2:172.21.15.69:3300/0 conn(0x7fe1300d28f0 0x7fe1300d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 936983747 0 0) 0x7fe1300d4d70 con 0x7fe1300b47b0 2024-04-02T13:26:49.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe118003520 con 0x7fe1300b47b0 2024-04-02T13:26:49.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3784148997 0 0) 0x7fe118003520 con 0x7fe1300b47b0 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 >> v2:172.21.15.69:3300/0 conn(0x7fe1300d28f0 msgr2=0x7fe1300d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12e7fc640 1 --2- 172.21.15.44:0/2027887452 >> v2:172.21.15.69:3300/0 conn(0x7fe1300d28f0 0x7fe1300d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 >> v1:172.21.15.69:6789/0 conn(0x7fe1300b5200 legacy=0x7fe1300b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1300b4cb0 con 0x7fe1300b47b0 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fe1280019f0 con 0x7fe1300b47b0 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 >> v1:172.21.15.44:6789/0 conn(0x7fe1300b47b0 legacy=0x7fe1300b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12e7fc640 1 --2- 172.21.15.44:0/2027887452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1180039b0 0x7fe118003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12e7fc640 1 --2- 172.21.15.44:0/2027887452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe118004800 0x7fe118004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.506+0000 7fe12e7fc640 1 --2- 172.21.15.44:0/2027887452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1180051c0 0x7fe11800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe1300d5050 con 0x7fe118004800 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe11800c110 con 0x7fe1180051c0 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe11800c310 con 0x7fe1180039b0 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12effd640 1 --2- 172.21.15.44:0/2027887452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe118004800 0x7fe118004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12f7fe640 1 --2- 172.21.15.44:0/2027887452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1180039b0 0x7fe118003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12ffff640 1 --2- 172.21.15.44:0/2027887452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1180051c0 0x7fe11800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12effd640 1 -- 172.21.15.44:0/2027887452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1180039b0 msgr2=0x7fe118003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12effd640 1 --2- 172.21.15.44:0/2027887452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1180039b0 0x7fe118003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12effd640 1 -- 172.21.15.44:0/2027887452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1180051c0 msgr2=0x7fe11800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12effd640 1 --2- 172.21.15.44:0/2027887452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1180051c0 0x7fe11800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12effd640 1 -- 172.21.15.44:0/2027887452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe118003520 con 0x7fe118004800 2024-04-02T13:26:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12f7fe640 1 --2- 172.21.15.44:0/2027887452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1180039b0 0x7fe118003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12ffff640 1 --2- 172.21.15.44:0/2027887452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1180051c0 0x7fe11800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.507+0000 7fe12ffff640 1 -- 172.21.15.44:0/2027887452 reap_dead start 2024-04-02T13:26:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.508+0000 7fe12effd640 1 --2- 172.21.15.44:0/2027887452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe118004800 0x7fe118004c80 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fe120001910 tx=0x7fe118005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade3735c27ca2c26 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.508+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1200014e0 con 0x7fe118004800 2024-04-02T13:26:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.508+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe120037070 con 0x7fe118004800 2024-04-02T13:26:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.508+0000 7fe12e7fc640 1 -- 172.21.15.44:0/2027887452 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120032860 con 0x7fe118004800 2024-04-02T13:26:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.508+0000 7fe135e81640 1 -- 172.21.15.44:0/2027887452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe118004800 msgr2=0x7fe118004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.508+0000 7fe135e81640 1 --2- 172.21.15.44:0/2027887452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe118004800 0x7fe118004c80 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fe120001910 tx=0x7fe118005830 comp rx=0 tx=0).stop 2024-04-02T13:26:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.508+0000 7fe135e81640 1 -- 172.21.15.44:0/2027887452 shutdown_connections 2024-04-02T13:26:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.508+0000 7fe135e81640 1 --2- 172.21.15.44:0/2027887452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe118004800 0x7fe118004c80 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.508+0000 7fe135e81640 1 -- 172.21.15.44:0/2027887452 >> 172.21.15.44:0/2027887452 conn(0x7fe1300d4120 msgr2=0x7fe1300d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.508+0000 7fe135e81640 1 -- 172.21.15.44:0/2027887452 shutdown_connections 2024-04-02T13:26:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.508+0000 7fe135e81640 1 -- 172.21.15.44:0/2027887452 wait complete. 2024-04-02T13:26:49.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.509+0000 7fe135e81640 1 Processor -- start 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.510+0000 7fe135e81640 1 -- start start 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.510+0000 7fe135e81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe118004800 0x7fe1301f7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.510+0000 7fe135e81640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe13015c110 0x7fe1301f68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.510+0000 7fe135e81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13015c590 0x7fe1301f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.510+0000 7fe135e81640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe1300d23a0 con 0x7fe13015c110 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.510+0000 7fe135e81640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe1300d1470 con 0x7fe118004800 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.510+0000 7fe135e81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe1300b5020 con 0x7fe13015c590 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.510+0000 7fe12effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe13015c110 0x7fe1301f68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.510+0000 7fe12effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe13015c110 0x7fe1301f68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49402/0 (socket says 172.21.15.44:49402) 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.510+0000 7fe12effd640 1 -- 172.21.15.44:0/450209522 learned_addr learned my addr 172.21.15.44:0/450209522 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.511+0000 7fe12f7fe640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe118004800 0x7fe1301f7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.511+0000 7fe12ffff640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13015c590 0x7fe1301f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.511+0000 7fe12effd640 1 -- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13015c590 msgr2=0x7fe1301f6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.511+0000 7fe12effd640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13015c590 0x7fe1301f6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.511+0000 7fe12effd640 1 -- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe118004800 msgr2=0x7fe1301f7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.511+0000 7fe12effd640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe118004800 0x7fe1301f7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.511+0000 7fe12effd640 1 -- 172.21.15.44:0/450209522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1301f8390 con 0x7fe13015c110 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.511+0000 7fe12effd640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe13015c110 0x7fe1301f68d0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fe120034040 tx=0x7fe1200012b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.511+0000 7fe117fff640 1 -- 172.21.15.44:0/450209522 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120007a70 con 0x7fe13015c110 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.511+0000 7fe117fff640 1 -- 172.21.15.44:0/450209522 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe120047480 con 0x7fe13015c110 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.511+0000 7fe117fff640 1 -- 172.21.15.44:0/450209522 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120047870 con 0x7fe13015c110 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.511+0000 7fe135e81640 1 -- 172.21.15.44:0/450209522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1300b37e0 con 0x7fe13015c110 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.511+0000 7fe135e81640 1 -- 172.21.15.44:0/450209522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1300cb290 con 0x7fe13015c110 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.512+0000 7fe135e81640 1 -- 172.21.15.44:0/450209522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1300684f0 con 0x7fe13015c110 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.512+0000 7fe117fff640 1 -- 172.21.15.44:0/450209522 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe120031070 con 0x7fe13015c110 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.513+0000 7fe117fff640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe11c06c8f0 0x7fe11c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.513+0000 7fe117fff640 1 -- 172.21.15.44:0/450209522 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7fe1200adb70 con 0x7fe13015c110 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.513+0000 7fe12f7fe640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe11c06c8f0 0x7fe11c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:49.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.514+0000 7fe12f7fe640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe11c06c8f0 0x7fe11c06edb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe124009b10 tx=0x7fe124008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:49.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.517+0000 7fe117fff640 1 -- 172.21.15.44:0/450209522 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe12007bea0 con 0x7fe13015c110 2024-04-02T13:26:49.647 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:26:49.645+0000 7f91a4f1f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:26:49.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.742+0000 7fe135e81640 1 -- 172.21.15.44:0/450209522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe1301e6f90 con 0x7fe13015c110 2024-04-02T13:26:49.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.742+0000 7fe117fff640 1 -- 172.21.15.44:0/450209522 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v61) ==== 53+0+73 (secure 0 0 0) 0x7fe12007b7a0 con 0x7fe13015c110 2024-04-02T13:26:49.744 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:26:49.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.744+0000 7fe135e81640 1 -- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe11c06c8f0 msgr2=0x7fe11c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.744+0000 7fe135e81640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe11c06c8f0 0x7fe11c06edb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe124009b10 tx=0x7fe124008040 comp rx=0 tx=0).stop 2024-04-02T13:26:49.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.744+0000 7fe135e81640 1 -- 172.21.15.44:0/450209522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe13015c110 msgr2=0x7fe1301f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:49.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.744+0000 7fe135e81640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe13015c110 0x7fe1301f68d0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fe120034040 tx=0x7fe1200012b0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.744+0000 7fe135e81640 1 -- 172.21.15.44:0/450209522 shutdown_connections 2024-04-02T13:26:49.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.744+0000 7fe135e81640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe11c06c8f0 0x7fe11c06edb0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.744+0000 7fe135e81640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13015c590 0x7fe1301f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.744+0000 7fe135e81640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe13015c110 0x7fe1301f68d0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.744+0000 7fe135e81640 1 --2- 172.21.15.44:0/450209522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe118004800 0x7fe1301f7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:49.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.744+0000 7fe135e81640 1 -- 172.21.15.44:0/450209522 >> 172.21.15.44:0/450209522 conn(0x7fe1300d4120 msgr2=0x7fe1300b7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:49.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.744+0000 7fe135e81640 1 -- 172.21.15.44:0/450209522 shutdown_connections 2024-04-02T13:26:49.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49.744+0000 7fe135e81640 1 -- 172.21.15.44:0/450209522 wait complete. 2024-04-02T13:26:49.757 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:26:49.757 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:26:49.757 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:26:49.757 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:26:49.757 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:49.757 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:26:49.757 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:26:49.758 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:26:49.758 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:26:49.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:49.758 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:26:49.773 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42369sec preferred_lft 42369sec 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:49.774 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:26:49.775 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:26:49.775 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:26:49.775 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:49.775 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:26:49.775 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:26:49.775 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:26:49.775 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:26:49.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:49.775 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:26:49.839 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:26:49.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:49.840 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:26:49.896 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:26:49.896 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:26:49.896 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:26:49.897 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:26:49.897 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:26:49.897 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:26:49.897 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:26:49.897 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:26:49.897 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:26:49.897 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:26:49.897 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:26:49.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:26:50.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:26:50.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:50.031 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:26:50.046 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:26:50.046 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:26:50.046 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:26:50.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:50.046 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:26:50.112 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:26:50.112 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:26:50.112 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:50.168 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:26:50.168 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:26:50.168 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:50.223 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:26:50.290 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:26:50.345 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:26:50.345 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:26:50.345 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:26:50.345 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:26:50.345 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:26:50.345 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:26:50.346 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:26:50.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:50.348 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:26:50.399 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:26:50.399 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:26:50.441 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:26:50.468 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:26:50.523 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:26:50.523 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:26:50.523 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:26:50.524 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:26:50.524 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:26:50.524 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:26:50.524 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:26:50.524 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:26:50.524 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:26:50.524 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:26:50.524 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:26:50.524 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:26:50.524 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:26:50.524 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:26:50.524 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:26:50.524 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:26:50.525 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:26:50.526 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:26:50.526 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:26:50.526 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:26:50.526 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:26:50.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:50.526 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:26:50.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:26:50.524+0000 7fe9469db480 -1 init, newargv = 0x55d17c4b9100 newargc=15 2024-04-02T13:26:50.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[69777]: starting ceph client 2024-04-02T13:26:50.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[69777]: starting fuse 2024-04-02T13:26:50.541 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:26:50.541 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:26:50.541 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:26:50.541 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:26:50.542 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:26:50.543 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:26:50.543 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:26:50.543 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:26:50.543 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:26:50.543 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:26:50.543 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:26:50.543 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:26:50.543 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:26:50.543 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:26:50.543 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:26:50.543 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:26:50.662 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.69777.asok 2024-04-02T13:26:50.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:26:50.673 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.69777.asok 2024-04-02T13:26:50.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:50.673 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.69777.asok status 2024-04-02T13:26:50.810 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:26:50.810 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:26:50.810 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:26:50.810 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:26:50.811 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:26:50.811 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:26:50.811 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:26:50.811 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "69777", 2024-04-02T13:26:50.811 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:26:50.811 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:50.811 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:26:50.811 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:26:50.811 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 7983, 2024-04-02T13:26:50.811 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:26:50.811 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:26:50.812 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:26:50.812 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 7983 2024-04-02T13:26:50.812 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:50.812 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:26:50.812 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:26:50.812 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:26:50.812 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2300085654 2024-04-02T13:26:50.812 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:26:50.812 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:50.812 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:26:50.812 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:26:50.812 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:26:50.812 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2300085654 2024-04-02T13:26:50.813 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:26:50.813 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.7983 192.168.0.1:0/2300085654", 2024-04-02T13:26:50.813 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/2300085654", 2024-04-02T13:26:50.813 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:26:50.813 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 61, 2024-04-02T13:26:50.813 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 91, 2024-04-02T13:26:50.813 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:26:50.813 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:26:50.813 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:26:50.813 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:26:50.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:50.819 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:26:50.819 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:50.834 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:26:50.834 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:50.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:50.834 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:50.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:50.901 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir) 2024-04-02T13:26:50.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:50.961 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir/file) 2024-04-02T13:26:51.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:51.018 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir testdir) 2024-04-02T13:26:51.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:51.075 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir/file testdir/hardlink) 2024-04-02T13:26:51.137 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:26:51.137 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:26:51.137 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:26:51.137 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:26:51.137 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir").st_ino) 2024-04-02T13:26:51.137 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:26:51.241 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627776 2024-04-02T13:26:51.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:26:51.249 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:26:51.250 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:26:51.305 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:26:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.444+0000 7f8ec1c86640 1 Processor -- start 2024-04-02T13:26:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.444+0000 7f8ec1c86640 1 -- start start 2024-04-02T13:26:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.444+0000 7f8ec1c86640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ebc0b2d00 0x7f8ebc0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.444+0000 7f8ec1c86640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8ebc0ce110 0x7f8ebc071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.444+0000 7f8ec1c86640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ebc0b3160 con 0x7f8ebc15aa00 2024-04-02T13:26:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.444+0000 7f8ec1c86640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f8ebc15ade0 con 0x7f8ebc0ce110 2024-04-02T13:26:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.444+0000 7f8ec1c86640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8ebc0af1b0 con 0x7f8ebc0b2d00 2024-04-02T13:26:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.445+0000 7f8ebbfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8ebc0ce110 0x7f8ebc071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.445+0000 7f8ebbfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8ebc0ce110 0x7f8ebc071190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49414/0 (socket says 172.21.15.44:49414) 2024-04-02T13:26:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.445+0000 7f8ebbfff640 1 -- 172.21.15.44:0/3712684364 learned_addr learned my addr 172.21.15.44:0/3712684364 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.445+0000 7f8ebaffd640 1 --2- 172.21.15.44:0/3712684364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ebc0b2d00 0x7f8ebc0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.445+0000 7f8ebbfff640 1 -- 172.21.15.44:0/3712684364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ebc0b2d00 msgr2=0x7f8ebc0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.445+0000 7f8ebbfff640 1 --2- 172.21.15.44:0/3712684364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ebc0b2d00 0x7f8ebc0cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.445+0000 7f8ebbfff640 1 -- 172.21.15.44:0/3712684364 >> v1:172.21.15.69:6789/0 conn(0x7f8ebc15aa00 legacy=0x7f8ebc072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.445+0000 7f8ebbfff640 1 -- 172.21.15.44:0/3712684364 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ebc0af490 con 0x7f8ebc0ce110 2024-04-02T13:26:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.446+0000 7f8ebbfff640 1 --2- 172.21.15.44:0/3712684364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ebc0ce110 0x7f8ebc071190 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400d150 tx=0x7f8eb402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b68502c8d2fb931 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.446+0000 7f8eba7fc640 1 -- 172.21.15.44:0/3712684364 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb402ea80 con 0x7f8ebc0ce110 2024-04-02T13:26:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.446+0000 7f8eba7fc640 1 -- 172.21.15.44:0/3712684364 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8eb402ec20 con 0x7f8ebc0ce110 2024-04-02T13:26:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.446+0000 7f8eba7fc640 1 -- 172.21.15.44:0/3712684364 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4009600 con 0x7f8ebc0ce110 2024-04-02T13:26:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.446+0000 7f8ec1c86640 1 -- 172.21.15.44:0/3712684364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ebc0ce110 msgr2=0x7f8ebc071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.446+0000 7f8ec1c86640 1 --2- 172.21.15.44:0/3712684364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ebc0ce110 0x7f8ebc071190 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400d150 tx=0x7f8eb402e0e0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.446+0000 7f8ec1c86640 1 -- 172.21.15.44:0/3712684364 shutdown_connections 2024-04-02T13:26:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.446+0000 7f8ec1c86640 1 --2- 172.21.15.44:0/3712684364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ebc0ce110 0x7f8ebc071190 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.446+0000 7f8ec1c86640 1 --2- 172.21.15.44:0/3712684364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ebc0b2d00 0x7f8ebc0cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.446+0000 7f8ec1c86640 1 -- 172.21.15.44:0/3712684364 >> 172.21.15.44:0/3712684364 conn(0x7f8ebc0cfcd0 msgr2=0x7f8ebc0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.446+0000 7f8ec1c86640 1 -- 172.21.15.44:0/3712684364 shutdown_connections 2024-04-02T13:26:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.446+0000 7f8ec1c86640 1 -- 172.21.15.44:0/3712684364 wait complete. 2024-04-02T13:26:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.447+0000 7f8ec1c86640 1 Processor -- start 2024-04-02T13:26:51.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.448+0000 7f8ec1c86640 1 -- start start 2024-04-02T13:26:51.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.448+0000 7f8ec1c86640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ebc0b2d00 0x7f8ebc09cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.448+0000 7f8ec1c86640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ebc0ce110 0x7f8ebc0b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.448+0000 7f8ec1c86640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ebc15aa00 0x7f8ebc14c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.448+0000 7f8ec1c86640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc0ce5a0 con 0x7f8ebc0ce110 2024-04-02T13:26:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.448+0000 7f8ec1c86640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc0726d0 con 0x7f8ebc15aa00 2024-04-02T13:26:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.448+0000 7f8ec1c86640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8ebc0cc750 con 0x7f8ebc0b2d00 2024-04-02T13:26:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.448+0000 7f8ebaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ebc0ce110 0x7f8ebc0b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.448+0000 7f8ebaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ebc0ce110 0x7f8ebc0b0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49422/0 (socket says 172.21.15.44:49422) 2024-04-02T13:26:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.448+0000 7f8ebaffd640 1 -- 172.21.15.44:0/1197160676 learned_addr learned my addr 172.21.15.44:0/1197160676 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.449+0000 7f8ebb7fe640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ebc0b2d00 0x7f8ebc09cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.449+0000 7f8ebbfff640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ebc15aa00 0x7f8ebc14c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.449+0000 7f8ebaffd640 1 -- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ebc0b2d00 msgr2=0x7f8ebc09cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.449+0000 7f8ebaffd640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ebc0b2d00 0x7f8ebc09cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.449+0000 7f8ebaffd640 1 -- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ebc15aa00 msgr2=0x7f8ebc14c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:51.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.449+0000 7f8ebaffd640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ebc15aa00 0x7f8ebc14c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.449+0000 7f8ebaffd640 1 -- 172.21.15.44:0/1197160676 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ebc09d480 con 0x7f8ebc0ce110 2024-04-02T13:26:51.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.449+0000 7f8ebaffd640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ebc0ce110 0x7f8ebc0b0fc0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000d830 tx=0x7f8eb000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:51.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.449+0000 7f8ea3fff640 1 -- 172.21.15.44:0/1197160676 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb000b6c0 con 0x7f8ebc0ce110 2024-04-02T13:26:51.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.449+0000 7f8ea3fff640 1 -- 172.21.15.44:0/1197160676 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8eb0019070 con 0x7f8ebc0ce110 2024-04-02T13:26:51.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.449+0000 7f8ec1c86640 1 -- 172.21.15.44:0/1197160676 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ebc14d3a0 con 0x7f8ebc0ce110 2024-04-02T13:26:51.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.449+0000 7f8ea3fff640 1 -- 172.21.15.44:0/1197160676 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb000ba80 con 0x7f8ebc0ce110 2024-04-02T13:26:51.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.449+0000 7f8ec1c86640 1 -- 172.21.15.44:0/1197160676 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ebc09d760 con 0x7f8ebc0ce110 2024-04-02T13:26:51.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.450+0000 7f8ec1c86640 1 -- 172.21.15.44:0/1197160676 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ebc074f20 con 0x7f8ebc0ce110 2024-04-02T13:26:51.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.451+0000 7f8ea3fff640 1 -- 172.21.15.44:0/1197160676 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8eb001e460 con 0x7f8ebc0ce110 2024-04-02T13:26:51.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.451+0000 7f8ea3fff640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8ea806c8f0 0x7f8ea806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:51.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.451+0000 7f8ea3fff640 1 -- 172.21.15.44:0/1197160676 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7f8eb008c430 con 0x7f8ebc0ce110 2024-04-02T13:26:51.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.451+0000 7f8ebb7fe640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8ea806c8f0 0x7f8ea806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:51.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.452+0000 7f8ebb7fe640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8ea806c8f0 0x7f8ea806edb0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8eac001500 tx=0x7f8eac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:51.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.455+0000 7f8ea3fff640 1 -- 172.21.15.44:0/1197160676 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8eb005a6d0 con 0x7f8ebc0ce110 2024-04-02T13:26:51.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.642+0000 7f8ec1c86640 1 -- 172.21.15.44:0/1197160676 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ebc0684f0 con 0x7f8ebc0ce110 2024-04-02T13:26:51.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.643+0000 7f8ea3fff640 1 -- 172.21.15.44:0/1197160676 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+16837 (secure 0 0 0) 0x7f8eb0059fd0 con 0x7f8ebc0ce110 2024-04-02T13:26:51.645 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:51.645 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:26:44.676976+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:26:41.300970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-02T13:26:41.979586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:26:51.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.645+0000 7f8ec1c86640 1 -- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8ea806c8f0 msgr2=0x7f8ea806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:51.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.645+0000 7f8ec1c86640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8ea806c8f0 0x7f8ea806edb0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8eac001500 tx=0x7f8eac009040 comp rx=0 tx=0).stop 2024-04-02T13:26:51.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.645+0000 7f8ec1c86640 1 -- 172.21.15.44:0/1197160676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ebc0ce110 msgr2=0x7f8ebc0b0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:51.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.645+0000 7f8ec1c86640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ebc0ce110 0x7f8ebc0b0fc0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000d830 tx=0x7f8eb000dcf0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.645+0000 7f8ec1c86640 1 -- 172.21.15.44:0/1197160676 shutdown_connections 2024-04-02T13:26:51.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.645+0000 7f8ec1c86640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8ea806c8f0 0x7f8ea806edb0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.645+0000 7f8ec1c86640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ebc15aa00 0x7f8ebc14c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.645+0000 7f8ec1c86640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ebc0ce110 0x7f8ebc0b0fc0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.645+0000 7f8ec1c86640 1 --2- 172.21.15.44:0/1197160676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ebc0b2d00 0x7f8ebc09cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.645+0000 7f8ec1c86640 1 -- 172.21.15.44:0/1197160676 >> 172.21.15.44:0/1197160676 conn(0x7f8ebc0cfcd0 msgr2=0x7f8ebc0afba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:51.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.645+0000 7f8ec1c86640 1 -- 172.21.15.44:0/1197160676 shutdown_connections 2024-04-02T13:26:51.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.645+0000 7f8ec1c86640 1 -- 172.21.15.44:0/1197160676 wait complete. 2024-04-02T13:26:51.658 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:26:41.300970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:26:41.979586+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:26:51.658 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:26:51.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.764+0000 7f9eb1d4d640 1 Processor -- start 2024-04-02T13:26:51.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.764+0000 7f9eb1d4d640 1 -- start start 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.765+0000 7f9eb1d4d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9eac0cec60 0x7f9eac0d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.765+0000 7f9eb1d4d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9eac0cf1f0 con 0x7f9eac15e1d0 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.765+0000 7f9eb1d4d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9eac0cf4d0 con 0x7f9eac15e5b0 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.765+0000 7f9eb1d4d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f9eac0cfe00 con 0x7f9eac0cec60 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.765+0000 7f9eab7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f9eac15e1d0 0x7f9eac0d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42976/0 (socket says 172.21.15.44:42976) 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.765+0000 7f9eab7fe640 1 -- 172.21.15.44:0/4193544021 learned_addr learned my addr 172.21.15.44:0/4193544021 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.765+0000 7f9eaaffd640 1 --2- 172.21.15.44:0/4193544021 >> v2:172.21.15.44:3300/0 conn(0x7f9eac0cec60 0x7f9eac0d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.766+0000 7f9eaa7fc640 1 -- 172.21.15.44:0/4193544021 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1192995035 0 0) 0x7f9eac0cf1f0 con 0x7f9eac15e1d0 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.766+0000 7f9eaa7fc640 1 -- 172.21.15.44:0/4193544021 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e94003520 con 0x7f9eac15e1d0 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.766+0000 7f9eaaffd640 1 -- 172.21.15.44:0/4193544021 >> v1:172.21.15.69:6789/0 conn(0x7f9eac15e5b0 legacy=0x7f9eac0b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.766+0000 7f9eaaffd640 1 -- 172.21.15.44:0/4193544021 >> v1:172.21.15.44:6789/0 conn(0x7f9eac15e1d0 legacy=0x7f9eac0d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.766+0000 7f9eaaffd640 1 -- 172.21.15.44:0/4193544021 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac0cf7b0 con 0x7f9eac0cec60 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.766+0000 7f9eaaffd640 1 --2- 172.21.15.44:0/4193544021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9eac0cec60 0x7f9eac0d26c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f9ea00097a0 tx=0x7f9ea002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da0ecf789c7d825 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:51.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.767+0000 7f9eaa7fc640 1 -- 172.21.15.44:0/4193544021 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0035020 con 0x7f9eac0cec60 2024-04-02T13:26:51.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.767+0000 7f9eaa7fc640 1 -- 172.21.15.44:0/4193544021 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea0031830 con 0x7f9eac0cec60 2024-04-02T13:26:51.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.767+0000 7f9eaa7fc640 1 -- 172.21.15.44:0/4193544021 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0031be0 con 0x7f9eac0cec60 2024-04-02T13:26:51.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.767+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/4193544021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9eac0cec60 msgr2=0x7f9eac0d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:51.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.767+0000 7f9eb1d4d640 1 --2- 172.21.15.44:0/4193544021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9eac0cec60 0x7f9eac0d26c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f9ea00097a0 tx=0x7f9ea002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.767+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/4193544021 shutdown_connections 2024-04-02T13:26:51.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.767+0000 7f9eb1d4d640 1 --2- 172.21.15.44:0/4193544021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9eac0cec60 0x7f9eac0d26c0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.767+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/4193544021 >> 172.21.15.44:0/4193544021 conn(0x7f9eac09d3b0 msgr2=0x7f9eac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:51.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.767+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/4193544021 shutdown_connections 2024-04-02T13:26:51.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.767+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/4193544021 wait complete. 2024-04-02T13:26:51.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.768+0000 7f9eb1d4d640 1 Processor -- start 2024-04-02T13:26:51.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.768+0000 7f9eb1d4d640 1 -- start start 2024-04-02T13:26:51.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.769+0000 7f9eb1d4d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0cec60 0x7f9eac1f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:51.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.769+0000 7f9eb1d4d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eac15e1d0 0x7f9eac0b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:51.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.769+0000 7f9eb1d4d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9eac15e5b0 0x7f9eac0b8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:51.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.769+0000 7f9eb1d4d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac0b8150 con 0x7f9eac15e5b0 2024-04-02T13:26:51.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.769+0000 7f9eb1d4d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac0b7fd0 con 0x7f9eac0cec60 2024-04-02T13:26:51.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.769+0000 7f9eb1d4d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9eac0b82d0 con 0x7f9eac15e1d0 2024-04-02T13:26:51.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.769+0000 7f9eabfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9eac15e5b0 0x7f9eac0b8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:51.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.769+0000 7f9eabfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9eac15e5b0 0x7f9eac0b8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49434/0 (socket says 172.21.15.44:49434) 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.769+0000 7f9eabfff640 1 -- 172.21.15.44:0/1798566803 learned_addr learned my addr 172.21.15.44:0/1798566803 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.769+0000 7f9eab7fe640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0cec60 0x7f9eac1f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.769+0000 7f9eaaffd640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eac15e1d0 0x7f9eac0b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.770+0000 7f9eabfff640 1 -- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eac15e1d0 msgr2=0x7f9eac0b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.770+0000 7f9eabfff640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eac15e1d0 0x7f9eac0b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.770+0000 7f9eabfff640 1 -- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0cec60 msgr2=0x7f9eac1f4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.770+0000 7f9eabfff640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0cec60 0x7f9eac1f4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.770+0000 7f9eabfff640 1 -- 172.21.15.44:0/1798566803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac1f4fe0 con 0x7f9eac15e5b0 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.770+0000 7f9eabfff640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9eac15e5b0 0x7f9eac0b8960 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400c6e0 tx=0x7f9ea400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.770+0000 7f9e93fff640 1 -- 172.21.15.44:0/1798566803 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4003b70 con 0x7f9eac15e5b0 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.770+0000 7f9e93fff640 1 -- 172.21.15.44:0/1798566803 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea4003d30 con 0x7f9eac15e5b0 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.770+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/1798566803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eac0b56c0 con 0x7f9eac15e5b0 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.770+0000 7f9e93fff640 1 -- 172.21.15.44:0/1798566803 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4005310 con 0x7f9eac15e5b0 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.770+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/1798566803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eac161160 con 0x7f9eac15e5b0 2024-04-02T13:26:51.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.771+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/1798566803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eac0684f0 con 0x7f9eac15e5b0 2024-04-02T13:26:51.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.771+0000 7f9e93fff640 1 -- 172.21.15.44:0/1798566803 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9ea40054b0 con 0x7f9eac15e5b0 2024-04-02T13:26:51.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.772+0000 7f9e93fff640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9e9806c920 0x7f9e9806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:51.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.772+0000 7f9e93fff640 1 -- 172.21.15.44:0/1798566803 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7f9ea408b940 con 0x7f9eac15e5b0 2024-04-02T13:26:51.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.772+0000 7f9eab7fe640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9e9806c920 0x7f9e9806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:51.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.773+0000 7f9eab7fe640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9e9806c920 0x7f9e9806ede0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0011e0 tx=0x7f9e9c007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:51.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.776+0000 7f9e93fff640 1 -- 172.21.15.44:0/1798566803 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9eac1f4fe0 con 0x7f9eac15e5b0 2024-04-02T13:26:51.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.962+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/1798566803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9eac0d44f0 con 0x7f9eac15e5b0 2024-04-02T13:26:51.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.962+0000 7f9e93fff640 1 -- 172.21.15.44:0/1798566803 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f9ea4059620 con 0x7f9eac15e5b0 2024-04-02T13:26:51.965 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:26:51.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.964+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9e9806c920 msgr2=0x7f9e9806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:51.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.964+0000 7f9eb1d4d640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9e9806c920 0x7f9e9806ede0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0011e0 tx=0x7f9e9c007d30 comp rx=0 tx=0).stop 2024-04-02T13:26:51.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.964+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/1798566803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9eac15e5b0 msgr2=0x7f9eac0b8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:51.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.964+0000 7f9eb1d4d640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9eac15e5b0 0x7f9eac0b8960 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400c6e0 tx=0x7f9ea400cba0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.964+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/1798566803 shutdown_connections 2024-04-02T13:26:51.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.964+0000 7f9eb1d4d640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9e9806c920 0x7f9e9806ede0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.964+0000 7f9eb1d4d640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9eac15e5b0 0x7f9eac0b8960 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.964+0000 7f9eb1d4d640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eac15e1d0 0x7f9eac0b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.964+0000 7f9eb1d4d640 1 --2- 172.21.15.44:0/1798566803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0cec60 0x7f9eac1f4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:51.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.964+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/1798566803 >> 172.21.15.44:0/1798566803 conn(0x7f9eac09d3b0 msgr2=0x7f9eac0cd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:51.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.964+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/1798566803 shutdown_connections 2024-04-02T13:26:51.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:51.964+0000 7f9eb1d4d640 1 -- 172.21.15.44:0/1798566803 wait complete. 2024-04-02T13:26:51.977 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:26:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.079+0000 7ff92dad6640 1 Processor -- start 2024-04-02T13:26:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.079+0000 7ff92dad6640 1 -- start start 2024-04-02T13:26:52.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff92dad6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff92815e260 0x7ff9280d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff92dad6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff92815e640 0x7ff9280d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff92dad6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff9280cec60 0x7ff9280d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff92dad6640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff9280b5a50 con 0x7ff9280cec60 2024-04-02T13:26:52.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff92dad6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff9280d01c0 con 0x7ff92815e640 2024-04-02T13:26:52.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff92dad6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff9280cf120 con 0x7ff92815e260 2024-04-02T13:26:52.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff927fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff9280cec60 0x7ff9280d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff927fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff9280cec60 0x7ff9280d1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49444/0 (socket says 172.21.15.44:49444) 2024-04-02T13:26:52.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff927fff640 1 -- 172.21.15.44:0/2038757981 learned_addr learned my addr 172.21.15.44:0/2038757981 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:52.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff926ffd640 1 --2- 172.21.15.44:0/2038757981 >> v2:172.21.15.69:3300/0 conn(0x7ff92815e640 0x7ff9280d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff9277fe640 1 --2- 172.21.15.44:0/2038757981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff92815e260 0x7ff9280d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff927fff640 1 -- 172.21.15.44:0/2038757981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff92815e260 msgr2=0x7ff9280d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:52.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff927fff640 1 --2- 172.21.15.44:0/2038757981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff92815e260 0x7ff9280d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff927fff640 1 -- 172.21.15.44:0/2038757981 >> v2:172.21.15.69:3300/0 conn(0x7ff92815e640 msgr2=0x7ff9280d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:52.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff927fff640 1 --2- 172.21.15.44:0/2038757981 >> v2:172.21.15.69:3300/0 conn(0x7ff92815e640 0x7ff9280d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.080+0000 7ff927fff640 1 -- 172.21.15.44:0/2038757981 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9280cf400 con 0x7ff9280cec60 2024-04-02T13:26:52.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.081+0000 7ff927fff640 1 --2- 172.21.15.44:0/2038757981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9280cec60 0x7ff9280d1010 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7ff92808c670 tx=0x7ff92002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e827ce0f79f8c93 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:52.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.081+0000 7ff9267fc640 1 -- 172.21.15.44:0/2038757981 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92002ca70 con 0x7ff9280cec60 2024-04-02T13:26:52.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.081+0000 7ff9267fc640 1 -- 172.21.15.44:0/2038757981 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff92002cc10 con 0x7ff9280cec60 2024-04-02T13:26:52.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.081+0000 7ff9267fc640 1 -- 172.21.15.44:0/2038757981 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200343f0 con 0x7ff9280cec60 2024-04-02T13:26:52.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.082+0000 7ff92dad6640 1 -- 172.21.15.44:0/2038757981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9280cec60 msgr2=0x7ff9280d1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.082+0000 7ff92dad6640 1 --2- 172.21.15.44:0/2038757981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9280cec60 0x7ff9280d1010 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7ff92808c670 tx=0x7ff92002bed0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.082+0000 7ff92dad6640 1 -- 172.21.15.44:0/2038757981 shutdown_connections 2024-04-02T13:26:52.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.082+0000 7ff92dad6640 1 --2- 172.21.15.44:0/2038757981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9280cec60 0x7ff9280d1010 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.082+0000 7ff92dad6640 1 --2- 172.21.15.44:0/2038757981 >> v2:172.21.15.69:3300/0 conn(0x7ff92815e640 0x7ff9280d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.082+0000 7ff92dad6640 1 --2- 172.21.15.44:0/2038757981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff92815e260 0x7ff9280d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.082+0000 7ff92dad6640 1 -- 172.21.15.44:0/2038757981 >> 172.21.15.44:0/2038757981 conn(0x7ff92809d3c0 msgr2=0x7ff9280d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:52.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.082+0000 7ff92dad6640 1 -- 172.21.15.44:0/2038757981 shutdown_connections 2024-04-02T13:26:52.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.082+0000 7ff92dad6640 1 -- 172.21.15.44:0/2038757981 wait complete. 2024-04-02T13:26:52.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.083+0000 7ff92dad6640 1 Processor -- start 2024-04-02T13:26:52.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.083+0000 7ff92dad6640 1 -- start start 2024-04-02T13:26:52.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff92dad6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9280cec60 0x7ff92815ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff92dad6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92815e260 0x7ff9281f7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff92dad6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff92815e640 0x7ff9280ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff92dad6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff9280cfca0 con 0x7ff92815e640 2024-04-02T13:26:52.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff92dad6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff9280cf9c0 con 0x7ff92815e260 2024-04-02T13:26:52.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff92dad6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff9280d2ea0 con 0x7ff9280cec60 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff927fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff92815e640 0x7ff9280ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff927fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff92815e640 0x7ff9280ca800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49450/0 (socket says 172.21.15.44:49450) 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff927fff640 1 -- 172.21.15.44:0/3387802553 learned_addr learned my addr 172.21.15.44:0/3387802553 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff9277fe640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9280cec60 0x7ff92815ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff926ffd640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92815e260 0x7ff9281f7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff927fff640 1 -- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9280cec60 msgr2=0x7ff92815ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff927fff640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9280cec60 0x7ff92815ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff927fff640 1 -- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92815e260 msgr2=0x7ff9281f7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff927fff640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92815e260 0x7ff9281f7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.084+0000 7ff927fff640 1 -- 172.21.15.44:0/3387802553 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9281f89d0 con 0x7ff92815e640 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.085+0000 7ff927fff640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff92815e640 0x7ff9280ca800 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7ff920009f40 tx=0x7ff920004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.085+0000 7ff90ffff640 1 -- 172.21.15.44:0/3387802553 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff920043020 con 0x7ff92815e640 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.085+0000 7ff90ffff640 1 -- 172.21.15.44:0/3387802553 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff92003e450 con 0x7ff92815e640 2024-04-02T13:26:52.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.085+0000 7ff92dad6640 1 -- 172.21.15.44:0/3387802553 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9281f8410 con 0x7ff92815e640 2024-04-02T13:26:52.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.085+0000 7ff90ffff640 1 -- 172.21.15.44:0/3387802553 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff920009510 con 0x7ff92815e640 2024-04-02T13:26:52.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.085+0000 7ff92dad6640 1 -- 172.21.15.44:0/3387802553 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9280b7e60 con 0x7ff92815e640 2024-04-02T13:26:52.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.086+0000 7ff92dad6640 1 -- 172.21.15.44:0/3387802553 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9280684f0 con 0x7ff92815e640 2024-04-02T13:26:52.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.086+0000 7ff90ffff640 1 -- 172.21.15.44:0/3387802553 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff920046020 con 0x7ff92815e640 2024-04-02T13:26:52.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.086+0000 7ff90ffff640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff91406c8f0 0x7ff91406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.087+0000 7ff90ffff640 1 -- 172.21.15.44:0/3387802553 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7ff9200ae460 con 0x7ff92815e640 2024-04-02T13:26:52.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.087+0000 7ff9277fe640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff91406c8f0 0x7ff91406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.087+0000 7ff9277fe640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff91406c8f0 0x7ff91406edb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff918009ac0 tx=0x7ff918009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:52.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.091+0000 7ff90ffff640 1 -- 172.21.15.44:0/3387802553 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff92004d020 con 0x7ff92815e640 2024-04-02T13:26:52.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.275+0000 7ff92dad6640 1 -- 172.21.15.44:0/3387802553 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff928068c00 con 0x7ff92815e640 2024-04-02T13:26:52.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.275+0000 7ff90ffff640 1 -- 172.21.15.44:0/3387802553 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7ff92007c140 con 0x7ff92815e640 2024-04-02T13:26:52.277 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:26:52.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.277+0000 7ff92dad6640 1 -- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff91406c8f0 msgr2=0x7ff91406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.277+0000 7ff92dad6640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff91406c8f0 0x7ff91406edb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff918009ac0 tx=0x7ff918009190 comp rx=0 tx=0).stop 2024-04-02T13:26:52.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.277+0000 7ff92dad6640 1 -- 172.21.15.44:0/3387802553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff92815e640 msgr2=0x7ff9280ca800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.277+0000 7ff92dad6640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff92815e640 0x7ff9280ca800 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7ff920009f40 tx=0x7ff920004a40 comp rx=0 tx=0).stop 2024-04-02T13:26:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.277+0000 7ff92dad6640 1 -- 172.21.15.44:0/3387802553 shutdown_connections 2024-04-02T13:26:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.277+0000 7ff92dad6640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff91406c8f0 0x7ff91406edb0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.277+0000 7ff92dad6640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff92815e640 0x7ff9280ca800 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.277+0000 7ff92dad6640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92815e260 0x7ff9281f7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.277+0000 7ff92dad6640 1 --2- 172.21.15.44:0/3387802553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9280cec60 0x7ff92815ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.277+0000 7ff92dad6640 1 -- 172.21.15.44:0/3387802553 >> 172.21.15.44:0/3387802553 conn(0x7ff92809d3c0 msgr2=0x7ff92806c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.277+0000 7ff92dad6640 1 -- 172.21.15.44:0/3387802553 shutdown_connections 2024-04-02T13:26:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.277+0000 7ff92dad6640 1 -- 172.21.15.44:0/3387802553 wait complete. 2024-04-02T13:26:52.290 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:26:52.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.392+0000 7f799c455640 1 Processor -- start 2024-04-02T13:26:52.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.392+0000 7f799c455640 1 -- start start 2024-04-02T13:26:52.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f799c455640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f79941565d0 0x7f79940d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f799c455640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f79940d05e0 0x7f79940ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f799c455640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7994160b10 con 0x7f79940cf400 2024-04-02T13:26:52.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f799c455640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f79940cfad0 con 0x7f79941565d0 2024-04-02T13:26:52.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f799c455640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f79940cf870 con 0x7f79940d05e0 2024-04-02T13:26:52.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f799a1f1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f79941565d0 0x7f79940d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f799a9f2640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f79940cf400 0x7f79940cc660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42982/0 (socket says 172.21.15.44:42982) 2024-04-02T13:26:52.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f799a9f2640 1 -- 172.21.15.44:0/4204728971 learned_addr learned my addr 172.21.15.44:0/4204728971 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:52.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f79999f0640 1 --2- 172.21.15.44:0/4204728971 >> v2:172.21.15.69:3300/0 conn(0x7f79940d05e0 0x7f79940ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f799a1f1640 1 -- 172.21.15.44:0/4204728971 >> v2:172.21.15.69:3300/0 conn(0x7f79940d05e0 msgr2=0x7f79940ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:52.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f799a1f1640 1 --2- 172.21.15.44:0/4204728971 >> v2:172.21.15.69:3300/0 conn(0x7f79940d05e0 0x7f79940ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f799a1f1640 1 -- 172.21.15.44:0/4204728971 >> v1:172.21.15.44:6789/0 conn(0x7f79940cf400 legacy=0x7f79940cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f79991ef640 1 -- 172.21.15.44:0/4204728971 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1010241736 0 0) 0x7f7994160b10 con 0x7f79940cf400 2024-04-02T13:26:52.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f799a1f1640 1 -- 172.21.15.44:0/4204728971 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799415fd50 con 0x7f79941565d0 2024-04-02T13:26:52.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.393+0000 7f79999f0640 1 --2- 172.21.15.44:0/4204728971 >> v2:172.21.15.69:3300/0 conn(0x7f79940d05e0 0x7f79940ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:52.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.394+0000 7f799a1f1640 1 --2- 172.21.15.44:0/4204728971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79941565d0 0x7f79940d00a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f79940b2a30 tx=0x7f798c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee18b80cf957a11f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:52.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.394+0000 7f79991ef640 1 -- 172.21.15.44:0/4204728971 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c0315a0 con 0x7f79941565d0 2024-04-02T13:26:52.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.394+0000 7f79991ef640 1 -- 172.21.15.44:0/4204728971 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f798c037070 con 0x7f79941565d0 2024-04-02T13:26:52.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.394+0000 7f79991ef640 1 -- 172.21.15.44:0/4204728971 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c031960 con 0x7f79941565d0 2024-04-02T13:26:52.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.395+0000 7f799c455640 1 -- 172.21.15.44:0/4204728971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79941565d0 msgr2=0x7f79940d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.395+0000 7f799c455640 1 --2- 172.21.15.44:0/4204728971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79941565d0 0x7f79940d00a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f79940b2a30 tx=0x7f798c02b840 comp rx=0 tx=0).stop 2024-04-02T13:26:52.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.395+0000 7f799c455640 1 -- 172.21.15.44:0/4204728971 shutdown_connections 2024-04-02T13:26:52.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.395+0000 7f799c455640 1 --2- 172.21.15.44:0/4204728971 >> v2:172.21.15.69:3300/0 conn(0x7f79940d05e0 0x7f79940ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.395+0000 7f799c455640 1 --2- 172.21.15.44:0/4204728971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79941565d0 0x7f79940d00a0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.395+0000 7f799c455640 1 -- 172.21.15.44:0/4204728971 >> 172.21.15.44:0/4204728971 conn(0x7f799406b920 msgr2=0x7f799406bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:52.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.395+0000 7f799c455640 1 -- 172.21.15.44:0/4204728971 shutdown_connections 2024-04-02T13:26:52.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.395+0000 7f799c455640 1 -- 172.21.15.44:0/4204728971 wait complete. 2024-04-02T13:26:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.396+0000 7f799c455640 1 Processor -- start 2024-04-02T13:26:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.396+0000 7f799c455640 1 -- start start 2024-04-02T13:26:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799c455640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79940cf400 0x7f799414f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799c455640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79940d05e0 0x7f799409d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799c455640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79941565d0 0x7f7994153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799c455640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f79941605f0 con 0x7f79940cf400 2024-04-02T13:26:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799c455640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7994160310 con 0x7f79940d05e0 2024-04-02T13:26:52.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799c455640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f79941569b0 con 0x7f79941565d0 2024-04-02T13:26:52.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799a1f1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79940cf400 0x7f799414f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799a1f1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79940cf400 0x7f799414f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49476/0 (socket says 172.21.15.44:49476) 2024-04-02T13:26:52.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799a1f1640 1 -- 172.21.15.44:0/3882611056 learned_addr learned my addr 172.21.15.44:0/3882611056 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:52.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799a9f2640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79941565d0 0x7f7994153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f79999f0640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79940d05e0 0x7f799409d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799a1f1640 1 -- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79941565d0 msgr2=0x7f7994153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799a1f1640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79941565d0 0x7f7994153c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799a1f1640 1 -- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79940d05e0 msgr2=0x7f799409d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799a1f1640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79940d05e0 0x7f799409d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799a1f1640 1 -- 172.21.15.44:0/3882611056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7994154b60 con 0x7f79940cf400 2024-04-02T13:26:52.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.397+0000 7f799a9f2640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79941565d0 0x7f7994153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:52.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.398+0000 7f799a1f1640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79940cf400 0x7f799414f720 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f798c034040 tx=0x7f798c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:52.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.398+0000 7f798affd640 1 -- 172.21.15.44:0/3882611056 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c004230 con 0x7f79940cf400 2024-04-02T13:26:52.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.398+0000 7f798affd640 1 -- 172.21.15.44:0/3882611056 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f798c0043d0 con 0x7f79940cf400 2024-04-02T13:26:52.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.398+0000 7f799c455640 1 -- 172.21.15.44:0/3882611056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7994154e40 con 0x7f79940cf400 2024-04-02T13:26:52.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.398+0000 7f798affd640 1 -- 172.21.15.44:0/3882611056 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c003850 con 0x7f79940cf400 2024-04-02T13:26:52.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.398+0000 7f799c455640 1 -- 172.21.15.44:0/3882611056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7994153710 con 0x7f79940cf400 2024-04-02T13:26:52.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.399+0000 7f799c455640 1 -- 172.21.15.44:0/3882611056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79940684f0 con 0x7f79940cf400 2024-04-02T13:26:52.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.399+0000 7f798affd640 1 -- 172.21.15.44:0/3882611056 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f798c0039f0 con 0x7f79940cf400 2024-04-02T13:26:52.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.400+0000 7f798affd640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f797806c8f0 0x7f797806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.400+0000 7f798affd640 1 -- 172.21.15.44:0/3882611056 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7f798c02f030 con 0x7f79940cf400 2024-04-02T13:26:52.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.400+0000 7f79999f0640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f797806c8f0 0x7f797806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.400+0000 7f79999f0640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f797806c8f0 0x7f797806edb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f799406f080 tx=0x7f7984008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:52.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.404+0000 7f798affd640 1 -- 172.21.15.44:0/3882611056 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f798c041070 con 0x7f79940cf400 2024-04-02T13:26:52.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.588+0000 7f799c455640 1 -- 172.21.15.44:0/3882611056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f79940ce540 con 0x7f79940cf400 2024-04-02T13:26:52.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.589+0000 7f798affd640 1 -- 172.21.15.44:0/3882611056 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f798c07c070 con 0x7f79940cf400 2024-04-02T13:26:52.591 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:26:52.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.590+0000 7f799c455640 1 -- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f797806c8f0 msgr2=0x7f797806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.591+0000 7f799c455640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f797806c8f0 0x7f797806edb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f799406f080 tx=0x7f7984008040 comp rx=0 tx=0).stop 2024-04-02T13:26:52.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.591+0000 7f799c455640 1 -- 172.21.15.44:0/3882611056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79940cf400 msgr2=0x7f799414f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.591+0000 7f799c455640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79940cf400 0x7f799414f720 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f798c034040 tx=0x7f798c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.591+0000 7f799c455640 1 -- 172.21.15.44:0/3882611056 shutdown_connections 2024-04-02T13:26:52.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.591+0000 7f799c455640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f797806c8f0 0x7f797806edb0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.591+0000 7f799c455640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79941565d0 0x7f7994153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.591+0000 7f799c455640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79940d05e0 0x7f799409d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.591+0000 7f799c455640 1 --2- 172.21.15.44:0/3882611056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79940cf400 0x7f799414f720 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.591+0000 7f799c455640 1 -- 172.21.15.44:0/3882611056 >> 172.21.15.44:0/3882611056 conn(0x7f799406b920 msgr2=0x7f799415a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:52.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.591+0000 7f799c455640 1 -- 172.21.15.44:0/3882611056 shutdown_connections 2024-04-02T13:26:52.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.591+0000 7f799c455640 1 -- 172.21.15.44:0/3882611056 wait complete. 2024-04-02T13:26:52.604 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:26:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.704+0000 7f6da31d4640 1 Processor -- start 2024-04-02T13:26:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.704+0000 7f6da31d4640 1 -- start start 2024-04-02T13:26:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.705+0000 7f6da31d4640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6d9c1617e0 0x7f6d9c0ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.705+0000 7f6da31d4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6d9c0cffd0 0x7f6d9c1629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.705+0000 7f6da31d4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d9c0cef30 con 0x7f6d9c0ccc90 2024-04-02T13:26:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.705+0000 7f6da31d4640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f6d9c162f00 con 0x7f6d9c1617e0 2024-04-02T13:26:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.705+0000 7f6da31d4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6d9c0aea90 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.705+0000 7f6da0f70640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6d9c1617e0 0x7f6d9c0ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.705+0000 7f6da1771640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6d9c0ccc90 0x7f6d9c0cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42984/0 (socket says 172.21.15.44:42984) 2024-04-02T13:26:52.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.705+0000 7f6da1771640 1 -- 172.21.15.44:0/1301086785 learned_addr learned my addr 172.21.15.44:0/1301086785 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:52.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.706+0000 7f6d9bfff640 1 --2- 172.21.15.44:0/1301086785 >> v2:172.21.15.69:3300/0 conn(0x7f6d9c0cffd0 0x7f6d9c1629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.706+0000 7f6da0f70640 1 -- 172.21.15.44:0/1301086785 >> v2:172.21.15.69:3300/0 conn(0x7f6d9c0cffd0 msgr2=0x7f6d9c1629c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:52.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.706+0000 7f6da0f70640 1 --2- 172.21.15.44:0/1301086785 >> v2:172.21.15.69:3300/0 conn(0x7f6d9c0cffd0 0x7f6d9c1629c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.706+0000 7f6da0f70640 1 -- 172.21.15.44:0/1301086785 >> v1:172.21.15.44:6789/0 conn(0x7f6d9c0ccc90 legacy=0x7f6d9c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.706+0000 7f6d9b7fe640 1 -- 172.21.15.44:0/1301086785 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2359448165 0 0) 0x7f6d9c0cef30 con 0x7f6d9c0ccc90 2024-04-02T13:26:52.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.706+0000 7f6da0f70640 1 -- 172.21.15.44:0/1301086785 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d9c0cf210 con 0x7f6d9c1617e0 2024-04-02T13:26:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.706+0000 7f6da0f70640 1 --2- 172.21.15.44:0/1301086785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9c1617e0 0x7f6d9c0ce8c0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c0cee00 tx=0x7f6d9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f75b99656000aed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.707+0000 7f6d9b7fe640 1 -- 172.21.15.44:0/1301086785 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d90035020 con 0x7f6d9c1617e0 2024-04-02T13:26:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.707+0000 7f6d9b7fe640 1 -- 172.21.15.44:0/1301086785 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d90031830 con 0x7f6d9c1617e0 2024-04-02T13:26:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.707+0000 7f6d9b7fe640 1 -- 172.21.15.44:0/1301086785 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d90031c00 con 0x7f6d9c1617e0 2024-04-02T13:26:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.707+0000 7f6da31d4640 1 -- 172.21.15.44:0/1301086785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9c1617e0 msgr2=0x7f6d9c0ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.707+0000 7f6da31d4640 1 --2- 172.21.15.44:0/1301086785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9c1617e0 0x7f6d9c0ce8c0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c0cee00 tx=0x7f6d9002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.707+0000 7f6da31d4640 1 -- 172.21.15.44:0/1301086785 shutdown_connections 2024-04-02T13:26:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.707+0000 7f6da31d4640 1 --2- 172.21.15.44:0/1301086785 >> v2:172.21.15.69:3300/0 conn(0x7f6d9c0cffd0 0x7f6d9c1629c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.707+0000 7f6da31d4640 1 --2- 172.21.15.44:0/1301086785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9c1617e0 0x7f6d9c0ce8c0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.707+0000 7f6da31d4640 1 -- 172.21.15.44:0/1301086785 >> 172.21.15.44:0/1301086785 conn(0x7f6d9c0b20e0 msgr2=0x7f6d9c0b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:52.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.707+0000 7f6da31d4640 1 -- 172.21.15.44:0/1301086785 shutdown_connections 2024-04-02T13:26:52.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.707+0000 7f6da31d4640 1 -- 172.21.15.44:0/1301086785 wait complete. 2024-04-02T13:26:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.708+0000 7f6da31d4640 1 Processor -- start 2024-04-02T13:26:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.708+0000 7f6da31d4640 1 -- start start 2024-04-02T13:26:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.709+0000 7f6da31d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d9c0ccc90 0x7f6d9c1f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.709+0000 7f6da31d4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9c0cffd0 0x7f6d9c0d0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.709+0000 7f6da31d4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d9c1617e0 0x7f6d9c0d09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.709+0000 7f6da31d4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c0cfab0 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.709+0000 7f6da31d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c0cf7d0 con 0x7f6d9c0ccc90 2024-04-02T13:26:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.709+0000 7f6da31d4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6d9c161bc0 con 0x7f6d9c1617e0 2024-04-02T13:26:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.709+0000 7f6d9bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9c0cffd0 0x7f6d9c0d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.709+0000 7f6d9bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9c0cffd0 0x7f6d9c0d0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49506/0 (socket says 172.21.15.44:49506) 2024-04-02T13:26:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.709+0000 7f6d9bfff640 1 -- 172.21.15.44:0/4020979054 learned_addr learned my addr 172.21.15.44:0/4020979054 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6da0f70640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d9c0ccc90 0x7f6d9c1f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6da1771640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d9c1617e0 0x7f6d9c0d09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6d9bfff640 1 -- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d9c1617e0 msgr2=0x7f6d9c0d09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6d9bfff640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d9c1617e0 0x7f6d9c0d09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6d9bfff640 1 -- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d9c0ccc90 msgr2=0x7f6d9c1f38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6d9bfff640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d9c0ccc90 0x7f6d9c1f38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6d9bfff640 1 -- 172.21.15.44:0/4020979054 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d9c0b2ff0 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6da0f70640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d9c0ccc90 0x7f6d9c1f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6da1771640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d9c1617e0 0x7f6d9c0d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6d9bfff640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9c0cffd0 0x7f6d9c0d0480 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f6d8800d800 tx=0x7f6d8800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6d997fa640 1 -- 172.21.15.44:0/4020979054 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8800b6c0 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6d997fa640 1 -- 172.21.15.44:0/4020979054 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d88019070 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6da31d4640 1 -- 172.21.15.44:0/4020979054 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d9c0b32d0 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6d997fa640 1 -- 172.21.15.44:0/4020979054 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8800baa0 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.710+0000 7f6da31d4640 1 -- 172.21.15.44:0/4020979054 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d9c0b3890 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.711+0000 7f6da31d4640 1 -- 172.21.15.44:0/4020979054 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d9c0b1e90 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.711+0000 7f6d997fa640 1 -- 172.21.15.44:0/4020979054 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6d8800bc40 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.712+0000 7f6d997fa640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6d8006c920 0x7f6d8006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.712+0000 7f6d997fa640 1 -- 172.21.15.44:0/4020979054 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7f6d8808c050 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.712+0000 7f6da0f70640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6d8006c920 0x7f6d8006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.713+0000 7f6da0f70640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6d8006c920 0x7f6d8006ede0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6d90036000 tx=0x7f6d90008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:52.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.716+0000 7f6d997fa640 1 -- 172.21.15.44:0/4020979054 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6d9c0b1e90 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.860 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:26:52.863 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:26:52.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.913+0000 7f6da31d4640 1 -- 172.21.15.44:0/4020979054 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6d9c0684f0 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.914+0000 7f6d997fa640 1 -- 172.21.15.44:0/4020979054 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v91) ==== 80+0+0 (secure 0 0 0) 0x7f6d8805a380 con 0x7f6d9c0cffd0 2024-04-02T13:26:52.916 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:26:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.916+0000 7f6da31d4640 1 -- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6d8006c920 msgr2=0x7f6d8006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.916+0000 7f6da31d4640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6d8006c920 0x7f6d8006ede0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6d90036000 tx=0x7f6d90008030 comp rx=0 tx=0).stop 2024-04-02T13:26:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.916+0000 7f6da31d4640 1 -- 172.21.15.44:0/4020979054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9c0cffd0 msgr2=0x7f6d9c0d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.916+0000 7f6da31d4640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9c0cffd0 0x7f6d9c0d0480 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f6d8800d800 tx=0x7f6d8800dcc0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.916+0000 7f6da31d4640 1 -- 172.21.15.44:0/4020979054 shutdown_connections 2024-04-02T13:26:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.916+0000 7f6da31d4640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6d8006c920 0x7f6d8006ede0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.916+0000 7f6da31d4640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d9c1617e0 0x7f6d9c0d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.916+0000 7f6da31d4640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d9c0cffd0 0x7f6d9c0d0480 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.916+0000 7f6da31d4640 1 --2- 172.21.15.44:0/4020979054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d9c0ccc90 0x7f6d9c1f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.916+0000 7f6da31d4640 1 -- 172.21.15.44:0/4020979054 >> 172.21.15.44:0/4020979054 conn(0x7f6d9c0b20e0 msgr2=0x7f6d9c0cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:52.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.916+0000 7f6da31d4640 1 -- 172.21.15.44:0/4020979054 shutdown_connections 2024-04-02T13:26:52.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:52.916+0000 7f6da31d4640 1 -- 172.21.15.44:0/4020979054 wait complete. 2024-04-02T13:26:52.929 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:26:52.929 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:26:52.985 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:26:53.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.123+0000 7f1f912f2640 1 Processor -- start 2024-04-02T13:26:53.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.124+0000 7f1f912f2640 1 -- start start 2024-04-02T13:26:53.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.124+0000 7f1f912f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f8c159db0 0x7f1f8c15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.124+0000 7f1f912f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1f8c071e50 0x7f1f8c0cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.124+0000 7f1f912f2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c0726c0 con 0x7f1f8c0ccc20 2024-04-02T13:26:53.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.124+0000 7f1f912f2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f1f8c073240 con 0x7f1f8c071e50 2024-04-02T13:26:53.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.124+0000 7f1f912f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1f8c0cd050 con 0x7f1f8c159db0 2024-04-02T13:26:53.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.124+0000 7f1f8a7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1f8c071e50 0x7f1f8c0cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.124+0000 7f1f8b7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f1f8c0ccc20 0x7f1f8c06d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:43000/0 (socket says 172.21.15.44:43000) 2024-04-02T13:26:53.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.124+0000 7f1f8b7fe640 1 -- 172.21.15.44:0/835577234 learned_addr learned my addr 172.21.15.44:0/835577234 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:53.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.125+0000 7f1f8a7fc640 1 -- 172.21.15.44:0/835577234 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f8c159db0 msgr2=0x7f1f8c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:53.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.125+0000 7f1f8a7fc640 1 --2- 172.21.15.44:0/835577234 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f8c159db0 0x7f1f8c15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.125+0000 7f1f8a7fc640 1 -- 172.21.15.44:0/835577234 >> v1:172.21.15.44:6789/0 conn(0x7f1f8c0ccc20 legacy=0x7f1f8c06d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.125+0000 7f1f8a7fc640 1 -- 172.21.15.44:0/835577234 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8c0729a0 con 0x7f1f8c071e50 2024-04-02T13:26:53.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.125+0000 7f1f8a7fc640 1 --2- 172.21.15.44:0/835577234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f8c071e50 0x7f1f8c0cc6e0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f1f800097a0 tx=0x7f1f8002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83c9c3ca57ad8bf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:53.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.126+0000 7f1f89ffb640 1 -- 172.21.15.44:0/835577234 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f800315a0 con 0x7f1f8c071e50 2024-04-02T13:26:53.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.126+0000 7f1f89ffb640 1 -- 172.21.15.44:0/835577234 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f80037070 con 0x7f1f8c071e50 2024-04-02T13:26:53.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.126+0000 7f1f89ffb640 1 -- 172.21.15.44:0/835577234 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f80031960 con 0x7f1f8c071e50 2024-04-02T13:26:53.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.126+0000 7f1f912f2640 1 -- 172.21.15.44:0/835577234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f8c071e50 msgr2=0x7f1f8c0cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.126+0000 7f1f912f2640 1 --2- 172.21.15.44:0/835577234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f8c071e50 0x7f1f8c0cc6e0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f1f800097a0 tx=0x7f1f8002b840 comp rx=0 tx=0).stop 2024-04-02T13:26:53.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.126+0000 7f1f912f2640 1 -- 172.21.15.44:0/835577234 shutdown_connections 2024-04-02T13:26:53.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.126+0000 7f1f912f2640 1 --2- 172.21.15.44:0/835577234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f8c071e50 0x7f1f8c0cc6e0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.126+0000 7f1f912f2640 1 --2- 172.21.15.44:0/835577234 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f8c159db0 0x7f1f8c15c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.126+0000 7f1f912f2640 1 -- 172.21.15.44:0/835577234 >> 172.21.15.44:0/835577234 conn(0x7f1f8c0cfcd0 msgr2=0x7f1f8c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:53.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.126+0000 7f1f912f2640 1 -- 172.21.15.44:0/835577234 shutdown_connections 2024-04-02T13:26:53.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.126+0000 7f1f912f2640 1 -- 172.21.15.44:0/835577234 wait complete. 2024-04-02T13:26:53.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.127+0000 7f1f912f2640 1 Processor -- start 2024-04-02T13:26:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.128+0000 7f1f912f2640 1 -- start start 2024-04-02T13:26:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.128+0000 7f1f912f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f8c071e50 0x7f1f8c0b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.128+0000 7f1f912f2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f8c0ccc20 0x7f1f8c0ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.128+0000 7f1f912f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f8c159db0 0x7f1f8c14b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.128+0000 7f1f912f2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c0722d0 con 0x7f1f8c071e50 2024-04-02T13:26:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.128+0000 7f1f912f2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c072f60 con 0x7f1f8c0ccc20 2024-04-02T13:26:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.128+0000 7f1f912f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1f8c072450 con 0x7f1f8c159db0 2024-04-02T13:26:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.128+0000 7f1f8affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f8c071e50 0x7f1f8c0b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.128+0000 7f1f8affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f8c071e50 0x7f1f8c0b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49526/0 (socket says 172.21.15.44:49526) 2024-04-02T13:26:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.128+0000 7f1f8affd640 1 -- 172.21.15.44:0/1358213072 learned_addr learned my addr 172.21.15.44:0/1358213072 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f8a7fc640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f8c0ccc20 0x7f1f8c0ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f8affd640 1 -- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f8c159db0 msgr2=0x7f1f8c14b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f8b7fe640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f8c159db0 0x7f1f8c14b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f8affd640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f8c159db0 0x7f1f8c14b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f8affd640 1 -- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f8c0ccc20 msgr2=0x7f1f8c0ae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f8affd640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f8c0ccc20 0x7f1f8c0ae010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f8affd640 1 -- 172.21.15.44:0/1358213072 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8c0c93d0 con 0x7f1f8c071e50 2024-04-02T13:26:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f8a7fc640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f8c0ccc20 0x7f1f8c0ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f8b7fe640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f8c159db0 0x7f1f8c14b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f8affd640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f8c071e50 0x7f1f8c0b3810 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00b410 tx=0x7f1f7c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f737fe640 1 -- 172.21.15.44:0/1358213072 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c015020 con 0x7f1f8c071e50 2024-04-02T13:26:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f737fe640 1 -- 172.21.15.44:0/1358213072 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7c011850 con 0x7f1f8c071e50 2024-04-02T13:26:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f912f2640 1 -- 172.21.15.44:0/1358213072 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f8c0c96b0 con 0x7f1f8c071e50 2024-04-02T13:26:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f737fe640 1 -- 172.21.15.44:0/1358213072 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c011c40 con 0x7f1f8c071e50 2024-04-02T13:26:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.129+0000 7f1f912f2640 1 -- 172.21.15.44:0/1358213072 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f8c0c9c70 con 0x7f1f8c071e50 2024-04-02T13:26:53.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.130+0000 7f1f912f2640 1 -- 172.21.15.44:0/1358213072 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f8c074f20 con 0x7f1f8c071e50 2024-04-02T13:26:53.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.130+0000 7f1f737fe640 1 -- 172.21.15.44:0/1358213072 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1f7c018020 con 0x7f1f8c071e50 2024-04-02T13:26:53.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.131+0000 7f1f737fe640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1f7806c920 0x7f1f7806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.131+0000 7f1f737fe640 1 -- 172.21.15.44:0/1358213072 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7f1f7c08bcd0 con 0x7f1f8c071e50 2024-04-02T13:26:53.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.131+0000 7f1f8a7fc640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1f7806c920 0x7f1f7806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.132+0000 7f1f8a7fc640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1f7806c920 0x7f1f7806ede0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1f80036000 tx=0x7f1f80008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:53.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.135+0000 7f1f737fe640 1 -- 172.21.15.44:0/1358213072 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1f7c01f020 con 0x7f1f8c071e50 2024-04-02T13:26:53.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.319+0000 7f1f912f2640 1 -- 172.21.15.44:0/1358213072 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f8c15f330 con 0x7f1f8c071e50 2024-04-02T13:26:53.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.320+0000 7f1f737fe640 1 -- 172.21.15.44:0/1358213072 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+16837 (secure 0 0 0) 0x7f1f7c0172c0 con 0x7f1f8c071e50 2024-04-02T13:26:53.323 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:53.323 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:26:44.676976+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:26:41.300970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-02T13:26:41.979586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:26:53.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.322+0000 7f1f912f2640 1 -- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1f7806c920 msgr2=0x7f1f7806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.322+0000 7f1f912f2640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1f7806c920 0x7f1f7806ede0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1f80036000 tx=0x7f1f80008030 comp rx=0 tx=0).stop 2024-04-02T13:26:53.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.322+0000 7f1f912f2640 1 -- 172.21.15.44:0/1358213072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f8c071e50 msgr2=0x7f1f8c0b3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.322+0000 7f1f912f2640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f8c071e50 0x7f1f8c0b3810 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00b410 tx=0x7f1f7c00b8d0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.322+0000 7f1f912f2640 1 -- 172.21.15.44:0/1358213072 shutdown_connections 2024-04-02T13:26:53.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.322+0000 7f1f912f2640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1f7806c920 0x7f1f7806ede0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.322+0000 7f1f912f2640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f8c159db0 0x7f1f8c14b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.322+0000 7f1f912f2640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f8c0ccc20 0x7f1f8c0ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.322+0000 7f1f912f2640 1 --2- 172.21.15.44:0/1358213072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f8c071e50 0x7f1f8c0b3810 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.322+0000 7f1f912f2640 1 -- 172.21.15.44:0/1358213072 >> 172.21.15.44:0/1358213072 conn(0x7f1f8c0cfcd0 msgr2=0x7f1f8c0cdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:53.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.322+0000 7f1f912f2640 1 -- 172.21.15.44:0/1358213072 shutdown_connections 2024-04-02T13:26:53.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.322+0000 7f1f912f2640 1 -- 172.21.15.44:0/1358213072 wait complete. 2024-04-02T13:26:53.335 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:26:41.300970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:26:41.979586+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:26:53.336 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:26:53.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.437+0000 7f41360a0640 1 Processor -- start 2024-04-02T13:26:53.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.437+0000 7f41360a0640 1 -- start start 2024-04-02T13:26:53.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.438+0000 7f41360a0640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f413015e1d0 0x7f41300d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.438+0000 7f41360a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41300cec60 0x7f41300d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.438+0000 7f41360a0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f413015e5b0 0x7f41300d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.438+0000 7f41360a0640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f41300cfe00 con 0x7f413015e1d0 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.438+0000 7f41360a0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f41300d1640 con 0x7f413015e5b0 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.438+0000 7f41360a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f41300cf1f0 con 0x7f41300cec60 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.438+0000 7f412f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f413015e1d0 0x7f41300d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.438+0000 7f412f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f413015e1d0 0x7f41300d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49538/0 (socket says 172.21.15.44:49538) 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.438+0000 7f412f7fe640 1 -- 172.21.15.44:0/1826292419 learned_addr learned my addr 172.21.15.44:0/1826292419 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.439+0000 7f412ffff640 1 --2- 172.21.15.44:0/1826292419 >> v2:172.21.15.69:3300/0 conn(0x7f413015e5b0 0x7f41300d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.439+0000 7f412f7fe640 1 -- 172.21.15.44:0/1826292419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41300cec60 msgr2=0x7f41300d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.439+0000 7f412effd640 1 --2- 172.21.15.44:0/1826292419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41300cec60 0x7f41300d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.439+0000 7f412f7fe640 1 --2- 172.21.15.44:0/1826292419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41300cec60 0x7f41300d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.439+0000 7f412f7fe640 1 -- 172.21.15.44:0/1826292419 >> v2:172.21.15.69:3300/0 conn(0x7f413015e5b0 msgr2=0x7f41300d2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.439+0000 7f412f7fe640 1 --2- 172.21.15.44:0/1826292419 >> v2:172.21.15.69:3300/0 conn(0x7f413015e5b0 0x7f41300d2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.439+0000 7f412f7fe640 1 -- 172.21.15.44:0/1826292419 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41300cf4d0 con 0x7f413015e1d0 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.439+0000 7f412ffff640 1 --2- 172.21.15.44:0/1826292419 >> v2:172.21.15.69:3300/0 conn(0x7f413015e5b0 0x7f41300d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:53.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.439+0000 7f412f7fe640 1 --2- 172.21.15.44:0/1826292419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f413015e1d0 0x7f41300d1100 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f41300cb160 tx=0x7f412002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c9e0275c0f08e1f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:53.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.440+0000 7f412e7fc640 1 -- 172.21.15.44:0/1826292419 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412002e070 con 0x7f413015e1d0 2024-04-02T13:26:53.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.440+0000 7f412e7fc640 1 -- 172.21.15.44:0/1826292419 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4120037070 con 0x7f413015e1d0 2024-04-02T13:26:53.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.440+0000 7f412e7fc640 1 -- 172.21.15.44:0/1826292419 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4120031a00 con 0x7f413015e1d0 2024-04-02T13:26:53.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.440+0000 7f41360a0640 1 -- 172.21.15.44:0/1826292419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f413015e1d0 msgr2=0x7f41300d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.440+0000 7f41360a0640 1 --2- 172.21.15.44:0/1826292419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f413015e1d0 0x7f41300d1100 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f41300cb160 tx=0x7f412002b880 comp rx=0 tx=0).stop 2024-04-02T13:26:53.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.440+0000 7f41360a0640 1 -- 172.21.15.44:0/1826292419 shutdown_connections 2024-04-02T13:26:53.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.440+0000 7f41360a0640 1 --2- 172.21.15.44:0/1826292419 >> v2:172.21.15.69:3300/0 conn(0x7f413015e5b0 0x7f41300d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.440+0000 7f41360a0640 1 --2- 172.21.15.44:0/1826292419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41300cec60 0x7f41300d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.440+0000 7f41360a0640 1 --2- 172.21.15.44:0/1826292419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f413015e1d0 0x7f41300d1100 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.440+0000 7f41360a0640 1 -- 172.21.15.44:0/1826292419 >> 172.21.15.44:0/1826292419 conn(0x7f413009d3b0 msgr2=0x7f41300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:53.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.440+0000 7f41360a0640 1 -- 172.21.15.44:0/1826292419 shutdown_connections 2024-04-02T13:26:53.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.440+0000 7f41360a0640 1 -- 172.21.15.44:0/1826292419 wait complete. 2024-04-02T13:26:53.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.441+0000 7f41360a0640 1 Processor -- start 2024-04-02T13:26:53.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.441+0000 7f41360a0640 1 -- start start 2024-04-02T13:26:53.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.442+0000 7f41360a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41300cec60 0x7f41301f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.442+0000 7f41360a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f413015e1d0 0x7f41300b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.442+0000 7f41360a0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f413015e5b0 0x7f41300b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.442+0000 7f41360a0640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f41300b4ce0 con 0x7f413015e5b0 2024-04-02T13:26:53.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.442+0000 7f41360a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f41300cfa90 con 0x7f413015e1d0 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.442+0000 7f41360a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f41300b4e60 con 0x7f41300cec60 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.442+0000 7f412ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f413015e5b0 0x7f41300b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.442+0000 7f412ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f413015e5b0 0x7f41300b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49550/0 (socket says 172.21.15.44:49550) 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.442+0000 7f412ffff640 1 -- 172.21.15.44:0/744665286 learned_addr learned my addr 172.21.15.44:0/744665286 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.442+0000 7f412f7fe640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41300cec60 0x7f41301f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.442+0000 7f412effd640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f413015e1d0 0x7f41300b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.443+0000 7f412ffff640 1 -- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41300cec60 msgr2=0x7f41301f4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.443+0000 7f412ffff640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41300cec60 0x7f41301f4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.443+0000 7f412ffff640 1 -- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f413015e1d0 msgr2=0x7f41300b2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.443+0000 7f412ffff640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f413015e1d0 0x7f41300b2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.443+0000 7f412ffff640 1 -- 172.21.15.44:0/744665286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41301f50e0 con 0x7f413015e5b0 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.443+0000 7f412ffff640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f413015e5b0 0x7f41300b3290 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f412800cd50 tx=0x7f412800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.443+0000 7f4117fff640 1 -- 172.21.15.44:0/744665286 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412800f680 con 0x7f413015e5b0 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.443+0000 7f4117fff640 1 -- 172.21.15.44:0/744665286 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f412800f840 con 0x7f413015e5b0 2024-04-02T13:26:53.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.443+0000 7f4117fff640 1 -- 172.21.15.44:0/744665286 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412800fc30 con 0x7f413015e5b0 2024-04-02T13:26:53.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.443+0000 7f41360a0640 1 -- 172.21.15.44:0/744665286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41300b2190 con 0x7f413015e5b0 2024-04-02T13:26:53.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.443+0000 7f41360a0640 1 -- 172.21.15.44:0/744665286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41300b13d0 con 0x7f413015e5b0 2024-04-02T13:26:53.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.444+0000 7f41360a0640 1 -- 172.21.15.44:0/744665286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41300684f0 con 0x7f413015e5b0 2024-04-02T13:26:53.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.445+0000 7f4117fff640 1 -- 172.21.15.44:0/744665286 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f412800e3e0 con 0x7f413015e5b0 2024-04-02T13:26:53.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.445+0000 7f4117fff640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f411c06c8f0 0x7f411c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.445+0000 7f4117fff640 1 -- 172.21.15.44:0/744665286 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7f412808c870 con 0x7f413015e5b0 2024-04-02T13:26:53.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.445+0000 7f412f7fe640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f411c06c8f0 0x7f411c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.446+0000 7f412f7fe640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f411c06c8f0 0x7f411c06edb0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4120036000 tx=0x7f412003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:53.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.449+0000 7f4117fff640 1 -- 172.21.15.44:0/744665286 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f41301f50e0 con 0x7f413015e5b0 2024-04-02T13:26:53.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.635+0000 7f41360a0640 1 -- 172.21.15.44:0/744665286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f41300d47f0 con 0x7f413015e5b0 2024-04-02T13:26:53.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.635+0000 7f4117fff640 1 -- 172.21.15.44:0/744665286 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f4128014030 con 0x7f413015e5b0 2024-04-02T13:26:53.638 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:26:53.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.637+0000 7f41360a0640 1 -- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f411c06c8f0 msgr2=0x7f411c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.637+0000 7f41360a0640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f411c06c8f0 0x7f411c06edb0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4120036000 tx=0x7f412003d040 comp rx=0 tx=0).stop 2024-04-02T13:26:53.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.637+0000 7f41360a0640 1 -- 172.21.15.44:0/744665286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f413015e5b0 msgr2=0x7f41300b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.637+0000 7f41360a0640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f413015e5b0 0x7f41300b3290 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f412800cd50 tx=0x7f412800a650 comp rx=0 tx=0).stop 2024-04-02T13:26:53.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.637+0000 7f41360a0640 1 -- 172.21.15.44:0/744665286 shutdown_connections 2024-04-02T13:26:53.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.637+0000 7f41360a0640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f411c06c8f0 0x7f411c06edb0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.637+0000 7f41360a0640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f413015e5b0 0x7f41300b3290 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.637+0000 7f41360a0640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f413015e1d0 0x7f41300b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.637+0000 7f41360a0640 1 --2- 172.21.15.44:0/744665286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41300cec60 0x7f41301f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.637+0000 7f41360a0640 1 -- 172.21.15.44:0/744665286 >> 172.21.15.44:0/744665286 conn(0x7f413009d3b0 msgr2=0x7f413015d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:53.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.638+0000 7f41360a0640 1 -- 172.21.15.44:0/744665286 shutdown_connections 2024-04-02T13:26:53.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.638+0000 7f41360a0640 1 -- 172.21.15.44:0/744665286 wait complete. 2024-04-02T13:26:53.650 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:26:53.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.753+0000 7fc2580e4640 1 Processor -- start 2024-04-02T13:26:53.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.754+0000 7fc2580e4640 1 -- start start 2024-04-02T13:26:53.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.754+0000 7fc2580e4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc25015e1d0 0x7fc2500d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.754+0000 7fc2580e4640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc2500cec60 0x7fc2500d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.754+0000 7fc2580e4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2500d1640 con 0x7fc25015e5b0 2024-04-02T13:26:53.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.754+0000 7fc2580e4640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc2500cfe00 con 0x7fc2500cec60 2024-04-02T13:26:53.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.754+0000 7fc2580e4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc2500cf1f0 con 0x7fc25015e1d0 2024-04-02T13:26:53.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.755+0000 7fc25567f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc2500cec60 0x7fc2500d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.755+0000 7fc25567f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc2500cec60 0x7fc2500d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49554/0 (socket says 172.21.15.44:49554) 2024-04-02T13:26:53.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.755+0000 7fc25567f640 1 -- 172.21.15.44:0/3833033247 learned_addr learned my addr 172.21.15.44:0/3833033247 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:53.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.755+0000 7fc255e80640 1 --2- 172.21.15.44:0/3833033247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc25015e1d0 0x7fc2500d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.755+0000 7fc25567f640 1 -- 172.21.15.44:0/3833033247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc25015e1d0 msgr2=0x7fc2500d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:53.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.755+0000 7fc25567f640 1 --2- 172.21.15.44:0/3833033247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc25015e1d0 0x7fc2500d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.755+0000 7fc25567f640 1 -- 172.21.15.44:0/3833033247 >> v1:172.21.15.69:6789/0 conn(0x7fc25015e5b0 legacy=0x7fc2500b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.755+0000 7fc25567f640 1 -- 172.21.15.44:0/3833033247 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2500cf4d0 con 0x7fc2500cec60 2024-04-02T13:26:53.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.756+0000 7fc25567f640 1 --2- 172.21.15.44:0/3833033247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2500cec60 0x7fc2500d26c0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fc244009870 tx=0x7fc24402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd4531a1be704cc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:53.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.756+0000 7fc254e7e640 1 -- 172.21.15.44:0/3833033247 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244035020 con 0x7fc2500cec60 2024-04-02T13:26:53.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.756+0000 7fc254e7e640 1 -- 172.21.15.44:0/3833033247 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc244031640 con 0x7fc2500cec60 2024-04-02T13:26:53.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.756+0000 7fc254e7e640 1 -- 172.21.15.44:0/3833033247 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24402d040 con 0x7fc2500cec60 2024-04-02T13:26:53.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.756+0000 7fc2580e4640 1 -- 172.21.15.44:0/3833033247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2500cec60 msgr2=0x7fc2500d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.756+0000 7fc2580e4640 1 --2- 172.21.15.44:0/3833033247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2500cec60 0x7fc2500d26c0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fc244009870 tx=0x7fc24402b8f0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.756+0000 7fc2580e4640 1 -- 172.21.15.44:0/3833033247 shutdown_connections 2024-04-02T13:26:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.756+0000 7fc2580e4640 1 --2- 172.21.15.44:0/3833033247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2500cec60 0x7fc2500d26c0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.756+0000 7fc2580e4640 1 --2- 172.21.15.44:0/3833033247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc25015e1d0 0x7fc2500d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.756+0000 7fc2580e4640 1 -- 172.21.15.44:0/3833033247 >> 172.21.15.44:0/3833033247 conn(0x7fc25009d3b0 msgr2=0x7fc2500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.756+0000 7fc2580e4640 1 -- 172.21.15.44:0/3833033247 shutdown_connections 2024-04-02T13:26:53.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.756+0000 7fc2580e4640 1 -- 172.21.15.44:0/3833033247 wait complete. 2024-04-02T13:26:53.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.758+0000 7fc2580e4640 1 Processor -- start 2024-04-02T13:26:53.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.758+0000 7fc2580e4640 1 -- start start 2024-04-02T13:26:53.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.758+0000 7fc2580e4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2500cec60 0x7fc250153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.758+0000 7fc2580e4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc25015e1d0 0x7fc2501610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.758+0000 7fc2580e4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc25015e5b0 0x7fc25015b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.758+0000 7fc2580e4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc2500d2c00 con 0x7fc25015e1d0 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.758+0000 7fc2580e4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc2500cfa90 con 0x7fc2500cec60 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.758+0000 7fc2580e4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc2500d2d80 con 0x7fc25015e5b0 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.758+0000 7fc25567f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc25015e1d0 0x7fc2501610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.758+0000 7fc25567f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc25015e1d0 0x7fc2501610c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49560/0 (socket says 172.21.15.44:49560) 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.758+0000 7fc25567f640 1 -- 172.21.15.44:0/3208392602 learned_addr learned my addr 172.21.15.44:0/3208392602 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.758+0000 7fc255e80640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2500cec60 0x7fc250153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.758+0000 7fc256681640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc25015e5b0 0x7fc25015b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.759+0000 7fc25567f640 1 -- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc25015e5b0 msgr2=0x7fc25015b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.759+0000 7fc25567f640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc25015e5b0 0x7fc25015b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.759+0000 7fc25567f640 1 -- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2500cec60 msgr2=0x7fc250153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.759+0000 7fc25567f640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2500cec60 0x7fc250153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.759+0000 7fc25567f640 1 -- 172.21.15.44:0/3208392602 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25015f4f0 con 0x7fc25015e1d0 2024-04-02T13:26:53.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.759+0000 7fc255e80640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2500cec60 0x7fc250153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:53.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.759+0000 7fc256681640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc25015e5b0 0x7fc25015b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:53.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.759+0000 7fc25567f640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc25015e1d0 0x7fc2501610c0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fc244000c00 tx=0x7fc2440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:53.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.759+0000 7fc24e7fc640 1 -- 172.21.15.44:0/3208392602 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244040070 con 0x7fc25015e1d0 2024-04-02T13:26:53.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.759+0000 7fc24e7fc640 1 -- 172.21.15.44:0/3208392602 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2440399f0 con 0x7fc25015e1d0 2024-04-02T13:26:53.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.759+0000 7fc2580e4640 1 -- 172.21.15.44:0/3208392602 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc25015f7d0 con 0x7fc25015e1d0 2024-04-02T13:26:53.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.759+0000 7fc24e7fc640 1 -- 172.21.15.44:0/3208392602 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244035040 con 0x7fc25015e1d0 2024-04-02T13:26:53.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.759+0000 7fc2580e4640 1 -- 172.21.15.44:0/3208392602 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2500cb9b0 con 0x7fc25015e1d0 2024-04-02T13:26:53.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.760+0000 7fc2580e4640 1 -- 172.21.15.44:0/3208392602 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2500684f0 con 0x7fc25015e1d0 2024-04-02T13:26:53.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.761+0000 7fc24e7fc640 1 -- 172.21.15.44:0/3208392602 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc24404b060 con 0x7fc25015e1d0 2024-04-02T13:26:53.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.761+0000 7fc24e7fc640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc23c06c920 0x7fc23c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:53.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.761+0000 7fc24e7fc640 1 -- 172.21.15.44:0/3208392602 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7fc2440ad630 con 0x7fc25015e1d0 2024-04-02T13:26:53.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.761+0000 7fc255e80640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc23c06c920 0x7fc23c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:53.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.762+0000 7fc255e80640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc23c06c920 0x7fc23c06ede0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc248006fd0 tx=0x7fc248008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:53.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.765+0000 7fc24e7fc640 1 -- 172.21.15.44:0/3208392602 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc24407b8d0 con 0x7fc25015e1d0 2024-04-02T13:26:53.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.949+0000 7fc2580e4640 1 -- 172.21.15.44:0/3208392602 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc2500b86d0 con 0x7fc25015e1d0 2024-04-02T13:26:53.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.949+0000 7fc24e7fc640 1 -- 172.21.15.44:0/3208392602 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fc24407b1d0 con 0x7fc25015e1d0 2024-04-02T13:26:53.952 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:26:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.951+0000 7fc2580e4640 1 -- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc23c06c920 msgr2=0x7fc23c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.951+0000 7fc2580e4640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc23c06c920 0x7fc23c06ede0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc248006fd0 tx=0x7fc248008040 comp rx=0 tx=0).stop 2024-04-02T13:26:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.951+0000 7fc2580e4640 1 -- 172.21.15.44:0/3208392602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc25015e1d0 msgr2=0x7fc2501610c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.951+0000 7fc2580e4640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc25015e1d0 0x7fc2501610c0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fc244000c00 tx=0x7fc2440046c0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.951+0000 7fc2580e4640 1 -- 172.21.15.44:0/3208392602 shutdown_connections 2024-04-02T13:26:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.951+0000 7fc2580e4640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc23c06c920 0x7fc23c06ede0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.951+0000 7fc2580e4640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc25015e5b0 0x7fc25015b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.951+0000 7fc2580e4640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc25015e1d0 0x7fc2501610c0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.951+0000 7fc2580e4640 1 --2- 172.21.15.44:0/3208392602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2500cec60 0x7fc250153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.951+0000 7fc2580e4640 1 -- 172.21.15.44:0/3208392602 >> 172.21.15.44:0/3208392602 conn(0x7fc25009d3b0 msgr2=0x7fc2500b9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.951+0000 7fc2580e4640 1 -- 172.21.15.44:0/3208392602 shutdown_connections 2024-04-02T13:26:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:53.951+0000 7fc2580e4640 1 -- 172.21.15.44:0/3208392602 wait complete. 2024-04-02T13:26:53.964 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:26:54.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.065+0000 7fb54b91c640 1 Processor -- start 2024-04-02T13:26:54.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.066+0000 7fb54b91c640 1 -- start start 2024-04-02T13:26:54.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.066+0000 7fb54b91c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5440cec60 0x7fb5440d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.066+0000 7fb54b91c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb54415e5b0 0x7fb5440d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.066+0000 7fb54b91c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5440cf1f0 con 0x7fb54415e1d0 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.066+0000 7fb54b91c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb5440cfe00 con 0x7fb54415e5b0 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.066+0000 7fb54b91c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb5440cf4d0 con 0x7fb5440cec60 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.066+0000 7fb5496b8640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb54415e1d0 0x7fb5440d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:43002/0 (socket says 172.21.15.44:43002) 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.066+0000 7fb5496b8640 1 -- 172.21.15.44:0/4219697105 learned_addr learned my addr 172.21.15.44:0/4219697105 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb548eb7640 1 --2- 172.21.15.44:0/4219697105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5440cec60 0x7fb5440d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb549eb9640 1 --2- 172.21.15.44:0/4219697105 >> v2:172.21.15.69:3300/0 conn(0x7fb54415e5b0 0x7fb5440d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2625828584 0 0) 0x7fb5440cf1f0 con 0x7fb54415e1d0 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb52c003520 con 0x7fb54415e1d0 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2844721629 0 0) 0x7fb52c003520 con 0x7fb54415e1d0 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5440cec60 msgr2=0x7fb5440d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 --2- 172.21.15.44:0/4219697105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5440cec60 0x7fb5440d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 >> v2:172.21.15.69:3300/0 conn(0x7fb54415e5b0 msgr2=0x7fb5440d2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 --2- 172.21.15.44:0/4219697105 >> v2:172.21.15.69:3300/0 conn(0x7fb54415e5b0 0x7fb5440d2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5440cf7b0 con 0x7fb54415e1d0 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fb53c001670 con 0x7fb54415e1d0 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 >> v1:172.21.15.44:6789/0 conn(0x7fb54415e1d0 legacy=0x7fb5440d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 --2- 172.21.15.44:0/4219697105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5440cec60 0x7fb52c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 --2- 172.21.15.44:0/4219697105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb52c003f30 0x7fb52c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 --2- 172.21.15.44:0/4219697105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb52c004d40 0x7fb52c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb5440cfe00 con 0x7fb52c003f30 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb5440cf4d0 con 0x7fb52c004d40 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb52c00ba10 con 0x7fb5440cec60 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb548eb7640 1 --2- 172.21.15.44:0/4219697105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb52c003f30 0x7fb52c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb5496b8640 1 --2- 172.21.15.44:0/4219697105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5440cec60 0x7fb52c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb549eb9640 1 --2- 172.21.15.44:0/4219697105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb52c004d40 0x7fb52c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb548eb7640 1 -- 172.21.15.44:0/4219697105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5440cec60 msgr2=0x7fb52c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb548eb7640 1 --2- 172.21.15.44:0/4219697105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5440cec60 0x7fb52c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb548eb7640 1 -- 172.21.15.44:0/4219697105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb52c004d40 msgr2=0x7fb52c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:54.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb548eb7640 1 --2- 172.21.15.44:0/4219697105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb52c004d40 0x7fb52c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.067+0000 7fb548eb7640 1 -- 172.21.15.44:0/4219697105 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52c003520 con 0x7fb52c003f30 2024-04-02T13:26:54.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.068+0000 7fb548eb7640 1 --2- 172.21.15.44:0/4219697105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb52c003f30 0x7fb52c004800 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fb538009af0 tx=0x7fb52c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f42386dac0b43f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:54.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.068+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb538039070 con 0x7fb52c003f30 2024-04-02T13:26:54.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.068+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb53802c6b0 con 0x7fb52c003f30 2024-04-02T13:26:54.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.068+0000 7fb543fff640 1 -- 172.21.15.44:0/4219697105 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb538009690 con 0x7fb52c003f30 2024-04-02T13:26:54.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.069+0000 7fb54b91c640 1 -- 172.21.15.44:0/4219697105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb52c003f30 msgr2=0x7fb52c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.069+0000 7fb54b91c640 1 --2- 172.21.15.44:0/4219697105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb52c003f30 0x7fb52c004800 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fb538009af0 tx=0x7fb52c005830 comp rx=0 tx=0).stop 2024-04-02T13:26:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.069+0000 7fb549eb9640 1 -- 172.21.15.44:0/4219697105 reap_dead start 2024-04-02T13:26:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.069+0000 7fb54b91c640 1 -- 172.21.15.44:0/4219697105 shutdown_connections 2024-04-02T13:26:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.069+0000 7fb54b91c640 1 -- 172.21.15.44:0/4219697105 >> 172.21.15.44:0/4219697105 conn(0x7fb54409d3b0 msgr2=0x7fb5440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.069+0000 7fb54b91c640 1 -- 172.21.15.44:0/4219697105 shutdown_connections 2024-04-02T13:26:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.069+0000 7fb54b91c640 1 -- 172.21.15.44:0/4219697105 wait complete. 2024-04-02T13:26:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.070+0000 7fb54b91c640 1 Processor -- start 2024-04-02T13:26:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.070+0000 7fb54b91c640 1 -- start start 2024-04-02T13:26:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb54b91c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb52c003f30 0x7fb54415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb54b91c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5441f4440 0x7fb5441f48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb54b91c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5441609b0 0x7fb544160e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb54b91c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb5440d2840 con 0x7fb5441609b0 2024-04-02T13:26:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb54b91c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb5440d26c0 con 0x7fb5441f4440 2024-04-02T13:26:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb54b91c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb5440d29c0 con 0x7fb52c003f30 2024-04-02T13:26:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb549eb9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5441609b0 0x7fb544160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb549eb9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5441609b0 0x7fb544160e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49576/0 (socket says 172.21.15.44:49576) 2024-04-02T13:26:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb549eb9640 1 -- 172.21.15.44:0/547380963 learned_addr learned my addr 172.21.15.44:0/547380963 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb548eb7640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5441f4440 0x7fb5441f48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb5496b8640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb52c003f30 0x7fb54415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb549eb9640 1 -- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb52c003f30 msgr2=0x7fb54415ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb549eb9640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb52c003f30 0x7fb54415ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb549eb9640 1 -- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5441f4440 msgr2=0x7fb5441f48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb549eb9640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5441f4440 0x7fb5441f48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb549eb9640 1 -- 172.21.15.44:0/547380963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb544161370 con 0x7fb5441609b0 2024-04-02T13:26:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.071+0000 7fb549eb9640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5441609b0 0x7fb544160e30 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fb54408e610 tx=0x7fb53400f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.072+0000 7fb541ffb640 1 -- 172.21.15.44:0/547380963 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53400fd10 con 0x7fb5441609b0 2024-04-02T13:26:54.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.072+0000 7fb541ffb640 1 -- 172.21.15.44:0/547380963 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb534015430 con 0x7fb5441609b0 2024-04-02T13:26:54.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.072+0000 7fb54b91c640 1 -- 172.21.15.44:0/547380963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb54415fbf0 con 0x7fb5441609b0 2024-04-02T13:26:54.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.072+0000 7fb541ffb640 1 -- 172.21.15.44:0/547380963 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb534015820 con 0x7fb5441609b0 2024-04-02T13:26:54.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.072+0000 7fb54b91c640 1 -- 172.21.15.44:0/547380963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5441601b0 con 0x7fb5441609b0 2024-04-02T13:26:54.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.073+0000 7fb54b91c640 1 -- 172.21.15.44:0/547380963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb544073730 con 0x7fb5441609b0 2024-04-02T13:26:54.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.073+0000 7fb541ffb640 1 -- 172.21.15.44:0/547380963 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb5340159c0 con 0x7fb5441609b0 2024-04-02T13:26:54.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.074+0000 7fb541ffb640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb53006c8f0 0x7fb53006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.074+0000 7fb541ffb640 1 -- 172.21.15.44:0/547380963 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7fb53408c9e0 con 0x7fb5441609b0 2024-04-02T13:26:54.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.074+0000 7fb5496b8640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb53006c8f0 0x7fb53006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.074+0000 7fb5496b8640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb53006c8f0 0x7fb53006edb0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb53c007fd0 tx=0x7fb53c0078f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:54.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.078+0000 7fb541ffb640 1 -- 172.21.15.44:0/547380963 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb544073730 con 0x7fb5441609b0 2024-04-02T13:26:54.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.261+0000 7fb54b91c640 1 -- 172.21.15.44:0/547380963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb5440684f0 con 0x7fb5441609b0 2024-04-02T13:26:54.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.262+0000 7fb541ffb640 1 -- 172.21.15.44:0/547380963 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fb544161370 con 0x7fb5441609b0 2024-04-02T13:26:54.264 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:26:54.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.264+0000 7fb54b91c640 1 -- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb53006c8f0 msgr2=0x7fb53006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.264+0000 7fb54b91c640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb53006c8f0 0x7fb53006edb0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb53c007fd0 tx=0x7fb53c0078f0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.264+0000 7fb54b91c640 1 -- 172.21.15.44:0/547380963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5441609b0 msgr2=0x7fb544160e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.264+0000 7fb54b91c640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5441609b0 0x7fb544160e30 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fb54408e610 tx=0x7fb53400f1b0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.264+0000 7fb54b91c640 1 -- 172.21.15.44:0/547380963 shutdown_connections 2024-04-02T13:26:54.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.264+0000 7fb54b91c640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb53006c8f0 0x7fb53006edb0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.264+0000 7fb54b91c640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5441609b0 0x7fb544160e30 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.264+0000 7fb54b91c640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5441f4440 0x7fb5441f48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.264+0000 7fb54b91c640 1 --2- 172.21.15.44:0/547380963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb52c003f30 0x7fb54415ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.264+0000 7fb54b91c640 1 -- 172.21.15.44:0/547380963 >> 172.21.15.44:0/547380963 conn(0x7fb54409d3b0 msgr2=0x7fb5440b7740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:54.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.264+0000 7fb54b91c640 1 -- 172.21.15.44:0/547380963 shutdown_connections 2024-04-02T13:26:54.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.264+0000 7fb54b91c640 1 -- 172.21.15.44:0/547380963 wait complete. 2024-04-02T13:26:54.277 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:26:54.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.376+0000 7f61b9433640 1 Processor -- start 2024-04-02T13:26:54.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.376+0000 7f61b9433640 1 -- start start 2024-04-02T13:26:54.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b9433640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f61b415e4d0 0x7f61b406b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b9433640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f61b4072f60 0x7f61b406ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b9433640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b406dfe0 con 0x7f61b415e0f0 2024-04-02T13:26:54.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b9433640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f61b40aed70 con 0x7f61b415e4d0 2024-04-02T13:26:54.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b9433640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f61b406e2c0 con 0x7f61b4072f60 2024-04-02T13:26:54.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b27fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f61b415e4d0 0x7f61b406b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b27fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f61b415e4d0 0x7f61b406b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49578/0 (socket says 172.21.15.44:49578) 2024-04-02T13:26:54.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b27fc640 1 -- 172.21.15.44:0/3754933426 learned_addr learned my addr 172.21.15.44:0/3754933426 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:54.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b37fe640 1 --2- 172.21.15.44:0/3754933426 >> v2:172.21.15.69:3300/0 conn(0x7f61b4072f60 0x7f61b406ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b27fc640 1 -- 172.21.15.44:0/3754933426 >> v2:172.21.15.69:3300/0 conn(0x7f61b4072f60 msgr2=0x7f61b406ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:54.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b27fc640 1 --2- 172.21.15.44:0/3754933426 >> v2:172.21.15.69:3300/0 conn(0x7f61b4072f60 0x7f61b406ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b27fc640 1 -- 172.21.15.44:0/3754933426 >> v1:172.21.15.69:6789/0 conn(0x7f61b415e0f0 legacy=0x7f61b40d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b27fc640 1 -- 172.21.15.44:0/3754933426 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b406e5a0 con 0x7f61b415e4d0 2024-04-02T13:26:54.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.377+0000 7f61b37fe640 1 --2- 172.21.15.44:0/3754933426 >> v2:172.21.15.69:3300/0 conn(0x7f61b4072f60 0x7f61b406ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:54.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.378+0000 7f61b27fc640 1 --2- 172.21.15.44:0/3754933426 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61b415e4d0 0x7f61b406b4d0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f61b40cb390 tx=0x7f619c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8245b5ab54577ea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:54.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.378+0000 7f61b1ffb640 1 -- 172.21.15.44:0/3754933426 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c02f850 con 0x7f61b415e4d0 2024-04-02T13:26:54.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.378+0000 7f61b1ffb640 1 -- 172.21.15.44:0/3754933426 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f619c02f9f0 con 0x7f61b415e4d0 2024-04-02T13:26:54.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.378+0000 7f61b1ffb640 1 -- 172.21.15.44:0/3754933426 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c02fdc0 con 0x7f61b415e4d0 2024-04-02T13:26:54.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.379+0000 7f61b9433640 1 -- 172.21.15.44:0/3754933426 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61b415e4d0 msgr2=0x7f61b406b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.379+0000 7f61b9433640 1 --2- 172.21.15.44:0/3754933426 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61b415e4d0 0x7f61b406b4d0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f61b40cb390 tx=0x7f619c02ba30 comp rx=0 tx=0).stop 2024-04-02T13:26:54.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.379+0000 7f61b9433640 1 -- 172.21.15.44:0/3754933426 shutdown_connections 2024-04-02T13:26:54.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.379+0000 7f61b9433640 1 --2- 172.21.15.44:0/3754933426 >> v2:172.21.15.69:3300/0 conn(0x7f61b4072f60 0x7f61b406ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.379+0000 7f61b9433640 1 --2- 172.21.15.44:0/3754933426 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61b415e4d0 0x7f61b406b4d0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.379+0000 7f61b9433640 1 -- 172.21.15.44:0/3754933426 >> 172.21.15.44:0/3754933426 conn(0x7f61b40b2980 msgr2=0x7f61b40b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:54.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.379+0000 7f61b9433640 1 -- 172.21.15.44:0/3754933426 shutdown_connections 2024-04-02T13:26:54.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.379+0000 7f61b9433640 1 -- 172.21.15.44:0/3754933426 wait complete. 2024-04-02T13:26:54.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.380+0000 7f61b9433640 1 Processor -- start 2024-04-02T13:26:54.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.380+0000 7f61b9433640 1 -- start start 2024-04-02T13:26:54.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b9433640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61b4072f60 0x7f61b41f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b9433640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61b415e0f0 0x7f61b41f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b9433640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61b415e4d0 0x7f61b40c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b9433640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f61b40d0800 con 0x7f61b4072f60 2024-04-02T13:26:54.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b9433640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f61b40d0af0 con 0x7f61b415e4d0 2024-04-02T13:26:54.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b9433640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f61b406eda0 con 0x7f61b415e0f0 2024-04-02T13:26:54.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61b4072f60 0x7f61b41f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61b4072f60 0x7f61b41f54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49580/0 (socket says 172.21.15.44:49580) 2024-04-02T13:26:54.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b2ffd640 1 -- 172.21.15.44:0/2049954186 learned_addr learned my addr 172.21.15.44:0/2049954186 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:54.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b37fe640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61b415e4d0 0x7f61b40c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b27fc640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61b415e0f0 0x7f61b41f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b2ffd640 1 -- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61b415e0f0 msgr2=0x7f61b41f7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b2ffd640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61b415e0f0 0x7f61b41f7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b2ffd640 1 -- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61b415e4d0 msgr2=0x7f61b40c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b2ffd640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61b415e4d0 0x7f61b40c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b2ffd640 1 -- 172.21.15.44:0/2049954186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b40c60c0 con 0x7f61b4072f60 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61b2ffd640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61b4072f60 0x7f61b41f54c0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00c790 tx=0x7f61ac00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61ab7fe640 1 -- 172.21.15.44:0/2049954186 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac016020 con 0x7f61b4072f60 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.381+0000 7f61ab7fe640 1 -- 172.21.15.44:0/2049954186 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61ac006d50 con 0x7f61b4072f60 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.382+0000 7f61b9433640 1 -- 172.21.15.44:0/2049954186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b40cb480 con 0x7f61b4072f60 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.382+0000 7f61ab7fe640 1 -- 172.21.15.44:0/2049954186 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac003bb0 con 0x7f61b4072f60 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.382+0000 7f61b9433640 1 -- 172.21.15.44:0/2049954186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b40cba40 con 0x7f61b4072f60 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.382+0000 7f61b9433640 1 -- 172.21.15.44:0/2049954186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b40c4570 con 0x7f61b4072f60 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.383+0000 7f61ab7fe640 1 -- 172.21.15.44:0/2049954186 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f61ac00a850 con 0x7f61b4072f60 2024-04-02T13:26:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.383+0000 7f61ab7fe640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f619406c8f0 0x7f619406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.384+0000 7f61ab7fe640 1 -- 172.21.15.44:0/2049954186 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7f61ac08b9a0 con 0x7f61b4072f60 2024-04-02T13:26:54.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.384+0000 7f61b27fc640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f619406c8f0 0x7f619406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.384+0000 7f61b27fc640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f619406c8f0 0x7f619406edb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f619c036000 tx=0x7f619c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:54.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.388+0000 7f61ab7fe640 1 -- 172.21.15.44:0/2049954186 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f61b40c4570 con 0x7f61b4072f60 2024-04-02T13:26:54.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.584+0000 7f61b9433640 1 -- 172.21.15.44:0/2049954186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f61b40684f0 con 0x7f61b4072f60 2024-04-02T13:26:54.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.584+0000 7f61ab7fe640 1 -- 172.21.15.44:0/2049954186 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v91) ==== 80+0+0 (secure 0 0 0) 0x7f61ac020020 con 0x7f61b4072f60 2024-04-02T13:26:54.587 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:26:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.586+0000 7f61b9433640 1 -- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f619406c8f0 msgr2=0x7f619406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.586+0000 7f61b9433640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f619406c8f0 0x7f619406edb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f619c036000 tx=0x7f619c008030 comp rx=0 tx=0).stop 2024-04-02T13:26:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.586+0000 7f61b9433640 1 -- 172.21.15.44:0/2049954186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61b4072f60 msgr2=0x7f61b41f54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.586+0000 7f61b9433640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61b4072f60 0x7f61b41f54c0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00c790 tx=0x7f61ac00cc50 comp rx=0 tx=0).stop 2024-04-02T13:26:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.586+0000 7f61b9433640 1 -- 172.21.15.44:0/2049954186 shutdown_connections 2024-04-02T13:26:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.586+0000 7f61b9433640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f619406c8f0 0x7f619406edb0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.586+0000 7f61b9433640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61b415e4d0 0x7f61b40c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.586+0000 7f61b9433640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61b415e0f0 0x7f61b41f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.586+0000 7f61b9433640 1 --2- 172.21.15.44:0/2049954186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61b4072f60 0x7f61b41f54c0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.586+0000 7f61b9433640 1 -- 172.21.15.44:0/2049954186 >> 172.21.15.44:0/2049954186 conn(0x7f61b40b2980 msgr2=0x7f61b40ae3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.586+0000 7f61b9433640 1 -- 172.21.15.44:0/2049954186 shutdown_connections 2024-04-02T13:26:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.586+0000 7f61b9433640 1 -- 172.21.15.44:0/2049954186 wait complete. 2024-04-02T13:26:54.599 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:26:54.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:54.599 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:26:54.624 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[69777]: fuse finished with error 0 and tester_r 0 2024-04-02T13:26:54.626 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:26:54.720 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:26:54.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:26:54.721 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:26:54.735 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:26:54.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.875+0000 7f6b8c247640 1 Processor -- start 2024-04-02T13:26:54.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.875+0000 7f6b8c247640 1 -- start start 2024-04-02T13:26:54.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.876+0000 7f6b8c247640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b840b2d00 0x7f6b840cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.876+0000 7f6b8c247640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b840b3160 con 0x7f6b8415aa00 2024-04-02T13:26:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.876+0000 7f6b8c247640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8415ade0 con 0x7f6b840ce110 2024-04-02T13:26:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.876+0000 7f6b8c247640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6b840af1b0 con 0x7f6b840b2d00 2024-04-02T13:26:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.876+0000 7f6b89fe3640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6b8415aa00 0x7f6b84072bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37628/0 (socket says 172.21.15.44:37628) 2024-04-02T13:26:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.876+0000 7f6b89fe3640 1 -- 172.21.15.44:0/512357927 learned_addr learned my addr 172.21.15.44:0/512357927 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.877+0000 7f6b897e2640 1 --2- 172.21.15.44:0/512357927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b840b2d00 0x7f6b840cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.877+0000 7f6b88fe1640 1 -- 172.21.15.44:0/512357927 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1371871349 0 0) 0x7f6b840b3160 con 0x7f6b8415aa00 2024-04-02T13:26:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.877+0000 7f6b88fe1640 1 -- 172.21.15.44:0/512357927 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b6c005730 con 0x7f6b8415aa00 2024-04-02T13:26:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.877+0000 7f6b88fe1640 1 -- 172.21.15.44:0/512357927 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3988254637 0 0) 0x7f6b8415ade0 con 0x7f6b840ce110 2024-04-02T13:26:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.877+0000 7f6b88fe1640 1 -- 172.21.15.44:0/512357927 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b840b3160 con 0x7f6b840ce110 2024-04-02T13:26:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.877+0000 7f6b897e2640 1 -- 172.21.15.44:0/512357927 >> v1:172.21.15.69:6789/0 conn(0x7f6b840ce110 legacy=0x7f6b84071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.877+0000 7f6b897e2640 1 -- 172.21.15.44:0/512357927 >> v1:172.21.15.44:6789/0 conn(0x7f6b8415aa00 legacy=0x7f6b84072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.877+0000 7f6b897e2640 1 -- 172.21.15.44:0/512357927 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b840af490 con 0x7f6b840b2d00 2024-04-02T13:26:54.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.878+0000 7f6b897e2640 1 --2- 172.21.15.44:0/512357927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b840b2d00 0x7f6b840cdbd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6b780097d0 tx=0x7f6b7802bac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11c2fe056a5a91f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:54.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.878+0000 7f6b88fe1640 1 -- 172.21.15.44:0/512357927 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7802d880 con 0x7f6b840b2d00 2024-04-02T13:26:54.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.878+0000 7f6b88fe1640 1 -- 172.21.15.44:0/512357927 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b7802da20 con 0x7f6b840b2d00 2024-04-02T13:26:54.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.878+0000 7f6b88fe1640 1 -- 172.21.15.44:0/512357927 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7802ddd0 con 0x7f6b840b2d00 2024-04-02T13:26:54.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.879+0000 7f6b8c247640 1 -- 172.21.15.44:0/512357927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b840b2d00 msgr2=0x7f6b840cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.879+0000 7f6b8c247640 1 --2- 172.21.15.44:0/512357927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b840b2d00 0x7f6b840cdbd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6b780097d0 tx=0x7f6b7802bac0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.879+0000 7f6b8c247640 1 -- 172.21.15.44:0/512357927 shutdown_connections 2024-04-02T13:26:54.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.879+0000 7f6b8c247640 1 --2- 172.21.15.44:0/512357927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b840b2d00 0x7f6b840cdbd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.879+0000 7f6b8c247640 1 -- 172.21.15.44:0/512357927 >> 172.21.15.44:0/512357927 conn(0x7f6b840cfcd0 msgr2=0x7f6b840d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:54.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.879+0000 7f6b8c247640 1 -- 172.21.15.44:0/512357927 shutdown_connections 2024-04-02T13:26:54.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.879+0000 7f6b8c247640 1 -- 172.21.15.44:0/512357927 wait complete. 2024-04-02T13:26:54.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.880+0000 7f6b8c247640 1 Processor -- start 2024-04-02T13:26:54.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.880+0000 7f6b8c247640 1 -- start start 2024-04-02T13:26:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.880+0000 7f6b8c247640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b840b2d00 0x7f6b841f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8c247640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b840ce110 0x7f6b841592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8c247640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b8415aa00 0x7f6b841f1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8c247640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6b840ce5a0 con 0x7f6b8415aa00 2024-04-02T13:26:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8c247640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6b840726d0 con 0x7f6b840ce110 2024-04-02T13:26:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8c247640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6b840cc750 con 0x7f6b840b2d00 2024-04-02T13:26:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8a7e4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b8415aa00 0x7f6b841f1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8a7e4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b8415aa00 0x7f6b841f1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52014/0 (socket says 172.21.15.44:52014) 2024-04-02T13:26:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8a7e4640 1 -- 172.21.15.44:0/4292398197 learned_addr learned my addr 172.21.15.44:0/4292398197 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b897e2640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b840ce110 0x7f6b841592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b89fe3640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b840b2d00 0x7f6b841f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8a7e4640 1 -- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b840b2d00 msgr2=0x7f6b841f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8a7e4640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b840b2d00 0x7f6b841f2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8a7e4640 1 -- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b840ce110 msgr2=0x7f6b841592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8a7e4640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b840ce110 0x7f6b841592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8a7e4640 1 -- 172.21.15.44:0/4292398197 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b841f1e10 con 0x7f6b8415aa00 2024-04-02T13:26:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b897e2640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b840ce110 0x7f6b841592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b89fe3640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b840b2d00 0x7f6b841f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8a7e4640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b8415aa00 0x7f6b841f1350 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400c7c0 tx=0x7f6b7400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b827fc640 1 -- 172.21.15.44:0/4292398197 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b74003b70 con 0x7f6b8415aa00 2024-04-02T13:26:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b827fc640 1 -- 172.21.15.44:0/4292398197 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b74003d10 con 0x7f6b8415aa00 2024-04-02T13:26:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b827fc640 1 -- 172.21.15.44:0/4292398197 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7400aa50 con 0x7f6b8415aa00 2024-04-02T13:26:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8c247640 1 -- 172.21.15.44:0/4292398197 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b841f20f0 con 0x7f6b8415aa00 2024-04-02T13:26:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.881+0000 7f6b8c247640 1 -- 172.21.15.44:0/4292398197 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b841f2990 con 0x7f6b8415aa00 2024-04-02T13:26:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.883+0000 7f6b8c247640 1 -- 172.21.15.44:0/4292398197 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b84074f20 con 0x7f6b8415aa00 2024-04-02T13:26:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.883+0000 7f6b827fc640 1 -- 172.21.15.44:0/4292398197 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6b7400abf0 con 0x7f6b8415aa00 2024-04-02T13:26:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.883+0000 7f6b827fc640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b7006c8f0 0x7f6b7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.883+0000 7f6b827fc640 1 -- 172.21.15.44:0/4292398197 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7f6b7408bce0 con 0x7f6b8415aa00 2024-04-02T13:26:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.884+0000 7f6b89fe3640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b7006c8f0 0x7f6b7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.884+0000 7f6b89fe3640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b7006c8f0 0x7f6b7006edb0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0011e0 tx=0x7f6b7c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:54.888+0000 7f6b827fc640 1 -- 172.21.15.44:0/4292398197 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6b74059f80 con 0x7f6b8415aa00 2024-04-02T13:26:55.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.111+0000 7f6b8c247640 1 -- 172.21.15.44:0/4292398197 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b841f2b90 con 0x7f6b8415aa00 2024-04-02T13:26:55.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.112+0000 7f6b827fc640 1 -- 172.21.15.44:0/4292398197 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5177 (secure 0 0 0) 0x7f6b74059880 con 0x7f6b8415aa00 2024-04-02T13:26:55.115 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:55.115 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":61,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:26:46.063220+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6833/719680934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":719680934},{"type":"v1","addr":"172.21.15.44:6833","nonce":719680934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":6}]} 2024-04-02T13:26:55.117 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 61 2024-04-02T13:26:55.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.113+0000 7f6b8c247640 1 -- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b7006c8f0 msgr2=0x7f6b7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.113+0000 7f6b8c247640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b7006c8f0 0x7f6b7006edb0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0011e0 tx=0x7f6b7c009040 comp rx=0 tx=0).stop 2024-04-02T13:26:55.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.113+0000 7f6b8c247640 1 -- 172.21.15.44:0/4292398197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b8415aa00 msgr2=0x7f6b841f1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.113+0000 7f6b8c247640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b8415aa00 0x7f6b841f1350 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400c7c0 tx=0x7f6b7400cc80 comp rx=0 tx=0).stop 2024-04-02T13:26:55.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.113+0000 7f6b8c247640 1 -- 172.21.15.44:0/4292398197 shutdown_connections 2024-04-02T13:26:55.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.113+0000 7f6b8c247640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6b7006c8f0 0x7f6b7006edb0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.114+0000 7f6b8c247640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b8415aa00 0x7f6b841f1350 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.114+0000 7f6b8c247640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b840ce110 0x7f6b841592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.114+0000 7f6b8c247640 1 --2- 172.21.15.44:0/4292398197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b840b2d00 0x7f6b841f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.114+0000 7f6b8c247640 1 -- 172.21.15.44:0/4292398197 >> 172.21.15.44:0/4292398197 conn(0x7f6b840cfcd0 msgr2=0x7f6b84150830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:55.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.114+0000 7f6b8c247640 1 -- 172.21.15.44:0/4292398197 shutdown_connections 2024-04-02T13:26:55.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.114+0000 7f6b8c247640 1 -- 172.21.15.44:0/4292398197 wait complete. 2024-04-02T13:26:55.126 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-04-02T13:26:55.254 INFO:teuthology.orchestra.run.smithi044.stdout:{"message":"","return_code":0} 2024-04-02T13:26:55.254 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-02T13:26:55.254 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:26:55.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.354+0000 7ff4f607b640 1 Processor -- start 2024-04-02T13:26:55.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.355+0000 7ff4f607b640 1 -- start start 2024-04-02T13:26:55.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.355+0000 7ff4f607b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4f00d0a30 0x7ff4f006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.355+0000 7ff4f607b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff4f00d0e10 0x7ff4f00cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.355+0000 7ff4f607b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff4f00cfe30 0x7ff4f00d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.355+0000 7ff4f607b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff4f006cb00 con 0x7ff4f00cfe30 2024-04-02T13:26:55.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.355+0000 7ff4f607b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff4f006c1d0 con 0x7ff4f00d0e10 2024-04-02T13:26:55.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.355+0000 7ff4f607b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff4f006c4b0 con 0x7ff4f00d0a30 2024-04-02T13:26:55.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.355+0000 7ff4effff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff4f00cfe30 0x7ff4f00d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.355+0000 7ff4effff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff4f00cfe30 0x7ff4f00d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52024/0 (socket says 172.21.15.44:52024) 2024-04-02T13:26:55.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.356+0000 7ff4effff640 1 -- 172.21.15.44:0/4239672400 learned_addr learned my addr 172.21.15.44:0/4239672400 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:55.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.356+0000 7ff4eeffd640 1 --2- 172.21.15.44:0/4239672400 >> v2:172.21.15.69:3300/0 conn(0x7ff4f00d0e10 0x7ff4f00cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.356+0000 7ff4effff640 1 -- 172.21.15.44:0/4239672400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4f00d0a30 msgr2=0x7ff4f006cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:55.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.356+0000 7ff4ef7fe640 1 --2- 172.21.15.44:0/4239672400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4f00d0a30 0x7ff4f006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.356+0000 7ff4effff640 1 --2- 172.21.15.44:0/4239672400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4f00d0a30 0x7ff4f006cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.356+0000 7ff4effff640 1 -- 172.21.15.44:0/4239672400 >> v2:172.21.15.69:3300/0 conn(0x7ff4f00d0e10 msgr2=0x7ff4f00cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.356+0000 7ff4effff640 1 --2- 172.21.15.44:0/4239672400 >> v2:172.21.15.69:3300/0 conn(0x7ff4f00d0e10 0x7ff4f00cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.356+0000 7ff4effff640 1 -- 172.21.15.44:0/4239672400 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f006c790 con 0x7ff4f00cfe30 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.356+0000 7ff4eeffd640 1 --2- 172.21.15.44:0/4239672400 >> v2:172.21.15.69:3300/0 conn(0x7ff4f00d0e10 0x7ff4f00cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.356+0000 7ff4ef7fe640 1 --2- 172.21.15.44:0/4239672400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4f00d0a30 0x7ff4f006cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.357+0000 7ff4effff640 1 --2- 172.21.15.44:0/4239672400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4f00cfe30 0x7ff4f00d2040 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7ff4f006bdf0 tx=0x7ff4e802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416ddffa2360f5fa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.357+0000 7ff4ee7fc640 1 -- 172.21.15.44:0/4239672400 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e802c9e0 con 0x7ff4f00cfe30 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.357+0000 7ff4ee7fc640 1 -- 172.21.15.44:0/4239672400 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4e802cb80 con 0x7ff4f00cfe30 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.357+0000 7ff4ee7fc640 1 -- 172.21.15.44:0/4239672400 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e80343f0 con 0x7ff4f00cfe30 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.357+0000 7ff4f607b640 1 -- 172.21.15.44:0/4239672400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4f00cfe30 msgr2=0x7ff4f00d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.357+0000 7ff4f607b640 1 --2- 172.21.15.44:0/4239672400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4f00cfe30 0x7ff4f00d2040 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7ff4f006bdf0 tx=0x7ff4e802be40 comp rx=0 tx=0).stop 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.357+0000 7ff4f607b640 1 -- 172.21.15.44:0/4239672400 shutdown_connections 2024-04-02T13:26:55.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.357+0000 7ff4f607b640 1 --2- 172.21.15.44:0/4239672400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4f00cfe30 0x7ff4f00d2040 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.357+0000 7ff4f607b640 1 --2- 172.21.15.44:0/4239672400 >> v2:172.21.15.69:3300/0 conn(0x7ff4f00d0e10 0x7ff4f00cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.357+0000 7ff4f607b640 1 --2- 172.21.15.44:0/4239672400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4f00d0a30 0x7ff4f006cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.357+0000 7ff4f607b640 1 -- 172.21.15.44:0/4239672400 >> 172.21.15.44:0/4239672400 conn(0x7ff4f009d470 msgr2=0x7ff4f00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:55.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.357+0000 7ff4f607b640 1 -- 172.21.15.44:0/4239672400 shutdown_connections 2024-04-02T13:26:55.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.357+0000 7ff4f607b640 1 -- 172.21.15.44:0/4239672400 wait complete. 2024-04-02T13:26:55.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.358+0000 7ff4f607b640 1 Processor -- start 2024-04-02T13:26:55.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.359+0000 7ff4f607b640 1 -- start start 2024-04-02T13:26:55.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.359+0000 7ff4f607b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4f00cfe30 0x7ff4f014e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.359+0000 7ff4f607b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4f00d0a30 0x7ff4f00b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.359+0000 7ff4f607b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4f00d0e10 0x7ff4f00d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.359+0000 7ff4f607b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f015e870 con 0x7ff4f00cfe30 2024-04-02T13:26:55.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.359+0000 7ff4f607b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f015e6f0 con 0x7ff4f00d0e10 2024-04-02T13:26:55.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.359+0000 7ff4f607b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff4f00d4da0 con 0x7ff4f00d0a30 2024-04-02T13:26:55.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.359+0000 7ff4ef7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4f00cfe30 0x7ff4f014e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.359+0000 7ff4ef7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4f00cfe30 0x7ff4f014e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52030/0 (socket says 172.21.15.44:52030) 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.359+0000 7ff4ef7fe640 1 -- 172.21.15.44:0/1152283514 learned_addr learned my addr 172.21.15.44:0/1152283514 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.360+0000 7ff4eeffd640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4f00d0a30 0x7ff4f00b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.360+0000 7ff4effff640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4f00d0e10 0x7ff4f00d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.360+0000 7ff4ef7fe640 1 -- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4f00d0a30 msgr2=0x7ff4f00b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.360+0000 7ff4ef7fe640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4f00d0a30 0x7ff4f00b3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.360+0000 7ff4ef7fe640 1 -- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4f00d0e10 msgr2=0x7ff4f00d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.360+0000 7ff4ef7fe640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4f00d0e10 0x7ff4f00d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.360+0000 7ff4ef7fe640 1 -- 172.21.15.44:0/1152283514 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f00b6460 con 0x7ff4f00cfe30 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.360+0000 7ff4ef7fe640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4f00cfe30 0x7ff4f014e680 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000c7f0 tx=0x7ff4e000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.360+0000 7ff4d7fff640 1 -- 172.21.15.44:0/1152283514 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0015070 con 0x7ff4f00cfe30 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.360+0000 7ff4d7fff640 1 -- 172.21.15.44:0/1152283514 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4e0007a30 con 0x7ff4f00cfe30 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.360+0000 7ff4d7fff640 1 -- 172.21.15.44:0/1152283514 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0007e00 con 0x7ff4f00cfe30 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.360+0000 7ff4f607b640 1 -- 172.21.15.44:0/1152283514 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f00b6740 con 0x7ff4f00cfe30 2024-04-02T13:26:55.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.360+0000 7ff4f607b640 1 -- 172.21.15.44:0/1152283514 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f00c7f00 con 0x7ff4f00cfe30 2024-04-02T13:26:55.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.361+0000 7ff4f607b640 1 -- 172.21.15.44:0/1152283514 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4f00b2af0 con 0x7ff4f00cfe30 2024-04-02T13:26:55.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.362+0000 7ff4d7fff640 1 -- 172.21.15.44:0/1152283514 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff4e0018070 con 0x7ff4f00cfe30 2024-04-02T13:26:55.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.362+0000 7ff4d7fff640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff4dc06c8f0 0x7ff4dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.362+0000 7ff4d7fff640 1 -- 172.21.15.44:0/1152283514 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6101+0+0 (secure 0 0 0) 0x7ff4e008bc00 con 0x7ff4f00cfe30 2024-04-02T13:26:55.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.362+0000 7ff4eeffd640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff4dc06c8f0 0x7ff4dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.363+0000 7ff4eeffd640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff4dc06c8f0 0x7ff4dc06edb0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff4f015c470 tx=0x7ff4e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.366+0000 7ff4d7fff640 1 -- 172.21.15.44:0/1152283514 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff4f00b2af0 con 0x7ff4f00cfe30 2024-04-02T13:26:55.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.588+0000 7ff4f607b640 1 -- 172.21.15.44:0/1152283514 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff4f00684f0 con 0x7ff4f00cfe30 2024-04-02T13:26:55.591 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:26:55.588+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:26:55.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.590+0000 7ff4d7fff640 1 -- 172.21.15.44:0/1152283514 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v62) ==== 161+0+0 (secure 0 0 0) 0x7ff4e0092080 con 0x7ff4f00cfe30 2024-04-02T13:26:55.593 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:26:55.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.593+0000 7ff4f607b640 1 -- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff4dc06c8f0 msgr2=0x7ff4dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.593+0000 7ff4f607b640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff4dc06c8f0 0x7ff4dc06edb0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff4f015c470 tx=0x7ff4e4008040 comp rx=0 tx=0).stop 2024-04-02T13:26:55.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.593+0000 7ff4f607b640 1 -- 172.21.15.44:0/1152283514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4f00cfe30 msgr2=0x7ff4f014e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.593+0000 7ff4f607b640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4f00cfe30 0x7ff4f014e680 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000c7f0 tx=0x7ff4e000ccb0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.593+0000 7ff4f607b640 1 -- 172.21.15.44:0/1152283514 shutdown_connections 2024-04-02T13:26:55.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.593+0000 7ff4f607b640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff4dc06c8f0 0x7ff4dc06edb0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.593+0000 7ff4f607b640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4f00d0e10 0x7ff4f00d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.593+0000 7ff4f607b640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4f00d0a30 0x7ff4f00b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.593+0000 7ff4f607b640 1 --2- 172.21.15.44:0/1152283514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4f00cfe30 0x7ff4f014e680 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.593+0000 7ff4f607b640 1 -- 172.21.15.44:0/1152283514 >> 172.21.15.44:0/1152283514 conn(0x7ff4f009d470 msgr2=0x7ff4f00b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:55.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.593+0000 7ff4f607b640 1 -- 172.21.15.44:0/1152283514 shutdown_connections 2024-04-02T13:26:55.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.593+0000 7ff4f607b640 1 -- 172.21.15.44:0/1152283514 wait complete. 2024-04-02T13:26:55.598 INFO:tasks.ceph.mds.a.smithi044.stderr: -3710> 2024-04-02T13:26:49.645+0000 7f91a4f1f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:26:55.607 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 10000000000.00000000 2024-04-02T13:26:55.626 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:26:55.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.646+0000 7f39a837ff40 1 Processor -- start 2024-04-02T13:26:55.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.646+0000 7f39a837ff40 1 -- start start 2024-04-02T13:26:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a837ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5618fe676eb0 0x5618fe6332e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a837ff40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5618fe7034e0 0x5618fe705290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a837ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5618fe7092e0 0x5618fe7085c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a837ff40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5618fe67b920 con 0x5618fe7034e0 2024-04-02T13:26:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a837ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5618fe5d73e0 con 0x5618fe676eb0 2024-04-02T13:26:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a837ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5618fe679970 con 0x5618fe7092e0 2024-04-02T13:26:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a5681640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5618fe7034e0 0x5618fe705290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a5681640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5618fe7034e0 0x5618fe705290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52072/0 (socket says 172.21.15.44:52072) 2024-04-02T13:26:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a5681640 1 -- 172.21.15.44:0/3759590452 learned_addr learned my addr 172.21.15.44:0/3759590452 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a5e82640 1 --2- 172.21.15.44:0/3759590452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5618fe676eb0 0x5618fe6332e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a6683640 1 --2- 172.21.15.44:0/3759590452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5618fe7092e0 0x5618fe7085c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a5681640 1 -- 172.21.15.44:0/3759590452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5618fe7092e0 msgr2=0x5618fe7085c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a5681640 1 --2- 172.21.15.44:0/3759590452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5618fe7092e0 0x5618fe7085c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a5681640 1 -- 172.21.15.44:0/3759590452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5618fe676eb0 msgr2=0x5618fe6332e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a5681640 1 --2- 172.21.15.44:0/3759590452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5618fe676eb0 0x5618fe6332e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a5681640 1 -- 172.21.15.44:0/3759590452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618fe660e10 con 0x5618fe7034e0 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f39a5681640 1 --2- 172.21.15.44:0/3759590452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5618fe7034e0 0x5618fe705290 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f399000b340 tx=0x7f399000b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3718d504222f5af8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f399ffff640 1 -- 172.21.15.44:0/3759590452 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3990008070 con 0x5618fe7034e0 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f399ffff640 1 -- 172.21.15.44:0/3759590452 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f399000d5c0 con 0x5618fe7034e0 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.647+0000 7f399ffff640 1 -- 172.21.15.44:0/3759590452 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399000d970 con 0x5618fe7034e0 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.648+0000 7f39a837ff40 1 -- 172.21.15.44:0/3759590452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5618fe7034e0 msgr2=0x5618fe705290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.648+0000 7f39a837ff40 1 --2- 172.21.15.44:0/3759590452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5618fe7034e0 0x5618fe705290 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f399000b340 tx=0x7f399000b800 comp rx=0 tx=0).stop 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.648+0000 7f39a837ff40 1 -- 172.21.15.44:0/3759590452 shutdown_connections 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.648+0000 7f39a837ff40 1 --2- 172.21.15.44:0/3759590452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5618fe7092e0 0x5618fe7085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.648+0000 7f39a837ff40 1 --2- 172.21.15.44:0/3759590452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5618fe7034e0 0x5618fe705290 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.648+0000 7f39a837ff40 1 --2- 172.21.15.44:0/3759590452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5618fe676eb0 0x5618fe6332e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.648+0000 7f39a837ff40 1 -- 172.21.15.44:0/3759590452 >> 172.21.15.44:0/3759590452 conn(0x5618fe65bb50 msgr2=0x5618fe7096c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:55.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.648+0000 7f39a837ff40 1 -- 172.21.15.44:0/3759590452 shutdown_connections 2024-04-02T13:26:55.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.648+0000 7f39a837ff40 1 -- 172.21.15.44:0/3759590452 wait complete. 2024-04-02T13:26:55.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.648+0000 7f39a837ff40 1 Processor -- start 2024-04-02T13:26:55.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.649+0000 7f39a837ff40 1 -- start start 2024-04-02T13:26:55.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.649+0000 7f39a837ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5618fe676eb0 0x5618fe661570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.649+0000 7f39a837ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5618fe7034e0 0x5618fe6fb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.649+0000 7f39a837ff40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5618fe7092e0 0x5618fe632630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.649+0000 7f39a837ff40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5618fe710930 con 0x5618fe7092e0 2024-04-02T13:26:55.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.649+0000 7f39a837ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5618fe7106e0 con 0x5618fe676eb0 2024-04-02T13:26:55.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.649+0000 7f39a837ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5618fe710090 con 0x5618fe7034e0 2024-04-02T13:26:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.649+0000 7f39a6683640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5618fe7092e0 0x5618fe632630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.649+0000 7f39a6683640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5618fe7092e0 0x5618fe632630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52082/0 (socket says 172.21.15.44:52082) 2024-04-02T13:26:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.649+0000 7f39a6683640 1 -- 172.21.15.44:0/3158413230 learned_addr learned my addr 172.21.15.44:0/3158413230 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.649+0000 7f39a5681640 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5618fe7034e0 0x5618fe6fb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.649+0000 7f39a5e82640 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5618fe676eb0 0x5618fe661570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.650+0000 7f39a6683640 1 -- 172.21.15.44:0/3158413230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5618fe7034e0 msgr2=0x5618fe6fb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.650+0000 7f39a6683640 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5618fe7034e0 0x5618fe6fb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.650+0000 7f39a6683640 1 -- 172.21.15.44:0/3158413230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5618fe676eb0 msgr2=0x5618fe661570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.650+0000 7f39a6683640 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5618fe676eb0 0x5618fe661570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.650+0000 7f39a6683640 1 -- 172.21.15.44:0/3158413230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618fe676480 con 0x5618fe7092e0 2024-04-02T13:26:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.650+0000 7f39a6683640 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5618fe7092e0 0x5618fe632630 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f39a000bfd0 tx=0x7f39a000aaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.650+0000 7f399effd640 1 -- 172.21.15.44:0/3158413230 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a0019070 con 0x5618fe7092e0 2024-04-02T13:26:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.650+0000 7f399effd640 1 -- 172.21.15.44:0/3158413230 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39a0008780 con 0x5618fe7092e0 2024-04-02T13:26:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.650+0000 7f39a837ff40 1 -- 172.21.15.44:0/3158413230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5618fe705250 con 0x5618fe7092e0 2024-04-02T13:26:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.650+0000 7f399effd640 1 -- 172.21.15.44:0/3158413230 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a0008b90 con 0x5618fe7092e0 2024-04-02T13:26:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.650+0000 7f39a837ff40 1 -- 172.21.15.44:0/3158413230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618fe676720 con 0x5618fe7092e0 2024-04-02T13:26:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.651+0000 7f399effd640 1 -- 172.21.15.44:0/3158413230 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f39a001c360 con 0x5618fe7092e0 2024-04-02T13:26:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.652+0000 7f399effd640 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f398806b300 0x7f398806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.652+0000 7f399effd640 1 -- 172.21.15.44:0/3158413230 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7f39a008a770 con 0x5618fe7092e0 2024-04-02T13:26:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.652+0000 7f39a837ff40 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5618fe675260 0x5618fe675640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.652+0000 7f39a837ff40 1 -- 172.21.15.44:0/3158413230 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:1 13.10 13:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5618fe675b80 con 0x5618fe675260 2024-04-02T13:26:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.652+0000 7f39a5e82640 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f398806b300 0x7f398806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.652+0000 7f39a5681640 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5618fe675260 0x5618fe675640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.652+0000 7f39a5681640 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5618fe675260 0x5618fe675640 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.653+0000 7f39a5e82640 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f398806b300 0x7f398806d7c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3998053000 tx=0x7f399804e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.655+0000 7f39a5681640 1 -- 172.21.15.44:0/3158413230 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(1 10000000000.00000000 [delete] v92'11 uv11 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f399000d870 con 0x5618fe675260 2024-04-02T13:26:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.655+0000 7f39a837ff40 1 -- 172.21.15.44:0/3158413230 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5618fe675260 msgr2=0x5618fe675640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.655+0000 7f39a837ff40 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5618fe675260 0x5618fe675640 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.655+0000 7f39a837ff40 1 -- 172.21.15.44:0/3158413230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f398806b300 msgr2=0x7f398806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.655+0000 7f39a837ff40 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f398806b300 0x7f398806d7c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3998053000 tx=0x7f399804e000 comp rx=0 tx=0).stop 2024-04-02T13:26:55.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.656+0000 7f39a837ff40 1 -- 172.21.15.44:0/3158413230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5618fe7092e0 msgr2=0x5618fe632630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.656+0000 7f39a837ff40 1 --2- 172.21.15.44:0/3158413230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5618fe7092e0 0x5618fe632630 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f39a000bfd0 tx=0x7f39a000aaa0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.656+0000 7f39a6683640 1 -- 172.21.15.44:0/3158413230 reap_dead start 2024-04-02T13:26:55.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.656+0000 7f39a837ff40 1 -- 172.21.15.44:0/3158413230 shutdown_connections 2024-04-02T13:26:55.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.656+0000 7f39a837ff40 1 -- 172.21.15.44:0/3158413230 >> 172.21.15.44:0/3158413230 conn(0x5618fe65bb50 msgr2=0x5618fe7096c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:55.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.656+0000 7f39a837ff40 1 -- 172.21.15.44:0/3158413230 shutdown_connections 2024-04-02T13:26:55.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.656+0000 7f39a837ff40 1 -- 172.21.15.44:0/3158413230 wait complete. 2024-04-02T13:26:55.668 DEBUG:teuthology.orchestra.run.smithi044:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset 2024-04-02T13:26:55.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.739+0000 7f9cbe541280 1 -- start start 2024-04-02T13:26:55.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.739+0000 7f9cbe541280 1 Processor -- start 2024-04-02T13:26:55.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.739+0000 7f9cbe541280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb81eec00 0x55eeb8fb0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.739+0000 7f9cbe541280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb81ef800 0x55eeb8fb0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.739+0000 7f9cbe541280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb81efc00 0x55eeb8fb0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.739+0000 7f9cbe541280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eeb8fd0180 con 0x55eeb81efc00 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.739+0000 7f9cbe541280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55eeb8fd0300 con 0x55eeb81ef800 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.739+0000 7f9cbe541280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55eeb8fd0480 con 0x55eeb81eec00 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbd4f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb81efc00 0x55eeb8fb0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbd4f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb81efc00 0x55eeb8fb0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52094/0 (socket says 172.21.15.44:52094) 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbd4f0640 1 -- 172.21.15.44:0/188312505 learned_addr learned my addr 172.21.15.44:0/188312505 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/188312505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb81ef800 0x55eeb8fb0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbd4f0640 1 -- 172.21.15.44:0/188312505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb81eec00 msgr2=0x55eeb8fb0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbccef640 1 --2- 172.21.15.44:0/188312505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb81eec00 0x55eeb8fb0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbd4f0640 1 --2- 172.21.15.44:0/188312505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb81eec00 0x55eeb8fb0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbd4f0640 1 -- 172.21.15.44:0/188312505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb81ef800 msgr2=0x55eeb8fb0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbd4f0640 1 --2- 172.21.15.44:0/188312505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb81ef800 0x55eeb8fb0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbd4f0640 1 -- 172.21.15.44:0/188312505 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eeb8224780 con 0x55eeb81efc00 2024-04-02T13:26:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/188312505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb81ef800 0x55eeb8fb0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbd4f0640 1 --2- 172.21.15.44:0/188312505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb81efc00 0x55eeb8fb0b00 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x55eeb90113b0 tx=0x55eeb8f848a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40264ad3235ce264 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cba4ea640 1 -- 172.21.15.44:0/188312505 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55eeb8214680 con 0x55eeb81efc00 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cba4ea640 1 -- 172.21.15.44:0/188312505 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55eeb821aa80 con 0x55eeb81efc00 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cba4ea640 1 -- 172.21.15.44:0/188312505 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55eeb8214b60 con 0x55eeb81efc00 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbe541280 1 -- 172.21.15.44:0/188312505 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eeb82254a0 con 0x55eeb81efc00 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbe541280 1 -- 172.21.15.44:0/188312505 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55eeb8224000 con 0x55eeb81efc00 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.740+0000 7f9cbe541280 4 waiting for MDS map... 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.741+0000 7f9cba4ea640 1 -- 172.21.15.44:0/188312505 <== mon.0 v2:172.21.15.44:3300/0 4 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x55eeb90b4600 con 0x55eeb81efc00 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.741+0000 7f9cba4ea640 1 -- 172.21.15.44:0/188312505 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 62) ==== 2885+0+0 (secure 0 0 0) 0x55eeb8249180 con 0x55eeb81efc00 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.741+0000 7f9cbe541280 4 Got MDS map 62 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.741+0000 7f9cbe541280 10 main: JournalTool::main 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.741+0000 7f9cbe541280 4 main: JournalTool: connecting to RADOS... 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.741+0000 7f9cbe541280 1 Processor -- start 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.741+0000 7f9cbe541280 1 -- start start 2024-04-02T13:26:55.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbe541280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf400 0x55eeb8fb1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbe541280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb90bf800 0x55eeb8fb1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbe541280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb90bfc00 0x55eeb8fb1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbe541280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eeb8fd0600 con 0x55eeb90bf400 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbe541280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55eeb8fd0780 con 0x55eeb90bf800 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbe541280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55eeb8fd0900 con 0x55eeb90bfc00 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbc4ee640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf400 0x55eeb8fb1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbc4ee640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf400 0x55eeb8fb1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52100/0 (socket says 172.21.15.44:52100) 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbc4ee640 1 -- 172.21.15.44:0/506488734 learned_addr learned my addr 172.21.15.44:0/506488734 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbccef640 1 --2- 172.21.15.44:0/506488734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb90bf800 0x55eeb8fb1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/506488734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb90bfc00 0x55eeb8fb1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbc4ee640 1 -- 172.21.15.44:0/506488734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb90bfc00 msgr2=0x55eeb8fb1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/506488734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb90bfc00 0x55eeb8fb1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbc4ee640 1 -- 172.21.15.44:0/506488734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb90bf800 msgr2=0x55eeb8fb1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/506488734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb90bf800 0x55eeb8fb1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.742+0000 7f9cbc4ee640 1 -- 172.21.15.44:0/506488734 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eeb82243c0 con 0x55eeb90bf400 2024-04-02T13:26:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.743+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/506488734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf400 0x55eeb8fb1080 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x55eeb90d8690 tx=0x55eeb90dac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4dfde851797cc4a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.743+0000 7f9cb84e6640 1 -- 172.21.15.44:0/506488734 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55eeb8214680 con 0x55eeb90bf400 2024-04-02T13:26:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.743+0000 7f9cb84e6640 1 -- 172.21.15.44:0/506488734 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55eeb821afc0 con 0x55eeb90bf400 2024-04-02T13:26:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.743+0000 7f9cb84e6640 1 -- 172.21.15.44:0/506488734 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55eeb8214b60 con 0x55eeb90bf400 2024-04-02T13:26:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.743+0000 7f9cbe541280 1 -- 172.21.15.44:0/506488734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf400 msgr2=0x55eeb8fb1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.743+0000 7f9cbe541280 1 --2- 172.21.15.44:0/506488734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf400 0x55eeb8fb1080 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x55eeb90d8690 tx=0x55eeb90dac60 comp rx=0 tx=0).stop 2024-04-02T13:26:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.743+0000 7f9cbe541280 1 -- 172.21.15.44:0/506488734 shutdown_connections 2024-04-02T13:26:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.743+0000 7f9cbe541280 1 --2- 172.21.15.44:0/506488734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb90bfc00 0x55eeb8fb1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.743+0000 7f9cbe541280 1 --2- 172.21.15.44:0/506488734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb90bf800 0x55eeb8fb1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.743+0000 7f9cbe541280 1 --2- 172.21.15.44:0/506488734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf400 0x55eeb8fb1080 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.743+0000 7f9cbe541280 1 -- 172.21.15.44:0/506488734 >> 172.21.15.44:0/506488734 conn(0x55eeb90bf000 msgr2=0x55eeb82db000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.743+0000 7f9cbe541280 1 -- 172.21.15.44:0/506488734 shutdown_connections 2024-04-02T13:26:55.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.743+0000 7f9cbe541280 1 -- 172.21.15.44:0/506488734 wait complete. 2024-04-02T13:26:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.744+0000 7f9cbe541280 1 Processor -- start 2024-04-02T13:26:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.744+0000 7f9cbe541280 1 -- start start 2024-04-02T13:26:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.744+0000 7f9cbe541280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb90bf400 0x55eeb8fb1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.744+0000 7f9cbe541280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf800 0x55eeb8fb1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.744+0000 7f9cbe541280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb90bfc00 0x55eeb8fb1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.744+0000 7f9cbe541280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eeb8fd1b00 con 0x55eeb90bf800 2024-04-02T13:26:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.744+0000 7f9cbe541280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55eeb8fd1980 con 0x55eeb90bf400 2024-04-02T13:26:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.744+0000 7f9cbe541280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55eeb8fd1800 con 0x55eeb90bfc00 2024-04-02T13:26:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.744+0000 7f9cbccef640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf800 0x55eeb8fb1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.744+0000 7f9cbccef640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf800 0x55eeb8fb1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52108/0 (socket says 172.21.15.44:52108) 2024-04-02T13:26:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.744+0000 7f9cbccef640 1 -- 172.21.15.44:0/1665177105 learned_addr learned my addr 172.21.15.44:0/1665177105 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.744+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb90bf400 0x55eeb8fb1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.744+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb90bfc00 0x55eeb8fb1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.745+0000 7f9cbccef640 1 -- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb90bfc00 msgr2=0x55eeb8fb1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.745+0000 7f9cbccef640 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb90bfc00 0x55eeb8fb1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.745+0000 7f9cbccef640 1 -- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb90bf400 msgr2=0x55eeb8fb1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.745+0000 7f9cbccef640 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb90bf400 0x55eeb8fb1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.745+0000 7f9cbccef640 1 -- 172.21.15.44:0/1665177105 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eeb8224780 con 0x55eeb90bf800 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.745+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb90bf400 0x55eeb8fb1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.745+0000 7f9cbccef640 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf800 0x55eeb8fb1600 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x55eeb9010ab0 tx=0x55eeb8f829c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.745+0000 7f9cb6ce3640 1 -- 172.21.15.44:0/1665177105 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55eeb82149c0 con 0x55eeb90bf800 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.745+0000 7f9cb6ce3640 1 -- 172.21.15.44:0/1665177105 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55eeb821ae00 con 0x55eeb90bf800 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.745+0000 7f9cb6ce3640 1 -- 172.21.15.44:0/1665177105 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55eeb8214680 con 0x55eeb90bf800 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.745+0000 7f9cbe541280 1 -- 172.21.15.44:0/1665177105 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55eeb82254a0 con 0x55eeb90bf800 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.745+0000 7f9cbe541280 1 -- 172.21.15.44:0/1665177105 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eeb8224960 con 0x55eeb90bf800 2024-04-02T13:26:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.745+0000 7f9cbe541280 4 main: JournalTool: resolving pool 13 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.746+0000 7f9cb6ce3640 1 -- 172.21.15.44:0/1665177105 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x55eeb9136c00 con 0x55eeb90bf800 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.746+0000 7f9cb6ce3640 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55eeb8fe6c00 0x55eeb8fb2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.747+0000 7f9cb6ce3640 1 -- 172.21.15.44:0/1665177105 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x55eeb9134c00 con 0x55eeb90bf800 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.747+0000 7f9cbe541280 4 main: JournalTool: creating IoCtx.. 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.747+0000 7f9cbe541280 4 main: Executing for rank 0 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.747+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55eeb8fe6c00 0x55eeb8fb2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.747+0000 7f9cbe541280 1 -- start start 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.747+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55eeb8fe6c00 0x55eeb8fb2100 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x55eeb90d8690 tx=0x55eeb8f82a20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.747+0000 7f9cbe541280 1 Processor -- start 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbe541280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb8fe7400 0x55eeb8fb2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbe541280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb8fe7800 0x55eeb8fb2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbe541280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb8fe7c00 0x55eeb8fb3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbe541280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eeb8fd0600 con 0x55eeb8fe7400 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbe541280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55eeb8fd0780 con 0x55eeb8fe7800 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbe541280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55eeb8fd0a80 con 0x55eeb8fe7c00 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbd4f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb8fe7400 0x55eeb8fb2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbd4f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb8fe7400 0x55eeb8fb2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52114/0 (socket says 172.21.15.44:52114) 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbd4f0640 1 -- 172.21.15.44:0/910762921 learned_addr learned my addr 172.21.15.44:0/910762921 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbccef640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb8fe7800 0x55eeb8fb2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb8fe7c00 0x55eeb8fb3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbd4f0640 1 -- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb8fe7c00 msgr2=0x55eeb8fb3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbd4f0640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb8fe7c00 0x55eeb8fb3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbd4f0640 1 -- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb8fe7800 msgr2=0x55eeb8fb2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbd4f0640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb8fe7800 0x55eeb8fb2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbd4f0640 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eeb8225680 con 0x55eeb8fe7400 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.748+0000 7f9cbd4f0640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb8fe7400 0x55eeb8fb2680 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x55eeb8517710 tx=0x55eeb82a4de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b5401e04359652 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.749+0000 7f9cb3cdd640 1 -- 172.21.15.44:0/910762921 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55eeb8214820 con 0x55eeb8fe7400 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.749+0000 7f9cb3cdd640 1 -- 172.21.15.44:0/910762921 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55eeb821a1c0 con 0x55eeb8fe7400 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.749+0000 7f9cb3cdd640 1 -- 172.21.15.44:0/910762921 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55eeb84e4ea0 con 0x55eeb8fe7400 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.749+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eeb8225860 con 0x55eeb8fe7400 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.749+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55eeb8225a40 con 0x55eeb8fe7400 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.749+0000 7f9cbe541280 4 waiting for MDS map... 2024-04-02T13:26:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.749+0000 7f9cb3cdd640 1 -- 172.21.15.44:0/910762921 <== mon.0 v2:172.21.15.44:3300/0 4 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x55eeb85b4c00 con 0x55eeb8fe7400 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.749+0000 7f9cb3cdd640 1 -- 172.21.15.44:0/910762921 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 62) ==== 2885+0+0 (secure 0 0 0) 0x55eeb8249500 con 0x55eeb8fe7400 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.749+0000 7f9cbe541280 4 Got MDS map 62 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.749+0000 7f9cbe541280 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55eeb85cb800 0x55eeb8fb3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.749+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:1 13.f 13:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92) -- 0x55eeb85cbc00 con 0x55eeb85cb800 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.749+0000 7f9cbccef640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55eeb85cb800 0x55eeb8fb3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.750+0000 7f9cbccef640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55eeb85cb800 0x55eeb8fb3700 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=bd16d66d5856d34d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.752+0000 7f9cbccef640 1 -- 172.21.15.44:0/910762921 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x55eeb82f0000 con 0x55eeb85cb800 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.752+0000 7f9cbe541280 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55eeb85e6000 0x55eeb85ec000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.752+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:2 13.14 13:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92) -- 0x55eeb85e6400 con 0x55eeb85e6000 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.752+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55eeb85e6000 0x55eeb85ec000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.753+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55eeb85e6000 0x55eeb85ec000 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=703fbed391151f05 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.753+0000 7f9cbc4ee640 1 -- 172.21.15.44:0/910762921 <== osd.0 v2:172.21.15.44:6800/2303805525 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv19 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x55eeb82f0500 con 0x55eeb85e6000 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.753+0000 7f9cb24da640 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:3 13.14 13:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92) -- 0x55eeb85e7c00 con 0x55eeb85e6000 2024-04-02T13:26:55.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.753+0000 7f9cb24da640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x55eeb85f0000 0x55eeb85edb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.753+0000 7f9cb24da640 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:4 13.1b 13:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92) -- 0x55eeb85f0400 con 0x55eeb85f0000 2024-04-02T13:26:55.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.753+0000 7f9cbc4ee640 1 -- 172.21.15.44:0/910762921 <== osd.0 v2:172.21.15.44:6800/2303805525 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv16 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x55eeb82f0500 con 0x55eeb85e6000 2024-04-02T13:26:55.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.753+0000 7f9cbd4f0640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x55eeb85f0000 0x55eeb85edb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.753+0000 7f9cbd4f0640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x55eeb85f0000 0x55eeb85edb80 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=5f12e2292532049e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.756+0000 7f9cbd4f0640 1 -- 172.21.15.44:0/910762921 <== osd.4 v2:172.21.15.69:6816/1041350007 1 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55eeb82f0780 con 0x55eeb85f0000 2024-04-02T13:26:55.768 INFO:teuthology.orchestra.run.smithi044.stdout:old journal was 4208696~3054 2024-04-02T13:26:55.768 INFO:teuthology.orchestra.run.smithi044.stdout:new journal start will be 8388608 (4176858 bytes past old end) 2024-04-02T13:26:55.768 INFO:teuthology.orchestra.run.smithi044.stdout:writing journal head 2024-04-02T13:26:55.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.756+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:5 13.14 13:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x55eeb85e6800 con 0x55eeb85e6000 2024-04-02T13:26:55.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.757+0000 7f9cbc4ee640 1 -- 172.21.15.44:0/910762921 <== osd.0 v2:172.21.15.44:6800/2303805525 3 ==== osd_op_reply(5 200.00000000 [writefull 0~90] v92'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55eeb82f0500 con 0x55eeb85e6000 2024-04-02T13:26:55.769 INFO:teuthology.orchestra.run.smithi044.stdout:writing EResetJournal entry 2024-04-02T13:26:55.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.757+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:6 13.1b 13:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x55eeb85e6c00 con 0x55eeb85f0000 2024-04-02T13:26:55.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.757+0000 7f9cbe541280 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55eeb85e7000 0x55eeb85ec580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.757+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:7 13.b 13:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x55eeb85e7400 con 0x55eeb85e7000 2024-04-02T13:26:55.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.757+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:8 13.1f 13:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x55eeb85e7800 con 0x55eeb85cb800 2024-04-02T13:26:55.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.757+0000 7f9cbe541280 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55eeb85f0800 0x55eeb85ecb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.757+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:9 13.1a 13:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x55eeb85f0c00 con 0x55eeb85f0800 2024-04-02T13:26:55.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.757+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:10 13.1e 13:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x55eeb85f1000 con 0x55eeb85f0000 2024-04-02T13:26:55.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.757+0000 7f9cbe541280 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55eeb85f1400 0x55eeb85ed080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.757+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:11 13.1d 13:bec1251c:::200.00000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x55eeb85e6400 con 0x55eeb85f1400 2024-04-02T13:26:55.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.757+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55eeb85f0800 0x55eeb85ecb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.757+0000 7f9cbccef640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55eeb85e7000 0x55eeb85ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.757+0000 7f9cbd4f0640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55eeb85f1400 0x55eeb85ed080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.758+0000 7f9cbc4ee640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55eeb85f0800 0x55eeb85ecb00 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=7a0b6d3a203447d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.758+0000 7f9cbccef640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55eeb85e7000 0x55eeb85ec580 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=8a72ff04ddbfa001 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.758+0000 7f9cbd4f0640 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55eeb85f1400 0x55eeb85ed080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=8be607de26873b37 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.759+0000 7f9cbd4f0640 1 -- 172.21.15.44:0/910762921 <== osd.4 v2:172.21.15.69:6816/1041350007 2 ==== osd_op_reply(6 200.00000002 [delete] v92'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55eeb82f0780 con 0x55eeb85f0000 2024-04-02T13:26:55.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.759+0000 7f9cbc4ee640 1 -- 172.21.15.44:0/910762921 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(9 200.00000005 [delete] v92'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55eeb82f0500 con 0x55eeb85f0800 2024-04-02T13:26:55.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.759+0000 7f9cbccef640 1 -- 172.21.15.44:0/910762921 <== osd.6 v2:172.21.15.69:6802/2604158175 2 ==== osd_op_reply(8 200.00000004 [delete] v92'7 uv6 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55eeb82f0000 con 0x55eeb85cb800 2024-04-02T13:26:55.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.759+0000 7f9cbd4f0640 1 -- 172.21.15.44:0/910762921 <== osd.4 v2:172.21.15.69:6816/1041350007 3 ==== osd_op_reply(10 200.00000006 [delete] v92'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55eeb82f0780 con 0x55eeb85f0000 2024-04-02T13:26:55.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.759+0000 7f9cbccef640 1 -- 172.21.15.44:0/910762921 <== osd.5 v2:172.21.15.69:6800/2841767499 1 ==== osd_op_reply(7 200.00000003 [delete] v92'4 uv3 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55eeb82f0000 con 0x55eeb85e7000 2024-04-02T13:26:55.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.759+0000 7f9cb24da640 1 -- 172.21.15.44:0/910762921 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:12 13.1b 13:ddb4e4d7:::200.00000002:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x55eeb85f1800 con 0x55eeb85f0000 2024-04-02T13:26:55.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.759+0000 7f9cbd4f0640 1 -- 172.21.15.44:0/910762921 <== osd.7 v2:172.21.15.69:6824/1292475712 1 ==== osd_op_reply(11 200.00000007 [delete] v92'3 uv2 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55eeb82f0780 con 0x55eeb85f1400 2024-04-02T13:26:55.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbd4f0640 1 -- 172.21.15.44:0/910762921 <== osd.4 v2:172.21.15.69:6816/1041350007 4 ==== osd_op_reply(12 200.00000002 [write 0~48] v92'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55eeb82f0780 con 0x55eeb85f0000 2024-04-02T13:26:55.773 INFO:teuthology.orchestra.run.smithi044.stdout:done 2024-04-02T13:26:55.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55eeb85e6000 msgr2=0x55eeb85ec000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55eeb85e6000 0x55eeb85ec000 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55eeb85f0800 msgr2=0x55eeb85ecb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55eeb85f0800 0x55eeb85ecb00 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x55eeb85f0000 msgr2=0x55eeb85edb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x55eeb85f0000 0x55eeb85edb80 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55eeb85e7000 msgr2=0x55eeb85ec580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55eeb85e7000 0x55eeb85ec580 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55eeb85cb800 msgr2=0x55eeb8fb3700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55eeb85cb800 0x55eeb8fb3700 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55eeb85f1400 msgr2=0x55eeb85ed080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55eeb85f1400 0x55eeb85ed080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbc4ee640 1 -- 172.21.15.44:0/910762921 reap_dead start 2024-04-02T13:26:55.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb8fe7400 msgr2=0x55eeb8fb2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb8fe7400 0x55eeb8fb2680 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x55eeb8517710 tx=0x55eeb82a4de0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbc4ee640 1 -- 172.21.15.44:0/910762921 reap_dead start 2024-04-02T13:26:55.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 shutdown_connections 2024-04-02T13:26:55.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/910762921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb8fe7400 0x55eeb8fb2680 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 >> 172.21.15.44:0/910762921 conn(0x55eeb8fe7000 msgr2=0x55eeb82db800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:55.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 shutdown_connections 2024-04-02T13:26:55.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/910762921 wait complete. 2024-04-02T13:26:55.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55eeb8fe6c00 msgr2=0x55eeb8fb2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55eeb8fe6c00 0x55eeb8fb2100 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x55eeb90d8690 tx=0x55eeb8f82a20 comp rx=0 tx=0).stop 2024-04-02T13:26:55.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/1665177105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf800 msgr2=0x55eeb8fb1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf800 0x55eeb8fb1600 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x55eeb9010ab0 tx=0x55eeb8f829c0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/1665177105 shutdown_connections 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55eeb8fe6c00 0x55eeb8fb2100 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb90bfc00 0x55eeb8fb1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb90bf800 0x55eeb8fb1600 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 --2- 172.21.15.44:0/1665177105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb90bf400 0x55eeb8fb1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.761+0000 7f9cbe541280 1 -- 172.21.15.44:0/1665177105 >> 172.21.15.44:0/1665177105 conn(0x55eeb90bf000 msgr2=0x55eeb82db000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.762+0000 7f9cbe541280 1 -- 172.21.15.44:0/1665177105 shutdown_connections 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.762+0000 7f9cbe541280 1 -- 172.21.15.44:0/1665177105 wait complete. 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.762+0000 7f9cbe541280 1 -- 172.21.15.44:0/188312505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb81efc00 msgr2=0x55eeb8fb0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.762+0000 7f9cbe541280 1 --2- 172.21.15.44:0/188312505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb81efc00 0x55eeb8fb0b00 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x55eeb90113b0 tx=0x55eeb8f848a0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.762+0000 7f9cbe541280 1 -- 172.21.15.44:0/188312505 shutdown_connections 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.762+0000 7f9cbe541280 1 --2- 172.21.15.44:0/188312505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eeb81efc00 0x55eeb8fb0b00 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.762+0000 7f9cbe541280 1 --2- 172.21.15.44:0/188312505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55eeb81ef800 0x55eeb8fb0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.762+0000 7f9cbe541280 1 --2- 172.21.15.44:0/188312505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55eeb81eec00 0x55eeb8fb0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.762+0000 7f9cbe541280 1 -- 172.21.15.44:0/188312505 >> 172.21.15.44:0/188312505 conn(0x55eeb81ef000 msgr2=0x55eeb82da800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:55.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.762+0000 7f9cbe541280 1 -- 172.21.15.44:0/188312505 shutdown_connections 2024-04-02T13:26:55.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.762+0000 7f9cbe541280 1 -- 172.21.15.44:0/188312505 wait complete. 2024-04-02T13:26:55.779 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset'] in time 0:00:00.110763, result: old journal was 4208696~3054 new journal start will be 8388608 (4176858 bytes past old end) writing journal head writing EResetJournal entry done 2024-04-02T13:26:55.779 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-02T13:26:55.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.918+0000 7f765de22640 1 Processor -- start 2024-04-02T13:26:55.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.918+0000 7f765de22640 1 -- start start 2024-04-02T13:26:55.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.919+0000 7f765de22640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765806d700 0x7f765815e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.919+0000 7f765de22640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f765815ee70 0x7f76580d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.919+0000 7f765de22640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f765815fb50 con 0x7f765815e550 2024-04-02T13:26:55.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.919+0000 7f765de22640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f765806ca90 con 0x7f765815ee70 2024-04-02T13:26:55.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.919+0000 7f765de22640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f765815f590 con 0x7f765806d700 2024-04-02T13:26:55.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.919+0000 7f7656ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765806d700 0x7f765815e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.919+0000 7f7657fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f765815ee70 0x7f76580d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.919+0000 7f76577fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f765815e550 0x7f765806cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:43006/0 (socket says 172.21.15.44:43006) 2024-04-02T13:26:55.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.919+0000 7f76577fe640 1 -- 172.21.15.44:0/2030633371 learned_addr learned my addr 172.21.15.44:0/2030633371 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:55.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.919+0000 7f7656ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765806d700 0x7f765815e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:49664/0 (socket says 172.21.15.44:49664) 2024-04-02T13:26:55.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.920+0000 7f76567fc640 1 -- 172.21.15.44:0/2030633371 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2924484529 0 0) 0x7f765815fb50 con 0x7f765815e550 2024-04-02T13:26:55.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.920+0000 7f76567fc640 1 -- 172.21.15.44:0/2030633371 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7640003520 con 0x7f765815e550 2024-04-02T13:26:55.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.920+0000 7f7656ffd640 1 -- 172.21.15.44:0/2030633371 >> v2:172.21.15.69:3300/0 conn(0x7f765815ee70 msgr2=0x7f76580d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:55.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.920+0000 7f7656ffd640 1 --2- 172.21.15.44:0/2030633371 >> v2:172.21.15.69:3300/0 conn(0x7f765815ee70 0x7f76580d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.920+0000 7f7656ffd640 1 -- 172.21.15.44:0/2030633371 >> v1:172.21.15.69:6789/0 conn(0x7f765815e550 legacy=0x7f765806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.920+0000 7f7656ffd640 1 -- 172.21.15.44:0/2030633371 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765815f320 con 0x7f765806d700 2024-04-02T13:26:55.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.920+0000 7f7657fff640 1 --2- 172.21.15.44:0/2030633371 >> v2:172.21.15.69:3300/0 conn(0x7f765815ee70 0x7f76580d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:26:55.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.920+0000 7f7656ffd640 1 --2- 172.21.15.44:0/2030633371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765806d700 0x7f765815e930 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f764c0097a0 tx=0x7f764c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e58f663cbcc84098 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.921+0000 7f76567fc640 1 -- 172.21.15.44:0/2030633371 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c034070 con 0x7f765806d700 2024-04-02T13:26:55.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.921+0000 7f76567fc640 1 -- 172.21.15.44:0/2030633371 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f764c030610 con 0x7f765806d700 2024-04-02T13:26:55.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.921+0000 7f76567fc640 1 -- 172.21.15.44:0/2030633371 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c0309c0 con 0x7f765806d700 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.921+0000 7f765de22640 1 -- 172.21.15.44:0/2030633371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765806d700 msgr2=0x7f765815e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.921+0000 7f765de22640 1 --2- 172.21.15.44:0/2030633371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765806d700 0x7f765815e930 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f764c0097a0 tx=0x7f764c02b960 comp rx=0 tx=0).stop 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.921+0000 7f765de22640 1 -- 172.21.15.44:0/2030633371 shutdown_connections 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.921+0000 7f765de22640 1 --2- 172.21.15.44:0/2030633371 >> v2:172.21.15.69:3300/0 conn(0x7f765815ee70 0x7f76580d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.921+0000 7f765de22640 1 --2- 172.21.15.44:0/2030633371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765806d700 0x7f765815e930 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.921+0000 7f765de22640 1 -- 172.21.15.44:0/2030633371 >> 172.21.15.44:0/2030633371 conn(0x7f765809aa20 msgr2=0x7f765809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.921+0000 7f765de22640 1 -- 172.21.15.44:0/2030633371 shutdown_connections 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.921+0000 7f765de22640 1 -- 172.21.15.44:0/2030633371 wait complete. 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.922+0000 7f765de22640 1 Processor -- start 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.923+0000 7f765de22640 1 -- start start 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.923+0000 7f765de22640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f765806d700 0x7f76581f7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.923+0000 7f765de22640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765815e550 0x7f765814e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.923+0000 7f765de22640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f765815ee70 0x7f7658070620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.923+0000 7f765de22640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f76580d0590 con 0x7f765815ee70 2024-04-02T13:26:55.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.923+0000 7f765de22640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f76580d02b0 con 0x7f765806d700 2024-04-02T13:26:55.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.923+0000 7f765de22640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f76580cc4b0 con 0x7f765815e550 2024-04-02T13:26:55.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.923+0000 7f7657fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f765815ee70 0x7f7658070620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.923+0000 7f7657fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f765815ee70 0x7f7658070620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52120/0 (socket says 172.21.15.44:52120) 2024-04-02T13:26:55.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.923+0000 7f7657fff640 1 -- 172.21.15.44:0/569521660 learned_addr learned my addr 172.21.15.44:0/569521660 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:55.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f76577fe640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f765806d700 0x7f76581f7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f7656ffd640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765815e550 0x7f765814e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f7657fff640 1 -- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765815e550 msgr2=0x7f765814e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f7657fff640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765815e550 0x7f765814e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f7657fff640 1 -- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f765806d700 msgr2=0x7f76581f7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:55.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f7657fff640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f765806d700 0x7f76581f7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:55.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f7657fff640 1 -- 172.21.15.44:0/569521660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7658151140 con 0x7f765815ee70 2024-04-02T13:26:55.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f76577fe640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f765806d700 0x7f76581f7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f7657fff640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f765815ee70 0x7f7658070620 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f7650009e60 tx=0x7f765000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f763ffff640 1 -- 172.21.15.44:0/569521660 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650018030 con 0x7f765815ee70 2024-04-02T13:26:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f763ffff640 1 -- 172.21.15.44:0/569521660 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f765000cdf0 con 0x7f765815ee70 2024-04-02T13:26:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f765de22640 1 -- 172.21.15.44:0/569521660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7658151420 con 0x7f765815ee70 2024-04-02T13:26:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f763ffff640 1 -- 172.21.15.44:0/569521660 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650003bb0 con 0x7f765815ee70 2024-04-02T13:26:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.924+0000 7f765de22640 1 -- 172.21.15.44:0/569521660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76581519e0 con 0x7f765815ee70 2024-04-02T13:26:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.925+0000 7f765de22640 1 -- 172.21.15.44:0/569521660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76580684f0 con 0x7f765815ee70 2024-04-02T13:26:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.926+0000 7f763ffff640 1 -- 172.21.15.44:0/569521660 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f765001b070 con 0x7f765815ee70 2024-04-02T13:26:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.926+0000 7f763ffff640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f764406c8f0 0x7f764406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.926+0000 7f763ffff640 1 -- 172.21.15.44:0/569521660 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7f765008c6b0 con 0x7f765815ee70 2024-04-02T13:26:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.926+0000 7f76577fe640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f764406c8f0 0x7f764406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.927+0000 7f76577fe640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f764406c8f0 0x7f764406edb0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7648007fd0 tx=0x7f7648009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:55.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:55.930+0000 7f763ffff640 1 -- 172.21.15.44:0/569521660 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7650010040 con 0x7f765815ee70 2024-04-02T13:26:56.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.158+0000 7f765de22640 1 -- 172.21.15.44:0/569521660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f76580cccf0 con 0x7f765815ee70 2024-04-02T13:26:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.592+0000 7f763ffff640 1 -- 172.21.15.44:0/569521660 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v63) ==== 162+0+0 (secure 0 0 0) 0x7f765008ae20 con 0x7f765815ee70 2024-04-02T13:26:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-02T13:26:56.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.594+0000 7f765de22640 1 -- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f764406c8f0 msgr2=0x7f764406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:56.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.594+0000 7f765de22640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f764406c8f0 0x7f764406edb0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7648007fd0 tx=0x7f7648009040 comp rx=0 tx=0).stop 2024-04-02T13:26:56.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.594+0000 7f765de22640 1 -- 172.21.15.44:0/569521660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f765815ee70 msgr2=0x7f7658070620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:56.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.594+0000 7f765de22640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f765815ee70 0x7f7658070620 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f7650009e60 tx=0x7f765000c490 comp rx=0 tx=0).stop 2024-04-02T13:26:56.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.594+0000 7f765de22640 1 -- 172.21.15.44:0/569521660 shutdown_connections 2024-04-02T13:26:56.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.594+0000 7f765de22640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f764406c8f0 0x7f764406edb0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:56.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.594+0000 7f765de22640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f765815ee70 0x7f7658070620 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:56.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.594+0000 7f765de22640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765815e550 0x7f765814e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:56.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.594+0000 7f765de22640 1 --2- 172.21.15.44:0/569521660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f765806d700 0x7f76581f7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:56.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.594+0000 7f765de22640 1 -- 172.21.15.44:0/569521660 >> 172.21.15.44:0/569521660 conn(0x7f765809aa20 msgr2=0x7f76580ced60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:56.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.594+0000 7f765de22640 1 -- 172.21.15.44:0/569521660 shutdown_connections 2024-04-02T13:26:56.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.594+0000 7f765de22640 1 -- 172.21.15.44:0/569521660 wait complete. 2024-04-02T13:26:56.610 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:26:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.710+0000 7f2f3342a640 1 Processor -- start 2024-04-02T13:26:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.710+0000 7f2f3342a640 1 -- start start 2024-04-02T13:26:56.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.711+0000 7f2f3342a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f2c15e550 0x7f2f2c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:56.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.711+0000 7f2f3342a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2f2c06d530 0x7f2f2c15e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:56.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.711+0000 7f2f3342a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f2c15fb50 con 0x7f2f2c15ee70 2024-04-02T13:26:56.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.711+0000 7f2f3342a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2f2c06d970 con 0x7f2f2c06d530 2024-04-02T13:26:56.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.711+0000 7f2f3342a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2f2c15f300 con 0x7f2f2c15e550 2024-04-02T13:26:56.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.711+0000 7f2f319c7640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2f2c15ee70 0x7f2f2c0ce6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:43018/0 (socket says 172.21.15.44:43018) 2024-04-02T13:26:56.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.711+0000 7f2f319c7640 1 -- 172.21.15.44:0/3952713133 learned_addr learned my addr 172.21.15.44:0/3952713133 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:56.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.711+0000 7f2f309c5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2f2c06d530 0x7f2f2c15e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.711+0000 7f2f311c6640 1 --2- 172.21.15.44:0/3952713133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f2c15e550 0x7f2f2c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.712+0000 7f2f2bfff640 1 -- 172.21.15.44:0/3952713133 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1862984472 0 0) 0x7f2f2c15fb50 con 0x7f2f2c15ee70 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.712+0000 7f2f2bfff640 1 -- 172.21.15.44:0/3952713133 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f14003520 con 0x7f2f2c15ee70 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.712+0000 7f2f311c6640 1 -- 172.21.15.44:0/3952713133 >> v2:172.21.15.69:3300/0 conn(0x7f2f2c06d530 msgr2=0x7f2f2c15e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.712+0000 7f2f311c6640 1 --2- 172.21.15.44:0/3952713133 >> v2:172.21.15.69:3300/0 conn(0x7f2f2c06d530 0x7f2f2c15e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.712+0000 7f2f311c6640 1 -- 172.21.15.44:0/3952713133 >> v1:172.21.15.69:6789/0 conn(0x7f2f2c15ee70 legacy=0x7f2f2c0ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.712+0000 7f2f311c6640 1 -- 172.21.15.44:0/3952713133 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2c0d1ac0 con 0x7f2f2c15e550 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.712+0000 7f2f309c5640 1 --2- 172.21.15.44:0/3952713133 >> v2:172.21.15.69:3300/0 conn(0x7f2f2c06d530 0x7f2f2c15e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.713+0000 7f2f311c6640 1 --2- 172.21.15.44:0/3952713133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f2c15e550 0x7f2f2c06cff0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2f240097a0 tx=0x7f2f2402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9011ee0952000a92 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.713+0000 7f2f2bfff640 1 -- 172.21.15.44:0/3952713133 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f24034070 con 0x7f2f2c15e550 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.713+0000 7f2f2bfff640 1 -- 172.21.15.44:0/3952713133 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f24030610 con 0x7f2f2c15e550 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.713+0000 7f2f2bfff640 1 -- 172.21.15.44:0/3952713133 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f240309e0 con 0x7f2f2c15e550 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.713+0000 7f2f3342a640 1 -- 172.21.15.44:0/3952713133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f2c15e550 msgr2=0x7f2f2c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.713+0000 7f2f3342a640 1 --2- 172.21.15.44:0/3952713133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f2c15e550 0x7f2f2c06cff0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2f240097a0 tx=0x7f2f2402b960 comp rx=0 tx=0).stop 2024-04-02T13:26:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.713+0000 7f2f3342a640 1 -- 172.21.15.44:0/3952713133 shutdown_connections 2024-04-02T13:26:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.713+0000 7f2f3342a640 1 --2- 172.21.15.44:0/3952713133 >> v2:172.21.15.69:3300/0 conn(0x7f2f2c06d530 0x7f2f2c15e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.713+0000 7f2f3342a640 1 --2- 172.21.15.44:0/3952713133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f2c15e550 0x7f2f2c06cff0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.713+0000 7f2f3342a640 1 -- 172.21.15.44:0/3952713133 >> 172.21.15.44:0/3952713133 conn(0x7f2f2c09aa20 msgr2=0x7f2f2c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.713+0000 7f2f3342a640 1 -- 172.21.15.44:0/3952713133 shutdown_connections 2024-04-02T13:26:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.713+0000 7f2f3342a640 1 -- 172.21.15.44:0/3952713133 wait complete. 2024-04-02T13:26:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.714+0000 7f2f3342a640 1 Processor -- start 2024-04-02T13:26:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.715+0000 7f2f3342a640 1 -- start start 2024-04-02T13:26:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.715+0000 7f2f3342a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f2c06d530 0x7f2f2c0b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.715+0000 7f2f3342a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f2c15e550 0x7f2f2c09be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.715+0000 7f2f3342a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f2c15ee70 0x7f2f2c09e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.715+0000 7f2f3342a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c15f870 con 0x7f2f2c15ee70 2024-04-02T13:26:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.715+0000 7f2f3342a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c15f590 con 0x7f2f2c06d530 2024-04-02T13:26:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.715+0000 7f2f3342a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2f2c0d02b0 con 0x7f2f2c15e550 2024-04-02T13:26:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.715+0000 7f2f319c7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f2c15ee70 0x7f2f2c09e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.715+0000 7f2f319c7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f2c15ee70 0x7f2f2c09e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52134/0 (socket says 172.21.15.44:52134) 2024-04-02T13:26:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.715+0000 7f2f319c7640 1 -- 172.21.15.44:0/2396091326 learned_addr learned my addr 172.21.15.44:0/2396091326 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.716+0000 7f2f311c6640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f2c06d530 0x7f2f2c0b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.716+0000 7f2f309c5640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f2c15e550 0x7f2f2c09be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.716+0000 7f2f319c7640 1 -- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f2c15e550 msgr2=0x7f2f2c09be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.716+0000 7f2f319c7640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f2c15e550 0x7f2f2c09be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.716+0000 7f2f319c7640 1 -- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f2c06d530 msgr2=0x7f2f2c0b2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.716+0000 7f2f319c7640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f2c06d530 0x7f2f2c0b2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.716+0000 7f2f319c7640 1 -- 172.21.15.44:0/2396091326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2c15c970 con 0x7f2f2c15ee70 2024-04-02T13:26:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.716+0000 7f2f319c7640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f2c15ee70 0x7f2f2c09e870 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00caa0 tx=0x7f2f1c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.716+0000 7f2f29ffb640 1 -- 172.21.15.44:0/2396091326 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c017070 con 0x7f2f2c15ee70 2024-04-02T13:26:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.716+0000 7f2f29ffb640 1 -- 172.21.15.44:0/2396091326 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f1c00ac00 con 0x7f2f2c15ee70 2024-04-02T13:26:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.716+0000 7f2f29ffb640 1 -- 172.21.15.44:0/2396091326 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c012400 con 0x7f2f2c15ee70 2024-04-02T13:26:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.716+0000 7f2f3342a640 1 -- 172.21.15.44:0/2396091326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2c0b6d30 con 0x7f2f2c15ee70 2024-04-02T13:26:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.716+0000 7f2f3342a640 1 -- 172.21.15.44:0/2396091326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f2c0b72f0 con 0x7f2f2c15ee70 2024-04-02T13:26:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.717+0000 7f2f3342a640 1 -- 172.21.15.44:0/2396091326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f2c0684f0 con 0x7f2f2c15ee70 2024-04-02T13:26:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.718+0000 7f2f29ffb640 1 -- 172.21.15.44:0/2396091326 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2f1c01a070 con 0x7f2f2c15ee70 2024-04-02T13:26:56.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.718+0000 7f2f29ffb640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f1806c920 0x7f2f1806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:56.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.718+0000 7f2f29ffb640 1 -- 172.21.15.44:0/2396091326 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7f2f1c08ba50 con 0x7f2f2c15ee70 2024-04-02T13:26:56.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.718+0000 7f2f311c6640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f1806c920 0x7f2f1806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:56.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.719+0000 7f2f311c6640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f1806c920 0x7f2f1806ede0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2f24036000 tx=0x7f2f2403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:56.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.722+0000 7f2f29ffb640 1 -- 172.21.15.44:0/2396091326 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2f2c15c970 con 0x7f2f2c15ee70 2024-04-02T13:26:56.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.945+0000 7f2f3342a640 1 -- 172.21.15.44:0/2396091326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f2c06d180 con 0x7f2f2c15ee70 2024-04-02T13:26:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.946+0000 7f2f29ffb640 1 -- 172.21.15.44:0/2396091326 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5174 (secure 0 0 0) 0x7f2f1c059d30 con 0x7f2f2c15ee70 2024-04-02T13:26:56.948 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:56.948 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:26:56.594635+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8073":{"gid":8073,"name":"a","rank":0,"incarnation":64,"state":"up:replay","state_seq":1,"addr":"172.21.15.44:6833/2250279975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2250279975},{"type":"v1","addr":"172.21.15.44:6833","nonce":2250279975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-04-02T13:26:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 64 2024-04-02T13:26:56.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.948+0000 7f2f3342a640 1 -- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f1806c920 msgr2=0x7f2f1806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:56.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.948+0000 7f2f3342a640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f1806c920 0x7f2f1806ede0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2f24036000 tx=0x7f2f2403b040 comp rx=0 tx=0).stop 2024-04-02T13:26:56.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.948+0000 7f2f3342a640 1 -- 172.21.15.44:0/2396091326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f2c15ee70 msgr2=0x7f2f2c09e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:56.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.948+0000 7f2f3342a640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f2c15ee70 0x7f2f2c09e870 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00caa0 tx=0x7f2f1c00cf60 comp rx=0 tx=0).stop 2024-04-02T13:26:56.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.948+0000 7f2f3342a640 1 -- 172.21.15.44:0/2396091326 shutdown_connections 2024-04-02T13:26:56.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.948+0000 7f2f3342a640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f1806c920 0x7f2f1806ede0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:56.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.948+0000 7f2f3342a640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f2c15ee70 0x7f2f2c09e870 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:56.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.948+0000 7f2f3342a640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f2c15e550 0x7f2f2c09be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:56.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.948+0000 7f2f3342a640 1 --2- 172.21.15.44:0/2396091326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f2c06d530 0x7f2f2c0b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:56.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.948+0000 7f2f3342a640 1 -- 172.21.15.44:0/2396091326 >> 172.21.15.44:0/2396091326 conn(0x7f2f2c09aa20 msgr2=0x7f2f2c0d09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:56.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.948+0000 7f2f3342a640 1 -- 172.21.15.44:0/2396091326 shutdown_connections 2024-04-02T13:26:56.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:56.948+0000 7f2f3342a640 1 -- 172.21.15.44:0/2396091326 wait complete. 2024-04-02T13:26:57.364 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:26:43.025860+0000', 'modified': '2024-04-02T13:26:56.594635+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 92, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8073}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8073': {'gid': 8073, 'name': 'a', 'rank': 0, 'incarnation': 64, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.44:6833/2250279975', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6832', 'nonce': 2250279975}, {'type': 'v1', 'addr': '172.21.15.44:6833', 'nonce': 2250279975}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-02T13:26:57.364 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8073:up:replay 2024-04-02T13:26:58.365 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:26:58.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.477+0000 7fd98f89a640 1 Processor -- start 2024-04-02T13:26:58.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.477+0000 7fd98f89a640 1 -- start start 2024-04-02T13:26:58.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.478+0000 7fd98f89a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9880d0a30 0x7fd98806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:58.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.478+0000 7fd98f89a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9880d0e10 0x7fd9880cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:58.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.478+0000 7fd98f89a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd98806c1d0 con 0x7fd9880cfe30 2024-04-02T13:26:58.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.478+0000 7fd98f89a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd98806cb00 con 0x7fd9880d0a30 2024-04-02T13:26:58.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.478+0000 7fd98f89a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd98806c4b0 con 0x7fd9880d0e10 2024-04-02T13:26:58.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.478+0000 7fd98ce35640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9880d0e10 0x7fd9880cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:58.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.478+0000 7fd98d636640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9880d0a30 0x7fd98806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:58.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.478+0000 7fd98de37640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd9880cfe30 0x7fd9880d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:43026/0 (socket says 172.21.15.44:43026) 2024-04-02T13:26:58.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.478+0000 7fd98de37640 1 -- 172.21.15.44:0/3931381562 learned_addr learned my addr 172.21.15.44:0/3931381562 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:26:58.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.479+0000 7fd98d636640 1 -- 172.21.15.44:0/3931381562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9880d0e10 msgr2=0x7fd9880cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:26:58.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.479+0000 7fd98d636640 1 --2- 172.21.15.44:0/3931381562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9880d0e10 0x7fd9880cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:58.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.479+0000 7fd98d636640 1 -- 172.21.15.44:0/3931381562 >> v1:172.21.15.69:6789/0 conn(0x7fd9880cfe30 legacy=0x7fd9880d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:58.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.479+0000 7fd98d636640 1 -- 172.21.15.44:0/3931381562 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98806c790 con 0x7fd9880d0a30 2024-04-02T13:26:58.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.479+0000 7fd97ffff640 1 -- 172.21.15.44:0/3931381562 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1987159886 0 0) 0x7fd98806c1d0 con 0x7fd9880cfe30 2024-04-02T13:26:58.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.479+0000 7fd98ce35640 1 --2- 172.21.15.44:0/3931381562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9880d0e10 0x7fd9880cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:26:58.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.479+0000 7fd98d636640 1 --2- 172.21.15.44:0/3931381562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9880d0a30 0x7fd98806cf90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd9840097a0 tx=0x7fd98402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5a2a996bdd118bf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:58.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.480+0000 7fd97ffff640 1 -- 172.21.15.44:0/3931381562 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9840315a0 con 0x7fd9880d0a30 2024-04-02T13:26:58.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.480+0000 7fd97ffff640 1 -- 172.21.15.44:0/3931381562 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd984037070 con 0x7fd9880d0a30 2024-04-02T13:26:58.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.480+0000 7fd97ffff640 1 -- 172.21.15.44:0/3931381562 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984031960 con 0x7fd9880d0a30 2024-04-02T13:26:58.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.480+0000 7fd98f89a640 1 -- 172.21.15.44:0/3931381562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9880d0a30 msgr2=0x7fd98806cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:58.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.480+0000 7fd98f89a640 1 --2- 172.21.15.44:0/3931381562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9880d0a30 0x7fd98806cf90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd9840097a0 tx=0x7fd98402b840 comp rx=0 tx=0).stop 2024-04-02T13:26:58.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.480+0000 7fd98f89a640 1 -- 172.21.15.44:0/3931381562 shutdown_connections 2024-04-02T13:26:58.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.480+0000 7fd98f89a640 1 --2- 172.21.15.44:0/3931381562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9880d0e10 0x7fd9880cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:58.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.480+0000 7fd98f89a640 1 --2- 172.21.15.44:0/3931381562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9880d0a30 0x7fd98806cf90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:58.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.480+0000 7fd98f89a640 1 -- 172.21.15.44:0/3931381562 >> 172.21.15.44:0/3931381562 conn(0x7fd98809d470 msgr2=0x7fd9880d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:58.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.480+0000 7fd98f89a640 1 -- 172.21.15.44:0/3931381562 shutdown_connections 2024-04-02T13:26:58.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.480+0000 7fd98f89a640 1 -- 172.21.15.44:0/3931381562 wait complete. 2024-04-02T13:26:58.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.481+0000 7fd98f89a640 1 Processor -- start 2024-04-02T13:26:58.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.482+0000 7fd98f89a640 1 -- start start 2024-04-02T13:26:58.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.482+0000 7fd98f89a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9880cfe30 0x7fd9881578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:58.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.482+0000 7fd98f89a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9880d0a30 0x7fd9880b8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:58.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.482+0000 7fd98f89a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9880d0e10 0x7fd9880b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:58.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.482+0000 7fd98f89a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd98815e870 con 0x7fd9880cfe30 2024-04-02T13:26:58.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.482+0000 7fd98f89a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd98815e6f0 con 0x7fd9880d0a30 2024-04-02T13:26:58.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.482+0000 7fd98f89a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd9880d4da0 con 0x7fd9880d0e10 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.482+0000 7fd98d636640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9880cfe30 0x7fd9881578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.482+0000 7fd98d636640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9880cfe30 0x7fd9881578c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52148/0 (socket says 172.21.15.44:52148) 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.482+0000 7fd98d636640 1 -- 172.21.15.44:0/2680378467 learned_addr learned my addr 172.21.15.44:0/2680378467 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd98ce35640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9880d0a30 0x7fd9880b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd98de37640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9880d0e10 0x7fd9880b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd98d636640 1 -- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9880d0e10 msgr2=0x7fd9880b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd98d636640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9880d0e10 0x7fd9880b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd98d636640 1 -- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9880d0a30 msgr2=0x7fd9880b8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd98d636640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9880d0a30 0x7fd9880b8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd98d636640 1 -- 172.21.15.44:0/2680378467 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd988157e00 con 0x7fd9880cfe30 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd98de37640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9880d0e10 0x7fd9880b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd98d636640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9880cfe30 0x7fd9881578c0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fd984034040 tx=0x7fd9840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd97dffb640 1 -- 172.21.15.44:0/2680378467 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984032630 con 0x7fd9880cfe30 2024-04-02T13:26:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd97dffb640 1 -- 172.21.15.44:0/2680378467 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9840327d0 con 0x7fd9880cfe30 2024-04-02T13:26:58.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd98f89a640 1 -- 172.21.15.44:0/2680378467 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9880b9790 con 0x7fd9880cfe30 2024-04-02T13:26:58.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd97dffb640 1 -- 172.21.15.44:0/2680378467 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984032bc0 con 0x7fd9880cfe30 2024-04-02T13:26:58.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.483+0000 7fd98f89a640 1 -- 172.21.15.44:0/2680378467 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9880b9d50 con 0x7fd9880cfe30 2024-04-02T13:26:58.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.484+0000 7fd98f89a640 1 -- 172.21.15.44:0/2680378467 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9880684f0 con 0x7fd9880cfe30 2024-04-02T13:26:58.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.485+0000 7fd97dffb640 1 -- 172.21.15.44:0/2680378467 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd98403b3f0 con 0x7fd9880cfe30 2024-04-02T13:26:58.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.485+0000 7fd97dffb640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd96c06c8f0 0x7fd96c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:26:58.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.485+0000 7fd97dffb640 1 -- 172.21.15.44:0/2680378467 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7fd98402f030 con 0x7fd9880cfe30 2024-04-02T13:26:58.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.486+0000 7fd98ce35640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd96c06c8f0 0x7fd96c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:26:58.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.486+0000 7fd98ce35640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd96c06c8f0 0x7fd96c06edb0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd978009560 tx=0x7fd978009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:26:58.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.490+0000 7fd97dffb640 1 -- 172.21.15.44:0/2680378467 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd984040070 con 0x7fd9880cfe30 2024-04-02T13:26:58.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.717+0000 7fd98f89a640 1 -- 172.21.15.44:0/2680378467 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9880cef00 con 0x7fd9880cfe30 2024-04-02T13:26:58.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.718+0000 7fd97dffb640 1 -- 172.21.15.44:0/2680378467 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+5174 (secure 0 0 0) 0x7fd98407bf40 con 0x7fd9880cfe30 2024-04-02T13:26:58.721 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:26:58.721 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":66,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:26:57.599923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8073":{"gid":8073,"name":"a","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.44:6833/2250279975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2250279975},{"type":"v1","addr":"172.21.15.44:6833","nonce":2250279975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-04-02T13:26:58.721 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 66 2024-04-02T13:26:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.720+0000 7fd98f89a640 1 -- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd96c06c8f0 msgr2=0x7fd96c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.720+0000 7fd98f89a640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd96c06c8f0 0x7fd96c06edb0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd978009560 tx=0x7fd978009190 comp rx=0 tx=0).stop 2024-04-02T13:26:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.720+0000 7fd98f89a640 1 -- 172.21.15.44:0/2680378467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9880cfe30 msgr2=0x7fd9881578c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:26:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.720+0000 7fd98f89a640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9880cfe30 0x7fd9881578c0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fd984034040 tx=0x7fd9840046c0 comp rx=0 tx=0).stop 2024-04-02T13:26:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.720+0000 7fd98f89a640 1 -- 172.21.15.44:0/2680378467 shutdown_connections 2024-04-02T13:26:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.720+0000 7fd98f89a640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd96c06c8f0 0x7fd96c06edb0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.720+0000 7fd98f89a640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9880d0e10 0x7fd9880b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.720+0000 7fd98f89a640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9880d0a30 0x7fd9880b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.720+0000 7fd98f89a640 1 --2- 172.21.15.44:0/2680378467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9880cfe30 0x7fd9881578c0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:26:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.720+0000 7fd98f89a640 1 -- 172.21.15.44:0/2680378467 >> 172.21.15.44:0/2680378467 conn(0x7fd98809d470 msgr2=0x7fd9880d3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:26:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.720+0000 7fd98f89a640 1 -- 172.21.15.44:0/2680378467 shutdown_connections 2024-04-02T13:26:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:26:58.720+0000 7fd98f89a640 1 -- 172.21.15.44:0/2680378467 wait complete. 2024-04-02T13:26:59.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:26:43.025860+0000', 'modified': '2024-04-02T13:26:57.599923+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 92, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8073}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8073': {'gid': 8073, 'name': 'a', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.44:6833/2250279975', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6832', 'nonce': 2250279975}, {'type': 'v1', 'addr': '172.21.15.44:6833', 'nonce': 2250279975}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-02T13:26:59.136 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8073:up:rejoin 2024-04-02T13:27:00.137 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:27:00.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.248+0000 7f8a71167640 1 Processor -- start 2024-04-02T13:27:00.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.248+0000 7f8a71167640 1 -- start start 2024-04-02T13:27:00.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.249+0000 7f8a71167640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a6c0d09f0 0x7f8a6c06d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:00.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.249+0000 7f8a71167640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8a6c06ece0 0x7f8a6c0acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:00.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.249+0000 7f8a71167640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a6c06dc40 con 0x7f8a6c0cece0 2024-04-02T13:27:00.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.249+0000 7f8a71167640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f8a6c06df20 con 0x7f8a6c06ece0 2024-04-02T13:27:00.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.249+0000 7f8a71167640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8a6c06e200 con 0x7f8a6c0d09f0 2024-04-02T13:27:00.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.249+0000 7f8a6b59e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8a6c06ece0 0x7f8a6c0acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:00.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.249+0000 7f8a6ad9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f8a6c0cece0 0x7f8a6c0cf0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37658/0 (socket says 172.21.15.44:37658) 2024-04-02T13:27:00.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.249+0000 7f8a6ad9d640 1 -- 172.21.15.44:0/429267408 learned_addr learned my addr 172.21.15.44:0/429267408 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:27:00.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.249+0000 7f8a63fff640 1 --2- 172.21.15.44:0/429267408 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a6c0d09f0 0x7f8a6c06d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:00.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.249+0000 7f8a6b59e640 1 -- 172.21.15.44:0/429267408 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a6c0d09f0 msgr2=0x7f8a6c06d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:00.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.249+0000 7f8a6b59e640 1 --2- 172.21.15.44:0/429267408 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a6c0d09f0 0x7f8a6c06d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:00.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.249+0000 7f8a6b59e640 1 -- 172.21.15.44:0/429267408 >> v1:172.21.15.44:6789/0 conn(0x7f8a6c0cece0 legacy=0x7f8a6c0cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:00.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.249+0000 7f8a6b59e640 1 -- 172.21.15.44:0/429267408 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6c06e4e0 con 0x7f8a6c06ece0 2024-04-02T13:27:00.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.250+0000 7f8a6b59e640 1 --2- 172.21.15.44:0/429267408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a6c06ece0 0x7f8a6c0acd50 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f8a64009d90 tx=0x7f8a6402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c508a70c620eae5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:00.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.250+0000 7f8a6a59c640 1 -- 172.21.15.44:0/429267408 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64039070 con 0x7f8a6c06ece0 2024-04-02T13:27:00.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.250+0000 7f8a6a59c640 1 -- 172.21.15.44:0/429267408 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a6402cc50 con 0x7f8a6c06ece0 2024-04-02T13:27:00.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.250+0000 7f8a6a59c640 1 -- 172.21.15.44:0/429267408 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64034440 con 0x7f8a6c06ece0 2024-04-02T13:27:00.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.251+0000 7f8a71167640 1 -- 172.21.15.44:0/429267408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a6c06ece0 msgr2=0x7f8a6c0acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:00.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.251+0000 7f8a71167640 1 --2- 172.21.15.44:0/429267408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a6c06ece0 0x7f8a6c0acd50 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f8a64009d90 tx=0x7f8a6402bea0 comp rx=0 tx=0).stop 2024-04-02T13:27:00.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.251+0000 7f8a71167640 1 -- 172.21.15.44:0/429267408 shutdown_connections 2024-04-02T13:27:00.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.251+0000 7f8a71167640 1 --2- 172.21.15.44:0/429267408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a6c06ece0 0x7f8a6c0acd50 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:00.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.251+0000 7f8a71167640 1 --2- 172.21.15.44:0/429267408 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a6c0d09f0 0x7f8a6c06d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:00.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.251+0000 7f8a71167640 1 -- 172.21.15.44:0/429267408 >> 172.21.15.44:0/429267408 conn(0x7f8a6c0cfcd0 msgr2=0x7f8a6c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:00.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.251+0000 7f8a71167640 1 -- 172.21.15.44:0/429267408 shutdown_connections 2024-04-02T13:27:00.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.251+0000 7f8a71167640 1 -- 172.21.15.44:0/429267408 wait complete. 2024-04-02T13:27:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.252+0000 7f8a71167640 1 Processor -- start 2024-04-02T13:27:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.252+0000 7f8a71167640 1 -- start start 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a71167640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a6c06ece0 0x7f8a6c15dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a71167640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a6c0cece0 0x7f8a6c15cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a71167640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a6c0d09f0 0x7f8a6c15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a71167640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c06d990 con 0x7f8a6c06ece0 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a71167640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c06d810 con 0x7f8a6c0d09f0 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a71167640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8a6c15b220 con 0x7f8a6c0cece0 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a6ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a6c06ece0 0x7f8a6c15dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a6ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a6c06ece0 0x7f8a6c15dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52168/0 (socket says 172.21.15.44:52168) 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a6ad9d640 1 -- 172.21.15.44:0/2950774534 learned_addr learned my addr 172.21.15.44:0/2950774534 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a6b59e640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a6c0d09f0 0x7f8a6c15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a6ad9d640 1 -- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a6c0cece0 msgr2=0x7f8a6c15cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a63fff640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a6c0cece0 0x7f8a6c15cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a6ad9d640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a6c0cece0 0x7f8a6c15cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a6ad9d640 1 -- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a6c0d09f0 msgr2=0x7f8a6c15b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a6ad9d640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a6c0d09f0 0x7f8a6c15b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a6ad9d640 1 -- 172.21.15.44:0/2950774534 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6c15e350 con 0x7f8a6c06ece0 2024-04-02T13:27:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a6b59e640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a6c0d09f0 0x7f8a6c15b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:27:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.253+0000 7f8a6ad9d640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a6c06ece0 0x7f8a6c15dc50 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00c400 tx=0x7f8a5c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.254+0000 7f8a637fe640 1 -- 172.21.15.44:0/2950774534 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c017070 con 0x7f8a6c06ece0 2024-04-02T13:27:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.254+0000 7f8a637fe640 1 -- 172.21.15.44:0/2950774534 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a5c005ce0 con 0x7f8a6c06ece0 2024-04-02T13:27:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.254+0000 7f8a71167640 1 -- 172.21.15.44:0/2950774534 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6c143f40 con 0x7f8a6c06ece0 2024-04-02T13:27:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.254+0000 7f8a637fe640 1 -- 172.21.15.44:0/2950774534 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c0072f0 con 0x7f8a6c06ece0 2024-04-02T13:27:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.254+0000 7f8a71167640 1 -- 172.21.15.44:0/2950774534 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a6c144500 con 0x7f8a6c06ece0 2024-04-02T13:27:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.255+0000 7f8a71167640 1 -- 172.21.15.44:0/2950774534 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a6c0684f0 con 0x7f8a6c06ece0 2024-04-02T13:27:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.255+0000 7f8a637fe640 1 -- 172.21.15.44:0/2950774534 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8a5c01a070 con 0x7f8a6c06ece0 2024-04-02T13:27:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.256+0000 7f8a637fe640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8a4c06c8f0 0x7f8a4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.256+0000 7f8a637fe640 1 -- 172.21.15.44:0/2950774534 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7f8a5c08b120 con 0x7f8a6c06ece0 2024-04-02T13:27:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.256+0000 7f8a63fff640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8a4c06c8f0 0x7f8a4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.257+0000 7f8a63fff640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8a4c06c8f0 0x7f8a4c06edb0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8a54006fd0 tx=0x7f8a54008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:00.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.260+0000 7f8a637fe640 1 -- 172.21.15.44:0/2950774534 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8a5c059370 con 0x7f8a6c06ece0 2024-04-02T13:27:00.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.484+0000 7f8a71167640 1 -- 172.21.15.44:0/2950774534 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a6c0b2f50 con 0x7f8a6c06ece0 2024-04-02T13:27:00.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.485+0000 7f8a637fe640 1 -- 172.21.15.44:0/2950774534 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+5181 (secure 0 0 0) 0x7f8a5c05f070 con 0x7f8a6c06ece0 2024-04-02T13:27:00.487 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:27:00.487 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":67,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:26:59.599493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8073":{"gid":8073,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6833/2250279975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2250279975},{"type":"v1","addr":"172.21.15.44:6833","nonce":2250279975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8073,"qdb_cluster":[8073]},"id":6}]} 2024-04-02T13:27:00.488 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 67 2024-04-02T13:27:00.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.486+0000 7f8a71167640 1 -- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8a4c06c8f0 msgr2=0x7f8a4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:00.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.486+0000 7f8a71167640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8a4c06c8f0 0x7f8a4c06edb0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8a54006fd0 tx=0x7f8a54008040 comp rx=0 tx=0).stop 2024-04-02T13:27:00.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.486+0000 7f8a71167640 1 -- 172.21.15.44:0/2950774534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a6c06ece0 msgr2=0x7f8a6c15dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:00.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.486+0000 7f8a71167640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a6c06ece0 0x7f8a6c15dc50 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00c400 tx=0x7f8a5c00c8c0 comp rx=0 tx=0).stop 2024-04-02T13:27:00.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.486+0000 7f8a71167640 1 -- 172.21.15.44:0/2950774534 shutdown_connections 2024-04-02T13:27:00.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.486+0000 7f8a71167640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8a4c06c8f0 0x7f8a4c06edb0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:00.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.486+0000 7f8a71167640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a6c0d09f0 0x7f8a6c15b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:00.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.486+0000 7f8a71167640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a6c0cece0 0x7f8a6c15cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:00.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.486+0000 7f8a71167640 1 --2- 172.21.15.44:0/2950774534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a6c06ece0 0x7f8a6c15dc50 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:00.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.486+0000 7f8a71167640 1 -- 172.21.15.44:0/2950774534 >> 172.21.15.44:0/2950774534 conn(0x7f8a6c0cfcd0 msgr2=0x7f8a6c0acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:00.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.487+0000 7f8a71167640 1 -- 172.21.15.44:0/2950774534 shutdown_connections 2024-04-02T13:27:00.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:00.487+0000 7f8a71167640 1 -- 172.21.15.44:0/2950774534 wait complete. 2024-04-02T13:27:00.903 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:26:43.025860+0000', 'modified': '2024-04-02T13:26:59.599493+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 92, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8073}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8073': {'gid': 8073, 'name': 'a', 'rank': 0, 'incarnation': 64, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.44:6833/2250279975', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6832', 'nonce': 2250279975}, {'type': 'v1', 'addr': '172.21.15.44:6833', 'nonce': 2250279975}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8073, 'qdb_cluster': [8073]} 2024-04-02T13:27:00.903 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:27:00.903 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T13:27:01.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.005+0000 7fd1d7332640 1 Processor -- start 2024-04-02T13:27:01.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.006+0000 7fd1d7332640 1 -- start start 2024-04-02T13:27:01.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.006+0000 7fd1d7332640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1d015c160 0x7fd1d006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.006+0000 7fd1d7332640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd1d00d0890 0x7fd1d00cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.006+0000 7fd1d7332640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d006bab0 con 0x7fd1d0160600 2024-04-02T13:27:01.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.007+0000 7fd1d7332640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd1d006bd90 con 0x7fd1d00d0890 2024-04-02T13:27:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.007+0000 7fd1d7332640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d00ccec0 con 0x7fd1d015c160 2024-04-02T13:27:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.007+0000 7fd1d58cf640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd1d00d0890 0x7fd1d00cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.007+0000 7fd1d48cd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd1d0160600 0x7fd1d00d0180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37666/0 (socket says 172.21.15.44:37666) 2024-04-02T13:27:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.007+0000 7fd1d48cd640 1 -- 172.21.15.44:0/2987264963 learned_addr learned my addr 172.21.15.44:0/2987264963 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:27:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.007+0000 7fd1d58cf640 1 -- 172.21.15.44:0/2987264963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1d015c160 msgr2=0x7fd1d006b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.007+0000 7fd1d58cf640 1 --2- 172.21.15.44:0/2987264963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1d015c160 0x7fd1d006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.007+0000 7fd1d58cf640 1 -- 172.21.15.44:0/2987264963 >> v1:172.21.15.44:6789/0 conn(0x7fd1d0160600 legacy=0x7fd1d00d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.007+0000 7fd1d58cf640 1 -- 172.21.15.44:0/2987264963 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d00cd1a0 con 0x7fd1d00d0890 2024-04-02T13:27:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.007+0000 7fd1c7fff640 1 -- 172.21.15.44:0/2987264963 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1603667265 0 0) 0x7fd1d006bab0 con 0x7fd1d0160600 2024-04-02T13:27:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.008+0000 7fd1d58cf640 1 --2- 172.21.15.44:0/2987264963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1d00d0890 0x7fd1d00cd6c0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009d90 tx=0x7fd1c802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f2b6f0b1c631894 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.008+0000 7fd1c7fff640 1 -- 172.21.15.44:0/2987264963 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8039070 con 0x7fd1d00d0890 2024-04-02T13:27:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.008+0000 7fd1c7fff640 1 -- 172.21.15.44:0/2987264963 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c802cbc0 con 0x7fd1d00d0890 2024-04-02T13:27:01.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.008+0000 7fd1c7fff640 1 -- 172.21.15.44:0/2987264963 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8034440 con 0x7fd1d00d0890 2024-04-02T13:27:01.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.008+0000 7fd1d7332640 1 -- 172.21.15.44:0/2987264963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1d00d0890 msgr2=0x7fd1d00cd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.008+0000 7fd1d7332640 1 --2- 172.21.15.44:0/2987264963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1d00d0890 0x7fd1d00cd6c0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009d90 tx=0x7fd1c802be10 comp rx=0 tx=0).stop 2024-04-02T13:27:01.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.008+0000 7fd1d7332640 1 -- 172.21.15.44:0/2987264963 shutdown_connections 2024-04-02T13:27:01.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.008+0000 7fd1d7332640 1 --2- 172.21.15.44:0/2987264963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1d00d0890 0x7fd1d00cd6c0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.008+0000 7fd1d7332640 1 --2- 172.21.15.44:0/2987264963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1d015c160 0x7fd1d006b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.008+0000 7fd1d7332640 1 -- 172.21.15.44:0/2987264963 >> 172.21.15.44:0/2987264963 conn(0x7fd1d00b2980 msgr2=0x7fd1d00b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:01.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.009+0000 7fd1d7332640 1 -- 172.21.15.44:0/2987264963 shutdown_connections 2024-04-02T13:27:01.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.009+0000 7fd1d7332640 1 -- 172.21.15.44:0/2987264963 wait complete. 2024-04-02T13:27:01.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.010+0000 7fd1d7332640 1 Processor -- start 2024-04-02T13:27:01.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.010+0000 7fd1d7332640 1 -- start start 2024-04-02T13:27:01.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.010+0000 7fd1d7332640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1d00d0890 0x7fd1d00c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.010+0000 7fd1d7332640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1d015c160 0x7fd1d01487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.010+0000 7fd1d7332640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1d0160600 0x7fd1d0148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.010+0000 7fd1d7332640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d00add60 con 0x7fd1d00d0890 2024-04-02T13:27:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.010+0000 7fd1d7332640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d0160a60 con 0x7fd1d0160600 2024-04-02T13:27:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.010+0000 7fd1d7332640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d00adee0 con 0x7fd1d015c160 2024-04-02T13:27:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.010+0000 7fd1d50ce640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1d00d0890 0x7fd1d00c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.010+0000 7fd1d50ce640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1d00d0890 0x7fd1d00c6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52176/0 (socket says 172.21.15.44:52176) 2024-04-02T13:27:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.010+0000 7fd1d50ce640 1 -- 172.21.15.44:0/1997921368 learned_addr learned my addr 172.21.15.44:0/1997921368 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.011+0000 7fd1d50ce640 1 -- 172.21.15.44:0/1997921368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1d015c160 msgr2=0x7fd1d01487c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.011+0000 7fd1d50ce640 1 --2- 172.21.15.44:0/1997921368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1d015c160 0x7fd1d01487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.011+0000 7fd1d58cf640 1 --2- 172.21.15.44:0/1997921368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1d0160600 0x7fd1d0148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.011+0000 7fd1d50ce640 1 -- 172.21.15.44:0/1997921368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1d0160600 msgr2=0x7fd1d0148d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.011+0000 7fd1d50ce640 1 --2- 172.21.15.44:0/1997921368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1d0160600 0x7fd1d0148d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.011+0000 7fd1d50ce640 1 -- 172.21.15.44:0/1997921368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d015ec40 con 0x7fd1d00d0890 2024-04-02T13:27:01.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.011+0000 7fd1d50ce640 1 --2- 172.21.15.44:0/1997921368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1d00d0890 0x7fd1d00c6b00 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00b440 tx=0x7fd1cc00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:01.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.011+0000 7fd1c5ffb640 1 -- 172.21.15.44:0/1997921368 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc015020 con 0x7fd1d00d0890 2024-04-02T13:27:01.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.011+0000 7fd1c5ffb640 1 -- 172.21.15.44:0/1997921368 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1cc011890 con 0x7fd1d00d0890 2024-04-02T13:27:01.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.011+0000 7fd1c5ffb640 1 -- 172.21.15.44:0/1997921368 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc011c80 con 0x7fd1d00d0890 2024-04-02T13:27:01.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.011+0000 7fd1d7332640 1 -- 172.21.15.44:0/1997921368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d015ef20 con 0x7fd1d00d0890 2024-04-02T13:27:01.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.011+0000 7fd1d7332640 1 -- 172.21.15.44:0/1997921368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d015f4e0 con 0x7fd1d00d0890 2024-04-02T13:27:01.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.013+0000 7fd1c5ffb640 1 -- 172.21.15.44:0/1997921368 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd1cc018020 con 0x7fd1d00d0890 2024-04-02T13:27:01.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.013+0000 7fd1c5ffb640 1 --2- 172.21.15.44:0/1997921368 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd1b406c8f0 0x7fd1b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.013+0000 7fd1c5ffb640 1 -- 172.21.15.44:0/1997921368 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7fd1cc08beb0 con 0x7fd1d00d0890 2024-04-02T13:27:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.013+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/1997921368 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd1b406c8f0 0x7fd1b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.014+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/1997921368 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd1b406c8f0 0x7fd1b406edb0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0007fd0 tx=0x7fd1c0007920 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.014+0000 7fd1d9ff7740 1 Processor -- start 2024-04-02T13:27:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.014+0000 7fd1d9ff7740 1 -- start start 2024-04-02T13:27:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d9ff7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca752a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d9ff7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 0x55dca74e3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d9ff7740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca750bf50 0x55dca74e4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d9ff7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55dca7448060 con 0x55dca752c9f0 2024-04-02T13:27:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d9ff7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dca73f0650 con 0x55dca752c610 2024-04-02T13:27:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d9ff7740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55dca7444010 con 0x55dca750bf50 2024-04-02T13:27:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d48cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 0x55dca74e3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d48cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 0x55dca74e3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52192/0 (socket says 172.21.15.44:52192) 2024-04-02T13:27:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d48cd640 1 -- 172.21.15.44:0/1599232881 learned_addr learned my addr 172.21.15.44:0/1599232881 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d58cf640 1 --2- 172.21.15.44:0/1599232881 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca750bf50 0x55dca74e4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d50ce640 1 --2- 172.21.15.44:0/1599232881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca752a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d48cd640 1 -- 172.21.15.44:0/1599232881 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca750bf50 msgr2=0x55dca74e4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/1599232881 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca750bf50 0x55dca74e4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d48cd640 1 -- 172.21.15.44:0/1599232881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 msgr2=0x55dca752a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/1599232881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca752a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d48cd640 1 -- 172.21.15.44:0/1599232881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dca715c2d0 con 0x55dca752c9f0 2024-04-02T13:27:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d50ce640 1 --2- 172.21.15.44:0/1599232881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca752a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:27:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/1599232881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 0x55dca74e3c40 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0007580 tx=0x7fd1c000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174b4b9a4376d44 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.015+0000 7fd1b2ffd640 1 -- 172.21.15.44:0/1599232881 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0007670 con 0x55dca752c9f0 2024-04-02T13:27:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.016+0000 7fd1b2ffd640 1 -- 172.21.15.44:0/1599232881 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c0005c80 con 0x55dca752c9f0 2024-04-02T13:27:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.016+0000 7fd1b2ffd640 1 -- 172.21.15.44:0/1599232881 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0016b50 con 0x55dca752c9f0 2024-04-02T13:27:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.016+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/1599232881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 msgr2=0x55dca74e3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.016+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/1599232881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 0x55dca74e3c40 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0007580 tx=0x7fd1c000f000 comp rx=0 tx=0).stop 2024-04-02T13:27:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.016+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/1599232881 shutdown_connections 2024-04-02T13:27:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.016+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/1599232881 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca750bf50 0x55dca74e4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.016+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/1599232881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 0x55dca74e3c40 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.016+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/1599232881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca752a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.016+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/1599232881 >> 172.21.15.44:0/1599232881 conn(0x55dca74e0da0 msgr2=0x55dca7529920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:01.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.016+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/1599232881 shutdown_connections 2024-04-02T13:27:01.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.016+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/1599232881 wait complete. 2024-04-02T13:27:01.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.017+0000 7fd1d9ff7740 1 -- start start 2024-04-02T13:27:01.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.017+0000 7fd1d9ff7740 1 Processor -- start 2024-04-02T13:27:01.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d9ff7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca75011a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d9ff7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 0x55dca7500330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d9ff7740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca75090e0 0x55dca75095c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d9ff7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55dca74df3d0 con 0x55dca752c9f0 2024-04-02T13:27:01.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d9ff7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dca7403a10 con 0x55dca752c610 2024-04-02T13:27:01.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d9ff7740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55dca74e31d0 con 0x55dca75090e0 2024-04-02T13:27:01.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d9ff7740 10 client.0 _ll_register_callbacks cb 0x55dca74400b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:01.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d48cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 0x55dca7500330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d48cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 0x55dca7500330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52202/0 (socket says 172.21.15.44:52202) 2024-04-02T13:27:01.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d48cd640 1 -- 172.21.15.44:0/1673457340 learned_addr learned my addr 172.21.15.44:0/1673457340 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:01.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d58cf640 1 --2- 172.21.15.44:0/1673457340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca75090e0 0x55dca75095c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d50ce640 1 --2- 172.21.15.44:0/1673457340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca75011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d48cd640 1 -- 172.21.15.44:0/1673457340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca75090e0 msgr2=0x55dca75095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:01.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/1673457340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca75090e0 0x55dca75095c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d48cd640 1 -- 172.21.15.44:0/1673457340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 msgr2=0x55dca75011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:01.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/1673457340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca75011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d48cd640 1 -- 172.21.15.44:0/1673457340 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dca72266b0 con 0x55dca752c9f0 2024-04-02T13:27:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/1673457340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 0x55dca7500330 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x55dca74e1d30 tx=0x7fd1c00010f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c130c74dd1a005 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1abfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:27:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1abfff640 1 -- 172.21.15.44:0/1673457340 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c002e060 con 0x55dca752c9f0 2024-04-02T13:27:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/1673457340 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1c0005c80 con 0x55dca752c9f0 2024-04-02T13:27:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.018+0000 7fd1abfff640 1 -- 172.21.15.44:0/1673457340 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7fd1c001c040 con 0x55dca752c9f0 2024-04-02T13:27:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1abfff640 1 -- 172.21.15.44:0/1673457340 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fd1c0034050 con 0x55dca752c9f0 2024-04-02T13:27:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1d9ff7740 10 client.8133 fetch_fsmap learned FSMap version 67 2024-04-02T13:27:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/1673457340 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x55dca752b5a0 con 0x55dca752c9f0 2024-04-02T13:27:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1abfff640 1 -- 172.21.15.44:0/1673457340 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c0005c80 con 0x55dca752c9f0 2024-04-02T13:27:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1abfff640 1 -- 172.21.15.44:0/1673457340 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7fd1c0037020 con 0x55dca752c9f0 2024-04-02T13:27:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1d9ff7740 10 client.8133 fetch_fsmap finished waiting for FSMap version 67 2024-04-02T13:27:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1d9ff7740 10 client.8133 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:27:01.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1d9ff7740 20 client.8133 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:01.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/1673457340 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55dca753e6c0 0x55dca754bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1d9ff7740 4 client.8133 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:27:01.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/1673457340 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dca73efcf0 con 0x55dca753e6c0 2024-04-02T13:27:01.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1d50ce640 1 --2- 172.21.15.44:0/1673457340 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55dca753e6c0 0x55dca754bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1abfff640 10 client.8133 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:01.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.019+0000 7fd1d50ce640 1 --2- 172.21.15.44:0/1673457340 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55dca753e6c0 0x55dca754bfc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd541b284438629a server_cookie=845e2709f133ba36 in_seq=0 out_seq=0 2024-04-02T13:27:01.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1abfff640 1 -- 172.21.15.44:0/1673457340 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dca73efcf0 con 0x55dca753e6c0 2024-04-02T13:27:01.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1abfff640 10 client.8133 handle_command_reply: tid=0 2024-04-02T13:27:01.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1d9ff7740 1 client.8133 shutdown 2024-04-02T13:27:01.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/1673457340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 msgr2=0x55dca7500330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/1673457340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 0x55dca7500330 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x55dca74e1d30 tx=0x7fd1c00010f0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/1673457340 shutdown_connections 2024-04-02T13:27:01.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/1673457340 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55dca753e6c0 0x55dca754bfc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/1673457340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca75090e0 0x55dca75095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/1673457340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c9f0 0x55dca7500330 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1abfff640 0 client.8133 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:01.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/1673457340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca75011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/1673457340 >> 172.21.15.44:0/1673457340 conn(0x55dca750bf50 msgr2=0x55dca74dd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:01.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/1673457340 shutdown_connections 2024-04-02T13:27:01.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/1673457340 wait complete. 2024-04-02T13:27:01.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.020+0000 7fd1d9ff7740 20 client.8133 trim_cache size 0 max 16384 2024-04-02T13:27:01.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.039+0000 7fd1d9ff7740 1 Processor -- start 2024-04-02T13:27:01.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.039+0000 7fd1d9ff7740 1 -- start start 2024-04-02T13:27:01.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.039+0000 7fd1d9ff7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca75090e0 0x55dca74e4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.039+0000 7fd1d9ff7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c610 0x55dca75011d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.039+0000 7fd1d9ff7740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca752c9f0 0x55dca7529910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.039+0000 7fd1d9ff7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d006f1a0 con 0x55dca752c610 2024-04-02T13:27:01.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.039+0000 7fd1d9ff7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d00c8d50 con 0x55dca75090e0 2024-04-02T13:27:01.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.039+0000 7fd1d9ff7740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d00c93a0 con 0x55dca752c9f0 2024-04-02T13:27:01.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.039+0000 7fd1d48cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c610 0x55dca75011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.040+0000 7fd1d48cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c610 0x55dca75011d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52214/0 (socket says 172.21.15.44:52214) 2024-04-02T13:27:01.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.040+0000 7fd1d48cd640 1 -- 172.21.15.44:0/126986342 learned_addr learned my addr 172.21.15.44:0/126986342 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:01.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.040+0000 7fd1d48cd640 1 -- 172.21.15.44:0/126986342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca752c9f0 msgr2=0x55dca7529910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:01.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.040+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/126986342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca752c9f0 0x55dca7529910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.040+0000 7fd1d48cd640 1 -- 172.21.15.44:0/126986342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca75090e0 msgr2=0x55dca74e4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:01.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.040+0000 7fd1d50ce640 1 --2- 172.21.15.44:0/126986342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca75090e0 0x55dca74e4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.040+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/126986342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca75090e0 0x55dca74e4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.040+0000 7fd1d48cd640 1 -- 172.21.15.44:0/126986342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dca752c140 con 0x55dca752c610 2024-04-02T13:27:01.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.040+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/126986342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c610 0x55dca75011d0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0016820 tx=0x7fd1c0025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6278ccc047e72b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:01.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.040+0000 7fd1b27fc640 1 -- 172.21.15.44:0/126986342 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0005c80 con 0x55dca752c610 2024-04-02T13:27:01.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.040+0000 7fd1b27fc640 1 -- 172.21.15.44:0/126986342 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c0012770 con 0x55dca752c610 2024-04-02T13:27:01.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.040+0000 7fd1b27fc640 1 -- 172.21.15.44:0/126986342 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0012b20 con 0x55dca752c610 2024-04-02T13:27:01.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.041+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/126986342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c610 msgr2=0x55dca75011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.041+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/126986342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c610 0x55dca75011d0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0016820 tx=0x7fd1c0025000 comp rx=0 tx=0).stop 2024-04-02T13:27:01.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.041+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/126986342 shutdown_connections 2024-04-02T13:27:01.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.041+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/126986342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca752c9f0 0x55dca7529910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.041+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/126986342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca752c610 0x55dca75011d0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.041+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/126986342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca75090e0 0x55dca74e4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.041+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/126986342 >> 172.21.15.44:0/126986342 conn(0x55dca750bf50 msgr2=0x55dca74d9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:01.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.041+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/126986342 shutdown_connections 2024-04-02T13:27:01.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.041+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/126986342 wait complete. 2024-04-02T13:27:01.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.041+0000 7fd1d9ff7740 1 -- start start 2024-04-02T13:27:01.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.041+0000 7fd1d9ff7740 1 Processor -- start 2024-04-02T13:27:01.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.042+0000 7fd1d9ff7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca75090e0 0x55dca7590d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.042+0000 7fd1d9ff7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca74fecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.042+0000 7fd1d9ff7740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca752c9f0 0x55dca75ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.042+0000 7fd1d9ff7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55dca753e540 con 0x55dca75090e0 2024-04-02T13:27:01.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.042+0000 7fd1d9ff7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dca753ee90 con 0x55dca752c610 2024-04-02T13:27:01.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.042+0000 7fd1d9ff7740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55dca753eb60 con 0x55dca752c9f0 2024-04-02T13:27:01.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.042+0000 7fd1d9ff7740 10 client.0 _ll_register_callbacks cb 0x7fd1c00073f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:01.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.042+0000 7fd1d50ce640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca75090e0 0x55dca7590d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.042+0000 7fd1d50ce640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca75090e0 0x55dca7590d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52228/0 (socket says 172.21.15.44:52228) 2024-04-02T13:27:01.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.042+0000 7fd1d50ce640 1 -- 172.21.15.44:0/178782468 learned_addr learned my addr 172.21.15.44:0/178782468 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:01.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.042+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca74fecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.042+0000 7fd1d58cf640 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca752c9f0 0x55dca75ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d50ce640 1 -- 172.21.15.44:0/178782468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca752c9f0 msgr2=0x55dca75ac1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:01.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d50ce640 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca752c9f0 0x55dca75ac1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d50ce640 1 -- 172.21.15.44:0/178782468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 msgr2=0x55dca74fecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:01.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d50ce640 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca74fecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d50ce640 1 -- 172.21.15.44:0/178782468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dca7527d60 con 0x55dca75090e0 2024-04-02T13:27:01.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca74fecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:27:01.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d50ce640 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca75090e0 0x55dca7590d10 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00be00 tx=0x7fd1cc01c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f358a93ab63de7ce server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:01.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1abfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:27:01.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1abfff640 1 -- 172.21.15.44:0/178782468 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc0625e0 con 0x55dca75090e0 2024-04-02T13:27:01.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1abfff640 1 -- 172.21.15.44:0/178782468 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7fd1cc057d00 con 0x55dca75090e0 2024-04-02T13:27:01.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/178782468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1c0012770 con 0x55dca75090e0 2024-04-02T13:27:01.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1abfff640 1 -- 172.21.15.44:0/178782468 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fd1cc056070 con 0x55dca75090e0 2024-04-02T13:27:01.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d9ff7740 10 client.8139 fetch_fsmap learned FSMap version 67 2024-04-02T13:27:01.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/178782468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x55dca75282c0 con 0x55dca75090e0 2024-04-02T13:27:01.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1abfff640 1 -- 172.21.15.44:0/178782468 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c0012770 con 0x55dca75090e0 2024-04-02T13:27:01.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1abfff640 1 -- 172.21.15.44:0/178782468 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7fd1cc062150 con 0x55dca75090e0 2024-04-02T13:27:01.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d9ff7740 10 client.8139 fetch_fsmap finished waiting for FSMap version 67 2024-04-02T13:27:01.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d9ff7740 10 client.8139 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:27:01.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d9ff7740 20 client.8139 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:01.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55dca75517d0 0x55dca7502b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d9ff7740 4 client.8139 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:27:01.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/178782468 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "status"}) -- 0x55dca74555a0 con 0x55dca75517d0 2024-04-02T13:27:01.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.043+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55dca75517d0 0x55dca7502b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1d48cd640 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55dca75517d0 0x55dca7502b10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a943cae21e6d0624 server_cookie=14b73b164033eee8 in_seq=0 out_seq=0 2024-04-02T13:27:01.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1abfff640 10 client.8139 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:01.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1abfff640 1 -- 172.21.15.44:0/178782468 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55dca74555a0 con 0x55dca75517d0 2024-04-02T13:27:01.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1abfff640 10 client.8139 handle_command_reply: tid=0 2024-04-02T13:27:01.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1d9ff7740 1 client.8139 shutdown 2024-04-02T13:27:01.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/178782468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca75090e0 msgr2=0x55dca7590d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca75090e0 0x55dca7590d10 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00be00 tx=0x7fd1cc01c040 comp rx=0 tx=0).stop 2024-04-02T13:27:01.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/178782468 shutdown_connections 2024-04-02T13:27:01.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55dca75517d0 0x55dca7502b10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55dca752c9f0 0x55dca75ac1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dca752c610 0x55dca74fecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1d9ff7740 1 --2- 172.21.15.44:0/178782468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dca75090e0 0x55dca7590d10 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1abfff640 0 client.8139 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:01.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/178782468 >> 172.21.15.44:0/178782468 conn(0x55dca750bf50 msgr2=0x55dca750ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:01.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/178782468 shutdown_connections 2024-04-02T13:27:01.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1d9ff7740 1 -- 172.21.15.44:0/178782468 wait complete. 2024-04-02T13:27:01.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.044+0000 7fd1d9ff7740 20 client.8139 trim_cache size 0 max 16384 2024-04-02T13:27:01.056 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:27:01.056 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:27:01.056 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:27:01.056 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 8073, 2024-04-02T13:27:01.056 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:27:01.056 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:27:01.056 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:27:01.056 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 4.4459595780000001, 2024-04-02T13:27:01.056 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 67, 2024-04-02T13:27:01.056 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 92, 2024-04-02T13:27:01.056 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 92, 2024-04-02T13:27:01.057 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 5.4186530599999996 2024-04-02T13:27:01.057 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:27:01.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.045+0000 7fd1d7332640 1 -- 172.21.15.44:0/1997921368 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd1b406c8f0 msgr2=0x7fd1b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.045+0000 7fd1d7332640 1 --2- 172.21.15.44:0/1997921368 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd1b406c8f0 0x7fd1b406edb0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0007fd0 tx=0x7fd1c0007920 comp rx=0 tx=0).stop 2024-04-02T13:27:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.045+0000 7fd1d7332640 1 -- 172.21.15.44:0/1997921368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1d00d0890 msgr2=0x7fd1d00c6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.045+0000 7fd1d7332640 1 --2- 172.21.15.44:0/1997921368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1d00d0890 0x7fd1d00c6b00 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00b440 tx=0x7fd1cc00b900 comp rx=0 tx=0).stop 2024-04-02T13:27:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.045+0000 7fd1d7332640 1 -- 172.21.15.44:0/1997921368 shutdown_connections 2024-04-02T13:27:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.045+0000 7fd1d7332640 1 --2- 172.21.15.44:0/1997921368 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd1b406c8f0 0x7fd1b406edb0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.045+0000 7fd1d7332640 1 --2- 172.21.15.44:0/1997921368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1d0160600 0x7fd1d0148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.045+0000 7fd1d7332640 1 --2- 172.21.15.44:0/1997921368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1d015c160 0x7fd1d01487c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.045+0000 7fd1d7332640 1 --2- 172.21.15.44:0/1997921368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1d00d0890 0x7fd1d00c6b00 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.045+0000 7fd1d7332640 1 -- 172.21.15.44:0/1997921368 >> 172.21.15.44:0/1997921368 conn(0x7fd1d00b2980 msgr2=0x7fd1d00ae120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.045+0000 7fd1d7332640 1 -- 172.21.15.44:0/1997921368 shutdown_connections 2024-04-02T13:27:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.045+0000 7fd1d7332640 1 -- 172.21.15.44:0/1997921368 wait complete. 2024-04-02T13:27:01.060 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:27:01.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.159+0000 7f386d167640 1 Processor -- start 2024-04-02T13:27:01.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.159+0000 7f386d167640 1 -- start start 2024-04-02T13:27:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.159+0000 7f386d167640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f38680b47b0 0x7f38680d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.159+0000 7f386d167640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38680b5200 0x7f38680b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.159+0000 7f386d167640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38680d4d70 con 0x7f38680d28f0 2024-04-02T13:27:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.159+0000 7f386d167640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f38680d5050 con 0x7f38680b47b0 2024-04-02T13:27:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.159+0000 7f386d167640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f38680d23a0 con 0x7f38680b5200 2024-04-02T13:27:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.160+0000 7f386759e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38680b5200 0x7f38680b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.160+0000 7f385ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f38680b47b0 0x7f38680d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.160+0000 7f386759e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38680b5200 0x7f38680b7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:49764/0 (socket says 172.21.15.44:49764) 2024-04-02T13:27:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.160+0000 7f3866d9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f38680d28f0 0x7f38680b40a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:43034/0 (socket says 172.21.15.44:43034) 2024-04-02T13:27:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.160+0000 7f386759e640 1 -- 172.21.15.44:0/908980103 learned_addr learned my addr 172.21.15.44:0/908980103 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.160+0000 7f386659c640 1 -- 172.21.15.44:0/908980103 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2887709044 0 0) 0x7f38680d4d70 con 0x7f38680d28f0 2024-04-02T13:27:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.160+0000 7f386759e640 1 -- 172.21.15.44:0/908980103 >> v2:172.21.15.69:3300/0 conn(0x7f38680b47b0 msgr2=0x7f38680d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.160+0000 7f386759e640 1 --2- 172.21.15.44:0/908980103 >> v2:172.21.15.69:3300/0 conn(0x7f38680b47b0 0x7f38680d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.160+0000 7f386759e640 1 -- 172.21.15.44:0/908980103 >> v1:172.21.15.69:6789/0 conn(0x7f38680d28f0 legacy=0x7f38680b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.160+0000 7f386759e640 1 -- 172.21.15.44:0/908980103 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38680d2030 con 0x7f38680b5200 2024-04-02T13:27:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.161+0000 7f386759e640 1 --2- 172.21.15.44:0/908980103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38680b5200 0x7f38680b7120 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3860009e50 tx=0x7f386002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3be20d69fc5f6a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.161+0000 7f386659c640 1 -- 172.21.15.44:0/908980103 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386002c9e0 con 0x7f38680b5200 2024-04-02T13:27:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.161+0000 7f386659c640 1 -- 172.21.15.44:0/908980103 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f386002cb80 con 0x7f38680b5200 2024-04-02T13:27:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.161+0000 7f386659c640 1 -- 172.21.15.44:0/908980103 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38600343f0 con 0x7f38680b5200 2024-04-02T13:27:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.161+0000 7f386d167640 1 -- 172.21.15.44:0/908980103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38680b5200 msgr2=0x7f38680b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.161+0000 7f386d167640 1 --2- 172.21.15.44:0/908980103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38680b5200 0x7f38680b7120 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3860009e50 tx=0x7f386002be40 comp rx=0 tx=0).stop 2024-04-02T13:27:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.162+0000 7f386d167640 1 -- 172.21.15.44:0/908980103 shutdown_connections 2024-04-02T13:27:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.162+0000 7f386d167640 1 --2- 172.21.15.44:0/908980103 >> v2:172.21.15.69:3300/0 conn(0x7f38680b47b0 0x7f38680d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.162+0000 7f386d167640 1 --2- 172.21.15.44:0/908980103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38680b5200 0x7f38680b7120 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.162+0000 7f386d167640 1 -- 172.21.15.44:0/908980103 >> 172.21.15.44:0/908980103 conn(0x7f38680d4120 msgr2=0x7f38680d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:01.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.162+0000 7f386d167640 1 -- 172.21.15.44:0/908980103 shutdown_connections 2024-04-02T13:27:01.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.162+0000 7f386d167640 1 -- 172.21.15.44:0/908980103 wait complete. 2024-04-02T13:27:01.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.163+0000 7f386d167640 1 Processor -- start 2024-04-02T13:27:01.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.163+0000 7f386d167640 1 -- start start 2024-04-02T13:27:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f386d167640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38680b47b0 0x7f38680b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f386d167640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38680b5200 0x7f386815f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f386d167640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38680d28f0 0x7f386809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f386d167640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f38680d1260 con 0x7f38680b5200 2024-04-02T13:27:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f386d167640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f38680b5020 con 0x7f38680d28f0 2024-04-02T13:27:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f386d167640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f38680d13e0 con 0x7f38680b47b0 2024-04-02T13:27:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f385ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38680b5200 0x7f386815f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f385ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38680b5200 0x7f386815f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52234/0 (socket says 172.21.15.44:52234) 2024-04-02T13:27:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f385ffff640 1 -- 172.21.15.44:0/182490680 learned_addr learned my addr 172.21.15.44:0/182490680 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f386759e640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38680d28f0 0x7f386809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f3866d9d640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38680b47b0 0x7f38680b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f385ffff640 1 -- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38680b47b0 msgr2=0x7f38680b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f385ffff640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38680b47b0 0x7f38680b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f385ffff640 1 -- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38680d28f0 msgr2=0x7f386809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f385ffff640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38680d28f0 0x7f386809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f385ffff640 1 -- 172.21.15.44:0/182490680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f386809e240 con 0x7f38680b5200 2024-04-02T13:27:01.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f385ffff640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38680b5200 0x7f386815f990 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f385000e870 tx=0x7f385000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:01.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f385f7fe640 1 -- 172.21.15.44:0/182490680 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3850017070 con 0x7f38680b5200 2024-04-02T13:27:01.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f385f7fe640 1 -- 172.21.15.44:0/182490680 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f385000c940 con 0x7f38680b5200 2024-04-02T13:27:01.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f386d167640 1 -- 172.21.15.44:0/182490680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f386809e520 con 0x7f38680b5200 2024-04-02T13:27:01.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f385f7fe640 1 -- 172.21.15.44:0/182490680 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385000cd30 con 0x7f38680b5200 2024-04-02T13:27:01.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.164+0000 7f386d167640 1 -- 172.21.15.44:0/182490680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f386809edc0 con 0x7f38680b5200 2024-04-02T13:27:01.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.165+0000 7f386d167640 1 -- 172.21.15.44:0/182490680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f386809e6f0 con 0x7f38680b5200 2024-04-02T13:27:01.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.166+0000 7f385f7fe640 1 -- 172.21.15.44:0/182490680 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f385001a070 con 0x7f38680b5200 2024-04-02T13:27:01.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.166+0000 7f385f7fe640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f384806c8f0 0x7f384806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:01.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.166+0000 7f385f7fe640 1 -- 172.21.15.44:0/182490680 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7f385008bfe0 con 0x7f38680b5200 2024-04-02T13:27:01.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.166+0000 7f3866d9d640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f384806c8f0 0x7f384806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:01.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.167+0000 7f3866d9d640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f384806c8f0 0x7f384806edb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3858000f60 tx=0x7f3858009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:01.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.171+0000 7f385f7fe640 1 -- 172.21.15.44:0/182490680 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f385005a1d0 con 0x7f38680b5200 2024-04-02T13:27:01.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.394+0000 7f386d167640 1 -- 172.21.15.44:0/182490680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f38680684f0 con 0x7f38680b5200 2024-04-02T13:27:01.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.395+0000 7f385f7fe640 1 -- 172.21.15.44:0/182490680 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v67) ==== 53+0+73 (secure 0 0 0) 0x7f3850059ad0 con 0x7f38680b5200 2024-04-02T13:27:01.397 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:27:01.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.397+0000 7f386d167640 1 -- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f384806c8f0 msgr2=0x7f384806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.397+0000 7f386d167640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f384806c8f0 0x7f384806edb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3858000f60 tx=0x7f3858009040 comp rx=0 tx=0).stop 2024-04-02T13:27:01.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.397+0000 7f386d167640 1 -- 172.21.15.44:0/182490680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38680b5200 msgr2=0x7f386815f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:01.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.397+0000 7f386d167640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38680b5200 0x7f386815f990 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f385000e870 tx=0x7f385000ed30 comp rx=0 tx=0).stop 2024-04-02T13:27:01.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.397+0000 7f386d167640 1 -- 172.21.15.44:0/182490680 shutdown_connections 2024-04-02T13:27:01.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.397+0000 7f386d167640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f384806c8f0 0x7f384806edb0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.397+0000 7f386d167640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38680d28f0 0x7f386809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.397+0000 7f386d167640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38680b5200 0x7f386815f990 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.397+0000 7f386d167640 1 --2- 172.21.15.44:0/182490680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38680b47b0 0x7f38680b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:01.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.397+0000 7f386d167640 1 -- 172.21.15.44:0/182490680 >> 172.21.15.44:0/182490680 conn(0x7f38680d4120 msgr2=0x7f3868161840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:01.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.397+0000 7f386d167640 1 -- 172.21.15.44:0/182490680 shutdown_connections 2024-04-02T13:27:01.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01.397+0000 7f386d167640 1 -- 172.21.15.44:0/182490680 wait complete. 2024-04-02T13:27:01.410 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:27:01.410 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:27:01.410 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:27:01.410 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:27:01.410 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:27:01.410 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:27:01.411 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:27:01.411 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:27:01.411 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:27:01.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:01.411 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:27:01.426 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:27:01.426 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:27:01.426 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:27:01.426 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:27:01.426 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42357sec preferred_lft 42357sec 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:27:01.427 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:27:01.428 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:27:01.428 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:27:01.428 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:27:01.428 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:27:01.428 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:27:01.428 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:27:01.428 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:27:01.428 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:27:01.428 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:27:01.428 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:27:01.428 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:27:01.429 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:27:01.429 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:27:01.429 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:27:01.429 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:27:01.429 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:27:01.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:01.429 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:27:01.491 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:27:01.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:01.493 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:27:01.548 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:27:01.548 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:27:01.548 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:27:01.548 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:27:01.548 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:27:01.549 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:27:01.549 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:27:01.549 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:27:01.549 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:27:01.549 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:27:01.549 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:27:01.601 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:27:01.599+0000 7fe0b4a36640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:27:01.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:27:01.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:27:01.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:01.685 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:27:01.740 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:27:01.740 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:27:01.740 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:27:01.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:01.741 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:27:01.807 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:27:01.807 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:27:01.807 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:27:01.861 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:27:01.862 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:27:01.862 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:27:01.916 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:27:01.984 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:27:02.039 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:27:02.039 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:27:02.039 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:27:02.040 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:27:02.041 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:27:02.042 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:27:02.042 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:27:02.042 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:27:02.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:02.042 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:27:02.095 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:27:02.095 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:27:02.138 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:27:02.170 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:27:02.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:27:02.220+0000 7f653f235480 -1 init, newargv = 0x55e7a6be2210 newargc=15 2024-04-02T13:27:02.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[71243]: starting ceph client 2024-04-02T13:27:02.226 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:27:02.226 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:27:02.226 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:27:02.226 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:27:02.226 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:27:02.226 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:27:02.226 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:27:02.226 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:27:02.227 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:27:02.227 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:27:02.227 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:27:02.227 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:27:02.227 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:27:02.227 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:27:02.227 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:27:02.227 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:27:02.227 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:27:02.227 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:27:02.227 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:27:02.227 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:27:02.227 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:27:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:27:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:27:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:27:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:27:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:27:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:27:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:27:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:27:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:27:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:27:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:27:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:27:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:27:02.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:02.229 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:27:02.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[71243]: starting fuse 2024-04-02T13:27:02.243 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:27:02.243 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:27:02.243 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:27:02.243 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:27:02.244 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:27:02.244 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:27:02.244 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:27:02.244 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:27:02.244 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:27:02.244 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:27:02.244 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:27:02.244 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:27:02.244 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:27:02.244 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:27:02.244 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:27:02.245 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:27:02.363 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.71243.asok 2024-04-02T13:27:02.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:27:02.373 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.71243.asok 2024-04-02T13:27:02.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:02.373 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.71243.asok status 2024-04-02T13:27:02.510 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:27:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:27:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:27:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:27:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:27:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:27:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:27:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "71243", 2024-04-02T13:27:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:27:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:27:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:27:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:27:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 8151, 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 8151 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3605293844 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:27:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3605293844 2024-04-02T13:27:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:27:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.8151 192.168.0.1:0/3605293844", 2024-04-02T13:27:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/3605293844", 2024-04-02T13:27:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:27:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 67, 2024-04-02T13:27:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 92, 2024-04-02T13:27:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:27:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:27:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:27:02.513 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:27:02.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:02.519 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:27:02.519 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:27:02.533 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:27:02.534 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:27:02.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:02.534 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:27:02.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:02.601 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat testdir/hardlink) 2024-04-02T13:27:02.658 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:27:02.655+0000 7fe0b0a2e640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-04-02T13:27:02.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:27:02.660 INFO:teuthology.orchestra.run.smithi044.stderr:stat: cannot statx 'testdir/hardlink': Input/output error 2024-04-02T13:27:02.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:02.660 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive 2024-04-02T13:27:02.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.801+0000 7f13e38ba640 1 Processor -- start 2024-04-02T13:27:02.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.802+0000 7f13e38ba640 1 -- start start 2024-04-02T13:27:02.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.802+0000 7f13e38ba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f13dc0cec60 0x7f13dc0d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.802+0000 7f13e38ba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13dc0cf1f0 con 0x7f13dc15e1d0 2024-04-02T13:27:02.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.802+0000 7f13e38ba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13dc0cf4d0 con 0x7f13dc15e5b0 2024-04-02T13:27:02.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.802+0000 7f13e38ba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f13dc0cfe00 con 0x7f13dc0cec60 2024-04-02T13:27:02.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13e1656640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f13dc15e1d0 0x7f13dc0d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37672/0 (socket says 172.21.15.44:37672) 2024-04-02T13:27:02.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13e1656640 1 -- 172.21.15.44:0/2845040767 learned_addr learned my addr 172.21.15.44:0/2845040767 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:27:02.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1943888473 0 0) 0x7f13dc0cf1f0 con 0x7f13dc15e1d0 2024-04-02T13:27:02.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13c4003520 con 0x7f13dc15e1d0 2024-04-02T13:27:02.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13e0e55640 1 --2- 172.21.15.44:0/2845040767 >> v2:172.21.15.69:3300/0 conn(0x7f13dc0cec60 0x7f13dc0d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2745406102 0 0) 0x7f13c4003520 con 0x7f13dc15e1d0 2024-04-02T13:27:02.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 >> v2:172.21.15.69:3300/0 conn(0x7f13dc0cec60 msgr2=0x7f13dc0d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 --2- 172.21.15.44:0/2845040767 >> v2:172.21.15.69:3300/0 conn(0x7f13dc0cec60 0x7f13dc0d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 >> v1:172.21.15.69:6789/0 conn(0x7f13dc15e5b0 legacy=0x7f13dc0b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13dc0cf7b0 con 0x7f13dc15e1d0 2024-04-02T13:27:02.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f13d4001670 con 0x7f13dc15e1d0 2024-04-02T13:27:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 >> v1:172.21.15.44:6789/0 conn(0x7f13dc15e1d0 legacy=0x7f13dc0d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 --2- 172.21.15.44:0/2845040767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c40039b0 0x7f13c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 --2- 172.21.15.44:0/2845040767 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f13c4004800 0x7f13c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 --2- 172.21.15.44:0/2845040767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13c40051c0 0x7f13c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc0cfe00 con 0x7f13c4004800 2024-04-02T13:27:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f13c400c160 con 0x7f13c40051c0 2024-04-02T13:27:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f13c400c360 con 0x7f13c40039b0 2024-04-02T13:27:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13e0e55640 1 --2- 172.21.15.44:0/2845040767 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f13c4004800 0x7f13c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13e1656640 1 --2- 172.21.15.44:0/2845040767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c40039b0 0x7f13c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.803+0000 7f13e1e57640 1 --2- 172.21.15.44:0/2845040767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13c40051c0 0x7f13c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.804+0000 7f13e0e55640 1 -- 172.21.15.44:0/2845040767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c40039b0 msgr2=0x7f13c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.804+0000 7f13e0e55640 1 --2- 172.21.15.44:0/2845040767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c40039b0 0x7f13c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.804+0000 7f13e0e55640 1 -- 172.21.15.44:0/2845040767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13c40051c0 msgr2=0x7f13c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.804+0000 7f13e0e55640 1 --2- 172.21.15.44:0/2845040767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13c40051c0 0x7f13c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.804+0000 7f13e0e55640 1 -- 172.21.15.44:0/2845040767 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c4003520 con 0x7f13c4004800 2024-04-02T13:27:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.804+0000 7f13e1e57640 1 -- 172.21.15.44:0/2845040767 reap_dead start 2024-04-02T13:27:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.804+0000 7f13e0e55640 1 --2- 172.21.15.44:0/2845040767 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f13c4004800 0x7f13c4004c80 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009ac0 tx=0x7f13c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ee775e4606a8ba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.804+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0039070 con 0x7f13c4004800 2024-04-02T13:27:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.805+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d002c6b0 con 0x7f13c4004800 2024-04-02T13:27:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.805+0000 7f13dbfff640 1 -- 172.21.15.44:0/2845040767 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0009690 con 0x7f13c4004800 2024-04-02T13:27:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.805+0000 7f13e38ba640 1 -- 172.21.15.44:0/2845040767 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f13c4004800 msgr2=0x7f13c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.805+0000 7f13e38ba640 1 --2- 172.21.15.44:0/2845040767 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f13c4004800 0x7f13c4004c80 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009ac0 tx=0x7f13c4005830 comp rx=0 tx=0).stop 2024-04-02T13:27:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.805+0000 7f13e38ba640 1 -- 172.21.15.44:0/2845040767 shutdown_connections 2024-04-02T13:27:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.805+0000 7f13e38ba640 1 --2- 172.21.15.44:0/2845040767 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f13c4004800 0x7f13c4004c80 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.805+0000 7f13e38ba640 1 -- 172.21.15.44:0/2845040767 >> 172.21.15.44:0/2845040767 conn(0x7f13dc09d3b0 msgr2=0x7f13dc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.805+0000 7f13e38ba640 1 -- 172.21.15.44:0/2845040767 shutdown_connections 2024-04-02T13:27:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.805+0000 7f13e38ba640 1 -- 172.21.15.44:0/2845040767 wait complete. 2024-04-02T13:27:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.806+0000 7f13e38ba640 1 Processor -- start 2024-04-02T13:27:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.806+0000 7f13e38ba640 1 -- start start 2024-04-02T13:27:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e38ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f13c4004800 0x7f13dc15b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e38ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13dc160a30 0x7f13dc09e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e38ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13dc151190 0x7f13dc0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e38ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc0d1280 con 0x7f13c4004800 2024-04-02T13:27:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e38ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc0d1100 con 0x7f13dc151190 2024-04-02T13:27:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e38ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f13dc0d1400 con 0x7f13dc160a30 2024-04-02T13:27:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e1656640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f13c4004800 0x7f13dc15b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e1656640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f13c4004800 0x7f13dc15b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52248/0 (socket says 172.21.15.44:52248) 2024-04-02T13:27:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e1656640 1 -- 172.21.15.44:0/777811887 learned_addr learned my addr 172.21.15.44:0/777811887 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e1e57640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13dc151190 0x7f13dc0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e0e55640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13dc160a30 0x7f13dc09e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e1656640 1 -- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13dc160a30 msgr2=0x7f13dc09e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e1656640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13dc160a30 0x7f13dc09e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e1656640 1 -- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13dc151190 msgr2=0x7f13dc0d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.807+0000 7f13e1656640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13dc151190 0x7f13dc0d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.808+0000 7f13e1656640 1 -- 172.21.15.44:0/777811887 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13dc161180 con 0x7f13c4004800 2024-04-02T13:27:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.808+0000 7f13e1656640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f13c4004800 0x7f13dc15b820 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f13d40072c0 tx=0x7f13d4007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.808+0000 7f13d9ffb640 1 -- 172.21.15.44:0/777811887 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d400fa20 con 0x7f13c4004800 2024-04-02T13:27:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.808+0000 7f13d9ffb640 1 -- 172.21.15.44:0/777811887 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d4018070 con 0x7f13c4004800 2024-04-02T13:27:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.808+0000 7f13e38ba640 1 -- 172.21.15.44:0/777811887 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13dc161460 con 0x7f13c4004800 2024-04-02T13:27:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.808+0000 7f13d9ffb640 1 -- 172.21.15.44:0/777811887 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d400fe20 con 0x7f13c4004800 2024-04-02T13:27:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.808+0000 7f13e38ba640 1 -- 172.21.15.44:0/777811887 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13dc09e230 con 0x7f13c4004800 2024-04-02T13:27:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.809+0000 7f13d9ffb640 1 -- 172.21.15.44:0/777811887 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f13d401d460 con 0x7f13c4004800 2024-04-02T13:27:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.810+0000 7f13d9ffb640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f13c806c920 0x7f13c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.810+0000 7f13d9ffb640 1 -- 172.21.15.44:0/777811887 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7f13d408bea0 con 0x7f13c4004800 2024-04-02T13:27:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.810+0000 7f13e0e55640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f13c806c920 0x7f13c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.810+0000 7f13e0e55640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f13c806c920 0x7f13c806ede0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f13d0036000 tx=0x7f13d0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:02.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.811+0000 7f13e657d740 1 Processor -- start 2024-04-02T13:27:02.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.811+0000 7f13e657d740 1 -- start start 2024-04-02T13:27:02.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.811+0000 7f13e657d740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7420 0x55a0893ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.811+0000 7f13e657d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893fa1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.811+0000 7f13e657d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893ca900 0x55a0893fa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.811+0000 7f13e657d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a089333430 con 0x55a0893ca900 2024-04-02T13:27:02.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.811+0000 7f13e657d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a0892d1a40 con 0x55a0893f7800 2024-04-02T13:27:02.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.811+0000 7f13e657d740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a08932f3e0 con 0x55a0893f7420 2024-04-02T13:27:02.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13e1e57640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893ca900 0x55a0893fa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13e1e57640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893ca900 0x55a0893fa6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52258/0 (socket says 172.21.15.44:52258) 2024-04-02T13:27:02.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13e1656640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7420 0x55a0893ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13e1e57640 1 -- 172.21.15.44:0/4232901696 learned_addr learned my addr 172.21.15.44:0/4232901696 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13e0e55640 1 --2- 172.21.15.44:0/4232901696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893fa1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13e1e57640 1 -- 172.21.15.44:0/4232901696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7420 msgr2=0x55a0893ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13e1e57640 1 --2- 172.21.15.44:0/4232901696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7420 0x55a0893ca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13e1e57640 1 -- 172.21.15.44:0/4232901696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 msgr2=0x55a0893fa1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13e1e57640 1 --2- 172.21.15.44:0/4232901696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893fa1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13e1e57640 1 -- 172.21.15.44:0/4232901696 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0892fb960 con 0x55a0893ca900 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13e0e55640 1 --2- 172.21.15.44:0/4232901696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893fa1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13e1e57640 1 --2- 172.21.15.44:0/4232901696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893ca900 0x55a0893fa6f0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f13dc08e610 tx=0x7f13cc001610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eedb2ff334ff4bd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13c2ffd640 1 -- 172.21.15.44:0/4232901696 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc01b070 con 0x55a0893ca900 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13c2ffd640 1 -- 172.21.15.44:0/4232901696 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13cc005280 con 0x55a0893ca900 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.812+0000 7f13c2ffd640 1 -- 172.21.15.44:0/4232901696 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc005650 con 0x55a0893ca900 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.813+0000 7f13e657d740 1 -- 172.21.15.44:0/4232901696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893ca900 msgr2=0x55a0893fa6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.813+0000 7f13e657d740 1 --2- 172.21.15.44:0/4232901696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893ca900 0x55a0893fa6f0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f13dc08e610 tx=0x7f13cc001610 comp rx=0 tx=0).stop 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.813+0000 7f13e657d740 1 -- 172.21.15.44:0/4232901696 shutdown_connections 2024-04-02T13:27:02.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.813+0000 7f13e657d740 1 --2- 172.21.15.44:0/4232901696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893ca900 0x55a0893fa6f0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.813+0000 7f13e657d740 1 --2- 172.21.15.44:0/4232901696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893fa1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.813+0000 7f13e657d740 1 --2- 172.21.15.44:0/4232901696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7420 0x55a0893ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.813+0000 7f13e657d740 1 -- 172.21.15.44:0/4232901696 >> 172.21.15.44:0/4232901696 conn(0x55a0893cc630 msgr2=0x55a0894151b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.813+0000 7f13e657d740 1 -- 172.21.15.44:0/4232901696 shutdown_connections 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.813+0000 7f13e657d740 1 -- 172.21.15.44:0/4232901696 wait complete. 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.813+0000 7f13e657d740 1 -- start start 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.814+0000 7f13e657d740 1 Processor -- start 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.814+0000 7f13e657d740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7420 0x55a0893f46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.814+0000 7f13e657d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893fa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.814+0000 7f13e657d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0894008e0 0x55a089400d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.814+0000 7f13e657d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a0893cecd0 con 0x55a0894008e0 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.814+0000 7f13e657d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a0892cf710 con 0x55a0893f7800 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.814+0000 7f13e657d740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a0893d1510 con 0x55a0893f7420 2024-04-02T13:27:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e657d740 10 client.0 _ll_register_callbacks cb 0x55a089046b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e1e57640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0894008e0 0x55a089400d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e1e57640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0894008e0 0x55a089400d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52266/0 (socket says 172.21.15.44:52266) 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e1e57640 1 -- 172.21.15.44:0/3728670352 learned_addr learned my addr 172.21.15.44:0/3728670352 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e0e55640 1 --2- 172.21.15.44:0/3728670352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893fa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e1656640 1 --2- 172.21.15.44:0/3728670352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7420 0x55a0893f46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e1e57640 1 -- 172.21.15.44:0/3728670352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7420 msgr2=0x55a0893f46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e1e57640 1 --2- 172.21.15.44:0/3728670352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7420 0x55a0893f46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e1e57640 1 -- 172.21.15.44:0/3728670352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 msgr2=0x55a0893fa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e1e57640 1 --2- 172.21.15.44:0/3728670352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893fa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e1e57640 1 -- 172.21.15.44:0/3728670352 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0891137b0 con 0x55a0894008e0 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e1e57640 1 --2- 172.21.15.44:0/3728670352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0894008e0 0x55a089400d60 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f13cc001270 tx=0x7f13cc001510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8876100edc5a6735 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13bbfff640 1 -- 172.21.15.44:0/3728670352 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc01b040 con 0x55a0894008e0 2024-04-02T13:27:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13bbfff640 1 -- 172.21.15.44:0/3728670352 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7f13cc006b50 con 0x55a0894008e0 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e657d740 1 -- 172.21.15.44:0/3728670352 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13cc005280 con 0x55a0894008e0 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13bbfff640 1 -- 172.21.15.44:0/3728670352 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f13cc02d050 con 0x55a0894008e0 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e657d740 10 client.8163 fetch_fsmap learned FSMap version 67 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13e657d740 1 -- 172.21.15.44:0/3728670352 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x55a089173f40 con 0x55a0894008e0 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.815+0000 7f13bbfff640 1 -- 172.21.15.44:0/3728670352 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13cc005280 con 0x55a0894008e0 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.816+0000 7f13bbfff640 1 -- 172.21.15.44:0/3728670352 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7f13cc030020 con 0x55a0894008e0 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.816+0000 7f13e657d740 10 client.8163 fetch_fsmap finished waiting for FSMap version 67 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.816+0000 7f13e657d740 10 client.8163 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.816+0000 7f13e657d740 20 client.8163 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.816+0000 7f13e657d740 1 --2- 172.21.15.44:0/3728670352 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a0893f8810 0x55a0893f8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.816+0000 7f13e657d740 4 client.8163 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.816+0000 7f13e657d740 1 -- 172.21.15.44:0/3728670352 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a089340970 con 0x55a0893f8810 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.816+0000 7f13e1656640 1 --2- 172.21.15.44:0/3728670352 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a0893f8810 0x55a0893f8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.816+0000 7f13bbfff640 10 client.8163 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.816+0000 7f13e1656640 1 --2- 172.21.15.44:0/3728670352 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a0893f8810 0x55a0893f8c20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=896f9b4c56bd207c server_cookie=f494f0feea63f3ba in_seq=0 out_seq=0 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13bbfff640 1 -- 172.21.15.44:0/3728670352 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a089340970 con 0x55a0893f8810 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13bbfff640 10 client.8163 handle_command_reply: tid=0 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13e657d740 1 client.8163 shutdown 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13e657d740 1 -- 172.21.15.44:0/3728670352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0894008e0 msgr2=0x55a089400d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13e657d740 1 --2- 172.21.15.44:0/3728670352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0894008e0 0x55a089400d60 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f13cc001270 tx=0x7f13cc001510 comp rx=0 tx=0).stop 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13e657d740 1 -- 172.21.15.44:0/3728670352 shutdown_connections 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13e657d740 1 --2- 172.21.15.44:0/3728670352 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a0893f8810 0x55a0893f8c20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13e657d740 1 --2- 172.21.15.44:0/3728670352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0894008e0 0x55a089400d60 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13e657d740 1 --2- 172.21.15.44:0/3728670352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893fa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13bbfff640 0 client.8163 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13e657d740 1 --2- 172.21.15.44:0/3728670352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7420 0x55a0893f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13e657d740 1 -- 172.21.15.44:0/3728670352 >> 172.21.15.44:0/3728670352 conn(0x55a0893ca900 msgr2=0x55a0893ce2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13e657d740 1 -- 172.21.15.44:0/3728670352 shutdown_connections 2024-04-02T13:27:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13e657d740 1 -- 172.21.15.44:0/3728670352 wait complete. 2024-04-02T13:27:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.817+0000 7f13e657d740 20 client.8163 trim_cache size 0 max 16384 2024-04-02T13:27:02.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.835+0000 7f13e657d740 1 Processor -- start 2024-04-02T13:27:02.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.835+0000 7f13e657d740 1 -- start start 2024-04-02T13:27:02.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.835+0000 7f13e657d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 0x55a089428ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.835+0000 7f13e657d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.835+0000 7f13e657d740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0894008e0 0x55a0894238d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.835+0000 7f13e657d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a089401730 con 0x55a0893f7420 2024-04-02T13:27:02.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.835+0000 7f13e657d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc201400 con 0x55a0893f7800 2024-04-02T13:27:02.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.835+0000 7f13e657d740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f13dc200420 con 0x55a0894008e0 2024-04-02T13:27:02.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13e1656640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 0x55a089428ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13e1656640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 0x55a089428ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52272/0 (socket says 172.21.15.44:52272) 2024-04-02T13:27:02.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13e1656640 1 -- 172.21.15.44:0/1416494533 learned_addr learned my addr 172.21.15.44:0/1416494533 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:02.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13e0e55640 1 --2- 172.21.15.44:0/1416494533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13e1e57640 1 --2- 172.21.15.44:0/1416494533 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0894008e0 0x55a0894238d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13e1656640 1 -- 172.21.15.44:0/1416494533 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0894008e0 msgr2=0x55a0894238d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13e1656640 1 --2- 172.21.15.44:0/1416494533 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0894008e0 0x55a0894238d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13e1656640 1 -- 172.21.15.44:0/1416494533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 msgr2=0x55a0893ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13e1656640 1 --2- 172.21.15.44:0/1416494533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13e1656640 1 -- 172.21.15.44:0/1416494533 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0893f4a50 con 0x55a0893f7420 2024-04-02T13:27:02.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13e0e55640 1 --2- 172.21.15.44:0/1416494533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:27:02.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13e1656640 1 --2- 172.21.15.44:0/1416494533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 0x55a089428ba0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f13d4007290 tx=0x7f13d408a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c4d2142a67a0c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:02.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13c27fc640 1 -- 172.21.15.44:0/1416494533 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d405de30 con 0x55a0893f7420 2024-04-02T13:27:02.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13c27fc640 1 -- 172.21.15.44:0/1416494533 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d4062660 con 0x55a0893f7420 2024-04-02T13:27:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.836+0000 7f13c27fc640 1 -- 172.21.15.44:0/1416494533 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d4057d80 con 0x55a0893f7420 2024-04-02T13:27:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.837+0000 7f13e657d740 1 -- 172.21.15.44:0/1416494533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 msgr2=0x55a089428ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.837+0000 7f13e657d740 1 --2- 172.21.15.44:0/1416494533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 0x55a089428ba0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f13d4007290 tx=0x7f13d408a150 comp rx=0 tx=0).stop 2024-04-02T13:27:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.837+0000 7f13e657d740 1 -- 172.21.15.44:0/1416494533 shutdown_connections 2024-04-02T13:27:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.837+0000 7f13e657d740 1 --2- 172.21.15.44:0/1416494533 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0894008e0 0x55a0894238d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.837+0000 7f13e657d740 1 --2- 172.21.15.44:0/1416494533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0893f7800 0x55a0893ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.837+0000 7f13e657d740 1 --2- 172.21.15.44:0/1416494533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 0x55a089428ba0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.837+0000 7f13e657d740 1 -- 172.21.15.44:0/1416494533 >> 172.21.15.44:0/1416494533 conn(0x55a0893ca900 msgr2=0x55a0893c56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.837+0000 7f13e657d740 1 -- 172.21.15.44:0/1416494533 shutdown_connections 2024-04-02T13:27:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.837+0000 7f13e657d740 1 -- 172.21.15.44:0/1416494533 wait complete. 2024-04-02T13:27:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.837+0000 7f13e657d740 1 -- start start 2024-04-02T13:27:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.837+0000 7f13e657d740 1 Processor -- start 2024-04-02T13:27:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.838+0000 7f13e657d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 0x55a089418b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.838+0000 7f13e657d740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7800 0x55a08948e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.838+0000 7f13e657d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0894008e0 0x55a089414ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.838+0000 7f13e657d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc1ffe00 con 0x55a0893f7420 2024-04-02T13:27:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.838+0000 7f13e657d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc15f730 con 0x55a0894008e0 2024-04-02T13:27:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.838+0000 7f13e657d740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a0893f40c0 con 0x55a0893f7800 2024-04-02T13:27:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.838+0000 7f13e657d740 10 client.0 _ll_register_callbacks cb 0x7f13cc003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.838+0000 7f13e1656640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 0x55a089418b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.838+0000 7f13e1656640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 0x55a089418b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52278/0 (socket says 172.21.15.44:52278) 2024-04-02T13:27:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.838+0000 7f13e1656640 1 -- 172.21.15.44:0/2341436417 learned_addr learned my addr 172.21.15.44:0/2341436417 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.838+0000 7f13e1e57640 1 --2- 172.21.15.44:0/2341436417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0894008e0 0x55a089414ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.838+0000 7f13e0e55640 1 --2- 172.21.15.44:0/2341436417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7800 0x55a08948e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e1656640 1 -- 172.21.15.44:0/2341436417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7800 msgr2=0x55a08948e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e1656640 1 --2- 172.21.15.44:0/2341436417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7800 0x55a08948e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e1656640 1 -- 172.21.15.44:0/2341436417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0894008e0 msgr2=0x55a089414ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e1656640 1 --2- 172.21.15.44:0/2341436417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0894008e0 0x55a089414ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e1656640 1 -- 172.21.15.44:0/2341436417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0890070a0 con 0x55a0893f7420 2024-04-02T13:27:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e1656640 1 --2- 172.21.15.44:0/2341436417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 0x55a089418b30 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f13d400f5a0 tx=0x7f13cc005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ae7dcb58cfd83e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:27:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13bbfff640 1 -- 172.21.15.44:0/2341436417 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d40560f0 con 0x55a0893f7420 2024-04-02T13:27:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13bbfff640 1 -- 172.21.15.44:0/2341436417 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7f13d4060070 con 0x55a0893f7420 2024-04-02T13:27:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e657d740 1 -- 172.21.15.44:0/2341436417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13d4062660 con 0x55a0893f7420 2024-04-02T13:27:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13bbfff640 1 -- 172.21.15.44:0/2341436417 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f13d4056290 con 0x55a0893f7420 2024-04-02T13:27:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e657d740 10 client.8169 fetch_fsmap learned FSMap version 67 2024-04-02T13:27:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e657d740 1 -- 172.21.15.44:0/2341436417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x55a089418610 con 0x55a0893f7420 2024-04-02T13:27:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13bbfff640 1 -- 172.21.15.44:0/2341436417 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d4062660 con 0x55a0893f7420 2024-04-02T13:27:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13bbfff640 1 -- 172.21.15.44:0/2341436417 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7f13d4058ed0 con 0x55a0893f7420 2024-04-02T13:27:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e657d740 10 client.8169 fetch_fsmap finished waiting for FSMap version 67 2024-04-02T13:27:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e657d740 10 client.8169 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:27:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e657d740 20 client.8169 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e657d740 1 --2- 172.21.15.44:0/2341436417 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a089415220 0x55a0893f05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e657d740 4 client.8169 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive"]}] 2024-04-02T13:27:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e657d740 1 -- 172.21.15.44:0/2341436417 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive"]}) -- 0x55a08933acc0 con 0x55a089415220 2024-04-02T13:27:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.839+0000 7f13e0e55640 1 --2- 172.21.15.44:0/2341436417 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a089415220 0x55a0893f05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13e0e55640 1 --2- 172.21.15.44:0/2341436417 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a089415220 0x55a0893f05d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50a6b0841c83eb7b server_cookie=97357ec815596475 in_seq=0 out_seq=0 2024-04-02T13:27:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13bbfff640 10 client.8169 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13bbfff640 1 -- 172.21.15.44:0/2341436417 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a08933acc0 con 0x55a089415220 2024-04-02T13:27:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13bbfff640 10 client.8169 handle_command_reply: tid=0 2024-04-02T13:27:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13e657d740 1 client.8169 shutdown 2024-04-02T13:27:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13e657d740 1 -- 172.21.15.44:0/2341436417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 msgr2=0x55a089418b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13e657d740 1 --2- 172.21.15.44:0/2341436417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 0x55a089418b30 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f13d400f5a0 tx=0x7f13cc005000 comp rx=0 tx=0).stop 2024-04-02T13:27:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13e657d740 1 -- 172.21.15.44:0/2341436417 shutdown_connections 2024-04-02T13:27:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13e657d740 1 --2- 172.21.15.44:0/2341436417 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a089415220 0x55a0893f05d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13e657d740 1 --2- 172.21.15.44:0/2341436417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0894008e0 0x55a089414ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13e657d740 1 --2- 172.21.15.44:0/2341436417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0893f7800 0x55a08948e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13e657d740 1 --2- 172.21.15.44:0/2341436417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0893f7420 0x55a089418b30 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13e657d740 1 -- 172.21.15.44:0/2341436417 >> 172.21.15.44:0/2341436417 conn(0x55a0893ca900 msgr2=0x55a0893cdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:02.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13bbfff640 0 client.8169 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:02.855 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:27:02.840+0000 7fe0b0a2e640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 ap=1 f(v0 m2024-04-02T13:26:51.014993+0000 1=1+0) n(v0 rc2024-04-02T13:26:51.131857+0000 2=1+1)/n(v0 rc2024-04-02T13:26:51.014993+0000 2=1+1) | dirfrag=0 dirtyrstat=1 authpin=1 scrubqueue=0 0x5647a570fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-04-02T13:26:51.014993+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-04-02T13:26:51.131857+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2} 2024-04-02T13:27:02.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.840+0000 7f13e657d740 1 -- 172.21.15.44:0/2341436417 shutdown_connections 2024-04-02T13:27:02.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e657d740 1 -- 172.21.15.44:0/2341436417 wait complete. 2024-04-02T13:27:02.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e657d740 20 client.8169 trim_cache size 0 max 16384 2024-04-02T13:27:02.857 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:27:02.857 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-02T13:27:02.857 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "4900ac83-1f26-4bec-bc9e-c07e989ab81b", 2024-04-02T13:27:02.857 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-02T13:27:02.857 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:27:02.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e38ba640 1 -- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f13c806c920 msgr2=0x7f13c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e38ba640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f13c806c920 0x7f13c806ede0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f13d0036000 tx=0x7f13d0030000 comp rx=0 tx=0).stop 2024-04-02T13:27:02.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e38ba640 1 -- 172.21.15.44:0/777811887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f13c4004800 msgr2=0x7f13dc15b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e38ba640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f13c4004800 0x7f13dc15b820 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f13d40072c0 tx=0x7f13d4007780 comp rx=0 tx=0).stop 2024-04-02T13:27:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e38ba640 1 -- 172.21.15.44:0/777811887 shutdown_connections 2024-04-02T13:27:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e38ba640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f13c806c920 0x7f13c806ede0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e38ba640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13dc151190 0x7f13dc0d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e38ba640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13dc160a30 0x7f13dc09e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e38ba640 1 --2- 172.21.15.44:0/777811887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f13c4004800 0x7f13dc15b820 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e38ba640 1 -- 172.21.15.44:0/777811887 >> 172.21.15.44:0/777811887 conn(0x7f13dc09d3b0 msgr2=0x7f13dc0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e38ba640 1 -- 172.21.15.44:0/777811887 shutdown_connections 2024-04-02T13:27:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.841+0000 7f13e38ba640 1 -- 172.21.15.44:0/777811887 wait complete. 2024-04-02T13:27:02.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:02.861 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-02T13:27:02.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.961+0000 7fa17339d640 1 Processor -- start 2024-04-02T13:27:02.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.961+0000 7fa17339d640 1 -- start start 2024-04-02T13:27:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.962+0000 7fa17339d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa16c161ac0 0x7fa16c0b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.962+0000 7fa17339d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa16c0d1ce0 0x7fa16c0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.962+0000 7fa17339d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16c0d2a30 con 0x7fa16c1616e0 2024-04-02T13:27:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.962+0000 7fa17339d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa16c0d2d10 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.962+0000 7fa17339d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa16c0cee60 con 0x7fa16c161ac0 2024-04-02T13:27:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.962+0000 7fa172b9c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa16c0d1ce0 0x7fa16c0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.962+0000 7fa17239b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa16c1616e0 0x7fa16c0d2170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37678/0 (socket says 172.21.15.44:37678) 2024-04-02T13:27:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.962+0000 7fa17239b640 1 -- 172.21.15.44:0/1571125340 learned_addr learned my addr 172.21.15.44:0/1571125340 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:27:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.962+0000 7fa171b9a640 1 --2- 172.21.15.44:0/1571125340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa16c161ac0 0x7fa16c0b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.963+0000 7fa172b9c640 1 -- 172.21.15.44:0/1571125340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa16c161ac0 msgr2=0x7fa16c0b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.963+0000 7fa172b9c640 1 --2- 172.21.15.44:0/1571125340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa16c161ac0 0x7fa16c0b0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.963+0000 7fa172b9c640 1 -- 172.21.15.44:0/1571125340 >> v1:172.21.15.44:6789/0 conn(0x7fa16c1616e0 legacy=0x7fa16c0d2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.963+0000 7fa171399640 1 -- 172.21.15.44:0/1571125340 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3857293002 0 0) 0x7fa16c0d2a30 con 0x7fa16c1616e0 2024-04-02T13:27:02.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.963+0000 7fa172b9c640 1 -- 172.21.15.44:0/1571125340 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16c0cf140 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.963+0000 7fa172b9c640 1 --2- 172.21.15.44:0/1571125340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c0d1ce0 0x7fa16c0cf660 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fa15c009d90 tx=0x7fa15c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8512f9312d026f35 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:02.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.963+0000 7fa171399640 1 -- 172.21.15.44:0/1571125340 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c039070 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.963+0000 7fa171399640 1 -- 172.21.15.44:0/1571125340 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa15c02cc50 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.963+0000 7fa171399640 1 -- 172.21.15.44:0/1571125340 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c034440 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.964+0000 7fa17339d640 1 -- 172.21.15.44:0/1571125340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c0d1ce0 msgr2=0x7fa16c0cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.964+0000 7fa17339d640 1 --2- 172.21.15.44:0/1571125340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c0d1ce0 0x7fa16c0cf660 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fa15c009d90 tx=0x7fa15c02bea0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.964+0000 7fa17339d640 1 -- 172.21.15.44:0/1571125340 shutdown_connections 2024-04-02T13:27:02.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.964+0000 7fa17339d640 1 --2- 172.21.15.44:0/1571125340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c0d1ce0 0x7fa16c0cf660 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.964+0000 7fa17339d640 1 --2- 172.21.15.44:0/1571125340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa16c161ac0 0x7fa16c0b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.964+0000 7fa17339d640 1 -- 172.21.15.44:0/1571125340 >> 172.21.15.44:0/1571125340 conn(0x7fa16c09d210 msgr2=0x7fa16c0d3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:02.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.964+0000 7fa17339d640 1 -- 172.21.15.44:0/1571125340 shutdown_connections 2024-04-02T13:27:02.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.964+0000 7fa17339d640 1 -- 172.21.15.44:0/1571125340 wait complete. 2024-04-02T13:27:02.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.965+0000 7fa17339d640 1 Processor -- start 2024-04-02T13:27:02.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.965+0000 7fa17339d640 1 -- start start 2024-04-02T13:27:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17339d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c0d1ce0 0x7fa16c0ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17339d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa16c1616e0 0x7fa16c0c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17339d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa16c161ac0 0x7fa16c0c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17339d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c0b1230 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17339d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c0b10b0 con 0x7fa16c1616e0 2024-04-02T13:27:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17339d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa16c0b13b0 con 0x7fa16c161ac0 2024-04-02T13:27:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17239b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c0d1ce0 0x7fa16c0ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17239b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c0d1ce0 0x7fa16c0ca8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52302/0 (socket says 172.21.15.44:52302) 2024-04-02T13:27:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17239b640 1 -- 172.21.15.44:0/3162524353 learned_addr learned my addr 172.21.15.44:0/3162524353 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa172b9c640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa16c161ac0 0x7fa16c0c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa171b9a640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa16c1616e0 0x7fa16c0c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17239b640 1 -- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa16c161ac0 msgr2=0x7fa16c0c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17239b640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa16c161ac0 0x7fa16c0c9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17239b640 1 -- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa16c1616e0 msgr2=0x7fa16c0c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17239b640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa16c1616e0 0x7fa16c0c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa17239b640 1 -- 172.21.15.44:0/3162524353 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16c0c8200 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa171b9a640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa16c1616e0 0x7fa16c0c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:27:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.966+0000 7fa172b9c640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa16c161ac0 0x7fa16c0c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:27:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.967+0000 7fa17239b640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c0d1ce0 0x7fa16c0ca8d0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fa16400c4d0 tx=0x7fa16400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.967+0000 7fa16affd640 1 -- 172.21.15.44:0/3162524353 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164018070 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.967+0000 7fa16affd640 1 -- 172.21.15.44:0/3162524353 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa164007500 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.967+0000 7fa16affd640 1 -- 172.21.15.44:0/3162524353 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164004a20 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.967+0000 7fa17339d640 1 -- 172.21.15.44:0/3162524353 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa16c1f76e0 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.967+0000 7fa17339d640 1 -- 172.21.15.44:0/3162524353 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa16c1f7ca0 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.968+0000 7fa16affd640 1 -- 172.21.15.44:0/3162524353 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa16401b070 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.969+0000 7fa16affd640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa15806c8f0 0x7fa15806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.969+0000 7fa16affd640 1 -- 172.21.15.44:0/3162524353 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7fa16408afd0 con 0x7fa16c0d1ce0 2024-04-02T13:27:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.969+0000 7fa171b9a640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa15806c8f0 0x7fa15806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.969+0000 7fa171b9a640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa15806c8f0 0x7fa15806edb0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa16c15d0b0 tx=0x7fa160008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:02.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.970+0000 7fa1771e7740 1 Processor -- start 2024-04-02T13:27:02.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.970+0000 7fa1771e7740 1 -- start start 2024-04-02T13:27:02.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa1771e7740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0a9bb0 0x56481a07cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa1771e7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9f90 0x56481a0ac940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa1771e7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a07d090 0x56481a0ace80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa1771e7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564819fe6830 con 0x56481a07d090 2024-04-02T13:27:02.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa1771e7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564819f8e820 con 0x56481a0a9f90 2024-04-02T13:27:02.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa1771e7740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564819fe27e0 con 0x56481a0a9bb0 2024-04-02T13:27:02.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa172b9c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a07d090 0x56481a0ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa172b9c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a07d090 0x56481a0ace80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52316/0 (socket says 172.21.15.44:52316) 2024-04-02T13:27:02.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa172b9c640 1 -- 172.21.15.44:0/2801516499 learned_addr learned my addr 172.21.15.44:0/2801516499 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:02.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa171b9a640 1 --2- 172.21.15.44:0/2801516499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9f90 0x56481a0ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa17239b640 1 --2- 172.21.15.44:0/2801516499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0a9bb0 0x56481a07cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa172b9c640 1 -- 172.21.15.44:0/2801516499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0a9bb0 msgr2=0x56481a07cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa172b9c640 1 --2- 172.21.15.44:0/2801516499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0a9bb0 0x56481a07cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa172b9c640 1 -- 172.21.15.44:0/2801516499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9f90 msgr2=0x56481a0ac940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa172b9c640 1 --2- 172.21.15.44:0/2801516499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9f90 0x56481a0ac940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa172b9c640 1 -- 172.21.15.44:0/2801516499 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564819dc47d0 con 0x56481a07d090 2024-04-02T13:27:02.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.971+0000 7fa172b9c640 1 --2- 172.21.15.44:0/2801516499 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a07d090 0x56481a0ace80 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fa15c037040 tx=0x7fa15c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce378554057fc48 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:02.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.972+0000 7fa153fff640 1 -- 172.21.15.44:0/2801516499 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c039040 con 0x56481a07d090 2024-04-02T13:27:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.972+0000 7fa153fff640 1 -- 172.21.15.44:0/2801516499 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa15c02cbd0 con 0x56481a07d090 2024-04-02T13:27:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.972+0000 7fa153fff640 1 -- 172.21.15.44:0/2801516499 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c03f660 con 0x56481a07d090 2024-04-02T13:27:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.972+0000 7fa1771e7740 1 -- 172.21.15.44:0/2801516499 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a07d090 msgr2=0x56481a0ace80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.972+0000 7fa1771e7740 1 --2- 172.21.15.44:0/2801516499 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a07d090 0x56481a0ace80 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fa15c037040 tx=0x7fa15c02e000 comp rx=0 tx=0).stop 2024-04-02T13:27:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.972+0000 7fa1771e7740 1 -- 172.21.15.44:0/2801516499 shutdown_connections 2024-04-02T13:27:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.972+0000 7fa1771e7740 1 --2- 172.21.15.44:0/2801516499 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a07d090 0x56481a0ace80 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.972+0000 7fa1771e7740 1 --2- 172.21.15.44:0/2801516499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9f90 0x56481a0ac940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.972+0000 7fa1771e7740 1 --2- 172.21.15.44:0/2801516499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0a9bb0 0x56481a07cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.972+0000 7fa1771e7740 1 -- 172.21.15.44:0/2801516499 >> 172.21.15.44:0/2801516499 conn(0x56481a07edc0 msgr2=0x56481a0c7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.972+0000 7fa1771e7740 1 -- 172.21.15.44:0/2801516499 shutdown_connections 2024-04-02T13:27:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.972+0000 7fa1771e7740 1 -- 172.21.15.44:0/2801516499 wait complete. 2024-04-02T13:27:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.973+0000 7fa1771e7740 1 -- start start 2024-04-02T13:27:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.973+0000 7fa1771e7740 1 Processor -- start 2024-04-02T13:27:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa1771e7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9bb0 0x56481a09ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa1771e7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9f90 0x56481a0ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa1771e7740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0ae900 0x56481a0c0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa1771e7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56481a081360 con 0x56481a0a9bb0 2024-04-02T13:27:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa1771e7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564819fa1d30 con 0x56481a0a9f90 2024-04-02T13:27:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa1771e7740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x56481a080750 con 0x56481a0ae900 2024-04-02T13:27:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa1771e7740 10 client.0 _ll_register_callbacks cb 0x564819fa1720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa17239b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9bb0 0x56481a09ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa17239b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9bb0 0x56481a09ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52326/0 (socket says 172.21.15.44:52326) 2024-04-02T13:27:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa17239b640 1 -- 172.21.15.44:0/3984400750 learned_addr learned my addr 172.21.15.44:0/3984400750 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa171b9a640 1 --2- 172.21.15.44:0/3984400750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9f90 0x56481a0ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa172b9c640 1 --2- 172.21.15.44:0/3984400750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0ae900 0x56481a0c0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa17239b640 1 -- 172.21.15.44:0/3984400750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0ae900 msgr2=0x56481a0c0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa17239b640 1 --2- 172.21.15.44:0/3984400750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0ae900 0x56481a0c0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa17239b640 1 -- 172.21.15.44:0/3984400750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9f90 msgr2=0x56481a0ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa17239b640 1 --2- 172.21.15.44:0/3984400750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9f90 0x56481a0ae3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.974+0000 7fa17239b640 1 -- 172.21.15.44:0/3984400750 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564819cfa3f0 con 0x56481a0a9bb0 2024-04-02T13:27:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa17239b640 1 --2- 172.21.15.44:0/3984400750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9bb0 0x56481a09ce60 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fa164055010 tx=0x7fa164058bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c4d4fcc5e4b5c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1517fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:27:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1517fa640 1 -- 172.21.15.44:0/3984400750 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164071020 con 0x56481a0a9bb0 2024-04-02T13:27:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1517fa640 1 -- 172.21.15.44:0/3984400750 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7fa164004dc0 con 0x56481a0a9bb0 2024-04-02T13:27:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1771e7740 1 -- 172.21.15.44:0/3984400750 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa15c02cbd0 con 0x56481a0a9bb0 2024-04-02T13:27:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1517fa640 1 -- 172.21.15.44:0/3984400750 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fa1640890d0 con 0x56481a0a9bb0 2024-04-02T13:27:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1771e7740 10 client.8184 fetch_fsmap learned FSMap version 67 2024-04-02T13:27:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1771e7740 1 -- 172.21.15.44:0/3984400750 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x564819cba0f0 con 0x56481a0a9bb0 2024-04-02T13:27:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1517fa640 1 -- 172.21.15.44:0/3984400750 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa15c02cbd0 con 0x56481a0a9bb0 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1517fa640 1 -- 172.21.15.44:0/3984400750 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7fa164058000 con 0x56481a0a9bb0 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1771e7740 10 client.8184 fetch_fsmap finished waiting for FSMap version 67 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1771e7740 10 client.8184 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1771e7740 20 client.8184 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1771e7740 1 --2- 172.21.15.44:0/3984400750 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x56481a0db3e0 0x56481a0db800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1771e7740 4 client.8184 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa1771e7740 1 -- 172.21.15.44:0/3984400750 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564819ff3d70 con 0x56481a0db3e0 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.975+0000 7fa171b9a640 1 --2- 172.21.15.44:0/3984400750 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x56481a0db3e0 0x56481a0db800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa171b9a640 1 --2- 172.21.15.44:0/3984400750 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x56481a0db3e0 0x56481a0db800 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61c9150d29e7694e server_cookie=342ce4420dcb0a7 in_seq=0 out_seq=0 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa1517fa640 10 client.8184 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa1517fa640 1 -- 172.21.15.44:0/3984400750 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564819ff3d70 con 0x56481a0db3e0 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa1517fa640 10 client.8184 handle_command_reply: tid=0 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa1771e7740 1 client.8184 shutdown 2024-04-02T13:27:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa1771e7740 1 -- 172.21.15.44:0/3984400750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9bb0 msgr2=0x56481a09ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa1771e7740 1 --2- 172.21.15.44:0/3984400750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9bb0 0x56481a09ce60 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fa164055010 tx=0x7fa164058bb0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa1771e7740 1 -- 172.21.15.44:0/3984400750 shutdown_connections 2024-04-02T13:27:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa1771e7740 1 --2- 172.21.15.44:0/3984400750 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x56481a0db3e0 0x56481a0db800 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa1771e7740 1 --2- 172.21.15.44:0/3984400750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0ae900 0x56481a0c0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa1771e7740 1 --2- 172.21.15.44:0/3984400750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9f90 0x56481a0ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa1517fa640 0 client.8184 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa1771e7740 1 --2- 172.21.15.44:0/3984400750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9bb0 0x56481a09ce60 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.976+0000 7fa1771e7740 1 -- 172.21.15.44:0/3984400750 >> 172.21.15.44:0/3984400750 conn(0x56481a07d090 msgr2=0x56481a0ac930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.977+0000 7fa1771e7740 1 -- 172.21.15.44:0/3984400750 shutdown_connections 2024-04-02T13:27:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.977+0000 7fa1771e7740 1 -- 172.21.15.44:0/3984400750 wait complete. 2024-04-02T13:27:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.977+0000 7fa1771e7740 20 client.8184 trim_cache size 0 max 16384 2024-04-02T13:27:02.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.994+0000 7fa1771e7740 1 Processor -- start 2024-04-02T13:27:02.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.994+0000 7fa1771e7740 1 -- start start 2024-04-02T13:27:03.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa1771e7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9bb0 0x56481a0ac440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:03.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa1771e7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9f90 0x56481a0ac980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:03.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa1771e7740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0ae900 0x56481a0bd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:03.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa1771e7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c200c30 con 0x56481a0a9f90 2024-04-02T13:27:03.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa1771e7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c1ffc70 con 0x56481a0a9bb0 2024-04-02T13:27:03.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa1771e7740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa16c0b3c00 con 0x56481a0ae900 2024-04-02T13:27:03.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa171b9a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9f90 0x56481a0ac980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:03.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa171b9a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9f90 0x56481a0ac980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52334/0 (socket says 172.21.15.44:52334) 2024-04-02T13:27:03.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa171b9a640 1 -- 172.21.15.44:0/838846142 learned_addr learned my addr 172.21.15.44:0/838846142 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:03.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa17239b640 1 --2- 172.21.15.44:0/838846142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9bb0 0x56481a0ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:03.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa172b9c640 1 --2- 172.21.15.44:0/838846142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0ae900 0x56481a0bd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:03.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa171b9a640 1 -- 172.21.15.44:0/838846142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0ae900 msgr2=0x56481a0bd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:03.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa171b9a640 1 --2- 172.21.15.44:0/838846142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0ae900 0x56481a0bd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa171b9a640 1 -- 172.21.15.44:0/838846142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9bb0 msgr2=0x56481a0ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:03.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa171b9a640 1 --2- 172.21.15.44:0/838846142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9bb0 0x56481a0ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.995+0000 7fa171b9a640 1 -- 172.21.15.44:0/838846142 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56481a0ae730 con 0x56481a0a9f90 2024-04-02T13:27:03.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.996+0000 7fa171b9a640 1 --2- 172.21.15.44:0/838846142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9f90 0x56481a0ac980 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0b44b0 tx=0x7fa160014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65800466d943697c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:03.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.996+0000 7fa1537fe640 1 -- 172.21.15.44:0/838846142 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa160015ad0 con 0x56481a0a9f90 2024-04-02T13:27:03.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.996+0000 7fa1537fe640 1 -- 172.21.15.44:0/838846142 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa16001a040 con 0x56481a0a9f90 2024-04-02T13:27:03.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.996+0000 7fa1537fe640 1 -- 172.21.15.44:0/838846142 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa160015c70 con 0x56481a0a9f90 2024-04-02T13:27:03.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.996+0000 7fa1771e7740 1 -- 172.21.15.44:0/838846142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9f90 msgr2=0x56481a0ac980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:03.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.996+0000 7fa1771e7740 1 --2- 172.21.15.44:0/838846142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9f90 0x56481a0ac980 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0b44b0 tx=0x7fa160014000 comp rx=0 tx=0).stop 2024-04-02T13:27:03.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.996+0000 7fa1771e7740 1 -- 172.21.15.44:0/838846142 shutdown_connections 2024-04-02T13:27:03.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.996+0000 7fa1771e7740 1 --2- 172.21.15.44:0/838846142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0ae900 0x56481a0bd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.996+0000 7fa1771e7740 1 --2- 172.21.15.44:0/838846142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0a9f90 0x56481a0ac980 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.996+0000 7fa1771e7740 1 --2- 172.21.15.44:0/838846142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9bb0 0x56481a0ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.996+0000 7fa1771e7740 1 -- 172.21.15.44:0/838846142 >> 172.21.15.44:0/838846142 conn(0x56481a07d090 msgr2=0x56481a0d6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:03.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.996+0000 7fa1771e7740 1 -- 172.21.15.44:0/838846142 shutdown_connections 2024-04-02T13:27:03.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.996+0000 7fa1771e7740 1 -- 172.21.15.44:0/838846142 wait complete. 2024-04-02T13:27:03.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.997+0000 7fa1771e7740 1 -- start start 2024-04-02T13:27:03.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.997+0000 7fa1771e7740 1 Processor -- start 2024-04-02T13:27:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa1771e7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9bb0 0x56481a1413f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa1771e7740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0a9f90 0x56481a0eb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa1771e7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0ae900 0x56481a0eb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa1771e7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56481a078e90 con 0x56481a0ae900 2024-04-02T13:27:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa1771e7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c1ff670 con 0x56481a0a9bb0 2024-04-02T13:27:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa1771e7740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x56481a07ceb0 con 0x56481a0a9f90 2024-04-02T13:27:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa1771e7740 10 client.0 _ll_register_callbacks cb 0x7fa164058600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa172b9c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0ae900 0x56481a0eb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa172b9c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0ae900 0x56481a0eb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52336/0 (socket says 172.21.15.44:52336) 2024-04-02T13:27:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa172b9c640 1 -- 172.21.15.44:0/3430653874 learned_addr learned my addr 172.21.15.44:0/3430653874 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa171b9a640 1 --2- 172.21.15.44:0/3430653874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0a9f90 0x56481a0eb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa17239b640 1 --2- 172.21.15.44:0/3430653874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9bb0 0x56481a1413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa172b9c640 1 -- 172.21.15.44:0/3430653874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0a9f90 msgr2=0x56481a0eb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa172b9c640 1 --2- 172.21.15.44:0/3430653874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0a9f90 0x56481a0eb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa172b9c640 1 -- 172.21.15.44:0/3430653874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9bb0 msgr2=0x56481a1413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa172b9c640 1 --2- 172.21.15.44:0/3430653874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9bb0 0x56481a1413f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa172b9c640 1 -- 172.21.15.44:0/3430653874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56481a0af040 con 0x56481a0ae900 2024-04-02T13:27:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa172b9c640 1 --2- 172.21.15.44:0/3430653874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0ae900 0x56481a0eb780 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0095c0 tx=0x7fa15c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d951b70ba25d632 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.998+0000 7fa1517fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:27:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1517fa640 1 -- 172.21.15.44:0/3430653874 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c039040 con 0x56481a0ae900 2024-04-02T13:27:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1517fa640 1 -- 172.21.15.44:0/3430653874 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 6187+0+0 (secure 0 0 0) 0x7fa15c051c80 con 0x56481a0ae900 2024-04-02T13:27:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1771e7740 1 -- 172.21.15.44:0/3430653874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa16001a040 con 0x56481a0ae900 2024-04-02T13:27:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1517fa640 1 -- 172.21.15.44:0/3430653874 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fa15c058050 con 0x56481a0ae900 2024-04-02T13:27:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1771e7740 10 client.8190 fetch_fsmap learned FSMap version 67 2024-04-02T13:27:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1771e7740 1 -- 172.21.15.44:0/3430653874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x56481a0c4c20 con 0x56481a0ae900 2024-04-02T13:27:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1517fa640 1 -- 172.21.15.44:0/3430653874 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa16001a040 con 0x56481a0ae900 2024-04-02T13:27:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1517fa640 1 -- 172.21.15.44:0/3430653874 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7fa15c05b020 con 0x56481a0ae900 2024-04-02T13:27:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1771e7740 10 client.8190 fetch_fsmap finished waiting for FSMap version 67 2024-04-02T13:27:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1771e7740 10 client.8190 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:27:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1771e7740 20 client.8190 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1771e7740 1 --2- 172.21.15.44:0/3430653874 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x56481a07c180 0x56481a0af500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1771e7740 4 client.8190 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:27:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1771e7740 1 -- 172.21.15.44:0/3430653874 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564819fee0c0 con 0x56481a07c180 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa17239b640 1 --2- 172.21.15.44:0/3430653874 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x56481a07c180 0x56481a0af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa1517fa640 10 client.8190 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:02.999+0000 7fa17239b640 1 --2- 172.21.15.44:0/3430653874 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x56481a07c180 0x56481a0af500 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eb55f5ad03adb1c server_cookie=cba526a76088444c in_seq=0 out_seq=0 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1517fa640 1 -- 172.21.15.44:0/3430653874 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+256 (crc 0 0 0) 0x564819fee0c0 con 0x56481a07c180 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1517fa640 10 client.8190 handle_command_reply: tid=0 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1771e7740 1 client.8190 shutdown 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1771e7740 1 -- 172.21.15.44:0/3430653874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0ae900 msgr2=0x56481a0eb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1771e7740 1 --2- 172.21.15.44:0/3430653874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0ae900 0x56481a0eb780 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0095c0 tx=0x7fa15c004ab0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1771e7740 1 -- 172.21.15.44:0/3430653874 shutdown_connections 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1771e7740 1 --2- 172.21.15.44:0/3430653874 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x56481a07c180 0x56481a0af500 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1771e7740 1 --2- 172.21.15.44:0/3430653874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56481a0ae900 0x56481a0eb780 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1771e7740 1 --2- 172.21.15.44:0/3430653874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56481a0a9f90 0x56481a0eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1771e7740 1 --2- 172.21.15.44:0/3430653874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56481a0a9bb0 0x56481a1413f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1517fa640 0 client.8190 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:03.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1771e7740 1 -- 172.21.15.44:0/3430653874 >> 172.21.15.44:0/3430653874 conn(0x56481a07d090 msgr2=0x56481a0a9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:03.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1771e7740 1 -- 172.21.15.44:0/3430653874 shutdown_connections 2024-04-02T13:27:03.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1771e7740 1 -- 172.21.15.44:0/3430653874 wait complete. 2024-04-02T13:27:03.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa1771e7740 20 client.8190 trim_cache size 0 max 16384 2024-04-02T13:27:03.011 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:27:03.011 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T13:27:03.011 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-02T13:27:03.011 INFO:teuthology.orchestra.run.smithi044.stdout: "4900ac83-1f26-4bec-bc9e-c07e989ab81b": { 2024-04-02T13:27:03.011 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/dir", 2024-04-02T13:27:03.011 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "4900ac83-1f26-4bec-bc9e-c07e989ab81b", 2024-04-02T13:27:03.012 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive" 2024-04-02T13:27:03.012 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:27:03.012 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:27:03.012 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:27:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa17339d640 1 -- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa15806c8f0 msgr2=0x7fa15806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa17339d640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa15806c8f0 0x7fa15806edb0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa16c15d0b0 tx=0x7fa160008040 comp rx=0 tx=0).stop 2024-04-02T13:27:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa17339d640 1 -- 172.21.15.44:0/3162524353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c0d1ce0 msgr2=0x7fa16c0ca8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.000+0000 7fa17339d640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c0d1ce0 0x7fa16c0ca8d0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fa16400c4d0 tx=0x7fa16400c990 comp rx=0 tx=0).stop 2024-04-02T13:27:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.001+0000 7fa17339d640 1 -- 172.21.15.44:0/3162524353 shutdown_connections 2024-04-02T13:27:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.001+0000 7fa17339d640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa15806c8f0 0x7fa15806edb0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.001+0000 7fa17339d640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa16c161ac0 0x7fa16c0c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.001+0000 7fa17339d640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa16c1616e0 0x7fa16c0c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.001+0000 7fa17339d640 1 --2- 172.21.15.44:0/3162524353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c0d1ce0 0x7fa16c0ca8d0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.001+0000 7fa17339d640 1 -- 172.21.15.44:0/3162524353 >> 172.21.15.44:0/3162524353 conn(0x7fa16c09d210 msgr2=0x7fa16c067230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.001+0000 7fa17339d640 1 -- 172.21.15.44:0/3162524353 shutdown_connections 2024-04-02T13:27:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:03.001+0000 7fa17339d640 1 -- 172.21.15.44:0/3162524353 wait complete. 2024-04-02T13:27:03.014 INFO:tasks.cephfs.filesystem:scrub status for tag:4900ac83-1f26-4bec-bc9e-c07e989ab81b - {'path': '/dir', 'tag': '4900ac83-1f26-4bec-bc9e-c07e989ab81b', 'options': 'recursive'} 2024-04-02T13:27:06.615 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:27:06.612+0000 7fe12a6e8640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-02T13:27:22.975 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:27:22.979 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:27:33.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:33.015 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-02T13:27:33.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.128+0000 7f01fd55f640 1 Processor -- start 2024-04-02T13:27:33.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.128+0000 7f01fd55f640 1 -- start start 2024-04-02T13:27:33.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.128+0000 7f01fd55f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f80d0a30 0x7f01f80b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.128+0000 7f01fd55f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f80d0e10 con 0x7f01f8144130 2024-04-02T13:27:33.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.129+0000 7f01fd55f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f8144750 con 0x7f01f80b3990 2024-04-02T13:27:33.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.129+0000 7f01fd55f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f01f80b3df0 con 0x7f01f80d0a30 2024-04-02T13:27:33.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.129+0000 7f01f77fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f01f8144130 0x7f01f815e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52608/0 (socket says 172.21.15.44:52608) 2024-04-02T13:27:33.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.129+0000 7f01f77fe640 1 -- 172.21.15.44:0/2740260553 learned_addr learned my addr 172.21.15.44:0/2740260553 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:27:33.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.129+0000 7f01f6ffd640 1 --2- 172.21.15.44:0/2740260553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f80d0a30 0x7f01f80b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.129+0000 7f01f5ffb640 1 -- 172.21.15.44:0/2740260553 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1432866557 0 0) 0x7f01f0001490 con 0x7f01f8144130 2024-04-02T13:27:33.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.129+0000 7f01f5ffb640 1 -- 172.21.15.44:0/2740260553 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01e0003520 con 0x7f01f8144130 2024-04-02T13:27:33.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.129+0000 7f01f5ffb640 1 -- 172.21.15.44:0/2740260553 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 754477562 0 0) 0x7f01f8144750 con 0x7f01f80b3990 2024-04-02T13:27:33.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.129+0000 7f01f5ffb640 1 -- 172.21.15.44:0/2740260553 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01f0001490 con 0x7f01f80b3990 2024-04-02T13:27:33.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.129+0000 7f01f6ffd640 1 -- 172.21.15.44:0/2740260553 >> v1:172.21.15.69:6789/0 conn(0x7f01f80b3990 legacy=0x7f01f8143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.129+0000 7f01f6ffd640 1 -- 172.21.15.44:0/2740260553 >> v1:172.21.15.44:6789/0 conn(0x7f01f8144130 legacy=0x7f01f815e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.129+0000 7f01f6ffd640 1 -- 172.21.15.44:0/2740260553 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f8162060 con 0x7f01f80d0a30 2024-04-02T13:27:33.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.130+0000 7f01f6ffd640 1 --2- 172.21.15.44:0/2740260553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f80d0a30 0x7f01f80b3450 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f01e80097d0 tx=0x7f01e802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e510754322c6eb77 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.130+0000 7f01f5ffb640 1 -- 172.21.15.44:0/2740260553 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8035020 con 0x7f01f80d0a30 2024-04-02T13:27:33.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.130+0000 7f01f5ffb640 1 -- 172.21.15.44:0/2740260553 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01e8031800 con 0x7f01f80d0a30 2024-04-02T13:27:33.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.130+0000 7f01f5ffb640 1 -- 172.21.15.44:0/2740260553 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8031bb0 con 0x7f01f80d0a30 2024-04-02T13:27:33.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.131+0000 7f01fd55f640 1 -- 172.21.15.44:0/2740260553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f80d0a30 msgr2=0x7f01f80b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.131+0000 7f01fd55f640 1 --2- 172.21.15.44:0/2740260553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f80d0a30 0x7f01f80b3450 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f01e80097d0 tx=0x7f01e802b8a0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.131+0000 7f01fd55f640 1 -- 172.21.15.44:0/2740260553 shutdown_connections 2024-04-02T13:27:33.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.131+0000 7f01fd55f640 1 --2- 172.21.15.44:0/2740260553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f80d0a30 0x7f01f80b3450 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.131+0000 7f01fd55f640 1 -- 172.21.15.44:0/2740260553 >> 172.21.15.44:0/2740260553 conn(0x7f01f80b5d90 msgr2=0x7f01f806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.131+0000 7f01fd55f640 1 -- 172.21.15.44:0/2740260553 shutdown_connections 2024-04-02T13:27:33.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.131+0000 7f01fd55f640 1 -- 172.21.15.44:0/2740260553 wait complete. 2024-04-02T13:27:33.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.132+0000 7f01fd55f640 1 Processor -- start 2024-04-02T13:27:33.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.132+0000 7f01fd55f640 1 -- start start 2024-04-02T13:27:33.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01fd55f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01f80b3990 0x7f01f80b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01fd55f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01f80d0a30 0x7f01f81f4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01fd55f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f8144130 0x7f01f81475e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01fd55f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f01f8162620 con 0x7f01f80b3990 2024-04-02T13:27:33.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01fd55f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f01f8161e10 con 0x7f01f80d0a30 2024-04-02T13:27:33.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01fd55f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f01f8162b20 con 0x7f01f8144130 2024-04-02T13:27:33.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01f6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01f80b3990 0x7f01f80b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01f6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01f80b3990 0x7f01f80b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52314/0 (socket says 172.21.15.44:52314) 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01f6ffd640 1 -- 172.21.15.44:0/1581077074 learned_addr learned my addr 172.21.15.44:0/1581077074 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01f67fc640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01f80d0a30 0x7f01f81f4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01f77fe640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f8144130 0x7f01f81475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01f6ffd640 1 -- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f8144130 msgr2=0x7f01f81475e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01f6ffd640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f8144130 0x7f01f81475e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01f6ffd640 1 -- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01f80d0a30 msgr2=0x7f01f81f4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01f6ffd640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01f80d0a30 0x7f01f81f4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.133+0000 7f01f6ffd640 1 -- 172.21.15.44:0/1581077074 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f81f4780 con 0x7f01f80b3990 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.134+0000 7f01f6ffd640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01f80b3990 0x7f01f80b0b20 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f01e80098f0 tx=0x7f01e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.134+0000 7f01df7fe640 1 -- 172.21.15.44:0/1581077074 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8035040 con 0x7f01f80b3990 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.134+0000 7f01df7fe640 1 -- 172.21.15.44:0/1581077074 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01e803ad00 con 0x7f01f80b3990 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.134+0000 7f01df7fe640 1 -- 172.21.15.44:0/1581077074 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e80466c0 con 0x7f01f80b3990 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.134+0000 7f01fd55f640 1 -- 172.21.15.44:0/1581077074 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f80b1370 con 0x7f01f80b3990 2024-04-02T13:27:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.134+0000 7f01fd55f640 1 -- 172.21.15.44:0/1581077074 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f80b1930 con 0x7f01f80b3990 2024-04-02T13:27:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.136+0000 7f01df7fe640 1 -- 172.21.15.44:0/1581077074 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f01e8038020 con 0x7f01f80b3990 2024-04-02T13:27:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.136+0000 7f01df7fe640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f01e406c8f0 0x7f01e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.136+0000 7f01df7fe640 1 -- 172.21.15.44:0/1581077074 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f01e80ade80 con 0x7f01f80b3990 2024-04-02T13:27:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.136+0000 7f01f67fc640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f01e406c8f0 0x7f01e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.137+0000 7f020021f740 1 Processor -- start 2024-04-02T13:27:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.137+0000 7f01f67fc640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f01e406c8f0 0x7f01e406edb0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f01f80b4ce0 tx=0x7f01ec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.137+0000 7f020021f740 1 -- start start 2024-04-02T13:27:33.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f020021f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb19a0 0x557bf6b84940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f020021f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb1d80 0x557bf6bb4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f020021f740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6b84e80 0x557bf6bb4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f020021f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557bf6aee830 con 0x557bf6bb1d80 2024-04-02T13:27:33.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f020021f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557bf6a96820 con 0x557bf6bb19a0 2024-04-02T13:27:33.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f020021f740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x557bf6aea7e0 con 0x557bf6b84e80 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f01f67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb1d80 0x557bf6bb4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f01f67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb1d80 0x557bf6bb4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52320/0 (socket says 172.21.15.44:52320) 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f01f67fc640 1 -- 172.21.15.44:0/2896879947 learned_addr learned my addr 172.21.15.44:0/2896879947 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f01f6ffd640 1 --2- 172.21.15.44:0/2896879947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb19a0 0x557bf6b84940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f01f77fe640 1 --2- 172.21.15.44:0/2896879947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6b84e80 0x557bf6bb4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f01f67fc640 1 -- 172.21.15.44:0/2896879947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6b84e80 msgr2=0x557bf6bb4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f01f67fc640 1 --2- 172.21.15.44:0/2896879947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6b84e80 0x557bf6bb4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f01f67fc640 1 -- 172.21.15.44:0/2896879947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb19a0 msgr2=0x557bf6b84940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f01f67fc640 1 --2- 172.21.15.44:0/2896879947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb19a0 0x557bf6b84940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f01f67fc640 1 -- 172.21.15.44:0/2896879947 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bf68cc7d0 con 0x557bf6bb1d80 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.138+0000 7f01f67fc640 1 --2- 172.21.15.44:0/2896879947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb1d80 0x557bf6bb4730 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0052f0 tx=0x7f01ec011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ce69115772128 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.139+0000 7f01dcff9640 1 -- 172.21.15.44:0/2896879947 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec00e500 con 0x557bf6bb1d80 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.139+0000 7f01dcff9640 1 -- 172.21.15.44:0/2896879947 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01ec00e6a0 con 0x557bf6bb1d80 2024-04-02T13:27:33.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.139+0000 7f01dcff9640 1 -- 172.21.15.44:0/2896879947 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec00ea70 con 0x557bf6bb1d80 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.139+0000 7f020021f740 1 -- 172.21.15.44:0/2896879947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb1d80 msgr2=0x557bf6bb4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.139+0000 7f020021f740 1 --2- 172.21.15.44:0/2896879947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb1d80 0x557bf6bb4730 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0052f0 tx=0x7f01ec011000 comp rx=0 tx=0).stop 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.139+0000 7f020021f740 1 -- 172.21.15.44:0/2896879947 shutdown_connections 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.139+0000 7f020021f740 1 --2- 172.21.15.44:0/2896879947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6b84e80 0x557bf6bb4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.139+0000 7f020021f740 1 --2- 172.21.15.44:0/2896879947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb1d80 0x557bf6bb4730 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.139+0000 7f020021f740 1 --2- 172.21.15.44:0/2896879947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb19a0 0x557bf6b84940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.139+0000 7f020021f740 1 -- 172.21.15.44:0/2896879947 >> 172.21.15.44:0/2896879947 conn(0x557bf6b86bb0 msgr2=0x557bf6bcf730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.139+0000 7f020021f740 1 -- 172.21.15.44:0/2896879947 shutdown_connections 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.139+0000 7f020021f740 1 -- 172.21.15.44:0/2896879947 wait complete. 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.140+0000 7f020021f740 1 -- start start 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.140+0000 7f020021f740 1 Processor -- start 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f020021f740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6bb19a0 0x557bf6be03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f020021f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6ba2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f020021f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 0x557bf6bc6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f020021f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557bf6b89150 con 0x557bf6bc9260 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f020021f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557bf6aa9d30 con 0x557bf6bb1d80 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f020021f740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x557bf6b88540 con 0x557bf6bb19a0 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f020021f740 10 client.0 _ll_register_callbacks cb 0x557bf6aa9720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f01f77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 0x557bf6bc6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f01f6ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6bb19a0 0x557bf6be03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f01f77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 0x557bf6bc6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52328/0 (socket says 172.21.15.44:52328) 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f01f77fe640 1 -- 172.21.15.44:0/1483577252 learned_addr learned my addr 172.21.15.44:0/1483577252 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f01f67fc640 1 --2- 172.21.15.44:0/1483577252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6ba2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f01f77fe640 1 -- 172.21.15.44:0/1483577252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6bb19a0 msgr2=0x557bf6be03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f01f77fe640 1 --2- 172.21.15.44:0/1483577252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6bb19a0 0x557bf6be03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f01f77fe640 1 -- 172.21.15.44:0/1483577252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 msgr2=0x557bf6ba2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f01f77fe640 1 --2- 172.21.15.44:0/1483577252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6ba2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f01f77fe640 1 -- 172.21.15.44:0/1483577252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bf68023f0 con 0x557bf6bc9260 2024-04-02T13:27:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f01f77fe640 1 --2- 172.21.15.44:0/1483577252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 0x557bf6bc6d80 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f01f0011ce0 tx=0x7f01f000f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178a6b4fa2ed4eed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.141+0000 7f01d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f01d5ffb640 1 -- 172.21.15.44:0/1483577252 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f0001e70 con 0x557bf6bc9260 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f01d5ffb640 1 -- 172.21.15.44:0/1483577252 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f01f0002010 con 0x557bf6bc9260 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f020021f740 1 -- 172.21.15.44:0/1483577252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01ec00e6a0 con 0x557bf6bc9260 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f01d5ffb640 1 -- 172.21.15.44:0/1483577252 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f01f0022050 con 0x557bf6bc9260 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f020021f740 10 client.8202 fetch_fsmap learned FSMap version 68 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f020021f740 1 -- 172.21.15.44:0/1483577252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x557bf67c20f0 con 0x557bf6bc9260 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f01d5ffb640 1 -- 172.21.15.44:0/1483577252 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01ec00e6a0 con 0x557bf6bc9260 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f01d5ffb640 1 -- 172.21.15.44:0/1483577252 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f01f0025020 con 0x557bf6bc9260 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f020021f740 10 client.8202 fetch_fsmap finished waiting for FSMap version 68 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f020021f740 10 client.8202 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f020021f740 20 client.8202 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f020021f740 1 --2- 172.21.15.44:0/1483577252 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x557bf6bd3110 0x557bf6bb6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f020021f740 4 client.8202 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:27:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f020021f740 1 -- 172.21.15.44:0/1483577252 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bf6afbd70 con 0x557bf6bd3110 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.142+0000 7f01f6ffd640 1 --2- 172.21.15.44:0/1483577252 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x557bf6bd3110 0x557bf6bb6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f01f6ffd640 1 --2- 172.21.15.44:0/1483577252 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x557bf6bd3110 0x557bf6bb6770 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=315de28b501194c3 server_cookie=4429445551e68c15 in_seq=0 out_seq=0 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f01d5ffb640 10 client.8202 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f01d5ffb640 1 -- 172.21.15.44:0/1483577252 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557bf6afbd70 con 0x557bf6bd3110 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f01d5ffb640 10 client.8202 handle_command_reply: tid=0 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f020021f740 1 client.8202 shutdown 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f020021f740 1 -- 172.21.15.44:0/1483577252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 msgr2=0x557bf6bc6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f020021f740 1 --2- 172.21.15.44:0/1483577252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 0x557bf6bc6d80 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f01f0011ce0 tx=0x7f01f000f5a0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f020021f740 1 -- 172.21.15.44:0/1483577252 shutdown_connections 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f020021f740 1 --2- 172.21.15.44:0/1483577252 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x557bf6bd3110 0x557bf6bb6770 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f020021f740 1 --2- 172.21.15.44:0/1483577252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 0x557bf6bc6d80 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f020021f740 1 --2- 172.21.15.44:0/1483577252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6ba2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f01d5ffb640 0 client.8202 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f020021f740 1 --2- 172.21.15.44:0/1483577252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6bb19a0 0x557bf6be03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f020021f740 1 -- 172.21.15.44:0/1483577252 >> 172.21.15.44:0/1483577252 conn(0x557bf6b84e80 msgr2=0x557bf6bb4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f020021f740 1 -- 172.21.15.44:0/1483577252 shutdown_connections 2024-04-02T13:27:33.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f020021f740 1 -- 172.21.15.44:0/1483577252 wait complete. 2024-04-02T13:27:33.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.143+0000 7f020021f740 20 client.8202 trim_cache size 0 max 16384 2024-04-02T13:27:33.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.161+0000 7f020021f740 1 Processor -- start 2024-04-02T13:27:33.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.161+0000 7f020021f740 1 -- start start 2024-04-02T13:27:33.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f020021f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb19a0 0x557bf6bb4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f020021f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6bcf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f020021f740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6bc9260 0x557bf6b8a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f020021f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f01f8201160 con 0x557bf6bb19a0 2024-04-02T13:27:33.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f020021f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f01f8200140 con 0x557bf6bb1d80 2024-04-02T13:27:33.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f020021f740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f01f8146090 con 0x557bf6bc9260 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f01f6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb19a0 0x557bf6bb4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f01f6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb19a0 0x557bf6bb4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52340/0 (socket says 172.21.15.44:52340) 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f01f6ffd640 1 -- 172.21.15.44:0/248183948 learned_addr learned my addr 172.21.15.44:0/248183948 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f01f6ffd640 1 -- 172.21.15.44:0/248183948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6bc9260 msgr2=0x557bf6b8a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f01f67fc640 1 --2- 172.21.15.44:0/248183948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6bcf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f01f6ffd640 1 --2- 172.21.15.44:0/248183948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6bc9260 0x557bf6b8a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f01f6ffd640 1 -- 172.21.15.44:0/248183948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 msgr2=0x557bf6bcf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f01f6ffd640 1 --2- 172.21.15.44:0/248183948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6bcf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f01f6ffd640 1 -- 172.21.15.44:0/248183948 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bf6bb0d10 con 0x557bf6bb19a0 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f01f67fc640 1 --2- 172.21.15.44:0/248183948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6bcf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.162+0000 7f01f6ffd640 1 --2- 172.21.15.44:0/248183948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb19a0 0x557bf6bb4bd0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f01e807b660 tx=0x7f01e807e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306de0d339cfaaf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.163+0000 7f01d7fff640 1 -- 172.21.15.44:0/248183948 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e804b040 con 0x557bf6bb19a0 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.163+0000 7f01d7fff640 1 -- 172.21.15.44:0/248183948 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01e807fd10 con 0x557bf6bb19a0 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.163+0000 7f01d7fff640 1 -- 172.21.15.44:0/248183948 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8084540 con 0x557bf6bb19a0 2024-04-02T13:27:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.163+0000 7f020021f740 1 -- 172.21.15.44:0/248183948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb19a0 msgr2=0x557bf6bb4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.163+0000 7f020021f740 1 --2- 172.21.15.44:0/248183948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb19a0 0x557bf6bb4bd0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f01e807b660 tx=0x7f01e807e040 comp rx=0 tx=0).stop 2024-04-02T13:27:33.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.163+0000 7f020021f740 1 -- 172.21.15.44:0/248183948 shutdown_connections 2024-04-02T13:27:33.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.163+0000 7f020021f740 1 --2- 172.21.15.44:0/248183948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6bc9260 0x557bf6b8a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.163+0000 7f020021f740 1 --2- 172.21.15.44:0/248183948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6bcf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.163+0000 7f020021f740 1 --2- 172.21.15.44:0/248183948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bb19a0 0x557bf6bb4bd0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.163+0000 7f020021f740 1 -- 172.21.15.44:0/248183948 >> 172.21.15.44:0/248183948 conn(0x557bf6b84e80 msgr2=0x557bf6ba2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.163+0000 7f020021f740 1 -- 172.21.15.44:0/248183948 shutdown_connections 2024-04-02T13:27:33.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.163+0000 7f020021f740 1 -- 172.21.15.44:0/248183948 wait complete. 2024-04-02T13:27:33.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.164+0000 7f020021f740 1 -- start start 2024-04-02T13:27:33.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.164+0000 7f020021f740 1 Processor -- start 2024-04-02T13:27:33.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.164+0000 7f020021f740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6bb19a0 0x557bf6c49f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.164+0000 7f020021f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6c4a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f020021f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 0x557bf6be89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f020021f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557bf6bb7730 con 0x557bf6bc9260 2024-04-02T13:27:33.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f020021f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f01f81f8670 con 0x557bf6bb1d80 2024-04-02T13:27:33.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f020021f740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x557bf6bb10a0 con 0x557bf6bb19a0 2024-04-02T13:27:33.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f020021f740 10 client.0 _ll_register_callbacks cb 0x7f01f0011940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01f77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 0x557bf6be89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01f77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 0x557bf6be89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52346/0 (socket says 172.21.15.44:52346) 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01f77fe640 1 -- 172.21.15.44:0/3180347551 learned_addr learned my addr 172.21.15.44:0/3180347551 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01f67fc640 1 --2- 172.21.15.44:0/3180347551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6c4a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01f77fe640 1 -- 172.21.15.44:0/3180347551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6bb19a0 msgr2=0x557bf6c49f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01f77fe640 1 --2- 172.21.15.44:0/3180347551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6bb19a0 0x557bf6c49f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01f77fe640 1 -- 172.21.15.44:0/3180347551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 msgr2=0x557bf6c4a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01f77fe640 1 --2- 172.21.15.44:0/3180347551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6c4a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01f77fe640 1 -- 172.21.15.44:0/3180347551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bf6bf30b0 con 0x557bf6bc9260 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01f67fc640 1 --2- 172.21.15.44:0/3180347551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6c4a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01f77fe640 1 --2- 172.21.15.44:0/3180347551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 0x557bf6be89b0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f01f0011cb0 tx=0x7f01f0001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a90ee1257bab2f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01d5ffb640 1 -- 172.21.15.44:0/3180347551 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f002b060 con 0x557bf6bc9260 2024-04-02T13:27:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01d5ffb640 1 -- 172.21.15.44:0/3180347551 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f01f001d760 con 0x557bf6bc9260 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f020021f740 1 -- 172.21.15.44:0/3180347551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01e807fd10 con 0x557bf6bc9260 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f01d5ffb640 1 -- 172.21.15.44:0/3180347551 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f01f0011070 con 0x557bf6bc9260 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f020021f740 10 client.8208 fetch_fsmap learned FSMap version 68 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.165+0000 7f020021f740 1 -- 172.21.15.44:0/3180347551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x557bf6c49a20 con 0x557bf6bc9260 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f01d5ffb640 1 -- 172.21.15.44:0/3180347551 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01e807fd10 con 0x557bf6bc9260 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f01d5ffb640 1 -- 172.21.15.44:0/3180347551 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f01f0025040 con 0x557bf6bc9260 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 10 client.8208 fetch_fsmap finished waiting for FSMap version 68 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 10 client.8208 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 20 client.8208 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 1 --2- 172.21.15.44:0/3180347551 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x557bf6c47100 0x557bf6be8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 4 client.8208 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 1 -- 172.21.15.44:0/3180347551 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557bf6af60c0 con 0x557bf6c47100 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f01f6ffd640 1 --2- 172.21.15.44:0/3180347551 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x557bf6c47100 0x557bf6be8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f01d5ffb640 10 client.8208 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f01f6ffd640 1 --2- 172.21.15.44:0/3180347551 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x557bf6c47100 0x557bf6be8ef0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5e4e690762a0329 server_cookie=ae972405cf223c03 in_seq=0 out_seq=0 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f01d5ffb640 1 -- 172.21.15.44:0/3180347551 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557bf6af60c0 con 0x557bf6c47100 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f01d5ffb640 10 client.8208 handle_command_reply: tid=0 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 1 client.8208 shutdown 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 1 -- 172.21.15.44:0/3180347551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 msgr2=0x557bf6be89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 1 --2- 172.21.15.44:0/3180347551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 0x557bf6be89b0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f01f0011cb0 tx=0x7f01f0001250 comp rx=0 tx=0).stop 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 1 -- 172.21.15.44:0/3180347551 shutdown_connections 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 1 --2- 172.21.15.44:0/3180347551 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x557bf6c47100 0x557bf6be8ef0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 1 --2- 172.21.15.44:0/3180347551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bf6bc9260 0x557bf6be89b0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 1 --2- 172.21.15.44:0/3180347551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bf6bb1d80 0x557bf6c4a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f01d5ffb640 0 client.8208 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 1 --2- 172.21.15.44:0/3180347551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557bf6bb19a0 0x557bf6c49f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 1 -- 172.21.15.44:0/3180347551 >> 172.21.15.44:0/3180347551 conn(0x557bf6b84e80 msgr2=0x557bf6bcf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 1 -- 172.21.15.44:0/3180347551 shutdown_connections 2024-04-02T13:27:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.166+0000 7f020021f740 1 -- 172.21.15.44:0/3180347551 wait complete. 2024-04-02T13:27:33.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.167+0000 7f020021f740 20 client.8208 trim_cache size 0 max 16384 2024-04-02T13:27:33.177 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:27:33.177 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-02T13:27:33.177 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-02T13:27:33.177 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:27:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.167+0000 7f01fd55f640 1 -- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f01e406c8f0 msgr2=0x7f01e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.167+0000 7f01fd55f640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f01e406c8f0 0x7f01e406edb0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f01f80b4ce0 tx=0x7f01ec009040 comp rx=0 tx=0).stop 2024-04-02T13:27:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.167+0000 7f01fd55f640 1 -- 172.21.15.44:0/1581077074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01f80b3990 msgr2=0x7f01f80b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.167+0000 7f01fd55f640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01f80b3990 0x7f01f80b0b20 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f01e80098f0 tx=0x7f01e80046c0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.167+0000 7f01fd55f640 1 -- 172.21.15.44:0/1581077074 shutdown_connections 2024-04-02T13:27:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.167+0000 7f01fd55f640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f01e406c8f0 0x7f01e406edb0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.167+0000 7f01fd55f640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f8144130 0x7f01f81475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.167+0000 7f01fd55f640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01f80d0a30 0x7f01f81f4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.167+0000 7f01fd55f640 1 --2- 172.21.15.44:0/1581077074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01f80b3990 0x7f01f80b0b20 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.167+0000 7f01fd55f640 1 -- 172.21.15.44:0/1581077074 >> 172.21.15.44:0/1581077074 conn(0x7f01f80b5d90 msgr2=0x7f01f81600a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.167+0000 7f01fd55f640 1 -- 172.21.15.44:0/1581077074 shutdown_connections 2024-04-02T13:27:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.167+0000 7f01fd55f640 1 -- 172.21.15.44:0/1581077074 wait complete. 2024-04-02T13:27:33.181 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T13:27:33.182 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:27:33.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.280+0000 7fa637488640 1 Processor -- start 2024-04-02T13:27:33.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.280+0000 7fa637488640 1 -- start start 2024-04-02T13:27:33.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.281+0000 7fa637488640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa63015e550 0x7fa63006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.281+0000 7fa637488640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa63015f040 0x7fa6300d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.281+0000 7fa637488640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa63015fb50 con 0x7fa63006d530 2024-04-02T13:27:33.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.281+0000 7fa637488640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa63006d970 con 0x7fa63015e550 2024-04-02T13:27:33.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.281+0000 7fa637488640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa63006ca90 con 0x7fa63015f040 2024-04-02T13:27:33.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.281+0000 7fa635224640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa63015e550 0x7fa63006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.281+0000 7fa635224640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa63015e550 0x7fa63006cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52354/0 (socket says 172.21.15.44:52354) 2024-04-02T13:27:33.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.281+0000 7fa635224640 1 -- 172.21.15.44:0/3379862904 learned_addr learned my addr 172.21.15.44:0/3379862904 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.281+0000 7fa635a25640 1 --2- 172.21.15.44:0/3379862904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa63015f040 0x7fa6300d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.281+0000 7fa635224640 1 -- 172.21.15.44:0/3379862904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa63015f040 msgr2=0x7fa6300d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.281+0000 7fa635224640 1 --2- 172.21.15.44:0/3379862904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa63015f040 0x7fa6300d0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.281+0000 7fa635224640 1 -- 172.21.15.44:0/3379862904 >> v1:172.21.15.69:6789/0 conn(0x7fa63006d530 legacy=0x7fa63015e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.281+0000 7fa635224640 1 -- 172.21.15.44:0/3379862904 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63015f590 con 0x7fa63015e550 2024-04-02T13:27:33.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.282+0000 7fa635224640 1 --2- 172.21.15.44:0/3379862904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa63015e550 0x7fa63006cff0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fa62c009870 tx=0x7fa62c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9e093fb06deda6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.282+0000 7fa627fff640 1 -- 172.21.15.44:0/3379862904 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c034070 con 0x7fa63015e550 2024-04-02T13:27:33.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.282+0000 7fa627fff640 1 -- 172.21.15.44:0/3379862904 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa62c030840 con 0x7fa63015e550 2024-04-02T13:27:33.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.282+0000 7fa627fff640 1 -- 172.21.15.44:0/3379862904 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c030c10 con 0x7fa63015e550 2024-04-02T13:27:33.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.283+0000 7fa637488640 1 -- 172.21.15.44:0/3379862904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa63015e550 msgr2=0x7fa63006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.283+0000 7fa637488640 1 --2- 172.21.15.44:0/3379862904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa63015e550 0x7fa63006cff0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fa62c009870 tx=0x7fa62c02b980 comp rx=0 tx=0).stop 2024-04-02T13:27:33.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.283+0000 7fa637488640 1 -- 172.21.15.44:0/3379862904 shutdown_connections 2024-04-02T13:27:33.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.283+0000 7fa637488640 1 --2- 172.21.15.44:0/3379862904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa63015f040 0x7fa6300d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.283+0000 7fa637488640 1 --2- 172.21.15.44:0/3379862904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa63015e550 0x7fa63006cff0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.283+0000 7fa637488640 1 -- 172.21.15.44:0/3379862904 >> 172.21.15.44:0/3379862904 conn(0x7fa63009aa20 msgr2=0x7fa63009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.283+0000 7fa637488640 1 -- 172.21.15.44:0/3379862904 shutdown_connections 2024-04-02T13:27:33.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.283+0000 7fa637488640 1 -- 172.21.15.44:0/3379862904 wait complete. 2024-04-02T13:27:33.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.284+0000 7fa637488640 1 Processor -- start 2024-04-02T13:27:33.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.284+0000 7fa637488640 1 -- start start 2024-04-02T13:27:33.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.284+0000 7fa637488640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa63006d530 0x7fa630150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa637488640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa63015e550 0x7fa6300af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa637488640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa63015f040 0x7fa63015c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa637488640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa6300d0590 con 0x7fa63015f040 2024-04-02T13:27:33.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa637488640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa6300d02b0 con 0x7fa63015e550 2024-04-02T13:27:33.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa637488640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa6300cc4b0 con 0x7fa63006d530 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa635a25640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa63015f040 0x7fa63015c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa635a25640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa63015f040 0x7fa63015c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52370/0 (socket says 172.21.15.44:52370) 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa635a25640 1 -- 172.21.15.44:0/467895521 learned_addr learned my addr 172.21.15.44:0/467895521 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa634a23640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa63015e550 0x7fa6300af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa635224640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa63006d530 0x7fa630150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa635a25640 1 -- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa63006d530 msgr2=0x7fa630150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa635a25640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa63006d530 0x7fa630150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa635a25640 1 -- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa63015e550 msgr2=0x7fa6300af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa635a25640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa63015e550 0x7fa6300af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa635a25640 1 -- 172.21.15.44:0/467895521 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6300b4d80 con 0x7fa63015f040 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa635a25640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa63015f040 0x7fa63015c040 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fa62800cc30 tx=0x7fa62800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa625ffb640 1 -- 172.21.15.44:0/467895521 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62800ce40 con 0x7fa63015f040 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa625ffb640 1 -- 172.21.15.44:0/467895521 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa628003990 con 0x7fa63015f040 2024-04-02T13:27:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa637488640 1 -- 172.21.15.44:0/467895521 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6300b5060 con 0x7fa63015f040 2024-04-02T13:27:33.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.285+0000 7fa625ffb640 1 -- 172.21.15.44:0/467895521 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628003d40 con 0x7fa63015f040 2024-04-02T13:27:33.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.286+0000 7fa637488640 1 -- 172.21.15.44:0/467895521 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6300b5620 con 0x7fa63015f040 2024-04-02T13:27:33.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.286+0000 7fa637488640 1 -- 172.21.15.44:0/467895521 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6300684f0 con 0x7fa63015f040 2024-04-02T13:27:33.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.287+0000 7fa625ffb640 1 -- 172.21.15.44:0/467895521 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa628003ee0 con 0x7fa63015f040 2024-04-02T13:27:33.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.287+0000 7fa625ffb640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa61406c8f0 0x7fa61406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.287+0000 7fa625ffb640 1 -- 172.21.15.44:0/467895521 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7fa62808c170 con 0x7fa63015f040 2024-04-02T13:27:33.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.288+0000 7fa635224640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa61406c8f0 0x7fa61406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.288+0000 7fa635224640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa61406c8f0 0x7fa61406edb0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa62c036000 tx=0x7fa62c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.292+0000 7fa625ffb640 1 -- 172.21.15.44:0/467895521 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa628014030 con 0x7fa63015f040 2024-04-02T13:27:33.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.515+0000 7fa637488640 1 -- 172.21.15.44:0/467895521 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6300af9d0 con 0x7fa63015f040 2024-04-02T13:27:33.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.516+0000 7fa625ffb640 1 -- 172.21.15.44:0/467895521 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5181 (secure 0 0 0) 0x7fa628059d00 con 0x7fa63015f040 2024-04-02T13:27:33.519 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:27:33.519 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":68,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:27:06.599726+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8073":{"gid":8073,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":6,"addr":"172.21.15.44:6833/2250279975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2250279975},{"type":"v1","addr":"172.21.15.44:6833","nonce":2250279975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8073,"qdb_cluster":[8073]},"id":6}]} 2024-04-02T13:27:33.520 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 68 2024-04-02T13:27:33.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.518+0000 7fa637488640 1 -- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa61406c8f0 msgr2=0x7fa61406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.518+0000 7fa637488640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa61406c8f0 0x7fa61406edb0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa62c036000 tx=0x7fa62c03b040 comp rx=0 tx=0).stop 2024-04-02T13:27:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.518+0000 7fa637488640 1 -- 172.21.15.44:0/467895521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa63015f040 msgr2=0x7fa63015c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.518+0000 7fa637488640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa63015f040 0x7fa63015c040 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fa62800cc30 tx=0x7fa62800a5c0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.518+0000 7fa637488640 1 -- 172.21.15.44:0/467895521 shutdown_connections 2024-04-02T13:27:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.518+0000 7fa637488640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa61406c8f0 0x7fa61406edb0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.518+0000 7fa637488640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa63015f040 0x7fa63015c040 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.518+0000 7fa637488640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa63015e550 0x7fa6300af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.518+0000 7fa637488640 1 --2- 172.21.15.44:0/467895521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa63006d530 0x7fa630150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.518+0000 7fa637488640 1 -- 172.21.15.44:0/467895521 >> 172.21.15.44:0/467895521 conn(0x7fa63009aa20 msgr2=0x7fa63006f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.518+0000 7fa637488640 1 -- 172.21.15.44:0/467895521 shutdown_connections 2024-04-02T13:27:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.518+0000 7fa637488640 1 -- 172.21.15.44:0/467895521 wait complete. 2024-04-02T13:27:33.532 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-04-02T13:27:33.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.632+0000 7f9e66750640 1 Processor -- start 2024-04-02T13:27:33.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.632+0000 7f9e66750640 1 -- start start 2024-04-02T13:27:33.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.633+0000 7f9e66750640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9e6015e4a0 0x7f9e600aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.633+0000 7f9e66750640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9e6006d550 0x7f9e600d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.633+0000 7f9e66750640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e600cc600 con 0x7f9e600d0c50 2024-04-02T13:27:33.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.633+0000 7f9e66750640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f9e600cc8e0 con 0x7f9e6006d550 2024-04-02T13:27:33.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.633+0000 7f9e66750640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9e600ccbc0 con 0x7f9e6015e4a0 2024-04-02T13:27:33.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.633+0000 7f9e64ced640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9e6006d550 0x7f9e600d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.633+0000 7f9e64ced640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9e6006d550 0x7f9e600d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52372/0 (socket says 172.21.15.44:52372) 2024-04-02T13:27:33.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.633+0000 7f9e64ced640 1 -- 172.21.15.44:0/1866799300 learned_addr learned my addr 172.21.15.44:0/1866799300 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.633+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/1866799300 >> v2:172.21.15.69:3300/0 conn(0x7f9e6015e4a0 0x7f9e600aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.633+0000 7f9e64ced640 1 -- 172.21.15.44:0/1866799300 >> v2:172.21.15.69:3300/0 conn(0x7f9e6015e4a0 msgr2=0x7f9e600aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.633+0000 7f9e64ced640 1 --2- 172.21.15.44:0/1866799300 >> v2:172.21.15.69:3300/0 conn(0x7f9e6015e4a0 0x7f9e600aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.633+0000 7f9e64ced640 1 -- 172.21.15.44:0/1866799300 >> v1:172.21.15.69:6789/0 conn(0x7f9e600d0c50 legacy=0x7f9e6006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.633+0000 7f9e64ced640 1 -- 172.21.15.44:0/1866799300 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e600ccea0 con 0x7f9e6006d550 2024-04-02T13:27:33.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.634+0000 7f9e64ced640 1 --2- 172.21.15.44:0/1866799300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e6006d550 0x7f9e600d10b0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009e50 tx=0x7f9e5802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78a537936b6cb9d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.634+0000 7f9e5effd640 1 -- 172.21.15.44:0/1866799300 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5802cb20 con 0x7f9e6006d550 2024-04-02T13:27:33.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.634+0000 7f9e5effd640 1 -- 172.21.15.44:0/1866799300 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e5802ccc0 con 0x7f9e6006d550 2024-04-02T13:27:33.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.634+0000 7f9e5effd640 1 -- 172.21.15.44:0/1866799300 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58034610 con 0x7f9e6006d550 2024-04-02T13:27:33.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.635+0000 7f9e66750640 1 -- 172.21.15.44:0/1866799300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e6006d550 msgr2=0x7f9e600d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.635+0000 7f9e66750640 1 --2- 172.21.15.44:0/1866799300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e6006d550 0x7f9e600d10b0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009e50 tx=0x7f9e5802bf80 comp rx=0 tx=0).stop 2024-04-02T13:27:33.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.635+0000 7f9e66750640 1 -- 172.21.15.44:0/1866799300 shutdown_connections 2024-04-02T13:27:33.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.635+0000 7f9e66750640 1 --2- 172.21.15.44:0/1866799300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e6006d550 0x7f9e600d10b0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.635+0000 7f9e66750640 1 --2- 172.21.15.44:0/1866799300 >> v2:172.21.15.69:3300/0 conn(0x7f9e6015e4a0 0x7f9e600aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.635+0000 7f9e66750640 1 -- 172.21.15.44:0/1866799300 >> 172.21.15.44:0/1866799300 conn(0x7f9e6009aa20 msgr2=0x7f9e6009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.635+0000 7f9e66750640 1 -- 172.21.15.44:0/1866799300 shutdown_connections 2024-04-02T13:27:33.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.635+0000 7f9e66750640 1 -- 172.21.15.44:0/1866799300 wait complete. 2024-04-02T13:27:33.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.636+0000 7f9e66750640 1 Processor -- start 2024-04-02T13:27:33.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.636+0000 7f9e66750640 1 -- start start 2024-04-02T13:27:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e66750640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6006d550 0x7f9e60154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e66750640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e600d0c50 0x7f9e600ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e66750640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e6015e4a0 0x7f9e600cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e66750640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6015e880 con 0x7f9e600d0c50 2024-04-02T13:27:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e66750640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9e600cd460 con 0x7f9e6015e4a0 2024-04-02T13:27:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e66750640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9e600af130 con 0x7f9e6006d550 2024-04-02T13:27:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e5f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e600d0c50 0x7f9e600ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e5f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e600d0c50 0x7f9e600ce8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52384/0 (socket says 172.21.15.44:52384) 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/2611088139 learned_addr learned my addr 172.21.15.44:0/2611088139 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e64ced640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e6015e4a0 0x7f9e600cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6006d550 0x7f9e60154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6006d550 msgr2=0x7f9e60154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e5f7fe640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6006d550 0x7f9e60154890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e6015e4a0 msgr2=0x7f9e600cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e5f7fe640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e6015e4a0 0x7f9e600cee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/2611088139 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e600b68c0 con 0x7f9e600d0c50 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.637+0000 7f9e5f7fe640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e600d0c50 0x7f9e600ce8d0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000c440 tx=0x7f9e50001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.638+0000 7f9e5cff9640 1 -- 172.21.15.44:0/2611088139 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e500052d0 con 0x7f9e600d0c50 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.638+0000 7f9e5cff9640 1 -- 172.21.15.44:0/2611088139 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e50005470 con 0x7f9e600d0c50 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.638+0000 7f9e66750640 1 -- 172.21.15.44:0/2611088139 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e600b5820 con 0x7f9e600d0c50 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.638+0000 7f9e5cff9640 1 -- 172.21.15.44:0/2611088139 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50007260 con 0x7f9e600d0c50 2024-04-02T13:27:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.638+0000 7f9e66750640 1 -- 172.21.15.44:0/2611088139 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e600b5de0 con 0x7f9e600d0c50 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.639+0000 7f9e5cff9640 1 -- 172.21.15.44:0/2611088139 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9e50007400 con 0x7f9e600d0c50 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.639+0000 7f9e5cff9640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9e4006c8f0 0x7f9e4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.640+0000 7f9e5cff9640 1 -- 172.21.15.44:0/2611088139 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f9e5008b1b0 con 0x7f9e600d0c50 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.640+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9e4006c8f0 0x7f9e4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.640+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9e4006c8f0 0x7f9e4006edb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9e600cc080 tx=0x7f9e48009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.640+0000 7f9e6940e740 1 Processor -- start 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.640+0000 7f9e6940e740 1 -- start start 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e6940e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb6fbdc0 0x5644fb6fc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e6940e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 0x5644fb6ffe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e6940e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb7494d0 0x5644fb749950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e6940e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5644fb6c35d0 con 0x5644fb701be0 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e6940e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5644fb6656b0 con 0x5644fb6fbdc0 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e6940e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5644fb603890 con 0x5644fb7494d0 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e5f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 0x5644fb6ffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e5f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 0x5644fb6ffe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52386/0 (socket says 172.21.15.44:52386) 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/1756510959 learned_addr learned my addr 172.21.15.44:0/1756510959 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e64ced640 1 --2- 172.21.15.44:0/1756510959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb7494d0 0x5644fb749950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/1756510959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb6fbdc0 0x5644fb6fc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/1756510959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb7494d0 msgr2=0x5644fb749950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e5f7fe640 1 --2- 172.21.15.44:0/1756510959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb7494d0 0x5644fb749950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/1756510959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb6fbdc0 msgr2=0x5644fb6fc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e5f7fe640 1 --2- 172.21.15.44:0/1756510959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb6fbdc0 0x5644fb6fc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/1756510959 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644fb6321d0 con 0x5644fb701be0 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.641+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/1756510959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb6fbdc0 0x5644fb6fc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.642+0000 7f9e5f7fe640 1 --2- 172.21.15.44:0/1756510959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 0x5644fb6ffe10 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f9e50089520 tx=0x7f9e50000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22be8ffb8ec3f823 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.642+0000 7f9e55ffb640 1 -- 172.21.15.44:0/1756510959 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50056df0 con 0x5644fb701be0 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.642+0000 7f9e55ffb640 1 -- 172.21.15.44:0/1756510959 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e50057050 con 0x5644fb701be0 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.642+0000 7f9e55ffb640 1 -- 172.21.15.44:0/1756510959 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e500780c0 con 0x5644fb701be0 2024-04-02T13:27:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.642+0000 7f9e6940e740 1 -- 172.21.15.44:0/1756510959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 msgr2=0x5644fb6ffe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.642+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1756510959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 0x5644fb6ffe10 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f9e50089520 tx=0x7f9e50000ed0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.642+0000 7f9e6940e740 1 -- 172.21.15.44:0/1756510959 shutdown_connections 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.642+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1756510959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb7494d0 0x5644fb749950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.642+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1756510959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 0x5644fb6ffe10 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.642+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1756510959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb6fbdc0 0x5644fb6fc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.642+0000 7f9e6940e740 1 -- 172.21.15.44:0/1756510959 >> 172.21.15.44:0/1756510959 conn(0x5644fb6fde40 msgr2=0x5644fb6ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.642+0000 7f9e6940e740 1 -- 172.21.15.44:0/1756510959 shutdown_connections 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.642+0000 7f9e6940e740 1 -- 172.21.15.44:0/1756510959 wait complete. 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.643+0000 7f9e6940e740 1 -- start start 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.643+0000 7f9e6940e740 1 Processor -- start 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e6940e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb701be0 0x5644fb741350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e6940e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb7494d0 0x5644fb741890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e6940e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb721280 0x5644fb74a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e6940e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5644fb601200 con 0x5644fb721280 2024-04-02T13:27:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e6940e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5644fb661660 con 0x5644fb7494d0 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e6940e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5644fb728530 con 0x5644fb701be0 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e6940e740 10 client.0 _ll_register_callbacks cb 0x5644fb624040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e64ced640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb721280 0x5644fb74a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e5f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb7494d0 0x5644fb741890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e64ced640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb721280 0x5644fb74a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52388/0 (socket says 172.21.15.44:52388) 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e64ced640 1 -- 172.21.15.44:0/1819961968 learned_addr learned my addr 172.21.15.44:0/1819961968 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/1819961968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb701be0 0x5644fb741350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/1819961968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb701be0 msgr2=0x5644fb741350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e5f7fe640 1 --2- 172.21.15.44:0/1819961968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb701be0 0x5644fb741350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/1819961968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb721280 msgr2=0x5644fb74a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e5f7fe640 1 --2- 172.21.15.44:0/1819961968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb721280 0x5644fb74a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/1819961968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644fb4457c0 con 0x5644fb7494d0 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.644+0000 7f9e64ced640 1 --2- 172.21.15.44:0/1819961968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb721280 0x5644fb74a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:27:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e5f7fe640 1 --2- 172.21.15.44:0/1819961968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb7494d0 0x5644fb741890 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000c410 tx=0x7f9e500893f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71040e113e60c1bb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e3effd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e3effd640 1 -- 172.21.15.44:0/1819961968 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e500553c0 con 0x5644fb7494d0 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e3effd640 1 -- 172.21.15.44:0/1819961968 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f9e50072d80 con 0x5644fb7494d0 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e6940e740 1 -- 172.21.15.44:0/1819961968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e50057050 con 0x5644fb7494d0 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e3effd640 1 -- 172.21.15.44:0/1819961968 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f9e50055560 con 0x5644fb7494d0 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e6940e740 10 client.7246 fetch_fsmap learned FSMap version 68 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e6940e740 1 -- 172.21.15.44:0/1819961968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x5644fb4a5f50 con 0x5644fb7494d0 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e3effd640 1 -- 172.21.15.44:0/1819961968 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e50057050 con 0x5644fb7494d0 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e3effd640 1 -- 172.21.15.44:0/1819961968 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f9e500780c0 con 0x5644fb7494d0 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e6940e740 10 client.7246 fetch_fsmap finished waiting for FSMap version 68 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e6940e740 10 client.7246 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e6940e740 20 client.7246 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1819961968 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5644fb72cc90 0x5644fb72d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e6940e740 4 client.7246 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:27:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e6940e740 1 -- 172.21.15.44:0/1819961968 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644fb678ee0 con 0x5644fb72cc90 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.645+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/1819961968 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5644fb72cc90 0x5644fb72d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/1819961968 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5644fb72cc90 0x5644fb72d070 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e036eb70ea75efd server_cookie=d16627544dab0335 in_seq=0 out_seq=0 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e3effd640 10 client.7246 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e3effd640 1 -- 172.21.15.44:0/1819961968 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5644fb678ee0 con 0x5644fb72cc90 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e3effd640 10 client.7246 handle_command_reply: tid=0 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e6940e740 1 client.7246 shutdown 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e6940e740 1 -- 172.21.15.44:0/1819961968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb7494d0 msgr2=0x5644fb741890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1819961968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb7494d0 0x5644fb741890 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000c410 tx=0x7f9e500893f0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e6940e740 1 -- 172.21.15.44:0/1819961968 shutdown_connections 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1819961968 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5644fb72cc90 0x5644fb72d070 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1819961968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb721280 0x5644fb74a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1819961968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb7494d0 0x5644fb741890 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e3effd640 0 client.7246 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1819961968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb701be0 0x5644fb741350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.646+0000 7f9e6940e740 1 -- 172.21.15.44:0/1819961968 >> 172.21.15.44:0/1819961968 conn(0x5644fb6fbdc0 msgr2=0x5644fb6fb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.647+0000 7f9e6940e740 1 -- 172.21.15.44:0/1819961968 shutdown_connections 2024-04-02T13:27:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.647+0000 7f9e6940e740 1 -- 172.21.15.44:0/1819961968 wait complete. 2024-04-02T13:27:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.647+0000 7f9e6940e740 20 client.7246 trim_cache size 0 max 16384 2024-04-02T13:27:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.660+0000 7f9e6940e740 1 Processor -- start 2024-04-02T13:27:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.660+0000 7f9e6940e740 1 -- start start 2024-04-02T13:27:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e6940e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb6fbdc0 0x5644fb72c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e6940e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb701be0 0x5644fb7288a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e6940e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb721280 0x5644fb700b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e6940e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9e60200370 con 0x5644fb6fbdc0 2024-04-02T13:27:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e6940e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6015fbc0 con 0x5644fb701be0 2024-04-02T13:27:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e6940e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9e601ffd50 con 0x5644fb721280 2024-04-02T13:27:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e5ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb6fbdc0 0x5644fb72c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e5ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb6fbdc0 0x5644fb72c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52390/0 (socket says 172.21.15.44:52390) 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e5ffff640 1 -- 172.21.15.44:0/1763170510 learned_addr learned my addr 172.21.15.44:0/1763170510 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e5ffff640 1 -- 172.21.15.44:0/1763170510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb721280 msgr2=0x5644fb700b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/1763170510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb721280 0x5644fb700b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e5ffff640 1 -- 172.21.15.44:0/1763170510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb701be0 msgr2=0x5644fb7288a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e5f7fe640 1 --2- 172.21.15.44:0/1763170510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb701be0 0x5644fb7288a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/1763170510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb701be0 0x5644fb7288a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e5ffff640 1 -- 172.21.15.44:0/1763170510 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644fb3390b0 con 0x5644fb6fbdc0 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.661+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/1763170510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb6fbdc0 0x5644fb72c070 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009fd0 tx=0x7f9e48018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19681c569e9911dd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.662+0000 7f9e557fa640 1 -- 172.21.15.44:0/1763170510 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e48032070 con 0x5644fb6fbdc0 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.662+0000 7f9e557fa640 1 -- 172.21.15.44:0/1763170510 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e4801b040 con 0x5644fb6fbdc0 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.662+0000 7f9e557fa640 1 -- 172.21.15.44:0/1763170510 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4802c660 con 0x5644fb6fbdc0 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.662+0000 7f9e6940e740 1 -- 172.21.15.44:0/1763170510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb6fbdc0 msgr2=0x5644fb72c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.662+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1763170510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb6fbdc0 0x5644fb72c070 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009fd0 tx=0x7f9e48018000 comp rx=0 tx=0).stop 2024-04-02T13:27:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.662+0000 7f9e6940e740 1 -- 172.21.15.44:0/1763170510 shutdown_connections 2024-04-02T13:27:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.662+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1763170510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb721280 0x5644fb700b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.662+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1763170510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb701be0 0x5644fb7288a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.662+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1763170510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb6fbdc0 0x5644fb72c070 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.662+0000 7f9e6940e740 1 -- 172.21.15.44:0/1763170510 >> 172.21.15.44:0/1763170510 conn(0x5644fb728150 msgr2=0x5644fb6ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.662+0000 7f9e6940e740 1 -- 172.21.15.44:0/1763170510 shutdown_connections 2024-04-02T13:27:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.662+0000 7f9e6940e740 1 -- 172.21.15.44:0/1763170510 wait complete. 2024-04-02T13:27:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.663+0000 7f9e6940e740 1 -- start start 2024-04-02T13:27:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.663+0000 7f9e6940e740 1 Processor -- start 2024-04-02T13:27:33.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e6940e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb6fbdc0 0x5644fb748500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e6940e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 0x5644fb6fae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e6940e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb721280 0x5644fb76a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e6940e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5644fb661660 con 0x5644fb701be0 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e6940e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5644fb728530 con 0x5644fb721280 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e6940e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5644fb601200 con 0x5644fb6fbdc0 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e6940e740 10 client.0 _ll_register_callbacks cb 0x7f9e58009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e5f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 0x5644fb6fae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e5f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 0x5644fb6fae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52406/0 (socket says 172.21.15.44:52406) 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/1079544469 learned_addr learned my addr 172.21.15.44:0/1079544469 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e64ced640 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb721280 0x5644fb76a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb6fbdc0 0x5644fb748500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/1079544469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb6fbdc0 msgr2=0x5644fb748500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e5f7fe640 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb6fbdc0 0x5644fb748500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/1079544469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb721280 msgr2=0x5644fb76a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e5f7fe640 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb721280 0x5644fb76a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e5f7fe640 1 -- 172.21.15.44:0/1079544469 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644fb6fb3a0 con 0x5644fb701be0 2024-04-02T13:27:33.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e64ced640 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb721280 0x5644fb76a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb6fbdc0 0x5644fb748500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e5f7fe640 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 0x5644fb6fae60 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f9e50061350 tx=0x7f9e50000f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c5ef80fcc644361 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e3effd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e3effd640 1 -- 172.21.15.44:0/1079544469 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e500553c0 con 0x5644fb701be0 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e3effd640 1 -- 172.21.15.44:0/1079544469 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f9e50013d20 con 0x5644fb701be0 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.664+0000 7f9e6940e740 1 -- 172.21.15.44:0/1079544469 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e4801b040 con 0x5644fb701be0 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e3effd640 1 -- 172.21.15.44:0/1079544469 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f9e50058b20 con 0x5644fb701be0 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e6940e740 10 client.8232 fetch_fsmap learned FSMap version 68 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e6940e740 1 -- 172.21.15.44:0/1079544469 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x5644fb6fb3a0 con 0x5644fb701be0 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e3effd640 1 -- 172.21.15.44:0/1079544469 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e4801b040 con 0x5644fb701be0 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e3effd640 1 -- 172.21.15.44:0/1079544469 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f9e500581a0 con 0x5644fb701be0 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e6940e740 10 client.8232 fetch_fsmap finished waiting for FSMap version 68 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e6940e740 10 client.8232 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:27:33.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e6940e740 20 client.8232 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5644fb7494d0 0x5644fb7bac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e6940e740 4 client.8232 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e6940e740 1 -- 172.21.15.44:0/1079544469 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x5644fb6064d0 con 0x5644fb7494d0 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5644fb7494d0 0x5644fb7bac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e3effd640 10 client.8232 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.665+0000 7f9e5ffff640 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5644fb7494d0 0x5644fb7bac20 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e209da8de1b34f65 server_cookie=b13b152cf6ece8fe in_seq=0 out_seq=0 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e3effd640 1 -- 172.21.15.44:0/1079544469 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+420 (crc 0 0 0) 0x5644fb6064d0 con 0x5644fb7494d0 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e3effd640 10 client.8232 handle_command_reply: tid=0 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e6940e740 1 client.8232 shutdown 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e6940e740 1 -- 172.21.15.44:0/1079544469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 msgr2=0x5644fb6fae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 0x5644fb6fae60 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f9e50061350 tx=0x7f9e50000f70 comp rx=0 tx=0).stop 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e6940e740 1 -- 172.21.15.44:0/1079544469 shutdown_connections 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5644fb7494d0 0x5644fb7bac20 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644fb721280 0x5644fb76a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644fb701be0 0x5644fb6fae60 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e3effd640 0 client.8232 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e6940e740 1 --2- 172.21.15.44:0/1079544469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5644fb6fbdc0 0x5644fb748500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e6940e740 1 -- 172.21.15.44:0/1079544469 >> 172.21.15.44:0/1079544469 conn(0x5644fb728150 msgr2=0x5644fb728bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e6940e740 1 -- 172.21.15.44:0/1079544469 shutdown_connections 2024-04-02T13:27:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e6940e740 1 -- 172.21.15.44:0/1079544469 wait complete. 2024-04-02T13:27:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e6940e740 20 client.8232 trim_cache size 0 max 16384 2024-04-02T13:27:33.676 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:27:33.676 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-04-02T13:27:33.676 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:27:33.676 INFO:teuthology.orchestra.run.smithi044.stdout: "damage_type": "backtrace", 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 3713808342, 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: "ino": 1099511627776, 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/dir" 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: "damage_type": "dir_frag", 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 3741410865, 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: "ino": 1099511627776, 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: "frag": "*", 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/dir" 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: "damage_type": "backtrace", 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 4251987405, 2024-04-02T13:27:33.677 INFO:teuthology.orchestra.run.smithi044.stdout: "ino": 1099511627777, 2024-04-02T13:27:33.678 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/testdir/hardlink" 2024-04-02T13:27:33.678 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:27:33.678 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-04-02T13:27:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e66750640 1 -- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9e4006c8f0 msgr2=0x7f9e4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e66750640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9e4006c8f0 0x7f9e4006edb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9e600cc080 tx=0x7f9e48009190 comp rx=0 tx=0).stop 2024-04-02T13:27:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e66750640 1 -- 172.21.15.44:0/2611088139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e600d0c50 msgr2=0x7f9e600ce8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e66750640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e600d0c50 0x7f9e600ce8d0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000c440 tx=0x7f9e50001e70 comp rx=0 tx=0).stop 2024-04-02T13:27:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e66750640 1 -- 172.21.15.44:0/2611088139 shutdown_connections 2024-04-02T13:27:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e66750640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9e4006c8f0 0x7f9e4006edb0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e66750640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e6015e4a0 0x7f9e600cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e66750640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e600d0c50 0x7f9e600ce8d0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e66750640 1 --2- 172.21.15.44:0/2611088139 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6006d550 0x7f9e60154890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.666+0000 7f9e66750640 1 -- 172.21.15.44:0/2611088139 >> 172.21.15.44:0/2611088139 conn(0x7f9e6009aa20 msgr2=0x7f9e600b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.667+0000 7f9e66750640 1 -- 172.21.15.44:0/2611088139 shutdown_connections 2024-04-02T13:27:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.667+0000 7f9e66750640 1 -- 172.21.15.44:0/2611088139 wait complete. 2024-04-02T13:27:33.681 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-02T13:27:33.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.777+0000 7fe98bf4d640 1 Processor -- start 2024-04-02T13:27:33.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.778+0000 7fe98bf4d640 1 -- start start 2024-04-02T13:27:33.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.778+0000 7fe98bf4d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe9840d1e00 0x7fe9840d2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.778+0000 7fe98bf4d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe9840ce8b0 0x7fe9840ced40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.778+0000 7fe98bf4d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9840b1dc0 con 0x7fe9840d15f0 2024-04-02T13:27:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.778+0000 7fe98bf4d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe9840b26d0 con 0x7fe9840ce8b0 2024-04-02T13:27:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.778+0000 7fe98bf4d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe9840cf600 con 0x7fe9840d1e00 2024-04-02T13:27:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.779+0000 7fe98a4ea640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe9840ce8b0 0x7fe9840ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.779+0000 7fe98a4ea640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe9840ce8b0 0x7fe9840ced40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52422/0 (socket says 172.21.15.44:52422) 2024-04-02T13:27:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.779+0000 7fe98a4ea640 1 -- 172.21.15.44:0/80249714 learned_addr learned my addr 172.21.15.44:0/80249714 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.779+0000 7fe9894e8640 1 --2- 172.21.15.44:0/80249714 >> v2:172.21.15.69:3300/0 conn(0x7fe9840d1e00 0x7fe9840d2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.779+0000 7fe98a4ea640 1 -- 172.21.15.44:0/80249714 >> v2:172.21.15.69:3300/0 conn(0x7fe9840d1e00 msgr2=0x7fe9840d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.779+0000 7fe98a4ea640 1 --2- 172.21.15.44:0/80249714 >> v2:172.21.15.69:3300/0 conn(0x7fe9840d1e00 0x7fe9840d2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.779+0000 7fe98a4ea640 1 -- 172.21.15.44:0/80249714 >> v1:172.21.15.69:6789/0 conn(0x7fe9840d15f0 legacy=0x7fe9840b2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.779+0000 7fe98a4ea640 1 -- 172.21.15.44:0/80249714 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9840b2080 con 0x7fe9840ce8b0 2024-04-02T13:27:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.780+0000 7fe98a4ea640 1 --2- 172.21.15.44:0/80249714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9840ce8b0 0x7fe9840ced40 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe9840737a0 tx=0x7fe97c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9fb88bb79d67a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.780+0000 7fe988ce7640 1 -- 172.21.15.44:0/80249714 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c034070 con 0x7fe9840ce8b0 2024-04-02T13:27:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.780+0000 7fe988ce7640 1 -- 172.21.15.44:0/80249714 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe97c030860 con 0x7fe9840ce8b0 2024-04-02T13:27:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.780+0000 7fe988ce7640 1 -- 172.21.15.44:0/80249714 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c030c30 con 0x7fe9840ce8b0 2024-04-02T13:27:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.780+0000 7fe98bf4d640 1 -- 172.21.15.44:0/80249714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9840ce8b0 msgr2=0x7fe9840ced40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.780+0000 7fe98bf4d640 1 --2- 172.21.15.44:0/80249714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9840ce8b0 0x7fe9840ced40 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe9840737a0 tx=0x7fe97c02b9a0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.780+0000 7fe98bf4d640 1 -- 172.21.15.44:0/80249714 shutdown_connections 2024-04-02T13:27:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.780+0000 7fe98bf4d640 1 --2- 172.21.15.44:0/80249714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9840ce8b0 0x7fe9840ced40 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.780+0000 7fe98bf4d640 1 --2- 172.21.15.44:0/80249714 >> v2:172.21.15.69:3300/0 conn(0x7fe9840d1e00 0x7fe9840d2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.780+0000 7fe98bf4d640 1 -- 172.21.15.44:0/80249714 >> 172.21.15.44:0/80249714 conn(0x7fe98409aa20 msgr2=0x7fe98409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.780+0000 7fe98bf4d640 1 -- 172.21.15.44:0/80249714 shutdown_connections 2024-04-02T13:27:33.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.780+0000 7fe98bf4d640 1 -- 172.21.15.44:0/80249714 wait complete. 2024-04-02T13:27:33.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.782+0000 7fe98bf4d640 1 Processor -- start 2024-04-02T13:27:33.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.782+0000 7fe98bf4d640 1 -- start start 2024-04-02T13:27:33.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.782+0000 7fe98bf4d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9840ce8b0 0x7fe984156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.782+0000 7fe98bf4d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840d15f0 0x7fe9841f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.782+0000 7fe98bf4d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9840d1e00 0x7fe9840cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.782+0000 7fe98bf4d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe9840cf280 con 0x7fe9840ce8b0 2024-04-02T13:27:33.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.782+0000 7fe98bf4d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe9840d19d0 con 0x7fe9840d1e00 2024-04-02T13:27:33.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.782+0000 7fe98bf4d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe9840d0140 con 0x7fe9840d15f0 2024-04-02T13:27:33.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.782+0000 7fe989ce9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9840ce8b0 0x7fe984156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.782+0000 7fe989ce9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9840ce8b0 0x7fe984156350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52424/0 (socket says 172.21.15.44:52424) 2024-04-02T13:27:33.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.782+0000 7fe989ce9640 1 -- 172.21.15.44:0/3378100730 learned_addr learned my addr 172.21.15.44:0/3378100730 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:33.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe9894e8640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840d15f0 0x7fe9841f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe98a4ea640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9840d1e00 0x7fe9840cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe989ce9640 1 -- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840d15f0 msgr2=0x7fe9841f5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe989ce9640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840d15f0 0x7fe9841f5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe989ce9640 1 -- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9840d1e00 msgr2=0x7fe9840cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe989ce9640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9840d1e00 0x7fe9840cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe989ce9640 1 -- 172.21.15.44:0/3378100730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9840c7a90 con 0x7fe9840ce8b0 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe98a4ea640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9840d1e00 0x7fe9840cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe989ce9640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9840ce8b0 0x7fe984156350 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fe98000c7c0 tx=0x7fe98000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe97a7fc640 1 -- 172.21.15.44:0/3378100730 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe980016020 con 0x7fe9840ce8b0 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe97a7fc640 1 -- 172.21.15.44:0/3378100730 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe980006d50 con 0x7fe9840ce8b0 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe98bf4d640 1 -- 172.21.15.44:0/3378100730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9840c7d70 con 0x7fe9840ce8b0 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe97a7fc640 1 -- 172.21.15.44:0/3378100730 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe980003bb0 con 0x7fe9840ce8b0 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.783+0000 7fe98bf4d640 1 -- 172.21.15.44:0/3378100730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9840c8330 con 0x7fe9840ce8b0 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.784+0000 7fe98bf4d640 1 -- 172.21.15.44:0/3378100730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9840684f0 con 0x7fe9840ce8b0 2024-04-02T13:27:33.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.785+0000 7fe97a7fc640 1 -- 172.21.15.44:0/3378100730 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe98000a850 con 0x7fe9840ce8b0 2024-04-02T13:27:33.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.785+0000 7fe97a7fc640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe96806c8f0 0x7fe96806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:33.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.785+0000 7fe97a7fc640 1 -- 172.21.15.44:0/3378100730 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7fe98008ba30 con 0x7fe9840ce8b0 2024-04-02T13:27:33.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.785+0000 7fe9894e8640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe96806c8f0 0x7fe96806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:33.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.786+0000 7fe9894e8640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe96806c8f0 0x7fe96806edb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe974006fd0 tx=0x7fe974008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:33.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:33.789+0000 7fe97a7fc640 1 -- 172.21.15.44:0/3378100730 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe980020020 con 0x7fe9840ce8b0 2024-04-02T13:27:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.013+0000 7fe98bf4d640 1 -- 172.21.15.44:0/3378100730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe9840b18d0 con 0x7fe9840ce8b0 2024-04-02T13:27:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.014+0000 7fe97a7fc640 1 -- 172.21.15.44:0/3378100730 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+164 (secure 0 0 0) 0x7fe980019310 con 0x7fe9840ce8b0 2024-04-02T13:27:34.017 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:27:34.017 INFO:teuthology.orchestra.run.smithi044.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-04-02T13:27:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.016+0000 7fe98bf4d640 1 -- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe96806c8f0 msgr2=0x7fe96806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.016+0000 7fe98bf4d640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe96806c8f0 0x7fe96806edb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe974006fd0 tx=0x7fe974008040 comp rx=0 tx=0).stop 2024-04-02T13:27:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.016+0000 7fe98bf4d640 1 -- 172.21.15.44:0/3378100730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9840ce8b0 msgr2=0x7fe984156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.016+0000 7fe98bf4d640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9840ce8b0 0x7fe984156350 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fe98000c7c0 tx=0x7fe98000cc80 comp rx=0 tx=0).stop 2024-04-02T13:27:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.016+0000 7fe98bf4d640 1 -- 172.21.15.44:0/3378100730 shutdown_connections 2024-04-02T13:27:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.016+0000 7fe98bf4d640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe96806c8f0 0x7fe96806edb0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.016+0000 7fe98bf4d640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9840d1e00 0x7fe9840cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.016+0000 7fe98bf4d640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840d15f0 0x7fe9841f5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.016+0000 7fe98bf4d640 1 --2- 172.21.15.44:0/3378100730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9840ce8b0 0x7fe984156350 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.016+0000 7fe98bf4d640 1 -- 172.21.15.44:0/3378100730 >> 172.21.15.44:0/3378100730 conn(0x7fe98409aa20 msgr2=0x7fe984160010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:34.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.016+0000 7fe98bf4d640 1 -- 172.21.15.44:0/3378100730 shutdown_connections 2024-04-02T13:27:34.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.016+0000 7fe98bf4d640 1 -- 172.21.15.44:0/3378100730 wait complete. 2024-04-02T13:27:34.030 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-02T13:27:34.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:34.030 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive,repair 2024-04-02T13:27:34.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.132+0000 7f517468f640 1 Processor -- start 2024-04-02T13:27:34.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.132+0000 7f517468f640 1 -- start start 2024-04-02T13:27:34.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f517468f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f516c0d0840 0x7f516c165c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f517468f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f516c0cdc20 con 0x7f516c1554c0 2024-04-02T13:27:34.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f517468f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f516c0cdec0 con 0x7f516c0ce9a0 2024-04-02T13:27:34.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f517468f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f516c0ce1a0 con 0x7f516c0d0840 2024-04-02T13:27:34.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f517242b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f516c1554c0 0x7f516c071120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52618/0 (socket says 172.21.15.44:52618) 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f517242b640 1 -- 172.21.15.44:0/643431569 learned_addr learned my addr 172.21.15.44:0/643431569 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1036755248 0 0) 0x7f516c0cdc20 con 0x7f516c1554c0 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5158003520 con 0x7f516c1554c0 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f5172c2c640 1 --2- 172.21.15.44:0/643431569 >> v2:172.21.15.69:3300/0 conn(0x7f516c0d0840 0x7f516c165c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2423113584 0 0) 0x7f5158003520 con 0x7f516c1554c0 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 >> v2:172.21.15.69:3300/0 conn(0x7f516c0d0840 msgr2=0x7f516c165c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f5171429640 1 --2- 172.21.15.44:0/643431569 >> v2:172.21.15.69:3300/0 conn(0x7f516c0d0840 0x7f516c165c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 >> v1:172.21.15.69:6789/0 conn(0x7f516c0ce9a0 legacy=0x7f516c0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516c0ce480 con 0x7f516c1554c0 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f5168001690 con 0x7f516c1554c0 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 >> v1:172.21.15.44:6789/0 conn(0x7f516c1554c0 legacy=0x7f516c071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f5171429640 1 --2- 172.21.15.44:0/643431569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51580039b0 0x7f5158003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f5171429640 1 --2- 172.21.15.44:0/643431569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5158004800 0x7f5158004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.133+0000 7f5171429640 1 --2- 172.21.15.44:0/643431569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51580051c0 0x7f515800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.134+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f516c0ce1a0 con 0x7f51580039b0 2024-04-02T13:27:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.134+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f515800c060 con 0x7f5158004800 2024-04-02T13:27:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.134+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f515800c260 con 0x7f51580051c0 2024-04-02T13:27:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.134+0000 7f517242b640 1 --2- 172.21.15.44:0/643431569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51580039b0 0x7f5158003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.134+0000 7f5171c2a640 1 --2- 172.21.15.44:0/643431569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5158004800 0x7f5158004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.134+0000 7f5172c2c640 1 --2- 172.21.15.44:0/643431569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51580051c0 0x7f515800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.134+0000 7f517242b640 1 -- 172.21.15.44:0/643431569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51580051c0 msgr2=0x7f515800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.134+0000 7f517242b640 1 --2- 172.21.15.44:0/643431569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51580051c0 0x7f515800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.134+0000 7f517242b640 1 -- 172.21.15.44:0/643431569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5158004800 msgr2=0x7f5158004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.134+0000 7f517242b640 1 --2- 172.21.15.44:0/643431569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5158004800 0x7f5158004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.134+0000 7f5172c2c640 1 -- 172.21.15.44:0/643431569 reap_dead start 2024-04-02T13:27:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.134+0000 7f517242b640 1 -- 172.21.15.44:0/643431569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5158003520 con 0x7f51580039b0 2024-04-02T13:27:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.135+0000 7f517242b640 1 --2- 172.21.15.44:0/643431569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51580039b0 0x7f5158003dd0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f516800c210 tx=0x7f5158005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa97d3f00159cf8d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.135+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168007d30 con 0x7f51580039b0 2024-04-02T13:27:34.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.135+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5168030070 con 0x7f51580039b0 2024-04-02T13:27:34.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.135+0000 7f5171429640 1 -- 172.21.15.44:0/643431569 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168005d10 con 0x7f51580039b0 2024-04-02T13:27:34.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.135+0000 7f517468f640 1 -- 172.21.15.44:0/643431569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51580039b0 msgr2=0x7f5158003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.135+0000 7f517468f640 1 --2- 172.21.15.44:0/643431569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51580039b0 0x7f5158003dd0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f516800c210 tx=0x7f5158005830 comp rx=0 tx=0).stop 2024-04-02T13:27:34.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.135+0000 7f517468f640 1 -- 172.21.15.44:0/643431569 shutdown_connections 2024-04-02T13:27:34.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.135+0000 7f517468f640 1 --2- 172.21.15.44:0/643431569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51580039b0 0x7f5158003dd0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.135+0000 7f517468f640 1 -- 172.21.15.44:0/643431569 >> 172.21.15.44:0/643431569 conn(0x7f516c06b920 msgr2=0x7f516c06bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:34.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.135+0000 7f517468f640 1 -- 172.21.15.44:0/643431569 shutdown_connections 2024-04-02T13:27:34.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.135+0000 7f517468f640 1 -- 172.21.15.44:0/643431569 wait complete. 2024-04-02T13:27:34.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.137+0000 7f517468f640 1 Processor -- start 2024-04-02T13:27:34.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.137+0000 7f517468f640 1 -- start start 2024-04-02T13:27:34.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.137+0000 7f517468f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51580039b0 0x7f516c0ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.137+0000 7f517468f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f516c156da0 0x7f516c1f7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.137+0000 7f517468f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c154120 0x7f516c1545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517468f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f516c15c290 con 0x7f51580039b0 2024-04-02T13:27:34.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517468f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f516c071be0 con 0x7f516c156da0 2024-04-02T13:27:34.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517468f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f516c0cb000 con 0x7f516c154120 2024-04-02T13:27:34.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517242b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51580039b0 0x7f516c0ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517242b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51580039b0 0x7f516c0ae9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52440/0 (socket says 172.21.15.44:52440) 2024-04-02T13:27:34.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517242b640 1 -- 172.21.15.44:0/1059761948 learned_addr learned my addr 172.21.15.44:0/1059761948 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:34.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f5172c2c640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c154120 0x7f516c1545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f5171c2a640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f516c156da0 0x7f516c1f7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517242b640 1 -- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c154120 msgr2=0x7f516c1545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517242b640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c154120 0x7f516c1545a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517242b640 1 -- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f516c156da0 msgr2=0x7f516c1f7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517242b640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f516c156da0 0x7f516c1f7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517242b640 1 -- 172.21.15.44:0/1059761948 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516c157560 con 0x7f51580039b0 2024-04-02T13:27:34.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517242b640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51580039b0 0x7f516c0ae9c0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f516802c4c0 tx=0x7f516c0ce650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f5162ffd640 1 -- 172.21.15.44:0/1059761948 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168001c70 con 0x7f51580039b0 2024-04-02T13:27:34.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f5162ffd640 1 -- 172.21.15.44:0/1059761948 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51680049f0 con 0x7f51580039b0 2024-04-02T13:27:34.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f5162ffd640 1 -- 172.21.15.44:0/1059761948 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516802de60 con 0x7f51580039b0 2024-04-02T13:27:34.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517468f640 1 -- 172.21.15.44:0/1059761948 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f516c157840 con 0x7f51580039b0 2024-04-02T13:27:34.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.138+0000 7f517468f640 1 -- 172.21.15.44:0/1059761948 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f516c157e00 con 0x7f51580039b0 2024-04-02T13:27:34.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.140+0000 7f5162ffd640 1 -- 172.21.15.44:0/1059761948 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5168007070 con 0x7f51580039b0 2024-04-02T13:27:34.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.141+0000 7f5162ffd640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f515006c920 0x7f515006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.141+0000 7f5162ffd640 1 -- 172.21.15.44:0/1059761948 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f51680ad0e0 con 0x7f51580039b0 2024-04-02T13:27:34.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.141+0000 7f5171c2a640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f515006c920 0x7f515006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.141+0000 7f5171c2a640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f515006c920 0x7f515006ede0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f515c000c00 tx=0x7f515c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.141+0000 7f5177276740 1 Processor -- start 2024-04-02T13:27:34.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.142+0000 7f5177276740 1 -- start start 2024-04-02T13:27:34.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.142+0000 7f5177276740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d98310 0x55a804d986f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.142+0000 7f5177276740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d68330 0x55a804d95580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.142+0000 7f5177276740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 0x55a804d95ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.142+0000 7f5177276740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a804cd2770 con 0x55a804d6be90 2024-04-02T13:27:34.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.142+0000 7f5177276740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a804c7a960 con 0x55a804d98310 2024-04-02T13:27:34.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.142+0000 7f5177276740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a804cce8c0 con 0x55a804d68330 2024-04-02T13:27:34.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.142+0000 7f5172c2c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 0x55a804d95ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.142+0000 7f5172c2c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 0x55a804d95ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52442/0 (socket says 172.21.15.44:52442) 2024-04-02T13:27:34.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.143+0000 7f5172c2c640 1 -- 172.21.15.44:0/3864139115 learned_addr learned my addr 172.21.15.44:0/3864139115 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:34.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.143+0000 7f5171c2a640 1 --2- 172.21.15.44:0/3864139115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d68330 0x55a804d95580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.143+0000 7f517242b640 1 --2- 172.21.15.44:0/3864139115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d98310 0x55a804d986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.143+0000 7f5172c2c640 1 -- 172.21.15.44:0/3864139115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d68330 msgr2=0x55a804d95580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.143+0000 7f5172c2c640 1 --2- 172.21.15.44:0/3864139115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d68330 0x55a804d95580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.143+0000 7f5172c2c640 1 -- 172.21.15.44:0/3864139115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d98310 msgr2=0x55a804d986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.143+0000 7f5172c2c640 1 --2- 172.21.15.44:0/3864139115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d98310 0x55a804d986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.143+0000 7f5172c2c640 1 -- 172.21.15.44:0/3864139115 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a804aae7e0 con 0x55a804d6be90 2024-04-02T13:27:34.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.143+0000 7f5172c2c640 1 --2- 172.21.15.44:0/3864139115 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 0x55a804d95ac0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f516c06cf30 tx=0x7f5164009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4686988cc2f94d0d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.143+0000 7f514ffff640 1 -- 172.21.15.44:0/3864139115 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164020070 con 0x55a804d6be90 2024-04-02T13:27:34.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.143+0000 7f514ffff640 1 -- 172.21.15.44:0/3864139115 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5164003b90 con 0x55a804d6be90 2024-04-02T13:27:34.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.143+0000 7f514ffff640 1 -- 172.21.15.44:0/3864139115 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164004030 con 0x55a804d6be90 2024-04-02T13:27:34.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.144+0000 7f5177276740 1 -- 172.21.15.44:0/3864139115 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 msgr2=0x55a804d95ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.144+0000 7f5177276740 1 --2- 172.21.15.44:0/3864139115 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 0x55a804d95ac0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f516c06cf30 tx=0x7f5164009750 comp rx=0 tx=0).stop 2024-04-02T13:27:34.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.144+0000 7f5177276740 1 -- 172.21.15.44:0/3864139115 shutdown_connections 2024-04-02T13:27:34.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.144+0000 7f5177276740 1 --2- 172.21.15.44:0/3864139115 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 0x55a804d95ac0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.144+0000 7f5177276740 1 --2- 172.21.15.44:0/3864139115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d68330 0x55a804d95580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.144+0000 7f5177276740 1 --2- 172.21.15.44:0/3864139115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d98310 0x55a804d986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.144+0000 7f5177276740 1 -- 172.21.15.44:0/3864139115 >> 172.21.15.44:0/3864139115 conn(0x55a804d6a5a0 msgr2=0x55a804db3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:34.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.144+0000 7f5177276740 1 -- 172.21.15.44:0/3864139115 shutdown_connections 2024-04-02T13:27:34.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.144+0000 7f5177276740 1 -- 172.21.15.44:0/3864139115 wait complete. 2024-04-02T13:27:34.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.144+0000 7f5177276740 1 -- start start 2024-04-02T13:27:34.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.145+0000 7f5177276740 1 Processor -- start 2024-04-02T13:27:34.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.145+0000 7f5177276740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d6be90 0x55a804d8f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.145+0000 7f5177276740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 0x55a804d928f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.145+0000 7f5177276740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d8f8d0 0x55a804d8fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.145+0000 7f5177276740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a804d6d040 con 0x55a804d98310 2024-04-02T13:27:34.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.145+0000 7f5177276740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a804c8dd20 con 0x55a804d6be90 2024-04-02T13:27:34.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.145+0000 7f5177276740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a804d6c430 con 0x55a804d8f8d0 2024-04-02T13:27:34.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5177276740 10 client.0 _ll_register_callbacks cb 0x55a804c5aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:34.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5171c2a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 0x55a804d928f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5171c2a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 0x55a804d928f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52450/0 (socket says 172.21.15.44:52450) 2024-04-02T13:27:34.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5171c2a640 1 -- 172.21.15.44:0/1421474672 learned_addr learned my addr 172.21.15.44:0/1421474672 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:34.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f517242b640 1 --2- 172.21.15.44:0/1421474672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d6be90 0x55a804d8f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5172c2c640 1 --2- 172.21.15.44:0/1421474672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d8f8d0 0x55a804d8fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5171c2a640 1 -- 172.21.15.44:0/1421474672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d8f8d0 msgr2=0x55a804d8fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5171c2a640 1 --2- 172.21.15.44:0/1421474672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d8f8d0 0x55a804d8fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5171c2a640 1 -- 172.21.15.44:0/1421474672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d6be90 msgr2=0x55a804d8f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5171c2a640 1 --2- 172.21.15.44:0/1421474672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d6be90 0x55a804d8f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5171c2a640 1 -- 172.21.15.44:0/1421474672 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a804b0ef70 con 0x55a804d98310 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5171c2a640 1 --2- 172.21.15.44:0/1421474672 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 0x55a804d928f0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f515c0073b0 tx=0x7f515c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=372a12f679de5aa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f514d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f514d7fa640 1 -- 172.21.15.44:0/1421474672 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515c006ba0 con 0x55a804d98310 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f514d7fa640 1 -- 172.21.15.44:0/1421474672 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f515c010040 con 0x55a804d98310 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5177276740 1 -- 172.21.15.44:0/1421474672 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5164003b90 con 0x55a804d98310 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f514d7fa640 1 -- 172.21.15.44:0/1421474672 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f515c006d40 con 0x55a804d98310 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5177276740 10 client.8250 fetch_fsmap learned FSMap version 68 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.146+0000 7f5177276740 1 -- 172.21.15.44:0/1421474672 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55a8049a6170 con 0x55a804d98310 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.147+0000 7f514d7fa640 1 -- 172.21.15.44:0/1421474672 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5164003b90 con 0x55a804d98310 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.147+0000 7f514d7fa640 1 -- 172.21.15.44:0/1421474672 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f515c015440 con 0x55a804d98310 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.147+0000 7f5177276740 10 client.8250 fetch_fsmap finished waiting for FSMap version 68 2024-04-02T13:27:34.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.147+0000 7f5177276740 10 client.8250 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.147+0000 7f5177276740 20 client.8250 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.147+0000 7f5177276740 1 --2- 172.21.15.44:0/1421474672 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a804d9bc40 0x55a804dc6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.147+0000 7f5177276740 4 client.8250 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.147+0000 7f5177276740 1 -- 172.21.15.44:0/1421474672 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a804cdfcb0 con 0x55a804d9bc40 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.147+0000 7f517242b640 1 --2- 172.21.15.44:0/1421474672 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a804d9bc40 0x55a804dc6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.147+0000 7f514d7fa640 10 client.8250 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.147+0000 7f517242b640 1 --2- 172.21.15.44:0/1421474672 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a804d9bc40 0x55a804dc6c50 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4e26cc7071ffcd8 server_cookie=b94384359a2e6330 in_seq=0 out_seq=0 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f514d7fa640 1 -- 172.21.15.44:0/1421474672 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a804cdfcb0 con 0x55a804d9bc40 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f514d7fa640 10 client.8250 handle_command_reply: tid=0 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f5177276740 1 client.8250 shutdown 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f5177276740 1 -- 172.21.15.44:0/1421474672 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 msgr2=0x55a804d928f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f5177276740 1 --2- 172.21.15.44:0/1421474672 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 0x55a804d928f0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f515c0073b0 tx=0x7f515c001120 comp rx=0 tx=0).stop 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f5177276740 1 -- 172.21.15.44:0/1421474672 shutdown_connections 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f5177276740 1 --2- 172.21.15.44:0/1421474672 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a804d9bc40 0x55a804dc6c50 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f5177276740 1 --2- 172.21.15.44:0/1421474672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d8f8d0 0x55a804d8fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f5177276740 1 --2- 172.21.15.44:0/1421474672 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 0x55a804d928f0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f514d7fa640 0 client.8250 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:34.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f5177276740 1 --2- 172.21.15.44:0/1421474672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d6be90 0x55a804d8f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f5177276740 1 -- 172.21.15.44:0/1421474672 >> 172.21.15.44:0/1421474672 conn(0x55a804d68330 msgr2=0x55a804db6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:34.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f5177276740 1 -- 172.21.15.44:0/1421474672 shutdown_connections 2024-04-02T13:27:34.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f5177276740 1 -- 172.21.15.44:0/1421474672 wait complete. 2024-04-02T13:27:34.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.148+0000 7f5177276740 20 client.8250 trim_cache size 0 max 16384 2024-04-02T13:27:34.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.166+0000 7f5177276740 1 Processor -- start 2024-04-02T13:27:34.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.166+0000 7f5177276740 1 -- start start 2024-04-02T13:27:34.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.166+0000 7f5177276740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d6be90 0x55a804d8f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.166+0000 7f5177276740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d8f8d0 0x55a804db3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.166+0000 7f5177276740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 0x55a804dc5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.166+0000 7f5177276740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f516c201900 con 0x55a804d98310 2024-04-02T13:27:34.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.166+0000 7f5177276740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f516c200900 con 0x55a804d8f8d0 2024-04-02T13:27:34.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.166+0000 7f5177276740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f516c1ffc90 con 0x55a804d6be90 2024-04-02T13:27:34.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f5172c2c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 0x55a804dc5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f5172c2c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 0x55a804dc5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52464/0 (socket says 172.21.15.44:52464) 2024-04-02T13:27:34.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f5172c2c640 1 -- 172.21.15.44:0/2565296666 learned_addr learned my addr 172.21.15.44:0/2565296666 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:34.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f517242b640 1 --2- 172.21.15.44:0/2565296666 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d6be90 0x55a804d8f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f5172c2c640 1 -- 172.21.15.44:0/2565296666 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d6be90 msgr2=0x55a804d8f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f5171c2a640 1 --2- 172.21.15.44:0/2565296666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d8f8d0 0x55a804db3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f5172c2c640 1 --2- 172.21.15.44:0/2565296666 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d6be90 0x55a804d8f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f5172c2c640 1 -- 172.21.15.44:0/2565296666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d8f8d0 msgr2=0x55a804db3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f5172c2c640 1 --2- 172.21.15.44:0/2565296666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d8f8d0 0x55a804db3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f5172c2c640 1 -- 172.21.15.44:0/2565296666 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a804d954f0 con 0x55a804d98310 2024-04-02T13:27:34.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f5172c2c640 1 --2- 172.21.15.44:0/2565296666 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 0x55a804dc5670 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f5164004000 tx=0x7f5164019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f08c402d4d78989 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f514f7fe640 1 -- 172.21.15.44:0/2565296666 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164020050 con 0x55a804d98310 2024-04-02T13:27:34.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f514f7fe640 1 -- 172.21.15.44:0/2565296666 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5164021040 con 0x55a804d98310 2024-04-02T13:27:34.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.167+0000 7f514f7fe640 1 -- 172.21.15.44:0/2565296666 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164003bd0 con 0x55a804d98310 2024-04-02T13:27:34.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.168+0000 7f5177276740 1 -- 172.21.15.44:0/2565296666 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 msgr2=0x55a804dc5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.168+0000 7f5177276740 1 --2- 172.21.15.44:0/2565296666 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 0x55a804dc5670 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f5164004000 tx=0x7f5164019000 comp rx=0 tx=0).stop 2024-04-02T13:27:34.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.168+0000 7f5177276740 1 -- 172.21.15.44:0/2565296666 shutdown_connections 2024-04-02T13:27:34.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.168+0000 7f5177276740 1 --2- 172.21.15.44:0/2565296666 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d98310 0x55a804dc5670 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.168+0000 7f5177276740 1 --2- 172.21.15.44:0/2565296666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d8f8d0 0x55a804db3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.168+0000 7f5177276740 1 --2- 172.21.15.44:0/2565296666 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d6be90 0x55a804d8f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.168+0000 7f5177276740 1 -- 172.21.15.44:0/2565296666 >> 172.21.15.44:0/2565296666 conn(0x55a804d68330 msgr2=0x55a804d92700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:34.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.168+0000 7f5177276740 1 -- 172.21.15.44:0/2565296666 shutdown_connections 2024-04-02T13:27:34.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.168+0000 7f5177276740 1 -- 172.21.15.44:0/2565296666 wait complete. 2024-04-02T13:27:34.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.168+0000 7f5177276740 1 -- start start 2024-04-02T13:27:34.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.168+0000 7f5177276740 1 Processor -- start 2024-04-02T13:27:34.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.169+0000 7f5177276740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 0x55a804e2b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.169+0000 7f5177276740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d8f8d0 0x55a804e2e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.169+0000 7f5177276740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d98310 0x55a804e2e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.169+0000 7f5177276740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a804d958c0 con 0x55a804d6be90 2024-04-02T13:27:34.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.169+0000 7f5177276740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f516c2002b0 con 0x55a804d98310 2024-04-02T13:27:34.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.169+0000 7f5177276740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a804d642b0 con 0x55a804d8f8d0 2024-04-02T13:27:34.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.169+0000 7f5177276740 10 client.0 _ll_register_callbacks cb 0x7f515c007030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:34.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.169+0000 7f517242b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 0x55a804e2b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.169+0000 7f517242b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 0x55a804e2b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52476/0 (socket says 172.21.15.44:52476) 2024-04-02T13:27:34.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.169+0000 7f517242b640 1 -- 172.21.15.44:0/607139415 learned_addr learned my addr 172.21.15.44:0/607139415 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:34.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f5172c2c640 1 --2- 172.21.15.44:0/607139415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d98310 0x55a804e2e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f517242b640 1 -- 172.21.15.44:0/607139415 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d8f8d0 msgr2=0x55a804e2e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f517242b640 1 --2- 172.21.15.44:0/607139415 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d8f8d0 0x55a804e2e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f517242b640 1 -- 172.21.15.44:0/607139415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d98310 msgr2=0x55a804e2e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f517242b640 1 --2- 172.21.15.44:0/607139415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d98310 0x55a804e2e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f517242b640 1 -- 172.21.15.44:0/607139415 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a804e2ec10 con 0x55a804d6be90 2024-04-02T13:27:34.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f517242b640 1 --2- 172.21.15.44:0/607139415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 0x55a804e2b350 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f516807ace0 tx=0x7f516807f1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82013f74b7d74612 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f514d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:27:34.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f514d7fa640 1 -- 172.21.15.44:0/607139415 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51680aa060 con 0x55a804d6be90 2024-04-02T13:27:34.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f514d7fa640 1 -- 172.21.15.44:0/607139415 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f51680931e0 con 0x55a804d6be90 2024-04-02T13:27:34.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f5177276740 1 -- 172.21.15.44:0/607139415 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5164021040 con 0x55a804d6be90 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f514d7fa640 1 -- 172.21.15.44:0/607139415 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f5168079050 con 0x55a804d6be90 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f5177276740 10 client.8256 fetch_fsmap learned FSMap version 68 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f5177276740 1 -- 172.21.15.44:0/607139415 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55a804dc25d0 con 0x55a804d6be90 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f514d7fa640 1 -- 172.21.15.44:0/607139415 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5164021040 con 0x55a804d6be90 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f514d7fa640 1 -- 172.21.15.44:0/607139415 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f5168049070 con 0x55a804d6be90 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f5177276740 10 client.8256 fetch_fsmap finished waiting for FSMap version 68 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f5177276740 10 client.8256 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f5177276740 20 client.8256 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f5177276740 1 --2- 172.21.15.44:0/607139415 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a804d90680 0x55a804dc17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f5177276740 4 client.8256 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}] 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.170+0000 7f5177276740 1 -- 172.21.15.44:0/607139415 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}) -- 0x55a804cda000 con 0x55a804d90680 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.171+0000 7f5171c2a640 1 --2- 172.21.15.44:0/607139415 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a804d90680 0x55a804dc17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.171+0000 7f514d7fa640 10 client.8256 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:34.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.171+0000 7f5171c2a640 1 --2- 172.21.15.44:0/607139415 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a804d90680 0x55a804dc17f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=500394a7f61c493c server_cookie=c828b2972f2a4273 in_seq=0 out_seq=0 2024-04-02T13:27:34.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.171+0000 7f514d7fa640 1 -- 172.21.15.44:0/607139415 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a804cda000 con 0x55a804d90680 2024-04-02T13:27:34.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.171+0000 7f514d7fa640 10 client.8256 handle_command_reply: tid=0 2024-04-02T13:27:34.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.171+0000 7f5177276740 1 client.8256 shutdown 2024-04-02T13:27:34.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.171+0000 7f5177276740 1 -- 172.21.15.44:0/607139415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 msgr2=0x55a804e2b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.171+0000 7f5177276740 1 --2- 172.21.15.44:0/607139415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 0x55a804e2b350 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f516807ace0 tx=0x7f516807f1d0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f5177276740 1 -- 172.21.15.44:0/607139415 shutdown_connections 2024-04-02T13:27:34.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f5177276740 1 --2- 172.21.15.44:0/607139415 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x55a804d90680 0x55a804dc17f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f5177276740 1 --2- 172.21.15.44:0/607139415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a804d98310 0x55a804e2e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f5177276740 1 --2- 172.21.15.44:0/607139415 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a804d8f8d0 0x55a804e2e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f514d7fa640 0 client.8256 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:34.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f5177276740 1 --2- 172.21.15.44:0/607139415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a804d6be90 0x55a804e2b350 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f5177276740 1 -- 172.21.15.44:0/607139415 >> 172.21.15.44:0/607139415 conn(0x55a804d68330 msgr2=0x55a804d8ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:34.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f5177276740 1 -- 172.21.15.44:0/607139415 shutdown_connections 2024-04-02T13:27:34.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f5177276740 1 -- 172.21.15.44:0/607139415 wait complete. 2024-04-02T13:27:34.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f5177276740 20 client.8256 trim_cache size 0 max 16384 2024-04-02T13:27:34.183 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:27:34.183 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-02T13:27:34.183 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "d2d686a1-3479-4fc7-9e78-87e5581c59ed", 2024-04-02T13:27:34.183 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-02T13:27:34.183 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:27:34.184 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:27:34.172+0000 7fe0b0a2e640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v17 pv19 ap=3 DIRTYPARENT f(v0 m2024-04-02T13:26:51.014993+0000 1=1+0)->f(v1 m2024-04-02T13:26:51.014993+0000) n(v0 rc2024-04-02T13:26:51.131857+0000 2=1+1)->n(v0 rc2024-04-02T13:26:51.131857+0000 2=1+1) (inest lock dirty) (ifile lock->sync w=1 flushing) | dirtyscattered=2 lock=1 dirfrag=1 dirtyrstat=0 dirtyparent=1 dirty=1 waiter=1 authpin=1 scrubqueue=0 0x5647a570fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000000:[<0x1/dir v17>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-04-02T13:26:51.014993+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-04-02T13:26:51.131857+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2024-04-02T13:27:34.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f517468f640 1 -- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f515006c920 msgr2=0x7f515006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f517468f640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f515006c920 0x7f515006ede0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f515c000c00 tx=0x7f515c009040 comp rx=0 tx=0).stop 2024-04-02T13:27:34.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f517468f640 1 -- 172.21.15.44:0/1059761948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51580039b0 msgr2=0x7f516c0ae9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f517468f640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51580039b0 0x7f516c0ae9c0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f516802c4c0 tx=0x7f516c0ce650 comp rx=0 tx=0).stop 2024-04-02T13:27:34.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f517468f640 1 -- 172.21.15.44:0/1059761948 shutdown_connections 2024-04-02T13:27:34.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f517468f640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f515006c920 0x7f515006ede0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f517468f640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f516c154120 0x7f516c1545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f517468f640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f516c156da0 0x7f516c1f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f517468f640 1 --2- 172.21.15.44:0/1059761948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51580039b0 0x7f516c0ae9c0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.172+0000 7f517468f640 1 -- 172.21.15.44:0/1059761948 >> 172.21.15.44:0/1059761948 conn(0x7f516c06b920 msgr2=0x7f516c06e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:34.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.173+0000 7f517468f640 1 -- 172.21.15.44:0/1059761948 shutdown_connections 2024-04-02T13:27:34.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.173+0000 7f517468f640 1 -- 172.21.15.44:0/1059761948 wait complete. 2024-04-02T13:27:34.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:27:34.187 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-02T13:27:34.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.287+0000 7f62c46f2640 1 Processor -- start 2024-04-02T13:27:34.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.287+0000 7f62c46f2640 1 -- start start 2024-04-02T13:27:34.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c46f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc0b4a40 0x7f62bc0af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c46f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f62bc0ae230 0x7f62bc06b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c46f2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62bc0aff40 con 0x7f62bc0addd0 2024-04-02T13:27:34.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c46f2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f62bc0b0220 con 0x7f62bc0ae230 2024-04-02T13:27:34.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c46f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f62bc0b0500 con 0x7f62bc0b4a40 2024-04-02T13:27:34.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c2c8f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f62bc0ae230 0x7f62bc06b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c2c8f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f62bc0ae230 0x7f62bc06b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52484/0 (socket says 172.21.15.44:52484) 2024-04-02T13:27:34.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c2c8f640 1 -- 172.21.15.44:0/1463137884 learned_addr learned my addr 172.21.15.44:0/1463137884 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:34.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c248e640 1 --2- 172.21.15.44:0/1463137884 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc0b4a40 0x7f62bc0af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c2c8f640 1 -- 172.21.15.44:0/1463137884 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc0b4a40 msgr2=0x7f62bc0af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c2c8f640 1 --2- 172.21.15.44:0/1463137884 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc0b4a40 0x7f62bc0af8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c2c8f640 1 -- 172.21.15.44:0/1463137884 >> v1:172.21.15.69:6789/0 conn(0x7f62bc0addd0 legacy=0x7f62bc0b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c2c8f640 1 -- 172.21.15.44:0/1463137884 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62bc0d0ce0 con 0x7f62bc0ae230 2024-04-02T13:27:34.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.288+0000 7f62c248e640 1 --2- 172.21.15.44:0/1463137884 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc0b4a40 0x7f62bc0af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:27:34.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.289+0000 7f62c2c8f640 1 --2- 172.21.15.44:0/1463137884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62bc0ae230 0x7f62bc06b4d0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f62b4009e50 tx=0x7f62b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f383764ba98ca03e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.289+0000 7f62c148c640 1 -- 172.21.15.44:0/1463137884 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b402ca70 con 0x7f62bc0ae230 2024-04-02T13:27:34.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.289+0000 7f62c148c640 1 -- 172.21.15.44:0/1463137884 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b402cc10 con 0x7f62bc0ae230 2024-04-02T13:27:34.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.289+0000 7f62c148c640 1 -- 172.21.15.44:0/1463137884 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b40343f0 con 0x7f62bc0ae230 2024-04-02T13:27:34.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.290+0000 7f62c46f2640 1 -- 172.21.15.44:0/1463137884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62bc0ae230 msgr2=0x7f62bc06b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.290+0000 7f62c46f2640 1 --2- 172.21.15.44:0/1463137884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62bc0ae230 0x7f62bc06b4d0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f62b4009e50 tx=0x7f62b402bed0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.290+0000 7f62c46f2640 1 -- 172.21.15.44:0/1463137884 shutdown_connections 2024-04-02T13:27:34.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.290+0000 7f62c46f2640 1 --2- 172.21.15.44:0/1463137884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62bc0ae230 0x7f62bc06b4d0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.290+0000 7f62c46f2640 1 --2- 172.21.15.44:0/1463137884 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc0b4a40 0x7f62bc0af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.290+0000 7f62c46f2640 1 -- 172.21.15.44:0/1463137884 >> 172.21.15.44:0/1463137884 conn(0x7f62bc06c5e0 msgr2=0x7f62bc06c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:34.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.290+0000 7f62c46f2640 1 -- 172.21.15.44:0/1463137884 shutdown_connections 2024-04-02T13:27:34.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.290+0000 7f62c46f2640 1 -- 172.21.15.44:0/1463137884 wait complete. 2024-04-02T13:27:34.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.291+0000 7f62c46f2640 1 Processor -- start 2024-04-02T13:27:34.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.291+0000 7f62c46f2640 1 -- start start 2024-04-02T13:27:34.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c46f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc0addd0 0x7f62bc0cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c46f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62bc0ae230 0x7f62bc152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c46f2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62bc0b4a40 0x7f62bc142320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c46f2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f62bc0b4fa0 con 0x7f62bc0ae230 2024-04-02T13:27:34.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c46f2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f62bc0b4e20 con 0x7f62bc0b4a40 2024-04-02T13:27:34.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c46f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f62bc06ba10 con 0x7f62bc0addd0 2024-04-02T13:27:34.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c1c8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62bc0ae230 0x7f62bc152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c1c8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62bc0ae230 0x7f62bc152f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52500/0 (socket says 172.21.15.44:52500) 2024-04-02T13:27:34.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c1c8d640 1 -- 172.21.15.44:0/4126388379 learned_addr learned my addr 172.21.15.44:0/4126388379 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:34.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c248e640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc0addd0 0x7f62bc0cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c2c8f640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62bc0b4a40 0x7f62bc142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c1c8d640 1 -- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc0addd0 msgr2=0x7f62bc0cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c1c8d640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc0addd0 0x7f62bc0cf7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c1c8d640 1 -- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62bc0b4a40 msgr2=0x7f62bc142320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c1c8d640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62bc0b4a40 0x7f62bc142320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c1c8d640 1 -- 172.21.15.44:0/4126388379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62bc1493d0 con 0x7f62bc0ae230 2024-04-02T13:27:34.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c1c8d640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62bc0ae230 0x7f62bc152f00 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00c3f0 tx=0x7f62ac00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62b2ffd640 1 -- 172.21.15.44:0/4126388379 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac0052d0 con 0x7f62bc0ae230 2024-04-02T13:27:34.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62b2ffd640 1 -- 172.21.15.44:0/4126388379 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62ac005490 con 0x7f62bc0ae230 2024-04-02T13:27:34.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c46f2640 1 -- 172.21.15.44:0/4126388379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62bc1f5710 con 0x7f62bc0ae230 2024-04-02T13:27:34.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62b2ffd640 1 -- 172.21.15.44:0/4126388379 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac007260 con 0x7f62bc0ae230 2024-04-02T13:27:34.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.292+0000 7f62c46f2640 1 -- 172.21.15.44:0/4126388379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62bc1f5f50 con 0x7f62bc0ae230 2024-04-02T13:27:34.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.294+0000 7f62b2ffd640 1 -- 172.21.15.44:0/4126388379 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f62ac007400 con 0x7f62bc0ae230 2024-04-02T13:27:34.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.294+0000 7f62b2ffd640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f62a006c8f0 0x7f62a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.294+0000 7f62b2ffd640 1 -- 172.21.15.44:0/4126388379 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f62ac08b250 con 0x7f62bc0ae230 2024-04-02T13:27:34.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.295+0000 7f62c248e640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f62a006c8f0 0x7f62a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.295+0000 7f62c248e640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f62a006c8f0 0x7f62a006edb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f62b800d8b0 tx=0x7f62b8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.295+0000 7f62c73b4740 1 Processor -- start 2024-04-02T13:27:34.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.295+0000 7f62c73b4740 1 -- start start 2024-04-02T13:27:34.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c73b4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbeaf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c73b4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbe90d60 0x5557cbe68a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c73b4740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbe68f90 0x5557cbe93730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c73b4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5557cbdcd830 con 0x5557cbeb1520 2024-04-02T13:27:34.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c73b4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5557cbd75820 con 0x5557cbe90d60 2024-04-02T13:27:34.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c73b4740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5557cbdc97e0 con 0x5557cbe68f90 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c248e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbeaf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c248e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbeaf0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52508/0 (socket says 172.21.15.44:52508) 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c248e640 1 -- 172.21.15.44:0/2664165567 learned_addr learned my addr 172.21.15.44:0/2664165567 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c1c8d640 1 --2- 172.21.15.44:0/2664165567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbe90d60 0x5557cbe68a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c2c8f640 1 --2- 172.21.15.44:0/2664165567 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbe68f90 0x5557cbe93730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c248e640 1 -- 172.21.15.44:0/2664165567 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbe68f90 msgr2=0x5557cbe93730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c248e640 1 --2- 172.21.15.44:0/2664165567 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbe68f90 0x5557cbe93730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c248e640 1 -- 172.21.15.44:0/2664165567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbe90d60 msgr2=0x5557cbe68a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c248e640 1 --2- 172.21.15.44:0/2664165567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbe90d60 0x5557cbe68a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.296+0000 7f62c248e640 1 -- 172.21.15.44:0/2664165567 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557cbbab7d0 con 0x5557cbeb1520 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.297+0000 7f62c248e640 1 --2- 172.21.15.44:0/2664165567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbeaf0e0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f62b800fa20 tx=0x7f62b8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb743b654a3cb9d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.297+0000 7f629ffff640 1 -- 172.21.15.44:0/2664165567 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b801b050 con 0x5557cbeb1520 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.297+0000 7f629ffff640 1 -- 172.21.15.44:0/2664165567 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b8025070 con 0x5557cbeb1520 2024-04-02T13:27:34.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.297+0000 7f629ffff640 1 -- 172.21.15.44:0/2664165567 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b8020440 con 0x5557cbeb1520 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.297+0000 7f62c73b4740 1 -- 172.21.15.44:0/2664165567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 msgr2=0x5557cbeaf0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.297+0000 7f62c73b4740 1 --2- 172.21.15.44:0/2664165567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbeaf0e0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f62b800fa20 tx=0x7f62b8012000 comp rx=0 tx=0).stop 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.297+0000 7f62c73b4740 1 -- 172.21.15.44:0/2664165567 shutdown_connections 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.297+0000 7f62c73b4740 1 --2- 172.21.15.44:0/2664165567 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbe68f90 0x5557cbe93730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.297+0000 7f62c73b4740 1 --2- 172.21.15.44:0/2664165567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbe90d60 0x5557cbe68a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.297+0000 7f62c73b4740 1 --2- 172.21.15.44:0/2664165567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbeaf0e0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.297+0000 7f62c73b4740 1 -- 172.21.15.44:0/2664165567 >> 172.21.15.44:0/2664165567 conn(0x5557cbe65bb0 msgr2=0x5557cbeae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.298+0000 7f62c73b4740 1 -- 172.21.15.44:0/2664165567 shutdown_connections 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.298+0000 7f62c73b4740 1 -- 172.21.15.44:0/2664165567 wait complete. 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.298+0000 7f62c73b4740 1 -- start start 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.298+0000 7f62c73b4740 1 Processor -- start 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.299+0000 7f62c73b4740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbe90d60 0x5557cbe8c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.299+0000 7f62c73b4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbe92550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.299+0000 7f62c73b4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbec56b0 0x5557cbe92a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.299+0000 7f62c73b4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5557cbe67eb0 con 0x5557cbeb1520 2024-04-02T13:27:34.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.299+0000 7f62c73b4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5557cbd88d30 con 0x5557cbec56b0 2024-04-02T13:27:34.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.299+0000 7f62c73b4740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5557cbe641e0 con 0x5557cbe90d60 2024-04-02T13:27:34.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.299+0000 7f62c73b4740 10 client.0 _ll_register_callbacks cb 0x5557cbd88720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:34.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.299+0000 7f62c1c8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbe92550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.299+0000 7f62c1c8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbe92550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52514/0 (socket says 172.21.15.44:52514) 2024-04-02T13:27:34.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.299+0000 7f62c1c8d640 1 -- 172.21.15.44:0/643962812 learned_addr learned my addr 172.21.15.44:0/643962812 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:34.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.299+0000 7f62c248e640 1 --2- 172.21.15.44:0/643962812 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbe90d60 0x5557cbe8c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.299+0000 7f62c2c8f640 1 --2- 172.21.15.44:0/643962812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbec56b0 0x5557cbe92a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c1c8d640 1 -- 172.21.15.44:0/643962812 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbe90d60 msgr2=0x5557cbe8c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c1c8d640 1 --2- 172.21.15.44:0/643962812 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbe90d60 0x5557cbe8c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c1c8d640 1 -- 172.21.15.44:0/643962812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbec56b0 msgr2=0x5557cbe92a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c1c8d640 1 --2- 172.21.15.44:0/643962812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbec56b0 0x5557cbe92a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c1c8d640 1 -- 172.21.15.44:0/643962812 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557cbae13f0 con 0x5557cbeb1520 2024-04-02T13:27:34.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c1c8d640 1 --2- 172.21.15.44:0/643962812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbe92550 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f62ac058c60 tx=0x7f62ac059570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ffa0089ac8b2858 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f629d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:27:34.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f629d7fa640 1 -- 172.21.15.44:0/643962812 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac087060 con 0x5557cbeb1520 2024-04-02T13:27:34.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f629d7fa640 1 -- 172.21.15.44:0/643962812 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f62ac055390 con 0x5557cbeb1520 2024-04-02T13:27:34.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c73b4740 1 -- 172.21.15.44:0/643962812 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62b8025070 con 0x5557cbeb1520 2024-04-02T13:27:34.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f629d7fa640 1 -- 172.21.15.44:0/643962812 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f62ac076050 con 0x5557cbeb1520 2024-04-02T13:27:34.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c73b4740 10 client.8268 fetch_fsmap learned FSMap version 68 2024-04-02T13:27:34.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c73b4740 1 -- 172.21.15.44:0/643962812 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x5557cbaa10f0 con 0x5557cbeb1520 2024-04-02T13:27:34.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f629d7fa640 1 -- 172.21.15.44:0/643962812 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b8025070 con 0x5557cbeb1520 2024-04-02T13:27:34.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f629d7fa640 1 -- 172.21.15.44:0/643962812 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f62ac058170 con 0x5557cbeb1520 2024-04-02T13:27:34.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c73b4740 10 client.8268 fetch_fsmap finished waiting for FSMap version 68 2024-04-02T13:27:34.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c73b4740 10 client.8268 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:27:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c73b4740 20 client.8268 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c73b4740 1 --2- 172.21.15.44:0/643962812 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5557cbec6080 0x5557cbec25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c73b4740 4 client.8268 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:27:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c73b4740 1 -- 172.21.15.44:0/643962812 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557cbddad70 con 0x5557cbec6080 2024-04-02T13:27:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.300+0000 7f62c248e640 1 --2- 172.21.15.44:0/643962812 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5557cbec6080 0x5557cbec25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.301+0000 7f62c248e640 1 --2- 172.21.15.44:0/643962812 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5557cbec6080 0x5557cbec25c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c98a23adf23750a server_cookie=4741f4a95944838a in_seq=0 out_seq=0 2024-04-02T13:27:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.301+0000 7f629d7fa640 10 client.8268 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.301+0000 7f629d7fa640 1 -- 172.21.15.44:0/643962812 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5557cbddad70 con 0x5557cbec6080 2024-04-02T13:27:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.301+0000 7f629d7fa640 10 client.8268 handle_command_reply: tid=0 2024-04-02T13:27:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.301+0000 7f62c73b4740 1 client.8268 shutdown 2024-04-02T13:27:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.301+0000 7f62c73b4740 1 -- 172.21.15.44:0/643962812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 msgr2=0x5557cbe92550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.301+0000 7f62c73b4740 1 --2- 172.21.15.44:0/643962812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbe92550 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f62ac058c60 tx=0x7f62ac059570 comp rx=0 tx=0).stop 2024-04-02T13:27:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.302+0000 7f62c73b4740 1 -- 172.21.15.44:0/643962812 shutdown_connections 2024-04-02T13:27:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.302+0000 7f62c73b4740 1 --2- 172.21.15.44:0/643962812 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5557cbec6080 0x5557cbec25c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.302+0000 7f62c73b4740 1 --2- 172.21.15.44:0/643962812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbec56b0 0x5557cbe92a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.302+0000 7f62c73b4740 1 --2- 172.21.15.44:0/643962812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbe92550 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.302+0000 7f629d7fa640 0 client.8268 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.302+0000 7f62c73b4740 1 --2- 172.21.15.44:0/643962812 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbe90d60 0x5557cbe8c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.302+0000 7f62c73b4740 1 -- 172.21.15.44:0/643962812 >> 172.21.15.44:0/643962812 conn(0x5557cbe68f90 msgr2=0x5557cbeae650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.302+0000 7f62c73b4740 1 -- 172.21.15.44:0/643962812 shutdown_connections 2024-04-02T13:27:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.302+0000 7f62c73b4740 1 -- 172.21.15.44:0/643962812 wait complete. 2024-04-02T13:27:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.302+0000 7f62c73b4740 20 client.8268 trim_cache size 0 max 16384 2024-04-02T13:27:34.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.319+0000 7f62c73b4740 1 Processor -- start 2024-04-02T13:27:34.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.319+0000 7f62c73b4740 1 -- start start 2024-04-02T13:27:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c73b4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbe90d60 0x5557cbe8bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c73b4740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbeb1520 0x5557cbe8c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c73b4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbec56b0 0x5557cbeb1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c73b4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f62bc1fcd50 con 0x5557cbe90d60 2024-04-02T13:27:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c73b4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f62bc06fc50 con 0x5557cbec56b0 2024-04-02T13:27:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c73b4740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f62bc145f50 con 0x5557cbeb1520 2024-04-02T13:27:34.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c248e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbe90d60 0x5557cbe8bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c248e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbe90d60 0x5557cbe8bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52528/0 (socket says 172.21.15.44:52528) 2024-04-02T13:27:34.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c248e640 1 -- 172.21.15.44:0/3241084714 learned_addr learned my addr 172.21.15.44:0/3241084714 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:34.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c248e640 1 -- 172.21.15.44:0/3241084714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbeb1520 msgr2=0x5557cbe8c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c248e640 1 --2- 172.21.15.44:0/3241084714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbeb1520 0x5557cbe8c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c248e640 1 -- 172.21.15.44:0/3241084714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbec56b0 msgr2=0x5557cbeb1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c248e640 1 --2- 172.21.15.44:0/3241084714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbec56b0 0x5557cbeb1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c248e640 1 -- 172.21.15.44:0/3241084714 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557cbe657f0 con 0x5557cbe90d60 2024-04-02T13:27:34.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.320+0000 7f62c248e640 1 --2- 172.21.15.44:0/3241084714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbe90d60 0x5557cbe8bb40 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f62b8021000 tx=0x7f62b8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270d55001c5a3d5b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.321+0000 7f629f7fe640 1 -- 172.21.15.44:0/3241084714 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b801b070 con 0x5557cbe90d60 2024-04-02T13:27:34.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.321+0000 7f629f7fe640 1 -- 172.21.15.44:0/3241084714 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b8025070 con 0x5557cbe90d60 2024-04-02T13:27:34.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.321+0000 7f629f7fe640 1 -- 172.21.15.44:0/3241084714 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b8004030 con 0x5557cbe90d60 2024-04-02T13:27:34.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.321+0000 7f62c73b4740 1 -- 172.21.15.44:0/3241084714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbe90d60 msgr2=0x5557cbe8bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.321+0000 7f62c73b4740 1 --2- 172.21.15.44:0/3241084714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbe90d60 0x5557cbe8bb40 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f62b8021000 tx=0x7f62b8014000 comp rx=0 tx=0).stop 2024-04-02T13:27:34.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.321+0000 7f62c73b4740 1 -- 172.21.15.44:0/3241084714 shutdown_connections 2024-04-02T13:27:34.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.321+0000 7f62c73b4740 1 --2- 172.21.15.44:0/3241084714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbec56b0 0x5557cbeb1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.321+0000 7f62c73b4740 1 --2- 172.21.15.44:0/3241084714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbeb1520 0x5557cbe8c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.321+0000 7f62c73b4740 1 --2- 172.21.15.44:0/3241084714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbe90d60 0x5557cbe8bb40 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.321+0000 7f62c73b4740 1 -- 172.21.15.44:0/3241084714 >> 172.21.15.44:0/3241084714 conn(0x5557cbe68f90 msgr2=0x5557cbeae650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:34.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.321+0000 7f62c73b4740 1 -- 172.21.15.44:0/3241084714 shutdown_connections 2024-04-02T13:27:34.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.321+0000 7f62c73b4740 1 -- 172.21.15.44:0/3241084714 wait complete. 2024-04-02T13:27:34.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.322+0000 7f62c73b4740 1 -- start start 2024-04-02T13:27:34.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.322+0000 7f62c73b4740 1 Processor -- start 2024-04-02T13:27:34.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c73b4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbe90d60 0x5557cbe96880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c73b4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbeae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c73b4740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbec56b0 0x5557cbf14c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c73b4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f62bc06f600 con 0x5557cbeb1520 2024-04-02T13:27:34.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c73b4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5557cbe8cb30 con 0x5557cbe90d60 2024-04-02T13:27:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c73b4740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5557cbf2ca40 con 0x5557cbec56b0 2024-04-02T13:27:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c73b4740 10 client.0 _ll_register_callbacks cb 0x7f62ac059420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:27:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c1c8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbeae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c1c8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbeae6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52536/0 (socket says 172.21.15.44:52536) 2024-04-02T13:27:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c1c8d640 1 -- 172.21.15.44:0/58656149 learned_addr learned my addr 172.21.15.44:0/58656149 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:27:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c248e640 1 --2- 172.21.15.44:0/58656149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbe90d60 0x5557cbe96880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c2c8f640 1 --2- 172.21.15.44:0/58656149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbec56b0 0x5557cbf14c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c1c8d640 1 -- 172.21.15.44:0/58656149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbec56b0 msgr2=0x5557cbf14c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c1c8d640 1 --2- 172.21.15.44:0/58656149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbec56b0 0x5557cbf14c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c1c8d640 1 -- 172.21.15.44:0/58656149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbe90d60 msgr2=0x5557cbe96880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:27:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c1c8d640 1 --2- 172.21.15.44:0/58656149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbe90d60 0x5557cbe96880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c1c8d640 1 -- 172.21.15.44:0/58656149 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557cbeaec20 con 0x5557cbeb1520 2024-04-02T13:27:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c1c8d640 1 --2- 172.21.15.44:0/58656149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbeae6e0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f62ac055180 tx=0x7f62ac000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d005c3ab94ceb7f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:27:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f629d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:27:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f629d7fa640 1 -- 172.21.15.44:0/58656149 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac061900 con 0x5557cbeb1520 2024-04-02T13:27:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f629d7fa640 1 -- 172.21.15.44:0/58656149 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f62ac05f2d0 con 0x5557cbeb1520 2024-04-02T13:27:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.323+0000 7f62c73b4740 1 -- 172.21.15.44:0/58656149 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62b8025070 con 0x5557cbeb1520 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f629d7fa640 1 -- 172.21.15.44:0/58656149 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f62ac081050 con 0x5557cbeb1520 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f62c73b4740 10 client.8274 fetch_fsmap learned FSMap version 68 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f62c73b4740 1 -- 172.21.15.44:0/58656149 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x5557cbf2c780 con 0x5557cbeb1520 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f629d7fa640 1 -- 172.21.15.44:0/58656149 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b8025070 con 0x5557cbeb1520 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f629d7fa640 1 -- 172.21.15.44:0/58656149 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f62ac054a70 con 0x5557cbeb1520 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f62c73b4740 10 client.8274 fetch_fsmap finished waiting for FSMap version 68 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f62c73b4740 10 client.8274 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f62c73b4740 20 client.8274 populate_metadata read hostname 'smithi044' 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f62c73b4740 1 --2- 172.21.15.44:0/58656149 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5557cbf15180 0x5557cbe8fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f62c73b4740 4 client.8274 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f62c73b4740 1 -- 172.21.15.44:0/58656149 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5557cbdd50c0 con 0x5557cbf15180 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f62c248e640 1 --2- 172.21.15.44:0/58656149 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5557cbf15180 0x5557cbe8fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f629d7fa640 10 client.8274 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:34.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f62c248e640 1 --2- 172.21.15.44:0/58656149 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5557cbf15180 0x5557cbe8fa10 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0a2c35f19097cf5 server_cookie=336000f77a3fa60a in_seq=0 out_seq=0 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f629d7fa640 1 -- 172.21.15.44:0/58656149 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+263 (crc 0 0 0) 0x5557cbdd50c0 con 0x5557cbf15180 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f629d7fa640 10 client.8274 handle_command_reply: tid=0 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.324+0000 7f62c73b4740 1 client.8274 shutdown 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c73b4740 1 -- 172.21.15.44:0/58656149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 msgr2=0x5557cbeae6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c73b4740 1 --2- 172.21.15.44:0/58656149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbeae6e0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f62ac055180 tx=0x7f62ac000bc0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c73b4740 1 -- 172.21.15.44:0/58656149 shutdown_connections 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c73b4740 1 --2- 172.21.15.44:0/58656149 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5557cbf15180 0x5557cbe8fa10 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c73b4740 1 --2- 172.21.15.44:0/58656149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5557cbec56b0 0x5557cbf14c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c73b4740 1 --2- 172.21.15.44:0/58656149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557cbeb1520 0x5557cbeae6e0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c73b4740 1 --2- 172.21.15.44:0/58656149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557cbe90d60 0x5557cbe96880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f629d7fa640 0 client.8274 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c73b4740 1 -- 172.21.15.44:0/58656149 >> 172.21.15.44:0/58656149 conn(0x5557cbe68f90 msgr2=0x5557cbebac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c73b4740 1 -- 172.21.15.44:0/58656149 shutdown_connections 2024-04-02T13:27:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c73b4740 1 -- 172.21.15.44:0/58656149 wait complete. 2024-04-02T13:27:34.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c73b4740 20 client.8274 trim_cache size 0 max 16384 2024-04-02T13:27:34.336 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:27:34.336 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T13:27:34.336 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-02T13:27:34.336 INFO:teuthology.orchestra.run.smithi044.stdout: "d2d686a1-3479-4fc7-9e78-87e5581c59ed": { 2024-04-02T13:27:34.336 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/dir", 2024-04-02T13:27:34.336 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "d2d686a1-3479-4fc7-9e78-87e5581c59ed", 2024-04-02T13:27:34.336 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,repair" 2024-04-02T13:27:34.336 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:27:34.336 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:27:34.337 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:27:34.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c46f2640 1 -- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f62a006c8f0 msgr2=0x7f62a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c46f2640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f62a006c8f0 0x7f62a006edb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f62b800d8b0 tx=0x7f62b8008040 comp rx=0 tx=0).stop 2024-04-02T13:27:34.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c46f2640 1 -- 172.21.15.44:0/4126388379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62bc0ae230 msgr2=0x7f62bc152f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:27:34.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c46f2640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62bc0ae230 0x7f62bc152f00 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00c3f0 tx=0x7f62ac00c8b0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c46f2640 1 -- 172.21.15.44:0/4126388379 shutdown_connections 2024-04-02T13:27:34.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c46f2640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f62a006c8f0 0x7f62a006edb0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c46f2640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62bc0b4a40 0x7f62bc142320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c46f2640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62bc0ae230 0x7f62bc152f00 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c46f2640 1 --2- 172.21.15.44:0/4126388379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc0addd0 0x7f62bc0cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:27:34.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c46f2640 1 -- 172.21.15.44:0/4126388379 >> 172.21.15.44:0/4126388379 conn(0x7f62bc06c5e0 msgr2=0x7f62bc0d02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:27:34.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c46f2640 1 -- 172.21.15.44:0/4126388379 shutdown_connections 2024-04-02T13:27:34.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:27:34.325+0000 7f62c46f2640 1 -- 172.21.15.44:0/4126388379 wait complete. 2024-04-02T13:27:34.339 INFO:tasks.cephfs.filesystem:scrub status for tag:d2d686a1-3479-4fc7-9e78-87e5581c59ed - {'path': '/dir', 'tag': 'd2d686a1-3479-4fc7-9e78-87e5581c59ed', 'options': 'recursive,repair'} 2024-04-02T13:27:53.038 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:27:53.041 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:28:04.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:04.341 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-02T13:28:04.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.455+0000 7f53a359e640 1 Processor -- start 2024-04-02T13:28:04.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.455+0000 7f53a359e640 1 -- start start 2024-04-02T13:28:04.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.456+0000 7f53a359e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53a406e8d0 0x7f53a406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.456+0000 7f53a359e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f53a4071df0 0x7f53a4072260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.456+0000 7f53a359e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53a40caf60 con 0x7f53a4155380 2024-04-02T13:28:04.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.456+0000 7f53a359e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f53a406f270 con 0x7f53a4071df0 2024-04-02T13:28:04.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.456+0000 7f53a359e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f53a4155760 con 0x7f53a406e8d0 2024-04-02T13:28:04.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.456+0000 7f53a1d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53a406e8d0 0x7f53a406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.456+0000 7f53a2d9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f53a4071df0 0x7f53a4072260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.456+0000 7f53a259c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f53a4155380 0x7f53a40ca510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:57208/0 (socket says 172.21.15.44:57208) 2024-04-02T13:28:04.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.456+0000 7f53a259c640 1 -- 172.21.15.44:0/698030755 learned_addr learned my addr 172.21.15.44:0/698030755 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:04.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.457+0000 7f53a1d9b640 1 -- 172.21.15.44:0/698030755 >> v2:172.21.15.69:3300/0 conn(0x7f53a4071df0 msgr2=0x7f53a4072260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.457+0000 7f53a1d9b640 1 --2- 172.21.15.44:0/698030755 >> v2:172.21.15.69:3300/0 conn(0x7f53a4071df0 0x7f53a4072260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.457+0000 7f53a1d9b640 1 -- 172.21.15.44:0/698030755 >> v1:172.21.15.69:6789/0 conn(0x7f53a4155380 legacy=0x7f53a40ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.457+0000 7f53a1d9b640 1 -- 172.21.15.44:0/698030755 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a40cb240 con 0x7f53a406e8d0 2024-04-02T13:28:04.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.457+0000 7f53a2d9d640 1 --2- 172.21.15.44:0/698030755 >> v2:172.21.15.69:3300/0 conn(0x7f53a4071df0 0x7f53a4072260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:28:04.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.457+0000 7f53a1d9b640 1 --2- 172.21.15.44:0/698030755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53a406e8d0 0x7f53a406ed30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f53980097a0 tx=0x7f539802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ede1f74964f5ebb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.458+0000 7f53a159a640 1 -- 172.21.15.44:0/698030755 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398035020 con 0x7f53a406e8d0 2024-04-02T13:28:04.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.458+0000 7f53a159a640 1 -- 172.21.15.44:0/698030755 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5398031830 con 0x7f53a406e8d0 2024-04-02T13:28:04.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.458+0000 7f53a159a640 1 -- 172.21.15.44:0/698030755 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398031c00 con 0x7f53a406e8d0 2024-04-02T13:28:04.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.458+0000 7f53a359e640 1 -- 172.21.15.44:0/698030755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53a406e8d0 msgr2=0x7f53a406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.458+0000 7f53a359e640 1 --2- 172.21.15.44:0/698030755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53a406e8d0 0x7f53a406ed30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f53980097a0 tx=0x7f539802b8d0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.458+0000 7f53a359e640 1 -- 172.21.15.44:0/698030755 shutdown_connections 2024-04-02T13:28:04.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.458+0000 7f53a359e640 1 --2- 172.21.15.44:0/698030755 >> v2:172.21.15.69:3300/0 conn(0x7f53a4071df0 0x7f53a4072260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.458+0000 7f53a359e640 1 --2- 172.21.15.44:0/698030755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53a406e8d0 0x7f53a406ed30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.458+0000 7f53a359e640 1 -- 172.21.15.44:0/698030755 >> 172.21.15.44:0/698030755 conn(0x7f53a40cfa90 msgr2=0x7f53a40cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:04.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.458+0000 7f53a359e640 1 -- 172.21.15.44:0/698030755 shutdown_connections 2024-04-02T13:28:04.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.458+0000 7f53a359e640 1 -- 172.21.15.44:0/698030755 wait complete. 2024-04-02T13:28:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.460+0000 7f53a359e640 1 Processor -- start 2024-04-02T13:28:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.460+0000 7f53a359e640 1 -- start start 2024-04-02T13:28:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.460+0000 7f53a359e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53a406e8d0 0x7f53a41f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.460+0000 7f53a359e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53a4071df0 0x7f53a40bddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.460+0000 7f53a359e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53a4155380 0x7f53a40b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.460+0000 7f53a359e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f53a40cac20 con 0x7f53a4071df0 2024-04-02T13:28:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.460+0000 7f53a359e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f53a40718d0 con 0x7f53a406e8d0 2024-04-02T13:28:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.460+0000 7f53a359e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f53a40cd660 con 0x7f53a4155380 2024-04-02T13:28:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.460+0000 7f53a1d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53a4071df0 0x7f53a40bddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f53a1d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53a4071df0 0x7f53a40bddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37474/0 (socket says 172.21.15.44:37474) 2024-04-02T13:28:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f53a1d9b640 1 -- 172.21.15.44:0/3086607221 learned_addr learned my addr 172.21.15.44:0/3086607221 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f53a259c640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53a406e8d0 0x7f53a41f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f53a2d9d640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53a4155380 0x7f53a40b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f53a1d9b640 1 -- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53a4155380 msgr2=0x7f53a40b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f53a1d9b640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53a4155380 0x7f53a40b02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f53a1d9b640 1 -- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53a406e8d0 msgr2=0x7f53a41f1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f53a1d9b640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53a406e8d0 0x7f53a41f1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f53a1d9b640 1 -- 172.21.15.44:0/3086607221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a40c6e80 con 0x7f53a4071df0 2024-04-02T13:28:04.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f53a1d9b640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53a4071df0 0x7f53a40bddb0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f539802bee0 tx=0x7f53980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f5392ffd640 1 -- 172.21.15.44:0/3086607221 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398035040 con 0x7f53a4071df0 2024-04-02T13:28:04.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f5392ffd640 1 -- 172.21.15.44:0/3086607221 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f539803b860 con 0x7f53a4071df0 2024-04-02T13:28:04.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f53a359e640 1 -- 172.21.15.44:0/3086607221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a41f6200 con 0x7f53a4071df0 2024-04-02T13:28:04.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f5392ffd640 1 -- 172.21.15.44:0/3086607221 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539803bc50 con 0x7f53a4071df0 2024-04-02T13:28:04.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.461+0000 7f53a359e640 1 -- 172.21.15.44:0/3086607221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a4153270 con 0x7f53a4071df0 2024-04-02T13:28:04.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.463+0000 7f5392ffd640 1 -- 172.21.15.44:0/3086607221 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5398038020 con 0x7f53a4071df0 2024-04-02T13:28:04.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.463+0000 7f5392ffd640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f538406c920 0x7f538406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.463+0000 7f5392ffd640 1 -- 172.21.15.44:0/3086607221 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f53980ade00 con 0x7f53a4071df0 2024-04-02T13:28:04.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.464+0000 7f53a259c640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f538406c920 0x7f538406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.464+0000 7f53a259c640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f538406c920 0x7f538406ede0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f53a40c40a0 tx=0x7f5394007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.464+0000 7f53ab3df740 1 Processor -- start 2024-04-02T13:28:04.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.464+0000 7f53ab3df740 1 -- start start 2024-04-02T13:28:04.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.465+0000 7f53ab3df740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a70715730 0x555a707132f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.465+0000 7f53ab3df740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a706f4f70 0x555a706ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.465+0000 7f53ab3df740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706cd1a0 0x555a706f7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.465+0000 7f53ab3df740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555a70631830 con 0x555a706cd1a0 2024-04-02T13:28:04.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.465+0000 7f53ab3df740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555a705d9820 con 0x555a706f4f70 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.465+0000 7f53ab3df740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x555a7062d7e0 con 0x555a70715730 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.465+0000 7f53a2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706cd1a0 0x555a706f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.465+0000 7f53a259c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a70715730 0x555a707132f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.465+0000 7f53a2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706cd1a0 0x555a706f7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37488/0 (socket says 172.21.15.44:37488) 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.465+0000 7f53a2d9d640 1 -- 172.21.15.44:0/1247011330 learned_addr learned my addr 172.21.15.44:0/1247011330 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.465+0000 7f53a1d9b640 1 --2- 172.21.15.44:0/1247011330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a706f4f70 0x555a706ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.466+0000 7f53a2d9d640 1 -- 172.21.15.44:0/1247011330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a70715730 msgr2=0x555a707132f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.466+0000 7f53a2d9d640 1 --2- 172.21.15.44:0/1247011330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a70715730 0x555a707132f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.466+0000 7f53a2d9d640 1 -- 172.21.15.44:0/1247011330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a706f4f70 msgr2=0x555a706ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.466+0000 7f53a2d9d640 1 --2- 172.21.15.44:0/1247011330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a706f4f70 0x555a706ccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.466+0000 7f53a2d9d640 1 -- 172.21.15.44:0/1247011330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a7040f7d0 con 0x555a706cd1a0 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.466+0000 7f53a259c640 1 --2- 172.21.15.44:0/1247011330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a70715730 0x555a707132f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.466+0000 7f53a2d9d640 1 --2- 172.21.15.44:0/1247011330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706cd1a0 0x555a706f7940 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f539c0115b0 tx=0x7f539c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c098380d90d617a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.466+0000 7f5383fff640 1 -- 172.21.15.44:0/1247011330 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c003b90 con 0x555a706cd1a0 2024-04-02T13:28:04.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.466+0000 7f5383fff640 1 -- 172.21.15.44:0/1247011330 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f539c003d30 con 0x555a706cd1a0 2024-04-02T13:28:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.466+0000 7f5383fff640 1 -- 172.21.15.44:0/1247011330 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c004240 con 0x555a706cd1a0 2024-04-02T13:28:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.467+0000 7f53ab3df740 1 -- 172.21.15.44:0/1247011330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706cd1a0 msgr2=0x555a706f7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.467+0000 7f53ab3df740 1 --2- 172.21.15.44:0/1247011330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706cd1a0 0x555a706f7940 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f539c0115b0 tx=0x7f539c009750 comp rx=0 tx=0).stop 2024-04-02T13:28:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.467+0000 7f53ab3df740 1 -- 172.21.15.44:0/1247011330 shutdown_connections 2024-04-02T13:28:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.467+0000 7f53ab3df740 1 --2- 172.21.15.44:0/1247011330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706cd1a0 0x555a706f7940 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.467+0000 7f53ab3df740 1 --2- 172.21.15.44:0/1247011330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a706f4f70 0x555a706ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.467+0000 7f53ab3df740 1 --2- 172.21.15.44:0/1247011330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a70715730 0x555a707132f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.467+0000 7f53ab3df740 1 -- 172.21.15.44:0/1247011330 >> 172.21.15.44:0/1247011330 conn(0x555a706c9dc0 msgr2=0x555a70712940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.467+0000 7f53ab3df740 1 -- 172.21.15.44:0/1247011330 shutdown_connections 2024-04-02T13:28:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.467+0000 7f53ab3df740 1 -- 172.21.15.44:0/1247011330 wait complete. 2024-04-02T13:28:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.467+0000 7f53ab3df740 1 -- start start 2024-04-02T13:28:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.467+0000 7f53ab3df740 1 Processor -- start 2024-04-02T13:28:04.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.468+0000 7f53ab3df740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 0x555a706e1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.468+0000 7f53ab3df740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a70715730 0x555a7070c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.468+0000 7f53ab3df740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a7070ee50 0x555a7070f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.468+0000 7f53ab3df740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555a706cc0c0 con 0x555a706f4f70 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.468+0000 7f53ab3df740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555a705ecd30 con 0x555a70715730 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.468+0000 7f53ab3df740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x555a706c83f0 con 0x555a7070ee50 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.468+0000 7f53ab3df740 10 client.0 _ll_register_callbacks cb 0x555a705ec720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.468+0000 7f53a259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 0x555a706e1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.468+0000 7f53a259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 0x555a706e1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37502/0 (socket says 172.21.15.44:37502) 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.468+0000 7f53a259c640 1 -- 172.21.15.44:0/4262881762 learned_addr learned my addr 172.21.15.44:0/4262881762 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53a1d9b640 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a70715730 0x555a7070c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53a2d9d640 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a7070ee50 0x555a7070f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53a259c640 1 -- 172.21.15.44:0/4262881762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a7070ee50 msgr2=0x555a7070f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53a259c640 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a7070ee50 0x555a7070f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53a259c640 1 -- 172.21.15.44:0/4262881762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a70715730 msgr2=0x555a7070c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53a259c640 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a70715730 0x555a7070c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53a259c640 1 -- 172.21.15.44:0/4262881762 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a703453f0 con 0x555a706f4f70 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53a2d9d640 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a7070ee50 0x555a7070f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53a259c640 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 0x555a706e1e10 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f53940134f0 tx=0x7f5394000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e530a56724b74200 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53817fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53817fa640 1 -- 172.21.15.44:0/4262881762 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394020580 con 0x555a706f4f70 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53817fa640 1 -- 172.21.15.44:0/4262881762 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f5394020720 con 0x555a706f4f70 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53ab3df740 1 -- 172.21.15.44:0/4262881762 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f539c003d30 con 0x555a706f4f70 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53817fa640 1 -- 172.21.15.44:0/4262881762 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f539402e050 con 0x555a706f4f70 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53ab3df740 10 client.8283 fetch_fsmap learned FSMap version 68 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53ab3df740 1 -- 172.21.15.44:0/4262881762 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x555a703050f0 con 0x555a706f4f70 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.469+0000 7f53817fa640 1 -- 172.21.15.44:0/4262881762 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f539c003d30 con 0x555a706f4f70 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.470+0000 7f53817fa640 1 -- 172.21.15.44:0/4262881762 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f5394031020 con 0x555a706f4f70 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.470+0000 7f53ab3df740 10 client.8283 fetch_fsmap finished waiting for FSMap version 68 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.470+0000 7f53ab3df740 10 client.8283 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.470+0000 7f53ab3df740 20 client.8283 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:04.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.470+0000 7f53ab3df740 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x555a7070a960 0x555a7070ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.470+0000 7f53ab3df740 4 client.8283 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:28:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.470+0000 7f53ab3df740 1 -- 172.21.15.44:0/4262881762 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a7063ed70 con 0x555a7070a960 2024-04-02T13:28:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.470+0000 7f53a1d9b640 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x555a7070a960 0x555a7070ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.470+0000 7f53a1d9b640 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x555a7070a960 0x555a7070ada0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c0af29ed5533608 server_cookie=461969cb447e4e2 in_seq=0 out_seq=0 2024-04-02T13:28:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.470+0000 7f53817fa640 10 client.8283 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:28:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53817fa640 1 -- 172.21.15.44:0/4262881762 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a7063ed70 con 0x555a7070a960 2024-04-02T13:28:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53817fa640 10 client.8283 handle_command_reply: tid=0 2024-04-02T13:28:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53ab3df740 1 client.8283 shutdown 2024-04-02T13:28:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53ab3df740 1 -- 172.21.15.44:0/4262881762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 msgr2=0x555a706e1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53ab3df740 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 0x555a706e1e10 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f53940134f0 tx=0x7f5394000bc0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53ab3df740 1 -- 172.21.15.44:0/4262881762 shutdown_connections 2024-04-02T13:28:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53ab3df740 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x555a7070a960 0x555a7070ada0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53ab3df740 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a7070ee50 0x555a7070f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53ab3df740 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a70715730 0x555a7070c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53817fa640 0 client.8283 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:28:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53ab3df740 1 --2- 172.21.15.44:0/4262881762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 0x555a706e1e10 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53ab3df740 1 -- 172.21.15.44:0/4262881762 >> 172.21.15.44:0/4262881762 conn(0x555a706cd1a0 msgr2=0x555a70712860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53ab3df740 1 -- 172.21.15.44:0/4262881762 shutdown_connections 2024-04-02T13:28:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53ab3df740 1 -- 172.21.15.44:0/4262881762 wait complete. 2024-04-02T13:28:04.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.471+0000 7f53ab3df740 20 client.8283 trim_cache size 0 max 16384 2024-04-02T13:28:04.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.489+0000 7f53ab3df740 1 Processor -- start 2024-04-02T13:28:04.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.489+0000 7f53ab3df740 1 -- start start 2024-04-02T13:28:04.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53ab3df740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 0x555a7073a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53ab3df740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a7070ee50 0x555a7070c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53ab3df740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a70715730 0x555a7070c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53ab3df740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f53a4150f10 con 0x555a706f4f70 2024-04-02T13:28:04.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53ab3df740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f53a4151e20 con 0x555a7070ee50 2024-04-02T13:28:04.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53ab3df740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f53a41f4d20 con 0x555a70715730 2024-04-02T13:28:04.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53a259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 0x555a7073a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53a259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 0x555a7073a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37512/0 (socket says 172.21.15.44:37512) 2024-04-02T13:28:04.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53a259c640 1 -- 172.21.15.44:0/2551376122 learned_addr learned my addr 172.21.15.44:0/2551376122 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:04.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53a1d9b640 1 --2- 172.21.15.44:0/2551376122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a7070ee50 0x555a7070c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53a2d9d640 1 --2- 172.21.15.44:0/2551376122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a70715730 0x555a7070c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53a259c640 1 -- 172.21.15.44:0/2551376122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a70715730 msgr2=0x555a7070c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53a259c640 1 --2- 172.21.15.44:0/2551376122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a70715730 0x555a7070c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53a259c640 1 -- 172.21.15.44:0/2551376122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a7070ee50 msgr2=0x555a7070c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53a259c640 1 --2- 172.21.15.44:0/2551376122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a7070ee50 0x555a7070c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53a259c640 1 -- 172.21.15.44:0/2551376122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a706fbe30 con 0x555a706f4f70 2024-04-02T13:28:04.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53a1d9b640 1 --2- 172.21.15.44:0/2551376122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a7070ee50 0x555a7070c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:04.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53a259c640 1 --2- 172.21.15.44:0/2551376122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 0x555a7073a970 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f5394007f50 tx=0x7f5394021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ba3c326aa6b934 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53837fe640 1 -- 172.21.15.44:0/2551376122 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394020d30 con 0x555a706f4f70 2024-04-02T13:28:04.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53837fe640 1 -- 172.21.15.44:0/2551376122 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5394026070 con 0x555a706f4f70 2024-04-02T13:28:04.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.490+0000 7f53837fe640 1 -- 172.21.15.44:0/2551376122 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394020780 con 0x555a706f4f70 2024-04-02T13:28:04.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.491+0000 7f53ab3df740 1 -- 172.21.15.44:0/2551376122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 msgr2=0x555a7073a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.491+0000 7f53ab3df740 1 --2- 172.21.15.44:0/2551376122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 0x555a7073a970 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f5394007f50 tx=0x7f5394021000 comp rx=0 tx=0).stop 2024-04-02T13:28:04.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.491+0000 7f53ab3df740 1 -- 172.21.15.44:0/2551376122 shutdown_connections 2024-04-02T13:28:04.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.491+0000 7f53ab3df740 1 --2- 172.21.15.44:0/2551376122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a70715730 0x555a7070c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.491+0000 7f53ab3df740 1 --2- 172.21.15.44:0/2551376122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a7070ee50 0x555a7070c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.491+0000 7f53ab3df740 1 --2- 172.21.15.44:0/2551376122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a706f4f70 0x555a7073a970 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.491+0000 7f53ab3df740 1 -- 172.21.15.44:0/2551376122 >> 172.21.15.44:0/2551376122 conn(0x555a706cd1a0 msgr2=0x555a70712860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:04.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.491+0000 7f53ab3df740 1 -- 172.21.15.44:0/2551376122 shutdown_connections 2024-04-02T13:28:04.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.491+0000 7f53ab3df740 1 -- 172.21.15.44:0/2551376122 wait complete. 2024-04-02T13:28:04.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.491+0000 7f53ab3df740 1 -- start start 2024-04-02T13:28:04.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.492+0000 7f53ab3df740 1 Processor -- start 2024-04-02T13:28:04.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.492+0000 7f53ab3df740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a706f4f70 0x555a7070c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.492+0000 7f53ab3df740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a7070ee50 0x555a7070c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.492+0000 7f53ab3df740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a70715730 0x555a707293f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.492+0000 7f53ab3df740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555a706e21f0 con 0x555a70715730 2024-04-02T13:28:04.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.492+0000 7f53ab3df740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f53a41517d0 con 0x555a706f4f70 2024-04-02T13:28:04.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.492+0000 7f53ab3df740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x555a706ce730 con 0x555a7070ee50 2024-04-02T13:28:04.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53ab3df740 10 client.0 _ll_register_callbacks cb 0x7f5394001bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:04.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53a2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a70715730 0x555a707293f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53a2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a70715730 0x555a707293f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37522/0 (socket says 172.21.15.44:37522) 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53a2d9d640 1 -- 172.21.15.44:0/2991156657 learned_addr learned my addr 172.21.15.44:0/2991156657 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53a259c640 1 --2- 172.21.15.44:0/2991156657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a706f4f70 0x555a7070c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53a1d9b640 1 --2- 172.21.15.44:0/2991156657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a7070ee50 0x555a7070c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53a2d9d640 1 -- 172.21.15.44:0/2991156657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a7070ee50 msgr2=0x555a7070c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53a2d9d640 1 --2- 172.21.15.44:0/2991156657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a7070ee50 0x555a7070c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53a2d9d640 1 -- 172.21.15.44:0/2991156657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a706f4f70 msgr2=0x555a7070c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53a2d9d640 1 --2- 172.21.15.44:0/2991156657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a706f4f70 0x555a7070c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53a2d9d640 1 -- 172.21.15.44:0/2991156657 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a70728f10 con 0x555a70715730 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53a2d9d640 1 --2- 172.21.15.44:0/2991156657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a70715730 0x555a707293f0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f539c011580 tx=0x7f539c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73322912f78656e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53817fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53817fa640 1 -- 172.21.15.44:0/2991156657 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c015a50 con 0x555a70715730 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53817fa640 1 -- 172.21.15.44:0/2991156657 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f539c015bf0 con 0x555a70715730 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53ab3df740 1 -- 172.21.15.44:0/2991156657 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5394026070 con 0x555a70715730 2024-04-02T13:28:04.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53817fa640 1 -- 172.21.15.44:0/2991156657 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f539c033050 con 0x555a70715730 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53ab3df740 10 client.8289 fetch_fsmap learned FSMap version 68 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.493+0000 7f53ab3df740 1 -- 172.21.15.44:0/2991156657 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x555a7071e940 con 0x555a70715730 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53817fa640 1 -- 172.21.15.44:0/2991156657 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5394026070 con 0x555a70715730 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53817fa640 1 -- 172.21.15.44:0/2991156657 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f539c036020 con 0x555a70715730 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53ab3df740 10 client.8289 fetch_fsmap finished waiting for FSMap version 68 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53ab3df740 10 client.8289 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53ab3df740 20 client.8289 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53ab3df740 1 --2- 172.21.15.44:0/2991156657 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x555a70793c40 0x555a70729930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53ab3df740 4 client.8289 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53ab3df740 1 -- 172.21.15.44:0/2991156657 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555a706390c0 con 0x555a70793c40 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53a259c640 1 --2- 172.21.15.44:0/2991156657 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x555a70793c40 0x555a70729930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53817fa640 10 client.8289 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53a259c640 1 --2- 172.21.15.44:0/2991156657 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x555a70793c40 0x555a70729930 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36284b3446225587 server_cookie=869b425cdbd8d9da in_seq=0 out_seq=0 2024-04-02T13:28:04.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53817fa640 1 -- 172.21.15.44:0/2991156657 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555a706390c0 con 0x555a70793c40 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53817fa640 10 client.8289 handle_command_reply: tid=0 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53ab3df740 1 client.8289 shutdown 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53ab3df740 1 -- 172.21.15.44:0/2991156657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a70715730 msgr2=0x555a707293f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.494+0000 7f53ab3df740 1 --2- 172.21.15.44:0/2991156657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a70715730 0x555a707293f0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f539c011580 tx=0x7f539c004ab0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53ab3df740 1 -- 172.21.15.44:0/2991156657 shutdown_connections 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53ab3df740 1 --2- 172.21.15.44:0/2991156657 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x555a70793c40 0x555a70729930 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53ab3df740 1 --2- 172.21.15.44:0/2991156657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a70715730 0x555a707293f0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53ab3df740 1 --2- 172.21.15.44:0/2991156657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555a7070ee50 0x555a7070c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53817fa640 0 client.8289 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53ab3df740 1 --2- 172.21.15.44:0/2991156657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a706f4f70 0x555a7070c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53ab3df740 1 -- 172.21.15.44:0/2991156657 >> 172.21.15.44:0/2991156657 conn(0x555a706cd1a0 msgr2=0x555a70708ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53ab3df740 1 -- 172.21.15.44:0/2991156657 shutdown_connections 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53ab3df740 1 -- 172.21.15.44:0/2991156657 wait complete. 2024-04-02T13:28:04.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53ab3df740 20 client.8289 trim_cache size 0 max 16384 2024-04-02T13:28:04.507 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:28:04.507 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-02T13:28:04.507 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-02T13:28:04.507 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:28:04.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53a359e640 1 -- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f538406c920 msgr2=0x7f538406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53a359e640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f538406c920 0x7f538406ede0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f53a40c40a0 tx=0x7f5394007e80 comp rx=0 tx=0).stop 2024-04-02T13:28:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53a359e640 1 -- 172.21.15.44:0/3086607221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53a4071df0 msgr2=0x7f53a40bddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53a359e640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53a4071df0 0x7f53a40bddb0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f539802bee0 tx=0x7f53980046c0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53a359e640 1 -- 172.21.15.44:0/3086607221 shutdown_connections 2024-04-02T13:28:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53a359e640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f538406c920 0x7f538406ede0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53a359e640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53a4155380 0x7f53a40b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53a359e640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53a4071df0 0x7f53a40bddb0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53a359e640 1 --2- 172.21.15.44:0/3086607221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53a406e8d0 0x7f53a41f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53a359e640 1 -- 172.21.15.44:0/3086607221 >> 172.21.15.44:0/3086607221 conn(0x7f53a40cfa90 msgr2=0x7f53a40cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53a359e640 1 -- 172.21.15.44:0/3086607221 shutdown_connections 2024-04-02T13:28:04.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.495+0000 7f53a359e640 1 -- 172.21.15.44:0/3086607221 wait complete. 2024-04-02T13:28:04.510 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T13:28:04.510 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:28:04.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.608+0000 7fe0f3275640 1 Processor -- start 2024-04-02T13:28:04.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.608+0000 7fe0f3275640 1 -- start start 2024-04-02T13:28:04.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.609+0000 7fe0f3275640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe0ec15c050 0x7fe0ec0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.609+0000 7fe0f3275640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe0ec15c430 0x7fe0ec0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.609+0000 7fe0f3275640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0ec0ccb10 con 0x7fe0ec0b17c0 2024-04-02T13:28:04.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.609+0000 7fe0f3275640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe0ec0b2480 con 0x7fe0ec15c050 2024-04-02T13:28:04.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.609+0000 7fe0f3275640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe0ec0b2760 con 0x7fe0ec15c430 2024-04-02T13:28:04.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.609+0000 7fe0f1011640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe0ec15c050 0x7fe0ec0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.609+0000 7fe0f1812640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe0ec0b17c0 0x7fe0ec0b1c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42602/0 (socket says 172.21.15.44:42602) 2024-04-02T13:28:04.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.609+0000 7fe0f1812640 1 -- 172.21.15.44:0/2533606810 learned_addr learned my addr 172.21.15.44:0/2533606810 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:04.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.609+0000 7fe0f0810640 1 --2- 172.21.15.44:0/2533606810 >> v2:172.21.15.69:3300/0 conn(0x7fe0ec15c430 0x7fe0ec0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.609+0000 7fe0f1011640 1 -- 172.21.15.44:0/2533606810 >> v2:172.21.15.69:3300/0 conn(0x7fe0ec15c430 msgr2=0x7fe0ec0ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.609+0000 7fe0f1011640 1 --2- 172.21.15.44:0/2533606810 >> v2:172.21.15.69:3300/0 conn(0x7fe0ec15c430 0x7fe0ec0ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.609+0000 7fe0f1011640 1 -- 172.21.15.44:0/2533606810 >> v1:172.21.15.44:6789/0 conn(0x7fe0ec0b17c0 legacy=0x7fe0ec0b1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.609+0000 7fe0f1011640 1 -- 172.21.15.44:0/2533606810 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0ec0b2a40 con 0x7fe0ec15c050 2024-04-02T13:28:04.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.610+0000 7fe0f1011640 1 --2- 172.21.15.44:0/2533606810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0ec15c050 0x7fe0ec0ce850 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0b4a20 tx=0x7fe0e402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3a6abb2382e569 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.610+0000 7fe0ebfff640 1 -- 172.21.15.44:0/2533606810 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4034070 con 0x7fe0ec15c050 2024-04-02T13:28:04.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.610+0000 7fe0ebfff640 1 -- 172.21.15.44:0/2533606810 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0e4030610 con 0x7fe0ec15c050 2024-04-02T13:28:04.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.610+0000 7fe0ebfff640 1 -- 172.21.15.44:0/2533606810 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e40309e0 con 0x7fe0ec15c050 2024-04-02T13:28:04.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.611+0000 7fe0f3275640 1 -- 172.21.15.44:0/2533606810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0ec15c050 msgr2=0x7fe0ec0ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.611+0000 7fe0f3275640 1 --2- 172.21.15.44:0/2533606810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0ec15c050 0x7fe0ec0ce850 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0b4a20 tx=0x7fe0e402b960 comp rx=0 tx=0).stop 2024-04-02T13:28:04.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.611+0000 7fe0f3275640 1 -- 172.21.15.44:0/2533606810 shutdown_connections 2024-04-02T13:28:04.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.611+0000 7fe0f3275640 1 --2- 172.21.15.44:0/2533606810 >> v2:172.21.15.69:3300/0 conn(0x7fe0ec15c430 0x7fe0ec0ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.611+0000 7fe0f3275640 1 --2- 172.21.15.44:0/2533606810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0ec15c050 0x7fe0ec0ce850 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.611+0000 7fe0f3275640 1 -- 172.21.15.44:0/2533606810 >> 172.21.15.44:0/2533606810 conn(0x7fe0ec09aa20 msgr2=0x7fe0ec09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:04.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.611+0000 7fe0f3275640 1 -- 172.21.15.44:0/2533606810 shutdown_connections 2024-04-02T13:28:04.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.611+0000 7fe0f3275640 1 -- 172.21.15.44:0/2533606810 wait complete. 2024-04-02T13:28:04.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.612+0000 7fe0f3275640 1 Processor -- start 2024-04-02T13:28:04.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.612+0000 7fe0f3275640 1 -- start start 2024-04-02T13:28:04.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f3275640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ec0b17c0 0x7fe0ec06c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f3275640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ec15c050 0x7fe0ec151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f3275640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0ec15c430 0x7fe0ec157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f3275640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ec0cd140 con 0x7fe0ec15c430 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f3275640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ec0ccfc0 con 0x7fe0ec0b17c0 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f3275640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe0ec0cd2c0 con 0x7fe0ec15c050 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f1812640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0ec15c430 0x7fe0ec157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f1812640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0ec15c430 0x7fe0ec157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48072/0 (socket says 172.21.15.44:48072) 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f1812640 1 -- 172.21.15.44:0/970551798 learned_addr learned my addr 172.21.15.44:0/970551798 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f1011640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ec0b17c0 0x7fe0ec06c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f0810640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ec15c050 0x7fe0ec151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f1812640 1 -- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ec15c050 msgr2=0x7fe0ec151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f1812640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ec15c050 0x7fe0ec151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f1812640 1 -- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ec0b17c0 msgr2=0x7fe0ec06c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f1812640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ec0b17c0 0x7fe0ec06c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.613+0000 7fe0f1812640 1 -- 172.21.15.44:0/970551798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0ec06cb20 con 0x7fe0ec15c430 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.614+0000 7fe0f1812640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0ec15c430 0x7fe0ec157f60 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00cbf0 tx=0x7fe0dc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.614+0000 7fe0e9ffb640 1 -- 172.21.15.44:0/970551798 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc017070 con 0x7fe0ec15c430 2024-04-02T13:28:04.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.614+0000 7fe0e9ffb640 1 -- 172.21.15.44:0/970551798 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0dc00ad40 con 0x7fe0ec15c430 2024-04-02T13:28:04.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.614+0000 7fe0f3275640 1 -- 172.21.15.44:0/970551798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0ec06cdf0 con 0x7fe0ec15c430 2024-04-02T13:28:04.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.614+0000 7fe0e9ffb640 1 -- 172.21.15.44:0/970551798 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc012600 con 0x7fe0ec15c430 2024-04-02T13:28:04.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.614+0000 7fe0f3275640 1 -- 172.21.15.44:0/970551798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0ec06d3b0 con 0x7fe0ec15c430 2024-04-02T13:28:04.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.615+0000 7fe0f3275640 1 -- 172.21.15.44:0/970551798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0ec0684f0 con 0x7fe0ec15c430 2024-04-02T13:28:04.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.615+0000 7fe0e9ffb640 1 -- 172.21.15.44:0/970551798 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe0dc01a070 con 0x7fe0ec15c430 2024-04-02T13:28:04.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.615+0000 7fe0e9ffb640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe0d806c8f0 0x7fe0d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.616+0000 7fe0e9ffb640 1 -- 172.21.15.44:0/970551798 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7fe0dc08b970 con 0x7fe0ec15c430 2024-04-02T13:28:04.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.616+0000 7fe0f1011640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe0d806c8f0 0x7fe0d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.616+0000 7fe0f1011640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe0d806c8f0 0x7fe0d806edb0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4036000 tx=0x7fe0e403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.620+0000 7fe0e9ffb640 1 -- 172.21.15.44:0/970551798 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe0dc059590 con 0x7fe0ec15c430 2024-04-02T13:28:04.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.846+0000 7fe0f3275640 1 -- 172.21.15.44:0/970551798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0ec06d5f0 con 0x7fe0ec15c430 2024-04-02T13:28:04.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.846+0000 7fe0e9ffb640 1 -- 172.21.15.44:0/970551798 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5181 (secure 0 0 0) 0x7fe0dc05d890 con 0x7fe0ec15c430 2024-04-02T13:28:04.849 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:04.849 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":68,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:27:06.599726+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8073":{"gid":8073,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":6,"addr":"172.21.15.44:6833/2250279975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2250279975},{"type":"v1","addr":"172.21.15.44:6833","nonce":2250279975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8073,"qdb_cluster":[8073]},"id":6}]} 2024-04-02T13:28:04.850 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 68 2024-04-02T13:28:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.848+0000 7fe0f3275640 1 -- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe0d806c8f0 msgr2=0x7fe0d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.848+0000 7fe0f3275640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe0d806c8f0 0x7fe0d806edb0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4036000 tx=0x7fe0e403b040 comp rx=0 tx=0).stop 2024-04-02T13:28:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.848+0000 7fe0f3275640 1 -- 172.21.15.44:0/970551798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0ec15c430 msgr2=0x7fe0ec157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.848+0000 7fe0f3275640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0ec15c430 0x7fe0ec157f60 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00cbf0 tx=0x7fe0dc00a430 comp rx=0 tx=0).stop 2024-04-02T13:28:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.849+0000 7fe0f3275640 1 -- 172.21.15.44:0/970551798 shutdown_connections 2024-04-02T13:28:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.849+0000 7fe0f3275640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe0d806c8f0 0x7fe0d806edb0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.849+0000 7fe0f3275640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0ec15c430 0x7fe0ec157f60 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.849+0000 7fe0f3275640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ec15c050 0x7fe0ec151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.849+0000 7fe0f3275640 1 --2- 172.21.15.44:0/970551798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ec0b17c0 0x7fe0ec06c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.849+0000 7fe0f3275640 1 -- 172.21.15.44:0/970551798 >> 172.21.15.44:0/970551798 conn(0x7fe0ec09aa20 msgr2=0x7fe0ec0b4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.849+0000 7fe0f3275640 1 -- 172.21.15.44:0/970551798 shutdown_connections 2024-04-02T13:28:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.849+0000 7fe0f3275640 1 -- 172.21.15.44:0/970551798 wait complete. 2024-04-02T13:28:04.862 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-04-02T13:28:04.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.962+0000 7f9902e60640 1 Processor -- start 2024-04-02T13:28:04.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.962+0000 7f9902e60640 1 -- start start 2024-04-02T13:28:04.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.963+0000 7f9902e60640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f98fc0d09a0 0x7f98fc0d0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.963+0000 7f9902e60640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98fc162d90 con 0x7f98fc0d4850 2024-04-02T13:28:04.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.963+0000 7f9902e60640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98fc0cee00 con 0x7f98fc0d4cc0 2024-04-02T13:28:04.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.963+0000 7f9902e60640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f98fc0cfcb0 con 0x7f98fc0d09a0 2024-04-02T13:28:04.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.963+0000 7f9900bfc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f98fc0d09a0 0x7f98fc0d0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.963+0000 7f98fbfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f98fc0d4850 0x7f98fc161f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42610/0 (socket says 172.21.15.44:42610) 2024-04-02T13:28:04.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.963+0000 7f98fbfff640 1 -- 172.21.15.44:0/381176341 learned_addr learned my addr 172.21.15.44:0/381176341 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:04.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.964+0000 7f9900bfc640 1 -- 172.21.15.44:0/381176341 >> v1:172.21.15.69:6789/0 conn(0x7f98fc0d4cc0 legacy=0x7f98fc162680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.964+0000 7f9900bfc640 1 -- 172.21.15.44:0/381176341 >> v1:172.21.15.44:6789/0 conn(0x7f98fc0d4850 legacy=0x7f98fc161f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.964+0000 7f9900bfc640 1 -- 172.21.15.44:0/381176341 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98fc0d12c0 con 0x7f98fc0d09a0 2024-04-02T13:28:04.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.964+0000 7f9900bfc640 1 --2- 172.21.15.44:0/381176341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98fc0d09a0 0x7f98fc0d0d80 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f98ec0097a0 tx=0x7f98ec02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb31bad4aefe1fd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.965+0000 7f98fb7fe640 1 -- 172.21.15.44:0/381176341 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec037070 con 0x7f98fc0d09a0 2024-04-02T13:28:04.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.965+0000 7f98fb7fe640 1 -- 172.21.15.44:0/381176341 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98ec030710 con 0x7f98fc0d09a0 2024-04-02T13:28:04.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.965+0000 7f98fb7fe640 1 -- 172.21.15.44:0/381176341 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec030ae0 con 0x7f98fc0d09a0 2024-04-02T13:28:04.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.965+0000 7f9902e60640 1 -- 172.21.15.44:0/381176341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98fc0d09a0 msgr2=0x7f98fc0d0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.965+0000 7f9902e60640 1 --2- 172.21.15.44:0/381176341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98fc0d09a0 0x7f98fc0d0d80 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f98ec0097a0 tx=0x7f98ec02b790 comp rx=0 tx=0).stop 2024-04-02T13:28:04.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.965+0000 7f9902e60640 1 -- 172.21.15.44:0/381176341 shutdown_connections 2024-04-02T13:28:04.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.965+0000 7f9902e60640 1 --2- 172.21.15.44:0/381176341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98fc0d09a0 0x7f98fc0d0d80 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.965+0000 7f9902e60640 1 -- 172.21.15.44:0/381176341 >> 172.21.15.44:0/381176341 conn(0x7f98fc09d470 msgr2=0x7f98fc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:04.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.965+0000 7f9902e60640 1 -- 172.21.15.44:0/381176341 shutdown_connections 2024-04-02T13:28:04.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.965+0000 7f9902e60640 1 -- 172.21.15.44:0/381176341 wait complete. 2024-04-02T13:28:04.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.966+0000 7f9902e60640 1 Processor -- start 2024-04-02T13:28:04.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.966+0000 7f9902e60640 1 -- start start 2024-04-02T13:28:04.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.967+0000 7f9902e60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98fc0d09a0 0x7f98fc0b8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.967+0000 7f9902e60640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98fc0d4850 0x7f98fc1578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.967+0000 7f9902e60640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98fc0d4cc0 0x7f98fc0ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.967+0000 7f9902e60640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc0cf660 con 0x7f98fc0d4850 2024-04-02T13:28:04.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.967+0000 7f9902e60640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc0cf380 con 0x7f98fc0d4cc0 2024-04-02T13:28:04.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.967+0000 7f9902e60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f98fc0cf940 con 0x7f98fc0d09a0 2024-04-02T13:28:04.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.967+0000 7f98fbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98fc0d4850 0x7f98fc1578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.967+0000 7f98fbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98fc0d4850 0x7f98fc1578c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48082/0 (socket says 172.21.15.44:48082) 2024-04-02T13:28:04.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.967+0000 7f98fbfff640 1 -- 172.21.15.44:0/2954839453 learned_addr learned my addr 172.21.15.44:0/2954839453 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:04.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.968+0000 7f99013fd640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98fc0d4cc0 0x7f98fc0ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.968+0000 7f9900bfc640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98fc0d09a0 0x7f98fc0b8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.968+0000 7f98fbfff640 1 -- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98fc0d09a0 msgr2=0x7f98fc0b8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.968+0000 7f98fbfff640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98fc0d09a0 0x7f98fc0b8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.968+0000 7f98fbfff640 1 -- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98fc0d4cc0 msgr2=0x7f98fc0ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.968+0000 7f98fbfff640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98fc0d4cc0 0x7f98fc0ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.968+0000 7f98fbfff640 1 -- 172.21.15.44:0/2954839453 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98fc0ce200 con 0x7f98fc0d4850 2024-04-02T13:28:04.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.968+0000 7f98fbfff640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98fc0d4850 0x7f98fc1578c0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f98f000c800 tx=0x7f98f000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.968+0000 7f98f97fa640 1 -- 172.21.15.44:0/2954839453 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f0018070 con 0x7f98fc0d4850 2024-04-02T13:28:04.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.968+0000 7f98f97fa640 1 -- 172.21.15.44:0/2954839453 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f00072d0 con 0x7f98fc0d4850 2024-04-02T13:28:04.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.968+0000 7f9902e60640 1 -- 172.21.15.44:0/2954839453 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98fc0cd960 con 0x7f98fc0d4850 2024-04-02T13:28:04.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.968+0000 7f98f97fa640 1 -- 172.21.15.44:0/2954839453 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f00047e0 con 0x7f98fc0d4850 2024-04-02T13:28:04.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.968+0000 7f9902e60640 1 -- 172.21.15.44:0/2954839453 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98fc0cdf20 con 0x7f98fc0d4850 2024-04-02T13:28:04.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.970+0000 7f98f97fa640 1 -- 172.21.15.44:0/2954839453 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f98f001b070 con 0x7f98fc0d4850 2024-04-02T13:28:04.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.970+0000 7f98f97fa640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98e806c8f0 0x7f98e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.970+0000 7f98f97fa640 1 -- 172.21.15.44:0/2954839453 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f98f008b2c0 con 0x7f98fc0d4850 2024-04-02T13:28:04.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.970+0000 7f9900bfc640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98e806c8f0 0x7f98e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.971+0000 7f9900bfc640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98e806c8f0 0x7f98e806edb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f98ec034000 tx=0x7f98ec02f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.971+0000 7f9905a48740 1 Processor -- start 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.971+0000 7f9905a48740 1 -- start start 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9905a48740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265044dc0 0x5612650451a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9905a48740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56126504abe0 0x561265048e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9905a48740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5612650924d0 0x561265092950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9905a48740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56126500c5d0 con 0x561265044dc0 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9905a48740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561264fae6b0 con 0x56126504abe0 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9905a48740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x561264f4c890 con 0x5612650924d0 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9900bfc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265044dc0 0x5612650451a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9900bfc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265044dc0 0x5612650451a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48094/0 (socket says 172.21.15.44:48094) 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9900bfc640 1 -- 172.21.15.44:0/4129447487 learned_addr learned my addr 172.21.15.44:0/4129447487 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f99013fd640 1 --2- 172.21.15.44:0/4129447487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5612650924d0 0x561265092950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f98fbfff640 1 --2- 172.21.15.44:0/4129447487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56126504abe0 0x561265048e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9900bfc640 1 -- 172.21.15.44:0/4129447487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5612650924d0 msgr2=0x561265092950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9900bfc640 1 --2- 172.21.15.44:0/4129447487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5612650924d0 0x561265092950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9900bfc640 1 -- 172.21.15.44:0/4129447487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56126504abe0 msgr2=0x561265048e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9900bfc640 1 --2- 172.21.15.44:0/4129447487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56126504abe0 0x561265048e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9900bfc640 1 -- 172.21.15.44:0/4129447487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561264f7b1d0 con 0x561265044dc0 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f99013fd640 1 --2- 172.21.15.44:0/4129447487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5612650924d0 0x561265092950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f9900bfc640 1 --2- 172.21.15.44:0/4129447487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265044dc0 0x5612650451a0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f98ec02bda0 tx=0x7f98ec02d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8df367629a1a47 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.972+0000 7f98e27fc640 1 -- 172.21.15.44:0/4129447487 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec04e070 con 0x561265044dc0 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.973+0000 7f98e27fc640 1 -- 172.21.15.44:0/4129447487 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98ec049470 con 0x561265044dc0 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.973+0000 7f98e27fc640 1 -- 172.21.15.44:0/4129447487 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec037040 con 0x561265044dc0 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.973+0000 7f9905a48740 1 -- 172.21.15.44:0/4129447487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265044dc0 msgr2=0x5612650451a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.973+0000 7f9905a48740 1 --2- 172.21.15.44:0/4129447487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265044dc0 0x5612650451a0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f98ec02bda0 tx=0x7f98ec02d000 comp rx=0 tx=0).stop 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.973+0000 7f9905a48740 1 -- 172.21.15.44:0/4129447487 shutdown_connections 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.973+0000 7f9905a48740 1 --2- 172.21.15.44:0/4129447487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5612650924d0 0x561265092950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.973+0000 7f9905a48740 1 --2- 172.21.15.44:0/4129447487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56126504abe0 0x561265048e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.973+0000 7f9905a48740 1 --2- 172.21.15.44:0/4129447487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265044dc0 0x5612650451a0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.973+0000 7f9905a48740 1 -- 172.21.15.44:0/4129447487 >> 172.21.15.44:0/4129447487 conn(0x561265046e40 msgr2=0x561265048700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:04.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.973+0000 7f9905a48740 1 -- 172.21.15.44:0/4129447487 shutdown_connections 2024-04-02T13:28:04.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.973+0000 7f9905a48740 1 -- 172.21.15.44:0/4129447487 wait complete. 2024-04-02T13:28:04.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.974+0000 7f9905a48740 1 -- start start 2024-04-02T13:28:04.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.974+0000 7f9905a48740 1 Processor -- start 2024-04-02T13:28:04.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f9905a48740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x561265076800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f9905a48740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612650924d0 0x5612650747e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f9905a48740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 0x561265074d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f9905a48740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561264f4a200 con 0x561265070120 2024-04-02T13:28:04.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f9905a48740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561264faa660 con 0x5612650924d0 2024-04-02T13:28:04.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f9905a48740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x561265071530 con 0x56126504abe0 2024-04-02T13:28:04.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f9905a48740 10 client.0 _ll_register_callbacks cb 0x561264f6d040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:04.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f99013fd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 0x561265074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f99013fd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 0x561265074d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48108/0 (socket says 172.21.15.44:48108) 2024-04-02T13:28:04.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f99013fd640 1 -- 172.21.15.44:0/1316124079 learned_addr learned my addr 172.21.15.44:0/1316124079 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:04.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f98fbfff640 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612650924d0 0x5612650747e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f9900bfc640 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x561265076800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f99013fd640 1 -- 172.21.15.44:0/1316124079 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 msgr2=0x561265076800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f99013fd640 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x561265076800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f99013fd640 1 -- 172.21.15.44:0/1316124079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612650924d0 msgr2=0x5612650747e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f99013fd640 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612650924d0 0x5612650747e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f99013fd640 1 -- 172.21.15.44:0/1316124079 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561264d8e7c0 con 0x561265070120 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f9900bfc640 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x561265076800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f99013fd640 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 0x561265074d20 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f98f4005950 tx=0x7f98f4001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d504d05b0de518 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f98db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f98db7fe640 1 -- 172.21.15.44:0/1316124079 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f401b060 con 0x561265070120 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f98db7fe640 1 -- 172.21.15.44:0/1316124079 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f98f4016d60 con 0x561265070120 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.975+0000 7f9905a48740 1 -- 172.21.15.44:0/1316124079 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98ec049470 con 0x561265070120 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f98db7fe640 1 -- 172.21.15.44:0/1316124079 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f98f40124d0 con 0x561265070120 2024-04-02T13:28:04.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f9905a48740 10 client.8313 fetch_fsmap learned FSMap version 68 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f9905a48740 1 -- 172.21.15.44:0/1316124079 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x561264deef50 con 0x561265070120 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f98db7fe640 1 -- 172.21.15.44:0/1316124079 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98ec049470 con 0x561265070120 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f98db7fe640 1 -- 172.21.15.44:0/1316124079 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f98f4012670 con 0x561265070120 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f9905a48740 10 client.8313 fetch_fsmap finished waiting for FSMap version 68 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f9905a48740 10 client.8313 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f9905a48740 20 client.8313 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f9905a48740 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5612650784a0 0x56126506bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f9905a48740 4 client.8313 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f9905a48740 1 -- 172.21.15.44:0/1316124079 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561264fc1ee0 con 0x5612650784a0 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f9900bfc640 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5612650784a0 0x56126506bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f98db7fe640 10 client.8313 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.976+0000 7f9900bfc640 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5612650784a0 0x56126506bcd0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86cdca62c60f3dca server_cookie=c2895ce0063ed0c9 in_seq=0 out_seq=0 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f98db7fe640 1 -- 172.21.15.44:0/1316124079 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561264fc1ee0 con 0x5612650784a0 2024-04-02T13:28:04.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f98db7fe640 10 client.8313 handle_command_reply: tid=0 2024-04-02T13:28:04.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f9905a48740 1 client.8313 shutdown 2024-04-02T13:28:04.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f9905a48740 1 -- 172.21.15.44:0/1316124079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 msgr2=0x561265074d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f9905a48740 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 0x561265074d20 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f98f4005950 tx=0x7f98f4001a30 comp rx=0 tx=0).stop 2024-04-02T13:28:04.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f9905a48740 1 -- 172.21.15.44:0/1316124079 shutdown_connections 2024-04-02T13:28:04.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f9905a48740 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5612650784a0 0x56126506bcd0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f9905a48740 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 0x561265074d20 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f9905a48740 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612650924d0 0x5612650747e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f9905a48740 1 --2- 172.21.15.44:0/1316124079 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x561265076800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f98db7fe640 0 client.8313 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:28:04.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f9905a48740 1 -- 172.21.15.44:0/1316124079 >> 172.21.15.44:0/1316124079 conn(0x561265044dc0 msgr2=0x561265070ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:04.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f9905a48740 1 -- 172.21.15.44:0/1316124079 shutdown_connections 2024-04-02T13:28:04.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f9905a48740 1 -- 172.21.15.44:0/1316124079 wait complete. 2024-04-02T13:28:04.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.977+0000 7f9905a48740 20 client.8313 trim_cache size 0 max 16384 2024-04-02T13:28:04.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.991+0000 7f9905a48740 1 Processor -- start 2024-04-02T13:28:04.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.991+0000 7f9905a48740 1 -- start start 2024-04-02T13:28:04.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.991+0000 7f9905a48740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x561265090f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.991+0000 7f9905a48740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 0x561265070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.991+0000 7f9905a48740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612650924d0 0x561265071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:04.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.991+0000 7f9905a48740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc2002c0 con 0x561265070120 2024-04-02T13:28:04.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.991+0000 7f9905a48740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc0b5520 con 0x5612650924d0 2024-04-02T13:28:04.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.991+0000 7f9905a48740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f98fc1ffcf0 con 0x56126504abe0 2024-04-02T13:28:04.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.991+0000 7f98fbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 0x561265070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f98fbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 0x561265070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48112/0 (socket says 172.21.15.44:48112) 2024-04-02T13:28:04.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f98fbfff640 1 -- 172.21.15.44:0/441056950 learned_addr learned my addr 172.21.15.44:0/441056950 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:04.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f9900bfc640 1 --2- 172.21.15.44:0/441056950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x561265090f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f99013fd640 1 --2- 172.21.15.44:0/441056950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612650924d0 0x561265071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:04.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f98fbfff640 1 -- 172.21.15.44:0/441056950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 msgr2=0x561265090f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f98fbfff640 1 --2- 172.21.15.44:0/441056950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x561265090f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f98fbfff640 1 -- 172.21.15.44:0/441056950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612650924d0 msgr2=0x561265071aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:04.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f98fbfff640 1 --2- 172.21.15.44:0/441056950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612650924d0 0x561265071aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f98fbfff640 1 -- 172.21.15.44:0/441056950 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561264c820b0 con 0x561265070120 2024-04-02T13:28:04.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f9900bfc640 1 --2- 172.21.15.44:0/441056950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x561265090f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f99013fd640 1 --2- 172.21.15.44:0/441056950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612650924d0 0x561265071aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f98fbfff640 1 --2- 172.21.15.44:0/441056950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 0x561265070f50 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f98f00895a0 tx=0x7f98f0058da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce9db09a6b0e535 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f98e1ffb640 1 -- 172.21.15.44:0/441056950 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f0061980 con 0x561265070120 2024-04-02T13:28:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f98e1ffb640 1 -- 172.21.15.44:0/441056950 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f0056e40 con 0x561265070120 2024-04-02T13:28:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.992+0000 7f98e1ffb640 1 -- 172.21.15.44:0/441056950 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f0055410 con 0x561265070120 2024-04-02T13:28:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.993+0000 7f9905a48740 1 -- 172.21.15.44:0/441056950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 msgr2=0x561265070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.993+0000 7f9905a48740 1 --2- 172.21.15.44:0/441056950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 0x561265070f50 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f98f00895a0 tx=0x7f98f0058da0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.993+0000 7f9905a48740 1 -- 172.21.15.44:0/441056950 shutdown_connections 2024-04-02T13:28:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.993+0000 7f9905a48740 1 --2- 172.21.15.44:0/441056950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612650924d0 0x561265071aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.993+0000 7f9905a48740 1 --2- 172.21.15.44:0/441056950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561265070120 0x561265070f50 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.993+0000 7f9905a48740 1 --2- 172.21.15.44:0/441056950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x561265090f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.993+0000 7f9905a48740 1 -- 172.21.15.44:0/441056950 >> 172.21.15.44:0/441056950 conn(0x561265044dc0 msgr2=0x56126503ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.993+0000 7f9905a48740 1 -- 172.21.15.44:0/441056950 shutdown_connections 2024-04-02T13:28:04.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.993+0000 7f9905a48740 1 -- 172.21.15.44:0/441056950 wait complete. 2024-04-02T13:28:04.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.993+0000 7f9905a48740 1 -- start start 2024-04-02T13:28:04.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.993+0000 7f9905a48740 1 Processor -- start 2024-04-02T13:28:05.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.994+0000 7f9905a48740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x5612650485d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.994+0000 7f9905a48740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561265070120 0x561265048b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.994+0000 7f9905a48740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612650924d0 0x5612650a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.994+0000 7f9905a48740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561265078320 con 0x5612650924d0 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.994+0000 7f9905a48740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56126506af90 con 0x561265070120 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.994+0000 7f9905a48740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x561265078970 con 0x56126504abe0 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.994+0000 7f9905a48740 10 client.0 _ll_register_callbacks cb 0x7f98f4003a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.994+0000 7f99013fd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612650924d0 0x5612650a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.994+0000 7f99013fd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612650924d0 0x5612650a20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48122/0 (socket says 172.21.15.44:48122) 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.994+0000 7f99013fd640 1 -- 172.21.15.44:0/547727388 learned_addr learned my addr 172.21.15.44:0/547727388 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.994+0000 7f98fbfff640 1 --2- 172.21.15.44:0/547727388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561265070120 0x561265048b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.994+0000 7f9900bfc640 1 --2- 172.21.15.44:0/547727388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x5612650485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f99013fd640 1 -- 172.21.15.44:0/547727388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 msgr2=0x5612650485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f99013fd640 1 --2- 172.21.15.44:0/547727388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x5612650485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f99013fd640 1 -- 172.21.15.44:0/547727388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561265070120 msgr2=0x561265048b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f99013fd640 1 --2- 172.21.15.44:0/547727388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561265070120 0x561265048b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f99013fd640 1 -- 172.21.15.44:0/547727388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56126504a9e0 con 0x5612650924d0 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f99013fd640 1 --2- 172.21.15.44:0/547727388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612650924d0 0x5612650a20b0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x561265077500 tx=0x7f98f4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f799d4184a03a8b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f98db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f98db7fe640 1 -- 172.21.15.44:0/547727388 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f402f060 con 0x5612650924d0 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f98db7fe640 1 -- 172.21.15.44:0/547727388 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f98f4005510 con 0x5612650924d0 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f9905a48740 1 -- 172.21.15.44:0/547727388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98f0056e40 con 0x5612650924d0 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f98db7fe640 1 -- 172.21.15.44:0/547727388 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f98f4035050 con 0x5612650924d0 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f9905a48740 10 client.8319 fetch_fsmap learned FSMap version 68 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f9905a48740 1 -- 172.21.15.44:0/547727388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x56126506b400 con 0x5612650924d0 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f98db7fe640 1 -- 172.21.15.44:0/547727388 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f0056e40 con 0x5612650924d0 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f98db7fe640 1 -- 172.21.15.44:0/547727388 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f98f4038020 con 0x5612650924d0 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f9905a48740 10 client.8319 fetch_fsmap finished waiting for FSMap version 68 2024-04-02T13:28:05.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f9905a48740 10 client.8319 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f9905a48740 20 client.8319 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f9905a48740 1 --2- 172.21.15.44:0/547727388 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5612650b2990 0x5612650b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.995+0000 7f9905a48740 4 client.8319 mds_command: new command op to 8073 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9905a48740 1 -- 172.21.15.44:0/547727388 --> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x561264f4f4d0 con 0x5612650b2990 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9900bfc640 1 --2- 172.21.15.44:0/547727388 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5612650b2990 0x5612650b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f98db7fe640 10 client.8319 ms_handle_connect on v2:172.21.15.44:6832/2250279975 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9900bfc640 1 --2- 172.21.15.44:0/547727388 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5612650b2990 0x5612650b2d70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d30937fe3854d407 server_cookie=4b8ac6dc59e1a553 in_seq=0 out_seq=0 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f98db7fe640 1 -- 172.21.15.44:0/547727388 <== mds.0 v2:172.21.15.44:6832/2250279975 1 ==== command_reply(tid 0: 0 ) ==== 8+0+144 (crc 0 0 0) 0x561264f4f4d0 con 0x5612650b2990 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f98db7fe640 10 client.8319 handle_command_reply: tid=0 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9905a48740 1 client.8319 shutdown 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9905a48740 1 -- 172.21.15.44:0/547727388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612650924d0 msgr2=0x5612650a20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9905a48740 1 --2- 172.21.15.44:0/547727388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612650924d0 0x5612650a20b0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x561265077500 tx=0x7f98f4013000 comp rx=0 tx=0).stop 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9905a48740 1 -- 172.21.15.44:0/547727388 shutdown_connections 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9905a48740 1 --2- 172.21.15.44:0/547727388 >> [v2:172.21.15.44:6832/2250279975,v1:172.21.15.44:6833/2250279975] conn(0x5612650b2990 0x5612650b2d70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9905a48740 1 --2- 172.21.15.44:0/547727388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612650924d0 0x5612650a20b0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9905a48740 1 --2- 172.21.15.44:0/547727388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561265070120 0x561265048b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f98db7fe640 0 client.8319 ms_handle_reset on v2:172.21.15.44:6832/2250279975 2024-04-02T13:28:05.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9905a48740 1 --2- 172.21.15.44:0/547727388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56126504abe0 0x5612650485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9905a48740 1 -- 172.21.15.44:0/547727388 >> 172.21.15.44:0/547727388 conn(0x561265044dc0 msgr2=0x56126506c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:05.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9905a48740 1 -- 172.21.15.44:0/547727388 shutdown_connections 2024-04-02T13:28:05.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9905a48740 1 -- 172.21.15.44:0/547727388 wait complete. 2024-04-02T13:28:05.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.996+0000 7f9905a48740 20 client.8319 trim_cache size 0 max 16384 2024-04-02T13:28:05.006 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:05.006 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-04-02T13:28:05.006 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:28:05.007 INFO:teuthology.orchestra.run.smithi044.stdout: "damage_type": "backtrace", 2024-04-02T13:28:05.007 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 4251987405, 2024-04-02T13:28:05.007 INFO:teuthology.orchestra.run.smithi044.stdout: "ino": 1099511627777, 2024-04-02T13:28:05.007 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/testdir/hardlink" 2024-04-02T13:28:05.007 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:28:05.007 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-04-02T13:28:05.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.997+0000 7f9902e60640 1 -- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98e806c8f0 msgr2=0x7f98e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.997+0000 7f9902e60640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98e806c8f0 0x7f98e806edb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f98ec034000 tx=0x7f98ec02f000 comp rx=0 tx=0).stop 2024-04-02T13:28:05.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.997+0000 7f9902e60640 1 -- 172.21.15.44:0/2954839453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98fc0d4850 msgr2=0x7f98fc1578c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.997+0000 7f9902e60640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98fc0d4850 0x7f98fc1578c0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f98f000c800 tx=0x7f98f000ccc0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.997+0000 7f9902e60640 1 -- 172.21.15.44:0/2954839453 shutdown_connections 2024-04-02T13:28:05.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.997+0000 7f9902e60640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98e806c8f0 0x7f98e806edb0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.997+0000 7f9902e60640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98fc0d4cc0 0x7f98fc0ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.997+0000 7f9902e60640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98fc0d4850 0x7f98fc1578c0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.997+0000 7f9902e60640 1 --2- 172.21.15.44:0/2954839453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98fc0d09a0 0x7f98fc0b8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.997+0000 7f9902e60640 1 -- 172.21.15.44:0/2954839453 >> 172.21.15.44:0/2954839453 conn(0x7f98fc09d470 msgr2=0x7f98fc15c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:05.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.997+0000 7f9902e60640 1 -- 172.21.15.44:0/2954839453 shutdown_connections 2024-04-02T13:28:05.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:04.997+0000 7f9902e60640 1 -- 172.21.15.44:0/2954839453 wait complete. 2024-04-02T13:28:05.010 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:28:05.010 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:28:05.025 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:28:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.165+0000 7f15157a8640 1 Processor -- start 2024-04-02T13:28:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.165+0000 7f15157a8640 1 -- start start 2024-04-02T13:28:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.166+0000 7f15157a8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f151015e520 0x7f15100d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.166+0000 7f15157a8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15100b70b0 con 0x7f151015e140 2024-04-02T13:28:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.166+0000 7f15157a8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15100b7390 con 0x7f15100d0f80 2024-04-02T13:28:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.166+0000 7f15157a8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f15100b9f50 con 0x7f151015e520 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.166+0000 7f1507fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f151015e520 0x7f15100d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.166+0000 7f150effd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f151015e140 0x7f15100d2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42620/0 (socket says 172.21.15.44:42620) 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.166+0000 7f150effd640 1 -- 172.21.15.44:0/2306691258 learned_addr learned my addr 172.21.15.44:0/2306691258 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.166+0000 7f1507fff640 1 -- 172.21.15.44:0/2306691258 >> v1:172.21.15.69:6789/0 conn(0x7f15100d0f80 legacy=0x7f15100b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.166+0000 7f1507fff640 1 -- 172.21.15.44:0/2306691258 >> v1:172.21.15.44:6789/0 conn(0x7f151015e140 legacy=0x7f15100d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.166+0000 7f1507fff640 1 -- 172.21.15.44:0/2306691258 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15100b7670 con 0x7f151015e520 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.167+0000 7f1507fff640 1 --2- 172.21.15.44:0/2306691258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f151015e520 0x7f15100d0a40 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f14f80097a0 tx=0x7f14f802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a2467858fa683c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.167+0000 7f150e7fc640 1 -- 172.21.15.44:0/2306691258 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f8037070 con 0x7f151015e520 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.167+0000 7f150e7fc640 1 -- 172.21.15.44:0/2306691258 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14f8030710 con 0x7f151015e520 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.167+0000 7f150e7fc640 1 -- 172.21.15.44:0/2306691258 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f8030ae0 con 0x7f151015e520 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.168+0000 7f15157a8640 1 -- 172.21.15.44:0/2306691258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f151015e520 msgr2=0x7f15100d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.168+0000 7f15157a8640 1 --2- 172.21.15.44:0/2306691258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f151015e520 0x7f15100d0a40 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f14f80097a0 tx=0x7f14f802b790 comp rx=0 tx=0).stop 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.168+0000 7f15157a8640 1 -- 172.21.15.44:0/2306691258 shutdown_connections 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.168+0000 7f15157a8640 1 --2- 172.21.15.44:0/2306691258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f151015e520 0x7f15100d0a40 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.168+0000 7f15157a8640 1 -- 172.21.15.44:0/2306691258 >> 172.21.15.44:0/2306691258 conn(0x7f151009d390 msgr2=0x7f15100d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:05.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.168+0000 7f15157a8640 1 -- 172.21.15.44:0/2306691258 shutdown_connections 2024-04-02T13:28:05.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.168+0000 7f15157a8640 1 -- 172.21.15.44:0/2306691258 wait complete. 2024-04-02T13:28:05.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.169+0000 7f15157a8640 1 Processor -- start 2024-04-02T13:28:05.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.169+0000 7f15157a8640 1 -- start start 2024-04-02T13:28:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f15157a8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15100d0f80 0x7f151015c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f15157a8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f151015e140 0x7f15100a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f15157a8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f151015e520 0x7f15100c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f15157a8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f15100d2d40 con 0x7f15100d0f80 2024-04-02T13:28:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f15157a8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f15100d4ee0 con 0x7f151015e520 2024-04-02T13:28:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f15157a8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f15100b8e60 con 0x7f151015e140 2024-04-02T13:28:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f150effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15100d0f80 0x7f151015c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f150effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15100d0f80 0x7f151015c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48146/0 (socket says 172.21.15.44:48146) 2024-04-02T13:28:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f150effd640 1 -- 172.21.15.44:0/1346648354 learned_addr learned my addr 172.21.15.44:0/1346648354 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f150f7fe640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f151015e520 0x7f15100c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f150effd640 1 -- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f151015e140 msgr2=0x7f15100a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f1507fff640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f151015e140 0x7f15100a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f150effd640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f151015e140 0x7f15100a0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f150effd640 1 -- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f151015e520 msgr2=0x7f15100c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f150effd640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f151015e520 0x7f15100c9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f150effd640 1 -- 172.21.15.44:0/1346648354 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15100a1960 con 0x7f15100d0f80 2024-04-02T13:28:05.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f150effd640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15100d0f80 0x7f151015c5c0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f150000c800 tx=0x7f150000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:05.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f15077fe640 1 -- 172.21.15.44:0/1346648354 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1500018070 con 0x7f15100d0f80 2024-04-02T13:28:05.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f15077fe640 1 -- 172.21.15.44:0/1346648354 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15000072d0 con 0x7f15100d0f80 2024-04-02T13:28:05.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f15157a8640 1 -- 172.21.15.44:0/1346648354 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15100a1c40 con 0x7f15100d0f80 2024-04-02T13:28:05.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f15077fe640 1 -- 172.21.15.44:0/1346648354 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15000047e0 con 0x7f15100d0f80 2024-04-02T13:28:05.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.170+0000 7f15157a8640 1 -- 172.21.15.44:0/1346648354 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f151009dfe0 con 0x7f15100d0f80 2024-04-02T13:28:05.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.172+0000 7f15157a8640 1 -- 172.21.15.44:0/1346648354 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15100684f0 con 0x7f15100d0f80 2024-04-02T13:28:05.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.172+0000 7f15077fe640 1 -- 172.21.15.44:0/1346648354 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f150001b070 con 0x7f15100d0f80 2024-04-02T13:28:05.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.172+0000 7f15077fe640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f14f006c8f0 0x7f14f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.172+0000 7f15077fe640 1 -- 172.21.15.44:0/1346648354 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f150008b270 con 0x7f15100d0f80 2024-04-02T13:28:05.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.173+0000 7f1507fff640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f14f006c8f0 0x7f14f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.173+0000 7f1507fff640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f14f006c8f0 0x7f14f006edb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f14f8034000 tx=0x7f14f802f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:05.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.177+0000 7f15077fe640 1 -- 172.21.15.44:0/1346648354 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f15000594e0 con 0x7f15100d0f80 2024-04-02T13:28:05.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.364+0000 7f15157a8640 1 -- 172.21.15.44:0/1346648354 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f151006dd50 con 0x7f15100d0f80 2024-04-02T13:28:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.365+0000 7f15077fe640 1 -- 172.21.15.44:0/1346648354 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+17158 (secure 0 0 0) 0x7f1500058de0 con 0x7f15100d0f80 2024-04-02T13:28:05.368 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:05.368 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":94,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:27:10.627871+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:26:41.300970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-02T13:26:41.979586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.24","mappings":[{"from":1,"to":5}]},{"pgid":"14.2c","mappings":[{"from":1,"to":2}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6833/719680934":"2024-04-03T13:26:55.589090+0000","172.21.15.44:6832/719680934":"2024-04-03T13:26:55.589090+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:28:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.367+0000 7f15157a8640 1 -- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f14f006c8f0 msgr2=0x7f14f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.367+0000 7f15157a8640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f14f006c8f0 0x7f14f006edb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f14f8034000 tx=0x7f14f802f000 comp rx=0 tx=0).stop 2024-04-02T13:28:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.367+0000 7f15157a8640 1 -- 172.21.15.44:0/1346648354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15100d0f80 msgr2=0x7f151015c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.367+0000 7f15157a8640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15100d0f80 0x7f151015c5c0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f150000c800 tx=0x7f150000ccc0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.367+0000 7f15157a8640 1 -- 172.21.15.44:0/1346648354 shutdown_connections 2024-04-02T13:28:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.367+0000 7f15157a8640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f14f006c8f0 0x7f14f006edb0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.367+0000 7f15157a8640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f151015e520 0x7f15100c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.367+0000 7f15157a8640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f151015e140 0x7f15100a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.367+0000 7f15157a8640 1 --2- 172.21.15.44:0/1346648354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15100d0f80 0x7f151015c5c0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.367+0000 7f15157a8640 1 -- 172.21.15.44:0/1346648354 >> 172.21.15.44:0/1346648354 conn(0x7f151009d390 msgr2=0x7f15100d1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:05.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.367+0000 7f15157a8640 1 -- 172.21.15.44:0/1346648354 shutdown_connections 2024-04-02T13:28:05.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.367+0000 7f15157a8640 1 -- 172.21.15.44:0/1346648354 wait complete. 2024-04-02T13:28:05.380 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:26:41.300970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:26:41.979586+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:28:05.381 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:28:05.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.483+0000 7f354aa47640 1 Processor -- start 2024-04-02T13:28:05.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.483+0000 7f354aa47640 1 -- start start 2024-04-02T13:28:05.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f354aa47640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f354415e260 0x7f35440d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f354aa47640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f35440cec60 0x7f3544168280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f354aa47640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35440cf120 con 0x7f354415e640 2024-04-02T13:28:05.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f354aa47640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f35440b5a50 con 0x7f35440cec60 2024-04-02T13:28:05.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f354aa47640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f35440d01c0 con 0x7f354415e260 2024-04-02T13:28:05.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f35437fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f354415e640 0x7f35440d0ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42634/0 (socket says 172.21.15.44:42634) 2024-04-02T13:28:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f35437fe640 1 -- 172.21.15.44:0/3221787761 learned_addr learned my addr 172.21.15.44:0/3221787761 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f3548fe4640 1 --2- 172.21.15.44:0/3221787761 >> v2:172.21.15.69:3300/0 conn(0x7f35440cec60 0x7f3544168280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f3543fff640 1 --2- 172.21.15.44:0/3221787761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f354415e260 0x7f35440d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f3542ffd640 1 -- 172.21.15.44:0/3221787761 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 630108611 0 0) 0x7f35440cf120 con 0x7f354415e640 2024-04-02T13:28:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f3542ffd640 1 -- 172.21.15.44:0/3221787761 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3530003520 con 0x7f354415e640 2024-04-02T13:28:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f3542ffd640 1 -- 172.21.15.44:0/3221787761 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 240484915 0 0) 0x7f3530003520 con 0x7f354415e640 2024-04-02T13:28:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f3548fe4640 1 -- 172.21.15.44:0/3221787761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f354415e260 msgr2=0x7f35440d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f3548fe4640 1 --2- 172.21.15.44:0/3221787761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f354415e260 0x7f35440d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f3548fe4640 1 -- 172.21.15.44:0/3221787761 >> v1:172.21.15.44:6789/0 conn(0x7f354415e640 legacy=0x7f35440d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f3548fe4640 1 -- 172.21.15.44:0/3221787761 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35440cf400 con 0x7f35440cec60 2024-04-02T13:28:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.484+0000 7f3543fff640 1 --2- 172.21.15.44:0/3221787761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f354415e260 0x7f35440d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:28:05.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.485+0000 7f3548fe4640 1 --2- 172.21.15.44:0/3221787761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35440cec60 0x7f3544168280 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f353c009d90 tx=0x7f353c02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61ab2f8954d08507 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.485+0000 7f3542ffd640 1 -- 172.21.15.44:0/3221787761 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c039070 con 0x7f35440cec60 2024-04-02T13:28:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.485+0000 7f3542ffd640 1 -- 172.21.15.44:0/3221787761 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f353c02c8a0 con 0x7f35440cec60 2024-04-02T13:28:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.485+0000 7f3542ffd640 1 -- 172.21.15.44:0/3221787761 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534001670 con 0x7f35440cec60 2024-04-02T13:28:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.486+0000 7f354aa47640 1 -- 172.21.15.44:0/3221787761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35440cec60 msgr2=0x7f3544168280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.486+0000 7f354aa47640 1 --2- 172.21.15.44:0/3221787761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35440cec60 0x7f3544168280 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f353c009d90 tx=0x7f353c02bd70 comp rx=0 tx=0).stop 2024-04-02T13:28:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.486+0000 7f354aa47640 1 -- 172.21.15.44:0/3221787761 shutdown_connections 2024-04-02T13:28:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.486+0000 7f354aa47640 1 --2- 172.21.15.44:0/3221787761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35440cec60 0x7f3544168280 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.486+0000 7f354aa47640 1 --2- 172.21.15.44:0/3221787761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f354415e260 0x7f35440d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.486+0000 7f354aa47640 1 -- 172.21.15.44:0/3221787761 >> 172.21.15.44:0/3221787761 conn(0x7f354409d3c0 msgr2=0x7f35440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.486+0000 7f354aa47640 1 -- 172.21.15.44:0/3221787761 shutdown_connections 2024-04-02T13:28:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.486+0000 7f354aa47640 1 -- 172.21.15.44:0/3221787761 wait complete. 2024-04-02T13:28:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.487+0000 7f354aa47640 1 Processor -- start 2024-04-02T13:28:05.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.487+0000 7f354aa47640 1 -- start start 2024-04-02T13:28:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.488+0000 7f354aa47640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35440cec60 0x7f35440cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.488+0000 7f354aa47640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f354415e260 0x7f35441f82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.488+0000 7f354aa47640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f354415e640 0x7f354415ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.488+0000 7f354aa47640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f35440cfca0 con 0x7f354415e640 2024-04-02T13:28:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.488+0000 7f354aa47640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f35440cf9c0 con 0x7f354415e260 2024-04-02T13:28:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.488+0000 7f354aa47640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f35440d11e0 con 0x7f35440cec60 2024-04-02T13:28:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.488+0000 7f3548fe4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f354415e640 0x7f354415ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.488+0000 7f3548fe4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f354415e640 0x7f354415ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48156/0 (socket says 172.21.15.44:48156) 2024-04-02T13:28:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.488+0000 7f3548fe4640 1 -- 172.21.15.44:0/927782327 learned_addr learned my addr 172.21.15.44:0/927782327 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.488+0000 7f3543fff640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35440cec60 0x7f35440cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.488+0000 7f35437fe640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f354415e260 0x7f35441f82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.489+0000 7f3548fe4640 1 -- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35440cec60 msgr2=0x7f35440cb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.489+0000 7f3548fe4640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35440cec60 0x7f35440cb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.489+0000 7f3548fe4640 1 -- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f354415e260 msgr2=0x7f35441f82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.489+0000 7f3548fe4640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f354415e260 0x7f35441f82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.489+0000 7f3548fe4640 1 -- 172.21.15.44:0/927782327 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35440cbcb0 con 0x7f354415e640 2024-04-02T13:28:05.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.489+0000 7f3543fff640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35440cec60 0x7f35440cb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:05.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.489+0000 7f3548fe4640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f354415e640 0x7f354415ad00 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f353c009d90 tx=0x7f353c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:05.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.489+0000 7f3540ff9640 1 -- 172.21.15.44:0/927782327 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c039050 con 0x7f354415e640 2024-04-02T13:28:05.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.489+0000 7f3540ff9640 1 -- 172.21.15.44:0/927782327 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f353c045070 con 0x7f354415e640 2024-04-02T13:28:05.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.489+0000 7f3540ff9640 1 -- 172.21.15.44:0/927782327 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c009450 con 0x7f354415e640 2024-04-02T13:28:05.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.489+0000 7f354aa47640 1 -- 172.21.15.44:0/927782327 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35440cbf90 con 0x7f354415e640 2024-04-02T13:28:05.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.489+0000 7f354aa47640 1 -- 172.21.15.44:0/927782327 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f354415faf0 con 0x7f354415e640 2024-04-02T13:28:05.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.490+0000 7f354aa47640 1 -- 172.21.15.44:0/927782327 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35440684f0 con 0x7f354415e640 2024-04-02T13:28:05.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.490+0000 7f3540ff9640 1 -- 172.21.15.44:0/927782327 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f353c040690 con 0x7f354415e640 2024-04-02T13:28:05.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.491+0000 7f3540ff9640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f352406c8f0 0x7f352406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.491+0000 7f3540ff9640 1 -- 172.21.15.44:0/927782327 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f353c0ae570 con 0x7f354415e640 2024-04-02T13:28:05.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.491+0000 7f3543fff640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f352406c8f0 0x7f352406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.491+0000 7f3543fff640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f352406c8f0 0x7f352406edb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f352c009ba0 tx=0x7f352c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.495+0000 7f3540ff9640 1 -- 172.21.15.44:0/927782327 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f353c07c7e0 con 0x7f354415e640 2024-04-02T13:28:05.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.677+0000 7f354aa47640 1 -- 172.21.15.44:0/927782327 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f35440b5230 con 0x7f354415e640 2024-04-02T13:28:05.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.677+0000 7f3540ff9640 1 -- 172.21.15.44:0/927782327 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f353c07c0e0 con 0x7f354415e640 2024-04-02T13:28:05.680 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:28:05.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.679+0000 7f354aa47640 1 -- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f352406c8f0 msgr2=0x7f352406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.679+0000 7f354aa47640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f352406c8f0 0x7f352406edb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f352c009ba0 tx=0x7f352c008040 comp rx=0 tx=0).stop 2024-04-02T13:28:05.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.679+0000 7f354aa47640 1 -- 172.21.15.44:0/927782327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f354415e640 msgr2=0x7f354415ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.679+0000 7f354aa47640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f354415e640 0x7f354415ad00 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f353c009d90 tx=0x7f353c004a40 comp rx=0 tx=0).stop 2024-04-02T13:28:05.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.679+0000 7f354aa47640 1 -- 172.21.15.44:0/927782327 shutdown_connections 2024-04-02T13:28:05.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.679+0000 7f354aa47640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f352406c8f0 0x7f352406edb0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.679+0000 7f354aa47640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f354415e640 0x7f354415ad00 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.679+0000 7f354aa47640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f354415e260 0x7f35441f82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.679+0000 7f354aa47640 1 --2- 172.21.15.44:0/927782327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35440cec60 0x7f35440cb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.679+0000 7f354aa47640 1 -- 172.21.15.44:0/927782327 >> 172.21.15.44:0/927782327 conn(0x7f354409d3c0 msgr2=0x7f354415d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.679+0000 7f354aa47640 1 -- 172.21.15.44:0/927782327 shutdown_connections 2024-04-02T13:28:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.679+0000 7f354aa47640 1 -- 172.21.15.44:0/927782327 wait complete. 2024-04-02T13:28:05.693 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:28:05.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.795+0000 7f4dae806640 1 Processor -- start 2024-04-02T13:28:05.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.795+0000 7f4dae806640 1 -- start start 2024-04-02T13:28:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4dae806640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4da80b4050 0x7f4da80ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4dae806640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4da80cd030 0x7f4da80c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4dae806640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4da80cd4a0 con 0x7f4da815c260 2024-04-02T13:28:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4dae806640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4da80cd760 con 0x7f4da80cd030 2024-04-02T13:28:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4dae806640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4da80cda40 con 0x7f4da80b4050 2024-04-02T13:28:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4dacda3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4da80cd030 0x7f4da80c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4da7fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4da815c260 0x7f4da80b3940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42644/0 (socket says 172.21.15.44:42644) 2024-04-02T13:28:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4da7fff640 1 -- 172.21.15.44:0/2430853770 learned_addr learned my addr 172.21.15.44:0/2430853770 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4da77fe640 1 --2- 172.21.15.44:0/2430853770 >> v2:172.21.15.69:3300/0 conn(0x7f4da80b4050 0x7f4da80ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4dacda3640 1 -- 172.21.15.44:0/2430853770 >> v2:172.21.15.69:3300/0 conn(0x7f4da80b4050 msgr2=0x7f4da80ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4dacda3640 1 --2- 172.21.15.44:0/2430853770 >> v2:172.21.15.69:3300/0 conn(0x7f4da80b4050 0x7f4da80ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4dacda3640 1 -- 172.21.15.44:0/2430853770 >> v1:172.21.15.44:6789/0 conn(0x7f4da815c260 legacy=0x7f4da80b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4dacda3640 1 -- 172.21.15.44:0/2430853770 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da80cdd20 con 0x7f4da80cd030 2024-04-02T13:28:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4da6ffd640 1 -- 172.21.15.44:0/2430853770 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2341022551 0 0) 0x7f4da80cd4a0 con 0x7f4da815c260 2024-04-02T13:28:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.796+0000 7f4da77fe640 1 --2- 172.21.15.44:0/2430853770 >> v2:172.21.15.69:3300/0 conn(0x7f4da80b4050 0x7f4da80ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.797+0000 7f4dacda3640 1 --2- 172.21.15.44:0/2430853770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4da80cd030 0x7f4da80c5b30 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f4da809b6a0 tx=0x7f4da002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e8fe36af711008 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.797+0000 7f4da6ffd640 1 -- 172.21.15.44:0/2430853770 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da0039070 con 0x7f4da80cd030 2024-04-02T13:28:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.797+0000 7f4da6ffd640 1 -- 172.21.15.44:0/2430853770 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4da002cc80 con 0x7f4da80cd030 2024-04-02T13:28:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.797+0000 7f4da6ffd640 1 -- 172.21.15.44:0/2430853770 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da0034660 con 0x7f4da80cd030 2024-04-02T13:28:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.797+0000 7f4dae806640 1 -- 172.21.15.44:0/2430853770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4da80cd030 msgr2=0x7f4da80c5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.797+0000 7f4dae806640 1 --2- 172.21.15.44:0/2430853770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4da80cd030 0x7f4da80c5b30 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f4da809b6a0 tx=0x7f4da002bed0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.798+0000 7f4dae806640 1 -- 172.21.15.44:0/2430853770 shutdown_connections 2024-04-02T13:28:05.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.798+0000 7f4dae806640 1 --2- 172.21.15.44:0/2430853770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4da80cd030 0x7f4da80c5b30 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.798+0000 7f4dae806640 1 --2- 172.21.15.44:0/2430853770 >> v2:172.21.15.69:3300/0 conn(0x7f4da80b4050 0x7f4da80ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.798+0000 7f4dae806640 1 -- 172.21.15.44:0/2430853770 >> 172.21.15.44:0/2430853770 conn(0x7f4da809aa20 msgr2=0x7f4da809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:05.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.798+0000 7f4dae806640 1 -- 172.21.15.44:0/2430853770 shutdown_connections 2024-04-02T13:28:05.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.798+0000 7f4dae806640 1 -- 172.21.15.44:0/2430853770 wait complete. 2024-04-02T13:28:05.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.799+0000 7f4dae806640 1 Processor -- start 2024-04-02T13:28:05.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.799+0000 7f4dae806640 1 -- start start 2024-04-02T13:28:05.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dae806640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4da80b4050 0x7f4da80b4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dae806640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4da80cd030 0x7f4da81442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dae806640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4da815c260 0x7f4da8144800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dae806640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4da80c6070 con 0x7f4da815c260 2024-04-02T13:28:05.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dae806640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4da80cfe60 con 0x7f4da80b4050 2024-04-02T13:28:05.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dae806640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4da80c61f0 con 0x7f4da80cd030 2024-04-02T13:28:05.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dacda3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4da815c260 0x7f4da8144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dacda3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4da815c260 0x7f4da8144800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48170/0 (socket says 172.21.15.44:48170) 2024-04-02T13:28:05.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dacda3640 1 -- 172.21.15.44:0/1767476184 learned_addr learned my addr 172.21.15.44:0/1767476184 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4da7fff640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4da80b4050 0x7f4da80b4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4da77fe640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4da80cd030 0x7f4da81442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dacda3640 1 -- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4da80cd030 msgr2=0x7f4da81442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dacda3640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4da80cd030 0x7f4da81442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dacda3640 1 -- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4da80b4050 msgr2=0x7f4da80b4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dacda3640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4da80b4050 0x7f4da80b4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dacda3640 1 -- 172.21.15.44:0/1767476184 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da80cafe0 con 0x7f4da815c260 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4da77fe640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4da80cd030 0x7f4da81442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dacda3640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4da815c260 0x7f4da8144800 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f4da0009e80 tx=0x7f4da0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4da4ff9640 1 -- 172.21.15.44:0/1767476184 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da0039050 con 0x7f4da815c260 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4da4ff9640 1 -- 172.21.15.44:0/1767476184 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4da0045070 con 0x7f4da815c260 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dae806640 1 -- 172.21.15.44:0/1767476184 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da80cb2c0 con 0x7f4da815c260 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4da4ff9640 1 -- 172.21.15.44:0/1767476184 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da003e830 con 0x7f4da815c260 2024-04-02T13:28:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.800+0000 7f4dae806640 1 -- 172.21.15.44:0/1767476184 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da80cb880 con 0x7f4da815c260 2024-04-02T13:28:05.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.802+0000 7f4dae806640 1 -- 172.21.15.44:0/1767476184 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da80684f0 con 0x7f4da815c260 2024-04-02T13:28:05.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.802+0000 7f4da4ff9640 1 -- 172.21.15.44:0/1767476184 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4da003e9d0 con 0x7f4da815c260 2024-04-02T13:28:05.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.802+0000 7f4da4ff9640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4d9406c8f0 0x7f4d9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:05.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.802+0000 7f4da4ff9640 1 -- 172.21.15.44:0/1767476184 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f4da00aeb00 con 0x7f4da815c260 2024-04-02T13:28:05.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.803+0000 7f4da7fff640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4d9406c8f0 0x7f4d9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:05.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.803+0000 7f4da7fff640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4d9406c8f0 0x7f4d9406edb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4d98000f60 tx=0x7f4d98007c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:05.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.807+0000 7f4da4ff9640 1 -- 172.21.15.44:0/1767476184 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4da007ce90 con 0x7f4da815c260 2024-04-02T13:28:05.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.991+0000 7f4dae806640 1 -- 172.21.15.44:0/1767476184 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4da815b690 con 0x7f4da815c260 2024-04-02T13:28:05.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.991+0000 7f4da4ff9640 1 -- 172.21.15.44:0/1767476184 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f4da007c790 con 0x7f4da815c260 2024-04-02T13:28:05.994 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:28:05.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.993+0000 7f4dae806640 1 -- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4d9406c8f0 msgr2=0x7f4d9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.993+0000 7f4dae806640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4d9406c8f0 0x7f4d9406edb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4d98000f60 tx=0x7f4d98007c00 comp rx=0 tx=0).stop 2024-04-02T13:28:05.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.993+0000 7f4dae806640 1 -- 172.21.15.44:0/1767476184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4da815c260 msgr2=0x7f4da8144800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:05.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.993+0000 7f4dae806640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4da815c260 0x7f4da8144800 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f4da0009e80 tx=0x7f4da0004a40 comp rx=0 tx=0).stop 2024-04-02T13:28:05.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.993+0000 7f4dae806640 1 -- 172.21.15.44:0/1767476184 shutdown_connections 2024-04-02T13:28:05.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.993+0000 7f4dae806640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4d9406c8f0 0x7f4d9406edb0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.993+0000 7f4dae806640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4da815c260 0x7f4da8144800 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.993+0000 7f4dae806640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4da80cd030 0x7f4da81442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.993+0000 7f4dae806640 1 --2- 172.21.15.44:0/1767476184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4da80b4050 0x7f4da80b4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:05.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.993+0000 7f4dae806640 1 -- 172.21.15.44:0/1767476184 >> 172.21.15.44:0/1767476184 conn(0x7f4da809aa20 msgr2=0x7f4da80c6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:05.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.993+0000 7f4dae806640 1 -- 172.21.15.44:0/1767476184 shutdown_connections 2024-04-02T13:28:05.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:05.993+0000 7f4dae806640 1 -- 172.21.15.44:0/1767476184 wait complete. 2024-04-02T13:28:06.006 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:28:06.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.111+0000 7f395e688640 1 Processor -- start 2024-04-02T13:28:06.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.111+0000 7f395e688640 1 -- start start 2024-04-02T13:28:06.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.112+0000 7f395e688640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f395815e1d0 0x7f39580d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.112+0000 7f395e688640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f39580d2dd0 0x7f39580b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.112+0000 7f395e688640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39580d1640 con 0x7f39580cec60 2024-04-02T13:28:06.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.112+0000 7f395e688640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f39580cfe00 con 0x7f39580d2dd0 2024-04-02T13:28:06.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.112+0000 7f395e688640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f39580cf1f0 con 0x7f395815e1d0 2024-04-02T13:28:06.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.112+0000 7f395cc25640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f39580d2dd0 0x7f39580b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.112+0000 7f395cc25640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f39580d2dd0 0x7f39580b4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48176/0 (socket says 172.21.15.44:48176) 2024-04-02T13:28:06.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.112+0000 7f395cc25640 1 -- 172.21.15.44:0/137927301 learned_addr learned my addr 172.21.15.44:0/137927301 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:06.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.113+0000 7f3957fff640 1 --2- 172.21.15.44:0/137927301 >> v2:172.21.15.69:3300/0 conn(0x7f395815e1d0 0x7f39580d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.113+0000 7f395cc25640 1 -- 172.21.15.44:0/137927301 >> v2:172.21.15.69:3300/0 conn(0x7f395815e1d0 msgr2=0x7f39580d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:06.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.113+0000 7f395cc25640 1 --2- 172.21.15.44:0/137927301 >> v2:172.21.15.69:3300/0 conn(0x7f395815e1d0 0x7f39580d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.113+0000 7f395cc25640 1 -- 172.21.15.44:0/137927301 >> v1:172.21.15.69:6789/0 conn(0x7f39580cec60 legacy=0x7f39580d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:06.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.113+0000 7f395cc25640 1 -- 172.21.15.44:0/137927301 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39580cf4d0 con 0x7f39580d2dd0 2024-04-02T13:28:06.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.113+0000 7f395cc25640 1 --2- 172.21.15.44:0/137927301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39580d2dd0 0x7f39580b4ce0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f395808c9c0 tx=0x7f395002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8fdc8fdf7397214 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:06.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.114+0000 7f3956ffd640 1 -- 172.21.15.44:0/137927301 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395002c9f0 con 0x7f39580d2dd0 2024-04-02T13:28:06.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.114+0000 7f3956ffd640 1 -- 172.21.15.44:0/137927301 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f395002cb90 con 0x7f39580d2dd0 2024-04-02T13:28:06.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.114+0000 7f3956ffd640 1 -- 172.21.15.44:0/137927301 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39500343f0 con 0x7f39580d2dd0 2024-04-02T13:28:06.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.114+0000 7f395e688640 1 -- 172.21.15.44:0/137927301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39580d2dd0 msgr2=0x7f39580b4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.114+0000 7f395e688640 1 --2- 172.21.15.44:0/137927301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39580d2dd0 0x7f39580b4ce0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f395808c9c0 tx=0x7f395002be50 comp rx=0 tx=0).stop 2024-04-02T13:28:06.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.114+0000 7f395e688640 1 -- 172.21.15.44:0/137927301 shutdown_connections 2024-04-02T13:28:06.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.114+0000 7f395e688640 1 --2- 172.21.15.44:0/137927301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39580d2dd0 0x7f39580b4ce0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.114+0000 7f395e688640 1 --2- 172.21.15.44:0/137927301 >> v2:172.21.15.69:3300/0 conn(0x7f395815e1d0 0x7f39580d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.114+0000 7f395e688640 1 -- 172.21.15.44:0/137927301 >> 172.21.15.44:0/137927301 conn(0x7f395809d3b0 msgr2=0x7f39580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:06.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.114+0000 7f395e688640 1 -- 172.21.15.44:0/137927301 shutdown_connections 2024-04-02T13:28:06.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.114+0000 7f395e688640 1 -- 172.21.15.44:0/137927301 wait complete. 2024-04-02T13:28:06.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.115+0000 7f395e688640 1 Processor -- start 2024-04-02T13:28:06.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.115+0000 7f395e688640 1 -- start start 2024-04-02T13:28:06.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.116+0000 7f395e688640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39580cec60 0x7f395815a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.116+0000 7f395e688640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39580d2dd0 0x7f39581604c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.116+0000 7f395e688640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f395815e1d0 0x7f395815b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.116+0000 7f395e688640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f395815e5b0 con 0x7f39580d2dd0 2024-04-02T13:28:06.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.116+0000 7f395e688640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f39580cfa90 con 0x7f39580cec60 2024-04-02T13:28:06.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.116+0000 7f395e688640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f395815e730 con 0x7f395815e1d0 2024-04-02T13:28:06.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.116+0000 7f39577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39580d2dd0 0x7f39581604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.116+0000 7f39577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39580d2dd0 0x7f39581604c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48178/0 (socket says 172.21.15.44:48178) 2024-04-02T13:28:06.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.116+0000 7f39577fe640 1 -- 172.21.15.44:0/1956676365 learned_addr learned my addr 172.21.15.44:0/1956676365 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:06.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.116+0000 7f3957fff640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39580cec60 0x7f395815a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.116+0000 7f395cc25640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f395815e1d0 0x7f395815b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.117+0000 7f39577fe640 1 -- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f395815e1d0 msgr2=0x7f395815b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.117+0000 7f39577fe640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f395815e1d0 0x7f395815b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.117+0000 7f39577fe640 1 -- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39580cec60 msgr2=0x7f395815a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.117+0000 7f39577fe640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39580cec60 0x7f395815a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.117+0000 7f39577fe640 1 -- 172.21.15.44:0/1956676365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3958160c10 con 0x7f39580d2dd0 2024-04-02T13:28:06.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.117+0000 7f39577fe640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39580d2dd0 0x7f39581604c0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f394c00b3f0 tx=0x7f394c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:06.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.117+0000 7f3954ff9640 1 -- 172.21.15.44:0/1956676365 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c015020 con 0x7f39580d2dd0 2024-04-02T13:28:06.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.117+0000 7f3954ff9640 1 -- 172.21.15.44:0/1956676365 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f394c011830 con 0x7f39580d2dd0 2024-04-02T13:28:06.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.117+0000 7f395e688640 1 -- 172.21.15.44:0/1956676365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3958160ef0 con 0x7f39580d2dd0 2024-04-02T13:28:06.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.117+0000 7f3954ff9640 1 -- 172.21.15.44:0/1956676365 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c011c20 con 0x7f39580d2dd0 2024-04-02T13:28:06.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.117+0000 7f395e688640 1 -- 172.21.15.44:0/1956676365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f395809f710 con 0x7f39580d2dd0 2024-04-02T13:28:06.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.118+0000 7f395e688640 1 -- 172.21.15.44:0/1956676365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f395809f910 con 0x7f39580d2dd0 2024-04-02T13:28:06.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.119+0000 7f3954ff9640 1 -- 172.21.15.44:0/1956676365 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f394c018020 con 0x7f39580d2dd0 2024-04-02T13:28:06.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.119+0000 7f3954ff9640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f394406c8f0 0x7f394406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.119+0000 7f3954ff9640 1 -- 172.21.15.44:0/1956676365 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f394c08b950 con 0x7f39580d2dd0 2024-04-02T13:28:06.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.119+0000 7f3957fff640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f394406c8f0 0x7f394406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.120+0000 7f3957fff640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f394406c8f0 0x7f394406edb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3958158eb0 tx=0x7f3948008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:06.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.124+0000 7f3954ff9640 1 -- 172.21.15.44:0/1956676365 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f395809f910 con 0x7f39580d2dd0 2024-04-02T13:28:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.308+0000 7f395e688640 1 -- 172.21.15.44:0/1956676365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f39580684f0 con 0x7f39580d2dd0 2024-04-02T13:28:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.309+0000 7f3954ff9640 1 -- 172.21.15.44:0/1956676365 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f394c01f030 con 0x7f39580d2dd0 2024-04-02T13:28:06.311 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:28:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.311+0000 7f395e688640 1 -- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f394406c8f0 msgr2=0x7f394406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.311+0000 7f395e688640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f394406c8f0 0x7f394406edb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3958158eb0 tx=0x7f3948008040 comp rx=0 tx=0).stop 2024-04-02T13:28:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.311+0000 7f395e688640 1 -- 172.21.15.44:0/1956676365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39580d2dd0 msgr2=0x7f39581604c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.311+0000 7f395e688640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39580d2dd0 0x7f39581604c0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f394c00b3f0 tx=0x7f394c00b8b0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.311+0000 7f395e688640 1 -- 172.21.15.44:0/1956676365 shutdown_connections 2024-04-02T13:28:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.311+0000 7f395e688640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f394406c8f0 0x7f394406edb0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.311+0000 7f395e688640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f395815e1d0 0x7f395815b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.311+0000 7f395e688640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39580d2dd0 0x7f39581604c0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.311+0000 7f395e688640 1 --2- 172.21.15.44:0/1956676365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39580cec60 0x7f395815a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.311+0000 7f395e688640 1 -- 172.21.15.44:0/1956676365 >> 172.21.15.44:0/1956676365 conn(0x7f395809d3b0 msgr2=0x7f39580cd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.311+0000 7f395e688640 1 -- 172.21.15.44:0/1956676365 shutdown_connections 2024-04-02T13:28:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.311+0000 7f395e688640 1 -- 172.21.15.44:0/1956676365 wait complete. 2024-04-02T13:28:06.324 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:28:06.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.423+0000 7f757c2d3640 1 Processor -- start 2024-04-02T13:28:06.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.423+0000 7f757c2d3640 1 -- start start 2024-04-02T13:28:06.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.424+0000 7f757c2d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f757406cf90 0x7f75740cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.424+0000 7f757c2d3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f75740cfe30 0x7f75740d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.424+0000 7f757c2d3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f757406c1d0 con 0x7f75740d0a30 2024-04-02T13:28:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.424+0000 7f757c2d3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f757406cb00 con 0x7f75740cfe30 2024-04-02T13:28:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.424+0000 7f757c2d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f757406d3f0 con 0x7f757406cf90 2024-04-02T13:28:06.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.425+0000 7f757a870640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f75740cfe30 0x7f75740d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.425+0000 7f757986e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f757406cf90 0x7f75740cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.425+0000 7f757a06f640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f75740d0a30 0x7f75740d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:33178/0 (socket says 172.21.15.44:33178) 2024-04-02T13:28:06.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.425+0000 7f757a06f640 1 -- 172.21.15.44:0/506956636 learned_addr learned my addr 172.21.15.44:0/506956636 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:06.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.425+0000 7f757a870640 1 -- 172.21.15.44:0/506956636 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f757406cf90 msgr2=0x7f75740cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:06.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.425+0000 7f757a870640 1 --2- 172.21.15.44:0/506956636 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f757406cf90 0x7f75740cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.425+0000 7f757a870640 1 -- 172.21.15.44:0/506956636 >> v1:172.21.15.69:6789/0 conn(0x7f75740d0a30 legacy=0x7f75740d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.425+0000 7f757906d640 1 -- 172.21.15.44:0/506956636 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2758803556 0 0) 0x7f757406c1d0 con 0x7f75740d0a30 2024-04-02T13:28:06.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.425+0000 7f757a870640 1 -- 172.21.15.44:0/506956636 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f757406c4b0 con 0x7f75740cfe30 2024-04-02T13:28:06.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.425+0000 7f757986e640 1 --2- 172.21.15.44:0/506956636 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f757406cf90 0x7f75740cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:28:06.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.426+0000 7f757a870640 1 --2- 172.21.15.44:0/506956636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75740cfe30 0x7f75740d2040 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7564009d90 tx=0x7f756402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d62af745e4fa48f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:06.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.426+0000 7f757906d640 1 -- 172.21.15.44:0/506956636 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7564039070 con 0x7f75740cfe30 2024-04-02T13:28:06.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.426+0000 7f757906d640 1 -- 172.21.15.44:0/506956636 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f756402cbc0 con 0x7f75740cfe30 2024-04-02T13:28:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.426+0000 7f757906d640 1 -- 172.21.15.44:0/506956636 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7564034440 con 0x7f75740cfe30 2024-04-02T13:28:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.427+0000 7f757c2d3640 1 -- 172.21.15.44:0/506956636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75740cfe30 msgr2=0x7f75740d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.427+0000 7f757c2d3640 1 --2- 172.21.15.44:0/506956636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75740cfe30 0x7f75740d2040 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7564009d90 tx=0x7f756402be10 comp rx=0 tx=0).stop 2024-04-02T13:28:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.427+0000 7f757c2d3640 1 -- 172.21.15.44:0/506956636 shutdown_connections 2024-04-02T13:28:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.427+0000 7f757c2d3640 1 --2- 172.21.15.44:0/506956636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75740cfe30 0x7f75740d2040 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.427+0000 7f757c2d3640 1 --2- 172.21.15.44:0/506956636 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f757406cf90 0x7f75740cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.427+0000 7f757c2d3640 1 -- 172.21.15.44:0/506956636 >> 172.21.15.44:0/506956636 conn(0x7f757409d470 msgr2=0x7f75740d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.427+0000 7f757c2d3640 1 -- 172.21.15.44:0/506956636 shutdown_connections 2024-04-02T13:28:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.427+0000 7f757c2d3640 1 -- 172.21.15.44:0/506956636 wait complete. 2024-04-02T13:28:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.428+0000 7f757c2d3640 1 Processor -- start 2024-04-02T13:28:06.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.428+0000 7f757c2d3640 1 -- start start 2024-04-02T13:28:06.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.428+0000 7f757c2d3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f757406cf90 0x7f75740b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757c2d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75740cfe30 0x7f757414e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757c2d3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75740d0a30 0x7f75740ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757c2d3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f757415e870 con 0x7f757406cf90 2024-04-02T13:28:06.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757c2d3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f757415e6f0 con 0x7f75740d0a30 2024-04-02T13:28:06.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757c2d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f75740d4da0 con 0x7f75740cfe30 2024-04-02T13:28:06.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757a06f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f757406cf90 0x7f75740b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757a06f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f757406cf90 0x7f75740b5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48182/0 (socket says 172.21.15.44:48182) 2024-04-02T13:28:06.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757a06f640 1 -- 172.21.15.44:0/3806370834 learned_addr learned my addr 172.21.15.44:0/3806370834 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:06.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757a870640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75740d0a30 0x7f75740ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757986e640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75740cfe30 0x7f757414e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757a06f640 1 -- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75740cfe30 msgr2=0x7f757414e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757a06f640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75740cfe30 0x7f757414e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757a06f640 1 -- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75740d0a30 msgr2=0x7f75740ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757a06f640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75740d0a30 0x7f75740ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757a06f640 1 -- 172.21.15.44:0/3806370834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75740b68a0 con 0x7f757406cf90 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757a06f640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f757406cf90 0x7f75740b5230 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f756c00c500 tx=0x7f756c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f7572ffd640 1 -- 172.21.15.44:0/3806370834 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f756c018070 con 0x7f757406cf90 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f7572ffd640 1 -- 172.21.15.44:0/3806370834 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f756c007580 con 0x7f757406cf90 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757c2d3640 1 -- 172.21.15.44:0/3806370834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75740b4750 con 0x7f757406cf90 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f7572ffd640 1 -- 172.21.15.44:0/3806370834 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f756c004a20 con 0x7f757406cf90 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.429+0000 7f757c2d3640 1 -- 172.21.15.44:0/3806370834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75740b4d10 con 0x7f757406cf90 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.431+0000 7f757c2d3640 1 -- 172.21.15.44:0/3806370834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75740684f0 con 0x7f757406cf90 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.431+0000 7f7572ffd640 1 -- 172.21.15.44:0/3806370834 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f756c01b070 con 0x7f757406cf90 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.431+0000 7f7572ffd640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f756006c8f0 0x7f756006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.431+0000 7f7572ffd640 1 -- 172.21.15.44:0/3806370834 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f756c08afc0 con 0x7f757406cf90 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.432+0000 7f757986e640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f756006c8f0 0x7f756006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.432+0000 7f757986e640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f756006c8f0 0x7f756006edb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7568009560 tx=0x7f7568009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:06.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.436+0000 7f7572ffd640 1 -- 172.21.15.44:0/3806370834 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f756c058be0 con 0x7f757406cf90 2024-04-02T13:28:06.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.632+0000 7f757c2d3640 1 -- 172.21.15.44:0/3806370834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f75740ce440 con 0x7f757406cf90 2024-04-02T13:28:06.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.632+0000 7f7572ffd640 1 -- 172.21.15.44:0/3806370834 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v94) ==== 80+0+120 (secure 0 0 0) 0x7f756c05cee0 con 0x7f757406cf90 2024-04-02T13:28:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6833/719680934 2024-04-03T13:26:55.589090+0000 2024-04-02T13:28:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6832/719680934 2024-04-03T13:26:55.589090+0000 2024-04-02T13:28:06.635 INFO:teuthology.orchestra.run.smithi044.stderr:listed 2 entries 2024-04-02T13:28:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.634+0000 7f757c2d3640 1 -- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f756006c8f0 msgr2=0x7f756006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.634+0000 7f757c2d3640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f756006c8f0 0x7f756006edb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7568009560 tx=0x7f7568009190 comp rx=0 tx=0).stop 2024-04-02T13:28:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.634+0000 7f757c2d3640 1 -- 172.21.15.44:0/3806370834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f757406cf90 msgr2=0x7f75740b5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.634+0000 7f757c2d3640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f757406cf90 0x7f75740b5230 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f756c00c500 tx=0x7f756c00c9c0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.634+0000 7f757c2d3640 1 -- 172.21.15.44:0/3806370834 shutdown_connections 2024-04-02T13:28:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.634+0000 7f757c2d3640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f756006c8f0 0x7f756006edb0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.634+0000 7f757c2d3640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75740d0a30 0x7f75740ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.634+0000 7f757c2d3640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75740cfe30 0x7f757414e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.634+0000 7f757c2d3640 1 --2- 172.21.15.44:0/3806370834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f757406cf90 0x7f75740b5230 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.634+0000 7f757c2d3640 1 -- 172.21.15.44:0/3806370834 >> 172.21.15.44:0/3806370834 conn(0x7f757409d470 msgr2=0x7f75740d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.634+0000 7f757c2d3640 1 -- 172.21.15.44:0/3806370834 shutdown_connections 2024-04-02T13:28:06.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.634+0000 7f757c2d3640 1 -- 172.21.15.44:0/3806370834 wait complete. 2024-04-02T13:28:06.647 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:28:06.647 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:28:06.662 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:28:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.800+0000 7fd5c763c640 1 Processor -- start 2024-04-02T13:28:06.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.801+0000 7fd5c763c640 1 -- start start 2024-04-02T13:28:06.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.801+0000 7fd5c763c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd5c006d550 0x7fd5c00d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.801+0000 7fd5c763c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5c00cc600 con 0x7fd5c015e4a0 2024-04-02T13:28:06.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.801+0000 7fd5c763c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5c00cc8e0 con 0x7fd5c00d0c50 2024-04-02T13:28:06.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.801+0000 7fd5c763c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd5c00af300 con 0x7fd5c006d550 2024-04-02T13:28:06.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.802+0000 7fd5c53d8640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd5c015e4a0 0x7fd5c00aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42656/0 (socket says 172.21.15.44:42656) 2024-04-02T13:28:06.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.802+0000 7fd5c53d8640 1 -- 172.21.15.44:0/3966510868 learned_addr learned my addr 172.21.15.44:0/3966510868 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:06.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.802+0000 7fd5c5bd9640 1 --2- 172.21.15.44:0/3966510868 >> v2:172.21.15.69:3300/0 conn(0x7fd5c006d550 0x7fd5c00d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.802+0000 7fd5b7fff640 1 -- 172.21.15.44:0/3966510868 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3484115849 0 0) 0x7fd5c00cc600 con 0x7fd5c015e4a0 2024-04-02T13:28:06.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.802+0000 7fd5b7fff640 1 -- 172.21.15.44:0/3966510868 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5ac003520 con 0x7fd5c015e4a0 2024-04-02T13:28:06.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.802+0000 7fd5b7fff640 1 -- 172.21.15.44:0/3966510868 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2838618871 0 0) 0x7fd5ac003520 con 0x7fd5c015e4a0 2024-04-02T13:28:06.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.802+0000 7fd5c5bd9640 1 -- 172.21.15.44:0/3966510868 >> v1:172.21.15.69:6789/0 conn(0x7fd5c00d0c50 legacy=0x7fd5c006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.802+0000 7fd5c5bd9640 1 -- 172.21.15.44:0/3966510868 >> v1:172.21.15.44:6789/0 conn(0x7fd5c015e4a0 legacy=0x7fd5c00aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.802+0000 7fd5c5bd9640 1 -- 172.21.15.44:0/3966510868 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c00ccbc0 con 0x7fd5c006d550 2024-04-02T13:28:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.803+0000 7fd5c5bd9640 1 --2- 172.21.15.44:0/3966510868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c006d550 0x7fd5c00d10b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009dc0 tx=0x7fd5b802be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2068b4b8f2bf4b5f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.803+0000 7fd5b7fff640 1 -- 172.21.15.44:0/3966510868 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8039070 con 0x7fd5c006d550 2024-04-02T13:28:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.803+0000 7fd5b7fff640 1 -- 172.21.15.44:0/3966510868 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5b802c930 con 0x7fd5c006d550 2024-04-02T13:28:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.803+0000 7fd5b7fff640 1 -- 172.21.15.44:0/3966510868 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc001670 con 0x7fd5c006d550 2024-04-02T13:28:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.804+0000 7fd5c763c640 1 -- 172.21.15.44:0/3966510868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c006d550 msgr2=0x7fd5c00d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.804+0000 7fd5c763c640 1 --2- 172.21.15.44:0/3966510868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c006d550 0x7fd5c00d10b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009dc0 tx=0x7fd5b802be00 comp rx=0 tx=0).stop 2024-04-02T13:28:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.804+0000 7fd5c763c640 1 -- 172.21.15.44:0/3966510868 shutdown_connections 2024-04-02T13:28:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.804+0000 7fd5c763c640 1 --2- 172.21.15.44:0/3966510868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c006d550 0x7fd5c00d10b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.804+0000 7fd5c763c640 1 -- 172.21.15.44:0/3966510868 >> 172.21.15.44:0/3966510868 conn(0x7fd5c009aa20 msgr2=0x7fd5c009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.804+0000 7fd5c763c640 1 -- 172.21.15.44:0/3966510868 shutdown_connections 2024-04-02T13:28:06.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.804+0000 7fd5c763c640 1 -- 172.21.15.44:0/3966510868 wait complete. 2024-04-02T13:28:06.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.805+0000 7fd5c763c640 1 Processor -- start 2024-04-02T13:28:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.805+0000 7fd5c763c640 1 -- start start 2024-04-02T13:28:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c763c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5c006d550 0x7fd5c00b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c763c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c00d0c50 0x7fd5c0160f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c763c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c015e4a0 0x7fd5c00c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c763c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c00cd460 con 0x7fd5c015e4a0 2024-04-02T13:28:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c763c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c00cd180 con 0x7fd5c00d0c50 2024-04-02T13:28:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c763c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd5c015e880 con 0x7fd5c006d550 2024-04-02T13:28:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c5bd9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c015e4a0 0x7fd5c00c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c5bd9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c015e4a0 0x7fd5c00c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48188/0 (socket says 172.21.15.44:48188) 2024-04-02T13:28:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c5bd9640 1 -- 172.21.15.44:0/152817266 learned_addr learned my addr 172.21.15.44:0/152817266 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c4bd7640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c00d0c50 0x7fd5c0160f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c53d8640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5c006d550 0x7fd5c00b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c5bd9640 1 -- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5c006d550 msgr2=0x7fd5c00b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c5bd9640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5c006d550 0x7fd5c00b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c5bd9640 1 -- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c00d0c50 msgr2=0x7fd5c0160f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:06.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c5bd9640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c00d0c50 0x7fd5c0160f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:06.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c5bd9640 1 -- 172.21.15.44:0/152817266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c00c7550 con 0x7fd5c015e4a0 2024-04-02T13:28:06.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c5bd9640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c015e4a0 0x7fd5c00c7a00 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fd5b802c410 tx=0x7fd5b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:06.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5b5ffb640 1 -- 172.21.15.44:0/152817266 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8044020 con 0x7fd5c015e4a0 2024-04-02T13:28:06.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5b5ffb640 1 -- 172.21.15.44:0/152817266 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5b8034d70 con 0x7fd5c015e4a0 2024-04-02T13:28:06.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5b5ffb640 1 -- 172.21.15.44:0/152817266 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8039040 con 0x7fd5c015e4a0 2024-04-02T13:28:06.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.806+0000 7fd5c763c640 1 -- 172.21.15.44:0/152817266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5c00c5e10 con 0x7fd5c015e4a0 2024-04-02T13:28:06.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.807+0000 7fd5c763c640 1 -- 172.21.15.44:0/152817266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c01f7ad0 con 0x7fd5c015e4a0 2024-04-02T13:28:06.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.808+0000 7fd5c763c640 1 -- 172.21.15.44:0/152817266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5c00684f0 con 0x7fd5c015e4a0 2024-04-02T13:28:06.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.808+0000 7fd5b5ffb640 1 -- 172.21.15.44:0/152817266 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd5b8005d10 con 0x7fd5c015e4a0 2024-04-02T13:28:06.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.808+0000 7fd5b5ffb640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd5a406c8f0 0x7fd5a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:06.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.809+0000 7fd5b5ffb640 1 -- 172.21.15.44:0/152817266 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7fd5b80ae4d0 con 0x7fd5c015e4a0 2024-04-02T13:28:06.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.809+0000 7fd5c53d8640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd5a406c8f0 0x7fd5a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:06.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.809+0000 7fd5c53d8640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd5a406c8f0 0x7fd5a406edb0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc005880 tx=0x7fd5bc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:06.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.813+0000 7fd5b5ffb640 1 -- 172.21.15.44:0/152817266 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd5b804e070 con 0x7fd5c015e4a0 2024-04-02T13:28:07.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:06.999+0000 7fd5c763c640 1 -- 172.21.15.44:0/152817266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5c00b4b50 con 0x7fd5c015e4a0 2024-04-02T13:28:07.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.000+0000 7fd5b5ffb640 1 -- 172.21.15.44:0/152817266 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+17158 (secure 0 0 0) 0x7fd5c00c5e10 con 0x7fd5c015e4a0 2024-04-02T13:28:07.003 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:07.003 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":94,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:27:10.627871+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:26:41.300970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-02T13:26:41.979586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.24","mappings":[{"from":1,"to":5}]},{"pgid":"14.2c","mappings":[{"from":1,"to":2}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6833/719680934":"2024-04-03T13:26:55.589090+0000","172.21.15.44:6832/719680934":"2024-04-03T13:26:55.589090+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:28:07.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.002+0000 7fd5c763c640 1 -- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd5a406c8f0 msgr2=0x7fd5a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.002+0000 7fd5c763c640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd5a406c8f0 0x7fd5a406edb0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc005880 tx=0x7fd5bc009040 comp rx=0 tx=0).stop 2024-04-02T13:28:07.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.002+0000 7fd5c763c640 1 -- 172.21.15.44:0/152817266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c015e4a0 msgr2=0x7fd5c00c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.002+0000 7fd5c763c640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c015e4a0 0x7fd5c00c7a00 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fd5b802c410 tx=0x7fd5b8004a40 comp rx=0 tx=0).stop 2024-04-02T13:28:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.002+0000 7fd5c763c640 1 -- 172.21.15.44:0/152817266 shutdown_connections 2024-04-02T13:28:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.002+0000 7fd5c763c640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd5a406c8f0 0x7fd5a406edb0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.002+0000 7fd5c763c640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c015e4a0 0x7fd5c00c7a00 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.002+0000 7fd5c763c640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c00d0c50 0x7fd5c0160f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.002+0000 7fd5c763c640 1 --2- 172.21.15.44:0/152817266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5c006d550 0x7fd5c00b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.002+0000 7fd5c763c640 1 -- 172.21.15.44:0/152817266 >> 172.21.15.44:0/152817266 conn(0x7fd5c009aa20 msgr2=0x7fd5c00cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.002+0000 7fd5c763c640 1 -- 172.21.15.44:0/152817266 shutdown_connections 2024-04-02T13:28:07.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.002+0000 7fd5c763c640 1 -- 172.21.15.44:0/152817266 wait complete. 2024-04-02T13:28:07.015 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:26:41.300970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:26:41.979586+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:28:07.016 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:28:07.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.118+0000 7f524b60d640 1 Processor -- start 2024-04-02T13:28:07.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.118+0000 7f524b60d640 1 -- start start 2024-04-02T13:28:07.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.119+0000 7f524b60d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f524406b4d0 0x7f524406b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.119+0000 7f524b60d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52440ceec0 0x7f52440cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.119+0000 7f524b60d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f524406bdf0 con 0x7f52440cd770 2024-04-02T13:28:07.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.119+0000 7f524b60d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f524406f3f0 con 0x7f524406b4d0 2024-04-02T13:28:07.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.119+0000 7f524b60d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f524406f6d0 con 0x7f52440ceec0 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.119+0000 7f5248ba8640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f52440cd770 0x7f524406eba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42666/0 (socket says 172.21.15.44:42666) 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.119+0000 7f5248ba8640 1 -- 172.21.15.44:0/863957494 learned_addr learned my addr 172.21.15.44:0/863957494 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.119+0000 7f52493a9640 1 --2- 172.21.15.44:0/863957494 >> v2:172.21.15.69:3300/0 conn(0x7f524406b4d0 0x7f524406b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.119+0000 7f5249baa640 1 --2- 172.21.15.44:0/863957494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52440ceec0 0x7f52440cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.119+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282490858 0 0) 0x7f524406bdf0 con 0x7f52440cd770 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f522c003520 con 0x7f52440cd770 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3982376008 0 0) 0x7f522c003520 con 0x7f52440cd770 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52440ceec0 msgr2=0x7f52440cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 --2- 172.21.15.44:0/863957494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52440ceec0 0x7f52440cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 >> v2:172.21.15.69:3300/0 conn(0x7f524406b4d0 msgr2=0x7f524406b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 --2- 172.21.15.44:0/863957494 >> v2:172.21.15.69:3300/0 conn(0x7f524406b4d0 0x7f524406b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524406f9b0 con 0x7f52440cd770 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f5238001670 con 0x7f52440cd770 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 >> v1:172.21.15.44:6789/0 conn(0x7f52440cd770 legacy=0x7f524406eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 --2- 172.21.15.44:0/863957494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52440ceec0 0x7f522c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 --2- 172.21.15.44:0/863957494 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f522c003f30 0x7f522c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 --2- 172.21.15.44:0/863957494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f522c004d40 0x7f522c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f524406f3f0 con 0x7f522c003f30 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f524406f6d0 con 0x7f522c004d40 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f522c00b9c0 con 0x7f52440ceec0 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5248ba8640 1 --2- 172.21.15.44:0/863957494 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f522c003f30 0x7f522c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5249baa640 1 --2- 172.21.15.44:0/863957494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f522c004d40 0x7f522c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f52493a9640 1 --2- 172.21.15.44:0/863957494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52440ceec0 0x7f522c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5248ba8640 1 -- 172.21.15.44:0/863957494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52440ceec0 msgr2=0x7f522c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5248ba8640 1 --2- 172.21.15.44:0/863957494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52440ceec0 0x7f522c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5248ba8640 1 -- 172.21.15.44:0/863957494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f522c004d40 msgr2=0x7f522c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5248ba8640 1 --2- 172.21.15.44:0/863957494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f522c004d40 0x7f522c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f5248ba8640 1 -- 172.21.15.44:0/863957494 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522c003520 con 0x7f522c003f30 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.120+0000 7f52493a9640 1 --2- 172.21.15.44:0/863957494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52440ceec0 0x7f522c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:07.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.121+0000 7f5248ba8640 1 --2- 172.21.15.44:0/863957494 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f522c003f30 0x7f522c004800 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f523800c210 tx=0x7f522c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=515ea37673023dd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:07.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.121+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5238007d50 con 0x7f522c003f30 2024-04-02T13:28:07.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.121+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5238030070 con 0x7f522c003f30 2024-04-02T13:28:07.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.121+0000 7f5243fff640 1 -- 172.21.15.44:0/863957494 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523802cdb0 con 0x7f522c003f30 2024-04-02T13:28:07.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.122+0000 7f524b60d640 1 -- 172.21.15.44:0/863957494 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f522c003f30 msgr2=0x7f522c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.122+0000 7f524b60d640 1 --2- 172.21.15.44:0/863957494 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f522c003f30 0x7f522c004800 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f523800c210 tx=0x7f522c005830 comp rx=0 tx=0).stop 2024-04-02T13:28:07.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.122+0000 7f5249baa640 1 -- 172.21.15.44:0/863957494 reap_dead start 2024-04-02T13:28:07.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.122+0000 7f524b60d640 1 -- 172.21.15.44:0/863957494 shutdown_connections 2024-04-02T13:28:07.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.122+0000 7f524b60d640 1 -- 172.21.15.44:0/863957494 >> 172.21.15.44:0/863957494 conn(0x7f52440b2a10 msgr2=0x7f52440b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:07.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.122+0000 7f524b60d640 1 -- 172.21.15.44:0/863957494 shutdown_connections 2024-04-02T13:28:07.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.122+0000 7f524b60d640 1 -- 172.21.15.44:0/863957494 wait complete. 2024-04-02T13:28:07.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.123+0000 7f524b60d640 1 Processor -- start 2024-04-02T13:28:07.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.123+0000 7f524b60d640 1 -- start start 2024-04-02T13:28:07.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f524b60d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f522c003f30 0x7f5244145a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f524b60d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f524415ee90 0x7f52441f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f524b60d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52441f5670 0x7f52441f5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f524b60d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f524415c470 con 0x7f522c003f30 2024-04-02T13:28:07.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f524b60d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f524406ff70 con 0x7f52441f5670 2024-04-02T13:28:07.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f524b60d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f524415c5f0 con 0x7f524415ee90 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f52493a9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f522c003f30 0x7f5244145a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f52493a9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f522c003f30 0x7f5244145a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48200/0 (socket says 172.21.15.44:48200) 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f52493a9640 1 -- 172.21.15.44:0/82053159 learned_addr learned my addr 172.21.15.44:0/82053159 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f5249baa640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52441f5670 0x7f52441f5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f52493a9640 1 -- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f524415ee90 msgr2=0x7f52441f5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f5248ba8640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f524415ee90 0x7f52441f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f52493a9640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f524415ee90 0x7f52441f5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f52493a9640 1 -- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52441f5670 msgr2=0x7f52441f5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f52493a9640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52441f5670 0x7f52441f5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f52493a9640 1 -- 172.21.15.44:0/82053159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5244154640 con 0x7f522c003f30 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f5249baa640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52441f5670 0x7f52441f5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f5248ba8640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f524415ee90 0x7f52441f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f52493a9640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f522c003f30 0x7f5244145a20 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f523c011630 tx=0x7f523c011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f5241ffb640 1 -- 172.21.15.44:0/82053159 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c00f920 con 0x7f522c003f30 2024-04-02T13:28:07.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f5241ffb640 1 -- 172.21.15.44:0/82053159 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f523c00fac0 con 0x7f522c003f30 2024-04-02T13:28:07.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f524b60d640 1 -- 172.21.15.44:0/82053159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5244154920 con 0x7f522c003f30 2024-04-02T13:28:07.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.124+0000 7f5241ffb640 1 -- 172.21.15.44:0/82053159 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c00fe90 con 0x7f522c003f30 2024-04-02T13:28:07.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.125+0000 7f524b60d640 1 -- 172.21.15.44:0/82053159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5244154ee0 con 0x7f522c003f30 2024-04-02T13:28:07.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.126+0000 7f524b60d640 1 -- 172.21.15.44:0/82053159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52440ccfb0 con 0x7f522c003f30 2024-04-02T13:28:07.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.126+0000 7f5241ffb640 1 -- 172.21.15.44:0/82053159 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f523c015460 con 0x7f522c003f30 2024-04-02T13:28:07.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.126+0000 7f5241ffb640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f523006cac0 0x7f523006ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.127+0000 7f5241ffb640 1 -- 172.21.15.44:0/82053159 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f523c08c780 con 0x7f522c003f30 2024-04-02T13:28:07.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.127+0000 7f5248ba8640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f523006cac0 0x7f523006ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.127+0000 7f5248ba8640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f523006cac0 0x7f523006ef80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5238034000 tx=0x7f5238007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:07.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.131+0000 7f5241ffb640 1 -- 172.21.15.44:0/82053159 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f52440ccfb0 con 0x7f522c003f30 2024-04-02T13:28:07.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.313+0000 7f524b60d640 1 -- 172.21.15.44:0/82053159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f524406c2b0 con 0x7f522c003f30 2024-04-02T13:28:07.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.314+0000 7f5241ffb640 1 -- 172.21.15.44:0/82053159 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f523c019030 con 0x7f522c003f30 2024-04-02T13:28:07.317 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:28:07.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.316+0000 7f524b60d640 1 -- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f523006cac0 msgr2=0x7f523006ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.316+0000 7f524b60d640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f523006cac0 0x7f523006ef80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5238034000 tx=0x7f5238007000 comp rx=0 tx=0).stop 2024-04-02T13:28:07.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.316+0000 7f524b60d640 1 -- 172.21.15.44:0/82053159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f522c003f30 msgr2=0x7f5244145a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.316+0000 7f524b60d640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f522c003f30 0x7f5244145a20 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f523c011630 tx=0x7f523c011af0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.316+0000 7f524b60d640 1 -- 172.21.15.44:0/82053159 shutdown_connections 2024-04-02T13:28:07.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.316+0000 7f524b60d640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f523006cac0 0x7f523006ef80 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.316+0000 7f524b60d640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52441f5670 0x7f52441f5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.316+0000 7f524b60d640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f524415ee90 0x7f52441f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.316+0000 7f524b60d640 1 --2- 172.21.15.44:0/82053159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f522c003f30 0x7f5244145a20 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.316+0000 7f524b60d640 1 -- 172.21.15.44:0/82053159 >> 172.21.15.44:0/82053159 conn(0x7f52440b2a10 msgr2=0x7f5244159bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:07.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.316+0000 7f524b60d640 1 -- 172.21.15.44:0/82053159 shutdown_connections 2024-04-02T13:28:07.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.316+0000 7f524b60d640 1 -- 172.21.15.44:0/82053159 wait complete. 2024-04-02T13:28:07.329 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:28:07.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.431+0000 7f3feca2a640 1 Processor -- start 2024-04-02T13:28:07.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.431+0000 7f3feca2a640 1 -- start start 2024-04-02T13:28:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.431+0000 7f3feca2a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3fe815e470 0x7f3fe80b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.431+0000 7f3feca2a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fe80b1540 0x7f3fe80d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.431+0000 7f3feca2a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3fe80ce790 0x7f3fe80d20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.431+0000 7f3feca2a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f3fe80d2630 con 0x7f3fe80ce790 2024-04-02T13:28:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.431+0000 7f3feca2a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3fe80ced00 con 0x7f3fe815e470 2024-04-02T13:28:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.431+0000 7f3feca2a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3fe80cefe0 con 0x7f3fe80b1540 2024-04-02T13:28:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.431+0000 7f3fe6d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3fe80ce790 0x7f3fe80d20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.432+0000 7f3fe6d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3fe80ce790 0x7f3fe80d20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48214/0 (socket says 172.21.15.44:48214) 2024-04-02T13:28:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.432+0000 7f3fe6d9d640 1 -- 172.21.15.44:0/2211072932 learned_addr learned my addr 172.21.15.44:0/2211072932 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.432+0000 7f3fe659c640 1 --2- 172.21.15.44:0/2211072932 >> v2:172.21.15.69:3300/0 conn(0x7f3fe815e470 0x7f3fe80b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.432+0000 7f3fdffff640 1 --2- 172.21.15.44:0/2211072932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fe80b1540 0x7f3fe80d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.432+0000 7f3fe6d9d640 1 -- 172.21.15.44:0/2211072932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fe80b1540 msgr2=0x7f3fe80d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.432+0000 7f3fe6d9d640 1 --2- 172.21.15.44:0/2211072932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fe80b1540 0x7f3fe80d1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.432+0000 7f3fe6d9d640 1 -- 172.21.15.44:0/2211072932 >> v2:172.21.15.69:3300/0 conn(0x7f3fe815e470 msgr2=0x7f3fe80b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.432+0000 7f3fe6d9d640 1 --2- 172.21.15.44:0/2211072932 >> v2:172.21.15.69:3300/0 conn(0x7f3fe815e470 0x7f3fe80b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.432+0000 7f3fe6d9d640 1 -- 172.21.15.44:0/2211072932 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe80cf2c0 con 0x7f3fe80ce790 2024-04-02T13:28:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.432+0000 7f3fdffff640 1 --2- 172.21.15.44:0/2211072932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fe80b1540 0x7f3fe80d1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.433+0000 7f3fe6d9d640 1 --2- 172.21.15.44:0/2211072932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fe80ce790 0x7f3fe80d20f0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f3fe806f1c0 tx=0x7f3fe002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272d969ea74150cd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.433+0000 7f3fe5d9b640 1 -- 172.21.15.44:0/2211072932 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe002c9e0 con 0x7f3fe80ce790 2024-04-02T13:28:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.433+0000 7f3fe5d9b640 1 -- 172.21.15.44:0/2211072932 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fe002cb80 con 0x7f3fe80ce790 2024-04-02T13:28:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.433+0000 7f3fe5d9b640 1 -- 172.21.15.44:0/2211072932 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe00343f0 con 0x7f3fe80ce790 2024-04-02T13:28:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.433+0000 7f3feca2a640 1 -- 172.21.15.44:0/2211072932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fe80ce790 msgr2=0x7f3fe80d20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.433+0000 7f3feca2a640 1 --2- 172.21.15.44:0/2211072932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fe80ce790 0x7f3fe80d20f0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f3fe806f1c0 tx=0x7f3fe002be40 comp rx=0 tx=0).stop 2024-04-02T13:28:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.433+0000 7f3feca2a640 1 -- 172.21.15.44:0/2211072932 shutdown_connections 2024-04-02T13:28:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.433+0000 7f3feca2a640 1 --2- 172.21.15.44:0/2211072932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fe80ce790 0x7f3fe80d20f0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.433+0000 7f3feca2a640 1 --2- 172.21.15.44:0/2211072932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fe80b1540 0x7f3fe80d1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.433+0000 7f3feca2a640 1 --2- 172.21.15.44:0/2211072932 >> v2:172.21.15.69:3300/0 conn(0x7f3fe815e470 0x7f3fe80b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.433+0000 7f3feca2a640 1 -- 172.21.15.44:0/2211072932 >> 172.21.15.44:0/2211072932 conn(0x7f3fe809aa20 msgr2=0x7f3fe809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:07.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.433+0000 7f3feca2a640 1 -- 172.21.15.44:0/2211072932 shutdown_connections 2024-04-02T13:28:07.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.433+0000 7f3feca2a640 1 -- 172.21.15.44:0/2211072932 wait complete. 2024-04-02T13:28:07.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.435+0000 7f3feca2a640 1 Processor -- start 2024-04-02T13:28:07.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.435+0000 7f3feca2a640 1 -- start start 2024-04-02T13:28:07.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.435+0000 7f3feca2a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fe80b1540 0x7f3fe8156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.435+0000 7f3feca2a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fe80ce790 0x7f3fe80b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.435+0000 7f3feca2a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fe815e470 0x7f3fe80b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.435+0000 7f3feca2a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe806cb60 con 0x7f3fe80b1540 2024-04-02T13:28:07.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.435+0000 7f3feca2a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe815e850 con 0x7f3fe815e470 2024-04-02T13:28:07.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.435+0000 7f3feca2a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3fe8161fd0 con 0x7f3fe80ce790 2024-04-02T13:28:07.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.435+0000 7f3fe659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fe80b1540 0x7f3fe8156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.435+0000 7f3fe659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fe80b1540 0x7f3fe8156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48226/0 (socket says 172.21.15.44:48226) 2024-04-02T13:28:07.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.435+0000 7f3fe659c640 1 -- 172.21.15.44:0/1914137093 learned_addr learned my addr 172.21.15.44:0/1914137093 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:07.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.436+0000 7f3fe6d9d640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fe815e470 0x7f3fe80b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.436+0000 7f3fdffff640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fe80ce790 0x7f3fe80b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.436+0000 7f3fe659c640 1 -- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fe80ce790 msgr2=0x7f3fe80b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.436+0000 7f3fe659c640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fe80ce790 0x7f3fe80b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.436+0000 7f3fe659c640 1 -- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fe815e470 msgr2=0x7f3fe80b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.436+0000 7f3fe659c640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fe815e470 0x7f3fe80b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.436+0000 7f3fe659c640 1 -- 172.21.15.44:0/1914137093 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe8157aa0 con 0x7f3fe80b1540 2024-04-02T13:28:07.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.436+0000 7f3fdffff640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fe80ce790 0x7f3fe80b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:07.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.436+0000 7f3fe659c640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fe80b1540 0x7f3fe8156cf0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800d5f0 tx=0x7f3fd800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:07.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.438+0000 7f3fdeffd640 1 -- 172.21.15.44:0/1914137093 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd80125f0 con 0x7f3fe80b1540 2024-04-02T13:28:07.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.438+0000 7f3fdeffd640 1 -- 172.21.15.44:0/1914137093 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fd8019070 con 0x7f3fe80b1540 2024-04-02T13:28:07.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.438+0000 7f3feca2a640 1 -- 172.21.15.44:0/1914137093 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fe81f3e20 con 0x7f3fe80b1540 2024-04-02T13:28:07.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.438+0000 7f3fdeffd640 1 -- 172.21.15.44:0/1914137093 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd80129d0 con 0x7f3fe80b1540 2024-04-02T13:28:07.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.438+0000 7f3feca2a640 1 -- 172.21.15.44:0/1914137093 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe80c5b50 con 0x7f3fe80b1540 2024-04-02T13:28:07.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.439+0000 7f3feca2a640 1 -- 172.21.15.44:0/1914137093 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fe80684f0 con 0x7f3fe80b1540 2024-04-02T13:28:07.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.439+0000 7f3fdeffd640 1 -- 172.21.15.44:0/1914137093 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3fd8012b70 con 0x7f3fe80b1540 2024-04-02T13:28:07.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.439+0000 7f3fdeffd640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fc806c8f0 0x7f3fc806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.440+0000 7f3fdffff640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fc806c8f0 0x7f3fc806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.440+0000 7f3fdeffd640 1 -- 172.21.15.44:0/1914137093 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f3fd808a5d0 con 0x7f3fe80b1540 2024-04-02T13:28:07.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.440+0000 7f3fdffff640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fc806c8f0 0x7f3fc806edb0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0006fd0 tx=0x7f3fd0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:07.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.445+0000 7f3fdeffd640 1 -- 172.21.15.44:0/1914137093 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3fd805a330 con 0x7f3fe80b1540 2024-04-02T13:28:07.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.628+0000 7f3feca2a640 1 -- 172.21.15.44:0/1914137093 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3fe80d2d10 con 0x7f3fe80b1540 2024-04-02T13:28:07.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.629+0000 7f3fdeffd640 1 -- 172.21.15.44:0/1914137093 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f3fd8059c30 con 0x7f3fe80b1540 2024-04-02T13:28:07.631 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:28:07.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.631+0000 7f3feca2a640 1 -- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fc806c8f0 msgr2=0x7f3fc806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.631+0000 7f3feca2a640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fc806c8f0 0x7f3fc806edb0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0006fd0 tx=0x7f3fd0008040 comp rx=0 tx=0).stop 2024-04-02T13:28:07.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.631+0000 7f3feca2a640 1 -- 172.21.15.44:0/1914137093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fe80b1540 msgr2=0x7f3fe8156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.631+0000 7f3feca2a640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fe80b1540 0x7f3fe8156cf0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800d5f0 tx=0x7f3fd800dab0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.631+0000 7f3feca2a640 1 -- 172.21.15.44:0/1914137093 shutdown_connections 2024-04-02T13:28:07.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.631+0000 7f3feca2a640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fc806c8f0 0x7f3fc806edb0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.631+0000 7f3feca2a640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fe815e470 0x7f3fe80b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.631+0000 7f3feca2a640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fe80ce790 0x7f3fe80b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.631+0000 7f3feca2a640 1 --2- 172.21.15.44:0/1914137093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fe80b1540 0x7f3fe8156cf0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.631+0000 7f3feca2a640 1 -- 172.21.15.44:0/1914137093 >> 172.21.15.44:0/1914137093 conn(0x7f3fe809aa20 msgr2=0x7f3fe80b2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:07.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.631+0000 7f3feca2a640 1 -- 172.21.15.44:0/1914137093 shutdown_connections 2024-04-02T13:28:07.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.631+0000 7f3feca2a640 1 -- 172.21.15.44:0/1914137093 wait complete. 2024-04-02T13:28:07.644 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:28:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.746+0000 7fc22eeba640 1 Processor -- start 2024-04-02T13:28:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.746+0000 7fc22eeba640 1 -- start start 2024-04-02T13:28:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.747+0000 7fc22eeba640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc2281565d0 0x7fc2280d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.747+0000 7fc22eeba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2280cf5d0 0x7fc22815c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.747+0000 7fc22eeba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc228160b10 con 0x7fc2280d05e0 2024-04-02T13:28:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.747+0000 7fc22eeba640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc2280cfad0 con 0x7fc2281565d0 2024-04-02T13:28:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.747+0000 7fc22eeba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc22815fd50 con 0x7fc2280cf5d0 2024-04-02T13:28:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.747+0000 7fc22cc56640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc2281565d0 0x7fc2280d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.747+0000 7fc227fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc2280d05e0 0x7fc2280ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42668/0 (socket says 172.21.15.44:42668) 2024-04-02T13:28:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.747+0000 7fc227fff640 1 -- 172.21.15.44:0/3173381135 learned_addr learned my addr 172.21.15.44:0/3173381135 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.747+0000 7fc22d457640 1 --2- 172.21.15.44:0/3173381135 >> v2:172.21.15.69:3300/0 conn(0x7fc2280cf5d0 0x7fc22815c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.747+0000 7fc22cc56640 1 -- 172.21.15.44:0/3173381135 >> v2:172.21.15.69:3300/0 conn(0x7fc2280cf5d0 msgr2=0x7fc22815c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.747+0000 7fc22cc56640 1 --2- 172.21.15.44:0/3173381135 >> v2:172.21.15.69:3300/0 conn(0x7fc2280cf5d0 0x7fc22815c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.747+0000 7fc22cc56640 1 -- 172.21.15.44:0/3173381135 >> v1:172.21.15.44:6789/0 conn(0x7fc2280d05e0 legacy=0x7fc2280ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.747+0000 7fc22cc56640 1 -- 172.21.15.44:0/3173381135 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc228160030 con 0x7fc2281565d0 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.748+0000 7fc22cc56640 1 --2- 172.21.15.44:0/3173381135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2281565d0 0x7fc2280d00a0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fc22806f3c0 tx=0x7fc21c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e64c9979c80a4e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.748+0000 7fc2277fe640 1 -- 172.21.15.44:0/3173381135 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21c0315a0 con 0x7fc2281565d0 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.748+0000 7fc2277fe640 1 -- 172.21.15.44:0/3173381135 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc21c037070 con 0x7fc2281565d0 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.748+0000 7fc2277fe640 1 -- 172.21.15.44:0/3173381135 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21c031960 con 0x7fc2281565d0 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.749+0000 7fc22eeba640 1 -- 172.21.15.44:0/3173381135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2281565d0 msgr2=0x7fc2280d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.749+0000 7fc22eeba640 1 --2- 172.21.15.44:0/3173381135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2281565d0 0x7fc2280d00a0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fc22806f3c0 tx=0x7fc21c02b840 comp rx=0 tx=0).stop 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.749+0000 7fc22eeba640 1 -- 172.21.15.44:0/3173381135 shutdown_connections 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.749+0000 7fc22eeba640 1 --2- 172.21.15.44:0/3173381135 >> v2:172.21.15.69:3300/0 conn(0x7fc2280cf5d0 0x7fc22815c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.749+0000 7fc22eeba640 1 --2- 172.21.15.44:0/3173381135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2281565d0 0x7fc2280d00a0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.749+0000 7fc22eeba640 1 -- 172.21.15.44:0/3173381135 >> 172.21.15.44:0/3173381135 conn(0x7fc22806b920 msgr2=0x7fc22806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:07.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.749+0000 7fc22eeba640 1 -- 172.21.15.44:0/3173381135 shutdown_connections 2024-04-02T13:28:07.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.749+0000 7fc22eeba640 1 -- 172.21.15.44:0/3173381135 wait complete. 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.750+0000 7fc22eeba640 1 Processor -- start 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.750+0000 7fc22eeba640 1 -- start start 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22eeba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2280cf5d0 0x7fc22814d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22eeba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2280d05e0 0x7fc22809cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22eeba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2281565d0 0x7fc22809d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22eeba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc2281569b0 con 0x7fc2281565d0 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22eeba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc2281605f0 con 0x7fc2280d05e0 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22eeba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc228156b30 con 0x7fc2280cf5d0 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22d457640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2281565d0 0x7fc22809d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22d457640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2281565d0 0x7fc22809d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48252/0 (socket says 172.21.15.44:48252) 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22d457640 1 -- 172.21.15.44:0/1438981506 learned_addr learned my addr 172.21.15.44:0/1438981506 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc227fff640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2280d05e0 0x7fc22809cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22d457640 1 -- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2280cf5d0 msgr2=0x7fc22814d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22cc56640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2280cf5d0 0x7fc22814d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22d457640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2280cf5d0 0x7fc22814d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22d457640 1 -- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2280d05e0 msgr2=0x7fc22809cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22d457640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2280d05e0 0x7fc22809cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22d457640 1 -- 172.21.15.44:0/1438981506 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22809c6a0 con 0x7fc2281565d0 2024-04-02T13:28:07.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc227fff640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2280d05e0 0x7fc22809cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:07.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22cc56640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2280cf5d0 0x7fc22814d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:07.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22d457640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2281565d0 0x7fc22809d2c0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fc22000ccb0 tx=0x7fc22000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:07.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc2257fa640 1 -- 172.21.15.44:0/1438981506 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22000ce60 con 0x7fc2281565d0 2024-04-02T13:28:07.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc2257fa640 1 -- 172.21.15.44:0/1438981506 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc220003990 con 0x7fc2281565d0 2024-04-02T13:28:07.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc22eeba640 1 -- 172.21.15.44:0/1438981506 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc22809b600 con 0x7fc2281565d0 2024-04-02T13:28:07.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.751+0000 7fc2257fa640 1 -- 172.21.15.44:0/1438981506 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220003da0 con 0x7fc2281565d0 2024-04-02T13:28:07.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.752+0000 7fc22eeba640 1 -- 172.21.15.44:0/1438981506 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc22809bbc0 con 0x7fc2281565d0 2024-04-02T13:28:07.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.752+0000 7fc22eeba640 1 -- 172.21.15.44:0/1438981506 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2280b0210 con 0x7fc2281565d0 2024-04-02T13:28:07.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.753+0000 7fc2257fa640 1 -- 172.21.15.44:0/1438981506 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc220003f40 con 0x7fc2281565d0 2024-04-02T13:28:07.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.753+0000 7fc2257fa640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc20c06c920 0x7fc20c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:07.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.753+0000 7fc2257fa640 1 -- 172.21.15.44:0/1438981506 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7fc22008c200 con 0x7fc2281565d0 2024-04-02T13:28:07.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.753+0000 7fc22cc56640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc20c06c920 0x7fc20c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:07.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.754+0000 7fc22cc56640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc20c06c920 0x7fc20c06ede0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc21c036000 tx=0x7fc21c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:07.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.758+0000 7fc2257fa640 1 -- 172.21.15.44:0/1438981506 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc2280b0210 con 0x7fc2281565d0 2024-04-02T13:28:07.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.943+0000 7fc22eeba640 1 -- 172.21.15.44:0/1438981506 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc2280684f0 con 0x7fc2281565d0 2024-04-02T13:28:07.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.944+0000 7fc2257fa640 1 -- 172.21.15.44:0/1438981506 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fc220014030 con 0x7fc2281565d0 2024-04-02T13:28:07.946 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:28:07.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.945+0000 7fc22eeba640 1 -- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc20c06c920 msgr2=0x7fc20c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.945+0000 7fc22eeba640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc20c06c920 0x7fc20c06ede0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc21c036000 tx=0x7fc21c008030 comp rx=0 tx=0).stop 2024-04-02T13:28:07.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.945+0000 7fc22eeba640 1 -- 172.21.15.44:0/1438981506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2281565d0 msgr2=0x7fc22809d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:07.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.945+0000 7fc22eeba640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2281565d0 0x7fc22809d2c0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fc22000ccb0 tx=0x7fc22000a5c0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.946+0000 7fc22eeba640 1 -- 172.21.15.44:0/1438981506 shutdown_connections 2024-04-02T13:28:07.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.946+0000 7fc22eeba640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc20c06c920 0x7fc20c06ede0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.946+0000 7fc22eeba640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2281565d0 0x7fc22809d2c0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.946+0000 7fc22eeba640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2280d05e0 0x7fc22809cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.946+0000 7fc22eeba640 1 --2- 172.21.15.44:0/1438981506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2280cf5d0 0x7fc22814d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:07.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.946+0000 7fc22eeba640 1 -- 172.21.15.44:0/1438981506 >> 172.21.15.44:0/1438981506 conn(0x7fc22806b920 msgr2=0x7fc2280afb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:07.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.946+0000 7fc22eeba640 1 -- 172.21.15.44:0/1438981506 shutdown_connections 2024-04-02T13:28:07.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:07.946+0000 7fc22eeba640 1 -- 172.21.15.44:0/1438981506 wait complete. 2024-04-02T13:28:07.959 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:28:08.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.059+0000 7f5a0f59e640 1 Processor -- start 2024-04-02T13:28:08.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.059+0000 7f5a0f59e640 1 -- start start 2024-04-02T13:28:08.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0f59e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5a1015e2d0 0x7f5a100d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:08.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0f59e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5a100d49e0 0x7f5a100d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:08.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0f59e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a100d1770 con 0x7f5a100d29e0 2024-04-02T13:28:08.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0f59e640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f5a1006bec0 con 0x7f5a100d49e0 2024-04-02T13:28:08.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0f59e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5a100d2e10 con 0x7f5a1015e2d0 2024-04-02T13:28:08.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0dd9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5a100d49e0 0x7f5a100d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:08.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0dd9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5a100d49e0 0x7f5a100d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48256/0 (socket says 172.21.15.44:48256) 2024-04-02T13:28:08.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0dd9b640 1 -- 172.21.15.44:0/4214528912 learned_addr learned my addr 172.21.15.44:0/4214528912 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:08.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0e59c640 1 --2- 172.21.15.44:0/4214528912 >> v2:172.21.15.69:3300/0 conn(0x7f5a1015e2d0 0x7f5a100d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:08.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0dd9b640 1 -- 172.21.15.44:0/4214528912 >> v2:172.21.15.69:3300/0 conn(0x7f5a1015e2d0 msgr2=0x7f5a100d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:08.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0dd9b640 1 --2- 172.21.15.44:0/4214528912 >> v2:172.21.15.69:3300/0 conn(0x7f5a1015e2d0 0x7f5a100d1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:08.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0dd9b640 1 -- 172.21.15.44:0/4214528912 >> v1:172.21.15.69:6789/0 conn(0x7f5a100d29e0 legacy=0x7f5a100d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:08.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0dd9b640 1 -- 172.21.15.44:0/4214528912 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a100cf680 con 0x7f5a100d49e0 2024-04-02T13:28:08.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.060+0000 7f5a0e59c640 1 --2- 172.21.15.44:0/4214528912 >> v2:172.21.15.69:3300/0 conn(0x7f5a1015e2d0 0x7f5a100d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:08.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.061+0000 7f5a0dd9b640 1 --2- 172.21.15.44:0/4214528912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a100d49e0 0x7f5a100d24a0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f5a1006e110 tx=0x7f5a0402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3133f2044ddcf892 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:08.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.061+0000 7f5a0d59a640 1 -- 172.21.15.44:0/4214528912 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04034070 con 0x7f5a100d49e0 2024-04-02T13:28:08.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.061+0000 7f5a0d59a640 1 -- 172.21.15.44:0/4214528912 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a04030860 con 0x7f5a100d49e0 2024-04-02T13:28:08.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.061+0000 7f5a0d59a640 1 -- 172.21.15.44:0/4214528912 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04030c30 con 0x7f5a100d49e0 2024-04-02T13:28:08.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.062+0000 7f5a0f59e640 1 -- 172.21.15.44:0/4214528912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a100d49e0 msgr2=0x7f5a100d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:08.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.062+0000 7f5a0f59e640 1 --2- 172.21.15.44:0/4214528912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a100d49e0 0x7f5a100d24a0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f5a1006e110 tx=0x7f5a0402b9a0 comp rx=0 tx=0).stop 2024-04-02T13:28:08.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.062+0000 7f5a0f59e640 1 -- 172.21.15.44:0/4214528912 shutdown_connections 2024-04-02T13:28:08.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.062+0000 7f5a0f59e640 1 --2- 172.21.15.44:0/4214528912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a100d49e0 0x7f5a100d24a0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:08.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.062+0000 7f5a0f59e640 1 --2- 172.21.15.44:0/4214528912 >> v2:172.21.15.69:3300/0 conn(0x7f5a1015e2d0 0x7f5a100d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:08.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.062+0000 7f5a0f59e640 1 -- 172.21.15.44:0/4214528912 >> 172.21.15.44:0/4214528912 conn(0x7f5a100d3ec0 msgr2=0x7f5a100d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:08.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.062+0000 7f5a0f59e640 1 -- 172.21.15.44:0/4214528912 shutdown_connections 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.062+0000 7f5a0f59e640 1 -- 172.21.15.44:0/4214528912 wait complete. 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.063+0000 7f5a0f59e640 1 Processor -- start 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.063+0000 7f5a0f59e640 1 -- start start 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0f59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a100d29e0 0x7f5a100cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a100d49e0 0x7f5a101f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0f59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a1015e2d0 0x7f5a100b9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0f59e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1015e6b0 con 0x7f5a100d49e0 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0f59e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5a100cfc40 con 0x7f5a1015e2d0 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0f59e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5a1015e830 con 0x7f5a100d29e0 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0dd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a100d49e0 0x7f5a101f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0dd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a100d49e0 0x7f5a101f48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48272/0 (socket says 172.21.15.44:48272) 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0dd9b640 1 -- 172.21.15.44:0/301472979 learned_addr learned my addr 172.21.15.44:0/301472979 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0ed9d640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a1015e2d0 0x7f5a100b9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:08.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0e59c640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a100d29e0 0x7f5a100cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:08.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0dd9b640 1 -- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a100d29e0 msgr2=0x7f5a100cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:08.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0dd9b640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a100d29e0 0x7f5a100cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:08.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0dd9b640 1 -- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a1015e2d0 msgr2=0x7f5a100b9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:08.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0dd9b640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a1015e2d0 0x7f5a100b9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:08.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0dd9b640 1 -- 172.21.15.44:0/301472979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a100cc2d0 con 0x7f5a100d49e0 2024-04-02T13:28:08.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0ed9d640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a1015e2d0 0x7f5a100b9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:08.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.064+0000 7f5a0dd9b640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a100d49e0 0x7f5a101f48d0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f5a04000c00 tx=0x7f5a040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:08.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.065+0000 7f59feffd640 1 -- 172.21.15.44:0/301472979 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04034040 con 0x7f5a100d49e0 2024-04-02T13:28:08.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.065+0000 7f59feffd640 1 -- 172.21.15.44:0/301472979 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a04039d90 con 0x7f5a100d49e0 2024-04-02T13:28:08.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.065+0000 7f5a0f59e640 1 -- 172.21.15.44:0/301472979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a100cc5b0 con 0x7f5a100d49e0 2024-04-02T13:28:08.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.065+0000 7f59feffd640 1 -- 172.21.15.44:0/301472979 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04043610 con 0x7f5a100d49e0 2024-04-02T13:28:08.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.065+0000 7f5a0f59e640 1 -- 172.21.15.44:0/301472979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a100ccb70 con 0x7f5a100d49e0 2024-04-02T13:28:08.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.066+0000 7f5a0f59e640 1 -- 172.21.15.44:0/301472979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a1004f8d0 con 0x7f5a100d49e0 2024-04-02T13:28:08.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.066+0000 7f59feffd640 1 -- 172.21.15.44:0/301472979 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5a04037070 con 0x7f5a100d49e0 2024-04-02T13:28:08.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.066+0000 7f59feffd640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59f006c8f0 0x7f59f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:08.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.066+0000 7f59feffd640 1 -- 172.21.15.44:0/301472979 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f5a040adf90 con 0x7f5a100d49e0 2024-04-02T13:28:08.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.067+0000 7f5a0e59c640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59f006c8f0 0x7f59f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:08.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.067+0000 7f5a0e59c640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59f006c8f0 0x7f59f006edb0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5a00006fd0 tx=0x7f5a00008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:08.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.071+0000 7f59feffd640 1 -- 172.21.15.44:0/301472979 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5a0403e070 con 0x7f5a100d49e0 2024-04-02T13:28:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.268+0000 7f5a0f59e640 1 -- 172.21.15.44:0/301472979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5a100d42a0 con 0x7f5a100d49e0 2024-04-02T13:28:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.268+0000 7f59feffd640 1 -- 172.21.15.44:0/301472979 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v94) ==== 80+0+120 (secure 0 0 0) 0x7f5a0407c0f0 con 0x7f5a100d49e0 2024-04-02T13:28:08.272 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6833/719680934 2024-04-03T13:26:55.589090+0000 2024-04-02T13:28:08.272 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6832/719680934 2024-04-03T13:26:55.589090+0000 2024-04-02T13:28:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:listed 2 entries 2024-04-02T13:28:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.270+0000 7f5a0f59e640 1 -- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59f006c8f0 msgr2=0x7f59f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.270+0000 7f5a0f59e640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59f006c8f0 0x7f59f006edb0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5a00006fd0 tx=0x7f5a00008040 comp rx=0 tx=0).stop 2024-04-02T13:28:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.270+0000 7f5a0f59e640 1 -- 172.21.15.44:0/301472979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a100d49e0 msgr2=0x7f5a101f48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.270+0000 7f5a0f59e640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a100d49e0 0x7f5a101f48d0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f5a04000c00 tx=0x7f5a040046c0 comp rx=0 tx=0).stop 2024-04-02T13:28:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.270+0000 7f5a0f59e640 1 -- 172.21.15.44:0/301472979 shutdown_connections 2024-04-02T13:28:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.270+0000 7f5a0f59e640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f59f006c8f0 0x7f59f006edb0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.270+0000 7f5a0f59e640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a1015e2d0 0x7f5a100b9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.270+0000 7f5a0f59e640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a100d49e0 0x7f5a101f48d0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:08.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.270+0000 7f5a0f59e640 1 --2- 172.21.15.44:0/301472979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a100d29e0 0x7f5a100cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:08.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.270+0000 7f5a0f59e640 1 -- 172.21.15.44:0/301472979 >> 172.21.15.44:0/301472979 conn(0x7f5a100d3ec0 msgr2=0x7f5a101613c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:08.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.270+0000 7f5a0f59e640 1 -- 172.21.15.44:0/301472979 shutdown_connections 2024-04-02T13:28:08.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:08.270+0000 7f5a0f59e640 1 -- 172.21.15.44:0/301472979 wait complete. 2024-04-02T13:28:08.283 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:28:08.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:08.284 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:28:08.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[71243]: fuse finished with error 0 and tester_r 0 2024-04-02T13:28:08.311 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:28:14.413 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:28:14.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:14.413 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:28:14.432 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:28:14.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.580+0000 7fc458b3d640 1 Processor -- start 2024-04-02T13:28:14.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.580+0000 7fc458b3d640 1 -- start start 2024-04-02T13:28:14.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.581+0000 7fc458b3d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc45415e0f0 0x7fc4540d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:14.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.581+0000 7fc458b3d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4540d0630 0x7fc45406b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:14.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.581+0000 7fc458b3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45415e4d0 0x7fc45406ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:14.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.581+0000 7fc458b3d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc45406dfe0 con 0x7fc45415e0f0 2024-04-02T13:28:14.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.581+0000 7fc458b3d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc45406e2c0 con 0x7fc4540d0630 2024-04-02T13:28:14.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.581+0000 7fc458b3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc45406e5a0 con 0x7fc45415e4d0 2024-04-02T13:28:14.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.581+0000 7fc45259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc45415e0f0 0x7fc4540d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:14.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.581+0000 7fc45259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc45415e0f0 0x7fc4540d00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34556/0 (socket says 172.21.15.44:34556) 2024-04-02T13:28:14.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.581+0000 7fc45259c640 1 -- 172.21.15.44:0/2659131754 learned_addr learned my addr 172.21.15.44:0/2659131754 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:14.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.581+0000 7fc451d9b640 1 --2- 172.21.15.44:0/2659131754 >> v2:172.21.15.69:3300/0 conn(0x7fc4540d0630 0x7fc45406b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:14.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.581+0000 7fc452d9d640 1 --2- 172.21.15.44:0/2659131754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45415e4d0 0x7fc45406ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:14.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.582+0000 7fc45259c640 1 -- 172.21.15.44:0/2659131754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45415e4d0 msgr2=0x7fc45406ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:14.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.582+0000 7fc45259c640 1 --2- 172.21.15.44:0/2659131754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45415e4d0 0x7fc45406ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:14.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.582+0000 7fc45259c640 1 -- 172.21.15.44:0/2659131754 >> v2:172.21.15.69:3300/0 conn(0x7fc4540d0630 msgr2=0x7fc45406b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:14.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.582+0000 7fc45259c640 1 --2- 172.21.15.44:0/2659131754 >> v2:172.21.15.69:3300/0 conn(0x7fc4540d0630 0x7fc45406b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:14.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.582+0000 7fc45259c640 1 -- 172.21.15.44:0/2659131754 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc45406e880 con 0x7fc45415e0f0 2024-04-02T13:28:14.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.583+0000 7fc45259c640 1 --2- 172.21.15.44:0/2659131754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc45415e0f0 0x7fc4540d00f0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fc45409ee50 tx=0x7fc44c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41aeb748644c1cc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:14.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.583+0000 7fc45159a640 1 -- 172.21.15.44:0/2659131754 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c034070 con 0x7fc45415e0f0 2024-04-02T13:28:14.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.583+0000 7fc45159a640 1 -- 172.21.15.44:0/2659131754 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc44c030860 con 0x7fc45415e0f0 2024-04-02T13:28:14.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.583+0000 7fc45159a640 1 -- 172.21.15.44:0/2659131754 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c030c30 con 0x7fc45415e0f0 2024-04-02T13:28:14.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.583+0000 7fc458b3d640 1 -- 172.21.15.44:0/2659131754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc45415e0f0 msgr2=0x7fc4540d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:14.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.583+0000 7fc458b3d640 1 --2- 172.21.15.44:0/2659131754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc45415e0f0 0x7fc4540d00f0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fc45409ee50 tx=0x7fc44c02b9a0 comp rx=0 tx=0).stop 2024-04-02T13:28:14.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.583+0000 7fc458b3d640 1 -- 172.21.15.44:0/2659131754 shutdown_connections 2024-04-02T13:28:14.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.583+0000 7fc458b3d640 1 --2- 172.21.15.44:0/2659131754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45415e4d0 0x7fc45406ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:14.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.583+0000 7fc458b3d640 1 --2- 172.21.15.44:0/2659131754 >> v2:172.21.15.69:3300/0 conn(0x7fc4540d0630 0x7fc45406b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:14.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.583+0000 7fc458b3d640 1 --2- 172.21.15.44:0/2659131754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc45415e0f0 0x7fc4540d00f0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:14.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.583+0000 7fc458b3d640 1 -- 172.21.15.44:0/2659131754 >> 172.21.15.44:0/2659131754 conn(0x7fc4540b2980 msgr2=0x7fc4540b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:14.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.583+0000 7fc458b3d640 1 -- 172.21.15.44:0/2659131754 shutdown_connections 2024-04-02T13:28:14.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.583+0000 7fc458b3d640 1 -- 172.21.15.44:0/2659131754 wait complete. 2024-04-02T13:28:14.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.585+0000 7fc458b3d640 1 Processor -- start 2024-04-02T13:28:14.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.585+0000 7fc458b3d640 1 -- start start 2024-04-02T13:28:14.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc458b3d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4540d0630 0x7fc45414b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:14.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc458b3d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc45415e0f0 0x7fc4540c4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:14.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc458b3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45415e4d0 0x7fc4540c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:14.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc458b3d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc454072f60 con 0x7fc4540d0630 2024-04-02T13:28:14.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc458b3d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc45406eda0 con 0x7fc45415e0f0 2024-04-02T13:28:14.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc458b3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc4540730e0 con 0x7fc45415e4d0 2024-04-02T13:28:14.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc45259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4540d0630 0x7fc45414b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:14.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc45259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4540d0630 0x7fc45414b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34558/0 (socket says 172.21.15.44:34558) 2024-04-02T13:28:14.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc45259c640 1 -- 172.21.15.44:0/3019649787 learned_addr learned my addr 172.21.15.44:0/3019649787 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:14.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc451d9b640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc45415e0f0 0x7fc4540c4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:14.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc452d9d640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45415e4d0 0x7fc4540c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:14.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc45259c640 1 -- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45415e4d0 msgr2=0x7fc4540c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:14.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc45259c640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45415e4d0 0x7fc4540c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:14.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc45259c640 1 -- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc45415e0f0 msgr2=0x7fc4540c4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:14.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc45259c640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc45415e0f0 0x7fc4540c4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:14.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc45259c640 1 -- 172.21.15.44:0/3019649787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4540c4fc0 con 0x7fc4540d0630 2024-04-02T13:28:14.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc45259c640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4540d0630 0x7fc45414b260 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fc44c02b9e0 tx=0x7fc44c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:14.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc442ffd640 1 -- 172.21.15.44:0/3019649787 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c048070 con 0x7fc4540d0630 2024-04-02T13:28:14.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.586+0000 7fc442ffd640 1 -- 172.21.15.44:0/3019649787 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc44c030e20 con 0x7fc4540d0630 2024-04-02T13:28:14.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.587+0000 7fc458b3d640 1 -- 172.21.15.44:0/3019649787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4540c52a0 con 0x7fc4540d0630 2024-04-02T13:28:14.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.587+0000 7fc442ffd640 1 -- 172.21.15.44:0/3019649787 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c034040 con 0x7fc4540d0630 2024-04-02T13:28:14.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.587+0000 7fc458b3d640 1 -- 172.21.15.44:0/3019649787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4540c5860 con 0x7fc4540d0630 2024-04-02T13:28:14.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.588+0000 7fc458b3d640 1 -- 172.21.15.44:0/3019649787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4540684f0 con 0x7fc4540d0630 2024-04-02T13:28:14.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.588+0000 7fc442ffd640 1 -- 172.21.15.44:0/3019649787 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc44c04b070 con 0x7fc4540d0630 2024-04-02T13:28:14.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.588+0000 7fc442ffd640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc43406c8f0 0x7fc43406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:14.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.588+0000 7fc442ffd640 1 -- 172.21.15.44:0/3019649787 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7fc44c0adc20 con 0x7fc4540d0630 2024-04-02T13:28:14.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.589+0000 7fc451d9b640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc43406c8f0 0x7fc43406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:14.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.589+0000 7fc451d9b640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc43406c8f0 0x7fc43406edb0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc4540c9d50 tx=0x7fc448008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:14.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.593+0000 7fc442ffd640 1 -- 172.21.15.44:0/3019649787 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc44c07be90 con 0x7fc4540d0630 2024-04-02T13:28:14.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.816+0000 7fc458b3d640 1 -- 172.21.15.44:0/3019649787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4540727f0 con 0x7fc4540d0630 2024-04-02T13:28:14.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.817+0000 7fc442ffd640 1 -- 172.21.15.44:0/3019649787 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5181 (secure 0 0 0) 0x7fc44c07b790 con 0x7fc4540d0630 2024-04-02T13:28:14.821 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:14.821 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":68,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:27:06.599726+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8073":{"gid":8073,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":6,"addr":"172.21.15.44:6833/2250279975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2250279975},{"type":"v1","addr":"172.21.15.44:6833","nonce":2250279975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8073,"qdb_cluster":[8073]},"id":6}]} 2024-04-02T13:28:14.821 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 68 2024-04-02T13:28:14.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.819+0000 7fc458b3d640 1 -- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc43406c8f0 msgr2=0x7fc43406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:14.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.819+0000 7fc458b3d640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc43406c8f0 0x7fc43406edb0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc4540c9d50 tx=0x7fc448008040 comp rx=0 tx=0).stop 2024-04-02T13:28:14.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.819+0000 7fc458b3d640 1 -- 172.21.15.44:0/3019649787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4540d0630 msgr2=0x7fc45414b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:14.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.819+0000 7fc458b3d640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4540d0630 0x7fc45414b260 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fc44c02b9e0 tx=0x7fc44c004730 comp rx=0 tx=0).stop 2024-04-02T13:28:14.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.819+0000 7fc458b3d640 1 -- 172.21.15.44:0/3019649787 shutdown_connections 2024-04-02T13:28:14.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.819+0000 7fc458b3d640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc43406c8f0 0x7fc43406edb0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:14.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.819+0000 7fc458b3d640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45415e4d0 0x7fc4540c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:14.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.819+0000 7fc458b3d640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc45415e0f0 0x7fc4540c4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:14.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.819+0000 7fc458b3d640 1 --2- 172.21.15.44:0/3019649787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4540d0630 0x7fc45414b260 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:14.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.819+0000 7fc458b3d640 1 -- 172.21.15.44:0/3019649787 >> 172.21.15.44:0/3019649787 conn(0x7fc4540b2980 msgr2=0x7fc4540b0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:14.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.819+0000 7fc458b3d640 1 -- 172.21.15.44:0/3019649787 shutdown_connections 2024-04-02T13:28:14.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:14.819+0000 7fc458b3d640 1 -- 172.21.15.44:0/3019649787 wait complete. 2024-04-02T13:28:14.834 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-04-02T13:28:14.964 INFO:teuthology.orchestra.run.smithi044.stdout:{"message":"","return_code":0} 2024-04-02T13:28:14.965 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-02T13:28:14.965 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:28:15.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.064+0000 7f52978f2640 1 Processor -- start 2024-04-02T13:28:15.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.064+0000 7f52978f2640 1 -- start start 2024-04-02T13:28:15.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.065+0000 7f52978f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f529015e0f0 0x7f52900d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.065+0000 7f52978f2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f529006bbe0 0x7f52900add60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.065+0000 7f52978f2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f529006dfe0 con 0x7f52900d0630 2024-04-02T13:28:15.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.065+0000 7f52978f2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f529006e2c0 con 0x7f529015e0f0 2024-04-02T13:28:15.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.065+0000 7f52978f2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f529006e5a0 con 0x7f529006bbe0 2024-04-02T13:28:15.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.065+0000 7f529568e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f529015e0f0 0x7f52900d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.065+0000 7f5294e8d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f52900d0630 0x7f529006b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41442/0 (socket says 172.21.15.44:41442) 2024-04-02T13:28:15.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.065+0000 7f5294e8d640 1 -- 172.21.15.44:0/668749126 learned_addr learned my addr 172.21.15.44:0/668749126 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:15.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.065+0000 7f5295e8f640 1 --2- 172.21.15.44:0/668749126 >> v2:172.21.15.69:3300/0 conn(0x7f529006bbe0 0x7f52900add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.065+0000 7f529568e640 1 -- 172.21.15.44:0/668749126 >> v2:172.21.15.69:3300/0 conn(0x7f529006bbe0 msgr2=0x7f52900add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.065+0000 7f529568e640 1 --2- 172.21.15.44:0/668749126 >> v2:172.21.15.69:3300/0 conn(0x7f529006bbe0 0x7f52900add60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.065+0000 7f529568e640 1 -- 172.21.15.44:0/668749126 >> v1:172.21.15.44:6789/0 conn(0x7f52900d0630 legacy=0x7f529006b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.066+0000 7f5287fff640 1 -- 172.21.15.44:0/668749126 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778761856 0 0) 0x7f529006dfe0 con 0x7f52900d0630 2024-04-02T13:28:15.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.066+0000 7f529568e640 1 -- 172.21.15.44:0/668749126 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529006e880 con 0x7f529015e0f0 2024-04-02T13:28:15.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.066+0000 7f529568e640 1 --2- 172.21.15.44:0/668749126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f529015e0f0 0x7f52900d00f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f5290162dc0 tx=0x7f528802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65aa0f8de91f97ed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.066+0000 7f5287fff640 1 -- 172.21.15.44:0/668749126 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288034070 con 0x7f529015e0f0 2024-04-02T13:28:15.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.066+0000 7f5287fff640 1 -- 172.21.15.44:0/668749126 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5288030610 con 0x7f529015e0f0 2024-04-02T13:28:15.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.066+0000 7f5287fff640 1 -- 172.21.15.44:0/668749126 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52880309e0 con 0x7f529015e0f0 2024-04-02T13:28:15.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.067+0000 7f52978f2640 1 -- 172.21.15.44:0/668749126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f529015e0f0 msgr2=0x7f52900d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.067+0000 7f52978f2640 1 --2- 172.21.15.44:0/668749126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f529015e0f0 0x7f52900d00f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f5290162dc0 tx=0x7f528802b960 comp rx=0 tx=0).stop 2024-04-02T13:28:15.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.067+0000 7f52978f2640 1 -- 172.21.15.44:0/668749126 shutdown_connections 2024-04-02T13:28:15.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.067+0000 7f52978f2640 1 --2- 172.21.15.44:0/668749126 >> v2:172.21.15.69:3300/0 conn(0x7f529006bbe0 0x7f52900add60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.067+0000 7f52978f2640 1 --2- 172.21.15.44:0/668749126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f529015e0f0 0x7f52900d00f0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.067+0000 7f52978f2640 1 -- 172.21.15.44:0/668749126 >> 172.21.15.44:0/668749126 conn(0x7f52900b2980 msgr2=0x7f52900b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:15.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.067+0000 7f52978f2640 1 -- 172.21.15.44:0/668749126 shutdown_connections 2024-04-02T13:28:15.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.067+0000 7f52978f2640 1 -- 172.21.15.44:0/668749126 wait complete. 2024-04-02T13:28:15.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.068+0000 7f52978f2640 1 Processor -- start 2024-04-02T13:28:15.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.068+0000 7f52978f2640 1 -- start start 2024-04-02T13:28:15.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f52978f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f529006bbe0 0x7f529015d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f52978f2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52900d0630 0x7f52901e8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f52978f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f529015e0f0 0x7f52900c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f52978f2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f529015e4d0 con 0x7f529015e0f0 2024-04-02T13:28:15.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f52978f2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f529006eda0 con 0x7f52900d0630 2024-04-02T13:28:15.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f52978f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f529015e650 con 0x7f529006bbe0 2024-04-02T13:28:15.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f5295e8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f529015e0f0 0x7f52900c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f5295e8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f529015e0f0 0x7f52900c9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34574/0 (socket says 172.21.15.44:34574) 2024-04-02T13:28:15.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f5295e8f640 1 -- 172.21.15.44:0/3687200397 learned_addr learned my addr 172.21.15.44:0/3687200397 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:15.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f529568e640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f529006bbe0 0x7f529015d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f5294e8d640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52900d0630 0x7f52901e8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f5295e8f640 1 -- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f529006bbe0 msgr2=0x7f529015d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f5295e8f640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f529006bbe0 0x7f529015d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f5295e8f640 1 -- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52900d0630 msgr2=0x7f52901e8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f5295e8f640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52900d0630 0x7f52901e8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.069+0000 7f5295e8f640 1 -- 172.21.15.44:0/3687200397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52901e9110 con 0x7f529015e0f0 2024-04-02T13:28:15.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.070+0000 7f5295e8f640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f529015e0f0 0x7f52900c9bd0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f528c00cda0 tx=0x7f528c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.070+0000 7f5285ffb640 1 -- 172.21.15.44:0/3687200397 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c003b90 con 0x7f529015e0f0 2024-04-02T13:28:15.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.070+0000 7f5285ffb640 1 -- 172.21.15.44:0/3687200397 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f528c003d30 con 0x7f529015e0f0 2024-04-02T13:28:15.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.070+0000 7f52978f2640 1 -- 172.21.15.44:0/3687200397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52900c5330 con 0x7f529015e0f0 2024-04-02T13:28:15.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.070+0000 7f5285ffb640 1 -- 172.21.15.44:0/3687200397 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c004240 con 0x7f529015e0f0 2024-04-02T13:28:15.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.070+0000 7f52978f2640 1 -- 172.21.15.44:0/3687200397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52900c4f50 con 0x7f529015e0f0 2024-04-02T13:28:15.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.071+0000 7f52978f2640 1 -- 172.21.15.44:0/3687200397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52900684f0 con 0x7f529015e0f0 2024-04-02T13:28:15.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.071+0000 7f5285ffb640 1 -- 172.21.15.44:0/3687200397 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f528c0043e0 con 0x7f529015e0f0 2024-04-02T13:28:15.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.072+0000 7f5285ffb640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f527406c8f0 0x7f527406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.072+0000 7f5285ffb640 1 -- 172.21.15.44:0/3687200397 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6303+0+0 (secure 0 0 0) 0x7f528c08c7e0 con 0x7f529015e0f0 2024-04-02T13:28:15.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.072+0000 7f529568e640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f527406c8f0 0x7f527406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.072+0000 7f529568e640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f527406c8f0 0x7f527406edb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5288036000 tx=0x7f5288008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.076+0000 7f5285ffb640 1 -- 172.21.15.44:0/3687200397 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f528c05a2d0 con 0x7f529015e0f0 2024-04-02T13:28:15.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.298+0000 7f52978f2640 1 -- 172.21.15.44:0/3687200397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f52900c3f50 con 0x7f529015e0f0 2024-04-02T13:28:15.302 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:28:15.299+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:28:15.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.300+0000 7f5285ffb640 1 -- 172.21.15.44:0/3687200397 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v69) ==== 161+0+0 (secure 0 0 0) 0x7f528c05e5d0 con 0x7f529015e0f0 2024-04-02T13:28:15.304 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:28:15.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.302+0000 7f52978f2640 1 -- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f527406c8f0 msgr2=0x7f527406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.302+0000 7f52978f2640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f527406c8f0 0x7f527406edb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5288036000 tx=0x7f5288008030 comp rx=0 tx=0).stop 2024-04-02T13:28:15.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.302+0000 7f52978f2640 1 -- 172.21.15.44:0/3687200397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f529015e0f0 msgr2=0x7f52900c9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.302+0000 7f52978f2640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f529015e0f0 0x7f52900c9bd0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f528c00cda0 tx=0x7f528c00a660 comp rx=0 tx=0).stop 2024-04-02T13:28:15.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.302+0000 7f52978f2640 1 -- 172.21.15.44:0/3687200397 shutdown_connections 2024-04-02T13:28:15.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.302+0000 7f52978f2640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f527406c8f0 0x7f527406edb0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.302+0000 7f52978f2640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f529015e0f0 0x7f52900c9bd0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.302+0000 7f52978f2640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52900d0630 0x7f52901e8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.303+0000 7f52978f2640 1 --2- 172.21.15.44:0/3687200397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f529006bbe0 0x7f529015d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.303+0000 7f52978f2640 1 -- 172.21.15.44:0/3687200397 >> 172.21.15.44:0/3687200397 conn(0x7f52900b2980 msgr2=0x7f52900b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:15.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.303+0000 7f52978f2640 1 -- 172.21.15.44:0/3687200397 shutdown_connections 2024-04-02T13:28:15.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.303+0000 7f52978f2640 1 -- 172.21.15.44:0/3687200397 wait complete. 2024-04-02T13:28:15.310 INFO:tasks.ceph.mds.a.smithi044.stderr: -4860> 2024-04-02T13:27:01.599+0000 7fe0b4a36640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:28:15.310 INFO:tasks.ceph.mds.a.smithi044.stderr: -4510> 2024-04-02T13:27:02.655+0000 7fe0b0a2e640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-04-02T13:28:15.310 INFO:tasks.ceph.mds.a.smithi044.stderr: -4340> 2024-04-02T13:27:02.840+0000 7fe0b0a2e640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 ap=1 f(v0 m2024-04-02T13:26:51.014993+0000 1=1+0) n(v0 rc2024-04-02T13:26:51.131857+0000 2=1+1)/n(v0 rc2024-04-02T13:26:51.014993+0000 2=1+1) | dirfrag=0 dirtyrstat=1 authpin=1 scrubqueue=0 0x5647a570fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-04-02T13:26:51.014993+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-04-02T13:26:51.131857+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2} 2024-04-02T13:28:15.312 INFO:tasks.ceph.mds.a.smithi044.stderr: -2476> 2024-04-02T13:27:34.172+0000 7fe0b0a2e640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v17 pv19 ap=3 DIRTYPARENT f(v0 m2024-04-02T13:26:51.014993+0000 1=1+0)->f(v1 m2024-04-02T13:26:51.014993+0000) n(v0 rc2024-04-02T13:26:51.131857+0000 2=1+1)->n(v0 rc2024-04-02T13:26:51.131857+0000 2=1+1) (inest lock dirty) (ifile lock->sync w=1 flushing) | dirtyscattered=2 lock=1 dirfrag=1 dirtyrstat=0 dirtyparent=1 dirty=1 waiter=1 authpin=1 scrubqueue=0 0x5647a570fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000000:[<0x1/dir v17>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-04-02T13:26:51.014993+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-04-02T13:26:51.131857+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2024-04-02T13:28:15.320 DEBUG:teuthology.orchestra.run.smithi044:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents cephfs_data 2024-04-02T13:28:15.339 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:28:15.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.355+0000 7f421a431740 1 -- start start 2024-04-02T13:28:15.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.355+0000 7f421a431740 1 Processor -- start 2024-04-02T13:28:15.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.355+0000 7f421a431740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559503242c00 0x5595033b2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.355+0000 7f421a431740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559503243800 0x5595033b2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.355+0000 7f421a431740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559503243c00 0x5595033b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.355+0000 7f421a431740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559504028780 con 0x559503243800 2024-04-02T13:28:15.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.355+0000 7f421a431740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559504028600 con 0x559503243c00 2024-04-02T13:28:15.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.355+0000 7f421a431740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x559504028480 con 0x559503242c00 2024-04-02T13:28:15.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42174ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559503243800 0x5595033b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42174ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559503243800 0x5595033b2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34606/0 (socket says 172.21.15.44:34606) 2024-04-02T13:28:15.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42174ed640 1 -- 172.21.15.44:0/2504100702 learned_addr learned my addr 172.21.15.44:0/2504100702 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:15.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42184ef640 1 --2- 172.21.15.44:0/2504100702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559503243c00 0x5595033b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f4217cee640 1 --2- 172.21.15.44:0/2504100702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559503242c00 0x5595033b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42174ed640 1 -- 172.21.15.44:0/2504100702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559503242c00 msgr2=0x5595033b2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42174ed640 1 --2- 172.21.15.44:0/2504100702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559503242c00 0x5595033b2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42174ed640 1 -- 172.21.15.44:0/2504100702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559503243c00 msgr2=0x5595033b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42174ed640 1 --2- 172.21.15.44:0/2504100702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559503243c00 0x5595033b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42174ed640 1 -- 172.21.15.44:0/2504100702 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55950326f680 con 0x559503243800 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42174ed640 1 --2- 172.21.15.44:0/2504100702 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559503243800 0x5595033b2580 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x5595040599e0 tx=0x559503fdc600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc0eea7ffe5b527 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42154e9640 1 -- 172.21.15.44:0/2504100702 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x559503268680 con 0x559503243800 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42154e9640 1 -- 172.21.15.44:0/2504100702 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55950328ce00 con 0x559503243800 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42154e9640 1 -- 172.21.15.44:0/2504100702 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x559503268b60 con 0x559503243800 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f421a431740 1 -- 172.21.15.44:0/2504100702 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55950326f860 con 0x559503243800 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f421a431740 1 -- 172.21.15.44:0/2504100702 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55950326f2c0 con 0x559503243800 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f421a431740 4 waiting for MDS map... 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42154e9640 1 -- 172.21.15.44:0/2504100702 <== mon.0 v2:172.21.15.44:3300/0 4 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x559504118600 con 0x559503243800 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f42154e9640 1 -- 172.21.15.44:0/2504100702 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 69) ==== 2885+0+0 (secure 0 0 0) 0x55950329b180 con 0x559503243800 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.356+0000 7f421a431740 4 Got MDS map 69 2024-04-02T13:28:15.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.357+0000 7f421a431740 4 datascan.main: Using metadata pool output 2024-04-02T13:28:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.357+0000 7f421a431740 4 datascan.main: connecting to RADOS... 2024-04-02T13:28:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.357+0000 7f421a431740 1 Processor -- start 2024-04-02T13:28:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.357+0000 7f421a431740 1 -- start start 2024-04-02T13:28:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f421a431740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559504105400 0x5595033b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f421a431740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559504105800 0x5595033b3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f421a431740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 0x5595033b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f421a431740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559504028300 con 0x559504105c00 2024-04-02T13:28:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f421a431740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559504028000 con 0x559504105800 2024-04-02T13:28:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f421a431740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5595032a2900 con 0x559504105400 2024-04-02T13:28:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f42174ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 0x5595033b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f42174ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 0x5595033b3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34616/0 (socket says 172.21.15.44:34616) 2024-04-02T13:28:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f42174ed640 1 -- 172.21.15.44:0/3696028950 learned_addr learned my addr 172.21.15.44:0/3696028950 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:15.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f42184ef640 1 --2- 172.21.15.44:0/3696028950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559504105400 0x5595033b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f4217cee640 1 --2- 172.21.15.44:0/3696028950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559504105800 0x5595033b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f42174ed640 1 -- 172.21.15.44:0/3696028950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559504105400 msgr2=0x5595033b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f42174ed640 1 --2- 172.21.15.44:0/3696028950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559504105400 0x5595033b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f42174ed640 1 -- 172.21.15.44:0/3696028950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559504105800 msgr2=0x5595033b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f42174ed640 1 --2- 172.21.15.44:0/3696028950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559504105800 0x5595033b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f42174ed640 1 -- 172.21.15.44:0/3696028950 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55950326f4a0 con 0x559504105c00 2024-04-02T13:28:15.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f42174ed640 1 --2- 172.21.15.44:0/3696028950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 0x5595033b3b80 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x55950410ede0 tx=0x559503fdca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8e48d91a6a2468 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f42134e5640 1 -- 172.21.15.44:0/3696028950 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x559503268820 con 0x559504105c00 2024-04-02T13:28:15.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f42134e5640 1 -- 172.21.15.44:0/3696028950 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55950328cc40 con 0x559504105c00 2024-04-02T13:28:15.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.358+0000 7f42134e5640 1 -- 172.21.15.44:0/3696028950 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x559503268680 con 0x559504105c00 2024-04-02T13:28:15.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.359+0000 7f421a431740 1 -- 172.21.15.44:0/3696028950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 msgr2=0x5595033b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.359+0000 7f421a431740 1 --2- 172.21.15.44:0/3696028950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 0x5595033b3b80 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x55950410ede0 tx=0x559503fdca20 comp rx=0 tx=0).stop 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.359+0000 7f421a431740 1 -- 172.21.15.44:0/3696028950 shutdown_connections 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.359+0000 7f421a431740 1 --2- 172.21.15.44:0/3696028950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 0x5595033b3b80 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.359+0000 7f421a431740 1 --2- 172.21.15.44:0/3696028950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559504105800 0x5595033b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.359+0000 7f421a431740 1 --2- 172.21.15.44:0/3696028950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559504105400 0x5595033b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.359+0000 7f421a431740 1 -- 172.21.15.44:0/3696028950 >> 172.21.15.44:0/3696028950 conn(0x559504105000 msgr2=0x559503327000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.359+0000 7f421a431740 1 -- 172.21.15.44:0/3696028950 shutdown_connections 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.359+0000 7f421a431740 1 -- 172.21.15.44:0/3696028950 wait complete. 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.359+0000 7f421a431740 1 Processor -- start 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.359+0000 7f421a431740 1 -- start start 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f421a431740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559504105400 0x5595033b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f421a431740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559504105800 0x5595033b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f421a431740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 0x5595033b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f421a431740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559504029680 con 0x559504105c00 2024-04-02T13:28:15.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f421a431740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559504029800 con 0x559504105400 2024-04-02T13:28:15.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f421a431740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x559504029500 con 0x559504105800 2024-04-02T13:28:15.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f42174ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 0x5595033b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f42174ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 0x5595033b3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34618/0 (socket says 172.21.15.44:34618) 2024-04-02T13:28:15.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 learned_addr learned my addr 172.21.15.44:0/2829702913 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:15.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f42184ef640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559504105400 0x5595033b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f4217cee640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559504105800 0x5595033b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559504105800 msgr2=0x5595033b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f42174ed640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559504105800 0x5595033b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559504105400 msgr2=0x5595033b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f42174ed640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559504105400 0x5595033b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55950326f680 con 0x559504105c00 2024-04-02T13:28:15.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.360+0000 7f42184ef640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559504105400 0x5595033b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.361+0000 7f42174ed640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 0x5595033b3b80 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x55950410faa0 tx=0x559504131800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.361+0000 7f4211ce2640 1 -- 172.21.15.44:0/2829702913 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x559503268b60 con 0x559504105c00 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.361+0000 7f4211ce2640 1 -- 172.21.15.44:0/2829702913 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55950328ca80 con 0x559504105c00 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.361+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55950326f860 con 0x559504105c00 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.361+0000 7f4211ce2640 1 -- 172.21.15.44:0/2829702913 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5595032684e0 con 0x559504105c00 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.361+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55950326f0e0 con 0x559504105c00 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.361+0000 7f421a431740 4 datascan.init: resolving metadata pool 13 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.362+0000 7f4211ce2640 1 -- 172.21.15.44:0/2829702913 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x55950418cc00 con 0x559504105c00 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.362+0000 7f4211ce2640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55950418b000 0x5595033b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.362+0000 7f4211ce2640 1 -- 172.21.15.44:0/2829702913 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x55950419e600 con 0x559504105c00 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.362+0000 7f421a431740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.362+0000 7f421a431740 4 datascan.main: data pool 'cephfs_data' has ID 14 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.362+0000 7f421a431740 4 datascan.main: opening data pool 'cephfs_data' 2024-04-02T13:28:15.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.362+0000 7f421a431740 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x55950418b400 0x5595033b4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.362+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:1 14.0 14:00000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950418b800 con 0x55950418b400 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.362+0000 7f42184ef640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55950418b000 0x5595033b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.363+0000 7f4217cee640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x55950418b400 0x5595033b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.363+0000 7f4217cee640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x55950418b400 0x5595033b4680 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.363+0000 7f42184ef640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55950418b000 0x5595033b4100 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x559503fca210 tx=0x5595032f8d20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.363+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.2 v2:172.21.15.44:6816/2091834331 1 ==== osd_op_reply(1 [pgnls start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950418b400 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.363+0000 7f4212ce4640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55950403c000 0x5595035d8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.363+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:2 14.20 14:04000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950403c400 con 0x55950403c000 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.363+0000 7f42184ef640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55950403c000 0x5595035d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.364+0000 7f42184ef640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55950403c000 0x5595035d8000 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.364+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 1 ==== osd_op_reply(2 [pgnls start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.364+0000 7f42134e5640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55950403cc00 0x5595035d8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.364+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:3 14.10 14:08000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950403d000 con 0x55950403cc00 2024-04-02T13:28:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.364+0000 7f4217cee640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55950403cc00 0x5595035d8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.364+0000 7f4217cee640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55950403cc00 0x5595035d8580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.366+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(3 [pgnls start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950403cc00 2024-04-02T13:28:15.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.366+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:4 14.30 14:0c000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950403c800 con 0x55950403c000 2024-04-02T13:28:15.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.367+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 2 ==== osd_op_reply(4 [pgnls start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.367+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:5 14.8 14:10000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950403d400 con 0x55950403c000 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.367+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 3 ==== osd_op_reply(5 [pgnls start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.367+0000 7f4212ce4640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x55950403d000 0x5595035d8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.367+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:6 14.28 14:14000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950403d800 con 0x55950403d000 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.367+0000 7f42174ed640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x55950403d000 0x5595035d8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.367+0000 7f42174ed640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x55950403d000 0x5595035d8b00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.369+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.3 v2:172.21.15.44:6824/1588952513 1 ==== osd_op_reply(6 [pgnls start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4780 con 0x55950403d000 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.369+0000 7f42134e5640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55950403dc00 0x5595035d9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.369+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:7 14.18 14:18000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ec000 con 0x55950403dc00 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.369+0000 7f42184ef640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55950403dc00 0x5595035d9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.370+0000 7f42184ef640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55950403dc00 0x5595035d9600 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.372+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.7 v2:172.21.15.69:6824/1292475712 1 ==== osd_op_reply(7 [pgnls start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403dc00 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.372+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:8 14.38 14:1c000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950418b800 con 0x55950403cc00 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.372+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.1 v2:172.21.15.44:6801/704457846 2 ==== osd_op_reply(8 [pgnls start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950403cc00 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.372+0000 7f42134e5640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x5595035ec400 0x5595035d9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.372+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:9 14.4 14:20000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ec800 con 0x5595035ec400 2024-04-02T13:28:15.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.373+0000 7f4217cee640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x5595035ec400 0x5595035d9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.373+0000 7f4217cee640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x5595035ec400 0x5595035d9b80 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.375+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== osd_op_reply(9 [pgnls start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x5595035ec400 2024-04-02T13:28:15.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.375+0000 7f4212ce4640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55950403c400 0x5595035d9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.375+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:10 14.24 14:24000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950403c800 con 0x55950403c400 2024-04-02T13:28:15.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.375+0000 7f42174ed640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55950403c400 0x5595035d9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.376+0000 7f42174ed640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55950403c400 0x5595035d9080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.376+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.5 v2:172.21.15.69:6800/2841767499 1 ==== osd_op_reply(10 [pgnls start_epoch 95 out=49b] v91'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4780 con 0x55950403c400 2024-04-02T13:28:15.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.376+0000 7f42134e5640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x5595035ecc00 0x5595035da100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.376+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:11 14.14 14:28000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ec000 con 0x5595035ecc00 2024-04-02T13:28:15.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.377+0000 7f42184ef640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x5595035ecc00 0x5595035da100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.377+0000 7f42184ef640 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x5595035ecc00 0x5595035da100 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.379+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.4 v2:172.21.15.69:6816/1041350007 1 ==== osd_op_reply(11 [pgnls start_epoch 95 out=49b] v91'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x5595035ecc00 2024-04-02T13:28:15.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.379+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:12 14.34 14:2c000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950403d400 con 0x55950418b400 2024-04-02T13:28:15.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.380+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.2 v2:172.21.15.44:6816/2091834331 2 ==== osd_op_reply(12 [pgnls start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950418b400 2024-04-02T13:28:15.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.380+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:13 14.c 14:30000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ed000 con 0x55950403c000 2024-04-02T13:28:15.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.380+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 4 ==== osd_op_reply(13 [pgnls start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.380+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:14 14.2c 14:34000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x559504007400 con 0x55950418b400 2024-04-02T13:28:15.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.380+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.2 v2:172.21.15.44:6816/2091834331 3 ==== osd_op_reply(14 [pgnls start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950418b400 2024-04-02T13:28:15.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.380+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:15 14.1c 14:38000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ed400 con 0x55950403c000 2024-04-02T13:28:15.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.380+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 5 ==== osd_op_reply(15 [pgnls start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.380+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:16 14.3c 14:3c000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950403d400 con 0x55950403cc00 2024-04-02T13:28:15.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.380+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.1 v2:172.21.15.44:6801/704457846 3 ==== osd_op_reply(16 [pgnls start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950403cc00 2024-04-02T13:28:15.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.381+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:17 14.2 14:40000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ed800 con 0x5595035ec400 2024-04-02T13:28:15.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.381+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.6 v2:172.21.15.69:6802/2604158175 2 ==== osd_op_reply(17 [pgnls start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x5595035ec400 2024-04-02T13:28:15.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.381+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:18 14.22 14:44000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ec800 con 0x5595035ecc00 2024-04-02T13:28:15.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.381+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.4 v2:172.21.15.69:6816/1041350007 2 ==== osd_op_reply(18 [pgnls start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x5595035ecc00 2024-04-02T13:28:15.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.381+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:19 14.12 14:48000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035edc00 con 0x55950418b400 2024-04-02T13:28:15.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.382+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.2 v2:172.21.15.44:6816/2091834331 4 ==== osd_op_reply(19 [pgnls start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950418b400 2024-04-02T13:28:15.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.382+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:20 14.32 14:4c000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950418b800 con 0x5595035ecc00 2024-04-02T13:28:15.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.382+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.4 v2:172.21.15.69:6816/1041350007 3 ==== osd_op_reply(20 [pgnls start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x5595035ecc00 2024-04-02T13:28:15.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.382+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:21 14.a 14:50000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ed800 con 0x55950403c400 2024-04-02T13:28:15.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.382+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.5 v2:172.21.15.69:6800/2841767499 2 ==== osd_op_reply(21 [pgnls start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4780 con 0x55950403c400 2024-04-02T13:28:15.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.382+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:22 14.2a 14:54000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035fe000 con 0x5595035ecc00 2024-04-02T13:28:15.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.383+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.4 v2:172.21.15.69:6816/1041350007 4 ==== osd_op_reply(22 [pgnls start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x5595035ecc00 2024-04-02T13:28:15.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.383+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:23 14.1a 14:58000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035edc00 con 0x55950403c000 2024-04-02T13:28:15.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.383+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 6 ==== osd_op_reply(23 [pgnls start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.383+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:24 14.3a 14:5c000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035fe400 con 0x55950403c000 2024-04-02T13:28:15.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.384+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 7 ==== osd_op_reply(24 [pgnls start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.384+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:25 14.6 14:60000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950403d400 con 0x55950403c000 2024-04-02T13:28:15.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.384+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 8 ==== osd_op_reply(25 [pgnls start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.384+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:26 14.26 14:64000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035fe800 con 0x5595035ecc00 2024-04-02T13:28:15.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.384+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.4 v2:172.21.15.69:6816/1041350007 5 ==== osd_op_reply(26 [pgnls start_epoch 95 out=49b] v91'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x5595035ecc00 2024-04-02T13:28:15.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.384+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:27 14.16 14:68000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x559504007400 con 0x55950403d000 2024-04-02T13:28:15.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.384+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.3 v2:172.21.15.44:6824/1588952513 2 ==== osd_op_reply(27 [pgnls start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4780 con 0x55950403d000 2024-04-02T13:28:15.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.384+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:28 14.36 14:6c000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035fec00 con 0x55950418b400 2024-04-02T13:28:15.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.385+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.2 v2:172.21.15.44:6816/2091834331 5 ==== osd_op_reply(28 [pgnls start_epoch 95 out=49b] v91'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950418b400 2024-04-02T13:28:15.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.385+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:29 14.e 14:70000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ed800 con 0x5595035ecc00 2024-04-02T13:28:15.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.385+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.4 v2:172.21.15.69:6816/1041350007 6 ==== osd_op_reply(29 [pgnls start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x5595035ecc00 2024-04-02T13:28:15.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.385+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:30 14.2e 14:74000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ff000 con 0x55950403d000 2024-04-02T13:28:15.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.385+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.3 v2:172.21.15.44:6824/1588952513 3 ==== osd_op_reply(30 [pgnls start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4780 con 0x55950403d000 2024-04-02T13:28:15.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.385+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:31 14.1e 14:78000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ec000 con 0x55950403dc00 2024-04-02T13:28:15.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.386+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.7 v2:172.21.15.69:6824/1292475712 2 ==== osd_op_reply(31 [pgnls start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403dc00 2024-04-02T13:28:15.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.386+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:32 14.3e 14:7c000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ff400 con 0x55950403c400 2024-04-02T13:28:15.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.386+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.5 v2:172.21.15.69:6800/2841767499 3 ==== osd_op_reply(32 [pgnls start_epoch 95 out=49b] v91'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4780 con 0x55950403c400 2024-04-02T13:28:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.386+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:33 14.1 14:80000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ed000 con 0x55950403d000 2024-04-02T13:28:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.386+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.3 v2:172.21.15.44:6824/1588952513 4 ==== osd_op_reply(33 [pgnls start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4780 con 0x55950403d000 2024-04-02T13:28:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.386+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:34 14.21 14:84000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035fe000 con 0x55950403c400 2024-04-02T13:28:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.387+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.5 v2:172.21.15.69:6800/2841767499 4 ==== osd_op_reply(34 [pgnls start_epoch 95 out=49b] v91'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4780 con 0x55950403c400 2024-04-02T13:28:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.387+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:35 14.11 14:88000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ed400 con 0x5595035ecc00 2024-04-02T13:28:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.387+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.4 v2:172.21.15.69:6816/1041350007 7 ==== osd_op_reply(35 [pgnls start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x5595035ecc00 2024-04-02T13:28:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.387+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:36 14.31 14:8c000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ff800 con 0x55950403cc00 2024-04-02T13:28:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.387+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.1 v2:172.21.15.44:6801/704457846 4 ==== osd_op_reply(36 [pgnls start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950403cc00 2024-04-02T13:28:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.387+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:37 14.9 14:90000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x559504007400 con 0x55950403c400 2024-04-02T13:28:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.388+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.5 v2:172.21.15.69:6800/2841767499 5 ==== osd_op_reply(37 [pgnls start_epoch 95 out=49b] v91'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4780 con 0x55950403c400 2024-04-02T13:28:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.388+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:38 14.29 14:94000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ffc00 con 0x5595035ec400 2024-04-02T13:28:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.388+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.6 v2:172.21.15.69:6802/2604158175 3 ==== osd_op_reply(38 [pgnls start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x5595035ec400 2024-04-02T13:28:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.388+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:39 14.19 14:98000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950403d800 con 0x5595035ec400 2024-04-02T13:28:15.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.388+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.6 v2:172.21.15.69:6802/2604158175 4 ==== osd_op_reply(39 [pgnls start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x5595035ec400 2024-04-02T13:28:15.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.388+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:40 14.39 14:9c000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950418ac00 con 0x55950403dc00 2024-04-02T13:28:15.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.389+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.7 v2:172.21.15.69:6824/1292475712 3 ==== osd_op_reply(40 [pgnls start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403dc00 2024-04-02T13:28:15.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.389+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:41 14.5 14:a0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950403c800 con 0x55950403c000 2024-04-02T13:28:15.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.389+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 9 ==== osd_op_reply(41 [pgnls start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.389+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:42 14.25 14:a4000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950418a800 con 0x55950418b400 2024-04-02T13:28:15.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.389+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.2 v2:172.21.15.44:6816/2091834331 6 ==== osd_op_reply(42 [pgnls start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950418b400 2024-04-02T13:28:15.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.389+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:43 14.15 14:a8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035fec00 con 0x55950403dc00 2024-04-02T13:28:15.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.390+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.7 v2:172.21.15.69:6824/1292475712 4 ==== osd_op_reply(43 [pgnls start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403dc00 2024-04-02T13:28:15.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.390+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:44 14.35 14:ac000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950418b800 con 0x5595035ecc00 2024-04-02T13:28:15.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.390+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.4 v2:172.21.15.69:6816/1041350007 8 ==== osd_op_reply(44 [pgnls start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x5595035ecc00 2024-04-02T13:28:15.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.390+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:45 14.d 14:b0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ff800 con 0x5595035ec400 2024-04-02T13:28:15.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.391+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.6 v2:172.21.15.69:6802/2604158175 5 ==== osd_op_reply(45 [pgnls start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x5595035ec400 2024-04-02T13:28:15.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.391+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:46 14.2d 14:b4000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ec800 con 0x5595035ecc00 2024-04-02T13:28:15.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.391+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.4 v2:172.21.15.69:6816/1041350007 9 ==== osd_op_reply(46 [pgnls start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x5595035ecc00 2024-04-02T13:28:15.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.391+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:47 14.1d 14:b8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ffc00 con 0x55950403c000 2024-04-02T13:28:15.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.392+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 10 ==== osd_op_reply(47 [pgnls start_epoch 95 out=49b] v91'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.392+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:48 14.3d 14:bc000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035edc00 con 0x55950403d000 2024-04-02T13:28:15.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.392+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.3 v2:172.21.15.44:6824/1588952513 5 ==== osd_op_reply(48 [pgnls start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4780 con 0x55950403d000 2024-04-02T13:28:15.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.392+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:49 14.3 14:c0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035fe400 con 0x55950403c000 2024-04-02T13:28:15.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.392+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 11 ==== osd_op_reply(49 [pgnls start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.392+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:50 14.23 14:c4000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ed800 con 0x55950403dc00 2024-04-02T13:28:15.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.393+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.7 v2:172.21.15.69:6824/1292475712 5 ==== osd_op_reply(50 [pgnls start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403dc00 2024-04-02T13:28:15.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.393+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:51 14.13 14:c8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035fe800 con 0x55950418b400 2024-04-02T13:28:15.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.393+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.2 v2:172.21.15.44:6816/2091834331 7 ==== osd_op_reply(51 [pgnls start_epoch 95 out=49b] v91'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950418b400 2024-04-02T13:28:15.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.393+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:52 14.33 14:cc000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ec000 con 0x55950403cc00 2024-04-02T13:28:15.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.393+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.1 v2:172.21.15.44:6801/704457846 5 ==== osd_op_reply(52 [pgnls start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950403cc00 2024-04-02T13:28:15.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.393+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:53 14.b 14:d0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950403d800 con 0x55950403c000 2024-04-02T13:28:15.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.393+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 12 ==== osd_op_reply(53 [pgnls start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.393+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:54 14.2b 14:d4000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ed400 con 0x5595035ecc00 2024-04-02T13:28:15.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.394+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.4 v2:172.21.15.69:6816/1041350007 10 ==== osd_op_reply(54 [pgnls start_epoch 95 out=49b] v91'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x5595035ecc00 2024-04-02T13:28:15.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.394+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:55 14.1b 14:d8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950403d400 con 0x55950403cc00 2024-04-02T13:28:15.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.394+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.1 v2:172.21.15.44:6801/704457846 6 ==== osd_op_reply(55 [pgnls start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950403cc00 2024-04-02T13:28:15.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.394+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:56 14.3b 14:dc000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ec800 con 0x5595035ecc00 2024-04-02T13:28:15.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.395+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.4 v2:172.21.15.69:6816/1041350007 11 ==== osd_op_reply(56 [pgnls start_epoch 95 out=49b] v91'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x5595035ecc00 2024-04-02T13:28:15.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.395+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:57 14.7 14:e0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950418a800 con 0x55950403c400 2024-04-02T13:28:15.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.395+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.5 v2:172.21.15.69:6800/2841767499 6 ==== osd_op_reply(57 [pgnls start_epoch 95 out=81b] v91'6 uv6 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x559503fd4780 con 0x55950403c400 2024-04-02T13:28:15.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.395+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:58 14.7 14:e2e07930:::10000000001.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x55950418bc00 con 0x55950403c400 2024-04-02T13:28:15.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.395+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.5 v2:172.21.15.69:6800/2841767499 7 ==== osd_op_reply(58 10000000001.00000000 [stat out=16b] v0'0 uv6 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x559503fd4780 con 0x55950403c400 2024-04-02T13:28:15.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.395+0000 7f421a431740 10 datascan.operator(): handling object 1099511627777.0 2024-04-02T13:28:15.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.395+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:59 14.7 14:e2e07930:::10000000001.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5595035ed000 con 0x55950403c400 2024-04-02T13:28:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.397+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.5 v2:172.21.15.69:6800/2841767499 8 ==== osd_op_reply(59 10000000001.00000000 [call] v95'7 uv7 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x559503fd4780 con 0x55950403c400 2024-04-02T13:28:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.397+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:60 14.27 14:e4000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035edc00 con 0x55950403c400 2024-04-02T13:28:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.397+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.5 v2:172.21.15.69:6800/2841767499 9 ==== osd_op_reply(60 [pgnls start_epoch 95 out=49b] v91'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4780 con 0x55950403c400 2024-04-02T13:28:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.397+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:61 14.17 14:e8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55950418a800 con 0x55950403cc00 2024-04-02T13:28:15.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.398+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 <== osd.1 v2:172.21.15.44:6801/704457846 7 ==== osd_op_reply(61 [pgnls start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4000 con 0x55950403cc00 2024-04-02T13:28:15.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.398+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:62 14.37 14:ec000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ed800 con 0x5595035ecc00 2024-04-02T13:28:15.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.398+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.4 v2:172.21.15.69:6816/1041350007 12 ==== osd_op_reply(62 [pgnls start_epoch 95 out=49b] v91'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x5595035ecc00 2024-04-02T13:28:15.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.398+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:63 14.f 14:f0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x559503606000 con 0x55950403dc00 2024-04-02T13:28:15.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.7 v2:172.21.15.69:6824/1292475712 6 ==== osd_op_reply(63 [pgnls start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403dc00 2024-04-02T13:28:15.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:64 14.2f 14:f4000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035fec00 con 0x55950403c000 2024-04-02T13:28:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 13 ==== osd_op_reply(64 [pgnls start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f42134e5640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:65 14.1f 14:f8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x559503606400 con 0x55950403c000 2024-04-02T13:28:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f42184ef640 1 -- 172.21.15.44:0/2829702913 <== osd.0 v2:172.21.15.44:6800/2303805525 14 ==== osd_op_reply(65 [pgnls start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4500 con 0x55950403c000 2024-04-02T13:28:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f4212ce4640 1 -- 172.21.15.44:0/2829702913 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:66 14.3f 14:fc000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x5595035ffc00 con 0x55950403d000 2024-04-02T13:28:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f42174ed640 1 -- 172.21.15.44:0/2829702913 <== osd.3 v2:172.21.15.44:6824/1588952513 6 ==== osd_op_reply(66 [pgnls start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x559503fd4780 con 0x55950403d000 2024-04-02T13:28:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55950403c000 msgr2=0x5595035d8000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55950403c000 0x5595035d8000 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55950403cc00 msgr2=0x5595035d8580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55950403cc00 0x5595035d8580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x55950418b400 msgr2=0x5595033b4680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x55950418b400 0x5595033b4680 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x55950403d000 msgr2=0x5595035d8b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x55950403d000 0x5595035d8b00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x5595035ecc00 msgr2=0x5595035da100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x5595035ecc00 0x5595035da100 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55950403c400 msgr2=0x5595035d9080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55950403c400 0x5595035d9080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x5595035ec400 msgr2=0x5595035d9b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x5595035ec400 0x5595035d9b80 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 reap_dead start 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55950403dc00 msgr2=0x5595035d9600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.399+0000 7f421a431740 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55950403dc00 0x5595035d9600 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 reap_dead start 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f4217cee640 1 -- 172.21.15.44:0/2829702913 reap_dead start 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55950418b000 msgr2=0x5595033b4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55950418b000 0x5595033b4100 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x559503fca210 tx=0x5595032f8d20 comp rx=0 tx=0).stop 2024-04-02T13:28:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 msgr2=0x5595033b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 0x5595033b3b80 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x55950410faa0 tx=0x559504131800 comp rx=0 tx=0).stop 2024-04-02T13:28:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 shutdown_connections 2024-04-02T13:28:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55950418b000 0x5595033b4100 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 --2- 172.21.15.44:0/2829702913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559504105c00 0x5595033b3b80 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 >> 172.21.15.44:0/2829702913 conn(0x559504105000 msgr2=0x559503327000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 shutdown_connections 2024-04-02T13:28:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 -- 172.21.15.44:0/2829702913 wait complete. 2024-04-02T13:28:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 -- 172.21.15.44:0/2504100702 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559503243800 msgr2=0x5595033b2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 --2- 172.21.15.44:0/2504100702 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559503243800 0x5595033b2580 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x5595040599e0 tx=0x559503fdc600 comp rx=0 tx=0).stop 2024-04-02T13:28:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 -- 172.21.15.44:0/2504100702 shutdown_connections 2024-04-02T13:28:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 --2- 172.21.15.44:0/2504100702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559503243c00 0x5595033b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 --2- 172.21.15.44:0/2504100702 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559503243800 0x5595033b2580 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 --2- 172.21.15.44:0/2504100702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559503242c00 0x5595033b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 -- 172.21.15.44:0/2504100702 >> 172.21.15.44:0/2504100702 conn(0x559503243000 msgr2=0x559503326800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 -- 172.21.15.44:0/2504100702 shutdown_connections 2024-04-02T13:28:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.400+0000 7f421a431740 1 -- 172.21.15.44:0/2504100702 wait complete. 2024-04-02T13:28:15.409 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', 'cephfs_data'] in time 0:00:00.089336, result: 2024-04-02T13:28:15.409 DEBUG:teuthology.orchestra.run.smithi044:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes cephfs_data 2024-04-02T13:28:15.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.485+0000 7fc777c39740 1 -- start start 2024-04-02T13:28:15.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.485+0000 7fc777c39740 1 Processor -- start 2024-04-02T13:28:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.485+0000 7fc777c39740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce34a2c00 0x558ce4272000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.485+0000 7fc777c39740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce34a3800 0x558ce4272580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.485+0000 7fc777c39740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce34a3c00 0x558ce4272b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.485+0000 7fc777c39740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558ce4364180 con 0x558ce34a3c00 2024-04-02T13:28:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.485+0000 7fc777c39740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558ce4364300 con 0x558ce34a2c00 2024-04-02T13:28:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.485+0000 7fc777c39740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x558ce4364480 con 0x558ce34a3800 2024-04-02T13:28:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.485+0000 7fc7764d1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce34a3c00 0x558ce4272b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7764d1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce34a3c00 0x558ce4272b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34644/0 (socket says 172.21.15.44:34644) 2024-04-02T13:28:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7764d1640 1 -- 172.21.15.44:0/2129546525 learned_addr learned my addr 172.21.15.44:0/2129546525 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7754cf640 1 --2- 172.21.15.44:0/2129546525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce34a3800 0x558ce4272580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc775cd0640 1 --2- 172.21.15.44:0/2129546525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce34a2c00 0x558ce4272000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7764d1640 1 -- 172.21.15.44:0/2129546525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce34a3800 msgr2=0x558ce4272580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7764d1640 1 --2- 172.21.15.44:0/2129546525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce34a3800 0x558ce4272580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7764d1640 1 -- 172.21.15.44:0/2129546525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce34a2c00 msgr2=0x558ce4272000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7764d1640 1 --2- 172.21.15.44:0/2129546525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce34a2c00 0x558ce4272000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7764d1640 1 -- 172.21.15.44:0/2129546525 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ce34cf680 con 0x558ce34a3c00 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7764d1640 1 --2- 172.21.15.44:0/2129546525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce34a3c00 0x558ce4272b00 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x558ce42c1920 tx=0x558ce423ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e7012b7e62f69d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7734cb640 1 -- 172.21.15.44:0/2129546525 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558ce34c8680 con 0x558ce34a3c00 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7734cb640 1 -- 172.21.15.44:0/2129546525 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x558ce34ec540 con 0x558ce34a3c00 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7734cb640 1 -- 172.21.15.44:0/2129546525 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558ce34c8b60 con 0x558ce34a3c00 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc777c39740 1 -- 172.21.15.44:0/2129546525 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ce34cf4a0 con 0x558ce34a3c00 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc777c39740 1 -- 172.21.15.44:0/2129546525 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=0}) -- 0x558ce34cfa40 con 0x558ce34a3c00 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc777c39740 4 waiting for MDS map... 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7734cb640 1 -- 172.21.15.44:0/2129546525 <== mon.0 v2:172.21.15.44:3300/0 4 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x558ce4382a00 con 0x558ce34a3c00 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc7734cb640 1 -- 172.21.15.44:0/2129546525 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 69) ==== 2885+0+0 (secure 0 0 0) 0x558ce34fb180 con 0x558ce34a3c00 2024-04-02T13:28:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.486+0000 7fc777c39740 4 Got MDS map 69 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.487+0000 7fc777c39740 4 datascan.main: Using metadata pool output 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.487+0000 7fc777c39740 4 datascan.main: connecting to RADOS... 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.487+0000 7fc777c39740 1 Processor -- start 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.487+0000 7fc777c39740 1 -- start start 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc777c39740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436f400 0x558ce4273080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc777c39740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce436f800 0x558ce4273600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc777c39740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce436fc00 0x558ce4273b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc777c39740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558ce4364600 con 0x558ce436f400 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc777c39740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558ce4364780 con 0x558ce436f800 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc777c39740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x558ce4364900 con 0x558ce436fc00 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc7754cf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436f400 0x558ce4273080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc7754cf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436f400 0x558ce4273080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34658/0 (socket says 172.21.15.44:34658) 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc7754cf640 1 -- 172.21.15.44:0/3378987942 learned_addr learned my addr 172.21.15.44:0/3378987942 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc775cd0640 1 --2- 172.21.15.44:0/3378987942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce436f800 0x558ce4273600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc7754cf640 1 --2- 172.21.15.44:0/3378987942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce436fc00 0x558ce4273b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc7754cf640 1 -- 172.21.15.44:0/3378987942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce436fc00 msgr2=0x558ce4273b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc7754cf640 1 --2- 172.21.15.44:0/3378987942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce436fc00 0x558ce4273b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc7754cf640 1 -- 172.21.15.44:0/3378987942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce436f800 msgr2=0x558ce4273600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc7754cf640 1 --2- 172.21.15.44:0/3378987942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce436f800 0x558ce4273600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc7754cf640 1 -- 172.21.15.44:0/3378987942 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ce34cf860 con 0x558ce436f400 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc775cd0640 1 --2- 172.21.15.44:0/3378987942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce436f800 0x558ce4273600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc7754cf640 1 --2- 172.21.15.44:0/3378987942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436f400 0x558ce4273080 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x558ce4394d20 tx=0x558ce423df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f3361934fc910cb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc7714c7640 1 -- 172.21.15.44:0/3378987942 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558ce34c89c0 con 0x558ce436f400 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc7714c7640 1 -- 172.21.15.44:0/3378987942 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x558ce34ece00 con 0x558ce436f400 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.488+0000 7fc7714c7640 1 -- 172.21.15.44:0/3378987942 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558ce34c8680 con 0x558ce436f400 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.489+0000 7fc777c39740 1 -- 172.21.15.44:0/3378987942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436f400 msgr2=0x558ce4273080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.489+0000 7fc777c39740 1 --2- 172.21.15.44:0/3378987942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436f400 0x558ce4273080 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x558ce4394d20 tx=0x558ce423df80 comp rx=0 tx=0).stop 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.489+0000 7fc777c39740 1 -- 172.21.15.44:0/3378987942 shutdown_connections 2024-04-02T13:28:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.489+0000 7fc777c39740 1 --2- 172.21.15.44:0/3378987942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce436fc00 0x558ce4273b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.489+0000 7fc777c39740 1 --2- 172.21.15.44:0/3378987942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce436f800 0x558ce4273600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.489+0000 7fc777c39740 1 --2- 172.21.15.44:0/3378987942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436f400 0x558ce4273080 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.489+0000 7fc777c39740 1 -- 172.21.15.44:0/3378987942 >> 172.21.15.44:0/3378987942 conn(0x558ce436f000 msgr2=0x558ce3587000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.489+0000 7fc777c39740 1 -- 172.21.15.44:0/3378987942 shutdown_connections 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.489+0000 7fc777c39740 1 -- 172.21.15.44:0/3378987942 wait complete. 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.489+0000 7fc777c39740 1 Processor -- start 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.489+0000 7fc777c39740 1 -- start start 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc777c39740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce436f400 0x558ce4273080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc777c39740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce436f800 0x558ce4273600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc777c39740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436fc00 0x558ce4273b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc777c39740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558ce4365980 con 0x558ce436fc00 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc777c39740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558ce4365800 con 0x558ce436f800 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc777c39740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x558ce4365680 con 0x558ce436f400 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc7754cf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436fc00 0x558ce4273b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc7754cf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436fc00 0x558ce4273b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34670/0 (socket says 172.21.15.44:34670) 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 learned_addr learned my addr 172.21.15.44:0/636479158 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc775cd0640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce436f800 0x558ce4273600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc7754cf640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce436f400 0x558ce4273080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce436f400 msgr2=0x558ce4273080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc7754cf640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce436f400 0x558ce4273080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce436f800 msgr2=0x558ce4273600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc7754cf640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce436f800 0x558ce4273600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ce34cf680 con 0x558ce436fc00 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.490+0000 7fc7754cf640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436fc00 0x558ce4273b80 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x558ce4395050 tx=0x558ce423c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.491+0000 7fc76fcc4640 1 -- 172.21.15.44:0/636479158 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558ce34c8b60 con 0x558ce436fc00 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.491+0000 7fc76fcc4640 1 -- 172.21.15.44:0/636479158 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x558ce34ecc40 con 0x558ce436fc00 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.491+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558ce34cf4a0 con 0x558ce436fc00 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.491+0000 7fc76fcc4640 1 -- 172.21.15.44:0/636479158 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558ce34c84e0 con 0x558ce436fc00 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.491+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ce34cfa40 con 0x558ce436fc00 2024-04-02T13:28:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.491+0000 7fc777c39740 4 datascan.init: resolving metadata pool 13 2024-04-02T13:28:15.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.492+0000 7fc76fcc4640 1 -- 172.21.15.44:0/636479158 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x558ce43eec00 con 0x558ce436fc00 2024-04-02T13:28:15.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.492+0000 7fc76fcc4640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x558ce3822000 0x558ce4274100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.492+0000 7fc76fcc4640 1 -- 172.21.15.44:0/636479158 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x558ce42a8600 con 0x558ce436fc00 2024-04-02T13:28:15.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.492+0000 7fc777c39740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-04-02T13:28:15.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.492+0000 7fc777c39740 4 datascan.main: data pool 'cephfs_data' has ID 14 2024-04-02T13:28:15.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.492+0000 7fc777c39740 4 datascan.main: opening data pool 'cephfs_data' 2024-04-02T13:28:15.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.492+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x558ce3822400 0x558ce4274680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.492+0000 7fc775cd0640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x558ce3822000 0x558ce4274100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.492+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:1 13.14 13:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x558ce3822800 con 0x558ce3822400 2024-04-02T13:28:15.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.492+0000 7fc7754cf640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x558ce3822400 0x558ce4274680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.493+0000 7fc7754cf640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x558ce3822400 0x558ce4274680 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.493+0000 7fc775cd0640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x558ce3822000 0x558ce4274100 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x558ce422ade0 tx=0x558ce37b4a80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.493+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv24 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.493+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x558ce3822c00 0x558ce4274c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.493+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:2 13.1f 13:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x558ce3823000 con 0x558ce3822c00 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.493+0000 7fc7764d1640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x558ce3822c00 0x558ce4274c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.494+0000 7fc7764d1640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x558ce3822c00 0x558ce4274c00 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.494+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) ==== 162+0+16 (crc 0 0 0) 0x558ce4224500 con 0x558ce3822c00 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.494+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x558ce3823400 0x558ce4275180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.494+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:3 14.0 14:00000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383e000 con 0x558ce3823400 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.494+0000 7fc775cd0640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x558ce3823400 0x558ce4275180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.494+0000 7fc775cd0640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x558ce3823400 0x558ce4275180 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.495+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.495+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:4 14.20 14:04000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383f800 con 0x558ce3822400 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.495+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 2 ==== osd_op_reply(4 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.495+0000 7fc7714c7640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x558ce383fc00 0x558ce384e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.495+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:5 14.10 14:08000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3852000 con 0x558ce383fc00 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.495+0000 7fc7754cf640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x558ce383fc00 0x558ce384e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.495+0000 7fc7754cf640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x558ce383fc00 0x558ce384e000 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.496+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(5 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.496+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:6 14.30 14:0c000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3852800 con 0x558ce3822400 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.496+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 3 ==== osd_op_reply(6 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.496+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:7 14.8 14:10000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3852400 con 0x558ce3822400 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.496+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 4 ==== osd_op_reply(7 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.496+0000 7fc770cc6640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x558ce3852c00 0x558ce384e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.496+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:8 14.28 14:14000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383f800 con 0x558ce3852c00 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.496+0000 7fc7764d1640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x558ce3852c00 0x558ce384e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.496+0000 7fc7764d1640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x558ce3852c00 0x558ce384e580 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.497+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.3 v2:172.21.15.44:6824/1588952513 1 ==== osd_op_reply(8 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3852c00 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.497+0000 7fc7714c7640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x558ce3852000 0x558ce384eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.497+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:9 14.18 14:18000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3852800 con 0x558ce3852000 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.497+0000 7fc775cd0640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x558ce3852000 0x558ce384eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.497+0000 7fc775cd0640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x558ce3852000 0x558ce384eb00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.498+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.7 v2:172.21.15.69:6824/1292475712 1 ==== osd_op_reply(9 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3852000 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.498+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:10 14.38 14:1c000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3822800 con 0x558ce383fc00 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.498+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 2 ==== osd_op_reply(10 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.498+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:11 14.4 14:20000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853000 con 0x558ce3822c00 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.498+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.6 v2:172.21.15.69:6802/2604158175 2 ==== osd_op_reply(11 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3822c00 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.498+0000 7fc770cc6640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x558ce4283400 0x558ce384f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.498+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:12 14.24 14:24000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853400 con 0x558ce4283400 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.498+0000 7fc7754cf640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x558ce4283400 0x558ce384f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.499+0000 7fc7754cf640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x558ce4283400 0x558ce384f600 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.499+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 1 ==== osd_op_reply(12 [pgnls-filter start_epoch 95 out=49b] v91'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.499+0000 7fc7714c7640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x558ce3862000 0x558ce384f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.499+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:13 14.14 14:28000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862400 con 0x558ce3862000 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.499+0000 7fc7764d1640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x558ce3862000 0x558ce384f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.500+0000 7fc7764d1640 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x558ce3862000 0x558ce384f080 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.500+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 1 ==== osd_op_reply(13 [pgnls-filter start_epoch 95 out=49b] v91'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.500+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:14 14.34 14:2c000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853800 con 0x558ce3823400 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.500+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 2 ==== osd_op_reply(14 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.500+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:15 14.c 14:30000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862800 con 0x558ce3822400 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.500+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 5 ==== osd_op_reply(15 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.500+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:16 14.2c 14:34000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853c00 con 0x558ce3823400 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.501+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 3 ==== osd_op_reply(16 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.501+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:17 14.1c 14:38000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862c00 con 0x558ce3822400 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.501+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 6 ==== osd_op_reply(17 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.501+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:18 14.3c 14:3c000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3852800 con 0x558ce383fc00 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.501+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 3 ==== osd_op_reply(18 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.501+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:19 14.2 14:40000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3852400 con 0x558ce3822c00 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.502+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.6 v2:172.21.15.69:6802/2604158175 3 ==== osd_op_reply(19 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3822c00 2024-04-02T13:28:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.502+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:20 14.22 14:44000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383e000 con 0x558ce3862000 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.502+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 2 ==== osd_op_reply(20 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.502+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:21 14.12 14:48000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853400 con 0x558ce3823400 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.502+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 4 ==== osd_op_reply(21 [pgnls-filter start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.502+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:22 14.32 14:4c000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383f800 con 0x558ce3862000 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.502+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 3 ==== osd_op_reply(22 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.502+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:23 14.a 14:50000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3852800 con 0x558ce4283400 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.503+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 2 ==== osd_op_reply(23 [pgnls-filter start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.503+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:24 14.2a 14:54000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3823000 con 0x558ce3862000 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.503+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 4 ==== osd_op_reply(24 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.504+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:25 14.1a 14:58000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853800 con 0x558ce3822400 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.504+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 7 ==== osd_op_reply(25 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.504+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:26 14.3a 14:5c000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce436e800 con 0x558ce3822400 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.504+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 8 ==== osd_op_reply(26 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.504+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:27 14.6 14:60000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3822800 con 0x558ce3822400 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.505+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 9 ==== osd_op_reply(27 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.505+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:28 14.26 14:64000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862800 con 0x558ce3862000 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.505+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 5 ==== osd_op_reply(28 [pgnls-filter start_epoch 95 out=49b] v91'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.505+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:29 14.16 14:68000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863400 con 0x558ce3852c00 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.505+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.3 v2:172.21.15.44:6824/1588952513 2 ==== osd_op_reply(29 [pgnls-filter start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3852c00 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.505+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:30 14.36 14:6c000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862c00 con 0x558ce3823400 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.506+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 5 ==== osd_op_reply(30 [pgnls-filter start_epoch 95 out=49b] v91'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.506+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:31 14.e 14:70000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863800 con 0x558ce3862000 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.506+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 6 ==== osd_op_reply(31 [pgnls-filter start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.506+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:32 14.2e 14:74000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862400 con 0x558ce3852c00 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.506+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.3 v2:172.21.15.44:6824/1588952513 3 ==== osd_op_reply(32 [pgnls-filter start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3852c00 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.506+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:33 14.1e 14:78000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863c00 con 0x558ce3852000 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.507+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.7 v2:172.21.15.69:6824/1292475712 2 ==== osd_op_reply(33 [pgnls-filter start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3852000 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.507+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:34 14.3e 14:7c000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863000 con 0x558ce4283400 2024-04-02T13:28:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.507+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 3 ==== osd_op_reply(34 [pgnls-filter start_epoch 95 out=49b] v91'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.507+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:35 14.1 14:80000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862800 con 0x558ce3852c00 2024-04-02T13:28:15.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.507+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.3 v2:172.21.15.44:6824/1588952513 4 ==== osd_op_reply(35 [pgnls-filter start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3852c00 2024-04-02T13:28:15.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.507+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:36 14.21 14:84000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853c00 con 0x558ce4283400 2024-04-02T13:28:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.508+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 4 ==== osd_op_reply(36 [pgnls-filter start_epoch 95 out=49b] v91'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.508+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:37 14.11 14:88000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863400 con 0x558ce3862000 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.508+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 7 ==== osd_op_reply(37 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.508+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:38 14.31 14:8c000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853000 con 0x558ce383fc00 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.508+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 4 ==== osd_op_reply(38 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.508+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:39 14.9 14:90000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3823000 con 0x558ce4283400 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.509+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 5 ==== osd_op_reply(39 [pgnls-filter start_epoch 95 out=49b] v91'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.509+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:40 14.29 14:94000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3852800 con 0x558ce3822c00 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.509+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.6 v2:172.21.15.69:6802/2604158175 4 ==== osd_op_reply(40 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3822c00 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.509+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:41 14.19 14:98000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383e000 con 0x558ce3822c00 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.509+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.6 v2:172.21.15.69:6802/2604158175 5 ==== osd_op_reply(41 [pgnls-filter start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3822c00 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.509+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:42 14.39 14:9c000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853800 con 0x558ce3852000 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.510+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.7 v2:172.21.15.69:6824/1292475712 3 ==== osd_op_reply(42 [pgnls-filter start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3852000 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.510+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:43 14.5 14:a0000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383f800 con 0x558ce3822400 2024-04-02T13:28:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.510+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 10 ==== osd_op_reply(43 [pgnls-filter start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.510+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:44 14.25 14:a4000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853c00 con 0x558ce3823400 2024-04-02T13:28:15.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.510+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 6 ==== osd_op_reply(44 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.510+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:45 14.15 14:a8000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3852400 con 0x558ce3852000 2024-04-02T13:28:15.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.511+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.7 v2:172.21.15.69:6824/1292475712 4 ==== osd_op_reply(45 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3852000 2024-04-02T13:28:15.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.511+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:46 14.35 14:ac000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853000 con 0x558ce3862000 2024-04-02T13:28:15.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.511+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 8 ==== osd_op_reply(46 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.511+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:47 14.d 14:b0000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3852800 con 0x558ce3822c00 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.511+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.6 v2:172.21.15.69:6802/2604158175 6 ==== osd_op_reply(47 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3822c00 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.512+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:48 14.2d 14:b4000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863000 con 0x558ce3862000 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.512+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 9 ==== osd_op_reply(48 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.512+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:49 14.1d 14:b8000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863800 con 0x558ce3822400 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.512+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 11 ==== osd_op_reply(49 [pgnls-filter start_epoch 95 out=49b] v91'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.512+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:50 14.3d 14:bc000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3822800 con 0x558ce3852c00 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.513+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.3 v2:172.21.15.44:6824/1588952513 5 ==== osd_op_reply(50 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3852c00 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.513+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:51 14.3 14:c0000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862400 con 0x558ce3822400 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.513+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 12 ==== osd_op_reply(51 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.513+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:52 14.23 14:c4000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce436e800 con 0x558ce3852000 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.513+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.7 v2:172.21.15.69:6824/1292475712 5 ==== osd_op_reply(52 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3852000 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.513+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:53 14.13 14:c8000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862800 con 0x558ce3823400 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.514+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 7 ==== osd_op_reply(53 [pgnls-filter start_epoch 95 out=49b] v91'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.514+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:54 14.33 14:cc000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853000 con 0x558ce383fc00 2024-04-02T13:28:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.514+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 5 ==== osd_op_reply(54 [pgnls-filter start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.514+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:55 14.b 14:d0000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863400 con 0x558ce3822400 2024-04-02T13:28:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.514+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 13 ==== osd_op_reply(55 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.514+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:56 14.2b 14:d4000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853400 con 0x558ce3862000 2024-04-02T13:28:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.515+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 10 ==== osd_op_reply(56 [pgnls-filter start_epoch 95 out=49b] v91'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.515+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:57 14.1b 14:d8000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862c00 con 0x558ce383fc00 2024-04-02T13:28:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.515+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 6 ==== osd_op_reply(57 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.515+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:58 14.3b 14:dc000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853800 con 0x558ce3862000 2024-04-02T13:28:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.515+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 11 ==== osd_op_reply(58 [pgnls-filter start_epoch 95 out=49b] v91'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.515+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:59 14.7 14:e0000000::::head [pgnls-filter start_epoch 95 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863c00 con 0x558ce4283400 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.516+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 6 ==== osd_op_reply(59 [pgnls-filter start_epoch 95 out=81b] v95'7 uv7 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.516+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:60 14.0 14:00000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383e400 con 0x558ce3823400 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.516+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 8 ==== osd_op_reply(60 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.516+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:61 14.20 14:04000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383e000 con 0x558ce3822400 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.516+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 14 ==== osd_op_reply(61 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.516+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:62 14.10 14:08000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853c00 con 0x558ce383fc00 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.516+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 7 ==== osd_op_reply(62 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.516+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:63 14.30 14:0c000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863800 con 0x558ce3822400 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.517+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 15 ==== osd_op_reply(63 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.517+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:64 14.8 14:10000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386a000 con 0x558ce3822400 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.517+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 16 ==== osd_op_reply(64 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.517+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:65 14.28 14:14000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862400 con 0x558ce3852c00 2024-04-02T13:28:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.517+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.3 v2:172.21.15.44:6824/1588952513 6 ==== osd_op_reply(65 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3852c00 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.517+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:66 14.18 14:18000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386a400 con 0x558ce3852000 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.518+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.7 v2:172.21.15.69:6824/1292475712 6 ==== osd_op_reply(66 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3852000 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.518+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:67 14.38 14:1c000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863400 con 0x558ce383fc00 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.518+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 8 ==== osd_op_reply(67 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.518+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:68 14.4 14:20000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386a800 con 0x558ce3822c00 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.518+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.6 v2:172.21.15.69:6802/2604158175 7 ==== osd_op_reply(68 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3822c00 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.518+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:69 14.24 14:24000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862c00 con 0x558ce4283400 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.519+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 7 ==== osd_op_reply(69 [pgnls-filter start_epoch 95 out=49b] v91'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.519+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:70 14.14 14:28000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386ac00 con 0x558ce3862000 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.519+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 12 ==== osd_op_reply(70 [pgnls-filter start_epoch 95 out=49b] v91'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.519+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:71 14.34 14:2c000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863c00 con 0x558ce3823400 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.519+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 9 ==== osd_op_reply(71 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.520+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:72 14.c 14:30000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386b000 con 0x558ce3822400 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.520+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 17 ==== osd_op_reply(72 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.520+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:73 14.2c 14:34000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853000 con 0x558ce3823400 2024-04-02T13:28:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.520+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 10 ==== osd_op_reply(73 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.520+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:74 14.1c 14:38000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386b400 con 0x558ce3822400 2024-04-02T13:28:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.520+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 18 ==== osd_op_reply(74 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.520+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:75 14.3c 14:3c000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383f800 con 0x558ce383fc00 2024-04-02T13:28:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.520+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 9 ==== osd_op_reply(75 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.520+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:76 14.2 14:40000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386b800 con 0x558ce3822c00 2024-04-02T13:28:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.521+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.6 v2:172.21.15.69:6802/2604158175 8 ==== osd_op_reply(76 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3822c00 2024-04-02T13:28:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.521+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:77 14.22 14:44000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3823000 con 0x558ce3862000 2024-04-02T13:28:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.521+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 13 ==== osd_op_reply(77 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.522+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:78 14.12 14:48000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386bc00 con 0x558ce3823400 2024-04-02T13:28:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.522+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 11 ==== osd_op_reply(78 [pgnls-filter start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.522+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:79 14.32 14:4c000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853c00 con 0x558ce3862000 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.522+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 14 ==== osd_op_reply(79 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.522+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:80 14.a 14:50000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386a000 con 0x558ce4283400 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.522+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 8 ==== osd_op_reply(80 [pgnls-filter start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.523+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:81 14.2a 14:54000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3852400 con 0x558ce3862000 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.523+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 15 ==== osd_op_reply(81 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.523+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:82 14.1a 14:58000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386b000 con 0x558ce3822400 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.523+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 19 ==== osd_op_reply(82 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.523+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:83 14.3a 14:5c000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3852800 con 0x558ce3822400 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.523+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 20 ==== osd_op_reply(83 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.523+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:84 14.6 14:60000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386b400 con 0x558ce3822400 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.524+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 21 ==== osd_op_reply(84 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.524+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:85 14.26 14:64000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853400 con 0x558ce3862000 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.524+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 16 ==== osd_op_reply(85 [pgnls-filter start_epoch 95 out=49b] v91'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.524+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:86 14.16 14:68000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863800 con 0x558ce3852c00 2024-04-02T13:28:15.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.524+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.3 v2:172.21.15.44:6824/1588952513 7 ==== osd_op_reply(86 [pgnls-filter start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3852c00 2024-04-02T13:28:15.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.524+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:87 14.36 14:6c000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853800 con 0x558ce3823400 2024-04-02T13:28:15.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.524+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 12 ==== osd_op_reply(87 [pgnls-filter start_epoch 95 out=49b] v91'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.524+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:88 14.e 14:70000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863400 con 0x558ce3862000 2024-04-02T13:28:15.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.525+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 17 ==== osd_op_reply(88 [pgnls-filter start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.525+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:89 14.2e 14:74000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3822800 con 0x558ce3852c00 2024-04-02T13:28:15.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.525+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.3 v2:172.21.15.44:6824/1588952513 8 ==== osd_op_reply(89 [pgnls-filter start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3852c00 2024-04-02T13:28:15.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.525+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:90 14.1e 14:78000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862c00 con 0x558ce3852000 2024-04-02T13:28:15.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.526+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.7 v2:172.21.15.69:6824/1292475712 7 ==== osd_op_reply(90 [pgnls-filter start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3852000 2024-04-02T13:28:15.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.526+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:91 14.3e 14:7c000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383e000 con 0x558ce4283400 2024-04-02T13:28:15.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.526+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 9 ==== osd_op_reply(91 [pgnls-filter start_epoch 95 out=49b] v91'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.526+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:92 14.1 14:80000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862800 con 0x558ce3852c00 2024-04-02T13:28:15.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.526+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.3 v2:172.21.15.44:6824/1588952513 9 ==== osd_op_reply(92 [pgnls-filter start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3852c00 2024-04-02T13:28:15.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.527+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:93 14.21 14:84000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383f800 con 0x558ce4283400 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.527+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 10 ==== osd_op_reply(93 [pgnls-filter start_epoch 95 out=49b] v91'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.527+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:94 14.11 14:88000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863c00 con 0x558ce3862000 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.527+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 18 ==== osd_op_reply(94 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.527+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:95 14.31 14:8c000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383e400 con 0x558ce383fc00 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.527+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 10 ==== osd_op_reply(95 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.527+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:96 14.9 14:90000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863000 con 0x558ce4283400 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.528+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 11 ==== osd_op_reply(96 [pgnls-filter start_epoch 95 out=49b] v91'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.528+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:97 14.29 14:94000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce436e800 con 0x558ce3822c00 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.528+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.6 v2:172.21.15.69:6802/2604158175 9 ==== osd_op_reply(97 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3822c00 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.528+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:98 14.19 14:98000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862400 con 0x558ce3822c00 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.529+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.6 v2:172.21.15.69:6802/2604158175 10 ==== osd_op_reply(98 [pgnls-filter start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3822c00 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.529+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:99 14.39 14:9c000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386b800 con 0x558ce3852000 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.529+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.7 v2:172.21.15.69:6824/1292475712 8 ==== osd_op_reply(99 [pgnls-filter start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3852000 2024-04-02T13:28:15.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.529+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:100 14.5 14:a0000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386ac00 con 0x558ce3822400 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.530+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 22 ==== osd_op_reply(100 [pgnls-filter start_epoch 95 out=49b] v91'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.530+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:101 14.25 14:a4000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386a400 con 0x558ce3823400 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.530+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 13 ==== osd_op_reply(101 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.530+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:102 14.15 14:a8000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3874000 con 0x558ce3852000 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.530+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.7 v2:172.21.15.69:6824/1292475712 9 ==== osd_op_reply(102 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3852000 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.530+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:103 14.35 14:ac000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce386a800 con 0x558ce3862000 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.531+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 19 ==== osd_op_reply(103 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.531+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:104 14.d 14:b0000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3874400 con 0x558ce3822c00 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.531+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.6 v2:172.21.15.69:6802/2604158175 11 ==== osd_op_reply(104 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3822c00 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.531+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:105 14.2d 14:b4000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863800 con 0x558ce3862000 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.531+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 20 ==== osd_op_reply(105 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.531+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:106 14.1d 14:b8000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3874800 con 0x558ce3822400 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.532+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 23 ==== osd_op_reply(106 [pgnls-filter start_epoch 95 out=49b] v91'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.532+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:107 14.3d 14:bc000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863400 con 0x558ce3852c00 2024-04-02T13:28:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.532+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.3 v2:172.21.15.44:6824/1588952513 10 ==== osd_op_reply(107 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3852c00 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.532+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:108 14.3 14:c0000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3874c00 con 0x558ce3822400 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.532+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 24 ==== osd_op_reply(108 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.532+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:109 14.23 14:c4000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862800 con 0x558ce3852000 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.532+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.7 v2:172.21.15.69:6824/1292475712 10 ==== osd_op_reply(109 [pgnls-filter start_epoch 95 out=49b] v91'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3852000 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.532+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:110 14.13 14:c8000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3875000 con 0x558ce3823400 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.533+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.2 v2:172.21.15.44:6816/2091834331 14 ==== osd_op_reply(110 [pgnls-filter start_epoch 95 out=49b] v91'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3823400 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.533+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:111 14.33 14:cc000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3823000 con 0x558ce383fc00 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.533+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 11 ==== osd_op_reply(111 [pgnls-filter start_epoch 95 out=49b] v91'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.533+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:112 14.b 14:d0000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3875400 con 0x558ce3822400 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.533+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 25 ==== osd_op_reply(112 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.533+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:113 14.2b 14:d4000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3822800 con 0x558ce3862000 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.533+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 21 ==== osd_op_reply(113 [pgnls-filter start_epoch 95 out=49b] v91'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.533+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:114 14.1b 14:d8000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3875800 con 0x558ce383fc00 2024-04-02T13:28:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.534+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 12 ==== osd_op_reply(114 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.534+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:115 14.3b 14:dc000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853c00 con 0x558ce3862000 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.534+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 22 ==== osd_op_reply(115 [pgnls-filter start_epoch 95 out=49b] v91'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.534+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:116 14.7 14:e0000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3875c00 con 0x558ce4283400 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.534+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 12 ==== osd_op_reply(116 [pgnls-filter start_epoch 95 out=81b] v95'7 uv7 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.534+0000 7fc777c39740 20 datascan.forall_objects: OSD matched oid 10000000001.00000000 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.534+0000 7fc777c39740 10 datascan.operator(): handling object 10000000001.0 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.534+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:117 14.7 14:e2e07930:::10000000001.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x558ce383ec00 con 0x558ce4283400 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.535+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 13 ==== osd_op_reply(117 10000000001.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (89) out=89b,getxattr (30) out=30b,getxattr] v0'0 uv7 ondisk = 0) ==== 416+0+157 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.535+0000 7fc777c39740 10 datascan.inject_with_backtrace: inode: 0x0x10000000001 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.535+0000 7fc777c39740 10 datascan.inject_with_backtrace: backptr: 0x0x10000000000/file 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.535+0000 7fc777c39740 20 datascan.get_frag_of: dirino=0x10000000000 target_dname=file 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.535+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:118 13.10 13:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x558ce383f000 con 0x558ce383fc00 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.535+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 13 ==== osd_op_reply(118 10000000000.00000000 [getxattr (59) out=59b] v0'0 uv17 ondisk = 0) ==== 164+0+59 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.535+0000 7fc777c39740 20 datascan.get_frag_of: got backtrace parent 0x1/dir 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.535+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:119 13.1f 13:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x558ce383f400 con 0x558ce3822c00 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.535+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.6 v2:172.21.15.69:6802/2604158175 12 ==== osd_op_reply(119 1.00000000 [omap-get-vals-by-keys out=520b] v0'0 uv9 ondisk = 0) ==== 154+0+520 (crc 0 0 0) 0x558ce4224500 con 0x558ce3822c00 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.535+0000 7fc777c39740 20 datascan.read_dentry: oid=1.00000000 dname=dir frag=*, r=0 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.535+0000 7fc777c39740 20 datascan.get_frag_of: fast path, fragtree is fragtree_t(*) 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.535+0000 7fc777c39740 20 datascan.get_frag_of: frag is * 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.535+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:120 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853c00 con 0x558ce383fc00 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.536+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 14 ==== osd_op_reply(120 10000000000.00000000 [omap-get-header out=274b] v0'0 uv17 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.536+0000 7fc777c39740 10 datascan.find_or_create_dirfrag: read_version = 17 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.536+0000 7fc777c39740 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000000 * 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.536+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:121 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x558ce383f400 con 0x558ce383fc00 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.536+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 15 ==== osd_op_reply(121 10000000000.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv17 ondisk = 0) ==== 164+0+4 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.536+0000 7fc777c39740 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.536+0000 7fc777c39740 20 datascan.read_dentry: file_head not found in result 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.536+0000 7fc777c39740 10 datascan.inject_with_backtrace: Linking inode 0x0x10000000001 at 0x0x10000000000/file with size=0 bytes 2024-04-02T13:28:15.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.536+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:122 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x558ce383e000 con 0x558ce383fc00 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.537+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 16 ==== osd_op_reply(122 10000000000.00000000 [omap-set-vals] v95'18 uv18 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.537+0000 7fc777c39740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.537+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:123 14.27 14:e4000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce383f800 con 0x558ce4283400 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.538+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.5 v2:172.21.15.69:6800/2841767499 14 ==== osd_op_reply(123 [pgnls-filter start_epoch 95 out=49b] v91'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce4283400 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.538+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:124 14.17 14:e8000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863c00 con 0x558ce383fc00 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.538+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.1 v2:172.21.15.44:6801/704457846 17 ==== osd_op_reply(124 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce383fc00 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.538+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:125 14.37 14:ec000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3852400 con 0x558ce3862000 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.538+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.4 v2:172.21.15.69:6816/1041350007 23 ==== osd_op_reply(125 [pgnls-filter start_epoch 95 out=49b] v91'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3862000 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.538+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:126 14.f 14:f0000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3862400 con 0x558ce3852000 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.539+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 <== osd.7 v2:172.21.15.69:6824/1292475712 11 ==== osd_op_reply(126 [pgnls-filter start_epoch 95 out=49b] v91'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224780 con 0x558ce3852000 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.539+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:127 14.2f 14:f4000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3853400 con 0x558ce3822400 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.539+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 26 ==== osd_op_reply(127 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.539+0000 7fc770cc6640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:128 14.1f 14:f8000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3863800 con 0x558ce3822400 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.539+0000 7fc7754cf640 1 -- 172.21.15.44:0/636479158 <== osd.0 v2:172.21.15.44:6800/2303805525 27 ==== osd_op_reply(128 [pgnls-filter start_epoch 95 out=49b] v91'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224000 con 0x558ce3822400 2024-04-02T13:28:15.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc7714c7640 1 -- 172.21.15.44:0/636479158 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:129 14.3f 14:fc000000::::head [pgnls-filter start_epoch 95 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x558ce3874c00 con 0x558ce3852c00 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc7764d1640 1 -- 172.21.15.44:0/636479158 <== osd.3 v2:172.21.15.44:6824/1588952513 11 ==== osd_op_reply(129 [pgnls-filter start_epoch 95 out=49b] v91'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x558ce4224500 con 0x558ce3852c00 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x558ce3822400 msgr2=0x558ce4274680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x558ce3822400 0x558ce4274680 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x558ce383fc00 msgr2=0x558ce384e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x558ce383fc00 0x558ce384e000 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x558ce3823400 msgr2=0x558ce4275180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x558ce3823400 0x558ce4275180 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x558ce3852c00 msgr2=0x558ce384e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x558ce3852c00 0x558ce384e580 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 reap_dead start 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x558ce3862000 msgr2=0x558ce384f080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x558ce3862000 0x558ce384f080 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x558ce4283400 msgr2=0x558ce384f600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x558ce4283400 0x558ce384f600 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x558ce3822c00 msgr2=0x558ce4274c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x558ce3822c00 0x558ce4274c00 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x558ce3852000 msgr2=0x558ce384eb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x558ce3852000 0x558ce384eb00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc775cd0640 1 -- 172.21.15.44:0/636479158 reap_dead start 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x558ce3822000 msgr2=0x558ce4274100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x558ce3822000 0x558ce4274100 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x558ce422ade0 tx=0x558ce37b4a80 comp rx=0 tx=0).stop 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436fc00 msgr2=0x558ce4273b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436fc00 0x558ce4273b80 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x558ce4395050 tx=0x558ce423c900 comp rx=0 tx=0).stop 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 shutdown_connections 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x558ce3822000 0x558ce4274100 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 --2- 172.21.15.44:0/636479158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce436fc00 0x558ce4273b80 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 >> 172.21.15.44:0/636479158 conn(0x558ce436f000 msgr2=0x558ce3587000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 shutdown_connections 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/636479158 wait complete. 2024-04-02T13:28:15.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 -- 172.21.15.44:0/2129546525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce34a3c00 msgr2=0x558ce4272b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.540+0000 7fc777c39740 1 --2- 172.21.15.44:0/2129546525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce34a3c00 0x558ce4272b00 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x558ce42c1920 tx=0x558ce423ea20 comp rx=0 tx=0).stop 2024-04-02T13:28:15.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.541+0000 7fc777c39740 1 -- 172.21.15.44:0/2129546525 shutdown_connections 2024-04-02T13:28:15.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.541+0000 7fc777c39740 1 --2- 172.21.15.44:0/2129546525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ce34a3c00 0x558ce4272b00 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.541+0000 7fc777c39740 1 --2- 172.21.15.44:0/2129546525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558ce34a3800 0x558ce4272580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.541+0000 7fc777c39740 1 --2- 172.21.15.44:0/2129546525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558ce34a2c00 0x558ce4272000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.541+0000 7fc777c39740 1 -- 172.21.15.44:0/2129546525 >> 172.21.15.44:0/2129546525 conn(0x558ce34a3000 msgr2=0x558ce3586800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:15.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.541+0000 7fc777c39740 1 -- 172.21.15.44:0/2129546525 shutdown_connections 2024-04-02T13:28:15.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.541+0000 7fc777c39740 1 -- 172.21.15.44:0/2129546525 wait complete. 2024-04-02T13:28:15.556 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', 'cephfs_data'] in time 0:00:00.147188, result: 2024-04-02T13:28:15.556 DEBUG:teuthology.orchestra.run.smithi044:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links 2024-04-02T13:28:15.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.621+0000 7f45f9e0d740 1 -- start start 2024-04-02T13:28:15.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.621+0000 7f45f9e0d740 1 Processor -- start 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f9e0d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a800 0x5565815db080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f9e0d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658146fc00 0x5565815dab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f9e0d740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658224bc00 0x5565815da580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f9e0d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556582246780 con 0x55658224a800 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f9e0d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556582246600 con 0x55658146fc00 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f9e0d740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x556582246180 con 0x55658224bc00 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f7c76640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a800 0x5565815db080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f7c76640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a800 0x5565815db080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34686/0 (socket says 172.21.15.44:34686) 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f7c76640 1 -- 172.21.15.44:0/3891032402 learned_addr learned my addr 172.21.15.44:0/3891032402 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f8477640 1 --2- 172.21.15.44:0/3891032402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658224bc00 0x5565815da580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f7475640 1 --2- 172.21.15.44:0/3891032402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658146fc00 0x5565815dab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f7c76640 1 -- 172.21.15.44:0/3891032402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658224bc00 msgr2=0x5565815da580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f7c76640 1 --2- 172.21.15.44:0/3891032402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658224bc00 0x5565815da580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f7c76640 1 -- 172.21.15.44:0/3891032402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658146fc00 msgr2=0x5565815dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f7c76640 1 --2- 172.21.15.44:0/3891032402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658146fc00 0x5565815dab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f7c76640 1 -- 172.21.15.44:0/3891032402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55658149b680 con 0x55658224a800 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.622+0000 7f45f7475640 1 --2- 172.21.15.44:0/3891032402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658146fc00 0x5565815dab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.623+0000 7f45f7c76640 1 --2- 172.21.15.44:0/3891032402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a800 0x5565815db080 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x556582295b60 tx=0x55658220a0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3501e65f4b74089 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.623+0000 7f45f5471640 1 -- 172.21.15.44:0/3891032402 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556581494d00 con 0x55658224a800 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.623+0000 7f45f5471640 1 -- 172.21.15.44:0/3891032402 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5565814b8540 con 0x55658224a800 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.623+0000 7f45f5471640 1 -- 172.21.15.44:0/3891032402 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556581494680 con 0x55658224a800 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.623+0000 7f45f9e0d740 1 -- 172.21.15.44:0/3891032402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55658149b2c0 con 0x55658224a800 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.623+0000 7f45f9e0d740 1 -- 172.21.15.44:0/3891032402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55658149ab40 con 0x55658224a800 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.623+0000 7f45f9e0d740 4 waiting for MDS map... 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.623+0000 7f45f5471640 1 -- 172.21.15.44:0/3891032402 <== mon.0 v2:172.21.15.44:3300/0 4 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x55658227c600 con 0x55658224a800 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.623+0000 7f45f5471640 1 -- 172.21.15.44:0/3891032402 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 69) ==== 2885+0+0 (secure 0 0 0) 0x5565814c7180 con 0x55658224a800 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.623+0000 7f45f9e0d740 4 Got MDS map 69 2024-04-02T13:28:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.623+0000 7f45f9e0d740 4 datascan.main: Using metadata pool output 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.623+0000 7f45f9e0d740 4 datascan.main: connecting to RADOS... 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.624+0000 7f45f9e0d740 1 Processor -- start 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.624+0000 7f45f9e0d740 1 -- start start 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.624+0000 7f45f9e0d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a000 0x5565815db600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.624+0000 7f45f9e0d740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658233b800 0x5565815dbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.624+0000 7f45f9e0d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658233bc00 0x5565815dc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.624+0000 7f45f9e0d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5565814ce480 con 0x55658224a000 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.624+0000 7f45f9e0d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5565814ce900 con 0x55658233bc00 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.624+0000 7f45f9e0d740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x556582247200 con 0x55658233b800 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.624+0000 7f45f8477640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a000 0x5565815db600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.624+0000 7f45f8477640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a000 0x5565815db600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34696/0 (socket says 172.21.15.44:34696) 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.624+0000 7f45f8477640 1 -- 172.21.15.44:0/3687788171 learned_addr learned my addr 172.21.15.44:0/3687788171 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.624+0000 7f45f7475640 1 --2- 172.21.15.44:0/3687788171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658233bc00 0x5565815dc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.624+0000 7f45f7c76640 1 --2- 172.21.15.44:0/3687788171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658233b800 0x5565815dbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f8477640 1 -- 172.21.15.44:0/3687788171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658233b800 msgr2=0x5565815dbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f8477640 1 --2- 172.21.15.44:0/3687788171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658233b800 0x5565815dbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f8477640 1 -- 172.21.15.44:0/3687788171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658233bc00 msgr2=0x5565815dc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f8477640 1 --2- 172.21.15.44:0/3687788171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658233bc00 0x5565815dc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f8477640 1 -- 172.21.15.44:0/3687788171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55658149a3c0 con 0x55658224a000 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f8477640 1 --2- 172.21.15.44:0/3687788171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a000 0x5565815db600 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x5565821f6150 tx=0x556582208a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6376277d365e0f66 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f346d640 1 -- 172.21.15.44:0/3687788171 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556581494b60 con 0x55658224a000 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f346d640 1 -- 172.21.15.44:0/3687788171 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5565814b8c40 con 0x55658224a000 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f346d640 1 -- 172.21.15.44:0/3687788171 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556581494d00 con 0x55658224a000 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f9e0d740 1 -- 172.21.15.44:0/3687788171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a000 msgr2=0x5565815db600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/3687788171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a000 0x5565815db600 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x5565821f6150 tx=0x556582208a20 comp rx=0 tx=0).stop 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f9e0d740 1 -- 172.21.15.44:0/3687788171 shutdown_connections 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/3687788171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658233bc00 0x5565815dc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/3687788171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658233b800 0x5565815dbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/3687788171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a000 0x5565815db600 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f9e0d740 1 -- 172.21.15.44:0/3687788171 >> 172.21.15.44:0/3687788171 conn(0x55658224a400 msgr2=0x556581553800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f9e0d740 1 -- 172.21.15.44:0/3687788171 shutdown_connections 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.625+0000 7f45f9e0d740 1 -- 172.21.15.44:0/3687788171 wait complete. 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.626+0000 7f45f9e0d740 1 Processor -- start 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.626+0000 7f45f9e0d740 1 -- start start 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.626+0000 7f45f9e0d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658224a000 0x5565815db600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.626+0000 7f45f9e0d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658233b800 0x5565815dbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.626+0000 7f45f9e0d740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658233bc00 0x5565815dc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f9e0d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556582247500 con 0x55658233b800 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f9e0d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556582247380 con 0x55658224a000 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f9e0d740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x556582247680 con 0x55658233bc00 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f7c76640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658233b800 0x5565815dbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f7c76640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658233b800 0x5565815dbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34702/0 (socket says 172.21.15.44:34702) 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 learned_addr learned my addr 172.21.15.44:0/194658039 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f8477640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658224a000 0x5565815db600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f7475640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658233bc00 0x5565815dc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658233bc00 msgr2=0x5565815dc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f7c76640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658233bc00 0x5565815dc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658224a000 msgr2=0x5565815db600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f7c76640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658224a000 0x5565815db600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55658149b680 con 0x55658233b800 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f7475640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658233bc00 0x5565815dc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f7c76640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658233b800 0x5565815dbb80 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x556582356180 tx=0x556582208ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f1c6a640 1 -- 172.21.15.44:0/194658039 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556581494820 con 0x55658233b800 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f1c6a640 1 -- 172.21.15.44:0/194658039 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5565814b9340 con 0x55658233b800 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55658149b2c0 con 0x55658233b800 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f1c6a640 1 -- 172.21.15.44:0/194658039 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556581494b60 con 0x55658233b800 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55658149ba40 con 0x55658233b800 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.627+0000 7f45f9e0d740 4 datascan.init: resolving metadata pool 13 2024-04-02T13:28:15.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.628+0000 7f45f1c6a640 1 -- 172.21.15.44:0/194658039 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x556582260c00 con 0x55658233b800 2024-04-02T13:28:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.629+0000 7f45f1c6a640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x556582339000 0x5565815dc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.629+0000 7f45f1c6a640 1 -- 172.21.15.44:0/194658039 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x55658225ea00 con 0x55658233b800 2024-04-02T13:28:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.629+0000 7f45f9e0d740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-04-02T13:28:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.629+0000 7f45f9e0d740 4 datascan.main: resolving metadata pool 13 2024-04-02T13:28:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.629+0000 7f45f8477640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x556582339000 0x5565815dc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.629+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x556582339400 0x5565815dcc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.629+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:1 13.0 13:00000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339800 con 0x556582339400 2024-04-02T13:28:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.629+0000 7f45f7475640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x556582339400 0x5565815dcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.629+0000 7f45f8477640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x556582339000 0x5565815dc680 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x5565821f6150 tx=0x556582208a80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.629+0000 7f45f7475640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x556582339400 0x5565815dcc00 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.630+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 1 ==== osd_op_reply(1 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.630+0000 7f45f7475640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x556582339800 0x5565817fe000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.630+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:2 13.10 13:08000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581802000 con 0x556582339800 2024-04-02T13:28:15.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.630+0000 7f45f8477640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x556582339800 0x5565817fe000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.630+0000 7f45f8477640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x556582339800 0x5565817fe000 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.631+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(2 [pgnls start_epoch 95 out=177b] v95'18 uv18 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.631+0000 7f45f8477640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x556581802000 0x5565815da000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.631+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:3 13.8 13:10000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55658146ec00 con 0x556581802000 2024-04-02T13:28:15.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.631+0000 7f45f7c76640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x556581802000 0x5565815da000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.631+0000 7f45f7c76640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x556581802000 0x5565815da000 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.631+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.2 v2:172.21.15.44:6816/2091834331 1 ==== osd_op_reply(3 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200780 con 0x556581802000 2024-04-02T13:28:15.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.632+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:4 13.18 13:18000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x55658146ec00 con 0x556582339400 2024-04-02T13:28:15.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.632+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 2 ==== osd_op_reply(4 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.632+0000 7f45f7475640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55658146ec00 0x5565817fe580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.632+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:5 13.4 13:20000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581802400 con 0x55658146ec00 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.632+0000 7f45f7475640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55658146ec00 0x5565817fe580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.632+0000 7f45f7475640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55658146ec00 0x5565817fe580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.633+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== osd_op_reply(5 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.633+0000 7f45f7475640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x556581802400 0x5565817feb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.633+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:6 13.14 13:28000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581802800 con 0x556581802400 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.633+0000 7f45f8477640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x556581802400 0x5565817feb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.633+0000 7f45f8477640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x556581802400 0x5565817feb00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.633+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.0 v2:172.21.15.44:6800/2303805525 1 ==== osd_op_reply(6 [pgnls start_epoch 95 out=149b] v94'27 uv27 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x556582200500 con 0x556581802400 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.633+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:7 13.c 13:30000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581802800 con 0x556582339800 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.634+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 2 ==== osd_op_reply(7 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.634+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:8 13.1c 13:38000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581802800 con 0x55658146ec00 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.634+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 2 ==== osd_op_reply(8 [pgnls start_epoch 95 out=73b] v94'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.634+0000 7f45f7475640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x556581802800 0x5565817ff080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.634+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:9 13.2 13:40000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581802c00 con 0x556581802800 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.634+0000 7f45f7c76640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x556581802800 0x5565817ff080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.635+0000 7f45f7c76640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x556581802800 0x5565817ff080 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.635+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.5 v2:172.21.15.69:6800/2841767499 1 ==== osd_op_reply(9 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200780 con 0x556581802800 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.635+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:10 13.12 13:48000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581802c00 con 0x556581802400 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.635+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.0 v2:172.21.15.44:6800/2303805525 2 ==== osd_op_reply(10 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556581802400 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.635+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:11 13.a 13:50000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581802c00 con 0x556581802000 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.635+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.2 v2:172.21.15.44:6816/2091834331 2 ==== osd_op_reply(11 [pgnls start_epoch 95 out=81b] v94'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x556582200780 con 0x556581802000 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.635+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:12 13.1a 13:58000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581802c00 con 0x556582339800 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.636+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 3 ==== osd_op_reply(12 [pgnls start_epoch 95 out=49b] v92'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.636+0000 7f45f8477640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x556581802c00 0x556581800100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.636+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:13 13.6 13:60000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803800 con 0x556581802c00 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.636+0000 7f45f7475640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x556581802c00 0x556581800100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.636+0000 7f45f7475640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x556581802c00 0x556581800100 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.636+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 1 ==== osd_op_reply(13 [pgnls start_epoch 95 out=77b] v94'8 uv8 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.636+0000 7f45f7475640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x556581803800 0x5565817ff600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.636+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:14 13.16 13:68000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556581803800 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.637+0000 7f45f8477640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x556581803800 0x5565817ff600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.637+0000 7f45f8477640 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x556581803800 0x5565817ff600 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.637+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.3 v2:172.21.15.44:6824/1588952513 1 ==== osd_op_reply(14 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556581803800 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.637+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:15 13.e 13:70000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556582339400 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.637+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 3 ==== osd_op_reply(15 [pgnls start_epoch 95 out=73b] v94'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.637+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:16 13.1e 13:78000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556581802c00 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.638+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 2 ==== osd_op_reply(16 [pgnls start_epoch 95 out=49b] v92'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.638+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:17 13.1 13:80000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x55658146ec00 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.638+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 3 ==== osd_op_reply(17 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.638+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:18 13.11 13:88000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556582339800 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.638+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 4 ==== osd_op_reply(18 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.638+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:19 13.9 13:90000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556581802800 2024-04-02T13:28:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.639+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.5 v2:172.21.15.69:6800/2841767499 2 ==== osd_op_reply(19 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200780 con 0x556581802800 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.639+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:20 13.19 13:98000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556581803800 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.639+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.3 v2:172.21.15.44:6824/1588952513 2 ==== osd_op_reply(20 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556581803800 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.639+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:21 13.5 13:a0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556581802c00 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.639+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 3 ==== osd_op_reply(21 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.639+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:22 13.15 13:a8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556581802c00 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.639+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 4 ==== osd_op_reply(22 [pgnls start_epoch 95 out=97b] v94'4 uv4 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.639+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:23 13.d 13:b0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556581802c00 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.640+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 5 ==== osd_op_reply(23 [pgnls start_epoch 95 out=74b] v90'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.640+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:24 13.1d 13:b8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556582339400 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.640+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 4 ==== osd_op_reply(24 [pgnls start_epoch 95 out=73b] v94'5 uv5 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.640+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:25 13.3 13:c0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556581802400 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.640+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.0 v2:172.21.15.44:6800/2303805525 3 ==== osd_op_reply(25 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556581802400 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.640+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:26 13.13 13:c8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556582339400 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.641+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 5 ==== osd_op_reply(26 [pgnls start_epoch 95 out=98b] v94'4 uv4 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.641+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:27 13.b 13:d0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556581802800 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.641+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.5 v2:172.21.15.69:6800/2841767499 3 ==== osd_op_reply(27 [pgnls start_epoch 95 out=76b] v94'6 uv6 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x556582200780 con 0x556581802800 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.641+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:28 13.1b 13:d8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556581802c00 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.641+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 6 ==== osd_op_reply(28 [pgnls start_epoch 95 out=73b] v94'10 uv10 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.641+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:29 13.7 13:e0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556582339400 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.642+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 6 ==== osd_op_reply(29 [pgnls start_epoch 95 out=73b] v94'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.642+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:30 13.17 13:e8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556581802800 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.642+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.5 v2:172.21.15.69:6800/2841767499 4 ==== osd_op_reply(30 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200780 con 0x556581802800 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.642+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:31 13.f 13:f0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x55658146ec00 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.643+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 4 ==== osd_op_reply(31 [pgnls start_epoch 95 out=73b] v90'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.643+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:32 13.1f 13:f8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x55658146ec00 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.643+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 5 ==== osd_op_reply(32 [pgnls start_epoch 95 out=101b] v94'9 uv9 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.643+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 601.00000000 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.643+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:33 13.10 13:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556582339800 2024-04-02T13:28:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.643+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 5 ==== osd_op_reply(33 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.643+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 602.00000000 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.643+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:34 13.10 13:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581822000 con 0x556582339800 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 6 ==== osd_op_reply(34 602.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:35 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581822400 con 0x556582339800 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 7 ==== osd_op_reply(35 10000000000.00000000 [omap-get-vals out=506b] v0'0 uv18 ondisk = 0) ==== 164+0+506 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 20 datascan.scan_links: injected ino detected 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 600.00000000 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:36 13.10 13:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581822800 con 0x556582339800 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 8 ==== osd_op_reply(36 600.00000000 [omap-get-vals out=5b] v0'0 uv12 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 603.00000000 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:37 13.10 13:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581822c00 con 0x556582339800 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 9 ==== osd_op_reply(37 603.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 200.00000000 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 200.00000001 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 606.00000000 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.644+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:38 13.14 13:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581823000 con 0x556581802400 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.645+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.0 v2:172.21.15.44:6800/2303805525 4 ==== osd_op_reply(38 606.00000000 [omap-get-vals out=5b] v0'0 uv25 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200500 con 0x556581802400 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.645+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 607.00000000 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.645+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:39 13.1c 13:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581823400 con 0x55658146ec00 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.645+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 6 ==== osd_op_reply(39 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.645+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 100000001f6.00000000 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.645+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:40 13.a 13:57fe7535:::100000001f6.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581823800 con 0x556581802000 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.645+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.2 v2:172.21.15.44:6816/2091834331 3 ==== osd_op_reply(40 100000001f6.00000000 [omap-get-vals out=54b] v0'0 uv3 ondisk = 0) ==== 164+0+54 (crc 0 0 0) 0x556582200780 con 0x556581802000 2024-04-02T13:28:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.645+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object mds0_openfiles.0 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.645+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.645+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 608.00000000 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.645+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:41 13.e 13:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581822000 con 0x556582339400 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.646+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 7 ==== osd_op_reply(41 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.646+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 604.00000000 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.646+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:42 13.15 13:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581822400 con 0x556581802c00 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.646+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 7 ==== osd_op_reply(42 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.646+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 500.00000000 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.646+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.646+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.646+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.646+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 605.00000000 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.646+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:43 13.1d 13:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581822800 con 0x556582339400 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 8 ==== osd_op_reply(43 605.00000000 [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 100.00000000 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:44 13.13 13:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581822c00 con 0x556582339400 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 9 ==== osd_op_reply(44 100.00000000 [omap-get-vals out=5195b] v0'0 uv4 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 200.00000002 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000002' 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 609.00000000 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:45 13.7 13:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581823000 con 0x556582339400 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 10 ==== osd_op_reply(45 609.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 400.00000000 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-04-02T13:28:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 10 datascan.scan_links: step 1: handling object 1.00000000 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.647+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:46 13.1f 13:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581823c00 con 0x55658146ec00 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.648+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 7 ==== osd_op_reply(46 1.00000000 [omap-get-vals out=1041b] v0'0 uv9 ondisk = 0) ==== 154+0+1041 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.648+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:47 13.0 13:00000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556582339400 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.648+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 11 ==== osd_op_reply(47 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.648+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:48 13.10 13:08000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556582339800 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.648+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 10 ==== osd_op_reply(48 [pgnls start_epoch 95 out=177b] v95'18 uv18 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.648+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:49 13.8 13:10000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802000 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.649+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.2 v2:172.21.15.44:6816/2091834331 4 ==== osd_op_reply(49 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200780 con 0x556581802000 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.649+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:50 13.18 13:18000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556582339400 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.649+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 12 ==== osd_op_reply(50 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.649+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:51 13.4 13:20000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x55658146ec00 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.649+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 8 ==== osd_op_reply(51 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.649+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:52 13.14 13:28000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802400 2024-04-02T13:28:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.650+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.0 v2:172.21.15.44:6800/2303805525 5 ==== osd_op_reply(52 [pgnls start_epoch 95 out=149b] v94'27 uv27 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x556582200500 con 0x556581802400 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.650+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:53 13.c 13:30000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556582339800 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.650+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 11 ==== osd_op_reply(53 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.650+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:54 13.1c 13:38000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x55658146ec00 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.650+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 9 ==== osd_op_reply(54 [pgnls start_epoch 95 out=73b] v94'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.650+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:55 13.2 13:40000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802800 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.651+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.5 v2:172.21.15.69:6800/2841767499 5 ==== osd_op_reply(55 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200780 con 0x556581802800 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.651+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:56 13.12 13:48000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802400 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.651+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.0 v2:172.21.15.44:6800/2303805525 6 ==== osd_op_reply(56 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556581802400 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.651+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:57 13.a 13:50000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802000 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.651+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.2 v2:172.21.15.44:6816/2091834331 5 ==== osd_op_reply(57 [pgnls start_epoch 95 out=81b] v94'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x556582200780 con 0x556581802000 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.651+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:58 13.1a 13:58000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556582339800 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.651+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 12 ==== osd_op_reply(58 [pgnls start_epoch 95 out=49b] v92'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.651+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:59 13.6 13:60000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802c00 2024-04-02T13:28:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.652+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 8 ==== osd_op_reply(59 [pgnls start_epoch 95 out=77b] v94'8 uv8 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.652+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:60 13.16 13:68000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581803800 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.652+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.3 v2:172.21.15.44:6824/1588952513 3 ==== osd_op_reply(60 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556581803800 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.652+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:61 13.e 13:70000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556582339400 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.652+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 13 ==== osd_op_reply(61 [pgnls start_epoch 95 out=73b] v94'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.653+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:62 13.1e 13:78000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802c00 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.653+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 9 ==== osd_op_reply(62 [pgnls start_epoch 95 out=49b] v92'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.653+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:63 13.1 13:80000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x55658146ec00 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.653+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 10 ==== osd_op_reply(63 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.653+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:64 13.11 13:88000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556582339800 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.653+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 13 ==== osd_op_reply(64 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.653+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:65 13.9 13:90000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802800 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.654+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.5 v2:172.21.15.69:6800/2841767499 6 ==== osd_op_reply(65 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200780 con 0x556581802800 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.654+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:66 13.19 13:98000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581803800 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.654+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.3 v2:172.21.15.44:6824/1588952513 4 ==== osd_op_reply(66 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556581803800 2024-04-02T13:28:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.654+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:67 13.5 13:a0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802c00 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.654+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 10 ==== osd_op_reply(67 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.654+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:68 13.15 13:a8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802c00 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.654+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 11 ==== osd_op_reply(68 [pgnls start_epoch 95 out=97b] v94'4 uv4 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.654+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:69 13.d 13:b0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802c00 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.655+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 12 ==== osd_op_reply(69 [pgnls start_epoch 95 out=74b] v90'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.655+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:70 13.1d 13:b8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556582339400 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.655+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 14 ==== osd_op_reply(70 [pgnls start_epoch 95 out=73b] v94'5 uv5 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.655+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:71 13.3 13:c0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802400 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.655+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.0 v2:172.21.15.44:6800/2303805525 7 ==== osd_op_reply(71 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200500 con 0x556581802400 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.655+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:72 13.13 13:c8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556582339400 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.655+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 15 ==== osd_op_reply(72 [pgnls start_epoch 95 out=98b] v94'4 uv4 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.655+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:73 13.b 13:d0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802800 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.656+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.5 v2:172.21.15.69:6800/2841767499 7 ==== osd_op_reply(73 [pgnls start_epoch 95 out=76b] v94'6 uv6 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x556582200780 con 0x556581802800 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.656+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:74 13.1b 13:d8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802c00 2024-04-02T13:28:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.656+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 13 ==== osd_op_reply(74 [pgnls start_epoch 95 out=73b] v94'10 uv10 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.656+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:75 13.7 13:e0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556582339400 2024-04-02T13:28:15.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.656+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 16 ==== osd_op_reply(75 [pgnls start_epoch 95 out=73b] v94'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.656+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:76 13.17 13:e8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556581802800 2024-04-02T13:28:15.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.657+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.5 v2:172.21.15.69:6800/2841767499 8 ==== osd_op_reply(76 [pgnls start_epoch 95 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x556582200780 con 0x556581802800 2024-04-02T13:28:15.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.657+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:77 13.f 13:f0000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x55658146ec00 2024-04-02T13:28:15.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.657+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 11 ==== osd_op_reply(77 [pgnls start_epoch 95 out=73b] v90'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.657+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:78 13.1f 13:f8000000::::head [pgnls start_epoch 95 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x55658146ec00 2024-04-02T13:28:15.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.657+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 12 ==== osd_op_reply(78 [pgnls start_epoch 95 out=101b] v94'9 uv9 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.657+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 601.00000000 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.657+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:79 13.10 13:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581803c00 con 0x556582339800 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.657+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 14 ==== osd_op_reply(79 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 602.00000000 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:80 13.10 13:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581823400 con 0x556582339800 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 15 ==== osd_op_reply(80 602.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:81 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581822000 con 0x556582339800 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 16 ==== osd_op_reply(81 10000000000.00000000 [omap-get-vals out=506b] v0'0 uv18 ondisk = 0) ==== 164+0+506 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 20 datascan.scan_links: injected ino detected 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 -1 datascan.scan_links: Bad nlink on 0x10000000001 expected 2 has 1 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 20 datascan.scan_links: adding 0x10000000001 for future processing to fix dnfirst 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 600.00000000 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:82 13.10 13:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581822400 con 0x556582339800 2024-04-02T13:28:15.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 17 ==== osd_op_reply(82 600.00000000 [omap-get-vals out=5b] v0'0 uv12 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 603.00000000 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:83 13.10 13:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581803400 con 0x556582339800 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 18 ==== osd_op_reply(83 603.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 200.00000000 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 200.00000001 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 606.00000000 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.658+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:84 13.14 13:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581803000 con 0x556581802400 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.659+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.0 v2:172.21.15.44:6800/2303805525 8 ==== osd_op_reply(84 606.00000000 [omap-get-vals out=5b] v0'0 uv25 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200500 con 0x556581802400 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.659+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 607.00000000 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.659+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:85 13.1c 13:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x55658182a000 con 0x55658146ec00 2024-04-02T13:28:15.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.659+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 13 ==== osd_op_reply(85 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 100000001f6.00000000 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:86 13.a 13:57fe7535:::100000001f6.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x55658182a400 con 0x556581802000 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f7c76640 1 -- 172.21.15.44:0/194658039 <== osd.2 v2:172.21.15.44:6816/2091834331 6 ==== osd_op_reply(86 100000001f6.00000000 [omap-get-vals out=54b] v0'0 uv3 ondisk = 0) ==== 164+0+54 (crc 0 0 0) 0x556582200780 con 0x556581802000 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object mds0_openfiles.0 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 608.00000000 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:87 13.e 13:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x55658182a800 con 0x556582339400 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 17 ==== osd_op_reply(87 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 604.00000000 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:88 13.15 13:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x55658182ac00 con 0x556581802c00 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 14 ==== osd_op_reply(88 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 500.00000000 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-04-02T13:28:15.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.660+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 605.00000000 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:89 13.1d 13:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x55658182b000 con 0x556582339400 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 18 ==== osd_op_reply(89 605.00000000 [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 100.00000000 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:90 13.13 13:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x55658182b400 con 0x556582339400 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 19 ==== osd_op_reply(90 100.00000000 [omap-get-vals out=5195b] v0'0 uv4 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 200.00000002 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000002' 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 609.00000000 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:91 13.7 13:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x55658182b800 con 0x556582339400 2024-04-02T13:28:15.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 20 ==== osd_op_reply(91 609.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 400.00000000 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 10 datascan.scan_links: step 2: handling object 1.00000000 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.661+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:92 13.1f 13:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x55658182bc00 con 0x55658146ec00 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.662+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.6 v2:172.21.15.69:6802/2604158175 14 ==== osd_op_reply(92 1.00000000 [omap-get-vals out=1041b] v0'0 uv9 ondisk = 0) ==== 154+0+1041 (crc 0 0 0) 0x556582200000 con 0x55658146ec00 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.662+0000 7f45f9e0d740 10 datascan.scan_links: processing 0 dup_primaries, 1 remote_links 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.662+0000 7f45f9e0d740 10 datascan.scan_links: removing dup dentries from 0 objects 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.662+0000 7f45f9e0d740 10 datascan.scan_links: processing 1 bad_nlink_inos 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.662+0000 7f45f9e0d740 10 datascan.scan_links: handling bad_nlink_ino 0x10000000001 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.662+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:93 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x55658182a000 con 0x556582339800 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.662+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 19 ==== osd_op_reply(93 10000000000.00000000 [omap-get-vals-by-keys out=505b] v0'0 uv18 ondisk = 0) ==== 164+0+505 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.662+0000 7f45f9e0d740 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.662+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:94 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x556582339c00 con 0x556582339800 2024-04-02T13:28:15.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.663+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 20 ==== osd_op_reply(94 10000000000.00000000 [omap-set-vals] v95'19 uv19 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.663+0000 7f45f9e0d740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.663+0000 7f45f9e0d740 10 datascan.scan_links: processing 1 injected_inos 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.663+0000 7f45f9e0d740 10 datascan.scan_links: handling injected_ino 0x10000000001 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.663+0000 7f45f9e0d740 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000000:file 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.663+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:95 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581822800 con 0x556582339800 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.664+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 21 ==== osd_op_reply(95 10000000000.00000000 [omap-get-vals-by-keys out=505b] v0'0 uv19 ondisk = 0) ==== 164+0+505 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.664+0000 7f45f9e0d740 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.664+0000 7f45f9e0d740 20 datascan.scan_links: first is now 2 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.664+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:96 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x556581822c00 con 0x556582339800 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.665+0000 7f45f8477640 1 -- 172.21.15.44:0/194658039 <== osd.1 v2:172.21.15.44:6801/704457846 22 ==== osd_op_reply(96 10000000000.00000000 [omap-set-vals] v95'20 uv20 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x556582200500 con 0x556582339800 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.665+0000 7f45f9e0d740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.665+0000 7f45f9e0d740 10 datascan.scan_links: updating inotable 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.665+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:97 13.13 13:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x556581823000 con 0x556582339400 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.665+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 21 ==== osd_op_reply(97 mds0_inotable [read 0~50 out=50b] v0'0 uv3 ondisk = 0) ==== 157+0+50 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.665+0000 7f45f9e0d740 10 mds.0.inotable: skip_inos was [0x10000000002~0x1f4,0x100000001f7~0xfffffffe09] 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.665+0000 7f45f9e0d740 10 mds.0.inotable: skip_inos now [0x100000001f7~0xfffffffe09] 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.665+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:98 13.13 13:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x556581823c00 con 0x556582339400 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.666+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.7 v2:172.21.15.69:6824/1292475712 22 ==== osd_op_reply(98 mds0_inotable [writefull 0~34] v95'5 uv5 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x556582200000 con 0x556582339400 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.666+0000 7f45f9e0d740 10 datascan.scan_links: updating snaptable 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.666+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:99 13.d 13:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) -- 0x55658182a800 con 0x556581802c00 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 <== osd.4 v2:172.21.15.69:6816/1041350007 15 ==== osd_op_reply(99 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) ==== 157+0+70 (crc 0 0 0) 0x556582200000 con 0x556581802c00 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x556581802400 msgr2=0x5565817feb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x556581802400 0x5565817feb00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x556582339800 msgr2=0x5565817fe000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x556582339800 0x5565817fe000 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x556581802000 msgr2=0x5565815da000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x556581802000 0x5565815da000 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x556581803800 msgr2=0x5565817ff600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x556581803800 0x5565817ff600 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 reap_dead start 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x556581802c00 msgr2=0x556581800100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x556581802c00 0x556581800100 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 reap_dead start 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x556581802800 msgr2=0x5565817ff080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x556581802800 0x5565817ff080 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55658146ec00 msgr2=0x5565817fe580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55658146ec00 0x5565817fe580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x556582339400 msgr2=0x5565815dcc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x556582339400 0x5565815dcc00 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x556582339000 msgr2=0x5565815dc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x556582339000 0x5565815dc680 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x5565821f6150 tx=0x556582208a80 comp rx=0 tx=0).stop 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658233b800 msgr2=0x5565815dbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/194658039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658233b800 0x5565815dbb80 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x556582356180 tx=0x556582208ae0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f7475640 1 -- 172.21.15.44:0/194658039 reap_dead start 2024-04-02T13:28:15.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 shutdown_connections 2024-04-02T13:28:15.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 >> 172.21.15.44:0/194658039 conn(0x55658224a400 msgr2=0x556581553800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:15.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 shutdown_connections 2024-04-02T13:28:15.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/194658039 wait complete. 2024-04-02T13:28:15.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 -- 172.21.15.44:0/3891032402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a800 msgr2=0x5565815db080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.667+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/3891032402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a800 0x5565815db080 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x556582295b60 tx=0x55658220a0c0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.668+0000 7f45f9e0d740 1 -- 172.21.15.44:0/3891032402 shutdown_connections 2024-04-02T13:28:15.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.668+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/3891032402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55658224bc00 0x5565815da580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.668+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/3891032402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55658146fc00 0x5565815dab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.668+0000 7f45f9e0d740 1 --2- 172.21.15.44:0/3891032402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55658224a800 0x5565815db080 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.668+0000 7f45f9e0d740 1 -- 172.21.15.44:0/3891032402 >> 172.21.15.44:0/3891032402 conn(0x55658146f000 msgr2=0x556581552000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:15.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.668+0000 7f45f9e0d740 1 -- 172.21.15.44:0/3891032402 shutdown_connections 2024-04-02T13:28:15.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.668+0000 7f45f9e0d740 1 -- 172.21.15.44:0/3891032402 wait complete. 2024-04-02T13:28:15.692 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links'] in time 0:00:00.136086, result: 2024-04-02T13:28:15.693 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-02T13:28:15.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.824+0000 7fbe8faf2640 1 Processor -- start 2024-04-02T13:28:15.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.825+0000 7fbe8faf2640 1 -- start start 2024-04-02T13:28:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.825+0000 7fbe8faf2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbe88161ab0 0x7fbe880d4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.825+0000 7fbe8faf2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbe880d2aa0 0x7fbe880c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.825+0000 7fbe8faf2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe880b5500 con 0x7fbe880d1f00 2024-04-02T13:28:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.825+0000 7fbe8faf2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fbe880d2fa0 con 0x7fbe88161ab0 2024-04-02T13:28:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.825+0000 7fbe8faf2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbe880cfab0 con 0x7fbe880d2aa0 2024-04-02T13:28:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.825+0000 7fbe8d88e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbe88161ab0 0x7fbe880d4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.825+0000 7fbe8d88e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbe88161ab0 0x7fbe880d4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34712/0 (socket says 172.21.15.44:34712) 2024-04-02T13:28:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.825+0000 7fbe8d88e640 1 -- 172.21.15.44:0/196246904 learned_addr learned my addr 172.21.15.44:0/196246904 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.826+0000 7fbe8d88e640 1 -- 172.21.15.44:0/196246904 >> v2:172.21.15.69:3300/0 conn(0x7fbe880d2aa0 msgr2=0x7fbe880c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.826+0000 7fbe8d88e640 1 --2- 172.21.15.44:0/196246904 >> v2:172.21.15.69:3300/0 conn(0x7fbe880d2aa0 0x7fbe880c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.826+0000 7fbe8d88e640 1 -- 172.21.15.44:0/196246904 >> v1:172.21.15.69:6789/0 conn(0x7fbe880d1f00 legacy=0x7fbe880d2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.826+0000 7fbe8d88e640 1 -- 172.21.15.44:0/196246904 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe880cfd90 con 0x7fbe88161ab0 2024-04-02T13:28:15.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.826+0000 7fbe8d88e640 1 --2- 172.21.15.44:0/196246904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe88161ab0 0x7fbe880d4c40 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fbe880a1fe0 tx=0x7fbe8402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d123de4ecf2416da server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.827+0000 7fbe8c88c640 1 -- 172.21.15.44:0/196246904 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84035020 con 0x7fbe88161ab0 2024-04-02T13:28:15.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.827+0000 7fbe8c88c640 1 -- 172.21.15.44:0/196246904 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe84031870 con 0x7fbe88161ab0 2024-04-02T13:28:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.827+0000 7fbe8c88c640 1 -- 172.21.15.44:0/196246904 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84031c40 con 0x7fbe88161ab0 2024-04-02T13:28:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.827+0000 7fbe8faf2640 1 -- 172.21.15.44:0/196246904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe88161ab0 msgr2=0x7fbe880d4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.827+0000 7fbe8faf2640 1 --2- 172.21.15.44:0/196246904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe88161ab0 0x7fbe880d4c40 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fbe880a1fe0 tx=0x7fbe8402b910 comp rx=0 tx=0).stop 2024-04-02T13:28:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.827+0000 7fbe8faf2640 1 -- 172.21.15.44:0/196246904 shutdown_connections 2024-04-02T13:28:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.827+0000 7fbe8faf2640 1 --2- 172.21.15.44:0/196246904 >> v2:172.21.15.69:3300/0 conn(0x7fbe880d2aa0 0x7fbe880c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.827+0000 7fbe8faf2640 1 --2- 172.21.15.44:0/196246904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe88161ab0 0x7fbe880d4c40 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.827+0000 7fbe8faf2640 1 -- 172.21.15.44:0/196246904 >> 172.21.15.44:0/196246904 conn(0x7fbe880d4120 msgr2=0x7fbe880d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.827+0000 7fbe8faf2640 1 -- 172.21.15.44:0/196246904 shutdown_connections 2024-04-02T13:28:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.827+0000 7fbe8faf2640 1 -- 172.21.15.44:0/196246904 wait complete. 2024-04-02T13:28:15.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.828+0000 7fbe8faf2640 1 Processor -- start 2024-04-02T13:28:15.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.828+0000 7fbe8faf2640 1 -- start start 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.829+0000 7fbe8faf2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe880d1f00 0x7fbe881461d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.829+0000 7fbe8faf2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe880d2aa0 0x7fbe881f4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.829+0000 7fbe8faf2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe88161ab0 0x7fbe881f7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.829+0000 7fbe8faf2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbe88161e90 con 0x7fbe880d2aa0 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.829+0000 7fbe8faf2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbe880cf7a0 con 0x7fbe880d1f00 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.829+0000 7fbe8faf2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbe88162010 con 0x7fbe88161ab0 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.829+0000 7fbe8d08d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe880d2aa0 0x7fbe881f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.829+0000 7fbe8d08d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe880d2aa0 0x7fbe881f4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34720/0 (socket says 172.21.15.44:34720) 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.829+0000 7fbe8d08d640 1 -- 172.21.15.44:0/1273599168 learned_addr learned my addr 172.21.15.44:0/1273599168 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.829+0000 7fbe8d88e640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe880d1f00 0x7fbe881461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.829+0000 7fbe8e08f640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe88161ab0 0x7fbe881f7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.830+0000 7fbe8d08d640 1 -- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe88161ab0 msgr2=0x7fbe881f7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.830+0000 7fbe8d08d640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe88161ab0 0x7fbe881f7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.830+0000 7fbe8d08d640 1 -- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe880d1f00 msgr2=0x7fbe881461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:15.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.830+0000 7fbe8d08d640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe880d1f00 0x7fbe881461d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:15.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.830+0000 7fbe8d08d640 1 -- 172.21.15.44:0/1273599168 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe880b9bd0 con 0x7fbe880d2aa0 2024-04-02T13:28:15.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.830+0000 7fbe8d08d640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe880d2aa0 0x7fbe881f4340 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800b3e0 tx=0x7fbe7800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.830+0000 7fbe7e7fc640 1 -- 172.21.15.44:0/1273599168 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78015020 con 0x7fbe880d2aa0 2024-04-02T13:28:15.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.830+0000 7fbe7e7fc640 1 -- 172.21.15.44:0/1273599168 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe78011850 con 0x7fbe880d2aa0 2024-04-02T13:28:15.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.830+0000 7fbe7e7fc640 1 -- 172.21.15.44:0/1273599168 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78011c40 con 0x7fbe880d2aa0 2024-04-02T13:28:15.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.830+0000 7fbe8faf2640 1 -- 172.21.15.44:0/1273599168 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe880b6f60 con 0x7fbe880d2aa0 2024-04-02T13:28:15.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.830+0000 7fbe8faf2640 1 -- 172.21.15.44:0/1273599168 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe880b7520 con 0x7fbe880d2aa0 2024-04-02T13:28:15.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.831+0000 7fbe8faf2640 1 -- 172.21.15.44:0/1273599168 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe880684f0 con 0x7fbe880d2aa0 2024-04-02T13:28:15.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.832+0000 7fbe7e7fc640 1 -- 172.21.15.44:0/1273599168 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbe78018020 con 0x7fbe880d2aa0 2024-04-02T13:28:15.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.832+0000 7fbe7e7fc640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbe6c06c8f0 0x7fbe6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:15.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.832+0000 7fbe7e7fc640 1 -- 172.21.15.44:0/1273599168 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fbe7808bb90 con 0x7fbe880d2aa0 2024-04-02T13:28:15.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.832+0000 7fbe8d88e640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbe6c06c8f0 0x7fbe6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:15.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.833+0000 7fbe8d88e640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbe6c06c8f0 0x7fbe6c06edb0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fbe84036000 tx=0x7fbe8403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:15.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:15.837+0000 7fbe7e7fc640 1 -- 172.21.15.44:0/1273599168 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbe7801f020 con 0x7fbe880d2aa0 2024-04-02T13:28:16.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.060+0000 7fbe8faf2640 1 -- 172.21.15.44:0/1273599168 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fbe880caa80 con 0x7fbe880d2aa0 2024-04-02T13:28:16.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.303+0000 7fbe7e7fc640 1 -- 172.21.15.44:0/1273599168 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v70) ==== 162+0+0 (secure 0 0 0) 0x7fbe78059750 con 0x7fbe880d2aa0 2024-04-02T13:28:16.307 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-02T13:28:16.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.305+0000 7fbe8faf2640 1 -- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbe6c06c8f0 msgr2=0x7fbe6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:16.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.305+0000 7fbe8faf2640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbe6c06c8f0 0x7fbe6c06edb0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fbe84036000 tx=0x7fbe8403c040 comp rx=0 tx=0).stop 2024-04-02T13:28:16.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.305+0000 7fbe8faf2640 1 -- 172.21.15.44:0/1273599168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe880d2aa0 msgr2=0x7fbe881f4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:16.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.305+0000 7fbe8faf2640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe880d2aa0 0x7fbe881f4340 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800b3e0 tx=0x7fbe7800b8a0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.305+0000 7fbe8faf2640 1 -- 172.21.15.44:0/1273599168 shutdown_connections 2024-04-02T13:28:16.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.305+0000 7fbe8faf2640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbe6c06c8f0 0x7fbe6c06edb0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.305+0000 7fbe8faf2640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe88161ab0 0x7fbe881f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.305+0000 7fbe8faf2640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe880d2aa0 0x7fbe881f4340 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.305+0000 7fbe8faf2640 1 --2- 172.21.15.44:0/1273599168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe880d1f00 0x7fbe881461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.306+0000 7fbe8faf2640 1 -- 172.21.15.44:0/1273599168 >> 172.21.15.44:0/1273599168 conn(0x7fbe880d4120 msgr2=0x7fbe8815f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:16.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.306+0000 7fbe8faf2640 1 -- 172.21.15.44:0/1273599168 shutdown_connections 2024-04-02T13:28:16.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.306+0000 7fbe8faf2640 1 -- 172.21.15.44:0/1273599168 wait complete. 2024-04-02T13:28:16.322 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:28:16.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.420+0000 7f868f1f6640 1 Processor -- start 2024-04-02T13:28:16.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.420+0000 7f868f1f6640 1 -- start start 2024-04-02T13:28:16.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.421+0000 7f868f1f6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f86880cc8b0 0x7f86880cab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:16.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.421+0000 7f868f1f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880cbb50 0x7f86880b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:16.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.421+0000 7f868f1f6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f868806fc60 con 0x7f86880ccf00 2024-04-02T13:28:16.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.421+0000 7f868f1f6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f86880cb100 con 0x7f86880cc8b0 2024-04-02T13:28:16.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.421+0000 7f868f1f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f86880cb6c0 con 0x7f86880cbb50 2024-04-02T13:28:16.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.421+0000 7f868d793640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880cbb50 0x7f86880b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:16.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.421+0000 7f8687fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f86880cc8b0 0x7f86880cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:16.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.421+0000 7f868cf92640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f86880ccf00 0x7f86880cd2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:48944/0 (socket says 172.21.15.44:48944) 2024-04-02T13:28:16.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.422+0000 7f868cf92640 1 -- 172.21.15.44:0/4039610723 learned_addr learned my addr 172.21.15.44:0/4039610723 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:16.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.422+0000 7f868d793640 1 --2- 172.21.15.44:0/4039610723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880cbb50 0x7f86880b33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:53174/0 (socket says 172.21.15.44:53174) 2024-04-02T13:28:16.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.422+0000 7f8687fff640 1 --2- 172.21.15.44:0/4039610723 >> v2:172.21.15.69:3300/0 conn(0x7f86880cc8b0 0x7f86880cab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:36990/0 (socket says 172.21.15.44:36990) 2024-04-02T13:28:16.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.422+0000 7f86877fe640 1 -- 172.21.15.44:0/4039610723 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 149332010 0 0) 0x7f868806fc60 con 0x7f86880ccf00 2024-04-02T13:28:16.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.422+0000 7f86877fe640 1 -- 172.21.15.44:0/4039610723 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8678003520 con 0x7f86880ccf00 2024-04-02T13:28:16.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.422+0000 7f868d793640 1 -- 172.21.15.44:0/4039610723 >> v2:172.21.15.69:3300/0 conn(0x7f86880cc8b0 msgr2=0x7f86880cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:16.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.422+0000 7f868d793640 1 --2- 172.21.15.44:0/4039610723 >> v2:172.21.15.69:3300/0 conn(0x7f86880cc8b0 0x7f86880cab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.422+0000 7f868d793640 1 -- 172.21.15.44:0/4039610723 >> v1:172.21.15.69:6789/0 conn(0x7f86880ccf00 legacy=0x7f86880cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:16.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.422+0000 7f868d793640 1 -- 172.21.15.44:0/4039610723 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f868806ff00 con 0x7f86880cbb50 2024-04-02T13:28:16.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.422+0000 7f8687fff640 1 --2- 172.21.15.44:0/4039610723 >> v2:172.21.15.69:3300/0 conn(0x7f86880cc8b0 0x7f86880cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:28:16.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.423+0000 7f868d793640 1 --2- 172.21.15.44:0/4039610723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880cbb50 0x7f86880b33e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8680009d90 tx=0x7f868002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b17a882c011ca3f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:16.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.423+0000 7f86877fe640 1 -- 172.21.15.44:0/4039610723 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8680039070 con 0x7f86880cbb50 2024-04-02T13:28:16.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.423+0000 7f86877fe640 1 -- 172.21.15.44:0/4039610723 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f868002cbc0 con 0x7f86880cbb50 2024-04-02T13:28:16.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.423+0000 7f86877fe640 1 -- 172.21.15.44:0/4039610723 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8680034440 con 0x7f86880cbb50 2024-04-02T13:28:16.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.423+0000 7f868f1f6640 1 -- 172.21.15.44:0/4039610723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880cbb50 msgr2=0x7f86880b33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:16.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.423+0000 7f868f1f6640 1 --2- 172.21.15.44:0/4039610723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880cbb50 0x7f86880b33e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8680009d90 tx=0x7f868002be10 comp rx=0 tx=0).stop 2024-04-02T13:28:16.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.423+0000 7f868f1f6640 1 -- 172.21.15.44:0/4039610723 shutdown_connections 2024-04-02T13:28:16.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.423+0000 7f868f1f6640 1 --2- 172.21.15.44:0/4039610723 >> v2:172.21.15.69:3300/0 conn(0x7f86880cc8b0 0x7f86880cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.423+0000 7f868f1f6640 1 --2- 172.21.15.44:0/4039610723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880cbb50 0x7f86880b33e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.423+0000 7f868f1f6640 1 -- 172.21.15.44:0/4039610723 >> 172.21.15.44:0/4039610723 conn(0x7f86880b20e0 msgr2=0x7f86880b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:16.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.424+0000 7f868f1f6640 1 -- 172.21.15.44:0/4039610723 shutdown_connections 2024-04-02T13:28:16.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.424+0000 7f868f1f6640 1 -- 172.21.15.44:0/4039610723 wait complete. 2024-04-02T13:28:16.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.425+0000 7f868f1f6640 1 Processor -- start 2024-04-02T13:28:16.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.425+0000 7f868f1f6640 1 -- start start 2024-04-02T13:28:16.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.425+0000 7f868f1f6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86880cbb50 0x7f86880c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:16.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.425+0000 7f868f1f6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86880cc8b0 0x7f86881f1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:16.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.425+0000 7f868f1f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880ccf00 0x7f86880b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:16.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.425+0000 7f868f1f6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f86880cc610 con 0x7f86880cc8b0 2024-04-02T13:28:16.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.425+0000 7f868f1f6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f86880cb350 con 0x7f86880cbb50 2024-04-02T13:28:16.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.425+0000 7f868f1f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f868815c550 con 0x7f86880ccf00 2024-04-02T13:28:16.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f8687fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86880cc8b0 0x7f86881f1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:16.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f8687fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86880cc8b0 0x7f86881f1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34734/0 (socket says 172.21.15.44:34734) 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f8687fff640 1 -- 172.21.15.44:0/1348856301 learned_addr learned my addr 172.21.15.44:0/1348856301 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f868cf92640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86880cbb50 0x7f86880c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f868d793640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880ccf00 0x7f86880b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f8687fff640 1 -- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880ccf00 msgr2=0x7f86880b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f8687fff640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880ccf00 0x7f86880b4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f8687fff640 1 -- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86880cbb50 msgr2=0x7f86880c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f8687fff640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86880cbb50 0x7f86880c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f8687fff640 1 -- 172.21.15.44:0/1348856301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86881f16a0 con 0x7f86880cc8b0 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f868cf92640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86880cbb50 0x7f86880c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f8687fff640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86880cc8b0 0x7f86881f1160 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f867400ec10 tx=0x7f867400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f86857fa640 1 -- 172.21.15.44:0/1348856301 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674017070 con 0x7f86880cc8b0 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f86857fa640 1 -- 172.21.15.44:0/1348856301 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f867400cc30 con 0x7f86880cc8b0 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f868f1f6640 1 -- 172.21.15.44:0/1348856301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86881f4880 con 0x7f86880cc8b0 2024-04-02T13:28:16.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f86857fa640 1 -- 172.21.15.44:0/1348856301 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674013470 con 0x7f86880cc8b0 2024-04-02T13:28:16.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.426+0000 7f868f1f6640 1 -- 172.21.15.44:0/1348856301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86880c60b0 con 0x7f86880cc8b0 2024-04-02T13:28:16.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.427+0000 7f868f1f6640 1 -- 172.21.15.44:0/1348856301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86880aef60 con 0x7f86880cc8b0 2024-04-02T13:28:16.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.428+0000 7f86857fa640 1 -- 172.21.15.44:0/1348856301 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f867401a070 con 0x7f86880cc8b0 2024-04-02T13:28:16.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.428+0000 7f86857fa640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f866c06c8f0 0x7f866c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:16.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.429+0000 7f86857fa640 1 -- 172.21.15.44:0/1348856301 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f867408c110 con 0x7f86880cc8b0 2024-04-02T13:28:16.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.429+0000 7f868cf92640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f866c06c8f0 0x7f866c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:16.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.429+0000 7f868cf92640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f866c06c8f0 0x7f866c06edb0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f867c007fd0 tx=0x7f867c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:16.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.433+0000 7f86857fa640 1 -- 172.21.15.44:0/1348856301 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f86880aef60 con 0x7f86880cc8b0 2024-04-02T13:28:16.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.659+0000 7f868f1f6640 1 -- 172.21.15.44:0/1348856301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86880684f0 con 0x7f86880cc8b0 2024-04-02T13:28:16.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.659+0000 7f86857fa640 1 -- 172.21.15.44:0/1348856301 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5174 (secure 0 0 0) 0x7f867405a290 con 0x7f86880cc8b0 2024-04-02T13:28:16.663 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:16.663 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":71,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:28:16.307101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8403":{"gid":8403,"name":"a","rank":0,"incarnation":71,"state":"up:replay","state_seq":1,"addr":"172.21.15.44:6833/3044289683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":3044289683},{"type":"v1","addr":"172.21.15.44:6833","nonce":3044289683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-04-02T13:28:16.663 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 71 2024-04-02T13:28:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.661+0000 7f868f1f6640 1 -- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f866c06c8f0 msgr2=0x7f866c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.661+0000 7f868f1f6640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f866c06c8f0 0x7f866c06edb0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f867c007fd0 tx=0x7f867c009040 comp rx=0 tx=0).stop 2024-04-02T13:28:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.661+0000 7f868f1f6640 1 -- 172.21.15.44:0/1348856301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86880cc8b0 msgr2=0x7f86881f1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.661+0000 7f868f1f6640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86880cc8b0 0x7f86881f1160 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f867400ec10 tx=0x7f867400c520 comp rx=0 tx=0).stop 2024-04-02T13:28:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.661+0000 7f868f1f6640 1 -- 172.21.15.44:0/1348856301 shutdown_connections 2024-04-02T13:28:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.661+0000 7f868f1f6640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f866c06c8f0 0x7f866c06edb0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.661+0000 7f868f1f6640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880ccf00 0x7f86880b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.661+0000 7f868f1f6640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86880cc8b0 0x7f86881f1160 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.661+0000 7f868f1f6640 1 --2- 172.21.15.44:0/1348856301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86880cbb50 0x7f86880c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.661+0000 7f868f1f6640 1 -- 172.21.15.44:0/1348856301 >> 172.21.15.44:0/1348856301 conn(0x7f86880b20e0 msgr2=0x7f86880d0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:16.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.661+0000 7f868f1f6640 1 -- 172.21.15.44:0/1348856301 shutdown_connections 2024-04-02T13:28:16.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:16.661+0000 7f868f1f6640 1 -- 172.21.15.44:0/1348856301 wait complete. 2024-04-02T13:28:17.081 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:26:43.025860+0000', 'modified': '2024-04-02T13:28:16.307101+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 95, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8403}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8403': {'gid': 8403, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.44:6833/3044289683', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6832', 'nonce': 3044289683}, {'type': 'v1', 'addr': '172.21.15.44:6833', 'nonce': 3044289683}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-02T13:28:17.082 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8403:up:replay 2024-04-02T13:28:18.083 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:28:18.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.193+0000 7f1d1d6c5640 1 Processor -- start 2024-04-02T13:28:18.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.194+0000 7f1d1d6c5640 1 -- start start 2024-04-02T13:28:18.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.194+0000 7f1d1d6c5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d1815e4d0 0x7f1d180cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:18.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.194+0000 7f1d1d6c5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d180d1230 0x7f1d180b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:18.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.194+0000 7f1d1d6c5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d1815e8b0 con 0x7f1d180cfd20 2024-04-02T13:28:18.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.194+0000 7f1d1d6c5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1d180b3a90 con 0x7f1d180d1230 2024-04-02T13:28:18.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.194+0000 7f1d1d6c5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1d180b4610 con 0x7f1d1815e4d0 2024-04-02T13:28:18.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.195+0000 7f1d177fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d180d1230 0x7f1d180b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:18.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.195+0000 7f1d16ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d1815e4d0 0x7f1d180cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:18.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.195+0000 7f1d177fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d180d1230 0x7f1d180b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:37020/0 (socket says 172.21.15.44:37020) 2024-04-02T13:28:18.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.195+0000 7f1d16ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d1815e4d0 0x7f1d180cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:53188/0 (socket says 172.21.15.44:53188) 2024-04-02T13:28:18.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.195+0000 7f1d177fe640 1 -- 172.21.15.44:0/420594566 learned_addr learned my addr 172.21.15.44:0/420594566 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:18.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.195+0000 7f1d167fc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1d180cfd20 0x7f1d180d0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:48956/0 (socket says 172.21.15.44:48956) 2024-04-02T13:28:18.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.195+0000 7f1d15ffb640 1 -- 172.21.15.44:0/420594566 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 470493472 0 0) 0x7f1d1815e8b0 con 0x7f1d180cfd20 2024-04-02T13:28:18.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.195+0000 7f1d15ffb640 1 -- 172.21.15.44:0/420594566 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d00003520 con 0x7f1d180cfd20 2024-04-02T13:28:18.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.195+0000 7f1d177fe640 1 -- 172.21.15.44:0/420594566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d1815e4d0 msgr2=0x7f1d180cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:18.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.195+0000 7f1d177fe640 1 --2- 172.21.15.44:0/420594566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d1815e4d0 0x7f1d180cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:18.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.195+0000 7f1d177fe640 1 -- 172.21.15.44:0/420594566 >> v1:172.21.15.69:6789/0 conn(0x7f1d180cfd20 legacy=0x7f1d180d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:18.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.195+0000 7f1d177fe640 1 -- 172.21.15.44:0/420594566 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d180d2fb0 con 0x7f1d180d1230 2024-04-02T13:28:18.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.195+0000 7f1d16ffd640 1 --2- 172.21.15.44:0/420594566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d1815e4d0 0x7f1d180cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:28:18.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.196+0000 7f1d177fe640 1 --2- 172.21.15.44:0/420594566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d180d1230 0x7f1d180b4aa0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1d08009e50 tx=0x7f1d0802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=674f0faa51e0375a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:18.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.196+0000 7f1d15ffb640 1 -- 172.21.15.44:0/420594566 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0802cb00 con 0x7f1d180d1230 2024-04-02T13:28:18.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.196+0000 7f1d15ffb640 1 -- 172.21.15.44:0/420594566 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d0802cca0 con 0x7f1d180d1230 2024-04-02T13:28:18.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.196+0000 7f1d15ffb640 1 -- 172.21.15.44:0/420594566 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d08034610 con 0x7f1d180d1230 2024-04-02T13:28:18.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.197+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/420594566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d180d1230 msgr2=0x7f1d180b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:18.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.197+0000 7f1d1d6c5640 1 --2- 172.21.15.44:0/420594566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d180d1230 0x7f1d180b4aa0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1d08009e50 tx=0x7f1d0802bf60 comp rx=0 tx=0).stop 2024-04-02T13:28:18.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.197+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/420594566 shutdown_connections 2024-04-02T13:28:18.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.197+0000 7f1d1d6c5640 1 --2- 172.21.15.44:0/420594566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d180d1230 0x7f1d180b4aa0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:18.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.197+0000 7f1d1d6c5640 1 --2- 172.21.15.44:0/420594566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d1815e4d0 0x7f1d180cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:18.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.197+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/420594566 >> 172.21.15.44:0/420594566 conn(0x7f1d1809d410 msgr2=0x7f1d180d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:18.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.197+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/420594566 shutdown_connections 2024-04-02T13:28:18.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.197+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/420594566 wait complete. 2024-04-02T13:28:18.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.198+0000 7f1d1d6c5640 1 Processor -- start 2024-04-02T13:28:18.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.198+0000 7f1d1d6c5640 1 -- start start 2024-04-02T13:28:18.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d1d6c5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d180cfd20 0x7f1d181614d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:18.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d1d6c5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d180d1230 0x7f1d180ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:18.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d1d6c5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d1815e4d0 0x7f1d18160830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:18.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d1d6c5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1d180b42a0 con 0x7f1d180cfd20 2024-04-02T13:28:18.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d1d6c5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1d180b3fc0 con 0x7f1d1815e4d0 2024-04-02T13:28:18.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d1d6c5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1d180b0d00 con 0x7f1d180d1230 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d16ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d180cfd20 0x7f1d181614d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d177fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d1815e4d0 0x7f1d18160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d16ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d180cfd20 0x7f1d181614d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34748/0 (socket says 172.21.15.44:34748) 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d16ffd640 1 -- 172.21.15.44:0/95991497 learned_addr learned my addr 172.21.15.44:0/95991497 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d167fc640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d180d1230 0x7f1d180ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d16ffd640 1 -- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d180d1230 msgr2=0x7f1d180ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d16ffd640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d180d1230 0x7f1d180ca3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d16ffd640 1 -- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d1815e4d0 msgr2=0x7f1d18160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d16ffd640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d1815e4d0 0x7f1d18160830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d16ffd640 1 -- 172.21.15.44:0/95991497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d18161a10 con 0x7f1d180cfd20 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.199+0000 7f1d177fe640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d1815e4d0 0x7f1d18160830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.200+0000 7f1d16ffd640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d180cfd20 0x7f1d181614d0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000e960 tx=0x7f1d1000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.200+0000 7f1cff7fe640 1 -- 172.21.15.44:0/95991497 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d10017070 con 0x7f1d180cfd20 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.200+0000 7f1cff7fe640 1 -- 172.21.15.44:0/95991497 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d1000ca00 con 0x7f1d180cfd20 2024-04-02T13:28:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.200+0000 7f1cff7fe640 1 -- 172.21.15.44:0/95991497 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1000cdd0 con 0x7f1d180cfd20 2024-04-02T13:28:18.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.200+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/95991497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d180b88a0 con 0x7f1d180cfd20 2024-04-02T13:28:18.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.200+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/95991497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d180cbeb0 con 0x7f1d180cfd20 2024-04-02T13:28:18.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.201+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/95991497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d180730c0 con 0x7f1d180cfd20 2024-04-02T13:28:18.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.201+0000 7f1cff7fe640 1 -- 172.21.15.44:0/95991497 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1d1001a070 con 0x7f1d180cfd20 2024-04-02T13:28:18.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.202+0000 7f1cff7fe640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d0406c8f0 0x7f1d0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:18.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.202+0000 7f1cff7fe640 1 -- 172.21.15.44:0/95991497 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f1d1008bf30 con 0x7f1d180cfd20 2024-04-02T13:28:18.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.202+0000 7f1d167fc640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d0406c8f0 0x7f1d0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:18.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.203+0000 7f1d167fc640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d0406c8f0 0x7f1d0406edb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1d181f78c0 tx=0x7f1d0c007cf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:18.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.206+0000 7f1cff7fe640 1 -- 172.21.15.44:0/95991497 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1d180730c0 con 0x7f1d180cfd20 2024-04-02T13:28:18.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.432+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/95991497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d180cbcc0 con 0x7f1d180cfd20 2024-04-02T13:28:18.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.432+0000 7f1cff7fe640 1 -- 172.21.15.44:0/95991497 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5174 (secure 0 0 0) 0x7f1d10059af0 con 0x7f1d180cfd20 2024-04-02T13:28:18.436 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:18.436 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":73,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:28:17.315346+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8403":{"gid":8403,"name":"a","rank":0,"incarnation":71,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.44:6833/3044289683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":3044289683},{"type":"v1","addr":"172.21.15.44:6833","nonce":3044289683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-04-02T13:28:18.437 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 73 2024-04-02T13:28:18.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.434+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d0406c8f0 msgr2=0x7f1d0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:18.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.434+0000 7f1d1d6c5640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d0406c8f0 0x7f1d0406edb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1d181f78c0 tx=0x7f1d0c007cf0 comp rx=0 tx=0).stop 2024-04-02T13:28:18.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.434+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/95991497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d180cfd20 msgr2=0x7f1d181614d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:18.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.434+0000 7f1d1d6c5640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d180cfd20 0x7f1d181614d0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000e960 tx=0x7f1d1000ee20 comp rx=0 tx=0).stop 2024-04-02T13:28:18.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.434+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/95991497 shutdown_connections 2024-04-02T13:28:18.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.434+0000 7f1d1d6c5640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1d0406c8f0 0x7f1d0406edb0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:18.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.434+0000 7f1d1d6c5640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d1815e4d0 0x7f1d18160830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:18.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.434+0000 7f1d1d6c5640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d180d1230 0x7f1d180ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:18.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.434+0000 7f1d1d6c5640 1 --2- 172.21.15.44:0/95991497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d180cfd20 0x7f1d181614d0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:18.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.434+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/95991497 >> 172.21.15.44:0/95991497 conn(0x7f1d1809d410 msgr2=0x7f1d1806ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:18.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.434+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/95991497 shutdown_connections 2024-04-02T13:28:18.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:18.434+0000 7f1d1d6c5640 1 -- 172.21.15.44:0/95991497 wait complete. 2024-04-02T13:28:18.852 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:26:43.025860+0000', 'modified': '2024-04-02T13:28:17.315346+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 95, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8403}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8403': {'gid': 8403, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.44:6833/3044289683', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6832', 'nonce': 3044289683}, {'type': 'v1', 'addr': '172.21.15.44:6833', 'nonce': 3044289683}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-02T13:28:18.852 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8403:up:rejoin 2024-04-02T13:28:19.853 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:28:19.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.963+0000 7f0f34c9a640 1 Processor -- start 2024-04-02T13:28:19.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.963+0000 7f0f34c9a640 1 -- start start 2024-04-02T13:28:19.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f34c9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f300d09f0 0x7f0f3006d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:19.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f34c9a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0f3006ece0 0x7f0f300acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:19.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f34c9a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f3006dc40 con 0x7f0f300cece0 2024-04-02T13:28:19.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f34c9a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0f3006df20 con 0x7f0f3006ece0 2024-04-02T13:28:19.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f34c9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0f3006e200 con 0x7f0f300d09f0 2024-04-02T13:28:19.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f2ed9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0f3006ece0 0x7f0f300acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:19.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f2e59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0f300cece0 0x7f0f300cf0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41458/0 (socket says 172.21.15.44:41458) 2024-04-02T13:28:19.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f2e59c640 1 -- 172.21.15.44:0/919770177 learned_addr learned my addr 172.21.15.44:0/919770177 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:19.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f27fff640 1 --2- 172.21.15.44:0/919770177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f300d09f0 0x7f0f3006d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:19.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f2ed9d640 1 -- 172.21.15.44:0/919770177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f300d09f0 msgr2=0x7f0f3006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:19.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f2ed9d640 1 --2- 172.21.15.44:0/919770177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f300d09f0 0x7f0f3006d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:19.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f2ed9d640 1 -- 172.21.15.44:0/919770177 >> v1:172.21.15.44:6789/0 conn(0x7f0f300cece0 legacy=0x7f0f300cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:19.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f2ed9d640 1 -- 172.21.15.44:0/919770177 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f3006e4e0 con 0x7f0f3006ece0 2024-04-02T13:28:19.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.964+0000 7f0f2dd9b640 1 -- 172.21.15.44:0/919770177 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 720455954 0 0) 0x7f0f3006dc40 con 0x7f0f300cece0 2024-04-02T13:28:19.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.965+0000 7f0f2ed9d640 1 --2- 172.21.15.44:0/919770177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3006ece0 0x7f0f300acd50 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009d90 tx=0x7f0f2802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf5ae55cd3dd8f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:19.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.965+0000 7f0f2dd9b640 1 -- 172.21.15.44:0/919770177 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28039070 con 0x7f0f3006ece0 2024-04-02T13:28:19.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.965+0000 7f0f2dd9b640 1 -- 172.21.15.44:0/919770177 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f2802cc50 con 0x7f0f3006ece0 2024-04-02T13:28:19.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.965+0000 7f0f2dd9b640 1 -- 172.21.15.44:0/919770177 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28034440 con 0x7f0f3006ece0 2024-04-02T13:28:19.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.966+0000 7f0f34c9a640 1 -- 172.21.15.44:0/919770177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3006ece0 msgr2=0x7f0f300acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:19.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.966+0000 7f0f34c9a640 1 --2- 172.21.15.44:0/919770177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3006ece0 0x7f0f300acd50 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009d90 tx=0x7f0f2802bea0 comp rx=0 tx=0).stop 2024-04-02T13:28:19.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.966+0000 7f0f34c9a640 1 -- 172.21.15.44:0/919770177 shutdown_connections 2024-04-02T13:28:19.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.966+0000 7f0f34c9a640 1 --2- 172.21.15.44:0/919770177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3006ece0 0x7f0f300acd50 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:19.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.966+0000 7f0f34c9a640 1 --2- 172.21.15.44:0/919770177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f300d09f0 0x7f0f3006d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:19.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.966+0000 7f0f34c9a640 1 -- 172.21.15.44:0/919770177 >> 172.21.15.44:0/919770177 conn(0x7f0f300cfcd0 msgr2=0x7f0f300d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:19.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.966+0000 7f0f34c9a640 1 -- 172.21.15.44:0/919770177 shutdown_connections 2024-04-02T13:28:19.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.966+0000 7f0f34c9a640 1 -- 172.21.15.44:0/919770177 wait complete. 2024-04-02T13:28:19.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.967+0000 7f0f34c9a640 1 Processor -- start 2024-04-02T13:28:19.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.967+0000 7f0f34c9a640 1 -- start start 2024-04-02T13:28:19.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f34c9a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3006ece0 0x7f0f30145840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:19.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f34c9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f300cece0 0x7f0f3009d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:19.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f34c9a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f300d09f0 0x7f0f301516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:19.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f34c9a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3006d990 con 0x7f0f3006ece0 2024-04-02T13:28:19.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f34c9a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3006d810 con 0x7f0f300d09f0 2024-04-02T13:28:19.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f34c9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0f3015b220 con 0x7f0f300cece0 2024-04-02T13:28:19.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f2e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3006ece0 0x7f0f30145840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:19.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f2e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3006ece0 0x7f0f30145840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34758/0 (socket says 172.21.15.44:34758) 2024-04-02T13:28:19.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f2e59c640 1 -- 172.21.15.44:0/246098516 learned_addr learned my addr 172.21.15.44:0/246098516 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:19.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f2ed9d640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f300d09f0 0x7f0f301516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f27fff640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f300cece0 0x7f0f3009d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f2e59c640 1 -- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f300cece0 msgr2=0x7f0f3009d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f2e59c640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f300cece0 0x7f0f3009d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f2e59c640 1 -- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f300d09f0 msgr2=0x7f0f301516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f2e59c640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f300d09f0 0x7f0f301516f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f2e59c640 1 -- 172.21.15.44:0/246098516 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f30151e60 con 0x7f0f3006ece0 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f2ed9d640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f300d09f0 0x7f0f301516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.968+0000 7f0f27fff640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f300cece0 0x7f0f3009d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.969+0000 7f0f2e59c640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3006ece0 0x7f0f30145840 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000c500 tx=0x7f0f2000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.969+0000 7f0f26ffd640 1 -- 172.21.15.44:0/246098516 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f20018070 con 0x7f0f3006ece0 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.969+0000 7f0f26ffd640 1 -- 172.21.15.44:0/246098516 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f20007580 con 0x7f0f3006ece0 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.969+0000 7f0f26ffd640 1 -- 172.21.15.44:0/246098516 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f20004a20 con 0x7f0f3006ece0 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.969+0000 7f0f34c9a640 1 -- 172.21.15.44:0/246098516 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f30152140 con 0x7f0f3006ece0 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.969+0000 7f0f34c9a640 1 -- 172.21.15.44:0/246098516 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f30152700 con 0x7f0f3006ece0 2024-04-02T13:28:19.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.970+0000 7f0f34c9a640 1 -- 172.21.15.44:0/246098516 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f30074f20 con 0x7f0f3006ece0 2024-04-02T13:28:19.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.970+0000 7f0f26ffd640 1 -- 172.21.15.44:0/246098516 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0f2001b070 con 0x7f0f3006ece0 2024-04-02T13:28:19.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.971+0000 7f0f26ffd640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0f1006c8f0 0x7f0f1006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:19.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.971+0000 7f0f26ffd640 1 -- 172.21.15.44:0/246098516 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f0f2008b0d0 con 0x7f0f3006ece0 2024-04-02T13:28:19.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.971+0000 7f0f27fff640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0f1006c8f0 0x7f0f1006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:19.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.972+0000 7f0f27fff640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0f1006c8f0 0x7f0f1006edb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0f18006fd0 tx=0x7f0f18008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:19.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:19.975+0000 7f0f26ffd640 1 -- 172.21.15.44:0/246098516 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0f200592e0 con 0x7f0f3006ece0 2024-04-02T13:28:20.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.203+0000 7f0f34c9a640 1 -- 172.21.15.44:0/246098516 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f300afd80 con 0x7f0f3006ece0 2024-04-02T13:28:20.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.203+0000 7f0f26ffd640 1 -- 172.21.15.44:0/246098516 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5181 (secure 0 0 0) 0x7f0f20058be0 con 0x7f0f3006ece0 2024-04-02T13:28:20.207 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:20.207 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:28:19.321536+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8403":{"gid":8403,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6833/3044289683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":3044289683},{"type":"v1","addr":"172.21.15.44:6833","nonce":3044289683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8403,"qdb_cluster":[8403]},"id":6}]} 2024-04-02T13:28:20.207 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 74 2024-04-02T13:28:20.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.205+0000 7f0f34c9a640 1 -- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0f1006c8f0 msgr2=0x7f0f1006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.205+0000 7f0f34c9a640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0f1006c8f0 0x7f0f1006edb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0f18006fd0 tx=0x7f0f18008040 comp rx=0 tx=0).stop 2024-04-02T13:28:20.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.205+0000 7f0f34c9a640 1 -- 172.21.15.44:0/246098516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3006ece0 msgr2=0x7f0f30145840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.205+0000 7f0f34c9a640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3006ece0 0x7f0f30145840 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000c500 tx=0x7f0f2000c9c0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.205+0000 7f0f34c9a640 1 -- 172.21.15.44:0/246098516 shutdown_connections 2024-04-02T13:28:20.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.205+0000 7f0f34c9a640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0f1006c8f0 0x7f0f1006edb0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.205+0000 7f0f34c9a640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f300d09f0 0x7f0f301516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.205+0000 7f0f34c9a640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f300cece0 0x7f0f3009d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.205+0000 7f0f34c9a640 1 --2- 172.21.15.44:0/246098516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3006ece0 0x7f0f30145840 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.205+0000 7f0f34c9a640 1 -- 172.21.15.44:0/246098516 >> 172.21.15.44:0/246098516 conn(0x7f0f300cfcd0 msgr2=0x7f0f30072300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:20.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.205+0000 7f0f34c9a640 1 -- 172.21.15.44:0/246098516 shutdown_connections 2024-04-02T13:28:20.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.205+0000 7f0f34c9a640 1 -- 172.21.15.44:0/246098516 wait complete. 2024-04-02T13:28:20.623 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 74, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:26:43.025860+0000', 'modified': '2024-04-02T13:28:19.321536+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 95, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8403}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8403': {'gid': 8403, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.44:6833/3044289683', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6832', 'nonce': 3044289683}, {'type': 'v1', 'addr': '172.21.15.44:6833', 'nonce': 3044289683}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8403, 'qdb_cluster': [8403]} 2024-04-02T13:28:20.623 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:28:20.623 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T13:28:20.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.728+0000 7fa1a82d9640 1 Processor -- start 2024-04-02T13:28:20.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.728+0000 7fa1a82d9640 1 -- start start 2024-04-02T13:28:20.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a82d9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1a00b5200 0x7fa1a00d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a82d9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1a00d4d70 con 0x7fa1a00b47b0 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a82d9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1a00d2030 con 0x7fa1a00d28f0 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a82d9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa1a00d5050 con 0x7fa1a00b5200 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a5874640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa1a00b47b0 0x7fa1a00b7120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41464/0 (socket says 172.21.15.44:41464) 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a5874640 1 -- 172.21.15.44:0/3748608708 learned_addr learned my addr 172.21.15.44:0/3748608708 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a6876640 1 --2- 172.21.15.44:0/3748608708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1a00b5200 0x7fa1a00d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 113132097 0 0) 0x7fa1a00d4d70 con 0x7fa1a00b47b0 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa18c003520 con 0x7fa1a00b47b0 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4244224102 0 0) 0x7fa1a00d2030 con 0x7fa1a00d28f0 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1a00d4d70 con 0x7fa1a00d28f0 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2526767392 0 0) 0x7fa18c003520 con 0x7fa1a00b47b0 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1a00b5200 msgr2=0x7fa1a00d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.729+0000 7fa1a5073640 1 --2- 172.21.15.44:0/3748608708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1a00b5200 0x7fa1a00d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 >> v1:172.21.15.69:6789/0 conn(0x7fa1a00d28f0 legacy=0x7fa1a00b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a00b4cb0 con 0x7fa1a00b47b0 2024-04-02T13:28:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fa190001670 con 0x7fa1a00b47b0 2024-04-02T13:28:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 >> v1:172.21.15.44:6789/0 conn(0x7fa1a00b47b0 legacy=0x7fa1a00b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5073640 1 --2- 172.21.15.44:0/3748608708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa18c003b70 0x7fa18c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5073640 1 --2- 172.21.15.44:0/3748608708 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa18c004800 0x7fa18c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5073640 1 --2- 172.21.15.44:0/3748608708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1a00b5200 0x7fa18c005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a00d5050 con 0x7fa18c004800 2024-04-02T13:28:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa18c00bb80 con 0x7fa18c003b70 2024-04-02T13:28:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa18c00bd80 con 0x7fa1a00b5200 2024-04-02T13:28:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5874640 1 --2- 172.21.15.44:0/3748608708 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa18c004800 0x7fa18c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a6075640 1 --2- 172.21.15.44:0/3748608708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa18c003b70 0x7fa18c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a6876640 1 --2- 172.21.15.44:0/3748608708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1a00b5200 0x7fa18c005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5874640 1 -- 172.21.15.44:0/3748608708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1a00b5200 msgr2=0x7fa18c005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5874640 1 --2- 172.21.15.44:0/3748608708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1a00b5200 0x7fa18c005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5874640 1 -- 172.21.15.44:0/3748608708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa18c003b70 msgr2=0x7fa18c003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5874640 1 --2- 172.21.15.44:0/3748608708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa18c003b70 0x7fa18c003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a5874640 1 -- 172.21.15.44:0/3748608708 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa18c003520 con 0x7fa18c004800 2024-04-02T13:28:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.730+0000 7fa1a6876640 1 --2- 172.21.15.44:0/3748608708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1a00b5200 0x7fa18c005270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.731+0000 7fa1a5874640 1 --2- 172.21.15.44:0/3748608708 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa18c004800 0x7fa18c004c80 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fa19000c190 tx=0x7fa18c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bfc4d7cb296c216 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.731+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa190007ce0 con 0x7fa18c004800 2024-04-02T13:28:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.731+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa190030070 con 0x7fa18c004800 2024-04-02T13:28:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.731+0000 7fa1a5073640 1 -- 172.21.15.44:0/3748608708 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa190005d50 con 0x7fa18c004800 2024-04-02T13:28:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.731+0000 7fa1a82d9640 1 -- 172.21.15.44:0/3748608708 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa18c004800 msgr2=0x7fa18c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.731+0000 7fa1a82d9640 1 --2- 172.21.15.44:0/3748608708 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa18c004800 0x7fa18c004c80 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fa19000c190 tx=0x7fa18c005830 comp rx=0 tx=0).stop 2024-04-02T13:28:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.732+0000 7fa1a6876640 1 -- 172.21.15.44:0/3748608708 reap_dead start 2024-04-02T13:28:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.732+0000 7fa1a82d9640 1 -- 172.21.15.44:0/3748608708 shutdown_connections 2024-04-02T13:28:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.732+0000 7fa1a82d9640 1 -- 172.21.15.44:0/3748608708 >> 172.21.15.44:0/3748608708 conn(0x7fa1a00d4120 msgr2=0x7fa1a00d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.732+0000 7fa1a82d9640 1 -- 172.21.15.44:0/3748608708 shutdown_connections 2024-04-02T13:28:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.732+0000 7fa1a82d9640 1 -- 172.21.15.44:0/3748608708 wait complete. 2024-04-02T13:28:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.733+0000 7fa1a82d9640 1 Processor -- start 2024-04-02T13:28:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.733+0000 7fa1a82d9640 1 -- start start 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a82d9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa18c004800 0x7fa1a00cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a82d9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1a0160df0 0x7fa1a01f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a82d9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1a01f4640 0x7fa1a00b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a82d9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a00d23a0 con 0x7fa18c004800 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a82d9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a00d14f0 con 0x7fa1a0160df0 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a82d9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa1a00b5020 con 0x7fa1a01f4640 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a6075640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa18c004800 0x7fa1a00cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a6075640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa18c004800 0x7fa1a00cd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34776/0 (socket says 172.21.15.44:34776) 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a6075640 1 -- 172.21.15.44:0/697140331 learned_addr learned my addr 172.21.15.44:0/697140331 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a5874640 1 --2- 172.21.15.44:0/697140331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1a0160df0 0x7fa1a01f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a6075640 1 -- 172.21.15.44:0/697140331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1a01f4640 msgr2=0x7fa1a00b2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a6075640 1 --2- 172.21.15.44:0/697140331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1a01f4640 0x7fa1a00b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a6075640 1 -- 172.21.15.44:0/697140331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1a0160df0 msgr2=0x7fa1a01f4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a6075640 1 --2- 172.21.15.44:0/697140331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1a0160df0 0x7fa1a01f4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a6075640 1 -- 172.21.15.44:0/697140331 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a01f4c30 con 0x7fa18c004800 2024-04-02T13:28:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a6075640 1 --2- 172.21.15.44:0/697140331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa18c004800 0x7fa1a00cd0f0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fa19c00f7a0 tx=0x7fa19c00fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa196ffd640 1 -- 172.21.15.44:0/697140331 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c0050d0 con 0x7fa18c004800 2024-04-02T13:28:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa196ffd640 1 -- 172.21.15.44:0/697140331 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa19c018070 con 0x7fa18c004800 2024-04-02T13:28:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa196ffd640 1 -- 172.21.15.44:0/697140331 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c00d650 con 0x7fa18c004800 2024-04-02T13:28:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a82d9640 1 -- 172.21.15.44:0/697140331 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1a01f4f10 con 0x7fa18c004800 2024-04-02T13:28:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.734+0000 7fa1a82d9640 1 -- 172.21.15.44:0/697140331 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1a00ccbd0 con 0x7fa18c004800 2024-04-02T13:28:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.736+0000 7fa196ffd640 1 -- 172.21.15.44:0/697140331 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa19c00d7f0 con 0x7fa18c004800 2024-04-02T13:28:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.736+0000 7fa196ffd640 1 --2- 172.21.15.44:0/697140331 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa18406c920 0x7fa18406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.736+0000 7fa196ffd640 1 -- 172.21.15.44:0/697140331 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fa19c08bba0 con 0x7fa18c004800 2024-04-02T13:28:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.736+0000 7fa1a5874640 1 --2- 172.21.15.44:0/697140331 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa18406c920 0x7fa18406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.737+0000 7fa1a5874640 1 --2- 172.21.15.44:0/697140331 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa18406c920 0x7fa18406ede0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa190034000 tx=0x7fa190007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.737+0000 7fa1aafa0740 1 Processor -- start 2024-04-02T13:28:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.737+0000 7fa1aafa0740 1 -- start start 2024-04-02T13:28:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1aafa0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633b90 0x55678b606b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1aafa0740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633f70 0x55678b636920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1aafa0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b607070 0x55678b636e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1aafa0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55678b570060 con 0x55678b633b90 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1aafa0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55678b518650 con 0x55678b607070 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1aafa0740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55678b56c010 con 0x55678b633f70 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1a6075640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633b90 0x55678b606b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1a6075640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633b90 0x55678b606b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34792/0 (socket says 172.21.15.44:34792) 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1a6075640 1 -- 172.21.15.44:0/3924783192 learned_addr learned my addr 172.21.15.44:0/3924783192 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1a6876640 1 --2- 172.21.15.44:0/3924783192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b607070 0x55678b636e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1a5874640 1 --2- 172.21.15.44:0/3924783192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633f70 0x55678b636920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1a6075640 1 -- 172.21.15.44:0/3924783192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633f70 msgr2=0x55678b636920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1a6075640 1 --2- 172.21.15.44:0/3924783192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633f70 0x55678b636920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1a6075640 1 -- 172.21.15.44:0/3924783192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b607070 msgr2=0x55678b636e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1a6075640 1 --2- 172.21.15.44:0/3924783192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b607070 0x55678b636e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1a6075640 1 -- 172.21.15.44:0/3924783192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55678b2842d0 con 0x55678b633b90 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.738+0000 7fa1a5874640 1 --2- 172.21.15.44:0/3924783192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633f70 0x55678b636920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.739+0000 7fa1a6075640 1 --2- 172.21.15.44:0/3924783192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633b90 0x55678b606b30 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009010 tx=0x7fa19c0591c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9b766fdc90aaf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.739+0000 7fa183fff640 1 -- 172.21.15.44:0/3924783192 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c05d930 con 0x55678b633b90 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.739+0000 7fa183fff640 1 -- 172.21.15.44:0/3924783192 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa19c062160 con 0x55678b633b90 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.739+0000 7fa183fff640 1 -- 172.21.15.44:0/3924783192 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c012e10 con 0x55678b633b90 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.739+0000 7fa1aafa0740 1 -- 172.21.15.44:0/3924783192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633b90 msgr2=0x55678b606b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.739+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/3924783192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633b90 0x55678b606b30 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009010 tx=0x7fa19c0591c0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.739+0000 7fa1aafa0740 1 -- 172.21.15.44:0/3924783192 shutdown_connections 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.739+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/3924783192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b607070 0x55678b636e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.739+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/3924783192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633f70 0x55678b636920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.740+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/3924783192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633b90 0x55678b606b30 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.740+0000 7fa1aafa0740 1 -- 172.21.15.44:0/3924783192 >> 172.21.15.44:0/3924783192 conn(0x55678b608da0 msgr2=0x55678b651920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.740+0000 7fa1aafa0740 1 -- 172.21.15.44:0/3924783192 shutdown_connections 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.740+0000 7fa1aafa0740 1 -- 172.21.15.44:0/3924783192 wait complete. 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.740+0000 7fa1aafa0740 1 -- start start 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.740+0000 7fa1aafa0740 1 Processor -- start 2024-04-02T13:28:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.741+0000 7fa1aafa0740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633b90 0x55678b62d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.741+0000 7fa1aafa0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 0x55678b62f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.741+0000 7fa1aafa0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b62c350 0x55678b62f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.741+0000 7fa1aafa0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55678b60b440 con 0x55678b633f70 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.741+0000 7fa1aafa0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55678b52ba10 con 0x55678b62c350 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.741+0000 7fa1aafa0740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55678b60dc80 con 0x55678b633b90 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.741+0000 7fa1aafa0740 10 client.0 _ll_register_callbacks cb 0x55678b5680b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.741+0000 7fa1a5874640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 0x55678b62f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.741+0000 7fa1a5874640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 0x55678b62f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34802/0 (socket says 172.21.15.44:34802) 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.741+0000 7fa1a5874640 1 -- 172.21.15.44:0/5865043 learned_addr learned my addr 172.21.15.44:0/5865043 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.741+0000 7fa1a6876640 1 --2- 172.21.15.44:0/5865043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b62c350 0x55678b62f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.741+0000 7fa1a6075640 1 --2- 172.21.15.44:0/5865043 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633b90 0x55678b62d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1a5874640 1 -- 172.21.15.44:0/5865043 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633b90 msgr2=0x55678b62d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1a5874640 1 --2- 172.21.15.44:0/5865043 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633b90 0x55678b62d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1a5874640 1 -- 172.21.15.44:0/5865043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b62c350 msgr2=0x55678b62f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1a5874640 1 --2- 172.21.15.44:0/5865043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b62c350 0x55678b62f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1a5874640 1 -- 172.21.15.44:0/5865043 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55678b34e6b0 con 0x55678b633f70 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1a5874640 1 --2- 172.21.15.44:0/5865043 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 0x55678b62f120 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fa1900018f0 tx=0x7fa190001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4038cf36a27611ba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1817fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1817fa640 1 -- 172.21.15.44:0/5865043 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa190052060 con 0x55678b633f70 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1817fa640 1 -- 172.21.15.44:0/5865043 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fa190007270 con 0x55678b633f70 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1aafa0740 1 -- 172.21.15.44:0/5865043 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa19c062160 con 0x55678b633f70 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1817fa640 1 -- 172.21.15.44:0/5865043 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7fa190058050 con 0x55678b633f70 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1aafa0740 10 client.8472 fetch_fsmap learned FSMap version 74 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1aafa0740 1 -- 172.21.15.44:0/5865043 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55678b639cb0 con 0x55678b633f70 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1817fa640 1 -- 172.21.15.44:0/5865043 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa19c062160 con 0x55678b633f70 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1817fa640 1 -- 172.21.15.44:0/5865043 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7fa19005b020 con 0x55678b633f70 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1aafa0740 10 client.8472 fetch_fsmap finished waiting for FSMap version 74 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1aafa0740 10 client.8472 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:28:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1aafa0740 20 client.8472 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/5865043 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55678b635ed0 0x55678b670a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1aafa0740 4 client.8472 mds_command: new command op to 8403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.742+0000 7fa1aafa0740 1 -- 172.21.15.44:0/5865043 --> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55678b517cf0 con 0x55678b635ed0 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.743+0000 7fa1a6075640 1 --2- 172.21.15.44:0/5865043 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55678b635ed0 0x55678b670a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.743+0000 7fa1817fa640 10 client.8472 ms_handle_connect on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.743+0000 7fa1a6075640 1 --2- 172.21.15.44:0/5865043 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55678b635ed0 0x55678b670a80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a2ae543a3b98528 server_cookie=b18f77fcbde1e975 in_seq=0 out_seq=0 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.743+0000 7fa1817fa640 1 -- 172.21.15.44:0/5865043 <== mds.0 v2:172.21.15.44:6832/3044289683 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55678b517cf0 con 0x55678b635ed0 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.743+0000 7fa1817fa640 10 client.8472 handle_command_reply: tid=0 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.743+0000 7fa1aafa0740 1 client.8472 shutdown 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.744+0000 7fa1aafa0740 1 -- 172.21.15.44:0/5865043 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 msgr2=0x55678b62f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.744+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/5865043 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 0x55678b62f120 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fa1900018f0 tx=0x7fa190001a30 comp rx=0 tx=0).stop 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.744+0000 7fa1aafa0740 1 -- 172.21.15.44:0/5865043 shutdown_connections 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.744+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/5865043 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55678b635ed0 0x55678b670a80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.744+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/5865043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b62c350 0x55678b62f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.744+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/5865043 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 0x55678b62f120 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.744+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/5865043 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633b90 0x55678b62d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.744+0000 7fa1817fa640 0 client.8472 ms_handle_reset on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.744+0000 7fa1aafa0740 1 -- 172.21.15.44:0/5865043 >> 172.21.15.44:0/5865043 conn(0x55678b607070 msgr2=0x55678b60aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.744+0000 7fa1aafa0740 1 -- 172.21.15.44:0/5865043 shutdown_connections 2024-04-02T13:28:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.744+0000 7fa1aafa0740 1 -- 172.21.15.44:0/5865043 wait complete. 2024-04-02T13:28:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.744+0000 7fa1aafa0740 20 client.8472 trim_cache size 0 max 16384 2024-04-02T13:28:20.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.762+0000 7fa1aafa0740 1 Processor -- start 2024-04-02T13:28:20.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.762+0000 7fa1aafa0740 1 -- start start 2024-04-02T13:28:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1aafa0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b62c350 0x55678b62f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1aafa0740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633b90 0x55678b651300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1aafa0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 0x55678b651840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1aafa0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a02019f0 con 0x55678b633f70 2024-04-02T13:28:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1aafa0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a02009d0 con 0x55678b62c350 2024-04-02T13:28:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1aafa0740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa1a01ffd20 con 0x55678b633b90 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1a6876640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 0x55678b651840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1a6876640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 0x55678b651840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34808/0 (socket says 172.21.15.44:34808) 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1a6876640 1 -- 172.21.15.44:0/2040318645 learned_addr learned my addr 172.21.15.44:0/2040318645 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1a5874640 1 --2- 172.21.15.44:0/2040318645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633b90 0x55678b651300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1a6075640 1 --2- 172.21.15.44:0/2040318645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b62c350 0x55678b62f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1a6876640 1 -- 172.21.15.44:0/2040318645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633b90 msgr2=0x55678b651300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1a6876640 1 --2- 172.21.15.44:0/2040318645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633b90 0x55678b651300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1a6876640 1 -- 172.21.15.44:0/2040318645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b62c350 msgr2=0x55678b62f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1a6876640 1 --2- 172.21.15.44:0/2040318645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b62c350 0x55678b62f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.763+0000 7fa1a6876640 1 -- 172.21.15.44:0/2040318645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55678b665410 con 0x55678b633f70 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.764+0000 7fa1a6876640 1 --2- 172.21.15.44:0/2040318645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 0x55678b651840 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fa1a00ce800 tx=0x7fa198016510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11299cc841423eb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.764+0000 7fa1837fe640 1 -- 172.21.15.44:0/2040318645 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa198020670 con 0x55678b633f70 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.764+0000 7fa1837fe640 1 -- 172.21.15.44:0/2040318645 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa19801b070 con 0x55678b633f70 2024-04-02T13:28:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.764+0000 7fa1837fe640 1 -- 172.21.15.44:0/2040318645 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa198020a50 con 0x55678b633f70 2024-04-02T13:28:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.764+0000 7fa1aafa0740 1 -- 172.21.15.44:0/2040318645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 msgr2=0x55678b651840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.764+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/2040318645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 0x55678b651840 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fa1a00ce800 tx=0x7fa198016510 comp rx=0 tx=0).stop 2024-04-02T13:28:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.764+0000 7fa1aafa0740 1 -- 172.21.15.44:0/2040318645 shutdown_connections 2024-04-02T13:28:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.764+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/2040318645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b633f70 0x55678b651840 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.764+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/2040318645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633b90 0x55678b651300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.764+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/2040318645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b62c350 0x55678b62f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.764+0000 7fa1aafa0740 1 -- 172.21.15.44:0/2040318645 >> 172.21.15.44:0/2040318645 conn(0x55678b607070 msgr2=0x55678b636480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.764+0000 7fa1aafa0740 1 -- 172.21.15.44:0/2040318645 shutdown_connections 2024-04-02T13:28:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.764+0000 7fa1aafa0740 1 -- 172.21.15.44:0/2040318645 wait complete. 2024-04-02T13:28:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.765+0000 7fa1aafa0740 1 -- start start 2024-04-02T13:28:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.765+0000 7fa1aafa0740 1 Processor -- start 2024-04-02T13:28:20.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1aafa0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b62c350 0x55678b6b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1aafa0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b633b90 0x55678b634350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1aafa0740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633f70 0x55678b62afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1aafa0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55678b639f90 con 0x55678b62c350 2024-04-02T13:28:20.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1aafa0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a0200320 con 0x55678b633b90 2024-04-02T13:28:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1aafa0740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55678b63a5a0 con 0x55678b633f70 2024-04-02T13:28:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1aafa0740 10 client.0 _ll_register_callbacks cb 0x7fa18c0036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1a6075640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b62c350 0x55678b6b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1a6075640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b62c350 0x55678b6b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34814/0 (socket says 172.21.15.44:34814) 2024-04-02T13:28:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1a6075640 1 -- 172.21.15.44:0/3917225091 learned_addr learned my addr 172.21.15.44:0/3917225091 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1a6876640 1 --2- 172.21.15.44:0/3917225091 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633f70 0x55678b62afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1a5874640 1 --2- 172.21.15.44:0/3917225091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b633b90 0x55678b634350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1a6075640 1 -- 172.21.15.44:0/3917225091 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633f70 msgr2=0x55678b62afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1a6075640 1 --2- 172.21.15.44:0/3917225091 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633f70 0x55678b62afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1a6075640 1 -- 172.21.15.44:0/3917225091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b633b90 msgr2=0x55678b634350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1a6075640 1 --2- 172.21.15.44:0/3917225091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b633b90 0x55678b634350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1a6075640 1 -- 172.21.15.44:0/3917225091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55678b634890 con 0x55678b62c350 2024-04-02T13:28:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1a6075640 1 --2- 172.21.15.44:0/3917225091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b62c350 0x55678b6b12c0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fa19c00f770 tx=0x7fa19c0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a3cc5026e69aac server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1817fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1817fa640 1 -- 172.21.15.44:0/3917225091 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c075070 con 0x55678b62c350 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1817fa640 1 -- 172.21.15.44:0/3917225091 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fa19c057880 con 0x55678b62c350 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.766+0000 7fa1aafa0740 1 -- 172.21.15.44:0/3917225091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa19801b070 con 0x55678b62c350 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1817fa640 1 -- 172.21.15.44:0/3917225091 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7fa19c07f050 con 0x55678b62c350 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1aafa0740 10 client.8478 fetch_fsmap learned FSMap version 74 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1aafa0740 1 -- 172.21.15.44:0/3917225091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55678b6c0200 con 0x55678b62c350 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1817fa640 1 -- 172.21.15.44:0/3917225091 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa19801b070 con 0x55678b62c350 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1817fa640 1 -- 172.21.15.44:0/3917225091 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7fa19c082020 con 0x55678b62c350 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1aafa0740 10 client.8478 fetch_fsmap finished waiting for FSMap version 74 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1aafa0740 10 client.8478 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1aafa0740 20 client.8478 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/3917225091 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55678b651d90 0x55678b6bf180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1aafa0740 4 client.8478 mds_command: new command op to 8403 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1aafa0740 1 -- 172.21.15.44:0/3917225091 --> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] -- command(tid 0: {"prefix": "status"}) -- 0x55678b57d5a0 con 0x55678b651d90 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1a5874640 1 --2- 172.21.15.44:0/3917225091 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55678b651d90 0x55678b6bf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1817fa640 10 client.8478 ms_handle_connect on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1a5874640 1 --2- 172.21.15.44:0/3917225091 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55678b651d90 0x55678b6bf180 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=243adee61914c8f2 server_cookie=961ec9b9a622a1a6 in_seq=0 out_seq=0 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1817fa640 1 -- 172.21.15.44:0/3917225091 <== mds.0 v2:172.21.15.44:6832/3044289683 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55678b57d5a0 con 0x55678b651d90 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1817fa640 10 client.8478 handle_command_reply: tid=0 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.767+0000 7fa1aafa0740 1 client.8478 shutdown 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1aafa0740 1 -- 172.21.15.44:0/3917225091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b62c350 msgr2=0x55678b6b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/3917225091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b62c350 0x55678b6b12c0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fa19c00f770 tx=0x7fa19c0012d0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1aafa0740 1 -- 172.21.15.44:0/3917225091 shutdown_connections 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/3917225091 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55678b651d90 0x55678b6bf180 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/3917225091 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55678b633f70 0x55678b62afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/3917225091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55678b633b90 0x55678b634350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1aafa0740 1 --2- 172.21.15.44:0/3917225091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55678b62c350 0x55678b6b12c0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1817fa640 0 client.8478 ms_handle_reset on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:20.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1aafa0740 1 -- 172.21.15.44:0/3917225091 >> 172.21.15.44:0/3917225091 conn(0x55678b607070 msgr2=0x55678b676190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:20.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1aafa0740 1 -- 172.21.15.44:0/3917225091 shutdown_connections 2024-04-02T13:28:20.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1aafa0740 1 -- 172.21.15.44:0/3917225091 wait complete. 2024-04-02T13:28:20.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1aafa0740 20 client.8478 trim_cache size 0 max 16384 2024-04-02T13:28:20.778 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:28:20.778 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:28:20.778 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:28:20.778 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 8403, 2024-04-02T13:28:20.778 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:28:20.779 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:28:20.779 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:28:20.779 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 4.4583397209999998, 2024-04-02T13:28:20.779 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 74, 2024-04-02T13:28:20.779 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 95, 2024-04-02T13:28:20.779 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 95, 2024-04-02T13:28:20.779 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 5.4306332599999996 2024-04-02T13:28:20.779 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:28:20.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1a82d9640 1 -- 172.21.15.44:0/697140331 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa18406c920 msgr2=0x7fa18406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1a82d9640 1 --2- 172.21.15.44:0/697140331 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa18406c920 0x7fa18406ede0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa190034000 tx=0x7fa190007000 comp rx=0 tx=0).stop 2024-04-02T13:28:20.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1a82d9640 1 -- 172.21.15.44:0/697140331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa18c004800 msgr2=0x7fa1a00cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1a82d9640 1 --2- 172.21.15.44:0/697140331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa18c004800 0x7fa1a00cd0f0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fa19c00f7a0 tx=0x7fa19c00fc60 comp rx=0 tx=0).stop 2024-04-02T13:28:20.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1a82d9640 1 -- 172.21.15.44:0/697140331 shutdown_connections 2024-04-02T13:28:20.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1a82d9640 1 --2- 172.21.15.44:0/697140331 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa18406c920 0x7fa18406ede0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1a82d9640 1 --2- 172.21.15.44:0/697140331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1a01f4640 0x7fa1a00b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1a82d9640 1 --2- 172.21.15.44:0/697140331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1a0160df0 0x7fa1a01f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1a82d9640 1 --2- 172.21.15.44:0/697140331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa18c004800 0x7fa1a00cd0f0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1a82d9640 1 -- 172.21.15.44:0/697140331 >> 172.21.15.44:0/697140331 conn(0x7fa1a00d4120 msgr2=0x7fa1a00b7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:20.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1a82d9640 1 -- 172.21.15.44:0/697140331 shutdown_connections 2024-04-02T13:28:20.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.768+0000 7fa1a82d9640 1 -- 172.21.15.44:0/697140331 wait complete. 2024-04-02T13:28:20.782 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:28:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.880+0000 7f225ad40640 1 Processor -- start 2024-04-02T13:28:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.880+0000 7f225ad40640 1 -- start start 2024-04-02T13:28:20.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.881+0000 7f225ad40640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f225415e4d0 0x7f225406b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.881+0000 7f225ad40640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2254072f60 0x7f225406ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.881+0000 7f225ad40640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f225406dfe0 con 0x7f225415e0f0 2024-04-02T13:28:20.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.881+0000 7f225ad40640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f22540aed70 con 0x7f225415e4d0 2024-04-02T13:28:20.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.881+0000 7f225ad40640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f225406e2c0 con 0x7f2254072f60 2024-04-02T13:28:20.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.881+0000 7f22592dd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2254072f60 0x7f225406ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.881+0000 7f2253fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f225415e4d0 0x7f225406b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.881+0000 7f2258adc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f225415e0f0 0x7f22540d00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:48978/0 (socket says 172.21.15.44:48978) 2024-04-02T13:28:20.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.881+0000 7f2258adc640 1 -- 172.21.15.44:0/2900746866 learned_addr learned my addr 172.21.15.44:0/2900746866 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:20.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.882+0000 7f2253fff640 1 -- 172.21.15.44:0/2900746866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2254072f60 msgr2=0x7f225406ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:20.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.882+0000 7f2253fff640 1 --2- 172.21.15.44:0/2900746866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2254072f60 0x7f225406ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.882+0000 7f2253fff640 1 -- 172.21.15.44:0/2900746866 >> v1:172.21.15.69:6789/0 conn(0x7f225415e0f0 legacy=0x7f22540d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.882+0000 7f2253fff640 1 -- 172.21.15.44:0/2900746866 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f225406e5a0 con 0x7f225415e4d0 2024-04-02T13:28:20.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.882+0000 7f22537fe640 1 -- 172.21.15.44:0/2900746866 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 550499118 0 0) 0x7f225406dfe0 con 0x7f225415e0f0 2024-04-02T13:28:20.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.882+0000 7f22592dd640 1 --2- 172.21.15.44:0/2900746866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2254072f60 0x7f225406ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:28:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.882+0000 7f2253fff640 1 --2- 172.21.15.44:0/2900746866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f225415e4d0 0x7f225406b4d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f22400097a0 tx=0x7f224002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9445157c362faca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.883+0000 7f22537fe640 1 -- 172.21.15.44:0/2900746866 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2240034070 con 0x7f225415e4d0 2024-04-02T13:28:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.883+0000 7f22537fe640 1 -- 172.21.15.44:0/2900746866 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2240030610 con 0x7f225415e4d0 2024-04-02T13:28:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.883+0000 7f22537fe640 1 -- 172.21.15.44:0/2900746866 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22400309e0 con 0x7f225415e4d0 2024-04-02T13:28:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.883+0000 7f225ad40640 1 -- 172.21.15.44:0/2900746866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f225415e4d0 msgr2=0x7f225406b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.883+0000 7f225ad40640 1 --2- 172.21.15.44:0/2900746866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f225415e4d0 0x7f225406b4d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f22400097a0 tx=0x7f224002b960 comp rx=0 tx=0).stop 2024-04-02T13:28:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.883+0000 7f225ad40640 1 -- 172.21.15.44:0/2900746866 shutdown_connections 2024-04-02T13:28:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.883+0000 7f225ad40640 1 --2- 172.21.15.44:0/2900746866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f225415e4d0 0x7f225406b4d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.883+0000 7f225ad40640 1 --2- 172.21.15.44:0/2900746866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2254072f60 0x7f225406ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.883+0000 7f225ad40640 1 -- 172.21.15.44:0/2900746866 >> 172.21.15.44:0/2900746866 conn(0x7f22540b2980 msgr2=0x7f22540b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.883+0000 7f225ad40640 1 -- 172.21.15.44:0/2900746866 shutdown_connections 2024-04-02T13:28:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.883+0000 7f225ad40640 1 -- 172.21.15.44:0/2900746866 wait complete. 2024-04-02T13:28:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.884+0000 7f225ad40640 1 Processor -- start 2024-04-02T13:28:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.884+0000 7f225ad40640 1 -- start start 2024-04-02T13:28:20.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.885+0000 7f225ad40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2254072f60 0x7f2254156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.885+0000 7f225ad40640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f225415e0f0 0x7f22541f83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.885+0000 7f225ad40640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f225415e4d0 0x7f2254152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.885+0000 7f225ad40640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f22540d0800 con 0x7f225415e0f0 2024-04-02T13:28:20.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.885+0000 7f225ad40640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f22540d0af0 con 0x7f2254072f60 2024-04-02T13:28:20.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.885+0000 7f225ad40640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f225406eda0 con 0x7f225415e4d0 2024-04-02T13:28:20.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.885+0000 7f2253fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f225415e0f0 0x7f22541f83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.885+0000 7f2253fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f225415e0f0 0x7f22541f83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34826/0 (socket says 172.21.15.44:34826) 2024-04-02T13:28:20.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.885+0000 7f2253fff640 1 -- 172.21.15.44:0/1987757 learned_addr learned my addr 172.21.15.44:0/1987757 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:20.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f2258adc640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2254072f60 0x7f2254156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f22592dd640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f225415e4d0 0x7f2254152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f2253fff640 1 -- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f225415e4d0 msgr2=0x7f2254152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f2253fff640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f225415e4d0 0x7f2254152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f2253fff640 1 -- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2254072f60 msgr2=0x7f2254156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f2253fff640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2254072f60 0x7f2254156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f2253fff640 1 -- 172.21.15.44:0/1987757 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2254152670 con 0x7f225415e0f0 2024-04-02T13:28:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f2258adc640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2254072f60 0x7f2254156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f2253fff640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f225415e0f0 0x7f22541f83c0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f22400097a0 tx=0x7f22400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f22517fa640 1 -- 172.21.15.44:0/1987757 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2240047070 con 0x7f225415e0f0 2024-04-02T13:28:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f22517fa640 1 -- 172.21.15.44:0/1987757 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2240030410 con 0x7f225415e0f0 2024-04-02T13:28:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f225ad40640 1 -- 172.21.15.44:0/1987757 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2254157100 con 0x7f225415e0f0 2024-04-02T13:28:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f22517fa640 1 -- 172.21.15.44:0/1987757 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2240034040 con 0x7f225415e0f0 2024-04-02T13:28:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.886+0000 7f225ad40640 1 -- 172.21.15.44:0/1987757 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22541576c0 con 0x7f225415e0f0 2024-04-02T13:28:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.887+0000 7f225ad40640 1 -- 172.21.15.44:0/1987757 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22540684f0 con 0x7f225415e0f0 2024-04-02T13:28:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.888+0000 7f22517fa640 1 -- 172.21.15.44:0/1987757 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f224004a070 con 0x7f225415e0f0 2024-04-02T13:28:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.888+0000 7f22517fa640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f223806c8f0 0x7f223806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.888+0000 7f22517fa640 1 -- 172.21.15.44:0/1987757 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f22400ad810 con 0x7f225415e0f0 2024-04-02T13:28:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.888+0000 7f2258adc640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f223806c8f0 0x7f223806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.889+0000 7f2258adc640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f223806c8f0 0x7f223806edb0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2248007fd0 tx=0x7f2248009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:20.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:20.892+0000 7f22517fa640 1 -- 172.21.15.44:0/1987757 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f22400305e0 con 0x7f225415e0f0 2024-04-02T13:28:21.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.117+0000 7f225ad40640 1 -- 172.21.15.44:0/1987757 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f22540727f0 con 0x7f225415e0f0 2024-04-02T13:28:21.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.117+0000 7f22517fa640 1 -- 172.21.15.44:0/1987757 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v74) ==== 53+0+73 (secure 0 0 0) 0x7f224007b910 con 0x7f225415e0f0 2024-04-02T13:28:21.121 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:28:21.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.119+0000 7f225ad40640 1 -- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f223806c8f0 msgr2=0x7f223806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:21.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.119+0000 7f225ad40640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f223806c8f0 0x7f223806edb0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2248007fd0 tx=0x7f2248009040 comp rx=0 tx=0).stop 2024-04-02T13:28:21.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.119+0000 7f225ad40640 1 -- 172.21.15.44:0/1987757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f225415e0f0 msgr2=0x7f22541f83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:21.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.119+0000 7f225ad40640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f225415e0f0 0x7f22541f83c0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f22400097a0 tx=0x7f22400046c0 comp rx=0 tx=0).stop 2024-04-02T13:28:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.120+0000 7f225ad40640 1 -- 172.21.15.44:0/1987757 shutdown_connections 2024-04-02T13:28:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.120+0000 7f225ad40640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f223806c8f0 0x7f223806edb0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.120+0000 7f225ad40640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f225415e4d0 0x7f2254152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.120+0000 7f225ad40640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f225415e0f0 0x7f22541f83c0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.120+0000 7f225ad40640 1 --2- 172.21.15.44:0/1987757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2254072f60 0x7f2254156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.120+0000 7f225ad40640 1 -- 172.21.15.44:0/1987757 >> 172.21.15.44:0/1987757 conn(0x7f22540b2980 msgr2=0x7f22540b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.120+0000 7f225ad40640 1 -- 172.21.15.44:0/1987757 shutdown_connections 2024-04-02T13:28:21.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21.120+0000 7f225ad40640 1 -- 172.21.15.44:0/1987757 wait complete. 2024-04-02T13:28:21.134 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:28:21.134 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:28:21.134 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:28:21.134 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:28:21.134 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:28:21.134 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:28:21.134 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:28:21.134 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:28:21.135 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:28:21.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:21.135 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:28:21.150 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:28:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:28:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:28:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:28:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:28:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:28:21.150 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:28:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:28:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:28:21.150 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:28:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42278sec preferred_lft 42278sec 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:28:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:28:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:28:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:28:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:28:21.152 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:28:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:28:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:28:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:28:21.152 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:28:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:28:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:28:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:28:21.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:21.153 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:28:21.217 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:28:21.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:21.218 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:28:21.273 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:28:21.274 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:28:21.274 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:28:21.274 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:28:21.274 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:28:21.274 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:28:21.274 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:28:21.274 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:28:21.274 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:28:21.274 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:28:21.274 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:28:21.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:28:21.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:28:21.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:21.408 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:28:21.423 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:28:21.423 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:28:21.423 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:28:21.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:21.423 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:28:21.490 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:28:21.490 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:28:21.490 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:28:21.544 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:28:21.545 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:28:21.545 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:28:21.600 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:28:21.667 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:28:21.722 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:28:21.722 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:28:21.722 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:28:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:28:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:28:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:28:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:28:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:28:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:28:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:28:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:28:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:28:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:28:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:28:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:28:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:28:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:28:21.725 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:28:21.725 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:28:21.725 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:28:21.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:21.725 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:28:21.776 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:28:21.777 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:28:21.820 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:28:21.846 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:28:21.902 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:28:21.902 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:28:21.902 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:28:21.902 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:28:21.902 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:28:21.902 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:28:21.902 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:28:21.903 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:28:21.904 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:28:21.904 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:28:21.904 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:28:21.904 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:28:21.904 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:28:21.904 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:28:21.904 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:28:21.904 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:28:21.904 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:28:21.904 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:28:21.904 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:28:21.904 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:28:21.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:21.905 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:28:21.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:28:21.902+0000 7f4f2e0da480 -1 init, newargv = 0x555a91cbd150 newargc=15 2024-04-02T13:28:21.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[73358]: starting ceph client 2024-04-02T13:28:21.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[73358]: starting fuse 2024-04-02T13:28:21.921 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:28:21.921 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:28:21.921 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:28:21.921 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:28:21.921 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:28:21.921 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:28:21.921 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:28:21.921 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:28:21.921 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:28:21.922 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:28:21.923 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:28:21.923 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:28:21.923 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:28:21.923 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:28:21.923 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:28:22.042 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.73358.asok 2024-04-02T13:28:22.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:28:22.052 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.73358.asok 2024-04-02T13:28:22.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:22.052 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.73358.asok status 2024-04-02T13:28:22.191 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:28:22.191 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:28:22.191 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:28:22.191 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:28:22.191 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:28:22.191 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "73358", 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 8490, 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 8490 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:28:22.192 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 4144075036 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 4144075036 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.8490 192.168.0.1:0/4144075036", 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/4144075036", 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 74, 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 95, 2024-04-02T13:28:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:28:22.194 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:28:22.194 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:28:22.194 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:28:22.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:22.202 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:28:22.202 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:28:22.220 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:28:22.220 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:28:22.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:22.220 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:28:22.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:22.303 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive,repair 2024-04-02T13:28:22.313 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:28:22.310+0000 7f856a5a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:28:22.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.420+0000 7ff0d8515640 1 Processor -- start 2024-04-02T13:28:22.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.420+0000 7ff0d8515640 1 -- start start 2024-04-02T13:28:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.421+0000 7ff0d8515640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0d006cff0 0x7ff0d006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.421+0000 7ff0d8515640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff0d015e5d0 0x7ff0d00d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.421+0000 7ff0d8515640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d006d910 con 0x7ff0d00b4180 2024-04-02T13:28:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.421+0000 7ff0d8515640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff0d00d1bf0 con 0x7ff0d006cff0 2024-04-02T13:28:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.421+0000 7ff0d8515640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff0d00d1630 con 0x7ff0d015e5d0 2024-04-02T13:28:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.421+0000 7ff0d62b1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0d006cff0 0x7ff0d006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.421+0000 7ff0d62b1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff0d006cff0 0x7ff0d006d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34838/0 (socket says 172.21.15.44:34838) 2024-04-02T13:28:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.421+0000 7ff0d62b1640 1 -- 172.21.15.44:0/1425691894 learned_addr learned my addr 172.21.15.44:0/1425691894 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.421+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/1425691894 >> v2:172.21.15.69:3300/0 conn(0x7ff0d015e5d0 0x7ff0d00d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.421+0000 7ff0d62b1640 1 -- 172.21.15.44:0/1425691894 >> v2:172.21.15.69:3300/0 conn(0x7ff0d015e5d0 msgr2=0x7ff0d00d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.421+0000 7ff0d62b1640 1 --2- 172.21.15.44:0/1425691894 >> v2:172.21.15.69:3300/0 conn(0x7ff0d015e5d0 0x7ff0d00d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.421+0000 7ff0d62b1640 1 -- 172.21.15.44:0/1425691894 >> v1:172.21.15.69:6789/0 conn(0x7ff0d00b4180 legacy=0x7ff0d00ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.421+0000 7ff0d62b1640 1 -- 172.21.15.44:0/1425691894 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d00d1910 con 0x7ff0d006cff0 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.422+0000 7ff0d62b1640 1 --2- 172.21.15.44:0/1425691894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d006cff0 0x7ff0d006d3d0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0161e40 tx=0x7ff0cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f52f1959f9b1026 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.422+0000 7ff0d52af640 1 -- 172.21.15.44:0/1425691894 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc035020 con 0x7ff0d006cff0 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.422+0000 7ff0d52af640 1 -- 172.21.15.44:0/1425691894 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0cc031870 con 0x7ff0d006cff0 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.422+0000 7ff0d52af640 1 -- 172.21.15.44:0/1425691894 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc031c40 con 0x7ff0d006cff0 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.423+0000 7ff0d8515640 1 -- 172.21.15.44:0/1425691894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d006cff0 msgr2=0x7ff0d006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.423+0000 7ff0d8515640 1 --2- 172.21.15.44:0/1425691894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d006cff0 0x7ff0d006d3d0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0161e40 tx=0x7ff0cc02b910 comp rx=0 tx=0).stop 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.423+0000 7ff0d8515640 1 -- 172.21.15.44:0/1425691894 shutdown_connections 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.423+0000 7ff0d8515640 1 --2- 172.21.15.44:0/1425691894 >> v2:172.21.15.69:3300/0 conn(0x7ff0d015e5d0 0x7ff0d00d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.423+0000 7ff0d8515640 1 --2- 172.21.15.44:0/1425691894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d006cff0 0x7ff0d006d3d0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.423+0000 7ff0d8515640 1 -- 172.21.15.44:0/1425691894 >> 172.21.15.44:0/1425691894 conn(0x7ff0d009aa20 msgr2=0x7ff0d009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.423+0000 7ff0d8515640 1 -- 172.21.15.44:0/1425691894 shutdown_connections 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.423+0000 7ff0d8515640 1 -- 172.21.15.44:0/1425691894 wait complete. 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.424+0000 7ff0d8515640 1 Processor -- start 2024-04-02T13:28:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.424+0000 7ff0d8515640 1 -- start start 2024-04-02T13:28:22.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d8515640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d006cff0 0x7ff0d015fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d8515640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d00b4180 0x7ff0d01601e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d8515640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d015e5d0 0x7ff0d00b6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d8515640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d00d1270 con 0x7ff0d006cff0 2024-04-02T13:28:22.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d8515640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d00b1820 con 0x7ff0d015e5d0 2024-04-02T13:28:22.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d8515640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d00d13f0 con 0x7ff0d00b4180 2024-04-02T13:28:22.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d62b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d006cff0 0x7ff0d015fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d62b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d006cff0 0x7ff0d015fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34842/0 (socket says 172.21.15.44:34842) 2024-04-02T13:28:22.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d62b1640 1 -- 172.21.15.44:0/1299353487 learned_addr learned my addr 172.21.15.44:0/1299353487 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d6ab2640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d015e5d0 0x7ff0d00b6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d00b4180 0x7ff0d01601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d62b1640 1 -- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d00b4180 msgr2=0x7ff0d01601e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d62b1640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d00b4180 0x7ff0d01601e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d62b1640 1 -- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d015e5d0 msgr2=0x7ff0d00b6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d62b1640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d015e5d0 0x7ff0d00b6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d62b1640 1 -- 172.21.15.44:0/1299353487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d0160a40 con 0x7ff0d006cff0 2024-04-02T13:28:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d6ab2640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d015e5d0 0x7ff0d00b6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d62b1640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d006cff0 0x7ff0d015fca0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc02b950 tx=0x7ff0cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0c6ffd640 1 -- 172.21.15.44:0/1299353487 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc040070 con 0x7ff0d006cff0 2024-04-02T13:28:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0c6ffd640 1 -- 172.21.15.44:0/1299353487 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0cc030d00 con 0x7ff0d006cff0 2024-04-02T13:28:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.425+0000 7ff0d8515640 1 -- 172.21.15.44:0/1299353487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d0161000 con 0x7ff0d006cff0 2024-04-02T13:28:22.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.426+0000 7ff0c6ffd640 1 -- 172.21.15.44:0/1299353487 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc035040 con 0x7ff0d006cff0 2024-04-02T13:28:22.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.426+0000 7ff0d8515640 1 -- 172.21.15.44:0/1299353487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d00c6030 con 0x7ff0d006cff0 2024-04-02T13:28:22.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.427+0000 7ff0c6ffd640 1 -- 172.21.15.44:0/1299353487 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0cc04b060 con 0x7ff0d006cff0 2024-04-02T13:28:22.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.427+0000 7ff0c6ffd640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0b406c8f0 0x7ff0b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.427+0000 7ff0c6ffd640 1 -- 172.21.15.44:0/1299353487 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7ff0cc0adb50 con 0x7ff0d006cff0 2024-04-02T13:28:22.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.428+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0b406c8f0 0x7ff0b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.428+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0b406c8f0 0x7ff0b406edb0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0161600 tx=0x7ff0c0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.428+0000 7ff0db1de740 1 Processor -- start 2024-04-02T13:28:22.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.429+0000 7ff0db1de740 1 -- start start 2024-04-02T13:28:22.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.429+0000 7ff0db1de740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba88b310 0x557cba88b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.429+0000 7ff0db1de740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba85b330 0x557cba888580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.429+0000 7ff0db1de740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba85ee90 0x557cba888ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.429+0000 7ff0db1de740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557cba7c5770 con 0x557cba85ee90 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.429+0000 7ff0db1de740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557cba76d960 con 0x557cba85b330 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.429+0000 7ff0db1de740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x557cba7c18c0 con 0x557cba88b310 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0d6ab2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba85ee90 0x557cba888ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0d6ab2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba85ee90 0x557cba888ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34850/0 (socket says 172.21.15.44:34850) 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0d6ab2640 1 -- 172.21.15.44:0/2242070907 learned_addr learned my addr 172.21.15.44:0/2242070907 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0d62b1640 1 --2- 172.21.15.44:0/2242070907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba88b310 0x557cba88b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/2242070907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba85b330 0x557cba888580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0d6ab2640 1 -- 172.21.15.44:0/2242070907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba88b310 msgr2=0x557cba88b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0d6ab2640 1 --2- 172.21.15.44:0/2242070907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba88b310 0x557cba88b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0d6ab2640 1 -- 172.21.15.44:0/2242070907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba85b330 msgr2=0x557cba888580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0d6ab2640 1 --2- 172.21.15.44:0/2242070907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba85b330 0x557cba888580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0d6ab2640 1 -- 172.21.15.44:0/2242070907 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cba5a17e0 con 0x557cba85ee90 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0d6ab2640 1 --2- 172.21.15.44:0/2242070907 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba85ee90 0x557cba888ac0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8001ce0 tx=0x7ff0c80020e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78dfe205819326e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0b3fff640 1 -- 172.21.15.44:0/2242070907 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8019070 con 0x557cba85ee90 2024-04-02T13:28:22.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0b3fff640 1 -- 172.21.15.44:0/2242070907 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c8003b80 con 0x557cba85ee90 2024-04-02T13:28:22.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.430+0000 7ff0b3fff640 1 -- 172.21.15.44:0/2242070907 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c800e860 con 0x557cba85ee90 2024-04-02T13:28:22.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.431+0000 7ff0db1de740 1 -- 172.21.15.44:0/2242070907 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba85ee90 msgr2=0x557cba888ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.431+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2242070907 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba85ee90 0x557cba888ac0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8001ce0 tx=0x7ff0c80020e0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.431+0000 7ff0db1de740 1 -- 172.21.15.44:0/2242070907 shutdown_connections 2024-04-02T13:28:22.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.431+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2242070907 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba85ee90 0x557cba888ac0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.431+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2242070907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba85b330 0x557cba888580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.431+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2242070907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba88b310 0x557cba88b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.431+0000 7ff0db1de740 1 -- 172.21.15.44:0/2242070907 >> 172.21.15.44:0/2242070907 conn(0x557cba85d5a0 msgr2=0x557cba8a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:22.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.431+0000 7ff0db1de740 1 -- 172.21.15.44:0/2242070907 shutdown_connections 2024-04-02T13:28:22.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.431+0000 7ff0db1de740 1 -- 172.21.15.44:0/2242070907 wait complete. 2024-04-02T13:28:22.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.432+0000 7ff0db1de740 1 -- start start 2024-04-02T13:28:22.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.432+0000 7ff0db1de740 1 Processor -- start 2024-04-02T13:28:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.432+0000 7ff0db1de740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 0x557cba88efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.432+0000 7ff0db1de740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba88b310 0x557cba885b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.432+0000 7ff0db1de740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba886090 0x557cba88be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.432+0000 7ff0db1de740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557cba860040 con 0x557cba88b310 2024-04-02T13:28:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.432+0000 7ff0db1de740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557cba780d20 con 0x557cba886090 2024-04-02T13:28:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.432+0000 7ff0db1de740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x557cba85f430 con 0x557cba85ee90 2024-04-02T13:28:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0db1de740 10 client.0 _ll_register_callbacks cb 0x557cba74da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0d5ab0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba88b310 0x557cba885b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0d5ab0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba88b310 0x557cba885b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34856/0 (socket says 172.21.15.44:34856) 2024-04-02T13:28:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0d5ab0640 1 -- 172.21.15.44:0/3819531862 learned_addr learned my addr 172.21.15.44:0/3819531862 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0d62b1640 1 --2- 172.21.15.44:0/3819531862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 0x557cba88efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0d6ab2640 1 --2- 172.21.15.44:0/3819531862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba886090 0x557cba88be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0d5ab0640 1 -- 172.21.15.44:0/3819531862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 msgr2=0x557cba88efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/3819531862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 0x557cba88efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0d5ab0640 1 -- 172.21.15.44:0/3819531862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba886090 msgr2=0x557cba88be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/3819531862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba886090 0x557cba88be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0d5ab0640 1 -- 172.21.15.44:0/3819531862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cba601f70 con 0x557cba88b310 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/3819531862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba88b310 0x557cba885b50 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00156f0 tx=0x7ff0c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76286836d53fc67 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0b17fa640 1 -- 172.21.15.44:0/3819531862 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c00145a0 con 0x557cba88b310 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0b17fa640 1 -- 172.21.15.44:0/3819531862 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7ff0c0014740 con 0x557cba88b310 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0db1de740 1 -- 172.21.15.44:0/3819531862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0c8003b80 con 0x557cba88b310 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0b17fa640 1 -- 172.21.15.44:0/3819531862 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7ff0c0032050 con 0x557cba88b310 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0db1de740 10 client.8502 fetch_fsmap learned FSMap version 74 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.433+0000 7ff0db1de740 1 -- 172.21.15.44:0/3819531862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x557cba499170 con 0x557cba88b310 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.434+0000 7ff0b17fa640 1 -- 172.21.15.44:0/3819531862 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c8003b80 con 0x557cba88b310 2024-04-02T13:28:22.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.434+0000 7ff0b17fa640 1 -- 172.21.15.44:0/3819531862 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7ff0c0014940 con 0x557cba88b310 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.434+0000 7ff0db1de740 10 client.8502 fetch_fsmap finished waiting for FSMap version 74 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.434+0000 7ff0db1de740 10 client.8502 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.434+0000 7ff0db1de740 20 client.8502 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.434+0000 7ff0db1de740 1 --2- 172.21.15.44:0/3819531862 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x557cba881700 0x557cba881b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.434+0000 7ff0db1de740 4 client.8502 mds_command: new command op to 8403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.434+0000 7ff0db1de740 1 -- 172.21.15.44:0/3819531862 --> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557cba7d2cb0 con 0x557cba881700 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.434+0000 7ff0d62b1640 1 --2- 172.21.15.44:0/3819531862 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x557cba881700 0x557cba881b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.434+0000 7ff0b17fa640 10 client.8502 ms_handle_connect on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.434+0000 7ff0d62b1640 1 --2- 172.21.15.44:0/3819531862 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x557cba881700 0x557cba881b20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=969ca8d8b3b23fc3 server_cookie=9cc3d4a7066f599e in_seq=0 out_seq=0 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0b17fa640 1 -- 172.21.15.44:0/3819531862 <== mds.0 v2:172.21.15.44:6832/3044289683 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557cba7d2cb0 con 0x557cba881700 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0b17fa640 10 client.8502 handle_command_reply: tid=0 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0db1de740 1 client.8502 shutdown 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0db1de740 1 -- 172.21.15.44:0/3819531862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba88b310 msgr2=0x557cba885b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0db1de740 1 --2- 172.21.15.44:0/3819531862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba88b310 0x557cba885b50 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00156f0 tx=0x7ff0c00046c0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0db1de740 1 -- 172.21.15.44:0/3819531862 shutdown_connections 2024-04-02T13:28:22.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0db1de740 1 --2- 172.21.15.44:0/3819531862 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x557cba881700 0x557cba881b20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0db1de740 1 --2- 172.21.15.44:0/3819531862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba886090 0x557cba88be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0db1de740 1 --2- 172.21.15.44:0/3819531862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba88b310 0x557cba885b50 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0b17fa640 0 client.8502 ms_handle_reset on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:22.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0db1de740 1 --2- 172.21.15.44:0/3819531862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 0x557cba88efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0db1de740 1 -- 172.21.15.44:0/3819531862 >> 172.21.15.44:0/3819531862 conn(0x557cba85b330 msgr2=0x557cba8a7630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:22.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0db1de740 1 -- 172.21.15.44:0/3819531862 shutdown_connections 2024-04-02T13:28:22.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0db1de740 1 -- 172.21.15.44:0/3819531862 wait complete. 2024-04-02T13:28:22.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.435+0000 7ff0db1de740 20 client.8502 trim_cache size 0 max 16384 2024-04-02T13:28:22.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.453+0000 7ff0db1de740 1 Processor -- start 2024-04-02T13:28:22.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.453+0000 7ff0db1de740 1 -- start start 2024-04-02T13:28:22.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.453+0000 7ff0db1de740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 0x557cba88cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.453+0000 7ff0db1de740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 0x557cba8a75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.453+0000 7ff0db1de740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba88b310 0x557cba881ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.453+0000 7ff0db1de740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d02012f0 con 0x557cba886090 2024-04-02T13:28:22.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.453+0000 7ff0db1de740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d02002d0 con 0x557cba88b310 2024-04-02T13:28:22.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.453+0000 7ff0db1de740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d01ff7d0 con 0x557cba85ee90 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.454+0000 7ff0d5ab0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 0x557cba8a75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.454+0000 7ff0d5ab0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 0x557cba8a75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34860/0 (socket says 172.21.15.44:34860) 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.454+0000 7ff0d5ab0640 1 -- 172.21.15.44:0/2048461517 learned_addr learned my addr 172.21.15.44:0/2048461517 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.454+0000 7ff0d5ab0640 1 -- 172.21.15.44:0/2048461517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 msgr2=0x557cba88cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.454+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/2048461517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 0x557cba88cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.454+0000 7ff0d5ab0640 1 -- 172.21.15.44:0/2048461517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba88b310 msgr2=0x557cba881ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.454+0000 7ff0d6ab2640 1 --2- 172.21.15.44:0/2048461517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba88b310 0x557cba881ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.454+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/2048461517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba88b310 0x557cba881ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.454+0000 7ff0d5ab0640 1 -- 172.21.15.44:0/2048461517 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cba88c170 con 0x557cba886090 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.454+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/2048461517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 0x557cba8a75a0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0008010 tx=0x7ff0c0023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30fa28e6ab3e7baa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.454+0000 7ff0b37fe640 1 -- 172.21.15.44:0/2048461517 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c002f070 con 0x557cba886090 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.454+0000 7ff0b37fe640 1 -- 172.21.15.44:0/2048461517 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c003a070 con 0x557cba886090 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.454+0000 7ff0b37fe640 1 -- 172.21.15.44:0/2048461517 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c00147c0 con 0x557cba886090 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.455+0000 7ff0db1de740 1 -- 172.21.15.44:0/2048461517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 msgr2=0x557cba8a75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.455+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2048461517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 0x557cba8a75a0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0008010 tx=0x7ff0c0023000 comp rx=0 tx=0).stop 2024-04-02T13:28:22.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.455+0000 7ff0db1de740 1 -- 172.21.15.44:0/2048461517 shutdown_connections 2024-04-02T13:28:22.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.455+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2048461517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba88b310 0x557cba881ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.455+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2048461517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 0x557cba8a75a0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.455+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2048461517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 0x557cba88cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.455+0000 7ff0db1de740 1 -- 172.21.15.44:0/2048461517 >> 172.21.15.44:0/2048461517 conn(0x557cba85b330 msgr2=0x557cba8803f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:22.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.455+0000 7ff0db1de740 1 -- 172.21.15.44:0/2048461517 shutdown_connections 2024-04-02T13:28:22.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.455+0000 7ff0db1de740 1 -- 172.21.15.44:0/2048461517 wait complete. 2024-04-02T13:28:22.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.455+0000 7ff0db1de740 1 -- start start 2024-04-02T13:28:22.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.455+0000 7ff0db1de740 1 Processor -- start 2024-04-02T13:28:22.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.456+0000 7ff0db1de740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 0x557cba871f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.456+0000 7ff0db1de740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 0x557cba8c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.456+0000 7ff0db1de740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba88b310 0x557cba8c5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.456+0000 7ff0db1de740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557cba8b6c70 con 0x557cba886090 2024-04-02T13:28:22.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.456+0000 7ff0db1de740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d01ffcc0 con 0x557cba88b310 2024-04-02T13:28:22.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.456+0000 7ff0db1de740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x557cba8733b0 con 0x557cba85ee90 2024-04-02T13:28:22.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.456+0000 7ff0db1de740 10 client.0 _ll_register_callbacks cb 0x7ff0c00118b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:22.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.456+0000 7ff0d5ab0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 0x557cba8c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.456+0000 7ff0d5ab0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 0x557cba8c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34864/0 (socket says 172.21.15.44:34864) 2024-04-02T13:28:22.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.456+0000 7ff0d5ab0640 1 -- 172.21.15.44:0/2149326870 learned_addr learned my addr 172.21.15.44:0/2149326870 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:22.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0d62b1640 1 --2- 172.21.15.44:0/2149326870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 0x557cba871f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0d6ab2640 1 --2- 172.21.15.44:0/2149326870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba88b310 0x557cba8c5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0d5ab0640 1 -- 172.21.15.44:0/2149326870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 msgr2=0x557cba871f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/2149326870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 0x557cba871f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0d5ab0640 1 -- 172.21.15.44:0/2149326870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba88b310 msgr2=0x557cba8c5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/2149326870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba88b310 0x557cba8c5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0d5ab0640 1 -- 172.21.15.44:0/2149326870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cba88b6f0 con 0x557cba886090 2024-04-02T13:28:22.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0d5ab0640 1 --2- 172.21.15.44:0/2149326870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 0x557cba8c8420 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0000c00 tx=0x7ff0c0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d385d610a146322 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:22.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0b17fa640 1 -- 172.21.15.44:0/2149326870 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0043070 con 0x557cba886090 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0b17fa640 1 -- 172.21.15.44:0/2149326870 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7ff0c003f460 con 0x557cba886090 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0db1de740 1 -- 172.21.15.44:0/2149326870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0c003a070 con 0x557cba886090 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0b17fa640 1 -- 172.21.15.44:0/2149326870 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7ff0c002f040 con 0x557cba886090 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0db1de740 10 client.8508 fetch_fsmap learned FSMap version 74 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0db1de740 1 -- 172.21.15.44:0/2149326870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x557cba8c8d60 con 0x557cba886090 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0b17fa640 1 -- 172.21.15.44:0/2149326870 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c003a070 con 0x557cba886090 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0b17fa640 1 -- 172.21.15.44:0/2149326870 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7ff0c003f6a0 con 0x557cba886090 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0db1de740 10 client.8508 fetch_fsmap finished waiting for FSMap version 74 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0db1de740 10 client.8508 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0db1de740 20 client.8508 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2149326870 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x557cba888920 0x557cba8c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0db1de740 4 client.8508 mds_command: new command op to 8403 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}] 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.457+0000 7ff0db1de740 1 -- 172.21.15.44:0/2149326870 --> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}) -- 0x557cba7cd000 con 0x557cba888920 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.458+0000 7ff0d62b1640 1 --2- 172.21.15.44:0/2149326870 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x557cba888920 0x557cba8c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.458+0000 7ff0b17fa640 10 client.8508 ms_handle_connect on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.458+0000 7ff0d62b1640 1 --2- 172.21.15.44:0/2149326870 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x557cba888920 0x557cba8c5f60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ac66e9c87f816a server_cookie=ea9f4cbf5655f0a8 in_seq=0 out_seq=0 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.458+0000 7ff0b17fa640 1 -- 172.21.15.44:0/2149326870 <== mds.0 v2:172.21.15.44:6832/3044289683 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557cba7cd000 con 0x557cba888920 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.458+0000 7ff0b17fa640 10 client.8508 handle_command_reply: tid=0 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.458+0000 7ff0db1de740 1 client.8508 shutdown 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.459+0000 7ff0db1de740 1 -- 172.21.15.44:0/2149326870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 msgr2=0x557cba8c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.459+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2149326870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 0x557cba8c8420 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0000c00 tx=0x7ff0c0016000 comp rx=0 tx=0).stop 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.459+0000 7ff0db1de740 1 -- 172.21.15.44:0/2149326870 shutdown_connections 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.459+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2149326870 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x557cba888920 0x557cba8c5f60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.459+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2149326870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557cba88b310 0x557cba8c5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.459+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2149326870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557cba886090 0x557cba8c8420 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.459+0000 7ff0b17fa640 0 client.8508 ms_handle_reset on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.459+0000 7ff0db1de740 1 --2- 172.21.15.44:0/2149326870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557cba85ee90 0x557cba871f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.459+0000 7ff0db1de740 1 -- 172.21.15.44:0/2149326870 >> 172.21.15.44:0/2149326870 conn(0x557cba85b330 msgr2=0x557cba87f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.459+0000 7ff0db1de740 1 -- 172.21.15.44:0/2149326870 shutdown_connections 2024-04-02T13:28:22.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.459+0000 7ff0db1de740 1 -- 172.21.15.44:0/2149326870 wait complete. 2024-04-02T13:28:22.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.459+0000 7ff0db1de740 20 client.8508 trim_cache size 0 max 16384 2024-04-02T13:28:22.469 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:28:22.469 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-02T13:28:22.469 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "df583992-1aee-4cd3-953f-a17d376ae176", 2024-04-02T13:28:22.469 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-02T13:28:22.469 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:28:22.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.460+0000 7ff0d8515640 1 -- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0b406c8f0 msgr2=0x7ff0b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.460+0000 7ff0d8515640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0b406c8f0 0x7ff0b406edb0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0161600 tx=0x7ff0c0008040 comp rx=0 tx=0).stop 2024-04-02T13:28:22.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.460+0000 7ff0d8515640 1 -- 172.21.15.44:0/1299353487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d006cff0 msgr2=0x7ff0d015fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.460+0000 7ff0d8515640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d006cff0 0x7ff0d015fca0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc02b950 tx=0x7ff0cc0046c0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.460+0000 7ff0d8515640 1 -- 172.21.15.44:0/1299353487 shutdown_connections 2024-04-02T13:28:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.460+0000 7ff0d8515640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff0b406c8f0 0x7ff0b406edb0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.460+0000 7ff0d8515640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d015e5d0 0x7ff0d00b6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.460+0000 7ff0d8515640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0d00b4180 0x7ff0d01601e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.460+0000 7ff0d8515640 1 --2- 172.21.15.44:0/1299353487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0d006cff0 0x7ff0d015fca0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.460+0000 7ff0d8515640 1 -- 172.21.15.44:0/1299353487 >> 172.21.15.44:0/1299353487 conn(0x7ff0d009aa20 msgr2=0x7ff0d00b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.460+0000 7ff0d8515640 1 -- 172.21.15.44:0/1299353487 shutdown_connections 2024-04-02T13:28:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.460+0000 7ff0d8515640 1 -- 172.21.15.44:0/1299353487 wait complete. 2024-04-02T13:28:22.471 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:28:22.460+0000 7f856659a640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [...2,head] /dir/file auth v1 ap=1 DIRTYPARENT s=0 nl=2 n() | dirtyparent=1 authpin=1 scrubqueue=0 0x55788b8de100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(14)0x10000000001:[<0x10000000000/file v6>,<0x1/dir v13>]//[]","memoryvalue":"(14)0x10000000001:[<0x10000000000/file v1>,<0x1/dir v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-02T13:28:22.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:22.474 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-02T13:28:22.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.580+0000 7f201ba3a640 1 Processor -- start 2024-04-02T13:28:22.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.580+0000 7f201ba3a640 1 -- start start 2024-04-02T13:28:22.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f201ba3a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20140d0530 0x7f20140ad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f201ba3a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20140d09a0 con 0x7f20140d00a0 2024-04-02T13:28:22.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f201ba3a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20140cdd60 con 0x7f20140aeca0 2024-04-02T13:28:22.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f201ba3a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f201415b3b0 con 0x7f20140d0530 2024-04-02T13:28:22.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f2018fd5640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f20140d00a0 0x7f20140acde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41478/0 (socket says 172.21.15.44:41478) 2024-04-02T13:28:22.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f2018fd5640 1 -- 172.21.15.44:0/4124838378 learned_addr learned my addr 172.21.15.44:0/4124838378 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:22.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 194998354 0 0) 0x7f20140d09a0 con 0x7f20140d00a0 2024-04-02T13:28:22.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f2019fd7640 1 --2- 172.21.15.44:0/4124838378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20140d0530 0x7f20140ad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2000003520 con 0x7f20140d00a0 2024-04-02T13:28:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 93051822 0 0) 0x7f20140cdd60 con 0x7f20140aeca0 2024-04-02T13:28:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20140d09a0 con 0x7f20140aeca0 2024-04-02T13:28:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4215128600 0 0) 0x7f2000003520 con 0x7f20140d00a0 2024-04-02T13:28:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20140d0530 msgr2=0x7f20140ad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 --2- 172.21.15.44:0/4124838378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20140d0530 0x7f20140ad4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 >> v1:172.21.15.69:6789/0 conn(0x7f20140aeca0 legacy=0x7f201406e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20140ce040 con 0x7f20140d00a0 2024-04-02T13:28:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f20040015f0 con 0x7f20140d00a0 2024-04-02T13:28:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 >> v1:172.21.15.44:6789/0 conn(0x7f20140d00a0 legacy=0x7f20140acde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 --2- 172.21.15.44:0/4124838378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2000003b70 0x7f2000003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 --2- 172.21.15.44:0/4124838378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20140d0530 0x7f2000004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 --2- 172.21.15.44:0/4124838378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2000004d40 0x7f20000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f201415b3b0 con 0x7f2000003b70 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f200000ba70 con 0x7f2000004d40 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f200000bc70 con 0x7f20140d0530 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.581+0000 7f20197d6640 1 --2- 172.21.15.44:0/4124838378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2000003b70 0x7f2000003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.582+0000 7f2018fd5640 1 --2- 172.21.15.44:0/4124838378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20140d0530 0x7f2000004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.582+0000 7f2019fd7640 1 --2- 172.21.15.44:0/4124838378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2000004d40 0x7f20000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.582+0000 7f20197d6640 1 -- 172.21.15.44:0/4124838378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20140d0530 msgr2=0x7f2000004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.582+0000 7f20197d6640 1 --2- 172.21.15.44:0/4124838378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20140d0530 0x7f2000004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.582+0000 7f20197d6640 1 -- 172.21.15.44:0/4124838378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2000004d40 msgr2=0x7f20000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.582+0000 7f20197d6640 1 --2- 172.21.15.44:0/4124838378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2000004d40 0x7f20000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.582+0000 7f20197d6640 1 -- 172.21.15.44:0/4124838378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2000003520 con 0x7f2000003b70 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.583+0000 7f20197d6640 1 --2- 172.21.15.44:0/4124838378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2000003b70 0x7f2000003f70 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f20100018d0 tx=0x7f2000005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be91becbaac65ba3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.583+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2010037070 con 0x7f2000003b70 2024-04-02T13:28:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.583+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20100052d0 con 0x7f2000003b70 2024-04-02T13:28:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.583+0000 7f200bfff640 1 -- 172.21.15.44:0/4124838378 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20100014f0 con 0x7f2000003b70 2024-04-02T13:28:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.583+0000 7f201ba3a640 1 -- 172.21.15.44:0/4124838378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2000003b70 msgr2=0x7f2000003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.583+0000 7f201ba3a640 1 --2- 172.21.15.44:0/4124838378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2000003b70 0x7f2000003f70 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f20100018d0 tx=0x7f2000005830 comp rx=0 tx=0).stop 2024-04-02T13:28:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.583+0000 7f2019fd7640 1 -- 172.21.15.44:0/4124838378 reap_dead start 2024-04-02T13:28:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.583+0000 7f201ba3a640 1 -- 172.21.15.44:0/4124838378 shutdown_connections 2024-04-02T13:28:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.583+0000 7f201ba3a640 1 -- 172.21.15.44:0/4124838378 >> 172.21.15.44:0/4124838378 conn(0x7f201406b920 msgr2=0x7f201406bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.583+0000 7f201ba3a640 1 -- 172.21.15.44:0/4124838378 shutdown_connections 2024-04-02T13:28:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.583+0000 7f201ba3a640 1 -- 172.21.15.44:0/4124838378 wait complete. 2024-04-02T13:28:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.584+0000 7f201ba3a640 1 Processor -- start 2024-04-02T13:28:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.585+0000 7f201ba3a640 1 -- start start 2024-04-02T13:28:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.585+0000 7f201ba3a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2000003b70 0x7f201409c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.585+0000 7f201ba3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f201409b3a0 0x7f20141e34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.585+0000 7f201ba3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20141e39e0 0x7f2014144800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.585+0000 7f201ba3a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f20140ce8e0 con 0x7f201409b3a0 2024-04-02T13:28:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.585+0000 7f201ba3a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f20140ce600 con 0x7f20141e39e0 2024-04-02T13:28:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.585+0000 7f201ba3a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f201406ece0 con 0x7f2000003b70 2024-04-02T13:28:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.585+0000 7f2018fd5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f201409b3a0 0x7f20141e34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.585+0000 7f2018fd5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f201409b3a0 0x7f20141e34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34888/0 (socket says 172.21.15.44:34888) 2024-04-02T13:28:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.585+0000 7f2018fd5640 1 -- 172.21.15.44:0/2886888669 learned_addr learned my addr 172.21.15.44:0/2886888669 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.585+0000 7f2019fd7640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20141e39e0 0x7f2014144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.585+0000 7f20197d6640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2000003b70 0x7f201409c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.586+0000 7f2018fd5640 1 -- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2000003b70 msgr2=0x7f201409c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.586+0000 7f2018fd5640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2000003b70 0x7f201409c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.586+0000 7f2018fd5640 1 -- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20141e39e0 msgr2=0x7f2014144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.586+0000 7f2018fd5640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20141e39e0 0x7f2014144800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.586+0000 7f2018fd5640 1 -- 172.21.15.44:0/2886888669 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201409b8a0 con 0x7f201409b3a0 2024-04-02T13:28:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.586+0000 7f20197d6640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2000003b70 0x7f201409c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.586+0000 7f2018fd5640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f201409b3a0 0x7f20141e34a0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f20040072c0 tx=0x7f2004007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.586+0000 7f2009ffb640 1 -- 172.21.15.44:0/2886888669 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200400f940 con 0x7f201409b3a0 2024-04-02T13:28:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.586+0000 7f2009ffb640 1 -- 172.21.15.44:0/2886888669 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2004018070 con 0x7f201409b3a0 2024-04-02T13:28:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.586+0000 7f201ba3a640 1 -- 172.21.15.44:0/2886888669 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201409bb80 con 0x7f201409b3a0 2024-04-02T13:28:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.586+0000 7f2009ffb640 1 -- 172.21.15.44:0/2886888669 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200400fd40 con 0x7f201409b3a0 2024-04-02T13:28:22.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.586+0000 7f201ba3a640 1 -- 172.21.15.44:0/2886888669 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f201409c140 con 0x7f201409b3a0 2024-04-02T13:28:22.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.588+0000 7f2009ffb640 1 -- 172.21.15.44:0/2886888669 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f200401d460 con 0x7f201409b3a0 2024-04-02T13:28:22.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.588+0000 7f2009ffb640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1ff806c8f0 0x7f1ff806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.588+0000 7f2009ffb640 1 -- 172.21.15.44:0/2886888669 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f200408bcb0 con 0x7f201409b3a0 2024-04-02T13:28:22.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.588+0000 7f20197d6640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1ff806c8f0 0x7f1ff806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.589+0000 7f20197d6640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1ff806c8f0 0x7f1ff806edb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2010034000 tx=0x7f2010031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.589+0000 7f201e61a740 1 Processor -- start 2024-04-02T13:28:22.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.589+0000 7f201e61a740 1 -- start start 2024-04-02T13:28:22.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f201e61a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3493099a0 0x55b3492dc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f201e61a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 0x55b34930c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f201e61a740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3492dce80 0x55b34930cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f201e61a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b349246830 con 0x55b349309d80 2024-04-02T13:28:22.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f201e61a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b3491ee820 con 0x55b3493099a0 2024-04-02T13:28:22.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f201e61a740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55b3492427e0 con 0x55b3492dce80 2024-04-02T13:28:22.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f2018fd5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 0x55b34930c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f2018fd5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 0x55b34930c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34902/0 (socket says 172.21.15.44:34902) 2024-04-02T13:28:22.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f2018fd5640 1 -- 172.21.15.44:0/1790964633 learned_addr learned my addr 172.21.15.44:0/1790964633 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:22.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f20197d6640 1 --2- 172.21.15.44:0/1790964633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3493099a0 0x55b3492dc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f2019fd7640 1 --2- 172.21.15.44:0/1790964633 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3492dce80 0x55b34930cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f2018fd5640 1 -- 172.21.15.44:0/1790964633 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3492dce80 msgr2=0x55b34930cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f2018fd5640 1 --2- 172.21.15.44:0/1790964633 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3492dce80 0x55b34930cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f2018fd5640 1 -- 172.21.15.44:0/1790964633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3493099a0 msgr2=0x55b3492dc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f2018fd5640 1 --2- 172.21.15.44:0/1790964633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3493099a0 0x55b3492dc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f2018fd5640 1 -- 172.21.15.44:0/1790964633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3490247d0 con 0x55b349309d80 2024-04-02T13:28:22.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.590+0000 7f2018fd5640 1 --2- 172.21.15.44:0/1790964633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 0x55b34930c730 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f2004089f60 tx=0x7f2004059120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3add5488c63d4a48 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.591+0000 7f1ffeffd640 1 -- 172.21.15.44:0/1790964633 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200405db70 con 0x55b349309d80 2024-04-02T13:28:22.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.591+0000 7f1ffeffd640 1 -- 172.21.15.44:0/1790964633 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20040623a0 con 0x55b349309d80 2024-04-02T13:28:22.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.591+0000 7f1ffeffd640 1 -- 172.21.15.44:0/1790964633 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2004057ac0 con 0x55b349309d80 2024-04-02T13:28:22.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.591+0000 7f201e61a740 1 -- 172.21.15.44:0/1790964633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 msgr2=0x55b34930c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.591+0000 7f201e61a740 1 --2- 172.21.15.44:0/1790964633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 0x55b34930c730 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f2004089f60 tx=0x7f2004059120 comp rx=0 tx=0).stop 2024-04-02T13:28:22.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.591+0000 7f201e61a740 1 -- 172.21.15.44:0/1790964633 shutdown_connections 2024-04-02T13:28:22.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.591+0000 7f201e61a740 1 --2- 172.21.15.44:0/1790964633 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3492dce80 0x55b34930cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.591+0000 7f201e61a740 1 --2- 172.21.15.44:0/1790964633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 0x55b34930c730 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.591+0000 7f201e61a740 1 --2- 172.21.15.44:0/1790964633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3493099a0 0x55b3492dc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.591+0000 7f201e61a740 1 -- 172.21.15.44:0/1790964633 >> 172.21.15.44:0/1790964633 conn(0x55b3492debb0 msgr2=0x55b349327730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.591+0000 7f201e61a740 1 -- 172.21.15.44:0/1790964633 shutdown_connections 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.591+0000 7f201e61a740 1 -- 172.21.15.44:0/1790964633 wait complete. 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.592+0000 7f201e61a740 1 -- start start 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.592+0000 7f201e61a740 1 Processor -- start 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f201e61a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3493099a0 0x55b3492f76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f201e61a740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b349309d80 0x55b34930e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f201e61a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b34930ecb0 0x55b34930f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f201e61a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b3492e1150 con 0x55b3493099a0 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f201e61a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b349201d30 con 0x55b34930ecb0 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f201e61a740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55b3492e0540 con 0x55b349309d80 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f201e61a740 10 client.0 _ll_register_callbacks cb 0x55b349201720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f20197d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3493099a0 0x55b3492f76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f20197d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3493099a0 0x55b3492f76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34906/0 (socket says 172.21.15.44:34906) 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f20197d6640 1 -- 172.21.15.44:0/2628078916 learned_addr learned my addr 172.21.15.44:0/2628078916 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f2019fd7640 1 --2- 172.21.15.44:0/2628078916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b34930ecb0 0x55b34930f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f2018fd5640 1 --2- 172.21.15.44:0/2628078916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b349309d80 0x55b34930e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f20197d6640 1 -- 172.21.15.44:0/2628078916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b349309d80 msgr2=0x55b34930e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f20197d6640 1 --2- 172.21.15.44:0/2628078916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b349309d80 0x55b34930e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f20197d6640 1 -- 172.21.15.44:0/2628078916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b34930ecb0 msgr2=0x55b34930f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f20197d6640 1 --2- 172.21.15.44:0/2628078916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b34930ecb0 0x55b34930f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f20197d6640 1 -- 172.21.15.44:0/2628078916 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b348f5a3f0 con 0x55b3493099a0 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f20197d6640 1 --2- 172.21.15.44:0/2628078916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3493099a0 0x55b3492f76b0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f20100019c0 tx=0x7f201002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f448c787ddd275 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f1ff3fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.593+0000 7f1ff3fff640 1 -- 172.21.15.44:0/2628078916 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201004d070 con 0x55b3493099a0 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f1ff3fff640 1 -- 172.21.15.44:0/2628078916 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f2010040990 con 0x55b3493099a0 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f201e61a740 1 -- 172.21.15.44:0/2628078916 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20040623a0 con 0x55b3493099a0 2024-04-02T13:28:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f1ff3fff640 1 -- 172.21.15.44:0/2628078916 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f2010057050 con 0x55b3493099a0 2024-04-02T13:28:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f201e61a740 10 client.8520 fetch_fsmap learned FSMap version 74 2024-04-02T13:28:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f201e61a740 1 -- 172.21.15.44:0/2628078916 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55b348f1a0f0 con 0x55b3493099a0 2024-04-02T13:28:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f1ff3fff640 1 -- 172.21.15.44:0/2628078916 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20040623a0 con 0x55b3493099a0 2024-04-02T13:28:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f1ff3fff640 1 -- 172.21.15.44:0/2628078916 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f201005a020 con 0x55b3493099a0 2024-04-02T13:28:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f201e61a740 10 client.8520 fetch_fsmap finished waiting for FSMap version 74 2024-04-02T13:28:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f201e61a740 10 client.8520 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:28:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f201e61a740 20 client.8520 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f201e61a740 1 --2- 172.21.15.44:0/2628078916 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55b3492fa2d0 0x55b3492fa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f201e61a740 4 client.8520 mds_command: new command op to 8403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:28:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f201e61a740 1 -- 172.21.15.44:0/2628078916 --> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b349253d70 con 0x55b3492fa2d0 2024-04-02T13:28:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f2018fd5640 1 --2- 172.21.15.44:0/2628078916 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55b3492fa2d0 0x55b3492fa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f1ff3fff640 10 client.8520 ms_handle_connect on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.594+0000 7f2018fd5640 1 --2- 172.21.15.44:0/2628078916 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55b3492fa2d0 0x55b3492fa6d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b589d848838865dc server_cookie=5ab79522aa6368c5 in_seq=0 out_seq=0 2024-04-02T13:28:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f1ff3fff640 1 -- 172.21.15.44:0/2628078916 <== mds.0 v2:172.21.15.44:6832/3044289683 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b349253d70 con 0x55b3492fa2d0 2024-04-02T13:28:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f1ff3fff640 10 client.8520 handle_command_reply: tid=0 2024-04-02T13:28:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f201e61a740 1 client.8520 shutdown 2024-04-02T13:28:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f201e61a740 1 -- 172.21.15.44:0/2628078916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3493099a0 msgr2=0x55b3492f76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f201e61a740 1 --2- 172.21.15.44:0/2628078916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3493099a0 0x55b3492f76b0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f20100019c0 tx=0x7f201002e000 comp rx=0 tx=0).stop 2024-04-02T13:28:22.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f201e61a740 1 -- 172.21.15.44:0/2628078916 shutdown_connections 2024-04-02T13:28:22.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f201e61a740 1 --2- 172.21.15.44:0/2628078916 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55b3492fa2d0 0x55b3492fa6d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f201e61a740 1 --2- 172.21.15.44:0/2628078916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b34930ecb0 0x55b34930f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f1ff3fff640 0 client.8520 ms_handle_reset on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:22.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f201e61a740 1 --2- 172.21.15.44:0/2628078916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b349309d80 0x55b34930e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f201e61a740 1 --2- 172.21.15.44:0/2628078916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3493099a0 0x55b3492f76b0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f201e61a740 1 -- 172.21.15.44:0/2628078916 >> 172.21.15.44:0/2628078916 conn(0x55b3492dce80 msgr2=0x55b34932ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:22.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f201e61a740 1 -- 172.21.15.44:0/2628078916 shutdown_connections 2024-04-02T13:28:22.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f201e61a740 1 -- 172.21.15.44:0/2628078916 wait complete. 2024-04-02T13:28:22.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.595+0000 7f201e61a740 20 client.8520 trim_cache size 0 max 16384 2024-04-02T13:28:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.613+0000 7f201e61a740 1 Processor -- start 2024-04-02T13:28:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.613+0000 7f201e61a740 1 -- start start 2024-04-02T13:28:22.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f201e61a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3493099a0 0x55b34932ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f201e61a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 0x55b34932b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f201e61a740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b34930ecb0 0x55b349336b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f201e61a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f20141ff590 con 0x55b349309d80 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f201e61a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f20141fe650 con 0x55b3493099a0 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f201e61a740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f20141fd9b0 con 0x55b34930ecb0 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f2018fd5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 0x55b34932b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f2018fd5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 0x55b34932b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34922/0 (socket says 172.21.15.44:34922) 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f2018fd5640 1 -- 172.21.15.44:0/686651469 learned_addr learned my addr 172.21.15.44:0/686651469 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f2018fd5640 1 -- 172.21.15.44:0/686651469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b34930ecb0 msgr2=0x55b349336b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f2018fd5640 1 --2- 172.21.15.44:0/686651469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b34930ecb0 0x55b349336b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f20197d6640 1 --2- 172.21.15.44:0/686651469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3493099a0 0x55b34932ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f2018fd5640 1 -- 172.21.15.44:0/686651469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3493099a0 msgr2=0x55b34932ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f2018fd5640 1 --2- 172.21.15.44:0/686651469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3493099a0 0x55b34932ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f2018fd5640 1 -- 172.21.15.44:0/686651469 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3492d8c50 con 0x55b349309d80 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f20197d6640 1 --2- 172.21.15.44:0/686651469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3493099a0 0x55b34932ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f2018fd5640 1 --2- 172.21.15.44:0/686651469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 0x55b34932b180 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f2004059750 tx=0x7f200405c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6d00d328ef8c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f1ffe7fc640 1 -- 172.21.15.44:0/686651469 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20040623a0 con 0x55b349309d80 2024-04-02T13:28:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f1ffe7fc640 1 -- 172.21.15.44:0/686651469 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2004057860 con 0x55b349309d80 2024-04-02T13:28:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.614+0000 7f1ffe7fc640 1 -- 172.21.15.44:0/686651469 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2004055e30 con 0x55b349309d80 2024-04-02T13:28:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.615+0000 7f201e61a740 1 -- 172.21.15.44:0/686651469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 msgr2=0x55b34932b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.615+0000 7f201e61a740 1 --2- 172.21.15.44:0/686651469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 0x55b34932b180 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f2004059750 tx=0x7f200405c000 comp rx=0 tx=0).stop 2024-04-02T13:28:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.615+0000 7f201e61a740 1 -- 172.21.15.44:0/686651469 shutdown_connections 2024-04-02T13:28:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.615+0000 7f201e61a740 1 --2- 172.21.15.44:0/686651469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b34930ecb0 0x55b349336b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.615+0000 7f201e61a740 1 --2- 172.21.15.44:0/686651469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b349309d80 0x55b34932b180 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.615+0000 7f201e61a740 1 --2- 172.21.15.44:0/686651469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3493099a0 0x55b34932ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.615+0000 7f201e61a740 1 -- 172.21.15.44:0/686651469 >> 172.21.15.44:0/686651469 conn(0x55b3492dce80 msgr2=0x55b34930fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.615+0000 7f201e61a740 1 -- 172.21.15.44:0/686651469 shutdown_connections 2024-04-02T13:28:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.615+0000 7f201e61a740 1 -- 172.21.15.44:0/686651469 wait complete. 2024-04-02T13:28:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.615+0000 7f201e61a740 1 -- start start 2024-04-02T13:28:22.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.616+0000 7f201e61a740 1 Processor -- start 2024-04-02T13:28:22.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.616+0000 7f201e61a740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3493099a0 0x55b3493a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.616+0000 7f201e61a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b349309d80 0x55b3492f8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.616+0000 7f201e61a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b34930ecb0 0x55b3492f94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.616+0000 7f201e61a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f20141fe000 con 0x55b34930ecb0 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.616+0000 7f201e61a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b34931e5a0 con 0x55b349309d80 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.616+0000 7f201e61a740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55b3492e1bc0 con 0x55b3493099a0 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.616+0000 7f201e61a740 10 client.0 _ll_register_callbacks cb 0x7f200000b9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.616+0000 7f2019fd7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b34930ecb0 0x55b3492f94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f2019fd7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b34930ecb0 0x55b3492f94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34934/0 (socket says 172.21.15.44:34934) 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f2019fd7640 1 -- 172.21.15.44:0/3680427386 learned_addr learned my addr 172.21.15.44:0/3680427386 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f20197d6640 1 --2- 172.21.15.44:0/3680427386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3493099a0 0x55b3493a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f2018fd5640 1 --2- 172.21.15.44:0/3680427386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b349309d80 0x55b3492f8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f2019fd7640 1 -- 172.21.15.44:0/3680427386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3493099a0 msgr2=0x55b3493a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f2019fd7640 1 --2- 172.21.15.44:0/3680427386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3493099a0 0x55b3493a2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f2019fd7640 1 -- 172.21.15.44:0/3680427386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b349309d80 msgr2=0x55b3492f8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f2019fd7640 1 --2- 172.21.15.44:0/3680427386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b349309d80 0x55b3492f8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f2019fd7640 1 -- 172.21.15.44:0/3680427386 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3493374d0 con 0x55b34930ecb0 2024-04-02T13:28:22.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f2019fd7640 1 --2- 172.21.15.44:0/3680427386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b34930ecb0 0x55b3492f94a0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f201406f630 tx=0x7f200c018f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932cceba78bf93eb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f1ff3fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f1ff3fff640 1 -- 172.21.15.44:0/3680427386 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c016a70 con 0x55b34930ecb0 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f1ff3fff640 1 -- 172.21.15.44:0/3680427386 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f200c016c10 con 0x55b34930ecb0 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f201e61a740 1 -- 172.21.15.44:0/3680427386 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2004057860 con 0x55b34930ecb0 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f1ff3fff640 1 -- 172.21.15.44:0/3680427386 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f200c016e10 con 0x55b34930ecb0 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f201e61a740 10 client.8526 fetch_fsmap learned FSMap version 74 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f201e61a740 1 -- 172.21.15.44:0/3680427386 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55b3492e1640 con 0x55b34930ecb0 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.617+0000 7f1ff3fff640 1 -- 172.21.15.44:0/3680427386 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2004057860 con 0x55b34930ecb0 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f1ff3fff640 1 -- 172.21.15.44:0/3680427386 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f200c02f080 con 0x55b34930ecb0 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 10 client.8526 fetch_fsmap finished waiting for FSMap version 74 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 10 client.8526 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 20 client.8526 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 1 --2- 172.21.15.44:0/3680427386 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55b3493417b0 0x55b3493a8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 4 client.8526 mds_command: new command op to 8403 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:28:22.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 1 -- 172.21.15.44:0/3680427386 --> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b34924e0c0 con 0x55b3493417b0 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f20197d6640 1 --2- 172.21.15.44:0/3680427386 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55b3493417b0 0x55b3493a8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f20197d6640 1 --2- 172.21.15.44:0/3680427386 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55b3493417b0 0x55b3493a8fd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ded9387092af98be server_cookie=3fd1252f787cd2fd in_seq=0 out_seq=0 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f1ff3fff640 10 client.8526 ms_handle_connect on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f1ff3fff640 1 -- 172.21.15.44:0/3680427386 <== mds.0 v2:172.21.15.44:6832/3044289683 1 ==== command_reply(tid 0: 0 ) ==== 8+0+263 (crc 0 0 0) 0x55b34924e0c0 con 0x55b3493417b0 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f1ff3fff640 10 client.8526 handle_command_reply: tid=0 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 1 client.8526 shutdown 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 1 -- 172.21.15.44:0/3680427386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b34930ecb0 msgr2=0x55b3492f94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 1 --2- 172.21.15.44:0/3680427386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b34930ecb0 0x55b3492f94a0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f201406f630 tx=0x7f200c018f40 comp rx=0 tx=0).stop 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 1 -- 172.21.15.44:0/3680427386 shutdown_connections 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 1 --2- 172.21.15.44:0/3680427386 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x55b3493417b0 0x55b3493a8fd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 1 --2- 172.21.15.44:0/3680427386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b34930ecb0 0x55b3492f94a0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 1 --2- 172.21.15.44:0/3680427386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b349309d80 0x55b3492f8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f1ff3fff640 0 client.8526 ms_handle_reset on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:22.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 1 --2- 172.21.15.44:0/3680427386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55b3493099a0 0x55b3493a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.618+0000 7f201e61a740 1 -- 172.21.15.44:0/3680427386 >> 172.21.15.44:0/3680427386 conn(0x55b3492dce80 msgr2=0x55b349308e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:22.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201e61a740 1 -- 172.21.15.44:0/3680427386 shutdown_connections 2024-04-02T13:28:22.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201e61a740 1 -- 172.21.15.44:0/3680427386 wait complete. 2024-04-02T13:28:22.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201e61a740 20 client.8526 trim_cache size 0 max 16384 2024-04-02T13:28:22.629 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:28:22.629 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T13:28:22.629 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-02T13:28:22.629 INFO:teuthology.orchestra.run.smithi044.stdout: "df583992-1aee-4cd3-953f-a17d376ae176": { 2024-04-02T13:28:22.629 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/dir", 2024-04-02T13:28:22.629 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "df583992-1aee-4cd3-953f-a17d376ae176", 2024-04-02T13:28:22.630 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,repair" 2024-04-02T13:28:22.630 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:28:22.630 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:28:22.630 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:28:22.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201ba3a640 1 -- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1ff806c8f0 msgr2=0x7f1ff806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201ba3a640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1ff806c8f0 0x7f1ff806edb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2010034000 tx=0x7f2010031000 comp rx=0 tx=0).stop 2024-04-02T13:28:22.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201ba3a640 1 -- 172.21.15.44:0/2886888669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f201409b3a0 msgr2=0x7f20141e34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:22.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201ba3a640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f201409b3a0 0x7f20141e34a0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f20040072c0 tx=0x7f2004007780 comp rx=0 tx=0).stop 2024-04-02T13:28:22.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201ba3a640 1 -- 172.21.15.44:0/2886888669 shutdown_connections 2024-04-02T13:28:22.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201ba3a640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1ff806c8f0 0x7f1ff806edb0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201ba3a640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20141e39e0 0x7f2014144800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201ba3a640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f201409b3a0 0x7f20141e34a0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201ba3a640 1 --2- 172.21.15.44:0/2886888669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2000003b70 0x7f201409c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:22.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201ba3a640 1 -- 172.21.15.44:0/2886888669 >> 172.21.15.44:0/2886888669 conn(0x7f201406b920 msgr2=0x7f201415ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:22.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201ba3a640 1 -- 172.21.15.44:0/2886888669 shutdown_connections 2024-04-02T13:28:22.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:22.619+0000 7f201ba3a640 1 -- 172.21.15.44:0/2886888669 wait complete. 2024-04-02T13:28:22.634 INFO:tasks.cephfs.filesystem:scrub status for tag:df583992-1aee-4cd3-953f-a17d376ae176 - {'path': '/dir', 'tag': 'df583992-1aee-4cd3-953f-a17d376ae176', 'options': 'recursive,repair'} 2024-04-02T13:28:23.219 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:28:23.223 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:28:52.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:52.635 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-02T13:28:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.750+0000 7f0c4f4f2640 1 Processor -- start 2024-04-02T13:28:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.751+0000 7f0c4f4f2640 1 -- start start 2024-04-02T13:28:52.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.751+0000 7f0c4f4f2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0c481554c0 0x7f0c48071120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.751+0000 7f0c4f4f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c480d0840 0x7f0c48164c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.751+0000 7f0c4f4f2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c480cdc20 con 0x7f0c480ce9a0 2024-04-02T13:28:52.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.751+0000 7f0c4f4f2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0c480cdec0 con 0x7f0c481554c0 2024-04-02T13:28:52.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.751+0000 7f0c4f4f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0c480ce1a0 con 0x7f0c480d0840 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c4ca8d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0c480ce9a0 0x7f0c480d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:38808/0 (socket says 172.21.15.44:38808) 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c4ca8d640 1 -- 172.21.15.44:0/3897865725 learned_addr learned my addr 172.21.15.44:0/3897865725 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/3897865725 >> v2:172.21.15.69:3300/0 conn(0x7f0c481554c0 0x7f0c48071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c4da8f640 1 --2- 172.21.15.44:0/3897865725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c480d0840 0x7f0c48164c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 175131848 0 0) 0x7f0c480cdc20 con 0x7f0c480ce9a0 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c30003520 con 0x7f0c480ce9a0 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 708701387 0 0) 0x7f0c30003520 con 0x7f0c480ce9a0 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c480d0840 msgr2=0x7f0c48164c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 --2- 172.21.15.44:0/3897865725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c480d0840 0x7f0c48164c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 >> v2:172.21.15.69:3300/0 conn(0x7f0c481554c0 msgr2=0x7f0c48071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 --2- 172.21.15.44:0/3897865725 >> v2:172.21.15.69:3300/0 conn(0x7f0c481554c0 0x7f0c48071120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c480ce480 con 0x7f0c480ce9a0 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f0c3c001670 con 0x7f0c480ce9a0 2024-04-02T13:28:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/3897865725 >> v2:172.21.15.69:3300/0 conn(0x7f0c481554c0 0x7f0c48071120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 >> v1:172.21.15.44:6789/0 conn(0x7f0c480ce9a0 legacy=0x7f0c480d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 --2- 172.21.15.44:0/3897865725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c300039b0 0x7f0c30003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 --2- 172.21.15.44:0/3897865725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c480d0840 0x7f0c30004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 --2- 172.21.15.44:0/3897865725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c30004d40 0x7f0c300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c480cdec0 con 0x7f0c300039b0 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0c480ce1a0 con 0x7f0c30004d40 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0c3000b9c0 con 0x7f0c480d0840 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.752+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/3897865725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c300039b0 0x7f0c30003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.753+0000 7f0c4ca8d640 1 --2- 172.21.15.44:0/3897865725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c480d0840 0x7f0c30004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.753+0000 7f0c4da8f640 1 --2- 172.21.15.44:0/3897865725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c30004d40 0x7f0c300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.753+0000 7f0c4d28e640 1 -- 172.21.15.44:0/3897865725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c480d0840 msgr2=0x7f0c30004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.753+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/3897865725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c480d0840 0x7f0c30004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.753+0000 7f0c4d28e640 1 -- 172.21.15.44:0/3897865725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c30004d40 msgr2=0x7f0c300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.753+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/3897865725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c30004d40 0x7f0c300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.753+0000 7f0c4d28e640 1 -- 172.21.15.44:0/3897865725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c30003520 con 0x7f0c300039b0 2024-04-02T13:28:52.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.754+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/3897865725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c300039b0 0x7f0c30003df0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009820 tx=0x7f0c30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa2bd7dec05b64e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:52.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.754+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4003b070 con 0x7f0c300039b0 2024-04-02T13:28:52.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.754+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c4002eb90 con 0x7f0c300039b0 2024-04-02T13:28:52.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.754+0000 7f0c47fff640 1 -- 172.21.15.44:0/3897865725 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c40009860 con 0x7f0c300039b0 2024-04-02T13:28:52.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.754+0000 7f0c4f4f2640 1 -- 172.21.15.44:0/3897865725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c300039b0 msgr2=0x7f0c30003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.754+0000 7f0c4f4f2640 1 --2- 172.21.15.44:0/3897865725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c300039b0 0x7f0c30003df0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009820 tx=0x7f0c30005830 comp rx=0 tx=0).stop 2024-04-02T13:28:52.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.754+0000 7f0c4da8f640 1 -- 172.21.15.44:0/3897865725 reap_dead start 2024-04-02T13:28:52.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.754+0000 7f0c4f4f2640 1 -- 172.21.15.44:0/3897865725 shutdown_connections 2024-04-02T13:28:52.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.754+0000 7f0c4f4f2640 1 -- 172.21.15.44:0/3897865725 >> 172.21.15.44:0/3897865725 conn(0x7f0c4806b920 msgr2=0x7f0c4806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.754+0000 7f0c4f4f2640 1 -- 172.21.15.44:0/3897865725 shutdown_connections 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.754+0000 7f0c4f4f2640 1 -- 172.21.15.44:0/3897865725 wait complete. 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.756+0000 7f0c4f4f2640 1 Processor -- start 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.756+0000 7f0c4f4f2640 1 -- start start 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.756+0000 7f0c4f4f2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c300039b0 0x7f0c4814d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.756+0000 7f0c4f4f2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c48156000 0x7f0c480af550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.756+0000 7f0c4f4f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c48157740 0x7f0c480adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.756+0000 7f0c4f4f2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4815c290 con 0x7f0c48157740 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.756+0000 7f0c4f4f2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0c48071be0 con 0x7f0c48156000 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.756+0000 7f0c4f4f2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0c4814dd20 con 0x7f0c300039b0 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.756+0000 7f0c4da8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c48157740 0x7f0c480adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c4da8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c48157740 0x7f0c480adda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51866/0 (socket says 172.21.15.44:51866) 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c4da8f640 1 -- 172.21.15.44:0/930133732 learned_addr learned my addr 172.21.15.44:0/930133732 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c4ca8d640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c48156000 0x7f0c480af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c300039b0 0x7f0c4814d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c4da8f640 1 -- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c300039b0 msgr2=0x7f0c4814d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c4da8f640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c300039b0 0x7f0c4814d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c4da8f640 1 -- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c48156000 msgr2=0x7f0c480af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c4da8f640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c48156000 0x7f0c480af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c4da8f640 1 -- 172.21.15.44:0/930133732 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c48158a50 con 0x7f0c48157740 2024-04-02T13:28:52.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c4da8f640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c48157740 0x7f0c480adda0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f0c4806f630 tx=0x7f0c3800f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:52.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c45ffb640 1 -- 172.21.15.44:0/930133732 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c38019070 con 0x7f0c48157740 2024-04-02T13:28:52.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c45ffb640 1 -- 172.21.15.44:0/930133732 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c3800f8d0 con 0x7f0c48157740 2024-04-02T13:28:52.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c4f4f2640 1 -- 172.21.15.44:0/930133732 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c481581b0 con 0x7f0c48157740 2024-04-02T13:28:52.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c45ffb640 1 -- 172.21.15.44:0/930133732 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3800fcc0 con 0x7f0c48157740 2024-04-02T13:28:52.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.757+0000 7f0c4f4f2640 1 -- 172.21.15.44:0/930133732 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c48158770 con 0x7f0c48157740 2024-04-02T13:28:52.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.759+0000 7f0c45ffb640 1 -- 172.21.15.44:0/930133732 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0c3801c070 con 0x7f0c48157740 2024-04-02T13:28:52.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.760+0000 7f0c45ffb640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c3406c920 0x7f0c3406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.760+0000 7f0c45ffb640 1 -- 172.21.15.44:0/930133732 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f0c3808c510 con 0x7f0c48157740 2024-04-02T13:28:52.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.760+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c3406c920 0x7f0c3406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.761+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c3406c920 0x7f0c3406ede0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0c40038000 tx=0x7f0c40032000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.761+0000 7f0c521b5740 1 Processor -- start 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.761+0000 7f0c521b5740 1 -- start start 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.761+0000 7f0c521b5740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1db9a0 0x5647ac1ae940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.761+0000 7f0c521b5740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac1de730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.761+0000 7f0c521b5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1aee80 0x5647ac1dec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.761+0000 7f0c521b5740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5647ac118830 con 0x5647ac1dbd80 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.761+0000 7f0c521b5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5647ac0c0820 con 0x5647ac1aee80 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.761+0000 7f0c521b5740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5647ac1147e0 con 0x5647ac1db9a0 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c4ca8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac1de730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c4ca8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac1de730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51882/0 (socket says 172.21.15.44:51882) 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c4ca8d640 1 -- 172.21.15.44:0/2564956313 learned_addr learned my addr 172.21.15.44:0/2564956313 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/2564956313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1db9a0 0x5647ac1ae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c4da8f640 1 --2- 172.21.15.44:0/2564956313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1aee80 0x5647ac1dec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c4ca8d640 1 -- 172.21.15.44:0/2564956313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1db9a0 msgr2=0x5647ac1ae940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c4ca8d640 1 --2- 172.21.15.44:0/2564956313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1db9a0 0x5647ac1ae940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c4ca8d640 1 -- 172.21.15.44:0/2564956313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1aee80 msgr2=0x5647ac1dec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c4ca8d640 1 --2- 172.21.15.44:0/2564956313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1aee80 0x5647ac1dec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c4ca8d640 1 -- 172.21.15.44:0/2564956313 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647abef67d0 con 0x5647ac1dbd80 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c4ca8d640 1 --2- 172.21.15.44:0/2564956313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac1de730 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c001760 tx=0x7f0c3c004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf743d227301ac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c2effd640 1 -- 172.21.15.44:0/2564956313 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c015440 con 0x5647ac1dbd80 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c2effd640 1 -- 172.21.15.44:0/2564956313 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c3c01a070 con 0x5647ac1dbd80 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.762+0000 7f0c2effd640 1 -- 172.21.15.44:0/2564956313 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c0157e0 con 0x5647ac1dbd80 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.763+0000 7f0c521b5740 1 -- 172.21.15.44:0/2564956313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 msgr2=0x5647ac1de730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.763+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2564956313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac1de730 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c001760 tx=0x7f0c3c004bb0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.763+0000 7f0c521b5740 1 -- 172.21.15.44:0/2564956313 shutdown_connections 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.763+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2564956313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1aee80 0x5647ac1dec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.763+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2564956313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac1de730 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.763+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2564956313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1db9a0 0x5647ac1ae940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.763+0000 7f0c521b5740 1 -- 172.21.15.44:0/2564956313 >> 172.21.15.44:0/2564956313 conn(0x5647ac1b0bb0 msgr2=0x5647ac1f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:52.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.763+0000 7f0c521b5740 1 -- 172.21.15.44:0/2564956313 shutdown_connections 2024-04-02T13:28:52.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.763+0000 7f0c521b5740 1 -- 172.21.15.44:0/2564956313 wait complete. 2024-04-02T13:28:52.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.763+0000 7f0c521b5740 1 -- start start 2024-04-02T13:28:52.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.764+0000 7f0c521b5740 1 Processor -- start 2024-04-02T13:28:52.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.764+0000 7f0c521b5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1db9a0 0x5647ac1e22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.764+0000 7f0c521b5740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac1d9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.764+0000 7f0c521b5740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1d98b0 0x5647ac20ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.764+0000 7f0c521b5740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5647ac1b3150 con 0x5647ac1dbd80 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.764+0000 7f0c521b5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5647ac0d3d30 con 0x5647ac1db9a0 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.764+0000 7f0c521b5740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5647ac1b2540 con 0x5647ac1d98b0 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.764+0000 7f0c521b5740 10 client.0 _ll_register_callbacks cb 0x5647ac0d3720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c4ca8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac1d9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c4ca8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac1d9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51884/0 (socket says 172.21.15.44:51884) 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c4ca8d640 1 -- 172.21.15.44:0/3698127880 learned_addr learned my addr 172.21.15.44:0/3698127880 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/3698127880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1db9a0 0x5647ac1e22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c4da8f640 1 --2- 172.21.15.44:0/3698127880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1d98b0 0x5647ac20ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c4ca8d640 1 -- 172.21.15.44:0/3698127880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1d98b0 msgr2=0x5647ac20ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c4ca8d640 1 --2- 172.21.15.44:0/3698127880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1d98b0 0x5647ac20ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c4ca8d640 1 -- 172.21.15.44:0/3698127880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1db9a0 msgr2=0x5647ac1e22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c4ca8d640 1 --2- 172.21.15.44:0/3698127880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1db9a0 0x5647ac1e22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c4ca8d640 1 -- 172.21.15.44:0/3698127880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647abe2c3f0 con 0x5647ac1dbd80 2024-04-02T13:28:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c4ca8d640 1 --2- 172.21.15.44:0/3698127880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac1d9120 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c016000 tx=0x7f0c4815b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff168802c420b1a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:52.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c27fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:52.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c27fff640 1 -- 172.21.15.44:0/3698127880 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c021420 con 0x5647ac1dbd80 2024-04-02T13:28:52.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c27fff640 1 -- 172.21.15.44:0/3698127880 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f0c3c0215c0 con 0x5647ac1dbd80 2024-04-02T13:28:52.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c521b5740 1 -- 172.21.15.44:0/3698127880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c3c01a070 con 0x5647ac1dbd80 2024-04-02T13:28:52.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c27fff640 1 -- 172.21.15.44:0/3698127880 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f0c3c02d050 con 0x5647ac1dbd80 2024-04-02T13:28:52.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c521b5740 10 client.8538 fetch_fsmap learned FSMap version 74 2024-04-02T13:28:52.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.765+0000 7f0c521b5740 1 -- 172.21.15.44:0/3698127880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x5647abdec0f0 con 0x5647ac1dbd80 2024-04-02T13:28:52.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.766+0000 7f0c27fff640 1 -- 172.21.15.44:0/3698127880 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c3c01a070 con 0x5647ac1dbd80 2024-04-02T13:28:52.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.766+0000 7f0c27fff640 1 -- 172.21.15.44:0/3698127880 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f0c3c030020 con 0x5647ac1dbd80 2024-04-02T13:28:52.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.766+0000 7f0c521b5740 10 client.8538 fetch_fsmap finished waiting for FSMap version 74 2024-04-02T13:28:52.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.766+0000 7f0c521b5740 10 client.8538 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:28:52.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.766+0000 7f0c521b5740 20 client.8538 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:52.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.766+0000 7f0c521b5740 1 --2- 172.21.15.44:0/3698127880 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5647ac1e5720 0x5647ac1e5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.766+0000 7f0c521b5740 4 client.8538 mds_command: new command op to 8403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:28:52.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.766+0000 7f0c521b5740 1 -- 172.21.15.44:0/3698127880 --> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647ac125d70 con 0x5647ac1e5720 2024-04-02T13:28:52.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.766+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/3698127880 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5647ac1e5720 0x5647ac1e5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.766+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/3698127880 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5647ac1e5720 0x5647ac1e5b00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cc9a60a2242d4b9 server_cookie=93290b2d0a97a480 in_seq=0 out_seq=0 2024-04-02T13:28:52.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.766+0000 7f0c27fff640 10 client.8538 ms_handle_connect on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:52.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c27fff640 1 -- 172.21.15.44:0/3698127880 <== mds.0 v2:172.21.15.44:6832/3044289683 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5647ac125d70 con 0x5647ac1e5720 2024-04-02T13:28:52.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c27fff640 10 client.8538 handle_command_reply: tid=0 2024-04-02T13:28:52.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c521b5740 1 client.8538 shutdown 2024-04-02T13:28:52.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c521b5740 1 -- 172.21.15.44:0/3698127880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 msgr2=0x5647ac1d9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c521b5740 1 --2- 172.21.15.44:0/3698127880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac1d9120 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c016000 tx=0x7f0c4815b4c0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c521b5740 1 -- 172.21.15.44:0/3698127880 shutdown_connections 2024-04-02T13:28:52.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c521b5740 1 --2- 172.21.15.44:0/3698127880 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5647ac1e5720 0x5647ac1e5b00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c521b5740 1 --2- 172.21.15.44:0/3698127880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1d98b0 0x5647ac20ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c521b5740 1 --2- 172.21.15.44:0/3698127880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac1d9120 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c27fff640 0 client.8538 ms_handle_reset on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:52.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c521b5740 1 --2- 172.21.15.44:0/3698127880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1db9a0 0x5647ac1e22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c521b5740 1 -- 172.21.15.44:0/3698127880 >> 172.21.15.44:0/3698127880 conn(0x5647ac1aee80 msgr2=0x5647ac1de720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:52.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c521b5740 1 -- 172.21.15.44:0/3698127880 shutdown_connections 2024-04-02T13:28:52.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c521b5740 1 -- 172.21.15.44:0/3698127880 wait complete. 2024-04-02T13:28:52.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.767+0000 7f0c521b5740 20 client.8538 trim_cache size 0 max 16384 2024-04-02T13:28:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.785+0000 7f0c521b5740 1 Processor -- start 2024-04-02T13:28:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.785+0000 7f0c521b5740 1 -- start start 2024-04-02T13:28:52.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.785+0000 7f0c521b5740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1d98b0 0x5647ac1fce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.785+0000 7f0c521b5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1db9a0 0x5647ac20c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.785+0000 7f0c521b5740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac20c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c521b5740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c481fd480 con 0x5647ac1dbd80 2024-04-02T13:28:52.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c521b5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0c481fc460 con 0x5647ac1db9a0 2024-04-02T13:28:52.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c521b5740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0c481fb7c0 con 0x5647ac1d98b0 2024-04-02T13:28:52.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c4da8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac20c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c4da8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac20c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51894/0 (socket says 172.21.15.44:51894) 2024-04-02T13:28:52.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c4da8f640 1 -- 172.21.15.44:0/2568712610 learned_addr learned my addr 172.21.15.44:0/2568712610 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c4ca8d640 1 --2- 172.21.15.44:0/2568712610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1db9a0 0x5647ac20c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c4da8f640 1 -- 172.21.15.44:0/2568712610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1d98b0 msgr2=0x5647ac1fce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c4da8f640 1 --2- 172.21.15.44:0/2568712610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1d98b0 0x5647ac1fce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c4da8f640 1 -- 172.21.15.44:0/2568712610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1db9a0 msgr2=0x5647ac20c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c4da8f640 1 --2- 172.21.15.44:0/2568712610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1db9a0 0x5647ac20c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c4da8f640 1 -- 172.21.15.44:0/2568712610 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647ac1e5e70 con 0x5647ac1dbd80 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c4da8f640 1 --2- 172.21.15.44:0/2568712610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac20c560 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f0c3805a060 tx=0x7f0c38008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553927da61ebb2ad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c2e7fc640 1 -- 172.21.15.44:0/2568712610 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c38023040 con 0x5647ac1dbd80 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c2e7fc640 1 -- 172.21.15.44:0/2568712610 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c38085020 con 0x5647ac1dbd80 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.786+0000 7f0c2e7fc640 1 -- 172.21.15.44:0/2568712610 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c38062c00 con 0x5647ac1dbd80 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.787+0000 7f0c521b5740 1 -- 172.21.15.44:0/2568712610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 msgr2=0x5647ac20c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.787+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2568712610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac20c560 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f0c3805a060 tx=0x7f0c38008000 comp rx=0 tx=0).stop 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.787+0000 7f0c521b5740 1 -- 172.21.15.44:0/2568712610 shutdown_connections 2024-04-02T13:28:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.787+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2568712610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1dbd80 0x5647ac20c560 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.787+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2568712610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1db9a0 0x5647ac20c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.787+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2568712610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1d98b0 0x5647ac1fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.787+0000 7f0c521b5740 1 -- 172.21.15.44:0/2568712610 >> 172.21.15.44:0/2568712610 conn(0x5647ac1aee80 msgr2=0x5647ac1b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:52.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.787+0000 7f0c521b5740 1 -- 172.21.15.44:0/2568712610 shutdown_connections 2024-04-02T13:28:52.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.787+0000 7f0c521b5740 1 -- 172.21.15.44:0/2568712610 wait complete. 2024-04-02T13:28:52.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.787+0000 7f0c521b5740 1 -- start start 2024-04-02T13:28:52.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.787+0000 7f0c521b5740 1 Processor -- start 2024-04-02T13:28:52.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.788+0000 7f0c521b5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1d98b0 0x5647ac1adc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.788+0000 7f0c521b5740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1db9a0 0x5647ac2768f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.788+0000 7f0c521b5740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1dbd80 0x5647ac1fce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.788+0000 7f0c521b5740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c481fbe10 con 0x5647ac1db9a0 2024-04-02T13:28:52.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.788+0000 7f0c521b5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5647ac125d70 con 0x5647ac1d98b0 2024-04-02T13:28:52.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.788+0000 7f0c521b5740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5647ac21c2c0 con 0x5647ac1dbd80 2024-04-02T13:28:52.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.788+0000 7f0c521b5740 10 client.0 _ll_register_callbacks cb 0x7f0c3c001650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:52.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.788+0000 7f0c4ca8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1db9a0 0x5647ac2768f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c4ca8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1db9a0 0x5647ac2768f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51896/0 (socket says 172.21.15.44:51896) 2024-04-02T13:28:52.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c4ca8d640 1 -- 172.21.15.44:0/2230545665 learned_addr learned my addr 172.21.15.44:0/2230545665 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:52.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c4da8f640 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1dbd80 0x5647ac1fce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1d98b0 0x5647ac1adc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c4ca8d640 1 -- 172.21.15.44:0/2230545665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1dbd80 msgr2=0x5647ac1fce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c4ca8d640 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1dbd80 0x5647ac1fce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c4ca8d640 1 -- 172.21.15.44:0/2230545665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1d98b0 msgr2=0x5647ac1adc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c4ca8d640 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1d98b0 0x5647ac1adc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c4ca8d640 1 -- 172.21.15.44:0/2230545665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647ac1d8900 con 0x5647ac1db9a0 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1d98b0 0x5647ac1adc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c4da8f640 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1dbd80 0x5647ac1fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c4ca8d640 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1db9a0 0x5647ac2768f0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c016000 tx=0x7f0c3c0078f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae95a8a30cb75fb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c27fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c27fff640 1 -- 172.21.15.44:0/2230545665 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c0159a0 con 0x5647ac1db9a0 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c27fff640 1 -- 172.21.15.44:0/2230545665 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f0c3c015d50 con 0x5647ac1db9a0 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c521b5740 1 -- 172.21.15.44:0/2230545665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c38085020 con 0x5647ac1db9a0 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c27fff640 1 -- 172.21.15.44:0/2230545665 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f0c3c007490 con 0x5647ac1db9a0 2024-04-02T13:28:52.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c521b5740 10 client.8544 fetch_fsmap learned FSMap version 74 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c521b5740 1 -- 172.21.15.44:0/2230545665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x5647ac1d2ae0 con 0x5647ac1db9a0 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.789+0000 7f0c27fff640 1 -- 172.21.15.44:0/2230545665 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c38085020 con 0x5647ac1db9a0 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c27fff640 1 -- 172.21.15.44:0/2230545665 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f0c3c030040 con 0x5647ac1db9a0 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 10 client.8544 fetch_fsmap finished waiting for FSMap version 74 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 10 client.8544 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 20 client.8544 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5647ac1fd370 0x5647ac1b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 4 client.8544 mds_command: new command op to 8403 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 1 -- 172.21.15.44:0/2230545665 --> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5647ac1200c0 con 0x5647ac1fd370 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5647ac1fd370 0x5647ac1b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c27fff640 10 client.8544 ms_handle_connect on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c4d28e640 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5647ac1fd370 0x5647ac1b3640 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=729bb698cf74d108 server_cookie=302c85e735649e7c in_seq=0 out_seq=0 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c27fff640 1 -- 172.21.15.44:0/2230545665 <== mds.0 v2:172.21.15.44:6832/3044289683 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5647ac1200c0 con 0x5647ac1fd370 2024-04-02T13:28:52.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c27fff640 10 client.8544 handle_command_reply: tid=0 2024-04-02T13:28:52.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 1 client.8544 shutdown 2024-04-02T13:28:52.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 1 -- 172.21.15.44:0/2230545665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1db9a0 msgr2=0x5647ac2768f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1db9a0 0x5647ac2768f0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c016000 tx=0x7f0c3c0078f0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 1 -- 172.21.15.44:0/2230545665 shutdown_connections 2024-04-02T13:28:52.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5647ac1fd370 0x5647ac1b3640 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5647ac1dbd80 0x5647ac1fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647ac1db9a0 0x5647ac2768f0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c27fff640 0 client.8544 ms_handle_reset on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:52.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 1 --2- 172.21.15.44:0/2230545665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5647ac1d98b0 0x5647ac1adc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.790+0000 7f0c521b5740 1 -- 172.21.15.44:0/2230545665 >> 172.21.15.44:0/2230545665 conn(0x5647ac1aee80 msgr2=0x5647ac1b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:52.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c521b5740 1 -- 172.21.15.44:0/2230545665 shutdown_connections 2024-04-02T13:28:52.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c521b5740 1 -- 172.21.15.44:0/2230545665 wait complete. 2024-04-02T13:28:52.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c521b5740 20 client.8544 trim_cache size 0 max 16384 2024-04-02T13:28:52.800 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:28:52.800 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-02T13:28:52.800 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-02T13:28:52.800 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:28:52.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c4f4f2640 1 -- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c3406c920 msgr2=0x7f0c3406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c4f4f2640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c3406c920 0x7f0c3406ede0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0c40038000 tx=0x7f0c40032000 comp rx=0 tx=0).stop 2024-04-02T13:28:52.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c4f4f2640 1 -- 172.21.15.44:0/930133732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c48157740 msgr2=0x7f0c480adda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c4f4f2640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c48157740 0x7f0c480adda0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f0c4806f630 tx=0x7f0c3800f1b0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c4f4f2640 1 -- 172.21.15.44:0/930133732 shutdown_connections 2024-04-02T13:28:52.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c4f4f2640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c3406c920 0x7f0c3406ede0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c4f4f2640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c48157740 0x7f0c480adda0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c4f4f2640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c48156000 0x7f0c480af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c4f4f2640 1 --2- 172.21.15.44:0/930133732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c300039b0 0x7f0c4814d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c4f4f2640 1 -- 172.21.15.44:0/930133732 >> 172.21.15.44:0/930133732 conn(0x7f0c4806b920 msgr2=0x7f0c4806e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:52.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c4f4f2640 1 -- 172.21.15.44:0/930133732 shutdown_connections 2024-04-02T13:28:52.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.791+0000 7f0c4f4f2640 1 -- 172.21.15.44:0/930133732 wait complete. 2024-04-02T13:28:52.805 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T13:28:52.805 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:28:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.904+0000 7f4b11cd3640 1 Processor -- start 2024-04-02T13:28:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.904+0000 7f4b11cd3640 1 -- start start 2024-04-02T13:28:52.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b11cd3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4b0c15e550 0x7f4b0c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b11cd3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b0c15f040 0x7f4b0c0d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b11cd3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b0c15fb50 con 0x7f4b0c06d530 2024-04-02T13:28:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b11cd3640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4b0c06d970 con 0x7f4b0c15e550 2024-04-02T13:28:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b11cd3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4b0c06ca90 con 0x7f4b0c15f040 2024-04-02T13:28:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b0affd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4b0c06d530 0x7f4b0c15e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:38820/0 (socket says 172.21.15.44:38820) 2024-04-02T13:28:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b0b7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4b0c15e550 0x7f4b0c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b0affd640 1 -- 172.21.15.44:0/3271219176 learned_addr learned my addr 172.21.15.44:0/3271219176 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b0bfff640 1 --2- 172.21.15.44:0/3271219176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b0c15f040 0x7f4b0c0d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b0a7fc640 1 -- 172.21.15.44:0/3271219176 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2768354460 0 0) 0x7f4b0c15fb50 con 0x7f4b0c06d530 2024-04-02T13:28:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b0a7fc640 1 -- 172.21.15.44:0/3271219176 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4af4003520 con 0x7f4b0c06d530 2024-04-02T13:28:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b0b7fe640 1 -- 172.21.15.44:0/3271219176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b0c15f040 msgr2=0x7f4b0c0d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b0b7fe640 1 --2- 172.21.15.44:0/3271219176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b0c15f040 0x7f4b0c0d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b0b7fe640 1 -- 172.21.15.44:0/3271219176 >> v1:172.21.15.44:6789/0 conn(0x7f4b0c06d530 legacy=0x7f4b0c15e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.905+0000 7f4b0b7fe640 1 -- 172.21.15.44:0/3271219176 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0c15f590 con 0x7f4b0c15e550 2024-04-02T13:28:52.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.906+0000 7f4b0b7fe640 1 --2- 172.21.15.44:0/3271219176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c15e550 0x7f4b0c06cff0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f4b000097a0 tx=0x7f4b0002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7d6c6e2abea4ca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:52.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.906+0000 7f4b0a7fc640 1 -- 172.21.15.44:0/3271219176 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b00034070 con 0x7f4b0c15e550 2024-04-02T13:28:52.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.906+0000 7f4b0a7fc640 1 -- 172.21.15.44:0/3271219176 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b00030610 con 0x7f4b0c15e550 2024-04-02T13:28:52.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.906+0000 7f4b0a7fc640 1 -- 172.21.15.44:0/3271219176 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b000309e0 con 0x7f4b0c15e550 2024-04-02T13:28:52.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.906+0000 7f4b11cd3640 1 -- 172.21.15.44:0/3271219176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c15e550 msgr2=0x7f4b0c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.906+0000 7f4b11cd3640 1 --2- 172.21.15.44:0/3271219176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c15e550 0x7f4b0c06cff0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f4b000097a0 tx=0x7f4b0002b960 comp rx=0 tx=0).stop 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.907+0000 7f4b11cd3640 1 -- 172.21.15.44:0/3271219176 shutdown_connections 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.907+0000 7f4b11cd3640 1 --2- 172.21.15.44:0/3271219176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b0c15f040 0x7f4b0c0d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.907+0000 7f4b11cd3640 1 --2- 172.21.15.44:0/3271219176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c15e550 0x7f4b0c06cff0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.907+0000 7f4b11cd3640 1 -- 172.21.15.44:0/3271219176 >> 172.21.15.44:0/3271219176 conn(0x7f4b0c09aa20 msgr2=0x7f4b0c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.907+0000 7f4b11cd3640 1 -- 172.21.15.44:0/3271219176 shutdown_connections 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.907+0000 7f4b11cd3640 1 -- 172.21.15.44:0/3271219176 wait complete. 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.908+0000 7f4b11cd3640 1 Processor -- start 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.908+0000 7f4b11cd3640 1 -- start start 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b11cd3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b0c06d530 0x7f4b0c15c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b11cd3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b0c15e550 0x7f4b0c09d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b11cd3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c15f040 0x7f4b0c0b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b11cd3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c0d0590 con 0x7f4b0c15f040 2024-04-02T13:28:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b11cd3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c0d02b0 con 0x7f4b0c15e550 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b11cd3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4b0c0cc4b0 con 0x7f4b0c06d530 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b0bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c15f040 0x7f4b0c0b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b0bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c15f040 0x7f4b0c0b0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51916/0 (socket says 172.21.15.44:51916) 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b0bfff640 1 -- 172.21.15.44:0/4257855140 learned_addr learned my addr 172.21.15.44:0/4257855140 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b0affd640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b0c15e550 0x7f4b0c09d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b0b7fe640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b0c06d530 0x7f4b0c15c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b0bfff640 1 -- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b0c06d530 msgr2=0x7f4b0c15c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b0bfff640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b0c06d530 0x7f4b0c15c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b0bfff640 1 -- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b0c15e550 msgr2=0x7f4b0c09d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b0bfff640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b0c15e550 0x7f4b0c09d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b0bfff640 1 -- 172.21.15.44:0/4257855140 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0c0b5cf0 con 0x7f4b0c15f040 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b0affd640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b0c15e550 0x7f4b0c09d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b0bfff640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c15f040 0x7f4b0c0b0030 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400efc0 tx=0x7f4b0400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4af3fff640 1 -- 172.21.15.44:0/4257855140 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b04018020 con 0x7f4b0c15f040 2024-04-02T13:28:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4af3fff640 1 -- 172.21.15.44:0/4257855140 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b04014480 con 0x7f4b0c15f040 2024-04-02T13:28:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b11cd3640 1 -- 172.21.15.44:0/4257855140 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0c0b5fd0 con 0x7f4b0c15f040 2024-04-02T13:28:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4af3fff640 1 -- 172.21.15.44:0/4257855140 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b04014870 con 0x7f4b0c15f040 2024-04-02T13:28:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.909+0000 7f4b11cd3640 1 -- 172.21.15.44:0/4257855140 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b0c0b6590 con 0x7f4b0c15f040 2024-04-02T13:28:52.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.911+0000 7f4b11cd3640 1 -- 172.21.15.44:0/4257855140 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b0c0b6790 con 0x7f4b0c15f040 2024-04-02T13:28:52.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.911+0000 7f4af3fff640 1 -- 172.21.15.44:0/4257855140 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4b0401b020 con 0x7f4b0c15f040 2024-04-02T13:28:52.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.911+0000 7f4af3fff640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4af806c8f0 0x7f4af806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:52.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.911+0000 7f4af3fff640 1 -- 172.21.15.44:0/4257855140 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f4b0408cc00 con 0x7f4b0c15f040 2024-04-02T13:28:52.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.912+0000 7f4b0b7fe640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4af806c8f0 0x7f4af806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:52.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.912+0000 7f4b0b7fe640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4af806c8f0 0x7f4af806edb0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4b00036000 tx=0x7f4b00008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:52.916+0000 7f4af3fff640 1 -- 172.21.15.44:0/4257855140 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4b0c0b6790 con 0x7f4b0c15f040 2024-04-02T13:28:53.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.141+0000 7f4b11cd3640 1 -- 172.21.15.44:0/4257855140 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b0c0684f0 con 0x7f4b0c15f040 2024-04-02T13:28:53.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.141+0000 7f4af3fff640 1 -- 172.21.15.44:0/4257855140 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5181 (secure 0 0 0) 0x7f4b04093080 con 0x7f4b0c15f040 2024-04-02T13:28:53.144 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:53.144 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:28:19.321536+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8403":{"gid":8403,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6833/3044289683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":3044289683},{"type":"v1","addr":"172.21.15.44:6833","nonce":3044289683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8403,"qdb_cluster":[8403]},"id":6}]} 2024-04-02T13:28:53.144 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 74 2024-04-02T13:28:53.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.143+0000 7f4b11cd3640 1 -- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4af806c8f0 msgr2=0x7f4af806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.143+0000 7f4b11cd3640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4af806c8f0 0x7f4af806edb0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4b00036000 tx=0x7f4b00008030 comp rx=0 tx=0).stop 2024-04-02T13:28:53.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.143+0000 7f4b11cd3640 1 -- 172.21.15.44:0/4257855140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c15f040 msgr2=0x7f4b0c0b0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.143+0000 7f4b11cd3640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c15f040 0x7f4b0c0b0030 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400efc0 tx=0x7f4b0400c490 comp rx=0 tx=0).stop 2024-04-02T13:28:53.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.143+0000 7f4b11cd3640 1 -- 172.21.15.44:0/4257855140 shutdown_connections 2024-04-02T13:28:53.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.143+0000 7f4b11cd3640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4af806c8f0 0x7f4af806edb0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.143+0000 7f4b11cd3640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c15f040 0x7f4b0c0b0030 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.143+0000 7f4b11cd3640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b0c15e550 0x7f4b0c09d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.143+0000 7f4b11cd3640 1 --2- 172.21.15.44:0/4257855140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b0c06d530 0x7f4b0c15c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.143+0000 7f4b11cd3640 1 -- 172.21.15.44:0/4257855140 >> 172.21.15.44:0/4257855140 conn(0x7f4b0c09aa20 msgr2=0x7f4b0c0cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:53.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.143+0000 7f4b11cd3640 1 -- 172.21.15.44:0/4257855140 shutdown_connections 2024-04-02T13:28:53.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.144+0000 7f4b11cd3640 1 -- 172.21.15.44:0/4257855140 wait complete. 2024-04-02T13:28:53.157 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-04-02T13:28:53.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.257+0000 7f6802360640 1 Processor -- start 2024-04-02T13:28:53.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.257+0000 7f6802360640 1 -- start start 2024-04-02T13:28:53.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.258+0000 7f6802360640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67fc15e4a0 0x7f67fc0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.258+0000 7f6802360640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67fc06d550 0x7f67fc0d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.258+0000 7f6802360640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67fc0cc600 con 0x7f67fc0d0c50 2024-04-02T13:28:53.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.258+0000 7f6802360640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f67fc0cc8e0 con 0x7f67fc06d550 2024-04-02T13:28:53.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.258+0000 7f6802360640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f67fc0ccbc0 con 0x7f67fc15e4a0 2024-04-02T13:28:53.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.258+0000 7f67fb7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f67fc0d0c50 0x7f67fc06ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:38822/0 (socket says 172.21.15.44:38822) 2024-04-02T13:28:53.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.258+0000 7f67fb7fe640 1 -- 172.21.15.44:0/795611413 learned_addr learned my addr 172.21.15.44:0/795611413 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:53.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.259+0000 7f68008fd640 1 --2- 172.21.15.44:0/795611413 >> v2:172.21.15.69:3300/0 conn(0x7f67fc06d550 0x7f67fc0d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.259+0000 7f67fbfff640 1 --2- 172.21.15.44:0/795611413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67fc15e4a0 0x7f67fc0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.259+0000 7f67faffd640 1 -- 172.21.15.44:0/795611413 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3622267474 0 0) 0x7f67fc0cc600 con 0x7f67fc0d0c50 2024-04-02T13:28:53.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.259+0000 7f67faffd640 1 -- 172.21.15.44:0/795611413 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67e4003520 con 0x7f67fc0d0c50 2024-04-02T13:28:53.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.259+0000 7f67faffd640 1 -- 172.21.15.44:0/795611413 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4123310842 0 0) 0x7f67e4003520 con 0x7f67fc0d0c50 2024-04-02T13:28:53.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.259+0000 7f68008fd640 1 -- 172.21.15.44:0/795611413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67fc15e4a0 msgr2=0x7f67fc0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:53.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.259+0000 7f68008fd640 1 --2- 172.21.15.44:0/795611413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67fc15e4a0 0x7f67fc0aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.259+0000 7f68008fd640 1 -- 172.21.15.44:0/795611413 >> v1:172.21.15.44:6789/0 conn(0x7f67fc0d0c50 legacy=0x7f67fc06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.259+0000 7f68008fd640 1 -- 172.21.15.44:0/795611413 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67fc0ccea0 con 0x7f67fc06d550 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.260+0000 7f68008fd640 1 --2- 172.21.15.44:0/795611413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67fc06d550 0x7f67fc0d10b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f67f4009d90 tx=0x7f67f402bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38e00e2d98061f4a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.260+0000 7f67faffd640 1 -- 172.21.15.44:0/795611413 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f4039070 con 0x7f67fc06d550 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.260+0000 7f67faffd640 1 -- 172.21.15.44:0/795611413 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67f402c8a0 con 0x7f67fc06d550 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.260+0000 7f67faffd640 1 -- 172.21.15.44:0/795611413 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0001670 con 0x7f67fc06d550 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.260+0000 7f6802360640 1 -- 172.21.15.44:0/795611413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67fc06d550 msgr2=0x7f67fc0d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.260+0000 7f6802360640 1 --2- 172.21.15.44:0/795611413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67fc06d550 0x7f67fc0d10b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f67f4009d90 tx=0x7f67f402bd70 comp rx=0 tx=0).stop 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.261+0000 7f6802360640 1 -- 172.21.15.44:0/795611413 shutdown_connections 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.261+0000 7f6802360640 1 --2- 172.21.15.44:0/795611413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67fc06d550 0x7f67fc0d10b0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.261+0000 7f6802360640 1 --2- 172.21.15.44:0/795611413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67fc15e4a0 0x7f67fc0aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.261+0000 7f6802360640 1 -- 172.21.15.44:0/795611413 >> 172.21.15.44:0/795611413 conn(0x7f67fc09aa20 msgr2=0x7f67fc09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.261+0000 7f6802360640 1 -- 172.21.15.44:0/795611413 shutdown_connections 2024-04-02T13:28:53.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.261+0000 7f6802360640 1 -- 172.21.15.44:0/795611413 wait complete. 2024-04-02T13:28:53.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.262+0000 7f6802360640 1 Processor -- start 2024-04-02T13:28:53.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.262+0000 7f6802360640 1 -- start start 2024-04-02T13:28:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f6802360640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67fc06d550 0x7f67fc15c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f6802360640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67fc0d0c50 0x7f67fc15b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f6802360640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67fc15e4a0 0x7f67fc15b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f6802360640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc15e880 con 0x7f67fc0d0c50 2024-04-02T13:28:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f6802360640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc0cd460 con 0x7f67fc06d550 2024-04-02T13:28:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f6802360640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f67fc0af130 con 0x7f67fc15e4a0 2024-04-02T13:28:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67fb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67fc0d0c50 0x7f67fc15b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67fb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67fc0d0c50 0x7f67fc15b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51924/0 (socket says 172.21.15.44:51924) 2024-04-02T13:28:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67fb7fe640 1 -- 172.21.15.44:0/775239447 learned_addr learned my addr 172.21.15.44:0/775239447 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67fbfff640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67fc06d550 0x7f67fc15c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67fb7fe640 1 -- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67fc15e4a0 msgr2=0x7f67fc15b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67fb7fe640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67fc15e4a0 0x7f67fc15b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67fb7fe640 1 -- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67fc06d550 msgr2=0x7f67fc15c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67fb7fe640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67fc06d550 0x7f67fc15c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67fb7fe640 1 -- 172.21.15.44:0/775239447 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67fc0b5680 con 0x7f67fc0d0c50 2024-04-02T13:28:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67fbfff640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67fc06d550 0x7f67fc15c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67fb7fe640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67fc0d0c50 0x7f67fc15b110 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f67f000c780 tx=0x7f67f000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67f8ff9640 1 -- 172.21.15.44:0/775239447 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0005ad0 con 0x7f67fc0d0c50 2024-04-02T13:28:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67f8ff9640 1 -- 172.21.15.44:0/775239447 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67f0005c90 con 0x7f67fc0d0c50 2024-04-02T13:28:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f67f8ff9640 1 -- 172.21.15.44:0/775239447 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f00072b0 con 0x7f67fc0d0c50 2024-04-02T13:28:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f6802360640 1 -- 172.21.15.44:0/775239447 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67fc0b79e0 con 0x7f67fc0d0c50 2024-04-02T13:28:53.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.263+0000 7f6802360640 1 -- 172.21.15.44:0/775239447 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67fc0b7700 con 0x7f67fc0d0c50 2024-04-02T13:28:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.265+0000 7f67f8ff9640 1 -- 172.21.15.44:0/775239447 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f67f0007450 con 0x7f67fc0d0c50 2024-04-02T13:28:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.265+0000 7f67f8ff9640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f67e806c8f0 0x7f67e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.265+0000 7f67f8ff9640 1 -- 172.21.15.44:0/775239447 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f67f008bc80 con 0x7f67fc0d0c50 2024-04-02T13:28:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.266+0000 7f67fbfff640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f67e806c8f0 0x7f67e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.266+0000 7f67fbfff640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f67e806c8f0 0x7f67e806edb0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f67fc070c80 tx=0x7f67ec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:53.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.266+0000 7f6805028740 1 Processor -- start 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.266+0000 7f6805028740 1 -- start start 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f6805028740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc3dc0 0x5605adcc41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f6805028740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 0x5605adcc7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f6805028740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 0x5605add11950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f6805028740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5605adc8b5d0 con 0x5605adcc9be0 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f6805028740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5605adc2d6b0 con 0x5605adcc3dc0 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f6805028740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5605adbcb890 con 0x5605add114d0 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f67fb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 0x5605adcc7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f67fb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 0x5605adcc7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51934/0 (socket says 172.21.15.44:51934) 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f67fb7fe640 1 -- 172.21.15.44:0/2257919443 learned_addr learned my addr 172.21.15.44:0/2257919443 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f67fbfff640 1 --2- 172.21.15.44:0/2257919443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc3dc0 0x5605adcc41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f68008fd640 1 --2- 172.21.15.44:0/2257919443 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 0x5605add11950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f67fb7fe640 1 -- 172.21.15.44:0/2257919443 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 msgr2=0x5605add11950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:53.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f67fb7fe640 1 --2- 172.21.15.44:0/2257919443 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 0x5605add11950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f67fb7fe640 1 -- 172.21.15.44:0/2257919443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc3dc0 msgr2=0x5605adcc41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f67fb7fe640 1 --2- 172.21.15.44:0/2257919443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc3dc0 0x5605adcc41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.267+0000 7f67fb7fe640 1 -- 172.21.15.44:0/2257919443 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605adbfa1d0 con 0x5605adcc9be0 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.268+0000 7f67fb7fe640 1 --2- 172.21.15.44:0/2257919443 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 0x5605adcc7e10 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f67f0089f30 tx=0x7f67f0059690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529837c3448a8455 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.268+0000 7f67e1ffb640 1 -- 172.21.15.44:0/2257919443 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0088070 con 0x5605adcc9be0 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.268+0000 7f67e1ffb640 1 -- 172.21.15.44:0/2257919443 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67f0059790 con 0x5605adcc9be0 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.268+0000 7f67e1ffb640 1 -- 172.21.15.44:0/2257919443 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f005db40 con 0x5605adcc9be0 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.268+0000 7f6805028740 1 -- 172.21.15.44:0/2257919443 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 msgr2=0x5605adcc7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.268+0000 7f6805028740 1 --2- 172.21.15.44:0/2257919443 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 0x5605adcc7e10 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f67f0089f30 tx=0x7f67f0059690 comp rx=0 tx=0).stop 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.268+0000 7f6805028740 1 -- 172.21.15.44:0/2257919443 shutdown_connections 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.268+0000 7f6805028740 1 --2- 172.21.15.44:0/2257919443 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 0x5605add11950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.268+0000 7f6805028740 1 --2- 172.21.15.44:0/2257919443 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 0x5605adcc7e10 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.268+0000 7f6805028740 1 --2- 172.21.15.44:0/2257919443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc3dc0 0x5605adcc41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.268+0000 7f6805028740 1 -- 172.21.15.44:0/2257919443 >> 172.21.15.44:0/2257919443 conn(0x5605adcc5e40 msgr2=0x5605adcc7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:53.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.268+0000 7f6805028740 1 -- 172.21.15.44:0/2257919443 shutdown_connections 2024-04-02T13:28:53.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.268+0000 7f6805028740 1 -- 172.21.15.44:0/2257919443 wait complete. 2024-04-02T13:28:53.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.269+0000 7f6805028740 1 -- start start 2024-04-02T13:28:53.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.269+0000 7f6805028740 1 Processor -- start 2024-04-02T13:28:53.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f6805028740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc9be0 0x5605adcf3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f6805028740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 0x5605add097e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f6805028740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adce93f0 0x5605add09d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f6805028740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5605adbc9200 con 0x5605adce93f0 2024-04-02T13:28:53.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f6805028740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5605adc29660 con 0x5605adcc9be0 2024-04-02T13:28:53.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f6805028740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5605adcf0530 con 0x5605add114d0 2024-04-02T13:28:53.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f6805028740 10 client.0 _ll_register_callbacks cb 0x5605adbec040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:53.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f68008fd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adce93f0 0x5605add09d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f68008fd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adce93f0 0x5605add09d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51948/0 (socket says 172.21.15.44:51948) 2024-04-02T13:28:53.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f68008fd640 1 -- 172.21.15.44:0/4094792665 learned_addr learned my addr 172.21.15.44:0/4094792665 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f67fbfff640 1 --2- 172.21.15.44:0/4094792665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc9be0 0x5605adcf3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f67fb7fe640 1 --2- 172.21.15.44:0/4094792665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 0x5605add097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f68008fd640 1 -- 172.21.15.44:0/4094792665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 msgr2=0x5605add097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f68008fd640 1 --2- 172.21.15.44:0/4094792665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 0x5605add097e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f68008fd640 1 -- 172.21.15.44:0/4094792665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc9be0 msgr2=0x5605adcf3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f68008fd640 1 --2- 172.21.15.44:0/4094792665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc9be0 0x5605adcf3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f68008fd640 1 -- 172.21.15.44:0/4094792665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605ada0d7c0 con 0x5605adce93f0 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f68008fd640 1 --2- 172.21.15.44:0/4094792665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adce93f0 0x5605add09d20 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f67f4009eb0 tx=0x7f67f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2d759b6854fd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.270+0000 7f67daffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f67daffd640 1 -- 172.21.15.44:0/4094792665 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f4045070 con 0x5605adce93f0 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f67daffd640 1 -- 172.21.15.44:0/4094792665 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f67f4034850 con 0x5605adce93f0 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f6805028740 1 -- 172.21.15.44:0/4094792665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67f0059790 con 0x5605adce93f0 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f67daffd640 1 -- 172.21.15.44:0/4094792665 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f67f4050050 con 0x5605adce93f0 2024-04-02T13:28:53.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f6805028740 10 client.8565 fetch_fsmap learned FSMap version 74 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f6805028740 1 -- 172.21.15.44:0/4094792665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x5605ada6df50 con 0x5605adce93f0 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f67daffd640 1 -- 172.21.15.44:0/4094792665 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67f0059790 con 0x5605adce93f0 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f67daffd640 1 -- 172.21.15.44:0/4094792665 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f67f4053020 con 0x5605adce93f0 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f6805028740 10 client.8565 fetch_fsmap finished waiting for FSMap version 74 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f6805028740 10 client.8565 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f6805028740 20 client.8565 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f6805028740 1 --2- 172.21.15.44:0/4094792665 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5605adcf7290 0x5605adcf7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f6805028740 4 client.8565 mds_command: new command op to 8403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f6805028740 1 -- 172.21.15.44:0/4094792665 --> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605adc40ee0 con 0x5605adcf7290 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.271+0000 7f67fbfff640 1 --2- 172.21.15.44:0/4094792665 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5605adcf7290 0x5605adcf7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f67fbfff640 1 --2- 172.21.15.44:0/4094792665 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5605adcf7290 0x5605adcf7670 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3494723929923fa server_cookie=dcf3e6c42d9ab56f in_seq=0 out_seq=0 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f67daffd640 10 client.8565 ms_handle_connect on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f67daffd640 1 -- 172.21.15.44:0/4094792665 <== mds.0 v2:172.21.15.44:6832/3044289683 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5605adc40ee0 con 0x5605adcf7290 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f67daffd640 10 client.8565 handle_command_reply: tid=0 2024-04-02T13:28:53.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f6805028740 1 client.8565 shutdown 2024-04-02T13:28:53.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f6805028740 1 -- 172.21.15.44:0/4094792665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adce93f0 msgr2=0x5605add09d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f6805028740 1 --2- 172.21.15.44:0/4094792665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adce93f0 0x5605add09d20 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f67f4009eb0 tx=0x7f67f4004a40 comp rx=0 tx=0).stop 2024-04-02T13:28:53.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f6805028740 1 -- 172.21.15.44:0/4094792665 shutdown_connections 2024-04-02T13:28:53.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f6805028740 1 --2- 172.21.15.44:0/4094792665 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5605adcf7290 0x5605adcf7670 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f6805028740 1 --2- 172.21.15.44:0/4094792665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adce93f0 0x5605add09d20 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f6805028740 1 --2- 172.21.15.44:0/4094792665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 0x5605add097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f67daffd640 0 client.8565 ms_handle_reset on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:53.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f6805028740 1 --2- 172.21.15.44:0/4094792665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc9be0 0x5605adcf3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.272+0000 7f6805028740 1 -- 172.21.15.44:0/4094792665 >> 172.21.15.44:0/4094792665 conn(0x5605adcc3dc0 msgr2=0x5605adcc32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:53.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.273+0000 7f6805028740 1 -- 172.21.15.44:0/4094792665 shutdown_connections 2024-04-02T13:28:53.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.273+0000 7f6805028740 1 -- 172.21.15.44:0/4094792665 wait complete. 2024-04-02T13:28:53.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.273+0000 7f6805028740 20 client.8565 trim_cache size 0 max 16384 2024-04-02T13:28:53.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.286+0000 7f6805028740 1 Processor -- start 2024-04-02T13:28:53.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.286+0000 7f6805028740 1 -- start start 2024-04-02T13:28:53.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f6805028740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc9be0 0x5605adcf0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f6805028740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605adce93f0 0x5605adcf0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f6805028740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605add114d0 0x5605adcf33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f6805028740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc201000 con 0x5605add114d0 2024-04-02T13:28:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f6805028740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc200030 con 0x5605adcc9be0 2024-04-02T13:28:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f6805028740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f67fc15f720 con 0x5605adce93f0 2024-04-02T13:28:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f68008fd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605add114d0 0x5605adcf33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f68008fd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605add114d0 0x5605adcf33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51952/0 (socket says 172.21.15.44:51952) 2024-04-02T13:28:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f68008fd640 1 -- 172.21.15.44:0/1342400459 learned_addr learned my addr 172.21.15.44:0/1342400459 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f67fbfff640 1 --2- 172.21.15.44:0/1342400459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc9be0 0x5605adcf0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f67fb7fe640 1 --2- 172.21.15.44:0/1342400459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605adce93f0 0x5605adcf0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f68008fd640 1 -- 172.21.15.44:0/1342400459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605adce93f0 msgr2=0x5605adcf0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f68008fd640 1 --2- 172.21.15.44:0/1342400459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605adce93f0 0x5605adcf0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f68008fd640 1 -- 172.21.15.44:0/1342400459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc9be0 msgr2=0x5605adcf0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f68008fd640 1 --2- 172.21.15.44:0/1342400459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc9be0 0x5605adcf0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f68008fd640 1 -- 172.21.15.44:0/1342400459 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605ad9010b0 con 0x5605add114d0 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f67fbfff640 1 --2- 172.21.15.44:0/1342400459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc9be0 0x5605adcf0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f68008fd640 1 --2- 172.21.15.44:0/1342400459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605add114d0 0x5605adcf33a0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f67f4002230 tx=0x7f67f4042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7019998d00fb44fd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f67e17fa640 1 -- 172.21.15.44:0/1342400459 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f4059070 con 0x5605add114d0 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.287+0000 7f67e17fa640 1 -- 172.21.15.44:0/1342400459 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67f403a040 con 0x5605add114d0 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.288+0000 7f67e17fa640 1 -- 172.21.15.44:0/1342400459 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f4045040 con 0x5605add114d0 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.288+0000 7f6805028740 1 -- 172.21.15.44:0/1342400459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605add114d0 msgr2=0x5605adcf33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.288+0000 7f6805028740 1 --2- 172.21.15.44:0/1342400459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605add114d0 0x5605adcf33a0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f67f4002230 tx=0x7f67f4042000 comp rx=0 tx=0).stop 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.288+0000 7f6805028740 1 -- 172.21.15.44:0/1342400459 shutdown_connections 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.288+0000 7f6805028740 1 --2- 172.21.15.44:0/1342400459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605add114d0 0x5605adcf33a0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.288+0000 7f6805028740 1 --2- 172.21.15.44:0/1342400459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605adce93f0 0x5605adcf0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.288+0000 7f6805028740 1 --2- 172.21.15.44:0/1342400459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adcc9be0 0x5605adcf0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.288+0000 7f6805028740 1 -- 172.21.15.44:0/1342400459 >> 172.21.15.44:0/1342400459 conn(0x5605adcc3dc0 msgr2=0x5605adcef8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:53.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.288+0000 7f6805028740 1 -- 172.21.15.44:0/1342400459 shutdown_connections 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.288+0000 7f6805028740 1 -- 172.21.15.44:0/1342400459 wait complete. 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.289+0000 7f6805028740 1 -- start start 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.289+0000 7f6805028740 1 Processor -- start 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f6805028740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 0x5605adcca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f6805028740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adce93f0 0x5605adccace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f6805028740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 0x5605add32130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f6805028740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5605adcf7cb0 con 0x5605adcc9be0 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f6805028740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc1ffa30 con 0x5605adce93f0 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f6805028740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5605adcf6430 con 0x5605add114d0 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f6805028740 10 client.0 _ll_register_callbacks cb 0x7f67e40036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67fbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 0x5605adcca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67fbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 0x5605adcca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51956/0 (socket says 172.21.15.44:51956) 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67fbfff640 1 -- 172.21.15.44:0/2804003299 learned_addr learned my addr 172.21.15.44:0/2804003299 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67fb7fe640 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adce93f0 0x5605adccace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f68008fd640 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 0x5605add32130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67fbfff640 1 -- 172.21.15.44:0/2804003299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 msgr2=0x5605add32130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67fbfff640 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 0x5605add32130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67fbfff640 1 -- 172.21.15.44:0/2804003299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adce93f0 msgr2=0x5605adccace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67fbfff640 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adce93f0 0x5605adccace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67fbfff640 1 -- 172.21.15.44:0/2804003299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605add71b50 con 0x5605adcc9be0 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f68008fd640 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 0x5605add32130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67fbfff640 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 0x5605adcca7a0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x5605adcf6880 tx=0x7f67ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ded416ccf11bba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67daffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67daffd640 1 -- 172.21.15.44:0/2804003299 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec00f040 con 0x5605adcc9be0 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67daffd640 1 -- 172.21.15.44:0/2804003299 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f67ec02c9a0 con 0x5605adcc9be0 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f6805028740 1 -- 172.21.15.44:0/2804003299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67f403a040 con 0x5605adcc9be0 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f67daffd640 1 -- 172.21.15.44:0/2804003299 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f67ec02cba0 con 0x5605adcc9be0 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.290+0000 7f6805028740 10 client.8571 fetch_fsmap learned FSMap version 74 2024-04-02T13:28:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f6805028740 1 -- 172.21.15.44:0/2804003299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x5605adcc2f10 con 0x5605adcc9be0 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f67daffd640 1 -- 172.21.15.44:0/2804003299 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67f403a040 con 0x5605adcc9be0 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f67daffd640 1 -- 172.21.15.44:0/2804003299 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f67ec033050 con 0x5605adcc9be0 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f6805028740 10 client.8571 fetch_fsmap finished waiting for FSMap version 74 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f6805028740 10 client.8571 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f6805028740 20 client.8571 populate_metadata read hostname 'smithi044' 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f6805028740 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5605add32920 0x5605adce14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f6805028740 4 client.8571 mds_command: new command op to 8403 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f6805028740 1 -- 172.21.15.44:0/2804003299 --> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x5605add0f2d0 con 0x5605add32920 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f67fb7fe640 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5605add32920 0x5605adce14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f67daffd640 10 client.8571 ms_handle_connect on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f67fb7fe640 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5605add32920 0x5605adce14c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81bad7b01c63b97d server_cookie=9acc7ef36415b737 in_seq=0 out_seq=0 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f67daffd640 1 -- 172.21.15.44:0/2804003299 <== mds.0 v2:172.21.15.44:6832/3044289683 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5605add0f2d0 con 0x5605add32920 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f67daffd640 10 client.8571 handle_command_reply: tid=0 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.291+0000 7f6805028740 1 client.8571 shutdown 2024-04-02T13:28:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6805028740 1 -- 172.21.15.44:0/2804003299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 msgr2=0x5605adcca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6805028740 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 0x5605adcca7a0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x5605adcf6880 tx=0x7f67ec0046c0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6805028740 1 -- 172.21.15.44:0/2804003299 shutdown_connections 2024-04-02T13:28:53.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6805028740 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.44:6832/3044289683,v1:172.21.15.44:6833/3044289683] conn(0x5605add32920 0x5605adce14c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6805028740 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5605add114d0 0x5605add32130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6805028740 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605adce93f0 0x5605adccace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f67daffd640 0 client.8571 ms_handle_reset on v2:172.21.15.44:6832/3044289683 2024-04-02T13:28:53.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6805028740 1 --2- 172.21.15.44:0/2804003299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5605adcc9be0 0x5605adcca7a0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6805028740 1 -- 172.21.15.44:0/2804003299 >> 172.21.15.44:0/2804003299 conn(0x5605adcc3dc0 msgr2=0x5605add32ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:53.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6805028740 1 -- 172.21.15.44:0/2804003299 shutdown_connections 2024-04-02T13:28:53.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6805028740 1 -- 172.21.15.44:0/2804003299 wait complete. 2024-04-02T13:28:53.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6805028740 20 client.8571 trim_cache size 0 max 16384 2024-04-02T13:28:53.302 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:53.302 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-02T13:28:53.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6802360640 1 -- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f67e806c8f0 msgr2=0x7f67e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6802360640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f67e806c8f0 0x7f67e806edb0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f67fc070c80 tx=0x7f67ec008040 comp rx=0 tx=0).stop 2024-04-02T13:28:53.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6802360640 1 -- 172.21.15.44:0/775239447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67fc0d0c50 msgr2=0x7f67fc15b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6802360640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67fc0d0c50 0x7f67fc15b110 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f67f000c780 tx=0x7f67f000cc40 comp rx=0 tx=0).stop 2024-04-02T13:28:53.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6802360640 1 -- 172.21.15.44:0/775239447 shutdown_connections 2024-04-02T13:28:53.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6802360640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f67e806c8f0 0x7f67e806edb0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6802360640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67fc15e4a0 0x7f67fc15b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6802360640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67fc0d0c50 0x7f67fc15b110 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6802360640 1 --2- 172.21.15.44:0/775239447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67fc06d550 0x7f67fc15c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6802360640 1 -- 172.21.15.44:0/775239447 >> 172.21.15.44:0/775239447 conn(0x7f67fc09aa20 msgr2=0x7f67fc06c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:53.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6802360640 1 -- 172.21.15.44:0/775239447 shutdown_connections 2024-04-02T13:28:53.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.292+0000 7f6802360640 1 -- 172.21.15.44:0/775239447 wait complete. 2024-04-02T13:28:53.305 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-02T13:28:53.358 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:28:53.361 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:28:53.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.403+0000 7fce9daa5640 1 Processor -- start 2024-04-02T13:28:53.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.403+0000 7fce9daa5640 1 -- start start 2024-04-02T13:28:53.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.404+0000 7fce9daa5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fce9815b080 0x7fce9815f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.404+0000 7fce9daa5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fce980cdc60 0x7fce98072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.404+0000 7fce9daa5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce9815fab0 con 0x7fce980cd800 2024-04-02T13:28:53.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.404+0000 7fce9daa5640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fce9815b460 con 0x7fce9815b080 2024-04-02T13:28:53.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.404+0000 7fce9daa5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fce980d0590 con 0x7fce980cdc60 2024-04-02T13:28:53.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.404+0000 7fce977fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fce9815b080 0x7fce9815f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.404+0000 7fce977fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fce9815b080 0x7fce9815f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51960/0 (socket says 172.21.15.44:51960) 2024-04-02T13:28:53.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.404+0000 7fce977fe640 1 -- 172.21.15.44:0/4063619263 learned_addr learned my addr 172.21.15.44:0/4063619263 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:53.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.405+0000 7fce97fff640 1 --2- 172.21.15.44:0/4063619263 >> v2:172.21.15.69:3300/0 conn(0x7fce980cdc60 0x7fce98072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.405+0000 7fce977fe640 1 -- 172.21.15.44:0/4063619263 >> v2:172.21.15.69:3300/0 conn(0x7fce980cdc60 msgr2=0x7fce98072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:53.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.405+0000 7fce977fe640 1 --2- 172.21.15.44:0/4063619263 >> v2:172.21.15.69:3300/0 conn(0x7fce980cdc60 0x7fce98072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.405+0000 7fce977fe640 1 -- 172.21.15.44:0/4063619263 >> v1:172.21.15.69:6789/0 conn(0x7fce980cd800 legacy=0x7fce98072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:53.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.405+0000 7fce977fe640 1 -- 172.21.15.44:0/4063619263 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce980cf980 con 0x7fce9815b080 2024-04-02T13:28:53.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.405+0000 7fce97fff640 1 --2- 172.21.15.44:0/4063619263 >> v2:172.21.15.69:3300/0 conn(0x7fce980cdc60 0x7fce98072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:53.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.405+0000 7fce977fe640 1 --2- 172.21.15.44:0/4063619263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce9815b080 0x7fce9815f570 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fce980cf470 tx=0x7fce8802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2602281b17261317 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:53.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.406+0000 7fce967fc640 1 -- 172.21.15.44:0/4063619263 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce88035020 con 0x7fce9815b080 2024-04-02T13:28:53.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.406+0000 7fce967fc640 1 -- 172.21.15.44:0/4063619263 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce88031870 con 0x7fce9815b080 2024-04-02T13:28:53.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.406+0000 7fce967fc640 1 -- 172.21.15.44:0/4063619263 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce88031c40 con 0x7fce9815b080 2024-04-02T13:28:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.406+0000 7fce9daa5640 1 -- 172.21.15.44:0/4063619263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce9815b080 msgr2=0x7fce9815f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.406+0000 7fce9daa5640 1 --2- 172.21.15.44:0/4063619263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce9815b080 0x7fce9815f570 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fce980cf470 tx=0x7fce8802b910 comp rx=0 tx=0).stop 2024-04-02T13:28:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.406+0000 7fce9daa5640 1 -- 172.21.15.44:0/4063619263 shutdown_connections 2024-04-02T13:28:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.406+0000 7fce9daa5640 1 --2- 172.21.15.44:0/4063619263 >> v2:172.21.15.69:3300/0 conn(0x7fce980cdc60 0x7fce98072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.406+0000 7fce9daa5640 1 --2- 172.21.15.44:0/4063619263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce9815b080 0x7fce9815f570 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.406+0000 7fce9daa5640 1 -- 172.21.15.44:0/4063619263 >> 172.21.15.44:0/4063619263 conn(0x7fce980b2a80 msgr2=0x7fce980b2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:53.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.407+0000 7fce9daa5640 1 -- 172.21.15.44:0/4063619263 shutdown_connections 2024-04-02T13:28:53.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.407+0000 7fce9daa5640 1 -- 172.21.15.44:0/4063619263 wait complete. 2024-04-02T13:28:53.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.408+0000 7fce9daa5640 1 Processor -- start 2024-04-02T13:28:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.408+0000 7fce9daa5640 1 -- start start 2024-04-02T13:28:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce9daa5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce980cd800 0x7fce98144530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce9daa5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce980cdc60 0x7fce981413c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce9daa5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce9815b080 0x7fce9815dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce9daa5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fce980d0220 con 0x7fce980cd800 2024-04-02T13:28:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce9daa5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fce980cff40 con 0x7fce980cdc60 2024-04-02T13:28:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce9daa5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fce980ce0f0 con 0x7fce9815b080 2024-04-02T13:28:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce977fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce980cd800 0x7fce98144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce977fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce980cd800 0x7fce98144530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51974/0 (socket says 172.21.15.44:51974) 2024-04-02T13:28:53.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce977fe640 1 -- 172.21.15.44:0/4216426377 learned_addr learned my addr 172.21.15.44:0/4216426377 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:53.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce96ffd640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce980cdc60 0x7fce981413c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce97fff640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce9815b080 0x7fce9815dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce977fe640 1 -- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce9815b080 msgr2=0x7fce9815dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce977fe640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce9815b080 0x7fce9815dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce977fe640 1 -- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce980cdc60 msgr2=0x7fce981413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce977fe640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce980cdc60 0x7fce981413c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce977fe640 1 -- 172.21.15.44:0/4216426377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce9815e4f0 con 0x7fce980cd800 2024-04-02T13:28:53.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.409+0000 7fce977fe640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce980cd800 0x7fce98144530 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fce88000c00 tx=0x7fce88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:53.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.410+0000 7fce7ffff640 1 -- 172.21.15.44:0/4216426377 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce8804a070 con 0x7fce980cd800 2024-04-02T13:28:53.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.410+0000 7fce7ffff640 1 -- 172.21.15.44:0/4216426377 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce880453f0 con 0x7fce980cd800 2024-04-02T13:28:53.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.410+0000 7fce9daa5640 1 -- 172.21.15.44:0/4216426377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce98141d10 con 0x7fce980cd800 2024-04-02T13:28:53.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.410+0000 7fce7ffff640 1 -- 172.21.15.44:0/4216426377 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce880457e0 con 0x7fce980cd800 2024-04-02T13:28:53.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.410+0000 7fce9daa5640 1 -- 172.21.15.44:0/4216426377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce981422d0 con 0x7fce980cd800 2024-04-02T13:28:53.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.411+0000 7fce9daa5640 1 -- 172.21.15.44:0/4216426377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce98068c00 con 0x7fce980cd800 2024-04-02T13:28:53.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.411+0000 7fce7ffff640 1 -- 172.21.15.44:0/4216426377 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fce8804d070 con 0x7fce980cd800 2024-04-02T13:28:53.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.411+0000 7fce7ffff640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce8406c8f0 0x7fce8406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.412+0000 7fce7ffff640 1 -- 172.21.15.44:0/4216426377 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fce880afd80 con 0x7fce980cd800 2024-04-02T13:28:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.412+0000 7fce96ffd640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce8406c8f0 0x7fce8406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.412+0000 7fce96ffd640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce8406c8f0 0x7fce8406edb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fce981f44e0 tx=0x7fce8c007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:53.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.416+0000 7fce7ffff640 1 -- 172.21.15.44:0/4216426377 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fce8807d940 con 0x7fce980cd800 2024-04-02T13:28:53.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.638+0000 7fce9daa5640 1 -- 172.21.15.44:0/4216426377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fce980cef90 con 0x7fce980cd800 2024-04-02T13:28:53.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.638+0000 7fce7ffff640 1 -- 172.21.15.44:0/4216426377 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fce88081c40 con 0x7fce980cd800 2024-04-02T13:28:53.641 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:53.642 INFO:teuthology.orchestra.run.smithi044.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-02T13:28:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.640+0000 7fce9daa5640 1 -- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce8406c8f0 msgr2=0x7fce8406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.640+0000 7fce9daa5640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce8406c8f0 0x7fce8406edb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fce981f44e0 tx=0x7fce8c007e30 comp rx=0 tx=0).stop 2024-04-02T13:28:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.640+0000 7fce9daa5640 1 -- 172.21.15.44:0/4216426377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce980cd800 msgr2=0x7fce98144530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.640+0000 7fce9daa5640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce980cd800 0x7fce98144530 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fce88000c00 tx=0x7fce88004730 comp rx=0 tx=0).stop 2024-04-02T13:28:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.640+0000 7fce9daa5640 1 -- 172.21.15.44:0/4216426377 shutdown_connections 2024-04-02T13:28:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.640+0000 7fce9daa5640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fce8406c8f0 0x7fce8406edb0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.640+0000 7fce9daa5640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce9815b080 0x7fce9815dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.640+0000 7fce9daa5640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce980cdc60 0x7fce981413c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.640+0000 7fce9daa5640 1 --2- 172.21.15.44:0/4216426377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce980cd800 0x7fce98144530 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.640+0000 7fce9daa5640 1 -- 172.21.15.44:0/4216426377 >> 172.21.15.44:0/4216426377 conn(0x7fce980b2a80 msgr2=0x7fce9806eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.640+0000 7fce9daa5640 1 -- 172.21.15.44:0/4216426377 shutdown_connections 2024-04-02T13:28:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.640+0000 7fce9daa5640 1 -- 172.21.15.44:0/4216426377 wait complete. 2024-04-02T13:28:53.654 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-02T13:28:53.654 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:28:53.654 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:28:53.732 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:28:53.777 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:28:53.778 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:28:53.778 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:28:53.778 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:28:53.793 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:28:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.932+0000 7fd17fa2c640 1 Processor -- start 2024-04-02T13:28:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.932+0000 7fd17fa2c640 1 -- start start 2024-04-02T13:28:53.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.933+0000 7fd17fa2c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17815e4a0 0x7fd1780aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.933+0000 7fd17fa2c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd1780d0c50 0x7fd1780d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.933+0000 7fd17fa2c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1780cc600 con 0x7fd17806ce40 2024-04-02T13:28:53.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.933+0000 7fd17fa2c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd1780cc8e0 con 0x7fd1780d0c50 2024-04-02T13:28:53.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.933+0000 7fd17fa2c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd1780ccbc0 con 0x7fd17815e4a0 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.933+0000 7fd17d7c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17815e4a0 0x7fd1780aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.933+0000 7fd17cfc7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd1780d0c50 0x7fd1780d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.933+0000 7fd17d7c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17815e4a0 0x7fd1780aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:50104/0 (socket says 172.21.15.44:50104) 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.933+0000 7fd17d7c8640 1 -- 172.21.15.44:0/2207079222 learned_addr learned my addr 172.21.15.44:0/2207079222 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.933+0000 7fd17cfc7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd1780d0c50 0x7fd1780d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:37460/0 (socket says 172.21.15.44:37460) 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.933+0000 7fd17dfc9640 1 --1- 172.21.15.44:0/2207079222 >> v1:172.21.15.69:6789/0 conn(0x7fd17806ce40 0x7fd1780cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:57596/0 (socket says 172.21.15.44:57596) 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.934+0000 7fd177fff640 1 -- 172.21.15.44:0/2207079222 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 258740531 0 0) 0x7fd1780cc600 con 0x7fd17806ce40 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.934+0000 7fd177fff640 1 -- 172.21.15.44:0/2207079222 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd160003520 con 0x7fd17806ce40 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.934+0000 7fd17cfc7640 1 -- 172.21.15.44:0/2207079222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17815e4a0 msgr2=0x7fd1780aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.934+0000 7fd17cfc7640 1 --2- 172.21.15.44:0/2207079222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17815e4a0 0x7fd1780aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.934+0000 7fd17cfc7640 1 -- 172.21.15.44:0/2207079222 >> v1:172.21.15.69:6789/0 conn(0x7fd17806ce40 legacy=0x7fd1780cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.934+0000 7fd17cfc7640 1 -- 172.21.15.44:0/2207079222 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1780ccea0 con 0x7fd1780d0c50 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.934+0000 7fd17d7c8640 1 --2- 172.21.15.44:0/2207079222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17815e4a0 0x7fd1780aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:28:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.934+0000 7fd17cfc7640 1 --2- 172.21.15.44:0/2207079222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1780d0c50 0x7fd1780d10b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0097a0 tx=0x7fd16c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94a06a348cb9f4ac server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:53.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.935+0000 7fd177fff640 1 -- 172.21.15.44:0/2207079222 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c035020 con 0x7fd1780d0c50 2024-04-02T13:28:53.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.935+0000 7fd177fff640 1 -- 172.21.15.44:0/2207079222 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd16c031830 con 0x7fd1780d0c50 2024-04-02T13:28:53.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.935+0000 7fd177fff640 1 -- 172.21.15.44:0/2207079222 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c031be0 con 0x7fd1780d0c50 2024-04-02T13:28:53.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.935+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2207079222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1780d0c50 msgr2=0x7fd1780d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.935+0000 7fd17fa2c640 1 --2- 172.21.15.44:0/2207079222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1780d0c50 0x7fd1780d10b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0097a0 tx=0x7fd16c02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.935+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2207079222 shutdown_connections 2024-04-02T13:28:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.935+0000 7fd17fa2c640 1 --2- 172.21.15.44:0/2207079222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1780d0c50 0x7fd1780d10b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.935+0000 7fd17fa2c640 1 --2- 172.21.15.44:0/2207079222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17815e4a0 0x7fd1780aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.935+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2207079222 >> 172.21.15.44:0/2207079222 conn(0x7fd17809aa20 msgr2=0x7fd17809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.935+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2207079222 shutdown_connections 2024-04-02T13:28:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.935+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2207079222 wait complete. 2024-04-02T13:28:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.936+0000 7fd17fa2c640 1 Processor -- start 2024-04-02T13:28:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.936+0000 7fd17fa2c640 1 -- start start 2024-04-02T13:28:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.937+0000 7fd17fa2c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd17806ce40 0x7fd1780c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.937+0000 7fd17fa2c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1780d0c50 0x7fd178160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.937+0000 7fd17fa2c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17815e4a0 0x7fd178160950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.937+0000 7fd17fa2c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd17815e880 con 0x7fd17806ce40 2024-04-02T13:28:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.937+0000 7fd17fa2c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd1780cd460 con 0x7fd1780d0c50 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.937+0000 7fd17fa2c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd1780af130 con 0x7fd17815e4a0 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.937+0000 7fd17d7c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd17806ce40 0x7fd1780c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.937+0000 7fd17d7c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd17806ce40 0x7fd1780c83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51990/0 (socket says 172.21.15.44:51990) 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.937+0000 7fd17d7c8640 1 -- 172.21.15.44:0/2347672176 learned_addr learned my addr 172.21.15.44:0/2347672176 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.937+0000 7fd17cfc7640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1780d0c50 0x7fd178160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.937+0000 7fd17dfc9640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17815e4a0 0x7fd178160950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.938+0000 7fd17d7c8640 1 -- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17815e4a0 msgr2=0x7fd178160950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.938+0000 7fd17d7c8640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17815e4a0 0x7fd178160950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.938+0000 7fd17d7c8640 1 -- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1780d0c50 msgr2=0x7fd178160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.938+0000 7fd17d7c8640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1780d0c50 0x7fd178160410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.938+0000 7fd17d7c8640 1 -- 172.21.15.44:0/2347672176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1780cb3c0 con 0x7fd17806ce40 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.938+0000 7fd17d7c8640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd17806ce40 0x7fd1780c83b0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fd17000e9e0 tx=0x7fd17000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.938+0000 7fd175ffb640 1 -- 172.21.15.44:0/2347672176 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd170017070 con 0x7fd17806ce40 2024-04-02T13:28:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.938+0000 7fd175ffb640 1 -- 172.21.15.44:0/2347672176 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd17000c9d0 con 0x7fd17806ce40 2024-04-02T13:28:53.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.938+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2347672176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1780cb6a0 con 0x7fd17806ce40 2024-04-02T13:28:53.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.938+0000 7fd175ffb640 1 -- 172.21.15.44:0/2347672176 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17000cd80 con 0x7fd17806ce40 2024-04-02T13:28:53.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.938+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2347672176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1780c5e30 con 0x7fd17806ce40 2024-04-02T13:28:53.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.939+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2347672176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1780684f0 con 0x7fd17806ce40 2024-04-02T13:28:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.940+0000 7fd175ffb640 1 -- 172.21.15.44:0/2347672176 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd17001a070 con 0x7fd17806ce40 2024-04-02T13:28:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.940+0000 7fd175ffb640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd16406c8f0 0x7fd16406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.940+0000 7fd175ffb640 1 -- 172.21.15.44:0/2347672176 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fd17008bbd0 con 0x7fd17806ce40 2024-04-02T13:28:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.940+0000 7fd17cfc7640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd16406c8f0 0x7fd16406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.941+0000 7fd17cfc7640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd16406c8f0 0x7fd16406edb0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd16c036000 tx=0x7fd16c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:53.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:53.945+0000 7fd175ffb640 1 -- 172.21.15.44:0/2347672176 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd170059710 con 0x7fd17806ce40 2024-04-02T13:28:54.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.129+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2347672176 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd17815dba0 con 0x7fd17806ce40 2024-04-02T13:28:54.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.131+0000 7fd175ffb640 1 -- 172.21.15.44:0/2347672176 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+17288 (secure 0 0 0) 0x7fd17005da10 con 0x7fd17806ce40 2024-04-02T13:28:54.134 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:54.134 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":95,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:28:15.300549+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:26:41.300970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-02T13:26:41.979586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.24","mappings":[{"from":1,"to":5}]},{"pgid":"14.2c","mappings":[{"from":1,"to":2}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6833/2250279975":"2024-04-03T13:28:15.300517+0000","172.21.15.44:6832/2250279975":"2024-04-03T13:28:15.300517+0000","172.21.15.44:6833/719680934":"2024-04-03T13:26:55.589090+0000","172.21.15.44:6832/719680934":"2024-04-03T13:26:55.589090+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:28:54.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.132+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd16406c8f0 msgr2=0x7fd16406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.132+0000 7fd17fa2c640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd16406c8f0 0x7fd16406edb0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd16c036000 tx=0x7fd16c008030 comp rx=0 tx=0).stop 2024-04-02T13:28:54.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.132+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2347672176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd17806ce40 msgr2=0x7fd1780c83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.132+0000 7fd17fa2c640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd17806ce40 0x7fd1780c83b0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fd17000e9e0 tx=0x7fd17000eea0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.133+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2347672176 shutdown_connections 2024-04-02T13:28:54.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.133+0000 7fd17fa2c640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd16406c8f0 0x7fd16406edb0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.133+0000 7fd17fa2c640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17815e4a0 0x7fd178160950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.133+0000 7fd17fa2c640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1780d0c50 0x7fd178160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.133+0000 7fd17fa2c640 1 --2- 172.21.15.44:0/2347672176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd17806ce40 0x7fd1780c83b0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.133+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2347672176 >> 172.21.15.44:0/2347672176 conn(0x7fd17809aa20 msgr2=0x7fd17809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:54.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.133+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2347672176 shutdown_connections 2024-04-02T13:28:54.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.133+0000 7fd17fa2c640 1 -- 172.21.15.44:0/2347672176 wait complete. 2024-04-02T13:28:54.146 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:26:41.300970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:26:41.979586+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:28:54.147 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:28:54.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.249+0000 7eff17c55640 1 Processor -- start 2024-04-02T13:28:54.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.249+0000 7eff17c55640 1 -- start start 2024-04-02T13:28:54.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.250+0000 7eff17c55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff1015e420 0x7eff100ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.250+0000 7eff17c55640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7eff100cfe10 0x7eff100cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.250+0000 7eff17c55640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff1006c220 con 0x7eff100721f0 2024-04-02T13:28:54.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.250+0000 7eff17c55640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7eff1015f7e0 con 0x7eff100cfe10 2024-04-02T13:28:54.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.250+0000 7eff17c55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7eff1015ec60 con 0x7eff1015e420 2024-04-02T13:28:54.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.250+0000 7eff161f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7eff100cfe10 0x7eff100cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.250+0000 7eff161f2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7eff100cfe10 0x7eff100cdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52004/0 (socket says 172.21.15.44:52004) 2024-04-02T13:28:54.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.250+0000 7eff161f2640 1 -- 172.21.15.44:0/2108651949 learned_addr learned my addr 172.21.15.44:0/2108651949 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:54.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.250+0000 7eff161f2640 1 -- 172.21.15.44:0/2108651949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff1015e420 msgr2=0x7eff100ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:54.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.250+0000 7eff161f2640 1 --2- 172.21.15.44:0/2108651949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff1015e420 0x7eff100ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.250+0000 7eff161f2640 1 -- 172.21.15.44:0/2108651949 >> v1:172.21.15.69:6789/0 conn(0x7eff100721f0 legacy=0x7eff100cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:54.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.250+0000 7eff161f2640 1 -- 172.21.15.44:0/2108651949 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1015ef40 con 0x7eff100cfe10 2024-04-02T13:28:54.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.251+0000 7eff161f2640 1 --2- 172.21.15.44:0/2108651949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff100cfe10 0x7eff100cdae0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7eff08009870 tx=0x7eff0802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f581601a2c2ffc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:54.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.251+0000 7eff149ef640 1 -- 172.21.15.44:0/2108651949 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08038070 con 0x7eff100cfe10 2024-04-02T13:28:54.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.251+0000 7eff149ef640 1 -- 172.21.15.44:0/2108651949 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff08031740 con 0x7eff100cfe10 2024-04-02T13:28:54.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.251+0000 7eff149ef640 1 -- 172.21.15.44:0/2108651949 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0802f050 con 0x7eff100cfe10 2024-04-02T13:28:54.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.252+0000 7eff17c55640 1 -- 172.21.15.44:0/2108651949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff100cfe10 msgr2=0x7eff100cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.252+0000 7eff17c55640 1 --2- 172.21.15.44:0/2108651949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff100cfe10 0x7eff100cdae0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7eff08009870 tx=0x7eff0802b7e0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.252+0000 7eff17c55640 1 -- 172.21.15.44:0/2108651949 shutdown_connections 2024-04-02T13:28:54.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.252+0000 7eff17c55640 1 --2- 172.21.15.44:0/2108651949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff100cfe10 0x7eff100cdae0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.252+0000 7eff17c55640 1 --2- 172.21.15.44:0/2108651949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff1015e420 0x7eff100ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.252+0000 7eff17c55640 1 -- 172.21.15.44:0/2108651949 >> 172.21.15.44:0/2108651949 conn(0x7eff100b2a10 msgr2=0x7eff100b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:54.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.252+0000 7eff17c55640 1 -- 172.21.15.44:0/2108651949 shutdown_connections 2024-04-02T13:28:54.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.252+0000 7eff17c55640 1 -- 172.21.15.44:0/2108651949 wait complete. 2024-04-02T13:28:54.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.253+0000 7eff17c55640 1 Processor -- start 2024-04-02T13:28:54.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.253+0000 7eff17c55640 1 -- start start 2024-04-02T13:28:54.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff17c55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff100721f0 0x7eff10154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff17c55640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff100cfe10 0x7eff10151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff17c55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff1015e420 0x7eff10154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff17c55640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7eff100cf1e0 con 0x7eff100cfe10 2024-04-02T13:28:54.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff17c55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7eff1015f500 con 0x7eff100721f0 2024-04-02T13:28:54.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff17c55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7eff100cf360 con 0x7eff1015e420 2024-04-02T13:28:54.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff151f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff100cfe10 0x7eff10151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff151f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff100cfe10 0x7eff10151e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52018/0 (socket says 172.21.15.44:52018) 2024-04-02T13:28:54.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff151f0640 1 -- 172.21.15.44:0/1691568236 learned_addr learned my addr 172.21.15.44:0/1691568236 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:54.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff161f2640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff1015e420 0x7eff10154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff159f1640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff100721f0 0x7eff10154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff151f0640 1 -- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff1015e420 msgr2=0x7eff10154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff151f0640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff1015e420 0x7eff10154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff151f0640 1 -- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff100721f0 msgr2=0x7eff10154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff151f0640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff100721f0 0x7eff10154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.254+0000 7eff151f0640 1 -- 172.21.15.44:0/1691568236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff101526e0 con 0x7eff100cfe10 2024-04-02T13:28:54.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.255+0000 7eff151f0640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff100cfe10 0x7eff10151e90 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7eff0400c460 tx=0x7eff0400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:54.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.255+0000 7eff0e7fc640 1 -- 172.21.15.44:0/1691568236 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff04016020 con 0x7eff100cfe10 2024-04-02T13:28:54.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.255+0000 7eff0e7fc640 1 -- 172.21.15.44:0/1691568236 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff04012870 con 0x7eff100cfe10 2024-04-02T13:28:54.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.255+0000 7eff17c55640 1 -- 172.21.15.44:0/1691568236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff100b46c0 con 0x7eff100cfe10 2024-04-02T13:28:54.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.255+0000 7eff0e7fc640 1 -- 172.21.15.44:0/1691568236 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff04012c60 con 0x7eff100cfe10 2024-04-02T13:28:54.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.255+0000 7eff17c55640 1 -- 172.21.15.44:0/1691568236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff100b4c80 con 0x7eff100cfe10 2024-04-02T13:28:54.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.256+0000 7eff17c55640 1 -- 172.21.15.44:0/1691568236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff100684f0 con 0x7eff100cfe10 2024-04-02T13:28:54.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.256+0000 7eff0e7fc640 1 -- 172.21.15.44:0/1691568236 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7eff04019020 con 0x7eff100cfe10 2024-04-02T13:28:54.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.257+0000 7eff0e7fc640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efefc06c8f0 0x7efefc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.257+0000 7eff0e7fc640 1 -- 172.21.15.44:0/1691568236 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7eff0408be80 con 0x7eff100cfe10 2024-04-02T13:28:54.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.257+0000 7eff159f1640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efefc06c8f0 0x7efefc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.257+0000 7eff159f1640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efefc06c8f0 0x7efefc06edb0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7eff101f81f0 tx=0x7eff000094f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:54.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.261+0000 7eff0e7fc640 1 -- 172.21.15.44:0/1691568236 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7eff04020020 con 0x7eff100cfe10 2024-04-02T13:28:54.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.445+0000 7eff17c55640 1 -- 172.21.15.44:0/1691568236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7eff100b4e50 con 0x7eff100cfe10 2024-04-02T13:28:54.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.445+0000 7eff0e7fc640 1 -- 172.21.15.44:0/1691568236 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7eff040182c0 con 0x7eff100cfe10 2024-04-02T13:28:54.448 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:28:54.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.447+0000 7eff17c55640 1 -- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efefc06c8f0 msgr2=0x7efefc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.447+0000 7eff17c55640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efefc06c8f0 0x7efefc06edb0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7eff101f81f0 tx=0x7eff000094f0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.447+0000 7eff17c55640 1 -- 172.21.15.44:0/1691568236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff100cfe10 msgr2=0x7eff10151e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.447+0000 7eff17c55640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff100cfe10 0x7eff10151e90 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7eff0400c460 tx=0x7eff0400c920 comp rx=0 tx=0).stop 2024-04-02T13:28:54.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.447+0000 7eff17c55640 1 -- 172.21.15.44:0/1691568236 shutdown_connections 2024-04-02T13:28:54.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.447+0000 7eff17c55640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efefc06c8f0 0x7efefc06edb0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.447+0000 7eff17c55640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff1015e420 0x7eff10154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.447+0000 7eff17c55640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff100cfe10 0x7eff10151e90 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.447+0000 7eff17c55640 1 --2- 172.21.15.44:0/1691568236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff100721f0 0x7eff10154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.447+0000 7eff17c55640 1 -- 172.21.15.44:0/1691568236 >> 172.21.15.44:0/1691568236 conn(0x7eff100b2a10 msgr2=0x7eff100af6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:54.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.447+0000 7eff17c55640 1 -- 172.21.15.44:0/1691568236 shutdown_connections 2024-04-02T13:28:54.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.447+0000 7eff17c55640 1 -- 172.21.15.44:0/1691568236 wait complete. 2024-04-02T13:28:54.460 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:28:54.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.562+0000 7fde2982a640 1 Processor -- start 2024-04-02T13:28:54.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.562+0000 7fde2982a640 1 -- start start 2024-04-02T13:28:54.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.563+0000 7fde2982a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde2415c1a0 0x7fde2406f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.563+0000 7fde2982a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fde240cfd60 0x7fde240d1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.563+0000 7fde2982a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde2415c580 con 0x7fde240d1ff0 2024-04-02T13:28:54.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.563+0000 7fde2982a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fde24070ad0 con 0x7fde240cfd60 2024-04-02T13:28:54.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.563+0000 7fde2982a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fde240d01c0 con 0x7fde2415c1a0 2024-04-02T13:28:54.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.564+0000 7fde22ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde2415c1a0 0x7fde2406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.564+0000 7fde227fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fde240cfd60 0x7fde240d1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.564+0000 7fde237fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fde240d1ff0 0x7fde240ce780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:35698/0 (socket says 172.21.15.44:35698) 2024-04-02T13:28:54.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.564+0000 7fde237fe640 1 -- 172.21.15.44:0/625958779 learned_addr learned my addr 172.21.15.44:0/625958779 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:54.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.564+0000 7fde22ffd640 1 -- 172.21.15.44:0/625958779 >> v2:172.21.15.69:3300/0 conn(0x7fde240cfd60 msgr2=0x7fde240d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:54.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.564+0000 7fde22ffd640 1 --2- 172.21.15.44:0/625958779 >> v2:172.21.15.69:3300/0 conn(0x7fde240cfd60 0x7fde240d1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.564+0000 7fde22ffd640 1 -- 172.21.15.44:0/625958779 >> v1:172.21.15.69:6789/0 conn(0x7fde240d1ff0 legacy=0x7fde240ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.564+0000 7fde22ffd640 1 -- 172.21.15.44:0/625958779 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde240d2480 con 0x7fde2415c1a0 2024-04-02T13:28:54.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.564+0000 7fde227fc640 1 --2- 172.21.15.44:0/625958779 >> v2:172.21.15.69:3300/0 conn(0x7fde240cfd60 0x7fde240d1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:28:54.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.565+0000 7fde22ffd640 1 --2- 172.21.15.44:0/625958779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde2415c1a0 0x7fde2406f370 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0097a0 tx=0x7fde1c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7838dc5dd93590bc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:54.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.565+0000 7fde21ffb640 1 -- 172.21.15.44:0/625958779 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c035020 con 0x7fde2415c1a0 2024-04-02T13:28:54.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.565+0000 7fde21ffb640 1 -- 172.21.15.44:0/625958779 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde1c031830 con 0x7fde2415c1a0 2024-04-02T13:28:54.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.565+0000 7fde21ffb640 1 -- 172.21.15.44:0/625958779 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c031c00 con 0x7fde2415c1a0 2024-04-02T13:28:54.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.565+0000 7fde2982a640 1 -- 172.21.15.44:0/625958779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde2415c1a0 msgr2=0x7fde2406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.565+0000 7fde2982a640 1 --2- 172.21.15.44:0/625958779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde2415c1a0 0x7fde2406f370 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0097a0 tx=0x7fde1c02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.565+0000 7fde2982a640 1 -- 172.21.15.44:0/625958779 shutdown_connections 2024-04-02T13:28:54.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.565+0000 7fde2982a640 1 --2- 172.21.15.44:0/625958779 >> v2:172.21.15.69:3300/0 conn(0x7fde240cfd60 0x7fde240d1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.565+0000 7fde2982a640 1 --2- 172.21.15.44:0/625958779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde2415c1a0 0x7fde2406f370 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.565+0000 7fde2982a640 1 -- 172.21.15.44:0/625958779 >> 172.21.15.44:0/625958779 conn(0x7fde2409aa20 msgr2=0x7fde2409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:54.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.565+0000 7fde2982a640 1 -- 172.21.15.44:0/625958779 shutdown_connections 2024-04-02T13:28:54.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.566+0000 7fde2982a640 1 -- 172.21.15.44:0/625958779 wait complete. 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.567+0000 7fde2982a640 1 Processor -- start 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.567+0000 7fde2982a640 1 -- start start 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.567+0000 7fde2982a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde240cfd60 0x7fde24152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.567+0000 7fde2982a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde240d1ff0 0x7fde240c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.567+0000 7fde2982a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde2415c1a0 0x7fde2415eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.567+0000 7fde2982a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fde2406fff0 con 0x7fde2415c1a0 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.567+0000 7fde2982a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fde2406fd10 con 0x7fde240d1ff0 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.567+0000 7fde2982a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fde240702d0 con 0x7fde240cfd60 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde237fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde2415c1a0 0x7fde2415eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde237fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde2415c1a0 0x7fde2415eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38502/0 (socket says 172.21.15.44:38502) 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde237fe640 1 -- 172.21.15.44:0/2954709769 learned_addr learned my addr 172.21.15.44:0/2954709769 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde22ffd640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde240cfd60 0x7fde24152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde227fc640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde240d1ff0 0x7fde240c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde237fe640 1 -- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde240cfd60 msgr2=0x7fde24152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde237fe640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde240cfd60 0x7fde24152a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde237fe640 1 -- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde240d1ff0 msgr2=0x7fde240c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde237fe640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde240d1ff0 0x7fde240c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde237fe640 1 -- 172.21.15.44:0/2954709769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2415da10 con 0x7fde2415c1a0 2024-04-02T13:28:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde227fc640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde240d1ff0 0x7fde240c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde237fe640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde2415c1a0 0x7fde2415eab0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fde1400cbf0 tx=0x7fde1400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde1bfff640 1 -- 172.21.15.44:0/2954709769 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14017070 con 0x7fde2415c1a0 2024-04-02T13:28:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde1bfff640 1 -- 172.21.15.44:0/2954709769 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde1400ad40 con 0x7fde2415c1a0 2024-04-02T13:28:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde2982a640 1 -- 172.21.15.44:0/2954709769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde2415dcf0 con 0x7fde2415c1a0 2024-04-02T13:28:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde1bfff640 1 -- 172.21.15.44:0/2954709769 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14012600 con 0x7fde2415c1a0 2024-04-02T13:28:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.568+0000 7fde2982a640 1 -- 172.21.15.44:0/2954709769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde2415e2b0 con 0x7fde2415c1a0 2024-04-02T13:28:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.569+0000 7fde2982a640 1 -- 172.21.15.44:0/2954709769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde24068c00 con 0x7fde2415c1a0 2024-04-02T13:28:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.570+0000 7fde1bfff640 1 -- 172.21.15.44:0/2954709769 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fde1401a070 con 0x7fde2415c1a0 2024-04-02T13:28:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.570+0000 7fde1bfff640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fde0406c8f0 0x7fde0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.570+0000 7fde1bfff640 1 -- 172.21.15.44:0/2954709769 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fde1408bac0 con 0x7fde2415c1a0 2024-04-02T13:28:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.570+0000 7fde22ffd640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fde0406c8f0 0x7fde0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.571+0000 7fde22ffd640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fde0406c8f0 0x7fde0406edb0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fde1c036000 tx=0x7fde1c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.575+0000 7fde1bfff640 1 -- 172.21.15.44:0/2954709769 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fde140595f0 con 0x7fde2415c1a0 2024-04-02T13:28:54.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.758+0000 7fde2982a640 1 -- 172.21.15.44:0/2954709769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fde240cddb0 con 0x7fde2415c1a0 2024-04-02T13:28:54.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.759+0000 7fde1bfff640 1 -- 172.21.15.44:0/2954709769 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7fde1405d8f0 con 0x7fde2415c1a0 2024-04-02T13:28:54.763 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:28:54.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.760+0000 7fde2982a640 1 -- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fde0406c8f0 msgr2=0x7fde0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.760+0000 7fde2982a640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fde0406c8f0 0x7fde0406edb0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fde1c036000 tx=0x7fde1c03c040 comp rx=0 tx=0).stop 2024-04-02T13:28:54.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.760+0000 7fde2982a640 1 -- 172.21.15.44:0/2954709769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde2415c1a0 msgr2=0x7fde2415eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.760+0000 7fde2982a640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde2415c1a0 0x7fde2415eab0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fde1400cbf0 tx=0x7fde1400a430 comp rx=0 tx=0).stop 2024-04-02T13:28:54.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.760+0000 7fde2982a640 1 -- 172.21.15.44:0/2954709769 shutdown_connections 2024-04-02T13:28:54.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.760+0000 7fde2982a640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fde0406c8f0 0x7fde0406edb0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.760+0000 7fde2982a640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde2415c1a0 0x7fde2415eab0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.760+0000 7fde2982a640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde240d1ff0 0x7fde240c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.760+0000 7fde2982a640 1 --2- 172.21.15.44:0/2954709769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde240cfd60 0x7fde24152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.760+0000 7fde2982a640 1 -- 172.21.15.44:0/2954709769 >> 172.21.15.44:0/2954709769 conn(0x7fde2409aa20 msgr2=0x7fde240ccde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:54.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.761+0000 7fde2982a640 1 -- 172.21.15.44:0/2954709769 shutdown_connections 2024-04-02T13:28:54.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.761+0000 7fde2982a640 1 -- 172.21.15.44:0/2954709769 wait complete. 2024-04-02T13:28:54.774 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:28:54.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.876+0000 7f3354ced640 1 Processor -- start 2024-04-02T13:28:54.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.876+0000 7f3354ced640 1 -- start start 2024-04-02T13:28:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.876+0000 7f3354ced640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f335015e1d0 0x7f33500d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.876+0000 7f3354ced640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f33500d2dd0 0x7f33500b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.876+0000 7f3354ced640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33500d1640 con 0x7f33500cec60 2024-04-02T13:28:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.876+0000 7f3354ced640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f33500cfe00 con 0x7f335015e1d0 2024-04-02T13:28:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.876+0000 7f3354ced640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f33500cf1f0 con 0x7f33500d2dd0 2024-04-02T13:28:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.877+0000 7f334e59c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f335015e1d0 0x7f33500d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.877+0000 7f334e59c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f335015e1d0 0x7f33500d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38504/0 (socket says 172.21.15.44:38504) 2024-04-02T13:28:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.877+0000 7f334e59c640 1 -- 172.21.15.44:0/841910648 learned_addr learned my addr 172.21.15.44:0/841910648 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.877+0000 7f334ed9d640 1 --2- 172.21.15.44:0/841910648 >> v2:172.21.15.69:3300/0 conn(0x7f33500d2dd0 0x7f33500b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.877+0000 7f334e59c640 1 -- 172.21.15.44:0/841910648 >> v2:172.21.15.69:3300/0 conn(0x7f33500d2dd0 msgr2=0x7f33500b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.877+0000 7f334e59c640 1 --2- 172.21.15.44:0/841910648 >> v2:172.21.15.69:3300/0 conn(0x7f33500d2dd0 0x7f33500b4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.877+0000 7f334e59c640 1 -- 172.21.15.44:0/841910648 >> v1:172.21.15.69:6789/0 conn(0x7f33500cec60 legacy=0x7f33500d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.877+0000 7f334e59c640 1 -- 172.21.15.44:0/841910648 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33500cf4d0 con 0x7f335015e1d0 2024-04-02T13:28:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.877+0000 7f334ed9d640 1 --2- 172.21.15.44:0/841910648 >> v2:172.21.15.69:3300/0 conn(0x7f33500d2dd0 0x7f33500b4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.878+0000 7f334e59c640 1 --2- 172.21.15.44:0/841910648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f335015e1d0 0x7f33500d1100 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f33500d4080 tx=0x7f334002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0d88833ec831b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.878+0000 7f334dd9b640 1 -- 172.21.15.44:0/841910648 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334002e070 con 0x7f335015e1d0 2024-04-02T13:28:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.878+0000 7f334dd9b640 1 -- 172.21.15.44:0/841910648 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3340037070 con 0x7f335015e1d0 2024-04-02T13:28:54.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.878+0000 7f334dd9b640 1 -- 172.21.15.44:0/841910648 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3340031a00 con 0x7f335015e1d0 2024-04-02T13:28:54.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.878+0000 7f3354ced640 1 -- 172.21.15.44:0/841910648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f335015e1d0 msgr2=0x7f33500d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.878+0000 7f3354ced640 1 --2- 172.21.15.44:0/841910648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f335015e1d0 0x7f33500d1100 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f33500d4080 tx=0x7f334002b880 comp rx=0 tx=0).stop 2024-04-02T13:28:54.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.878+0000 7f3354ced640 1 -- 172.21.15.44:0/841910648 shutdown_connections 2024-04-02T13:28:54.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.878+0000 7f3354ced640 1 --2- 172.21.15.44:0/841910648 >> v2:172.21.15.69:3300/0 conn(0x7f33500d2dd0 0x7f33500b4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.878+0000 7f3354ced640 1 --2- 172.21.15.44:0/841910648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f335015e1d0 0x7f33500d1100 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.878+0000 7f3354ced640 1 -- 172.21.15.44:0/841910648 >> 172.21.15.44:0/841910648 conn(0x7f335009d3b0 msgr2=0x7f33500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:54.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.878+0000 7f3354ced640 1 -- 172.21.15.44:0/841910648 shutdown_connections 2024-04-02T13:28:54.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.878+0000 7f3354ced640 1 -- 172.21.15.44:0/841910648 wait complete. 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.879+0000 7f3354ced640 1 Processor -- start 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.880+0000 7f3354ced640 1 -- start start 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.880+0000 7f3354ced640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33500cec60 0x7f33501598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.880+0000 7f3354ced640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33500d2dd0 0x7f335015f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.880+0000 7f3354ced640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f335015e1d0 0x7f33500cd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.880+0000 7f3354ced640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f335015e5b0 con 0x7f335015e1d0 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.880+0000 7f3354ced640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f33500cfa90 con 0x7f33500d2dd0 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.880+0000 7f3354ced640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f335015e730 con 0x7f33500cec60 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.880+0000 7f334ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f335015e1d0 0x7f33500cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.880+0000 7f334ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f335015e1d0 0x7f33500cd340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38510/0 (socket says 172.21.15.44:38510) 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.880+0000 7f334ed9d640 1 -- 172.21.15.44:0/326915628 learned_addr learned my addr 172.21.15.44:0/326915628 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f334e59c640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33500cec60 0x7f33501598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f3347fff640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33500d2dd0 0x7f335015f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f334ed9d640 1 -- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33500cec60 msgr2=0x7f33501598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f334ed9d640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33500cec60 0x7f33501598f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f334ed9d640 1 -- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33500d2dd0 msgr2=0x7f335015f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f334ed9d640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33500d2dd0 0x7f335015f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f334ed9d640 1 -- 172.21.15.44:0/326915628 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33500cc2a0 con 0x7f335015e1d0 2024-04-02T13:28:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f3347fff640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33500d2dd0 0x7f335015f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f334ed9d640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f335015e1d0 0x7f33500cd340 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f334800cdb0 tx=0x7f334800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f3346ffd640 1 -- 172.21.15.44:0/326915628 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334800f680 con 0x7f335015e1d0 2024-04-02T13:28:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f3346ffd640 1 -- 172.21.15.44:0/326915628 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f334800f840 con 0x7f335015e1d0 2024-04-02T13:28:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f3354ced640 1 -- 172.21.15.44:0/326915628 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33500cc580 con 0x7f335015e1d0 2024-04-02T13:28:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f3346ffd640 1 -- 172.21.15.44:0/326915628 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334800fc30 con 0x7f335015e1d0 2024-04-02T13:28:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.881+0000 7f3354ced640 1 -- 172.21.15.44:0/326915628 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33500ccb40 con 0x7f335015e1d0 2024-04-02T13:28:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.882+0000 7f3354ced640 1 -- 172.21.15.44:0/326915628 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33500684f0 con 0x7f335015e1d0 2024-04-02T13:28:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.882+0000 7f3346ffd640 1 -- 172.21.15.44:0/326915628 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f33480213f0 con 0x7f335015e1d0 2024-04-02T13:28:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.883+0000 7f3346ffd640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f333006c8f0 0x7f333006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.883+0000 7f3346ffd640 1 -- 172.21.15.44:0/326915628 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f334808cbe0 con 0x7f335015e1d0 2024-04-02T13:28:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.883+0000 7f334e59c640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f333006c8f0 0x7f333006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.883+0000 7f334e59c640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f333006c8f0 0x7f333006edb0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3340036000 tx=0x7f3340008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:54.887+0000 7f3346ffd640 1 -- 172.21.15.44:0/326915628 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3348014030 con 0x7f335015e1d0 2024-04-02T13:28:55.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.071+0000 7f3354ced640 1 -- 172.21.15.44:0/326915628 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f33500b99a0 con 0x7f335015e1d0 2024-04-02T13:28:55.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.071+0000 7f3346ffd640 1 -- 172.21.15.44:0/326915628 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f334805ad60 con 0x7f335015e1d0 2024-04-02T13:28:55.074 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:28:55.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.073+0000 7f3354ced640 1 -- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f333006c8f0 msgr2=0x7f333006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:55.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.073+0000 7f3354ced640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f333006c8f0 0x7f333006edb0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3340036000 tx=0x7f3340008030 comp rx=0 tx=0).stop 2024-04-02T13:28:55.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.073+0000 7f3354ced640 1 -- 172.21.15.44:0/326915628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f335015e1d0 msgr2=0x7f33500cd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:55.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.073+0000 7f3354ced640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f335015e1d0 0x7f33500cd340 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f334800cdb0 tx=0x7f334800a650 comp rx=0 tx=0).stop 2024-04-02T13:28:55.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.073+0000 7f3354ced640 1 -- 172.21.15.44:0/326915628 shutdown_connections 2024-04-02T13:28:55.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.073+0000 7f3354ced640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f333006c8f0 0x7f333006edb0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.073+0000 7f3354ced640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f335015e1d0 0x7f33500cd340 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.073+0000 7f3354ced640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33500d2dd0 0x7f335015f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.073+0000 7f3354ced640 1 --2- 172.21.15.44:0/326915628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33500cec60 0x7f33501598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.073+0000 7f3354ced640 1 -- 172.21.15.44:0/326915628 >> 172.21.15.44:0/326915628 conn(0x7f335009d3b0 msgr2=0x7f33500b8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:55.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.073+0000 7f3354ced640 1 -- 172.21.15.44:0/326915628 shutdown_connections 2024-04-02T13:28:55.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.073+0000 7f3354ced640 1 -- 172.21.15.44:0/326915628 wait complete. 2024-04-02T13:28:55.087 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:28:55.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.187+0000 7fb0786a5640 1 Processor -- start 2024-04-02T13:28:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.187+0000 7fb0786a5640 1 -- start start 2024-04-02T13:28:55.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb0786a5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb07015e550 0x7fb07006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:55.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb0786a5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb07006d530 0x7fb07015e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:55.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb0786a5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb07015ee70 0x7fb0700ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:55.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb0786a5640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fb07006d970 con 0x7fb07015ee70 2024-04-02T13:28:55.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb0786a5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb07015fb50 con 0x7fb07006d530 2024-04-02T13:28:55.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb0786a5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb07015f300 con 0x7fb07015e550 2024-04-02T13:28:55.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb076c42640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb07015ee70 0x7fb0700ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:55.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb076c42640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb07015ee70 0x7fb0700ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38512/0 (socket says 172.21.15.44:38512) 2024-04-02T13:28:55.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb076c42640 1 -- 172.21.15.44:0/2844803251 learned_addr learned my addr 172.21.15.44:0/2844803251 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:55.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb075c40640 1 --2- 172.21.15.44:0/2844803251 >> v2:172.21.15.69:3300/0 conn(0x7fb07006d530 0x7fb07015e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:55.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb076441640 1 --2- 172.21.15.44:0/2844803251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb07015e550 0x7fb07006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:55.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb076c42640 1 -- 172.21.15.44:0/2844803251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb07015e550 msgr2=0x7fb07006cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:55.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb076c42640 1 --2- 172.21.15.44:0/2844803251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb07015e550 0x7fb07006cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb076c42640 1 -- 172.21.15.44:0/2844803251 >> v2:172.21.15.69:3300/0 conn(0x7fb07006d530 msgr2=0x7fb07015e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:55.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb076c42640 1 --2- 172.21.15.44:0/2844803251 >> v2:172.21.15.69:3300/0 conn(0x7fb07006d530 0x7fb07015e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.188+0000 7fb076c42640 1 -- 172.21.15.44:0/2844803251 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07006ca90 con 0x7fb07015ee70 2024-04-02T13:28:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.189+0000 7fb076c42640 1 --2- 172.21.15.44:0/2844803251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb07015ee70 0x7fb0700ce6e0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fb0700758e0 tx=0x7fb06802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80595eb6d57921d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.189+0000 7fb07543f640 1 -- 172.21.15.44:0/2844803251 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06802cb00 con 0x7fb07015ee70 2024-04-02T13:28:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.189+0000 7fb07543f640 1 -- 172.21.15.44:0/2844803251 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb06802cca0 con 0x7fb07015ee70 2024-04-02T13:28:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.189+0000 7fb07543f640 1 -- 172.21.15.44:0/2844803251 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb068034610 con 0x7fb07015ee70 2024-04-02T13:28:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.190+0000 7fb0786a5640 1 -- 172.21.15.44:0/2844803251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb07015ee70 msgr2=0x7fb0700ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.190+0000 7fb0786a5640 1 --2- 172.21.15.44:0/2844803251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb07015ee70 0x7fb0700ce6e0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fb0700758e0 tx=0x7fb06802bf60 comp rx=0 tx=0).stop 2024-04-02T13:28:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.190+0000 7fb0786a5640 1 -- 172.21.15.44:0/2844803251 shutdown_connections 2024-04-02T13:28:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.190+0000 7fb0786a5640 1 --2- 172.21.15.44:0/2844803251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb07015ee70 0x7fb0700ce6e0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.190+0000 7fb0786a5640 1 --2- 172.21.15.44:0/2844803251 >> v2:172.21.15.69:3300/0 conn(0x7fb07006d530 0x7fb07015e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.190+0000 7fb0786a5640 1 --2- 172.21.15.44:0/2844803251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb07015e550 0x7fb07006cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.190+0000 7fb0786a5640 1 -- 172.21.15.44:0/2844803251 >> 172.21.15.44:0/2844803251 conn(0x7fb07009aa20 msgr2=0x7fb07009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.190+0000 7fb0786a5640 1 -- 172.21.15.44:0/2844803251 shutdown_connections 2024-04-02T13:28:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.190+0000 7fb0786a5640 1 -- 172.21.15.44:0/2844803251 wait complete. 2024-04-02T13:28:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.191+0000 7fb0786a5640 1 Processor -- start 2024-04-02T13:28:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.191+0000 7fb0786a5640 1 -- start start 2024-04-02T13:28:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb0786a5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb07006d530 0x7fb0700cb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb0786a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb07015e550 0x7fb07009bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb0786a5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb07015ee70 0x7fb07014b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb0786a5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb0700d02b0 con 0x7fb07015ee70 2024-04-02T13:28:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb0786a5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb07015f870 con 0x7fb07015e550 2024-04-02T13:28:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb0786a5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb0700d0590 con 0x7fb07006d530 2024-04-02T13:28:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb076c42640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb07015ee70 0x7fb07014b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb076c42640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb07015ee70 0x7fb07014b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38524/0 (socket says 172.21.15.44:38524) 2024-04-02T13:28:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb076c42640 1 -- 172.21.15.44:0/3740217383 learned_addr learned my addr 172.21.15.44:0/3740217383 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb076441640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb07006d530 0x7fb0700cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb075c40640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb07015e550 0x7fb07009bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb076c42640 1 -- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb07006d530 msgr2=0x7fb0700cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb076c42640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb07006d530 0x7fb0700cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb076c42640 1 -- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb07015e550 msgr2=0x7fb07009bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb076c42640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb07015e550 0x7fb07009bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb076c42640 1 -- 172.21.15.44:0/3740217383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07014d1a0 con 0x7fb07015ee70 2024-04-02T13:28:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.192+0000 7fb076c42640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb07015ee70 0x7fb07014b9b0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fb068009f40 tx=0x7fb068004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.193+0000 7fb066ffd640 1 -- 172.21.15.44:0/3740217383 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb068044020 con 0x7fb07015ee70 2024-04-02T13:28:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.193+0000 7fb066ffd640 1 -- 172.21.15.44:0/3740217383 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb06802ce50 con 0x7fb07015ee70 2024-04-02T13:28:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.193+0000 7fb0786a5640 1 -- 172.21.15.44:0/3740217383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb07014d4b0 con 0x7fb07015ee70 2024-04-02T13:28:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.193+0000 7fb066ffd640 1 -- 172.21.15.44:0/3740217383 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb068009510 con 0x7fb07015ee70 2024-04-02T13:28:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.193+0000 7fb0786a5640 1 -- 172.21.15.44:0/3740217383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb07014dcf0 con 0x7fb07015ee70 2024-04-02T13:28:55.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.194+0000 7fb0786a5640 1 -- 172.21.15.44:0/3740217383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb07014def0 con 0x7fb07015ee70 2024-04-02T13:28:55.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.194+0000 7fb066ffd640 1 -- 172.21.15.44:0/3740217383 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb068047020 con 0x7fb07015ee70 2024-04-02T13:28:55.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.194+0000 7fb066ffd640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb05406c8f0 0x7fb05406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:55.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.195+0000 7fb066ffd640 1 -- 172.21.15.44:0/3740217383 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fb0680ae540 con 0x7fb07015ee70 2024-04-02T13:28:55.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.195+0000 7fb076441640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb05406c8f0 0x7fb05406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:55.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.195+0000 7fb076441640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb05406c8f0 0x7fb05406edb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb06c009a20 tx=0x7fb06c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:55.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.199+0000 7fb066ffd640 1 -- 172.21.15.44:0/3740217383 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb07014def0 con 0x7fb07015ee70 2024-04-02T13:28:55.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.394+0000 7fb0786a5640 1 -- 172.21.15.44:0/3740217383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb07014d690 con 0x7fb07015ee70 2024-04-02T13:28:55.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.394+0000 7fb066ffd640 1 -- 172.21.15.44:0/3740217383 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v95) ==== 80+0+242 (secure 0 0 0) 0x7fb06804e030 con 0x7fb07015ee70 2024-04-02T13:28:55.397 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6833/2250279975 2024-04-03T13:28:15.300517+0000 2024-04-02T13:28:55.397 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6832/2250279975 2024-04-03T13:28:15.300517+0000 2024-04-02T13:28:55.397 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6833/719680934 2024-04-03T13:26:55.589090+0000 2024-04-02T13:28:55.397 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6832/719680934 2024-04-03T13:26:55.589090+0000 2024-04-02T13:28:55.398 INFO:teuthology.orchestra.run.smithi044.stderr:listed 4 entries 2024-04-02T13:28:55.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.396+0000 7fb0786a5640 1 -- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb05406c8f0 msgr2=0x7fb05406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:55.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.396+0000 7fb0786a5640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb05406c8f0 0x7fb05406edb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb06c009a20 tx=0x7fb06c009190 comp rx=0 tx=0).stop 2024-04-02T13:28:55.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.396+0000 7fb0786a5640 1 -- 172.21.15.44:0/3740217383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb07015ee70 msgr2=0x7fb07014b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:55.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.396+0000 7fb0786a5640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb07015ee70 0x7fb07014b9b0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fb068009f40 tx=0x7fb068004a40 comp rx=0 tx=0).stop 2024-04-02T13:28:55.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.396+0000 7fb0786a5640 1 -- 172.21.15.44:0/3740217383 shutdown_connections 2024-04-02T13:28:55.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.396+0000 7fb0786a5640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb05406c8f0 0x7fb05406edb0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.396+0000 7fb0786a5640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb07015ee70 0x7fb07014b9b0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.396+0000 7fb0786a5640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb07015e550 0x7fb07009bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.396+0000 7fb0786a5640 1 --2- 172.21.15.44:0/3740217383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb07006d530 0x7fb0700cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.396+0000 7fb0786a5640 1 -- 172.21.15.44:0/3740217383 >> 172.21.15.44:0/3740217383 conn(0x7fb07009aa20 msgr2=0x7fb07015d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:55.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.396+0000 7fb0786a5640 1 -- 172.21.15.44:0/3740217383 shutdown_connections 2024-04-02T13:28:55.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.396+0000 7fb0786a5640 1 -- 172.21.15.44:0/3740217383 wait complete. 2024-04-02T13:28:55.410 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:28:55.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:55.410 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:28:55.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[73358]: fuse finished with error 0 and tester_r 0 2024-04-02T13:28:55.437 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:28:55.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:55.437 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:28:55.540 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:28:55.541 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:28:55.541 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:28:55.559 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:28:55.560 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:28:55.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:55.560 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-02T13:28:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:28:55.617 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-02T13:28:55.617 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:28:55.617 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:28:55.618 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:28:55.632 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:28:55.632 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:28:55.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:55.633 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-02T13:28:55.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:28:55.687 INFO:teuthology.orchestra.run.smithi044.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-02T13:28:55.687 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:28:55.687 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:28:55.687 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:28:55.704 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:28:55.705 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:28:55.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:28:55.705 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-02T13:28:55.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:28:55.762 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-02T13:28:55.763 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:28:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.862+0000 7fd6ccfc8640 1 Processor -- start 2024-04-02T13:28:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.862+0000 7fd6ccfc8640 1 -- start start 2024-04-02T13:28:55.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.863+0000 7fd6ccfc8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd6c80d0a30 0x7fd6c806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:55.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.863+0000 7fd6ccfc8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd6c80d2040 0x7fd6c80d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:55.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.863+0000 7fd6ccfc8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c80d0000 con 0x7fd6c80d0e10 2024-04-02T13:28:55.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.863+0000 7fd6ccfc8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd6c806cb00 con 0x7fd6c80d0a30 2024-04-02T13:28:55.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.863+0000 7fd6ccfc8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd6c806c1d0 con 0x7fd6c80d2040 2024-04-02T13:28:55.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.863+0000 7fd6c659c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd6c80d0a30 0x7fd6c806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:55.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.863+0000 7fd6c5d9b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd6c80d0e10 0x7fd6c80cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54940/0 (socket says 172.21.15.44:54940) 2024-04-02T13:28:55.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.863+0000 7fd6c5d9b640 1 -- 172.21.15.44:0/3653965491 learned_addr learned my addr 172.21.15.44:0/3653965491 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:55.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.864+0000 7fd6c6d9d640 1 --2- 172.21.15.44:0/3653965491 >> v2:172.21.15.69:3300/0 conn(0x7fd6c80d2040 0x7fd6c80d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:55.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.864+0000 7fd6c659c640 1 -- 172.21.15.44:0/3653965491 >> v2:172.21.15.69:3300/0 conn(0x7fd6c80d2040 msgr2=0x7fd6c80d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:55.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.864+0000 7fd6c659c640 1 --2- 172.21.15.44:0/3653965491 >> v2:172.21.15.69:3300/0 conn(0x7fd6c80d2040 0x7fd6c80d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.864+0000 7fd6c659c640 1 -- 172.21.15.44:0/3653965491 >> v1:172.21.15.44:6789/0 conn(0x7fd6c80d0e10 legacy=0x7fd6c80cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:55.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.864+0000 7fd6c659c640 1 -- 172.21.15.44:0/3653965491 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c806c4b0 con 0x7fd6c80d0a30 2024-04-02T13:28:55.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.864+0000 7fd6c559a640 1 -- 172.21.15.44:0/3653965491 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2368650489 0 0) 0x7fd6c80d0000 con 0x7fd6c80d0e10 2024-04-02T13:28:55.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.864+0000 7fd6c6d9d640 1 --2- 172.21.15.44:0/3653965491 >> v2:172.21.15.69:3300/0 conn(0x7fd6c80d2040 0x7fd6c80d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:55.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.864+0000 7fd6c659c640 1 --2- 172.21.15.44:0/3653965491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6c80d0a30 0x7fd6c806cf90 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fd6c808c000 tx=0x7fd6b802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19916ac6cde3ad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.865+0000 7fd6c559a640 1 -- 172.21.15.44:0/3653965491 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8039070 con 0x7fd6c80d0a30 2024-04-02T13:28:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.865+0000 7fd6c559a640 1 -- 172.21.15.44:0/3653965491 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6b802cbf0 con 0x7fd6c80d0a30 2024-04-02T13:28:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.865+0000 7fd6c559a640 1 -- 172.21.15.44:0/3653965491 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8034440 con 0x7fd6c80d0a30 2024-04-02T13:28:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.865+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/3653965491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6c80d0a30 msgr2=0x7fd6c806cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.865+0000 7fd6ccfc8640 1 --2- 172.21.15.44:0/3653965491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6c80d0a30 0x7fd6c806cf90 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fd6c808c000 tx=0x7fd6b802be40 comp rx=0 tx=0).stop 2024-04-02T13:28:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.865+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/3653965491 shutdown_connections 2024-04-02T13:28:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.865+0000 7fd6ccfc8640 1 --2- 172.21.15.44:0/3653965491 >> v2:172.21.15.69:3300/0 conn(0x7fd6c80d2040 0x7fd6c80d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.865+0000 7fd6ccfc8640 1 --2- 172.21.15.44:0/3653965491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6c80d0a30 0x7fd6c806cf90 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.865+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/3653965491 >> 172.21.15.44:0/3653965491 conn(0x7fd6c809d470 msgr2=0x7fd6c80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.865+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/3653965491 shutdown_connections 2024-04-02T13:28:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.865+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/3653965491 wait complete. 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.867+0000 7fd6ccfc8640 1 Processor -- start 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.867+0000 7fd6ccfc8640 1 -- start start 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.867+0000 7fd6ccfc8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6c80d0a30 0x7fd6c80b4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.867+0000 7fd6ccfc8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6c80d0e10 0x7fd6c81e99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.867+0000 7fd6ccfc8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6c80d2040 0x7fd6c80cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.867+0000 7fd6ccfc8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c815e6f0 con 0x7fd6c80d0e10 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.867+0000 7fd6ccfc8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c80d1270 con 0x7fd6c80d0a30 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.867+0000 7fd6ccfc8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd6c815e870 con 0x7fd6c80d2040 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.867+0000 7fd6c5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6c80d0e10 0x7fd6c81e99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.867+0000 7fd6c5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6c80d0e10 0x7fd6c81e99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38544/0 (socket says 172.21.15.44:38544) 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.867+0000 7fd6c5d9b640 1 -- 172.21.15.44:0/154071659 learned_addr learned my addr 172.21.15.44:0/154071659 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6c659c640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6c80d0a30 0x7fd6c80b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6c6d9d640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6c80d2040 0x7fd6c80cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6c5d9b640 1 -- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6c80d2040 msgr2=0x7fd6c80cb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6c5d9b640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6c80d2040 0x7fd6c80cb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6c5d9b640 1 -- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6c80d0a30 msgr2=0x7fd6c80b4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6c5d9b640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6c80d0a30 0x7fd6c80b4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6c5d9b640 1 -- 172.21.15.44:0/154071659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c80b4770 con 0x7fd6c80d0e10 2024-04-02T13:28:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6c659c640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6c80d0a30 0x7fd6c80b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6c5d9b640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6c80d0e10 0x7fd6c81e99a0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000c440 tx=0x7fd6b000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6beffd640 1 -- 172.21.15.44:0/154071659 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0018070 con 0x7fd6c80d0e10 2024-04-02T13:28:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6beffd640 1 -- 172.21.15.44:0/154071659 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6b00074e0 con 0x7fd6c80d0e10 2024-04-02T13:28:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6beffd640 1 -- 172.21.15.44:0/154071659 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0004a20 con 0x7fd6c80d0e10 2024-04-02T13:28:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/154071659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c80b4a50 con 0x7fd6c80d0e10 2024-04-02T13:28:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.868+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/154071659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c80b5010 con 0x7fd6c80d0e10 2024-04-02T13:28:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.869+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/154071659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6c81e9f40 con 0x7fd6c80d0e10 2024-04-02T13:28:55.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.870+0000 7fd6beffd640 1 -- 172.21.15.44:0/154071659 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd6b001b070 con 0x7fd6c80d0e10 2024-04-02T13:28:55.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.870+0000 7fd6beffd640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd6a806c920 0x7fd6a806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:55.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.870+0000 7fd6beffd640 1 -- 172.21.15.44:0/154071659 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fd6b008b100 con 0x7fd6c80d0e10 2024-04-02T13:28:55.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.870+0000 7fd6c659c640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd6a806c920 0x7fd6a806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:55.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.871+0000 7fd6c659c640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd6a806c920 0x7fd6a806ede0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8036000 tx=0x7fd6b8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:55.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:55.874+0000 7fd6beffd640 1 -- 172.21.15.44:0/154071659 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd6c81e9f40 con 0x7fd6c80d0e10 2024-04-02T13:28:56.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.101+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/154071659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6c80684f0 con 0x7fd6c80d0e10 2024-04-02T13:28:56.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.101+0000 7fd6beffd640 1 -- 172.21.15.44:0/154071659 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5181 (secure 0 0 0) 0x7fd6b00591f0 con 0x7fd6c80d0e10 2024-04-02T13:28:56.104 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:56.104 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:28:19.321536+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8403":{"gid":8403,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6833/3044289683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":3044289683},{"type":"v1","addr":"172.21.15.44:6833","nonce":3044289683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8403,"qdb_cluster":[8403]},"id":6}]} 2024-04-02T13:28:56.104 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 74 2024-04-02T13:28:56.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.103+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd6a806c920 msgr2=0x7fd6a806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.103+0000 7fd6ccfc8640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd6a806c920 0x7fd6a806ede0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8036000 tx=0x7fd6b8030000 comp rx=0 tx=0).stop 2024-04-02T13:28:56.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.103+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/154071659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6c80d0e10 msgr2=0x7fd6c81e99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.103+0000 7fd6ccfc8640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6c80d0e10 0x7fd6c81e99a0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000c440 tx=0x7fd6b000c900 comp rx=0 tx=0).stop 2024-04-02T13:28:56.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.104+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/154071659 shutdown_connections 2024-04-02T13:28:56.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.104+0000 7fd6ccfc8640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd6a806c920 0x7fd6a806ede0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.104+0000 7fd6ccfc8640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6c80d2040 0x7fd6c80cb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.104+0000 7fd6ccfc8640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6c80d0e10 0x7fd6c81e99a0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.104+0000 7fd6ccfc8640 1 --2- 172.21.15.44:0/154071659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6c80d0a30 0x7fd6c80b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.104+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/154071659 >> 172.21.15.44:0/154071659 conn(0x7fd6c809d470 msgr2=0x7fd6c815ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:56.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.104+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/154071659 shutdown_connections 2024-04-02T13:28:56.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.104+0000 7fd6ccfc8640 1 -- 172.21.15.44:0/154071659 wait complete. 2024-04-02T13:28:56.117 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:28:56.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.216+0000 7faf7031c640 1 Processor -- start 2024-04-02T13:28:56.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.216+0000 7faf7031c640 1 -- start start 2024-04-02T13:28:56.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf7031c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faf6815e4a0 0x7faf680aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf7031c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7faf6806d550 0x7faf680d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf7031c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf680cc600 con 0x7faf680d0c50 2024-04-02T13:28:56.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf7031c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7faf680cc8e0 con 0x7faf6806d550 2024-04-02T13:28:56.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf7031c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7faf680ccbc0 con 0x7faf6815e4a0 2024-04-02T13:28:56.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf6d8b7640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7faf680d0c50 0x7faf6806ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54952/0 (socket says 172.21.15.44:54952) 2024-04-02T13:28:56.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf6e8b9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7faf6806d550 0x7faf680d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf6d8b7640 1 -- 172.21.15.44:0/3679022029 learned_addr learned my addr 172.21.15.44:0/3679022029 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:56.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf6e0b8640 1 --2- 172.21.15.44:0/3679022029 >> v2:172.21.15.69:3300/0 conn(0x7faf6815e4a0 0x7faf680aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf6e8b9640 1 -- 172.21.15.44:0/3679022029 >> v2:172.21.15.69:3300/0 conn(0x7faf6815e4a0 msgr2=0x7faf680aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:56.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf6e8b9640 1 --2- 172.21.15.44:0/3679022029 >> v2:172.21.15.69:3300/0 conn(0x7faf6815e4a0 0x7faf680aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf6e8b9640 1 -- 172.21.15.44:0/3679022029 >> v1:172.21.15.44:6789/0 conn(0x7faf680d0c50 legacy=0x7faf6806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf6e8b9640 1 -- 172.21.15.44:0/3679022029 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf680ccea0 con 0x7faf6806d550 2024-04-02T13:28:56.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.217+0000 7faf6d0b6640 1 -- 172.21.15.44:0/3679022029 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 937210796 0 0) 0x7faf680cc600 con 0x7faf680d0c50 2024-04-02T13:28:56.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.218+0000 7faf6e8b9640 1 --2- 172.21.15.44:0/3679022029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf6806d550 0x7faf680d10b0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7faf60009d90 tx=0x7faf6002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da668ed99ce39fe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:56.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.218+0000 7faf6d0b6640 1 -- 172.21.15.44:0/3679022029 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf60039070 con 0x7faf6806d550 2024-04-02T13:28:56.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.218+0000 7faf6d0b6640 1 -- 172.21.15.44:0/3679022029 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf6002cc60 con 0x7faf6806d550 2024-04-02T13:28:56.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.218+0000 7faf6d0b6640 1 -- 172.21.15.44:0/3679022029 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf60034660 con 0x7faf6806d550 2024-04-02T13:28:56.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.218+0000 7faf7031c640 1 -- 172.21.15.44:0/3679022029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf6806d550 msgr2=0x7faf680d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.218+0000 7faf7031c640 1 --2- 172.21.15.44:0/3679022029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf6806d550 0x7faf680d10b0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7faf60009d90 tx=0x7faf6002beb0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.219+0000 7faf7031c640 1 -- 172.21.15.44:0/3679022029 shutdown_connections 2024-04-02T13:28:56.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.219+0000 7faf7031c640 1 --2- 172.21.15.44:0/3679022029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf6806d550 0x7faf680d10b0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.219+0000 7faf7031c640 1 --2- 172.21.15.44:0/3679022029 >> v2:172.21.15.69:3300/0 conn(0x7faf6815e4a0 0x7faf680aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.219+0000 7faf7031c640 1 -- 172.21.15.44:0/3679022029 >> 172.21.15.44:0/3679022029 conn(0x7faf6809aa20 msgr2=0x7faf6809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:56.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.219+0000 7faf7031c640 1 -- 172.21.15.44:0/3679022029 shutdown_connections 2024-04-02T13:28:56.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.219+0000 7faf7031c640 1 -- 172.21.15.44:0/3679022029 wait complete. 2024-04-02T13:28:56.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.220+0000 7faf7031c640 1 Processor -- start 2024-04-02T13:28:56.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.220+0000 7faf7031c640 1 -- start start 2024-04-02T13:28:56.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.220+0000 7faf7031c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf6806d550 0x7faf680c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.220+0000 7faf7031c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf680d0c50 0x7faf680c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.220+0000 7faf7031c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf6815e4a0 0x7faf68157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.220+0000 7faf7031c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faf6815e880 con 0x7faf6806d550 2024-04-02T13:28:56.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.220+0000 7faf7031c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faf680cd460 con 0x7faf680d0c50 2024-04-02T13:28:56.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.220+0000 7faf7031c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faf680af130 con 0x7faf6815e4a0 2024-04-02T13:28:56.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf6e0b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf6806d550 0x7faf680c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf6e0b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf6806d550 0x7faf680c8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38558/0 (socket says 172.21.15.44:38558) 2024-04-02T13:28:56.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf6e0b8640 1 -- 172.21.15.44:0/2782994123 learned_addr learned my addr 172.21.15.44:0/2782994123 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:56.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf6d8b7640 1 --2- 172.21.15.44:0/2782994123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf680d0c50 0x7faf680c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf6e0b8640 1 -- 172.21.15.44:0/2782994123 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf6815e4a0 msgr2=0x7faf68157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf6e0b8640 1 --2- 172.21.15.44:0/2782994123 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf6815e4a0 0x7faf68157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf6e0b8640 1 -- 172.21.15.44:0/2782994123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf680d0c50 msgr2=0x7faf680c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf6e0b8640 1 --2- 172.21.15.44:0/2782994123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf680d0c50 0x7faf680c7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf6e0b8640 1 -- 172.21.15.44:0/2782994123 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf680b51e0 con 0x7faf6806d550 2024-04-02T13:28:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf6e0b8640 1 --2- 172.21.15.44:0/2782994123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf6806d550 0x7faf680c8510 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7faf6400e870 tx=0x7faf6400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf5effd640 1 -- 172.21.15.44:0/2782994123 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf64017070 con 0x7faf6806d550 2024-04-02T13:28:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf5effd640 1 -- 172.21.15.44:0/2782994123 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf6400c940 con 0x7faf6806d550 2024-04-02T13:28:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf5effd640 1 -- 172.21.15.44:0/2782994123 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6400cd30 con 0x7faf6806d550 2024-04-02T13:28:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf7031c640 1 -- 172.21.15.44:0/2782994123 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf680b54c0 con 0x7faf6806d550 2024-04-02T13:28:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.221+0000 7faf7031c640 1 -- 172.21.15.44:0/2782994123 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf680b5fa0 con 0x7faf6806d550 2024-04-02T13:28:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.222+0000 7faf7031c640 1 -- 172.21.15.44:0/2782994123 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf680684f0 con 0x7faf6806d550 2024-04-02T13:28:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.223+0000 7faf5effd640 1 -- 172.21.15.44:0/2782994123 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faf6401a070 con 0x7faf6806d550 2024-04-02T13:28:56.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.223+0000 7faf5effd640 1 --2- 172.21.15.44:0/2782994123 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf4c06c8f0 0x7faf4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.223+0000 7faf5effd640 1 -- 172.21.15.44:0/2782994123 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7faf6408bcd0 con 0x7faf6806d550 2024-04-02T13:28:56.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.223+0000 7faf6d8b7640 1 --2- 172.21.15.44:0/2782994123 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf4c06c8f0 0x7faf4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.224+0000 7faf6d8b7640 1 --2- 172.21.15.44:0/2782994123 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf4c06c8f0 0x7faf4c06edb0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7faf58007fd0 tx=0x7faf58009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:56.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.227+0000 7faf5effd640 1 -- 172.21.15.44:0/2782994123 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7faf64059e50 con 0x7faf6806d550 2024-04-02T13:28:56.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.413+0000 7faf7031c640 1 -- 172.21.15.44:0/2782994123 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf680c77c0 con 0x7faf6806d550 2024-04-02T13:28:56.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.414+0000 7faf5effd640 1 -- 172.21.15.44:0/2782994123 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+17288 (secure 0 0 0) 0x7faf64059750 con 0x7faf6806d550 2024-04-02T13:28:56.417 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:56.417 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":95,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:28:15.300549+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:26:41.300970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-02T13:26:41.979586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.24","mappings":[{"from":1,"to":5}]},{"pgid":"14.2c","mappings":[{"from":1,"to":2}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6833/2250279975":"2024-04-03T13:28:15.300517+0000","172.21.15.44:6832/2250279975":"2024-04-03T13:28:15.300517+0000","172.21.15.44:6833/719680934":"2024-04-03T13:26:55.589090+0000","172.21.15.44:6832/719680934":"2024-04-03T13:26:55.589090+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:28:56.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.416+0000 7faf7031c640 1 -- 172.21.15.44:0/2782994123 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf4c06c8f0 msgr2=0x7faf4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.416+0000 7faf7031c640 1 --2- 172.21.15.44:0/2782994123 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf4c06c8f0 0x7faf4c06edb0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7faf58007fd0 tx=0x7faf58009040 comp rx=0 tx=0).stop 2024-04-02T13:28:56.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.416+0000 7faf7031c640 1 -- 172.21.15.44:0/2782994123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf6806d550 msgr2=0x7faf680c8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.416+0000 7faf7031c640 1 --2- 172.21.15.44:0/2782994123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf6806d550 0x7faf680c8510 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7faf6400e870 tx=0x7faf6400ed30 comp rx=0 tx=0).stop 2024-04-02T13:28:56.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.416+0000 7faf7031c640 1 -- 172.21.15.44:0/2782994123 shutdown_connections 2024-04-02T13:28:56.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.416+0000 7faf7031c640 1 --2- 172.21.15.44:0/2782994123 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faf4c06c8f0 0x7faf4c06edb0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.416+0000 7faf7031c640 1 --2- 172.21.15.44:0/2782994123 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf6815e4a0 0x7faf68157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.416+0000 7faf7031c640 1 --2- 172.21.15.44:0/2782994123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf680d0c50 0x7faf680c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.416+0000 7faf7031c640 1 --2- 172.21.15.44:0/2782994123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf6806d550 0x7faf680c8510 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.416+0000 7faf7031c640 1 -- 172.21.15.44:0/2782994123 >> 172.21.15.44:0/2782994123 conn(0x7faf6809aa20 msgr2=0x7faf6815d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:56.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.416+0000 7faf7031c640 1 -- 172.21.15.44:0/2782994123 shutdown_connections 2024-04-02T13:28:56.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.416+0000 7faf7031c640 1 -- 172.21.15.44:0/2782994123 wait complete. 2024-04-02T13:28:56.430 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:26:41.300970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:26:41.979586+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:28:56.430 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:28:56.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.532+0000 7f929dd52640 1 Processor -- start 2024-04-02T13:28:56.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.532+0000 7f929dd52640 1 -- start start 2024-04-02T13:28:56.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.533+0000 7f929dd52640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f929815e1d0 0x7f92980d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.533+0000 7f929dd52640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f92980cec60 0x7f92980d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.533+0000 7f929dd52640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92980d1640 con 0x7f929815e5b0 2024-04-02T13:28:56.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.533+0000 7f929dd52640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f92980cfe00 con 0x7f929815e1d0 2024-04-02T13:28:56.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.533+0000 7f929dd52640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f92980cf1f0 con 0x7f92980cec60 2024-04-02T13:28:56.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.533+0000 7f92977fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f929815e1d0 0x7f92980d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.533+0000 7f9297fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f929815e5b0 0x7f92980b4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54968/0 (socket says 172.21.15.44:54968) 2024-04-02T13:28:56.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.533+0000 7f9297fff640 1 -- 172.21.15.44:0/3164601748 learned_addr learned my addr 172.21.15.44:0/3164601748 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:56.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.534+0000 7f9296ffd640 1 --2- 172.21.15.44:0/3164601748 >> v2:172.21.15.69:3300/0 conn(0x7f92980cec60 0x7f92980d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.534+0000 7f92977fe640 1 -- 172.21.15.44:0/3164601748 >> v2:172.21.15.69:3300/0 conn(0x7f92980cec60 msgr2=0x7f92980d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:56.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.534+0000 7f92977fe640 1 --2- 172.21.15.44:0/3164601748 >> v2:172.21.15.69:3300/0 conn(0x7f92980cec60 0x7f92980d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.534+0000 7f92977fe640 1 -- 172.21.15.44:0/3164601748 >> v1:172.21.15.44:6789/0 conn(0x7f929815e5b0 legacy=0x7f92980b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.534+0000 7f92967fc640 1 -- 172.21.15.44:0/3164601748 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 122447240 0 0) 0x7f92980d1640 con 0x7f929815e5b0 2024-04-02T13:28:56.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.534+0000 7f92977fe640 1 -- 172.21.15.44:0/3164601748 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92980cf4d0 con 0x7f929815e1d0 2024-04-02T13:28:56.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.534+0000 7f92977fe640 1 --2- 172.21.15.44:0/3164601748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929815e1d0 0x7f92980d1100 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f9298161f20 tx=0x7f928802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8fd8ed75966adcd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:56.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.535+0000 7f92967fc640 1 -- 172.21.15.44:0/3164601748 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92880315a0 con 0x7f929815e1d0 2024-04-02T13:28:56.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.535+0000 7f92967fc640 1 -- 172.21.15.44:0/3164601748 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9288037070 con 0x7f929815e1d0 2024-04-02T13:28:56.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.535+0000 7f92967fc640 1 -- 172.21.15.44:0/3164601748 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288031960 con 0x7f929815e1d0 2024-04-02T13:28:56.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.535+0000 7f929dd52640 1 -- 172.21.15.44:0/3164601748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929815e1d0 msgr2=0x7f92980d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.535+0000 7f929dd52640 1 --2- 172.21.15.44:0/3164601748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929815e1d0 0x7f92980d1100 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f9298161f20 tx=0x7f928802b840 comp rx=0 tx=0).stop 2024-04-02T13:28:56.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.535+0000 7f929dd52640 1 -- 172.21.15.44:0/3164601748 shutdown_connections 2024-04-02T13:28:56.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.535+0000 7f929dd52640 1 --2- 172.21.15.44:0/3164601748 >> v2:172.21.15.69:3300/0 conn(0x7f92980cec60 0x7f92980d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.535+0000 7f929dd52640 1 --2- 172.21.15.44:0/3164601748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929815e1d0 0x7f92980d1100 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.535+0000 7f929dd52640 1 -- 172.21.15.44:0/3164601748 >> 172.21.15.44:0/3164601748 conn(0x7f929809d3b0 msgr2=0x7f92980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:56.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.535+0000 7f929dd52640 1 -- 172.21.15.44:0/3164601748 shutdown_connections 2024-04-02T13:28:56.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.535+0000 7f929dd52640 1 -- 172.21.15.44:0/3164601748 wait complete. 2024-04-02T13:28:56.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.536+0000 7f929dd52640 1 Processor -- start 2024-04-02T13:28:56.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.536+0000 7f929dd52640 1 -- start start 2024-04-02T13:28:56.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.537+0000 7f929dd52640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92980cec60 0x7f92980a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.537+0000 7f929dd52640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f929815e1d0 0x7f929809fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.537+0000 7f929dd52640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929815e5b0 0x7f929814e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.537+0000 7f929dd52640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f92980d2c00 con 0x7f929815e5b0 2024-04-02T13:28:56.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.537+0000 7f929dd52640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f92980cfa90 con 0x7f92980cec60 2024-04-02T13:28:56.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.537+0000 7f929dd52640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f92980d2d80 con 0x7f929815e1d0 2024-04-02T13:28:56.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.537+0000 7f9297fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929815e5b0 0x7f929814e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.537+0000 7f9297fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929815e5b0 0x7f929814e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38578/0 (socket says 172.21.15.44:38578) 2024-04-02T13:28:56.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.537+0000 7f9297fff640 1 -- 172.21.15.44:0/2468939152 learned_addr learned my addr 172.21.15.44:0/2468939152 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:56.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.537+0000 7f9296ffd640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f929815e1d0 0x7f929809fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.537+0000 7f92977fe640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92980cec60 0x7f92980a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.538+0000 7f9297fff640 1 -- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f929815e1d0 msgr2=0x7f929809fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.538+0000 7f9297fff640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f929815e1d0 0x7f929809fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.538+0000 7f9297fff640 1 -- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92980cec60 msgr2=0x7f92980a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.538+0000 7f9297fff640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92980cec60 0x7f92980a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.538+0000 7f9297fff640 1 -- 172.21.15.44:0/2468939152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929809e860 con 0x7f929815e5b0 2024-04-02T13:28:56.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.538+0000 7f92977fe640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92980cec60 0x7f92980a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:56.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.538+0000 7f9297fff640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929815e5b0 0x7f929814e110 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f929000c700 tx=0x7f929000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:56.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.538+0000 7f927ffff640 1 -- 172.21.15.44:0/2468939152 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290018070 con 0x7f929815e5b0 2024-04-02T13:28:56.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.538+0000 7f927ffff640 1 -- 172.21.15.44:0/2468939152 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f929000ce10 con 0x7f929815e5b0 2024-04-02T13:28:56.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.538+0000 7f929dd52640 1 -- 172.21.15.44:0/2468939152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929809eb40 con 0x7f929815e5b0 2024-04-02T13:28:56.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.538+0000 7f927ffff640 1 -- 172.21.15.44:0/2468939152 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929000a860 con 0x7f929815e5b0 2024-04-02T13:28:56.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.538+0000 7f929dd52640 1 -- 172.21.15.44:0/2468939152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f929809f100 con 0x7f929815e5b0 2024-04-02T13:28:56.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.539+0000 7f929dd52640 1 -- 172.21.15.44:0/2468939152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9298072fd0 con 0x7f929815e5b0 2024-04-02T13:28:56.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.539+0000 7f927ffff640 1 -- 172.21.15.44:0/2468939152 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f929001b070 con 0x7f929815e5b0 2024-04-02T13:28:56.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.540+0000 7f927ffff640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f928406c8f0 0x7f928406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.540+0000 7f927ffff640 1 -- 172.21.15.44:0/2468939152 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f929008b3e0 con 0x7f929815e5b0 2024-04-02T13:28:56.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.540+0000 7f92977fe640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f928406c8f0 0x7f928406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.541+0000 7f92977fe640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f928406c8f0 0x7f928406edb0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9288036000 tx=0x7f928803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:56.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.544+0000 7f927ffff640 1 -- 172.21.15.44:0/2468939152 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9298072fd0 con 0x7f929815e5b0 2024-04-02T13:28:56.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.726+0000 7f929dd52640 1 -- 172.21.15.44:0/2468939152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f92980684f0 con 0x7f929815e5b0 2024-04-02T13:28:56.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.727+0000 7f927ffff640 1 -- 172.21.15.44:0/2468939152 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f9290059550 con 0x7f929815e5b0 2024-04-02T13:28:56.730 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:28:56.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.729+0000 7f929dd52640 1 -- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f928406c8f0 msgr2=0x7f928406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.729+0000 7f929dd52640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f928406c8f0 0x7f928406edb0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9288036000 tx=0x7f928803d040 comp rx=0 tx=0).stop 2024-04-02T13:28:56.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.729+0000 7f929dd52640 1 -- 172.21.15.44:0/2468939152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929815e5b0 msgr2=0x7f929814e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.729+0000 7f929dd52640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929815e5b0 0x7f929814e110 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f929000c700 tx=0x7f929000cbc0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.729+0000 7f929dd52640 1 -- 172.21.15.44:0/2468939152 shutdown_connections 2024-04-02T13:28:56.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.729+0000 7f929dd52640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f928406c8f0 0x7f928406edb0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.729+0000 7f929dd52640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929815e5b0 0x7f929814e110 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.729+0000 7f929dd52640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f929815e1d0 0x7f929809fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.729+0000 7f929dd52640 1 --2- 172.21.15.44:0/2468939152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92980cec60 0x7f92980a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.729+0000 7f929dd52640 1 -- 172.21.15.44:0/2468939152 >> 172.21.15.44:0/2468939152 conn(0x7f929809d3b0 msgr2=0x7f929806c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:56.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.729+0000 7f929dd52640 1 -- 172.21.15.44:0/2468939152 shutdown_connections 2024-04-02T13:28:56.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.729+0000 7f929dd52640 1 -- 172.21.15.44:0/2468939152 wait complete. 2024-04-02T13:28:56.742 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:28:56.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.843+0000 7fab77f1c640 1 Processor -- start 2024-04-02T13:28:56.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.843+0000 7fab77f1c640 1 -- start start 2024-04-02T13:28:56.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.844+0000 7fab77f1c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fab700b1440 0x7fab700b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.844+0000 7fab77f1c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fab700d03d0 0x7fab700d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.844+0000 7fab77f1c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab700cca50 con 0x7fab7015c220 2024-04-02T13:28:56.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.844+0000 7fab77f1c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fab700ccd60 con 0x7fab700b1440 2024-04-02T13:28:56.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.844+0000 7fab77f1c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fab700cef40 con 0x7fab700d03d0 2024-04-02T13:28:56.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.844+0000 7fab754b7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fab700b1440 0x7fab700b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.844+0000 7fab754b7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fab700b1440 0x7fab700b1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38580/0 (socket says 172.21.15.44:38580) 2024-04-02T13:28:56.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.844+0000 7fab754b7640 1 -- 172.21.15.44:0/2789722452 learned_addr learned my addr 172.21.15.44:0/2789722452 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:56.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.844+0000 7fab764b9640 1 --2- 172.21.15.44:0/2789722452 >> v2:172.21.15.69:3300/0 conn(0x7fab700d03d0 0x7fab700d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.845+0000 7fab754b7640 1 -- 172.21.15.44:0/2789722452 >> v2:172.21.15.69:3300/0 conn(0x7fab700d03d0 msgr2=0x7fab700d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:56.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.845+0000 7fab754b7640 1 --2- 172.21.15.44:0/2789722452 >> v2:172.21.15.69:3300/0 conn(0x7fab700d03d0 0x7fab700d0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.845+0000 7fab754b7640 1 -- 172.21.15.44:0/2789722452 >> v1:172.21.15.69:6789/0 conn(0x7fab7015c220 legacy=0x7fab700ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.845+0000 7fab754b7640 1 -- 172.21.15.44:0/2789722452 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab700cf220 con 0x7fab700b1440 2024-04-02T13:28:56.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.845+0000 7fab754b7640 1 --2- 172.21.15.44:0/2789722452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab700b1440 0x7fab700b1db0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fab7009b5d0 tx=0x7fab6402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a9e49f3b1ac51e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:56.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.846+0000 7fab74cb6640 1 -- 172.21.15.44:0/2789722452 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab64034070 con 0x7fab700b1440 2024-04-02T13:28:56.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.846+0000 7fab74cb6640 1 -- 172.21.15.44:0/2789722452 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab64030860 con 0x7fab700b1440 2024-04-02T13:28:56.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.846+0000 7fab74cb6640 1 -- 172.21.15.44:0/2789722452 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab64030c30 con 0x7fab700b1440 2024-04-02T13:28:56.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.846+0000 7fab77f1c640 1 -- 172.21.15.44:0/2789722452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab700b1440 msgr2=0x7fab700b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.846+0000 7fab77f1c640 1 --2- 172.21.15.44:0/2789722452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab700b1440 0x7fab700b1db0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fab7009b5d0 tx=0x7fab6402b9a0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.846+0000 7fab77f1c640 1 -- 172.21.15.44:0/2789722452 shutdown_connections 2024-04-02T13:28:56.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.846+0000 7fab77f1c640 1 --2- 172.21.15.44:0/2789722452 >> v2:172.21.15.69:3300/0 conn(0x7fab700d03d0 0x7fab700d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.846+0000 7fab77f1c640 1 --2- 172.21.15.44:0/2789722452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab700b1440 0x7fab700b1db0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.846+0000 7fab77f1c640 1 -- 172.21.15.44:0/2789722452 >> 172.21.15.44:0/2789722452 conn(0x7fab7009aa20 msgr2=0x7fab7009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:56.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.846+0000 7fab77f1c640 1 -- 172.21.15.44:0/2789722452 shutdown_connections 2024-04-02T13:28:56.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.846+0000 7fab77f1c640 1 -- 172.21.15.44:0/2789722452 wait complete. 2024-04-02T13:28:56.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.847+0000 7fab77f1c640 1 Processor -- start 2024-04-02T13:28:56.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.847+0000 7fab77f1c640 1 -- start start 2024-04-02T13:28:56.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab77f1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab700b1440 0x7fab700c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab77f1c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab700d03d0 0x7fab700c7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab77f1c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab7015c220 0x7fab700afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab77f1c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fab700cd1f0 con 0x7fab7015c220 2024-04-02T13:28:56.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab77f1c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab7015c600 con 0x7fab700d03d0 2024-04-02T13:28:56.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab77f1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fab700cd370 con 0x7fab700b1440 2024-04-02T13:28:56.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab764b9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab7015c220 0x7fab700afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab764b9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab7015c220 0x7fab700afe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38582/0 (socket says 172.21.15.44:38582) 2024-04-02T13:28:56.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab764b9640 1 -- 172.21.15.44:0/1812687267 learned_addr learned my addr 172.21.15.44:0/1812687267 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:56.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab75cb8640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab700b1440 0x7fab700c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab754b7640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab700d03d0 0x7fab700c7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab764b9640 1 -- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab700b1440 msgr2=0x7fab700c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab764b9640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab700b1440 0x7fab700c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab764b9640 1 -- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab700d03d0 msgr2=0x7fab700c7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:56.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab764b9640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab700d03d0 0x7fab700c7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:56.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.848+0000 7fab764b9640 1 -- 172.21.15.44:0/1812687267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab700b0be0 con 0x7fab7015c220 2024-04-02T13:28:56.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.849+0000 7fab764b9640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab7015c220 0x7fab700afe80 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fab6000efc0 tx=0x7fab6000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:56.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.849+0000 7fab6e7fc640 1 -- 172.21.15.44:0/1812687267 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60018020 con 0x7fab7015c220 2024-04-02T13:28:56.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.849+0000 7fab6e7fc640 1 -- 172.21.15.44:0/1812687267 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab60014480 con 0x7fab7015c220 2024-04-02T13:28:56.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.849+0000 7fab77f1c640 1 -- 172.21.15.44:0/1812687267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab700b0ec0 con 0x7fab7015c220 2024-04-02T13:28:56.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.849+0000 7fab6e7fc640 1 -- 172.21.15.44:0/1812687267 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60014870 con 0x7fab7015c220 2024-04-02T13:28:56.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.849+0000 7fab77f1c640 1 -- 172.21.15.44:0/1812687267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab700c94f0 con 0x7fab7015c220 2024-04-02T13:28:56.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.850+0000 7fab77f1c640 1 -- 172.21.15.44:0/1812687267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab700684f0 con 0x7fab7015c220 2024-04-02T13:28:56.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.850+0000 7fab6e7fc640 1 -- 172.21.15.44:0/1812687267 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fab60005d10 con 0x7fab7015c220 2024-04-02T13:28:56.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.851+0000 7fab6e7fc640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fab5c06c8f0 0x7fab5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:56.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.851+0000 7fab6e7fc640 1 -- 172.21.15.44:0/1812687267 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fab6008cac0 con 0x7fab7015c220 2024-04-02T13:28:56.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.851+0000 7fab75cb8640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fab5c06c8f0 0x7fab5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:56.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.851+0000 7fab75cb8640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fab5c06c8f0 0x7fab5c06edb0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fab700aed60 tx=0x7fab68009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:56.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:56.855+0000 7fab6e7fc640 1 -- 172.21.15.44:0/1812687267 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fab60022030 con 0x7fab7015c220 2024-04-02T13:28:57.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.039+0000 7fab77f1c640 1 -- 172.21.15.44:0/1812687267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fab700b1090 con 0x7fab7015c220 2024-04-02T13:28:57.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.039+0000 7fab6e7fc640 1 -- 172.21.15.44:0/1812687267 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7fab60010040 con 0x7fab7015c220 2024-04-02T13:28:57.042 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:28:57.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.041+0000 7fab77f1c640 1 -- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fab5c06c8f0 msgr2=0x7fab5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.041+0000 7fab77f1c640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fab5c06c8f0 0x7fab5c06edb0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fab700aed60 tx=0x7fab68009040 comp rx=0 tx=0).stop 2024-04-02T13:28:57.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.041+0000 7fab77f1c640 1 -- 172.21.15.44:0/1812687267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab7015c220 msgr2=0x7fab700afe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.041+0000 7fab77f1c640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab7015c220 0x7fab700afe80 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fab6000efc0 tx=0x7fab6000c490 comp rx=0 tx=0).stop 2024-04-02T13:28:57.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.041+0000 7fab77f1c640 1 -- 172.21.15.44:0/1812687267 shutdown_connections 2024-04-02T13:28:57.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.041+0000 7fab77f1c640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fab5c06c8f0 0x7fab5c06edb0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.041+0000 7fab77f1c640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab7015c220 0x7fab700afe80 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.041+0000 7fab77f1c640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab700d03d0 0x7fab700c7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.041+0000 7fab77f1c640 1 --2- 172.21.15.44:0/1812687267 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab700b1440 0x7fab700c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.041+0000 7fab77f1c640 1 -- 172.21.15.44:0/1812687267 >> 172.21.15.44:0/1812687267 conn(0x7fab7009aa20 msgr2=0x7fab7006f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:57.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.041+0000 7fab77f1c640 1 -- 172.21.15.44:0/1812687267 shutdown_connections 2024-04-02T13:28:57.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.041+0000 7fab77f1c640 1 -- 172.21.15.44:0/1812687267 wait complete. 2024-04-02T13:28:57.055 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:28:57.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.157+0000 7fed229cd640 1 Processor -- start 2024-04-02T13:28:57.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.157+0000 7fed229cd640 1 -- start start 2024-04-02T13:28:57.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed229cd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fed1c15e5d0 0x7fed1c0d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed229cd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed1c0b4180 0x7fed1c0b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed229cd640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed1c0d1630 con 0x7fed1c06cff0 2024-04-02T13:28:57.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed229cd640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fed1c0d1bf0 con 0x7fed1c15e5d0 2024-04-02T13:28:57.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed229cd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fed1c0d1910 con 0x7fed1c0b4180 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed1bfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fed1c06cff0 0x7fed1c06d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54970/0 (socket says 172.21.15.44:54970) 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed1bfff640 1 -- 172.21.15.44:0/779795521 learned_addr learned my addr 172.21.15.44:0/779795521 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed20f6a640 1 --2- 172.21.15.44:0/779795521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed1c0b4180 0x7fed1c0b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed1b7fe640 1 --2- 172.21.15.44:0/779795521 >> v2:172.21.15.69:3300/0 conn(0x7fed1c15e5d0 0x7fed1c0d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4007867296 0 0) 0x7fed1c0d1630 con 0x7fed1c06cff0 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed08003520 con 0x7fed1c06cff0 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1589446421 0 0) 0x7fed08003520 con 0x7fed1c06cff0 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed1c0b4180 msgr2=0x7fed1c0b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed1affd640 1 --2- 172.21.15.44:0/779795521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed1c0b4180 0x7fed1c0b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 >> v2:172.21.15.69:3300/0 conn(0x7fed1c15e5d0 msgr2=0x7fed1c0d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed1affd640 1 --2- 172.21.15.44:0/779795521 >> v2:172.21.15.69:3300/0 conn(0x7fed1c15e5d0 0x7fed1c0d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1c0b1540 con 0x7fed1c06cff0 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fed0c001670 con 0x7fed1c06cff0 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.158+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 >> v1:172.21.15.44:6789/0 conn(0x7fed1c06cff0 legacy=0x7fed1c06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed1affd640 1 --2- 172.21.15.44:0/779795521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed1c0b4180 0x7fed080039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed1affd640 1 --2- 172.21.15.44:0/779795521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed08003f30 0x7fed08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed1affd640 1 --2- 172.21.15.44:0/779795521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed08004d40 0x7fed080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c0d1bf0 con 0x7fed08003f30 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c0d1910 con 0x7fed08004d40 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fed0800ba10 con 0x7fed1c0b4180 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed1b7fe640 1 --2- 172.21.15.44:0/779795521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed08003f30 0x7fed08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed1bfff640 1 --2- 172.21.15.44:0/779795521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed1c0b4180 0x7fed080039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed20f6a640 1 --2- 172.21.15.44:0/779795521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed08004d40 0x7fed080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed1b7fe640 1 -- 172.21.15.44:0/779795521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed1c0b4180 msgr2=0x7fed080039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed1b7fe640 1 --2- 172.21.15.44:0/779795521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed1c0b4180 0x7fed080039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed1b7fe640 1 -- 172.21.15.44:0/779795521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed08004d40 msgr2=0x7fed080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed1b7fe640 1 --2- 172.21.15.44:0/779795521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed08004d40 0x7fed080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.159+0000 7fed1b7fe640 1 -- 172.21.15.44:0/779795521 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed08003520 con 0x7fed08003f30 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.160+0000 7fed1b7fe640 1 --2- 172.21.15.44:0/779795521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed08003f30 0x7fed08004800 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fed04009ac0 tx=0x7fed08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae501829df873a22 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.160+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04039070 con 0x7fed08003f30 2024-04-02T13:28:57.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.160+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed0402c6b0 con 0x7fed08003f30 2024-04-02T13:28:57.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.160+0000 7fed1affd640 1 -- 172.21.15.44:0/779795521 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04009690 con 0x7fed08003f30 2024-04-02T13:28:57.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.160+0000 7fed229cd640 1 -- 172.21.15.44:0/779795521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed08003f30 msgr2=0x7fed08004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.160+0000 7fed229cd640 1 --2- 172.21.15.44:0/779795521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed08003f30 0x7fed08004800 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fed04009ac0 tx=0x7fed08005830 comp rx=0 tx=0).stop 2024-04-02T13:28:57.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.160+0000 7fed20f6a640 1 -- 172.21.15.44:0/779795521 reap_dead start 2024-04-02T13:28:57.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.160+0000 7fed229cd640 1 -- 172.21.15.44:0/779795521 shutdown_connections 2024-04-02T13:28:57.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.160+0000 7fed229cd640 1 -- 172.21.15.44:0/779795521 >> 172.21.15.44:0/779795521 conn(0x7fed1c09aa20 msgr2=0x7fed1c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:57.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.160+0000 7fed229cd640 1 -- 172.21.15.44:0/779795521 shutdown_connections 2024-04-02T13:28:57.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.160+0000 7fed229cd640 1 -- 172.21.15.44:0/779795521 wait complete. 2024-04-02T13:28:57.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.162+0000 7fed229cd640 1 Processor -- start 2024-04-02T13:28:57.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.162+0000 7fed229cd640 1 -- start start 2024-04-02T13:28:57.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.162+0000 7fed229cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed08003f30 0x7fed1c14dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.162+0000 7fed229cd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed1c14a950 0x7fed1c0c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.162+0000 7fed229cd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed1c160cb0 0x7fed1c14d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.162+0000 7fed229cd640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c0d13f0 con 0x7fed08003f30 2024-04-02T13:28:57.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.162+0000 7fed229cd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c0d1270 con 0x7fed1c14a950 2024-04-02T13:28:57.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.162+0000 7fed229cd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fed1c0b1d40 con 0x7fed1c160cb0 2024-04-02T13:28:57.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed1bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed08003f30 0x7fed1c14dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed1bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed08003f30 0x7fed1c14dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38598/0 (socket says 172.21.15.44:38598) 2024-04-02T13:28:57.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed1bfff640 1 -- 172.21.15.44:0/2964038870 learned_addr learned my addr 172.21.15.44:0/2964038870 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:57.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed1b7fe640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed1c14a950 0x7fed1c0c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed20f6a640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed1c160cb0 0x7fed1c14d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed1bfff640 1 -- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed1c160cb0 msgr2=0x7fed1c14d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed1bfff640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed1c160cb0 0x7fed1c14d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed1bfff640 1 -- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed1c14a950 msgr2=0x7fed1c0c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed1bfff640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed1c14a950 0x7fed1c0c8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed1bfff640 1 -- 172.21.15.44:0/2964038870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1c0c6300 con 0x7fed08003f30 2024-04-02T13:28:57.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed1bfff640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed08003f30 0x7fed1c14dde0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fed0c007370 tx=0x7fed0c007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:57.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed18ff9640 1 -- 172.21.15.44:0/2964038870 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c00f9c0 con 0x7fed08003f30 2024-04-02T13:28:57.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed18ff9640 1 -- 172.21.15.44:0/2964038870 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed0c018070 con 0x7fed08003f30 2024-04-02T13:28:57.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed229cd640 1 -- 172.21.15.44:0/2964038870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1c0c5d40 con 0x7fed08003f30 2024-04-02T13:28:57.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed18ff9640 1 -- 172.21.15.44:0/2964038870 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c00fda0 con 0x7fed08003f30 2024-04-02T13:28:57.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.163+0000 7fed229cd640 1 -- 172.21.15.44:0/2964038870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed1c1604b0 con 0x7fed08003f30 2024-04-02T13:28:57.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.164+0000 7fed229cd640 1 -- 172.21.15.44:0/2964038870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed1c0684f0 con 0x7fed08003f30 2024-04-02T13:28:57.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.165+0000 7fed18ff9640 1 -- 172.21.15.44:0/2964038870 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fed0c01d460 con 0x7fed08003f30 2024-04-02T13:28:57.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.165+0000 7fed18ff9640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fecfc06c8f0 0x7fecfc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.165+0000 7fed18ff9640 1 -- 172.21.15.44:0/2964038870 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fed0c08bee0 con 0x7fed08003f30 2024-04-02T13:28:57.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.165+0000 7fed1b7fe640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fecfc06c8f0 0x7fecfc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.166+0000 7fed1b7fe640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fecfc06c8f0 0x7fecfc06edb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fed04036000 tx=0x7fed04030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:57.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.170+0000 7fed18ff9640 1 -- 172.21.15.44:0/2964038870 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fed0c05a0f0 con 0x7fed08003f30 2024-04-02T13:28:57.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.354+0000 7fed229cd640 1 -- 172.21.15.44:0/2964038870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fed1c067770 con 0x7fed08003f30 2024-04-02T13:28:57.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.354+0000 7fed18ff9640 1 -- 172.21.15.44:0/2964038870 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7fed0c0599f0 con 0x7fed08003f30 2024-04-02T13:28:57.357 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:28:57.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.356+0000 7fed229cd640 1 -- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fecfc06c8f0 msgr2=0x7fecfc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.356+0000 7fed229cd640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fecfc06c8f0 0x7fecfc06edb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fed04036000 tx=0x7fed04030000 comp rx=0 tx=0).stop 2024-04-02T13:28:57.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.356+0000 7fed229cd640 1 -- 172.21.15.44:0/2964038870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed08003f30 msgr2=0x7fed1c14dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.356+0000 7fed229cd640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed08003f30 0x7fed1c14dde0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fed0c007370 tx=0x7fed0c007830 comp rx=0 tx=0).stop 2024-04-02T13:28:57.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.356+0000 7fed229cd640 1 -- 172.21.15.44:0/2964038870 shutdown_connections 2024-04-02T13:28:57.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.356+0000 7fed229cd640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fecfc06c8f0 0x7fecfc06edb0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.356+0000 7fed229cd640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed1c160cb0 0x7fed1c14d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.356+0000 7fed229cd640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed1c14a950 0x7fed1c0c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.356+0000 7fed229cd640 1 --2- 172.21.15.44:0/2964038870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed08003f30 0x7fed1c14dde0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.356+0000 7fed229cd640 1 -- 172.21.15.44:0/2964038870 >> 172.21.15.44:0/2964038870 conn(0x7fed1c09aa20 msgr2=0x7fed1c070650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:57.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.356+0000 7fed229cd640 1 -- 172.21.15.44:0/2964038870 shutdown_connections 2024-04-02T13:28:57.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.356+0000 7fed229cd640 1 -- 172.21.15.44:0/2964038870 wait complete. 2024-04-02T13:28:57.369 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:28:57.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.468+0000 7fd15c332640 1 Processor -- start 2024-04-02T13:28:57.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.468+0000 7fd15c332640 1 -- start start 2024-04-02T13:28:57.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd15c332640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd1540d28f0 0x7fd1540d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd15c332640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1540b5200 0x7fd1540b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd15c332640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1540d4d70 con 0x7fd1540d1260 2024-04-02T13:28:57.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd15c332640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd1540d5050 con 0x7fd1540d28f0 2024-04-02T13:28:57.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd15c332640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd1540d23a0 con 0x7fd1540b5200 2024-04-02T13:28:57.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd15a8cf640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd1540d1260 0x7fd1540ced30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54972/0 (socket says 172.21.15.44:54972) 2024-04-02T13:28:57.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd15a8cf640 1 -- 172.21.15.44:0/1797689895 learned_addr learned my addr 172.21.15.44:0/1797689895 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd15a0ce640 1 --2- 172.21.15.44:0/1797689895 >> v2:172.21.15.69:3300/0 conn(0x7fd1540d28f0 0x7fd1540d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd1598cd640 1 --2- 172.21.15.44:0/1797689895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1540b5200 0x7fd1540b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 289064238 0 0) 0x7fd1540d4d70 con 0x7fd1540d1260 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd140003520 con 0x7fd1540d1260 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1006542310 0 0) 0x7fd140003520 con 0x7fd1540d1260 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1540b5200 msgr2=0x7fd1540b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd1590cc640 1 --2- 172.21.15.44:0/1797689895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1540b5200 0x7fd1540b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 >> v2:172.21.15.69:3300/0 conn(0x7fd1540d28f0 msgr2=0x7fd1540d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd1590cc640 1 --2- 172.21.15.44:0/1797689895 >> v2:172.21.15.69:3300/0 conn(0x7fd1540d28f0 0x7fd1540d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1540d2030 con 0x7fd1540d1260 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.469+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fd14c0019f0 con 0x7fd1540d1260 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 >> v1:172.21.15.44:6789/0 conn(0x7fd1540d1260 legacy=0x7fd1540ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd1590cc640 1 --2- 172.21.15.44:0/1797689895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1400039b0 0x7fd140003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd1590cc640 1 --2- 172.21.15.44:0/1797689895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd140004800 0x7fd140004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd1590cc640 1 --2- 172.21.15.44:0/1797689895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1540b5200 0x7fd1400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd1540d5050 con 0x7fd140004800 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd1540d23a0 con 0x7fd1400039b0 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd14000ba70 con 0x7fd1540b5200 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd1598cd640 1 --2- 172.21.15.44:0/1797689895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd140004800 0x7fd140004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd15a0ce640 1 --2- 172.21.15.44:0/1797689895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1400039b0 0x7fd140003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd15a8cf640 1 --2- 172.21.15.44:0/1797689895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1540b5200 0x7fd1400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd1598cd640 1 -- 172.21.15.44:0/1797689895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1540b5200 msgr2=0x7fd1400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd1598cd640 1 --2- 172.21.15.44:0/1797689895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd1540b5200 0x7fd1400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd1598cd640 1 -- 172.21.15.44:0/1797689895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1400039b0 msgr2=0x7fd140003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd1598cd640 1 --2- 172.21.15.44:0/1797689895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1400039b0 0x7fd140003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.470+0000 7fd1598cd640 1 -- 172.21.15.44:0/1797689895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd140003520 con 0x7fd140004800 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.471+0000 7fd1598cd640 1 --2- 172.21.15.44:0/1797689895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd140004800 0x7fd140004c80 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fd144009ac0 tx=0x7fd140005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c0af4371624117f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.471+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144039070 con 0x7fd140004800 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.471+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd14402c6b0 con 0x7fd140004800 2024-04-02T13:28:57.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.471+0000 7fd1590cc640 1 -- 172.21.15.44:0/1797689895 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144009690 con 0x7fd140004800 2024-04-02T13:28:57.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.471+0000 7fd15c332640 1 -- 172.21.15.44:0/1797689895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd140004800 msgr2=0x7fd140004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.471+0000 7fd15c332640 1 --2- 172.21.15.44:0/1797689895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd140004800 0x7fd140004c80 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fd144009ac0 tx=0x7fd140005830 comp rx=0 tx=0).stop 2024-04-02T13:28:57.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.471+0000 7fd15a8cf640 1 -- 172.21.15.44:0/1797689895 reap_dead start 2024-04-02T13:28:57.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.471+0000 7fd15c332640 1 -- 172.21.15.44:0/1797689895 shutdown_connections 2024-04-02T13:28:57.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.471+0000 7fd15c332640 1 -- 172.21.15.44:0/1797689895 >> 172.21.15.44:0/1797689895 conn(0x7fd1540d4120 msgr2=0x7fd1540d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:57.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.472+0000 7fd15c332640 1 -- 172.21.15.44:0/1797689895 shutdown_connections 2024-04-02T13:28:57.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.472+0000 7fd15c332640 1 -- 172.21.15.44:0/1797689895 wait complete. 2024-04-02T13:28:57.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.473+0000 7fd15c332640 1 Processor -- start 2024-04-02T13:28:57.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.473+0000 7fd15c332640 1 -- start start 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd15c332640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd140004800 0x7fd1541f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd15c332640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1541f67d0 0x7fd1540b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd15c332640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1540b6150 0x7fd1540cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd15c332640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd1540b8130 con 0x7fd1541f67d0 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd15c332640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd1540b5020 con 0x7fd1540b6150 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd15c332640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd1540b82b0 con 0x7fd140004800 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd1598cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1541f67d0 0x7fd1540b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd1598cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1541f67d0 0x7fd1540b5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38612/0 (socket says 172.21.15.44:38612) 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd1598cd640 1 -- 172.21.15.44:0/2343793725 learned_addr learned my addr 172.21.15.44:0/2343793725 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd15a8cf640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1540b6150 0x7fd1540cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd15a0ce640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd140004800 0x7fd1541f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd1598cd640 1 -- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd140004800 msgr2=0x7fd1541f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd1598cd640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd140004800 0x7fd1541f3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd1598cd640 1 -- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1540b6150 msgr2=0x7fd1540cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd1598cd640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1540b6150 0x7fd1540cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd1598cd640 1 -- 172.21.15.44:0/2343793725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15415fea0 con 0x7fd1541f67d0 2024-04-02T13:28:57.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd15a0ce640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd140004800 0x7fd1541f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:57.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.474+0000 7fd1598cd640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1541f67d0 0x7fd1540b5c10 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fd144009be0 tx=0x7fd144004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:57.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.475+0000 7fd14affd640 1 -- 172.21.15.44:0/2343793725 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144042020 con 0x7fd1541f67d0 2024-04-02T13:28:57.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.475+0000 7fd15c332640 1 -- 172.21.15.44:0/2343793725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd154160180 con 0x7fd1541f67d0 2024-04-02T13:28:57.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.475+0000 7fd14affd640 1 -- 172.21.15.44:0/2343793725 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd14403a040 con 0x7fd1541f67d0 2024-04-02T13:28:57.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.475+0000 7fd14affd640 1 -- 172.21.15.44:0/2343793725 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144039070 con 0x7fd1541f67d0 2024-04-02T13:28:57.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.475+0000 7fd15c332640 1 -- 172.21.15.44:0/2343793725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd154160740 con 0x7fd1541f67d0 2024-04-02T13:28:57.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.476+0000 7fd15c332640 1 -- 172.21.15.44:0/2343793725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1540684f0 con 0x7fd1541f67d0 2024-04-02T13:28:57.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.477+0000 7fd14affd640 1 -- 172.21.15.44:0/2343793725 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd144045020 con 0x7fd1541f67d0 2024-04-02T13:28:57.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.477+0000 7fd14affd640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd13806c920 0x7fd13806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.477+0000 7fd14affd640 1 -- 172.21.15.44:0/2343793725 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fd1440ad470 con 0x7fd1541f67d0 2024-04-02T13:28:57.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.477+0000 7fd15a0ce640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd13806c920 0x7fd13806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.478+0000 7fd15a0ce640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd13806c920 0x7fd13806ede0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd150009bc0 tx=0x7fd150008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:57.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.482+0000 7fd14affd640 1 -- 172.21.15.44:0/2343793725 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd14404c030 con 0x7fd1541f67d0 2024-04-02T13:28:57.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.705+0000 7fd15c332640 1 -- 172.21.15.44:0/2343793725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd15406cc00 con 0x7fd1541f67d0 2024-04-02T13:28:57.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.705+0000 7fd14affd640 1 -- 172.21.15.44:0/2343793725 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5181 (secure 0 0 0) 0x7fd14407b570 con 0x7fd1541f67d0 2024-04-02T13:28:57.708 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:57.708 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:28:19.321536+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8403":{"gid":8403,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6833/3044289683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":3044289683},{"type":"v1","addr":"172.21.15.44:6833","nonce":3044289683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8403,"qdb_cluster":[8403]},"id":6}]} 2024-04-02T13:28:57.708 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 74 2024-04-02T13:28:57.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.707+0000 7fd15c332640 1 -- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd13806c920 msgr2=0x7fd13806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.707+0000 7fd15c332640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd13806c920 0x7fd13806ede0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd150009bc0 tx=0x7fd150008040 comp rx=0 tx=0).stop 2024-04-02T13:28:57.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.707+0000 7fd15c332640 1 -- 172.21.15.44:0/2343793725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1541f67d0 msgr2=0x7fd1540b5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.707+0000 7fd15c332640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1541f67d0 0x7fd1540b5c10 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fd144009be0 tx=0x7fd144004730 comp rx=0 tx=0).stop 2024-04-02T13:28:57.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.707+0000 7fd15c332640 1 -- 172.21.15.44:0/2343793725 shutdown_connections 2024-04-02T13:28:57.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.707+0000 7fd15c332640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd13806c920 0x7fd13806ede0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.707+0000 7fd15c332640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1540b6150 0x7fd1540cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.707+0000 7fd15c332640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd1541f67d0 0x7fd1540b5c10 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.707+0000 7fd15c332640 1 --2- 172.21.15.44:0/2343793725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd140004800 0x7fd1541f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.707+0000 7fd15c332640 1 -- 172.21.15.44:0/2343793725 >> 172.21.15.44:0/2343793725 conn(0x7fd1540d4120 msgr2=0x7fd15406c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:57.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.707+0000 7fd15c332640 1 -- 172.21.15.44:0/2343793725 shutdown_connections 2024-04-02T13:28:57.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.707+0000 7fd15c332640 1 -- 172.21.15.44:0/2343793725 wait complete. 2024-04-02T13:28:57.721 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:28:57.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.820+0000 7fc8f6a4a640 1 Processor -- start 2024-04-02T13:28:57.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.820+0000 7fc8f6a4a640 1 -- start start 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.820+0000 7fc8f6a4a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8f015f040 0x7fc8f0168f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.820+0000 7fc8f6a4a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8f015fb50 con 0x7fc8f006d700 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.821+0000 7fc8f6a4a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8f00d1ac0 con 0x7fc8f015e550 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.821+0000 7fc8f6a4a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc8f006ca90 con 0x7fc8f015f040 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.821+0000 7fc8ef7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc8f006d700 0x7fc8f015e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54974/0 (socket says 172.21.15.44:54974) 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.821+0000 7fc8ef7fe640 1 -- 172.21.15.44:0/635932428 learned_addr learned my addr 172.21.15.44:0/635932428 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.821+0000 7fc8f4fe7640 1 --2- 172.21.15.44:0/635932428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8f015f040 0x7fc8f0168f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.821+0000 7fc8eeffd640 1 -- 172.21.15.44:0/635932428 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2703185807 0 0) 0x7fc8f015fb50 con 0x7fc8f006d700 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.821+0000 7fc8eeffd640 1 -- 172.21.15.44:0/635932428 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8d8003520 con 0x7fc8f006d700 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.821+0000 7fc8eeffd640 1 -- 172.21.15.44:0/635932428 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145740226 0 0) 0x7fc8f00d1ac0 con 0x7fc8f015e550 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.821+0000 7fc8eeffd640 1 -- 172.21.15.44:0/635932428 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8f015fb50 con 0x7fc8f015e550 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.821+0000 7fc8f4fe7640 1 -- 172.21.15.44:0/635932428 >> v1:172.21.15.69:6789/0 conn(0x7fc8f015e550 legacy=0x7fc8f006cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.821+0000 7fc8f4fe7640 1 -- 172.21.15.44:0/635932428 >> v1:172.21.15.44:6789/0 conn(0x7fc8f006d700 legacy=0x7fc8f015e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.821+0000 7fc8f4fe7640 1 -- 172.21.15.44:0/635932428 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8f015f590 con 0x7fc8f015f040 2024-04-02T13:28:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.822+0000 7fc8f4fe7640 1 --2- 172.21.15.44:0/635932428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8f015f040 0x7fc8f0168f60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009dc0 tx=0x7fc8e802bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=288c179ae893d0b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.822+0000 7fc8eeffd640 1 -- 172.21.15.44:0/635932428 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8039070 con 0x7fc8f015f040 2024-04-02T13:28:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.822+0000 7fc8eeffd640 1 -- 172.21.15.44:0/635932428 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8e802cd50 con 0x7fc8f015f040 2024-04-02T13:28:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.822+0000 7fc8eeffd640 1 -- 172.21.15.44:0/635932428 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8034640 con 0x7fc8f015f040 2024-04-02T13:28:57.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.823+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/635932428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8f015f040 msgr2=0x7fc8f0168f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.823+0000 7fc8f6a4a640 1 --2- 172.21.15.44:0/635932428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8f015f040 0x7fc8f0168f60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009dc0 tx=0x7fc8e802bfa0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.823+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/635932428 shutdown_connections 2024-04-02T13:28:57.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.823+0000 7fc8f6a4a640 1 --2- 172.21.15.44:0/635932428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8f015f040 0x7fc8f0168f60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.823+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/635932428 >> 172.21.15.44:0/635932428 conn(0x7fc8f009aa20 msgr2=0x7fc8f009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:57.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.823+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/635932428 shutdown_connections 2024-04-02T13:28:57.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.823+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/635932428 wait complete. 2024-04-02T13:28:57.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.824+0000 7fc8f6a4a640 1 Processor -- start 2024-04-02T13:28:57.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.824+0000 7fc8f6a4a640 1 -- start start 2024-04-02T13:28:57.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8f6a4a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8f006d700 0x7fc8f009f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8f6a4a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8f015e550 0x7fc8f00caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8f6a4a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8f015f040 0x7fc8f00705d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8f6a4a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f00d0590 con 0x7fc8f015e550 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8f6a4a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f00d02b0 con 0x7fc8f006d700 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8f6a4a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc8f00cc4b0 con 0x7fc8f015f040 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8ef7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8f015e550 0x7fc8f00caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8ef7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8f015e550 0x7fc8f00caf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38618/0 (socket says 172.21.15.44:38618) 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8ef7fe640 1 -- 172.21.15.44:0/3478016927 learned_addr learned my addr 172.21.15.44:0/3478016927 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8f4fe7640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8f015f040 0x7fc8f00705d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8effff640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8f006d700 0x7fc8f009f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8ef7fe640 1 -- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8f015f040 msgr2=0x7fc8f00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8ef7fe640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8f015f040 0x7fc8f00705d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8ef7fe640 1 -- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8f006d700 msgr2=0x7fc8f009f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8ef7fe640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8f006d700 0x7fc8f009f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8ef7fe640 1 -- 172.21.15.44:0/3478016927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8f009bc30 con 0x7fc8f015e550 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.825+0000 7fc8ef7fe640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8f015e550 0x7fc8f00caf30 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400c410 tx=0x7fc8e400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.826+0000 7fc8ecff9640 1 -- 172.21.15.44:0/3478016927 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4017070 con 0x7fc8f015e550 2024-04-02T13:28:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.826+0000 7fc8ecff9640 1 -- 172.21.15.44:0/3478016927 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8e4005ce0 con 0x7fc8f015e550 2024-04-02T13:28:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.826+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/3478016927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8f00cb780 con 0x7fc8f015e550 2024-04-02T13:28:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.826+0000 7fc8ecff9640 1 -- 172.21.15.44:0/3478016927 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e40072d0 con 0x7fc8f015e550 2024-04-02T13:28:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.826+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/3478016927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8f00cb470 con 0x7fc8f015e550 2024-04-02T13:28:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.827+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/3478016927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8f00cc200 con 0x7fc8f015e550 2024-04-02T13:28:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.827+0000 7fc8ecff9640 1 -- 172.21.15.44:0/3478016927 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc8e401a070 con 0x7fc8f015e550 2024-04-02T13:28:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.827+0000 7fc8ecff9640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc8dc06c8f0 0x7fc8dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.827+0000 7fc8ecff9640 1 -- 172.21.15.44:0/3478016927 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fc8e408b120 con 0x7fc8f015e550 2024-04-02T13:28:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.828+0000 7fc8effff640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc8dc06c8f0 0x7fc8dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.828+0000 7fc8effff640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc8dc06c8f0 0x7fc8dc06edb0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc8e00011e0 tx=0x7fc8e0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:57.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:57.832+0000 7fc8ecff9640 1 -- 172.21.15.44:0/3478016927 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc8f00cc200 con 0x7fc8f015e550 2024-04-02T13:28:58.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.017+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/3478016927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8f0067d70 con 0x7fc8f015e550 2024-04-02T13:28:58.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.019+0000 7fc8ecff9640 1 -- 172.21.15.44:0/3478016927 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+17288 (secure 0 0 0) 0x7fc8e4058c50 con 0x7fc8f015e550 2024-04-02T13:28:58.022 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:58.022 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":95,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:28:15.300549+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:26:41.300970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-02T13:26:41.979586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.24","mappings":[{"from":1,"to":5}]},{"pgid":"14.2c","mappings":[{"from":1,"to":2}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6833/2250279975":"2024-04-03T13:28:15.300517+0000","172.21.15.44:6832/2250279975":"2024-04-03T13:28:15.300517+0000","172.21.15.44:6833/719680934":"2024-04-03T13:26:55.589090+0000","172.21.15.44:6832/719680934":"2024-04-03T13:26:55.589090+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:28:58.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.020+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc8dc06c8f0 msgr2=0x7fc8dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.020+0000 7fc8f6a4a640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc8dc06c8f0 0x7fc8dc06edb0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc8e00011e0 tx=0x7fc8e0009040 comp rx=0 tx=0).stop 2024-04-02T13:28:58.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.021+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/3478016927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8f015e550 msgr2=0x7fc8f00caf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.021+0000 7fc8f6a4a640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8f015e550 0x7fc8f00caf30 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400c410 tx=0x7fc8e400c8d0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.021+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/3478016927 shutdown_connections 2024-04-02T13:28:58.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.021+0000 7fc8f6a4a640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc8dc06c8f0 0x7fc8dc06edb0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.021+0000 7fc8f6a4a640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8f015f040 0x7fc8f00705d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.021+0000 7fc8f6a4a640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8f015e550 0x7fc8f00caf30 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.021+0000 7fc8f6a4a640 1 --2- 172.21.15.44:0/3478016927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8f006d700 0x7fc8f009f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.021+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/3478016927 >> 172.21.15.44:0/3478016927 conn(0x7fc8f009aa20 msgr2=0x7fc8f006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:58.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.021+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/3478016927 shutdown_connections 2024-04-02T13:28:58.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.021+0000 7fc8f6a4a640 1 -- 172.21.15.44:0/3478016927 wait complete. 2024-04-02T13:28:58.034 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T13:28:58.035 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:28:58.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.134+0000 7f6fe89fe640 1 Processor -- start 2024-04-02T13:28:58.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.134+0000 7f6fe89fe640 1 -- start start 2024-04-02T13:28:58.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.134+0000 7f6fe89fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6fe4162550 0x7f6fe40d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.134+0000 7f6fe89fe640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe406d1d0 con 0x7f6fe415e270 2024-04-02T13:28:58.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.134+0000 7f6fe89fe640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe406d4b0 con 0x7f6fe41629e0 2024-04-02T13:28:58.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.134+0000 7f6fe89fe640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f6fe406dde0 con 0x7f6fe4162550 2024-04-02T13:28:58.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.134+0000 7f6fe259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6fe4162550 0x7f6fe40d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.134+0000 7f6fe2d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6fe415e270 0x7f6fe4161e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54984/0 (socket says 172.21.15.44:54984) 2024-04-02T13:28:58.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.135+0000 7f6fe2d9d640 1 -- 172.21.15.44:0/3125622598 learned_addr learned my addr 172.21.15.44:0/3125622598 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:58.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.135+0000 7f6fe259c640 1 --2- 172.21.15.44:0/3125622598 >> v2:172.21.15.44:3300/0 conn(0x7f6fe4162550 0x7f6fe40d4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38634/0 (socket says 172.21.15.44:38634) 2024-04-02T13:28:58.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.135+0000 7f6fe259c640 1 -- 172.21.15.44:0/3125622598 >> v1:172.21.15.69:6789/0 conn(0x7f6fe41629e0 legacy=0x7f6fe40b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:58.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.135+0000 7f6fe259c640 1 -- 172.21.15.44:0/3125622598 >> v1:172.21.15.44:6789/0 conn(0x7f6fe415e270 legacy=0x7f6fe4161e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.135+0000 7f6fe259c640 1 -- 172.21.15.44:0/3125622598 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe406d790 con 0x7f6fe4162550 2024-04-02T13:28:58.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.135+0000 7f6fe259c640 1 --2- 172.21.15.44:0/3125622598 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fe4162550 0x7f6fe40d4a10 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80097a0 tx=0x7f6fd802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8080b3eb8147d3ee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:58.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.136+0000 7f6fe1d9b640 1 -- 172.21.15.44:0/3125622598 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8035020 con 0x7f6fe4162550 2024-04-02T13:28:58.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.136+0000 7f6fe1d9b640 1 -- 172.21.15.44:0/3125622598 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd8031830 con 0x7f6fe4162550 2024-04-02T13:28:58.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.136+0000 7f6fe1d9b640 1 -- 172.21.15.44:0/3125622598 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8031c00 con 0x7f6fe4162550 2024-04-02T13:28:58.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.136+0000 7f6fe89fe640 1 -- 172.21.15.44:0/3125622598 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fe4162550 msgr2=0x7f6fe40d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.136+0000 7f6fe89fe640 1 --2- 172.21.15.44:0/3125622598 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fe4162550 0x7f6fe40d4a10 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80097a0 tx=0x7f6fd802b8d0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.136+0000 7f6fe89fe640 1 -- 172.21.15.44:0/3125622598 shutdown_connections 2024-04-02T13:28:58.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.136+0000 7f6fe89fe640 1 --2- 172.21.15.44:0/3125622598 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fe4162550 0x7f6fe40d4a10 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.136+0000 7f6fe89fe640 1 -- 172.21.15.44:0/3125622598 >> 172.21.15.44:0/3125622598 conn(0x7f6fe40d3ef0 msgr2=0x7f6fe40d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:58.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.136+0000 7f6fe89fe640 1 -- 172.21.15.44:0/3125622598 shutdown_connections 2024-04-02T13:28:58.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.136+0000 7f6fe89fe640 1 -- 172.21.15.44:0/3125622598 wait complete. 2024-04-02T13:28:58.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.138+0000 7f6fe89fe640 1 Processor -- start 2024-04-02T13:28:58.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.138+0000 7f6fe89fe640 1 -- start start 2024-04-02T13:28:58.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.138+0000 7f6fe89fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fe415e270 0x7f6fe41f3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.138+0000 7f6fe89fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe4162550 0x7f6fe40b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.138+0000 7f6fe89fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe41629e0 0x7f6fe4144ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.138+0000 7f6fe89fe640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe415e650 con 0x7f6fe415e270 2024-04-02T13:28:58.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.138+0000 7f6fe89fe640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe406cef0 con 0x7f6fe4162550 2024-04-02T13:28:58.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.138+0000 7f6fe89fe640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe40b1560 con 0x7f6fe41629e0 2024-04-02T13:28:58.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.138+0000 7f6fe2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fe415e270 0x7f6fe41f3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fe2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fe415e270 0x7f6fe41f3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38644/0 (socket says 172.21.15.44:38644) 2024-04-02T13:28:58.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fe2d9d640 1 -- 172.21.15.44:0/380156675 learned_addr learned my addr 172.21.15.44:0/380156675 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:58.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fe259c640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe4162550 0x7f6fe40b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fe359e640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe41629e0 0x7f6fe4144ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fe2d9d640 1 -- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe41629e0 msgr2=0x7f6fe4144ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fe2d9d640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe41629e0 0x7f6fe4144ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fe2d9d640 1 -- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe4162550 msgr2=0x7f6fe40b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fe2d9d640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe4162550 0x7f6fe40b4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fe2d9d640 1 -- 172.21.15.44:0/380156675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe41f42d0 con 0x7f6fe415e270 2024-04-02T13:28:58.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fe2d9d640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fe415e270 0x7f6fe41f3d90 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00c430 tx=0x7f6fdc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:58.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fcb7fe640 1 -- 172.21.15.44:0/380156675 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc017070 con 0x7f6fe415e270 2024-04-02T13:28:58.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fcb7fe640 1 -- 172.21.15.44:0/380156675 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fdc005ce0 con 0x7f6fe415e270 2024-04-02T13:28:58.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fe89fe640 1 -- 172.21.15.44:0/380156675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe40b4d00 con 0x7f6fe415e270 2024-04-02T13:28:58.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fcb7fe640 1 -- 172.21.15.44:0/380156675 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0072f0 con 0x7f6fe415e270 2024-04-02T13:28:58.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.139+0000 7f6fe89fe640 1 -- 172.21.15.44:0/380156675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe40b52c0 con 0x7f6fe415e270 2024-04-02T13:28:58.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.140+0000 7f6fe89fe640 1 -- 172.21.15.44:0/380156675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe404f8d0 con 0x7f6fe415e270 2024-04-02T13:28:58.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.141+0000 7f6fcb7fe640 1 -- 172.21.15.44:0/380156675 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6fdc01a070 con 0x7f6fe415e270 2024-04-02T13:28:58.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.141+0000 7f6fcb7fe640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6fd006c8f0 0x7f6fd006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.141+0000 7f6fcb7fe640 1 -- 172.21.15.44:0/380156675 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7f6fdc08b220 con 0x7f6fe415e270 2024-04-02T13:28:58.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.141+0000 7f6fe259c640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6fd006c8f0 0x7f6fd006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.142+0000 7f6fe259c640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6fd006c8f0 0x7f6fd006edb0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8036000 tx=0x7f6fd8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:58.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.146+0000 7f6fcb7fe640 1 -- 172.21.15.44:0/380156675 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6fdc0593a0 con 0x7f6fe415e270 2024-04-02T13:28:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.369+0000 7f6fe89fe640 1 -- 172.21.15.44:0/380156675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fe40b2be0 con 0x7f6fe415e270 2024-04-02T13:28:58.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.370+0000 7f6fcb7fe640 1 -- 172.21.15.44:0/380156675 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5181 (secure 0 0 0) 0x7f6fdc058ca0 con 0x7f6fe415e270 2024-04-02T13:28:58.373 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:58.373 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:28:19.321536+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8403":{"gid":8403,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6833/3044289683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":3044289683},{"type":"v1","addr":"172.21.15.44:6833","nonce":3044289683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8403,"qdb_cluster":[8403]},"id":6}]} 2024-04-02T13:28:58.373 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 74 2024-04-02T13:28:58.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.372+0000 7f6fe89fe640 1 -- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6fd006c8f0 msgr2=0x7f6fd006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.372+0000 7f6fe89fe640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6fd006c8f0 0x7f6fd006edb0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8036000 tx=0x7f6fd8008030 comp rx=0 tx=0).stop 2024-04-02T13:28:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.372+0000 7f6fe89fe640 1 -- 172.21.15.44:0/380156675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fe415e270 msgr2=0x7f6fe41f3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.372+0000 7f6fe89fe640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fe415e270 0x7f6fe41f3d90 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00c430 tx=0x7f6fdc00c8f0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.372+0000 7f6fe89fe640 1 -- 172.21.15.44:0/380156675 shutdown_connections 2024-04-02T13:28:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.372+0000 7f6fe89fe640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6fd006c8f0 0x7f6fd006edb0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.372+0000 7f6fe89fe640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe41629e0 0x7f6fe4144ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.372+0000 7f6fe89fe640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe4162550 0x7f6fe40b4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.372+0000 7f6fe89fe640 1 --2- 172.21.15.44:0/380156675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fe415e270 0x7f6fe41f3d90 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.372+0000 7f6fe89fe640 1 -- 172.21.15.44:0/380156675 >> 172.21.15.44:0/380156675 conn(0x7f6fe40d3ef0 msgr2=0x7f6fe40b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.372+0000 7f6fe89fe640 1 -- 172.21.15.44:0/380156675 shutdown_connections 2024-04-02T13:28:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.372+0000 7f6fe89fe640 1 -- 172.21.15.44:0/380156675 wait complete. 2024-04-02T13:28:58.386 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:28:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.484+0000 7faab6bcd640 1 Processor -- start 2024-04-02T13:28:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.484+0000 7faab6bcd640 1 -- start start 2024-04-02T13:28:58.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.485+0000 7faab6bcd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab015c250 0x7faab00aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.485+0000 7faab6bcd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faab00b3510 0x7faab00d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.485+0000 7faab6bcd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7faab00d08d0 0x7faab00b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.485+0000 7faab6bcd640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7faab015c630 con 0x7faab00d08d0 2024-04-02T13:28:58.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.485+0000 7faab6bcd640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7faab00af420 con 0x7faab00b3510 2024-04-02T13:28:58.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.485+0000 7faab6bcd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faab00af700 con 0x7faab015c250 2024-04-02T13:28:58.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.485+0000 7faab516a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7faab00d08d0 0x7faab00b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.485+0000 7faab516a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7faab00d08d0 0x7faab00b1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38646/0 (socket says 172.21.15.44:38646) 2024-04-02T13:28:58.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.485+0000 7faab516a640 1 -- 172.21.15.44:0/1460080639 learned_addr learned my addr 172.21.15.44:0/1460080639 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:58.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.485+0000 7faab4969640 1 --2- 172.21.15.44:0/1460080639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab015c250 0x7faab00aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.485+0000 7faaaffff640 1 --2- 172.21.15.44:0/1460080639 >> v2:172.21.15.69:3300/0 conn(0x7faab00b3510 0x7faab00d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.486+0000 7faab516a640 1 -- 172.21.15.44:0/1460080639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab015c250 msgr2=0x7faab00aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:58.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.486+0000 7faab516a640 1 --2- 172.21.15.44:0/1460080639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab015c250 0x7faab00aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.486+0000 7faab516a640 1 -- 172.21.15.44:0/1460080639 >> v2:172.21.15.69:3300/0 conn(0x7faab00b3510 msgr2=0x7faab00d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:58.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.486+0000 7faab516a640 1 --2- 172.21.15.44:0/1460080639 >> v2:172.21.15.69:3300/0 conn(0x7faab00b3510 0x7faab00d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.486+0000 7faab516a640 1 -- 172.21.15.44:0/1460080639 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab00af9e0 con 0x7faab00d08d0 2024-04-02T13:28:58.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.486+0000 7faab516a640 1 --2- 172.21.15.44:0/1460080639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab00d08d0 0x7faab00b1e40 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7faab00b4ad0 tx=0x7faaa802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7de60a9f1c1b8ef server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:58.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.487+0000 7faaaf7fe640 1 -- 172.21.15.44:0/1460080639 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa802c9e0 con 0x7faab00d08d0 2024-04-02T13:28:58.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.487+0000 7faaaf7fe640 1 -- 172.21.15.44:0/1460080639 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaa802cb80 con 0x7faab00d08d0 2024-04-02T13:28:58.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.487+0000 7faaaf7fe640 1 -- 172.21.15.44:0/1460080639 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa80343f0 con 0x7faab00d08d0 2024-04-02T13:28:58.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.487+0000 7faab6bcd640 1 -- 172.21.15.44:0/1460080639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab00d08d0 msgr2=0x7faab00b1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.487+0000 7faab6bcd640 1 --2- 172.21.15.44:0/1460080639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab00d08d0 0x7faab00b1e40 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7faab00b4ad0 tx=0x7faaa802be40 comp rx=0 tx=0).stop 2024-04-02T13:28:58.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.487+0000 7faab6bcd640 1 -- 172.21.15.44:0/1460080639 shutdown_connections 2024-04-02T13:28:58.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.487+0000 7faab6bcd640 1 --2- 172.21.15.44:0/1460080639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab00d08d0 0x7faab00b1e40 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.487+0000 7faab6bcd640 1 --2- 172.21.15.44:0/1460080639 >> v2:172.21.15.69:3300/0 conn(0x7faab00b3510 0x7faab00d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.487+0000 7faab6bcd640 1 --2- 172.21.15.44:0/1460080639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab015c250 0x7faab00aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.487+0000 7faab6bcd640 1 -- 172.21.15.44:0/1460080639 >> 172.21.15.44:0/1460080639 conn(0x7faab0073850 msgr2=0x7faab009aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.487+0000 7faab6bcd640 1 -- 172.21.15.44:0/1460080639 shutdown_connections 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.487+0000 7faab6bcd640 1 -- 172.21.15.44:0/1460080639 wait complete. 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.488+0000 7faab6bcd640 1 Processor -- start 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.488+0000 7faab6bcd640 1 -- start start 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.489+0000 7faab6bcd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab00b3510 0x7faab006cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.489+0000 7faab6bcd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab00d08d0 0x7faab015d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.489+0000 7faab6bcd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab015c250 0x7faab015dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.489+0000 7faab6bcd640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faab00b2380 con 0x7faab015c250 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.489+0000 7faab6bcd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faab00af130 con 0x7faab00b3510 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.489+0000 7faab6bcd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faab00b2500 con 0x7faab00d08d0 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.489+0000 7faab516a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab015c250 0x7faab015dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.489+0000 7faab516a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab015c250 0x7faab015dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38662/0 (socket says 172.21.15.44:38662) 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.489+0000 7faab516a640 1 -- 172.21.15.44:0/276366256 learned_addr learned my addr 172.21.15.44:0/276366256 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:58.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faaaffff640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab00d08d0 0x7faab015d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faab4969640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab00b3510 0x7faab006cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faab516a640 1 -- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab00d08d0 msgr2=0x7faab015d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faab516a640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab00d08d0 0x7faab015d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faab516a640 1 -- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab00b3510 msgr2=0x7faab006cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faab516a640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab00b3510 0x7faab006cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faab516a640 1 -- 172.21.15.44:0/276366256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab006d4c0 con 0x7faab015c250 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faab4969640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab00b3510 0x7faab006cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faaaffff640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab00d08d0 0x7faab015d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faab516a640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab015c250 0x7faab015dea0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7faaa8036040 tx=0x7faaa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faaad7fa640 1 -- 172.21.15.44:0/276366256 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa8043020 con 0x7faab015c250 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faaad7fa640 1 -- 172.21.15.44:0/276366256 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaa803e450 con 0x7faab015c250 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faaad7fa640 1 -- 172.21.15.44:0/276366256 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa8009510 con 0x7faab015c250 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faab6bcd640 1 -- 172.21.15.44:0/276366256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab006d780 con 0x7faab015c250 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.490+0000 7faab6bcd640 1 -- 172.21.15.44:0/276366256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab006dd40 con 0x7faab015c250 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.491+0000 7faab6bcd640 1 -- 172.21.15.44:0/276366256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab00711c0 con 0x7faab015c250 2024-04-02T13:28:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.491+0000 7faaad7fa640 1 -- 172.21.15.44:0/276366256 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faaa8046020 con 0x7faab015c250 2024-04-02T13:28:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.492+0000 7faaad7fa640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faa9406c920 0x7faa9406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.492+0000 7faaad7fa640 1 -- 172.21.15.44:0/276366256 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7faaa80ae230 con 0x7faab015c250 2024-04-02T13:28:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.492+0000 7faab4969640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faa9406c920 0x7faa9406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.493+0000 7faab4969640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faa9406c920 0x7faa9406ede0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7faaa4009a20 tx=0x7faaa4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:58.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.497+0000 7faaad7fa640 1 -- 172.21.15.44:0/276366256 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7faab00711c0 con 0x7faab015c250 2024-04-02T13:28:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.720+0000 7faab6bcd640 1 -- 172.21.15.44:0/276366256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faab0068c00 con 0x7faab015c250 2024-04-02T13:28:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.721+0000 7faaad7fa640 1 -- 172.21.15.44:0/276366256 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5181 (secure 0 0 0) 0x7faaa804d070 con 0x7faab015c250 2024-04-02T13:28:58.724 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:58.724 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:26:43.025860+0000","modified":"2024-04-02T13:28:19.321536+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8403":{"gid":8403,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6833/3044289683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":3044289683},{"type":"v1","addr":"172.21.15.44:6833","nonce":3044289683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8403,"qdb_cluster":[8403]},"id":6}]} 2024-04-02T13:28:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 74 2024-04-02T13:28:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.723+0000 7faab6bcd640 1 -- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faa9406c920 msgr2=0x7faa9406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.723+0000 7faab6bcd640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faa9406c920 0x7faa9406ede0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7faaa4009a20 tx=0x7faaa4008040 comp rx=0 tx=0).stop 2024-04-02T13:28:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.723+0000 7faab6bcd640 1 -- 172.21.15.44:0/276366256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab015c250 msgr2=0x7faab015dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.723+0000 7faab6bcd640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab015c250 0x7faab015dea0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7faaa8036040 tx=0x7faaa8004a40 comp rx=0 tx=0).stop 2024-04-02T13:28:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.723+0000 7faab6bcd640 1 -- 172.21.15.44:0/276366256 shutdown_connections 2024-04-02T13:28:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.723+0000 7faab6bcd640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7faa9406c920 0x7faa9406ede0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.723+0000 7faab6bcd640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab015c250 0x7faab015dea0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.723+0000 7faab6bcd640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab00d08d0 0x7faab015d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.723+0000 7faab6bcd640 1 --2- 172.21.15.44:0/276366256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab00b3510 0x7faab006cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.723+0000 7faab6bcd640 1 -- 172.21.15.44:0/276366256 >> 172.21.15.44:0/276366256 conn(0x7faab0073850 msgr2=0x7faab0070ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.723+0000 7faab6bcd640 1 -- 172.21.15.44:0/276366256 shutdown_connections 2024-04-02T13:28:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.723+0000 7faab6bcd640 1 -- 172.21.15.44:0/276366256 wait complete. 2024-04-02T13:28:58.736 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:28:58.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.835+0000 7fa81ead9640 1 Processor -- start 2024-04-02T13:28:58.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.835+0000 7fa81ead9640 1 -- start start 2024-04-02T13:28:58.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa81ead9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa8180d0a30 0x7fa81806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa81ead9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa8180d2040 0x7fa8180d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa81ead9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8180d0000 con 0x7fa8180d0e10 2024-04-02T13:28:58.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa81ead9640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa81806cb00 con 0x7fa8180d2040 2024-04-02T13:28:58.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa81ead9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa81806c1d0 con 0x7fa8180d0a30 2024-04-02T13:28:58.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa81d076640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa8180d2040 0x7fa8180d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa817fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa8180d0e10 0x7fa8180cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54992/0 (socket says 172.21.15.44:54992) 2024-04-02T13:28:58.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa817fff640 1 -- 172.21.15.44:0/1473903737 learned_addr learned my addr 172.21.15.44:0/1473903737 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:28:58.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa81c875640 1 --2- 172.21.15.44:0/1473903737 >> v2:172.21.15.69:3300/0 conn(0x7fa8180d0a30 0x7fa81806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa81d076640 1 -- 172.21.15.44:0/1473903737 >> v2:172.21.15.69:3300/0 conn(0x7fa8180d0a30 msgr2=0x7fa81806cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:58.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa81d076640 1 --2- 172.21.15.44:0/1473903737 >> v2:172.21.15.69:3300/0 conn(0x7fa8180d0a30 0x7fa81806cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa81d076640 1 -- 172.21.15.44:0/1473903737 >> v1:172.21.15.44:6789/0 conn(0x7fa8180d0e10 legacy=0x7fa8180cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa81d076640 1 -- 172.21.15.44:0/1473903737 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81806c4b0 con 0x7fa8180d2040 2024-04-02T13:28:58.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa8177fe640 1 -- 172.21.15.44:0/1473903737 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1021487965 0 0) 0x7fa8180d0000 con 0x7fa8180d0e10 2024-04-02T13:28:58.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.836+0000 7fa81c875640 1 --2- 172.21.15.44:0/1473903737 >> v2:172.21.15.69:3300/0 conn(0x7fa8180d0a30 0x7fa81806cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:28:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.837+0000 7fa81d076640 1 --2- 172.21.15.44:0/1473903737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8180d2040 0x7fa8180d24f0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fa81806de20 tx=0x7fa80802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38faa202b95b1b9d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.837+0000 7fa8177fe640 1 -- 172.21.15.44:0/1473903737 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8080315a0 con 0x7fa8180d2040 2024-04-02T13:28:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.837+0000 7fa8177fe640 1 -- 172.21.15.44:0/1473903737 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa808037070 con 0x7fa8180d2040 2024-04-02T13:28:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.837+0000 7fa8177fe640 1 -- 172.21.15.44:0/1473903737 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808031960 con 0x7fa8180d2040 2024-04-02T13:28:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.838+0000 7fa81ead9640 1 -- 172.21.15.44:0/1473903737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8180d2040 msgr2=0x7fa8180d24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.838+0000 7fa81ead9640 1 --2- 172.21.15.44:0/1473903737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8180d2040 0x7fa8180d24f0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fa81806de20 tx=0x7fa80802b840 comp rx=0 tx=0).stop 2024-04-02T13:28:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.838+0000 7fa81ead9640 1 -- 172.21.15.44:0/1473903737 shutdown_connections 2024-04-02T13:28:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.838+0000 7fa81ead9640 1 --2- 172.21.15.44:0/1473903737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8180d2040 0x7fa8180d24f0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.838+0000 7fa81ead9640 1 --2- 172.21.15.44:0/1473903737 >> v2:172.21.15.69:3300/0 conn(0x7fa8180d0a30 0x7fa81806cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.838+0000 7fa81ead9640 1 -- 172.21.15.44:0/1473903737 >> 172.21.15.44:0/1473903737 conn(0x7fa81809d470 msgr2=0x7fa8180d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.838+0000 7fa81ead9640 1 -- 172.21.15.44:0/1473903737 shutdown_connections 2024-04-02T13:28:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.838+0000 7fa81ead9640 1 -- 172.21.15.44:0/1473903737 wait complete. 2024-04-02T13:28:58.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.839+0000 7fa81ead9640 1 Processor -- start 2024-04-02T13:28:58.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.839+0000 7fa81ead9640 1 -- start start 2024-04-02T13:28:58.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa81ead9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8180d0a30 0x7fa8180cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa81ead9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8180d0e10 0x7fa8180cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa81ead9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8180d2040 0x7fa8181e92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa81ead9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa81815e6f0 con 0x7fa8180d0e10 2024-04-02T13:28:58.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa81ead9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa8180d1270 con 0x7fa8180d0a30 2024-04-02T13:28:58.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa81ead9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa81815e870 con 0x7fa8180d2040 2024-04-02T13:28:58.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa817fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8180d0e10 0x7fa8180cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa817fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8180d0e10 0x7fa8180cb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38684/0 (socket says 172.21.15.44:38684) 2024-04-02T13:28:58.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa817fff640 1 -- 172.21.15.44:0/601081999 learned_addr learned my addr 172.21.15.44:0/601081999 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:58.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa81c875640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8180d0a30 0x7fa8180cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa81d076640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8180d2040 0x7fa8181e92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa817fff640 1 -- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8180d2040 msgr2=0x7fa8181e92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa817fff640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8180d2040 0x7fa8181e92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa817fff640 1 -- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8180d0a30 msgr2=0x7fa8180cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:58.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa817fff640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8180d0a30 0x7fa8180cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:58.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa817fff640 1 -- 172.21.15.44:0/601081999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8180b49e0 con 0x7fa8180d0e10 2024-04-02T13:28:58.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.840+0000 7fa817fff640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8180d0e10 0x7fa8180cb750 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00c410 tx=0x7fa80c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:58.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.841+0000 7fa8157fa640 1 -- 172.21.15.44:0/601081999 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c0074e0 con 0x7fa8180d0e10 2024-04-02T13:28:58.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.841+0000 7fa8157fa640 1 -- 172.21.15.44:0/601081999 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa80c0047e0 con 0x7fa8180d0e10 2024-04-02T13:28:58.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.841+0000 7fa81ead9640 1 -- 172.21.15.44:0/601081999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8180b4cc0 con 0x7fa8180d0e10 2024-04-02T13:28:58.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.841+0000 7fa8157fa640 1 -- 172.21.15.44:0/601081999 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c004bd0 con 0x7fa8180d0e10 2024-04-02T13:28:58.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.841+0000 7fa81ead9640 1 -- 172.21.15.44:0/601081999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8180b5280 con 0x7fa8180d0e10 2024-04-02T13:28:58.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.842+0000 7fa81ead9640 1 -- 172.21.15.44:0/601081999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8180684f0 con 0x7fa8180d0e10 2024-04-02T13:28:58.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.842+0000 7fa8157fa640 1 -- 172.21.15.44:0/601081999 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa80c011070 con 0x7fa8180d0e10 2024-04-02T13:28:58.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.843+0000 7fa8157fa640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa80406c8f0 0x7fa80406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:58.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.843+0000 7fa8157fa640 1 -- 172.21.15.44:0/601081999 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fa80c08b910 con 0x7fa8180d0e10 2024-04-02T13:28:58.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.843+0000 7fa81c875640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa80406c8f0 0x7fa80406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:58.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.843+0000 7fa81c875640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa80406c8f0 0x7fa80406edb0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa810009fd0 tx=0x7fa8100097f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:58.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:58.847+0000 7fa8157fa640 1 -- 172.21.15.44:0/601081999 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa80c059b20 con 0x7fa8180d0e10 2024-04-02T13:28:59.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.036+0000 7fa81ead9640 1 -- 172.21.15.44:0/601081999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa8180d17c0 con 0x7fa8180d0e10 2024-04-02T13:28:59.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.037+0000 7fa8157fa640 1 -- 172.21.15.44:0/601081999 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+17288 (secure 0 0 0) 0x7fa80c059420 con 0x7fa8180d0e10 2024-04-02T13:28:59.040 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:28:59.040 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":95,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:28:15.300549+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:26:41.300970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-02T13:26:41.979586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.24","mappings":[{"from":1,"to":5}]},{"pgid":"14.2c","mappings":[{"from":1,"to":2}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6833/2250279975":"2024-04-03T13:28:15.300517+0000","172.21.15.44:6832/2250279975":"2024-04-03T13:28:15.300517+0000","172.21.15.44:6833/719680934":"2024-04-03T13:26:55.589090+0000","172.21.15.44:6832/719680934":"2024-04-03T13:26:55.589090+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:28:59.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.039+0000 7fa81ead9640 1 -- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa80406c8f0 msgr2=0x7fa80406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:59.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.039+0000 7fa81ead9640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa80406c8f0 0x7fa80406edb0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa810009fd0 tx=0x7fa8100097f0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.039+0000 7fa81ead9640 1 -- 172.21.15.44:0/601081999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8180d0e10 msgr2=0x7fa8180cb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:59.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.039+0000 7fa81ead9640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8180d0e10 0x7fa8180cb750 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00c410 tx=0x7fa80c00c8d0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.039+0000 7fa81ead9640 1 -- 172.21.15.44:0/601081999 shutdown_connections 2024-04-02T13:28:59.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.039+0000 7fa81ead9640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa80406c8f0 0x7fa80406edb0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.039+0000 7fa81ead9640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8180d2040 0x7fa8181e92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.039+0000 7fa81ead9640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8180d0e10 0x7fa8180cb750 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.039+0000 7fa81ead9640 1 --2- 172.21.15.44:0/601081999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8180d0a30 0x7fa8180cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.039+0000 7fa81ead9640 1 -- 172.21.15.44:0/601081999 >> 172.21.15.44:0/601081999 conn(0x7fa81809d470 msgr2=0x7fa81815c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:59.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.039+0000 7fa81ead9640 1 -- 172.21.15.44:0/601081999 shutdown_connections 2024-04-02T13:28:59.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.039+0000 7fa81ead9640 1 -- 172.21.15.44:0/601081999 wait complete. 2024-04-02T13:28:59.053 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:28:59.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.153+0000 7fea5d58f640 1 Processor -- start 2024-04-02T13:28:59.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.153+0000 7fea5d58f640 1 -- start start 2024-04-02T13:28:59.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.154+0000 7fea5d58f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fea5815c050 0x7fea580ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:59.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.154+0000 7fea5d58f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea580b1ed0 0x7fea580ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:59.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.154+0000 7fea5d58f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea580ccb10 con 0x7fea5815c430 2024-04-02T13:28:59.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.154+0000 7fea5d58f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fea580b2480 con 0x7fea5815c050 2024-04-02T13:28:59.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.154+0000 7fea5d58f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fea580b2760 con 0x7fea580b1ed0 2024-04-02T13:28:59.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.154+0000 7fea56ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fea5815c050 0x7fea580ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:59.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.154+0000 7fea56ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fea5815c050 0x7fea580ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38686/0 (socket says 172.21.15.44:38686) 2024-04-02T13:28:59.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.154+0000 7fea56ffd640 1 -- 172.21.15.44:0/2647416132 learned_addr learned my addr 172.21.15.44:0/2647416132 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:59.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.154+0000 7fea56ffd640 1 -- 172.21.15.44:0/2647416132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea580b1ed0 msgr2=0x7fea580ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:59.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.154+0000 7fea577fe640 1 --2- 172.21.15.44:0/2647416132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea580b1ed0 0x7fea580ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:59.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.154+0000 7fea56ffd640 1 --2- 172.21.15.44:0/2647416132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea580b1ed0 0x7fea580ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.154+0000 7fea56ffd640 1 -- 172.21.15.44:0/2647416132 >> v1:172.21.15.69:6789/0 conn(0x7fea5815c430 legacy=0x7fea580b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:59.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.154+0000 7fea56ffd640 1 -- 172.21.15.44:0/2647416132 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea580b2a40 con 0x7fea5815c050 2024-04-02T13:28:59.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.155+0000 7fea56ffd640 1 --2- 172.21.15.44:0/2647416132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5815c050 0x7fea580ce850 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fea580b49c0 tx=0x7fea4802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8928c53682b63d7d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:59.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.155+0000 7fea55ffb640 1 -- 172.21.15.44:0/2647416132 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea48034070 con 0x7fea5815c050 2024-04-02T13:28:59.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.155+0000 7fea55ffb640 1 -- 172.21.15.44:0/2647416132 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea48030610 con 0x7fea5815c050 2024-04-02T13:28:59.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.155+0000 7fea55ffb640 1 -- 172.21.15.44:0/2647416132 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea480309e0 con 0x7fea5815c050 2024-04-02T13:28:59.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.155+0000 7fea5d58f640 1 -- 172.21.15.44:0/2647416132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5815c050 msgr2=0x7fea580ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:59.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.155+0000 7fea5d58f640 1 --2- 172.21.15.44:0/2647416132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5815c050 0x7fea580ce850 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fea580b49c0 tx=0x7fea4802b960 comp rx=0 tx=0).stop 2024-04-02T13:28:59.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.156+0000 7fea5d58f640 1 -- 172.21.15.44:0/2647416132 shutdown_connections 2024-04-02T13:28:59.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.156+0000 7fea5d58f640 1 --2- 172.21.15.44:0/2647416132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea580b1ed0 0x7fea580ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.156+0000 7fea5d58f640 1 --2- 172.21.15.44:0/2647416132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5815c050 0x7fea580ce850 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.156+0000 7fea5d58f640 1 -- 172.21.15.44:0/2647416132 >> 172.21.15.44:0/2647416132 conn(0x7fea5809aa20 msgr2=0x7fea5809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:59.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.156+0000 7fea5d58f640 1 -- 172.21.15.44:0/2647416132 shutdown_connections 2024-04-02T13:28:59.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.156+0000 7fea5d58f640 1 -- 172.21.15.44:0/2647416132 wait complete. 2024-04-02T13:28:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.157+0000 7fea5d58f640 1 Processor -- start 2024-04-02T13:28:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.157+0000 7fea5d58f640 1 -- start start 2024-04-02T13:28:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea5d58f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea580b1ed0 0x7fea58150470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea5d58f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea5815c050 0x7fea580b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea5d58f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5815c430 0x7fea580b5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea5d58f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fea580cd140 con 0x7fea5815c430 2024-04-02T13:28:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea5d58f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fea580ccfc0 con 0x7fea580b1ed0 2024-04-02T13:28:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea5d58f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fea580cd2c0 con 0x7fea5815c050 2024-04-02T13:28:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5815c430 0x7fea580b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5815c430 0x7fea580b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38690/0 (socket says 172.21.15.44:38690) 2024-04-02T13:28:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea577fe640 1 -- 172.21.15.44:0/3010288744 learned_addr learned my addr 172.21.15.44:0/3010288744 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea56ffd640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea580b1ed0 0x7fea58150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea567fc640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea5815c050 0x7fea580b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea577fe640 1 -- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea5815c050 msgr2=0x7fea580b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea577fe640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea5815c050 0x7fea580b0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea577fe640 1 -- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea580b1ed0 msgr2=0x7fea58150470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea577fe640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea580b1ed0 0x7fea58150470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea577fe640 1 -- 172.21.15.44:0/3010288744 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea580aecb0 con 0x7fea5815c430 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea56ffd640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea580b1ed0 0x7fea58150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea577fe640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5815c430 0x7fea580b5960 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fea5000cd70 tx=0x7fea5000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea3f7fe640 1 -- 172.21.15.44:0/3010288744 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea50003b90 con 0x7fea5815c430 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea3f7fe640 1 -- 172.21.15.44:0/3010288744 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea50003d30 con 0x7fea5815c430 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea5d58f640 1 -- 172.21.15.44:0/3010288744 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea580aef90 con 0x7fea5815c430 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea3f7fe640 1 -- 172.21.15.44:0/3010288744 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea50004240 con 0x7fea5815c430 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.158+0000 7fea5d58f640 1 -- 172.21.15.44:0/3010288744 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea580af550 con 0x7fea5815c430 2024-04-02T13:28:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.159+0000 7fea5d58f640 1 -- 172.21.15.44:0/3010288744 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea580684f0 con 0x7fea5815c430 2024-04-02T13:28:59.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.160+0000 7fea3f7fe640 1 -- 172.21.15.44:0/3010288744 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fea500043e0 con 0x7fea5815c430 2024-04-02T13:28:59.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.160+0000 7fea3f7fe640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fea4406c8f0 0x7fea4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:59.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.160+0000 7fea3f7fe640 1 -- 172.21.15.44:0/3010288744 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6389+0+0 (secure 0 0 0) 0x7fea5008c930 con 0x7fea5815c430 2024-04-02T13:28:59.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.160+0000 7fea56ffd640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fea4406c8f0 0x7fea4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:59.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.161+0000 7fea56ffd640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fea4406c8f0 0x7fea4406edb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fea48036000 tx=0x7fea4803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:59.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.164+0000 7fea3f7fe640 1 -- 172.21.15.44:0/3010288744 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fea5001d340 con 0x7fea5815c430 2024-04-02T13:28:59.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.390+0000 7fea5d58f640 1 -- 172.21.15.44:0/3010288744 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fea58067d70 con 0x7fea5815c430 2024-04-02T13:28:59.393 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:28:59.391+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:28:59.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.392+0000 7fea3f7fe640 1 -- 172.21.15.44:0/3010288744 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v75) ==== 161+0+0 (secure 0 0 0) 0x7fea5005aab0 con 0x7fea5815c430 2024-04-02T13:28:59.394 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:28:59.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.394+0000 7fea5d58f640 1 -- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fea4406c8f0 msgr2=0x7fea4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:59.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.394+0000 7fea5d58f640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fea4406c8f0 0x7fea4406edb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fea48036000 tx=0x7fea4803b040 comp rx=0 tx=0).stop 2024-04-02T13:28:59.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.394+0000 7fea5d58f640 1 -- 172.21.15.44:0/3010288744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5815c430 msgr2=0x7fea580b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:59.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.394+0000 7fea5d58f640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5815c430 0x7fea580b5960 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fea5000cd70 tx=0x7fea5000a660 comp rx=0 tx=0).stop 2024-04-02T13:28:59.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.394+0000 7fea5d58f640 1 -- 172.21.15.44:0/3010288744 shutdown_connections 2024-04-02T13:28:59.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.394+0000 7fea5d58f640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fea4406c8f0 0x7fea4406edb0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.394+0000 7fea5d58f640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5815c430 0x7fea580b5960 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.394+0000 7fea5d58f640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea5815c050 0x7fea580b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.394+0000 7fea5d58f640 1 --2- 172.21.15.44:0/3010288744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea580b1ed0 0x7fea58150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.394+0000 7fea5d58f640 1 -- 172.21.15.44:0/3010288744 >> 172.21.15.44:0/3010288744 conn(0x7fea5809aa20 msgr2=0x7fea580b31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:59.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.394+0000 7fea5d58f640 1 -- 172.21.15.44:0/3010288744 shutdown_connections 2024-04-02T13:28:59.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.394+0000 7fea5d58f640 1 -- 172.21.15.44:0/3010288744 wait complete. 2024-04-02T13:28:59.398 INFO:tasks.ceph.mds.a.smithi044.stderr: -2168> 2024-04-02T13:28:22.310+0000 7f856a5a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:28:59.399 INFO:tasks.ceph.mds.a.smithi044.stderr: -1923> 2024-04-02T13:28:22.460+0000 7f856659a640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [...2,head] /dir/file auth v1 ap=1 DIRTYPARENT s=0 nl=2 n() | dirtyparent=1 authpin=1 scrubqueue=0 0x55788b8de100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(14)0x10000000001:[<0x10000000000/file v6>,<0x1/dir v13>]//[]","memoryvalue":"(14)0x10000000001:[<0x10000000000/file v1>,<0x1/dir v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-02T13:28:59.408 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T13:28:59.424 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:28:59.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.511+0000 7f75fb332640 1 Processor -- start 2024-04-02T13:28:59.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.511+0000 7f75fb332640 1 -- start start 2024-04-02T13:28:59.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.512+0000 7f75fb332640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f75f40d2e00 0x7f75f40b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:59.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.512+0000 7f75fb332640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f75f40cfdc0 0x7f75f40d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:59.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.512+0000 7f75fb332640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75f406de80 con 0x7f75f406daa0 2024-04-02T13:28:59.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.512+0000 7f75fb332640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f75f40cf930 con 0x7f75f40d2e00 2024-04-02T13:28:59.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.512+0000 7f75fb332640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f75f40cf000 con 0x7f75f40cfdc0 2024-04-02T13:28:59.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.512+0000 7f75f88cd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f75f40d2e00 0x7f75f40b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:59.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.512+0000 7f75f88cd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f75f40d2e00 0x7f75f40b35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38706/0 (socket says 172.21.15.44:38706) 2024-04-02T13:28:59.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.512+0000 7f75f88cd640 1 -- 172.21.15.44:0/458579924 learned_addr learned my addr 172.21.15.44:0/458579924 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:59.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.513+0000 7f75f98cf640 1 --2- 172.21.15.44:0/458579924 >> v2:172.21.15.69:3300/0 conn(0x7f75f40cfdc0 0x7f75f40d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:59.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.513+0000 7f75f88cd640 1 -- 172.21.15.44:0/458579924 >> v2:172.21.15.69:3300/0 conn(0x7f75f40cfdc0 msgr2=0x7f75f40d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:59.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.513+0000 7f75f88cd640 1 --2- 172.21.15.44:0/458579924 >> v2:172.21.15.69:3300/0 conn(0x7f75f40cfdc0 0x7f75f40d11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.513+0000 7f75f88cd640 1 -- 172.21.15.44:0/458579924 >> v1:172.21.15.69:6789/0 conn(0x7f75f406daa0 legacy=0x7f75f40d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:28:59.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.513+0000 7f75f88cd640 1 -- 172.21.15.44:0/458579924 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f40cf2e0 con 0x7f75f40d2e00 2024-04-02T13:28:59.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.513+0000 7f75f98cf640 1 --2- 172.21.15.44:0/458579924 >> v2:172.21.15.69:3300/0 conn(0x7f75f40cfdc0 0x7f75f40d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:28:59.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.513+0000 7f75f88cd640 1 --2- 172.21.15.44:0/458579924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75f40d2e00 0x7f75f40b35b0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f75f4163100 tx=0x7f75e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e3324056664fe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:59.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.513+0000 7f75ebfff640 1 -- 172.21.15.44:0/458579924 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4035020 con 0x7f75f40d2e00 2024-04-02T13:28:59.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.514+0000 7f75ebfff640 1 -- 172.21.15.44:0/458579924 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e4031870 con 0x7f75f40d2e00 2024-04-02T13:28:59.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.514+0000 7f75ebfff640 1 -- 172.21.15.44:0/458579924 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4031c40 con 0x7f75f40d2e00 2024-04-02T13:28:59.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.514+0000 7f75fb332640 1 -- 172.21.15.44:0/458579924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75f40d2e00 msgr2=0x7f75f40b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:59.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.514+0000 7f75fb332640 1 --2- 172.21.15.44:0/458579924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75f40d2e00 0x7f75f40b35b0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f75f4163100 tx=0x7f75e402b910 comp rx=0 tx=0).stop 2024-04-02T13:28:59.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.514+0000 7f75fb332640 1 -- 172.21.15.44:0/458579924 shutdown_connections 2024-04-02T13:28:59.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.514+0000 7f75fb332640 1 --2- 172.21.15.44:0/458579924 >> v2:172.21.15.69:3300/0 conn(0x7f75f40cfdc0 0x7f75f40d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.514+0000 7f75fb332640 1 --2- 172.21.15.44:0/458579924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75f40d2e00 0x7f75f40b35b0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.514+0000 7f75fb332640 1 -- 172.21.15.44:0/458579924 >> 172.21.15.44:0/458579924 conn(0x7f75f409d430 msgr2=0x7f75f40d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:28:59.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.514+0000 7f75fb332640 1 -- 172.21.15.44:0/458579924 shutdown_connections 2024-04-02T13:28:59.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.514+0000 7f75fb332640 1 -- 172.21.15.44:0/458579924 wait complete. 2024-04-02T13:28:59.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.515+0000 7f75fb332640 1 Processor -- start 2024-04-02T13:28:59.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.515+0000 7f75fb332640 1 -- start start 2024-04-02T13:28:59.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75fb332640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75f406daa0 0x7f75f41e9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:59.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75fb332640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75f40cfdc0 0x7f75f41f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:59.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75fb332640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75f40d2e00 0x7f75f41594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:59.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75fb332640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f75f40ceda0 con 0x7f75f40cfdc0 2024-04-02T13:28:59.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75fb332640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f75f40cec20 con 0x7f75f406daa0 2024-04-02T13:28:59.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75fb332640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f75f415dd60 con 0x7f75f40d2e00 2024-04-02T13:28:59.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75f88cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75f40cfdc0 0x7f75f41f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:59.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75f88cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75f40cfdc0 0x7f75f41f5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38710/0 (socket says 172.21.15.44:38710) 2024-04-02T13:28:59.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75f88cd640 1 -- 172.21.15.44:0/3180860518 learned_addr learned my addr 172.21.15.44:0/3180860518 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:28:59.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75f90ce640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75f406daa0 0x7f75f41e9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:59.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75f98cf640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75f40d2e00 0x7f75f41594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:59.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75f88cd640 1 -- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75f40d2e00 msgr2=0x7f75f41594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:59.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75f88cd640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75f40d2e00 0x7f75f41594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75f88cd640 1 -- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75f406daa0 msgr2=0x7f75f41e9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:28:59.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75f88cd640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75f406daa0 0x7f75f41e9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:28:59.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.516+0000 7f75f88cd640 1 -- 172.21.15.44:0/3180860518 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f41f57d0 con 0x7f75f40cfdc0 2024-04-02T13:28:59.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.517+0000 7f75f88cd640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75f40cfdc0 0x7f75f41f5290 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f75e402b950 tx=0x7f75e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:59.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.517+0000 7f75e9ffb640 1 -- 172.21.15.44:0/3180860518 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4040070 con 0x7f75f40cfdc0 2024-04-02T13:28:59.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.517+0000 7f75e9ffb640 1 -- 172.21.15.44:0/3180860518 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e403a420 con 0x7f75f40cfdc0 2024-04-02T13:28:59.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.517+0000 7f75fb332640 1 -- 172.21.15.44:0/3180860518 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f41e83e0 con 0x7f75f40cfdc0 2024-04-02T13:28:59.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.517+0000 7f75e9ffb640 1 -- 172.21.15.44:0/3180860518 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4035040 con 0x7f75f40cfdc0 2024-04-02T13:28:59.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.517+0000 7f75fb332640 1 -- 172.21.15.44:0/3180860518 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f41e89a0 con 0x7f75f40cfdc0 2024-04-02T13:28:59.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.518+0000 7f75fb332640 1 -- 172.21.15.44:0/3180860518 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f40684f0 con 0x7f75f40cfdc0 2024-04-02T13:28:59.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.519+0000 7f75e9ffb640 1 -- 172.21.15.44:0/3180860518 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f75e404b060 con 0x7f75f40cfdc0 2024-04-02T13:28:59.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.519+0000 7f75e9ffb640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75d806c8f0 0x7f75d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:28:59.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.519+0000 7f75e9ffb640 1 -- 172.21.15.44:0/3180860518 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6475+0+0 (secure 0 0 0) 0x7f75e40adc60 con 0x7f75f40cfdc0 2024-04-02T13:28:59.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.519+0000 7f75f90ce640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75d806c8f0 0x7f75d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:28:59.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.520+0000 7f75f90ce640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75d806c8f0 0x7f75d806edb0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f75f4161c80 tx=0x7f75ec007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:28:59.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.524+0000 7f75e9ffb640 1 -- 172.21.15.44:0/3180860518 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f75f41f57d0 con 0x7f75f40cfdc0 2024-04-02T13:28:59.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:28:59.751+0000 7f75fb332640 1 -- 172.21.15.44:0/3180860518 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f75f40d18c0 con 0x7f75f40cfdc0 2024-04-02T13:29:00.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.393+0000 7f75e9ffb640 1 -- 172.21.15.44:0/3180860518 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v76) ==== 104+0+0 (secure 0 0 0) 0x7f75e407b7d0 con 0x7f75f40cfdc0 2024-04-02T13:29:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.396+0000 7f75fb332640 1 -- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75d806c8f0 msgr2=0x7f75d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.396+0000 7f75fb332640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75d806c8f0 0x7f75d806edb0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f75f4161c80 tx=0x7f75ec007e30 comp rx=0 tx=0).stop 2024-04-02T13:29:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.396+0000 7f75fb332640 1 -- 172.21.15.44:0/3180860518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75f40cfdc0 msgr2=0x7f75f41f5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.396+0000 7f75fb332640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75f40cfdc0 0x7f75f41f5290 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f75e402b950 tx=0x7f75e40046c0 comp rx=0 tx=0).stop 2024-04-02T13:29:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.396+0000 7f75fb332640 1 -- 172.21.15.44:0/3180860518 shutdown_connections 2024-04-02T13:29:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.396+0000 7f75fb332640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75d806c8f0 0x7f75d806edb0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.396+0000 7f75fb332640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75f40d2e00 0x7f75f41594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.396+0000 7f75fb332640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75f40cfdc0 0x7f75f41f5290 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.396+0000 7f75fb332640 1 --2- 172.21.15.44:0/3180860518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75f406daa0 0x7f75f41e9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.396+0000 7f75fb332640 1 -- 172.21.15.44:0/3180860518 >> 172.21.15.44:0/3180860518 conn(0x7f75f409d430 msgr2=0x7f75f40d0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:00.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.396+0000 7f75fb332640 1 -- 172.21.15.44:0/3180860518 shutdown_connections 2024-04-02T13:29:00.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.396+0000 7f75fb332640 1 -- 172.21.15.44:0/3180860518 wait complete. 2024-04-02T13:29:00.412 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T13:29:00.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.562+0000 7f05e67e2640 1 Processor -- start 2024-04-02T13:29:00.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.563+0000 7f05e67e2640 1 -- start start 2024-04-02T13:29:00.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.563+0000 7f05e67e2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f05e015e1d0 0x7f05e00d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:00.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.563+0000 7f05e67e2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05e00cec60 0x7f05e00d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:00.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.563+0000 7f05e67e2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f05e015e5b0 0x7f05e00d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:00.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.563+0000 7f05e67e2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f05e00cfe00 con 0x7f05e015e1d0 2024-04-02T13:29:00.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.563+0000 7f05e67e2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f05e00d1640 con 0x7f05e015e5b0 2024-04-02T13:29:00.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.563+0000 7f05e67e2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f05e00cf1f0 con 0x7f05e00cec60 2024-04-02T13:29:00.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.563+0000 7f05dffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f05e015e1d0 0x7f05e00d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:00.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.563+0000 7f05dffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f05e015e1d0 0x7f05e00d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38718/0 (socket says 172.21.15.44:38718) 2024-04-02T13:29:00.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.563+0000 7f05dffff640 1 -- 172.21.15.44:0/1116216729 learned_addr learned my addr 172.21.15.44:0/1116216729 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:00.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.564+0000 7f05df7fe640 1 --2- 172.21.15.44:0/1116216729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05e00cec60 0x7f05e00d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:00.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.564+0000 7f05e4d7f640 1 --2- 172.21.15.44:0/1116216729 >> v2:172.21.15.69:3300/0 conn(0x7f05e015e5b0 0x7f05e00d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:00.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.564+0000 7f05dffff640 1 -- 172.21.15.44:0/1116216729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05e00cec60 msgr2=0x7f05e00d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:00.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.564+0000 7f05dffff640 1 --2- 172.21.15.44:0/1116216729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05e00cec60 0x7f05e00d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:00.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.564+0000 7f05dffff640 1 -- 172.21.15.44:0/1116216729 >> v2:172.21.15.69:3300/0 conn(0x7f05e015e5b0 msgr2=0x7f05e00d2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:00.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.564+0000 7f05dffff640 1 --2- 172.21.15.44:0/1116216729 >> v2:172.21.15.69:3300/0 conn(0x7f05e015e5b0 0x7f05e00d2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:00.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.564+0000 7f05dffff640 1 -- 172.21.15.44:0/1116216729 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e00cf4d0 con 0x7f05e015e1d0 2024-04-02T13:29:00.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.564+0000 7f05dffff640 1 --2- 172.21.15.44:0/1116216729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05e015e1d0 0x7f05e00d1100 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f05e00cb160 tx=0x7f05d002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197f7c089b52d169 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:00.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.565+0000 7f05deffd640 1 -- 172.21.15.44:0/1116216729 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d002e070 con 0x7f05e015e1d0 2024-04-02T13:29:00.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.565+0000 7f05deffd640 1 -- 172.21.15.44:0/1116216729 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05d0037070 con 0x7f05e015e1d0 2024-04-02T13:29:00.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.565+0000 7f05deffd640 1 -- 172.21.15.44:0/1116216729 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d0031a00 con 0x7f05e015e1d0 2024-04-02T13:29:00.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.565+0000 7f05e67e2640 1 -- 172.21.15.44:0/1116216729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05e015e1d0 msgr2=0x7f05e00d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:00.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.565+0000 7f05e67e2640 1 --2- 172.21.15.44:0/1116216729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05e015e1d0 0x7f05e00d1100 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f05e00cb160 tx=0x7f05d002b880 comp rx=0 tx=0).stop 2024-04-02T13:29:00.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.565+0000 7f05e67e2640 1 -- 172.21.15.44:0/1116216729 shutdown_connections 2024-04-02T13:29:00.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.565+0000 7f05e67e2640 1 --2- 172.21.15.44:0/1116216729 >> v2:172.21.15.69:3300/0 conn(0x7f05e015e5b0 0x7f05e00d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:00.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.565+0000 7f05e67e2640 1 --2- 172.21.15.44:0/1116216729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05e00cec60 0x7f05e00d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:00.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.565+0000 7f05e67e2640 1 --2- 172.21.15.44:0/1116216729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05e015e1d0 0x7f05e00d1100 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:00.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.565+0000 7f05e67e2640 1 -- 172.21.15.44:0/1116216729 >> 172.21.15.44:0/1116216729 conn(0x7f05e009d3b0 msgr2=0x7f05e00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:00.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.565+0000 7f05e67e2640 1 -- 172.21.15.44:0/1116216729 shutdown_connections 2024-04-02T13:29:00.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.565+0000 7f05e67e2640 1 -- 172.21.15.44:0/1116216729 wait complete. 2024-04-02T13:29:00.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.566+0000 7f05e67e2640 1 Processor -- start 2024-04-02T13:29:00.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.567+0000 7f05e67e2640 1 -- start start 2024-04-02T13:29:00.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.567+0000 7f05e67e2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05e00cec60 0x7f05e00cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:00.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.567+0000 7f05e67e2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05e015e1d0 0x7f05e00cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:00.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.567+0000 7f05e67e2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e015e5b0 0x7f05e00c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:00.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.567+0000 7f05e67e2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f05e00b4ce0 con 0x7f05e00cec60 2024-04-02T13:29:00.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.567+0000 7f05e67e2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f05e00cfa90 con 0x7f05e015e5b0 2024-04-02T13:29:00.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.567+0000 7f05e67e2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f05e00b4e60 con 0x7f05e015e1d0 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.567+0000 7f05dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05e00cec60 0x7f05e00cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.567+0000 7f05dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05e00cec60 0x7f05e00cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38728/0 (socket says 172.21.15.44:38728) 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05dffff640 1 -- 172.21.15.44:0/4007149800 learned_addr learned my addr 172.21.15.44:0/4007149800 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05e4d7f640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e015e5b0 0x7f05e00c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05df7fe640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05e015e1d0 0x7f05e00cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05dffff640 1 -- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05e015e1d0 msgr2=0x7f05e00cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05dffff640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05e015e1d0 0x7f05e00cd910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05dffff640 1 -- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e015e5b0 msgr2=0x7f05e00c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05dffff640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e015e5b0 0x7f05e00c9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05dffff640 1 -- 172.21.15.44:0/4007149800 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e00cb8b0 con 0x7f05e00cec60 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05e4d7f640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e015e5b0 0x7f05e00c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05dffff640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05e00cec60 0x7f05e00cb1b0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f05d0000c00 tx=0x7f05d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05dcff9640 1 -- 172.21.15.44:0/4007149800 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d0032670 con 0x7f05e00cec60 2024-04-02T13:29:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05dcff9640 1 -- 172.21.15.44:0/4007149800 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05d0032850 con 0x7f05e00cec60 2024-04-02T13:29:00.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05dcff9640 1 -- 172.21.15.44:0/4007149800 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d0032c40 con 0x7f05e00cec60 2024-04-02T13:29:00.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05e67e2640 1 -- 172.21.15.44:0/4007149800 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05e00cbb90 con 0x7f05e00cec60 2024-04-02T13:29:00.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.568+0000 7f05e67e2640 1 -- 172.21.15.44:0/4007149800 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05e00cc150 con 0x7f05e00cec60 2024-04-02T13:29:00.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.569+0000 7f05e67e2640 1 -- 172.21.15.44:0/4007149800 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05e00684f0 con 0x7f05e00cec60 2024-04-02T13:29:00.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.570+0000 7f05dcff9640 1 -- 172.21.15.44:0/4007149800 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f05d004c060 con 0x7f05e00cec60 2024-04-02T13:29:00.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.570+0000 7f05dcff9640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f05cc06c8f0 0x7f05cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:00.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.570+0000 7f05dcff9640 1 -- 172.21.15.44:0/4007149800 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6475+0+0 (secure 0 0 0) 0x7f05d00ad900 con 0x7f05e00cec60 2024-04-02T13:29:00.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.570+0000 7f05df7fe640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f05cc06c8f0 0x7f05cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:00.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.571+0000 7f05df7fe640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f05cc06c8f0 0x7f05cc06edb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f05e00ca8c0 tx=0x7f05d4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:00.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.574+0000 7f05dcff9640 1 -- 172.21.15.44:0/4007149800 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f05d0040070 con 0x7f05e00cec60 2024-04-02T13:29:00.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:00.759+0000 7f05e67e2640 1 -- 172.21.15.44:0/4007149800 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f05e00d47f0 con 0x7f05e00cec60 2024-04-02T13:29:01.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.496+0000 7f05dcff9640 1 -- 172.21.15.44:0/4007149800 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v97) ==== 181+0+0 (secure 0 0 0) 0x7f05d007bb50 con 0x7f05e00cec60 2024-04-02T13:29:01.499 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' removed 2024-04-02T13:29:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.500+0000 7f05e67e2640 1 -- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f05cc06c8f0 msgr2=0x7f05cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.500+0000 7f05e67e2640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f05cc06c8f0 0x7f05cc06edb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f05e00ca8c0 tx=0x7f05d4008040 comp rx=0 tx=0).stop 2024-04-02T13:29:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.500+0000 7f05e67e2640 1 -- 172.21.15.44:0/4007149800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05e00cec60 msgr2=0x7f05e00cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.500+0000 7f05e67e2640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05e00cec60 0x7f05e00cb1b0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f05d0000c00 tx=0x7f05d00046c0 comp rx=0 tx=0).stop 2024-04-02T13:29:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.500+0000 7f05e67e2640 1 -- 172.21.15.44:0/4007149800 shutdown_connections 2024-04-02T13:29:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.500+0000 7f05e67e2640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f05cc06c8f0 0x7f05cc06edb0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.500+0000 7f05e67e2640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e015e5b0 0x7f05e00c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.500+0000 7f05e67e2640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05e015e1d0 0x7f05e00cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.500+0000 7f05e67e2640 1 --2- 172.21.15.44:0/4007149800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05e00cec60 0x7f05e00cb1b0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.500+0000 7f05e67e2640 1 -- 172.21.15.44:0/4007149800 >> 172.21.15.44:0/4007149800 conn(0x7f05e009d3b0 msgr2=0x7f05e00d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.501+0000 7f05e67e2640 1 -- 172.21.15.44:0/4007149800 shutdown_connections 2024-04-02T13:29:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.501+0000 7f05e67e2640 1 -- 172.21.15.44:0/4007149800 wait complete. 2024-04-02T13:29:01.518 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T13:29:01.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.628+0000 7f0b6c596640 1 Processor -- start 2024-04-02T13:29:01.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.628+0000 7f0b6c596640 1 -- start start 2024-04-02T13:29:01.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.628+0000 7f0b6c596640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0b640d2dd0 0x7f0b640b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:01.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.628+0000 7f0b6c596640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b640cf1f0 con 0x7f0b640cec60 2024-04-02T13:29:01.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b6c596640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b640cf4d0 con 0x7f0b6415e1d0 2024-04-02T13:29:01.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b6c596640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0b640cfe00 con 0x7f0b640d2dd0 2024-04-02T13:29:01.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69b31640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0b640cec60 0x7f0b640d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:55008/0 (socket says 172.21.15.44:55008) 2024-04-02T13:29:01.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69b31640 1 -- 172.21.15.44:0/4273795663 learned_addr learned my addr 172.21.15.44:0/4273795663 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:01.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b6ab33640 1 --2- 172.21.15.44:0/4273795663 >> v2:172.21.15.69:3300/0 conn(0x7f0b640d2dd0 0x7f0b640b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:01.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2722775396 0 0) 0x7f0b640cf1f0 con 0x7f0b640cec60 2024-04-02T13:29:01.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b4c003520 con 0x7f0b640cec60 2024-04-02T13:29:01.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 188214887 0 0) 0x7f0b4c003520 con 0x7f0b640cec60 2024-04-02T13:29:01.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 >> v2:172.21.15.69:3300/0 conn(0x7f0b640d2dd0 msgr2=0x7f0b640b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 --2- 172.21.15.44:0/4273795663 >> v2:172.21.15.69:3300/0 conn(0x7f0b640d2dd0 0x7f0b640b4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 >> v1:172.21.15.69:6789/0 conn(0x7f0b6415e1d0 legacy=0x7f0b640d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b640cf7b0 con 0x7f0b640cec60 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f0b58001670 con 0x7f0b640cec60 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 >> v1:172.21.15.44:6789/0 conn(0x7f0b640cec60 legacy=0x7f0b640d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 --2- 172.21.15.44:0/4273795663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b4c0039b0 0x7f0b4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 --2- 172.21.15.44:0/4273795663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b4c004800 0x7f0b4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 --2- 172.21.15.44:0/4273795663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b4c0051c0 0x7f0b4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0b640cfe00 con 0x7f0b4c0051c0 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4c00c160 con 0x7f0b4c0039b0 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0b4c00c360 con 0x7f0b4c004800 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b6ab33640 1 --2- 172.21.15.44:0/4273795663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b4c0051c0 0x7f0b4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.629+0000 7f0b6a332640 1 --2- 172.21.15.44:0/4273795663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b4c0039b0 0x7f0b4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.630+0000 7f0b69b31640 1 --2- 172.21.15.44:0/4273795663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b4c004800 0x7f0b4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.630+0000 7f0b6ab33640 1 -- 172.21.15.44:0/4273795663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b4c004800 msgr2=0x7f0b4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.630+0000 7f0b6ab33640 1 --2- 172.21.15.44:0/4273795663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b4c004800 0x7f0b4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.630+0000 7f0b6ab33640 1 -- 172.21.15.44:0/4273795663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b4c0039b0 msgr2=0x7f0b4c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.630+0000 7f0b6ab33640 1 --2- 172.21.15.44:0/4273795663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b4c0039b0 0x7f0b4c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.630+0000 7f0b6ab33640 1 -- 172.21.15.44:0/4273795663 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4c003520 con 0x7f0b4c0051c0 2024-04-02T13:29:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.630+0000 7f0b69b31640 1 --2- 172.21.15.44:0/4273795663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b4c004800 0x7f0b4c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.630+0000 7f0b6ab33640 1 -- 172.21.15.44:0/4273795663 reap_dead start 2024-04-02T13:29:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.630+0000 7f0b6ab33640 1 --2- 172.21.15.44:0/4273795663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b4c0051c0 0x7f0b4c00b9c0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f0b54009f90 tx=0x7f0b4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c1d06739aead81 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.631+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b54009c10 con 0x7f0b4c0051c0 2024-04-02T13:29:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.631+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5402cc20 con 0x7f0b4c0051c0 2024-04-02T13:29:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.631+0000 7f0b69330640 1 -- 172.21.15.44:0/4273795663 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b54003990 con 0x7f0b4c0051c0 2024-04-02T13:29:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.631+0000 7f0b6c596640 1 -- 172.21.15.44:0/4273795663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b4c0051c0 msgr2=0x7f0b4c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.631+0000 7f0b6c596640 1 --2- 172.21.15.44:0/4273795663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b4c0051c0 0x7f0b4c00b9c0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f0b54009f90 tx=0x7f0b4c005830 comp rx=0 tx=0).stop 2024-04-02T13:29:01.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.631+0000 7f0b6c596640 1 -- 172.21.15.44:0/4273795663 shutdown_connections 2024-04-02T13:29:01.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.631+0000 7f0b6c596640 1 --2- 172.21.15.44:0/4273795663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b4c0051c0 0x7f0b4c00b9c0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:01.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.631+0000 7f0b6c596640 1 -- 172.21.15.44:0/4273795663 >> 172.21.15.44:0/4273795663 conn(0x7f0b6409d3b0 msgr2=0x7f0b640d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:01.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.631+0000 7f0b6c596640 1 -- 172.21.15.44:0/4273795663 shutdown_connections 2024-04-02T13:29:01.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.631+0000 7f0b6c596640 1 -- 172.21.15.44:0/4273795663 wait complete. 2024-04-02T13:29:01.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.632+0000 7f0b6c596640 1 Processor -- start 2024-04-02T13:29:01.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.632+0000 7f0b6c596640 1 -- start start 2024-04-02T13:29:01.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.633+0000 7f0b6c596640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b4c0051c0 0x7f0b641505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:01.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.633+0000 7f0b6c596640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b641f4010 0x7f0b640a1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:01.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.633+0000 7f0b6c596640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b6415f540 0x7f0b6415c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:01.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.633+0000 7f0b6c596640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6415e730 con 0x7f0b641f4010 2024-04-02T13:29:01.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.633+0000 7f0b6c596640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6415e5b0 con 0x7f0b4c0051c0 2024-04-02T13:29:01.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.633+0000 7f0b6c596640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0b6415e8b0 con 0x7f0b6415f540 2024-04-02T13:29:01.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.633+0000 7f0b69b31640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b641f4010 0x7f0b640a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:01.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.633+0000 7f0b69b31640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b641f4010 0x7f0b640a1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38736/0 (socket says 172.21.15.44:38736) 2024-04-02T13:29:01.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.633+0000 7f0b69b31640 1 -- 172.21.15.44:0/4063056392 learned_addr learned my addr 172.21.15.44:0/4063056392 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:01.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.633+0000 7f0b6ab33640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b6415f540 0x7f0b6415c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:01.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.633+0000 7f0b6a332640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b4c0051c0 0x7f0b641505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:01.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.634+0000 7f0b69b31640 1 -- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b6415f540 msgr2=0x7f0b6415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:01.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.634+0000 7f0b69b31640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b6415f540 0x7f0b6415c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:01.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.634+0000 7f0b69b31640 1 -- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b4c0051c0 msgr2=0x7f0b641505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:01.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.634+0000 7f0b69b31640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b4c0051c0 0x7f0b641505d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:01.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.634+0000 7f0b69b31640 1 -- 172.21.15.44:0/4063056392 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6415f9f0 con 0x7f0b641f4010 2024-04-02T13:29:01.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.634+0000 7f0b6a332640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b4c0051c0 0x7f0b641505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:29:01.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.634+0000 7f0b69b31640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b641f4010 0x7f0b640a1400 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800e780 tx=0x7f0b5800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:01.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.634+0000 7f0b62ffd640 1 -- 172.21.15.44:0/4063056392 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5800c870 con 0x7f0b641f4010 2024-04-02T13:29:01.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.634+0000 7f0b62ffd640 1 -- 172.21.15.44:0/4063056392 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b58018070 con 0x7f0b641f4010 2024-04-02T13:29:01.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.634+0000 7f0b62ffd640 1 -- 172.21.15.44:0/4063056392 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5800cc50 con 0x7f0b641f4010 2024-04-02T13:29:01.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.634+0000 7f0b6c596640 1 -- 172.21.15.44:0/4063056392 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b640a1b50 con 0x7f0b641f4010 2024-04-02T13:29:01.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.634+0000 7f0b6c596640 1 -- 172.21.15.44:0/4063056392 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b641600b0 con 0x7f0b641f4010 2024-04-02T13:29:01.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.635+0000 7f0b6c596640 1 -- 172.21.15.44:0/4063056392 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b640ce8b0 con 0x7f0b641f4010 2024-04-02T13:29:01.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.635+0000 7f0b62ffd640 1 -- 172.21.15.44:0/4063056392 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0b5801d460 con 0x7f0b641f4010 2024-04-02T13:29:01.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.636+0000 7f0b62ffd640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b5006c8f0 0x7f0b5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:01.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.636+0000 7f0b62ffd640 1 -- 172.21.15.44:0/4063056392 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6037+0+0 (secure 0 0 0) 0x7f0b5808b9f0 con 0x7f0b641f4010 2024-04-02T13:29:01.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.636+0000 7f0b6a332640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b5006c8f0 0x7f0b5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:01.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.637+0000 7f0b6a332640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b5006c8f0 0x7f0b5006edb0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c000c00 tx=0x7f0b5c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:01.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.640+0000 7f0b62ffd640 1 -- 172.21.15.44:0/4063056392 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0b58059680 con 0x7f0b641f4010 2024-04-02T13:29:01.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:01.825+0000 7f0b6c596640 1 -- 172.21.15.44:0/4063056392 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0b640684f0 con 0x7f0b641f4010 2024-04-02T13:29:02.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.502+0000 7f0b62ffd640 1 -- 172.21.15.44:0/4063056392 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v98) ==== 169+0+0 (secure 0 0 0) 0x7f0b5805d980 con 0x7f0b641f4010 2024-04-02T13:29:02.505 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' removed 2024-04-02T13:29:02.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.509+0000 7f0b6c596640 1 -- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b5006c8f0 msgr2=0x7f0b5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.509+0000 7f0b6c596640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b5006c8f0 0x7f0b5006edb0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c000c00 tx=0x7f0b5c009040 comp rx=0 tx=0).stop 2024-04-02T13:29:02.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.509+0000 7f0b6c596640 1 -- 172.21.15.44:0/4063056392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b641f4010 msgr2=0x7f0b640a1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.509+0000 7f0b6c596640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b641f4010 0x7f0b640a1400 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800e780 tx=0x7f0b5800ec40 comp rx=0 tx=0).stop 2024-04-02T13:29:02.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.509+0000 7f0b6c596640 1 -- 172.21.15.44:0/4063056392 shutdown_connections 2024-04-02T13:29:02.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.509+0000 7f0b6c596640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b5006c8f0 0x7f0b5006edb0 secure :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c000c00 tx=0x7f0b5c009040 comp rx=0 tx=0).stop 2024-04-02T13:29:02.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.509+0000 7f0b6c596640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b6415f540 0x7f0b6415c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.509+0000 7f0b6c596640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b641f4010 0x7f0b640a1400 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.509+0000 7f0b6c596640 1 --2- 172.21.15.44:0/4063056392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b4c0051c0 0x7f0b641505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.509+0000 7f0b6c596640 1 -- 172.21.15.44:0/4063056392 >> 172.21.15.44:0/4063056392 conn(0x7f0b6409d3b0 msgr2=0x7f0b640d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:02.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.510+0000 7f0b6c596640 1 -- 172.21.15.44:0/4063056392 shutdown_connections 2024-04-02T13:29:02.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.510+0000 7f0b6c596640 1 -- 172.21.15.44:0/4063056392 wait complete. 2024-04-02T13:29:02.531 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T13:29:02.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.637+0000 7f93463ba640 1 Processor -- start 2024-04-02T13:29:02.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.637+0000 7f93463ba640 1 -- start start 2024-04-02T13:29:02.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f93463ba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9340144130 0x7f93401691a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f93463ba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93400d0e10 con 0x7f93400b3b60 2024-04-02T13:29:02.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f93463ba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9340144750 con 0x7f93400d0a30 2024-04-02T13:29:02.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f93463ba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9340161e10 con 0x7f9340144130 2024-04-02T13:29:02.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f933f7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f93400b3b60 0x7f9340143a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:55018/0 (socket says 172.21.15.44:55018) 2024-04-02T13:29:02.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f933f7fe640 1 -- 172.21.15.44:0/2270522844 learned_addr learned my addr 172.21.15.44:0/2270522844 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:02.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3783737675 0 0) 0x7f9330001110 con 0x7f93400b3b60 2024-04-02T13:29:02.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f9344957640 1 --2- 172.21.15.44:0/2270522844 >> v2:172.21.15.69:3300/0 conn(0x7f9340144130 0x7f93401691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9328003520 con 0x7f93400b3b60 2024-04-02T13:29:02.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3894810258 0 0) 0x7f9328003520 con 0x7f93400b3b60 2024-04-02T13:29:02.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 >> v2:172.21.15.69:3300/0 conn(0x7f9340144130 msgr2=0x7f93401691a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:02.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f933effd640 1 --2- 172.21.15.44:0/2270522844 >> v2:172.21.15.69:3300/0 conn(0x7f9340144130 0x7f93401691a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 >> v1:172.21.15.69:6789/0 conn(0x7f93400d0a30 legacy=0x7f93400b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9340162060 con 0x7f93400b3b60 2024-04-02T13:29:02.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.638+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f9330001840 con 0x7f93400b3b60 2024-04-02T13:29:02.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 >> v1:172.21.15.44:6789/0 conn(0x7f93400b3b60 legacy=0x7f9340143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933effd640 1 --2- 172.21.15.44:0/2270522844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93280039b0 0x7f9328003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933effd640 1 --2- 172.21.15.44:0/2270522844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9328004800 0x7f9328004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933effd640 1 --2- 172.21.15.44:0/2270522844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93280051c0 0x7f932800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9340161e10 con 0x7f93280039b0 2024-04-02T13:29:02.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f932800c160 con 0x7f93280051c0 2024-04-02T13:29:02.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f932800c360 con 0x7f9328004800 2024-04-02T13:29:02.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933ffff640 1 --2- 172.21.15.44:0/2270522844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93280039b0 0x7f9328003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933f7fe640 1 --2- 172.21.15.44:0/2270522844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9328004800 0x7f9328004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f9344957640 1 --2- 172.21.15.44:0/2270522844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93280051c0 0x7f932800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933ffff640 1 -- 172.21.15.44:0/2270522844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9328004800 msgr2=0x7f9328004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:02.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933ffff640 1 --2- 172.21.15.44:0/2270522844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9328004800 0x7f9328004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933ffff640 1 -- 172.21.15.44:0/2270522844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93280051c0 msgr2=0x7f932800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:02.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933ffff640 1 --2- 172.21.15.44:0/2270522844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93280051c0 0x7f932800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f933ffff640 1 -- 172.21.15.44:0/2270522844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9328003520 con 0x7f93280039b0 2024-04-02T13:29:02.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f9344957640 1 --2- 172.21.15.44:0/2270522844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93280051c0 0x7f932800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:02.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.639+0000 7f9344957640 1 -- 172.21.15.44:0/2270522844 reap_dead start 2024-04-02T13:29:02.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.640+0000 7f933ffff640 1 --2- 172.21.15.44:0/2270522844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93280039b0 0x7f9328003dd0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f9334001750 tx=0x7f9328005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986c55157edd009b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.640+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9334001370 con 0x7f93280039b0 2024-04-02T13:29:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.640+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93340052d0 con 0x7f93280039b0 2024-04-02T13:29:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.640+0000 7f933effd640 1 -- 172.21.15.44:0/2270522844 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9334003830 con 0x7f93280039b0 2024-04-02T13:29:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.640+0000 7f93463ba640 1 -- 172.21.15.44:0/2270522844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93280039b0 msgr2=0x7f9328003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.640+0000 7f93463ba640 1 --2- 172.21.15.44:0/2270522844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93280039b0 0x7f9328003dd0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f9334001750 tx=0x7f9328005830 comp rx=0 tx=0).stop 2024-04-02T13:29:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.640+0000 7f93463ba640 1 -- 172.21.15.44:0/2270522844 shutdown_connections 2024-04-02T13:29:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.640+0000 7f93463ba640 1 --2- 172.21.15.44:0/2270522844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93280039b0 0x7f9328003dd0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.640+0000 7f93463ba640 1 -- 172.21.15.44:0/2270522844 >> 172.21.15.44:0/2270522844 conn(0x7f93400b5d90 msgr2=0x7f934006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.640+0000 7f93463ba640 1 -- 172.21.15.44:0/2270522844 shutdown_connections 2024-04-02T13:29:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.640+0000 7f93463ba640 1 -- 172.21.15.44:0/2270522844 wait complete. 2024-04-02T13:29:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.642+0000 7f93463ba640 1 Processor -- start 2024-04-02T13:29:02.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.642+0000 7f93463ba640 1 -- start start 2024-04-02T13:29:02.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.642+0000 7f93463ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93280039b0 0x7f93401f5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.642+0000 7f93463ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9340140ff0 0x7f93400b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f93463ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93400b16f0 0x7f93401e84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f93463ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9340162b20 con 0x7f9340140ff0 2024-04-02T13:29:02.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f93463ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9340162620 con 0x7f93400b16f0 2024-04-02T13:29:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f93463ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9340162ca0 con 0x7f93280039b0 2024-04-02T13:29:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f933f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9340140ff0 0x7f93400b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f933f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9340140ff0 0x7f93400b11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38750/0 (socket says 172.21.15.44:38750) 2024-04-02T13:29:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f933f7fe640 1 -- 172.21.15.44:0/1150514253 learned_addr learned my addr 172.21.15.44:0/1150514253 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f933f7fe640 1 -- 172.21.15.44:0/1150514253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93280039b0 msgr2=0x7f93401f5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f933f7fe640 1 --2- 172.21.15.44:0/1150514253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93280039b0 0x7f93401f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f933f7fe640 1 -- 172.21.15.44:0/1150514253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93400b16f0 msgr2=0x7f93401e84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f933f7fe640 1 --2- 172.21.15.44:0/1150514253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93400b16f0 0x7f93401e84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f933f7fe640 1 -- 172.21.15.44:0/1150514253 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9340141470 con 0x7f9340140ff0 2024-04-02T13:29:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f933f7fe640 1 --2- 172.21.15.44:0/1150514253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9340140ff0 0x7f93400b11b0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f9330007330 tx=0x7f93300077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f933cff9640 1 -- 172.21.15.44:0/1150514253 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93300017e0 con 0x7f9340140ff0 2024-04-02T13:29:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f933cff9640 1 -- 172.21.15.44:0/1150514253 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9330018070 con 0x7f9340140ff0 2024-04-02T13:29:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f93463ba640 1 -- 172.21.15.44:0/1150514253 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93401e9270 con 0x7f9340140ff0 2024-04-02T13:29:02.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f933cff9640 1 -- 172.21.15.44:0/1150514253 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933000eb10 con 0x7f9340140ff0 2024-04-02T13:29:02.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.643+0000 7f93463ba640 1 -- 172.21.15.44:0/1150514253 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9340142940 con 0x7f9340140ff0 2024-04-02T13:29:02.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.644+0000 7f93463ba640 1 -- 172.21.15.44:0/1150514253 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93400b1b10 con 0x7f9340140ff0 2024-04-02T13:29:02.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.645+0000 7f933cff9640 1 -- 172.21.15.44:0/1150514253 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f933000ecb0 con 0x7f9340140ff0 2024-04-02T13:29:02.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.645+0000 7f933cff9640 1 --2- 172.21.15.44:0/1150514253 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f932c06c920 0x7f932c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.645+0000 7f933cff9640 1 -- 172.21.15.44:0/1150514253 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5507+0+0 (secure 0 0 0) 0x7f933008b970 con 0x7f9340140ff0 2024-04-02T13:29:02.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.646+0000 7f933ffff640 1 --2- 172.21.15.44:0/1150514253 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f932c06c920 0x7f932c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.646+0000 7f933ffff640 1 --2- 172.21.15.44:0/1150514253 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f932c06c920 0x7f932c06ede0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9334033000 tx=0x7f933402f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:02.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.650+0000 7f933cff9640 1 -- 172.21.15.44:0/1150514253 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f93400d0e10 con 0x7f9340140ff0 2024-04-02T13:29:02.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.835+0000 7f93463ba640 1 -- 172.21.15.44:0/1150514253 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f934006c140 con 0x7f9340140ff0 2024-04-02T13:29:02.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.838+0000 7f933cff9640 1 -- 172.21.15.44:0/1150514253 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f9330059dc0 con 0x7f9340140ff0 2024-04-02T13:29:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.840+0000 7f93463ba640 1 -- 172.21.15.44:0/1150514253 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f932c06c920 msgr2=0x7f932c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.840+0000 7f93463ba640 1 --2- 172.21.15.44:0/1150514253 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f932c06c920 0x7f932c06ede0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9334033000 tx=0x7f933402f000 comp rx=0 tx=0).stop 2024-04-02T13:29:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.840+0000 7f93463ba640 1 -- 172.21.15.44:0/1150514253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9340140ff0 msgr2=0x7f93400b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.840+0000 7f93463ba640 1 --2- 172.21.15.44:0/1150514253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9340140ff0 0x7f93400b11b0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f9330007330 tx=0x7f93300077f0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.840+0000 7f93463ba640 1 -- 172.21.15.44:0/1150514253 shutdown_connections 2024-04-02T13:29:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.840+0000 7f93463ba640 1 --2- 172.21.15.44:0/1150514253 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f932c06c920 0x7f932c06ede0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.840+0000 7f93463ba640 1 --2- 172.21.15.44:0/1150514253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93400b16f0 0x7f93401e84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.840+0000 7f93463ba640 1 --2- 172.21.15.44:0/1150514253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9340140ff0 0x7f93400b11b0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.840+0000 7f93463ba640 1 --2- 172.21.15.44:0/1150514253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93280039b0 0x7f93401f5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.840+0000 7f93463ba640 1 -- 172.21.15.44:0/1150514253 >> 172.21.15.44:0/1150514253 conn(0x7f93400b5d90 msgr2=0x7f9340161250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.840+0000 7f93463ba640 1 -- 172.21.15.44:0/1150514253 shutdown_connections 2024-04-02T13:29:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.840+0000 7f93463ba640 1 -- 172.21.15.44:0/1150514253 wait complete. 2024-04-02T13:29:02.854 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair' 2024-04-02T13:29:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.953+0000 7fe21b17c640 1 Processor -- start 2024-04-02T13:29:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.953+0000 7fe21b17c640 1 -- start start 2024-04-02T13:29:02.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.954+0000 7fe21b17c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe21415e550 0x7fe21406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.954+0000 7fe21b17c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe21415fb50 con 0x7fe21415f040 2024-04-02T13:29:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.954+0000 7fe21b17c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe21406ca90 con 0x7fe21406d530 2024-04-02T13:29:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.954+0000 7fe21b17c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe21406d970 con 0x7fe21415e550 2024-04-02T13:29:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.954+0000 7fe219719640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe21415f040 0x7fe2140d0ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:55022/0 (socket says 172.21.15.44:55022) 2024-04-02T13:29:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.954+0000 7fe219719640 1 -- 172.21.15.44:0/121702595 learned_addr learned my addr 172.21.15.44:0/121702595 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.954+0000 7fe218f18640 1 --2- 172.21.15.44:0/121702595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe21415e550 0x7fe21406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.954+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1909417259 0 0) 0x7fe21415fb50 con 0x7fe21415f040 2024-04-02T13:29:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.954+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1fc003520 con 0x7fe21415f040 2024-04-02T13:29:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2138134839 0 0) 0x7fe21406ca90 con 0x7fe21406d530 2024-04-02T13:29:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe21415fb50 con 0x7fe21406d530 2024-04-02T13:29:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 749600126 0 0) 0x7fe1fc003520 con 0x7fe21415f040 2024-04-02T13:29:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe21415e550 msgr2=0x7fe21406cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 --2- 172.21.15.44:0/121702595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe21415e550 0x7fe21406cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 >> v1:172.21.15.69:6789/0 conn(0x7fe21406d530 legacy=0x7fe21415e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe21415f590 con 0x7fe21415f040 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fe204001670 con 0x7fe21415f040 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 >> v1:172.21.15.44:6789/0 conn(0x7fe21415f040 legacy=0x7fe2140d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 --2- 172.21.15.44:0/121702595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc003b70 0x7fe1fc003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 --2- 172.21.15.44:0/121702595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1fc004800 0x7fe1fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 --2- 172.21.15.44:0/121702595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe21415e550 0x7fe1fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe21406d970 con 0x7fe1fc004800 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc00ba70 con 0x7fe1fc003b70 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe1fc00bc70 con 0x7fe21415e550 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe213fff640 1 --2- 172.21.15.44:0/121702595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1fc004800 0x7fe1fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe218f18640 1 --2- 172.21.15.44:0/121702595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc003b70 0x7fe1fc003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe219719640 1 --2- 172.21.15.44:0/121702595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe21415e550 0x7fe1fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe213fff640 1 -- 172.21.15.44:0/121702595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe21415e550 msgr2=0x7fe1fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe213fff640 1 --2- 172.21.15.44:0/121702595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe21415e550 0x7fe1fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe213fff640 1 -- 172.21.15.44:0/121702595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc003b70 msgr2=0x7fe1fc003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe213fff640 1 --2- 172.21.15.44:0/121702595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc003b70 0x7fe1fc003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.955+0000 7fe213fff640 1 -- 172.21.15.44:0/121702595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1fc003520 con 0x7fe1fc004800 2024-04-02T13:29:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.956+0000 7fe213fff640 1 --2- 172.21.15.44:0/121702595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1fc004800 0x7fe1fc004c80 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fe2080018d0 tx=0x7fe1fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1cee56f8563952 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.956+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe208037070 con 0x7fe1fc004800 2024-04-02T13:29:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.956+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2080052d0 con 0x7fe1fc004800 2024-04-02T13:29:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.956+0000 7fe2137fe640 1 -- 172.21.15.44:0/121702595 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2080014f0 con 0x7fe1fc004800 2024-04-02T13:29:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.957+0000 7fe21b17c640 1 -- 172.21.15.44:0/121702595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1fc004800 msgr2=0x7fe1fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.957+0000 7fe21b17c640 1 --2- 172.21.15.44:0/121702595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1fc004800 0x7fe1fc004c80 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fe2080018d0 tx=0x7fe1fc005830 comp rx=0 tx=0).stop 2024-04-02T13:29:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.957+0000 7fe219719640 1 -- 172.21.15.44:0/121702595 reap_dead start 2024-04-02T13:29:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.957+0000 7fe21b17c640 1 -- 172.21.15.44:0/121702595 shutdown_connections 2024-04-02T13:29:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.957+0000 7fe21b17c640 1 -- 172.21.15.44:0/121702595 >> 172.21.15.44:0/121702595 conn(0x7fe21409aa20 msgr2=0x7fe21409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.957+0000 7fe21b17c640 1 -- 172.21.15.44:0/121702595 shutdown_connections 2024-04-02T13:29:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.957+0000 7fe21b17c640 1 -- 172.21.15.44:0/121702595 wait complete. 2024-04-02T13:29:02.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.958+0000 7fe21b17c640 1 Processor -- start 2024-04-02T13:29:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.958+0000 7fe21b17c640 1 -- start start 2024-04-02T13:29:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe21b17c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc004800 0x7fe2140c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe21b17c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2140b6cd0 0x7fe2140b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe21b17c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2140b4a80 0x7fe2140b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe21b17c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe2140d0590 con 0x7fe2140b6cd0 2024-04-02T13:29:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe21b17c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2140d02b0 con 0x7fe1fc004800 2024-04-02T13:29:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe21b17c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe2140cc4b0 con 0x7fe2140b4a80 2024-04-02T13:29:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe213fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2140b6cd0 0x7fe2140b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe213fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2140b6cd0 0x7fe2140b7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38772/0 (socket says 172.21.15.44:38772) 2024-04-02T13:29:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe213fff640 1 -- 172.21.15.44:0/4141362377 learned_addr learned my addr 172.21.15.44:0/4141362377 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe218f18640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc004800 0x7fe2140c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe219719640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2140b4a80 0x7fe2140b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe213fff640 1 -- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2140b4a80 msgr2=0x7fe2140b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe213fff640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2140b4a80 0x7fe2140b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe213fff640 1 -- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc004800 msgr2=0x7fe2140c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe213fff640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc004800 0x7fe2140c9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe213fff640 1 -- 172.21.15.44:0/4141362377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2140b5770 con 0x7fe2140b6cd0 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe218f18640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc004800 0x7fe2140c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe213fff640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2140b6cd0 0x7fe2140b7150 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fe208035040 tx=0x7fe2080012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe2117fa640 1 -- 172.21.15.44:0/4141362377 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe208042020 con 0x7fe2140b6cd0 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.959+0000 7fe2117fa640 1 -- 172.21.15.44:0/4141362377 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe208038040 con 0x7fe2140b6cd0 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.960+0000 7fe21b17c640 1 -- 172.21.15.44:0/4141362377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2140b5a50 con 0x7fe2140b6cd0 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.960+0000 7fe2117fa640 1 -- 172.21.15.44:0/4141362377 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe208037070 con 0x7fe2140b6cd0 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.960+0000 7fe21b17c640 1 -- 172.21.15.44:0/4141362377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2140b6010 con 0x7fe2140b6cd0 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.960+0000 7fe21b17c640 1 -- 172.21.15.44:0/4141362377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2140ce2c0 con 0x7fe2140b6cd0 2024-04-02T13:29:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.961+0000 7fe2117fa640 1 -- 172.21.15.44:0/4141362377 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe208045020 con 0x7fe2140b6cd0 2024-04-02T13:29:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.961+0000 7fe2117fa640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe20006c8f0 0x7fe20006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.961+0000 7fe2117fa640 1 -- 172.21.15.44:0/4141362377 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5507+0+0 (secure 0 0 0) 0x7fe2080ad610 con 0x7fe2140b6cd0 2024-04-02T13:29:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.962+0000 7fe218f18640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe20006c8f0 0x7fe20006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.962+0000 7fe218f18640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe20006c8f0 0x7fe20006edb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009f60 tx=0x7fe20c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:02.966+0000 7fe2117fa640 1 -- 172.21.15.44:0/4141362377 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe20804c030 con 0x7fe2140b6cd0 2024-04-02T13:29:03.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.233+0000 7fe21b17c640 1 -- 172.21.15.44:0/4141362377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]} v 0) -- 0x7fe2140684f0 con 0x7fe2140b6cd0 2024-04-02T13:29:03.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.503+0000 7fe2117fa640 1 -- 172.21.15.44:0/4141362377 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]}]=0 v415) ==== 167+0+0 (secure 0 0 0) 0x7fe20807ba60 con 0x7fe2140b6cd0 2024-04-02T13:29:03.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.505+0000 7fe21b17c640 1 -- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe20006c8f0 msgr2=0x7fe20006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:03.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.505+0000 7fe21b17c640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe20006c8f0 0x7fe20006edb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009f60 tx=0x7fe20c00b040 comp rx=0 tx=0).stop 2024-04-02T13:29:03.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.505+0000 7fe21b17c640 1 -- 172.21.15.44:0/4141362377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2140b6cd0 msgr2=0x7fe2140b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:03.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.505+0000 7fe21b17c640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2140b6cd0 0x7fe2140b7150 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fe208035040 tx=0x7fe2080012d0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.506+0000 7fe21b17c640 1 -- 172.21.15.44:0/4141362377 shutdown_connections 2024-04-02T13:29:03.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.506+0000 7fe21b17c640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe20006c8f0 0x7fe20006edb0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.506+0000 7fe21b17c640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2140b4a80 0x7fe2140b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.506+0000 7fe21b17c640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2140b6cd0 0x7fe2140b7150 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.506+0000 7fe21b17c640 1 --2- 172.21.15.44:0/4141362377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc004800 0x7fe2140c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.506+0000 7fe21b17c640 1 -- 172.21.15.44:0/4141362377 >> 172.21.15.44:0/4141362377 conn(0x7fe21409aa20 msgr2=0x7fe2140cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:03.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.506+0000 7fe21b17c640 1 -- 172.21.15.44:0/4141362377 shutdown_connections 2024-04-02T13:29:03.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.506+0000 7fe21b17c640 1 -- 172.21.15.44:0/4141362377 wait complete. 2024-04-02T13:29:03.520 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-04-02T13:29:03.520 INFO:tasks.cephfs_test_runner:Starting test: test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-02T13:29:03.521 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:29:03.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.621+0000 7ff61cb32640 1 Processor -- start 2024-04-02T13:29:03.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.621+0000 7ff61cb32640 1 -- start start 2024-04-02T13:29:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff61cb32640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff61815e4d0 0x7ff61806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff61cb32640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff618072f60 0x7ff61806ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff61cb32640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff61806dfe0 con 0x7ff61815e0f0 2024-04-02T13:29:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff61cb32640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff6180aed70 con 0x7ff618072f60 2024-04-02T13:29:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff61cb32640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff61806e2c0 con 0x7ff61815e4d0 2024-04-02T13:29:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff61659c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff61815e0f0 0x7ff6180d00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:55028/0 (socket says 172.21.15.44:55028) 2024-04-02T13:29:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff61659c640 1 -- 172.21.15.44:0/4260458493 learned_addr learned my addr 172.21.15.44:0/4260458493 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff616d9d640 1 --2- 172.21.15.44:0/4260458493 >> v2:172.21.15.69:3300/0 conn(0x7ff618072f60 0x7ff61806ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff60ffff640 1 --2- 172.21.15.44:0/4260458493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff61815e4d0 0x7ff61806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:03.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff615d9b640 1 -- 172.21.15.44:0/4260458493 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1475906212 0 0) 0x7ff61806dfe0 con 0x7ff61815e0f0 2024-04-02T13:29:03.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff615d9b640 1 -- 172.21.15.44:0/4260458493 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff604003520 con 0x7ff61815e0f0 2024-04-02T13:29:03.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff60ffff640 1 -- 172.21.15.44:0/4260458493 >> v2:172.21.15.69:3300/0 conn(0x7ff618072f60 msgr2=0x7ff61806ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:03.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff60ffff640 1 --2- 172.21.15.44:0/4260458493 >> v2:172.21.15.69:3300/0 conn(0x7ff618072f60 0x7ff61806ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff60ffff640 1 -- 172.21.15.44:0/4260458493 >> v1:172.21.15.44:6789/0 conn(0x7ff61815e0f0 legacy=0x7ff6180d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:03.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.622+0000 7ff60ffff640 1 -- 172.21.15.44:0/4260458493 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61806e5a0 con 0x7ff61815e4d0 2024-04-02T13:29:03.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.623+0000 7ff60ffff640 1 --2- 172.21.15.44:0/4260458493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff61815e4d0 0x7ff61806b4d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff6000097a0 tx=0x7ff60002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e505601ff3c9562 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:03.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.623+0000 7ff615d9b640 1 -- 172.21.15.44:0/4260458493 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600034070 con 0x7ff61815e4d0 2024-04-02T13:29:03.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.623+0000 7ff615d9b640 1 -- 172.21.15.44:0/4260458493 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff600030610 con 0x7ff61815e4d0 2024-04-02T13:29:03.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.623+0000 7ff615d9b640 1 -- 172.21.15.44:0/4260458493 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6000309c0 con 0x7ff61815e4d0 2024-04-02T13:29:03.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.624+0000 7ff61cb32640 1 -- 172.21.15.44:0/4260458493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff61815e4d0 msgr2=0x7ff61806b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.624+0000 7ff61cb32640 1 --2- 172.21.15.44:0/4260458493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff61815e4d0 0x7ff61806b4d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff6000097a0 tx=0x7ff60002b960 comp rx=0 tx=0).stop 2024-04-02T13:29:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.624+0000 7ff61cb32640 1 -- 172.21.15.44:0/4260458493 shutdown_connections 2024-04-02T13:29:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.624+0000 7ff61cb32640 1 --2- 172.21.15.44:0/4260458493 >> v2:172.21.15.69:3300/0 conn(0x7ff618072f60 0x7ff61806ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.624+0000 7ff61cb32640 1 --2- 172.21.15.44:0/4260458493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff61815e4d0 0x7ff61806b4d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.624+0000 7ff61cb32640 1 -- 172.21.15.44:0/4260458493 >> 172.21.15.44:0/4260458493 conn(0x7ff6180b2980 msgr2=0x7ff6180b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.624+0000 7ff61cb32640 1 -- 172.21.15.44:0/4260458493 shutdown_connections 2024-04-02T13:29:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.624+0000 7ff61cb32640 1 -- 172.21.15.44:0/4260458493 wait complete. 2024-04-02T13:29:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.625+0000 7ff61cb32640 1 Processor -- start 2024-04-02T13:29:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.625+0000 7ff61cb32640 1 -- start start 2024-04-02T13:29:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff61cb32640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff618072f60 0x7ff618155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff61cb32640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff61815e0f0 0x7ff6180cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff61cb32640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff61815e4d0 0x7ff6181f4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff61cb32640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff6180d0800 con 0x7ff61815e4d0 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff61cb32640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff6180d0af0 con 0x7ff618072f60 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff61cb32640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff61806eda0 con 0x7ff61815e0f0 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff616d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff61815e4d0 0x7ff6181f4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff616d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff61815e4d0 0x7ff6181f4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38776/0 (socket says 172.21.15.44:38776) 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff616d9d640 1 -- 172.21.15.44:0/2778317166 learned_addr learned my addr 172.21.15.44:0/2778317166 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff61659c640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff618072f60 0x7ff618155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff60ffff640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff61815e0f0 0x7ff6180cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff616d9d640 1 -- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff61815e0f0 msgr2=0x7ff6180cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff616d9d640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff61815e0f0 0x7ff6180cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff616d9d640 1 -- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff618072f60 msgr2=0x7ff618155390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff616d9d640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff618072f60 0x7ff618155390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.626+0000 7ff616d9d640 1 -- 172.21.15.44:0/2778317166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6180c81f0 con 0x7ff61815e4d0 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.627+0000 7ff616d9d640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff61815e4d0 0x7ff6181f4d20 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7ff61000efc0 tx=0x7ff61000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.627+0000 7ff60effd640 1 -- 172.21.15.44:0/2778317166 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610018020 con 0x7ff61815e4d0 2024-04-02T13:29:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.627+0000 7ff60effd640 1 -- 172.21.15.44:0/2778317166 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff610014480 con 0x7ff61815e4d0 2024-04-02T13:29:03.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.627+0000 7ff61cb32640 1 -- 172.21.15.44:0/2778317166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6180c84d0 con 0x7ff61815e4d0 2024-04-02T13:29:03.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.627+0000 7ff60effd640 1 -- 172.21.15.44:0/2778317166 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610014870 con 0x7ff61815e4d0 2024-04-02T13:29:03.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.627+0000 7ff61cb32640 1 -- 172.21.15.44:0/2778317166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6180c8a90 con 0x7ff61815e4d0 2024-04-02T13:29:03.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.628+0000 7ff61cb32640 1 -- 172.21.15.44:0/2778317166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6180684f0 con 0x7ff61815e4d0 2024-04-02T13:29:03.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.628+0000 7ff60effd640 1 -- 172.21.15.44:0/2778317166 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff610005d10 con 0x7ff61815e4d0 2024-04-02T13:29:03.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.628+0000 7ff60effd640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff5f806c8f0 0x7ff5f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:03.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.629+0000 7ff60effd640 1 -- 172.21.15.44:0/2778317166 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5507+0+0 (secure 0 0 0) 0x7ff61008c3f0 con 0x7ff61815e4d0 2024-04-02T13:29:03.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.629+0000 7ff61659c640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff5f806c8f0 0x7ff5f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:03.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.629+0000 7ff61659c640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff5f806c8f0 0x7ff5f806edb0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff608005880 tx=0x7ff608009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.633+0000 7ff60effd640 1 -- 172.21.15.44:0/2778317166 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff610022070 con 0x7ff61815e4d0 2024-04-02T13:29:03.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.818+0000 7ff61cb32640 1 -- 172.21.15.44:0/2778317166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff6180727f0 con 0x7ff61815e4d0 2024-04-02T13:29:03.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.819+0000 7ff60effd640 1 -- 172.21.15.44:0/2778317166 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+13504 (secure 0 0 0) 0x7ff610010040 con 0x7ff61815e4d0 2024-04-02T13:29:03.822 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:03.822 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":98,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:29:02.497587+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6833/3044289683":"2024-04-03T13:28:59.391162+0000","172.21.15.44:6832/3044289683":"2024-04-03T13:28:59.391162+0000","172.21.15.44:6833/2250279975":"2024-04-03T13:28:15.300517+0000","172.21.15.44:6832/2250279975":"2024-04-03T13:28:15.300517+0000","172.21.15.44:6833/719680934":"2024-04-03T13:26:55.589090+0000","172.21.15.44:6832/719680934":"2024-04-03T13:26:55.589090+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:29:03.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.821+0000 7ff61cb32640 1 -- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff5f806c8f0 msgr2=0x7ff5f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:03.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.821+0000 7ff61cb32640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff5f806c8f0 0x7ff5f806edb0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff608005880 tx=0x7ff608009040 comp rx=0 tx=0).stop 2024-04-02T13:29:03.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.821+0000 7ff61cb32640 1 -- 172.21.15.44:0/2778317166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff61815e4d0 msgr2=0x7ff6181f4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:03.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.821+0000 7ff61cb32640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff61815e4d0 0x7ff6181f4d20 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7ff61000efc0 tx=0x7ff61000c490 comp rx=0 tx=0).stop 2024-04-02T13:29:03.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.821+0000 7ff61cb32640 1 -- 172.21.15.44:0/2778317166 shutdown_connections 2024-04-02T13:29:03.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.821+0000 7ff61cb32640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff5f806c8f0 0x7ff5f806edb0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.821+0000 7ff61cb32640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff61815e4d0 0x7ff6181f4d20 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.821+0000 7ff61cb32640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff61815e0f0 0x7ff6180cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.821+0000 7ff61cb32640 1 --2- 172.21.15.44:0/2778317166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff618072f60 0x7ff618155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.821+0000 7ff61cb32640 1 -- 172.21.15.44:0/2778317166 >> 172.21.15.44:0/2778317166 conn(0x7ff6180b2980 msgr2=0x7ff6180d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:03.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.821+0000 7ff61cb32640 1 -- 172.21.15.44:0/2778317166 shutdown_connections 2024-04-02T13:29:03.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.821+0000 7ff61cb32640 1 -- 172.21.15.44:0/2778317166 wait complete. 2024-04-02T13:29:03.835 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:29:03.835 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:29:03.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.936+0000 7f39486e4640 1 Processor -- start 2024-04-02T13:29:03.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.936+0000 7f39486e4640 1 -- start start 2024-04-02T13:29:03.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.937+0000 7f39486e4640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f39400aeca0 0x7f394006e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:03.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.937+0000 7f39486e4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f394006eb10 0x7f39400d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:03.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.937+0000 7f39486e4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39400cdd60 con 0x7f39400d05e0 2024-04-02T13:29:03.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.937+0000 7f39486e4640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f394015b3b0 con 0x7f39400aeca0 2024-04-02T13:29:03.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.937+0000 7f39486e4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f39400d0a10 con 0x7f394006eb10 2024-04-02T13:29:03.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.938+0000 7f3946480640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f39400aeca0 0x7f394006e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:03.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.938+0000 7f3946480640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f39400aeca0 0x7f394006e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38780/0 (socket says 172.21.15.44:38780) 2024-04-02T13:29:03.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.938+0000 7f3946480640 1 -- 172.21.15.44:0/1256131662 learned_addr learned my addr 172.21.15.44:0/1256131662 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:03.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.938+0000 7f3945c7f640 1 --2- 172.21.15.44:0/1256131662 >> v2:172.21.15.69:3300/0 conn(0x7f394006eb10 0x7f39400d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:03.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.938+0000 7f3946480640 1 -- 172.21.15.44:0/1256131662 >> v2:172.21.15.69:3300/0 conn(0x7f394006eb10 msgr2=0x7f39400d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:03.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.938+0000 7f3946480640 1 --2- 172.21.15.44:0/1256131662 >> v2:172.21.15.69:3300/0 conn(0x7f394006eb10 0x7f39400d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.938+0000 7f3946480640 1 -- 172.21.15.44:0/1256131662 >> v1:172.21.15.69:6789/0 conn(0x7f39400d05e0 legacy=0x7f394006d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:03.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.938+0000 7f3946480640 1 -- 172.21.15.44:0/1256131662 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39400ce040 con 0x7f39400aeca0 2024-04-02T13:29:03.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.939+0000 7f3946480640 1 --2- 172.21.15.44:0/1256131662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39400aeca0 0x7f394006e5d0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f39400d0c90 tx=0x7f393802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5be17360ed6c358 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:03.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.939+0000 7f394547e640 1 -- 172.21.15.44:0/1256131662 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3938034070 con 0x7f39400aeca0 2024-04-02T13:29:03.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.939+0000 7f394547e640 1 -- 172.21.15.44:0/1256131662 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3938030860 con 0x7f39400aeca0 2024-04-02T13:29:03.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.939+0000 7f394547e640 1 -- 172.21.15.44:0/1256131662 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3938030c30 con 0x7f39400aeca0 2024-04-02T13:29:03.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.939+0000 7f39486e4640 1 -- 172.21.15.44:0/1256131662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39400aeca0 msgr2=0x7f394006e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:03.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.939+0000 7f39486e4640 1 --2- 172.21.15.44:0/1256131662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39400aeca0 0x7f394006e5d0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f39400d0c90 tx=0x7f393802b9a0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.940+0000 7f39486e4640 1 -- 172.21.15.44:0/1256131662 shutdown_connections 2024-04-02T13:29:03.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.940+0000 7f39486e4640 1 --2- 172.21.15.44:0/1256131662 >> v2:172.21.15.69:3300/0 conn(0x7f394006eb10 0x7f39400d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.940+0000 7f39486e4640 1 --2- 172.21.15.44:0/1256131662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39400aeca0 0x7f394006e5d0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.940+0000 7f39486e4640 1 -- 172.21.15.44:0/1256131662 >> 172.21.15.44:0/1256131662 conn(0x7f394006b920 msgr2=0x7f394006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:03.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.940+0000 7f39486e4640 1 -- 172.21.15.44:0/1256131662 shutdown_connections 2024-04-02T13:29:03.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.940+0000 7f39486e4640 1 -- 172.21.15.44:0/1256131662 wait complete. 2024-04-02T13:29:03.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.941+0000 7f39486e4640 1 Processor -- start 2024-04-02T13:29:03.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.941+0000 7f39486e4640 1 -- start start 2024-04-02T13:29:03.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f39486e4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f394006eb10 0x7f394015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:03.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f39486e4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39400aeca0 0x7f3940159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:03.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f39486e4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39400d05e0 0x7f39401e78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f39486e4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f39400ce8e0 con 0x7f39400d05e0 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f39486e4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f39400ce600 con 0x7f394006eb10 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f39486e4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f39400addf0 con 0x7f39400aeca0 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f3946c81640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39400d05e0 0x7f39401e78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f3946c81640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39400d05e0 0x7f39401e78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38796/0 (socket says 172.21.15.44:38796) 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f3946c81640 1 -- 172.21.15.44:0/825588664 learned_addr learned my addr 172.21.15.44:0/825588664 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f3946480640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f394006eb10 0x7f394015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f3945c7f640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39400aeca0 0x7f3940159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f3946c81640 1 -- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39400aeca0 msgr2=0x7f3940159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f3946c81640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39400aeca0 0x7f3940159570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f3946c81640 1 -- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f394006eb10 msgr2=0x7f394015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f3946c81640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f394006eb10 0x7f394015a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f3946c81640 1 -- 172.21.15.44:0/825588664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39400b2a60 con 0x7f39400d05e0 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f3946c81640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39400d05e0 0x7f39401e78b0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f393c00c6b0 tx=0x7f393c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:03.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f3936ffd640 1 -- 172.21.15.44:0/825588664 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c003b70 con 0x7f39400d05e0 2024-04-02T13:29:03.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f3936ffd640 1 -- 172.21.15.44:0/825588664 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f393c003d30 con 0x7f39400d05e0 2024-04-02T13:29:03.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.942+0000 7f39486e4640 1 -- 172.21.15.44:0/825588664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39400c4be0 con 0x7f39400d05e0 2024-04-02T13:29:03.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.943+0000 7f3936ffd640 1 -- 172.21.15.44:0/825588664 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c005310 con 0x7f39400d05e0 2024-04-02T13:29:03.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.943+0000 7f39486e4640 1 -- 172.21.15.44:0/825588664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39400b2fe0 con 0x7f39400d05e0 2024-04-02T13:29:03.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.943+0000 7f39486e4640 1 -- 172.21.15.44:0/825588664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39400684f0 con 0x7f39400d05e0 2024-04-02T13:29:03.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.944+0000 7f3936ffd640 1 -- 172.21.15.44:0/825588664 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f393c0054b0 con 0x7f39400d05e0 2024-04-02T13:29:03.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.944+0000 7f3936ffd640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f392406c8f0 0x7f392406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:03.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.945+0000 7f3936ffd640 1 -- 172.21.15.44:0/825588664 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5507+0+0 (secure 0 0 0) 0x7f393c08b760 con 0x7f39400d05e0 2024-04-02T13:29:03.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.945+0000 7f3946480640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f392406c8f0 0x7f392406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:03.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.945+0000 7f3946480640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f392406c8f0 0x7f392406edb0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3938036000 tx=0x7f3938008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:03.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:03.949+0000 7f3936ffd640 1 -- 172.21.15.44:0/825588664 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f393c0595f0 con 0x7f39400d05e0 2024-04-02T13:29:04.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.134+0000 7f39486e4640 1 -- 172.21.15.44:0/825588664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f39400b31b0 con 0x7f39400d05e0 2024-04-02T13:29:04.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.134+0000 7f3936ffd640 1 -- 172.21.15.44:0/825588664 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) ==== 93+0+10 (secure 0 0 0) 0x7f393c05d8f0 con 0x7f39400d05e0 2024-04-02T13:29:04.137 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:29:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.136+0000 7f39486e4640 1 -- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f392406c8f0 msgr2=0x7f392406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.136+0000 7f39486e4640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f392406c8f0 0x7f392406edb0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3938036000 tx=0x7f3938008030 comp rx=0 tx=0).stop 2024-04-02T13:29:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.136+0000 7f39486e4640 1 -- 172.21.15.44:0/825588664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39400d05e0 msgr2=0x7f39401e78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.136+0000 7f39486e4640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39400d05e0 0x7f39401e78b0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f393c00c6b0 tx=0x7f393c00cb70 comp rx=0 tx=0).stop 2024-04-02T13:29:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.136+0000 7f39486e4640 1 -- 172.21.15.44:0/825588664 shutdown_connections 2024-04-02T13:29:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.136+0000 7f39486e4640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f392406c8f0 0x7f392406edb0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.136+0000 7f39486e4640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39400d05e0 0x7f39401e78b0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.136+0000 7f39486e4640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39400aeca0 0x7f3940159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.136+0000 7f39486e4640 1 --2- 172.21.15.44:0/825588664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f394006eb10 0x7f394015a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.136+0000 7f39486e4640 1 -- 172.21.15.44:0/825588664 >> 172.21.15.44:0/825588664 conn(0x7f394006b920 msgr2=0x7f39400b4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.136+0000 7f39486e4640 1 -- 172.21.15.44:0/825588664 shutdown_connections 2024-04-02T13:29:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.136+0000 7f39486e4640 1 -- 172.21.15.44:0/825588664 wait complete. 2024-04-02T13:29:04.150 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync' 2024-04-02T13:29:04.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.249+0000 7f31347e9640 1 Processor -- start 2024-04-02T13:29:04.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.249+0000 7f31347e9640 1 -- start start 2024-04-02T13:29:04.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.249+0000 7f31347e9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f312c0d28f0 0x7f312c0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:04.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.249+0000 7f31347e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312c0b5200 0x7f312c0b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:04.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.249+0000 7f31347e9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f312c0d1260 0x7f312c0b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:04.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.249+0000 7f31347e9640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f312c0d5050 con 0x7f312c0d28f0 2024-04-02T13:29:04.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.249+0000 7f31347e9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f312c0d23a0 con 0x7f312c0d1260 2024-04-02T13:29:04.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.249+0000 7f31347e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f312c0b5020 con 0x7f312c0b5200 2024-04-02T13:29:04.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.250+0000 7f3132585640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f312c0d28f0 0x7f312c0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:04.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.250+0000 7f3132585640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f312c0d28f0 0x7f312c0d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38810/0 (socket says 172.21.15.44:38810) 2024-04-02T13:29:04.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.250+0000 7f3132585640 1 -- 172.21.15.44:0/2435719053 learned_addr learned my addr 172.21.15.44:0/2435719053 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:04.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.250+0000 7f3132d86640 1 --2- 172.21.15.44:0/2435719053 >> v2:172.21.15.69:3300/0 conn(0x7f312c0d1260 0x7f312c0b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:04.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.250+0000 7f3131d84640 1 --2- 172.21.15.44:0/2435719053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312c0b5200 0x7f312c0b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:04.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.250+0000 7f3132585640 1 -- 172.21.15.44:0/2435719053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312c0b5200 msgr2=0x7f312c0b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:04.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.250+0000 7f3132585640 1 --2- 172.21.15.44:0/2435719053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312c0b5200 0x7f312c0b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:04.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.250+0000 7f3132585640 1 -- 172.21.15.44:0/2435719053 >> v2:172.21.15.69:3300/0 conn(0x7f312c0d1260 msgr2=0x7f312c0b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:04.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.250+0000 7f3132585640 1 --2- 172.21.15.44:0/2435719053 >> v2:172.21.15.69:3300/0 conn(0x7f312c0d1260 0x7f312c0b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:04.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.250+0000 7f3132585640 1 -- 172.21.15.44:0/2435719053 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312c0d4d70 con 0x7f312c0d28f0 2024-04-02T13:29:04.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.250+0000 7f3131d84640 1 --2- 172.21.15.44:0/2435719053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312c0b5200 0x7f312c0b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:04.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.251+0000 7f3132585640 1 --2- 172.21.15.44:0/2435719053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f312c0d28f0 0x7f312c0d2cd0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f312c0ce800 tx=0x7f312802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc38524bda885fb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:04.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.251+0000 7f3131583640 1 -- 172.21.15.44:0/2435719053 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3128038070 con 0x7f312c0d28f0 2024-04-02T13:29:04.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.251+0000 7f3131583640 1 -- 172.21.15.44:0/2435719053 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f312802f050 con 0x7f312c0d28f0 2024-04-02T13:29:04.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.251+0000 7f3131583640 1 -- 172.21.15.44:0/2435719053 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3128031970 con 0x7f312c0d28f0 2024-04-02T13:29:04.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.251+0000 7f31347e9640 1 -- 172.21.15.44:0/2435719053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f312c0d28f0 msgr2=0x7f312c0d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:04.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.251+0000 7f31347e9640 1 --2- 172.21.15.44:0/2435719053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f312c0d28f0 0x7f312c0d2cd0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f312c0ce800 tx=0x7f312802b7f0 comp rx=0 tx=0).stop 2024-04-02T13:29:04.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.252+0000 7f31347e9640 1 -- 172.21.15.44:0/2435719053 shutdown_connections 2024-04-02T13:29:04.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.252+0000 7f31347e9640 1 --2- 172.21.15.44:0/2435719053 >> v2:172.21.15.69:3300/0 conn(0x7f312c0d1260 0x7f312c0b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:04.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.252+0000 7f31347e9640 1 --2- 172.21.15.44:0/2435719053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312c0b5200 0x7f312c0b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:04.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.252+0000 7f31347e9640 1 --2- 172.21.15.44:0/2435719053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f312c0d28f0 0x7f312c0d2cd0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:04.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.252+0000 7f31347e9640 1 -- 172.21.15.44:0/2435719053 >> 172.21.15.44:0/2435719053 conn(0x7f312c0d4120 msgr2=0x7f312c0d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:04.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.252+0000 7f31347e9640 1 -- 172.21.15.44:0/2435719053 shutdown_connections 2024-04-02T13:29:04.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.252+0000 7f31347e9640 1 -- 172.21.15.44:0/2435719053 wait complete. 2024-04-02T13:29:04.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.253+0000 7f31347e9640 1 Processor -- start 2024-04-02T13:29:04.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.253+0000 7f31347e9640 1 -- start start 2024-04-02T13:29:04.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.253+0000 7f31347e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c0b5200 0x7f312c0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:04.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.253+0000 7f31347e9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f312c0d1260 0x7f312c15b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.253+0000 7f31347e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312c0d28f0 0x7f312c0cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.253+0000 7f31347e9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f312c0ced30 con 0x7f312c0d1260 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.253+0000 7f31347e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f312c0b4cb0 con 0x7f312c0b5200 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.253+0000 7f31347e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f312c0ceeb0 con 0x7f312c0d28f0 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3131d84640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f312c0d1260 0x7f312c15b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3131d84640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f312c0d1260 0x7f312c15b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38820/0 (socket says 172.21.15.44:38820) 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3131d84640 1 -- 172.21.15.44:0/3188082032 learned_addr learned my addr 172.21.15.44:0/3188082032 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3132585640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c0b5200 0x7f312c0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3131d84640 1 -- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312c0d28f0 msgr2=0x7f312c0cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3131d84640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312c0d28f0 0x7f312c0cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3131d84640 1 -- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c0b5200 msgr2=0x7f312c0b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3131d84640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c0b5200 0x7f312c0b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3131d84640 1 -- 172.21.15.44:0/3188082032 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312c0b2730 con 0x7f312c0d1260 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3132585640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c0b5200 0x7f312c0b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:04.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3131d84640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f312c0d1260 0x7f312c15b3f0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f311c00e7c0 tx=0x7f311c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:04.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3122ffd640 1 -- 172.21.15.44:0/3188082032 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c00f040 con 0x7f312c0d1260 2024-04-02T13:29:04.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3122ffd640 1 -- 172.21.15.44:0/3188082032 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f311c004030 con 0x7f312c0d1260 2024-04-02T13:29:04.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f31347e9640 1 -- 172.21.15.44:0/3188082032 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f312c0b2a10 con 0x7f312c0d1260 2024-04-02T13:29:04.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f3122ffd640 1 -- 172.21.15.44:0/3188082032 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c004400 con 0x7f312c0d1260 2024-04-02T13:29:04.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.254+0000 7f31347e9640 1 -- 172.21.15.44:0/3188082032 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f312c0b36a0 con 0x7f312c0d1260 2024-04-02T13:29:04.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.255+0000 7f31347e9640 1 -- 172.21.15.44:0/3188082032 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f312c0684f0 con 0x7f312c0d1260 2024-04-02T13:29:04.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.256+0000 7f3122ffd640 1 -- 172.21.15.44:0/3188082032 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f311c00c6d0 con 0x7f312c0d1260 2024-04-02T13:29:04.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.256+0000 7f3122ffd640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f311006c8f0 0x7f311006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:04.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.256+0000 7f3122ffd640 1 -- 172.21.15.44:0/3188082032 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5507+0+0 (secure 0 0 0) 0x7f311c08b6f0 con 0x7f312c0d1260 2024-04-02T13:29:04.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.256+0000 7f3132585640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f311006c8f0 0x7f311006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:04.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.257+0000 7f3132585640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f311006c8f0 0x7f311006edb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3128035000 tx=0x7f3128008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:04.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.260+0000 7f3122ffd640 1 -- 172.21.15.44:0/3188082032 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f311c017020 con 0x7f312c0d1260 2024-04-02T13:29:04.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:04.517+0000 7f31347e9640 1 -- 172.21.15.44:0/3188082032 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]} v 0) -- 0x7f312c0b2be0 con 0x7f312c0d1260 2024-04-02T13:29:05.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.509+0000 7f3122ffd640 1 -- 172.21.15.44:0/3188082032 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]}]=0 v417) ==== 149+0+0 (secure 0 0 0) 0x7f311c059be0 con 0x7f312c0d1260 2024-04-02T13:29:05.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.512+0000 7f31347e9640 1 -- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f311006c8f0 msgr2=0x7f311006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:05.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.512+0000 7f31347e9640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f311006c8f0 0x7f311006edb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3128035000 tx=0x7f3128008030 comp rx=0 tx=0).stop 2024-04-02T13:29:05.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.512+0000 7f31347e9640 1 -- 172.21.15.44:0/3188082032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f312c0d1260 msgr2=0x7f312c15b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:05.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.512+0000 7f31347e9640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f312c0d1260 0x7f312c15b3f0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f311c00e7c0 tx=0x7f311c00ec80 comp rx=0 tx=0).stop 2024-04-02T13:29:05.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.512+0000 7f31347e9640 1 -- 172.21.15.44:0/3188082032 shutdown_connections 2024-04-02T13:29:05.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.512+0000 7f31347e9640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f311006c8f0 0x7f311006edb0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.512+0000 7f31347e9640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312c0d28f0 0x7f312c0cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.512+0000 7f31347e9640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f312c0d1260 0x7f312c15b3f0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.512+0000 7f31347e9640 1 --2- 172.21.15.44:0/3188082032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c0b5200 0x7f312c0b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.512+0000 7f31347e9640 1 -- 172.21.15.44:0/3188082032 >> 172.21.15.44:0/3188082032 conn(0x7f312c0d4120 msgr2=0x7f312c1615f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:05.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.512+0000 7f31347e9640 1 -- 172.21.15.44:0/3188082032 shutdown_connections 2024-04-02T13:29:05.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.512+0000 7f31347e9640 1 -- 172.21.15.44:0/3188082032 wait complete. 2024-04-02T13:29:05.529 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T13:29:05.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.636+0000 7fe410b22640 1 Processor -- start 2024-04-02T13:29:05.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.636+0000 7fe410b22640 1 -- start start 2024-04-02T13:29:05.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.637+0000 7fe410b22640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe40c15e470 0x7fe40c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:05.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.637+0000 7fe410b22640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe40c0b1540 0x7fe40c0d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:05.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.637+0000 7fe410b22640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe40c0ce790 0x7fe40c0d20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:05.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.637+0000 7fe410b22640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe40c0d2630 con 0x7fe40c0b1540 2024-04-02T13:29:05.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.637+0000 7fe410b22640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe40c0ced00 con 0x7fe40c0ce790 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.637+0000 7fe410b22640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe40c0cefe0 con 0x7fe40c15e470 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.637+0000 7fe409d9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe40c0b1540 0x7fe40c0d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.637+0000 7fe409d9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe40c0b1540 0x7fe40c0d1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51712/0 (socket says 172.21.15.44:51712) 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.637+0000 7fe409d9b640 1 -- 172.21.15.44:0/668367202 learned_addr learned my addr 172.21.15.44:0/668367202 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.638+0000 7fe40ad9d640 1 --2- 172.21.15.44:0/668367202 >> v2:172.21.15.69:3300/0 conn(0x7fe40c0ce790 0x7fe40c0d20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.638+0000 7fe40a59c640 1 --2- 172.21.15.44:0/668367202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe40c15e470 0x7fe40c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.638+0000 7fe409d9b640 1 -- 172.21.15.44:0/668367202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe40c15e470 msgr2=0x7fe40c0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.638+0000 7fe409d9b640 1 --2- 172.21.15.44:0/668367202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe40c15e470 0x7fe40c0b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.638+0000 7fe409d9b640 1 -- 172.21.15.44:0/668367202 >> v2:172.21.15.69:3300/0 conn(0x7fe40c0ce790 msgr2=0x7fe40c0d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.638+0000 7fe409d9b640 1 --2- 172.21.15.44:0/668367202 >> v2:172.21.15.69:3300/0 conn(0x7fe40c0ce790 0x7fe40c0d20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.638+0000 7fe409d9b640 1 -- 172.21.15.44:0/668367202 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40c0cf2c0 con 0x7fe40c0b1540 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.638+0000 7fe40ad9d640 1 --2- 172.21.15.44:0/668367202 >> v2:172.21.15.69:3300/0 conn(0x7fe40c0ce790 0x7fe40c0d20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.638+0000 7fe409d9b640 1 --2- 172.21.15.44:0/668367202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe40c0b1540 0x7fe40c0d1bb0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fe40c071b40 tx=0x7fe3f402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be864285c6dfb13 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:05.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.639+0000 7fe40959a640 1 -- 172.21.15.44:0/668367202 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f40317d0 con 0x7fe40c0b1540 2024-04-02T13:29:05.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.639+0000 7fe40959a640 1 -- 172.21.15.44:0/668367202 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f4037070 con 0x7fe40c0b1540 2024-04-02T13:29:05.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.639+0000 7fe40959a640 1 -- 172.21.15.44:0/668367202 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4031b90 con 0x7fe40c0b1540 2024-04-02T13:29:05.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.639+0000 7fe410b22640 1 -- 172.21.15.44:0/668367202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe40c0b1540 msgr2=0x7fe40c0d1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:05.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.639+0000 7fe410b22640 1 --2- 172.21.15.44:0/668367202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe40c0b1540 0x7fe40c0d1bb0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fe40c071b40 tx=0x7fe3f402b870 comp rx=0 tx=0).stop 2024-04-02T13:29:05.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.639+0000 7fe410b22640 1 -- 172.21.15.44:0/668367202 shutdown_connections 2024-04-02T13:29:05.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.639+0000 7fe410b22640 1 --2- 172.21.15.44:0/668367202 >> v2:172.21.15.69:3300/0 conn(0x7fe40c0ce790 0x7fe40c0d20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.639+0000 7fe410b22640 1 --2- 172.21.15.44:0/668367202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe40c0b1540 0x7fe40c0d1bb0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.639+0000 7fe410b22640 1 --2- 172.21.15.44:0/668367202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe40c15e470 0x7fe40c0b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.639+0000 7fe410b22640 1 -- 172.21.15.44:0/668367202 >> 172.21.15.44:0/668367202 conn(0x7fe40c09aa20 msgr2=0x7fe40c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.639+0000 7fe410b22640 1 -- 172.21.15.44:0/668367202 shutdown_connections 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.639+0000 7fe410b22640 1 -- 172.21.15.44:0/668367202 wait complete. 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.641+0000 7fe410b22640 1 Processor -- start 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.641+0000 7fe410b22640 1 -- start start 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.641+0000 7fe410b22640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe40c0b1540 0x7fe40c160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.641+0000 7fe410b22640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe40c0ce790 0x7fe40c1f4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.641+0000 7fe410b22640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe40c15e470 0x7fe40c15f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.641+0000 7fe410b22640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe40c06cb60 con 0x7fe40c0ce790 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.641+0000 7fe410b22640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe40c15e850 con 0x7fe40c15e470 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.641+0000 7fe410b22640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe40c161fd0 con 0x7fe40c0b1540 2024-04-02T13:29:05.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.641+0000 7fe409d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe40c0ce790 0x7fe40c1f4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.641+0000 7fe409d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe40c0ce790 0x7fe40c1f4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51716/0 (socket says 172.21.15.44:51716) 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe409d9b640 1 -- 172.21.15.44:0/1455399112 learned_addr learned my addr 172.21.15.44:0/1455399112 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe40ad9d640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe40c15e470 0x7fe40c15f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe40a59c640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe40c0b1540 0x7fe40c160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe409d9b640 1 -- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe40c0b1540 msgr2=0x7fe40c160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe409d9b640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe40c0b1540 0x7fe40c160fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe409d9b640 1 -- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe40c15e470 msgr2=0x7fe40c15f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe409d9b640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe40c15e470 0x7fe40c15f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe409d9b640 1 -- 172.21.15.44:0/1455399112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40c0b4b40 con 0x7fe40c0ce790 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe40ad9d640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe40c15e470 0x7fe40c15f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe409d9b640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe40c0ce790 0x7fe40c1f4070 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4000c00 tx=0x7fe3f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe402ffd640 1 -- 172.21.15.44:0/1455399112 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4032660 con 0x7fe40c0ce790 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe402ffd640 1 -- 172.21.15.44:0/1455399112 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f4032840 con 0x7fe40c0ce790 2024-04-02T13:29:05.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe410b22640 1 -- 172.21.15.44:0/1455399112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe40c0b4e20 con 0x7fe40c0ce790 2024-04-02T13:29:05.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe402ffd640 1 -- 172.21.15.44:0/1455399112 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4032c30 con 0x7fe40c0ce790 2024-04-02T13:29:05.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.642+0000 7fe410b22640 1 -- 172.21.15.44:0/1455399112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe40c0b53e0 con 0x7fe40c0ce790 2024-04-02T13:29:05.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.643+0000 7fe410b22640 1 -- 172.21.15.44:0/1455399112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe40c15f980 con 0x7fe40c0ce790 2024-04-02T13:29:05.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.644+0000 7fe402ffd640 1 -- 172.21.15.44:0/1455399112 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe3f404c060 con 0x7fe40c0ce790 2024-04-02T13:29:05.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.644+0000 7fe402ffd640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe3ec06c8f0 0x7fe3ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:05.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.644+0000 7fe402ffd640 1 -- 172.21.15.44:0/1455399112 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5507+0+0 (secure 0 0 0) 0x7fe3f40ad440 con 0x7fe40c0ce790 2024-04-02T13:29:05.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.644+0000 7fe40a59c640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe3ec06c8f0 0x7fe3ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:05.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.645+0000 7fe40a59c640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe3ec06c8f0 0x7fe3ec06edb0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe404006fd0 tx=0x7fe404008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:05.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.648+0000 7fe402ffd640 1 -- 172.21.15.44:0/1455399112 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe3f4040070 con 0x7fe40c0ce790 2024-04-02T13:29:05.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.830+0000 7fe410b22640 1 -- 172.21.15.44:0/1455399112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe40c0d2d10 con 0x7fe40c0ce790 2024-04-02T13:29:05.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.836+0000 7fe402ffd640 1 -- 172.21.15.44:0/1455399112 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) ==== 122+0+0 (secure 0 0 0) 0x7fe3f407b9c0 con 0x7fe40c0ce790 2024-04-02T13:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.838+0000 7fe410b22640 1 -- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe3ec06c8f0 msgr2=0x7fe3ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.838+0000 7fe410b22640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe3ec06c8f0 0x7fe3ec06edb0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe404006fd0 tx=0x7fe404008040 comp rx=0 tx=0).stop 2024-04-02T13:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.838+0000 7fe410b22640 1 -- 172.21.15.44:0/1455399112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe40c0ce790 msgr2=0x7fe40c1f4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.838+0000 7fe410b22640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe40c0ce790 0x7fe40c1f4070 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4000c00 tx=0x7fe3f40046c0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.838+0000 7fe410b22640 1 -- 172.21.15.44:0/1455399112 shutdown_connections 2024-04-02T13:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.838+0000 7fe410b22640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe3ec06c8f0 0x7fe3ec06edb0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.838+0000 7fe410b22640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe40c15e470 0x7fe40c15f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.838+0000 7fe410b22640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe40c0ce790 0x7fe40c1f4070 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.838+0000 7fe410b22640 1 --2- 172.21.15.44:0/1455399112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe40c0b1540 0x7fe40c160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.838+0000 7fe410b22640 1 -- 172.21.15.44:0/1455399112 >> 172.21.15.44:0/1455399112 conn(0x7fe40c09aa20 msgr2=0x7fe40c0aebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.838+0000 7fe410b22640 1 -- 172.21.15.44:0/1455399112 shutdown_connections 2024-04-02T13:29:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:05.838+0000 7fe410b22640 1 -- 172.21.15.44:0/1455399112 wait complete. 2024-04-02T13:29:05.852 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:29:05.852 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:29:05.909 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:29:05.971 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:29:05.971 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:29:05.971 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:29:05.986 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:29:05.986 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:29:05.986 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:29:06.003 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:29:06.003 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:29:06.003 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:29:06.041 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:29:06.041 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:29:06.041 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:29:06.060 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:29:06.060 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T13:29:06.060 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:29:06.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.181+0000 7f542735f640 1 Processor -- start 2024-04-02T13:29:06.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.181+0000 7f542735f640 1 -- start start 2024-04-02T13:29:06.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.182+0000 7f542735f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f542006cf90 0x7f54200cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.182+0000 7f542735f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f54200cfe30 0x7f54200d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.182+0000 7f542735f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542006c1d0 con 0x7f54200d0a30 2024-04-02T13:29:06.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.182+0000 7f542735f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f542006cb00 con 0x7f54200cfe30 2024-04-02T13:29:06.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.182+0000 7f542735f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f542006d3f0 con 0x7f542006cf90 2024-04-02T13:29:06.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.182+0000 7f54258fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f54200cfe30 0x7f54200d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.182+0000 7f54258fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f54200cfe30 0x7f54200d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51720/0 (socket says 172.21.15.44:51720) 2024-04-02T13:29:06.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.182+0000 7f54258fc640 1 -- 172.21.15.44:0/32754078 learned_addr learned my addr 172.21.15.44:0/32754078 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:06.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.183+0000 7f54248fa640 1 --2- 172.21.15.44:0/32754078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f542006cf90 0x7f54200cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.183+0000 7f54258fc640 1 -- 172.21.15.44:0/32754078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f542006cf90 msgr2=0x7f54200cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:06.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.183+0000 7f54258fc640 1 --2- 172.21.15.44:0/32754078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f542006cf90 0x7f54200cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.183+0000 7f54258fc640 1 -- 172.21.15.44:0/32754078 >> v1:172.21.15.69:6789/0 conn(0x7f54200d0a30 legacy=0x7f54200d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:06.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.183+0000 7f54258fc640 1 -- 172.21.15.44:0/32754078 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542006c4b0 con 0x7f54200cfe30 2024-04-02T13:29:06.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.183+0000 7f54258fc640 1 --2- 172.21.15.44:0/32754078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54200cfe30 0x7f54200d2040 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f5418009870 tx=0x7f541802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4eed9848671bd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:06.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.184+0000 7f541ffff640 1 -- 172.21.15.44:0/32754078 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418035020 con 0x7f54200cfe30 2024-04-02T13:29:06.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.184+0000 7f541ffff640 1 -- 172.21.15.44:0/32754078 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5418031640 con 0x7f54200cfe30 2024-04-02T13:29:06.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.184+0000 7f541ffff640 1 -- 172.21.15.44:0/32754078 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541802d040 con 0x7f54200cfe30 2024-04-02T13:29:06.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.184+0000 7f542735f640 1 -- 172.21.15.44:0/32754078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54200cfe30 msgr2=0x7f54200d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:06.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.184+0000 7f542735f640 1 --2- 172.21.15.44:0/32754078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54200cfe30 0x7f54200d2040 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f5418009870 tx=0x7f541802b8f0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.184+0000 7f542735f640 1 -- 172.21.15.44:0/32754078 shutdown_connections 2024-04-02T13:29:06.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.184+0000 7f542735f640 1 --2- 172.21.15.44:0/32754078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54200cfe30 0x7f54200d2040 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.184+0000 7f542735f640 1 --2- 172.21.15.44:0/32754078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f542006cf90 0x7f54200cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.184+0000 7f542735f640 1 -- 172.21.15.44:0/32754078 >> 172.21.15.44:0/32754078 conn(0x7f542009d470 msgr2=0x7f54200d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:06.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.184+0000 7f542735f640 1 -- 172.21.15.44:0/32754078 shutdown_connections 2024-04-02T13:29:06.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.184+0000 7f542735f640 1 -- 172.21.15.44:0/32754078 wait complete. 2024-04-02T13:29:06.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.185+0000 7f542735f640 1 Processor -- start 2024-04-02T13:29:06.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.185+0000 7f542735f640 1 -- start start 2024-04-02T13:29:06.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f542735f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f542006cf90 0x7f5420154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f542735f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54200cfe30 0x7f54200c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f542735f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54200d0a30 0x7f54200cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f542735f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f542015e870 con 0x7f54200d0a30 2024-04-02T13:29:06.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f542735f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f542015e6f0 con 0x7f54200cfe30 2024-04-02T13:29:06.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f542735f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f54200d4da0 con 0x7f542006cf90 2024-04-02T13:29:06.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f54258fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54200d0a30 0x7f54200cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f54258fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54200d0a30 0x7f54200cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51736/0 (socket says 172.21.15.44:51736) 2024-04-02T13:29:06.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f54258fc640 1 -- 172.21.15.44:0/2371547861 learned_addr learned my addr 172.21.15.44:0/2371547861 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:06.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f54248fa640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54200cfe30 0x7f54200c7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f54250fb640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f542006cf90 0x7f5420154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f54258fc640 1 -- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f542006cf90 msgr2=0x7f5420154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:06.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f54258fc640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f542006cf90 0x7f5420154440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f54258fc640 1 -- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54200cfe30 msgr2=0x7f54200c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:06.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f54258fc640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54200cfe30 0x7f54200c7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f54258fc640 1 -- 172.21.15.44:0/2371547861 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54200cdfb0 con 0x7f54200d0a30 2024-04-02T13:29:06.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.186+0000 7f54250fb640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f542006cf90 0x7f5420154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:06.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.187+0000 7f54258fc640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54200d0a30 0x7f54200cd640 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f5418009870 tx=0x7f54180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:06.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.187+0000 7f541dffb640 1 -- 172.21.15.44:0/2371547861 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418040070 con 0x7f54200d0a30 2024-04-02T13:29:06.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.187+0000 7f541dffb640 1 -- 172.21.15.44:0/2371547861 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54180399f0 con 0x7f54200d0a30 2024-04-02T13:29:06.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.187+0000 7f542735f640 1 -- 172.21.15.44:0/2371547861 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54200b3d80 con 0x7f54200d0a30 2024-04-02T13:29:06.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.187+0000 7f541dffb640 1 -- 172.21.15.44:0/2371547861 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418035040 con 0x7f54200d0a30 2024-04-02T13:29:06.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.187+0000 7f542735f640 1 -- 172.21.15.44:0/2371547861 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54200b4340 con 0x7f54200d0a30 2024-04-02T13:29:06.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.188+0000 7f542735f640 1 -- 172.21.15.44:0/2371547861 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54200684f0 con 0x7f54200d0a30 2024-04-02T13:29:06.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.188+0000 7f541dffb640 1 -- 172.21.15.44:0/2371547861 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5418038020 con 0x7f54200d0a30 2024-04-02T13:29:06.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.189+0000 7f541dffb640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f540c06c8f0 0x7f540c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.189+0000 7f541dffb640 1 -- 172.21.15.44:0/2371547861 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5507+0+0 (secure 0 0 0) 0x7f54180ad290 con 0x7f54200d0a30 2024-04-02T13:29:06.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.189+0000 7f54250fb640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f540c06c8f0 0x7f540c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.189+0000 7f54250fb640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f540c06c8f0 0x7f540c06edb0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f5410001d70 tx=0x7f5410009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:06.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.193+0000 7f541dffb640 1 -- 172.21.15.44:0/2371547861 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f541807b8d0 con 0x7f54200d0a30 2024-04-02T13:29:06.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.417+0000 7f542735f640 1 -- 172.21.15.44:0/2371547861 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54200d17c0 con 0x7f54200d0a30 2024-04-02T13:29:06.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.418+0000 7f541dffb640 1 -- 172.21.15.44:0/2371547861 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+3863 (secure 0 0 0) 0x7f541807b1d0 con 0x7f54200d0a30 2024-04-02T13:29:06.421 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:06.421 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":76,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2076636272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2076636272},{"type":"v1","addr":"172.21.15.44:6835","nonce":2076636272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/286589675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":286589675},{"type":"v1","addr":"172.21.15.69:6837","nonce":286589675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/4249055743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":4249055743},{"type":"v1","addr":"172.21.15.69:6836","nonce":4249055743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":8688,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6833/158636023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":158636023},{"type":"v1","addr":"172.21.15.44:6833","nonce":158636023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[]} 2024-04-02T13:29:06.421 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 76 2024-04-02T13:29:06.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.420+0000 7f542735f640 1 -- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f540c06c8f0 msgr2=0x7f540c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:06.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.420+0000 7f542735f640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f540c06c8f0 0x7f540c06edb0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f5410001d70 tx=0x7f5410009040 comp rx=0 tx=0).stop 2024-04-02T13:29:06.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.420+0000 7f542735f640 1 -- 172.21.15.44:0/2371547861 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54200d0a30 msgr2=0x7f54200cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:06.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.420+0000 7f542735f640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54200d0a30 0x7f54200cd640 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f5418009870 tx=0x7f54180046c0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.420+0000 7f542735f640 1 -- 172.21.15.44:0/2371547861 shutdown_connections 2024-04-02T13:29:06.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.420+0000 7f542735f640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f540c06c8f0 0x7f540c06edb0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.420+0000 7f542735f640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54200d0a30 0x7f54200cd640 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.420+0000 7f542735f640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54200cfe30 0x7f54200c7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.420+0000 7f542735f640 1 --2- 172.21.15.44:0/2371547861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f542006cf90 0x7f5420154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.420+0000 7f542735f640 1 -- 172.21.15.44:0/2371547861 >> 172.21.15.44:0/2371547861 conn(0x7f542009d470 msgr2=0x7f54200d28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:06.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.420+0000 7f542735f640 1 -- 172.21.15.44:0/2371547861 shutdown_connections 2024-04-02T13:29:06.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.420+0000 7f542735f640 1 -- 172.21.15.44:0/2371547861 wait complete. 2024-04-02T13:29:06.434 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T13:29:06.434 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T13:29:06.434 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T13:29:06.434 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:29:06.434 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T13:29:06.434 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T13:29:06.434 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T13:29:06.435 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:29:06.435 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T13:29:06.435 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T13:29:06.435 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T13:29:06.435 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:29:06.435 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T13:29:06.435 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T13:29:06.435 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T13:29:06.436 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:29:06.443 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T13:29:06.443 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T13:29:06.444 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T13:29:06.444 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T13:29:06.447 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T13:29:06.447 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T13:29:06.447 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T13:29:06.448 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T13:29:06.485 INFO:tasks.ceph.mds.a:Started 2024-04-02T13:29:06.486 DEBUG:teuthology.parallel:result is None 2024-04-02T13:29:06.486 INFO:tasks.ceph.mds.c:Started 2024-04-02T13:29:06.486 DEBUG:teuthology.parallel:result is None 2024-04-02T13:29:06.489 INFO:tasks.ceph.mds.b:Started 2024-04-02T13:29:06.489 DEBUG:teuthology.parallel:result is None 2024-04-02T13:29:06.490 INFO:tasks.ceph.mds.d:Started 2024-04-02T13:29:06.490 DEBUG:teuthology.parallel:result is None 2024-04-02T13:29:06.490 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T13:29:06.563 INFO:tasks.ceph.mds.c.smithi044.stdout:starting mds.c at 2024-04-02T13:29:06.563 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:29:06.573 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-04-02T13:29:06.574 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-04-02T13:29:06.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.595+0000 7f8828455640 1 Processor -- start 2024-04-02T13:29:06.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.595+0000 7f8828455640 1 -- start start 2024-04-02T13:29:06.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.595+0000 7f8828455640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f88200d0a30 0x7f882006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f8828455640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88200d0000 con 0x7f88200d0e10 2024-04-02T13:29:06.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f8828455640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f882006c1d0 con 0x7f88200d2040 2024-04-02T13:29:06.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f8828455640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f882006cb00 con 0x7f88200d0a30 2024-04-02T13:29:06.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88259f0640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f88200d0e10 0x7f88200cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32792/0 (socket says 172.21.15.44:32792) 2024-04-02T13:29:06.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88259f0640 1 -- 172.21.15.44:0/695085487 learned_addr learned my addr 172.21.15.44:0/695085487 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:06.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3504021898 0 0) 0x7f88200d0000 con 0x7f88200d0e10 2024-04-02T13:29:06.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f880c003520 con 0x7f88200d0e10 2024-04-02T13:29:06.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88261f1640 1 --2- 172.21.15.44:0/695085487 >> v2:172.21.15.69:3300/0 conn(0x7f88200d0a30 0x7f882006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2055605369 0 0) 0x7f880c003520 con 0x7f88200d0e10 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 >> v2:172.21.15.69:3300/0 conn(0x7f88200d0a30 msgr2=0x7f882006cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 --2- 172.21.15.44:0/695085487 >> v2:172.21.15.69:3300/0 conn(0x7f88200d0a30 0x7f882006cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 >> v1:172.21.15.69:6789/0 conn(0x7f88200d2040 legacy=0x7f88200d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f882006c4b0 con 0x7f88200d0e10 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f881c001670 con 0x7f88200d0e10 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 >> v1:172.21.15.44:6789/0 conn(0x7f88200d0e10 legacy=0x7f88200cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 --2- 172.21.15.44:0/695085487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f880c0039b0 0x7f880c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 --2- 172.21.15.44:0/695085487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f880c004800 0x7f880c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 --2- 172.21.15.44:0/695085487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f880c0051c0 0x7f880c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f882006cb00 con 0x7f880c004800 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f880c00c160 con 0x7f880c0051c0 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f880c00c360 con 0x7f880c0039b0 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.596+0000 7f88259f0640 1 --2- 172.21.15.44:0/695085487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f880c004800 0x7f880c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.597+0000 7f88261f1640 1 --2- 172.21.15.44:0/695085487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f880c0039b0 0x7f880c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.597+0000 7f88269f2640 1 --2- 172.21.15.44:0/695085487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f880c0051c0 0x7f880c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.597+0000 7f88259f0640 1 -- 172.21.15.44:0/695085487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f880c0039b0 msgr2=0x7f880c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.597+0000 7f88259f0640 1 --2- 172.21.15.44:0/695085487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f880c0039b0 0x7f880c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.597+0000 7f88259f0640 1 -- 172.21.15.44:0/695085487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f880c0051c0 msgr2=0x7f880c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.597+0000 7f88259f0640 1 --2- 172.21.15.44:0/695085487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f880c0051c0 0x7f880c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.597+0000 7f88259f0640 1 -- 172.21.15.44:0/695085487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880c003520 con 0x7f880c004800 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.597+0000 7f88269f2640 1 -- 172.21.15.44:0/695085487 reap_dead start 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.597+0000 7f88259f0640 1 --2- 172.21.15.44:0/695085487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f880c004800 0x7f880c004c80 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f882008e630 tx=0x7f880c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d3c9a7bb6bb627 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.598+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881c007d30 con 0x7f880c004800 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.598+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f881c030070 con 0x7f880c004800 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.598+0000 7f88251ef640 1 -- 172.21.15.44:0/695085487 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881c005d70 con 0x7f880c004800 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.598+0000 7f8828455640 1 -- 172.21.15.44:0/695085487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f880c004800 msgr2=0x7f880c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.598+0000 7f8828455640 1 --2- 172.21.15.44:0/695085487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f880c004800 0x7f880c004c80 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f882008e630 tx=0x7f880c005830 comp rx=0 tx=0).stop 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.598+0000 7f8828455640 1 -- 172.21.15.44:0/695085487 shutdown_connections 2024-04-02T13:29:06.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.598+0000 7f8828455640 1 --2- 172.21.15.44:0/695085487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f880c004800 0x7f880c004c80 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.598+0000 7f8828455640 1 -- 172.21.15.44:0/695085487 >> 172.21.15.44:0/695085487 conn(0x7f882009d470 msgr2=0x7f88200d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:06.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.598+0000 7f8828455640 1 -- 172.21.15.44:0/695085487 shutdown_connections 2024-04-02T13:29:06.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.598+0000 7f8828455640 1 -- 172.21.15.44:0/695085487 wait complete. 2024-04-02T13:29:06.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.599+0000 7f8828455640 1 Processor -- start 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.599+0000 7f8828455640 1 -- start start 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f8828455640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f880c004800 0x7f8820160de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f8828455640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88200b3ed0 0x7f88200b99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f8828455640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88200cc7a0 0x7f88200ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f8828455640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f882015e6f0 con 0x7f88200b3ed0 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f8828455640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f88200d1270 con 0x7f88200cc7a0 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f8828455640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f882015e870 con 0x7f880c004800 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f88259f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88200b3ed0 0x7f88200b99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f88259f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88200b3ed0 0x7f88200b99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51780/0 (socket says 172.21.15.44:51780) 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f88259f0640 1 -- 172.21.15.44:0/3820914733 learned_addr learned my addr 172.21.15.44:0/3820914733 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f88261f1640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f880c004800 0x7f8820160de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f88269f2640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88200cc7a0 0x7f88200ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f88259f0640 1 -- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f880c004800 msgr2=0x7f8820160de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:06.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f88259f0640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f880c004800 0x7f8820160de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f88259f0640 1 -- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88200cc7a0 msgr2=0x7f88200ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.600+0000 7f88259f0640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88200cc7a0 0x7f88200ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.601+0000 7f88259f0640 1 -- 172.21.15.44:0/3820914733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88200b4400 con 0x7f88200b3ed0 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.601+0000 7f88261f1640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f880c004800 0x7f8820160de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.601+0000 7f88269f2640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88200cc7a0 0x7f88200ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.601+0000 7f88259f0640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88200b3ed0 0x7f88200b99e0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f881c00c300 tx=0x7f881c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.601+0000 7f8816ffd640 1 -- 172.21.15.44:0/3820914733 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881c046070 con 0x7f88200b3ed0 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.601+0000 7f8816ffd640 1 -- 172.21.15.44:0/3820914733 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f881c004bf0 con 0x7f88200b3ed0 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.601+0000 7f8816ffd640 1 -- 172.21.15.44:0/3820914733 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881c007d10 con 0x7f88200b3ed0 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.601+0000 7f8828455640 1 -- 172.21.15.44:0/3820914733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88200b46e0 con 0x7f88200b3ed0 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.601+0000 7f8828455640 1 -- 172.21.15.44:0/3820914733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88200b4ca0 con 0x7f88200b3ed0 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.602+0000 7f8828455640 1 -- 172.21.15.44:0/3820914733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88200684f0 con 0x7f88200b3ed0 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.602+0000 7f8816ffd640 1 -- 172.21.15.44:0/3820914733 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f881c007070 con 0x7f88200b3ed0 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.603+0000 7f8816ffd640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f880406c920 0x7f880406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:06.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.603+0000 7f8816ffd640 1 -- 172.21.15.44:0/3820914733 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5507+0+0 (secure 0 0 0) 0x7f881c0acf20 con 0x7f88200b3ed0 2024-04-02T13:29:06.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.603+0000 7f88261f1640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f880406c920 0x7f880406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:06.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.603+0000 7f88261f1640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f880406c920 0x7f880406ede0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f882006ad00 tx=0x7f8810008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:06.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.607+0000 7f8816ffd640 1 -- 172.21.15.44:0/3820914733 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f881c07ad30 con 0x7f88200b3ed0 2024-04-02T13:29:06.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:06.803+0000 7f8828455640 1 -- 172.21.15.44:0/3820914733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f88200ce240 con 0x7f88200b3ed0 2024-04-02T13:29:07.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.528+0000 7f8816ffd640 1 -- 172.21.15.44:0/3820914733 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v100) ==== 97+0+0 (secure 0 0 0) 0x7f881c07f030 con 0x7f88200b3ed0 2024-04-02T13:29:07.531 INFO:teuthology.orchestra.run.smithi044.stderr: removed all blocklist entries 2024-04-02T13:29:07.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.531+0000 7f8828455640 1 -- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f880406c920 msgr2=0x7f880406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:07.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.531+0000 7f8828455640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f880406c920 0x7f880406ede0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f882006ad00 tx=0x7f8810008040 comp rx=0 tx=0).stop 2024-04-02T13:29:07.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.531+0000 7f8828455640 1 -- 172.21.15.44:0/3820914733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88200b3ed0 msgr2=0x7f88200b99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:07.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.531+0000 7f8828455640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88200b3ed0 0x7f88200b99e0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f881c00c300 tx=0x7f881c02e070 comp rx=0 tx=0).stop 2024-04-02T13:29:07.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.531+0000 7f8828455640 1 -- 172.21.15.44:0/3820914733 shutdown_connections 2024-04-02T13:29:07.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.531+0000 7f8828455640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f880406c920 0x7f880406ede0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.531+0000 7f8828455640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88200cc7a0 0x7f88200ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.531+0000 7f8828455640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88200b3ed0 0x7f88200b99e0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.531+0000 7f8828455640 1 --2- 172.21.15.44:0/3820914733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f880c004800 0x7f8820160de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.531+0000 7f8828455640 1 -- 172.21.15.44:0/3820914733 >> 172.21.15.44:0/3820914733 conn(0x7f882009d470 msgr2=0x7f88200d2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:07.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.531+0000 7f8828455640 1 -- 172.21.15.44:0/3820914733 shutdown_connections 2024-04-02T13:29:07.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.531+0000 7f8828455640 1 -- 172.21.15.44:0/3820914733 wait complete. 2024-04-02T13:29:07.550 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T13:29:07.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.655+0000 7f87ec2dc640 1 Processor -- start 2024-04-02T13:29:07.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.655+0000 7f87ec2dc640 1 -- start start 2024-04-02T13:29:07.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.656+0000 7f87ec2dc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87e40d0c50 0x7f87e40d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:07.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.656+0000 7f87ec2dc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f87e406ce40 0x7f87e406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:07.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.656+0000 7f87ec2dc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87e40cc600 con 0x7f87e415e4a0 2024-04-02T13:29:07.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.656+0000 7f87ec2dc640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f87e40af300 con 0x7f87e406ce40 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.656+0000 7f87ec2dc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f87e40cc8e0 con 0x7f87e40d0c50 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.656+0000 7f87ea879640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f87e406ce40 0x7f87e406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.656+0000 7f87ea078640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f87e415e4a0 0x7f87e40aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32804/0 (socket says 172.21.15.44:32804) 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.656+0000 7f87ea078640 1 -- 172.21.15.44:0/1217803241 learned_addr learned my addr 172.21.15.44:0/1217803241 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.656+0000 7f87e9076640 1 -- 172.21.15.44:0/1217803241 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3574300913 0 0) 0x7f87e40cc600 con 0x7f87e415e4a0 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.656+0000 7f87e9076640 1 -- 172.21.15.44:0/1217803241 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87cc003520 con 0x7f87e415e4a0 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.656+0000 7f87e9877640 1 --2- 172.21.15.44:0/1217803241 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87e40d0c50 0x7f87e40d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.657+0000 7f87ea879640 1 -- 172.21.15.44:0/1217803241 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87e40d0c50 msgr2=0x7f87e40d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.657+0000 7f87ea879640 1 --2- 172.21.15.44:0/1217803241 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87e40d0c50 0x7f87e40d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.657+0000 7f87ea879640 1 -- 172.21.15.44:0/1217803241 >> v1:172.21.15.44:6789/0 conn(0x7f87e415e4a0 legacy=0x7f87e40aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.657+0000 7f87ea879640 1 -- 172.21.15.44:0/1217803241 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e40ccbc0 con 0x7f87e406ce40 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.657+0000 7f87e9877640 1 --2- 172.21.15.44:0/1217803241 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87e40d0c50 0x7f87e40d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.657+0000 7f87ea879640 1 --2- 172.21.15.44:0/1217803241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87e406ce40 0x7f87e406d2c0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009d90 tx=0x7f87d402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=414a72c9784352ab server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:07.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.657+0000 7f87e9076640 1 -- 172.21.15.44:0/1217803241 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4039070 con 0x7f87e406ce40 2024-04-02T13:29:07.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.657+0000 7f87e9076640 1 -- 172.21.15.44:0/1217803241 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87d402cc50 con 0x7f87e406ce40 2024-04-02T13:29:07.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.658+0000 7f87e9076640 1 -- 172.21.15.44:0/1217803241 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4034440 con 0x7f87e406ce40 2024-04-02T13:29:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.658+0000 7f87ec2dc640 1 -- 172.21.15.44:0/1217803241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87e406ce40 msgr2=0x7f87e406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.658+0000 7f87ec2dc640 1 --2- 172.21.15.44:0/1217803241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87e406ce40 0x7f87e406d2c0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009d90 tx=0x7f87d402bea0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.658+0000 7f87ec2dc640 1 -- 172.21.15.44:0/1217803241 shutdown_connections 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.658+0000 7f87ec2dc640 1 --2- 172.21.15.44:0/1217803241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87e406ce40 0x7f87e406d2c0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.658+0000 7f87ec2dc640 1 --2- 172.21.15.44:0/1217803241 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87e40d0c50 0x7f87e40d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.658+0000 7f87ec2dc640 1 -- 172.21.15.44:0/1217803241 >> 172.21.15.44:0/1217803241 conn(0x7f87e409aa20 msgr2=0x7f87e409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.658+0000 7f87ec2dc640 1 -- 172.21.15.44:0/1217803241 shutdown_connections 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.658+0000 7f87ec2dc640 1 -- 172.21.15.44:0/1217803241 wait complete. 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.659+0000 7f87ec2dc640 1 Processor -- start 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.659+0000 7f87ec2dc640 1 -- start start 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.660+0000 7f87ec2dc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87e406ce40 0x7f87e415f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.660+0000 7f87ec2dc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87e40d0c50 0x7f87e4159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.660+0000 7f87ec2dc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87e415e4a0 0x7f87e40cb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.660+0000 7f87ec2dc640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f87e40cd460 con 0x7f87e406ce40 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.660+0000 7f87ec2dc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f87e40cd180 con 0x7f87e40d0c50 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.660+0000 7f87ec2dc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f87e415e880 con 0x7f87e415e4a0 2024-04-02T13:29:07.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.660+0000 7f87ea078640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87e406ce40 0x7f87e415f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.660+0000 7f87ea078640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87e406ce40 0x7f87e415f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51804/0 (socket says 172.21.15.44:51804) 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.660+0000 7f87ea078640 1 -- 172.21.15.44:0/759396730 learned_addr learned my addr 172.21.15.44:0/759396730 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.660+0000 7f87ea879640 1 --2- 172.21.15.44:0/759396730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87e415e4a0 0x7f87e40cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.661+0000 7f87ea078640 1 -- 172.21.15.44:0/759396730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87e415e4a0 msgr2=0x7f87e40cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.661+0000 7f87ea078640 1 --2- 172.21.15.44:0/759396730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87e415e4a0 0x7f87e40cb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.661+0000 7f87ea078640 1 -- 172.21.15.44:0/759396730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87e40d0c50 msgr2=0x7f87e4159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.661+0000 7f87ea078640 1 --2- 172.21.15.44:0/759396730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87e40d0c50 0x7f87e4159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.661+0000 7f87ea078640 1 -- 172.21.15.44:0/759396730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e40c9c40 con 0x7f87e406ce40 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.661+0000 7f87ea078640 1 --2- 172.21.15.44:0/759396730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87e406ce40 0x7f87e415f480 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00c390 tx=0x7f87dc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.661+0000 7f87e2ffd640 1 -- 172.21.15.44:0/759396730 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc017070 con 0x7f87e406ce40 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.661+0000 7f87e2ffd640 1 -- 172.21.15.44:0/759396730 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87dc005ce0 con 0x7f87e406ce40 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.661+0000 7f87e2ffd640 1 -- 172.21.15.44:0/759396730 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc0072d0 con 0x7f87e406ce40 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.661+0000 7f87ec2dc640 1 -- 172.21.15.44:0/759396730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e415feb0 con 0x7f87e406ce40 2024-04-02T13:29:07.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.661+0000 7f87ec2dc640 1 -- 172.21.15.44:0/759396730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e4160470 con 0x7f87e406ce40 2024-04-02T13:29:07.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.662+0000 7f87ec2dc640 1 -- 172.21.15.44:0/759396730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87e4160670 con 0x7f87e406ce40 2024-04-02T13:29:07.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.662+0000 7f87e2ffd640 1 -- 172.21.15.44:0/759396730 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f87dc01a070 con 0x7f87e406ce40 2024-04-02T13:29:07.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.663+0000 7f87e2ffd640 1 --2- 172.21.15.44:0/759396730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f87d006c8f0 0x7f87d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:07.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.663+0000 7f87e2ffd640 1 -- 172.21.15.44:0/759396730 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5249+0+0 (secure 0 0 0) 0x7f87dc08aba0 con 0x7f87e406ce40 2024-04-02T13:29:07.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.663+0000 7f87e9877640 1 --2- 172.21.15.44:0/759396730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f87d006c8f0 0x7f87d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:07.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.664+0000 7f87e9877640 1 --2- 172.21.15.44:0/759396730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f87d006c8f0 0x7f87d006edb0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f87d8006fd0 tx=0x7f87d8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:07.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.667+0000 7f87e2ffd640 1 -- 172.21.15.44:0/759396730 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f87e4160670 con 0x7f87e406ce40 2024-04-02T13:29:07.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.925+0000 7f87ec2dc640 1 -- 172.21.15.44:0/759396730 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f87e40684f0 con 0x7f87e406ce40 2024-04-02T13:29:07.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.926+0000 7f87e2ffd640 1 -- 172.21.15.44:0/759396730 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5304 (secure 0 0 0) 0x7f87dc058bd0 con 0x7f87e406ce40 2024-04-02T13:29:07.928 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout: "auth_dump": [ 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.a", 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmVhZrMhAAklqx5uKCFA5kiqYwyLBKpg==", 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.929 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.b", 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmalQ+AxAA2SqH9CX/66OIXR7SDID97w==", 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.c", 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxm8BLKORAAjMIrmdQ/bXbPk2sAnYJWTw==", 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:29:07.930 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.d", 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmNoaRChAAEqxWmBjBWG79Tkj6dU8FQg==", 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.931 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.932 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.932 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.0", 2024-04-02T13:29:07.932 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDfBQxm4GP9KxAA/AmOunAfVaczAWRGVXvFVg==", 2024-04-02T13:29:07.932 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.932 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:29:07.932 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:29:07.932 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.932 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.932 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.932 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.1", 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDiBQxmDL/nKRAADF2C30+IGAXL9XwDQDKCUA==", 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.2", 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDlBQxm1gOCKBAAuYxN9jNHuRYEqc8WrsYErQ==", 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:29:07.933 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.3", 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDoBQxmLLFQJxAAFMCzphGU7TOt4nC0Nt+y+g==", 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.4", 2024-04-02T13:29:07.934 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDuBQxmtk8QLhAAR4gm0irR6C3D2SkhFjvDKQ==", 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.5", 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDxBQxm6Ee/LBAAX5h+9ML70ztw2QH/50OzOg==", 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.935 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.6", 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD0BQxm35hwKxAAvsgMmiXdLo+c7wlvb9RtVQ==", 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.7", 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD3BQxmYLBOKhAA1vuSOBDcFf2zT78eoTHtHQ==", 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.936 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.0", 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A==", 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow r", 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.1", 2024-04-02T13:29:07.937 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxm3sYsGBAAeRCNbP5qiQ8YnRVsDoaaFQ==", 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.2", 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmKNS5FBAAKvsCNmuG+fI1D/vwu6MC5w==", 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:29:07.938 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.3", 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmslsSHhAAPsAfALW7RimJDyXdOa2wbQ==", 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.939 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.admin", 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmq+VcFRAA9CgeH75Q0sXf27cZdJthJg==", 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.x", 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmR5ICLxAAtQeg/gq0OAq6e0mroBZVxQ==", 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:29:07.940 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.941 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.941 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:07.941 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:07.941 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.y", 2024-04-02T13:29:07.941 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmxoquKhAAE5fMnJIuCy5hGJ1aEAx8Vg==", 2024-04-02T13:29:07.941 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:29:07.941 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:29:07.941 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:29:07.941 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:29:07.941 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.941 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:07.941 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-02T13:29:07.941 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:29:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.928+0000 7f87ec2dc640 1 -- 172.21.15.44:0/759396730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f87d006c8f0 msgr2=0x7f87d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.928+0000 7f87ec2dc640 1 --2- 172.21.15.44:0/759396730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f87d006c8f0 0x7f87d006edb0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f87d8006fd0 tx=0x7f87d8008040 comp rx=0 tx=0).stop 2024-04-02T13:29:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.928+0000 7f87ec2dc640 1 -- 172.21.15.44:0/759396730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87e406ce40 msgr2=0x7f87e415f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.928+0000 7f87ec2dc640 1 --2- 172.21.15.44:0/759396730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87e406ce40 0x7f87e415f480 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00c390 tx=0x7f87dc00c850 comp rx=0 tx=0).stop 2024-04-02T13:29:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.928+0000 7f87ec2dc640 1 -- 172.21.15.44:0/759396730 shutdown_connections 2024-04-02T13:29:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.928+0000 7f87ec2dc640 1 --2- 172.21.15.44:0/759396730 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f87d006c8f0 0x7f87d006edb0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.928+0000 7f87ec2dc640 1 --2- 172.21.15.44:0/759396730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87e415e4a0 0x7f87e40cb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.928+0000 7f87ec2dc640 1 --2- 172.21.15.44:0/759396730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87e40d0c50 0x7f87e4159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.928+0000 7f87ec2dc640 1 --2- 172.21.15.44:0/759396730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87e406ce40 0x7f87e415f480 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.928+0000 7f87ec2dc640 1 -- 172.21.15.44:0/759396730 >> 172.21.15.44:0/759396730 conn(0x7f87e409aa20 msgr2=0x7f87e40704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.928+0000 7f87ec2dc640 1 -- 172.21.15.44:0/759396730 shutdown_connections 2024-04-02T13:29:07.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:07.928+0000 7f87ec2dc640 1 -- 172.21.15.44:0/759396730 wait complete. 2024-04-02T13:29:07.944 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:29:08.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.042+0000 7fcf00122640 1 Processor -- start 2024-04-02T13:29:08.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.043+0000 7fcf00122640 1 -- start start 2024-04-02T13:29:08.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.043+0000 7fcf00122640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcef815e550 0x7fcef806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.043+0000 7fcf00122640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcef806d530 0x7fcef815e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.043+0000 7fcf00122640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef815fb50 con 0x7fcef815ee70 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.043+0000 7fcf00122640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fcef806d970 con 0x7fcef806d530 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.043+0000 7fcf00122640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcef815f300 con 0x7fcef815e550 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.043+0000 7fcefd6bd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcef806d530 0x7fcef815e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.044+0000 7fcefd6bd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcef806d530 0x7fcef815e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51810/0 (socket says 172.21.15.44:51810) 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.044+0000 7fcefd6bd640 1 -- 172.21.15.44:0/837106840 learned_addr learned my addr 172.21.15.44:0/837106840 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.044+0000 7fcefdebe640 1 --2- 172.21.15.44:0/837106840 >> v2:172.21.15.69:3300/0 conn(0x7fcef815e550 0x7fcef806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.044+0000 7fcefd6bd640 1 -- 172.21.15.44:0/837106840 >> v2:172.21.15.69:3300/0 conn(0x7fcef815e550 msgr2=0x7fcef806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.044+0000 7fcefd6bd640 1 --2- 172.21.15.44:0/837106840 >> v2:172.21.15.69:3300/0 conn(0x7fcef815e550 0x7fcef806cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.044+0000 7fcefd6bd640 1 -- 172.21.15.44:0/837106840 >> v1:172.21.15.69:6789/0 conn(0x7fcef815ee70 legacy=0x7fcef80ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.044+0000 7fcefd6bd640 1 -- 172.21.15.44:0/837106840 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef80d1ac0 con 0x7fcef806d530 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.044+0000 7fcefdebe640 1 --2- 172.21.15.44:0/837106840 >> v2:172.21.15.69:3300/0 conn(0x7fcef815e550 0x7fcef806cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.045+0000 7fcefd6bd640 1 --2- 172.21.15.44:0/837106840 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcef806d530 0x7fcef815e930 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fcef80b49c0 tx=0x7fceec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9cd76dd394a148 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:08.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.045+0000 7fcefcebc640 1 -- 172.21.15.44:0/837106840 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec034070 con 0x7fcef806d530 2024-04-02T13:29:08.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.045+0000 7fcefcebc640 1 -- 172.21.15.44:0/837106840 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceec030860 con 0x7fcef806d530 2024-04-02T13:29:08.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.045+0000 7fcefcebc640 1 -- 172.21.15.44:0/837106840 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec030c30 con 0x7fcef806d530 2024-04-02T13:29:08.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.045+0000 7fcf00122640 1 -- 172.21.15.44:0/837106840 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcef806d530 msgr2=0x7fcef815e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.045+0000 7fcf00122640 1 --2- 172.21.15.44:0/837106840 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcef806d530 0x7fcef815e930 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fcef80b49c0 tx=0x7fceec02b9a0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.045+0000 7fcf00122640 1 -- 172.21.15.44:0/837106840 shutdown_connections 2024-04-02T13:29:08.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.045+0000 7fcf00122640 1 --2- 172.21.15.44:0/837106840 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcef806d530 0x7fcef815e930 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.045+0000 7fcf00122640 1 --2- 172.21.15.44:0/837106840 >> v2:172.21.15.69:3300/0 conn(0x7fcef815e550 0x7fcef806cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.045+0000 7fcf00122640 1 -- 172.21.15.44:0/837106840 >> 172.21.15.44:0/837106840 conn(0x7fcef809aa20 msgr2=0x7fcef809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:08.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.045+0000 7fcf00122640 1 -- 172.21.15.44:0/837106840 shutdown_connections 2024-04-02T13:29:08.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.045+0000 7fcf00122640 1 -- 172.21.15.44:0/837106840 wait complete. 2024-04-02T13:29:08.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.047+0000 7fcf00122640 1 Processor -- start 2024-04-02T13:29:08.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.047+0000 7fcf00122640 1 -- start start 2024-04-02T13:29:08.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.047+0000 7fcf00122640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcef806d530 0x7fcef80b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.047+0000 7fcf00122640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcef815e550 0x7fcef815b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.047+0000 7fcf00122640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcef815ee70 0x7fcef80cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.047+0000 7fcf00122640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcef815f870 con 0x7fcef815e550 2024-04-02T13:29:08.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.047+0000 7fcf00122640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcef815f590 con 0x7fcef806d530 2024-04-02T13:29:08.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.047+0000 7fcf00122640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcef80d02b0 con 0x7fcef815ee70 2024-04-02T13:29:08.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.047+0000 7fcefd6bd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcef815e550 0x7fcef815b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcefd6bd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcef815e550 0x7fcef815b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51814/0 (socket says 172.21.15.44:51814) 2024-04-02T13:29:08.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcefd6bd640 1 -- 172.21.15.44:0/2134196976 learned_addr learned my addr 172.21.15.44:0/2134196976 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:08.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcefdebe640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcef806d530 0x7fcef80b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcefd6bd640 1 -- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcef815ee70 msgr2=0x7fcef80cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcefe6bf640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcef815ee70 0x7fcef80cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcefd6bd640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcef815ee70 0x7fcef80cb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcefd6bd640 1 -- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcef806d530 msgr2=0x7fcef80b5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcefd6bd640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcef806d530 0x7fcef80b5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcefd6bd640 1 -- 172.21.15.44:0/2134196976 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef815b630 con 0x7fcef815e550 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcefe6bf640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcef815ee70 0x7fcef80cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcefdebe640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcef806d530 0x7fcef80b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcefd6bd640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcef815e550 0x7fcef815b0f0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fceec02b9e0 tx=0x7fceec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcef67fc640 1 -- 172.21.15.44:0/2134196976 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec034040 con 0x7fcef815e550 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcef67fc640 1 -- 172.21.15.44:0/2134196976 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceec0433f0 con 0x7fcef815e550 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcef67fc640 1 -- 172.21.15.44:0/2134196976 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec0437e0 con 0x7fcef815e550 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcf00122640 1 -- 172.21.15.44:0/2134196976 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef81f74a0 con 0x7fcef815e550 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.048+0000 7fcf00122640 1 -- 172.21.15.44:0/2134196976 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef81f7a60 con 0x7fcef815e550 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.049+0000 7fcf00122640 1 -- 172.21.15.44:0/2134196976 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef80684f0 con 0x7fcef815e550 2024-04-02T13:29:08.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.050+0000 7fcef67fc640 1 -- 172.21.15.44:0/2134196976 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fceec037070 con 0x7fcef815e550 2024-04-02T13:29:08.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.050+0000 7fcef67fc640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcee406c920 0x7fcee406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.050+0000 7fcef67fc640 1 -- 172.21.15.44:0/2134196976 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5249+0+0 (secure 0 0 0) 0x7fceec0adbf0 con 0x7fcef815e550 2024-04-02T13:29:08.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.050+0000 7fcefdebe640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcee406c920 0x7fcee406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.051+0000 7fcefdebe640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcee406c920 0x7fcee406ede0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fcef0006fd0 tx=0x7fcef0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:08.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.055+0000 7fcef67fc640 1 -- 172.21.15.44:0/2134196976 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fceec03e070 con 0x7fcef815e550 2024-04-02T13:29:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.240+0000 7fcf00122640 1 -- 172.21.15.44:0/2134196976 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcef8067d70 con 0x7fcef815e550 2024-04-02T13:29:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.241+0000 7fcef67fc640 1 -- 172.21.15.44:0/2134196976 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+13118 (secure 0 0 0) 0x7fceec07c390 con 0x7fcef815e550 2024-04-02T13:29:08.243 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:08.243 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":102,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:29:07.543291+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:29:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.242+0000 7fcf00122640 1 -- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcee406c920 msgr2=0x7fcee406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.242+0000 7fcf00122640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcee406c920 0x7fcee406ede0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fcef0006fd0 tx=0x7fcef0008040 comp rx=0 tx=0).stop 2024-04-02T13:29:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.242+0000 7fcf00122640 1 -- 172.21.15.44:0/2134196976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcef815e550 msgr2=0x7fcef815b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.242+0000 7fcf00122640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcef815e550 0x7fcef815b0f0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fceec02b9e0 tx=0x7fceec0046c0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.242+0000 7fcf00122640 1 -- 172.21.15.44:0/2134196976 shutdown_connections 2024-04-02T13:29:08.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.242+0000 7fcf00122640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcee406c920 0x7fcee406ede0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.242+0000 7fcf00122640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcef815ee70 0x7fcef80cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.242+0000 7fcf00122640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcef815e550 0x7fcef815b0f0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.242+0000 7fcf00122640 1 --2- 172.21.15.44:0/2134196976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcef806d530 0x7fcef80b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.242+0000 7fcf00122640 1 -- 172.21.15.44:0/2134196976 >> 172.21.15.44:0/2134196976 conn(0x7fcef809aa20 msgr2=0x7fcef80cdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:08.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.243+0000 7fcf00122640 1 -- 172.21.15.44:0/2134196976 shutdown_connections 2024-04-02T13:29:08.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.243+0000 7fcf00122640 1 -- 172.21.15.44:0/2134196976 wait complete. 2024-04-02T13:29:08.256 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:29:08.256 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:29:08.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.358+0000 7f61ac57c640 1 Processor -- start 2024-04-02T13:29:08.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.358+0000 7f61ac57c640 1 -- start start 2024-04-02T13:29:08.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.359+0000 7f61ac57c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a415c220 0x7f61a40b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.359+0000 7f61ac57c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f61a40d03d0 0x7f61a40d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.359+0000 7f61ac57c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61a40cca50 con 0x7f61a40b1440 2024-04-02T13:29:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.359+0000 7f61ac57c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f61a40d0d70 con 0x7f61a40d03d0 2024-04-02T13:29:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.359+0000 7f61ac57c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f61a40ccd60 con 0x7f61a415c220 2024-04-02T13:29:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.359+0000 7f61aab19640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f61a40d03d0 0x7f61a40d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.359+0000 7f61aab19640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f61a40d03d0 0x7f61a40d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51818/0 (socket says 172.21.15.44:51818) 2024-04-02T13:29:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.359+0000 7f61a9b17640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f61a40b1440 0x7f61a40ce780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32808/0 (socket says 172.21.15.44:32808) 2024-04-02T13:29:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.359+0000 7f61aab19640 1 -- 172.21.15.44:0/118865325 learned_addr learned my addr 172.21.15.44:0/118865325 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.359+0000 7f61aa318640 1 --2- 172.21.15.44:0/118865325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a415c220 0x7f61a40b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.360+0000 7f61aab19640 1 -- 172.21.15.44:0/118865325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a415c220 msgr2=0x7f61a40b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.360+0000 7f61aab19640 1 --2- 172.21.15.44:0/118865325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a415c220 0x7f61a40b1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.360+0000 7f61aab19640 1 -- 172.21.15.44:0/118865325 >> v1:172.21.15.44:6789/0 conn(0x7f61a40b1440 legacy=0x7f61a40ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.360+0000 7f61aab19640 1 -- 172.21.15.44:0/118865325 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a40cef40 con 0x7f61a40d03d0 2024-04-02T13:29:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.360+0000 7f61a9316640 1 -- 172.21.15.44:0/118865325 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 340884789 0 0) 0x7f6194001110 con 0x7f61a40b1440 2024-04-02T13:29:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.360+0000 7f61aa318640 1 --2- 172.21.15.44:0/118865325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a415c220 0x7f61a40b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:08.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.360+0000 7f61aab19640 1 --2- 172.21.15.44:0/118865325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61a40d03d0 0x7f61a40d0830 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f619c009870 tx=0x7f619c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87bd0c64f9ff596 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:08.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.360+0000 7f61a9316640 1 -- 172.21.15.44:0/118865325 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c034070 con 0x7f61a40d03d0 2024-04-02T13:29:08.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.360+0000 7f61a9316640 1 -- 172.21.15.44:0/118865325 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f619c030840 con 0x7f61a40d03d0 2024-04-02T13:29:08.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.360+0000 7f61a9316640 1 -- 172.21.15.44:0/118865325 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c030c10 con 0x7f61a40d03d0 2024-04-02T13:29:08.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.361+0000 7f61ac57c640 1 -- 172.21.15.44:0/118865325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61a40d03d0 msgr2=0x7f61a40d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.361+0000 7f61ac57c640 1 --2- 172.21.15.44:0/118865325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61a40d03d0 0x7f61a40d0830 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f619c009870 tx=0x7f619c02b980 comp rx=0 tx=0).stop 2024-04-02T13:29:08.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.361+0000 7f61ac57c640 1 -- 172.21.15.44:0/118865325 shutdown_connections 2024-04-02T13:29:08.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.361+0000 7f61ac57c640 1 --2- 172.21.15.44:0/118865325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61a40d03d0 0x7f61a40d0830 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.361+0000 7f61ac57c640 1 --2- 172.21.15.44:0/118865325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a415c220 0x7f61a40b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.361+0000 7f61ac57c640 1 -- 172.21.15.44:0/118865325 >> 172.21.15.44:0/118865325 conn(0x7f61a409aa20 msgr2=0x7f61a409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:08.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.361+0000 7f61ac57c640 1 -- 172.21.15.44:0/118865325 shutdown_connections 2024-04-02T13:29:08.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.361+0000 7f61ac57c640 1 -- 172.21.15.44:0/118865325 wait complete. 2024-04-02T13:29:08.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.362+0000 7f61ac57c640 1 Processor -- start 2024-04-02T13:29:08.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.362+0000 7f61ac57c640 1 -- start start 2024-04-02T13:29:08.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61ac57c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61a40b1440 0x7f61a415d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61ac57c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61a40d03d0 0x7f61a40c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61ac57c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a415c220 0x7f61a41513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61ac57c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f61a415c600 con 0x7f61a40b1440 2024-04-02T13:29:08.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61ac57c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f61a40cf500 con 0x7f61a40d03d0 2024-04-02T13:29:08.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61ac57c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f61a40cd1f0 con 0x7f61a415c220 2024-04-02T13:29:08.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61aa318640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61a40b1440 0x7f61a415d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61aa318640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61a40b1440 0x7f61a415d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51830/0 (socket says 172.21.15.44:51830) 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61aa318640 1 -- 172.21.15.44:0/3614712540 learned_addr learned my addr 172.21.15.44:0/3614712540 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61aab19640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a415c220 0x7f61a41513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61a9b17640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61a40d03d0 0x7f61a40c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61aa318640 1 -- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a415c220 msgr2=0x7f61a41513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61aa318640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a415c220 0x7f61a41513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61aa318640 1 -- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61a40d03d0 msgr2=0x7f61a40c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61aa318640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61a40d03d0 0x7f61a40c68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61aa318640 1 -- 172.21.15.44:0/3614712540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a415e150 con 0x7f61a40b1440 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61aab19640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a415c220 0x7f61a41513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.363+0000 7f61aa318640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61a40b1440 0x7f61a415d9e0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009fd0 tx=0x7f61a000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.364+0000 7f619affd640 1 -- 172.21.15.44:0/3614712540 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a001a070 con 0x7f61a40b1440 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.364+0000 7f619affd640 1 -- 172.21.15.44:0/3614712540 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61a0003bb0 con 0x7f61a40b1440 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.364+0000 7f619affd640 1 -- 172.21.15.44:0/3614712540 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a0004030 con 0x7f61a40b1440 2024-04-02T13:29:08.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.364+0000 7f61ac57c640 1 -- 172.21.15.44:0/3614712540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a415e430 con 0x7f61a40b1440 2024-04-02T13:29:08.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.364+0000 7f61ac57c640 1 -- 172.21.15.44:0/3614712540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a415e9f0 con 0x7f61a40b1440 2024-04-02T13:29:08.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.365+0000 7f61ac57c640 1 -- 172.21.15.44:0/3614712540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61a40684f0 con 0x7f61a40b1440 2024-04-02T13:29:08.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.365+0000 7f619affd640 1 -- 172.21.15.44:0/3614712540 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f61a001d070 con 0x7f61a40b1440 2024-04-02T13:29:08.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.365+0000 7f619affd640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f618806c8f0 0x7f618806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.366+0000 7f619affd640 1 -- 172.21.15.44:0/3614712540 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5249+0+0 (secure 0 0 0) 0x7f61a008bad0 con 0x7f61a40b1440 2024-04-02T13:29:08.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.366+0000 7f61a9b17640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f618806c8f0 0x7f618806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.366+0000 7f61a9b17640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f618806c8f0 0x7f618806edb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6194000f60 tx=0x7f6194007970 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:08.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.370+0000 7f619affd640 1 -- 172.21.15.44:0/3614712540 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f61a0010070 con 0x7f61a40b1440 2024-04-02T13:29:08.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.556+0000 7f61ac57c640 1 -- 172.21.15.44:0/3614712540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f61a406fbe0 con 0x7f61a40b1440 2024-04-02T13:29:08.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.556+0000 7f619affd640 1 -- 172.21.15.44:0/3614712540 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) ==== 93+0+10 (secure 0 0 0) 0x7f61a0059b00 con 0x7f61a40b1440 2024-04-02T13:29:08.559 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:29:08.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.558+0000 7f61ac57c640 1 -- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f618806c8f0 msgr2=0x7f618806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.558+0000 7f61ac57c640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f618806c8f0 0x7f618806edb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6194000f60 tx=0x7f6194007970 comp rx=0 tx=0).stop 2024-04-02T13:29:08.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.558+0000 7f61ac57c640 1 -- 172.21.15.44:0/3614712540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61a40b1440 msgr2=0x7f61a415d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.558+0000 7f61ac57c640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61a40b1440 0x7f61a415d9e0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009fd0 tx=0x7f61a000d560 comp rx=0 tx=0).stop 2024-04-02T13:29:08.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.558+0000 7f61ac57c640 1 -- 172.21.15.44:0/3614712540 shutdown_connections 2024-04-02T13:29:08.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.558+0000 7f61ac57c640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f618806c8f0 0x7f618806edb0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.558+0000 7f61ac57c640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a415c220 0x7f61a41513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.558+0000 7f61ac57c640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61a40d03d0 0x7f61a40c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.558+0000 7f61ac57c640 1 --2- 172.21.15.44:0/3614712540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61a40b1440 0x7f61a415d9e0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.558+0000 7f61ac57c640 1 -- 172.21.15.44:0/3614712540 >> 172.21.15.44:0/3614712540 conn(0x7f61a409aa20 msgr2=0x7f61a40cd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:08.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.558+0000 7f61ac57c640 1 -- 172.21.15.44:0/3614712540 shutdown_connections 2024-04-02T13:29:08.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.558+0000 7f61ac57c640 1 -- 172.21.15.44:0/3614712540 wait complete. 2024-04-02T13:29:08.572 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T13:29:08.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.671+0000 7f35f9504640 1 Processor -- start 2024-04-02T13:29:08.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.671+0000 7f35f9504640 1 -- start start 2024-04-02T13:29:08.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.672+0000 7f35f9504640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f35f415e0f0 0x7f35f40d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.672+0000 7f35f9504640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35f406bbe0 0x7f35f40add60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.672+0000 7f35f9504640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35f406dfe0 con 0x7f35f40d0630 2024-04-02T13:29:08.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.672+0000 7f35f9504640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f35f406e2c0 con 0x7f35f415e0f0 2024-04-02T13:29:08.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.672+0000 7f35f9504640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f35f406e5a0 con 0x7f35f406bbe0 2024-04-02T13:29:08.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.672+0000 7f35f2ffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f35f415e0f0 0x7f35f40d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.672+0000 7f35f37fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35f406bbe0 0x7f35f40add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.672+0000 7f35ebfff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f35f40d0630 0x7f35f406b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:51800/0 (socket says 172.21.15.44:51800) 2024-04-02T13:29:08.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.672+0000 7f35ebfff640 1 -- 172.21.15.44:0/1843550353 learned_addr learned my addr 172.21.15.44:0/1843550353 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:08.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.673+0000 7f35f37fe640 1 -- 172.21.15.44:0/1843550353 >> v2:172.21.15.69:3300/0 conn(0x7f35f415e0f0 msgr2=0x7f35f40d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:08.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.673+0000 7f35f37fe640 1 --2- 172.21.15.44:0/1843550353 >> v2:172.21.15.69:3300/0 conn(0x7f35f415e0f0 0x7f35f40d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.673+0000 7f35f37fe640 1 -- 172.21.15.44:0/1843550353 >> v1:172.21.15.69:6789/0 conn(0x7f35f40d0630 legacy=0x7f35f406b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.673+0000 7f35f37fe640 1 -- 172.21.15.44:0/1843550353 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f406e880 con 0x7f35f406bbe0 2024-04-02T13:29:08.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.673+0000 7f35f27fc640 1 -- 172.21.15.44:0/1843550353 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2075311956 0 0) 0x7f35f406dfe0 con 0x7f35f40d0630 2024-04-02T13:29:08.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.673+0000 7f35f2ffd640 1 --2- 172.21.15.44:0/1843550353 >> v2:172.21.15.69:3300/0 conn(0x7f35f415e0f0 0x7f35f40d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:29:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.674+0000 7f35f37fe640 1 --2- 172.21.15.44:0/1843550353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35f406bbe0 0x7f35f40add60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f35ec009d90 tx=0x7f35ec02bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f9a8f3cd9fb595b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.674+0000 7f35f27fc640 1 -- 172.21.15.44:0/1843550353 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec039070 con 0x7f35f406bbe0 2024-04-02T13:29:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.674+0000 7f35f27fc640 1 -- 172.21.15.44:0/1843550353 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35ec02cce0 con 0x7f35f406bbe0 2024-04-02T13:29:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.674+0000 7f35f27fc640 1 -- 172.21.15.44:0/1843550353 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec034660 con 0x7f35f406bbe0 2024-04-02T13:29:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.674+0000 7f35f9504640 1 -- 172.21.15.44:0/1843550353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35f406bbe0 msgr2=0x7f35f40add60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.674+0000 7f35f9504640 1 --2- 172.21.15.44:0/1843550353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35f406bbe0 0x7f35f40add60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f35ec009d90 tx=0x7f35ec02bf30 comp rx=0 tx=0).stop 2024-04-02T13:29:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.674+0000 7f35f9504640 1 -- 172.21.15.44:0/1843550353 shutdown_connections 2024-04-02T13:29:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.674+0000 7f35f9504640 1 --2- 172.21.15.44:0/1843550353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35f406bbe0 0x7f35f40add60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.674+0000 7f35f9504640 1 --2- 172.21.15.44:0/1843550353 >> v2:172.21.15.69:3300/0 conn(0x7f35f415e0f0 0x7f35f40d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.674+0000 7f35f9504640 1 -- 172.21.15.44:0/1843550353 >> 172.21.15.44:0/1843550353 conn(0x7f35f40b2980 msgr2=0x7f35f40b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.674+0000 7f35f9504640 1 -- 172.21.15.44:0/1843550353 shutdown_connections 2024-04-02T13:29:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.674+0000 7f35f9504640 1 -- 172.21.15.44:0/1843550353 wait complete. 2024-04-02T13:29:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.675+0000 7f35f9504640 1 Processor -- start 2024-04-02T13:29:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.676+0000 7f35f9504640 1 -- start start 2024-04-02T13:29:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.676+0000 7f35f9504640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35f406bbe0 0x7f35f41f7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.676+0000 7f35f9504640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f40d0630 0x7f35f40c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.676+0000 7f35f9504640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35f415e0f0 0x7f35f40cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.676+0000 7f35f9504640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f35f415e4d0 con 0x7f35f406bbe0 2024-04-02T13:29:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.676+0000 7f35f9504640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f35f406eda0 con 0x7f35f40d0630 2024-04-02T13:29:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.676+0000 7f35f9504640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f35f415e650 con 0x7f35f415e0f0 2024-04-02T13:29:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.676+0000 7f35f2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35f406bbe0 0x7f35f41f7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.676+0000 7f35f2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35f406bbe0 0x7f35f41f7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51838/0 (socket says 172.21.15.44:51838) 2024-04-02T13:29:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.676+0000 7f35f2ffd640 1 -- 172.21.15.44:0/1895922552 learned_addr learned my addr 172.21.15.44:0/1895922552 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35ebfff640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f40d0630 0x7f35f40c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35f37fe640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35f415e0f0 0x7f35f40cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35f2ffd640 1 -- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35f415e0f0 msgr2=0x7f35f40cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35f2ffd640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35f415e0f0 0x7f35f40cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35f2ffd640 1 -- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f40d0630 msgr2=0x7f35f40c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35f2ffd640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f40d0630 0x7f35f40c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35f2ffd640 1 -- 172.21.15.44:0/1895922552 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f40b4d90 con 0x7f35f406bbe0 2024-04-02T13:29:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35f37fe640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35f415e0f0 0x7f35f40cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35ebfff640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f40d0630 0x7f35f40c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35f2ffd640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35f406bbe0 0x7f35f41f7e40 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f35e400e960 tx=0x7f35e400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35eb7fe640 1 -- 172.21.15.44:0/1895922552 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e4017070 con 0x7f35f406bbe0 2024-04-02T13:29:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35eb7fe640 1 -- 172.21.15.44:0/1895922552 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35e400ca00 con 0x7f35f406bbe0 2024-04-02T13:29:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35eb7fe640 1 -- 172.21.15.44:0/1895922552 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e400cdd0 con 0x7f35f406bbe0 2024-04-02T13:29:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35f9504640 1 -- 172.21.15.44:0/1895922552 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f40b5070 con 0x7f35f406bbe0 2024-04-02T13:29:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.677+0000 7f35f9504640 1 -- 172.21.15.44:0/1895922552 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f40b5630 con 0x7f35f406bbe0 2024-04-02T13:29:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.678+0000 7f35f9504640 1 -- 172.21.15.44:0/1895922552 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35f40684f0 con 0x7f35f406bbe0 2024-04-02T13:29:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.679+0000 7f35eb7fe640 1 -- 172.21.15.44:0/1895922552 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f35e401a070 con 0x7f35f406bbe0 2024-04-02T13:29:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.679+0000 7f35eb7fe640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35d406c8f0 0x7f35d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.679+0000 7f35eb7fe640 1 -- 172.21.15.44:0/1895922552 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5249+0+0 (secure 0 0 0) 0x7f35e408bb50 con 0x7f35f406bbe0 2024-04-02T13:29:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.679+0000 7f35ebfff640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35d406c8f0 0x7f35d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.680+0000 7f35ebfff640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35d406c8f0 0x7f35d406edb0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f35dc000f60 tx=0x7f35dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:08.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.684+0000 7f35eb7fe640 1 -- 172.21.15.44:0/1895922552 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f35e405a1d0 con 0x7f35f406bbe0 2024-04-02T13:29:08.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.869+0000 7f35f9504640 1 -- 172.21.15.44:0/1895922552 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f35f40727f0 con 0x7f35f406bbe0 2024-04-02T13:29:08.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.869+0000 7f35eb7fe640 1 -- 172.21.15.44:0/1895922552 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v102) ==== 84+0+65 (secure 0 0 0) 0x7f35e4059ad0 con 0x7f35f406bbe0 2024-04-02T13:29:08.872 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:08.872 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-04-02T13:29:08.872 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:29:08.872 INFO:teuthology.orchestra.run.smithi044.stdout: "poolnum": 1, 2024-04-02T13:29:08.872 INFO:teuthology.orchestra.run.smithi044.stdout: "poolname": ".mgr" 2024-04-02T13:29:08.872 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:08.872 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-04-02T13:29:08.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.871+0000 7f35f9504640 1 -- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35d406c8f0 msgr2=0x7f35d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.871+0000 7f35f9504640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35d406c8f0 0x7f35d406edb0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f35dc000f60 tx=0x7f35dc009040 comp rx=0 tx=0).stop 2024-04-02T13:29:08.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.871+0000 7f35f9504640 1 -- 172.21.15.44:0/1895922552 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35f406bbe0 msgr2=0x7f35f41f7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.871+0000 7f35f9504640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35f406bbe0 0x7f35f41f7e40 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f35e400e960 tx=0x7f35e400ee20 comp rx=0 tx=0).stop 2024-04-02T13:29:08.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.871+0000 7f35f9504640 1 -- 172.21.15.44:0/1895922552 shutdown_connections 2024-04-02T13:29:08.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.871+0000 7f35f9504640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35d406c8f0 0x7f35d406edb0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.871+0000 7f35f9504640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35f415e0f0 0x7f35f40cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.871+0000 7f35f9504640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f40d0630 0x7f35f40c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.871+0000 7f35f9504640 1 --2- 172.21.15.44:0/1895922552 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35f406bbe0 0x7f35f41f7e40 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.871+0000 7f35f9504640 1 -- 172.21.15.44:0/1895922552 >> 172.21.15.44:0/1895922552 conn(0x7f35f40b2980 msgr2=0x7f35f40b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:08.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.871+0000 7f35f9504640 1 -- 172.21.15.44:0/1895922552 shutdown_connections 2024-04-02T13:29:08.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.871+0000 7f35f9504640 1 -- 172.21.15.44:0/1895922552 wait complete. 2024-04-02T13:29:08.884 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T13:29:08.884 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T13:29:08.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.991+0000 7f11da7e9640 1 Processor -- start 2024-04-02T13:29:08.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.991+0000 7f11da7e9640 1 -- start start 2024-04-02T13:29:08.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.992+0000 7f11da7e9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f11d4166e40 0x7f11d4169230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.992+0000 7f11da7e9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d40b39a0 con 0x7f11d40cd810 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.992+0000 7f11da7e9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d406e530 con 0x7f11d40b1c30 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.992+0000 7f11da7e9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f11d40ce800 con 0x7f11d4166e40 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.992+0000 7f11d37fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f11d40cd810 0x7f11d40b3290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32814/0 (socket says 172.21.15.44:32814) 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.992+0000 7f11d37fe640 1 -- 172.21.15.44:0/565807186 learned_addr learned my addr 172.21.15.44:0/565807186 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.992+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 508752819 0 0) 0x7f11d40b39a0 con 0x7f11d40cd810 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.992+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11bc003520 con 0x7f11d40cd810 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.992+0000 7f11d8d86640 1 --2- 172.21.15.44:0/565807186 >> v2:172.21.15.69:3300/0 conn(0x7f11d4166e40 0x7f11d4169230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 640511374 0 0) 0x7f11bc003520 con 0x7f11d40cd810 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 >> v2:172.21.15.69:3300/0 conn(0x7f11d4166e40 msgr2=0x7f11d4169230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d2ffd640 1 --2- 172.21.15.44:0/565807186 >> v2:172.21.15.69:3300/0 conn(0x7f11d4166e40 0x7f11d4169230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 >> v1:172.21.15.69:6789/0 conn(0x7f11d40b1c30 legacy=0x7f11d40b2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d40cea30 con 0x7f11d40cd810 2024-04-02T13:29:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f11c4001670 con 0x7f11d40cd810 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 >> v1:172.21.15.44:6789/0 conn(0x7f11d40cd810 legacy=0x7f11d40b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d2ffd640 1 --2- 172.21.15.44:0/565807186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11bc0039b0 0x7f11bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d2ffd640 1 --2- 172.21.15.44:0/565807186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11bc0043b0 0x7f11bc004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d2ffd640 1 --2- 172.21.15.44:0/565807186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11bc005080 0x7f11bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc00bfb0 con 0x7f11bc0043b0 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc00c160 con 0x7f11bc005080 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f11bc00c360 con 0x7f11bc0039b0 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d37fe640 1 --2- 172.21.15.44:0/565807186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11bc0043b0 0x7f11bc004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d8d86640 1 --2- 172.21.15.44:0/565807186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11bc005080 0x7f11bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d3fff640 1 --2- 172.21.15.44:0/565807186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11bc0039b0 0x7f11bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d37fe640 1 -- 172.21.15.44:0/565807186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11bc0039b0 msgr2=0x7f11bc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d37fe640 1 --2- 172.21.15.44:0/565807186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11bc0039b0 0x7f11bc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d37fe640 1 -- 172.21.15.44:0/565807186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11bc005080 msgr2=0x7f11bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d37fe640 1 --2- 172.21.15.44:0/565807186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11bc005080 0x7f11bc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d8d86640 1 -- 172.21.15.44:0/565807186 reap_dead start 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d37fe640 1 -- 172.21.15.44:0/565807186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11bc003520 con 0x7f11bc0043b0 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.993+0000 7f11d3fff640 1 --2- 172.21.15.44:0/565807186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11bc0039b0 0x7f11bc003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.994+0000 7f11d37fe640 1 --2- 172.21.15.44:0/565807186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11bc0043b0 0x7f11bc004880 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f11c400c240 tx=0x7f11bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937324320ad06f3e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.994+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c4007d10 con 0x7f11bc0043b0 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.994+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c4030070 con 0x7f11bc0043b0 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.994+0000 7f11d2ffd640 1 -- 172.21.15.44:0/565807186 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c4005d70 con 0x7f11bc0043b0 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.995+0000 7f11da7e9640 1 -- 172.21.15.44:0/565807186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11bc0043b0 msgr2=0x7f11bc004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.995+0000 7f11da7e9640 1 --2- 172.21.15.44:0/565807186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11bc0043b0 0x7f11bc004880 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f11c400c240 tx=0x7f11bc005830 comp rx=0 tx=0).stop 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.995+0000 7f11da7e9640 1 -- 172.21.15.44:0/565807186 shutdown_connections 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.995+0000 7f11da7e9640 1 --2- 172.21.15.44:0/565807186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11bc0043b0 0x7f11bc004880 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.995+0000 7f11da7e9640 1 -- 172.21.15.44:0/565807186 >> 172.21.15.44:0/565807186 conn(0x7f11d406b4d0 msgr2=0x7f11d406b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.995+0000 7f11da7e9640 1 -- 172.21.15.44:0/565807186 shutdown_connections 2024-04-02T13:29:09.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.995+0000 7f11da7e9640 1 -- 172.21.15.44:0/565807186 wait complete. 2024-04-02T13:29:09.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.996+0000 7f11da7e9640 1 Processor -- start 2024-04-02T13:29:09.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.996+0000 7f11da7e9640 1 -- start start 2024-04-02T13:29:09.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11da7e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11bc0043b0 0x7f11d409aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:09.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11da7e9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11d414cf90 0x7f11d4149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:09.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11da7e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11d415f720 0x7f11d415fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:09.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11da7e9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f11d40cf2d0 con 0x7f11d414cf90 2024-04-02T13:29:09.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11da7e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f11d40ceff0 con 0x7f11d415f720 2024-04-02T13:29:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11da7e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f11d40cf5b0 con 0x7f11bc0043b0 2024-04-02T13:29:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11d37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11d414cf90 0x7f11d4149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11d37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11d414cf90 0x7f11d4149d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51848/0 (socket says 172.21.15.44:51848) 2024-04-02T13:29:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11d37fe640 1 -- 172.21.15.44:0/3889048071 learned_addr learned my addr 172.21.15.44:0/3889048071 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11d8d86640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11d415f720 0x7f11d415fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11d3fff640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11bc0043b0 0x7f11d409aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11d37fe640 1 -- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11bc0043b0 msgr2=0x7f11d409aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11d37fe640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11bc0043b0 0x7f11d409aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11d37fe640 1 -- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11d415f720 msgr2=0x7f11d415fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11d37fe640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11d415f720 0x7f11d415fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11d37fe640 1 -- 172.21.15.44:0/3889048071 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d409c310 con 0x7f11d414cf90 2024-04-02T13:29:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.997+0000 7f11d37fe640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11d414cf90 0x7f11d4149d40 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f11c400c240 tx=0x7f11d40ce420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.998+0000 7f11d0ff9640 1 -- 172.21.15.44:0/3889048071 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c4001c50 con 0x7f11d414cf90 2024-04-02T13:29:09.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.998+0000 7f11d0ff9640 1 -- 172.21.15.44:0/3889048071 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c4004bf0 con 0x7f11d414cf90 2024-04-02T13:29:09.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.998+0000 7f11d0ff9640 1 -- 172.21.15.44:0/3889048071 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c4007070 con 0x7f11d414cf90 2024-04-02T13:29:09.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.998+0000 7f11da7e9640 1 -- 172.21.15.44:0/3889048071 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d414d490 con 0x7f11d414cf90 2024-04-02T13:29:09.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.998+0000 7f11da7e9640 1 -- 172.21.15.44:0/3889048071 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d40c9ee0 con 0x7f11d414cf90 2024-04-02T13:29:09.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.999+0000 7f11da7e9640 1 -- 172.21.15.44:0/3889048071 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d40684f0 con 0x7f11d414cf90 2024-04-02T13:29:09.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.999+0000 7f11d0ff9640 1 -- 172.21.15.44:0/3889048071 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f11c4048020 con 0x7f11d414cf90 2024-04-02T13:29:09.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.999+0000 7f11d0ff9640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f11c006c8f0 0x7f11c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:09.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:08.999+0000 7f11d0ff9640 1 -- 172.21.15.44:0/3889048071 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5249+0+0 (secure 0 0 0) 0x7f11c40acdf0 con 0x7f11d414cf90 2024-04-02T13:29:09.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.000+0000 7f11d3fff640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f11c006c8f0 0x7f11c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:09.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.000+0000 7f11d3fff640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f11c006c8f0 0x7f11c006edb0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f11c8007fd0 tx=0x7f11c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:09.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.004+0000 7f11d0ff9640 1 -- 172.21.15.44:0/3889048071 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f11c407b3e0 con 0x7f11d414cf90 2024-04-02T13:29:09.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.188+0000 7f11da7e9640 1 -- 172.21.15.44:0/3889048071 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f11d40b1870 con 0x7f11d414cf90 2024-04-02T13:29:09.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.531+0000 7f11d0ff9640 1 -- 172.21.15.44:0/3889048071 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v103) ==== 138+0+0 (secure 0 0 0) 0x7f11c407ace0 con 0x7f11d414cf90 2024-04-02T13:29:09.534 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' created 2024-04-02T13:29:09.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.533+0000 7f11da7e9640 1 -- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f11c006c8f0 msgr2=0x7f11c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:09.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.533+0000 7f11da7e9640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f11c006c8f0 0x7f11c006edb0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f11c8007fd0 tx=0x7f11c8009040 comp rx=0 tx=0).stop 2024-04-02T13:29:09.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.533+0000 7f11da7e9640 1 -- 172.21.15.44:0/3889048071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11d414cf90 msgr2=0x7f11d4149d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:09.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.533+0000 7f11da7e9640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11d414cf90 0x7f11d4149d40 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f11c400c240 tx=0x7f11d40ce420 comp rx=0 tx=0).stop 2024-04-02T13:29:09.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.533+0000 7f11da7e9640 1 -- 172.21.15.44:0/3889048071 shutdown_connections 2024-04-02T13:29:09.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.533+0000 7f11da7e9640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f11c006c8f0 0x7f11c006edb0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.533+0000 7f11da7e9640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11d415f720 0x7f11d415fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.533+0000 7f11da7e9640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11d414cf90 0x7f11d4149d40 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.533+0000 7f11da7e9640 1 --2- 172.21.15.44:0/3889048071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11bc0043b0 0x7f11d409aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.533+0000 7f11da7e9640 1 -- 172.21.15.44:0/3889048071 >> 172.21.15.44:0/3889048071 conn(0x7f11d406b4d0 msgr2=0x7f11d40b4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:09.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.533+0000 7f11da7e9640 1 -- 172.21.15.44:0/3889048071 shutdown_connections 2024-04-02T13:29:09.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.533+0000 7f11da7e9640 1 -- 172.21.15.44:0/3889048071 wait complete. 2024-04-02T13:29:09.551 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T13:29:09.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.658+0000 7fd2c5d60640 1 Processor -- start 2024-04-02T13:29:09.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.658+0000 7fd2c5d60640 1 -- start start 2024-04-02T13:29:09.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.658+0000 7fd2c5d60640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd2c015e420 0x7fd2c00ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:09.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.658+0000 7fd2c5d60640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd2c00cfe10 0x7fd2c00cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:09.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.658+0000 7fd2c5d60640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2c006c220 con 0x7fd2c00721f0 2024-04-02T13:29:09.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.659+0000 7fd2c5d60640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd2c015f7e0 con 0x7fd2c00cfe10 2024-04-02T13:29:09.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.659+0000 7fd2c5d60640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd2c015ec60 con 0x7fd2c015e420 2024-04-02T13:29:09.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.659+0000 7fd2bffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd2c00cfe10 0x7fd2c00cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:09.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.659+0000 7fd2beffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd2c00721f0 0x7fd2c00cf700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32826/0 (socket says 172.21.15.44:32826) 2024-04-02T13:29:09.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.659+0000 7fd2beffd640 1 -- 172.21.15.44:0/129492566 learned_addr learned my addr 172.21.15.44:0/129492566 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:09.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.659+0000 7fd2bf7fe640 1 --2- 172.21.15.44:0/129492566 >> v2:172.21.15.69:3300/0 conn(0x7fd2c015e420 0x7fd2c00ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:09.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.659+0000 7fd2bffff640 1 -- 172.21.15.44:0/129492566 >> v2:172.21.15.69:3300/0 conn(0x7fd2c015e420 msgr2=0x7fd2c00ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:09.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.659+0000 7fd2bffff640 1 --2- 172.21.15.44:0/129492566 >> v2:172.21.15.69:3300/0 conn(0x7fd2c015e420 0x7fd2c00ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.659+0000 7fd2bffff640 1 -- 172.21.15.44:0/129492566 >> v1:172.21.15.44:6789/0 conn(0x7fd2c00721f0 legacy=0x7fd2c00cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:09.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.659+0000 7fd2bffff640 1 -- 172.21.15.44:0/129492566 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c015ef40 con 0x7fd2c00cfe10 2024-04-02T13:29:09.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.660+0000 7fd2bffff640 1 --2- 172.21.15.44:0/129492566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2c00cfe10 0x7fd2c00cdae0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fd2c006dc40 tx=0x7fd2b802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bdc34ebfcbdd783 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:09.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.660+0000 7fd2be7fc640 1 -- 172.21.15.44:0/129492566 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8039070 con 0x7fd2c00cfe10 2024-04-02T13:29:09.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.660+0000 7fd2be7fc640 1 -- 172.21.15.44:0/129492566 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2b802cb60 con 0x7fd2c00cfe10 2024-04-02T13:29:09.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.660+0000 7fd2be7fc640 1 -- 172.21.15.44:0/129492566 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8034440 con 0x7fd2c00cfe10 2024-04-02T13:29:09.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.660+0000 7fd2c5d60640 1 -- 172.21.15.44:0/129492566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2c00cfe10 msgr2=0x7fd2c00cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:09.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.660+0000 7fd2c5d60640 1 --2- 172.21.15.44:0/129492566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2c00cfe10 0x7fd2c00cdae0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fd2c006dc40 tx=0x7fd2b802bdb0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.660+0000 7fd2c5d60640 1 -- 172.21.15.44:0/129492566 shutdown_connections 2024-04-02T13:29:09.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.660+0000 7fd2c5d60640 1 --2- 172.21.15.44:0/129492566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2c00cfe10 0x7fd2c00cdae0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.660+0000 7fd2c5d60640 1 --2- 172.21.15.44:0/129492566 >> v2:172.21.15.69:3300/0 conn(0x7fd2c015e420 0x7fd2c00ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.660+0000 7fd2c5d60640 1 -- 172.21.15.44:0/129492566 >> 172.21.15.44:0/129492566 conn(0x7fd2c00b2a10 msgr2=0x7fd2c00b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:09.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.660+0000 7fd2c5d60640 1 -- 172.21.15.44:0/129492566 shutdown_connections 2024-04-02T13:29:09.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.661+0000 7fd2c5d60640 1 -- 172.21.15.44:0/129492566 wait complete. 2024-04-02T13:29:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.662+0000 7fd2c5d60640 1 Processor -- start 2024-04-02T13:29:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.662+0000 7fd2c5d60640 1 -- start start 2024-04-02T13:29:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.662+0000 7fd2c5d60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2c00721f0 0x7fd2c01f8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.662+0000 7fd2c5d60640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2c00cfe10 0x7fd2c014e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.662+0000 7fd2c5d60640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2c015e420 0x7fd2c00af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.662+0000 7fd2c5d60640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c00cf1e0 con 0x7fd2c015e420 2024-04-02T13:29:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.662+0000 7fd2c5d60640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c015f500 con 0x7fd2c00cfe10 2024-04-02T13:29:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.662+0000 7fd2c5d60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c00cf360 con 0x7fd2c00721f0 2024-04-02T13:29:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2bffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2c015e420 0x7fd2c00af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2bffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2c015e420 0x7fd2c00af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51872/0 (socket says 172.21.15.44:51872) 2024-04-02T13:29:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2bffff640 1 -- 172.21.15.44:0/3313741145 learned_addr learned my addr 172.21.15.44:0/3313741145 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2bf7fe640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2c00721f0 0x7fd2c01f8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2beffd640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2c00cfe10 0x7fd2c014e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:09.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2bffff640 1 -- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2c00721f0 msgr2=0x7fd2c01f8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:09.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2bffff640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2c00721f0 0x7fd2c01f8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2bffff640 1 -- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2c00cfe10 msgr2=0x7fd2c014e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:09.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2bffff640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2c00cfe10 0x7fd2c014e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:09.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2bffff640 1 -- 172.21.15.44:0/3313741145 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c00afba0 con 0x7fd2c015e420 2024-04-02T13:29:09.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2bf7fe640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2c00721f0 0x7fd2c01f8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:09.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2bffff640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2c015e420 0x7fd2c00af660 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009e80 tx=0x7fd2b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:09.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2a7fff640 1 -- 172.21.15.44:0/3313741145 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8039050 con 0x7fd2c015e420 2024-04-02T13:29:09.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2a7fff640 1 -- 172.21.15.44:0/3313741145 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2b8045070 con 0x7fd2c015e420 2024-04-02T13:29:09.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2c5d60640 1 -- 172.21.15.44:0/3313741145 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c00afe70 con 0x7fd2c015e420 2024-04-02T13:29:09.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2a7fff640 1 -- 172.21.15.44:0/3313741145 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b803e830 con 0x7fd2c015e420 2024-04-02T13:29:09.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.663+0000 7fd2c5d60640 1 -- 172.21.15.44:0/3313741145 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c00b0430 con 0x7fd2c015e420 2024-04-02T13:29:09.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.664+0000 7fd2c5d60640 1 -- 172.21.15.44:0/3313741145 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c00684f0 con 0x7fd2c015e420 2024-04-02T13:29:09.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.665+0000 7fd2a7fff640 1 -- 172.21.15.44:0/3313741145 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd2b802cd10 con 0x7fd2c015e420 2024-04-02T13:29:09.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.665+0000 7fd2a7fff640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd2ac06c8f0 0x7fd2ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:09.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.665+0000 7fd2a7fff640 1 -- 172.21.15.44:0/3313741145 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5619+0+0 (secure 0 0 0) 0x7fd2b80ae6d0 con 0x7fd2c015e420 2024-04-02T13:29:09.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.665+0000 7fd2bf7fe640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd2ac06c8f0 0x7fd2ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:09.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.666+0000 7fd2bf7fe640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd2ac06c8f0 0x7fd2ac06edb0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0006fd0 tx=0x7fd2b0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:09.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.670+0000 7fd2a7fff640 1 -- 172.21.15.44:0/3313741145 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd2b8032e20 con 0x7fd2c015e420 2024-04-02T13:29:09.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:09.856+0000 7fd2c5d60640 1 -- 172.21.15.44:0/3313741145 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd2c00b0670 con 0x7fd2c015e420 2024-04-02T13:29:10.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.533+0000 7fd2a7fff640 1 -- 172.21.15.44:0/3313741145 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v104) ==== 170+0+0 (secure 0 0 0) 0x7fd2b807cab0 con 0x7fd2c015e420 2024-04-02T13:29:10.536 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' created 2024-04-02T13:29:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.535+0000 7fd2c5d60640 1 -- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd2ac06c8f0 msgr2=0x7fd2ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.535+0000 7fd2c5d60640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd2ac06c8f0 0x7fd2ac06edb0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0006fd0 tx=0x7fd2b0008040 comp rx=0 tx=0).stop 2024-04-02T13:29:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.535+0000 7fd2c5d60640 1 -- 172.21.15.44:0/3313741145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2c015e420 msgr2=0x7fd2c00af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.535+0000 7fd2c5d60640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2c015e420 0x7fd2c00af660 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009e80 tx=0x7fd2b8004a40 comp rx=0 tx=0).stop 2024-04-02T13:29:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.535+0000 7fd2c5d60640 1 -- 172.21.15.44:0/3313741145 shutdown_connections 2024-04-02T13:29:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.535+0000 7fd2c5d60640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd2ac06c8f0 0x7fd2ac06edb0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.535+0000 7fd2c5d60640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2c015e420 0x7fd2c00af660 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.535+0000 7fd2c5d60640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2c00cfe10 0x7fd2c014e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.535+0000 7fd2c5d60640 1 --2- 172.21.15.44:0/3313741145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2c00721f0 0x7fd2c01f8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.535+0000 7fd2c5d60640 1 -- 172.21.15.44:0/3313741145 >> 172.21.15.44:0/3313741145 conn(0x7fd2c00b2a10 msgr2=0x7fd2c00b14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:10.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.536+0000 7fd2c5d60640 1 -- 172.21.15.44:0/3313741145 shutdown_connections 2024-04-02T13:29:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.536+0000 7fd2c5d60640 1 -- 172.21.15.44:0/3313741145 wait complete. 2024-04-02T13:29:10.559 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T13:29:10.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.667+0000 7f2113ceb640 1 Processor -- start 2024-04-02T13:29:10.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.667+0000 7f2113ceb640 1 -- start start 2024-04-02T13:29:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.668+0000 7f2113ceb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f210c15e090 0x7f210c06df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.668+0000 7f2113ceb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f210c06e590 con 0x7f210c06bbe0 2024-04-02T13:29:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.668+0000 7f2113ceb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f210c06f860 con 0x7f210c072890 2024-04-02T13:29:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.668+0000 7f2113ceb640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f210c0d0cf0 con 0x7f210c15e090 2024-04-02T13:29:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.668+0000 7f2111a87640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f210c15e090 0x7f210c06df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.668+0000 7f2112288640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f210c06bbe0 0x7f210c0d1180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32840/0 (socket says 172.21.15.44:32840) 2024-04-02T13:29:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.668+0000 7f2112288640 1 -- 172.21.15.44:0/742511835 learned_addr learned my addr 172.21.15.44:0/742511835 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.668+0000 7f2111a87640 1 -- 172.21.15.44:0/742511835 >> v1:172.21.15.69:6789/0 conn(0x7f210c072890 legacy=0x7f210c06b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.668+0000 7f2111a87640 1 -- 172.21.15.44:0/742511835 >> v1:172.21.15.44:6789/0 conn(0x7f210c06bbe0 legacy=0x7f210c0d1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.668+0000 7f2111a87640 1 -- 172.21.15.44:0/742511835 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f210c06fb40 con 0x7f210c15e090 2024-04-02T13:29:10.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.669+0000 7f2111a87640 1 --2- 172.21.15.44:0/742511835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210c15e090 0x7f210c06df20 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f21040097a0 tx=0x7f210402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3f8eeee487649b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:10.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.669+0000 7f2110a85640 1 -- 172.21.15.44:0/742511835 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2104035020 con 0x7f210c15e090 2024-04-02T13:29:10.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.669+0000 7f2110a85640 1 -- 172.21.15.44:0/742511835 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2104031830 con 0x7f210c15e090 2024-04-02T13:29:10.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.669+0000 7f2110a85640 1 -- 172.21.15.44:0/742511835 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2104031c00 con 0x7f210c15e090 2024-04-02T13:29:10.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.670+0000 7f2113ceb640 1 -- 172.21.15.44:0/742511835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210c15e090 msgr2=0x7f210c06df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:10.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.670+0000 7f2113ceb640 1 --2- 172.21.15.44:0/742511835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210c15e090 0x7f210c06df20 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f21040097a0 tx=0x7f210402b8d0 comp rx=0 tx=0).stop 2024-04-02T13:29:10.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.670+0000 7f2113ceb640 1 -- 172.21.15.44:0/742511835 shutdown_connections 2024-04-02T13:29:10.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.670+0000 7f2113ceb640 1 --2- 172.21.15.44:0/742511835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210c15e090 0x7f210c06df20 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:10.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.670+0000 7f2113ceb640 1 -- 172.21.15.44:0/742511835 >> 172.21.15.44:0/742511835 conn(0x7f210c06c5e0 msgr2=0x7f210c06c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:10.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.670+0000 7f2113ceb640 1 -- 172.21.15.44:0/742511835 shutdown_connections 2024-04-02T13:29:10.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.670+0000 7f2113ceb640 1 -- 172.21.15.44:0/742511835 wait complete. 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.671+0000 7f2113ceb640 1 Processor -- start 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.671+0000 7f2113ceb640 1 -- start start 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2113ceb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f210c06bbe0 0x7f210c1ea570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2113ceb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210c072890 0x7f210c1e85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2113ceb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f210c15e090 0x7f210c159190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2113ceb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f210c15e5f0 con 0x7f210c072890 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2113ceb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f210c15e470 con 0x7f210c06bbe0 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2113ceb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f210c0b3dd0 con 0x7f210c15e090 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2111286640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210c072890 0x7f210c1e85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2111286640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210c072890 0x7f210c1e85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51886/0 (socket says 172.21.15.44:51886) 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2111286640 1 -- 172.21.15.44:0/3260637059 learned_addr learned my addr 172.21.15.44:0/3260637059 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2112288640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f210c15e090 0x7f210c159190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2111a87640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f210c06bbe0 0x7f210c1ea570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2111286640 1 -- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f210c15e090 msgr2=0x7f210c159190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2111286640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f210c15e090 0x7f210c159190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2111286640 1 -- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f210c06bbe0 msgr2=0x7f210c1ea570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2111286640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f210c06bbe0 0x7f210c1ea570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2111286640 1 -- 172.21.15.44:0/3260637059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f210c1e8b00 con 0x7f210c072890 2024-04-02T13:29:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2112288640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f210c15e090 0x7f210c159190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2111a87640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f210c06bbe0 0x7f210c1ea570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2111286640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210c072890 0x7f210c1e85c0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f210000c3c0 tx=0x7f210000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f210a7fc640 1 -- 172.21.15.44:0/3260637059 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21000052d0 con 0x7f210c072890 2024-04-02T13:29:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f210a7fc640 1 -- 172.21.15.44:0/3260637059 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2100005490 con 0x7f210c072890 2024-04-02T13:29:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.672+0000 7f2113ceb640 1 -- 172.21.15.44:0/3260637059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f210c1e8da0 con 0x7f210c072890 2024-04-02T13:29:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.673+0000 7f210a7fc640 1 -- 172.21.15.44:0/3260637059 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100007260 con 0x7f210c072890 2024-04-02T13:29:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.673+0000 7f2113ceb640 1 -- 172.21.15.44:0/3260637059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f210c1e9360 con 0x7f210c072890 2024-04-02T13:29:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.674+0000 7f2113ceb640 1 -- 172.21.15.44:0/3260637059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f210c1e96a0 con 0x7f210c072890 2024-04-02T13:29:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.674+0000 7f210a7fc640 1 -- 172.21.15.44:0/3260637059 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2100007400 con 0x7f210c072890 2024-04-02T13:29:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.674+0000 7f210a7fc640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20f806c8f0 0x7f20f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.674+0000 7f210a7fc640 1 -- 172.21.15.44:0/3260637059 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 6001+0+0 (secure 0 0 0) 0x7f210008b120 con 0x7f210c072890 2024-04-02T13:29:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.675+0000 7f2111a87640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20f806c8f0 0x7f20f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.675+0000 7f2111a87640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20f806c8f0 0x7f20f806edb0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2104036000 tx=0x7f2104008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:10.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.679+0000 7f210a7fc640 1 -- 172.21.15.44:0/3260637059 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f210c1e96a0 con 0x7f210c072890 2024-04-02T13:29:10.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:10.906+0000 7f2113ceb640 1 -- 172.21.15.44:0/3260637059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f210c068c00 con 0x7f210c072890 2024-04-02T13:29:11.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.564+0000 7f210a7fc640 1 -- 172.21.15.44:0/3260637059 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v79) ==== 174+0+0 (secure 0 0 0) 0x7f2100059420 con 0x7f210c072890 2024-04-02T13:29:11.569 INFO:teuthology.orchestra.run.smithi044.stderr:new fs with metadata pool 15 and data pool 16 2024-04-02T13:29:11.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.575+0000 7f2113ceb640 1 -- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20f806c8f0 msgr2=0x7f20f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:11.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.575+0000 7f2113ceb640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20f806c8f0 0x7f20f806edb0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2104036000 tx=0x7f2104008030 comp rx=0 tx=0).stop 2024-04-02T13:29:11.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.576+0000 7f2113ceb640 1 -- 172.21.15.44:0/3260637059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210c072890 msgr2=0x7f210c1e85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:11.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.576+0000 7f2113ceb640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210c072890 0x7f210c1e85c0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f210000c3c0 tx=0x7f210000c880 comp rx=0 tx=0).stop 2024-04-02T13:29:11.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.577+0000 7f2113ceb640 1 -- 172.21.15.44:0/3260637059 shutdown_connections 2024-04-02T13:29:11.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.577+0000 7f2113ceb640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20f806c8f0 0x7f20f806edb0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:11.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.577+0000 7f2113ceb640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f210c15e090 0x7f210c159190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:11.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.578+0000 7f2113ceb640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f210c072890 0x7f210c1e85c0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:11.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.578+0000 7f2113ceb640 1 --2- 172.21.15.44:0/3260637059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f210c06bbe0 0x7f210c1ea570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:11.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.578+0000 7f2113ceb640 1 -- 172.21.15.44:0/3260637059 >> 172.21.15.44:0/3260637059 conn(0x7f210c06c5e0 msgr2=0x7f210c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:11.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.579+0000 7f2113ceb640 1 -- 172.21.15.44:0/3260637059 shutdown_connections 2024-04-02T13:29:11.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.579+0000 7f2113ceb640 1 -- 172.21.15.44:0/3260637059 wait complete. 2024-04-02T13:29:11.596 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:29:11.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.696+0000 7f8096132640 1 Processor -- start 2024-04-02T13:29:11.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.696+0000 7f8096132640 1 -- start start 2024-04-02T13:29:11.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.697+0000 7f8096132640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f809015e550 0x7f809006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:11.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.697+0000 7f8096132640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f809015f040 0x7f80900d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:11.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.697+0000 7f8096132640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f809015fb50 con 0x7f809006d530 2024-04-02T13:29:11.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.697+0000 7f8096132640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f809006d970 con 0x7f809015e550 2024-04-02T13:29:11.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.697+0000 7f8096132640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f809006ca90 con 0x7f809015f040 2024-04-02T13:29:11.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.697+0000 7f808f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f809015e550 0x7f809006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:11.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.697+0000 7f808effd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f809006d530 0x7f809015e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32854/0 (socket says 172.21.15.44:32854) 2024-04-02T13:29:11.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.697+0000 7f808effd640 1 -- 172.21.15.44:0/208784283 learned_addr learned my addr 172.21.15.44:0/208784283 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:11.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.697+0000 7f808ffff640 1 --2- 172.21.15.44:0/208784283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f809015f040 0x7f80900d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:11.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.697+0000 7f808f7fe640 1 -- 172.21.15.44:0/208784283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f809015f040 msgr2=0x7f80900d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:11.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.697+0000 7f808f7fe640 1 --2- 172.21.15.44:0/208784283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f809015f040 0x7f80900d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:11.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.697+0000 7f808f7fe640 1 -- 172.21.15.44:0/208784283 >> v1:172.21.15.44:6789/0 conn(0x7f809006d530 legacy=0x7f809015e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:11.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.697+0000 7f808f7fe640 1 -- 172.21.15.44:0/208784283 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809015f590 con 0x7f809015e550 2024-04-02T13:29:11.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.698+0000 7f808f7fe640 1 --2- 172.21.15.44:0/208784283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f809015e550 0x7f809006cff0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f80800097a0 tx=0x7f808002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e5c8b84c225d4a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:11.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.698+0000 7f808e7fc640 1 -- 172.21.15.44:0/208784283 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080034070 con 0x7f809015e550 2024-04-02T13:29:11.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.698+0000 7f808e7fc640 1 -- 172.21.15.44:0/208784283 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8080030610 con 0x7f809015e550 2024-04-02T13:29:11.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.698+0000 7f808e7fc640 1 -- 172.21.15.44:0/208784283 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80800309e0 con 0x7f809015e550 2024-04-02T13:29:11.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.699+0000 7f8096132640 1 -- 172.21.15.44:0/208784283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f809015e550 msgr2=0x7f809006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:11.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.699+0000 7f8096132640 1 --2- 172.21.15.44:0/208784283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f809015e550 0x7f809006cff0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f80800097a0 tx=0x7f808002b960 comp rx=0 tx=0).stop 2024-04-02T13:29:11.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.699+0000 7f8096132640 1 -- 172.21.15.44:0/208784283 shutdown_connections 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.699+0000 7f8096132640 1 --2- 172.21.15.44:0/208784283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f809015f040 0x7f80900d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.699+0000 7f8096132640 1 --2- 172.21.15.44:0/208784283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f809015e550 0x7f809006cff0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.699+0000 7f8096132640 1 -- 172.21.15.44:0/208784283 >> 172.21.15.44:0/208784283 conn(0x7f809009aa20 msgr2=0x7f809009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.699+0000 7f8096132640 1 -- 172.21.15.44:0/208784283 shutdown_connections 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.699+0000 7f8096132640 1 -- 172.21.15.44:0/208784283 wait complete. 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.700+0000 7f8096132640 1 Processor -- start 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.700+0000 7f8096132640 1 -- start start 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f8096132640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f809006d530 0x7f80900b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f8096132640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f809015e550 0x7f8090159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f8096132640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f809015f040 0x7f809015c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f8096132640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f80900d0590 con 0x7f809015f040 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f8096132640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f80900d02b0 con 0x7f809015e550 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f8096132640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f80900cc4b0 con 0x7f809006d530 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f808ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f809015f040 0x7f809015c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:11.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f808ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f809015f040 0x7f809015c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51898/0 (socket says 172.21.15.44:51898) 2024-04-02T13:29:11.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f808ffff640 1 -- 172.21.15.44:0/732975159 learned_addr learned my addr 172.21.15.44:0/732975159 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:11.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f808f7fe640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f809006d530 0x7f80900b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:11.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f808effd640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f809015e550 0x7f8090159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:11.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f808ffff640 1 -- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f809006d530 msgr2=0x7f80900b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:11.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f808ffff640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f809006d530 0x7f80900b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:11.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f808ffff640 1 -- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f809015e550 msgr2=0x7f8090159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:11.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f808ffff640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f809015e550 0x7f8090159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:11.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f808ffff640 1 -- 172.21.15.44:0/732975159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80900b2930 con 0x7f809015f040 2024-04-02T13:29:11.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f808effd640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f809015e550 0x7f8090159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:11.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f808ffff640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f809015f040 0x7f809015c4e0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f808800efc0 tx=0x7f808800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:11.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f8077fff640 1 -- 172.21.15.44:0/732975159 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088018020 con 0x7f809015f040 2024-04-02T13:29:11.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.701+0000 7f8077fff640 1 -- 172.21.15.44:0/732975159 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8088014480 con 0x7f809015f040 2024-04-02T13:29:11.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.702+0000 7f8077fff640 1 -- 172.21.15.44:0/732975159 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088014870 con 0x7f809015f040 2024-04-02T13:29:11.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.702+0000 7f8096132640 1 -- 172.21.15.44:0/732975159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80900b2090 con 0x7f809015f040 2024-04-02T13:29:11.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.702+0000 7f8096132640 1 -- 172.21.15.44:0/732975159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80900b2650 con 0x7f809015f040 2024-04-02T13:29:11.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.702+0000 7f8096132640 1 -- 172.21.15.44:0/732975159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80900d20a0 con 0x7f809015f040 2024-04-02T13:29:11.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.703+0000 7f8077fff640 1 -- 172.21.15.44:0/732975159 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8088005d10 con 0x7f809015f040 2024-04-02T13:29:11.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.703+0000 7f8077fff640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f807c06c8f0 0x7f807c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:11.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.704+0000 7f8077fff640 1 -- 172.21.15.44:0/732975159 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6101+0+0 (secure 0 0 0) 0x7f808808c9f0 con 0x7f809015f040 2024-04-02T13:29:11.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.704+0000 7f808f7fe640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f807c06c8f0 0x7f807c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:11.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.704+0000 7f808f7fe640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f807c06c8f0 0x7f807c06edb0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8080036000 tx=0x7f808003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:11.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.708+0000 7f8077fff640 1 -- 172.21.15.44:0/732975159 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f80900d20a0 con 0x7f809015f040 2024-04-02T13:29:11.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.892+0000 7f8096132640 1 -- 172.21.15.44:0/732975159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80900684f0 con 0x7f809015f040 2024-04-02T13:29:11.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.893+0000 7f8077fff640 1 -- 172.21.15.44:0/732975159 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+16849 (secure 0 0 0) 0x7f8088022030 con 0x7f809015f040 2024-04-02T13:29:11.896 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:11.896 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":105,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:29:10.907607+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:29:09.192430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-02T13:29:09.860748+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:29:11.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.895+0000 7f8096132640 1 -- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f807c06c8f0 msgr2=0x7f807c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:11.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.895+0000 7f8096132640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f807c06c8f0 0x7f807c06edb0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8080036000 tx=0x7f808003b040 comp rx=0 tx=0).stop 2024-04-02T13:29:11.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.895+0000 7f8096132640 1 -- 172.21.15.44:0/732975159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f809015f040 msgr2=0x7f809015c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:11.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.895+0000 7f8096132640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f809015f040 0x7f809015c4e0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f808800efc0 tx=0x7f808800c490 comp rx=0 tx=0).stop 2024-04-02T13:29:11.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.895+0000 7f8096132640 1 -- 172.21.15.44:0/732975159 shutdown_connections 2024-04-02T13:29:11.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.895+0000 7f8096132640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f807c06c8f0 0x7f807c06edb0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:11.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.895+0000 7f8096132640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f809015f040 0x7f809015c4e0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:11.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.895+0000 7f8096132640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f809015e550 0x7f8090159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:11.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.895+0000 7f8096132640 1 --2- 172.21.15.44:0/732975159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f809006d530 0x7f80900b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:11.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.895+0000 7f8096132640 1 -- 172.21.15.44:0/732975159 >> 172.21.15.44:0/732975159 conn(0x7f809009aa20 msgr2=0x7f80900ccd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:11.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.895+0000 7f8096132640 1 -- 172.21.15.44:0/732975159 shutdown_connections 2024-04-02T13:29:11.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:11.895+0000 7f8096132640 1 -- 172.21.15.44:0/732975159 wait complete. 2024-04-02T13:29:11.909 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:29:12.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.008+0000 7f24458eb640 1 Processor -- start 2024-04-02T13:29:12.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.008+0000 7f24458eb640 1 -- start start 2024-04-02T13:29:12.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.009+0000 7f24458eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f244015e430 0x7f24400ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.009+0000 7f24458eb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f24400b1ef0 0x7f24400b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.009+0000 7f24458eb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f244015e810 con 0x7f244006c7f0 2024-04-02T13:29:12.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.009+0000 7f24458eb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f24400d1410 con 0x7f24400b1ef0 2024-04-02T13:29:12.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.009+0000 7f24458eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f244006cff0 con 0x7f244015e430 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.009+0000 7f2437fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f244006c7f0 0x7f24400ced00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32868/0 (socket says 172.21.15.44:32868) 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.009+0000 7f2437fff640 1 -- 172.21.15.44:0/2176321490 learned_addr learned my addr 172.21.15.44:0/2176321490 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243effd640 1 --2- 172.21.15.44:0/2176321490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f244015e430 0x7f24400ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2657582014 0 0) 0x7f244015e810 con 0x7f244006c7f0 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243f7fe640 1 --2- 172.21.15.44:0/2176321490 >> v2:172.21.15.69:3300/0 conn(0x7f24400b1ef0 0x7f24400b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f242c003520 con 0x7f244006c7f0 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4225017842 0 0) 0x7f242c003520 con 0x7f244006c7f0 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f244015e430 msgr2=0x7f24400ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 --2- 172.21.15.44:0/2176321490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f244015e430 0x7f24400ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 >> v2:172.21.15.69:3300/0 conn(0x7f24400b1ef0 msgr2=0x7f24400b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 --2- 172.21.15.44:0/2176321490 >> v2:172.21.15.69:3300/0 conn(0x7f24400b1ef0 0x7f24400b2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244006d240 con 0x7f244006c7f0 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f2428001670 con 0x7f244006c7f0 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 >> v1:172.21.15.44:6789/0 conn(0x7f244006c7f0 legacy=0x7f24400ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:12.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 --2- 172.21.15.44:0/2176321490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f242c0039b0 0x7f242c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 --2- 172.21.15.44:0/2176321490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f244015e430 0x7f242c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 --2- 172.21.15.44:0/2176321490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f242c004d40 0x7f242c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f244006cff0 con 0x7f242c004d40 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f242c00b9c0 con 0x7f242c0039b0 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f242c00bb40 con 0x7f244015e430 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243f7fe640 1 --2- 172.21.15.44:0/2176321490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f242c004d40 0x7f242c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243effd640 1 --2- 172.21.15.44:0/2176321490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f242c0039b0 0x7f242c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f2437fff640 1 --2- 172.21.15.44:0/2176321490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f244015e430 0x7f242c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243f7fe640 1 -- 172.21.15.44:0/2176321490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f244015e430 msgr2=0x7f242c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243f7fe640 1 --2- 172.21.15.44:0/2176321490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f244015e430 0x7f242c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243f7fe640 1 -- 172.21.15.44:0/2176321490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f242c0039b0 msgr2=0x7f242c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243f7fe640 1 --2- 172.21.15.44:0/2176321490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f242c0039b0 0x7f242c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.010+0000 7f243f7fe640 1 -- 172.21.15.44:0/2176321490 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242c003520 con 0x7f242c004d40 2024-04-02T13:29:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.011+0000 7f243f7fe640 1 --2- 172.21.15.44:0/2176321490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f242c004d40 0x7f242c0051c0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f2438009fc0 tx=0x7f242c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b1287b74ed08c27 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:12.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.011+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438039070 con 0x7f242c004d40 2024-04-02T13:29:12.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.011+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f243802cc60 con 0x7f242c004d40 2024-04-02T13:29:12.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.011+0000 7f243e7fc640 1 -- 172.21.15.44:0/2176321490 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438009c10 con 0x7f242c004d40 2024-04-02T13:29:12.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.012+0000 7f24458eb640 1 -- 172.21.15.44:0/2176321490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f242c004d40 msgr2=0x7f242c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:12.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.012+0000 7f24458eb640 1 --2- 172.21.15.44:0/2176321490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f242c004d40 0x7f242c0051c0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f2438009fc0 tx=0x7f242c005830 comp rx=0 tx=0).stop 2024-04-02T13:29:12.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.012+0000 7f243f7fe640 1 -- 172.21.15.44:0/2176321490 reap_dead start 2024-04-02T13:29:12.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.012+0000 7f24458eb640 1 -- 172.21.15.44:0/2176321490 shutdown_connections 2024-04-02T13:29:12.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.012+0000 7f24458eb640 1 -- 172.21.15.44:0/2176321490 >> 172.21.15.44:0/2176321490 conn(0x7f2440073850 msgr2=0x7f244009aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:12.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.012+0000 7f24458eb640 1 -- 172.21.15.44:0/2176321490 shutdown_connections 2024-04-02T13:29:12.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.012+0000 7f24458eb640 1 -- 172.21.15.44:0/2176321490 wait complete. 2024-04-02T13:29:12.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.013+0000 7f24458eb640 1 Processor -- start 2024-04-02T13:29:12.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.013+0000 7f24458eb640 1 -- start start 2024-04-02T13:29:12.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f24458eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f242c004d40 0x7f24401f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f24458eb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24401538b0 0x7f2440155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f24458eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24401581a0 0x7f244015a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f24458eb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f24400d0ef0 con 0x7f24401538b0 2024-04-02T13:29:12.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f24458eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f24400d0c10 con 0x7f24401581a0 2024-04-02T13:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f24458eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f244006d800 con 0x7f242c004d40 2024-04-02T13:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f2437fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24401538b0 0x7f2440155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f2437fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24401538b0 0x7f2440155380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51912/0 (socket says 172.21.15.44:51912) 2024-04-02T13:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f2437fff640 1 -- 172.21.15.44:0/2335595169 learned_addr learned my addr 172.21.15.44:0/2335595169 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f243effd640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f242c004d40 0x7f24401f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f243f7fe640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24401581a0 0x7f244015a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f2437fff640 1 -- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f242c004d40 msgr2=0x7f24401f6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f2437fff640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f242c004d40 0x7f24401f6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f2437fff640 1 -- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24401581a0 msgr2=0x7f244015a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f2437fff640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24401581a0 0x7f244015a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f2437fff640 1 -- 172.21.15.44:0/2335595169 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24401f77c0 con 0x7f24401538b0 2024-04-02T13:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.014+0000 7f2437fff640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24401538b0 0x7f2440155380 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f2428007370 tx=0x7f2428007770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.015+0000 7f243cff9640 1 -- 172.21.15.44:0/2335595169 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242801a070 con 0x7f24401538b0 2024-04-02T13:29:12.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.015+0000 7f243cff9640 1 -- 172.21.15.44:0/2335595169 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f242800fa20 con 0x7f24401538b0 2024-04-02T13:29:12.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.015+0000 7f243cff9640 1 -- 172.21.15.44:0/2335595169 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242800fdf0 con 0x7f24401538b0 2024-04-02T13:29:12.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.015+0000 7f24458eb640 1 -- 172.21.15.44:0/2335595169 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24401f7aa0 con 0x7f24401538b0 2024-04-02T13:29:12.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.015+0000 7f24458eb640 1 -- 172.21.15.44:0/2335595169 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24401f74b0 con 0x7f24401538b0 2024-04-02T13:29:12.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.016+0000 7f24458eb640 1 -- 172.21.15.44:0/2335595169 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2440068c00 con 0x7f24401538b0 2024-04-02T13:29:12.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.016+0000 7f243cff9640 1 -- 172.21.15.44:0/2335595169 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f242801d070 con 0x7f24401538b0 2024-04-02T13:29:12.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.016+0000 7f243cff9640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f242006c8f0 0x7f242006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.017+0000 7f243cff9640 1 -- 172.21.15.44:0/2335595169 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6101+0+0 (secure 0 0 0) 0x7f242808b610 con 0x7f24401538b0 2024-04-02T13:29:12.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.017+0000 7f243effd640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f242006c8f0 0x7f242006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.017+0000 7f243effd640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f242006c8f0 0x7f242006edb0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2430009b40 tx=0x7f2430009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:12.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.021+0000 7f243cff9640 1 -- 172.21.15.44:0/2335595169 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2428059260 con 0x7f24401538b0 2024-04-02T13:29:12.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.207+0000 7f24458eb640 1 -- 172.21.15.44:0/2335595169 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f244006f530 con 0x7f24401538b0 2024-04-02T13:29:12.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.208+0000 7f243cff9640 1 -- 172.21.15.44:0/2335595169 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+16849 (secure 0 0 0) 0x7f242805d560 con 0x7f24401538b0 2024-04-02T13:29:12.211 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:12.211 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":105,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:29:10.907607+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:29:09.192430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-02T13:29:09.860748+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:29:12.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.210+0000 7f24458eb640 1 -- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f242006c8f0 msgr2=0x7f242006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:12.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.210+0000 7f24458eb640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f242006c8f0 0x7f242006edb0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2430009b40 tx=0x7f2430009190 comp rx=0 tx=0).stop 2024-04-02T13:29:12.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.210+0000 7f24458eb640 1 -- 172.21.15.44:0/2335595169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24401538b0 msgr2=0x7f2440155380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:12.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.210+0000 7f24458eb640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24401538b0 0x7f2440155380 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f2428007370 tx=0x7f2428007770 comp rx=0 tx=0).stop 2024-04-02T13:29:12.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.210+0000 7f24458eb640 1 -- 172.21.15.44:0/2335595169 shutdown_connections 2024-04-02T13:29:12.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.210+0000 7f24458eb640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f242006c8f0 0x7f242006edb0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.210+0000 7f24458eb640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24401581a0 0x7f244015a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.210+0000 7f24458eb640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24401538b0 0x7f2440155380 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.210+0000 7f24458eb640 1 --2- 172.21.15.44:0/2335595169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f242c004d40 0x7f24401f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.210+0000 7f24458eb640 1 -- 172.21.15.44:0/2335595169 >> 172.21.15.44:0/2335595169 conn(0x7f2440073850 msgr2=0x7f24400b0230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.210+0000 7f24458eb640 1 -- 172.21.15.44:0/2335595169 shutdown_connections 2024-04-02T13:29:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.210+0000 7f24458eb640 1 -- 172.21.15.44:0/2335595169 wait complete. 2024-04-02T13:29:12.224 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T13:29:12.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.326+0000 7f2c903c8640 1 Processor -- start 2024-04-02T13:29:12.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.326+0000 7f2c903c8640 1 -- start start 2024-04-02T13:29:12.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c903c8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2c8815e550 0x7f2c8806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c903c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c8815f040 0x7f2c880d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c903c8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c8815fb50 con 0x7f2c8806d530 2024-04-02T13:29:12.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c903c8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2c8806d970 con 0x7f2c8815e550 2024-04-02T13:29:12.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c903c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2c8806ca90 con 0x7f2c8815f040 2024-04-02T13:29:12.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c8e164640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2c8815e550 0x7f2c8806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c8e164640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2c8815e550 0x7f2c8806cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51914/0 (socket says 172.21.15.44:51914) 2024-04-02T13:29:12.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c8e164640 1 -- 172.21.15.44:0/536167730 learned_addr learned my addr 172.21.15.44:0/536167730 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:12.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c8e965640 1 --2- 172.21.15.44:0/536167730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c8815f040 0x7f2c880d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c8e164640 1 -- 172.21.15.44:0/536167730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c8815f040 msgr2=0x7f2c880d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:12.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c8e164640 1 --2- 172.21.15.44:0/536167730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c8815f040 0x7f2c880d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c8e164640 1 -- 172.21.15.44:0/536167730 >> v1:172.21.15.69:6789/0 conn(0x7f2c8806d530 legacy=0x7f2c8815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:12.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c8e164640 1 -- 172.21.15.44:0/536167730 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c8815f590 con 0x7f2c8815e550 2024-04-02T13:29:12.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.327+0000 7f2c8e965640 1 --2- 172.21.15.44:0/536167730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c8815f040 0x7f2c880d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:12.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.328+0000 7f2c8e164640 1 --2- 172.21.15.44:0/536167730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c8815e550 0x7f2c8806cff0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f2c80009e50 tx=0x7f2c8002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2e347a94367c24 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:12.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.328+0000 7f2c8d162640 1 -- 172.21.15.44:0/536167730 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8002cb90 con 0x7f2c8815e550 2024-04-02T13:29:12.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.328+0000 7f2c8d162640 1 -- 172.21.15.44:0/536167730 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c8002cd30 con 0x7f2c8815e550 2024-04-02T13:29:12.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.328+0000 7f2c8d162640 1 -- 172.21.15.44:0/536167730 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c80034610 con 0x7f2c8815e550 2024-04-02T13:29:12.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.329+0000 7f2c903c8640 1 -- 172.21.15.44:0/536167730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c8815e550 msgr2=0x7f2c8806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:12.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.329+0000 7f2c903c8640 1 --2- 172.21.15.44:0/536167730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c8815e550 0x7f2c8806cff0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f2c80009e50 tx=0x7f2c8002bff0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.329+0000 7f2c903c8640 1 -- 172.21.15.44:0/536167730 shutdown_connections 2024-04-02T13:29:12.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.329+0000 7f2c903c8640 1 --2- 172.21.15.44:0/536167730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c8815f040 0x7f2c880d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.329+0000 7f2c903c8640 1 --2- 172.21.15.44:0/536167730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c8815e550 0x7f2c8806cff0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.329+0000 7f2c903c8640 1 -- 172.21.15.44:0/536167730 >> 172.21.15.44:0/536167730 conn(0x7f2c8809aa20 msgr2=0x7f2c8809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:12.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.329+0000 7f2c903c8640 1 -- 172.21.15.44:0/536167730 shutdown_connections 2024-04-02T13:29:12.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.329+0000 7f2c903c8640 1 -- 172.21.15.44:0/536167730 wait complete. 2024-04-02T13:29:12.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.330+0000 7f2c903c8640 1 Processor -- start 2024-04-02T13:29:12.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.330+0000 7f2c903c8640 1 -- start start 2024-04-02T13:29:12.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c903c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c8806d530 0x7f2c8814ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c903c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c8815e550 0x7f2c88151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c903c8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c8815f040 0x7f2c88151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c903c8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2c880d0590 con 0x7f2c8815e550 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c903c8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2c880d02b0 con 0x7f2c8815f040 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c903c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2c880cc4b0 con 0x7f2c8806d530 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c8d963640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c8815e550 0x7f2c88151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c8d963640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c8815e550 0x7f2c88151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51926/0 (socket says 172.21.15.44:51926) 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c8d963640 1 -- 172.21.15.44:0/1369703004 learned_addr learned my addr 172.21.15.44:0/1369703004 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c8e164640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c8806d530 0x7f2c8814ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c8e965640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c8815f040 0x7f2c88151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c8d963640 1 -- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c8806d530 msgr2=0x7f2c8814ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c8d963640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c8806d530 0x7f2c8814ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c8d963640 1 -- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c8815f040 msgr2=0x7f2c88151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c8d963640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c8815f040 0x7f2c88151da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:12.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.331+0000 7f2c8d963640 1 -- 172.21.15.44:0/1369703004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c88150490 con 0x7f2c8815e550 2024-04-02T13:29:12.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.332+0000 7f2c8d963640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c8815e550 0x7f2c88151860 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c00c400 tx=0x7f2c7c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:12.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.332+0000 7f2c86ffd640 1 -- 172.21.15.44:0/1369703004 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c017070 con 0x7f2c8815e550 2024-04-02T13:29:12.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.332+0000 7f2c86ffd640 1 -- 172.21.15.44:0/1369703004 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c7c0052d0 con 0x7f2c8815e550 2024-04-02T13:29:12.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.332+0000 7f2c86ffd640 1 -- 172.21.15.44:0/1369703004 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c003830 con 0x7f2c8815e550 2024-04-02T13:29:12.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.332+0000 7f2c903c8640 1 -- 172.21.15.44:0/1369703004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c88150770 con 0x7f2c8815e550 2024-04-02T13:29:12.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.332+0000 7f2c903c8640 1 -- 172.21.15.44:0/1369703004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c88150d30 con 0x7f2c8815e550 2024-04-02T13:29:12.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.333+0000 7f2c903c8640 1 -- 172.21.15.44:0/1369703004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c880684f0 con 0x7f2c8815e550 2024-04-02T13:29:12.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.333+0000 7f2c86ffd640 1 -- 172.21.15.44:0/1369703004 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2c7c01a070 con 0x7f2c8815e550 2024-04-02T13:29:12.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.333+0000 7f2c86ffd640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2c7406c8f0 0x7f2c7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:12.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.334+0000 7f2c86ffd640 1 -- 172.21.15.44:0/1369703004 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6101+0+0 (secure 0 0 0) 0x7f2c7c08af00 con 0x7f2c8815e550 2024-04-02T13:29:12.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.334+0000 7f2c8e164640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2c7406c8f0 0x7f2c7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:12.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.334+0000 7f2c8e164640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2c7406c8f0 0x7f2c7406edb0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2c80035000 tx=0x7f2c80030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:12.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.338+0000 7f2c86ffd640 1 -- 172.21.15.44:0/1369703004 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2c7c058be0 con 0x7f2c8815e550 2024-04-02T13:29:12.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:12.567+0000 7f2c903c8640 1 -- 172.21.15.44:0/1369703004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f2c880cf4d0 con 0x7f2c8815e550 2024-04-02T13:29:13.533 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:29:13.530+0000 7fe21e740640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:29:13.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.565+0000 7f2c86ffd640 1 -- 172.21.15.44:0/1369703004 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v81) ==== 118+0+0 (secure 0 0 0) 0x7f2c7c05cee0 con 0x7f2c8815e550 2024-04-02T13:29:13.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.568+0000 7f2c903c8640 1 -- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2c7406c8f0 msgr2=0x7f2c7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:13.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.568+0000 7f2c903c8640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2c7406c8f0 0x7f2c7406edb0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2c80035000 tx=0x7f2c80030000 comp rx=0 tx=0).stop 2024-04-02T13:29:13.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.568+0000 7f2c903c8640 1 -- 172.21.15.44:0/1369703004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c8815e550 msgr2=0x7f2c88151860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:13.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.568+0000 7f2c903c8640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c8815e550 0x7f2c88151860 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c00c400 tx=0x7f2c7c00c8c0 comp rx=0 tx=0).stop 2024-04-02T13:29:13.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.568+0000 7f2c903c8640 1 -- 172.21.15.44:0/1369703004 shutdown_connections 2024-04-02T13:29:13.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.568+0000 7f2c903c8640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2c7406c8f0 0x7f2c7406edb0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:13.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.568+0000 7f2c903c8640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c8815f040 0x7f2c88151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:13.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.568+0000 7f2c903c8640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c8815e550 0x7f2c88151860 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:13.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.568+0000 7f2c903c8640 1 --2- 172.21.15.44:0/1369703004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c8806d530 0x7f2c8814ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:13.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.568+0000 7f2c903c8640 1 -- 172.21.15.44:0/1369703004 >> 172.21.15.44:0/1369703004 conn(0x7f2c8809aa20 msgr2=0x7f2c880cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:13.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.568+0000 7f2c903c8640 1 -- 172.21.15.44:0/1369703004 shutdown_connections 2024-04-02T13:29:13.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.569+0000 7f2c903c8640 1 -- 172.21.15.44:0/1369703004 wait complete. 2024-04-02T13:29:13.584 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T13:29:13.585 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T13:29:13.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.693+0000 7f823aa77640 1 Processor -- start 2024-04-02T13:29:13.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.693+0000 7f823aa77640 1 -- start start 2024-04-02T13:29:13.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.693+0000 7f823aa77640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f823406d700 0x7f823415e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:13.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.693+0000 7f823aa77640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f823415ee70 0x7f82340d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:13.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.693+0000 7f823aa77640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f823415fb50 con 0x7f823415e550 2024-04-02T13:29:13.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.693+0000 7f823aa77640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f823406ca90 con 0x7f823415ee70 2024-04-02T13:29:13.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.693+0000 7f823aa77640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f823415f590 con 0x7f823406d700 2024-04-02T13:29:13.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.694+0000 7f8233fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f823406d700 0x7f823415e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:13.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.694+0000 7f8238813640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f823415e550 0x7f823406cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:51832/0 (socket says 172.21.15.44:51832) 2024-04-02T13:29:13.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.694+0000 7f8239014640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f823415ee70 0x7f82340d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:13.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.694+0000 7f8238813640 1 -- 172.21.15.44:0/1267461783 learned_addr learned my addr 172.21.15.44:0/1267461783 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:13.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.694+0000 7f82337fe640 1 -- 172.21.15.44:0/1267461783 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1803090522 0 0) 0x7f823415fb50 con 0x7f823415e550 2024-04-02T13:29:13.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.694+0000 7f82337fe640 1 -- 172.21.15.44:0/1267461783 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8224003520 con 0x7f823415e550 2024-04-02T13:29:13.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.694+0000 7f8239014640 1 -- 172.21.15.44:0/1267461783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f823406d700 msgr2=0x7f823415e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:13.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.694+0000 7f8239014640 1 --2- 172.21.15.44:0/1267461783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f823406d700 0x7f823415e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:13.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.694+0000 7f8239014640 1 -- 172.21.15.44:0/1267461783 >> v1:172.21.15.69:6789/0 conn(0x7f823415e550 legacy=0x7f823406cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:13.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.694+0000 7f8239014640 1 -- 172.21.15.44:0/1267461783 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823415f320 con 0x7f823415ee70 2024-04-02T13:29:13.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.694+0000 7f8233fff640 1 --2- 172.21.15.44:0/1267461783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f823406d700 0x7f823415e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:29:13.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.695+0000 7f8239014640 1 --2- 172.21.15.44:0/1267461783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f823415ee70 0x7f82340d0ab0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f822c009d90 tx=0x7f822c02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5643ef0b32f75ae9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:13.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.695+0000 7f82337fe640 1 -- 172.21.15.44:0/1267461783 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c039070 con 0x7f823415ee70 2024-04-02T13:29:13.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.695+0000 7f82337fe640 1 -- 172.21.15.44:0/1267461783 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f822c02cce0 con 0x7f823415ee70 2024-04-02T13:29:13.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.695+0000 7f82337fe640 1 -- 172.21.15.44:0/1267461783 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c034660 con 0x7f823415ee70 2024-04-02T13:29:13.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.696+0000 7f823aa77640 1 -- 172.21.15.44:0/1267461783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f823415ee70 msgr2=0x7f82340d0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:13.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.696+0000 7f823aa77640 1 --2- 172.21.15.44:0/1267461783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f823415ee70 0x7f82340d0ab0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f822c009d90 tx=0x7f822c02bf30 comp rx=0 tx=0).stop 2024-04-02T13:29:13.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.696+0000 7f823aa77640 1 -- 172.21.15.44:0/1267461783 shutdown_connections 2024-04-02T13:29:13.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.696+0000 7f823aa77640 1 --2- 172.21.15.44:0/1267461783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f823415ee70 0x7f82340d0ab0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:13.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.696+0000 7f823aa77640 1 --2- 172.21.15.44:0/1267461783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f823406d700 0x7f823415e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:13.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.696+0000 7f823aa77640 1 -- 172.21.15.44:0/1267461783 >> 172.21.15.44:0/1267461783 conn(0x7f823409aa20 msgr2=0x7f823409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:13.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.696+0000 7f823aa77640 1 -- 172.21.15.44:0/1267461783 shutdown_connections 2024-04-02T13:29:13.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.696+0000 7f823aa77640 1 -- 172.21.15.44:0/1267461783 wait complete. 2024-04-02T13:29:13.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.697+0000 7f823aa77640 1 Processor -- start 2024-04-02T13:29:13.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.697+0000 7f823aa77640 1 -- start start 2024-04-02T13:29:13.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f823aa77640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f823406d700 0x7f82341510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:13.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f823aa77640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f823415e550 0x7f82341f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:13.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f823aa77640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f823415ee70 0x7f82341f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:13.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f823aa77640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f82340d0590 con 0x7f823406d700 2024-04-02T13:29:13.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f823aa77640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f82340d02b0 con 0x7f823415e550 2024-04-02T13:29:13.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f823aa77640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f82340cc4b0 con 0x7f823415ee70 2024-04-02T13:29:13.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f8238813640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f823406d700 0x7f82341510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:13.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f8238813640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f823406d700 0x7f82341510c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51928/0 (socket says 172.21.15.44:51928) 2024-04-02T13:29:13.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f8238813640 1 -- 172.21.15.44:0/2567683888 learned_addr learned my addr 172.21.15.44:0/2567683888 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:13.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f8233fff640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f823415e550 0x7f82341f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:13.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f8239014640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f823415ee70 0x7f82341f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:13.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f8238813640 1 -- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f823415ee70 msgr2=0x7f82341f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:13.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f8238813640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f823415ee70 0x7f82341f3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:13.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f8238813640 1 -- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f823415e550 msgr2=0x7f82341f7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:13.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f8238813640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f823415e550 0x7f82341f7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:13.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.698+0000 7f8238813640 1 -- 172.21.15.44:0/2567683888 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82341f7d50 con 0x7f823406d700 2024-04-02T13:29:13.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.699+0000 7f8238813640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f823406d700 0x7f82341510c0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f822800c390 tx=0x7f822800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:13.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.699+0000 7f82317fa640 1 -- 172.21.15.44:0/2567683888 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8228017070 con 0x7f823406d700 2024-04-02T13:29:13.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.699+0000 7f82317fa640 1 -- 172.21.15.44:0/2567683888 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8228005ce0 con 0x7f823406d700 2024-04-02T13:29:13.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.699+0000 7f82317fa640 1 -- 172.21.15.44:0/2567683888 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82280072f0 con 0x7f823406d700 2024-04-02T13:29:13.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.699+0000 7f823aa77640 1 -- 172.21.15.44:0/2567683888 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82341f8030 con 0x7f823406d700 2024-04-02T13:29:13.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.699+0000 7f823aa77640 1 -- 172.21.15.44:0/2567683888 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82340b4ee0 con 0x7f823406d700 2024-04-02T13:29:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.700+0000 7f823aa77640 1 -- 172.21.15.44:0/2567683888 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82340684f0 con 0x7f823406d700 2024-04-02T13:29:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.700+0000 7f82317fa640 1 -- 172.21.15.44:0/2567683888 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f822801a070 con 0x7f823406d700 2024-04-02T13:29:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.701+0000 7f82317fa640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f821806c8f0 0x7f821806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.701+0000 7f82317fa640 1 -- 172.21.15.44:0/2567683888 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f822808afc0 con 0x7f823406d700 2024-04-02T13:29:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.701+0000 7f8233fff640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f821806c8f0 0x7f821806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.701+0000 7f8233fff640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f821806c8f0 0x7f821806edb0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8220009560 tx=0x7f8220009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:13.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.705+0000 7f82317fa640 1 -- 172.21.15.44:0/2567683888 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8228058ca0 con 0x7f823406d700 2024-04-02T13:29:13.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:13.930+0000 7f823aa77640 1 -- 172.21.15.44:0/2567683888 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f82340cf2e0 con 0x7f823406d700 2024-04-02T13:29:14.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.567+0000 7f82317fa640 1 -- 172.21.15.44:0/2567683888 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v82) ==== 122+0+0 (secure 0 0 0) 0x7f822805cfa0 con 0x7f823406d700 2024-04-02T13:29:14.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.570+0000 7f823aa77640 1 -- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f821806c8f0 msgr2=0x7f821806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:14.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.570+0000 7f823aa77640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f821806c8f0 0x7f821806edb0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8220009560 tx=0x7f8220009190 comp rx=0 tx=0).stop 2024-04-02T13:29:14.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.570+0000 7f823aa77640 1 -- 172.21.15.44:0/2567683888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f823406d700 msgr2=0x7f82341510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:14.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.570+0000 7f823aa77640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f823406d700 0x7f82341510c0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f822800c390 tx=0x7f822800c850 comp rx=0 tx=0).stop 2024-04-02T13:29:14.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.570+0000 7f823aa77640 1 -- 172.21.15.44:0/2567683888 shutdown_connections 2024-04-02T13:29:14.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.570+0000 7f823aa77640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f821806c8f0 0x7f821806edb0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:14.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.570+0000 7f823aa77640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f823415ee70 0x7f82341f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:14.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.570+0000 7f823aa77640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f823415e550 0x7f82341f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:14.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.570+0000 7f823aa77640 1 --2- 172.21.15.44:0/2567683888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f823406d700 0x7f82341510c0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:14.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.570+0000 7f823aa77640 1 -- 172.21.15.44:0/2567683888 >> 172.21.15.44:0/2567683888 conn(0x7f823409aa20 msgr2=0x7f82340d09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:14.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.570+0000 7f823aa77640 1 -- 172.21.15.44:0/2567683888 shutdown_connections 2024-04-02T13:29:14.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.570+0000 7f823aa77640 1 -- 172.21.15.44:0/2567683888 wait complete. 2024-04-02T13:29:14.586 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T13:29:14.587 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:29:14.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.691+0000 7f2896d22640 1 Processor -- start 2024-04-02T13:29:14.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.691+0000 7f2896d22640 1 -- start start 2024-04-02T13:29:14.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.692+0000 7f2896d22640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f28900cece0 0x7f28900cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:14.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.692+0000 7f2896d22640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f289006ece0 0x7f28900acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:14.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.692+0000 7f2896d22640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28900cf600 con 0x7f28900d09f0 2024-04-02T13:29:14.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.692+0000 7f2896d22640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f289006d9e0 con 0x7f28900cece0 2024-04-02T13:29:14.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.692+0000 7f2896d22640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f289006dc40 con 0x7f289006ece0 2024-04-02T13:29:14.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.692+0000 7f2894abe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f28900cece0 0x7f28900cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:14.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.692+0000 7f2894abe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f28900cece0 0x7f28900cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49006/0 (socket says 172.21.15.44:49006) 2024-04-02T13:29:14.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.692+0000 7f2894abe640 1 -- 172.21.15.44:0/2981587295 learned_addr learned my addr 172.21.15.44:0/2981587295 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:14.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.692+0000 7f2894abe640 1 -- 172.21.15.44:0/2981587295 >> v2:172.21.15.69:3300/0 conn(0x7f289006ece0 msgr2=0x7f28900acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:14.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.692+0000 7f28952bf640 1 --2- 172.21.15.44:0/2981587295 >> v2:172.21.15.69:3300/0 conn(0x7f289006ece0 0x7f28900acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:14.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.692+0000 7f2894abe640 1 --2- 172.21.15.44:0/2981587295 >> v2:172.21.15.69:3300/0 conn(0x7f289006ece0 0x7f28900acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:14.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.692+0000 7f2894abe640 1 -- 172.21.15.44:0/2981587295 >> v1:172.21.15.69:6789/0 conn(0x7f28900d09f0 legacy=0x7f289006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:14.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.692+0000 7f2894abe640 1 -- 172.21.15.44:0/2981587295 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f289006df20 con 0x7f28900cece0 2024-04-02T13:29:14.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.693+0000 7f2894abe640 1 --2- 172.21.15.44:0/2981587295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28900cece0 0x7f28900cf0c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f289006f1a0 tx=0x7f288402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753d692ed0953517 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:14.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.693+0000 7f288f7fe640 1 -- 172.21.15.44:0/2981587295 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288402e070 con 0x7f28900cece0 2024-04-02T13:29:14.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.693+0000 7f288f7fe640 1 -- 172.21.15.44:0/2981587295 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2884037070 con 0x7f28900cece0 2024-04-02T13:29:14.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.693+0000 7f288f7fe640 1 -- 172.21.15.44:0/2981587295 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2884031a00 con 0x7f28900cece0 2024-04-02T13:29:14.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.694+0000 7f2896d22640 1 -- 172.21.15.44:0/2981587295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28900cece0 msgr2=0x7f28900cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:14.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.694+0000 7f2896d22640 1 --2- 172.21.15.44:0/2981587295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28900cece0 0x7f28900cf0c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f289006f1a0 tx=0x7f288402b880 comp rx=0 tx=0).stop 2024-04-02T13:29:14.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.694+0000 7f2896d22640 1 -- 172.21.15.44:0/2981587295 shutdown_connections 2024-04-02T13:29:14.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.694+0000 7f2896d22640 1 --2- 172.21.15.44:0/2981587295 >> v2:172.21.15.69:3300/0 conn(0x7f289006ece0 0x7f28900acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:14.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.694+0000 7f2896d22640 1 --2- 172.21.15.44:0/2981587295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28900cece0 0x7f28900cf0c0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:14.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.694+0000 7f2896d22640 1 -- 172.21.15.44:0/2981587295 >> 172.21.15.44:0/2981587295 conn(0x7f28900cfcd0 msgr2=0x7f28900d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:14.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.694+0000 7f2896d22640 1 -- 172.21.15.44:0/2981587295 shutdown_connections 2024-04-02T13:29:14.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.694+0000 7f2896d22640 1 -- 172.21.15.44:0/2981587295 wait complete. 2024-04-02T13:29:14.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.695+0000 7f2896d22640 1 Processor -- start 2024-04-02T13:29:14.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.695+0000 7f2896d22640 1 -- start start 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f2896d22640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f289006ece0 0x7f28901ee490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f2896d22640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28900cece0 0x7f28901e3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f2896d22640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28900d09f0 0x7f28901e3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f2896d22640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f289006e7c0 con 0x7f28900d09f0 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f2896d22640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f289006e4e0 con 0x7f289006ece0 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f2896d22640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f289015b220 con 0x7f28900cece0 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f28952bf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28900d09f0 0x7f28901e3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f28952bf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28900d09f0 0x7f28901e3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49020/0 (socket says 172.21.15.44:49020) 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f28952bf640 1 -- 172.21.15.44:0/1291512616 learned_addr learned my addr 172.21.15.44:0/1291512616 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f2894abe640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f289006ece0 0x7f28901ee490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f288ffff640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28900cece0 0x7f28901e3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f28952bf640 1 -- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28900cece0 msgr2=0x7f28901e3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f28952bf640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28900cece0 0x7f28901e3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f28952bf640 1 -- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f289006ece0 msgr2=0x7f28901ee490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f28952bf640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f289006ece0 0x7f28901ee490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:14.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f28952bf640 1 -- 172.21.15.44:0/1291512616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f289009c880 con 0x7f28900d09f0 2024-04-02T13:29:14.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.696+0000 7f28952bf640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28900d09f0 0x7f28901e3f50 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f288800cd70 tx=0x7f288800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:14.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.697+0000 7f288d7fa640 1 -- 172.21.15.44:0/1291512616 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2888003b90 con 0x7f28900d09f0 2024-04-02T13:29:14.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.697+0000 7f288d7fa640 1 -- 172.21.15.44:0/1291512616 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2888003d30 con 0x7f28900d09f0 2024-04-02T13:29:14.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.697+0000 7f2896d22640 1 -- 172.21.15.44:0/1291512616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f289009cb60 con 0x7f28900d09f0 2024-04-02T13:29:14.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.697+0000 7f288d7fa640 1 -- 172.21.15.44:0/1291512616 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2888004240 con 0x7f28900d09f0 2024-04-02T13:29:14.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.697+0000 7f2896d22640 1 -- 172.21.15.44:0/1291512616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f289009d120 con 0x7f28900d09f0 2024-04-02T13:29:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.698+0000 7f2896d22640 1 -- 172.21.15.44:0/1291512616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28900684f0 con 0x7f28900d09f0 2024-04-02T13:29:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.698+0000 7f288d7fa640 1 -- 172.21.15.44:0/1291512616 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f28880043e0 con 0x7f28900d09f0 2024-04-02T13:29:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.698+0000 7f288d7fa640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f287406c8f0 0x7f287406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.699+0000 7f288d7fa640 1 -- 172.21.15.44:0/1291512616 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f288808c7a0 con 0x7f28900d09f0 2024-04-02T13:29:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.699+0000 7f2894abe640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f287406c8f0 0x7f287406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.699+0000 7f2894abe640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f287406c8f0 0x7f287406edb0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2884036000 tx=0x7f2884008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:14.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.703+0000 7f288d7fa640 1 -- 172.21.15.44:0/1291512616 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f288805aad0 con 0x7f28900d09f0 2024-04-02T13:29:14.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.925+0000 7f2896d22640 1 -- 172.21.15.44:0/1291512616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2890075550 con 0x7f28900d09f0 2024-04-02T13:29:14.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.926+0000 7f288d7fa640 1 -- 172.21.15.44:0/1291512616 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5183 (secure 0 0 0) 0x7f288805a3d0 con 0x7f28900d09f0 2024-04-02T13:29:14.928 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:14.929 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7696,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2715855087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2715855087},{"type":"v1","addr":"172.21.15.69:6836","nonce":2715855087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7699,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3850680553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3850680553},{"type":"v1","addr":"172.21.15.69:6837","nonce":3850680553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/1330712383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1330712383},{"type":"v1","addr":"172.21.15.44:6835","nonce":1330712383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:29:10.907920+0000","modified":"2024-04-02T13:29:13.930909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8760":{"gid":8760,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6834/3259420283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":3259420283},{"type":"v1","addr":"172.21.15.44:6834","nonce":3259420283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8760,"qdb_cluster":[8760]},"id":7}]} 2024-04-02T13:29:14.929 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 82 2024-04-02T13:29:14.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.927+0000 7f2896d22640 1 -- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f287406c8f0 msgr2=0x7f287406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:14.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.927+0000 7f2896d22640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f287406c8f0 0x7f287406edb0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2884036000 tx=0x7f2884008030 comp rx=0 tx=0).stop 2024-04-02T13:29:14.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.927+0000 7f2896d22640 1 -- 172.21.15.44:0/1291512616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28900d09f0 msgr2=0x7f28901e3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:14.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.927+0000 7f2896d22640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28900d09f0 0x7f28901e3f50 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f288800cd70 tx=0x7f288800a660 comp rx=0 tx=0).stop 2024-04-02T13:29:14.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.928+0000 7f2896d22640 1 -- 172.21.15.44:0/1291512616 shutdown_connections 2024-04-02T13:29:14.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.928+0000 7f2896d22640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f287406c8f0 0x7f287406edb0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:14.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.928+0000 7f2896d22640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28900d09f0 0x7f28901e3f50 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:14.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.928+0000 7f2896d22640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28900cece0 0x7f28901e3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:14.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.928+0000 7f2896d22640 1 --2- 172.21.15.44:0/1291512616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f289006ece0 0x7f28901ee490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:14.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.928+0000 7f2896d22640 1 -- 172.21.15.44:0/1291512616 >> 172.21.15.44:0/1291512616 conn(0x7f28900cfcd0 msgr2=0x7f28900b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:14.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.928+0000 7f2896d22640 1 -- 172.21.15.44:0/1291512616 shutdown_connections 2024-04-02T13:29:14.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:14.928+0000 7f2896d22640 1 -- 172.21.15.44:0/1291512616 wait complete. 2024-04-02T13:29:14.941 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:29:15.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.041+0000 7f1e0b18d640 1 Processor -- start 2024-04-02T13:29:15.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.041+0000 7f1e0b18d640 1 -- start start 2024-04-02T13:29:15.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.041+0000 7f1e0b18d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1e0415e4a0 0x7f1e040aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.041+0000 7f1e0b18d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1e0406d550 0x7f1e040d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.041+0000 7f1e0b18d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e040cc600 con 0x7f1e040d0c50 2024-04-02T13:29:15.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.041+0000 7f1e0b18d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f1e040cc8e0 con 0x7f1e0406d550 2024-04-02T13:29:15.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.041+0000 7f1e0b18d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1e040ccbc0 con 0x7f1e0415e4a0 2024-04-02T13:29:15.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.042+0000 7f1e0972a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1e0406d550 0x7f1e040d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.042+0000 7f1e0972a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f1e0406d550 0x7f1e040d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49036/0 (socket says 172.21.15.44:49036) 2024-04-02T13:29:15.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.042+0000 7f1e0972a640 1 -- 172.21.15.44:0/3344304578 learned_addr learned my addr 172.21.15.44:0/3344304578 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:15.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.042+0000 7f1e08f29640 1 --2- 172.21.15.44:0/3344304578 >> v2:172.21.15.69:3300/0 conn(0x7f1e0415e4a0 0x7f1e040aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.042+0000 7f1e0972a640 1 -- 172.21.15.44:0/3344304578 >> v2:172.21.15.69:3300/0 conn(0x7f1e0415e4a0 msgr2=0x7f1e040aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:15.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.042+0000 7f1e0972a640 1 --2- 172.21.15.44:0/3344304578 >> v2:172.21.15.69:3300/0 conn(0x7f1e0415e4a0 0x7f1e040aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.042+0000 7f1e0972a640 1 -- 172.21.15.44:0/3344304578 >> v1:172.21.15.69:6789/0 conn(0x7f1e040d0c50 legacy=0x7f1e0406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.042+0000 7f1e0972a640 1 -- 172.21.15.44:0/3344304578 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e040ccea0 con 0x7f1e0406d550 2024-04-02T13:29:15.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.043+0000 7f1e0972a640 1 --2- 172.21.15.44:0/3344304578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e0406d550 0x7f1e040d10b0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009e50 tx=0x7f1dfc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d99e3380dc360ea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:15.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.043+0000 7f1e037fe640 1 -- 172.21.15.44:0/3344304578 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc02cb20 con 0x7f1e0406d550 2024-04-02T13:29:15.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.043+0000 7f1e037fe640 1 -- 172.21.15.44:0/3344304578 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dfc02ccc0 con 0x7f1e0406d550 2024-04-02T13:29:15.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.043+0000 7f1e037fe640 1 -- 172.21.15.44:0/3344304578 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc034610 con 0x7f1e0406d550 2024-04-02T13:29:15.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.043+0000 7f1e0b18d640 1 -- 172.21.15.44:0/3344304578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e0406d550 msgr2=0x7f1e040d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.043+0000 7f1e0b18d640 1 --2- 172.21.15.44:0/3344304578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e0406d550 0x7f1e040d10b0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009e50 tx=0x7f1dfc02bf80 comp rx=0 tx=0).stop 2024-04-02T13:29:15.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.043+0000 7f1e0b18d640 1 -- 172.21.15.44:0/3344304578 shutdown_connections 2024-04-02T13:29:15.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.043+0000 7f1e0b18d640 1 --2- 172.21.15.44:0/3344304578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e0406d550 0x7f1e040d10b0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.043+0000 7f1e0b18d640 1 --2- 172.21.15.44:0/3344304578 >> v2:172.21.15.69:3300/0 conn(0x7f1e0415e4a0 0x7f1e040aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.043+0000 7f1e0b18d640 1 -- 172.21.15.44:0/3344304578 >> 172.21.15.44:0/3344304578 conn(0x7f1e0409aa20 msgr2=0x7f1e0409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:15.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.043+0000 7f1e0b18d640 1 -- 172.21.15.44:0/3344304578 shutdown_connections 2024-04-02T13:29:15.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.043+0000 7f1e0b18d640 1 -- 172.21.15.44:0/3344304578 wait complete. 2024-04-02T13:29:15.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.045+0000 7f1e0b18d640 1 Processor -- start 2024-04-02T13:29:15.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.045+0000 7f1e0b18d640 1 -- start start 2024-04-02T13:29:15.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.045+0000 7f1e0b18d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e0406d550 0x7f1e040b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.045+0000 7f1e0b18d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e040d0c50 0x7f1e04160170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.045+0000 7f1e0b18d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e0415e4a0 0x7f1e0415ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.045+0000 7f1e0b18d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0415e880 con 0x7f1e040d0c50 2024-04-02T13:29:15.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.045+0000 7f1e0b18d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1e040cd460 con 0x7f1e0415e4a0 2024-04-02T13:29:15.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.045+0000 7f1e0b18d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1e040af130 con 0x7f1e0406d550 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e03fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e040d0c50 0x7f1e04160170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e03fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e040d0c50 0x7f1e04160170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49046/0 (socket says 172.21.15.44:49046) 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e03fff640 1 -- 172.21.15.44:0/1908634890 learned_addr learned my addr 172.21.15.44:0/1908634890 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e08f29640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e0406d550 0x7f1e040b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e0972a640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e0415e4a0 0x7f1e0415ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e03fff640 1 -- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e0406d550 msgr2=0x7f1e040b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e03fff640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e0406d550 0x7f1e040b79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e03fff640 1 -- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e0415e4a0 msgr2=0x7f1e0415ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e03fff640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e0415e4a0 0x7f1e0415ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e03fff640 1 -- 172.21.15.44:0/1908634890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e041606b0 con 0x7f1e040d0c50 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e03fff640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e040d0c50 0x7f1e04160170 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f1df000c430 tx=0x7f1df000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e017fa640 1 -- 172.21.15.44:0/1908634890 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0017070 con 0x7f1e040d0c50 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e017fa640 1 -- 172.21.15.44:0/1908634890 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1df00052d0 con 0x7f1e040d0c50 2024-04-02T13:29:15.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e0b18d640 1 -- 172.21.15.44:0/1908634890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e040c85e0 con 0x7f1e040d0c50 2024-04-02T13:29:15.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e017fa640 1 -- 172.21.15.44:0/1908634890 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0003830 con 0x7f1e040d0c50 2024-04-02T13:29:15.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.046+0000 7f1e0b18d640 1 -- 172.21.15.44:0/1908634890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e040b6ef0 con 0x7f1e040d0c50 2024-04-02T13:29:15.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.047+0000 7f1e0b18d640 1 -- 172.21.15.44:0/1908634890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e040684f0 con 0x7f1e040d0c50 2024-04-02T13:29:15.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.048+0000 7f1e017fa640 1 -- 172.21.15.44:0/1908634890 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1df001a070 con 0x7f1e040d0c50 2024-04-02T13:29:15.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.048+0000 7f1e017fa640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1de806c8f0 0x7f1de806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.048+0000 7f1e017fa640 1 -- 172.21.15.44:0/1908634890 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f1df008b070 con 0x7f1e040d0c50 2024-04-02T13:29:15.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.048+0000 7f1e08f29640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1de806c8f0 0x7f1de806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.049+0000 7f1e08f29640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1de806c8f0 0x7f1de806edb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1df8006fd0 tx=0x7f1df8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:15.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.053+0000 7f1e017fa640 1 -- 172.21.15.44:0/1908634890 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1e041606b0 con 0x7f1e040d0c50 2024-04-02T13:29:15.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.236+0000 7f1e0b18d640 1 -- 172.21.15.44:0/1908634890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e0415d850 con 0x7f1e040d0c50 2024-04-02T13:29:15.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.238+0000 7f1e017fa640 1 -- 172.21.15.44:0/1908634890 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+16836 (secure 0 0 0) 0x7f1df0059310 con 0x7f1e040d0c50 2024-04-02T13:29:15.241 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:15.241 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":106,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:29:12.549182+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:29:09.192430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-02T13:29:09.860748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:29:15.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.239+0000 7f1e0b18d640 1 -- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1de806c8f0 msgr2=0x7f1de806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.239+0000 7f1e0b18d640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1de806c8f0 0x7f1de806edb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1df8006fd0 tx=0x7f1df8008040 comp rx=0 tx=0).stop 2024-04-02T13:29:15.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.239+0000 7f1e0b18d640 1 -- 172.21.15.44:0/1908634890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e040d0c50 msgr2=0x7f1e04160170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.239+0000 7f1e0b18d640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e040d0c50 0x7f1e04160170 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f1df000c430 tx=0x7f1df000c8f0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.240+0000 7f1e0b18d640 1 -- 172.21.15.44:0/1908634890 shutdown_connections 2024-04-02T13:29:15.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.240+0000 7f1e0b18d640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1de806c8f0 0x7f1de806edb0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.240+0000 7f1e0b18d640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e0415e4a0 0x7f1e0415ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.240+0000 7f1e0b18d640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e040d0c50 0x7f1e04160170 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.240+0000 7f1e0b18d640 1 --2- 172.21.15.44:0/1908634890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e0406d550 0x7f1e040b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.240+0000 7f1e0b18d640 1 -- 172.21.15.44:0/1908634890 >> 172.21.15.44:0/1908634890 conn(0x7f1e0409aa20 msgr2=0x7f1e04067230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:15.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.240+0000 7f1e0b18d640 1 -- 172.21.15.44:0/1908634890 shutdown_connections 2024-04-02T13:29:15.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.240+0000 7f1e0b18d640 1 -- 172.21.15.44:0/1908634890 wait complete. 2024-04-02T13:29:15.253 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T13:29:15.254 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:29:15.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.352+0000 7f7b7bb74640 1 Processor -- start 2024-04-02T13:29:15.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.352+0000 7f7b7bb74640 1 -- start start 2024-04-02T13:29:15.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.353+0000 7f7b7bb74640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7b7406cf90 0x7f7b740cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.353+0000 7f7b7bb74640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7b740cfe30 0x7f7b740d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.353+0000 7f7b7bb74640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b7406c1d0 con 0x7f7b740d0a30 2024-04-02T13:29:15.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.353+0000 7f7b7bb74640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7b7406cb00 con 0x7f7b7406cf90 2024-04-02T13:29:15.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.353+0000 7f7b7bb74640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7b7406d3f0 con 0x7f7b740cfe30 2024-04-02T13:29:15.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.353+0000 7f7b7910f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7b7406cf90 0x7f7b740cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.353+0000 7f7b79910640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7b740d0a30 0x7f7b740d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40426/0 (socket says 172.21.15.44:40426) 2024-04-02T13:29:15.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.353+0000 7f7b79910640 1 -- 172.21.15.44:0/743803270 learned_addr learned my addr 172.21.15.44:0/743803270 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:15.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.354+0000 7f7b7a111640 1 --2- 172.21.15.44:0/743803270 >> v2:172.21.15.69:3300/0 conn(0x7f7b740cfe30 0x7f7b740d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.354+0000 7f7b7910f640 1 -- 172.21.15.44:0/743803270 >> v2:172.21.15.69:3300/0 conn(0x7f7b740cfe30 msgr2=0x7f7b740d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:15.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.354+0000 7f7b7910f640 1 --2- 172.21.15.44:0/743803270 >> v2:172.21.15.69:3300/0 conn(0x7f7b740cfe30 0x7f7b740d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.354+0000 7f7b7910f640 1 -- 172.21.15.44:0/743803270 >> v1:172.21.15.44:6789/0 conn(0x7f7b740d0a30 legacy=0x7f7b740d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.354+0000 7f7b7910f640 1 -- 172.21.15.44:0/743803270 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7406c4b0 con 0x7f7b7406cf90 2024-04-02T13:29:15.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.354+0000 7f7b7890e640 1 -- 172.21.15.44:0/743803270 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2613503447 0 0) 0x7f7b7406c1d0 con 0x7f7b740d0a30 2024-04-02T13:29:15.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.354+0000 7f7b7a111640 1 --2- 172.21.15.44:0/743803270 >> v2:172.21.15.69:3300/0 conn(0x7f7b740cfe30 0x7f7b740d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:15.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.354+0000 7f7b7910f640 1 --2- 172.21.15.44:0/743803270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b7406cf90 0x7f7b740cf8f0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f7b7408c030 tx=0x7f7b6802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89558b9c796a4f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:15.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.355+0000 7f7b7890e640 1 -- 172.21.15.44:0/743803270 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b680315a0 con 0x7f7b7406cf90 2024-04-02T13:29:15.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.355+0000 7f7b7890e640 1 -- 172.21.15.44:0/743803270 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b68037070 con 0x7f7b7406cf90 2024-04-02T13:29:15.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.355+0000 7f7b7890e640 1 -- 172.21.15.44:0/743803270 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b68031960 con 0x7f7b7406cf90 2024-04-02T13:29:15.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.355+0000 7f7b7bb74640 1 -- 172.21.15.44:0/743803270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b7406cf90 msgr2=0x7f7b740cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.355+0000 7f7b7bb74640 1 --2- 172.21.15.44:0/743803270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b7406cf90 0x7f7b740cf8f0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f7b7408c030 tx=0x7f7b6802b840 comp rx=0 tx=0).stop 2024-04-02T13:29:15.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.355+0000 7f7b7bb74640 1 -- 172.21.15.44:0/743803270 shutdown_connections 2024-04-02T13:29:15.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.355+0000 7f7b7bb74640 1 --2- 172.21.15.44:0/743803270 >> v2:172.21.15.69:3300/0 conn(0x7f7b740cfe30 0x7f7b740d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.355+0000 7f7b7bb74640 1 --2- 172.21.15.44:0/743803270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b7406cf90 0x7f7b740cf8f0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.355+0000 7f7b7bb74640 1 -- 172.21.15.44:0/743803270 >> 172.21.15.44:0/743803270 conn(0x7f7b7409d470 msgr2=0x7f7b740d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:15.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.355+0000 7f7b7bb74640 1 -- 172.21.15.44:0/743803270 shutdown_connections 2024-04-02T13:29:15.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.355+0000 7f7b7bb74640 1 -- 172.21.15.44:0/743803270 wait complete. 2024-04-02T13:29:15.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.356+0000 7f7b7bb74640 1 Processor -- start 2024-04-02T13:29:15.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.356+0000 7f7b7bb74640 1 -- start start 2024-04-02T13:29:15.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.357+0000 7f7b7bb74640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b7406cf90 0x7f7b7414f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.357+0000 7f7b7bb74640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b740cfe30 0x7f7b740c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.357+0000 7f7b7bb74640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b740d0a30 0x7f7b740cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.357+0000 7f7b7bb74640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7415e870 con 0x7f7b740cfe30 2024-04-02T13:29:15.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.357+0000 7f7b7bb74640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7415e6f0 con 0x7f7b740d0a30 2024-04-02T13:29:15.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.357+0000 7f7b7bb74640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7b740d4da0 con 0x7f7b7406cf90 2024-04-02T13:29:15.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.357+0000 7f7b7910f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b740cfe30 0x7f7b740c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.357+0000 7f7b7910f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b740cfe30 0x7f7b740c84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49064/0 (socket says 172.21.15.44:49064) 2024-04-02T13:29:15.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.357+0000 7f7b7910f640 1 -- 172.21.15.44:0/1931865089 learned_addr learned my addr 172.21.15.44:0/1931865089 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.357+0000 7f7b7a111640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b740d0a30 0x7f7b740cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.357+0000 7f7b79910640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b7406cf90 0x7f7b7414f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.358+0000 7f7b7910f640 1 -- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b7406cf90 msgr2=0x7f7b7414f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.358+0000 7f7b7910f640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b7406cf90 0x7f7b7414f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.358+0000 7f7b7910f640 1 -- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b740d0a30 msgr2=0x7f7b740cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.358+0000 7f7b7910f640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b740d0a30 0x7f7b740cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.358+0000 7f7b7910f640 1 -- 172.21.15.44:0/1931865089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b740b81b0 con 0x7f7b740cfe30 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.358+0000 7f7b7910f640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b740cfe30 0x7f7b740c84a0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009890 tx=0x7f7b680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.358+0000 7f7b727fc640 1 -- 172.21.15.44:0/1931865089 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b68032840 con 0x7f7b740cfe30 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.358+0000 7f7b727fc640 1 -- 172.21.15.44:0/1931865089 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b680329e0 con 0x7f7b740cfe30 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.358+0000 7f7b7bb74640 1 -- 172.21.15.44:0/1931865089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b740b8490 con 0x7f7b740cfe30 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.358+0000 7f7b727fc640 1 -- 172.21.15.44:0/1931865089 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b68032dd0 con 0x7f7b740cfe30 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.358+0000 7f7b7bb74640 1 -- 172.21.15.44:0/1931865089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b740b8a50 con 0x7f7b740cfe30 2024-04-02T13:29:15.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.359+0000 7f7b7bb74640 1 -- 172.21.15.44:0/1931865089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b740684f0 con 0x7f7b740cfe30 2024-04-02T13:29:15.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.359+0000 7f7b727fc640 1 -- 172.21.15.44:0/1931865089 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7b6803b3f0 con 0x7f7b740cfe30 2024-04-02T13:29:15.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.360+0000 7f7b727fc640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b6006c920 0x7f7b6006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.360+0000 7f7b727fc640 1 -- 172.21.15.44:0/1931865089 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f7b6802f030 con 0x7f7b740cfe30 2024-04-02T13:29:15.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.360+0000 7f7b79910640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b6006c920 0x7f7b6006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.361+0000 7f7b79910640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b6006c920 0x7f7b6006ede0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c007fd0 tx=0x7f7b6c007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:15.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.364+0000 7f7b727fc640 1 -- 172.21.15.44:0/1931865089 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7b68040070 con 0x7f7b740cfe30 2024-04-02T13:29:15.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.544+0000 7f7b7bb74640 1 -- 172.21.15.44:0/1931865089 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7b740d2ca0 con 0x7f7b6006c920 2024-04-02T13:29:15.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.551+0000 7f7b727fc640 1 -- 172.21.15.44:0/1931865089 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252483 (secure 0 0 0) 0x7f7b740d2ca0 con 0x7f7b6006c920 2024-04-02T13:29:15.558 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:15.560 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:29:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.554+0000 7f7b7bb74640 1 -- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b6006c920 msgr2=0x7f7b6006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.554+0000 7f7b7bb74640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b6006c920 0x7f7b6006ede0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c007fd0 tx=0x7f7b6c007de0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.554+0000 7f7b7bb74640 1 -- 172.21.15.44:0/1931865089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b740cfe30 msgr2=0x7f7b740c84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.554+0000 7f7b7bb74640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b740cfe30 0x7f7b740c84a0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009890 tx=0x7f7b680046c0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.554+0000 7f7b7bb74640 1 -- 172.21.15.44:0/1931865089 shutdown_connections 2024-04-02T13:29:15.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.554+0000 7f7b7bb74640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7b6006c920 0x7f7b6006ede0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.554+0000 7f7b7bb74640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b740d0a30 0x7f7b740cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.554+0000 7f7b7bb74640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b740cfe30 0x7f7b740c84a0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.554+0000 7f7b7bb74640 1 --2- 172.21.15.44:0/1931865089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b7406cf90 0x7f7b7414f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.554+0000 7f7b7bb74640 1 -- 172.21.15.44:0/1931865089 >> 172.21.15.44:0/1931865089 conn(0x7f7b7409d470 msgr2=0x7f7b740cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:15.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.554+0000 7f7b7bb74640 1 -- 172.21.15.44:0/1931865089 shutdown_connections 2024-04-02T13:29:15.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.554+0000 7f7b7bb74640 1 -- 172.21.15.44:0/1931865089 wait complete. 2024-04-02T13:29:15.567 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-04-02T13:29:15.008080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260984,"kb_used_data":11496,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713736,"statfs":{"total":773094113280,"available":772826865664,"internally_reserved":0,"allocated":11771904,"data_stored":6073156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.475726"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555405+0000","last_change":"2024-04-02T13:29:11.572158+0000","last_active":"2024-04-02T13:29:12.555405+0000","last_peered":"2024-04-02T13:29:12.555405+0000","last_clean":"2024-04-02T13:29:12.555405+0000","last_became_active":"2024-04-02T13:29:11.571759+0000","last_became_peered":"2024-04-02T13:29:11.571759+0000","last_unstale":"2024-04-02T13:29:12.555405+0000","last_undegraded":"2024-04-02T13:29:12.555405+0000","last_fullsized":"2024-04-02T13:29:12.555405+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:53:26.735220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655515+0000","last_change":"2024-04-02T13:29:11.570652+0000","last_active":"2024-04-02T13:29:12.655515+0000","last_peered":"2024-04-02T13:29:12.655515+0000","last_clean":"2024-04-02T13:29:12.655515+0000","last_became_active":"2024-04-02T13:29:11.570392+0000","last_became_peered":"2024-04-02T13:29:11.570392+0000","last_unstale":"2024-04-02T13:29:12.655515+0000","last_undegraded":"2024-04-02T13:29:12.655515+0000","last_fullsized":"2024-04-02T13:29:12.655515+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:05:16.048100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555167+0000","last_change":"2024-04-02T13:29:11.564487+0000","last_active":"2024-04-02T13:29:12.555167+0000","last_peered":"2024-04-02T13:29:12.555167+0000","last_clean":"2024-04-02T13:29:12.555167+0000","last_became_active":"2024-04-02T13:29:11.564194+0000","last_became_peered":"2024-04-02T13:29:11.564194+0000","last_unstale":"2024-04-02T13:29:12.555167+0000","last_undegraded":"2024-04-02T13:29:12.555167+0000","last_fullsized":"2024-04-02T13:29:12.555167+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:11:38.798164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552571+0000","last_change":"2024-04-02T13:29:11.570909+0000","last_active":"2024-04-02T13:29:12.552571+0000","last_peered":"2024-04-02T13:29:12.552571+0000","last_clean":"2024-04-02T13:29:12.552571+0000","last_became_active":"2024-04-02T13:29:11.570624+0000","last_became_peered":"2024-04-02T13:29:11.570624+0000","last_unstale":"2024-04-02T13:29:12.552571+0000","last_undegraded":"2024-04-02T13:29:12.552571+0000","last_fullsized":"2024-04-02T13:29:12.552571+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:00.080193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091574+0000","last_change":"2024-04-02T13:29:11.570175+0000","last_active":"2024-04-02T13:29:13.091574+0000","last_peered":"2024-04-02T13:29:13.091574+0000","last_clean":"2024-04-02T13:29:13.091574+0000","last_became_active":"2024-04-02T13:29:11.569345+0000","last_became_peered":"2024-04-02T13:29:11.569345+0000","last_unstale":"2024-04-02T13:29:13.091574+0000","last_undegraded":"2024-04-02T13:29:13.091574+0000","last_fullsized":"2024-04-02T13:29:13.091574+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:41.907332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655089+0000","last_change":"2024-04-02T13:29:11.567175+0000","last_active":"2024-04-02T13:29:12.655089+0000","last_peered":"2024-04-02T13:29:12.655089+0000","last_clean":"2024-04-02T13:29:12.655089+0000","last_became_active":"2024-04-02T13:29:11.566897+0000","last_became_peered":"2024-04-02T13:29:11.566897+0000","last_unstale":"2024-04-02T13:29:12.655089+0000","last_undegraded":"2024-04-02T13:29:12.655089+0000","last_fullsized":"2024-04-02T13:29:12.655089+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:07:12.001409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655016+0000","last_change":"2024-04-02T13:29:11.570937+0000","last_active":"2024-04-02T13:29:12.655016+0000","last_peered":"2024-04-02T13:29:12.655016+0000","last_clean":"2024-04-02T13:29:12.655016+0000","last_became_active":"2024-04-02T13:29:11.568973+0000","last_became_peered":"2024-04-02T13:29:11.568973+0000","last_unstale":"2024-04-02T13:29:12.655016+0000","last_undegraded":"2024-04-02T13:29:12.655016+0000","last_fullsized":"2024-04-02T13:29:12.655016+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:54:37.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091369+0000","last_change":"2024-04-02T13:29:11.571629+0000","last_active":"2024-04-02T13:29:13.091369+0000","last_peered":"2024-04-02T13:29:13.091369+0000","last_clean":"2024-04-02T13:29:13.091369+0000","last_became_active":"2024-04-02T13:29:11.571373+0000","last_became_peered":"2024-04-02T13:29:11.571373+0000","last_unstale":"2024-04-02T13:29:13.091369+0000","last_undegraded":"2024-04-02T13:29:13.091369+0000","last_fullsized":"2024-04-02T13:29:13.091369+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:52:19.518479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655489+0000","last_change":"2024-04-02T13:29:11.568280+0000","last_active":"2024-04-02T13:29:12.655489+0000","last_peered":"2024-04-02T13:29:12.655489+0000","last_clean":"2024-04-02T13:29:12.655489+0000","last_became_active":"2024-04-02T13:29:11.567211+0000","last_became_peered":"2024-04-02T13:29:11.567211+0000","last_unstale":"2024-04-02T13:29:12.655489+0000","last_undegraded":"2024-04-02T13:29:12.655489+0000","last_fullsized":"2024-04-02T13:29:12.655489+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:31.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091828+0000","last_change":"2024-04-02T13:29:11.563645+0000","last_active":"2024-04-02T13:29:13.091828+0000","last_peered":"2024-04-02T13:29:13.091828+0000","last_clean":"2024-04-02T13:29:13.091828+0000","last_became_active":"2024-04-02T13:29:11.563378+0000","last_became_peered":"2024-04-02T13:29:11.563378+0000","last_unstale":"2024-04-02T13:29:13.091828+0000","last_undegraded":"2024-04-02T13:29:13.091828+0000","last_fullsized":"2024-04-02T13:29:13.091828+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:52.672917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556913+0000","last_change":"2024-04-02T13:29:11.564144+0000","last_active":"2024-04-02T13:29:12.556913+0000","last_peered":"2024-04-02T13:29:12.556913+0000","last_clean":"2024-04-02T13:29:12.556913+0000","last_became_active":"2024-04-02T13:29:11.563863+0000","last_became_peered":"2024-04-02T13:29:11.563863+0000","last_unstale":"2024-04-02T13:29:12.556913+0000","last_undegraded":"2024-04-02T13:29:12.556913+0000","last_fullsized":"2024-04-02T13:29:12.556913+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:49.582030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092500+0000","last_change":"2024-04-02T13:29:11.572938+0000","last_active":"2024-04-02T13:29:13.092500+0000","last_peered":"2024-04-02T13:29:13.092500+0000","last_clean":"2024-04-02T13:29:13.092500+0000","last_became_active":"2024-04-02T13:29:11.571439+0000","last_became_peered":"2024-04-02T13:29:11.571439+0000","last_unstale":"2024-04-02T13:29:13.092500+0000","last_undegraded":"2024-04-02T13:29:13.092500+0000","last_fullsized":"2024-04-02T13:29:13.092500+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:20.988426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552385+0000","last_change":"2024-04-02T13:29:11.565274+0000","last_active":"2024-04-02T13:29:12.552385+0000","last_peered":"2024-04-02T13:29:12.552385+0000","last_clean":"2024-04-02T13:29:12.552385+0000","last_became_active":"2024-04-02T13:29:11.565025+0000","last_became_peered":"2024-04-02T13:29:11.565025+0000","last_unstale":"2024-04-02T13:29:12.552385+0000","last_undegraded":"2024-04-02T13:29:12.552385+0000","last_fullsized":"2024-04-02T13:29:12.552385+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:49:39.658389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091660+0000","last_change":"2024-04-02T13:29:11.562082+0000","last_active":"2024-04-02T13:29:13.091660+0000","last_peered":"2024-04-02T13:29:13.091660+0000","last_clean":"2024-04-02T13:29:13.091660+0000","last_became_active":"2024-04-02T13:29:11.561781+0000","last_became_peered":"2024-04-02T13:29:11.561781+0000","last_unstale":"2024-04-02T13:29:13.091660+0000","last_undegraded":"2024-04-02T13:29:13.091660+0000","last_fullsized":"2024-04-02T13:29:13.091660+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:29:24.686822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091910+0000","last_change":"2024-04-02T13:29:11.573376+0000","last_active":"2024-04-02T13:29:13.091910+0000","last_peered":"2024-04-02T13:29:13.091910+0000","last_clean":"2024-04-02T13:29:13.091910+0000","last_became_active":"2024-04-02T13:29:11.573084+0000","last_became_peered":"2024-04-02T13:29:11.573084+0000","last_unstale":"2024-04-02T13:29:13.091910+0000","last_undegraded":"2024-04-02T13:29:13.091910+0000","last_fullsized":"2024-04-02T13:29:13.091910+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:46:50.450645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091790+0000","last_change":"2024-04-02T13:29:11.563245+0000","last_active":"2024-04-02T13:29:13.091790+0000","last_peered":"2024-04-02T13:29:13.091790+0000","last_clean":"2024-04-02T13:29:13.091790+0000","last_became_active":"2024-04-02T13:29:11.562973+0000","last_became_peered":"2024-04-02T13:29:11.562973+0000","last_unstale":"2024-04-02T13:29:13.091790+0000","last_undegraded":"2024-04-02T13:29:13.091790+0000","last_fullsized":"2024-04-02T13:29:13.091790+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:00:05.121231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555513+0000","last_change":"2024-04-02T13:29:11.564633+0000","last_active":"2024-04-02T13:29:12.555513+0000","last_peered":"2024-04-02T13:29:12.555513+0000","last_clean":"2024-04-02T13:29:12.555513+0000","last_became_active":"2024-04-02T13:29:11.564360+0000","last_became_peered":"2024-04-02T13:29:11.564360+0000","last_unstale":"2024-04-02T13:29:12.555513+0000","last_undegraded":"2024-04-02T13:29:12.555513+0000","last_fullsized":"2024-04-02T13:29:12.555513+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:21.325009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655523+0000","last_change":"2024-04-02T13:29:11.569256+0000","last_active":"2024-04-02T13:29:12.655523+0000","last_peered":"2024-04-02T13:29:12.655523+0000","last_clean":"2024-04-02T13:29:12.655523+0000","last_became_active":"2024-04-02T13:29:11.568992+0000","last_became_peered":"2024-04-02T13:29:11.568992+0000","last_unstale":"2024-04-02T13:29:12.655523+0000","last_undegraded":"2024-04-02T13:29:12.655523+0000","last_fullsized":"2024-04-02T13:29:12.655523+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:27:20.320062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555354+0000","last_change":"2024-04-02T13:29:11.569227+0000","last_active":"2024-04-02T13:29:12.555354+0000","last_peered":"2024-04-02T13:29:12.555354+0000","last_clean":"2024-04-02T13:29:12.555354+0000","last_became_active":"2024-04-02T13:29:11.568966+0000","last_became_peered":"2024-04-02T13:29:11.568966+0000","last_unstale":"2024-04-02T13:29:12.555354+0000","last_undegraded":"2024-04-02T13:29:12.555354+0000","last_fullsized":"2024-04-02T13:29:12.555354+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:12:18.457537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092582+0000","last_change":"2024-04-02T13:29:11.566637+0000","last_active":"2024-04-02T13:29:13.092582+0000","last_peered":"2024-04-02T13:29:13.092582+0000","last_clean":"2024-04-02T13:29:13.092582+0000","last_became_active":"2024-04-02T13:29:11.565531+0000","last_became_peered":"2024-04-02T13:29:11.565531+0000","last_unstale":"2024-04-02T13:29:13.092582+0000","last_undegraded":"2024-04-02T13:29:13.092582+0000","last_fullsized":"2024-04-02T13:29:13.092582+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:31.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092189+0000","last_change":"2024-04-02T13:29:11.564849+0000","last_active":"2024-04-02T13:29:13.092189+0000","last_peered":"2024-04-02T13:29:13.092189+0000","last_clean":"2024-04-02T13:29:13.092189+0000","last_became_active":"2024-04-02T13:29:11.564597+0000","last_became_peered":"2024-04-02T13:29:11.564597+0000","last_unstale":"2024-04-02T13:29:13.092189+0000","last_undegraded":"2024-04-02T13:29:13.092189+0000","last_fullsized":"2024-04-02T13:29:13.092189+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:45:48.117521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091774+0000","last_change":"2024-04-02T13:29:11.556789+0000","last_active":"2024-04-02T13:29:13.091774+0000","last_peered":"2024-04-02T13:29:13.091774+0000","last_clean":"2024-04-02T13:29:13.091774+0000","last_became_active":"2024-04-02T13:29:11.556490+0000","last_became_peered":"2024-04-02T13:29:11.556490+0000","last_unstale":"2024-04-02T13:29:13.091774+0000","last_undegraded":"2024-04-02T13:29:13.091774+0000","last_fullsized":"2024-04-02T13:29:13.091774+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:07:12.001409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555172+0000","last_change":"2024-04-02T13:29:11.557712+0000","last_active":"2024-04-02T13:29:12.555172+0000","last_peered":"2024-04-02T13:29:12.555172+0000","last_clean":"2024-04-02T13:29:12.555172+0000","last_became_active":"2024-04-02T13:29:11.557443+0000","last_became_peered":"2024-04-02T13:29:11.557443+0000","last_unstale":"2024-04-02T13:29:12.555172+0000","last_undegraded":"2024-04-02T13:29:12.555172+0000","last_fullsized":"2024-04-02T13:29:12.555172+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:16:10.761598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091377+0000","last_change":"2024-04-02T13:29:11.556553+0000","last_active":"2024-04-02T13:29:13.091377+0000","last_peered":"2024-04-02T13:29:13.091377+0000","last_clean":"2024-04-02T13:29:13.091377+0000","last_became_active":"2024-04-02T13:29:11.556284+0000","last_became_peered":"2024-04-02T13:29:11.556284+0000","last_unstale":"2024-04-02T13:29:13.091377+0000","last_undegraded":"2024-04-02T13:29:13.091377+0000","last_fullsized":"2024-04-02T13:29:13.091377+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:53:56.947267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092287+0000","last_change":"2024-04-02T13:29:11.575767+0000","last_active":"2024-04-02T13:29:13.092287+0000","last_peered":"2024-04-02T13:29:13.092287+0000","last_clean":"2024-04-02T13:29:13.092287+0000","last_became_active":"2024-04-02T13:29:11.575491+0000","last_became_peered":"2024-04-02T13:29:11.575491+0000","last_unstale":"2024-04-02T13:29:13.092287+0000","last_undegraded":"2024-04-02T13:29:13.092287+0000","last_fullsized":"2024-04-02T13:29:13.092287+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:07:12.001409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092329+0000","last_change":"2024-04-02T13:29:11.568347+0000","last_active":"2024-04-02T13:29:13.092329+0000","last_peered":"2024-04-02T13:29:13.092329+0000","last_clean":"2024-04-02T13:29:13.092329+0000","last_became_active":"2024-04-02T13:29:11.568030+0000","last_became_peered":"2024-04-02T13:29:11.568030+0000","last_unstale":"2024-04-02T13:29:13.092329+0000","last_undegraded":"2024-04-02T13:29:13.092329+0000","last_fullsized":"2024-04-02T13:29:13.092329+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:49.582030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556914+0000","last_change":"2024-04-02T13:29:11.564763+0000","last_active":"2024-04-02T13:29:12.556914+0000","last_peered":"2024-04-02T13:29:12.556914+0000","last_clean":"2024-04-02T13:29:12.556914+0000","last_became_active":"2024-04-02T13:29:11.562744+0000","last_became_peered":"2024-04-02T13:29:11.562744+0000","last_unstale":"2024-04-02T13:29:12.556914+0000","last_undegraded":"2024-04-02T13:29:12.556914+0000","last_fullsized":"2024-04-02T13:29:12.556914+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:20.988426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655422+0000","last_change":"2024-04-02T13:29:11.568275+0000","last_active":"2024-04-02T13:29:12.655422+0000","last_peered":"2024-04-02T13:29:12.655422+0000","last_clean":"2024-04-02T13:29:12.655422+0000","last_became_active":"2024-04-02T13:29:11.567148+0000","last_became_peered":"2024-04-02T13:29:11.567148+0000","last_unstale":"2024-04-02T13:29:12.655422+0000","last_undegraded":"2024-04-02T13:29:12.655422+0000","last_fullsized":"2024-04-02T13:29:12.655422+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:20.988426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092232+0000","last_change":"2024-04-02T13:29:11.561927+0000","last_active":"2024-04-02T13:29:13.092232+0000","last_peered":"2024-04-02T13:29:13.092232+0000","last_clean":"2024-04-02T13:29:13.092232+0000","last_became_active":"2024-04-02T13:29:11.561641+0000","last_became_peered":"2024-04-02T13:29:11.561641+0000","last_unstale":"2024-04-02T13:29:13.092232+0000","last_undegraded":"2024-04-02T13:29:13.092232+0000","last_fullsized":"2024-04-02T13:29:13.092232+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:18:45.068600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092079+0000","last_change":"2024-04-02T13:29:11.575792+0000","last_active":"2024-04-02T13:29:13.092079+0000","last_peered":"2024-04-02T13:29:13.092079+0000","last_clean":"2024-04-02T13:29:13.092079+0000","last_became_active":"2024-04-02T13:29:11.575527+0000","last_became_peered":"2024-04-02T13:29:11.575527+0000","last_unstale":"2024-04-02T13:29:13.092079+0000","last_undegraded":"2024-04-02T13:29:13.092079+0000","last_fullsized":"2024-04-02T13:29:13.092079+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:54:37.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655137+0000","last_change":"2024-04-02T13:29:11.571601+0000","last_active":"2024-04-02T13:29:12.655137+0000","last_peered":"2024-04-02T13:29:12.655137+0000","last_clean":"2024-04-02T13:29:12.655137+0000","last_became_active":"2024-04-02T13:29:11.571323+0000","last_became_peered":"2024-04-02T13:29:11.571323+0000","last_unstale":"2024-04-02T13:29:12.655137+0000","last_undegraded":"2024-04-02T13:29:12.655137+0000","last_fullsized":"2024-04-02T13:29:12.655137+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:52.672917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091386+0000","last_change":"2024-04-02T13:29:11.572161+0000","last_active":"2024-04-02T13:29:13.091386+0000","last_peered":"2024-04-02T13:29:13.091386+0000","last_clean":"2024-04-02T13:29:13.091386+0000","last_became_active":"2024-04-02T13:29:11.571918+0000","last_became_peered":"2024-04-02T13:29:11.571918+0000","last_unstale":"2024-04-02T13:29:13.091386+0000","last_undegraded":"2024-04-02T13:29:13.091386+0000","last_fullsized":"2024-04-02T13:29:13.091386+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:43:49.496031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555267+0000","last_change":"2024-04-02T13:29:11.566629+0000","last_active":"2024-04-02T13:29:12.555267+0000","last_peered":"2024-04-02T13:29:12.555267+0000","last_clean":"2024-04-02T13:29:12.555267+0000","last_became_active":"2024-04-02T13:29:11.566278+0000","last_became_peered":"2024-04-02T13:29:11.566278+0000","last_unstale":"2024-04-02T13:29:12.555267+0000","last_undegraded":"2024-04-02T13:29:12.555267+0000","last_fullsized":"2024-04-02T13:29:12.555267+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:17:21.218582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"105'4","reported_seq":18,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552702+0000","last_change":"2024-04-02T13:29:10.551574+0000","last_active":"2024-04-02T13:29:12.552702+0000","last_peered":"2024-04-02T13:29:12.552702+0000","last_clean":"2024-04-02T13:29:12.552702+0000","last_became_active":"2024-04-02T13:29:10.551292+0000","last_became_peered":"2024-04-02T13:29:10.551292+0000","last_unstale":"2024-04-02T13:29:12.552702+0000","last_undegraded":"2024-04-02T13:29:12.552702+0000","last_fullsized":"2024-04-02T13:29:12.552702+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:12:46.096389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092003+0000","last_change":"2024-04-02T13:29:11.571460+0000","last_active":"2024-04-02T13:29:13.092003+0000","last_peered":"2024-04-02T13:29:13.092003+0000","last_clean":"2024-04-02T13:29:13.092003+0000","last_became_active":"2024-04-02T13:29:11.571125+0000","last_became_peered":"2024-04-02T13:29:11.571125+0000","last_unstale":"2024-04-02T13:29:13.092003+0000","last_undegraded":"2024-04-02T13:29:13.092003+0000","last_fullsized":"2024-04-02T13:29:13.092003+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:46:59.213999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092360+0000","last_change":"2024-04-02T13:29:10.553591+0000","last_active":"2024-04-02T13:29:13.092360+0000","last_peered":"2024-04-02T13:29:13.092360+0000","last_clean":"2024-04-02T13:29:13.092360+0000","last_became_active":"2024-04-02T13:29:10.553297+0000","last_became_peered":"2024-04-02T13:29:10.553297+0000","last_unstale":"2024-04-02T13:29:13.092360+0000","last_undegraded":"2024-04-02T13:29:13.092360+0000","last_fullsized":"2024-04-02T13:29:13.092360+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:53.392833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092300+0000","last_change":"2024-04-02T13:29:11.570664+0000","last_active":"2024-04-02T13:29:13.092300+0000","last_peered":"2024-04-02T13:29:13.092300+0000","last_clean":"2024-04-02T13:29:13.092300+0000","last_became_active":"2024-04-02T13:29:11.570377+0000","last_became_peered":"2024-04-02T13:29:11.570377+0000","last_unstale":"2024-04-02T13:29:13.092300+0000","last_undegraded":"2024-04-02T13:29:13.092300+0000","last_fullsized":"2024-04-02T13:29:13.092300+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:52.672917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092177+0000","last_change":"2024-04-02T13:29:10.561945+0000","last_active":"2024-04-02T13:29:13.092177+0000","last_peered":"2024-04-02T13:29:13.092177+0000","last_clean":"2024-04-02T13:29:13.092177+0000","last_became_active":"2024-04-02T13:29:10.561668+0000","last_became_peered":"2024-04-02T13:29:10.561668+0000","last_unstale":"2024-04-02T13:29:13.092177+0000","last_undegraded":"2024-04-02T13:29:13.092177+0000","last_fullsized":"2024-04-02T13:29:13.092177+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:34:04.714892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556734+0000","last_change":"2024-04-02T13:29:11.566493+0000","last_active":"2024-04-02T13:29:12.556734+0000","last_peered":"2024-04-02T13:29:12.556734+0000","last_clean":"2024-04-02T13:29:12.556734+0000","last_became_active":"2024-04-02T13:29:11.566087+0000","last_became_peered":"2024-04-02T13:29:11.566087+0000","last_unstale":"2024-04-02T13:29:12.556734+0000","last_undegraded":"2024-04-02T13:29:12.556734+0000","last_fullsized":"2024-04-02T13:29:12.556734+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:52.672917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092018+0000","last_change":"2024-04-02T13:29:10.554736+0000","last_active":"2024-04-02T13:29:13.092018+0000","last_peered":"2024-04-02T13:29:13.092018+0000","last_clean":"2024-04-02T13:29:13.092018+0000","last_became_active":"2024-04-02T13:29:10.554153+0000","last_became_peered":"2024-04-02T13:29:10.554153+0000","last_unstale":"2024-04-02T13:29:13.092018+0000","last_undegraded":"2024-04-02T13:29:13.092018+0000","last_fullsized":"2024-04-02T13:29:13.092018+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:15:26.524288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552637+0000","last_change":"2024-04-02T13:29:11.568599+0000","last_active":"2024-04-02T13:29:12.552637+0000","last_peered":"2024-04-02T13:29:12.552637+0000","last_clean":"2024-04-02T13:29:12.552637+0000","last_became_active":"2024-04-02T13:29:11.568293+0000","last_became_peered":"2024-04-02T13:29:11.568293+0000","last_unstale":"2024-04-02T13:29:12.552637+0000","last_undegraded":"2024-04-02T13:29:12.552637+0000","last_fullsized":"2024-04-02T13:29:12.552637+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:59:41.813799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091980+0000","last_change":"2024-04-02T13:29:10.549037+0000","last_active":"2024-04-02T13:29:13.091980+0000","last_peered":"2024-04-02T13:29:13.091980+0000","last_clean":"2024-04-02T13:29:13.091980+0000","last_became_active":"2024-04-02T13:29:10.548744+0000","last_became_peered":"2024-04-02T13:29:10.548744+0000","last_unstale":"2024-04-02T13:29:13.091980+0000","last_undegraded":"2024-04-02T13:29:13.091980+0000","last_fullsized":"2024-04-02T13:29:13.091980+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:07:10.997075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555463+0000","last_change":"2024-04-02T13:29:11.567262+0000","last_active":"2024-04-02T13:29:12.555463+0000","last_peered":"2024-04-02T13:29:12.555463+0000","last_clean":"2024-04-02T13:29:12.555463+0000","last_became_active":"2024-04-02T13:29:11.567010+0000","last_became_peered":"2024-04-02T13:29:11.567010+0000","last_unstale":"2024-04-02T13:29:12.555463+0000","last_undegraded":"2024-04-02T13:29:12.555463+0000","last_fullsized":"2024-04-02T13:29:12.555463+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:10:28.246157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556550+0000","last_change":"2024-04-02T13:29:10.556750+0000","last_active":"2024-04-02T13:29:12.556550+0000","last_peered":"2024-04-02T13:29:12.556550+0000","last_clean":"2024-04-02T13:29:12.556550+0000","last_became_active":"2024-04-02T13:29:10.556478+0000","last_became_peered":"2024-04-02T13:29:10.556478+0000","last_unstale":"2024-04-02T13:29:12.556550+0000","last_undegraded":"2024-04-02T13:29:12.556550+0000","last_fullsized":"2024-04-02T13:29:12.556550+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:48:36.435520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655606+0000","last_change":"2024-04-02T13:29:11.569527+0000","last_active":"2024-04-02T13:29:12.655606+0000","last_peered":"2024-04-02T13:29:12.655606+0000","last_clean":"2024-04-02T13:29:12.655606+0000","last_became_active":"2024-04-02T13:29:11.568434+0000","last_became_peered":"2024-04-02T13:29:11.568434+0000","last_unstale":"2024-04-02T13:29:12.655606+0000","last_undegraded":"2024-04-02T13:29:12.655606+0000","last_fullsized":"2024-04-02T13:29:12.655606+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:29:24.686822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091959+0000","last_change":"2024-04-02T13:29:10.556247+0000","last_active":"2024-04-02T13:29:13.091959+0000","last_peered":"2024-04-02T13:29:13.091959+0000","last_clean":"2024-04-02T13:29:13.091959+0000","last_became_active":"2024-04-02T13:29:10.554542+0000","last_became_peered":"2024-04-02T13:29:10.554542+0000","last_unstale":"2024-04-02T13:29:13.091959+0000","last_undegraded":"2024-04-02T13:29:13.091959+0000","last_fullsized":"2024-04-02T13:29:13.091959+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:13:59.952423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552694+0000","last_change":"2024-04-02T13:29:11.563227+0000","last_active":"2024-04-02T13:29:12.552694+0000","last_peered":"2024-04-02T13:29:12.552694+0000","last_clean":"2024-04-02T13:29:12.552694+0000","last_became_active":"2024-04-02T13:29:11.562915+0000","last_became_peered":"2024-04-02T13:29:11.562915+0000","last_unstale":"2024-04-02T13:29:12.552694+0000","last_undegraded":"2024-04-02T13:29:12.552694+0000","last_fullsized":"2024-04-02T13:29:12.552694+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:23:55.267998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552305+0000","last_change":"2024-04-02T13:29:10.556556+0000","last_active":"2024-04-02T13:29:12.552305+0000","last_peered":"2024-04-02T13:29:12.552305+0000","last_clean":"2024-04-02T13:29:12.552305+0000","last_became_active":"2024-04-02T13:29:10.556240+0000","last_became_peered":"2024-04-02T13:29:10.556240+0000","last_unstale":"2024-04-02T13:29:12.552305+0000","last_undegraded":"2024-04-02T13:29:12.552305+0000","last_fullsized":"2024-04-02T13:29:12.552305+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:30.633172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555151+0000","last_change":"2024-04-02T13:29:11.570296+0000","last_active":"2024-04-02T13:29:12.555151+0000","last_peered":"2024-04-02T13:29:12.555151+0000","last_clean":"2024-04-02T13:29:12.555151+0000","last_became_active":"2024-04-02T13:29:11.569844+0000","last_became_peered":"2024-04-02T13:29:11.569844+0000","last_unstale":"2024-04-02T13:29:12.555151+0000","last_undegraded":"2024-04-02T13:29:12.555151+0000","last_fullsized":"2024-04-02T13:29:12.555151+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:06:15.210019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092376+0000","last_change":"2024-04-02T13:29:10.562040+0000","last_active":"2024-04-02T13:29:13.092376+0000","last_peered":"2024-04-02T13:29:13.092376+0000","last_clean":"2024-04-02T13:29:13.092376+0000","last_became_active":"2024-04-02T13:29:10.561482+0000","last_became_peered":"2024-04-02T13:29:10.561482+0000","last_unstale":"2024-04-02T13:29:13.092376+0000","last_undegraded":"2024-04-02T13:29:13.092376+0000","last_fullsized":"2024-04-02T13:29:13.092376+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:48:36.435520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091569+0000","last_change":"2024-04-02T13:29:11.566957+0000","last_active":"2024-04-02T13:29:13.091569+0000","last_peered":"2024-04-02T13:29:13.091569+0000","last_clean":"2024-04-02T13:29:13.091569+0000","last_became_active":"2024-04-02T13:29:11.566670+0000","last_became_peered":"2024-04-02T13:29:11.566670+0000","last_unstale":"2024-04-02T13:29:13.091569+0000","last_undegraded":"2024-04-02T13:29:13.091569+0000","last_fullsized":"2024-04-02T13:29:13.091569+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:48:49.409715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555149+0000","last_change":"2024-04-02T13:29:10.560147+0000","last_active":"2024-04-02T13:29:12.555149+0000","last_peered":"2024-04-02T13:29:12.555149+0000","last_clean":"2024-04-02T13:29:12.555149+0000","last_became_active":"2024-04-02T13:29:10.559595+0000","last_became_peered":"2024-04-02T13:29:10.559595+0000","last_unstale":"2024-04-02T13:29:12.555149+0000","last_undegraded":"2024-04-02T13:29:12.555149+0000","last_fullsized":"2024-04-02T13:29:12.555149+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:26:29.859886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091780+0000","last_change":"2024-04-02T13:29:11.563658+0000","last_active":"2024-04-02T13:29:13.091780+0000","last_peered":"2024-04-02T13:29:13.091780+0000","last_clean":"2024-04-02T13:29:13.091780+0000","last_became_active":"2024-04-02T13:29:11.562304+0000","last_became_peered":"2024-04-02T13:29:11.562304+0000","last_unstale":"2024-04-02T13:29:13.091780+0000","last_undegraded":"2024-04-02T13:29:13.091780+0000","last_fullsized":"2024-04-02T13:29:13.091780+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:04:05.591117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"105'3","reported_seq":17,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091797+0000","last_change":"2024-04-02T13:29:10.559984+0000","last_active":"2024-04-02T13:29:13.091797+0000","last_peered":"2024-04-02T13:29:13.091797+0000","last_clean":"2024-04-02T13:29:13.091797+0000","last_became_active":"2024-04-02T13:29:10.559665+0000","last_became_peered":"2024-04-02T13:29:10.559665+0000","last_unstale":"2024-04-02T13:29:13.091797+0000","last_undegraded":"2024-04-02T13:29:13.091797+0000","last_fullsized":"2024-04-02T13:29:13.091797+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:51.668583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091890+0000","last_change":"2024-04-02T13:29:11.567784+0000","last_active":"2024-04-02T13:29:13.091890+0000","last_peered":"2024-04-02T13:29:13.091890+0000","last_clean":"2024-04-02T13:29:13.091890+0000","last_became_active":"2024-04-02T13:29:11.567533+0000","last_became_peered":"2024-04-02T13:29:11.567533+0000","last_unstale":"2024-04-02T13:29:13.091890+0000","last_undegraded":"2024-04-02T13:29:13.091890+0000","last_fullsized":"2024-04-02T13:29:13.091890+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:49.582030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"105'6","reported_seq":20,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655370+0000","last_change":"2024-04-02T13:29:10.560138+0000","last_active":"2024-04-02T13:29:12.655370+0000","last_peered":"2024-04-02T13:29:12.655370+0000","last_clean":"2024-04-02T13:29:12.655370+0000","last_became_active":"2024-04-02T13:29:10.559858+0000","last_became_peered":"2024-04-02T13:29:10.559858+0000","last_unstale":"2024-04-02T13:29:12.655370+0000","last_undegraded":"2024-04-02T13:29:12.655370+0000","last_fullsized":"2024-04-02T13:29:12.655370+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:34:04.714892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556625+0000","last_change":"2024-04-02T13:29:11.562690+0000","last_active":"2024-04-02T13:29:12.556625+0000","last_peered":"2024-04-02T13:29:12.556625+0000","last_clean":"2024-04-02T13:29:12.556625+0000","last_became_active":"2024-04-02T13:29:11.562359+0000","last_became_peered":"2024-04-02T13:29:11.562359+0000","last_unstale":"2024-04-02T13:29:12.556625+0000","last_undegraded":"2024-04-02T13:29:12.556625+0000","last_fullsized":"2024-04-02T13:29:12.556625+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:54:37.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091889+0000","last_change":"2024-04-02T13:29:10.555092+0000","last_active":"2024-04-02T13:29:13.091889+0000","last_peered":"2024-04-02T13:29:13.091889+0000","last_clean":"2024-04-02T13:29:13.091889+0000","last_became_active":"2024-04-02T13:29:10.554536+0000","last_became_peered":"2024-04-02T13:29:10.554536+0000","last_unstale":"2024-04-02T13:29:13.091889+0000","last_undegraded":"2024-04-02T13:29:13.091889+0000","last_fullsized":"2024-04-02T13:29:13.091889+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:48:36.435520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655485+0000","last_change":"2024-04-02T13:29:11.566424+0000","last_active":"2024-04-02T13:29:12.655485+0000","last_peered":"2024-04-02T13:29:12.655485+0000","last_clean":"2024-04-02T13:29:12.655485+0000","last_became_active":"2024-04-02T13:29:11.566119+0000","last_became_peered":"2024-04-02T13:29:11.566119+0000","last_unstale":"2024-04-02T13:29:12.655485+0000","last_undegraded":"2024-04-02T13:29:12.655485+0000","last_fullsized":"2024-04-02T13:29:12.655485+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:12:47.100724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552386+0000","last_change":"2024-04-02T13:29:10.549688+0000","last_active":"2024-04-02T13:29:12.552386+0000","last_peered":"2024-04-02T13:29:12.552386+0000","last_clean":"2024-04-02T13:29:12.552386+0000","last_became_active":"2024-04-02T13:29:10.549214+0000","last_became_peered":"2024-04-02T13:29:10.549214+0000","last_unstale":"2024-04-02T13:29:12.552386+0000","last_undegraded":"2024-04-02T13:29:12.552386+0000","last_fullsized":"2024-04-02T13:29:12.552386+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:19.984092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092207+0000","last_change":"2024-04-02T13:29:11.572314+0000","last_active":"2024-04-02T13:29:13.092207+0000","last_peered":"2024-04-02T13:29:13.092207+0000","last_clean":"2024-04-02T13:29:13.092207+0000","last_became_active":"2024-04-02T13:29:11.572069+0000","last_became_peered":"2024-04-02T13:29:11.572069+0000","last_unstale":"2024-04-02T13:29:13.092207+0000","last_undegraded":"2024-04-02T13:29:13.092207+0000","last_fullsized":"2024-04-02T13:29:13.092207+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:23:59.604573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092029+0000","last_change":"2024-04-02T13:29:10.556788+0000","last_active":"2024-04-02T13:29:13.092029+0000","last_peered":"2024-04-02T13:29:13.092029+0000","last_clean":"2024-04-02T13:29:13.092029+0000","last_became_active":"2024-04-02T13:29:10.556482+0000","last_became_peered":"2024-04-02T13:29:10.556482+0000","last_unstale":"2024-04-02T13:29:13.092029+0000","last_undegraded":"2024-04-02T13:29:13.092029+0000","last_fullsized":"2024-04-02T13:29:13.092029+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:55.085324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555499+0000","last_change":"2024-04-02T13:29:11.565050+0000","last_active":"2024-04-02T13:29:12.555499+0000","last_peered":"2024-04-02T13:29:12.555499+0000","last_clean":"2024-04-02T13:29:12.555499+0000","last_became_active":"2024-04-02T13:29:11.564781+0000","last_became_peered":"2024-04-02T13:29:11.564781+0000","last_unstale":"2024-04-02T13:29:12.555499+0000","last_undegraded":"2024-04-02T13:29:12.555499+0000","last_fullsized":"2024-04-02T13:29:12.555499+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:55:08.573393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"105'8","reported_seq":22,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092012+0000","last_change":"2024-04-02T13:29:10.561888+0000","last_active":"2024-04-02T13:29:13.092012+0000","last_peered":"2024-04-02T13:29:13.092012+0000","last_clean":"2024-04-02T13:29:13.092012+0000","last_became_active":"2024-04-02T13:29:10.561599+0000","last_became_peered":"2024-04-02T13:29:10.561599+0000","last_unstale":"2024-04-02T13:29:13.092012+0000","last_undegraded":"2024-04-02T13:29:13.092012+0000","last_fullsized":"2024-04-02T13:29:13.092012+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:17:07.568385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091748+0000","last_change":"2024-04-02T13:29:11.562382+0000","last_active":"2024-04-02T13:29:13.091748+0000","last_peered":"2024-04-02T13:29:13.091748+0000","last_clean":"2024-04-02T13:29:13.091748+0000","last_became_active":"2024-04-02T13:29:11.562133+0000","last_became_peered":"2024-04-02T13:29:11.562133+0000","last_unstale":"2024-04-02T13:29:13.091748+0000","last_undegraded":"2024-04-02T13:29:13.091748+0000","last_fullsized":"2024-04-02T13:29:13.091748+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:13:08.050156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655413+0000","last_change":"2024-04-02T13:29:10.555844+0000","last_active":"2024-04-02T13:29:12.655413+0000","last_peered":"2024-04-02T13:29:12.655413+0000","last_clean":"2024-04-02T13:29:12.655413+0000","last_became_active":"2024-04-02T13:29:10.555491+0000","last_became_peered":"2024-04-02T13:29:10.555491+0000","last_unstale":"2024-04-02T13:29:12.655413+0000","last_undegraded":"2024-04-02T13:29:12.655413+0000","last_fullsized":"2024-04-02T13:29:12.655413+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:11.502526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655261+0000","last_change":"2024-04-02T13:29:11.572559+0000","last_active":"2024-04-02T13:29:12.655261+0000","last_peered":"2024-04-02T13:29:12.655261+0000","last_clean":"2024-04-02T13:29:12.655261+0000","last_became_active":"2024-04-02T13:29:11.572297+0000","last_became_peered":"2024-04-02T13:29:11.572297+0000","last_unstale":"2024-04-02T13:29:12.655261+0000","last_undegraded":"2024-04-02T13:29:12.655261+0000","last_fullsized":"2024-04-02T13:29:12.655261+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:49.582030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091843+0000","last_change":"2024-04-02T13:29:10.555120+0000","last_active":"2024-04-02T13:29:13.091843+0000","last_peered":"2024-04-02T13:29:13.091843+0000","last_clean":"2024-04-02T13:29:13.091843+0000","last_became_active":"2024-04-02T13:29:10.554785+0000","last_became_peered":"2024-04-02T13:29:10.554785+0000","last_unstale":"2024-04-02T13:29:13.091843+0000","last_undegraded":"2024-04-02T13:29:13.091843+0000","last_fullsized":"2024-04-02T13:29:13.091843+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:48.577696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091733+0000","last_change":"2024-04-02T13:29:11.560685+0000","last_active":"2024-04-02T13:29:13.091733+0000","last_peered":"2024-04-02T13:29:13.091733+0000","last_clean":"2024-04-02T13:29:13.091733+0000","last_became_active":"2024-04-02T13:29:11.560084+0000","last_became_peered":"2024-04-02T13:29:11.560084+0000","last_unstale":"2024-04-02T13:29:13.091733+0000","last_undegraded":"2024-04-02T13:29:13.091733+0000","last_fullsized":"2024-04-02T13:29:13.091733+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:54:37.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556610+0000","last_change":"2024-04-02T13:29:10.552003+0000","last_active":"2024-04-02T13:29:12.556610+0000","last_peered":"2024-04-02T13:29:12.556610+0000","last_clean":"2024-04-02T13:29:12.556610+0000","last_became_active":"2024-04-02T13:29:10.551700+0000","last_became_peered":"2024-04-02T13:29:10.551700+0000","last_unstale":"2024-04-02T13:29:12.556610+0000","last_undegraded":"2024-04-02T13:29:12.556610+0000","last_fullsized":"2024-04-02T13:29:12.556610+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:13:59.952423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555237+0000","last_change":"2024-04-02T13:29:11.562698+0000","last_active":"2024-04-02T13:29:12.555237+0000","last_peered":"2024-04-02T13:29:12.555237+0000","last_clean":"2024-04-02T13:29:12.555237+0000","last_became_active":"2024-04-02T13:29:11.562343+0000","last_became_peered":"2024-04-02T13:29:11.562343+0000","last_unstale":"2024-04-02T13:29:12.555237+0000","last_undegraded":"2024-04-02T13:29:12.555237+0000","last_fullsized":"2024-04-02T13:29:12.555237+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:17.848752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092300+0000","last_change":"2024-04-02T13:29:10.560067+0000","last_active":"2024-04-02T13:29:13.092300+0000","last_peered":"2024-04-02T13:29:13.092300+0000","last_clean":"2024-04-02T13:29:13.092300+0000","last_became_active":"2024-04-02T13:29:10.559665+0000","last_became_peered":"2024-04-02T13:29:10.559665+0000","last_unstale":"2024-04-02T13:29:13.092300+0000","last_undegraded":"2024-04-02T13:29:13.092300+0000","last_fullsized":"2024-04-02T13:29:13.092300+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:04.764639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555399+0000","last_change":"2024-04-02T13:29:11.558019+0000","last_active":"2024-04-02T13:29:12.555399+0000","last_peered":"2024-04-02T13:29:12.555399+0000","last_clean":"2024-04-02T13:29:12.555399+0000","last_became_active":"2024-04-02T13:29:11.557706+0000","last_became_peered":"2024-04-02T13:29:11.557706+0000","last_unstale":"2024-04-02T13:29:12.555399+0000","last_undegraded":"2024-04-02T13:29:12.555399+0000","last_fullsized":"2024-04-02T13:29:12.555399+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:54:54.418033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091622+0000","last_change":"2024-04-02T13:29:10.550724+0000","last_active":"2024-04-02T13:29:13.091622+0000","last_peered":"2024-04-02T13:29:13.091622+0000","last_clean":"2024-04-02T13:29:13.091622+0000","last_became_active":"2024-04-02T13:29:10.550452+0000","last_became_peered":"2024-04-02T13:29:10.550452+0000","last_unstale":"2024-04-02T13:29:13.091622+0000","last_undegraded":"2024-04-02T13:29:13.091622+0000","last_fullsized":"2024-04-02T13:29:13.091622+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:38:39.890323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552628+0000","last_change":"2024-04-02T13:29:11.569238+0000","last_active":"2024-04-02T13:29:12.552628+0000","last_peered":"2024-04-02T13:29:12.552628+0000","last_clean":"2024-04-02T13:29:12.552628+0000","last_became_active":"2024-04-02T13:29:11.568966+0000","last_became_peered":"2024-04-02T13:29:11.568966+0000","last_unstale":"2024-04-02T13:29:12.552628+0000","last_undegraded":"2024-04-02T13:29:12.552628+0000","last_fullsized":"2024-04-02T13:29:12.552628+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:26:30.864220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655168+0000","last_change":"2024-04-02T13:29:10.549120+0000","last_active":"2024-04-02T13:29:12.655168+0000","last_peered":"2024-04-02T13:29:12.655168+0000","last_clean":"2024-04-02T13:29:12.655168+0000","last_became_active":"2024-04-02T13:29:10.548847+0000","last_became_peered":"2024-04-02T13:29:10.548847+0000","last_unstale":"2024-04-02T13:29:12.655168+0000","last_undegraded":"2024-04-02T13:29:12.655168+0000","last_fullsized":"2024-04-02T13:29:12.655168+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:06:00.732939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092285+0000","last_change":"2024-04-02T13:29:11.569888+0000","last_active":"2024-04-02T13:29:13.092285+0000","last_peered":"2024-04-02T13:29:13.092285+0000","last_clean":"2024-04-02T13:29:13.092285+0000","last_became_active":"2024-04-02T13:29:11.569621+0000","last_became_peered":"2024-04-02T13:29:11.569621+0000","last_unstale":"2024-04-02T13:29:13.092285+0000","last_undegraded":"2024-04-02T13:29:13.092285+0000","last_fullsized":"2024-04-02T13:29:13.092285+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:57.961338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091507+0000","last_change":"2024-04-02T13:29:10.552735+0000","last_active":"2024-04-02T13:29:13.091507+0000","last_peered":"2024-04-02T13:29:13.091507+0000","last_clean":"2024-04-02T13:29:13.091507+0000","last_became_active":"2024-04-02T13:29:10.552465+0000","last_became_peered":"2024-04-02T13:29:10.552465+0000","last_unstale":"2024-04-02T13:29:13.091507+0000","last_undegraded":"2024-04-02T13:29:13.091507+0000","last_fullsized":"2024-04-02T13:29:13.091507+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:20.296695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555178+0000","last_change":"2024-04-02T13:29:11.560581+0000","last_active":"2024-04-02T13:29:12.555178+0000","last_peered":"2024-04-02T13:29:12.555178+0000","last_clean":"2024-04-02T13:29:12.555178+0000","last_became_active":"2024-04-02T13:29:11.560299+0000","last_became_peered":"2024-04-02T13:29:11.560299+0000","last_unstale":"2024-04-02T13:29:12.555178+0000","last_undegraded":"2024-04-02T13:29:12.555178+0000","last_fullsized":"2024-04-02T13:29:12.555178+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:31:57.351494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091760+0000","last_change":"2024-04-02T13:29:10.557197+0000","last_active":"2024-04-02T13:29:13.091760+0000","last_peered":"2024-04-02T13:29:13.091760+0000","last_clean":"2024-04-02T13:29:13.091760+0000","last_became_active":"2024-04-02T13:29:10.554163+0000","last_became_peered":"2024-04-02T13:29:10.554163+0000","last_unstale":"2024-04-02T13:29:13.091760+0000","last_undegraded":"2024-04-02T13:29:13.091760+0000","last_fullsized":"2024-04-02T13:29:13.091760+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:34:04.714892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091794+0000","last_change":"2024-04-02T13:29:11.568696+0000","last_active":"2024-04-02T13:29:13.091794+0000","last_peered":"2024-04-02T13:29:13.091794+0000","last_clean":"2024-04-02T13:29:13.091794+0000","last_became_active":"2024-04-02T13:29:11.567935+0000","last_became_peered":"2024-04-02T13:29:11.567935+0000","last_unstale":"2024-04-02T13:29:13.091794+0000","last_undegraded":"2024-04-02T13:29:13.091794+0000","last_fullsized":"2024-04-02T13:29:13.091794+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:41.907332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"106'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.577646+0000","last_change":"2024-04-02T13:29:10.559371+0000","last_active":"2024-04-02T13:29:12.577646+0000","last_peered":"2024-04-02T13:29:12.577646+0000","last_clean":"2024-04-02T13:29:12.577646+0000","last_became_active":"2024-04-02T13:29:10.559051+0000","last_became_peered":"2024-04-02T13:29:10.559051+0000","last_unstale":"2024-04-02T13:29:12.577646+0000","last_undegraded":"2024-04-02T13:29:12.577646+0000","last_fullsized":"2024-04-02T13:29:12.577646+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:59:40.809465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552396+0000","last_change":"2024-04-02T13:29:11.561544+0000","last_active":"2024-04-02T13:29:12.552396+0000","last_peered":"2024-04-02T13:29:12.552396+0000","last_clean":"2024-04-02T13:29:12.552396+0000","last_became_active":"2024-04-02T13:29:11.561274+0000","last_became_peered":"2024-04-02T13:29:11.561274+0000","last_unstale":"2024-04-02T13:29:12.552396+0000","last_undegraded":"2024-04-02T13:29:12.552396+0000","last_fullsized":"2024-04-02T13:29:12.552396+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:13:24.816639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655459+0000","last_change":"2024-04-02T13:29:10.549906+0000","last_active":"2024-04-02T13:29:12.655459+0000","last_peered":"2024-04-02T13:29:12.655459+0000","last_clean":"2024-04-02T13:29:12.655459+0000","last_became_active":"2024-04-02T13:29:10.549622+0000","last_became_peered":"2024-04-02T13:29:10.549622+0000","last_unstale":"2024-04-02T13:29:12.655459+0000","last_undegraded":"2024-04-02T13:29:12.655459+0000","last_fullsized":"2024-04-02T13:29:12.655459+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:01:19.156677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556630+0000","last_change":"2024-04-02T13:29:11.561639+0000","last_active":"2024-04-02T13:29:12.556630+0000","last_peered":"2024-04-02T13:29:12.556630+0000","last_clean":"2024-04-02T13:29:12.556630+0000","last_became_active":"2024-04-02T13:29:11.561357+0000","last_became_peered":"2024-04-02T13:29:11.561357+0000","last_unstale":"2024-04-02T13:29:12.556630+0000","last_undegraded":"2024-04-02T13:29:12.556630+0000","last_fullsized":"2024-04-02T13:29:12.556630+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:07:12.001409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092022+0000","last_change":"2024-04-02T13:29:10.556359+0000","last_active":"2024-04-02T13:29:13.092022+0000","last_peered":"2024-04-02T13:29:13.092022+0000","last_clean":"2024-04-02T13:29:13.092022+0000","last_became_active":"2024-04-02T13:29:10.555656+0000","last_became_peered":"2024-04-02T13:29:10.555656+0000","last_unstale":"2024-04-02T13:29:13.092022+0000","last_undegraded":"2024-04-02T13:29:13.092022+0000","last_fullsized":"2024-04-02T13:29:13.092022+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:11.502526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092216+0000","last_change":"2024-04-02T13:29:11.570565+0000","last_active":"2024-04-02T13:29:13.092216+0000","last_peered":"2024-04-02T13:29:13.092216+0000","last_clean":"2024-04-02T13:29:13.092216+0000","last_became_active":"2024-04-02T13:29:11.570292+0000","last_became_peered":"2024-04-02T13:29:11.570292+0000","last_unstale":"2024-04-02T13:29:13.092216+0000","last_undegraded":"2024-04-02T13:29:13.092216+0000","last_fullsized":"2024-04-02T13:29:13.092216+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:47:27.598123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092242+0000","last_change":"2024-04-02T13:29:10.554912+0000","last_active":"2024-04-02T13:29:13.092242+0000","last_peered":"2024-04-02T13:29:13.092242+0000","last_clean":"2024-04-02T13:29:13.092242+0000","last_became_active":"2024-04-02T13:29:10.554610+0000","last_became_peered":"2024-04-02T13:29:10.554610+0000","last_unstale":"2024-04-02T13:29:13.092242+0000","last_undegraded":"2024-04-02T13:29:13.092242+0000","last_fullsized":"2024-04-02T13:29:13.092242+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:11.502526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092226+0000","last_change":"2024-04-02T13:29:11.575099+0000","last_active":"2024-04-02T13:29:13.092226+0000","last_peered":"2024-04-02T13:29:13.092226+0000","last_clean":"2024-04-02T13:29:13.092226+0000","last_became_active":"2024-04-02T13:29:11.574791+0000","last_became_peered":"2024-04-02T13:29:11.574791+0000","last_unstale":"2024-04-02T13:29:13.092226+0000","last_undegraded":"2024-04-02T13:29:13.092226+0000","last_fullsized":"2024-04-02T13:29:13.092226+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:36.295285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091877+0000","last_change":"2024-04-02T13:29:10.555944+0000","last_active":"2024-04-02T13:29:13.091877+0000","last_peered":"2024-04-02T13:29:13.091877+0000","last_clean":"2024-04-02T13:29:13.091877+0000","last_became_active":"2024-04-02T13:29:10.555666+0000","last_became_peered":"2024-04-02T13:29:10.555666+0000","last_unstale":"2024-04-02T13:29:13.091877+0000","last_undegraded":"2024-04-02T13:29:13.091877+0000","last_fullsized":"2024-04-02T13:29:13.091877+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:26.195781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091875+0000","last_change":"2024-04-02T13:29:11.572201+0000","last_active":"2024-04-02T13:29:13.091875+0000","last_peered":"2024-04-02T13:29:13.091875+0000","last_clean":"2024-04-02T13:29:13.091875+0000","last_became_active":"2024-04-02T13:29:11.571910+0000","last_became_peered":"2024-04-02T13:29:11.571910+0000","last_unstale":"2024-04-02T13:29:13.091875+0000","last_undegraded":"2024-04-02T13:29:13.091875+0000","last_fullsized":"2024-04-02T13:29:13.091875+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:37.386743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655329+0000","last_change":"2024-04-02T13:29:10.549745+0000","last_active":"2024-04-02T13:29:12.655329+0000","last_peered":"2024-04-02T13:29:12.655329+0000","last_clean":"2024-04-02T13:29:12.655329+0000","last_became_active":"2024-04-02T13:29:10.549480+0000","last_became_peered":"2024-04-02T13:29:10.549480+0000","last_unstale":"2024-04-02T13:29:12.655329+0000","last_undegraded":"2024-04-02T13:29:12.655329+0000","last_fullsized":"2024-04-02T13:29:12.655329+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:15:26.524288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556991+0000","last_change":"2024-04-02T13:29:11.562383+0000","last_active":"2024-04-02T13:29:12.556991+0000","last_peered":"2024-04-02T13:29:12.556991+0000","last_clean":"2024-04-02T13:29:12.556991+0000","last_became_active":"2024-04-02T13:29:11.562098+0000","last_became_peered":"2024-04-02T13:29:11.562098+0000","last_unstale":"2024-04-02T13:29:12.556991+0000","last_undegraded":"2024-04-02T13:29:12.556991+0000","last_fullsized":"2024-04-02T13:29:12.556991+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:31.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091608+0000","last_change":"2024-04-02T13:29:10.561749+0000","last_active":"2024-04-02T13:29:13.091608+0000","last_peered":"2024-04-02T13:29:13.091608+0000","last_clean":"2024-04-02T13:29:13.091608+0000","last_became_active":"2024-04-02T13:29:10.561518+0000","last_became_peered":"2024-04-02T13:29:10.561518+0000","last_unstale":"2024-04-02T13:29:13.091608+0000","last_undegraded":"2024-04-02T13:29:13.091608+0000","last_fullsized":"2024-04-02T13:29:13.091608+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:19.438340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556416+0000","last_change":"2024-04-02T13:29:11.564242+0000","last_active":"2024-04-02T13:29:12.556416+0000","last_peered":"2024-04-02T13:29:12.556416+0000","last_clean":"2024-04-02T13:29:12.556416+0000","last_became_active":"2024-04-02T13:29:11.563263+0000","last_became_peered":"2024-04-02T13:29:11.563263+0000","last_unstale":"2024-04-02T13:29:12.556416+0000","last_undegraded":"2024-04-02T13:29:12.556416+0000","last_fullsized":"2024-04-02T13:29:12.556416+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:41.907332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092361+0000","last_change":"2024-04-02T13:29:10.552044+0000","last_active":"2024-04-02T13:29:13.092361+0000","last_peered":"2024-04-02T13:29:13.092361+0000","last_clean":"2024-04-02T13:29:13.092361+0000","last_became_active":"2024-04-02T13:29:10.550405+0000","last_became_peered":"2024-04-02T13:29:10.550405+0000","last_unstale":"2024-04-02T13:29:13.092361+0000","last_undegraded":"2024-04-02T13:29:13.092361+0000","last_fullsized":"2024-04-02T13:29:13.092361+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:47:39.094392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":335,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091520+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:29:13.091520+0000","last_peered":"2024-04-02T13:29:13.091520+0000","last_clean":"2024-04-02T13:29:13.091520+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:29:13.091520+0000","last_undegraded":"2024-04-02T13:29:13.091520+0000","last_fullsized":"2024-04-02T13:29:13.091520+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1916928,"data_stored":1201478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574958,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1335296,"data_stored":613166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574959,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1314816,"data_stored":610950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574959,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32480,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339360,"statfs":{"total":96636764160,"available":96603504640,"internally_reserved":0,"allocated":1323008,"data_stored":611042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574959,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1318912,"data_stored":611532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574958,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1318912,"data_stored":610984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":610984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574959,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33064,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338776,"statfs":{"total":96636764160,"available":96602906624,"internally_reserved":0,"allocated":1925120,"data_stored":1203020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:29:15.572 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:29:15.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.707+0000 7f3d4de3c640 1 Processor -- start 2024-04-02T13:29:15.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.707+0000 7f3d4de3c640 1 -- start start 2024-04-02T13:29:15.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.708+0000 7f3d4de3c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3d480b47b0 0x7f3d480d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.708+0000 7f3d4de3c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d480d4d70 con 0x7f3d480d28f0 2024-04-02T13:29:15.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.708+0000 7f3d4de3c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d480d2030 con 0x7f3d480b5200 2024-04-02T13:29:15.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.708+0000 7f3d4de3c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f3d480d5050 con 0x7f3d480b47b0 2024-04-02T13:29:15.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.708+0000 7f3d46ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3d480b47b0 0x7f3d480d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.708+0000 7f3d477fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f3d480d28f0 0x7f3d480b40a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40440/0 (socket says 172.21.15.44:40440) 2024-04-02T13:29:15.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.708+0000 7f3d477fe640 1 -- 172.21.15.44:0/3746355954 learned_addr learned my addr 172.21.15.44:0/3746355954 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:15.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.708+0000 7f3d46ffd640 1 -- 172.21.15.44:0/3746355954 >> v1:172.21.15.69:6789/0 conn(0x7f3d480b5200 legacy=0x7f3d480b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:15.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.708+0000 7f3d46ffd640 1 -- 172.21.15.44:0/3746355954 >> v1:172.21.15.44:6789/0 conn(0x7f3d480d28f0 legacy=0x7f3d480b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.708+0000 7f3d46ffd640 1 -- 172.21.15.44:0/3746355954 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d480b4cb0 con 0x7f3d480b47b0 2024-04-02T13:29:15.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.708+0000 7f3d467fc640 1 -- 172.21.15.44:0/3746355954 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 329477104 0 0) 0x7f3d480d4d70 con 0x7f3d480d28f0 2024-04-02T13:29:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.709+0000 7f3d46ffd640 1 --2- 172.21.15.44:0/3746355954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d480b47b0 0x7f3d480d2cd0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f3d380097a0 tx=0x7f3d3802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8302a2e838dd13 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.709+0000 7f3d467fc640 1 -- 172.21.15.44:0/3746355954 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38038070 con 0x7f3d480b47b0 2024-04-02T13:29:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.709+0000 7f3d467fc640 1 -- 172.21.15.44:0/3746355954 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d38031700 con 0x7f3d480b47b0 2024-04-02T13:29:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.709+0000 7f3d467fc640 1 -- 172.21.15.44:0/3746355954 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38031ad0 con 0x7f3d480b47b0 2024-04-02T13:29:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.709+0000 7f3d4de3c640 1 -- 172.21.15.44:0/3746355954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d480b47b0 msgr2=0x7f3d480d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.709+0000 7f3d4de3c640 1 --2- 172.21.15.44:0/3746355954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d480b47b0 0x7f3d480d2cd0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f3d380097a0 tx=0x7f3d3802b7a0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.709+0000 7f3d4de3c640 1 -- 172.21.15.44:0/3746355954 shutdown_connections 2024-04-02T13:29:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.709+0000 7f3d4de3c640 1 --2- 172.21.15.44:0/3746355954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d480b47b0 0x7f3d480d2cd0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.709+0000 7f3d4de3c640 1 -- 172.21.15.44:0/3746355954 >> 172.21.15.44:0/3746355954 conn(0x7f3d480d4120 msgr2=0x7f3d480d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.709+0000 7f3d4de3c640 1 -- 172.21.15.44:0/3746355954 shutdown_connections 2024-04-02T13:29:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.710+0000 7f3d4de3c640 1 -- 172.21.15.44:0/3746355954 wait complete. 2024-04-02T13:29:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.711+0000 7f3d4de3c640 1 Processor -- start 2024-04-02T13:29:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.711+0000 7f3d4de3c640 1 -- start start 2024-04-02T13:29:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.711+0000 7f3d4de3c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d480b47b0 0x7f3d480cb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.711+0000 7f3d4de3c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d480b5200 0x7f3d480b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.711+0000 7f3d4de3c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d480d28f0 0x7f3d480b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.711+0000 7f3d4de3c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3d480d23a0 con 0x7f3d480d28f0 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.711+0000 7f3d4de3c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3d480d14f0 con 0x7f3d480b47b0 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.711+0000 7f3d4de3c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3d480b5020 con 0x7f3d480b5200 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d47fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d480d28f0 0x7f3d480b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d47fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d480d28f0 0x7f3d480b2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49090/0 (socket says 172.21.15.44:49090) 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d47fff640 1 -- 172.21.15.44:0/1410960742 learned_addr learned my addr 172.21.15.44:0/1410960742 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d46ffd640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d480b5200 0x7f3d480b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d477fe640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d480b47b0 0x7f3d480cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d47fff640 1 -- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d480b5200 msgr2=0x7f3d480b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d47fff640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d480b5200 0x7f3d480b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d47fff640 1 -- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d480b47b0 msgr2=0x7f3d480cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d47fff640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d480b47b0 0x7f3d480cb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d47fff640 1 -- 172.21.15.44:0/1410960742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d480b29c0 con 0x7f3d480d28f0 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d47fff640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d480d28f0 0x7f3d480b2240 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000b9c0 tx=0x7f3d4000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d2ffff640 1 -- 172.21.15.44:0/1410960742 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d40019070 con 0x7f3d480d28f0 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d2ffff640 1 -- 172.21.15.44:0/1410960742 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d4000cba0 con 0x7f3d480d28f0 2024-04-02T13:29:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d2ffff640 1 -- 172.21.15.44:0/1410960742 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d40014440 con 0x7f3d480d28f0 2024-04-02T13:29:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d4de3c640 1 -- 172.21.15.44:0/1410960742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d480b2ca0 con 0x7f3d480d28f0 2024-04-02T13:29:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.712+0000 7f3d4de3c640 1 -- 172.21.15.44:0/1410960742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d480b3260 con 0x7f3d480d28f0 2024-04-02T13:29:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.713+0000 7f3d4de3c640 1 -- 172.21.15.44:0/1410960742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d48073820 con 0x7f3d480d28f0 2024-04-02T13:29:15.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.714+0000 7f3d2ffff640 1 -- 172.21.15.44:0/1410960742 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d4001c070 con 0x7f3d480d28f0 2024-04-02T13:29:15.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.714+0000 7f3d2ffff640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d3406c8f0 0x7f3d3406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:15.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.714+0000 7f3d2ffff640 1 -- 172.21.15.44:0/1410960742 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f3d4008bab0 con 0x7f3d480d28f0 2024-04-02T13:29:15.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.714+0000 7f3d477fe640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d3406c8f0 0x7f3d3406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:15.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.715+0000 7f3d477fe640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d3406c8f0 0x7f3d3406edb0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c000f60 tx=0x7f3d3c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.719+0000 7f3d2ffff640 1 -- 172.21.15.44:0/1410960742 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3d48073820 con 0x7f3d480d28f0 2024-04-02T13:29:15.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.903+0000 7f3d4de3c640 1 -- 172.21.15.44:0/1410960742 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3d480684f0 con 0x7f3d3406c8f0 2024-04-02T13:29:15.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.909+0000 7f3d2ffff640 1 -- 172.21.15.44:0/1410960742 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252483 (secure 0 0 0) 0x7f3d480684f0 con 0x7f3d3406c8f0 2024-04-02T13:29:15.915 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:15.916 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:29:15.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.913+0000 7f3d4de3c640 1 -- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d3406c8f0 msgr2=0x7f3d3406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.913+0000 7f3d4de3c640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d3406c8f0 0x7f3d3406edb0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c000f60 tx=0x7f3d3c009040 comp rx=0 tx=0).stop 2024-04-02T13:29:15.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.913+0000 7f3d4de3c640 1 -- 172.21.15.44:0/1410960742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d480d28f0 msgr2=0x7f3d480b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:15.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.913+0000 7f3d4de3c640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d480d28f0 0x7f3d480b2240 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000b9c0 tx=0x7f3d4000be80 comp rx=0 tx=0).stop 2024-04-02T13:29:15.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.913+0000 7f3d4de3c640 1 -- 172.21.15.44:0/1410960742 shutdown_connections 2024-04-02T13:29:15.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.913+0000 7f3d4de3c640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3d3406c8f0 0x7f3d3406edb0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.913+0000 7f3d4de3c640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d480d28f0 0x7f3d480b2240 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.913+0000 7f3d4de3c640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d480b5200 0x7f3d480b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.913+0000 7f3d4de3c640 1 --2- 172.21.15.44:0/1410960742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d480b47b0 0x7f3d480cb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:15.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.913+0000 7f3d4de3c640 1 -- 172.21.15.44:0/1410960742 >> 172.21.15.44:0/1410960742 conn(0x7f3d480d4120 msgr2=0x7f3d480d01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:15.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.913+0000 7f3d4de3c640 1 -- 172.21.15.44:0/1410960742 shutdown_connections 2024-04-02T13:29:15.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:15.913+0000 7f3d4de3c640 1 -- 172.21.15.44:0/1410960742 wait complete. 2024-04-02T13:29:15.926 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-04-02T13:29:15.008080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260984,"kb_used_data":11496,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713736,"statfs":{"total":773094113280,"available":772826865664,"internally_reserved":0,"allocated":11771904,"data_stored":6073156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.475726"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555405+0000","last_change":"2024-04-02T13:29:11.572158+0000","last_active":"2024-04-02T13:29:12.555405+0000","last_peered":"2024-04-02T13:29:12.555405+0000","last_clean":"2024-04-02T13:29:12.555405+0000","last_became_active":"2024-04-02T13:29:11.571759+0000","last_became_peered":"2024-04-02T13:29:11.571759+0000","last_unstale":"2024-04-02T13:29:12.555405+0000","last_undegraded":"2024-04-02T13:29:12.555405+0000","last_fullsized":"2024-04-02T13:29:12.555405+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:53:26.735220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655515+0000","last_change":"2024-04-02T13:29:11.570652+0000","last_active":"2024-04-02T13:29:12.655515+0000","last_peered":"2024-04-02T13:29:12.655515+0000","last_clean":"2024-04-02T13:29:12.655515+0000","last_became_active":"2024-04-02T13:29:11.570392+0000","last_became_peered":"2024-04-02T13:29:11.570392+0000","last_unstale":"2024-04-02T13:29:12.655515+0000","last_undegraded":"2024-04-02T13:29:12.655515+0000","last_fullsized":"2024-04-02T13:29:12.655515+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:05:16.048100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555167+0000","last_change":"2024-04-02T13:29:11.564487+0000","last_active":"2024-04-02T13:29:12.555167+0000","last_peered":"2024-04-02T13:29:12.555167+0000","last_clean":"2024-04-02T13:29:12.555167+0000","last_became_active":"2024-04-02T13:29:11.564194+0000","last_became_peered":"2024-04-02T13:29:11.564194+0000","last_unstale":"2024-04-02T13:29:12.555167+0000","last_undegraded":"2024-04-02T13:29:12.555167+0000","last_fullsized":"2024-04-02T13:29:12.555167+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:11:38.798164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552571+0000","last_change":"2024-04-02T13:29:11.570909+0000","last_active":"2024-04-02T13:29:12.552571+0000","last_peered":"2024-04-02T13:29:12.552571+0000","last_clean":"2024-04-02T13:29:12.552571+0000","last_became_active":"2024-04-02T13:29:11.570624+0000","last_became_peered":"2024-04-02T13:29:11.570624+0000","last_unstale":"2024-04-02T13:29:12.552571+0000","last_undegraded":"2024-04-02T13:29:12.552571+0000","last_fullsized":"2024-04-02T13:29:12.552571+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:00.080193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091574+0000","last_change":"2024-04-02T13:29:11.570175+0000","last_active":"2024-04-02T13:29:13.091574+0000","last_peered":"2024-04-02T13:29:13.091574+0000","last_clean":"2024-04-02T13:29:13.091574+0000","last_became_active":"2024-04-02T13:29:11.569345+0000","last_became_peered":"2024-04-02T13:29:11.569345+0000","last_unstale":"2024-04-02T13:29:13.091574+0000","last_undegraded":"2024-04-02T13:29:13.091574+0000","last_fullsized":"2024-04-02T13:29:13.091574+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:41.907332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655089+0000","last_change":"2024-04-02T13:29:11.567175+0000","last_active":"2024-04-02T13:29:12.655089+0000","last_peered":"2024-04-02T13:29:12.655089+0000","last_clean":"2024-04-02T13:29:12.655089+0000","last_became_active":"2024-04-02T13:29:11.566897+0000","last_became_peered":"2024-04-02T13:29:11.566897+0000","last_unstale":"2024-04-02T13:29:12.655089+0000","last_undegraded":"2024-04-02T13:29:12.655089+0000","last_fullsized":"2024-04-02T13:29:12.655089+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:07:12.001409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655016+0000","last_change":"2024-04-02T13:29:11.570937+0000","last_active":"2024-04-02T13:29:12.655016+0000","last_peered":"2024-04-02T13:29:12.655016+0000","last_clean":"2024-04-02T13:29:12.655016+0000","last_became_active":"2024-04-02T13:29:11.568973+0000","last_became_peered":"2024-04-02T13:29:11.568973+0000","last_unstale":"2024-04-02T13:29:12.655016+0000","last_undegraded":"2024-04-02T13:29:12.655016+0000","last_fullsized":"2024-04-02T13:29:12.655016+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:54:37.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091369+0000","last_change":"2024-04-02T13:29:11.571629+0000","last_active":"2024-04-02T13:29:13.091369+0000","last_peered":"2024-04-02T13:29:13.091369+0000","last_clean":"2024-04-02T13:29:13.091369+0000","last_became_active":"2024-04-02T13:29:11.571373+0000","last_became_peered":"2024-04-02T13:29:11.571373+0000","last_unstale":"2024-04-02T13:29:13.091369+0000","last_undegraded":"2024-04-02T13:29:13.091369+0000","last_fullsized":"2024-04-02T13:29:13.091369+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:52:19.518479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655489+0000","last_change":"2024-04-02T13:29:11.568280+0000","last_active":"2024-04-02T13:29:12.655489+0000","last_peered":"2024-04-02T13:29:12.655489+0000","last_clean":"2024-04-02T13:29:12.655489+0000","last_became_active":"2024-04-02T13:29:11.567211+0000","last_became_peered":"2024-04-02T13:29:11.567211+0000","last_unstale":"2024-04-02T13:29:12.655489+0000","last_undegraded":"2024-04-02T13:29:12.655489+0000","last_fullsized":"2024-04-02T13:29:12.655489+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:31.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091828+0000","last_change":"2024-04-02T13:29:11.563645+0000","last_active":"2024-04-02T13:29:13.091828+0000","last_peered":"2024-04-02T13:29:13.091828+0000","last_clean":"2024-04-02T13:29:13.091828+0000","last_became_active":"2024-04-02T13:29:11.563378+0000","last_became_peered":"2024-04-02T13:29:11.563378+0000","last_unstale":"2024-04-02T13:29:13.091828+0000","last_undegraded":"2024-04-02T13:29:13.091828+0000","last_fullsized":"2024-04-02T13:29:13.091828+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:52.672917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556913+0000","last_change":"2024-04-02T13:29:11.564144+0000","last_active":"2024-04-02T13:29:12.556913+0000","last_peered":"2024-04-02T13:29:12.556913+0000","last_clean":"2024-04-02T13:29:12.556913+0000","last_became_active":"2024-04-02T13:29:11.563863+0000","last_became_peered":"2024-04-02T13:29:11.563863+0000","last_unstale":"2024-04-02T13:29:12.556913+0000","last_undegraded":"2024-04-02T13:29:12.556913+0000","last_fullsized":"2024-04-02T13:29:12.556913+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:49.582030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092500+0000","last_change":"2024-04-02T13:29:11.572938+0000","last_active":"2024-04-02T13:29:13.092500+0000","last_peered":"2024-04-02T13:29:13.092500+0000","last_clean":"2024-04-02T13:29:13.092500+0000","last_became_active":"2024-04-02T13:29:11.571439+0000","last_became_peered":"2024-04-02T13:29:11.571439+0000","last_unstale":"2024-04-02T13:29:13.092500+0000","last_undegraded":"2024-04-02T13:29:13.092500+0000","last_fullsized":"2024-04-02T13:29:13.092500+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:20.988426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552385+0000","last_change":"2024-04-02T13:29:11.565274+0000","last_active":"2024-04-02T13:29:12.552385+0000","last_peered":"2024-04-02T13:29:12.552385+0000","last_clean":"2024-04-02T13:29:12.552385+0000","last_became_active":"2024-04-02T13:29:11.565025+0000","last_became_peered":"2024-04-02T13:29:11.565025+0000","last_unstale":"2024-04-02T13:29:12.552385+0000","last_undegraded":"2024-04-02T13:29:12.552385+0000","last_fullsized":"2024-04-02T13:29:12.552385+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:49:39.658389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091660+0000","last_change":"2024-04-02T13:29:11.562082+0000","last_active":"2024-04-02T13:29:13.091660+0000","last_peered":"2024-04-02T13:29:13.091660+0000","last_clean":"2024-04-02T13:29:13.091660+0000","last_became_active":"2024-04-02T13:29:11.561781+0000","last_became_peered":"2024-04-02T13:29:11.561781+0000","last_unstale":"2024-04-02T13:29:13.091660+0000","last_undegraded":"2024-04-02T13:29:13.091660+0000","last_fullsized":"2024-04-02T13:29:13.091660+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:29:24.686822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091910+0000","last_change":"2024-04-02T13:29:11.573376+0000","last_active":"2024-04-02T13:29:13.091910+0000","last_peered":"2024-04-02T13:29:13.091910+0000","last_clean":"2024-04-02T13:29:13.091910+0000","last_became_active":"2024-04-02T13:29:11.573084+0000","last_became_peered":"2024-04-02T13:29:11.573084+0000","last_unstale":"2024-04-02T13:29:13.091910+0000","last_undegraded":"2024-04-02T13:29:13.091910+0000","last_fullsized":"2024-04-02T13:29:13.091910+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:46:50.450645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091790+0000","last_change":"2024-04-02T13:29:11.563245+0000","last_active":"2024-04-02T13:29:13.091790+0000","last_peered":"2024-04-02T13:29:13.091790+0000","last_clean":"2024-04-02T13:29:13.091790+0000","last_became_active":"2024-04-02T13:29:11.562973+0000","last_became_peered":"2024-04-02T13:29:11.562973+0000","last_unstale":"2024-04-02T13:29:13.091790+0000","last_undegraded":"2024-04-02T13:29:13.091790+0000","last_fullsized":"2024-04-02T13:29:13.091790+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:00:05.121231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555513+0000","last_change":"2024-04-02T13:29:11.564633+0000","last_active":"2024-04-02T13:29:12.555513+0000","last_peered":"2024-04-02T13:29:12.555513+0000","last_clean":"2024-04-02T13:29:12.555513+0000","last_became_active":"2024-04-02T13:29:11.564360+0000","last_became_peered":"2024-04-02T13:29:11.564360+0000","last_unstale":"2024-04-02T13:29:12.555513+0000","last_undegraded":"2024-04-02T13:29:12.555513+0000","last_fullsized":"2024-04-02T13:29:12.555513+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:21.325009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655523+0000","last_change":"2024-04-02T13:29:11.569256+0000","last_active":"2024-04-02T13:29:12.655523+0000","last_peered":"2024-04-02T13:29:12.655523+0000","last_clean":"2024-04-02T13:29:12.655523+0000","last_became_active":"2024-04-02T13:29:11.568992+0000","last_became_peered":"2024-04-02T13:29:11.568992+0000","last_unstale":"2024-04-02T13:29:12.655523+0000","last_undegraded":"2024-04-02T13:29:12.655523+0000","last_fullsized":"2024-04-02T13:29:12.655523+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:27:20.320062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555354+0000","last_change":"2024-04-02T13:29:11.569227+0000","last_active":"2024-04-02T13:29:12.555354+0000","last_peered":"2024-04-02T13:29:12.555354+0000","last_clean":"2024-04-02T13:29:12.555354+0000","last_became_active":"2024-04-02T13:29:11.568966+0000","last_became_peered":"2024-04-02T13:29:11.568966+0000","last_unstale":"2024-04-02T13:29:12.555354+0000","last_undegraded":"2024-04-02T13:29:12.555354+0000","last_fullsized":"2024-04-02T13:29:12.555354+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:12:18.457537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092582+0000","last_change":"2024-04-02T13:29:11.566637+0000","last_active":"2024-04-02T13:29:13.092582+0000","last_peered":"2024-04-02T13:29:13.092582+0000","last_clean":"2024-04-02T13:29:13.092582+0000","last_became_active":"2024-04-02T13:29:11.565531+0000","last_became_peered":"2024-04-02T13:29:11.565531+0000","last_unstale":"2024-04-02T13:29:13.092582+0000","last_undegraded":"2024-04-02T13:29:13.092582+0000","last_fullsized":"2024-04-02T13:29:13.092582+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:31.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092189+0000","last_change":"2024-04-02T13:29:11.564849+0000","last_active":"2024-04-02T13:29:13.092189+0000","last_peered":"2024-04-02T13:29:13.092189+0000","last_clean":"2024-04-02T13:29:13.092189+0000","last_became_active":"2024-04-02T13:29:11.564597+0000","last_became_peered":"2024-04-02T13:29:11.564597+0000","last_unstale":"2024-04-02T13:29:13.092189+0000","last_undegraded":"2024-04-02T13:29:13.092189+0000","last_fullsized":"2024-04-02T13:29:13.092189+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:45:48.117521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091774+0000","last_change":"2024-04-02T13:29:11.556789+0000","last_active":"2024-04-02T13:29:13.091774+0000","last_peered":"2024-04-02T13:29:13.091774+0000","last_clean":"2024-04-02T13:29:13.091774+0000","last_became_active":"2024-04-02T13:29:11.556490+0000","last_became_peered":"2024-04-02T13:29:11.556490+0000","last_unstale":"2024-04-02T13:29:13.091774+0000","last_undegraded":"2024-04-02T13:29:13.091774+0000","last_fullsized":"2024-04-02T13:29:13.091774+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:07:12.001409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555172+0000","last_change":"2024-04-02T13:29:11.557712+0000","last_active":"2024-04-02T13:29:12.555172+0000","last_peered":"2024-04-02T13:29:12.555172+0000","last_clean":"2024-04-02T13:29:12.555172+0000","last_became_active":"2024-04-02T13:29:11.557443+0000","last_became_peered":"2024-04-02T13:29:11.557443+0000","last_unstale":"2024-04-02T13:29:12.555172+0000","last_undegraded":"2024-04-02T13:29:12.555172+0000","last_fullsized":"2024-04-02T13:29:12.555172+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:16:10.761598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091377+0000","last_change":"2024-04-02T13:29:11.556553+0000","last_active":"2024-04-02T13:29:13.091377+0000","last_peered":"2024-04-02T13:29:13.091377+0000","last_clean":"2024-04-02T13:29:13.091377+0000","last_became_active":"2024-04-02T13:29:11.556284+0000","last_became_peered":"2024-04-02T13:29:11.556284+0000","last_unstale":"2024-04-02T13:29:13.091377+0000","last_undegraded":"2024-04-02T13:29:13.091377+0000","last_fullsized":"2024-04-02T13:29:13.091377+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:53:56.947267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092287+0000","last_change":"2024-04-02T13:29:11.575767+0000","last_active":"2024-04-02T13:29:13.092287+0000","last_peered":"2024-04-02T13:29:13.092287+0000","last_clean":"2024-04-02T13:29:13.092287+0000","last_became_active":"2024-04-02T13:29:11.575491+0000","last_became_peered":"2024-04-02T13:29:11.575491+0000","last_unstale":"2024-04-02T13:29:13.092287+0000","last_undegraded":"2024-04-02T13:29:13.092287+0000","last_fullsized":"2024-04-02T13:29:13.092287+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:07:12.001409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092329+0000","last_change":"2024-04-02T13:29:11.568347+0000","last_active":"2024-04-02T13:29:13.092329+0000","last_peered":"2024-04-02T13:29:13.092329+0000","last_clean":"2024-04-02T13:29:13.092329+0000","last_became_active":"2024-04-02T13:29:11.568030+0000","last_became_peered":"2024-04-02T13:29:11.568030+0000","last_unstale":"2024-04-02T13:29:13.092329+0000","last_undegraded":"2024-04-02T13:29:13.092329+0000","last_fullsized":"2024-04-02T13:29:13.092329+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:49.582030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556914+0000","last_change":"2024-04-02T13:29:11.564763+0000","last_active":"2024-04-02T13:29:12.556914+0000","last_peered":"2024-04-02T13:29:12.556914+0000","last_clean":"2024-04-02T13:29:12.556914+0000","last_became_active":"2024-04-02T13:29:11.562744+0000","last_became_peered":"2024-04-02T13:29:11.562744+0000","last_unstale":"2024-04-02T13:29:12.556914+0000","last_undegraded":"2024-04-02T13:29:12.556914+0000","last_fullsized":"2024-04-02T13:29:12.556914+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:20.988426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655422+0000","last_change":"2024-04-02T13:29:11.568275+0000","last_active":"2024-04-02T13:29:12.655422+0000","last_peered":"2024-04-02T13:29:12.655422+0000","last_clean":"2024-04-02T13:29:12.655422+0000","last_became_active":"2024-04-02T13:29:11.567148+0000","last_became_peered":"2024-04-02T13:29:11.567148+0000","last_unstale":"2024-04-02T13:29:12.655422+0000","last_undegraded":"2024-04-02T13:29:12.655422+0000","last_fullsized":"2024-04-02T13:29:12.655422+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:20.988426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092232+0000","last_change":"2024-04-02T13:29:11.561927+0000","last_active":"2024-04-02T13:29:13.092232+0000","last_peered":"2024-04-02T13:29:13.092232+0000","last_clean":"2024-04-02T13:29:13.092232+0000","last_became_active":"2024-04-02T13:29:11.561641+0000","last_became_peered":"2024-04-02T13:29:11.561641+0000","last_unstale":"2024-04-02T13:29:13.092232+0000","last_undegraded":"2024-04-02T13:29:13.092232+0000","last_fullsized":"2024-04-02T13:29:13.092232+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:18:45.068600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092079+0000","last_change":"2024-04-02T13:29:11.575792+0000","last_active":"2024-04-02T13:29:13.092079+0000","last_peered":"2024-04-02T13:29:13.092079+0000","last_clean":"2024-04-02T13:29:13.092079+0000","last_became_active":"2024-04-02T13:29:11.575527+0000","last_became_peered":"2024-04-02T13:29:11.575527+0000","last_unstale":"2024-04-02T13:29:13.092079+0000","last_undegraded":"2024-04-02T13:29:13.092079+0000","last_fullsized":"2024-04-02T13:29:13.092079+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:54:37.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655137+0000","last_change":"2024-04-02T13:29:11.571601+0000","last_active":"2024-04-02T13:29:12.655137+0000","last_peered":"2024-04-02T13:29:12.655137+0000","last_clean":"2024-04-02T13:29:12.655137+0000","last_became_active":"2024-04-02T13:29:11.571323+0000","last_became_peered":"2024-04-02T13:29:11.571323+0000","last_unstale":"2024-04-02T13:29:12.655137+0000","last_undegraded":"2024-04-02T13:29:12.655137+0000","last_fullsized":"2024-04-02T13:29:12.655137+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:52.672917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091386+0000","last_change":"2024-04-02T13:29:11.572161+0000","last_active":"2024-04-02T13:29:13.091386+0000","last_peered":"2024-04-02T13:29:13.091386+0000","last_clean":"2024-04-02T13:29:13.091386+0000","last_became_active":"2024-04-02T13:29:11.571918+0000","last_became_peered":"2024-04-02T13:29:11.571918+0000","last_unstale":"2024-04-02T13:29:13.091386+0000","last_undegraded":"2024-04-02T13:29:13.091386+0000","last_fullsized":"2024-04-02T13:29:13.091386+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:43:49.496031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555267+0000","last_change":"2024-04-02T13:29:11.566629+0000","last_active":"2024-04-02T13:29:12.555267+0000","last_peered":"2024-04-02T13:29:12.555267+0000","last_clean":"2024-04-02T13:29:12.555267+0000","last_became_active":"2024-04-02T13:29:11.566278+0000","last_became_peered":"2024-04-02T13:29:11.566278+0000","last_unstale":"2024-04-02T13:29:12.555267+0000","last_undegraded":"2024-04-02T13:29:12.555267+0000","last_fullsized":"2024-04-02T13:29:12.555267+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:17:21.218582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"105'4","reported_seq":18,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552702+0000","last_change":"2024-04-02T13:29:10.551574+0000","last_active":"2024-04-02T13:29:12.552702+0000","last_peered":"2024-04-02T13:29:12.552702+0000","last_clean":"2024-04-02T13:29:12.552702+0000","last_became_active":"2024-04-02T13:29:10.551292+0000","last_became_peered":"2024-04-02T13:29:10.551292+0000","last_unstale":"2024-04-02T13:29:12.552702+0000","last_undegraded":"2024-04-02T13:29:12.552702+0000","last_fullsized":"2024-04-02T13:29:12.552702+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:12:46.096389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092003+0000","last_change":"2024-04-02T13:29:11.571460+0000","last_active":"2024-04-02T13:29:13.092003+0000","last_peered":"2024-04-02T13:29:13.092003+0000","last_clean":"2024-04-02T13:29:13.092003+0000","last_became_active":"2024-04-02T13:29:11.571125+0000","last_became_peered":"2024-04-02T13:29:11.571125+0000","last_unstale":"2024-04-02T13:29:13.092003+0000","last_undegraded":"2024-04-02T13:29:13.092003+0000","last_fullsized":"2024-04-02T13:29:13.092003+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:46:59.213999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092360+0000","last_change":"2024-04-02T13:29:10.553591+0000","last_active":"2024-04-02T13:29:13.092360+0000","last_peered":"2024-04-02T13:29:13.092360+0000","last_clean":"2024-04-02T13:29:13.092360+0000","last_became_active":"2024-04-02T13:29:10.553297+0000","last_became_peered":"2024-04-02T13:29:10.553297+0000","last_unstale":"2024-04-02T13:29:13.092360+0000","last_undegraded":"2024-04-02T13:29:13.092360+0000","last_fullsized":"2024-04-02T13:29:13.092360+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:53.392833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092300+0000","last_change":"2024-04-02T13:29:11.570664+0000","last_active":"2024-04-02T13:29:13.092300+0000","last_peered":"2024-04-02T13:29:13.092300+0000","last_clean":"2024-04-02T13:29:13.092300+0000","last_became_active":"2024-04-02T13:29:11.570377+0000","last_became_peered":"2024-04-02T13:29:11.570377+0000","last_unstale":"2024-04-02T13:29:13.092300+0000","last_undegraded":"2024-04-02T13:29:13.092300+0000","last_fullsized":"2024-04-02T13:29:13.092300+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:52.672917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092177+0000","last_change":"2024-04-02T13:29:10.561945+0000","last_active":"2024-04-02T13:29:13.092177+0000","last_peered":"2024-04-02T13:29:13.092177+0000","last_clean":"2024-04-02T13:29:13.092177+0000","last_became_active":"2024-04-02T13:29:10.561668+0000","last_became_peered":"2024-04-02T13:29:10.561668+0000","last_unstale":"2024-04-02T13:29:13.092177+0000","last_undegraded":"2024-04-02T13:29:13.092177+0000","last_fullsized":"2024-04-02T13:29:13.092177+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:34:04.714892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556734+0000","last_change":"2024-04-02T13:29:11.566493+0000","last_active":"2024-04-02T13:29:12.556734+0000","last_peered":"2024-04-02T13:29:12.556734+0000","last_clean":"2024-04-02T13:29:12.556734+0000","last_became_active":"2024-04-02T13:29:11.566087+0000","last_became_peered":"2024-04-02T13:29:11.566087+0000","last_unstale":"2024-04-02T13:29:12.556734+0000","last_undegraded":"2024-04-02T13:29:12.556734+0000","last_fullsized":"2024-04-02T13:29:12.556734+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:52.672917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092018+0000","last_change":"2024-04-02T13:29:10.554736+0000","last_active":"2024-04-02T13:29:13.092018+0000","last_peered":"2024-04-02T13:29:13.092018+0000","last_clean":"2024-04-02T13:29:13.092018+0000","last_became_active":"2024-04-02T13:29:10.554153+0000","last_became_peered":"2024-04-02T13:29:10.554153+0000","last_unstale":"2024-04-02T13:29:13.092018+0000","last_undegraded":"2024-04-02T13:29:13.092018+0000","last_fullsized":"2024-04-02T13:29:13.092018+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:15:26.524288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552637+0000","last_change":"2024-04-02T13:29:11.568599+0000","last_active":"2024-04-02T13:29:12.552637+0000","last_peered":"2024-04-02T13:29:12.552637+0000","last_clean":"2024-04-02T13:29:12.552637+0000","last_became_active":"2024-04-02T13:29:11.568293+0000","last_became_peered":"2024-04-02T13:29:11.568293+0000","last_unstale":"2024-04-02T13:29:12.552637+0000","last_undegraded":"2024-04-02T13:29:12.552637+0000","last_fullsized":"2024-04-02T13:29:12.552637+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:59:41.813799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091980+0000","last_change":"2024-04-02T13:29:10.549037+0000","last_active":"2024-04-02T13:29:13.091980+0000","last_peered":"2024-04-02T13:29:13.091980+0000","last_clean":"2024-04-02T13:29:13.091980+0000","last_became_active":"2024-04-02T13:29:10.548744+0000","last_became_peered":"2024-04-02T13:29:10.548744+0000","last_unstale":"2024-04-02T13:29:13.091980+0000","last_undegraded":"2024-04-02T13:29:13.091980+0000","last_fullsized":"2024-04-02T13:29:13.091980+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:07:10.997075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555463+0000","last_change":"2024-04-02T13:29:11.567262+0000","last_active":"2024-04-02T13:29:12.555463+0000","last_peered":"2024-04-02T13:29:12.555463+0000","last_clean":"2024-04-02T13:29:12.555463+0000","last_became_active":"2024-04-02T13:29:11.567010+0000","last_became_peered":"2024-04-02T13:29:11.567010+0000","last_unstale":"2024-04-02T13:29:12.555463+0000","last_undegraded":"2024-04-02T13:29:12.555463+0000","last_fullsized":"2024-04-02T13:29:12.555463+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:10:28.246157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556550+0000","last_change":"2024-04-02T13:29:10.556750+0000","last_active":"2024-04-02T13:29:12.556550+0000","last_peered":"2024-04-02T13:29:12.556550+0000","last_clean":"2024-04-02T13:29:12.556550+0000","last_became_active":"2024-04-02T13:29:10.556478+0000","last_became_peered":"2024-04-02T13:29:10.556478+0000","last_unstale":"2024-04-02T13:29:12.556550+0000","last_undegraded":"2024-04-02T13:29:12.556550+0000","last_fullsized":"2024-04-02T13:29:12.556550+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:48:36.435520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655606+0000","last_change":"2024-04-02T13:29:11.569527+0000","last_active":"2024-04-02T13:29:12.655606+0000","last_peered":"2024-04-02T13:29:12.655606+0000","last_clean":"2024-04-02T13:29:12.655606+0000","last_became_active":"2024-04-02T13:29:11.568434+0000","last_became_peered":"2024-04-02T13:29:11.568434+0000","last_unstale":"2024-04-02T13:29:12.655606+0000","last_undegraded":"2024-04-02T13:29:12.655606+0000","last_fullsized":"2024-04-02T13:29:12.655606+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:29:24.686822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091959+0000","last_change":"2024-04-02T13:29:10.556247+0000","last_active":"2024-04-02T13:29:13.091959+0000","last_peered":"2024-04-02T13:29:13.091959+0000","last_clean":"2024-04-02T13:29:13.091959+0000","last_became_active":"2024-04-02T13:29:10.554542+0000","last_became_peered":"2024-04-02T13:29:10.554542+0000","last_unstale":"2024-04-02T13:29:13.091959+0000","last_undegraded":"2024-04-02T13:29:13.091959+0000","last_fullsized":"2024-04-02T13:29:13.091959+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:13:59.952423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552694+0000","last_change":"2024-04-02T13:29:11.563227+0000","last_active":"2024-04-02T13:29:12.552694+0000","last_peered":"2024-04-02T13:29:12.552694+0000","last_clean":"2024-04-02T13:29:12.552694+0000","last_became_active":"2024-04-02T13:29:11.562915+0000","last_became_peered":"2024-04-02T13:29:11.562915+0000","last_unstale":"2024-04-02T13:29:12.552694+0000","last_undegraded":"2024-04-02T13:29:12.552694+0000","last_fullsized":"2024-04-02T13:29:12.552694+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:23:55.267998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552305+0000","last_change":"2024-04-02T13:29:10.556556+0000","last_active":"2024-04-02T13:29:12.552305+0000","last_peered":"2024-04-02T13:29:12.552305+0000","last_clean":"2024-04-02T13:29:12.552305+0000","last_became_active":"2024-04-02T13:29:10.556240+0000","last_became_peered":"2024-04-02T13:29:10.556240+0000","last_unstale":"2024-04-02T13:29:12.552305+0000","last_undegraded":"2024-04-02T13:29:12.552305+0000","last_fullsized":"2024-04-02T13:29:12.552305+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:30.633172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555151+0000","last_change":"2024-04-02T13:29:11.570296+0000","last_active":"2024-04-02T13:29:12.555151+0000","last_peered":"2024-04-02T13:29:12.555151+0000","last_clean":"2024-04-02T13:29:12.555151+0000","last_became_active":"2024-04-02T13:29:11.569844+0000","last_became_peered":"2024-04-02T13:29:11.569844+0000","last_unstale":"2024-04-02T13:29:12.555151+0000","last_undegraded":"2024-04-02T13:29:12.555151+0000","last_fullsized":"2024-04-02T13:29:12.555151+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:06:15.210019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092376+0000","last_change":"2024-04-02T13:29:10.562040+0000","last_active":"2024-04-02T13:29:13.092376+0000","last_peered":"2024-04-02T13:29:13.092376+0000","last_clean":"2024-04-02T13:29:13.092376+0000","last_became_active":"2024-04-02T13:29:10.561482+0000","last_became_peered":"2024-04-02T13:29:10.561482+0000","last_unstale":"2024-04-02T13:29:13.092376+0000","last_undegraded":"2024-04-02T13:29:13.092376+0000","last_fullsized":"2024-04-02T13:29:13.092376+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:48:36.435520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091569+0000","last_change":"2024-04-02T13:29:11.566957+0000","last_active":"2024-04-02T13:29:13.091569+0000","last_peered":"2024-04-02T13:29:13.091569+0000","last_clean":"2024-04-02T13:29:13.091569+0000","last_became_active":"2024-04-02T13:29:11.566670+0000","last_became_peered":"2024-04-02T13:29:11.566670+0000","last_unstale":"2024-04-02T13:29:13.091569+0000","last_undegraded":"2024-04-02T13:29:13.091569+0000","last_fullsized":"2024-04-02T13:29:13.091569+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:48:49.409715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555149+0000","last_change":"2024-04-02T13:29:10.560147+0000","last_active":"2024-04-02T13:29:12.555149+0000","last_peered":"2024-04-02T13:29:12.555149+0000","last_clean":"2024-04-02T13:29:12.555149+0000","last_became_active":"2024-04-02T13:29:10.559595+0000","last_became_peered":"2024-04-02T13:29:10.559595+0000","last_unstale":"2024-04-02T13:29:12.555149+0000","last_undegraded":"2024-04-02T13:29:12.555149+0000","last_fullsized":"2024-04-02T13:29:12.555149+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:26:29.859886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091780+0000","last_change":"2024-04-02T13:29:11.563658+0000","last_active":"2024-04-02T13:29:13.091780+0000","last_peered":"2024-04-02T13:29:13.091780+0000","last_clean":"2024-04-02T13:29:13.091780+0000","last_became_active":"2024-04-02T13:29:11.562304+0000","last_became_peered":"2024-04-02T13:29:11.562304+0000","last_unstale":"2024-04-02T13:29:13.091780+0000","last_undegraded":"2024-04-02T13:29:13.091780+0000","last_fullsized":"2024-04-02T13:29:13.091780+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:04:05.591117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"105'3","reported_seq":17,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091797+0000","last_change":"2024-04-02T13:29:10.559984+0000","last_active":"2024-04-02T13:29:13.091797+0000","last_peered":"2024-04-02T13:29:13.091797+0000","last_clean":"2024-04-02T13:29:13.091797+0000","last_became_active":"2024-04-02T13:29:10.559665+0000","last_became_peered":"2024-04-02T13:29:10.559665+0000","last_unstale":"2024-04-02T13:29:13.091797+0000","last_undegraded":"2024-04-02T13:29:13.091797+0000","last_fullsized":"2024-04-02T13:29:13.091797+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:54:51.668583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091890+0000","last_change":"2024-04-02T13:29:11.567784+0000","last_active":"2024-04-02T13:29:13.091890+0000","last_peered":"2024-04-02T13:29:13.091890+0000","last_clean":"2024-04-02T13:29:13.091890+0000","last_became_active":"2024-04-02T13:29:11.567533+0000","last_became_peered":"2024-04-02T13:29:11.567533+0000","last_unstale":"2024-04-02T13:29:13.091890+0000","last_undegraded":"2024-04-02T13:29:13.091890+0000","last_fullsized":"2024-04-02T13:29:13.091890+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:49.582030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"105'6","reported_seq":20,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655370+0000","last_change":"2024-04-02T13:29:10.560138+0000","last_active":"2024-04-02T13:29:12.655370+0000","last_peered":"2024-04-02T13:29:12.655370+0000","last_clean":"2024-04-02T13:29:12.655370+0000","last_became_active":"2024-04-02T13:29:10.559858+0000","last_became_peered":"2024-04-02T13:29:10.559858+0000","last_unstale":"2024-04-02T13:29:12.655370+0000","last_undegraded":"2024-04-02T13:29:12.655370+0000","last_fullsized":"2024-04-02T13:29:12.655370+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:34:04.714892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556625+0000","last_change":"2024-04-02T13:29:11.562690+0000","last_active":"2024-04-02T13:29:12.556625+0000","last_peered":"2024-04-02T13:29:12.556625+0000","last_clean":"2024-04-02T13:29:12.556625+0000","last_became_active":"2024-04-02T13:29:11.562359+0000","last_became_peered":"2024-04-02T13:29:11.562359+0000","last_unstale":"2024-04-02T13:29:12.556625+0000","last_undegraded":"2024-04-02T13:29:12.556625+0000","last_fullsized":"2024-04-02T13:29:12.556625+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:54:37.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091889+0000","last_change":"2024-04-02T13:29:10.555092+0000","last_active":"2024-04-02T13:29:13.091889+0000","last_peered":"2024-04-02T13:29:13.091889+0000","last_clean":"2024-04-02T13:29:13.091889+0000","last_became_active":"2024-04-02T13:29:10.554536+0000","last_became_peered":"2024-04-02T13:29:10.554536+0000","last_unstale":"2024-04-02T13:29:13.091889+0000","last_undegraded":"2024-04-02T13:29:13.091889+0000","last_fullsized":"2024-04-02T13:29:13.091889+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:48:36.435520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655485+0000","last_change":"2024-04-02T13:29:11.566424+0000","last_active":"2024-04-02T13:29:12.655485+0000","last_peered":"2024-04-02T13:29:12.655485+0000","last_clean":"2024-04-02T13:29:12.655485+0000","last_became_active":"2024-04-02T13:29:11.566119+0000","last_became_peered":"2024-04-02T13:29:11.566119+0000","last_unstale":"2024-04-02T13:29:12.655485+0000","last_undegraded":"2024-04-02T13:29:12.655485+0000","last_fullsized":"2024-04-02T13:29:12.655485+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:12:47.100724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552386+0000","last_change":"2024-04-02T13:29:10.549688+0000","last_active":"2024-04-02T13:29:12.552386+0000","last_peered":"2024-04-02T13:29:12.552386+0000","last_clean":"2024-04-02T13:29:12.552386+0000","last_became_active":"2024-04-02T13:29:10.549214+0000","last_became_peered":"2024-04-02T13:29:10.549214+0000","last_unstale":"2024-04-02T13:29:12.552386+0000","last_undegraded":"2024-04-02T13:29:12.552386+0000","last_fullsized":"2024-04-02T13:29:12.552386+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:19.984092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092207+0000","last_change":"2024-04-02T13:29:11.572314+0000","last_active":"2024-04-02T13:29:13.092207+0000","last_peered":"2024-04-02T13:29:13.092207+0000","last_clean":"2024-04-02T13:29:13.092207+0000","last_became_active":"2024-04-02T13:29:11.572069+0000","last_became_peered":"2024-04-02T13:29:11.572069+0000","last_unstale":"2024-04-02T13:29:13.092207+0000","last_undegraded":"2024-04-02T13:29:13.092207+0000","last_fullsized":"2024-04-02T13:29:13.092207+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:23:59.604573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092029+0000","last_change":"2024-04-02T13:29:10.556788+0000","last_active":"2024-04-02T13:29:13.092029+0000","last_peered":"2024-04-02T13:29:13.092029+0000","last_clean":"2024-04-02T13:29:13.092029+0000","last_became_active":"2024-04-02T13:29:10.556482+0000","last_became_peered":"2024-04-02T13:29:10.556482+0000","last_unstale":"2024-04-02T13:29:13.092029+0000","last_undegraded":"2024-04-02T13:29:13.092029+0000","last_fullsized":"2024-04-02T13:29:13.092029+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:55.085324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555499+0000","last_change":"2024-04-02T13:29:11.565050+0000","last_active":"2024-04-02T13:29:12.555499+0000","last_peered":"2024-04-02T13:29:12.555499+0000","last_clean":"2024-04-02T13:29:12.555499+0000","last_became_active":"2024-04-02T13:29:11.564781+0000","last_became_peered":"2024-04-02T13:29:11.564781+0000","last_unstale":"2024-04-02T13:29:12.555499+0000","last_undegraded":"2024-04-02T13:29:12.555499+0000","last_fullsized":"2024-04-02T13:29:12.555499+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:55:08.573393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"105'8","reported_seq":22,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092012+0000","last_change":"2024-04-02T13:29:10.561888+0000","last_active":"2024-04-02T13:29:13.092012+0000","last_peered":"2024-04-02T13:29:13.092012+0000","last_clean":"2024-04-02T13:29:13.092012+0000","last_became_active":"2024-04-02T13:29:10.561599+0000","last_became_peered":"2024-04-02T13:29:10.561599+0000","last_unstale":"2024-04-02T13:29:13.092012+0000","last_undegraded":"2024-04-02T13:29:13.092012+0000","last_fullsized":"2024-04-02T13:29:13.092012+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:17:07.568385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091748+0000","last_change":"2024-04-02T13:29:11.562382+0000","last_active":"2024-04-02T13:29:13.091748+0000","last_peered":"2024-04-02T13:29:13.091748+0000","last_clean":"2024-04-02T13:29:13.091748+0000","last_became_active":"2024-04-02T13:29:11.562133+0000","last_became_peered":"2024-04-02T13:29:11.562133+0000","last_unstale":"2024-04-02T13:29:13.091748+0000","last_undegraded":"2024-04-02T13:29:13.091748+0000","last_fullsized":"2024-04-02T13:29:13.091748+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:13:08.050156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655413+0000","last_change":"2024-04-02T13:29:10.555844+0000","last_active":"2024-04-02T13:29:12.655413+0000","last_peered":"2024-04-02T13:29:12.655413+0000","last_clean":"2024-04-02T13:29:12.655413+0000","last_became_active":"2024-04-02T13:29:10.555491+0000","last_became_peered":"2024-04-02T13:29:10.555491+0000","last_unstale":"2024-04-02T13:29:12.655413+0000","last_undegraded":"2024-04-02T13:29:12.655413+0000","last_fullsized":"2024-04-02T13:29:12.655413+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:11.502526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655261+0000","last_change":"2024-04-02T13:29:11.572559+0000","last_active":"2024-04-02T13:29:12.655261+0000","last_peered":"2024-04-02T13:29:12.655261+0000","last_clean":"2024-04-02T13:29:12.655261+0000","last_became_active":"2024-04-02T13:29:11.572297+0000","last_became_peered":"2024-04-02T13:29:11.572297+0000","last_unstale":"2024-04-02T13:29:12.655261+0000","last_undegraded":"2024-04-02T13:29:12.655261+0000","last_fullsized":"2024-04-02T13:29:12.655261+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:49.582030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091843+0000","last_change":"2024-04-02T13:29:10.555120+0000","last_active":"2024-04-02T13:29:13.091843+0000","last_peered":"2024-04-02T13:29:13.091843+0000","last_clean":"2024-04-02T13:29:13.091843+0000","last_became_active":"2024-04-02T13:29:10.554785+0000","last_became_peered":"2024-04-02T13:29:10.554785+0000","last_unstale":"2024-04-02T13:29:13.091843+0000","last_undegraded":"2024-04-02T13:29:13.091843+0000","last_fullsized":"2024-04-02T13:29:13.091843+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:48.577696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091733+0000","last_change":"2024-04-02T13:29:11.560685+0000","last_active":"2024-04-02T13:29:13.091733+0000","last_peered":"2024-04-02T13:29:13.091733+0000","last_clean":"2024-04-02T13:29:13.091733+0000","last_became_active":"2024-04-02T13:29:11.560084+0000","last_became_peered":"2024-04-02T13:29:11.560084+0000","last_unstale":"2024-04-02T13:29:13.091733+0000","last_undegraded":"2024-04-02T13:29:13.091733+0000","last_fullsized":"2024-04-02T13:29:13.091733+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:54:37.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556610+0000","last_change":"2024-04-02T13:29:10.552003+0000","last_active":"2024-04-02T13:29:12.556610+0000","last_peered":"2024-04-02T13:29:12.556610+0000","last_clean":"2024-04-02T13:29:12.556610+0000","last_became_active":"2024-04-02T13:29:10.551700+0000","last_became_peered":"2024-04-02T13:29:10.551700+0000","last_unstale":"2024-04-02T13:29:12.556610+0000","last_undegraded":"2024-04-02T13:29:12.556610+0000","last_fullsized":"2024-04-02T13:29:12.556610+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:13:59.952423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555237+0000","last_change":"2024-04-02T13:29:11.562698+0000","last_active":"2024-04-02T13:29:12.555237+0000","last_peered":"2024-04-02T13:29:12.555237+0000","last_clean":"2024-04-02T13:29:12.555237+0000","last_became_active":"2024-04-02T13:29:11.562343+0000","last_became_peered":"2024-04-02T13:29:11.562343+0000","last_unstale":"2024-04-02T13:29:12.555237+0000","last_undegraded":"2024-04-02T13:29:12.555237+0000","last_fullsized":"2024-04-02T13:29:12.555237+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:17.848752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092300+0000","last_change":"2024-04-02T13:29:10.560067+0000","last_active":"2024-04-02T13:29:13.092300+0000","last_peered":"2024-04-02T13:29:13.092300+0000","last_clean":"2024-04-02T13:29:13.092300+0000","last_became_active":"2024-04-02T13:29:10.559665+0000","last_became_peered":"2024-04-02T13:29:10.559665+0000","last_unstale":"2024-04-02T13:29:13.092300+0000","last_undegraded":"2024-04-02T13:29:13.092300+0000","last_fullsized":"2024-04-02T13:29:13.092300+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:04.764639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555399+0000","last_change":"2024-04-02T13:29:11.558019+0000","last_active":"2024-04-02T13:29:12.555399+0000","last_peered":"2024-04-02T13:29:12.555399+0000","last_clean":"2024-04-02T13:29:12.555399+0000","last_became_active":"2024-04-02T13:29:11.557706+0000","last_became_peered":"2024-04-02T13:29:11.557706+0000","last_unstale":"2024-04-02T13:29:12.555399+0000","last_undegraded":"2024-04-02T13:29:12.555399+0000","last_fullsized":"2024-04-02T13:29:12.555399+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:54:54.418033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091622+0000","last_change":"2024-04-02T13:29:10.550724+0000","last_active":"2024-04-02T13:29:13.091622+0000","last_peered":"2024-04-02T13:29:13.091622+0000","last_clean":"2024-04-02T13:29:13.091622+0000","last_became_active":"2024-04-02T13:29:10.550452+0000","last_became_peered":"2024-04-02T13:29:10.550452+0000","last_unstale":"2024-04-02T13:29:13.091622+0000","last_undegraded":"2024-04-02T13:29:13.091622+0000","last_fullsized":"2024-04-02T13:29:13.091622+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:38:39.890323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552628+0000","last_change":"2024-04-02T13:29:11.569238+0000","last_active":"2024-04-02T13:29:12.552628+0000","last_peered":"2024-04-02T13:29:12.552628+0000","last_clean":"2024-04-02T13:29:12.552628+0000","last_became_active":"2024-04-02T13:29:11.568966+0000","last_became_peered":"2024-04-02T13:29:11.568966+0000","last_unstale":"2024-04-02T13:29:12.552628+0000","last_undegraded":"2024-04-02T13:29:12.552628+0000","last_fullsized":"2024-04-02T13:29:12.552628+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:26:30.864220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655168+0000","last_change":"2024-04-02T13:29:10.549120+0000","last_active":"2024-04-02T13:29:12.655168+0000","last_peered":"2024-04-02T13:29:12.655168+0000","last_clean":"2024-04-02T13:29:12.655168+0000","last_became_active":"2024-04-02T13:29:10.548847+0000","last_became_peered":"2024-04-02T13:29:10.548847+0000","last_unstale":"2024-04-02T13:29:12.655168+0000","last_undegraded":"2024-04-02T13:29:12.655168+0000","last_fullsized":"2024-04-02T13:29:12.655168+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:06:00.732939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092285+0000","last_change":"2024-04-02T13:29:11.569888+0000","last_active":"2024-04-02T13:29:13.092285+0000","last_peered":"2024-04-02T13:29:13.092285+0000","last_clean":"2024-04-02T13:29:13.092285+0000","last_became_active":"2024-04-02T13:29:11.569621+0000","last_became_peered":"2024-04-02T13:29:11.569621+0000","last_unstale":"2024-04-02T13:29:13.092285+0000","last_undegraded":"2024-04-02T13:29:13.092285+0000","last_fullsized":"2024-04-02T13:29:13.092285+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:57.961338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091507+0000","last_change":"2024-04-02T13:29:10.552735+0000","last_active":"2024-04-02T13:29:13.091507+0000","last_peered":"2024-04-02T13:29:13.091507+0000","last_clean":"2024-04-02T13:29:13.091507+0000","last_became_active":"2024-04-02T13:29:10.552465+0000","last_became_peered":"2024-04-02T13:29:10.552465+0000","last_unstale":"2024-04-02T13:29:13.091507+0000","last_undegraded":"2024-04-02T13:29:13.091507+0000","last_fullsized":"2024-04-02T13:29:13.091507+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:20.296695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.555178+0000","last_change":"2024-04-02T13:29:11.560581+0000","last_active":"2024-04-02T13:29:12.555178+0000","last_peered":"2024-04-02T13:29:12.555178+0000","last_clean":"2024-04-02T13:29:12.555178+0000","last_became_active":"2024-04-02T13:29:11.560299+0000","last_became_peered":"2024-04-02T13:29:11.560299+0000","last_unstale":"2024-04-02T13:29:12.555178+0000","last_undegraded":"2024-04-02T13:29:12.555178+0000","last_fullsized":"2024-04-02T13:29:12.555178+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:31:57.351494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091760+0000","last_change":"2024-04-02T13:29:10.557197+0000","last_active":"2024-04-02T13:29:13.091760+0000","last_peered":"2024-04-02T13:29:13.091760+0000","last_clean":"2024-04-02T13:29:13.091760+0000","last_became_active":"2024-04-02T13:29:10.554163+0000","last_became_peered":"2024-04-02T13:29:10.554163+0000","last_unstale":"2024-04-02T13:29:13.091760+0000","last_undegraded":"2024-04-02T13:29:13.091760+0000","last_fullsized":"2024-04-02T13:29:13.091760+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:34:04.714892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091794+0000","last_change":"2024-04-02T13:29:11.568696+0000","last_active":"2024-04-02T13:29:13.091794+0000","last_peered":"2024-04-02T13:29:13.091794+0000","last_clean":"2024-04-02T13:29:13.091794+0000","last_became_active":"2024-04-02T13:29:11.567935+0000","last_became_peered":"2024-04-02T13:29:11.567935+0000","last_unstale":"2024-04-02T13:29:13.091794+0000","last_undegraded":"2024-04-02T13:29:13.091794+0000","last_fullsized":"2024-04-02T13:29:13.091794+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:41.907332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"106'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.577646+0000","last_change":"2024-04-02T13:29:10.559371+0000","last_active":"2024-04-02T13:29:12.577646+0000","last_peered":"2024-04-02T13:29:12.577646+0000","last_clean":"2024-04-02T13:29:12.577646+0000","last_became_active":"2024-04-02T13:29:10.559051+0000","last_became_peered":"2024-04-02T13:29:10.559051+0000","last_unstale":"2024-04-02T13:29:12.577646+0000","last_undegraded":"2024-04-02T13:29:12.577646+0000","last_fullsized":"2024-04-02T13:29:12.577646+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:59:40.809465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.552396+0000","last_change":"2024-04-02T13:29:11.561544+0000","last_active":"2024-04-02T13:29:12.552396+0000","last_peered":"2024-04-02T13:29:12.552396+0000","last_clean":"2024-04-02T13:29:12.552396+0000","last_became_active":"2024-04-02T13:29:11.561274+0000","last_became_peered":"2024-04-02T13:29:11.561274+0000","last_unstale":"2024-04-02T13:29:12.552396+0000","last_undegraded":"2024-04-02T13:29:12.552396+0000","last_fullsized":"2024-04-02T13:29:12.552396+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:13:24.816639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655459+0000","last_change":"2024-04-02T13:29:10.549906+0000","last_active":"2024-04-02T13:29:12.655459+0000","last_peered":"2024-04-02T13:29:12.655459+0000","last_clean":"2024-04-02T13:29:12.655459+0000","last_became_active":"2024-04-02T13:29:10.549622+0000","last_became_peered":"2024-04-02T13:29:10.549622+0000","last_unstale":"2024-04-02T13:29:12.655459+0000","last_undegraded":"2024-04-02T13:29:12.655459+0000","last_fullsized":"2024-04-02T13:29:12.655459+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:01:19.156677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556630+0000","last_change":"2024-04-02T13:29:11.561639+0000","last_active":"2024-04-02T13:29:12.556630+0000","last_peered":"2024-04-02T13:29:12.556630+0000","last_clean":"2024-04-02T13:29:12.556630+0000","last_became_active":"2024-04-02T13:29:11.561357+0000","last_became_peered":"2024-04-02T13:29:11.561357+0000","last_unstale":"2024-04-02T13:29:12.556630+0000","last_undegraded":"2024-04-02T13:29:12.556630+0000","last_fullsized":"2024-04-02T13:29:12.556630+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:07:12.001409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092022+0000","last_change":"2024-04-02T13:29:10.556359+0000","last_active":"2024-04-02T13:29:13.092022+0000","last_peered":"2024-04-02T13:29:13.092022+0000","last_clean":"2024-04-02T13:29:13.092022+0000","last_became_active":"2024-04-02T13:29:10.555656+0000","last_became_peered":"2024-04-02T13:29:10.555656+0000","last_unstale":"2024-04-02T13:29:13.092022+0000","last_undegraded":"2024-04-02T13:29:13.092022+0000","last_fullsized":"2024-04-02T13:29:13.092022+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:11.502526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092216+0000","last_change":"2024-04-02T13:29:11.570565+0000","last_active":"2024-04-02T13:29:13.092216+0000","last_peered":"2024-04-02T13:29:13.092216+0000","last_clean":"2024-04-02T13:29:13.092216+0000","last_became_active":"2024-04-02T13:29:11.570292+0000","last_became_peered":"2024-04-02T13:29:11.570292+0000","last_unstale":"2024-04-02T13:29:13.092216+0000","last_undegraded":"2024-04-02T13:29:13.092216+0000","last_fullsized":"2024-04-02T13:29:13.092216+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:47:27.598123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092242+0000","last_change":"2024-04-02T13:29:10.554912+0000","last_active":"2024-04-02T13:29:13.092242+0000","last_peered":"2024-04-02T13:29:13.092242+0000","last_clean":"2024-04-02T13:29:13.092242+0000","last_became_active":"2024-04-02T13:29:10.554610+0000","last_became_peered":"2024-04-02T13:29:10.554610+0000","last_unstale":"2024-04-02T13:29:13.092242+0000","last_undegraded":"2024-04-02T13:29:13.092242+0000","last_fullsized":"2024-04-02T13:29:13.092242+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:11.502526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092226+0000","last_change":"2024-04-02T13:29:11.575099+0000","last_active":"2024-04-02T13:29:13.092226+0000","last_peered":"2024-04-02T13:29:13.092226+0000","last_clean":"2024-04-02T13:29:13.092226+0000","last_became_active":"2024-04-02T13:29:11.574791+0000","last_became_peered":"2024-04-02T13:29:11.574791+0000","last_unstale":"2024-04-02T13:29:13.092226+0000","last_undegraded":"2024-04-02T13:29:13.092226+0000","last_fullsized":"2024-04-02T13:29:13.092226+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:46:36.295285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091877+0000","last_change":"2024-04-02T13:29:10.555944+0000","last_active":"2024-04-02T13:29:13.091877+0000","last_peered":"2024-04-02T13:29:13.091877+0000","last_clean":"2024-04-02T13:29:13.091877+0000","last_became_active":"2024-04-02T13:29:10.555666+0000","last_became_peered":"2024-04-02T13:29:10.555666+0000","last_unstale":"2024-04-02T13:29:13.091877+0000","last_undegraded":"2024-04-02T13:29:13.091877+0000","last_fullsized":"2024-04-02T13:29:13.091877+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:26.195781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091875+0000","last_change":"2024-04-02T13:29:11.572201+0000","last_active":"2024-04-02T13:29:13.091875+0000","last_peered":"2024-04-02T13:29:13.091875+0000","last_clean":"2024-04-02T13:29:13.091875+0000","last_became_active":"2024-04-02T13:29:11.571910+0000","last_became_peered":"2024-04-02T13:29:11.571910+0000","last_unstale":"2024-04-02T13:29:13.091875+0000","last_undegraded":"2024-04-02T13:29:13.091875+0000","last_fullsized":"2024-04-02T13:29:13.091875+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:37.386743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.655329+0000","last_change":"2024-04-02T13:29:10.549745+0000","last_active":"2024-04-02T13:29:12.655329+0000","last_peered":"2024-04-02T13:29:12.655329+0000","last_clean":"2024-04-02T13:29:12.655329+0000","last_became_active":"2024-04-02T13:29:10.549480+0000","last_became_peered":"2024-04-02T13:29:10.549480+0000","last_unstale":"2024-04-02T13:29:12.655329+0000","last_undegraded":"2024-04-02T13:29:12.655329+0000","last_fullsized":"2024-04-02T13:29:12.655329+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:15:26.524288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556991+0000","last_change":"2024-04-02T13:29:11.562383+0000","last_active":"2024-04-02T13:29:12.556991+0000","last_peered":"2024-04-02T13:29:12.556991+0000","last_clean":"2024-04-02T13:29:12.556991+0000","last_became_active":"2024-04-02T13:29:11.562098+0000","last_became_peered":"2024-04-02T13:29:11.562098+0000","last_unstale":"2024-04-02T13:29:12.556991+0000","last_undegraded":"2024-04-02T13:29:12.556991+0000","last_fullsized":"2024-04-02T13:29:12.556991+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:09:31.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091608+0000","last_change":"2024-04-02T13:29:10.561749+0000","last_active":"2024-04-02T13:29:13.091608+0000","last_peered":"2024-04-02T13:29:13.091608+0000","last_clean":"2024-04-02T13:29:13.091608+0000","last_became_active":"2024-04-02T13:29:10.561518+0000","last_became_peered":"2024-04-02T13:29:10.561518+0000","last_unstale":"2024-04-02T13:29:13.091608+0000","last_undegraded":"2024-04-02T13:29:13.091608+0000","last_fullsized":"2024-04-02T13:29:13.091608+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:19.438340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:12.556416+0000","last_change":"2024-04-02T13:29:11.564242+0000","last_active":"2024-04-02T13:29:12.556416+0000","last_peered":"2024-04-02T13:29:12.556416+0000","last_clean":"2024-04-02T13:29:12.556416+0000","last_became_active":"2024-04-02T13:29:11.563263+0000","last_became_peered":"2024-04-02T13:29:11.563263+0000","last_unstale":"2024-04-02T13:29:12.556416+0000","last_undegraded":"2024-04-02T13:29:12.556416+0000","last_fullsized":"2024-04-02T13:29:12.556416+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:10.531442+0000","last_clean_scrub_stamp":"2024-04-02T13:29:10.531442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:41.907332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.092361+0000","last_change":"2024-04-02T13:29:10.552044+0000","last_active":"2024-04-02T13:29:13.092361+0000","last_peered":"2024-04-02T13:29:13.092361+0000","last_clean":"2024-04-02T13:29:13.092361+0000","last_became_active":"2024-04-02T13:29:10.550405+0000","last_became_peered":"2024-04-02T13:29:10.550405+0000","last_unstale":"2024-04-02T13:29:13.092361+0000","last_undegraded":"2024-04-02T13:29:13.092361+0000","last_fullsized":"2024-04-02T13:29:13.092361+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:29:09.527108+0000","last_clean_scrub_stamp":"2024-04-02T13:29:09.527108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:47:39.094392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":335,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-02T13:29:13.091520+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:29:13.091520+0000","last_peered":"2024-04-02T13:29:13.091520+0000","last_clean":"2024-04-02T13:29:13.091520+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:29:13.091520+0000","last_undegraded":"2024-04-02T13:29:13.091520+0000","last_fullsized":"2024-04-02T13:29:13.091520+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1916928,"data_stored":1201478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574958,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1335296,"data_stored":613166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574959,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1314816,"data_stored":610950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574959,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32480,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339360,"statfs":{"total":96636764160,"available":96603504640,"internally_reserved":0,"allocated":1323008,"data_stored":611042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574959,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1318912,"data_stored":611532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574958,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1318912,"data_stored":610984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":610984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574959,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33064,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338776,"statfs":{"total":96636764160,"available":96602906624,"internally_reserved":0,"allocated":1925120,"data_stored":1203020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:29:15.930 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T13:29:15.931 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T13:29:16.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.072+0000 7f9c05231640 1 Processor -- start 2024-04-02T13:29:16.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.072+0000 7f9c05231640 1 -- start start 2024-04-02T13:29:16.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.073+0000 7f9c05231640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c000d0a30 0x7f9c0006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:16.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.073+0000 7f9c05231640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c000d0000 con 0x7f9c000d2040 2024-04-02T13:29:16.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.073+0000 7f9c05231640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c0006c1d0 con 0x7f9c000d0e10 2024-04-02T13:29:16.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.073+0000 7f9c05231640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9c0006cb00 con 0x7f9c000d0a30 2024-04-02T13:29:16.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.073+0000 7f9bff59e640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f9c000d2040 0x7f9c000d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40448/0 (socket says 172.21.15.44:40448) 2024-04-02T13:29:16.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.073+0000 7f9bff59e640 1 -- 172.21.15.44:0/110838136 learned_addr learned my addr 172.21.15.44:0/110838136 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:16.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.073+0000 7f9bfed9d640 1 --2- 172.21.15.44:0/110838136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c000d0a30 0x7f9c0006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:16.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.073+0000 7f9bfdd9b640 1 -- 172.21.15.44:0/110838136 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 785171149 0 0) 0x7f9c000d0000 con 0x7f9c000d2040 2024-04-02T13:29:16.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.073+0000 7f9bfdd9b640 1 -- 172.21.15.44:0/110838136 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bec003520 con 0x7f9c000d2040 2024-04-02T13:29:16.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.074+0000 7f9bfdd9b640 1 -- 172.21.15.44:0/110838136 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1372203722 0 0) 0x7f9c0006c1d0 con 0x7f9c000d0e10 2024-04-02T13:29:16.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.074+0000 7f9bfed9d640 1 -- 172.21.15.44:0/110838136 >> v1:172.21.15.69:6789/0 conn(0x7f9c000d0e10 legacy=0x7f9c000cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.074+0000 7f9bfed9d640 1 -- 172.21.15.44:0/110838136 >> v1:172.21.15.44:6789/0 conn(0x7f9c000d2040 legacy=0x7f9c000d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.074+0000 7f9bfed9d640 1 -- 172.21.15.44:0/110838136 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c0006c4b0 con 0x7f9c000d0a30 2024-04-02T13:29:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.074+0000 7f9bfed9d640 1 --2- 172.21.15.44:0/110838136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c000d0a30 0x7f9c0006cf90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009dc0 tx=0x7f9bf002bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d58198d908d79fd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.075+0000 7f9bfdd9b640 1 -- 172.21.15.44:0/110838136 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0039070 con 0x7f9c000d0a30 2024-04-02T13:29:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.075+0000 7f9bfdd9b640 1 -- 172.21.15.44:0/110838136 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf002c7b0 con 0x7f9c000d0a30 2024-04-02T13:29:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.075+0000 7f9bfdd9b640 1 -- 172.21.15.44:0/110838136 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf80015f0 con 0x7f9c000d0a30 2024-04-02T13:29:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.075+0000 7f9c05231640 1 -- 172.21.15.44:0/110838136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c000d0a30 msgr2=0x7f9c0006cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.075+0000 7f9c05231640 1 --2- 172.21.15.44:0/110838136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c000d0a30 0x7f9c0006cf90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009dc0 tx=0x7f9bf002bc80 comp rx=0 tx=0).stop 2024-04-02T13:29:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.075+0000 7f9c05231640 1 -- 172.21.15.44:0/110838136 shutdown_connections 2024-04-02T13:29:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.075+0000 7f9c05231640 1 --2- 172.21.15.44:0/110838136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c000d0a30 0x7f9c0006cf90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.075+0000 7f9c05231640 1 -- 172.21.15.44:0/110838136 >> 172.21.15.44:0/110838136 conn(0x7f9c0009d470 msgr2=0x7f9c000d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.075+0000 7f9c05231640 1 -- 172.21.15.44:0/110838136 shutdown_connections 2024-04-02T13:29:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.075+0000 7f9c05231640 1 -- 172.21.15.44:0/110838136 wait complete. 2024-04-02T13:29:16.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.077+0000 7f9c05231640 1 Processor -- start 2024-04-02T13:29:16.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.077+0000 7f9c05231640 1 -- start start 2024-04-02T13:29:16.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.077+0000 7f9c05231640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c000d0a30 0x7f9c00159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:16.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.077+0000 7f9c05231640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c000d0e10 0x7f9c000b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:16.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.077+0000 7f9c05231640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c000d2040 0x7f9c0015b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:16.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.077+0000 7f9c05231640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0015e6f0 con 0x7f9c000d0e10 2024-04-02T13:29:16.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.077+0000 7f9c05231640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9c000d1270 con 0x7f9c000d2040 2024-04-02T13:29:16.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.077+0000 7f9c05231640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9c0015e870 con 0x7f9c000d0a30 2024-04-02T13:29:16.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.077+0000 7f9bfe59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c000d0e10 0x7f9c000b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:16.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bfe59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c000d0e10 0x7f9c000b63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49096/0 (socket says 172.21.15.44:49096) 2024-04-02T13:29:16.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bfe59c640 1 -- 172.21.15.44:0/1472637562 learned_addr learned my addr 172.21.15.44:0/1472637562 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:16.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bfed9d640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c000d0a30 0x7f9c00159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:16.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bff59e640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c000d2040 0x7f9c0015b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:16.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bfe59c640 1 -- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c000d0a30 msgr2=0x7f9c00159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bfe59c640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c000d0a30 0x7f9c00159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bfe59c640 1 -- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c000d2040 msgr2=0x7f9c0015b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bfe59c640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c000d2040 0x7f9c0015b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bfe59c640 1 -- 172.21.15.44:0/1472637562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c000b40a0 con 0x7f9c000d0e10 2024-04-02T13:29:16.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bff59e640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c000d2040 0x7f9c0015b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:16.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bfe59c640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c000d0e10 0x7f9c000b63d0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f9be800c520 tx=0x7f9be800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:16.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bf77fe640 1 -- 172.21.15.44:0/1472637562 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be8017070 con 0x7f9c000d0e10 2024-04-02T13:29:16.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bf77fe640 1 -- 172.21.15.44:0/1472637562 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9be80035f0 con 0x7f9c000d0e10 2024-04-02T13:29:16.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9c05231640 1 -- 172.21.15.44:0/1472637562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c000b4380 con 0x7f9c000d0e10 2024-04-02T13:29:16.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9bf77fe640 1 -- 172.21.15.44:0/1472637562 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be8003a00 con 0x7f9c000d0e10 2024-04-02T13:29:16.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.078+0000 7f9c05231640 1 -- 172.21.15.44:0/1472637562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c000b4940 con 0x7f9c000d0e10 2024-04-02T13:29:16.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.079+0000 7f9c05231640 1 -- 172.21.15.44:0/1472637562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c00073730 con 0x7f9c000d0e10 2024-04-02T13:29:16.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.080+0000 7f9bf77fe640 1 -- 172.21.15.44:0/1472637562 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9be801a070 con 0x7f9c000d0e10 2024-04-02T13:29:16.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.080+0000 7f9bf77fe640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9be006c8f0 0x7f9be006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:16.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.080+0000 7f9bf77fe640 1 -- 172.21.15.44:0/1472637562 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f9be808b0a0 con 0x7f9c000d0e10 2024-04-02T13:29:16.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.080+0000 7f9bfed9d640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9be006c8f0 0x7f9be006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:16.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.081+0000 7f9bfed9d640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9be006c8f0 0x7f9be006edb0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0036000 tx=0x7f9bf0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:16.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.084+0000 7f9bf77fe640 1 -- 172.21.15.44:0/1472637562 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9c00073730 con 0x7f9c000d0e10 2024-04-02T13:29:16.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.337+0000 7f9c05231640 1 -- 172.21.15.44:0/1472637562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f9c000684f0 con 0x7f9c000d0e10 2024-04-02T13:29:16.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.339+0000 7f9bf77fe640 1 -- 172.21.15.44:0/1472637562 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7f9be80593d0 con 0x7f9c000d0e10 2024-04-02T13:29:16.343 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-04-02T13:29:16.343 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A== 2024-04-02T13:29:16.343 INFO:teuthology.orchestra.run.smithi044.stdout: caps mds = "allow" 2024-04-02T13:29:16.343 INFO:teuthology.orchestra.run.smithi044.stdout: caps mon = "allow r" 2024-04-02T13:29:16.343 INFO:teuthology.orchestra.run.smithi044.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-02T13:29:16.343 INFO:teuthology.orchestra.run.smithi044.stderr:updated caps for client.0 2024-04-02T13:29:16.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.341+0000 7f9c05231640 1 -- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9be006c8f0 msgr2=0x7f9be006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.341+0000 7f9c05231640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9be006c8f0 0x7f9be006edb0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0036000 tx=0x7f9bf0030000 comp rx=0 tx=0).stop 2024-04-02T13:29:16.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.341+0000 7f9c05231640 1 -- 172.21.15.44:0/1472637562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c000d0e10 msgr2=0x7f9c000b63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.341+0000 7f9c05231640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c000d0e10 0x7f9c000b63d0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f9be800c520 tx=0x7f9be800c9e0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.341+0000 7f9c05231640 1 -- 172.21.15.44:0/1472637562 shutdown_connections 2024-04-02T13:29:16.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.341+0000 7f9c05231640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9be006c8f0 0x7f9be006edb0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.341+0000 7f9c05231640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c000d2040 0x7f9c0015b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.341+0000 7f9c05231640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c000d0e10 0x7f9c000b63d0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.341+0000 7f9c05231640 1 --2- 172.21.15.44:0/1472637562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c000d0a30 0x7f9c00159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.341+0000 7f9c05231640 1 -- 172.21.15.44:0/1472637562 >> 172.21.15.44:0/1472637562 conn(0x7f9c0009d470 msgr2=0x7f9c000b1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:16.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.341+0000 7f9c05231640 1 -- 172.21.15.44:0/1472637562 shutdown_connections 2024-04-02T13:29:16.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.341+0000 7f9c05231640 1 -- 172.21.15.44:0/1472637562 wait complete. 2024-04-02T13:29:16.356 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:29:16.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.454+0000 7ff23f07b640 1 Processor -- start 2024-04-02T13:29:16.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.454+0000 7ff23f07b640 1 -- start start 2024-04-02T13:29:16.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff23f07b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2380ce2e0 0x7ff238165820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:16.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff23f07b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2380b3160 con 0x7ff23815aa00 2024-04-02T13:29:16.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff23f07b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff23815ade0 con 0x7ff2380b2d00 2024-04-02T13:29:16.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff23f07b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff2380721a0 con 0x7ff2380ce2e0 2024-04-02T13:29:16.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff23ce17640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff23815aa00 0x7ff238072bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40456/0 (socket says 172.21.15.44:40456) 2024-04-02T13:29:16.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff23ce17640 1 -- 172.21.15.44:0/115561952 learned_addr learned my addr 172.21.15.44:0/115561952 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:16.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff23d618640 1 --2- 172.21.15.44:0/115561952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2380ce2e0 0x7ff238165820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1606616475 0 0) 0x7ff2380b3160 con 0x7ff23815aa00 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff224003520 con 0x7ff23815aa00 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4289152317 0 0) 0x7ff224003520 con 0x7ff23815aa00 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2380ce2e0 msgr2=0x7ff238165820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 --2- 172.21.15.44:0/115561952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2380ce2e0 0x7ff238165820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 >> v1:172.21.15.69:6789/0 conn(0x7ff2380b2d00 legacy=0x7ff2380cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2380af1b0 con 0x7ff23815aa00 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7ff220001670 con 0x7ff23815aa00 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 >> v1:172.21.15.44:6789/0 conn(0x7ff23815aa00 legacy=0x7ff238072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 --2- 172.21.15.44:0/115561952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2380ce2e0 0x7ff2240039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 --2- 172.21.15.44:0/115561952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff224003f20 0x7ff224004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 --2- 172.21.15.44:0/115561952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff224004d40 0x7ff2240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff2380721a0 con 0x7ff224003f20 2024-04-02T13:29:16.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff22400bad0 con 0x7ff224004d40 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff22400bcd0 con 0x7ff2380ce2e0 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.455+0000 7ff237fff640 1 --2- 172.21.15.44:0/115561952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff224003f20 0x7ff224004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.456+0000 7ff23ce17640 1 --2- 172.21.15.44:0/115561952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2380ce2e0 0x7ff2240039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.456+0000 7ff23d618640 1 --2- 172.21.15.44:0/115561952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff224004d40 0x7ff2240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.456+0000 7ff237fff640 1 -- 172.21.15.44:0/115561952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2380ce2e0 msgr2=0x7ff2240039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.456+0000 7ff237fff640 1 --2- 172.21.15.44:0/115561952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2380ce2e0 0x7ff2240039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.456+0000 7ff237fff640 1 -- 172.21.15.44:0/115561952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff224004d40 msgr2=0x7ff2240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.456+0000 7ff237fff640 1 --2- 172.21.15.44:0/115561952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff224004d40 0x7ff2240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.456+0000 7ff237fff640 1 -- 172.21.15.44:0/115561952 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff224003520 con 0x7ff224003f20 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.457+0000 7ff237fff640 1 --2- 172.21.15.44:0/115561952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff224003f20 0x7ff224004800 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7ff228001860 tx=0x7ff224005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edea83dbc746b2e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.457+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228038070 con 0x7ff224003f20 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.457+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff228033410 con 0x7ff224003f20 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.457+0000 7ff2377fe640 1 -- 172.21.15.44:0/115561952 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228001430 con 0x7ff224003f20 2024-04-02T13:29:16.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.457+0000 7ff23f07b640 1 -- 172.21.15.44:0/115561952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff224003f20 msgr2=0x7ff224004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.457+0000 7ff23f07b640 1 --2- 172.21.15.44:0/115561952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff224003f20 0x7ff224004800 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7ff228001860 tx=0x7ff224005830 comp rx=0 tx=0).stop 2024-04-02T13:29:16.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.457+0000 7ff23d618640 1 -- 172.21.15.44:0/115561952 reap_dead start 2024-04-02T13:29:16.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.457+0000 7ff23f07b640 1 -- 172.21.15.44:0/115561952 shutdown_connections 2024-04-02T13:29:16.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.457+0000 7ff23f07b640 1 -- 172.21.15.44:0/115561952 >> 172.21.15.44:0/115561952 conn(0x7ff2380cfcd0 msgr2=0x7ff2380d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:16.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.457+0000 7ff23f07b640 1 -- 172.21.15.44:0/115561952 shutdown_connections 2024-04-02T13:29:16.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.457+0000 7ff23f07b640 1 -- 172.21.15.44:0/115561952 wait complete. 2024-04-02T13:29:16.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.458+0000 7ff23f07b640 1 Processor -- start 2024-04-02T13:29:16.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.458+0000 7ff23f07b640 1 -- start start 2024-04-02T13:29:16.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.459+0000 7ff23f07b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff224003f20 0x7ff23809aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:16.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.459+0000 7ff23f07b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2381e4380 0x7ff23806e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:16.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.459+0000 7ff23f07b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2381e47e0 0x7ff238146320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:16.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.459+0000 7ff23f07b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff2380726d0 con 0x7ff2381e47e0 2024-04-02T13:29:16.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.459+0000 7ff23f07b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff2380723f0 con 0x7ff2381e4380 2024-04-02T13:29:16.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.459+0000 7ff23f07b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff2380cc750 con 0x7ff224003f20 2024-04-02T13:29:16.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.459+0000 7ff23d618640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2381e47e0 0x7ff238146320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:16.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.459+0000 7ff23d618640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2381e47e0 0x7ff238146320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49112/0 (socket says 172.21.15.44:49112) 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.459+0000 7ff23d618640 1 -- 172.21.15.44:0/325510796 learned_addr learned my addr 172.21.15.44:0/325510796 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff23ce17640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff224003f20 0x7ff23809aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff23d618640 1 -- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff224003f20 msgr2=0x7ff23809aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff237fff640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2381e4380 0x7ff23806e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff23d618640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff224003f20 0x7ff23809aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff23d618640 1 -- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2381e4380 msgr2=0x7ff23806e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff23d618640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2381e4380 0x7ff23806e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff23d618640 1 -- 172.21.15.44:0/325510796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff238150b30 con 0x7ff2381e47e0 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff23ce17640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff224003f20 0x7ff23809aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff23d618640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2381e47e0 0x7ff238146320 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7ff238073a80 tx=0x7ff23000af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff2357fa640 1 -- 172.21.15.44:0/325510796 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff230003da0 con 0x7ff2381e47e0 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff2357fa640 1 -- 172.21.15.44:0/325510796 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff23001b070 con 0x7ff2381e47e0 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff23f07b640 1 -- 172.21.15.44:0/325510796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2381f3d50 con 0x7ff2381e47e0 2024-04-02T13:29:16.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff2357fa640 1 -- 172.21.15.44:0/325510796 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff230007b00 con 0x7ff2381e47e0 2024-04-02T13:29:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.460+0000 7ff23f07b640 1 -- 172.21.15.44:0/325510796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2381f4310 con 0x7ff2381e47e0 2024-04-02T13:29:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.461+0000 7ff23f07b640 1 -- 172.21.15.44:0/325510796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff238074f20 con 0x7ff2381e47e0 2024-04-02T13:29:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.461+0000 7ff2357fa640 1 -- 172.21.15.44:0/325510796 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff230007ca0 con 0x7ff2381e47e0 2024-04-02T13:29:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.462+0000 7ff2357fa640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff21806c920 0x7ff21806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.462+0000 7ff2357fa640 1 -- 172.21.15.44:0/325510796 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7ff23008c6e0 con 0x7ff2381e47e0 2024-04-02T13:29:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.462+0000 7ff23ce17640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff21806c920 0x7ff21806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:16.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.463+0000 7ff23ce17640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff21806c920 0x7ff21806ede0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff220007bf0 tx=0x7ff220009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:16.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.466+0000 7ff2357fa640 1 -- 172.21.15.44:0/325510796 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff238150b30 con 0x7ff2381e47e0 2024-04-02T13:29:16.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.691+0000 7ff23f07b640 1 -- 172.21.15.44:0/325510796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2380d09a0 con 0x7ff2381e47e0 2024-04-02T13:29:16.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.692+0000 7ff2357fa640 1 -- 172.21.15.44:0/325510796 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5183 (secure 0 0 0) 0x7ff230011050 con 0x7ff2381e47e0 2024-04-02T13:29:16.696 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:16.696 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7696,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2715855087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2715855087},{"type":"v1","addr":"172.21.15.69:6836","nonce":2715855087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7699,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3850680553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3850680553},{"type":"v1","addr":"172.21.15.69:6837","nonce":3850680553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/1330712383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1330712383},{"type":"v1","addr":"172.21.15.44:6835","nonce":1330712383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:29:10.907920+0000","modified":"2024-04-02T13:29:13.930909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8760":{"gid":8760,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6834/3259420283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":3259420283},{"type":"v1","addr":"172.21.15.44:6834","nonce":3259420283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8760,"qdb_cluster":[8760]},"id":7}]} 2024-04-02T13:29:16.696 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 82 2024-04-02T13:29:16.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.694+0000 7ff23f07b640 1 -- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff21806c920 msgr2=0x7ff21806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.694+0000 7ff23f07b640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff21806c920 0x7ff21806ede0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff220007bf0 tx=0x7ff220009040 comp rx=0 tx=0).stop 2024-04-02T13:29:16.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.694+0000 7ff23f07b640 1 -- 172.21.15.44:0/325510796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2381e47e0 msgr2=0x7ff238146320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:16.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.694+0000 7ff23f07b640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2381e47e0 0x7ff238146320 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7ff238073a80 tx=0x7ff23000af60 comp rx=0 tx=0).stop 2024-04-02T13:29:16.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.694+0000 7ff23f07b640 1 -- 172.21.15.44:0/325510796 shutdown_connections 2024-04-02T13:29:16.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.694+0000 7ff23f07b640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff21806c920 0x7ff21806ede0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.694+0000 7ff23f07b640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2381e47e0 0x7ff238146320 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.694+0000 7ff23f07b640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2381e4380 0x7ff23806e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.694+0000 7ff23f07b640 1 --2- 172.21.15.44:0/325510796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff224003f20 0x7ff23809aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:16.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.694+0000 7ff23f07b640 1 -- 172.21.15.44:0/325510796 >> 172.21.15.44:0/325510796 conn(0x7ff2380cfcd0 msgr2=0x7ff2380b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:16.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.694+0000 7ff23f07b640 1 -- 172.21.15.44:0/325510796 shutdown_connections 2024-04-02T13:29:16.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:16.694+0000 7ff23f07b640 1 -- 172.21.15.44:0/325510796 wait complete. 2024-04-02T13:29:17.111 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:29:10.907920+0000', 'modified': '2024-04-02T13:29:13.930909+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8760}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8760': {'gid': 8760, 'name': 'a', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.44:6834/3259420283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6833', 'nonce': 3259420283}, {'type': 'v1', 'addr': '172.21.15.44:6834', 'nonce': 3259420283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8760, 'qdb_cluster': [8760]} 2024-04-02T13:29:17.111 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:29:17.111 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T13:29:17.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.213+0000 7f7b2182a640 1 Processor -- start 2024-04-02T13:29:17.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.213+0000 7f7b2182a640 1 -- start start 2024-04-02T13:29:17.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.214+0000 7f7b2182a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7b1c0b2d00 0x7f7b1c0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.214+0000 7f7b2182a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b1c0b3160 con 0x7f7b1c0ce110 2024-04-02T13:29:17.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.214+0000 7f7b2182a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b1c15ade0 con 0x7f7b1c15aa00 2024-04-02T13:29:17.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.214+0000 7f7b2182a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7b1c0af1b0 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.214+0000 7f7b1a7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7b1c0b2d00 0x7f7b1c0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.214+0000 7f7b1b7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7b1c0ce110 0x7f7b1c071190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40462/0 (socket says 172.21.15.44:40462) 2024-04-02T13:29:17.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.214+0000 7f7b1b7fe640 1 -- 172.21.15.44:0/555228013 learned_addr learned my addr 172.21.15.44:0/555228013 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:17.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.214+0000 7f7b1a7fc640 1 -- 172.21.15.44:0/555228013 >> v1:172.21.15.69:6789/0 conn(0x7f7b1c15aa00 legacy=0x7f7b1c072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:17.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.214+0000 7f7b1a7fc640 1 -- 172.21.15.44:0/555228013 >> v1:172.21.15.44:6789/0 conn(0x7f7b1c0ce110 legacy=0x7f7b1c071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.214+0000 7f7b1a7fc640 1 -- 172.21.15.44:0/555228013 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1c0af490 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.214+0000 7f7b19ffb640 1 -- 172.21.15.44:0/555228013 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1555597826 0 0) 0x7f7b1c0b3160 con 0x7f7b1c0ce110 2024-04-02T13:29:17.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.215+0000 7f7b1a7fc640 1 --2- 172.21.15.44:0/555228013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b1c0b2d00 0x7f7b1c0cdbd0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400ba40 tx=0x7f7b0402ddf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2b615f175ec520 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:17.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.215+0000 7f7b19ffb640 1 -- 172.21.15.44:0/555228013 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b04034070 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.215+0000 7f7b19ffb640 1 -- 172.21.15.44:0/555228013 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b0402eba0 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.215+0000 7f7b19ffb640 1 -- 172.21.15.44:0/555228013 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b04038400 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.216+0000 7f7b2182a640 1 -- 172.21.15.44:0/555228013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b1c0b2d00 msgr2=0x7f7b1c0cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.216+0000 7f7b2182a640 1 --2- 172.21.15.44:0/555228013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b1c0b2d00 0x7f7b1c0cdbd0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400ba40 tx=0x7f7b0402ddf0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.216+0000 7f7b2182a640 1 -- 172.21.15.44:0/555228013 shutdown_connections 2024-04-02T13:29:17.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.216+0000 7f7b2182a640 1 --2- 172.21.15.44:0/555228013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b1c0b2d00 0x7f7b1c0cdbd0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.216+0000 7f7b2182a640 1 -- 172.21.15.44:0/555228013 >> 172.21.15.44:0/555228013 conn(0x7f7b1c0cfcd0 msgr2=0x7f7b1c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:17.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.216+0000 7f7b2182a640 1 -- 172.21.15.44:0/555228013 shutdown_connections 2024-04-02T13:29:17.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.216+0000 7f7b2182a640 1 -- 172.21.15.44:0/555228013 wait complete. 2024-04-02T13:29:17.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.217+0000 7f7b2182a640 1 Processor -- start 2024-04-02T13:29:17.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.217+0000 7f7b2182a640 1 -- start start 2024-04-02T13:29:17.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b2182a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b1c0b2d00 0x7f7b1c154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b2182a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b1c0ce110 0x7f7b1c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b2182a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b1c15aa00 0x7f7b1c159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b2182a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c0ce560 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b2182a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c0726d0 con 0x7f7b1c0ce110 2024-04-02T13:29:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b2182a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7b1c0cc750 con 0x7f7b1c15aa00 2024-04-02T13:29:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b1affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b1c0b2d00 0x7f7b1c154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b1affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b1c0b2d00 0x7f7b1c154d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49126/0 (socket says 172.21.15.44:49126) 2024-04-02T13:29:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b1affd640 1 -- 172.21.15.44:0/211569823 learned_addr learned my addr 172.21.15.44:0/211569823 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b1a7fc640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b1c0ce110 0x7f7b1c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b1b7fe640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b1c15aa00 0x7f7b1c159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b1affd640 1 -- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b1c15aa00 msgr2=0x7f7b1c159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b1affd640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b1c15aa00 0x7f7b1c159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b1affd640 1 -- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b1c0ce110 msgr2=0x7f7b1c06e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b1affd640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b1c0ce110 0x7f7b1c06e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b1affd640 1 -- 172.21.15.44:0/211569823 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1c14df70 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.218+0000 7f7b1b7fe640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b1c15aa00 0x7f7b1c159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:17.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.219+0000 7f7b1affd640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b1c0b2d00 0x7f7b1c154d50 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c00ba50 tx=0x7f7b0c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:17.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.219+0000 7f7b13fff640 1 -- 172.21.15.44:0/211569823 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c019070 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.219+0000 7f7b13fff640 1 -- 172.21.15.44:0/211569823 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b0c00cc20 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.219+0000 7f7b2182a640 1 -- 172.21.15.44:0/211569823 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b1c14e210 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.219+0000 7f7b13fff640 1 -- 172.21.15.44:0/211569823 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c014440 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.219+0000 7f7b2182a640 1 -- 172.21.15.44:0/211569823 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b1c14eab0 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.220+0000 7f7b13fff640 1 -- 172.21.15.44:0/211569823 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7b0c01c070 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.221+0000 7f7b13fff640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7afc06c8f0 0x7f7afc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.221+0000 7f7b13fff640 1 -- 172.21.15.44:0/211569823 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f7b0c08baf0 con 0x7f7b1c0b2d00 2024-04-02T13:29:17.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.221+0000 7f7b1a7fc640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7afc06c8f0 0x7f7afc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.221+0000 7f7b1a7fc640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7afc06c8f0 0x7f7afc06edb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7b04039000 tx=0x7f7b04034000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:17.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.222+0000 7f7b24413740 1 Processor -- start 2024-04-02T13:29:17.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.222+0000 7f7b24413740 1 -- start start 2024-04-02T13:29:17.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.222+0000 7f7b24413740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb18b90 0x55a77caebb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.222+0000 7f7b24413740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 0x55a77cb1b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.222+0000 7f7b24413740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77caec070 0x55a77cb1be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.222+0000 7f7b24413740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a77ca55060 con 0x55a77caec070 2024-04-02T13:29:17.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.222+0000 7f7b24413740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a77c9fd650 con 0x55a77cb18b90 2024-04-02T13:29:17.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.222+0000 7f7b24413740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a77ca51010 con 0x55a77cb18f70 2024-04-02T13:29:17.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b1b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77caec070 0x55a77cb1be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b1b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77caec070 0x55a77cb1be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49130/0 (socket says 172.21.15.44:49130) 2024-04-02T13:29:17.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b1b7fe640 1 -- 172.21.15.44:0/1858492852 learned_addr learned my addr 172.21.15.44:0/1858492852 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b1affd640 1 --2- 172.21.15.44:0/1858492852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb18b90 0x55a77caebb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b1a7fc640 1 --2- 172.21.15.44:0/1858492852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 0x55a77cb1b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b1b7fe640 1 -- 172.21.15.44:0/1858492852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 msgr2=0x55a77cb1b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b1b7fe640 1 --2- 172.21.15.44:0/1858492852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 0x55a77cb1b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b1b7fe640 1 -- 172.21.15.44:0/1858492852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb18b90 msgr2=0x55a77caebb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b1b7fe640 1 --2- 172.21.15.44:0/1858492852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb18b90 0x55a77caebb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b1b7fe640 1 -- 172.21.15.44:0/1858492852 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a77c7692d0 con 0x55a77caec070 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b1affd640 1 --2- 172.21.15.44:0/1858492852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb18b90 0x55a77caebb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b1b7fe640 1 --2- 172.21.15.44:0/1858492852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77caec070 0x55a77cb1be60 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7b140074c0 tx=0x7f7b14005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a710c085dd82d82 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b117fa640 1 -- 172.21.15.44:0/1858492852 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b14019070 con 0x55a77caec070 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b117fa640 1 -- 172.21.15.44:0/1858492852 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b14001830 con 0x55a77caec070 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.223+0000 7f7b117fa640 1 -- 172.21.15.44:0/1858492852 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b14007670 con 0x55a77caec070 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.224+0000 7f7b24413740 1 -- 172.21.15.44:0/1858492852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77caec070 msgr2=0x55a77cb1be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.224+0000 7f7b24413740 1 --2- 172.21.15.44:0/1858492852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77caec070 0x55a77cb1be60 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7b140074c0 tx=0x7f7b14005bc0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.224+0000 7f7b24413740 1 -- 172.21.15.44:0/1858492852 shutdown_connections 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.224+0000 7f7b24413740 1 --2- 172.21.15.44:0/1858492852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77caec070 0x55a77cb1be60 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.224+0000 7f7b24413740 1 --2- 172.21.15.44:0/1858492852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 0x55a77cb1b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.224+0000 7f7b24413740 1 --2- 172.21.15.44:0/1858492852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb18b90 0x55a77caebb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.224+0000 7f7b24413740 1 -- 172.21.15.44:0/1858492852 >> 172.21.15.44:0/1858492852 conn(0x55a77caedda0 msgr2=0x55a77cb36920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.224+0000 7f7b24413740 1 -- 172.21.15.44:0/1858492852 shutdown_connections 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.224+0000 7f7b24413740 1 -- 172.21.15.44:0/1858492852 wait complete. 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.225+0000 7f7b24413740 1 -- start start 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.225+0000 7f7b24413740 1 Processor -- start 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.225+0000 7f7b24413740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 0x55a77cb23d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.225+0000 7f7b24413740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 0x55a77cb1e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.225+0000 7f7b24413740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 0x55a77cb0c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.225+0000 7f7b24413740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a77caf0440 con 0x55a77cb18b90 2024-04-02T13:29:17.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.225+0000 7f7b24413740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a77ca10a10 con 0x55a77cb1ed20 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.225+0000 7f7b24413740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a77caf2c80 con 0x55a77cb18f70 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.225+0000 7f7b24413740 10 client.0 _ll_register_callbacks cb 0x55a77ca4d0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.225+0000 7f7b1affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 0x55a77cb23d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.225+0000 7f7b1affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 0x55a77cb23d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49142/0 (socket says 172.21.15.44:49142) 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.225+0000 7f7b1affd640 1 -- 172.21.15.44:0/4125246640 learned_addr learned my addr 172.21.15.44:0/4125246640 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b1a7fc640 1 --2- 172.21.15.44:0/4125246640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 0x55a77cb1e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b1b7fe640 1 --2- 172.21.15.44:0/4125246640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 0x55a77cb0c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b1affd640 1 -- 172.21.15.44:0/4125246640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 msgr2=0x55a77cb1e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b1affd640 1 --2- 172.21.15.44:0/4125246640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 0x55a77cb1e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b1affd640 1 -- 172.21.15.44:0/4125246640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 msgr2=0x55a77cb0c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b1affd640 1 --2- 172.21.15.44:0/4125246640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 0x55a77cb0c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b1affd640 1 -- 172.21.15.44:0/4125246640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a77c8336b0 con 0x55a77cb18b90 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b1affd640 1 --2- 172.21.15.44:0/4125246640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 0x55a77cb23d50 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c055a90 tx=0x7f7b0c05da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e250912aa5bc641 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:17.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7afa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7afa7fc640 1 -- 172.21.15.44:0/4125246640 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c058a80 con 0x55a77cb18b90 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7afa7fc640 1 -- 172.21.15.44:0/4125246640 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f7b0c058c20 con 0x55a77cb18b90 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b24413740 1 -- 172.21.15.44:0/4125246640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b14001830 con 0x55a77cb18b90 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7afa7fc640 1 -- 172.21.15.44:0/4125246640 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f7b0c088050 con 0x55a77cb18b90 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b24413740 10 client.8907 fetch_fsmap learned FSMap version 82 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b24413740 1 -- 172.21.15.44:0/4125246640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) -- 0x55a77cb1f7b0 con 0x55a77cb18b90 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7afa7fc640 1 -- 172.21.15.44:0/4125246640 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b14001830 con 0x55a77cb18b90 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7afa7fc640 1 -- 172.21.15.44:0/4125246640 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 82) ==== 3494+0+0 (secure 0 0 0) 0x7f7b0c0772b0 con 0x55a77cb18b90 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b24413740 10 client.8907 fetch_fsmap finished waiting for FSMap version 82 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b24413740 10 client.8907 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.226+0000 7f7b24413740 20 client.8907 populate_metadata read hostname 'smithi044' 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.227+0000 7f7b24413740 1 --2- 172.21.15.44:0/4125246640 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55a77cb4a5c0 0x55a77cb22380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.227+0000 7f7b24413740 4 client.8907 mds_command: new command op to 8760 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:29:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.227+0000 7f7b24413740 1 -- 172.21.15.44:0/4125246640 --> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a77c9fccf0 con 0x55a77cb4a5c0 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.227+0000 7f7b1a7fc640 1 --2- 172.21.15.44:0/4125246640 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55a77cb4a5c0 0x55a77cb22380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.227+0000 7f7afa7fc640 10 client.8907 ms_handle_connect on v2:172.21.15.44:6833/3259420283 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.227+0000 7f7b1a7fc640 1 --2- 172.21.15.44:0/4125246640 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55a77cb4a5c0 0x55a77cb22380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ba036a338473cd1 server_cookie=37e23f146dae760c in_seq=0 out_seq=0 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7afa7fc640 1 -- 172.21.15.44:0/4125246640 <== mds.0 v2:172.21.15.44:6833/3259420283 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a77c9fccf0 con 0x55a77cb4a5c0 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7afa7fc640 10 client.8907 handle_command_reply: tid=0 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7b24413740 1 client.8907 shutdown 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7b24413740 1 -- 172.21.15.44:0/4125246640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 msgr2=0x55a77cb23d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7b24413740 1 --2- 172.21.15.44:0/4125246640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 0x55a77cb23d50 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c055a90 tx=0x7f7b0c05da40 comp rx=0 tx=0).stop 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7b24413740 1 -- 172.21.15.44:0/4125246640 shutdown_connections 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7b24413740 1 --2- 172.21.15.44:0/4125246640 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55a77cb4a5c0 0x55a77cb22380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7b24413740 1 --2- 172.21.15.44:0/4125246640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 0x55a77cb0c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7afa7fc640 0 client.8907 ms_handle_reset on v2:172.21.15.44:6833/3259420283 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7b24413740 1 --2- 172.21.15.44:0/4125246640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 0x55a77cb1e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7b24413740 1 --2- 172.21.15.44:0/4125246640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 0x55a77cb23d50 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7b24413740 1 -- 172.21.15.44:0/4125246640 >> 172.21.15.44:0/4125246640 conn(0x55a77caec070 msgr2=0x55a77caefbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:17.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7b24413740 1 -- 172.21.15.44:0/4125246640 shutdown_connections 2024-04-02T13:29:17.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7b24413740 1 -- 172.21.15.44:0/4125246640 wait complete. 2024-04-02T13:29:17.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.228+0000 7f7b24413740 20 client.8907 trim_cache size 0 max 16384 2024-04-02T13:29:17.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.246+0000 7f7b24413740 1 Processor -- start 2024-04-02T13:29:17.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.246+0000 7f7b24413740 1 -- start start 2024-04-02T13:29:17.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.247+0000 7f7b24413740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 0x55a77cb177c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.247+0000 7f7b24413740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 0x55a77cb17d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.247+0000 7f7b24413740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 0x55a77cb489b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.247+0000 7f7b24413740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1fc8f0 con 0x55a77cb18b90 2024-04-02T13:29:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.247+0000 7f7b24413740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c14fc70 con 0x55a77cb1ed20 2024-04-02T13:29:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.247+0000 7f7b24413740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7b1c14efd0 con 0x55a77cb18f70 2024-04-02T13:29:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.247+0000 7f7b1affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 0x55a77cb177c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.247+0000 7f7b1affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 0x55a77cb177c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49144/0 (socket says 172.21.15.44:49144) 2024-04-02T13:29:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.247+0000 7f7b1affd640 1 -- 172.21.15.44:0/4265946051 learned_addr learned my addr 172.21.15.44:0/4265946051 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b1affd640 1 -- 172.21.15.44:0/4265946051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 msgr2=0x55a77cb17d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b1affd640 1 --2- 172.21.15.44:0/4265946051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 0x55a77cb17d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b1affd640 1 -- 172.21.15.44:0/4265946051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 msgr2=0x55a77cb489b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b1affd640 1 --2- 172.21.15.44:0/4265946051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 0x55a77cb489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b1affd640 1 -- 172.21.15.44:0/4265946051 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a77cb20f20 con 0x55a77cb18b90 2024-04-02T13:29:17.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b1affd640 1 --2- 172.21.15.44:0/4265946051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 0x55a77cb177c0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c05dad0 tx=0x7f7b0c05c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d3d498d5072e46 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b10ff9640 1 -- 172.21.15.44:0/4265946051 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c07d030 con 0x55a77cb18b90 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b10ff9640 1 -- 172.21.15.44:0/4265946051 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b0c061f70 con 0x55a77cb18b90 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b10ff9640 1 -- 172.21.15.44:0/4265946051 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c0558d0 con 0x55a77cb18b90 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b24413740 1 -- 172.21.15.44:0/4265946051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 msgr2=0x55a77cb177c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b24413740 1 --2- 172.21.15.44:0/4265946051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 0x55a77cb177c0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c05dad0 tx=0x7f7b0c05c000 comp rx=0 tx=0).stop 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b24413740 1 -- 172.21.15.44:0/4265946051 shutdown_connections 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b24413740 1 --2- 172.21.15.44:0/4265946051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 0x55a77cb489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b24413740 1 --2- 172.21.15.44:0/4265946051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18f70 0x55a77cb17d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b24413740 1 --2- 172.21.15.44:0/4265946051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18b90 0x55a77cb177c0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.248+0000 7f7b24413740 1 -- 172.21.15.44:0/4265946051 >> 172.21.15.44:0/4265946051 conn(0x55a77caec070 msgr2=0x55a77cb39730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.249+0000 7f7b24413740 1 -- 172.21.15.44:0/4265946051 shutdown_connections 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.249+0000 7f7b24413740 1 -- 172.21.15.44:0/4265946051 wait complete. 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.249+0000 7f7b24413740 1 -- start start 2024-04-02T13:29:17.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.249+0000 7f7b24413740 1 Processor -- start 2024-04-02T13:29:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b24413740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18b90 0x55a77cb1bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b24413740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18f70 0x55a77cb59400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b24413740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 0x55a77cb59940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b24413740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a77cb4a440 con 0x55a77cb18f70 2024-04-02T13:29:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b24413740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c14f620 con 0x55a77cb1ed20 2024-04-02T13:29:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b24413740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a77cb4aa60 con 0x55a77cb18b90 2024-04-02T13:29:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b24413740 10 client.0 _ll_register_callbacks cb 0x7f7b0c059200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:29:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b1a7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18f70 0x55a77cb59400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b1a7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18f70 0x55a77cb59400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49152/0 (socket says 172.21.15.44:49152) 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b1a7fc640 1 -- 172.21.15.44:0/2144058014 learned_addr learned my addr 172.21.15.44:0/2144058014 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b1b7fe640 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 0x55a77cb59940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b1affd640 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18b90 0x55a77cb1bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b1a7fc640 1 -- 172.21.15.44:0/2144058014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18b90 msgr2=0x55a77cb1bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b1a7fc640 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18b90 0x55a77cb1bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b1a7fc640 1 -- 172.21.15.44:0/2144058014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 msgr2=0x55a77cb59940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b1a7fc640 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 0x55a77cb59940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b1a7fc640 1 -- 172.21.15.44:0/2144058014 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a77cb17ea0 con 0x55a77cb18f70 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b1b7fe640 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 0x55a77cb59940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7b1a7fc640 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18f70 0x55a77cb59400 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f7b0403a040 tx=0x7f7b040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea9b87096733fe6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.250+0000 7f7afa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7afa7fc640 1 -- 172.21.15.44:0/2144058014 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b040562f0 con 0x55a77cb18f70 2024-04-02T13:29:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7afa7fc640 1 -- 172.21.15.44:0/2144058014 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f7b04056490 con 0x55a77cb18f70 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7b24413740 1 -- 172.21.15.44:0/2144058014 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b0c061f70 con 0x55a77cb18f70 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7afa7fc640 1 -- 172.21.15.44:0/2144058014 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f7b04056690 con 0x55a77cb18f70 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7b24413740 10 client.8913 fetch_fsmap learned FSMap version 82 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7b24413740 1 -- 172.21.15.44:0/2144058014 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) -- 0x55a77cae6e50 con 0x55a77cb18f70 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7afa7fc640 1 -- 172.21.15.44:0/2144058014 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b0c061f70 con 0x55a77cb18f70 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7afa7fc640 1 -- 172.21.15.44:0/2144058014 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 82) ==== 3494+0+0 (secure 0 0 0) 0x7f7b04003610 con 0x55a77cb18f70 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7b24413740 10 client.8913 fetch_fsmap finished waiting for FSMap version 82 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7b24413740 10 client.8913 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7b24413740 20 client.8913 populate_metadata read hostname 'smithi044' 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7b24413740 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55a77cbb0470 0x55a77cbb0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7b24413740 4 client.8913 mds_command: new command op to 8760 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7b24413740 1 -- 172.21.15.44:0/2144058014 --> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] -- command(tid 0: {"prefix": "status"}) -- 0x55a77ca625a0 con 0x55a77cbb0470 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.251+0000 7f7b1affd640 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55a77cbb0470 0x55a77cbb0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7afa7fc640 10 client.8913 ms_handle_connect on v2:172.21.15.44:6833/3259420283 2024-04-02T13:29:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7b1affd640 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55a77cbb0470 0x55a77cbb0850 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b2559305570ed33 server_cookie=42dfcda6daeef0d2 in_seq=0 out_seq=0 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7afa7fc640 1 -- 172.21.15.44:0/2144058014 <== mds.0 v2:172.21.15.44:6833/3259420283 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a77ca625a0 con 0x55a77cbb0470 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7afa7fc640 10 client.8913 handle_command_reply: tid=0 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7b24413740 1 client.8913 shutdown 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7b24413740 1 -- 172.21.15.44:0/2144058014 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18f70 msgr2=0x55a77cb59400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7b24413740 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18f70 0x55a77cb59400 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f7b0403a040 tx=0x7f7b040046c0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7b24413740 1 -- 172.21.15.44:0/2144058014 shutdown_connections 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7b24413740 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55a77cbb0470 0x55a77cbb0850 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7b24413740 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a77cb1ed20 0x55a77cb59940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7afa7fc640 0 client.8913 ms_handle_reset on v2:172.21.15.44:6833/3259420283 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7b24413740 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a77cb18f70 0x55a77cb59400 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7b24413740 1 --2- 172.21.15.44:0/2144058014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a77cb18b90 0x55a77cb1bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7b24413740 1 -- 172.21.15.44:0/2144058014 >> 172.21.15.44:0/2144058014 conn(0x55a77caec070 msgr2=0x55a77cb388a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7b24413740 1 -- 172.21.15.44:0/2144058014 shutdown_connections 2024-04-02T13:29:17.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7b24413740 1 -- 172.21.15.44:0/2144058014 wait complete. 2024-04-02T13:29:17.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.252+0000 7f7b24413740 20 client.8913 trim_cache size 0 max 16384 2024-04-02T13:29:17.263 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:29:17.263 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:29:17.263 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:29:17.263 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 8760, 2024-04-02T13:29:17.263 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:29:17.263 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:29:17.263 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:29:17.263 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 5.6919798559999997, 2024-04-02T13:29:17.263 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 82, 2024-04-02T13:29:17.264 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 106, 2024-04-02T13:29:17.264 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 106, 2024-04-02T13:29:17.264 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 10.691428083 2024-04-02T13:29:17.264 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:29:17.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.253+0000 7f7b2182a640 1 -- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7afc06c8f0 msgr2=0x7f7afc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.253+0000 7f7b2182a640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7afc06c8f0 0x7f7afc06edb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7b04039000 tx=0x7f7b04034000 comp rx=0 tx=0).stop 2024-04-02T13:29:17.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.253+0000 7f7b2182a640 1 -- 172.21.15.44:0/211569823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b1c0b2d00 msgr2=0x7f7b1c154d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.253+0000 7f7b2182a640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b1c0b2d00 0x7f7b1c154d50 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c00ba50 tx=0x7f7b0c00bf10 comp rx=0 tx=0).stop 2024-04-02T13:29:17.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.253+0000 7f7b2182a640 1 -- 172.21.15.44:0/211569823 shutdown_connections 2024-04-02T13:29:17.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.253+0000 7f7b2182a640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7afc06c8f0 0x7f7afc06edb0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.253+0000 7f7b2182a640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b1c15aa00 0x7f7b1c159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.253+0000 7f7b2182a640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b1c0ce110 0x7f7b1c06e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.253+0000 7f7b2182a640 1 --2- 172.21.15.44:0/211569823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b1c0b2d00 0x7f7b1c154d50 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.253+0000 7f7b2182a640 1 -- 172.21.15.44:0/211569823 >> 172.21.15.44:0/211569823 conn(0x7f7b1c0cfcd0 msgr2=0x7f7b1c0b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:17.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.253+0000 7f7b2182a640 1 -- 172.21.15.44:0/211569823 shutdown_connections 2024-04-02T13:29:17.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.253+0000 7f7b2182a640 1 -- 172.21.15.44:0/211569823 wait complete. 2024-04-02T13:29:17.267 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:29:17.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.364+0000 7f5370511640 1 Processor -- start 2024-04-02T13:29:17.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.364+0000 7f5370511640 1 -- start start 2024-04-02T13:29:17.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.365+0000 7f5370511640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f53680d28f0 0x7f53680d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.365+0000 7f5370511640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53680b5200 0x7f53680b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.365+0000 7f5370511640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53680d4d70 con 0x7f53680d1260 2024-04-02T13:29:17.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.365+0000 7f5370511640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f53680d5050 con 0x7f53680d28f0 2024-04-02T13:29:17.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.365+0000 7f5370511640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f53680d23a0 con 0x7f53680b5200 2024-04-02T13:29:17.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.365+0000 7f536daac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53680b5200 0x7f53680b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.365+0000 7f536e2ad640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f53680d28f0 0x7f53680d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.365+0000 7f536eaae640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f53680d1260 0x7f53680ced30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:40778/0 (socket says 172.21.15.44:40778) 2024-04-02T13:29:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.365+0000 7f536eaae640 1 -- 172.21.15.44:0/977350787 learned_addr learned my addr 172.21.15.44:0/977350787 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.366+0000 7f536d2ab640 1 -- 172.21.15.44:0/977350787 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 744657215 0 0) 0x7f53680d4d70 con 0x7f53680d1260 2024-04-02T13:29:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.366+0000 7f536d2ab640 1 -- 172.21.15.44:0/977350787 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5354003520 con 0x7f53680d1260 2024-04-02T13:29:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.366+0000 7f536e2ad640 1 -- 172.21.15.44:0/977350787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53680b5200 msgr2=0x7f53680b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.366+0000 7f536e2ad640 1 --2- 172.21.15.44:0/977350787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53680b5200 0x7f53680b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.366+0000 7f536e2ad640 1 -- 172.21.15.44:0/977350787 >> v1:172.21.15.69:6789/0 conn(0x7f53680d1260 legacy=0x7f53680ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.366+0000 7f536e2ad640 1 -- 172.21.15.44:0/977350787 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53680d2030 con 0x7f53680d28f0 2024-04-02T13:29:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.366+0000 7f536e2ad640 1 --2- 172.21.15.44:0/977350787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53680d28f0 0x7f53680d2cd0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f5364009d90 tx=0x7f536402bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bab4d2587fecaf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.367+0000 7f536d2ab640 1 -- 172.21.15.44:0/977350787 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5364039070 con 0x7f53680d28f0 2024-04-02T13:29:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.367+0000 7f536d2ab640 1 -- 172.21.15.44:0/977350787 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f536402cb30 con 0x7f53680d28f0 2024-04-02T13:29:17.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.367+0000 7f536d2ab640 1 -- 172.21.15.44:0/977350787 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5364034440 con 0x7f53680d28f0 2024-04-02T13:29:17.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.367+0000 7f5370511640 1 -- 172.21.15.44:0/977350787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53680d28f0 msgr2=0x7f53680d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.367+0000 7f5370511640 1 --2- 172.21.15.44:0/977350787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53680d28f0 0x7f53680d2cd0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f5364009d90 tx=0x7f536402bd80 comp rx=0 tx=0).stop 2024-04-02T13:29:17.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.367+0000 7f5370511640 1 -- 172.21.15.44:0/977350787 shutdown_connections 2024-04-02T13:29:17.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.367+0000 7f5370511640 1 --2- 172.21.15.44:0/977350787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53680b5200 0x7f53680b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.367+0000 7f5370511640 1 --2- 172.21.15.44:0/977350787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53680d28f0 0x7f53680d2cd0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.367+0000 7f5370511640 1 -- 172.21.15.44:0/977350787 >> 172.21.15.44:0/977350787 conn(0x7f53680d4120 msgr2=0x7f53680d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:17.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.367+0000 7f5370511640 1 -- 172.21.15.44:0/977350787 shutdown_connections 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.367+0000 7f5370511640 1 -- 172.21.15.44:0/977350787 wait complete. 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.368+0000 7f5370511640 1 Processor -- start 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.369+0000 7f5370511640 1 -- start start 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.369+0000 7f5370511640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53680b5200 0x7f5368156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.369+0000 7f5370511640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53680d1260 0x7f53680b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.369+0000 7f5370511640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53680d28f0 0x7f53680b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.369+0000 7f5370511640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f53680b8130 con 0x7f53680d28f0 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.369+0000 7f5370511640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f53680b5020 con 0x7f53680b5200 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.369+0000 7f5370511640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f53680b82b0 con 0x7f53680d1260 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.369+0000 7f536eaae640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53680d28f0 0x7f53680b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.369+0000 7f536eaae640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53680d28f0 0x7f53680b3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49162/0 (socket says 172.21.15.44:49162) 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.369+0000 7f536eaae640 1 -- 172.21.15.44:0/1712604573 learned_addr learned my addr 172.21.15.44:0/1712604573 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.369+0000 7f536daac640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53680d1260 0x7f53680b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.369+0000 7f536e2ad640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53680b5200 0x7f5368156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.370+0000 7f536eaae640 1 -- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53680d1260 msgr2=0x7f53680b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.370+0000 7f536eaae640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53680d1260 0x7f53680b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.370+0000 7f536eaae640 1 -- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53680b5200 msgr2=0x7f5368156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.370+0000 7f536eaae640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53680b5200 0x7f5368156a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.370+0000 7f536eaae640 1 -- 172.21.15.44:0/1712604573 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53680cbab0 con 0x7f53680d28f0 2024-04-02T13:29:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.370+0000 7f536eaae640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53680d28f0 0x7f53680b3690 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f536000c390 tx=0x7f536000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.370+0000 7f535effd640 1 -- 172.21.15.44:0/1712604573 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5360017070 con 0x7f53680d28f0 2024-04-02T13:29:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.370+0000 7f535effd640 1 -- 172.21.15.44:0/1712604573 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5360005ce0 con 0x7f53680d28f0 2024-04-02T13:29:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.370+0000 7f5370511640 1 -- 172.21.15.44:0/1712604573 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53680b1b70 con 0x7f53680d28f0 2024-04-02T13:29:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.370+0000 7f535effd640 1 -- 172.21.15.44:0/1712604573 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5360007310 con 0x7f53680d28f0 2024-04-02T13:29:17.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.370+0000 7f5370511640 1 -- 172.21.15.44:0/1712604573 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53680b2130 con 0x7f53680d28f0 2024-04-02T13:29:17.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.371+0000 7f5370511640 1 -- 172.21.15.44:0/1712604573 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53680684f0 con 0x7f53680d28f0 2024-04-02T13:29:17.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.372+0000 7f535effd640 1 -- 172.21.15.44:0/1712604573 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f536001a070 con 0x7f53680d28f0 2024-04-02T13:29:17.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.372+0000 7f535effd640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f534c06c920 0x7f534c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:17.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.372+0000 7f535effd640 1 -- 172.21.15.44:0/1712604573 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f536008aff0 con 0x7f53680d28f0 2024-04-02T13:29:17.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.372+0000 7f536e2ad640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f534c06c920 0x7f534c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:17.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.373+0000 7f536e2ad640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f534c06c920 0x7f534c06ede0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5364036000 tx=0x7f5364030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:17.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.376+0000 7f535effd640 1 -- 172.21.15.44:0/1712604573 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5360058c40 con 0x7f53680d28f0 2024-04-02T13:29:17.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.604+0000 7f5370511640 1 -- 172.21.15.44:0/1712604573 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f53680d0e10 con 0x7f53680d28f0 2024-04-02T13:29:17.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.604+0000 7f535effd640 1 -- 172.21.15.44:0/1712604573 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v82) ==== 53+0+73 (secure 0 0 0) 0x7f536005cf40 con 0x7f53680d28f0 2024-04-02T13:29:17.608 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:29:17.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.606+0000 7f5370511640 1 -- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f534c06c920 msgr2=0x7f534c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.606+0000 7f5370511640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f534c06c920 0x7f534c06ede0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5364036000 tx=0x7f5364030000 comp rx=0 tx=0).stop 2024-04-02T13:29:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.606+0000 7f5370511640 1 -- 172.21.15.44:0/1712604573 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53680d28f0 msgr2=0x7f53680b3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.606+0000 7f5370511640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53680d28f0 0x7f53680b3690 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f536000c390 tx=0x7f536000c850 comp rx=0 tx=0).stop 2024-04-02T13:29:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.606+0000 7f5370511640 1 -- 172.21.15.44:0/1712604573 shutdown_connections 2024-04-02T13:29:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.606+0000 7f5370511640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f534c06c920 0x7f534c06ede0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.606+0000 7f5370511640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53680d28f0 0x7f53680b3690 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.606+0000 7f5370511640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53680d1260 0x7f53680b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.606+0000 7f5370511640 1 --2- 172.21.15.44:0/1712604573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53680b5200 0x7f5368156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.606+0000 7f5370511640 1 -- 172.21.15.44:0/1712604573 >> 172.21.15.44:0/1712604573 conn(0x7f53680d4120 msgr2=0x7f5368161820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.607+0000 7f5370511640 1 -- 172.21.15.44:0/1712604573 shutdown_connections 2024-04-02T13:29:17.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17.607+0000 7f5370511640 1 -- 172.21.15.44:0/1712604573 wait complete. 2024-04-02T13:29:17.621 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:29:17.621 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:29:17.621 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:29:17.621 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:29:17.621 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:17.621 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:29:17.621 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:29:17.621 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:29:17.621 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:29:17.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:17.622 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:29:17.636 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:29:17.636 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:29:17.637 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:29:17.637 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:17.637 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:29:17.637 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:17.637 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:29:17.637 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:29:17.637 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:29:17.637 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:29:17.637 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:29:17.637 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:29:17.637 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:29:17.637 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:29:17.637 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:29:17.638 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:29:17.638 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42221sec preferred_lft 42221sec 2024-04-02T13:29:17.638 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:29:17.638 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:17.638 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:29:17.638 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:29:17.638 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:29:17.638 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:29:17.638 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:29:17.638 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:29:17.638 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:17.638 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:29:17.638 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:17.639 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:29:17.639 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:29:17.639 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:29:17.639 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:17.639 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:29:17.639 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:29:17.639 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:29:17.639 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:17.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:17.639 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:29:17.702 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:29:17.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:17.704 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:29:17.759 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:29:17.759 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:29:17.759 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:29:17.759 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:29:17.759 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:29:17.759 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:29:17.760 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:29:17.760 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:29:17.760 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:29:17.760 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:29:17.760 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:29:17.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:29:17.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:29:17.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:17.894 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:29:17.909 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:29:17.909 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:29:17.909 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:29:17.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:17.909 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:29:17.976 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:29:17.976 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:29:17.976 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:18.031 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:29:18.031 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:29:18.031 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:18.086 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:29:18.153 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:29:18.209 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:29:18.209 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:29:18.209 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:29:18.209 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:29:18.209 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:29:18.210 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:29:18.210 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:29:18.210 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:29:18.210 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:29:18.210 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:29:18.210 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:29:18.210 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:29:18.210 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:29:18.210 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:29:18.210 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:29:18.210 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:29:18.210 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:29:18.211 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:29:18.211 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:29:18.211 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:29:18.211 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:29:18.211 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:29:18.211 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:29:18.211 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:29:18.211 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:29:18.211 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:29:18.211 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:29:18.211 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:29:18.211 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:29:18.211 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:29:18.212 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:29:18.212 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:29:18.212 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:29:18.212 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:29:18.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:18.212 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:29:18.264 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:29:18.264 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:29:18.307 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:29:18.333 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:29:18.389 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:29:18.389 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:29:18.389 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:29:18.389 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:29:18.389 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:29:18.389 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:29:18.390 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:29:18.391 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:29:18.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:18.392 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:29:18.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:29:18.389+0000 7f6b94039480 -1 init, newargv = 0x555e94ab54d0 newargc=15 2024-04-02T13:29:18.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[76360]: starting ceph client 2024-04-02T13:29:18.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[76360]: starting fuse 2024-04-02T13:29:18.407 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:29:18.407 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:29:18.407 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:29:18.407 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:29:18.407 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:29:18.407 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:29:18.407 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:29:18.408 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:29:18.409 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:29:18.409 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:29:18.409 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:29:18.409 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:29:18.409 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:29:18.409 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:29:18.529 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.76360.asok 2024-04-02T13:29:18.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:29:18.539 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.76360.asok 2024-04-02T13:29:18.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:18.540 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.76360.asok status 2024-04-02T13:29:18.677 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:29:18.677 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:29:18.677 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:29:18.677 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:29:18.677 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:29:18.677 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:29:18.677 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:29:18.677 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "76360", 2024-04-02T13:29:18.678 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:29:18.678 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:18.678 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:29:18.678 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:29:18.678 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 8922, 2024-04-02T13:29:18.678 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:29:18.678 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:29:18.678 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:29:18.678 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 8922 2024-04-02T13:29:18.678 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:18.678 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:29:18.678 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:29:18.678 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:29:18.679 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 704056504 2024-04-02T13:29:18.679 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:18.679 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:18.679 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:29:18.679 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:29:18.679 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:29:18.679 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 704056504 2024-04-02T13:29:18.679 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:18.679 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.8922 192.168.0.1:0/704056504", 2024-04-02T13:29:18.679 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/704056504", 2024-04-02T13:29:18.680 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:29:18.680 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 82, 2024-04-02T13:29:18.680 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 106, 2024-04-02T13:29:18.680 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:29:18.680 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:29:18.680 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:29:18.680 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:29:18.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:18.685 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:29:18.685 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:18.700 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:29:18.700 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:18.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:18.701 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:18.768 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:29:18.768 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:29:18.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:18.824 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file1_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-04-02T13:29:18.925 INFO:teuthology.orchestra.run.smithi044.stderr:6+0 records in 2024-04-02T13:29:18.925 INFO:teuthology.orchestra.run.smithi044.stderr:6+0 records out 2024-04-02T13:29:18.925 INFO:teuthology.orchestra.run.smithi044.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0432839 s, 145 MB/s 2024-04-02T13:29:18.926 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:29:18.927 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:29:18.948 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:29:19.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.087+0000 7feee8675640 1 Processor -- start 2024-04-02T13:29:19.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.087+0000 7feee8675640 1 -- start start 2024-04-02T13:29:19.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.088+0000 7feee8675640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feee006cf90 0x7feee00cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.088+0000 7feee8675640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7feee00cfe30 0x7feee00d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.088+0000 7feee8675640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feee006c1d0 con 0x7feee00d0a30 2024-04-02T13:29:19.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.088+0000 7feee8675640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7feee006cb00 con 0x7feee00cfe30 2024-04-02T13:29:19.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.088+0000 7feee8675640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7feee006d3f0 con 0x7feee006cf90 2024-04-02T13:29:19.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.088+0000 7feee6c12640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7feee00cfe30 0x7feee00d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.088+0000 7feee6c12640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7feee00cfe30 0x7feee00d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49178/0 (socket says 172.21.15.44:49178) 2024-04-02T13:29:19.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.088+0000 7feee6c12640 1 -- 172.21.15.44:0/1204363569 learned_addr learned my addr 172.21.15.44:0/1204363569 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:19.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.089+0000 7feee5c10640 1 --2- 172.21.15.44:0/1204363569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feee006cf90 0x7feee00cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.089+0000 7feee6c12640 1 -- 172.21.15.44:0/1204363569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feee006cf90 msgr2=0x7feee00cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:19.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.089+0000 7feee6c12640 1 --2- 172.21.15.44:0/1204363569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feee006cf90 0x7feee00cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.089+0000 7feee6c12640 1 -- 172.21.15.44:0/1204363569 >> v1:172.21.15.69:6789/0 conn(0x7feee00d0a30 legacy=0x7feee00d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.089+0000 7feee6c12640 1 -- 172.21.15.44:0/1204363569 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee006c4b0 con 0x7feee00cfe30 2024-04-02T13:29:19.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.089+0000 7feee6c12640 1 --2- 172.21.15.44:0/1204363569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feee00cfe30 0x7feee00d2040 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7feed0009e50 tx=0x7feed002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f21b5491f9d3b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:19.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.090+0000 7feee540f640 1 -- 172.21.15.44:0/1204363569 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed002ca70 con 0x7feee00cfe30 2024-04-02T13:29:19.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.090+0000 7feee540f640 1 -- 172.21.15.44:0/1204363569 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed002cc10 con 0x7feee00cfe30 2024-04-02T13:29:19.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.090+0000 7feee540f640 1 -- 172.21.15.44:0/1204363569 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed00343f0 con 0x7feee00cfe30 2024-04-02T13:29:19.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.090+0000 7feee8675640 1 -- 172.21.15.44:0/1204363569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feee00cfe30 msgr2=0x7feee00d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.090+0000 7feee8675640 1 --2- 172.21.15.44:0/1204363569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feee00cfe30 0x7feee00d2040 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7feed0009e50 tx=0x7feed002bed0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.090+0000 7feee8675640 1 -- 172.21.15.44:0/1204363569 shutdown_connections 2024-04-02T13:29:19.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.090+0000 7feee8675640 1 --2- 172.21.15.44:0/1204363569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feee00cfe30 0x7feee00d2040 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.090+0000 7feee8675640 1 --2- 172.21.15.44:0/1204363569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feee006cf90 0x7feee00cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.090+0000 7feee8675640 1 -- 172.21.15.44:0/1204363569 >> 172.21.15.44:0/1204363569 conn(0x7feee009d470 msgr2=0x7feee00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:19.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.090+0000 7feee8675640 1 -- 172.21.15.44:0/1204363569 shutdown_connections 2024-04-02T13:29:19.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.090+0000 7feee8675640 1 -- 172.21.15.44:0/1204363569 wait complete. 2024-04-02T13:29:19.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.091+0000 7feee8675640 1 Processor -- start 2024-04-02T13:29:19.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.091+0000 7feee8675640 1 -- start start 2024-04-02T13:29:19.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.092+0000 7feee8675640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feee006cf90 0x7feee00cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.092+0000 7feee8675640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feee00cfe30 0x7feee014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.092+0000 7feee8675640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feee00d0a30 0x7feee014edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.092+0000 7feee8675640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feee015e870 con 0x7feee00cfe30 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.092+0000 7feee8675640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7feee015e6f0 con 0x7feee006cf90 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.092+0000 7feee8675640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7feee00d4da0 con 0x7feee00d0a30 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.092+0000 7feee5c10640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feee00cfe30 0x7feee014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.092+0000 7feee5c10640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feee00cfe30 0x7feee014e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49184/0 (socket says 172.21.15.44:49184) 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.092+0000 7feee5c10640 1 -- 172.21.15.44:0/130079959 learned_addr learned my addr 172.21.15.44:0/130079959 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.092+0000 7feee6c12640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feee00d0a30 0x7feee014edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.092+0000 7feee6411640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feee006cf90 0x7feee00cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.093+0000 7feee5c10640 1 -- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feee00d0a30 msgr2=0x7feee014edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.093+0000 7feee5c10640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feee00d0a30 0x7feee014edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.093+0000 7feee5c10640 1 -- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feee006cf90 msgr2=0x7feee00cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.093+0000 7feee5c10640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feee006cf90 0x7feee00cc860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.093+0000 7feee5c10640 1 -- 172.21.15.44:0/130079959 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee00ccf50 con 0x7feee00cfe30 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.093+0000 7feee6c12640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feee00d0a30 0x7feee014edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:19.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.093+0000 7feee5c10640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feee00cfe30 0x7feee014e1b0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7feed400d5d0 tx=0x7feed400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:19.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.093+0000 7feedeffd640 1 -- 172.21.15.44:0/130079959 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed4004240 con 0x7feee00cfe30 2024-04-02T13:29:19.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.093+0000 7feedeffd640 1 -- 172.21.15.44:0/130079959 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed40043e0 con 0x7feee00cfe30 2024-04-02T13:29:19.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.093+0000 7feee8675640 1 -- 172.21.15.44:0/130079959 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee00cd230 con 0x7feee00cfe30 2024-04-02T13:29:19.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.093+0000 7feedeffd640 1 -- 172.21.15.44:0/130079959 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed4003830 con 0x7feee00cfe30 2024-04-02T13:29:19.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.093+0000 7feee8675640 1 -- 172.21.15.44:0/130079959 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feee00cd7f0 con 0x7feee00cfe30 2024-04-02T13:29:19.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.094+0000 7feee8675640 1 -- 172.21.15.44:0/130079959 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feee00684f0 con 0x7feee00cfe30 2024-04-02T13:29:19.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.094+0000 7feedeffd640 1 -- 172.21.15.44:0/130079959 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feed40039d0 con 0x7feee00cfe30 2024-04-02T13:29:19.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.095+0000 7feedeffd640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feecc06c920 0x7feecc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.095+0000 7feedeffd640 1 -- 172.21.15.44:0/130079959 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7feed408bf60 con 0x7feee00cfe30 2024-04-02T13:29:19.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.095+0000 7feee6411640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feecc06c920 0x7feecc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.095+0000 7feee6411640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feecc06c920 0x7feecc06ede0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7feed80015e0 tx=0x7feed8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:19.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.099+0000 7feedeffd640 1 -- 172.21.15.44:0/130079959 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7feed4059c40 con 0x7feee00cfe30 2024-04-02T13:29:19.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.285+0000 7feee8675640 1 -- 172.21.15.44:0/130079959 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feee00d17c0 con 0x7feee00cfe30 2024-04-02T13:29:19.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.286+0000 7feedeffd640 1 -- 172.21.15.44:0/130079959 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+16836 (secure 0 0 0) 0x7feed405df40 con 0x7feee00cfe30 2024-04-02T13:29:19.290 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:19.291 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":106,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:29:12.549182+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:29:09.192430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-02T13:29:09.860748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:29:19.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.288+0000 7feee8675640 1 -- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feecc06c920 msgr2=0x7feecc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.288+0000 7feee8675640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feecc06c920 0x7feecc06ede0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7feed80015e0 tx=0x7feed8009040 comp rx=0 tx=0).stop 2024-04-02T13:29:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.288+0000 7feee8675640 1 -- 172.21.15.44:0/130079959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feee00cfe30 msgr2=0x7feee014e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.288+0000 7feee8675640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feee00cfe30 0x7feee014e1b0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7feed400d5d0 tx=0x7feed400da90 comp rx=0 tx=0).stop 2024-04-02T13:29:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.288+0000 7feee8675640 1 -- 172.21.15.44:0/130079959 shutdown_connections 2024-04-02T13:29:19.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.288+0000 7feee8675640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7feecc06c920 0x7feecc06ede0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.288+0000 7feee8675640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feee00d0a30 0x7feee014edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.288+0000 7feee8675640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feee00cfe30 0x7feee014e1b0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.288+0000 7feee8675640 1 --2- 172.21.15.44:0/130079959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feee006cf90 0x7feee00cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.288+0000 7feee8675640 1 -- 172.21.15.44:0/130079959 >> 172.21.15.44:0/130079959 conn(0x7feee009d470 msgr2=0x7feee015c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:19.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.288+0000 7feee8675640 1 -- 172.21.15.44:0/130079959 shutdown_connections 2024-04-02T13:29:19.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.288+0000 7feee8675640 1 -- 172.21.15.44:0/130079959 wait complete. 2024-04-02T13:29:19.303 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:29:09.192430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:29:09.860748+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:29:19.303 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:29:19.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.405+0000 7f8c8dde2640 1 Processor -- start 2024-04-02T13:29:19.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.405+0000 7f8c8dde2640 1 -- start start 2024-04-02T13:29:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.406+0000 7f8c8dde2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c880cec60 0x7f8c880d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.406+0000 7f8c8dde2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8c8815e5b0 0x7f8c880d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.406+0000 7f8c8dde2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c880cf1f0 con 0x7f8c8815e1d0 2024-04-02T13:29:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.406+0000 7f8c8dde2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f8c880cfe00 con 0x7f8c8815e5b0 2024-04-02T13:29:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.406+0000 7f8c8dde2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8c880cf4d0 con 0x7f8c880cec60 2024-04-02T13:29:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.406+0000 7f8c87fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8c8815e5b0 0x7f8c880d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.406+0000 7f8c87fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8c8815e5b0 0x7f8c880d2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49190/0 (socket says 172.21.15.44:49190) 2024-04-02T13:29:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.406+0000 7f8c87fff640 1 -- 172.21.15.44:0/4093473517 learned_addr learned my addr 172.21.15.44:0/4093473517 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.407+0000 7f8c87fff640 1 -- 172.21.15.44:0/4093473517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c880cec60 msgr2=0x7f8c880d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.407+0000 7f8c86ffd640 1 --2- 172.21.15.44:0/4093473517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c880cec60 0x7f8c880d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.407+0000 7f8c87fff640 1 --2- 172.21.15.44:0/4093473517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c880cec60 0x7f8c880d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.407+0000 7f8c87fff640 1 -- 172.21.15.44:0/4093473517 >> v1:172.21.15.69:6789/0 conn(0x7f8c8815e1d0 legacy=0x7f8c880d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.407+0000 7f8c87fff640 1 -- 172.21.15.44:0/4093473517 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c880cf7b0 con 0x7f8c8815e5b0 2024-04-02T13:29:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.407+0000 7f8c86ffd640 1 --2- 172.21.15.44:0/4093473517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c880cec60 0x7f8c880d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.407+0000 7f8c87fff640 1 --2- 172.21.15.44:0/4093473517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8815e5b0 0x7f8c880d2c00 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f8c80009e50 tx=0x7f8c8002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e24b2abfe0c177a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.408+0000 7f8c867fc640 1 -- 172.21.15.44:0/4093473517 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8002c9e0 con 0x7f8c8815e5b0 2024-04-02T13:29:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.408+0000 7f8c867fc640 1 -- 172.21.15.44:0/4093473517 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c8002cb80 con 0x7f8c8815e5b0 2024-04-02T13:29:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.408+0000 7f8c867fc640 1 -- 172.21.15.44:0/4093473517 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c800343f0 con 0x7f8c8815e5b0 2024-04-02T13:29:19.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.408+0000 7f8c8dde2640 1 -- 172.21.15.44:0/4093473517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8815e5b0 msgr2=0x7f8c880d2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.408+0000 7f8c8dde2640 1 --2- 172.21.15.44:0/4093473517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8815e5b0 0x7f8c880d2c00 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f8c80009e50 tx=0x7f8c8002be40 comp rx=0 tx=0).stop 2024-04-02T13:29:19.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.408+0000 7f8c8dde2640 1 -- 172.21.15.44:0/4093473517 shutdown_connections 2024-04-02T13:29:19.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.408+0000 7f8c8dde2640 1 --2- 172.21.15.44:0/4093473517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8815e5b0 0x7f8c880d2c00 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.408+0000 7f8c8dde2640 1 --2- 172.21.15.44:0/4093473517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c880cec60 0x7f8c880d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.408+0000 7f8c8dde2640 1 -- 172.21.15.44:0/4093473517 >> 172.21.15.44:0/4093473517 conn(0x7f8c8809d3b0 msgr2=0x7f8c880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:19.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.408+0000 7f8c8dde2640 1 -- 172.21.15.44:0/4093473517 shutdown_connections 2024-04-02T13:29:19.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.408+0000 7f8c8dde2640 1 -- 172.21.15.44:0/4093473517 wait complete. 2024-04-02T13:29:19.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.409+0000 7f8c8dde2640 1 Processor -- start 2024-04-02T13:29:19.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.409+0000 7f8c8dde2640 1 -- start start 2024-04-02T13:29:19.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.410+0000 7f8c8dde2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c880cec60 0x7f8c880cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.410+0000 7f8c8dde2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8815e1d0 0x7f8c881485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.410+0000 7f8c8dde2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8815e5b0 0x7f8c880cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.410+0000 7f8c8dde2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8c880b4e60 con 0x7f8c8815e5b0 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.410+0000 7f8c8dde2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8c880b4ce0 con 0x7f8c880cec60 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.410+0000 7f8c8dde2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8c880b4fe0 con 0x7f8c8815e1d0 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.410+0000 7f8c87fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8815e5b0 0x7f8c880cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.410+0000 7f8c87fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8815e5b0 0x7f8c880cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49196/0 (socket says 172.21.15.44:49196) 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.410+0000 7f8c87fff640 1 -- 172.21.15.44:0/3623233419 learned_addr learned my addr 172.21.15.44:0/3623233419 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.410+0000 7f8c877fe640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c880cec60 0x7f8c880cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.410+0000 7f8c86ffd640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8815e1d0 0x7f8c881485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.411+0000 7f8c87fff640 1 -- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8815e1d0 msgr2=0x7f8c881485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.411+0000 7f8c87fff640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8815e1d0 0x7f8c881485c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.411+0000 7f8c87fff640 1 -- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c880cec60 msgr2=0x7f8c880cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.411+0000 7f8c87fff640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c880cec60 0x7f8c880cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.411+0000 7f8c87fff640 1 -- 172.21.15.44:0/3623233419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c880cb6f0 con 0x7f8c8815e5b0 2024-04-02T13:29:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.411+0000 7f8c86ffd640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8815e1d0 0x7f8c881485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.411+0000 7f8c87fff640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8815e5b0 0x7f8c880cb1b0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f8c80036040 tx=0x7f8c80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.411+0000 7f8c6ffff640 1 -- 172.21.15.44:0/3623233419 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c80043020 con 0x7f8c8815e5b0 2024-04-02T13:29:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.411+0000 7f8c6ffff640 1 -- 172.21.15.44:0/3623233419 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c8003e450 con 0x7f8c8815e5b0 2024-04-02T13:29:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.411+0000 7f8c8dde2640 1 -- 172.21.15.44:0/3623233419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c880cb990 con 0x7f8c8815e5b0 2024-04-02T13:29:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.411+0000 7f8c6ffff640 1 -- 172.21.15.44:0/3623233419 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8003e860 con 0x7f8c8815e5b0 2024-04-02T13:29:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.411+0000 7f8c8dde2640 1 -- 172.21.15.44:0/3623233419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c880cbf50 con 0x7f8c8815e5b0 2024-04-02T13:29:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.412+0000 7f8c8dde2640 1 -- 172.21.15.44:0/3623233419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c880684f0 con 0x7f8c8815e5b0 2024-04-02T13:29:19.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.412+0000 7f8c6ffff640 1 -- 172.21.15.44:0/3623233419 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8c80046020 con 0x7f8c8815e5b0 2024-04-02T13:29:19.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.413+0000 7f8c6ffff640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8c7406c8f0 0x7f8c7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.413+0000 7f8c6ffff640 1 -- 172.21.15.44:0/3623233419 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f8c800ae510 con 0x7f8c8815e5b0 2024-04-02T13:29:19.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.413+0000 7f8c877fe640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8c7406c8f0 0x7f8c7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.413+0000 7f8c877fe640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8c7406c8f0 0x7f8c7406edb0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f8c880cc4e0 tx=0x7f8c78007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:19.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.417+0000 7f8c6ffff640 1 -- 172.21.15.44:0/3623233419 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8c8004d030 con 0x7f8c8815e5b0 2024-04-02T13:29:19.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.608+0000 7f8c8dde2640 1 -- 172.21.15.44:0/3623233419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8c880d47f0 con 0x7f8c8815e5b0 2024-04-02T13:29:19.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.608+0000 7f8c6ffff640 1 -- 172.21.15.44:0/3623233419 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7f8c8007c820 con 0x7f8c8815e5b0 2024-04-02T13:29:19.612 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:29:19.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.610+0000 7f8c8dde2640 1 -- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8c7406c8f0 msgr2=0x7f8c7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.610+0000 7f8c8dde2640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8c7406c8f0 0x7f8c7406edb0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f8c880cc4e0 tx=0x7f8c78007e30 comp rx=0 tx=0).stop 2024-04-02T13:29:19.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.610+0000 7f8c8dde2640 1 -- 172.21.15.44:0/3623233419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8815e5b0 msgr2=0x7f8c880cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.610+0000 7f8c8dde2640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8815e5b0 0x7f8c880cb1b0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f8c80036040 tx=0x7f8c80004a40 comp rx=0 tx=0).stop 2024-04-02T13:29:19.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.610+0000 7f8c8dde2640 1 -- 172.21.15.44:0/3623233419 shutdown_connections 2024-04-02T13:29:19.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.610+0000 7f8c8dde2640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8c7406c8f0 0x7f8c7406edb0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.610+0000 7f8c8dde2640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8815e5b0 0x7f8c880cb1b0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.610+0000 7f8c8dde2640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8815e1d0 0x7f8c881485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.610+0000 7f8c8dde2640 1 --2- 172.21.15.44:0/3623233419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c880cec60 0x7f8c880cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.610+0000 7f8c8dde2640 1 -- 172.21.15.44:0/3623233419 >> 172.21.15.44:0/3623233419 conn(0x7f8c8809d3b0 msgr2=0x7f8c880b5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:19.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.611+0000 7f8c8dde2640 1 -- 172.21.15.44:0/3623233419 shutdown_connections 2024-04-02T13:29:19.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.611+0000 7f8c8dde2640 1 -- 172.21.15.44:0/3623233419 wait complete. 2024-04-02T13:29:19.625 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:29:19.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.727+0000 7f56a7270640 1 Processor -- start 2024-04-02T13:29:19.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.727+0000 7f56a7270640 1 -- start start 2024-04-02T13:29:19.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.728+0000 7f56a7270640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f56a00d2dd0 0x7f56a00b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.728+0000 7f56a7270640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a00cf1f0 con 0x7f56a015e1d0 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.728+0000 7f56a7270640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a00cf4d0 con 0x7f56a00cec60 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.728+0000 7f56a7270640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f56a00cfe00 con 0x7f56a00d2dd0 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.728+0000 7f56a500c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f56a015e1d0 0x7f56a00d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40478/0 (socket says 172.21.15.44:40478) 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.728+0000 7f56a500c640 1 -- 172.21.15.44:0/1286930312 learned_addr learned my addr 172.21.15.44:0/1286930312 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3677778426 0 0) 0x7f56a00cf1f0 con 0x7f56a015e1d0 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f56a580d640 1 --2- 172.21.15.44:0/1286930312 >> v2:172.21.15.69:3300/0 conn(0x7f56a00d2dd0 0x7f56a00b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5688003520 con 0x7f56a015e1d0 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3804866616 0 0) 0x7f5688003520 con 0x7f56a015e1d0 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 >> v2:172.21.15.69:3300/0 conn(0x7f56a00d2dd0 msgr2=0x7f56a00b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 --2- 172.21.15.44:0/1286930312 >> v2:172.21.15.69:3300/0 conn(0x7f56a00d2dd0 0x7f56a00b4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 >> v1:172.21.15.69:6789/0 conn(0x7f56a00cec60 legacy=0x7f56a00d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a00cf7b0 con 0x7f56a015e1d0 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f5698001670 con 0x7f56a015e1d0 2024-04-02T13:29:19.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 >> v1:172.21.15.44:6789/0 conn(0x7f56a015e1d0 legacy=0x7f56a00d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 --2- 172.21.15.44:0/1286930312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56880039b0 0x7f5688003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 --2- 172.21.15.44:0/1286930312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5688004800 0x7f5688004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 --2- 172.21.15.44:0/1286930312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56880051c0 0x7f568800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f56a00cfe00 con 0x7f56880051c0 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f568800c160 con 0x7f56880039b0 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f568800c360 con 0x7f5688004800 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f56a580d640 1 --2- 172.21.15.44:0/1286930312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56880051c0 0x7f568800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f56a500c640 1 --2- 172.21.15.44:0/1286930312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56880039b0 0x7f5688003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f56a480b640 1 --2- 172.21.15.44:0/1286930312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5688004800 0x7f5688004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f56a580d640 1 -- 172.21.15.44:0/1286930312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5688004800 msgr2=0x7f5688004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f56a580d640 1 --2- 172.21.15.44:0/1286930312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5688004800 0x7f5688004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f56a580d640 1 -- 172.21.15.44:0/1286930312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56880039b0 msgr2=0x7f5688003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f56a580d640 1 --2- 172.21.15.44:0/1286930312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56880039b0 0x7f5688003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.729+0000 7f56a580d640 1 -- 172.21.15.44:0/1286930312 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5688003520 con 0x7f56880051c0 2024-04-02T13:29:19.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.730+0000 7f56a580d640 1 -- 172.21.15.44:0/1286930312 reap_dead start 2024-04-02T13:29:19.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.730+0000 7f56a580d640 1 --2- 172.21.15.44:0/1286930312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56880051c0 0x7f568800b9c0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f5690009f90 tx=0x7f5688005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d107a87cd7721209 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:19.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.730+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5690009c10 con 0x7f56880051c0 2024-04-02T13:29:19.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.730+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f569002cc20 con 0x7f56880051c0 2024-04-02T13:29:19.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.730+0000 7f569ffff640 1 -- 172.21.15.44:0/1286930312 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5690003990 con 0x7f56880051c0 2024-04-02T13:29:19.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.731+0000 7f56a7270640 1 -- 172.21.15.44:0/1286930312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56880051c0 msgr2=0x7f568800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.731+0000 7f56a7270640 1 --2- 172.21.15.44:0/1286930312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56880051c0 0x7f568800b9c0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f5690009f90 tx=0x7f5688005830 comp rx=0 tx=0).stop 2024-04-02T13:29:19.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.731+0000 7f56a7270640 1 -- 172.21.15.44:0/1286930312 shutdown_connections 2024-04-02T13:29:19.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.731+0000 7f56a7270640 1 --2- 172.21.15.44:0/1286930312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56880051c0 0x7f568800b9c0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.731+0000 7f56a7270640 1 -- 172.21.15.44:0/1286930312 >> 172.21.15.44:0/1286930312 conn(0x7f56a009d3b0 msgr2=0x7f56a00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:19.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.731+0000 7f56a7270640 1 -- 172.21.15.44:0/1286930312 shutdown_connections 2024-04-02T13:29:19.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.731+0000 7f56a7270640 1 -- 172.21.15.44:0/1286930312 wait complete. 2024-04-02T13:29:19.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.732+0000 7f56a7270640 1 Processor -- start 2024-04-02T13:29:19.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.732+0000 7f56a7270640 1 -- start start 2024-04-02T13:29:19.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a7270640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56880051c0 0x7f56a00a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a7270640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56a009dd20 0x7f56a0150640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a7270640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56a00cc200 0x7f56a00cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a7270640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f56a015e730 con 0x7f56a009dd20 2024-04-02T13:29:19.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a7270640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f56a015e5b0 con 0x7f56880051c0 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a7270640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f56a015e8b0 con 0x7f56a00cc200 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a480b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56a009dd20 0x7f56a0150640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a480b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56a009dd20 0x7f56a0150640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49218/0 (socket says 172.21.15.44:49218) 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a480b640 1 -- 172.21.15.44:0/3189574614 learned_addr learned my addr 172.21.15.44:0/3189574614 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a500c640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56880051c0 0x7f56a00a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a580d640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56a00cc200 0x7f56a00cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a480b640 1 -- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56a00cc200 msgr2=0x7f56a00cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a480b640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56a00cc200 0x7f56a00cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a480b640 1 -- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56880051c0 msgr2=0x7f56a00a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a480b640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56880051c0 0x7f56a00a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a480b640 1 -- 172.21.15.44:0/3189574614 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a015aa80 con 0x7f56a009dd20 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a500c640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56880051c0 0x7f56a00a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a580d640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56a00cc200 0x7f56a00cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f56a480b640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56a009dd20 0x7f56a0150640 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f569400f620 tx=0x7f569400fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.733+0000 7f569dffb640 1 -- 172.21.15.44:0/3189574614 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56940052e0 con 0x7f56a009dd20 2024-04-02T13:29:19.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.734+0000 7f569dffb640 1 -- 172.21.15.44:0/3189574614 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5694018070 con 0x7f56a009dd20 2024-04-02T13:29:19.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.734+0000 7f56a7270640 1 -- 172.21.15.44:0/3189574614 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56a009e180 con 0x7f56a009dd20 2024-04-02T13:29:19.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.734+0000 7f569dffb640 1 -- 172.21.15.44:0/3189574614 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5694003810 con 0x7f56a009dd20 2024-04-02T13:29:19.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.734+0000 7f56a7270640 1 -- 172.21.15.44:0/3189574614 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56a00a1300 con 0x7f56a009dd20 2024-04-02T13:29:19.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.735+0000 7f56a7270640 1 -- 172.21.15.44:0/3189574614 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56a0073730 con 0x7f56a009dd20 2024-04-02T13:29:19.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.735+0000 7f569dffb640 1 -- 172.21.15.44:0/3189574614 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f56940039b0 con 0x7f56a009dd20 2024-04-02T13:29:19.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.735+0000 7f569dffb640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f568c06c8f0 0x7f568c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:19.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.736+0000 7f569dffb640 1 -- 172.21.15.44:0/3189574614 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f569408ba00 con 0x7f56a009dd20 2024-04-02T13:29:19.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.736+0000 7f56a500c640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f568c06c8f0 0x7f568c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:19.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.736+0000 7f56a500c640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f568c06c8f0 0x7f568c06edb0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5698001eb0 tx=0x7f5698009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:19.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.740+0000 7f569dffb640 1 -- 172.21.15.44:0/3189574614 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f56a0073730 con 0x7f56a009dd20 2024-04-02T13:29:19.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.923+0000 7f56a7270640 1 -- 172.21.15.44:0/3189574614 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f56a00684f0 con 0x7f56a009dd20 2024-04-02T13:29:19.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.923+0000 7f569dffb640 1 -- 172.21.15.44:0/3189574614 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) ==== 104+0+11 (secure 0 0 0) 0x7f5694059d30 con 0x7f56a009dd20 2024-04-02T13:29:19.927 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:29:19.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.925+0000 7f56a7270640 1 -- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f568c06c8f0 msgr2=0x7f568c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.925+0000 7f56a7270640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f568c06c8f0 0x7f568c06edb0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5698001eb0 tx=0x7f5698009040 comp rx=0 tx=0).stop 2024-04-02T13:29:19.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.925+0000 7f56a7270640 1 -- 172.21.15.44:0/3189574614 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56a009dd20 msgr2=0x7f56a0150640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:19.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.925+0000 7f56a7270640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56a009dd20 0x7f56a0150640 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f569400f620 tx=0x7f569400fae0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.925+0000 7f56a7270640 1 -- 172.21.15.44:0/3189574614 shutdown_connections 2024-04-02T13:29:19.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.925+0000 7f56a7270640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f568c06c8f0 0x7f568c06edb0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.925+0000 7f56a7270640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56a00cc200 0x7f56a00cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.925+0000 7f56a7270640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56a009dd20 0x7f56a0150640 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.925+0000 7f56a7270640 1 --2- 172.21.15.44:0/3189574614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56880051c0 0x7f56a00a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:19.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.925+0000 7f56a7270640 1 -- 172.21.15.44:0/3189574614 >> 172.21.15.44:0/3189574614 conn(0x7f56a009d3b0 msgr2=0x7f56a006d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:19.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.925+0000 7f56a7270640 1 -- 172.21.15.44:0/3189574614 shutdown_connections 2024-04-02T13:29:19.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:19.925+0000 7f56a7270640 1 -- 172.21.15.44:0/3189574614 wait complete. 2024-04-02T13:29:19.939 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:29:20.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.039+0000 7f2fadbf0640 1 Processor -- start 2024-04-02T13:29:20.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.039+0000 7f2fadbf0640 1 -- start start 2024-04-02T13:29:20.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.040+0000 7f2fadbf0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80cec60 0x7f2fa80d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.040+0000 7f2fadbf0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2fa815e5b0 0x7f2fa80d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.040+0000 7f2fadbf0640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa80cf1f0 con 0x7f2fa815e1d0 2024-04-02T13:29:20.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.040+0000 7f2fadbf0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2fa80cfe00 con 0x7f2fa815e5b0 2024-04-02T13:29:20.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.040+0000 7f2fadbf0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa80cf4d0 con 0x7f2fa80cec60 2024-04-02T13:29:20.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.040+0000 7f2fa77fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f2fa815e1d0 0x7f2fa80d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40494/0 (socket says 172.21.15.44:40494) 2024-04-02T13:29:20.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.040+0000 7f2fa77fe640 1 -- 172.21.15.44:0/210958568 learned_addr learned my addr 172.21.15.44:0/210958568 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:20.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3956201218 0 0) 0x7f2fa80cf1f0 con 0x7f2fa815e1d0 2024-04-02T13:29:20.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa6ffd640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80cec60 0x7f2fa80d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa7fff640 1 --2- 172.21.15.44:0/210958568 >> v2:172.21.15.69:3300/0 conn(0x7f2fa815e5b0 0x7f2fa80d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f90003520 con 0x7f2fa815e1d0 2024-04-02T13:29:20.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4221692022 0 0) 0x7f2f90003520 con 0x7f2fa815e1d0 2024-04-02T13:29:20.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80cec60 msgr2=0x7f2fa80d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:20.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80cec60 0x7f2fa80d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 >> v2:172.21.15.69:3300/0 conn(0x7f2fa815e5b0 msgr2=0x7f2fa80d2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:20.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 --2- 172.21.15.44:0/210958568 >> v2:172.21.15.69:3300/0 conn(0x7f2fa815e5b0 0x7f2fa80d2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa80cf7b0 con 0x7f2fa815e1d0 2024-04-02T13:29:20.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f2f98001670 con 0x7f2fa815e1d0 2024-04-02T13:29:20.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 >> v1:172.21.15.44:6789/0 conn(0x7f2fa815e1d0 legacy=0x7f2fa80d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f900039b0 0x7f2f90003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80cec60 0x7f2f90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f90004d40 0x7f2f900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa80cfe00 con 0x7f2f90004d40 2024-04-02T13:29:20.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa80cf4d0 con 0x7f2f900039b0 2024-04-02T13:29:20.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9000b9c0 con 0x7f2fa80cec60 2024-04-02T13:29:20.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa7fff640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f90004d40 0x7f2f900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa6ffd640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80cec60 0x7f2f90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa77fe640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f900039b0 0x7f2f90003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa7fff640 1 -- 172.21.15.44:0/210958568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80cec60 msgr2=0x7f2f90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:20.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.041+0000 7f2fa7fff640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80cec60 0x7f2f90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.042+0000 7f2fa7fff640 1 -- 172.21.15.44:0/210958568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f900039b0 msgr2=0x7f2f90003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:20.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.042+0000 7f2fa7fff640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f900039b0 0x7f2f90003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.042+0000 7f2fa7fff640 1 -- 172.21.15.44:0/210958568 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f90003520 con 0x7f2f90004d40 2024-04-02T13:29:20.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.042+0000 7f2fa6ffd640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80cec60 0x7f2f90004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:29:20.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.042+0000 7f2fa77fe640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f900039b0 0x7f2f90003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:29:20.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.042+0000 7f2fa7fff640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f90004d40 0x7f2f900051c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0009fc0 tx=0x7f2f90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2fba15fe066f77 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:20.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.042+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0039070 con 0x7f2f90004d40 2024-04-02T13:29:20.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.042+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa002cc60 con 0x7f2f90004d40 2024-04-02T13:29:20.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.042+0000 7f2fa67fc640 1 -- 172.21.15.44:0/210958568 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0009c10 con 0x7f2f90004d40 2024-04-02T13:29:20.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.043+0000 7f2fadbf0640 1 -- 172.21.15.44:0/210958568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f90004d40 msgr2=0x7f2f900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.043+0000 7f2fadbf0640 1 --2- 172.21.15.44:0/210958568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f90004d40 0x7f2f900051c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0009fc0 tx=0x7f2f90005830 comp rx=0 tx=0).stop 2024-04-02T13:29:20.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.043+0000 7f2fa7fff640 1 -- 172.21.15.44:0/210958568 reap_dead start 2024-04-02T13:29:20.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.043+0000 7f2fadbf0640 1 -- 172.21.15.44:0/210958568 shutdown_connections 2024-04-02T13:29:20.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.043+0000 7f2fadbf0640 1 -- 172.21.15.44:0/210958568 >> 172.21.15.44:0/210958568 conn(0x7f2fa809d3b0 msgr2=0x7f2fa80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:20.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.043+0000 7f2fadbf0640 1 -- 172.21.15.44:0/210958568 shutdown_connections 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.043+0000 7f2fadbf0640 1 -- 172.21.15.44:0/210958568 wait complete. 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.044+0000 7f2fadbf0640 1 Processor -- start 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.044+0000 7f2fadbf0640 1 -- start start 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fadbf0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80a0af0 0x7f2fa8160230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fadbf0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fa8160770 0x7f2fa815cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fadbf0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa80a0ed0 0x7f2fa815d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fadbf0640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa80d2840 con 0x7f2fa80a0ed0 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fadbf0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa80d26c0 con 0x7f2fa8160770 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fadbf0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa80d29c0 con 0x7f2fa80a0af0 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fa7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa80a0ed0 0x7f2fa815d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fa7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa80a0ed0 0x7f2fa815d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49234/0 (socket says 172.21.15.44:49234) 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fa7fff640 1 -- 172.21.15.44:0/4220699194 learned_addr learned my addr 172.21.15.44:0/4220699194 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fa77fe640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80a0af0 0x7f2fa8160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fa6ffd640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fa8160770 0x7f2fa815cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fa7fff640 1 -- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80a0af0 msgr2=0x7f2fa8160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fa7fff640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80a0af0 0x7f2fa8160230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fa7fff640 1 -- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fa8160770 msgr2=0x7f2fa815cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fa7fff640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fa8160770 0x7f2fa815cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.045+0000 7f2fa7fff640 1 -- 172.21.15.44:0/4220699194 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa809ef50 con 0x7f2fa80a0ed0 2024-04-02T13:29:20.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.046+0000 7f2fa7fff640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa80a0ed0 0x7f2fa815d0d0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000e230 tx=0x7f2fa0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:20.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.046+0000 7f2f8ffff640 1 -- 172.21.15.44:0/4220699194 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0039030 con 0x7f2fa80a0ed0 2024-04-02T13:29:20.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.046+0000 7f2f8ffff640 1 -- 172.21.15.44:0/4220699194 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa003a040 con 0x7f2fa80a0ed0 2024-04-02T13:29:20.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.046+0000 7f2fadbf0640 1 -- 172.21.15.44:0/4220699194 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa809f230 con 0x7f2fa80a0ed0 2024-04-02T13:29:20.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.046+0000 7f2f8ffff640 1 -- 172.21.15.44:0/4220699194 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98007d60 con 0x7f2fa80a0ed0 2024-04-02T13:29:20.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.046+0000 7f2fadbf0640 1 -- 172.21.15.44:0/4220699194 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa809f7f0 con 0x7f2fa80a0ed0 2024-04-02T13:29:20.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.047+0000 7f2fadbf0640 1 -- 172.21.15.44:0/4220699194 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa80684f0 con 0x7f2fa80a0ed0 2024-04-02T13:29:20.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.047+0000 7f2f8ffff640 1 -- 172.21.15.44:0/4220699194 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2fa0005d10 con 0x7f2fa80a0ed0 2024-04-02T13:29:20.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.048+0000 7f2f8ffff640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f9406c920 0x7f2f9406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.048+0000 7f2f8ffff640 1 -- 172.21.15.44:0/4220699194 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f2fa00ad540 con 0x7f2fa80a0ed0 2024-04-02T13:29:20.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.048+0000 7f2fa77fe640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f9406c920 0x7f2f9406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.049+0000 7f2fa77fe640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f9406c920 0x7f2f9406ede0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2f98000c00 tx=0x7f2f98001c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:20.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.053+0000 7f2f8ffff640 1 -- 172.21.15.44:0/4220699194 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2fa007b820 con 0x7f2fa80a0ed0 2024-04-02T13:29:20.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.234+0000 7f2fadbf0640 1 -- 172.21.15.44:0/4220699194 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2fa80b8580 con 0x7f2fa80a0ed0 2024-04-02T13:29:20.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.235+0000 7f2f8ffff640 1 -- 172.21.15.44:0/4220699194 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) ==== 100+0+11 (secure 0 0 0) 0x7f2fa007b120 con 0x7f2fa80a0ed0 2024-04-02T13:29:20.239 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:29:20.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.236+0000 7f2fadbf0640 1 -- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f9406c920 msgr2=0x7f2f9406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.236+0000 7f2fadbf0640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f9406c920 0x7f2f9406ede0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2f98000c00 tx=0x7f2f98001c00 comp rx=0 tx=0).stop 2024-04-02T13:29:20.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.236+0000 7f2fadbf0640 1 -- 172.21.15.44:0/4220699194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa80a0ed0 msgr2=0x7f2fa815d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.236+0000 7f2fadbf0640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa80a0ed0 0x7f2fa815d0d0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000e230 tx=0x7f2fa0009750 comp rx=0 tx=0).stop 2024-04-02T13:29:20.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.236+0000 7f2fadbf0640 1 -- 172.21.15.44:0/4220699194 shutdown_connections 2024-04-02T13:29:20.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.237+0000 7f2fadbf0640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f9406c920 0x7f2f9406ede0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.237+0000 7f2fadbf0640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa80a0ed0 0x7f2fa815d0d0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.237+0000 7f2fadbf0640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fa8160770 0x7f2fa815cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.237+0000 7f2fadbf0640 1 --2- 172.21.15.44:0/4220699194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa80a0af0 0x7f2fa8160230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.237+0000 7f2fadbf0640 1 -- 172.21.15.44:0/4220699194 >> 172.21.15.44:0/4220699194 conn(0x7f2fa809d3b0 msgr2=0x7f2fa80b7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:20.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.237+0000 7f2fadbf0640 1 -- 172.21.15.44:0/4220699194 shutdown_connections 2024-04-02T13:29:20.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.237+0000 7f2fadbf0640 1 -- 172.21.15.44:0/4220699194 wait complete. 2024-04-02T13:29:20.251 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:29:20.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.350+0000 7f33d23ba640 1 Processor -- start 2024-04-02T13:29:20.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.350+0000 7f33d23ba640 1 -- start start 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.351+0000 7f33d23ba640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f33cc0b47b0 0x7f33cc0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.351+0000 7f33d23ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33cc0b5200 0x7f33cc0b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.351+0000 7f33d23ba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33cc0d4d70 con 0x7f33cc0d28f0 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.351+0000 7f33d23ba640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f33cc0d5050 con 0x7f33cc0b47b0 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.351+0000 7f33d23ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f33cc0d23a0 con 0x7f33cc0b5200 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.351+0000 7f33cb7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f33cc0b47b0 0x7f33cc0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.351+0000 7f33cb7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f33cc0b47b0 0x7f33cc0d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49250/0 (socket says 172.21.15.44:49250) 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.351+0000 7f33cb7fe640 1 -- 172.21.15.44:0/1983310003 learned_addr learned my addr 172.21.15.44:0/1983310003 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.352+0000 7f33d0957640 1 --2- 172.21.15.44:0/1983310003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33cc0b5200 0x7f33cc0b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.352+0000 7f33cb7fe640 1 -- 172.21.15.44:0/1983310003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33cc0b5200 msgr2=0x7f33cc0b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.352+0000 7f33cb7fe640 1 --2- 172.21.15.44:0/1983310003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33cc0b5200 0x7f33cc0b7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.352+0000 7f33cb7fe640 1 -- 172.21.15.44:0/1983310003 >> v1:172.21.15.69:6789/0 conn(0x7f33cc0d28f0 legacy=0x7f33cc0b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.352+0000 7f33cb7fe640 1 -- 172.21.15.44:0/1983310003 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33cc0d2030 con 0x7f33cc0b47b0 2024-04-02T13:29:20.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.352+0000 7f33cb7fe640 1 --2- 172.21.15.44:0/1983310003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33cc0b47b0 0x7f33cc0d2cd0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f33bc009870 tx=0x7f33bc02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef1463e08b2c88f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:20.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.352+0000 7f33caffd640 1 -- 172.21.15.44:0/1983310003 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33bc038070 con 0x7f33cc0b47b0 2024-04-02T13:29:20.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.353+0000 7f33caffd640 1 -- 172.21.15.44:0/1983310003 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33bc031740 con 0x7f33cc0b47b0 2024-04-02T13:29:20.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.353+0000 7f33caffd640 1 -- 172.21.15.44:0/1983310003 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33bc02f050 con 0x7f33cc0b47b0 2024-04-02T13:29:20.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.353+0000 7f33d23ba640 1 -- 172.21.15.44:0/1983310003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33cc0b47b0 msgr2=0x7f33cc0d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.353+0000 7f33d23ba640 1 --2- 172.21.15.44:0/1983310003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33cc0b47b0 0x7f33cc0d2cd0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f33bc009870 tx=0x7f33bc02b7e0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.353+0000 7f33d23ba640 1 -- 172.21.15.44:0/1983310003 shutdown_connections 2024-04-02T13:29:20.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.353+0000 7f33d23ba640 1 --2- 172.21.15.44:0/1983310003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33cc0b47b0 0x7f33cc0d2cd0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.353+0000 7f33d23ba640 1 --2- 172.21.15.44:0/1983310003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33cc0b5200 0x7f33cc0b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.353+0000 7f33d23ba640 1 -- 172.21.15.44:0/1983310003 >> 172.21.15.44:0/1983310003 conn(0x7f33cc0d4120 msgr2=0x7f33cc0d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:20.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.353+0000 7f33d23ba640 1 -- 172.21.15.44:0/1983310003 shutdown_connections 2024-04-02T13:29:20.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.353+0000 7f33d23ba640 1 -- 172.21.15.44:0/1983310003 wait complete. 2024-04-02T13:29:20.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.354+0000 7f33d23ba640 1 Processor -- start 2024-04-02T13:29:20.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.354+0000 7f33d23ba640 1 -- start start 2024-04-02T13:29:20.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.355+0000 7f33d23ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33cc0b47b0 0x7f33cc0b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.355+0000 7f33d23ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33cc0b5200 0x7f33cc15c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.355+0000 7f33d23ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33cc0d28f0 0x7f33cc15cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.355+0000 7f33d23ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc0d1260 con 0x7f33cc0b47b0 2024-04-02T13:29:20.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.355+0000 7f33d23ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc0b5020 con 0x7f33cc0b5200 2024-04-02T13:29:20.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.355+0000 7f33d23ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f33cc0d13e0 con 0x7f33cc0d28f0 2024-04-02T13:29:20.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.355+0000 7f33cbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33cc0b47b0 0x7f33cc0b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.355+0000 7f33cbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33cc0b47b0 0x7f33cc0b5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49260/0 (socket says 172.21.15.44:49260) 2024-04-02T13:29:20.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.355+0000 7f33cbfff640 1 -- 172.21.15.44:0/2660136292 learned_addr learned my addr 172.21.15.44:0/2660136292 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:20.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.355+0000 7f33cb7fe640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33cc0b5200 0x7f33cc15c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.355+0000 7f33d0957640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33cc0d28f0 0x7f33cc15cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.355+0000 7f33cbfff640 1 -- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33cc0d28f0 msgr2=0x7f33cc15cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.356+0000 7f33cbfff640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33cc0d28f0 0x7f33cc15cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.356+0000 7f33cbfff640 1 -- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33cc0b5200 msgr2=0x7f33cc15c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.356+0000 7f33cbfff640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33cc0b5200 0x7f33cc15c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.356+0000 7f33cbfff640 1 -- 172.21.15.44:0/2660136292 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33cc158080 con 0x7f33cc0b47b0 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.356+0000 7f33cb7fe640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33cc0b5200 0x7f33cc15c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.356+0000 7f33d0957640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33cc0d28f0 0x7f33cc15cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.356+0000 7f33cbfff640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33cc0b47b0 0x7f33cc0b5bd0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f33c000c430 tx=0x7f33c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.356+0000 7f33c8ff9640 1 -- 172.21.15.44:0/2660136292 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c0011790 con 0x7f33cc0b47b0 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.356+0000 7f33c8ff9640 1 -- 172.21.15.44:0/2660136292 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c0017070 con 0x7f33cc0b47b0 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.356+0000 7f33d23ba640 1 -- 172.21.15.44:0/2660136292 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33cc158360 con 0x7f33cc0b47b0 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.356+0000 7f33c8ff9640 1 -- 172.21.15.44:0/2660136292 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c0011b70 con 0x7f33cc0b47b0 2024-04-02T13:29:20.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.356+0000 7f33d23ba640 1 -- 172.21.15.44:0/2660136292 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33cc158920 con 0x7f33cc0b47b0 2024-04-02T13:29:20.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.357+0000 7f33d23ba640 1 -- 172.21.15.44:0/2660136292 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33cc0684f0 con 0x7f33cc0b47b0 2024-04-02T13:29:20.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.358+0000 7f33c8ff9640 1 -- 172.21.15.44:0/2660136292 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f33c0011d10 con 0x7f33cc0b47b0 2024-04-02T13:29:20.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.358+0000 7f33c8ff9640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f33b806c8f0 0x7f33b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.358+0000 7f33c8ff9640 1 -- 172.21.15.44:0/2660136292 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f33c008bbb0 con 0x7f33cc0b47b0 2024-04-02T13:29:20.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.358+0000 7f33cb7fe640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f33b806c8f0 0x7f33b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.359+0000 7f33cb7fe640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f33b806c8f0 0x7f33b806edb0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f33bc035000 tx=0x7f33bc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:20.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.363+0000 7f33c8ff9640 1 -- 172.21.15.44:0/2660136292 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f33c0059e50 con 0x7f33cc0b47b0 2024-04-02T13:29:20.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.561+0000 7f33d23ba640 1 -- 172.21.15.44:0/2660136292 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f33cc0b8eb0 con 0x7f33cc0b47b0 2024-04-02T13:29:20.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.561+0000 7f33c8ff9640 1 -- 172.21.15.44:0/2660136292 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v106) ==== 80+0+0 (secure 0 0 0) 0x7f33c0059750 con 0x7f33cc0b47b0 2024-04-02T13:29:20.565 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:29:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.563+0000 7f33d23ba640 1 -- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f33b806c8f0 msgr2=0x7f33b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.563+0000 7f33d23ba640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f33b806c8f0 0x7f33b806edb0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f33bc035000 tx=0x7f33bc008030 comp rx=0 tx=0).stop 2024-04-02T13:29:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.563+0000 7f33d23ba640 1 -- 172.21.15.44:0/2660136292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33cc0b47b0 msgr2=0x7f33cc0b5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.563+0000 7f33d23ba640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33cc0b47b0 0x7f33cc0b5bd0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f33c000c430 tx=0x7f33c000c8f0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.563+0000 7f33d23ba640 1 -- 172.21.15.44:0/2660136292 shutdown_connections 2024-04-02T13:29:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.563+0000 7f33d23ba640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f33b806c8f0 0x7f33b806edb0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.563+0000 7f33d23ba640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33cc0d28f0 0x7f33cc15cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.563+0000 7f33d23ba640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33cc0b5200 0x7f33cc15c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.563+0000 7f33d23ba640 1 --2- 172.21.15.44:0/2660136292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33cc0b47b0 0x7f33cc0b5bd0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.563+0000 7f33d23ba640 1 -- 172.21.15.44:0/2660136292 >> 172.21.15.44:0/2660136292 conn(0x7f33cc0d4120 msgr2=0x7f33cc0d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.563+0000 7f33d23ba640 1 -- 172.21.15.44:0/2660136292 shutdown_connections 2024-04-02T13:29:20.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.563+0000 7f33d23ba640 1 -- 172.21.15.44:0/2660136292 wait complete. 2024-04-02T13:29:20.577 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:29:20.577 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:29:20.592 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:29:20.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.731+0000 7fb84c614640 1 Processor -- start 2024-04-02T13:29:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.731+0000 7fb84c614640 1 -- start start 2024-04-02T13:29:20.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb84c614640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb84415e4a0 0x7fb8440aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb84c614640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8440cc600 con 0x7fb8440d0c50 2024-04-02T13:29:20.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb84c614640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8440cc8e0 con 0x7fb84406d550 2024-04-02T13:29:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb84c614640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb8440ccbc0 con 0x7fb84415e4a0 2024-04-02T13:29:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb849baf640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb8440d0c50 0x7fb84406ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40508/0 (socket says 172.21.15.44:40508) 2024-04-02T13:29:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb849baf640 1 -- 172.21.15.44:0/2956968681 learned_addr learned my addr 172.21.15.44:0/2956968681 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb84a3b0640 1 --2- 172.21.15.44:0/2956968681 >> v2:172.21.15.69:3300/0 conn(0x7fb84415e4a0 0x7fb8440aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 968131189 0 0) 0x7fb8440cc600 con 0x7fb8440d0c50 2024-04-02T13:29:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb830003520 con 0x7fb8440d0c50 2024-04-02T13:29:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1664248069 0 0) 0x7fb830003520 con 0x7fb8440d0c50 2024-04-02T13:29:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 >> v2:172.21.15.69:3300/0 conn(0x7fb84415e4a0 msgr2=0x7fb8440aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 --2- 172.21.15.44:0/2956968681 >> v2:172.21.15.69:3300/0 conn(0x7fb84415e4a0 0x7fb8440aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 >> v1:172.21.15.69:6789/0 conn(0x7fb84406d550 legacy=0x7fb8440cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8440ccea0 con 0x7fb8440d0c50 2024-04-02T13:29:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fb834001670 con 0x7fb8440d0c50 2024-04-02T13:29:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 >> v1:172.21.15.44:6789/0 conn(0x7fb8440d0c50 legacy=0x7fb84406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 --2- 172.21.15.44:0/2956968681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8300039b0 0x7fb830003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 --2- 172.21.15.44:0/2956968681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb830004800 0x7fb830004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 --2- 172.21.15.44:0/2956968681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8300051c0 0x7fb83000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb8440ccbc0 con 0x7fb8300051c0 2024-04-02T13:29:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb83000c060 con 0x7fb830004800 2024-04-02T13:29:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.732+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb83000c260 con 0x7fb8300039b0 2024-04-02T13:29:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.733+0000 7fb849baf640 1 --2- 172.21.15.44:0/2956968681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb830004800 0x7fb830004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.733+0000 7fb84abb1640 1 --2- 172.21.15.44:0/2956968681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8300051c0 0x7fb83000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.733+0000 7fb84a3b0640 1 --2- 172.21.15.44:0/2956968681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8300039b0 0x7fb830003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.733+0000 7fb84abb1640 1 -- 172.21.15.44:0/2956968681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8300039b0 msgr2=0x7fb830003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.733+0000 7fb84abb1640 1 --2- 172.21.15.44:0/2956968681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8300039b0 0x7fb830003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.733+0000 7fb84abb1640 1 -- 172.21.15.44:0/2956968681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb830004800 msgr2=0x7fb830004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.733+0000 7fb84abb1640 1 --2- 172.21.15.44:0/2956968681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb830004800 0x7fb830004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.733+0000 7fb84abb1640 1 -- 172.21.15.44:0/2956968681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb830003520 con 0x7fb8300051c0 2024-04-02T13:29:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.733+0000 7fb84abb1640 1 -- 172.21.15.44:0/2956968681 reap_dead start 2024-04-02T13:29:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.734+0000 7fb84abb1640 1 --2- 172.21.15.44:0/2956968681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8300051c0 0x7fb83000b9c0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fb8440cc5b0 tx=0x7fb830005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b2002c3d8d96da server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.734+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c001a70 con 0x7fb8300051c0 2024-04-02T13:29:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.734+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb83c003b70 con 0x7fb8300051c0 2024-04-02T13:29:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.734+0000 7fb8493ae640 1 -- 172.21.15.44:0/2956968681 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c0050d0 con 0x7fb8300051c0 2024-04-02T13:29:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.734+0000 7fb84c614640 1 -- 172.21.15.44:0/2956968681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8300051c0 msgr2=0x7fb83000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.734+0000 7fb84c614640 1 --2- 172.21.15.44:0/2956968681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8300051c0 0x7fb83000b9c0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fb8440cc5b0 tx=0x7fb830005830 comp rx=0 tx=0).stop 2024-04-02T13:29:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.734+0000 7fb84c614640 1 -- 172.21.15.44:0/2956968681 shutdown_connections 2024-04-02T13:29:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.734+0000 7fb84c614640 1 --2- 172.21.15.44:0/2956968681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8300051c0 0x7fb83000b9c0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.734+0000 7fb84c614640 1 -- 172.21.15.44:0/2956968681 >> 172.21.15.44:0/2956968681 conn(0x7fb84409aa20 msgr2=0x7fb84409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.734+0000 7fb84c614640 1 -- 172.21.15.44:0/2956968681 shutdown_connections 2024-04-02T13:29:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.734+0000 7fb84c614640 1 -- 172.21.15.44:0/2956968681 wait complete. 2024-04-02T13:29:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.736+0000 7fb84c614640 1 Processor -- start 2024-04-02T13:29:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.736+0000 7fb84c614640 1 -- start start 2024-04-02T13:29:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.736+0000 7fb84c614640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8300051c0 0x7fb84415a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.736+0000 7fb84c614640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8440c65b0 0x7fb8440b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.736+0000 7fb84c614640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8440c6a30 0x7fb8440c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.736+0000 7fb84c614640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb84415e880 con 0x7fb8300051c0 2024-04-02T13:29:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.736+0000 7fb84c614640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb8440cd460 con 0x7fb8440c6a30 2024-04-02T13:29:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.736+0000 7fb84c614640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb8440d10b0 con 0x7fb8440c65b0 2024-04-02T13:29:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.736+0000 7fb84a3b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8300051c0 0x7fb84415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.736+0000 7fb84a3b0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8300051c0 0x7fb84415a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49278/0 (socket says 172.21.15.44:49278) 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.736+0000 7fb84a3b0640 1 -- 172.21.15.44:0/3352877421 learned_addr learned my addr 172.21.15.44:0/3352877421 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.737+0000 7fb849baf640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8440c65b0 0x7fb8440b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.737+0000 7fb84abb1640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8440c6a30 0x7fb8440c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.737+0000 7fb84a3b0640 1 -- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8440c65b0 msgr2=0x7fb8440b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.737+0000 7fb84a3b0640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8440c65b0 0x7fb8440b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.737+0000 7fb84a3b0640 1 -- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8440c6a30 msgr2=0x7fb8440c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.737+0000 7fb84a3b0640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8440c6a30 0x7fb8440c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.737+0000 7fb84a3b0640 1 -- 172.21.15.44:0/3352877421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8440c6f90 con 0x7fb8300051c0 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.737+0000 7fb84a3b0640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8300051c0 0x7fb84415a310 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fb8400118e0 tx=0x7fb840011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.737+0000 7fb83affd640 1 -- 172.21.15.44:0/3352877421 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84000faf0 con 0x7fb8300051c0 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.737+0000 7fb83affd640 1 -- 172.21.15.44:0/3352877421 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb84000fc90 con 0x7fb8300051c0 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.737+0000 7fb83affd640 1 -- 172.21.15.44:0/3352877421 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb840014440 con 0x7fb8300051c0 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.737+0000 7fb84c614640 1 -- 172.21.15.44:0/3352877421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8440c7270 con 0x7fb8300051c0 2024-04-02T13:29:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.737+0000 7fb84c614640 1 -- 172.21.15.44:0/3352877421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8440c7830 con 0x7fb8300051c0 2024-04-02T13:29:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.738+0000 7fb84c614640 1 -- 172.21.15.44:0/3352877421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8440684f0 con 0x7fb8300051c0 2024-04-02T13:29:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.739+0000 7fb83affd640 1 -- 172.21.15.44:0/3352877421 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb840012040 con 0x7fb8300051c0 2024-04-02T13:29:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.739+0000 7fb83affd640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb82806c920 0x7fb82806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.739+0000 7fb83affd640 1 -- 172.21.15.44:0/3352877421 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7fb84008c2b0 con 0x7fb8300051c0 2024-04-02T13:29:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.739+0000 7fb849baf640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb82806c920 0x7fb82806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.740+0000 7fb849baf640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb82806c920 0x7fb82806ede0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb8440b7a60 tx=0x7fb8340078f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.744+0000 7fb83affd640 1 -- 172.21.15.44:0/3352877421 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb840019030 con 0x7fb8300051c0 2024-04-02T13:29:20.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.928+0000 7fb84c614640 1 -- 172.21.15.44:0/3352877421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb84409b180 con 0x7fb8300051c0 2024-04-02T13:29:20.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.930+0000 7fb83affd640 1 -- 172.21.15.44:0/3352877421 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+16836 (secure 0 0 0) 0x7fb840059f00 con 0x7fb8300051c0 2024-04-02T13:29:20.933 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:20.934 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":106,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:29:12.549182+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:29:09.192430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-02T13:29:09.860748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:29:20.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.932+0000 7fb84c614640 1 -- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb82806c920 msgr2=0x7fb82806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.932+0000 7fb84c614640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb82806c920 0x7fb82806ede0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb8440b7a60 tx=0x7fb8340078f0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.932+0000 7fb84c614640 1 -- 172.21.15.44:0/3352877421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8300051c0 msgr2=0x7fb84415a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:20.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.932+0000 7fb84c614640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8300051c0 0x7fb84415a310 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fb8400118e0 tx=0x7fb840011da0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.932+0000 7fb84c614640 1 -- 172.21.15.44:0/3352877421 shutdown_connections 2024-04-02T13:29:20.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.932+0000 7fb84c614640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb82806c920 0x7fb82806ede0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.932+0000 7fb84c614640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8440c6a30 0x7fb8440c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.932+0000 7fb84c614640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8440c65b0 0x7fb8440b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.932+0000 7fb84c614640 1 --2- 172.21.15.44:0/3352877421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8300051c0 0x7fb84415a310 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:20.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.932+0000 7fb84c614640 1 -- 172.21.15.44:0/3352877421 >> 172.21.15.44:0/3352877421 conn(0x7fb84409aa20 msgr2=0x7fb8440cebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:20.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.932+0000 7fb84c614640 1 -- 172.21.15.44:0/3352877421 shutdown_connections 2024-04-02T13:29:20.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:20.932+0000 7fb84c614640 1 -- 172.21.15.44:0/3352877421 wait complete. 2024-04-02T13:29:20.946 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:29:09.192430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:29:09.860748+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:29:20.947 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:29:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.047+0000 7f0846689640 1 Processor -- start 2024-04-02T13:29:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.047+0000 7f0846689640 1 -- start start 2024-04-02T13:29:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.048+0000 7f0846689640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f08400cec60 0x7f08400d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.048+0000 7f0846689640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f084015e5b0 0x7f08400d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.048+0000 7f0846689640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08400cf1f0 con 0x7f084015e1d0 2024-04-02T13:29:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.048+0000 7f0846689640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f08400cfe00 con 0x7f08400cec60 2024-04-02T13:29:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.048+0000 7f0846689640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f08400cf4d0 con 0x7f084015e5b0 2024-04-02T13:29:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.048+0000 7f083f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f08400cec60 0x7f08400d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.048+0000 7f083ffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f084015e1d0 0x7f08400d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40514/0 (socket says 172.21.15.44:40514) 2024-04-02T13:29:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.048+0000 7f083ffff640 1 -- 172.21.15.44:0/697032413 learned_addr learned my addr 172.21.15.44:0/697032413 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.048+0000 7f083f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f08400cec60 0x7f08400d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49280/0 (socket says 172.21.15.44:49280) 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.049+0000 7f0844c26640 1 --2- 172.21.15.44:0/697032413 >> v2:172.21.15.69:3300/0 conn(0x7f084015e5b0 0x7f08400d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.049+0000 7f083f7fe640 1 -- 172.21.15.44:0/697032413 >> v2:172.21.15.69:3300/0 conn(0x7f084015e5b0 msgr2=0x7f08400d2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.049+0000 7f083f7fe640 1 --2- 172.21.15.44:0/697032413 >> v2:172.21.15.69:3300/0 conn(0x7f084015e5b0 0x7f08400d2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.049+0000 7f083f7fe640 1 -- 172.21.15.44:0/697032413 >> v1:172.21.15.44:6789/0 conn(0x7f084015e1d0 legacy=0x7f08400d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.049+0000 7f083effd640 1 -- 172.21.15.44:0/697032413 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2783807374 0 0) 0x7f08400cf1f0 con 0x7f084015e1d0 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.049+0000 7f083f7fe640 1 -- 172.21.15.44:0/697032413 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08400cf7b0 con 0x7f08400cec60 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.049+0000 7f083f7fe640 1 --2- 172.21.15.44:0/697032413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08400cec60 0x7f08400d26c0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f08400d0350 tx=0x7f083402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f1c13febc5f7bd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.050+0000 7f083effd640 1 -- 172.21.15.44:0/697032413 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08340315a0 con 0x7f08400cec60 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.050+0000 7f083effd640 1 -- 172.21.15.44:0/697032413 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0834037070 con 0x7f08400cec60 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.050+0000 7f083effd640 1 -- 172.21.15.44:0/697032413 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834031960 con 0x7f08400cec60 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.050+0000 7f0846689640 1 -- 172.21.15.44:0/697032413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08400cec60 msgr2=0x7f08400d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.050+0000 7f0846689640 1 --2- 172.21.15.44:0/697032413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08400cec60 0x7f08400d26c0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f08400d0350 tx=0x7f083402b840 comp rx=0 tx=0).stop 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.050+0000 7f0846689640 1 -- 172.21.15.44:0/697032413 shutdown_connections 2024-04-02T13:29:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.050+0000 7f0846689640 1 --2- 172.21.15.44:0/697032413 >> v2:172.21.15.69:3300/0 conn(0x7f084015e5b0 0x7f08400d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.050+0000 7f0846689640 1 --2- 172.21.15.44:0/697032413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08400cec60 0x7f08400d26c0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.050+0000 7f0846689640 1 -- 172.21.15.44:0/697032413 >> 172.21.15.44:0/697032413 conn(0x7f084009d3b0 msgr2=0x7f08400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:21.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.050+0000 7f0846689640 1 -- 172.21.15.44:0/697032413 shutdown_connections 2024-04-02T13:29:21.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.050+0000 7f0846689640 1 -- 172.21.15.44:0/697032413 wait complete. 2024-04-02T13:29:21.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.051+0000 7f0846689640 1 Processor -- start 2024-04-02T13:29:21.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.051+0000 7f0846689640 1 -- start start 2024-04-02T13:29:21.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.052+0000 7f0846689640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08400cec60 0x7f0840158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.052+0000 7f0846689640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f084015e1d0 0x7f084014c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.052+0000 7f0846689640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f084015e5b0 0x7f084015f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.052+0000 7f0846689640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f08400b4e60 con 0x7f084015e5b0 2024-04-02T13:29:21.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.052+0000 7f0846689640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f08400b4ce0 con 0x7f08400cec60 2024-04-02T13:29:21.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.052+0000 7f0846689640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f08400b4fe0 con 0x7f084015e1d0 2024-04-02T13:29:21.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.052+0000 7f0844c26640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f084015e5b0 0x7f084015f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.052+0000 7f0844c26640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f084015e5b0 0x7f084015f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49286/0 (socket says 172.21.15.44:49286) 2024-04-02T13:29:21.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.052+0000 7f0844c26640 1 -- 172.21.15.44:0/536100286 learned_addr learned my addr 172.21.15.44:0/536100286 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:21.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.052+0000 7f083f7fe640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f084015e1d0 0x7f084014c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.052+0000 7f083ffff640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08400cec60 0x7f0840158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.053+0000 7f0844c26640 1 -- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f084015e1d0 msgr2=0x7f084014c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.053+0000 7f0844c26640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f084015e1d0 0x7f084014c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.053+0000 7f0844c26640 1 -- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08400cec60 msgr2=0x7f0840158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.053+0000 7f0844c26640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08400cec60 0x7f0840158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.053+0000 7f0844c26640 1 -- 172.21.15.44:0/536100286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0840158e20 con 0x7f084015e5b0 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.053+0000 7f083f7fe640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f084015e1d0 0x7f084014c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.053+0000 7f083ffff640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08400cec60 0x7f0840158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.053+0000 7f0844c26640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f084015e5b0 0x7f084015f370 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f0838007e90 tx=0x7f083800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.053+0000 7f083cff9640 1 -- 172.21.15.44:0/536100286 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838014420 con 0x7f084015e5b0 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.053+0000 7f083cff9640 1 -- 172.21.15.44:0/536100286 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0838019070 con 0x7f084015e5b0 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.053+0000 7f0846689640 1 -- 172.21.15.44:0/536100286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0840159100 con 0x7f084015e5b0 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.053+0000 7f083cff9640 1 -- 172.21.15.44:0/536100286 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838014800 con 0x7f084015e5b0 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.053+0000 7f0846689640 1 -- 172.21.15.44:0/536100286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f084015ee50 con 0x7f084015e5b0 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.054+0000 7f0846689640 1 -- 172.21.15.44:0/536100286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f084015f050 con 0x7f084015e5b0 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.054+0000 7f083cff9640 1 -- 172.21.15.44:0/536100286 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f08380149a0 con 0x7f084015e5b0 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.055+0000 7f083cff9640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f082c06c8f0 0x7f082c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.055+0000 7f083cff9640 1 -- 172.21.15.44:0/536100286 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f083808c500 con 0x7f084015e5b0 2024-04-02T13:29:21.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.055+0000 7f083ffff640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f082c06c8f0 0x7f082c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.056+0000 7f083ffff640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f082c06c8f0 0x7f082c06edb0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f0830000f60 tx=0x7f0830007c30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:21.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.059+0000 7f083cff9640 1 -- 172.21.15.44:0/536100286 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f084015f050 con 0x7f084015e5b0 2024-04-02T13:29:21.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.241+0000 7f0846689640 1 -- 172.21.15.44:0/536100286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f08400d47f0 con 0x7f084015e5b0 2024-04-02T13:29:21.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.242+0000 7f083cff9640 1 -- 172.21.15.44:0/536100286 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7f083805a7b0 con 0x7f084015e5b0 2024-04-02T13:29:21.246 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:29:21.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.244+0000 7f0846689640 1 -- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f082c06c8f0 msgr2=0x7f082c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.244+0000 7f0846689640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f082c06c8f0 0x7f082c06edb0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f0830000f60 tx=0x7f0830007c30 comp rx=0 tx=0).stop 2024-04-02T13:29:21.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.244+0000 7f0846689640 1 -- 172.21.15.44:0/536100286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f084015e5b0 msgr2=0x7f084015f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.244+0000 7f0846689640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f084015e5b0 0x7f084015f370 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f0838007e90 tx=0x7f083800e4b0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.244+0000 7f0846689640 1 -- 172.21.15.44:0/536100286 shutdown_connections 2024-04-02T13:29:21.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.244+0000 7f0846689640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f082c06c8f0 0x7f082c06edb0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.244+0000 7f0846689640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f084015e5b0 0x7f084015f370 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.244+0000 7f0846689640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f084015e1d0 0x7f084014c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.244+0000 7f0846689640 1 --2- 172.21.15.44:0/536100286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08400cec60 0x7f0840158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.244+0000 7f0846689640 1 -- 172.21.15.44:0/536100286 >> 172.21.15.44:0/536100286 conn(0x7f084009d3b0 msgr2=0x7f084006d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:21.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.244+0000 7f0846689640 1 -- 172.21.15.44:0/536100286 shutdown_connections 2024-04-02T13:29:21.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.244+0000 7f0846689640 1 -- 172.21.15.44:0/536100286 wait complete. 2024-04-02T13:29:21.258 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:29:21.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.358+0000 7f20e8d22640 1 Processor -- start 2024-04-02T13:29:21.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.358+0000 7f20e8d22640 1 -- start start 2024-04-02T13:29:21.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.359+0000 7f20e8d22640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f20e41565d0 0x7f20e40d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.359+0000 7f20e8d22640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f20e40d05e0 0x7f20e40ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.359+0000 7f20e8d22640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e4160b10 con 0x7f20e40cf400 2024-04-02T13:29:21.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.359+0000 7f20e8d22640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f20e40cfad0 con 0x7f20e41565d0 2024-04-02T13:29:21.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.359+0000 7f20e8d22640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f20e40cf870 con 0x7f20e40d05e0 2024-04-02T13:29:21.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.360+0000 7f20e259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f20e41565d0 0x7f20e40d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.360+0000 7f20e2d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f20e40cf400 0x7f20e40cc660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40528/0 (socket says 172.21.15.44:40528) 2024-04-02T13:29:21.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.360+0000 7f20e2d9d640 1 -- 172.21.15.44:0/676133183 learned_addr learned my addr 172.21.15.44:0/676133183 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:21.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.360+0000 7f20e259c640 1 -- 172.21.15.44:0/676133183 >> v2:172.21.15.69:3300/0 conn(0x7f20e40d05e0 msgr2=0x7f20e40ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:21.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.360+0000 7f20e1d9b640 1 --2- 172.21.15.44:0/676133183 >> v2:172.21.15.69:3300/0 conn(0x7f20e40d05e0 0x7f20e40ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.360+0000 7f20e259c640 1 --2- 172.21.15.44:0/676133183 >> v2:172.21.15.69:3300/0 conn(0x7f20e40d05e0 0x7f20e40ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.360+0000 7f20e259c640 1 -- 172.21.15.44:0/676133183 >> v1:172.21.15.44:6789/0 conn(0x7f20e40cf400 legacy=0x7f20e40cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.360+0000 7f20e159a640 1 -- 172.21.15.44:0/676133183 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1909544642 0 0) 0x7f20e4160b10 con 0x7f20e40cf400 2024-04-02T13:29:21.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.360+0000 7f20e259c640 1 -- 172.21.15.44:0/676133183 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e415fd50 con 0x7f20e41565d0 2024-04-02T13:29:21.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.361+0000 7f20e259c640 1 --2- 172.21.15.44:0/676133183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20e41565d0 0x7f20e40d00a0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f20e40b2a30 tx=0x7f20dc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11053c3f99583bf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:21.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.361+0000 7f20e159a640 1 -- 172.21.15.44:0/676133183 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20dc0315a0 con 0x7f20e41565d0 2024-04-02T13:29:21.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.361+0000 7f20e159a640 1 -- 172.21.15.44:0/676133183 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20dc037070 con 0x7f20e41565d0 2024-04-02T13:29:21.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.361+0000 7f20e159a640 1 -- 172.21.15.44:0/676133183 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20dc031960 con 0x7f20e41565d0 2024-04-02T13:29:21.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.361+0000 7f20e8d22640 1 -- 172.21.15.44:0/676133183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20e41565d0 msgr2=0x7f20e40d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.361+0000 7f20e8d22640 1 --2- 172.21.15.44:0/676133183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20e41565d0 0x7f20e40d00a0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f20e40b2a30 tx=0x7f20dc02b840 comp rx=0 tx=0).stop 2024-04-02T13:29:21.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.361+0000 7f20e8d22640 1 -- 172.21.15.44:0/676133183 shutdown_connections 2024-04-02T13:29:21.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.361+0000 7f20e8d22640 1 --2- 172.21.15.44:0/676133183 >> v2:172.21.15.69:3300/0 conn(0x7f20e40d05e0 0x7f20e40ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.361+0000 7f20e8d22640 1 --2- 172.21.15.44:0/676133183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20e41565d0 0x7f20e40d00a0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.361+0000 7f20e8d22640 1 -- 172.21.15.44:0/676133183 >> 172.21.15.44:0/676133183 conn(0x7f20e406b920 msgr2=0x7f20e406bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:21.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.361+0000 7f20e8d22640 1 -- 172.21.15.44:0/676133183 shutdown_connections 2024-04-02T13:29:21.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.361+0000 7f20e8d22640 1 -- 172.21.15.44:0/676133183 wait complete. 2024-04-02T13:29:21.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.363+0000 7f20e8d22640 1 Processor -- start 2024-04-02T13:29:21.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.363+0000 7f20e8d22640 1 -- start start 2024-04-02T13:29:21.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.363+0000 7f20e8d22640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e40cf400 0x7f20e4154950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.363+0000 7f20e8d22640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20e40d05e0 0x7f20e409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.363+0000 7f20e8d22640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20e41565d0 0x7f20e409ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.363+0000 7f20e8d22640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f20e41605f0 con 0x7f20e40d05e0 2024-04-02T13:29:21.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.363+0000 7f20e8d22640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f20e4160310 con 0x7f20e41565d0 2024-04-02T13:29:21.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.363+0000 7f20e8d22640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f20e41569b0 con 0x7f20e40cf400 2024-04-02T13:29:21.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e1d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20e40d05e0 0x7f20e409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e1d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20e40d05e0 0x7f20e409b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49302/0 (socket says 172.21.15.44:49302) 2024-04-02T13:29:21.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e1d9b640 1 -- 172.21.15.44:0/2510767128 learned_addr learned my addr 172.21.15.44:0/2510767128 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:21.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e259c640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e40cf400 0x7f20e4154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e1d9b640 1 -- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e40cf400 msgr2=0x7f20e4154950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e2d9d640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20e41565d0 0x7f20e409ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e1d9b640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e40cf400 0x7f20e4154950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e1d9b640 1 -- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20e41565d0 msgr2=0x7f20e409ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e1d9b640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20e41565d0 0x7f20e409ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e1d9b640 1 -- 172.21.15.44:0/2510767128 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e409af50 con 0x7f20e40d05e0 2024-04-02T13:29:21.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e259c640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e40cf400 0x7f20e4154950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:21.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e2d9d640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20e41565d0 0x7f20e409ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:21.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e1d9b640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20e40d05e0 0x7f20e409b4c0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00c7f0 tx=0x7f20cc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:21.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20daffd640 1 -- 172.21.15.44:0/2510767128 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc015070 con 0x7f20e40d05e0 2024-04-02T13:29:21.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20daffd640 1 -- 172.21.15.44:0/2510767128 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20cc004220 con 0x7f20e40d05e0 2024-04-02T13:29:21.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e8d22640 1 -- 172.21.15.44:0/2510767128 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20e406d360 con 0x7f20e40d05e0 2024-04-02T13:29:21.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20daffd640 1 -- 172.21.15.44:0/2510767128 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc007620 con 0x7f20e40d05e0 2024-04-02T13:29:21.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.364+0000 7f20e8d22640 1 -- 172.21.15.44:0/2510767128 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20e406d920 con 0x7f20e40d05e0 2024-04-02T13:29:21.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.365+0000 7f20e8d22640 1 -- 172.21.15.44:0/2510767128 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e40684f0 con 0x7f20e40d05e0 2024-04-02T13:29:21.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.366+0000 7f20daffd640 1 -- 172.21.15.44:0/2510767128 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f20cc0079d0 con 0x7f20e40d05e0 2024-04-02T13:29:21.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.366+0000 7f20daffd640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20c406c8f0 0x7f20c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.366+0000 7f20daffd640 1 -- 172.21.15.44:0/2510767128 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f20cc08c130 con 0x7f20e40d05e0 2024-04-02T13:29:21.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.366+0000 7f20e259c640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20c406c8f0 0x7f20c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.367+0000 7f20e259c640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20c406c8f0 0x7f20c406edb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f20dc036000 tx=0x7f20dc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:21.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.371+0000 7f20daffd640 1 -- 172.21.15.44:0/2510767128 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f20cc05a340 con 0x7f20e40d05e0 2024-04-02T13:29:21.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.552+0000 7f20e8d22640 1 -- 172.21.15.44:0/2510767128 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f20e40ce540 con 0x7f20e40d05e0 2024-04-02T13:29:21.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.552+0000 7f20daffd640 1 -- 172.21.15.44:0/2510767128 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) ==== 104+0+11 (secure 0 0 0) 0x7f20cc059c40 con 0x7f20e40d05e0 2024-04-02T13:29:21.556 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:29:21.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.554+0000 7f20e8d22640 1 -- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20c406c8f0 msgr2=0x7f20c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.554+0000 7f20e8d22640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20c406c8f0 0x7f20c406edb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f20dc036000 tx=0x7f20dc008030 comp rx=0 tx=0).stop 2024-04-02T13:29:21.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.554+0000 7f20e8d22640 1 -- 172.21.15.44:0/2510767128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20e40d05e0 msgr2=0x7f20e409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.554+0000 7f20e8d22640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20e40d05e0 0x7f20e409b4c0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00c7f0 tx=0x7f20cc00ccb0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.554+0000 7f20e8d22640 1 -- 172.21.15.44:0/2510767128 shutdown_connections 2024-04-02T13:29:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.554+0000 7f20e8d22640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20c406c8f0 0x7f20c406edb0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.554+0000 7f20e8d22640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20e41565d0 0x7f20e409ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.554+0000 7f20e8d22640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20e40d05e0 0x7f20e409b4c0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.554+0000 7f20e8d22640 1 --2- 172.21.15.44:0/2510767128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e40cf400 0x7f20e4154950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.554+0000 7f20e8d22640 1 -- 172.21.15.44:0/2510767128 >> 172.21.15.44:0/2510767128 conn(0x7f20e406b920 msgr2=0x7f20e406bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.554+0000 7f20e8d22640 1 -- 172.21.15.44:0/2510767128 shutdown_connections 2024-04-02T13:29:21.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.554+0000 7f20e8d22640 1 -- 172.21.15.44:0/2510767128 wait complete. 2024-04-02T13:29:21.569 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:29:21.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.670+0000 7f15e5711640 1 Processor -- start 2024-04-02T13:29:21.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.670+0000 7f15e5711640 1 -- start start 2024-04-02T13:29:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15e5711640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f15e00cd770 0x7f15e006eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15e5711640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15e00ceec0 0x7f15e00cf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15e5711640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15e006f3f0 con 0x7f15e006b4d0 2024-04-02T13:29:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15e5711640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f15e006f6d0 con 0x7f15e00cd770 2024-04-02T13:29:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15e5711640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f15e006f9b0 con 0x7f15e00ceec0 2024-04-02T13:29:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15de7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f15e00cd770 0x7f15e006eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15de7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f15e00cd770 0x7f15e006eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49310/0 (socket says 172.21.15.44:49310) 2024-04-02T13:29:21.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15deffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f15e006b4d0 0x7f15e006b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40544/0 (socket says 172.21.15.44:40544) 2024-04-02T13:29:21.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15de7fc640 1 -- 172.21.15.44:0/3809968080 learned_addr learned my addr 172.21.15.44:0/3809968080 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:21.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15df7fe640 1 --2- 172.21.15.44:0/3809968080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15e00ceec0 0x7f15e00cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15de7fc640 1 -- 172.21.15.44:0/3809968080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15e00ceec0 msgr2=0x7f15e00cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:21.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15de7fc640 1 --2- 172.21.15.44:0/3809968080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15e00ceec0 0x7f15e00cf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15de7fc640 1 -- 172.21.15.44:0/3809968080 >> v1:172.21.15.44:6789/0 conn(0x7f15e006b4d0 legacy=0x7f15e006b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15de7fc640 1 -- 172.21.15.44:0/3809968080 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e006fc90 con 0x7f15e00cd770 2024-04-02T13:29:21.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.671+0000 7f15ddffb640 1 -- 172.21.15.44:0/3809968080 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3959201229 0 0) 0x7f15e006f3f0 con 0x7f15e006b4d0 2024-04-02T13:29:21.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.672+0000 7f15de7fc640 1 --2- 172.21.15.44:0/3809968080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15e00cd770 0x7f15e006eba0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f15c8009870 tx=0x7f15c802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcde463416309bb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:21.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.672+0000 7f15ddffb640 1 -- 172.21.15.44:0/3809968080 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c8034070 con 0x7f15e00cd770 2024-04-02T13:29:21.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.672+0000 7f15ddffb640 1 -- 172.21.15.44:0/3809968080 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15c8030840 con 0x7f15e00cd770 2024-04-02T13:29:21.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.672+0000 7f15ddffb640 1 -- 172.21.15.44:0/3809968080 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c8030c10 con 0x7f15e00cd770 2024-04-02T13:29:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.673+0000 7f15e5711640 1 -- 172.21.15.44:0/3809968080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15e00cd770 msgr2=0x7f15e006eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.673+0000 7f15e5711640 1 --2- 172.21.15.44:0/3809968080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15e00cd770 0x7f15e006eba0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f15c8009870 tx=0x7f15c802b980 comp rx=0 tx=0).stop 2024-04-02T13:29:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.673+0000 7f15e5711640 1 -- 172.21.15.44:0/3809968080 shutdown_connections 2024-04-02T13:29:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.673+0000 7f15e5711640 1 --2- 172.21.15.44:0/3809968080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15e00ceec0 0x7f15e00cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.673+0000 7f15e5711640 1 --2- 172.21.15.44:0/3809968080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15e00cd770 0x7f15e006eba0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.673+0000 7f15e5711640 1 -- 172.21.15.44:0/3809968080 >> 172.21.15.44:0/3809968080 conn(0x7f15e00b2a10 msgr2=0x7f15e00b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.673+0000 7f15e5711640 1 -- 172.21.15.44:0/3809968080 shutdown_connections 2024-04-02T13:29:21.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.673+0000 7f15e5711640 1 -- 172.21.15.44:0/3809968080 wait complete. 2024-04-02T13:29:21.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.674+0000 7f15e5711640 1 Processor -- start 2024-04-02T13:29:21.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.674+0000 7f15e5711640 1 -- start start 2024-04-02T13:29:21.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15e5711640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15e006b4d0 0x7f15e01487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15e5711640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15e00cd770 0x7f15e0159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15e5711640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15e00ceec0 0x7f15e015e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15e5711640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f15e015c470 con 0x7f15e006b4d0 2024-04-02T13:29:21.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15e5711640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f15e006f0e0 con 0x7f15e00cd770 2024-04-02T13:29:21.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15e5711640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f15e015c5f0 con 0x7f15e00ceec0 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15deffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15e006b4d0 0x7f15e01487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15deffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15e006b4d0 0x7f15e01487c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49324/0 (socket says 172.21.15.44:49324) 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15deffd640 1 -- 172.21.15.44:0/2903300987 learned_addr learned my addr 172.21.15.44:0/2903300987 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15df7fe640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15e00ceec0 0x7f15e015e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15de7fc640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15e00cd770 0x7f15e0159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15deffd640 1 -- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15e00ceec0 msgr2=0x7f15e015e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15deffd640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15e00ceec0 0x7f15e015e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15deffd640 1 -- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15e00cd770 msgr2=0x7f15e0159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15deffd640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15e00cd770 0x7f15e0159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15deffd640 1 -- 172.21.15.44:0/2903300987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e015d150 con 0x7f15e006b4d0 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.675+0000 7f15deffd640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15e006b4d0 0x7f15e01487c0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f15d800c4d0 tx=0x7f15d800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.676+0000 7f15d77fe640 1 -- 172.21.15.44:0/2903300987 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d8018070 con 0x7f15e006b4d0 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.676+0000 7f15d77fe640 1 -- 172.21.15.44:0/2903300987 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15d8007500 con 0x7f15e006b4d0 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.676+0000 7f15d77fe640 1 -- 172.21.15.44:0/2903300987 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d8004a20 con 0x7f15e006b4d0 2024-04-02T13:29:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.676+0000 7f15e5711640 1 -- 172.21.15.44:0/2903300987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e015d3d0 con 0x7f15e006b4d0 2024-04-02T13:29:21.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.676+0000 7f15e5711640 1 -- 172.21.15.44:0/2903300987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e015d990 con 0x7f15e006b4d0 2024-04-02T13:29:21.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.677+0000 7f15e5711640 1 -- 172.21.15.44:0/2903300987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e00684f0 con 0x7f15e006b4d0 2024-04-02T13:29:21.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.677+0000 7f15d77fe640 1 -- 172.21.15.44:0/2903300987 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f15d801b070 con 0x7f15e006b4d0 2024-04-02T13:29:21.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.677+0000 7f15d77fe640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f15c006c8f0 0x7f15c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.678+0000 7f15d77fe640 1 -- 172.21.15.44:0/2903300987 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f15d808af80 con 0x7f15e006b4d0 2024-04-02T13:29:21.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.678+0000 7f15de7fc640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f15c006c8f0 0x7f15c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.678+0000 7f15de7fc640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f15c006c8f0 0x7f15c006edb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f15c8036000 tx=0x7f15c803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:21.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.682+0000 7f15d77fe640 1 -- 172.21.15.44:0/2903300987 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f15e015d150 con 0x7f15e006b4d0 2024-04-02T13:29:21.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.866+0000 7f15e5711640 1 -- 172.21.15.44:0/2903300987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f15e006c2b0 con 0x7f15e006b4d0 2024-04-02T13:29:21.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.866+0000 7f15d77fe640 1 -- 172.21.15.44:0/2903300987 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) ==== 100+0+11 (secure 0 0 0) 0x7f15d80592b0 con 0x7f15e006b4d0 2024-04-02T13:29:21.871 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:29:21.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.868+0000 7f15e5711640 1 -- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f15c006c8f0 msgr2=0x7f15c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.868+0000 7f15e5711640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f15c006c8f0 0x7f15c006edb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f15c8036000 tx=0x7f15c803b040 comp rx=0 tx=0).stop 2024-04-02T13:29:21.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.868+0000 7f15e5711640 1 -- 172.21.15.44:0/2903300987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15e006b4d0 msgr2=0x7f15e01487c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.868+0000 7f15e5711640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15e006b4d0 0x7f15e01487c0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f15d800c4d0 tx=0x7f15d800c990 comp rx=0 tx=0).stop 2024-04-02T13:29:21.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.868+0000 7f15e5711640 1 -- 172.21.15.44:0/2903300987 shutdown_connections 2024-04-02T13:29:21.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.868+0000 7f15e5711640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f15c006c8f0 0x7f15c006edb0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.868+0000 7f15e5711640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15e00ceec0 0x7f15e015e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.868+0000 7f15e5711640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15e00cd770 0x7f15e0159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.868+0000 7f15e5711640 1 --2- 172.21.15.44:0/2903300987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15e006b4d0 0x7f15e01487c0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.868+0000 7f15e5711640 1 -- 172.21.15.44:0/2903300987 >> 172.21.15.44:0/2903300987 conn(0x7f15e00b2a10 msgr2=0x7f15e00b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:21.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.868+0000 7f15e5711640 1 -- 172.21.15.44:0/2903300987 shutdown_connections 2024-04-02T13:29:21.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.868+0000 7f15e5711640 1 -- 172.21.15.44:0/2903300987 wait complete. 2024-04-02T13:29:21.883 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:29:21.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.981+0000 7f36a940d640 1 Processor -- start 2024-04-02T13:29:21.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.981+0000 7f36a940d640 1 -- start start 2024-04-02T13:29:21.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a940d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f36a40d28f0 0x7f36a40d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a940d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36a40b5200 0x7f36a40b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a940d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36a40d4d70 con 0x7f36a40d1260 2024-04-02T13:29:21.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a940d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f36a40d5050 con 0x7f36a40d28f0 2024-04-02T13:29:21.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a940d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f36a40d23a0 con 0x7f36a40b5200 2024-04-02T13:29:21.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a2ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f36a40d28f0 0x7f36a40d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a2ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f36a40d28f0 0x7f36a40d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49336/0 (socket says 172.21.15.44:49336) 2024-04-02T13:29:21.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a2ffd640 1 -- 172.21.15.44:0/4228457599 learned_addr learned my addr 172.21.15.44:0/4228457599 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:21.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a27fc640 1 --2- 172.21.15.44:0/4228457599 >> v2:172.21.15.69:3300/0 conn(0x7f36a40b5200 0x7f36a40b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a2ffd640 1 -- 172.21.15.44:0/4228457599 >> v2:172.21.15.69:3300/0 conn(0x7f36a40b5200 msgr2=0x7f36a40b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:21.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a2ffd640 1 --2- 172.21.15.44:0/4228457599 >> v2:172.21.15.69:3300/0 conn(0x7f36a40b5200 0x7f36a40b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a2ffd640 1 -- 172.21.15.44:0/4228457599 >> v1:172.21.15.69:6789/0 conn(0x7f36a40d1260 legacy=0x7f36a40ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a2ffd640 1 -- 172.21.15.44:0/4228457599 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a40d2030 con 0x7f36a40d28f0 2024-04-02T13:29:21.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.982+0000 7f36a27fc640 1 --2- 172.21.15.44:0/4228457599 >> v2:172.21.15.69:3300/0 conn(0x7f36a40b5200 0x7f36a40b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:21.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.983+0000 7f36a2ffd640 1 --2- 172.21.15.44:0/4228457599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36a40d28f0 0x7f36a40d2cd0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f36a40d1c00 tx=0x7f369402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4844d6239d7b617d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:21.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.983+0000 7f36a1ffb640 1 -- 172.21.15.44:0/4228457599 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369402c9f0 con 0x7f36a40d28f0 2024-04-02T13:29:21.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.983+0000 7f36a1ffb640 1 -- 172.21.15.44:0/4228457599 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f369402cb90 con 0x7f36a40d28f0 2024-04-02T13:29:21.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.983+0000 7f36a1ffb640 1 -- 172.21.15.44:0/4228457599 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36940343f0 con 0x7f36a40d28f0 2024-04-02T13:29:21.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.984+0000 7f36a940d640 1 -- 172.21.15.44:0/4228457599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36a40d28f0 msgr2=0x7f36a40d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.984+0000 7f36a940d640 1 --2- 172.21.15.44:0/4228457599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36a40d28f0 0x7f36a40d2cd0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f36a40d1c00 tx=0x7f369402be50 comp rx=0 tx=0).stop 2024-04-02T13:29:21.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.984+0000 7f36a940d640 1 -- 172.21.15.44:0/4228457599 shutdown_connections 2024-04-02T13:29:21.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.984+0000 7f36a940d640 1 --2- 172.21.15.44:0/4228457599 >> v2:172.21.15.69:3300/0 conn(0x7f36a40b5200 0x7f36a40b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.984+0000 7f36a940d640 1 --2- 172.21.15.44:0/4228457599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36a40d28f0 0x7f36a40d2cd0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.984+0000 7f36a940d640 1 -- 172.21.15.44:0/4228457599 >> 172.21.15.44:0/4228457599 conn(0x7f36a40d4120 msgr2=0x7f36a40d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:21.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.984+0000 7f36a940d640 1 -- 172.21.15.44:0/4228457599 shutdown_connections 2024-04-02T13:29:21.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.984+0000 7f36a940d640 1 -- 172.21.15.44:0/4228457599 wait complete. 2024-04-02T13:29:21.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.985+0000 7f36a940d640 1 Processor -- start 2024-04-02T13:29:21.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.985+0000 7f36a940d640 1 -- start start 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a940d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a40b5200 0x7f36a415d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a940d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36a40d1260 0x7f36a415fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a940d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36a40d28f0 0x7f36a4160180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a940d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f36a40b8130 con 0x7f36a40d28f0 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a940d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f36a40b5020 con 0x7f36a40b5200 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a940d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f36a40b82b0 con 0x7f36a40d1260 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36a40d28f0 0x7f36a4160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36a40d28f0 0x7f36a4160180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49346/0 (socket says 172.21.15.44:49346) 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a37fe640 1 -- 172.21.15.44:0/3311882519 learned_addr learned my addr 172.21.15.44:0/3311882519 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a2ffd640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a40b5200 0x7f36a415d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a27fc640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36a40d1260 0x7f36a415fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a37fe640 1 -- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36a40d1260 msgr2=0x7f36a415fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a37fe640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36a40d1260 0x7f36a415fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a37fe640 1 -- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a40b5200 msgr2=0x7f36a415d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:21.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a37fe640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a40b5200 0x7f36a415d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:21.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a37fe640 1 -- 172.21.15.44:0/3311882519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a40ccf10 con 0x7f36a40d28f0 2024-04-02T13:29:21.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a2ffd640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a40b5200 0x7f36a415d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:21.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a27fc640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36a40d1260 0x7f36a415fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:21.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.986+0000 7f36a37fe640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36a40d28f0 0x7f36a4160180 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f369c00c3c0 tx=0x7f369c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:21.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.987+0000 7f369b7fe640 1 -- 172.21.15.44:0/3311882519 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c017070 con 0x7f36a40d28f0 2024-04-02T13:29:21.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.987+0000 7f369b7fe640 1 -- 172.21.15.44:0/3311882519 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f369c0052d0 con 0x7f36a40d28f0 2024-04-02T13:29:21.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.987+0000 7f36a940d640 1 -- 172.21.15.44:0/3311882519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36a40cd1a0 con 0x7f36a40d28f0 2024-04-02T13:29:21.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.987+0000 7f369b7fe640 1 -- 172.21.15.44:0/3311882519 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c003810 con 0x7f36a40d28f0 2024-04-02T13:29:21.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.987+0000 7f36a940d640 1 -- 172.21.15.44:0/3311882519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36a40cd760 con 0x7f36a40d28f0 2024-04-02T13:29:21.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.987+0000 7f36a940d640 1 -- 172.21.15.44:0/3311882519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36a40684f0 con 0x7f36a40d28f0 2024-04-02T13:29:21.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.988+0000 7f369b7fe640 1 -- 172.21.15.44:0/3311882519 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f369c01a070 con 0x7f36a40d28f0 2024-04-02T13:29:21.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.988+0000 7f369b7fe640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f368406c8f0 0x7f368406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:21.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.988+0000 7f369b7fe640 1 -- 172.21.15.44:0/3311882519 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f369c08afe0 con 0x7f36a40d28f0 2024-04-02T13:29:21.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.989+0000 7f36a2ffd640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f368406c8f0 0x7f368406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:21.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.989+0000 7f36a2ffd640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f368406c8f0 0x7f368406edb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3694035000 tx=0x7f3694030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:21.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:21.993+0000 7f369b7fe640 1 -- 172.21.15.44:0/3311882519 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f369c059280 con 0x7f36a40d28f0 2024-04-02T13:29:22.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.187+0000 7f36a940d640 1 -- 172.21.15.44:0/3311882519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f36a40b9e20 con 0x7f36a40d28f0 2024-04-02T13:29:22.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.188+0000 7f369b7fe640 1 -- 172.21.15.44:0/3311882519 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v106) ==== 80+0+0 (secure 0 0 0) 0x7f369c058b80 con 0x7f36a40d28f0 2024-04-02T13:29:22.191 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:29:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.190+0000 7f36a940d640 1 -- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f368406c8f0 msgr2=0x7f368406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.190+0000 7f36a940d640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f368406c8f0 0x7f368406edb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3694035000 tx=0x7f3694030000 comp rx=0 tx=0).stop 2024-04-02T13:29:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.190+0000 7f36a940d640 1 -- 172.21.15.44:0/3311882519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36a40d28f0 msgr2=0x7f36a4160180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.190+0000 7f36a940d640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36a40d28f0 0x7f36a4160180 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f369c00c3c0 tx=0x7f369c00c880 comp rx=0 tx=0).stop 2024-04-02T13:29:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.190+0000 7f36a940d640 1 -- 172.21.15.44:0/3311882519 shutdown_connections 2024-04-02T13:29:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.190+0000 7f36a940d640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f368406c8f0 0x7f368406edb0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.190+0000 7f36a940d640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f36a40d28f0 0x7f36a4160180 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.190+0000 7f36a940d640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36a40d1260 0x7f36a415fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.190+0000 7f36a940d640 1 --2- 172.21.15.44:0/3311882519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a40b5200 0x7f36a415d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.190+0000 7f36a940d640 1 -- 172.21.15.44:0/3311882519 >> 172.21.15.44:0/3311882519 conn(0x7f36a40d4120 msgr2=0x7f36a415e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.190+0000 7f36a940d640 1 -- 172.21.15.44:0/3311882519 shutdown_connections 2024-04-02T13:29:22.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:22.190+0000 7f36a940d640 1 -- 172.21.15.44:0/3311882519 wait complete. 2024-04-02T13:29:22.204 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:29:22.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:22.204 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:22.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[76360]: fuse finished with error 0 and tester_r 0 2024-04-02T13:29:22.231 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:29:23.421 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:29:23.424 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:29:28.333 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:29:28.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:28.334 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:29:28.352 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:29:28.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.502+0000 7fd8b8e4a640 1 Processor -- start 2024-04-02T13:29:28.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.502+0000 7fd8b8e4a640 1 -- start start 2024-04-02T13:29:28.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.503+0000 7fd8b8e4a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd8b406d550 0x7fd8b40d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.503+0000 7fd8b8e4a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8b40cc600 con 0x7fd8b415e4a0 2024-04-02T13:29:28.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.503+0000 7fd8b8e4a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8b40cc8e0 con 0x7fd8b40d0c50 2024-04-02T13:29:28.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.503+0000 7fd8b8e4a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd8b40af300 con 0x7fd8b406d550 2024-04-02T13:29:28.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.503+0000 7fd8b259c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd8b415e4a0 0x7fd8b40aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32832/0 (socket says 172.21.15.44:32832) 2024-04-02T13:29:28.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.503+0000 7fd8b259c640 1 -- 172.21.15.44:0/332508504 learned_addr learned my addr 172.21.15.44:0/332508504 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:28.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.503+0000 7fd8b2d9d640 1 --2- 172.21.15.44:0/332508504 >> v2:172.21.15.69:3300/0 conn(0x7fd8b406d550 0x7fd8b40d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:28.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.503+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 14760830 0 0) 0x7fd8b40cc600 con 0x7fd8b415e4a0 2024-04-02T13:29:28.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.503+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8a0003520 con 0x7fd8b415e4a0 2024-04-02T13:29:28.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1362644144 0 0) 0x7fd8b40cc8e0 con 0x7fd8b40d0c50 2024-04-02T13:29:28.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8b40cc600 con 0x7fd8b40d0c50 2024-04-02T13:29:28.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 643688721 0 0) 0x7fd8a0003520 con 0x7fd8b415e4a0 2024-04-02T13:29:28.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 >> v2:172.21.15.69:3300/0 conn(0x7fd8b406d550 msgr2=0x7fd8b40d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:28.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 --2- 172.21.15.44:0/332508504 >> v2:172.21.15.69:3300/0 conn(0x7fd8b406d550 0x7fd8b40d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 >> v1:172.21.15.69:6789/0 conn(0x7fd8b40d0c50 legacy=0x7fd8b406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:28.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b40ccbc0 con 0x7fd8b415e4a0 2024-04-02T13:29:28.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fd8ac001670 con 0x7fd8b415e4a0 2024-04-02T13:29:28.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b2d9d640 1 --2- 172.21.15.44:0/332508504 >> v2:172.21.15.69:3300/0 conn(0x7fd8b406d550 0x7fd8b40d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:29:28.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 >> v1:172.21.15.44:6789/0 conn(0x7fd8b415e4a0 legacy=0x7fd8b40aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:28.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 --2- 172.21.15.44:0/332508504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a0003b70 0x7fd8a0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 --2- 172.21.15.44:0/332508504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8a0004800 0x7fd8a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 --2- 172.21.15.44:0/332508504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8a00051c0 0x7fd8a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b40af300 con 0x7fd8a0004800 2024-04-02T13:29:28.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a000c060 con 0x7fd8a00051c0 2024-04-02T13:29:28.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a000c260 con 0x7fd8a0003b70 2024-04-02T13:29:28.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b1d9b640 1 --2- 172.21.15.44:0/332508504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8a0004800 0x7fd8a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:28.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b2d9d640 1 --2- 172.21.15.44:0/332508504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8a00051c0 0x7fd8a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:28.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b259c640 1 --2- 172.21.15.44:0/332508504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a0003b70 0x7fd8a0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:28.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b1d9b640 1 -- 172.21.15.44:0/332508504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a0003b70 msgr2=0x7fd8a0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:28.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b1d9b640 1 --2- 172.21.15.44:0/332508504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a0003b70 0x7fd8a0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b1d9b640 1 -- 172.21.15.44:0/332508504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8a00051c0 msgr2=0x7fd8a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:28.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b1d9b640 1 --2- 172.21.15.44:0/332508504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8a00051c0 0x7fd8a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b1d9b640 1 -- 172.21.15.44:0/332508504 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a0003520 con 0x7fd8a0004800 2024-04-02T13:29:28.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b2d9d640 1 --2- 172.21.15.44:0/332508504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8a00051c0 0x7fd8a000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:28.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.504+0000 7fd8b2d9d640 1 -- 172.21.15.44:0/332508504 reap_dead start 2024-04-02T13:29:28.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.505+0000 7fd8b1d9b640 1 --2- 172.21.15.44:0/332508504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8a0004800 0x7fd8a0004c80 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fd89c0018a0 tx=0x7fd8a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea0947e86684ac8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:28.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.505+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89c0014f0 con 0x7fd8a0004800 2024-04-02T13:29:28.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.505+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd89c0050d0 con 0x7fd8a0004800 2024-04-02T13:29:28.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.505+0000 7fd8b159a640 1 -- 172.21.15.44:0/332508504 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89c0035f0 con 0x7fd8a0004800 2024-04-02T13:29:28.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.506+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/332508504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8a0004800 msgr2=0x7fd8a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:28.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.506+0000 7fd8b8e4a640 1 --2- 172.21.15.44:0/332508504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8a0004800 0x7fd8a0004c80 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fd89c0018a0 tx=0x7fd8a0005830 comp rx=0 tx=0).stop 2024-04-02T13:29:28.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.506+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/332508504 shutdown_connections 2024-04-02T13:29:28.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.506+0000 7fd8b8e4a640 1 --2- 172.21.15.44:0/332508504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8a0004800 0x7fd8a0004c80 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.506+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/332508504 >> 172.21.15.44:0/332508504 conn(0x7fd8b409aa20 msgr2=0x7fd8b409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:28.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.506+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/332508504 shutdown_connections 2024-04-02T13:29:28.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.506+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/332508504 wait complete. 2024-04-02T13:29:28.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.507+0000 7fd8b8e4a640 1 Processor -- start 2024-04-02T13:29:28.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.507+0000 7fd8b8e4a640 1 -- start start 2024-04-02T13:29:28.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b8e4a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a0004800 0x7fd8b4157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b8e4a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8b40b4bd0 0x7fd8b415ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b8e4a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b40b04d0 0x7fd8b40b0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b8e4a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b40cd460 con 0x7fd8b40b04d0 2024-04-02T13:29:28.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b8e4a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b40cd180 con 0x7fd8b40b4bd0 2024-04-02T13:29:28.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b8e4a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd8b415e880 con 0x7fd8a0004800 2024-04-02T13:29:28.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b40b04d0 0x7fd8b40b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:28.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b40b04d0 0x7fd8b40b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36092/0 (socket says 172.21.15.44:36092) 2024-04-02T13:29:28.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b2d9d640 1 -- 172.21.15.44:0/3836045429 learned_addr learned my addr 172.21.15.44:0/3836045429 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:28.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b259c640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a0004800 0x7fd8b4157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:28.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b1d9b640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8b40b4bd0 0x7fd8b415ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:28.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b2d9d640 1 -- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a0004800 msgr2=0x7fd8b4157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:28.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b2d9d640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a0004800 0x7fd8b4157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b2d9d640 1 -- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8b40b4bd0 msgr2=0x7fd8b415ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:28.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b2d9d640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8b40b4bd0 0x7fd8b415ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.508+0000 7fd8b2d9d640 1 -- 172.21.15.44:0/3836045429 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b40b50d0 con 0x7fd8b40b04d0 2024-04-02T13:29:28.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.509+0000 7fd8b2d9d640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b40b04d0 0x7fd8b40b0980 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fd8b406da50 tx=0x7fd8a40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:28.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.509+0000 7fd8aaffd640 1 -- 172.21.15.44:0/3836045429 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4003990 con 0x7fd8b40b04d0 2024-04-02T13:29:28.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.509+0000 7fd8aaffd640 1 -- 172.21.15.44:0/3836045429 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8a401b070 con 0x7fd8b40b04d0 2024-04-02T13:29:28.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.509+0000 7fd8aaffd640 1 -- 172.21.15.44:0/3836045429 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4003b90 con 0x7fd8b40b04d0 2024-04-02T13:29:28.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.509+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/3836045429 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8b40b53b0 con 0x7fd8b40b04d0 2024-04-02T13:29:28.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.509+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/3836045429 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8b40b5970 con 0x7fd8b40b04d0 2024-04-02T13:29:28.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.510+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/3836045429 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8b40684f0 con 0x7fd8b40b04d0 2024-04-02T13:29:28.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.510+0000 7fd8aaffd640 1 -- 172.21.15.44:0/3836045429 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd8a4005d10 con 0x7fd8b40b04d0 2024-04-02T13:29:28.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.511+0000 7fd8aaffd640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd89406c920 0x7fd89406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.511+0000 7fd8aaffd640 1 -- 172.21.15.44:0/3836045429 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7fd8a408c1a0 con 0x7fd8b40b04d0 2024-04-02T13:29:28.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.511+0000 7fd8b259c640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd89406c920 0x7fd89406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:28.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.511+0000 7fd8b259c640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd89406c920 0x7fd89406ede0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd8b415d6b0 tx=0x7fd8ac007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:28.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.515+0000 7fd8aaffd640 1 -- 172.21.15.44:0/3836045429 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd8a4025020 con 0x7fd8b40b04d0 2024-04-02T13:29:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.739+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/3836045429 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8b40ce890 con 0x7fd8b40b04d0 2024-04-02T13:29:28.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.739+0000 7fd8aaffd640 1 -- 172.21.15.44:0/3836045429 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5183 (secure 0 0 0) 0x7fd8a4059e80 con 0x7fd8b40b04d0 2024-04-02T13:29:28.743 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:28.743 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7696,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2715855087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2715855087},{"type":"v1","addr":"172.21.15.69:6836","nonce":2715855087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7699,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3850680553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3850680553},{"type":"v1","addr":"172.21.15.69:6837","nonce":3850680553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/1330712383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1330712383},{"type":"v1","addr":"172.21.15.44:6835","nonce":1330712383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:29:10.907920+0000","modified":"2024-04-02T13:29:13.930909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8760":{"gid":8760,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6834/3259420283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":3259420283},{"type":"v1","addr":"172.21.15.44:6834","nonce":3259420283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8760,"qdb_cluster":[8760]},"id":7}]} 2024-04-02T13:29:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 82 2024-04-02T13:29:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.741+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd89406c920 msgr2=0x7fd89406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.741+0000 7fd8b8e4a640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd89406c920 0x7fd89406ede0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd8b415d6b0 tx=0x7fd8ac007040 comp rx=0 tx=0).stop 2024-04-02T13:29:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.741+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/3836045429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b40b04d0 msgr2=0x7fd8b40b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.741+0000 7fd8b8e4a640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b40b04d0 0x7fd8b40b0980 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fd8b406da50 tx=0x7fd8a40115a0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.741+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/3836045429 shutdown_connections 2024-04-02T13:29:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.741+0000 7fd8b8e4a640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd89406c920 0x7fd89406ede0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.741+0000 7fd8b8e4a640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b40b04d0 0x7fd8b40b0980 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.741+0000 7fd8b8e4a640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8b40b4bd0 0x7fd8b415ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.741+0000 7fd8b8e4a640 1 --2- 172.21.15.44:0/3836045429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8a0004800 0x7fd8b4157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.741+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/3836045429 >> 172.21.15.44:0/3836045429 conn(0x7fd8b409aa20 msgr2=0x7fd8b40b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:28.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.741+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/3836045429 shutdown_connections 2024-04-02T13:29:28.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.741+0000 7fd8b8e4a640 1 -- 172.21.15.44:0/3836045429 wait complete. 2024-04-02T13:29:28.756 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-04-02T13:29:28.882 INFO:teuthology.orchestra.run.smithi044.stdout:{"message":"","return_code":0} 2024-04-02T13:29:28.883 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-02T13:29:28.883 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:29:28.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.981+0000 7fa423463640 1 Processor -- start 2024-04-02T13:29:28.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.981+0000 7fa423463640 1 -- start start 2024-04-02T13:29:28.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa423463640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa41c0d0a30 0x7fa41c06cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa423463640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa41c0d0e10 0x7fa41c0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa423463640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41c06c1d0 con 0x7fa41c0cfe30 2024-04-02T13:29:28.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa423463640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa41c06cb00 con 0x7fa41c0d0a30 2024-04-02T13:29:28.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa423463640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa41c06c4b0 con 0x7fa41c0d0e10 2024-04-02T13:29:28.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa421a00640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa41c0cfe30 0x7fa41c0d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32834/0 (socket says 172.21.15.44:32834) 2024-04-02T13:29:28.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa4211ff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa41c0d0a30 0x7fa41c06cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:28.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa421a00640 1 -- 172.21.15.44:0/2700840829 learned_addr learned my addr 172.21.15.44:0/2700840829 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:28.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa4209fe640 1 --2- 172.21.15.44:0/2700840829 >> v2:172.21.15.69:3300/0 conn(0x7fa41c0d0e10 0x7fa41c0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:28.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa4211ff640 1 -- 172.21.15.44:0/2700840829 >> v2:172.21.15.69:3300/0 conn(0x7fa41c0d0e10 msgr2=0x7fa41c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:28.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa4211ff640 1 --2- 172.21.15.44:0/2700840829 >> v2:172.21.15.69:3300/0 conn(0x7fa41c0d0e10 0x7fa41c0cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa4211ff640 1 -- 172.21.15.44:0/2700840829 >> v1:172.21.15.44:6789/0 conn(0x7fa41c0cfe30 legacy=0x7fa41c0d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:28.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa413fff640 1 -- 172.21.15.44:0/2700840829 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3675920801 0 0) 0x7fa41c06c1d0 con 0x7fa41c0cfe30 2024-04-02T13:29:28.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa4211ff640 1 -- 172.21.15.44:0/2700840829 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41c06c790 con 0x7fa41c0d0a30 2024-04-02T13:29:28.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.982+0000 7fa4209fe640 1 --2- 172.21.15.44:0/2700840829 >> v2:172.21.15.69:3300/0 conn(0x7fa41c0d0e10 0x7fa41c0cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:28.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.983+0000 7fa4211ff640 1 --2- 172.21.15.44:0/2700840829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41c0d0a30 0x7fa41c06cf90 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0b6d30 tx=0x7fa41802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99effa704cb26dd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:28.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.983+0000 7fa413fff640 1 -- 172.21.15.44:0/2700840829 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4180315a0 con 0x7fa41c0d0a30 2024-04-02T13:29:28.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.983+0000 7fa413fff640 1 -- 172.21.15.44:0/2700840829 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa418037070 con 0x7fa41c0d0a30 2024-04-02T13:29:28.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.983+0000 7fa413fff640 1 -- 172.21.15.44:0/2700840829 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418031960 con 0x7fa41c0d0a30 2024-04-02T13:29:28.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.984+0000 7fa423463640 1 -- 172.21.15.44:0/2700840829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41c0d0a30 msgr2=0x7fa41c06cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:28.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.984+0000 7fa423463640 1 --2- 172.21.15.44:0/2700840829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41c0d0a30 0x7fa41c06cf90 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0b6d30 tx=0x7fa41802b840 comp rx=0 tx=0).stop 2024-04-02T13:29:28.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.984+0000 7fa423463640 1 -- 172.21.15.44:0/2700840829 shutdown_connections 2024-04-02T13:29:28.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.984+0000 7fa423463640 1 --2- 172.21.15.44:0/2700840829 >> v2:172.21.15.69:3300/0 conn(0x7fa41c0d0e10 0x7fa41c0cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.984+0000 7fa423463640 1 --2- 172.21.15.44:0/2700840829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41c0d0a30 0x7fa41c06cf90 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.984+0000 7fa423463640 1 -- 172.21.15.44:0/2700840829 >> 172.21.15.44:0/2700840829 conn(0x7fa41c09d470 msgr2=0x7fa41c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:28.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.984+0000 7fa423463640 1 -- 172.21.15.44:0/2700840829 shutdown_connections 2024-04-02T13:29:28.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.984+0000 7fa423463640 1 -- 172.21.15.44:0/2700840829 wait complete. 2024-04-02T13:29:28.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.985+0000 7fa423463640 1 Processor -- start 2024-04-02T13:29:28.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.985+0000 7fa423463640 1 -- start start 2024-04-02T13:29:28.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa423463640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41c0cfe30 0x7fa41c0cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa423463640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa41c0d0a30 0x7fa41c0b6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa423463640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41c0d0e10 0x7fa41c1f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa423463640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa41c15e870 con 0x7fa41c0cfe30 2024-04-02T13:29:28.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa423463640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa41c15e6f0 con 0x7fa41c0d0a30 2024-04-02T13:29:28.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa423463640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa41c0d4da0 con 0x7fa41c0d0e10 2024-04-02T13:29:28.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa4211ff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41c0cfe30 0x7fa41c0cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:28.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa4211ff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41c0cfe30 0x7fa41c0cdb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36102/0 (socket says 172.21.15.44:36102) 2024-04-02T13:29:28.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa4211ff640 1 -- 172.21.15.44:0/559234180 learned_addr learned my addr 172.21.15.44:0/559234180 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:28.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa4211ff640 1 -- 172.21.15.44:0/559234180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41c0d0e10 msgr2=0x7fa41c1f3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:28.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa4211ff640 1 --2- 172.21.15.44:0/559234180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41c0d0e10 0x7fa41c1f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa4211ff640 1 -- 172.21.15.44:0/559234180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa41c0d0a30 msgr2=0x7fa41c0b6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa4209fe640 1 --2- 172.21.15.44:0/559234180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa41c0d0a30 0x7fa41c0b6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa4211ff640 1 --2- 172.21.15.44:0/559234180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa41c0d0a30 0x7fa41c0b6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa4211ff640 1 -- 172.21.15.44:0/559234180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41c0b74a0 con 0x7fa41c0cfe30 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa4211ff640 1 --2- 172.21.15.44:0/559234180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41c0cfe30 0x7fa41c0cdb60 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fa418000c00 tx=0x7fa4180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.986+0000 7fa411ffb640 1 -- 172.21.15.44:0/559234180 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418032840 con 0x7fa41c0cfe30 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.987+0000 7fa411ffb640 1 -- 172.21.15.44:0/559234180 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa418032a00 con 0x7fa41c0cfe30 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.987+0000 7fa423463640 1 -- 172.21.15.44:0/559234180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa41c0b7780 con 0x7fa41c0cfe30 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.987+0000 7fa411ffb640 1 -- 172.21.15.44:0/559234180 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418032df0 con 0x7fa41c0cfe30 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.987+0000 7fa423463640 1 -- 172.21.15.44:0/559234180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa41c0b7d40 con 0x7fa41c0cfe30 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.988+0000 7fa423463640 1 -- 172.21.15.44:0/559234180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa41c0684f0 con 0x7fa41c0cfe30 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.988+0000 7fa411ffb640 1 -- 172.21.15.44:0/559234180 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa41803b3f0 con 0x7fa41c0cfe30 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.988+0000 7fa411ffb640 1 --2- 172.21.15.44:0/559234180 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa40006c8f0 0x7fa40006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.988+0000 7fa411ffb640 1 -- 172.21.15.44:0/559234180 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7fa41802f030 con 0x7fa41c0cfe30 2024-04-02T13:29:28.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.989+0000 7fa4209fe640 1 --2- 172.21.15.44:0/559234180 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa40006c8f0 0x7fa40006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:28.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.989+0000 7fa4209fe640 1 --2- 172.21.15.44:0/559234180 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa40006c8f0 0x7fa40006edb0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa41c1f7420 tx=0x7fa40c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:28.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:28.993+0000 7fa411ffb640 1 -- 172.21.15.44:0/559234180 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa418040070 con 0x7fa41c0cfe30 2024-04-02T13:29:29.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.217+0000 7fa423463640 1 -- 172.21.15.44:0/559234180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa41c0d17c0 con 0x7fa41c0cfe30 2024-04-02T13:29:29.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.218+0000 7fa411ffb640 1 -- 172.21.15.44:0/559234180 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5183 (secure 0 0 0) 0x7fa41807bfd0 con 0x7fa41c0cfe30 2024-04-02T13:29:29.221 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:29.222 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7696,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2715855087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2715855087},{"type":"v1","addr":"172.21.15.69:6836","nonce":2715855087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7699,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3850680553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3850680553},{"type":"v1","addr":"172.21.15.69:6837","nonce":3850680553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/1330712383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1330712383},{"type":"v1","addr":"172.21.15.44:6835","nonce":1330712383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:29:10.907920+0000","modified":"2024-04-02T13:29:13.930909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8760":{"gid":8760,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6834/3259420283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":3259420283},{"type":"v1","addr":"172.21.15.44:6834","nonce":3259420283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8760,"qdb_cluster":[8760]},"id":7}]} 2024-04-02T13:29:29.222 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 82 2024-04-02T13:29:29.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.219+0000 7fa423463640 1 -- 172.21.15.44:0/559234180 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa40006c8f0 msgr2=0x7fa40006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.219+0000 7fa423463640 1 --2- 172.21.15.44:0/559234180 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa40006c8f0 0x7fa40006edb0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa41c1f7420 tx=0x7fa40c008040 comp rx=0 tx=0).stop 2024-04-02T13:29:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.220+0000 7fa423463640 1 -- 172.21.15.44:0/559234180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41c0cfe30 msgr2=0x7fa41c0cdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.220+0000 7fa423463640 1 --2- 172.21.15.44:0/559234180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41c0cfe30 0x7fa41c0cdb60 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fa418000c00 tx=0x7fa4180046c0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.220+0000 7fa423463640 1 -- 172.21.15.44:0/559234180 shutdown_connections 2024-04-02T13:29:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.220+0000 7fa423463640 1 --2- 172.21.15.44:0/559234180 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa40006c8f0 0x7fa40006edb0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.220+0000 7fa423463640 1 --2- 172.21.15.44:0/559234180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa41c0d0e10 0x7fa41c1f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.220+0000 7fa423463640 1 --2- 172.21.15.44:0/559234180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa41c0d0a30 0x7fa41c0b6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.220+0000 7fa423463640 1 --2- 172.21.15.44:0/559234180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa41c0cfe30 0x7fa41c0cdb60 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.220+0000 7fa423463640 1 -- 172.21.15.44:0/559234180 >> 172.21.15.44:0/559234180 conn(0x7fa41c09d470 msgr2=0x7fa41c0d3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.220+0000 7fa423463640 1 -- 172.21.15.44:0/559234180 shutdown_connections 2024-04-02T13:29:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.220+0000 7fa423463640 1 -- 172.21.15.44:0/559234180 wait complete. 2024-04-02T13:29:29.234 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata get mds0_inotable - 2024-04-02T13:29:29.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.272+0000 7f950e00ff40 1 Processor -- start 2024-04-02T13:29:29.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.272+0000 7f950e00ff40 1 -- start start 2024-04-02T13:29:29.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950e00ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5640daad74e0 0x5640dab00b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:29.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950e00ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640dab8f6c0 0x5640dabac150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:29.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950e00ff40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640daba3bb0 0x5640dabac690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950e00ff40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5640dabaa460 con 0x5640daba3bb0 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950e00ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5640daa7c3e0 con 0x5640dab8f6c0 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950e00ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5640dabaa700 con 0x5640daad74e0 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950b6ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640daba3bb0 0x5640dabac690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950b6ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640daba3bb0 0x5640dabac690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36118/0 (socket says 172.21.15.44:36118) 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950b6ea640 1 -- 172.21.15.44:0/2133556568 learned_addr learned my addr 172.21.15.44:0/2133556568 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950aee9640 1 --2- 172.21.15.44:0/2133556568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5640daad74e0 0x5640dab00b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950a6e8640 1 --2- 172.21.15.44:0/2133556568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640dab8f6c0 0x5640dabac150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950b6ea640 1 -- 172.21.15.44:0/2133556568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5640daad74e0 msgr2=0x5640dab00b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950b6ea640 1 --2- 172.21.15.44:0/2133556568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5640daad74e0 0x5640dab00b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950b6ea640 1 -- 172.21.15.44:0/2133556568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640dab8f6c0 msgr2=0x5640dabac150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950b6ea640 1 --2- 172.21.15.44:0/2133556568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640dab8f6c0 0x5640dabac150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950b6ea640 1 -- 172.21.15.44:0/2133556568 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640daad6770 con 0x5640daba3bb0 2024-04-02T13:29:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.273+0000 7f950a6e8640 1 --2- 172.21.15.44:0/2133556568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640dab8f6c0 0x5640dabac150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.274+0000 7f950b6ea640 1 --2- 172.21.15.44:0/2133556568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640daba3bb0 0x5640dabac690 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f950400a810 tx=0x7f9504005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbaa98b92ba7540 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.274+0000 7f95096e6640 1 -- 172.21.15.44:0/2133556568 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950400d9c0 con 0x5640daba3bb0 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.274+0000 7f95096e6640 1 -- 172.21.15.44:0/2133556568 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f950400db60 con 0x5640daba3bb0 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.274+0000 7f95096e6640 1 -- 172.21.15.44:0/2133556568 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9504012450 con 0x5640daba3bb0 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.274+0000 7f950e00ff40 1 -- 172.21.15.44:0/2133556568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640daba3bb0 msgr2=0x5640dabac690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.274+0000 7f950e00ff40 1 --2- 172.21.15.44:0/2133556568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640daba3bb0 0x5640dabac690 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f950400a810 tx=0x7f9504005d80 comp rx=0 tx=0).stop 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.274+0000 7f950e00ff40 1 -- 172.21.15.44:0/2133556568 shutdown_connections 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.274+0000 7f950e00ff40 1 --2- 172.21.15.44:0/2133556568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640daba3bb0 0x5640dabac690 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.274+0000 7f950e00ff40 1 --2- 172.21.15.44:0/2133556568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640dab8f6c0 0x5640dabac150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.274+0000 7f950e00ff40 1 --2- 172.21.15.44:0/2133556568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5640daad74e0 0x5640dab00b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.274+0000 7f950e00ff40 1 -- 172.21.15.44:0/2133556568 >> 172.21.15.44:0/2133556568 conn(0x5640dab01c80 msgr2=0x5640dabaad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.274+0000 7f950e00ff40 1 -- 172.21.15.44:0/2133556568 shutdown_connections 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.274+0000 7f950e00ff40 1 -- 172.21.15.44:0/2133556568 wait complete. 2024-04-02T13:29:29.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.275+0000 7f950e00ff40 1 Processor -- start 2024-04-02T13:29:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.275+0000 7f950e00ff40 1 -- start start 2024-04-02T13:29:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.275+0000 7f950e00ff40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640daad74e0 0x5640dab89e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.275+0000 7f950e00ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5640dab8f6c0 0x5640daba2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950e00ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640daba3bb0 0x5640daba85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950e00ff40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5640dabab520 con 0x5640daad74e0 2024-04-02T13:29:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950e00ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5640dabacbd0 con 0x5640daba3bb0 2024-04-02T13:29:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950e00ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5640dabab800 con 0x5640dab8f6c0 2024-04-02T13:29:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950aee9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640daad74e0 0x5640dab89e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950aee9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640daad74e0 0x5640dab89e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36134/0 (socket says 172.21.15.44:36134) 2024-04-02T13:29:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950aee9640 1 -- 172.21.15.44:0/547246584 learned_addr learned my addr 172.21.15.44:0/547246584 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950a6e8640 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5640dab8f6c0 0x5640daba2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950aee9640 1 -- 172.21.15.44:0/547246584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5640dab8f6c0 msgr2=0x5640daba2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950aee9640 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5640dab8f6c0 0x5640daba2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950b6ea640 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640daba3bb0 0x5640daba85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950aee9640 1 -- 172.21.15.44:0/547246584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640daba3bb0 msgr2=0x5640daba85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950aee9640 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640daba3bb0 0x5640daba85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950aee9640 1 -- 172.21.15.44:0/547246584 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640dab8a360 con 0x5640daad74e0 2024-04-02T13:29:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950aee9640 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640daad74e0 0x5640dab89e20 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f9500055040 tx=0x7f9500002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f94f3fff640 1 -- 172.21.15.44:0/547246584 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500061030 con 0x5640daad74e0 2024-04-02T13:29:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f94f3fff640 1 -- 172.21.15.44:0/547246584 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9500058040 con 0x5640daad74e0 2024-04-02T13:29:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950e00ff40 1 -- 172.21.15.44:0/547246584 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5640dab87e00 con 0x5640daad74e0 2024-04-02T13:29:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f94f3fff640 1 -- 172.21.15.44:0/547246584 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500057070 con 0x5640daad74e0 2024-04-02T13:29:29.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.276+0000 7f950e00ff40 1 -- 172.21.15.44:0/547246584 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640dab883a0 con 0x5640daad74e0 2024-04-02T13:29:29.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.278+0000 7f94f3fff640 1 -- 172.21.15.44:0/547246584 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9500064030 con 0x5640daad74e0 2024-04-02T13:29:29.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.278+0000 7f94f3fff640 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f94f806b2d0 0x7f94f806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:29.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.278+0000 7f94f3fff640 1 -- 172.21.15.44:0/547246584 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f95000cbae0 con 0x5640daad74e0 2024-04-02T13:29:29.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.278+0000 7f950e00ff40 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5640dabac760 0x5640dab8e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:29.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.278+0000 7f950a6e8640 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f94f806b2d0 0x7f94f806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:29.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.278+0000 7f950e00ff40 1 -- 172.21.15.44:0/547246584 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:1 15.13 15:c91d4a1d:::mds0_inotable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106) -- 0x5640dab8e5c0 con 0x5640dabac760 2024-04-02T13:29:29.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.278+0000 7f950b6ea640 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5640dabac760 0x5640dab8e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:29.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950b6ea640 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5640dabac760 0x5640dab8e080 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:29.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950a6e8640 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f94f806b2d0 0x7f94f806d790 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x5640daba9400 tx=0x7f94fc001b30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:29.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950b6ea640 1 -- 172.21.15.44:0/547246584 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(1 mds0_inotable [read 0~34 out=34b] v0'0 uv3 ondisk = 0) ==== 157+0+34 (crc 0 0 0) 0x7f9504019020 con 0x5640dabac760 2024-04-02T13:29:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950e00ff40 1 -- 172.21.15.44:0/547246584 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5640dabac760 msgr2=0x5640dab8e080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950e00ff40 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5640dabac760 0x5640dab8e080 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950e00ff40 1 -- 172.21.15.44:0/547246584 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f94f806b2d0 msgr2=0x7f94f806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950e00ff40 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f94f806b2d0 0x7f94f806d790 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x5640daba9400 tx=0x7f94fc001b30 comp rx=0 tx=0).stop 2024-04-02T13:29:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950e00ff40 1 -- 172.21.15.44:0/547246584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640daad74e0 msgr2=0x5640dab89e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950e00ff40 1 --2- 172.21.15.44:0/547246584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640daad74e0 0x5640dab89e20 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f9500055040 tx=0x7f9500002fb0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950b6ea640 1 -- 172.21.15.44:0/547246584 reap_dead start 2024-04-02T13:29:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950e00ff40 1 -- 172.21.15.44:0/547246584 shutdown_connections 2024-04-02T13:29:29.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950e00ff40 1 -- 172.21.15.44:0/547246584 >> 172.21.15.44:0/547246584 conn(0x5640dab01c80 msgr2=0x5640dabaad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:29.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950e00ff40 1 -- 172.21.15.44:0/547246584 shutdown_connections 2024-04-02T13:29:29.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.279+0000 7f950e00ff40 1 -- 172.21.15.44:0/547246584 wait complete. 2024-04-02T13:29:29.289 INFO:teuthology.orchestra.run.smithi044.stdout:����� 2024-04-02T13:29:29.289 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:29:29.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.428+0000 7f24f59a3640 1 Processor -- start 2024-04-02T13:29:29.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.428+0000 7f24f59a3640 1 -- start start 2024-04-02T13:29:29.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24f59a3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f24f00d0c50 0x7f24f00d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:29.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24f59a3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24f00cc600 con 0x7f24f015e4a0 2024-04-02T13:29:29.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24f59a3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24f00cc8e0 con 0x7f24f006ce40 2024-04-02T13:29:29.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24f59a3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f24f00af300 con 0x7f24f00d0c50 2024-04-02T13:29:29.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24eeffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f24f015e4a0 0x7f24f00aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32846/0 (socket says 172.21.15.44:32846) 2024-04-02T13:29:29.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24eeffd640 1 -- 172.21.15.44:0/2558910363 learned_addr learned my addr 172.21.15.44:0/2558910363 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:29.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24e7fff640 1 --2- 172.21.15.44:0/2558910363 >> v2:172.21.15.69:3300/0 conn(0x7f24f00d0c50 0x7f24f00d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:29.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24ee7fc640 1 -- 172.21.15.44:0/2558910363 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2133414705 0 0) 0x7f24f00cc600 con 0x7f24f015e4a0 2024-04-02T13:29:29.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24ee7fc640 1 -- 172.21.15.44:0/2558910363 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24dc003520 con 0x7f24f015e4a0 2024-04-02T13:29:29.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24ee7fc640 1 -- 172.21.15.44:0/2558910363 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1246704640 0 0) 0x7f24f00cc8e0 con 0x7f24f006ce40 2024-04-02T13:29:29.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24ee7fc640 1 -- 172.21.15.44:0/2558910363 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24f00cc600 con 0x7f24f006ce40 2024-04-02T13:29:29.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24ee7fc640 1 -- 172.21.15.44:0/2558910363 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1956639447 0 0) 0x7f24dc003520 con 0x7f24f015e4a0 2024-04-02T13:29:29.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24e7fff640 1 -- 172.21.15.44:0/2558910363 >> v1:172.21.15.69:6789/0 conn(0x7f24f006ce40 legacy=0x7f24f00cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24e7fff640 1 -- 172.21.15.44:0/2558910363 >> v1:172.21.15.44:6789/0 conn(0x7f24f015e4a0 legacy=0x7f24f00aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.429+0000 7f24e7fff640 1 -- 172.21.15.44:0/2558910363 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f00ccbc0 con 0x7f24f00d0c50 2024-04-02T13:29:29.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.430+0000 7f24e7fff640 1 --2- 172.21.15.44:0/2558910363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24f00d0c50 0x7f24f00d10b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f24d80097a0 tx=0x7f24d802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9f2cc1a1ea3ad27 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:29.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.430+0000 7f24ee7fc640 1 -- 172.21.15.44:0/2558910363 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d8038070 con 0x7f24f00d0c50 2024-04-02T13:29:29.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.430+0000 7f24ee7fc640 1 -- 172.21.15.44:0/2558910363 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24d8031500 con 0x7f24f00d0c50 2024-04-02T13:29:29.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.430+0000 7f24ee7fc640 1 -- 172.21.15.44:0/2558910363 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0001670 con 0x7f24f00d0c50 2024-04-02T13:29:29.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.431+0000 7f24f59a3640 1 -- 172.21.15.44:0/2558910363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24f00d0c50 msgr2=0x7f24f00d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.431+0000 7f24f59a3640 1 --2- 172.21.15.44:0/2558910363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24f00d0c50 0x7f24f00d10b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f24d80097a0 tx=0x7f24d802b820 comp rx=0 tx=0).stop 2024-04-02T13:29:29.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.431+0000 7f24f59a3640 1 -- 172.21.15.44:0/2558910363 shutdown_connections 2024-04-02T13:29:29.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.431+0000 7f24f59a3640 1 --2- 172.21.15.44:0/2558910363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24f00d0c50 0x7f24f00d10b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.431+0000 7f24f59a3640 1 -- 172.21.15.44:0/2558910363 >> 172.21.15.44:0/2558910363 conn(0x7f24f009aa20 msgr2=0x7f24f009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:29.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.431+0000 7f24f59a3640 1 -- 172.21.15.44:0/2558910363 shutdown_connections 2024-04-02T13:29:29.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.431+0000 7f24f59a3640 1 -- 172.21.15.44:0/2558910363 wait complete. 2024-04-02T13:29:29.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.432+0000 7f24f59a3640 1 Processor -- start 2024-04-02T13:29:29.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.432+0000 7f24f59a3640 1 -- start start 2024-04-02T13:29:29.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24f59a3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f006ce40 0x7f24f015a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:29.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24f59a3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24f00d0c50 0x7f24f015bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:29.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24f59a3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24f015e4a0 0x7f24f015c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:29.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24f59a3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f24f00cd460 con 0x7f24f006ce40 2024-04-02T13:29:29.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24f59a3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f24f00cd180 con 0x7f24f00d0c50 2024-04-02T13:29:29.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24f59a3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f24f015e880 con 0x7f24f015e4a0 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24eeffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f006ce40 0x7f24f015a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24eeffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f006ce40 0x7f24f015a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36144/0 (socket says 172.21.15.44:36144) 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24eeffd640 1 -- 172.21.15.44:0/43897816 learned_addr learned my addr 172.21.15.44:0/43897816 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24e7fff640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24f00d0c50 0x7f24f015bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24ef7fe640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24f015e4a0 0x7f24f015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24eeffd640 1 -- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24f015e4a0 msgr2=0x7f24f015c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24eeffd640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24f015e4a0 0x7f24f015c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24eeffd640 1 -- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24f00d0c50 msgr2=0x7f24f015bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24eeffd640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24f00d0c50 0x7f24f015bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24eeffd640 1 -- 172.21.15.44:0/43897816 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f01609b0 con 0x7f24f006ce40 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24e7fff640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24f00d0c50 0x7f24f015bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24ef7fe640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24f015e4a0 0x7f24f015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24eeffd640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f006ce40 0x7f24f015a660 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f24e000c780 tx=0x7f24e000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24ecff9640 1 -- 172.21.15.44:0/43897816 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0017070 con 0x7f24f006ce40 2024-04-02T13:29:29.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24ecff9640 1 -- 172.21.15.44:0/43897816 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e0005af0 con 0x7f24f006ce40 2024-04-02T13:29:29.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24f59a3640 1 -- 172.21.15.44:0/43897816 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f0160c90 con 0x7f24f006ce40 2024-04-02T13:29:29.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24ecff9640 1 -- 172.21.15.44:0/43897816 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0007070 con 0x7f24f006ce40 2024-04-02T13:29:29.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.433+0000 7f24f59a3640 1 -- 172.21.15.44:0/43897816 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f0161250 con 0x7f24f006ce40 2024-04-02T13:29:29.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.435+0000 7f24f59a3640 1 -- 172.21.15.44:0/43897816 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f00684f0 con 0x7f24f006ce40 2024-04-02T13:29:29.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.435+0000 7f24ecff9640 1 -- 172.21.15.44:0/43897816 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f24e001a070 con 0x7f24f006ce40 2024-04-02T13:29:29.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.435+0000 7f24ecff9640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f24d006c8f0 0x7f24d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:29.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.435+0000 7f24ecff9640 1 -- 172.21.15.44:0/43897816 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f24e008b230 con 0x7f24f006ce40 2024-04-02T13:29:29.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.435+0000 7f24e7fff640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f24d006c8f0 0x7f24d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:29.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.436+0000 7f24e7fff640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f24d006c8f0 0x7f24d006edb0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f24d8035000 tx=0x7f24d8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:29.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.440+0000 7f24ecff9640 1 -- 172.21.15.44:0/43897816 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f24e00594d0 con 0x7f24f006ce40 2024-04-02T13:29:29.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.662+0000 7f24f59a3640 1 -- 172.21.15.44:0/43897816 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f24f00d1800 con 0x7f24f006ce40 2024-04-02T13:29:29.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.662+0000 7f24ecff9640 1 -- 172.21.15.44:0/43897816 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v82) ==== 53+0+73 (secure 0 0 0) 0x7f24e0058dd0 con 0x7f24f006ce40 2024-04-02T13:29:29.666 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:29:29.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.664+0000 7f24f59a3640 1 -- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f24d006c8f0 msgr2=0x7f24d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.664+0000 7f24f59a3640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f24d006c8f0 0x7f24d006edb0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f24d8035000 tx=0x7f24d8008030 comp rx=0 tx=0).stop 2024-04-02T13:29:29.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.664+0000 7f24f59a3640 1 -- 172.21.15.44:0/43897816 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f006ce40 msgr2=0x7f24f015a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:29.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.664+0000 7f24f59a3640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f006ce40 0x7f24f015a660 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f24e000c780 tx=0x7f24e000cc40 comp rx=0 tx=0).stop 2024-04-02T13:29:29.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.664+0000 7f24f59a3640 1 -- 172.21.15.44:0/43897816 shutdown_connections 2024-04-02T13:29:29.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.664+0000 7f24f59a3640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f24d006c8f0 0x7f24d006edb0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.664+0000 7f24f59a3640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24f015e4a0 0x7f24f015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.664+0000 7f24f59a3640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24f00d0c50 0x7f24f015bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.664+0000 7f24f59a3640 1 --2- 172.21.15.44:0/43897816 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f006ce40 0x7f24f015a660 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:29.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.664+0000 7f24f59a3640 1 -- 172.21.15.44:0/43897816 >> 172.21.15.44:0/43897816 conn(0x7f24f009aa20 msgr2=0x7f24f00aebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:29.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.664+0000 7f24f59a3640 1 -- 172.21.15.44:0/43897816 shutdown_connections 2024-04-02T13:29:29.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29.664+0000 7f24f59a3640 1 -- 172.21.15.44:0/43897816 wait complete. 2024-04-02T13:29:29.678 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:29:29.678 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:29:29.678 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:29:29.678 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:29:29.678 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:29.678 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:29:29.678 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:29:29.678 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:29:29.679 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:29:29.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:29.679 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:29:29.694 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:29:29.694 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:29:29.694 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:29:29.694 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:29.694 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:29:29.694 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:29.695 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:29:29.695 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:29:29.695 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:29:29.695 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:29:29.695 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:29:29.695 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:29:29.695 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:29:29.695 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:29:29.695 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:29:29.695 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:29:29.695 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42209sec preferred_lft 42209sec 2024-04-02T13:29:29.695 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:29:29.696 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:29.696 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:29:29.696 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:29:29.696 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:29:29.696 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:29:29.696 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:29:29.696 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:29:29.696 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:29.696 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:29:29.696 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:29.697 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:29:29.697 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:29:29.697 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:29:29.697 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:29.697 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:29:29.697 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:29:29.697 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:29:29.697 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:29:29.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:29.697 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:29:29.760 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:29:29.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:29.761 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:29:29.816 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:29:29.816 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:29:29.816 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:29:29.816 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:29:29.816 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:29:29.817 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:29:29.817 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:29:29.817 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:29:29.817 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:29:29.817 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:29:29.817 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:29:29.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:29:29.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:29:29.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:29.951 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:29:29.965 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:29:29.965 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:29:29.966 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:29:29.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:29.966 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:29:30.032 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:29:30.032 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:29:30.032 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:30.087 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:29:30.089 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:29:30.089 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:30.141 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:29:30.208 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:29:30.264 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:29:30.264 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:29:30.264 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:29:30.264 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:29:30.264 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:29:30.264 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:29:30.265 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:29:30.265 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:29:30.265 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:29:30.265 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:29:30.265 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:29:30.265 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:29:30.265 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:29:30.265 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:29:30.265 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:29:30.266 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:29:30.266 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:29:30.266 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:29:30.266 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:29:30.266 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:29:30.266 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:29:30.266 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:29:30.266 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:29:30.266 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:29:30.266 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:29:30.267 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:29:30.267 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:29:30.267 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:29:30.267 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:29:30.267 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:29:30.267 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:29:30.267 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:29:30.267 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:29:30.267 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:29:30.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:30.268 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:29:30.318 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:29:30.319 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:29:30.361 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:29:30.387 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:29:30.443 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:29:30.443 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:29:30.443 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:29:30.443 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:29:30.443 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:29:30.443 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:29:30.443 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:29:30.443 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:29:30.444 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:29:30.445 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:29:30.445 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:29:30.445 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:29:30.445 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:29:30.445 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:29:30.445 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:29:30.445 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:29:30.445 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:29:30.445 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:29:30.445 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:29:30.445 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:29:30.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:30.446 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:29:30.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:29:30.442+0000 7f67f643a480 -1 init, newargv = 0x5574760650a0 newargc=15 2024-04-02T13:29:30.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[77459]: starting ceph client 2024-04-02T13:29:30.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[77459]: starting fuse 2024-04-02T13:29:30.460 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:29:30.460 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:29:30.461 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:29:30.461 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:29:30.461 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:29:30.461 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:29:30.461 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:29:30.461 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:29:30.461 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:29:30.461 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:29:30.461 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:29:30.461 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:29:30.461 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:29:30.461 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:29:30.461 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:29:30.462 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:29:30.582 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.77459.asok 2024-04-02T13:29:30.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:29:30.593 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.77459.asok 2024-04-02T13:29:30.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:30.593 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.77459.asok status 2024-04-02T13:29:30.730 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:29:30.730 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:29:30.730 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:29:30.730 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:29:30.730 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:29:30.730 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:29:30.730 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:29:30.730 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "77459", 2024-04-02T13:29:30.730 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:29:30.731 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:30.731 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:29:30.731 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:29:30.731 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 9030, 2024-04-02T13:29:30.731 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:29:30.731 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:29:30.731 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:29:30.731 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 9030 2024-04-02T13:29:30.731 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:30.731 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:29:30.731 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:29:30.731 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:29:30.732 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2570671089 2024-04-02T13:29:30.732 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:30.732 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:30.732 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:29:30.732 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:29:30.732 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:29:30.732 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2570671089 2024-04-02T13:29:30.732 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:29:30.732 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.9030 192.168.0.1:0/2570671089", 2024-04-02T13:29:30.733 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/2570671089", 2024-04-02T13:29:30.733 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:29:30.733 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 82, 2024-04-02T13:29:30.733 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 106, 2024-04-02T13:29:30.733 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:29:30.733 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:29:30.733 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:29:30.733 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:29:30.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:30.738 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:29:30.738 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:30.753 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:29:30.753 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:30.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:30.753 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:30.822 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:29:30.822 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:29:30.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:30.878 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file2_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-04-02T13:29:30.977 INFO:teuthology.orchestra.run.smithi044.stderr:6+0 records in 2024-04-02T13:29:30.977 INFO:teuthology.orchestra.run.smithi044.stderr:6+0 records out 2024-04-02T13:29:30.977 INFO:teuthology.orchestra.run.smithi044.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0400541 s, 157 MB/s 2024-04-02T13:29:30.978 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:29:30.978 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:29:30.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:30.993 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file3_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-04-02T13:29:31.090 INFO:teuthology.orchestra.run.smithi044.stderr:6+0 records in 2024-04-02T13:29:31.091 INFO:teuthology.orchestra.run.smithi044.stderr:6+0 records out 2024-04-02T13:29:31.091 INFO:teuthology.orchestra.run.smithi044.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0408058 s, 154 MB/s 2024-04-02T13:29:31.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:31.091 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-04-02T13:29:31.109 INFO:teuthology.orchestra.run.smithi044.stdout:./ 2024-04-02T13:29:31.109 INFO:teuthology.orchestra.run.smithi044.stdout:./file2_sixmegs 2024-04-02T13:29:31.109 INFO:teuthology.orchestra.run.smithi044.stdout:./file3_sixmegs 2024-04-02T13:29:31.109 INFO:teuthology.orchestra.run.smithi044.stdout:./file1_sixmegs 2024-04-02T13:29:31.110 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:29:31.110 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:29:31.110 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:29:31.110 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:29:31.110 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-04-02T13:29:31.110 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:29:31.216 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:29:31.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:29:31.226 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:29:31.226 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:29:31.226 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:29:31.226 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:29:31.226 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file2_sixmegs").st_ino) 2024-04-02T13:29:31.226 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:29:31.331 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627777 2024-04-02T13:29:31.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:29:31.340 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:29:31.340 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:29:31.340 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:29:31.340 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:29:31.340 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file3_sixmegs").st_ino) 2024-04-02T13:29:31.340 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:29:31.445 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627778 2024-04-02T13:29:31.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:29:31.459 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:29:31.459 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:29:31.459 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:29:31.459 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:29:31.459 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file1_sixmegs").st_ino) 2024-04-02T13:29:31.459 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:29:31.565 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627776 2024-04-02T13:29:31.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:29:31.574 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:29:31.574 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:29:31.629 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:29:31.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.767+0000 7f1860915640 1 Processor -- start 2024-04-02T13:29:31.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.767+0000 7f1860915640 1 -- start start 2024-04-02T13:29:31.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.768+0000 7f1860915640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f185c1617e0 0x7f185c0ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:31.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.768+0000 7f1860915640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f185c0cffd0 0x7f185c1629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:31.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.768+0000 7f1860915640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f185c0cef30 con 0x7f185c0ccc90 2024-04-02T13:29:31.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.768+0000 7f1860915640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f185c162f00 con 0x7f185c0cffd0 2024-04-02T13:29:31.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.768+0000 7f1860915640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f185c0aea90 con 0x7f185c1617e0 2024-04-02T13:29:31.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.768+0000 7f1853fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f185c0cffd0 0x7f185c1629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:31.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.768+0000 7f1853fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f185c0cffd0 0x7f185c1629c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36158/0 (socket says 172.21.15.44:36158) 2024-04-02T13:29:31.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.768+0000 7f1853fff640 1 -- 172.21.15.44:0/2891579736 learned_addr learned my addr 172.21.15.44:0/2891579736 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:31.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.768+0000 7f185a59c640 1 --2- 172.21.15.44:0/2891579736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f185c1617e0 0x7f185c0ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:31.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.768+0000 7f1853fff640 1 -- 172.21.15.44:0/2891579736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f185c1617e0 msgr2=0x7f185c0ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:31.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.768+0000 7f1853fff640 1 --2- 172.21.15.44:0/2891579736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f185c1617e0 0x7f185c0ce8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:31.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.768+0000 7f1853fff640 1 -- 172.21.15.44:0/2891579736 >> v1:172.21.15.69:6789/0 conn(0x7f185c0ccc90 legacy=0x7f185c0cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:31.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.768+0000 7f1853fff640 1 -- 172.21.15.44:0/2891579736 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185c0cf210 con 0x7f185c0cffd0 2024-04-02T13:29:31.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.769+0000 7f1853fff640 1 --2- 172.21.15.44:0/2891579736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f185c0cffd0 0x7f185c1629c0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f1844009870 tx=0x7f184402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4acc2cb999e92d71 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:31.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.769+0000 7f1859d9b640 1 -- 172.21.15.44:0/2891579736 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18440317c0 con 0x7f185c0cffd0 2024-04-02T13:29:31.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.769+0000 7f1859d9b640 1 -- 172.21.15.44:0/2891579736 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1844037070 con 0x7f185c0cffd0 2024-04-02T13:29:31.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.769+0000 7f1859d9b640 1 -- 172.21.15.44:0/2891579736 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844031b80 con 0x7f185c0cffd0 2024-04-02T13:29:31.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.770+0000 7f1860915640 1 -- 172.21.15.44:0/2891579736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f185c0cffd0 msgr2=0x7f185c1629c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:31.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.770+0000 7f1860915640 1 --2- 172.21.15.44:0/2891579736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f185c0cffd0 0x7f185c1629c0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f1844009870 tx=0x7f184402b860 comp rx=0 tx=0).stop 2024-04-02T13:29:31.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.770+0000 7f1860915640 1 -- 172.21.15.44:0/2891579736 shutdown_connections 2024-04-02T13:29:31.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.770+0000 7f1860915640 1 --2- 172.21.15.44:0/2891579736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f185c0cffd0 0x7f185c1629c0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:31.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.770+0000 7f1860915640 1 --2- 172.21.15.44:0/2891579736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f185c1617e0 0x7f185c0ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:31.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.770+0000 7f1860915640 1 -- 172.21.15.44:0/2891579736 >> 172.21.15.44:0/2891579736 conn(0x7f185c0b20e0 msgr2=0x7f185c0b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:31.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.770+0000 7f1860915640 1 -- 172.21.15.44:0/2891579736 shutdown_connections 2024-04-02T13:29:31.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.770+0000 7f1860915640 1 -- 172.21.15.44:0/2891579736 wait complete. 2024-04-02T13:29:31.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.771+0000 7f1860915640 1 Processor -- start 2024-04-02T13:29:31.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.771+0000 7f1860915640 1 -- start start 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1860915640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f185c0ccc90 0x7f185c15c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1860915640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f185c0cffd0 0x7f185c15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1860915640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f185c1617e0 0x7f185c159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1860915640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f185c0cfab0 con 0x7f185c0cffd0 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1860915640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f185c0cf7d0 con 0x7f185c1617e0 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1860915640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f185c161bc0 con 0x7f185c0ccc90 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1853fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f185c0cffd0 0x7f185c15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1853fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f185c0cffd0 0x7f185c15b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36166/0 (socket says 172.21.15.44:36166) 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1853fff640 1 -- 172.21.15.44:0/893813480 learned_addr learned my addr 172.21.15.44:0/893813480 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f185a59c640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f185c0ccc90 0x7f185c15c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f185ad9d640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f185c1617e0 0x7f185c159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1853fff640 1 -- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f185c0ccc90 msgr2=0x7f185c15c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1853fff640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f185c0ccc90 0x7f185c15c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1853fff640 1 -- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f185c1617e0 msgr2=0x7f185c159860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:31.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1853fff640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f185c1617e0 0x7f185c159860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:31.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1853fff640 1 -- 172.21.15.44:0/893813480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185c06e4b0 con 0x7f185c0cffd0 2024-04-02T13:29:31.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f185a59c640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f185c0ccc90 0x7f185c15c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:31.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f185ad9d640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f185c1617e0 0x7f185c159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:31.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1853fff640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f185c0cffd0 0x7f185c15b1d0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f1844000c00 tx=0x7f18440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:31.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.772+0000 7f1852ffd640 1 -- 172.21.15.44:0/893813480 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844032860 con 0x7f185c0cffd0 2024-04-02T13:29:31.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.773+0000 7f1852ffd640 1 -- 172.21.15.44:0/893813480 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1844032a20 con 0x7f185c0cffd0 2024-04-02T13:29:31.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.773+0000 7f1860915640 1 -- 172.21.15.44:0/893813480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f185c06f230 con 0x7f185c0cffd0 2024-04-02T13:29:31.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.773+0000 7f1852ffd640 1 -- 172.21.15.44:0/893813480 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844032e10 con 0x7f185c0cffd0 2024-04-02T13:29:31.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.773+0000 7f1860915640 1 -- 172.21.15.44:0/893813480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f185c06ea30 con 0x7f185c0cffd0 2024-04-02T13:29:31.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.774+0000 7f1860915640 1 -- 172.21.15.44:0/893813480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f185c0684f0 con 0x7f185c0cffd0 2024-04-02T13:29:31.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.774+0000 7f1852ffd640 1 -- 172.21.15.44:0/893813480 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f184404c060 con 0x7f185c0cffd0 2024-04-02T13:29:31.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.774+0000 7f1852ffd640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f183c06c920 0x7f183c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:31.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.775+0000 7f1852ffd640 1 -- 172.21.15.44:0/893813480 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f18440ad540 con 0x7f185c0cffd0 2024-04-02T13:29:31.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.775+0000 7f185a59c640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f183c06c920 0x7f183c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:31.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.775+0000 7f185a59c640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f183c06c920 0x7f183c06ede0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f184c00b820 tx=0x7f184c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:31.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.779+0000 7f1852ffd640 1 -- 172.21.15.44:0/893813480 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f1844040070 con 0x7f185c0cffd0 2024-04-02T13:29:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.968+0000 7f1860915640 1 -- 172.21.15.44:0/893813480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f185c0ce190 con 0x7f185c0cffd0 2024-04-02T13:29:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.969+0000 7f1852ffd640 1 -- 172.21.15.44:0/893813480 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+16836 (secure 0 0 0) 0x7f184407b870 con 0x7f185c0cffd0 2024-04-02T13:29:31.973 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:31.973 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":106,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:29:12.549182+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:29:09.192430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-02T13:29:09.860748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:29:31.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.972+0000 7f1860915640 1 -- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f183c06c920 msgr2=0x7f183c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:31.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.972+0000 7f1860915640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f183c06c920 0x7f183c06ede0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f184c00b820 tx=0x7f184c008040 comp rx=0 tx=0).stop 2024-04-02T13:29:31.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.972+0000 7f1860915640 1 -- 172.21.15.44:0/893813480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f185c0cffd0 msgr2=0x7f185c15b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:31.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.972+0000 7f1860915640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f185c0cffd0 0x7f185c15b1d0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f1844000c00 tx=0x7f18440046c0 comp rx=0 tx=0).stop 2024-04-02T13:29:31.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.972+0000 7f1860915640 1 -- 172.21.15.44:0/893813480 shutdown_connections 2024-04-02T13:29:31.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.972+0000 7f1860915640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f183c06c920 0x7f183c06ede0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:31.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.972+0000 7f1860915640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f185c1617e0 0x7f185c159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:31.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.972+0000 7f1860915640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f185c0cffd0 0x7f185c15b1d0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:31.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.972+0000 7f1860915640 1 --2- 172.21.15.44:0/893813480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f185c0ccc90 0x7f185c15c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:31.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.972+0000 7f1860915640 1 -- 172.21.15.44:0/893813480 >> 172.21.15.44:0/893813480 conn(0x7f185c0b20e0 msgr2=0x7f185c071f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:31.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.972+0000 7f1860915640 1 -- 172.21.15.44:0/893813480 shutdown_connections 2024-04-02T13:29:31.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:31.972+0000 7f1860915640 1 -- 172.21.15.44:0/893813480 wait complete. 2024-04-02T13:29:31.986 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:29:09.192430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:29:09.860748+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:29:31.987 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:29:32.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.086+0000 7f90043cd640 1 Processor -- start 2024-04-02T13:29:32.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.087+0000 7f90043cd640 1 -- start start 2024-04-02T13:29:32.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.087+0000 7f90043cd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8ffc0b1440 0x7f8ffc0b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.087+0000 7f90043cd640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ffc0cca50 con 0x7f8ffc0d03d0 2024-04-02T13:29:32.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.087+0000 7f90043cd640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ffc0cef40 con 0x7f8ffc15c220 2024-04-02T13:29:32.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.087+0000 7f90043cd640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f8ffc0ccd60 con 0x7f8ffc0b1440 2024-04-02T13:29:32.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.088+0000 7f900296a640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f8ffc0d03d0 0x7f8ffc0cd1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32852/0 (socket says 172.21.15.44:32852) 2024-04-02T13:29:32.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.088+0000 7f9001968640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8ffc0b1440 0x7f8ffc0b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.088+0000 7f900296a640 1 -- 172.21.15.44:0/815621592 learned_addr learned my addr 172.21.15.44:0/815621592 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:32.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.088+0000 7f9001968640 1 -- 172.21.15.44:0/815621592 >> v1:172.21.15.69:6789/0 conn(0x7f8ffc15c220 legacy=0x7f8ffc0ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:32.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.088+0000 7f9001968640 1 -- 172.21.15.44:0/815621592 >> v1:172.21.15.44:6789/0 conn(0x7f8ffc0d03d0 legacy=0x7f8ffc0cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.088+0000 7f9001968640 1 -- 172.21.15.44:0/815621592 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ffc0cf220 con 0x7f8ffc0b1440 2024-04-02T13:29:32.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.088+0000 7f9001167640 1 -- 172.21.15.44:0/815621592 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2877565724 0 0) 0x7f8ff4001490 con 0x7f8ffc0d03d0 2024-04-02T13:29:32.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.089+0000 7f9001968640 1 --2- 172.21.15.44:0/815621592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ffc0b1440 0x7f8ffc0b1db0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0097a0 tx=0x7f8fec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a60626cf247280 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:32.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.089+0000 7f9001167640 1 -- 172.21.15.44:0/815621592 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec035020 con 0x7f8ffc0b1440 2024-04-02T13:29:32.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.089+0000 7f9001167640 1 -- 172.21.15.44:0/815621592 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fec031830 con 0x7f8ffc0b1440 2024-04-02T13:29:32.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.089+0000 7f9001167640 1 -- 172.21.15.44:0/815621592 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec031c00 con 0x7f8ffc0b1440 2024-04-02T13:29:32.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.089+0000 7f90043cd640 1 -- 172.21.15.44:0/815621592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ffc0b1440 msgr2=0x7f8ffc0b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.089+0000 7f90043cd640 1 --2- 172.21.15.44:0/815621592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ffc0b1440 0x7f8ffc0b1db0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0097a0 tx=0x7f8fec02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.090+0000 7f90043cd640 1 -- 172.21.15.44:0/815621592 shutdown_connections 2024-04-02T13:29:32.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.090+0000 7f90043cd640 1 --2- 172.21.15.44:0/815621592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ffc0b1440 0x7f8ffc0b1db0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.090+0000 7f90043cd640 1 -- 172.21.15.44:0/815621592 >> 172.21.15.44:0/815621592 conn(0x7f8ffc09aa20 msgr2=0x7f8ffc09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:32.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.090+0000 7f90043cd640 1 -- 172.21.15.44:0/815621592 shutdown_connections 2024-04-02T13:29:32.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.090+0000 7f90043cd640 1 -- 172.21.15.44:0/815621592 wait complete. 2024-04-02T13:29:32.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.091+0000 7f90043cd640 1 Processor -- start 2024-04-02T13:29:32.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.091+0000 7f90043cd640 1 -- start start 2024-04-02T13:29:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.091+0000 7f90043cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ffc0b1440 0x7f8ffc156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.091+0000 7f90043cd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ffc0d03d0 0x7f8ffc15d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.091+0000 7f90043cd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ffc15c220 0x7f8ffc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.091+0000 7f90043cd640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc0d0880 con 0x7f8ffc0b1440 2024-04-02T13:29:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.091+0000 7f90043cd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc15c600 con 0x7f8ffc15c220 2024-04-02T13:29:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.091+0000 7f90043cd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8ffc0d0a00 con 0x7f8ffc0d03d0 2024-04-02T13:29:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.091+0000 7f9002169640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ffc0b1440 0x7f8ffc156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f9002169640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ffc0b1440 0x7f8ffc156f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36180/0 (socket says 172.21.15.44:36180) 2024-04-02T13:29:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f9002169640 1 -- 172.21.15.44:0/3569189214 learned_addr learned my addr 172.21.15.44:0/3569189214 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f9001968640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ffc0d03d0 0x7f8ffc15d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f900296a640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ffc15c220 0x7f8ffc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f9002169640 1 -- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ffc0d03d0 msgr2=0x7f8ffc15d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f9002169640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ffc0d03d0 0x7f8ffc15d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f9002169640 1 -- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ffc15c220 msgr2=0x7f8ffc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f9002169640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ffc15c220 0x7f8ffc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f9002169640 1 -- 172.21.15.44:0/3569189214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ffc15e160 con 0x7f8ffc0b1440 2024-04-02T13:29:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f9001968640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ffc0d03d0 0x7f8ffc15d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f900296a640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ffc15c220 0x7f8ffc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f9002169640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ffc0b1440 0x7f8ffc156f20 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800b3f0 tx=0x7f8ff800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f8ff2ffd640 1 -- 172.21.15.44:0/3569189214 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8015020 con 0x7f8ffc0b1440 2024-04-02T13:29:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f8ff2ffd640 1 -- 172.21.15.44:0/3569189214 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ff8011830 con 0x7f8ffc0b1440 2024-04-02T13:29:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f90043cd640 1 -- 172.21.15.44:0/3569189214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ffc15e440 con 0x7f8ffc0b1440 2024-04-02T13:29:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f8ff2ffd640 1 -- 172.21.15.44:0/3569189214 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8011c20 con 0x7f8ffc0b1440 2024-04-02T13:29:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.092+0000 7f90043cd640 1 -- 172.21.15.44:0/3569189214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ffc15ece0 con 0x7f8ffc0b1440 2024-04-02T13:29:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.093+0000 7f90043cd640 1 -- 172.21.15.44:0/3569189214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ffc15e830 con 0x7f8ffc0b1440 2024-04-02T13:29:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.094+0000 7f8ff2ffd640 1 -- 172.21.15.44:0/3569189214 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8ff8018020 con 0x7f8ffc0b1440 2024-04-02T13:29:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.094+0000 7f8ff2ffd640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8fe006c8f0 0x7f8fe006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.094+0000 7f8ff2ffd640 1 -- 172.21.15.44:0/3569189214 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f8ff808b900 con 0x7f8ffc0b1440 2024-04-02T13:29:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.094+0000 7f9001968640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8fe006c8f0 0x7f8fe006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.095+0000 7f9001968640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8fe006c8f0 0x7f8fe006edb0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8fec036000 tx=0x7f8fec008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:32.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.098+0000 7f8ff2ffd640 1 -- 172.21.15.44:0/3569189214 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8ff801f030 con 0x7f8ffc0b1440 2024-04-02T13:29:32.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.279+0000 7f90043cd640 1 -- 172.21.15.44:0/3569189214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8ffc0d2d10 con 0x7f8ffc0b1440 2024-04-02T13:29:32.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.280+0000 7f8ff2ffd640 1 -- 172.21.15.44:0/3569189214 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7f8ff80172c0 con 0x7f8ffc0b1440 2024-04-02T13:29:32.284 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:29:32.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.282+0000 7f90043cd640 1 -- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8fe006c8f0 msgr2=0x7f8fe006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.282+0000 7f90043cd640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8fe006c8f0 0x7f8fe006edb0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8fec036000 tx=0x7f8fec008030 comp rx=0 tx=0).stop 2024-04-02T13:29:32.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.282+0000 7f90043cd640 1 -- 172.21.15.44:0/3569189214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ffc0b1440 msgr2=0x7f8ffc156f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.282+0000 7f90043cd640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ffc0b1440 0x7f8ffc156f20 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800b3f0 tx=0x7f8ff800b8b0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.282+0000 7f90043cd640 1 -- 172.21.15.44:0/3569189214 shutdown_connections 2024-04-02T13:29:32.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.282+0000 7f90043cd640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8fe006c8f0 0x7f8fe006edb0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.282+0000 7f90043cd640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ffc15c220 0x7f8ffc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.282+0000 7f90043cd640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ffc0d03d0 0x7f8ffc15d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.282+0000 7f90043cd640 1 --2- 172.21.15.44:0/3569189214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ffc0b1440 0x7f8ffc156f20 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.282+0000 7f90043cd640 1 -- 172.21.15.44:0/3569189214 >> 172.21.15.44:0/3569189214 conn(0x7f8ffc09aa20 msgr2=0x7f8ffc070550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:32.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.282+0000 7f90043cd640 1 -- 172.21.15.44:0/3569189214 shutdown_connections 2024-04-02T13:29:32.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.282+0000 7f90043cd640 1 -- 172.21.15.44:0/3569189214 wait complete. 2024-04-02T13:29:32.297 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:29:32.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.397+0000 7f5ba2888640 1 Processor -- start 2024-04-02T13:29:32.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.397+0000 7f5ba2888640 1 -- start start 2024-04-02T13:29:32.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5ba2888640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5b9c15c220 0x7f5b9c0b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5ba2888640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5b9c0d03d0 0x7f5b9c0d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5ba2888640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b9c0cca50 con 0x7f5b9c0b1440 2024-04-02T13:29:32.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5ba2888640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f5b9c0ccd60 con 0x7f5b9c0d03d0 2024-04-02T13:29:32.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5ba2888640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5b9c0cef40 con 0x7f5b9c15c220 2024-04-02T13:29:32.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5ba0e25640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5b9c0d03d0 0x7f5b9c0d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5b9b7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f5b9c0b1440 0x7f5b9c0ce780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32866/0 (socket says 172.21.15.44:32866) 2024-04-02T13:29:32.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5b9b7fe640 1 -- 172.21.15.44:0/596774803 learned_addr learned my addr 172.21.15.44:0/596774803 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:32.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5ba0e25640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5b9c0d03d0 0x7f5b9c0d0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36184/0 (socket says 172.21.15.44:36184) 2024-04-02T13:29:32.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5b9bfff640 1 --2- 172.21.15.44:0/596774803 >> v2:172.21.15.69:3300/0 conn(0x7f5b9c15c220 0x7f5b9c0b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5ba0e25640 1 -- 172.21.15.44:0/596774803 >> v2:172.21.15.69:3300/0 conn(0x7f5b9c15c220 msgr2=0x7f5b9c0b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:32.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5ba0e25640 1 --2- 172.21.15.44:0/596774803 >> v2:172.21.15.69:3300/0 conn(0x7f5b9c15c220 0x7f5b9c0b1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5ba0e25640 1 -- 172.21.15.44:0/596774803 >> v1:172.21.15.44:6789/0 conn(0x7f5b9c0b1440 legacy=0x7f5b9c0ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.398+0000 7f5ba0e25640 1 -- 172.21.15.44:0/596774803 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9c0cf220 con 0x7f5b9c0d03d0 2024-04-02T13:29:32.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.399+0000 7f5ba0e25640 1 --2- 172.21.15.44:0/596774803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c0d03d0 0x7f5b9c0d0880 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0ce540 tx=0x7f5b9402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a093a35df762a9f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:32.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.399+0000 7f5b9affd640 1 -- 172.21.15.44:0/596774803 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b94039070 con 0x7f5b9c0d03d0 2024-04-02T13:29:32.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.399+0000 7f5b9affd640 1 -- 172.21.15.44:0/596774803 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b9402cc80 con 0x7f5b9c0d03d0 2024-04-02T13:29:32.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.399+0000 7f5b9affd640 1 -- 172.21.15.44:0/596774803 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b94034660 con 0x7f5b9c0d03d0 2024-04-02T13:29:32.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.400+0000 7f5ba2888640 1 -- 172.21.15.44:0/596774803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c0d03d0 msgr2=0x7f5b9c0d0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.400+0000 7f5ba2888640 1 --2- 172.21.15.44:0/596774803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c0d03d0 0x7f5b9c0d0880 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0ce540 tx=0x7f5b9402bed0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.400+0000 7f5ba2888640 1 -- 172.21.15.44:0/596774803 shutdown_connections 2024-04-02T13:29:32.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.400+0000 7f5ba2888640 1 --2- 172.21.15.44:0/596774803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c0d03d0 0x7f5b9c0d0880 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.400+0000 7f5ba2888640 1 --2- 172.21.15.44:0/596774803 >> v2:172.21.15.69:3300/0 conn(0x7f5b9c15c220 0x7f5b9c0b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.400+0000 7f5ba2888640 1 -- 172.21.15.44:0/596774803 >> 172.21.15.44:0/596774803 conn(0x7f5b9c09aa20 msgr2=0x7f5b9c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:32.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.400+0000 7f5ba2888640 1 -- 172.21.15.44:0/596774803 shutdown_connections 2024-04-02T13:29:32.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.400+0000 7f5ba2888640 1 -- 172.21.15.44:0/596774803 wait complete. 2024-04-02T13:29:32.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.401+0000 7f5ba2888640 1 Processor -- start 2024-04-02T13:29:32.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.401+0000 7f5ba2888640 1 -- start start 2024-04-02T13:29:32.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5ba2888640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c0b1440 0x7f5b9c0c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5ba2888640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b9c0d03d0 0x7f5b9c0b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5ba2888640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b9c15c220 0x7f5b9c15c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5ba2888640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c0cd1f0 con 0x7f5b9c0b1440 2024-04-02T13:29:32.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5ba2888640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c15c600 con 0x7f5b9c15c220 2024-04-02T13:29:32.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5ba2888640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5b9c0cd370 con 0x7f5b9c0d03d0 2024-04-02T13:29:32.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5b9bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c0b1440 0x7f5b9c0c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5b9bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c0b1440 0x7f5b9c0c59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36192/0 (socket says 172.21.15.44:36192) 2024-04-02T13:29:32.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5b9bfff640 1 -- 172.21.15.44:0/1166270804 learned_addr learned my addr 172.21.15.44:0/1166270804 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:32.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5b9b7fe640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b9c0d03d0 0x7f5b9c0b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5ba0e25640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b9c15c220 0x7f5b9c15c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5b9bfff640 1 -- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b9c0d03d0 msgr2=0x7f5b9c0b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5b9bfff640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b9c0d03d0 0x7f5b9c0b0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.402+0000 7f5b9bfff640 1 -- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b9c15c220 msgr2=0x7f5b9c15c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.403+0000 7f5b9bfff640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b9c15c220 0x7f5b9c15c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.403+0000 7f5b9bfff640 1 -- 172.21.15.44:0/1166270804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9c15ccc0 con 0x7f5b9c0b1440 2024-04-02T13:29:32.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.403+0000 7f5ba0e25640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b9c15c220 0x7f5b9c15c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:32.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.403+0000 7f5b9bfff640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c0b1440 0x7f5b9c0c59b0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400e870 tx=0x7f5b8400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:32.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.403+0000 7f5b98ff9640 1 -- 172.21.15.44:0/1166270804 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84017070 con 0x7f5b9c0b1440 2024-04-02T13:29:32.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.403+0000 7f5b98ff9640 1 -- 172.21.15.44:0/1166270804 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b8400c940 con 0x7f5b9c0b1440 2024-04-02T13:29:32.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.403+0000 7f5b98ff9640 1 -- 172.21.15.44:0/1166270804 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8400cd30 con 0x7f5b9c0b1440 2024-04-02T13:29:32.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.403+0000 7f5ba2888640 1 -- 172.21.15.44:0/1166270804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b9c0c6070 con 0x7f5b9c0b1440 2024-04-02T13:29:32.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.403+0000 7f5ba2888640 1 -- 172.21.15.44:0/1166270804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b9c0aedb0 con 0x7f5b9c0b1440 2024-04-02T13:29:32.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.404+0000 7f5ba2888640 1 -- 172.21.15.44:0/1166270804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b9c06c000 con 0x7f5b9c0b1440 2024-04-02T13:29:32.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.404+0000 7f5b98ff9640 1 -- 172.21.15.44:0/1166270804 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5b8401a070 con 0x7f5b9c0b1440 2024-04-02T13:29:32.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.405+0000 7f5b98ff9640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b7c06c8f0 0x7f5b7c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.405+0000 7f5b98ff9640 1 -- 172.21.15.44:0/1166270804 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f5b8408bac0 con 0x7f5b9c0b1440 2024-04-02T13:29:32.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.405+0000 7f5b9b7fe640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b7c06c8f0 0x7f5b7c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.405+0000 7f5b9b7fe640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b7c06c8f0 0x7f5b7c06edb0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c000f60 tx=0x7f5b8c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:32.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.409+0000 7f5b98ff9640 1 -- 172.21.15.44:0/1166270804 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5b9c06c000 con 0x7f5b9c0b1440 2024-04-02T13:29:32.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.592+0000 7f5ba2888640 1 -- 172.21.15.44:0/1166270804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5b9c0b1260 con 0x7f5b9c0b1440 2024-04-02T13:29:32.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.593+0000 7f5b98ff9640 1 -- 172.21.15.44:0/1166270804 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) ==== 104+0+11 (secure 0 0 0) 0x7f5b9c0b1260 con 0x7f5b9c0b1440 2024-04-02T13:29:32.597 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:29:32.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.594+0000 7f5ba2888640 1 -- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b7c06c8f0 msgr2=0x7f5b7c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.594+0000 7f5ba2888640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b7c06c8f0 0x7f5b7c06edb0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c000f60 tx=0x7f5b8c009040 comp rx=0 tx=0).stop 2024-04-02T13:29:32.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.594+0000 7f5ba2888640 1 -- 172.21.15.44:0/1166270804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c0b1440 msgr2=0x7f5b9c0c59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.594+0000 7f5ba2888640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c0b1440 0x7f5b9c0c59b0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400e870 tx=0x7f5b8400ed30 comp rx=0 tx=0).stop 2024-04-02T13:29:32.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.595+0000 7f5ba2888640 1 -- 172.21.15.44:0/1166270804 shutdown_connections 2024-04-02T13:29:32.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.595+0000 7f5ba2888640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5b7c06c8f0 0x7f5b7c06edb0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.595+0000 7f5ba2888640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b9c15c220 0x7f5b9c15c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.595+0000 7f5ba2888640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b9c0d03d0 0x7f5b9c0b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.595+0000 7f5ba2888640 1 --2- 172.21.15.44:0/1166270804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c0b1440 0x7f5b9c0c59b0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.595+0000 7f5ba2888640 1 -- 172.21.15.44:0/1166270804 >> 172.21.15.44:0/1166270804 conn(0x7f5b9c09aa20 msgr2=0x7f5b9c0d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:32.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.595+0000 7f5ba2888640 1 -- 172.21.15.44:0/1166270804 shutdown_connections 2024-04-02T13:29:32.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.595+0000 7f5ba2888640 1 -- 172.21.15.44:0/1166270804 wait complete. 2024-04-02T13:29:32.609 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:29:32.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.709+0000 7f42a5711640 1 Processor -- start 2024-04-02T13:29:32.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.710+0000 7f42a5711640 1 -- start start 2024-04-02T13:29:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.710+0000 7f42a5711640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f42a00cd810 0x7f42a00b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.710+0000 7f42a5711640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42a006d520 0x7f42a006d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.710+0000 7f42a5711640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42a00cea30 con 0x7f42a00b1c30 2024-04-02T13:29:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.710+0000 7f42a5711640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f42a00ce800 con 0x7f42a00cd810 2024-04-02T13:29:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.710+0000 7f42a5711640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f42a00ced10 con 0x7f42a006d520 2024-04-02T13:29:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.710+0000 7f429e7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f42a00cd810 0x7f42a00b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.710+0000 7f429e7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f42a00cd810 0x7f42a00b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36196/0 (socket says 172.21.15.44:36196) 2024-04-02T13:29:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.710+0000 7f429e7fc640 1 -- 172.21.15.44:0/2665700588 learned_addr learned my addr 172.21.15.44:0/2665700588 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.711+0000 7f429f7fe640 1 --2- 172.21.15.44:0/2665700588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42a006d520 0x7f42a006d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.711+0000 7f429e7fc640 1 -- 172.21.15.44:0/2665700588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42a006d520 msgr2=0x7f42a006d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.711+0000 7f429e7fc640 1 --2- 172.21.15.44:0/2665700588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42a006d520 0x7f42a006d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.711+0000 7f429e7fc640 1 -- 172.21.15.44:0/2665700588 >> v1:172.21.15.69:6789/0 conn(0x7f42a00b1c30 legacy=0x7f42a00b2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.711+0000 7f429e7fc640 1 -- 172.21.15.44:0/2665700588 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a00ceff0 con 0x7f42a00cd810 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.711+0000 7f429e7fc640 1 --2- 172.21.15.44:0/2665700588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a00cd810 0x7f42a00b3290 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f4294009870 tx=0x7f429402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b613905a24888ff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.712+0000 7f429dffb640 1 -- 172.21.15.44:0/2665700588 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4294035020 con 0x7f42a00cd810 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.712+0000 7f429dffb640 1 -- 172.21.15.44:0/2665700588 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4294031640 con 0x7f42a00cd810 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.712+0000 7f429dffb640 1 -- 172.21.15.44:0/2665700588 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429402d040 con 0x7f42a00cd810 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.712+0000 7f42a5711640 1 -- 172.21.15.44:0/2665700588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a00cd810 msgr2=0x7f42a00b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.712+0000 7f42a5711640 1 --2- 172.21.15.44:0/2665700588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a00cd810 0x7f42a00b3290 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f4294009870 tx=0x7f429402b8f0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.712+0000 7f42a5711640 1 -- 172.21.15.44:0/2665700588 shutdown_connections 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.712+0000 7f42a5711640 1 --2- 172.21.15.44:0/2665700588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a00cd810 0x7f42a00b3290 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.712+0000 7f42a5711640 1 --2- 172.21.15.44:0/2665700588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42a006d520 0x7f42a006d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.712+0000 7f42a5711640 1 -- 172.21.15.44:0/2665700588 >> 172.21.15.44:0/2665700588 conn(0x7f42a006b4d0 msgr2=0x7f42a006b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.712+0000 7f42a5711640 1 -- 172.21.15.44:0/2665700588 shutdown_connections 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.712+0000 7f42a5711640 1 -- 172.21.15.44:0/2665700588 wait complete. 2024-04-02T13:29:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.713+0000 7f42a5711640 1 Processor -- start 2024-04-02T13:29:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.714+0000 7f42a5711640 1 -- start start 2024-04-02T13:29:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.714+0000 7f42a5711640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42a006d520 0x7f42a014a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.714+0000 7f42a5711640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a00b1c30 0x7f42a00c7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.714+0000 7f42a5711640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42a00cd810 0x7f42a006dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.714+0000 7f42a5711640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f42a00cfad0 con 0x7f42a00b1c30 2024-04-02T13:29:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.714+0000 7f42a5711640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f42a00cf5b0 con 0x7f42a006d520 2024-04-02T13:29:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.714+0000 7f42a5711640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f42a00cfc50 con 0x7f42a00cd810 2024-04-02T13:29:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.714+0000 7f429e7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a00b1c30 0x7f42a00c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.714+0000 7f429e7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a00b1c30 0x7f42a00c7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36198/0 (socket says 172.21.15.44:36198) 2024-04-02T13:29:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.714+0000 7f429e7fc640 1 -- 172.21.15.44:0/868471512 learned_addr learned my addr 172.21.15.44:0/868471512 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.714+0000 7f429f7fe640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42a00cd810 0x7f42a006dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.714+0000 7f429effd640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42a006d520 0x7f42a014a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.715+0000 7f429e7fc640 1 -- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42a00cd810 msgr2=0x7f42a006dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.715+0000 7f429e7fc640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42a00cd810 0x7f42a006dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.715+0000 7f429e7fc640 1 -- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42a006d520 msgr2=0x7f42a014a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.715+0000 7f429e7fc640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42a006d520 0x7f42a014a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.715+0000 7f429e7fc640 1 -- 172.21.15.44:0/868471512 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a00c9a50 con 0x7f42a00b1c30 2024-04-02T13:29:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.715+0000 7f429e7fc640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a00b1c30 0x7f42a00c7280 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f4294000c00 tx=0x7f42940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.715+0000 7f42877fe640 1 -- 172.21.15.44:0/868471512 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4294040070 con 0x7f42a00b1c30 2024-04-02T13:29:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.715+0000 7f42877fe640 1 -- 172.21.15.44:0/868471512 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42940399f0 con 0x7f42a00b1c30 2024-04-02T13:29:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.715+0000 7f42a5711640 1 -- 172.21.15.44:0/868471512 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a00c9d30 con 0x7f42a00b1c30 2024-04-02T13:29:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.715+0000 7f42877fe640 1 -- 172.21.15.44:0/868471512 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4294035040 con 0x7f42a00b1c30 2024-04-02T13:29:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.715+0000 7f42a5711640 1 -- 172.21.15.44:0/868471512 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a00c77c0 con 0x7f42a00b1c30 2024-04-02T13:29:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.716+0000 7f42a5711640 1 -- 172.21.15.44:0/868471512 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a00684f0 con 0x7f42a00b1c30 2024-04-02T13:29:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.717+0000 7f42877fe640 1 -- 172.21.15.44:0/868471512 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4294038020 con 0x7f42a00b1c30 2024-04-02T13:29:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.717+0000 7f42877fe640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f428c06c920 0x7f428c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.717+0000 7f42877fe640 1 -- 172.21.15.44:0/868471512 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f42940ad4a0 con 0x7f42a00b1c30 2024-04-02T13:29:32.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.717+0000 7f429effd640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f428c06c920 0x7f428c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:32.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.718+0000 7f429effd640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f428c06c920 0x7f428c06ede0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4290004f60 tx=0x7f4290007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:32.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.721+0000 7f42877fe640 1 -- 172.21.15.44:0/868471512 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f429407b180 con 0x7f42a00b1c30 2024-04-02T13:29:32.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.905+0000 7f42a5711640 1 -- 172.21.15.44:0/868471512 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f42a0068ee0 con 0x7f42a00b1c30 2024-04-02T13:29:32.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.906+0000 7f42877fe640 1 -- 172.21.15.44:0/868471512 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) ==== 100+0+11 (secure 0 0 0) 0x7f429407f480 con 0x7f42a00b1c30 2024-04-02T13:29:32.910 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:29:32.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.907+0000 7f42a5711640 1 -- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f428c06c920 msgr2=0x7f428c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.907+0000 7f42a5711640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f428c06c920 0x7f428c06ede0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4290004f60 tx=0x7f4290007e30 comp rx=0 tx=0).stop 2024-04-02T13:29:32.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.907+0000 7f42a5711640 1 -- 172.21.15.44:0/868471512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a00b1c30 msgr2=0x7f42a00c7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:32.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.908+0000 7f42a5711640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a00b1c30 0x7f42a00c7280 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f4294000c00 tx=0x7f42940046c0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.908+0000 7f42a5711640 1 -- 172.21.15.44:0/868471512 shutdown_connections 2024-04-02T13:29:32.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.908+0000 7f42a5711640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f428c06c920 0x7f428c06ede0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.908+0000 7f42a5711640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42a00cd810 0x7f42a006dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.908+0000 7f42a5711640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a00b1c30 0x7f42a00c7280 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.908+0000 7f42a5711640 1 --2- 172.21.15.44:0/868471512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42a006d520 0x7f42a014a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:32.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.908+0000 7f42a5711640 1 -- 172.21.15.44:0/868471512 >> 172.21.15.44:0/868471512 conn(0x7f42a006b4d0 msgr2=0x7f42a00b3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:32.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.908+0000 7f42a5711640 1 -- 172.21.15.44:0/868471512 shutdown_connections 2024-04-02T13:29:32.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:32.908+0000 7f42a5711640 1 -- 172.21.15.44:0/868471512 wait complete. 2024-04-02T13:29:32.922 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:29:33.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.020+0000 7fdb88774640 1 Processor -- start 2024-04-02T13:29:33.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.020+0000 7fdb88774640 1 -- start start 2024-04-02T13:29:33.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.021+0000 7fdb88774640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8006f1d0 0x7fdb800cc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.021+0000 7fdb88774640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb8015fb30 con 0x7fdb800af680 2024-04-02T13:29:33.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.021+0000 7fdb88774640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb8015fdd0 con 0x7fdb800733f0 2024-04-02T13:29:33.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.021+0000 7fdb88774640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdb801600b0 con 0x7fdb8006f1d0 2024-04-02T13:29:33.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.021+0000 7fdb86510640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fdb800af680 0x7fdb80072ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32870/0 (socket says 172.21.15.44:32870) 2024-04-02T13:29:33.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.021+0000 7fdb86510640 1 -- 172.21.15.44:0/1142598551 learned_addr learned my addr 172.21.15.44:0/1142598551 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:33.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.021+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1948817580 0 0) 0x7fdb8015fb30 con 0x7fdb800af680 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.021+0000 7fdb85d0f640 1 --2- 172.21.15.44:0/1142598551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8006f1d0 0x7fdb800cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.021+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb6c003520 con 0x7fdb800af680 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 651493282 0 0) 0x7fdb6c003520 con 0x7fdb800af680 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8006f1d0 msgr2=0x7fdb800cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb8550e640 1 --2- 172.21.15.44:0/1142598551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8006f1d0 0x7fdb800cc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 >> v1:172.21.15.69:6789/0 conn(0x7fdb800733f0 legacy=0x7fdb8006d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb80160390 con 0x7fdb800af680 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fdb78001690 con 0x7fdb800af680 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 >> v1:172.21.15.44:6789/0 conn(0x7fdb800af680 legacy=0x7fdb80072ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb8550e640 1 --2- 172.21.15.44:0/1142598551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c0039e0 0x7fdb6c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb8550e640 1 --2- 172.21.15.44:0/1142598551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb6c004800 0x7fdb6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb8550e640 1 --2- 172.21.15.44:0/1142598551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8006f1d0 0x7fdb6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdb801600b0 con 0x7fdb6c0039e0 2024-04-02T13:29:33.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c00baa0 con 0x7fdb6c004800 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdb6c00bca0 con 0x7fdb8006f1d0 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb86510640 1 --2- 172.21.15.44:0/1142598551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c0039e0 0x7fdb6c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb85d0f640 1 --2- 172.21.15.44:0/1142598551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb6c004800 0x7fdb6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb86d11640 1 --2- 172.21.15.44:0/1142598551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8006f1d0 0x7fdb6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb86510640 1 -- 172.21.15.44:0/1142598551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8006f1d0 msgr2=0x7fdb6c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb86510640 1 --2- 172.21.15.44:0/1142598551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8006f1d0 0x7fdb6c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb86510640 1 -- 172.21.15.44:0/1142598551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb6c004800 msgr2=0x7fdb6c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb86510640 1 --2- 172.21.15.44:0/1142598551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb6c004800 0x7fdb6c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.022+0000 7fdb86510640 1 -- 172.21.15.44:0/1142598551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb6c003520 con 0x7fdb6c0039e0 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.023+0000 7fdb86510640 1 --2- 172.21.15.44:0/1142598551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c0039e0 0x7fdb6c003e00 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fdb8015fd00 tx=0x7fdb6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3f745fc0e68e85 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.023+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb78007d30 con 0x7fdb6c0039e0 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.023+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb78030070 con 0x7fdb6c0039e0 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.023+0000 7fdb8550e640 1 -- 172.21.15.44:0/1142598551 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb78005cf0 con 0x7fdb6c0039e0 2024-04-02T13:29:33.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.023+0000 7fdb88774640 1 -- 172.21.15.44:0/1142598551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c0039e0 msgr2=0x7fdb6c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.023+0000 7fdb88774640 1 --2- 172.21.15.44:0/1142598551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c0039e0 0x7fdb6c003e00 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fdb8015fd00 tx=0x7fdb6c005830 comp rx=0 tx=0).stop 2024-04-02T13:29:33.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.023+0000 7fdb86d11640 1 -- 172.21.15.44:0/1142598551 reap_dead start 2024-04-02T13:29:33.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.024+0000 7fdb88774640 1 -- 172.21.15.44:0/1142598551 shutdown_connections 2024-04-02T13:29:33.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.024+0000 7fdb88774640 1 -- 172.21.15.44:0/1142598551 >> 172.21.15.44:0/1142598551 conn(0x7fdb800b1c30 msgr2=0x7fdb800b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:33.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.024+0000 7fdb88774640 1 -- 172.21.15.44:0/1142598551 shutdown_connections 2024-04-02T13:29:33.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.024+0000 7fdb88774640 1 -- 172.21.15.44:0/1142598551 wait complete. 2024-04-02T13:29:33.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.025+0000 7fdb88774640 1 Processor -- start 2024-04-02T13:29:33.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.025+0000 7fdb88774640 1 -- start start 2024-04-02T13:29:33.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.025+0000 7fdb88774640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c0039e0 0x7fdb800aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.025+0000 7fdb88774640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb800b54e0 0x7fdb8015e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.025+0000 7fdb88774640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb800b5940 0x7fdb8009bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.025+0000 7fdb88774640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdb800cc140 con 0x7fdb6c0039e0 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.025+0000 7fdb88774640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdb80160950 con 0x7fdb800b5940 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.025+0000 7fdb88774640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdb800cc2c0 con 0x7fdb800b54e0 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb86510640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c0039e0 0x7fdb800aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb86510640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c0039e0 0x7fdb800aeac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36222/0 (socket says 172.21.15.44:36222) 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb86510640 1 -- 172.21.15.44:0/2490533797 learned_addr learned my addr 172.21.15.44:0/2490533797 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb86510640 1 -- 172.21.15.44:0/2490533797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb800b54e0 msgr2=0x7fdb8015e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb86510640 1 --2- 172.21.15.44:0/2490533797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb800b54e0 0x7fdb8015e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb86510640 1 -- 172.21.15.44:0/2490533797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb800b5940 msgr2=0x7fdb8009bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb86510640 1 --2- 172.21.15.44:0/2490533797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb800b5940 0x7fdb8009bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb86510640 1 -- 172.21.15.44:0/2490533797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb80155eb0 con 0x7fdb6c0039e0 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb86510640 1 --2- 172.21.15.44:0/2490533797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c0039e0 0x7fdb800aeac0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800c300 tx=0x7fdb80160560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:33.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb76ffd640 1 -- 172.21.15.44:0/2490533797 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb78007d30 con 0x7fdb6c0039e0 2024-04-02T13:29:33.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb76ffd640 1 -- 172.21.15.44:0/2490533797 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb78004c10 con 0x7fdb6c0039e0 2024-04-02T13:29:33.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb76ffd640 1 -- 172.21.15.44:0/2490533797 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb78007070 con 0x7fdb6c0039e0 2024-04-02T13:29:33.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb88774640 1 -- 172.21.15.44:0/2490533797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb800ae020 con 0x7fdb6c0039e0 2024-04-02T13:29:33.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.026+0000 7fdb88774640 1 -- 172.21.15.44:0/2490533797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb8015dd10 con 0x7fdb6c0039e0 2024-04-02T13:29:33.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.027+0000 7fdb88774640 1 -- 172.21.15.44:0/2490533797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb800684f0 con 0x7fdb6c0039e0 2024-04-02T13:29:33.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.028+0000 7fdb76ffd640 1 -- 172.21.15.44:0/2490533797 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb78048020 con 0x7fdb6c0039e0 2024-04-02T13:29:33.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.028+0000 7fdb76ffd640 1 --2- 172.21.15.44:0/2490533797 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdb6406c8f0 0x7fdb6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.028+0000 7fdb76ffd640 1 -- 172.21.15.44:0/2490533797 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7fdb780ad130 con 0x7fdb6c0039e0 2024-04-02T13:29:33.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.028+0000 7fdb85d0f640 1 --2- 172.21.15.44:0/2490533797 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdb6406c8f0 0x7fdb6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.029+0000 7fdb85d0f640 1 --2- 172.21.15.44:0/2490533797 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdb6406c8f0 0x7fdb6406edb0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fdb801555c0 tx=0x7fdb70008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:33.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.033+0000 7fdb76ffd640 1 -- 172.21.15.44:0/2490533797 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fdb80155eb0 con 0x7fdb6c0039e0 2024-04-02T13:29:33.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.227+0000 7fdb88774640 1 -- 172.21.15.44:0/2490533797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdb800cbf20 con 0x7fdb6c0039e0 2024-04-02T13:29:33.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.228+0000 7fdb76ffd640 1 -- 172.21.15.44:0/2490533797 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v106) ==== 80+0+0 (secure 0 0 0) 0x7fdb78081020 con 0x7fdb6c0039e0 2024-04-02T13:29:33.232 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:29:33.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.229+0000 7fdb88774640 1 -- 172.21.15.44:0/2490533797 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdb6406c8f0 msgr2=0x7fdb6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.229+0000 7fdb88774640 1 --2- 172.21.15.44:0/2490533797 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdb6406c8f0 0x7fdb6406edb0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fdb801555c0 tx=0x7fdb70008040 comp rx=0 tx=0).stop 2024-04-02T13:29:33.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.230+0000 7fdb88774640 1 -- 172.21.15.44:0/2490533797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c0039e0 msgr2=0x7fdb800aeac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.230+0000 7fdb88774640 1 --2- 172.21.15.44:0/2490533797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c0039e0 0x7fdb800aeac0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800c300 tx=0x7fdb80160560 comp rx=0 tx=0).stop 2024-04-02T13:29:33.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.230+0000 7fdb88774640 1 -- 172.21.15.44:0/2490533797 shutdown_connections 2024-04-02T13:29:33.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.230+0000 7fdb88774640 1 --2- 172.21.15.44:0/2490533797 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdb6406c8f0 0x7fdb6406edb0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.230+0000 7fdb88774640 1 --2- 172.21.15.44:0/2490533797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb800b5940 0x7fdb8009bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.230+0000 7fdb88774640 1 --2- 172.21.15.44:0/2490533797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb800b54e0 0x7fdb8015e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.230+0000 7fdb88774640 1 --2- 172.21.15.44:0/2490533797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c0039e0 0x7fdb800aeac0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.230+0000 7fdb88774640 1 -- 172.21.15.44:0/2490533797 >> 172.21.15.44:0/2490533797 conn(0x7fdb800b1c30 msgr2=0x7fdb800afa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:33.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.230+0000 7fdb88774640 1 -- 172.21.15.44:0/2490533797 shutdown_connections 2024-04-02T13:29:33.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.230+0000 7fdb88774640 1 -- 172.21.15.44:0/2490533797 wait complete. 2024-04-02T13:29:33.244 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:29:33.244 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:29:33.259 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:29:33.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.396+0000 7fda85bba640 1 Processor -- start 2024-04-02T13:29:33.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.396+0000 7fda85bba640 1 -- start start 2024-04-02T13:29:33.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.397+0000 7fda85bba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fda8015c160 0x7fda8006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.397+0000 7fda85bba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda800d0890 0x7fda800cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.397+0000 7fda85bba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda8006bab0 con 0x7fda80160600 2024-04-02T13:29:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.397+0000 7fda85bba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fda8006bd90 con 0x7fda8015c160 2024-04-02T13:29:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.397+0000 7fda85bba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fda800ccec0 con 0x7fda800d0890 2024-04-02T13:29:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.397+0000 7fda7effd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fda80160600 0x7fda800d0180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32878/0 (socket says 172.21.15.44:32878) 2024-04-02T13:29:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.397+0000 7fda7effd640 1 -- 172.21.15.44:0/1461351595 learned_addr learned my addr 172.21.15.44:0/1461351595 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.397+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 577689033 0 0) 0x7fda8006bab0 con 0x7fda80160600 2024-04-02T13:29:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.397+0000 7fda7f7fe640 1 --2- 172.21.15.44:0/1461351595 >> v2:172.21.15.69:3300/0 conn(0x7fda8015c160 0x7fda8006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.397+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda68003520 con 0x7fda80160600 2024-04-02T13:29:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.397+0000 7fda7ffff640 1 --2- 172.21.15.44:0/1461351595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda800d0890 0x7fda800cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3862122019 0 0) 0x7fda68003520 con 0x7fda80160600 2024-04-02T13:29:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda800d0890 msgr2=0x7fda800cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:33.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 --2- 172.21.15.44:0/1461351595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda800d0890 0x7fda800cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 >> v2:172.21.15.69:3300/0 conn(0x7fda8015c160 msgr2=0x7fda8006b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:33.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 --2- 172.21.15.44:0/1461351595 >> v2:172.21.15.69:3300/0 conn(0x7fda8015c160 0x7fda8006b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda800cd1a0 con 0x7fda80160600 2024-04-02T13:29:33.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fda74001670 con 0x7fda80160600 2024-04-02T13:29:33.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 >> v1:172.21.15.44:6789/0 conn(0x7fda80160600 legacy=0x7fda800d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 --2- 172.21.15.44:0/1461351595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda680039b0 0x7fda68003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 --2- 172.21.15.44:0/1461351595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda68004800 0x7fda68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 --2- 172.21.15.44:0/1461351595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda800d0890 0x7fda680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fda8006bd90 con 0x7fda68004800 2024-04-02T13:29:33.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fda800ccec0 con 0x7fda680039b0 2024-04-02T13:29:33.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fda6800b9c0 con 0x7fda800d0890 2024-04-02T13:29:33.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7effd640 1 --2- 172.21.15.44:0/1461351595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda68004800 0x7fda68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7f7fe640 1 --2- 172.21.15.44:0/1461351595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda680039b0 0x7fda68003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7ffff640 1 --2- 172.21.15.44:0/1461351595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda800d0890 0x7fda680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7effd640 1 -- 172.21.15.44:0/1461351595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda800d0890 msgr2=0x7fda680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:33.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7effd640 1 --2- 172.21.15.44:0/1461351595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda800d0890 0x7fda680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7effd640 1 -- 172.21.15.44:0/1461351595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda680039b0 msgr2=0x7fda68003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:33.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7effd640 1 --2- 172.21.15.44:0/1461351595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda680039b0 0x7fda68003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.398+0000 7fda7effd640 1 -- 172.21.15.44:0/1461351595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda68003520 con 0x7fda68004800 2024-04-02T13:29:33.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.399+0000 7fda7effd640 1 --2- 172.21.15.44:0/1461351595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda68004800 0x7fda68004c80 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fda7400c210 tx=0x7fda68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f02abccb4b64a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:33.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.399+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74007d70 con 0x7fda68004800 2024-04-02T13:29:33.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.399+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda74030070 con 0x7fda68004800 2024-04-02T13:29:33.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.399+0000 7fda7e7fc640 1 -- 172.21.15.44:0/1461351595 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7402cd60 con 0x7fda68004800 2024-04-02T13:29:33.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.400+0000 7fda85bba640 1 -- 172.21.15.44:0/1461351595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda68004800 msgr2=0x7fda68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.400+0000 7fda85bba640 1 --2- 172.21.15.44:0/1461351595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda68004800 0x7fda68004c80 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fda7400c210 tx=0x7fda68005830 comp rx=0 tx=0).stop 2024-04-02T13:29:33.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.400+0000 7fda7ffff640 1 -- 172.21.15.44:0/1461351595 reap_dead start 2024-04-02T13:29:33.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.400+0000 7fda85bba640 1 -- 172.21.15.44:0/1461351595 shutdown_connections 2024-04-02T13:29:33.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.400+0000 7fda85bba640 1 -- 172.21.15.44:0/1461351595 >> 172.21.15.44:0/1461351595 conn(0x7fda800b2980 msgr2=0x7fda800b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:33.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.400+0000 7fda85bba640 1 -- 172.21.15.44:0/1461351595 shutdown_connections 2024-04-02T13:29:33.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.400+0000 7fda85bba640 1 -- 172.21.15.44:0/1461351595 wait complete. 2024-04-02T13:29:33.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.401+0000 7fda85bba640 1 Processor -- start 2024-04-02T13:29:33.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.401+0000 7fda85bba640 1 -- start start 2024-04-02T13:29:33.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda85bba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda68004800 0x7fda801f6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda85bba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda8015ef10 0x7fda800c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda85bba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda8015f330 0x7fda800c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda85bba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fda800add60 con 0x7fda8015ef10 2024-04-02T13:29:33.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda85bba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fda80160a60 con 0x7fda68004800 2024-04-02T13:29:33.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda85bba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fda800adee0 con 0x7fda8015f330 2024-04-02T13:29:33.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda7effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda8015ef10 0x7fda800c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda7effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda8015ef10 0x7fda800c6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36226/0 (socket says 172.21.15.44:36226) 2024-04-02T13:29:33.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda7effd640 1 -- 172.21.15.44:0/3105520937 learned_addr learned my addr 172.21.15.44:0/3105520937 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:33.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda7f7fe640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda68004800 0x7fda801f6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda7ffff640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda8015f330 0x7fda800c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda7effd640 1 -- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda8015f330 msgr2=0x7fda800c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda7effd640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda8015f330 0x7fda800c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda7effd640 1 -- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda68004800 msgr2=0x7fda801f6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda7effd640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda68004800 0x7fda801f6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda7effd640 1 -- 172.21.15.44:0/3105520937 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8006f8b0 con 0x7fda8015ef10 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda7f7fe640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda68004800 0x7fda801f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda7ffff640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda8015f330 0x7fda800c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.402+0000 7fda7effd640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda8015ef10 0x7fda800c6220 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fda74001970 tx=0x7fda74000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.403+0000 7fda67fff640 1 -- 172.21.15.44:0/3105520937 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74004bf0 con 0x7fda8015ef10 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.403+0000 7fda67fff640 1 -- 172.21.15.44:0/3105520937 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda74004d90 con 0x7fda8015ef10 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.403+0000 7fda85bba640 1 -- 172.21.15.44:0/3105520937 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8006f2f0 con 0x7fda8015ef10 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.403+0000 7fda67fff640 1 -- 172.21.15.44:0/3105520937 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74007290 con 0x7fda8015ef10 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.403+0000 7fda85bba640 1 -- 172.21.15.44:0/3105520937 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda8015c710 con 0x7fda8015ef10 2024-04-02T13:29:33.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.404+0000 7fda85bba640 1 -- 172.21.15.44:0/3105520937 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda800684f0 con 0x7fda8015ef10 2024-04-02T13:29:33.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.404+0000 7fda67fff640 1 -- 172.21.15.44:0/3105520937 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fda74007430 con 0x7fda8015ef10 2024-04-02T13:29:33.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.404+0000 7fda67fff640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fda6c06c920 0x7fda6c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.404+0000 7fda67fff640 1 -- 172.21.15.44:0/3105520937 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7fda740ace10 con 0x7fda8015ef10 2024-04-02T13:29:33.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.405+0000 7fda7f7fe640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fda6c06c920 0x7fda6c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.405+0000 7fda7f7fe640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fda6c06c920 0x7fda6c06ede0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fda70009bc0 tx=0x7fda70008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:33.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.409+0000 7fda67fff640 1 -- 172.21.15.44:0/3105520937 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fda7407a9b0 con 0x7fda8015ef10 2024-04-02T13:29:33.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.595+0000 7fda85bba640 1 -- 172.21.15.44:0/3105520937 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda800728d0 con 0x7fda8015ef10 2024-04-02T13:29:33.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.596+0000 7fda67fff640 1 -- 172.21.15.44:0/3105520937 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+16836 (secure 0 0 0) 0x7fda7407ecb0 con 0x7fda8015ef10 2024-04-02T13:29:33.600 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:33.600 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":106,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:29:12.549182+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:29:09.192430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-02T13:29:09.860748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:29:33.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.597+0000 7fda85bba640 1 -- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fda6c06c920 msgr2=0x7fda6c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.597+0000 7fda85bba640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fda6c06c920 0x7fda6c06ede0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fda70009bc0 tx=0x7fda70008040 comp rx=0 tx=0).stop 2024-04-02T13:29:33.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.597+0000 7fda85bba640 1 -- 172.21.15.44:0/3105520937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda8015ef10 msgr2=0x7fda800c6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.597+0000 7fda85bba640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda8015ef10 0x7fda800c6220 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fda74001970 tx=0x7fda74000fe0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.598+0000 7fda85bba640 1 -- 172.21.15.44:0/3105520937 shutdown_connections 2024-04-02T13:29:33.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.598+0000 7fda85bba640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fda6c06c920 0x7fda6c06ede0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.598+0000 7fda85bba640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda8015f330 0x7fda800c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.598+0000 7fda85bba640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda8015ef10 0x7fda800c6220 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.598+0000 7fda85bba640 1 --2- 172.21.15.44:0/3105520937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda68004800 0x7fda801f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.598+0000 7fda85bba640 1 -- 172.21.15.44:0/3105520937 >> 172.21.15.44:0/3105520937 conn(0x7fda800b2980 msgr2=0x7fda800ae400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:33.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.598+0000 7fda85bba640 1 -- 172.21.15.44:0/3105520937 shutdown_connections 2024-04-02T13:29:33.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.598+0000 7fda85bba640 1 -- 172.21.15.44:0/3105520937 wait complete. 2024-04-02T13:29:33.613 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:29:09.192430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:29:09.860748+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:29:33.613 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:29:33.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.715+0000 7f6628c2c640 1 Processor -- start 2024-04-02T13:29:33.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.715+0000 7f6628c2c640 1 -- start start 2024-04-02T13:29:33.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f6628c2c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f662415e420 0x7f66240ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f6628c2c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66240cfe10 0x7f66240cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f6628c2c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f662406c220 con 0x7f66240721f0 2024-04-02T13:29:33.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f6628c2c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f662415f7e0 con 0x7f662415e420 2024-04-02T13:29:33.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f6628c2c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f662415ec60 con 0x7f66240cfe10 2024-04-02T13:29:33.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f662259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f662415e420 0x7f66240ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f6621d9b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f66240721f0 0x7f66240cf700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32894/0 (socket says 172.21.15.44:32894) 2024-04-02T13:29:33.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f6621d9b640 1 -- 172.21.15.44:0/916078956 learned_addr learned my addr 172.21.15.44:0/916078956 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:33.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f662259c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f662415e420 0x7f66240ceca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36230/0 (socket says 172.21.15.44:36230) 2024-04-02T13:29:33.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f6622d9d640 1 --2- 172.21.15.44:0/916078956 >> v2:172.21.15.69:3300/0 conn(0x7f66240cfe10 0x7f66240cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f662259c640 1 -- 172.21.15.44:0/916078956 >> v2:172.21.15.69:3300/0 conn(0x7f66240cfe10 msgr2=0x7f66240cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:33.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f662259c640 1 --2- 172.21.15.44:0/916078956 >> v2:172.21.15.69:3300/0 conn(0x7f66240cfe10 0x7f66240cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f662259c640 1 -- 172.21.15.44:0/916078956 >> v1:172.21.15.44:6789/0 conn(0x7f66240721f0 legacy=0x7f66240cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f662259c640 1 -- 172.21.15.44:0/916078956 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662415ef40 con 0x7f662415e420 2024-04-02T13:29:33.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.716+0000 7f662159a640 1 -- 172.21.15.44:0/916078956 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3698569991 0 0) 0x7f6618001110 con 0x7f66240721f0 2024-04-02T13:29:33.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.717+0000 7f662259c640 1 --2- 172.21.15.44:0/916078956 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662415e420 0x7f66240ceca0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f66240d0bf0 tx=0x7f661402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96967a9a883ee62b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:33.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.717+0000 7f662159a640 1 -- 172.21.15.44:0/916078956 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6614038070 con 0x7f662415e420 2024-04-02T13:29:33.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.717+0000 7f662159a640 1 -- 172.21.15.44:0/916078956 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6614031710 con 0x7f662415e420 2024-04-02T13:29:33.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.717+0000 7f662159a640 1 -- 172.21.15.44:0/916078956 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6614031ae0 con 0x7f662415e420 2024-04-02T13:29:33.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.718+0000 7f6628c2c640 1 -- 172.21.15.44:0/916078956 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662415e420 msgr2=0x7f66240ceca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.718+0000 7f6628c2c640 1 --2- 172.21.15.44:0/916078956 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662415e420 0x7f66240ceca0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f66240d0bf0 tx=0x7f661402b7b0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.718+0000 7f6628c2c640 1 -- 172.21.15.44:0/916078956 shutdown_connections 2024-04-02T13:29:33.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.718+0000 7f6628c2c640 1 --2- 172.21.15.44:0/916078956 >> v2:172.21.15.69:3300/0 conn(0x7f66240cfe10 0x7f66240cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.718+0000 7f6628c2c640 1 --2- 172.21.15.44:0/916078956 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662415e420 0x7f66240ceca0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.718+0000 7f6628c2c640 1 -- 172.21.15.44:0/916078956 >> 172.21.15.44:0/916078956 conn(0x7f66240b2a10 msgr2=0x7f66240b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:33.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.718+0000 7f6628c2c640 1 -- 172.21.15.44:0/916078956 shutdown_connections 2024-04-02T13:29:33.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.718+0000 7f6628c2c640 1 -- 172.21.15.44:0/916078956 wait complete. 2024-04-02T13:29:33.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.719+0000 7f6628c2c640 1 Processor -- start 2024-04-02T13:29:33.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.719+0000 7f6628c2c640 1 -- start start 2024-04-02T13:29:33.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6628c2c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66240721f0 0x7f66241f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6628c2c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66240cfe10 0x7f66241f7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6628c2c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662415e420 0x7f66240b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6628c2c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f66240cf1e0 con 0x7f662415e420 2024-04-02T13:29:33.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6628c2c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f662415f500 con 0x7f66240cfe10 2024-04-02T13:29:33.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6628c2c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f66240cf360 con 0x7f66240721f0 2024-04-02T13:29:33.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6622d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662415e420 0x7f66240b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6622d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662415e420 0x7f66240b4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36238/0 (socket says 172.21.15.44:36238) 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6622d9d640 1 -- 172.21.15.44:0/270572548 learned_addr learned my addr 172.21.15.44:0/270572548 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6621d9b640 1 --2- 172.21.15.44:0/270572548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66240cfe10 0x7f66241f7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6622d9d640 1 -- 172.21.15.44:0/270572548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66240721f0 msgr2=0x7f66241f56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6622d9d640 1 --2- 172.21.15.44:0/270572548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66240721f0 0x7f66241f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6622d9d640 1 -- 172.21.15.44:0/270572548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66240cfe10 msgr2=0x7f66241f7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6622d9d640 1 --2- 172.21.15.44:0/270572548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66240cfe10 0x7f66241f7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6622d9d640 1 -- 172.21.15.44:0/270572548 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66241535b0 con 0x7f662415e420 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6622d9d640 1 --2- 172.21.15.44:0/270572548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662415e420 0x7f66240b4b10 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f661c007e40 tx=0x7f661c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6612ffd640 1 -- 172.21.15.44:0/270572548 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661c014620 con 0x7f662415e420 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6612ffd640 1 -- 172.21.15.44:0/270572548 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f661c019070 con 0x7f662415e420 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6628c2c640 1 -- 172.21.15.44:0/270572548 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6624153850 con 0x7f662415e420 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.720+0000 7f6612ffd640 1 -- 172.21.15.44:0/270572548 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661c014a20 con 0x7f662415e420 2024-04-02T13:29:33.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.721+0000 7f6628c2c640 1 -- 172.21.15.44:0/270572548 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66241540f0 con 0x7f662415e420 2024-04-02T13:29:33.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.721+0000 7f6628c2c640 1 -- 172.21.15.44:0/270572548 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6624153c40 con 0x7f662415e420 2024-04-02T13:29:33.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.722+0000 7f6612ffd640 1 -- 172.21.15.44:0/270572548 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f661c014bc0 con 0x7f662415e420 2024-04-02T13:29:33.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.722+0000 7f6612ffd640 1 --2- 172.21.15.44:0/270572548 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f660406c8f0 0x7f660406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:33.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.722+0000 7f6612ffd640 1 -- 172.21.15.44:0/270572548 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f661c08c560 con 0x7f662415e420 2024-04-02T13:29:33.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.723+0000 7f662259c640 1 --2- 172.21.15.44:0/270572548 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f660406c8f0 0x7f660406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:33.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.723+0000 7f662259c640 1 --2- 172.21.15.44:0/270572548 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f660406c8f0 0x7f660406edb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6614035000 tx=0x7f6614030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:33.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.727+0000 7f6612ffd640 1 -- 172.21.15.44:0/270572548 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f661c05a890 con 0x7f662415e420 2024-04-02T13:29:33.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.908+0000 7f6628c2c640 1 -- 172.21.15.44:0/270572548 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6624153a20 con 0x7f662415e420 2024-04-02T13:29:33.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.909+0000 7f6612ffd640 1 -- 172.21.15.44:0/270572548 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7f661c05a190 con 0x7f662415e420 2024-04-02T13:29:33.913 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:29:33.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.910+0000 7f6628c2c640 1 -- 172.21.15.44:0/270572548 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f660406c8f0 msgr2=0x7f660406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.910+0000 7f6628c2c640 1 --2- 172.21.15.44:0/270572548 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f660406c8f0 0x7f660406edb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6614035000 tx=0x7f6614030000 comp rx=0 tx=0).stop 2024-04-02T13:29:33.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.910+0000 7f6628c2c640 1 -- 172.21.15.44:0/270572548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662415e420 msgr2=0x7f66240b4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:33.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.910+0000 7f6628c2c640 1 --2- 172.21.15.44:0/270572548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662415e420 0x7f66240b4b10 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f661c007e40 tx=0x7f661c00e4b0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.910+0000 7f6628c2c640 1 -- 172.21.15.44:0/270572548 shutdown_connections 2024-04-02T13:29:33.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.910+0000 7f6628c2c640 1 --2- 172.21.15.44:0/270572548 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f660406c8f0 0x7f660406edb0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.910+0000 7f6628c2c640 1 --2- 172.21.15.44:0/270572548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662415e420 0x7f66240b4b10 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.910+0000 7f6628c2c640 1 --2- 172.21.15.44:0/270572548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66240cfe10 0x7f66241f7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.910+0000 7f6628c2c640 1 --2- 172.21.15.44:0/270572548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66240721f0 0x7f66241f56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:33.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.910+0000 7f6628c2c640 1 -- 172.21.15.44:0/270572548 >> 172.21.15.44:0/270572548 conn(0x7f66240b2a10 msgr2=0x7f66240ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:33.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.911+0000 7f6628c2c640 1 -- 172.21.15.44:0/270572548 shutdown_connections 2024-04-02T13:29:33.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:33.911+0000 7f6628c2c640 1 -- 172.21.15.44:0/270572548 wait complete. 2024-04-02T13:29:33.925 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:29:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.026+0000 7f931a44a640 1 Processor -- start 2024-04-02T13:29:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.026+0000 7f931a44a640 1 -- start start 2024-04-02T13:29:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.027+0000 7f931a44a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f931415e1d0 0x7f93140d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.027+0000 7f931a44a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f93140cec60 0x7f93140d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.027+0000 7f931a44a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93140d1640 con 0x7f931415e5b0 2024-04-02T13:29:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.027+0000 7f931a44a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f93140cfe00 con 0x7f93140cec60 2024-04-02T13:29:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.027+0000 7f931a44a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f93140cf1f0 con 0x7f931415e1d0 2024-04-02T13:29:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.027+0000 7f93137fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f93140cec60 0x7f93140d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.027+0000 7f93137fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f93140cec60 0x7f93140d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36254/0 (socket says 172.21.15.44:36254) 2024-04-02T13:29:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.027+0000 7f93137fe640 1 -- 172.21.15.44:0/4287250333 learned_addr learned my addr 172.21.15.44:0/4287250333 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.028+0000 7f9313fff640 1 --2- 172.21.15.44:0/4287250333 >> v2:172.21.15.69:3300/0 conn(0x7f931415e1d0 0x7f93140d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.028+0000 7f93137fe640 1 -- 172.21.15.44:0/4287250333 >> v2:172.21.15.69:3300/0 conn(0x7f931415e1d0 msgr2=0x7f93140d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.028+0000 7f93137fe640 1 --2- 172.21.15.44:0/4287250333 >> v2:172.21.15.69:3300/0 conn(0x7f931415e1d0 0x7f93140d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.028+0000 7f93137fe640 1 -- 172.21.15.44:0/4287250333 >> v1:172.21.15.69:6789/0 conn(0x7f931415e5b0 legacy=0x7f93140b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.028+0000 7f93137fe640 1 -- 172.21.15.44:0/4287250333 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93140cf4d0 con 0x7f93140cec60 2024-04-02T13:29:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.028+0000 7f93137fe640 1 --2- 172.21.15.44:0/4287250333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93140cec60 0x7f93140d26c0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f931408c9c0 tx=0x7f930402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f94548d08634a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.029+0000 7f9312ffd640 1 -- 172.21.15.44:0/4287250333 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930402e070 con 0x7f93140cec60 2024-04-02T13:29:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.029+0000 7f9312ffd640 1 -- 172.21.15.44:0/4287250333 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9304037070 con 0x7f93140cec60 2024-04-02T13:29:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.029+0000 7f9312ffd640 1 -- 172.21.15.44:0/4287250333 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9304031a00 con 0x7f93140cec60 2024-04-02T13:29:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.029+0000 7f931a44a640 1 -- 172.21.15.44:0/4287250333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93140cec60 msgr2=0x7f93140d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.029+0000 7f931a44a640 1 --2- 172.21.15.44:0/4287250333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93140cec60 0x7f93140d26c0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f931408c9c0 tx=0x7f930402b880 comp rx=0 tx=0).stop 2024-04-02T13:29:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.029+0000 7f931a44a640 1 -- 172.21.15.44:0/4287250333 shutdown_connections 2024-04-02T13:29:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.029+0000 7f931a44a640 1 --2- 172.21.15.44:0/4287250333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93140cec60 0x7f93140d26c0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.029+0000 7f931a44a640 1 --2- 172.21.15.44:0/4287250333 >> v2:172.21.15.69:3300/0 conn(0x7f931415e1d0 0x7f93140d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.029+0000 7f931a44a640 1 -- 172.21.15.44:0/4287250333 >> 172.21.15.44:0/4287250333 conn(0x7f931409d3b0 msgr2=0x7f93140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:34.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.029+0000 7f931a44a640 1 -- 172.21.15.44:0/4287250333 shutdown_connections 2024-04-02T13:29:34.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.029+0000 7f931a44a640 1 -- 172.21.15.44:0/4287250333 wait complete. 2024-04-02T13:29:34.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.030+0000 7f931a44a640 1 Processor -- start 2024-04-02T13:29:34.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.031+0000 7f931a44a640 1 -- start start 2024-04-02T13:29:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.031+0000 7f931a44a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93140cec60 0x7f931415fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.031+0000 7f931a44a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f931415e1d0 0x7f931406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.031+0000 7f931a44a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f931415e5b0 0x7f931409e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.031+0000 7f931a44a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f93140d2c00 con 0x7f931415e5b0 2024-04-02T13:29:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.031+0000 7f931a44a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f93140cfa90 con 0x7f93140cec60 2024-04-02T13:29:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.031+0000 7f931a44a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f93140d2d80 con 0x7f931415e1d0 2024-04-02T13:29:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.031+0000 7f93189e7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f931415e5b0 0x7f931409e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.031+0000 7f93189e7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f931415e5b0 0x7f931409e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36270/0 (socket says 172.21.15.44:36270) 2024-04-02T13:29:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.031+0000 7f93189e7640 1 -- 172.21.15.44:0/4145529351 learned_addr learned my addr 172.21.15.44:0/4145529351 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.031+0000 7f9313fff640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93140cec60 0x7f931415fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.032+0000 7f93137fe640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f931415e1d0 0x7f931406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.032+0000 7f93189e7640 1 -- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f931415e1d0 msgr2=0x7f931406d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.032+0000 7f93189e7640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f931415e1d0 0x7f931406d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.032+0000 7f93189e7640 1 -- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93140cec60 msgr2=0x7f931415fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.032+0000 7f93189e7640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93140cec60 0x7f931415fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.032+0000 7f93189e7640 1 -- 172.21.15.44:0/4145529351 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9314160910 con 0x7f931415e5b0 2024-04-02T13:29:34.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.032+0000 7f93137fe640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f931415e1d0 0x7f931406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:34.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.032+0000 7f93189e7640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f931415e5b0 0x7f931409e5c0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f930c00b970 tx=0x7f930c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:34.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.032+0000 7f9310ff9640 1 -- 172.21.15.44:0/4145529351 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c019070 con 0x7f931415e5b0 2024-04-02T13:29:34.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.032+0000 7f9310ff9640 1 -- 172.21.15.44:0/4145529351 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f930c00cb40 con 0x7f931415e5b0 2024-04-02T13:29:34.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.032+0000 7f931a44a640 1 -- 172.21.15.44:0/4145529351 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9314160350 con 0x7f931415e5b0 2024-04-02T13:29:34.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.032+0000 7f9310ff9640 1 -- 172.21.15.44:0/4145529351 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c014440 con 0x7f931415e5b0 2024-04-02T13:29:34.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.032+0000 7f931a44a640 1 -- 172.21.15.44:0/4145529351 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f931409ddc0 con 0x7f931415e5b0 2024-04-02T13:29:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.033+0000 7f931a44a640 1 -- 172.21.15.44:0/4145529351 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93140684f0 con 0x7f931415e5b0 2024-04-02T13:29:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.033+0000 7f9310ff9640 1 -- 172.21.15.44:0/4145529351 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f930c01c070 con 0x7f931415e5b0 2024-04-02T13:29:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.034+0000 7f9310ff9640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f930006c8f0 0x7f930006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.034+0000 7f9310ff9640 1 -- 172.21.15.44:0/4145529351 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f930c08b9a0 con 0x7f931415e5b0 2024-04-02T13:29:34.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.034+0000 7f9313fff640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f930006c8f0 0x7f930006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.035+0000 7f9313fff640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f930006c8f0 0x7f930006edb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f930800aac0 tx=0x7f9308008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:34.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.038+0000 7f9310ff9640 1 -- 172.21.15.44:0/4145529351 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f930c0595f0 con 0x7f931415e5b0 2024-04-02T13:29:34.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.223+0000 7f931a44a640 1 -- 172.21.15.44:0/4145529351 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f93140d47f0 con 0x7f931415e5b0 2024-04-02T13:29:34.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.223+0000 7f9310ff9640 1 -- 172.21.15.44:0/4145529351 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) ==== 104+0+11 (secure 0 0 0) 0x7f930c05d8f0 con 0x7f931415e5b0 2024-04-02T13:29:34.227 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:29:34.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.225+0000 7f931a44a640 1 -- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f930006c8f0 msgr2=0x7f930006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.225+0000 7f931a44a640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f930006c8f0 0x7f930006edb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f930800aac0 tx=0x7f9308008040 comp rx=0 tx=0).stop 2024-04-02T13:29:34.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.225+0000 7f931a44a640 1 -- 172.21.15.44:0/4145529351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f931415e5b0 msgr2=0x7f931409e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.225+0000 7f931a44a640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f931415e5b0 0x7f931409e5c0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f930c00b970 tx=0x7f930c00be30 comp rx=0 tx=0).stop 2024-04-02T13:29:34.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.225+0000 7f931a44a640 1 -- 172.21.15.44:0/4145529351 shutdown_connections 2024-04-02T13:29:34.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.225+0000 7f931a44a640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f930006c8f0 0x7f930006edb0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.225+0000 7f931a44a640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f931415e5b0 0x7f931409e5c0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.225+0000 7f931a44a640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f931415e1d0 0x7f931406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.225+0000 7f931a44a640 1 --2- 172.21.15.44:0/4145529351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93140cec60 0x7f931415fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.225+0000 7f931a44a640 1 -- 172.21.15.44:0/4145529351 >> 172.21.15.44:0/4145529351 conn(0x7f931409d3b0 msgr2=0x7f93140cd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:34.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.225+0000 7f931a44a640 1 -- 172.21.15.44:0/4145529351 shutdown_connections 2024-04-02T13:29:34.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.225+0000 7f931a44a640 1 -- 172.21.15.44:0/4145529351 wait complete. 2024-04-02T13:29:34.239 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:29:34.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.340+0000 7fee31d3d640 1 Processor -- start 2024-04-02T13:29:34.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.340+0000 7fee31d3d640 1 -- start start 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.341+0000 7fee31d3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c15e640 0x7fee2c0d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.341+0000 7fee31d3d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fee2c0cec60 0x7fee2c0d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.341+0000 7fee31d3d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2c0cf120 con 0x7fee2c15e260 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.341+0000 7fee31d3d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fee2c0b5a50 con 0x7fee2c0cec60 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.341+0000 7fee31d3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fee2c0d01c0 con 0x7fee2c15e640 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.341+0000 7fee2b7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fee2c15e260 0x7fee2c0d2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32906/0 (socket says 172.21.15.44:32906) 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.341+0000 7fee2b7fe640 1 -- 172.21.15.44:0/3230981926 learned_addr learned my addr 172.21.15.44:0/3230981926 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.341+0000 7fee2bfff640 1 --2- 172.21.15.44:0/3230981926 >> v2:172.21.15.69:3300/0 conn(0x7fee2c0cec60 0x7fee2c0d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.341+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2424110533 0 0) 0x7fee2c0cf120 con 0x7fee2c15e260 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.341+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee14003520 con 0x7fee2c15e260 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.341+0000 7fee2affd640 1 --2- 172.21.15.44:0/3230981926 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c15e640 0x7fee2c0d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 689879580 0 0) 0x7fee14003520 con 0x7fee2c15e260 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c15e640 msgr2=0x7fee2c0d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:34.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 --2- 172.21.15.44:0/3230981926 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c15e640 0x7fee2c0d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 >> v2:172.21.15.69:3300/0 conn(0x7fee2c0cec60 msgr2=0x7fee2c0d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 --2- 172.21.15.44:0/3230981926 >> v2:172.21.15.69:3300/0 conn(0x7fee2c0cec60 0x7fee2c0d1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c0cf400 con 0x7fee2c15e260 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fee1c001670 con 0x7fee2c15e260 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2affd640 1 --2- 172.21.15.44:0/3230981926 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c15e640 0x7fee2c0d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 >> v1:172.21.15.44:6789/0 conn(0x7fee2c15e260 legacy=0x7fee2c0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 --2- 172.21.15.44:0/3230981926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee140039b0 0x7fee14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 --2- 172.21.15.44:0/3230981926 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c15e640 0x7fee14004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 --2- 172.21.15.44:0/3230981926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee14004d40 0x7fee140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c0b5a50 con 0x7fee14004d40 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fee1400b9c0 con 0x7fee140039b0 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fee1400bb40 con 0x7fee2c15e640 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2bfff640 1 --2- 172.21.15.44:0/3230981926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee14004d40 0x7fee140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2affd640 1 --2- 172.21.15.44:0/3230981926 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c15e640 0x7fee14004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2b7fe640 1 --2- 172.21.15.44:0/3230981926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee140039b0 0x7fee14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2bfff640 1 -- 172.21.15.44:0/3230981926 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c15e640 msgr2=0x7fee14004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:34.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2bfff640 1 --2- 172.21.15.44:0/3230981926 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c15e640 0x7fee14004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2bfff640 1 -- 172.21.15.44:0/3230981926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee140039b0 msgr2=0x7fee14003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:34.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2bfff640 1 --2- 172.21.15.44:0/3230981926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee140039b0 0x7fee14003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.342+0000 7fee2bfff640 1 -- 172.21.15.44:0/3230981926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee14003520 con 0x7fee14004d40 2024-04-02T13:29:34.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.343+0000 7fee2bfff640 1 --2- 172.21.15.44:0/3230981926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee14004d40 0x7fee140051c0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fee2c0ce9d0 tx=0x7fee14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447852455ce2e72 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:34.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.343+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee24039070 con 0x7fee14004d40 2024-04-02T13:29:34.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.343+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee2402cbd0 con 0x7fee14004d40 2024-04-02T13:29:34.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.343+0000 7fee2a7fc640 1 -- 172.21.15.44:0/3230981926 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee24009c10 con 0x7fee14004d40 2024-04-02T13:29:34.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.344+0000 7fee31d3d640 1 -- 172.21.15.44:0/3230981926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee14004d40 msgr2=0x7fee140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.344+0000 7fee31d3d640 1 --2- 172.21.15.44:0/3230981926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee14004d40 0x7fee140051c0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fee2c0ce9d0 tx=0x7fee14005830 comp rx=0 tx=0).stop 2024-04-02T13:29:34.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.344+0000 7fee2bfff640 1 -- 172.21.15.44:0/3230981926 reap_dead start 2024-04-02T13:29:34.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.344+0000 7fee31d3d640 1 -- 172.21.15.44:0/3230981926 shutdown_connections 2024-04-02T13:29:34.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.344+0000 7fee31d3d640 1 -- 172.21.15.44:0/3230981926 >> 172.21.15.44:0/3230981926 conn(0x7fee2c09d3c0 msgr2=0x7fee2c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:34.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.344+0000 7fee31d3d640 1 -- 172.21.15.44:0/3230981926 shutdown_connections 2024-04-02T13:29:34.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.344+0000 7fee31d3d640 1 -- 172.21.15.44:0/3230981926 wait complete. 2024-04-02T13:29:34.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.345+0000 7fee31d3d640 1 Processor -- start 2024-04-02T13:29:34.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.345+0000 7fee31d3d640 1 -- start start 2024-04-02T13:29:34.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee31d3d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee14004d40 0x7fee2c14b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee31d3d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2c15cad0 0x7fee2c0b7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee31d3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c0cd970 0x7fee2c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee31d3d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c0cfca0 con 0x7fee2c15cad0 2024-04-02T13:29:34.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee31d3d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c0cf9c0 con 0x7fee14004d40 2024-04-02T13:29:34.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee31d3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fee2c0d3070 con 0x7fee2c0cd970 2024-04-02T13:29:34.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee2affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2c15cad0 0x7fee2c0b7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee2affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2c15cad0 0x7fee2c0b7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36280/0 (socket says 172.21.15.44:36280) 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee2affd640 1 -- 172.21.15.44:0/4154829041 learned_addr learned my addr 172.21.15.44:0/4154829041 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee2b7fe640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee14004d40 0x7fee2c14b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee2bfff640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c0cd970 0x7fee2c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee2affd640 1 -- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c0cd970 msgr2=0x7fee2c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee2affd640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c0cd970 0x7fee2c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee2affd640 1 -- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee14004d40 msgr2=0x7fee2c14b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee2affd640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee14004d40 0x7fee2c14b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee2affd640 1 -- 172.21.15.44:0/4154829041 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c14b5e0 con 0x7fee2c15cad0 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee2affd640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2c15cad0 0x7fee2c0b7b20 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fee200115e0 tx=0x7fee200119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee13fff640 1 -- 172.21.15.44:0/4154829041 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2000f040 con 0x7fee2c15cad0 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.346+0000 7fee13fff640 1 -- 172.21.15.44:0/4154829041 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee20004230 con 0x7fee2c15cad0 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.347+0000 7fee31d3d640 1 -- 172.21.15.44:0/4154829041 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee2c0b84c0 con 0x7fee2c15cad0 2024-04-02T13:29:34.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.347+0000 7fee13fff640 1 -- 172.21.15.44:0/4154829041 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20015670 con 0x7fee2c15cad0 2024-04-02T13:29:34.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.347+0000 7fee31d3d640 1 -- 172.21.15.44:0/4154829041 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee2c0b8a80 con 0x7fee2c15cad0 2024-04-02T13:29:34.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.348+0000 7fee31d3d640 1 -- 172.21.15.44:0/4154829041 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee2c0684f0 con 0x7fee2c15cad0 2024-04-02T13:29:34.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.348+0000 7fee13fff640 1 -- 172.21.15.44:0/4154829041 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fee20015810 con 0x7fee2c15cad0 2024-04-02T13:29:34.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.348+0000 7fee13fff640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fee1806c8f0 0x7fee1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.348+0000 7fee13fff640 1 -- 172.21.15.44:0/4154829041 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7fee2008c080 con 0x7fee2c15cad0 2024-04-02T13:29:34.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.348+0000 7fee2b7fe640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fee1806c8f0 0x7fee1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.349+0000 7fee2b7fe640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fee1806c8f0 0x7fee1806edb0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fee1c007d90 tx=0x7fee1c0078f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:34.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.353+0000 7fee13fff640 1 -- 172.21.15.44:0/4154829041 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fee20019020 con 0x7fee2c15cad0 2024-04-02T13:29:34.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.535+0000 7fee31d3d640 1 -- 172.21.15.44:0/4154829041 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fee2c0d47f0 con 0x7fee2c15cad0 2024-04-02T13:29:34.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.536+0000 7fee13fff640 1 -- 172.21.15.44:0/4154829041 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) ==== 100+0+11 (secure 0 0 0) 0x7fee2005a320 con 0x7fee2c15cad0 2024-04-02T13:29:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:29:34.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.537+0000 7fee31d3d640 1 -- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fee1806c8f0 msgr2=0x7fee1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.537+0000 7fee31d3d640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fee1806c8f0 0x7fee1806edb0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fee1c007d90 tx=0x7fee1c0078f0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.538+0000 7fee31d3d640 1 -- 172.21.15.44:0/4154829041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2c15cad0 msgr2=0x7fee2c0b7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.538+0000 7fee31d3d640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2c15cad0 0x7fee2c0b7b20 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fee200115e0 tx=0x7fee200119e0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.538+0000 7fee31d3d640 1 -- 172.21.15.44:0/4154829041 shutdown_connections 2024-04-02T13:29:34.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.538+0000 7fee31d3d640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fee1806c8f0 0x7fee1806edb0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.538+0000 7fee31d3d640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2c0cd970 0x7fee2c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.538+0000 7fee31d3d640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2c15cad0 0x7fee2c0b7b20 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.538+0000 7fee31d3d640 1 --2- 172.21.15.44:0/4154829041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee14004d40 0x7fee2c14b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.538+0000 7fee31d3d640 1 -- 172.21.15.44:0/4154829041 >> 172.21.15.44:0/4154829041 conn(0x7fee2c09d3c0 msgr2=0x7fee2c06d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:34.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.538+0000 7fee31d3d640 1 -- 172.21.15.44:0/4154829041 shutdown_connections 2024-04-02T13:29:34.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.538+0000 7fee31d3d640 1 -- 172.21.15.44:0/4154829041 wait complete. 2024-04-02T13:29:34.552 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:29:34.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.657+0000 7f0c68cf2640 1 Processor -- start 2024-04-02T13:29:34.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.657+0000 7f0c68cf2640 1 -- start start 2024-04-02T13:29:34.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.657+0000 7f0c68cf2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6415e140 0x7f0c640d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.657+0000 7f0c68cf2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c640b70b0 con 0x7f0c6415e520 2024-04-02T13:29:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.657+0000 7f0c68cf2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c640b7390 con 0x7f0c640d1150 2024-04-02T13:29:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.657+0000 7f0c68cf2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0c640b9f50 con 0x7f0c6415e140 2024-04-02T13:29:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c61d9b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0c6415e520 0x7f0c640d0a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56036/0 (socket says 172.21.15.44:56036) 2024-04-02T13:29:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c61d9b640 1 -- 172.21.15.44:0/2720022082 learned_addr learned my addr 172.21.15.44:0/2720022082 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6259c640 1 --2- 172.21.15.44:0/2720022082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6415e140 0x7f0c640d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2168524554 0 0) 0x7f0c640b70b0 con 0x7f0c6415e520 2024-04-02T13:29:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c50003520 con 0x7f0c6415e520 2024-04-02T13:29:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2000708372 0 0) 0x7f0c50003520 con 0x7f0c6415e520 2024-04-02T13:29:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6415e140 msgr2=0x7f0c640d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 --2- 172.21.15.44:0/2720022082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6415e140 0x7f0c640d2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 >> v1:172.21.15.69:6789/0 conn(0x7f0c640d1150 legacy=0x7f0c640b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c640b7670 con 0x7f0c6415e520 2024-04-02T13:29:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f0c4c001670 con 0x7f0c6415e520 2024-04-02T13:29:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 >> v1:172.21.15.44:6789/0 conn(0x7f0c6415e520 legacy=0x7f0c640d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 --2- 172.21.15.44:0/2720022082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500039e0 0x7f0c50003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 --2- 172.21.15.44:0/2720022082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c50004800 0x7f0c50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 --2- 172.21.15.44:0/2720022082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6415e140 0x7f0c500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c640b9f50 con 0x7f0c500039e0 2024-04-02T13:29:34.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5000ba70 con 0x7f0c50004800 2024-04-02T13:29:34.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0c5000bc70 con 0x7f0c6415e140 2024-04-02T13:29:34.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c6259c640 1 --2- 172.21.15.44:0/2720022082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500039e0 0x7f0c50003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c61d9b640 1 --2- 172.21.15.44:0/2720022082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c50004800 0x7f0c50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.658+0000 7f0c62d9d640 1 --2- 172.21.15.44:0/2720022082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6415e140 0x7f0c500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.659+0000 7f0c6259c640 1 -- 172.21.15.44:0/2720022082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6415e140 msgr2=0x7f0c500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:34.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.659+0000 7f0c6259c640 1 --2- 172.21.15.44:0/2720022082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6415e140 0x7f0c500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.659+0000 7f0c6259c640 1 -- 172.21.15.44:0/2720022082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c50004800 msgr2=0x7f0c50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:34.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.659+0000 7f0c6259c640 1 --2- 172.21.15.44:0/2720022082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c50004800 0x7f0c50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.659+0000 7f0c6259c640 1 -- 172.21.15.44:0/2720022082 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c50003520 con 0x7f0c500039e0 2024-04-02T13:29:34.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.659+0000 7f0c62d9d640 1 --2- 172.21.15.44:0/2720022082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6415e140 0x7f0c500051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:34.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.659+0000 7f0c6259c640 1 --2- 172.21.15.44:0/2720022082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500039e0 0x7f0c50003e00 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009a90 tx=0x7f0c50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc7dc7a559ac9f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:34.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.660+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c039070 con 0x7f0c500039e0 2024-04-02T13:29:34.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.660+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c5c02c680 con 0x7f0c500039e0 2024-04-02T13:29:34.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.660+0000 7f0c6159a640 1 -- 172.21.15.44:0/2720022082 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c009690 con 0x7f0c500039e0 2024-04-02T13:29:34.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.660+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2720022082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500039e0 msgr2=0x7f0c50003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.660+0000 7f0c68cf2640 1 --2- 172.21.15.44:0/2720022082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500039e0 0x7f0c50003e00 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009a90 tx=0x7f0c50005830 comp rx=0 tx=0).stop 2024-04-02T13:29:34.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.660+0000 7f0c62d9d640 1 -- 172.21.15.44:0/2720022082 reap_dead start 2024-04-02T13:29:34.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.660+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2720022082 shutdown_connections 2024-04-02T13:29:34.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.660+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2720022082 >> 172.21.15.44:0/2720022082 conn(0x7f0c6409d390 msgr2=0x7f0c640d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:34.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.660+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2720022082 shutdown_connections 2024-04-02T13:29:34.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.660+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2720022082 wait complete. 2024-04-02T13:29:34.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.661+0000 7f0c68cf2640 1 Processor -- start 2024-04-02T13:29:34.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.661+0000 7f0c68cf2640 1 -- start start 2024-04-02T13:29:34.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.662+0000 7f0c68cf2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c500039e0 0x7f0c64151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.662+0000 7f0c68cf2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c640cb120 0x7f0c640a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.662+0000 7f0c68cf2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c640cc650 0x7f0c640cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.662+0000 7f0c68cf2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c640b8e60 con 0x7f0c640cb120 2024-04-02T13:29:34.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.662+0000 7f0c68cf2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0c640d4ee0 con 0x7f0c640cc650 2024-04-02T13:29:34.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.662+0000 7f0c68cf2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0c640b8fe0 con 0x7f0c500039e0 2024-04-02T13:29:34.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.662+0000 7f0c61d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c640cb120 0x7f0c640a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.662+0000 7f0c61d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c640cb120 0x7f0c640a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40158/0 (socket says 172.21.15.44:40158) 2024-04-02T13:29:34.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.662+0000 7f0c61d9b640 1 -- 172.21.15.44:0/2146548307 learned_addr learned my addr 172.21.15.44:0/2146548307 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:34.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.662+0000 7f0c62d9d640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c640cc650 0x7f0c640cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.662+0000 7f0c6259c640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c500039e0 0x7f0c64151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.663+0000 7f0c61d9b640 1 -- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c500039e0 msgr2=0x7f0c64151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.663+0000 7f0c61d9b640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c500039e0 0x7f0c64151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.663+0000 7f0c61d9b640 1 -- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c640cc650 msgr2=0x7f0c640cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.663+0000 7f0c61d9b640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c640cc650 0x7f0c640cd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.663+0000 7f0c61d9b640 1 -- 172.21.15.44:0/2146548307 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c64151a90 con 0x7f0c640cb120 2024-04-02T13:29:34.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.663+0000 7f0c62d9d640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c640cc650 0x7f0c640cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:34.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.663+0000 7f0c61d9b640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c640cb120 0x7f0c640a1830 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c007340 tx=0x7f0c4c007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:34.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.663+0000 7f0c5affd640 1 -- 172.21.15.44:0/2146548307 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c00f860 con 0x7f0c640cb120 2024-04-02T13:29:34.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.663+0000 7f0c5affd640 1 -- 172.21.15.44:0/2146548307 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c4c018070 con 0x7f0c640cb120 2024-04-02T13:29:34.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.663+0000 7f0c5affd640 1 -- 172.21.15.44:0/2146548307 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c00fc80 con 0x7f0c640cb120 2024-04-02T13:29:34.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.663+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2146548307 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c640a1d70 con 0x7f0c640cb120 2024-04-02T13:29:34.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.663+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2146548307 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c640cbb70 con 0x7f0c640cb120 2024-04-02T13:29:34.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.664+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2146548307 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c640684f0 con 0x7f0c640cb120 2024-04-02T13:29:34.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.665+0000 7f0c5affd640 1 -- 172.21.15.44:0/2146548307 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0c4c01d460 con 0x7f0c640cb120 2024-04-02T13:29:34.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.665+0000 7f0c5affd640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c4406c8f0 0x7f0c4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:34.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.665+0000 7f0c5affd640 1 -- 172.21.15.44:0/2146548307 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f0c4c08bd60 con 0x7f0c640cb120 2024-04-02T13:29:34.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.665+0000 7f0c6259c640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c4406c8f0 0x7f0c4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:34.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.666+0000 7f0c6259c640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c4406c8f0 0x7f0c4406edb0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c036000 tx=0x7f0c5c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:34.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.669+0000 7f0c5affd640 1 -- 172.21.15.44:0/2146548307 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0c64151a90 con 0x7f0c640cb120 2024-04-02T13:29:34.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.863+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2146548307 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0c640d2af0 con 0x7f0c640cb120 2024-04-02T13:29:34.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.864+0000 7f0c5affd640 1 -- 172.21.15.44:0/2146548307 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v106) ==== 80+0+0 (secure 0 0 0) 0x7f0c4c05a000 con 0x7f0c640cb120 2024-04-02T13:29:34.868 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:29:34.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.865+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c4406c8f0 msgr2=0x7f0c4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.865+0000 7f0c68cf2640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c4406c8f0 0x7f0c4406edb0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c036000 tx=0x7f0c5c030000 comp rx=0 tx=0).stop 2024-04-02T13:29:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.865+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2146548307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c640cb120 msgr2=0x7f0c640a1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.865+0000 7f0c68cf2640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c640cb120 0x7f0c640a1830 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c007340 tx=0x7f0c4c007800 comp rx=0 tx=0).stop 2024-04-02T13:29:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.865+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2146548307 shutdown_connections 2024-04-02T13:29:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.865+0000 7f0c68cf2640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c4406c8f0 0x7f0c4406edb0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.865+0000 7f0c68cf2640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c640cc650 0x7f0c640cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.865+0000 7f0c68cf2640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c640cb120 0x7f0c640a1830 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.865+0000 7f0c68cf2640 1 --2- 172.21.15.44:0/2146548307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c500039e0 0x7f0c64151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.865+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2146548307 >> 172.21.15.44:0/2146548307 conn(0x7f0c6409d390 msgr2=0x7f0c6415e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.866+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2146548307 shutdown_connections 2024-04-02T13:29:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:34.866+0000 7f0c68cf2640 1 -- 172.21.15.44:0/2146548307 wait complete. 2024-04-02T13:29:34.880 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:29:34.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:34.880 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:34.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[77459]: fuse finished with error 0 and tester_r 0 2024-04-02T13:29:34.908 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:29:41.010 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:29:41.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:41.010 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:29:41.030 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:29:41.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.181+0000 7fc2c14eb640 1 Processor -- start 2024-04-02T13:29:41.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.181+0000 7fc2c14eb640 1 -- start start 2024-04-02T13:29:41.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2c14eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2bc1617e0 0x7fc2bc0ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2c14eb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc2bc0cffd0 0x7fc2bc1629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2c14eb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2bc0ccc90 0x7fc2bc0cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2c14eb640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc2bc162f00 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2c14eb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc2bc0cef30 con 0x7fc2bc0ccc90 2024-04-02T13:29:41.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2c14eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc2bc0cf210 con 0x7fc2bc1617e0 2024-04-02T13:29:41.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2ba7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc2bc0cffd0 0x7fc2bc1629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2ba7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc2bc0cffd0 0x7fc2bc1629c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40172/0 (socket says 172.21.15.44:40172) 2024-04-02T13:29:41.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2ba7fc640 1 -- 172.21.15.44:0/1024537527 learned_addr learned my addr 172.21.15.44:0/1024537527 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:41.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2baffd640 1 --2- 172.21.15.44:0/1024537527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2bc1617e0 0x7fc2bc0ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2bb7fe640 1 --2- 172.21.15.44:0/1024537527 >> v2:172.21.15.69:3300/0 conn(0x7fc2bc0ccc90 0x7fc2bc0cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2ba7fc640 1 -- 172.21.15.44:0/1024537527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2bc1617e0 msgr2=0x7fc2bc0ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2ba7fc640 1 --2- 172.21.15.44:0/1024537527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2bc1617e0 0x7fc2bc0ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2ba7fc640 1 -- 172.21.15.44:0/1024537527 >> v2:172.21.15.69:3300/0 conn(0x7fc2bc0ccc90 msgr2=0x7fc2bc0cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2ba7fc640 1 --2- 172.21.15.44:0/1024537527 >> v2:172.21.15.69:3300/0 conn(0x7fc2bc0ccc90 0x7fc2bc0cd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.182+0000 7fc2ba7fc640 1 -- 172.21.15.44:0/1024537527 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2bc0cf4f0 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.183+0000 7fc2ba7fc640 1 --2- 172.21.15.44:0/1024537527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2bc0cffd0 0x7fc2bc1629c0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0cc9e0 tx=0x7fc2b002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8eb42ce575edb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.183+0000 7fc2b9ffb640 1 -- 172.21.15.44:0/1024537527 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b002e070 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.183+0000 7fc2b9ffb640 1 -- 172.21.15.44:0/1024537527 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2b0037070 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.183+0000 7fc2b9ffb640 1 -- 172.21.15.44:0/1024537527 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0031a00 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.184+0000 7fc2c14eb640 1 -- 172.21.15.44:0/1024537527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2bc0cffd0 msgr2=0x7fc2bc1629c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.184+0000 7fc2c14eb640 1 --2- 172.21.15.44:0/1024537527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2bc0cffd0 0x7fc2bc1629c0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0cc9e0 tx=0x7fc2b002b880 comp rx=0 tx=0).stop 2024-04-02T13:29:41.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.184+0000 7fc2c14eb640 1 -- 172.21.15.44:0/1024537527 shutdown_connections 2024-04-02T13:29:41.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.184+0000 7fc2c14eb640 1 --2- 172.21.15.44:0/1024537527 >> v2:172.21.15.69:3300/0 conn(0x7fc2bc0ccc90 0x7fc2bc0cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.184+0000 7fc2c14eb640 1 --2- 172.21.15.44:0/1024537527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2bc0cffd0 0x7fc2bc1629c0 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.184+0000 7fc2c14eb640 1 --2- 172.21.15.44:0/1024537527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2bc1617e0 0x7fc2bc0ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.184+0000 7fc2c14eb640 1 -- 172.21.15.44:0/1024537527 >> 172.21.15.44:0/1024537527 conn(0x7fc2bc0b20e0 msgr2=0x7fc2bc0b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:41.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.184+0000 7fc2c14eb640 1 -- 172.21.15.44:0/1024537527 shutdown_connections 2024-04-02T13:29:41.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.184+0000 7fc2c14eb640 1 -- 172.21.15.44:0/1024537527 wait complete. 2024-04-02T13:29:41.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.185+0000 7fc2c14eb640 1 Processor -- start 2024-04-02T13:29:41.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.185+0000 7fc2c14eb640 1 -- start start 2024-04-02T13:29:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.186+0000 7fc2c14eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2bc0ccc90 0x7fc2bc0b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.186+0000 7fc2c14eb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2bc0cffd0 0x7fc2bc1f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.186+0000 7fc2c14eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2bc1617e0 0x7fc2bc1f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.186+0000 7fc2c14eb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc0cd620 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.186+0000 7fc2c14eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc0cfab0 con 0x7fc2bc0ccc90 2024-04-02T13:29:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.186+0000 7fc2c14eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc2bc161bc0 con 0x7fc2bc1617e0 2024-04-02T13:29:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.186+0000 7fc2ba7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2bc0cffd0 0x7fc2bc1f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.186+0000 7fc2ba7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2bc0cffd0 0x7fc2bc1f3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40178/0 (socket says 172.21.15.44:40178) 2024-04-02T13:29:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.186+0000 7fc2ba7fc640 1 -- 172.21.15.44:0/835993178 learned_addr learned my addr 172.21.15.44:0/835993178 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.186+0000 7fc2baffd640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2bc0ccc90 0x7fc2bc0b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.186+0000 7fc2bb7fe640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2bc1617e0 0x7fc2bc1f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.186+0000 7fc2ba7fc640 1 -- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2bc1617e0 msgr2=0x7fc2bc1f4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.186+0000 7fc2ba7fc640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2bc1617e0 0x7fc2bc1f4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.187+0000 7fc2ba7fc640 1 -- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2bc0ccc90 msgr2=0x7fc2bc0b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.187+0000 7fc2ba7fc640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2bc0ccc90 0x7fc2bc0b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.187+0000 7fc2ba7fc640 1 -- 172.21.15.44:0/835993178 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2bc0c8870 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.187+0000 7fc2baffd640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2bc0ccc90 0x7fc2bc0b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.187+0000 7fc2ba7fc640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2bc0cffd0 0x7fc2bc1f3bc0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fc2b002b8c0 tx=0x7fc2b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.187+0000 7fc2a37fe640 1 -- 172.21.15.44:0/835993178 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0032670 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.187+0000 7fc2a37fe640 1 -- 172.21.15.44:0/835993178 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2b0032850 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.187+0000 7fc2a37fe640 1 -- 172.21.15.44:0/835993178 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0032c40 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.187+0000 7fc2c14eb640 1 -- 172.21.15.44:0/835993178 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2bc0c8b50 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.187+0000 7fc2c14eb640 1 -- 172.21.15.44:0/835993178 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2bc0c8580 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.188+0000 7fc2c14eb640 1 -- 172.21.15.44:0/835993178 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2bc0c9630 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.188+0000 7fc2a37fe640 1 -- 172.21.15.44:0/835993178 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc2b00453f0 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.189+0000 7fc2a37fe640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc2a806c8f0 0x7fc2a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.189+0000 7fc2a37fe640 1 -- 172.21.15.44:0/835993178 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7fc2b00adb10 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.189+0000 7fc2baffd640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc2a806c8f0 0x7fc2a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.190+0000 7fc2baffd640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc2a806c8f0 0x7fc2a806edb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac003f00 tx=0x7fc2ac0098d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.194+0000 7fc2a37fe640 1 -- 172.21.15.44:0/835993178 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc2b0040070 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.416+0000 7fc2c14eb640 1 -- 172.21.15.44:0/835993178 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2bc0684f0 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.417+0000 7fc2a37fe640 1 -- 172.21.15.44:0/835993178 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5183 (secure 0 0 0) 0x7fc2b00500e0 con 0x7fc2bc0cffd0 2024-04-02T13:29:41.421 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:41.421 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7696,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2715855087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2715855087},{"type":"v1","addr":"172.21.15.69:6836","nonce":2715855087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7699,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3850680553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3850680553},{"type":"v1","addr":"172.21.15.69:6837","nonce":3850680553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/1330712383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1330712383},{"type":"v1","addr":"172.21.15.44:6835","nonce":1330712383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:29:10.907920+0000","modified":"2024-04-02T13:29:13.930909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8760":{"gid":8760,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.44:6834/3259420283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":3259420283},{"type":"v1","addr":"172.21.15.44:6834","nonce":3259420283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8760,"qdb_cluster":[8760]},"id":7}]} 2024-04-02T13:29:41.421 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 82 2024-04-02T13:29:41.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.419+0000 7fc2c14eb640 1 -- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc2a806c8f0 msgr2=0x7fc2a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.419+0000 7fc2c14eb640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc2a806c8f0 0x7fc2a806edb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac003f00 tx=0x7fc2ac0098d0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.419+0000 7fc2c14eb640 1 -- 172.21.15.44:0/835993178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2bc0cffd0 msgr2=0x7fc2bc1f3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.419+0000 7fc2c14eb640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2bc0cffd0 0x7fc2bc1f3bc0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fc2b002b8c0 tx=0x7fc2b00046c0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.419+0000 7fc2c14eb640 1 -- 172.21.15.44:0/835993178 shutdown_connections 2024-04-02T13:29:41.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.419+0000 7fc2c14eb640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc2a806c8f0 0x7fc2a806edb0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.419+0000 7fc2c14eb640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2bc1617e0 0x7fc2bc1f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.419+0000 7fc2c14eb640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2bc0cffd0 0x7fc2bc1f3bc0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.419+0000 7fc2c14eb640 1 --2- 172.21.15.44:0/835993178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2bc0ccc90 0x7fc2bc0b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.419+0000 7fc2c14eb640 1 -- 172.21.15.44:0/835993178 >> 172.21.15.44:0/835993178 conn(0x7fc2bc0b20e0 msgr2=0x7fc2bc0cab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:41.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.419+0000 7fc2c14eb640 1 -- 172.21.15.44:0/835993178 shutdown_connections 2024-04-02T13:29:41.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.419+0000 7fc2c14eb640 1 -- 172.21.15.44:0/835993178 wait complete. 2024-04-02T13:29:41.434 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-04-02T13:29:41.560 INFO:teuthology.orchestra.run.smithi044.stdout:{"message":"","return_code":0} 2024-04-02T13:29:41.560 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-02T13:29:41.560 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:29:41.560 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:29:41.579 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:29:41.579 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi044.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-04-02T13:29:41.579 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:29:41.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:41.580 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:29:41.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:29:41.639 INFO:teuthology.orchestra.run.smithi044.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-02T13:29:41.639 DEBUG:teuthology.orchestra.run.smithi044:> sudo daemon-helper kill ceph --cluster ceph -w 2024-04-02T13:29:41.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:41.687 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub start / repair,recursive 2024-04-02T13:29:41.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.790+0000 7fdf0051c640 1 Processor -- start 2024-04-02T13:29:41.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.790+0000 7fdf0051c640 1 -- start start 2024-04-02T13:29:41.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.790+0000 7fdf0051c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdef815e1d0 0x7fdef80d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.791+0000 7fdf0051c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdef80d1640 con 0x7fdef80cec60 2024-04-02T13:29:41.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.791+0000 7fdf0051c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdef80cf1f0 con 0x7fdef80d2dd0 2024-04-02T13:29:41.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.791+0000 7fdf0051c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fdef80cfe00 con 0x7fdef815e1d0 2024-04-02T13:29:41.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.791+0000 7fdefe2b8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdef815e1d0 0x7fdef80d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.791+0000 7fdefdab7640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fdef80cec60 0x7fdef80d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56046/0 (socket says 172.21.15.44:56046) 2024-04-02T13:29:41.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.791+0000 7fdefdab7640 1 -- 172.21.15.44:0/3618764679 learned_addr learned my addr 172.21.15.44:0/3618764679 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:41.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.791+0000 7fdefe2b8640 1 -- 172.21.15.44:0/3618764679 >> v1:172.21.15.69:6789/0 conn(0x7fdef80d2dd0 legacy=0x7fdef80b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.791+0000 7fdefe2b8640 1 -- 172.21.15.44:0/3618764679 >> v1:172.21.15.44:6789/0 conn(0x7fdef80cec60 legacy=0x7fdef80d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.791+0000 7fdefe2b8640 1 -- 172.21.15.44:0/3618764679 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef80cf4d0 con 0x7fdef815e1d0 2024-04-02T13:29:41.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.792+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/3618764679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdef815e1d0 0x7fdef80d1100 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fdef0009d90 tx=0x7fdef002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd288cf6edf235e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.792+0000 7fdefd2b6640 1 -- 172.21.15.44:0/3618764679 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0039070 con 0x7fdef815e1d0 2024-04-02T13:29:41.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.792+0000 7fdefd2b6640 1 -- 172.21.15.44:0/3618764679 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef002cc60 con 0x7fdef815e1d0 2024-04-02T13:29:41.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.792+0000 7fdefd2b6640 1 -- 172.21.15.44:0/3618764679 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0034660 con 0x7fdef815e1d0 2024-04-02T13:29:41.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.793+0000 7fdf0051c640 1 -- 172.21.15.44:0/3618764679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdef815e1d0 msgr2=0x7fdef80d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.793+0000 7fdf0051c640 1 --2- 172.21.15.44:0/3618764679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdef815e1d0 0x7fdef80d1100 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fdef0009d90 tx=0x7fdef002beb0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.793+0000 7fdf0051c640 1 -- 172.21.15.44:0/3618764679 shutdown_connections 2024-04-02T13:29:41.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.793+0000 7fdf0051c640 1 --2- 172.21.15.44:0/3618764679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdef815e1d0 0x7fdef80d1100 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.793+0000 7fdf0051c640 1 -- 172.21.15.44:0/3618764679 >> 172.21.15.44:0/3618764679 conn(0x7fdef809d3b0 msgr2=0x7fdef80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:41.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.793+0000 7fdf0051c640 1 -- 172.21.15.44:0/3618764679 shutdown_connections 2024-04-02T13:29:41.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.793+0000 7fdf0051c640 1 -- 172.21.15.44:0/3618764679 wait complete. 2024-04-02T13:29:41.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.794+0000 7fdf0051c640 1 Processor -- start 2024-04-02T13:29:41.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.794+0000 7fdf0051c640 1 -- start start 2024-04-02T13:29:41.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdf0051c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdef80cec60 0x7fdef81e8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdf0051c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdef80d2dd0 0x7fdef81621c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdf0051c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdef815e1d0 0x7fdef81f7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdf0051c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdef80b7fd0 con 0x7fdef80cec60 2024-04-02T13:29:41.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdf0051c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdef80cfa90 con 0x7fdef80d2dd0 2024-04-02T13:29:41.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdf0051c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdef80b8150 con 0x7fdef815e1d0 2024-04-02T13:29:41.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdefe2b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdef80cec60 0x7fdef81e8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdefe2b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdef80cec60 0x7fdef81e8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40188/0 (socket says 172.21.15.44:40188) 2024-04-02T13:29:41.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdefe2b8640 1 -- 172.21.15.44:0/4202593355 learned_addr learned my addr 172.21.15.44:0/4202593355 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:41.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdefdab7640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdef80d2dd0 0x7fdef81621c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdefeab9640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdef815e1d0 0x7fdef81f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdefe2b8640 1 -- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdef815e1d0 msgr2=0x7fdef81f7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdef815e1d0 0x7fdef81f7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdefe2b8640 1 -- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdef80d2dd0 msgr2=0x7fdef81621c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdef80d2dd0 0x7fdef81621c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdefe2b8640 1 -- 172.21.15.44:0/4202593355 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef81f8650 con 0x7fdef80cec60 2024-04-02T13:29:41.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdefdab7640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdef80d2dd0 0x7fdef81621c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:41.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdefeab9640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdef815e1d0 0x7fdef81f7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:41.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdef80cec60 0x7fdef81e8a40 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fdef0037040 tx=0x7fdef0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdef6ffd640 1 -- 172.21.15.44:0/4202593355 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0039050 con 0x7fdef80cec60 2024-04-02T13:29:41.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdef6ffd640 1 -- 172.21.15.44:0/4202593355 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef0045070 con 0x7fdef80cec60 2024-04-02T13:29:41.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdf0051c640 1 -- 172.21.15.44:0/4202593355 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef81e7f60 con 0x7fdef80cec60 2024-04-02T13:29:41.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.795+0000 7fdef6ffd640 1 -- 172.21.15.44:0/4202593355 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef003ea50 con 0x7fdef80cec60 2024-04-02T13:29:41.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.796+0000 7fdf0051c640 1 -- 172.21.15.44:0/4202593355 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef81e8520 con 0x7fdef80cec60 2024-04-02T13:29:41.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.798+0000 7fdef6ffd640 1 -- 172.21.15.44:0/4202593355 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdef002cae0 con 0x7fdef80cec60 2024-04-02T13:29:41.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.798+0000 7fdef6ffd640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdee406c8f0 0x7fdee406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.798+0000 7fdef6ffd640 1 -- 172.21.15.44:0/4202593355 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7fdef00aeaf0 con 0x7fdef80cec60 2024-04-02T13:29:41.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.798+0000 7fdefdab7640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdee406c8f0 0x7fdee406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.799+0000 7fdf031e4740 1 Processor -- start 2024-04-02T13:29:41.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.799+0000 7fdefdab7640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdee406c8f0 0x7fdee406edb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fdef81613a0 tx=0x7fdeec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.799+0000 7fdf031e4740 1 -- start start 2024-04-02T13:29:41.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.799+0000 7fdf031e4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 0x55f50d4946f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.799+0000 7fdf031e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d464330 0x55f50d491580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.799+0000 7fdf031e4740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d467e90 0x55f50d491ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdf031e4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f50d3ce770 con 0x55f50d494310 2024-04-02T13:29:41.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdf031e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f50d376960 con 0x55f50d464330 2024-04-02T13:29:41.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdf031e4740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55f50d3ca8c0 con 0x55f50d467e90 2024-04-02T13:29:41.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdefe2b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 0x55f50d4946f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdefe2b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 0x55f50d4946f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40190/0 (socket says 172.21.15.44:40190) 2024-04-02T13:29:41.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdefe2b8640 1 -- 172.21.15.44:0/1955161144 learned_addr learned my addr 172.21.15.44:0/1955161144 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:41.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdefdab7640 1 --2- 172.21.15.44:0/1955161144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d464330 0x55f50d491580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdefeab9640 1 --2- 172.21.15.44:0/1955161144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d467e90 0x55f50d491ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdefe2b8640 1 -- 172.21.15.44:0/1955161144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d467e90 msgr2=0x55f50d491ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/1955161144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d467e90 0x55f50d491ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdefe2b8640 1 -- 172.21.15.44:0/1955161144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d464330 msgr2=0x55f50d491580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/1955161144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d464330 0x55f50d491580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdefe2b8640 1 -- 172.21.15.44:0/1955161144 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f50d1aa7e0 con 0x55f50d494310 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdefeab9640 1 --2- 172.21.15.44:0/1955161144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d467e90 0x55f50d491ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/1955161144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 0x55f50d4946f0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fdef0084e60 tx=0x7fdef007ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77d13b48b3fd914 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdedffff640 1 -- 172.21.15.44:0/1955161144 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef00ab070 con 0x55f50d494310 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdedffff640 1 -- 172.21.15.44:0/1955161144 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef003a040 con 0x55f50d494310 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.800+0000 7fdedffff640 1 -- 172.21.15.44:0/1955161144 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef00830c0 con 0x55f50d494310 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.801+0000 7fdf031e4740 1 -- 172.21.15.44:0/1955161144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 msgr2=0x55f50d4946f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.801+0000 7fdf031e4740 1 --2- 172.21.15.44:0/1955161144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 0x55f50d4946f0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fdef0084e60 tx=0x7fdef007ceb0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.801+0000 7fdf031e4740 1 -- 172.21.15.44:0/1955161144 shutdown_connections 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.801+0000 7fdf031e4740 1 --2- 172.21.15.44:0/1955161144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d467e90 0x55f50d491ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.801+0000 7fdf031e4740 1 --2- 172.21.15.44:0/1955161144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d464330 0x55f50d491580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.801+0000 7fdf031e4740 1 --2- 172.21.15.44:0/1955161144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 0x55f50d4946f0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.801+0000 7fdf031e4740 1 -- 172.21.15.44:0/1955161144 >> 172.21.15.44:0/1955161144 conn(0x55f50d4665a0 msgr2=0x55f50d4af120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.801+0000 7fdf031e4740 1 -- 172.21.15.44:0/1955161144 shutdown_connections 2024-04-02T13:29:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.801+0000 7fdf031e4740 1 -- 172.21.15.44:0/1955161144 wait complete. 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.802+0000 7fdf031e4740 1 -- start start 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.802+0000 7fdf031e4740 1 Processor -- start 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdf031e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d467e90 0x55f50d48a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdf031e4740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d494310 0x55f50d48ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdf031e4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d48c470 0x55f50d4af100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdf031e4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f50d469040 con 0x55f50d48c470 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdf031e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f50d389d20 con 0x55f50d467e90 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdf031e4740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55f50d468430 con 0x55f50d494310 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdf031e4740 10 client.0 _ll_register_callbacks cb 0x55f50d356a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdefeab9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d48c470 0x55f50d4af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdefeab9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d48c470 0x55f50d4af100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40204/0 (socket says 172.21.15.44:40204) 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdefeab9640 1 -- 172.21.15.44:0/338518338 learned_addr learned my addr 172.21.15.44:0/338518338 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdefdab7640 1 --2- 172.21.15.44:0/338518338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d494310 0x55f50d48ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/338518338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d467e90 0x55f50d48a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdefeab9640 1 -- 172.21.15.44:0/338518338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d494310 msgr2=0x55f50d48ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdefeab9640 1 --2- 172.21.15.44:0/338518338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d494310 0x55f50d48ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdefeab9640 1 -- 172.21.15.44:0/338518338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d467e90 msgr2=0x55f50d48a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdefeab9640 1 --2- 172.21.15.44:0/338518338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d467e90 0x55f50d48a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdefeab9640 1 -- 172.21.15.44:0/338518338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f50d20af70 con 0x55f50d48c470 2024-04-02T13:29:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdefeab9640 1 --2- 172.21.15.44:0/338518338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d48c470 0x55f50d4af100 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fdee8001dc0 tx=0x7fdee800f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b05e89595e79db server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.803+0000 7fdedd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:29:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdedd7fa640 1 -- 172.21.15.44:0/338518338 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee80050d0 con 0x55f50d48c470 2024-04-02T13:29:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdf031e4740 1 -- 172.21.15.44:0/338518338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdef003a040 con 0x55f50d48c470 2024-04-02T13:29:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdedd7fa640 1 -- 172.21.15.44:0/338518338 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7fdee8005270 con 0x55f50d48c470 2024-04-02T13:29:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdedd7fa640 1 -- 172.21.15.44:0/338518338 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fdee8020050 con 0x55f50d48c470 2024-04-02T13:29:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdf031e4740 10 client.9120 fetch_fsmap learned FSMap version 82 2024-04-02T13:29:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdf031e4740 1 -- 172.21.15.44:0/338518338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) -- 0x55f50d0a2170 con 0x55f50d48c470 2024-04-02T13:29:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdedd7fa640 1 -- 172.21.15.44:0/338518338 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef003a040 con 0x55f50d48c470 2024-04-02T13:29:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdedd7fa640 1 -- 172.21.15.44:0/338518338 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 82) ==== 3494+0+0 (secure 0 0 0) 0x7fdee8023020 con 0x55f50d48c470 2024-04-02T13:29:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdf031e4740 10 client.9120 fetch_fsmap finished waiting for FSMap version 82 2024-04-02T13:29:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdf031e4740 10 client.9120 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-02T13:29:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdf031e4740 20 client.9120 populate_metadata read hostname 'smithi044' 2024-04-02T13:29:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdf031e4740 1 --2- 172.21.15.44:0/338518338 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55f50d495d80 0x55f50d496190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdf031e4740 4 client.9120 mds_command: new command op to 8760 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:29:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdf031e4740 1 -- 172.21.15.44:0/338518338 --> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f50d3dbcb0 con 0x55f50d495d80 2024-04-02T13:29:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.804+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/338518338 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55f50d495d80 0x55f50d496190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.805+0000 7fdedd7fa640 10 client.9120 ms_handle_connect on v2:172.21.15.44:6833/3259420283 2024-04-02T13:29:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.805+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/338518338 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55f50d495d80 0x55f50d496190 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2902b4f03d5fcabd server_cookie=dd9ea2f03d3f9ed6 in_seq=0 out_seq=0 2024-04-02T13:29:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.805+0000 7fdedd7fa640 1 -- 172.21.15.44:0/338518338 <== mds.0 v2:172.21.15.44:6833/3259420283 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f50d3dbcb0 con 0x55f50d495d80 2024-04-02T13:29:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.805+0000 7fdedd7fa640 10 client.9120 handle_command_reply: tid=0 2024-04-02T13:29:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.805+0000 7fdf031e4740 1 client.9120 shutdown 2024-04-02T13:29:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.806+0000 7fdf031e4740 1 -- 172.21.15.44:0/338518338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d48c470 msgr2=0x55f50d4af100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.806+0000 7fdf031e4740 1 --2- 172.21.15.44:0/338518338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d48c470 0x55f50d4af100 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fdee8001dc0 tx=0x7fdee800f5a0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.806+0000 7fdf031e4740 1 -- 172.21.15.44:0/338518338 shutdown_connections 2024-04-02T13:29:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.806+0000 7fdf031e4740 1 --2- 172.21.15.44:0/338518338 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55f50d495d80 0x55f50d496190 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.806+0000 7fdf031e4740 1 --2- 172.21.15.44:0/338518338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d48c470 0x55f50d4af100 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.806+0000 7fdf031e4740 1 --2- 172.21.15.44:0/338518338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d494310 0x55f50d48ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.806+0000 7fdedd7fa640 0 client.9120 ms_handle_reset on v2:172.21.15.44:6833/3259420283 2024-04-02T13:29:41.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.806+0000 7fdf031e4740 1 --2- 172.21.15.44:0/338518338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d467e90 0x55f50d48a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.806+0000 7fdf031e4740 1 -- 172.21.15.44:0/338518338 >> 172.21.15.44:0/338518338 conn(0x55f50d464330 msgr2=0x55f50d48fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:41.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.806+0000 7fdf031e4740 1 -- 172.21.15.44:0/338518338 shutdown_connections 2024-04-02T13:29:41.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.806+0000 7fdf031e4740 1 -- 172.21.15.44:0/338518338 wait complete. 2024-04-02T13:29:41.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.806+0000 7fdf031e4740 20 client.9120 trim_cache size 0 max 16384 2024-04-02T13:29:41.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.810+0000 7f3432f60640 1 Processor -- start 2024-04-02T13:29:41.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.810+0000 7f3432f60640 1 -- start start 2024-04-02T13:29:41.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f3432f60640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f342c161920 0x7f342c0cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f3432f60640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f342c0d1120 0x7f342c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f3432f60640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f342c0cf8c0 con 0x7f342c0cf320 2024-04-02T13:29:41.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f3432f60640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f342c0d1e20 con 0x7f342c161920 2024-04-02T13:29:41.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f3432f60640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f342c0cfba0 con 0x7f342c0d1120 2024-04-02T13:29:41.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f3430cfc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f342c161920 0x7f342c0cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f342bfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f342c0cf320 0x7f342c0d0a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56058/0 (socket says 172.21.15.44:56058) 2024-04-02T13:29:41.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f342bfff640 1 -- 172.21.15.44:0/2202464462 learned_addr learned my addr 172.21.15.44:0/2202464462 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:41.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f34314fd640 1 --2- 172.21.15.44:0/2202464462 >> v2:172.21.15.69:3300/0 conn(0x7f342c0d1120 0x7f342c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f3430cfc640 1 -- 172.21.15.44:0/2202464462 >> v2:172.21.15.69:3300/0 conn(0x7f342c0d1120 msgr2=0x7f342c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f3430cfc640 1 --2- 172.21.15.44:0/2202464462 >> v2:172.21.15.69:3300/0 conn(0x7f342c0d1120 0x7f342c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f3430cfc640 1 -- 172.21.15.44:0/2202464462 >> v1:172.21.15.44:6789/0 conn(0x7f342c0cf320 legacy=0x7f342c0d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f342b7fe640 1 -- 172.21.15.44:0/2202464462 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1057386894 0 0) 0x7f342c0cf8c0 con 0x7f342c0cf320 2024-04-02T13:29:41.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.811+0000 7f3430cfc640 1 -- 172.21.15.44:0/2202464462 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342c0cfe80 con 0x7f342c161920 2024-04-02T13:29:41.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.813+0000 7f3430cfc640 1 --2- 172.21.15.44:0/2202464462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c161920 0x7f342c0cede0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f342c08e970 tx=0x7f341c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527e2acfe8622b57 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.813+0000 7f342b7fe640 1 -- 172.21.15.44:0/2202464462 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c035020 con 0x7f342c161920 2024-04-02T13:29:41.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.813+0000 7f342b7fe640 1 -- 172.21.15.44:0/2202464462 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f341c031830 con 0x7f342c161920 2024-04-02T13:29:41.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.813+0000 7f342b7fe640 1 -- 172.21.15.44:0/2202464462 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c031c00 con 0x7f342c161920 2024-04-02T13:29:41.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.813+0000 7f3432f60640 1 -- 172.21.15.44:0/2202464462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c161920 msgr2=0x7f342c0cede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.813+0000 7f3432f60640 1 --2- 172.21.15.44:0/2202464462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c161920 0x7f342c0cede0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f342c08e970 tx=0x7f341c02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.813+0000 7f3432f60640 1 -- 172.21.15.44:0/2202464462 shutdown_connections 2024-04-02T13:29:41.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.813+0000 7f3432f60640 1 --2- 172.21.15.44:0/2202464462 >> v2:172.21.15.69:3300/0 conn(0x7f342c0d1120 0x7f342c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.813+0000 7f3432f60640 1 --2- 172.21.15.44:0/2202464462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c161920 0x7f342c0cede0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.813+0000 7f3432f60640 1 -- 172.21.15.44:0/2202464462 >> 172.21.15.44:0/2202464462 conn(0x7f342c09d470 msgr2=0x7f342c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:41.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.813+0000 7f3432f60640 1 -- 172.21.15.44:0/2202464462 shutdown_connections 2024-04-02T13:29:41.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.813+0000 7f3432f60640 1 -- 172.21.15.44:0/2202464462 wait complete. 2024-04-02T13:29:41.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.814+0000 7f3432f60640 1 Processor -- start 2024-04-02T13:29:41.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.815+0000 7f3432f60640 1 -- start start 2024-04-02T13:29:41.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.815+0000 7f3432f60640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c0cf320 0x7f342c159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.815+0000 7f3432f60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f342c0d1120 0x7f342c0cd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.815+0000 7f3432f60640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c161920 0x7f342c0c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.815+0000 7f3432f60640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f342c161d00 con 0x7f342c0cf320 2024-04-02T13:29:41.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.815+0000 7f3432f60640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f342c0d0160 con 0x7f342c161920 2024-04-02T13:29:41.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.815+0000 7f3432f60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f342c161e80 con 0x7f342c0d1120 2024-04-02T13:29:41.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.815+0000 7f3430cfc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c0cf320 0x7f342c159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.815+0000 7f3430cfc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c0cf320 0x7f342c159ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40226/0 (socket says 172.21.15.44:40226) 2024-04-02T13:29:41.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.815+0000 7f3430cfc640 1 -- 172.21.15.44:0/259763857 learned_addr learned my addr 172.21.15.44:0/259763857 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:41.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f342bfff640 1 --2- 172.21.15.44:0/259763857 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f342c0d1120 0x7f342c0cd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f34314fd640 1 --2- 172.21.15.44:0/259763857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c161920 0x7f342c0c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f3430cfc640 1 -- 172.21.15.44:0/259763857 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f342c0d1120 msgr2=0x7f342c0cd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f3430cfc640 1 --2- 172.21.15.44:0/259763857 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f342c0d1120 0x7f342c0cd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f3430cfc640 1 -- 172.21.15.44:0/259763857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c161920 msgr2=0x7f342c0c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f3430cfc640 1 --2- 172.21.15.44:0/259763857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c161920 0x7f342c0c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f3430cfc640 1 -- 172.21.15.44:0/259763857 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342c0c8e20 con 0x7f342c0cf320 2024-04-02T13:29:41.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f34314fd640 1 --2- 172.21.15.44:0/259763857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c161920 0x7f342c0c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:41.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f3430cfc640 1 --2- 172.21.15.44:0/259763857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c0cf320 0x7f342c159ad0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f341c000c00 tx=0x7f341c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f34297fa640 1 -- 172.21.15.44:0/259763857 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c040070 con 0x7f342c0cf320 2024-04-02T13:29:41.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f34297fa640 1 -- 172.21.15.44:0/259763857 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f341c039a20 con 0x7f342c0cf320 2024-04-02T13:29:41.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f3432f60640 1 -- 172.21.15.44:0/259763857 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f342c0c9100 con 0x7f342c0cf320 2024-04-02T13:29:41.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f34297fa640 1 -- 172.21.15.44:0/259763857 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c035040 con 0x7f342c0cf320 2024-04-02T13:29:41.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.816+0000 7f3432f60640 1 -- 172.21.15.44:0/259763857 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f342c0c96c0 con 0x7f342c0cf320 2024-04-02T13:29:41.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.817+0000 7f3432f60640 1 -- 172.21.15.44:0/259763857 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f342c0684f0 con 0x7f342c0cf320 2024-04-02T13:29:41.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.818+0000 7f34297fa640 1 -- 172.21.15.44:0/259763857 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f341c04b060 con 0x7f342c0cf320 2024-04-02T13:29:41.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.818+0000 7f34297fa640 1 --2- 172.21.15.44:0/259763857 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f341806c920 0x7f341806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.819+0000 7f34297fa640 1 -- 172.21.15.44:0/259763857 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7f341c0adbe0 con 0x7f342c0cf320 2024-04-02T13:29:41.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.819+0000 7f34297fa640 1 -- 172.21.15.44:0/259763857 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+511 (secure 0 0 0) 0x7f341c052020 con 0x7f342c0cf320 2024-04-02T13:29:41.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.819+0000 7f3432f60640 1 -- 172.21.15.44:0/259763857 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7f342c0d17a0 con 0x7f342c0cf320 2024-04-02T13:29:41.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.819+0000 7f342bfff640 1 --2- 172.21.15.44:0/259763857 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f341806c920 0x7f341806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.819+0000 7f342bfff640 1 --2- 172.21.15.44:0/259763857 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f341806c920 0x7f341806ede0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3420007fd0 tx=0x7f3420009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.826+0000 7fdf031e4740 1 Processor -- start 2024-04-02T13:29:41.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.826+0000 7fdf031e4740 1 -- start start 2024-04-02T13:29:41.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdf031e4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d467e90 0x55f50d48ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdf031e4740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d48c470 0x55f50d4b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdf031e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d494310 0x55f50d4c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdf031e4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f50d494a60 con 0x55f50d467e90 2024-04-02T13:29:41.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdf031e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdef8201160 con 0x55f50d494310 2024-04-02T13:29:41.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdf031e4740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdef8200180 con 0x55f50d48c470 2024-04-02T13:29:41.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdefe2b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d467e90 0x55f50d48ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdefe2b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d467e90 0x55f50d48ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40238/0 (socket says 172.21.15.44:40238) 2024-04-02T13:29:41.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdefe2b8640 1 -- 172.21.15.44:0/2213084930 learned_addr learned my addr 172.21.15.44:0/2213084930 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:41.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdefe2b8640 1 -- 172.21.15.44:0/2213084930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d48c470 msgr2=0x55f50d4b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/2213084930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d48c470 0x55f50d4b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdefe2b8640 1 -- 172.21.15.44:0/2213084930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d494310 msgr2=0x55f50d4c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/2213084930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d494310 0x55f50d4c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdefe2b8640 1 -- 172.21.15.44:0/2213084930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f50d465e40 con 0x55f50d467e90 2024-04-02T13:29:41.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/2213084930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d467e90 0x55f50d48ea40 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fdef0000f80 tx=0x7fdef0092000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367833b9c137dcf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.827+0000 7fdedf7fe640 1 -- 172.21.15.44:0/2213084930 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef008a070 con 0x55f50d467e90 2024-04-02T13:29:41.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.828+0000 7fdedf7fe640 1 -- 172.21.15.44:0/2213084930 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef0085390 con 0x55f50d467e90 2024-04-02T13:29:41.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.828+0000 7fdedf7fe640 1 -- 172.21.15.44:0/2213084930 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0090030 con 0x55f50d467e90 2024-04-02T13:29:41.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.828+0000 7fdf031e4740 1 -- 172.21.15.44:0/2213084930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d467e90 msgr2=0x55f50d48ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.828+0000 7fdf031e4740 1 --2- 172.21.15.44:0/2213084930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d467e90 0x55f50d48ea40 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fdef0000f80 tx=0x7fdef0092000 comp rx=0 tx=0).stop 2024-04-02T13:29:41.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.828+0000 7fdf031e4740 1 -- 172.21.15.44:0/2213084930 shutdown_connections 2024-04-02T13:29:41.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.828+0000 7fdf031e4740 1 --2- 172.21.15.44:0/2213084930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d494310 0x55f50d4c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.828+0000 7fdf031e4740 1 --2- 172.21.15.44:0/2213084930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d48c470 0x55f50d4b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.828+0000 7fdf031e4740 1 --2- 172.21.15.44:0/2213084930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d467e90 0x55f50d48ea40 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.828+0000 7fdf031e4740 1 -- 172.21.15.44:0/2213084930 >> 172.21.15.44:0/2213084930 conn(0x55f50d464330 msgr2=0x55f50d48cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.828+0000 7fdf031e4740 1 -- 172.21.15.44:0/2213084930 shutdown_connections 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.828+0000 7fdf031e4740 1 -- 172.21.15.44:0/2213084930 wait complete. 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.829+0000 7fdf031e4740 1 -- start start 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.829+0000 7fdf031e4740 1 Processor -- start 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdf031e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d467e90 0x55f50d529b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdf031e4740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d48c470 0x55f50d48f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdf031e4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 0x55f50d52d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdf031e4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdef81ffd20 con 0x55f50d494310 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdf031e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdef81f4d30 con 0x55f50d467e90 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdf031e4740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55f50d493c60 con 0x55f50d48c470 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdf031e4740 10 client.0 _ll_register_callbacks cb 0x7fdee8007d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdefeab9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 0x55f50d52d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdefeab9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 0x55f50d52d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40244/0 (socket says 172.21.15.44:40244) 2024-04-02T13:29:41.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdefeab9640 1 -- 172.21.15.44:0/4293137742 learned_addr learned my addr 172.21.15.44:0/4293137742 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:41.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdefeab9640 1 -- 172.21.15.44:0/4293137742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d48c470 msgr2=0x55f50d48f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdefeab9640 1 --2- 172.21.15.44:0/4293137742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d48c470 0x55f50d48f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/4293137742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d467e90 0x55f50d529b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdefeab9640 1 -- 172.21.15.44:0/4293137742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d467e90 msgr2=0x55f50d529b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdefeab9640 1 --2- 172.21.15.44:0/4293137742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d467e90 0x55f50d529b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdefeab9640 1 -- 172.21.15.44:0/4293137742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f50d52a050 con 0x55f50d494310 2024-04-02T13:29:41.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdefeab9640 1 --2- 172.21.15.44:0/4293137742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 0x55f50d52d730 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fdee8000f60 tx=0x7fdee8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd7c87057f4e39a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.830+0000 7fdedd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:29:41.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdedd7fa640 1 -- 172.21.15.44:0/4293137742 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee800faa0 con 0x55f50d494310 2024-04-02T13:29:41.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdedd7fa640 1 -- 172.21.15.44:0/4293137742 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7fdee8003a10 con 0x55f50d494310 2024-04-02T13:29:41.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdf031e4740 1 -- 172.21.15.44:0/4293137742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdef0085390 con 0x55f50d494310 2024-04-02T13:29:41.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdedd7fa640 1 -- 172.21.15.44:0/4293137742 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fdee80035f0 con 0x55f50d494310 2024-04-02T13:29:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdf031e4740 10 client.9135 fetch_fsmap learned FSMap version 82 2024-04-02T13:29:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdf031e4740 1 -- 172.21.15.44:0/4293137742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) -- 0x55f50d4c9f80 con 0x55f50d494310 2024-04-02T13:29:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdedd7fa640 1 -- 172.21.15.44:0/4293137742 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef0085390 con 0x55f50d494310 2024-04-02T13:29:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdedd7fa640 1 -- 172.21.15.44:0/4293137742 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 82) ==== 3494+0+0 (secure 0 0 0) 0x7fdee8035080 con 0x55f50d494310 2024-04-02T13:29:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdf031e4740 10 client.9135 fetch_fsmap finished waiting for FSMap version 82 2024-04-02T13:29:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdf031e4740 10 client.9135 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-02T13:29:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdf031e4740 20 client.9135 populate_metadata read hostname 'smithi044' 2024-04-02T13:29:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdf031e4740 1 --2- 172.21.15.44:0/4293137742 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55f50d4b2dc0 0x55f50d492090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdf031e4740 4 client.9135 mds_command: new command op to 8760 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-04-02T13:29:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdf031e4740 1 -- 172.21.15.44:0/4293137742 --> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) -- 0x55f50d464b30 con 0x55f50d4b2dc0 2024-04-02T13:29:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/4293137742 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55f50d4b2dc0 0x55f50d492090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.831+0000 7fdedd7fa640 10 client.9135 ms_handle_connect on v2:172.21.15.44:6833/3259420283 2024-04-02T13:29:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.832+0000 7fdefe2b8640 1 --2- 172.21.15.44:0/4293137742 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55f50d4b2dc0 0x55f50d492090 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85d4f5882935e622 server_cookie=25e892637823e572 in_seq=0 out_seq=0 2024-04-02T13:29:41.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.832+0000 7fdedd7fa640 1 -- 172.21.15.44:0/4293137742 <== mds.0 v2:172.21.15.44:6833/3259420283 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f50d464b30 con 0x55f50d4b2dc0 2024-04-02T13:29:41.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.832+0000 7fdedd7fa640 10 client.9135 handle_command_reply: tid=0 2024-04-02T13:29:41.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.832+0000 7fdf031e4740 1 client.9135 shutdown 2024-04-02T13:29:41.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.832+0000 7fdf031e4740 1 -- 172.21.15.44:0/4293137742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 msgr2=0x55f50d52d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.832+0000 7fdf031e4740 1 --2- 172.21.15.44:0/4293137742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 0x55f50d52d730 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fdee8000f60 tx=0x7fdee8000ed0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf031e4740 1 -- 172.21.15.44:0/4293137742 shutdown_connections 2024-04-02T13:29:41.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf031e4740 1 --2- 172.21.15.44:0/4293137742 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x55f50d4b2dc0 0x55f50d492090 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdedd7fa640 0 client.9135 ms_handle_reset on v2:172.21.15.44:6833/3259420283 2024-04-02T13:29:41.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf031e4740 1 --2- 172.21.15.44:0/4293137742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f50d494310 0x55f50d52d730 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf031e4740 1 --2- 172.21.15.44:0/4293137742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f50d48c470 0x55f50d48f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf031e4740 1 --2- 172.21.15.44:0/4293137742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f50d467e90 0x55f50d529b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf031e4740 1 -- 172.21.15.44:0/4293137742 >> 172.21.15.44:0/4293137742 conn(0x55f50d464330 msgr2=0x55f50d48cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:41.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf031e4740 1 -- 172.21.15.44:0/4293137742 shutdown_connections 2024-04-02T13:29:41.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf031e4740 1 -- 172.21.15.44:0/4293137742 wait complete. 2024-04-02T13:29:41.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf031e4740 20 client.9135 trim_cache size 0 max 16384 2024-04-02T13:29:41.849 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:29:41.849 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-02T13:29:41.849 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "a0fedb08-35ef-4301-8fc9-b3d05ed842a9", 2024-04-02T13:29:41.849 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-02T13:29:41.849 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:29:41.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf0051c640 1 -- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdee406c8f0 msgr2=0x7fdee406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf0051c640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdee406c8f0 0x7fdee406edb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fdef81613a0 tx=0x7fdeec009040 comp rx=0 tx=0).stop 2024-04-02T13:29:41.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf0051c640 1 -- 172.21.15.44:0/4202593355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdef80cec60 msgr2=0x7fdef81e8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf0051c640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdef80cec60 0x7fdef81e8a40 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fdef0037040 tx=0x7fdef0004a40 comp rx=0 tx=0).stop 2024-04-02T13:29:41.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf0051c640 1 -- 172.21.15.44:0/4202593355 shutdown_connections 2024-04-02T13:29:41.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf0051c640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdee406c8f0 0x7fdee406edb0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf0051c640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdef815e1d0 0x7fdef81f7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf0051c640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdef80d2dd0 0x7fdef81621c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf0051c640 1 --2- 172.21.15.44:0/4202593355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdef80cec60 0x7fdef81e8a40 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf0051c640 1 -- 172.21.15.44:0/4202593355 >> 172.21.15.44:0/4202593355 conn(0x7fdef809d3b0 msgr2=0x7fdef806c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:41.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf0051c640 1 -- 172.21.15.44:0/4202593355 shutdown_connections 2024-04-02T13:29:41.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.833+0000 7fdf0051c640 1 -- 172.21.15.44:0/4202593355 wait complete. 2024-04-02T13:29:41.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:29:41.852 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-02T13:29:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.956+0000 7fb9c1a8f640 1 Processor -- start 2024-04-02T13:29:41.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.956+0000 7fb9c1a8f640 1 -- start start 2024-04-02T13:29:41.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.957+0000 7fb9c1a8f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb9bc15e1d0 0x7fb9bc0d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.957+0000 7fb9c1a8f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb9bc0cec60 0x7fb9bc0d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.957+0000 7fb9c1a8f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9bc0d1640 con 0x7fb9bc15e5b0 2024-04-02T13:29:41.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.957+0000 7fb9c1a8f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fb9bc0cfe00 con 0x7fb9bc15e1d0 2024-04-02T13:29:41.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.957+0000 7fb9c1a8f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb9bc0cf1f0 con 0x7fb9bc0cec60 2024-04-02T13:29:41.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.957+0000 7fb9bb7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb9bc15e1d0 0x7fb9bc0d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.957+0000 7fb9bbfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb9bc15e5b0 0x7fb9bc0b4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56068/0 (socket says 172.21.15.44:56068) 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.957+0000 7fb9bbfff640 1 -- 172.21.15.44:0/1334175865 learned_addr learned my addr 172.21.15.44:0/1334175865 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.958+0000 7fb9baffd640 1 --2- 172.21.15.44:0/1334175865 >> v2:172.21.15.69:3300/0 conn(0x7fb9bc0cec60 0x7fb9bc0d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.958+0000 7fb9bb7fe640 1 -- 172.21.15.44:0/1334175865 >> v2:172.21.15.69:3300/0 conn(0x7fb9bc0cec60 msgr2=0x7fb9bc0d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.958+0000 7fb9bb7fe640 1 --2- 172.21.15.44:0/1334175865 >> v2:172.21.15.69:3300/0 conn(0x7fb9bc0cec60 0x7fb9bc0d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.958+0000 7fb9bb7fe640 1 -- 172.21.15.44:0/1334175865 >> v1:172.21.15.44:6789/0 conn(0x7fb9bc15e5b0 legacy=0x7fb9bc0b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.958+0000 7fb9bb7fe640 1 -- 172.21.15.44:0/1334175865 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9bc0cf4d0 con 0x7fb9bc15e1d0 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.958+0000 7fb9ba7fc640 1 -- 172.21.15.44:0/1334175865 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3229859876 0 0) 0x7fb9bc0d1640 con 0x7fb9bc15e5b0 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.958+0000 7fb9baffd640 1 --2- 172.21.15.44:0/1334175865 >> v2:172.21.15.69:3300/0 conn(0x7fb9bc0cec60 0x7fb9bc0d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.958+0000 7fb9bb7fe640 1 --2- 172.21.15.44:0/1334175865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9bc15e1d0 0x7fb9bc0d1100 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc161f20 tx=0x7fb9ac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4357d10b690dbc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.959+0000 7fb9ba7fc640 1 -- 172.21.15.44:0/1334175865 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac0315a0 con 0x7fb9bc15e1d0 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.959+0000 7fb9ba7fc640 1 -- 172.21.15.44:0/1334175865 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ac037070 con 0x7fb9bc15e1d0 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.959+0000 7fb9ba7fc640 1 -- 172.21.15.44:0/1334175865 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac031960 con 0x7fb9bc15e1d0 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.959+0000 7fb9c1a8f640 1 -- 172.21.15.44:0/1334175865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9bc15e1d0 msgr2=0x7fb9bc0d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.959+0000 7fb9c1a8f640 1 --2- 172.21.15.44:0/1334175865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9bc15e1d0 0x7fb9bc0d1100 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc161f20 tx=0x7fb9ac02b840 comp rx=0 tx=0).stop 2024-04-02T13:29:41.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.959+0000 7fb9c1a8f640 1 -- 172.21.15.44:0/1334175865 shutdown_connections 2024-04-02T13:29:41.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.959+0000 7fb9c1a8f640 1 --2- 172.21.15.44:0/1334175865 >> v2:172.21.15.69:3300/0 conn(0x7fb9bc0cec60 0x7fb9bc0d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.959+0000 7fb9c1a8f640 1 --2- 172.21.15.44:0/1334175865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9bc15e1d0 0x7fb9bc0d1100 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.959+0000 7fb9c1a8f640 1 -- 172.21.15.44:0/1334175865 >> 172.21.15.44:0/1334175865 conn(0x7fb9bc09d3b0 msgr2=0x7fb9bc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:41.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.959+0000 7fb9c1a8f640 1 -- 172.21.15.44:0/1334175865 shutdown_connections 2024-04-02T13:29:41.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.959+0000 7fb9c1a8f640 1 -- 172.21.15.44:0/1334175865 wait complete. 2024-04-02T13:29:41.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.960+0000 7fb9c1a8f640 1 Processor -- start 2024-04-02T13:29:41.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.961+0000 7fb9c1a8f640 1 -- start start 2024-04-02T13:29:41.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.961+0000 7fb9c1a8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9bc0cec60 0x7fb9bc15a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.961+0000 7fb9c1a8f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9bc15e1d0 0x7fb9bc0a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.961+0000 7fb9c1a8f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9bc15e5b0 0x7fb9bc0a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.961+0000 7fb9c1a8f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc0d2c00 con 0x7fb9bc0cec60 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.961+0000 7fb9c1a8f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc0cfa90 con 0x7fb9bc15e1d0 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.961+0000 7fb9c1a8f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb9bc0d2d80 con 0x7fb9bc15e5b0 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.961+0000 7fb9bb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9bc0cec60 0x7fb9bc15a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.961+0000 7fb9bb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9bc0cec60 0x7fb9bc15a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40274/0 (socket says 172.21.15.44:40274) 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.961+0000 7fb9bb7fe640 1 -- 172.21.15.44:0/2156250381 learned_addr learned my addr 172.21.15.44:0/2156250381 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9bbfff640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9bc15e5b0 0x7fb9bc0a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9baffd640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9bc15e1d0 0x7fb9bc0a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9bb7fe640 1 -- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9bc15e5b0 msgr2=0x7fb9bc0a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9bb7fe640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9bc15e5b0 0x7fb9bc0a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9bb7fe640 1 -- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9bc15e1d0 msgr2=0x7fb9bc0a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9bb7fe640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9bc15e1d0 0x7fb9bc0a1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9bb7fe640 1 -- 172.21.15.44:0/2156250381 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9bc1609c0 con 0x7fb9bc0cec60 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9bbfff640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9bc15e5b0 0x7fb9bc0a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:41.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9baffd640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9bc15e1d0 0x7fb9bc0a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:29:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9bb7fe640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9bc0cec60 0x7fb9bc15a550 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac009890 tx=0x7fb9ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9a3fff640 1 -- 172.21.15.44:0/2156250381 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac004230 con 0x7fb9bc0cec60 2024-04-02T13:29:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9a3fff640 1 -- 172.21.15.44:0/2156250381 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ac0043d0 con 0x7fb9bc0cec60 2024-04-02T13:29:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9c1a8f640 1 -- 172.21.15.44:0/2156250381 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9bc1f7ea0 con 0x7fb9bc0cec60 2024-04-02T13:29:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9a3fff640 1 -- 172.21.15.44:0/2156250381 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac003850 con 0x7fb9bc0cec60 2024-04-02T13:29:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.962+0000 7fb9c1a8f640 1 -- 172.21.15.44:0/2156250381 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9bc09e230 con 0x7fb9bc0cec60 2024-04-02T13:29:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.963+0000 7fb9a3fff640 1 -- 172.21.15.44:0/2156250381 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb9ac0039f0 con 0x7fb9bc0cec60 2024-04-02T13:29:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.964+0000 7fb9a3fff640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb9a806c920 0x7fb9a806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.964+0000 7fb9a3fff640 1 -- 172.21.15.44:0/2156250381 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7fb9ac02f030 con 0x7fb9bc0cec60 2024-04-02T13:29:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.964+0000 7fb9baffd640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb9a806c920 0x7fb9a806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.965+0000 7fb9baffd640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb9a806c920 0x7fb9a806ede0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0006fd0 tx=0x7fb9b0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.965+0000 7fb9c4670740 1 Processor -- start 2024-04-02T13:29:41.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.965+0000 7fb9c4670740 1 -- start start 2024-04-02T13:29:41.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9c4670740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42e39a0 0x562be42b6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9c4670740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 0x562be42e6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9c4670740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42b6e80 0x562be42e6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9c4670740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562be4220830 con 0x562be42e3d80 2024-04-02T13:29:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9c4670740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562be41c8820 con 0x562be42b6e80 2024-04-02T13:29:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9c4670740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x562be421c7e0 con 0x562be42e39a0 2024-04-02T13:29:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9baffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 0x562be42e6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9baffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 0x562be42e6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40290/0 (socket says 172.21.15.44:40290) 2024-04-02T13:29:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9baffd640 1 -- 172.21.15.44:0/2122807322 learned_addr learned my addr 172.21.15.44:0/2122807322 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:41.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9bbfff640 1 --2- 172.21.15.44:0/2122807322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42b6e80 0x562be42e6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9bb7fe640 1 --2- 172.21.15.44:0/2122807322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42e39a0 0x562be42b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9baffd640 1 -- 172.21.15.44:0/2122807322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42e39a0 msgr2=0x562be42b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9baffd640 1 --2- 172.21.15.44:0/2122807322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42e39a0 0x562be42b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9baffd640 1 -- 172.21.15.44:0/2122807322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42b6e80 msgr2=0x562be42e6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9baffd640 1 --2- 172.21.15.44:0/2122807322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42b6e80 0x562be42e6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9baffd640 1 -- 172.21.15.44:0/2122807322 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562be3ffe7d0 con 0x562be42e3d80 2024-04-02T13:29:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9bbfff640 1 --2- 172.21.15.44:0/2122807322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42b6e80 0x562be42e6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:29:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9baffd640 1 --2- 172.21.15.44:0/2122807322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 0x562be42e6730 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000f9e0 tx=0x7fb9b0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a34244181a0be6f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9a17fa640 1 -- 172.21.15.44:0/2122807322 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b001b050 con 0x562be42e3d80 2024-04-02T13:29:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9a17fa640 1 -- 172.21.15.44:0/2122807322 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9b0025070 con 0x562be42e3d80 2024-04-02T13:29:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.966+0000 7fb9a17fa640 1 -- 172.21.15.44:0/2122807322 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0014de0 con 0x562be42e3d80 2024-04-02T13:29:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.967+0000 7fb9c4670740 1 -- 172.21.15.44:0/2122807322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 msgr2=0x562be42e6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.967+0000 7fb9c4670740 1 --2- 172.21.15.44:0/2122807322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 0x562be42e6730 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000f9e0 tx=0x7fb9b0012000 comp rx=0 tx=0).stop 2024-04-02T13:29:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.967+0000 7fb9c4670740 1 -- 172.21.15.44:0/2122807322 shutdown_connections 2024-04-02T13:29:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.967+0000 7fb9c4670740 1 --2- 172.21.15.44:0/2122807322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42b6e80 0x562be42e6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.967+0000 7fb9c4670740 1 --2- 172.21.15.44:0/2122807322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 0x562be42e6730 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.967+0000 7fb9c4670740 1 --2- 172.21.15.44:0/2122807322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42e39a0 0x562be42b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.967+0000 7fb9c4670740 1 -- 172.21.15.44:0/2122807322 >> 172.21.15.44:0/2122807322 conn(0x562be42b8bb0 msgr2=0x562be4301730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:41.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.967+0000 7fb9c4670740 1 -- 172.21.15.44:0/2122807322 shutdown_connections 2024-04-02T13:29:41.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.967+0000 7fb9c4670740 1 -- 172.21.15.44:0/2122807322 wait complete. 2024-04-02T13:29:41.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.968+0000 7fb9c4670740 1 -- start start 2024-04-02T13:29:41.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.968+0000 7fb9c4670740 1 Processor -- start 2024-04-02T13:29:41.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.968+0000 7fb9c4670740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 0x562be42e93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.968+0000 7fb9c4670740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42e3d80 0x562be42e9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.968+0000 7fb9c4670740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42d4dd0 0x562be42fbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.968+0000 7fb9c4670740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562be42bb150 con 0x562be42e39a0 2024-04-02T13:29:41.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9c4670740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562be41dbd30 con 0x562be42e3d80 2024-04-02T13:29:41.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9c4670740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x562be42ba540 con 0x562be42d4dd0 2024-04-02T13:29:41.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9c4670740 10 client.0 _ll_register_callbacks cb 0x562be41db720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9bb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 0x562be42e93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9bb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 0x562be42e93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40306/0 (socket says 172.21.15.44:40306) 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9bb7fe640 1 -- 172.21.15.44:0/3662232756 learned_addr learned my addr 172.21.15.44:0/3662232756 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9baffd640 1 --2- 172.21.15.44:0/3662232756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42e3d80 0x562be42e9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9bbfff640 1 --2- 172.21.15.44:0/3662232756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42d4dd0 0x562be42fbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9bb7fe640 1 -- 172.21.15.44:0/3662232756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42d4dd0 msgr2=0x562be42fbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9bb7fe640 1 --2- 172.21.15.44:0/3662232756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42d4dd0 0x562be42fbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9bb7fe640 1 -- 172.21.15.44:0/3662232756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42e3d80 msgr2=0x562be42e9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9bb7fe640 1 --2- 172.21.15.44:0/3662232756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42e3d80 0x562be42e9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9bb7fe640 1 -- 172.21.15.44:0/3662232756 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562be3f343f0 con 0x562be42e39a0 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9bb7fe640 1 --2- 172.21.15.44:0/3662232756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 0x562be42e93c0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac07b6f0 tx=0x7fb9ac07b640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a24813d3cb13757 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb99a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb99a7fc640 1 -- 172.21.15.44:0/3662232756 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac091030 con 0x562be42e39a0 2024-04-02T13:29:41.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb99a7fc640 1 -- 172.21.15.44:0/3662232756 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7fb9ac092070 con 0x562be42e39a0 2024-04-02T13:29:41.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9c4670740 1 -- 172.21.15.44:0/3662232756 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9b0025070 con 0x562be42e39a0 2024-04-02T13:29:41.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb99a7fc640 1 -- 172.21.15.44:0/3662232756 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fb9ac09a050 con 0x562be42e39a0 2024-04-02T13:29:41.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9c4670740 10 client.9150 fetch_fsmap learned FSMap version 82 2024-04-02T13:29:41.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.969+0000 7fb9c4670740 1 -- 172.21.15.44:0/3662232756 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) -- 0x562be3ef40f0 con 0x562be42e39a0 2024-04-02T13:29:41.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.970+0000 7fb99a7fc640 1 -- 172.21.15.44:0/3662232756 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9b0025070 con 0x562be42e39a0 2024-04-02T13:29:41.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.970+0000 7fb99a7fc640 1 -- 172.21.15.44:0/3662232756 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 82) ==== 3494+0+0 (secure 0 0 0) 0x7fb9ac031070 con 0x562be42e39a0 2024-04-02T13:29:41.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.970+0000 7fb9c4670740 10 client.9150 fetch_fsmap finished waiting for FSMap version 82 2024-04-02T13:29:41.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.970+0000 7fb9c4670740 10 client.9150 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-02T13:29:41.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.970+0000 7fb9c4670740 20 client.9150 populate_metadata read hostname 'smithi044' 2024-04-02T13:29:41.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.970+0000 7fb9c4670740 1 --2- 172.21.15.44:0/3662232756 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x562be4314f50 0x562be43153c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:41.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.970+0000 7fb9c4670740 4 client.9150 mds_command: new command op to 8760 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:29:41.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.970+0000 7fb9c4670740 1 -- 172.21.15.44:0/3662232756 --> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562be422dd70 con 0x562be4314f50 2024-04-02T13:29:41.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.970+0000 7fb9baffd640 1 --2- 172.21.15.44:0/3662232756 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x562be4314f50 0x562be43153c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:41.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.970+0000 7fb9baffd640 1 --2- 172.21.15.44:0/3662232756 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x562be4314f50 0x562be43153c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6138ab001a0de6b server_cookie=56fbf0ab90fcb3e5 in_seq=0 out_seq=0 2024-04-02T13:29:41.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.970+0000 7fb99a7fc640 10 client.9150 ms_handle_connect on v2:172.21.15.44:6833/3259420283 2024-04-02T13:29:41.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb99a7fc640 1 -- 172.21.15.44:0/3662232756 <== mds.0 v2:172.21.15.44:6833/3259420283 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562be422dd70 con 0x562be4314f50 2024-04-02T13:29:41.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb99a7fc640 10 client.9150 handle_command_reply: tid=0 2024-04-02T13:29:41.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb9c4670740 1 client.9150 shutdown 2024-04-02T13:29:41.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb9c4670740 1 -- 172.21.15.44:0/3662232756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 msgr2=0x562be42e93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:41.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb9c4670740 1 --2- 172.21.15.44:0/3662232756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 0x562be42e93c0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac07b6f0 tx=0x7fb9ac07b640 comp rx=0 tx=0).stop 2024-04-02T13:29:41.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb9c4670740 1 -- 172.21.15.44:0/3662232756 shutdown_connections 2024-04-02T13:29:41.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb9c4670740 1 --2- 172.21.15.44:0/3662232756 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x562be4314f50 0x562be43153c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb9c4670740 1 --2- 172.21.15.44:0/3662232756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42d4dd0 0x562be42fbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb99a7fc640 0 client.9150 ms_handle_reset on v2:172.21.15.44:6833/3259420283 2024-04-02T13:29:41.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb9c4670740 1 --2- 172.21.15.44:0/3662232756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42e3d80 0x562be42e9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb9c4670740 1 --2- 172.21.15.44:0/3662232756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 0x562be42e93c0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:41.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb9c4670740 1 -- 172.21.15.44:0/3662232756 >> 172.21.15.44:0/3662232756 conn(0x562be42b6e80 msgr2=0x562be42e6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:41.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb9c4670740 1 -- 172.21.15.44:0/3662232756 shutdown_connections 2024-04-02T13:29:41.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb9c4670740 1 -- 172.21.15.44:0/3662232756 wait complete. 2024-04-02T13:29:41.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.971+0000 7fb9c4670740 20 client.9150 trim_cache size 0 max 16384 2024-04-02T13:29:41.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.989+0000 7fb9c4670740 1 Processor -- start 2024-04-02T13:29:41.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.989+0000 7fb9c4670740 1 -- start start 2024-04-02T13:29:42.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.989+0000 7fb9c4670740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42d4dd0 0x562be43040d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:42.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.989+0000 7fb9c4670740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 0x562be4304610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.989+0000 7fb9c4670740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42e3d80 0x562be42e9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.989+0000 7fb9c4670740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc200cb0 con 0x562be42e39a0 2024-04-02T13:29:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.989+0000 7fb9c4670740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc1ffc90 con 0x562be42e3d80 2024-04-02T13:29:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.989+0000 7fb9c4670740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb9bc09f950 con 0x562be42d4dd0 2024-04-02T13:29:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.990+0000 7fb9baffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 0x562be4304610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.990+0000 7fb9baffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 0x562be4304610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40310/0 (socket says 172.21.15.44:40310) 2024-04-02T13:29:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.990+0000 7fb9baffd640 1 -- 172.21.15.44:0/1959228560 learned_addr learned my addr 172.21.15.44:0/1959228560 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.990+0000 7fb9baffd640 1 -- 172.21.15.44:0/1959228560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42d4dd0 msgr2=0x562be43040d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.990+0000 7fb9baffd640 1 --2- 172.21.15.44:0/1959228560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42d4dd0 0x562be43040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.990+0000 7fb9baffd640 1 -- 172.21.15.44:0/1959228560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42e3d80 msgr2=0x562be42e9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.990+0000 7fb9baffd640 1 --2- 172.21.15.44:0/1959228560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42e3d80 0x562be42e9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.990+0000 7fb9baffd640 1 -- 172.21.15.44:0/1959228560 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562be42b8940 con 0x562be42e39a0 2024-04-02T13:29:42.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.990+0000 7fb9baffd640 1 --2- 172.21.15.44:0/1959228560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 0x562be4304610 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009780 tx=0x7fb9b001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f6cfbd79a6c1fb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:42.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.990+0000 7fb9a0ff9640 1 -- 172.21.15.44:0/1959228560 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b003a070 con 0x562be42e39a0 2024-04-02T13:29:42.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.990+0000 7fb9a0ff9640 1 -- 172.21.15.44:0/1959228560 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9b0025040 con 0x562be42e39a0 2024-04-02T13:29:42.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.990+0000 7fb9a0ff9640 1 -- 172.21.15.44:0/1959228560 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b001b050 con 0x562be42e39a0 2024-04-02T13:29:42.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.991+0000 7fb9c4670740 1 -- 172.21.15.44:0/1959228560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 msgr2=0x562be4304610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:42.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.991+0000 7fb9c4670740 1 --2- 172.21.15.44:0/1959228560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 0x562be4304610 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009780 tx=0x7fb9b001e000 comp rx=0 tx=0).stop 2024-04-02T13:29:42.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.991+0000 7fb9c4670740 1 -- 172.21.15.44:0/1959228560 shutdown_connections 2024-04-02T13:29:42.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.991+0000 7fb9c4670740 1 --2- 172.21.15.44:0/1959228560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42e3d80 0x562be42e9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.991+0000 7fb9c4670740 1 --2- 172.21.15.44:0/1959228560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e39a0 0x562be4304610 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.991+0000 7fb9c4670740 1 --2- 172.21.15.44:0/1959228560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42d4dd0 0x562be43040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.991+0000 7fb9c4670740 1 -- 172.21.15.44:0/1959228560 >> 172.21.15.44:0/1959228560 conn(0x562be42b6e80 msgr2=0x562be42bd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:42.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.991+0000 7fb9c4670740 1 -- 172.21.15.44:0/1959228560 shutdown_connections 2024-04-02T13:29:42.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.991+0000 7fb9c4670740 1 -- 172.21.15.44:0/1959228560 wait complete. 2024-04-02T13:29:42.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.991+0000 7fb9c4670740 1 -- start start 2024-04-02T13:29:42.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.991+0000 7fb9c4670740 1 Processor -- start 2024-04-02T13:29:42.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.992+0000 7fb9c4670740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42d4dd0 0x562be437afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:42.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.992+0000 7fb9c4670740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42e39a0 0x562be4304c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:42.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.992+0000 7fb9c4670740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 0x562be4305180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:42.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.992+0000 7fb9c4670740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562be4313c60 con 0x562be42e3d80 2024-04-02T13:29:42.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.992+0000 7fb9c4670740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc1ff6d0 con 0x562be42d4dd0 2024-04-02T13:29:42.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.992+0000 7fb9c4670740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x562be422dd70 con 0x562be42e39a0 2024-04-02T13:29:42.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.992+0000 7fb9c4670740 10 client.0 _ll_register_callbacks cb 0x7fb9ac07b0c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:29:42.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.992+0000 7fb9bbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 0x562be4305180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:42.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.992+0000 7fb9bbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 0x562be4305180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40322/0 (socket says 172.21.15.44:40322) 2024-04-02T13:29:42.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.992+0000 7fb9bbfff640 1 -- 172.21.15.44:0/3386379147 learned_addr learned my addr 172.21.15.44:0/3386379147 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:29:42.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.992+0000 7fb9baffd640 1 --2- 172.21.15.44:0/3386379147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42e39a0 0x562be4304c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:42.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.992+0000 7fb9bb7fe640 1 --2- 172.21.15.44:0/3386379147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42d4dd0 0x562be437afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:42.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9bbfff640 1 -- 172.21.15.44:0/3386379147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42e39a0 msgr2=0x562be4304c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:42.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9bbfff640 1 --2- 172.21.15.44:0/3386379147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42e39a0 0x562be4304c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9bbfff640 1 -- 172.21.15.44:0/3386379147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42d4dd0 msgr2=0x562be437afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9bbfff640 1 --2- 172.21.15.44:0/3386379147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42d4dd0 0x562be437afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9bbfff640 1 -- 172.21.15.44:0/3386379147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562be437b520 con 0x562be42e3d80 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9bbfff640 1 --2- 172.21.15.44:0/3386379147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 0x562be4305180 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x562be42b93c0 tx=0x7fb9b4001890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32543ebd2875908e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb99a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb99a7fc640 1 -- 172.21.15.44:0/3386379147 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4002090 con 0x562be42e3d80 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb99a7fc640 1 -- 172.21.15.44:0/3386379147 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 6101+0+0 (secure 0 0 0) 0x7fb9b40145e0 con 0x562be42e3d80 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9c4670740 1 -- 172.21.15.44:0/3386379147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9b0025040 con 0x562be42e3d80 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb99a7fc640 1 -- 172.21.15.44:0/3386379147 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fb9b4022050 con 0x562be42e3d80 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9c4670740 10 client.9156 fetch_fsmap learned FSMap version 82 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9c4670740 1 -- 172.21.15.44:0/3386379147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) -- 0x562be4304670 con 0x562be42e3d80 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb99a7fc640 1 -- 172.21.15.44:0/3386379147 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9b0025040 con 0x562be42e3d80 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb99a7fc640 1 -- 172.21.15.44:0/3386379147 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 82) ==== 3494+0+0 (secure 0 0 0) 0x7fb9b4025020 con 0x562be42e3d80 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9c4670740 10 client.9156 fetch_fsmap finished waiting for FSMap version 82 2024-04-02T13:29:42.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9c4670740 10 client.9156 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9c4670740 20 client.9156 populate_metadata read hostname 'smithi044' 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9c4670740 1 --2- 172.21.15.44:0/3386379147 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x562be42e2a10 0x562be437ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9c4670740 4 client.9156 mds_command: new command op to 8760 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.993+0000 7fb9c4670740 1 -- 172.21.15.44:0/3386379147 --> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562be42280c0 con 0x562be42e2a10 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9bb7fe640 1 --2- 172.21.15.44:0/3386379147 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x562be42e2a10 0x562be437ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb99a7fc640 10 client.9156 ms_handle_connect on v2:172.21.15.44:6833/3259420283 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9bb7fe640 1 --2- 172.21.15.44:0/3386379147 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x562be42e2a10 0x562be437ecc0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfcedca46cb75ab3 server_cookie=1f03db3b704d332a in_seq=0 out_seq=0 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb99a7fc640 1 -- 172.21.15.44:0/3386379147 <== mds.0 v2:172.21.15.44:6833/3259420283 1 ==== command_reply(tid 0: 0 ) ==== 8+0+260 (crc 0 0 0) 0x562be42280c0 con 0x562be42e2a10 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb99a7fc640 10 client.9156 handle_command_reply: tid=0 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9c4670740 1 client.9156 shutdown 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9c4670740 1 -- 172.21.15.44:0/3386379147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 msgr2=0x562be4305180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9c4670740 1 --2- 172.21.15.44:0/3386379147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 0x562be4305180 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x562be42b93c0 tx=0x7fb9b4001890 comp rx=0 tx=0).stop 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9c4670740 1 -- 172.21.15.44:0/3386379147 shutdown_connections 2024-04-02T13:29:42.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9c4670740 1 --2- 172.21.15.44:0/3386379147 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x562be42e2a10 0x562be437ecc0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9c4670740 1 --2- 172.21.15.44:0/3386379147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562be42e3d80 0x562be4305180 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9c4670740 1 --2- 172.21.15.44:0/3386379147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562be42e39a0 0x562be4304c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb99a7fc640 0 client.9156 ms_handle_reset on v2:172.21.15.44:6833/3259420283 2024-04-02T13:29:42.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9c4670740 1 --2- 172.21.15.44:0/3386379147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562be42d4dd0 0x562be437afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9c4670740 1 -- 172.21.15.44:0/3386379147 >> 172.21.15.44:0/3386379147 conn(0x562be42b6e80 msgr2=0x562be42fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:42.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9c4670740 1 -- 172.21.15.44:0/3386379147 shutdown_connections 2024-04-02T13:29:42.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9c4670740 1 -- 172.21.15.44:0/3386379147 wait complete. 2024-04-02T13:29:42.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.994+0000 7fb9c4670740 20 client.9156 trim_cache size 0 max 16384 2024-04-02T13:29:42.011 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:29:42.012 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T13:29:42.012 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-02T13:29:42.012 INFO:teuthology.orchestra.run.smithi044.stdout: "a0fedb08-35ef-4301-8fc9-b3d05ed842a9": { 2024-04-02T13:29:42.012 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-02T13:29:42.012 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "a0fedb08-35ef-4301-8fc9-b3d05ed842a9", 2024-04-02T13:29:42.012 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,repair" 2024-04-02T13:29:42.012 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:42.012 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:29:42.012 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:29:42.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.995+0000 7fb9c1a8f640 1 -- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb9a806c920 msgr2=0x7fb9a806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:42.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.995+0000 7fb9c1a8f640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb9a806c920 0x7fb9a806ede0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0006fd0 tx=0x7fb9b0008040 comp rx=0 tx=0).stop 2024-04-02T13:29:42.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.995+0000 7fb9c1a8f640 1 -- 172.21.15.44:0/2156250381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9bc0cec60 msgr2=0x7fb9bc15a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:29:42.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.995+0000 7fb9c1a8f640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9bc0cec60 0x7fb9bc15a550 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac009890 tx=0x7fb9ac0046c0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.995+0000 7fb9c1a8f640 1 -- 172.21.15.44:0/2156250381 shutdown_connections 2024-04-02T13:29:42.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.995+0000 7fb9c1a8f640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb9a806c920 0x7fb9a806ede0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.995+0000 7fb9c1a8f640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9bc15e5b0 0x7fb9bc0a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.995+0000 7fb9c1a8f640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9bc15e1d0 0x7fb9bc0a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.995+0000 7fb9c1a8f640 1 --2- 172.21.15.44:0/2156250381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9bc0cec60 0x7fb9bc15a550 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:29:42.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.995+0000 7fb9c1a8f640 1 -- 172.21.15.44:0/2156250381 >> 172.21.15.44:0/2156250381 conn(0x7fb9bc09d3b0 msgr2=0x7fb9bc0b9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:29:42.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.995+0000 7fb9c1a8f640 1 -- 172.21.15.44:0/2156250381 shutdown_connections 2024-04-02T13:29:42.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:41.995+0000 7fb9c1a8f640 1 -- 172.21.15.44:0/2156250381 wait complete. 2024-04-02T13:29:42.015 INFO:tasks.cephfs.filesystem:scrub status for tag:a0fedb08-35ef-4301-8fc9-b3d05ed842a9 - {'path': '/', 'tag': 'a0fedb08-35ef-4301-8fc9-b3d05ed842a9', 'options': 'recursive,repair'} 2024-04-02T13:29:42.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:42.641+0000 7f34297fa640 1 -- 172.21.15.44:0/259763857 <== mon.0 v2:172.21.15.44:3300/0 7 ==== log(4 entries from seq 1 at 2024-04-02T13:29:41.833709+0000) ==== 695+0+0 (secure 0 0 0) 0x7f341c0317d0 con 0x7f342c0cf320 2024-04-02T13:29:42.648 INFO:teuthology.orchestra.run.smithi044.stdout: cluster: 2024-04-02T13:29:42.649 INFO:teuthology.orchestra.run.smithi044.stdout: id: 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:29:42.649 INFO:teuthology.orchestra.run.smithi044.stdout: health: HEALTH_OK 2024-04-02T13:29:42.649 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:42.649 INFO:teuthology.orchestra.run.smithi044.stdout: services: 2024-04-02T13:29:42.649 INFO:teuthology.orchestra.run.smithi044.stdout: mon: 3 daemons, quorum a,b,c (age 9m) 2024-04-02T13:29:42.650 INFO:teuthology.orchestra.run.smithi044.stdout: mgr: x(active, since 9m), standbys: y 2024-04-02T13:29:42.650 INFO:teuthology.orchestra.run.smithi044.stdout: mds: 1/1 daemons up, 3 standby 2024-04-02T13:29:42.650 INFO:teuthology.orchestra.run.smithi044.stdout: osd: 8 osds: 8 up (since 9m), 8 in (since 9m) 2024-04-02T13:29:42.650 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:42.650 INFO:teuthology.orchestra.run.smithi044.stdout: data: 2024-04-02T13:29:42.651 INFO:teuthology.orchestra.run.smithi044.stdout: volumes: 1/1 healthy 2024-04-02T13:29:42.651 INFO:teuthology.orchestra.run.smithi044.stdout: pools: 3 pools, 97 pgs 2024-04-02T13:29:42.651 INFO:teuthology.orchestra.run.smithi044.stdout: objects: 30 objects, 19 MiB 2024-04-02T13:29:42.651 INFO:teuthology.orchestra.run.smithi044.stdout: usage: 291 MiB used, 720 GiB / 720 GiB avail 2024-04-02T13:29:42.652 INFO:teuthology.orchestra.run.smithi044.stdout: pgs: 97 active+clean 2024-04-02T13:29:42.652 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:42.652 INFO:teuthology.orchestra.run.smithi044.stdout: io: 2024-04-02T13:29:42.652 INFO:teuthology.orchestra.run.smithi044.stdout: client: 85 B/s rd, 1.0 MiB/s wr, 0 op/s rd, 1 op/s wr 2024-04-02T13:29:42.652 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:42.653 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:29:42.653 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:29:41.833709+0000 mds.a [INF] scrub queued for path: / 2024-04-02T13:29:42.653 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:29:41.833714+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-04-02T13:29:42.653 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:29:41.833721+0000 mds.a [INF] scrub summary: active paths [/] 2024-04-02T13:29:42.653 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:29:41.834852+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-04-02T13:29:47.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:29:47.652+0000 7f34297fa640 1 -- 172.21.15.44:0/259763857 <== mon.0 v2:172.21.15.44:3300/0 8 ==== log(1 entries from seq 5 at 2024-04-02T13:29:46.576001+0000) ==== 189+0+0 (secure 0 0 0) 0x7f341c0ac090 con 0x7f342c0cf320 2024-04-02T13:29:47.657 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:29:46.576001+0000 mds.a [INF] scrub summary: idle 2024-04-02T13:29:53.560 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:29:53.563 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:30:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:00.674+0000 7f34297fa640 1 -- 172.21.15.44:0/259763857 <== mon.0 v2:172.21.15.44:3300/0 9 ==== log(1 entries from seq 1137 at 2024-04-02T13:30:00.000155+0000) ==== 187+0+0 (secure 0 0 0) 0x7f341c039e10 con 0x7f342c0cf320 2024-04-02T13:30:00.678 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:30:00.000155+0000 mon.a [INF] overall HEALTH_OK 2024-04-02T13:30:09.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:09.685+0000 7f34297fa640 1 -- 172.21.15.44:0/259763857 <== mon.0 v2:172.21.15.44:3300/0 10 ==== log(6 entries from seq 146 at 2024-04-02T13:30:09.649561+0000) ==== 1663+0+0 (secure 0 0 0) 0x7f341c0ae000 con 0x7f342c0cf320 2024-04-02T13:30:10.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:10.700+0000 7f34297fa640 1 -- 172.21.15.44:0/259763857 <== mon.0 v2:172.21.15.44:3300/0 11 ==== log(3 entries from seq 1141 at 2024-04-02T13:30:09.691617+0000) ==== 819+0+0 (secure 0 0 0) 0x7f341c079820 con 0x7f342c0cf320 2024-04-02T13:30:12.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:30:12.017 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-02T13:30:12.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.132+0000 7fb7ddba3640 1 Processor -- start 2024-04-02T13:30:12.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.132+0000 7fb7ddba3640 1 -- start start 2024-04-02T13:30:12.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.133+0000 7fb7ddba3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb7d80d07b0 0x7fb7d80ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.133+0000 7fb7ddba3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7d80cf400 0x7fb7d815c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.133+0000 7fb7ddba3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7d8160b10 con 0x7fb7d81565d0 2024-04-02T13:30:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.133+0000 7fb7ddba3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb7d80cf8b0 con 0x7fb7d80d07b0 2024-04-02T13:30:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.133+0000 7fb7ddba3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb7d80cfad0 con 0x7fb7d80cf400 2024-04-02T13:30:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.133+0000 7fb7d77fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb7d81565d0 0x7fb7d80d00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48524/0 (socket says 172.21.15.44:48524) 2024-04-02T13:30:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.133+0000 7fb7d77fe640 1 -- 172.21.15.44:0/2408385748 learned_addr learned my addr 172.21.15.44:0/2408385748 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.133+0000 7fb7d7fff640 1 --2- 172.21.15.44:0/2408385748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7d80cf400 0x7fb7d815c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.133+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/2408385748 >> v2:172.21.15.69:3300/0 conn(0x7fb7d80d07b0 0x7fb7d80ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.133+0000 7fb7d67fc640 1 -- 172.21.15.44:0/2408385748 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1981994014 0 0) 0x7fb7d8160b10 con 0x7fb7d81565d0 2024-04-02T13:30:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.133+0000 7fb7d67fc640 1 -- 172.21.15.44:0/2408385748 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7c0003520 con 0x7fb7d81565d0 2024-04-02T13:30:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.134+0000 7fb7d7fff640 1 -- 172.21.15.44:0/2408385748 >> v2:172.21.15.69:3300/0 conn(0x7fb7d80d07b0 msgr2=0x7fb7d80ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.134+0000 7fb7d7fff640 1 --2- 172.21.15.44:0/2408385748 >> v2:172.21.15.69:3300/0 conn(0x7fb7d80d07b0 0x7fb7d80ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.134+0000 7fb7d7fff640 1 -- 172.21.15.44:0/2408385748 >> v1:172.21.15.44:6789/0 conn(0x7fb7d81565d0 legacy=0x7fb7d80d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.134+0000 7fb7d7fff640 1 -- 172.21.15.44:0/2408385748 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d815fd50 con 0x7fb7d80cf400 2024-04-02T13:30:12.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.134+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/2408385748 >> v2:172.21.15.69:3300/0 conn(0x7fb7d80d07b0 0x7fb7d80ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:30:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.134+0000 7fb7d7fff640 1 --2- 172.21.15.44:0/2408385748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7d80cf400 0x7fb7d815c7a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009d90 tx=0x7fb7d002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55c156c7394c8abb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.135+0000 7fb7d67fc640 1 -- 172.21.15.44:0/2408385748 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0039070 con 0x7fb7d80cf400 2024-04-02T13:30:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.135+0000 7fb7d67fc640 1 -- 172.21.15.44:0/2408385748 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7d002cbc0 con 0x7fb7d80cf400 2024-04-02T13:30:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.135+0000 7fb7d67fc640 1 -- 172.21.15.44:0/2408385748 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0034440 con 0x7fb7d80cf400 2024-04-02T13:30:12.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.135+0000 7fb7ddba3640 1 -- 172.21.15.44:0/2408385748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7d80cf400 msgr2=0x7fb7d815c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.135+0000 7fb7ddba3640 1 --2- 172.21.15.44:0/2408385748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7d80cf400 0x7fb7d815c7a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009d90 tx=0x7fb7d002be10 comp rx=0 tx=0).stop 2024-04-02T13:30:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.135+0000 7fb7ddba3640 1 -- 172.21.15.44:0/2408385748 shutdown_connections 2024-04-02T13:30:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.135+0000 7fb7ddba3640 1 --2- 172.21.15.44:0/2408385748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7d80cf400 0x7fb7d815c7a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.135+0000 7fb7ddba3640 1 --2- 172.21.15.44:0/2408385748 >> v2:172.21.15.69:3300/0 conn(0x7fb7d80d07b0 0x7fb7d80ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.135+0000 7fb7ddba3640 1 -- 172.21.15.44:0/2408385748 >> 172.21.15.44:0/2408385748 conn(0x7fb7d806b920 msgr2=0x7fb7d806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.135+0000 7fb7ddba3640 1 -- 172.21.15.44:0/2408385748 shutdown_connections 2024-04-02T13:30:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.135+0000 7fb7ddba3640 1 -- 172.21.15.44:0/2408385748 wait complete. 2024-04-02T13:30:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.136+0000 7fb7ddba3640 1 Processor -- start 2024-04-02T13:30:12.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.137+0000 7fb7ddba3640 1 -- start start 2024-04-02T13:30:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.137+0000 7fb7ddba3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7d80cf400 0x7fb7d806e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.137+0000 7fb7ddba3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d80d07b0 0x7fb7d80b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.137+0000 7fb7ddba3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7d81565d0 0x7fb7d814a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.137+0000 7fb7ddba3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d81605f0 con 0x7fb7d80d07b0 2024-04-02T13:30:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.137+0000 7fb7ddba3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d8160310 con 0x7fb7d81565d0 2024-04-02T13:30:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.137+0000 7fb7ddba3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb7d81569b0 con 0x7fb7d80cf400 2024-04-02T13:30:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.137+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d80d07b0 0x7fb7d80b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.137+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d80d07b0 0x7fb7d80b0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47592/0 (socket says 172.21.15.44:47592) 2024-04-02T13:30:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.137+0000 7fb7d6ffd640 1 -- 172.21.15.44:0/3240534771 learned_addr learned my addr 172.21.15.44:0/3240534771 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.138+0000 7fb7d7fff640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7d81565d0 0x7fb7d814a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.138+0000 7fb7d77fe640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7d80cf400 0x7fb7d806e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.138+0000 7fb7d6ffd640 1 -- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7d80cf400 msgr2=0x7fb7d806e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.138+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7d80cf400 0x7fb7d806e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.138+0000 7fb7d6ffd640 1 -- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7d81565d0 msgr2=0x7fb7d814a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.138+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7d81565d0 0x7fb7d814a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.138+0000 7fb7d6ffd640 1 -- 172.21.15.44:0/3240534771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d80ca150 con 0x7fb7d80d07b0 2024-04-02T13:30:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.138+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d80d07b0 0x7fb7d80b0c10 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00e960 tx=0x7fb7cc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.138+0000 7fb7bffff640 1 -- 172.21.15.44:0/3240534771 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc017070 con 0x7fb7d80d07b0 2024-04-02T13:30:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.138+0000 7fb7bffff640 1 -- 172.21.15.44:0/3240534771 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7cc00ca00 con 0x7fb7d80d07b0 2024-04-02T13:30:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.138+0000 7fb7bffff640 1 -- 172.21.15.44:0/3240534771 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc00cdd0 con 0x7fb7d80d07b0 2024-04-02T13:30:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.138+0000 7fb7ddba3640 1 -- 172.21.15.44:0/3240534771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7d80ca430 con 0x7fb7d80d07b0 2024-04-02T13:30:12.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.138+0000 7fb7ddba3640 1 -- 172.21.15.44:0/3240534771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7d80ca9f0 con 0x7fb7d80d07b0 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.140+0000 7fb7bffff640 1 -- 172.21.15.44:0/3240534771 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb7cc01a070 con 0x7fb7d80d07b0 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.140+0000 7fb7bffff640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb7c406c8f0 0x7fb7c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.140+0000 7fb7bffff640 1 -- 172.21.15.44:0/3240534771 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6188+0+0 (secure 0 0 0) 0x7fb7cc08bb10 con 0x7fb7d80d07b0 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.140+0000 7fb7d77fe640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb7c406c8f0 0x7fb7c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.141+0000 7fb7d77fe640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb7c406c8f0 0x7fb7c406edb0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005880 tx=0x7fb7c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.141+0000 7fb7e0781740 1 Processor -- start 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.141+0000 7fb7e0781740 1 -- start start 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7e0781740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5625467859a0 0x562546758940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7e0781740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 0x562546788730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7e0781740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562546758e80 0x562546788c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7e0781740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5625466c2830 con 0x562546785d80 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7e0781740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56254666a820 con 0x5625467859a0 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7e0781740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5625466be7e0 con 0x562546758e80 2024-04-02T13:30:12.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 0x562546788730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 0x562546788730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47600/0 (socket says 172.21.15.44:47600) 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7d6ffd640 1 -- 172.21.15.44:0/3795592565 learned_addr learned my addr 172.21.15.44:0/3795592565 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7d77fe640 1 --2- 172.21.15.44:0/3795592565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5625467859a0 0x562546758940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7d7fff640 1 --2- 172.21.15.44:0/3795592565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562546758e80 0x562546788c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7d6ffd640 1 -- 172.21.15.44:0/3795592565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562546758e80 msgr2=0x562546788c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/3795592565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562546758e80 0x562546788c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7d6ffd640 1 -- 172.21.15.44:0/3795592565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5625467859a0 msgr2=0x562546758940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/3795592565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5625467859a0 0x562546758940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.142+0000 7fb7d6ffd640 1 -- 172.21.15.44:0/3795592565 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625464a07d0 con 0x562546785d80 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.143+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/3795592565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 0x562546788730 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc089e00 tx=0x7fb7cc0591a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b7a11036816b1a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.143+0000 7fb7bd7fa640 1 -- 172.21.15.44:0/3795592565 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc021070 con 0x562546785d80 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.143+0000 7fb7bd7fa640 1 -- 172.21.15.44:0/3795592565 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7cc0577b0 con 0x562546785d80 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.143+0000 7fb7bd7fa640 1 -- 172.21.15.44:0/3795592565 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc055d80 con 0x562546785d80 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.143+0000 7fb7e0781740 1 -- 172.21.15.44:0/3795592565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 msgr2=0x562546788730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.143+0000 7fb7e0781740 1 --2- 172.21.15.44:0/3795592565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 0x562546788730 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc089e00 tx=0x7fb7cc0591a0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.143+0000 7fb7e0781740 1 -- 172.21.15.44:0/3795592565 shutdown_connections 2024-04-02T13:30:12.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.143+0000 7fb7e0781740 1 --2- 172.21.15.44:0/3795592565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x562546758e80 0x562546788c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.143+0000 7fb7e0781740 1 --2- 172.21.15.44:0/3795592565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 0x562546788730 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.143+0000 7fb7e0781740 1 --2- 172.21.15.44:0/3795592565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5625467859a0 0x562546758940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.144+0000 7fb7e0781740 1 -- 172.21.15.44:0/3795592565 >> 172.21.15.44:0/3795592565 conn(0x56254675abb0 msgr2=0x5625467a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.144+0000 7fb7e0781740 1 -- 172.21.15.44:0/3795592565 shutdown_connections 2024-04-02T13:30:12.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.144+0000 7fb7e0781740 1 -- 172.21.15.44:0/3795592565 wait complete. 2024-04-02T13:30:12.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.144+0000 7fb7e0781740 1 -- start start 2024-04-02T13:30:12.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.144+0000 7fb7e0781740 1 Processor -- start 2024-04-02T13:30:12.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7e0781740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 0x56254679d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7e0781740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562546785d80 0x5625467a6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7e0781740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 0x5625467a70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7e0781740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56254675d150 con 0x5625467859a0 2024-04-02T13:30:12.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7e0781740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56254667dd30 con 0x562546785d80 2024-04-02T13:30:12.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7e0781740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x56254675c540 con 0x5625467b65d0 2024-04-02T13:30:12.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7e0781740 10 client.0 _ll_register_callbacks cb 0x56254667d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:30:12.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7d77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 0x56254679d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7d77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 0x56254679d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47604/0 (socket says 172.21.15.44:47604) 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7d77fe640 1 -- 172.21.15.44:0/4242303083 learned_addr learned my addr 172.21.15.44:0/4242303083 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562546785d80 0x5625467a6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7d7fff640 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 0x5625467a70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7d77fe640 1 -- 172.21.15.44:0/4242303083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 msgr2=0x5625467a70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7d77fe640 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 0x5625467a70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7d77fe640 1 -- 172.21.15.44:0/4242303083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562546785d80 msgr2=0x5625467a6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7d77fe640 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562546785d80 0x5625467a6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7d77fe640 1 -- 172.21.15.44:0/4242303083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625463d63f0 con 0x5625467859a0 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7d7fff640 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 0x5625467a70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.145+0000 7fb7d77fe640 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 0x56254679d730 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8015770 tx=0x7fb7c8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f688f0bbd6c9995d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7b67fc640 1 -- 172.21.15.44:0/4242303083 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8013800 con 0x5625467859a0 2024-04-02T13:30:12.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7b67fc640 1 -- 172.21.15.44:0/4242303083 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 6188+0+0 (secure 0 0 0) 0x7fb7c8010040 con 0x5625467859a0 2024-04-02T13:30:12.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7e0781740 1 -- 172.21.15.44:0/4242303083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7cc0577b0 con 0x5625467859a0 2024-04-02T13:30:12.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7b67fc640 1 -- 172.21.15.44:0/4242303083 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fb7c802e050 con 0x5625467859a0 2024-04-02T13:30:12.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7e0781740 10 client.9168 fetch_fsmap learned FSMap version 82 2024-04-02T13:30:12.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7e0781740 1 -- 172.21.15.44:0/4242303083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) -- 0x5625463960f0 con 0x5625467859a0 2024-04-02T13:30:12.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7b67fc640 1 -- 172.21.15.44:0/4242303083 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7cc0577b0 con 0x5625467859a0 2024-04-02T13:30:12.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7b67fc640 1 -- 172.21.15.44:0/4242303083 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 82) ==== 3494+0+0 (secure 0 0 0) 0x7fb7c8031020 con 0x5625467859a0 2024-04-02T13:30:12.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7e0781740 10 client.9168 fetch_fsmap finished waiting for FSMap version 82 2024-04-02T13:30:12.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7e0781740 10 client.9168 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-02T13:30:12.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7e0781740 20 client.9168 populate_metadata read hostname 'smithi044' 2024-04-02T13:30:12.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7e0781740 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x5625467a5960 0x5625467a5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7e0781740 4 client.9168 mds_command: new command op to 8760 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:30:12.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7e0781740 1 -- 172.21.15.44:0/4242303083 --> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625466cfd70 con 0x5625467a5960 2024-04-02T13:30:12.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.146+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x5625467a5960 0x5625467a5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x5625467a5960 0x5625467a5d40 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5873fe4e6cab0f85 server_cookie=b0e48ba9158a9c7d in_seq=0 out_seq=0 2024-04-02T13:30:12.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7b67fc640 10 client.9168 ms_handle_connect on v2:172.21.15.44:6833/3259420283 2024-04-02T13:30:12.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7b67fc640 1 -- 172.21.15.44:0/4242303083 <== mds.0 v2:172.21.15.44:6833/3259420283 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5625466cfd70 con 0x5625467a5960 2024-04-02T13:30:12.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7b67fc640 10 client.9168 handle_command_reply: tid=0 2024-04-02T13:30:12.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7e0781740 1 client.9168 shutdown 2024-04-02T13:30:12.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7e0781740 1 -- 172.21.15.44:0/4242303083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 msgr2=0x56254679d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7e0781740 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 0x56254679d730 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8015770 tx=0x7fb7c8000bc0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7e0781740 1 -- 172.21.15.44:0/4242303083 shutdown_connections 2024-04-02T13:30:12.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7e0781740 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x5625467a5960 0x5625467a5d40 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7e0781740 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 0x5625467a70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7e0781740 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562546785d80 0x5625467a6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7b67fc640 0 client.9168 ms_handle_reset on v2:172.21.15.44:6833/3259420283 2024-04-02T13:30:12.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7e0781740 1 --2- 172.21.15.44:0/4242303083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 0x56254679d730 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7e0781740 1 -- 172.21.15.44:0/4242303083 >> 172.21.15.44:0/4242303083 conn(0x562546758e80 msgr2=0x5625467a4980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7e0781740 1 -- 172.21.15.44:0/4242303083 shutdown_connections 2024-04-02T13:30:12.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.147+0000 7fb7e0781740 1 -- 172.21.15.44:0/4242303083 wait complete. 2024-04-02T13:30:12.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.148+0000 7fb7e0781740 20 client.9168 trim_cache size 0 max 16384 2024-04-02T13:30:12.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.165+0000 7fb7e0781740 1 Processor -- start 2024-04-02T13:30:12.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.165+0000 7fb7e0781740 1 -- start start 2024-04-02T13:30:12.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7e0781740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5625467859a0 0x562546757d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7e0781740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 0x5625467a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7e0781740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 0x5625467a39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7e0781740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d8200ee0 con 0x562546785d80 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7e0781740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d81fff40 con 0x5625467859a0 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7e0781740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb7d8152800 con 0x5625467b65d0 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 0x5625467a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 0x5625467a34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47608/0 (socket says 172.21.15.44:47608) 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7d6ffd640 1 -- 172.21.15.44:0/1695465473 learned_addr learned my addr 172.21.15.44:0/1695465473 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7d77fe640 1 --2- 172.21.15.44:0/1695465473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5625467859a0 0x562546757d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7d6ffd640 1 -- 172.21.15.44:0/1695465473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 msgr2=0x5625467a39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7d7fff640 1 --2- 172.21.15.44:0/1695465473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 0x5625467a39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/1695465473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 0x5625467a39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7d6ffd640 1 -- 172.21.15.44:0/1695465473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5625467859a0 msgr2=0x562546757d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/1695465473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5625467859a0 0x562546757d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.166+0000 7fb7d6ffd640 1 -- 172.21.15.44:0/1695465473 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562546682790 con 0x562546785d80 2024-04-02T13:30:12.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.167+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/1695465473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 0x5625467a34b0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc089dd0 tx=0x7fb7cc05c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c15dda74ec02df9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.167+0000 7fb7bcff9640 1 -- 172.21.15.44:0/1695465473 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc055d80 con 0x562546785d80 2024-04-02T13:30:12.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.167+0000 7fb7bcff9640 1 -- 172.21.15.44:0/1695465473 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7cc05eba0 con 0x562546785d80 2024-04-02T13:30:12.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.167+0000 7fb7bcff9640 1 -- 172.21.15.44:0/1695465473 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc055f20 con 0x562546785d80 2024-04-02T13:30:12.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.167+0000 7fb7e0781740 1 -- 172.21.15.44:0/1695465473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 msgr2=0x5625467a34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.167+0000 7fb7e0781740 1 --2- 172.21.15.44:0/1695465473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 0x5625467a34b0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc089dd0 tx=0x7fb7cc05c000 comp rx=0 tx=0).stop 2024-04-02T13:30:12.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.167+0000 7fb7e0781740 1 -- 172.21.15.44:0/1695465473 shutdown_connections 2024-04-02T13:30:12.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.167+0000 7fb7e0781740 1 --2- 172.21.15.44:0/1695465473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 0x5625467a39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.167+0000 7fb7e0781740 1 --2- 172.21.15.44:0/1695465473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562546785d80 0x5625467a34b0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.167+0000 7fb7e0781740 1 --2- 172.21.15.44:0/1695465473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5625467859a0 0x562546757d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.167+0000 7fb7e0781740 1 -- 172.21.15.44:0/1695465473 >> 172.21.15.44:0/1695465473 conn(0x562546758e80 msgr2=0x56254679a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.167+0000 7fb7e0781740 1 -- 172.21.15.44:0/1695465473 shutdown_connections 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.167+0000 7fb7e0781740 1 -- 172.21.15.44:0/1695465473 wait complete. 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.168+0000 7fb7e0781740 1 -- start start 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.168+0000 7fb7e0781740 1 Processor -- start 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7e0781740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 0x56254679acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7e0781740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562546785d80 0x56254675c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7e0781740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 0x56254675c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7e0781740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d81ff8f0 con 0x5625467859a0 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7e0781740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5625466c7ab0 con 0x562546785d80 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7e0781740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x56254681da00 con 0x5625467b65d0 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7e0781740 10 client.0 _ll_register_callbacks cb 0x7fb7c800fed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7d77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 0x56254679acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7d77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 0x56254679acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47616/0 (socket says 172.21.15.44:47616) 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7d77fe640 1 -- 172.21.15.44:0/1519306365 learned_addr learned my addr 172.21.15.44:0/1519306365 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7d7fff640 1 --2- 172.21.15.44:0/1519306365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 0x56254675c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/1519306365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562546785d80 0x56254675c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7d77fe640 1 -- 172.21.15.44:0/1519306365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 msgr2=0x56254675c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7d77fe640 1 --2- 172.21.15.44:0/1519306365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 0x56254675c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7d77fe640 1 -- 172.21.15.44:0/1519306365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562546785d80 msgr2=0x56254675c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7d77fe640 1 --2- 172.21.15.44:0/1519306365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562546785d80 0x56254675c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7d77fe640 1 -- 172.21.15.44:0/1519306365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562546784460 con 0x5625467859a0 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7d77fe640 1 --2- 172.21.15.44:0/1519306365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 0x56254679acf0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8001770 tx=0x7fb7c800a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8506a090da1583 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.169+0000 7fb7b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7b67fc640 1 -- 172.21.15.44:0/1519306365 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8037060 con 0x5625467859a0 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7b67fc640 1 -- 172.21.15.44:0/1519306365 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 6188+0+0 (secure 0 0 0) 0x7fb7c8004bf0 con 0x5625467859a0 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7e0781740 1 -- 172.21.15.44:0/1519306365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7cc05eba0 con 0x5625467859a0 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7b67fc640 1 -- 172.21.15.44:0/1519306365 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fb7c8013800 con 0x5625467859a0 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7e0781740 10 client.9174 fetch_fsmap learned FSMap version 82 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7e0781740 1 -- 172.21.15.44:0/1519306365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) -- 0x56254681d740 con 0x5625467859a0 2024-04-02T13:30:12.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7b67fc640 1 -- 172.21.15.44:0/1519306365 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7cc05eba0 con 0x5625467859a0 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7b67fc640 1 -- 172.21.15.44:0/1519306365 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 82) ==== 3494+0+0 (secure 0 0 0) 0x7fb7c8031040 con 0x5625467859a0 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7e0781740 10 client.9174 fetch_fsmap finished waiting for FSMap version 82 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7e0781740 10 client.9174 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7e0781740 20 client.9174 populate_metadata read hostname 'smithi044' 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7e0781740 1 --2- 172.21.15.44:0/1519306365 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x56254678b8c0 0x56254681c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7e0781740 4 client.9174 mds_command: new command op to 8760 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7e0781740 1 -- 172.21.15.44:0/1519306365 --> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5625467a67f0 con 0x56254678b8c0 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/1519306365 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x56254678b8c0 0x56254681c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7d6ffd640 1 --2- 172.21.15.44:0/1519306365 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x56254678b8c0 0x56254681c860 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ffa3a53ca2e3abc server_cookie=242b8264b84f0f4 in_seq=0 out_seq=0 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.170+0000 7fb7b67fc640 10 client.9174 ms_handle_connect on v2:172.21.15.44:6833/3259420283 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7b67fc640 1 -- 172.21.15.44:0/1519306365 <== mds.0 v2:172.21.15.44:6833/3259420283 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5625467a67f0 con 0x56254678b8c0 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7b67fc640 10 client.9174 handle_command_reply: tid=0 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7e0781740 1 client.9174 shutdown 2024-04-02T13:30:12.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7e0781740 1 -- 172.21.15.44:0/1519306365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 msgr2=0x56254679acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7e0781740 1 --2- 172.21.15.44:0/1519306365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 0x56254679acf0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8001770 tx=0x7fb7c800a000 comp rx=0 tx=0).stop 2024-04-02T13:30:12.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7e0781740 1 -- 172.21.15.44:0/1519306365 shutdown_connections 2024-04-02T13:30:12.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7e0781740 1 --2- 172.21.15.44:0/1519306365 >> [v2:172.21.15.44:6833/3259420283,v1:172.21.15.44:6834/3259420283] conn(0x56254678b8c0 0x56254681c860 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7e0781740 1 --2- 172.21.15.44:0/1519306365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5625467b65d0 0x56254675c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7e0781740 1 --2- 172.21.15.44:0/1519306365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562546785d80 0x56254675c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7b67fc640 0 client.9174 ms_handle_reset on v2:172.21.15.44:6833/3259420283 2024-04-02T13:30:12.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7e0781740 1 --2- 172.21.15.44:0/1519306365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625467859a0 0x56254679acf0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7e0781740 1 -- 172.21.15.44:0/1519306365 >> 172.21.15.44:0/1519306365 conn(0x562546758e80 msgr2=0x5625467a6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7e0781740 1 -- 172.21.15.44:0/1519306365 shutdown_connections 2024-04-02T13:30:12.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7e0781740 1 -- 172.21.15.44:0/1519306365 wait complete. 2024-04-02T13:30:12.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7e0781740 20 client.9174 trim_cache size 0 max 16384 2024-04-02T13:30:12.179 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:30:12.180 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-02T13:30:12.180 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-02T13:30:12.180 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:30:12.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7ddba3640 1 -- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb7c406c8f0 msgr2=0x7fb7c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7ddba3640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb7c406c8f0 0x7fb7c406edb0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005880 tx=0x7fb7c8009040 comp rx=0 tx=0).stop 2024-04-02T13:30:12.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7ddba3640 1 -- 172.21.15.44:0/3240534771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d80d07b0 msgr2=0x7fb7d80b0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7ddba3640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d80d07b0 0x7fb7d80b0c10 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00e960 tx=0x7fb7cc00ee20 comp rx=0 tx=0).stop 2024-04-02T13:30:12.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7ddba3640 1 -- 172.21.15.44:0/3240534771 shutdown_connections 2024-04-02T13:30:12.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7ddba3640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb7c406c8f0 0x7fb7c406edb0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7ddba3640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7d81565d0 0x7fb7d814a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7ddba3640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d80d07b0 0x7fb7d80b0c10 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7ddba3640 1 --2- 172.21.15.44:0/3240534771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7d80cf400 0x7fb7d806e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.171+0000 7fb7ddba3640 1 -- 172.21.15.44:0/3240534771 >> 172.21.15.44:0/3240534771 conn(0x7fb7d806b920 msgr2=0x7fb7d80b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.172+0000 7fb7ddba3640 1 -- 172.21.15.44:0/3240534771 shutdown_connections 2024-04-02T13:30:12.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.172+0000 7fb7ddba3640 1 -- 172.21.15.44:0/3240534771 wait complete. 2024-04-02T13:30:12.185 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T13:30:12.286 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:30:12.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.388+0000 7f0ace83a640 1 Processor -- start 2024-04-02T13:30:12.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.388+0000 7f0ace83a640 1 -- start start 2024-04-02T13:30:12.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ace83a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0ac80d0c50 0x7f0ac80d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ace83a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ac806ce40 0x7f0ac806d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ace83a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ac80cc600 con 0x7f0ac815e4a0 2024-04-02T13:30:12.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ace83a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0ac80af300 con 0x7f0ac80d0c50 2024-04-02T13:30:12.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ace83a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0ac80cc8e0 con 0x7f0ac806ce40 2024-04-02T13:30:12.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ac77fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0ac80d0c50 0x7f0ac80d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ac77fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0ac80d0c50 0x7f0ac80d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47632/0 (socket says 172.21.15.44:47632) 2024-04-02T13:30:12.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ac7fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0ac815e4a0 0x7f0ac80aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48528/0 (socket says 172.21.15.44:48528) 2024-04-02T13:30:12.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ac77fe640 1 -- 172.21.15.44:0/215812983 learned_addr learned my addr 172.21.15.44:0/215812983 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0accdd7640 1 --2- 172.21.15.44:0/215812983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ac806ce40 0x7f0ac806d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ac77fe640 1 -- 172.21.15.44:0/215812983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ac806ce40 msgr2=0x7f0ac806d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ac77fe640 1 --2- 172.21.15.44:0/215812983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ac806ce40 0x7f0ac806d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ac77fe640 1 -- 172.21.15.44:0/215812983 >> v1:172.21.15.44:6789/0 conn(0x7f0ac815e4a0 legacy=0x7f0ac80aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ac77fe640 1 -- 172.21.15.44:0/215812983 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac80ccbc0 con 0x7f0ac80d0c50 2024-04-02T13:30:12.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.389+0000 7f0ac6ffd640 1 -- 172.21.15.44:0/215812983 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3800938065 0 0) 0x7f0ac80cc600 con 0x7f0ac815e4a0 2024-04-02T13:30:12.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.390+0000 7f0ac77fe640 1 --2- 172.21.15.44:0/215812983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ac80d0c50 0x7f0ac80d10b0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009870 tx=0x7f0ab802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cafa447c3d66a089 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.390+0000 7f0ac6ffd640 1 -- 172.21.15.44:0/215812983 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8034070 con 0x7f0ac80d0c50 2024-04-02T13:30:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.390+0000 7f0ac6ffd640 1 -- 172.21.15.44:0/215812983 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ab8030840 con 0x7f0ac80d0c50 2024-04-02T13:30:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.390+0000 7f0ac6ffd640 1 -- 172.21.15.44:0/215812983 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8030c10 con 0x7f0ac80d0c50 2024-04-02T13:30:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.391+0000 7f0ace83a640 1 -- 172.21.15.44:0/215812983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ac80d0c50 msgr2=0x7f0ac80d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.391+0000 7f0ace83a640 1 --2- 172.21.15.44:0/215812983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ac80d0c50 0x7f0ac80d10b0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009870 tx=0x7f0ab802b980 comp rx=0 tx=0).stop 2024-04-02T13:30:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.391+0000 7f0ace83a640 1 -- 172.21.15.44:0/215812983 shutdown_connections 2024-04-02T13:30:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.391+0000 7f0ace83a640 1 --2- 172.21.15.44:0/215812983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ac806ce40 0x7f0ac806d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.391+0000 7f0ace83a640 1 --2- 172.21.15.44:0/215812983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ac80d0c50 0x7f0ac80d10b0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.391+0000 7f0ace83a640 1 -- 172.21.15.44:0/215812983 >> 172.21.15.44:0/215812983 conn(0x7f0ac809aa20 msgr2=0x7f0ac809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.391+0000 7f0ace83a640 1 -- 172.21.15.44:0/215812983 shutdown_connections 2024-04-02T13:30:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.391+0000 7f0ace83a640 1 -- 172.21.15.44:0/215812983 wait complete. 2024-04-02T13:30:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.392+0000 7f0ace83a640 1 Processor -- start 2024-04-02T13:30:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.392+0000 7f0ace83a640 1 -- start start 2024-04-02T13:30:12.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ace83a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ac806ce40 0x7f0ac81512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ace83a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ac80d0c50 0x7f0ac80b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ace83a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ac815e4a0 0x7f0ac815f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ace83a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac80cd460 con 0x7f0ac806ce40 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ace83a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac80cd180 con 0x7f0ac80d0c50 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ace83a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0ac815e880 con 0x7f0ac815e4a0 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ac7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ac806ce40 0x7f0ac81512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ac7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ac806ce40 0x7f0ac81512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47640/0 (socket says 172.21.15.44:47640) 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ac7fff640 1 -- 172.21.15.44:0/2238484531 learned_addr learned my addr 172.21.15.44:0/2238484531 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ac7fff640 1 -- 172.21.15.44:0/2238484531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ac815e4a0 msgr2=0x7f0ac815f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ac7fff640 1 --2- 172.21.15.44:0/2238484531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ac815e4a0 0x7f0ac815f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ac7fff640 1 -- 172.21.15.44:0/2238484531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ac80d0c50 msgr2=0x7f0ac80b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ac77fe640 1 --2- 172.21.15.44:0/2238484531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ac80d0c50 0x7f0ac80b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ac7fff640 1 --2- 172.21.15.44:0/2238484531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ac80d0c50 0x7f0ac80b05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ac7fff640 1 -- 172.21.15.44:0/2238484531 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac80c76a0 con 0x7f0ac806ce40 2024-04-02T13:30:12.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ac7fff640 1 --2- 172.21.15.44:0/2238484531 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ac806ce40 0x7f0ac81512f0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000c500 tx=0x7f0ab000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ac4ff9640 1 -- 172.21.15.44:0/2238484531 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0018070 con 0x7f0ac806ce40 2024-04-02T13:30:12.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ac4ff9640 1 -- 172.21.15.44:0/2238484531 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ab0007580 con 0x7f0ac806ce40 2024-04-02T13:30:12.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ace83a640 1 -- 172.21.15.44:0/2238484531 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac80c7940 con 0x7f0ac806ce40 2024-04-02T13:30:12.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.393+0000 7f0ac4ff9640 1 -- 172.21.15.44:0/2238484531 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0004a20 con 0x7f0ac806ce40 2024-04-02T13:30:12.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.394+0000 7f0ace83a640 1 -- 172.21.15.44:0/2238484531 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac80c7f00 con 0x7f0ac806ce40 2024-04-02T13:30:12.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.394+0000 7f0ace83a640 1 -- 172.21.15.44:0/2238484531 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ac80684f0 con 0x7f0ac806ce40 2024-04-02T13:30:12.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.395+0000 7f0ac4ff9640 1 -- 172.21.15.44:0/2238484531 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ab0004bc0 con 0x7f0ac806ce40 2024-04-02T13:30:12.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.395+0000 7f0ac4ff9640 1 --2- 172.21.15.44:0/2238484531 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0aa806c8f0 0x7f0aa806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.395+0000 7f0ac4ff9640 1 -- 172.21.15.44:0/2238484531 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6188+0+0 (secure 0 0 0) 0x7f0ab008b060 con 0x7f0ac806ce40 2024-04-02T13:30:12.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.396+0000 7f0ac77fe640 1 --2- 172.21.15.44:0/2238484531 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0aa806c8f0 0x7f0aa806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.396+0000 7f0ac77fe640 1 --2- 172.21.15.44:0/2238484531 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0aa806c8f0 0x7f0aa806edb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8036000 tx=0x7f0ab803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.400+0000 7f0ac4ff9640 1 -- 172.21.15.44:0/2238484531 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0ab0059220 con 0x7f0ac806ce40 2024-04-02T13:30:12.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.627+0000 7f0ace83a640 1 -- 172.21.15.44:0/2238484531 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0ac80aef30 con 0x7f0ac806ce40 2024-04-02T13:30:12.630 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:30:12.628+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:30:12.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.629+0000 7f0ac4ff9640 1 -- 172.21.15.44:0/2238484531 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v83) ==== 161+0+0 (secure 0 0 0) 0x7f0ab0058b20 con 0x7f0ac806ce40 2024-04-02T13:30:12.631 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:30:12.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.631+0000 7f0ace83a640 1 -- 172.21.15.44:0/2238484531 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0aa806c8f0 msgr2=0x7f0aa806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.631+0000 7f0ace83a640 1 --2- 172.21.15.44:0/2238484531 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0aa806c8f0 0x7f0aa806edb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8036000 tx=0x7f0ab803b040 comp rx=0 tx=0).stop 2024-04-02T13:30:12.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.631+0000 7f0ace83a640 1 -- 172.21.15.44:0/2238484531 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ac806ce40 msgr2=0x7f0ac81512f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.631+0000 7f0ace83a640 1 --2- 172.21.15.44:0/2238484531 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ac806ce40 0x7f0ac81512f0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000c500 tx=0x7f0ab000c9c0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.631+0000 7f0ace83a640 1 -- 172.21.15.44:0/2238484531 shutdown_connections 2024-04-02T13:30:12.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.631+0000 7f0ace83a640 1 --2- 172.21.15.44:0/2238484531 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0aa806c8f0 0x7f0aa806edb0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.631+0000 7f0ace83a640 1 --2- 172.21.15.44:0/2238484531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ac815e4a0 0x7f0ac815f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.632+0000 7f0ace83a640 1 --2- 172.21.15.44:0/2238484531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ac80d0c50 0x7f0ac80b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.632+0000 7f0ace83a640 1 --2- 172.21.15.44:0/2238484531 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ac806ce40 0x7f0ac81512f0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.632+0000 7f0ace83a640 1 -- 172.21.15.44:0/2238484531 >> 172.21.15.44:0/2238484531 conn(0x7f0ac809aa20 msgr2=0x7f0ac806dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.632+0000 7f0ace83a640 1 -- 172.21.15.44:0/2238484531 shutdown_connections 2024-04-02T13:30:12.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.632+0000 7f0ace83a640 1 -- 172.21.15.44:0/2238484531 wait complete. 2024-04-02T13:30:12.637 INFO:tasks.ceph.mds.a.smithi044.stderr: -6417> 2024-04-02T13:29:13.530+0000 7fe21e740640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:30:12.646 DEBUG:teuthology.orchestra.run.smithi044:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event splice --inode=1099511627777 summary 2024-04-02T13:30:12.668 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:30:12.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.686+0000 7fd043a48280 1 -- start start 2024-04-02T13:30:12.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.686+0000 7fd043a48280 1 Processor -- start 2024-04-02T13:30:12.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.686+0000 7fd043a48280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a70126cc00 0x55a7013de000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.686+0000 7fd043a48280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a70126d800 0x55a7013de580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.686+0000 7fd043a48280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a70126dc00 0x55a7013deb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.686+0000 7fd043a48280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a701ffe600 con 0x55a70126cc00 2024-04-02T13:30:12.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.686+0000 7fd043a48280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a701ffe480 con 0x55a70126dc00 2024-04-02T13:30:12.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.686+0000 7fd043a48280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a701ffe300 con 0x55a70126d800 2024-04-02T13:30:12.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.686+0000 7fd041a4e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a70126cc00 0x55a7013de000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.686+0000 7fd041a4e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a70126cc00 0x55a7013de000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47676/0 (socket says 172.21.15.44:47676) 2024-04-02T13:30:12.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.686+0000 7fd041a4e640 1 -- 172.21.15.44:0/910159285 learned_addr learned my addr 172.21.15.44:0/910159285 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd04124d640 1 --2- 172.21.15.44:0/910159285 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a70126d800 0x55a7013de580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd04224f640 1 --2- 172.21.15.44:0/910159285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a70126dc00 0x55a7013deb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd041a4e640 1 -- 172.21.15.44:0/910159285 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a70126d800 msgr2=0x55a7013de580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd041a4e640 1 --2- 172.21.15.44:0/910159285 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a70126d800 0x55a7013de580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd041a4e640 1 -- 172.21.15.44:0/910159285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a70126dc00 msgr2=0x55a7013deb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd041a4e640 1 --2- 172.21.15.44:0/910159285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a70126dc00 0x55a7013deb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd041a4e640 1 -- 172.21.15.44:0/910159285 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7012a2780 con 0x55a70126cc00 2024-04-02T13:30:12.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd04224f640 1 --2- 172.21.15.44:0/910159285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a70126dc00 0x55a7013deb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:12.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd041a4e640 1 --2- 172.21.15.44:0/910159285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a70126cc00 0x55a7013de000 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x55a7020813b0 tx=0x55a7020006c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a985ce6a7956cb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd03f249640 1 -- 172.21.15.44:0/910159285 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a701292680 con 0x55a70126cc00 2024-04-02T13:30:12.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd03f249640 1 -- 172.21.15.44:0/910159285 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55a701298a80 con 0x55a70126cc00 2024-04-02T13:30:12.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd043a48280 1 -- 172.21.15.44:0/910159285 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7012a34a0 con 0x55a70126cc00 2024-04-02T13:30:12.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd043a48280 1 -- 172.21.15.44:0/910159285 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55a7012a2000 con 0x55a70126cc00 2024-04-02T13:30:12.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd043a48280 4 waiting for MDS map... 2024-04-02T13:30:12.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd03f249640 1 -- 172.21.15.44:0/910159285 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a701292b60 con 0x55a70126cc00 2024-04-02T13:30:12.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd03f249640 1 -- 172.21.15.44:0/910159285 <== mon.0 v2:172.21.15.44:3300/0 4 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x55a702130600 con 0x55a70126cc00 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd03f249640 1 -- 172.21.15.44:0/910159285 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 83) ==== 2885+0+0 (secure 0 0 0) 0x55a7012c7180 con 0x55a70126cc00 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd043a48280 4 Got MDS map 83 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.687+0000 7fd043a48280 10 main: JournalTool::main 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.688+0000 7fd043a48280 4 main: JournalTool: connecting to RADOS... 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.688+0000 7fd043a48280 1 Processor -- start 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.688+0000 7fd043a48280 1 -- start start 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd043a48280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a702133400 0x55a7013df080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd043a48280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a702133800 0x55a7013df600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd043a48280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a702133c00 0x55a7013dfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd043a48280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a701ffe180 con 0x55a702133c00 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd043a48280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a7012ce480 con 0x55a702133400 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd043a48280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a7012ce900 con 0x55a702133800 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd04124d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a702133c00 0x55a7013dfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd04124d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a702133c00 0x55a7013dfb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47688/0 (socket says 172.21.15.44:47688) 2024-04-02T13:30:12.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd04124d640 1 -- 172.21.15.44:0/3603223665 learned_addr learned my addr 172.21.15.44:0/3603223665 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd04224f640 1 --2- 172.21.15.44:0/3603223665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a702133400 0x55a7013df080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd041a4e640 1 --2- 172.21.15.44:0/3603223665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a702133800 0x55a7013df600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd04124d640 1 -- 172.21.15.44:0/3603223665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a702133800 msgr2=0x55a7013df600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd04124d640 1 --2- 172.21.15.44:0/3603223665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a702133800 0x55a7013df600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd04124d640 1 -- 172.21.15.44:0/3603223665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a702133400 msgr2=0x55a7013df080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd04124d640 1 --2- 172.21.15.44:0/3603223665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a702133400 0x55a7013df080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd04124d640 1 -- 172.21.15.44:0/3603223665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7012a23c0 con 0x55a702133c00 2024-04-02T13:30:12.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd041a4e640 1 --2- 172.21.15.44:0/3603223665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a702133800 0x55a7013df600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd04224f640 1 --2- 172.21.15.44:0/3603223665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a702133400 0x55a7013df080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd04124d640 1 --2- 172.21.15.44:0/3603223665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a702133c00 0x55a7013dfb80 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x55a7020809f0 tx=0x55a7020009c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5cf2fe4136eb4d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd03d245640 1 -- 172.21.15.44:0/3603223665 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a7012929c0 con 0x55a702133c00 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd03d245640 1 -- 172.21.15.44:0/3603223665 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55a701298e00 con 0x55a702133c00 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.689+0000 7fd03d245640 1 -- 172.21.15.44:0/3603223665 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a701292680 con 0x55a702133c00 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.690+0000 7fd043a48280 1 -- 172.21.15.44:0/3603223665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a702133c00 msgr2=0x55a7013dfb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.690+0000 7fd043a48280 1 --2- 172.21.15.44:0/3603223665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a702133c00 0x55a7013dfb80 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x55a7020809f0 tx=0x55a7020009c0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.690+0000 7fd043a48280 1 -- 172.21.15.44:0/3603223665 shutdown_connections 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.690+0000 7fd043a48280 1 --2- 172.21.15.44:0/3603223665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a702133c00 0x55a7013dfb80 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.690+0000 7fd043a48280 1 --2- 172.21.15.44:0/3603223665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a702133800 0x55a7013df600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.690+0000 7fd043a48280 1 --2- 172.21.15.44:0/3603223665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a702133400 0x55a7013df080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.690+0000 7fd043a48280 1 -- 172.21.15.44:0/3603223665 >> 172.21.15.44:0/3603223665 conn(0x55a702133000 msgr2=0x55a701356800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.690+0000 7fd043a48280 1 -- 172.21.15.44:0/3603223665 shutdown_connections 2024-04-02T13:30:12.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.690+0000 7fd043a48280 1 -- 172.21.15.44:0/3603223665 wait complete. 2024-04-02T13:30:12.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.690+0000 7fd043a48280 1 Processor -- start 2024-04-02T13:30:12.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.690+0000 7fd043a48280 1 -- start start 2024-04-02T13:30:12.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd043a48280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a702133400 0x55a7013df080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd043a48280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a702133800 0x55a7013df600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd043a48280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a702133c00 0x55a7013dfb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd043a48280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a701fff200 con 0x55a702133800 2024-04-02T13:30:12.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd043a48280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a701ffe180 con 0x55a702133400 2024-04-02T13:30:12.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd043a48280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55a701fff380 con 0x55a702133c00 2024-04-02T13:30:12.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd041a4e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a702133800 0x55a7013df600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd041a4e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a702133800 0x55a7013df600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47700/0 (socket says 172.21.15.44:47700) 2024-04-02T13:30:12.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd041a4e640 1 -- 172.21.15.44:0/2670199034 learned_addr learned my addr 172.21.15.44:0/2670199034 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd04224f640 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a702133400 0x55a7013df080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd04124d640 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a702133c00 0x55a7013dfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd041a4e640 1 -- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a702133c00 msgr2=0x55a7013dfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd041a4e640 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a702133c00 0x55a7013dfb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd041a4e640 1 -- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a702133400 msgr2=0x55a7013df080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd041a4e640 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a702133400 0x55a7013df080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd041a4e640 1 -- 172.21.15.44:0/2670199034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7012a2780 con 0x55a702133800 2024-04-02T13:30:12.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd04124d640 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a702133c00 0x55a7013dfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:12.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.691+0000 7fd041a4e640 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a702133800 0x55a7013df600 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x55a702123320 tx=0x55a701322cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.692+0000 7fd03ba42640 1 -- 172.21.15.44:0/2670199034 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a701292820 con 0x55a702133800 2024-04-02T13:30:12.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.692+0000 7fd03ba42640 1 -- 172.21.15.44:0/2670199034 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55a701298c40 con 0x55a702133800 2024-04-02T13:30:12.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.692+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a7012a34a0 con 0x55a702133800 2024-04-02T13:30:12.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.692+0000 7fd03ba42640 1 -- 172.21.15.44:0/2670199034 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a7012929c0 con 0x55a702133800 2024-04-02T13:30:12.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.692+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7012a2960 con 0x55a702133800 2024-04-02T13:30:12.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.692+0000 7fd043a48280 4 main: JournalTool: resolving pool 15 2024-04-02T13:30:12.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.693+0000 7fd03ba42640 1 -- 172.21.15.44:0/2670199034 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x55a702128c00 con 0x55a702133800 2024-04-02T13:30:12.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.693+0000 7fd03ba42640 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55a70205b000 0x55a7013e0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.693+0000 7fd03ba42640 1 -- 172.21.15.44:0/2670199034 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x55a702058600 con 0x55a702133800 2024-04-02T13:30:12.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.693+0000 7fd043a48280 4 main: JournalTool: creating IoCtx.. 2024-04-02T13:30:12.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.693+0000 7fd043a48280 4 main: Executing for rank 0 2024-04-02T13:30:12.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.693+0000 7fd043a48280 4 Filtering by inode '1099511627777' 2024-04-02T13:30:12.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.693+0000 7fd043a48280 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55a70205b400 0x55a7013e0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.693+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:1 15.f 15:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e109) -- 0x55a70205b800 con 0x55a70205b400 2024-04-02T13:30:12.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.694+0000 7fd04224f640 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55a70205b000 0x55a7013e0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.694+0000 7fd04124d640 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55a70205b400 0x55a7013e0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.694+0000 7fd04124d640 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55a70205b400 0x55a7013e0680 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.694+0000 7fd04224f640 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55a70205b000 0x55a7013e0100 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x55a7020812f0 tx=0x55a70156cf00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.695+0000 7fd04124d640 1 -- 172.21.15.44:0/2670199034 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x55a701ff0000 con 0x55a70205b400 2024-04-02T13:30:12.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.695+0000 7fd043a48280 4 Pointer 400.00000000 is readable 2024-04-02T13:30:12.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.695+0000 7fd043a48280 4 JournalScanner::scan: reading header object '200.00000000' 2024-04-02T13:30:12.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.695+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:2 15.14 15:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e109) -- 0x55a70205bc00 con 0x55a70205b400 2024-04-02T13:30:12.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.695+0000 7fd04124d640 1 -- 172.21.15.44:0/2670199034 <== osd.6 v2:172.21.15.69:6802/2604158175 2 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv31 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x55a701ff0000 con 0x55a70205b400 2024-04-02T13:30:12.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.695+0000 7fd043a48280 10 Header 0x400000 0x405f0e 0x40628a 2024-04-02T13:30:12.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.695+0000 7fd043a48280 10 Starting journal scan from offset 0x405f0e 2024-04-02T13:30:12.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.695+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:3 15.14 15:2e2fa760:::200.00000001:head [read 24334~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e109) -- 0x55a7015dec00 con 0x55a70205b400 2024-04-02T13:30:12.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.696+0000 7fd04124d640 1 -- 172.21.15.44:0/2670199034 <== osd.6 v2:172.21.15.69:6802/2604158175 3 ==== osd_op_reply(3 200.00000001 [read 24334~892 out=892b] v0'0 uv28 ondisk = 0) ==== 156+0+892 (crc 0 0 0) 0x55a701ff0000 con 0x55a70205b400 2024-04-02T13:30:12.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.696+0000 7fd043a48280 4 Read 0x37c bytes from 200.00000001 gap=0 2024-04-02T13:30:12.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.696+0000 7fd043a48280 10 Parsing data, 0x37c bytes available 2024-04-02T13:30:12.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.696+0000 7fd043a48280 10 Attempting decode at 0x405f0e 2024-04-02T13:30:12.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.696+0000 7fd043a48280 10 Consumed 0x37c bytes 2024-04-02T13:30:12.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.696+0000 7fd043a48280 10 Valid entry at 0x405f0e 2024-04-02T13:30:12.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.696+0000 7fd043a48280 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55a70205b800 0x55a7013e0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.696+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:4 15.1b 15:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e109) -- 0x55a702045400 con 0x55a70205b800 2024-04-02T13:30:12.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.696+0000 7fd04224f640 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55a70205b800 0x55a7013e0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.696+0000 7fd04224f640 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55a70205b800 0x55a7013e0c00 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.708 INFO:teuthology.orchestra.run.smithi044.stdout:Events by type: 2024-04-02T13:30:12.708 INFO:teuthology.orchestra.run.smithi044.stdout:Errors: 0 2024-04-02T13:30:12.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.698+0000 7fd04224f640 1 -- 172.21.15.44:0/2670199034 <== osd.7 v2:172.21.15.69:6824/1292475712 1 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55a701ff0500 con 0x55a70205b800 2024-04-02T13:30:12.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.698+0000 7fd043a48280 4 Reached end of journal objects 2024-04-02T13:30:12.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.698+0000 7fd043a48280 4 Scanned objects, 0 missing, 1 valid 2024-04-02T13:30:12.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.698+0000 7fd043a48280 4 Events scanned, 0 gaps 2024-04-02T13:30:12.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.698+0000 7fd043a48280 4 Found 1 valid events 2024-04-02T13:30:12.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.698+0000 7fd043a48280 4 Selected 0 events events for processing 2024-04-02T13:30:12.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.698+0000 7fd043a48280 4 0 events 2024-04-02T13:30:12.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.698+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55a70205b400 msgr2=0x55a7013e0680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.698+0000 7fd043a48280 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55a70205b400 0x55a7013e0680 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.698+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55a70205b800 msgr2=0x55a7013e0c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.698+0000 7fd043a48280 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55a70205b800 0x55a7013e0c00 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55a70205b000 msgr2=0x55a7013e0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55a70205b000 0x55a7013e0100 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x55a7020812f0 tx=0x55a70156cf00 comp rx=0 tx=0).stop 2024-04-02T13:30:12.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a702133800 msgr2=0x55a7013df600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 --2- 172.21.15.44:0/2670199034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a702133800 0x55a7013df600 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x55a702123320 tx=0x55a701322cc0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd04124d640 1 -- 172.21.15.44:0/2670199034 reap_dead start 2024-04-02T13:30:12.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd04124d640 1 -- 172.21.15.44:0/2670199034 reap_dead start 2024-04-02T13:30:12.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 shutdown_connections 2024-04-02T13:30:12.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 >> 172.21.15.44:0/2670199034 conn(0x55a702133000 msgr2=0x55a701356800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 shutdown_connections 2024-04-02T13:30:12.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 -- 172.21.15.44:0/2670199034 wait complete. 2024-04-02T13:30:12.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 -- 172.21.15.44:0/910159285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a70126cc00 msgr2=0x55a7013de000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 --2- 172.21.15.44:0/910159285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a70126cc00 0x55a7013de000 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x55a7020813b0 tx=0x55a7020006c0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 -- 172.21.15.44:0/910159285 shutdown_connections 2024-04-02T13:30:12.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 --2- 172.21.15.44:0/910159285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a70126dc00 0x55a7013deb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 --2- 172.21.15.44:0/910159285 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a70126d800 0x55a7013de580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 --2- 172.21.15.44:0/910159285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a70126cc00 0x55a7013de000 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 -- 172.21.15.44:0/910159285 >> 172.21.15.44:0/910159285 conn(0x55a70126d000 msgr2=0x55a701357000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 -- 172.21.15.44:0/910159285 shutdown_connections 2024-04-02T13:30:12.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.699+0000 7fd043a48280 1 -- 172.21.15.44:0/910159285 wait complete. 2024-04-02T13:30:12.715 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'event', 'splice', '--inode=1099511627777', 'summary'] in time 0:00:00.069059, result: Events by type: Errors: 0 2024-04-02T13:30:12.715 DEBUG:teuthology.orchestra.run.smithi044:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event splice --inode=1099511627778 summary 2024-04-02T13:30:12.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.790+0000 7f5a5ff3d280 1 -- start start 2024-04-02T13:30:12.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.790+0000 7f5a5ff3d280 1 Processor -- start 2024-04-02T13:30:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.790+0000 7f5a5ff3d280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5e98eac00 0x55e5ea6b6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.790+0000 7f5a5ff3d280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5e98eb800 0x55e5ea6b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.790+0000 7f5a5ff3d280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5e98ebc00 0x55e5ea6b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.790+0000 7f5a5ff3d280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e5ea6c4180 con 0x55e5e98eb800 2024-04-02T13:30:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.790+0000 7f5a5ff3d280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e5ea6c4300 con 0x55e5e98eac00 2024-04-02T13:30:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.790+0000 7f5a5ff3d280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55e5ea6c4480 con 0x55e5e98ebc00 2024-04-02T13:30:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.790+0000 7f5a5dee8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5e98eb800 0x55e5ea6b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5dee8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5e98eb800 0x55e5ea6b6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47718/0 (socket says 172.21.15.44:47718) 2024-04-02T13:30:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5dee8640 1 -- 172.21.15.44:0/320591586 learned_addr learned my addr 172.21.15.44:0/320591586 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5dee8640 1 -- 172.21.15.44:0/320591586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5e98ebc00 msgr2=0x55e5ea6b6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5dee8640 1 --2- 172.21.15.44:0/320591586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5e98ebc00 0x55e5ea6b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5dee8640 1 -- 172.21.15.44:0/320591586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5e98eac00 msgr2=0x55e5ea6b6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5dee8640 1 --2- 172.21.15.44:0/320591586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5e98eac00 0x55e5ea6b6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5dee8640 1 -- 172.21.15.44:0/320591586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5e9920780 con 0x55e5e98eb800 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5dee8640 1 --2- 172.21.15.44:0/320591586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5e98eb800 0x55e5ea6b6580 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x55e5ea71b4a0 tx=0x55e5ea680d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef88322def3a232 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5bee4640 1 -- 172.21.15.44:0/320591586 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55e5e9910680 con 0x55e5e98eb800 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5bee4640 1 -- 172.21.15.44:0/320591586 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55e5e9916700 con 0x55e5e98eb800 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5bee4640 1 -- 172.21.15.44:0/320591586 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55e5e9910b60 con 0x55e5e98eb800 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/320591586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5e99214a0 con 0x55e5e98eb800 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/320591586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55e5e9920000 con 0x55e5e98eb800 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5ff3d280 4 waiting for MDS map... 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5bee4640 1 -- 172.21.15.44:0/320591586 <== mon.0 v2:172.21.15.44:3300/0 4 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x55e5ea7b6600 con 0x55e5e98eb800 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5bee4640 1 -- 172.21.15.44:0/320591586 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 83) ==== 2885+0+0 (secure 0 0 0) 0x55e5e9945180 con 0x55e5e98eb800 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5ff3d280 4 Got MDS map 83 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.791+0000 7f5a5ff3d280 10 main: JournalTool::main 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.792+0000 7f5a5ff3d280 4 main: JournalTool: connecting to RADOS... 2024-04-02T13:30:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.792+0000 7f5a5ff3d280 1 Processor -- start 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.792+0000 7f5a5ff3d280 1 -- start start 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5ff3d280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5ea7b9400 0x55e5ea6b7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5ff3d280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5ea7b9800 0x55e5ea6b7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5ff3d280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5ea7b9c00 0x55e5ea6b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5ff3d280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e5ea6c4600 con 0x55e5ea7b9c00 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5ff3d280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e5ea6c4780 con 0x55e5ea7b9400 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5ff3d280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55e5ea6c4900 con 0x55e5ea7b9800 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5dee8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5ea7b9c00 0x55e5ea6b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5dee8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5ea7b9c00 0x55e5ea6b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47732/0 (socket says 172.21.15.44:47732) 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5dee8640 1 -- 172.21.15.44:0/2359901502 learned_addr learned my addr 172.21.15.44:0/2359901502 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5e6e9640 1 --2- 172.21.15.44:0/2359901502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5ea7b9800 0x55e5ea6b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5eeea640 1 --2- 172.21.15.44:0/2359901502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5ea7b9400 0x55e5ea6b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5dee8640 1 -- 172.21.15.44:0/2359901502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5ea7b9800 msgr2=0x55e5ea6b7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5dee8640 1 --2- 172.21.15.44:0/2359901502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5ea7b9800 0x55e5ea6b7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5dee8640 1 -- 172.21.15.44:0/2359901502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5ea7b9400 msgr2=0x55e5ea6b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5dee8640 1 --2- 172.21.15.44:0/2359901502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5ea7b9400 0x55e5ea6b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5dee8640 1 -- 172.21.15.44:0/2359901502 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5e99203c0 con 0x55e5ea7b9c00 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5e6e9640 1 --2- 172.21.15.44:0/2359901502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5ea7b9800 0x55e5ea6b7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5eeea640 1 --2- 172.21.15.44:0/2359901502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5ea7b9400 0x55e5ea6b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a5dee8640 1 --2- 172.21.15.44:0/2359901502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5ea7b9c00 0x55e5ea6b7b80 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x55e5ea7d42d0 tx=0x55e5ea680120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a45341421c0551 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a59ee0640 1 -- 172.21.15.44:0/2359901502 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55e5e9910820 con 0x55e5ea7b9c00 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a59ee0640 1 -- 172.21.15.44:0/2359901502 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55e5e9916fc0 con 0x55e5ea7b9c00 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.793+0000 7f5a59ee0640 1 -- 172.21.15.44:0/2359901502 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55e5e9910680 con 0x55e5ea7b9c00 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.794+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/2359901502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5ea7b9c00 msgr2=0x55e5ea6b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.794+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/2359901502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5ea7b9c00 0x55e5ea6b7b80 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x55e5ea7d42d0 tx=0x55e5ea680120 comp rx=0 tx=0).stop 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.794+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/2359901502 shutdown_connections 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.794+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/2359901502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5ea7b9c00 0x55e5ea6b7b80 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.794+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/2359901502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5ea7b9800 0x55e5ea6b7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.794+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/2359901502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5ea7b9400 0x55e5ea6b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.794+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/2359901502 >> 172.21.15.44:0/2359901502 conn(0x55e5ea7b9000 msgr2=0x55e5e99d5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.794+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/2359901502 shutdown_connections 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.794+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/2359901502 wait complete. 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.794+0000 7f5a5ff3d280 1 Processor -- start 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.794+0000 7f5a5ff3d280 1 -- start start 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5ff3d280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5ea7b9400 0x55e5ea6b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5ff3d280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5ea7b9800 0x55e5ea6b7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5ff3d280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5ea7b9c00 0x55e5ea6b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5ff3d280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e5ea6c5c80 con 0x55e5ea7b9800 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5ff3d280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e5ea6c5b00 con 0x55e5ea7b9400 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5ff3d280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55e5ea6c5980 con 0x55e5ea7b9c00 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5e6e9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5ea7b9800 0x55e5ea6b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5e6e9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5ea7b9800 0x55e5ea6b7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47748/0 (socket says 172.21.15.44:47748) 2024-04-02T13:30:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5e6e9640 1 -- 172.21.15.44:0/1074680028 learned_addr learned my addr 172.21.15.44:0/1074680028 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5eeea640 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5ea7b9400 0x55e5ea6b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5dee8640 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5ea7b9c00 0x55e5ea6b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5e6e9640 1 -- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5ea7b9c00 msgr2=0x55e5ea6b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5e6e9640 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5ea7b9c00 0x55e5ea6b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5e6e9640 1 -- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5ea7b9400 msgr2=0x55e5ea6b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5e6e9640 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5ea7b9400 0x55e5ea6b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5e6e9640 1 -- 172.21.15.44:0/1074680028 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5e9920780 con 0x55e5ea7b9800 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.795+0000 7f5a5e6e9640 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5ea7b9800 0x55e5ea6b7600 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x55e5ea7d5590 tx=0x55e5ea7d8d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.796+0000 7f5a586dd640 1 -- 172.21.15.44:0/1074680028 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55e5e99109c0 con 0x55e5ea7b9800 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.796+0000 7f5a586dd640 1 -- 172.21.15.44:0/1074680028 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55e5e9916c40 con 0x55e5ea7b9800 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.796+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e5e99214a0 con 0x55e5ea7b9800 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.796+0000 7f5a586dd640 1 -- 172.21.15.44:0/1074680028 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55e5e9910820 con 0x55e5ea7b9800 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.796+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5e9920960 con 0x55e5ea7b9800 2024-04-02T13:30:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.796+0000 7f5a5ff3d280 4 main: JournalTool: resolving pool 15 2024-04-02T13:30:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.797+0000 7f5a586dd640 1 -- 172.21.15.44:0/1074680028 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x55e5ea7e8c00 con 0x55e5ea7b9800 2024-04-02T13:30:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.797+0000 7f5a586dd640 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55e5ea837000 0x55e5ea6b8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.797+0000 7f5a586dd640 1 -- 172.21.15.44:0/1074680028 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x55e5ea846600 con 0x55e5ea7b9800 2024-04-02T13:30:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.797+0000 7f5a5ff3d280 4 main: JournalTool: creating IoCtx.. 2024-04-02T13:30:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.797+0000 7f5a5ff3d280 4 main: Executing for rank 0 2024-04-02T13:30:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.797+0000 7f5a5ff3d280 4 Filtering by inode '1099511627778' 2024-04-02T13:30:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.797+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55e5ea837400 0x55e5ea6b8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.797+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:1 15.f 15:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e109) -- 0x55e5ea837800 con 0x55e5ea837400 2024-04-02T13:30:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.797+0000 7f5a5eeea640 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55e5ea837000 0x55e5ea6b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.797+0000 7f5a5dee8640 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55e5ea837400 0x55e5ea6b8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.798+0000 7f5a5eeea640 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55e5ea837000 0x55e5ea6b8100 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x55e5ea668810 tx=0x55e5e9bd9260 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.798+0000 7f5a5dee8640 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55e5ea837400 0x55e5ea6b8680 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5dee8640 1 -- 172.21.15.44:0/1074680028 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x55e5ea66e000 con 0x55e5ea837400 2024-04-02T13:30:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5ff3d280 4 Pointer 400.00000000 is readable 2024-04-02T13:30:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5ff3d280 4 JournalScanner::scan: reading header object '200.00000000' 2024-04-02T13:30:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:2 15.14 15:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e109) -- 0x55e5ea837c00 con 0x55e5ea837400 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5dee8640 1 -- 172.21.15.44:0/1074680028 <== osd.6 v2:172.21.15.69:6802/2604158175 2 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv31 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x55e5ea66e000 con 0x55e5ea837400 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5ff3d280 10 Header 0x400000 0x405f0e 0x40628a 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5ff3d280 10 Starting journal scan from offset 0x405f0e 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:3 15.14 15:2e2fa760:::200.00000001:head [read 24334~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e109) -- 0x55e5ea836c00 con 0x55e5ea837400 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5dee8640 1 -- 172.21.15.44:0/1074680028 <== osd.6 v2:172.21.15.69:6802/2604158175 3 ==== osd_op_reply(3 200.00000001 [read 24334~892 out=892b] v0'0 uv28 ondisk = 0) ==== 156+0+892 (crc 0 0 0) 0x55e5ea66e000 con 0x55e5ea837400 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5ff3d280 4 Read 0x37c bytes from 200.00000001 gap=0 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5ff3d280 10 Parsing data, 0x37c bytes available 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5ff3d280 10 Attempting decode at 0x405f0e 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5ff3d280 10 Consumed 0x37c bytes 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5ff3d280 10 Valid entry at 0x405f0e 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55e5ea836800 0x55e5ea6b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.799+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:4 15.1b 15:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e109) -- 0x55e5ea836400 con 0x55e5ea836800 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5eeea640 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55e5ea836800 0x55e5ea6b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5eeea640 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55e5ea836800 0x55e5ea6b8c00 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5eeea640 1 -- 172.21.15.44:0/1074680028 <== osd.7 v2:172.21.15.69:6824/1292475712 1 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55e5ea66e500 con 0x55e5ea836800 2024-04-02T13:30:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 4 Reached end of journal objects 2024-04-02T13:30:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 4 Scanned objects, 0 missing, 1 valid 2024-04-02T13:30:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 4 Events scanned, 0 gaps 2024-04-02T13:30:12.810 INFO:teuthology.orchestra.run.smithi044.stdout:Events by type: 2024-04-02T13:30:12.810 INFO:teuthology.orchestra.run.smithi044.stdout:Errors: 0 2024-04-02T13:30:12.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 4 Found 1 valid events 2024-04-02T13:30:12.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 4 Selected 0 events events for processing 2024-04-02T13:30:12.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 4 0 events 2024-04-02T13:30:12.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55e5ea837400 msgr2=0x55e5ea6b8680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55e5ea837400 0x55e5ea6b8680 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55e5ea836800 msgr2=0x55e5ea6b8c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55e5ea836800 0x55e5ea6b8c00 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55e5ea837000 msgr2=0x55e5ea6b8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55e5ea837000 0x55e5ea6b8100 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x55e5ea668810 tx=0x55e5e9bd9260 comp rx=0 tx=0).stop 2024-04-02T13:30:12.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5ea7b9800 msgr2=0x55e5ea6b7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/1074680028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5ea7b9800 0x55e5ea6b7600 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x55e5ea7d5590 tx=0x55e5ea7d8d20 comp rx=0 tx=0).stop 2024-04-02T13:30:12.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5e6e9640 1 -- 172.21.15.44:0/1074680028 reap_dead start 2024-04-02T13:30:12.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5e6e9640 1 -- 172.21.15.44:0/1074680028 reap_dead start 2024-04-02T13:30:12.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 shutdown_connections 2024-04-02T13:30:12.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 >> 172.21.15.44:0/1074680028 conn(0x55e5ea7b9000 msgr2=0x55e5e99d5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 shutdown_connections 2024-04-02T13:30:12.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.800+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/1074680028 wait complete. 2024-04-02T13:30:12.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.801+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/320591586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5e98eb800 msgr2=0x55e5ea6b6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.801+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/320591586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5e98eb800 0x55e5ea6b6580 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x55e5ea71b4a0 tx=0x55e5ea680d80 comp rx=0 tx=0).stop 2024-04-02T13:30:12.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.801+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/320591586 shutdown_connections 2024-04-02T13:30:12.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.801+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/320591586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e5e98ebc00 0x55e5ea6b6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.801+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/320591586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5e98eb800 0x55e5ea6b6580 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.801+0000 7f5a5ff3d280 1 --2- 172.21.15.44:0/320591586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e5e98eac00 0x55e5ea6b6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.801+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/320591586 >> 172.21.15.44:0/320591586 conn(0x55e5e98eb000 msgr2=0x55e5e99d5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.801+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/320591586 shutdown_connections 2024-04-02T13:30:12.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.801+0000 7f5a5ff3d280 1 -- 172.21.15.44:0/320591586 wait complete. 2024-04-02T13:30:12.815 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'event', 'splice', '--inode=1099511627778', 'summary'] in time 0:00:00.100414, result: Events by type: Errors: 0 2024-04-02T13:30:12.816 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put mds0_inotable - 2024-04-02T13:30:12.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.892+0000 7f1c1844af40 1 Processor -- start 2024-04-02T13:30:12.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.892+0000 7f1c1844af40 1 -- start start 2024-04-02T13:30:12.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c1844af40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558443287090 0x5584432a0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c1844af40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558443287ed0 0x5584433249e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c1844af40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55844332e8c0 0x5584433231f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c1844af40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5584432874a0 con 0x558443287ed0 2024-04-02T13:30:12.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c1844af40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5584431fe3e0 con 0x55844332e8c0 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c1844af40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x558443256d80 con 0x558443287090 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c15015640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558443287ed0 0x5584433249e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c15015640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558443287ed0 0x5584433249e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47770/0 (socket says 172.21.15.44:47770) 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c15015640 1 -- 172.21.15.44:0/277104278 learned_addr learned my addr 172.21.15.44:0/277104278 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c16017640 1 --2- 172.21.15.44:0/277104278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55844332e8c0 0x5584433231f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c15816640 1 --2- 172.21.15.44:0/277104278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558443287090 0x5584432a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c15015640 1 -- 172.21.15.44:0/277104278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558443287090 msgr2=0x5584432a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c15015640 1 --2- 172.21.15.44:0/277104278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558443287090 0x5584432a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c15015640 1 -- 172.21.15.44:0/277104278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55844332e8c0 msgr2=0x5584433231f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c15015640 1 --2- 172.21.15.44:0/277104278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55844332e8c0 0x5584433231f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c15015640 1 -- 172.21.15.44:0/277104278 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558443339410 con 0x558443287ed0 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.893+0000 7f1c15816640 1 --2- 172.21.15.44:0/277104278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558443287090 0x5584432a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.894+0000 7f1c15015640 1 --2- 172.21.15.44:0/277104278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558443287ed0 0x5584433249e0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c002de0 tx=0x7f1c0c00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127c0976d0b2f603 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.894+0000 7f1c07fff640 1 -- 172.21.15.44:0/277104278 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c00f400 con 0x558443287ed0 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.894+0000 7f1c07fff640 1 -- 172.21.15.44:0/277104278 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c0c014070 con 0x558443287ed0 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.894+0000 7f1c07fff640 1 -- 172.21.15.44:0/277104278 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c00f7e0 con 0x558443287ed0 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.894+0000 7f1c1844af40 1 -- 172.21.15.44:0/277104278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558443287ed0 msgr2=0x5584433249e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.894+0000 7f1c1844af40 1 --2- 172.21.15.44:0/277104278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558443287ed0 0x5584433249e0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c002de0 tx=0x7f1c0c00b5b0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.894+0000 7f1c1844af40 1 -- 172.21.15.44:0/277104278 shutdown_connections 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.894+0000 7f1c1844af40 1 --2- 172.21.15.44:0/277104278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55844332e8c0 0x5584433231f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.894+0000 7f1c1844af40 1 --2- 172.21.15.44:0/277104278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558443287ed0 0x5584433249e0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.894+0000 7f1c1844af40 1 --2- 172.21.15.44:0/277104278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558443287090 0x5584432a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.894+0000 7f1c1844af40 1 -- 172.21.15.44:0/277104278 >> 172.21.15.44:0/277104278 conn(0x558443282b50 msgr2=0x5584432584c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.894+0000 7f1c1844af40 1 -- 172.21.15.44:0/277104278 shutdown_connections 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.894+0000 7f1c1844af40 1 -- 172.21.15.44:0/277104278 wait complete. 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.895+0000 7f1c1844af40 1 Processor -- start 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.895+0000 7f1c1844af40 1 -- start start 2024-04-02T13:30:12.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.895+0000 7f1c1844af40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558443287090 0x5584433238b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.895+0000 7f1c1844af40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558443287ed0 0x55844331f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.895+0000 7f1c1844af40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55844332e8c0 0x55844331fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.895+0000 7f1c1844af40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55844325ae40 con 0x55844332e8c0 2024-04-02T13:30:12.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.895+0000 7f1c1844af40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5584433391a0 con 0x558443287ed0 2024-04-02T13:30:12.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.895+0000 7f1c1844af40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55844325b120 con 0x558443287090 2024-04-02T13:30:12.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c16017640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55844332e8c0 0x55844331fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c16017640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55844332e8c0 0x55844331fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47774/0 (socket says 172.21.15.44:47774) 2024-04-02T13:30:12.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c16017640 1 -- 172.21.15.44:0/1559459943 learned_addr learned my addr 172.21.15.44:0/1559459943 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:12.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c15816640 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558443287090 0x5584433238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c15015640 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558443287ed0 0x55844331f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c16017640 1 -- 172.21.15.44:0/1559459943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558443287090 msgr2=0x5584433238b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c16017640 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558443287090 0x5584433238b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c16017640 1 -- 172.21.15.44:0/1559459943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558443287ed0 msgr2=0x55844331f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c16017640 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558443287ed0 0x55844331f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c16017640 1 -- 172.21.15.44:0/1559459943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55844329e670 con 0x55844332e8c0 2024-04-02T13:30:12.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c15015640 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558443287ed0 0x55844331f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:12.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c16017640 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55844332e8c0 0x55844331fcb0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000e650 tx=0x7f1c1000eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c06ffd640 1 -- 172.21.15.44:0/1559459943 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c10016070 con 0x55844332e8c0 2024-04-02T13:30:12.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c06ffd640 1 -- 172.21.15.44:0/1559459943 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c1000c9a0 con 0x55844332e8c0 2024-04-02T13:30:12.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c1844af40 1 -- 172.21.15.44:0/1559459943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5584433219e0 con 0x55844332e8c0 2024-04-02T13:30:12.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c06ffd640 1 -- 172.21.15.44:0/1559459943 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1000cd90 con 0x55844332e8c0 2024-04-02T13:30:12.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.896+0000 7f1c1844af40 1 -- 172.21.15.44:0/1559459943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558443334700 con 0x55844332e8c0 2024-04-02T13:30:12.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.898+0000 7f1c06ffd640 1 -- 172.21.15.44:0/1559459943 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1c10019070 con 0x55844332e8c0 2024-04-02T13:30:12.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.898+0000 7f1c06ffd640 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1bf806b2d0 0x7f1bf806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.898+0000 7f1c06ffd640 1 -- 172.21.15.44:0/1559459943 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f1c1008afa0 con 0x55844332e8c0 2024-04-02T13:30:12.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.898+0000 7f1c1844af40 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5584432836b0 0x558443330500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:12.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.898+0000 7f1c1844af40 1 -- 172.21.15.44:0/1559459943 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:1 15.13 15:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x558443323df0 con 0x5584432836b0 2024-04-02T13:30:12.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.898+0000 7f1c15816640 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1bf806b2d0 0x7f1bf806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.898+0000 7f1c15015640 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5584432836b0 0x558443330500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:12.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.899+0000 7f1c15015640 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5584432836b0 0x558443330500 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.899+0000 7f1c15816640 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1bf806b2d0 0x7f1bf806d790 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1c08054000 tx=0x7f1c0804e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:12.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.900+0000 7f1c15015640 1 -- 172.21.15.44:0/1559459943 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(1 mds0_inotable [writefull 0~34] v109'5 uv5 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1c0c01a410 con 0x5584432836b0 2024-04-02T13:30:12.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.900+0000 7f1c1844af40 1 -- 172.21.15.44:0/1559459943 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5584432836b0 msgr2=0x558443330500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.900+0000 7f1c1844af40 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x5584432836b0 0x558443330500 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:12.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.900+0000 7f1c1844af40 1 -- 172.21.15.44:0/1559459943 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1bf806b2d0 msgr2=0x7f1bf806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.900+0000 7f1c1844af40 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f1bf806b2d0 0x7f1bf806d790 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1c08054000 tx=0x7f1c0804e000 comp rx=0 tx=0).stop 2024-04-02T13:30:12.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.900+0000 7f1c1844af40 1 -- 172.21.15.44:0/1559459943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55844332e8c0 msgr2=0x55844331fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:12.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.900+0000 7f1c1844af40 1 --2- 172.21.15.44:0/1559459943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55844332e8c0 0x55844331fcb0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000e650 tx=0x7f1c1000eb10 comp rx=0 tx=0).stop 2024-04-02T13:30:12.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.900+0000 7f1c16017640 1 -- 172.21.15.44:0/1559459943 reap_dead start 2024-04-02T13:30:12.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.900+0000 7f1c1844af40 1 -- 172.21.15.44:0/1559459943 shutdown_connections 2024-04-02T13:30:12.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.900+0000 7f1c1844af40 1 -- 172.21.15.44:0/1559459943 >> 172.21.15.44:0/1559459943 conn(0x558443282b50 msgr2=0x558443282f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:12.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.900+0000 7f1c1844af40 1 -- 172.21.15.44:0/1559459943 shutdown_connections 2024-04-02T13:30:12.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:12.900+0000 7f1c1844af40 1 -- 172.21.15.44:0/1559459943 wait complete. 2024-04-02T13:30:12.907 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-02T13:30:13.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.048+0000 7f22e1e93640 1 Processor -- start 2024-04-02T13:30:13.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.048+0000 7f22e1e93640 1 -- start start 2024-04-02T13:30:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.049+0000 7f22e1e93640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f22dc0d0a30 0x7f22dc06cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.049+0000 7f22e1e93640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22dc0d0e10 0x7f22dc0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.049+0000 7f22e1e93640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22dc06c1d0 con 0x7f22dc0cfe30 2024-04-02T13:30:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.049+0000 7f22e1e93640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f22dc06cb00 con 0x7f22dc0d0a30 2024-04-02T13:30:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.049+0000 7f22e1e93640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f22dc06c4b0 con 0x7f22dc0d0e10 2024-04-02T13:30:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.049+0000 7f22db7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f22dc0d0a30 0x7f22dc06cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.049+0000 7f22dbfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f22dc0cfe30 0x7f22dc0d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48568/0 (socket says 172.21.15.44:48568) 2024-04-02T13:30:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.049+0000 7f22dbfff640 1 -- 172.21.15.44:0/4270123692 learned_addr learned my addr 172.21.15.44:0/4270123692 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.049+0000 7f22daffd640 1 --2- 172.21.15.44:0/4270123692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22dc0d0e10 0x7f22dc0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.049+0000 7f22db7fe640 1 -- 172.21.15.44:0/4270123692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22dc0d0e10 msgr2=0x7f22dc0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.049+0000 7f22db7fe640 1 --2- 172.21.15.44:0/4270123692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22dc0d0e10 0x7f22dc0cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.049+0000 7f22db7fe640 1 -- 172.21.15.44:0/4270123692 >> v1:172.21.15.44:6789/0 conn(0x7f22dc0cfe30 legacy=0x7f22dc0d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.050+0000 7f22db7fe640 1 -- 172.21.15.44:0/4270123692 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22dc06c790 con 0x7f22dc0d0a30 2024-04-02T13:30:13.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.049+0000 7f22da7fc640 1 -- 172.21.15.44:0/4270123692 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2407712055 0 0) 0x7f22dc06c1d0 con 0x7f22dc0cfe30 2024-04-02T13:30:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.050+0000 7f22db7fe640 1 --2- 172.21.15.44:0/4270123692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22dc0d0a30 0x7f22dc06cf90 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f22cc0097a0 tx=0x7f22cc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153a413872371b42 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.050+0000 7f22da7fc640 1 -- 172.21.15.44:0/4270123692 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc0315a0 con 0x7f22dc0d0a30 2024-04-02T13:30:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.050+0000 7f22da7fc640 1 -- 172.21.15.44:0/4270123692 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22cc037070 con 0x7f22dc0d0a30 2024-04-02T13:30:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.050+0000 7f22da7fc640 1 -- 172.21.15.44:0/4270123692 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc031960 con 0x7f22dc0d0a30 2024-04-02T13:30:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.051+0000 7f22e1e93640 1 -- 172.21.15.44:0/4270123692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22dc0d0a30 msgr2=0x7f22dc06cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.051+0000 7f22e1e93640 1 --2- 172.21.15.44:0/4270123692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22dc0d0a30 0x7f22dc06cf90 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f22cc0097a0 tx=0x7f22cc02b840 comp rx=0 tx=0).stop 2024-04-02T13:30:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.051+0000 7f22e1e93640 1 -- 172.21.15.44:0/4270123692 shutdown_connections 2024-04-02T13:30:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.051+0000 7f22e1e93640 1 --2- 172.21.15.44:0/4270123692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22dc0d0e10 0x7f22dc0cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.051+0000 7f22e1e93640 1 --2- 172.21.15.44:0/4270123692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22dc0d0a30 0x7f22dc06cf90 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.051+0000 7f22e1e93640 1 -- 172.21.15.44:0/4270123692 >> 172.21.15.44:0/4270123692 conn(0x7f22dc09d470 msgr2=0x7f22dc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.051+0000 7f22e1e93640 1 -- 172.21.15.44:0/4270123692 shutdown_connections 2024-04-02T13:30:13.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.051+0000 7f22e1e93640 1 -- 172.21.15.44:0/4270123692 wait complete. 2024-04-02T13:30:13.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.052+0000 7f22e1e93640 1 Processor -- start 2024-04-02T13:30:13.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.052+0000 7f22e1e93640 1 -- start start 2024-04-02T13:30:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22e1e93640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22dc0cfe30 0x7f22dc0c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22e1e93640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22dc0d0a30 0x7f22dc1505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22e1e93640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22dc0d0e10 0x7f22dc0c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22e1e93640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc15e870 con 0x7f22dc0d0e10 2024-04-02T13:30:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22e1e93640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc15e6f0 con 0x7f22dc0d0a30 2024-04-02T13:30:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22e1e93640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f22dc0d4da0 con 0x7f22dc0cfe30 2024-04-02T13:30:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22dbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22dc0d0e10 0x7f22dc0c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22dbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22dc0d0e10 0x7f22dc0c7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47794/0 (socket says 172.21.15.44:47794) 2024-04-02T13:30:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22dbfff640 1 -- 172.21.15.44:0/83839833 learned_addr learned my addr 172.21.15.44:0/83839833 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22db7fe640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22dc0cfe30 0x7f22dc0c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22daffd640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22dc0d0a30 0x7f22dc1505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22dbfff640 1 -- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22dc0cfe30 msgr2=0x7f22dc0c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22dbfff640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22dc0cfe30 0x7f22dc0c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22dbfff640 1 -- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22dc0d0a30 msgr2=0x7f22dc1505a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22dbfff640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22dc0d0a30 0x7f22dc1505a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.053+0000 7f22dbfff640 1 -- 172.21.15.44:0/83839833 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22dc150ae0 con 0x7f22dc0d0e10 2024-04-02T13:30:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.054+0000 7f22dbfff640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22dc0d0e10 0x7f22dc0c7c70 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f22d400c730 tx=0x7f22d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.054+0000 7f22c3fff640 1 -- 172.21.15.44:0/83839833 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d4018070 con 0x7f22dc0d0e10 2024-04-02T13:30:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.054+0000 7f22c3fff640 1 -- 172.21.15.44:0/83839833 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22d400ce40 con 0x7f22dc0d0e10 2024-04-02T13:30:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.054+0000 7f22e1e93640 1 -- 172.21.15.44:0/83839833 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22dc0c84c0 con 0x7f22dc0d0e10 2024-04-02T13:30:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.054+0000 7f22c3fff640 1 -- 172.21.15.44:0/83839833 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d400a860 con 0x7f22dc0d0e10 2024-04-02T13:30:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.054+0000 7f22e1e93640 1 -- 172.21.15.44:0/83839833 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22dc0c8a80 con 0x7f22dc0d0e10 2024-04-02T13:30:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.055+0000 7f22e1e93640 1 -- 172.21.15.44:0/83839833 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22dc0684f0 con 0x7f22dc0d0e10 2024-04-02T13:30:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.055+0000 7f22c3fff640 1 -- 172.21.15.44:0/83839833 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f22d401b070 con 0x7f22dc0d0e10 2024-04-02T13:30:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.056+0000 7f22c3fff640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f22c806c8f0 0x7f22c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.056+0000 7f22c3fff640 1 -- 172.21.15.44:0/83839833 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f22d408b2f0 con 0x7f22dc0d0e10 2024-04-02T13:30:13.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.056+0000 7f22db7fe640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f22c806c8f0 0x7f22c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.056+0000 7f22db7fe640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f22c806c8f0 0x7f22c806edb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f22cc036000 tx=0x7f22cc03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:13.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.060+0000 7f22c3fff640 1 -- 172.21.15.44:0/83839833 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f22dc150ae0 con 0x7f22dc0d0e10 2024-04-02T13:30:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.283+0000 7f22e1e93640 1 -- 172.21.15.44:0/83839833 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f22dc0d17c0 con 0x7f22dc0d0e10 2024-04-02T13:30:13.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.630+0000 7f22c3fff640 1 -- 172.21.15.44:0/83839833 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v84) ==== 162+0+0 (secure 0 0 0) 0x7f22d4058df0 con 0x7f22dc0d0e10 2024-04-02T13:30:13.633 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-02T13:30:13.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.632+0000 7f22e1e93640 1 -- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f22c806c8f0 msgr2=0x7f22c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:13.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.632+0000 7f22e1e93640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f22c806c8f0 0x7f22c806edb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f22cc036000 tx=0x7f22cc03d040 comp rx=0 tx=0).stop 2024-04-02T13:30:13.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.632+0000 7f22e1e93640 1 -- 172.21.15.44:0/83839833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22dc0d0e10 msgr2=0x7f22dc0c7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:13.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.632+0000 7f22e1e93640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22dc0d0e10 0x7f22dc0c7c70 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f22d400c730 tx=0x7f22d400cbf0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.633+0000 7f22e1e93640 1 -- 172.21.15.44:0/83839833 shutdown_connections 2024-04-02T13:30:13.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.633+0000 7f22e1e93640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f22c806c8f0 0x7f22c806edb0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.633+0000 7f22e1e93640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22dc0d0e10 0x7f22dc0c7c70 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.633+0000 7f22e1e93640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22dc0d0a30 0x7f22dc1505a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.633+0000 7f22e1e93640 1 --2- 172.21.15.44:0/83839833 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22dc0cfe30 0x7f22dc0c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.633+0000 7f22e1e93640 1 -- 172.21.15.44:0/83839833 >> 172.21.15.44:0/83839833 conn(0x7f22dc09d470 msgr2=0x7f22dc15ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:13.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.633+0000 7f22e1e93640 1 -- 172.21.15.44:0/83839833 shutdown_connections 2024-04-02T13:30:13.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.633+0000 7f22e1e93640 1 -- 172.21.15.44:0/83839833 wait complete. 2024-04-02T13:30:13.650 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:30:13.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.752+0000 7f3b3d4d3640 1 Processor -- start 2024-04-02T13:30:13.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.752+0000 7f3b3d4d3640 1 -- start start 2024-04-02T13:30:13.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.753+0000 7f3b3d4d3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3b380cece0 0x7f3b380cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.753+0000 7f3b3d4d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b380d09f0 0x7f3b3806d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.753+0000 7f3b3d4d3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b380cf600 con 0x7f3b3806ece0 2024-04-02T13:30:13.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.753+0000 7f3b3d4d3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3b3806dc40 con 0x7f3b380cece0 2024-04-02T13:30:13.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.753+0000 7f3b3d4d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3b3806df20 con 0x7f3b380d09f0 2024-04-02T13:30:13.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.753+0000 7f3b377fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f3b3806ece0 0x7f3b3815eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48570/0 (socket says 172.21.15.44:48570) 2024-04-02T13:30:13.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.753+0000 7f3b377fe640 1 -- 172.21.15.44:0/2173760915 learned_addr learned my addr 172.21.15.44:0/2173760915 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:13.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.753+0000 7f3b367fc640 1 --2- 172.21.15.44:0/2173760915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b380d09f0 0x7f3b3806d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.753+0000 7f3b36ffd640 1 --2- 172.21.15.44:0/2173760915 >> v2:172.21.15.69:3300/0 conn(0x7f3b380cece0 0x7f3b380cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.753+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3872319182 0 0) 0x7f3b380cf600 con 0x7f3b3806ece0 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.753+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b20003520 con 0x7f3b3806ece0 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 919344406 0 0) 0x7f3b20003520 con 0x7f3b3806ece0 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b380d09f0 msgr2=0x7f3b3806d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 --2- 172.21.15.44:0/2173760915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b380d09f0 0x7f3b3806d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 >> v2:172.21.15.69:3300/0 conn(0x7f3b380cece0 msgr2=0x7f3b380cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 --2- 172.21.15.44:0/2173760915 >> v2:172.21.15.69:3300/0 conn(0x7f3b380cece0 0x7f3b380cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3806e200 con 0x7f3b3806ece0 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f3b300019f0 con 0x7f3b3806ece0 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 >> v1:172.21.15.44:6789/0 conn(0x7f3b3806ece0 legacy=0x7f3b3815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 --2- 172.21.15.44:0/2173760915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b380d09f0 0x7f3b200039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 --2- 172.21.15.44:0/2173760915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b20003f30 0x7f3b20004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 --2- 172.21.15.44:0/2173760915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b20004d40 0x7f3b200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3806dc40 con 0x7f3b20003f30 2024-04-02T13:30:13.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3806df20 con 0x7f3b20004d40 2024-04-02T13:30:13.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3b2000b9c0 con 0x7f3b380d09f0 2024-04-02T13:30:13.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b367fc640 1 --2- 172.21.15.44:0/2173760915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b20003f30 0x7f3b20004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b36ffd640 1 --2- 172.21.15.44:0/2173760915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b380d09f0 0x7f3b200039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b377fe640 1 --2- 172.21.15.44:0/2173760915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b20004d40 0x7f3b200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b367fc640 1 -- 172.21.15.44:0/2173760915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b380d09f0 msgr2=0x7f3b200039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:13.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b367fc640 1 --2- 172.21.15.44:0/2173760915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b380d09f0 0x7f3b200039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b367fc640 1 -- 172.21.15.44:0/2173760915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b20004d40 msgr2=0x7f3b200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:13.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b367fc640 1 --2- 172.21.15.44:0/2173760915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b20004d40 0x7f3b200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b367fc640 1 -- 172.21.15.44:0/2173760915 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b20003520 con 0x7f3b20003f30 2024-04-02T13:30:13.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.754+0000 7f3b377fe640 1 --2- 172.21.15.44:0/2173760915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b20004d40 0x7f3b200051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:13.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.755+0000 7f3b367fc640 1 --2- 172.21.15.44:0/2173760915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b20003f30 0x7f3b20004800 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009a90 tx=0x7f3b20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77862c81b591266d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:13.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.755+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28039070 con 0x7f3b20003f30 2024-04-02T13:30:13.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.755+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b2802c6b0 con 0x7f3b20003f30 2024-04-02T13:30:13.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.755+0000 7f3b35ffb640 1 -- 172.21.15.44:0/2173760915 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28009690 con 0x7f3b20003f30 2024-04-02T13:30:13.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.755+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/2173760915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b20003f30 msgr2=0x7f3b20004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:13.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.755+0000 7f3b3d4d3640 1 --2- 172.21.15.44:0/2173760915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b20003f30 0x7f3b20004800 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009a90 tx=0x7f3b20005830 comp rx=0 tx=0).stop 2024-04-02T13:30:13.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.755+0000 7f3b377fe640 1 -- 172.21.15.44:0/2173760915 reap_dead start 2024-04-02T13:30:13.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.756+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/2173760915 shutdown_connections 2024-04-02T13:30:13.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.756+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/2173760915 >> 172.21.15.44:0/2173760915 conn(0x7f3b380cfcd0 msgr2=0x7f3b380d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:13.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.756+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/2173760915 shutdown_connections 2024-04-02T13:30:13.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.756+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/2173760915 wait complete. 2024-04-02T13:30:13.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.757+0000 7f3b3d4d3640 1 Processor -- start 2024-04-02T13:30:13.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.757+0000 7f3b3d4d3640 1 -- start start 2024-04-02T13:30:13.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b3d4d3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b20003f30 0x7f3b3809ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b3d4d3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b3809cf70 0x7f3b3815d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b3d4d3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b3815dde0 0x7f3b38138d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b3d4d3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3815b220 con 0x7f3b3809cf70 2024-04-02T13:30:13.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b3d4d3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3806e7c0 con 0x7f3b3815dde0 2024-04-02T13:30:13.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b3d4d3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3b3815b3a0 con 0x7f3b20003f30 2024-04-02T13:30:13.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b367fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b3809cf70 0x7f3b3815d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b367fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b3809cf70 0x7f3b3815d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47820/0 (socket says 172.21.15.44:47820) 2024-04-02T13:30:13.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b367fc640 1 -- 172.21.15.44:0/90912854 learned_addr learned my addr 172.21.15.44:0/90912854 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:13.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b36ffd640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b20003f30 0x7f3b3809ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b377fe640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b3815dde0 0x7f3b38138d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b367fc640 1 -- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b20003f30 msgr2=0x7f3b3809ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:13.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b367fc640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b20003f30 0x7f3b3809ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b367fc640 1 -- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b3815dde0 msgr2=0x7f3b38138d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:13.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b367fc640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b3815dde0 0x7f3b38138d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b367fc640 1 -- 172.21.15.44:0/90912854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3814b070 con 0x7f3b3809cf70 2024-04-02T13:30:13.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b36ffd640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b20003f30 0x7f3b3809ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:13.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b377fe640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b3815dde0 0x7f3b38138d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:13.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b367fc640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b3809cf70 0x7f3b3815d8a0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f3b28000c00 tx=0x7f3b28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:13.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b1f7fe640 1 -- 172.21.15.44:0/90912854 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28042020 con 0x7f3b3809cf70 2024-04-02T13:30:13.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b1f7fe640 1 -- 172.21.15.44:0/90912854 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b2803a040 con 0x7f3b3809cf70 2024-04-02T13:30:13.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b1f7fe640 1 -- 172.21.15.44:0/90912854 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28039070 con 0x7f3b3809cf70 2024-04-02T13:30:13.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.758+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/90912854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b3814b350 con 0x7f3b3809cf70 2024-04-02T13:30:13.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.759+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/90912854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b3814b910 con 0x7f3b3809cf70 2024-04-02T13:30:13.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.759+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/90912854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b38074f20 con 0x7f3b3809cf70 2024-04-02T13:30:13.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.760+0000 7f3b1f7fe640 1 -- 172.21.15.44:0/90912854 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3b28045020 con 0x7f3b3809cf70 2024-04-02T13:30:13.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.760+0000 7f3b1f7fe640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b2406c920 0x7f3b2406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:13.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.760+0000 7f3b1f7fe640 1 -- 172.21.15.44:0/90912854 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f3b280ad510 con 0x7f3b3809cf70 2024-04-02T13:30:13.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.760+0000 7f3b36ffd640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b2406c920 0x7f3b2406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:13.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.761+0000 7f3b36ffd640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b2406c920 0x7f3b2406ede0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009b30 tx=0x7f3b2c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:13.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.765+0000 7f3b1f7fe640 1 -- 172.21.15.44:0/90912854 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3b2804c030 con 0x7f3b3809cf70 2024-04-02T13:30:13.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.988+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/90912854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b380684f0 con 0x7f3b3809cf70 2024-04-02T13:30:13.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.988+0000 7f3b1f7fe640 1 -- 172.21.15.44:0/90912854 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+5178 (secure 0 0 0) 0x7f3b2807b700 con 0x7f3b3809cf70 2024-04-02T13:30:13.991 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:13.991 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":85,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7696,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2715855087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2715855087},{"type":"v1","addr":"172.21.15.69:6836","nonce":2715855087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7699,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3850680553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3850680553},{"type":"v1","addr":"172.21.15.69:6837","nonce":3850680553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/1330712383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1330712383},{"type":"v1","addr":"172.21.15.44:6835","nonce":1330712383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:29:10.907920+0000","modified":"2024-04-02T13:30:13.633273+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9192":{"gid":9192,"name":"a","rank":0,"incarnation":85,"state":"up:replay","state_seq":1,"addr":"172.21.15.44:6834/1745009605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":1745009605},{"type":"v1","addr":"172.21.15.44:6834","nonce":1745009605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-04-02T13:30:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 85 2024-04-02T13:30:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.990+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b2406c920 msgr2=0x7f3b2406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.990+0000 7f3b3d4d3640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b2406c920 0x7f3b2406ede0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009b30 tx=0x7f3b2c008040 comp rx=0 tx=0).stop 2024-04-02T13:30:13.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.990+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/90912854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b3809cf70 msgr2=0x7f3b3815d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:13.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.990+0000 7f3b3d4d3640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b3809cf70 0x7f3b3815d8a0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f3b28000c00 tx=0x7f3b28004730 comp rx=0 tx=0).stop 2024-04-02T13:30:13.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.990+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/90912854 shutdown_connections 2024-04-02T13:30:13.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.990+0000 7f3b3d4d3640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3b2406c920 0x7f3b2406ede0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.990+0000 7f3b3d4d3640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b3815dde0 0x7f3b38138d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.990+0000 7f3b3d4d3640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b3809cf70 0x7f3b3815d8a0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.990+0000 7f3b3d4d3640 1 --2- 172.21.15.44:0/90912854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b20003f30 0x7f3b3809ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.991+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/90912854 >> 172.21.15.44:0/90912854 conn(0x7f3b380cfcd0 msgr2=0x7f3b38071810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.991+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/90912854 shutdown_connections 2024-04-02T13:30:13.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:13.991+0000 7f3b3d4d3640 1 -- 172.21.15.44:0/90912854 wait complete. 2024-04-02T13:30:14.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 85, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:29:10.907920+0000', 'modified': '2024-04-02T13:30:13.633273+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 109, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9192}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9192': {'gid': 9192, 'name': 'a', 'rank': 0, 'incarnation': 85, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.44:6834/1745009605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6833', 'nonce': 1745009605}, {'type': 'v1', 'addr': '172.21.15.44:6834', 'nonce': 1745009605}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-02T13:30:14.407 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9192:up:replay 2024-04-02T13:30:15.408 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:30:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.519+0000 7fe57cd1c640 1 Processor -- start 2024-04-02T13:30:15.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.519+0000 7fe57cd1c640 1 -- start start 2024-04-02T13:30:15.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.519+0000 7fe57cd1c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe5780d0a30 0x7fe57806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:15.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.519+0000 7fe57cd1c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5780d0000 con 0x7fe5780d2040 2024-04-02T13:30:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.519+0000 7fe57cd1c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe57806c1d0 con 0x7fe5780d0e10 2024-04-02T13:30:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.519+0000 7fe57cd1c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe57806cb00 con 0x7fe5780d0a30 2024-04-02T13:30:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe576d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe5780d2040 0x7fe5780d24f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:50394/0 (socket says 172.21.15.44:50394) 2024-04-02T13:30:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe576d9d640 1 -- 172.21.15.44:0/668944101 learned_addr learned my addr 172.21.15.44:0/668944101 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe57659c640 1 --2- 172.21.15.44:0/668944101 >> v2:172.21.15.69:3300/0 conn(0x7fe5780d0a30 0x7fe57806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2738776512 0 0) 0x7fe5780d0000 con 0x7fe5780d2040 2024-04-02T13:30:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe564003520 con 0x7fe5780d2040 2024-04-02T13:30:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3709736739 0 0) 0x7fe57806c1d0 con 0x7fe5780d0e10 2024-04-02T13:30:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5780d0000 con 0x7fe5780d0e10 2024-04-02T13:30:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 214275363 0 0) 0x7fe564003520 con 0x7fe5780d2040 2024-04-02T13:30:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 >> v2:172.21.15.69:3300/0 conn(0x7fe5780d0a30 msgr2=0x7fe57806cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 --2- 172.21.15.44:0/668944101 >> v2:172.21.15.69:3300/0 conn(0x7fe5780d0a30 0x7fe57806cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 >> v1:172.21.15.69:6789/0 conn(0x7fe5780d0e10 legacy=0x7fe5780cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe57806c4b0 con 0x7fe5780d2040 2024-04-02T13:30:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fe5700019f0 con 0x7fe5780d2040 2024-04-02T13:30:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 >> v1:172.21.15.44:6789/0 conn(0x7fe5780d2040 legacy=0x7fe5780d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 --2- 172.21.15.44:0/668944101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe564003b90 0x7fe564003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 --2- 172.21.15.44:0/668944101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe564004800 0x7fe564004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 --2- 172.21.15.44:0/668944101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe564005210 0x7fe56400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe57806cb00 con 0x7fe564004800 2024-04-02T13:30:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe56400c140 con 0x7fe564005210 2024-04-02T13:30:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe56400c340 con 0x7fe564003b90 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe56ffff640 1 --2- 172.21.15.44:0/668944101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe564004800 0x7fe564004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe576d9d640 1 --2- 172.21.15.44:0/668944101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe564005210 0x7fe56400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.520+0000 7fe57659c640 1 --2- 172.21.15.44:0/668944101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe564003b90 0x7fe564003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.521+0000 7fe56ffff640 1 -- 172.21.15.44:0/668944101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe564003b90 msgr2=0x7fe564003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.521+0000 7fe56ffff640 1 --2- 172.21.15.44:0/668944101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe564003b90 0x7fe564003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.521+0000 7fe56ffff640 1 -- 172.21.15.44:0/668944101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe564005210 msgr2=0x7fe56400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.521+0000 7fe56ffff640 1 --2- 172.21.15.44:0/668944101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe564005210 0x7fe56400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.521+0000 7fe56ffff640 1 -- 172.21.15.44:0/668944101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe564003520 con 0x7fe564004800 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.521+0000 7fe576d9d640 1 -- 172.21.15.44:0/668944101 reap_dead start 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.521+0000 7fe56ffff640 1 --2- 172.21.15.44:0/668944101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe564004800 0x7fe564004c80 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fe5600018a0 tx=0x7fe564005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682eb84505ce460e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.522+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe560037070 con 0x7fe564004800 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.522+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5600052d0 con 0x7fe564004800 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.522+0000 7fe575d9b640 1 -- 172.21.15.44:0/668944101 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5600014f0 con 0x7fe564004800 2024-04-02T13:30:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.522+0000 7fe57cd1c640 1 -- 172.21.15.44:0/668944101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe564004800 msgr2=0x7fe564004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.522+0000 7fe57cd1c640 1 --2- 172.21.15.44:0/668944101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe564004800 0x7fe564004c80 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fe5600018a0 tx=0x7fe564005830 comp rx=0 tx=0).stop 2024-04-02T13:30:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.522+0000 7fe57cd1c640 1 -- 172.21.15.44:0/668944101 shutdown_connections 2024-04-02T13:30:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.522+0000 7fe57cd1c640 1 --2- 172.21.15.44:0/668944101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe564004800 0x7fe564004c80 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.522+0000 7fe57cd1c640 1 -- 172.21.15.44:0/668944101 >> 172.21.15.44:0/668944101 conn(0x7fe57809d470 msgr2=0x7fe5780d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.522+0000 7fe57cd1c640 1 -- 172.21.15.44:0/668944101 shutdown_connections 2024-04-02T13:30:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.522+0000 7fe57cd1c640 1 -- 172.21.15.44:0/668944101 wait complete. 2024-04-02T13:30:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.524+0000 7fe57cd1c640 1 Processor -- start 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.524+0000 7fe57cd1c640 1 -- start start 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.524+0000 7fe57cd1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe564004800 0x7fe5781502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.524+0000 7fe57cd1c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5780c85f0 0x7fe5780b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.524+0000 7fe57cd1c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5780c8a70 0x7fe5780b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.524+0000 7fe57cd1c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe57815e6f0 con 0x7fe5780c8a70 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.524+0000 7fe57cd1c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe5780d1270 con 0x7fe5780c85f0 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.524+0000 7fe57cd1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe57815e870 con 0x7fe564004800 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.524+0000 7fe576d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5780c8a70 0x7fe5780b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe576d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5780c8a70 0x7fe5780b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45990/0 (socket says 172.21.15.44:45990) 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe576d9d640 1 -- 172.21.15.44:0/3125364389 learned_addr learned my addr 172.21.15.44:0/3125364389 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe56ffff640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5780c85f0 0x7fe5780b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe57659c640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe564004800 0x7fe5781502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe576d9d640 1 -- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe564004800 msgr2=0x7fe5781502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe576d9d640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe564004800 0x7fe5781502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe576d9d640 1 -- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5780c85f0 msgr2=0x7fe5780b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe576d9d640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5780c85f0 0x7fe5780b44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe576d9d640 1 -- 172.21.15.44:0/3125364389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5780c9020 con 0x7fe5780c8a70 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe57659c640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe564004800 0x7fe5781502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe56ffff640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5780c85f0 0x7fe5780b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe576d9d640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5780c8a70 0x7fe5780b49f0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fe57809d9a0 tx=0x7fe570007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe56effd640 1 -- 172.21.15.44:0/3125364389 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57000aa30 con 0x7fe5780c8a70 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe56effd640 1 -- 172.21.15.44:0/3125364389 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe570019070 con 0x7fe5780c8a70 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe57cd1c640 1 -- 172.21.15.44:0/3125364389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5780c9300 con 0x7fe5780c8a70 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe56effd640 1 -- 172.21.15.44:0/3125364389 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57000ae30 con 0x7fe5780c8a70 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.525+0000 7fe57cd1c640 1 -- 172.21.15.44:0/3125364389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5780c98c0 con 0x7fe5780c8a70 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.526+0000 7fe57cd1c640 1 -- 172.21.15.44:0/3125364389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5780684f0 con 0x7fe5780c8a70 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.527+0000 7fe56effd640 1 -- 172.21.15.44:0/3125364389 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe57001e460 con 0x7fe5780c8a70 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.527+0000 7fe56effd640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe55806c8f0 0x7fe55806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.527+0000 7fe56effd640 1 -- 172.21.15.44:0/3125364389 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7fe57008c1d0 con 0x7fe5780c8a70 2024-04-02T13:30:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.527+0000 7fe57659c640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe55806c8f0 0x7fe55806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.528+0000 7fe57659c640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe55806c8f0 0x7fe55806edb0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe5680099e0 tx=0x7fe568008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.532+0000 7fe56effd640 1 -- 172.21.15.44:0/3125364389 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe57005a450 con 0x7fe5780c8a70 2024-04-02T13:30:15.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.758+0000 7fe57cd1c640 1 -- 172.21.15.44:0/3125364389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe57815def0 con 0x7fe5780c8a70 2024-04-02T13:30:15.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.758+0000 7fe56effd640 1 -- 172.21.15.44:0/3125364389 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5178 (secure 0 0 0) 0x7fe570059d50 con 0x7fe5780c8a70 2024-04-02T13:30:15.762 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:15.762 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":87,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7696,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2715855087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2715855087},{"type":"v1","addr":"172.21.15.69:6836","nonce":2715855087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7699,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3850680553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3850680553},{"type":"v1","addr":"172.21.15.69:6837","nonce":3850680553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/1330712383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1330712383},{"type":"v1","addr":"172.21.15.44:6835","nonce":1330712383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:29:10.907920+0000","modified":"2024-04-02T13:30:14.638700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9192":{"gid":9192,"name":"a","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.44:6834/1745009605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":1745009605},{"type":"v1","addr":"172.21.15.44:6834","nonce":1745009605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-04-02T13:30:15.762 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 87 2024-04-02T13:30:15.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.761+0000 7fe57cd1c640 1 -- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe55806c8f0 msgr2=0x7fe55806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:15.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.761+0000 7fe57cd1c640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe55806c8f0 0x7fe55806edb0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe5680099e0 tx=0x7fe568008040 comp rx=0 tx=0).stop 2024-04-02T13:30:15.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.761+0000 7fe57cd1c640 1 -- 172.21.15.44:0/3125364389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5780c8a70 msgr2=0x7fe5780b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:15.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.761+0000 7fe57cd1c640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5780c8a70 0x7fe5780b49f0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fe57809d9a0 tx=0x7fe570007b10 comp rx=0 tx=0).stop 2024-04-02T13:30:15.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.761+0000 7fe57cd1c640 1 -- 172.21.15.44:0/3125364389 shutdown_connections 2024-04-02T13:30:15.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.761+0000 7fe57cd1c640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe55806c8f0 0x7fe55806edb0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:15.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.761+0000 7fe57cd1c640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5780c8a70 0x7fe5780b49f0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:15.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.761+0000 7fe57cd1c640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5780c85f0 0x7fe5780b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:15.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.761+0000 7fe57cd1c640 1 --2- 172.21.15.44:0/3125364389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe564004800 0x7fe5781502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:15.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.761+0000 7fe57cd1c640 1 -- 172.21.15.44:0/3125364389 >> 172.21.15.44:0/3125364389 conn(0x7fe57809d470 msgr2=0x7fe57815c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:15.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.761+0000 7fe57cd1c640 1 -- 172.21.15.44:0/3125364389 shutdown_connections 2024-04-02T13:30:15.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:15.761+0000 7fe57cd1c640 1 -- 172.21.15.44:0/3125364389 wait complete. 2024-04-02T13:30:16.178 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:29:10.907920+0000', 'modified': '2024-04-02T13:30:14.638700+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 109, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9192}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9192': {'gid': 9192, 'name': 'a', 'rank': 0, 'incarnation': 85, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.44:6834/1745009605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6833', 'nonce': 1745009605}, {'type': 'v1', 'addr': '172.21.15.44:6834', 'nonce': 1745009605}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-02T13:30:16.178 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9192:up:rejoin 2024-04-02T13:30:17.179 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:30:17.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.290+0000 7f232fb67640 1 Processor -- start 2024-04-02T13:30:17.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.290+0000 7f232fb67640 1 -- start start 2024-04-02T13:30:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.291+0000 7f232fb67640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f23280d28f0 0x7f23280d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.291+0000 7f232fb67640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23280b47b0 0x7f23280b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.291+0000 7f232fb67640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23280d4d70 con 0x7f23280b5200 2024-04-02T13:30:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.291+0000 7f232fb67640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f23280d5050 con 0x7f23280d28f0 2024-04-02T13:30:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.291+0000 7f232fb67640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f23280d23a0 con 0x7f23280b47b0 2024-04-02T13:30:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.291+0000 7f232e104640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23280b47b0 0x7f23280b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.291+0000 7f232d903640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f23280d28f0 0x7f23280d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.291+0000 7f232e104640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23280b47b0 0x7f23280b7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:56466/0 (socket says 172.21.15.44:56466) 2024-04-02T13:30:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.291+0000 7f232d903640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f23280d28f0 0x7f23280d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:37774/0 (socket says 172.21.15.44:37774) 2024-04-02T13:30:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.291+0000 7f232e104640 1 -- 172.21.15.44:0/4131761129 learned_addr learned my addr 172.21.15.44:0/4131761129 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.291+0000 7f232d102640 1 --1- 172.21.15.44:0/4131761129 >> v1:172.21.15.69:6789/0 conn(0x7f23280b5200 0x7f23280b40a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:34628/0 (socket says 172.21.15.44:34628) 2024-04-02T13:30:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.292+0000 7f232e104640 1 -- 172.21.15.44:0/4131761129 >> v2:172.21.15.69:3300/0 conn(0x7f23280d28f0 msgr2=0x7f23280d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.292+0000 7f232e104640 1 --2- 172.21.15.44:0/4131761129 >> v2:172.21.15.69:3300/0 conn(0x7f23280d28f0 0x7f23280d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.292+0000 7f232e104640 1 -- 172.21.15.44:0/4131761129 >> v1:172.21.15.69:6789/0 conn(0x7f23280b5200 legacy=0x7f23280b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.292+0000 7f232e104640 1 -- 172.21.15.44:0/4131761129 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23280d2030 con 0x7f23280b47b0 2024-04-02T13:30:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.292+0000 7f232d903640 1 --2- 172.21.15.44:0/4131761129 >> v2:172.21.15.69:3300/0 conn(0x7f23280d28f0 0x7f23280d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:30:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.292+0000 7f232e104640 1 --2- 172.21.15.44:0/4131761129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23280b47b0 0x7f23280b7120 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2320009e50 tx=0x7f232002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=248853efd25c7a80 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.293+0000 7f232c901640 1 -- 172.21.15.44:0/4131761129 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232002c9e0 con 0x7f23280b47b0 2024-04-02T13:30:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.293+0000 7f232c901640 1 -- 172.21.15.44:0/4131761129 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f232002cb80 con 0x7f23280b47b0 2024-04-02T13:30:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.293+0000 7f232c901640 1 -- 172.21.15.44:0/4131761129 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23200343f0 con 0x7f23280b47b0 2024-04-02T13:30:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.293+0000 7f232fb67640 1 -- 172.21.15.44:0/4131761129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23280b47b0 msgr2=0x7f23280b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.293+0000 7f232fb67640 1 --2- 172.21.15.44:0/4131761129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23280b47b0 0x7f23280b7120 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2320009e50 tx=0x7f232002be40 comp rx=0 tx=0).stop 2024-04-02T13:30:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.293+0000 7f232fb67640 1 -- 172.21.15.44:0/4131761129 shutdown_connections 2024-04-02T13:30:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.293+0000 7f232fb67640 1 --2- 172.21.15.44:0/4131761129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23280b47b0 0x7f23280b7120 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.293+0000 7f232fb67640 1 --2- 172.21.15.44:0/4131761129 >> v2:172.21.15.69:3300/0 conn(0x7f23280d28f0 0x7f23280d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.293+0000 7f232fb67640 1 -- 172.21.15.44:0/4131761129 >> 172.21.15.44:0/4131761129 conn(0x7f23280d4120 msgr2=0x7f23280d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.293+0000 7f232fb67640 1 -- 172.21.15.44:0/4131761129 shutdown_connections 2024-04-02T13:30:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.293+0000 7f232fb67640 1 -- 172.21.15.44:0/4131761129 wait complete. 2024-04-02T13:30:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.295+0000 7f232fb67640 1 Processor -- start 2024-04-02T13:30:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.295+0000 7f232fb67640 1 -- start start 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.295+0000 7f232fb67640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23280b47b0 0x7f23280b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.295+0000 7f232fb67640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23280b5200 0x7f232815b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.295+0000 7f232fb67640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23280d28f0 0x7f232815ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.295+0000 7f232fb67640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f23280d1260 con 0x7f23280b47b0 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.295+0000 7f232fb67640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f23280b5020 con 0x7f23280d28f0 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.295+0000 7f232fb67640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f23280d13e0 con 0x7f23280b5200 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.295+0000 7f232d903640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23280b47b0 0x7f23280b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.295+0000 7f232d903640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23280b47b0 0x7f23280b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45998/0 (socket says 172.21.15.44:45998) 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.295+0000 7f232d903640 1 -- 172.21.15.44:0/2337617474 learned_addr learned my addr 172.21.15.44:0/2337617474 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.296+0000 7f232e104640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23280d28f0 0x7f232815ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.296+0000 7f232d102640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23280b5200 0x7f232815b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.296+0000 7f232d903640 1 -- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23280b5200 msgr2=0x7f232815b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.296+0000 7f232d903640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23280b5200 0x7f232815b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.296+0000 7f232d903640 1 -- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23280d28f0 msgr2=0x7f232815ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.296+0000 7f232d903640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23280d28f0 0x7f232815ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.296+0000 7f232d903640 1 -- 172.21.15.44:0/2337617474 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23280b2390 con 0x7f23280b47b0 2024-04-02T13:30:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.296+0000 7f232d903640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23280b47b0 0x7f23280b1d00 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f232400e960 tx=0x7f232400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.296+0000 7f231e7fc640 1 -- 172.21.15.44:0/2337617474 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2324017070 con 0x7f23280b47b0 2024-04-02T13:30:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.296+0000 7f231e7fc640 1 -- 172.21.15.44:0/2337617474 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f232400ca00 con 0x7f23280b47b0 2024-04-02T13:30:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.296+0000 7f232fb67640 1 -- 172.21.15.44:0/2337617474 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23280b2670 con 0x7f23280b47b0 2024-04-02T13:30:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.296+0000 7f231e7fc640 1 -- 172.21.15.44:0/2337617474 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232400cdd0 con 0x7f23280b47b0 2024-04-02T13:30:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.296+0000 7f232fb67640 1 -- 172.21.15.44:0/2337617474 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23280b2c30 con 0x7f23280b47b0 2024-04-02T13:30:17.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.297+0000 7f232fb67640 1 -- 172.21.15.44:0/2337617474 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23280684f0 con 0x7f23280b47b0 2024-04-02T13:30:17.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.298+0000 7f231e7fc640 1 -- 172.21.15.44:0/2337617474 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f232401a070 con 0x7f23280b47b0 2024-04-02T13:30:17.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.298+0000 7f231e7fc640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f230c06c8f0 0x7f230c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:17.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.298+0000 7f231e7fc640 1 -- 172.21.15.44:0/2337617474 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f232408bf00 con 0x7f23280b47b0 2024-04-02T13:30:17.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.298+0000 7f232d102640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f230c06c8f0 0x7f230c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:17.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.299+0000 7f232d102640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f230c06c8f0 0x7f230c06edb0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f23280b3900 tx=0x7f2318007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:17.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.303+0000 7f231e7fc640 1 -- 172.21.15.44:0/2337617474 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2324059a90 con 0x7f23280b47b0 2024-04-02T13:30:17.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.527+0000 7f232fb67640 1 -- 172.21.15.44:0/2337617474 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23280b8fa0 con 0x7f23280b47b0 2024-04-02T13:30:17.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.528+0000 7f231e7fc640 1 -- 172.21.15.44:0/2337617474 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+5185 (secure 0 0 0) 0x7f232405dd90 con 0x7f23280b47b0 2024-04-02T13:30:17.938 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:17.938 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":88,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7696,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2715855087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2715855087},{"type":"v1","addr":"172.21.15.69:6836","nonce":2715855087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7699,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3850680553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3850680553},{"type":"v1","addr":"172.21.15.69:6837","nonce":3850680553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/1330712383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1330712383},{"type":"v1","addr":"172.21.15.44:6835","nonce":1330712383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:29:10.907920+0000","modified":"2024-04-02T13:30:16.638019+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9192":{"gid":9192,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6834/1745009605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":1745009605},{"type":"v1","addr":"172.21.15.44:6834","nonce":1745009605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9192,"qdb_cluster":[9192]},"id":7}]} 2024-04-02T13:30:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 88 2024-04-02T13:30:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.530+0000 7f232fb67640 1 -- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f230c06c8f0 msgr2=0x7f230c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.530+0000 7f232fb67640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f230c06c8f0 0x7f230c06edb0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f23280b3900 tx=0x7f2318007e80 comp rx=0 tx=0).stop 2024-04-02T13:30:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.530+0000 7f232fb67640 1 -- 172.21.15.44:0/2337617474 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23280b47b0 msgr2=0x7f23280b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.530+0000 7f232fb67640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23280b47b0 0x7f23280b1d00 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f232400e960 tx=0x7f232400ee20 comp rx=0 tx=0).stop 2024-04-02T13:30:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.530+0000 7f232fb67640 1 -- 172.21.15.44:0/2337617474 shutdown_connections 2024-04-02T13:30:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.530+0000 7f232fb67640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f230c06c8f0 0x7f230c06edb0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.530+0000 7f232fb67640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23280d28f0 0x7f232815ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.530+0000 7f232fb67640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23280b5200 0x7f232815b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.530+0000 7f232fb67640 1 --2- 172.21.15.44:0/2337617474 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23280b47b0 0x7f23280b1d00 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.530+0000 7f232fb67640 1 -- 172.21.15.44:0/2337617474 >> 172.21.15.44:0/2337617474 conn(0x7f23280d4120 msgr2=0x7f23280b7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.530+0000 7f232fb67640 1 -- 172.21.15.44:0/2337617474 shutdown_connections 2024-04-02T13:30:17.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:17.530+0000 7f232fb67640 1 -- 172.21.15.44:0/2337617474 wait complete. 2024-04-02T13:30:18.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:29:10.907920+0000', 'modified': '2024-04-02T13:30:16.638019+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 109, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9192}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9192': {'gid': 9192, 'name': 'a', 'rank': 0, 'incarnation': 85, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.44:6834/1745009605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6833', 'nonce': 1745009605}, {'type': 'v1', 'addr': '172.21.15.44:6834', 'nonce': 1745009605}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9192, 'qdb_cluster': [9192]} 2024-04-02T13:30:18.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:30:18.344 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T13:30:18.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.456+0000 7f2b6ff1c640 1 Processor -- start 2024-04-02T13:30:18.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.456+0000 7f2b6ff1c640 1 -- start start 2024-04-02T13:30:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6ff1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b6806cf90 0x7f2b680cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6ff1c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2b680cfe30 0x7f2b680d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6ff1c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b6806c1d0 con 0x7f2b680d0a30 2024-04-02T13:30:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6ff1c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2b6806cb00 con 0x7f2b680cfe30 2024-04-02T13:30:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6ff1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2b6806d3f0 con 0x7f2b6806cf90 2024-04-02T13:30:18.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6e4b9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2b680cfe30 0x7f2b680d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6dcb8640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f2b680d0a30 0x7f2b680d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:50402/0 (socket says 172.21.15.44:50402) 2024-04-02T13:30:18.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6dcb8640 1 -- 172.21.15.44:0/3695014549 learned_addr learned my addr 172.21.15.44:0/3695014549 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:18.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6d4b7640 1 --2- 172.21.15.44:0/3695014549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b6806cf90 0x7f2b680cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/3695014549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b6806cf90 msgr2=0x7f2b680cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6e4b9640 1 --2- 172.21.15.44:0/3695014549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b6806cf90 0x7f2b680cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/3695014549 >> v1:172.21.15.44:6789/0 conn(0x7f2b680d0a30 legacy=0x7f2b680d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6ccb6640 1 -- 172.21.15.44:0/3695014549 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 503273679 0 0) 0x7f2b6806c1d0 con 0x7f2b680d0a30 2024-04-02T13:30:18.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.457+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/3695014549 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b6806c4b0 con 0x7f2b680cfe30 2024-04-02T13:30:18.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.458+0000 7f2b6e4b9640 1 --2- 172.21.15.44:0/3695014549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b680cfe30 0x7f2b680d2040 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009d90 tx=0x7f2b6002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a21e18ffc380abd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.458+0000 7f2b6ccb6640 1 -- 172.21.15.44:0/3695014549 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60039070 con 0x7f2b680cfe30 2024-04-02T13:30:18.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.458+0000 7f2b6ccb6640 1 -- 172.21.15.44:0/3695014549 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b6002cbc0 con 0x7f2b680cfe30 2024-04-02T13:30:18.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.458+0000 7f2b6ccb6640 1 -- 172.21.15.44:0/3695014549 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60034440 con 0x7f2b680cfe30 2024-04-02T13:30:18.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.459+0000 7f2b6ff1c640 1 -- 172.21.15.44:0/3695014549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b680cfe30 msgr2=0x7f2b680d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.459+0000 7f2b6ff1c640 1 --2- 172.21.15.44:0/3695014549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b680cfe30 0x7f2b680d2040 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009d90 tx=0x7f2b6002be10 comp rx=0 tx=0).stop 2024-04-02T13:30:18.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.459+0000 7f2b6ff1c640 1 -- 172.21.15.44:0/3695014549 shutdown_connections 2024-04-02T13:30:18.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.459+0000 7f2b6ff1c640 1 --2- 172.21.15.44:0/3695014549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b680cfe30 0x7f2b680d2040 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.459+0000 7f2b6ff1c640 1 --2- 172.21.15.44:0/3695014549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b6806cf90 0x7f2b680cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.459+0000 7f2b6ff1c640 1 -- 172.21.15.44:0/3695014549 >> 172.21.15.44:0/3695014549 conn(0x7f2b6809d470 msgr2=0x7f2b680d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.459+0000 7f2b6ff1c640 1 -- 172.21.15.44:0/3695014549 shutdown_connections 2024-04-02T13:30:18.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.459+0000 7f2b6ff1c640 1 -- 172.21.15.44:0/3695014549 wait complete. 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.460+0000 7f2b6ff1c640 1 Processor -- start 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.460+0000 7f2b6ff1c640 1 -- start start 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6ff1c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b6806cf90 0x7f2b68158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6ff1c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b680cfe30 0x7f2b680b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6ff1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b680d0a30 0x7f2b680b5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6ff1c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6815e870 con 0x7f2b680cfe30 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6ff1c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6815e6f0 con 0x7f2b6806cf90 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6ff1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2b680d4da0 con 0x7f2b680d0a30 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6d4b7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b680cfe30 0x7f2b680b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6d4b7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b680cfe30 0x7f2b680b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46020/0 (socket says 172.21.15.44:46020) 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6d4b7640 1 -- 172.21.15.44:0/1696639007 learned_addr learned my addr 172.21.15.44:0/1696639007 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6e4b9640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b680d0a30 0x7f2b680b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6dcb8640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b6806cf90 0x7f2b68158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6d4b7640 1 -- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b680d0a30 msgr2=0x7f2b680b5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6d4b7640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b680d0a30 0x7f2b680b5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6d4b7640 1 -- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b6806cf90 msgr2=0x7f2b68158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6d4b7640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b6806cf90 0x7f2b68158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6d4b7640 1 -- 172.21.15.44:0/1696639007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b681594f0 con 0x7f2b680cfe30 2024-04-02T13:30:18.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.461+0000 7f2b6d4b7640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b680cfe30 0x7f2b680b42a0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f2b5800d600 tx=0x7f2b5800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.462+0000 7f2b5e7fc640 1 -- 172.21.15.44:0/1696639007 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b58004240 con 0x7f2b680cfe30 2024-04-02T13:30:18.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.462+0000 7f2b5e7fc640 1 -- 172.21.15.44:0/1696639007 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b580043e0 con 0x7f2b680cfe30 2024-04-02T13:30:18.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.462+0000 7f2b5e7fc640 1 -- 172.21.15.44:0/1696639007 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b58003830 con 0x7f2b680cfe30 2024-04-02T13:30:18.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.462+0000 7f2b6ff1c640 1 -- 172.21.15.44:0/1696639007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b680b6290 con 0x7f2b680cfe30 2024-04-02T13:30:18.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.462+0000 7f2b6ff1c640 1 -- 172.21.15.44:0/1696639007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b680b6850 con 0x7f2b680cfe30 2024-04-02T13:30:18.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.463+0000 7f2b5e7fc640 1 -- 172.21.15.44:0/1696639007 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2b580039d0 con 0x7f2b680cfe30 2024-04-02T13:30:18.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.463+0000 7f2b5e7fc640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2b4c06c8f0 0x7f2b4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.464+0000 7f2b5e7fc640 1 -- 172.21.15.44:0/1696639007 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f2b5808c020 con 0x7f2b680cfe30 2024-04-02T13:30:18.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.464+0000 7f2b6dcb8640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2b4c06c8f0 0x7f2b4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.464+0000 7f2b6dcb8640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2b4c06c8f0 0x7f2b4c06edb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000f60 tx=0x7f2b64007c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.465+0000 7f2b72be3740 1 Processor -- start 2024-04-02T13:30:18.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.465+0000 7f2b72be3740 1 -- start start 2024-04-02T13:30:18.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.465+0000 7f2b72be3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc0fe8b90 0x55ebc0fbbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.465+0000 7f2b72be3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8f70 0x55ebc0feb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.465+0000 7f2b72be3740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fbc070 0x55ebc0febe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.465+0000 7f2b72be3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ebc0f25060 con 0x55ebc0fe8b90 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.465+0000 7f2b72be3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ebc0ecd650 con 0x55ebc0fe8f70 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.465+0000 7f2b72be3740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ebc0f21010 con 0x55ebc0fbc070 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b6dcb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc0fe8b90 0x55ebc0fbbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b6dcb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc0fe8b90 0x55ebc0fbbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46034/0 (socket says 172.21.15.44:46034) 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b6dcb8640 1 -- 172.21.15.44:0/3566935827 learned_addr learned my addr 172.21.15.44:0/3566935827 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b6e4b9640 1 --2- 172.21.15.44:0/3566935827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fbc070 0x55ebc0febe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b6d4b7640 1 --2- 172.21.15.44:0/3566935827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8f70 0x55ebc0feb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b6dcb8640 1 -- 172.21.15.44:0/3566935827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fbc070 msgr2=0x55ebc0febe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b6dcb8640 1 --2- 172.21.15.44:0/3566935827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fbc070 0x55ebc0febe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b6dcb8640 1 -- 172.21.15.44:0/3566935827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8f70 msgr2=0x55ebc0feb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b6dcb8640 1 --2- 172.21.15.44:0/3566935827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8f70 0x55ebc0feb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b6dcb8640 1 -- 172.21.15.44:0/3566935827 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc0c392d0 con 0x55ebc0fe8b90 2024-04-02T13:30:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b6dcb8640 1 --2- 172.21.15.44:0/3566935827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc0fe8b90 0x55ebc0fbbb30 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f2b64005db0 tx=0x7f2b64012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479c1462918d9c06 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b4b7fe640 1 -- 172.21.15.44:0/3566935827 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b64015110 con 0x55ebc0fe8b90 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b4b7fe640 1 -- 172.21.15.44:0/3566935827 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b640159c0 con 0x55ebc0fe8b90 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.466+0000 7f2b4b7fe640 1 -- 172.21.15.44:0/3566935827 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b64015d70 con 0x55ebc0fe8b90 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.467+0000 7f2b72be3740 1 -- 172.21.15.44:0/3566935827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc0fe8b90 msgr2=0x55ebc0fbbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.467+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3566935827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc0fe8b90 0x55ebc0fbbb30 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f2b64005db0 tx=0x7f2b64012000 comp rx=0 tx=0).stop 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.467+0000 7f2b72be3740 1 -- 172.21.15.44:0/3566935827 shutdown_connections 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.467+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3566935827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fbc070 0x55ebc0febe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.467+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3566935827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8f70 0x55ebc0feb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.467+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3566935827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc0fe8b90 0x55ebc0fbbb30 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.467+0000 7f2b72be3740 1 -- 172.21.15.44:0/3566935827 >> 172.21.15.44:0/3566935827 conn(0x55ebc0fbdda0 msgr2=0x55ebc1006920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.467+0000 7f2b72be3740 1 -- 172.21.15.44:0/3566935827 shutdown_connections 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.467+0000 7f2b72be3740 1 -- 172.21.15.44:0/3566935827 wait complete. 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.468+0000 7f2b72be3740 1 -- start start 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.468+0000 7f2b72be3740 1 Processor -- start 2024-04-02T13:30:18.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b72be3740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8b90 0x55ebc0fed4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b72be3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8f70 0x55ebc101a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b72be3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc10161a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b72be3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ebc0fc0440 con 0x55ebc101a9c0 2024-04-02T13:30:18.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b72be3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ebc0ee0a10 con 0x55ebc0fe8f70 2024-04-02T13:30:18.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b72be3740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ebc0fc2c80 con 0x55ebc0fe8b90 2024-04-02T13:30:18.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b72be3740 10 client.0 _ll_register_callbacks cb 0x55ebc0f1d0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:30:18.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b6e4b9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc10161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b6e4b9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc10161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46042/0 (socket says 172.21.15.44:46042) 2024-04-02T13:30:18.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/3837383661 learned_addr learned my addr 172.21.15.44:0/3837383661 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b6d4b7640 1 --2- 172.21.15.44:0/3837383661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8f70 0x55ebc101a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b6dcb8640 1 --2- 172.21.15.44:0/3837383661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8b90 0x55ebc0fed4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/3837383661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8b90 msgr2=0x55ebc0fed4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b6e4b9640 1 --2- 172.21.15.44:0/3837383661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8b90 0x55ebc0fed4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/3837383661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8f70 msgr2=0x55ebc101a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b6e4b9640 1 --2- 172.21.15.44:0/3837383661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8f70 0x55ebc101a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/3837383661 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc0d036b0 con 0x55ebc101a9c0 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b6e4b9640 1 --2- 172.21.15.44:0/3837383661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc10161a0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009d90 tx=0x7f2b60004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8deaea34bbc2edca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.469+0000 7f2b48ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b48ff9640 1 -- 172.21.15.44:0/3837383661 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60039040 con 0x55ebc101a9c0 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b48ff9640 1 -- 172.21.15.44:0/3837383661 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f2b60034d00 con 0x55ebc101a9c0 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b72be3740 1 -- 172.21.15.44:0/3837383661 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b640159c0 con 0x55ebc101a9c0 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b48ff9640 1 -- 172.21.15.44:0/3837383661 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f2b60052050 con 0x55ebc101a9c0 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b72be3740 10 client.9267 fetch_fsmap learned FSMap version 88 2024-04-02T13:30:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b72be3740 1 -- 172.21.15.44:0/3837383661 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) -- 0x55ebc0fedd00 con 0x55ebc101a9c0 2024-04-02T13:30:18.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b48ff9640 1 -- 172.21.15.44:0/3837383661 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b640159c0 con 0x55ebc101a9c0 2024-04-02T13:30:18.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b48ff9640 1 -- 172.21.15.44:0/3837383661 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 88) ==== 3494+0+0 (secure 0 0 0) 0x7f2b60055020 con 0x55ebc101a9c0 2024-04-02T13:30:18.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b72be3740 10 client.9267 fetch_fsmap finished waiting for FSMap version 88 2024-04-02T13:30:18.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b72be3740 10 client.9267 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:30:18.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b72be3740 20 client.9267 populate_metadata read hostname 'smithi044' 2024-04-02T13:30:18.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3837383661 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55ebc10056a0 0x55ebc1005a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b72be3740 4 client.9267 mds_command: new command op to 9192 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:30:18.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b72be3740 1 -- 172.21.15.44:0/3837383661 --> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebc0ecccf0 con 0x55ebc10056a0 2024-04-02T13:30:18.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.470+0000 7f2b6dcb8640 1 --2- 172.21.15.44:0/3837383661 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55ebc10056a0 0x55ebc1005a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.471+0000 7f2b48ff9640 10 client.9267 ms_handle_connect on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:18.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.471+0000 7f2b6dcb8640 1 --2- 172.21.15.44:0/3837383661 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55ebc10056a0 0x55ebc1005a80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3567da97951cad3 server_cookie=1c6c813dbd8074bb in_seq=0 out_seq=0 2024-04-02T13:30:18.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.471+0000 7f2b48ff9640 1 -- 172.21.15.44:0/3837383661 <== mds.0 v2:172.21.15.44:6833/1745009605 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ebc0ecccf0 con 0x55ebc10056a0 2024-04-02T13:30:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.471+0000 7f2b48ff9640 10 client.9267 handle_command_reply: tid=0 2024-04-02T13:30:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.471+0000 7f2b72be3740 1 client.9267 shutdown 2024-04-02T13:30:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.471+0000 7f2b72be3740 1 -- 172.21.15.44:0/3837383661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 msgr2=0x55ebc10161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.471+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3837383661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc10161a0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009d90 tx=0x7f2b60004ab0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.472+0000 7f2b72be3740 1 -- 172.21.15.44:0/3837383661 shutdown_connections 2024-04-02T13:30:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.472+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3837383661 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55ebc10056a0 0x55ebc1005a80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.472+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3837383661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc10161a0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.472+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3837383661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8f70 0x55ebc101a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.472+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3837383661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8b90 0x55ebc0fed4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.472+0000 7f2b48ff9640 0 client.9267 ms_handle_reset on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.472+0000 7f2b72be3740 1 -- 172.21.15.44:0/3837383661 >> 172.21.15.44:0/3837383661 conn(0x55ebc0fbc070 msgr2=0x55ebc0fbfc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.472+0000 7f2b72be3740 1 -- 172.21.15.44:0/3837383661 shutdown_connections 2024-04-02T13:30:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.472+0000 7f2b72be3740 1 -- 172.21.15.44:0/3837383661 wait complete. 2024-04-02T13:30:18.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.472+0000 7f2b72be3740 20 client.9267 trim_cache size 0 max 16384 2024-04-02T13:30:18.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.490+0000 7f2b72be3740 1 Processor -- start 2024-04-02T13:30:18.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.490+0000 7f2b72be3740 1 -- start start 2024-04-02T13:30:18.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b72be3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8b90 0x55ebc0fbfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b72be3740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8f70 0x55ebc1005a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b72be3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc0fee600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b72be3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2b681fffe0 con 0x55ebc101a9c0 2024-04-02T13:30:18.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b72be3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2b681e8ae0 con 0x55ebc0fe8b90 2024-04-02T13:30:18.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b72be3740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2b680b74e0 con 0x55ebc0fe8f70 2024-04-02T13:30:18.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b6e4b9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc0fee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b6e4b9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc0fee600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46050/0 (socket says 172.21.15.44:46050) 2024-04-02T13:30:18.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/3196738352 learned_addr learned my addr 172.21.15.44:0/3196738352 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b6d4b7640 1 --2- 172.21.15.44:0/3196738352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8f70 0x55ebc1005a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b6dcb8640 1 --2- 172.21.15.44:0/3196738352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8b90 0x55ebc0fbfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/3196738352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8f70 msgr2=0x55ebc1005a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b6e4b9640 1 --2- 172.21.15.44:0/3196738352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8f70 0x55ebc1005a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/3196738352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8b90 msgr2=0x55ebc0fbfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b6e4b9640 1 --2- 172.21.15.44:0/3196738352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8b90 0x55ebc0fbfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/3196738352 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc102b7e0 con 0x55ebc101a9c0 2024-04-02T13:30:18.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.491+0000 7f2b6d4b7640 1 --2- 172.21.15.44:0/3196738352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8f70 0x55ebc1005a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:18.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.492+0000 7f2b6e4b9640 1 --2- 172.21.15.44:0/3196738352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc0fee600 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f2b60034550 tx=0x7f2b60041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33d792584c344f8f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.492+0000 7f2b4affd640 1 -- 172.21.15.44:0/3196738352 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6005b070 con 0x55ebc101a9c0 2024-04-02T13:30:18.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.492+0000 7f2b4affd640 1 -- 172.21.15.44:0/3196738352 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b6003a040 con 0x55ebc101a9c0 2024-04-02T13:30:18.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.492+0000 7f2b4affd640 1 -- 172.21.15.44:0/3196738352 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60039070 con 0x55ebc101a9c0 2024-04-02T13:30:18.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.492+0000 7f2b72be3740 1 -- 172.21.15.44:0/3196738352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 msgr2=0x55ebc0fee600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.492+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3196738352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc0fee600 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f2b60034550 tx=0x7f2b60041000 comp rx=0 tx=0).stop 2024-04-02T13:30:18.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.492+0000 7f2b72be3740 1 -- 172.21.15.44:0/3196738352 shutdown_connections 2024-04-02T13:30:18.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.492+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3196738352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc0fee600 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.492+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3196738352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8f70 0x55ebc1005a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.492+0000 7f2b72be3740 1 --2- 172.21.15.44:0/3196738352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8b90 0x55ebc0fbfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.492+0000 7f2b72be3740 1 -- 172.21.15.44:0/3196738352 >> 172.21.15.44:0/3196738352 conn(0x55ebc0fbc070 msgr2=0x55ebc0fef2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.492+0000 7f2b72be3740 1 -- 172.21.15.44:0/3196738352 shutdown_connections 2024-04-02T13:30:18.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.492+0000 7f2b72be3740 1 -- 172.21.15.44:0/3196738352 wait complete. 2024-04-02T13:30:18.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.493+0000 7f2b72be3740 1 -- start start 2024-04-02T13:30:18.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.493+0000 7f2b72be3740 1 Processor -- start 2024-04-02T13:30:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b72be3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8b90 0x55ebc1080f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b72be3740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8f70 0x55ebc1081440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b72be3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc1014ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b72be3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ebc1006720 con 0x55ebc101a9c0 2024-04-02T13:30:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b72be3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ebc0feb3d0 con 0x55ebc0fe8b90 2024-04-02T13:30:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b72be3740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ebc0fecce0 con 0x55ebc0fe8f70 2024-04-02T13:30:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b72be3740 10 client.0 _ll_register_callbacks cb 0x7f2b60009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:30:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b6e4b9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc1014ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b6e4b9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc1014ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46056/0 (socket says 172.21.15.44:46056) 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/888539088 learned_addr learned my addr 172.21.15.44:0/888539088 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b6dcb8640 1 --2- 172.21.15.44:0/888539088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8b90 0x55ebc1080f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b6d4b7640 1 --2- 172.21.15.44:0/888539088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8f70 0x55ebc1081440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/888539088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8f70 msgr2=0x55ebc1081440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b6e4b9640 1 --2- 172.21.15.44:0/888539088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8f70 0x55ebc1081440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/888539088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8b90 msgr2=0x55ebc1080f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b6e4b9640 1 --2- 172.21.15.44:0/888539088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8b90 0x55ebc1080f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.494+0000 7f2b6e4b9640 1 -- 172.21.15.44:0/888539088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc1004fd0 con 0x55ebc101a9c0 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b6e4b9640 1 --2- 172.21.15.44:0/888539088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc1014ff0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009920 tx=0x7f2b60009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567d7ba3b68c08b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b48ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b48ff9640 1 -- 172.21.15.44:0/888539088 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6005b040 con 0x55ebc101a9c0 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b48ff9640 1 -- 172.21.15.44:0/888539088 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f2b6004cd70 con 0x55ebc101a9c0 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b72be3740 1 -- 172.21.15.44:0/888539088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b6003a040 con 0x55ebc101a9c0 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b48ff9640 1 -- 172.21.15.44:0/888539088 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f2b60039040 con 0x55ebc101a9c0 2024-04-02T13:30:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b72be3740 10 client.9273 fetch_fsmap learned FSMap version 88 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b72be3740 1 -- 172.21.15.44:0/888539088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) -- 0x55ebc1004cb0 con 0x55ebc101a9c0 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b48ff9640 1 -- 172.21.15.44:0/888539088 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b6003a040 con 0x55ebc101a9c0 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b48ff9640 1 -- 172.21.15.44:0/888539088 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 88) ==== 3494+0+0 (secure 0 0 0) 0x7f2b60055050 con 0x55ebc101a9c0 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b72be3740 10 client.9273 fetch_fsmap finished waiting for FSMap version 88 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b72be3740 10 client.9273 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b72be3740 20 client.9273 populate_metadata read hostname 'smithi044' 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b72be3740 1 --2- 172.21.15.44:0/888539088 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55ebc10218b0 0x55ebc1084b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b72be3740 4 client.9273 mds_command: new command op to 9192 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b72be3740 1 -- 172.21.15.44:0/888539088 --> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] -- command(tid 0: {"prefix": "status"}) -- 0x55ebc0f325a0 con 0x55ebc10218b0 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.495+0000 7f2b6dcb8640 1 --2- 172.21.15.44:0/888539088 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55ebc10218b0 0x55ebc1084b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b48ff9640 10 client.9273 ms_handle_connect on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b6dcb8640 1 --2- 172.21.15.44:0/888539088 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55ebc10218b0 0x55ebc1084b20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac1bc18b7413128e server_cookie=67f736690e4f283d in_seq=0 out_seq=0 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b48ff9640 1 -- 172.21.15.44:0/888539088 <== mds.0 v2:172.21.15.44:6833/1745009605 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x55ebc0f325a0 con 0x55ebc10218b0 2024-04-02T13:30:18.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b48ff9640 10 client.9273 handle_command_reply: tid=0 2024-04-02T13:30:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b72be3740 1 client.9273 shutdown 2024-04-02T13:30:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b72be3740 1 -- 172.21.15.44:0/888539088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 msgr2=0x55ebc1014ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b72be3740 1 --2- 172.21.15.44:0/888539088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc1014ff0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009920 tx=0x7f2b60009750 comp rx=0 tx=0).stop 2024-04-02T13:30:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b72be3740 1 -- 172.21.15.44:0/888539088 shutdown_connections 2024-04-02T13:30:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b72be3740 1 --2- 172.21.15.44:0/888539088 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55ebc10218b0 0x55ebc1084b20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b72be3740 1 --2- 172.21.15.44:0/888539088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ebc101a9c0 0x55ebc1014ff0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b72be3740 1 --2- 172.21.15.44:0/888539088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ebc0fe8f70 0x55ebc1081440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b48ff9640 0 client.9273 ms_handle_reset on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b72be3740 1 --2- 172.21.15.44:0/888539088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ebc0fe8b90 0x55ebc1080f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b72be3740 1 -- 172.21.15.44:0/888539088 >> 172.21.15.44:0/888539088 conn(0x55ebc0fbc070 msgr2=0x55ebc0fc1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b72be3740 1 -- 172.21.15.44:0/888539088 shutdown_connections 2024-04-02T13:30:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b72be3740 1 -- 172.21.15.44:0/888539088 wait complete. 2024-04-02T13:30:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.496+0000 7f2b72be3740 20 client.9273 trim_cache size 0 max 16384 2024-04-02T13:30:18.507 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:30:18.507 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:30:18.507 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:30:18.507 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 9192, 2024-04-02T13:30:18.507 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:30:18.507 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:30:18.507 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:30:18.507 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 4.8595477330000003, 2024-04-02T13:30:18.507 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 88, 2024-04-02T13:30:18.507 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 109, 2024-04-02T13:30:18.507 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 109, 2024-04-02T13:30:18.507 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 5.8294167479999999 2024-04-02T13:30:18.508 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:30:18.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.497+0000 7f2b6ff1c640 1 -- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2b4c06c8f0 msgr2=0x7f2b4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.497+0000 7f2b6ff1c640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2b4c06c8f0 0x7f2b4c06edb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000f60 tx=0x7f2b64007c00 comp rx=0 tx=0).stop 2024-04-02T13:30:18.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.497+0000 7f2b6ff1c640 1 -- 172.21.15.44:0/1696639007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b680cfe30 msgr2=0x7f2b680b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.497+0000 7f2b6ff1c640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b680cfe30 0x7f2b680b42a0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f2b5800d600 tx=0x7f2b5800dac0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.497+0000 7f2b6ff1c640 1 -- 172.21.15.44:0/1696639007 shutdown_connections 2024-04-02T13:30:18.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.497+0000 7f2b6ff1c640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2b4c06c8f0 0x7f2b4c06edb0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.497+0000 7f2b6ff1c640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b680d0a30 0x7f2b680b5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.497+0000 7f2b6ff1c640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b680cfe30 0x7f2b680b42a0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.497+0000 7f2b6ff1c640 1 --2- 172.21.15.44:0/1696639007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b6806cf90 0x7f2b68158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.497+0000 7f2b6ff1c640 1 -- 172.21.15.44:0/1696639007 >> 172.21.15.44:0/1696639007 conn(0x7f2b6809d470 msgr2=0x7f2b680b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.497+0000 7f2b6ff1c640 1 -- 172.21.15.44:0/1696639007 shutdown_connections 2024-04-02T13:30:18.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.497+0000 7f2b6ff1c640 1 -- 172.21.15.44:0/1696639007 wait complete. 2024-04-02T13:30:18.511 DEBUG:teuthology.orchestra.run.smithi044:> sudo daemon-helper kill ceph --cluster ceph -w 2024-04-02T13:30:18.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:30:18.513 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub start / repair,recursive 2024-04-02T13:30:18.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.616+0000 7f4a91d00640 1 Processor -- start 2024-04-02T13:30:18.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.616+0000 7f4a91d00640 1 -- start start 2024-04-02T13:30:18.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a91d00640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4a8c15c1a0 0x7f4a8c06f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a91d00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4a8c0cfd60 0x7f4a8c0d1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a91d00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a8c0d1ff0 0x7f4a8c0ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a91d00640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4a8c070ad0 con 0x7f4a8c15c1a0 2024-04-02T13:30:18.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a91d00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4a8c15c580 con 0x7f4a8c0cfd60 2024-04-02T13:30:18.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a91d00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4a8c0d01c0 con 0x7f4a8c0d1ff0 2024-04-02T13:30:18.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a8b7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4a8c15c1a0 0x7f4a8c06f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a8b7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4a8c15c1a0 0x7f4a8c06f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46070/0 (socket says 172.21.15.44:46070) 2024-04-02T13:30:18.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/3990949973 learned_addr learned my addr 172.21.15.44:0/3990949973 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a8affd640 1 --2- 172.21.15.44:0/3990949973 >> v2:172.21.15.69:3300/0 conn(0x7f4a8c0cfd60 0x7f4a8c0d1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a8bfff640 1 --2- 172.21.15.44:0/3990949973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a8c0d1ff0 0x7f4a8c0ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/3990949973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a8c0d1ff0 msgr2=0x7f4a8c0ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/3990949973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a8c0d1ff0 0x7f4a8c0ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/3990949973 >> v2:172.21.15.69:3300/0 conn(0x7f4a8c0cfd60 msgr2=0x7f4a8c0d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/3990949973 >> v2:172.21.15.69:3300/0 conn(0x7f4a8c0cfd60 0x7f4a8c0d1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.617+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/3990949973 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c0d2480 con 0x7f4a8c15c1a0 2024-04-02T13:30:18.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.618+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/3990949973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a8c15c1a0 0x7f4a8c06f370 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0d2f10 tx=0x7f4a8002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e004de6e3f3e4db8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.618+0000 7f4a8a7fc640 1 -- 172.21.15.44:0/3990949973 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80035020 con 0x7f4a8c15c1a0 2024-04-02T13:30:18.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.618+0000 7f4a8a7fc640 1 -- 172.21.15.44:0/3990949973 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a80031870 con 0x7f4a8c15c1a0 2024-04-02T13:30:18.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.618+0000 7f4a8a7fc640 1 -- 172.21.15.44:0/3990949973 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80031c40 con 0x7f4a8c15c1a0 2024-04-02T13:30:18.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.619+0000 7f4a91d00640 1 -- 172.21.15.44:0/3990949973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a8c15c1a0 msgr2=0x7f4a8c06f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.619+0000 7f4a91d00640 1 --2- 172.21.15.44:0/3990949973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a8c15c1a0 0x7f4a8c06f370 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0d2f10 tx=0x7f4a8002b910 comp rx=0 tx=0).stop 2024-04-02T13:30:18.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.619+0000 7f4a91d00640 1 -- 172.21.15.44:0/3990949973 shutdown_connections 2024-04-02T13:30:18.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.619+0000 7f4a91d00640 1 --2- 172.21.15.44:0/3990949973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a8c0d1ff0 0x7f4a8c0ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.619+0000 7f4a91d00640 1 --2- 172.21.15.44:0/3990949973 >> v2:172.21.15.69:3300/0 conn(0x7f4a8c0cfd60 0x7f4a8c0d1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.619+0000 7f4a91d00640 1 --2- 172.21.15.44:0/3990949973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a8c15c1a0 0x7f4a8c06f370 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.619+0000 7f4a91d00640 1 -- 172.21.15.44:0/3990949973 >> 172.21.15.44:0/3990949973 conn(0x7f4a8c09aa20 msgr2=0x7f4a8c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.619+0000 7f4a91d00640 1 -- 172.21.15.44:0/3990949973 shutdown_connections 2024-04-02T13:30:18.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.619+0000 7f4a91d00640 1 -- 172.21.15.44:0/3990949973 wait complete. 2024-04-02T13:30:18.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.620+0000 7f4a91d00640 1 Processor -- start 2024-04-02T13:30:18.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.620+0000 7f4a91d00640 1 -- start start 2024-04-02T13:30:18.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a91d00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a8c0cfd60 0x7f4a8c0cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a91d00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a8c0d1ff0 0x7f4a8c0c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a91d00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a8c15c1a0 0x7f4a8c0c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a91d00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c06fff0 con 0x7f4a8c0cfd60 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a91d00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c06fd10 con 0x7f4a8c0d1ff0 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a91d00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4a8c0702d0 con 0x7f4a8c15c1a0 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a8b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a8c0cfd60 0x7f4a8c0cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a8b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a8c0cfd60 0x7f4a8c0cdee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46084/0 (socket says 172.21.15.44:46084) 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/4034787473 learned_addr learned my addr 172.21.15.44:0/4034787473 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a8affd640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a8c0d1ff0 0x7f4a8c0c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a8bfff640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a8c15c1a0 0x7f4a8c0c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a8c15c1a0 msgr2=0x7f4a8c0c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a8c15c1a0 0x7f4a8c0c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a8c0d1ff0 msgr2=0x7f4a8c0c59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a8c0d1ff0 0x7f4a8c0c59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/4034787473 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c15c8d0 con 0x7f4a8c0cfd60 2024-04-02T13:30:18.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.621+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a8c0cfd60 0x7f4a8c0cdee0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f4a8002b950 tx=0x7f4a800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.622+0000 7f4a73fff640 1 -- 172.21.15.44:0/4034787473 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80040070 con 0x7f4a8c0cfd60 2024-04-02T13:30:18.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.622+0000 7f4a73fff640 1 -- 172.21.15.44:0/4034787473 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a80030d00 con 0x7f4a8c0cfd60 2024-04-02T13:30:18.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.622+0000 7f4a73fff640 1 -- 172.21.15.44:0/4034787473 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80035040 con 0x7f4a8c0cfd60 2024-04-02T13:30:18.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.622+0000 7f4a91d00640 1 -- 172.21.15.44:0/4034787473 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8c15cbb0 con 0x7f4a8c0cfd60 2024-04-02T13:30:18.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.622+0000 7f4a91d00640 1 -- 172.21.15.44:0/4034787473 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a8c15d170 con 0x7f4a8c0cfd60 2024-04-02T13:30:18.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.623+0000 7f4a73fff640 1 -- 172.21.15.44:0/4034787473 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4a80038020 con 0x7f4a8c0cfd60 2024-04-02T13:30:18.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.624+0000 7f4a73fff640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4a7806c8f0 0x7f4a7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.624+0000 7f4a73fff640 1 -- 172.21.15.44:0/4034787473 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f4a800adba0 con 0x7f4a8c0cfd60 2024-04-02T13:30:18.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.624+0000 7f4a8affd640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4a7806c8f0 0x7f4a7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.624+0000 7f4a8affd640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4a7806c8f0 0x7f4a7806edb0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0ce720 tx=0x7f4a7c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.625+0000 7f4a949c4740 1 Processor -- start 2024-04-02T13:30:18.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.625+0000 7f4a949c4740 1 -- start start 2024-04-02T13:30:18.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a949c4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 0x555ebf3d46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a949c4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3a4330 0x555ebf3d1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a949c4740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3a7e90 0x555ebf3d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a949c4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555ebf30e770 con 0x555ebf3d4310 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a949c4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555ebf2b6960 con 0x555ebf3a4330 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a949c4740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x555ebf30a8c0 con 0x555ebf3a7e90 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a8b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 0x555ebf3d46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a8b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 0x555ebf3d46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46096/0 (socket says 172.21.15.44:46096) 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/1891033089 learned_addr learned my addr 172.21.15.44:0/1891033089 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a8bfff640 1 --2- 172.21.15.44:0/1891033089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3a7e90 0x555ebf3d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/1891033089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3a7e90 msgr2=0x555ebf3d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/1891033089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3a7e90 0x555ebf3d1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/1891033089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3a4330 msgr2=0x555ebf3d1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/1891033089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3a4330 0x555ebf3d1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/1891033089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ebf0ea7e0 con 0x555ebf3d4310 2024-04-02T13:30:18.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/1891033089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 0x555ebf3d46f0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f4a80031fd0 tx=0x7f4a80077a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376a3260dc7aef2e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a717fa640 1 -- 172.21.15.44:0/1891033089 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8007b690 con 0x555ebf3d4310 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a717fa640 1 -- 172.21.15.44:0/1891033089 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a8007fa40 con 0x555ebf3d4310 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.626+0000 7f4a717fa640 1 -- 172.21.15.44:0/1891033089 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80084270 con 0x555ebf3d4310 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.627+0000 7f4a949c4740 1 -- 172.21.15.44:0/1891033089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 msgr2=0x555ebf3d46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.627+0000 7f4a949c4740 1 --2- 172.21.15.44:0/1891033089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 0x555ebf3d46f0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f4a80031fd0 tx=0x7f4a80077a60 comp rx=0 tx=0).stop 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.627+0000 7f4a949c4740 1 -- 172.21.15.44:0/1891033089 shutdown_connections 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.627+0000 7f4a949c4740 1 --2- 172.21.15.44:0/1891033089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3a7e90 0x555ebf3d1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.627+0000 7f4a949c4740 1 --2- 172.21.15.44:0/1891033089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3a4330 0x555ebf3d1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.627+0000 7f4a949c4740 1 --2- 172.21.15.44:0/1891033089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 0x555ebf3d46f0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.627+0000 7f4a949c4740 1 -- 172.21.15.44:0/1891033089 >> 172.21.15.44:0/1891033089 conn(0x555ebf3a65a0 msgr2=0x555ebf3ef120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.627+0000 7f4a949c4740 1 -- 172.21.15.44:0/1891033089 shutdown_connections 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.627+0000 7f4a949c4740 1 -- 172.21.15.44:0/1891033089 wait complete. 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.628+0000 7f4a949c4740 1 -- start start 2024-04-02T13:30:18.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.628+0000 7f4a949c4740 1 Processor -- start 2024-04-02T13:30:18.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.628+0000 7f4a949c4740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3a7e90 0x555ebf3c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.628+0000 7f4a949c4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 0x555ebf3d66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.628+0000 7f4a949c4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d0820 0x555ebf3d8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.628+0000 7f4a949c4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555ebf2b6960 con 0x555ebf3d4310 2024-04-02T13:30:18.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.628+0000 7f4a949c4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555ebf2c9d20 con 0x555ebf3d0820 2024-04-02T13:30:18.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.628+0000 7f4a949c4740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x555ebf30e770 con 0x555ebf3a7e90 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.628+0000 7f4a949c4740 10 client.0 _ll_register_callbacks cb 0x555ebf296a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a8affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 0x555ebf3d66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a8affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 0x555ebf3d66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46104/0 (socket says 172.21.15.44:46104) 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a8affd640 1 -- 172.21.15.44:0/2326216611 learned_addr learned my addr 172.21.15.44:0/2326216611 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a8bfff640 1 --2- 172.21.15.44:0/2326216611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d0820 0x555ebf3d8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/2326216611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3a7e90 0x555ebf3c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a8affd640 1 -- 172.21.15.44:0/2326216611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3a7e90 msgr2=0x555ebf3c56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a8affd640 1 --2- 172.21.15.44:0/2326216611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3a7e90 0x555ebf3c56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a8affd640 1 -- 172.21.15.44:0/2326216611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d0820 msgr2=0x555ebf3d8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a8affd640 1 --2- 172.21.15.44:0/2326216611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d0820 0x555ebf3d8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a8affd640 1 -- 172.21.15.44:0/2326216611 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ebf14af70 con 0x555ebf3d4310 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a8affd640 1 --2- 172.21.15.44:0/2326216611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 0x555ebf3d66b0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00ffd0 tx=0x7f4a7c014830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=187adfb573fee5ba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a6a7fc640 1 -- 172.21.15.44:0/2326216611 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c025060 con 0x555ebf3d4310 2024-04-02T13:30:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a6a7fc640 1 -- 172.21.15.44:0/2326216611 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f4a7c015820 con 0x555ebf3d4310 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.629+0000 7f4a949c4740 1 -- 172.21.15.44:0/2326216611 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a8007fa40 con 0x555ebf3d4310 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a6a7fc640 1 -- 172.21.15.44:0/2326216611 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f4a7c02b050 con 0x555ebf3d4310 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a949c4740 10 client.9285 fetch_fsmap learned FSMap version 88 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a949c4740 1 -- 172.21.15.44:0/2326216611 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) -- 0x555ebefe2170 con 0x555ebf3d4310 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a6a7fc640 1 -- 172.21.15.44:0/2326216611 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a8007fa40 con 0x555ebf3d4310 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a6a7fc640 1 -- 172.21.15.44:0/2326216611 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 88) ==== 3494+0+0 (secure 0 0 0) 0x7f4a7c02e020 con 0x555ebf3d4310 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a949c4740 10 client.9285 fetch_fsmap finished waiting for FSMap version 88 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a949c4740 10 client.9285 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a949c4740 20 client.9285 populate_metadata read hostname 'smithi044' 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a949c4740 1 --2- 172.21.15.44:0/2326216611 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x555ebf402b30 0x555ebf402f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a949c4740 4 client.9285 mds_command: new command op to 9192 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a949c4740 1 -- 172.21.15.44:0/2326216611 --> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ebf31bcb0 con 0x555ebf402b30 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/2326216611 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x555ebf402b30 0x555ebf402f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a6a7fc640 10 client.9285 ms_handle_connect on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:18.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.630+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/2326216611 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x555ebf402b30 0x555ebf402f40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c4e6c39d00f62a4 server_cookie=be8473bf59874478 in_seq=0 out_seq=0 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a6a7fc640 1 -- 172.21.15.44:0/2326216611 <== mds.0 v2:172.21.15.44:6833/1745009605 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555ebf31bcb0 con 0x555ebf402b30 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a6a7fc640 10 client.9285 handle_command_reply: tid=0 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a949c4740 1 client.9285 shutdown 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a949c4740 1 -- 172.21.15.44:0/2326216611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 msgr2=0x555ebf3d66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a949c4740 1 --2- 172.21.15.44:0/2326216611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 0x555ebf3d66b0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00ffd0 tx=0x7f4a7c014830 comp rx=0 tx=0).stop 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a949c4740 1 -- 172.21.15.44:0/2326216611 shutdown_connections 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a949c4740 1 --2- 172.21.15.44:0/2326216611 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x555ebf402b30 0x555ebf402f40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a949c4740 1 --2- 172.21.15.44:0/2326216611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d0820 0x555ebf3d8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a949c4740 1 --2- 172.21.15.44:0/2326216611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3d4310 0x555ebf3d66b0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a6a7fc640 0 client.9285 ms_handle_reset on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a949c4740 1 --2- 172.21.15.44:0/2326216611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3a7e90 0x555ebf3c56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a949c4740 1 -- 172.21.15.44:0/2326216611 >> 172.21.15.44:0/2326216611 conn(0x555ebf3a4330 msgr2=0x555ebf3a8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a949c4740 1 -- 172.21.15.44:0/2326216611 shutdown_connections 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a949c4740 1 -- 172.21.15.44:0/2326216611 wait complete. 2024-04-02T13:30:18.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.631+0000 7f4a949c4740 20 client.9285 trim_cache size 0 max 16384 2024-04-02T13:30:18.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.637+0000 7f2a14fe2640 1 Processor -- start 2024-04-02T13:30:18.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.637+0000 7f2a14fe2640 1 -- start start 2024-04-02T13:30:18.640 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:30:18.637+0000 7f7d36f9b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:30:18.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a14fe2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a1015e4d0 0x7f2a100d2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a14fe2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2a100d1230 0x7f2a100b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a14fe2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a1015e8b0 con 0x7f2a100d2c70 2024-04-02T13:30:18.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a14fe2640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2a100d3180 con 0x7f2a100d1230 2024-04-02T13:30:18.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a14fe2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2a100b4560 con 0x7f2a1015e4d0 2024-04-02T13:30:18.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a0ed9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2a100d1230 0x7f2a100b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a0ed9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2a100d1230 0x7f2a100b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46112/0 (socket says 172.21.15.44:46112) 2024-04-02T13:30:18.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a07fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f2a100d2c70 0x7f2a100d0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:50416/0 (socket says 172.21.15.44:50416) 2024-04-02T13:30:18.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a0ed9d640 1 -- 172.21.15.44:0/547905881 learned_addr learned my addr 172.21.15.44:0/547905881 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a0e59c640 1 --2- 172.21.15.44:0/547905881 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a1015e4d0 0x7f2a100d2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a0ed9d640 1 -- 172.21.15.44:0/547905881 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a1015e4d0 msgr2=0x7f2a100d2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a0ed9d640 1 --2- 172.21.15.44:0/547905881 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a1015e4d0 0x7f2a100d2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a0ed9d640 1 -- 172.21.15.44:0/547905881 >> v1:172.21.15.44:6789/0 conn(0x7f2a100d2c70 legacy=0x7f2a100d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a0ed9d640 1 -- 172.21.15.44:0/547905881 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a100b3c30 con 0x7f2a100d1230 2024-04-02T13:30:18.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.638+0000 7f2a0e59c640 1 --2- 172.21.15.44:0/547905881 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a1015e4d0 0x7f2a100d2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:18.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.639+0000 7f2a0ed9d640 1 --2- 172.21.15.44:0/547905881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a100d1230 0x7f2a100b49f0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f2a08009e50 tx=0x7f2a0802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3abcbdeb907ba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.639+0000 7f2a0dd9b640 1 -- 172.21.15.44:0/547905881 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0802cb00 con 0x7f2a100d1230 2024-04-02T13:30:18.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.639+0000 7f2a0dd9b640 1 -- 172.21.15.44:0/547905881 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a0802cca0 con 0x7f2a100d1230 2024-04-02T13:30:18.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.639+0000 7f2a0dd9b640 1 -- 172.21.15.44:0/547905881 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a08034610 con 0x7f2a100d1230 2024-04-02T13:30:18.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.640+0000 7f2a14fe2640 1 -- 172.21.15.44:0/547905881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a100d1230 msgr2=0x7f2a100b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.640+0000 7f2a14fe2640 1 --2- 172.21.15.44:0/547905881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a100d1230 0x7f2a100b49f0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f2a08009e50 tx=0x7f2a0802bf60 comp rx=0 tx=0).stop 2024-04-02T13:30:18.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.640+0000 7f2a14fe2640 1 -- 172.21.15.44:0/547905881 shutdown_connections 2024-04-02T13:30:18.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.640+0000 7f2a14fe2640 1 --2- 172.21.15.44:0/547905881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a100d1230 0x7f2a100b49f0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.640+0000 7f2a14fe2640 1 --2- 172.21.15.44:0/547905881 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a1015e4d0 0x7f2a100d2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.640+0000 7f2a14fe2640 1 -- 172.21.15.44:0/547905881 >> 172.21.15.44:0/547905881 conn(0x7f2a1009d430 msgr2=0x7f2a100d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.640+0000 7f2a14fe2640 1 -- 172.21.15.44:0/547905881 shutdown_connections 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.640+0000 7f2a14fe2640 1 -- 172.21.15.44:0/547905881 wait complete. 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.641+0000 7f2a14fe2640 1 Processor -- start 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.641+0000 7f2a14fe2640 1 -- start start 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.642+0000 7f2a14fe2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a100d1230 0x7f2a100b6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.642+0000 7f2a14fe2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a100d2c70 0x7f2a100c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.642+0000 7f2a14fe2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a1015e4d0 0x7f2a100ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.642+0000 7f2a14fe2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2a100b41f0 con 0x7f2a1015e4d0 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.642+0000 7f2a14fe2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a100b3f10 con 0x7f2a100d1230 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.642+0000 7f2a14fe2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2a100b0d20 con 0x7f2a100d2c70 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.642+0000 7f2a0ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a1015e4d0 0x7f2a100ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.642+0000 7f2a0ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a1015e4d0 0x7f2a100ca010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46128/0 (socket says 172.21.15.44:46128) 2024-04-02T13:30:18.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.642+0000 7f2a0ed9d640 1 -- 172.21.15.44:0/503258263 learned_addr learned my addr 172.21.15.44:0/503258263 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.642+0000 7f2a07fff640 1 --2- 172.21.15.44:0/503258263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a100d2c70 0x7f2a100c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.643+0000 7f2a0e59c640 1 --2- 172.21.15.44:0/503258263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a100d1230 0x7f2a100b6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.643+0000 7f2a0ed9d640 1 -- 172.21.15.44:0/503258263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a100d2c70 msgr2=0x7f2a100c9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.643+0000 7f2a0ed9d640 1 --2- 172.21.15.44:0/503258263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a100d2c70 0x7f2a100c9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.643+0000 7f2a0ed9d640 1 -- 172.21.15.44:0/503258263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a100d1230 msgr2=0x7f2a100b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.643+0000 7f2a0ed9d640 1 --2- 172.21.15.44:0/503258263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a100d1230 0x7f2a100b6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.643+0000 7f2a0ed9d640 1 -- 172.21.15.44:0/503258263 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a100c96b0 con 0x7f2a1015e4d0 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.643+0000 7f2a0e59c640 1 --2- 172.21.15.44:0/503258263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a100d1230 0x7f2a100b6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.643+0000 7f2a0ed9d640 1 --2- 172.21.15.44:0/503258263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a1015e4d0 0x7f2a100ca010 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f2a08036040 tx=0x7f2a08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.643+0000 7f2a06ffd640 1 -- 172.21.15.44:0/503258263 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a08043020 con 0x7f2a1015e4d0 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.643+0000 7f2a06ffd640 1 -- 172.21.15.44:0/503258263 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a0802ce50 con 0x7f2a1015e4d0 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.643+0000 7f2a06ffd640 1 -- 172.21.15.44:0/503258263 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a08009510 con 0x7f2a1015e4d0 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.643+0000 7f2a14fe2640 1 -- 172.21.15.44:0/503258263 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a100cab80 con 0x7f2a1015e4d0 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.643+0000 7f2a14fe2640 1 -- 172.21.15.44:0/503258263 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a100ca8a0 con 0x7f2a1015e4d0 2024-04-02T13:30:18.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.644+0000 7f2a04ff9640 1 -- 172.21.15.44:0/503258263 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f29e8005270 con 0x7f2a1015e4d0 2024-04-02T13:30:18.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.645+0000 7f2a06ffd640 1 -- 172.21.15.44:0/503258263 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a08046020 con 0x7f2a1015e4d0 2024-04-02T13:30:18.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.645+0000 7f2a06ffd640 1 --2- 172.21.15.44:0/503258263 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29f006c8f0 0x7f29f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.645+0000 7f2a06ffd640 1 -- 172.21.15.44:0/503258263 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f2a080ae840 con 0x7f2a1015e4d0 2024-04-02T13:30:18.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.646+0000 7f2a06ffd640 1 -- 172.21.15.44:0/503258263 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+516 (secure 0 0 0) 0x7f2a0804d030 con 0x7f2a1015e4d0 2024-04-02T13:30:18.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.646+0000 7f2a0e59c640 1 --2- 172.21.15.44:0/503258263 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29f006c8f0 0x7f29f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.646+0000 7f2a14fe2640 1 -- 172.21.15.44:0/503258263 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7f2a10068c00 con 0x7f2a1015e4d0 2024-04-02T13:30:18.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.646+0000 7f2a06ffd640 1 -- 172.21.15.44:0/503258263 <== mon.0 v2:172.21.15.44:3300/0 7 ==== log(2 entries from seq 1166 at 2024-04-02T13:30:16.638088+0000) ==== 389+0+0 (secure 0 0 0) 0x7f2a080acc40 con 0x7f2a1015e4d0 2024-04-02T13:30:18.650 INFO:teuthology.orchestra.run.smithi044.stdout: cluster: 2024-04-02T13:30:18.650 INFO:teuthology.orchestra.run.smithi044.stdout: id: 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:30:18.650 INFO:teuthology.orchestra.run.smithi044.stdout: health: HEALTH_OK 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: services: 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: mon: 3 daemons, quorum a,b,c (age 10m) 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: mgr: x(active, since 10m), standbys: y 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: mds: 1/1 daemons up, 3 standby 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: osd: 8 osds: 8 up (since 10m), 8 in (since 10m) 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: data: 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: volumes: 1/1 healthy 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: pools: 3 pools, 97 pgs 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: objects: 30 objects, 19 MiB 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: usage: 291 MiB used, 720 GiB / 720 GiB avail 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: pgs: 97 active+clean 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:18.651 INFO:teuthology.orchestra.run.smithi044.stdout: io: 2024-04-02T13:30:18.652 INFO:teuthology.orchestra.run.smithi044.stdout: client: 2.5 KiB/s rd, 139 B/s wr, 2 op/s rd, 0 op/s wr 2024-04-02T13:30:18.652 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:18.652 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:18.652 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:30:16.638088+0000 mon.a [INF] Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-02T13:30:18.652 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:30:16.638110+0000 mon.a [INF] Cluster is now healthy 2024-04-02T13:30:18.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.646+0000 7f2a0e59c640 1 --2- 172.21.15.44:0/503258263 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29f006c8f0 0x7f29f006edb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2a10160b70 tx=0x7f2a00008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.652+0000 7f4a949c4740 1 Processor -- start 2024-04-02T13:30:18.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.652+0000 7f4a949c4740 1 -- start start 2024-04-02T13:30:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a949c4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 0x555ebf3d6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a949c4740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3d0820 0x555ebf3a8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a949c4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d4310 0x555ebf3a87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a949c4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c1fee90 con 0x555ebf3a7e90 2024-04-02T13:30:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a949c4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c1fdea0 con 0x555ebf3d4310 2024-04-02T13:30:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a949c4740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4a8c15e700 con 0x555ebf3d0820 2024-04-02T13:30:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a8b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 0x555ebf3d6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a8b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 0x555ebf3d6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46134/0 (socket says 172.21.15.44:46134) 2024-04-02T13:30:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/3255614077 learned_addr learned my addr 172.21.15.44:0/3255614077 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a8affd640 1 --2- 172.21.15.44:0/3255614077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3d0820 0x555ebf3a8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a8bfff640 1 --2- 172.21.15.44:0/3255614077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d4310 0x555ebf3a87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/3255614077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3d0820 msgr2=0x555ebf3a8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/3255614077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3d0820 0x555ebf3a8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/3255614077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d4310 msgr2=0x555ebf3a87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.653+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/3255614077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d4310 0x555ebf3a87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.654+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/3255614077 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ebf3d8dc0 con 0x555ebf3a7e90 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.654+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/3255614077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 0x555ebf3d6660 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f4a8007b390 tx=0x7f4a8007d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1b758e4318f8af server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.654+0000 7f4a70ff9640 1 -- 172.21.15.44:0/3255614077 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8007fa40 con 0x555ebf3a7e90 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.654+0000 7f4a70ff9640 1 -- 172.21.15.44:0/3255614077 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a8007b690 con 0x555ebf3a7e90 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.654+0000 7f4a70ff9640 1 -- 172.21.15.44:0/3255614077 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80084270 con 0x555ebf3a7e90 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.654+0000 7f4a949c4740 1 -- 172.21.15.44:0/3255614077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 msgr2=0x555ebf3d6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.654+0000 7f4a949c4740 1 --2- 172.21.15.44:0/3255614077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 0x555ebf3d6660 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f4a8007b390 tx=0x7f4a8007d000 comp rx=0 tx=0).stop 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.655+0000 7f4a949c4740 1 -- 172.21.15.44:0/3255614077 shutdown_connections 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.655+0000 7f4a949c4740 1 --2- 172.21.15.44:0/3255614077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d4310 0x555ebf3a87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.655+0000 7f4a949c4740 1 --2- 172.21.15.44:0/3255614077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3d0820 0x555ebf3a8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.655+0000 7f4a949c4740 1 --2- 172.21.15.44:0/3255614077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 0x555ebf3d6660 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.655+0000 7f4a949c4740 1 -- 172.21.15.44:0/3255614077 >> 172.21.15.44:0/3255614077 conn(0x555ebf3a4330 msgr2=0x555ebf3f2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.655+0000 7f4a949c4740 1 -- 172.21.15.44:0/3255614077 shutdown_connections 2024-04-02T13:30:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.655+0000 7f4a949c4740 1 -- 172.21.15.44:0/3255614077 wait complete. 2024-04-02T13:30:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.655+0000 7f4a949c4740 1 -- start start 2024-04-02T13:30:18.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.655+0000 7f4a949c4740 1 Processor -- start 2024-04-02T13:30:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a949c4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 0x555ebf3d6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a949c4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d0820 0x555ebf401860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a949c4740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3d4310 0x555ebf401da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a949c4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c1fd960 con 0x555ebf3a7e90 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a949c4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555ebf46ab40 con 0x555ebf3d0820 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a949c4740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x555ebf3f2310 con 0x555ebf3d4310 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a949c4740 10 client.0 _ll_register_callbacks cb 0x7f4a7c00b660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a8b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 0x555ebf3d6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a8b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 0x555ebf3d6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46140/0 (socket says 172.21.15.44:46140) 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/2325853791 learned_addr learned my addr 172.21.15.44:0/2325853791 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a8affd640 1 --2- 172.21.15.44:0/2325853791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d0820 0x555ebf401860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a8bfff640 1 --2- 172.21.15.44:0/2325853791 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3d4310 0x555ebf401da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/2325853791 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3d4310 msgr2=0x555ebf401da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/2325853791 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3d4310 0x555ebf401da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/2325853791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d0820 msgr2=0x555ebf401860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/2325853791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d0820 0x555ebf401860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.656+0000 7f4a8b7fe640 1 -- 172.21.15.44:0/2325853791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ebf3d4a60 con 0x555ebf3a7e90 2024-04-02T13:30:18.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a8b7fe640 1 --2- 172.21.15.44:0/2325853791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 0x555ebf3d6660 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f4a800abee0 tx=0x7f4a8003c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8c614cac60a04c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a6a7fc640 1 -- 172.21.15.44:0/2325853791 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80084270 con 0x555ebf3a7e90 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a6a7fc640 1 -- 172.21.15.44:0/2325853791 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f4a80079990 con 0x555ebf3a7e90 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a949c4740 1 -- 172.21.15.44:0/2325853791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a8007b690 con 0x555ebf3a7e90 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a6a7fc640 1 -- 172.21.15.44:0/2325853791 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f4a800a1050 con 0x555ebf3a7e90 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a949c4740 10 client.9300 fetch_fsmap learned FSMap version 88 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a949c4740 1 -- 172.21.15.44:0/2325853791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) -- 0x555ebf412430 con 0x555ebf3a7e90 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a6a7fc640 1 -- 172.21.15.44:0/2325853791 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a8007b690 con 0x555ebf3a7e90 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a6a7fc640 1 -- 172.21.15.44:0/2325853791 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 88) ==== 3494+0+0 (secure 0 0 0) 0x7f4a800a4020 con 0x555ebf3a7e90 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a949c4740 10 client.9300 fetch_fsmap finished waiting for FSMap version 88 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a949c4740 10 client.9300 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a949c4740 20 client.9300 populate_metadata read hostname 'smithi044' 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a949c4740 1 --2- 172.21.15.44:0/2325853791 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x555ebf469220 0x555ebf469800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a949c4740 4 client.9300 mds_command: new command op to 9192 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-04-02T13:30:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a949c4740 1 -- 172.21.15.44:0/2325853791 --> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) -- 0x555ebf316000 con 0x555ebf469220 2024-04-02T13:30:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.657+0000 7f4a8affd640 1 --2- 172.21.15.44:0/2325853791 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x555ebf469220 0x555ebf469800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.658+0000 7f4a6a7fc640 10 client.9300 ms_handle_connect on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.658+0000 7f4a8affd640 1 --2- 172.21.15.44:0/2325853791 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x555ebf469220 0x555ebf469800 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c62b19bf56aec55 server_cookie=e9ef6717a0f89563 in_seq=0 out_seq=0 2024-04-02T13:30:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.658+0000 7f4a6a7fc640 1 -- 172.21.15.44:0/2325853791 <== mds.0 v2:172.21.15.44:6833/1745009605 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555ebf316000 con 0x555ebf469220 2024-04-02T13:30:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.658+0000 7f4a6a7fc640 10 client.9300 handle_command_reply: tid=0 2024-04-02T13:30:18.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.658+0000 7f4a949c4740 1 client.9300 shutdown 2024-04-02T13:30:18.666 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:30:18.658+0000 7f7d32f93640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000001 2024-04-02T13:30:18.666 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:30:18.658+0000 7f7d32f93640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000001 2024-04-02T13:30:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f4a949c4740 1 -- 172.21.15.44:0/2325853791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 msgr2=0x555ebf3d6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f4a949c4740 1 --2- 172.21.15.44:0/2325853791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 0x555ebf3d6660 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f4a800abee0 tx=0x7f4a8003c040 comp rx=0 tx=0).stop 2024-04-02T13:30:18.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f4a949c4740 1 -- 172.21.15.44:0/2325853791 shutdown_connections 2024-04-02T13:30:18.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f4a949c4740 1 --2- 172.21.15.44:0/2325853791 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x555ebf469220 0x555ebf469800 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f4a949c4740 1 --2- 172.21.15.44:0/2325853791 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x555ebf3d4310 0x555ebf401da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f4a949c4740 1 --2- 172.21.15.44:0/2325853791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555ebf3d0820 0x555ebf401860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f4a949c4740 1 --2- 172.21.15.44:0/2325853791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555ebf3a7e90 0x555ebf3d6660 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f4a949c4740 1 -- 172.21.15.44:0/2325853791 >> 172.21.15.44:0/2325853791 conn(0x555ebf3a4330 msgr2=0x555ebf3d7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f4a6a7fc640 0 client.9300 ms_handle_reset on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:18.668 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f7d32f93640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000002 2024-04-02T13:30:18.668 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f7d32f93640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000002 2024-04-02T13:30:18.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f4a949c4740 1 -- 172.21.15.44:0/2325853791 shutdown_connections 2024-04-02T13:30:18.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f4a949c4740 1 -- 172.21.15.44:0/2325853791 wait complete. 2024-04-02T13:30:18.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.659+0000 7f4a949c4740 20 client.9300 trim_cache size 0 max 16384 2024-04-02T13:30:18.669 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:30:18.669 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-02T13:30:18.669 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "b70ed285-5c2e-47b7-b888-d823c1e21df1", 2024-04-02T13:30:18.669 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-02T13:30:18.669 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:30:18.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.660+0000 7f4a91d00640 1 -- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4a7806c8f0 msgr2=0x7f4a7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.660+0000 7f4a91d00640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4a7806c8f0 0x7f4a7806edb0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0ce720 tx=0x7f4a7c008040 comp rx=0 tx=0).stop 2024-04-02T13:30:18.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.660+0000 7f4a91d00640 1 -- 172.21.15.44:0/4034787473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a8c0cfd60 msgr2=0x7f4a8c0cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.660+0000 7f4a91d00640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a8c0cfd60 0x7f4a8c0cdee0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f4a8002b950 tx=0x7f4a800046c0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.660+0000 7f4a91d00640 1 -- 172.21.15.44:0/4034787473 shutdown_connections 2024-04-02T13:30:18.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.660+0000 7f4a91d00640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4a7806c8f0 0x7f4a7806edb0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.660+0000 7f4a91d00640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a8c15c1a0 0x7f4a8c0c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.660+0000 7f4a91d00640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a8c0d1ff0 0x7f4a8c0c59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.660+0000 7f4a91d00640 1 --2- 172.21.15.44:0/4034787473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a8c0cfd60 0x7f4a8c0cdee0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.660+0000 7f4a91d00640 1 -- 172.21.15.44:0/4034787473 >> 172.21.15.44:0/4034787473 conn(0x7f4a8c09aa20 msgr2=0x7f4a8c0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.660+0000 7f4a91d00640 1 -- 172.21.15.44:0/4034787473 shutdown_connections 2024-04-02T13:30:18.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.660+0000 7f4a91d00640 1 -- 172.21.15.44:0/4034787473 wait complete. 2024-04-02T13:30:18.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:30:18.674 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-02T13:30:18.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.775+0000 7f6a76493640 1 Processor -- start 2024-04-02T13:30:18.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.775+0000 7f6a76493640 1 -- start start 2024-04-02T13:30:18.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.775+0000 7f6a76493640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7015a940 0x7f6a700ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.775+0000 7f6a76493640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6a70071200 0x7f6a7006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.775+0000 7f6a76493640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a700ce9f0 con 0x7f6a700cc640 2024-04-02T13:30:18.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.775+0000 7f6a76493640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6a7006ef80 con 0x7f6a70071200 2024-04-02T13:30:18.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.775+0000 7f6a76493640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6a700ce020 con 0x7f6a7015a940 2024-04-02T13:30:18.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a74a30640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6a700cc640 0x7f6a700acd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:50418/0 (socket says 172.21.15.44:50418) 2024-04-02T13:30:18.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a74a30640 1 -- 172.21.15.44:0/1991105369 learned_addr learned my addr 172.21.15.44:0/1991105369 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:18.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2039853685 0 0) 0x7f6a700ce9f0 con 0x7f6a700cc640 2024-04-02T13:30:18.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a58003520 con 0x7f6a700cc640 2024-04-02T13:30:18.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/1991105369 >> v2:172.21.15.69:3300/0 conn(0x7f6a70071200 0x7f6a7006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/1991105369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7015a940 0x7f6a700ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1711495389 0 0) 0x7f6a58003520 con 0x7f6a700cc640 2024-04-02T13:30:18.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7015a940 msgr2=0x7f6a700ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 --2- 172.21.15.44:0/1991105369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7015a940 0x7f6a700ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 >> v2:172.21.15.69:3300/0 conn(0x7f6a70071200 msgr2=0x7f6a7006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 --2- 172.21.15.44:0/1991105369 >> v2:172.21.15.69:3300/0 conn(0x7f6a70071200 0x7f6a7006e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a7006e9c0 con 0x7f6a700cc640 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f6a680019f0 con 0x7f6a700cc640 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 >> v1:172.21.15.44:6789/0 conn(0x7f6a700cc640 legacy=0x7f6a700acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 --2- 172.21.15.44:0/1991105369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a580039b0 0x7f6a58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 --2- 172.21.15.44:0/1991105369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a58004800 0x7f6a58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 --2- 172.21.15.44:0/1991105369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7015a940 0x7f6a580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7006ef80 con 0x7f6a580039b0 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6a700ce020 con 0x7f6a58004800 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6a5800b9c0 con 0x7f6a7015a940 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/1991105369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a580039b0 0x7f6a58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/1991105369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a58004800 0x7f6a58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.776+0000 7f6a74a30640 1 --2- 172.21.15.44:0/1991105369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7015a940 0x7f6a580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.777+0000 7f6a6ffff640 1 -- 172.21.15.44:0/1991105369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7015a940 msgr2=0x7f6a580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.777+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/1991105369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7015a940 0x7f6a580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.777+0000 7f6a6ffff640 1 -- 172.21.15.44:0/1991105369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a58004800 msgr2=0x7f6a58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.777+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/1991105369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a58004800 0x7f6a58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.777+0000 7f6a6ffff640 1 -- 172.21.15.44:0/1991105369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a58003520 con 0x7f6a580039b0 2024-04-02T13:30:18.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.777+0000 7f6a74a30640 1 --2- 172.21.15.44:0/1991105369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7015a940 0x7f6a580051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:18.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.777+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/1991105369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a580039b0 0x7f6a58003dd0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009af0 tx=0x7f6a58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9747e4e18522e26 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.777+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60039070 con 0x7f6a580039b0 2024-04-02T13:30:18.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.777+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a6002c6b0 con 0x7f6a580039b0 2024-04-02T13:30:18.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.777+0000 7f6a6effd640 1 -- 172.21.15.44:0/1991105369 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60009690 con 0x7f6a580039b0 2024-04-02T13:30:18.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.778+0000 7f6a76493640 1 -- 172.21.15.44:0/1991105369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a580039b0 msgr2=0x7f6a58003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.778+0000 7f6a76493640 1 --2- 172.21.15.44:0/1991105369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a580039b0 0x7f6a58003dd0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009af0 tx=0x7f6a58005830 comp rx=0 tx=0).stop 2024-04-02T13:30:18.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.778+0000 7f6a74a30640 1 -- 172.21.15.44:0/1991105369 reap_dead start 2024-04-02T13:30:18.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.778+0000 7f6a76493640 1 -- 172.21.15.44:0/1991105369 shutdown_connections 2024-04-02T13:30:18.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.778+0000 7f6a76493640 1 -- 172.21.15.44:0/1991105369 >> 172.21.15.44:0/1991105369 conn(0x7f6a7006b920 msgr2=0x7f6a7006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.778+0000 7f6a76493640 1 -- 172.21.15.44:0/1991105369 shutdown_connections 2024-04-02T13:30:18.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.778+0000 7f6a76493640 1 -- 172.21.15.44:0/1991105369 wait complete. 2024-04-02T13:30:18.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.779+0000 7f6a76493640 1 Processor -- start 2024-04-02T13:30:18.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.779+0000 7f6a76493640 1 -- start start 2024-04-02T13:30:18.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.780+0000 7f6a76493640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70151ac0 0x7f6a701f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.780+0000 7f6a76493640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70153a40 0x7f6a700ad760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.780+0000 7f6a76493640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a7015ad20 0x7f6a7015b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.780+0000 7f6a76493640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7006e600 con 0x7f6a7015ad20 2024-04-02T13:30:18.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.780+0000 7f6a76493640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6a700cdcb0 con 0x7f6a70153a40 2024-04-02T13:30:18.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.780+0000 7f6a76493640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6a7006e780 con 0x7f6a70151ac0 2024-04-02T13:30:18.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.780+0000 7f6a74a30640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a7015ad20 0x7f6a7015b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.780+0000 7f6a74a30640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a7015ad20 0x7f6a7015b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46148/0 (socket says 172.21.15.44:46148) 2024-04-02T13:30:18.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.780+0000 7f6a74a30640 1 -- 172.21.15.44:0/3341155960 learned_addr learned my addr 172.21.15.44:0/3341155960 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.780+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70153a40 0x7f6a700ad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.780+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70151ac0 0x7f6a701f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.780+0000 7f6a74a30640 1 -- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70151ac0 msgr2=0x7f6a701f44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.781+0000 7f6a74a30640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70151ac0 0x7f6a701f44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.781+0000 7f6a74a30640 1 -- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70153a40 msgr2=0x7f6a700ad760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.781+0000 7f6a74a30640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70153a40 0x7f6a700ad760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.781+0000 7f6a74a30640 1 -- 172.21.15.44:0/3341155960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a70151ea0 con 0x7f6a7015ad20 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.781+0000 7f6a74a30640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a7015ad20 0x7f6a7015b1b0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f6a700d0d80 tx=0x7f6a6800e390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.781+0000 7f6a6cff9640 1 -- 172.21.15.44:0/3341155960 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6800eb40 con 0x7f6a7015ad20 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.781+0000 7f6a6cff9640 1 -- 172.21.15.44:0/3341155960 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a6800ece0 con 0x7f6a7015ad20 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.781+0000 7f6a6cff9640 1 -- 172.21.15.44:0/3341155960 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68013420 con 0x7f6a7015ad20 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.781+0000 7f6a76493640 1 -- 172.21.15.44:0/3341155960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a70156520 con 0x7f6a7015ad20 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.781+0000 7f6a76493640 1 -- 172.21.15.44:0/3341155960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a70156ae0 con 0x7f6a7015ad20 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.782+0000 7f6a6cff9640 1 -- 172.21.15.44:0/3341155960 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6a6801a020 con 0x7f6a7015ad20 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.783+0000 7f6a6cff9640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6a5c06c920 0x7f6a5c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.783+0000 7f6a6cff9640 1 -- 172.21.15.44:0/3341155960 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f6a6808b9c0 con 0x7f6a7015ad20 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.783+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6a5c06c920 0x7f6a5c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.783+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6a5c06c920 0x7f6a5c06ede0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6a60036000 tx=0x7f6a60030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.784+0000 7f6a79084740 1 Processor -- start 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.784+0000 7f6a79084740 1 -- start start 2024-04-02T13:30:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.784+0000 7f6a79084740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 0x558eef75d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.784+0000 7f6a79084740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef78d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.784+0000 7f6a79084740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef75de80 0x558eef78dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.784+0000 7f6a79084740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558eef6c7830 con 0x558eef78ad80 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.784+0000 7f6a79084740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558eef66f820 con 0x558eef78a9a0 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.784+0000 7f6a79084740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x558eef6c37e0 con 0x558eef75de80 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a6f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef78d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a6f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef78d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46162/0 (socket says 172.21.15.44:46162) 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/1573587895 learned_addr learned my addr 172.21.15.44:0/1573587895 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/1573587895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef75de80 msgr2=0x558eef78dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a74a30640 1 --2- 172.21.15.44:0/1573587895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef75de80 0x558eef78dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/1573587895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 0x558eef75d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/1573587895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef75de80 0x558eef78dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/1573587895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 msgr2=0x558eef75d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/1573587895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 0x558eef75d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/1573587895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eef4a57d0 con 0x558eef78ad80 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/1573587895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef78d730 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009b70 tx=0x7f6a64014780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=515b3641b1228d7b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a55ffb640 1 -- 172.21.15.44:0/1573587895 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64022070 con 0x558eef78ad80 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a55ffb640 1 -- 172.21.15.44:0/1573587895 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a6401d440 con 0x558eef78ad80 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.785+0000 7f6a55ffb640 1 -- 172.21.15.44:0/1573587895 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6401d810 con 0x558eef78ad80 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.786+0000 7f6a79084740 1 -- 172.21.15.44:0/1573587895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 msgr2=0x558eef78d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.786+0000 7f6a79084740 1 --2- 172.21.15.44:0/1573587895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef78d730 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009b70 tx=0x7f6a64014780 comp rx=0 tx=0).stop 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.786+0000 7f6a79084740 1 -- 172.21.15.44:0/1573587895 shutdown_connections 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.786+0000 7f6a79084740 1 --2- 172.21.15.44:0/1573587895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef75de80 0x558eef78dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.786+0000 7f6a79084740 1 --2- 172.21.15.44:0/1573587895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef78d730 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.786+0000 7f6a79084740 1 --2- 172.21.15.44:0/1573587895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 0x558eef75d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.786+0000 7f6a79084740 1 -- 172.21.15.44:0/1573587895 >> 172.21.15.44:0/1573587895 conn(0x558eef75fbb0 msgr2=0x558eef7a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.786+0000 7f6a79084740 1 -- 172.21.15.44:0/1573587895 shutdown_connections 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.786+0000 7f6a79084740 1 -- 172.21.15.44:0/1573587895 wait complete. 2024-04-02T13:30:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.786+0000 7f6a79084740 1 -- start start 2024-04-02T13:30:18.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.787+0000 7f6a79084740 1 Processor -- start 2024-04-02T13:30:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.787+0000 7f6a79084740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef78a9a0 0x558eef790f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.787+0000 7f6a79084740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef7a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.787+0000 7f6a79084740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef7a2180 0x558eef78f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.787+0000 7f6a79084740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558eef762150 con 0x558eef78ad80 2024-04-02T13:30:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.787+0000 7f6a79084740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558eef682d30 con 0x558eef7a2180 2024-04-02T13:30:18.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.787+0000 7f6a79084740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x558eef761540 con 0x558eef78a9a0 2024-04-02T13:30:18.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.787+0000 7f6a79084740 10 client.0 _ll_register_callbacks cb 0x558eef682720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:30:18.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.787+0000 7f6a6f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef7a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.787+0000 7f6a6f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef7a2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46168/0 (socket says 172.21.15.44:46168) 2024-04-02T13:30:18.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/1824234700 learned_addr learned my addr 172.21.15.44:0/1824234700 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/1824234700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef78a9a0 0x558eef790f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a74a30640 1 --2- 172.21.15.44:0/1824234700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef7a2180 0x558eef78f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/1824234700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef78a9a0 msgr2=0x558eef790f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/1824234700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef78a9a0 0x558eef790f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/1824234700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef7a2180 msgr2=0x558eef78f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/1824234700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef7a2180 0x558eef78f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/1824234700 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eef3db3f0 con 0x558eef78ad80 2024-04-02T13:30:18.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/1824234700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef7a2d60 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f6a64020040 tx=0x7f6a640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34dd2eee76f455f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a4effd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a4effd640 1 -- 172.21.15.44:0/1824234700 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64022040 con 0x558eef78ad80 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a4effd640 1 -- 172.21.15.44:0/1824234700 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f6a64028450 con 0x558eef78ad80 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a79084740 1 -- 172.21.15.44:0/1824234700 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a6401d440 con 0x558eef78ad80 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a4effd640 1 -- 172.21.15.44:0/1824234700 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f6a64035050 con 0x558eef78ad80 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a79084740 10 client.9312 fetch_fsmap learned FSMap version 88 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a79084740 1 -- 172.21.15.44:0/1824234700 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) -- 0x558eef39b0f0 con 0x558eef78ad80 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a4effd640 1 -- 172.21.15.44:0/1824234700 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a6401d440 con 0x558eef78ad80 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.788+0000 7f6a4effd640 1 -- 172.21.15.44:0/1824234700 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 88) ==== 3494+0+0 (secure 0 0 0) 0x7f6a64038020 con 0x558eef78ad80 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.789+0000 7f6a79084740 10 client.9312 fetch_fsmap finished waiting for FSMap version 88 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.789+0000 7f6a79084740 10 client.9312 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.789+0000 7f6a79084740 20 client.9312 populate_metadata read hostname 'smithi044' 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.789+0000 7f6a79084740 1 --2- 172.21.15.44:0/1824234700 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x558eef7a0bc0 0x558eef7a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.789+0000 7f6a79084740 4 client.9312 mds_command: new command op to 9192 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:30:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.789+0000 7f6a79084740 1 -- 172.21.15.44:0/1824234700 --> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558eef6d4d70 con 0x558eef7a0bc0 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.789+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/1824234700 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x558eef7a0bc0 0x558eef7a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.789+0000 7f6a4effd640 10 client.9312 ms_handle_connect on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.789+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/1824234700 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x558eef7a0bc0 0x558eef7a0ff0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1a8c8549950df88 server_cookie=6bb42114cf00c425 in_seq=0 out_seq=0 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a4effd640 1 -- 172.21.15.44:0/1824234700 <== mds.0 v2:172.21.15.44:6833/1745009605 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558eef6d4d70 con 0x558eef7a0bc0 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a4effd640 10 client.9312 handle_command_reply: tid=0 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a79084740 1 client.9312 shutdown 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a79084740 1 -- 172.21.15.44:0/1824234700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 msgr2=0x558eef7a2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a79084740 1 --2- 172.21.15.44:0/1824234700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef7a2d60 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f6a64020040 tx=0x7f6a640046c0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a79084740 1 -- 172.21.15.44:0/1824234700 shutdown_connections 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a79084740 1 --2- 172.21.15.44:0/1824234700 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x558eef7a0bc0 0x558eef7a0ff0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a79084740 1 --2- 172.21.15.44:0/1824234700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef7a2180 0x558eef78f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a79084740 1 --2- 172.21.15.44:0/1824234700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef7a2d60 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a4effd640 0 client.9312 ms_handle_reset on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a79084740 1 --2- 172.21.15.44:0/1824234700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef78a9a0 0x558eef790f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a79084740 1 -- 172.21.15.44:0/1824234700 >> 172.21.15.44:0/1824234700 conn(0x558eef75de80 msgr2=0x558eef78d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a79084740 1 -- 172.21.15.44:0/1824234700 shutdown_connections 2024-04-02T13:30:18.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a79084740 1 -- 172.21.15.44:0/1824234700 wait complete. 2024-04-02T13:30:18.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.790+0000 7f6a79084740 20 client.9312 trim_cache size 0 max 16384 2024-04-02T13:30:18.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.808+0000 7f6a79084740 1 Processor -- start 2024-04-02T13:30:18.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.808+0000 7f6a79084740 1 -- start start 2024-04-02T13:30:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.808+0000 7f6a79084740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 0x558eef7abc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.808+0000 7f6a79084740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef7ac180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.808+0000 7f6a79084740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef7a2180 0x558eef77e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.808+0000 7f6a79084740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6a70201c40 con 0x558eef78ad80 2024-04-02T13:30:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.808+0000 7f6a79084740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6a70200d00 con 0x558eef78a9a0 2024-04-02T13:30:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.808+0000 7f6a79084740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6a70200060 con 0x558eef7a2180 2024-04-02T13:30:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.808+0000 7f6a6f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef7ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.809+0000 7f6a6f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef7ac180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46180/0 (socket says 172.21.15.44:46180) 2024-04-02T13:30:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.809+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/3643864352 learned_addr learned my addr 172.21.15.44:0/3643864352 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.809+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/3643864352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 0x558eef7abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.809+0000 7f6a74a30640 1 --2- 172.21.15.44:0/3643864352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef7a2180 0x558eef77e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.809+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/3643864352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef7a2180 msgr2=0x558eef77e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.809+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/3643864352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef7a2180 0x558eef77e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.809+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/3643864352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 msgr2=0x558eef7abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.809+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/3643864352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 0x558eef7abc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.809+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/3643864352 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eef7a30d0 con 0x558eef78ad80 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.809+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/3643864352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef7ac180 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6a640147c0 tx=0x7f6a64023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf4d51231d1e57b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.809+0000 7f6a557fa640 1 -- 172.21.15.44:0/3643864352 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64022040 con 0x558eef78ad80 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.809+0000 7f6a557fa640 1 -- 172.21.15.44:0/3643864352 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a6402d070 con 0x558eef78ad80 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.809+0000 7f6a557fa640 1 -- 172.21.15.44:0/3643864352 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64030a50 con 0x558eef78ad80 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.810+0000 7f6a79084740 1 -- 172.21.15.44:0/3643864352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 msgr2=0x558eef7ac180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.810+0000 7f6a79084740 1 --2- 172.21.15.44:0/3643864352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef7ac180 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6a640147c0 tx=0x7f6a64023000 comp rx=0 tx=0).stop 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.810+0000 7f6a79084740 1 -- 172.21.15.44:0/3643864352 shutdown_connections 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.810+0000 7f6a79084740 1 --2- 172.21.15.44:0/3643864352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef7a2180 0x558eef77e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.810+0000 7f6a79084740 1 --2- 172.21.15.44:0/3643864352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef7ac180 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.810+0000 7f6a79084740 1 --2- 172.21.15.44:0/3643864352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 0x558eef7abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.810+0000 7f6a79084740 1 -- 172.21.15.44:0/3643864352 >> 172.21.15.44:0/3643864352 conn(0x558eef75de80 msgr2=0x558eef790cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.810+0000 7f6a79084740 1 -- 172.21.15.44:0/3643864352 shutdown_connections 2024-04-02T13:30:18.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.810+0000 7f6a79084740 1 -- 172.21.15.44:0/3643864352 wait complete. 2024-04-02T13:30:18.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.810+0000 7f6a79084740 1 -- start start 2024-04-02T13:30:18.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.810+0000 7f6a79084740 1 Processor -- start 2024-04-02T13:30:18.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.811+0000 7f6a79084740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 0x558eef7b6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.811+0000 7f6a79084740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef77dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.811+0000 7f6a79084740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef7a2180 0x558eef7ba830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.811+0000 7f6a79084740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6a702006b0 con 0x558eef78ad80 2024-04-02T13:30:18.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.811+0000 7f6a79084740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558eef7b7570 con 0x558eef78a9a0 2024-04-02T13:30:18.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.811+0000 7f6a79084740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x558eef78e450 con 0x558eef7a2180 2024-04-02T13:30:18.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.811+0000 7f6a79084740 10 client.0 _ll_register_callbacks cb 0x7f6a6401d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:30:18.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.811+0000 7f6a6f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef77dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.811+0000 7f6a6f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef77dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46194/0 (socket says 172.21.15.44:46194) 2024-04-02T13:30:18.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.811+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/2993326208 learned_addr learned my addr 172.21.15.44:0/2993326208 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.811+0000 7f6a74a30640 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef7a2180 0x558eef7ba830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.811+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 0x558eef7b6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/2993326208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef7a2180 msgr2=0x558eef7ba830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef7a2180 0x558eef7ba830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/2993326208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 msgr2=0x558eef7b6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 0x558eef7b6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a6f7fe640 1 -- 172.21.15.44:0/2993326208 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eef7b7070 con 0x558eef78ad80 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 0x558eef7b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a6f7fe640 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef77dc00 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f6a640147c0 tx=0x7f6a64009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae812e1c8c2eed5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a4effd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a4effd640 1 -- 172.21.15.44:0/2993326208 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64046060 con 0x558eef78ad80 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a4effd640 1 -- 172.21.15.44:0/2993326208 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f6a64030670 con 0x558eef78ad80 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a79084740 1 -- 172.21.15.44:0/2993326208 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a6402d070 con 0x558eef78ad80 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a4effd640 1 -- 172.21.15.44:0/2993326208 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f6a6404c050 con 0x558eef78ad80 2024-04-02T13:30:18.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a79084740 10 client.9318 fetch_fsmap learned FSMap version 88 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a79084740 1 -- 172.21.15.44:0/2993326208 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) -- 0x558eef7b7310 con 0x558eef78ad80 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a4effd640 1 -- 172.21.15.44:0/2993326208 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a6402d070 con 0x558eef78ad80 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a4effd640 1 -- 172.21.15.44:0/2993326208 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 88) ==== 3494+0+0 (secure 0 0 0) 0x7f6a6404f020 con 0x558eef78ad80 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a79084740 10 client.9318 fetch_fsmap finished waiting for FSMap version 88 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a79084740 10 client.9318 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a79084740 20 client.9318 populate_metadata read hostname 'smithi044' 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a79084740 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x558eef8220c0 0x558eef7bad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a79084740 4 client.9318 mds_command: new command op to 9192 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.812+0000 7f6a79084740 1 -- 172.21.15.44:0/2993326208 --> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558eef6cf0c0 con 0x558eef8220c0 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x558eef8220c0 0x558eef7bad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a4effd640 10 client.9318 ms_handle_connect on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a6ffff640 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x558eef8220c0 0x558eef7bad70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9186b291fe52b505 server_cookie=aabe2643a88e5657 in_seq=0 out_seq=0 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a4effd640 1 -- 172.21.15.44:0/2993326208 <== mds.0 v2:172.21.15.44:6833/1745009605 1 ==== command_reply(tid 0: 0 ) ==== 8+0+260 (crc 0 0 0) 0x558eef6cf0c0 con 0x558eef8220c0 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a4effd640 10 client.9318 handle_command_reply: tid=0 2024-04-02T13:30:18.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a79084740 1 client.9318 shutdown 2024-04-02T13:30:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a79084740 1 -- 172.21.15.44:0/2993326208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 msgr2=0x558eef77dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a79084740 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef77dc00 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f6a640147c0 tx=0x7f6a64009190 comp rx=0 tx=0).stop 2024-04-02T13:30:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a79084740 1 -- 172.21.15.44:0/2993326208 shutdown_connections 2024-04-02T13:30:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a79084740 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x558eef8220c0 0x558eef7bad70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a79084740 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558eef7a2180 0x558eef7ba830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a79084740 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558eef78ad80 0x558eef77dc00 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a4effd640 0 client.9318 ms_handle_reset on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a79084740 1 --2- 172.21.15.44:0/2993326208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558eef78a9a0 0x558eef7b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a79084740 1 -- 172.21.15.44:0/2993326208 >> 172.21.15.44:0/2993326208 conn(0x558eef75de80 msgr2=0x558eef7900a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a79084740 1 -- 172.21.15.44:0/2993326208 shutdown_connections 2024-04-02T13:30:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a79084740 1 -- 172.21.15.44:0/2993326208 wait complete. 2024-04-02T13:30:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.813+0000 7f6a79084740 20 client.9318 trim_cache size 0 max 16384 2024-04-02T13:30:18.823 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:30:18.823 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T13:30:18.823 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-02T13:30:18.823 INFO:teuthology.orchestra.run.smithi044.stdout: "b70ed285-5c2e-47b7-b888-d823c1e21df1": { 2024-04-02T13:30:18.823 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-02T13:30:18.823 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "b70ed285-5c2e-47b7-b888-d823c1e21df1", 2024-04-02T13:30:18.823 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,repair" 2024-04-02T13:30:18.823 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:30:18.823 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:30:18.823 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:30:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.814+0000 7f6a76493640 1 -- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6a5c06c920 msgr2=0x7f6a5c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.814+0000 7f6a76493640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6a5c06c920 0x7f6a5c06ede0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6a60036000 tx=0x7f6a60030000 comp rx=0 tx=0).stop 2024-04-02T13:30:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.814+0000 7f6a76493640 1 -- 172.21.15.44:0/3341155960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a7015ad20 msgr2=0x7f6a7015b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.814+0000 7f6a76493640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a7015ad20 0x7f6a7015b1b0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f6a700d0d80 tx=0x7f6a6800e390 comp rx=0 tx=0).stop 2024-04-02T13:30:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.814+0000 7f6a76493640 1 -- 172.21.15.44:0/3341155960 shutdown_connections 2024-04-02T13:30:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.814+0000 7f6a76493640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6a5c06c920 0x7f6a5c06ede0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.814+0000 7f6a76493640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a7015ad20 0x7f6a7015b1b0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.814+0000 7f6a76493640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70153a40 0x7f6a700ad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.814+0000 7f6a76493640 1 --2- 172.21.15.44:0/3341155960 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70151ac0 0x7f6a701f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:18.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.814+0000 7f6a76493640 1 -- 172.21.15.44:0/3341155960 >> 172.21.15.44:0/3341155960 conn(0x7f6a7006b920 msgr2=0x7f6a7006dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:18.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.814+0000 7f6a76493640 1 -- 172.21.15.44:0/3341155960 shutdown_connections 2024-04-02T13:30:18.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:18.814+0000 7f6a76493640 1 -- 172.21.15.44:0/3341155960 wait complete. 2024-04-02T13:30:18.827 INFO:tasks.cephfs.filesystem:scrub status for tag:b70ed285-5c2e-47b7-b888-d823c1e21df1 - {'path': '/', 'tag': 'b70ed285-5c2e-47b7-b888-d823c1e21df1', 'options': 'recursive,repair'} 2024-04-02T13:30:19.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:19.656+0000 7f2a06ffd640 1 -- 172.21.15.44:0/503258263 <== mon.0 v2:172.21.15.44:3300/0 8 ==== log(8 entries from seq 1 at 2024-04-02T13:30:18.658782+0000) ==== 1407+0+0 (secure 0 0 0) 0x7f2a080af000 con 0x7f2a1015e4d0 2024-04-02T13:30:19.659 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:30:18.658782+0000 mds.a [INF] scrub queued for path: / 2024-04-02T13:30:19.659 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:30:18.658786+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-04-02T13:30:19.659 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:30:18.658790+0000 mds.a [INF] scrub summary: active paths [/] 2024-04-02T13:30:19.660 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:30:18.659424+0000 mds.a [ERR] scrub: inode wrongly marked free: 0x10000000001 2024-04-02T13:30:19.660 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:30:18.659435+0000 mds.a [ERR] inode table repaired for inode: 0x10000000001 2024-04-02T13:30:19.660 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:30:18.659878+0000 mds.a [ERR] scrub: inode wrongly marked free: 0x10000000002 2024-04-02T13:30:19.660 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:30:18.659886+0000 mds.a [ERR] inode table repaired for inode: 0x10000000002 2024-04-02T13:30:19.660 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:30:18.660199+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-04-02T13:30:23.733 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:30:23.736 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:30:24.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:24.664+0000 7f2a06ffd640 1 -- 172.21.15.44:0/503258263 <== mon.0 v2:172.21.15.44:3300/0 9 ==== log(1 entries from seq 9 at 2024-04-02T13:30:22.669416+0000) ==== 189+0+0 (secure 0 0 0) 0x7f2a0807a460 con 0x7f2a1015e4d0 2024-04-02T13:30:24.668 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:30:22.669416+0000 mds.a [INF] scrub summary: idle 2024-04-02T13:30:48.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:30:48.829 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-02T13:30:48.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.941+0000 7f714038f640 1 Processor -- start 2024-04-02T13:30:48.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.941+0000 7f714038f640 1 -- start start 2024-04-02T13:30:48.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f714038f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f713806d520 0x7f71380b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f714038f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f713815c6f0 con 0x7f71380af360 2024-04-02T13:30:48.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f714038f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f713806da90 con 0x7f71380b2340 2024-04-02T13:30:48.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f714038f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f71380b4840 con 0x7f713806d520 2024-04-02T13:30:48.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713e12b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f71380af360 0x7f71380b1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37782/0 (socket says 172.21.15.44:37782) 2024-04-02T13:30:48.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713e12b640 1 -- 172.21.15.44:0/3891311340 learned_addr learned my addr 172.21.15.44:0/3891311340 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:48.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713e92c640 1 --2- 172.21.15.44:0/3891311340 >> v2:172.21.15.69:3300/0 conn(0x7f713806d520 0x7f71380b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4071953177 0 0) 0x7f713815c6f0 con 0x7f71380af360 2024-04-02T13:30:48.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7124003520 con 0x7f71380af360 2024-04-02T13:30:48.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3561235616 0 0) 0x7f7124003520 con 0x7f71380af360 2024-04-02T13:30:48.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 >> v2:172.21.15.69:3300/0 conn(0x7f713806d520 msgr2=0x7f71380b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:48.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 --2- 172.21.15.44:0/3891311340 >> v2:172.21.15.69:3300/0 conn(0x7f713806d520 0x7f71380b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 >> v1:172.21.15.69:6789/0 conn(0x7f71380b2340 legacy=0x7f7138164110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:48.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713806dd70 con 0x7f71380af360 2024-04-02T13:30:48.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f7134001690 con 0x7f71380af360 2024-04-02T13:30:48.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713e92c640 1 --2- 172.21.15.44:0/3891311340 >> v2:172.21.15.69:3300/0 conn(0x7f713806d520 0x7f71380b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:30:48.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 >> v1:172.21.15.44:6789/0 conn(0x7f71380af360 legacy=0x7f71380b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:48.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 --2- 172.21.15.44:0/3891311340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71240039b0 0x7f7124003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 --2- 172.21.15.44:0/3891311340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7124004800 0x7f7124004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 --2- 172.21.15.44:0/3891311340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71240051c0 0x7f712400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f71380b4840 con 0x7f7124004800 2024-04-02T13:30:48.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f712400c060 con 0x7f71240039b0 2024-04-02T13:30:48.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.942+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f712400c260 con 0x7f71240051c0 2024-04-02T13:30:48.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.943+0000 7f713d92a640 1 --2- 172.21.15.44:0/3891311340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7124004800 0x7f7124004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.943+0000 7f713e12b640 1 --2- 172.21.15.44:0/3891311340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71240039b0 0x7f7124003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.943+0000 7f713e92c640 1 --2- 172.21.15.44:0/3891311340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71240051c0 0x7f712400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.943+0000 7f713d92a640 1 -- 172.21.15.44:0/3891311340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71240051c0 msgr2=0x7f712400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:48.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.943+0000 7f713d92a640 1 --2- 172.21.15.44:0/3891311340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71240051c0 0x7f712400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.943+0000 7f713d92a640 1 -- 172.21.15.44:0/3891311340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71240039b0 msgr2=0x7f7124003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:48.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.943+0000 7f713d92a640 1 --2- 172.21.15.44:0/3891311340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71240039b0 0x7f7124003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.943+0000 7f713d92a640 1 -- 172.21.15.44:0/3891311340 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7124003520 con 0x7f7124004800 2024-04-02T13:30:48.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.943+0000 7f713e92c640 1 --2- 172.21.15.44:0/3891311340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71240051c0 0x7f712400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:48.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.943+0000 7f713e92c640 1 -- 172.21.15.44:0/3891311340 reap_dead start 2024-04-02T13:30:48.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.944+0000 7f713d92a640 1 --2- 172.21.15.44:0/3891311340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7124004800 0x7f7124004c80 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f7128001750 tx=0x7f7124005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69d986b4e908e9e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:48.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.944+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7128001370 con 0x7f7124004800 2024-04-02T13:30:48.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.944+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71280052d0 con 0x7f7124004800 2024-04-02T13:30:48.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.944+0000 7f713d129640 1 -- 172.21.15.44:0/3891311340 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7128003830 con 0x7f7124004800 2024-04-02T13:30:48.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.944+0000 7f714038f640 1 -- 172.21.15.44:0/3891311340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7124004800 msgr2=0x7f7124004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:48.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.944+0000 7f714038f640 1 --2- 172.21.15.44:0/3891311340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7124004800 0x7f7124004c80 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f7128001750 tx=0x7f7124005830 comp rx=0 tx=0).stop 2024-04-02T13:30:48.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.944+0000 7f714038f640 1 -- 172.21.15.44:0/3891311340 shutdown_connections 2024-04-02T13:30:48.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.944+0000 7f714038f640 1 --2- 172.21.15.44:0/3891311340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7124004800 0x7f7124004c80 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.944+0000 7f714038f640 1 -- 172.21.15.44:0/3891311340 >> 172.21.15.44:0/3891311340 conn(0x7f713806b4d0 msgr2=0x7f713806b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:48.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.944+0000 7f714038f640 1 -- 172.21.15.44:0/3891311340 shutdown_connections 2024-04-02T13:30:48.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.944+0000 7f714038f640 1 -- 172.21.15.44:0/3891311340 wait complete. 2024-04-02T13:30:48.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.946+0000 7f714038f640 1 Processor -- start 2024-04-02T13:30:48.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.946+0000 7f714038f640 1 -- start start 2024-04-02T13:30:48.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.946+0000 7f714038f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7124004800 0x7f7138150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.946+0000 7f714038f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f713815c8f0 0x7f71380c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.946+0000 7f714038f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71380b0ac0 0x7f71380b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.946+0000 7f714038f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f713806e330 con 0x7f71380b0ac0 2024-04-02T13:30:48.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.946+0000 7f714038f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f713806e6a0 con 0x7f7124004800 2024-04-02T13:30:48.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.946+0000 7f714038f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f71380af740 con 0x7f713815c8f0 2024-04-02T13:30:48.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f713e92c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71380b0ac0 0x7f71380b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f713e92c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71380b0ac0 0x7f71380b0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58736/0 (socket says 172.21.15.44:58736) 2024-04-02T13:30:48.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f713e92c640 1 -- 172.21.15.44:0/3719874041 learned_addr learned my addr 172.21.15.44:0/3719874041 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f713d92a640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f713815c8f0 0x7f71380c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f713e12b640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7124004800 0x7f7138150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f713e92c640 1 -- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f713815c8f0 msgr2=0x7f71380c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f713e92c640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f713815c8f0 0x7f71380c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f713e92c640 1 -- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7124004800 msgr2=0x7f7138150860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f713e92c640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7124004800 0x7f7138150860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f713e92c640 1 -- 172.21.15.44:0/3719874041 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7138150da0 con 0x7f71380b0ac0 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f713e92c640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71380b0ac0 0x7f71380b0f40 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f71380b3840 tx=0x7f71300115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f712effd640 1 -- 172.21.15.44:0/3719874041 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7130003990 con 0x7f71380b0ac0 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f712effd640 1 -- 172.21.15.44:0/3719874041 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f713001b070 con 0x7f71380b0ac0 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f714038f640 1 -- 172.21.15.44:0/3719874041 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71380c8910 con 0x7f71380b0ac0 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f712effd640 1 -- 172.21.15.44:0/3719874041 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7130003b90 con 0x7f71380b0ac0 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.947+0000 7f714038f640 1 -- 172.21.15.44:0/3719874041 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71380c8ed0 con 0x7f71380b0ac0 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.949+0000 7f712effd640 1 -- 172.21.15.44:0/3719874041 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7130005d10 con 0x7f71380b0ac0 2024-04-02T13:30:48.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.949+0000 7f712effd640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f711c06c8f0 0x7f711c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.949+0000 7f712effd640 1 -- 172.21.15.44:0/3719874041 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f713008c200 con 0x7f71380b0ac0 2024-04-02T13:30:48.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.950+0000 7f713e12b640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f711c06c8f0 0x7f711c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.950+0000 7f713e12b640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f711c06c8f0 0x7f711c06edb0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7134001800 tx=0x7f71340014f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:48.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.951+0000 7f7142f79740 1 Processor -- start 2024-04-02T13:30:48.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.951+0000 7f7142f79740 1 -- start start 2024-04-02T13:30:48.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f7142f79740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 0x55e9ca1ba940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f7142f79740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 0x55e9ca1ea730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f7142f79740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1bae80 0x55e9ca1eac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f7142f79740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e9ca124830 con 0x55e9ca1e7d80 2024-04-02T13:30:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f7142f79740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e9ca0cc820 con 0x55e9ca1bae80 2024-04-02T13:30:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f7142f79740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55e9ca1207e0 con 0x55e9ca1e79a0 2024-04-02T13:30:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f713d92a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 0x55e9ca1ea730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f713d92a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 0x55e9ca1ea730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58752/0 (socket says 172.21.15.44:58752) 2024-04-02T13:30:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f713d92a640 1 -- 172.21.15.44:0/4167023229 learned_addr learned my addr 172.21.15.44:0/4167023229 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f713e12b640 1 --2- 172.21.15.44:0/4167023229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 0x55e9ca1ba940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f713e92c640 1 --2- 172.21.15.44:0/4167023229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1bae80 0x55e9ca1eac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f713d92a640 1 -- 172.21.15.44:0/4167023229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 msgr2=0x55e9ca1ba940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f713d92a640 1 --2- 172.21.15.44:0/4167023229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 0x55e9ca1ba940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f713d92a640 1 -- 172.21.15.44:0/4167023229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1bae80 msgr2=0x55e9ca1eac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f713d92a640 1 --2- 172.21.15.44:0/4167023229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1bae80 0x55e9ca1eac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f713d92a640 1 -- 172.21.15.44:0/4167023229 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9c9f027d0 con 0x55e9ca1e7d80 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f713d92a640 1 --2- 172.21.15.44:0/4167023229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 0x55e9ca1ea730 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f7128001750 tx=0x7f712802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ad13af2b1cf3b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f711bfff640 1 -- 172.21.15.44:0/4167023229 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71280052d0 con 0x55e9ca1e7d80 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f711bfff640 1 -- 172.21.15.44:0/4167023229 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7128005470 con 0x55e9ca1e7d80 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.952+0000 7f711bfff640 1 -- 172.21.15.44:0/4167023229 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7128007240 con 0x55e9ca1e7d80 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.953+0000 7f7142f79740 1 -- 172.21.15.44:0/4167023229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 msgr2=0x55e9ca1ea730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.953+0000 7f7142f79740 1 --2- 172.21.15.44:0/4167023229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 0x55e9ca1ea730 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f7128001750 tx=0x7f712802f000 comp rx=0 tx=0).stop 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.953+0000 7f7142f79740 1 -- 172.21.15.44:0/4167023229 shutdown_connections 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.953+0000 7f7142f79740 1 --2- 172.21.15.44:0/4167023229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1bae80 0x55e9ca1eac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.953+0000 7f7142f79740 1 --2- 172.21.15.44:0/4167023229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 0x55e9ca1ea730 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.953+0000 7f7142f79740 1 --2- 172.21.15.44:0/4167023229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 0x55e9ca1ba940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.953+0000 7f7142f79740 1 -- 172.21.15.44:0/4167023229 >> 172.21.15.44:0/4167023229 conn(0x55e9ca1bcbb0 msgr2=0x55e9ca205730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.953+0000 7f7142f79740 1 -- 172.21.15.44:0/4167023229 shutdown_connections 2024-04-02T13:30:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.953+0000 7f7142f79740 1 -- 172.21.15.44:0/4167023229 wait complete. 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.954+0000 7f7142f79740 1 -- start start 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.954+0000 7f7142f79740 1 Processor -- start 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f7142f79740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 0x55e9ca205f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f7142f79740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 0x55e9ca1e8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f7142f79740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e9330 0x55e9ca1fc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f7142f79740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e9ca1bf150 con 0x55e9ca1e7d80 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f7142f79740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e9ca0dfd30 con 0x55e9ca1e9330 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f7142f79740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55e9ca1be540 con 0x55e9ca1e79a0 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f7142f79740 10 client.0 _ll_register_callbacks cb 0x55e9ca0df720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f713d92a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 0x55e9ca1e8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f713d92a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 0x55e9ca1e8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58760/0 (socket says 172.21.15.44:58760) 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f713d92a640 1 -- 172.21.15.44:0/2595562334 learned_addr learned my addr 172.21.15.44:0/2595562334 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f713e12b640 1 --2- 172.21.15.44:0/2595562334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 0x55e9ca205f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f713e92c640 1 --2- 172.21.15.44:0/2595562334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e9330 0x55e9ca1fc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f713d92a640 1 -- 172.21.15.44:0/2595562334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 msgr2=0x55e9ca205f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f713d92a640 1 --2- 172.21.15.44:0/2595562334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 0x55e9ca205f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f713d92a640 1 -- 172.21.15.44:0/2595562334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e9330 msgr2=0x55e9ca1fc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f713d92a640 1 --2- 172.21.15.44:0/2595562334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e9330 0x55e9ca1fc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f713d92a640 1 -- 172.21.15.44:0/2595562334 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9c9e383f0 con 0x55e9ca1e7d80 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f713d92a640 1 --2- 172.21.15.44:0/2595562334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 0x55e9ca1e8df0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f7128007ba0 tx=0x7f7128001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a54ca54200d2d98 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.955+0000 7f71197fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f71197fa640 1 -- 172.21.15.44:0/2595562334 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71280052d0 con 0x55e9ca1e7d80 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f71197fa640 1 -- 172.21.15.44:0/2595562334 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f7128007230 con 0x55e9ca1e7d80 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f7142f79740 1 -- 172.21.15.44:0/2595562334 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7128005470 con 0x55e9ca1e7d80 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f71197fa640 1 -- 172.21.15.44:0/2595562334 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f7128057050 con 0x55e9ca1e7d80 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f7142f79740 10 client.9330 fetch_fsmap learned FSMap version 88 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f7142f79740 1 -- 172.21.15.44:0/2595562334 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) -- 0x55e9c9df80f0 con 0x55e9ca1e7d80 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f71197fa640 1 -- 172.21.15.44:0/2595562334 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7128005470 con 0x55e9ca1e7d80 2024-04-02T13:30:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f71197fa640 1 -- 172.21.15.44:0/2595562334 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 88) ==== 3494+0+0 (secure 0 0 0) 0x7f7128007430 con 0x55e9ca1e7d80 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f7142f79740 10 client.9330 fetch_fsmap finished waiting for FSMap version 88 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f7142f79740 10 client.9330 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f7142f79740 20 client.9330 populate_metadata read hostname 'smithi044' 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f7142f79740 1 --2- 172.21.15.44:0/2595562334 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55e9ca219010 0x55e9ca219420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f7142f79740 4 client.9330 mds_command: new command op to 9192 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f7142f79740 1 -- 172.21.15.44:0/2595562334 --> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9ca131d70 con 0x55e9ca219010 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.956+0000 7f713e12b640 1 --2- 172.21.15.44:0/2595562334 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55e9ca219010 0x55e9ca219420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f71197fa640 10 client.9330 ms_handle_connect on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f713e12b640 1 --2- 172.21.15.44:0/2595562334 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55e9ca219010 0x55e9ca219420 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9667155439b8cdbf server_cookie=ab069bb8d19b2973 in_seq=0 out_seq=0 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f71197fa640 1 -- 172.21.15.44:0/2595562334 <== mds.0 v2:172.21.15.44:6833/1745009605 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e9ca131d70 con 0x55e9ca219010 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f71197fa640 10 client.9330 handle_command_reply: tid=0 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f7142f79740 1 client.9330 shutdown 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f7142f79740 1 -- 172.21.15.44:0/2595562334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 msgr2=0x55e9ca1e8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f7142f79740 1 --2- 172.21.15.44:0/2595562334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 0x55e9ca1e8df0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f7128007ba0 tx=0x7f7128001190 comp rx=0 tx=0).stop 2024-04-02T13:30:48.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f7142f79740 1 -- 172.21.15.44:0/2595562334 shutdown_connections 2024-04-02T13:30:48.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f7142f79740 1 --2- 172.21.15.44:0/2595562334 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55e9ca219010 0x55e9ca219420 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f7142f79740 1 --2- 172.21.15.44:0/2595562334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e9330 0x55e9ca1fc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f7142f79740 1 --2- 172.21.15.44:0/2595562334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e7d80 0x55e9ca1e8df0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f71197fa640 0 client.9330 ms_handle_reset on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:48.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f7142f79740 1 --2- 172.21.15.44:0/2595562334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 0x55e9ca205f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f7142f79740 1 -- 172.21.15.44:0/2595562334 >> 172.21.15.44:0/2595562334 conn(0x55e9ca1bae80 msgr2=0x55e9ca208bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:48.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f7142f79740 1 -- 172.21.15.44:0/2595562334 shutdown_connections 2024-04-02T13:30:48.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.957+0000 7f7142f79740 1 -- 172.21.15.44:0/2595562334 wait complete. 2024-04-02T13:30:48.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.958+0000 7f7142f79740 20 client.9330 trim_cache size 0 max 16384 2024-04-02T13:30:48.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.975+0000 7f7142f79740 1 Processor -- start 2024-04-02T13:30:48.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.975+0000 7f7142f79740 1 -- start start 2024-04-02T13:30:48.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f7142f79740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 0x55e9ca1e2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f7142f79740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e7d80 0x55e9ca207990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f7142f79740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e9330 0x55e9ca207ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f7142f79740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e9ca1bad00 con 0x55e9ca1e9330 2024-04-02T13:30:48.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f7142f79740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7138201890 con 0x55e9ca1e7d80 2024-04-02T13:30:48.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f7142f79740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7138200870 con 0x55e9ca1e79a0 2024-04-02T13:30:48.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f713e92c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e9330 0x55e9ca207ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f713e92c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e9330 0x55e9ca207ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58766/0 (socket says 172.21.15.44:58766) 2024-04-02T13:30:48.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f713e92c640 1 -- 172.21.15.44:0/2894172106 learned_addr learned my addr 172.21.15.44:0/2894172106 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:48.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f713e12b640 1 --2- 172.21.15.44:0/2894172106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 0x55e9ca1e2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f713d92a640 1 --2- 172.21.15.44:0/2894172106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e7d80 0x55e9ca207990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f713e92c640 1 -- 172.21.15.44:0/2894172106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 msgr2=0x55e9ca1e2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:48.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f713e92c640 1 --2- 172.21.15.44:0/2894172106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 0x55e9ca1e2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f713e92c640 1 -- 172.21.15.44:0/2894172106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e7d80 msgr2=0x55e9ca207990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:48.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f713e92c640 1 --2- 172.21.15.44:0/2894172106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e7d80 0x55e9ca207990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f713e92c640 1 -- 172.21.15.44:0/2894172106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9ca219790 con 0x55e9ca1e9330 2024-04-02T13:30:48.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f713e12b640 1 --2- 172.21.15.44:0/2894172106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 0x55e9ca1e2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:48.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.976+0000 7f713d92a640 1 --2- 172.21.15.44:0/2894172106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e7d80 0x55e9ca207990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:48.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.977+0000 7f713e92c640 1 --2- 172.21.15.44:0/2894172106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e9330 0x55e9ca207ed0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f7130056200 tx=0x7f7130056040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33caf678d4cbe267 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:48.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.977+0000 7f711b7fe640 1 -- 172.21.15.44:0/2894172106 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713005e0a0 con 0x55e9ca1e9330 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.977+0000 7f711b7fe640 1 -- 172.21.15.44:0/2894172106 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7130083e50 con 0x55e9ca1e9330 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.977+0000 7f711b7fe640 1 -- 172.21.15.44:0/2894172106 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71300628d0 con 0x55e9ca1e9330 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.977+0000 7f7142f79740 1 -- 172.21.15.44:0/2894172106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e9330 msgr2=0x55e9ca207ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.977+0000 7f7142f79740 1 --2- 172.21.15.44:0/2894172106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e9330 0x55e9ca207ed0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f7130056200 tx=0x7f7130056040 comp rx=0 tx=0).stop 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.977+0000 7f7142f79740 1 -- 172.21.15.44:0/2894172106 shutdown_connections 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.977+0000 7f7142f79740 1 --2- 172.21.15.44:0/2894172106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e9330 0x55e9ca207ed0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.977+0000 7f7142f79740 1 --2- 172.21.15.44:0/2894172106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e7d80 0x55e9ca207990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.977+0000 7f7142f79740 1 --2- 172.21.15.44:0/2894172106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e79a0 0x55e9ca1e2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.977+0000 7f7142f79740 1 -- 172.21.15.44:0/2894172106 >> 172.21.15.44:0/2894172106 conn(0x55e9ca1bae80 msgr2=0x55e9ca1e8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.977+0000 7f7142f79740 1 -- 172.21.15.44:0/2894172106 shutdown_connections 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.977+0000 7f7142f79740 1 -- 172.21.15.44:0/2894172106 wait complete. 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.978+0000 7f7142f79740 1 -- start start 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.978+0000 7f7142f79740 1 Processor -- start 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f7142f79740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e79a0 0x55e9ca27e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f7142f79740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e7d80 0x55e9ca205d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f7142f79740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e9330 0x55e9ca206270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f7142f79740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7138200220 con 0x55e9ca1e79a0 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f7142f79740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f71381ffbd0 con 0x55e9ca1e7d80 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f7142f79740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55e9ca0aaa00 con 0x55e9ca1e9330 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f7142f79740 10 client.0 _ll_register_callbacks cb 0x7f71300098f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f713e12b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e79a0 0x55e9ca27e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f713e12b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e79a0 0x55e9ca27e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58776/0 (socket says 172.21.15.44:58776) 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f713e12b640 1 -- 172.21.15.44:0/661270404 learned_addr learned my addr 172.21.15.44:0/661270404 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f713d92a640 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e7d80 0x55e9ca205d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f713e92c640 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e9330 0x55e9ca206270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f713e12b640 1 -- 172.21.15.44:0/661270404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e9330 msgr2=0x55e9ca206270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f713e12b640 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e9330 0x55e9ca206270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f713e12b640 1 -- 172.21.15.44:0/661270404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e7d80 msgr2=0x55e9ca205d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:48.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f713e12b640 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e7d80 0x55e9ca205d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f713e12b640 1 -- 172.21.15.44:0/661270404 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9ca1ba780 con 0x55e9ca1e79a0 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f713e92c640 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e9330 0x55e9ca206270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f713e12b640 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e79a0 0x55e9ca27e640 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f713815c8c0 tx=0x7f7134015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b43750866780f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.979+0000 7f71197fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f71197fa640 1 -- 172.21.15.44:0/661270404 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7134015710 con 0x55e9ca1e79a0 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f71197fa640 1 -- 172.21.15.44:0/661270404 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f7134021c10 con 0x55e9ca1e79a0 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f7142f79740 1 -- 172.21.15.44:0/661270404 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7130083e50 con 0x55e9ca1e79a0 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f71197fa640 1 -- 172.21.15.44:0/661270404 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f7134021e30 con 0x55e9ca1e79a0 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f7142f79740 10 client.9336 fetch_fsmap learned FSMap version 88 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f7142f79740 1 -- 172.21.15.44:0/661270404 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) -- 0x55e9ca217610 con 0x55e9ca1e79a0 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f71197fa640 1 -- 172.21.15.44:0/661270404 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7130083e50 con 0x55e9ca1e79a0 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f71197fa640 1 -- 172.21.15.44:0/661270404 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 88) ==== 3494+0+0 (secure 0 0 0) 0x7f71340158b0 con 0x55e9ca1e79a0 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f7142f79740 10 client.9336 fetch_fsmap finished waiting for FSMap version 88 2024-04-02T13:30:48.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f7142f79740 10 client.9336 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f7142f79740 20 client.9336 populate_metadata read hostname 'smithi044' 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f7142f79740 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55e9ca1c1bc0 0x55e9ca1fb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f7142f79740 4 client.9336 mds_command: new command op to 9192 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f7142f79740 1 -- 172.21.15.44:0/661270404 --> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e9ca12c0c0 con 0x55e9ca1c1bc0 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f713d92a640 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55e9ca1c1bc0 0x55e9ca1fb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f713d92a640 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55e9ca1c1bc0 0x55e9ca1fb0c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11dafa8eb783aca8 server_cookie=6e3687f943597245 in_seq=0 out_seq=0 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.980+0000 7f71197fa640 10 client.9336 ms_handle_connect on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f71197fa640 1 -- 172.21.15.44:0/661270404 <== mds.0 v2:172.21.15.44:6833/1745009605 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e9ca12c0c0 con 0x55e9ca1c1bc0 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f71197fa640 10 client.9336 handle_command_reply: tid=0 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f7142f79740 1 client.9336 shutdown 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f7142f79740 1 -- 172.21.15.44:0/661270404 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e79a0 msgr2=0x55e9ca27e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f7142f79740 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e79a0 0x55e9ca27e640 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f713815c8c0 tx=0x7f7134015000 comp rx=0 tx=0).stop 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f7142f79740 1 -- 172.21.15.44:0/661270404 shutdown_connections 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f7142f79740 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.44:6833/1745009605,v1:172.21.15.44:6834/1745009605] conn(0x55e9ca1c1bc0 0x55e9ca1fb0c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f7142f79740 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55e9ca1e9330 0x55e9ca206270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f7142f79740 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9ca1e7d80 0x55e9ca205d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f7142f79740 1 --2- 172.21.15.44:0/661270404 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e9ca1e79a0 0x55e9ca27e640 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f71197fa640 0 client.9336 ms_handle_reset on v2:172.21.15.44:6833/1745009605 2024-04-02T13:30:48.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f7142f79740 1 -- 172.21.15.44:0/661270404 >> 172.21.15.44:0/661270404 conn(0x55e9ca1bae80 msgr2=0x55e9ca1e2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:48.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f7142f79740 1 -- 172.21.15.44:0/661270404 shutdown_connections 2024-04-02T13:30:48.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f7142f79740 1 -- 172.21.15.44:0/661270404 wait complete. 2024-04-02T13:30:48.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f7142f79740 20 client.9336 trim_cache size 0 max 16384 2024-04-02T13:30:48.997 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:30:48.998 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-02T13:30:48.998 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-02T13:30:48.998 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:30:48.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.981+0000 7f714038f640 1 -- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f711c06c8f0 msgr2=0x7f711c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:48.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.982+0000 7f714038f640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f711c06c8f0 0x7f711c06edb0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7134001800 tx=0x7f71340014f0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.982+0000 7f714038f640 1 -- 172.21.15.44:0/3719874041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71380b0ac0 msgr2=0x7f71380b0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:48.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.982+0000 7f714038f640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71380b0ac0 0x7f71380b0f40 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f71380b3840 tx=0x7f71300115a0 comp rx=0 tx=0).stop 2024-04-02T13:30:48.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.982+0000 7f714038f640 1 -- 172.21.15.44:0/3719874041 shutdown_connections 2024-04-02T13:30:49.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.982+0000 7f714038f640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f711c06c8f0 0x7f711c06edb0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.982+0000 7f714038f640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71380b0ac0 0x7f71380b0f40 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.982+0000 7f714038f640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f713815c8f0 0x7f71380c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.982+0000 7f714038f640 1 --2- 172.21.15.44:0/3719874041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7124004800 0x7f7138150860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.982+0000 7f714038f640 1 -- 172.21.15.44:0/3719874041 >> 172.21.15.44:0/3719874041 conn(0x7f713806b4d0 msgr2=0x7f71380ad430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:49.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.982+0000 7f714038f640 1 -- 172.21.15.44:0/3719874041 shutdown_connections 2024-04-02T13:30:49.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:48.982+0000 7f714038f640 1 -- 172.21.15.44:0/3719874041 wait complete. 2024-04-02T13:30:49.000 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T13:30:49.102 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:30:49.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.201+0000 7f2806132640 1 Processor -- start 2024-04-02T13:30:49.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.201+0000 7f2806132640 1 -- start start 2024-04-02T13:30:49.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.202+0000 7f2806132640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28001617e0 0x7f28000ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.202+0000 7f2806132640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f28000cd3a0 0x7f28001629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.202+0000 7f2806132640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28000cef30 con 0x7f28000cffd0 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.202+0000 7f2806132640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2800162f00 con 0x7f28000cd3a0 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.202+0000 7f2806132640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f28000aea90 con 0x7f28001617e0 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.202+0000 7f27fffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f28000cd3a0 0x7f28001629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.202+0000 7f27fffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f28000cd3a0 0x7f28001629c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58790/0 (socket says 172.21.15.44:58790) 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.202+0000 7f27fffff640 1 -- 172.21.15.44:0/3337149706 learned_addr learned my addr 172.21.15.44:0/3337149706 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.203+0000 7f27ff7fe640 1 --2- 172.21.15.44:0/3337149706 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28001617e0 0x7f28000ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.203+0000 7f27fffff640 1 -- 172.21.15.44:0/3337149706 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28001617e0 msgr2=0x7f28000ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.203+0000 7f27fffff640 1 --2- 172.21.15.44:0/3337149706 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28001617e0 0x7f28000ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.203+0000 7f27fffff640 1 -- 172.21.15.44:0/3337149706 >> v1:172.21.15.69:6789/0 conn(0x7f28000cffd0 legacy=0x7f28000ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.203+0000 7f27fffff640 1 -- 172.21.15.44:0/3337149706 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28000cf210 con 0x7f28000cd3a0 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.203+0000 7f27ff7fe640 1 --2- 172.21.15.44:0/3337149706 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28001617e0 0x7f28000ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.203+0000 7f27fffff640 1 --2- 172.21.15.44:0/3337149706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28000cd3a0 0x7f28001629c0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009e50 tx=0x7f27f802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0bf7e1fb5014d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.204+0000 7f27fe7fc640 1 -- 172.21.15.44:0/3337149706 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f802c9e0 con 0x7f28000cd3a0 2024-04-02T13:30:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.204+0000 7f27fe7fc640 1 -- 172.21.15.44:0/3337149706 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27f802cb80 con 0x7f28000cd3a0 2024-04-02T13:30:49.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.204+0000 7f27fe7fc640 1 -- 172.21.15.44:0/3337149706 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f80343f0 con 0x7f28000cd3a0 2024-04-02T13:30:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.204+0000 7f2806132640 1 -- 172.21.15.44:0/3337149706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28000cd3a0 msgr2=0x7f28001629c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.204+0000 7f2806132640 1 --2- 172.21.15.44:0/3337149706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28000cd3a0 0x7f28001629c0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009e50 tx=0x7f27f802be40 comp rx=0 tx=0).stop 2024-04-02T13:30:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.204+0000 7f2806132640 1 -- 172.21.15.44:0/3337149706 shutdown_connections 2024-04-02T13:30:49.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.204+0000 7f2806132640 1 --2- 172.21.15.44:0/3337149706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28000cd3a0 0x7f28001629c0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.204+0000 7f2806132640 1 --2- 172.21.15.44:0/3337149706 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28001617e0 0x7f28000ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.204+0000 7f2806132640 1 -- 172.21.15.44:0/3337149706 >> 172.21.15.44:0/3337149706 conn(0x7f28000b20e0 msgr2=0x7f28000b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.204+0000 7f2806132640 1 -- 172.21.15.44:0/3337149706 shutdown_connections 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.204+0000 7f2806132640 1 -- 172.21.15.44:0/3337149706 wait complete. 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.205+0000 7f2806132640 1 Processor -- start 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.206+0000 7f2806132640 1 -- start start 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.206+0000 7f2806132640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28000cd3a0 0x7f28000b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.206+0000 7f2806132640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28000cffd0 0x7f28000b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.206+0000 7f2806132640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28001617e0 0x7f28000b5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.206+0000 7f2806132640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f28000cfab0 con 0x7f28000cffd0 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.206+0000 7f2806132640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f28000cf7d0 con 0x7f28000cd3a0 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.206+0000 7f2806132640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2800161bc0 con 0x7f28001617e0 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.206+0000 7f27feffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28000cffd0 0x7f28000b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.206+0000 7f27feffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28000cffd0 0x7f28000b4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58796/0 (socket says 172.21.15.44:58796) 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.206+0000 7f27feffd640 1 -- 172.21.15.44:0/2703976524 learned_addr learned my addr 172.21.15.44:0/2703976524 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.206+0000 7f27fffff640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28001617e0 0x7f28000b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.206+0000 7f27ff7fe640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28000cd3a0 0x7f28000b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.207+0000 7f27feffd640 1 -- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28001617e0 msgr2=0x7f28000b5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.207+0000 7f27feffd640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28001617e0 0x7f28000b5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.207+0000 7f27feffd640 1 -- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28000cd3a0 msgr2=0x7f28000b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.207+0000 7f27feffd640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28000cd3a0 0x7f28000b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.207+0000 7f27feffd640 1 -- 172.21.15.44:0/2703976524 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28000b3710 con 0x7f28000cffd0 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.207+0000 7f27feffd640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28000cffd0 0x7f28000b4cc0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f27f400c430 tx=0x7f27f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.207+0000 7f27e7fff640 1 -- 172.21.15.44:0/2703976524 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f4011790 con 0x7f28000cffd0 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.207+0000 7f27e7fff640 1 -- 172.21.15.44:0/2703976524 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27f4017070 con 0x7f28000cffd0 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.207+0000 7f27e7fff640 1 -- 172.21.15.44:0/2703976524 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f4011b70 con 0x7f28000cffd0 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.207+0000 7f2806132640 1 -- 172.21.15.44:0/2703976524 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f280015d500 con 0x7f28000cffd0 2024-04-02T13:30:49.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.207+0000 7f2806132640 1 -- 172.21.15.44:0/2703976524 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f280015dd40 con 0x7f28000cffd0 2024-04-02T13:30:49.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.208+0000 7f2806132640 1 -- 172.21.15.44:0/2703976524 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f280015d7e0 con 0x7f28000cffd0 2024-04-02T13:30:49.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.208+0000 7f27e7fff640 1 -- 172.21.15.44:0/2703976524 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f27f4011d10 con 0x7f28000cffd0 2024-04-02T13:30:49.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.209+0000 7f27e7fff640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27ec06c8f0 0x7f27ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.209+0000 7f27e7fff640 1 -- 172.21.15.44:0/2703976524 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6274+0+0 (secure 0 0 0) 0x7f27f408ba50 con 0x7f28000cffd0 2024-04-02T13:30:49.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.209+0000 7f27ff7fe640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27ec06c8f0 0x7f27ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.210+0000 7f27ff7fe640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27ec06c8f0 0x7f27ec06edb0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f27f000aac0 tx=0x7f27f0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:49.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.213+0000 7f27e7fff640 1 -- 172.21.15.44:0/2703976524 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f280015d7e0 con 0x7f28000cffd0 2024-04-02T13:30:49.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.435+0000 7f2806132640 1 -- 172.21.15.44:0/2703976524 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f28000ce190 con 0x7f28000cffd0 2024-04-02T13:30:49.439 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:30:49.435+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:30:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.437+0000 7f27e7fff640 1 -- 172.21.15.44:0/2703976524 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v89) ==== 161+0+0 (secure 0 0 0) 0x7f280015d500 con 0x7f28000cffd0 2024-04-02T13:30:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:30:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.439+0000 7f2806132640 1 -- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27ec06c8f0 msgr2=0x7f27ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.439+0000 7f2806132640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27ec06c8f0 0x7f27ec06edb0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f27f000aac0 tx=0x7f27f0008040 comp rx=0 tx=0).stop 2024-04-02T13:30:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.439+0000 7f2806132640 1 -- 172.21.15.44:0/2703976524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28000cffd0 msgr2=0x7f28000b4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.439+0000 7f2806132640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28000cffd0 0x7f28000b4cc0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f27f400c430 tx=0x7f27f400c8f0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.439+0000 7f2806132640 1 -- 172.21.15.44:0/2703976524 shutdown_connections 2024-04-02T13:30:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.439+0000 7f2806132640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f27ec06c8f0 0x7f27ec06edb0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.439+0000 7f2806132640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28001617e0 0x7f28000b5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.439+0000 7f2806132640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f28000cffd0 0x7f28000b4cc0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.439+0000 7f2806132640 1 --2- 172.21.15.44:0/2703976524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28000cd3a0 0x7f28000b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.439+0000 7f2806132640 1 -- 172.21.15.44:0/2703976524 >> 172.21.15.44:0/2703976524 conn(0x7f28000b20e0 msgr2=0x7f2800162460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:49.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.439+0000 7f2806132640 1 -- 172.21.15.44:0/2703976524 shutdown_connections 2024-04-02T13:30:49.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.439+0000 7f2806132640 1 -- 172.21.15.44:0/2703976524 wait complete. 2024-04-02T13:30:49.444 INFO:tasks.ceph.mds.a.smithi044.stderr: -1259> 2024-04-02T13:30:18.637+0000 7f7d36f9b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:30:49.444 INFO:tasks.ceph.mds.a.smithi044.stderr: -1127> 2024-04-02T13:30:18.658+0000 7f7d32f93640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000001 2024-04-02T13:30:49.444 INFO:tasks.ceph.mds.a.smithi044.stderr: -1124> 2024-04-02T13:30:18.658+0000 7f7d32f93640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000001 2024-04-02T13:30:49.444 INFO:tasks.ceph.mds.a.smithi044.stderr: -1088> 2024-04-02T13:30:18.659+0000 7f7d32f93640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000002 2024-04-02T13:30:49.445 INFO:tasks.ceph.mds.a.smithi044.stderr: -1085> 2024-04-02T13:30:18.659+0000 7f7d32f93640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000002 2024-04-02T13:30:49.455 DEBUG:teuthology.orchestra.run.smithi044:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 0 show inode 2024-04-02T13:30:49.470 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:30:49.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.490+0000 7fe63b040280 1 -- start start 2024-04-02T13:30:49.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.490+0000 7fe63b040280 1 Processor -- start 2024-04-02T13:30:49.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe63b040280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56403052cc00 0x5640312f8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe63b040280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56403052d800 0x5640312f8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe63b040280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56403052dc00 0x5640312f8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe63b040280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564031312600 con 0x56403052dc00 2024-04-02T13:30:49.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe63b040280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564031312480 con 0x56403052d800 2024-04-02T13:30:49.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe63b040280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564031312300 con 0x56403052cc00 2024-04-02T13:30:49.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe639fed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56403052dc00 0x5640312f8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe639fed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56403052dc00 0x5640312f8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58820/0 (socket says 172.21.15.44:58820) 2024-04-02T13:30:49.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe639fed640 1 -- 172.21.15.44:0/149693528 learned_addr learned my addr 172.21.15.44:0/149693528 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:49.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe6397ec640 1 --2- 172.21.15.44:0/149693528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56403052cc00 0x5640312f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe638feb640 1 --2- 172.21.15.44:0/149693528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56403052d800 0x5640312f8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe639fed640 1 -- 172.21.15.44:0/149693528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56403052cc00 msgr2=0x5640312f8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:49.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe639fed640 1 --2- 172.21.15.44:0/149693528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56403052cc00 0x5640312f8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe639fed640 1 -- 172.21.15.44:0/149693528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56403052d800 msgr2=0x5640312f8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:49.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe639fed640 1 --2- 172.21.15.44:0/149693528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56403052d800 0x5640312f8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe639fed640 1 -- 172.21.15.44:0/149693528 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564030562780 con 0x56403052dc00 2024-04-02T13:30:49.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe6397ec640 1 --2- 172.21.15.44:0/149693528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56403052cc00 0x5640312f8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:49.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.491+0000 7fe639fed640 1 --2- 172.21.15.44:0/149693528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56403052dc00 0x5640312f8b00 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x5640312fc450 tx=0x5640312c1f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15a39956f8fef6b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:49.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.492+0000 7fe636fe7640 1 -- 172.21.15.44:0/149693528 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5640305529c0 con 0x56403052dc00 2024-04-02T13:30:49.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.492+0000 7fe636fe7640 1 -- 172.21.15.44:0/149693528 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x564030558c40 con 0x56403052dc00 2024-04-02T13:30:49.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.492+0000 7fe636fe7640 1 -- 172.21.15.44:0/149693528 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564030552680 con 0x56403052dc00 2024-04-02T13:30:49.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.492+0000 7fe63b040280 1 -- 172.21.15.44:0/149693528 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640305623c0 con 0x56403052dc00 2024-04-02T13:30:49.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.492+0000 7fe63b040280 1 -- 172.21.15.44:0/149693528 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=0}) -- 0x564030563680 con 0x56403052dc00 2024-04-02T13:30:49.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.492+0000 7fe63b040280 4 waiting for MDS map... 2024-04-02T13:30:49.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.492+0000 7fe636fe7640 1 -- 172.21.15.44:0/149693528 <== mon.0 v2:172.21.15.44:3300/0 4 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x5640313fe200 con 0x56403052dc00 2024-04-02T13:30:49.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.492+0000 7fe636fe7640 1 -- 172.21.15.44:0/149693528 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 89) ==== 2885+0+0 (secure 0 0 0) 0x564030587180 con 0x56403052dc00 2024-04-02T13:30:49.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.492+0000 7fe63b040280 4 Got MDS map 89 2024-04-02T13:30:49.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.492+0000 7fe63b040280 10 main: main 2024-04-02T13:30:49.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.492+0000 7fe63b040280 4 main: connecting to RADOS... 2024-04-02T13:30:49.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.492+0000 7fe63b040280 1 Processor -- start 2024-04-02T13:30:49.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.493+0000 7fe63b040280 1 -- start start 2024-04-02T13:30:49.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.493+0000 7fe63b040280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564031446000 0x5640312f9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.493+0000 7fe63b040280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564031446400 0x5640312f9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.493+0000 7fe63b040280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564031446800 0x5640312f9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.493+0000 7fe63b040280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564031312180 con 0x564031446400 2024-04-02T13:30:49.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.493+0000 7fe63b040280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56403058e480 con 0x564031446000 2024-04-02T13:30:49.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.493+0000 7fe63b040280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x56403058e900 con 0x564031446800 2024-04-02T13:30:49.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.493+0000 7fe6397ec640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564031446400 0x5640312f9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.493+0000 7fe6397ec640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564031446400 0x5640312f9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58822/0 (socket says 172.21.15.44:58822) 2024-04-02T13:30:49.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.493+0000 7fe6397ec640 1 -- 172.21.15.44:0/2026667480 learned_addr learned my addr 172.21.15.44:0/2026667480 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:49.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.493+0000 7fe638feb640 1 --2- 172.21.15.44:0/2026667480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564031446800 0x5640312f9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.493+0000 7fe638feb640 1 --2- 172.21.15.44:0/2026667480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564031446000 0x5640312f9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.493+0000 7fe6397ec640 1 -- 172.21.15.44:0/2026667480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564031446800 msgr2=0x5640312f9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe6397ec640 1 --2- 172.21.15.44:0/2026667480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564031446800 0x5640312f9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe6397ec640 1 -- 172.21.15.44:0/2026667480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564031446000 msgr2=0x5640312f9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe6397ec640 1 --2- 172.21.15.44:0/2026667480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564031446000 0x5640312f9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe6397ec640 1 -- 172.21.15.44:0/2026667480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640305634a0 con 0x564031446400 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe638feb640 1 --2- 172.21.15.44:0/2026667480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564031446000 0x5640312f9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe6397ec640 1 --2- 172.21.15.44:0/2026667480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564031446400 0x5640312f9600 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x564031450180 tx=0x56403140b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e87a0bb5a751bba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe634fe3640 1 -- 172.21.15.44:0/2026667480 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5640305529c0 con 0x564031446400 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe634fe3640 1 -- 172.21.15.44:0/2026667480 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x564030558a80 con 0x564031446400 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe634fe3640 1 -- 172.21.15.44:0/2026667480 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564030552680 con 0x564031446400 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe63b040280 1 -- 172.21.15.44:0/2026667480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564031446400 msgr2=0x5640312f9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe63b040280 1 --2- 172.21.15.44:0/2026667480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564031446400 0x5640312f9600 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x564031450180 tx=0x56403140b920 comp rx=0 tx=0).stop 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe63b040280 1 -- 172.21.15.44:0/2026667480 shutdown_connections 2024-04-02T13:30:49.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe63b040280 1 --2- 172.21.15.44:0/2026667480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564031446800 0x5640312f9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe63b040280 1 --2- 172.21.15.44:0/2026667480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564031446400 0x5640312f9600 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe63b040280 1 --2- 172.21.15.44:0/2026667480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564031446000 0x5640312f9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe63b040280 1 -- 172.21.15.44:0/2026667480 >> 172.21.15.44:0/2026667480 conn(0x56403130fc00 msgr2=0x564030616800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe63b040280 1 -- 172.21.15.44:0/2026667480 shutdown_connections 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.494+0000 7fe63b040280 1 -- 172.21.15.44:0/2026667480 wait complete. 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.495+0000 7fe63b040280 1 Processor -- start 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.495+0000 7fe63b040280 1 -- start start 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.495+0000 7fe63b040280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564031446000 0x5640312f9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.495+0000 7fe63b040280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564031446400 0x5640312f9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.495+0000 7fe63b040280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564031446800 0x5640312f9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.495+0000 7fe63b040280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564031313380 con 0x564031446400 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.495+0000 7fe63b040280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564031312180 con 0x564031446000 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.495+0000 7fe63b040280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564031313500 con 0x564031446800 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.495+0000 7fe6397ec640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564031446400 0x5640312f9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.495+0000 7fe6397ec640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564031446400 0x5640312f9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58828/0 (socket says 172.21.15.44:58828) 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.495+0000 7fe6397ec640 1 -- 172.21.15.44:0/3546661243 learned_addr learned my addr 172.21.15.44:0/3546661243 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe638feb640 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564031446000 0x5640312f9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe638feb640 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564031446800 0x5640312f9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe6397ec640 1 -- 172.21.15.44:0/3546661243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564031446800 msgr2=0x5640312f9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe6397ec640 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564031446800 0x5640312f9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe6397ec640 1 -- 172.21.15.44:0/3546661243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564031446000 msgr2=0x5640312f9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe6397ec640 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564031446000 0x5640312f9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe6397ec640 1 -- 172.21.15.44:0/3546661243 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564030562000 con 0x564031446400 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe638feb640 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564031446000 0x5640312f9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe6397ec640 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564031446400 0x5640312f9600 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x564031450e40 tx=0x5640312c05a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe6337e0640 1 -- 172.21.15.44:0/3546661243 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5640305524e0 con 0x564031446400 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe6337e0640 1 -- 172.21.15.44:0/3546661243 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5640305588c0 con 0x564031446400 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe63b040280 1 -- 172.21.15.44:0/3546661243 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564030562960 con 0x564031446400 2024-04-02T13:30:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe6337e0640 1 -- 172.21.15.44:0/3546661243 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5640305529c0 con 0x564031446400 2024-04-02T13:30:49.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe63b040280 1 -- 172.21.15.44:0/3546661243 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564030563860 con 0x564031446400 2024-04-02T13:30:49.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.496+0000 7fe63b040280 4 main: resolving pool 15 2024-04-02T13:30:49.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.497+0000 7fe6337e0640 1 -- 172.21.15.44:0/3546661243 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x56403145ac00 con 0x564031446400 2024-04-02T13:30:49.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.498+0000 7fe6337e0640 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x56403089a000 0x5640312fa100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.498+0000 7fe6337e0640 1 -- 172.21.15.44:0/3546661243 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x564031458a00 con 0x564031446400 2024-04-02T13:30:49.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.498+0000 7fe63b040280 4 main: creating IoCtx.. 2024-04-02T13:30:49.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.498+0000 7fe63b040280 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x564031446c00 0x5640312fa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:49.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.498+0000 7fe63b040280 1 -- 172.21.15.44:0/3546661243 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:1 15.13 15:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e110) -- 0x564031447000 con 0x564031446c00 2024-04-02T13:30:49.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.498+0000 7fe638feb640 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x56403089a000 0x5640312fa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.498+0000 7fe638feb640 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x564031446c00 0x5640312fa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:49.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.498+0000 7fe638feb640 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x56403089a000 0x5640312fa100 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x564031450cf0 tx=0x56403082ed80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:49.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.498+0000 7fe638feb640 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x564031446c00 0x5640312fa680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:49.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe638feb640 1 -- 172.21.15.44:0/3546661243 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(1 mds0_inotable [read 0~34 out=34b] v0'0 uv7 ondisk = 0) ==== 157+0+34 (crc 0 0 0) 0x5640312b0000 con 0x564031446c00 2024-04-02T13:30:49.510 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:30:49.510 INFO:teuthology.orchestra.run.smithi044.stdout: "0": { 2024-04-02T13:30:49.510 INFO:teuthology.orchestra.run.smithi044.stdout: "data": { 2024-04-02T13:30:49.510 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 5, 2024-04-02T13:30:49.510 INFO:teuthology.orchestra.run.smithi044.stdout: "inotable": { 2024-04-02T13:30:49.510 INFO:teuthology.orchestra.run.smithi044.stdout: "projected_free": [ 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: "start": 1099511627779, 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: "len": 1099511627773 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: "free": [ 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: "start": 1099511627779, 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: "len": 1099511627773 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:30:49.511 INFO:teuthology.orchestra.run.smithi044.stdout: "result": 0 2024-04-02T13:30:49.512 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:30:49.512 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:30:49.512 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 -- 172.21.15.44:0/3546661243 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x564031446c00 msgr2=0x5640312fa680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x564031446c00 0x5640312fa680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 -- 172.21.15.44:0/3546661243 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x56403089a000 msgr2=0x5640312fa100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x56403089a000 0x5640312fa100 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x564031450cf0 tx=0x56403082ed80 comp rx=0 tx=0).stop 2024-04-02T13:30:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 -- 172.21.15.44:0/3546661243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564031446400 msgr2=0x5640312f9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 --2- 172.21.15.44:0/3546661243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564031446400 0x5640312f9600 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x564031450e40 tx=0x5640312c05a0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe6397ec640 1 -- 172.21.15.44:0/3546661243 reap_dead start 2024-04-02T13:30:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 -- 172.21.15.44:0/3546661243 shutdown_connections 2024-04-02T13:30:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 -- 172.21.15.44:0/3546661243 >> 172.21.15.44:0/3546661243 conn(0x56403130fc00 msgr2=0x564030616800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 -- 172.21.15.44:0/3546661243 shutdown_connections 2024-04-02T13:30:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 -- 172.21.15.44:0/3546661243 wait complete. 2024-04-02T13:30:49.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 -- 172.21.15.44:0/149693528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56403052dc00 msgr2=0x5640312f8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:49.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 --2- 172.21.15.44:0/149693528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56403052dc00 0x5640312f8b00 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x5640312fc450 tx=0x5640312c1f20 comp rx=0 tx=0).stop 2024-04-02T13:30:49.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 -- 172.21.15.44:0/149693528 shutdown_connections 2024-04-02T13:30:49.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 --2- 172.21.15.44:0/149693528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56403052dc00 0x5640312f8b00 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 --2- 172.21.15.44:0/149693528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56403052d800 0x5640312f8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 --2- 172.21.15.44:0/149693528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56403052cc00 0x5640312f8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:49.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 -- 172.21.15.44:0/149693528 >> 172.21.15.44:0/149693528 conn(0x56403052d000 msgr2=0x564030617000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:49.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 -- 172.21.15.44:0/149693528 shutdown_connections 2024-04-02T13:30:49.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:49.501+0000 7fe63b040280 1 -- 172.21.15.44:0/149693528 wait complete. 2024-04-02T13:30:49.515 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '0', 'show', 'inode'] in time 0:00:00.060400, result: { "0": { "data": { "version": 5, "inotable": { "projected_free": [ { "start": 1099511627779, "len": 1099511627773 } ], "free": [ { "start": 1099511627779, "len": 1099511627773 } ] } }, "result": 0 } } 2024-04-02T13:30:49.515 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:30:49.516 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:30:49.592 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:30:49.638 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:30:49.638 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:30:49.638 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:30:49.638 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:30:49.653 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:30:49.653 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:30:49.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:30:49.653 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:30:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:30:49.707 INFO:teuthology.orchestra.run.smithi044.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-02T13:30:49.707 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:30:49.708 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:30:49.708 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:30:49.724 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:30:49.724 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:30:49.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:30:49.724 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-02T13:30:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:30:49.781 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-02T13:30:49.781 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:30:49.781 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:30:49.782 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:30:49.797 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:30:49.797 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:30:49.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:30:49.797 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-02T13:30:49.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:30:49.852 INFO:teuthology.orchestra.run.smithi044.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-02T13:30:49.852 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:30:49.853 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:30:49.853 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:30:49.868 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:30:49.868 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:30:49.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:30:49.869 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-02T13:30:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:30:49.924 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-02T13:30:49.924 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:30:50.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.023+0000 7ff54b2a5640 1 Processor -- start 2024-04-02T13:30:50.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.024+0000 7ff54b2a5640 1 -- start start 2024-04-02T13:30:50.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.024+0000 7ff54b2a5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff5440b3510 0x7ff5440d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.024+0000 7ff54b2a5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff5440d08d0 0x7ff5440b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.024+0000 7ff54b2a5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5440af420 con 0x7ff54415c250 2024-04-02T13:30:50.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.024+0000 7ff54b2a5640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff54415c630 con 0x7ff5440d08d0 2024-04-02T13:30:50.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.024+0000 7ff54b2a5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff5440d0d20 con 0x7ff5440b3510 2024-04-02T13:30:50.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.024+0000 7ff549842640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff5440d08d0 0x7ff5440b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.024+0000 7ff549041640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff54415c250 0x7ff5440aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37796/0 (socket says 172.21.15.44:37796) 2024-04-02T13:30:50.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.025+0000 7ff549041640 1 -- 172.21.15.44:0/2271751913 learned_addr learned my addr 172.21.15.44:0/2271751913 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:50.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.025+0000 7ff548840640 1 --2- 172.21.15.44:0/2271751913 >> v2:172.21.15.69:3300/0 conn(0x7ff5440b3510 0x7ff5440d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.025+0000 7ff549842640 1 -- 172.21.15.44:0/2271751913 >> v2:172.21.15.69:3300/0 conn(0x7ff5440b3510 msgr2=0x7ff5440d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:50.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.025+0000 7ff549842640 1 --2- 172.21.15.44:0/2271751913 >> v2:172.21.15.69:3300/0 conn(0x7ff5440b3510 0x7ff5440d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.025+0000 7ff549842640 1 -- 172.21.15.44:0/2271751913 >> v1:172.21.15.44:6789/0 conn(0x7ff54415c250 legacy=0x7ff5440aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.025+0000 7ff549842640 1 -- 172.21.15.44:0/2271751913 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5440af700 con 0x7ff5440d08d0 2024-04-02T13:30:50.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.025+0000 7ff53bfff640 1 -- 172.21.15.44:0/2271751913 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 836006208 0 0) 0x7ff5440af420 con 0x7ff54415c250 2024-04-02T13:30:50.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.025+0000 7ff548840640 1 --2- 172.21.15.44:0/2271751913 >> v2:172.21.15.69:3300/0 conn(0x7ff5440b3510 0x7ff5440d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:50.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.026+0000 7ff549842640 1 --2- 172.21.15.44:0/2271751913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5440d08d0 0x7ff5440b04c0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7ff5440c8b70 tx=0x7ff54002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ae9f625e0acb20 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:50.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.026+0000 7ff53bfff640 1 -- 172.21.15.44:0/2271751913 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff540039070 con 0x7ff5440d08d0 2024-04-02T13:30:50.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.026+0000 7ff53bfff640 1 -- 172.21.15.44:0/2271751913 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff54002cbf0 con 0x7ff5440d08d0 2024-04-02T13:30:50.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.026+0000 7ff53bfff640 1 -- 172.21.15.44:0/2271751913 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff540034440 con 0x7ff5440d08d0 2024-04-02T13:30:50.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.026+0000 7ff54b2a5640 1 -- 172.21.15.44:0/2271751913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5440d08d0 msgr2=0x7ff5440b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.026+0000 7ff54b2a5640 1 --2- 172.21.15.44:0/2271751913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5440d08d0 0x7ff5440b04c0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7ff5440c8b70 tx=0x7ff54002be40 comp rx=0 tx=0).stop 2024-04-02T13:30:50.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.026+0000 7ff54b2a5640 1 -- 172.21.15.44:0/2271751913 shutdown_connections 2024-04-02T13:30:50.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.026+0000 7ff54b2a5640 1 --2- 172.21.15.44:0/2271751913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5440d08d0 0x7ff5440b04c0 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.026+0000 7ff54b2a5640 1 --2- 172.21.15.44:0/2271751913 >> v2:172.21.15.69:3300/0 conn(0x7ff5440b3510 0x7ff5440d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.026+0000 7ff54b2a5640 1 -- 172.21.15.44:0/2271751913 >> 172.21.15.44:0/2271751913 conn(0x7ff544073850 msgr2=0x7ff54409aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:50.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.026+0000 7ff54b2a5640 1 -- 172.21.15.44:0/2271751913 shutdown_connections 2024-04-02T13:30:50.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.026+0000 7ff54b2a5640 1 -- 172.21.15.44:0/2271751913 wait complete. 2024-04-02T13:30:50.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.027+0000 7ff54b2a5640 1 Processor -- start 2024-04-02T13:30:50.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.028+0000 7ff54b2a5640 1 -- start start 2024-04-02T13:30:50.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.028+0000 7ff54b2a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5440b3510 0x7ff5440b4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.028+0000 7ff54b2a5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5440d08d0 0x7ff5440b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.028+0000 7ff54b2a5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff54415c250 0x7ff544158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.028+0000 7ff54b2a5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff5440b0a00 con 0x7ff5440d08d0 2024-04-02T13:30:50.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.028+0000 7ff54b2a5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff5440afcc0 con 0x7ff5440b3510 2024-04-02T13:30:50.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.028+0000 7ff54b2a5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff5440b0b80 con 0x7ff54415c250 2024-04-02T13:30:50.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.028+0000 7ff548840640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5440d08d0 0x7ff5440b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.028+0000 7ff548840640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5440d08d0 0x7ff5440b5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58846/0 (socket says 172.21.15.44:58846) 2024-04-02T13:30:50.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.028+0000 7ff548840640 1 -- 172.21.15.44:0/1241711927 learned_addr learned my addr 172.21.15.44:0/1241711927 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:50.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.028+0000 7ff549041640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5440b3510 0x7ff5440b4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.028+0000 7ff549842640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff54415c250 0x7ff544158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.029+0000 7ff548840640 1 -- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff54415c250 msgr2=0x7ff544158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.029+0000 7ff548840640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff54415c250 0x7ff544158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.029+0000 7ff548840640 1 -- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5440b3510 msgr2=0x7ff5440b4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.029+0000 7ff548840640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5440b3510 0x7ff5440b4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.029+0000 7ff548840640 1 -- 172.21.15.44:0/1241711927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5440b50a0 con 0x7ff5440d08d0 2024-04-02T13:30:50.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.029+0000 7ff549041640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5440b3510 0x7ff5440b4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:50.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.029+0000 7ff548840640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5440d08d0 0x7ff5440b5980 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7ff53400c820 tx=0x7ff53400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:50.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.029+0000 7ff539ffb640 1 -- 172.21.15.44:0/1241711927 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534015070 con 0x7ff5440d08d0 2024-04-02T13:30:50.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.029+0000 7ff539ffb640 1 -- 172.21.15.44:0/1241711927 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff534007a30 con 0x7ff5440d08d0 2024-04-02T13:30:50.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.029+0000 7ff54b2a5640 1 -- 172.21.15.44:0/1241711927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5440c4b40 con 0x7ff5440d08d0 2024-04-02T13:30:50.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.029+0000 7ff539ffb640 1 -- 172.21.15.44:0/1241711927 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534007e00 con 0x7ff5440d08d0 2024-04-02T13:30:50.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.029+0000 7ff54b2a5640 1 -- 172.21.15.44:0/1241711927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff54406cbb0 con 0x7ff5440d08d0 2024-04-02T13:30:50.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.030+0000 7ff54b2a5640 1 -- 172.21.15.44:0/1241711927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff544068c00 con 0x7ff5440d08d0 2024-04-02T13:30:50.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.030+0000 7ff539ffb640 1 -- 172.21.15.44:0/1241711927 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff534018070 con 0x7ff5440d08d0 2024-04-02T13:30:50.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.031+0000 7ff539ffb640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff52806c8f0 0x7ff52806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.031+0000 7ff539ffb640 1 -- 172.21.15.44:0/1241711927 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x7ff53408bd40 con 0x7ff5440d08d0 2024-04-02T13:30:50.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.031+0000 7ff549041640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff52806c8f0 0x7ff52806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.031+0000 7ff549041640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff52806c8f0 0x7ff52806edb0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff53c000f60 tx=0x7ff53c007ca0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:50.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.035+0000 7ff539ffb640 1 -- 172.21.15.44:0/1241711927 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff534059f50 con 0x7ff5440d08d0 2024-04-02T13:30:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.262+0000 7ff54b2a5640 1 -- 172.21.15.44:0/1241711927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff54406f600 con 0x7ff5440d08d0 2024-04-02T13:30:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.263+0000 7ff539ffb640 1 -- 172.21.15.44:0/1241711927 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+4345 (secure 0 0 0) 0x7ff534059850 con 0x7ff5440d08d0 2024-04-02T13:30:50.267 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:50.267 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":89,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7696,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2715855087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2715855087},{"type":"v1","addr":"172.21.15.69:6836","nonce":2715855087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7699,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3850680553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3850680553},{"type":"v1","addr":"172.21.15.69:6837","nonce":3850680553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/1330712383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1330712383},{"type":"v1","addr":"172.21.15.44:6835","nonce":1330712383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":89,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:29:10.907920+0000","modified":"2024-04-02T13:30:49.437157+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-04-02T13:30:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 89 2024-04-02T13:30:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.265+0000 7ff54b2a5640 1 -- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff52806c8f0 msgr2=0x7ff52806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.265+0000 7ff54b2a5640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff52806c8f0 0x7ff52806edb0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff53c000f60 tx=0x7ff53c007ca0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.265+0000 7ff54b2a5640 1 -- 172.21.15.44:0/1241711927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5440d08d0 msgr2=0x7ff5440b5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.265+0000 7ff54b2a5640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5440d08d0 0x7ff5440b5980 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7ff53400c820 tx=0x7ff53400cc20 comp rx=0 tx=0).stop 2024-04-02T13:30:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.265+0000 7ff54b2a5640 1 -- 172.21.15.44:0/1241711927 shutdown_connections 2024-04-02T13:30:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.265+0000 7ff54b2a5640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff52806c8f0 0x7ff52806edb0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.265+0000 7ff54b2a5640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff54415c250 0x7ff544158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.265+0000 7ff54b2a5640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5440d08d0 0x7ff5440b5980 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.265+0000 7ff54b2a5640 1 --2- 172.21.15.44:0/1241711927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5440b3510 0x7ff5440b4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.265+0000 7ff54b2a5640 1 -- 172.21.15.44:0/1241711927 >> 172.21.15.44:0/1241711927 conn(0x7ff544073850 msgr2=0x7ff54415b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.265+0000 7ff54b2a5640 1 -- 172.21.15.44:0/1241711927 shutdown_connections 2024-04-02T13:30:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.265+0000 7ff54b2a5640 1 -- 172.21.15.44:0/1241711927 wait complete. 2024-04-02T13:30:50.280 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:30:50.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.377+0000 7f574cd9d640 1 Processor -- start 2024-04-02T13:30:50.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.377+0000 7f574cd9d640 1 -- start start 2024-04-02T13:30:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.378+0000 7f574cd9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f574815c2b0 0x7f57480d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.378+0000 7f574cd9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57480d0980 0x7f57480af060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.378+0000 7f574cd9d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57480cfda0 con 0x7f57480b0c70 2024-04-02T13:30:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.378+0000 7f574cd9d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f57480cee00 con 0x7f574815c2b0 2024-04-02T13:30:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.378+0000 7f574cd9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f57480cf0a0 con 0x7f57480d0980 2024-04-02T13:30:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.378+0000 7f57477fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f574815c2b0 0x7f57480d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.378+0000 7f5746ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f57480b0c70 0x7f57480d0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37804/0 (socket says 172.21.15.44:37804) 2024-04-02T13:30:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.378+0000 7f5746ffd640 1 -- 172.21.15.44:0/2296879069 learned_addr learned my addr 172.21.15.44:0/2296879069 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.379+0000 7f5747fff640 1 --2- 172.21.15.44:0/2296879069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57480d0980 0x7f57480af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.379+0000 7f57477fe640 1 -- 172.21.15.44:0/2296879069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57480d0980 msgr2=0x7f57480af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.379+0000 7f57477fe640 1 --2- 172.21.15.44:0/2296879069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57480d0980 0x7f57480af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.379+0000 7f57477fe640 1 -- 172.21.15.44:0/2296879069 >> v1:172.21.15.44:6789/0 conn(0x7f57480b0c70 legacy=0x7f57480d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.379+0000 7f57467fc640 1 -- 172.21.15.44:0/2296879069 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2536355831 0 0) 0x7f573c001110 con 0x7f57480b0c70 2024-04-02T13:30:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.379+0000 7f57477fe640 1 -- 172.21.15.44:0/2296879069 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57480cf380 con 0x7f574815c2b0 2024-04-02T13:30:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.379+0000 7f57477fe640 1 --2- 172.21.15.44:0/2296879069 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f574815c2b0 0x7f57480d27b0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f57380097a0 tx=0x7f573802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa50363e470ecf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:50.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.380+0000 7f57467fc640 1 -- 172.21.15.44:0/2296879069 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738038070 con 0x7f574815c2b0 2024-04-02T13:30:50.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.380+0000 7f57467fc640 1 -- 172.21.15.44:0/2296879069 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5738031700 con 0x7f574815c2b0 2024-04-02T13:30:50.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.380+0000 7f57467fc640 1 -- 172.21.15.44:0/2296879069 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738031ad0 con 0x7f574815c2b0 2024-04-02T13:30:50.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.380+0000 7f574cd9d640 1 -- 172.21.15.44:0/2296879069 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f574815c2b0 msgr2=0x7f57480d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.380+0000 7f574cd9d640 1 --2- 172.21.15.44:0/2296879069 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f574815c2b0 0x7f57480d27b0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f57380097a0 tx=0x7f573802b7a0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.380+0000 7f574cd9d640 1 -- 172.21.15.44:0/2296879069 shutdown_connections 2024-04-02T13:30:50.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.380+0000 7f574cd9d640 1 --2- 172.21.15.44:0/2296879069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57480d0980 0x7f57480af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.380+0000 7f574cd9d640 1 --2- 172.21.15.44:0/2296879069 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f574815c2b0 0x7f57480d27b0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.380+0000 7f574cd9d640 1 -- 172.21.15.44:0/2296879069 >> 172.21.15.44:0/2296879069 conn(0x7f57480d1c90 msgr2=0x7f57480d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:50.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.380+0000 7f574cd9d640 1 -- 172.21.15.44:0/2296879069 shutdown_connections 2024-04-02T13:30:50.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.380+0000 7f574cd9d640 1 -- 172.21.15.44:0/2296879069 wait complete. 2024-04-02T13:30:50.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.381+0000 7f574cd9d640 1 Processor -- start 2024-04-02T13:30:50.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.381+0000 7f574cd9d640 1 -- start start 2024-04-02T13:30:50.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.382+0000 7f574cd9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57480b0c70 0x7f57481531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.382+0000 7f574cd9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57480d0980 0x7f574815db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.382+0000 7f574cd9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f574815c2b0 0x7f5748159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.382+0000 7f574cd9d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f57480af930 con 0x7f574815c2b0 2024-04-02T13:30:50.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.382+0000 7f574cd9d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f57480af7b0 con 0x7f57480d0980 2024-04-02T13:30:50.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.382+0000 7f574cd9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f57480afab0 con 0x7f57480b0c70 2024-04-02T13:30:50.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.382+0000 7f5747fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f574815c2b0 0x7f5748159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.382+0000 7f57477fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57480b0c70 0x7f57481531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.382+0000 7f5747fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f574815c2b0 0x7f5748159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58866/0 (socket says 172.21.15.44:58866) 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.382+0000 7f5747fff640 1 -- 172.21.15.44:0/3570542680 learned_addr learned my addr 172.21.15.44:0/3570542680 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.382+0000 7f5746ffd640 1 --2- 172.21.15.44:0/3570542680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57480d0980 0x7f574815db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.383+0000 7f5747fff640 1 -- 172.21.15.44:0/3570542680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57480b0c70 msgr2=0x7f57481531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.383+0000 7f5747fff640 1 --2- 172.21.15.44:0/3570542680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57480b0c70 0x7f57481531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.383+0000 7f5747fff640 1 -- 172.21.15.44:0/3570542680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57480d0980 msgr2=0x7f574815db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.383+0000 7f5747fff640 1 --2- 172.21.15.44:0/3570542680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57480d0980 0x7f574815db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.383+0000 7f5747fff640 1 -- 172.21.15.44:0/3570542680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574815e600 con 0x7f574815c2b0 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.383+0000 7f5747fff640 1 --2- 172.21.15.44:0/3570542680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f574815c2b0 0x7f5748159b20 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f5740007e40 tx=0x7f574000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.383+0000 7f572ffff640 1 -- 172.21.15.44:0/3570542680 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740014620 con 0x7f574815c2b0 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.383+0000 7f572ffff640 1 -- 172.21.15.44:0/3570542680 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5740019070 con 0x7f574815c2b0 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.383+0000 7f574cd9d640 1 -- 172.21.15.44:0/3570542680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f574815e8e0 con 0x7f574815c2b0 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.383+0000 7f572ffff640 1 -- 172.21.15.44:0/3570542680 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740014a20 con 0x7f574815c2b0 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.383+0000 7f574cd9d640 1 -- 172.21.15.44:0/3570542680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f574815eea0 con 0x7f574815c2b0 2024-04-02T13:30:50.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.384+0000 7f574cd9d640 1 -- 172.21.15.44:0/3570542680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f574804f8d0 con 0x7f574815c2b0 2024-04-02T13:30:50.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.384+0000 7f572ffff640 1 -- 172.21.15.44:0/3570542680 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5740014bc0 con 0x7f574815c2b0 2024-04-02T13:30:50.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.385+0000 7f572ffff640 1 --2- 172.21.15.44:0/3570542680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f573406c8f0 0x7f573406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.385+0000 7f572ffff640 1 -- 172.21.15.44:0/3570542680 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x7f574008c5a0 con 0x7f574815c2b0 2024-04-02T13:30:50.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.385+0000 7f57477fe640 1 --2- 172.21.15.44:0/3570542680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f573406c8f0 0x7f573406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.385+0000 7f57477fe640 1 --2- 172.21.15.44:0/3570542680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f573406c8f0 0x7f573406edb0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5738035000 tx=0x7f5738008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.389+0000 7f572ffff640 1 -- 172.21.15.44:0/3570542680 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f574005a0f0 con 0x7f574815c2b0 2024-04-02T13:30:50.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.579+0000 7f574cd9d640 1 -- 172.21.15.44:0/3570542680 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f57480d0390 con 0x7f574815c2b0 2024-04-02T13:30:50.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.580+0000 7f572ffff640 1 -- 172.21.15.44:0/3570542680 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+17239 (secure 0 0 0) 0x7f574005e3f0 con 0x7f574815c2b0 2024-04-02T13:30:50.584 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:50.584 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":110,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:30:49.436981+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:29:09.192430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-02T13:29:09.860748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.d","mappings":[{"from":6,"to":7}]},{"pgid":"16.f","mappings":[{"from":5,"to":4}]},{"pgid":"16.1b","mappings":[{"from":1,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6834/1745009605":"2024-04-03T13:30:49.436959+0000","172.21.15.44:6833/1745009605":"2024-04-03T13:30:49.436959+0000","172.21.15.44:6834/3259420283":"2024-04-03T13:30:12.628183+0000","172.21.15.44:6833/3259420283":"2024-04-03T13:30:12.628183+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:30:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.582+0000 7f574cd9d640 1 -- 172.21.15.44:0/3570542680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f573406c8f0 msgr2=0x7f573406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.582+0000 7f574cd9d640 1 --2- 172.21.15.44:0/3570542680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f573406c8f0 0x7f573406edb0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5738035000 tx=0x7f5738008030 comp rx=0 tx=0).stop 2024-04-02T13:30:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.582+0000 7f574cd9d640 1 -- 172.21.15.44:0/3570542680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f574815c2b0 msgr2=0x7f5748159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.582+0000 7f574cd9d640 1 --2- 172.21.15.44:0/3570542680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f574815c2b0 0x7f5748159b20 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f5740007e40 tx=0x7f574000e4b0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.582+0000 7f574cd9d640 1 -- 172.21.15.44:0/3570542680 shutdown_connections 2024-04-02T13:30:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.582+0000 7f574cd9d640 1 --2- 172.21.15.44:0/3570542680 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f573406c8f0 0x7f573406edb0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.582+0000 7f574cd9d640 1 --2- 172.21.15.44:0/3570542680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f574815c2b0 0x7f5748159b20 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.582+0000 7f574cd9d640 1 --2- 172.21.15.44:0/3570542680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57480d0980 0x7f574815db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.582+0000 7f574cd9d640 1 --2- 172.21.15.44:0/3570542680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57480b0c70 0x7f57481531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.582+0000 7f574cd9d640 1 -- 172.21.15.44:0/3570542680 >> 172.21.15.44:0/3570542680 conn(0x7f57480d1c90 msgr2=0x7f5748070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:50.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.582+0000 7f574cd9d640 1 -- 172.21.15.44:0/3570542680 shutdown_connections 2024-04-02T13:30:50.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.582+0000 7f574cd9d640 1 -- 172.21.15.44:0/3570542680 wait complete. 2024-04-02T13:30:50.597 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:29:09.192430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:29:09.860748+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:30:50.597 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:30:50.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.697+0000 7f6f2f60d640 1 Processor -- start 2024-04-02T13:30:50.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.697+0000 7f6f2f60d640 1 -- start start 2024-04-02T13:30:50.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.698+0000 7f6f2f60d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6f2815e420 0x7f6f280ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.698+0000 7f6f2f60d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f280cfe10 0x7f6f280cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.698+0000 7f6f2f60d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f2806c220 con 0x7f6f280721f0 2024-04-02T13:30:50.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.698+0000 7f6f2f60d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f6f2815f7e0 con 0x7f6f2815e420 2024-04-02T13:30:50.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.698+0000 7f6f2f60d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6f2815ec60 con 0x7f6f280cfe10 2024-04-02T13:30:50.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.698+0000 7f6f2d3a9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6f2815e420 0x7f6f280ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.698+0000 7f6f2d3a9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6f2815e420 0x7f6f280ceca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58872/0 (socket says 172.21.15.44:58872) 2024-04-02T13:30:50.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.698+0000 7f6f2d3a9640 1 -- 172.21.15.44:0/767389902 learned_addr learned my addr 172.21.15.44:0/767389902 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:50.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.698+0000 7f6f2dbaa640 1 --2- 172.21.15.44:0/767389902 >> v2:172.21.15.69:3300/0 conn(0x7f6f280cfe10 0x7f6f280cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.698+0000 7f6f2d3a9640 1 -- 172.21.15.44:0/767389902 >> v2:172.21.15.69:3300/0 conn(0x7f6f280cfe10 msgr2=0x7f6f280cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:50.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.698+0000 7f6f2d3a9640 1 --2- 172.21.15.44:0/767389902 >> v2:172.21.15.69:3300/0 conn(0x7f6f280cfe10 0x7f6f280cdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.698+0000 7f6f2d3a9640 1 -- 172.21.15.44:0/767389902 >> v1:172.21.15.69:6789/0 conn(0x7f6f280721f0 legacy=0x7f6f280cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:50.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.698+0000 7f6f2d3a9640 1 -- 172.21.15.44:0/767389902 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2815ef40 con 0x7f6f2815e420 2024-04-02T13:30:50.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.699+0000 7f6f2d3a9640 1 --2- 172.21.15.44:0/767389902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f2815e420 0x7f6f280ceca0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f6f280d0bf0 tx=0x7f6f2402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29cab816d0cda9e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:50.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.699+0000 7f6f1ffff640 1 -- 172.21.15.44:0/767389902 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2402c960 con 0x7f6f2815e420 2024-04-02T13:30:50.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.699+0000 7f6f1ffff640 1 -- 172.21.15.44:0/767389902 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f2402cb00 con 0x7f6f2815e420 2024-04-02T13:30:50.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.699+0000 7f6f1ffff640 1 -- 172.21.15.44:0/767389902 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f240343f0 con 0x7f6f2815e420 2024-04-02T13:30:50.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.700+0000 7f6f2f60d640 1 -- 172.21.15.44:0/767389902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f2815e420 msgr2=0x7f6f280ceca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.700+0000 7f6f2f60d640 1 --2- 172.21.15.44:0/767389902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f2815e420 0x7f6f280ceca0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f6f280d0bf0 tx=0x7f6f2402bdc0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.700+0000 7f6f2f60d640 1 -- 172.21.15.44:0/767389902 shutdown_connections 2024-04-02T13:30:50.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.700+0000 7f6f2f60d640 1 --2- 172.21.15.44:0/767389902 >> v2:172.21.15.69:3300/0 conn(0x7f6f280cfe10 0x7f6f280cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.700+0000 7f6f2f60d640 1 --2- 172.21.15.44:0/767389902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f2815e420 0x7f6f280ceca0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.700+0000 7f6f2f60d640 1 -- 172.21.15.44:0/767389902 >> 172.21.15.44:0/767389902 conn(0x7f6f280b2a10 msgr2=0x7f6f280b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:50.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.700+0000 7f6f2f60d640 1 -- 172.21.15.44:0/767389902 shutdown_connections 2024-04-02T13:30:50.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.700+0000 7f6f2f60d640 1 -- 172.21.15.44:0/767389902 wait complete. 2024-04-02T13:30:50.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.701+0000 7f6f2f60d640 1 Processor -- start 2024-04-02T13:30:50.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.701+0000 7f6f2f60d640 1 -- start start 2024-04-02T13:30:50.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2f60d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f280721f0 0x7f6f2814a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2f60d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f280cfe10 0x7f6f281f5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2f60d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f2815e420 0x7f6f28159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2f60d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6f280cf1e0 con 0x7f6f280cfe10 2024-04-02T13:30:50.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2f60d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2815f500 con 0x7f6f280721f0 2024-04-02T13:30:50.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2f60d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6f280cf360 con 0x7f6f2815e420 2024-04-02T13:30:50.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2cba8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f280cfe10 0x7f6f281f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2cba8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f280cfe10 0x7f6f281f5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58884/0 (socket says 172.21.15.44:58884) 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2cba8640 1 -- 172.21.15.44:0/2946129312 learned_addr learned my addr 172.21.15.44:0/2946129312 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2dbaa640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f2815e420 0x7f6f28159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2d3a9640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f280721f0 0x7f6f2814a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2cba8640 1 -- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f2815e420 msgr2=0x7f6f28159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2cba8640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f2815e420 0x7f6f28159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2cba8640 1 -- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f280721f0 msgr2=0x7f6f2814a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2cba8640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f280721f0 0x7f6f2814a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2cba8640 1 -- 172.21.15.44:0/2946129312 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f281f5d70 con 0x7f6f280cfe10 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f2cba8640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f280cfe10 0x7f6f281f5690 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c460 tx=0x7f6f1800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f1dffb640 1 -- 172.21.15.44:0/2946129312 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18016020 con 0x7f6f280cfe10 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.702+0000 7f6f1dffb640 1 -- 172.21.15.44:0/2946129312 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f18012870 con 0x7f6f280cfe10 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.703+0000 7f6f2f60d640 1 -- 172.21.15.44:0/2946129312 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f281f6050 con 0x7f6f280cfe10 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.703+0000 7f6f1dffb640 1 -- 172.21.15.44:0/2946129312 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18012c60 con 0x7f6f280cfe10 2024-04-02T13:30:50.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.703+0000 7f6f2f60d640 1 -- 172.21.15.44:0/2946129312 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f281f6610 con 0x7f6f280cfe10 2024-04-02T13:30:50.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.704+0000 7f6f2f60d640 1 -- 172.21.15.44:0/2946129312 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f280684f0 con 0x7f6f280cfe10 2024-04-02T13:30:50.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.704+0000 7f6f1dffb640 1 -- 172.21.15.44:0/2946129312 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6f18019020 con 0x7f6f280cfe10 2024-04-02T13:30:50.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.705+0000 7f6f1dffb640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f0c06c8f0 0x7f6f0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:50.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.705+0000 7f6f2d3a9640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f0c06c8f0 0x7f6f0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:50.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.705+0000 7f6f2d3a9640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f0c06c8f0 0x7f6f0c06edb0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f6f24035000 tx=0x7f6f24030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:50.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.706+0000 7f6f1dffb640 1 -- 172.21.15.44:0/2946129312 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x7f6f18012e00 con 0x7f6f280cfe10 2024-04-02T13:30:50.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.710+0000 7f6f1dffb640 1 -- 172.21.15.44:0/2946129312 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6f18020030 con 0x7f6f280cfe10 2024-04-02T13:30:50.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.895+0000 7f6f2f60d640 1 -- 172.21.15.44:0/2946129312 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6f280b3770 con 0x7f6f280cfe10 2024-04-02T13:30:50.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.895+0000 7f6f1dffb640 1 -- 172.21.15.44:0/2946129312 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) ==== 93+0+10 (secure 0 0 0) 0x7f6f18059510 con 0x7f6f280cfe10 2024-04-02T13:30:50.900 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:30:50.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.897+0000 7f6f2f60d640 1 -- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f0c06c8f0 msgr2=0x7f6f0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.897+0000 7f6f2f60d640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f0c06c8f0 0x7f6f0c06edb0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f6f24035000 tx=0x7f6f24030000 comp rx=0 tx=0).stop 2024-04-02T13:30:50.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.897+0000 7f6f2f60d640 1 -- 172.21.15.44:0/2946129312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f280cfe10 msgr2=0x7f6f281f5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:50.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.897+0000 7f6f2f60d640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f280cfe10 0x7f6f281f5690 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c460 tx=0x7f6f1800c920 comp rx=0 tx=0).stop 2024-04-02T13:30:50.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.897+0000 7f6f2f60d640 1 -- 172.21.15.44:0/2946129312 shutdown_connections 2024-04-02T13:30:50.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.897+0000 7f6f2f60d640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f0c06c8f0 0x7f6f0c06edb0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.897+0000 7f6f2f60d640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f2815e420 0x7f6f28159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.897+0000 7f6f2f60d640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f280cfe10 0x7f6f281f5690 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.897+0000 7f6f2f60d640 1 --2- 172.21.15.44:0/2946129312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f280721f0 0x7f6f2814a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:50.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.897+0000 7f6f2f60d640 1 -- 172.21.15.44:0/2946129312 >> 172.21.15.44:0/2946129312 conn(0x7f6f280b2a10 msgr2=0x7f6f2806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:50.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.898+0000 7f6f2f60d640 1 -- 172.21.15.44:0/2946129312 shutdown_connections 2024-04-02T13:30:50.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:50.898+0000 7f6f2f60d640 1 -- 172.21.15.44:0/2946129312 wait complete. 2024-04-02T13:30:50.912 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:30:51.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.014+0000 7f750fe4a640 1 Processor -- start 2024-04-02T13:30:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.014+0000 7f750fe4a640 1 -- start start 2024-04-02T13:30:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.015+0000 7f750fe4a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f75080cec60 0x7f75080d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.015+0000 7f750fe4a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750815e5b0 0x7f75080d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.015+0000 7f750fe4a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75080cf1f0 con 0x7f750815e1d0 2024-04-02T13:30:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.015+0000 7f750fe4a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f75080cfe00 con 0x7f75080cec60 2024-04-02T13:30:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.015+0000 7f750fe4a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f75080cf4d0 con 0x7f750815e5b0 2024-04-02T13:30:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.015+0000 7f750d3e5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f75080cec60 0x7f75080d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.015+0000 7f750dbe6640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f750815e1d0 0x7f75080d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37808/0 (socket says 172.21.15.44:37808) 2024-04-02T13:30:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.015+0000 7f750dbe6640 1 -- 172.21.15.44:0/1795599932 learned_addr learned my addr 172.21.15.44:0/1795599932 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.015+0000 7f750d3e5640 1 -- 172.21.15.44:0/1795599932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750815e5b0 msgr2=0x7f75080d2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:51.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.015+0000 7f750e3e7640 1 --2- 172.21.15.44:0/1795599932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750815e5b0 0x7f75080d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.015+0000 7f750d3e5640 1 --2- 172.21.15.44:0/1795599932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750815e5b0 0x7f75080d2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.015+0000 7f750cbe4640 1 -- 172.21.15.44:0/1795599932 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1919286515 0 0) 0x7f75080cf1f0 con 0x7f750815e1d0 2024-04-02T13:30:51.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.015+0000 7f750d3e5640 1 -- 172.21.15.44:0/1795599932 >> v1:172.21.15.44:6789/0 conn(0x7f750815e1d0 legacy=0x7f75080d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.016+0000 7f750d3e5640 1 -- 172.21.15.44:0/1795599932 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75080cf7b0 con 0x7f75080cec60 2024-04-02T13:30:51.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.016+0000 7f750e3e7640 1 --2- 172.21.15.44:0/1795599932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750815e5b0 0x7f75080d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:51.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.016+0000 7f750d3e5640 1 --2- 172.21.15.44:0/1795599932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75080cec60 0x7f75080d26c0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0097a0 tx=0x7f74fc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120db6203c299b11 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:51.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.016+0000 7f750cbe4640 1 -- 172.21.15.44:0/1795599932 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc0315a0 con 0x7f75080cec60 2024-04-02T13:30:51.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.016+0000 7f750cbe4640 1 -- 172.21.15.44:0/1795599932 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74fc037070 con 0x7f75080cec60 2024-04-02T13:30:51.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.016+0000 7f750cbe4640 1 -- 172.21.15.44:0/1795599932 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc031960 con 0x7f75080cec60 2024-04-02T13:30:51.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.017+0000 7f750fe4a640 1 -- 172.21.15.44:0/1795599932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75080cec60 msgr2=0x7f75080d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.017+0000 7f750fe4a640 1 --2- 172.21.15.44:0/1795599932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75080cec60 0x7f75080d26c0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0097a0 tx=0x7f74fc02b840 comp rx=0 tx=0).stop 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.017+0000 7f750fe4a640 1 -- 172.21.15.44:0/1795599932 shutdown_connections 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.017+0000 7f750fe4a640 1 --2- 172.21.15.44:0/1795599932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750815e5b0 0x7f75080d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.017+0000 7f750fe4a640 1 --2- 172.21.15.44:0/1795599932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75080cec60 0x7f75080d26c0 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.017+0000 7f750fe4a640 1 -- 172.21.15.44:0/1795599932 >> 172.21.15.44:0/1795599932 conn(0x7f750809d3b0 msgr2=0x7f75080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.017+0000 7f750fe4a640 1 -- 172.21.15.44:0/1795599932 shutdown_connections 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.017+0000 7f750fe4a640 1 -- 172.21.15.44:0/1795599932 wait complete. 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.018+0000 7f750fe4a640 1 Processor -- start 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.018+0000 7f750fe4a640 1 -- start start 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750fe4a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75080cec60 0x7f75080b8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750fe4a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750815e1d0 0x7f75081614b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750fe4a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f750815e5b0 0x7f75080cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750fe4a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f75080b4e60 con 0x7f750815e5b0 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750fe4a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f75080b4ce0 con 0x7f75080cec60 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750fe4a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f75080b4fe0 con 0x7f750815e1d0 2024-04-02T13:30:51.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750e3e7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f750815e5b0 0x7f75080cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750e3e7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f750815e5b0 0x7f75080cd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58904/0 (socket says 172.21.15.44:58904) 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750e3e7640 1 -- 172.21.15.44:0/3427784489 learned_addr learned my addr 172.21.15.44:0/3427784489 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750dbe6640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75080cec60 0x7f75080b8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750d3e5640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750815e1d0 0x7f75081614b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750e3e7640 1 -- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750815e1d0 msgr2=0x7f75081614b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750e3e7640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750815e1d0 0x7f75081614b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750e3e7640 1 -- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75080cec60 msgr2=0x7f75080b8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750e3e7640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75080cec60 0x7f75080b8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.019+0000 7f750e3e7640 1 -- 172.21.15.44:0/3427784489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7508160450 con 0x7f750815e5b0 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.020+0000 7f750e3e7640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f750815e5b0 0x7f75080cd8a0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f74f800cd50 tx=0x7f74f800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.020+0000 7f75067fc640 1 -- 172.21.15.44:0/3427784489 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f800f680 con 0x7f750815e5b0 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.020+0000 7f75067fc640 1 -- 172.21.15.44:0/3427784489 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74f800f840 con 0x7f750815e5b0 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.020+0000 7f750fe4a640 1 -- 172.21.15.44:0/3427784489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75081606f0 con 0x7f750815e5b0 2024-04-02T13:30:51.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.020+0000 7f75067fc640 1 -- 172.21.15.44:0/3427784489 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f800fc30 con 0x7f750815e5b0 2024-04-02T13:30:51.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.020+0000 7f750fe4a640 1 -- 172.21.15.44:0/3427784489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7508160f90 con 0x7f750815e5b0 2024-04-02T13:30:51.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.021+0000 7f750fe4a640 1 -- 172.21.15.44:0/3427784489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7508160ae0 con 0x7f750815e5b0 2024-04-02T13:30:51.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.021+0000 7f75067fc640 1 -- 172.21.15.44:0/3427784489 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f74f80213f0 con 0x7f750815e5b0 2024-04-02T13:30:51.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.021+0000 7f75067fc640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f74f406c8f0 0x7f74f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.022+0000 7f75067fc640 1 -- 172.21.15.44:0/3427784489 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x7f74f808c9c0 con 0x7f750815e5b0 2024-04-02T13:30:51.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.022+0000 7f750dbe6640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f74f406c8f0 0x7f74f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.022+0000 7f750dbe6640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f74f406c8f0 0x7f74f406edb0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f7500000f60 tx=0x7f7500007d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:51.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.026+0000 7f75067fc640 1 -- 172.21.15.44:0/3427784489 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f74f808ee20 con 0x7f750815e5b0 2024-04-02T13:30:51.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.209+0000 7f750fe4a640 1 -- 172.21.15.44:0/3427784489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f75080684f0 con 0x7f750815e5b0 2024-04-02T13:30:51.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.210+0000 7f75067fc640 1 -- 172.21.15.44:0/3427784489 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) ==== 104+0+11 (secure 0 0 0) 0x7f74f8014030 con 0x7f750815e5b0 2024-04-02T13:30:51.214 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:30:51.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.211+0000 7f750fe4a640 1 -- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f74f406c8f0 msgr2=0x7f74f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.211+0000 7f750fe4a640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f74f406c8f0 0x7f74f406edb0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f7500000f60 tx=0x7f7500007d50 comp rx=0 tx=0).stop 2024-04-02T13:30:51.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.211+0000 7f750fe4a640 1 -- 172.21.15.44:0/3427784489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f750815e5b0 msgr2=0x7f75080cd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.211+0000 7f750fe4a640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f750815e5b0 0x7f75080cd8a0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f74f800cd50 tx=0x7f74f800a650 comp rx=0 tx=0).stop 2024-04-02T13:30:51.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.211+0000 7f750fe4a640 1 -- 172.21.15.44:0/3427784489 shutdown_connections 2024-04-02T13:30:51.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.211+0000 7f750fe4a640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f74f406c8f0 0x7f74f406edb0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.211+0000 7f750fe4a640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f750815e5b0 0x7f75080cd8a0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.211+0000 7f750fe4a640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750815e1d0 0x7f75081614b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.211+0000 7f750fe4a640 1 --2- 172.21.15.44:0/3427784489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75080cec60 0x7f75080b8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.211+0000 7f750fe4a640 1 -- 172.21.15.44:0/3427784489 >> 172.21.15.44:0/3427784489 conn(0x7f750809d3b0 msgr2=0x7f750806c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:51.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.212+0000 7f750fe4a640 1 -- 172.21.15.44:0/3427784489 shutdown_connections 2024-04-02T13:30:51.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.212+0000 7f750fe4a640 1 -- 172.21.15.44:0/3427784489 wait complete. 2024-04-02T13:30:51.226 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:30:51.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.328+0000 7f3794255640 1 Processor -- start 2024-04-02T13:30:51.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.328+0000 7f3794255640 1 -- start start 2024-04-02T13:30:51.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.329+0000 7f3794255640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c15c220 0x7f378c0b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.329+0000 7f3794255640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f378c0d03d0 0x7f378c0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.329+0000 7f3794255640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f378c0cca50 con 0x7f378c0b1440 2024-04-02T13:30:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.329+0000 7f3794255640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f378c0d0d70 con 0x7f378c0d03d0 2024-04-02T13:30:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.329+0000 7f3794255640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f378c0ccd60 con 0x7f378c15c220 2024-04-02T13:30:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.329+0000 7f3791ff1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c15c220 0x7f378c0b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.329+0000 7f3791ff1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c15c220 0x7f378c0b1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:38692/0 (socket says 172.21.15.44:38692) 2024-04-02T13:30:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.329+0000 7f37927f2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f378c0d03d0 0x7f378c0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.329+0000 7f3791ff1640 1 -- 172.21.15.44:0/3134580050 learned_addr learned my addr 172.21.15.44:0/3134580050 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.329+0000 7f3791ff1640 1 -- 172.21.15.44:0/3134580050 >> v2:172.21.15.69:3300/0 conn(0x7f378c0d03d0 msgr2=0x7f378c0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.329+0000 7f3791ff1640 1 --2- 172.21.15.44:0/3134580050 >> v2:172.21.15.69:3300/0 conn(0x7f378c0d03d0 0x7f378c0d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.329+0000 7f3791ff1640 1 -- 172.21.15.44:0/3134580050 >> v1:172.21.15.69:6789/0 conn(0x7f378c0b1440 legacy=0x7f378c0ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.329+0000 7f3791ff1640 1 -- 172.21.15.44:0/3134580050 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378c0cef40 con 0x7f378c15c220 2024-04-02T13:30:51.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.330+0000 7f3791ff1640 1 --2- 172.21.15.44:0/3134580050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c15c220 0x7f378c0b1db0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3788009870 tx=0x7f378802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6b7611827b46332 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:51.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.330+0000 7f3790fef640 1 -- 172.21.15.44:0/3134580050 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788034070 con 0x7f378c15c220 2024-04-02T13:30:51.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.330+0000 7f3790fef640 1 -- 172.21.15.44:0/3134580050 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3788030840 con 0x7f378c15c220 2024-04-02T13:30:51.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.330+0000 7f3790fef640 1 -- 172.21.15.44:0/3134580050 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788030c10 con 0x7f378c15c220 2024-04-02T13:30:51.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.331+0000 7f3794255640 1 -- 172.21.15.44:0/3134580050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c15c220 msgr2=0x7f378c0b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.331+0000 7f3794255640 1 --2- 172.21.15.44:0/3134580050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c15c220 0x7f378c0b1db0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3788009870 tx=0x7f378802b980 comp rx=0 tx=0).stop 2024-04-02T13:30:51.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.331+0000 7f3794255640 1 -- 172.21.15.44:0/3134580050 shutdown_connections 2024-04-02T13:30:51.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.331+0000 7f3794255640 1 --2- 172.21.15.44:0/3134580050 >> v2:172.21.15.69:3300/0 conn(0x7f378c0d03d0 0x7f378c0d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.331+0000 7f3794255640 1 --2- 172.21.15.44:0/3134580050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c15c220 0x7f378c0b1db0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.331+0000 7f3794255640 1 -- 172.21.15.44:0/3134580050 >> 172.21.15.44:0/3134580050 conn(0x7f378c09aa20 msgr2=0x7f378c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:51.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.331+0000 7f3794255640 1 -- 172.21.15.44:0/3134580050 shutdown_connections 2024-04-02T13:30:51.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.331+0000 7f3794255640 1 -- 172.21.15.44:0/3134580050 wait complete. 2024-04-02T13:30:51.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.332+0000 7f3794255640 1 Processor -- start 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.332+0000 7f3794255640 1 -- start start 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.333+0000 7f3794255640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c0b1440 0x7f378c1f60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.333+0000 7f3794255640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378c0d03d0 0x7f378c15d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.333+0000 7f3794255640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378c15c220 0x7f378c0b0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.333+0000 7f3794255640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f378c15c600 con 0x7f378c0d03d0 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.333+0000 7f3794255640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f378c0cf500 con 0x7f378c15c220 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.333+0000 7f3794255640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f378c0cd1f0 con 0x7f378c0b1440 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.333+0000 7f3791ff1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c0b1440 0x7f378c1f60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.333+0000 7f37917f0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378c0d03d0 0x7f378c15d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.333+0000 7f3791ff1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c0b1440 0x7f378c1f60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:38702/0 (socket says 172.21.15.44:38702) 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.333+0000 7f3791ff1640 1 -- 172.21.15.44:0/723994283 learned_addr learned my addr 172.21.15.44:0/723994283 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.333+0000 7f37927f2640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378c15c220 0x7f378c0b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.334+0000 7f37917f0640 1 -- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c0b1440 msgr2=0x7f378c1f60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.334+0000 7f37917f0640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c0b1440 0x7f378c1f60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.334+0000 7f37917f0640 1 -- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378c15c220 msgr2=0x7f378c0b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.334+0000 7f37917f0640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378c15c220 0x7f378c0b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.334+0000 7f37917f0640 1 -- 172.21.15.44:0/723994283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378c0c94f0 con 0x7f378c0d03d0 2024-04-02T13:30:51.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.334+0000 7f3791ff1640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c0b1440 0x7f378c1f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:30:51.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.334+0000 7f37917f0640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378c0d03d0 0x7f378c15d020 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f377c00c340 tx=0x7f377c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:51.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.334+0000 7f37827fc640 1 -- 172.21.15.44:0/723994283 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c0052d0 con 0x7f378c0d03d0 2024-04-02T13:30:51.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.334+0000 7f37827fc640 1 -- 172.21.15.44:0/723994283 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f377c005490 con 0x7f378c0d03d0 2024-04-02T13:30:51.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.334+0000 7f3794255640 1 -- 172.21.15.44:0/723994283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f378c0c97d0 con 0x7f378c0d03d0 2024-04-02T13:30:51.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.334+0000 7f37827fc640 1 -- 172.21.15.44:0/723994283 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c007260 con 0x7f378c0d03d0 2024-04-02T13:30:51.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.334+0000 7f3794255640 1 -- 172.21.15.44:0/723994283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f378c0c9d90 con 0x7f378c0d03d0 2024-04-02T13:30:51.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.335+0000 7f3794255640 1 -- 172.21.15.44:0/723994283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f378c0684f0 con 0x7f378c0d03d0 2024-04-02T13:30:51.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.335+0000 7f37827fc640 1 -- 172.21.15.44:0/723994283 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f377c007400 con 0x7f378c0d03d0 2024-04-02T13:30:51.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.336+0000 7f37827fc640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f377006c8f0 0x7f377006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.336+0000 7f37827fc640 1 -- 172.21.15.44:0/723994283 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x7f377c08b120 con 0x7f378c0d03d0 2024-04-02T13:30:51.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.336+0000 7f3791ff1640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f377006c8f0 0x7f377006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.336+0000 7f3791ff1640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f377006c8f0 0x7f377006edb0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3788009870 tx=0x7f378803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:51.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.340+0000 7f37827fc640 1 -- 172.21.15.44:0/723994283 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f378c0cca50 con 0x7f378c0d03d0 2024-04-02T13:30:51.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.524+0000 7f3794255640 1 -- 172.21.15.44:0/723994283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f378c0c87c0 con 0x7f378c0d03d0 2024-04-02T13:30:51.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.525+0000 7f37827fc640 1 -- 172.21.15.44:0/723994283 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) ==== 100+0+11 (secure 0 0 0) 0x7f377c058d00 con 0x7f378c0d03d0 2024-04-02T13:30:51.529 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:30:51.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.526+0000 7f3794255640 1 -- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f377006c8f0 msgr2=0x7f377006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.526+0000 7f3794255640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f377006c8f0 0x7f377006edb0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3788009870 tx=0x7f378803b040 comp rx=0 tx=0).stop 2024-04-02T13:30:51.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.527+0000 7f3794255640 1 -- 172.21.15.44:0/723994283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378c0d03d0 msgr2=0x7f378c15d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.527+0000 7f3794255640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378c0d03d0 0x7f378c15d020 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f377c00c340 tx=0x7f377c00c800 comp rx=0 tx=0).stop 2024-04-02T13:30:51.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.527+0000 7f3794255640 1 -- 172.21.15.44:0/723994283 shutdown_connections 2024-04-02T13:30:51.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.527+0000 7f3794255640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f377006c8f0 0x7f377006edb0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.527+0000 7f3794255640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378c15c220 0x7f378c0b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.527+0000 7f3794255640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378c0d03d0 0x7f378c15d020 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.527+0000 7f3794255640 1 --2- 172.21.15.44:0/723994283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378c0b1440 0x7f378c1f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.527+0000 7f3794255640 1 -- 172.21.15.44:0/723994283 >> 172.21.15.44:0/723994283 conn(0x7f378c09aa20 msgr2=0x7f378c06fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:51.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.527+0000 7f3794255640 1 -- 172.21.15.44:0/723994283 shutdown_connections 2024-04-02T13:30:51.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.527+0000 7f3794255640 1 -- 172.21.15.44:0/723994283 wait complete. 2024-04-02T13:30:51.541 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:30:51.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.650+0000 7f7a6f567640 1 Processor -- start 2024-04-02T13:30:51.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.650+0000 7f7a6f567640 1 -- start start 2024-04-02T13:30:51.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6f567640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a681617e0 0x7f7a680ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6f567640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7a680cffd0 0x7f7a681629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6f567640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a680cef30 con 0x7f7a680ccc90 2024-04-02T13:30:51.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6f567640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7a68162f00 con 0x7f7a680cffd0 2024-04-02T13:30:51.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6f567640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7a680aea90 con 0x7f7a681617e0 2024-04-02T13:30:51.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6cb02640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7a680cffd0 0x7f7a681629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6d303640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a681617e0 0x7f7a680ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6db04640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7a680ccc90 0x7f7a680cd0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:53680/0 (socket says 172.21.15.44:53680) 2024-04-02T13:30:51.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6db04640 1 -- 172.21.15.44:0/4044794468 learned_addr learned my addr 172.21.15.44:0/4044794468 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:51.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6cb02640 1 -- 172.21.15.44:0/4044794468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a681617e0 msgr2=0x7f7a680ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:51.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6cb02640 1 --2- 172.21.15.44:0/4044794468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a681617e0 0x7f7a680ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6cb02640 1 -- 172.21.15.44:0/4044794468 >> v1:172.21.15.69:6789/0 conn(0x7f7a680ccc90 legacy=0x7f7a680cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6cb02640 1 -- 172.21.15.44:0/4044794468 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a680cf210 con 0x7f7a680cffd0 2024-04-02T13:30:51.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.651+0000 7f7a6d303640 1 --2- 172.21.15.44:0/4044794468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a681617e0 0x7f7a680ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:30:51.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.652+0000 7f7a6cb02640 1 --2- 172.21.15.44:0/4044794468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680cffd0 0x7f7a681629c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7a580097a0 tx=0x7f7a5802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea88ff1766268601 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:51.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.652+0000 7f7a5ffff640 1 -- 172.21.15.44:0/4044794468 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a580315a0 con 0x7f7a680cffd0 2024-04-02T13:30:51.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.652+0000 7f7a5ffff640 1 -- 172.21.15.44:0/4044794468 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a58037070 con 0x7f7a680cffd0 2024-04-02T13:30:51.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.652+0000 7f7a5ffff640 1 -- 172.21.15.44:0/4044794468 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a58031960 con 0x7f7a680cffd0 2024-04-02T13:30:51.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.653+0000 7f7a6f567640 1 -- 172.21.15.44:0/4044794468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680cffd0 msgr2=0x7f7a681629c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.653+0000 7f7a6f567640 1 --2- 172.21.15.44:0/4044794468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680cffd0 0x7f7a681629c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7a580097a0 tx=0x7f7a5802b840 comp rx=0 tx=0).stop 2024-04-02T13:30:51.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.653+0000 7f7a6f567640 1 -- 172.21.15.44:0/4044794468 shutdown_connections 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.653+0000 7f7a6f567640 1 --2- 172.21.15.44:0/4044794468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680cffd0 0x7f7a681629c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.653+0000 7f7a6f567640 1 --2- 172.21.15.44:0/4044794468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a681617e0 0x7f7a680ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.653+0000 7f7a6f567640 1 -- 172.21.15.44:0/4044794468 >> 172.21.15.44:0/4044794468 conn(0x7f7a680b20e0 msgr2=0x7f7a680b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.653+0000 7f7a6f567640 1 -- 172.21.15.44:0/4044794468 shutdown_connections 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.653+0000 7f7a6f567640 1 -- 172.21.15.44:0/4044794468 wait complete. 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.654+0000 7f7a6f567640 1 Processor -- start 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.654+0000 7f7a6f567640 1 -- start start 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6f567640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a680ccc90 0x7f7a680b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6f567640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680cffd0 0x7f7a680b5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6f567640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a681617e0 0x7f7a6814b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6f567640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7a680cfab0 con 0x7f7a681617e0 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6f567640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a680cf7d0 con 0x7f7a680cffd0 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6f567640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7a68161bc0 con 0x7f7a680ccc90 2024-04-02T13:30:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6db04640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a681617e0 0x7f7a6814b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6db04640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a681617e0 0x7f7a6814b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58922/0 (socket says 172.21.15.44:58922) 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6db04640 1 -- 172.21.15.44:0/2016513032 learned_addr learned my addr 172.21.15.44:0/2016513032 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6cb02640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680cffd0 0x7f7a680b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6d303640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a680ccc90 0x7f7a680b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6db04640 1 -- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a680ccc90 msgr2=0x7f7a680b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6db04640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a680ccc90 0x7f7a680b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6db04640 1 -- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680cffd0 msgr2=0x7f7a680b5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6db04640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680cffd0 0x7f7a680b5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6db04640 1 -- 172.21.15.44:0/2016513032 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a6814ba30 con 0x7f7a681617e0 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6cb02640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680cffd0 0x7f7a680b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.655+0000 7f7a6db04640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a681617e0 0x7f7a6814b4f0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000cbc0 tx=0x7f7a6000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.656+0000 7f7a5dffb640 1 -- 172.21.15.44:0/2016513032 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a60017070 con 0x7f7a681617e0 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.656+0000 7f7a5dffb640 1 -- 172.21.15.44:0/2016513032 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a6000ad40 con 0x7f7a681617e0 2024-04-02T13:30:51.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.656+0000 7f7a6f567640 1 -- 172.21.15.44:0/2016513032 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a6814bdb0 con 0x7f7a681617e0 2024-04-02T13:30:51.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.656+0000 7f7a5dffb640 1 -- 172.21.15.44:0/2016513032 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a60012600 con 0x7f7a681617e0 2024-04-02T13:30:51.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.656+0000 7f7a6f567640 1 -- 172.21.15.44:0/2016513032 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a6814c370 con 0x7f7a681617e0 2024-04-02T13:30:51.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.657+0000 7f7a6f567640 1 -- 172.21.15.44:0/2016513032 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a680684f0 con 0x7f7a681617e0 2024-04-02T13:30:51.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.657+0000 7f7a5dffb640 1 -- 172.21.15.44:0/2016513032 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7a6001a070 con 0x7f7a681617e0 2024-04-02T13:30:51.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.657+0000 7f7a5dffb640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7a4c06c8f0 0x7f7a4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:51.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.658+0000 7f7a5dffb640 1 -- 172.21.15.44:0/2016513032 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x7f7a6008ba70 con 0x7f7a681617e0 2024-04-02T13:30:51.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.658+0000 7f7a6d303640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7a4c06c8f0 0x7f7a4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:51.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.658+0000 7f7a6d303640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7a4c06c8f0 0x7f7a4c06edb0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7a680d1310 tx=0x7f7a64008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:51.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.662+0000 7f7a5dffb640 1 -- 172.21.15.44:0/2016513032 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7a6814ba30 con 0x7f7a681617e0 2024-04-02T13:30:51.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.883+0000 7f7a6f567640 1 -- 172.21.15.44:0/2016513032 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a680ce190 con 0x7f7a681617e0 2024-04-02T13:30:51.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.884+0000 7f7a5dffb640 1 -- 172.21.15.44:0/2016513032 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+5170 (secure 0 0 0) 0x7f7a600595c0 con 0x7f7a681617e0 2024-04-02T13:30:51.888 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:51.888 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7696,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2715855087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2715855087},{"type":"v1","addr":"172.21.15.69:6836","nonce":2715855087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7699,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3850680553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3850680553},{"type":"v1","addr":"172.21.15.69:6837","nonce":3850680553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/1330712383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1330712383},{"type":"v1","addr":"172.21.15.44:6835","nonce":1330712383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":9348,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/3518070032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":3518070032},{"type":"v1","addr":"172.21.15.44:6834","nonce":3518070032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:29:10.907920+0000","modified":"2024-04-02T13:30:49.437157+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-04-02T13:30:51.888 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 90 2024-04-02T13:30:51.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.886+0000 7f7a6f567640 1 -- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7a4c06c8f0 msgr2=0x7f7a4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.886+0000 7f7a6f567640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7a4c06c8f0 0x7f7a4c06edb0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7a680d1310 tx=0x7f7a64008040 comp rx=0 tx=0).stop 2024-04-02T13:30:51.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.886+0000 7f7a6f567640 1 -- 172.21.15.44:0/2016513032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a681617e0 msgr2=0x7f7a6814b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:51.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.886+0000 7f7a6f567640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a681617e0 0x7f7a6814b4f0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000cbc0 tx=0x7f7a6000a430 comp rx=0 tx=0).stop 2024-04-02T13:30:51.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.886+0000 7f7a6f567640 1 -- 172.21.15.44:0/2016513032 shutdown_connections 2024-04-02T13:30:51.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.886+0000 7f7a6f567640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7a4c06c8f0 0x7f7a4c06edb0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.886+0000 7f7a6f567640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a681617e0 0x7f7a6814b4f0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.886+0000 7f7a6f567640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a680cffd0 0x7f7a680b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.886+0000 7f7a6f567640 1 --2- 172.21.15.44:0/2016513032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a680ccc90 0x7f7a680b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:51.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.886+0000 7f7a6f567640 1 -- 172.21.15.44:0/2016513032 >> 172.21.15.44:0/2016513032 conn(0x7f7a680b20e0 msgr2=0x7f7a680b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:51.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.886+0000 7f7a6f567640 1 -- 172.21.15.44:0/2016513032 shutdown_connections 2024-04-02T13:30:51.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.886+0000 7f7a6f567640 1 -- 172.21.15.44:0/2016513032 wait complete. 2024-04-02T13:30:51.901 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:30:52.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.999+0000 7f500bc60640 1 Processor -- start 2024-04-02T13:30:52.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:51.999+0000 7f500bc60640 1 -- start start 2024-04-02T13:30:52.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f500bc60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f500415c160 0x7f500406b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f500bc60640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5004160600 0x7f50040d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f500bc60640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f500406bab0 con 0x7f50040d06c0 2024-04-02T13:30:52.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f500bc60640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f50040ce6d0 con 0x7f5004160600 2024-04-02T13:30:52.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f500bc60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f500406bd90 con 0x7f500415c160 2024-04-02T13:30:52.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f50091fb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5004160600 0x7f50040d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f50091fb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5004160600 0x7f50040d0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58932/0 (socket says 172.21.15.44:58932) 2024-04-02T13:30:52.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f50091fb640 1 -- 172.21.15.44:0/171437192 learned_addr learned my addr 172.21.15.44:0/171437192 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f50091fb640 1 -- 172.21.15.44:0/171437192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f500415c160 msgr2=0x7f500406b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f50099fc640 1 --2- 172.21.15.44:0/171437192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f500415c160 0x7f500406b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f50091fb640 1 --2- 172.21.15.44:0/171437192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f500415c160 0x7f500406b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f50091fb640 1 -- 172.21.15.44:0/171437192 >> v1:172.21.15.69:6789/0 conn(0x7f50040d06c0 legacy=0x7f50040ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f50091fb640 1 -- 172.21.15.44:0/171437192 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50040ccec0 con 0x7f5004160600 2024-04-02T13:30:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.000+0000 7f50099fc640 1 --2- 172.21.15.44:0/171437192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f500415c160 0x7f500406b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.001+0000 7f50091fb640 1 --2- 172.21.15.44:0/171437192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5004160600 0x7f50040d0180 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009870 tx=0x7f4ff802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a54953fe509e05 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:52.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.001+0000 7f50089fa640 1 -- 172.21.15.44:0/171437192 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff80317c0 con 0x7f5004160600 2024-04-02T13:30:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.001+0000 7f50089fa640 1 -- 172.21.15.44:0/171437192 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ff8037070 con 0x7f5004160600 2024-04-02T13:30:52.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.001+0000 7f50089fa640 1 -- 172.21.15.44:0/171437192 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8031b80 con 0x7f5004160600 2024-04-02T13:30:52.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.002+0000 7f500bc60640 1 -- 172.21.15.44:0/171437192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5004160600 msgr2=0x7f50040d0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.002+0000 7f500bc60640 1 --2- 172.21.15.44:0/171437192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5004160600 0x7f50040d0180 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009870 tx=0x7f4ff802b860 comp rx=0 tx=0).stop 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.002+0000 7f500bc60640 1 -- 172.21.15.44:0/171437192 shutdown_connections 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.002+0000 7f500bc60640 1 --2- 172.21.15.44:0/171437192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5004160600 0x7f50040d0180 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.002+0000 7f500bc60640 1 --2- 172.21.15.44:0/171437192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f500415c160 0x7f500406b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.002+0000 7f500bc60640 1 -- 172.21.15.44:0/171437192 >> 172.21.15.44:0/171437192 conn(0x7f50040b2980 msgr2=0x7f50040b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.002+0000 7f500bc60640 1 -- 172.21.15.44:0/171437192 shutdown_connections 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.002+0000 7f500bc60640 1 -- 172.21.15.44:0/171437192 wait complete. 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.003+0000 7f500bc60640 1 Processor -- start 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.003+0000 7f500bc60640 1 -- start start 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f500bc60640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50040d06c0 0x7f50041f2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f500bc60640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f500415c160 0x7f50040c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f500bc60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5004160600 0x7f50041f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f500bc60640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5004160a60 con 0x7f500415c160 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f500bc60640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f50040ccb70 con 0x7f50040d06c0 2024-04-02T13:30:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f500bc60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f50040add60 con 0x7f5004160600 2024-04-02T13:30:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f50091fb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f500415c160 0x7f50040c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f50091fb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f500415c160 0x7f50040c58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58934/0 (socket says 172.21.15.44:58934) 2024-04-02T13:30:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f50091fb640 1 -- 172.21.15.44:0/2402813583 learned_addr learned my addr 172.21.15.44:0/2402813583 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f50091fb640 1 -- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5004160600 msgr2=0x7f50041f21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f500a1fd640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5004160600 0x7f50041f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f50099fc640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50040d06c0 0x7f50041f2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f50091fb640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5004160600 0x7f50041f21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f50091fb640 1 -- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50040d06c0 msgr2=0x7f50041f2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f50091fb640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50040d06c0 0x7f50041f2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f50091fb640 1 -- 172.21.15.44:0/2402813583 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50040c5df0 con 0x7f500415c160 2024-04-02T13:30:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f50099fc640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50040d06c0 0x7f50041f2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.004+0000 7f50091fb640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f500415c160 0x7f50040c58b0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f4ff802b8a0 tx=0x7f4ff80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.005+0000 7f50027fc640 1 -- 172.21.15.44:0/2402813583 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8032860 con 0x7f500415c160 2024-04-02T13:30:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.005+0000 7f50027fc640 1 -- 172.21.15.44:0/2402813583 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ff8032a00 con 0x7f500415c160 2024-04-02T13:30:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.005+0000 7f50027fc640 1 -- 172.21.15.44:0/2402813583 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8032df0 con 0x7f500415c160 2024-04-02T13:30:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.005+0000 7f500bc60640 1 -- 172.21.15.44:0/2402813583 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50040c6070 con 0x7f500415c160 2024-04-02T13:30:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.005+0000 7f500bc60640 1 -- 172.21.15.44:0/2402813583 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50040c6910 con 0x7f500415c160 2024-04-02T13:30:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.006+0000 7f500bc60640 1 -- 172.21.15.44:0/2402813583 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50040684f0 con 0x7f500415c160 2024-04-02T13:30:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.006+0000 7f50027fc640 1 -- 172.21.15.44:0/2402813583 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4ff804c060 con 0x7f500415c160 2024-04-02T13:30:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.006+0000 7f50027fc640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ff006c8f0 0x7f4ff006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.006+0000 7f50027fc640 1 -- 172.21.15.44:0/2402813583 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x7f4ff80adb80 con 0x7f500415c160 2024-04-02T13:30:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.007+0000 7f50099fc640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ff006c8f0 0x7f4ff006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.007+0000 7f50099fc640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ff006c8f0 0x7f4ff006edb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc006fd0 tx=0x7f4ffc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.011+0000 7f50027fc640 1 -- 172.21.15.44:0/2402813583 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f50040c5df0 con 0x7f500415c160 2024-04-02T13:30:52.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.195+0000 7f500bc60640 1 -- 172.21.15.44:0/2402813583 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f500406b4d0 con 0x7f500415c160 2024-04-02T13:30:52.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.196+0000 7f50027fc640 1 -- 172.21.15.44:0/2402813583 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+17239 (secure 0 0 0) 0x7f4ff8040070 con 0x7f500415c160 2024-04-02T13:30:52.200 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:52.200 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":110,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:30:49.436981+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:29:09.192430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-02T13:29:09.860748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.d","mappings":[{"from":6,"to":7}]},{"pgid":"16.f","mappings":[{"from":5,"to":4}]},{"pgid":"16.1b","mappings":[{"from":1,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6834/1745009605":"2024-04-03T13:30:49.436959+0000","172.21.15.44:6833/1745009605":"2024-04-03T13:30:49.436959+0000","172.21.15.44:6834/3259420283":"2024-04-03T13:30:12.628183+0000","172.21.15.44:6833/3259420283":"2024-04-03T13:30:12.628183+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:30:52.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.198+0000 7f500bc60640 1 -- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ff006c8f0 msgr2=0x7f4ff006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.198+0000 7f500bc60640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ff006c8f0 0x7f4ff006edb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc006fd0 tx=0x7f4ffc008040 comp rx=0 tx=0).stop 2024-04-02T13:30:52.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.198+0000 7f500bc60640 1 -- 172.21.15.44:0/2402813583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f500415c160 msgr2=0x7f50040c58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.198+0000 7f500bc60640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f500415c160 0x7f50040c58b0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f4ff802b8a0 tx=0x7f4ff80046c0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.198+0000 7f500bc60640 1 -- 172.21.15.44:0/2402813583 shutdown_connections 2024-04-02T13:30:52.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.198+0000 7f500bc60640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ff006c8f0 0x7f4ff006edb0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.198+0000 7f500bc60640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5004160600 0x7f50041f21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.198+0000 7f500bc60640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f500415c160 0x7f50040c58b0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.198+0000 7f500bc60640 1 --2- 172.21.15.44:0/2402813583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50040d06c0 0x7f50041f2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.198+0000 7f500bc60640 1 -- 172.21.15.44:0/2402813583 >> 172.21.15.44:0/2402813583 conn(0x7f50040b2980 msgr2=0x7f50040ae530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:52.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.198+0000 7f500bc60640 1 -- 172.21.15.44:0/2402813583 shutdown_connections 2024-04-02T13:30:52.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.198+0000 7f500bc60640 1 -- 172.21.15.44:0/2402813583 wait complete. 2024-04-02T13:30:52.213 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T13:30:52.213 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:30:52.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.311+0000 7f0d76504640 1 Processor -- start 2024-04-02T13:30:52.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.311+0000 7f0d76504640 1 -- start start 2024-04-02T13:30:52.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d76504640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0d700cece0 0x7f0d700cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d76504640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d700d09f0 0x7f0d7006d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d76504640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d700cf600 con 0x7f0d7006ece0 2024-04-02T13:30:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d76504640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0d7006dc40 con 0x7f0d700cece0 2024-04-02T13:30:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d76504640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0d7006df20 con 0x7f0d700d09f0 2024-04-02T13:30:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d6ffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0d700cece0 0x7f0d700cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d6ffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0d700cece0 0x7f0d700cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58942/0 (socket says 172.21.15.44:58942) 2024-04-02T13:30:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d6ffff640 1 -- 172.21.15.44:0/2813104554 learned_addr learned my addr 172.21.15.44:0/2813104554 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d6f7fe640 1 --2- 172.21.15.44:0/2813104554 >> v2:172.21.15.69:3300/0 conn(0x7f0d700d09f0 0x7f0d7006d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d6ffff640 1 -- 172.21.15.44:0/2813104554 >> v2:172.21.15.69:3300/0 conn(0x7f0d700d09f0 msgr2=0x7f0d7006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d6ffff640 1 --2- 172.21.15.44:0/2813104554 >> v2:172.21.15.69:3300/0 conn(0x7f0d700d09f0 0x7f0d7006d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d6ffff640 1 -- 172.21.15.44:0/2813104554 >> v1:172.21.15.69:6789/0 conn(0x7f0d7006ece0 legacy=0x7f0d7015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d6ffff640 1 -- 172.21.15.44:0/2813104554 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7006e200 con 0x7f0d700cece0 2024-04-02T13:30:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.312+0000 7f0d6f7fe640 1 --2- 172.21.15.44:0/2813104554 >> v2:172.21.15.69:3300/0 conn(0x7f0d700d09f0 0x7f0d7006d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:52.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.313+0000 7f0d6ffff640 1 --2- 172.21.15.44:0/2813104554 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d700cece0 0x7f0d700cf0c0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f0d700b44f0 tx=0x7f0d6002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1e7f753c0c6426 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:52.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.313+0000 7f0d6effd640 1 -- 172.21.15.44:0/2813104554 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d60034070 con 0x7f0d700cece0 2024-04-02T13:30:52.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.313+0000 7f0d6effd640 1 -- 172.21.15.44:0/2813104554 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d60030860 con 0x7f0d700cece0 2024-04-02T13:30:52.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.313+0000 7f0d6effd640 1 -- 172.21.15.44:0/2813104554 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d60030c30 con 0x7f0d700cece0 2024-04-02T13:30:52.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.314+0000 7f0d76504640 1 -- 172.21.15.44:0/2813104554 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d700cece0 msgr2=0x7f0d700cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.314+0000 7f0d76504640 1 --2- 172.21.15.44:0/2813104554 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d700cece0 0x7f0d700cf0c0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f0d700b44f0 tx=0x7f0d6002b9a0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.314+0000 7f0d76504640 1 -- 172.21.15.44:0/2813104554 shutdown_connections 2024-04-02T13:30:52.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.314+0000 7f0d76504640 1 --2- 172.21.15.44:0/2813104554 >> v2:172.21.15.69:3300/0 conn(0x7f0d700d09f0 0x7f0d7006d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.314+0000 7f0d76504640 1 --2- 172.21.15.44:0/2813104554 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d700cece0 0x7f0d700cf0c0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.314+0000 7f0d76504640 1 -- 172.21.15.44:0/2813104554 >> 172.21.15.44:0/2813104554 conn(0x7f0d700cfcd0 msgr2=0x7f0d700d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:52.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.314+0000 7f0d76504640 1 -- 172.21.15.44:0/2813104554 shutdown_connections 2024-04-02T13:30:52.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.314+0000 7f0d76504640 1 -- 172.21.15.44:0/2813104554 wait complete. 2024-04-02T13:30:52.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.315+0000 7f0d76504640 1 Processor -- start 2024-04-02T13:30:52.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.315+0000 7f0d76504640 1 -- start start 2024-04-02T13:30:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d76504640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d7006ece0 0x7f0d70150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d76504640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d700cece0 0x7f0d701519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d76504640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d700d09f0 0x7f0d70154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d76504640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7006d810 con 0x7f0d700d09f0 2024-04-02T13:30:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d76504640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7006e7c0 con 0x7f0d7006ece0 2024-04-02T13:30:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d76504640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0d7006d990 con 0x7f0d700cece0 2024-04-02T13:30:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d74aa1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d700d09f0 0x7f0d70154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d74aa1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d700d09f0 0x7f0d70154500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58944/0 (socket says 172.21.15.44:58944) 2024-04-02T13:30:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d74aa1640 1 -- 172.21.15.44:0/843887129 learned_addr learned my addr 172.21.15.44:0/843887129 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d6f7fe640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d700cece0 0x7f0d701519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d6ffff640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d7006ece0 0x7f0d70150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d74aa1640 1 -- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d700cece0 msgr2=0x7f0d701519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d74aa1640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d700cece0 0x7f0d701519a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d74aa1640 1 -- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d7006ece0 msgr2=0x7f0d70150260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d74aa1640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d7006ece0 0x7f0d70150260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d74aa1640 1 -- 172.21.15.44:0/843887129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d701f7850 con 0x7f0d700d09f0 2024-04-02T13:30:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.316+0000 7f0d74aa1640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d700d09f0 0x7f0d70154500 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800c7a0 tx=0x7f0d6800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.317+0000 7f0d6cff9640 1 -- 172.21.15.44:0/843887129 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68003b70 con 0x7f0d700d09f0 2024-04-02T13:30:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.317+0000 7f0d6cff9640 1 -- 172.21.15.44:0/843887129 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d68003d30 con 0x7f0d700d09f0 2024-04-02T13:30:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.317+0000 7f0d76504640 1 -- 172.21.15.44:0/843887129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d701f7b30 con 0x7f0d700d09f0 2024-04-02T13:30:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.317+0000 7f0d6cff9640 1 -- 172.21.15.44:0/843887129 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6800aa90 con 0x7f0d700d09f0 2024-04-02T13:30:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.317+0000 7f0d76504640 1 -- 172.21.15.44:0/843887129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d701f80f0 con 0x7f0d700d09f0 2024-04-02T13:30:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.318+0000 7f0d76504640 1 -- 172.21.15.44:0/843887129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d70074f20 con 0x7f0d700d09f0 2024-04-02T13:30:52.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.318+0000 7f0d6cff9640 1 -- 172.21.15.44:0/843887129 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0d6800ac30 con 0x7f0d700d09f0 2024-04-02T13:30:52.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.318+0000 7f0d6cff9640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0d5c06c8f0 0x7f0d5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.319+0000 7f0d6cff9640 1 -- 172.21.15.44:0/843887129 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x7f0d6808bd80 con 0x7f0d700d09f0 2024-04-02T13:30:52.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.319+0000 7f0d6ffff640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0d5c06c8f0 0x7f0d5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.319+0000 7f0d6ffff640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0d5c06c8f0 0x7f0d5c06edb0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0d60036000 tx=0x7f0d6003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.323+0000 7f0d6cff9640 1 -- 172.21.15.44:0/843887129 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0d68059f20 con 0x7f0d700d09f0 2024-04-02T13:30:52.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.545+0000 7f0d76504640 1 -- 172.21.15.44:0/843887129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d7015ad90 con 0x7f0d700d09f0 2024-04-02T13:30:52.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.545+0000 7f0d6cff9640 1 -- 172.21.15.44:0/843887129 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+5170 (secure 0 0 0) 0x7f0d68059820 con 0x7f0d700d09f0 2024-04-02T13:30:52.550 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:52.550 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7696,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2715855087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2715855087},{"type":"v1","addr":"172.21.15.69:6836","nonce":2715855087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7699,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3850680553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3850680553},{"type":"v1","addr":"172.21.15.69:6837","nonce":3850680553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/1330712383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1330712383},{"type":"v1","addr":"172.21.15.44:6835","nonce":1330712383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":9348,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/3518070032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":3518070032},{"type":"v1","addr":"172.21.15.44:6834","nonce":3518070032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:29:10.907920+0000","modified":"2024-04-02T13:30:49.437157+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-04-02T13:30:52.550 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 90 2024-04-02T13:30:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.547+0000 7f0d76504640 1 -- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0d5c06c8f0 msgr2=0x7f0d5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.547+0000 7f0d76504640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0d5c06c8f0 0x7f0d5c06edb0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0d60036000 tx=0x7f0d6003b040 comp rx=0 tx=0).stop 2024-04-02T13:30:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.547+0000 7f0d76504640 1 -- 172.21.15.44:0/843887129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d700d09f0 msgr2=0x7f0d70154500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.547+0000 7f0d76504640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d700d09f0 0x7f0d70154500 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800c7a0 tx=0x7f0d6800cc60 comp rx=0 tx=0).stop 2024-04-02T13:30:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.547+0000 7f0d76504640 1 -- 172.21.15.44:0/843887129 shutdown_connections 2024-04-02T13:30:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.547+0000 7f0d76504640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0d5c06c8f0 0x7f0d5c06edb0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.547+0000 7f0d76504640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d700d09f0 0x7f0d70154500 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.547+0000 7f0d76504640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d700cece0 0x7f0d701519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.547+0000 7f0d76504640 1 --2- 172.21.15.44:0/843887129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d7006ece0 0x7f0d70150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.547+0000 7f0d76504640 1 -- 172.21.15.44:0/843887129 >> 172.21.15.44:0/843887129 conn(0x7f0d700cfcd0 msgr2=0x7f0d700ae1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:52.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.548+0000 7f0d76504640 1 -- 172.21.15.44:0/843887129 shutdown_connections 2024-04-02T13:30:52.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.548+0000 7f0d76504640 1 -- 172.21.15.44:0/843887129 wait complete. 2024-04-02T13:30:52.563 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:30:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.664+0000 7fca09ca5640 1 Processor -- start 2024-04-02T13:30:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.664+0000 7fca09ca5640 1 -- start start 2024-04-02T13:30:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.664+0000 7fca09ca5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fca040cece0 0x7fca040cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.664+0000 7fca09ca5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca040cf600 con 0x7fca040d09f0 2024-04-02T13:30:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.664+0000 7fca09ca5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca0406dc40 con 0x7fca0406ece0 2024-04-02T13:30:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.664+0000 7fca09ca5640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fca0406d9e0 con 0x7fca040cece0 2024-04-02T13:30:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.665+0000 7fca037fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fca040cece0 0x7fca040cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.665+0000 7fca02ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fca040d09f0 0x7fca0406d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37814/0 (socket says 172.21.15.44:37814) 2024-04-02T13:30:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.665+0000 7fca02ffd640 1 -- 172.21.15.44:0/2140463852 learned_addr learned my addr 172.21.15.44:0/2140463852 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.665+0000 7fca037fe640 1 -- 172.21.15.44:0/2140463852 >> v1:172.21.15.69:6789/0 conn(0x7fca0406ece0 legacy=0x7fca040acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.665+0000 7fca037fe640 1 -- 172.21.15.44:0/2140463852 >> v1:172.21.15.44:6789/0 conn(0x7fca040d09f0 legacy=0x7fca0406d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.665+0000 7fca037fe640 1 -- 172.21.15.44:0/2140463852 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0406df20 con 0x7fca040cece0 2024-04-02T13:30:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.666+0000 7fca037fe640 1 --2- 172.21.15.44:0/2140463852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca040cece0 0x7fca040cf0c0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fc9f40097a0 tx=0x7fc9f402b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d77d3d756407b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:52.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.666+0000 7fca027fc640 1 -- 172.21.15.44:0/2140463852 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4037070 con 0x7fca040cece0 2024-04-02T13:30:52.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.666+0000 7fca027fc640 1 -- 172.21.15.44:0/2140463852 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9f4030710 con 0x7fca040cece0 2024-04-02T13:30:52.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.666+0000 7fca027fc640 1 -- 172.21.15.44:0/2140463852 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4030ae0 con 0x7fca040cece0 2024-04-02T13:30:52.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.666+0000 7fca09ca5640 1 -- 172.21.15.44:0/2140463852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca040cece0 msgr2=0x7fca040cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.666+0000 7fca09ca5640 1 --2- 172.21.15.44:0/2140463852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca040cece0 0x7fca040cf0c0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fc9f40097a0 tx=0x7fc9f402b710 comp rx=0 tx=0).stop 2024-04-02T13:30:52.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.666+0000 7fca09ca5640 1 -- 172.21.15.44:0/2140463852 shutdown_connections 2024-04-02T13:30:52.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.666+0000 7fca09ca5640 1 --2- 172.21.15.44:0/2140463852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca040cece0 0x7fca040cf0c0 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.666+0000 7fca09ca5640 1 -- 172.21.15.44:0/2140463852 >> 172.21.15.44:0/2140463852 conn(0x7fca040cfcd0 msgr2=0x7fca040d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:52.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.666+0000 7fca09ca5640 1 -- 172.21.15.44:0/2140463852 shutdown_connections 2024-04-02T13:30:52.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.667+0000 7fca09ca5640 1 -- 172.21.15.44:0/2140463852 wait complete. 2024-04-02T13:30:52.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.668+0000 7fca09ca5640 1 Processor -- start 2024-04-02T13:30:52.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.668+0000 7fca09ca5640 1 -- start start 2024-04-02T13:30:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.668+0000 7fca09ca5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca0406ece0 0x7fca04149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.668+0000 7fca09ca5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca040cece0 0x7fca0415bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.668+0000 7fca09ca5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca040d09f0 0x7fca0415d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.668+0000 7fca09ca5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fca0406e7c0 con 0x7fca040cece0 2024-04-02T13:30:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.668+0000 7fca09ca5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fca0406e4e0 con 0x7fca0406ece0 2024-04-02T13:30:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.668+0000 7fca09ca5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fca0415b220 con 0x7fca040d09f0 2024-04-02T13:30:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fca02ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca040cece0 0x7fca0415bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fca02ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca040cece0 0x7fca0415bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58960/0 (socket says 172.21.15.44:58960) 2024-04-02T13:30:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fca02ffd640 1 -- 172.21.15.44:0/7838290 learned_addr learned my addr 172.21.15.44:0/7838290 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fca03fff640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca040d09f0 0x7fca0415d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fca037fe640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca0406ece0 0x7fca04149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fca02ffd640 1 -- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca040d09f0 msgr2=0x7fca0415d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fca02ffd640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca040d09f0 0x7fca0415d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fca02ffd640 1 -- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca0406ece0 msgr2=0x7fca04149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fca02ffd640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca0406ece0 0x7fca04149910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fca02ffd640 1 -- 172.21.15.44:0/7838290 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0415c880 con 0x7fca040cece0 2024-04-02T13:30:52.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fca02ffd640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca040cece0 0x7fca0415bfc0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fc9f800c5f0 tx=0x7fc9f800cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:52.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fc9ebfff640 1 -- 172.21.15.44:0/7838290 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f8018070 con 0x7fca040cece0 2024-04-02T13:30:52.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fc9ebfff640 1 -- 172.21.15.44:0/7838290 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9f80074e0 con 0x7fca040cece0 2024-04-02T13:30:52.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fca09ca5640 1 -- 172.21.15.44:0/7838290 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca0409abc0 con 0x7fca040cece0 2024-04-02T13:30:52.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fc9ebfff640 1 -- 172.21.15.44:0/7838290 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f8004a00 con 0x7fca040cece0 2024-04-02T13:30:52.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.669+0000 7fca09ca5640 1 -- 172.21.15.44:0/7838290 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca0409b180 con 0x7fca040cece0 2024-04-02T13:30:52.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.670+0000 7fca09ca5640 1 -- 172.21.15.44:0/7838290 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca04074f20 con 0x7fca040cece0 2024-04-02T13:30:52.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.671+0000 7fc9ebfff640 1 -- 172.21.15.44:0/7838290 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc9f801b070 con 0x7fca040cece0 2024-04-02T13:30:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.671+0000 7fc9ebfff640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9f006c8f0 0x7fc9f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.671+0000 7fc9ebfff640 1 -- 172.21.15.44:0/7838290 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x7fc9f808b130 con 0x7fca040cece0 2024-04-02T13:30:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.671+0000 7fca037fe640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9f006c8f0 0x7fc9f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.672+0000 7fca037fe640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9f006c8f0 0x7fc9f006edb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4034000 tx=0x7fc9f4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.676+0000 7fc9ebfff640 1 -- 172.21.15.44:0/7838290 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc9f8058d10 con 0x7fca040cece0 2024-04-02T13:30:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.902+0000 7fca09ca5640 1 -- 172.21.15.44:0/7838290 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca040684f0 con 0x7fca040cece0 2024-04-02T13:30:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.903+0000 7fc9ebfff640 1 -- 172.21.15.44:0/7838290 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+5170 (secure 0 0 0) 0x7fc9f805d010 con 0x7fca040cece0 2024-04-02T13:30:52.907 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:52.907 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7696,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2715855087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2715855087},{"type":"v1","addr":"172.21.15.69:6836","nonce":2715855087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7699,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3850680553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3850680553},{"type":"v1","addr":"172.21.15.69:6837","nonce":3850680553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/1330712383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1330712383},{"type":"v1","addr":"172.21.15.44:6835","nonce":1330712383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":9348,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/3518070032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":3518070032},{"type":"v1","addr":"172.21.15.44:6834","nonce":3518070032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:29:10.907920+0000","modified":"2024-04-02T13:30:49.437157+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-04-02T13:30:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 90 2024-04-02T13:30:52.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.904+0000 7fca09ca5640 1 -- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9f006c8f0 msgr2=0x7fc9f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.904+0000 7fca09ca5640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9f006c8f0 0x7fc9f006edb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4034000 tx=0x7fc9f4008030 comp rx=0 tx=0).stop 2024-04-02T13:30:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.904+0000 7fca09ca5640 1 -- 172.21.15.44:0/7838290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca040cece0 msgr2=0x7fca0415bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.904+0000 7fca09ca5640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca040cece0 0x7fca0415bfc0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fc9f800c5f0 tx=0x7fc9f800cab0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.904+0000 7fca09ca5640 1 -- 172.21.15.44:0/7838290 shutdown_connections 2024-04-02T13:30:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.904+0000 7fca09ca5640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9f006c8f0 0x7fc9f006edb0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.904+0000 7fca09ca5640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca040d09f0 0x7fca0415d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.904+0000 7fca09ca5640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca040cece0 0x7fca0415bfc0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.904+0000 7fca09ca5640 1 --2- 172.21.15.44:0/7838290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca0406ece0 0x7fca04149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.904+0000 7fca09ca5640 1 -- 172.21.15.44:0/7838290 >> 172.21.15.44:0/7838290 conn(0x7fca040cfcd0 msgr2=0x7fca04071190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.905+0000 7fca09ca5640 1 -- 172.21.15.44:0/7838290 shutdown_connections 2024-04-02T13:30:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:52.905+0000 7fca09ca5640 1 -- 172.21.15.44:0/7838290 wait complete. 2024-04-02T13:30:52.919 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:30:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.021+0000 7fee26e3c640 1 Processor -- start 2024-04-02T13:30:53.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.021+0000 7fee26e3c640 1 -- start start 2024-04-02T13:30:53.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee26e3c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fee2015e550 0x7fee2006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee26e3c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2015f040 0x7fee200d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee26e3c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2015fb50 con 0x7fee2006d530 2024-04-02T13:30:53.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee26e3c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fee2006d970 con 0x7fee2015e550 2024-04-02T13:30:53.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee26e3c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fee2006ca90 con 0x7fee2015f040 2024-04-02T13:30:53.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee1ffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fee2006d530 0x7fee2015e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37830/0 (socket says 172.21.15.44:37830) 2024-04-02T13:30:53.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee24bd8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fee2015e550 0x7fee2006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee1ffff640 1 -- 172.21.15.44:0/3054349332 learned_addr learned my addr 172.21.15.44:0/3054349332 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:53.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee253d9640 1 --2- 172.21.15.44:0/3054349332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2015f040 0x7fee200d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee24bd8640 1 -- 172.21.15.44:0/3054349332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2015f040 msgr2=0x7fee200d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:53.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee24bd8640 1 --2- 172.21.15.44:0/3054349332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2015f040 0x7fee200d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee24bd8640 1 -- 172.21.15.44:0/3054349332 >> v1:172.21.15.44:6789/0 conn(0x7fee2006d530 legacy=0x7fee2015e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee24bd8640 1 -- 172.21.15.44:0/3054349332 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2015f590 con 0x7fee2015e550 2024-04-02T13:30:53.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.022+0000 7fee1f7fe640 1 -- 172.21.15.44:0/3054349332 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778190208 0 0) 0x7fee2015fb50 con 0x7fee2006d530 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.023+0000 7fee24bd8640 1 --2- 172.21.15.44:0/3054349332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2015e550 0x7fee2006cff0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fee140097a0 tx=0x7fee1402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885b11ba8467e6d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.023+0000 7fee1f7fe640 1 -- 172.21.15.44:0/3054349332 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14034070 con 0x7fee2015e550 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.023+0000 7fee1f7fe640 1 -- 172.21.15.44:0/3054349332 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee14030610 con 0x7fee2015e550 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.023+0000 7fee1f7fe640 1 -- 172.21.15.44:0/3054349332 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee140309e0 con 0x7fee2015e550 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.024+0000 7fee26e3c640 1 -- 172.21.15.44:0/3054349332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2015e550 msgr2=0x7fee2006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.024+0000 7fee26e3c640 1 --2- 172.21.15.44:0/3054349332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2015e550 0x7fee2006cff0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fee140097a0 tx=0x7fee1402b960 comp rx=0 tx=0).stop 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.024+0000 7fee26e3c640 1 -- 172.21.15.44:0/3054349332 shutdown_connections 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.024+0000 7fee26e3c640 1 --2- 172.21.15.44:0/3054349332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2015f040 0x7fee200d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.024+0000 7fee26e3c640 1 --2- 172.21.15.44:0/3054349332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2015e550 0x7fee2006cff0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.024+0000 7fee26e3c640 1 -- 172.21.15.44:0/3054349332 >> 172.21.15.44:0/3054349332 conn(0x7fee2009aa20 msgr2=0x7fee2009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.024+0000 7fee26e3c640 1 -- 172.21.15.44:0/3054349332 shutdown_connections 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.024+0000 7fee26e3c640 1 -- 172.21.15.44:0/3054349332 wait complete. 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.025+0000 7fee26e3c640 1 Processor -- start 2024-04-02T13:30:53.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.025+0000 7fee26e3c640 1 -- start start 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee26e3c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2006d530 0x7fee200b59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee26e3c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2015e550 0x7fee200b7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee26e3c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2015f040 0x7fee201521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee26e3c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fee200d0590 con 0x7fee2006d530 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee26e3c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fee200d02b0 con 0x7fee2015e550 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee26e3c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fee200cc4b0 con 0x7fee2015f040 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee24bd8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2006d530 0x7fee200b59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee24bd8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2006d530 0x7fee200b59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58988/0 (socket says 172.21.15.44:58988) 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee24bd8640 1 -- 172.21.15.44:0/3941123303 learned_addr learned my addr 172.21.15.44:0/3941123303 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee24bd8640 1 -- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2015f040 msgr2=0x7fee201521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee253d9640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2015f040 0x7fee201521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee1ffff640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2015e550 0x7fee200b7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee24bd8640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2015f040 0x7fee201521d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee24bd8640 1 -- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2015e550 msgr2=0x7fee200b7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee24bd8640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2015e550 0x7fee200b7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee24bd8640 1 -- 172.21.15.44:0/3941123303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee200b7d60 con 0x7fee2006d530 2024-04-02T13:30:53.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee1ffff640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2015e550 0x7fee200b7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:30:53.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee24bd8640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2006d530 0x7fee200b59c0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fee140098c0 tx=0x7fee140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:53.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee1d7fa640 1 -- 172.21.15.44:0/3941123303 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14034040 con 0x7fee2006d530 2024-04-02T13:30:53.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee1d7fa640 1 -- 172.21.15.44:0/3941123303 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee1403aaa0 con 0x7fee2006d530 2024-04-02T13:30:53.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee26e3c640 1 -- 172.21.15.44:0/3941123303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee200b5fa0 con 0x7fee2006d530 2024-04-02T13:30:53.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee1d7fa640 1 -- 172.21.15.44:0/3941123303 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee140443f0 con 0x7fee2006d530 2024-04-02T13:30:53.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.026+0000 7fee26e3c640 1 -- 172.21.15.44:0/3941123303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee200b6560 con 0x7fee2006d530 2024-04-02T13:30:53.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.028+0000 7fee26e3c640 1 -- 172.21.15.44:0/3941123303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee200684f0 con 0x7fee2006d530 2024-04-02T13:30:53.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.028+0000 7fee1d7fa640 1 -- 172.21.15.44:0/3941123303 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fee1404b060 con 0x7fee2006d530 2024-04-02T13:30:53.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.028+0000 7fee1d7fa640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fee0406c8f0 0x7fee0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.029+0000 7fee1d7fa640 1 -- 172.21.15.44:0/3941123303 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x7fee140adca0 con 0x7fee2006d530 2024-04-02T13:30:53.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.029+0000 7fee1ffff640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fee0406c8f0 0x7fee0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.029+0000 7fee1ffff640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fee0406c8f0 0x7fee0406edb0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fee2015ce50 tx=0x7fee0c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:53.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.033+0000 7fee1d7fa640 1 -- 172.21.15.44:0/3941123303 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fee200b7d60 con 0x7fee2006d530 2024-04-02T13:30:53.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.219+0000 7fee26e3c640 1 -- 172.21.15.44:0/3941123303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fee2015dae0 con 0x7fee2006d530 2024-04-02T13:30:53.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.220+0000 7fee1d7fa640 1 -- 172.21.15.44:0/3941123303 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+17239 (secure 0 0 0) 0x7fee1403f070 con 0x7fee2006d530 2024-04-02T13:30:53.224 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:53.224 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":110,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:30:49.436981+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:29:09.192430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-02T13:29:09.860748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.d","mappings":[{"from":6,"to":7}]},{"pgid":"16.f","mappings":[{"from":5,"to":4}]},{"pgid":"16.1b","mappings":[{"from":1,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6834/1745009605":"2024-04-03T13:30:49.436959+0000","172.21.15.44:6833/1745009605":"2024-04-03T13:30:49.436959+0000","172.21.15.44:6834/3259420283":"2024-04-03T13:30:12.628183+0000","172.21.15.44:6833/3259420283":"2024-04-03T13:30:12.628183+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:30:53.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.222+0000 7fee26e3c640 1 -- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fee0406c8f0 msgr2=0x7fee0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.222+0000 7fee26e3c640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fee0406c8f0 0x7fee0406edb0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fee2015ce50 tx=0x7fee0c009040 comp rx=0 tx=0).stop 2024-04-02T13:30:53.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.222+0000 7fee26e3c640 1 -- 172.21.15.44:0/3941123303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2006d530 msgr2=0x7fee200b59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.222+0000 7fee26e3c640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2006d530 0x7fee200b59c0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fee140098c0 tx=0x7fee140046c0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.222+0000 7fee26e3c640 1 -- 172.21.15.44:0/3941123303 shutdown_connections 2024-04-02T13:30:53.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.222+0000 7fee26e3c640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fee0406c8f0 0x7fee0406edb0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.222+0000 7fee26e3c640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee2015f040 0x7fee201521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.222+0000 7fee26e3c640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2015e550 0x7fee200b7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.222+0000 7fee26e3c640 1 --2- 172.21.15.44:0/3941123303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee2006d530 0x7fee200b59c0 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.222+0000 7fee26e3c640 1 -- 172.21.15.44:0/3941123303 >> 172.21.15.44:0/3941123303 conn(0x7fee2009aa20 msgr2=0x7fee2009ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:53.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.222+0000 7fee26e3c640 1 -- 172.21.15.44:0/3941123303 shutdown_connections 2024-04-02T13:30:53.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.222+0000 7fee26e3c640 1 -- 172.21.15.44:0/3941123303 wait complete. 2024-04-02T13:30:53.237 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:30:53.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.338+0000 7f6c9bf1c640 1 Processor -- start 2024-04-02T13:30:53.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.338+0000 7f6c9bf1c640 1 -- start start 2024-04-02T13:30:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c9bf1c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6c9415c160 0x7f6c9406b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c9bf1c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c9406bab0 con 0x7f6c940d0890 2024-04-02T13:30:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c9bf1c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c9406bd90 con 0x7f6c94160600 2024-04-02T13:30:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c9bf1c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6c940ccec0 con 0x7f6c9415c160 2024-04-02T13:30:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c9a4b9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6c940d0890 0x7f6c940cd6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37838/0 (socket says 172.21.15.44:37838) 2024-04-02T13:30:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c9a4b9640 1 -- 172.21.15.44:0/480855700 learned_addr learned my addr 172.21.15.44:0/480855700 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c99cb8640 1 --2- 172.21.15.44:0/480855700 >> v2:172.21.15.69:3300/0 conn(0x7f6c9415c160 0x7f6c9406b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4198973031 0 0) 0x7f6c9406bab0 con 0x7f6c940d0890 2024-04-02T13:30:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c80003520 con 0x7f6c940d0890 2024-04-02T13:30:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 793362784 0 0) 0x7f6c9406bd90 con 0x7f6c94160600 2024-04-02T13:30:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c9406bab0 con 0x7f6c94160600 2024-04-02T13:30:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1524273559 0 0) 0x7f6c80003520 con 0x7f6c940d0890 2024-04-02T13:30:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 >> v2:172.21.15.69:3300/0 conn(0x7f6c9415c160 msgr2=0x7f6c9406b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c98cb6640 1 --2- 172.21.15.44:0/480855700 >> v2:172.21.15.69:3300/0 conn(0x7f6c9415c160 0x7f6c9406b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 >> v1:172.21.15.69:6789/0 conn(0x7f6c94160600 legacy=0x7f6c940d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c940cd1a0 con 0x7f6c940d0890 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f6c8c0019f0 con 0x7f6c940d0890 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c99cb8640 1 --2- 172.21.15.44:0/480855700 >> v2:172.21.15.69:3300/0 conn(0x7f6c9415c160 0x7f6c9406b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.339+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 >> v1:172.21.15.44:6789/0 conn(0x7f6c940d0890 legacy=0x7f6c940cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c98cb6640 1 --2- 172.21.15.44:0/480855700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c80003b40 0x7f6c80003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c98cb6640 1 --2- 172.21.15.44:0/480855700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c80004800 0x7f6c80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c98cb6640 1 --2- 172.21.15.44:0/480855700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c800051c0 0x7f6c8000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6c940ccec0 con 0x7f6c80004800 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8000c0f0 con 0x7f6c80003b40 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6c8000c2f0 con 0x7f6c800051c0 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c994b7640 1 --2- 172.21.15.44:0/480855700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c80004800 0x7f6c80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c99cb8640 1 --2- 172.21.15.44:0/480855700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c80003b40 0x7f6c80003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c9a4b9640 1 --2- 172.21.15.44:0/480855700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c800051c0 0x7f6c8000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c994b7640 1 -- 172.21.15.44:0/480855700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c800051c0 msgr2=0x7f6c8000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c994b7640 1 --2- 172.21.15.44:0/480855700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c800051c0 0x7f6c8000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c994b7640 1 -- 172.21.15.44:0/480855700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c80003b40 msgr2=0x7f6c80003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c994b7640 1 --2- 172.21.15.44:0/480855700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c80003b40 0x7f6c80003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c994b7640 1 -- 172.21.15.44:0/480855700 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c80003520 con 0x7f6c80004800 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.340+0000 7f6c9a4b9640 1 -- 172.21.15.44:0/480855700 reap_dead start 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.341+0000 7f6c994b7640 1 --2- 172.21.15.44:0/480855700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c80004800 0x7f6c80004c80 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f6c840018d0 tx=0x7f6c80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add9f556498eee06 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.341+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84037070 con 0x7f6c80004800 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.341+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c840052d0 con 0x7f6c80004800 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.341+0000 7f6c98cb6640 1 -- 172.21.15.44:0/480855700 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c840014f0 con 0x7f6c80004800 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.341+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/480855700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c80004800 msgr2=0x7f6c80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.341+0000 7f6c9bf1c640 1 --2- 172.21.15.44:0/480855700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c80004800 0x7f6c80004c80 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f6c840018d0 tx=0x7f6c80005830 comp rx=0 tx=0).stop 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.341+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/480855700 shutdown_connections 2024-04-02T13:30:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.341+0000 7f6c9bf1c640 1 --2- 172.21.15.44:0/480855700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c80004800 0x7f6c80004c80 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.341+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/480855700 >> 172.21.15.44:0/480855700 conn(0x7f6c940b2980 msgr2=0x7f6c940b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:53.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.341+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/480855700 shutdown_connections 2024-04-02T13:30:53.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.341+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/480855700 wait complete. 2024-04-02T13:30:53.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.343+0000 7f6c9bf1c640 1 Processor -- start 2024-04-02T13:30:53.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.343+0000 7f6c9bf1c640 1 -- start start 2024-04-02T13:30:53.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.343+0000 7f6c9bf1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c80004800 0x7f6c94156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.343+0000 7f6c9bf1c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c9406f2f0 0x7f6c940c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.343+0000 7f6c9bf1c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c940c83c0 0x7f6c9415d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.343+0000 7f6c9bf1c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6c940add60 con 0x7f6c9406f2f0 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.343+0000 7f6c9bf1c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6c94160a60 con 0x7f6c940c83c0 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.343+0000 7f6c9bf1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6c940adee0 con 0x7f6c80004800 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.343+0000 7f6c994b7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c9406f2f0 0x7f6c940c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.343+0000 7f6c994b7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c9406f2f0 0x7f6c940c7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59002/0 (socket says 172.21.15.44:59002) 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.343+0000 7f6c994b7640 1 -- 172.21.15.44:0/1704540256 learned_addr learned my addr 172.21.15.44:0/1704540256 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.344+0000 7f6c99cb8640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c80004800 0x7f6c94156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.344+0000 7f6c9a4b9640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c940c83c0 0x7f6c9415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.344+0000 7f6c994b7640 1 -- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c80004800 msgr2=0x7f6c94156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.344+0000 7f6c994b7640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c80004800 0x7f6c94156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.344+0000 7f6c994b7640 1 -- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c940c83c0 msgr2=0x7f6c9415d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.344+0000 7f6c994b7640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c940c83c0 0x7f6c9415d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.344+0000 7f6c994b7640 1 -- 172.21.15.44:0/1704540256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c940c8c90 con 0x7f6c9406f2f0 2024-04-02T13:30:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.344+0000 7f6c994b7640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c9406f2f0 0x7f6c940c7e80 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f6c84035040 tx=0x7f6c840012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.344+0000 7f6c8a7fc640 1 -- 172.21.15.44:0/1704540256 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84042020 con 0x7f6c9406f2f0 2024-04-02T13:30:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.344+0000 7f6c8a7fc640 1 -- 172.21.15.44:0/1704540256 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c84038040 con 0x7f6c9406f2f0 2024-04-02T13:30:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.344+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/1704540256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c940c8f70 con 0x7f6c9406f2f0 2024-04-02T13:30:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.344+0000 7f6c8a7fc640 1 -- 172.21.15.44:0/1704540256 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84037070 con 0x7f6c9406f2f0 2024-04-02T13:30:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.344+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/1704540256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c94157150 con 0x7f6c9406f2f0 2024-04-02T13:30:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.345+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/1704540256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c940684f0 con 0x7f6c9406f2f0 2024-04-02T13:30:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.345+0000 7f6c8a7fc640 1 -- 172.21.15.44:0/1704540256 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6c84045020 con 0x7f6c9406f2f0 2024-04-02T13:30:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.346+0000 7f6c8a7fc640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c7806c8f0 0x7f6c7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.346+0000 7f6c8a7fc640 1 -- 172.21.15.44:0/1704540256 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x7f6c840ad3c0 con 0x7f6c9406f2f0 2024-04-02T13:30:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.346+0000 7f6c99cb8640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c7806c8f0 0x7f6c7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.347+0000 7f6c99cb8640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c7806c8f0 0x7f6c7806edb0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6c940aeee0 tx=0x7f6c90008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.350+0000 7f6c8a7fc640 1 -- 172.21.15.44:0/1704540256 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6c8404c030 con 0x7f6c9406f2f0 2024-04-02T13:30:53.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.572+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/1704540256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6c940b1550 con 0x7f6c9406f2f0 2024-04-02T13:30:53.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.717+0000 7f6c8a7fc640 1 -- 172.21.15.44:0/1704540256 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v91) ==== 161+0+0 (secure 0 0 0) 0x7f6c8407b560 con 0x7f6c9406f2f0 2024-04-02T13:30:53.721 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:30:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.719+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c7806c8f0 msgr2=0x7f6c7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.719+0000 7f6c9bf1c640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c7806c8f0 0x7f6c7806edb0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6c940aeee0 tx=0x7f6c90008040 comp rx=0 tx=0).stop 2024-04-02T13:30:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.719+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/1704540256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c9406f2f0 msgr2=0x7f6c940c7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.719+0000 7f6c9bf1c640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c9406f2f0 0x7f6c940c7e80 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f6c84035040 tx=0x7f6c840012d0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.720+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/1704540256 shutdown_connections 2024-04-02T13:30:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.720+0000 7f6c9bf1c640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6c7806c8f0 0x7f6c7806edb0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.720+0000 7f6c9bf1c640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c940c83c0 0x7f6c9415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.720+0000 7f6c9bf1c640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c9406f2f0 0x7f6c940c7e80 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.720+0000 7f6c9bf1c640 1 --2- 172.21.15.44:0/1704540256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c80004800 0x7f6c94156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.720+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/1704540256 >> 172.21.15.44:0/1704540256 conn(0x7f6c940b2980 msgr2=0x7f6c9415a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.720+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/1704540256 shutdown_connections 2024-04-02T13:30:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.720+0000 7f6c9bf1c640 1 -- 172.21.15.44:0/1704540256 wait complete. 2024-04-02T13:30:53.735 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T13:30:53.822 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:30:53.826 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:30:53.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.835+0000 7f0e401b5640 1 Processor -- start 2024-04-02T13:30:53.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.835+0000 7f0e401b5640 1 -- start start 2024-04-02T13:30:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.836+0000 7f0e401b5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0e3815bfc0 0x7f0e38072cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.836+0000 7f0e401b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380cff60 0x7f0e380d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.836+0000 7f0e401b5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e380b5c80 con 0x7f0e3806d1e0 2024-04-02T13:30:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.836+0000 7f0e401b5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0e3815c3a0 con 0x7f0e3815bfc0 2024-04-02T13:30:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.836+0000 7f0e401b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0e380b5f60 con 0x7f0e380cff60 2024-04-02T13:30:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.836+0000 7f0e3d750640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0e3806d1e0 0x7f0e380b5210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37840/0 (socket says 172.21.15.44:37840) 2024-04-02T13:30:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.836+0000 7f0e3d750640 1 -- 172.21.15.44:0/3110285100 learned_addr learned my addr 172.21.15.44:0/3110285100 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.836+0000 7f0e3e752640 1 --2- 172.21.15.44:0/3110285100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380cff60 0x7f0e380d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.836+0000 7f0e3df51640 1 --2- 172.21.15.44:0/3110285100 >> v2:172.21.15.69:3300/0 conn(0x7f0e3815bfc0 0x7f0e38072cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.836+0000 7f0e3cf4f640 1 -- 172.21.15.44:0/3110285100 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2091969354 0 0) 0x7f0e380b5c80 con 0x7f0e3806d1e0 2024-04-02T13:30:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.836+0000 7f0e3cf4f640 1 -- 172.21.15.44:0/3110285100 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e24003520 con 0x7f0e3806d1e0 2024-04-02T13:30:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.837+0000 7f0e3df51640 1 -- 172.21.15.44:0/3110285100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380cff60 msgr2=0x7f0e380d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.837+0000 7f0e3df51640 1 --2- 172.21.15.44:0/3110285100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380cff60 0x7f0e380d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.837+0000 7f0e3df51640 1 -- 172.21.15.44:0/3110285100 >> v1:172.21.15.44:6789/0 conn(0x7f0e3806d1e0 legacy=0x7f0e380b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.837+0000 7f0e3df51640 1 -- 172.21.15.44:0/3110285100 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e380724b0 con 0x7f0e3815bfc0 2024-04-02T13:30:53.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.837+0000 7f0e3df51640 1 --2- 172.21.15.44:0/3110285100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3815bfc0 0x7f0e38072cb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0e280097a0 tx=0x7f0e2802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8df40803e8a6ba4f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:53.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.838+0000 7f0e3cf4f640 1 -- 172.21.15.44:0/3110285100 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e28035020 con 0x7f0e3815bfc0 2024-04-02T13:30:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.838+0000 7f0e3cf4f640 1 -- 172.21.15.44:0/3110285100 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e28031830 con 0x7f0e3815bfc0 2024-04-02T13:30:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.838+0000 7f0e3cf4f640 1 -- 172.21.15.44:0/3110285100 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e28031be0 con 0x7f0e3815bfc0 2024-04-02T13:30:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.838+0000 7f0e401b5640 1 -- 172.21.15.44:0/3110285100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3815bfc0 msgr2=0x7f0e38072cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.838+0000 7f0e401b5640 1 --2- 172.21.15.44:0/3110285100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3815bfc0 0x7f0e38072cb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0e280097a0 tx=0x7f0e2802b8d0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.838+0000 7f0e401b5640 1 -- 172.21.15.44:0/3110285100 shutdown_connections 2024-04-02T13:30:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.838+0000 7f0e401b5640 1 --2- 172.21.15.44:0/3110285100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380cff60 0x7f0e380d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.838+0000 7f0e401b5640 1 --2- 172.21.15.44:0/3110285100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3815bfc0 0x7f0e38072cb0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.838+0000 7f0e401b5640 1 -- 172.21.15.44:0/3110285100 >> 172.21.15.44:0/3110285100 conn(0x7f0e3806c5e0 msgr2=0x7f0e3806c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.838+0000 7f0e401b5640 1 -- 172.21.15.44:0/3110285100 shutdown_connections 2024-04-02T13:30:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.838+0000 7f0e401b5640 1 -- 172.21.15.44:0/3110285100 wait complete. 2024-04-02T13:30:53.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.839+0000 7f0e401b5640 1 Processor -- start 2024-04-02T13:30:53.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.840+0000 7f0e401b5640 1 -- start start 2024-04-02T13:30:53.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.840+0000 7f0e401b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3806d1e0 0x7f0e38146230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.840+0000 7f0e401b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380cff60 0x7f0e38149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.840+0000 7f0e401b5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e3815bfc0 0x7f0e38147f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.840+0000 7f0e401b5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3806b4d0 con 0x7f0e3815bfc0 2024-04-02T13:30:53.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.840+0000 7f0e401b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0e380b5920 con 0x7f0e3806d1e0 2024-04-02T13:30:53.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.840+0000 7f0e401b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0e3806b650 con 0x7f0e380cff60 2024-04-02T13:30:53.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.840+0000 7f0e3e752640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e3815bfc0 0x7f0e38147f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.840+0000 7f0e3e752640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e3815bfc0 0x7f0e38147f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59016/0 (socket says 172.21.15.44:59016) 2024-04-02T13:30:53.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.840+0000 7f0e3e752640 1 -- 172.21.15.44:0/1516125081 learned_addr learned my addr 172.21.15.44:0/1516125081 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:53.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.840+0000 7f0e3df51640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3806d1e0 0x7f0e38146230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.840+0000 7f0e3d750640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380cff60 0x7f0e38149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.841+0000 7f0e3e752640 1 -- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380cff60 msgr2=0x7f0e38149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.841+0000 7f0e3e752640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380cff60 0x7f0e38149e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.841+0000 7f0e3e752640 1 -- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3806d1e0 msgr2=0x7f0e38146230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:53.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.841+0000 7f0e3e752640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3806d1e0 0x7f0e38146230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:53.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.841+0000 7f0e3e752640 1 -- 172.21.15.44:0/1516125081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e381e1590 con 0x7f0e3815bfc0 2024-04-02T13:30:53.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.841+0000 7f0e3e752640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e3815bfc0 0x7f0e38147f20 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000efc0 tx=0x7f0e3000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:53.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.841+0000 7f0e2e7fc640 1 -- 172.21.15.44:0/1516125081 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e30018020 con 0x7f0e3815bfc0 2024-04-02T13:30:53.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.841+0000 7f0e2e7fc640 1 -- 172.21.15.44:0/1516125081 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e30014480 con 0x7f0e3815bfc0 2024-04-02T13:30:53.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.841+0000 7f0e401b5640 1 -- 172.21.15.44:0/1516125081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e381e1870 con 0x7f0e3815bfc0 2024-04-02T13:30:53.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.841+0000 7f0e2e7fc640 1 -- 172.21.15.44:0/1516125081 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e30014870 con 0x7f0e3815bfc0 2024-04-02T13:30:53.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.841+0000 7f0e401b5640 1 -- 172.21.15.44:0/1516125081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e381e1e30 con 0x7f0e3815bfc0 2024-04-02T13:30:53.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.842+0000 7f0e401b5640 1 -- 172.21.15.44:0/1516125081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e38068c00 con 0x7f0e3815bfc0 2024-04-02T13:30:53.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.843+0000 7f0e2e7fc640 1 -- 172.21.15.44:0/1516125081 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0e30005d10 con 0x7f0e3815bfc0 2024-04-02T13:30:53.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.843+0000 7f0e2e7fc640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e1c06c8f0 0x7f0e1c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:53.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.843+0000 7f0e2e7fc640 1 -- 172.21.15.44:0/1516125081 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x7f0e3008c6f0 con 0x7f0e3815bfc0 2024-04-02T13:30:53.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.843+0000 7f0e3df51640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e1c06c8f0 0x7f0e1c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:53.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.844+0000 7f0e3df51640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e1c06c8f0 0x7f0e1c06edb0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0e28036000 tx=0x7f0e28008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:53.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:53.848+0000 7f0e2e7fc640 1 -- 172.21.15.44:0/1516125081 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0e30022070 con 0x7f0e3815bfc0 2024-04-02T13:30:54.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.071+0000 7f0e401b5640 1 -- 172.21.15.44:0/1516125081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f0e38068650 con 0x7f0e3815bfc0 2024-04-02T13:30:54.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.723+0000 7f0e2e7fc640 1 -- 172.21.15.44:0/1516125081 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v92) ==== 104+0+0 (secure 0 0 0) 0x7f0e30010040 con 0x7f0e3815bfc0 2024-04-02T13:30:54.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.726+0000 7f0e401b5640 1 -- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e1c06c8f0 msgr2=0x7f0e1c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:54.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.726+0000 7f0e401b5640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e1c06c8f0 0x7f0e1c06edb0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0e28036000 tx=0x7f0e28008030 comp rx=0 tx=0).stop 2024-04-02T13:30:54.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.726+0000 7f0e401b5640 1 -- 172.21.15.44:0/1516125081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e3815bfc0 msgr2=0x7f0e38147f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:54.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.726+0000 7f0e401b5640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e3815bfc0 0x7f0e38147f20 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000efc0 tx=0x7f0e3000c490 comp rx=0 tx=0).stop 2024-04-02T13:30:54.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.726+0000 7f0e401b5640 1 -- 172.21.15.44:0/1516125081 shutdown_connections 2024-04-02T13:30:54.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.726+0000 7f0e401b5640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e1c06c8f0 0x7f0e1c06edb0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:54.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.726+0000 7f0e401b5640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e3815bfc0 0x7f0e38147f20 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:54.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.726+0000 7f0e401b5640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380cff60 0x7f0e38149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:54.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.726+0000 7f0e401b5640 1 --2- 172.21.15.44:0/1516125081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3806d1e0 0x7f0e38146230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:54.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.726+0000 7f0e401b5640 1 -- 172.21.15.44:0/1516125081 >> 172.21.15.44:0/1516125081 conn(0x7f0e3806c5e0 msgr2=0x7f0e380cd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:54.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.726+0000 7f0e401b5640 1 -- 172.21.15.44:0/1516125081 shutdown_connections 2024-04-02T13:30:54.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.726+0000 7f0e401b5640 1 -- 172.21.15.44:0/1516125081 wait complete. 2024-04-02T13:30:54.743 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T13:30:54.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.851+0000 7f3032932640 1 Processor -- start 2024-04-02T13:30:54.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.851+0000 7f3032932640 1 -- start start 2024-04-02T13:30:54.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f3032932640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f302c1565d0 0x7f302c0d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:54.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f3032932640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f302c0d05e0 0x7f302c0ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:54.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f3032932640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0cf400 0x7f302c0cc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:54.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f3032932640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f302c0cf890 con 0x7f302c1565d0 2024-04-02T13:30:54.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f3032932640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f302c0cfad0 con 0x7f302c0d05e0 2024-04-02T13:30:54.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f3032932640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f302c160b10 con 0x7f302c0cf400 2024-04-02T13:30:54.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f302bfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f302c1565d0 0x7f302c0d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:54.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f302bfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f302c1565d0 0x7f302c0d00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43288/0 (socket says 172.21.15.44:43288) 2024-04-02T13:30:54.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f302bfff640 1 -- 172.21.15.44:0/4145323081 learned_addr learned my addr 172.21.15.44:0/4145323081 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:54.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f302bfff640 1 -- 172.21.15.44:0/4145323081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0cf400 msgr2=0x7f302c0cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:54.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f302bfff640 1 --2- 172.21.15.44:0/4145323081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0cf400 0x7f302c0cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:54.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f302bfff640 1 -- 172.21.15.44:0/4145323081 >> v2:172.21.15.69:3300/0 conn(0x7f302c0d05e0 msgr2=0x7f302c0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:54.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f302bfff640 1 --2- 172.21.15.44:0/4145323081 >> v2:172.21.15.69:3300/0 conn(0x7f302c0d05e0 0x7f302c0ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:54.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.852+0000 7f302bfff640 1 -- 172.21.15.44:0/4145323081 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f302c15fd50 con 0x7f302c1565d0 2024-04-02T13:30:54.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.853+0000 7f302bfff640 1 --2- 172.21.15.44:0/4145323081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f302c1565d0 0x7f302c0d00a0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f302c0b2a30 tx=0x7f302002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568f76ec6f558a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:54.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.853+0000 7f302affd640 1 -- 172.21.15.44:0/4145323081 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302002e070 con 0x7f302c1565d0 2024-04-02T13:30:54.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.853+0000 7f302affd640 1 -- 172.21.15.44:0/4145323081 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3020037070 con 0x7f302c1565d0 2024-04-02T13:30:54.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.853+0000 7f302affd640 1 -- 172.21.15.44:0/4145323081 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020031a00 con 0x7f302c1565d0 2024-04-02T13:30:54.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.854+0000 7f3032932640 1 -- 172.21.15.44:0/4145323081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f302c1565d0 msgr2=0x7f302c0d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:54.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.854+0000 7f3032932640 1 --2- 172.21.15.44:0/4145323081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f302c1565d0 0x7f302c0d00a0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f302c0b2a30 tx=0x7f302002b880 comp rx=0 tx=0).stop 2024-04-02T13:30:54.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.854+0000 7f3032932640 1 -- 172.21.15.44:0/4145323081 shutdown_connections 2024-04-02T13:30:54.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.854+0000 7f3032932640 1 --2- 172.21.15.44:0/4145323081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0cf400 0x7f302c0cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:54.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.854+0000 7f3032932640 1 --2- 172.21.15.44:0/4145323081 >> v2:172.21.15.69:3300/0 conn(0x7f302c0d05e0 0x7f302c0ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:54.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.854+0000 7f3032932640 1 --2- 172.21.15.44:0/4145323081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f302c1565d0 0x7f302c0d00a0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:54.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.854+0000 7f3032932640 1 -- 172.21.15.44:0/4145323081 >> 172.21.15.44:0/4145323081 conn(0x7f302c06b920 msgr2=0x7f302c06bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:54.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.854+0000 7f3032932640 1 -- 172.21.15.44:0/4145323081 shutdown_connections 2024-04-02T13:30:54.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.854+0000 7f3032932640 1 -- 172.21.15.44:0/4145323081 wait complete. 2024-04-02T13:30:54.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.855+0000 7f3032932640 1 Processor -- start 2024-04-02T13:30:54.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.855+0000 7f3032932640 1 -- start start 2024-04-02T13:30:54.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f3032932640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f302c0cf400 0x7f302c1552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:54.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f3032932640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0d05e0 0x7f302c0cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:54.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f3032932640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c1565d0 0x7f302c0b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:54.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f3032932640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f302c1605f0 con 0x7f302c0cf400 2024-04-02T13:30:54.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f3032932640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f302c160310 con 0x7f302c1565d0 2024-04-02T13:30:54.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f3032932640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f302c1569b0 con 0x7f302c0d05e0 2024-04-02T13:30:54.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f302bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f302c0cf400 0x7f302c1552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:54.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f302bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f302c0cf400 0x7f302c1552f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43300/0 (socket says 172.21.15.44:43300) 2024-04-02T13:30:54.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f302bfff640 1 -- 172.21.15.44:0/1905747838 learned_addr learned my addr 172.21.15.44:0/1905747838 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:54.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f3030ecf640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c1565d0 0x7f302c0b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:54.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f302b7fe640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0d05e0 0x7f302c0cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:54.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f302bfff640 1 -- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0d05e0 msgr2=0x7f302c0cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:54.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f302bfff640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0d05e0 0x7f302c0cc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:54.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f302bfff640 1 -- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c1565d0 msgr2=0x7f302c0b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:54.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f302bfff640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c1565d0 0x7f302c0b3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:54.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f302bfff640 1 -- 172.21.15.44:0/1905747838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f302c159990 con 0x7f302c0cf400 2024-04-02T13:30:54.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f302b7fe640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0d05e0 0x7f302c0cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:54.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.856+0000 7f302bfff640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f302c0cf400 0x7f302c1552f0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f302002b8c0 tx=0x7f30200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:54.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.857+0000 7f3028ff9640 1 -- 172.21.15.44:0/1905747838 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020032880 con 0x7f302c0cf400 2024-04-02T13:30:54.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.857+0000 7f3028ff9640 1 -- 172.21.15.44:0/1905747838 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3020032a20 con 0x7f302c0cf400 2024-04-02T13:30:54.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.857+0000 7f3032932640 1 -- 172.21.15.44:0/1905747838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f302c06d860 con 0x7f302c0cf400 2024-04-02T13:30:54.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.857+0000 7f3028ff9640 1 -- 172.21.15.44:0/1905747838 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020032e10 con 0x7f302c0cf400 2024-04-02T13:30:54.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.857+0000 7f3032932640 1 -- 172.21.15.44:0/1905747838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f302c06dde0 con 0x7f302c0cf400 2024-04-02T13:30:54.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.858+0000 7f3032932640 1 -- 172.21.15.44:0/1905747838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f302c0684f0 con 0x7f302c0cf400 2024-04-02T13:30:54.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.858+0000 7f3028ff9640 1 -- 172.21.15.44:0/1905747838 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f302004c060 con 0x7f302c0cf400 2024-04-02T13:30:54.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.859+0000 7f3028ff9640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f301806c8f0 0x7f301806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:54.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.859+0000 7f3028ff9640 1 -- 172.21.15.44:0/1905747838 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6360+0+0 (secure 0 0 0) 0x7f30200ada80 con 0x7f302c0cf400 2024-04-02T13:30:54.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.859+0000 7f302b7fe640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f301806c8f0 0x7f301806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:54.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.859+0000 7f302b7fe640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f301806c8f0 0x7f301806edb0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f302c15b850 tx=0x7f301c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:54.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:54.863+0000 7f3028ff9640 1 -- 172.21.15.44:0/1905747838 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f30200afe20 con 0x7f302c0cf400 2024-04-02T13:30:55.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.046+0000 7f3032932640 1 -- 172.21.15.44:0/1905747838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f302c0b4760 con 0x7f302c0cf400 2024-04-02T13:30:55.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.726+0000 7f3028ff9640 1 -- 172.21.15.44:0/1905747838 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v111) ==== 181+0+0 (secure 0 0 0) 0x7f3020040070 con 0x7f302c0cf400 2024-04-02T13:30:55.730 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' removed 2024-04-02T13:30:55.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.730+0000 7f3032932640 1 -- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f301806c8f0 msgr2=0x7f301806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:55.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.730+0000 7f3032932640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f301806c8f0 0x7f301806edb0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f302c15b850 tx=0x7f301c008040 comp rx=0 tx=0).stop 2024-04-02T13:30:55.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.731+0000 7f3032932640 1 -- 172.21.15.44:0/1905747838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f302c0cf400 msgr2=0x7f302c1552f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:55.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.731+0000 7f3032932640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f302c0cf400 0x7f302c1552f0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f302002b8c0 tx=0x7f30200046c0 comp rx=0 tx=0).stop 2024-04-02T13:30:55.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.732+0000 7f3032932640 1 -- 172.21.15.44:0/1905747838 shutdown_connections 2024-04-02T13:30:55.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.732+0000 7f3032932640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f301806c8f0 0x7f301806edb0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:55.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.732+0000 7f3032932640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c1565d0 0x7f302c0b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:55.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.732+0000 7f3032932640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0d05e0 0x7f302c0cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:55.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.732+0000 7f3032932640 1 --2- 172.21.15.44:0/1905747838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f302c0cf400 0x7f302c1552f0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:55.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.732+0000 7f3032932640 1 -- 172.21.15.44:0/1905747838 >> 172.21.15.44:0/1905747838 conn(0x7f302c06b920 msgr2=0x7f302c06bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:55.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.732+0000 7f3032932640 1 -- 172.21.15.44:0/1905747838 shutdown_connections 2024-04-02T13:30:55.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.732+0000 7f3032932640 1 -- 172.21.15.44:0/1905747838 wait complete. 2024-04-02T13:30:55.750 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T13:30:55.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.854+0000 7f6cfb0f4640 1 Processor -- start 2024-04-02T13:30:55.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.854+0000 7f6cfb0f4640 1 -- start start 2024-04-02T13:30:55.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.855+0000 7f6cfb0f4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6cf415e1d0 0x7f6cf40d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.855+0000 7f6cfb0f4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf40d1640 con 0x7f6cf40cec60 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.855+0000 7f6cfb0f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf40cf1f0 con 0x7f6cf40d2dd0 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.855+0000 7f6cfb0f4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf40cfe00 con 0x7f6cf415e1d0 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.855+0000 7f6cf3fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6cf40cec60 0x7f6cf40d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51104/0 (socket says 172.21.15.44:51104) 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.855+0000 7f6cf3fff640 1 -- 172.21.15.44:0/3928670236 learned_addr learned my addr 172.21.15.44:0/3928670236 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.855+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3954263074 0 0) 0x7f6cf40d1640 con 0x7f6cf40cec60 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.855+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cdc003520 con 0x7f6cf40cec60 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf8e90640 1 --2- 172.21.15.44:0/3928670236 >> v2:172.21.15.69:3300/0 conn(0x7f6cf415e1d0 0x7f6cf40d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1370556550 0 0) 0x7f6cdc003520 con 0x7f6cf40cec60 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 >> v2:172.21.15.69:3300/0 conn(0x7f6cf415e1d0 msgr2=0x7f6cf40d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf37fe640 1 --2- 172.21.15.44:0/3928670236 >> v2:172.21.15.69:3300/0 conn(0x7f6cf415e1d0 0x7f6cf40d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 >> v1:172.21.15.69:6789/0 conn(0x7f6cf40d2dd0 legacy=0x7f6cf40b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf40cf4d0 con 0x7f6cf40cec60 2024-04-02T13:30:55.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f6ce8001670 con 0x7f6cf40cec60 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 >> v1:172.21.15.44:6789/0 conn(0x7f6cf40cec60 legacy=0x7f6cf40d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf37fe640 1 --2- 172.21.15.44:0/3928670236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cdc0039b0 0x7f6cdc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf37fe640 1 --2- 172.21.15.44:0/3928670236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc004800 0x7f6cdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf37fe640 1 --2- 172.21.15.44:0/3928670236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0051c0 0x7f6cdc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf40cfe00 con 0x7f6cdc0039b0 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc00c160 con 0x7f6cdc0051c0 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6cdc00c360 con 0x7f6cdc004800 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf8e90640 1 --2- 172.21.15.44:0/3928670236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cdc0039b0 0x7f6cdc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf9691640 1 --2- 172.21.15.44:0/3928670236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0051c0 0x7f6cdc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf3fff640 1 --2- 172.21.15.44:0/3928670236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc004800 0x7f6cdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf8e90640 1 -- 172.21.15.44:0/3928670236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc004800 msgr2=0x7f6cdc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf8e90640 1 --2- 172.21.15.44:0/3928670236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc004800 0x7f6cdc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf8e90640 1 -- 172.21.15.44:0/3928670236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0051c0 msgr2=0x7f6cdc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:55.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf8e90640 1 --2- 172.21.15.44:0/3928670236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0051c0 0x7f6cdc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf8e90640 1 -- 172.21.15.44:0/3928670236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc003520 con 0x7f6cdc0039b0 2024-04-02T13:30:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf9691640 1 --2- 172.21.15.44:0/3928670236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0051c0 0x7f6cdc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.856+0000 7f6cf9691640 1 -- 172.21.15.44:0/3928670236 reap_dead start 2024-04-02T13:30:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.857+0000 7f6cf8e90640 1 --2- 172.21.15.44:0/3928670236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cdc0039b0 0x7f6cdc003dd0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4009ac0 tx=0x7f6cdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf4c46dd5457020 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.857+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4039070 con 0x7f6cdc0039b0 2024-04-02T13:30:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.857+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce402c6b0 con 0x7f6cdc0039b0 2024-04-02T13:30:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.857+0000 7f6cf37fe640 1 -- 172.21.15.44:0/3928670236 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4009690 con 0x7f6cdc0039b0 2024-04-02T13:30:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.858+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/3928670236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cdc0039b0 msgr2=0x7f6cdc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.858+0000 7f6cfb0f4640 1 --2- 172.21.15.44:0/3928670236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cdc0039b0 0x7f6cdc003dd0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4009ac0 tx=0x7f6cdc005830 comp rx=0 tx=0).stop 2024-04-02T13:30:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.858+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/3928670236 shutdown_connections 2024-04-02T13:30:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.858+0000 7f6cfb0f4640 1 --2- 172.21.15.44:0/3928670236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cdc0039b0 0x7f6cdc003dd0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:55.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.858+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/3928670236 >> 172.21.15.44:0/3928670236 conn(0x7f6cf409d3b0 msgr2=0x7f6cf40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:55.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.858+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/3928670236 shutdown_connections 2024-04-02T13:30:55.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.858+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/3928670236 wait complete. 2024-04-02T13:30:55.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.859+0000 7f6cfb0f4640 1 Processor -- start 2024-04-02T13:30:55.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.859+0000 7f6cfb0f4640 1 -- start start 2024-04-02T13:30:55.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cfb0f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0039b0 0x7f6cf40b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:55.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cfb0f4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cf40caaf0 0x7f6cf40a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:55.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cfb0f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cf409e840 0x7f6cf409ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cfb0f4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf415e5b0 con 0x7f6cf40caaf0 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cfb0f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf40cfa90 con 0x7f6cdc0039b0 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cfb0f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf415e730 con 0x7f6cf409e840 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cf3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cf40caaf0 0x7f6cf40a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cf3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cf40caaf0 0x7f6cf40a16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43318/0 (socket says 172.21.15.44:43318) 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cf3fff640 1 -- 172.21.15.44:0/2965958545 learned_addr learned my addr 172.21.15.44:0/2965958545 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cf8e90640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0039b0 0x7f6cf40b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cf9691640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cf409e840 0x7f6cf409ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cf3fff640 1 -- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cf409e840 msgr2=0x7f6cf409ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cf3fff640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cf409e840 0x7f6cf409ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cf3fff640 1 -- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0039b0 msgr2=0x7f6cf40b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cf3fff640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0039b0 0x7f6cf40b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cf3fff640 1 -- 172.21.15.44:0/2965958545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf415ea90 con 0x7f6cf40caaf0 2024-04-02T13:30:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.860+0000 7f6cf3fff640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cf40caaf0 0x7f6cf40a16d0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800e820 tx=0x7f6ce800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.861+0000 7f6cf17fa640 1 -- 172.21.15.44:0/2965958545 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce800ee70 con 0x7f6cf40caaf0 2024-04-02T13:30:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.861+0000 7f6cf17fa640 1 -- 172.21.15.44:0/2965958545 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce8018070 con 0x7f6cf40caaf0 2024-04-02T13:30:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.861+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/2965958545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf41f5120 con 0x7f6cf40caaf0 2024-04-02T13:30:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.861+0000 7f6cf17fa640 1 -- 172.21.15.44:0/2965958545 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce800cad0 con 0x7f6cf40caaf0 2024-04-02T13:30:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.861+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/2965958545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf41f56e0 con 0x7f6cf40caaf0 2024-04-02T13:30:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.862+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/2965958545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf40684f0 con 0x7f6cf40caaf0 2024-04-02T13:30:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.862+0000 7f6cf17fa640 1 -- 172.21.15.44:0/2965958545 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ce800cc70 con 0x7f6cf40caaf0 2024-04-02T13:30:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.862+0000 7f6cf17fa640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6ce006c8f0 0x7f6ce006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.862+0000 7f6cf17fa640 1 -- 172.21.15.44:0/2965958545 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5922+0+0 (secure 0 0 0) 0x7f6ce808b9e0 con 0x7f6cf40caaf0 2024-04-02T13:30:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.863+0000 7f6cf8e90640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6ce006c8f0 0x7f6ce006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.863+0000 7f6cf8e90640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6ce006c8f0 0x7f6ce006edb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4036000 tx=0x7f6ce4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:55.867+0000 7f6cf17fa640 1 -- 172.21.15.44:0/2965958545 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6ce8059d30 con 0x7f6cf40caaf0 2024-04-02T13:30:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.050+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/2965958545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6cf40d47f0 con 0x7f6cf40caaf0 2024-04-02T13:30:56.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.731+0000 7f6cf17fa640 1 -- 172.21.15.44:0/2965958545 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v112) ==== 169+0+0 (secure 0 0 0) 0x7f6ce8059630 con 0x7f6cf40caaf0 2024-04-02T13:30:56.736 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' removed 2024-04-02T13:30:56.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.735+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6ce006c8f0 msgr2=0x7f6ce006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:56.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.735+0000 7f6cfb0f4640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6ce006c8f0 0x7f6ce006edb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4036000 tx=0x7f6ce4030000 comp rx=0 tx=0).stop 2024-04-02T13:30:56.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.735+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/2965958545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cf40caaf0 msgr2=0x7f6cf40a16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:56.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.735+0000 7f6cfb0f4640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cf40caaf0 0x7f6cf40a16d0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800e820 tx=0x7f6ce800ec20 comp rx=0 tx=0).stop 2024-04-02T13:30:56.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.735+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/2965958545 shutdown_connections 2024-04-02T13:30:56.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.735+0000 7f6cfb0f4640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6ce006c8f0 0x7f6ce006edb0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:56.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.735+0000 7f6cfb0f4640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cf409e840 0x7f6cf409ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:56.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.735+0000 7f6cfb0f4640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cf40caaf0 0x7f6cf40a16d0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:56.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.735+0000 7f6cfb0f4640 1 --2- 172.21.15.44:0/2965958545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0039b0 0x7f6cf40b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:56.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.735+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/2965958545 >> 172.21.15.44:0/2965958545 conn(0x7f6cf409d3b0 msgr2=0x7f6cf40d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:56.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.735+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/2965958545 shutdown_connections 2024-04-02T13:30:56.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.735+0000 7f6cfb0f4640 1 -- 172.21.15.44:0/2965958545 wait complete. 2024-04-02T13:30:56.754 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T13:30:56.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.860+0000 7f88c67c8640 1 Processor -- start 2024-04-02T13:30:56.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.860+0000 7f88c67c8640 1 -- start start 2024-04-02T13:30:56.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88c67c8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f88c015e260 0x7f88c00d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:56.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88c67c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88c00cec60 0x7f88c0168280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:56.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88c67c8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88c00cf120 con 0x7f88c015e640 2024-04-02T13:30:56.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88c67c8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f88c00b5a50 con 0x7f88c015e260 2024-04-02T13:30:56.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88c67c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f88c00d01c0 con 0x7f88c00cec60 2024-04-02T13:30:56.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88bf7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f88c015e640 0x7f88c00d0ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51110/0 (socket says 172.21.15.44:51110) 2024-04-02T13:30:56.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88bf7fe640 1 -- 172.21.15.44:0/976006487 learned_addr learned my addr 172.21.15.44:0/976006487 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:56.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1348388758 0 0) 0x7f88c00cf120 con 0x7f88c015e640 2024-04-02T13:30:56.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88ac003520 con 0x7f88c015e640 2024-04-02T13:30:56.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88bffff640 1 --2- 172.21.15.44:0/976006487 >> v2:172.21.15.69:3300/0 conn(0x7f88c015e260 0x7f88c00d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:56.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88c4d65640 1 --2- 172.21.15.44:0/976006487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88c00cec60 0x7f88c0168280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:56.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 484038867 0 0) 0x7f88ac003520 con 0x7f88c015e640 2024-04-02T13:30:56.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88c00cec60 msgr2=0x7f88c0168280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:56.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88beffd640 1 --2- 172.21.15.44:0/976006487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88c00cec60 0x7f88c0168280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:56.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 >> v2:172.21.15.69:3300/0 conn(0x7f88c015e260 msgr2=0x7f88c00d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:56.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88beffd640 1 --2- 172.21.15.44:0/976006487 >> v2:172.21.15.69:3300/0 conn(0x7f88c015e260 0x7f88c00d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:56.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88c00cf400 con 0x7f88c015e640 2024-04-02T13:30:56.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f88b0001670 con 0x7f88c015e640 2024-04-02T13:30:56.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88bffff640 1 --2- 172.21.15.44:0/976006487 >> v2:172.21.15.69:3300/0 conn(0x7f88c015e260 0x7f88c00d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:56.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.861+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 >> v1:172.21.15.44:6789/0 conn(0x7f88c015e640 legacy=0x7f88c00d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:56.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88beffd640 1 --2- 172.21.15.44:0/976006487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88ac0039b0 0x7f88ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:56.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88beffd640 1 --2- 172.21.15.44:0/976006487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ac004800 0x7f88ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:56.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88beffd640 1 --2- 172.21.15.44:0/976006487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88c00cec60 0x7f88ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:56.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f88c00b5a50 con 0x7f88ac004800 2024-04-02T13:30:56.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac00ba20 con 0x7f88ac0039b0 2024-04-02T13:30:56.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f88ac00bc20 con 0x7f88c00cec60 2024-04-02T13:30:56.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88bf7fe640 1 --2- 172.21.15.44:0/976006487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ac004800 0x7f88ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:56.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88bffff640 1 --2- 172.21.15.44:0/976006487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88ac0039b0 0x7f88ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:56.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88c4d65640 1 --2- 172.21.15.44:0/976006487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88c00cec60 0x7f88ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:56.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88bf7fe640 1 -- 172.21.15.44:0/976006487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88c00cec60 msgr2=0x7f88ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:56.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88bf7fe640 1 --2- 172.21.15.44:0/976006487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88c00cec60 0x7f88ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:56.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88bf7fe640 1 -- 172.21.15.44:0/976006487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88ac0039b0 msgr2=0x7f88ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:56.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88bf7fe640 1 --2- 172.21.15.44:0/976006487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88ac0039b0 0x7f88ac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:56.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.862+0000 7f88bf7fe640 1 -- 172.21.15.44:0/976006487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ac003520 con 0x7f88ac004800 2024-04-02T13:30:56.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.863+0000 7f88bf7fe640 1 --2- 172.21.15.44:0/976006487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ac004800 0x7f88ac004c80 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f88b0001eb0 tx=0x7f88ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4bab48c285f9139 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:56.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.863+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b0007d70 con 0x7f88ac004800 2024-04-02T13:30:56.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.863+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88b0030070 con 0x7f88ac004800 2024-04-02T13:30:56.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.863+0000 7f88beffd640 1 -- 172.21.15.44:0/976006487 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b0005d30 con 0x7f88ac004800 2024-04-02T13:30:56.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.863+0000 7f88c67c8640 1 -- 172.21.15.44:0/976006487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ac004800 msgr2=0x7f88ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:56.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.863+0000 7f88c67c8640 1 --2- 172.21.15.44:0/976006487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ac004800 0x7f88ac004c80 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f88b0001eb0 tx=0x7f88ac005830 comp rx=0 tx=0).stop 2024-04-02T13:30:56.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.863+0000 7f88c4d65640 1 -- 172.21.15.44:0/976006487 reap_dead start 2024-04-02T13:30:56.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.863+0000 7f88c67c8640 1 -- 172.21.15.44:0/976006487 shutdown_connections 2024-04-02T13:30:56.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.863+0000 7f88c67c8640 1 -- 172.21.15.44:0/976006487 >> 172.21.15.44:0/976006487 conn(0x7f88c009d3c0 msgr2=0x7f88c00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:56.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.863+0000 7f88c67c8640 1 -- 172.21.15.44:0/976006487 shutdown_connections 2024-04-02T13:30:56.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.864+0000 7f88c67c8640 1 -- 172.21.15.44:0/976006487 wait complete. 2024-04-02T13:30:56.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.865+0000 7f88c67c8640 1 Processor -- start 2024-04-02T13:30:56.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.865+0000 7f88c67c8640 1 -- start start 2024-04-02T13:30:56.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.865+0000 7f88c67c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ac004800 0x7f88c015fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:56.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.865+0000 7f88c67c8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88c00cb1b0 0x7f88c0161460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:56.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88c67c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88c00b5120 0x7f88c0152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:56.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88c67c8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f88c00cfca0 con 0x7f88ac004800 2024-04-02T13:30:56.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88c67c8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f88c00cf9c0 con 0x7f88c00cb1b0 2024-04-02T13:30:56.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88c67c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f88c00d11e0 con 0x7f88c00b5120 2024-04-02T13:30:56.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ac004800 0x7f88c015fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:56.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ac004800 0x7f88c015fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43330/0 (socket says 172.21.15.44:43330) 2024-04-02T13:30:56.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bffff640 1 -- 172.21.15.44:0/1685431613 learned_addr learned my addr 172.21.15.44:0/1685431613 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:56.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bf7fe640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88c00cb1b0 0x7f88c0161460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:56.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88c4d65640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88c00b5120 0x7f88c0152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:56.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bffff640 1 -- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88c00b5120 msgr2=0x7f88c0152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:56.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bffff640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88c00b5120 0x7f88c0152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:56.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bffff640 1 -- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88c00cb1b0 msgr2=0x7f88c0161460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:56.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bffff640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88c00cb1b0 0x7f88c0161460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:56.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bffff640 1 -- 172.21.15.44:0/1685431613 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88c01603e0 con 0x7f88ac004800 2024-04-02T13:30:56.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bf7fe640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88c00cb1b0 0x7f88c0161460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:56.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bffff640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ac004800 0x7f88c015fdc0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f88a80116d0 tx=0x7f88a8011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:56.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bcff9640 1 -- 172.21.15.44:0/1685431613 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8019070 con 0x7f88ac004800 2024-04-02T13:30:56.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bcff9640 1 -- 172.21.15.44:0/1685431613 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88a8004220 con 0x7f88ac004800 2024-04-02T13:30:56.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88c67c8640 1 -- 172.21.15.44:0/1685431613 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88c01606c0 con 0x7f88ac004800 2024-04-02T13:30:56.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88bcff9640 1 -- 172.21.15.44:0/1685431613 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a800f040 con 0x7f88ac004800 2024-04-02T13:30:56.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.866+0000 7f88c67c8640 1 -- 172.21.15.44:0/1685431613 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88c0160c80 con 0x7f88ac004800 2024-04-02T13:30:56.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.867+0000 7f88c67c8640 1 -- 172.21.15.44:0/1685431613 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88c00684f0 con 0x7f88ac004800 2024-04-02T13:30:56.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.868+0000 7f88bcff9640 1 -- 172.21.15.44:0/1685431613 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f88a800f6e0 con 0x7f88ac004800 2024-04-02T13:30:56.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.868+0000 7f88bcff9640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f88a006c8f0 0x7f88a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:56.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.868+0000 7f88bcff9640 1 -- 172.21.15.44:0/1685431613 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5421+0+0 (secure 0 0 0) 0x7f88a808bc00 con 0x7f88ac004800 2024-04-02T13:30:56.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.869+0000 7f88bf7fe640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f88a006c8f0 0x7f88a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:56.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.869+0000 7f88bf7fe640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f88a006c8f0 0x7f88a006edb0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f88b0034000 tx=0x7f88b0007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:56.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:56.873+0000 7f88bcff9640 1 -- 172.21.15.44:0/1685431613 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f88a8092080 con 0x7f88ac004800 2024-04-02T13:30:57.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.062+0000 7f88c67c8640 1 -- 172.21.15.44:0/1685431613 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f88c00d0d70 con 0x7f88ac004800 2024-04-02T13:30:57.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.065+0000 7f88bcff9640 1 -- 172.21.15.44:0/1685431613 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7f88a8059b00 con 0x7f88ac004800 2024-04-02T13:30:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.067+0000 7f88c67c8640 1 -- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f88a006c8f0 msgr2=0x7f88a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.067+0000 7f88c67c8640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f88a006c8f0 0x7f88a006edb0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f88b0034000 tx=0x7f88b0007000 comp rx=0 tx=0).stop 2024-04-02T13:30:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.067+0000 7f88c67c8640 1 -- 172.21.15.44:0/1685431613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ac004800 msgr2=0x7f88c015fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.067+0000 7f88c67c8640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ac004800 0x7f88c015fdc0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f88a80116d0 tx=0x7f88a8011b90 comp rx=0 tx=0).stop 2024-04-02T13:30:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.067+0000 7f88c67c8640 1 -- 172.21.15.44:0/1685431613 shutdown_connections 2024-04-02T13:30:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.067+0000 7f88c67c8640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f88a006c8f0 0x7f88a006edb0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.067+0000 7f88c67c8640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88c00b5120 0x7f88c0152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.067+0000 7f88c67c8640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88c00cb1b0 0x7f88c0161460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.067+0000 7f88c67c8640 1 --2- 172.21.15.44:0/1685431613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ac004800 0x7f88c015fdc0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.067+0000 7f88c67c8640 1 -- 172.21.15.44:0/1685431613 >> 172.21.15.44:0/1685431613 conn(0x7f88c009d3c0 msgr2=0x7f88c00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:57.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.067+0000 7f88c67c8640 1 -- 172.21.15.44:0/1685431613 shutdown_connections 2024-04-02T13:30:57.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.067+0000 7f88c67c8640 1 -- 172.21.15.44:0/1685431613 wait complete. 2024-04-02T13:30:57.082 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync' 2024-04-02T13:30:57.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.179+0000 7fc978ee4640 1 Processor -- start 2024-04-02T13:30:57.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.180+0000 7fc978ee4640 1 -- start start 2024-04-02T13:30:57.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.180+0000 7fc978ee4640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc97406e650 0x7fc97415c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:57.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.180+0000 7fc978ee4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc9740cc6e0 0x7fc9740ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:57.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.180+0000 7fc978ee4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9740cd020 con 0x7fc974159db0 2024-04-02T13:30:57.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.180+0000 7fc978ee4640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc974073240 con 0x7fc97406e650 2024-04-02T13:30:57.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.180+0000 7fc978ee4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc9740726c0 con 0x7fc9740cc6e0 2024-04-02T13:30:57.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.181+0000 7fc97259c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc974159db0 0x7fc974071e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51114/0 (socket says 172.21.15.44:51114) 2024-04-02T13:30:57.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.181+0000 7fc97259c640 1 -- 172.21.15.44:0/4162872947 learned_addr learned my addr 172.21.15.44:0/4162872947 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:57.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.181+0000 7fc971d9b640 1 --2- 172.21.15.44:0/4162872947 >> v2:172.21.15.44:3300/0 conn(0x7fc97406e650 0x7fc97415c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:57.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.181+0000 7fc972d9d640 1 --2- 172.21.15.44:0/4162872947 >> v2:172.21.15.69:3300/0 conn(0x7fc9740cc6e0 0x7fc9740ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:57.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.181+0000 7fc97159a640 1 -- 172.21.15.44:0/4162872947 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 964705523 0 0) 0x7fc9740cd020 con 0x7fc974159db0 2024-04-02T13:30:57.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.181+0000 7fc97159a640 1 -- 172.21.15.44:0/4162872947 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc960003520 con 0x7fc974159db0 2024-04-02T13:30:57.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.181+0000 7fc971d9b640 1 -- 172.21.15.44:0/4162872947 >> v2:172.21.15.69:3300/0 conn(0x7fc9740cc6e0 msgr2=0x7fc9740ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:57.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.181+0000 7fc971d9b640 1 --2- 172.21.15.44:0/4162872947 >> v2:172.21.15.69:3300/0 conn(0x7fc9740cc6e0 0x7fc9740ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.181+0000 7fc971d9b640 1 -- 172.21.15.44:0/4162872947 >> v1:172.21.15.44:6789/0 conn(0x7fc974159db0 legacy=0x7fc974071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:57.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.181+0000 7fc971d9b640 1 -- 172.21.15.44:0/4162872947 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9740729a0 con 0x7fc97406e650 2024-04-02T13:30:57.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.182+0000 7fc971d9b640 1 --2- 172.21.15.44:0/4162872947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc97406e650 0x7fc97415c6d0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fc9740aef40 tx=0x7fc95c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4482cde2bd5ebff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:57.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.182+0000 7fc97159a640 1 -- 172.21.15.44:0/4162872947 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c0315a0 con 0x7fc97406e650 2024-04-02T13:30:57.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.182+0000 7fc97159a640 1 -- 172.21.15.44:0/4162872947 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc95c037070 con 0x7fc97406e650 2024-04-02T13:30:57.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.182+0000 7fc97159a640 1 -- 172.21.15.44:0/4162872947 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c031960 con 0x7fc97406e650 2024-04-02T13:30:57.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.182+0000 7fc978ee4640 1 -- 172.21.15.44:0/4162872947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc97406e650 msgr2=0x7fc97415c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:57.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.182+0000 7fc978ee4640 1 --2- 172.21.15.44:0/4162872947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc97406e650 0x7fc97415c6d0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fc9740aef40 tx=0x7fc95c02b840 comp rx=0 tx=0).stop 2024-04-02T13:30:57.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.182+0000 7fc978ee4640 1 -- 172.21.15.44:0/4162872947 shutdown_connections 2024-04-02T13:30:57.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.182+0000 7fc978ee4640 1 --2- 172.21.15.44:0/4162872947 >> v2:172.21.15.69:3300/0 conn(0x7fc9740cc6e0 0x7fc9740ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.182+0000 7fc978ee4640 1 --2- 172.21.15.44:0/4162872947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc97406e650 0x7fc97415c6d0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.183+0000 7fc978ee4640 1 -- 172.21.15.44:0/4162872947 >> 172.21.15.44:0/4162872947 conn(0x7fc9740cfcd0 msgr2=0x7fc9740d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:57.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.183+0000 7fc978ee4640 1 -- 172.21.15.44:0/4162872947 shutdown_connections 2024-04-02T13:30:57.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.183+0000 7fc978ee4640 1 -- 172.21.15.44:0/4162872947 wait complete. 2024-04-02T13:30:57.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.184+0000 7fc978ee4640 1 Processor -- start 2024-04-02T13:30:57.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.184+0000 7fc978ee4640 1 -- start start 2024-04-02T13:30:57.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc978ee4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc97406e650 0x7fc97415f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:57.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc978ee4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9740cc6e0 0x7fc97414ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:57.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc978ee4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc974159db0 0x7fc974152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:57.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc978ee4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc974165cf0 con 0x7fc97406e650 2024-04-02T13:30:57.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc978ee4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc974072f60 con 0x7fc9740cc6e0 2024-04-02T13:30:57.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc978ee4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc974165e70 con 0x7fc974159db0 2024-04-02T13:30:57.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc97259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc97406e650 0x7fc97415f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:57.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc97259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc97406e650 0x7fc97415f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43342/0 (socket says 172.21.15.44:43342) 2024-04-02T13:30:57.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc97259c640 1 -- 172.21.15.44:0/2303699828 learned_addr learned my addr 172.21.15.44:0/2303699828 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:57.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc972d9d640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc974159db0 0x7fc974152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:57.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc971d9b640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9740cc6e0 0x7fc97414ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc97259c640 1 -- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc974159db0 msgr2=0x7fc974152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc97259c640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc974159db0 0x7fc974152700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc97259c640 1 -- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9740cc6e0 msgr2=0x7fc97414ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc97259c640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9740cc6e0 0x7fc97414ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc97259c640 1 -- 172.21.15.44:0/2303699828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc974157f50 con 0x7fc97406e650 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc97259c640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc97406e650 0x7fc97415f010 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fc96c00c3c0 tx=0x7fc96c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc96affd640 1 -- 172.21.15.44:0/2303699828 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c017070 con 0x7fc97406e650 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc96affd640 1 -- 172.21.15.44:0/2303699828 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc96c005ce0 con 0x7fc97406e650 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc978ee4640 1 -- 172.21.15.44:0/2303699828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc974158230 con 0x7fc97406e650 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc96affd640 1 -- 172.21.15.44:0/2303699828 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0072f0 con 0x7fc97406e650 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.185+0000 7fc978ee4640 1 -- 172.21.15.44:0/2303699828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc97414cfa0 con 0x7fc97406e650 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.186+0000 7fc978ee4640 1 -- 172.21.15.44:0/2303699828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc974074f20 con 0x7fc97406e650 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.187+0000 7fc96affd640 1 -- 172.21.15.44:0/2303699828 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc96c01a070 con 0x7fc97406e650 2024-04-02T13:30:57.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.187+0000 7fc96affd640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc95406c8f0 0x7fc95406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:57.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.187+0000 7fc96affd640 1 -- 172.21.15.44:0/2303699828 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5421+0+0 (secure 0 0 0) 0x7fc96c08ad70 con 0x7fc97406e650 2024-04-02T13:30:57.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.188+0000 7fc971d9b640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc95406c8f0 0x7fc95406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:57.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.188+0000 7fc971d9b640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc95406c8f0 0x7fc95406edb0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc95c036000 tx=0x7fc95c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:57.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.192+0000 7fc96affd640 1 -- 172.21.15.44:0/2303699828 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc96c059350 con 0x7fc97406e650 2024-04-02T13:30:57.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.447+0000 7fc978ee4640 1 -- 172.21.15.44:0/2303699828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]} v 0) -- 0x7fc9740684f0 con 0x7fc97406e650 2024-04-02T13:30:57.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.741+0000 7fc96affd640 1 -- 172.21.15.44:0/2303699828 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]}]=0 v496) ==== 146+0+0 (secure 0 0 0) 0x7fc96c058c50 con 0x7fc97406e650 2024-04-02T13:30:57.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.743+0000 7fc978ee4640 1 -- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc95406c8f0 msgr2=0x7fc95406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:57.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.743+0000 7fc978ee4640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc95406c8f0 0x7fc95406edb0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc95c036000 tx=0x7fc95c03d040 comp rx=0 tx=0).stop 2024-04-02T13:30:57.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.743+0000 7fc978ee4640 1 -- 172.21.15.44:0/2303699828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc97406e650 msgr2=0x7fc97415f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:57.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.743+0000 7fc978ee4640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc97406e650 0x7fc97415f010 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fc96c00c3c0 tx=0x7fc96c00c880 comp rx=0 tx=0).stop 2024-04-02T13:30:57.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.743+0000 7fc978ee4640 1 -- 172.21.15.44:0/2303699828 shutdown_connections 2024-04-02T13:30:57.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.743+0000 7fc978ee4640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc95406c8f0 0x7fc95406edb0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.743+0000 7fc978ee4640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc974159db0 0x7fc974152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.743+0000 7fc978ee4640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9740cc6e0 0x7fc97414ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.743+0000 7fc978ee4640 1 --2- 172.21.15.44:0/2303699828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc97406e650 0x7fc97415f010 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.743+0000 7fc978ee4640 1 -- 172.21.15.44:0/2303699828 >> 172.21.15.44:0/2303699828 conn(0x7fc9740cfcd0 msgr2=0x7fc9740b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:57.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.743+0000 7fc978ee4640 1 -- 172.21.15.44:0/2303699828 shutdown_connections 2024-04-02T13:30:57.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.743+0000 7fc978ee4640 1 -- 172.21.15.44:0/2303699828 wait complete. 2024-04-02T13:30:57.758 INFO:tasks.cephfs_test_runner:test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-04-02T13:30:57.759 INFO:tasks.cephfs_test_runner:Starting test: test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-02T13:30:57.759 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:30:57.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.858+0000 7f2d8530e640 1 Processor -- start 2024-04-02T13:30:57.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.858+0000 7f2d8530e640 1 -- start start 2024-04-02T13:30:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.858+0000 7f2d8530e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2d80159db0 0x7f2d8015c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.858+0000 7f2d8530e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d800ccdf0 0x7f2d80165cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.858+0000 7f2d8530e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d800726c0 con 0x7f2d80071e50 2024-04-02T13:30:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.858+0000 7f2d8530e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2d80073240 con 0x7f2d80159db0 2024-04-02T13:30:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.858+0000 7f2d8530e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2d800729a0 con 0x7f2d800ccdf0 2024-04-02T13:30:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.859+0000 7f2d7effd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2d80159db0 0x7f2d8015c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.859+0000 7f2d7f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d800ccdf0 0x7f2d80165cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:57.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.859+0000 7f2d7e7fc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2d80071e50 0x7f2d800cc6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:57326/0 (socket says 172.21.15.44:57326) 2024-04-02T13:30:57.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.859+0000 7f2d7e7fc640 1 -- 172.21.15.44:0/2609232540 learned_addr learned my addr 172.21.15.44:0/2609232540 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:57.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.859+0000 7f2d7f7fe640 1 -- 172.21.15.44:0/2609232540 >> v2:172.21.15.69:3300/0 conn(0x7f2d80159db0 msgr2=0x7f2d8015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:57.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.859+0000 7f2d7dffb640 1 -- 172.21.15.44:0/2609232540 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009337962 0 0) 0x7f2d800726c0 con 0x7f2d80071e50 2024-04-02T13:30:57.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.859+0000 7f2d7f7fe640 1 --2- 172.21.15.44:0/2609232540 >> v2:172.21.15.69:3300/0 conn(0x7f2d80159db0 0x7f2d8015c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.859+0000 7f2d7f7fe640 1 -- 172.21.15.44:0/2609232540 >> v1:172.21.15.69:6789/0 conn(0x7f2d80071e50 legacy=0x7f2d800cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:57.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.859+0000 7f2d7f7fe640 1 -- 172.21.15.44:0/2609232540 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d80072c80 con 0x7f2d800ccdf0 2024-04-02T13:30:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.860+0000 7f2d7f7fe640 1 --2- 172.21.15.44:0/2609232540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d800ccdf0 0x7f2d80165cf0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009d90 tx=0x7f2d7802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90612544f2297496 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.860+0000 7f2d7dffb640 1 -- 172.21.15.44:0/2609232540 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78039070 con 0x7f2d800ccdf0 2024-04-02T13:30:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.860+0000 7f2d7dffb640 1 -- 172.21.15.44:0/2609232540 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d7802cbc0 con 0x7f2d800ccdf0 2024-04-02T13:30:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.860+0000 7f2d7dffb640 1 -- 172.21.15.44:0/2609232540 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78034440 con 0x7f2d800ccdf0 2024-04-02T13:30:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.861+0000 7f2d8530e640 1 -- 172.21.15.44:0/2609232540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d800ccdf0 msgr2=0x7f2d80165cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.861+0000 7f2d8530e640 1 --2- 172.21.15.44:0/2609232540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d800ccdf0 0x7f2d80165cf0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009d90 tx=0x7f2d7802be10 comp rx=0 tx=0).stop 2024-04-02T13:30:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.861+0000 7f2d8530e640 1 -- 172.21.15.44:0/2609232540 shutdown_connections 2024-04-02T13:30:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.861+0000 7f2d8530e640 1 --2- 172.21.15.44:0/2609232540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d800ccdf0 0x7f2d80165cf0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.861+0000 7f2d8530e640 1 --2- 172.21.15.44:0/2609232540 >> v2:172.21.15.69:3300/0 conn(0x7f2d80159db0 0x7f2d8015c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.861+0000 7f2d8530e640 1 -- 172.21.15.44:0/2609232540 >> 172.21.15.44:0/2609232540 conn(0x7f2d800cfcd0 msgr2=0x7f2d800d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.861+0000 7f2d8530e640 1 -- 172.21.15.44:0/2609232540 shutdown_connections 2024-04-02T13:30:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.861+0000 7f2d8530e640 1 -- 172.21.15.44:0/2609232540 wait complete. 2024-04-02T13:30:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.862+0000 7f2d8530e640 1 Processor -- start 2024-04-02T13:30:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.862+0000 7f2d8530e640 1 -- start start 2024-04-02T13:30:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d8530e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d80071e50 0x7f2d800cb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d8530e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d800ccdf0 0x7f2d800b4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d8530e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d80159db0 0x7f2d8015b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d8530e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2d80072480 con 0x7f2d80159db0 2024-04-02T13:30:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d8530e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2d80072300 con 0x7f2d800ccdf0 2024-04-02T13:30:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d8530e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2d8006e650 con 0x7f2d80071e50 2024-04-02T13:30:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d7f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d80159db0 0x7f2d8015b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d7f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d80159db0 0x7f2d8015b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43344/0 (socket says 172.21.15.44:43344) 2024-04-02T13:30:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d7f7fe640 1 -- 172.21.15.44:0/3344178111 learned_addr learned my addr 172.21.15.44:0/3344178111 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d7effd640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d80071e50 0x7f2d800cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d7e7fc640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d800ccdf0 0x7f2d800b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d7f7fe640 1 -- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d80071e50 msgr2=0x7f2d800cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d7f7fe640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d80071e50 0x7f2d800cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d7f7fe640 1 -- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d800ccdf0 msgr2=0x7f2d800b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d7f7fe640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d800ccdf0 0x7f2d800b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d7f7fe640 1 -- 172.21.15.44:0/3344178111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d800cbd40 con 0x7f2d80159db0 2024-04-02T13:30:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.863+0000 7f2d7effd640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d80071e50 0x7f2d800cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.864+0000 7f2d7f7fe640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d80159db0 0x7f2d8015b680 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009d90 tx=0x7f2d78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.864+0000 7f2d677fe640 1 -- 172.21.15.44:0/3344178111 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78044020 con 0x7f2d80159db0 2024-04-02T13:30:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.864+0000 7f2d677fe640 1 -- 172.21.15.44:0/3344178111 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d7803a040 con 0x7f2d80159db0 2024-04-02T13:30:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.864+0000 7f2d677fe640 1 -- 172.21.15.44:0/3344178111 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78039050 con 0x7f2d80159db0 2024-04-02T13:30:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.864+0000 7f2d8530e640 1 -- 172.21.15.44:0/3344178111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d80158950 con 0x7f2d80159db0 2024-04-02T13:30:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.864+0000 7f2d8530e640 1 -- 172.21.15.44:0/3344178111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d80158f10 con 0x7f2d80159db0 2024-04-02T13:30:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.865+0000 7f2d8530e640 1 -- 172.21.15.44:0/3344178111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d800684f0 con 0x7f2d80159db0 2024-04-02T13:30:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.865+0000 7f2d677fe640 1 -- 172.21.15.44:0/3344178111 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d78003bb0 con 0x7f2d80159db0 2024-04-02T13:30:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.866+0000 7f2d677fe640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2d6c06c8f0 0x7f2d6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.866+0000 7f2d677fe640 1 -- 172.21.15.44:0/3344178111 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5421+0+0 (secure 0 0 0) 0x7f2d780adfb0 con 0x7f2d80159db0 2024-04-02T13:30:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.866+0000 7f2d7effd640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2d6c06c8f0 0x7f2d6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:57.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.866+0000 7f2d7effd640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2d6c06c8f0 0x7f2d6c06edb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2d800cc1a0 tx=0x7f2d70008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:57.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:57.870+0000 7f2d677fe640 1 -- 172.21.15.44:0/3344178111 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2d7804e060 con 0x7f2d80159db0 2024-04-02T13:30:58.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.054+0000 7f2d8530e640 1 -- 172.21.15.44:0/3344178111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d8006f050 con 0x7f2d80159db0 2024-04-02T13:30:58.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.055+0000 7f2d677fe640 1 -- 172.21.15.44:0/3344178111 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) ==== 74+0+13385 (secure 0 0 0) 0x7f2d7807c590 con 0x7f2d80159db0 2024-04-02T13:30:58.060 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:30:58.060 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":112,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:30:56.728411+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6834/1745009605":"2024-04-03T13:30:49.436959+0000","172.21.15.44:6833/1745009605":"2024-04-03T13:30:49.436959+0000","172.21.15.44:6834/3259420283":"2024-04-03T13:30:12.628183+0000","172.21.15.44:6833/3259420283":"2024-04-03T13:30:12.628183+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:30:58.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.057+0000 7f2d8530e640 1 -- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2d6c06c8f0 msgr2=0x7f2d6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:58.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.057+0000 7f2d8530e640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2d6c06c8f0 0x7f2d6c06edb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2d800cc1a0 tx=0x7f2d70008040 comp rx=0 tx=0).stop 2024-04-02T13:30:58.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.057+0000 7f2d8530e640 1 -- 172.21.15.44:0/3344178111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d80159db0 msgr2=0x7f2d8015b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:58.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.057+0000 7f2d8530e640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d80159db0 0x7f2d8015b680 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009d90 tx=0x7f2d78004a40 comp rx=0 tx=0).stop 2024-04-02T13:30:58.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.057+0000 7f2d8530e640 1 -- 172.21.15.44:0/3344178111 shutdown_connections 2024-04-02T13:30:58.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.057+0000 7f2d8530e640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2d6c06c8f0 0x7f2d6c06edb0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.057+0000 7f2d8530e640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d80159db0 0x7f2d8015b680 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.057+0000 7f2d8530e640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d800ccdf0 0x7f2d800b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.057+0000 7f2d8530e640 1 --2- 172.21.15.44:0/3344178111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d80071e50 0x7f2d800cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.057+0000 7f2d8530e640 1 -- 172.21.15.44:0/3344178111 >> 172.21.15.44:0/3344178111 conn(0x7f2d800cfcd0 msgr2=0x7f2d800cc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:58.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.057+0000 7f2d8530e640 1 -- 172.21.15.44:0/3344178111 shutdown_connections 2024-04-02T13:30:58.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.057+0000 7f2d8530e640 1 -- 172.21.15.44:0/3344178111 wait complete. 2024-04-02T13:30:58.072 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:30:58.073 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:30:58.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.173+0000 7f2425ac8640 1 Processor -- start 2024-04-02T13:30:58.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.173+0000 7f2425ac8640 1 -- start start 2024-04-02T13:30:58.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.174+0000 7f2425ac8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f242006daa0 0x7f24200b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.174+0000 7f2425ac8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f242006de80 con 0x7f24200cfdc0 2024-04-02T13:30:58.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.174+0000 7f2425ac8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24200cf000 con 0x7f24200d2e00 2024-04-02T13:30:58.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.174+0000 7f2425ac8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f24200cf930 con 0x7f242006daa0 2024-04-02T13:30:58.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.174+0000 7f241f7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f242006daa0 0x7f24200b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.174+0000 7f241effd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f24200cfdc0 0x7f24200d26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51116/0 (socket says 172.21.15.44:51116) 2024-04-02T13:30:58.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.174+0000 7f241effd640 1 -- 172.21.15.44:0/1243995699 learned_addr learned my addr 172.21.15.44:0/1243995699 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:58.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.174+0000 7f241f7fe640 1 -- 172.21.15.44:0/1243995699 >> v1:172.21.15.69:6789/0 conn(0x7f24200d2e00 legacy=0x7f242015dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:58.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.174+0000 7f241f7fe640 1 -- 172.21.15.44:0/1243995699 >> v1:172.21.15.44:6789/0 conn(0x7f24200cfdc0 legacy=0x7f24200d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:58.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.174+0000 7f241f7fe640 1 -- 172.21.15.44:0/1243995699 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24200cf2e0 con 0x7f242006daa0 2024-04-02T13:30:58.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.174+0000 7f241e7fc640 1 -- 172.21.15.44:0/1243995699 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3771388612 0 0) 0x7f242006de80 con 0x7f24200cfdc0 2024-04-02T13:30:58.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.175+0000 7f241f7fe640 1 --2- 172.21.15.44:0/1243995699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f242006daa0 0x7f24200b35b0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f24100097a0 tx=0x7f241002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65452b00ab015fc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:58.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.175+0000 7f241e7fc640 1 -- 172.21.15.44:0/1243995699 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2410035020 con 0x7f242006daa0 2024-04-02T13:30:58.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.175+0000 7f241e7fc640 1 -- 172.21.15.44:0/1243995699 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2410031830 con 0x7f242006daa0 2024-04-02T13:30:58.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.175+0000 7f241e7fc640 1 -- 172.21.15.44:0/1243995699 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2410031c00 con 0x7f242006daa0 2024-04-02T13:30:58.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.176+0000 7f2425ac8640 1 -- 172.21.15.44:0/1243995699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f242006daa0 msgr2=0x7f24200b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:58.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.176+0000 7f2425ac8640 1 --2- 172.21.15.44:0/1243995699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f242006daa0 0x7f24200b35b0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f24100097a0 tx=0x7f241002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.176+0000 7f2425ac8640 1 -- 172.21.15.44:0/1243995699 shutdown_connections 2024-04-02T13:30:58.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.176+0000 7f2425ac8640 1 --2- 172.21.15.44:0/1243995699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f242006daa0 0x7f24200b35b0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.176+0000 7f2425ac8640 1 -- 172.21.15.44:0/1243995699 >> 172.21.15.44:0/1243995699 conn(0x7f242009d430 msgr2=0x7f24200d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:58.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.176+0000 7f2425ac8640 1 -- 172.21.15.44:0/1243995699 shutdown_connections 2024-04-02T13:30:58.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.176+0000 7f2425ac8640 1 -- 172.21.15.44:0/1243995699 wait complete. 2024-04-02T13:30:58.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.177+0000 7f2425ac8640 1 Processor -- start 2024-04-02T13:30:58.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.177+0000 7f2425ac8640 1 -- start start 2024-04-02T13:30:58.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f2425ac8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f242006daa0 0x7f242015d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f2425ac8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24200cfdc0 0x7f24200a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f2425ac8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24200d2e00 0x7f24200cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f2425ac8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f24200ceda0 con 0x7f24200d2e00 2024-04-02T13:30:58.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f2425ac8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f24200cec20 con 0x7f242006daa0 2024-04-02T13:30:58.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f2425ac8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f24200d11c0 con 0x7f24200cfdc0 2024-04-02T13:30:58.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f241ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24200d2e00 0x7f24200cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f241ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24200d2e00 0x7f24200cd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43366/0 (socket says 172.21.15.44:43366) 2024-04-02T13:30:58.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f241ffff640 1 -- 172.21.15.44:0/1423552574 learned_addr learned my addr 172.21.15.44:0/1423552574 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:58.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f241f7fe640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f242006daa0 0x7f242015d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f241ffff640 1 -- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24200cfdc0 msgr2=0x7f24200a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:58.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f241effd640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24200cfdc0 0x7f24200a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f241ffff640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24200cfdc0 0x7f24200a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f241ffff640 1 -- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f242006daa0 msgr2=0x7f242015d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:58.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f241ffff640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f242006daa0 0x7f242015d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f241ffff640 1 -- 172.21.15.44:0/1423552574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24200b6f80 con 0x7f24200d2e00 2024-04-02T13:30:58.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.178+0000 7f241ffff640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24200d2e00 0x7f24200cd180 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f241800be60 tx=0x7f241800bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:58.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.179+0000 7f2407fff640 1 -- 172.21.15.44:0/1423552574 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418019070 con 0x7f24200d2e00 2024-04-02T13:30:58.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.179+0000 7f2407fff640 1 -- 172.21.15.44:0/1423552574 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2418014440 con 0x7f24200d2e00 2024-04-02T13:30:58.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.179+0000 7f2425ac8640 1 -- 172.21.15.44:0/1423552574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24200b7260 con 0x7f24200d2e00 2024-04-02T13:30:58.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.179+0000 7f2407fff640 1 -- 172.21.15.44:0/1423552574 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418014830 con 0x7f24200d2e00 2024-04-02T13:30:58.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.179+0000 7f2425ac8640 1 -- 172.21.15.44:0/1423552574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24200b7820 con 0x7f24200d2e00 2024-04-02T13:30:58.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.180+0000 7f2425ac8640 1 -- 172.21.15.44:0/1423552574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24200684f0 con 0x7f24200d2e00 2024-04-02T13:30:58.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.180+0000 7f2407fff640 1 -- 172.21.15.44:0/1423552574 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f241801c070 con 0x7f24200d2e00 2024-04-02T13:30:58.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.180+0000 7f2407fff640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f240c06c8f0 0x7f240c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.181+0000 7f2407fff640 1 -- 172.21.15.44:0/1423552574 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5421+0+0 (secure 0 0 0) 0x7f241808b930 con 0x7f24200d2e00 2024-04-02T13:30:58.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.181+0000 7f241f7fe640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f240c06c8f0 0x7f240c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.181+0000 7f241f7fe640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f240c06c8f0 0x7f240c06edb0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f2410036000 tx=0x7f2410008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:58.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.185+0000 7f2407fff640 1 -- 172.21.15.44:0/1423552574 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2418059f10 con 0x7f24200d2e00 2024-04-02T13:30:58.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.374+0000 7f2425ac8640 1 -- 172.21.15.44:0/1423552574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f24200d24d0 con 0x7f24200d2e00 2024-04-02T13:30:58.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.374+0000 7f2407fff640 1 -- 172.21.15.44:0/1423552574 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) ==== 93+0+10 (secure 0 0 0) 0x7f2418059810 con 0x7f24200d2e00 2024-04-02T13:30:58.379 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:30:58.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.376+0000 7f2425ac8640 1 -- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f240c06c8f0 msgr2=0x7f240c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:58.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.376+0000 7f2425ac8640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f240c06c8f0 0x7f240c06edb0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f2410036000 tx=0x7f2410008030 comp rx=0 tx=0).stop 2024-04-02T13:30:58.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.376+0000 7f2425ac8640 1 -- 172.21.15.44:0/1423552574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24200d2e00 msgr2=0x7f24200cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:58.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.376+0000 7f2425ac8640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24200d2e00 0x7f24200cd180 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f241800be60 tx=0x7f241800bf50 comp rx=0 tx=0).stop 2024-04-02T13:30:58.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.376+0000 7f2425ac8640 1 -- 172.21.15.44:0/1423552574 shutdown_connections 2024-04-02T13:30:58.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.376+0000 7f2425ac8640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f240c06c8f0 0x7f240c06edb0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.376+0000 7f2425ac8640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24200d2e00 0x7f24200cd180 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.376+0000 7f2425ac8640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24200cfdc0 0x7f24200a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.376+0000 7f2425ac8640 1 --2- 172.21.15.44:0/1423552574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f242006daa0 0x7f242015d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.376+0000 7f2425ac8640 1 -- 172.21.15.44:0/1423552574 >> 172.21.15.44:0/1423552574 conn(0x7f242009d430 msgr2=0x7f24200d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:58.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.376+0000 7f2425ac8640 1 -- 172.21.15.44:0/1423552574 shutdown_connections 2024-04-02T13:30:58.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.376+0000 7f2425ac8640 1 -- 172.21.15.44:0/1423552574 wait complete. 2024-04-02T13:30:58.391 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan' 2024-04-02T13:30:58.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.489+0000 7fb8e88eb640 1 Processor -- start 2024-04-02T13:30:58.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.489+0000 7fb8e88eb640 1 -- start start 2024-04-02T13:30:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.489+0000 7fb8e88eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8e415e4a0 0x7fb8e40aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.489+0000 7fb8e88eb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb8e406d550 0x7fb8e40d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.489+0000 7fb8e88eb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8e40cc600 con 0x7fb8e40d0c50 2024-04-02T13:30:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.489+0000 7fb8e88eb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb8e40cc8e0 con 0x7fb8e406d550 2024-04-02T13:30:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.489+0000 7fb8e88eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e40ccbc0 con 0x7fb8e415e4a0 2024-04-02T13:30:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8dbfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb8e40d0c50 0x7fb8e406ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51128/0 (socket says 172.21.15.44:51128) 2024-04-02T13:30:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8dbfff640 1 -- 172.21.15.44:0/1039188981 learned_addr learned my addr 172.21.15.44:0/1039188981 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:30:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3327508135 0 0) 0x7fb8e40cc600 con 0x7fb8e40d0c50 2024-04-02T13:30:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e259c640 1 --2- 172.21.15.44:0/1039188981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8e415e4a0 0x7fb8e40aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8d0003520 con 0x7fb8e40d0c50 2024-04-02T13:30:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e2d9d640 1 --2- 172.21.15.44:0/1039188981 >> v2:172.21.15.69:3300/0 conn(0x7fb8e406d550 0x7fb8e40d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 299833053 0 0) 0x7fb8d0003520 con 0x7fb8e40d0c50 2024-04-02T13:30:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8e415e4a0 msgr2=0x7fb8e40aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:58.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 --2- 172.21.15.44:0/1039188981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8e415e4a0 0x7fb8e40aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 >> v2:172.21.15.69:3300/0 conn(0x7fb8e406d550 msgr2=0x7fb8e40d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:58.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 --2- 172.21.15.44:0/1039188981 >> v2:172.21.15.69:3300/0 conn(0x7fb8e406d550 0x7fb8e40d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e40ccea0 con 0x7fb8e40d0c50 2024-04-02T13:30:58.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fb8cc001670 con 0x7fb8e40d0c50 2024-04-02T13:30:58.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e2d9d640 1 --2- 172.21.15.44:0/1039188981 >> v2:172.21.15.69:3300/0 conn(0x7fb8e406d550 0x7fb8e40d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:58.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 >> v1:172.21.15.44:6789/0 conn(0x7fb8e40d0c50 legacy=0x7fb8e406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:58.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 --2- 172.21.15.44:0/1039188981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d00039b0 0x7fb8d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 --2- 172.21.15.44:0/1039188981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8d0004800 0x7fb8d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 --2- 172.21.15.44:0/1039188981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8e415e4a0 0x7fb8d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e40cc8e0 con 0x7fb8d00039b0 2024-04-02T13:30:58.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e40ccbc0 con 0x7fb8d0004800 2024-04-02T13:30:58.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb8d000b9c0 con 0x7fb8e415e4a0 2024-04-02T13:30:58.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e259c640 1 --2- 172.21.15.44:0/1039188981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d00039b0 0x7fb8d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8dbfff640 1 --2- 172.21.15.44:0/1039188981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8d0004800 0x7fb8d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.490+0000 7fb8e2d9d640 1 --2- 172.21.15.44:0/1039188981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8e415e4a0 0x7fb8d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.491+0000 7fb8e259c640 1 -- 172.21.15.44:0/1039188981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8e415e4a0 msgr2=0x7fb8d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:58.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.491+0000 7fb8e259c640 1 --2- 172.21.15.44:0/1039188981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8e415e4a0 0x7fb8d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.491+0000 7fb8e259c640 1 -- 172.21.15.44:0/1039188981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8d0004800 msgr2=0x7fb8d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:58.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.491+0000 7fb8e259c640 1 --2- 172.21.15.44:0/1039188981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8d0004800 0x7fb8d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.491+0000 7fb8e259c640 1 -- 172.21.15.44:0/1039188981 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8d0003520 con 0x7fb8d00039b0 2024-04-02T13:30:58.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.491+0000 7fb8e259c640 1 --2- 172.21.15.44:0/1039188981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d00039b0 0x7fb8d0003dd0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4009ac0 tx=0x7fb8d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c6baf65d2315 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:58.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.491+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4039070 con 0x7fb8d00039b0 2024-04-02T13:30:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.491+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d402c6b0 con 0x7fb8d00039b0 2024-04-02T13:30:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.491+0000 7fb8e1d9b640 1 -- 172.21.15.44:0/1039188981 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4009690 con 0x7fb8d00039b0 2024-04-02T13:30:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.492+0000 7fb8e88eb640 1 -- 172.21.15.44:0/1039188981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d00039b0 msgr2=0x7fb8d0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.492+0000 7fb8e88eb640 1 --2- 172.21.15.44:0/1039188981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d00039b0 0x7fb8d0003dd0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4009ac0 tx=0x7fb8d0005830 comp rx=0 tx=0).stop 2024-04-02T13:30:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.492+0000 7fb8e2d9d640 1 -- 172.21.15.44:0/1039188981 reap_dead start 2024-04-02T13:30:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.492+0000 7fb8e88eb640 1 -- 172.21.15.44:0/1039188981 shutdown_connections 2024-04-02T13:30:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.492+0000 7fb8e88eb640 1 -- 172.21.15.44:0/1039188981 >> 172.21.15.44:0/1039188981 conn(0x7fb8e409aa20 msgr2=0x7fb8e409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.492+0000 7fb8e88eb640 1 -- 172.21.15.44:0/1039188981 shutdown_connections 2024-04-02T13:30:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.492+0000 7fb8e88eb640 1 -- 172.21.15.44:0/1039188981 wait complete. 2024-04-02T13:30:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.493+0000 7fb8e88eb640 1 Processor -- start 2024-04-02T13:30:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.493+0000 7fb8e88eb640 1 -- start start 2024-04-02T13:30:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e88eb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d00039b0 0x7fb8e40cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e88eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8e41584f0 0x7fb8e4157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e88eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e40b6f70 0x7fb8e40b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e88eb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e415e880 con 0x7fb8d00039b0 2024-04-02T13:30:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e88eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e40cd460 con 0x7fb8e40b6f70 2024-04-02T13:30:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e88eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e40aebf0 con 0x7fb8e41584f0 2024-04-02T13:30:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d00039b0 0x7fb8e40cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d00039b0 0x7fb8e40cb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43384/0 (socket says 172.21.15.44:43384) 2024-04-02T13:30:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e259c640 1 -- 172.21.15.44:0/20584412 learned_addr learned my addr 172.21.15.44:0/20584412 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e2d9d640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e40b6f70 0x7fb8e40b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8dbfff640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8e41584f0 0x7fb8e4157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e259c640 1 -- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8e41584f0 msgr2=0x7fb8e4157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e259c640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8e41584f0 0x7fb8e4157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e259c640 1 -- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e40b6f70 msgr2=0x7fb8e40b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e259c640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e40b6f70 0x7fb8e40b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e259c640 1 -- 172.21.15.44:0/20584412 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e40b67b0 con 0x7fb8d00039b0 2024-04-02T13:30:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.494+0000 7fb8e2d9d640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e40b6f70 0x7fb8e40b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:30:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.495+0000 7fb8e259c640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d00039b0 0x7fb8e40cb950 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4009be0 tx=0x7fb8d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.495+0000 7fb8daffd640 1 -- 172.21.15.44:0/20584412 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4042020 con 0x7fb8d00039b0 2024-04-02T13:30:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.495+0000 7fb8daffd640 1 -- 172.21.15.44:0/20584412 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d403a040 con 0x7fb8d00039b0 2024-04-02T13:30:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.495+0000 7fb8e88eb640 1 -- 172.21.15.44:0/20584412 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8e40b59a0 con 0x7fb8d00039b0 2024-04-02T13:30:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.495+0000 7fb8daffd640 1 -- 172.21.15.44:0/20584412 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4039070 con 0x7fb8d00039b0 2024-04-02T13:30:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.495+0000 7fb8e88eb640 1 -- 172.21.15.44:0/20584412 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8e4160df0 con 0x7fb8d00039b0 2024-04-02T13:30:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.496+0000 7fb8e88eb640 1 -- 172.21.15.44:0/20584412 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8e40684f0 con 0x7fb8d00039b0 2024-04-02T13:30:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.496+0000 7fb8daffd640 1 -- 172.21.15.44:0/20584412 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb8d4045020 con 0x7fb8d00039b0 2024-04-02T13:30:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.497+0000 7fb8daffd640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb8c406c8f0 0x7fb8c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.497+0000 7fb8daffd640 1 -- 172.21.15.44:0/20584412 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5421+0+0 (secure 0 0 0) 0x7fb8d40ac7b0 con 0x7fb8d00039b0 2024-04-02T13:30:58.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.497+0000 7fb8dbfff640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb8c406c8f0 0x7fb8c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:58.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.497+0000 7fb8dbfff640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb8c406c8f0 0x7fb8c406edb0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40cb430 tx=0x7fb8cc0078f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:58.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.501+0000 7fb8daffd640 1 -- 172.21.15.44:0/20584412 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb8d407ad90 con 0x7fb8d00039b0 2024-04-02T13:30:58.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:58.757+0000 7fb8e88eb640 1 -- 172.21.15.44:0/20584412 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]} v 0) -- 0x7fb8e40ce3c0 con 0x7fb8d00039b0 2024-04-02T13:30:59.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.746+0000 7fb8daffd640 1 -- 172.21.15.44:0/20584412 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]}]=0 v498) ==== 147+0+0 (secure 0 0 0) 0x7fb8d407a690 con 0x7fb8d00039b0 2024-04-02T13:30:59.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.750+0000 7fb8e88eb640 1 -- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb8c406c8f0 msgr2=0x7fb8c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:59.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.750+0000 7fb8e88eb640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb8c406c8f0 0x7fb8c406edb0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40cb430 tx=0x7fb8cc0078f0 comp rx=0 tx=0).stop 2024-04-02T13:30:59.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.750+0000 7fb8e88eb640 1 -- 172.21.15.44:0/20584412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d00039b0 msgr2=0x7fb8e40cb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:59.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.750+0000 7fb8e88eb640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d00039b0 0x7fb8e40cb950 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4009be0 tx=0x7fb8d4004730 comp rx=0 tx=0).stop 2024-04-02T13:30:59.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.750+0000 7fb8e88eb640 1 -- 172.21.15.44:0/20584412 shutdown_connections 2024-04-02T13:30:59.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.750+0000 7fb8e88eb640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb8c406c8f0 0x7fb8c406edb0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:59.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.750+0000 7fb8e88eb640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e40b6f70 0x7fb8e40b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:59.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.750+0000 7fb8e88eb640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8e41584f0 0x7fb8e4157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:59.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.750+0000 7fb8e88eb640 1 --2- 172.21.15.44:0/20584412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d00039b0 0x7fb8e40cb950 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:59.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.750+0000 7fb8e88eb640 1 -- 172.21.15.44:0/20584412 >> 172.21.15.44:0/20584412 conn(0x7fb8e409aa20 msgr2=0x7fb8e40b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:59.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.750+0000 7fb8e88eb640 1 -- 172.21.15.44:0/20584412 shutdown_connections 2024-04-02T13:30:59.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.750+0000 7fb8e88eb640 1 -- 172.21.15.44:0/20584412 wait complete. 2024-04-02T13:30:59.767 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T13:30:59.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.873+0000 7f35e65cd640 1 Processor -- start 2024-04-02T13:30:59.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.873+0000 7f35e65cd640 1 -- start start 2024-04-02T13:30:59.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.874+0000 7f35e65cd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f35e015e1d0 0x7f35e00d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:59.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.874+0000 7f35e65cd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f35e00cec60 0x7f35e00d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:59.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.874+0000 7f35e65cd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35e015e5b0 0x7f35e00d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:59.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.874+0000 7f35e65cd640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f35e00cfe00 con 0x7f35e015e1d0 2024-04-02T13:30:59.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.874+0000 7f35e65cd640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f35e00d1640 con 0x7f35e00cec60 2024-04-02T13:30:59.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.874+0000 7f35e65cd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f35e00cf1f0 con 0x7f35e015e5b0 2024-04-02T13:30:59.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.874+0000 7f35dffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f35e015e1d0 0x7f35e00d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.874+0000 7f35dffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f35e015e1d0 0x7f35e00d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43388/0 (socket says 172.21.15.44:43388) 2024-04-02T13:30:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.874+0000 7f35dffff640 1 -- 172.21.15.44:0/3007440942 learned_addr learned my addr 172.21.15.44:0/3007440942 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.875+0000 7f35e4b6a640 1 --2- 172.21.15.44:0/3007440942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35e015e5b0 0x7f35e00d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.875+0000 7f35df7fe640 1 --2- 172.21.15.44:0/3007440942 >> v2:172.21.15.69:3300/0 conn(0x7f35e00cec60 0x7f35e00d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.875+0000 7f35dffff640 1 -- 172.21.15.44:0/3007440942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35e015e5b0 msgr2=0x7f35e00d2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.875+0000 7f35dffff640 1 --2- 172.21.15.44:0/3007440942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35e015e5b0 0x7f35e00d2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.875+0000 7f35dffff640 1 -- 172.21.15.44:0/3007440942 >> v2:172.21.15.69:3300/0 conn(0x7f35e00cec60 msgr2=0x7f35e00d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:30:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.875+0000 7f35dffff640 1 --2- 172.21.15.44:0/3007440942 >> v2:172.21.15.69:3300/0 conn(0x7f35e00cec60 0x7f35e00d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.875+0000 7f35dffff640 1 -- 172.21.15.44:0/3007440942 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e00cf4d0 con 0x7f35e015e1d0 2024-04-02T13:30:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.875+0000 7f35dffff640 1 --2- 172.21.15.44:0/3007440942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35e015e1d0 0x7f35e00d1100 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f35e0161f20 tx=0x7f35c802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2078357ee7e788 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.875+0000 7f35deffd640 1 -- 172.21.15.44:0/3007440942 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c802e070 con 0x7f35e015e1d0 2024-04-02T13:30:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.875+0000 7f35deffd640 1 -- 172.21.15.44:0/3007440942 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c8037070 con 0x7f35e015e1d0 2024-04-02T13:30:59.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.876+0000 7f35deffd640 1 -- 172.21.15.44:0/3007440942 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8031a00 con 0x7f35e015e1d0 2024-04-02T13:30:59.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.876+0000 7f35e65cd640 1 -- 172.21.15.44:0/3007440942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35e015e1d0 msgr2=0x7f35e00d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:59.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.876+0000 7f35e65cd640 1 --2- 172.21.15.44:0/3007440942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35e015e1d0 0x7f35e00d1100 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f35e0161f20 tx=0x7f35c802b880 comp rx=0 tx=0).stop 2024-04-02T13:30:59.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.876+0000 7f35e65cd640 1 -- 172.21.15.44:0/3007440942 shutdown_connections 2024-04-02T13:30:59.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.876+0000 7f35e65cd640 1 --2- 172.21.15.44:0/3007440942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35e015e5b0 0x7f35e00d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:59.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.876+0000 7f35e65cd640 1 --2- 172.21.15.44:0/3007440942 >> v2:172.21.15.69:3300/0 conn(0x7f35e00cec60 0x7f35e00d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:59.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.876+0000 7f35e65cd640 1 --2- 172.21.15.44:0/3007440942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35e015e1d0 0x7f35e00d1100 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:59.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.876+0000 7f35e65cd640 1 -- 172.21.15.44:0/3007440942 >> 172.21.15.44:0/3007440942 conn(0x7f35e009d3b0 msgr2=0x7f35e00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:30:59.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.876+0000 7f35e65cd640 1 -- 172.21.15.44:0/3007440942 shutdown_connections 2024-04-02T13:30:59.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.876+0000 7f35e65cd640 1 -- 172.21.15.44:0/3007440942 wait complete. 2024-04-02T13:30:59.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.877+0000 7f35e65cd640 1 Processor -- start 2024-04-02T13:30:59.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.877+0000 7f35e65cd640 1 -- start start 2024-04-02T13:30:59.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.878+0000 7f35e65cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35e00cec60 0x7f35e01f4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:59.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.878+0000 7f35e65cd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e015e1d0 0x7f35e00d3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:59.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.878+0000 7f35e65cd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35e015e5b0 0x7f35e00d35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:59.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.878+0000 7f35e65cd640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f35e00b4ce0 con 0x7f35e00cec60 2024-04-02T13:30:59.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.878+0000 7f35e65cd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f35e00cfa90 con 0x7f35e015e1d0 2024-04-02T13:30:59.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.878+0000 7f35e65cd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f35e00b4e60 con 0x7f35e015e5b0 2024-04-02T13:30:59.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.878+0000 7f35dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35e00cec60 0x7f35e01f4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:59.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.878+0000 7f35dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35e00cec60 0x7f35e01f4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43396/0 (socket says 172.21.15.44:43396) 2024-04-02T13:30:59.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.878+0000 7f35dffff640 1 -- 172.21.15.44:0/3641333993 learned_addr learned my addr 172.21.15.44:0/3641333993 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:30:59.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.878+0000 7f35df7fe640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e015e1d0 0x7f35e00d3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:59.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.878+0000 7f35e4b6a640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35e015e5b0 0x7f35e00d35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:59.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.879+0000 7f35dffff640 1 -- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35e015e5b0 msgr2=0x7f35e00d35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.879+0000 7f35dffff640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35e015e5b0 0x7f35e00d35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.879+0000 7f35dffff640 1 -- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e015e1d0 msgr2=0x7f35e00d3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:30:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.879+0000 7f35dffff640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e015e1d0 0x7f35e00d3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:30:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.879+0000 7f35dffff640 1 -- 172.21.15.44:0/3641333993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e01f4170 con 0x7f35e00cec60 2024-04-02T13:30:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.879+0000 7f35dffff640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35e00cec60 0x7f35e01f4930 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f35c8000c00 tx=0x7f35c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.879+0000 7f35dcff9640 1 -- 172.21.15.44:0/3641333993 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8032880 con 0x7f35e00cec60 2024-04-02T13:30:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.879+0000 7f35dcff9640 1 -- 172.21.15.44:0/3641333993 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c8032a20 con 0x7f35e00cec60 2024-04-02T13:30:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.879+0000 7f35e65cd640 1 -- 172.21.15.44:0/3641333993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e0155c70 con 0x7f35e00cec60 2024-04-02T13:30:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.879+0000 7f35dcff9640 1 -- 172.21.15.44:0/3641333993 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8032e10 con 0x7f35e00cec60 2024-04-02T13:30:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.879+0000 7f35e65cd640 1 -- 172.21.15.44:0/3641333993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e0156230 con 0x7f35e00cec60 2024-04-02T13:30:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.880+0000 7f35e65cd640 1 -- 172.21.15.44:0/3641333993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35e0152ea0 con 0x7f35e00cec60 2024-04-02T13:30:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.880+0000 7f35dcff9640 1 -- 172.21.15.44:0/3641333993 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f35c804c060 con 0x7f35e00cec60 2024-04-02T13:30:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.881+0000 7f35dcff9640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35c006c8f0 0x7f35c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:30:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.881+0000 7f35dcff9640 1 -- 172.21.15.44:0/3641333993 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5421+0+0 (secure 0 0 0) 0x7f35c80ad600 con 0x7f35e00cec60 2024-04-02T13:30:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.881+0000 7f35df7fe640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35c006c8f0 0x7f35c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:30:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.882+0000 7f35df7fe640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35c006c8f0 0x7f35c006edb0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f35e0161ac0 tx=0x7f35d0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:30:59.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:30:59.886+0000 7f35dcff9640 1 -- 172.21.15.44:0/3641333993 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f35c8040070 con 0x7f35e00cec60 2024-04-02T13:31:00.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.070+0000 7f35e65cd640 1 -- 172.21.15.44:0/3641333993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f35e00d47f0 con 0x7f35e00cec60 2024-04-02T13:31:00.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.074+0000 7f35dcff9640 1 -- 172.21.15.44:0/3641333993 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) ==== 122+0+0 (secure 0 0 0) 0x7f35c807bd00 con 0x7f35e00cec60 2024-04-02T13:31:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.076+0000 7f35e65cd640 1 -- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35c006c8f0 msgr2=0x7f35c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.076+0000 7f35e65cd640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35c006c8f0 0x7f35c006edb0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f35e0161ac0 tx=0x7f35d0008040 comp rx=0 tx=0).stop 2024-04-02T13:31:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.076+0000 7f35e65cd640 1 -- 172.21.15.44:0/3641333993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35e00cec60 msgr2=0x7f35e01f4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.076+0000 7f35e65cd640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35e00cec60 0x7f35e01f4930 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f35c8000c00 tx=0x7f35c80046c0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.076+0000 7f35e65cd640 1 -- 172.21.15.44:0/3641333993 shutdown_connections 2024-04-02T13:31:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.076+0000 7f35e65cd640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f35c006c8f0 0x7f35c006edb0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.076+0000 7f35e65cd640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35e015e5b0 0x7f35e00d35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.077+0000 7f35e65cd640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e015e1d0 0x7f35e00d3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.077+0000 7f35e65cd640 1 --2- 172.21.15.44:0/3641333993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35e00cec60 0x7f35e01f4930 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.077+0000 7f35e65cd640 1 -- 172.21.15.44:0/3641333993 >> 172.21.15.44:0/3641333993 conn(0x7f35e009d3b0 msgr2=0x7f35e015c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.077+0000 7f35e65cd640 1 -- 172.21.15.44:0/3641333993 shutdown_connections 2024-04-02T13:31:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.077+0000 7f35e65cd640 1 -- 172.21.15.44:0/3641333993 wait complete. 2024-04-02T13:31:00.092 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:31:00.092 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:31:00.136 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:31:00.182 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:31:00.182 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:31:00.182 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:31:00.197 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:31:00.197 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:31:00.197 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:31:00.214 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:31:00.214 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:31:00.214 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:31:00.252 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:31:00.252 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:31:00.252 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:31:00.271 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:31:00.272 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T13:31:00.272 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:31:00.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.389+0000 7ff1c69f6640 1 Processor -- start 2024-04-02T13:31:00.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.389+0000 7ff1c69f6640 1 -- start start 2024-04-02T13:31:00.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.390+0000 7ff1c69f6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff1c00b2d00 0x7ff1c00cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:00.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.390+0000 7ff1c69f6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1c00b3160 con 0x7ff1c015aa00 2024-04-02T13:31:00.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.390+0000 7ff1c69f6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1c015ade0 con 0x7ff1c00ce110 2024-04-02T13:31:00.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.390+0000 7ff1c69f6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff1c00af1b0 con 0x7ff1c00b2d00 2024-04-02T13:31:00.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.390+0000 7ff1bffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff1c015aa00 0x7ff1c0072bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51130/0 (socket says 172.21.15.44:51130) 2024-04-02T13:31:00.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.390+0000 7ff1bffff640 1 -- 172.21.15.44:0/3839168215 learned_addr learned my addr 172.21.15.44:0/3839168215 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:00.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 57137543 0 0) 0x7ff1c00b3160 con 0x7ff1c015aa00 2024-04-02T13:31:00.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1a8003520 con 0x7ff1c015aa00 2024-04-02T13:31:00.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1bf7fe640 1 --2- 172.21.15.44:0/3839168215 >> v2:172.21.15.69:3300/0 conn(0x7ff1c00b2d00 0x7ff1c00cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:00.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2340204042 0 0) 0x7ff1a8003520 con 0x7ff1c015aa00 2024-04-02T13:31:00.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 >> v2:172.21.15.69:3300/0 conn(0x7ff1c00b2d00 msgr2=0x7ff1c00cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:00.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 --2- 172.21.15.44:0/3839168215 >> v2:172.21.15.69:3300/0 conn(0x7ff1c00b2d00 0x7ff1c00cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 >> v1:172.21.15.69:6789/0 conn(0x7ff1c00ce110 legacy=0x7ff1c0071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:00.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c00af490 con 0x7ff1c015aa00 2024-04-02T13:31:00.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7ff1b0001670 con 0x7ff1c015aa00 2024-04-02T13:31:00.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 >> v1:172.21.15.44:6789/0 conn(0x7ff1c015aa00 legacy=0x7ff1c0072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:00.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 --2- 172.21.15.44:0/3839168215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a80039b0 0x7ff1a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:00.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 --2- 172.21.15.44:0/3839168215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a8004800 0x7ff1a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:00.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 --2- 172.21.15.44:0/3839168215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1a80051c0 0x7ff1a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:00.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c00af1b0 con 0x7ff1a80051c0 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a800c160 con 0x7ff1a8004800 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff1a800c360 con 0x7ff1a80039b0 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1c4f93640 1 --2- 172.21.15.44:0/3839168215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1a80051c0 0x7ff1a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1bf7fe640 1 --2- 172.21.15.44:0/3839168215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a8004800 0x7ff1a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1bffff640 1 --2- 172.21.15.44:0/3839168215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a80039b0 0x7ff1a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1c4f93640 1 -- 172.21.15.44:0/3839168215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a80039b0 msgr2=0x7ff1a8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1c4f93640 1 --2- 172.21.15.44:0/3839168215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a80039b0 0x7ff1a8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1c4f93640 1 -- 172.21.15.44:0/3839168215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a8004800 msgr2=0x7ff1a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1c4f93640 1 --2- 172.21.15.44:0/3839168215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a8004800 0x7ff1a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.391+0000 7ff1c4f93640 1 -- 172.21.15.44:0/3839168215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a8003520 con 0x7ff1a80051c0 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.392+0000 7ff1c4f93640 1 -- 172.21.15.44:0/3839168215 reap_dead start 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.392+0000 7ff1c4f93640 1 --2- 172.21.15.44:0/3839168215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1a80051c0 0x7ff1a800b9c0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8001a30 tx=0x7ff1a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ae08f8fae65ef8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.392+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b8001a70 con 0x7ff1a80051c0 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.392+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1b8003bb0 con 0x7ff1a80051c0 2024-04-02T13:31:00.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.392+0000 7ff1beffd640 1 -- 172.21.15.44:0/3839168215 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b80050d0 con 0x7ff1a80051c0 2024-04-02T13:31:00.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.393+0000 7ff1c69f6640 1 -- 172.21.15.44:0/3839168215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1a80051c0 msgr2=0x7ff1a800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:00.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.393+0000 7ff1c69f6640 1 --2- 172.21.15.44:0/3839168215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1a80051c0 0x7ff1a800b9c0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8001a30 tx=0x7ff1a8005830 comp rx=0 tx=0).stop 2024-04-02T13:31:00.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.393+0000 7ff1c69f6640 1 -- 172.21.15.44:0/3839168215 shutdown_connections 2024-04-02T13:31:00.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.393+0000 7ff1c69f6640 1 --2- 172.21.15.44:0/3839168215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1a80051c0 0x7ff1a800b9c0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.393+0000 7ff1c69f6640 1 -- 172.21.15.44:0/3839168215 >> 172.21.15.44:0/3839168215 conn(0x7ff1c00cfcd0 msgr2=0x7ff1c00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:00.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.393+0000 7ff1c69f6640 1 -- 172.21.15.44:0/3839168215 shutdown_connections 2024-04-02T13:31:00.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.393+0000 7ff1c69f6640 1 -- 172.21.15.44:0/3839168215 wait complete. 2024-04-02T13:31:00.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.394+0000 7ff1c69f6640 1 Processor -- start 2024-04-02T13:31:00.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.394+0000 7ff1c69f6640 1 -- start start 2024-04-02T13:31:00.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c69f6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a80051c0 0x7ff1c0150c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:00.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c69f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1c015d2b0 0x7ff1c015d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:00.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c69f6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1c009c600 0x7ff1c009ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:00.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c69f6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c00ce560 con 0x7ff1c009c600 2024-04-02T13:31:00.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c69f6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c00726d0 con 0x7ff1a80051c0 2024-04-02T13:31:00.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c69f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff1c00cdbd0 con 0x7ff1c015d2b0 2024-04-02T13:31:00.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c4f93640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1c009c600 0x7ff1c009ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:00.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c4f93640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1c009c600 0x7ff1c009ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43412/0 (socket says 172.21.15.44:43412) 2024-04-02T13:31:00.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c4f93640 1 -- 172.21.15.44:0/1550323786 learned_addr learned my addr 172.21.15.44:0/1550323786 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:00.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1bffff640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a80051c0 0x7ff1c0150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1bf7fe640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1c015d2b0 0x7ff1c015d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c4f93640 1 -- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1c015d2b0 msgr2=0x7ff1c015d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c4f93640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1c015d2b0 0x7ff1c015d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c4f93640 1 -- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a80051c0 msgr2=0x7ff1c0150c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c4f93640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a80051c0 0x7ff1c0150c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.395+0000 7ff1c4f93640 1 -- 172.21.15.44:0/1550323786 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c009aac0 con 0x7ff1c009c600 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.396+0000 7ff1c4f93640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1c009c600 0x7ff1c009ca80 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8001f40 tx=0x7ff1b80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.396+0000 7ff1bcff9640 1 -- 172.21.15.44:0/1550323786 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b8007db0 con 0x7ff1c009c600 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.396+0000 7ff1bcff9640 1 -- 172.21.15.44:0/1550323786 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1b8003970 con 0x7ff1c009c600 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.396+0000 7ff1c69f6640 1 -- 172.21.15.44:0/1550323786 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1c009bb20 con 0x7ff1c009c600 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.396+0000 7ff1bcff9640 1 -- 172.21.15.44:0/1550323786 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b802e600 con 0x7ff1c009c600 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.396+0000 7ff1c69f6640 1 -- 172.21.15.44:0/1550323786 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1c009b040 con 0x7ff1c009c600 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.397+0000 7ff1c69f6640 1 -- 172.21.15.44:0/1550323786 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1c0074f20 con 0x7ff1c009c600 2024-04-02T13:31:00.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.397+0000 7ff1bcff9640 1 -- 172.21.15.44:0/1550323786 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff1b802e7a0 con 0x7ff1c009c600 2024-04-02T13:31:00.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.398+0000 7ff1bcff9640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff1ac06c8f0 0x7ff1ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:00.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.398+0000 7ff1bcff9640 1 -- 172.21.15.44:0/1550323786 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5421+0+0 (secure 0 0 0) 0x7ff1b80aeae0 con 0x7ff1c009c600 2024-04-02T13:31:00.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.398+0000 7ff1bffff640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff1ac06c8f0 0x7ff1ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:00.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.398+0000 7ff1bffff640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff1ac06c8f0 0x7ff1ac06edb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff1c015cdc0 tx=0x7ff1b0000ed0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:00.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.403+0000 7ff1bcff9640 1 -- 172.21.15.44:0/1550323786 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff1b807d0f0 con 0x7ff1c009c600 2024-04-02T13:31:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.625+0000 7ff1c69f6640 1 -- 172.21.15.44:0/1550323786 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1c00ae4a0 con 0x7ff1c009c600 2024-04-02T13:31:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.625+0000 7ff1bcff9640 1 -- 172.21.15.44:0/1550323786 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) ==== 94+0+3869 (secure 0 0 0) 0x7ff1b807c9f0 con 0x7ff1c009c600 2024-04-02T13:31:00.629 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:00.629 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":92,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7696,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2715855087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2715855087},{"type":"v1","addr":"172.21.15.69:6836","nonce":2715855087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7699,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3850680553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3850680553},{"type":"v1","addr":"172.21.15.69:6837","nonce":3850680553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/1330712383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1330712383},{"type":"v1","addr":"172.21.15.44:6835","nonce":1330712383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":9348,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/3518070032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":3518070032},{"type":"v1","addr":"172.21.15.44:6834","nonce":3518070032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[]} 2024-04-02T13:31:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 92 2024-04-02T13:31:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.627+0000 7ff1c69f6640 1 -- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff1ac06c8f0 msgr2=0x7ff1ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.627+0000 7ff1c69f6640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff1ac06c8f0 0x7ff1ac06edb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff1c015cdc0 tx=0x7ff1b0000ed0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.627+0000 7ff1c69f6640 1 -- 172.21.15.44:0/1550323786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1c009c600 msgr2=0x7ff1c009ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.627+0000 7ff1c69f6640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1c009c600 0x7ff1c009ca80 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8001f40 tx=0x7ff1b80014a0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.627+0000 7ff1c69f6640 1 -- 172.21.15.44:0/1550323786 shutdown_connections 2024-04-02T13:31:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.627+0000 7ff1c69f6640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff1ac06c8f0 0x7ff1ac06edb0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.627+0000 7ff1c69f6640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1c009c600 0x7ff1c009ca80 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.627+0000 7ff1c69f6640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1c015d2b0 0x7ff1c015d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.627+0000 7ff1c69f6640 1 --2- 172.21.15.44:0/1550323786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a80051c0 0x7ff1c0150c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.627+0000 7ff1c69f6640 1 -- 172.21.15.44:0/1550323786 >> 172.21.15.44:0/1550323786 conn(0x7ff1c00cfcd0 msgr2=0x7ff1c00b4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.627+0000 7ff1c69f6640 1 -- 172.21.15.44:0/1550323786 shutdown_connections 2024-04-02T13:31:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:00.627+0000 7ff1c69f6640 1 -- 172.21.15.44:0/1550323786 wait complete. 2024-04-02T13:31:00.641 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T13:31:00.641 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T13:31:00.642 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T13:31:00.642 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:31:00.642 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T13:31:00.642 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T13:31:00.642 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T13:31:00.642 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:31:00.642 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T13:31:00.643 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T13:31:00.643 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T13:31:00.643 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:31:00.643 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T13:31:00.643 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T13:31:00.643 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T13:31:00.643 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:31:00.650 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T13:31:00.651 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T13:31:00.652 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T13:31:00.652 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T13:31:00.655 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T13:31:00.655 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T13:31:00.693 INFO:tasks.ceph.mds.a:Started 2024-04-02T13:31:00.693 DEBUG:teuthology.parallel:result is None 2024-04-02T13:31:00.694 INFO:tasks.ceph.mds.c:Started 2024-04-02T13:31:00.694 DEBUG:teuthology.parallel:result is None 2024-04-02T13:31:00.697 INFO:tasks.ceph.mds.d:Started 2024-04-02T13:31:00.698 DEBUG:teuthology.parallel:result is None 2024-04-02T13:31:00.769 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:31:00.769 INFO:tasks.ceph.mds.c.smithi044.stdout:starting mds.c at 2024-04-02T13:31:00.780 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-04-02T13:31:03.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-04-02T13:31:06.655 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T13:31:06.655 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T13:31:06.658 INFO:tasks.ceph.mds.b:Started 2024-04-02T13:31:06.658 DEBUG:teuthology.parallel:result is None 2024-04-02T13:31:06.659 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T13:31:06.741 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-04-02T13:31:06.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.771+0000 7fb09ea0d640 1 Processor -- start 2024-04-02T13:31:06.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.771+0000 7fb09ea0d640 1 -- start start 2024-04-02T13:31:06.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.771+0000 7fb09ea0d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb09815e4a0 0x7fb0980aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.771+0000 7fb09ea0d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0980d0c50 0x7fb0980d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.771+0000 7fb09ea0d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0980cc600 con 0x7fb09806ce40 2024-04-02T13:31:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.771+0000 7fb09ea0d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb0980cc8e0 con 0x7fb09815e4a0 2024-04-02T13:31:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.771+0000 7fb09ea0d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb0980ccbc0 con 0x7fb0980d0c50 2024-04-02T13:31:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb09cfaa640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb09806ce40 0x7fb0980cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48660/0 (socket says 172.21.15.44:48660) 2024-04-02T13:31:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb09cfaa640 1 -- 172.21.15.44:0/218736059 learned_addr learned my addr 172.21.15.44:0/218736059 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb097fff640 1 --2- 172.21.15.44:0/218736059 >> v2:172.21.15.69:3300/0 conn(0x7fb09815e4a0 0x7fb0980aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb0977fe640 1 --2- 172.21.15.44:0/218736059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0980d0c50 0x7fb0980d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1517428096 0 0) 0x7fb0980cc600 con 0x7fb09806ce40 2024-04-02T13:31:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb080003520 con 0x7fb09806ce40 2024-04-02T13:31:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 991582022 0 0) 0x7fb080003520 con 0x7fb09806ce40 2024-04-02T13:31:06.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0980d0c50 msgr2=0x7fb0980d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 --2- 172.21.15.44:0/218736059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0980d0c50 0x7fb0980d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 >> v2:172.21.15.69:3300/0 conn(0x7fb09815e4a0 msgr2=0x7fb0980aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 --2- 172.21.15.44:0/218736059 >> v2:172.21.15.69:3300/0 conn(0x7fb09815e4a0 0x7fb0980aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0980ccea0 con 0x7fb09806ce40 2024-04-02T13:31:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fb0900019f0 con 0x7fb09806ce40 2024-04-02T13:31:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 >> v1:172.21.15.44:6789/0 conn(0x7fb09806ce40 legacy=0x7fb0980cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 --2- 172.21.15.44:0/218736059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0800039b0 0x7fb080003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 --2- 172.21.15.44:0/218736059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb080004800 0x7fb080004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 --2- 172.21.15.44:0/218736059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0980d0c50 0x7fb0800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb0980cc8e0 con 0x7fb080004800 2024-04-02T13:31:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb0980ccbc0 con 0x7fb0800039b0 2024-04-02T13:31:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb08000b9c0 con 0x7fb0980d0c50 2024-04-02T13:31:06.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb0977fe640 1 --2- 172.21.15.44:0/218736059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb080004800 0x7fb080004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.772+0000 7fb097fff640 1 --2- 172.21.15.44:0/218736059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0800039b0 0x7fb080003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.773+0000 7fb09cfaa640 1 --2- 172.21.15.44:0/218736059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0980d0c50 0x7fb0800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.773+0000 7fb097fff640 1 -- 172.21.15.44:0/218736059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0980d0c50 msgr2=0x7fb0800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.773+0000 7fb097fff640 1 --2- 172.21.15.44:0/218736059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0980d0c50 0x7fb0800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.773+0000 7fb097fff640 1 -- 172.21.15.44:0/218736059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb080004800 msgr2=0x7fb080004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.773+0000 7fb097fff640 1 --2- 172.21.15.44:0/218736059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb080004800 0x7fb080004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.773+0000 7fb097fff640 1 -- 172.21.15.44:0/218736059 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb080003520 con 0x7fb0800039b0 2024-04-02T13:31:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.773+0000 7fb0977fe640 1 --2- 172.21.15.44:0/218736059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb080004800 0x7fb080004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:31:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.774+0000 7fb097fff640 1 --2- 172.21.15.44:0/218736059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0800039b0 0x7fb080003dd0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb088009ac0 tx=0x7fb080005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bc6eb24b1d5bf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.774+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb088039070 con 0x7fb0800039b0 2024-04-02T13:31:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.774+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb08802c710 con 0x7fb0800039b0 2024-04-02T13:31:06.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.774+0000 7fb096ffd640 1 -- 172.21.15.44:0/218736059 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb088009690 con 0x7fb0800039b0 2024-04-02T13:31:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.774+0000 7fb09ea0d640 1 -- 172.21.15.44:0/218736059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0800039b0 msgr2=0x7fb080003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.774+0000 7fb09ea0d640 1 --2- 172.21.15.44:0/218736059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0800039b0 0x7fb080003dd0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb088009ac0 tx=0x7fb080005830 comp rx=0 tx=0).stop 2024-04-02T13:31:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.774+0000 7fb09cfaa640 1 -- 172.21.15.44:0/218736059 reap_dead start 2024-04-02T13:31:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.774+0000 7fb09ea0d640 1 -- 172.21.15.44:0/218736059 shutdown_connections 2024-04-02T13:31:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.774+0000 7fb09ea0d640 1 -- 172.21.15.44:0/218736059 >> 172.21.15.44:0/218736059 conn(0x7fb09809aa20 msgr2=0x7fb09809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.774+0000 7fb09ea0d640 1 -- 172.21.15.44:0/218736059 shutdown_connections 2024-04-02T13:31:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.774+0000 7fb09ea0d640 1 -- 172.21.15.44:0/218736059 wait complete. 2024-04-02T13:31:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.775+0000 7fb09ea0d640 1 Processor -- start 2024-04-02T13:31:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.776+0000 7fb09ea0d640 1 -- start start 2024-04-02T13:31:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.776+0000 7fb09ea0d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0800039b0 0x7fb098154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:06.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.776+0000 7fb09ea0d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0980ce9d0 0x7fb0980cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.776+0000 7fb09ea0d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09815fce0 0x7fb098160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.776+0000 7fb09ea0d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb09815e880 con 0x7fb0980ce9d0 2024-04-02T13:31:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.776+0000 7fb09ea0d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb0980cd460 con 0x7fb0800039b0 2024-04-02T13:31:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.776+0000 7fb09ea0d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb0980af130 con 0x7fb09815fce0 2024-04-02T13:31:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb0977fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0980ce9d0 0x7fb0980cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb0977fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0980ce9d0 0x7fb0980cee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44478/0 (socket says 172.21.15.44:44478) 2024-04-02T13:31:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb0977fe640 1 -- 172.21.15.44:0/3823194293 learned_addr learned my addr 172.21.15.44:0/3823194293 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb097fff640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0800039b0 0x7fb098154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb09cfaa640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09815fce0 0x7fb098160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb0977fe640 1 -- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09815fce0 msgr2=0x7fb098160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb0977fe640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09815fce0 0x7fb098160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:06.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb0977fe640 1 -- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0800039b0 msgr2=0x7fb098154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb0977fe640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0800039b0 0x7fb098154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb0977fe640 1 -- 172.21.15.44:0/3823194293 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0980b4bd0 con 0x7fb0980ce9d0 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb0977fe640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0980ce9d0 0x7fb0980cee30 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fb08c011a50 tx=0x7fb08c011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb094ff9640 1 -- 172.21.15.44:0/3823194293 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c00fa70 con 0x7fb0980ce9d0 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb094ff9640 1 -- 172.21.15.44:0/3823194293 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb08c00fc10 con 0x7fb0980ce9d0 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb09ea0d640 1 -- 172.21.15.44:0/3823194293 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0980b4eb0 con 0x7fb0980ce9d0 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb094ff9640 1 -- 172.21.15.44:0/3823194293 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c014440 con 0x7fb0980ce9d0 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.777+0000 7fb09ea0d640 1 -- 172.21.15.44:0/3823194293 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0980b5470 con 0x7fb0980ce9d0 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.778+0000 7fb09ea0d640 1 -- 172.21.15.44:0/3823194293 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb09806c110 con 0x7fb0980ce9d0 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.779+0000 7fb094ff9640 1 -- 172.21.15.44:0/3823194293 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb08c012040 con 0x7fb0980ce9d0 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.779+0000 7fb094ff9640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb08406c8f0 0x7fb08406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.779+0000 7fb094ff9640 1 -- 172.21.15.44:0/3823194293 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5421+0+0 (secure 0 0 0) 0x7fb08c08c0c0 con 0x7fb0980ce9d0 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.779+0000 7fb097fff640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb08406c8f0 0x7fb08406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:06.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.780+0000 7fb097fff640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb08406c8f0 0x7fb08406edb0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb088036000 tx=0x7fb088030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:06.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.784+0000 7fb094ff9640 1 -- 172.21.15.44:0/3823194293 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb09806c110 con 0x7fb0980ce9d0 2024-04-02T13:31:06.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:06.979+0000 7fb09ea0d640 1 -- 172.21.15.44:0/3823194293 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fb0980684f0 con 0x7fb0980ce9d0 2024-04-02T13:31:08.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.093+0000 7fb094ff9640 1 -- 172.21.15.44:0/3823194293 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v117) ==== 97+0+0 (secure 0 0 0) 0x7fb08c019030 con 0x7fb0980ce9d0 2024-04-02T13:31:08.096 INFO:teuthology.orchestra.run.smithi044.stderr: removed all blocklist entries 2024-04-02T13:31:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.096+0000 7fb09ea0d640 1 -- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb08406c8f0 msgr2=0x7fb08406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.096+0000 7fb09ea0d640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb08406c8f0 0x7fb08406edb0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb088036000 tx=0x7fb088030000 comp rx=0 tx=0).stop 2024-04-02T13:31:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.096+0000 7fb09ea0d640 1 -- 172.21.15.44:0/3823194293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0980ce9d0 msgr2=0x7fb0980cee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.096+0000 7fb09ea0d640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0980ce9d0 0x7fb0980cee30 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fb08c011a50 tx=0x7fb08c011e50 comp rx=0 tx=0).stop 2024-04-02T13:31:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.097+0000 7fb09ea0d640 1 -- 172.21.15.44:0/3823194293 shutdown_connections 2024-04-02T13:31:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.097+0000 7fb09ea0d640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb08406c8f0 0x7fb08406edb0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.097+0000 7fb09ea0d640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09815fce0 0x7fb098160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.097+0000 7fb09ea0d640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0980ce9d0 0x7fb0980cee30 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.097+0000 7fb09ea0d640 1 --2- 172.21.15.44:0/3823194293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0800039b0 0x7fb098154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.097+0000 7fb09ea0d640 1 -- 172.21.15.44:0/3823194293 >> 172.21.15.44:0/3823194293 conn(0x7fb09809aa20 msgr2=0x7fb09809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:08.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.097+0000 7fb09ea0d640 1 -- 172.21.15.44:0/3823194293 shutdown_connections 2024-04-02T13:31:08.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.097+0000 7fb09ea0d640 1 -- 172.21.15.44:0/3823194293 wait complete. 2024-04-02T13:31:08.113 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T13:31:08.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.217+0000 7f6382e69640 1 Processor -- start 2024-04-02T13:31:08.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.218+0000 7f6382e69640 1 -- start start 2024-04-02T13:31:08.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.218+0000 7f6382e69640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f637c06cf90 0x7f637c0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.218+0000 7f6382e69640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f637c06c1d0 con 0x7f637c0cfe30 2024-04-02T13:31:08.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.218+0000 7f6382e69640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f637c06c4b0 con 0x7f637c0d0a30 2024-04-02T13:31:08.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.218+0000 7f6382e69640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f637c06cb00 con 0x7f637c06cf90 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.218+0000 7f6381406640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f637c0cfe30 0x7f637c0d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48676/0 (socket says 172.21.15.44:48676) 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.218+0000 7f6381406640 1 -- 172.21.15.44:0/4012374712 learned_addr learned my addr 172.21.15.44:0/4012374712 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637bfff640 1 --2- 172.21.15.44:0/4012374712 >> v2:172.21.15.69:3300/0 conn(0x7f637c06cf90 0x7f637c0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1204220841 0 0) 0x7f637c06c1d0 con 0x7f637c0cfe30 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6364003520 con 0x7f637c0cfe30 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1223166600 0 0) 0x7f6364003520 con 0x7f637c0cfe30 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 >> v2:172.21.15.69:3300/0 conn(0x7f637c06cf90 msgr2=0x7f637c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 --2- 172.21.15.44:0/4012374712 >> v2:172.21.15.69:3300/0 conn(0x7f637c06cf90 0x7f637c0cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 >> v1:172.21.15.69:6789/0 conn(0x7f637c0d0a30 legacy=0x7f637c0d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637c06c790 con 0x7f637c0cfe30 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f636c001670 con 0x7f637c0cfe30 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 >> v1:172.21.15.44:6789/0 conn(0x7f637c0cfe30 legacy=0x7f637c0d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 --2- 172.21.15.44:0/4012374712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63640039b0 0x7f6364003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 --2- 172.21.15.44:0/4012374712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6364004800 0x7f6364004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 --2- 172.21.15.44:0/4012374712 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63640051c0 0x7f636400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f637c06cb00 con 0x7f63640051c0 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f636400c160 con 0x7f6364004800 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f636400c360 con 0x7f63640039b0 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f6381406640 1 --2- 172.21.15.44:0/4012374712 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63640051c0 0x7f636400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f6380c05640 1 --2- 172.21.15.44:0/4012374712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63640039b0 0x7f6364003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.219+0000 7f637bfff640 1 --2- 172.21.15.44:0/4012374712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6364004800 0x7f6364004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.220+0000 7f6381406640 1 -- 172.21.15.44:0/4012374712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63640039b0 msgr2=0x7f6364003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.220+0000 7f6381406640 1 --2- 172.21.15.44:0/4012374712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63640039b0 0x7f6364003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.220+0000 7f6381406640 1 -- 172.21.15.44:0/4012374712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6364004800 msgr2=0x7f6364004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.220+0000 7f6381406640 1 --2- 172.21.15.44:0/4012374712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6364004800 0x7f6364004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.220+0000 7f6381406640 1 -- 172.21.15.44:0/4012374712 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6364003520 con 0x7f63640051c0 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.220+0000 7f6381406640 1 -- 172.21.15.44:0/4012374712 reap_dead start 2024-04-02T13:31:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.220+0000 7f6381406640 1 --2- 172.21.15.44:0/4012374712 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63640051c0 0x7f636400b9c0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f636c00c240 tx=0x7f6364005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7af13bfcfba112b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.220+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c007d10 con 0x7f63640051c0 2024-04-02T13:31:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.220+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f636c030070 con 0x7f63640051c0 2024-04-02T13:31:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.220+0000 7f637b7fe640 1 -- 172.21.15.44:0/4012374712 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c02cd20 con 0x7f63640051c0 2024-04-02T13:31:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.221+0000 7f6382e69640 1 -- 172.21.15.44:0/4012374712 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63640051c0 msgr2=0x7f636400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.221+0000 7f6382e69640 1 --2- 172.21.15.44:0/4012374712 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63640051c0 0x7f636400b9c0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f636c00c240 tx=0x7f6364005830 comp rx=0 tx=0).stop 2024-04-02T13:31:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.221+0000 7f6382e69640 1 -- 172.21.15.44:0/4012374712 shutdown_connections 2024-04-02T13:31:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.221+0000 7f6382e69640 1 --2- 172.21.15.44:0/4012374712 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63640051c0 0x7f636400b9c0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.221+0000 7f6382e69640 1 -- 172.21.15.44:0/4012374712 >> 172.21.15.44:0/4012374712 conn(0x7f637c09d470 msgr2=0x7f637c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.221+0000 7f6382e69640 1 -- 172.21.15.44:0/4012374712 shutdown_connections 2024-04-02T13:31:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.221+0000 7f6382e69640 1 -- 172.21.15.44:0/4012374712 wait complete. 2024-04-02T13:31:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.222+0000 7f6382e69640 1 Processor -- start 2024-04-02T13:31:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.222+0000 7f6382e69640 1 -- start start 2024-04-02T13:31:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.223+0000 7f6382e69640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63640051c0 0x7f637c15cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.223+0000 7f6382e69640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f637c0cde30 0x7f637c0b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.223+0000 7f6382e69640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f637c1601a0 0x7f637c160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.223+0000 7f6382e69640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f637c0d4da0 con 0x7f637c0cde30 2024-04-02T13:31:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.223+0000 7f6382e69640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f637c06d3f0 con 0x7f637c1601a0 2024-04-02T13:31:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.223+0000 7f6382e69640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f637c0d4f20 con 0x7f63640051c0 2024-04-02T13:31:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.223+0000 7f637bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f637c0cde30 0x7f637c0b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.223+0000 7f637bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f637c0cde30 0x7f637c0b44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44496/0 (socket says 172.21.15.44:44496) 2024-04-02T13:31:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.223+0000 7f637bfff640 1 -- 172.21.15.44:0/1109718640 learned_addr learned my addr 172.21.15.44:0/1109718640 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.223+0000 7f6381406640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f637c1601a0 0x7f637c160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.223+0000 7f6380c05640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63640051c0 0x7f637c15cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.224+0000 7f637bfff640 1 -- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63640051c0 msgr2=0x7f637c15cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.224+0000 7f637bfff640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63640051c0 0x7f637c15cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.224+0000 7f637bfff640 1 -- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f637c1601a0 msgr2=0x7f637c160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.224+0000 7f637bfff640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f637c1601a0 0x7f637c160620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.224+0000 7f637bfff640 1 -- 172.21.15.44:0/1109718640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637c15d280 con 0x7f637c0cde30 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.224+0000 7f637bfff640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f637c0cde30 0x7f637c0b44b0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f6370011990 tx=0x7f6370011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.224+0000 7f63797fa640 1 -- 172.21.15.44:0/1109718640 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637000fa70 con 0x7f637c0cde30 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.224+0000 7f63797fa640 1 -- 172.21.15.44:0/1109718640 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f637000fc10 con 0x7f637c0cde30 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.224+0000 7f6382e69640 1 -- 172.21.15.44:0/1109718640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637c0b4db0 con 0x7f637c0cde30 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.224+0000 7f63797fa640 1 -- 172.21.15.44:0/1109718640 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6370014440 con 0x7f637c0cde30 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.224+0000 7f6382e69640 1 -- 172.21.15.44:0/1109718640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f637c0b5370 con 0x7f637c0cde30 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.225+0000 7f6382e69640 1 -- 172.21.15.44:0/1109718640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f637c0684f0 con 0x7f637c0cde30 2024-04-02T13:31:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.225+0000 7f63797fa640 1 -- 172.21.15.44:0/1109718640 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6370012040 con 0x7f637c0cde30 2024-04-02T13:31:08.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.226+0000 7f63797fa640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f636806c920 0x7f636806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.226+0000 7f63797fa640 1 -- 172.21.15.44:0/1109718640 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5249+0+0 (secure 0 0 0) 0x7f637008bdf0 con 0x7f637c0cde30 2024-04-02T13:31:08.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.226+0000 7f6380c05640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f636806c920 0x7f636806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.226+0000 7f6380c05640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f636806c920 0x7f636806ede0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f637c0cb140 tx=0x7f6374009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:08.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.230+0000 7f63797fa640 1 -- 172.21.15.44:0/1109718640 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f637c15d280 con 0x7f637c0cde30 2024-04-02T13:31:08.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.489+0000 7f6382e69640 1 -- 172.21.15.44:0/1109718640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f637c0d17c0 con 0x7f637c0cde30 2024-04-02T13:31:08.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.490+0000 7f63797fa640 1 -- 172.21.15.44:0/1109718640 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) ==== 80+0+5304 (secure 0 0 0) 0x7f6370019030 con 0x7f637c0cde30 2024-04-02T13:31:08.493 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:08.493 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:31:08.493 INFO:teuthology.orchestra.run.smithi044.stdout: "auth_dump": [ 2024-04-02T13:31:08.493 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.493 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.a", 2024-04-02T13:31:08.493 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmVhZrMhAAklqx5uKCFA5kiqYwyLBKpg==", 2024-04-02T13:31:08.493 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.493 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:31:08.493 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:31:08.493 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:31:08.493 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.493 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.b", 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmalQ+AxAA2SqH9CX/66OIXR7SDID97w==", 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.c", 2024-04-02T13:31:08.494 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxm8BLKORAAjMIrmdQ/bXbPk2sAnYJWTw==", 2024-04-02T13:31:08.495 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.495 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:31:08.495 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:31:08.495 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:31:08.495 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.495 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.495 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.495 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.495 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mds.d", 2024-04-02T13:31:08.495 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmNoaRChAAEqxWmBjBWG79Tkj6dU8FQg==", 2024-04-02T13:31:08.495 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.0", 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDfBQxm4GP9KxAA/AmOunAfVaczAWRGVXvFVg==", 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.496 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.1", 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDiBQxmDL/nKRAADF2C30+IGAXL9XwDQDKCUA==", 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.2", 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDlBQxm1gOCKBAAuYxN9jNHuRYEqc8WrsYErQ==", 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.497 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.3", 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDoBQxmLLFQJxAAFMCzphGU7TOt4nC0Nt+y+g==", 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.498 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.4", 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDuBQxmtk8QLhAAR4gm0irR6C3D2SkhFjvDKQ==", 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.5", 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDxBQxm6Ee/LBAAX5h+9ML70ztw2QH/50OzOg==", 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:31:08.499 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.6", 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD0BQxm35hwKxAAvsgMmiXdLo+c7wlvb9RtVQ==", 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "osd.7", 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQD3BQxmYLBOKhAA1vuSOBDcFf2zT78eoTHtHQ==", 2024-04-02T13:31:08.500 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow profile osd", 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile osd", 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.0", 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A==", 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow r", 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.501 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.1", 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxm3sYsGBAAeRCNbP5qiQ8YnRVsDoaaFQ==", 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.2", 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmKNS5FBAAKvsCNmuG+fI1D/vwu6MC5w==", 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:31:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.3", 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDcBQxmslsSHhAAPsAfALW7RimJDyXdOa2wbQ==", 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow", 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow r", 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow rw", 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow rwx" 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "client.admin", 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmq+VcFRAA9CgeH75Q0sXf27cZdJthJg==", 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": "allow *", 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow *", 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.x", 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmR5ICLxAAtQeg/gq0OAq6e0mroBZVxQ==", 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.504 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: "entity": "mgr.y", 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "AQDbBQxmxoquKhAAE5fMnJIuCy5hGJ1aEAx8Vg==", 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: "caps": { 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": "allow *", 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": "allow profile mgr", 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": "allow *" 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-02T13:31:08.505 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:31:08.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.492+0000 7f6382e69640 1 -- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f636806c920 msgr2=0x7f636806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.492+0000 7f6382e69640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f636806c920 0x7f636806ede0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f637c0cb140 tx=0x7f6374009040 comp rx=0 tx=0).stop 2024-04-02T13:31:08.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.492+0000 7f6382e69640 1 -- 172.21.15.44:0/1109718640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f637c0cde30 msgr2=0x7f637c0b44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.492+0000 7f6382e69640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f637c0cde30 0x7f637c0b44b0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f6370011990 tx=0x7f6370011e50 comp rx=0 tx=0).stop 2024-04-02T13:31:08.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.492+0000 7f6382e69640 1 -- 172.21.15.44:0/1109718640 shutdown_connections 2024-04-02T13:31:08.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.492+0000 7f6382e69640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f636806c920 0x7f636806ede0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.492+0000 7f6382e69640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f637c1601a0 0x7f637c160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.492+0000 7f6382e69640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f637c0cde30 0x7f637c0b44b0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.492+0000 7f6382e69640 1 --2- 172.21.15.44:0/1109718640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63640051c0 0x7f637c15cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.492+0000 7f6382e69640 1 -- 172.21.15.44:0/1109718640 >> 172.21.15.44:0/1109718640 conn(0x7f637c09d470 msgr2=0x7f637c0b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:08.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.492+0000 7f6382e69640 1 -- 172.21.15.44:0/1109718640 shutdown_connections 2024-04-02T13:31:08.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.492+0000 7f6382e69640 1 -- 172.21.15.44:0/1109718640 wait complete. 2024-04-02T13:31:08.508 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:31:08.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.608+0000 7fbe9ac0d640 1 Processor -- start 2024-04-02T13:31:08.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.608+0000 7fbe9ac0d640 1 -- start start 2024-04-02T13:31:08.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.608+0000 7fbe9ac0d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe9406cf90 0x7fbe940cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.608+0000 7fbe9ac0d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbe940cfe30 0x7fbe940d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.608+0000 7fbe9ac0d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe9406c1d0 con 0x7fbe940d0a30 2024-04-02T13:31:08.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.608+0000 7fbe9ac0d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fbe9406cb00 con 0x7fbe940cfe30 2024-04-02T13:31:08.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.608+0000 7fbe9ac0d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbe9406d3f0 con 0x7fbe9406cf90 2024-04-02T13:31:08.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.608+0000 7fbe991aa640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbe940cfe30 0x7fbe940d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.609+0000 7fbe991aa640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbe940cfe30 0x7fbe940d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44508/0 (socket says 172.21.15.44:44508) 2024-04-02T13:31:08.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.609+0000 7fbe991aa640 1 -- 172.21.15.44:0/514767823 learned_addr learned my addr 172.21.15.44:0/514767823 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:08.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.609+0000 7fbe93fff640 1 --2- 172.21.15.44:0/514767823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe9406cf90 0x7fbe940cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.609+0000 7fbe991aa640 1 -- 172.21.15.44:0/514767823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe9406cf90 msgr2=0x7fbe940cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:08.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.609+0000 7fbe991aa640 1 --2- 172.21.15.44:0/514767823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe9406cf90 0x7fbe940cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.609+0000 7fbe991aa640 1 -- 172.21.15.44:0/514767823 >> v1:172.21.15.69:6789/0 conn(0x7fbe940d0a30 legacy=0x7fbe940d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.609+0000 7fbe991aa640 1 -- 172.21.15.44:0/514767823 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9406c4b0 con 0x7fbe940cfe30 2024-04-02T13:31:08.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.610+0000 7fbe991aa640 1 --2- 172.21.15.44:0/514767823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe940cfe30 0x7fbe940d2040 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009e50 tx=0x7fbe8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5d05d9c2e58fc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:08.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.610+0000 7fbe937fe640 1 -- 172.21.15.44:0/514767823 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c02ca70 con 0x7fbe940cfe30 2024-04-02T13:31:08.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.610+0000 7fbe937fe640 1 -- 172.21.15.44:0/514767823 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe8c02cc10 con 0x7fbe940cfe30 2024-04-02T13:31:08.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.610+0000 7fbe937fe640 1 -- 172.21.15.44:0/514767823 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c0343f0 con 0x7fbe940cfe30 2024-04-02T13:31:08.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.610+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/514767823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe940cfe30 msgr2=0x7fbe940d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.610+0000 7fbe9ac0d640 1 --2- 172.21.15.44:0/514767823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe940cfe30 0x7fbe940d2040 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009e50 tx=0x7fbe8c02bed0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.610+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/514767823 shutdown_connections 2024-04-02T13:31:08.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.610+0000 7fbe9ac0d640 1 --2- 172.21.15.44:0/514767823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe940cfe30 0x7fbe940d2040 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.610+0000 7fbe9ac0d640 1 --2- 172.21.15.44:0/514767823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe9406cf90 0x7fbe940cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.610+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/514767823 >> 172.21.15.44:0/514767823 conn(0x7fbe9409d470 msgr2=0x7fbe940d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:08.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.610+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/514767823 shutdown_connections 2024-04-02T13:31:08.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.610+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/514767823 wait complete. 2024-04-02T13:31:08.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.611+0000 7fbe9ac0d640 1 Processor -- start 2024-04-02T13:31:08.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.612+0000 7fbe9ac0d640 1 -- start start 2024-04-02T13:31:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.612+0000 7fbe9ac0d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe9406cf90 0x7fbe941e9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.612+0000 7fbe9ac0d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe940cfe30 0x7fbe940b97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.612+0000 7fbe9ac0d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe940d0a30 0x7fbe941f8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.612+0000 7fbe9ac0d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9415e870 con 0x7fbe940d0a30 2024-04-02T13:31:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.612+0000 7fbe9ac0d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9415e6f0 con 0x7fbe9406cf90 2024-04-02T13:31:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.612+0000 7fbe9ac0d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbe940d4da0 con 0x7fbe940cfe30 2024-04-02T13:31:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.612+0000 7fbe991aa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe940d0a30 0x7fbe941f8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.612+0000 7fbe991aa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe940d0a30 0x7fbe941f8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44522/0 (socket says 172.21.15.44:44522) 2024-04-02T13:31:08.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.612+0000 7fbe991aa640 1 -- 172.21.15.44:0/375973558 learned_addr learned my addr 172.21.15.44:0/375973558 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe989a9640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe9406cf90 0x7fbe941e9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe93fff640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe940cfe30 0x7fbe940b97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe991aa640 1 -- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe940cfe30 msgr2=0x7fbe940b97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe991aa640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe940cfe30 0x7fbe940b97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe991aa640 1 -- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe9406cf90 msgr2=0x7fbe941e9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe991aa640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe9406cf90 0x7fbe941e9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe991aa640 1 -- 172.21.15.44:0/375973558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe940cb7d0 con 0x7fbe940d0a30 2024-04-02T13:31:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe989a9640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe9406cf90 0x7fbe941e9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe991aa640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe940d0a30 0x7fbe941f8010 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c036040 tx=0x7fbe8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe917fa640 1 -- 172.21.15.44:0/375973558 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c043020 con 0x7fbe940d0a30 2024-04-02T13:31:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe917fa640 1 -- 172.21.15.44:0/375973558 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe8c03e450 con 0x7fbe940d0a30 2024-04-02T13:31:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/375973558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe940cbab0 con 0x7fbe940d0a30 2024-04-02T13:31:08.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe917fa640 1 -- 172.21.15.44:0/375973558 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c009510 con 0x7fbe940d0a30 2024-04-02T13:31:08.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.613+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/375973558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe941f7af0 con 0x7fbe940d0a30 2024-04-02T13:31:08.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.614+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/375973558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe940684f0 con 0x7fbe940d0a30 2024-04-02T13:31:08.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.615+0000 7fbe917fa640 1 -- 172.21.15.44:0/375973558 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbe8c046020 con 0x7fbe940d0a30 2024-04-02T13:31:08.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.615+0000 7fbe917fa640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbe7806c8f0 0x7fbe7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.615+0000 7fbe917fa640 1 -- 172.21.15.44:0/375973558 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5249+0+0 (secure 0 0 0) 0x7fbe8c0ae130 con 0x7fbe940d0a30 2024-04-02T13:31:08.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.615+0000 7fbe989a9640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbe7806c8f0 0x7fbe7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.616+0000 7fbe989a9640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbe7806c8f0 0x7fbe7806edb0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fbe88001700 tx=0x7fbe88009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:08.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.620+0000 7fbe917fa640 1 -- 172.21.15.44:0/375973558 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbe8c04d030 con 0x7fbe940d0a30 2024-04-02T13:31:08.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.806+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/375973558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe940ce170 con 0x7fbe940d0a30 2024-04-02T13:31:08.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.806+0000 7fbe917fa640 1 -- 172.21.15.44:0/375973558 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) ==== 74+0+13126 (secure 0 0 0) 0x7fbe8c07c7b0 con 0x7fbe940d0a30 2024-04-02T13:31:08.809 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:08.809 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":117,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:08.091849+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:31:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.809+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbe7806c8f0 msgr2=0x7fbe7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.809+0000 7fbe9ac0d640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbe7806c8f0 0x7fbe7806edb0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fbe88001700 tx=0x7fbe88009040 comp rx=0 tx=0).stop 2024-04-02T13:31:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.809+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/375973558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe940d0a30 msgr2=0x7fbe941f8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.809+0000 7fbe9ac0d640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe940d0a30 0x7fbe941f8010 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c036040 tx=0x7fbe8c004a40 comp rx=0 tx=0).stop 2024-04-02T13:31:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.809+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/375973558 shutdown_connections 2024-04-02T13:31:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.809+0000 7fbe9ac0d640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbe7806c8f0 0x7fbe7806edb0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.809+0000 7fbe9ac0d640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe940d0a30 0x7fbe941f8010 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.809+0000 7fbe9ac0d640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe940cfe30 0x7fbe940b97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.809+0000 7fbe9ac0d640 1 --2- 172.21.15.44:0/375973558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe9406cf90 0x7fbe941e9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.809+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/375973558 >> 172.21.15.44:0/375973558 conn(0x7fbe9409d470 msgr2=0x7fbe940d3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.809+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/375973558 shutdown_connections 2024-04-02T13:31:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.809+0000 7fbe9ac0d640 1 -- 172.21.15.44:0/375973558 wait complete. 2024-04-02T13:31:08.823 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:31:08.823 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:31:08.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.924+0000 7f53df51c640 1 Processor -- start 2024-04-02T13:31:08.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.924+0000 7f53df51c640 1 -- start start 2024-04-02T13:31:08.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.925+0000 7f53df51c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53d815e1d0 0x7f53d80d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.925+0000 7f53df51c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d80d1640 con 0x7f53d80d2dd0 2024-04-02T13:31:08.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.925+0000 7f53df51c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d80cf1f0 con 0x7f53d80cec60 2024-04-02T13:31:08.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.925+0000 7f53df51c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f53d80cfe00 con 0x7f53d815e1d0 2024-04-02T13:31:08.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.925+0000 7f53ddab9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f53d80d2dd0 0x7f53d80b4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48688/0 (socket says 172.21.15.44:48688) 2024-04-02T13:31:08.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.925+0000 7f53ddab9640 1 -- 172.21.15.44:0/3885459348 learned_addr learned my addr 172.21.15.44:0/3885459348 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:08.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53dd2b8640 1 --2- 172.21.15.44:0/3885459348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53d815e1d0 0x7f53d80d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 152997986 0 0) 0x7f53d80d1640 con 0x7f53d80d2dd0 2024-04-02T13:31:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53c0003520 con 0x7f53d80d2dd0 2024-04-02T13:31:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4041217273 0 0) 0x7f53d80cf1f0 con 0x7f53d80cec60 2024-04-02T13:31:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53d80d1640 con 0x7f53d80cec60 2024-04-02T13:31:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2089460408 0 0) 0x7f53c0003520 con 0x7f53d80d2dd0 2024-04-02T13:31:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53d815e1d0 msgr2=0x7f53d80d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 --2- 172.21.15.44:0/3885459348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53d815e1d0 0x7f53d80d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 >> v1:172.21.15.69:6789/0 conn(0x7f53d80cec60 legacy=0x7f53d80d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d80cf4d0 con 0x7f53d80d2dd0 2024-04-02T13:31:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f53c80019f0 con 0x7f53d80d2dd0 2024-04-02T13:31:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 >> v1:172.21.15.44:6789/0 conn(0x7f53d80d2dd0 legacy=0x7f53d80b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 --2- 172.21.15.44:0/3885459348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53c0003b70 0x7f53c0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 --2- 172.21.15.44:0/3885459348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53d815e1d0 0x7f53c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 --2- 172.21.15.44:0/3885459348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53c0004d40 0x7f53c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f53d80cfe00 con 0x7f53c0003b70 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f53c000bac0 con 0x7f53c0004d40 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f53c000bcc0 con 0x7f53d815e1d0 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53dd2b8640 1 --2- 172.21.15.44:0/3885459348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53c0003b70 0x7f53c0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53ddab9640 1 --2- 172.21.15.44:0/3885459348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53c0004d40 0x7f53c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53dcab7640 1 --2- 172.21.15.44:0/3885459348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53d815e1d0 0x7f53c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53dd2b8640 1 -- 172.21.15.44:0/3885459348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53d815e1d0 msgr2=0x7f53c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53dd2b8640 1 --2- 172.21.15.44:0/3885459348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53d815e1d0 0x7f53c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53dd2b8640 1 -- 172.21.15.44:0/3885459348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53c0004d40 msgr2=0x7f53c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53dd2b8640 1 --2- 172.21.15.44:0/3885459348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53c0004d40 0x7f53c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53dd2b8640 1 -- 172.21.15.44:0/3885459348 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c0003520 con 0x7f53c0003b70 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.926+0000 7f53dcab7640 1 --2- 172.21.15.44:0/3885459348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53d815e1d0 0x7f53c0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:08.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.927+0000 7f53dd2b8640 1 --2- 172.21.15.44:0/3885459348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53c0003b70 0x7f53c0003f70 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009ac0 tx=0x7f53c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba37c65c5eb9c7e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.927+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0039070 con 0x7f53c0003b70 2024-04-02T13:31:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.927+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53d002c6b0 con 0x7f53c0003b70 2024-04-02T13:31:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.927+0000 7f53d7fff640 1 -- 172.21.15.44:0/3885459348 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0009690 con 0x7f53c0003b70 2024-04-02T13:31:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.928+0000 7f53df51c640 1 -- 172.21.15.44:0/3885459348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53c0003b70 msgr2=0x7f53c0003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.928+0000 7f53df51c640 1 --2- 172.21.15.44:0/3885459348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53c0003b70 0x7f53c0003f70 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009ac0 tx=0x7f53c0005830 comp rx=0 tx=0).stop 2024-04-02T13:31:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.928+0000 7f53ddab9640 1 -- 172.21.15.44:0/3885459348 reap_dead start 2024-04-02T13:31:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.928+0000 7f53df51c640 1 -- 172.21.15.44:0/3885459348 shutdown_connections 2024-04-02T13:31:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.928+0000 7f53df51c640 1 -- 172.21.15.44:0/3885459348 >> 172.21.15.44:0/3885459348 conn(0x7f53d809d3b0 msgr2=0x7f53d80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.928+0000 7f53df51c640 1 -- 172.21.15.44:0/3885459348 shutdown_connections 2024-04-02T13:31:08.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.928+0000 7f53df51c640 1 -- 172.21.15.44:0/3885459348 wait complete. 2024-04-02T13:31:08.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.929+0000 7f53df51c640 1 Processor -- start 2024-04-02T13:31:08.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.929+0000 7f53df51c640 1 -- start start 2024-04-02T13:31:08.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53df51c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53c0003b70 0x7f53d814c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53df51c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53d8160010 0x7f53d806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53df51c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53d814d800 0x7f53d814dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53df51c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f53d80b7fd0 con 0x7f53d8160010 2024-04-02T13:31:08.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53df51c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f53d80cfa90 con 0x7f53d814d800 2024-04-02T13:31:08.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53df51c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f53d80b8150 con 0x7f53c0003b70 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53dcab7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53d8160010 0x7f53d806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53dcab7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53d8160010 0x7f53d806c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44532/0 (socket says 172.21.15.44:44532) 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53dcab7640 1 -- 172.21.15.44:0/3822905894 learned_addr learned my addr 172.21.15.44:0/3822905894 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53dd2b8640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53c0003b70 0x7f53d814c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53ddab9640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53d814d800 0x7f53d814dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53dcab7640 1 -- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53c0003b70 msgr2=0x7f53d814c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53dcab7640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53c0003b70 0x7f53d814c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53dcab7640 1 -- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53d814d800 msgr2=0x7f53d814dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53dcab7640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53d814d800 0x7f53d814dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53dcab7640 1 -- 172.21.15.44:0/3822905894 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d814c760 con 0x7f53d8160010 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53ddab9640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53d814d800 0x7f53d814dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.930+0000 7f53dcab7640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53d8160010 0x7f53d806c2a0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f53cc00e7b0 tx=0x7f53cc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.931+0000 7f53d5ffb640 1 -- 172.21.15.44:0/3822905894 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53cc0050d0 con 0x7f53d8160010 2024-04-02T13:31:08.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.931+0000 7f53d5ffb640 1 -- 172.21.15.44:0/3822905894 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53cc005270 con 0x7f53d8160010 2024-04-02T13:31:08.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.931+0000 7f53df51c640 1 -- 172.21.15.44:0/3822905894 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d814ca40 con 0x7f53d8160010 2024-04-02T13:31:08.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.931+0000 7f53d5ffb640 1 -- 172.21.15.44:0/3822905894 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53cc005420 con 0x7f53d8160010 2024-04-02T13:31:08.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.931+0000 7f53df51c640 1 -- 172.21.15.44:0/3822905894 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d814d000 con 0x7f53d8160010 2024-04-02T13:31:08.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.932+0000 7f53df51c640 1 -- 172.21.15.44:0/3822905894 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53d80684f0 con 0x7f53d8160010 2024-04-02T13:31:08.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.932+0000 7f53d5ffb640 1 -- 172.21.15.44:0/3822905894 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f53cc01a020 con 0x7f53d8160010 2024-04-02T13:31:08.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.933+0000 7f53d5ffb640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f53c406c8f0 0x7f53c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:08.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.933+0000 7f53d5ffb640 1 -- 172.21.15.44:0/3822905894 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5249+0+0 (secure 0 0 0) 0x7f53cc08b0b0 con 0x7f53d8160010 2024-04-02T13:31:08.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.933+0000 7f53dd2b8640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f53c406c8f0 0x7f53c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:08.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.933+0000 7f53dd2b8640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f53c406c8f0 0x7f53c406edb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f53d0036000 tx=0x7f53d0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:08.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:08.937+0000 7f53d5ffb640 1 -- 172.21.15.44:0/3822905894 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f53cc0596a0 con 0x7f53d8160010 2024-04-02T13:31:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.122+0000 7f53df51c640 1 -- 172.21.15.44:0/3822905894 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f53d80b8f20 con 0x7f53d8160010 2024-04-02T13:31:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.123+0000 7f53d5ffb640 1 -- 172.21.15.44:0/3822905894 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) ==== 93+0+10 (secure 0 0 0) 0x7f53cc058fa0 con 0x7f53d8160010 2024-04-02T13:31:09.126 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:31:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.125+0000 7f53df51c640 1 -- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f53c406c8f0 msgr2=0x7f53c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.125+0000 7f53df51c640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f53c406c8f0 0x7f53c406edb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f53d0036000 tx=0x7f53d0030000 comp rx=0 tx=0).stop 2024-04-02T13:31:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.125+0000 7f53df51c640 1 -- 172.21.15.44:0/3822905894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53d8160010 msgr2=0x7f53d806c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.125+0000 7f53df51c640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53d8160010 0x7f53d806c2a0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f53cc00e7b0 tx=0x7f53cc00ec70 comp rx=0 tx=0).stop 2024-04-02T13:31:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.125+0000 7f53df51c640 1 -- 172.21.15.44:0/3822905894 shutdown_connections 2024-04-02T13:31:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.125+0000 7f53df51c640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f53c406c8f0 0x7f53c406edb0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.125+0000 7f53df51c640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53d814d800 0x7f53d814dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.125+0000 7f53df51c640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53d8160010 0x7f53d806c2a0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.125+0000 7f53df51c640 1 --2- 172.21.15.44:0/3822905894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53c0003b70 0x7f53d814c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.125+0000 7f53df51c640 1 -- 172.21.15.44:0/3822905894 >> 172.21.15.44:0/3822905894 conn(0x7f53d809d3b0 msgr2=0x7f53d806d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.125+0000 7f53df51c640 1 -- 172.21.15.44:0/3822905894 shutdown_connections 2024-04-02T13:31:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.125+0000 7f53df51c640 1 -- 172.21.15.44:0/3822905894 wait complete. 2024-04-02T13:31:09.139 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T13:31:09.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.237+0000 7f0ea35a8640 1 Processor -- start 2024-04-02T13:31:09.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.238+0000 7f0ea35a8640 1 -- start start 2024-04-02T13:31:09.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.238+0000 7f0ea35a8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e9c0d2040 0x7f0e9c0d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.238+0000 7f0ea35a8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e9c0d0000 con 0x7f0e9c06cf90 2024-04-02T13:31:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.238+0000 7f0ea35a8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e9c06c1d0 con 0x7f0e9c0d0a30 2024-04-02T13:31:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.238+0000 7f0ea35a8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0e9c06cb00 con 0x7f0e9c0d2040 2024-04-02T13:31:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.238+0000 7f0ea0b43640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0e9c06cf90 0x7f0e9c0cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48696/0 (socket says 172.21.15.44:48696) 2024-04-02T13:31:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.238+0000 7f0ea0b43640 1 -- 172.21.15.44:0/3274285625 learned_addr learned my addr 172.21.15.44:0/3274285625 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4074573740 0 0) 0x7f0e9c0d0000 con 0x7f0e9c06cf90 2024-04-02T13:31:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e84003520 con 0x7f0e9c06cf90 2024-04-02T13:31:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0ea1b45640 1 --2- 172.21.15.44:0/3274285625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e9c0d2040 0x7f0e9c0d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3282367241 0 0) 0x7f0e84003520 con 0x7f0e9c06cf90 2024-04-02T13:31:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e9c0d2040 msgr2=0x7f0e9c0d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 --2- 172.21.15.44:0/3274285625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e9c0d2040 0x7f0e9c0d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 >> v1:172.21.15.69:6789/0 conn(0x7f0e9c0d0a30 legacy=0x7f0e9c0d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:09.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e9c06c4b0 con 0x7f0e9c06cf90 2024-04-02T13:31:09.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f0e90001670 con 0x7f0e9c06cf90 2024-04-02T13:31:09.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 >> v1:172.21.15.44:6789/0 conn(0x7f0e9c06cf90 legacy=0x7f0e9c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:09.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 --2- 172.21.15.44:0/3274285625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e840039e0 0x7f0e84003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 --2- 172.21.15.44:0/3274285625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e84004800 0x7f0e84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 --2- 172.21.15.44:0/3274285625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e9c0d2040 0x7f0e8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9c06cb00 con 0x7f0e84004800 2024-04-02T13:31:09.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0e84005550 con 0x7f0e840039e0 2024-04-02T13:31:09.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0e8400bf00 con 0x7f0e9c0d2040 2024-04-02T13:31:09.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0ea0b43640 1 --2- 172.21.15.44:0/3274285625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e84004800 0x7f0e84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0ea1344640 1 --2- 172.21.15.44:0/3274285625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e840039e0 0x7f0e84003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0ea1b45640 1 --2- 172.21.15.44:0/3274285625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e9c0d2040 0x7f0e8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0ea0b43640 1 -- 172.21.15.44:0/3274285625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e9c0d2040 msgr2=0x7f0e8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:09.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0ea0b43640 1 --2- 172.21.15.44:0/3274285625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e9c0d2040 0x7f0e8400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0ea0b43640 1 -- 172.21.15.44:0/3274285625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e840039e0 msgr2=0x7f0e84003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:09.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0ea0b43640 1 --2- 172.21.15.44:0/3274285625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e840039e0 0x7f0e84003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.239+0000 7f0ea0b43640 1 -- 172.21.15.44:0/3274285625 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e84003520 con 0x7f0e84004800 2024-04-02T13:31:09.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.240+0000 7f0ea0b43640 1 --2- 172.21.15.44:0/3274285625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e84004800 0x7f0e84004c80 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c08e630 tx=0x7f0e84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ec708d47f40d51 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:09.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.240+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e90007d30 con 0x7f0e84004800 2024-04-02T13:31:09.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.240+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e90030070 con 0x7f0e84004800 2024-04-02T13:31:09.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.240+0000 7f0e9bfff640 1 -- 172.21.15.44:0/3274285625 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e90005d30 con 0x7f0e84004800 2024-04-02T13:31:09.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.241+0000 7f0ea35a8640 1 -- 172.21.15.44:0/3274285625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e84004800 msgr2=0x7f0e84004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:09.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.241+0000 7f0ea35a8640 1 --2- 172.21.15.44:0/3274285625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e84004800 0x7f0e84004c80 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c08e630 tx=0x7f0e84005830 comp rx=0 tx=0).stop 2024-04-02T13:31:09.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.241+0000 7f0ea1b45640 1 -- 172.21.15.44:0/3274285625 reap_dead start 2024-04-02T13:31:09.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.241+0000 7f0ea35a8640 1 -- 172.21.15.44:0/3274285625 shutdown_connections 2024-04-02T13:31:09.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.241+0000 7f0ea35a8640 1 -- 172.21.15.44:0/3274285625 >> 172.21.15.44:0/3274285625 conn(0x7f0e9c09d470 msgr2=0x7f0e9c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:09.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.241+0000 7f0ea35a8640 1 -- 172.21.15.44:0/3274285625 shutdown_connections 2024-04-02T13:31:09.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.241+0000 7f0ea35a8640 1 -- 172.21.15.44:0/3274285625 wait complete. 2024-04-02T13:31:09.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.242+0000 7f0ea35a8640 1 Processor -- start 2024-04-02T13:31:09.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.242+0000 7f0ea35a8640 1 -- start start 2024-04-02T13:31:09.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.243+0000 7f0ea35a8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e84004800 0x7f0e9c0c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.243+0000 7f0ea35a8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e9c160010 0x7f0e9c160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.243+0000 7f0ea35a8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e9c0c93e0 0x7f0e9c0c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.243+0000 7f0ea35a8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9c15e6f0 con 0x7f0e9c0c93e0 2024-04-02T13:31:09.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.243+0000 7f0ea35a8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9c06d3f0 con 0x7f0e84004800 2024-04-02T13:31:09.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.243+0000 7f0ea35a8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0e9c15e870 con 0x7f0e9c160010 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.243+0000 7f0ea1b45640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e9c0c93e0 0x7f0e9c0c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.243+0000 7f0ea1b45640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e9c0c93e0 0x7f0e9c0c9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44558/0 (socket says 172.21.15.44:44558) 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.243+0000 7f0ea1b45640 1 -- 172.21.15.44:0/221317108 learned_addr learned my addr 172.21.15.44:0/221317108 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.243+0000 7f0ea0b43640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e9c160010 0x7f0e9c160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.243+0000 7f0ea1344640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e84004800 0x7f0e9c0c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.244+0000 7f0ea1b45640 1 -- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e9c160010 msgr2=0x7f0e9c160490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.244+0000 7f0ea1b45640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e9c160010 0x7f0e9c160490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.244+0000 7f0ea1b45640 1 -- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e84004800 msgr2=0x7f0e9c0c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.244+0000 7f0ea1b45640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e84004800 0x7f0e9c0c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.244+0000 7f0ea1b45640 1 -- 172.21.15.44:0/221317108 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e9c0b3b80 con 0x7f0e9c0c93e0 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.244+0000 7f0ea1b45640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e9c0c93e0 0x7f0e9c0c9860 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c09d9a0 tx=0x7f0e8c007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.244+0000 7f0e99ffb640 1 -- 172.21.15.44:0/221317108 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c0046a0 con 0x7f0e9c0c93e0 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.244+0000 7f0e99ffb640 1 -- 172.21.15.44:0/221317108 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e8c01a070 con 0x7f0e9c0c93e0 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.244+0000 7f0e99ffb640 1 -- 172.21.15.44:0/221317108 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c015640 con 0x7f0e9c0c93e0 2024-04-02T13:31:09.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.244+0000 7f0ea35a8640 1 -- 172.21.15.44:0/221317108 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e9c0ca0a0 con 0x7f0e9c0c93e0 2024-04-02T13:31:09.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.244+0000 7f0ea35a8640 1 -- 172.21.15.44:0/221317108 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e9c0ca660 con 0x7f0e9c0c93e0 2024-04-02T13:31:09.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.245+0000 7f0ea35a8640 1 -- 172.21.15.44:0/221317108 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e9c0684f0 con 0x7f0e9c0c93e0 2024-04-02T13:31:09.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.246+0000 7f0e99ffb640 1 -- 172.21.15.44:0/221317108 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0e8c0157e0 con 0x7f0e9c0c93e0 2024-04-02T13:31:09.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.246+0000 7f0e99ffb640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e8806c8f0 0x7f0e8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.246+0000 7f0e99ffb640 1 -- 172.21.15.44:0/221317108 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5249+0+0 (secure 0 0 0) 0x7f0e8c08bee0 con 0x7f0e9c0c93e0 2024-04-02T13:31:09.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.246+0000 7f0ea1344640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e8806c8f0 0x7f0e8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.247+0000 7f0ea1344640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e8806c8f0 0x7f0e8806edb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f0e940017e0 tx=0x7f0e94009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:09.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.251+0000 7f0e99ffb640 1 -- 172.21.15.44:0/221317108 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0e8c059df0 con 0x7f0e9c0c93e0 2024-04-02T13:31:09.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.435+0000 7f0ea35a8640 1 -- 172.21.15.44:0/221317108 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f0e9c0d17c0 con 0x7f0e9c0c93e0 2024-04-02T13:31:09.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.435+0000 7f0e99ffb640 1 -- 172.21.15.44:0/221317108 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v117) ==== 84+0+65 (secure 0 0 0) 0x7f0e8c05e0f0 con 0x7f0e9c0c93e0 2024-04-02T13:31:09.442 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:09.442 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-04-02T13:31:09.442 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-02T13:31:09.443 INFO:teuthology.orchestra.run.smithi044.stdout: "poolnum": 1, 2024-04-02T13:31:09.443 INFO:teuthology.orchestra.run.smithi044.stdout: "poolname": ".mgr" 2024-04-02T13:31:09.443 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:09.443 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-04-02T13:31:09.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.437+0000 7f0ea35a8640 1 -- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e8806c8f0 msgr2=0x7f0e8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:09.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.437+0000 7f0ea35a8640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e8806c8f0 0x7f0e8806edb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f0e940017e0 tx=0x7f0e94009040 comp rx=0 tx=0).stop 2024-04-02T13:31:09.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.437+0000 7f0ea35a8640 1 -- 172.21.15.44:0/221317108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e9c0c93e0 msgr2=0x7f0e9c0c9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:09.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.437+0000 7f0ea35a8640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e9c0c93e0 0x7f0e9c0c9860 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c09d9a0 tx=0x7f0e8c007700 comp rx=0 tx=0).stop 2024-04-02T13:31:09.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.437+0000 7f0ea35a8640 1 -- 172.21.15.44:0/221317108 shutdown_connections 2024-04-02T13:31:09.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.437+0000 7f0ea35a8640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0e8806c8f0 0x7f0e8806edb0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.437+0000 7f0ea35a8640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e9c0c93e0 0x7f0e9c0c9860 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.437+0000 7f0ea35a8640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e9c160010 0x7f0e9c160490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.437+0000 7f0ea35a8640 1 --2- 172.21.15.44:0/221317108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e84004800 0x7f0e9c0c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.437+0000 7f0ea35a8640 1 -- 172.21.15.44:0/221317108 >> 172.21.15.44:0/221317108 conn(0x7f0e9c09d470 msgr2=0x7f0e9c0b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:09.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.437+0000 7f0ea35a8640 1 -- 172.21.15.44:0/221317108 shutdown_connections 2024-04-02T13:31:09.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.437+0000 7f0ea35a8640 1 -- 172.21.15.44:0/221317108 wait complete. 2024-04-02T13:31:09.451 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T13:31:09.451 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T13:31:09.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.552+0000 7f85f1104640 1 Processor -- start 2024-04-02T13:31:09.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.553+0000 7f85f1104640 1 -- start start 2024-04-02T13:31:09.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.553+0000 7f85f1104640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f85ec0ce9a0 0x7f85ec0d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.553+0000 7f85f1104640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0d0670 0x7f85ec164c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.553+0000 7f85f1104640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85ec0cdc20 con 0x7f85ec1554c0 2024-04-02T13:31:09.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.553+0000 7f85f1104640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f85ec0d0ae0 con 0x7f85ec0ce9a0 2024-04-02T13:31:09.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.553+0000 7f85f1104640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f85ec0cdec0 con 0x7f85ec0d0670 2024-04-02T13:31:09.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.553+0000 7f85ead9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f85ec1554c0 0x7f85ec071120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48704/0 (socket says 172.21.15.44:48704) 2024-04-02T13:31:09.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.553+0000 7f85ead9d640 1 -- 172.21.15.44:0/394588809 learned_addr learned my addr 172.21.15.44:0/394588809 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:09.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85ea59c640 1 --2- 172.21.15.44:0/394588809 >> v2:172.21.15.69:3300/0 conn(0x7f85ec0ce9a0 0x7f85ec0d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85eb59e640 1 --2- 172.21.15.44:0/394588809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0d0670 0x7f85ec164c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1750171680 0 0) 0x7f85ec0cdc20 con 0x7f85ec1554c0 2024-04-02T13:31:09.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85d4003520 con 0x7f85ec1554c0 2024-04-02T13:31:09.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3581117595 0 0) 0x7f85d4003520 con 0x7f85ec1554c0 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0d0670 msgr2=0x7f85ec164c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 --2- 172.21.15.44:0/394588809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0d0670 0x7f85ec164c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 >> v2:172.21.15.69:3300/0 conn(0x7f85ec0ce9a0 msgr2=0x7f85ec0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 --2- 172.21.15.44:0/394588809 >> v2:172.21.15.69:3300/0 conn(0x7f85ec0ce9a0 0x7f85ec0d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ec0ce1a0 con 0x7f85ec1554c0 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f85dc001670 con 0x7f85ec1554c0 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 >> v1:172.21.15.44:6789/0 conn(0x7f85ec1554c0 legacy=0x7f85ec071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 --2- 172.21.15.44:0/394588809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0d0670 0x7f85d40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 --2- 172.21.15.44:0/394588809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d4003f30 0x7f85d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 --2- 172.21.15.44:0/394588809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85d4004d40 0x7f85d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec0d0ae0 con 0x7f85d4004d40 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec0cdec0 con 0x7f85d4003f30 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f85d400dbd0 con 0x7f85ec0d0670 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85ea59c640 1 --2- 172.21.15.44:0/394588809 >> v2:172.21.15.69:3300/0 conn(0x7f85ec0ce9a0 0x7f85ec0d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:31:09.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85eb59e640 1 --2- 172.21.15.44:0/394588809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85d4004d40 0x7f85d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85ead9d640 1 --2- 172.21.15.44:0/394588809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0d0670 0x7f85d40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.554+0000 7f85ea59c640 1 --2- 172.21.15.44:0/394588809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d4003f30 0x7f85d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.555+0000 7f85eb59e640 1 -- 172.21.15.44:0/394588809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0d0670 msgr2=0x7f85d40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:09.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.555+0000 7f85eb59e640 1 --2- 172.21.15.44:0/394588809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0d0670 0x7f85d40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.555+0000 7f85eb59e640 1 -- 172.21.15.44:0/394588809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d4003f30 msgr2=0x7f85d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:09.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.555+0000 7f85eb59e640 1 --2- 172.21.15.44:0/394588809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d4003f30 0x7f85d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.555+0000 7f85eb59e640 1 -- 172.21.15.44:0/394588809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85d4003520 con 0x7f85d4004d40 2024-04-02T13:31:09.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.555+0000 7f85eb59e640 1 --2- 172.21.15.44:0/394588809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85d4004d40 0x7f85d40051c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f85e4009fc0 tx=0x7f85d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8743d118cd9c50df server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:09.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.555+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4039070 con 0x7f85d4004d40 2024-04-02T13:31:09.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.555+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e402cc00 con 0x7f85d4004d40 2024-04-02T13:31:09.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.555+0000 7f85e9d9b640 1 -- 172.21.15.44:0/394588809 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4009c10 con 0x7f85d4004d40 2024-04-02T13:31:09.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.556+0000 7f85f1104640 1 -- 172.21.15.44:0/394588809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85d4004d40 msgr2=0x7f85d40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:09.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.556+0000 7f85f1104640 1 --2- 172.21.15.44:0/394588809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85d4004d40 0x7f85d40051c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f85e4009fc0 tx=0x7f85d4005830 comp rx=0 tx=0).stop 2024-04-02T13:31:09.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.556+0000 7f85eb59e640 1 -- 172.21.15.44:0/394588809 reap_dead start 2024-04-02T13:31:09.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.556+0000 7f85f1104640 1 -- 172.21.15.44:0/394588809 shutdown_connections 2024-04-02T13:31:09.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.556+0000 7f85f1104640 1 -- 172.21.15.44:0/394588809 >> 172.21.15.44:0/394588809 conn(0x7f85ec06b920 msgr2=0x7f85ec06bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:09.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.556+0000 7f85f1104640 1 -- 172.21.15.44:0/394588809 shutdown_connections 2024-04-02T13:31:09.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.556+0000 7f85f1104640 1 -- 172.21.15.44:0/394588809 wait complete. 2024-04-02T13:31:09.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.557+0000 7f85f1104640 1 Processor -- start 2024-04-02T13:31:09.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.557+0000 7f85f1104640 1 -- start start 2024-04-02T13:31:09.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.558+0000 7f85f1104640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d4004d40 0x7f85ec15ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.558+0000 7f85f1104640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85ec1582b0 0x7f85ec150470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.558+0000 7f85f1104640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0ccc80 0x7f85ec0cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.558+0000 7f85f1104640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec15c290 con 0x7f85ec1582b0 2024-04-02T13:31:09.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.558+0000 7f85f1104640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec1558a0 con 0x7f85d4004d40 2024-04-02T13:31:09.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.558+0000 7f85f1104640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f85ec158a20 con 0x7f85ec0ccc80 2024-04-02T13:31:09.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.558+0000 7f85ea59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85ec1582b0 0x7f85ec150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.558+0000 7f85ea59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85ec1582b0 0x7f85ec150470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44586/0 (socket says 172.21.15.44:44586) 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.558+0000 7f85ea59c640 1 -- 172.21.15.44:0/2914169083 learned_addr learned my addr 172.21.15.44:0/2914169083 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.558+0000 7f85eb59e640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0ccc80 0x7f85ec0cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.558+0000 7f85ead9d640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d4004d40 0x7f85ec15ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.559+0000 7f85ea59c640 1 -- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0ccc80 msgr2=0x7f85ec0cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.559+0000 7f85ea59c640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0ccc80 0x7f85ec0cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.559+0000 7f85ea59c640 1 -- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d4004d40 msgr2=0x7f85ec15ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.559+0000 7f85ea59c640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d4004d40 0x7f85ec15ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.559+0000 7f85ea59c640 1 -- 172.21.15.44:0/2914169083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ec1f4420 con 0x7f85ec1582b0 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.559+0000 7f85eb59e640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0ccc80 0x7f85ec0cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.559+0000 7f85ead9d640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d4004d40 0x7f85ec15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.559+0000 7f85ea59c640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85ec1582b0 0x7f85ec150470 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f85e0011d00 tx=0x7f85e000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.559+0000 7f85d37fe640 1 -- 172.21.15.44:0/2914169083 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e000fd20 con 0x7f85ec1582b0 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.559+0000 7f85d37fe640 1 -- 172.21.15.44:0/2914169083 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e0004030 con 0x7f85ec1582b0 2024-04-02T13:31:09.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.559+0000 7f85f1104640 1 -- 172.21.15.44:0/2914169083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ec1f3660 con 0x7f85ec1582b0 2024-04-02T13:31:09.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.559+0000 7f85d37fe640 1 -- 172.21.15.44:0/2914169083 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e0004420 con 0x7f85ec1582b0 2024-04-02T13:31:09.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.559+0000 7f85f1104640 1 -- 172.21.15.44:0/2914169083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ec1f3c20 con 0x7f85ec1582b0 2024-04-02T13:31:09.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.560+0000 7f85f1104640 1 -- 172.21.15.44:0/2914169083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85ec0cd640 con 0x7f85ec1582b0 2024-04-02T13:31:09.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.560+0000 7f85d37fe640 1 -- 172.21.15.44:0/2914169083 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f85e0012040 con 0x7f85ec1582b0 2024-04-02T13:31:09.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.561+0000 7f85d37fe640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f85d806c8f0 0x7f85d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:09.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.561+0000 7f85d37fe640 1 -- 172.21.15.44:0/2914169083 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5249+0+0 (secure 0 0 0) 0x7f85e008bf20 con 0x7f85ec1582b0 2024-04-02T13:31:09.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.561+0000 7f85ead9d640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f85d806c8f0 0x7f85d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:09.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.562+0000 7f85ead9d640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f85d806c8f0 0x7f85d806edb0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f85dc001050 tx=0x7f85dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:09.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.565+0000 7f85d37fe640 1 -- 172.21.15.44:0/2914169083 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f85e0019070 con 0x7f85ec1582b0 2024-04-02T13:31:09.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:09.748+0000 7f85f1104640 1 -- 172.21.15.44:0/2914169083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f85ec06d830 con 0x7f85ec1582b0 2024-04-02T13:31:10.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.110+0000 7f85d37fe640 1 -- 172.21.15.44:0/2914169083 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v118) ==== 138+0+0 (secure 0 0 0) 0x7f85e005a500 con 0x7f85ec1582b0 2024-04-02T13:31:10.113 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' created 2024-04-02T13:31:10.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.112+0000 7f85f1104640 1 -- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f85d806c8f0 msgr2=0x7f85d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:10.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.112+0000 7f85f1104640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f85d806c8f0 0x7f85d806edb0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f85dc001050 tx=0x7f85dc009040 comp rx=0 tx=0).stop 2024-04-02T13:31:10.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.112+0000 7f85f1104640 1 -- 172.21.15.44:0/2914169083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85ec1582b0 msgr2=0x7f85ec150470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:10.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.112+0000 7f85f1104640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85ec1582b0 0x7f85ec150470 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f85e0011d00 tx=0x7f85e000f610 comp rx=0 tx=0).stop 2024-04-02T13:31:10.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.112+0000 7f85f1104640 1 -- 172.21.15.44:0/2914169083 shutdown_connections 2024-04-02T13:31:10.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.112+0000 7f85f1104640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f85d806c8f0 0x7f85d806edb0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:10.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.112+0000 7f85f1104640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85ec0ccc80 0x7f85ec0cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:10.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.112+0000 7f85f1104640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85ec1582b0 0x7f85ec150470 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:10.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.112+0000 7f85f1104640 1 --2- 172.21.15.44:0/2914169083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85d4004d40 0x7f85ec15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:10.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.112+0000 7f85f1104640 1 -- 172.21.15.44:0/2914169083 >> 172.21.15.44:0/2914169083 conn(0x7f85ec06b920 msgr2=0x7f85ec06e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:10.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.113+0000 7f85f1104640 1 -- 172.21.15.44:0/2914169083 shutdown_connections 2024-04-02T13:31:10.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.113+0000 7f85f1104640 1 -- 172.21.15.44:0/2914169083 wait complete. 2024-04-02T13:31:10.130 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T13:31:10.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.236+0000 7f6f9762c640 1 Processor -- start 2024-04-02T13:31:10.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.236+0000 7f6f9762c640 1 -- start start 2024-04-02T13:31:10.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.237+0000 7f6f9762c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f900b1540 0x7f6f900b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:10.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.237+0000 7f6f9762c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f900ced00 con 0x7f6f9015e470 2024-04-02T13:31:10.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.237+0000 7f6f9762c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f900cefe0 con 0x7f6f900ce790 2024-04-02T13:31:10.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.237+0000 7f6f9762c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6f900cf2c0 con 0x7f6f900b1540 2024-04-02T13:31:10.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.237+0000 7f6f953c8640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6f9015e470 0x7f6f900d1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48710/0 (socket says 172.21.15.44:48710) 2024-04-02T13:31:10.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.237+0000 7f6f953c8640 1 -- 172.21.15.44:0/1749696650 learned_addr learned my addr 172.21.15.44:0/1749696650 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:10.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.237+0000 7f6f94bc7640 1 --2- 172.21.15.44:0/1749696650 >> v2:172.21.15.69:3300/0 conn(0x7f6f900b1540 0x7f6f900b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:10.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.237+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2935725154 0 0) 0x7f6f900ced00 con 0x7f6f9015e470 2024-04-02T13:31:10.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.237+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f78003520 con 0x7f6f9015e470 2024-04-02T13:31:10.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4232096298 0 0) 0x7f6f78003520 con 0x7f6f9015e470 2024-04-02T13:31:10.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 >> v2:172.21.15.69:3300/0 conn(0x7f6f900b1540 msgr2=0x7f6f900b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:10.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f8ffff640 1 --2- 172.21.15.44:0/1749696650 >> v2:172.21.15.69:3300/0 conn(0x7f6f900b1540 0x7f6f900b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 >> v1:172.21.15.69:6789/0 conn(0x7f6f900ce790 legacy=0x7f6f9006d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f900cf5a0 con 0x7f6f9015e470 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f6f800019f0 con 0x7f6f9015e470 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 >> v1:172.21.15.44:6789/0 conn(0x7f6f9015e470 legacy=0x7f6f900d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f8ffff640 1 --2- 172.21.15.44:0/1749696650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f780039b0 0x7f6f78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f8ffff640 1 --2- 172.21.15.44:0/1749696650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f78004800 0x7f6f78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f8ffff640 1 --2- 172.21.15.44:0/1749696650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f780051c0 0x7f6f7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6f900cf2c0 con 0x7f6f78004800 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7800c160 con 0x7f6f780051c0 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6f7800c360 con 0x7f6f780039b0 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f94bc7640 1 --2- 172.21.15.44:0/1749696650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f78004800 0x7f6f78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f953c8640 1 --2- 172.21.15.44:0/1749696650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f780039b0 0x7f6f78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f95bc9640 1 --2- 172.21.15.44:0/1749696650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f780051c0 0x7f6f7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:10.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f94bc7640 1 -- 172.21.15.44:0/1749696650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f780039b0 msgr2=0x7f6f78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f94bc7640 1 --2- 172.21.15.44:0/1749696650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f780039b0 0x7f6f78003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f94bc7640 1 -- 172.21.15.44:0/1749696650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f780051c0 msgr2=0x7f6f7800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f94bc7640 1 --2- 172.21.15.44:0/1749696650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f780051c0 0x7f6f7800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f94bc7640 1 -- 172.21.15.44:0/1749696650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f78003520 con 0x7f6f78004800 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.238+0000 7f6f95bc9640 1 -- 172.21.15.44:0/1749696650 reap_dead start 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.239+0000 7f6f94bc7640 1 --2- 172.21.15.44:0/1749696650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f78004800 0x7f6f78004c80 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009ac0 tx=0x7f6f78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112aa2c1c79ca557 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.239+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f84039070 con 0x7f6f78004800 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.239+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f8402c6b0 con 0x7f6f78004800 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.239+0000 7f6f8ffff640 1 -- 172.21.15.44:0/1749696650 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f84009690 con 0x7f6f78004800 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.240+0000 7f6f9762c640 1 -- 172.21.15.44:0/1749696650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f78004800 msgr2=0x7f6f78004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.240+0000 7f6f9762c640 1 --2- 172.21.15.44:0/1749696650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f78004800 0x7f6f78004c80 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009ac0 tx=0x7f6f78005830 comp rx=0 tx=0).stop 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.240+0000 7f6f9762c640 1 -- 172.21.15.44:0/1749696650 shutdown_connections 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.240+0000 7f6f9762c640 1 --2- 172.21.15.44:0/1749696650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f78004800 0x7f6f78004c80 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:10.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.240+0000 7f6f9762c640 1 -- 172.21.15.44:0/1749696650 >> 172.21.15.44:0/1749696650 conn(0x7f6f9009aa20 msgr2=0x7f6f9009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:10.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.240+0000 7f6f9762c640 1 -- 172.21.15.44:0/1749696650 shutdown_connections 2024-04-02T13:31:10.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.240+0000 7f6f9762c640 1 -- 172.21.15.44:0/1749696650 wait complete. 2024-04-02T13:31:10.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.241+0000 7f6f9762c640 1 Processor -- start 2024-04-02T13:31:10.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.241+0000 7f6f9762c640 1 -- start start 2024-04-02T13:31:10.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f9762c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f78004800 0x7f6f9014f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:10.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f9762c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f900c6ab0 0x7f6f9015c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:10.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f9762c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f9015c710 0x7f6f900c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:10.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f9762c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6f900b32c0 con 0x7f6f900c6ab0 2024-04-02T13:31:10.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f9762c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9006cb60 con 0x7f6f9015c710 2024-04-02T13:31:10.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f9762c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6f900b3440 con 0x7f6f78004800 2024-04-02T13:31:10.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f94bc7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f900c6ab0 0x7f6f9015c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:10.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f94bc7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f900c6ab0 0x7f6f9015c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44612/0 (socket says 172.21.15.44:44612) 2024-04-02T13:31:10.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f94bc7640 1 -- 172.21.15.44:0/637658385 learned_addr learned my addr 172.21.15.44:0/637658385 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:10.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f953c8640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f78004800 0x7f6f9014f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:10.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f95bc9640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f9015c710 0x7f6f900c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:10.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f94bc7640 1 -- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f78004800 msgr2=0x7f6f9014f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:10.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f94bc7640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f78004800 0x7f6f9014f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:10.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f94bc7640 1 -- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f9015c710 msgr2=0x7f6f900c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:10.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f94bc7640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f9015c710 0x7f6f900c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:10.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f94bc7640 1 -- 172.21.15.44:0/637658385 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f900c5a10 con 0x7f6f900c6ab0 2024-04-02T13:31:10.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.242+0000 7f6f94bc7640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f900c6ab0 0x7f6f9015c1d0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f6f84037040 tx=0x7f6f84004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:10.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.243+0000 7f6f8dffb640 1 -- 172.21.15.44:0/637658385 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f84039050 con 0x7f6f900c6ab0 2024-04-02T13:31:10.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.243+0000 7f6f8dffb640 1 -- 172.21.15.44:0/637658385 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f84045070 con 0x7f6f900c6ab0 2024-04-02T13:31:10.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.243+0000 7f6f9762c640 1 -- 172.21.15.44:0/637658385 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f900c5cf0 con 0x7f6f900c6ab0 2024-04-02T13:31:10.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.243+0000 7f6f8dffb640 1 -- 172.21.15.44:0/637658385 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f840044e0 con 0x7f6f900c6ab0 2024-04-02T13:31:10.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.243+0000 7f6f9762c640 1 -- 172.21.15.44:0/637658385 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f900c62b0 con 0x7f6f900c6ab0 2024-04-02T13:31:10.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.244+0000 7f6f9762c640 1 -- 172.21.15.44:0/637658385 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f900684f0 con 0x7f6f900c6ab0 2024-04-02T13:31:10.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.244+0000 7f6f8dffb640 1 -- 172.21.15.44:0/637658385 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6f84003a50 con 0x7f6f900c6ab0 2024-04-02T13:31:10.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.245+0000 7f6f8dffb640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f7c06c920 0x7f6f7c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:10.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.245+0000 7f6f8dffb640 1 -- 172.21.15.44:0/637658385 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5619+0+0 (secure 0 0 0) 0x7f6f840ad980 con 0x7f6f900c6ab0 2024-04-02T13:31:10.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.245+0000 7f6f953c8640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f7c06c920 0x7f6f7c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:10.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.246+0000 7f6f953c8640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f7c06c920 0x7f6f7c06ede0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6f800013d0 tx=0x7f6f80006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:10.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.249+0000 7f6f8dffb640 1 -- 172.21.15.44:0/637658385 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6f8404f020 con 0x7f6f900c6ab0 2024-04-02T13:31:10.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:10.433+0000 7f6f9762c640 1 -- 172.21.15.44:0/637658385 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f6f9006e580 con 0x7f6f900c6ab0 2024-04-02T13:31:10.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-04-02T13:31:11.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.115+0000 7f6f8dffb640 1 -- 172.21.15.44:0/637658385 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v119) ==== 170+0+0 (secure 0 0 0) 0x7f6f8407b850 con 0x7f6f900c6ab0 2024-04-02T13:31:11.119 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' created 2024-04-02T13:31:11.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.118+0000 7f6f9762c640 1 -- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f7c06c920 msgr2=0x7f6f7c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:11.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.118+0000 7f6f9762c640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f7c06c920 0x7f6f7c06ede0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6f800013d0 tx=0x7f6f80006040 comp rx=0 tx=0).stop 2024-04-02T13:31:11.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.118+0000 7f6f9762c640 1 -- 172.21.15.44:0/637658385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f900c6ab0 msgr2=0x7f6f9015c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:11.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.118+0000 7f6f9762c640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f900c6ab0 0x7f6f9015c1d0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f6f84037040 tx=0x7f6f84004730 comp rx=0 tx=0).stop 2024-04-02T13:31:11.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.118+0000 7f6f9762c640 1 -- 172.21.15.44:0/637658385 shutdown_connections 2024-04-02T13:31:11.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.118+0000 7f6f9762c640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f7c06c920 0x7f6f7c06ede0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:11.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.118+0000 7f6f9762c640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f9015c710 0x7f6f900c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:11.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.118+0000 7f6f9762c640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f900c6ab0 0x7f6f9015c1d0 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:11.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.118+0000 7f6f9762c640 1 --2- 172.21.15.44:0/637658385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f78004800 0x7f6f9014f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:11.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.118+0000 7f6f9762c640 1 -- 172.21.15.44:0/637658385 >> 172.21.15.44:0/637658385 conn(0x7f6f9009aa20 msgr2=0x7f6f9006f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:11.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.118+0000 7f6f9762c640 1 -- 172.21.15.44:0/637658385 shutdown_connections 2024-04-02T13:31:11.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.118+0000 7f6f9762c640 1 -- 172.21.15.44:0/637658385 wait complete. 2024-04-02T13:31:11.152 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T13:31:11.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.256+0000 7f9431a06640 1 Processor -- start 2024-04-02T13:31:11.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.256+0000 7f9431a06640 1 -- start start 2024-04-02T13:31:11.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f9431a06640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f942c15e420 0x7f942c0ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:11.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f9431a06640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f942c0721f0 0x7f942c0cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:11.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f9431a06640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f942c0cfc40 0x7f942c161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:11.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f9431a06640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f942c0d00b0 con 0x7f942c0cfc40 2024-04-02T13:31:11.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f9431a06640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f942c06c220 con 0x7f942c0721f0 2024-04-02T13:31:11.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f9431a06640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f942c15f7e0 con 0x7f942c15e420 2024-04-02T13:31:11.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f942b7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f942c0cfc40 0x7f942c161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:11.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f942b7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f942c0cfc40 0x7f942c161eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44616/0 (socket says 172.21.15.44:44616) 2024-04-02T13:31:11.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f942b7fe640 1 -- 172.21.15.44:0/1887714249 learned_addr learned my addr 172.21.15.44:0/1887714249 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:11.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f942a7fc640 1 --2- 172.21.15.44:0/1887714249 >> v2:172.21.15.69:3300/0 conn(0x7f942c0721f0 0x7f942c0cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:11.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f942affd640 1 --2- 172.21.15.44:0/1887714249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f942c15e420 0x7f942c0ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:11.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f942b7fe640 1 -- 172.21.15.44:0/1887714249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f942c15e420 msgr2=0x7f942c0ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:11.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f942b7fe640 1 --2- 172.21.15.44:0/1887714249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f942c15e420 0x7f942c0ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:11.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f942b7fe640 1 -- 172.21.15.44:0/1887714249 >> v2:172.21.15.69:3300/0 conn(0x7f942c0721f0 msgr2=0x7f942c0cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:11.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f942b7fe640 1 --2- 172.21.15.44:0/1887714249 >> v2:172.21.15.69:3300/0 conn(0x7f942c0721f0 0x7f942c0cf700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f942b7fe640 1 -- 172.21.15.44:0/1887714249 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942c15ec60 con 0x7f942c0cfc40 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f942a7fc640 1 --2- 172.21.15.44:0/1887714249 >> v2:172.21.15.69:3300/0 conn(0x7f942c0721f0 0x7f942c0cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.257+0000 7f942affd640 1 --2- 172.21.15.44:0/1887714249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f942c15e420 0x7f942c0ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.258+0000 7f942b7fe640 1 --2- 172.21.15.44:0/1887714249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f942c0cfc40 0x7f942c161eb0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f942c0ceb70 tx=0x7f941c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf58ce6dc18b3f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.258+0000 7f9429ffb640 1 -- 172.21.15.44:0/1887714249 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c02c950 con 0x7f942c0cfc40 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.258+0000 7f9429ffb640 1 -- 172.21.15.44:0/1887714249 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f941c02caf0 con 0x7f942c0cfc40 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.258+0000 7f9429ffb640 1 -- 172.21.15.44:0/1887714249 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c0343f0 con 0x7f942c0cfc40 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.259+0000 7f9431a06640 1 -- 172.21.15.44:0/1887714249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f942c0cfc40 msgr2=0x7f942c161eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.259+0000 7f9431a06640 1 --2- 172.21.15.44:0/1887714249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f942c0cfc40 0x7f942c161eb0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f942c0ceb70 tx=0x7f941c02bdb0 comp rx=0 tx=0).stop 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.259+0000 7f9431a06640 1 -- 172.21.15.44:0/1887714249 shutdown_connections 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.259+0000 7f9431a06640 1 --2- 172.21.15.44:0/1887714249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f942c0cfc40 0x7f942c161eb0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.259+0000 7f9431a06640 1 --2- 172.21.15.44:0/1887714249 >> v2:172.21.15.69:3300/0 conn(0x7f942c0721f0 0x7f942c0cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.259+0000 7f9431a06640 1 --2- 172.21.15.44:0/1887714249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f942c15e420 0x7f942c0ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.259+0000 7f9431a06640 1 -- 172.21.15.44:0/1887714249 >> 172.21.15.44:0/1887714249 conn(0x7f942c0b2a10 msgr2=0x7f942c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.259+0000 7f9431a06640 1 -- 172.21.15.44:0/1887714249 shutdown_connections 2024-04-02T13:31:11.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.259+0000 7f9431a06640 1 -- 172.21.15.44:0/1887714249 wait complete. 2024-04-02T13:31:11.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.260+0000 7f9431a06640 1 Processor -- start 2024-04-02T13:31:11.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.260+0000 7f9431a06640 1 -- start start 2024-04-02T13:31:11.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f9431a06640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f942c0721f0 0x7f942c1554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:11.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f9431a06640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f942c0cfc40 0x7f942c14ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:11.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f9431a06640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942c15e420 0x7f942c0b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:11.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f9431a06640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f942c15f500 con 0x7f942c0721f0 2024-04-02T13:31:11.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f9431a06640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f942c15f220 con 0x7f942c15e420 2024-04-02T13:31:11.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f9431a06640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f942c0cf1e0 con 0x7f942c0cfc40 2024-04-02T13:31:11.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f942affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f942c0721f0 0x7f942c1554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:11.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f942affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f942c0721f0 0x7f942c1554b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44622/0 (socket says 172.21.15.44:44622) 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f942affd640 1 -- 172.21.15.44:0/3852946973 learned_addr learned my addr 172.21.15.44:0/3852946973 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f942a7fc640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f942c0cfc40 0x7f942c14ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f942b7fe640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942c15e420 0x7f942c0b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f942affd640 1 -- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f942c0cfc40 msgr2=0x7f942c14ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f942affd640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f942c0cfc40 0x7f942c14ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f942affd640 1 -- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942c15e420 msgr2=0x7f942c0b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f942affd640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942c15e420 0x7f942c0b3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f942affd640 1 -- 172.21.15.44:0/3852946973 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942c14cf40 con 0x7f942c0721f0 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.261+0000 7f942affd640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f942c0721f0 0x7f942c1554b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f942400e7e0 tx=0x7f942400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.262+0000 7f9423fff640 1 -- 172.21.15.44:0/3852946973 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942400f040 con 0x7f942c0721f0 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.262+0000 7f9423fff640 1 -- 172.21.15.44:0/3852946973 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9424004030 con 0x7f942c0721f0 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.262+0000 7f9423fff640 1 -- 172.21.15.44:0/3852946973 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424004400 con 0x7f942c0721f0 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.262+0000 7f9431a06640 1 -- 172.21.15.44:0/3852946973 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942c15c5c0 con 0x7f942c0721f0 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.262+0000 7f9431a06640 1 -- 172.21.15.44:0/3852946973 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f942c155d70 con 0x7f942c0721f0 2024-04-02T13:31:11.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.263+0000 7f9431a06640 1 -- 172.21.15.44:0/3852946973 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f942c0684f0 con 0x7f942c0721f0 2024-04-02T13:31:11.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.263+0000 7f9423fff640 1 -- 172.21.15.44:0/3852946973 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f942400c6d0 con 0x7f942c0721f0 2024-04-02T13:31:11.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.264+0000 7f9423fff640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f940c06c8f0 0x7f940c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:11.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.264+0000 7f9423fff640 1 -- 172.21.15.44:0/3852946973 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 6001+0+0 (secure 0 0 0) 0x7f942408ba10 con 0x7f942c0721f0 2024-04-02T13:31:11.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.264+0000 7f942a7fc640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f940c06c8f0 0x7f940c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:11.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.264+0000 7f942a7fc640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f940c06c8f0 0x7f940c06edb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9414006fd0 tx=0x7f9414008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:11.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.269+0000 7f9423fff640 1 -- 172.21.15.44:0/3852946973 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f942c14cf40 con 0x7f942c0721f0 2024-04-02T13:31:11.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:11.494+0000 7f9431a06640 1 -- 172.21.15.44:0/3852946973 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f942c0b3770 con 0x7f942c0721f0 2024-04-02T13:31:12.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.117+0000 7f9423fff640 1 -- 172.21.15.44:0/3852946973 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v96) ==== 174+0+0 (secure 0 0 0) 0x7f9424017020 con 0x7f942c0721f0 2024-04-02T13:31:12.120 INFO:teuthology.orchestra.run.smithi044.stderr:new fs with metadata pool 17 and data pool 18 2024-04-02T13:31:12.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.120+0000 7f9431a06640 1 -- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f940c06c8f0 msgr2=0x7f940c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.120+0000 7f9431a06640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f940c06c8f0 0x7f940c06edb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9414006fd0 tx=0x7f9414008040 comp rx=0 tx=0).stop 2024-04-02T13:31:12.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.120+0000 7f9431a06640 1 -- 172.21.15.44:0/3852946973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f942c0721f0 msgr2=0x7f942c1554b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.120+0000 7f9431a06640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f942c0721f0 0x7f942c1554b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f942400e7e0 tx=0x7f942400eca0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.120+0000 7f9431a06640 1 -- 172.21.15.44:0/3852946973 shutdown_connections 2024-04-02T13:31:12.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.120+0000 7f9431a06640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f940c06c8f0 0x7f940c06edb0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.120+0000 7f9431a06640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942c15e420 0x7f942c0b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.120+0000 7f9431a06640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f942c0cfc40 0x7f942c14ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.120+0000 7f9431a06640 1 --2- 172.21.15.44:0/3852946973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f942c0721f0 0x7f942c1554b0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.120+0000 7f9431a06640 1 -- 172.21.15.44:0/3852946973 >> 172.21.15.44:0/3852946973 conn(0x7f942c0b2a10 msgr2=0x7f942c157bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:12.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.120+0000 7f9431a06640 1 -- 172.21.15.44:0/3852946973 shutdown_connections 2024-04-02T13:31:12.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.120+0000 7f9431a06640 1 -- 172.21.15.44:0/3852946973 wait complete. 2024-04-02T13:31:12.152 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:31:12.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.254+0000 7ffb19a2b640 1 Processor -- start 2024-04-02T13:31:12.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.254+0000 7ffb19a2b640 1 -- start start 2024-04-02T13:31:12.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb19a2b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ffb1415e260 0x7ffb140d2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb19a2b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ffb140cf7c0 0x7ffb140d0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb19a2b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb140cfc70 0x7ffb140d0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb19a2b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ffb140d3250 con 0x7ffb140cf7c0 2024-04-02T13:31:12.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb19a2b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ffb140d2880 con 0x7ffb1415e260 2024-04-02T13:31:12.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb19a2b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ffb140d1cb0 con 0x7ffb140cfc70 2024-04-02T13:31:12.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb0a7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ffb140cf7c0 0x7ffb140d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb0a7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ffb140cf7c0 0x7ffb140d0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44636/0 (socket says 172.21.15.44:44636) 2024-04-02T13:31:12.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb0a7fc640 1 -- 172.21.15.44:0/2931711270 learned_addr learned my addr 172.21.15.44:0/2931711270 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:12.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb0a7fc640 1 -- 172.21.15.44:0/2931711270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb140cfc70 msgr2=0x7ffb140d0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:12.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb0a7fc640 1 --2- 172.21.15.44:0/2931711270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb140cfc70 0x7ffb140d0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb12ffd640 1 --2- 172.21.15.44:0/2931711270 >> v2:172.21.15.69:3300/0 conn(0x7ffb1415e260 0x7ffb140d2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb0a7fc640 1 -- 172.21.15.44:0/2931711270 >> v2:172.21.15.69:3300/0 conn(0x7ffb1415e260 msgr2=0x7ffb140d2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:12.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb0a7fc640 1 --2- 172.21.15.44:0/2931711270 >> v2:172.21.15.69:3300/0 conn(0x7ffb1415e260 0x7ffb140d2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.255+0000 7ffb0a7fc640 1 -- 172.21.15.44:0/2931711270 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb140d1f50 con 0x7ffb140cf7c0 2024-04-02T13:31:12.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.256+0000 7ffb0a7fc640 1 --2- 172.21.15.44:0/2931711270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb140cf7c0 0x7ffb140d0970 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7ffb140b6c60 tx=0x7ffafc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212c05e19dbcbd02 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:12.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.256+0000 7ffb127fc640 1 -- 172.21.15.44:0/2931711270 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc035020 con 0x7ffb140cf7c0 2024-04-02T13:31:12.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.256+0000 7ffb127fc640 1 -- 172.21.15.44:0/2931711270 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffafc031870 con 0x7ffb140cf7c0 2024-04-02T13:31:12.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.256+0000 7ffb127fc640 1 -- 172.21.15.44:0/2931711270 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc031c40 con 0x7ffb140cf7c0 2024-04-02T13:31:12.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.257+0000 7ffb19a2b640 1 -- 172.21.15.44:0/2931711270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb140cf7c0 msgr2=0x7ffb140d0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.257+0000 7ffb19a2b640 1 --2- 172.21.15.44:0/2931711270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb140cf7c0 0x7ffb140d0970 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7ffb140b6c60 tx=0x7ffafc02b910 comp rx=0 tx=0).stop 2024-04-02T13:31:12.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.257+0000 7ffb19a2b640 1 -- 172.21.15.44:0/2931711270 shutdown_connections 2024-04-02T13:31:12.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.257+0000 7ffb19a2b640 1 --2- 172.21.15.44:0/2931711270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb140cfc70 0x7ffb140d0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.257+0000 7ffb19a2b640 1 --2- 172.21.15.44:0/2931711270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb140cf7c0 0x7ffb140d0970 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.257+0000 7ffb19a2b640 1 --2- 172.21.15.44:0/2931711270 >> v2:172.21.15.69:3300/0 conn(0x7ffb1415e260 0x7ffb140d2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.257+0000 7ffb19a2b640 1 -- 172.21.15.44:0/2931711270 >> 172.21.15.44:0/2931711270 conn(0x7ffb1409d430 msgr2=0x7ffb140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:12.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.257+0000 7ffb19a2b640 1 -- 172.21.15.44:0/2931711270 shutdown_connections 2024-04-02T13:31:12.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.257+0000 7ffb19a2b640 1 -- 172.21.15.44:0/2931711270 wait complete. 2024-04-02T13:31:12.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.258+0000 7ffb19a2b640 1 Processor -- start 2024-04-02T13:31:12.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.258+0000 7ffb19a2b640 1 -- start start 2024-04-02T13:31:12.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb19a2b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb140cf7c0 0x7ffb14158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb19a2b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb140cfc70 0x7ffb14160380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb19a2b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb1415e260 0x7ffb140c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb19a2b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffb140d4a10 con 0x7ffb140cfc70 2024-04-02T13:31:12.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb19a2b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffb140d2510 con 0x7ffb140cf7c0 2024-04-02T13:31:12.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb19a2b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ffb1415e640 con 0x7ffb1415e260 2024-04-02T13:31:12.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb0a7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb140cfc70 0x7ffb14160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb0a7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb140cfc70 0x7ffb14160380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44648/0 (socket says 172.21.15.44:44648) 2024-04-02T13:31:12.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb0a7fc640 1 -- 172.21.15.44:0/3187604650 learned_addr learned my addr 172.21.15.44:0/3187604650 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:12.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb12ffd640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb140cf7c0 0x7ffb14158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb0a7fc640 1 -- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb1415e260 msgr2=0x7ffb140c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb137fe640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb1415e260 0x7ffb140c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb0a7fc640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb1415e260 0x7ffb140c98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb0a7fc640 1 -- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb140cf7c0 msgr2=0x7ffb14158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb0a7fc640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb140cf7c0 0x7ffb14158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb0a7fc640 1 -- 172.21.15.44:0/3187604650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb140cbf30 con 0x7ffb140cfc70 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb0a7fc640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb140cfc70 0x7ffb14160380 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7ffafc02b950 tx=0x7ffafc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb10ff9640 1 -- 172.21.15.44:0/3187604650 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc040070 con 0x7ffb140cfc70 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb10ff9640 1 -- 172.21.15.44:0/3187604650 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffafc03a420 con 0x7ffb140cfc70 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb19a2b640 1 -- 172.21.15.44:0/3187604650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb140cc210 con 0x7ffb140cfc70 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.259+0000 7ffb10ff9640 1 -- 172.21.15.44:0/3187604650 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc035040 con 0x7ffb140cfc70 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.260+0000 7ffb19a2b640 1 -- 172.21.15.44:0/3187604650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb140ccab0 con 0x7ffb140cfc70 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.261+0000 7ffb19a2b640 1 -- 172.21.15.44:0/3187604650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb140cc600 con 0x7ffb140cfc70 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.261+0000 7ffb10ff9640 1 -- 172.21.15.44:0/3187604650 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffafc038020 con 0x7ffb140cfc70 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.261+0000 7ffb10ff9640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ffaf406c8f0 0x7ffaf406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.261+0000 7ffb10ff9640 1 -- 172.21.15.44:0/3187604650 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6101+0+0 (secure 0 0 0) 0x7ffafc0adc80 con 0x7ffb140cfc70 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.261+0000 7ffb12ffd640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ffaf406c8f0 0x7ffaf406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.262+0000 7ffb12ffd640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ffaf406c8f0 0x7ffaf406edb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ffb04006fd0 tx=0x7ffb04008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:12.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.266+0000 7ffb10ff9640 1 -- 172.21.15.44:0/3187604650 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ffafc07bf20 con 0x7ffb140cfc70 2024-04-02T13:31:12.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.451+0000 7ffb19a2b640 1 -- 172.21.15.44:0/3187604650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffb140684f0 con 0x7ffb140cfc70 2024-04-02T13:31:12.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.453+0000 7ffb10ff9640 1 -- 172.21.15.44:0/3187604650 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) ==== 74+0+16917 (secure 0 0 0) 0x7ffafc07b820 con 0x7ffb140cfc70 2024-04-02T13:31:12.456 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:12.456 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":120,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:11.495381+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:31:09.752845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-02T13:31:10.437454+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:31:12.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.454+0000 7ffb19a2b640 1 -- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ffaf406c8f0 msgr2=0x7ffaf406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.454+0000 7ffb19a2b640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ffaf406c8f0 0x7ffaf406edb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ffb04006fd0 tx=0x7ffb04008040 comp rx=0 tx=0).stop 2024-04-02T13:31:12.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.454+0000 7ffb19a2b640 1 -- 172.21.15.44:0/3187604650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb140cfc70 msgr2=0x7ffb14160380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.454+0000 7ffb19a2b640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb140cfc70 0x7ffb14160380 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7ffafc02b950 tx=0x7ffafc0046c0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.455+0000 7ffb19a2b640 1 -- 172.21.15.44:0/3187604650 shutdown_connections 2024-04-02T13:31:12.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.455+0000 7ffb19a2b640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ffaf406c8f0 0x7ffaf406edb0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.455+0000 7ffb19a2b640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb1415e260 0x7ffb140c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.455+0000 7ffb19a2b640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb140cfc70 0x7ffb14160380 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.455+0000 7ffb19a2b640 1 --2- 172.21.15.44:0/3187604650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb140cf7c0 0x7ffb14158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.455+0000 7ffb19a2b640 1 -- 172.21.15.44:0/3187604650 >> 172.21.15.44:0/3187604650 conn(0x7ffb1409d430 msgr2=0x7ffb140b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:12.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.455+0000 7ffb19a2b640 1 -- 172.21.15.44:0/3187604650 shutdown_connections 2024-04-02T13:31:12.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.455+0000 7ffb19a2b640 1 -- 172.21.15.44:0/3187604650 wait complete. 2024-04-02T13:31:12.469 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:31:12.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.566+0000 7f8773150640 1 Processor -- start 2024-04-02T13:31:12.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.566+0000 7f8773150640 1 -- start start 2024-04-02T13:31:12.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.567+0000 7f8773150640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f876c06cf90 0x7f876c0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.567+0000 7f8773150640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f876c0cfe30 0x7f876c0d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.567+0000 7f8773150640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f876c06c1d0 con 0x7f876c0d0a30 2024-04-02T13:31:12.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.567+0000 7f8773150640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f876c06cb00 con 0x7f876c06cf90 2024-04-02T13:31:12.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.567+0000 7f8773150640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f876c06d3f0 con 0x7f876c0cfe30 2024-04-02T13:31:12.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.567+0000 7f876bfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f876c06cf90 0x7f876c0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.567+0000 7f8770eec640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f876c0d0a30 0x7f876c0d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48726/0 (socket says 172.21.15.44:48726) 2024-04-02T13:31:12.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.567+0000 7f8770eec640 1 -- 172.21.15.44:0/2595798202 learned_addr learned my addr 172.21.15.44:0/2595798202 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.568+0000 7f87716ed640 1 --2- 172.21.15.44:0/2595798202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f876c0cfe30 0x7f876c0d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.568+0000 7f876bfff640 1 -- 172.21.15.44:0/2595798202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f876c0cfe30 msgr2=0x7f876c0d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.568+0000 7f876bfff640 1 --2- 172.21.15.44:0/2595798202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f876c0cfe30 0x7f876c0d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.568+0000 7f876bfff640 1 -- 172.21.15.44:0/2595798202 >> v1:172.21.15.44:6789/0 conn(0x7f876c0d0a30 legacy=0x7f876c0d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.568+0000 7f876bfff640 1 -- 172.21.15.44:0/2595798202 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876c06c4b0 con 0x7f876c06cf90 2024-04-02T13:31:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.568+0000 7f87716ed640 1 --2- 172.21.15.44:0/2595798202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f876c0cfe30 0x7f876c0d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.568+0000 7f876bfff640 1 --2- 172.21.15.44:0/2595798202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f876c06cf90 0x7f876c0cf8f0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f87580097a0 tx=0x7f875802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b94d34e415dd10 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.569+0000 7f876b7fe640 1 -- 172.21.15.44:0/2595798202 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87580315a0 con 0x7f876c06cf90 2024-04-02T13:31:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.569+0000 7f876b7fe640 1 -- 172.21.15.44:0/2595798202 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8758037070 con 0x7f876c06cf90 2024-04-02T13:31:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.569+0000 7f876b7fe640 1 -- 172.21.15.44:0/2595798202 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8758031960 con 0x7f876c06cf90 2024-04-02T13:31:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.569+0000 7f8773150640 1 -- 172.21.15.44:0/2595798202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f876c06cf90 msgr2=0x7f876c0cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.569+0000 7f8773150640 1 --2- 172.21.15.44:0/2595798202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f876c06cf90 0x7f876c0cf8f0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f87580097a0 tx=0x7f875802b840 comp rx=0 tx=0).stop 2024-04-02T13:31:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.569+0000 7f8773150640 1 -- 172.21.15.44:0/2595798202 shutdown_connections 2024-04-02T13:31:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.569+0000 7f8773150640 1 --2- 172.21.15.44:0/2595798202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f876c0cfe30 0x7f876c0d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.569+0000 7f8773150640 1 --2- 172.21.15.44:0/2595798202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f876c06cf90 0x7f876c0cf8f0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.569+0000 7f8773150640 1 -- 172.21.15.44:0/2595798202 >> 172.21.15.44:0/2595798202 conn(0x7f876c09d470 msgr2=0x7f876c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.569+0000 7f8773150640 1 -- 172.21.15.44:0/2595798202 shutdown_connections 2024-04-02T13:31:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.569+0000 7f8773150640 1 -- 172.21.15.44:0/2595798202 wait complete. 2024-04-02T13:31:12.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.570+0000 7f8773150640 1 Processor -- start 2024-04-02T13:31:12.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.570+0000 7f8773150640 1 -- start start 2024-04-02T13:31:12.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.571+0000 7f8773150640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f876c06cf90 0x7f876c154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.571+0000 7f8773150640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f876c0cfe30 0x7f876c0b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.571+0000 7f8773150640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f876c0d0a30 0x7f876c0b8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.571+0000 7f8773150640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f876c15e870 con 0x7f876c0d0a30 2024-04-02T13:31:12.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.571+0000 7f8773150640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f876c15e6f0 con 0x7f876c06cf90 2024-04-02T13:31:12.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.571+0000 7f8773150640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f876c0d4da0 con 0x7f876c0cfe30 2024-04-02T13:31:12.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.571+0000 7f87716ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f876c0d0a30 0x7f876c0b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.571+0000 7f87716ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f876c0d0a30 0x7f876c0b8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44676/0 (socket says 172.21.15.44:44676) 2024-04-02T13:31:12.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.571+0000 7f87716ed640 1 -- 172.21.15.44:0/3239869883 learned_addr learned my addr 172.21.15.44:0/3239869883 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:12.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.571+0000 7f8770eec640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f876c06cf90 0x7f876c154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f876bfff640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f876c0cfe30 0x7f876c0b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f87716ed640 1 -- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f876c0cfe30 msgr2=0x7f876c0b91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f87716ed640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f876c0cfe30 0x7f876c0b91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f87716ed640 1 -- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f876c06cf90 msgr2=0x7f876c154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f87716ed640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f876c06cf90 0x7f876c154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f87716ed640 1 -- 172.21.15.44:0/3239869883 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876c0b5d80 con 0x7f876c0d0a30 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f8770eec640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f876c06cf90 0x7f876c154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f876bfff640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f876c0cfe30 0x7f876c0b91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f87716ed640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f876c0d0a30 0x7f876c0b8050 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f876400cd80 tx=0x7f876400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f87697fa640 1 -- 172.21.15.44:0/3239869883 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876400f680 con 0x7f876c0d0a30 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f87697fa640 1 -- 172.21.15.44:0/3239869883 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f876400f840 con 0x7f876c0d0a30 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f8773150640 1 -- 172.21.15.44:0/3239869883 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f876c0b6020 con 0x7f876c0d0a30 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f87697fa640 1 -- 172.21.15.44:0/3239869883 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876400fc30 con 0x7f876c0d0a30 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.572+0000 7f8773150640 1 -- 172.21.15.44:0/3239869883 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f876c0b4e40 con 0x7f876c0d0a30 2024-04-02T13:31:12.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.573+0000 7f8773150640 1 -- 172.21.15.44:0/3239869883 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f876c0684f0 con 0x7f876c0d0a30 2024-04-02T13:31:12.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.573+0000 7f87697fa640 1 -- 172.21.15.44:0/3239869883 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f87640213f0 con 0x7f876c0d0a30 2024-04-02T13:31:12.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.574+0000 7f87697fa640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f875006c8f0 0x7f875006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.574+0000 7f87697fa640 1 -- 172.21.15.44:0/3239869883 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6101+0+0 (secure 0 0 0) 0x7f876408ca70 con 0x7f876c0d0a30 2024-04-02T13:31:12.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.574+0000 7f8770eec640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f875006c8f0 0x7f875006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.574+0000 7f8770eec640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f875006c8f0 0x7f875006edb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f876c0b5cb0 tx=0x7f8760009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:12.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.578+0000 7f87697fa640 1 -- 172.21.15.44:0/3239869883 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8764014030 con 0x7f876c0d0a30 2024-04-02T13:31:12.767 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:31:12.764+0000 7fe30526c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:31:12.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.765+0000 7f8773150640 1 -- 172.21.15.44:0/3239869883 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f876c0d17c0 con 0x7f876c0d0a30 2024-04-02T13:31:12.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.766+0000 7f87697fa640 1 -- 172.21.15.44:0/3239869883 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) ==== 74+0+16917 (secure 0 0 0) 0x7f876405a730 con 0x7f876c0d0a30 2024-04-02T13:31:12.769 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:12.770 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":120,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:11.495381+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:31:09.752845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-02T13:31:10.437454+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:31:12.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.768+0000 7f8773150640 1 -- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f875006c8f0 msgr2=0x7f875006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.768+0000 7f8773150640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f875006c8f0 0x7f875006edb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f876c0b5cb0 tx=0x7f8760009040 comp rx=0 tx=0).stop 2024-04-02T13:31:12.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.769+0000 7f8773150640 1 -- 172.21.15.44:0/3239869883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f876c0d0a30 msgr2=0x7f876c0b8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.769+0000 7f8773150640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f876c0d0a30 0x7f876c0b8050 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f876400cd80 tx=0x7f876400a650 comp rx=0 tx=0).stop 2024-04-02T13:31:12.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.769+0000 7f8773150640 1 -- 172.21.15.44:0/3239869883 shutdown_connections 2024-04-02T13:31:12.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.769+0000 7f8773150640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f875006c8f0 0x7f875006edb0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.769+0000 7f8773150640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f876c0d0a30 0x7f876c0b8050 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.769+0000 7f8773150640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f876c0cfe30 0x7f876c0b91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.769+0000 7f8773150640 1 --2- 172.21.15.44:0/3239869883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f876c06cf90 0x7f876c154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.769+0000 7f8773150640 1 -- 172.21.15.44:0/3239869883 >> 172.21.15.44:0/3239869883 conn(0x7f876c09d470 msgr2=0x7f876c0b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:12.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.769+0000 7f8773150640 1 -- 172.21.15.44:0/3239869883 shutdown_connections 2024-04-02T13:31:12.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.769+0000 7f8773150640 1 -- 172.21.15.44:0/3239869883 wait complete. 2024-04-02T13:31:12.783 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T13:31:12.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.884+0000 7f4eafd50640 1 Processor -- start 2024-04-02T13:31:12.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.884+0000 7f4eafd50640 1 -- start start 2024-04-02T13:31:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.885+0000 7f4eafd50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ea815e550 0x7f4ea806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.885+0000 7f4eafd50640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4ea815f040 0x7f4ea80d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.885+0000 7f4eafd50640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ea815fb50 con 0x7f4ea806d530 2024-04-02T13:31:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.885+0000 7f4eafd50640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4ea806d970 con 0x7f4ea815f040 2024-04-02T13:31:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.885+0000 7f4eafd50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4ea806ca90 con 0x7f4ea815e550 2024-04-02T13:31:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.885+0000 7f4ead2eb640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4ea806d530 0x7f4ea815e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48734/0 (socket says 172.21.15.44:48734) 2024-04-02T13:31:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.885+0000 7f4ead2eb640 1 -- 172.21.15.44:0/2508733798 learned_addr learned my addr 172.21.15.44:0/2508733798 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.885+0000 7f4eae2ed640 1 --2- 172.21.15.44:0/2508733798 >> v2:172.21.15.69:3300/0 conn(0x7f4ea815f040 0x7f4ea80d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.885+0000 7f4eadaec640 1 --2- 172.21.15.44:0/2508733798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ea815e550 0x7f4ea806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.885+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 835974864 0 0) 0x7f4ea815fb50 con 0x7f4ea806d530 2024-04-02T13:31:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.885+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e90003520 con 0x7f4ea806d530 2024-04-02T13:31:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 760445493 0 0) 0x7f4e90003520 con 0x7f4ea806d530 2024-04-02T13:31:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ea815e550 msgr2=0x7f4ea806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 --2- 172.21.15.44:0/2508733798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ea815e550 0x7f4ea806cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 >> v2:172.21.15.69:3300/0 conn(0x7f4ea815f040 msgr2=0x7f4ea80d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 --2- 172.21.15.44:0/2508733798 >> v2:172.21.15.69:3300/0 conn(0x7f4ea815f040 0x7f4ea80d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea815f590 con 0x7f4ea806d530 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f4e9c001670 con 0x7f4ea806d530 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eae2ed640 1 --2- 172.21.15.44:0/2508733798 >> v2:172.21.15.69:3300/0 conn(0x7f4ea815f040 0x7f4ea80d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 >> v1:172.21.15.44:6789/0 conn(0x7f4ea806d530 legacy=0x7f4ea815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 --2- 172.21.15.44:0/2508733798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e900039b0 0x7f4e90003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 --2- 172.21.15.44:0/2508733798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ea815e550 0x7f4e90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 --2- 172.21.15.44:0/2508733798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90004d40 0x7f4e900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea806d970 con 0x7f4e900039b0 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea806ca90 con 0x7f4e90004d40 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9000b9c0 con 0x7f4ea815e550 2024-04-02T13:31:12.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eadaec640 1 --2- 172.21.15.44:0/2508733798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e900039b0 0x7f4e90003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eae2ed640 1 --2- 172.21.15.44:0/2508733798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90004d40 0x7f4e900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4ead2eb640 1 --2- 172.21.15.44:0/2508733798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ea815e550 0x7f4e90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eadaec640 1 -- 172.21.15.44:0/2508733798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ea815e550 msgr2=0x7f4e90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eadaec640 1 --2- 172.21.15.44:0/2508733798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ea815e550 0x7f4e90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eadaec640 1 -- 172.21.15.44:0/2508733798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90004d40 msgr2=0x7f4e900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eadaec640 1 --2- 172.21.15.44:0/2508733798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90004d40 0x7f4e900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.886+0000 7f4eadaec640 1 -- 172.21.15.44:0/2508733798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e90003520 con 0x7f4e900039b0 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.887+0000 7f4eadaec640 1 --2- 172.21.15.44:0/2508733798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e900039b0 0x7f4e90003dd0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0009ac0 tx=0x7f4e90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd3db9d68a06a75 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.887+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0039070 con 0x7f4e900039b0 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.887+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ea002c6b0 con 0x7f4e900039b0 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.887+0000 7f4eacaea640 1 -- 172.21.15.44:0/2508733798 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0009690 con 0x7f4e900039b0 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.888+0000 7f4eafd50640 1 -- 172.21.15.44:0/2508733798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e900039b0 msgr2=0x7f4e90003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.888+0000 7f4eafd50640 1 --2- 172.21.15.44:0/2508733798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e900039b0 0x7f4e90003dd0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0009ac0 tx=0x7f4e90005830 comp rx=0 tx=0).stop 2024-04-02T13:31:12.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.888+0000 7f4eae2ed640 1 -- 172.21.15.44:0/2508733798 reap_dead start 2024-04-02T13:31:12.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.888+0000 7f4eafd50640 1 -- 172.21.15.44:0/2508733798 shutdown_connections 2024-04-02T13:31:12.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.888+0000 7f4eafd50640 1 -- 172.21.15.44:0/2508733798 >> 172.21.15.44:0/2508733798 conn(0x7f4ea809aa20 msgr2=0x7f4ea809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:12.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.888+0000 7f4eafd50640 1 -- 172.21.15.44:0/2508733798 shutdown_connections 2024-04-02T13:31:12.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.888+0000 7f4eafd50640 1 -- 172.21.15.44:0/2508733798 wait complete. 2024-04-02T13:31:12.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.889+0000 7f4eafd50640 1 Processor -- start 2024-04-02T13:31:12.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.889+0000 7f4eafd50640 1 -- start start 2024-04-02T13:31:12.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eafd50640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e900039b0 0x7f4ea81f7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eafd50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ea81f7d90 0x7f4ea80cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eafd50640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ea80cb060 0x7f4ea80b6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eafd50640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea80d0590 con 0x7f4e900039b0 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eafd50640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea80d02b0 con 0x7f4ea80cb060 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eafd50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4ea80cc4b0 con 0x7f4ea81f7d90 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eadaec640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e900039b0 0x7f4ea81f7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eadaec640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e900039b0 0x7f4ea81f7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44700/0 (socket says 172.21.15.44:44700) 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eadaec640 1 -- 172.21.15.44:0/489945737 learned_addr learned my addr 172.21.15.44:0/489945737 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eae2ed640 1 --2- 172.21.15.44:0/489945737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ea80cb060 0x7f4ea80b6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eadaec640 1 -- 172.21.15.44:0/489945737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ea81f7d90 msgr2=0x7f4ea80cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eadaec640 1 --2- 172.21.15.44:0/489945737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ea81f7d90 0x7f4ea80cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eadaec640 1 -- 172.21.15.44:0/489945737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ea80cb060 msgr2=0x7f4ea80b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eadaec640 1 --2- 172.21.15.44:0/489945737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ea80cb060 0x7f4ea80b6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:12.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eadaec640 1 -- 172.21.15.44:0/489945737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea81f54c0 con 0x7f4e900039b0 2024-04-02T13:31:12.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eadaec640 1 --2- 172.21.15.44:0/489945737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e900039b0 0x7f4ea81f7850 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0037040 tx=0x7f4ea0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:12.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4ea67fc640 1 -- 172.21.15.44:0/489945737 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0042020 con 0x7f4e900039b0 2024-04-02T13:31:12.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4ea67fc640 1 -- 172.21.15.44:0/489945737 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ea003a040 con 0x7f4e900039b0 2024-04-02T13:31:12.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eafd50640 1 -- 172.21.15.44:0/489945737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ea81f5760 con 0x7f4e900039b0 2024-04-02T13:31:12.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4ea67fc640 1 -- 172.21.15.44:0/489945737 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0039070 con 0x7f4e900039b0 2024-04-02T13:31:12.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.890+0000 7f4eafd50640 1 -- 172.21.15.44:0/489945737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ea80cb880 con 0x7f4e900039b0 2024-04-02T13:31:12.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.891+0000 7f4eafd50640 1 -- 172.21.15.44:0/489945737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ea80684f0 con 0x7f4e900039b0 2024-04-02T13:31:12.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.892+0000 7f4ea67fc640 1 -- 172.21.15.44:0/489945737 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4ea0045020 con 0x7f4e900039b0 2024-04-02T13:31:12.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.892+0000 7f4ea67fc640 1 --2- 172.21.15.44:0/489945737 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e9406c920 0x7f4e9406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:12.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.892+0000 7f4ea67fc640 1 -- 172.21.15.44:0/489945737 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6101+0+0 (secure 0 0 0) 0x7f4ea00ad200 con 0x7f4e900039b0 2024-04-02T13:31:12.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.892+0000 7f4ead2eb640 1 --2- 172.21.15.44:0/489945737 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e9406c920 0x7f4e9406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:12.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.893+0000 7f4ead2eb640 1 --2- 172.21.15.44:0/489945737 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e9406c920 0x7f4e9406ede0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c000c00 tx=0x7f4e9c004bb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:12.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:12.897+0000 7f4ea67fc640 1 -- 172.21.15.44:0/489945737 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4ea004c070 con 0x7f4e900039b0 2024-04-02T13:31:13.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.120+0000 7f4eafd50640 1 -- 172.21.15.44:0/489945737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f4ea80cdcc0 con 0x7f4e900039b0 2024-04-02T13:31:13.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.472+0000 7f4ea67fc640 1 -- 172.21.15.44:0/489945737 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v98) ==== 118+0+0 (secure 0 0 0) 0x7f4ea007b400 con 0x7f4e900039b0 2024-04-02T13:31:13.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.474+0000 7f4eafd50640 1 -- 172.21.15.44:0/489945737 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e9406c920 msgr2=0x7f4e9406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:13.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.474+0000 7f4eafd50640 1 --2- 172.21.15.44:0/489945737 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e9406c920 0x7f4e9406ede0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c000c00 tx=0x7f4e9c004bb0 comp rx=0 tx=0).stop 2024-04-02T13:31:13.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.474+0000 7f4eafd50640 1 -- 172.21.15.44:0/489945737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e900039b0 msgr2=0x7f4ea81f7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:13.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.474+0000 7f4eafd50640 1 --2- 172.21.15.44:0/489945737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e900039b0 0x7f4ea81f7850 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0037040 tx=0x7f4ea0004730 comp rx=0 tx=0).stop 2024-04-02T13:31:13.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.474+0000 7f4eafd50640 1 -- 172.21.15.44:0/489945737 shutdown_connections 2024-04-02T13:31:13.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.474+0000 7f4eafd50640 1 --2- 172.21.15.44:0/489945737 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4e9406c920 0x7f4e9406ede0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:13.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.474+0000 7f4eafd50640 1 --2- 172.21.15.44:0/489945737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ea80cb060 0x7f4ea80b6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:13.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.474+0000 7f4eafd50640 1 --2- 172.21.15.44:0/489945737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ea81f7d90 0x7f4ea80cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:13.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.474+0000 7f4eafd50640 1 --2- 172.21.15.44:0/489945737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e900039b0 0x7f4ea81f7850 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:13.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.474+0000 7f4eafd50640 1 -- 172.21.15.44:0/489945737 >> 172.21.15.44:0/489945737 conn(0x7f4ea809aa20 msgr2=0x7f4ea80cd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:13.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.474+0000 7f4eafd50640 1 -- 172.21.15.44:0/489945737 shutdown_connections 2024-04-02T13:31:13.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.474+0000 7f4eafd50640 1 -- 172.21.15.44:0/489945737 wait complete. 2024-04-02T13:31:13.488 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T13:31:13.488 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T13:31:13.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.591+0000 7f6b1270e640 1 Processor -- start 2024-04-02T13:31:13.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.591+0000 7f6b1270e640 1 -- start start 2024-04-02T13:31:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.591+0000 7f6b1270e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6b0c0c8c80 0x7f6b0c1692c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.591+0000 7f6b1270e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b0c0d2d20 con 0x7f6b0c161ab0 2024-04-02T13:31:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.591+0000 7f6b1270e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b0c0b5500 con 0x7f6b0c0d4c40 2024-04-02T13:31:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.591+0000 7f6b1270e640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f6b0c0d50d0 con 0x7f6b0c0c8c80 2024-04-02T13:31:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.592+0000 7f6b10cab640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6b0c0c8c80 0x7f6b0c1692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.592+0000 7f6b10cab640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f6b0c0c8c80 0x7f6b0c1692c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44714/0 (socket says 172.21.15.44:44714) 2024-04-02T13:31:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.592+0000 7f6b0bfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6b0c161ab0 0x7f6b0c0d1f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48736/0 (socket says 172.21.15.44:48736) 2024-04-02T13:31:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.592+0000 7f6b10cab640 1 -- 172.21.15.44:0/3926518878 learned_addr learned my addr 172.21.15.44:0/3926518878 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.592+0000 7f6b10cab640 1 -- 172.21.15.44:0/3926518878 >> v1:172.21.15.69:6789/0 conn(0x7f6b0c0d4c40 legacy=0x7f6b0c0d2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.592+0000 7f6b10cab640 1 -- 172.21.15.44:0/3926518878 >> v1:172.21.15.44:6789/0 conn(0x7f6b0c161ab0 legacy=0x7f6b0c0d1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.592+0000 7f6b10cab640 1 -- 172.21.15.44:0/3926518878 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0c0d2fa0 con 0x7f6b0c0c8c80 2024-04-02T13:31:13.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.593+0000 7f6b10cab640 1 --2- 172.21.15.44:0/3926518878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b0c0c8c80 0x7f6b0c1692c0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009e50 tx=0x7f6b0402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2af659267b11eeb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:13.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.593+0000 7f6b0affd640 1 -- 172.21.15.44:0/3926518878 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0402cbb0 con 0x7f6b0c0c8c80 2024-04-02T13:31:13.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.593+0000 7f6b0affd640 1 -- 172.21.15.44:0/3926518878 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b0402cd50 con 0x7f6b0c0c8c80 2024-04-02T13:31:13.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.593+0000 7f6b0affd640 1 -- 172.21.15.44:0/3926518878 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b04034610 con 0x7f6b0c0c8c80 2024-04-02T13:31:13.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.593+0000 7f6b1270e640 1 -- 172.21.15.44:0/3926518878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b0c0c8c80 msgr2=0x7f6b0c1692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:13.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.593+0000 7f6b1270e640 1 --2- 172.21.15.44:0/3926518878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b0c0c8c80 0x7f6b0c1692c0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009e50 tx=0x7f6b0402c010 comp rx=0 tx=0).stop 2024-04-02T13:31:13.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.593+0000 7f6b1270e640 1 -- 172.21.15.44:0/3926518878 shutdown_connections 2024-04-02T13:31:13.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.593+0000 7f6b1270e640 1 --2- 172.21.15.44:0/3926518878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b0c0c8c80 0x7f6b0c1692c0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:13.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.593+0000 7f6b1270e640 1 -- 172.21.15.44:0/3926518878 >> 172.21.15.44:0/3926518878 conn(0x7f6b0c0d4120 msgr2=0x7f6b0c0d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:13.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.593+0000 7f6b1270e640 1 -- 172.21.15.44:0/3926518878 shutdown_connections 2024-04-02T13:31:13.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.593+0000 7f6b1270e640 1 -- 172.21.15.44:0/3926518878 wait complete. 2024-04-02T13:31:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.594+0000 7f6b1270e640 1 Processor -- start 2024-04-02T13:31:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.595+0000 7f6b1270e640 1 -- start start 2024-04-02T13:31:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.595+0000 7f6b1270e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b0c0c8c80 0x7f6b0c0b8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.595+0000 7f6b1270e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0d4c40 0x7f6b0c14b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.595+0000 7f6b1270e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b0c161ab0 0x7f6b0c15d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.595+0000 7f6b1270e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c0d0070 con 0x7f6b0c161ab0 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.595+0000 7f6b1270e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c0cfd90 con 0x7f6b0c0d4c40 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.595+0000 7f6b1270e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6b0c0cf7a0 con 0x7f6b0c0c8c80 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.595+0000 7f6b10cab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b0c161ab0 0x7f6b0c15d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b10cab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b0c161ab0 0x7f6b0c15d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44726/0 (socket says 172.21.15.44:44726) 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b10cab640 1 -- 172.21.15.44:0/3493597502 learned_addr learned my addr 172.21.15.44:0/3493597502 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b0b7fe640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0d4c40 0x7f6b0c14b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b0bfff640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b0c0c8c80 0x7f6b0c0b8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b10cab640 1 -- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b0c0c8c80 msgr2=0x7f6b0c0b8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b10cab640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b0c0c8c80 0x7f6b0c0b8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b10cab640 1 -- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0d4c40 msgr2=0x7f6b0c14b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b10cab640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0d4c40 0x7f6b0c14b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b10cab640 1 -- 172.21.15.44:0/3493597502 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0c0cb680 con 0x7f6b0c161ab0 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b0b7fe640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0d4c40 0x7f6b0c14b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:13.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b0bfff640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b0c0c8c80 0x7f6b0c0b8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:13.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b10cab640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b0c161ab0 0x7f6b0c15d030 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009e50 tx=0x7f6b04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:13.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b08ff9640 1 -- 172.21.15.44:0/3493597502 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b04043020 con 0x7f6b0c161ab0 2024-04-02T13:31:13.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b08ff9640 1 -- 172.21.15.44:0/3493597502 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b0403e450 con 0x7f6b0c161ab0 2024-04-02T13:31:13.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b1270e640 1 -- 172.21.15.44:0/3493597502 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b0c0cb960 con 0x7f6b0c161ab0 2024-04-02T13:31:13.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b08ff9640 1 -- 172.21.15.44:0/3493597502 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b04009510 con 0x7f6b0c161ab0 2024-04-02T13:31:13.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.596+0000 7f6b1270e640 1 -- 172.21.15.44:0/3493597502 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b0c0cbf20 con 0x7f6b0c161ab0 2024-04-02T13:31:13.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.597+0000 7f6b1270e640 1 -- 172.21.15.44:0/3493597502 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b0c072fd0 con 0x7f6b0c161ab0 2024-04-02T13:31:13.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.598+0000 7f6b08ff9640 1 -- 172.21.15.44:0/3493597502 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6b04046020 con 0x7f6b0c161ab0 2024-04-02T13:31:13.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.598+0000 7f6b08ff9640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6af806c8f0 0x7f6af806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:13.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.598+0000 7f6b08ff9640 1 -- 172.21.15.44:0/3493597502 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f6b040ae620 con 0x7f6b0c161ab0 2024-04-02T13:31:13.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.598+0000 7f6b0bfff640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6af806c8f0 0x7f6af806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:13.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.599+0000 7f6b0bfff640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6af806c8f0 0x7f6af806edb0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f6afc000f60 tx=0x7f6afc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:13.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.603+0000 7f6b08ff9640 1 -- 172.21.15.44:0/3493597502 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6b0c072fd0 con 0x7f6b0c161ab0 2024-04-02T13:31:13.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:13.825+0000 7f6b1270e640 1 -- 172.21.15.44:0/3493597502 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f6b0c0caa80 con 0x7f6b0c161ab0 2024-04-02T13:31:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.475+0000 7f6b08ff9640 1 -- 172.21.15.44:0/3493597502 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v99) ==== 122+0+0 (secure 0 0 0) 0x7f6b0404d030 con 0x7f6b0c161ab0 2024-04-02T13:31:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.478+0000 7f6b1270e640 1 -- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6af806c8f0 msgr2=0x7f6af806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.478+0000 7f6b1270e640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6af806c8f0 0x7f6af806edb0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f6afc000f60 tx=0x7f6afc009040 comp rx=0 tx=0).stop 2024-04-02T13:31:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.478+0000 7f6b1270e640 1 -- 172.21.15.44:0/3493597502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b0c161ab0 msgr2=0x7f6b0c15d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.478+0000 7f6b1270e640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b0c161ab0 0x7f6b0c15d030 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009e50 tx=0x7f6b04004a40 comp rx=0 tx=0).stop 2024-04-02T13:31:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.478+0000 7f6b1270e640 1 -- 172.21.15.44:0/3493597502 shutdown_connections 2024-04-02T13:31:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.478+0000 7f6b1270e640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6af806c8f0 0x7f6af806edb0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.478+0000 7f6b1270e640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b0c161ab0 0x7f6b0c15d030 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.478+0000 7f6b1270e640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0d4c40 0x7f6b0c14b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.478+0000 7f6b1270e640 1 --2- 172.21.15.44:0/3493597502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b0c0c8c80 0x7f6b0c0b8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.478+0000 7f6b1270e640 1 -- 172.21.15.44:0/3493597502 >> 172.21.15.44:0/3493597502 conn(0x7f6b0c0d4120 msgr2=0x7f6b0c15eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.478+0000 7f6b1270e640 1 -- 172.21.15.44:0/3493597502 shutdown_connections 2024-04-02T13:31:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.478+0000 7f6b1270e640 1 -- 172.21.15.44:0/3493597502 wait complete. 2024-04-02T13:31:14.494 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T13:31:14.495 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:31:14.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.599+0000 7f9764d3d640 1 Processor -- start 2024-04-02T13:31:14.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.599+0000 7f9764d3d640 1 -- start start 2024-04-02T13:31:14.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.600+0000 7f9764d3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97600b2d00 0x7f97600cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:14.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.600+0000 7f9764d3d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f97600ce110 0x7f9760071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:14.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.600+0000 7f9764d3d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97600b3160 con 0x7f976015aa00 2024-04-02T13:31:14.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.600+0000 7f9764d3d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f976015ade0 con 0x7f97600ce110 2024-04-02T13:31:14.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.600+0000 7f9764d3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f97600af1b0 con 0x7f97600b2d00 2024-04-02T13:31:14.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.600+0000 7f975ed9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f97600ce110 0x7f9760071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:14.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.600+0000 7f975ed9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f97600ce110 0x7f9760071190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53554/0 (socket says 172.21.15.44:53554) 2024-04-02T13:31:14.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.600+0000 7f975ed9d640 1 -- 172.21.15.44:0/2407103413 learned_addr learned my addr 172.21.15.44:0/2407103413 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:14.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.600+0000 7f975dd9b640 1 --2- 172.21.15.44:0/2407103413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97600b2d00 0x7f97600cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:14.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.600+0000 7f975ed9d640 1 -- 172.21.15.44:0/2407103413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97600b2d00 msgr2=0x7f97600cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:14.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.601+0000 7f975ed9d640 1 --2- 172.21.15.44:0/2407103413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97600b2d00 0x7f97600cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.601+0000 7f975ed9d640 1 -- 172.21.15.44:0/2407103413 >> v1:172.21.15.69:6789/0 conn(0x7f976015aa00 legacy=0x7f9760072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:14.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.601+0000 7f975ed9d640 1 -- 172.21.15.44:0/2407103413 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97600af490 con 0x7f97600ce110 2024-04-02T13:31:14.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.601+0000 7f975ed9d640 1 --2- 172.21.15.44:0/2407103413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97600ce110 0x7f9760071190 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f9750009e50 tx=0x7f975002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f984507fd5749714 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:14.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.601+0000 7f975d59a640 1 -- 172.21.15.44:0/2407103413 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975002ca70 con 0x7f97600ce110 2024-04-02T13:31:14.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.601+0000 7f975d59a640 1 -- 172.21.15.44:0/2407103413 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f975002cc10 con 0x7f97600ce110 2024-04-02T13:31:14.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.602+0000 7f975d59a640 1 -- 172.21.15.44:0/2407103413 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97500343f0 con 0x7f97600ce110 2024-04-02T13:31:14.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.602+0000 7f9764d3d640 1 -- 172.21.15.44:0/2407103413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97600ce110 msgr2=0x7f9760071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:14.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.602+0000 7f9764d3d640 1 --2- 172.21.15.44:0/2407103413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97600ce110 0x7f9760071190 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f9750009e50 tx=0x7f975002bed0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.602+0000 7f9764d3d640 1 -- 172.21.15.44:0/2407103413 shutdown_connections 2024-04-02T13:31:14.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.602+0000 7f9764d3d640 1 --2- 172.21.15.44:0/2407103413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97600ce110 0x7f9760071190 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.602+0000 7f9764d3d640 1 --2- 172.21.15.44:0/2407103413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97600b2d00 0x7f97600cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.602+0000 7f9764d3d640 1 -- 172.21.15.44:0/2407103413 >> 172.21.15.44:0/2407103413 conn(0x7f97600cfcd0 msgr2=0x7f97600d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:14.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.602+0000 7f9764d3d640 1 -- 172.21.15.44:0/2407103413 shutdown_connections 2024-04-02T13:31:14.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.602+0000 7f9764d3d640 1 -- 172.21.15.44:0/2407103413 wait complete. 2024-04-02T13:31:14.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.603+0000 7f9764d3d640 1 Processor -- start 2024-04-02T13:31:14.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.603+0000 7f9764d3d640 1 -- start start 2024-04-02T13:31:14.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.604+0000 7f9764d3d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97600b2d00 0x7f9760157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.604+0000 7f9764d3d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97600ce110 0x7f97601f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.604+0000 7f9764d3d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f976015aa00 0x7f97600b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.604+0000 7f9764d3d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f97600ce5a0 con 0x7f97600ce110 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.604+0000 7f9764d3d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f97600726d0 con 0x7f976015aa00 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.604+0000 7f9764d3d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f97600cc750 con 0x7f97600b2d00 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.604+0000 7f975dd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97600ce110 0x7f97601f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.604+0000 7f975dd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97600ce110 0x7f97601f5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53564/0 (socket says 172.21.15.44:53564) 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.604+0000 7f975dd9b640 1 -- 172.21.15.44:0/1112193085 learned_addr learned my addr 172.21.15.44:0/1112193085 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.604+0000 7f975e59c640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97600b2d00 0x7f9760157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.604+0000 7f975ed9d640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f976015aa00 0x7f97600b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.605+0000 7f975dd9b640 1 -- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97600b2d00 msgr2=0x7f9760157250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.605+0000 7f975dd9b640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97600b2d00 0x7f9760157250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.605+0000 7f975dd9b640 1 -- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f976015aa00 msgr2=0x7f97600b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:14.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.605+0000 7f975dd9b640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f976015aa00 0x7f97600b45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.605+0000 7f975dd9b640 1 -- 172.21.15.44:0/1112193085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97600ca320 con 0x7f97600ce110 2024-04-02T13:31:14.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.605+0000 7f975ed9d640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f976015aa00 0x7f97600b45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:14.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.605+0000 7f975dd9b640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97600ce110 0x7f97601f5170 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f974800e9e0 tx=0x7f974800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:14.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.605+0000 7f9756ffd640 1 -- 172.21.15.44:0/1112193085 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9748017070 con 0x7f97600ce110 2024-04-02T13:31:14.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.605+0000 7f9756ffd640 1 -- 172.21.15.44:0/1112193085 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f974800c9d0 con 0x7f97600ce110 2024-04-02T13:31:14.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.605+0000 7f9764d3d640 1 -- 172.21.15.44:0/1112193085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97600ca590 con 0x7f97600ce110 2024-04-02T13:31:14.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.605+0000 7f9756ffd640 1 -- 172.21.15.44:0/1112193085 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974800cdc0 con 0x7f97600ce110 2024-04-02T13:31:14.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.605+0000 7f9764d3d640 1 -- 172.21.15.44:0/1112193085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97601f4ea0 con 0x7f97600ce110 2024-04-02T13:31:14.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.606+0000 7f9764d3d640 1 -- 172.21.15.44:0/1112193085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9760074f20 con 0x7f97600ce110 2024-04-02T13:31:14.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.606+0000 7f9756ffd640 1 -- 172.21.15.44:0/1112193085 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f974801a070 con 0x7f97600ce110 2024-04-02T13:31:14.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.607+0000 7f9756ffd640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f974006c8f0 0x7f974006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:14.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.607+0000 7f9756ffd640 1 -- 172.21.15.44:0/1112193085 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f974808bc30 con 0x7f97600ce110 2024-04-02T13:31:14.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.607+0000 7f975e59c640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f974006c8f0 0x7f974006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:14.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.608+0000 7f975e59c640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f974006c8f0 0x7f974006edb0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f97600cf8c0 tx=0x7f9758009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:14.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.611+0000 7f9756ffd640 1 -- 172.21.15.44:0/1112193085 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9748059880 con 0x7f97600ce110 2024-04-02T13:31:14.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.833+0000 7f9764d3d640 1 -- 172.21.15.44:0/1112193085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97600d09a0 con 0x7f97600ce110 2024-04-02T13:31:14.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.833+0000 7f9756ffd640 1 -- 172.21.15.44:0/1112193085 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5180 (secure 0 0 0) 0x7f974805db80 con 0x7f97600ce110 2024-04-02T13:31:14.836 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:14.836 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":99,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:31:11.495701+0000","modified":"2024-04-02T13:31:13.826777+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9513},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9513":{"gid":9513,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6835/2109578276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":2109578276},{"type":"v1","addr":"172.21.15.44:6835","nonce":2109578276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9513,"qdb_cluster":[9513]},"id":8}]} 2024-04-02T13:31:14.837 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 99 2024-04-02T13:31:14.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.835+0000 7f9764d3d640 1 -- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f974006c8f0 msgr2=0x7f974006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:14.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.835+0000 7f9764d3d640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f974006c8f0 0x7f974006edb0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f97600cf8c0 tx=0x7f9758009040 comp rx=0 tx=0).stop 2024-04-02T13:31:14.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.835+0000 7f9764d3d640 1 -- 172.21.15.44:0/1112193085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97600ce110 msgr2=0x7f97601f5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:14.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.835+0000 7f9764d3d640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97600ce110 0x7f97601f5170 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f974800e9e0 tx=0x7f974800eea0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.835+0000 7f9764d3d640 1 -- 172.21.15.44:0/1112193085 shutdown_connections 2024-04-02T13:31:14.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.835+0000 7f9764d3d640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f974006c8f0 0x7f974006edb0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.835+0000 7f9764d3d640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f976015aa00 0x7f97600b45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.835+0000 7f9764d3d640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97600ce110 0x7f97601f5170 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.835+0000 7f9764d3d640 1 --2- 172.21.15.44:0/1112193085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97600b2d00 0x7f9760157250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.835+0000 7f9764d3d640 1 -- 172.21.15.44:0/1112193085 >> 172.21.15.44:0/1112193085 conn(0x7f97600cfcd0 msgr2=0x7f97600b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:14.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.835+0000 7f9764d3d640 1 -- 172.21.15.44:0/1112193085 shutdown_connections 2024-04-02T13:31:14.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.835+0000 7f9764d3d640 1 -- 172.21.15.44:0/1112193085 wait complete. 2024-04-02T13:31:14.849 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:31:14.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.947+0000 7f60a43e9640 1 Processor -- start 2024-04-02T13:31:14.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.947+0000 7f60a43e9640 1 -- start start 2024-04-02T13:31:14.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.948+0000 7f60a43e9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f609c15c050 0x7f609c0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:14.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.948+0000 7f60a43e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f609c0b1ed0 0x7f609c0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:14.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.948+0000 7f60a43e9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f609c0ccb10 con 0x7f609c15c430 2024-04-02T13:31:14.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.948+0000 7f60a43e9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f609c0b2480 con 0x7f609c15c050 2024-04-02T13:31:14.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.948+0000 7f60a43e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f609c0b2760 con 0x7f609c0b1ed0 2024-04-02T13:31:14.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.948+0000 7f60a2986640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f609c0b1ed0 0x7f609c0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:14.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.948+0000 7f60a2986640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f609c0b1ed0 0x7f609c0ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:39590/0 (socket says 172.21.15.44:39590) 2024-04-02T13:31:14.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.948+0000 7f60a2185640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f609c15c050 0x7f609c0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:14.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.948+0000 7f60a2986640 1 -- 172.21.15.44:0/1787378830 learned_addr learned my addr 172.21.15.44:0/1787378830 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:14.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.949+0000 7f60a2986640 1 -- 172.21.15.44:0/1787378830 >> v2:172.21.15.69:3300/0 conn(0x7f609c15c050 msgr2=0x7f609c0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:14.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.949+0000 7f60a2986640 1 --2- 172.21.15.44:0/1787378830 >> v2:172.21.15.69:3300/0 conn(0x7f609c15c050 0x7f609c0ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.949+0000 7f60a2986640 1 -- 172.21.15.44:0/1787378830 >> v1:172.21.15.69:6789/0 conn(0x7f609c15c430 legacy=0x7f609c0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.949+0000 7f60a1183640 1 -- 172.21.15.44:0/1787378830 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1454912684 0 0) 0x7f609c0ccb10 con 0x7f609c15c430 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.949+0000 7f60a2986640 1 -- 172.21.15.44:0/1787378830 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609c0b2a40 con 0x7f609c0b1ed0 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.950+0000 7f60a2986640 1 --2- 172.21.15.44:0/1787378830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f609c0b1ed0 0x7f609c0ced90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f608c009e50 tx=0x7f608c02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=231fa510589d8df3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.950+0000 7f60a1183640 1 -- 172.21.15.44:0/1787378830 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c02cb90 con 0x7f609c0b1ed0 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.950+0000 7f60a1183640 1 -- 172.21.15.44:0/1787378830 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f608c02cd30 con 0x7f609c0b1ed0 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.950+0000 7f60a1183640 1 -- 172.21.15.44:0/1787378830 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c034610 con 0x7f609c0b1ed0 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.950+0000 7f60a43e9640 1 -- 172.21.15.44:0/1787378830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f609c0b1ed0 msgr2=0x7f609c0ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.950+0000 7f60a43e9640 1 --2- 172.21.15.44:0/1787378830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f609c0b1ed0 0x7f609c0ced90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f608c009e50 tx=0x7f608c02bff0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.950+0000 7f60a43e9640 1 -- 172.21.15.44:0/1787378830 shutdown_connections 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.950+0000 7f60a43e9640 1 --2- 172.21.15.44:0/1787378830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f609c0b1ed0 0x7f609c0ced90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.950+0000 7f60a43e9640 1 --2- 172.21.15.44:0/1787378830 >> v2:172.21.15.69:3300/0 conn(0x7f609c15c050 0x7f609c0ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.950+0000 7f60a43e9640 1 -- 172.21.15.44:0/1787378830 >> 172.21.15.44:0/1787378830 conn(0x7f609c09aa20 msgr2=0x7f609c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.950+0000 7f60a43e9640 1 -- 172.21.15.44:0/1787378830 shutdown_connections 2024-04-02T13:31:14.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.950+0000 7f60a43e9640 1 -- 172.21.15.44:0/1787378830 wait complete. 2024-04-02T13:31:14.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.951+0000 7f60a43e9640 1 Processor -- start 2024-04-02T13:31:14.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.952+0000 7f60a43e9640 1 -- start start 2024-04-02T13:31:14.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.952+0000 7f60a43e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f609c0b1ed0 0x7f609c1f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:14.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.952+0000 7f60a43e9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f609c15c050 0x7f609c15cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:14.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.952+0000 7f60a43e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f609c15c430 0x7f609c1f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:14.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.952+0000 7f60a43e9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f609c0cd140 con 0x7f609c15c050 2024-04-02T13:31:14.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.952+0000 7f60a43e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f609c0ccfc0 con 0x7f609c15c430 2024-04-02T13:31:14.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.952+0000 7f60a43e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f609c0cd2c0 con 0x7f609c0b1ed0 2024-04-02T13:31:14.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.952+0000 7f60a1984640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f609c15c050 0x7f609c15cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:14.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.952+0000 7f60a1984640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f609c15c050 0x7f609c15cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53572/0 (socket says 172.21.15.44:53572) 2024-04-02T13:31:14.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.952+0000 7f60a1984640 1 -- 172.21.15.44:0/3038846081 learned_addr learned my addr 172.21.15.44:0/3038846081 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f60a2986640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f609c15c430 0x7f609c1f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f60a2185640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f609c0b1ed0 0x7f609c1f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f60a1984640 1 -- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f609c0b1ed0 msgr2=0x7f609c1f2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f60a1984640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f609c0b1ed0 0x7f609c1f2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f60a1984640 1 -- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f609c15c430 msgr2=0x7f609c1f4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f60a1984640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f609c15c430 0x7f609c1f4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f60a1984640 1 -- 172.21.15.44:0/3038846081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609c157250 con 0x7f609c15c050 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f60a2986640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f609c15c430 0x7f609c1f4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f60a1984640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f609c15c050 0x7f609c15cab0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f609000c3e0 tx=0x7f609000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f609affd640 1 -- 172.21.15.44:0/3038846081 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6090017070 con 0x7f609c15c050 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f609affd640 1 -- 172.21.15.44:0/3038846081 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60900052d0 con 0x7f609c15c050 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f60a43e9640 1 -- 172.21.15.44:0/3038846081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f609c1f2af0 con 0x7f609c15c050 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f609affd640 1 -- 172.21.15.44:0/3038846081 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6090003830 con 0x7f609c15c050 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.953+0000 7f60a43e9640 1 -- 172.21.15.44:0/3038846081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f609c1577f0 con 0x7f609c15c050 2024-04-02T13:31:14.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.954+0000 7f60a43e9640 1 -- 172.21.15.44:0/3038846081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f609c0684f0 con 0x7f609c15c050 2024-04-02T13:31:14.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.954+0000 7f609affd640 1 -- 172.21.15.44:0/3038846081 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f609001a070 con 0x7f609c15c050 2024-04-02T13:31:14.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.955+0000 7f609affd640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f608806c8f0 0x7f608806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:14.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.955+0000 7f609affd640 1 -- 172.21.15.44:0/3038846081 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f609008b0d0 con 0x7f609c15c050 2024-04-02T13:31:14.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.955+0000 7f60a2185640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f608806c8f0 0x7f608806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:14.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.956+0000 7f60a2185640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f608806c8f0 0x7f608806edb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6094009ad0 tx=0x7f6094008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:14.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:14.959+0000 7f609affd640 1 -- 172.21.15.44:0/3038846081 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6090059380 con 0x7f609c15c050 2024-04-02T13:31:15.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.145+0000 7f60a43e9640 1 -- 172.21.15.44:0/3038846081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f609c0b0b30 con 0x7f609c15c050 2024-04-02T13:31:15.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.146+0000 7f609affd640 1 -- 172.21.15.44:0/3038846081 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) ==== 74+0+16904 (secure 0 0 0) 0x7f6090058c80 con 0x7f609c15c050 2024-04-02T13:31:15.149 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:15.149 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":121,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:13.125622+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:31:09.752845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-02T13:31:10.437454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:31:15.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.148+0000 7f60a43e9640 1 -- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f608806c8f0 msgr2=0x7f608806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.148+0000 7f60a43e9640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f608806c8f0 0x7f608806edb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6094009ad0 tx=0x7f6094008040 comp rx=0 tx=0).stop 2024-04-02T13:31:15.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.148+0000 7f60a43e9640 1 -- 172.21.15.44:0/3038846081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f609c15c050 msgr2=0x7f609c15cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.148+0000 7f60a43e9640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f609c15c050 0x7f609c15cab0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f609000c3e0 tx=0x7f609000c8a0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.148+0000 7f60a43e9640 1 -- 172.21.15.44:0/3038846081 shutdown_connections 2024-04-02T13:31:15.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.148+0000 7f60a43e9640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f608806c8f0 0x7f608806edb0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.148+0000 7f60a43e9640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f609c15c430 0x7f609c1f4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.148+0000 7f60a43e9640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f609c15c050 0x7f609c15cab0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.148+0000 7f60a43e9640 1 --2- 172.21.15.44:0/3038846081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f609c0b1ed0 0x7f609c1f2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.148+0000 7f60a43e9640 1 -- 172.21.15.44:0/3038846081 >> 172.21.15.44:0/3038846081 conn(0x7f609c09aa20 msgr2=0x7f609c0b31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:15.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.148+0000 7f60a43e9640 1 -- 172.21.15.44:0/3038846081 shutdown_connections 2024-04-02T13:31:15.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.148+0000 7f60a43e9640 1 -- 172.21.15.44:0/3038846081 wait complete. 2024-04-02T13:31:15.162 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T13:31:15.162 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:31:15.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.260+0000 7f4ec4d7c640 1 Processor -- start 2024-04-02T13:31:15.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.260+0000 7f4ec4d7c640 1 -- start start 2024-04-02T13:31:15.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.261+0000 7f4ec4d7c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4ec00d0a30 0x7f4ec006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.261+0000 7f4ec4d7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec00d0e10 0x7f4ec00cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.261+0000 7f4ec4d7c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4ec00cfe30 0x7f4ec00d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.261+0000 7f4ec4d7c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f4ec006cb00 con 0x7f4ec00cfe30 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.261+0000 7f4ec4d7c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4ec006c1d0 con 0x7f4ec00d0a30 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.261+0000 7f4ec4d7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4ec006c4b0 con 0x7f4ec00d0e10 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.261+0000 7f4ebed9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4ec00cfe30 0x7f4ec00d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.261+0000 7f4ebed9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f4ec00cfe30 0x7f4ec00d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53576/0 (socket says 172.21.15.44:53576) 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.261+0000 7f4ebed9d640 1 -- 172.21.15.44:0/2880144642 learned_addr learned my addr 172.21.15.44:0/2880144642 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.262+0000 7f4ebe59c640 1 --2- 172.21.15.44:0/2880144642 >> v2:172.21.15.69:3300/0 conn(0x7f4ec00d0a30 0x7f4ec006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.262+0000 7f4ebdd9b640 1 --2- 172.21.15.44:0/2880144642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec00d0e10 0x7f4ec00cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.262+0000 7f4ebed9d640 1 -- 172.21.15.44:0/2880144642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec00d0e10 msgr2=0x7f4ec00cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.262+0000 7f4ebed9d640 1 --2- 172.21.15.44:0/2880144642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec00d0e10 0x7f4ec00cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.262+0000 7f4ebed9d640 1 -- 172.21.15.44:0/2880144642 >> v2:172.21.15.69:3300/0 conn(0x7f4ec00d0a30 msgr2=0x7f4ec006cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.262+0000 7f4ebed9d640 1 --2- 172.21.15.44:0/2880144642 >> v2:172.21.15.69:3300/0 conn(0x7f4ec00d0a30 0x7f4ec006cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.262+0000 7f4ebed9d640 1 -- 172.21.15.44:0/2880144642 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec006c790 con 0x7f4ec00cfe30 2024-04-02T13:31:15.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.262+0000 7f4ebdd9b640 1 --2- 172.21.15.44:0/2880144642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec00d0e10 0x7f4ec00cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:15.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.262+0000 7f4ebed9d640 1 --2- 172.21.15.44:0/2880144642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec00cfe30 0x7f4ec00d2040 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f4ec006de20 tx=0x7f4eb002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a663a34d1570396 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:15.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.263+0000 7f4ebd59a640 1 -- 172.21.15.44:0/2880144642 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb002c9e0 con 0x7f4ec00cfe30 2024-04-02T13:31:15.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.263+0000 7f4ebd59a640 1 -- 172.21.15.44:0/2880144642 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eb002cb80 con 0x7f4ec00cfe30 2024-04-02T13:31:15.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.263+0000 7f4ebd59a640 1 -- 172.21.15.44:0/2880144642 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb00343f0 con 0x7f4ec00cfe30 2024-04-02T13:31:15.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.263+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/2880144642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec00cfe30 msgr2=0x7f4ec00d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.263+0000 7f4ec4d7c640 1 --2- 172.21.15.44:0/2880144642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec00cfe30 0x7f4ec00d2040 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f4ec006de20 tx=0x7f4eb002be40 comp rx=0 tx=0).stop 2024-04-02T13:31:15.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.263+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/2880144642 shutdown_connections 2024-04-02T13:31:15.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.263+0000 7f4ec4d7c640 1 --2- 172.21.15.44:0/2880144642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec00cfe30 0x7f4ec00d2040 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.263+0000 7f4ec4d7c640 1 --2- 172.21.15.44:0/2880144642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec00d0e10 0x7f4ec00cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.263+0000 7f4ec4d7c640 1 --2- 172.21.15.44:0/2880144642 >> v2:172.21.15.69:3300/0 conn(0x7f4ec00d0a30 0x7f4ec006cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.263+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/2880144642 >> 172.21.15.44:0/2880144642 conn(0x7f4ec009d470 msgr2=0x7f4ec00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:15.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.263+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/2880144642 shutdown_connections 2024-04-02T13:31:15.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.263+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/2880144642 wait complete. 2024-04-02T13:31:15.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.265+0000 7f4ec4d7c640 1 Processor -- start 2024-04-02T13:31:15.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.265+0000 7f4ec4d7c640 1 -- start start 2024-04-02T13:31:15.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.265+0000 7f4ec4d7c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec00cfe30 0x7f4ec01f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.265+0000 7f4ec4d7c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec00d0a30 0x7f4ec0161160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.265+0000 7f4ec4d7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec00d0e10 0x7f4ec014f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.265+0000 7f4ec4d7c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec015e870 con 0x7f4ec00cfe30 2024-04-02T13:31:15.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.265+0000 7f4ec4d7c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec015e6f0 con 0x7f4ec00d0a30 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.265+0000 7f4ec4d7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4ec00d4da0 con 0x7f4ec00d0e10 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.265+0000 7f4ebe59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec00cfe30 0x7f4ec01f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.265+0000 7f4ebe59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec00cfe30 0x7f4ec01f4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53588/0 (socket says 172.21.15.44:53588) 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.265+0000 7f4ebe59c640 1 -- 172.21.15.44:0/1879155749 learned_addr learned my addr 172.21.15.44:0/1879155749 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4ebed9d640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec00d0e10 0x7f4ec014f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4ebdd9b640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec00d0a30 0x7f4ec0161160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4ebe59c640 1 -- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec00d0e10 msgr2=0x7f4ec014f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4ebe59c640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec00d0e10 0x7f4ec014f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4ebe59c640 1 -- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec00d0a30 msgr2=0x7f4ec0161160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4ebe59c640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec00d0a30 0x7f4ec0161160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4ebe59c640 1 -- 172.21.15.44:0/1879155749 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec01f50b0 con 0x7f4ec00cfe30 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4ebed9d640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec00d0e10 0x7f4ec014f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4ebdd9b640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec00d0a30 0x7f4ec0161160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:15.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4ebe59c640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec00cfe30 0x7f4ec01f4b70 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800d800 tx=0x7f4eb800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:15.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4eb6ffd640 1 -- 172.21.15.44:0/1879155749 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8018030 con 0x7f4ec00cfe30 2024-04-02T13:31:15.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4eb6ffd640 1 -- 172.21.15.44:0/1879155749 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eb80096a0 con 0x7f4ec00cfe30 2024-04-02T13:31:15.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/1879155749 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec01600c0 con 0x7f4ec00cfe30 2024-04-02T13:31:15.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4eb6ffd640 1 -- 172.21.15.44:0/1879155749 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8009870 con 0x7f4ec00cfe30 2024-04-02T13:31:15.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.266+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/1879155749 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec0160960 con 0x7f4ec00cfe30 2024-04-02T13:31:15.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.267+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/1879155749 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ec00684f0 con 0x7f4ec00cfe30 2024-04-02T13:31:15.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.267+0000 7f4eb6ffd640 1 -- 172.21.15.44:0/1879155749 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4eb801b070 con 0x7f4ec00cfe30 2024-04-02T13:31:15.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.268+0000 7f4eb6ffd640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ea006c920 0x7f4ea006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.268+0000 7f4eb6ffd640 1 -- 172.21.15.44:0/1879155749 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f4eb808bee0 con 0x7f4ec00cfe30 2024-04-02T13:31:15.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.268+0000 7f4ebdd9b640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ea006c920 0x7f4ea006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.268+0000 7f4ebdd9b640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ea006c920 0x7f4ea006ede0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00c7ce0 tx=0x7f4ea8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:15.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.272+0000 7f4eb6ffd640 1 -- 172.21.15.44:0/1879155749 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4ec01f50b0 con 0x7f4ec00cfe30 2024-04-02T13:31:15.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.453+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/1879155749 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4ec00d17c0 con 0x7f4ea006c920 2024-04-02T13:31:15.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.460+0000 7f4eb6ffd640 1 -- 172.21.15.44:0/1879155749 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252498 (secure 0 0 0) 0x7f4ec00d17c0 con 0x7f4ea006c920 2024-04-02T13:31:15.466 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:15.467 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:31:15.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.463+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ea006c920 msgr2=0x7f4ea006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.463+0000 7f4ec4d7c640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ea006c920 0x7f4ea006ede0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00c7ce0 tx=0x7f4ea8008040 comp rx=0 tx=0).stop 2024-04-02T13:31:15.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.463+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/1879155749 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec00cfe30 msgr2=0x7f4ec01f4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.463+0000 7f4ec4d7c640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec00cfe30 0x7f4ec01f4b70 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800d800 tx=0x7f4eb800dcc0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.463+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/1879155749 shutdown_connections 2024-04-02T13:31:15.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.463+0000 7f4ec4d7c640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f4ea006c920 0x7f4ea006ede0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.463+0000 7f4ec4d7c640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec00d0e10 0x7f4ec014f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.463+0000 7f4ec4d7c640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec00d0a30 0x7f4ec0161160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.463+0000 7f4ec4d7c640 1 --2- 172.21.15.44:0/1879155749 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec00cfe30 0x7f4ec01f4b70 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.463+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/1879155749 >> 172.21.15.44:0/1879155749 conn(0x7f4ec009d470 msgr2=0x7f4ec00d3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:15.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.463+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/1879155749 shutdown_connections 2024-04-02T13:31:15.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.463+0000 7f4ec4d7c640 1 -- 172.21.15.44:0/1879155749 wait complete. 2024-04-02T13:31:15.476 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2024-04-02T13:31:15.038952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262448,"kb_used_data":12964,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712272,"statfs":{"total":773094113280,"available":772825366528,"internally_reserved":0,"allocated":13275136,"data_stored":6801584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.922770"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441174+0000","last_change":"2024-04-02T13:31:12.142684+0000","last_active":"2024-04-02T13:31:13.441174+0000","last_peered":"2024-04-02T13:31:13.441174+0000","last_clean":"2024-04-02T13:31:13.441174+0000","last_became_active":"2024-04-02T13:31:12.142401+0000","last_became_peered":"2024-04-02T13:31:12.142401+0000","last_unstale":"2024-04-02T13:31:13.441174+0000","last_undegraded":"2024-04-02T13:31:13.441174+0000","last_fullsized":"2024-04-02T13:31:13.441174+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:45:45.336259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308830+0000","last_change":"2024-04-02T13:31:12.154695+0000","last_active":"2024-04-02T13:31:13.308830+0000","last_peered":"2024-04-02T13:31:13.308830+0000","last_clean":"2024-04-02T13:31:13.308830+0000","last_became_active":"2024-04-02T13:31:12.154371+0000","last_became_peered":"2024-04-02T13:31:12.154371+0000","last_unstale":"2024-04-02T13:31:13.308830+0000","last_undegraded":"2024-04-02T13:31:13.308830+0000","last_fullsized":"2024-04-02T13:31:13.308830+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:21:14.776344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.309022+0000","last_change":"2024-04-02T13:31:12.142513+0000","last_active":"2024-04-02T13:31:13.309022+0000","last_peered":"2024-04-02T13:31:13.309022+0000","last_clean":"2024-04-02T13:31:13.309022+0000","last_became_active":"2024-04-02T13:31:12.142238+0000","last_became_peered":"2024-04-02T13:31:12.142238+0000","last_unstale":"2024-04-02T13:31:13.309022+0000","last_undegraded":"2024-04-02T13:31:13.309022+0000","last_fullsized":"2024-04-02T13:31:13.309022+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:35:31.717737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140458+0000","last_change":"2024-04-02T13:31:12.133658+0000","last_active":"2024-04-02T13:31:13.140458+0000","last_peered":"2024-04-02T13:31:13.140458+0000","last_clean":"2024-04-02T13:31:13.140458+0000","last_became_active":"2024-04-02T13:31:12.133301+0000","last_became_peered":"2024-04-02T13:31:12.133301+0000","last_unstale":"2024-04-02T13:31:13.140458+0000","last_undegraded":"2024-04-02T13:31:13.140458+0000","last_fullsized":"2024-04-02T13:31:13.140458+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:15:32.709627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440763+0000","last_change":"2024-04-02T13:31:12.130711+0000","last_active":"2024-04-02T13:31:13.440763+0000","last_peered":"2024-04-02T13:31:13.440763+0000","last_clean":"2024-04-02T13:31:13.440763+0000","last_became_active":"2024-04-02T13:31:12.130422+0000","last_became_peered":"2024-04-02T13:31:12.130422+0000","last_unstale":"2024-04-02T13:31:13.440763+0000","last_undegraded":"2024-04-02T13:31:13.440763+0000","last_fullsized":"2024-04-02T13:31:13.440763+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:45.106103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138983+0000","last_change":"2024-04-02T13:31:12.147698+0000","last_active":"2024-04-02T13:31:13.138983+0000","last_peered":"2024-04-02T13:31:13.138983+0000","last_clean":"2024-04-02T13:31:13.138983+0000","last_became_active":"2024-04-02T13:31:12.147413+0000","last_became_peered":"2024-04-02T13:31:12.147413+0000","last_unstale":"2024-04-02T13:31:13.138983+0000","last_undegraded":"2024-04-02T13:31:13.138983+0000","last_fullsized":"2024-04-02T13:31:13.138983+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:12:24.001590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138929+0000","last_change":"2024-04-02T13:31:12.140695+0000","last_active":"2024-04-02T13:31:13.138929+0000","last_peered":"2024-04-02T13:31:13.138929+0000","last_clean":"2024-04-02T13:31:13.138929+0000","last_became_active":"2024-04-02T13:31:12.140423+0000","last_became_peered":"2024-04-02T13:31:12.140423+0000","last_unstale":"2024-04-02T13:31:13.138929+0000","last_undegraded":"2024-04-02T13:31:13.138929+0000","last_fullsized":"2024-04-02T13:31:13.138929+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:15:04.901097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308527+0000","last_change":"2024-04-02T13:31:12.145293+0000","last_active":"2024-04-02T13:31:13.308527+0000","last_peered":"2024-04-02T13:31:13.308527+0000","last_clean":"2024-04-02T13:31:13.308527+0000","last_became_active":"2024-04-02T13:31:12.145022+0000","last_became_peered":"2024-04-02T13:31:12.145022+0000","last_unstale":"2024-04-02T13:31:13.308527+0000","last_undegraded":"2024-04-02T13:31:13.308527+0000","last_fullsized":"2024-04-02T13:31:13.308527+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:32.364376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140759+0000","last_change":"2024-04-02T13:31:12.147556+0000","last_active":"2024-04-02T13:31:13.140759+0000","last_peered":"2024-04-02T13:31:13.140759+0000","last_clean":"2024-04-02T13:31:13.140759+0000","last_became_active":"2024-04-02T13:31:12.147230+0000","last_became_peered":"2024-04-02T13:31:12.147230+0000","last_unstale":"2024-04-02T13:31:13.140759+0000","last_undegraded":"2024-04-02T13:31:13.140759+0000","last_fullsized":"2024-04-02T13:31:13.140759+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:43:44.956087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345155+0000","last_change":"2024-04-02T13:31:12.143114+0000","last_active":"2024-04-02T13:31:13.345155+0000","last_peered":"2024-04-02T13:31:13.345155+0000","last_clean":"2024-04-02T13:31:13.345155+0000","last_became_active":"2024-04-02T13:31:12.142838+0000","last_became_peered":"2024-04-02T13:31:12.142838+0000","last_unstale":"2024-04-02T13:31:13.345155+0000","last_undegraded":"2024-04-02T13:31:13.345155+0000","last_fullsized":"2024-04-02T13:31:13.345155+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:45:45.336259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440889+0000","last_change":"2024-04-02T13:31:12.127457+0000","last_active":"2024-04-02T13:31:13.440889+0000","last_peered":"2024-04-02T13:31:13.440889+0000","last_clean":"2024-04-02T13:31:13.440889+0000","last_became_active":"2024-04-02T13:31:12.127164+0000","last_became_peered":"2024-04-02T13:31:12.127164+0000","last_unstale":"2024-04-02T13:31:13.440889+0000","last_undegraded":"2024-04-02T13:31:13.440889+0000","last_fullsized":"2024-04-02T13:31:13.440889+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:11:29.837470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139260+0000","last_change":"2024-04-02T13:31:12.140930+0000","last_active":"2024-04-02T13:31:13.139260+0000","last_peered":"2024-04-02T13:31:13.139260+0000","last_clean":"2024-04-02T13:31:13.139260+0000","last_became_active":"2024-04-02T13:31:12.140539+0000","last_became_peered":"2024-04-02T13:31:12.140539+0000","last_unstale":"2024-04-02T13:31:13.139260+0000","last_undegraded":"2024-04-02T13:31:13.139260+0000","last_fullsized":"2024-04-02T13:31:13.139260+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:20.933022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345142+0000","last_change":"2024-04-02T13:31:12.142675+0000","last_active":"2024-04-02T13:31:13.345142+0000","last_peered":"2024-04-02T13:31:13.345142+0000","last_clean":"2024-04-02T13:31:13.345142+0000","last_became_active":"2024-04-02T13:31:12.142221+0000","last_became_peered":"2024-04-02T13:31:12.142221+0000","last_unstale":"2024-04-02T13:31:13.345142+0000","last_undegraded":"2024-04-02T13:31:13.345142+0000","last_fullsized":"2024-04-02T13:31:13.345142+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:24:37.992698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308707+0000","last_change":"2024-04-02T13:31:12.141165+0000","last_active":"2024-04-02T13:31:13.308707+0000","last_peered":"2024-04-02T13:31:13.308707+0000","last_clean":"2024-04-02T13:31:13.308707+0000","last_became_active":"2024-04-02T13:31:12.139621+0000","last_became_peered":"2024-04-02T13:31:12.139621+0000","last_unstale":"2024-04-02T13:31:13.308707+0000","last_undegraded":"2024-04-02T13:31:13.308707+0000","last_fullsized":"2024-04-02T13:31:13.308707+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:22:13.109205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440595+0000","last_change":"2024-04-02T13:31:12.127366+0000","last_active":"2024-04-02T13:31:13.440595+0000","last_peered":"2024-04-02T13:31:13.440595+0000","last_clean":"2024-04-02T13:31:13.440595+0000","last_became_active":"2024-04-02T13:31:12.127065+0000","last_became_peered":"2024-04-02T13:31:12.127065+0000","last_unstale":"2024-04-02T13:31:13.440595+0000","last_undegraded":"2024-04-02T13:31:13.440595+0000","last_fullsized":"2024-04-02T13:31:13.440595+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:49:46.345894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138850+0000","last_change":"2024-04-02T13:31:12.136738+0000","last_active":"2024-04-02T13:31:13.138850+0000","last_peered":"2024-04-02T13:31:13.138850+0000","last_clean":"2024-04-02T13:31:13.138850+0000","last_became_active":"2024-04-02T13:31:12.136468+0000","last_became_peered":"2024-04-02T13:31:12.136468+0000","last_unstale":"2024-04-02T13:31:13.138850+0000","last_undegraded":"2024-04-02T13:31:13.138850+0000","last_fullsized":"2024-04-02T13:31:13.138850+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:11:05.916368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345161+0000","last_change":"2024-04-02T13:31:12.139477+0000","last_active":"2024-04-02T13:31:13.345161+0000","last_peered":"2024-04-02T13:31:13.345161+0000","last_clean":"2024-04-02T13:31:13.345161+0000","last_became_active":"2024-04-02T13:31:12.139148+0000","last_became_peered":"2024-04-02T13:31:12.139148+0000","last_unstale":"2024-04-02T13:31:13.345161+0000","last_undegraded":"2024-04-02T13:31:13.345161+0000","last_fullsized":"2024-04-02T13:31:13.345161+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:14:16.657501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308921+0000","last_change":"2024-04-02T13:31:12.153879+0000","last_active":"2024-04-02T13:31:13.308921+0000","last_peered":"2024-04-02T13:31:13.308921+0000","last_clean":"2024-04-02T13:31:13.308921+0000","last_became_active":"2024-04-02T13:31:12.153585+0000","last_became_peered":"2024-04-02T13:31:12.153585+0000","last_unstale":"2024-04-02T13:31:13.308921+0000","last_undegraded":"2024-04-02T13:31:13.308921+0000","last_fullsized":"2024-04-02T13:31:13.308921+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:42:04.836436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441446+0000","last_change":"2024-04-02T13:31:12.138789+0000","last_active":"2024-04-02T13:31:13.441446+0000","last_peered":"2024-04-02T13:31:13.441446+0000","last_clean":"2024-04-02T13:31:13.441446+0000","last_became_active":"2024-04-02T13:31:12.138442+0000","last_became_peered":"2024-04-02T13:31:12.138442+0000","last_unstale":"2024-04-02T13:31:13.441446+0000","last_undegraded":"2024-04-02T13:31:13.441446+0000","last_fullsized":"2024-04-02T13:31:13.441446+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:24:37.992698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440168+0000","last_change":"2024-04-02T13:31:12.143673+0000","last_active":"2024-04-02T13:31:13.440168+0000","last_peered":"2024-04-02T13:31:13.440168+0000","last_clean":"2024-04-02T13:31:13.440168+0000","last_became_active":"2024-04-02T13:31:12.143379+0000","last_became_peered":"2024-04-02T13:31:12.143379+0000","last_unstale":"2024-04-02T13:31:13.440168+0000","last_undegraded":"2024-04-02T13:31:13.440168+0000","last_fullsized":"2024-04-02T13:31:13.440168+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:03:44.414064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140272+0000","last_change":"2024-04-02T13:31:12.134035+0000","last_active":"2024-04-02T13:31:13.140272+0000","last_peered":"2024-04-02T13:31:13.140272+0000","last_clean":"2024-04-02T13:31:13.140272+0000","last_became_active":"2024-04-02T13:31:12.133754+0000","last_became_peered":"2024-04-02T13:31:12.133754+0000","last_unstale":"2024-04-02T13:31:13.140272+0000","last_undegraded":"2024-04-02T13:31:13.140272+0000","last_fullsized":"2024-04-02T13:31:13.140272+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:18:25.723461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440735+0000","last_change":"2024-04-02T13:31:12.144040+0000","last_active":"2024-04-02T13:31:13.440735+0000","last_peered":"2024-04-02T13:31:13.440735+0000","last_clean":"2024-04-02T13:31:13.440735+0000","last_became_active":"2024-04-02T13:31:12.143750+0000","last_became_peered":"2024-04-02T13:31:12.143750+0000","last_unstale":"2024-04-02T13:31:13.440735+0000","last_undegraded":"2024-04-02T13:31:13.440735+0000","last_fullsized":"2024-04-02T13:31:13.440735+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:42:09.725977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140177+0000","last_change":"2024-04-02T13:31:12.147651+0000","last_active":"2024-04-02T13:31:13.140177+0000","last_peered":"2024-04-02T13:31:13.140177+0000","last_clean":"2024-04-02T13:31:13.140177+0000","last_became_active":"2024-04-02T13:31:12.147194+0000","last_became_peered":"2024-04-02T13:31:12.147194+0000","last_unstale":"2024-04-02T13:31:13.140177+0000","last_undegraded":"2024-04-02T13:31:13.140177+0000","last_fullsized":"2024-04-02T13:31:13.140177+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:02.941259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138668+0000","last_change":"2024-04-02T13:31:12.131065+0000","last_active":"2024-04-02T13:31:13.138668+0000","last_peered":"2024-04-02T13:31:13.138668+0000","last_clean":"2024-04-02T13:31:13.138668+0000","last_became_active":"2024-04-02T13:31:12.130782+0000","last_became_peered":"2024-04-02T13:31:12.130782+0000","last_unstale":"2024-04-02T13:31:13.138668+0000","last_undegraded":"2024-04-02T13:31:13.138668+0000","last_fullsized":"2024-04-02T13:31:13.138668+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:11:29.837470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138931+0000","last_change":"2024-04-02T13:31:12.131975+0000","last_active":"2024-04-02T13:31:13.138931+0000","last_peered":"2024-04-02T13:31:13.138931+0000","last_clean":"2024-04-02T13:31:13.138931+0000","last_became_active":"2024-04-02T13:31:12.131677+0000","last_became_peered":"2024-04-02T13:31:12.131677+0000","last_unstale":"2024-04-02T13:31:13.138931+0000","last_undegraded":"2024-04-02T13:31:13.138931+0000","last_fullsized":"2024-04-02T13:31:13.138931+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:12.227157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440243+0000","last_change":"2024-04-02T13:31:12.133781+0000","last_active":"2024-04-02T13:31:13.440243+0000","last_peered":"2024-04-02T13:31:13.440243+0000","last_clean":"2024-04-02T13:31:13.440243+0000","last_became_active":"2024-04-02T13:31:12.133409+0000","last_became_peered":"2024-04-02T13:31:12.133409+0000","last_unstale":"2024-04-02T13:31:13.440243+0000","last_undegraded":"2024-04-02T13:31:13.440243+0000","last_fullsized":"2024-04-02T13:31:13.440243+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:12.171120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441512+0000","last_change":"2024-04-02T13:31:12.137877+0000","last_active":"2024-04-02T13:31:13.441512+0000","last_peered":"2024-04-02T13:31:13.441512+0000","last_clean":"2024-04-02T13:31:13.441512+0000","last_became_active":"2024-04-02T13:31:12.137515+0000","last_became_peered":"2024-04-02T13:31:12.137515+0000","last_unstale":"2024-04-02T13:31:13.441512+0000","last_undegraded":"2024-04-02T13:31:13.441512+0000","last_fullsized":"2024-04-02T13:31:13.441512+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:45:45.336259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139345+0000","last_change":"2024-04-02T13:31:12.141128+0000","last_active":"2024-04-02T13:31:13.139345+0000","last_peered":"2024-04-02T13:31:13.139345+0000","last_clean":"2024-04-02T13:31:13.139345+0000","last_became_active":"2024-04-02T13:31:12.140867+0000","last_became_peered":"2024-04-02T13:31:12.140867+0000","last_unstale":"2024-04-02T13:31:13.139345+0000","last_undegraded":"2024-04-02T13:31:13.139345+0000","last_fullsized":"2024-04-02T13:31:13.139345+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:27:51.844903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440283+0000","last_change":"2024-04-02T13:31:12.138279+0000","last_active":"2024-04-02T13:31:13.440283+0000","last_peered":"2024-04-02T13:31:13.440283+0000","last_clean":"2024-04-02T13:31:13.440283+0000","last_became_active":"2024-04-02T13:31:12.137977+0000","last_became_peered":"2024-04-02T13:31:12.137977+0000","last_unstale":"2024-04-02T13:31:13.440283+0000","last_undegraded":"2024-04-02T13:31:13.440283+0000","last_fullsized":"2024-04-02T13:31:13.440283+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:40:09.871586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308726+0000","last_change":"2024-04-02T13:31:12.138813+0000","last_active":"2024-04-02T13:31:13.308726+0000","last_peered":"2024-04-02T13:31:13.308726+0000","last_clean":"2024-04-02T13:31:13.308726+0000","last_became_active":"2024-04-02T13:31:12.138446+0000","last_became_peered":"2024-04-02T13:31:12.138446+0000","last_unstale":"2024-04-02T13:31:13.308726+0000","last_undegraded":"2024-04-02T13:31:13.308726+0000","last_fullsized":"2024-04-02T13:31:13.308726+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:31:06.221856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138634+0000","last_change":"2024-04-02T13:31:12.134311+0000","last_active":"2024-04-02T13:31:13.138634+0000","last_peered":"2024-04-02T13:31:13.138634+0000","last_clean":"2024-04-02T13:31:13.138634+0000","last_became_active":"2024-04-02T13:31:12.134048+0000","last_became_peered":"2024-04-02T13:31:12.134048+0000","last_unstale":"2024-04-02T13:31:13.138634+0000","last_undegraded":"2024-04-02T13:31:13.138634+0000","last_fullsized":"2024-04-02T13:31:13.138634+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:06:45.310008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138216+0000","last_change":"2024-04-02T13:31:12.130296+0000","last_active":"2024-04-02T13:31:13.138216+0000","last_peered":"2024-04-02T13:31:13.138216+0000","last_clean":"2024-04-02T13:31:13.138216+0000","last_became_active":"2024-04-02T13:31:12.130024+0000","last_became_peered":"2024-04-02T13:31:12.130024+0000","last_unstale":"2024-04-02T13:31:13.138216+0000","last_undegraded":"2024-04-02T13:31:13.138216+0000","last_fullsized":"2024-04-02T13:31:13.138216+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:24:37.992698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345053+0000","last_change":"2024-04-02T13:31:12.145199+0000","last_active":"2024-04-02T13:31:13.345053+0000","last_peered":"2024-04-02T13:31:13.345053+0000","last_clean":"2024-04-02T13:31:13.345053+0000","last_became_active":"2024-04-02T13:31:12.144863+0000","last_became_peered":"2024-04-02T13:31:12.144863+0000","last_unstale":"2024-04-02T13:31:13.345053+0000","last_undegraded":"2024-04-02T13:31:13.345053+0000","last_fullsized":"2024-04-02T13:31:13.345053+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:45.106103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"120'4","reported_seq":18,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441471+0000","last_change":"2024-04-02T13:31:11.135714+0000","last_active":"2024-04-02T13:31:13.441471+0000","last_peered":"2024-04-02T13:31:13.441471+0000","last_clean":"2024-04-02T13:31:13.441471+0000","last_became_active":"2024-04-02T13:31:11.135452+0000","last_became_peered":"2024-04-02T13:31:11.135452+0000","last_unstale":"2024-04-02T13:31:13.441471+0000","last_undegraded":"2024-04-02T13:31:13.441471+0000","last_fullsized":"2024-04-02T13:31:13.441471+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:57.514520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139215+0000","last_change":"2024-04-02T13:31:12.148237+0000","last_active":"2024-04-02T13:31:13.139215+0000","last_peered":"2024-04-02T13:31:13.139215+0000","last_clean":"2024-04-02T13:31:13.139215+0000","last_became_active":"2024-04-02T13:31:12.147970+0000","last_became_peered":"2024-04-02T13:31:12.147970+0000","last_unstale":"2024-04-02T13:31:13.139215+0000","last_undegraded":"2024-04-02T13:31:13.139215+0000","last_fullsized":"2024-04-02T13:31:13.139215+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:22:36.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345100+0000","last_change":"2024-04-02T13:31:11.129147+0000","last_active":"2024-04-02T13:31:13.345100+0000","last_peered":"2024-04-02T13:31:13.345100+0000","last_clean":"2024-04-02T13:31:13.345100+0000","last_became_active":"2024-04-02T13:31:11.128778+0000","last_became_peered":"2024-04-02T13:31:11.128778+0000","last_unstale":"2024-04-02T13:31:13.345100+0000","last_undegraded":"2024-04-02T13:31:13.345100+0000","last_fullsized":"2024-04-02T13:31:13.345100+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:13:08.961404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138567+0000","last_change":"2024-04-02T13:31:12.142166+0000","last_active":"2024-04-02T13:31:13.138567+0000","last_peered":"2024-04-02T13:31:13.138567+0000","last_clean":"2024-04-02T13:31:13.138567+0000","last_became_active":"2024-04-02T13:31:12.141872+0000","last_became_peered":"2024-04-02T13:31:12.141872+0000","last_unstale":"2024-04-02T13:31:13.138567+0000","last_undegraded":"2024-04-02T13:31:13.138567+0000","last_fullsized":"2024-04-02T13:31:13.138567+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:14:16.657501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140633+0000","last_change":"2024-04-02T13:31:11.132278+0000","last_active":"2024-04-02T13:31:13.140633+0000","last_peered":"2024-04-02T13:31:13.140633+0000","last_clean":"2024-04-02T13:31:13.140633+0000","last_became_active":"2024-04-02T13:31:11.131577+0000","last_became_peered":"2024-04-02T13:31:11.131577+0000","last_unstale":"2024-04-02T13:31:13.140633+0000","last_undegraded":"2024-04-02T13:31:13.140633+0000","last_fullsized":"2024-04-02T13:31:13.140633+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:24:36.987532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140778+0000","last_change":"2024-04-02T13:31:12.134345+0000","last_active":"2024-04-02T13:31:13.140778+0000","last_peered":"2024-04-02T13:31:13.140778+0000","last_clean":"2024-04-02T13:31:13.140778+0000","last_became_active":"2024-04-02T13:31:12.134063+0000","last_became_peered":"2024-04-02T13:31:12.134063+0000","last_unstale":"2024-04-02T13:31:13.140778+0000","last_undegraded":"2024-04-02T13:31:13.140778+0000","last_fullsized":"2024-04-02T13:31:13.140778+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:35:25.756038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345112+0000","last_change":"2024-04-02T13:31:11.131085+0000","last_active":"2024-04-02T13:31:13.345112+0000","last_peered":"2024-04-02T13:31:13.345112+0000","last_clean":"2024-04-02T13:31:13.345112+0000","last_became_active":"2024-04-02T13:31:11.130800+0000","last_became_peered":"2024-04-02T13:31:11.130800+0000","last_unstale":"2024-04-02T13:31:13.345112+0000","last_undegraded":"2024-04-02T13:31:13.345112+0000","last_fullsized":"2024-04-02T13:31:13.345112+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:46:50.026745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.344924+0000","last_change":"2024-04-02T13:31:12.140319+0000","last_active":"2024-04-02T13:31:13.344924+0000","last_peered":"2024-04-02T13:31:13.344924+0000","last_clean":"2024-04-02T13:31:13.344924+0000","last_became_active":"2024-04-02T13:31:12.140062+0000","last_became_peered":"2024-04-02T13:31:12.140062+0000","last_unstale":"2024-04-02T13:31:13.344924+0000","last_undegraded":"2024-04-02T13:31:13.344924+0000","last_fullsized":"2024-04-02T13:31:13.344924+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:49:46.345894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440973+0000","last_change":"2024-04-02T13:31:11.134176+0000","last_active":"2024-04-02T13:31:13.440973+0000","last_peered":"2024-04-02T13:31:13.440973+0000","last_clean":"2024-04-02T13:31:13.440973+0000","last_became_active":"2024-04-02T13:31:11.133891+0000","last_became_peered":"2024-04-02T13:31:11.133891+0000","last_unstale":"2024-04-02T13:31:13.440973+0000","last_undegraded":"2024-04-02T13:31:13.440973+0000","last_fullsized":"2024-04-02T13:31:13.440973+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:44:10.359503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440621+0000","last_change":"2024-04-02T13:31:12.134737+0000","last_active":"2024-04-02T13:31:13.440621+0000","last_peered":"2024-04-02T13:31:13.440621+0000","last_clean":"2024-04-02T13:31:13.440621+0000","last_became_active":"2024-04-02T13:31:12.134462+0000","last_became_peered":"2024-04-02T13:31:12.134462+0000","last_unstale":"2024-04-02T13:31:13.440621+0000","last_undegraded":"2024-04-02T13:31:13.440621+0000","last_fullsized":"2024-04-02T13:31:13.440621+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:34:25.158893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138987+0000","last_change":"2024-04-02T13:31:11.132878+0000","last_active":"2024-04-02T13:31:13.138987+0000","last_peered":"2024-04-02T13:31:13.138987+0000","last_clean":"2024-04-02T13:31:13.138987+0000","last_became_active":"2024-04-02T13:31:11.132602+0000","last_became_peered":"2024-04-02T13:31:11.132602+0000","last_unstale":"2024-04-02T13:31:13.138987+0000","last_undegraded":"2024-04-02T13:31:13.138987+0000","last_fullsized":"2024-04-02T13:31:13.138987+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:35:24.750872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308885+0000","last_change":"2024-04-02T13:31:12.150723+0000","last_active":"2024-04-02T13:31:13.308885+0000","last_peered":"2024-04-02T13:31:13.308885+0000","last_clean":"2024-04-02T13:31:13.308885+0000","last_became_active":"2024-04-02T13:31:12.150444+0000","last_became_peered":"2024-04-02T13:31:12.150444+0000","last_unstale":"2024-04-02T13:31:13.308885+0000","last_undegraded":"2024-04-02T13:31:13.308885+0000","last_fullsized":"2024-04-02T13:31:13.308885+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:25:27.156701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138256+0000","last_change":"2024-04-02T13:31:11.133786+0000","last_active":"2024-04-02T13:31:13.138256+0000","last_peered":"2024-04-02T13:31:13.138256+0000","last_clean":"2024-04-02T13:31:13.138256+0000","last_became_active":"2024-04-02T13:31:11.132953+0000","last_became_peered":"2024-04-02T13:31:11.132953+0000","last_unstale":"2024-04-02T13:31:13.138256+0000","last_undegraded":"2024-04-02T13:31:13.138256+0000","last_fullsized":"2024-04-02T13:31:13.138256+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:43:40.243740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345047+0000","last_change":"2024-04-02T13:31:12.150939+0000","last_active":"2024-04-02T13:31:13.345047+0000","last_peered":"2024-04-02T13:31:13.345047+0000","last_clean":"2024-04-02T13:31:13.345047+0000","last_became_active":"2024-04-02T13:31:12.150683+0000","last_became_peered":"2024-04-02T13:31:12.150683+0000","last_unstale":"2024-04-02T13:31:13.345047+0000","last_undegraded":"2024-04-02T13:31:13.345047+0000","last_fullsized":"2024-04-02T13:31:13.345047+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:42:09.725977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139068+0000","last_change":"2024-04-02T13:31:11.133664+0000","last_active":"2024-04-02T13:31:13.139068+0000","last_peered":"2024-04-02T13:31:13.139068+0000","last_clean":"2024-04-02T13:31:13.139068+0000","last_became_active":"2024-04-02T13:31:11.133402+0000","last_became_peered":"2024-04-02T13:31:11.133402+0000","last_unstale":"2024-04-02T13:31:13.139068+0000","last_undegraded":"2024-04-02T13:31:13.139068+0000","last_fullsized":"2024-04-02T13:31:13.139068+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:43:43.950920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441255+0000","last_change":"2024-04-02T13:31:12.133962+0000","last_active":"2024-04-02T13:31:13.441255+0000","last_peered":"2024-04-02T13:31:13.441255+0000","last_clean":"2024-04-02T13:31:13.441255+0000","last_became_active":"2024-04-02T13:31:12.133693+0000","last_became_peered":"2024-04-02T13:31:12.133693+0000","last_unstale":"2024-04-02T13:31:13.441255+0000","last_undegraded":"2024-04-02T13:31:13.441255+0000","last_fullsized":"2024-04-02T13:31:13.441255+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:11:29.837470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441552+0000","last_change":"2024-04-02T13:31:11.139525+0000","last_active":"2024-04-02T13:31:13.441552+0000","last_peered":"2024-04-02T13:31:13.441552+0000","last_clean":"2024-04-02T13:31:13.441552+0000","last_became_active":"2024-04-02T13:31:11.139238+0000","last_became_peered":"2024-04-02T13:31:11.139238+0000","last_unstale":"2024-04-02T13:31:13.441552+0000","last_undegraded":"2024-04-02T13:31:13.441552+0000","last_fullsized":"2024-04-02T13:31:13.441552+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:32:53.017171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138771+0000","last_change":"2024-04-02T13:31:12.143793+0000","last_active":"2024-04-02T13:31:13.138771+0000","last_peered":"2024-04-02T13:31:13.138771+0000","last_clean":"2024-04-02T13:31:13.138771+0000","last_became_active":"2024-04-02T13:31:12.143517+0000","last_became_peered":"2024-04-02T13:31:12.143517+0000","last_unstale":"2024-04-02T13:31:13.138771+0000","last_undegraded":"2024-04-02T13:31:13.138771+0000","last_fullsized":"2024-04-02T13:31:13.138771+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:57:24.583576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138547+0000","last_change":"2024-04-02T13:31:11.128784+0000","last_active":"2024-04-02T13:31:13.138547+0000","last_peered":"2024-04-02T13:31:13.138547+0000","last_clean":"2024-04-02T13:31:13.138547+0000","last_became_active":"2024-04-02T13:31:11.128512+0000","last_became_peered":"2024-04-02T13:31:11.128512+0000","last_unstale":"2024-04-02T13:31:13.138547+0000","last_undegraded":"2024-04-02T13:31:13.138547+0000","last_fullsized":"2024-04-02T13:31:13.138547+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:22.352670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.309004+0000","last_change":"2024-04-02T13:31:12.141447+0000","last_active":"2024-04-02T13:31:13.309004+0000","last_peered":"2024-04-02T13:31:13.309004+0000","last_clean":"2024-04-02T13:31:13.309004+0000","last_became_active":"2024-04-02T13:31:12.139677+0000","last_became_peered":"2024-04-02T13:31:12.139677+0000","last_unstale":"2024-04-02T13:31:13.309004+0000","last_undegraded":"2024-04-02T13:31:13.309004+0000","last_fullsized":"2024-04-02T13:31:13.309004+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:05:45.882774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"120'3","reported_seq":17,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138689+0000","last_change":"2024-04-02T13:31:11.130564+0000","last_active":"2024-04-02T13:31:13.138689+0000","last_peered":"2024-04-02T13:31:13.138689+0000","last_clean":"2024-04-02T13:31:13.138689+0000","last_became_active":"2024-04-02T13:31:11.130305+0000","last_became_peered":"2024-04-02T13:31:11.130305+0000","last_unstale":"2024-04-02T13:31:13.138689+0000","last_undegraded":"2024-04-02T13:31:13.138689+0000","last_fullsized":"2024-04-02T13:31:13.138689+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:44:32.667698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138872+0000","last_change":"2024-04-02T13:31:12.136750+0000","last_active":"2024-04-02T13:31:13.138872+0000","last_peered":"2024-04-02T13:31:13.138872+0000","last_clean":"2024-04-02T13:31:13.138872+0000","last_became_active":"2024-04-02T13:31:12.136283+0000","last_became_peered":"2024-04-02T13:31:12.136283+0000","last_unstale":"2024-04-02T13:31:13.138872+0000","last_undegraded":"2024-04-02T13:31:13.138872+0000","last_fullsized":"2024-04-02T13:31:13.138872+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:32:29.196579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"120'6","reported_seq":20,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308980+0000","last_change":"2024-04-02T13:31:11.132655+0000","last_active":"2024-04-02T13:31:13.308980+0000","last_peered":"2024-04-02T13:31:13.308980+0000","last_clean":"2024-04-02T13:31:13.308980+0000","last_became_active":"2024-04-02T13:31:11.132292+0000","last_became_peered":"2024-04-02T13:31:11.132292+0000","last_unstale":"2024-04-02T13:31:13.308980+0000","last_undegraded":"2024-04-02T13:31:13.308980+0000","last_fullsized":"2024-04-02T13:31:13.308980+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:06:44.304841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441167+0000","last_change":"2024-04-02T13:31:12.143424+0000","last_active":"2024-04-02T13:31:13.441167+0000","last_peered":"2024-04-02T13:31:13.441167+0000","last_clean":"2024-04-02T13:31:13.441167+0000","last_became_active":"2024-04-02T13:31:12.143081+0000","last_became_peered":"2024-04-02T13:31:12.143081+0000","last_unstale":"2024-04-02T13:31:13.441167+0000","last_undegraded":"2024-04-02T13:31:13.441167+0000","last_fullsized":"2024-04-02T13:31:13.441167+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:45.106103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308759+0000","last_change":"2024-04-02T13:31:11.135615+0000","last_active":"2024-04-02T13:31:13.308759+0000","last_peered":"2024-04-02T13:31:13.308759+0000","last_clean":"2024-04-02T13:31:13.308759+0000","last_became_active":"2024-04-02T13:31:11.135131+0000","last_became_peered":"2024-04-02T13:31:11.135131+0000","last_unstale":"2024-04-02T13:31:13.308759+0000","last_undegraded":"2024-04-02T13:31:13.308759+0000","last_fullsized":"2024-04-02T13:31:13.308759+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:14:15.652335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345040+0000","last_change":"2024-04-02T13:31:12.141980+0000","last_active":"2024-04-02T13:31:13.345040+0000","last_peered":"2024-04-02T13:31:13.345040+0000","last_clean":"2024-04-02T13:31:13.345040+0000","last_became_active":"2024-04-02T13:31:12.141621+0000","last_became_peered":"2024-04-02T13:31:12.141621+0000","last_unstale":"2024-04-02T13:31:13.345040+0000","last_undegraded":"2024-04-02T13:31:13.345040+0000","last_fullsized":"2024-04-02T13:31:13.345040+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:34:25.158893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308972+0000","last_change":"2024-04-02T13:31:11.128713+0000","last_active":"2024-04-02T13:31:13.308972+0000","last_peered":"2024-04-02T13:31:13.308972+0000","last_clean":"2024-04-02T13:31:13.308972+0000","last_became_active":"2024-04-02T13:31:11.128388+0000","last_became_peered":"2024-04-02T13:31:11.128388+0000","last_unstale":"2024-04-02T13:31:13.308972+0000","last_undegraded":"2024-04-02T13:31:13.308972+0000","last_fullsized":"2024-04-02T13:31:13.308972+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:33:55.018213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440090+0000","last_change":"2024-04-02T13:31:12.130202+0000","last_active":"2024-04-02T13:31:13.440090+0000","last_peered":"2024-04-02T13:31:13.440090+0000","last_clean":"2024-04-02T13:31:13.440090+0000","last_became_active":"2024-04-02T13:31:12.129938+0000","last_became_peered":"2024-04-02T13:31:12.129938+0000","last_unstale":"2024-04-02T13:31:13.440090+0000","last_undegraded":"2024-04-02T13:31:13.440090+0000","last_fullsized":"2024-04-02T13:31:13.440090+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:20:30.993334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308857+0000","last_change":"2024-04-02T13:31:11.133017+0000","last_active":"2024-04-02T13:31:13.308857+0000","last_peered":"2024-04-02T13:31:13.308857+0000","last_clean":"2024-04-02T13:31:13.308857+0000","last_became_active":"2024-04-02T13:31:11.132360+0000","last_became_peered":"2024-04-02T13:31:11.132360+0000","last_unstale":"2024-04-02T13:31:13.308857+0000","last_undegraded":"2024-04-02T13:31:13.308857+0000","last_fullsized":"2024-04-02T13:31:13.308857+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:57:23.578410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138745+0000","last_change":"2024-04-02T13:31:12.142395+0000","last_active":"2024-04-02T13:31:13.138745+0000","last_peered":"2024-04-02T13:31:13.138745+0000","last_clean":"2024-04-02T13:31:13.138745+0000","last_became_active":"2024-04-02T13:31:12.142108+0000","last_became_peered":"2024-04-02T13:31:12.142108+0000","last_unstale":"2024-04-02T13:31:13.138745+0000","last_undegraded":"2024-04-02T13:31:13.138745+0000","last_fullsized":"2024-04-02T13:31:13.138745+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:33:56.023379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345265+0000","last_change":"2024-04-02T13:31:11.135311+0000","last_active":"2024-04-02T13:31:13.345265+0000","last_peered":"2024-04-02T13:31:13.345265+0000","last_clean":"2024-04-02T13:31:13.345265+0000","last_became_active":"2024-04-02T13:31:11.135038+0000","last_became_peered":"2024-04-02T13:31:11.135038+0000","last_unstale":"2024-04-02T13:31:13.345265+0000","last_undegraded":"2024-04-02T13:31:13.345265+0000","last_fullsized":"2024-04-02T13:31:13.345265+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:04:37.616135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441074+0000","last_change":"2024-04-02T13:31:12.143962+0000","last_active":"2024-04-02T13:31:13.441074+0000","last_peered":"2024-04-02T13:31:13.441074+0000","last_clean":"2024-04-02T13:31:13.441074+0000","last_became_active":"2024-04-02T13:31:12.143694+0000","last_became_peered":"2024-04-02T13:31:12.143694+0000","last_unstale":"2024-04-02T13:31:13.441074+0000","last_undegraded":"2024-04-02T13:31:13.441074+0000","last_fullsized":"2024-04-02T13:31:13.441074+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:24:37.992698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440250+0000","last_change":"2024-04-02T13:31:11.135836+0000","last_active":"2024-04-02T13:31:13.440250+0000","last_peered":"2024-04-02T13:31:13.440250+0000","last_clean":"2024-04-02T13:31:13.440250+0000","last_became_active":"2024-04-02T13:31:11.135535+0000","last_became_peered":"2024-04-02T13:31:11.135535+0000","last_unstale":"2024-04-02T13:31:13.440250+0000","last_undegraded":"2024-04-02T13:31:13.440250+0000","last_fullsized":"2024-04-02T13:31:13.440250+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:21:08.296959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139253+0000","last_change":"2024-04-02T13:31:12.142067+0000","last_active":"2024-04-02T13:31:13.139253+0000","last_peered":"2024-04-02T13:31:13.139253+0000","last_clean":"2024-04-02T13:31:13.139253+0000","last_became_active":"2024-04-02T13:31:12.141782+0000","last_became_peered":"2024-04-02T13:31:12.141782+0000","last_unstale":"2024-04-02T13:31:13.139253+0000","last_undegraded":"2024-04-02T13:31:13.139253+0000","last_fullsized":"2024-04-02T13:31:13.139253+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:39:40.128741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140646+0000","last_change":"2024-04-02T13:31:11.126878+0000","last_active":"2024-04-02T13:31:13.140646+0000","last_peered":"2024-04-02T13:31:13.140646+0000","last_clean":"2024-04-02T13:31:13.140646+0000","last_became_active":"2024-04-02T13:31:11.126520+0000","last_became_peered":"2024-04-02T13:31:11.126520+0000","last_unstale":"2024-04-02T13:31:13.140646+0000","last_undegraded":"2024-04-02T13:31:13.140646+0000","last_fullsized":"2024-04-02T13:31:13.140646+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:45:44.331093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138979+0000","last_change":"2024-04-02T13:31:12.134022+0000","last_active":"2024-04-02T13:31:13.138979+0000","last_peered":"2024-04-02T13:31:13.138979+0000","last_clean":"2024-04-02T13:31:13.138979+0000","last_became_active":"2024-04-02T13:31:12.133677+0000","last_became_peered":"2024-04-02T13:31:12.133677+0000","last_unstale":"2024-04-02T13:31:13.138979+0000","last_undegraded":"2024-04-02T13:31:13.138979+0000","last_fullsized":"2024-04-02T13:31:13.138979+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:38:42.298292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140496+0000","last_change":"2024-04-02T13:31:11.136359+0000","last_active":"2024-04-02T13:31:13.140496+0000","last_peered":"2024-04-02T13:31:13.140496+0000","last_clean":"2024-04-02T13:31:13.140496+0000","last_became_active":"2024-04-02T13:31:11.136154+0000","last_became_peered":"2024-04-02T13:31:11.136154+0000","last_unstale":"2024-04-02T13:31:13.140496+0000","last_undegraded":"2024-04-02T13:31:13.140496+0000","last_fullsized":"2024-04-02T13:31:13.140496+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:11:28.832303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138860+0000","last_change":"2024-04-02T13:31:12.136151+0000","last_active":"2024-04-02T13:31:13.138860+0000","last_peered":"2024-04-02T13:31:13.138860+0000","last_clean":"2024-04-02T13:31:13.138860+0000","last_became_active":"2024-04-02T13:31:12.135868+0000","last_became_peered":"2024-04-02T13:31:12.135868+0000","last_unstale":"2024-04-02T13:31:13.138860+0000","last_undegraded":"2024-04-02T13:31:13.138860+0000","last_fullsized":"2024-04-02T13:31:13.138860+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:07.907914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139008+0000","last_change":"2024-04-02T13:31:11.136321+0000","last_active":"2024-04-02T13:31:13.139008+0000","last_peered":"2024-04-02T13:31:13.139008+0000","last_clean":"2024-04-02T13:31:13.139008+0000","last_became_active":"2024-04-02T13:31:11.135962+0000","last_became_peered":"2024-04-02T13:31:11.135962+0000","last_unstale":"2024-04-02T13:31:13.139008+0000","last_undegraded":"2024-04-02T13:31:13.139008+0000","last_fullsized":"2024-04-02T13:31:13.139008+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:33:23.184101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308610+0000","last_change":"2024-04-02T13:31:12.142057+0000","last_active":"2024-04-02T13:31:13.308610+0000","last_peered":"2024-04-02T13:31:13.308610+0000","last_clean":"2024-04-02T13:31:13.308610+0000","last_became_active":"2024-04-02T13:31:12.141532+0000","last_became_peered":"2024-04-02T13:31:12.141532+0000","last_unstale":"2024-04-02T13:31:13.308610+0000","last_undegraded":"2024-04-02T13:31:13.308610+0000","last_fullsized":"2024-04-02T13:31:13.308610+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:37:08.666978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140068+0000","last_change":"2024-04-02T13:31:11.133500+0000","last_active":"2024-04-02T13:31:13.140068+0000","last_peered":"2024-04-02T13:31:13.140068+0000","last_clean":"2024-04-02T13:31:13.140068+0000","last_became_active":"2024-04-02T13:31:11.133225+0000","last_became_peered":"2024-04-02T13:31:11.133225+0000","last_unstale":"2024-04-02T13:31:13.140068+0000","last_undegraded":"2024-04-02T13:31:13.140068+0000","last_fullsized":"2024-04-02T13:31:13.140068+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:44.100936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308967+0000","last_change":"2024-04-02T13:31:12.136866+0000","last_active":"2024-04-02T13:31:13.308967+0000","last_peered":"2024-04-02T13:31:13.308967+0000","last_clean":"2024-04-02T13:31:13.308967+0000","last_became_active":"2024-04-02T13:31:12.136496+0000","last_became_peered":"2024-04-02T13:31:12.136496+0000","last_unstale":"2024-04-02T13:31:13.308967+0000","last_undegraded":"2024-04-02T13:31:13.308967+0000","last_fullsized":"2024-04-02T13:31:13.308967+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:48:48.771050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308840+0000","last_change":"2024-04-02T13:31:11.134053+0000","last_active":"2024-04-02T13:31:13.308840+0000","last_peered":"2024-04-02T13:31:13.308840+0000","last_clean":"2024-04-02T13:31:13.308840+0000","last_became_active":"2024-04-02T13:31:11.133716+0000","last_became_peered":"2024-04-02T13:31:11.133716+0000","last_unstale":"2024-04-02T13:31:13.308840+0000","last_undegraded":"2024-04-02T13:31:13.308840+0000","last_fullsized":"2024-04-02T13:31:13.308840+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:32:28.191412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138339+0000","last_change":"2024-04-02T13:31:12.136384+0000","last_active":"2024-04-02T13:31:13.138339+0000","last_peered":"2024-04-02T13:31:13.138339+0000","last_clean":"2024-04-02T13:31:13.138339+0000","last_became_active":"2024-04-02T13:31:12.136132+0000","last_became_peered":"2024-04-02T13:31:12.136132+0000","last_unstale":"2024-04-02T13:31:13.138339+0000","last_undegraded":"2024-04-02T13:31:13.138339+0000","last_fullsized":"2024-04-02T13:31:13.138339+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:45:45.336259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139865+0000","last_change":"2024-04-02T13:31:11.131870+0000","last_active":"2024-04-02T13:31:13.139865+0000","last_peered":"2024-04-02T13:31:13.139865+0000","last_clean":"2024-04-02T13:31:13.139865+0000","last_became_active":"2024-04-02T13:31:11.131567+0000","last_became_peered":"2024-04-02T13:31:11.131567+0000","last_unstale":"2024-04-02T13:31:13.139865+0000","last_undegraded":"2024-04-02T13:31:13.139865+0000","last_fullsized":"2024-04-02T13:31:13.139865+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:42:08.720811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139403+0000","last_change":"2024-04-02T13:31:12.145022+0000","last_active":"2024-04-02T13:31:13.139403+0000","last_peered":"2024-04-02T13:31:13.139403+0000","last_clean":"2024-04-02T13:31:13.139403+0000","last_became_active":"2024-04-02T13:31:12.144768+0000","last_became_peered":"2024-04-02T13:31:12.144768+0000","last_unstale":"2024-04-02T13:31:13.139403+0000","last_undegraded":"2024-04-02T13:31:13.139403+0000","last_fullsized":"2024-04-02T13:31:13.139403+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:34.217067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139224+0000","last_change":"2024-04-02T13:31:11.138081+0000","last_active":"2024-04-02T13:31:13.139224+0000","last_peered":"2024-04-02T13:31:13.139224+0000","last_clean":"2024-04-02T13:31:13.139224+0000","last_became_active":"2024-04-02T13:31:11.137830+0000","last_became_peered":"2024-04-02T13:31:11.137830+0000","last_unstale":"2024-04-02T13:31:13.139224+0000","last_undegraded":"2024-04-02T13:31:13.139224+0000","last_fullsized":"2024-04-02T13:31:13.139224+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:31.359210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345094+0000","last_change":"2024-04-02T13:31:12.140943+0000","last_active":"2024-04-02T13:31:13.345094+0000","last_peered":"2024-04-02T13:31:13.345094+0000","last_clean":"2024-04-02T13:31:13.345094+0000","last_became_active":"2024-04-02T13:31:12.140505+0000","last_became_peered":"2024-04-02T13:31:12.140505+0000","last_unstale":"2024-04-02T13:31:13.345094+0000","last_undegraded":"2024-04-02T13:31:13.345094+0000","last_fullsized":"2024-04-02T13:31:13.345094+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:11:29.837470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441507+0000","last_change":"2024-04-02T13:31:11.140019+0000","last_active":"2024-04-02T13:31:13.441507+0000","last_peered":"2024-04-02T13:31:13.441507+0000","last_clean":"2024-04-02T13:31:13.441507+0000","last_became_active":"2024-04-02T13:31:11.139233+0000","last_became_peered":"2024-04-02T13:31:11.139233+0000","last_unstale":"2024-04-02T13:31:13.441507+0000","last_undegraded":"2024-04-02T13:31:13.441507+0000","last_fullsized":"2024-04-02T13:31:13.441507+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:38.434488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140698+0000","last_change":"2024-04-02T13:31:12.131347+0000","last_active":"2024-04-02T13:31:13.140698+0000","last_peered":"2024-04-02T13:31:13.140698+0000","last_clean":"2024-04-02T13:31:13.140698+0000","last_became_active":"2024-04-02T13:31:12.131054+0000","last_became_peered":"2024-04-02T13:31:12.131054+0000","last_unstale":"2024-04-02T13:31:13.140698+0000","last_undegraded":"2024-04-02T13:31:13.140698+0000","last_fullsized":"2024-04-02T13:31:13.140698+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:30:55.860990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441565+0000","last_change":"2024-04-02T13:31:11.139638+0000","last_active":"2024-04-02T13:31:13.441565+0000","last_peered":"2024-04-02T13:31:13.441565+0000","last_clean":"2024-04-02T13:31:13.441565+0000","last_became_active":"2024-04-02T13:31:11.139319+0000","last_became_peered":"2024-04-02T13:31:11.139319+0000","last_unstale":"2024-04-02T13:31:13.441565+0000","last_undegraded":"2024-04-02T13:31:13.441565+0000","last_fullsized":"2024-04-02T13:31:13.441565+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:00:25.458770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440331+0000","last_change":"2024-04-02T13:31:12.127598+0000","last_active":"2024-04-02T13:31:13.440331+0000","last_peered":"2024-04-02T13:31:13.440331+0000","last_clean":"2024-04-02T13:31:13.440331+0000","last_became_active":"2024-04-02T13:31:12.127333+0000","last_became_peered":"2024-04-02T13:31:12.127333+0000","last_unstale":"2024-04-02T13:31:13.440331+0000","last_undegraded":"2024-04-02T13:31:13.440331+0000","last_fullsized":"2024-04-02T13:31:13.440331+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:08:15.789954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139487+0000","last_change":"2024-04-02T13:31:11.134862+0000","last_active":"2024-04-02T13:31:13.139487+0000","last_peered":"2024-04-02T13:31:13.139487+0000","last_clean":"2024-04-02T13:31:13.139487+0000","last_became_active":"2024-04-02T13:31:11.134611+0000","last_became_peered":"2024-04-02T13:31:11.134611+0000","last_unstale":"2024-04-02T13:31:13.139487+0000","last_undegraded":"2024-04-02T13:31:13.139487+0000","last_fullsized":"2024-04-02T13:31:13.139487+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:38.536661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139027+0000","last_change":"2024-04-02T13:31:12.135944+0000","last_active":"2024-04-02T13:31:13.139027+0000","last_peered":"2024-04-02T13:31:13.139027+0000","last_clean":"2024-04-02T13:31:13.139027+0000","last_became_active":"2024-04-02T13:31:12.135628+0000","last_became_peered":"2024-04-02T13:31:12.135628+0000","last_unstale":"2024-04-02T13:31:13.139027+0000","last_undegraded":"2024-04-02T13:31:13.139027+0000","last_fullsized":"2024-04-02T13:31:13.139027+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:33:09.023674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441026+0000","last_change":"2024-04-02T13:31:11.139793+0000","last_active":"2024-04-02T13:31:13.441026+0000","last_peered":"2024-04-02T13:31:13.441026+0000","last_clean":"2024-04-02T13:31:13.441026+0000","last_became_active":"2024-04-02T13:31:11.139508+0000","last_became_peered":"2024-04-02T13:31:11.139508+0000","last_unstale":"2024-04-02T13:31:13.441026+0000","last_undegraded":"2024-04-02T13:31:13.441026+0000","last_fullsized":"2024-04-02T13:31:13.441026+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:43:40.243740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308629+0000","last_change":"2024-04-02T13:31:12.142629+0000","last_active":"2024-04-02T13:31:13.308629+0000","last_peered":"2024-04-02T13:31:13.308629+0000","last_clean":"2024-04-02T13:31:13.308629+0000","last_became_active":"2024-04-02T13:31:12.142367+0000","last_became_peered":"2024-04-02T13:31:12.142367+0000","last_unstale":"2024-04-02T13:31:13.308629+0000","last_undegraded":"2024-04-02T13:31:13.308629+0000","last_fullsized":"2024-04-02T13:31:13.308629+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:39.541828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440919+0000","last_change":"2024-04-02T13:31:11.138347+0000","last_active":"2024-04-02T13:31:13.440919+0000","last_peered":"2024-04-02T13:31:13.440919+0000","last_clean":"2024-04-02T13:31:13.440919+0000","last_became_active":"2024-04-02T13:31:11.138045+0000","last_became_peered":"2024-04-02T13:31:11.138045+0000","last_unstale":"2024-04-02T13:31:13.440919+0000","last_undegraded":"2024-04-02T13:31:13.440919+0000","last_fullsized":"2024-04-02T13:31:13.440919+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:04:37.616135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140120+0000","last_change":"2024-04-02T13:31:12.135698+0000","last_active":"2024-04-02T13:31:13.140120+0000","last_peered":"2024-04-02T13:31:13.140120+0000","last_clean":"2024-04-02T13:31:13.140120+0000","last_became_active":"2024-04-02T13:31:12.135427+0000","last_became_peered":"2024-04-02T13:31:12.135427+0000","last_unstale":"2024-04-02T13:31:13.140120+0000","last_undegraded":"2024-04-02T13:31:13.140120+0000","last_fullsized":"2024-04-02T13:31:13.140120+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:05.199702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138872+0000","last_change":"2024-04-02T13:31:11.134119+0000","last_active":"2024-04-02T13:31:13.138872+0000","last_peered":"2024-04-02T13:31:13.138872+0000","last_clean":"2024-04-02T13:31:13.138872+0000","last_became_active":"2024-04-02T13:31:11.133880+0000","last_became_peered":"2024-04-02T13:31:11.133880+0000","last_unstale":"2024-04-02T13:31:13.138872+0000","last_undegraded":"2024-04-02T13:31:13.138872+0000","last_fullsized":"2024-04-02T13:31:13.138872+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:37:07.661812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345208+0000","last_change":"2024-04-02T13:31:12.141647+0000","last_active":"2024-04-02T13:31:13.345208+0000","last_peered":"2024-04-02T13:31:13.345208+0000","last_clean":"2024-04-02T13:31:13.345208+0000","last_became_active":"2024-04-02T13:31:12.141380+0000","last_became_peered":"2024-04-02T13:31:12.141380+0000","last_unstale":"2024-04-02T13:31:13.345208+0000","last_undegraded":"2024-04-02T13:31:13.345208+0000","last_fullsized":"2024-04-02T13:31:13.345208+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:06:45.310008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139872+0000","last_change":"2024-04-02T13:31:11.135129+0000","last_active":"2024-04-02T13:31:13.139872+0000","last_peered":"2024-04-02T13:31:13.139872+0000","last_clean":"2024-04-02T13:31:13.139872+0000","last_became_active":"2024-04-02T13:31:11.134810+0000","last_became_peered":"2024-04-02T13:31:11.134810+0000","last_unstale":"2024-04-02T13:31:13.139872+0000","last_undegraded":"2024-04-02T13:31:13.139872+0000","last_fullsized":"2024-04-02T13:31:13.139872+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:14:15.652335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140487+0000","last_change":"2024-04-02T13:31:12.136674+0000","last_active":"2024-04-02T13:31:13.140487+0000","last_peered":"2024-04-02T13:31:13.140487+0000","last_clean":"2024-04-02T13:31:13.140487+0000","last_became_active":"2024-04-02T13:31:12.136399+0000","last_became_peered":"2024-04-02T13:31:12.136399+0000","last_unstale":"2024-04-02T13:31:13.140487+0000","last_undegraded":"2024-04-02T13:31:13.140487+0000","last_fullsized":"2024-04-02T13:31:13.140487+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:22:03.194530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"120'8","reported_seq":22,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308482+0000","last_change":"2024-04-02T13:31:11.135308+0000","last_active":"2024-04-02T13:31:13.308482+0000","last_peered":"2024-04-02T13:31:13.308482+0000","last_clean":"2024-04-02T13:31:13.308482+0000","last_became_active":"2024-04-02T13:31:11.135021+0000","last_became_peered":"2024-04-02T13:31:11.135021+0000","last_unstale":"2024-04-02T13:31:13.308482+0000","last_undegraded":"2024-04-02T13:31:13.308482+0000","last_fullsized":"2024-04-02T13:31:13.308482+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:11.221991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":365,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139987+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:31:13.139987+0000","last_peered":"2024-04-02T13:31:13.139987+0000","last_clean":"2024-04-02T13:31:13.139987+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:31:13.139987+0000","last_undegraded":"2024-04-02T13:31:13.139987+0000","last_fullsized":"2024-04-02T13:31:13.139987+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574983,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33252,"kb_used_data":2068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338588,"statfs":{"total":96636764160,"available":96602714112,"internally_reserved":0,"allocated":2117632,"data_stored":1295657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574982,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1499136,"data_stored":698403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574983,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1515520,"data_stored":704673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574983,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32656,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":1507328,"data_stored":699433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574983,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1523712,"data_stored":706159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574982,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1507328,"data_stored":704513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574982,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1499136,"data_stored":697843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574983,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33240,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338600,"statfs":{"total":96636764160,"available":96602726400,"internally_reserved":0,"allocated":2105344,"data_stored":1294903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:31:15.481 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:31:15.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.616+0000 7f20e639d640 1 Processor -- start 2024-04-02T13:31:15.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.616+0000 7f20e639d640 1 -- start start 2024-04-02T13:31:15.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20e639d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e015e4a0 0x7f20e00aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20e639d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e00cc600 con 0x7f20e006d550 2024-04-02T13:31:15.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20e639d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e00cc8e0 con 0x7f20e00d0c50 2024-04-02T13:31:15.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20e639d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f20e00ccbc0 con 0x7f20e015e4a0 2024-04-02T13:31:15.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20e493a640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f20e006d550 0x7f20e00cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40960/0 (socket says 172.21.15.44:40960) 2024-04-02T13:31:15.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20e493a640 1 -- 172.21.15.44:0/275241592 learned_addr learned my addr 172.21.15.44:0/275241592 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:15.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2509157950 0 0) 0x7f20e00cc600 con 0x7f20e006d550 2024-04-02T13:31:15.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20c8003520 con 0x7f20e006d550 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20dffff640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e015e4a0 0x7f20e00aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1932900950 0 0) 0x7f20c8003520 con 0x7f20e006d550 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e015e4a0 msgr2=0x7f20e00aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e015e4a0 0x7f20e00aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 >> v1:172.21.15.69:6789/0 conn(0x7f20e00d0c50 legacy=0x7f20e006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e00ccea0 con 0x7f20e006d550 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f20d80019f0 con 0x7f20e006d550 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 >> v1:172.21.15.44:6789/0 conn(0x7f20e006d550 legacy=0x7f20e00cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e015e4a0 0x7f20c80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20c8003f60 0x7f20c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20c8004d40 0x7f20c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f20e00ccbc0 con 0x7f20c8003f60 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f20c800baa0 con 0x7f20c8004d40 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f20c800bca0 con 0x7f20e015e4a0 2024-04-02T13:31:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.617+0000 7f20df7fe640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20c8003f60 0x7f20c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.618+0000 7f20dffff640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e015e4a0 0x7f20c80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.618+0000 7f20e493a640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20c8004d40 0x7f20c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.618+0000 7f20df7fe640 1 -- 172.21.15.44:0/275241592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e015e4a0 msgr2=0x7f20c80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.618+0000 7f20df7fe640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e015e4a0 0x7f20c80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.618+0000 7f20df7fe640 1 -- 172.21.15.44:0/275241592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20c8004d40 msgr2=0x7f20c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.618+0000 7f20df7fe640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20c8004d40 0x7f20c80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.618+0000 7f20df7fe640 1 -- 172.21.15.44:0/275241592 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c8003520 con 0x7f20c8003f60 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.618+0000 7f20dffff640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e015e4a0 0x7f20c80039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.618+0000 7f20e493a640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20c8004d40 0x7f20c80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.619+0000 7f20df7fe640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20c8003f60 0x7f20c8004800 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f20d4001750 tx=0x7f20c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ac671c2499bfba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.619+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4001370 con 0x7f20c8003f60 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.619+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d40052d0 con 0x7f20c8003f60 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.619+0000 7f20deffd640 1 -- 172.21.15.44:0/275241592 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4003830 con 0x7f20c8003f60 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.619+0000 7f20e639d640 1 -- 172.21.15.44:0/275241592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20c8003f60 msgr2=0x7f20c8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.619+0000 7f20e639d640 1 --2- 172.21.15.44:0/275241592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20c8003f60 0x7f20c8004800 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f20d4001750 tx=0x7f20c8005830 comp rx=0 tx=0).stop 2024-04-02T13:31:15.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.619+0000 7f20e493a640 1 -- 172.21.15.44:0/275241592 reap_dead start 2024-04-02T13:31:15.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.619+0000 7f20e639d640 1 -- 172.21.15.44:0/275241592 shutdown_connections 2024-04-02T13:31:15.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.619+0000 7f20e639d640 1 -- 172.21.15.44:0/275241592 >> 172.21.15.44:0/275241592 conn(0x7f20e009aa20 msgr2=0x7f20e009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:15.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.619+0000 7f20e639d640 1 -- 172.21.15.44:0/275241592 shutdown_connections 2024-04-02T13:31:15.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.619+0000 7f20e639d640 1 -- 172.21.15.44:0/275241592 wait complete. 2024-04-02T13:31:15.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.620+0000 7f20e639d640 1 Processor -- start 2024-04-02T13:31:15.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.621+0000 7f20e639d640 1 -- start start 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.621+0000 7f20e639d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20c8003f60 0x7f20e01f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.621+0000 7f20e639d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e014c7f0 0x7f20e01f5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.621+0000 7f20e639d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20e015ea00 0x7f20e014bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.621+0000 7f20e639d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f20e015e880 con 0x7f20c8003f60 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.621+0000 7f20e639d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f20e00cd460 con 0x7f20e015ea00 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.621+0000 7f20e639d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f20e00ce090 con 0x7f20e014c7f0 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.621+0000 7f20dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20c8003f60 0x7f20e01f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.621+0000 7f20dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20c8003f60 0x7f20e01f3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53600/0 (socket says 172.21.15.44:53600) 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.621+0000 7f20dffff640 1 -- 172.21.15.44:0/581972256 learned_addr learned my addr 172.21.15.44:0/581972256 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.621+0000 7f20df7fe640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e014c7f0 0x7f20e01f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.621+0000 7f20e493a640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20e015ea00 0x7f20e014bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.622+0000 7f20dffff640 1 -- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e014c7f0 msgr2=0x7f20e01f5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.622+0000 7f20dffff640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e014c7f0 0x7f20e01f5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.622+0000 7f20dffff640 1 -- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20e015ea00 msgr2=0x7f20e014bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.622+0000 7f20dffff640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20e015ea00 0x7f20e014bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.622+0000 7f20dffff640 1 -- 172.21.15.44:0/581972256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e00c73d0 con 0x7f20c8003f60 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.622+0000 7f20e493a640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20e015ea00 0x7f20e014bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.622+0000 7f20dffff640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20c8003f60 0x7f20e01f3fc0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f20d0011d00 tx=0x7f20d000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.622+0000 7f20dcff9640 1 -- 172.21.15.44:0/581972256 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d000fd20 con 0x7f20c8003f60 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.622+0000 7f20dcff9640 1 -- 172.21.15.44:0/581972256 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d0014440 con 0x7f20c8003f60 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.622+0000 7f20dcff9640 1 -- 172.21.15.44:0/581972256 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0014830 con 0x7f20c8003f60 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.622+0000 7f20e639d640 1 -- 172.21.15.44:0/581972256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20e00c76b0 con 0x7f20c8003f60 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.622+0000 7f20e639d640 1 -- 172.21.15.44:0/581972256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20e00c7c70 con 0x7f20c8003f60 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.623+0000 7f20e639d640 1 -- 172.21.15.44:0/581972256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e00684f0 con 0x7f20c8003f60 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.623+0000 7f20dcff9640 1 -- 172.21.15.44:0/581972256 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f20d0012040 con 0x7f20c8003f60 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.624+0000 7f20dcff9640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20cc06c8f0 0x7f20cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.624+0000 7f20dcff9640 1 -- 172.21.15.44:0/581972256 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f20d008c440 con 0x7f20c8003f60 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.624+0000 7f20df7fe640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20cc06c8f0 0x7f20cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.625+0000 7f20df7fe640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20cc06c8f0 0x7f20cc06edb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f20d4033000 tx=0x7f20d4009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:15.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.628+0000 7f20dcff9640 1 -- 172.21.15.44:0/581972256 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f20d0019030 con 0x7f20c8003f60 2024-04-02T13:31:15.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.812+0000 7f20e639d640 1 -- 172.21.15.44:0/581972256 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f20e01f4500 con 0x7f20cc06c8f0 2024-04-02T13:31:15.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.818+0000 7f20dcff9640 1 -- 172.21.15.44:0/581972256 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252498 (secure 0 0 0) 0x7f20e01f4500 con 0x7f20cc06c8f0 2024-04-02T13:31:15.824 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:15.825 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:31:15.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.821+0000 7f20e639d640 1 -- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20cc06c8f0 msgr2=0x7f20cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.821+0000 7f20e639d640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20cc06c8f0 0x7f20cc06edb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f20d4033000 tx=0x7f20d4009030 comp rx=0 tx=0).stop 2024-04-02T13:31:15.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.821+0000 7f20e639d640 1 -- 172.21.15.44:0/581972256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20c8003f60 msgr2=0x7f20e01f3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.821+0000 7f20e639d640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20c8003f60 0x7f20e01f3fc0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f20d0011d00 tx=0x7f20d000f610 comp rx=0 tx=0).stop 2024-04-02T13:31:15.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.821+0000 7f20e639d640 1 -- 172.21.15.44:0/581972256 shutdown_connections 2024-04-02T13:31:15.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.821+0000 7f20e639d640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f20cc06c8f0 0x7f20cc06edb0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.821+0000 7f20e639d640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20e015ea00 0x7f20e014bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.821+0000 7f20e639d640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e014c7f0 0x7f20e01f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.821+0000 7f20e639d640 1 --2- 172.21.15.44:0/581972256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20c8003f60 0x7f20e01f3fc0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.821+0000 7f20e639d640 1 -- 172.21.15.44:0/581972256 >> 172.21.15.44:0/581972256 conn(0x7f20e009aa20 msgr2=0x7f20e00d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:15.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.822+0000 7f20e639d640 1 -- 172.21.15.44:0/581972256 shutdown_connections 2024-04-02T13:31:15.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.822+0000 7f20e639d640 1 -- 172.21.15.44:0/581972256 wait complete. 2024-04-02T13:31:15.835 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2024-04-02T13:31:15.038952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262448,"kb_used_data":12964,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712272,"statfs":{"total":773094113280,"available":772825366528,"internally_reserved":0,"allocated":13275136,"data_stored":6801584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.922770"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441174+0000","last_change":"2024-04-02T13:31:12.142684+0000","last_active":"2024-04-02T13:31:13.441174+0000","last_peered":"2024-04-02T13:31:13.441174+0000","last_clean":"2024-04-02T13:31:13.441174+0000","last_became_active":"2024-04-02T13:31:12.142401+0000","last_became_peered":"2024-04-02T13:31:12.142401+0000","last_unstale":"2024-04-02T13:31:13.441174+0000","last_undegraded":"2024-04-02T13:31:13.441174+0000","last_fullsized":"2024-04-02T13:31:13.441174+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:45:45.336259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308830+0000","last_change":"2024-04-02T13:31:12.154695+0000","last_active":"2024-04-02T13:31:13.308830+0000","last_peered":"2024-04-02T13:31:13.308830+0000","last_clean":"2024-04-02T13:31:13.308830+0000","last_became_active":"2024-04-02T13:31:12.154371+0000","last_became_peered":"2024-04-02T13:31:12.154371+0000","last_unstale":"2024-04-02T13:31:13.308830+0000","last_undegraded":"2024-04-02T13:31:13.308830+0000","last_fullsized":"2024-04-02T13:31:13.308830+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:21:14.776344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.309022+0000","last_change":"2024-04-02T13:31:12.142513+0000","last_active":"2024-04-02T13:31:13.309022+0000","last_peered":"2024-04-02T13:31:13.309022+0000","last_clean":"2024-04-02T13:31:13.309022+0000","last_became_active":"2024-04-02T13:31:12.142238+0000","last_became_peered":"2024-04-02T13:31:12.142238+0000","last_unstale":"2024-04-02T13:31:13.309022+0000","last_undegraded":"2024-04-02T13:31:13.309022+0000","last_fullsized":"2024-04-02T13:31:13.309022+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:35:31.717737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140458+0000","last_change":"2024-04-02T13:31:12.133658+0000","last_active":"2024-04-02T13:31:13.140458+0000","last_peered":"2024-04-02T13:31:13.140458+0000","last_clean":"2024-04-02T13:31:13.140458+0000","last_became_active":"2024-04-02T13:31:12.133301+0000","last_became_peered":"2024-04-02T13:31:12.133301+0000","last_unstale":"2024-04-02T13:31:13.140458+0000","last_undegraded":"2024-04-02T13:31:13.140458+0000","last_fullsized":"2024-04-02T13:31:13.140458+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:15:32.709627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440763+0000","last_change":"2024-04-02T13:31:12.130711+0000","last_active":"2024-04-02T13:31:13.440763+0000","last_peered":"2024-04-02T13:31:13.440763+0000","last_clean":"2024-04-02T13:31:13.440763+0000","last_became_active":"2024-04-02T13:31:12.130422+0000","last_became_peered":"2024-04-02T13:31:12.130422+0000","last_unstale":"2024-04-02T13:31:13.440763+0000","last_undegraded":"2024-04-02T13:31:13.440763+0000","last_fullsized":"2024-04-02T13:31:13.440763+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:45.106103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138983+0000","last_change":"2024-04-02T13:31:12.147698+0000","last_active":"2024-04-02T13:31:13.138983+0000","last_peered":"2024-04-02T13:31:13.138983+0000","last_clean":"2024-04-02T13:31:13.138983+0000","last_became_active":"2024-04-02T13:31:12.147413+0000","last_became_peered":"2024-04-02T13:31:12.147413+0000","last_unstale":"2024-04-02T13:31:13.138983+0000","last_undegraded":"2024-04-02T13:31:13.138983+0000","last_fullsized":"2024-04-02T13:31:13.138983+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:12:24.001590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138929+0000","last_change":"2024-04-02T13:31:12.140695+0000","last_active":"2024-04-02T13:31:13.138929+0000","last_peered":"2024-04-02T13:31:13.138929+0000","last_clean":"2024-04-02T13:31:13.138929+0000","last_became_active":"2024-04-02T13:31:12.140423+0000","last_became_peered":"2024-04-02T13:31:12.140423+0000","last_unstale":"2024-04-02T13:31:13.138929+0000","last_undegraded":"2024-04-02T13:31:13.138929+0000","last_fullsized":"2024-04-02T13:31:13.138929+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:15:04.901097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308527+0000","last_change":"2024-04-02T13:31:12.145293+0000","last_active":"2024-04-02T13:31:13.308527+0000","last_peered":"2024-04-02T13:31:13.308527+0000","last_clean":"2024-04-02T13:31:13.308527+0000","last_became_active":"2024-04-02T13:31:12.145022+0000","last_became_peered":"2024-04-02T13:31:12.145022+0000","last_unstale":"2024-04-02T13:31:13.308527+0000","last_undegraded":"2024-04-02T13:31:13.308527+0000","last_fullsized":"2024-04-02T13:31:13.308527+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:32.364376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140759+0000","last_change":"2024-04-02T13:31:12.147556+0000","last_active":"2024-04-02T13:31:13.140759+0000","last_peered":"2024-04-02T13:31:13.140759+0000","last_clean":"2024-04-02T13:31:13.140759+0000","last_became_active":"2024-04-02T13:31:12.147230+0000","last_became_peered":"2024-04-02T13:31:12.147230+0000","last_unstale":"2024-04-02T13:31:13.140759+0000","last_undegraded":"2024-04-02T13:31:13.140759+0000","last_fullsized":"2024-04-02T13:31:13.140759+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:43:44.956087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345155+0000","last_change":"2024-04-02T13:31:12.143114+0000","last_active":"2024-04-02T13:31:13.345155+0000","last_peered":"2024-04-02T13:31:13.345155+0000","last_clean":"2024-04-02T13:31:13.345155+0000","last_became_active":"2024-04-02T13:31:12.142838+0000","last_became_peered":"2024-04-02T13:31:12.142838+0000","last_unstale":"2024-04-02T13:31:13.345155+0000","last_undegraded":"2024-04-02T13:31:13.345155+0000","last_fullsized":"2024-04-02T13:31:13.345155+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:45:45.336259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440889+0000","last_change":"2024-04-02T13:31:12.127457+0000","last_active":"2024-04-02T13:31:13.440889+0000","last_peered":"2024-04-02T13:31:13.440889+0000","last_clean":"2024-04-02T13:31:13.440889+0000","last_became_active":"2024-04-02T13:31:12.127164+0000","last_became_peered":"2024-04-02T13:31:12.127164+0000","last_unstale":"2024-04-02T13:31:13.440889+0000","last_undegraded":"2024-04-02T13:31:13.440889+0000","last_fullsized":"2024-04-02T13:31:13.440889+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:11:29.837470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139260+0000","last_change":"2024-04-02T13:31:12.140930+0000","last_active":"2024-04-02T13:31:13.139260+0000","last_peered":"2024-04-02T13:31:13.139260+0000","last_clean":"2024-04-02T13:31:13.139260+0000","last_became_active":"2024-04-02T13:31:12.140539+0000","last_became_peered":"2024-04-02T13:31:12.140539+0000","last_unstale":"2024-04-02T13:31:13.139260+0000","last_undegraded":"2024-04-02T13:31:13.139260+0000","last_fullsized":"2024-04-02T13:31:13.139260+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:20.933022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345142+0000","last_change":"2024-04-02T13:31:12.142675+0000","last_active":"2024-04-02T13:31:13.345142+0000","last_peered":"2024-04-02T13:31:13.345142+0000","last_clean":"2024-04-02T13:31:13.345142+0000","last_became_active":"2024-04-02T13:31:12.142221+0000","last_became_peered":"2024-04-02T13:31:12.142221+0000","last_unstale":"2024-04-02T13:31:13.345142+0000","last_undegraded":"2024-04-02T13:31:13.345142+0000","last_fullsized":"2024-04-02T13:31:13.345142+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:24:37.992698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308707+0000","last_change":"2024-04-02T13:31:12.141165+0000","last_active":"2024-04-02T13:31:13.308707+0000","last_peered":"2024-04-02T13:31:13.308707+0000","last_clean":"2024-04-02T13:31:13.308707+0000","last_became_active":"2024-04-02T13:31:12.139621+0000","last_became_peered":"2024-04-02T13:31:12.139621+0000","last_unstale":"2024-04-02T13:31:13.308707+0000","last_undegraded":"2024-04-02T13:31:13.308707+0000","last_fullsized":"2024-04-02T13:31:13.308707+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:22:13.109205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440595+0000","last_change":"2024-04-02T13:31:12.127366+0000","last_active":"2024-04-02T13:31:13.440595+0000","last_peered":"2024-04-02T13:31:13.440595+0000","last_clean":"2024-04-02T13:31:13.440595+0000","last_became_active":"2024-04-02T13:31:12.127065+0000","last_became_peered":"2024-04-02T13:31:12.127065+0000","last_unstale":"2024-04-02T13:31:13.440595+0000","last_undegraded":"2024-04-02T13:31:13.440595+0000","last_fullsized":"2024-04-02T13:31:13.440595+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:49:46.345894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138850+0000","last_change":"2024-04-02T13:31:12.136738+0000","last_active":"2024-04-02T13:31:13.138850+0000","last_peered":"2024-04-02T13:31:13.138850+0000","last_clean":"2024-04-02T13:31:13.138850+0000","last_became_active":"2024-04-02T13:31:12.136468+0000","last_became_peered":"2024-04-02T13:31:12.136468+0000","last_unstale":"2024-04-02T13:31:13.138850+0000","last_undegraded":"2024-04-02T13:31:13.138850+0000","last_fullsized":"2024-04-02T13:31:13.138850+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:11:05.916368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345161+0000","last_change":"2024-04-02T13:31:12.139477+0000","last_active":"2024-04-02T13:31:13.345161+0000","last_peered":"2024-04-02T13:31:13.345161+0000","last_clean":"2024-04-02T13:31:13.345161+0000","last_became_active":"2024-04-02T13:31:12.139148+0000","last_became_peered":"2024-04-02T13:31:12.139148+0000","last_unstale":"2024-04-02T13:31:13.345161+0000","last_undegraded":"2024-04-02T13:31:13.345161+0000","last_fullsized":"2024-04-02T13:31:13.345161+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:14:16.657501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308921+0000","last_change":"2024-04-02T13:31:12.153879+0000","last_active":"2024-04-02T13:31:13.308921+0000","last_peered":"2024-04-02T13:31:13.308921+0000","last_clean":"2024-04-02T13:31:13.308921+0000","last_became_active":"2024-04-02T13:31:12.153585+0000","last_became_peered":"2024-04-02T13:31:12.153585+0000","last_unstale":"2024-04-02T13:31:13.308921+0000","last_undegraded":"2024-04-02T13:31:13.308921+0000","last_fullsized":"2024-04-02T13:31:13.308921+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:42:04.836436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441446+0000","last_change":"2024-04-02T13:31:12.138789+0000","last_active":"2024-04-02T13:31:13.441446+0000","last_peered":"2024-04-02T13:31:13.441446+0000","last_clean":"2024-04-02T13:31:13.441446+0000","last_became_active":"2024-04-02T13:31:12.138442+0000","last_became_peered":"2024-04-02T13:31:12.138442+0000","last_unstale":"2024-04-02T13:31:13.441446+0000","last_undegraded":"2024-04-02T13:31:13.441446+0000","last_fullsized":"2024-04-02T13:31:13.441446+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:24:37.992698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440168+0000","last_change":"2024-04-02T13:31:12.143673+0000","last_active":"2024-04-02T13:31:13.440168+0000","last_peered":"2024-04-02T13:31:13.440168+0000","last_clean":"2024-04-02T13:31:13.440168+0000","last_became_active":"2024-04-02T13:31:12.143379+0000","last_became_peered":"2024-04-02T13:31:12.143379+0000","last_unstale":"2024-04-02T13:31:13.440168+0000","last_undegraded":"2024-04-02T13:31:13.440168+0000","last_fullsized":"2024-04-02T13:31:13.440168+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:03:44.414064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140272+0000","last_change":"2024-04-02T13:31:12.134035+0000","last_active":"2024-04-02T13:31:13.140272+0000","last_peered":"2024-04-02T13:31:13.140272+0000","last_clean":"2024-04-02T13:31:13.140272+0000","last_became_active":"2024-04-02T13:31:12.133754+0000","last_became_peered":"2024-04-02T13:31:12.133754+0000","last_unstale":"2024-04-02T13:31:13.140272+0000","last_undegraded":"2024-04-02T13:31:13.140272+0000","last_fullsized":"2024-04-02T13:31:13.140272+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:18:25.723461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440735+0000","last_change":"2024-04-02T13:31:12.144040+0000","last_active":"2024-04-02T13:31:13.440735+0000","last_peered":"2024-04-02T13:31:13.440735+0000","last_clean":"2024-04-02T13:31:13.440735+0000","last_became_active":"2024-04-02T13:31:12.143750+0000","last_became_peered":"2024-04-02T13:31:12.143750+0000","last_unstale":"2024-04-02T13:31:13.440735+0000","last_undegraded":"2024-04-02T13:31:13.440735+0000","last_fullsized":"2024-04-02T13:31:13.440735+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:42:09.725977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140177+0000","last_change":"2024-04-02T13:31:12.147651+0000","last_active":"2024-04-02T13:31:13.140177+0000","last_peered":"2024-04-02T13:31:13.140177+0000","last_clean":"2024-04-02T13:31:13.140177+0000","last_became_active":"2024-04-02T13:31:12.147194+0000","last_became_peered":"2024-04-02T13:31:12.147194+0000","last_unstale":"2024-04-02T13:31:13.140177+0000","last_undegraded":"2024-04-02T13:31:13.140177+0000","last_fullsized":"2024-04-02T13:31:13.140177+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:02.941259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138668+0000","last_change":"2024-04-02T13:31:12.131065+0000","last_active":"2024-04-02T13:31:13.138668+0000","last_peered":"2024-04-02T13:31:13.138668+0000","last_clean":"2024-04-02T13:31:13.138668+0000","last_became_active":"2024-04-02T13:31:12.130782+0000","last_became_peered":"2024-04-02T13:31:12.130782+0000","last_unstale":"2024-04-02T13:31:13.138668+0000","last_undegraded":"2024-04-02T13:31:13.138668+0000","last_fullsized":"2024-04-02T13:31:13.138668+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:11:29.837470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138931+0000","last_change":"2024-04-02T13:31:12.131975+0000","last_active":"2024-04-02T13:31:13.138931+0000","last_peered":"2024-04-02T13:31:13.138931+0000","last_clean":"2024-04-02T13:31:13.138931+0000","last_became_active":"2024-04-02T13:31:12.131677+0000","last_became_peered":"2024-04-02T13:31:12.131677+0000","last_unstale":"2024-04-02T13:31:13.138931+0000","last_undegraded":"2024-04-02T13:31:13.138931+0000","last_fullsized":"2024-04-02T13:31:13.138931+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:12.227157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440243+0000","last_change":"2024-04-02T13:31:12.133781+0000","last_active":"2024-04-02T13:31:13.440243+0000","last_peered":"2024-04-02T13:31:13.440243+0000","last_clean":"2024-04-02T13:31:13.440243+0000","last_became_active":"2024-04-02T13:31:12.133409+0000","last_became_peered":"2024-04-02T13:31:12.133409+0000","last_unstale":"2024-04-02T13:31:13.440243+0000","last_undegraded":"2024-04-02T13:31:13.440243+0000","last_fullsized":"2024-04-02T13:31:13.440243+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:12.171120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441512+0000","last_change":"2024-04-02T13:31:12.137877+0000","last_active":"2024-04-02T13:31:13.441512+0000","last_peered":"2024-04-02T13:31:13.441512+0000","last_clean":"2024-04-02T13:31:13.441512+0000","last_became_active":"2024-04-02T13:31:12.137515+0000","last_became_peered":"2024-04-02T13:31:12.137515+0000","last_unstale":"2024-04-02T13:31:13.441512+0000","last_undegraded":"2024-04-02T13:31:13.441512+0000","last_fullsized":"2024-04-02T13:31:13.441512+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:45:45.336259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139345+0000","last_change":"2024-04-02T13:31:12.141128+0000","last_active":"2024-04-02T13:31:13.139345+0000","last_peered":"2024-04-02T13:31:13.139345+0000","last_clean":"2024-04-02T13:31:13.139345+0000","last_became_active":"2024-04-02T13:31:12.140867+0000","last_became_peered":"2024-04-02T13:31:12.140867+0000","last_unstale":"2024-04-02T13:31:13.139345+0000","last_undegraded":"2024-04-02T13:31:13.139345+0000","last_fullsized":"2024-04-02T13:31:13.139345+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:27:51.844903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440283+0000","last_change":"2024-04-02T13:31:12.138279+0000","last_active":"2024-04-02T13:31:13.440283+0000","last_peered":"2024-04-02T13:31:13.440283+0000","last_clean":"2024-04-02T13:31:13.440283+0000","last_became_active":"2024-04-02T13:31:12.137977+0000","last_became_peered":"2024-04-02T13:31:12.137977+0000","last_unstale":"2024-04-02T13:31:13.440283+0000","last_undegraded":"2024-04-02T13:31:13.440283+0000","last_fullsized":"2024-04-02T13:31:13.440283+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:40:09.871586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308726+0000","last_change":"2024-04-02T13:31:12.138813+0000","last_active":"2024-04-02T13:31:13.308726+0000","last_peered":"2024-04-02T13:31:13.308726+0000","last_clean":"2024-04-02T13:31:13.308726+0000","last_became_active":"2024-04-02T13:31:12.138446+0000","last_became_peered":"2024-04-02T13:31:12.138446+0000","last_unstale":"2024-04-02T13:31:13.308726+0000","last_undegraded":"2024-04-02T13:31:13.308726+0000","last_fullsized":"2024-04-02T13:31:13.308726+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:31:06.221856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138634+0000","last_change":"2024-04-02T13:31:12.134311+0000","last_active":"2024-04-02T13:31:13.138634+0000","last_peered":"2024-04-02T13:31:13.138634+0000","last_clean":"2024-04-02T13:31:13.138634+0000","last_became_active":"2024-04-02T13:31:12.134048+0000","last_became_peered":"2024-04-02T13:31:12.134048+0000","last_unstale":"2024-04-02T13:31:13.138634+0000","last_undegraded":"2024-04-02T13:31:13.138634+0000","last_fullsized":"2024-04-02T13:31:13.138634+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:06:45.310008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138216+0000","last_change":"2024-04-02T13:31:12.130296+0000","last_active":"2024-04-02T13:31:13.138216+0000","last_peered":"2024-04-02T13:31:13.138216+0000","last_clean":"2024-04-02T13:31:13.138216+0000","last_became_active":"2024-04-02T13:31:12.130024+0000","last_became_peered":"2024-04-02T13:31:12.130024+0000","last_unstale":"2024-04-02T13:31:13.138216+0000","last_undegraded":"2024-04-02T13:31:13.138216+0000","last_fullsized":"2024-04-02T13:31:13.138216+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:24:37.992698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345053+0000","last_change":"2024-04-02T13:31:12.145199+0000","last_active":"2024-04-02T13:31:13.345053+0000","last_peered":"2024-04-02T13:31:13.345053+0000","last_clean":"2024-04-02T13:31:13.345053+0000","last_became_active":"2024-04-02T13:31:12.144863+0000","last_became_peered":"2024-04-02T13:31:12.144863+0000","last_unstale":"2024-04-02T13:31:13.345053+0000","last_undegraded":"2024-04-02T13:31:13.345053+0000","last_fullsized":"2024-04-02T13:31:13.345053+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:45.106103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"120'4","reported_seq":18,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441471+0000","last_change":"2024-04-02T13:31:11.135714+0000","last_active":"2024-04-02T13:31:13.441471+0000","last_peered":"2024-04-02T13:31:13.441471+0000","last_clean":"2024-04-02T13:31:13.441471+0000","last_became_active":"2024-04-02T13:31:11.135452+0000","last_became_peered":"2024-04-02T13:31:11.135452+0000","last_unstale":"2024-04-02T13:31:13.441471+0000","last_undegraded":"2024-04-02T13:31:13.441471+0000","last_fullsized":"2024-04-02T13:31:13.441471+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:57.514520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139215+0000","last_change":"2024-04-02T13:31:12.148237+0000","last_active":"2024-04-02T13:31:13.139215+0000","last_peered":"2024-04-02T13:31:13.139215+0000","last_clean":"2024-04-02T13:31:13.139215+0000","last_became_active":"2024-04-02T13:31:12.147970+0000","last_became_peered":"2024-04-02T13:31:12.147970+0000","last_unstale":"2024-04-02T13:31:13.139215+0000","last_undegraded":"2024-04-02T13:31:13.139215+0000","last_fullsized":"2024-04-02T13:31:13.139215+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:22:36.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345100+0000","last_change":"2024-04-02T13:31:11.129147+0000","last_active":"2024-04-02T13:31:13.345100+0000","last_peered":"2024-04-02T13:31:13.345100+0000","last_clean":"2024-04-02T13:31:13.345100+0000","last_became_active":"2024-04-02T13:31:11.128778+0000","last_became_peered":"2024-04-02T13:31:11.128778+0000","last_unstale":"2024-04-02T13:31:13.345100+0000","last_undegraded":"2024-04-02T13:31:13.345100+0000","last_fullsized":"2024-04-02T13:31:13.345100+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:13:08.961404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138567+0000","last_change":"2024-04-02T13:31:12.142166+0000","last_active":"2024-04-02T13:31:13.138567+0000","last_peered":"2024-04-02T13:31:13.138567+0000","last_clean":"2024-04-02T13:31:13.138567+0000","last_became_active":"2024-04-02T13:31:12.141872+0000","last_became_peered":"2024-04-02T13:31:12.141872+0000","last_unstale":"2024-04-02T13:31:13.138567+0000","last_undegraded":"2024-04-02T13:31:13.138567+0000","last_fullsized":"2024-04-02T13:31:13.138567+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:14:16.657501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140633+0000","last_change":"2024-04-02T13:31:11.132278+0000","last_active":"2024-04-02T13:31:13.140633+0000","last_peered":"2024-04-02T13:31:13.140633+0000","last_clean":"2024-04-02T13:31:13.140633+0000","last_became_active":"2024-04-02T13:31:11.131577+0000","last_became_peered":"2024-04-02T13:31:11.131577+0000","last_unstale":"2024-04-02T13:31:13.140633+0000","last_undegraded":"2024-04-02T13:31:13.140633+0000","last_fullsized":"2024-04-02T13:31:13.140633+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:24:36.987532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140778+0000","last_change":"2024-04-02T13:31:12.134345+0000","last_active":"2024-04-02T13:31:13.140778+0000","last_peered":"2024-04-02T13:31:13.140778+0000","last_clean":"2024-04-02T13:31:13.140778+0000","last_became_active":"2024-04-02T13:31:12.134063+0000","last_became_peered":"2024-04-02T13:31:12.134063+0000","last_unstale":"2024-04-02T13:31:13.140778+0000","last_undegraded":"2024-04-02T13:31:13.140778+0000","last_fullsized":"2024-04-02T13:31:13.140778+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:35:25.756038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345112+0000","last_change":"2024-04-02T13:31:11.131085+0000","last_active":"2024-04-02T13:31:13.345112+0000","last_peered":"2024-04-02T13:31:13.345112+0000","last_clean":"2024-04-02T13:31:13.345112+0000","last_became_active":"2024-04-02T13:31:11.130800+0000","last_became_peered":"2024-04-02T13:31:11.130800+0000","last_unstale":"2024-04-02T13:31:13.345112+0000","last_undegraded":"2024-04-02T13:31:13.345112+0000","last_fullsized":"2024-04-02T13:31:13.345112+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:46:50.026745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.344924+0000","last_change":"2024-04-02T13:31:12.140319+0000","last_active":"2024-04-02T13:31:13.344924+0000","last_peered":"2024-04-02T13:31:13.344924+0000","last_clean":"2024-04-02T13:31:13.344924+0000","last_became_active":"2024-04-02T13:31:12.140062+0000","last_became_peered":"2024-04-02T13:31:12.140062+0000","last_unstale":"2024-04-02T13:31:13.344924+0000","last_undegraded":"2024-04-02T13:31:13.344924+0000","last_fullsized":"2024-04-02T13:31:13.344924+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:49:46.345894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440973+0000","last_change":"2024-04-02T13:31:11.134176+0000","last_active":"2024-04-02T13:31:13.440973+0000","last_peered":"2024-04-02T13:31:13.440973+0000","last_clean":"2024-04-02T13:31:13.440973+0000","last_became_active":"2024-04-02T13:31:11.133891+0000","last_became_peered":"2024-04-02T13:31:11.133891+0000","last_unstale":"2024-04-02T13:31:13.440973+0000","last_undegraded":"2024-04-02T13:31:13.440973+0000","last_fullsized":"2024-04-02T13:31:13.440973+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:44:10.359503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440621+0000","last_change":"2024-04-02T13:31:12.134737+0000","last_active":"2024-04-02T13:31:13.440621+0000","last_peered":"2024-04-02T13:31:13.440621+0000","last_clean":"2024-04-02T13:31:13.440621+0000","last_became_active":"2024-04-02T13:31:12.134462+0000","last_became_peered":"2024-04-02T13:31:12.134462+0000","last_unstale":"2024-04-02T13:31:13.440621+0000","last_undegraded":"2024-04-02T13:31:13.440621+0000","last_fullsized":"2024-04-02T13:31:13.440621+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:34:25.158893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138987+0000","last_change":"2024-04-02T13:31:11.132878+0000","last_active":"2024-04-02T13:31:13.138987+0000","last_peered":"2024-04-02T13:31:13.138987+0000","last_clean":"2024-04-02T13:31:13.138987+0000","last_became_active":"2024-04-02T13:31:11.132602+0000","last_became_peered":"2024-04-02T13:31:11.132602+0000","last_unstale":"2024-04-02T13:31:13.138987+0000","last_undegraded":"2024-04-02T13:31:13.138987+0000","last_fullsized":"2024-04-02T13:31:13.138987+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:35:24.750872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308885+0000","last_change":"2024-04-02T13:31:12.150723+0000","last_active":"2024-04-02T13:31:13.308885+0000","last_peered":"2024-04-02T13:31:13.308885+0000","last_clean":"2024-04-02T13:31:13.308885+0000","last_became_active":"2024-04-02T13:31:12.150444+0000","last_became_peered":"2024-04-02T13:31:12.150444+0000","last_unstale":"2024-04-02T13:31:13.308885+0000","last_undegraded":"2024-04-02T13:31:13.308885+0000","last_fullsized":"2024-04-02T13:31:13.308885+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:25:27.156701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138256+0000","last_change":"2024-04-02T13:31:11.133786+0000","last_active":"2024-04-02T13:31:13.138256+0000","last_peered":"2024-04-02T13:31:13.138256+0000","last_clean":"2024-04-02T13:31:13.138256+0000","last_became_active":"2024-04-02T13:31:11.132953+0000","last_became_peered":"2024-04-02T13:31:11.132953+0000","last_unstale":"2024-04-02T13:31:13.138256+0000","last_undegraded":"2024-04-02T13:31:13.138256+0000","last_fullsized":"2024-04-02T13:31:13.138256+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:43:40.243740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345047+0000","last_change":"2024-04-02T13:31:12.150939+0000","last_active":"2024-04-02T13:31:13.345047+0000","last_peered":"2024-04-02T13:31:13.345047+0000","last_clean":"2024-04-02T13:31:13.345047+0000","last_became_active":"2024-04-02T13:31:12.150683+0000","last_became_peered":"2024-04-02T13:31:12.150683+0000","last_unstale":"2024-04-02T13:31:13.345047+0000","last_undegraded":"2024-04-02T13:31:13.345047+0000","last_fullsized":"2024-04-02T13:31:13.345047+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:42:09.725977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139068+0000","last_change":"2024-04-02T13:31:11.133664+0000","last_active":"2024-04-02T13:31:13.139068+0000","last_peered":"2024-04-02T13:31:13.139068+0000","last_clean":"2024-04-02T13:31:13.139068+0000","last_became_active":"2024-04-02T13:31:11.133402+0000","last_became_peered":"2024-04-02T13:31:11.133402+0000","last_unstale":"2024-04-02T13:31:13.139068+0000","last_undegraded":"2024-04-02T13:31:13.139068+0000","last_fullsized":"2024-04-02T13:31:13.139068+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:43:43.950920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441255+0000","last_change":"2024-04-02T13:31:12.133962+0000","last_active":"2024-04-02T13:31:13.441255+0000","last_peered":"2024-04-02T13:31:13.441255+0000","last_clean":"2024-04-02T13:31:13.441255+0000","last_became_active":"2024-04-02T13:31:12.133693+0000","last_became_peered":"2024-04-02T13:31:12.133693+0000","last_unstale":"2024-04-02T13:31:13.441255+0000","last_undegraded":"2024-04-02T13:31:13.441255+0000","last_fullsized":"2024-04-02T13:31:13.441255+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:11:29.837470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441552+0000","last_change":"2024-04-02T13:31:11.139525+0000","last_active":"2024-04-02T13:31:13.441552+0000","last_peered":"2024-04-02T13:31:13.441552+0000","last_clean":"2024-04-02T13:31:13.441552+0000","last_became_active":"2024-04-02T13:31:11.139238+0000","last_became_peered":"2024-04-02T13:31:11.139238+0000","last_unstale":"2024-04-02T13:31:13.441552+0000","last_undegraded":"2024-04-02T13:31:13.441552+0000","last_fullsized":"2024-04-02T13:31:13.441552+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:32:53.017171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138771+0000","last_change":"2024-04-02T13:31:12.143793+0000","last_active":"2024-04-02T13:31:13.138771+0000","last_peered":"2024-04-02T13:31:13.138771+0000","last_clean":"2024-04-02T13:31:13.138771+0000","last_became_active":"2024-04-02T13:31:12.143517+0000","last_became_peered":"2024-04-02T13:31:12.143517+0000","last_unstale":"2024-04-02T13:31:13.138771+0000","last_undegraded":"2024-04-02T13:31:13.138771+0000","last_fullsized":"2024-04-02T13:31:13.138771+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:57:24.583576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138547+0000","last_change":"2024-04-02T13:31:11.128784+0000","last_active":"2024-04-02T13:31:13.138547+0000","last_peered":"2024-04-02T13:31:13.138547+0000","last_clean":"2024-04-02T13:31:13.138547+0000","last_became_active":"2024-04-02T13:31:11.128512+0000","last_became_peered":"2024-04-02T13:31:11.128512+0000","last_unstale":"2024-04-02T13:31:13.138547+0000","last_undegraded":"2024-04-02T13:31:13.138547+0000","last_fullsized":"2024-04-02T13:31:13.138547+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:22.352670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.309004+0000","last_change":"2024-04-02T13:31:12.141447+0000","last_active":"2024-04-02T13:31:13.309004+0000","last_peered":"2024-04-02T13:31:13.309004+0000","last_clean":"2024-04-02T13:31:13.309004+0000","last_became_active":"2024-04-02T13:31:12.139677+0000","last_became_peered":"2024-04-02T13:31:12.139677+0000","last_unstale":"2024-04-02T13:31:13.309004+0000","last_undegraded":"2024-04-02T13:31:13.309004+0000","last_fullsized":"2024-04-02T13:31:13.309004+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:05:45.882774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"120'3","reported_seq":17,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138689+0000","last_change":"2024-04-02T13:31:11.130564+0000","last_active":"2024-04-02T13:31:13.138689+0000","last_peered":"2024-04-02T13:31:13.138689+0000","last_clean":"2024-04-02T13:31:13.138689+0000","last_became_active":"2024-04-02T13:31:11.130305+0000","last_became_peered":"2024-04-02T13:31:11.130305+0000","last_unstale":"2024-04-02T13:31:13.138689+0000","last_undegraded":"2024-04-02T13:31:13.138689+0000","last_fullsized":"2024-04-02T13:31:13.138689+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:44:32.667698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138872+0000","last_change":"2024-04-02T13:31:12.136750+0000","last_active":"2024-04-02T13:31:13.138872+0000","last_peered":"2024-04-02T13:31:13.138872+0000","last_clean":"2024-04-02T13:31:13.138872+0000","last_became_active":"2024-04-02T13:31:12.136283+0000","last_became_peered":"2024-04-02T13:31:12.136283+0000","last_unstale":"2024-04-02T13:31:13.138872+0000","last_undegraded":"2024-04-02T13:31:13.138872+0000","last_fullsized":"2024-04-02T13:31:13.138872+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:32:29.196579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"120'6","reported_seq":20,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308980+0000","last_change":"2024-04-02T13:31:11.132655+0000","last_active":"2024-04-02T13:31:13.308980+0000","last_peered":"2024-04-02T13:31:13.308980+0000","last_clean":"2024-04-02T13:31:13.308980+0000","last_became_active":"2024-04-02T13:31:11.132292+0000","last_became_peered":"2024-04-02T13:31:11.132292+0000","last_unstale":"2024-04-02T13:31:13.308980+0000","last_undegraded":"2024-04-02T13:31:13.308980+0000","last_fullsized":"2024-04-02T13:31:13.308980+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:06:44.304841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441167+0000","last_change":"2024-04-02T13:31:12.143424+0000","last_active":"2024-04-02T13:31:13.441167+0000","last_peered":"2024-04-02T13:31:13.441167+0000","last_clean":"2024-04-02T13:31:13.441167+0000","last_became_active":"2024-04-02T13:31:12.143081+0000","last_became_peered":"2024-04-02T13:31:12.143081+0000","last_unstale":"2024-04-02T13:31:13.441167+0000","last_undegraded":"2024-04-02T13:31:13.441167+0000","last_fullsized":"2024-04-02T13:31:13.441167+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:45.106103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308759+0000","last_change":"2024-04-02T13:31:11.135615+0000","last_active":"2024-04-02T13:31:13.308759+0000","last_peered":"2024-04-02T13:31:13.308759+0000","last_clean":"2024-04-02T13:31:13.308759+0000","last_became_active":"2024-04-02T13:31:11.135131+0000","last_became_peered":"2024-04-02T13:31:11.135131+0000","last_unstale":"2024-04-02T13:31:13.308759+0000","last_undegraded":"2024-04-02T13:31:13.308759+0000","last_fullsized":"2024-04-02T13:31:13.308759+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:14:15.652335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345040+0000","last_change":"2024-04-02T13:31:12.141980+0000","last_active":"2024-04-02T13:31:13.345040+0000","last_peered":"2024-04-02T13:31:13.345040+0000","last_clean":"2024-04-02T13:31:13.345040+0000","last_became_active":"2024-04-02T13:31:12.141621+0000","last_became_peered":"2024-04-02T13:31:12.141621+0000","last_unstale":"2024-04-02T13:31:13.345040+0000","last_undegraded":"2024-04-02T13:31:13.345040+0000","last_fullsized":"2024-04-02T13:31:13.345040+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:34:25.158893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308972+0000","last_change":"2024-04-02T13:31:11.128713+0000","last_active":"2024-04-02T13:31:13.308972+0000","last_peered":"2024-04-02T13:31:13.308972+0000","last_clean":"2024-04-02T13:31:13.308972+0000","last_became_active":"2024-04-02T13:31:11.128388+0000","last_became_peered":"2024-04-02T13:31:11.128388+0000","last_unstale":"2024-04-02T13:31:13.308972+0000","last_undegraded":"2024-04-02T13:31:13.308972+0000","last_fullsized":"2024-04-02T13:31:13.308972+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:33:55.018213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440090+0000","last_change":"2024-04-02T13:31:12.130202+0000","last_active":"2024-04-02T13:31:13.440090+0000","last_peered":"2024-04-02T13:31:13.440090+0000","last_clean":"2024-04-02T13:31:13.440090+0000","last_became_active":"2024-04-02T13:31:12.129938+0000","last_became_peered":"2024-04-02T13:31:12.129938+0000","last_unstale":"2024-04-02T13:31:13.440090+0000","last_undegraded":"2024-04-02T13:31:13.440090+0000","last_fullsized":"2024-04-02T13:31:13.440090+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:20:30.993334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308857+0000","last_change":"2024-04-02T13:31:11.133017+0000","last_active":"2024-04-02T13:31:13.308857+0000","last_peered":"2024-04-02T13:31:13.308857+0000","last_clean":"2024-04-02T13:31:13.308857+0000","last_became_active":"2024-04-02T13:31:11.132360+0000","last_became_peered":"2024-04-02T13:31:11.132360+0000","last_unstale":"2024-04-02T13:31:13.308857+0000","last_undegraded":"2024-04-02T13:31:13.308857+0000","last_fullsized":"2024-04-02T13:31:13.308857+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:57:23.578410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138745+0000","last_change":"2024-04-02T13:31:12.142395+0000","last_active":"2024-04-02T13:31:13.138745+0000","last_peered":"2024-04-02T13:31:13.138745+0000","last_clean":"2024-04-02T13:31:13.138745+0000","last_became_active":"2024-04-02T13:31:12.142108+0000","last_became_peered":"2024-04-02T13:31:12.142108+0000","last_unstale":"2024-04-02T13:31:13.138745+0000","last_undegraded":"2024-04-02T13:31:13.138745+0000","last_fullsized":"2024-04-02T13:31:13.138745+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:33:56.023379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345265+0000","last_change":"2024-04-02T13:31:11.135311+0000","last_active":"2024-04-02T13:31:13.345265+0000","last_peered":"2024-04-02T13:31:13.345265+0000","last_clean":"2024-04-02T13:31:13.345265+0000","last_became_active":"2024-04-02T13:31:11.135038+0000","last_became_peered":"2024-04-02T13:31:11.135038+0000","last_unstale":"2024-04-02T13:31:13.345265+0000","last_undegraded":"2024-04-02T13:31:13.345265+0000","last_fullsized":"2024-04-02T13:31:13.345265+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:04:37.616135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441074+0000","last_change":"2024-04-02T13:31:12.143962+0000","last_active":"2024-04-02T13:31:13.441074+0000","last_peered":"2024-04-02T13:31:13.441074+0000","last_clean":"2024-04-02T13:31:13.441074+0000","last_became_active":"2024-04-02T13:31:12.143694+0000","last_became_peered":"2024-04-02T13:31:12.143694+0000","last_unstale":"2024-04-02T13:31:13.441074+0000","last_undegraded":"2024-04-02T13:31:13.441074+0000","last_fullsized":"2024-04-02T13:31:13.441074+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:24:37.992698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440250+0000","last_change":"2024-04-02T13:31:11.135836+0000","last_active":"2024-04-02T13:31:13.440250+0000","last_peered":"2024-04-02T13:31:13.440250+0000","last_clean":"2024-04-02T13:31:13.440250+0000","last_became_active":"2024-04-02T13:31:11.135535+0000","last_became_peered":"2024-04-02T13:31:11.135535+0000","last_unstale":"2024-04-02T13:31:13.440250+0000","last_undegraded":"2024-04-02T13:31:13.440250+0000","last_fullsized":"2024-04-02T13:31:13.440250+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:21:08.296959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139253+0000","last_change":"2024-04-02T13:31:12.142067+0000","last_active":"2024-04-02T13:31:13.139253+0000","last_peered":"2024-04-02T13:31:13.139253+0000","last_clean":"2024-04-02T13:31:13.139253+0000","last_became_active":"2024-04-02T13:31:12.141782+0000","last_became_peered":"2024-04-02T13:31:12.141782+0000","last_unstale":"2024-04-02T13:31:13.139253+0000","last_undegraded":"2024-04-02T13:31:13.139253+0000","last_fullsized":"2024-04-02T13:31:13.139253+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:39:40.128741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140646+0000","last_change":"2024-04-02T13:31:11.126878+0000","last_active":"2024-04-02T13:31:13.140646+0000","last_peered":"2024-04-02T13:31:13.140646+0000","last_clean":"2024-04-02T13:31:13.140646+0000","last_became_active":"2024-04-02T13:31:11.126520+0000","last_became_peered":"2024-04-02T13:31:11.126520+0000","last_unstale":"2024-04-02T13:31:13.140646+0000","last_undegraded":"2024-04-02T13:31:13.140646+0000","last_fullsized":"2024-04-02T13:31:13.140646+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:45:44.331093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138979+0000","last_change":"2024-04-02T13:31:12.134022+0000","last_active":"2024-04-02T13:31:13.138979+0000","last_peered":"2024-04-02T13:31:13.138979+0000","last_clean":"2024-04-02T13:31:13.138979+0000","last_became_active":"2024-04-02T13:31:12.133677+0000","last_became_peered":"2024-04-02T13:31:12.133677+0000","last_unstale":"2024-04-02T13:31:13.138979+0000","last_undegraded":"2024-04-02T13:31:13.138979+0000","last_fullsized":"2024-04-02T13:31:13.138979+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:38:42.298292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140496+0000","last_change":"2024-04-02T13:31:11.136359+0000","last_active":"2024-04-02T13:31:13.140496+0000","last_peered":"2024-04-02T13:31:13.140496+0000","last_clean":"2024-04-02T13:31:13.140496+0000","last_became_active":"2024-04-02T13:31:11.136154+0000","last_became_peered":"2024-04-02T13:31:11.136154+0000","last_unstale":"2024-04-02T13:31:13.140496+0000","last_undegraded":"2024-04-02T13:31:13.140496+0000","last_fullsized":"2024-04-02T13:31:13.140496+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:11:28.832303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138860+0000","last_change":"2024-04-02T13:31:12.136151+0000","last_active":"2024-04-02T13:31:13.138860+0000","last_peered":"2024-04-02T13:31:13.138860+0000","last_clean":"2024-04-02T13:31:13.138860+0000","last_became_active":"2024-04-02T13:31:12.135868+0000","last_became_peered":"2024-04-02T13:31:12.135868+0000","last_unstale":"2024-04-02T13:31:13.138860+0000","last_undegraded":"2024-04-02T13:31:13.138860+0000","last_fullsized":"2024-04-02T13:31:13.138860+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:07.907914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139008+0000","last_change":"2024-04-02T13:31:11.136321+0000","last_active":"2024-04-02T13:31:13.139008+0000","last_peered":"2024-04-02T13:31:13.139008+0000","last_clean":"2024-04-02T13:31:13.139008+0000","last_became_active":"2024-04-02T13:31:11.135962+0000","last_became_peered":"2024-04-02T13:31:11.135962+0000","last_unstale":"2024-04-02T13:31:13.139008+0000","last_undegraded":"2024-04-02T13:31:13.139008+0000","last_fullsized":"2024-04-02T13:31:13.139008+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:33:23.184101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308610+0000","last_change":"2024-04-02T13:31:12.142057+0000","last_active":"2024-04-02T13:31:13.308610+0000","last_peered":"2024-04-02T13:31:13.308610+0000","last_clean":"2024-04-02T13:31:13.308610+0000","last_became_active":"2024-04-02T13:31:12.141532+0000","last_became_peered":"2024-04-02T13:31:12.141532+0000","last_unstale":"2024-04-02T13:31:13.308610+0000","last_undegraded":"2024-04-02T13:31:13.308610+0000","last_fullsized":"2024-04-02T13:31:13.308610+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:37:08.666978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140068+0000","last_change":"2024-04-02T13:31:11.133500+0000","last_active":"2024-04-02T13:31:13.140068+0000","last_peered":"2024-04-02T13:31:13.140068+0000","last_clean":"2024-04-02T13:31:13.140068+0000","last_became_active":"2024-04-02T13:31:11.133225+0000","last_became_peered":"2024-04-02T13:31:11.133225+0000","last_unstale":"2024-04-02T13:31:13.140068+0000","last_undegraded":"2024-04-02T13:31:13.140068+0000","last_fullsized":"2024-04-02T13:31:13.140068+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:58:44.100936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308967+0000","last_change":"2024-04-02T13:31:12.136866+0000","last_active":"2024-04-02T13:31:13.308967+0000","last_peered":"2024-04-02T13:31:13.308967+0000","last_clean":"2024-04-02T13:31:13.308967+0000","last_became_active":"2024-04-02T13:31:12.136496+0000","last_became_peered":"2024-04-02T13:31:12.136496+0000","last_unstale":"2024-04-02T13:31:13.308967+0000","last_undegraded":"2024-04-02T13:31:13.308967+0000","last_fullsized":"2024-04-02T13:31:13.308967+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:48:48.771050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308840+0000","last_change":"2024-04-02T13:31:11.134053+0000","last_active":"2024-04-02T13:31:13.308840+0000","last_peered":"2024-04-02T13:31:13.308840+0000","last_clean":"2024-04-02T13:31:13.308840+0000","last_became_active":"2024-04-02T13:31:11.133716+0000","last_became_peered":"2024-04-02T13:31:11.133716+0000","last_unstale":"2024-04-02T13:31:13.308840+0000","last_undegraded":"2024-04-02T13:31:13.308840+0000","last_fullsized":"2024-04-02T13:31:13.308840+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:32:28.191412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138339+0000","last_change":"2024-04-02T13:31:12.136384+0000","last_active":"2024-04-02T13:31:13.138339+0000","last_peered":"2024-04-02T13:31:13.138339+0000","last_clean":"2024-04-02T13:31:13.138339+0000","last_became_active":"2024-04-02T13:31:12.136132+0000","last_became_peered":"2024-04-02T13:31:12.136132+0000","last_unstale":"2024-04-02T13:31:13.138339+0000","last_undegraded":"2024-04-02T13:31:13.138339+0000","last_fullsized":"2024-04-02T13:31:13.138339+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:45:45.336259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139865+0000","last_change":"2024-04-02T13:31:11.131870+0000","last_active":"2024-04-02T13:31:13.139865+0000","last_peered":"2024-04-02T13:31:13.139865+0000","last_clean":"2024-04-02T13:31:13.139865+0000","last_became_active":"2024-04-02T13:31:11.131567+0000","last_became_peered":"2024-04-02T13:31:11.131567+0000","last_unstale":"2024-04-02T13:31:13.139865+0000","last_undegraded":"2024-04-02T13:31:13.139865+0000","last_fullsized":"2024-04-02T13:31:13.139865+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:42:08.720811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139403+0000","last_change":"2024-04-02T13:31:12.145022+0000","last_active":"2024-04-02T13:31:13.139403+0000","last_peered":"2024-04-02T13:31:13.139403+0000","last_clean":"2024-04-02T13:31:13.139403+0000","last_became_active":"2024-04-02T13:31:12.144768+0000","last_became_peered":"2024-04-02T13:31:12.144768+0000","last_unstale":"2024-04-02T13:31:13.139403+0000","last_undegraded":"2024-04-02T13:31:13.139403+0000","last_fullsized":"2024-04-02T13:31:13.139403+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:34.217067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139224+0000","last_change":"2024-04-02T13:31:11.138081+0000","last_active":"2024-04-02T13:31:13.139224+0000","last_peered":"2024-04-02T13:31:13.139224+0000","last_clean":"2024-04-02T13:31:13.139224+0000","last_became_active":"2024-04-02T13:31:11.137830+0000","last_became_peered":"2024-04-02T13:31:11.137830+0000","last_unstale":"2024-04-02T13:31:13.139224+0000","last_undegraded":"2024-04-02T13:31:13.139224+0000","last_fullsized":"2024-04-02T13:31:13.139224+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:31.359210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345094+0000","last_change":"2024-04-02T13:31:12.140943+0000","last_active":"2024-04-02T13:31:13.345094+0000","last_peered":"2024-04-02T13:31:13.345094+0000","last_clean":"2024-04-02T13:31:13.345094+0000","last_became_active":"2024-04-02T13:31:12.140505+0000","last_became_peered":"2024-04-02T13:31:12.140505+0000","last_unstale":"2024-04-02T13:31:13.345094+0000","last_undegraded":"2024-04-02T13:31:13.345094+0000","last_fullsized":"2024-04-02T13:31:13.345094+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:11:29.837470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441507+0000","last_change":"2024-04-02T13:31:11.140019+0000","last_active":"2024-04-02T13:31:13.441507+0000","last_peered":"2024-04-02T13:31:13.441507+0000","last_clean":"2024-04-02T13:31:13.441507+0000","last_became_active":"2024-04-02T13:31:11.139233+0000","last_became_peered":"2024-04-02T13:31:11.139233+0000","last_unstale":"2024-04-02T13:31:13.441507+0000","last_undegraded":"2024-04-02T13:31:13.441507+0000","last_fullsized":"2024-04-02T13:31:13.441507+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:38.434488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140698+0000","last_change":"2024-04-02T13:31:12.131347+0000","last_active":"2024-04-02T13:31:13.140698+0000","last_peered":"2024-04-02T13:31:13.140698+0000","last_clean":"2024-04-02T13:31:13.140698+0000","last_became_active":"2024-04-02T13:31:12.131054+0000","last_became_peered":"2024-04-02T13:31:12.131054+0000","last_unstale":"2024-04-02T13:31:13.140698+0000","last_undegraded":"2024-04-02T13:31:13.140698+0000","last_fullsized":"2024-04-02T13:31:13.140698+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:30:55.860990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441565+0000","last_change":"2024-04-02T13:31:11.139638+0000","last_active":"2024-04-02T13:31:13.441565+0000","last_peered":"2024-04-02T13:31:13.441565+0000","last_clean":"2024-04-02T13:31:13.441565+0000","last_became_active":"2024-04-02T13:31:11.139319+0000","last_became_peered":"2024-04-02T13:31:11.139319+0000","last_unstale":"2024-04-02T13:31:13.441565+0000","last_undegraded":"2024-04-02T13:31:13.441565+0000","last_fullsized":"2024-04-02T13:31:13.441565+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:00:25.458770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440331+0000","last_change":"2024-04-02T13:31:12.127598+0000","last_active":"2024-04-02T13:31:13.440331+0000","last_peered":"2024-04-02T13:31:13.440331+0000","last_clean":"2024-04-02T13:31:13.440331+0000","last_became_active":"2024-04-02T13:31:12.127333+0000","last_became_peered":"2024-04-02T13:31:12.127333+0000","last_unstale":"2024-04-02T13:31:13.440331+0000","last_undegraded":"2024-04-02T13:31:13.440331+0000","last_fullsized":"2024-04-02T13:31:13.440331+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:08:15.789954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139487+0000","last_change":"2024-04-02T13:31:11.134862+0000","last_active":"2024-04-02T13:31:13.139487+0000","last_peered":"2024-04-02T13:31:13.139487+0000","last_clean":"2024-04-02T13:31:13.139487+0000","last_became_active":"2024-04-02T13:31:11.134611+0000","last_became_peered":"2024-04-02T13:31:11.134611+0000","last_unstale":"2024-04-02T13:31:13.139487+0000","last_undegraded":"2024-04-02T13:31:13.139487+0000","last_fullsized":"2024-04-02T13:31:13.139487+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:38.536661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139027+0000","last_change":"2024-04-02T13:31:12.135944+0000","last_active":"2024-04-02T13:31:13.139027+0000","last_peered":"2024-04-02T13:31:13.139027+0000","last_clean":"2024-04-02T13:31:13.139027+0000","last_became_active":"2024-04-02T13:31:12.135628+0000","last_became_peered":"2024-04-02T13:31:12.135628+0000","last_unstale":"2024-04-02T13:31:13.139027+0000","last_undegraded":"2024-04-02T13:31:13.139027+0000","last_fullsized":"2024-04-02T13:31:13.139027+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:33:09.023674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.441026+0000","last_change":"2024-04-02T13:31:11.139793+0000","last_active":"2024-04-02T13:31:13.441026+0000","last_peered":"2024-04-02T13:31:13.441026+0000","last_clean":"2024-04-02T13:31:13.441026+0000","last_became_active":"2024-04-02T13:31:11.139508+0000","last_became_peered":"2024-04-02T13:31:11.139508+0000","last_unstale":"2024-04-02T13:31:13.441026+0000","last_undegraded":"2024-04-02T13:31:13.441026+0000","last_fullsized":"2024-04-02T13:31:13.441026+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:43:40.243740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308629+0000","last_change":"2024-04-02T13:31:12.142629+0000","last_active":"2024-04-02T13:31:13.308629+0000","last_peered":"2024-04-02T13:31:13.308629+0000","last_clean":"2024-04-02T13:31:13.308629+0000","last_became_active":"2024-04-02T13:31:12.142367+0000","last_became_peered":"2024-04-02T13:31:12.142367+0000","last_unstale":"2024-04-02T13:31:13.308629+0000","last_undegraded":"2024-04-02T13:31:13.308629+0000","last_fullsized":"2024-04-02T13:31:13.308629+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:39.541828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.440919+0000","last_change":"2024-04-02T13:31:11.138347+0000","last_active":"2024-04-02T13:31:13.440919+0000","last_peered":"2024-04-02T13:31:13.440919+0000","last_clean":"2024-04-02T13:31:13.440919+0000","last_became_active":"2024-04-02T13:31:11.138045+0000","last_became_peered":"2024-04-02T13:31:11.138045+0000","last_unstale":"2024-04-02T13:31:13.440919+0000","last_undegraded":"2024-04-02T13:31:13.440919+0000","last_fullsized":"2024-04-02T13:31:13.440919+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:04:37.616135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140120+0000","last_change":"2024-04-02T13:31:12.135698+0000","last_active":"2024-04-02T13:31:13.140120+0000","last_peered":"2024-04-02T13:31:13.140120+0000","last_clean":"2024-04-02T13:31:13.140120+0000","last_became_active":"2024-04-02T13:31:12.135427+0000","last_became_peered":"2024-04-02T13:31:12.135427+0000","last_unstale":"2024-04-02T13:31:13.140120+0000","last_undegraded":"2024-04-02T13:31:13.140120+0000","last_fullsized":"2024-04-02T13:31:13.140120+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:05.199702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.138872+0000","last_change":"2024-04-02T13:31:11.134119+0000","last_active":"2024-04-02T13:31:13.138872+0000","last_peered":"2024-04-02T13:31:13.138872+0000","last_clean":"2024-04-02T13:31:13.138872+0000","last_became_active":"2024-04-02T13:31:11.133880+0000","last_became_peered":"2024-04-02T13:31:11.133880+0000","last_unstale":"2024-04-02T13:31:13.138872+0000","last_undegraded":"2024-04-02T13:31:13.138872+0000","last_fullsized":"2024-04-02T13:31:13.138872+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:37:07.661812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.345208+0000","last_change":"2024-04-02T13:31:12.141647+0000","last_active":"2024-04-02T13:31:13.345208+0000","last_peered":"2024-04-02T13:31:13.345208+0000","last_clean":"2024-04-02T13:31:13.345208+0000","last_became_active":"2024-04-02T13:31:12.141380+0000","last_became_peered":"2024-04-02T13:31:12.141380+0000","last_unstale":"2024-04-02T13:31:13.345208+0000","last_undegraded":"2024-04-02T13:31:13.345208+0000","last_fullsized":"2024-04-02T13:31:13.345208+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:06:45.310008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139872+0000","last_change":"2024-04-02T13:31:11.135129+0000","last_active":"2024-04-02T13:31:13.139872+0000","last_peered":"2024-04-02T13:31:13.139872+0000","last_clean":"2024-04-02T13:31:13.139872+0000","last_became_active":"2024-04-02T13:31:11.134810+0000","last_became_peered":"2024-04-02T13:31:11.134810+0000","last_unstale":"2024-04-02T13:31:13.139872+0000","last_undegraded":"2024-04-02T13:31:13.139872+0000","last_fullsized":"2024-04-02T13:31:13.139872+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:14:15.652335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.140487+0000","last_change":"2024-04-02T13:31:12.136674+0000","last_active":"2024-04-02T13:31:13.140487+0000","last_peered":"2024-04-02T13:31:13.140487+0000","last_clean":"2024-04-02T13:31:13.140487+0000","last_became_active":"2024-04-02T13:31:12.136399+0000","last_became_peered":"2024-04-02T13:31:12.136399+0000","last_unstale":"2024-04-02T13:31:13.140487+0000","last_undegraded":"2024-04-02T13:31:13.140487+0000","last_fullsized":"2024-04-02T13:31:13.140487+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:11.111377+0000","last_clean_scrub_stamp":"2024-04-02T13:31:11.111377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:22:03.194530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"120'8","reported_seq":22,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.308482+0000","last_change":"2024-04-02T13:31:11.135308+0000","last_active":"2024-04-02T13:31:13.308482+0000","last_peered":"2024-04-02T13:31:13.308482+0000","last_clean":"2024-04-02T13:31:13.308482+0000","last_became_active":"2024-04-02T13:31:11.135021+0000","last_became_peered":"2024-04-02T13:31:11.135021+0000","last_unstale":"2024-04-02T13:31:13.308482+0000","last_undegraded":"2024-04-02T13:31:13.308482+0000","last_fullsized":"2024-04-02T13:31:13.308482+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:31:10.106211+0000","last_clean_scrub_stamp":"2024-04-02T13:31:10.106211+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:11.221991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":365,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T13:31:13.139987+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:31:13.139987+0000","last_peered":"2024-04-02T13:31:13.139987+0000","last_clean":"2024-04-02T13:31:13.139987+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:31:13.139987+0000","last_undegraded":"2024-04-02T13:31:13.139987+0000","last_fullsized":"2024-04-02T13:31:13.139987+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574983,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33252,"kb_used_data":2068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338588,"statfs":{"total":96636764160,"available":96602714112,"internally_reserved":0,"allocated":2117632,"data_stored":1295657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574982,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1499136,"data_stored":698403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574983,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1515520,"data_stored":704673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574983,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32656,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":1507328,"data_stored":699433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574983,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1523712,"data_stored":706159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574982,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1507328,"data_stored":704513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574982,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1499136,"data_stored":697843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574983,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33240,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338600,"statfs":{"total":96636764160,"available":96602726400,"internally_reserved":0,"allocated":2105344,"data_stored":1294903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:31:15.839 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T13:31:15.839 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T13:31:15.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.981+0000 7f8e05382640 1 Processor -- start 2024-04-02T13:31:15.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.981+0000 7f8e05382640 1 -- start start 2024-04-02T13:31:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.982+0000 7f8e05382640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8e0015c1c0 0x7f8e000d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.982+0000 7f8e05382640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8e000d0a40 0x7f8e000cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.982+0000 7f8e05382640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e000cd7f0 con 0x7f8e000ccf90 2024-04-02T13:31:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.982+0000 7f8e05382640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f8e000d0e80 con 0x7f8e000d0a40 2024-04-02T13:31:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.982+0000 7f8e05382640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8e000cdad0 con 0x7f8e0015c1c0 2024-04-02T13:31:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.982+0000 7f8dfe7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8e000d0a40 0x7f8e000cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.982+0000 7f8dff7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f8e000ccf90 0x7f8e000ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40972/0 (socket says 172.21.15.44:40972) 2024-04-02T13:31:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.982+0000 7f8dff7fe640 1 -- 172.21.15.44:0/657089675 learned_addr learned my addr 172.21.15.44:0/657089675 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.982+0000 7f8dfeffd640 1 --2- 172.21.15.44:0/657089675 >> v2:172.21.15.69:3300/0 conn(0x7f8e0015c1c0 0x7f8e000d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.982+0000 7f8dfe7fc640 1 -- 172.21.15.44:0/657089675 >> v2:172.21.15.69:3300/0 conn(0x7f8e0015c1c0 msgr2=0x7f8e000d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.982+0000 7f8dfe7fc640 1 --2- 172.21.15.44:0/657089675 >> v2:172.21.15.69:3300/0 conn(0x7f8e0015c1c0 0x7f8e000d0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.982+0000 7f8dfe7fc640 1 -- 172.21.15.44:0/657089675 >> v1:172.21.15.44:6789/0 conn(0x7f8e000ccf90 legacy=0x7f8e000ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.982+0000 7f8dfdffb640 1 -- 172.21.15.44:0/657089675 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1549998797 0 0) 0x7f8e000cd7f0 con 0x7f8e000ccf90 2024-04-02T13:31:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.983+0000 7f8dfe7fc640 1 -- 172.21.15.44:0/657089675 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e000cddb0 con 0x7f8e000d0a40 2024-04-02T13:31:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.983+0000 7f8dfe7fc640 1 --2- 172.21.15.44:0/657089675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e000d0a40 0x7f8e000cca50 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f8e000cfa90 tx=0x7f8de802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be1abc48a21928d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.983+0000 7f8dfdffb640 1 -- 172.21.15.44:0/657089675 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8034070 con 0x7f8e000d0a40 2024-04-02T13:31:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.983+0000 7f8dfdffb640 1 -- 172.21.15.44:0/657089675 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de8030610 con 0x7f8e000d0a40 2024-04-02T13:31:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.983+0000 7f8dfdffb640 1 -- 172.21.15.44:0/657089675 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de80309e0 con 0x7f8e000d0a40 2024-04-02T13:31:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.984+0000 7f8e05382640 1 -- 172.21.15.44:0/657089675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e000d0a40 msgr2=0x7f8e000cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.984+0000 7f8e05382640 1 --2- 172.21.15.44:0/657089675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e000d0a40 0x7f8e000cca50 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f8e000cfa90 tx=0x7f8de802b960 comp rx=0 tx=0).stop 2024-04-02T13:31:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.984+0000 7f8e05382640 1 -- 172.21.15.44:0/657089675 shutdown_connections 2024-04-02T13:31:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.984+0000 7f8e05382640 1 --2- 172.21.15.44:0/657089675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e000d0a40 0x7f8e000cca50 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.984+0000 7f8e05382640 1 --2- 172.21.15.44:0/657089675 >> v2:172.21.15.69:3300/0 conn(0x7f8e0015c1c0 0x7f8e000d0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.984+0000 7f8e05382640 1 -- 172.21.15.44:0/657089675 >> 172.21.15.44:0/657089675 conn(0x7f8e00073850 msgr2=0x7f8e0009aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:15.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.984+0000 7f8e05382640 1 -- 172.21.15.44:0/657089675 shutdown_connections 2024-04-02T13:31:15.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.984+0000 7f8e05382640 1 -- 172.21.15.44:0/657089675 wait complete. 2024-04-02T13:31:15.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.985+0000 7f8e05382640 1 Processor -- start 2024-04-02T13:31:15.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.985+0000 7f8e05382640 1 -- start start 2024-04-02T13:31:15.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.986+0000 7f8e05382640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e000ccf90 0x7f8e00159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.986+0000 7f8e05382640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e000d0a40 0x7f8e0015a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.986+0000 7f8e05382640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e0015c1c0 0x7f8e0015a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.986+0000 7f8e05382640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0015c5a0 con 0x7f8e000ccf90 2024-04-02T13:31:15.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.986+0000 7f8e05382640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8e000ce370 con 0x7f8e000d0a40 2024-04-02T13:31:15.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.986+0000 7f8e05382640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8e000cd420 con 0x7f8e0015c1c0 2024-04-02T13:31:15.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.986+0000 7f8dfeffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e000ccf90 0x7f8e00159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.986+0000 7f8dfeffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e000ccf90 0x7f8e00159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53616/0 (socket says 172.21.15.44:53616) 2024-04-02T13:31:15.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.986+0000 7f8dfeffd640 1 -- 172.21.15.44:0/1790570479 learned_addr learned my addr 172.21.15.44:0/1790570479 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:15.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.986+0000 7f8dfe7fc640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e000d0a40 0x7f8e0015a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.986+0000 7f8dff7fe640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e0015c1c0 0x7f8e0015a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.986+0000 7f8dfeffd640 1 -- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e0015c1c0 msgr2=0x7f8e0015a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.986+0000 7f8dfeffd640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e0015c1c0 0x7f8e0015a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.987+0000 7f8dfeffd640 1 -- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e000d0a40 msgr2=0x7f8e0015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.987+0000 7f8dfeffd640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e000d0a40 0x7f8e0015a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.987+0000 7f8dfeffd640 1 -- 172.21.15.44:0/1790570479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e000ca510 con 0x7f8e000ccf90 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.987+0000 7f8dfeffd640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e000ccf90 0x7f8e00159530 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f8df000ccd0 tx=0x7f8df000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.987+0000 7f8df77fe640 1 -- 172.21.15.44:0/1790570479 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df0003990 con 0x7f8e000ccf90 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.987+0000 7f8df77fe640 1 -- 172.21.15.44:0/1790570479 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8df0003b30 con 0x7f8e000ccf90 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.987+0000 7f8e05382640 1 -- 172.21.15.44:0/1790570479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e000ca7f0 con 0x7f8e000ccf90 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.987+0000 7f8df77fe640 1 -- 172.21.15.44:0/1790570479 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df0004020 con 0x7f8e000ccf90 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.987+0000 7f8e05382640 1 -- 172.21.15.44:0/1790570479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e000cadb0 con 0x7f8e000ccf90 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.988+0000 7f8e05382640 1 -- 172.21.15.44:0/1790570479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e000c5c00 con 0x7f8e000ccf90 2024-04-02T13:31:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.988+0000 7f8df77fe640 1 -- 172.21.15.44:0/1790570479 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8df00041c0 con 0x7f8e000ccf90 2024-04-02T13:31:15.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.989+0000 7f8df77fe640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8de006c8f0 0x7f8de006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:15.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.989+0000 7f8df77fe640 1 -- 172.21.15.44:0/1790570479 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f8df008c110 con 0x7f8e000ccf90 2024-04-02T13:31:15.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.989+0000 7f8dfe7fc640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8de006c8f0 0x7f8de006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:15.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.989+0000 7f8dfe7fc640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8de006c8f0 0x7f8de006edb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f8de8036000 tx=0x7f8de8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:15.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:15.993+0000 7f8df77fe640 1 -- 172.21.15.44:0/1790570479 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8e000c5c00 con 0x7f8e000ccf90 2024-04-02T13:31:16.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.247+0000 7f8e05382640 1 -- 172.21.15.44:0/1790570479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f8e00068c00 con 0x7f8e000ccf90 2024-04-02T13:31:16.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.249+0000 7f8df77fe640 1 -- 172.21.15.44:0/1790570479 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) ==== 190+0+147 (secure 0 0 0) 0x7f8df0014030 con 0x7f8e000ccf90 2024-04-02T13:31:16.252 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-04-02T13:31:16.252 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcBQxmo+PYDhAAuuQxyBfESid2DBKpd9Lz8A== 2024-04-02T13:31:16.252 INFO:teuthology.orchestra.run.smithi044.stdout: caps mds = "allow" 2024-04-02T13:31:16.252 INFO:teuthology.orchestra.run.smithi044.stdout: caps mon = "allow r" 2024-04-02T13:31:16.253 INFO:teuthology.orchestra.run.smithi044.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-02T13:31:16.253 INFO:teuthology.orchestra.run.smithi044.stderr:updated caps for client.0 2024-04-02T13:31:16.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.251+0000 7f8e05382640 1 -- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8de006c8f0 msgr2=0x7f8de006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:16.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.251+0000 7f8e05382640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8de006c8f0 0x7f8de006edb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f8de8036000 tx=0x7f8de8008030 comp rx=0 tx=0).stop 2024-04-02T13:31:16.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.251+0000 7f8e05382640 1 -- 172.21.15.44:0/1790570479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e000ccf90 msgr2=0x7f8e00159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:16.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.251+0000 7f8e05382640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e000ccf90 0x7f8e00159530 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f8df000ccd0 tx=0x7f8df000a5c0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.251+0000 7f8e05382640 1 -- 172.21.15.44:0/1790570479 shutdown_connections 2024-04-02T13:31:16.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.251+0000 7f8e05382640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8de006c8f0 0x7f8de006edb0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.251+0000 7f8e05382640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e0015c1c0 0x7f8e0015a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.251+0000 7f8e05382640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e000d0a40 0x7f8e0015a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.251+0000 7f8e05382640 1 --2- 172.21.15.44:0/1790570479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e000ccf90 0x7f8e00159530 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.251+0000 7f8e05382640 1 -- 172.21.15.44:0/1790570479 >> 172.21.15.44:0/1790570479 conn(0x7f8e00073850 msgr2=0x7f8e0015b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:16.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.251+0000 7f8e05382640 1 -- 172.21.15.44:0/1790570479 shutdown_connections 2024-04-02T13:31:16.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.251+0000 7f8e05382640 1 -- 172.21.15.44:0/1790570479 wait complete. 2024-04-02T13:31:16.265 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:31:16.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.364+0000 7fd046a55640 1 Processor -- start 2024-04-02T13:31:16.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.364+0000 7fd046a55640 1 -- start start 2024-04-02T13:31:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.365+0000 7fd046a55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd04015e4a0 0x7fd0400aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.365+0000 7fd046a55640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0400d0c50 0x7fd0400d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.365+0000 7fd046a55640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0400cc600 con 0x7fd04006ce40 2024-04-02T13:31:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.365+0000 7fd046a55640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd0400cc8e0 con 0x7fd0400d0c50 2024-04-02T13:31:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.365+0000 7fd046a55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd0400ccbc0 con 0x7fd04015e4a0 2024-04-02T13:31:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.365+0000 7fd044ff2640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd04006ce40 0x7fd0400cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40984/0 (socket says 172.21.15.44:40984) 2024-04-02T13:31:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.365+0000 7fd044ff2640 1 -- 172.21.15.44:0/1354464472 learned_addr learned my addr 172.21.15.44:0/1354464472 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.365+0000 7fd03f7fe640 1 --2- 172.21.15.44:0/1354464472 >> v2:172.21.15.69:3300/0 conn(0x7fd0400d0c50 0x7fd0400d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.365+0000 7fd03ffff640 1 --2- 172.21.15.44:0/1354464472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd04015e4a0 0x7fd0400aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.365+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2479341211 0 0) 0x7fd0400cc600 con 0x7fd04006ce40 2024-04-02T13:31:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.365+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd02c003520 con 0x7fd04006ce40 2024-04-02T13:31:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.365+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2605687764 0 0) 0x7fd02c003520 con 0x7fd04006ce40 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd04015e4a0 msgr2=0x7fd0400aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03effd640 1 --2- 172.21.15.44:0/1354464472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd04015e4a0 0x7fd0400aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 >> v2:172.21.15.69:3300/0 conn(0x7fd0400d0c50 msgr2=0x7fd0400d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03effd640 1 --2- 172.21.15.44:0/1354464472 >> v2:172.21.15.69:3300/0 conn(0x7fd0400d0c50 0x7fd0400d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0400ccea0 con 0x7fd04006ce40 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fd0380019f0 con 0x7fd04006ce40 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03f7fe640 1 --2- 172.21.15.44:0/1354464472 >> v2:172.21.15.69:3300/0 conn(0x7fd0400d0c50 0x7fd0400d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 >> v1:172.21.15.44:6789/0 conn(0x7fd04006ce40 legacy=0x7fd0400cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03effd640 1 --2- 172.21.15.44:0/1354464472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd04015e4a0 0x7fd02c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03effd640 1 --2- 172.21.15.44:0/1354464472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd02c003f30 0x7fd02c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03effd640 1 --2- 172.21.15.44:0/1354464472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd02c004d40 0x7fd02c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd0400cc8e0 con 0x7fd02c004d40 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd0400ccbc0 con 0x7fd02c003f30 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd02c00b9c0 con 0x7fd04015e4a0 2024-04-02T13:31:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd044ff2640 1 --2- 172.21.15.44:0/1354464472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd02c004d40 0x7fd02c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03ffff640 1 --2- 172.21.15.44:0/1354464472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd04015e4a0 0x7fd02c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd03f7fe640 1 --2- 172.21.15.44:0/1354464472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd02c003f30 0x7fd02c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd044ff2640 1 -- 172.21.15.44:0/1354464472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd04015e4a0 msgr2=0x7fd02c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd044ff2640 1 --2- 172.21.15.44:0/1354464472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd04015e4a0 0x7fd02c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd044ff2640 1 -- 172.21.15.44:0/1354464472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd02c003f30 msgr2=0x7fd02c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd044ff2640 1 --2- 172.21.15.44:0/1354464472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd02c003f30 0x7fd02c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.366+0000 7fd044ff2640 1 -- 172.21.15.44:0/1354464472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02c003520 con 0x7fd02c004d40 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.367+0000 7fd044ff2640 1 --2- 172.21.15.44:0/1354464472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd02c004d40 0x7fd02c0051c0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fd03800a880 tx=0x7fd02c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ceb765e39aead9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.367+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03800a4a0 con 0x7fd02c004d40 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.367+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd038034650 con 0x7fd02c004d40 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.367+0000 7fd03effd640 1 -- 172.21.15.44:0/1354464472 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038034a80 con 0x7fd02c004d40 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.368+0000 7fd046a55640 1 -- 172.21.15.44:0/1354464472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd02c004d40 msgr2=0x7fd02c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.368+0000 7fd046a55640 1 --2- 172.21.15.44:0/1354464472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd02c004d40 0x7fd02c0051c0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fd03800a880 tx=0x7fd02c005830 comp rx=0 tx=0).stop 2024-04-02T13:31:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.368+0000 7fd044ff2640 1 -- 172.21.15.44:0/1354464472 reap_dead start 2024-04-02T13:31:16.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.368+0000 7fd046a55640 1 -- 172.21.15.44:0/1354464472 shutdown_connections 2024-04-02T13:31:16.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.368+0000 7fd046a55640 1 -- 172.21.15.44:0/1354464472 >> 172.21.15.44:0/1354464472 conn(0x7fd04009aa20 msgr2=0x7fd04009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:16.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.368+0000 7fd046a55640 1 -- 172.21.15.44:0/1354464472 shutdown_connections 2024-04-02T13:31:16.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.368+0000 7fd046a55640 1 -- 172.21.15.44:0/1354464472 wait complete. 2024-04-02T13:31:16.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.369+0000 7fd046a55640 1 Processor -- start 2024-04-02T13:31:16.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.369+0000 7fd046a55640 1 -- start start 2024-04-02T13:31:16.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd046a55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd02c004d40 0x7fd0401614b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:16.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd046a55640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0401f7de0 0x7fd0400c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:16.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd046a55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd04014cfb0 0x7fd0400704a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:16.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd046a55640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd04015e880 con 0x7fd0401f7de0 2024-04-02T13:31:16.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd046a55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd0400cd460 con 0x7fd02c004d40 2024-04-02T13:31:16.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd046a55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd0400ce090 con 0x7fd04014cfb0 2024-04-02T13:31:16.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd03f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0401f7de0 0x7fd0400c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:16.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd03f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0401f7de0 0x7fd0400c9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53640/0 (socket says 172.21.15.44:53640) 2024-04-02T13:31:16.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd03f7fe640 1 -- 172.21.15.44:0/4175602375 learned_addr learned my addr 172.21.15.44:0/4175602375 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd044ff2640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd04014cfb0 0x7fd0400704a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd03ffff640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd02c004d40 0x7fd0401614b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd03f7fe640 1 -- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd04014cfb0 msgr2=0x7fd0400704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd03f7fe640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd04014cfb0 0x7fd0400704a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd03f7fe640 1 -- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd02c004d40 msgr2=0x7fd0401614b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd03f7fe640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd02c004d40 0x7fd0401614b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd03f7fe640 1 -- 172.21.15.44:0/4175602375 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0400c95c0 con 0x7fd0401f7de0 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd03ffff640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd02c004d40 0x7fd0401614b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd03f7fe640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0401f7de0 0x7fd0400c9080 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fd030011d00 tx=0x7fd03000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.370+0000 7fd03cff9640 1 -- 172.21.15.44:0/4175602375 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03000fcc0 con 0x7fd0401f7de0 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.371+0000 7fd03cff9640 1 -- 172.21.15.44:0/4175602375 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd03000fe60 con 0x7fd0401f7de0 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.371+0000 7fd03cff9640 1 -- 172.21.15.44:0/4175602375 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd030004270 con 0x7fd0401f7de0 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.371+0000 7fd046a55640 1 -- 172.21.15.44:0/4175602375 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd040150480 con 0x7fd0401f7de0 2024-04-02T13:31:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.371+0000 7fd046a55640 1 -- 172.21.15.44:0/4175602375 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0401f7300 con 0x7fd0401f7de0 2024-04-02T13:31:16.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.372+0000 7fd046a55640 1 -- 172.21.15.44:0/4175602375 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0400684f0 con 0x7fd0401f7de0 2024-04-02T13:31:16.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.372+0000 7fd03cff9640 1 -- 172.21.15.44:0/4175602375 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd030012040 con 0x7fd0401f7de0 2024-04-02T13:31:16.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.372+0000 7fd03cff9640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd02006c8f0 0x7fd02006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:16.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.373+0000 7fd03cff9640 1 -- 172.21.15.44:0/4175602375 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7fd03008c170 con 0x7fd0401f7de0 2024-04-02T13:31:16.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.373+0000 7fd03ffff640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd02006c8f0 0x7fd02006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:16.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.373+0000 7fd03ffff640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd02006c8f0 0x7fd02006edb0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd028009bc0 tx=0x7fd028008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:16.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.377+0000 7fd03cff9640 1 -- 172.21.15.44:0/4175602375 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd030019070 con 0x7fd0401f7de0 2024-04-02T13:31:16.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.605+0000 7fd046a55640 1 -- 172.21.15.44:0/4175602375 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0400b4230 con 0x7fd0401f7de0 2024-04-02T13:31:16.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.606+0000 7fd03cff9640 1 -- 172.21.15.44:0/4175602375 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5180 (secure 0 0 0) 0x7fd030059dc0 con 0x7fd0401f7de0 2024-04-02T13:31:16.609 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:16.609 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":99,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:31:11.495701+0000","modified":"2024-04-02T13:31:13.826777+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9513},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9513":{"gid":9513,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6835/2109578276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":2109578276},{"type":"v1","addr":"172.21.15.44:6835","nonce":2109578276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9513,"qdb_cluster":[9513]},"id":8}]} 2024-04-02T13:31:16.609 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 99 2024-04-02T13:31:16.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.608+0000 7fd046a55640 1 -- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd02006c8f0 msgr2=0x7fd02006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:16.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.608+0000 7fd046a55640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd02006c8f0 0x7fd02006edb0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd028009bc0 tx=0x7fd028008040 comp rx=0 tx=0).stop 2024-04-02T13:31:16.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.608+0000 7fd046a55640 1 -- 172.21.15.44:0/4175602375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0401f7de0 msgr2=0x7fd0400c9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:16.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.608+0000 7fd046a55640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0401f7de0 0x7fd0400c9080 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fd030011d00 tx=0x7fd03000f610 comp rx=0 tx=0).stop 2024-04-02T13:31:16.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.608+0000 7fd046a55640 1 -- 172.21.15.44:0/4175602375 shutdown_connections 2024-04-02T13:31:16.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.608+0000 7fd046a55640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd02006c8f0 0x7fd02006edb0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.608+0000 7fd046a55640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd04014cfb0 0x7fd0400704a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.608+0000 7fd046a55640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0401f7de0 0x7fd0400c9080 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.608+0000 7fd046a55640 1 --2- 172.21.15.44:0/4175602375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd02c004d40 0x7fd0401614b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:16.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.608+0000 7fd046a55640 1 -- 172.21.15.44:0/4175602375 >> 172.21.15.44:0/4175602375 conn(0x7fd04009aa20 msgr2=0x7fd0400d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:16.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.608+0000 7fd046a55640 1 -- 172.21.15.44:0/4175602375 shutdown_connections 2024-04-02T13:31:16.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:16.608+0000 7fd046a55640 1 -- 172.21.15.44:0/4175602375 wait complete. 2024-04-02T13:31:17.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:31:11.495701+0000', 'modified': '2024-04-02T13:31:13.826777+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9513}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9513': {'gid': 9513, 'name': 'a', 'rank': 0, 'incarnation': 96, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.44:6835/2109578276', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6833', 'nonce': 2109578276}, {'type': 'v1', 'addr': '172.21.15.44:6835', 'nonce': 2109578276}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9513, 'qdb_cluster': [9513]} 2024-04-02T13:31:17.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:31:17.026 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T13:31:17.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.131+0000 7f9c1ad7c640 1 Processor -- start 2024-04-02T13:31:17.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.131+0000 7f9c1ad7c640 1 -- start start 2024-04-02T13:31:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.131+0000 7f9c1ad7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c140cece0 0x7f9c140cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.131+0000 7f9c1ad7c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9c1406ece0 0x7f9c140acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.131+0000 7f9c1ad7c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c140cf600 con 0x7f9c140d09f0 2024-04-02T13:31:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.131+0000 7f9c1ad7c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f9c1406d9e0 con 0x7f9c1406ece0 2024-04-02T13:31:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.131+0000 7f9c1ad7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9c1406dc40 con 0x7f9c140cece0 2024-04-02T13:31:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.132+0000 7f9c19319640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9c1406ece0 0x7f9c140acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.132+0000 7f9c13fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f9c140d09f0 0x7f9c1406d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41000/0 (socket says 172.21.15.44:41000) 2024-04-02T13:31:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.132+0000 7f9c13fff640 1 -- 172.21.15.44:0/2805567201 learned_addr learned my addr 172.21.15.44:0/2805567201 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.132+0000 7f9c18b18640 1 --2- 172.21.15.44:0/2805567201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c140cece0 0x7f9c140cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.132+0000 7f9c19319640 1 -- 172.21.15.44:0/2805567201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c140cece0 msgr2=0x7f9c140cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.132+0000 7f9c19319640 1 --2- 172.21.15.44:0/2805567201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c140cece0 0x7f9c140cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.132+0000 7f9c19319640 1 -- 172.21.15.44:0/2805567201 >> v1:172.21.15.44:6789/0 conn(0x7f9c140d09f0 legacy=0x7f9c1406d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.132+0000 7f9c19319640 1 -- 172.21.15.44:0/2805567201 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c1406df20 con 0x7f9c1406ece0 2024-04-02T13:31:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.132+0000 7f9c137fe640 1 -- 172.21.15.44:0/2805567201 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 622880901 0 0) 0x7f9c140cf600 con 0x7f9c140d09f0 2024-04-02T13:31:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.133+0000 7f9c19319640 1 --2- 172.21.15.44:0/2805567201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1406ece0 0x7f9c140acd50 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f9c040097d0 tx=0x7f9c0402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce88b949aa599016 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.133+0000 7f9c137fe640 1 -- 172.21.15.44:0/2805567201 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c04035020 con 0x7f9c1406ece0 2024-04-02T13:31:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.133+0000 7f9c137fe640 1 -- 172.21.15.44:0/2805567201 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c04031800 con 0x7f9c1406ece0 2024-04-02T13:31:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.133+0000 7f9c137fe640 1 -- 172.21.15.44:0/2805567201 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c04031bd0 con 0x7f9c1406ece0 2024-04-02T13:31:17.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.133+0000 7f9c1ad7c640 1 -- 172.21.15.44:0/2805567201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1406ece0 msgr2=0x7f9c140acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.133+0000 7f9c1ad7c640 1 --2- 172.21.15.44:0/2805567201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1406ece0 0x7f9c140acd50 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f9c040097d0 tx=0x7f9c0402b8a0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.133+0000 7f9c1ad7c640 1 -- 172.21.15.44:0/2805567201 shutdown_connections 2024-04-02T13:31:17.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.133+0000 7f9c1ad7c640 1 --2- 172.21.15.44:0/2805567201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1406ece0 0x7f9c140acd50 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.133+0000 7f9c1ad7c640 1 --2- 172.21.15.44:0/2805567201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c140cece0 0x7f9c140cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.133+0000 7f9c1ad7c640 1 -- 172.21.15.44:0/2805567201 >> 172.21.15.44:0/2805567201 conn(0x7f9c140cfcd0 msgr2=0x7f9c140d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:17.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.133+0000 7f9c1ad7c640 1 -- 172.21.15.44:0/2805567201 shutdown_connections 2024-04-02T13:31:17.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.133+0000 7f9c1ad7c640 1 -- 172.21.15.44:0/2805567201 wait complete. 2024-04-02T13:31:17.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.134+0000 7f9c1ad7c640 1 Processor -- start 2024-04-02T13:31:17.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.135+0000 7f9c1ad7c640 1 -- start start 2024-04-02T13:31:17.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.135+0000 7f9c1ad7c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1406ece0 0x7f9c14148320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.135+0000 7f9c1ad7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c140cece0 0x7f9c141573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.135+0000 7f9c1ad7c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c140d09f0 0x7f9c1414b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.135+0000 7f9c1ad7c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1406e7c0 con 0x7f9c1406ece0 2024-04-02T13:31:17.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.135+0000 7f9c1ad7c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1406e4e0 con 0x7f9c140d09f0 2024-04-02T13:31:17.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.135+0000 7f9c1ad7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9c1415b220 con 0x7f9c140cece0 2024-04-02T13:31:17.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.135+0000 7f9c18b18640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1406ece0 0x7f9c14148320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c18b18640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1406ece0 0x7f9c14148320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53664/0 (socket says 172.21.15.44:53664) 2024-04-02T13:31:17.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c18b18640 1 -- 172.21.15.44:0/586151459 learned_addr learned my addr 172.21.15.44:0/586151459 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:17.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c19319640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c140d09f0 0x7f9c1414b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c13fff640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c140cece0 0x7f9c141573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c18b18640 1 -- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c140cece0 msgr2=0x7f9c141573f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c18b18640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c140cece0 0x7f9c141573f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c18b18640 1 -- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c140d09f0 msgr2=0x7f9c1414b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c18b18640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c140d09f0 0x7f9c1414b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c18b18640 1 -- 172.21.15.44:0/586151459 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c14157b90 con 0x7f9c1406ece0 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c19319640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c140d09f0 0x7f9c1414b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c18b18640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1406ece0 0x7f9c14148320 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c00ce40 tx=0x7f9c0c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c117fa640 1 -- 172.21.15.44:0/586151459 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c003b90 con 0x7f9c1406ece0 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c117fa640 1 -- 172.21.15.44:0/586151459 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c0c003d30 con 0x7f9c1406ece0 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c1ad7c640 1 -- 172.21.15.44:0/586151459 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c14157e70 con 0x7f9c1406ece0 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c117fa640 1 -- 172.21.15.44:0/586151459 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c004240 con 0x7f9c1406ece0 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.136+0000 7f9c1ad7c640 1 -- 172.21.15.44:0/586151459 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c14158430 con 0x7f9c1406ece0 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.138+0000 7f9c117fa640 1 -- 172.21.15.44:0/586151459 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9c0c0043e0 con 0x7f9c1406ece0 2024-04-02T13:31:17.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.138+0000 7f9c117fa640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c0006c8f0 0x7f9c0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.138+0000 7f9c117fa640 1 -- 172.21.15.44:0/586151459 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f9c0c08c830 con 0x7f9c1406ece0 2024-04-02T13:31:17.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.139+0000 7f9c13fff640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c0006c8f0 0x7f9c0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.139+0000 7f9c1da37740 1 Processor -- start 2024-04-02T13:31:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.139+0000 7f9c13fff640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c0006c8f0 0x7f9c0006edb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9c1415bd90 tx=0x7f9c08009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.139+0000 7f9c1da37740 1 -- start start 2024-04-02T13:31:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c1da37740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 0x55cf9303ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c1da37740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf93067f70 0x55cf9306a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c1da37740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf9303b070 0x55cf9306ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c1da37740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cf92fa4060 con 0x55cf93067f70 2024-04-02T13:31:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c1da37740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55cf92f4c650 con 0x55cf93067b90 2024-04-02T13:31:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c1da37740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55cf92fa0010 con 0x55cf9303b070 2024-04-02T13:31:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c13fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf93067f70 0x55cf9306a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c13fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf93067f70 0x55cf9306a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53674/0 (socket says 172.21.15.44:53674) 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c13fff640 1 -- 172.21.15.44:0/3211131686 learned_addr learned my addr 172.21.15.44:0/3211131686 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c18b18640 1 --2- 172.21.15.44:0/3211131686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 0x55cf9303ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c19319640 1 --2- 172.21.15.44:0/3211131686 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf9303b070 0x55cf9306ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c13fff640 1 -- 172.21.15.44:0/3211131686 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf9303b070 msgr2=0x55cf9306ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c13fff640 1 --2- 172.21.15.44:0/3211131686 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf9303b070 0x55cf9306ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c13fff640 1 -- 172.21.15.44:0/3211131686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 msgr2=0x55cf9303ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c13fff640 1 --2- 172.21.15.44:0/3211131686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 0x55cf9303ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.140+0000 7f9c13fff640 1 -- 172.21.15.44:0/3211131686 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf92cb82d0 con 0x55cf93067f70 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.141+0000 7f9c13fff640 1 --2- 172.21.15.44:0/3211131686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf93067f70 0x55cf9306a920 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f9c08007420 tx=0x7f9c08010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d3c65377b77b7c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.141+0000 7f9bfa7fc640 1 -- 172.21.15.44:0/3211131686 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c08023070 con 0x55cf93067f70 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.141+0000 7f9bfa7fc640 1 -- 172.21.15.44:0/3211131686 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c0800d8a0 con 0x55cf93067f70 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.141+0000 7f9bfa7fc640 1 -- 172.21.15.44:0/3211131686 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0800dc50 con 0x55cf93067f70 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.141+0000 7f9c1da37740 1 -- 172.21.15.44:0/3211131686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf93067f70 msgr2=0x55cf9306a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.141+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3211131686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf93067f70 0x55cf9306a920 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f9c08007420 tx=0x7f9c08010000 comp rx=0 tx=0).stop 2024-04-02T13:31:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.141+0000 7f9c1da37740 1 -- 172.21.15.44:0/3211131686 shutdown_connections 2024-04-02T13:31:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.141+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3211131686 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf9303b070 0x55cf9306ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.141+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3211131686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf93067f70 0x55cf9306a920 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.141+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3211131686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 0x55cf9303ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.141+0000 7f9c1da37740 1 -- 172.21.15.44:0/3211131686 >> 172.21.15.44:0/3211131686 conn(0x55cf9303cda0 msgr2=0x55cf93085920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.141+0000 7f9c1da37740 1 -- 172.21.15.44:0/3211131686 shutdown_connections 2024-04-02T13:31:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.141+0000 7f9c1da37740 1 -- 172.21.15.44:0/3211131686 wait complete. 2024-04-02T13:31:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.142+0000 7f9c1da37740 1 -- start start 2024-04-02T13:31:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.142+0000 7f9c1da37740 1 Processor -- start 2024-04-02T13:31:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c1da37740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 0x55cf93069610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c1da37740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067f70 0x55cf93083e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c1da37740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf93084340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c1da37740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cf9303f440 con 0x55cf930590f0 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c1da37740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55cf92f5fa10 con 0x55cf93067b90 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c1da37740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55cf93041c80 con 0x55cf93067f70 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c1da37740 10 client.0 _ll_register_callbacks cb 0x55cf92f9c0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c19319640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf93084340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c19319640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf93084340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53684/0 (socket says 172.21.15.44:53684) 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c19319640 1 -- 172.21.15.44:0/3649647652 learned_addr learned my addr 172.21.15.44:0/3649647652 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c13fff640 1 --2- 172.21.15.44:0/3649647652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067f70 0x55cf93083e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c18b18640 1 --2- 172.21.15.44:0/3649647652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 0x55cf93069610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c19319640 1 -- 172.21.15.44:0/3649647652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067f70 msgr2=0x55cf93083e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c19319640 1 --2- 172.21.15.44:0/3649647652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067f70 0x55cf93083e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c19319640 1 -- 172.21.15.44:0/3649647652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 msgr2=0x55cf93069610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c19319640 1 --2- 172.21.15.44:0/3649647652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 0x55cf93069610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c19319640 1 -- 172.21.15.44:0/3649647652 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf92d826b0 con 0x55cf930590f0 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9c19319640 1 --2- 172.21.15.44:0/3649647652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf93084340 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f9c04006ce0 tx=0x7f9c0402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89da9580b3ecc97 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.143+0000 7f9bf37fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9bf37fe640 1 -- 172.21.15.44:0/3649647652 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0404b060 con 0x55cf930590f0 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9bf37fe640 1 -- 172.21.15.44:0/3649647652 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f9c04003dd0 con 0x55cf930590f0 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9c1da37740 1 -- 172.21.15.44:0/3649647652 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c0800d8a0 con 0x55cf930590f0 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9bf37fe640 1 -- 172.21.15.44:0/3649647652 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f9c04051050 con 0x55cf930590f0 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9c1da37740 10 client.9654 fetch_fsmap learned FSMap version 99 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9c1da37740 1 -- 172.21.15.44:0/3649647652 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x55cf9306ab70 con 0x55cf930590f0 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9bf37fe640 1 -- 172.21.15.44:0/3649647652 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c0800d8a0 con 0x55cf930590f0 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9bf37fe640 1 -- 172.21.15.44:0/3649647652 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 99) ==== 3494+0+0 (secure 0 0 0) 0x7f9c04054020 con 0x55cf930590f0 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9c1da37740 10 client.9654 fetch_fsmap finished waiting for FSMap version 99 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9c1da37740 10 client.9654 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9c1da37740 20 client.9654 populate_metadata read hostname 'smithi044' 2024-04-02T13:31:17.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3649647652 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55cf9306d2a0 0x55cf93099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9c1da37740 4 client.9654 mds_command: new command op to 9513 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9c1da37740 1 -- 172.21.15.44:0/3649647652 --> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf92f4bcf0 con 0x55cf9306d2a0 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.144+0000 7f9c18b18640 1 --2- 172.21.15.44:0/3649647652 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55cf9306d2a0 0x55cf93099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9c18b18640 1 --2- 172.21.15.44:0/3649647652 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55cf9306d2a0 0x55cf93099a50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3fba9caaea57006 server_cookie=af1cf8947196993d in_seq=0 out_seq=0 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9bf37fe640 10 client.9654 ms_handle_connect on v2:172.21.15.44:6833/2109578276 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9bf37fe640 1 -- 172.21.15.44:0/3649647652 <== mds.0 v2:172.21.15.44:6833/2109578276 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf92f4bcf0 con 0x55cf9306d2a0 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9bf37fe640 10 client.9654 handle_command_reply: tid=0 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9c1da37740 1 client.9654 shutdown 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9c1da37740 1 -- 172.21.15.44:0/3649647652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 msgr2=0x55cf93084340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3649647652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf93084340 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f9c04006ce0 tx=0x7f9c0402e000 comp rx=0 tx=0).stop 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9c1da37740 1 -- 172.21.15.44:0/3649647652 shutdown_connections 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3649647652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf93084340 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3649647652 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55cf9306d2a0 0x55cf93099a50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3649647652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067f70 0x55cf93083e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3649647652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 0x55cf93069610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9bf37fe640 0 client.9654 ms_handle_reset on v2:172.21.15.44:6833/2109578276 2024-04-02T13:31:17.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9c1da37740 1 -- 172.21.15.44:0/3649647652 >> 172.21.15.44:0/3649647652 conn(0x55cf9303b070 msgr2=0x55cf9303ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:17.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9c1da37740 1 -- 172.21.15.44:0/3649647652 shutdown_connections 2024-04-02T13:31:17.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.145+0000 7f9c1da37740 1 -- 172.21.15.44:0/3649647652 wait complete. 2024-04-02T13:31:17.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.146+0000 7f9c1da37740 20 client.9654 trim_cache size 0 max 16384 2024-04-02T13:31:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.164+0000 7f9c1da37740 1 Processor -- start 2024-04-02T13:31:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.164+0000 7f9c1da37740 1 -- start start 2024-04-02T13:31:17.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.164+0000 7f9c1da37740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf9303a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.164+0000 7f9c1da37740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 0x55cf93039990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.164+0000 7f9c1da37740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067f70 0x55cf930f6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.164+0000 7f9c1da37740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9c141ffe70 con 0x55cf930590f0 2024-04-02T13:31:17.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.164+0000 7f9c1da37740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1409cea0 con 0x55cf93067b90 2024-04-02T13:31:17.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.164+0000 7f9c1da37740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9c141ff880 con 0x55cf93067f70 2024-04-02T13:31:17.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.164+0000 7f9c18b18640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf9303a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9c18b18640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf9303a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53696/0 (socket says 172.21.15.44:53696) 2024-04-02T13:31:17.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9c18b18640 1 -- 172.21.15.44:0/3695799050 learned_addr learned my addr 172.21.15.44:0/3695799050 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:17.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9c13fff640 1 --2- 172.21.15.44:0/3695799050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 0x55cf93039990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9c19319640 1 --2- 172.21.15.44:0/3695799050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067f70 0x55cf930f6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9c18b18640 1 -- 172.21.15.44:0/3695799050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067f70 msgr2=0x55cf930f6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:17.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9c18b18640 1 --2- 172.21.15.44:0/3695799050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067f70 0x55cf930f6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9c18b18640 1 -- 172.21.15.44:0/3695799050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 msgr2=0x55cf93039990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:17.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9c18b18640 1 --2- 172.21.15.44:0/3695799050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 0x55cf93039990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9c18b18640 1 -- 172.21.15.44:0/3695799050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf93086650 con 0x55cf930590f0 2024-04-02T13:31:17.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9c19319640 1 --2- 172.21.15.44:0/3695799050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067f70 0x55cf930f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:17.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9c18b18640 1 --2- 172.21.15.44:0/3695799050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf9303a320 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c056830 tx=0x7f9c0c070040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7fd47b5ab264fa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:17.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3695799050 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c088030 con 0x55cf930590f0 2024-04-02T13:31:17.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3695799050 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c0c062fb0 con 0x55cf930590f0 2024-04-02T13:31:17.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.165+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3695799050 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c058470 con 0x55cf930590f0 2024-04-02T13:31:17.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.166+0000 7f9c1da37740 1 -- 172.21.15.44:0/3695799050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 msgr2=0x55cf9303a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.166+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3695799050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf9303a320 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c056830 tx=0x7f9c0c070040 comp rx=0 tx=0).stop 2024-04-02T13:31:17.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.166+0000 7f9c1da37740 1 -- 172.21.15.44:0/3695799050 shutdown_connections 2024-04-02T13:31:17.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.166+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3695799050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067f70 0x55cf930f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.166+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3695799050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067b90 0x55cf93039990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.166+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3695799050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf9303a320 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.166+0000 7f9c1da37740 1 -- 172.21.15.44:0/3695799050 >> 172.21.15.44:0/3695799050 conn(0x55cf9303b070 msgr2=0x55cf93035e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:17.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.166+0000 7f9c1da37740 1 -- 172.21.15.44:0/3695799050 shutdown_connections 2024-04-02T13:31:17.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.166+0000 7f9c1da37740 1 -- 172.21.15.44:0/3695799050 wait complete. 2024-04-02T13:31:17.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.166+0000 7f9c1da37740 1 -- start start 2024-04-02T13:31:17.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.166+0000 7f9c1da37740 1 Processor -- start 2024-04-02T13:31:17.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.167+0000 7f9c1da37740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf93039f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.167+0000 7f9c1da37740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067b90 0x55cf93098010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.167+0000 7f9c1da37740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067f70 0x55cf93098550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.167+0000 7f9c1da37740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cf9306d120 con 0x55cf930590f0 2024-04-02T13:31:17.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.167+0000 7f9c1da37740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55cf9306d9c0 con 0x55cf93067f70 2024-04-02T13:31:17.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.167+0000 7f9c1da37740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55cf9306d740 con 0x55cf93067b90 2024-04-02T13:31:17.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.167+0000 7f9c1da37740 10 client.0 _ll_register_callbacks cb 0x7f9c040044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:31:17.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.167+0000 7f9c18b18640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf93039f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.167+0000 7f9c18b18640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf93039f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53706/0 (socket says 172.21.15.44:53706) 2024-04-02T13:31:17.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.167+0000 7f9c18b18640 1 -- 172.21.15.44:0/3455876465 learned_addr learned my addr 172.21.15.44:0/3455876465 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:17.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.167+0000 7f9c19319640 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067f70 0x55cf93098550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.167+0000 7f9c13fff640 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067b90 0x55cf93098010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c18b18640 1 -- 172.21.15.44:0/3455876465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067b90 msgr2=0x55cf93098010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:17.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c18b18640 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067b90 0x55cf93098010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c18b18640 1 -- 172.21.15.44:0/3455876465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067f70 msgr2=0x55cf93098550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:17.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c18b18640 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067f70 0x55cf93098550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c18b18640 1 -- 172.21.15.44:0/3455876465 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf9303a5f0 con 0x55cf930590f0 2024-04-02T13:31:17.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c13fff640 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067b90 0x55cf93098010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:17.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c18b18640 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf93039f30 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c05a360 tx=0x7f9c0c05d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa129ae14ea5fc55 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:17.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9bf37fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:31:17.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9bf37fe640 1 -- 172.21.15.44:0/3455876465 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c0046d0 con 0x55cf930590f0 2024-04-02T13:31:17.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9bf37fe640 1 -- 172.21.15.44:0/3455876465 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f9c0c056a40 con 0x55cf930590f0 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c1da37740 1 -- 172.21.15.44:0/3455876465 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c0c062fb0 con 0x55cf930590f0 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9bf37fe640 1 -- 172.21.15.44:0/3455876465 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f9c0c062b20 con 0x55cf930590f0 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c1da37740 10 client.9660 fetch_fsmap learned FSMap version 99 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c1da37740 1 -- 172.21.15.44:0/3455876465 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x55cf93082530 con 0x55cf930590f0 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9bf37fe640 1 -- 172.21.15.44:0/3455876465 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c0c062fb0 con 0x55cf930590f0 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9bf37fe640 1 -- 172.21.15.44:0/3455876465 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 99) ==== 3494+0+0 (secure 0 0 0) 0x7f9c0c072bb0 con 0x55cf930590f0 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c1da37740 10 client.9660 fetch_fsmap finished waiting for FSMap version 99 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c1da37740 10 client.9660 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c1da37740 20 client.9660 populate_metadata read hostname 'smithi044' 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55cf930f7110 0x55cf930f74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c1da37740 4 client.9660 mds_command: new command op to 9513 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.168+0000 7f9c1da37740 1 -- 172.21.15.44:0/3455876465 --> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] -- command(tid 0: {"prefix": "status"}) -- 0x55cf92fb15a0 con 0x55cf930f7110 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9c13fff640 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55cf930f7110 0x55cf930f74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9bf37fe640 10 client.9660 ms_handle_connect on v2:172.21.15.44:6833/2109578276 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9c13fff640 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55cf930f7110 0x55cf930f74f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af7726a03137cb6c server_cookie=4fe9b198af6fb826 in_seq=0 out_seq=0 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9bf37fe640 1 -- 172.21.15.44:0/3455876465 <== mds.0 v2:172.21.15.44:6833/2109578276 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55cf92fb15a0 con 0x55cf930f7110 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9bf37fe640 10 client.9660 handle_command_reply: tid=0 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9c1da37740 1 client.9660 shutdown 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9c1da37740 1 -- 172.21.15.44:0/3455876465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 msgr2=0x55cf93039f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf93039f30 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c05a360 tx=0x7f9c0c05d040 comp rx=0 tx=0).stop 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9c1da37740 1 -- 172.21.15.44:0/3455876465 shutdown_connections 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf93067f70 0x55cf93098550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55cf930f7110 0x55cf930f74f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55cf93067b90 0x55cf93098010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9bf37fe640 0 client.9660 ms_handle_reset on v2:172.21.15.44:6833/2109578276 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9c1da37740 1 --2- 172.21.15.44:0/3455876465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf930590f0 0x55cf93039f30 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9c1da37740 1 -- 172.21.15.44:0/3455876465 >> 172.21.15.44:0/3455876465 conn(0x55cf9303b070 msgr2=0x55cf930f7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9c1da37740 1 -- 172.21.15.44:0/3455876465 shutdown_connections 2024-04-02T13:31:17.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.169+0000 7f9c1da37740 1 -- 172.21.15.44:0/3455876465 wait complete. 2024-04-02T13:31:17.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.170+0000 7f9c1da37740 20 client.9660 trim_cache size 0 max 16384 2024-04-02T13:31:17.182 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:31:17.182 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:31:17.182 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:31:17.182 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 9513, 2024-04-02T13:31:17.182 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:31:17.182 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:31:17.182 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:31:17.182 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 5.0518478440000001, 2024-04-02T13:31:17.182 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 99, 2024-04-02T13:31:17.182 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 121, 2024-04-02T13:31:17.183 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 121, 2024-04-02T13:31:17.183 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 16.401552056 2024-04-02T13:31:17.183 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:31:17.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.170+0000 7f9c1ad7c640 1 -- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c0006c8f0 msgr2=0x7f9c0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.170+0000 7f9c1ad7c640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c0006c8f0 0x7f9c0006edb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9c1415bd90 tx=0x7f9c08009040 comp rx=0 tx=0).stop 2024-04-02T13:31:17.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.170+0000 7f9c1ad7c640 1 -- 172.21.15.44:0/586151459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1406ece0 msgr2=0x7f9c14148320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.170+0000 7f9c1ad7c640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1406ece0 0x7f9c14148320 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c00ce40 tx=0x7f9c0c00a660 comp rx=0 tx=0).stop 2024-04-02T13:31:17.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.170+0000 7f9c1ad7c640 1 -- 172.21.15.44:0/586151459 shutdown_connections 2024-04-02T13:31:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.170+0000 7f9c1ad7c640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9c0006c8f0 0x7f9c0006edb0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.170+0000 7f9c1ad7c640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c140d09f0 0x7f9c1414b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.170+0000 7f9c1ad7c640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c140cece0 0x7f9c141573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.170+0000 7f9c1ad7c640 1 --2- 172.21.15.44:0/586151459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1406ece0 0x7f9c14148320 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.170+0000 7f9c1ad7c640 1 -- 172.21.15.44:0/586151459 >> 172.21.15.44:0/586151459 conn(0x7f9c140cfcd0 msgr2=0x7f9c140cdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.170+0000 7f9c1ad7c640 1 -- 172.21.15.44:0/586151459 shutdown_connections 2024-04-02T13:31:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.170+0000 7f9c1ad7c640 1 -- 172.21.15.44:0/586151459 wait complete. 2024-04-02T13:31:17.185 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:31:17.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.283+0000 7f7f21f96640 1 Processor -- start 2024-04-02T13:31:17.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.283+0000 7f7f21f96640 1 -- start start 2024-04-02T13:31:17.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.284+0000 7f7f21f96640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7f1c06cf90 0x7f7f1c0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.284+0000 7f7f21f96640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f1c0cfe30 0x7f7f1c0d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.284+0000 7f7f21f96640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f1c06c1d0 con 0x7f7f1c0d0a30 2024-04-02T13:31:17.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.284+0000 7f7f21f96640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7f1c06cb00 con 0x7f7f1c06cf90 2024-04-02T13:31:17.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.284+0000 7f7f21f96640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7f1c06d3f0 con 0x7f7f1c0cfe30 2024-04-02T13:31:17.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.284+0000 7f7f1affd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7f1c06cf90 0x7f7f1c0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.284+0000 7f7f1b7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7f1c0d0a30 0x7f7f1c0d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41012/0 (socket says 172.21.15.44:41012) 2024-04-02T13:31:17.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.284+0000 7f7f1b7fe640 1 -- 172.21.15.44:0/2991558259 learned_addr learned my addr 172.21.15.44:0/2991558259 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.285+0000 7f7f1bfff640 1 --2- 172.21.15.44:0/2991558259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f1c0cfe30 0x7f7f1c0d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.285+0000 7f7f1affd640 1 -- 172.21.15.44:0/2991558259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f1c0cfe30 msgr2=0x7f7f1c0d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.285+0000 7f7f1affd640 1 --2- 172.21.15.44:0/2991558259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f1c0cfe30 0x7f7f1c0d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.285+0000 7f7f1affd640 1 -- 172.21.15.44:0/2991558259 >> v1:172.21.15.44:6789/0 conn(0x7f7f1c0d0a30 legacy=0x7f7f1c0d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.285+0000 7f7f1a7fc640 1 -- 172.21.15.44:0/2991558259 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 962476248 0 0) 0x7f7f1c06c1d0 con 0x7f7f1c0d0a30 2024-04-02T13:31:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.285+0000 7f7f1affd640 1 -- 172.21.15.44:0/2991558259 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1c06c4b0 con 0x7f7f1c06cf90 2024-04-02T13:31:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.285+0000 7f7f1bfff640 1 --2- 172.21.15.44:0/2991558259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f1c0cfe30 0x7f7f1c0d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.285+0000 7f7f1affd640 1 --2- 172.21.15.44:0/2991558259 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f1c06cf90 0x7f7f1c0cf8f0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f7f100097a0 tx=0x7f7f1002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec1b954e5d768f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.285+0000 7f7f1a7fc640 1 -- 172.21.15.44:0/2991558259 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f100315a0 con 0x7f7f1c06cf90 2024-04-02T13:31:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.286+0000 7f7f1a7fc640 1 -- 172.21.15.44:0/2991558259 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f10037070 con 0x7f7f1c06cf90 2024-04-02T13:31:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.286+0000 7f7f1a7fc640 1 -- 172.21.15.44:0/2991558259 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10031960 con 0x7f7f1c06cf90 2024-04-02T13:31:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.286+0000 7f7f21f96640 1 -- 172.21.15.44:0/2991558259 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f1c06cf90 msgr2=0x7f7f1c0cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.286+0000 7f7f21f96640 1 --2- 172.21.15.44:0/2991558259 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f1c06cf90 0x7f7f1c0cf8f0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f7f100097a0 tx=0x7f7f1002b840 comp rx=0 tx=0).stop 2024-04-02T13:31:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.286+0000 7f7f21f96640 1 -- 172.21.15.44:0/2991558259 shutdown_connections 2024-04-02T13:31:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.286+0000 7f7f21f96640 1 --2- 172.21.15.44:0/2991558259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f1c0cfe30 0x7f7f1c0d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.286+0000 7f7f21f96640 1 --2- 172.21.15.44:0/2991558259 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f1c06cf90 0x7f7f1c0cf8f0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.286+0000 7f7f21f96640 1 -- 172.21.15.44:0/2991558259 >> 172.21.15.44:0/2991558259 conn(0x7f7f1c09d470 msgr2=0x7f7f1c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.286+0000 7f7f21f96640 1 -- 172.21.15.44:0/2991558259 shutdown_connections 2024-04-02T13:31:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.286+0000 7f7f21f96640 1 -- 172.21.15.44:0/2991558259 wait complete. 2024-04-02T13:31:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.287+0000 7f7f21f96640 1 Processor -- start 2024-04-02T13:31:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.287+0000 7f7f21f96640 1 -- start start 2024-04-02T13:31:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.288+0000 7f7f21f96640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f1c06cf90 0x7f7f1c1527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.288+0000 7f7f21f96640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f1c0cfe30 0x7f7f1c0b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.288+0000 7f7f21f96640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f1c0d0a30 0x7f7f1c0b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.288+0000 7f7f21f96640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c15e870 con 0x7f7f1c06cf90 2024-04-02T13:31:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.288+0000 7f7f21f96640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c15e6f0 con 0x7f7f1c0cfe30 2024-04-02T13:31:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.288+0000 7f7f21f96640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7f1c0d4da0 con 0x7f7f1c0d0a30 2024-04-02T13:31:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.288+0000 7f7f1b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f1c06cf90 0x7f7f1c1527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.288+0000 7f7f1b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f1c06cf90 0x7f7f1c1527e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53724/0 (socket says 172.21.15.44:53724) 2024-04-02T13:31:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.288+0000 7f7f1b7fe640 1 -- 172.21.15.44:0/18007082 learned_addr learned my addr 172.21.15.44:0/18007082 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.288+0000 7f7f1bfff640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f1c0d0a30 0x7f7f1c0b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.288+0000 7f7f1affd640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f1c0cfe30 0x7f7f1c0b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.289+0000 7f7f1b7fe640 1 -- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f1c0d0a30 msgr2=0x7f7f1c0b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.289+0000 7f7f1b7fe640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f1c0d0a30 0x7f7f1c0b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.289+0000 7f7f1b7fe640 1 -- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f1c0cfe30 msgr2=0x7f7f1c0b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.289+0000 7f7f1b7fe640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f1c0cfe30 0x7f7f1c0b4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.289+0000 7f7f1b7fe640 1 -- 172.21.15.44:0/18007082 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1c0b4b30 con 0x7f7f1c06cf90 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.289+0000 7f7f1affd640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f1c0cfe30 0x7f7f1c0b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.289+0000 7f7f1b7fe640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f1c06cf90 0x7f7f1c1527e0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00c500 tx=0x7f7f0c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.289+0000 7f7f03fff640 1 -- 172.21.15.44:0/18007082 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c018070 con 0x7f7f1c06cf90 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.289+0000 7f7f03fff640 1 -- 172.21.15.44:0/18007082 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f0c007580 con 0x7f7f1c06cf90 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.289+0000 7f7f21f96640 1 -- 172.21.15.44:0/18007082 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1c0b58d0 con 0x7f7f1c06cf90 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.289+0000 7f7f03fff640 1 -- 172.21.15.44:0/18007082 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c004a20 con 0x7f7f1c06cf90 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.289+0000 7f7f21f96640 1 -- 172.21.15.44:0/18007082 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f1c0b5e90 con 0x7f7f1c06cf90 2024-04-02T13:31:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.290+0000 7f7f21f96640 1 -- 172.21.15.44:0/18007082 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f1c0684f0 con 0x7f7f1c06cf90 2024-04-02T13:31:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.290+0000 7f7f03fff640 1 -- 172.21.15.44:0/18007082 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7f0c01b070 con 0x7f7f1c06cf90 2024-04-02T13:31:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.291+0000 7f7f03fff640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f0806c8f0 0x7f7f0806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.291+0000 7f7f03fff640 1 -- 172.21.15.44:0/18007082 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f7f0c08aea0 con 0x7f7f1c06cf90 2024-04-02T13:31:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.291+0000 7f7f1affd640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f0806c8f0 0x7f7f0806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.291+0000 7f7f1affd640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f0806c8f0 0x7f7f0806edb0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7f10036000 tx=0x7f7f1003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.295+0000 7f7f03fff640 1 -- 172.21.15.44:0/18007082 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7f0c058b80 con 0x7f7f1c06cf90 2024-04-02T13:31:17.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.519+0000 7f7f21f96640 1 -- 172.21.15.44:0/18007082 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7f1c0d17c0 con 0x7f7f1c06cf90 2024-04-02T13:31:17.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.519+0000 7f7f03fff640 1 -- 172.21.15.44:0/18007082 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v99) ==== 53+0+73 (secure 0 0 0) 0x7f7f0c05ce80 con 0x7f7f1c06cf90 2024-04-02T13:31:17.522 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:31:17.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.521+0000 7f7f21f96640 1 -- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f0806c8f0 msgr2=0x7f7f0806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.521+0000 7f7f21f96640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f0806c8f0 0x7f7f0806edb0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7f10036000 tx=0x7f7f1003d040 comp rx=0 tx=0).stop 2024-04-02T13:31:17.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.521+0000 7f7f21f96640 1 -- 172.21.15.44:0/18007082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f1c06cf90 msgr2=0x7f7f1c1527e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:17.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.521+0000 7f7f21f96640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f1c06cf90 0x7f7f1c1527e0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00c500 tx=0x7f7f0c00c9c0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.521+0000 7f7f21f96640 1 -- 172.21.15.44:0/18007082 shutdown_connections 2024-04-02T13:31:17.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.521+0000 7f7f21f96640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7f0806c8f0 0x7f7f0806edb0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.521+0000 7f7f21f96640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f1c0d0a30 0x7f7f1c0b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.521+0000 7f7f21f96640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f1c0cfe30 0x7f7f1c0b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.521+0000 7f7f21f96640 1 --2- 172.21.15.44:0/18007082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f1c06cf90 0x7f7f1c1527e0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:17.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.521+0000 7f7f21f96640 1 -- 172.21.15.44:0/18007082 >> 172.21.15.44:0/18007082 conn(0x7f7f1c09d470 msgr2=0x7f7f1c0d3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:17.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.521+0000 7f7f21f96640 1 -- 172.21.15.44:0/18007082 shutdown_connections 2024-04-02T13:31:17.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17.521+0000 7f7f21f96640 1 -- 172.21.15.44:0/18007082 wait complete. 2024-04-02T13:31:17.535 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:31:17.535 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:31:17.535 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:31:17.535 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:31:17.535 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:17.535 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:31:17.535 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:31:17.535 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:31:17.535 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:31:17.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:17.536 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:31:17.551 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:31:17.551 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:31:17.551 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:31:17.551 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:17.551 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:31:17.551 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:17.551 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:31:17.551 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:17.551 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:31:17.551 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42101sec preferred_lft 42101sec 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:31:17.552 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:17.553 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:31:17.553 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:17.553 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:31:17.553 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:17.553 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:31:17.553 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:17.553 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:31:17.553 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:17.553 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:31:17.553 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:17.553 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:31:17.553 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:17.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:17.554 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:31:17.617 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:31:17.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:17.619 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:31:17.674 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:31:17.674 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:31:17.674 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:31:17.674 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:31:17.675 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:31:17.675 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:31:17.675 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:31:17.675 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:31:17.675 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:31:17.675 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:31:17.675 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:31:17.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:31:17.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:17.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:17.809 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:31:17.827 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:31:17.827 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:31:17.827 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:31:17.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:17.827 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:31:17.894 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:31:17.895 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:31:17.895 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:17.949 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:31:17.950 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:31:17.950 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:18.004 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:31:18.072 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:31:18.128 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:31:18.128 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:31:18.128 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:31:18.128 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:31:18.128 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:31:18.128 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:31:18.129 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:31:18.130 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:31:18.130 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:31:18.130 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:31:18.130 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:31:18.130 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:31:18.130 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:31:18.130 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:31:18.130 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:31:18.130 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:31:18.130 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:31:18.130 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:18.130 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:18.130 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:18.131 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:18.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:18.131 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:31:18.183 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:31:18.183 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:31:18.226 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:31:18.252 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:31:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:31:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:31:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:31:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:31:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:31:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:31:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:31:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:31:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:31:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:31:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:31:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:31:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:31:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:31:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:31:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:31:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:18.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:18.310 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:31:18.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:31:18.309+0000 7fe65c379480 -1 init, newargv = 0x55bf2e320240 newargc=15 2024-04-02T13:31:18.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[81616]: starting ceph client 2024-04-02T13:31:18.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[81616]: starting fuse 2024-04-02T13:31:18.326 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:31:18.326 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:31:18.326 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:31:18.326 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:31:18.326 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:31:18.326 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:18.326 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:31:18.327 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:31:18.328 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:31:18.328 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:31:18.328 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:31:18.328 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:31:18.328 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:18.328 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:31:18.328 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:18.447 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.81616.asok 2024-04-02T13:31:18.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:18.458 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.81616.asok 2024-04-02T13:31:18.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:18.458 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.81616.asok status 2024-04-02T13:31:18.596 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:31:18.596 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:31:18.597 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:31:18.597 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:31:18.597 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:31:18.597 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:31:18.597 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:31:18.597 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "81616", 2024-04-02T13:31:18.597 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:31:18.597 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:18.597 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:31:18.597 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:31:18.598 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 9675, 2024-04-02T13:31:18.598 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:31:18.598 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:31:18.598 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:31:18.598 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 9675 2024-04-02T13:31:18.598 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:18.598 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:31:18.598 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:31:18.598 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:31:18.598 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2507467008 2024-04-02T13:31:18.598 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:18.598 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2507467008 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.9675 192.168.0.1:0/2507467008", 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/2507467008", 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 99, 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 121, 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:31:18.599 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:31:18.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:18.605 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:31:18.605 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:18.620 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:31:18.620 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:18.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:18.620 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:18.687 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:18.687 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:18.687 DEBUG:teuthology.orchestra.run.smithi044:> mount_point = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:31:18.687 DEBUG:teuthology.orchestra.run.smithi044:> parent = os.path.join(mount_point, "parent") 2024-04-02T13:31:18.687 DEBUG:teuthology.orchestra.run.smithi044:> os.mkdir(parent) 2024-04-02T13:31:18.687 DEBUG:teuthology.orchestra.run.smithi044:> flushed = os.path.join(parent, "flushed") 2024-04-02T13:31:18.687 DEBUG:teuthology.orchestra.run.smithi044:> os.mkdir(flushed) 2024-04-02T13:31:18.687 DEBUG:teuthology.orchestra.run.smithi044:> for f in ["alpha", "bravo", "charlie"]: 2024-04-02T13:31:18.688 DEBUG:teuthology.orchestra.run.smithi044:> open(os.path.join(flushed, f), '"'"'w'"'"').write(f) 2024-04-02T13:31:18.688 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:18.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:18.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:18.813 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-04-02T13:31:18.877 INFO:teuthology.orchestra.run.smithi044.stdout:./ 2024-04-02T13:31:18.877 INFO:teuthology.orchestra.run.smithi044.stdout:./parent 2024-04-02T13:31:18.877 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/flushed 2024-04-02T13:31:18.877 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/flushed/charlie 2024-04-02T13:31:18.877 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/flushed/alpha 2024-04-02T13:31:18.878 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/flushed/bravo 2024-04-02T13:31:18.878 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:18.878 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:18.878 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:18.878 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:18.878 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-04-02T13:31:18.878 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:18.983 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:31:18.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:18.992 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:18.992 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:18.992 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:18.993 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:18.993 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-04-02T13:31:18.993 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:19.098 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627776 2024-04-02T13:31:19.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:19.107 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:19.107 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:19.107 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:19.107 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:19.107 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-04-02T13:31:19.107 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:19.212 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627777 2024-04-02T13:31:19.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:19.221 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:19.221 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:19.221 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:19.221 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:19.221 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-04-02T13:31:19.222 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:19.327 INFO:teuthology.orchestra.run.smithi044.stdout:1099511628280 2024-04-02T13:31:19.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:19.336 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:19.336 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:19.336 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:19.336 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:19.336 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-04-02T13:31:19.336 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:19.441 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627778 2024-04-02T13:31:19.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:19.450 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:19.450 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:19.450 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:19.450 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:19.450 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/bravo").st_ino) 2024-04-02T13:31:19.450 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:19.555 INFO:teuthology.orchestra.run.smithi044.stdout:1099511628279 2024-04-02T13:31:19.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:19.564 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:31:19.564 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:31:19.620 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:31:19.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.758+0000 7f5a78182640 1 Processor -- start 2024-04-02T13:31:19.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.758+0000 7f5a78182640 1 -- start start 2024-04-02T13:31:19.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.759+0000 7f5a78182640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5a701617e0 0x7f5a700ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:19.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.759+0000 7f5a78182640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5a700cffd0 0x7f5a701629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:19.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.759+0000 7f5a78182640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a700cef30 con 0x7f5a700ccc90 2024-04-02T13:31:19.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.759+0000 7f5a78182640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f5a70162f00 con 0x7f5a701617e0 2024-04-02T13:31:19.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.759+0000 7f5a78182640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5a700aea90 con 0x7f5a700cffd0 2024-04-02T13:31:19.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.759+0000 7f5a75f1e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5a701617e0 0x7f5a700ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:19.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.759+0000 7f5a75f1e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5a701617e0 0x7f5a700ce8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53732/0 (socket says 172.21.15.44:53732) 2024-04-02T13:31:19.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.759+0000 7f5a75f1e640 1 -- 172.21.15.44:0/3200761256 learned_addr learned my addr 172.21.15.44:0/3200761256 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:19.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.759+0000 7f5a7571d640 1 --2- 172.21.15.44:0/3200761256 >> v2:172.21.15.69:3300/0 conn(0x7f5a700cffd0 0x7f5a701629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:19.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.759+0000 7f5a75f1e640 1 -- 172.21.15.44:0/3200761256 >> v2:172.21.15.69:3300/0 conn(0x7f5a700cffd0 msgr2=0x7f5a701629c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:19.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.759+0000 7f5a75f1e640 1 --2- 172.21.15.44:0/3200761256 >> v2:172.21.15.69:3300/0 conn(0x7f5a700cffd0 0x7f5a701629c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:19.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.759+0000 7f5a75f1e640 1 -- 172.21.15.44:0/3200761256 >> v1:172.21.15.69:6789/0 conn(0x7f5a700ccc90 legacy=0x7f5a700cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:19.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.759+0000 7f5a75f1e640 1 -- 172.21.15.44:0/3200761256 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a700cf210 con 0x7f5a701617e0 2024-04-02T13:31:19.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.760+0000 7f5a75f1e640 1 --2- 172.21.15.44:0/3200761256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a701617e0 0x7f5a700ce8c0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f5a700cee00 tx=0x7f5a6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c8e18a4d7c6f1a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:19.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.760+0000 7f5a74f1c640 1 -- 172.21.15.44:0/3200761256 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c035020 con 0x7f5a701617e0 2024-04-02T13:31:19.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.760+0000 7f5a74f1c640 1 -- 172.21.15.44:0/3200761256 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a6c031870 con 0x7f5a701617e0 2024-04-02T13:31:19.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.760+0000 7f5a74f1c640 1 -- 172.21.15.44:0/3200761256 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c031c40 con 0x7f5a701617e0 2024-04-02T13:31:19.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.761+0000 7f5a78182640 1 -- 172.21.15.44:0/3200761256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a701617e0 msgr2=0x7f5a700ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:19.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.761+0000 7f5a78182640 1 --2- 172.21.15.44:0/3200761256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a701617e0 0x7f5a700ce8c0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f5a700cee00 tx=0x7f5a6c02b910 comp rx=0 tx=0).stop 2024-04-02T13:31:19.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.761+0000 7f5a78182640 1 -- 172.21.15.44:0/3200761256 shutdown_connections 2024-04-02T13:31:19.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.761+0000 7f5a78182640 1 --2- 172.21.15.44:0/3200761256 >> v2:172.21.15.69:3300/0 conn(0x7f5a700cffd0 0x7f5a701629c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:19.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.761+0000 7f5a78182640 1 --2- 172.21.15.44:0/3200761256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a701617e0 0x7f5a700ce8c0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:19.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.761+0000 7f5a78182640 1 -- 172.21.15.44:0/3200761256 >> 172.21.15.44:0/3200761256 conn(0x7f5a700b20e0 msgr2=0x7f5a700b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:19.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.761+0000 7f5a78182640 1 -- 172.21.15.44:0/3200761256 shutdown_connections 2024-04-02T13:31:19.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.761+0000 7f5a78182640 1 -- 172.21.15.44:0/3200761256 wait complete. 2024-04-02T13:31:19.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.762+0000 7f5a78182640 1 Processor -- start 2024-04-02T13:31:19.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.762+0000 7f5a78182640 1 -- start start 2024-04-02T13:31:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a78182640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a700ccc90 0x7f5a70156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a78182640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a700cffd0 0x7f5a700c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a78182640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a701617e0 0x7f5a700c8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a78182640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700cfab0 con 0x7f5a700ccc90 2024-04-02T13:31:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a78182640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700cf7d0 con 0x7f5a701617e0 2024-04-02T13:31:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a78182640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5a70161bc0 con 0x7f5a700cffd0 2024-04-02T13:31:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a75f1e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a700ccc90 0x7f5a70156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a75f1e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a700ccc90 0x7f5a70156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53748/0 (socket says 172.21.15.44:53748) 2024-04-02T13:31:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a75f1e640 1 -- 172.21.15.44:0/3383633862 learned_addr learned my addr 172.21.15.44:0/3383633862 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a7671f640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a701617e0 0x7f5a700c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a7571d640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a700cffd0 0x7f5a700c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a75f1e640 1 -- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a700cffd0 msgr2=0x7f5a700c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a75f1e640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a700cffd0 0x7f5a700c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a75f1e640 1 -- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a701617e0 msgr2=0x7f5a700c8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a75f1e640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a701617e0 0x7f5a700c8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a75f1e640 1 -- 172.21.15.44:0/3383633862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7006dc40 con 0x7f5a700ccc90 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.763+0000 7f5a7571d640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a700cffd0 0x7f5a700c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.764+0000 7f5a75f1e640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a700ccc90 0x7f5a70156a80 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c02b950 tx=0x7f5a6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.764+0000 7f5a667fc640 1 -- 172.21.15.44:0/3383633862 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c03ad60 con 0x7f5a700ccc90 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.764+0000 7f5a667fc640 1 -- 172.21.15.44:0/3383633862 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a6c0443f0 con 0x7f5a700ccc90 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.764+0000 7f5a78182640 1 -- 172.21.15.44:0/3383633862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a7015bca0 con 0x7f5a700ccc90 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.764+0000 7f5a667fc640 1 -- 172.21.15.44:0/3383633862 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c0447c0 con 0x7f5a700ccc90 2024-04-02T13:31:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.764+0000 7f5a78182640 1 -- 172.21.15.44:0/3383633862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a7006e1c0 con 0x7f5a700ccc90 2024-04-02T13:31:19.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.765+0000 7f5a78182640 1 -- 172.21.15.44:0/3383633862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a700684f0 con 0x7f5a700ccc90 2024-04-02T13:31:19.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.765+0000 7f5a667fc640 1 -- 172.21.15.44:0/3383633862 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5a6c038020 con 0x7f5a700ccc90 2024-04-02T13:31:19.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.766+0000 7f5a667fc640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5a5406c8f0 0x7f5a5406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:19.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.766+0000 7f5a667fc640 1 -- 172.21.15.44:0/3383633862 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f5a6c0adf20 con 0x7f5a700ccc90 2024-04-02T13:31:19.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.766+0000 7f5a7571d640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5a5406c8f0 0x7f5a5406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:19.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.767+0000 7f5a7571d640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5a5406c8f0 0x7f5a5406edb0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5a700c87e0 tx=0x7f5a60008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:19.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.770+0000 7f5a667fc640 1 -- 172.21.15.44:0/3383633862 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5a6c03f070 con 0x7f5a700ccc90 2024-04-02T13:31:19.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.955+0000 7f5a78182640 1 -- 172.21.15.44:0/3383633862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5a700ce190 con 0x7f5a700ccc90 2024-04-02T13:31:19.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.957+0000 7f5a667fc640 1 -- 172.21.15.44:0/3383633862 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) ==== 74+0+16904 (secure 0 0 0) 0x7f5a6c07bc00 con 0x7f5a700ccc90 2024-04-02T13:31:19.960 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:19.960 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":121,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:13.125622+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:31:09.752845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-02T13:31:10.437454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:31:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.959+0000 7f5a78182640 1 -- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5a5406c8f0 msgr2=0x7f5a5406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.959+0000 7f5a78182640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5a5406c8f0 0x7f5a5406edb0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5a700c87e0 tx=0x7f5a60008040 comp rx=0 tx=0).stop 2024-04-02T13:31:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.959+0000 7f5a78182640 1 -- 172.21.15.44:0/3383633862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a700ccc90 msgr2=0x7f5a70156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.959+0000 7f5a78182640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a700ccc90 0x7f5a70156a80 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c02b950 tx=0x7f5a6c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:31:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.959+0000 7f5a78182640 1 -- 172.21.15.44:0/3383633862 shutdown_connections 2024-04-02T13:31:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.959+0000 7f5a78182640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5a5406c8f0 0x7f5a5406edb0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.959+0000 7f5a78182640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a701617e0 0x7f5a700c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.959+0000 7f5a78182640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a700cffd0 0x7f5a700c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.959+0000 7f5a78182640 1 --2- 172.21.15.44:0/3383633862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a700ccc90 0x7f5a70156a80 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.959+0000 7f5a78182640 1 -- 172.21.15.44:0/3383633862 >> 172.21.15.44:0/3383633862 conn(0x7f5a700b20e0 msgr2=0x7f5a7015f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:19.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.959+0000 7f5a78182640 1 -- 172.21.15.44:0/3383633862 shutdown_connections 2024-04-02T13:31:19.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:19.959+0000 7f5a78182640 1 -- 172.21.15.44:0/3383633862 wait complete. 2024-04-02T13:31:19.972 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:31:09.752845+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:31:10.437454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:31:19.972 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:31:20.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.074+0000 7f47547ba640 1 Processor -- start 2024-04-02T13:31:20.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.074+0000 7f47547ba640 1 -- start start 2024-04-02T13:31:20.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f47547ba640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f474c15e5d0 0x7f474c0d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f47547ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f474c0b4180 0x7f474c0b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f47547ba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f474c0d1630 con 0x7f474c06cff0 2024-04-02T13:31:20.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f47547ba640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f474c0d1bf0 con 0x7f474c15e5d0 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f47547ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f474c0d1910 con 0x7f474c0b4180 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f4751d55640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f474c15e5d0 0x7f474c0d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f4751d55640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f474c15e5d0 0x7f474c0d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53760/0 (socket says 172.21.15.44:53760) 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f4751d55640 1 -- 172.21.15.44:0/3081588111 learned_addr learned my addr 172.21.15.44:0/3081588111 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f4752d57640 1 --2- 172.21.15.44:0/3081588111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f474c0b4180 0x7f474c0b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f4751d55640 1 -- 172.21.15.44:0/3081588111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f474c0b4180 msgr2=0x7f474c0b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f4751d55640 1 --2- 172.21.15.44:0/3081588111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f474c0b4180 0x7f474c0b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f4751d55640 1 -- 172.21.15.44:0/3081588111 >> v1:172.21.15.69:6789/0 conn(0x7f474c06cff0 legacy=0x7f474c06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f4751d55640 1 -- 172.21.15.44:0/3081588111 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474c0b1540 con 0x7f474c15e5d0 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.075+0000 7f4752d57640 1 --2- 172.21.15.44:0/3081588111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f474c0b4180 0x7f474c0b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.076+0000 7f4751d55640 1 --2- 172.21.15.44:0/3081588111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f474c15e5d0 0x7f474c0d0d30 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f4740009870 tx=0x7f474002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5518c2b2c4cb3ea1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.076+0000 7f4751554640 1 -- 172.21.15.44:0/3081588111 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4740035020 con 0x7f474c15e5d0 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.076+0000 7f4751554640 1 -- 172.21.15.44:0/3081588111 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4740031640 con 0x7f474c15e5d0 2024-04-02T13:31:20.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.076+0000 7f4751554640 1 -- 172.21.15.44:0/3081588111 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474002d040 con 0x7f474c15e5d0 2024-04-02T13:31:20.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.077+0000 7f47547ba640 1 -- 172.21.15.44:0/3081588111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f474c15e5d0 msgr2=0x7f474c0d0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.077+0000 7f47547ba640 1 --2- 172.21.15.44:0/3081588111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f474c15e5d0 0x7f474c0d0d30 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f4740009870 tx=0x7f474002b8f0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.077+0000 7f47547ba640 1 -- 172.21.15.44:0/3081588111 shutdown_connections 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.077+0000 7f47547ba640 1 --2- 172.21.15.44:0/3081588111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f474c15e5d0 0x7f474c0d0d30 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.077+0000 7f47547ba640 1 --2- 172.21.15.44:0/3081588111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f474c0b4180 0x7f474c0b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.077+0000 7f47547ba640 1 -- 172.21.15.44:0/3081588111 >> 172.21.15.44:0/3081588111 conn(0x7f474c09aa20 msgr2=0x7f474c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.077+0000 7f47547ba640 1 -- 172.21.15.44:0/3081588111 shutdown_connections 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.077+0000 7f47547ba640 1 -- 172.21.15.44:0/3081588111 wait complete. 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.078+0000 7f47547ba640 1 Processor -- start 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.078+0000 7f47547ba640 1 -- start start 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f47547ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f474c06cff0 0x7f474c14c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f47547ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f474c0b4180 0x7f474c15e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f47547ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f474c15e5d0 0x7f474c0b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f47547ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f474c0d13f0 con 0x7f474c06cff0 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f47547ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f474c0d1270 con 0x7f474c0b4180 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f47547ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f474c0b2280 con 0x7f474c15e5d0 2024-04-02T13:31:20.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f4752556640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f474c06cff0 0x7f474c14c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f4752556640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f474c06cff0 0x7f474c14c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53774/0 (socket says 172.21.15.44:53774) 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f4752556640 1 -- 172.21.15.44:0/2566890565 learned_addr learned my addr 172.21.15.44:0/2566890565 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f4752556640 1 -- 172.21.15.44:0/2566890565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f474c15e5d0 msgr2=0x7f474c0b6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f4752556640 1 --2- 172.21.15.44:0/2566890565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f474c15e5d0 0x7f474c0b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f4752556640 1 -- 172.21.15.44:0/2566890565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f474c0b4180 msgr2=0x7f474c15e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f4752556640 1 --2- 172.21.15.44:0/2566890565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f474c0b4180 0x7f474c15e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f4752556640 1 -- 172.21.15.44:0/2566890565 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474c0b5f60 con 0x7f474c06cff0 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f4752556640 1 --2- 172.21.15.44:0/2566890565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f474c06cff0 0x7f474c14c490 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f474400c790 tx=0x7f474400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.079+0000 7f474affd640 1 -- 172.21.15.44:0/2566890565 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744016020 con 0x7f474c06cff0 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.080+0000 7f474affd640 1 -- 172.21.15.44:0/2566890565 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4744006d50 con 0x7f474c06cff0 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.080+0000 7f474affd640 1 -- 172.21.15.44:0/2566890565 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744003bb0 con 0x7f474c06cff0 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.080+0000 7f47547ba640 1 -- 172.21.15.44:0/2566890565 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f474c0b6200 con 0x7f474c06cff0 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.080+0000 7f47547ba640 1 -- 172.21.15.44:0/2566890565 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f474c0b67c0 con 0x7f474c06cff0 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.080+0000 7f47547ba640 1 -- 172.21.15.44:0/2566890565 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f474c0684f0 con 0x7f474c06cff0 2024-04-02T13:31:20.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.081+0000 7f474affd640 1 -- 172.21.15.44:0/2566890565 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f474400a850 con 0x7f474c06cff0 2024-04-02T13:31:20.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.081+0000 7f474affd640 1 --2- 172.21.15.44:0/2566890565 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f473806c8f0 0x7f473806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.081+0000 7f474affd640 1 -- 172.21.15.44:0/2566890565 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f474408ba30 con 0x7f474c06cff0 2024-04-02T13:31:20.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.082+0000 7f4751d55640 1 --2- 172.21.15.44:0/2566890565 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f473806c8f0 0x7f473806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.082+0000 7f4751d55640 1 --2- 172.21.15.44:0/2566890565 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f473806c8f0 0x7f473806edb0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4740036000 tx=0x7f4740030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:20.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.086+0000 7f474affd640 1 -- 172.21.15.44:0/2566890565 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4744020020 con 0x7f474c06cff0 2024-04-02T13:31:20.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.271+0000 7f47547ba640 1 -- 172.21.15.44:0/2566890565 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f474c06cdd0 con 0x7f474c06cff0 2024-04-02T13:31:20.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.271+0000 7f474affd640 1 -- 172.21.15.44:0/2566890565 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) ==== 93+0+10 (secure 0 0 0) 0x7f4744019310 con 0x7f474c06cff0 2024-04-02T13:31:20.274 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:31:20.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.273+0000 7f47547ba640 1 -- 172.21.15.44:0/2566890565 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f473806c8f0 msgr2=0x7f473806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.273+0000 7f47547ba640 1 --2- 172.21.15.44:0/2566890565 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f473806c8f0 0x7f473806edb0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4740036000 tx=0x7f4740030000 comp rx=0 tx=0).stop 2024-04-02T13:31:20.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.273+0000 7f47547ba640 1 -- 172.21.15.44:0/2566890565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f474c06cff0 msgr2=0x7f474c14c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.273+0000 7f47547ba640 1 --2- 172.21.15.44:0/2566890565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f474c06cff0 0x7f474c14c490 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f474400c790 tx=0x7f474400cc50 comp rx=0 tx=0).stop 2024-04-02T13:31:20.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.273+0000 7f47547ba640 1 -- 172.21.15.44:0/2566890565 shutdown_connections 2024-04-02T13:31:20.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.273+0000 7f47547ba640 1 --2- 172.21.15.44:0/2566890565 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f473806c8f0 0x7f473806edb0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.273+0000 7f47547ba640 1 --2- 172.21.15.44:0/2566890565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f474c15e5d0 0x7f474c0b6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.273+0000 7f47547ba640 1 --2- 172.21.15.44:0/2566890565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f474c0b4180 0x7f474c15e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.273+0000 7f47547ba640 1 --2- 172.21.15.44:0/2566890565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f474c06cff0 0x7f474c14c490 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.273+0000 7f47547ba640 1 -- 172.21.15.44:0/2566890565 >> 172.21.15.44:0/2566890565 conn(0x7f474c09aa20 msgr2=0x7f474c0c7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:20.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.273+0000 7f47547ba640 1 -- 172.21.15.44:0/2566890565 shutdown_connections 2024-04-02T13:31:20.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.273+0000 7f47547ba640 1 -- 172.21.15.44:0/2566890565 wait complete. 2024-04-02T13:31:20.288 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:31:20.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.390+0000 7f9906796640 1 Processor -- start 2024-04-02T13:31:20.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.390+0000 7f9906796640 1 -- start start 2024-04-02T13:31:20.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.391+0000 7f9906796640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f99000b1440 0x7f99000b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.391+0000 7f9906796640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f99000d03d0 0x7f99000d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.391+0000 7f9906796640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99000cca50 con 0x7f990015c220 2024-04-02T13:31:20.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.391+0000 7f9906796640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f99000ccd60 con 0x7f99000d03d0 2024-04-02T13:31:20.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.391+0000 7f9906796640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f99000cef40 con 0x7f99000b1440 2024-04-02T13:31:20.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.391+0000 7f9904d33640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f99000d03d0 0x7f99000d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.391+0000 7f9904d33640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f99000d03d0 0x7f99000d0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53788/0 (socket says 172.21.15.44:53788) 2024-04-02T13:31:20.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.391+0000 7f9904d33640 1 -- 172.21.15.44:0/4142730778 learned_addr learned my addr 172.21.15.44:0/4142730778 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:20.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.391+0000 7f98ff7fe640 1 --2- 172.21.15.44:0/4142730778 >> v2:172.21.15.69:3300/0 conn(0x7f99000b1440 0x7f99000b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.391+0000 7f9904d33640 1 -- 172.21.15.44:0/4142730778 >> v2:172.21.15.69:3300/0 conn(0x7f99000b1440 msgr2=0x7f99000b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:20.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.391+0000 7f9904d33640 1 --2- 172.21.15.44:0/4142730778 >> v2:172.21.15.69:3300/0 conn(0x7f99000b1440 0x7f99000b1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.391+0000 7f9904d33640 1 -- 172.21.15.44:0/4142730778 >> v1:172.21.15.69:6789/0 conn(0x7f990015c220 legacy=0x7f99000ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.391+0000 7f9904d33640 1 -- 172.21.15.44:0/4142730778 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99000cf220 con 0x7f99000d03d0 2024-04-02T13:31:20.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.392+0000 7f9904d33640 1 --2- 172.21.15.44:0/4142730778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99000d03d0 0x7f99000d0880 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f990009ff40 tx=0x7f98f802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54065d010b6fc213 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:20.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.392+0000 7f98feffd640 1 -- 172.21.15.44:0/4142730778 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f802cb20 con 0x7f99000d03d0 2024-04-02T13:31:20.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.392+0000 7f98feffd640 1 -- 172.21.15.44:0/4142730778 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f802ccc0 con 0x7f99000d03d0 2024-04-02T13:31:20.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.392+0000 7f98feffd640 1 -- 172.21.15.44:0/4142730778 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8034610 con 0x7f99000d03d0 2024-04-02T13:31:20.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.393+0000 7f9906796640 1 -- 172.21.15.44:0/4142730778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99000d03d0 msgr2=0x7f99000d0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.393+0000 7f9906796640 1 --2- 172.21.15.44:0/4142730778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99000d03d0 0x7f99000d0880 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f990009ff40 tx=0x7f98f802bf80 comp rx=0 tx=0).stop 2024-04-02T13:31:20.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.393+0000 7f9906796640 1 -- 172.21.15.44:0/4142730778 shutdown_connections 2024-04-02T13:31:20.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.393+0000 7f9906796640 1 --2- 172.21.15.44:0/4142730778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99000d03d0 0x7f99000d0880 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.393+0000 7f9906796640 1 --2- 172.21.15.44:0/4142730778 >> v2:172.21.15.69:3300/0 conn(0x7f99000b1440 0x7f99000b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.393+0000 7f9906796640 1 -- 172.21.15.44:0/4142730778 >> 172.21.15.44:0/4142730778 conn(0x7f990009aa20 msgr2=0x7f990009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.393+0000 7f9906796640 1 -- 172.21.15.44:0/4142730778 shutdown_connections 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.393+0000 7f9906796640 1 -- 172.21.15.44:0/4142730778 wait complete. 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.394+0000 7f9906796640 1 Processor -- start 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.394+0000 7f9906796640 1 -- start start 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9906796640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99000b1440 0x7f99001532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9906796640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99000d03d0 0x7f99000c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9906796640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990015c220 0x7f9900153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9906796640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f99000cd1f0 con 0x7f990015c220 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9906796640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f990015c600 con 0x7f99000b1440 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9906796640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f99000cd370 con 0x7f99000d03d0 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9904d33640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990015c220 0x7f9900153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9904d33640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990015c220 0x7f9900153e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53800/0 (socket says 172.21.15.44:53800) 2024-04-02T13:31:20.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9904d33640 1 -- 172.21.15.44:0/2696271769 learned_addr learned my addr 172.21.15.44:0/2696271769 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f98ff7fe640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99000d03d0 0x7f99000c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f98fffff640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99000b1440 0x7f99001532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9904d33640 1 -- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99000d03d0 msgr2=0x7f99000c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9904d33640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99000d03d0 0x7f99000c7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9904d33640 1 -- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99000b1440 msgr2=0x7f99001532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9904d33640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99000b1440 0x7f99001532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9904d33640 1 -- 172.21.15.44:0/2696271769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9900153800 con 0x7f990015c220 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f98ff7fe640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99000d03d0 0x7f99000c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f98fffff640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99000b1440 0x7f99001532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f9904d33640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990015c220 0x7f9900153e10 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f98f8036040 tx=0x7f98f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.395+0000 7f98fcff9640 1 -- 172.21.15.44:0/2696271769 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8043020 con 0x7f990015c220 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.396+0000 7f98fcff9640 1 -- 172.21.15.44:0/2696271769 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f803e450 con 0x7f990015c220 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.396+0000 7f9906796640 1 -- 172.21.15.44:0/2696271769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990015c960 con 0x7f990015c220 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.396+0000 7f98fcff9640 1 -- 172.21.15.44:0/2696271769 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8009510 con 0x7f990015c220 2024-04-02T13:31:20.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.396+0000 7f9906796640 1 -- 172.21.15.44:0/2696271769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f990015cf20 con 0x7f990015c220 2024-04-02T13:31:20.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.397+0000 7f9906796640 1 -- 172.21.15.44:0/2696271769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99000684f0 con 0x7f990015c220 2024-04-02T13:31:20.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.397+0000 7f98fcff9640 1 -- 172.21.15.44:0/2696271769 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f98f8046020 con 0x7f990015c220 2024-04-02T13:31:20.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.397+0000 7f98fcff9640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98e006c8f0 0x7f98e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.397+0000 7f98fcff9640 1 -- 172.21.15.44:0/2696271769 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f98f80ae4f0 con 0x7f990015c220 2024-04-02T13:31:20.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.398+0000 7f98fffff640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98e006c8f0 0x7f98e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.398+0000 7f98fffff640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98e006c8f0 0x7f98e006edb0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f98e80015e0 tx=0x7f98e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:20.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.402+0000 7f98fcff9640 1 -- 172.21.15.44:0/2696271769 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f98f804d030 con 0x7f990015c220 2024-04-02T13:31:20.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.585+0000 7f9906796640 1 -- 172.21.15.44:0/2696271769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f99000d2d10 con 0x7f990015c220 2024-04-02T13:31:20.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.585+0000 7f98fcff9640 1 -- 172.21.15.44:0/2696271769 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v121) ==== 104+0+11 (secure 0 0 0) 0x7f98f807c710 con 0x7f990015c220 2024-04-02T13:31:20.588 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:31:20.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.587+0000 7f9906796640 1 -- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98e006c8f0 msgr2=0x7f98e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.587+0000 7f9906796640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98e006c8f0 0x7f98e006edb0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f98e80015e0 tx=0x7f98e8009040 comp rx=0 tx=0).stop 2024-04-02T13:31:20.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.587+0000 7f9906796640 1 -- 172.21.15.44:0/2696271769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990015c220 msgr2=0x7f9900153e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.587+0000 7f9906796640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990015c220 0x7f9900153e10 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f98f8036040 tx=0x7f98f8004a40 comp rx=0 tx=0).stop 2024-04-02T13:31:20.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.587+0000 7f9906796640 1 -- 172.21.15.44:0/2696271769 shutdown_connections 2024-04-02T13:31:20.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.587+0000 7f9906796640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98e006c8f0 0x7f98e006edb0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.587+0000 7f9906796640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990015c220 0x7f9900153e10 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.587+0000 7f9906796640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99000d03d0 0x7f99000c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.587+0000 7f9906796640 1 --2- 172.21.15.44:0/2696271769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99000b1440 0x7f99001532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.587+0000 7f9906796640 1 -- 172.21.15.44:0/2696271769 >> 172.21.15.44:0/2696271769 conn(0x7f990009aa20 msgr2=0x7f990006f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:20.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.587+0000 7f9906796640 1 -- 172.21.15.44:0/2696271769 shutdown_connections 2024-04-02T13:31:20.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.587+0000 7f9906796640 1 -- 172.21.15.44:0/2696271769 wait complete. 2024-04-02T13:31:20.601 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:31:20.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.702+0000 7fa5d4cc7640 1 Processor -- start 2024-04-02T13:31:20.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.702+0000 7fa5d4cc7640 1 -- start start 2024-04-02T13:31:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5d4cc7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5d015e470 0x7fa5d00b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5d4cc7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa5d00b1540 0x7fa5d00d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5d4cc7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5d00ce790 0x7fa5d00d20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5d4cc7640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa5d00d2630 con 0x7fa5d00b1540 2024-04-02T13:31:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5d4cc7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa5d00ced00 con 0x7fa5d015e470 2024-04-02T13:31:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5d4cc7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d00cefe0 con 0x7fa5d00ce790 2024-04-02T13:31:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5cdd9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa5d00b1540 0x7fa5d00d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5cdd9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa5d00b1540 0x7fa5d00d1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53804/0 (socket says 172.21.15.44:53804) 2024-04-02T13:31:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5cdd9b640 1 -- 172.21.15.44:0/2979139718 learned_addr learned my addr 172.21.15.44:0/2979139718 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:20.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5cdd9b640 1 -- 172.21.15.44:0/2979139718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5d00ce790 msgr2=0x7fa5d00d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:20.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5ce59c640 1 --2- 172.21.15.44:0/2979139718 >> v2:172.21.15.69:3300/0 conn(0x7fa5d015e470 0x7fa5d00b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5ced9d640 1 --2- 172.21.15.44:0/2979139718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5d00ce790 0x7fa5d00d20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5cdd9b640 1 --2- 172.21.15.44:0/2979139718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5d00ce790 0x7fa5d00d20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5cdd9b640 1 -- 172.21.15.44:0/2979139718 >> v2:172.21.15.69:3300/0 conn(0x7fa5d015e470 msgr2=0x7fa5d00b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:20.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5cdd9b640 1 --2- 172.21.15.44:0/2979139718 >> v2:172.21.15.69:3300/0 conn(0x7fa5d015e470 0x7fa5d00b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5cdd9b640 1 -- 172.21.15.44:0/2979139718 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d00cf2c0 con 0x7fa5d00b1540 2024-04-02T13:31:20.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.703+0000 7fa5ce59c640 1 --2- 172.21.15.44:0/2979139718 >> v2:172.21.15.69:3300/0 conn(0x7fa5d015e470 0x7fa5d00b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:20.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.704+0000 7fa5cdd9b640 1 --2- 172.21.15.44:0/2979139718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5d00b1540 0x7fa5d00d1bb0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fa5d006f1c0 tx=0x7fa5c402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362735b937d42ef8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:20.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.704+0000 7fa5cd59a640 1 -- 172.21.15.44:0/2979139718 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c402e070 con 0x7fa5d00b1540 2024-04-02T13:31:20.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.704+0000 7fa5cd59a640 1 -- 172.21.15.44:0/2979139718 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c4037070 con 0x7fa5d00b1540 2024-04-02T13:31:20.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.704+0000 7fa5cd59a640 1 -- 172.21.15.44:0/2979139718 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4031a00 con 0x7fa5d00b1540 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.705+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/2979139718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5d00b1540 msgr2=0x7fa5d00d1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.705+0000 7fa5d4cc7640 1 --2- 172.21.15.44:0/2979139718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5d00b1540 0x7fa5d00d1bb0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fa5d006f1c0 tx=0x7fa5c402b880 comp rx=0 tx=0).stop 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.705+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/2979139718 shutdown_connections 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.705+0000 7fa5d4cc7640 1 --2- 172.21.15.44:0/2979139718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5d00ce790 0x7fa5d00d20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.705+0000 7fa5d4cc7640 1 --2- 172.21.15.44:0/2979139718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5d00b1540 0x7fa5d00d1bb0 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.705+0000 7fa5d4cc7640 1 --2- 172.21.15.44:0/2979139718 >> v2:172.21.15.69:3300/0 conn(0x7fa5d015e470 0x7fa5d00b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.705+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/2979139718 >> 172.21.15.44:0/2979139718 conn(0x7fa5d009aa20 msgr2=0x7fa5d009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.705+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/2979139718 shutdown_connections 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.705+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/2979139718 wait complete. 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.706+0000 7fa5d4cc7640 1 Processor -- start 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.706+0000 7fa5d4cc7640 1 -- start start 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.707+0000 7fa5d4cc7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5d00b1540 0x7fa5d00c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.707+0000 7fa5d4cc7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5d00ce790 0x7fa5d01f7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.707+0000 7fa5d4cc7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5d015e470 0x7fa5d01f7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.707+0000 7fa5d4cc7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d006cb60 con 0x7fa5d00b1540 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.707+0000 7fa5d4cc7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d015e850 con 0x7fa5d015e470 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.707+0000 7fa5d4cc7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d0161fd0 con 0x7fa5d00ce790 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.707+0000 7fa5ce59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5d00b1540 0x7fa5d00c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.707+0000 7fa5ce59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5d00b1540 0x7fa5d00c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53820/0 (socket says 172.21.15.44:53820) 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.707+0000 7fa5ce59c640 1 -- 172.21.15.44:0/3137450299 learned_addr learned my addr 172.21.15.44:0/3137450299 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.707+0000 7fa5cdd9b640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5d00ce790 0x7fa5d01f7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.707+0000 7fa5ced9d640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5d015e470 0x7fa5d01f7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.708+0000 7fa5ce59c640 1 -- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5d00ce790 msgr2=0x7fa5d01f7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.708+0000 7fa5ce59c640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5d00ce790 0x7fa5d01f7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.708+0000 7fa5ce59c640 1 -- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5d015e470 msgr2=0x7fa5d01f7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.708+0000 7fa5ce59c640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5d015e470 0x7fa5d01f7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.708+0000 7fa5ce59c640 1 -- 172.21.15.44:0/3137450299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d00c77e0 con 0x7fa5d00b1540 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.708+0000 7fa5ce59c640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5d00b1540 0x7fa5d00c65f0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800c7f0 tx=0x7fa5c800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.708+0000 7fa5beffd640 1 -- 172.21.15.44:0/3137450299 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8015070 con 0x7fa5d00b1540 2024-04-02T13:31:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.708+0000 7fa5beffd640 1 -- 172.21.15.44:0/3137450299 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c8007820 con 0x7fa5d00b1540 2024-04-02T13:31:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.708+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/3137450299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d00c7ac0 con 0x7fa5d00b1540 2024-04-02T13:31:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.708+0000 7fa5beffd640 1 -- 172.21.15.44:0/3137450299 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8007bd0 con 0x7fa5d00b1540 2024-04-02T13:31:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.708+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/3137450299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d00c8080 con 0x7fa5d00b1540 2024-04-02T13:31:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.709+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/3137450299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d00684f0 con 0x7fa5d00b1540 2024-04-02T13:31:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.709+0000 7fa5beffd640 1 -- 172.21.15.44:0/3137450299 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa5c8018070 con 0x7fa5d00b1540 2024-04-02T13:31:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.710+0000 7fa5beffd640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa5b006c920 0x7fa5b006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.710+0000 7fa5beffd640 1 -- 172.21.15.44:0/3137450299 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7fa5c808bc30 con 0x7fa5d00b1540 2024-04-02T13:31:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.710+0000 7fa5cdd9b640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa5b006c920 0x7fa5b006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.711+0000 7fa5cdd9b640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa5b006c920 0x7fa5b006ede0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4036000 tx=0x7fa5c403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.714+0000 7fa5beffd640 1 -- 172.21.15.44:0/3137450299 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa5c8059880 con 0x7fa5d00b1540 2024-04-02T13:31:20.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.899+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/3137450299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa5d00d22f0 con 0x7fa5d00b1540 2024-04-02T13:31:20.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.899+0000 7fa5beffd640 1 -- 172.21.15.44:0/3137450299 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v121) ==== 100+0+11 (secure 0 0 0) 0x7fa5c805db80 con 0x7fa5d00b1540 2024-04-02T13:31:20.902 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:31:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.901+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa5b006c920 msgr2=0x7fa5b006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.901+0000 7fa5d4cc7640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa5b006c920 0x7fa5b006ede0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4036000 tx=0x7fa5c403d040 comp rx=0 tx=0).stop 2024-04-02T13:31:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.901+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/3137450299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5d00b1540 msgr2=0x7fa5d00c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.901+0000 7fa5d4cc7640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5d00b1540 0x7fa5d00c65f0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800c7f0 tx=0x7fa5c800ccb0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.901+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/3137450299 shutdown_connections 2024-04-02T13:31:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.901+0000 7fa5d4cc7640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa5b006c920 0x7fa5b006ede0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.901+0000 7fa5d4cc7640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5d015e470 0x7fa5d01f7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.901+0000 7fa5d4cc7640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5d00ce790 0x7fa5d01f7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.901+0000 7fa5d4cc7640 1 --2- 172.21.15.44:0/3137450299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5d00b1540 0x7fa5d00c65f0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.901+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/3137450299 >> 172.21.15.44:0/3137450299 conn(0x7fa5d009aa20 msgr2=0x7fa5d006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.901+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/3137450299 shutdown_connections 2024-04-02T13:31:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:20.902+0000 7fa5d4cc7640 1 -- 172.21.15.44:0/3137450299 wait complete. 2024-04-02T13:31:20.915 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:31:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.014+0000 7f6f34f1c640 1 Processor -- start 2024-04-02T13:31:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.014+0000 7f6f34f1c640 1 -- start start 2024-04-02T13:31:21.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.015+0000 7f6f34f1c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f300d0a30 0x7f6f3006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.015+0000 7f6f34f1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f300d2040 0x7f6f300d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.015+0000 7f6f34f1c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f300d0000 con 0x7f6f300d0e10 2024-04-02T13:31:21.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.015+0000 7f6f34f1c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6f3006cb00 con 0x7f6f300d0a30 2024-04-02T13:31:21.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.015+0000 7f6f34f1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6f3006c1d0 con 0x7f6f300d2040 2024-04-02T13:31:21.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.016+0000 7f6f2ed9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f300d2040 0x7f6f300d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.016+0000 7f6f2e59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f300d0a30 0x7f6f3006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.016+0000 7f6f2dd9b640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6f300d0e10 0x7f6f300cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:38290/0 (socket says 172.21.15.44:38290) 2024-04-02T13:31:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.016+0000 7f6f2e59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f300d0a30 0x7f6f3006cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:48334/0 (socket says 172.21.15.44:48334) 2024-04-02T13:31:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.016+0000 7f6f2ed9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f300d2040 0x7f6f300d24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:39832/0 (socket says 172.21.15.44:39832) 2024-04-02T13:31:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.016+0000 7f6f2dd9b640 1 -- 172.21.15.44:0/1690944434 learned_addr learned my addr 172.21.15.44:0/1690944434 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.016+0000 7f6f2e59c640 1 -- 172.21.15.44:0/1690944434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f300d2040 msgr2=0x7f6f300d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.016+0000 7f6f2e59c640 1 --2- 172.21.15.44:0/1690944434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f300d2040 0x7f6f300d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.016+0000 7f6f2e59c640 1 -- 172.21.15.44:0/1690944434 >> v1:172.21.15.69:6789/0 conn(0x7f6f300d0e10 legacy=0x7f6f300cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.016+0000 7f6f2e59c640 1 -- 172.21.15.44:0/1690944434 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f3006c4b0 con 0x7f6f300d0a30 2024-04-02T13:31:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.017+0000 7f6f2e59c640 1 --2- 172.21.15.44:0/1690944434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f300d0a30 0x7f6f3006cf90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6f200097a0 tx=0x7f6f2002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7594398456492ea7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.017+0000 7f6f2d59a640 1 -- 172.21.15.44:0/1690944434 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f200315a0 con 0x7f6f300d0a30 2024-04-02T13:31:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.017+0000 7f6f2d59a640 1 -- 172.21.15.44:0/1690944434 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f20037070 con 0x7f6f300d0a30 2024-04-02T13:31:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.017+0000 7f6f2d59a640 1 -- 172.21.15.44:0/1690944434 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f20031960 con 0x7f6f300d0a30 2024-04-02T13:31:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.017+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1690944434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f300d0a30 msgr2=0x7f6f3006cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.017+0000 7f6f34f1c640 1 --2- 172.21.15.44:0/1690944434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f300d0a30 0x7f6f3006cf90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6f200097a0 tx=0x7f6f2002b840 comp rx=0 tx=0).stop 2024-04-02T13:31:21.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.018+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1690944434 shutdown_connections 2024-04-02T13:31:21.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.018+0000 7f6f34f1c640 1 --2- 172.21.15.44:0/1690944434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f300d2040 0x7f6f300d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.018+0000 7f6f34f1c640 1 --2- 172.21.15.44:0/1690944434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f300d0a30 0x7f6f3006cf90 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.018+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1690944434 >> 172.21.15.44:0/1690944434 conn(0x7f6f3009d470 msgr2=0x7f6f300d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:21.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.018+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1690944434 shutdown_connections 2024-04-02T13:31:21.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.018+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1690944434 wait complete. 2024-04-02T13:31:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.019+0000 7f6f34f1c640 1 Processor -- start 2024-04-02T13:31:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.019+0000 7f6f34f1c640 1 -- start start 2024-04-02T13:31:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f34f1c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f300d0a30 0x7f6f30155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f34f1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f300d0e10 0x7f6f300b5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f34f1c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f300d2040 0x7f6f300b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f34f1c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3015e6f0 con 0x7f6f300d2040 2024-04-02T13:31:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f34f1c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f300d1270 con 0x7f6f300d0a30 2024-04-02T13:31:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f34f1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6f3015e870 con 0x7f6f300d0e10 2024-04-02T13:31:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f2ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f300d2040 0x7f6f300b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f2ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f300d2040 0x7f6f300b62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53830/0 (socket says 172.21.15.44:53830) 2024-04-02T13:31:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f2ed9d640 1 -- 172.21.15.44:0/1287743160 learned_addr learned my addr 172.21.15.44:0/1287743160 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f2e59c640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f300d0a30 0x7f6f30155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f2dd9b640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f300d0e10 0x7f6f300b5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f2ed9d640 1 -- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f300d0e10 msgr2=0x7f6f300b5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f2ed9d640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f300d0e10 0x7f6f300b5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f2ed9d640 1 -- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f300d0a30 msgr2=0x7f6f30155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f2ed9d640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f300d0a30 0x7f6f30155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f2ed9d640 1 -- 172.21.15.44:0/1287743160 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f30156640 con 0x7f6f300d2040 2024-04-02T13:31:21.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f2dd9b640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f300d0e10 0x7f6f300b5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:21.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f2ed9d640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f300d2040 0x7f6f300b62d0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f6f28009d30 tx=0x7f6f2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:21.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f1effd640 1 -- 172.21.15.44:0/1287743160 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f28018020 con 0x7f6f300d2040 2024-04-02T13:31:21.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f1effd640 1 -- 172.21.15.44:0/1287743160 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f28014480 con 0x7f6f300d2040 2024-04-02T13:31:21.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1287743160 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f30156920 con 0x7f6f300d2040 2024-04-02T13:31:21.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.020+0000 7f6f1effd640 1 -- 172.21.15.44:0/1287743160 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f28014870 con 0x7f6f300d2040 2024-04-02T13:31:21.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.021+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1287743160 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f301558e0 con 0x7f6f300d2040 2024-04-02T13:31:21.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.022+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1287743160 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f300c77d0 con 0x7f6f300d2040 2024-04-02T13:31:21.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.022+0000 7f6f1effd640 1 -- 172.21.15.44:0/1287743160 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6f28005d10 con 0x7f6f300d2040 2024-04-02T13:31:21.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.022+0000 7f6f1effd640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f1006c8f0 0x7f6f1006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.023+0000 7f6f1effd640 1 -- 172.21.15.44:0/1287743160 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f6f2808c720 con 0x7f6f300d2040 2024-04-02T13:31:21.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.023+0000 7f6f2e59c640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f1006c8f0 0x7f6f1006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.023+0000 7f6f2e59c640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f1006c8f0 0x7f6f1006edb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f6f20036000 tx=0x7f6f2003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:21.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.027+0000 7f6f1effd640 1 -- 172.21.15.44:0/1287743160 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6f300c77d0 con 0x7f6f300d2040 2024-04-02T13:31:21.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.224+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1287743160 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6f300b3b50 con 0x7f6f300d2040 2024-04-02T13:31:21.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.225+0000 7f6f1effd640 1 -- 172.21.15.44:0/1287743160 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v121) ==== 80+0+0 (secure 0 0 0) 0x7f6f28022070 con 0x7f6f300d2040 2024-04-02T13:31:21.228 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:31:21.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.227+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f1006c8f0 msgr2=0x7f6f1006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.227+0000 7f6f34f1c640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f1006c8f0 0x7f6f1006edb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f6f20036000 tx=0x7f6f2003d040 comp rx=0 tx=0).stop 2024-04-02T13:31:21.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.227+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1287743160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f300d2040 msgr2=0x7f6f300b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.227+0000 7f6f34f1c640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f300d2040 0x7f6f300b62d0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f6f28009d30 tx=0x7f6f2800c490 comp rx=0 tx=0).stop 2024-04-02T13:31:21.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.227+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1287743160 shutdown_connections 2024-04-02T13:31:21.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.227+0000 7f6f34f1c640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f6f1006c8f0 0x7f6f1006edb0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.227+0000 7f6f34f1c640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f300d2040 0x7f6f300b62d0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.227+0000 7f6f34f1c640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f300d0e10 0x7f6f300b5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.227+0000 7f6f34f1c640 1 --2- 172.21.15.44:0/1287743160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f300d0a30 0x7f6f30155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.227+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1287743160 >> 172.21.15.44:0/1287743160 conn(0x7f6f3009d470 msgr2=0x7f6f300b2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:21.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.227+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1287743160 shutdown_connections 2024-04-02T13:31:21.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.227+0000 7f6f34f1c640 1 -- 172.21.15.44:0/1287743160 wait complete. 2024-04-02T13:31:21.240 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:31:21.240 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:31:21.256 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:31:21.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.395+0000 7fd5b4aed640 1 Processor -- start 2024-04-02T13:31:21.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.395+0000 7fd5b4aed640 1 -- start start 2024-04-02T13:31:21.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5b4aed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5b015e550 0x7fd5b006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5b4aed640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd5b015f040 0x7fd5b00d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5b4aed640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5b015fb50 con 0x7fd5b006d530 2024-04-02T13:31:21.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5b4aed640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd5b006d970 con 0x7fd5b015f040 2024-04-02T13:31:21.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5b4aed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b006ca90 con 0x7fd5b015e550 2024-04-02T13:31:21.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5a7fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd5b006d530 0x7fd5b015e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41026/0 (socket says 172.21.15.44:41026) 2024-04-02T13:31:21.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5a7fff640 1 -- 172.21.15.44:0/874811968 learned_addr learned my addr 172.21.15.44:0/874811968 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:21.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5aed9d640 1 --2- 172.21.15.44:0/874811968 >> v2:172.21.15.69:3300/0 conn(0x7fd5b015f040 0x7fd5b00d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5ae59c640 1 --2- 172.21.15.44:0/874811968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5b015e550 0x7fd5b006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5add9b640 1 -- 172.21.15.44:0/874811968 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 135131924 0 0) 0x7fd5b015fb50 con 0x7fd5b006d530 2024-04-02T13:31:21.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5add9b640 1 -- 172.21.15.44:0/874811968 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd59c003520 con 0x7fd5b006d530 2024-04-02T13:31:21.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5aed9d640 1 -- 172.21.15.44:0/874811968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5b015e550 msgr2=0x7fd5b006cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:21.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5aed9d640 1 --2- 172.21.15.44:0/874811968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5b015e550 0x7fd5b006cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5aed9d640 1 -- 172.21.15.44:0/874811968 >> v1:172.21.15.44:6789/0 conn(0x7fd5b006d530 legacy=0x7fd5b015e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.396+0000 7fd5aed9d640 1 -- 172.21.15.44:0/874811968 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b015f590 con 0x7fd5b015f040 2024-04-02T13:31:21.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.397+0000 7fd5aed9d640 1 --2- 172.21.15.44:0/874811968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5b015f040 0x7fd5b00d0ab0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009d90 tx=0x7fd5a802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ace1f57c99f26a03 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:21.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.397+0000 7fd5add9b640 1 -- 172.21.15.44:0/874811968 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8039070 con 0x7fd5b015f040 2024-04-02T13:31:21.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.397+0000 7fd5add9b640 1 -- 172.21.15.44:0/874811968 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5a802cce0 con 0x7fd5b015f040 2024-04-02T13:31:21.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.397+0000 7fd5add9b640 1 -- 172.21.15.44:0/874811968 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8034660 con 0x7fd5b015f040 2024-04-02T13:31:21.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.398+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874811968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5b015f040 msgr2=0x7fd5b00d0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.398+0000 7fd5b4aed640 1 --2- 172.21.15.44:0/874811968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5b015f040 0x7fd5b00d0ab0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009d90 tx=0x7fd5a802bf30 comp rx=0 tx=0).stop 2024-04-02T13:31:21.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.398+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874811968 shutdown_connections 2024-04-02T13:31:21.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.398+0000 7fd5b4aed640 1 --2- 172.21.15.44:0/874811968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5b015f040 0x7fd5b00d0ab0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.398+0000 7fd5b4aed640 1 --2- 172.21.15.44:0/874811968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5b015e550 0x7fd5b006cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.398+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874811968 >> 172.21.15.44:0/874811968 conn(0x7fd5b009aa20 msgr2=0x7fd5b009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:21.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.398+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874811968 shutdown_connections 2024-04-02T13:31:21.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.398+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874811968 wait complete. 2024-04-02T13:31:21.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.399+0000 7fd5b4aed640 1 Processor -- start 2024-04-02T13:31:21.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.399+0000 7fd5b4aed640 1 -- start start 2024-04-02T13:31:21.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5b4aed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5b006d530 0x7fd5b015a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5b4aed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5b015e550 0x7fd5b00d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5b4aed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5b015f040 0x7fd5b00d1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5b4aed640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b00d0590 con 0x7fd5b015f040 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5b4aed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b00d02b0 con 0x7fd5b015e550 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5b4aed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b00cc4b0 con 0x7fd5b006d530 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5aed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5b015f040 0x7fd5b00d1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5aed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5b015f040 0x7fd5b00d1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53832/0 (socket says 172.21.15.44:53832) 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5aed9d640 1 -- 172.21.15.44:0/874195850 learned_addr learned my addr 172.21.15.44:0/874195850 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5ae59c640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5b006d530 0x7fd5b015a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5a7fff640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5b015e550 0x7fd5b00d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5aed9d640 1 -- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5b006d530 msgr2=0x7fd5b015a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5aed9d640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5b006d530 0x7fd5b015a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5aed9d640 1 -- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5b015e550 msgr2=0x7fd5b00d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5aed9d640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5b015e550 0x7fd5b00d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.400+0000 7fd5aed9d640 1 -- 172.21.15.44:0/874195850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b009bd30 con 0x7fd5b015f040 2024-04-02T13:31:21.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.401+0000 7fd5aed9d640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5b015f040 0x7fd5b00d1490 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009d90 tx=0x7fd5a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:21.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.401+0000 7fd5a6ffd640 1 -- 172.21.15.44:0/874195850 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8039050 con 0x7fd5b015f040 2024-04-02T13:31:21.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.401+0000 7fd5a6ffd640 1 -- 172.21.15.44:0/874195850 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5a8045070 con 0x7fd5b015f040 2024-04-02T13:31:21.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.401+0000 7fd5a6ffd640 1 -- 172.21.15.44:0/874195850 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8009450 con 0x7fd5b015f040 2024-04-02T13:31:21.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.401+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874195850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b009c010 con 0x7fd5b015f040 2024-04-02T13:31:21.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.401+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874195850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b009c5d0 con 0x7fd5b015f040 2024-04-02T13:31:21.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.402+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874195850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b00684f0 con 0x7fd5b015f040 2024-04-02T13:31:21.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.402+0000 7fd5a6ffd640 1 -- 172.21.15.44:0/874195850 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd5a803e850 con 0x7fd5b015f040 2024-04-02T13:31:21.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.403+0000 7fd5a6ffd640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd59006c8f0 0x7fd59006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.403+0000 7fd5a6ffd640 1 -- 172.21.15.44:0/874195850 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7fd5a80ae960 con 0x7fd5b015f040 2024-04-02T13:31:21.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.403+0000 7fd5ae59c640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd59006c8f0 0x7fd59006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.403+0000 7fd5ae59c640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd59006c8f0 0x7fd59006edb0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0009aa0 tx=0x7fd5a0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:21.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.407+0000 7fd5a6ffd640 1 -- 172.21.15.44:0/874195850 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd5a807c630 con 0x7fd5b015f040 2024-04-02T13:31:21.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.594+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874195850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5b0070ea0 con 0x7fd5b015f040 2024-04-02T13:31:21.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.595+0000 7fd5a6ffd640 1 -- 172.21.15.44:0/874195850 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) ==== 74+0+16904 (secure 0 0 0) 0x7fd5a8080930 con 0x7fd5b015f040 2024-04-02T13:31:21.598 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:21.598 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":121,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:13.125622+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:31:09.752845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-02T13:31:10.437454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:31:21.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.597+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd59006c8f0 msgr2=0x7fd59006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.597+0000 7fd5b4aed640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd59006c8f0 0x7fd59006edb0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0009aa0 tx=0x7fd5a0009190 comp rx=0 tx=0).stop 2024-04-02T13:31:21.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.597+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874195850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5b015f040 msgr2=0x7fd5b00d1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.597+0000 7fd5b4aed640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5b015f040 0x7fd5b00d1490 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009d90 tx=0x7fd5a8004a40 comp rx=0 tx=0).stop 2024-04-02T13:31:21.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.597+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874195850 shutdown_connections 2024-04-02T13:31:21.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.597+0000 7fd5b4aed640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd59006c8f0 0x7fd59006edb0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.597+0000 7fd5b4aed640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5b015f040 0x7fd5b00d1490 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.597+0000 7fd5b4aed640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5b015e550 0x7fd5b00d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.597+0000 7fd5b4aed640 1 --2- 172.21.15.44:0/874195850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5b006d530 0x7fd5b015a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.597+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874195850 >> 172.21.15.44:0/874195850 conn(0x7fd5b009aa20 msgr2=0x7fd5b00cc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:21.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.597+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874195850 shutdown_connections 2024-04-02T13:31:21.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.597+0000 7fd5b4aed640 1 -- 172.21.15.44:0/874195850 wait complete. 2024-04-02T13:31:21.611 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:31:09.752845+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:31:10.437454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:31:21.611 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:31:21.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.713+0000 7f29ce0ba640 1 Processor -- start 2024-04-02T13:31:21.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.713+0000 7f29ce0ba640 1 -- start start 2024-04-02T13:31:21.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29ce0ba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f29c815c260 0x7f29c80b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29ce0ba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29c80cd760 con 0x7f29c80cd200 2024-04-02T13:31:21.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29ce0ba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29c80cda40 con 0x7f29c80b3e80 2024-04-02T13:31:21.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29ce0ba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f29c80cdd20 con 0x7f29c815c260 2024-04-02T13:31:21.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c7fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f29c80cd200 0x7f29c80c5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41030/0 (socket says 172.21.15.44:41030) 2024-04-02T13:31:21.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c7fff640 1 -- 172.21.15.44:0/2446992246 learned_addr learned my addr 172.21.15.44:0/2446992246 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:21.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c77fe640 1 --2- 172.21.15.44:0/2446992246 >> v2:172.21.15.69:3300/0 conn(0x7f29c815c260 0x7f29c80b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2872745302 0 0) 0x7f29c80cd760 con 0x7f29c80cd200 2024-04-02T13:31:21.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29b0003520 con 0x7f29c80cd200 2024-04-02T13:31:21.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 626700506 0 0) 0x7f29b0003520 con 0x7f29c80cd200 2024-04-02T13:31:21.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 >> v2:172.21.15.69:3300/0 conn(0x7f29c815c260 msgr2=0x7f29c80b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:21.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 --2- 172.21.15.44:0/2446992246 >> v2:172.21.15.69:3300/0 conn(0x7f29c815c260 0x7f29c80b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 >> v1:172.21.15.69:6789/0 conn(0x7f29c80b3e80 legacy=0x7f29c80ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c80ce000 con 0x7f29c80cd200 2024-04-02T13:31:21.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f29c0001a10 con 0x7f29c80cd200 2024-04-02T13:31:21.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 >> v1:172.21.15.44:6789/0 conn(0x7f29c80cd200 legacy=0x7f29c80c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 --2- 172.21.15.44:0/2446992246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b00039b0 0x7f29b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 --2- 172.21.15.44:0/2446992246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b0004800 0x7f29b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 --2- 172.21.15.44:0/2446992246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b00051c0 0x7f29b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f29c80cdd20 con 0x7f29b00051c0 2024-04-02T13:31:21.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f29b000c060 con 0x7f29b00039b0 2024-04-02T13:31:21.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.714+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f29b000c260 con 0x7f29b0004800 2024-04-02T13:31:21.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.715+0000 7f29c7fff640 1 --2- 172.21.15.44:0/2446992246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b00051c0 0x7f29b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.715+0000 7f29c77fe640 1 --2- 172.21.15.44:0/2446992246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b00039b0 0x7f29b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.715+0000 7f29c6ffd640 1 --2- 172.21.15.44:0/2446992246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b0004800 0x7f29b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.715+0000 7f29c7fff640 1 -- 172.21.15.44:0/2446992246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b0004800 msgr2=0x7f29b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:21.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.715+0000 7f29c7fff640 1 --2- 172.21.15.44:0/2446992246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b0004800 0x7f29b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.715+0000 7f29c7fff640 1 -- 172.21.15.44:0/2446992246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b00039b0 msgr2=0x7f29b0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:21.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.715+0000 7f29c7fff640 1 --2- 172.21.15.44:0/2446992246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b00039b0 0x7f29b0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.715+0000 7f29c7fff640 1 -- 172.21.15.44:0/2446992246 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b0003520 con 0x7f29b00051c0 2024-04-02T13:31:21.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.715+0000 7f29c77fe640 1 --2- 172.21.15.44:0/2446992246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b00039b0 0x7f29b0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:31:21.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.715+0000 7f29c6ffd640 1 --2- 172.21.15.44:0/2446992246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b0004800 0x7f29b0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:21.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.715+0000 7f29c7fff640 1 -- 172.21.15.44:0/2446992246 reap_dead start 2024-04-02T13:31:21.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.716+0000 7f29c7fff640 1 --2- 172.21.15.44:0/2446992246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b00051c0 0x7f29b000b9c0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f29c000a8f0 tx=0x7f29b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d7009c7778a4e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:21.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.716+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc009690 con 0x7f29b00051c0 2024-04-02T13:31:21.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.716+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29c0034650 con 0x7f29b00051c0 2024-04-02T13:31:21.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.716+0000 7f29c67fc640 1 -- 172.21.15.44:0/2446992246 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c000a4c0 con 0x7f29b00051c0 2024-04-02T13:31:21.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.716+0000 7f29ce0ba640 1 -- 172.21.15.44:0/2446992246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b00051c0 msgr2=0x7f29b000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.716+0000 7f29ce0ba640 1 --2- 172.21.15.44:0/2446992246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b00051c0 0x7f29b000b9c0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f29c000a8f0 tx=0x7f29b0005830 comp rx=0 tx=0).stop 2024-04-02T13:31:21.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.716+0000 7f29ce0ba640 1 -- 172.21.15.44:0/2446992246 shutdown_connections 2024-04-02T13:31:21.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.716+0000 7f29ce0ba640 1 --2- 172.21.15.44:0/2446992246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b00051c0 0x7f29b000b9c0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.716+0000 7f29ce0ba640 1 -- 172.21.15.44:0/2446992246 >> 172.21.15.44:0/2446992246 conn(0x7f29c809aa20 msgr2=0x7f29c809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:21.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.716+0000 7f29ce0ba640 1 -- 172.21.15.44:0/2446992246 shutdown_connections 2024-04-02T13:31:21.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.716+0000 7f29ce0ba640 1 -- 172.21.15.44:0/2446992246 wait complete. 2024-04-02T13:31:21.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.717+0000 7f29ce0ba640 1 Processor -- start 2024-04-02T13:31:21.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.718+0000 7f29ce0ba640 1 -- start start 2024-04-02T13:31:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.718+0000 7f29ce0ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b00051c0 0x7f29c8159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.718+0000 7f29ce0ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29c815a0c0 0x7f29c815f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.718+0000 7f29ce0ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c80cb0c0 0x7f29c80cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.718+0000 7f29ce0ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f29c80b3ac0 con 0x7f29c80cb0c0 2024-04-02T13:31:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.718+0000 7f29ce0ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f29c80b3940 con 0x7f29b00051c0 2024-04-02T13:31:21.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.718+0000 7f29ce0ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f29c80b3c40 con 0x7f29c815a0c0 2024-04-02T13:31:21.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29c7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c80cb0c0 0x7f29c80cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29c7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c80cb0c0 0x7f29c80cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53842/0 (socket says 172.21.15.44:53842) 2024-04-02T13:31:21.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29c7fff640 1 -- 172.21.15.44:0/340034177 learned_addr learned my addr 172.21.15.44:0/340034177 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:21.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29c77fe640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b00051c0 0x7f29c8159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29c6ffd640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29c815a0c0 0x7f29c815f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29c7fff640 1 -- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29c815a0c0 msgr2=0x7f29c815f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29c7fff640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29c815a0c0 0x7f29c815f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29c7fff640 1 -- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b00051c0 msgr2=0x7f29c8159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29c7fff640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b00051c0 0x7f29c8159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29c7fff640 1 -- 172.21.15.44:0/340034177 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c80b4fe0 con 0x7f29c80cb0c0 2024-04-02T13:31:21.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29c6ffd640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29c815a0c0 0x7f29c815f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:21.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29c7fff640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c80cb0c0 0x7f29c80cb570 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f29c000aa10 tx=0x7f29c0001de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:21.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29affff640 1 -- 172.21.15.44:0/340034177 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c0045070 con 0x7f29c80cb0c0 2024-04-02T13:31:21.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29affff640 1 -- 172.21.15.44:0/340034177 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29c003a040 con 0x7f29c80cb0c0 2024-04-02T13:31:21.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29ce0ba640 1 -- 172.21.15.44:0/340034177 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c80b52c0 con 0x7f29c80cb0c0 2024-04-02T13:31:21.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29affff640 1 -- 172.21.15.44:0/340034177 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c0034c60 con 0x7f29c80cb0c0 2024-04-02T13:31:21.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.719+0000 7f29ce0ba640 1 -- 172.21.15.44:0/340034177 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c80b5880 con 0x7f29c80cb0c0 2024-04-02T13:31:21.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.720+0000 7f29ce0ba640 1 -- 172.21.15.44:0/340034177 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c80684f0 con 0x7f29c80cb0c0 2024-04-02T13:31:21.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.721+0000 7f29affff640 1 -- 172.21.15.44:0/340034177 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f29c0048070 con 0x7f29c80cb0c0 2024-04-02T13:31:21.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.721+0000 7f29affff640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29b406c920 0x7f29b406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:21.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.721+0000 7f29affff640 1 -- 172.21.15.44:0/340034177 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f29c00ad960 con 0x7f29c80cb0c0 2024-04-02T13:31:21.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.722+0000 7f29c77fe640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29b406c920 0x7f29b406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:21.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.722+0000 7f29c77fe640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29b406c920 0x7f29b406ede0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009a80 tx=0x7f29bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:21.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.726+0000 7f29affff640 1 -- 172.21.15.44:0/340034177 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f29c007b5b0 con 0x7f29c80cb0c0 2024-04-02T13:31:21.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.911+0000 7f29ce0ba640 1 -- 172.21.15.44:0/340034177 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f29c80c8830 con 0x7f29c80cb0c0 2024-04-02T13:31:21.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.911+0000 7f29affff640 1 -- 172.21.15.44:0/340034177 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) ==== 93+0+10 (secure 0 0 0) 0x7f29c007f8b0 con 0x7f29c80cb0c0 2024-04-02T13:31:21.914 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:31:21.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.913+0000 7f29ce0ba640 1 -- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29b406c920 msgr2=0x7f29b406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.913+0000 7f29ce0ba640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29b406c920 0x7f29b406ede0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009a80 tx=0x7f29bc008040 comp rx=0 tx=0).stop 2024-04-02T13:31:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.913+0000 7f29ce0ba640 1 -- 172.21.15.44:0/340034177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c80cb0c0 msgr2=0x7f29c80cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.913+0000 7f29ce0ba640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c80cb0c0 0x7f29c80cb570 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f29c000aa10 tx=0x7f29c0001de0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.913+0000 7f29ce0ba640 1 -- 172.21.15.44:0/340034177 shutdown_connections 2024-04-02T13:31:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.913+0000 7f29ce0ba640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29b406c920 0x7f29b406ede0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.913+0000 7f29ce0ba640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c80cb0c0 0x7f29c80cb570 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.913+0000 7f29ce0ba640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29c815a0c0 0x7f29c815f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.913+0000 7f29ce0ba640 1 --2- 172.21.15.44:0/340034177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b00051c0 0x7f29c8159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.913+0000 7f29ce0ba640 1 -- 172.21.15.44:0/340034177 >> 172.21.15.44:0/340034177 conn(0x7f29c809aa20 msgr2=0x7f29c806fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.913+0000 7f29ce0ba640 1 -- 172.21.15.44:0/340034177 shutdown_connections 2024-04-02T13:31:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:21.913+0000 7f29ce0ba640 1 -- 172.21.15.44:0/340034177 wait complete. 2024-04-02T13:31:21.927 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:31:22.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.029+0000 7f37109d6640 1 Processor -- start 2024-04-02T13:31:22.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.029+0000 7f37109d6640 1 -- start start 2024-04-02T13:31:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.029+0000 7f37109d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370c15c260 0x7f370c0b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.029+0000 7f37109d6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f370c0b3e80 0x7f370c0ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.029+0000 7f37109d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f370c0cd4c0 con 0x7f370c0cd030 2024-04-02T13:31:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.029+0000 7f37109d6640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f370c0b42e0 con 0x7f370c0b3e80 2024-04-02T13:31:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.029+0000 7f37109d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f370c0cd760 con 0x7f370c15c260 2024-04-02T13:31:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.030+0000 7f3709d9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f370c0b3e80 0x7f370c0ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.030+0000 7f3709d9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f370c0b3e80 0x7f370c0ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53858/0 (socket says 172.21.15.44:53858) 2024-04-02T13:31:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.030+0000 7f370ad9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f370c0cd030 0x7f370c0ce9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41034/0 (socket says 172.21.15.44:41034) 2024-04-02T13:31:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.030+0000 7f3709d9b640 1 -- 172.21.15.44:0/1652067569 learned_addr learned my addr 172.21.15.44:0/1652067569 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.030+0000 7f370a59c640 1 --2- 172.21.15.44:0/1652067569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370c15c260 0x7f370c0b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.030+0000 7f3709d9b640 1 -- 172.21.15.44:0/1652067569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370c15c260 msgr2=0x7f370c0b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.030+0000 7f3709d9b640 1 --2- 172.21.15.44:0/1652067569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370c15c260 0x7f370c0b3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.030+0000 7f3709d9b640 1 -- 172.21.15.44:0/1652067569 >> v1:172.21.15.44:6789/0 conn(0x7f370c0cd030 legacy=0x7f370c0ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.030+0000 7f3709d9b640 1 -- 172.21.15.44:0/1652067569 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370c0cda40 con 0x7f370c0b3e80 2024-04-02T13:31:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.031+0000 7f3709d9b640 1 --2- 172.21.15.44:0/1652067569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f370c0b3e80 0x7f370c0ccaf0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f36f4009870 tx=0x7f36f402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd12752399704d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.031+0000 7f370959a640 1 -- 172.21.15.44:0/1652067569 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f4034070 con 0x7f370c0b3e80 2024-04-02T13:31:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.031+0000 7f370959a640 1 -- 172.21.15.44:0/1652067569 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f4030840 con 0x7f370c0b3e80 2024-04-02T13:31:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.031+0000 7f370959a640 1 -- 172.21.15.44:0/1652067569 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f4030c10 con 0x7f370c0b3e80 2024-04-02T13:31:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.031+0000 7f37109d6640 1 -- 172.21.15.44:0/1652067569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f370c0b3e80 msgr2=0x7f370c0ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.031+0000 7f37109d6640 1 --2- 172.21.15.44:0/1652067569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f370c0b3e80 0x7f370c0ccaf0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f36f4009870 tx=0x7f36f402b980 comp rx=0 tx=0).stop 2024-04-02T13:31:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.031+0000 7f37109d6640 1 -- 172.21.15.44:0/1652067569 shutdown_connections 2024-04-02T13:31:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.031+0000 7f37109d6640 1 --2- 172.21.15.44:0/1652067569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f370c0b3e80 0x7f370c0ccaf0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.031+0000 7f37109d6640 1 --2- 172.21.15.44:0/1652067569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370c15c260 0x7f370c0b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.031+0000 7f37109d6640 1 -- 172.21.15.44:0/1652067569 >> 172.21.15.44:0/1652067569 conn(0x7f370c09aa20 msgr2=0x7f370c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.032+0000 7f37109d6640 1 -- 172.21.15.44:0/1652067569 shutdown_connections 2024-04-02T13:31:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.032+0000 7f37109d6640 1 -- 172.21.15.44:0/1652067569 wait complete. 2024-04-02T13:31:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.033+0000 7f37109d6640 1 Processor -- start 2024-04-02T13:31:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.033+0000 7f37109d6640 1 -- start start 2024-04-02T13:31:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.033+0000 7f37109d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370c0b3e80 0x7f370c14ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.033+0000 7f37109d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f370c0cd030 0x7f370c0b7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.033+0000 7f37109d6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f370c15c260 0x7f370c15d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.033+0000 7f37109d6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f370c0cfe60 con 0x7f370c0cd030 2024-04-02T13:31:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.033+0000 7f37109d6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f370c0ce000 con 0x7f370c15c260 2024-04-02T13:31:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f37109d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f370c0c6b40 con 0x7f370c0b3e80 2024-04-02T13:31:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f3709d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f370c0cd030 0x7f370c0b7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f3709d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f370c0cd030 0x7f370c0b7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53864/0 (socket says 172.21.15.44:53864) 2024-04-02T13:31:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f3709d9b640 1 -- 172.21.15.44:0/2638351927 learned_addr learned my addr 172.21.15.44:0/2638351927 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f3709d9b640 1 -- 172.21.15.44:0/2638351927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370c0b3e80 msgr2=0x7f370c14ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f3709d9b640 1 --2- 172.21.15.44:0/2638351927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370c0b3e80 0x7f370c14ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f370ad9d640 1 --2- 172.21.15.44:0/2638351927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f370c15c260 0x7f370c15d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f3709d9b640 1 -- 172.21.15.44:0/2638351927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f370c15c260 msgr2=0x7f370c15d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f3709d9b640 1 --2- 172.21.15.44:0/2638351927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f370c15c260 0x7f370c15d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f3709d9b640 1 -- 172.21.15.44:0/2638351927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370c15de00 con 0x7f370c0cd030 2024-04-02T13:31:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f3709d9b640 1 --2- 172.21.15.44:0/2638351927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f370c0cd030 0x7f370c0b7890 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f36f402b9c0 tx=0x7f36f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f3702ffd640 1 -- 172.21.15.44:0/2638351927 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f4048070 con 0x7f370c0cd030 2024-04-02T13:31:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f3702ffd640 1 -- 172.21.15.44:0/2638351927 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f4030640 con 0x7f370c0cd030 2024-04-02T13:31:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f37109d6640 1 -- 172.21.15.44:0/2638351927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f370c15e0e0 con 0x7f370c0cd030 2024-04-02T13:31:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f3702ffd640 1 -- 172.21.15.44:0/2638351927 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f4034040 con 0x7f370c0cd030 2024-04-02T13:31:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.034+0000 7f37109d6640 1 -- 172.21.15.44:0/2638351927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f370c15e6a0 con 0x7f370c0cd030 2024-04-02T13:31:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.035+0000 7f37109d6640 1 -- 172.21.15.44:0/2638351927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f370c0684f0 con 0x7f370c0cd030 2024-04-02T13:31:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.036+0000 7f3702ffd640 1 -- 172.21.15.44:0/2638351927 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f36f404b070 con 0x7f370c0cd030 2024-04-02T13:31:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.036+0000 7f3702ffd640 1 --2- 172.21.15.44:0/2638351927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36ec06c920 0x7f36ec06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.036+0000 7f3702ffd640 1 -- 172.21.15.44:0/2638351927 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f36f40add30 con 0x7f370c0cd030 2024-04-02T13:31:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.036+0000 7f370a59c640 1 --2- 172.21.15.44:0/2638351927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36ec06c920 0x7f36ec06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.037+0000 7f370a59c640 1 --2- 172.21.15.44:0/2638351927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36ec06c920 0x7f36ec06ede0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f370400a810 tx=0x7f3704008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.041+0000 7f3702ffd640 1 -- 172.21.15.44:0/2638351927 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f36f407c060 con 0x7f370c0cd030 2024-04-02T13:31:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.225+0000 7f37109d6640 1 -- 172.21.15.44:0/2638351927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f370c0c87b0 con 0x7f370c0cd030 2024-04-02T13:31:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.225+0000 7f3702ffd640 1 -- 172.21.15.44:0/2638351927 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v121) ==== 104+0+11 (secure 0 0 0) 0x7f36f407b960 con 0x7f370c0cd030 2024-04-02T13:31:22.259 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:31:22.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.227+0000 7f37109d6640 1 -- 172.21.15.44:0/2638351927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36ec06c920 msgr2=0x7f36ec06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.227+0000 7f37109d6640 1 --2- 172.21.15.44:0/2638351927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36ec06c920 0x7f36ec06ede0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f370400a810 tx=0x7f3704008040 comp rx=0 tx=0).stop 2024-04-02T13:31:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.227+0000 7f37109d6640 1 -- 172.21.15.44:0/2638351927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f370c0cd030 msgr2=0x7f370c0b7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.227+0000 7f37109d6640 1 --2- 172.21.15.44:0/2638351927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f370c0cd030 0x7f370c0b7890 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f36f402b9c0 tx=0x7f36f4004730 comp rx=0 tx=0).stop 2024-04-02T13:31:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.227+0000 7f37109d6640 1 -- 172.21.15.44:0/2638351927 shutdown_connections 2024-04-02T13:31:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.227+0000 7f37109d6640 1 --2- 172.21.15.44:0/2638351927 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f36ec06c920 0x7f36ec06ede0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.227+0000 7f37109d6640 1 --2- 172.21.15.44:0/2638351927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f370c15c260 0x7f370c15d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.227+0000 7f37109d6640 1 --2- 172.21.15.44:0/2638351927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f370c0cd030 0x7f370c0b7890 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.227+0000 7f37109d6640 1 --2- 172.21.15.44:0/2638351927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f370c0b3e80 0x7f370c14ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.227+0000 7f37109d6640 1 -- 172.21.15.44:0/2638351927 >> 172.21.15.44:0/2638351927 conn(0x7f370c09aa20 msgr2=0x7f370c0b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.227+0000 7f37109d6640 1 -- 172.21.15.44:0/2638351927 shutdown_connections 2024-04-02T13:31:22.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.227+0000 7f37109d6640 1 -- 172.21.15.44:0/2638351927 wait complete. 2024-04-02T13:31:22.262 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:31:22.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.368+0000 7f243c6e4640 1 Processor -- start 2024-04-02T13:31:22.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.368+0000 7f243c6e4640 1 -- start start 2024-04-02T13:31:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.369+0000 7f243c6e4640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f243406b4d0 0x7f243406b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.369+0000 7f243c6e4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24340cd770 0x7f243406eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.369+0000 7f243c6e4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f24340721f0 0x7f2434072660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.369+0000 7f243c6e4640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f243406bdf0 con 0x7f243406b4d0 2024-04-02T13:31:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.369+0000 7f243c6e4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f243406f3f0 con 0x7f24340721f0 2024-04-02T13:31:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.369+0000 7f243c6e4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f243406f6d0 con 0x7f24340cd770 2024-04-02T13:31:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.369+0000 7f243a480640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f243406b4d0 0x7f243406b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.369+0000 7f243a480640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f243406b4d0 0x7f243406b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53878/0 (socket says 172.21.15.44:53878) 2024-04-02T13:31:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.369+0000 7f243a480640 1 -- 172.21.15.44:0/3675150512 learned_addr learned my addr 172.21.15.44:0/3675150512 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.369+0000 7f243ac81640 1 --2- 172.21.15.44:0/3675150512 >> v2:172.21.15.69:3300/0 conn(0x7f24340721f0 0x7f2434072660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.369+0000 7f2439c7f640 1 --2- 172.21.15.44:0/3675150512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24340cd770 0x7f243406eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.370+0000 7f243a480640 1 -- 172.21.15.44:0/3675150512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24340cd770 msgr2=0x7f243406eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.370+0000 7f243a480640 1 --2- 172.21.15.44:0/3675150512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24340cd770 0x7f243406eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.370+0000 7f243a480640 1 -- 172.21.15.44:0/3675150512 >> v2:172.21.15.69:3300/0 conn(0x7f24340721f0 msgr2=0x7f2434072660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.370+0000 7f243a480640 1 --2- 172.21.15.44:0/3675150512 >> v2:172.21.15.69:3300/0 conn(0x7f24340721f0 0x7f2434072660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.370+0000 7f243a480640 1 -- 172.21.15.44:0/3675150512 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f243406f9b0 con 0x7f243406b4d0 2024-04-02T13:31:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.370+0000 7f2439c7f640 1 --2- 172.21.15.44:0/3675150512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24340cd770 0x7f243406eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.370+0000 7f243a480640 1 --2- 172.21.15.44:0/3675150512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f243406b4d0 0x7f243406b8b0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f2434072180 tx=0x7f242c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23cdf63ebec270c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:22.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.371+0000 7f243947e640 1 -- 172.21.15.44:0/3675150512 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c035020 con 0x7f243406b4d0 2024-04-02T13:31:22.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.371+0000 7f243947e640 1 -- 172.21.15.44:0/3675150512 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f242c031870 con 0x7f243406b4d0 2024-04-02T13:31:22.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.371+0000 7f243947e640 1 -- 172.21.15.44:0/3675150512 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c031c40 con 0x7f243406b4d0 2024-04-02T13:31:22.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.371+0000 7f243c6e4640 1 -- 172.21.15.44:0/3675150512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f243406b4d0 msgr2=0x7f243406b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.371+0000 7f243c6e4640 1 --2- 172.21.15.44:0/3675150512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f243406b4d0 0x7f243406b8b0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f2434072180 tx=0x7f242c02b910 comp rx=0 tx=0).stop 2024-04-02T13:31:22.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.371+0000 7f243c6e4640 1 -- 172.21.15.44:0/3675150512 shutdown_connections 2024-04-02T13:31:22.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.371+0000 7f243c6e4640 1 --2- 172.21.15.44:0/3675150512 >> v2:172.21.15.69:3300/0 conn(0x7f24340721f0 0x7f2434072660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.371+0000 7f243c6e4640 1 --2- 172.21.15.44:0/3675150512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24340cd770 0x7f243406eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.371+0000 7f243c6e4640 1 --2- 172.21.15.44:0/3675150512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f243406b4d0 0x7f243406b8b0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.371+0000 7f243c6e4640 1 -- 172.21.15.44:0/3675150512 >> 172.21.15.44:0/3675150512 conn(0x7f24340b2a10 msgr2=0x7f24340b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:22.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.371+0000 7f243c6e4640 1 -- 172.21.15.44:0/3675150512 shutdown_connections 2024-04-02T13:31:22.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.371+0000 7f243c6e4640 1 -- 172.21.15.44:0/3675150512 wait complete. 2024-04-02T13:31:22.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.372+0000 7f243c6e4640 1 Processor -- start 2024-04-02T13:31:22.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.372+0000 7f243c6e4640 1 -- start start 2024-04-02T13:31:22.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.373+0000 7f243c6e4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f243406b4d0 0x7f2434151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.373+0000 7f243c6e4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24340721f0 0x7f24341f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.373+0000 7f243c6e4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24340cd770 0x7f243415eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.373+0000 7f243c6e4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f243406f0e0 con 0x7f24340cd770 2024-04-02T13:31:22.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.373+0000 7f243c6e4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f243406ff70 con 0x7f24340721f0 2024-04-02T13:31:22.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.373+0000 7f243c6e4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f243415c470 con 0x7f243406b4d0 2024-04-02T13:31:22.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.373+0000 7f243ac81640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24340cd770 0x7f243415eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.373+0000 7f243ac81640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24340cd770 0x7f243415eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53892/0 (socket says 172.21.15.44:53892) 2024-04-02T13:31:22.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.373+0000 7f243ac81640 1 -- 172.21.15.44:0/3056605364 learned_addr learned my addr 172.21.15.44:0/3056605364 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:22.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f243a480640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f243406b4d0 0x7f2434151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f2439c7f640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24340721f0 0x7f24341f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f243ac81640 1 -- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f243406b4d0 msgr2=0x7f2434151520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f243ac81640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f243406b4d0 0x7f2434151520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f243ac81640 1 -- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24340721f0 msgr2=0x7f24341f7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f243ac81640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24340721f0 0x7f24341f7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f243ac81640 1 -- 172.21.15.44:0/3056605364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f243415c730 con 0x7f24340cd770 2024-04-02T13:31:22.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f2439c7f640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24340721f0 0x7f24341f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:22.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f243ac81640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24340cd770 0x7f243415eff0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f242400eed0 tx=0x7f242400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:22.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f2432ffd640 1 -- 172.21.15.44:0/3056605364 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2424018020 con 0x7f24340cd770 2024-04-02T13:31:22.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f2432ffd640 1 -- 172.21.15.44:0/3056605364 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2424014400 con 0x7f24340cd770 2024-04-02T13:31:22.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f243c6e4640 1 -- 172.21.15.44:0/3056605364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24341f83c0 con 0x7f24340cd770 2024-04-02T13:31:22.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f2432ffd640 1 -- 172.21.15.44:0/3056605364 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24240147f0 con 0x7f24340cd770 2024-04-02T13:31:22.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.374+0000 7f243c6e4640 1 -- 172.21.15.44:0/3056605364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24341f8980 con 0x7f24340cd770 2024-04-02T13:31:22.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.375+0000 7f243c6e4640 1 -- 172.21.15.44:0/3056605364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24340684f0 con 0x7f24340cd770 2024-04-02T13:31:22.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.375+0000 7f2432ffd640 1 -- 172.21.15.44:0/3056605364 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2424005d10 con 0x7f24340cd770 2024-04-02T13:31:22.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.376+0000 7f2432ffd640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f242006c8f0 0x7f242006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.376+0000 7f2432ffd640 1 -- 172.21.15.44:0/3056605364 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f242408c2f0 con 0x7f24340cd770 2024-04-02T13:31:22.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.376+0000 7f243a480640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f242006c8f0 0x7f242006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.376+0000 7f243a480640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f242006c8f0 0x7f242006edb0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f242c036000 tx=0x7f242c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:22.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.380+0000 7f2432ffd640 1 -- 172.21.15.44:0/3056605364 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2424022070 con 0x7f24340cd770 2024-04-02T13:31:22.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.566+0000 7f243c6e4640 1 -- 172.21.15.44:0/3056605364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f243406c2b0 con 0x7f24340cd770 2024-04-02T13:31:22.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.566+0000 7f2432ffd640 1 -- 172.21.15.44:0/3056605364 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v121) ==== 100+0+11 (secure 0 0 0) 0x7f242405a620 con 0x7f24340cd770 2024-04-02T13:31:22.569 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:31:22.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.568+0000 7f243c6e4640 1 -- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f242006c8f0 msgr2=0x7f242006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.568+0000 7f243c6e4640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f242006c8f0 0x7f242006edb0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f242c036000 tx=0x7f242c008030 comp rx=0 tx=0).stop 2024-04-02T13:31:22.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.568+0000 7f243c6e4640 1 -- 172.21.15.44:0/3056605364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24340cd770 msgr2=0x7f243415eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.568+0000 7f243c6e4640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24340cd770 0x7f243415eff0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f242400eed0 tx=0x7f242400b480 comp rx=0 tx=0).stop 2024-04-02T13:31:22.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.568+0000 7f243c6e4640 1 -- 172.21.15.44:0/3056605364 shutdown_connections 2024-04-02T13:31:22.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.568+0000 7f243c6e4640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f242006c8f0 0x7f242006edb0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.568+0000 7f243c6e4640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24340cd770 0x7f243415eff0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.568+0000 7f243c6e4640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24340721f0 0x7f24341f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.568+0000 7f243c6e4640 1 --2- 172.21.15.44:0/3056605364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f243406b4d0 0x7f2434151520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.568+0000 7f243c6e4640 1 -- 172.21.15.44:0/3056605364 >> 172.21.15.44:0/3056605364 conn(0x7f24340b2a10 msgr2=0x7f2434159c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:22.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.568+0000 7f243c6e4640 1 -- 172.21.15.44:0/3056605364 shutdown_connections 2024-04-02T13:31:22.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.568+0000 7f243c6e4640 1 -- 172.21.15.44:0/3056605364 wait complete. 2024-04-02T13:31:22.582 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:31:22.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.695+0000 7f2fbd655640 1 Processor -- start 2024-04-02T13:31:22.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.695+0000 7f2fbd655640 1 -- start start 2024-04-02T13:31:22.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.695+0000 7f2fbd655640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806cf90 0x7f2fb80cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.695+0000 7f2fbd655640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb806c1d0 con 0x7f2fb80d0a30 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.695+0000 7f2fbd655640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb806c4b0 con 0x7f2fb80cfe30 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.695+0000 7f2fbd655640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb806cb00 con 0x7f2fb806cf90 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb6ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f2fb80d0a30 0x7f2fb80d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41046/0 (socket says 172.21.15.44:41046) 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb6ffd640 1 -- 172.21.15.44:0/1965462647 learned_addr learned my addr 172.21.15.44:0/1965462647 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2725863219 0 0) 0x7f2fb806c1d0 con 0x7f2fb80d0a30 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb67fc640 1 --2- 172.21.15.44:0/1965462647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806cf90 0x7f2fb80cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fa4003520 con 0x7f2fb80d0a30 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3013512960 0 0) 0x7f2fa4003520 con 0x7f2fb80d0a30 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806cf90 msgr2=0x7f2fb80cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 --2- 172.21.15.44:0/1965462647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806cf90 0x7f2fb80cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 >> v1:172.21.15.69:6789/0 conn(0x7f2fb80cfe30 legacy=0x7f2fb80d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb806c790 con 0x7f2fb80d0a30 2024-04-02T13:31:22.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f2fb0001670 con 0x7f2fb80d0a30 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 >> v1:172.21.15.44:6789/0 conn(0x7f2fb80d0a30 legacy=0x7f2fb80d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 --2- 172.21.15.44:0/1965462647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa40039e0 0x7f2fa4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 --2- 172.21.15.44:0/1965462647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fa4004800 0x7f2fa4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 --2- 172.21.15.44:0/1965462647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806cf90 0x7f2fa400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb806cb00 con 0x7f2fa40039e0 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa4005520 con 0x7f2fa4004800 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa400bf00 con 0x7f2fb806cf90 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb6ffd640 1 --2- 172.21.15.44:0/1965462647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa40039e0 0x7f2fa4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb67fc640 1 --2- 172.21.15.44:0/1965462647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fa4004800 0x7f2fa4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.696+0000 7f2fb77fe640 1 --2- 172.21.15.44:0/1965462647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806cf90 0x7f2fa400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.697+0000 7f2fb6ffd640 1 -- 172.21.15.44:0/1965462647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806cf90 msgr2=0x7f2fa400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.697+0000 7f2fb6ffd640 1 --2- 172.21.15.44:0/1965462647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806cf90 0x7f2fa400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.697+0000 7f2fb6ffd640 1 -- 172.21.15.44:0/1965462647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fa4004800 msgr2=0x7f2fa4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:22.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.697+0000 7f2fb6ffd640 1 --2- 172.21.15.44:0/1965462647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fa4004800 0x7f2fa4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.697+0000 7f2fb6ffd640 1 -- 172.21.15.44:0/1965462647 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa4003520 con 0x7f2fa40039e0 2024-04-02T13:31:22.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.697+0000 7f2fb77fe640 1 --2- 172.21.15.44:0/1965462647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806cf90 0x7f2fa400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:22.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.697+0000 7f2fb6ffd640 1 --2- 172.21.15.44:0/1965462647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa40039e0 0x7f2fa4003e00 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f2fb000c210 tx=0x7f2fa4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7656dfff29be3ffd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:22.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.697+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0007d30 con 0x7f2fa40039e0 2024-04-02T13:31:22.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.697+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb0030070 con 0x7f2fa40039e0 2024-04-02T13:31:22.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.697+0000 7f2fb5ffb640 1 -- 172.21.15.44:0/1965462647 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb002cd60 con 0x7f2fa40039e0 2024-04-02T13:31:22.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.698+0000 7f2fbd655640 1 -- 172.21.15.44:0/1965462647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa40039e0 msgr2=0x7f2fa4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.698+0000 7f2fbd655640 1 --2- 172.21.15.44:0/1965462647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa40039e0 0x7f2fa4003e00 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f2fb000c210 tx=0x7f2fa4005830 comp rx=0 tx=0).stop 2024-04-02T13:31:22.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.698+0000 7f2fb77fe640 1 -- 172.21.15.44:0/1965462647 reap_dead start 2024-04-02T13:31:22.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.698+0000 7f2fbd655640 1 -- 172.21.15.44:0/1965462647 shutdown_connections 2024-04-02T13:31:22.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.698+0000 7f2fbd655640 1 -- 172.21.15.44:0/1965462647 >> 172.21.15.44:0/1965462647 conn(0x7f2fb809d470 msgr2=0x7f2fb80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:22.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.698+0000 7f2fbd655640 1 -- 172.21.15.44:0/1965462647 shutdown_connections 2024-04-02T13:31:22.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.698+0000 7f2fbd655640 1 -- 172.21.15.44:0/1965462647 wait complete. 2024-04-02T13:31:22.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.699+0000 7f2fbd655640 1 Processor -- start 2024-04-02T13:31:22.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.699+0000 7f2fbd655640 1 -- start start 2024-04-02T13:31:22.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fbd655640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa40039e0 0x7f2fb8158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fbd655640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb80b4360 0x7f2fb80b6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fbd655640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb80b6860 0x7f2fb80ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fbd655640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb80d4da0 con 0x7f2fb80b6860 2024-04-02T13:31:22.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fbd655640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb806d3f0 con 0x7f2fb80b4360 2024-04-02T13:31:22.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fbd655640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb80d4f20 con 0x7f2fa40039e0 2024-04-02T13:31:22.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fb77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb80b6860 0x7f2fb80ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fb77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb80b6860 0x7f2fb80ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53918/0 (socket says 172.21.15.44:53918) 2024-04-02T13:31:22.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fb77fe640 1 -- 172.21.15.44:0/2298373049 learned_addr learned my addr 172.21.15.44:0/2298373049 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:22.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fb6ffd640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa40039e0 0x7f2fb8158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fb67fc640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb80b4360 0x7f2fb80b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fb77fe640 1 -- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa40039e0 msgr2=0x7f2fb8158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fb77fe640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa40039e0 0x7f2fb8158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fb77fe640 1 -- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb80b4360 msgr2=0x7f2fb80b6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fb77fe640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb80b4360 0x7f2fb80b6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.700+0000 7f2fb77fe640 1 -- 172.21.15.44:0/2298373049 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb8159340 con 0x7f2fb80b6860 2024-04-02T13:31:22.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.701+0000 7f2fb77fe640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb80b6860 0x7f2fb80ccc70 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f2fb809d9a0 tx=0x7f2fa800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:22.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.701+0000 7f2faf7fe640 1 -- 172.21.15.44:0/2298373049 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8006b50 con 0x7f2fb80b6860 2024-04-02T13:31:22.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.701+0000 7f2faf7fe640 1 -- 172.21.15.44:0/2298373049 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa8006d10 con 0x7f2fb80b6860 2024-04-02T13:31:22.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.701+0000 7f2faf7fe640 1 -- 172.21.15.44:0/2298373049 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8003970 con 0x7f2fb80b6860 2024-04-02T13:31:22.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.701+0000 7f2fbd655640 1 -- 172.21.15.44:0/2298373049 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb80b54b0 con 0x7f2fb80b6860 2024-04-02T13:31:22.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.701+0000 7f2fbd655640 1 -- 172.21.15.44:0/2298373049 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb80b5750 con 0x7f2fb80b6860 2024-04-02T13:31:22.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.702+0000 7f2fbd655640 1 -- 172.21.15.44:0/2298373049 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb80684f0 con 0x7f2fb80b6860 2024-04-02T13:31:22.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.702+0000 7f2faf7fe640 1 -- 172.21.15.44:0/2298373049 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2fa801a020 con 0x7f2fb80b6860 2024-04-02T13:31:22.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.703+0000 7f2faf7fe640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f9806c8f0 0x7f2f9806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:22.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.703+0000 7f2faf7fe640 1 -- 172.21.15.44:0/2298373049 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f2fa808b0f0 con 0x7f2fb80b6860 2024-04-02T13:31:22.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.703+0000 7f2fb6ffd640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f9806c8f0 0x7f2f9806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:22.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.703+0000 7f2fb6ffd640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f9806c8f0 0x7f2f9806edb0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0034000 tx=0x7f2fb0009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:22.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.707+0000 7f2faf7fe640 1 -- 172.21.15.44:0/2298373049 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2fb8159340 con 0x7f2fb80b6860 2024-04-02T13:31:22.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.905+0000 7f2fbd655640 1 -- 172.21.15.44:0/2298373049 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2fb80d17c0 con 0x7f2fb80b6860 2024-04-02T13:31:22.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.905+0000 7f2faf7fe640 1 -- 172.21.15.44:0/2298373049 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v121) ==== 80+0+0 (secure 0 0 0) 0x7f2fa8058dd0 con 0x7f2fb80b6860 2024-04-02T13:31:22.909 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:31:22.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.907+0000 7f2fbd655640 1 -- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f9806c8f0 msgr2=0x7f2f9806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.907+0000 7f2fbd655640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f9806c8f0 0x7f2f9806edb0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0034000 tx=0x7f2fb0009030 comp rx=0 tx=0).stop 2024-04-02T13:31:22.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.907+0000 7f2fbd655640 1 -- 172.21.15.44:0/2298373049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb80b6860 msgr2=0x7f2fb80ccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:22.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.907+0000 7f2fbd655640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb80b6860 0x7f2fb80ccc70 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f2fb809d9a0 tx=0x7f2fa800eb90 comp rx=0 tx=0).stop 2024-04-02T13:31:22.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.907+0000 7f2fbd655640 1 -- 172.21.15.44:0/2298373049 shutdown_connections 2024-04-02T13:31:22.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.907+0000 7f2fbd655640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2f9806c8f0 0x7f2f9806edb0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.907+0000 7f2fbd655640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb80b6860 0x7f2fb80ccc70 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.907+0000 7f2fbd655640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb80b4360 0x7f2fb80b6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.907+0000 7f2fbd655640 1 --2- 172.21.15.44:0/2298373049 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fa40039e0 0x7f2fb8158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:22.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.907+0000 7f2fbd655640 1 -- 172.21.15.44:0/2298373049 >> 172.21.15.44:0/2298373049 conn(0x7f2fb809d470 msgr2=0x7f2fb80b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:22.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.907+0000 7f2fbd655640 1 -- 172.21.15.44:0/2298373049 shutdown_connections 2024-04-02T13:31:22.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:22.907+0000 7f2fbd655640 1 -- 172.21.15.44:0/2298373049 wait complete. 2024-04-02T13:31:22.921 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:31:22.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:22.922 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:22.947 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[81616]: fuse finished with error 0 and tester_r 0 2024-04-02T13:31:22.949 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:31:23.893 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:31:23.896 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:31:29.050 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:31:29.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:29.051 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:31:29.069 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 flush journal 2024-04-02T13:31:29.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.220+0000 7f0b0e3c8640 1 Processor -- start 2024-04-02T13:31:29.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.220+0000 7f0b0e3c8640 1 -- start start 2024-04-02T13:31:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.220+0000 7f0b0e3c8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0b081565d0 0x7f0b080d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.220+0000 7f0b0e3c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b080d05e0 0x7f0b080ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.220+0000 7f0b0e3c8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0b080cf400 0x7f0b080cc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.220+0000 7f0b0e3c8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0b080cf890 con 0x7f0b080cf400 2024-04-02T13:31:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.221+0000 7f0b0e3c8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0b080cfad0 con 0x7f0b081565d0 2024-04-02T13:31:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.221+0000 7f0b0e3c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0b08160b10 con 0x7f0b080d05e0 2024-04-02T13:31:29.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.221+0000 7f0b0c965640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0b080cf400 0x7f0b080cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.221+0000 7f0b0c965640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0b080cf400 0x7f0b080cc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46112/0 (socket says 172.21.15.44:46112) 2024-04-02T13:31:29.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.221+0000 7f0b0c965640 1 -- 172.21.15.44:0/3833161090 learned_addr learned my addr 172.21.15.44:0/3833161090 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:29.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.221+0000 7f0b07fff640 1 --2- 172.21.15.44:0/3833161090 >> v2:172.21.15.69:3300/0 conn(0x7f0b081565d0 0x7f0b080d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.221+0000 7f0b077fe640 1 --2- 172.21.15.44:0/3833161090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b080d05e0 0x7f0b080ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.221+0000 7f0b0c965640 1 -- 172.21.15.44:0/3833161090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b080d05e0 msgr2=0x7f0b080ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:29.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.221+0000 7f0b0c965640 1 --2- 172.21.15.44:0/3833161090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b080d05e0 0x7f0b080ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.221+0000 7f0b0c965640 1 -- 172.21.15.44:0/3833161090 >> v2:172.21.15.69:3300/0 conn(0x7f0b081565d0 msgr2=0x7f0b080d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:29.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.221+0000 7f0b0c965640 1 --2- 172.21.15.44:0/3833161090 >> v2:172.21.15.69:3300/0 conn(0x7f0b081565d0 0x7f0b080d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.221+0000 7f0b0c965640 1 -- 172.21.15.44:0/3833161090 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b0815fd50 con 0x7f0b080cf400 2024-04-02T13:31:29.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.222+0000 7f0b0c965640 1 --2- 172.21.15.44:0/3833161090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b080cf400 0x7f0b080cc660 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f0b080d0cd0 tx=0x7f0b0002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6aeb55cd5afb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:29.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.222+0000 7f0b06ffd640 1 -- 172.21.15.44:0/3833161090 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0002c9e0 con 0x7f0b080cf400 2024-04-02T13:31:29.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.222+0000 7f0b06ffd640 1 -- 172.21.15.44:0/3833161090 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b0002cb80 con 0x7f0b080cf400 2024-04-02T13:31:29.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.222+0000 7f0b06ffd640 1 -- 172.21.15.44:0/3833161090 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b000343f0 con 0x7f0b080cf400 2024-04-02T13:31:29.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.222+0000 7f0b0e3c8640 1 -- 172.21.15.44:0/3833161090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b080cf400 msgr2=0x7f0b080cc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.222+0000 7f0b0e3c8640 1 --2- 172.21.15.44:0/3833161090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b080cf400 0x7f0b080cc660 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f0b080d0cd0 tx=0x7f0b0002be40 comp rx=0 tx=0).stop 2024-04-02T13:31:29.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.223+0000 7f0b0e3c8640 1 -- 172.21.15.44:0/3833161090 shutdown_connections 2024-04-02T13:31:29.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.223+0000 7f0b0e3c8640 1 --2- 172.21.15.44:0/3833161090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b080cf400 0x7f0b080cc660 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.223+0000 7f0b0e3c8640 1 --2- 172.21.15.44:0/3833161090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b080d05e0 0x7f0b080ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.223+0000 7f0b0e3c8640 1 --2- 172.21.15.44:0/3833161090 >> v2:172.21.15.69:3300/0 conn(0x7f0b081565d0 0x7f0b080d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.223+0000 7f0b0e3c8640 1 -- 172.21.15.44:0/3833161090 >> 172.21.15.44:0/3833161090 conn(0x7f0b0806b920 msgr2=0x7f0b0806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:29.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.223+0000 7f0b0e3c8640 1 -- 172.21.15.44:0/3833161090 shutdown_connections 2024-04-02T13:31:29.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.223+0000 7f0b0e3c8640 1 -- 172.21.15.44:0/3833161090 wait complete. 2024-04-02T13:31:29.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.224+0000 7f0b0e3c8640 1 Processor -- start 2024-04-02T13:31:29.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.224+0000 7f0b0e3c8640 1 -- start start 2024-04-02T13:31:29.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b0e3c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b080cf400 0x7f0b08145760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b0e3c8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b080d05e0 0x7f0b081593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b0e3c8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b081565d0 0x7f0b08154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b0e3c8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0b081605f0 con 0x7f0b080cf400 2024-04-02T13:31:29.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b0e3c8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0b08160310 con 0x7f0b080d05e0 2024-04-02T13:31:29.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b0e3c8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0b081569b0 con 0x7f0b081565d0 2024-04-02T13:31:29.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b07fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b080cf400 0x7f0b08145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b07fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b080cf400 0x7f0b08145760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46122/0 (socket says 172.21.15.44:46122) 2024-04-02T13:31:29.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b07fff640 1 -- 172.21.15.44:0/2604857839 learned_addr learned my addr 172.21.15.44:0/2604857839 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:29.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b0c965640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b081565d0 0x7f0b08154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b077fe640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b080d05e0 0x7f0b081593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b07fff640 1 -- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b081565d0 msgr2=0x7f0b08154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b07fff640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b081565d0 0x7f0b08154480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b07fff640 1 -- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b080d05e0 msgr2=0x7f0b081593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b07fff640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b080d05e0 0x7f0b081593c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b07fff640 1 -- 172.21.15.44:0/2604857839 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b0809b6a0 con 0x7f0b080cf400 2024-04-02T13:31:29.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b07fff640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b080cf400 0x7f0b08145760 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f0af800d5f0 tx=0x7f0af800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:29.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.225+0000 7f0b04ff9640 1 -- 172.21.15.44:0/2604857839 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af80125f0 con 0x7f0b080cf400 2024-04-02T13:31:29.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.226+0000 7f0b04ff9640 1 -- 172.21.15.44:0/2604857839 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0af8019070 con 0x7f0b080cf400 2024-04-02T13:31:29.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.226+0000 7f0b0e3c8640 1 -- 172.21.15.44:0/2604857839 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b0809b980 con 0x7f0b080cf400 2024-04-02T13:31:29.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.226+0000 7f0b04ff9640 1 -- 172.21.15.44:0/2604857839 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af80129d0 con 0x7f0b080cf400 2024-04-02T13:31:29.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.226+0000 7f0b0e3c8640 1 -- 172.21.15.44:0/2604857839 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b08158ea0 con 0x7f0b080cf400 2024-04-02T13:31:29.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.227+0000 7f0b04ff9640 1 -- 172.21.15.44:0/2604857839 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0af8012b70 con 0x7f0b080cf400 2024-04-02T13:31:29.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.227+0000 7f0b04ff9640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0af406c8f0 0x7f0af406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.227+0000 7f0b04ff9640 1 -- 172.21.15.44:0/2604857839 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f0af808c180 con 0x7f0b080cf400 2024-04-02T13:31:29.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.228+0000 7f0b077fe640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0af406c8f0 0x7f0af406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.228+0000 7f0b10fae740 1 Processor -- start 2024-04-02T13:31:29.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.228+0000 7f0b10fae740 1 -- start start 2024-04-02T13:31:29.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b077fe640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0af406c8f0 0x7f0af406edb0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f0afc006fd0 tx=0x7f0afc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:29.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b10fae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b59520 0x55bb09b570e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b10fae740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 0x55bb09b10a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b10fae740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b10f90 0x55bb09b3b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b10fae740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bb09a75830 con 0x55bb09b10f90 2024-04-02T13:31:29.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b10fae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bb09a1d820 con 0x55bb09b59520 2024-04-02T13:31:29.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b10fae740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55bb09a717e0 con 0x55bb09b38d60 2024-04-02T13:31:29.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b0c965640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b10f90 0x55bb09b3b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b0c965640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b10f90 0x55bb09b3b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46136/0 (socket says 172.21.15.44:46136) 2024-04-02T13:31:29.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b0c965640 1 -- 172.21.15.44:0/3154232841 learned_addr learned my addr 172.21.15.44:0/3154232841 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:29.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b07fff640 1 --2- 172.21.15.44:0/3154232841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b59520 0x55bb09b570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b077fe640 1 --2- 172.21.15.44:0/3154232841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 0x55bb09b10a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b0c965640 1 -- 172.21.15.44:0/3154232841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 msgr2=0x55bb09b10a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b0c965640 1 --2- 172.21.15.44:0/3154232841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 0x55bb09b10a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b0c965640 1 -- 172.21.15.44:0/3154232841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b59520 msgr2=0x55bb09b570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b0c965640 1 --2- 172.21.15.44:0/3154232841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b59520 0x55bb09b570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.229+0000 7f0b0c965640 1 -- 172.21.15.44:0/3154232841 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb098537d0 con 0x55bb09b10f90 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.230+0000 7f0b0c965640 1 --2- 172.21.15.44:0/3154232841 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b10f90 0x55bb09b3b730 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f0b0002c450 tx=0x7f0b0002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec0186330056c3d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.230+0000 7f0aedffb640 1 -- 172.21.15.44:0/3154232841 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b00045070 con 0x55bb09b10f90 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.230+0000 7f0aedffb640 1 -- 172.21.15.44:0/3154232841 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b0002c9e0 con 0x55bb09b10f90 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.230+0000 7f0aedffb640 1 -- 172.21.15.44:0/3154232841 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b00009510 con 0x55bb09b10f90 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.230+0000 7f0b10fae740 1 -- 172.21.15.44:0/3154232841 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b10f90 msgr2=0x55bb09b3b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.230+0000 7f0b10fae740 1 --2- 172.21.15.44:0/3154232841 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b10f90 0x55bb09b3b730 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f0b0002c450 tx=0x7f0b0002e000 comp rx=0 tx=0).stop 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.230+0000 7f0b10fae740 1 -- 172.21.15.44:0/3154232841 shutdown_connections 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.230+0000 7f0b10fae740 1 --2- 172.21.15.44:0/3154232841 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b10f90 0x55bb09b3b730 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.230+0000 7f0b10fae740 1 --2- 172.21.15.44:0/3154232841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 0x55bb09b10a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.230+0000 7f0b10fae740 1 --2- 172.21.15.44:0/3154232841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b59520 0x55bb09b570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.230+0000 7f0b10fae740 1 -- 172.21.15.44:0/3154232841 >> 172.21.15.44:0/3154232841 conn(0x55bb09b0dbb0 msgr2=0x55bb09b56730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:29.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.230+0000 7f0b10fae740 1 -- 172.21.15.44:0/3154232841 shutdown_connections 2024-04-02T13:31:29.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.230+0000 7f0b10fae740 1 -- 172.21.15.44:0/3154232841 wait complete. 2024-04-02T13:31:29.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.231+0000 7f0b10fae740 1 -- start start 2024-04-02T13:31:29.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.231+0000 7f0b10fae740 1 Processor -- start 2024-04-02T13:31:29.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b10fae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b38d60 0x55bb09b53160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b10fae740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b59520 0x55bb09b503b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b10fae740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b25780 0x55bb09b25c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b10fae740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bb09b0feb0 con 0x55bb09b25780 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b10fae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bb09a30d30 con 0x55bb09b38d60 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b10fae740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55bb09b0c1e0 con 0x55bb09b59520 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b10fae740 10 client.0 _ll_register_callbacks cb 0x55bb09a30720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b0c965640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b25780 0x55bb09b25c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b0c965640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b25780 0x55bb09b25c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46148/0 (socket says 172.21.15.44:46148) 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b07fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b38d60 0x55bb09b53160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b0c965640 1 -- 172.21.15.44:0/1697159482 learned_addr learned my addr 172.21.15.44:0/1697159482 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b077fe640 1 --2- 172.21.15.44:0/1697159482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b59520 0x55bb09b503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b0c965640 1 -- 172.21.15.44:0/1697159482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b59520 msgr2=0x55bb09b503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b0c965640 1 --2- 172.21.15.44:0/1697159482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b59520 0x55bb09b503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b0c965640 1 -- 172.21.15.44:0/1697159482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b38d60 msgr2=0x55bb09b53160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:29.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b0c965640 1 --2- 172.21.15.44:0/1697159482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b38d60 0x55bb09b53160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.232+0000 7f0b0c965640 1 -- 172.21.15.44:0/1697159482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb097893f0 con 0x55bb09b25780 2024-04-02T13:31:29.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0b0c965640 1 --2- 172.21.15.44:0/1697159482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b25780 0x55bb09b25c00 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f0b00043040 tx=0x7f0b00004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1501b53df371f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:29.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0ae6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:31:29.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0ae6ffd640 1 -- 172.21.15.44:0/1697159482 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b00045070 con 0x55bb09b25780 2024-04-02T13:31:29.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0ae6ffd640 1 -- 172.21.15.44:0/1697159482 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f0b0004cc60 con 0x55bb09b25780 2024-04-02T13:31:29.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0b10fae740 1 -- 172.21.15.44:0/1697159482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b0002c9e0 con 0x55bb09b25780 2024-04-02T13:31:29.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0ae6ffd640 1 -- 172.21.15.44:0/1697159482 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f0b00053050 con 0x55bb09b25780 2024-04-02T13:31:29.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0b10fae740 10 client.9747 fetch_fsmap learned FSMap version 99 2024-04-02T13:31:29.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0b10fae740 1 -- 172.21.15.44:0/1697159482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x55bb097490f0 con 0x55bb09b25780 2024-04-02T13:31:29.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0ae6ffd640 1 -- 172.21.15.44:0/1697159482 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b0002c9e0 con 0x55bb09b25780 2024-04-02T13:31:29.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0ae6ffd640 1 -- 172.21.15.44:0/1697159482 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 99) ==== 3494+0+0 (secure 0 0 0) 0x7f0b00056020 con 0x55bb09b25780 2024-04-02T13:31:29.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0b10fae740 10 client.9747 fetch_fsmap finished waiting for FSMap version 99 2024-04-02T13:31:29.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0b10fae740 10 client.9747 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.a 2024-04-02T13:31:29.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0b10fae740 20 client.9747 populate_metadata read hostname 'smithi044' 2024-04-02T13:31:29.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0b10fae740 1 --2- 172.21.15.44:0/1697159482 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55bb09b4e050 0x55bb09b4e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0b10fae740 4 client.9747 mds_command: new command op to 9513 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:31:29.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0b10fae740 1 -- 172.21.15.44:0/1697159482 --> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb09a82d70 con 0x55bb09b4e050 2024-04-02T13:31:29.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.233+0000 7f0b07fff640 1 --2- 172.21.15.44:0/1697159482 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55bb09b4e050 0x55bb09b4e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.234+0000 7f0ae6ffd640 10 client.9747 ms_handle_connect on v2:172.21.15.44:6833/2109578276 2024-04-02T13:31:29.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.234+0000 7f0b07fff640 1 --2- 172.21.15.44:0/1697159482 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55bb09b4e050 0x55bb09b4e4c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5c5f0fd158c8be6 server_cookie=50eeb8168d36139e in_seq=0 out_seq=0 2024-04-02T13:31:29.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.234+0000 7f0ae6ffd640 1 -- 172.21.15.44:0/1697159482 <== mds.0 v2:172.21.15.44:6833/2109578276 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb09a82d70 con 0x55bb09b4e050 2024-04-02T13:31:29.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.234+0000 7f0ae6ffd640 10 client.9747 handle_command_reply: tid=0 2024-04-02T13:31:29.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.234+0000 7f0b10fae740 1 client.9747 shutdown 2024-04-02T13:31:29.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.234+0000 7f0b10fae740 1 -- 172.21.15.44:0/1697159482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b25780 msgr2=0x55bb09b25c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.234+0000 7f0b10fae740 1 --2- 172.21.15.44:0/1697159482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b25780 0x55bb09b25c00 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f0b00043040 tx=0x7f0b00004ab0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.235+0000 7f0b10fae740 1 -- 172.21.15.44:0/1697159482 shutdown_connections 2024-04-02T13:31:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.235+0000 7f0b10fae740 1 --2- 172.21.15.44:0/1697159482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b25780 0x55bb09b25c00 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.235+0000 7f0b10fae740 1 --2- 172.21.15.44:0/1697159482 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55bb09b4e050 0x55bb09b4e4c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.235+0000 7f0b10fae740 1 --2- 172.21.15.44:0/1697159482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b59520 0x55bb09b503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.235+0000 7f0b10fae740 1 --2- 172.21.15.44:0/1697159482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b38d60 0x55bb09b53160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.235+0000 7f0b10fae740 1 -- 172.21.15.44:0/1697159482 >> 172.21.15.44:0/1697159482 conn(0x55bb09b10f90 msgr2=0x55bb09b0a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.235+0000 7f0ae6ffd640 0 client.9747 ms_handle_reset on v2:172.21.15.44:6833/2109578276 2024-04-02T13:31:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.235+0000 7f0b10fae740 1 -- 172.21.15.44:0/1697159482 shutdown_connections 2024-04-02T13:31:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.235+0000 7f0b10fae740 1 -- 172.21.15.44:0/1697159482 wait complete. 2024-04-02T13:31:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.235+0000 7f0b10fae740 20 client.9747 trim_cache size 0 max 16384 2024-04-02T13:31:29.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.250+0000 7f0b10fae740 1 Processor -- start 2024-04-02T13:31:29.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.250+0000 7f0b10fae740 1 -- start start 2024-04-02T13:31:29.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.250+0000 7f0b10fae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b25780 0x55bb09b56650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.250+0000 7f0b10fae740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 0x55bb09b56b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.250+0000 7f0b10fae740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 0x55bb09b0b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.250+0000 7f0b10fae740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0b081fc950 con 0x55bb09b59520 2024-04-02T13:31:29.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.250+0000 7f0b10fae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0806e410 con 0x55bb09b25780 2024-04-02T13:31:29.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.250+0000 7f0b10fae740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0b0815d0c0 con 0x55bb09b38d60 2024-04-02T13:31:29.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.250+0000 7f0b0c965640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 0x55bb09b0b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.250+0000 7f0b0c965640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 0x55bb09b0b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46160/0 (socket says 172.21.15.44:46160) 2024-04-02T13:31:29.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.250+0000 7f0b0c965640 1 -- 172.21.15.44:0/2031841070 learned_addr learned my addr 172.21.15.44:0/2031841070 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:29.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.251+0000 7f0b07fff640 1 --2- 172.21.15.44:0/2031841070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b25780 0x55bb09b56650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.251+0000 7f0b0c965640 1 -- 172.21.15.44:0/2031841070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 msgr2=0x55bb09b56b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:29.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.251+0000 7f0b0c965640 1 --2- 172.21.15.44:0/2031841070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 0x55bb09b56b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.251+0000 7f0b0c965640 1 -- 172.21.15.44:0/2031841070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b25780 msgr2=0x55bb09b56650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:29.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.251+0000 7f0b0c965640 1 --2- 172.21.15.44:0/2031841070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b25780 0x55bb09b56650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.251+0000 7f0b0c965640 1 -- 172.21.15.44:0/2031841070 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb09b55b20 con 0x55bb09b59520 2024-04-02T13:31:29.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.251+0000 7f0b0c965640 1 --2- 172.21.15.44:0/2031841070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 0x55bb09b0b9c0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f0b000094c0 tx=0x7f0b00045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04063376565dd1e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:29.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.251+0000 7f0aed7fa640 1 -- 172.21.15.44:0/2031841070 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0005c070 con 0x55bb09b59520 2024-04-02T13:31:29.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.251+0000 7f0aed7fa640 1 -- 172.21.15.44:0/2031841070 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b000044f0 con 0x55bb09b59520 2024-04-02T13:31:29.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.251+0000 7f0aed7fa640 1 -- 172.21.15.44:0/2031841070 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0002cb90 con 0x55bb09b59520 2024-04-02T13:31:29.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.251+0000 7f0b10fae740 1 -- 172.21.15.44:0/2031841070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 msgr2=0x55bb09b0b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.251+0000 7f0b10fae740 1 --2- 172.21.15.44:0/2031841070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 0x55bb09b0b9c0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f0b000094c0 tx=0x7f0b00045000 comp rx=0 tx=0).stop 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.252+0000 7f0b10fae740 1 -- 172.21.15.44:0/2031841070 shutdown_connections 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.252+0000 7f0b10fae740 1 --2- 172.21.15.44:0/2031841070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 0x55bb09b0b9c0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.252+0000 7f0b10fae740 1 --2- 172.21.15.44:0/2031841070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 0x55bb09b56b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.252+0000 7f0b10fae740 1 --2- 172.21.15.44:0/2031841070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b25780 0x55bb09b56650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.252+0000 7f0b10fae740 1 -- 172.21.15.44:0/2031841070 >> 172.21.15.44:0/2031841070 conn(0x55bb09b10f90 msgr2=0x55bb09b4f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.252+0000 7f0b10fae740 1 -- 172.21.15.44:0/2031841070 shutdown_connections 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.252+0000 7f0b10fae740 1 -- 172.21.15.44:0/2031841070 wait complete. 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.252+0000 7f0b10fae740 1 -- start start 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.252+0000 7f0b10fae740 1 Processor -- start 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b10fae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b25780 0x55bb09b4c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b10fae740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 0x55bb09bd38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b10fae740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 0x55bb09b57940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b10fae740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0b08157a00 con 0x55bb09b59520 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b10fae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bb09b7ddd0 con 0x55bb09b25780 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b10fae740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55bb09b58740 con 0x55bb09b38d60 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b10fae740 10 client.0 _ll_register_callbacks cb 0x7f0b000092e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b0c965640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 0x55bb09b57940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b0c965640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 0x55bb09b57940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46172/0 (socket says 172.21.15.44:46172) 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b0c965640 1 -- 172.21.15.44:0/411712785 learned_addr learned my addr 172.21.15.44:0/411712785 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b07fff640 1 --2- 172.21.15.44:0/411712785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b25780 0x55bb09b4c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b077fe640 1 --2- 172.21.15.44:0/411712785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 0x55bb09bd38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b0c965640 1 -- 172.21.15.44:0/411712785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 msgr2=0x55bb09bd38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b0c965640 1 --2- 172.21.15.44:0/411712785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 0x55bb09bd38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b0c965640 1 -- 172.21.15.44:0/411712785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b25780 msgr2=0x55bb09b4c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b0c965640 1 --2- 172.21.15.44:0/411712785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b25780 0x55bb09b4c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.253+0000 7f0b0c965640 1 -- 172.21.15.44:0/411712785 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb09b38990 con 0x55bb09b59520 2024-04-02T13:31:29.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0b0c965640 1 --2- 172.21.15.44:0/411712785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 0x55bb09b57940 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f0b00040040 tx=0x7f0b00030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a09c4246632b298 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0ae6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0ae6ffd640 1 -- 172.21.15.44:0/411712785 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0005c070 con 0x55bb09b59520 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0ae6ffd640 1 -- 172.21.15.44:0/411712785 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f0b00040670 con 0x55bb09b59520 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0b10fae740 1 -- 172.21.15.44:0/411712785 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b000044f0 con 0x55bb09b59520 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0ae6ffd640 1 -- 172.21.15.44:0/411712785 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f0b00066050 con 0x55bb09b59520 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0b10fae740 10 client.9753 fetch_fsmap learned FSMap version 99 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0b10fae740 1 -- 172.21.15.44:0/411712785 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x55bb09b58480 con 0x55bb09b59520 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0ae6ffd640 1 -- 172.21.15.44:0/411712785 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b000044f0 con 0x55bb09b59520 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0ae6ffd640 1 -- 172.21.15.44:0/411712785 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 99) ==== 3494+0+0 (secure 0 0 0) 0x7f0b00069020 con 0x55bb09b59520 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0b10fae740 10 client.9753 fetch_fsmap finished waiting for FSMap version 99 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0b10fae740 10 client.9753 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.a 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0b10fae740 20 client.9753 populate_metadata read hostname 'smithi044' 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0b10fae740 1 --2- 172.21.15.44:0/411712785 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55bb09b4cec0 0x55bb09b57e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0b10fae740 4 client.9753 mds_command: new command op to 9513 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T13:31:29.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0b10fae740 1 -- 172.21.15.44:0/411712785 --> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bb09a7d0c0 con 0x55bb09b4cec0 2024-04-02T13:31:29.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.254+0000 7f0b07fff640 1 --2- 172.21.15.44:0/411712785 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55bb09b4cec0 0x55bb09b57e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.255+0000 7f0b07fff640 1 --2- 172.21.15.44:0/411712785 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55bb09b4cec0 0x55bb09b57e80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=debabcec70ab16d1 server_cookie=68a1c12504997744 in_seq=0 out_seq=0 2024-04-02T13:31:29.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.255+0000 7f0ae6ffd640 10 client.9753 ms_handle_connect on v2:172.21.15.44:6833/2109578276 2024-04-02T13:31:29.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0ae6ffd640 1 -- 172.21.15.44:0/411712785 <== mds.0 v2:172.21.15.44:6833/2109578276 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bb09a7d0c0 con 0x55bb09b4cec0 2024-04-02T13:31:29.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0ae6ffd640 10 client.9753 handle_command_reply: tid=0 2024-04-02T13:31:29.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0b10fae740 1 client.9753 shutdown 2024-04-02T13:31:29.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0b10fae740 1 -- 172.21.15.44:0/411712785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 msgr2=0x55bb09b57940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0b10fae740 1 --2- 172.21.15.44:0/411712785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 0x55bb09b57940 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f0b00040040 tx=0x7f0b00030000 comp rx=0 tx=0).stop 2024-04-02T13:31:29.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0b10fae740 1 -- 172.21.15.44:0/411712785 shutdown_connections 2024-04-02T13:31:29.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0b10fae740 1 --2- 172.21.15.44:0/411712785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bb09b59520 0x55bb09b57940 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0b10fae740 1 --2- 172.21.15.44:0/411712785 >> [v2:172.21.15.44:6833/2109578276,v1:172.21.15.44:6835/2109578276] conn(0x55bb09b4cec0 0x55bb09b57e80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0b10fae740 1 --2- 172.21.15.44:0/411712785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55bb09b38d60 0x55bb09bd38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0b10fae740 1 --2- 172.21.15.44:0/411712785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb09b25780 0x55bb09b4c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0b10fae740 1 -- 172.21.15.44:0/411712785 >> 172.21.15.44:0/411712785 conn(0x55bb09b10f90 msgr2=0x55bb09b53290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:29.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0ae6ffd640 0 client.9753 ms_handle_reset on v2:172.21.15.44:6833/2109578276 2024-04-02T13:31:29.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0b10fae740 1 -- 172.21.15.44:0/411712785 shutdown_connections 2024-04-02T13:31:29.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0b10fae740 1 -- 172.21.15.44:0/411712785 wait complete. 2024-04-02T13:31:29.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.263+0000 7f0b10fae740 20 client.9753 trim_cache size 0 max 16384 2024-04-02T13:31:29.268 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:31:29.268 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "", 2024-04-02T13:31:29.268 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0 2024-04-02T13:31:29.268 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:31:29.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.264+0000 7f0b0e3c8640 1 -- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0af406c8f0 msgr2=0x7f0af406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.264+0000 7f0b0e3c8640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0af406c8f0 0x7f0af406edb0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f0afc006fd0 tx=0x7f0afc008040 comp rx=0 tx=0).stop 2024-04-02T13:31:29.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.264+0000 7f0b0e3c8640 1 -- 172.21.15.44:0/2604857839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b080cf400 msgr2=0x7f0b08145760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.264+0000 7f0b0e3c8640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b080cf400 0x7f0b08145760 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f0af800d5f0 tx=0x7f0af800dab0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.264+0000 7f0b0e3c8640 1 -- 172.21.15.44:0/2604857839 shutdown_connections 2024-04-02T13:31:29.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.264+0000 7f0b0e3c8640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0af406c8f0 0x7f0af406edb0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.264+0000 7f0b0e3c8640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b081565d0 0x7f0b08154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.264+0000 7f0b0e3c8640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b080d05e0 0x7f0b081593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.264+0000 7f0b0e3c8640 1 --2- 172.21.15.44:0/2604857839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b080cf400 0x7f0b08145760 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.264+0000 7f0b0e3c8640 1 -- 172.21.15.44:0/2604857839 >> 172.21.15.44:0/2604857839 conn(0x7f0b0806b920 msgr2=0x7f0b08071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:29.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.264+0000 7f0b0e3c8640 1 -- 172.21.15.44:0/2604857839 shutdown_connections 2024-04-02T13:31:29.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.264+0000 7f0b0e3c8640 1 -- 172.21.15.44:0/2604857839 wait complete. 2024-04-02T13:31:29.278 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:31:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.376+0000 7f7fe5201640 1 Processor -- start 2024-04-02T13:31:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.376+0000 7f7fe5201640 1 -- start start 2024-04-02T13:31:29.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.377+0000 7f7fe5201640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7fe00d2c10 0x7f7fe0168f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.377+0000 7f7fe5201640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fe00b3880 con 0x7f7fe00d1290 2024-04-02T13:31:29.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.377+0000 7f7fe5201640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fe00b3b60 con 0x7f7fe015e2a0 2024-04-02T13:31:29.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.377+0000 7f7fe5201640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7fe00b4490 con 0x7f7fe00d2c10 2024-04-02T13:31:29.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.377+0000 7f7fde59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7fe00d1290 0x7f7fe00d2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51478/0 (socket says 172.21.15.44:51478) 2024-04-02T13:31:29.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.377+0000 7f7fde59c640 1 -- 172.21.15.44:0/4219376577 learned_addr learned my addr 172.21.15.44:0/4219376577 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:29.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2478056948 0 0) 0x7f7fe00b3880 con 0x7f7fe00d1290 2024-04-02T13:31:29.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fcc003520 con 0x7f7fe00d1290 2024-04-02T13:31:29.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fdf59e640 1 --2- 172.21.15.44:0/4219376577 >> v2:172.21.15.69:3300/0 conn(0x7f7fe00d2c10 0x7f7fe0168f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2940662945 0 0) 0x7f7fcc003520 con 0x7f7fe00d1290 2024-04-02T13:31:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 >> v2:172.21.15.69:3300/0 conn(0x7f7fe00d2c10 msgr2=0x7f7fe0168f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 --2- 172.21.15.44:0/4219376577 >> v2:172.21.15.69:3300/0 conn(0x7f7fe00d2c10 0x7f7fe0168f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 >> v1:172.21.15.69:6789/0 conn(0x7f7fe015e2a0 legacy=0x7f7fe00d4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe00b3e40 con 0x7f7fe00d1290 2024-04-02T13:31:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f7fc8001670 con 0x7f7fe00d1290 2024-04-02T13:31:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 >> v1:172.21.15.44:6789/0 conn(0x7f7fe00d1290 legacy=0x7f7fe00d2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 --2- 172.21.15.44:0/4219376577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc003a00 0x7f7fcc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 --2- 172.21.15.44:0/4219376577 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fcc004800 0x7f7fcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 --2- 172.21.15.44:0/4219376577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc0051c0 0x7f7fcc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe00b4490 con 0x7f7fcc0051c0 2024-04-02T13:31:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7fcc00c0c0 con 0x7f7fcc004800 2024-04-02T13:31:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7fcc00c2c0 con 0x7f7fcc003a00 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fdf59e640 1 --2- 172.21.15.44:0/4219376577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc0051c0 0x7f7fcc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fde59c640 1 --2- 172.21.15.44:0/4219376577 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fcc004800 0x7f7fcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fded9d640 1 --2- 172.21.15.44:0/4219376577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc003a00 0x7f7fcc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fdf59e640 1 -- 172.21.15.44:0/4219376577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc003a00 msgr2=0x7f7fcc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fdf59e640 1 --2- 172.21.15.44:0/4219376577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc003a00 0x7f7fcc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fdf59e640 1 -- 172.21.15.44:0/4219376577 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fcc004800 msgr2=0x7f7fcc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fdf59e640 1 --2- 172.21.15.44:0/4219376577 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fcc004800 0x7f7fcc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.378+0000 7f7fdf59e640 1 -- 172.21.15.44:0/4219376577 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fcc003520 con 0x7f7fcc0051c0 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.379+0000 7f7fdf59e640 1 -- 172.21.15.44:0/4219376577 reap_dead start 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.379+0000 7f7fdf59e640 1 --2- 172.21.15.44:0/4219376577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc0051c0 0x7f7fcc00b9c0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f7fe008e710 tx=0x7f7fcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5fdfa8632a05850 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.379+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0039070 con 0x7f7fcc0051c0 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.379+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fd002cb90 con 0x7f7fcc0051c0 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.379+0000 7f7fddd9b640 1 -- 172.21.15.44:0/4219376577 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0009c10 con 0x7f7fcc0051c0 2024-04-02T13:31:29.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.380+0000 7f7fe5201640 1 -- 172.21.15.44:0/4219376577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc0051c0 msgr2=0x7f7fcc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.380+0000 7f7fe5201640 1 --2- 172.21.15.44:0/4219376577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc0051c0 0x7f7fcc00b9c0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f7fe008e710 tx=0x7f7fcc005830 comp rx=0 tx=0).stop 2024-04-02T13:31:29.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.380+0000 7f7fe5201640 1 -- 172.21.15.44:0/4219376577 shutdown_connections 2024-04-02T13:31:29.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.380+0000 7f7fe5201640 1 --2- 172.21.15.44:0/4219376577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc0051c0 0x7f7fcc00b9c0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.380+0000 7f7fe5201640 1 -- 172.21.15.44:0/4219376577 >> 172.21.15.44:0/4219376577 conn(0x7f7fe009d270 msgr2=0x7f7fe00d3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:29.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.380+0000 7f7fe5201640 1 -- 172.21.15.44:0/4219376577 shutdown_connections 2024-04-02T13:31:29.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.380+0000 7f7fe5201640 1 -- 172.21.15.44:0/4219376577 wait complete. 2024-04-02T13:31:29.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.381+0000 7f7fe5201640 1 Processor -- start 2024-04-02T13:31:29.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.381+0000 7f7fe5201640 1 -- start start 2024-04-02T13:31:29.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fe5201640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc0051c0 0x7f7fe01525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fe5201640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe00ca140 0x7f7fe00cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fe5201640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fe00cc190 0x7f7fe00b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fe5201640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe00b0e30 con 0x7f7fe00cc190 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fe5201640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe00b0cb0 con 0x7f7fe00ca140 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fe5201640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe015e680 con 0x7f7fcc0051c0 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fdf59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fe00cc190 0x7f7fe00b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fdf59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fe00cc190 0x7f7fe00b4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46196/0 (socket says 172.21.15.44:46196) 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fdf59e640 1 -- 172.21.15.44:0/3563631986 learned_addr learned my addr 172.21.15.44:0/3563631986 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fded9d640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc0051c0 0x7f7fe01525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fde59c640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe00ca140 0x7f7fe00cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fdf59e640 1 -- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc0051c0 msgr2=0x7f7fe01525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fdf59e640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc0051c0 0x7f7fe01525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fdf59e640 1 -- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe00ca140 msgr2=0x7f7fe00cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fdf59e640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe00ca140 0x7f7fe00cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.382+0000 7f7fdf59e640 1 -- 172.21.15.44:0/3563631986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe0152b00 con 0x7f7fe00cc190 2024-04-02T13:31:29.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.383+0000 7f7fdf59e640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fe00cc190 0x7f7fe00b4ef0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f7fd002c600 tx=0x7f7fd0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:29.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.383+0000 7f7fd77fe640 1 -- 172.21.15.44:0/3563631986 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0039040 con 0x7f7fe00cc190 2024-04-02T13:31:29.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.383+0000 7f7fd77fe640 1 -- 172.21.15.44:0/3563631986 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fd0045070 con 0x7f7fe00cc190 2024-04-02T13:31:29.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.383+0000 7f7fd77fe640 1 -- 172.21.15.44:0/3563631986 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd003e6a0 con 0x7f7fe00cc190 2024-04-02T13:31:29.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.383+0000 7f7fe5201640 1 -- 172.21.15.44:0/3563631986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe00ca6b0 con 0x7f7fe00cc190 2024-04-02T13:31:29.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.383+0000 7f7fe5201640 1 -- 172.21.15.44:0/3563631986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe00cb450 con 0x7f7fe00cc190 2024-04-02T13:31:29.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.384+0000 7f7fe5201640 1 -- 172.21.15.44:0/3563631986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fe004f8d0 con 0x7f7fe00cc190 2024-04-02T13:31:29.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.384+0000 7f7fd77fe640 1 -- 172.21.15.44:0/3563631986 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7fd0005d10 con 0x7f7fe00cc190 2024-04-02T13:31:29.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.385+0000 7f7fd77fe640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7fc006c8f0 0x7f7fc006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:29.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.385+0000 7f7fd77fe640 1 -- 172.21.15.44:0/3563631986 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f7fd00adb60 con 0x7f7fe00cc190 2024-04-02T13:31:29.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.385+0000 7f7fded9d640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7fc006c8f0 0x7f7fc006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:29.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.386+0000 7f7fded9d640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7fc006c8f0 0x7f7fc006edb0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8001760 tx=0x7f7fd8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:29.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.389+0000 7f7fd77fe640 1 -- 172.21.15.44:0/3563631986 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7fd007b7b0 con 0x7f7fe00cc190 2024-04-02T13:31:29.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.613+0000 7f7fe5201640 1 -- 172.21.15.44:0/3563631986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7fe00d3d00 con 0x7f7fe00cc190 2024-04-02T13:31:29.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.613+0000 7f7fd77fe640 1 -- 172.21.15.44:0/3563631986 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v99) ==== 53+0+73 (secure 0 0 0) 0x7f7fd007fab0 con 0x7f7fe00cc190 2024-04-02T13:31:29.616 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:31:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.615+0000 7f7fe5201640 1 -- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7fc006c8f0 msgr2=0x7f7fc006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.615+0000 7f7fe5201640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7fc006c8f0 0x7f7fc006edb0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8001760 tx=0x7f7fd8009040 comp rx=0 tx=0).stop 2024-04-02T13:31:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.615+0000 7f7fe5201640 1 -- 172.21.15.44:0/3563631986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fe00cc190 msgr2=0x7f7fe00b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.615+0000 7f7fe5201640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fe00cc190 0x7f7fe00b4ef0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f7fd002c600 tx=0x7f7fd0009750 comp rx=0 tx=0).stop 2024-04-02T13:31:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.615+0000 7f7fe5201640 1 -- 172.21.15.44:0/3563631986 shutdown_connections 2024-04-02T13:31:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.615+0000 7f7fe5201640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f7fc006c8f0 0x7f7fc006edb0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.615+0000 7f7fe5201640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fe00cc190 0x7f7fe00b4ef0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.615+0000 7f7fe5201640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe00ca140 0x7f7fe00cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.615+0000 7f7fe5201640 1 --2- 172.21.15.44:0/3563631986 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc0051c0 0x7f7fe01525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.615+0000 7f7fe5201640 1 -- 172.21.15.44:0/3563631986 >> 172.21.15.44:0/3563631986 conn(0x7f7fe009d270 msgr2=0x7f7fe00d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.615+0000 7f7fe5201640 1 -- 172.21.15.44:0/3563631986 shutdown_connections 2024-04-02T13:31:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29.615+0000 7f7fe5201640 1 -- 172.21.15.44:0/3563631986 wait complete. 2024-04-02T13:31:29.629 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:31:29.629 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:31:29.629 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:31:29.629 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:31:29.629 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:29.629 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:31:29.629 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:31:29.629 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:31:29.629 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:31:29.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:29.630 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:31:29.644 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:29.645 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42089sec preferred_lft 42089sec 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:29.646 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:31:29.647 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:29.647 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:31:29.647 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:29.647 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:31:29.647 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:29.647 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:31:29.647 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:29.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:29.647 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:31:29.710 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:31:29.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:29.711 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:31:29.767 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:31:29.767 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:31:29.767 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:31:29.767 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:31:29.767 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:31:29.767 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:31:29.767 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:31:29.768 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:31:29.768 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:31:29.768 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:31:29.768 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:31:29.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:31:29.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:29.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:29.902 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:31:29.916 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:31:29.917 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:31:29.917 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:31:29.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:29.917 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:31:29.983 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:31:29.983 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:31:29.983 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:30.038 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:31:30.038 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:31:30.038 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:30.092 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:31:30.159 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:31:30.215 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:31:30.215 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:31:30.215 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:31:30.215 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:31:30.215 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:31:30.215 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:31:30.215 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:31:30.215 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:31:30.215 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:31:30.215 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:31:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:31:30.217 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:31:30.217 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:31:30.217 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:31:30.217 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:31:30.217 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:31:30.217 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:30.217 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:30.217 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:30.217 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:30.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:30.217 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:31:30.269 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:31:30.270 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:31:30.312 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:31:30.338 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:31:30.393 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:31:30.393 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:31:30.394 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:31:30.395 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:30.396 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:30.396 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:30.396 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:30.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:30.396 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:31:30.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:31:30.393+0000 7f83619f6480 -1 init, newargv = 0x55f4981d4810 newargc=15 2024-04-02T13:31:30.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[82743]: starting ceph client 2024-04-02T13:31:30.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[82743]: starting fuse 2024-04-02T13:31:30.412 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:31:30.412 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:31:30.412 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:31:30.412 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:31:30.412 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:31:30.412 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:30.412 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:31:30.412 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:30.412 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:31:30.412 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:31:30.412 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:31:30.412 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:30.413 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:31:30.414 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:30.533 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.82743.asok 2024-04-02T13:31:30.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:30.543 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.82743.asok 2024-04-02T13:31:30.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:30.543 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.82743.asok status 2024-04-02T13:31:30.686 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:31:30.687 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:31:30.687 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:31:30.687 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:31:30.687 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:31:30.687 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:31:30.687 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:31:30.687 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "82743", 2024-04-02T13:31:30.687 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:31:30.687 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:30.687 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:31:30.687 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:31:30.687 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 9765, 2024-04-02T13:31:30.687 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 9765 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 782187268 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:31:30.688 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 782187268 2024-04-02T13:31:30.689 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:30.689 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.9765 192.168.0.1:0/782187268", 2024-04-02T13:31:30.689 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/782187268", 2024-04-02T13:31:30.689 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:31:30.689 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 99, 2024-04-02T13:31:30.689 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 121, 2024-04-02T13:31:30.689 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:31:30.689 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:31:30.689 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:31:30.689 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:31:30.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:30.690 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:31:30.690 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:30.705 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:31:30.705 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:30.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:30.705 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:30.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:30.772 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent/unflushed) 2024-04-02T13:31:30.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:30.832 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=./parent/unflushed/jfile bs=1M count=8) 2024-04-02T13:31:30.919 INFO:teuthology.orchestra.run.smithi044.stderr:8+0 records in 2024-04-02T13:31:30.919 INFO:teuthology.orchestra.run.smithi044.stderr:8+0 records out 2024-04-02T13:31:30.919 INFO:teuthology.orchestra.run.smithi044.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.032228 s, 260 MB/s 2024-04-02T13:31:30.920 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:30.920 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:30.920 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:30.920 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:30.921 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-04-02T13:31:30.921 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:30.986 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627779 2024-04-02T13:31:30.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:30.994 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:30.995 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:30.995 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:30.995 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:30.995 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-04-02T13:31:30.995 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:31.101 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627780 2024-04-02T13:31:31.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:31.109 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:31:31.109 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:31:31.165 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:31:31.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.302+0000 7f5e8c78d640 1 Processor -- start 2024-04-02T13:31:31.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.302+0000 7f5e8c78d640 1 -- start start 2024-04-02T13:31:31.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e8c78d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e8415e4a0 0x7f5e840aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e8c78d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5e840d0c50 0x7f5e840d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e8c78d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e840cc600 con 0x7f5e8406ce40 2024-04-02T13:31:31.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e8c78d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f5e840cc8e0 con 0x7f5e840d0c50 2024-04-02T13:31:31.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e8c78d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5e840ccbc0 con 0x7f5e8415e4a0 2024-04-02T13:31:31.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e89d28640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5e840d0c50 0x7f5e840d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e89d28640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5e840d0c50 0x7f5e840d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46198/0 (socket says 172.21.15.44:46198) 2024-04-02T13:31:31.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e8ad2a640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f5e8406ce40 0x7f5e840cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51492/0 (socket says 172.21.15.44:51492) 2024-04-02T13:31:31.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e89d28640 1 -- 172.21.15.44:0/2671235500 learned_addr learned my addr 172.21.15.44:0/2671235500 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:31.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e89d28640 1 -- 172.21.15.44:0/2671235500 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e8415e4a0 msgr2=0x7f5e840aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:31.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e8a529640 1 --2- 172.21.15.44:0/2671235500 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e8415e4a0 0x7f5e840aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e89d28640 1 --2- 172.21.15.44:0/2671235500 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e8415e4a0 0x7f5e840aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e89527640 1 -- 172.21.15.44:0/2671235500 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 860279617 0 0) 0x7f5e840cc600 con 0x7f5e8406ce40 2024-04-02T13:31:31.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e89d28640 1 -- 172.21.15.44:0/2671235500 >> v1:172.21.15.44:6789/0 conn(0x7f5e8406ce40 legacy=0x7f5e840cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.303+0000 7f5e89d28640 1 -- 172.21.15.44:0/2671235500 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e840ccea0 con 0x7f5e840d0c50 2024-04-02T13:31:31.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.304+0000 7f5e89d28640 1 --2- 172.21.15.44:0/2671235500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e840d0c50 0x7f5e840d10b0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009870 tx=0x7f5e7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875a306546d80293 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:31.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.304+0000 7f5e89527640 1 -- 172.21.15.44:0/2671235500 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78034070 con 0x7f5e840d0c50 2024-04-02T13:31:31.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.304+0000 7f5e89527640 1 -- 172.21.15.44:0/2671235500 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e78030840 con 0x7f5e840d0c50 2024-04-02T13:31:31.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.304+0000 7f5e89527640 1 -- 172.21.15.44:0/2671235500 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78030c10 con 0x7f5e840d0c50 2024-04-02T13:31:31.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.305+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2671235500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e840d0c50 msgr2=0x7f5e840d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.305+0000 7f5e8c78d640 1 --2- 172.21.15.44:0/2671235500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e840d0c50 0x7f5e840d10b0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009870 tx=0x7f5e7802b980 comp rx=0 tx=0).stop 2024-04-02T13:31:31.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.305+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2671235500 shutdown_connections 2024-04-02T13:31:31.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.305+0000 7f5e8c78d640 1 --2- 172.21.15.44:0/2671235500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e840d0c50 0x7f5e840d10b0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.305+0000 7f5e8c78d640 1 --2- 172.21.15.44:0/2671235500 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e8415e4a0 0x7f5e840aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.305+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2671235500 >> 172.21.15.44:0/2671235500 conn(0x7f5e8409aa20 msgr2=0x7f5e8409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:31.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.305+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2671235500 shutdown_connections 2024-04-02T13:31:31.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.305+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2671235500 wait complete. 2024-04-02T13:31:31.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.306+0000 7f5e8c78d640 1 Processor -- start 2024-04-02T13:31:31.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.306+0000 7f5e8c78d640 1 -- start start 2024-04-02T13:31:31.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e8c78d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e8406ce40 0x7f5e841f7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e8c78d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e840d0c50 0x7f5e841535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e8c78d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e8415e4a0 0x7f5e841581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e8c78d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8415e880 con 0x7f5e840d0c50 2024-04-02T13:31:31.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e8c78d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5e840cd460 con 0x7f5e8406ce40 2024-04-02T13:31:31.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e8c78d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5e840af130 con 0x7f5e8415e4a0 2024-04-02T13:31:31.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e89d28640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e840d0c50 0x7f5e841535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e89d28640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e840d0c50 0x7f5e841535f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46210/0 (socket says 172.21.15.44:46210) 2024-04-02T13:31:31.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e89d28640 1 -- 172.21.15.44:0/2844668807 learned_addr learned my addr 172.21.15.44:0/2844668807 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:31.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e8a529640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e8406ce40 0x7f5e841f7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e8ad2a640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e8415e4a0 0x7f5e841581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e89d28640 1 -- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e8415e4a0 msgr2=0x7f5e841581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e89d28640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e8415e4a0 0x7f5e841581f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e89d28640 1 -- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e8406ce40 msgr2=0x7f5e841f7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e89d28640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e8406ce40 0x7f5e841f7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e89d28640 1 -- 172.21.15.44:0/2844668807 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e841f73c0 con 0x7f5e840d0c50 2024-04-02T13:31:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e89d28640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e840d0c50 0x7f5e841535f0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f5e7802b9c0 tx=0x7f5e78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.307+0000 7f5e82ffd640 1 -- 172.21.15.44:0/2844668807 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78048070 con 0x7f5e840d0c50 2024-04-02T13:31:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.308+0000 7f5e82ffd640 1 -- 172.21.15.44:0/2844668807 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e78030640 con 0x7f5e840d0c50 2024-04-02T13:31:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.308+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2844668807 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e84161850 con 0x7f5e840d0c50 2024-04-02T13:31:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.308+0000 7f5e82ffd640 1 -- 172.21.15.44:0/2844668807 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78034040 con 0x7f5e840d0c50 2024-04-02T13:31:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.308+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2844668807 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e8415fca0 con 0x7f5e840d0c50 2024-04-02T13:31:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.308+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2844668807 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e840684f0 con 0x7f5e840d0c50 2024-04-02T13:31:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.309+0000 7f5e82ffd640 1 -- 172.21.15.44:0/2844668807 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5e7804b070 con 0x7f5e840d0c50 2024-04-02T13:31:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.309+0000 7f5e82ffd640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5e7006c8f0 0x7f5e7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.309+0000 7f5e82ffd640 1 -- 172.21.15.44:0/2844668807 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f5e780ade50 con 0x7f5e840d0c50 2024-04-02T13:31:31.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.310+0000 7f5e8a529640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5e7006c8f0 0x7f5e7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.310+0000 7f5e8a529640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5e7006c8f0 0x7f5e7006edb0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c00b7c0 tx=0x7f5e7c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:31.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.314+0000 7f5e82ffd640 1 -- 172.21.15.44:0/2844668807 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5e7807c0e0 con 0x7f5e840d0c50 2024-04-02T13:31:31.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.501+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2844668807 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e8406db60 con 0x7f5e840d0c50 2024-04-02T13:31:31.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.503+0000 7f5e82ffd640 1 -- 172.21.15.44:0/2844668807 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) ==== 74+0+16904 (secure 0 0 0) 0x7f5e7807b9e0 con 0x7f5e840d0c50 2024-04-02T13:31:31.507 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:31.507 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":121,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:13.125622+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:31:09.752845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-02T13:31:10.437454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:31:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.505+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5e7006c8f0 msgr2=0x7f5e7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.505+0000 7f5e8c78d640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5e7006c8f0 0x7f5e7006edb0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c00b7c0 tx=0x7f5e7c008040 comp rx=0 tx=0).stop 2024-04-02T13:31:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.505+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2844668807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e840d0c50 msgr2=0x7f5e841535f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.505+0000 7f5e8c78d640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e840d0c50 0x7f5e841535f0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f5e7802b9c0 tx=0x7f5e78004730 comp rx=0 tx=0).stop 2024-04-02T13:31:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.505+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2844668807 shutdown_connections 2024-04-02T13:31:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.505+0000 7f5e8c78d640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5e7006c8f0 0x7f5e7006edb0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.505+0000 7f5e8c78d640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e8415e4a0 0x7f5e841581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.505+0000 7f5e8c78d640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e840d0c50 0x7f5e841535f0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.505+0000 7f5e8c78d640 1 --2- 172.21.15.44:0/2844668807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e8406ce40 0x7f5e841f7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.505+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2844668807 >> 172.21.15.44:0/2844668807 conn(0x7f5e8409aa20 msgr2=0x7f5e8406f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:31.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.505+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2844668807 shutdown_connections 2024-04-02T13:31:31.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.505+0000 7f5e8c78d640 1 -- 172.21.15.44:0/2844668807 wait complete. 2024-04-02T13:31:31.519 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:31:09.752845+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:31:10.437454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:31:31.520 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:31:31.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.620+0000 7f8b0fed6640 1 Processor -- start 2024-04-02T13:31:31.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.620+0000 7f8b0fed6640 1 -- start start 2024-04-02T13:31:31.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.621+0000 7f8b0fed6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8b0815e1d0 0x7f8b080d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.621+0000 7f8b0fed6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b080d1640 con 0x7f8b080d2dd0 2024-04-02T13:31:31.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.621+0000 7f8b0fed6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b080cf1f0 con 0x7f8b080cec60 2024-04-02T13:31:31.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.621+0000 7f8b0fed6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8b080cfe00 con 0x7f8b0815e1d0 2024-04-02T13:31:31.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.621+0000 7f8b0e473640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f8b080d2dd0 0x7f8b080b4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51494/0 (socket says 172.21.15.44:51494) 2024-04-02T13:31:31.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.621+0000 7f8b0e473640 1 -- 172.21.15.44:0/2500442457 learned_addr learned my addr 172.21.15.44:0/2500442457 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:31.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.622+0000 7f8b0dc72640 1 --2- 172.21.15.44:0/2500442457 >> v2:172.21.15.69:3300/0 conn(0x7f8b0815e1d0 0x7f8b080d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.622+0000 7f8b0cc70640 1 -- 172.21.15.44:0/2500442457 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 611854119 0 0) 0x7f8b080d1640 con 0x7f8b080d2dd0 2024-04-02T13:31:31.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.622+0000 7f8b0cc70640 1 -- 172.21.15.44:0/2500442457 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8af0003520 con 0x7f8b080d2dd0 2024-04-02T13:31:31.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.622+0000 7f8b0cc70640 1 -- 172.21.15.44:0/2500442457 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 578764123 0 0) 0x7f8af0003520 con 0x7f8b080d2dd0 2024-04-02T13:31:31.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.622+0000 7f8b0dc72640 1 -- 172.21.15.44:0/2500442457 >> v1:172.21.15.69:6789/0 conn(0x7f8b080cec60 legacy=0x7f8b080d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.622+0000 7f8b0dc72640 1 -- 172.21.15.44:0/2500442457 >> v1:172.21.15.44:6789/0 conn(0x7f8b080d2dd0 legacy=0x7f8b080b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.622+0000 7f8b0dc72640 1 -- 172.21.15.44:0/2500442457 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b080cf4d0 con 0x7f8b0815e1d0 2024-04-02T13:31:31.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.623+0000 7f8b0dc72640 1 --2- 172.21.15.44:0/2500442457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b0815e1d0 0x7f8b080d1100 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009d90 tx=0x7f8af802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1d4b737e141b3da server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:31.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.623+0000 7f8b0cc70640 1 -- 172.21.15.44:0/2500442457 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af8039070 con 0x7f8b0815e1d0 2024-04-02T13:31:31.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.623+0000 7f8b0cc70640 1 -- 172.21.15.44:0/2500442457 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af802c8b0 con 0x7f8b0815e1d0 2024-04-02T13:31:31.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.623+0000 7f8b0cc70640 1 -- 172.21.15.44:0/2500442457 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b00001670 con 0x7f8b0815e1d0 2024-04-02T13:31:31.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.623+0000 7f8b0fed6640 1 -- 172.21.15.44:0/2500442457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b0815e1d0 msgr2=0x7f8b080d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.623+0000 7f8b0fed6640 1 --2- 172.21.15.44:0/2500442457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b0815e1d0 0x7f8b080d1100 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009d90 tx=0x7f8af802bd80 comp rx=0 tx=0).stop 2024-04-02T13:31:31.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.624+0000 7f8b0fed6640 1 -- 172.21.15.44:0/2500442457 shutdown_connections 2024-04-02T13:31:31.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.624+0000 7f8b0fed6640 1 --2- 172.21.15.44:0/2500442457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b0815e1d0 0x7f8b080d1100 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.624+0000 7f8b0fed6640 1 -- 172.21.15.44:0/2500442457 >> 172.21.15.44:0/2500442457 conn(0x7f8b0809d3b0 msgr2=0x7f8b080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:31.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.624+0000 7f8b0fed6640 1 -- 172.21.15.44:0/2500442457 shutdown_connections 2024-04-02T13:31:31.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.624+0000 7f8b0fed6640 1 -- 172.21.15.44:0/2500442457 wait complete. 2024-04-02T13:31:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.625+0000 7f8b0fed6640 1 Processor -- start 2024-04-02T13:31:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.625+0000 7f8b0fed6640 1 -- start start 2024-04-02T13:31:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.625+0000 7f8b0fed6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b080cec60 0x7f8b0815f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.625+0000 7f8b0fed6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b080d2dd0 0x7f8b08161200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.625+0000 7f8b0fed6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b0815e1d0 0x7f8b080cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.625+0000 7f8b0fed6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8b080b7fd0 con 0x7f8b080d2dd0 2024-04-02T13:31:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.625+0000 7f8b0fed6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8b080cfa90 con 0x7f8b080cec60 2024-04-02T13:31:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.625+0000 7f8b0fed6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8b080b8150 con 0x7f8b0815e1d0 2024-04-02T13:31:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0d471640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b080d2dd0 0x7f8b08161200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0d471640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b080d2dd0 0x7f8b08161200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46222/0 (socket says 172.21.15.44:46222) 2024-04-02T13:31:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0d471640 1 -- 172.21.15.44:0/3904111493 learned_addr learned my addr 172.21.15.44:0/3904111493 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0dc72640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b080cec60 0x7f8b0815f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0e473640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b0815e1d0 0x7f8b080cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0d471640 1 -- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b0815e1d0 msgr2=0x7f8b080cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0d471640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b0815e1d0 0x7f8b080cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0d471640 1 -- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b080cec60 msgr2=0x7f8b0815f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0d471640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b080cec60 0x7f8b0815f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0d471640 1 -- 172.21.15.44:0/3904111493 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b08160160 con 0x7f8b080d2dd0 2024-04-02T13:31:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0e473640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b0815e1d0 0x7f8b080cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0d471640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b080d2dd0 0x7f8b08161200 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00c430 tx=0x7f8afc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b067fc640 1 -- 172.21.15.44:0/3904111493 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc017070 con 0x7f8b080d2dd0 2024-04-02T13:31:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b067fc640 1 -- 172.21.15.44:0/3904111493 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8afc0052d0 con 0x7f8b080d2dd0 2024-04-02T13:31:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b067fc640 1 -- 172.21.15.44:0/3904111493 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc003830 con 0x7f8b080d2dd0 2024-04-02T13:31:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0fed6640 1 -- 172.21.15.44:0/3904111493 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b08160440 con 0x7f8b080d2dd0 2024-04-02T13:31:31.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.626+0000 7f8b0fed6640 1 -- 172.21.15.44:0/3904111493 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b08160a00 con 0x7f8b080d2dd0 2024-04-02T13:31:31.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.627+0000 7f8b0fed6640 1 -- 172.21.15.44:0/3904111493 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b080684f0 con 0x7f8b080d2dd0 2024-04-02T13:31:31.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.628+0000 7f8b067fc640 1 -- 172.21.15.44:0/3904111493 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8afc01a070 con 0x7f8b080d2dd0 2024-04-02T13:31:31.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.628+0000 7f8b067fc640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8af406c8f0 0x7f8af406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.628+0000 7f8b067fc640 1 -- 172.21.15.44:0/3904111493 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f8afc08af70 con 0x7f8b080d2dd0 2024-04-02T13:31:31.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.628+0000 7f8b0dc72640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8af406c8f0 0x7f8af406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.629+0000 7f8b0dc72640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8af406c8f0 0x7f8af406edb0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f8af8036000 tx=0x7f8af8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:31.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.632+0000 7f8b067fc640 1 -- 172.21.15.44:0/3904111493 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8afc058bc0 con 0x7f8b080d2dd0 2024-04-02T13:31:31.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.815+0000 7f8b0fed6640 1 -- 172.21.15.44:0/3904111493 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8b0806d0c0 con 0x7f8b080d2dd0 2024-04-02T13:31:31.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.815+0000 7f8b067fc640 1 -- 172.21.15.44:0/3904111493 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) ==== 93+0+10 (secure 0 0 0) 0x7f8afc05cec0 con 0x7f8b080d2dd0 2024-04-02T13:31:31.819 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:31:31.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.817+0000 7f8b0fed6640 1 -- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8af406c8f0 msgr2=0x7f8af406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.817+0000 7f8b0fed6640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8af406c8f0 0x7f8af406edb0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f8af8036000 tx=0x7f8af8030000 comp rx=0 tx=0).stop 2024-04-02T13:31:31.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.817+0000 7f8b0fed6640 1 -- 172.21.15.44:0/3904111493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b080d2dd0 msgr2=0x7f8b08161200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.817+0000 7f8b0fed6640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b080d2dd0 0x7f8b08161200 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00c430 tx=0x7f8afc00c8f0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.817+0000 7f8b0fed6640 1 -- 172.21.15.44:0/3904111493 shutdown_connections 2024-04-02T13:31:31.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.817+0000 7f8b0fed6640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8af406c8f0 0x7f8af406edb0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.817+0000 7f8b0fed6640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b0815e1d0 0x7f8b080cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.817+0000 7f8b0fed6640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b080d2dd0 0x7f8b08161200 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.817+0000 7f8b0fed6640 1 --2- 172.21.15.44:0/3904111493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b080cec60 0x7f8b0815f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.817+0000 7f8b0fed6640 1 -- 172.21.15.44:0/3904111493 >> 172.21.15.44:0/3904111493 conn(0x7f8b0809d3b0 msgr2=0x7f8b08161d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:31.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.817+0000 7f8b0fed6640 1 -- 172.21.15.44:0/3904111493 shutdown_connections 2024-04-02T13:31:31.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.817+0000 7f8b0fed6640 1 -- 172.21.15.44:0/3904111493 wait complete. 2024-04-02T13:31:31.832 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:31:31.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.932+0000 7f1580088640 1 Processor -- start 2024-04-02T13:31:31.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.932+0000 7f1580088640 1 -- start start 2024-04-02T13:31:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.933+0000 7f1580088640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f15780cfe10 0x7f1578169240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.933+0000 7f1580088640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f157806c220 con 0x7f15780721f0 2024-04-02T13:31:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.933+0000 7f1580088640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f157815f7e0 con 0x7f157815e420 2024-04-02T13:31:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.933+0000 7f1580088640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f15780cf3b0 con 0x7f15780cfe10 2024-04-02T13:31:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.933+0000 7f157d623640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f15780721f0 0x7f15780cf700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51500/0 (socket says 172.21.15.44:51500) 2024-04-02T13:31:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.933+0000 7f157d623640 1 -- 172.21.15.44:0/2017790757 learned_addr learned my addr 172.21.15.44:0/2017790757 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.933+0000 7f157e625640 1 --2- 172.21.15.44:0/2017790757 >> v2:172.21.15.69:3300/0 conn(0x7f15780cfe10 0x7f1578169240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3726187106 0 0) 0x7f156c001110 con 0x7f15780721f0 2024-04-02T13:31:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1560003520 con 0x7f15780721f0 2024-04-02T13:31:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3789453292 0 0) 0x7f1560003520 con 0x7f15780721f0 2024-04-02T13:31:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 >> v2:172.21.15.69:3300/0 conn(0x7f15780cfe10 msgr2=0x7f1578169240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 --2- 172.21.15.44:0/2017790757 >> v2:172.21.15.69:3300/0 conn(0x7f15780cfe10 0x7f1578169240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 >> v1:172.21.15.69:6789/0 conn(0x7f157815e420 legacy=0x7f15780ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157815ec60 con 0x7f15780721f0 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f156c001840 con 0x7f15780721f0 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 >> v1:172.21.15.44:6789/0 conn(0x7f15780721f0 legacy=0x7f15780cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 --2- 172.21.15.44:0/2017790757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15600039e0 0x7f1560003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 --2- 172.21.15.44:0/2017790757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1560004800 0x7f1560004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 --2- 172.21.15.44:0/2017790757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15600052e0 0x7f156000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f15780cf3b0 con 0x7f1560004800 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f156000c1e0 con 0x7f15600039e0 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f156000c3e0 con 0x7f15600052e0 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157d623640 1 --2- 172.21.15.44:0/2017790757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1560004800 0x7f1560004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157de24640 1 --2- 172.21.15.44:0/2017790757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15600039e0 0x7f1560003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157e625640 1 --2- 172.21.15.44:0/2017790757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15600052e0 0x7f156000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157d623640 1 -- 172.21.15.44:0/2017790757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15600052e0 msgr2=0x7f156000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:31.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157d623640 1 --2- 172.21.15.44:0/2017790757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15600052e0 0x7f156000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157d623640 1 -- 172.21.15.44:0/2017790757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15600039e0 msgr2=0x7f1560003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157d623640 1 --2- 172.21.15.44:0/2017790757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15600039e0 0x7f1560003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157d623640 1 -- 172.21.15.44:0/2017790757 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1560003520 con 0x7f1560004800 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.934+0000 7f157e625640 1 -- 172.21.15.44:0/2017790757 reap_dead start 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.935+0000 7f157d623640 1 --2- 172.21.15.44:0/2017790757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1560004800 0x7f1560004c80 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f156c00c490 tx=0x7f1560005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23fadd93c5e72f57 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.935+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c00c0e0 con 0x7f1560004800 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.935+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f156c030070 con 0x7f1560004800 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.935+0000 7f157ce22640 1 -- 172.21.15.44:0/2017790757 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c0058d0 con 0x7f1560004800 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.936+0000 7f1580088640 1 -- 172.21.15.44:0/2017790757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1560004800 msgr2=0x7f1560004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.936+0000 7f1580088640 1 --2- 172.21.15.44:0/2017790757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1560004800 0x7f1560004c80 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f156c00c490 tx=0x7f1560005830 comp rx=0 tx=0).stop 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.936+0000 7f1580088640 1 -- 172.21.15.44:0/2017790757 shutdown_connections 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.936+0000 7f1580088640 1 --2- 172.21.15.44:0/2017790757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1560004800 0x7f1560004c80 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.936+0000 7f1580088640 1 -- 172.21.15.44:0/2017790757 >> 172.21.15.44:0/2017790757 conn(0x7f15780b2a10 msgr2=0x7f15780b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:31.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.936+0000 7f1580088640 1 -- 172.21.15.44:0/2017790757 shutdown_connections 2024-04-02T13:31:31.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.936+0000 7f1580088640 1 -- 172.21.15.44:0/2017790757 wait complete. 2024-04-02T13:31:31.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.937+0000 7f1580088640 1 Processor -- start 2024-04-02T13:31:31.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.937+0000 7f1580088640 1 -- start start 2024-04-02T13:31:31.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f1580088640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1560004800 0x7f1578141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f1580088640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1578141b90 0x7f157815c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f1580088640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f157815cb80 0x7f1578143be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f1580088640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f157815f500 con 0x7f1560004800 2024-04-02T13:31:31.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f1580088640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f157815f220 con 0x7f157815cb80 2024-04-02T13:31:31.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f1580088640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f157815e800 con 0x7f1578141b90 2024-04-02T13:31:31.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f157de24640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1560004800 0x7f1578141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f157de24640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1560004800 0x7f1578141370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46236/0 (socket says 172.21.15.44:46236) 2024-04-02T13:31:31.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f157de24640 1 -- 172.21.15.44:0/3813732842 learned_addr learned my addr 172.21.15.44:0/3813732842 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:31.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f157e625640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f157815cb80 0x7f1578143be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f157d623640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1578141b90 0x7f157815c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f157de24640 1 -- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1578141b90 msgr2=0x7f157815c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f157de24640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1578141b90 0x7f157815c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f157de24640 1 -- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f157815cb80 msgr2=0x7f1578143be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:31.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f157de24640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f157815cb80 0x7f1578143be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:31.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f157de24640 1 -- 172.21.15.44:0/3813732842 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15781418b0 con 0x7f1560004800 2024-04-02T13:31:31.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f157e625640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f157815cb80 0x7f1578143be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:31.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f157d623640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1578141b90 0x7f157815c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:31.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f157de24640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1560004800 0x7f1578141370 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f157000f650 tx=0x7f157000fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:31.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f15767fc640 1 -- 172.21.15.44:0/3813732842 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15700050d0 con 0x7f1560004800 2024-04-02T13:31:31.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f15767fc640 1 -- 172.21.15.44:0/3813732842 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1570018070 con 0x7f1560004800 2024-04-02T13:31:31.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f15767fc640 1 -- 172.21.15.44:0/3813732842 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1570005490 con 0x7f1560004800 2024-04-02T13:31:31.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f1580088640 1 -- 172.21.15.44:0/3813732842 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1578159f80 con 0x7f1560004800 2024-04-02T13:31:31.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.938+0000 7f1580088640 1 -- 172.21.15.44:0/3813732842 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1578157310 con 0x7f1560004800 2024-04-02T13:31:31.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.939+0000 7f1580088640 1 -- 172.21.15.44:0/3813732842 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15780684f0 con 0x7f1560004800 2024-04-02T13:31:31.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.940+0000 7f15767fc640 1 -- 172.21.15.44:0/3813732842 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f157000d6d0 con 0x7f1560004800 2024-04-02T13:31:31.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.940+0000 7f15767fc640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f156406c8f0 0x7f156406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:31.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.940+0000 7f15767fc640 1 -- 172.21.15.44:0/3813732842 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f157008ba30 con 0x7f1560004800 2024-04-02T13:31:31.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.941+0000 7f157d623640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f156406c8f0 0x7f156406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:31.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.941+0000 7f157d623640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f156406c8f0 0x7f156406edb0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f15600057e0 tx=0x7f156c039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:31.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:31.945+0000 7f15767fc640 1 -- 172.21.15.44:0/3813732842 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f15781418b0 con 0x7f1560004800 2024-04-02T13:31:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.131+0000 7f1580088640 1 -- 172.21.15.44:0/3813732842 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f15780b3770 con 0x7f1560004800 2024-04-02T13:31:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.131+0000 7f15767fc640 1 -- 172.21.15.44:0/3813732842 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v121) ==== 104+0+11 (secure 0 0 0) 0x7f1570059cd0 con 0x7f1560004800 2024-04-02T13:31:32.135 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:31:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.133+0000 7f1580088640 1 -- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f156406c8f0 msgr2=0x7f156406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.133+0000 7f1580088640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f156406c8f0 0x7f156406edb0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f15600057e0 tx=0x7f156c039040 comp rx=0 tx=0).stop 2024-04-02T13:31:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.133+0000 7f1580088640 1 -- 172.21.15.44:0/3813732842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1560004800 msgr2=0x7f1578141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.133+0000 7f1580088640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1560004800 0x7f1578141370 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f157000f650 tx=0x7f157000fb10 comp rx=0 tx=0).stop 2024-04-02T13:31:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.133+0000 7f1580088640 1 -- 172.21.15.44:0/3813732842 shutdown_connections 2024-04-02T13:31:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.133+0000 7f1580088640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f156406c8f0 0x7f156406edb0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.133+0000 7f1580088640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f157815cb80 0x7f1578143be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.133+0000 7f1580088640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1578141b90 0x7f157815c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.133+0000 7f1580088640 1 --2- 172.21.15.44:0/3813732842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1560004800 0x7f1578141370 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.133+0000 7f1580088640 1 -- 172.21.15.44:0/3813732842 >> 172.21.15.44:0/3813732842 conn(0x7f15780b2a10 msgr2=0x7f157806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.133+0000 7f1580088640 1 -- 172.21.15.44:0/3813732842 shutdown_connections 2024-04-02T13:31:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.133+0000 7f1580088640 1 -- 172.21.15.44:0/3813732842 wait complete. 2024-04-02T13:31:32.147 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:31:32.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.247+0000 7f16fee0f640 1 Processor -- start 2024-04-02T13:31:32.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.247+0000 7f16fee0f640 1 -- start start 2024-04-02T13:31:32.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.248+0000 7f16fee0f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f16f80cec60 0x7f16f80d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.248+0000 7f16fee0f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16f815e5b0 0x7f16f80d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.248+0000 7f16fee0f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16f80cf1f0 con 0x7f16f815e1d0 2024-04-02T13:31:32.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.248+0000 7f16fee0f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f16f80cfe00 con 0x7f16f80cec60 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.248+0000 7f16fee0f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f16f80cf4d0 con 0x7f16f815e5b0 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.248+0000 7f16f7fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f16f80cec60 0x7f16f80d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.248+0000 7f16f7fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f16f80cec60 0x7f16f80d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46242/0 (socket says 172.21.15.44:46242) 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.248+0000 7f16f7fff640 1 -- 172.21.15.44:0/4205122764 learned_addr learned my addr 172.21.15.44:0/4205122764 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.249+0000 7f16fd3ac640 1 --2- 172.21.15.44:0/4205122764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16f815e5b0 0x7f16f80d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.249+0000 7f16f7fff640 1 -- 172.21.15.44:0/4205122764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16f815e5b0 msgr2=0x7f16f80d2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.249+0000 7f16f7fff640 1 --2- 172.21.15.44:0/4205122764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16f815e5b0 0x7f16f80d2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.249+0000 7f16f7fff640 1 -- 172.21.15.44:0/4205122764 >> v1:172.21.15.69:6789/0 conn(0x7f16f815e1d0 legacy=0x7f16f80d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.249+0000 7f16f7fff640 1 -- 172.21.15.44:0/4205122764 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f80cf7b0 con 0x7f16f80cec60 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.249+0000 7f16fd3ac640 1 --2- 172.21.15.44:0/4205122764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16f815e5b0 0x7f16f80d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.249+0000 7f16f7fff640 1 --2- 172.21.15.44:0/4205122764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16f80cec60 0x7f16f80d26c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009870 tx=0x7f16ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c0ee26b7a43d4c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.250+0000 7f16f77fe640 1 -- 172.21.15.44:0/4205122764 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec035020 con 0x7f16f80cec60 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.250+0000 7f16f77fe640 1 -- 172.21.15.44:0/4205122764 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ec031640 con 0x7f16f80cec60 2024-04-02T13:31:32.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.250+0000 7f16f77fe640 1 -- 172.21.15.44:0/4205122764 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec02d040 con 0x7f16f80cec60 2024-04-02T13:31:32.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.250+0000 7f16fee0f640 1 -- 172.21.15.44:0/4205122764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16f80cec60 msgr2=0x7f16f80d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.250+0000 7f16fee0f640 1 --2- 172.21.15.44:0/4205122764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16f80cec60 0x7f16f80d26c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009870 tx=0x7f16ec02b8f0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.250+0000 7f16fee0f640 1 -- 172.21.15.44:0/4205122764 shutdown_connections 2024-04-02T13:31:32.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.250+0000 7f16fee0f640 1 --2- 172.21.15.44:0/4205122764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16f80cec60 0x7f16f80d26c0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.250+0000 7f16fee0f640 1 --2- 172.21.15.44:0/4205122764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16f815e5b0 0x7f16f80d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.250+0000 7f16fee0f640 1 -- 172.21.15.44:0/4205122764 >> 172.21.15.44:0/4205122764 conn(0x7f16f809d3b0 msgr2=0x7f16f80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:32.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.250+0000 7f16fee0f640 1 -- 172.21.15.44:0/4205122764 shutdown_connections 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.250+0000 7f16fee0f640 1 -- 172.21.15.44:0/4205122764 wait complete. 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.251+0000 7f16fee0f640 1 Processor -- start 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.251+0000 7f16fee0f640 1 -- start start 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.252+0000 7f16fee0f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16f80cec60 0x7f16f8152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.252+0000 7f16fee0f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16f815e1d0 0x7f16f815ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.252+0000 7f16fee0f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16f815e5b0 0x7f16f80cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.252+0000 7f16fee0f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f16f80b4e60 con 0x7f16f815e1d0 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.252+0000 7f16fee0f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f16f80b4ce0 con 0x7f16f815e5b0 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.252+0000 7f16fee0f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f16f80b4fe0 con 0x7f16f80cec60 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.252+0000 7f16f7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16f815e1d0 0x7f16f815ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.252+0000 7f16f7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16f815e1d0 0x7f16f815ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46246/0 (socket says 172.21.15.44:46246) 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.252+0000 7f16f7fff640 1 -- 172.21.15.44:0/3640140929 learned_addr learned my addr 172.21.15.44:0/3640140929 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.252+0000 7f16fcbab640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16f80cec60 0x7f16f8152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.252+0000 7f16fd3ac640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16f815e5b0 0x7f16f80cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.253+0000 7f16f7fff640 1 -- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16f80cec60 msgr2=0x7f16f8152910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.253+0000 7f16f7fff640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16f80cec60 0x7f16f8152910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.253+0000 7f16f7fff640 1 -- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16f815e5b0 msgr2=0x7f16f80cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.253+0000 7f16f7fff640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16f815e5b0 0x7f16f80cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.253+0000 7f16f7fff640 1 -- 172.21.15.44:0/3640140929 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f8152e50 con 0x7f16f815e1d0 2024-04-02T13:31:32.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.253+0000 7f16fcbab640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16f80cec60 0x7f16f8152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:32.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.253+0000 7f16fd3ac640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16f815e5b0 0x7f16f80cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:32.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.253+0000 7f16f7fff640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16f815e1d0 0x7f16f815ea20 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f16ec02b930 tx=0x7f16ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:32.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.253+0000 7f16f57fa640 1 -- 172.21.15.44:0/3640140929 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec035040 con 0x7f16f815e1d0 2024-04-02T13:31:32.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.253+0000 7f16f57fa640 1 -- 172.21.15.44:0/3640140929 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ec039bf0 con 0x7f16f815e1d0 2024-04-02T13:31:32.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.253+0000 7f16fee0f640 1 -- 172.21.15.44:0/3640140929 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f80cc620 con 0x7f16f815e1d0 2024-04-02T13:31:32.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.253+0000 7f16f57fa640 1 -- 172.21.15.44:0/3640140929 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec045480 con 0x7f16f815e1d0 2024-04-02T13:31:32.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.253+0000 7f16fee0f640 1 -- 172.21.15.44:0/3640140929 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f80ccbe0 con 0x7f16f815e1d0 2024-04-02T13:31:32.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.254+0000 7f16fee0f640 1 -- 172.21.15.44:0/3640140929 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f80684f0 con 0x7f16f815e1d0 2024-04-02T13:31:32.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.254+0000 7f16f57fa640 1 -- 172.21.15.44:0/3640140929 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f16ec038020 con 0x7f16f815e1d0 2024-04-02T13:31:32.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.255+0000 7f16f57fa640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f16e406c920 0x7f16e406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.255+0000 7f16f57fa640 1 -- 172.21.15.44:0/3640140929 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f16ec0adc70 con 0x7f16f815e1d0 2024-04-02T13:31:32.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.255+0000 7f16fcbab640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f16e406c920 0x7f16e406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.256+0000 7f16fcbab640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f16e406c920 0x7f16e406ede0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f16f00012e0 tx=0x7f16f0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:32.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.259+0000 7f16f57fa640 1 -- 172.21.15.44:0/3640140929 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f16ec031610 con 0x7f16f815e1d0 2024-04-02T13:31:32.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.443+0000 7f16fee0f640 1 -- 172.21.15.44:0/3640140929 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f16f80d47f0 con 0x7f16f815e1d0 2024-04-02T13:31:32.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.444+0000 7f16f57fa640 1 -- 172.21.15.44:0/3640140929 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v121) ==== 100+0+11 (secure 0 0 0) 0x7f16ec07bfa0 con 0x7f16f815e1d0 2024-04-02T13:31:32.448 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:31:32.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.446+0000 7f16fee0f640 1 -- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f16e406c920 msgr2=0x7f16e406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.446+0000 7f16fee0f640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f16e406c920 0x7f16e406ede0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f16f00012e0 tx=0x7f16f0009040 comp rx=0 tx=0).stop 2024-04-02T13:31:32.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.446+0000 7f16fee0f640 1 -- 172.21.15.44:0/3640140929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16f815e1d0 msgr2=0x7f16f815ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.446+0000 7f16fee0f640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16f815e1d0 0x7f16f815ea20 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f16ec02b930 tx=0x7f16ec0046c0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.446+0000 7f16fee0f640 1 -- 172.21.15.44:0/3640140929 shutdown_connections 2024-04-02T13:31:32.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.446+0000 7f16fee0f640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f16e406c920 0x7f16e406ede0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.446+0000 7f16fee0f640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16f815e5b0 0x7f16f80cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.446+0000 7f16fee0f640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16f815e1d0 0x7f16f815ea20 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.446+0000 7f16fee0f640 1 --2- 172.21.15.44:0/3640140929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16f80cec60 0x7f16f8152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.446+0000 7f16fee0f640 1 -- 172.21.15.44:0/3640140929 >> 172.21.15.44:0/3640140929 conn(0x7f16f809d3b0 msgr2=0x7f16f806cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:32.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.446+0000 7f16fee0f640 1 -- 172.21.15.44:0/3640140929 shutdown_connections 2024-04-02T13:31:32.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.446+0000 7f16fee0f640 1 -- 172.21.15.44:0/3640140929 wait complete. 2024-04-02T13:31:32.460 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:31:32.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.560+0000 7fb5125eb640 1 Processor -- start 2024-04-02T13:31:32.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.560+0000 7fb5125eb640 1 -- start start 2024-04-02T13:31:32.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.560+0000 7fb5125eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb50c15e0f0 0x7fb50c0d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.560+0000 7fb5125eb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb50c0d0630 0x7fb50c06b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.560+0000 7fb5125eb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb50c0aed70 con 0x7fb50c15e4d0 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.560+0000 7fb5125eb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb50c06dfe0 con 0x7fb50c0d0630 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.560+0000 7fb5125eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb50c06e2c0 con 0x7fb50c15e0f0 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.561+0000 7fb50b7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb50c0d0630 0x7fb50c06b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.561+0000 7fb50bfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb50c15e0f0 0x7fb50c0d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.561+0000 7fb510b88640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb50c15e4d0 0x7fb50c161db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:58194/0 (socket says 172.21.15.44:58194) 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.561+0000 7fb510b88640 1 -- 172.21.15.44:0/3303015871 learned_addr learned my addr 172.21.15.44:0/3303015871 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.561+0000 7fb50affd640 1 -- 172.21.15.44:0/3303015871 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2409996627 0 0) 0x7fb50c0aed70 con 0x7fb50c15e4d0 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.561+0000 7fb50affd640 1 -- 172.21.15.44:0/3303015871 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4f4003520 con 0x7fb50c15e4d0 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.561+0000 7fb50b7fe640 1 -- 172.21.15.44:0/3303015871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb50c15e0f0 msgr2=0x7fb50c0d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.561+0000 7fb50b7fe640 1 --2- 172.21.15.44:0/3303015871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb50c15e0f0 0x7fb50c0d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.561+0000 7fb50b7fe640 1 -- 172.21.15.44:0/3303015871 >> v1:172.21.15.69:6789/0 conn(0x7fb50c15e4d0 legacy=0x7fb50c161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.561+0000 7fb50b7fe640 1 -- 172.21.15.44:0/3303015871 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb50c06e5a0 con 0x7fb50c0d0630 2024-04-02T13:31:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.561+0000 7fb50bfff640 1 --2- 172.21.15.44:0/3303015871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb50c15e0f0 0x7fb50c0d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:31:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.562+0000 7fb50b7fe640 1 --2- 172.21.15.44:0/3303015871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb50c0d0630 0x7fb50c06b4d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb5000097d0 tx=0x7fb50002b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0afafb4b9b58e39 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.562+0000 7fb50affd640 1 -- 172.21.15.44:0/3303015871 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb500035020 con 0x7fb50c0d0630 2024-04-02T13:31:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.562+0000 7fb50affd640 1 -- 172.21.15.44:0/3303015871 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb500031890 con 0x7fb50c0d0630 2024-04-02T13:31:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.562+0000 7fb50affd640 1 -- 172.21.15.44:0/3303015871 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb500031c60 con 0x7fb50c0d0630 2024-04-02T13:31:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.563+0000 7fb5125eb640 1 -- 172.21.15.44:0/3303015871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb50c0d0630 msgr2=0x7fb50c06b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.563+0000 7fb5125eb640 1 --2- 172.21.15.44:0/3303015871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb50c0d0630 0x7fb50c06b4d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb5000097d0 tx=0x7fb50002b930 comp rx=0 tx=0).stop 2024-04-02T13:31:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.563+0000 7fb5125eb640 1 -- 172.21.15.44:0/3303015871 shutdown_connections 2024-04-02T13:31:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.563+0000 7fb5125eb640 1 --2- 172.21.15.44:0/3303015871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb50c0d0630 0x7fb50c06b4d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.563+0000 7fb5125eb640 1 --2- 172.21.15.44:0/3303015871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb50c15e0f0 0x7fb50c0d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.563+0000 7fb5125eb640 1 -- 172.21.15.44:0/3303015871 >> 172.21.15.44:0/3303015871 conn(0x7fb50c0b2980 msgr2=0x7fb50c0b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.563+0000 7fb5125eb640 1 -- 172.21.15.44:0/3303015871 shutdown_connections 2024-04-02T13:31:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.563+0000 7fb5125eb640 1 -- 172.21.15.44:0/3303015871 wait complete. 2024-04-02T13:31:32.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.564+0000 7fb5125eb640 1 Processor -- start 2024-04-02T13:31:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.564+0000 7fb5125eb640 1 -- start start 2024-04-02T13:31:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb5125eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb50c0d0630 0x7fb50c0b5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb5125eb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb50c15e0f0 0x7fb50c1e9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb5125eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb50c15e4d0 0x7fb50c1e9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb5125eb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c06eda0 con 0x7fb50c15e0f0 2024-04-02T13:31:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb5125eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c0d0af0 con 0x7fb50c0d0630 2024-04-02T13:31:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb5125eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb50c072f60 con 0x7fb50c15e4d0 2024-04-02T13:31:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb50b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb50c15e0f0 0x7fb50c1e9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb50b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb50c15e0f0 0x7fb50c1e9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46260/0 (socket says 172.21.15.44:46260) 2024-04-02T13:31:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb50b7fe640 1 -- 172.21.15.44:0/3006137850 learned_addr learned my addr 172.21.15.44:0/3006137850 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:32.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb50bfff640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb50c0d0630 0x7fb50c0b5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb510b88640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb50c15e4d0 0x7fb50c1e9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb50b7fe640 1 -- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb50c15e4d0 msgr2=0x7fb50c1e9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb50b7fe640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb50c15e4d0 0x7fb50c1e9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb50b7fe640 1 -- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb50c0d0630 msgr2=0x7fb50c0b5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb50b7fe640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb50c0d0630 0x7fb50c0b5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb50b7fe640 1 -- 172.21.15.44:0/3006137850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb50c0b62b0 con 0x7fb50c15e0f0 2024-04-02T13:31:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb50bfff640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb50c0d0630 0x7fb50c0b5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb510b88640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb50c15e4d0 0x7fb50c1e9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.565+0000 7fb50b7fe640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb50c15e0f0 0x7fb50c1e9610 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fb5000098f0 tx=0x7fb5000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.566+0000 7fb508ff9640 1 -- 172.21.15.44:0/3006137850 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb500035040 con 0x7fb50c15e0f0 2024-04-02T13:31:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.566+0000 7fb508ff9640 1 -- 172.21.15.44:0/3006137850 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb500030d20 con 0x7fb50c15e0f0 2024-04-02T13:31:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.566+0000 7fb5125eb640 1 -- 172.21.15.44:0/3006137850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb50c0b4770 con 0x7fb50c15e0f0 2024-04-02T13:31:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.566+0000 7fb508ff9640 1 -- 172.21.15.44:0/3006137850 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb500006e10 con 0x7fb50c15e0f0 2024-04-02T13:31:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.566+0000 7fb5125eb640 1 -- 172.21.15.44:0/3006137850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb50c0b4d30 con 0x7fb50c15e0f0 2024-04-02T13:31:32.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.567+0000 7fb5125eb640 1 -- 172.21.15.44:0/3006137850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb50c0684f0 con 0x7fb50c15e0f0 2024-04-02T13:31:32.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.567+0000 7fb508ff9640 1 -- 172.21.15.44:0/3006137850 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb500038020 con 0x7fb50c15e0f0 2024-04-02T13:31:32.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.568+0000 7fb508ff9640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb4f806c8f0 0x7fb4f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.568+0000 7fb508ff9640 1 -- 172.21.15.44:0/3006137850 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7fb5000ae340 con 0x7fb50c15e0f0 2024-04-02T13:31:32.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.568+0000 7fb50bfff640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb4f806c8f0 0x7fb4f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.569+0000 7fb50bfff640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb4f806c8f0 0x7fb4f806edb0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb50c0b1970 tx=0x7fb4fc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:32.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.572+0000 7fb508ff9640 1 -- 172.21.15.44:0/3006137850 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb50002d040 con 0x7fb50c15e0f0 2024-04-02T13:31:32.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.772+0000 7fb5125eb640 1 -- 172.21.15.44:0/3006137850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb50c1f51c0 con 0x7fb50c15e0f0 2024-04-02T13:31:32.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.772+0000 7fb508ff9640 1 -- 172.21.15.44:0/3006137850 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v121) ==== 80+0+0 (secure 0 0 0) 0x7fb50007c670 con 0x7fb50c15e0f0 2024-04-02T13:31:32.776 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:31:32.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.774+0000 7fb5125eb640 1 -- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb4f806c8f0 msgr2=0x7fb4f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.774+0000 7fb5125eb640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb4f806c8f0 0x7fb4f806edb0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb50c0b1970 tx=0x7fb4fc008040 comp rx=0 tx=0).stop 2024-04-02T13:31:32.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.774+0000 7fb5125eb640 1 -- 172.21.15.44:0/3006137850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb50c15e0f0 msgr2=0x7fb50c1e9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.774+0000 7fb5125eb640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb50c15e0f0 0x7fb50c1e9610 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fb5000098f0 tx=0x7fb5000046c0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.774+0000 7fb5125eb640 1 -- 172.21.15.44:0/3006137850 shutdown_connections 2024-04-02T13:31:32.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.774+0000 7fb5125eb640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb4f806c8f0 0x7fb4f806edb0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.774+0000 7fb5125eb640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb50c15e4d0 0x7fb50c1e9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.774+0000 7fb5125eb640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb50c15e0f0 0x7fb50c1e9610 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.774+0000 7fb5125eb640 1 --2- 172.21.15.44:0/3006137850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb50c0d0630 0x7fb50c0b5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.774+0000 7fb5125eb640 1 -- 172.21.15.44:0/3006137850 >> 172.21.15.44:0/3006137850 conn(0x7fb50c0b2980 msgr2=0x7fb50c0d0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:32.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.774+0000 7fb5125eb640 1 -- 172.21.15.44:0/3006137850 shutdown_connections 2024-04-02T13:31:32.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.774+0000 7fb5125eb640 1 -- 172.21.15.44:0/3006137850 wait complete. 2024-04-02T13:31:32.789 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:31:32.789 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:31:32.822 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:31:32.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.966+0000 7f2eb09e9640 1 Processor -- start 2024-04-02T13:31:32.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.966+0000 7f2eb09e9640 1 -- start start 2024-04-02T13:31:32.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2eb09e9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2eac06d700 0x7f2eac15e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2eb09e9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2eac15ee70 0x7f2eac0d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2eb09e9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eac15fb50 con 0x7f2eac15e550 2024-04-02T13:31:32.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2eb09e9640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f2eac06ca90 con 0x7f2eac15ee70 2024-04-02T13:31:32.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2eb09e9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2eac15f590 con 0x7f2eac06d700 2024-04-02T13:31:32.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2eaad9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f2eac15ee70 0x7f2eac0d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2eaa59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f2eac15e550 0x7f2eac06cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51516/0 (socket says 172.21.15.44:51516) 2024-04-02T13:31:32.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2eaa59c640 1 -- 172.21.15.44:0/3435370245 learned_addr learned my addr 172.21.15.44:0/3435370245 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:32.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2ea9d9b640 1 --2- 172.21.15.44:0/3435370245 >> v2:172.21.15.69:3300/0 conn(0x7f2eac06d700 0x7f2eac15e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2eaad9d640 1 -- 172.21.15.44:0/3435370245 >> v2:172.21.15.69:3300/0 conn(0x7f2eac06d700 msgr2=0x7f2eac15e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:32.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2eaad9d640 1 --2- 172.21.15.44:0/3435370245 >> v2:172.21.15.69:3300/0 conn(0x7f2eac06d700 0x7f2eac15e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2eaad9d640 1 -- 172.21.15.44:0/3435370245 >> v1:172.21.15.44:6789/0 conn(0x7f2eac15e550 legacy=0x7f2eac06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2eaad9d640 1 -- 172.21.15.44:0/3435370245 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac15f320 con 0x7f2eac15ee70 2024-04-02T13:31:32.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2ea959a640 1 -- 172.21.15.44:0/3435370245 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3421391852 0 0) 0x7f2eac15fb50 con 0x7f2eac15e550 2024-04-02T13:31:32.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.967+0000 7f2ea9d9b640 1 --2- 172.21.15.44:0/3435370245 >> v2:172.21.15.69:3300/0 conn(0x7f2eac06d700 0x7f2eac15e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:32.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.968+0000 7f2eaad9d640 1 --2- 172.21.15.44:0/3435370245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eac15ee70 0x7f2eac0d0ab0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f2eac0b4a20 tx=0x7f2ea402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa73dc47af63a14 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:32.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.968+0000 7f2ea959a640 1 -- 172.21.15.44:0/3435370245 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4039070 con 0x7f2eac15ee70 2024-04-02T13:31:32.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.968+0000 7f2ea959a640 1 -- 172.21.15.44:0/3435370245 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea402cd00 con 0x7f2eac15ee70 2024-04-02T13:31:32.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.968+0000 7f2ea959a640 1 -- 172.21.15.44:0/3435370245 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4034660 con 0x7f2eac15ee70 2024-04-02T13:31:32.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.969+0000 7f2eb09e9640 1 -- 172.21.15.44:0/3435370245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eac15ee70 msgr2=0x7f2eac0d0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.969+0000 7f2eb09e9640 1 --2- 172.21.15.44:0/3435370245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eac15ee70 0x7f2eac0d0ab0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f2eac0b4a20 tx=0x7f2ea402bf50 comp rx=0 tx=0).stop 2024-04-02T13:31:32.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.969+0000 7f2eb09e9640 1 -- 172.21.15.44:0/3435370245 shutdown_connections 2024-04-02T13:31:32.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.969+0000 7f2eb09e9640 1 --2- 172.21.15.44:0/3435370245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eac15ee70 0x7f2eac0d0ab0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.969+0000 7f2eb09e9640 1 --2- 172.21.15.44:0/3435370245 >> v2:172.21.15.69:3300/0 conn(0x7f2eac06d700 0x7f2eac15e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.969+0000 7f2eb09e9640 1 -- 172.21.15.44:0/3435370245 >> 172.21.15.44:0/3435370245 conn(0x7f2eac09aa20 msgr2=0x7f2eac09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:32.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.969+0000 7f2eb09e9640 1 -- 172.21.15.44:0/3435370245 shutdown_connections 2024-04-02T13:31:32.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.969+0000 7f2eb09e9640 1 -- 172.21.15.44:0/3435370245 wait complete. 2024-04-02T13:31:32.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.970+0000 7f2eb09e9640 1 Processor -- start 2024-04-02T13:31:32.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.970+0000 7f2eb09e9640 1 -- start start 2024-04-02T13:31:32.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eb09e9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eac06d700 0x7f2eac155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eb09e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eac15e550 0x7f2eac1f5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eb09e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eac15ee70 0x7f2eac0b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eb09e9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac0d0590 con 0x7f2eac06d700 2024-04-02T13:31:32.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eb09e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac0d02b0 con 0x7f2eac15ee70 2024-04-02T13:31:32.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eb09e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2eac0cc4b0 con 0x7f2eac15e550 2024-04-02T13:31:32.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eaa59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eac06d700 0x7f2eac155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eaa59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eac06d700 0x7f2eac155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46282/0 (socket says 172.21.15.44:46282) 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eaa59c640 1 -- 172.21.15.44:0/1070253030 learned_addr learned my addr 172.21.15.44:0/1070253030 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eaad9d640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eac15ee70 0x7f2eac0b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2ea9d9b640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eac15e550 0x7f2eac1f5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eaa59c640 1 -- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eac15e550 msgr2=0x7f2eac1f5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eaa59c640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eac15e550 0x7f2eac1f5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eaa59c640 1 -- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eac15ee70 msgr2=0x7f2eac0b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eaa59c640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eac15ee70 0x7f2eac0b5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eaa59c640 1 -- 172.21.15.44:0/1070253030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac0c90d0 con 0x7f2eac06d700 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eaa59c640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eac06d700 0x7f2eac155180 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c00c430 tx=0x7f2e9c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2e9affd640 1 -- 172.21.15.44:0/1070253030 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c0074e0 con 0x7f2eac06d700 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2e9affd640 1 -- 172.21.15.44:0/1070253030 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e9c0047e0 con 0x7f2eac06d700 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eb09e9640 1 -- 172.21.15.44:0/1070253030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eac0c93b0 con 0x7f2eac06d700 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2e9affd640 1 -- 172.21.15.44:0/1070253030 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c004bd0 con 0x7f2eac06d700 2024-04-02T13:31:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.971+0000 7f2eb09e9640 1 -- 172.21.15.44:0/1070253030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eac0c9970 con 0x7f2eac06d700 2024-04-02T13:31:32.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.972+0000 7f2eb09e9640 1 -- 172.21.15.44:0/1070253030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eac0684f0 con 0x7f2eac06d700 2024-04-02T13:31:32.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.973+0000 7f2e9affd640 1 -- 172.21.15.44:0/1070253030 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2e9c011070 con 0x7f2eac06d700 2024-04-02T13:31:32.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.973+0000 7f2e9affd640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2e8c06c8f0 0x7f2e8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:32.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.973+0000 7f2e9affd640 1 -- 172.21.15.44:0/1070253030 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f2e9c08b9a0 con 0x7f2eac06d700 2024-04-02T13:31:32.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.973+0000 7f2ea9d9b640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2e8c06c8f0 0x7f2e8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.974+0000 7f2ea9d9b640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2e8c06c8f0 0x7f2e8c06edb0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f2eac15c340 tx=0x7f2ea0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:32.978+0000 7f2e9affd640 1 -- 172.21.15.44:0/1070253030 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2e9c059c40 con 0x7f2eac06d700 2024-04-02T13:31:33.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.163+0000 7f2eb09e9640 1 -- 172.21.15.44:0/1070253030 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2eac0b2f80 con 0x7f2eac06d700 2024-04-02T13:31:33.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.164+0000 7f2e9affd640 1 -- 172.21.15.44:0/1070253030 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) ==== 74+0+16904 (secure 0 0 0) 0x7f2e9c059540 con 0x7f2eac06d700 2024-04-02T13:31:33.168 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:33.168 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":121,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:13.125622+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:31:09.752845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-02T13:31:10.437454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:31:33.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.166+0000 7f2eb09e9640 1 -- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2e8c06c8f0 msgr2=0x7f2e8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.166+0000 7f2eb09e9640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2e8c06c8f0 0x7f2e8c06edb0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f2eac15c340 tx=0x7f2ea0008040 comp rx=0 tx=0).stop 2024-04-02T13:31:33.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.166+0000 7f2eb09e9640 1 -- 172.21.15.44:0/1070253030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eac06d700 msgr2=0x7f2eac155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.166+0000 7f2eb09e9640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eac06d700 0x7f2eac155180 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c00c430 tx=0x7f2e9c00c8f0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.166+0000 7f2eb09e9640 1 -- 172.21.15.44:0/1070253030 shutdown_connections 2024-04-02T13:31:33.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.166+0000 7f2eb09e9640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2e8c06c8f0 0x7f2e8c06edb0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.166+0000 7f2eb09e9640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eac15ee70 0x7f2eac0b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.166+0000 7f2eb09e9640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eac15e550 0x7f2eac1f5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.166+0000 7f2eb09e9640 1 --2- 172.21.15.44:0/1070253030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eac06d700 0x7f2eac155180 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.166+0000 7f2eb09e9640 1 -- 172.21.15.44:0/1070253030 >> 172.21.15.44:0/1070253030 conn(0x7f2eac09aa20 msgr2=0x7f2eac15d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:33.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.166+0000 7f2eb09e9640 1 -- 172.21.15.44:0/1070253030 shutdown_connections 2024-04-02T13:31:33.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.166+0000 7f2eb09e9640 1 -- 172.21.15.44:0/1070253030 wait complete. 2024-04-02T13:31:33.181 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:31:09.752845+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:31:10.437454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:31:33.181 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:31:33.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.283+0000 7f8048504640 1 Processor -- start 2024-04-02T13:31:33.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.283+0000 7f8048504640 1 -- start start 2024-04-02T13:31:33.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f8048504640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80400d07b0 0x7f80400ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f8048504640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f80400cf400 0x7f804015c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f8048504640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8040160b10 con 0x7f80401565d0 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f8048504640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f80400cf8b0 con 0x7f80400cf400 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f8048504640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f80400cfad0 con 0x7f80400d07b0 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f8046aa1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f80400cf400 0x7f804015c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f80462a0640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f80401565d0 0x7f80400d00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51520/0 (socket says 172.21.15.44:51520) 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f80462a0640 1 -- 172.21.15.44:0/3313647277 learned_addr learned my addr 172.21.15.44:0/3313647277 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f8045a9f640 1 --2- 172.21.15.44:0/3313647277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80400d07b0 0x7f80400ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f8046aa1640 1 -- 172.21.15.44:0/3313647277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80400d07b0 msgr2=0x7f80400ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f8046aa1640 1 --2- 172.21.15.44:0/3313647277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80400d07b0 0x7f80400ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f8046aa1640 1 -- 172.21.15.44:0/3313647277 >> v1:172.21.15.44:6789/0 conn(0x7f80401565d0 legacy=0x7f80400d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f804529e640 1 -- 172.21.15.44:0/3313647277 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 480799605 0 0) 0x7f8040160b10 con 0x7f80401565d0 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f8046aa1640 1 -- 172.21.15.44:0/3313647277 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f804015fd50 con 0x7f80400cf400 2024-04-02T13:31:33.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.284+0000 7f8045a9f640 1 --2- 172.21.15.44:0/3313647277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80400d07b0 0x7f80400ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.285+0000 7f8046aa1640 1 --2- 172.21.15.44:0/3313647277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80400cf400 0x7f804015c7a0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f8038009d90 tx=0x7f803802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5fbd688f996e676 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.285+0000 7f804529e640 1 -- 172.21.15.44:0/3313647277 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8038039070 con 0x7f80400cf400 2024-04-02T13:31:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.285+0000 7f804529e640 1 -- 172.21.15.44:0/3313647277 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f803802cbc0 con 0x7f80400cf400 2024-04-02T13:31:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.285+0000 7f804529e640 1 -- 172.21.15.44:0/3313647277 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8038034440 con 0x7f80400cf400 2024-04-02T13:31:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.286+0000 7f8048504640 1 -- 172.21.15.44:0/3313647277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80400cf400 msgr2=0x7f804015c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.286+0000 7f8048504640 1 --2- 172.21.15.44:0/3313647277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80400cf400 0x7f804015c7a0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f8038009d90 tx=0x7f803802be10 comp rx=0 tx=0).stop 2024-04-02T13:31:33.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.286+0000 7f8048504640 1 -- 172.21.15.44:0/3313647277 shutdown_connections 2024-04-02T13:31:33.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.286+0000 7f8048504640 1 --2- 172.21.15.44:0/3313647277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80400cf400 0x7f804015c7a0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.286+0000 7f8048504640 1 --2- 172.21.15.44:0/3313647277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80400d07b0 0x7f80400ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.286+0000 7f8048504640 1 -- 172.21.15.44:0/3313647277 >> 172.21.15.44:0/3313647277 conn(0x7f804006b920 msgr2=0x7f804006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:33.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.286+0000 7f8048504640 1 -- 172.21.15.44:0/3313647277 shutdown_connections 2024-04-02T13:31:33.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.286+0000 7f8048504640 1 -- 172.21.15.44:0/3313647277 wait complete. 2024-04-02T13:31:33.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.287+0000 7f8048504640 1 Processor -- start 2024-04-02T13:31:33.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.287+0000 7f8048504640 1 -- start start 2024-04-02T13:31:33.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8048504640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80400cf400 0x7f80401576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8048504640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80400d07b0 0x7f8040154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8048504640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80401565d0 0x7f804014e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8048504640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f80401605f0 con 0x7f80400cf400 2024-04-02T13:31:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8048504640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8040160310 con 0x7f80400d07b0 2024-04-02T13:31:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8048504640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f80401569b0 con 0x7f80401565d0 2024-04-02T13:31:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f80462a0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80400cf400 0x7f80401576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f80462a0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80400cf400 0x7f80401576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46300/0 (socket says 172.21.15.44:46300) 2024-04-02T13:31:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f80462a0640 1 -- 172.21.15.44:0/764199733 learned_addr learned my addr 172.21.15.44:0/764199733 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8045a9f640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80400d07b0 0x7f8040154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8046aa1640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80401565d0 0x7f804014e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f80462a0640 1 -- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80401565d0 msgr2=0x7f804014e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f80462a0640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80401565d0 0x7f804014e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f80462a0640 1 -- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80400d07b0 msgr2=0x7f8040154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f80462a0640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80400d07b0 0x7f8040154660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f80462a0640 1 -- 172.21.15.44:0/764199733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f804014e900 con 0x7f80400cf400 2024-04-02T13:31:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8045a9f640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80400d07b0 0x7f8040154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f80462a0640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80400cf400 0x7f80401576b0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f803c00c500 tx=0x7f803c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8036ffd640 1 -- 172.21.15.44:0/764199733 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803c018070 con 0x7f80400cf400 2024-04-02T13:31:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8036ffd640 1 -- 172.21.15.44:0/764199733 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f803c007580 con 0x7f80400cf400 2024-04-02T13:31:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8048504640 1 -- 172.21.15.44:0/764199733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8040157d70 con 0x7f80400cf400 2024-04-02T13:31:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8036ffd640 1 -- 172.21.15.44:0/764199733 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803c004a20 con 0x7f80400cf400 2024-04-02T13:31:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.288+0000 7f8048504640 1 -- 172.21.15.44:0/764199733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8040158b00 con 0x7f80400cf400 2024-04-02T13:31:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.289+0000 7f8048504640 1 -- 172.21.15.44:0/764199733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80400684f0 con 0x7f80400cf400 2024-04-02T13:31:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.290+0000 7f8036ffd640 1 -- 172.21.15.44:0/764199733 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f803c01b070 con 0x7f80400cf400 2024-04-02T13:31:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.290+0000 7f8036ffd640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f802406c920 0x7f802406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.290+0000 7f8036ffd640 1 -- 172.21.15.44:0/764199733 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f803c08af90 con 0x7f80400cf400 2024-04-02T13:31:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.291+0000 7f8045a9f640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f802406c920 0x7f802406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.291+0000 7f8045a9f640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f802406c920 0x7f802406ede0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8030006fd0 tx=0x7f8030008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:33.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.295+0000 7f8036ffd640 1 -- 172.21.15.44:0/764199733 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f803c059230 con 0x7f80400cf400 2024-04-02T13:31:33.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.479+0000 7f8048504640 1 -- 172.21.15.44:0/764199733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f80400ce540 con 0x7f80400cf400 2024-04-02T13:31:33.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.479+0000 7f8036ffd640 1 -- 172.21.15.44:0/764199733 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) ==== 93+0+10 (secure 0 0 0) 0x7f803c058b30 con 0x7f80400cf400 2024-04-02T13:31:33.483 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:31:33.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.481+0000 7f8048504640 1 -- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f802406c920 msgr2=0x7f802406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.481+0000 7f8048504640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f802406c920 0x7f802406ede0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8030006fd0 tx=0x7f8030008040 comp rx=0 tx=0).stop 2024-04-02T13:31:33.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.481+0000 7f8048504640 1 -- 172.21.15.44:0/764199733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80400cf400 msgr2=0x7f80401576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.481+0000 7f8048504640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80400cf400 0x7f80401576b0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f803c00c500 tx=0x7f803c00c9c0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.481+0000 7f8048504640 1 -- 172.21.15.44:0/764199733 shutdown_connections 2024-04-02T13:31:33.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.481+0000 7f8048504640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f802406c920 0x7f802406ede0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.481+0000 7f8048504640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80401565d0 0x7f804014e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.481+0000 7f8048504640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80400d07b0 0x7f8040154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.481+0000 7f8048504640 1 --2- 172.21.15.44:0/764199733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80400cf400 0x7f80401576b0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.481+0000 7f8048504640 1 -- 172.21.15.44:0/764199733 >> 172.21.15.44:0/764199733 conn(0x7f804006b920 msgr2=0x7f804006bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:33.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.481+0000 7f8048504640 1 -- 172.21.15.44:0/764199733 shutdown_connections 2024-04-02T13:31:33.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.481+0000 7f8048504640 1 -- 172.21.15.44:0/764199733 wait complete. 2024-04-02T13:31:33.496 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:31:33.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.601+0000 7f91faa3c640 1 Processor -- start 2024-04-02T13:31:33.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.601+0000 7f91faa3c640 1 -- start start 2024-04-02T13:31:33.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.601+0000 7f91faa3c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91f40d0840 0x7f91f4165c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.601+0000 7f91faa3c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91f40cdc20 con 0x7f91f41554c0 2024-04-02T13:31:33.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.601+0000 7f91faa3c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91f40cdec0 con 0x7f91f40ce9a0 2024-04-02T13:31:33.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.601+0000 7f91faa3c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f91f40ce1a0 con 0x7f91f40d0840 2024-04-02T13:31:33.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f3fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f91f41554c0 0x7f91f4071120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51532/0 (socket says 172.21.15.44:51532) 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f3fff640 1 -- 172.21.15.44:0/1937628269 learned_addr learned my addr 172.21.15.44:0/1937628269 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f8fd9640 1 --2- 172.21.15.44:0/1937628269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91f40d0840 0x7f91f4165c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 360033164 0 0) 0x7f91f40cdc20 con 0x7f91f41554c0 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91dc003520 con 0x7f91f41554c0 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 820529464 0 0) 0x7f91dc003520 con 0x7f91f41554c0 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91f40d0840 msgr2=0x7f91f4165c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 --2- 172.21.15.44:0/1937628269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91f40d0840 0x7f91f4165c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 >> v1:172.21.15.69:6789/0 conn(0x7f91f40ce9a0 legacy=0x7f91f40d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f40ce480 con 0x7f91f41554c0 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f91e4001670 con 0x7f91f41554c0 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f8fd9640 1 --2- 172.21.15.44:0/1937628269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91f40d0840 0x7f91f4165c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 >> v1:172.21.15.44:6789/0 conn(0x7f91f41554c0 legacy=0x7f91f4071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 --2- 172.21.15.44:0/1937628269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91f40d0840 0x7f91dc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 --2- 172.21.15.44:0/1937628269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91dc003fd0 0x7f91dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 --2- 172.21.15.44:0/1937628269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91dc004d40 0x7f91dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f91f40ce1a0 con 0x7f91dc003fd0 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc00bb20 con 0x7f91dc004d40 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f91dc00bd20 con 0x7f91f40d0840 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f37fe640 1 --2- 172.21.15.44:0/1937628269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91dc003fd0 0x7f91dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f3fff640 1 --2- 172.21.15.44:0/1937628269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91f40d0840 0x7f91dc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.602+0000 7f91f8fd9640 1 --2- 172.21.15.44:0/1937628269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91dc004d40 0x7f91dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.603+0000 7f91f37fe640 1 -- 172.21.15.44:0/1937628269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91f40d0840 msgr2=0x7f91dc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.603+0000 7f91f37fe640 1 --2- 172.21.15.44:0/1937628269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91f40d0840 0x7f91dc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.603+0000 7f91f37fe640 1 -- 172.21.15.44:0/1937628269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91dc004d40 msgr2=0x7f91dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.603+0000 7f91f37fe640 1 --2- 172.21.15.44:0/1937628269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91dc004d40 0x7f91dc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.603+0000 7f91f37fe640 1 -- 172.21.15.44:0/1937628269 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91dc003520 con 0x7f91dc003fd0 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.603+0000 7f91f37fe640 1 --2- 172.21.15.44:0/1937628269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91dc003fd0 0x7f91dc004800 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f91e8001720 tx=0x7f91dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5478e90dca36e54b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.604+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8001370 con 0x7f91dc003fd0 2024-04-02T13:31:33.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.604+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91e80052d0 con 0x7f91dc003fd0 2024-04-02T13:31:33.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.604+0000 7f91f2ffd640 1 -- 172.21.15.44:0/1937628269 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8003830 con 0x7f91dc003fd0 2024-04-02T13:31:33.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.604+0000 7f91faa3c640 1 -- 172.21.15.44:0/1937628269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91dc003fd0 msgr2=0x7f91dc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.604+0000 7f91faa3c640 1 --2- 172.21.15.44:0/1937628269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91dc003fd0 0x7f91dc004800 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f91e8001720 tx=0x7f91dc005830 comp rx=0 tx=0).stop 2024-04-02T13:31:33.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.604+0000 7f91f8fd9640 1 -- 172.21.15.44:0/1937628269 reap_dead start 2024-04-02T13:31:33.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.604+0000 7f91faa3c640 1 -- 172.21.15.44:0/1937628269 shutdown_connections 2024-04-02T13:31:33.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.604+0000 7f91faa3c640 1 -- 172.21.15.44:0/1937628269 >> 172.21.15.44:0/1937628269 conn(0x7f91f406b920 msgr2=0x7f91f406bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:33.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.604+0000 7f91faa3c640 1 -- 172.21.15.44:0/1937628269 shutdown_connections 2024-04-02T13:31:33.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.604+0000 7f91faa3c640 1 -- 172.21.15.44:0/1937628269 wait complete. 2024-04-02T13:31:33.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.605+0000 7f91faa3c640 1 Processor -- start 2024-04-02T13:31:33.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.605+0000 7f91faa3c640 1 -- start start 2024-04-02T13:31:33.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.606+0000 7f91faa3c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91dc003fd0 0x7f91f415a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.606+0000 7f91faa3c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91f4155b40 0x7f91f4157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.606+0000 7f91faa3c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4157b60 0x7f91f414a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.606+0000 7f91faa3c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f91f415c290 con 0x7f91f4157b60 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.606+0000 7f91faa3c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f91f4071830 con 0x7f91f4155b40 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.606+0000 7f91faa3c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f91f4156c30 con 0x7f91dc003fd0 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.606+0000 7f91f8fd9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4157b60 0x7f91f414a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.606+0000 7f91f8fd9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4157b60 0x7f91f414a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46326/0 (socket says 172.21.15.44:46326) 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.606+0000 7f91f8fd9640 1 -- 172.21.15.44:0/3606013990 learned_addr learned my addr 172.21.15.44:0/3606013990 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.606+0000 7f91f37fe640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91f4155b40 0x7f91f4157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.606+0000 7f91f3fff640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91dc003fd0 0x7f91f415a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.607+0000 7f91f8fd9640 1 -- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91dc003fd0 msgr2=0x7f91f415a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.607+0000 7f91f8fd9640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91dc003fd0 0x7f91f415a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.607+0000 7f91f8fd9640 1 -- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91f4155b40 msgr2=0x7f91f4157620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.607+0000 7f91f8fd9640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91f4155b40 0x7f91f4157620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.607+0000 7f91f8fd9640 1 -- 172.21.15.44:0/3606013990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f4156020 con 0x7f91f4157b60 2024-04-02T13:31:33.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.607+0000 7f91f37fe640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91f4155b40 0x7f91f4157620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:33.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.607+0000 7f91f8fd9640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4157b60 0x7f91f414a2a0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f91f406cf30 tx=0x7f91ec010530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:33.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.607+0000 7f91f0ff9640 1 -- 172.21.15.44:0/3606013990 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec01c070 con 0x7f91f4157b60 2024-04-02T13:31:33.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.607+0000 7f91f0ff9640 1 -- 172.21.15.44:0/3606013990 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91ec010cc0 con 0x7f91f4157b60 2024-04-02T13:31:33.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.607+0000 7f91faa3c640 1 -- 172.21.15.44:0/3606013990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f4156300 con 0x7f91f4157b60 2024-04-02T13:31:33.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.607+0000 7f91f0ff9640 1 -- 172.21.15.44:0/3606013990 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec003bf0 con 0x7f91f4157b60 2024-04-02T13:31:33.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.607+0000 7f91faa3c640 1 -- 172.21.15.44:0/3606013990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f41568c0 con 0x7f91f4157b60 2024-04-02T13:31:33.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.608+0000 7f91faa3c640 1 -- 172.21.15.44:0/3606013990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91f40684f0 con 0x7f91f4157b60 2024-04-02T13:31:33.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.608+0000 7f91f0ff9640 1 -- 172.21.15.44:0/3606013990 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f91ec01f070 con 0x7f91f4157b60 2024-04-02T13:31:33.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.609+0000 7f91f0ff9640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f91e006c8f0 0x7f91e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.609+0000 7f91f0ff9640 1 -- 172.21.15.44:0/3606013990 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f91ec08bdb0 con 0x7f91f4157b60 2024-04-02T13:31:33.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.609+0000 7f91f3fff640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f91e006c8f0 0x7f91e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.610+0000 7f91f3fff640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f91e006c8f0 0x7f91e006edb0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f91e4007c30 tx=0x7f91e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:33.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.613+0000 7f91f0ff9640 1 -- 172.21.15.44:0/3606013990 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f91ec05a050 con 0x7f91f4157b60 2024-04-02T13:31:33.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.796+0000 7f91faa3c640 1 -- 172.21.15.44:0/3606013990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f91f40b5170 con 0x7f91f4157b60 2024-04-02T13:31:33.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.796+0000 7f91f0ff9640 1 -- 172.21.15.44:0/3606013990 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v121) ==== 104+0+11 (secure 0 0 0) 0x7f91ec059950 con 0x7f91f4157b60 2024-04-02T13:31:33.800 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:31:33.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.798+0000 7f91faa3c640 1 -- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f91e006c8f0 msgr2=0x7f91e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.798+0000 7f91faa3c640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f91e006c8f0 0x7f91e006edb0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f91e4007c30 tx=0x7f91e4009040 comp rx=0 tx=0).stop 2024-04-02T13:31:33.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.798+0000 7f91faa3c640 1 -- 172.21.15.44:0/3606013990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4157b60 msgr2=0x7f91f414a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.798+0000 7f91faa3c640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4157b60 0x7f91f414a2a0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f91f406cf30 tx=0x7f91ec010530 comp rx=0 tx=0).stop 2024-04-02T13:31:33.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.798+0000 7f91faa3c640 1 -- 172.21.15.44:0/3606013990 shutdown_connections 2024-04-02T13:31:33.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.798+0000 7f91faa3c640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f91e006c8f0 0x7f91e006edb0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.798+0000 7f91faa3c640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4157b60 0x7f91f414a2a0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.798+0000 7f91faa3c640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91f4155b40 0x7f91f4157620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.798+0000 7f91faa3c640 1 --2- 172.21.15.44:0/3606013990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91dc003fd0 0x7f91f415a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.798+0000 7f91faa3c640 1 -- 172.21.15.44:0/3606013990 >> 172.21.15.44:0/3606013990 conn(0x7f91f406b920 msgr2=0x7f91f4071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.798+0000 7f91faa3c640 1 -- 172.21.15.44:0/3606013990 shutdown_connections 2024-04-02T13:31:33.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.798+0000 7f91faa3c640 1 -- 172.21.15.44:0/3606013990 wait complete. 2024-04-02T13:31:33.813 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:31:33.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.913+0000 7fcd5ee31640 1 Processor -- start 2024-04-02T13:31:33.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.913+0000 7fcd5ee31640 1 -- start start 2024-04-02T13:31:33.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.914+0000 7fcd5ee31640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcd580ce9a0 0x7fcd580d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.914+0000 7fcd5ee31640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcd580d0670 0x7fcd58164c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.914+0000 7fcd5ee31640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd580cdc20 con 0x7fcd581554c0 2024-04-02T13:31:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.914+0000 7fcd5ee31640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fcd580d0aa0 con 0x7fcd580d0670 2024-04-02T13:31:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.914+0000 7fcd5ee31640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcd580cdec0 con 0x7fcd580ce9a0 2024-04-02T13:31:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.914+0000 7fcd5cbcd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fcd581554c0 0x7fcd58071120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51544/0 (socket says 172.21.15.44:51544) 2024-04-02T13:31:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.914+0000 7fcd5cbcd640 1 -- 172.21.15.44:0/1105626745 learned_addr learned my addr 172.21.15.44:0/1105626745 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.914+0000 7fcd5d3ce640 1 --2- 172.21.15.44:0/1105626745 >> v2:172.21.15.44:3300/0 conn(0x7fcd580d0670 0x7fcd58164c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.915+0000 7fcd57fff640 1 --2- 172.21.15.44:0/1105626745 >> v2:172.21.15.69:3300/0 conn(0x7fcd580ce9a0 0x7fcd580d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.915+0000 7fcd577fe640 1 -- 172.21.15.44:0/1105626745 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3448284770 0 0) 0x7fcd580cdc20 con 0x7fcd581554c0 2024-04-02T13:31:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.915+0000 7fcd5d3ce640 1 -- 172.21.15.44:0/1105626745 >> v2:172.21.15.69:3300/0 conn(0x7fcd580ce9a0 msgr2=0x7fcd580d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.915+0000 7fcd5d3ce640 1 --2- 172.21.15.44:0/1105626745 >> v2:172.21.15.69:3300/0 conn(0x7fcd580ce9a0 0x7fcd580d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.915+0000 7fcd5d3ce640 1 -- 172.21.15.44:0/1105626745 >> v1:172.21.15.44:6789/0 conn(0x7fcd581554c0 legacy=0x7fcd58071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.915+0000 7fcd5d3ce640 1 -- 172.21.15.44:0/1105626745 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd580ce1a0 con 0x7fcd580d0670 2024-04-02T13:31:33.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.915+0000 7fcd5d3ce640 1 --2- 172.21.15.44:0/1105626745 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd580d0670 0x7fcd58164c10 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fcd580cf460 tx=0x7fcd4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f543aa4185b28926 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:33.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.916+0000 7fcd577fe640 1 -- 172.21.15.44:0/1105626745 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd40035020 con 0x7fcd580d0670 2024-04-02T13:31:33.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.916+0000 7fcd577fe640 1 -- 172.21.15.44:0/1105626745 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd40031830 con 0x7fcd580d0670 2024-04-02T13:31:33.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.916+0000 7fcd577fe640 1 -- 172.21.15.44:0/1105626745 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd40031c00 con 0x7fcd580d0670 2024-04-02T13:31:33.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.916+0000 7fcd5ee31640 1 -- 172.21.15.44:0/1105626745 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd580d0670 msgr2=0x7fcd58164c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.916+0000 7fcd5ee31640 1 --2- 172.21.15.44:0/1105626745 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd580d0670 0x7fcd58164c10 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fcd580cf460 tx=0x7fcd4002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.916+0000 7fcd5ee31640 1 -- 172.21.15.44:0/1105626745 shutdown_connections 2024-04-02T13:31:33.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.916+0000 7fcd5ee31640 1 --2- 172.21.15.44:0/1105626745 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd580d0670 0x7fcd58164c10 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.916+0000 7fcd5ee31640 1 --2- 172.21.15.44:0/1105626745 >> v2:172.21.15.69:3300/0 conn(0x7fcd580ce9a0 0x7fcd580d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.916+0000 7fcd5ee31640 1 -- 172.21.15.44:0/1105626745 >> 172.21.15.44:0/1105626745 conn(0x7fcd5806b920 msgr2=0x7fcd5806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:33.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.916+0000 7fcd5ee31640 1 -- 172.21.15.44:0/1105626745 shutdown_connections 2024-04-02T13:31:33.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.916+0000 7fcd5ee31640 1 -- 172.21.15.44:0/1105626745 wait complete. 2024-04-02T13:31:33.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.917+0000 7fcd5ee31640 1 Processor -- start 2024-04-02T13:31:33.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.917+0000 7fcd5ee31640 1 -- start start 2024-04-02T13:31:33.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.918+0000 7fcd5ee31640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580ce9a0 0x7fcd5814d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.918+0000 7fcd5ee31640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd580d0670 0x7fcd58159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.918+0000 7fcd5ee31640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd581554c0 0x7fcd580aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.918+0000 7fcd5ee31640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5815c290 con 0x7fcd580d0670 2024-04-02T13:31:33.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.918+0000 7fcd5ee31640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcd581558a0 con 0x7fcd581554c0 2024-04-02T13:31:33.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.918+0000 7fcd5ee31640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcd5814dcb0 con 0x7fcd580ce9a0 2024-04-02T13:31:33.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.918+0000 7fcd57fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd580d0670 0x7fcd58159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.918+0000 7fcd57fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd580d0670 0x7fcd58159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46344/0 (socket says 172.21.15.44:46344) 2024-04-02T13:31:33.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.918+0000 7fcd57fff640 1 -- 172.21.15.44:0/461680009 learned_addr learned my addr 172.21.15.44:0/461680009 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:33.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.918+0000 7fcd5d3ce640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd581554c0 0x7fcd580aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.918+0000 7fcd5cbcd640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580ce9a0 0x7fcd5814d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.919+0000 7fcd57fff640 1 -- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580ce9a0 msgr2=0x7fcd5814d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.919+0000 7fcd57fff640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580ce9a0 0x7fcd5814d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.919+0000 7fcd57fff640 1 -- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd581554c0 msgr2=0x7fcd580aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:33.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.919+0000 7fcd57fff640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd581554c0 0x7fcd580aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:33.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.919+0000 7fcd57fff640 1 -- 172.21.15.44:0/461680009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5815abd0 con 0x7fcd580d0670 2024-04-02T13:31:33.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.919+0000 7fcd5cbcd640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580ce9a0 0x7fcd5814d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:33.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.919+0000 7fcd57fff640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd580d0670 0x7fcd58159a30 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fcd48007ef0 tx=0x7fcd4800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:33.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.919+0000 7fcd557fa640 1 -- 172.21.15.44:0/461680009 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48014620 con 0x7fcd580d0670 2024-04-02T13:31:33.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.919+0000 7fcd557fa640 1 -- 172.21.15.44:0/461680009 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd48019070 con 0x7fcd580d0670 2024-04-02T13:31:33.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.919+0000 7fcd5ee31640 1 -- 172.21.15.44:0/461680009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd5815a330 con 0x7fcd580d0670 2024-04-02T13:31:33.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.919+0000 7fcd557fa640 1 -- 172.21.15.44:0/461680009 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48014a00 con 0x7fcd580d0670 2024-04-02T13:31:33.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.919+0000 7fcd5ee31640 1 -- 172.21.15.44:0/461680009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd5815a8f0 con 0x7fcd580d0670 2024-04-02T13:31:33.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.920+0000 7fcd5ee31640 1 -- 172.21.15.44:0/461680009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd580684f0 con 0x7fcd580d0670 2024-04-02T13:31:33.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.920+0000 7fcd557fa640 1 -- 172.21.15.44:0/461680009 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcd48014ba0 con 0x7fcd580d0670 2024-04-02T13:31:33.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.921+0000 7fcd557fa640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcd3806c8f0 0x7fcd3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:33.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.921+0000 7fcd557fa640 1 -- 172.21.15.44:0/461680009 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7fcd4808c5b0 con 0x7fcd580d0670 2024-04-02T13:31:33.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.921+0000 7fcd5cbcd640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcd3806c8f0 0x7fcd3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:33.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.921+0000 7fcd5cbcd640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcd3806c8f0 0x7fcd3806edb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fcd50000f60 tx=0x7fcd50009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:33.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:33.925+0000 7fcd557fa640 1 -- 172.21.15.44:0/461680009 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fcd4805a850 con 0x7fcd580d0670 2024-04-02T13:31:34.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.109+0000 7fcd5ee31640 1 -- 172.21.15.44:0/461680009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcd580b5170 con 0x7fcd580d0670 2024-04-02T13:31:34.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.110+0000 7fcd557fa640 1 -- 172.21.15.44:0/461680009 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v121) ==== 100+0+11 (secure 0 0 0) 0x7fcd4805a150 con 0x7fcd580d0670 2024-04-02T13:31:34.114 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:31:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.112+0000 7fcd5ee31640 1 -- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcd3806c8f0 msgr2=0x7fcd3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.112+0000 7fcd5ee31640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcd3806c8f0 0x7fcd3806edb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fcd50000f60 tx=0x7fcd50009040 comp rx=0 tx=0).stop 2024-04-02T13:31:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.112+0000 7fcd5ee31640 1 -- 172.21.15.44:0/461680009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd580d0670 msgr2=0x7fcd58159a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.112+0000 7fcd5ee31640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd580d0670 0x7fcd58159a30 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fcd48007ef0 tx=0x7fcd4800e4b0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.112+0000 7fcd5ee31640 1 -- 172.21.15.44:0/461680009 shutdown_connections 2024-04-02T13:31:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.112+0000 7fcd5ee31640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcd3806c8f0 0x7fcd3806edb0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.112+0000 7fcd5ee31640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd581554c0 0x7fcd580aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.112+0000 7fcd5ee31640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd580d0670 0x7fcd58159a30 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.112+0000 7fcd5ee31640 1 --2- 172.21.15.44:0/461680009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580ce9a0 0x7fcd5814d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.112+0000 7fcd5ee31640 1 -- 172.21.15.44:0/461680009 >> 172.21.15.44:0/461680009 conn(0x7fcd5806b920 msgr2=0x7fcd5806da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:34.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.112+0000 7fcd5ee31640 1 -- 172.21.15.44:0/461680009 shutdown_connections 2024-04-02T13:31:34.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.112+0000 7fcd5ee31640 1 -- 172.21.15.44:0/461680009 wait complete. 2024-04-02T13:31:34.126 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:31:34.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.225+0000 7fe50ce3c640 1 Processor -- start 2024-04-02T13:31:34.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.225+0000 7fe50ce3c640 1 -- start start 2024-04-02T13:31:34.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.226+0000 7fe50ce3c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe5080d0a30 0x7fe50806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:34.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.226+0000 7fe50ce3c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe5080d0e10 0x7fe5080cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:34.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.226+0000 7fe50ce3c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe50806c1d0 con 0x7fe5080cfe30 2024-04-02T13:31:34.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.226+0000 7fe50ce3c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe50806cb00 con 0x7fe5080d0a30 2024-04-02T13:31:34.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.226+0000 7fe50ce3c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe50806c4b0 con 0x7fe5080d0e10 2024-04-02T13:31:34.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.226+0000 7fe50659c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe5080d0a30 0x7fe50806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:34.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.226+0000 7fe50659c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe5080d0a30 0x7fe50806cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46346/0 (socket says 172.21.15.44:46346) 2024-04-02T13:31:34.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.226+0000 7fe50659c640 1 -- 172.21.15.44:0/3509833739 learned_addr learned my addr 172.21.15.44:0/3509833739 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:34.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.227+0000 7fe505d9b640 1 --2- 172.21.15.44:0/3509833739 >> v2:172.21.15.69:3300/0 conn(0x7fe5080d0e10 0x7fe5080cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:34.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.227+0000 7fe50659c640 1 -- 172.21.15.44:0/3509833739 >> v2:172.21.15.69:3300/0 conn(0x7fe5080d0e10 msgr2=0x7fe5080cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:34.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.227+0000 7fe50659c640 1 --2- 172.21.15.44:0/3509833739 >> v2:172.21.15.69:3300/0 conn(0x7fe5080d0e10 0x7fe5080cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.227+0000 7fe50659c640 1 -- 172.21.15.44:0/3509833739 >> v1:172.21.15.44:6789/0 conn(0x7fe5080cfe30 legacy=0x7fe5080d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:34.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.227+0000 7fe50559a640 1 -- 172.21.15.44:0/3509833739 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 393306377 0 0) 0x7fe50806c1d0 con 0x7fe5080cfe30 2024-04-02T13:31:34.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.227+0000 7fe50659c640 1 -- 172.21.15.44:0/3509833739 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50806c790 con 0x7fe5080d0a30 2024-04-02T13:31:34.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.227+0000 7fe505d9b640 1 --2- 172.21.15.44:0/3509833739 >> v2:172.21.15.69:3300/0 conn(0x7fe5080d0e10 0x7fe5080cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:34.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.228+0000 7fe50659c640 1 --2- 172.21.15.44:0/3509833739 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5080d0a30 0x7fe50806cf90 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fe5080b6d30 tx=0x7fe50002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675831c180f618af server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:34.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.228+0000 7fe50559a640 1 -- 172.21.15.44:0/3509833739 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe500035020 con 0x7fe5080d0a30 2024-04-02T13:31:34.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.228+0000 7fe50559a640 1 -- 172.21.15.44:0/3509833739 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe500031870 con 0x7fe5080d0a30 2024-04-02T13:31:34.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.228+0000 7fe50559a640 1 -- 172.21.15.44:0/3509833739 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe500031c40 con 0x7fe5080d0a30 2024-04-02T13:31:34.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.228+0000 7fe50ce3c640 1 -- 172.21.15.44:0/3509833739 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5080d0a30 msgr2=0x7fe50806cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:34.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.228+0000 7fe50ce3c640 1 --2- 172.21.15.44:0/3509833739 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5080d0a30 0x7fe50806cf90 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fe5080b6d30 tx=0x7fe50002b910 comp rx=0 tx=0).stop 2024-04-02T13:31:34.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.228+0000 7fe50ce3c640 1 -- 172.21.15.44:0/3509833739 shutdown_connections 2024-04-02T13:31:34.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.228+0000 7fe50ce3c640 1 --2- 172.21.15.44:0/3509833739 >> v2:172.21.15.69:3300/0 conn(0x7fe5080d0e10 0x7fe5080cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.228+0000 7fe50ce3c640 1 --2- 172.21.15.44:0/3509833739 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5080d0a30 0x7fe50806cf90 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.228+0000 7fe50ce3c640 1 -- 172.21.15.44:0/3509833739 >> 172.21.15.44:0/3509833739 conn(0x7fe50809d470 msgr2=0x7fe5080d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:34.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.228+0000 7fe50ce3c640 1 -- 172.21.15.44:0/3509833739 shutdown_connections 2024-04-02T13:31:34.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.228+0000 7fe50ce3c640 1 -- 172.21.15.44:0/3509833739 wait complete. 2024-04-02T13:31:34.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.230+0000 7fe50ce3c640 1 Processor -- start 2024-04-02T13:31:34.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.230+0000 7fe50ce3c640 1 -- start start 2024-04-02T13:31:34.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.230+0000 7fe50ce3c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5080cfe30 0x7fe508152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:34.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.230+0000 7fe50ce3c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5080d0a30 0x7fe5080ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:34.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.230+0000 7fe50ce3c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5080d0e10 0x7fe5080cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.230+0000 7fe50ce3c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe50815e870 con 0x7fe5080d0e10 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.230+0000 7fe50ce3c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe50815e6f0 con 0x7fe5080cfe30 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.230+0000 7fe50ce3c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe5080d4da0 con 0x7fe5080d0a30 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.230+0000 7fe506d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5080d0e10 0x7fe5080cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.230+0000 7fe506d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5080d0e10 0x7fe5080cd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46352/0 (socket says 172.21.15.44:46352) 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.230+0000 7fe506d9d640 1 -- 172.21.15.44:0/1925078682 learned_addr learned my addr 172.21.15.44:0/1925078682 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe50659c640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5080cfe30 0x7fe508152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe505d9b640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5080d0a30 0x7fe5080ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe506d9d640 1 -- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5080d0a30 msgr2=0x7fe5080ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe506d9d640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5080d0a30 0x7fe5080ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe506d9d640 1 -- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5080cfe30 msgr2=0x7fe508152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe506d9d640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5080cfe30 0x7fe508152f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe506d9d640 1 -- 172.21.15.44:0/1925078682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe508153490 con 0x7fe5080d0e10 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe505d9b640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5080d0a30 0x7fe5080ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:34.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe506d9d640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5080d0e10 0x7fe5080cd2f0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fe4f800c780 tx=0x7fe4f800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:34.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe4feffd640 1 -- 172.21.15.44:0/1925078682 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f8003da0 con 0x7fe5080d0e10 2024-04-02T13:31:34.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe4feffd640 1 -- 172.21.15.44:0/1925078682 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4f800ce30 con 0x7fe5080d0e10 2024-04-02T13:31:34.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe50ce3c640 1 -- 172.21.15.44:0/1925078682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5080cbeb0 con 0x7fe5080d0e10 2024-04-02T13:31:34.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe4feffd640 1 -- 172.21.15.44:0/1925078682 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f8005290 con 0x7fe5080d0e10 2024-04-02T13:31:34.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.231+0000 7fe50ce3c640 1 -- 172.21.15.44:0/1925078682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5080c7cd0 con 0x7fe5080d0e10 2024-04-02T13:31:34.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.232+0000 7fe50ce3c640 1 -- 172.21.15.44:0/1925078682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5080684f0 con 0x7fe5080d0e10 2024-04-02T13:31:34.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.233+0000 7fe4feffd640 1 -- 172.21.15.44:0/1925078682 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe4f8005430 con 0x7fe5080d0e10 2024-04-02T13:31:34.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.233+0000 7fe4feffd640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe4e806c8f0 0x7fe4e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:34.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.233+0000 7fe4feffd640 1 -- 172.21.15.44:0/1925078682 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7fe4f808bac0 con 0x7fe5080d0e10 2024-04-02T13:31:34.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.233+0000 7fe50659c640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe4e806c8f0 0x7fe4e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:34.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.234+0000 7fe50659c640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe4e806c8f0 0x7fe4e806edb0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe500036000 tx=0x7fe50003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:34.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.238+0000 7fe4feffd640 1 -- 172.21.15.44:0/1925078682 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe4f8059d60 con 0x7fe5080d0e10 2024-04-02T13:31:34.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.433+0000 7fe50ce3c640 1 -- 172.21.15.44:0/1925078682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe5080ce120 con 0x7fe5080d0e10 2024-04-02T13:31:34.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.433+0000 7fe4feffd640 1 -- 172.21.15.44:0/1925078682 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v121) ==== 80+0+0 (secure 0 0 0) 0x7fe4f8059660 con 0x7fe5080d0e10 2024-04-02T13:31:34.438 INFO:teuthology.orchestra.run.smithi044.stderr:listed 0 entries 2024-04-02T13:31:34.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.435+0000 7fe50ce3c640 1 -- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe4e806c8f0 msgr2=0x7fe4e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:34.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.435+0000 7fe50ce3c640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe4e806c8f0 0x7fe4e806edb0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe500036000 tx=0x7fe50003c040 comp rx=0 tx=0).stop 2024-04-02T13:31:34.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.435+0000 7fe50ce3c640 1 -- 172.21.15.44:0/1925078682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5080d0e10 msgr2=0x7fe5080cd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:34.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.435+0000 7fe50ce3c640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5080d0e10 0x7fe5080cd2f0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fe4f800c780 tx=0x7fe4f800cc40 comp rx=0 tx=0).stop 2024-04-02T13:31:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.435+0000 7fe50ce3c640 1 -- 172.21.15.44:0/1925078682 shutdown_connections 2024-04-02T13:31:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.435+0000 7fe50ce3c640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe4e806c8f0 0x7fe4e806edb0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.435+0000 7fe50ce3c640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5080d0e10 0x7fe5080cd2f0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.435+0000 7fe50ce3c640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5080d0a30 0x7fe5080ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.435+0000 7fe50ce3c640 1 --2- 172.21.15.44:0/1925078682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5080cfe30 0x7fe508152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.435+0000 7fe50ce3c640 1 -- 172.21.15.44:0/1925078682 >> 172.21.15.44:0/1925078682 conn(0x7fe50809d470 msgr2=0x7fe5080d3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.435+0000 7fe50ce3c640 1 -- 172.21.15.44:0/1925078682 shutdown_connections 2024-04-02T13:31:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:34.435+0000 7fe50ce3c640 1 -- 172.21.15.44:0/1925078682 wait complete. 2024-04-02T13:31:34.450 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:31:34.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:34.450 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:34.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[82743]: fuse finished with error 0 and tester_r 0 2024-04-02T13:31:34.481 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:31:40.583 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:31:40.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:40.583 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:31:40.602 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:31:40.602 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:31:40.659 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:31:40.659 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi044.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:40.659 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:31:40.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:40.659 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:31:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-02T13:31:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:31:40.715 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:31:40.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.865+0000 7f0c8f447640 1 Processor -- start 2024-04-02T13:31:40.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.865+0000 7f0c8f447640 1 -- start start 2024-04-02T13:31:40.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.866+0000 7f0c8f447640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0c8815e4a0 0x7f0c880aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:40.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.866+0000 7f0c8f447640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c880cc600 con 0x7f0c880d0c50 2024-04-02T13:31:40.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.866+0000 7f0c8f447640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c880cc8e0 con 0x7f0c8806d550 2024-04-02T13:31:40.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.866+0000 7f0c8f447640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0c880ccbc0 con 0x7f0c8815e4a0 2024-04-02T13:31:40.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.866+0000 7f0c8d1e3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0c8815e4a0 0x7f0c880aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:40.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.866+0000 7f0c8c9e2640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0c880d0c50 0x7f0c8806ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48768/0 (socket says 172.21.15.44:48768) 2024-04-02T13:31:40.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.866+0000 7f0c8c9e2640 1 -- 172.21.15.44:0/2635076245 learned_addr learned my addr 172.21.15.44:0/2635076245 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:40.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.866+0000 7f0c8d1e3640 1 -- 172.21.15.44:0/2635076245 >> v1:172.21.15.69:6789/0 conn(0x7f0c8806d550 legacy=0x7f0c880cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:40.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.866+0000 7f0c8d1e3640 1 -- 172.21.15.44:0/2635076245 >> v1:172.21.15.44:6789/0 conn(0x7f0c880d0c50 legacy=0x7f0c8806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:40.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.866+0000 7f0c7ffff640 1 -- 172.21.15.44:0/2635076245 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2982059115 0 0) 0x7f0c880cc600 con 0x7f0c880d0c50 2024-04-02T13:31:40.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.866+0000 7f0c8d1e3640 1 -- 172.21.15.44:0/2635076245 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c880ccea0 con 0x7f0c8815e4a0 2024-04-02T13:31:40.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.867+0000 7f0c8d1e3640 1 --2- 172.21.15.44:0/2635076245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c8815e4a0 0x7f0c880aebf0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f0c840097d0 tx=0x7f0c8402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b1a289d98910dc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:40.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.867+0000 7f0c7ffff640 1 -- 172.21.15.44:0/2635076245 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84034070 con 0x7f0c8815e4a0 2024-04-02T13:31:40.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.867+0000 7f0c7ffff640 1 -- 172.21.15.44:0/2635076245 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c84030610 con 0x7f0c8815e4a0 2024-04-02T13:31:40.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.867+0000 7f0c7ffff640 1 -- 172.21.15.44:0/2635076245 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c840309e0 con 0x7f0c8815e4a0 2024-04-02T13:31:40.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.868+0000 7f0c8f447640 1 -- 172.21.15.44:0/2635076245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c8815e4a0 msgr2=0x7f0c880aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:40.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.868+0000 7f0c8f447640 1 --2- 172.21.15.44:0/2635076245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c8815e4a0 0x7f0c880aebf0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f0c840097d0 tx=0x7f0c8402b960 comp rx=0 tx=0).stop 2024-04-02T13:31:40.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.868+0000 7f0c8f447640 1 -- 172.21.15.44:0/2635076245 shutdown_connections 2024-04-02T13:31:40.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.868+0000 7f0c8f447640 1 --2- 172.21.15.44:0/2635076245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c8815e4a0 0x7f0c880aebf0 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:40.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.868+0000 7f0c8f447640 1 -- 172.21.15.44:0/2635076245 >> 172.21.15.44:0/2635076245 conn(0x7f0c8809aa20 msgr2=0x7f0c8809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:40.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.868+0000 7f0c8f447640 1 -- 172.21.15.44:0/2635076245 shutdown_connections 2024-04-02T13:31:40.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.868+0000 7f0c8f447640 1 -- 172.21.15.44:0/2635076245 wait complete. 2024-04-02T13:31:40.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.869+0000 7f0c8f447640 1 Processor -- start 2024-04-02T13:31:40.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.869+0000 7f0c8f447640 1 -- start start 2024-04-02T13:31:40.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8f447640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c8806d550 0x7f0c881f4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:40.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8f447640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c880d0c50 0x7f0c880c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:40.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8f447640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c8815e4a0 0x7f0c880c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:40.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8f447640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8815e880 con 0x7f0c8806d550 2024-04-02T13:31:40.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8f447640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0c880cd460 con 0x7f0c8815e4a0 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8f447640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0c880af130 con 0x7f0c880d0c50 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8d1e3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c8806d550 0x7f0c881f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8d1e3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c8806d550 0x7f0c881f4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46238/0 (socket says 172.21.15.44:46238) 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8d1e3640 1 -- 172.21.15.44:0/1718083063 learned_addr learned my addr 172.21.15.44:0/1718083063 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8c9e2640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c880d0c50 0x7f0c880c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8d9e4640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c8815e4a0 0x7f0c880c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8d1e3640 1 -- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c880d0c50 msgr2=0x7f0c880c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8d1e3640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c880d0c50 0x7f0c880c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8d1e3640 1 -- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c8815e4a0 msgr2=0x7f0c880c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8d1e3640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c8815e4a0 0x7f0c880c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8d1e3640 1 -- 172.21.15.44:0/1718083063 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c880c7b70 con 0x7f0c8806d550 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.870+0000 7f0c8d9e4640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c8815e4a0 0x7f0c880c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.871+0000 7f0c8d1e3640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c8806d550 0x7f0c881f4860 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f0c840098f0 tx=0x7f0c840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.871+0000 7f0c7dffb640 1 -- 172.21.15.44:0/1718083063 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84049070 con 0x7f0c8806d550 2024-04-02T13:31:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.871+0000 7f0c7dffb640 1 -- 172.21.15.44:0/1718083063 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c84039780 con 0x7f0c8806d550 2024-04-02T13:31:40.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.871+0000 7f0c7dffb640 1 -- 172.21.15.44:0/1718083063 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84034040 con 0x7f0c8806d550 2024-04-02T13:31:40.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.871+0000 7f0c8f447640 1 -- 172.21.15.44:0/1718083063 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c880c7e50 con 0x7f0c8806d550 2024-04-02T13:31:40.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.871+0000 7f0c8f447640 1 -- 172.21.15.44:0/1718083063 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c880c8410 con 0x7f0c8806d550 2024-04-02T13:31:40.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.872+0000 7f0c8f447640 1 -- 172.21.15.44:0/1718083063 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c880684f0 con 0x7f0c8806d550 2024-04-02T13:31:40.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.872+0000 7f0c7dffb640 1 -- 172.21.15.44:0/1718083063 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0c8404c070 con 0x7f0c8806d550 2024-04-02T13:31:40.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.873+0000 7f0c7dffb640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c6c06c8f0 0x7f0c6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:40.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.873+0000 7f0c7dffb640 1 -- 172.21.15.44:0/1718083063 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f0c840ad9f0 con 0x7f0c8806d550 2024-04-02T13:31:40.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.873+0000 7f0c8c9e2640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c6c06c8f0 0x7f0c6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:40.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.874+0000 7f0c8c9e2640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c6c06c8f0 0x7f0c6c06edb0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0c78007f70 tx=0x7f0c78007f00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:40.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:40.877+0000 7f0c7dffb640 1 -- 172.21.15.44:0/1718083063 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0c8407b640 con 0x7f0c8806d550 2024-04-02T13:31:41.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.101+0000 7f0c8f447640 1 -- 172.21.15.44:0/1718083063 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0c880cc370 con 0x7f0c8806d550 2024-04-02T13:31:41.105 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:31:41.101+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:31:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.102+0000 7f0c7dffb640 1 -- 172.21.15.44:0/1718083063 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v100) ==== 161+0+0 (secure 0 0 0) 0x7f0c8407f940 con 0x7f0c8806d550 2024-04-02T13:31:41.106 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:31:41.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.106+0000 7f0c8f447640 1 -- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c6c06c8f0 msgr2=0x7f0c6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:41.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.106+0000 7f0c8f447640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c6c06c8f0 0x7f0c6c06edb0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0c78007f70 tx=0x7f0c78007f00 comp rx=0 tx=0).stop 2024-04-02T13:31:41.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.106+0000 7f0c8f447640 1 -- 172.21.15.44:0/1718083063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c8806d550 msgr2=0x7f0c881f4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:41.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.106+0000 7f0c8f447640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c8806d550 0x7f0c881f4860 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f0c840098f0 tx=0x7f0c840046c0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.106+0000 7f0c8f447640 1 -- 172.21.15.44:0/1718083063 shutdown_connections 2024-04-02T13:31:41.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.106+0000 7f0c8f447640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0c6c06c8f0 0x7f0c6c06edb0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.106+0000 7f0c8f447640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c8815e4a0 0x7f0c880c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.106+0000 7f0c8f447640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c880d0c50 0x7f0c880c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.106+0000 7f0c8f447640 1 --2- 172.21.15.44:0/1718083063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c8806d550 0x7f0c881f4860 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.106+0000 7f0c8f447640 1 -- 172.21.15.44:0/1718083063 >> 172.21.15.44:0/1718083063 conn(0x7f0c8809aa20 msgr2=0x7f0c8815d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:41.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.106+0000 7f0c8f447640 1 -- 172.21.15.44:0/1718083063 shutdown_connections 2024-04-02T13:31:41.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.106+0000 7f0c8f447640 1 -- 172.21.15.44:0/1718083063 wait complete. 2024-04-02T13:31:41.114 INFO:tasks.ceph.mds.a.smithi044.stderr: -8168> 2024-04-02T13:31:12.764+0000 7fe30526c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:31:41.124 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T13:31:41.127 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T13:31:41.149 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:31:41.176 INFO:teuthology.orchestra.run.smithi044.stdout:[global] 2024-04-02T13:31:41.176 INFO:teuthology.orchestra.run.smithi044.stdout: chdir = "" 2024-04-02T13:31:41.176 INFO:teuthology.orchestra.run.smithi044.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T13:31:41.176 INFO:teuthology.orchestra.run.smithi044.stdout: auth supported = cephx 2024-04-02T13:31:41.176 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.176 INFO:teuthology.orchestra.run.smithi044.stdout: filestore xattr use omap = true 2024-04-02T13:31:41.176 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.176 INFO:teuthology.orchestra.run.smithi044.stdout: mon clock drift allowed = 1.000 2024-04-02T13:31:41.176 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: osd crush chooseleaf type = 0 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: auth debug = true 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on old message = true 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on bug = true 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: mon pg warn max object skew = 0 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default size = 2 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.177 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow primary affinity = true 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow pg remap = true 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on legacy crush tunables = false 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on crush straw calc version zero = false 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on no sortbitwise = false 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on osd down out interval zero = false 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on too few osds = false 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: mon_allow_pool_size_one = true 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: osd default data pool replay window = 5 2024-04-02T13:31:41.178 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout: mon allow pool delete = true 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout: mon cluster log file level = debug 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout: debug asserts on shutdown = true 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout: mon health detail to clog = false 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout: mon host = "172.21.15.44,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout: fsid = 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout:[osd] 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout: osd journal size = 100 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub load threshold = 5.0 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub max interval = 600 2024-04-02T13:31:41.179 INFO:teuthology.orchestra.run.smithi044.stdout: osd mclock profile = high_recovery_ops 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: osd recover clone overlap = true 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: osd recovery max chunk = 1048576 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug shutdown = true 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug op order = true 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug verify stray on activate = true 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug trim objects = true 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: osd open classes on start = true 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug pg log writeout = true 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: osd deep scrub update digest min age = 30 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: osd map max advance = 10 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: journal zero on create = true 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: filestore ondisk finisher threads = 3 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: filestore apply finisher threads = 3 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: bdev debug aio = true 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug misdirected ops = true 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore block size = 96636764160 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore fsck on mount = True 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluefs = 20 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluestore = 20 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:31:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: debug osd = 20 2024-04-02T13:31:41.182 INFO:teuthology.orchestra.run.smithi044.stdout: debug rocksdb = 10 2024-04-02T13:31:41.182 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T13:31:41.182 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd full ratio = 0.9 2024-04-02T13:31:41.182 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd nearfull ratio = 0.8 2024-04-02T13:31:41.182 INFO:teuthology.orchestra.run.smithi044.stdout: osd failsafe full ratio = 0.95 2024-04-02T13:31:41.182 INFO:teuthology.orchestra.run.smithi044.stdout: osd objectstore = bluestore 2024-04-02T13:31:41.182 INFO:teuthology.orchestra.run.smithi044.stdout: osd op complaint time = 180 2024-04-02T13:31:41.182 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.182 INFO:teuthology.orchestra.run.smithi044.stdout:[mgr] 2024-04-02T13:31:41.182 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:31:41.182 INFO:teuthology.orchestra.run.smithi044.stdout: debug mgr = 20 2024-04-02T13:31:41.182 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout: mgr/telemetry/nag = false 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout: debug client = 20 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout:[mon] 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout: debug paxos = 20 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout: mon data avail warn = 5 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout: mon mgr mkfs grace = 240 2024-04-02T13:31:41.183 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd reporter subtree level = osd 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd prime pg temp = true 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: # 1m isn't quite enough 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: mon_down_mkfs_grace = 2m 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_filestore_osds = false 2024-04-02T13:31:41.184 INFO:teuthology.orchestra.run.smithi044.stdout: mon down mkfs grace = 300 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout: mon op complaint time = 120 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout:[client] 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout: rgw cache enabled = true 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable ops log = true 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable usage log = true 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout: client mount timeout = 600 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout: debug client = 20 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout: pid file = "" 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout: rados mon op timeout = 900 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout: rados osd op timeout = 900 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.a] 2024-04-02T13:31:41.185 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.b] 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.c] 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.a] 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.c] 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.b] 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.d] 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout:[mds] 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout: debug mds = 20 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout: debug mds balancer = 20 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout: mds debug frag = True 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout: mds debug scatterstat = True 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout: mds op complaint time = 180 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout: mds verify scatter = False 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout: osd op complaint time = 180 2024-04-02T13:31:41.186 INFO:teuthology.orchestra.run.smithi044.stdout: rados mon op timeout = 900 2024-04-02T13:31:41.187 INFO:teuthology.orchestra.run.smithi044.stdout: rados osd op timeout = 900 2024-04-02T13:31:41.187 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-02T13:31:41.187 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-04-02T13:31:41.187 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T13:31:41.187 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-04-02T13:31:41.187 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.187 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-04-02T13:31:41.187 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.188 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-04-02T13:31:41.188 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.188 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-04-02T13:31:41.188 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-04-02T13:31:41.188 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.188 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-04-02T13:31:41.188 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-04-02T13:31:41.188 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.188 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T13:31:41.188 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-04-02T13:31:41.188 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.188 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.189 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T13:31:41.190 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.190 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-04-02T13:31:41.190 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.190 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-04-02T13:31:41.190 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.190 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-04-02T13:31:41.190 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-04-02T13:31:41.190 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-04-02T13:31:41.190 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.44,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-04-02T13:31:41.190 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:31:41.190 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.190 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-04-02T13:31:41.190 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-04-02T13:31:41.191 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 20 2024-04-02T13:31:41.192 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 20 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 10 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-04-02T13:31:41.193 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-04-02T13:31:41.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: mon down mkfs grace = 300 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-02T13:31:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = False 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-02T13:31:41.197 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-02T13:31:41.198 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-02T13:31:41.199 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T13:31:41.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T13:31:41.288 INFO:teuthology.orchestra.run.smithi044.stdout:[global] 2024-04-02T13:31:41.288 INFO:teuthology.orchestra.run.smithi044.stdout: chdir = "" 2024-04-02T13:31:41.288 INFO:teuthology.orchestra.run.smithi044.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: auth supported = cephx 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: filestore xattr use omap = true 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: mon clock drift allowed = 1.000 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: osd crush chooseleaf type = 0 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: auth debug = true 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on old message = true 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on bug = true 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: mon pg warn max object skew = 0 2024-04-02T13:31:41.289 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default size = 2 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow primary affinity = true 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow pg remap = true 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on legacy crush tunables = false 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on crush straw calc version zero = false 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on no sortbitwise = false 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on osd down out interval zero = false 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on too few osds = false 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T13:31:41.290 INFO:teuthology.orchestra.run.smithi044.stdout: mon_allow_pool_size_one = true 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: osd default data pool replay window = 5 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: mon allow pool delete = true 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: mon cluster log file level = debug 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: debug asserts on shutdown = true 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: mon health detail to clog = false 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: mon host = "172.21.15.44,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: fsid = 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout:[osd] 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: osd journal size = 100 2024-04-02T13:31:41.291 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub load threshold = 5.0 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub max interval = 600 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: osd mclock profile = high_recovery_ops 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: osd recover clone overlap = true 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: osd recovery max chunk = 1048576 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug shutdown = true 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug op order = true 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug verify stray on activate = true 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug trim objects = true 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: osd open classes on start = true 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug pg log writeout = true 2024-04-02T13:31:41.292 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: osd deep scrub update digest min age = 30 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: osd map max advance = 10 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: journal zero on create = true 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: filestore ondisk finisher threads = 3 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: filestore apply finisher threads = 3 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: bdev debug aio = true 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug misdirected ops = true 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore block size = 96636764160 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore fsck on mount = True 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluefs = 20 2024-04-02T13:31:41.293 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluestore = 20 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: debug osd = 20 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: debug rocksdb = 10 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd full ratio = 0.9 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd nearfull ratio = 0.8 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: osd failsafe full ratio = 0.95 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: osd objectstore = bluestore 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: osd op complaint time = 180 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout:[mgr] 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: debug mgr = 20 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-04-02T13:31:41.294 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: mgr/telemetry/nag = false 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: debug client = 20 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout:[mon] 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: debug paxos = 20 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: mon data avail warn = 5 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: mon mgr mkfs grace = 240 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd reporter subtree level = osd 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd prime pg temp = true 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:31:41.295 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: # 1m isn't quite enough 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: mon_down_mkfs_grace = 2m 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_filestore_osds = false 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: mon down mkfs grace = 300 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: mon op complaint time = 120 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout:[client] 2024-04-02T13:31:41.296 INFO:teuthology.orchestra.run.smithi044.stdout: rgw cache enabled = true 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable ops log = true 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable usage log = true 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout: client mount timeout = 600 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout: debug client = 20 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout: pid file = "" 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout: rados mon op timeout = 900 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout: rados osd op timeout = 900 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.a] 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.b] 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.c] 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.a] 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.c] 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.b] 2024-04-02T13:31:41.297 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.d] 2024-04-02T13:31:41.298 INFO:teuthology.orchestra.run.smithi044.stdout:[mds] 2024-04-02T13:31:41.298 INFO:teuthology.orchestra.run.smithi044.stdout: debug mds = 20 2024-04-02T13:31:41.298 INFO:teuthology.orchestra.run.smithi044.stdout: debug mds balancer = 20 2024-04-02T13:31:41.298 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:31:41.298 INFO:teuthology.orchestra.run.smithi044.stdout: mds debug frag = True 2024-04-02T13:31:41.298 INFO:teuthology.orchestra.run.smithi044.stdout: mds debug scatterstat = False 2024-04-02T13:31:41.298 INFO:teuthology.orchestra.run.smithi044.stdout: mds op complaint time = 180 2024-04-02T13:31:41.298 INFO:teuthology.orchestra.run.smithi044.stdout: mds verify scatter = False 2024-04-02T13:31:41.298 INFO:teuthology.orchestra.run.smithi044.stdout: osd op complaint time = 180 2024-04-02T13:31:41.298 INFO:teuthology.orchestra.run.smithi044.stdout: rados mon op timeout = 900 2024-04-02T13:31:41.298 INFO:teuthology.orchestra.run.smithi044.stdout: rados osd op timeout = 900 2024-04-02T13:31:41.299 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-02T13:31:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-04-02T13:31:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T13:31:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-04-02T13:31:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-04-02T13:31:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-04-02T13:31:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-04-02T13:31:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-04-02T13:31:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-04-02T13:31:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-04-02T13:31:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.44,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-04-02T13:31:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-04-02T13:31:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 20 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 20 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 10 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T13:31:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-02T13:31:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T13:31:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: mon down mkfs grace = 300 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-04-02T13:31:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-04-02T13:31:41.308 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-04-02T13:31:41.309 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-04-02T13:31:41.309 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-04-02T13:31:41.309 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-04-02T13:31:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-04-02T13:31:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-04-02T13:31:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:31:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-04-02T13:31:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = False 2024-04-02T13:31:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-04-02T13:31:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = False 2024-04-02T13:31:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-02T13:31:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-02T13:31:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-02T13:31:41.311 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmomapkey 10000000001.00000000 bravo_head 2024-04-02T13:31:41.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.389+0000 7f5cc0245f40 1 Processor -- start 2024-04-02T13:31:41.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.389+0000 7f5cc0245f40 1 -- start start 2024-04-02T13:31:41.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.389+0000 7f5cc0245f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cac0022d0 0x55c036e57430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.389+0000 7f5cc0245f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c036e56ac0 0x55c036dcae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.389+0000 7f5cc0245f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c036dcb380 0x55c036e59980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.389+0000 7f5cc0245f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55c036d323e0 con 0x55c036e56ac0 2024-04-02T13:31:41.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.389+0000 7f5cc0245f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c036d36430 con 0x7f5cac0022d0 2024-04-02T13:31:41.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.389+0000 7f5cc0245f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55c036dd3e20 con 0x55c036dcb380 2024-04-02T13:31:41.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.389+0000 7f5cbcd56640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c036e56ac0 0x55c036dcae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.389+0000 7f5cbcd56640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c036e56ac0 0x55c036dcae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46268/0 (socket says 172.21.15.44:46268) 2024-04-02T13:31:41.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.389+0000 7f5cbd557640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cac0022d0 0x55c036e57430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.389+0000 7f5cbcd56640 1 -- 172.21.15.44:0/828512841 learned_addr learned my addr 172.21.15.44:0/828512841 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:41.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.389+0000 7f5cbdd58640 1 --2- 172.21.15.44:0/828512841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c036dcb380 0x55c036e59980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cbd557640 1 -- 172.21.15.44:0/828512841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c036dcb380 msgr2=0x55c036e59980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cbd557640 1 --2- 172.21.15.44:0/828512841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c036dcb380 0x55c036e59980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cbd557640 1 -- 172.21.15.44:0/828512841 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c036e56ac0 msgr2=0x55c036dcae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cbd557640 1 --2- 172.21.15.44:0/828512841 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c036e56ac0 0x55c036dcae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cbd557640 1 -- 172.21.15.44:0/828512841 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c036e57060 con 0x7f5cac0022d0 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cbcd56640 1 --2- 172.21.15.44:0/828512841 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c036e56ac0 0x55c036dcae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cbd557640 1 --2- 172.21.15.44:0/828512841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cac0022d0 0x55c036e57430 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00a3b0 tx=0x7f5cb8002fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54b16eed470ef6a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cb77fe640 1 -- 172.21.15.44:0/828512841 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8057040 con 0x7f5cac0022d0 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cb77fe640 1 -- 172.21.15.44:0/828512841 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cb80525f0 con 0x7f5cac0022d0 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cb77fe640 1 -- 172.21.15.44:0/828512841 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb80529e0 con 0x7f5cac0022d0 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cc0245f40 1 -- 172.21.15.44:0/828512841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cac0022d0 msgr2=0x55c036e57430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cc0245f40 1 --2- 172.21.15.44:0/828512841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cac0022d0 0x55c036e57430 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00a3b0 tx=0x7f5cb8002fb0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cc0245f40 1 -- 172.21.15.44:0/828512841 shutdown_connections 2024-04-02T13:31:41.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cc0245f40 1 --2- 172.21.15.44:0/828512841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c036dcb380 0x55c036e59980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cc0245f40 1 --2- 172.21.15.44:0/828512841 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c036e56ac0 0x55c036dcae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cc0245f40 1 --2- 172.21.15.44:0/828512841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cac0022d0 0x55c036e57430 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.390+0000 7f5cc0245f40 1 -- 172.21.15.44:0/828512841 >> 172.21.15.44:0/828512841 conn(0x55c036d8b1c0 msgr2=0x55c036d8b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.391+0000 7f5cc0245f40 1 -- 172.21.15.44:0/828512841 shutdown_connections 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.391+0000 7f5cc0245f40 1 -- 172.21.15.44:0/828512841 wait complete. 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.391+0000 7f5cc0245f40 1 Processor -- start 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.391+0000 7f5cc0245f40 1 -- start start 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cc0245f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c036dcb380 0x55c036dd05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cc0245f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c036e56ac0 0x55c036e41130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cc0245f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cac0022d0 0x55c036e4a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cc0245f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55c036e56630 con 0x55c036e56ac0 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cc0245f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c036db77a0 con 0x7f5cac0022d0 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cc0245f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55c036db74c0 con 0x55c036dcb380 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cbcd56640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c036e56ac0 0x55c036e41130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cbcd56640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c036e56ac0 0x55c036e41130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46278/0 (socket says 172.21.15.44:46278) 2024-04-02T13:31:41.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cbcd56640 1 -- 172.21.15.44:0/2885483904 learned_addr learned my addr 172.21.15.44:0/2885483904 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:41.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cbd557640 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c036dcb380 0x55c036dd05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cbdd58640 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cac0022d0 0x55c036e4a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cbcd56640 1 -- 172.21.15.44:0/2885483904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c036dcb380 msgr2=0x55c036dd05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:41.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cbcd56640 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c036dcb380 0x55c036dd05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cbcd56640 1 -- 172.21.15.44:0/2885483904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cac0022d0 msgr2=0x55c036e4a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:41.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cbcd56640 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cac0022d0 0x55c036e4a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cbcd56640 1 -- 172.21.15.44:0/2885483904 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c036dd0d40 con 0x55c036e56ac0 2024-04-02T13:31:41.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cbcd56640 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c036e56ac0 0x55c036e41130 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f5ca800f6b0 tx=0x7f5ca800fb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:41.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.392+0000 7f5cb67fc640 1 -- 172.21.15.44:0/2885483904 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca800ebb0 con 0x55c036e56ac0 2024-04-02T13:31:41.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.393+0000 7f5cb67fc640 1 -- 172.21.15.44:0/2885483904 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ca800ed50 con 0x55c036e56ac0 2024-04-02T13:31:41.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.393+0000 7f5cc0245f40 1 -- 172.21.15.44:0/2885483904 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c036e48f80 con 0x55c036e56ac0 2024-04-02T13:31:41.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.393+0000 7f5cb67fc640 1 -- 172.21.15.44:0/2885483904 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca801a630 con 0x55c036e56ac0 2024-04-02T13:31:41.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.393+0000 7f5cc0245f40 1 -- 172.21.15.44:0/2885483904 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c036e49540 con 0x55c036e56ac0 2024-04-02T13:31:41.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.394+0000 7f5cb67fc640 1 -- 172.21.15.44:0/2885483904 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5ca801a7d0 con 0x55c036e56ac0 2024-04-02T13:31:41.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.394+0000 7f5cb67fc640 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5ca006b2d0 0x7f5ca006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.394+0000 7f5cb67fc640 1 -- 172.21.15.44:0/2885483904 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f5ca808bc70 con 0x55c036e56ac0 2024-04-02T13:31:41.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.394+0000 7f5cbd557640 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5ca006b2d0 0x7f5ca006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.395+0000 7f5cc0245f40 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55c036d8c860 0x55c036d8cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.395+0000 7f5cc0245f40 1 -- 172.21.15.44:0/2885483904 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [omap-rm-keys in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55c036d8d180 con 0x55c036d8c860 2024-04-02T13:31:41.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.395+0000 7f5cbdd58640 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55c036d8c860 0x55c036d8cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.395+0000 7f5cbd557640 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5ca006b2d0 0x7f5ca006d790 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x55c036e5da00 tx=0x7f5cb804c000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:41.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.395+0000 7f5cbdd58640 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55c036d8c860 0x55c036d8cc40 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:41.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.396+0000 7f5cbdd58640 1 -- 172.21.15.44:0/2885483904 <== osd.0 v2:172.21.15.44:6800/2303805525 1 ==== osd_op_reply(1 10000000001.00000000 [omap-rm-keys] v122'5 uv5 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5cb00143b0 con 0x55c036d8c860 2024-04-02T13:31:41.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.396+0000 7f5cc0245f40 1 -- 172.21.15.44:0/2885483904 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55c036d8c860 msgr2=0x55c036d8cc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:41.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.396+0000 7f5cc0245f40 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55c036d8c860 0x55c036d8cc40 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.396+0000 7f5cc0245f40 1 -- 172.21.15.44:0/2885483904 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5ca006b2d0 msgr2=0x7f5ca006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:41.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.396+0000 7f5cc0245f40 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5ca006b2d0 0x7f5ca006d790 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x55c036e5da00 tx=0x7f5cb804c000 comp rx=0 tx=0).stop 2024-04-02T13:31:41.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.396+0000 7f5cc0245f40 1 -- 172.21.15.44:0/2885483904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c036e56ac0 msgr2=0x55c036e41130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:41.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.396+0000 7f5cc0245f40 1 --2- 172.21.15.44:0/2885483904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c036e56ac0 0x55c036e41130 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f5ca800f6b0 tx=0x7f5ca800fb70 comp rx=0 tx=0).stop 2024-04-02T13:31:41.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.396+0000 7f5cbdd58640 1 -- 172.21.15.44:0/2885483904 reap_dead start 2024-04-02T13:31:41.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.396+0000 7f5cc0245f40 1 -- 172.21.15.44:0/2885483904 shutdown_connections 2024-04-02T13:31:41.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.396+0000 7f5cc0245f40 1 -- 172.21.15.44:0/2885483904 >> 172.21.15.44:0/2885483904 conn(0x55c036d8b1c0 msgr2=0x55c036d8b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:41.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.396+0000 7f5cc0245f40 1 -- 172.21.15.44:0/2885483904 shutdown_connections 2024-04-02T13:31:41.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.396+0000 7f5cc0245f40 1 -- 172.21.15.44:0/2885483904 wait complete. 2024-04-02T13:31:41.404 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-02T13:31:41.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.543+0000 7f0af1c55640 1 Processor -- start 2024-04-02T13:31:41.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.543+0000 7f0af1c55640 1 -- start start 2024-04-02T13:31:41.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.544+0000 7f0af1c55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aec0b5540 0x7f0aec0d1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.544+0000 7f0af1c55640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0aec0d2d40 0x7f0aec15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.544+0000 7f0af1c55640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0aec0cebd0 con 0x7f0aec0d21f0 2024-04-02T13:31:41.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.544+0000 7f0af1c55640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0aec15f920 con 0x7f0aec0d2d40 2024-04-02T13:31:41.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.544+0000 7f0af1c55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0aec0b5920 con 0x7f0aec0b5540 2024-04-02T13:31:41.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.544+0000 7f0aebfff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0aec0d2d40 0x7f0aec15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.544+0000 7f0aeaffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0aec0d21f0 0x7f0aec0d2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48790/0 (socket says 172.21.15.44:48790) 2024-04-02T13:31:41.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.544+0000 7f0aeaffd640 1 -- 172.21.15.44:0/3573355091 learned_addr learned my addr 172.21.15.44:0/3573355091 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:41.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.544+0000 7f0aeb7fe640 1 --2- 172.21.15.44:0/3573355091 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aec0b5540 0x7f0aec0d1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.545+0000 7f0aebfff640 1 -- 172.21.15.44:0/3573355091 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aec0b5540 msgr2=0x7f0aec0d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:41.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.545+0000 7f0aebfff640 1 --2- 172.21.15.44:0/3573355091 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aec0b5540 0x7f0aec0d1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.545+0000 7f0aebfff640 1 -- 172.21.15.44:0/3573355091 >> v1:172.21.15.44:6789/0 conn(0x7f0aec0d21f0 legacy=0x7f0aec0d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:41.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.545+0000 7f0aebfff640 1 -- 172.21.15.44:0/3573355091 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aec15eda0 con 0x7f0aec0d2d40 2024-04-02T13:31:41.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.545+0000 7f0aea7fc640 1 -- 172.21.15.44:0/3573355091 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3304389831 0 0) 0x7f0aec0cebd0 con 0x7f0aec0d21f0 2024-04-02T13:31:41.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.545+0000 7f0aeb7fe640 1 --2- 172.21.15.44:0/3573355091 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aec0b5540 0x7f0aec0d1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:31:41.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.545+0000 7f0aebfff640 1 --2- 172.21.15.44:0/3573355091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aec0d2d40 0x7f0aec15dca0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4009d90 tx=0x7f0ae402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225a0220fd1c2713 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:41.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.545+0000 7f0aea7fc640 1 -- 172.21.15.44:0/3573355091 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4039070 con 0x7f0aec0d2d40 2024-04-02T13:31:41.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.545+0000 7f0aea7fc640 1 -- 172.21.15.44:0/3573355091 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae402cc50 con 0x7f0aec0d2d40 2024-04-02T13:31:41.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.546+0000 7f0aea7fc640 1 -- 172.21.15.44:0/3573355091 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4034440 con 0x7f0aec0d2d40 2024-04-02T13:31:41.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.546+0000 7f0af1c55640 1 -- 172.21.15.44:0/3573355091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aec0d2d40 msgr2=0x7f0aec15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:41.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.546+0000 7f0af1c55640 1 --2- 172.21.15.44:0/3573355091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aec0d2d40 0x7f0aec15dca0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4009d90 tx=0x7f0ae402bea0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.546+0000 7f0af1c55640 1 -- 172.21.15.44:0/3573355091 shutdown_connections 2024-04-02T13:31:41.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.546+0000 7f0af1c55640 1 --2- 172.21.15.44:0/3573355091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aec0d2d40 0x7f0aec15dca0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.546+0000 7f0af1c55640 1 --2- 172.21.15.44:0/3573355091 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aec0b5540 0x7f0aec0d1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.546+0000 7f0af1c55640 1 -- 172.21.15.44:0/3573355091 >> 172.21.15.44:0/3573355091 conn(0x7f0aec09d390 msgr2=0x7f0aec0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:41.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.546+0000 7f0af1c55640 1 -- 172.21.15.44:0/3573355091 shutdown_connections 2024-04-02T13:31:41.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.546+0000 7f0af1c55640 1 -- 172.21.15.44:0/3573355091 wait complete. 2024-04-02T13:31:41.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.547+0000 7f0af1c55640 1 Processor -- start 2024-04-02T13:31:41.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.548+0000 7f0af1c55640 1 -- start start 2024-04-02T13:31:41.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.548+0000 7f0af1c55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec0b5540 0x7f0aec158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.548+0000 7f0af1c55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aec0d21f0 0x7f0aec15af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.548+0000 7f0af1c55640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aec0d2d40 0x7f0aec0cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.548+0000 7f0af1c55640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0aec15f640 con 0x7f0aec0d2d40 2024-04-02T13:31:41.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.548+0000 7f0af1c55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0aec15f360 con 0x7f0aec0b5540 2024-04-02T13:31:41.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.548+0000 7f0af1c55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0aec15fcc0 con 0x7f0aec0d21f0 2024-04-02T13:31:41.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.548+0000 7f0aebfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aec0d2d40 0x7f0aec0cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.548+0000 7f0aebfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aec0d2d40 0x7f0aec0cdf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46294/0 (socket says 172.21.15.44:46294) 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.548+0000 7f0aeb7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec0b5540 0x7f0aec158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.548+0000 7f0aebfff640 1 -- 172.21.15.44:0/3577700640 learned_addr learned my addr 172.21.15.44:0/3577700640 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.548+0000 7f0aeaffd640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aec0d21f0 0x7f0aec15af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.549+0000 7f0aebfff640 1 -- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aec0d21f0 msgr2=0x7f0aec15af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.549+0000 7f0aebfff640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aec0d21f0 0x7f0aec15af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.549+0000 7f0aebfff640 1 -- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec0b5540 msgr2=0x7f0aec158ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.549+0000 7f0aebfff640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec0b5540 0x7f0aec158ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.549+0000 7f0aebfff640 1 -- 172.21.15.44:0/3577700640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aec0ce450 con 0x7f0aec0d2d40 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.549+0000 7f0aeb7fe640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec0b5540 0x7f0aec158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.549+0000 7f0aebfff640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aec0d2d40 0x7f0aec0cdf10 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4037040 tx=0x7f0ae4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.549+0000 7f0ad3fff640 1 -- 172.21.15.44:0/3577700640 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4044020 con 0x7f0aec0d2d40 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.549+0000 7f0ad3fff640 1 -- 172.21.15.44:0/3577700640 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae402ca50 con 0x7f0aec0d2d40 2024-04-02T13:31:41.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.549+0000 7f0af1c55640 1 -- 172.21.15.44:0/3577700640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aec159450 con 0x7f0aec0d2d40 2024-04-02T13:31:41.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.549+0000 7f0ad3fff640 1 -- 172.21.15.44:0/3577700640 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4039040 con 0x7f0aec0d2d40 2024-04-02T13:31:41.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.549+0000 7f0af1c55640 1 -- 172.21.15.44:0/3577700640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aec159a10 con 0x7f0aec0d2d40 2024-04-02T13:31:41.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.550+0000 7f0af1c55640 1 -- 172.21.15.44:0/3577700640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0aec0684f0 con 0x7f0aec0d2d40 2024-04-02T13:31:41.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.551+0000 7f0ad3fff640 1 -- 172.21.15.44:0/3577700640 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ae4047020 con 0x7f0aec0d2d40 2024-04-02T13:31:41.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.551+0000 7f0ad3fff640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ad806c920 0x7f0ad806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:41.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.551+0000 7f0ad3fff640 1 -- 172.21.15.44:0/3577700640 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f0ae40ae290 con 0x7f0aec0d2d40 2024-04-02T13:31:41.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.551+0000 7f0aeb7fe640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ad806c920 0x7f0ad806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:41.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.552+0000 7f0aeb7fe640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ad806c920 0x7f0ad806ede0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009560 tx=0x7f0adc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:41.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.556+0000 7f0ad3fff640 1 -- 172.21.15.44:0/3577700640 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0ae404e070 con 0x7f0aec0d2d40 2024-04-02T13:31:41.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:41.780+0000 7f0af1c55640 1 -- 172.21.15.44:0/3577700640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f0aec0d4760 con 0x7f0aec0d2d40 2024-04-02T13:31:42.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.104+0000 7f0ad3fff640 1 -- 172.21.15.44:0/3577700640 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v101) ==== 162+0+0 (secure 0 0 0) 0x7f0aec0ce450 con 0x7f0aec0d2d40 2024-04-02T13:31:42.108 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-02T13:31:42.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.106+0000 7f0af1c55640 1 -- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ad806c920 msgr2=0x7f0ad806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:42.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.106+0000 7f0af1c55640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ad806c920 0x7f0ad806ede0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009560 tx=0x7f0adc009190 comp rx=0 tx=0).stop 2024-04-02T13:31:42.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.106+0000 7f0af1c55640 1 -- 172.21.15.44:0/3577700640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aec0d2d40 msgr2=0x7f0aec0cdf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:42.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.106+0000 7f0af1c55640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aec0d2d40 0x7f0aec0cdf10 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4037040 tx=0x7f0ae4004a40 comp rx=0 tx=0).stop 2024-04-02T13:31:42.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.106+0000 7f0af1c55640 1 -- 172.21.15.44:0/3577700640 shutdown_connections 2024-04-02T13:31:42.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.106+0000 7f0af1c55640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ad806c920 0x7f0ad806ede0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.106+0000 7f0af1c55640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0aec0d2d40 0x7f0aec0cdf10 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.106+0000 7f0af1c55640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aec0d21f0 0x7f0aec15af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.106+0000 7f0af1c55640 1 --2- 172.21.15.44:0/3577700640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec0b5540 0x7f0aec158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.106+0000 7f0af1c55640 1 -- 172.21.15.44:0/3577700640 >> 172.21.15.44:0/3577700640 conn(0x7f0aec09d390 msgr2=0x7f0aec0b8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:42.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.106+0000 7f0af1c55640 1 -- 172.21.15.44:0/3577700640 shutdown_connections 2024-04-02T13:31:42.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.106+0000 7f0af1c55640 1 -- 172.21.15.44:0/3577700640 wait complete. 2024-04-02T13:31:42.124 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:31:42.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.222+0000 7f403c0d5640 1 Processor -- start 2024-04-02T13:31:42.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.222+0000 7f403c0d5640 1 -- start start 2024-04-02T13:31:42.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f403c0d5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f403415e4a0 0x7f40340aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:42.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f403c0d5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f40340d0c50 0x7f40340d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:42.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f403c0d5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40340cc600 con 0x7f403406ce40 2024-04-02T13:31:42.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f403c0d5640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f40340cc8e0 con 0x7f403415e4a0 2024-04-02T13:31:42.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f403c0d5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f40340ccbc0 con 0x7f40340d0c50 2024-04-02T13:31:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f4039e71640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f403415e4a0 0x7f40340aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f4039e71640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f403415e4a0 0x7f40340aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46310/0 (socket says 172.21.15.44:46310) 2024-04-02T13:31:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f4039e71640 1 -- 172.21.15.44:0/1924901587 learned_addr learned my addr 172.21.15.44:0/1924901587 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f4039e71640 1 -- 172.21.15.44:0/1924901587 >> v2:172.21.15.69:3300/0 conn(0x7f40340d0c50 msgr2=0x7f40340d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f4039670640 1 --2- 172.21.15.44:0/1924901587 >> v2:172.21.15.69:3300/0 conn(0x7f40340d0c50 0x7f40340d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f4039e71640 1 --2- 172.21.15.44:0/1924901587 >> v2:172.21.15.69:3300/0 conn(0x7f40340d0c50 0x7f40340d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f4039e71640 1 -- 172.21.15.44:0/1924901587 >> v1:172.21.15.69:6789/0 conn(0x7f403406ce40 legacy=0x7f40340cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f4039e71640 1 -- 172.21.15.44:0/1924901587 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40340ccea0 con 0x7f403415e4a0 2024-04-02T13:31:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.223+0000 7f4039670640 1 --2- 172.21.15.44:0/1924901587 >> v2:172.21.15.69:3300/0 conn(0x7f40340d0c50 0x7f40340d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.224+0000 7f4039e71640 1 --2- 172.21.15.44:0/1924901587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f403415e4a0 0x7f40340aebf0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f40340b8060 tx=0x7f402c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84464fdf373486af server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.224+0000 7f4038e6f640 1 -- 172.21.15.44:0/1924901587 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c035020 con 0x7f403415e4a0 2024-04-02T13:31:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.224+0000 7f4038e6f640 1 -- 172.21.15.44:0/1924901587 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f402c031870 con 0x7f403415e4a0 2024-04-02T13:31:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.224+0000 7f4038e6f640 1 -- 172.21.15.44:0/1924901587 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c031c40 con 0x7f403415e4a0 2024-04-02T13:31:42.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.225+0000 7f403c0d5640 1 -- 172.21.15.44:0/1924901587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f403415e4a0 msgr2=0x7f40340aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:42.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.225+0000 7f403c0d5640 1 --2- 172.21.15.44:0/1924901587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f403415e4a0 0x7f40340aebf0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f40340b8060 tx=0x7f402c02b910 comp rx=0 tx=0).stop 2024-04-02T13:31:42.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.225+0000 7f403c0d5640 1 -- 172.21.15.44:0/1924901587 shutdown_connections 2024-04-02T13:31:42.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.225+0000 7f403c0d5640 1 --2- 172.21.15.44:0/1924901587 >> v2:172.21.15.69:3300/0 conn(0x7f40340d0c50 0x7f40340d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.225+0000 7f403c0d5640 1 --2- 172.21.15.44:0/1924901587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f403415e4a0 0x7f40340aebf0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.225+0000 7f403c0d5640 1 -- 172.21.15.44:0/1924901587 >> 172.21.15.44:0/1924901587 conn(0x7f403409aa20 msgr2=0x7f403409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.225+0000 7f403c0d5640 1 -- 172.21.15.44:0/1924901587 shutdown_connections 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.225+0000 7f403c0d5640 1 -- 172.21.15.44:0/1924901587 wait complete. 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.226+0000 7f403c0d5640 1 Processor -- start 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.226+0000 7f403c0d5640 1 -- start start 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f403c0d5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f403406ce40 0x7f40341614f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f403c0d5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40340d0c50 0x7f4034152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f403c0d5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f403415e4a0 0x7f40341532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f403c0d5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f403415e880 con 0x7f40340d0c50 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f403c0d5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f40340cd460 con 0x7f403415e4a0 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f403c0d5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f40340af130 con 0x7f403406ce40 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f4039670640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40340d0c50 0x7f4034152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f4039670640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40340d0c50 0x7f4034152da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46318/0 (socket says 172.21.15.44:46318) 2024-04-02T13:31:42.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f4039670640 1 -- 172.21.15.44:0/2851117710 learned_addr learned my addr 172.21.15.44:0/2851117710 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:42.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f403a672640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f403415e4a0 0x7f40341532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:42.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f4039e71640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f403406ce40 0x7f40341614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:42.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f4039670640 1 -- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f403406ce40 msgr2=0x7f40341614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:42.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f4039670640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f403406ce40 0x7f40341614f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f4039670640 1 -- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f403415e4a0 msgr2=0x7f40341532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:42.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f4039670640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f403415e4a0 0x7f40341532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.227+0000 7f4039670640 1 -- 172.21.15.44:0/2851117710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4034153820 con 0x7f40340d0c50 2024-04-02T13:31:42.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.228+0000 7f4039670640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40340d0c50 0x7f4034152da0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f402800c820 tx=0x7f402800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:42.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.228+0000 7f40327fc640 1 -- 172.21.15.44:0/2851117710 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028004230 con 0x7f40340d0c50 2024-04-02T13:31:42.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.228+0000 7f40327fc640 1 -- 172.21.15.44:0/2851117710 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40280043d0 con 0x7f40340d0c50 2024-04-02T13:31:42.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.228+0000 7f40327fc640 1 -- 172.21.15.44:0/2851117710 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028007a70 con 0x7f40340d0c50 2024-04-02T13:31:42.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.228+0000 7f403c0d5640 1 -- 172.21.15.44:0/2851117710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f403415ecb0 con 0x7f40340d0c50 2024-04-02T13:31:42.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.228+0000 7f403c0d5640 1 -- 172.21.15.44:0/2851117710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f403415ef50 con 0x7f40340d0c50 2024-04-02T13:31:42.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.229+0000 7f403c0d5640 1 -- 172.21.15.44:0/2851117710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40340684f0 con 0x7f40340d0c50 2024-04-02T13:31:42.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.229+0000 7f40327fc640 1 -- 172.21.15.44:0/2851117710 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4028007c10 con 0x7f40340d0c50 2024-04-02T13:31:42.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.230+0000 7f40327fc640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f402006c8f0 0x7f402006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:42.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.230+0000 7f40327fc640 1 -- 172.21.15.44:0/2851117710 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f402808bd60 con 0x7f40340d0c50 2024-04-02T13:31:42.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.230+0000 7f4039e71640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f402006c8f0 0x7f402006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:42.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.230+0000 7f4039e71640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f402006c8f0 0x7f402006edb0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f402c036000 tx=0x7f402c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:42.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.234+0000 7f40327fc640 1 -- 172.21.15.44:0/2851117710 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f4028015020 con 0x7f40340d0c50 2024-04-02T13:31:42.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.457+0000 7f403c0d5640 1 -- 172.21.15.44:0/2851117710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40340b4300 con 0x7f40340d0c50 2024-04-02T13:31:42.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.457+0000 7f40327fc640 1 -- 172.21.15.44:0/2851117710 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5178 (secure 0 0 0) 0x7f40280598d0 con 0x7f40340d0c50 2024-04-02T13:31:42.462 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:42.462 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":102,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:31:11.495701+0000","modified":"2024-04-02T13:31:42.107820+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9855},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9855":{"gid":9855,"name":"a","rank":0,"incarnation":102,"state":"up:replay","state_seq":1,"addr":"172.21.15.44:6835/2895101355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":2895101355},{"type":"v1","addr":"172.21.15.44:6835","nonce":2895101355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-04-02T13:31:42.462 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 102 2024-04-02T13:31:42.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.460+0000 7f403c0d5640 1 -- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f402006c8f0 msgr2=0x7f402006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:42.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.460+0000 7f403c0d5640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f402006c8f0 0x7f402006edb0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f402c036000 tx=0x7f402c03c040 comp rx=0 tx=0).stop 2024-04-02T13:31:42.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.460+0000 7f403c0d5640 1 -- 172.21.15.44:0/2851117710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40340d0c50 msgr2=0x7f4034152da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:42.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.460+0000 7f403c0d5640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40340d0c50 0x7f4034152da0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f402800c820 tx=0x7f402800cce0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.460+0000 7f403c0d5640 1 -- 172.21.15.44:0/2851117710 shutdown_connections 2024-04-02T13:31:42.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.460+0000 7f403c0d5640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f402006c8f0 0x7f402006edb0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.460+0000 7f403c0d5640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f403415e4a0 0x7f40341532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.460+0000 7f403c0d5640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40340d0c50 0x7f4034152da0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.460+0000 7f403c0d5640 1 --2- 172.21.15.44:0/2851117710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f403406ce40 0x7f40341614f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:42.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.460+0000 7f403c0d5640 1 -- 172.21.15.44:0/2851117710 >> 172.21.15.44:0/2851117710 conn(0x7f403409aa20 msgr2=0x7f403415d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:42.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.460+0000 7f403c0d5640 1 -- 172.21.15.44:0/2851117710 shutdown_connections 2024-04-02T13:31:42.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:42.460+0000 7f403c0d5640 1 -- 172.21.15.44:0/2851117710 wait complete. 2024-04-02T13:31:42.877 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:31:11.495701+0000', 'modified': '2024-04-02T13:31:42.107820+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 122, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9855}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9855': {'gid': 9855, 'name': 'a', 'rank': 0, 'incarnation': 102, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.44:6835/2895101355', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6833', 'nonce': 2895101355}, {'type': 'v1', 'addr': '172.21.15.44:6835', 'nonce': 2895101355}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-02T13:31:42.877 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9855:up:replay 2024-04-02T13:31:43.878 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:31:43.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.988+0000 7f43f0ae4640 1 Processor -- start 2024-04-02T13:31:43.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.988+0000 7f43f0ae4640 1 -- start start 2024-04-02T13:31:43.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.989+0000 7f43f0ae4640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f43ec06cf90 0x7f43ec0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:43.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.989+0000 7f43f0ae4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f43ec0cfe30 0x7f43ec0d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:43.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.989+0000 7f43f0ae4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43ec06c1d0 con 0x7f43ec0d0a30 2024-04-02T13:31:43.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.989+0000 7f43f0ae4640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f43ec06cb00 con 0x7f43ec06cf90 2024-04-02T13:31:43.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.989+0000 7f43f0ae4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f43ec06d3f0 con 0x7f43ec0cfe30 2024-04-02T13:31:43.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.989+0000 7f43ea59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f43ec0d0a30 0x7f43ec0d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48800/0 (socket says 172.21.15.44:48800) 2024-04-02T13:31:43.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.989+0000 7f43ea59c640 1 -- 172.21.15.44:0/3176447542 learned_addr learned my addr 172.21.15.44:0/3176447542 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:43.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.989+0000 7f43e9d9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f43ec06cf90 0x7f43ec0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:43.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.990+0000 7f43ead9d640 1 --2- 172.21.15.44:0/3176447542 >> v2:172.21.15.69:3300/0 conn(0x7f43ec0cfe30 0x7f43ec0d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:43.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.990+0000 7f43e9d9b640 1 -- 172.21.15.44:0/3176447542 >> v2:172.21.15.69:3300/0 conn(0x7f43ec0cfe30 msgr2=0x7f43ec0d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:43.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.990+0000 7f43e9d9b640 1 --2- 172.21.15.44:0/3176447542 >> v2:172.21.15.69:3300/0 conn(0x7f43ec0cfe30 0x7f43ec0d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:43.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.990+0000 7f43e9d9b640 1 -- 172.21.15.44:0/3176447542 >> v1:172.21.15.44:6789/0 conn(0x7f43ec0d0a30 legacy=0x7f43ec0d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:43.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.990+0000 7f43e959a640 1 -- 172.21.15.44:0/3176447542 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 473648993 0 0) 0x7f43ec06c1d0 con 0x7f43ec0d0a30 2024-04-02T13:31:43.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.990+0000 7f43e9d9b640 1 -- 172.21.15.44:0/3176447542 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ec06c4b0 con 0x7f43ec06cf90 2024-04-02T13:31:43.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.991+0000 7f43e9d9b640 1 --2- 172.21.15.44:0/3176447542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43ec06cf90 0x7f43ec0cf8f0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f43ec08c030 tx=0x7f43d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3112e444da460095 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:43.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.991+0000 7f43e959a640 1 -- 172.21.15.44:0/3176447542 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d40315a0 con 0x7f43ec06cf90 2024-04-02T13:31:43.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.991+0000 7f43e959a640 1 -- 172.21.15.44:0/3176447542 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43d4037070 con 0x7f43ec06cf90 2024-04-02T13:31:43.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.991+0000 7f43e959a640 1 -- 172.21.15.44:0/3176447542 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d4031960 con 0x7f43ec06cf90 2024-04-02T13:31:43.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.991+0000 7f43f0ae4640 1 -- 172.21.15.44:0/3176447542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43ec06cf90 msgr2=0x7f43ec0cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:43.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.991+0000 7f43f0ae4640 1 --2- 172.21.15.44:0/3176447542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43ec06cf90 0x7f43ec0cf8f0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f43ec08c030 tx=0x7f43d402b840 comp rx=0 tx=0).stop 2024-04-02T13:31:43.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.992+0000 7f43f0ae4640 1 -- 172.21.15.44:0/3176447542 shutdown_connections 2024-04-02T13:31:43.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.992+0000 7f43f0ae4640 1 --2- 172.21.15.44:0/3176447542 >> v2:172.21.15.69:3300/0 conn(0x7f43ec0cfe30 0x7f43ec0d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:43.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.992+0000 7f43f0ae4640 1 --2- 172.21.15.44:0/3176447542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43ec06cf90 0x7f43ec0cf8f0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:43.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.992+0000 7f43f0ae4640 1 -- 172.21.15.44:0/3176447542 >> 172.21.15.44:0/3176447542 conn(0x7f43ec09d470 msgr2=0x7f43ec0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:43.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.992+0000 7f43f0ae4640 1 -- 172.21.15.44:0/3176447542 shutdown_connections 2024-04-02T13:31:43.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.992+0000 7f43f0ae4640 1 -- 172.21.15.44:0/3176447542 wait complete. 2024-04-02T13:31:43.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.993+0000 7f43f0ae4640 1 Processor -- start 2024-04-02T13:31:43.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.993+0000 7f43f0ae4640 1 -- start start 2024-04-02T13:31:43.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43f0ae4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43ec06cf90 0x7f43ec1587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:43.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43f0ae4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43ec0cfe30 0x7f43ec0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:43.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43f0ae4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43ec0d0a30 0x7f43ec0c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:43.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43f0ae4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f43ec15e870 con 0x7f43ec0cfe30 2024-04-02T13:31:43.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43f0ae4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f43ec15e6f0 con 0x7f43ec06cf90 2024-04-02T13:31:43.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43f0ae4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f43ec0d4da0 con 0x7f43ec0d0a30 2024-04-02T13:31:43.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43e9d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43ec0cfe30 0x7f43ec0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:43.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43e9d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43ec0cfe30 0x7f43ec0ca3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46336/0 (socket says 172.21.15.44:46336) 2024-04-02T13:31:43.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43e9d9b640 1 -- 172.21.15.44:0/829165033 learned_addr learned my addr 172.21.15.44:0/829165033 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:43.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43ead9d640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43ec0d0a30 0x7f43ec0c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:44.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43ea59c640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43ec06cf90 0x7f43ec1587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:44.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43e9d9b640 1 -- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43ec0d0a30 msgr2=0x7f43ec0c87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:44.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43e9d9b640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43ec0d0a30 0x7f43ec0c87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:44.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43e9d9b640 1 -- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43ec06cf90 msgr2=0x7f43ec1587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:44.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43e9d9b640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43ec06cf90 0x7f43ec1587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:44.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43e9d9b640 1 -- 172.21.15.44:0/829165033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ec0c9350 con 0x7f43ec0cfe30 2024-04-02T13:31:44.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43e9d9b640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43ec0cfe30 0x7f43ec0ca3f0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009890 tx=0x7f43d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:44.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.994+0000 7f43e2ffd640 1 -- 172.21.15.44:0/829165033 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d4032630 con 0x7f43ec0cfe30 2024-04-02T13:31:44.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.995+0000 7f43e2ffd640 1 -- 172.21.15.44:0/829165033 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43d40327d0 con 0x7f43ec0cfe30 2024-04-02T13:31:44.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.995+0000 7f43f0ae4640 1 -- 172.21.15.44:0/829165033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43ec0c9630 con 0x7f43ec0cfe30 2024-04-02T13:31:44.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.995+0000 7f43e2ffd640 1 -- 172.21.15.44:0/829165033 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d4032ba0 con 0x7f43ec0cfe30 2024-04-02T13:31:44.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.995+0000 7f43f0ae4640 1 -- 172.21.15.44:0/829165033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43ec0c9bf0 con 0x7f43ec0cfe30 2024-04-02T13:31:44.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.995+0000 7f43f0ae4640 1 -- 172.21.15.44:0/829165033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43ec0684f0 con 0x7f43ec0cfe30 2024-04-02T13:31:44.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.996+0000 7f43e2ffd640 1 -- 172.21.15.44:0/829165033 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f43d403b3f0 con 0x7f43ec0cfe30 2024-04-02T13:31:44.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.996+0000 7f43e2ffd640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f43cc06c920 0x7f43cc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:44.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.996+0000 7f43e2ffd640 1 -- 172.21.15.44:0/829165033 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f43d402f030 con 0x7f43ec0cfe30 2024-04-02T13:31:44.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.997+0000 7f43ea59c640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f43cc06c920 0x7f43cc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:44.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:43.997+0000 7f43ea59c640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f43cc06c920 0x7f43cc06ede0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f43e4000f60 tx=0x7f43e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:44.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.001+0000 7f43e2ffd640 1 -- 172.21.15.44:0/829165033 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f43d4040070 con 0x7f43ec0cfe30 2024-04-02T13:31:44.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.226+0000 7f43f0ae4640 1 -- 172.21.15.44:0/829165033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43ec15dca0 con 0x7f43ec0cfe30 2024-04-02T13:31:44.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.227+0000 7f43e2ffd640 1 -- 172.21.15.44:0/829165033 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) ==== 95+0+5178 (secure 0 0 0) 0x7f43d407be50 con 0x7f43ec0cfe30 2024-04-02T13:31:44.231 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:44.231 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":104,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:31:11.495701+0000","modified":"2024-04-02T13:31:43.113471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9855},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9855":{"gid":9855,"name":"a","rank":0,"incarnation":102,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.44:6835/2895101355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":2895101355},{"type":"v1","addr":"172.21.15.44:6835","nonce":2895101355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-04-02T13:31:44.231 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 104 2024-04-02T13:31:44.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.229+0000 7f43f0ae4640 1 -- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f43cc06c920 msgr2=0x7f43cc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:44.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.229+0000 7f43f0ae4640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f43cc06c920 0x7f43cc06ede0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f43e4000f60 tx=0x7f43e4009040 comp rx=0 tx=0).stop 2024-04-02T13:31:44.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.229+0000 7f43f0ae4640 1 -- 172.21.15.44:0/829165033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43ec0cfe30 msgr2=0x7f43ec0ca3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:44.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.229+0000 7f43f0ae4640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43ec0cfe30 0x7f43ec0ca3f0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009890 tx=0x7f43d40046c0 comp rx=0 tx=0).stop 2024-04-02T13:31:44.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.229+0000 7f43f0ae4640 1 -- 172.21.15.44:0/829165033 shutdown_connections 2024-04-02T13:31:44.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.229+0000 7f43f0ae4640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f43cc06c920 0x7f43cc06ede0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:44.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.229+0000 7f43f0ae4640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43ec0d0a30 0x7f43ec0c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:44.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.229+0000 7f43f0ae4640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43ec0cfe30 0x7f43ec0ca3f0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:44.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.229+0000 7f43f0ae4640 1 --2- 172.21.15.44:0/829165033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43ec06cf90 0x7f43ec1587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:44.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.229+0000 7f43f0ae4640 1 -- 172.21.15.44:0/829165033 >> 172.21.15.44:0/829165033 conn(0x7f43ec09d470 msgr2=0x7f43ec0d3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:44.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.229+0000 7f43f0ae4640 1 -- 172.21.15.44:0/829165033 shutdown_connections 2024-04-02T13:31:44.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:44.229+0000 7f43f0ae4640 1 -- 172.21.15.44:0/829165033 wait complete. 2024-04-02T13:31:44.645 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:31:11.495701+0000', 'modified': '2024-04-02T13:31:43.113471+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 122, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9855}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9855': {'gid': 9855, 'name': 'a', 'rank': 0, 'incarnation': 102, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.44:6835/2895101355', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6833', 'nonce': 2895101355}, {'type': 'v1', 'addr': '172.21.15.44:6835', 'nonce': 2895101355}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-02T13:31:44.646 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9855:up:rejoin 2024-04-02T13:31:45.646 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:31:45.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.756+0000 7f2fbe88f640 1 Processor -- start 2024-04-02T13:31:45.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.756+0000 7f2fbe88f640 1 -- start start 2024-04-02T13:31:45.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.757+0000 7f2fbe88f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2fb815e4a0 0x7f2fb80aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:45.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.757+0000 7f2fbe88f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806d550 0x7f2fb80d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:45.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.757+0000 7f2fbe88f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb80cc600 con 0x7f2fb80d0c50 2024-04-02T13:31:45.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.757+0000 7f2fbe88f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2fb80cc8e0 con 0x7f2fb815e4a0 2024-04-02T13:31:45.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.757+0000 7f2fbe88f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb80ccbc0 con 0x7f2fb806d550 2024-04-02T13:31:45.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.757+0000 7f2fb77fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f2fb80d0c50 0x7f2fb806ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56592/0 (socket says 172.21.15.44:56592) 2024-04-02T13:31:45.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.757+0000 7f2fb77fe640 1 -- 172.21.15.44:0/1943128586 learned_addr learned my addr 172.21.15.44:0/1943128586 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:45.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.757+0000 7f2fb7fff640 1 --2- 172.21.15.44:0/1943128586 >> v2:172.21.15.69:3300/0 conn(0x7f2fb815e4a0 0x7f2fb80aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:45.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.757+0000 7f2fbce2c640 1 --2- 172.21.15.44:0/1943128586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806d550 0x7f2fb80d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:45.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.757+0000 7f2fb6ffd640 1 -- 172.21.15.44:0/1943128586 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1539322347 0 0) 0x7f2fb80cc600 con 0x7f2fb80d0c50 2024-04-02T13:31:45.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.757+0000 7f2fb6ffd640 1 -- 172.21.15.44:0/1943128586 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fa0003520 con 0x7f2fb80d0c50 2024-04-02T13:31:45.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.758+0000 7f2fb6ffd640 1 -- 172.21.15.44:0/1943128586 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1418278910 0 0) 0x7f2fa0003520 con 0x7f2fb80d0c50 2024-04-02T13:31:45.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.758+0000 7f2fbce2c640 1 -- 172.21.15.44:0/1943128586 >> v2:172.21.15.69:3300/0 conn(0x7f2fb815e4a0 msgr2=0x7f2fb80aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:45.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.758+0000 7f2fbce2c640 1 --2- 172.21.15.44:0/1943128586 >> v2:172.21.15.69:3300/0 conn(0x7f2fb815e4a0 0x7f2fb80aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:45.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.758+0000 7f2fbce2c640 1 -- 172.21.15.44:0/1943128586 >> v1:172.21.15.44:6789/0 conn(0x7f2fb80d0c50 legacy=0x7f2fb806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:45.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.758+0000 7f2fbce2c640 1 -- 172.21.15.44:0/1943128586 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb80ccea0 con 0x7f2fb806d550 2024-04-02T13:31:45.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.758+0000 7f2fb7fff640 1 --2- 172.21.15.44:0/1943128586 >> v2:172.21.15.69:3300/0 conn(0x7f2fb815e4a0 0x7f2fb80aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:31:45.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.758+0000 7f2fbce2c640 1 --2- 172.21.15.44:0/1943128586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806d550 0x7f2fb80d10b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009d90 tx=0x7f2fb002bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c69cbce19e4f43 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.759+0000 7f2fb6ffd640 1 -- 172.21.15.44:0/1943128586 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0039070 con 0x7f2fb806d550 2024-04-02T13:31:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.759+0000 7f2fb6ffd640 1 -- 172.21.15.44:0/1943128586 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb002c8a0 con 0x7f2fb806d550 2024-04-02T13:31:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.759+0000 7f2fb6ffd640 1 -- 172.21.15.44:0/1943128586 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8001670 con 0x7f2fb806d550 2024-04-02T13:31:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.759+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1943128586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806d550 msgr2=0x7f2fb80d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.759+0000 7f2fbe88f640 1 --2- 172.21.15.44:0/1943128586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806d550 0x7f2fb80d10b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009d90 tx=0x7f2fb002bd70 comp rx=0 tx=0).stop 2024-04-02T13:31:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.759+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1943128586 shutdown_connections 2024-04-02T13:31:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.759+0000 7f2fbe88f640 1 --2- 172.21.15.44:0/1943128586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806d550 0x7f2fb80d10b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.759+0000 7f2fbe88f640 1 --2- 172.21.15.44:0/1943128586 >> v2:172.21.15.69:3300/0 conn(0x7f2fb815e4a0 0x7f2fb80aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.759+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1943128586 >> 172.21.15.44:0/1943128586 conn(0x7f2fb809aa20 msgr2=0x7f2fb809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.759+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1943128586 shutdown_connections 2024-04-02T13:31:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.759+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1943128586 wait complete. 2024-04-02T13:31:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.760+0000 7f2fbe88f640 1 Processor -- start 2024-04-02T13:31:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.761+0000 7f2fbe88f640 1 -- start start 2024-04-02T13:31:45.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.761+0000 7f2fbe88f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806d550 0x7f2fb80c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:45.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.761+0000 7f2fbe88f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb80d0c50 0x7f2fb80c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.761+0000 7f2fbe88f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb815e4a0 0x7f2fb80cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.761+0000 7f2fbe88f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb815e880 con 0x7f2fb815e4a0 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.761+0000 7f2fbe88f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb80cd460 con 0x7f2fb80d0c50 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.761+0000 7f2fbe88f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb80af130 con 0x7f2fb806d550 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fbce2c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb815e4a0 0x7f2fb80cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fbce2c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb815e4a0 0x7f2fb80cae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50556/0 (socket says 172.21.15.44:50556) 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fbce2c640 1 -- 172.21.15.44:0/1802587186 learned_addr learned my addr 172.21.15.44:0/1802587186 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fb77fe640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb80d0c50 0x7f2fb80c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fb7fff640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806d550 0x7f2fb80c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fbce2c640 1 -- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806d550 msgr2=0x7f2fb80c5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fbce2c640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806d550 0x7f2fb80c5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fbce2c640 1 -- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb80d0c50 msgr2=0x7f2fb80c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fbce2c640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb80d0c50 0x7f2fb80c6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fbce2c640 1 -- 172.21.15.44:0/1802587186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb80c9510 con 0x7f2fb815e4a0 2024-04-02T13:31:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fbce2c640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb815e4a0 0x7f2fb80cae60 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0037040 tx=0x7f2fb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fb4ff9640 1 -- 172.21.15.44:0/1802587186 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0039050 con 0x7f2fb815e4a0 2024-04-02T13:31:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fb4ff9640 1 -- 172.21.15.44:0/1802587186 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb0045070 con 0x7f2fb815e4a0 2024-04-02T13:31:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1802587186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb80c97f0 con 0x7f2fb815e4a0 2024-04-02T13:31:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fb4ff9640 1 -- 172.21.15.44:0/1802587186 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0009450 con 0x7f2fb815e4a0 2024-04-02T13:31:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.762+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1802587186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb80c9db0 con 0x7f2fb815e4a0 2024-04-02T13:31:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.763+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1802587186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb80684f0 con 0x7f2fb815e4a0 2024-04-02T13:31:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.764+0000 7f2fb4ff9640 1 -- 172.21.15.44:0/1802587186 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2fb0040690 con 0x7f2fb815e4a0 2024-04-02T13:31:45.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.764+0000 7f2fb4ff9640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2fa406c8f0 0x7f2fa406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:45.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.764+0000 7f2fb4ff9640 1 -- 172.21.15.44:0/1802587186 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f2fb00ae6a0 con 0x7f2fb815e4a0 2024-04-02T13:31:45.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.765+0000 7f2fb7fff640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2fa406c8f0 0x7f2fa406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:45.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.765+0000 7f2fb7fff640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2fa406c8f0 0x7f2fa406edb0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009a30 tx=0x7f2fac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:45.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.769+0000 7f2fb4ff9640 1 -- 172.21.15.44:0/1802587186 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f2fb007ca10 con 0x7f2fb815e4a0 2024-04-02T13:31:45.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.992+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1802587186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fb80d1320 con 0x7f2fb815e4a0 2024-04-02T13:31:45.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.993+0000 7f2fb4ff9640 1 -- 172.21.15.44:0/1802587186 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5185 (secure 0 0 0) 0x7f2fb007c310 con 0x7f2fb815e4a0 2024-04-02T13:31:45.997 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:45.997 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":105,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:31:11.495701+0000","modified":"2024-04-02T13:31:45.116238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9855},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9855":{"gid":9855,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6835/2895101355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":2895101355},{"type":"v1","addr":"172.21.15.44:6835","nonce":2895101355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9855,"qdb_cluster":[9855]},"id":8}]} 2024-04-02T13:31:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 105 2024-04-02T13:31:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.995+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2fa406c8f0 msgr2=0x7f2fa406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.995+0000 7f2fbe88f640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2fa406c8f0 0x7f2fa406edb0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009a30 tx=0x7f2fac008040 comp rx=0 tx=0).stop 2024-04-02T13:31:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.995+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1802587186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb815e4a0 msgr2=0x7f2fb80cae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.995+0000 7f2fbe88f640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb815e4a0 0x7f2fb80cae60 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0037040 tx=0x7f2fb0004a40 comp rx=0 tx=0).stop 2024-04-02T13:31:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.995+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1802587186 shutdown_connections 2024-04-02T13:31:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.995+0000 7f2fbe88f640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f2fa406c8f0 0x7f2fa406edb0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.995+0000 7f2fbe88f640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fb815e4a0 0x7f2fb80cae60 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.995+0000 7f2fbe88f640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb80d0c50 0x7f2fb80c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.995+0000 7f2fbe88f640 1 --2- 172.21.15.44:0/1802587186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb806d550 0x7f2fb80c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.995+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1802587186 >> 172.21.15.44:0/1802587186 conn(0x7f2fb809aa20 msgr2=0x7f2fb806c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.995+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1802587186 shutdown_connections 2024-04-02T13:31:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:45.995+0000 7f2fbe88f640 1 -- 172.21.15.44:0/1802587186 wait complete. 2024-04-02T13:31:46.412 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:31:11.495701+0000', 'modified': '2024-04-02T13:31:45.116238+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 122, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9855}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9855': {'gid': 9855, 'name': 'a', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.44:6835/2895101355', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6833', 'nonce': 2895101355}, {'type': 'v1', 'addr': '172.21.15.44:6835', 'nonce': 2895101355}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9855, 'qdb_cluster': [9855]} 2024-04-02T13:31:46.412 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:31:46.412 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T13:31:46.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.514+0000 7f0952b15640 1 Processor -- start 2024-04-02T13:31:46.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.514+0000 7f0952b15640 1 -- start start 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.515+0000 7f0952b15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094c15c250 0x7f094c0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.515+0000 7f0952b15640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f094c0d0aa0 0x7f094c0b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.515+0000 7f0952b15640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f094c0af420 con 0x7f094c0b3510 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.515+0000 7f0952b15640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f094c15c630 con 0x7f094c0d0aa0 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.515+0000 7f0952b15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f094c0af700 con 0x7f094c15c250 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.515+0000 7f09510b2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f094c0d0aa0 0x7f094c0b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.515+0000 7f094bfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f094c0b3510 0x7f094c0d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56606/0 (socket says 172.21.15.44:56606) 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.515+0000 7f094bfff640 1 -- 172.21.15.44:0/2021892403 learned_addr learned my addr 172.21.15.44:0/2021892403 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.516+0000 7f09508b1640 1 --2- 172.21.15.44:0/2021892403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094c15c250 0x7f094c0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.516+0000 7f094b7fe640 1 -- 172.21.15.44:0/2021892403 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 565773236 0 0) 0x7f094c0af420 con 0x7f094c0b3510 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.516+0000 7f094b7fe640 1 -- 172.21.15.44:0/2021892403 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0934003520 con 0x7f094c0b3510 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.516+0000 7f09510b2640 1 -- 172.21.15.44:0/2021892403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094c15c250 msgr2=0x7f094c0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.516+0000 7f09510b2640 1 --2- 172.21.15.44:0/2021892403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094c15c250 0x7f094c0aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.516+0000 7f09510b2640 1 -- 172.21.15.44:0/2021892403 >> v1:172.21.15.44:6789/0 conn(0x7f094c0b3510 legacy=0x7f094c0d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.516+0000 7f09510b2640 1 -- 172.21.15.44:0/2021892403 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094c0af9e0 con 0x7f094c0d0aa0 2024-04-02T13:31:46.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.516+0000 7f09510b2640 1 --2- 172.21.15.44:0/2021892403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094c0d0aa0 0x7f094c0b04c0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f093c0097a0 tx=0x7f093c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fad39c00ca44d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:46.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.517+0000 7f094b7fe640 1 -- 172.21.15.44:0/2021892403 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c0315a0 con 0x7f094c0d0aa0 2024-04-02T13:31:46.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.517+0000 7f094b7fe640 1 -- 172.21.15.44:0/2021892403 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f093c037070 con 0x7f094c0d0aa0 2024-04-02T13:31:46.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.517+0000 7f094b7fe640 1 -- 172.21.15.44:0/2021892403 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c031960 con 0x7f094c0d0aa0 2024-04-02T13:31:46.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.517+0000 7f0952b15640 1 -- 172.21.15.44:0/2021892403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094c0d0aa0 msgr2=0x7f094c0b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.517+0000 7f0952b15640 1 --2- 172.21.15.44:0/2021892403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094c0d0aa0 0x7f094c0b04c0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f093c0097a0 tx=0x7f093c02b840 comp rx=0 tx=0).stop 2024-04-02T13:31:46.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.517+0000 7f0952b15640 1 -- 172.21.15.44:0/2021892403 shutdown_connections 2024-04-02T13:31:46.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.517+0000 7f0952b15640 1 --2- 172.21.15.44:0/2021892403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094c0d0aa0 0x7f094c0b04c0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.517+0000 7f0952b15640 1 --2- 172.21.15.44:0/2021892403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094c15c250 0x7f094c0aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.517+0000 7f0952b15640 1 -- 172.21.15.44:0/2021892403 >> 172.21.15.44:0/2021892403 conn(0x7f094c073850 msgr2=0x7f094c09aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:46.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.517+0000 7f0952b15640 1 -- 172.21.15.44:0/2021892403 shutdown_connections 2024-04-02T13:31:46.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.517+0000 7f0952b15640 1 -- 172.21.15.44:0/2021892403 wait complete. 2024-04-02T13:31:46.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.518+0000 7f0952b15640 1 Processor -- start 2024-04-02T13:31:46.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.518+0000 7f0952b15640 1 -- start start 2024-04-02T13:31:46.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.519+0000 7f0952b15640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f094c0b3510 0x7f094c0c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.519+0000 7f0952b15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094c0d0aa0 0x7f094c151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.519+0000 7f0952b15640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094c15c250 0x7f094c0ced10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.519+0000 7f0952b15640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f094c0b0a00 con 0x7f094c15c250 2024-04-02T13:31:46.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.519+0000 7f0952b15640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f094c0af130 con 0x7f094c0b3510 2024-04-02T13:31:46.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.519+0000 7f0952b15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f094c0b0b80 con 0x7f094c0d0aa0 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.519+0000 7f09510b2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094c15c250 0x7f094c0ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.519+0000 7f09510b2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094c15c250 0x7f094c0ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50584/0 (socket says 172.21.15.44:50584) 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.519+0000 7f09510b2640 1 -- 172.21.15.44:0/694212133 learned_addr learned my addr 172.21.15.44:0/694212133 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.519+0000 7f09508b1640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f094c0b3510 0x7f094c0c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.519+0000 7f094bfff640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094c0d0aa0 0x7f094c151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.520+0000 7f09510b2640 1 -- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094c0d0aa0 msgr2=0x7f094c151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.520+0000 7f09510b2640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094c0d0aa0 0x7f094c151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.520+0000 7f09510b2640 1 -- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f094c0b3510 msgr2=0x7f094c0c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.520+0000 7f09510b2640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f094c0b3510 0x7f094c0c77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.520+0000 7f09510b2640 1 -- 172.21.15.44:0/694212133 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094c0c6720 con 0x7f094c15c250 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.520+0000 7f09508b1640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f094c0b3510 0x7f094c0c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.520+0000 7f094bfff640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094c0d0aa0 0x7f094c151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.520+0000 7f09510b2640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094c15c250 0x7f094c0ced10 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f093c0098c0 tx=0x7f093c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:46.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.520+0000 7f09497fa640 1 -- 172.21.15.44:0/694212133 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c032630 con 0x7f094c15c250 2024-04-02T13:31:46.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.520+0000 7f09497fa640 1 -- 172.21.15.44:0/694212133 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f093c0327d0 con 0x7f094c15c250 2024-04-02T13:31:46.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.520+0000 7f0952b15640 1 -- 172.21.15.44:0/694212133 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f094c0c6a00 con 0x7f094c15c250 2024-04-02T13:31:46.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.520+0000 7f09497fa640 1 -- 172.21.15.44:0/694212133 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c032bc0 con 0x7f094c15c250 2024-04-02T13:31:46.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.520+0000 7f0952b15640 1 -- 172.21.15.44:0/694212133 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f094c0c6fc0 con 0x7f094c15c250 2024-04-02T13:31:46.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.521+0000 7f09497fa640 1 -- 172.21.15.44:0/694212133 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f093c04c020 con 0x7f094c15c250 2024-04-02T13:31:46.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.522+0000 7f09497fa640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f093806c8f0 0x7f093806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.522+0000 7f09497fa640 1 -- 172.21.15.44:0/694212133 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f093c02f020 con 0x7f094c15c250 2024-04-02T13:31:46.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.522+0000 7f09508b1640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f093806c8f0 0x7f093806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.523+0000 7f09508b1640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f093806c8f0 0x7f093806edb0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0944009a80 tx=0x7f09440039d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.523+0000 7f09557d8740 1 Processor -- start 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.523+0000 7f09557d8740 1 -- start start 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f09557d8740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1b90 0x563e5e874b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f09557d8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1f70 0x563e5e8a4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f09557d8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e875070 0x563e5e8a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f09557d8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563e5e7de060 con 0x563e5e8a1f70 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f09557d8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563e5e786650 con 0x563e5e875070 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f09557d8740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x563e5e7da010 con 0x563e5e8a1b90 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f094bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1f70 0x563e5e8a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f094bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1f70 0x563e5e8a4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50594/0 (socket says 172.21.15.44:50594) 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f094bfff640 1 -- 172.21.15.44:0/3338018129 learned_addr learned my addr 172.21.15.44:0/3338018129 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f09508b1640 1 --2- 172.21.15.44:0/3338018129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1b90 0x563e5e874b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f09510b2640 1 --2- 172.21.15.44:0/3338018129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e875070 0x563e5e8a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f094bfff640 1 -- 172.21.15.44:0/3338018129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1b90 msgr2=0x563e5e874b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f094bfff640 1 --2- 172.21.15.44:0/3338018129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1b90 0x563e5e874b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f094bfff640 1 -- 172.21.15.44:0/3338018129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e875070 msgr2=0x563e5e8a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f094bfff640 1 --2- 172.21.15.44:0/3338018129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e875070 0x563e5e8a4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f094bfff640 1 -- 172.21.15.44:0/3338018129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e5e4f22d0 con 0x563e5e8a1f70 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f094bfff640 1 --2- 172.21.15.44:0/3338018129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1f70 0x563e5e8a4920 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f0940001670 tx=0x7f0940000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c74579f027978dd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f09327fc640 1 -- 172.21.15.44:0/3338018129 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940019070 con 0x563e5e8a1f70 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f09327fc640 1 -- 172.21.15.44:0/3338018129 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09400058b0 con 0x563e5e8a1f70 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.524+0000 7f09327fc640 1 -- 172.21.15.44:0/3338018129 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940005c80 con 0x563e5e8a1f70 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.525+0000 7f09557d8740 1 -- 172.21.15.44:0/3338018129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1f70 msgr2=0x563e5e8a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.525+0000 7f09557d8740 1 --2- 172.21.15.44:0/3338018129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1f70 0x563e5e8a4920 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f0940001670 tx=0x7f0940000bc0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.525+0000 7f09557d8740 1 -- 172.21.15.44:0/3338018129 shutdown_connections 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.525+0000 7f09557d8740 1 --2- 172.21.15.44:0/3338018129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e875070 0x563e5e8a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.525+0000 7f09557d8740 1 --2- 172.21.15.44:0/3338018129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1f70 0x563e5e8a4920 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.525+0000 7f09557d8740 1 --2- 172.21.15.44:0/3338018129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1b90 0x563e5e874b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.525+0000 7f09557d8740 1 -- 172.21.15.44:0/3338018129 >> 172.21.15.44:0/3338018129 conn(0x563e5e876da0 msgr2=0x563e5e8bf920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.525+0000 7f09557d8740 1 -- 172.21.15.44:0/3338018129 shutdown_connections 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.525+0000 7f09557d8740 1 -- 172.21.15.44:0/3338018129 wait complete. 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.525+0000 7f09557d8740 1 -- start start 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.526+0000 7f09557d8740 1 Processor -- start 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.526+0000 7f09557d8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8a1b90 0x563e5e894db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.526+0000 7f09557d8740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1f70 0x563e5e8a64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.526+0000 7f09557d8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8ab4d0 0x563e5e8ab930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.526+0000 7f09557d8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563e5e879440 con 0x563e5e8ab4d0 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.526+0000 7f09557d8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563e5e799a10 con 0x563e5e8a1b90 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.526+0000 7f09557d8740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x563e5e87bc80 con 0x563e5e8a1f70 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.526+0000 7f09557d8740 10 client.0 _ll_register_callbacks cb 0x563e5e7d60b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09510b2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8ab4d0 0x563e5e8ab930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09510b2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8ab4d0 0x563e5e8ab930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50598/0 (socket says 172.21.15.44:50598) 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09510b2640 1 -- 172.21.15.44:0/2791414709 learned_addr learned my addr 172.21.15.44:0/2791414709 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09508b1640 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8a1b90 0x563e5e894db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f094bfff640 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1f70 0x563e5e8a64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09510b2640 1 -- 172.21.15.44:0/2791414709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1f70 msgr2=0x563e5e8a64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09510b2640 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1f70 0x563e5e8a64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09510b2640 1 -- 172.21.15.44:0/2791414709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8a1b90 msgr2=0x563e5e894db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09510b2640 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8a1b90 0x563e5e894db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09510b2640 1 -- 172.21.15.44:0/2791414709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e5e5bc6b0 con 0x563e5e8ab4d0 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09508b1640 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8a1b90 0x563e5e894db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09510b2640 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8ab4d0 0x563e5e8ab930 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f093c077910 tx=0x7f093c07bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c022e8a56b2198 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f092b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f092b7fe640 1 -- 172.21.15.44:0/2791414709 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c084090 con 0x563e5e8ab4d0 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f092b7fe640 1 -- 172.21.15.44:0/2791414709 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f093c04cd70 con 0x563e5e8ab4d0 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09557d8740 1 -- 172.21.15.44:0/2791414709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09400058b0 con 0x563e5e8ab4d0 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f092b7fe640 1 -- 172.21.15.44:0/2791414709 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f093c07b710 con 0x563e5e8ab4d0 2024-04-02T13:31:46.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09557d8740 10 client.9909 fetch_fsmap learned FSMap version 105 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f09557d8740 1 -- 172.21.15.44:0/2791414709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x563e5e8cfb60 con 0x563e5e8ab4d0 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.527+0000 7f092b7fe640 1 -- 172.21.15.44:0/2791414709 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09400058b0 con 0x563e5e8ab4d0 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.528+0000 7f092b7fe640 1 -- 172.21.15.44:0/2791414709 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 105) ==== 3494+0+0 (secure 0 0 0) 0x7f093c031070 con 0x563e5e8ab4d0 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.528+0000 7f09557d8740 10 client.9909 fetch_fsmap finished waiting for FSMap version 105 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.528+0000 7f09557d8740 10 client.9909 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.528+0000 7f09557d8740 20 client.9909 populate_metadata read hostname 'smithi044' 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.528+0000 7f09557d8740 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x563e5e8aa210 0x563e5e8aa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.528+0000 7f09557d8740 4 client.9909 mds_command: new command op to 9855 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.528+0000 7f09557d8740 1 -- 172.21.15.44:0/2791414709 --> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e5e785cf0 con 0x563e5e8aa210 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.528+0000 7f09508b1640 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x563e5e8aa210 0x563e5e8aa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.528+0000 7f092b7fe640 10 client.9909 ms_handle_connect on v2:172.21.15.44:6833/2895101355 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.528+0000 7f09508b1640 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x563e5e8aa210 0x563e5e8aa5f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed01013a05eece02 server_cookie=9dc501a7eacddff0 in_seq=0 out_seq=0 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f092b7fe640 1 -- 172.21.15.44:0/2791414709 <== mds.0 v2:172.21.15.44:6833/2895101355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563e5e785cf0 con 0x563e5e8aa210 2024-04-02T13:31:46.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f092b7fe640 10 client.9909 handle_command_reply: tid=0 2024-04-02T13:31:46.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f09557d8740 1 client.9909 shutdown 2024-04-02T13:31:46.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f09557d8740 1 -- 172.21.15.44:0/2791414709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8ab4d0 msgr2=0x563e5e8ab930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f09557d8740 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8ab4d0 0x563e5e8ab930 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f093c077910 tx=0x7f093c07bbb0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f09557d8740 1 -- 172.21.15.44:0/2791414709 shutdown_connections 2024-04-02T13:31:46.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f09557d8740 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x563e5e8aa210 0x563e5e8aa5f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f09557d8740 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8ab4d0 0x563e5e8ab930 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f09557d8740 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1f70 0x563e5e8a64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f092b7fe640 0 client.9909 ms_handle_reset on v2:172.21.15.44:6833/2895101355 2024-04-02T13:31:46.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f09557d8740 1 --2- 172.21.15.44:0/2791414709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8a1b90 0x563e5e894db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f09557d8740 1 -- 172.21.15.44:0/2791414709 >> 172.21.15.44:0/2791414709 conn(0x563e5e875070 msgr2=0x563e5e878b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:46.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f09557d8740 1 -- 172.21.15.44:0/2791414709 shutdown_connections 2024-04-02T13:31:46.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f09557d8740 1 -- 172.21.15.44:0/2791414709 wait complete. 2024-04-02T13:31:46.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.529+0000 7f09557d8740 20 client.9909 trim_cache size 0 max 16384 2024-04-02T13:31:46.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.548+0000 7f09557d8740 1 Processor -- start 2024-04-02T13:31:46.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.548+0000 7f09557d8740 1 -- start start 2024-04-02T13:31:46.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.548+0000 7f09557d8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 0x563e5e8a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.548+0000 7f09557d8740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1f70 0x563e5e8a82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.548+0000 7f09557d8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8ab4d0 0x563e5e873900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.548+0000 7f09557d8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f094c1fdbc0 con 0x563e5e8a1b90 2024-04-02T13:31:46.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.548+0000 7f09557d8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f094c0c9f40 con 0x563e5e8ab4d0 2024-04-02T13:31:46.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.548+0000 7f09557d8740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f094c1fd570 con 0x563e5e8a1f70 2024-04-02T13:31:46.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.548+0000 7f09508b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 0x563e5e8a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.548+0000 7f09508b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 0x563e5e8a7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50614/0 (socket says 172.21.15.44:50614) 2024-04-02T13:31:46.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.548+0000 7f09508b1640 1 -- 172.21.15.44:0/611308421 learned_addr learned my addr 172.21.15.44:0/611308421 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:46.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.549+0000 7f09510b2640 1 --2- 172.21.15.44:0/611308421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8ab4d0 0x563e5e873900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.549+0000 7f094bfff640 1 --2- 172.21.15.44:0/611308421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1f70 0x563e5e8a82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.549+0000 7f09508b1640 1 -- 172.21.15.44:0/611308421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1f70 msgr2=0x563e5e8a82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:46.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.549+0000 7f09508b1640 1 --2- 172.21.15.44:0/611308421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1f70 0x563e5e8a82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.549+0000 7f09508b1640 1 -- 172.21.15.44:0/611308421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8ab4d0 msgr2=0x563e5e873900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.549+0000 7f09508b1640 1 --2- 172.21.15.44:0/611308421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8ab4d0 0x563e5e873900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.549+0000 7f09508b1640 1 -- 172.21.15.44:0/611308421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e5e876a80 con 0x563e5e8a1b90 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.549+0000 7f094bfff640 1 --2- 172.21.15.44:0/611308421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1f70 0x563e5e8a82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.549+0000 7f09508b1640 1 --2- 172.21.15.44:0/611308421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 0x563e5e8a7da0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f0944004010 tx=0x7f0944015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43bcbba32893f232 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.549+0000 7f0931ffb640 1 -- 172.21.15.44:0/611308421 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0944030070 con 0x563e5e8a1b90 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.549+0000 7f0931ffb640 1 -- 172.21.15.44:0/611308421 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0944016df0 con 0x563e5e8a1b90 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.549+0000 7f0931ffb640 1 -- 172.21.15.44:0/611308421 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0944024610 con 0x563e5e8a1b90 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.550+0000 7f09557d8740 1 -- 172.21.15.44:0/611308421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 msgr2=0x563e5e8a7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.550+0000 7f09557d8740 1 --2- 172.21.15.44:0/611308421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 0x563e5e8a7da0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f0944004010 tx=0x7f0944015000 comp rx=0 tx=0).stop 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.550+0000 7f09557d8740 1 -- 172.21.15.44:0/611308421 shutdown_connections 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.550+0000 7f09557d8740 1 --2- 172.21.15.44:0/611308421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8ab4d0 0x563e5e873900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.550+0000 7f09557d8740 1 --2- 172.21.15.44:0/611308421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8a1f70 0x563e5e8a82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.550+0000 7f09557d8740 1 --2- 172.21.15.44:0/611308421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 0x563e5e8a7da0 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.550+0000 7f09557d8740 1 -- 172.21.15.44:0/611308421 >> 172.21.15.44:0/611308421 conn(0x563e5e875070 msgr2=0x563e5e8ab8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:46.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.550+0000 7f09557d8740 1 -- 172.21.15.44:0/611308421 shutdown_connections 2024-04-02T13:31:46.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.550+0000 7f09557d8740 1 -- 172.21.15.44:0/611308421 wait complete. 2024-04-02T13:31:46.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.550+0000 7f09557d8740 1 -- start start 2024-04-02T13:31:46.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.550+0000 7f09557d8740 1 Processor -- start 2024-04-02T13:31:46.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09557d8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 0x563e5e93a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09557d8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8a1f70 0x563e5e8cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09557d8740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8ab4d0 0x563e5e8a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09557d8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563e5e879440 con 0x563e5e8a1b90 2024-04-02T13:31:46.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09557d8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563e5e8aad90 con 0x563e5e8a1f70 2024-04-02T13:31:46.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09557d8740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x563e5e799a10 con 0x563e5e8ab4d0 2024-04-02T13:31:46.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09557d8740 10 client.0 _ll_register_callbacks cb 0x7f093c07b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:31:46.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09508b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 0x563e5e93a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09508b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 0x563e5e93a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50616/0 (socket says 172.21.15.44:50616) 2024-04-02T13:31:46.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09508b1640 1 -- 172.21.15.44:0/2867519722 learned_addr learned my addr 172.21.15.44:0/2867519722 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:46.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f094bfff640 1 --2- 172.21.15.44:0/2867519722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8a1f70 0x563e5e8cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09510b2640 1 --2- 172.21.15.44:0/2867519722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8ab4d0 0x563e5e8a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09508b1640 1 -- 172.21.15.44:0/2867519722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8ab4d0 msgr2=0x563e5e8a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09508b1640 1 --2- 172.21.15.44:0/2867519722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8ab4d0 0x563e5e8a7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09508b1640 1 -- 172.21.15.44:0/2867519722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8a1f70 msgr2=0x563e5e8cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09508b1640 1 --2- 172.21.15.44:0/2867519722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8a1f70 0x563e5e8cdff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.551+0000 7f09508b1640 1 -- 172.21.15.44:0/2867519722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e5e8d9e20 con 0x563e5e8a1b90 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f09508b1640 1 --2- 172.21.15.44:0/2867519722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 0x563e5e93a8f0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f094402e040 tx=0x7f0944004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6ffdb5fb35d642 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f092b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f092b7fe640 1 -- 172.21.15.44:0/2867519722 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0944039060 con 0x563e5e8a1b90 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f092b7fe640 1 -- 172.21.15.44:0/2867519722 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f0944025950 con 0x563e5e8a1b90 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f09557d8740 1 -- 172.21.15.44:0/2867519722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0944016df0 con 0x563e5e8a1b90 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f092b7fe640 1 -- 172.21.15.44:0/2867519722 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f094403f050 con 0x563e5e8a1b90 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f09557d8740 10 client.9915 fetch_fsmap learned FSMap version 105 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f09557d8740 1 -- 172.21.15.44:0/2867519722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x563e5e8ce930 con 0x563e5e8a1b90 2024-04-02T13:31:46.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f092b7fe640 1 -- 172.21.15.44:0/2867519722 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0944016df0 con 0x563e5e8a1b90 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f092b7fe640 1 -- 172.21.15.44:0/2867519722 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 105) ==== 3494+0+0 (secure 0 0 0) 0x7f0944042020 con 0x563e5e8a1b90 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f09557d8740 10 client.9915 fetch_fsmap finished waiting for FSMap version 105 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f09557d8740 10 client.9915 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f09557d8740 20 client.9915 populate_metadata read hostname 'smithi044' 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f09557d8740 1 --2- 172.21.15.44:0/2867519722 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x563e5e8a82e0 0x563e5e8c0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f09557d8740 4 client.9915 mds_command: new command op to 9855 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f09557d8740 1 -- 172.21.15.44:0/2867519722 --> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] -- command(tid 0: {"prefix": "status"}) -- 0x563e5e7eb5a0 con 0x563e5e8a82e0 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.552+0000 7f094bfff640 1 --2- 172.21.15.44:0/2867519722 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x563e5e8a82e0 0x563e5e8c0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f092b7fe640 10 client.9915 ms_handle_connect on v2:172.21.15.44:6833/2895101355 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f094bfff640 1 --2- 172.21.15.44:0/2867519722 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x563e5e8a82e0 0x563e5e8c0a50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3260b3768bca3aba server_cookie=91ec4a5118574fa3 in_seq=0 out_seq=0 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f092b7fe640 1 -- 172.21.15.44:0/2867519722 <== mds.0 v2:172.21.15.44:6833/2895101355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x563e5e7eb5a0 con 0x563e5e8a82e0 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f092b7fe640 10 client.9915 handle_command_reply: tid=0 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f09557d8740 1 client.9915 shutdown 2024-04-02T13:31:46.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f09557d8740 1 -- 172.21.15.44:0/2867519722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 msgr2=0x563e5e93a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f09557d8740 1 --2- 172.21.15.44:0/2867519722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 0x563e5e93a8f0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f094402e040 tx=0x7f0944004ab0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f09557d8740 1 -- 172.21.15.44:0/2867519722 shutdown_connections 2024-04-02T13:31:46.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f09557d8740 1 --2- 172.21.15.44:0/2867519722 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x563e5e8a82e0 0x563e5e8c0a50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f09557d8740 1 --2- 172.21.15.44:0/2867519722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563e5e8ab4d0 0x563e5e8a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f092b7fe640 0 client.9915 ms_handle_reset on v2:172.21.15.44:6833/2895101355 2024-04-02T13:31:46.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f09557d8740 1 --2- 172.21.15.44:0/2867519722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563e5e8a1f70 0x563e5e8cdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f09557d8740 1 --2- 172.21.15.44:0/2867519722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563e5e8a1b90 0x563e5e93a8f0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f09557d8740 1 -- 172.21.15.44:0/2867519722 >> 172.21.15.44:0/2867519722 conn(0x563e5e875070 msgr2=0x563e5e8a5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:46.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f09557d8740 1 -- 172.21.15.44:0/2867519722 shutdown_connections 2024-04-02T13:31:46.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f09557d8740 1 -- 172.21.15.44:0/2867519722 wait complete. 2024-04-02T13:31:46.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.553+0000 7f09557d8740 20 client.9915 trim_cache size 0 max 16384 2024-04-02T13:31:46.564 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:31:46.564 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:31:46.564 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:31:46.564 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 9855, 2024-04-02T13:31:46.564 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:31:46.564 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:31:46.564 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:31:46.564 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 4.4428972719999997, 2024-04-02T13:31:46.564 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 105, 2024-04-02T13:31:46.564 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 122, 2024-04-02T13:31:46.564 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 122, 2024-04-02T13:31:46.564 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 5.4058088550000001 2024-04-02T13:31:46.564 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:31:46.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.554+0000 7f0952b15640 1 -- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f093806c8f0 msgr2=0x7f093806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.554+0000 7f0952b15640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f093806c8f0 0x7f093806edb0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0944009a80 tx=0x7f09440039d0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.554+0000 7f0952b15640 1 -- 172.21.15.44:0/694212133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094c15c250 msgr2=0x7f094c0ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.554+0000 7f0952b15640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094c15c250 0x7f094c0ced10 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f093c0098c0 tx=0x7f093c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.554+0000 7f0952b15640 1 -- 172.21.15.44:0/694212133 shutdown_connections 2024-04-02T13:31:46.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.554+0000 7f0952b15640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f093806c8f0 0x7f093806edb0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.554+0000 7f0952b15640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f094c15c250 0x7f094c0ced10 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.554+0000 7f0952b15640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094c0d0aa0 0x7f094c151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.554+0000 7f0952b15640 1 --2- 172.21.15.44:0/694212133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f094c0b3510 0x7f094c0c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.554+0000 7f0952b15640 1 -- 172.21.15.44:0/694212133 >> 172.21.15.44:0/694212133 conn(0x7f094c073850 msgr2=0x7f094c06f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:46.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.554+0000 7f0952b15640 1 -- 172.21.15.44:0/694212133 shutdown_connections 2024-04-02T13:31:46.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.554+0000 7f0952b15640 1 -- 172.21.15.44:0/694212133 wait complete. 2024-04-02T13:31:46.569 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:31:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.666+0000 7ff727e0d640 1 Processor -- start 2024-04-02T13:31:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.666+0000 7ff727e0d640 1 -- start start 2024-04-02T13:31:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.667+0000 7ff727e0d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff720166f50 0x7ff720169340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.667+0000 7ff727e0d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7200cf5e0 con 0x7ff72015e430 2024-04-02T13:31:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.667+0000 7ff727e0d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff72015e810 con 0x7ff72006c7f0 2024-04-02T13:31:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.667+0000 7ff727e0d640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff7200d1410 con 0x7ff720166f50 2024-04-02T13:31:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.667+0000 7ff7263aa640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff720166f50 0x7ff720169340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.667+0000 7ff725ba9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff72015e430 0x7ff7200ce7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56610/0 (socket says 172.21.15.44:56610) 2024-04-02T13:31:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.667+0000 7ff725ba9640 1 -- 172.21.15.44:0/33930634 learned_addr learned my addr 172.21.15.44:0/33930634 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.668+0000 7ff7263aa640 1 -- 172.21.15.44:0/33930634 >> v1:172.21.15.69:6789/0 conn(0x7ff72006c7f0 legacy=0x7ff7200ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.668+0000 7ff7263aa640 1 -- 172.21.15.44:0/33930634 >> v1:172.21.15.44:6789/0 conn(0x7ff72015e430 legacy=0x7ff7200ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.668+0000 7ff724ba7640 1 -- 172.21.15.44:0/33930634 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1041113191 0 0) 0x7ff7200cf5e0 con 0x7ff72015e430 2024-04-02T13:31:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.668+0000 7ff7263aa640 1 -- 172.21.15.44:0/33930634 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff72006d240 con 0x7ff720166f50 2024-04-02T13:31:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.668+0000 7ff7263aa640 1 --2- 172.21.15.44:0/33930634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff720166f50 0x7ff720169340 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7ff71c009d90 tx=0x7ff71c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ccb55ecffd7667 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.669+0000 7ff724ba7640 1 -- 172.21.15.44:0/33930634 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c039070 con 0x7ff720166f50 2024-04-02T13:31:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.669+0000 7ff724ba7640 1 -- 172.21.15.44:0/33930634 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff71c02cc60 con 0x7ff720166f50 2024-04-02T13:31:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.669+0000 7ff724ba7640 1 -- 172.21.15.44:0/33930634 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c034660 con 0x7ff720166f50 2024-04-02T13:31:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.669+0000 7ff727e0d640 1 -- 172.21.15.44:0/33930634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff720166f50 msgr2=0x7ff720169340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.669+0000 7ff727e0d640 1 --2- 172.21.15.44:0/33930634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff720166f50 0x7ff720169340 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7ff71c009d90 tx=0x7ff71c02beb0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.669+0000 7ff727e0d640 1 -- 172.21.15.44:0/33930634 shutdown_connections 2024-04-02T13:31:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.669+0000 7ff727e0d640 1 --2- 172.21.15.44:0/33930634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff720166f50 0x7ff720169340 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.669+0000 7ff727e0d640 1 -- 172.21.15.44:0/33930634 >> 172.21.15.44:0/33930634 conn(0x7ff720073850 msgr2=0x7ff72009aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.669+0000 7ff727e0d640 1 -- 172.21.15.44:0/33930634 shutdown_connections 2024-04-02T13:31:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.669+0000 7ff727e0d640 1 -- 172.21.15.44:0/33930634 wait complete. 2024-04-02T13:31:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.670+0000 7ff727e0d640 1 Processor -- start 2024-04-02T13:31:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.670+0000 7ff727e0d640 1 -- start start 2024-04-02T13:31:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff727e0d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff72006c7f0 0x7ff7201566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff727e0d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff72015e430 0x7ff7201597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff727e0d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff720166f50 0x7ff720159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff727e0d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff7200d0c10 con 0x7ff72015e430 2024-04-02T13:31:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff727e0d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff72006cff0 con 0x7ff72006c7f0 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff727e0d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff7200d0ef0 con 0x7ff720166f50 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff7253a8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff72015e430 0x7ff7201597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff7253a8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff72015e430 0x7ff7201597f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50636/0 (socket says 172.21.15.44:50636) 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff7253a8640 1 -- 172.21.15.44:0/2888190028 learned_addr learned my addr 172.21.15.44:0/2888190028 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff725ba9640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff72006c7f0 0x7ff7201566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff7263aa640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff720166f50 0x7ff720159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff7253a8640 1 -- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff720166f50 msgr2=0x7ff720159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff7253a8640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff720166f50 0x7ff720159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff7253a8640 1 -- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff72006c7f0 msgr2=0x7ff7201566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff7253a8640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff72006c7f0 0x7ff7201566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.671+0000 7ff7253a8640 1 -- 172.21.15.44:0/2888190028 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff720156bf0 con 0x7ff72015e430 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.672+0000 7ff7253a8640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff72015e430 0x7ff7201597f0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7ff71000c400 tx=0x7ff71000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.672+0000 7ff7167fc640 1 -- 172.21.15.44:0/2888190028 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710017070 con 0x7ff72015e430 2024-04-02T13:31:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.672+0000 7ff7167fc640 1 -- 172.21.15.44:0/2888190028 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7100052d0 con 0x7ff72015e430 2024-04-02T13:31:46.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.672+0000 7ff727e0d640 1 -- 172.21.15.44:0/2888190028 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7200d2810 con 0x7ff72015e430 2024-04-02T13:31:46.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.672+0000 7ff7167fc640 1 -- 172.21.15.44:0/2888190028 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710003830 con 0x7ff72015e430 2024-04-02T13:31:46.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.672+0000 7ff727e0d640 1 -- 172.21.15.44:0/2888190028 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff72015bf00 con 0x7ff72015e430 2024-04-02T13:31:46.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.673+0000 7ff727e0d640 1 -- 172.21.15.44:0/2888190028 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff720068c00 con 0x7ff72015e430 2024-04-02T13:31:46.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.673+0000 7ff7167fc640 1 -- 172.21.15.44:0/2888190028 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff71001a070 con 0x7ff72015e430 2024-04-02T13:31:46.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.674+0000 7ff7167fc640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff70406c8f0 0x7ff70406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:46.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.674+0000 7ff7167fc640 1 -- 172.21.15.44:0/2888190028 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7ff71008afe0 con 0x7ff72015e430 2024-04-02T13:31:46.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.674+0000 7ff725ba9640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff70406c8f0 0x7ff70406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:46.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.674+0000 7ff725ba9640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff70406c8f0 0x7ff70406edb0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff718007f70 tx=0x7ff718007f00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:46.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.678+0000 7ff7167fc640 1 -- 172.21.15.44:0/2888190028 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff710058ba0 con 0x7ff72015e430 2024-04-02T13:31:46.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.900+0000 7ff727e0d640 1 -- 172.21.15.44:0/2888190028 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff720158790 con 0x7ff72015e430 2024-04-02T13:31:46.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.900+0000 7ff7167fc640 1 -- 172.21.15.44:0/2888190028 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v105) ==== 53+0+73 (secure 0 0 0) 0x7ff71005cea0 con 0x7ff72015e430 2024-04-02T13:31:46.904 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:31:46.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.902+0000 7ff727e0d640 1 -- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff70406c8f0 msgr2=0x7ff70406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.902+0000 7ff727e0d640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff70406c8f0 0x7ff70406edb0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff718007f70 tx=0x7ff718007f00 comp rx=0 tx=0).stop 2024-04-02T13:31:46.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.902+0000 7ff727e0d640 1 -- 172.21.15.44:0/2888190028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff72015e430 msgr2=0x7ff7201597f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:46.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.902+0000 7ff727e0d640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff72015e430 0x7ff7201597f0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7ff71000c400 tx=0x7ff71000c8c0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.902+0000 7ff727e0d640 1 -- 172.21.15.44:0/2888190028 shutdown_connections 2024-04-02T13:31:46.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.902+0000 7ff727e0d640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff70406c8f0 0x7ff70406edb0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.902+0000 7ff727e0d640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff720166f50 0x7ff720159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.902+0000 7ff727e0d640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff72015e430 0x7ff7201597f0 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.902+0000 7ff727e0d640 1 --2- 172.21.15.44:0/2888190028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff72006c7f0 0x7ff7201566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:46.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.902+0000 7ff727e0d640 1 -- 172.21.15.44:0/2888190028 >> 172.21.15.44:0/2888190028 conn(0x7ff720073850 msgr2=0x7ff7200b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:46.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.902+0000 7ff727e0d640 1 -- 172.21.15.44:0/2888190028 shutdown_connections 2024-04-02T13:31:46.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:46.902+0000 7ff727e0d640 1 -- 172.21.15.44:0/2888190028 wait complete. 2024-04-02T13:31:46.916 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:31:46.916 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:31:46.916 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:31:46.916 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:31:46.917 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:46.917 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:31:46.917 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:31:46.917 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:31:46.917 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:31:46.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:46.917 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:31:46.932 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:31:46.932 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:31:46.932 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:31:46.932 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42072sec preferred_lft 42072sec 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:46.933 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:31:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:46.935 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:31:46.935 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:31:46.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:46.935 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:31:46.997 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:31:46.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:46.999 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:31:47.055 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:31:47.055 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:31:47.055 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:31:47.055 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:31:47.055 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:31:47.056 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:31:47.056 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:31:47.056 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:31:47.056 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:31:47.056 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:31:47.056 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:31:47.115 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:31:47.111+0000 7f89f9a82640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:31:47.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:31:47.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:47.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:47.192 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:31:47.247 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:31:47.247 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:31:47.247 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:31:47.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:47.247 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:31:47.314 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:31:47.314 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:31:47.314 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:47.368 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:31:47.368 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:31:47.368 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:47.422 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:31:47.490 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:31:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:31:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:31:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:31:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:31:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:31:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:31:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:31:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:31:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:31:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:31:47.548 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:31:47.548 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:31:47.548 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:31:47.548 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:31:47.548 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:31:47.548 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:31:47.548 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:47.548 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:47.548 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:47.548 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:47.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:47.548 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:31:47.600 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:31:47.601 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:31:47.643 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:31:47.669 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:31:47.725 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:31:47.725 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:31:47.725 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:31:47.725 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:31:47.725 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:31:47.725 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:31:47.725 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:31:47.725 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:31:47.726 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:31:47.726 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:31:47.726 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:31:47.726 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:31:47.726 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:31:47.726 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:31:47.726 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:31:47.726 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:31:47.726 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:31:47.726 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:31:47.726 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:31:47.726 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:31:47.726 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:31:47.727 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:31:47.727 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:31:47.727 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:31:47.727 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:31:47.727 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:31:47.727 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:31:47.727 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:31:47.727 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:31:47.727 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:31:47.727 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:47.727 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:31:47.727 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:47.727 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:31:47.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:47.728 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:31:47.729 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:31:47.723+0000 7f2892660480 -1 init, newargv = 0x556fd8d08340 newargc=15 2024-04-02T13:31:47.729 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[84152]: starting ceph client 2024-04-02T13:31:47.731 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[84152]: starting fuse 2024-04-02T13:31:47.745 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:31:47.745 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:31:47.745 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:31:47.745 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:31:47.745 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:31:47.746 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:47.746 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:31:47.746 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:47.746 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:31:47.746 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:31:47.746 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:31:47.746 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:31:47.746 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:47.746 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:31:47.746 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:31:47.746 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:31:47.746 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:31:47.746 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:47.747 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:31:47.747 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:31:47.747 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:31:47.747 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:31:47.747 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:31:47.747 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:31:47.747 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:31:47.747 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:31:47.747 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:47.747 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:31:47.747 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:47.864 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.84152.asok 2024-04-02T13:31:47.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:47.874 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.84152.asok 2024-04-02T13:31:47.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:47.874 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.84152.asok status 2024-04-02T13:31:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:31:48.012 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:31:48.012 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:31:48.012 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:31:48.012 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:31:48.013 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:31:48.013 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:31:48.013 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "84152", 2024-04-02T13:31:48.013 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:31:48.013 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:48.013 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:31:48.013 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:31:48.013 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 9927, 2024-04-02T13:31:48.013 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:31:48.013 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:31:48.013 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:31:48.013 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 9927 2024-04-02T13:31:48.014 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:48.014 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:31:48.014 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:31:48.014 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:31:48.014 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 1838286368 2024-04-02T13:31:48.014 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:31:48.014 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:48.014 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:31:48.014 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:31:48.014 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:31:48.015 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 1838286368 2024-04-02T13:31:48.015 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:31:48.015 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.9927 192.168.0.1:0/1838286368", 2024-04-02T13:31:48.015 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/1838286368", 2024-04-02T13:31:48.015 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:31:48.015 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 105, 2024-04-02T13:31:48.015 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 122, 2024-04-02T13:31:48.015 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:31:48.015 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:31:48.015 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:31:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:31:48.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:48.021 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:31:48.021 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:31:48.036 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:48.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:48.036 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:48.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:48.103 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-04-02T13:31:48.166 INFO:teuthology.orchestra.run.smithi044.stdout:./ 2024-04-02T13:31:48.166 INFO:teuthology.orchestra.run.smithi044.stdout:./parent 2024-04-02T13:31:48.167 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/flushed 2024-04-02T13:31:48.167 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/flushed/charlie 2024-04-02T13:31:48.167 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/flushed/alpha 2024-04-02T13:31:48.167 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/unflushed 2024-04-02T13:31:48.167 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/unflushed/jfile 2024-04-02T13:31:48.167 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:48.167 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:48.167 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:48.167 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:48.167 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-04-02T13:31:48.167 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:48.272 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:31:48.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:48.281 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:48.281 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:48.281 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:48.281 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:48.281 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-04-02T13:31:48.281 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:48.386 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627776 2024-04-02T13:31:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:48.395 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:48.395 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:48.395 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:48.395 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:48.395 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-04-02T13:31:48.395 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:48.501 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627777 2024-04-02T13:31:48.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:48.510 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:48.510 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:48.510 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:48.510 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:48.510 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-04-02T13:31:48.510 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:48.615 INFO:teuthology.orchestra.run.smithi044.stdout:1099511628280 2024-04-02T13:31:48.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:48.624 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:48.624 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:48.624 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:48.624 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:48.624 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-04-02T13:31:48.624 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:48.729 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627778 2024-04-02T13:31:48.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:48.738 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:48.738 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:48.738 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:48.738 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:48.739 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-04-02T13:31:48.739 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:48.844 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627779 2024-04-02T13:31:48.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:48.853 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:31:48.853 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:31:48.853 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:31:48.853 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:31:48.853 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-04-02T13:31:48.853 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:31:48.958 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627780 2024-04-02T13:31:48.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:31:48.968 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:31:48.968 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:31:49.023 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:31:49.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.162+0000 7f338220d640 1 Processor -- start 2024-04-02T13:31:49.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.162+0000 7f338220d640 1 -- start start 2024-04-02T13:31:49.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f338220d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f337c0b2d00 0x7f337c0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f338220d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f337c0ce110 0x7f337c071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f338220d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f337c0b3160 con 0x7f337c15aa00 2024-04-02T13:31:49.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f338220d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f337c15ade0 con 0x7f337c0ce110 2024-04-02T13:31:49.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f338220d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f337c0af1b0 con 0x7f337c0b2d00 2024-04-02T13:31:49.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337b7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f337c15aa00 0x7f337c072bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56618/0 (socket says 172.21.15.44:56618) 2024-04-02T13:31:49.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337b7fe640 1 -- 172.21.15.44:0/3335023337 learned_addr learned my addr 172.21.15.44:0/3335023337 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:49.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242413609 0 0) 0x7f337c0b3160 con 0x7f337c15aa00 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3364003520 con 0x7f337c15aa00 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337bfff640 1 --2- 172.21.15.44:0/3335023337 >> v2:172.21.15.69:3300/0 conn(0x7f337c0ce110 0x7f337c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337affd640 1 --2- 172.21.15.44:0/3335023337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f337c0b2d00 0x7f337c0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 825537781 0 0) 0x7f3364003520 con 0x7f337c15aa00 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f337c0b2d00 msgr2=0x7f337c0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 --2- 172.21.15.44:0/3335023337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f337c0b2d00 0x7f337c0cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 >> v2:172.21.15.69:3300/0 conn(0x7f337c0ce110 msgr2=0x7f337c071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 --2- 172.21.15.44:0/3335023337 >> v2:172.21.15.69:3300/0 conn(0x7f337c0ce110 0x7f337c071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337c0af490 con 0x7f337c15aa00 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f336c001670 con 0x7f337c15aa00 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 >> v1:172.21.15.44:6789/0 conn(0x7f337c15aa00 legacy=0x7f337c072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 --2- 172.21.15.44:0/3335023337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33640039b0 0x7f3364003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 --2- 172.21.15.44:0/3335023337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f337c0b2d00 0x7f3364004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 --2- 172.21.15.44:0/3335023337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3364004d40 0x7f33640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f337c15ade0 con 0x7f33640039b0 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f337c0af1b0 con 0x7f3364004d40 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f336400b9c0 con 0x7f337c0b2d00 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.163+0000 7f337b7fe640 1 --2- 172.21.15.44:0/3335023337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33640039b0 0x7f3364003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.164+0000 7f337affd640 1 --2- 172.21.15.44:0/3335023337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f337c0b2d00 0x7f3364004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.164+0000 7f337bfff640 1 --2- 172.21.15.44:0/3335023337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3364004d40 0x7f33640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.164+0000 7f337b7fe640 1 -- 172.21.15.44:0/3335023337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f337c0b2d00 msgr2=0x7f3364004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.164+0000 7f337b7fe640 1 --2- 172.21.15.44:0/3335023337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f337c0b2d00 0x7f3364004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.164+0000 7f337b7fe640 1 -- 172.21.15.44:0/3335023337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3364004d40 msgr2=0x7f33640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.164+0000 7f337b7fe640 1 --2- 172.21.15.44:0/3335023337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3364004d40 0x7f33640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.164+0000 7f337b7fe640 1 -- 172.21.15.44:0/3335023337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3364003520 con 0x7f33640039b0 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.165+0000 7f337b7fe640 1 --2- 172.21.15.44:0/3335023337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33640039b0 0x7f3364003dd0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f336c007fd0 tx=0x7f3364005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=216d3eec10c799ce server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.165+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c007d50 con 0x7f33640039b0 2024-04-02T13:31:49.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.165+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f336c030070 con 0x7f33640039b0 2024-04-02T13:31:49.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.165+0000 7f337a7fc640 1 -- 172.21.15.44:0/3335023337 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c02cd40 con 0x7f33640039b0 2024-04-02T13:31:49.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.165+0000 7f338220d640 1 -- 172.21.15.44:0/3335023337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33640039b0 msgr2=0x7f3364003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.165+0000 7f338220d640 1 --2- 172.21.15.44:0/3335023337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33640039b0 0x7f3364003dd0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f336c007fd0 tx=0x7f3364005830 comp rx=0 tx=0).stop 2024-04-02T13:31:49.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.165+0000 7f337bfff640 1 -- 172.21.15.44:0/3335023337 reap_dead start 2024-04-02T13:31:49.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.165+0000 7f338220d640 1 -- 172.21.15.44:0/3335023337 shutdown_connections 2024-04-02T13:31:49.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.165+0000 7f338220d640 1 -- 172.21.15.44:0/3335023337 >> 172.21.15.44:0/3335023337 conn(0x7f337c0cfcd0 msgr2=0x7f337c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:49.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.165+0000 7f338220d640 1 -- 172.21.15.44:0/3335023337 shutdown_connections 2024-04-02T13:31:49.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.165+0000 7f338220d640 1 -- 172.21.15.44:0/3335023337 wait complete. 2024-04-02T13:31:49.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.166+0000 7f338220d640 1 Processor -- start 2024-04-02T13:31:49.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.167+0000 7f338220d640 1 -- start start 2024-04-02T13:31:49.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.167+0000 7f338220d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33640039b0 0x7f337c14cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.167+0000 7f338220d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f337c15c040 0x7f337c09c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.167+0000 7f338220d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f337c15c4c0 0x7f337c09cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.167+0000 7f338220d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f337c0ce5a0 con 0x7f33640039b0 2024-04-02T13:31:49.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.167+0000 7f338220d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f337c0726d0 con 0x7f337c15c4c0 2024-04-02T13:31:49.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.167+0000 7f338220d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f337c0cdbd0 con 0x7f337c15c040 2024-04-02T13:31:49.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.167+0000 7f337b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33640039b0 0x7f337c14cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.167+0000 7f337b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33640039b0 0x7f337c14cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50650/0 (socket says 172.21.15.44:50650) 2024-04-02T13:31:49.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.167+0000 7f337b7fe640 1 -- 172.21.15.44:0/3849584829 learned_addr learned my addr 172.21.15.44:0/3849584829 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:49.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.167+0000 7f337affd640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f337c15c040 0x7f337c09c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.167+0000 7f337bfff640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f337c15c4c0 0x7f337c09cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.168+0000 7f337b7fe640 1 -- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f337c15c040 msgr2=0x7f337c09c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.168+0000 7f337b7fe640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f337c15c040 0x7f337c09c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.168+0000 7f337b7fe640 1 -- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f337c15c4c0 msgr2=0x7f337c09cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.168+0000 7f337b7fe640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f337c15c4c0 0x7f337c09cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.168+0000 7f337b7fe640 1 -- 172.21.15.44:0/3849584829 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337c15ca20 con 0x7f33640039b0 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.168+0000 7f337b7fe640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33640039b0 0x7f337c14cbb0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f336c02c570 tx=0x7f336c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.168+0000 7f3363fff640 1 -- 172.21.15.44:0/3849584829 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c001cb0 con 0x7f33640039b0 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.168+0000 7f3363fff640 1 -- 172.21.15.44:0/3849584829 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f336c004bf0 con 0x7f33640039b0 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.168+0000 7f338220d640 1 -- 172.21.15.44:0/3849584829 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337c15cd00 con 0x7f33640039b0 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.168+0000 7f3363fff640 1 -- 172.21.15.44:0/3849584829 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c007070 con 0x7f33640039b0 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.168+0000 7f338220d640 1 -- 172.21.15.44:0/3849584829 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f337c15d2c0 con 0x7f33640039b0 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.169+0000 7f338220d640 1 -- 172.21.15.44:0/3849584829 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f337c074f20 con 0x7f33640039b0 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.169+0000 7f3363fff640 1 -- 172.21.15.44:0/3849584829 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f336c007210 con 0x7f33640039b0 2024-04-02T13:31:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.170+0000 7f3363fff640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f336806c8f0 0x7f336806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.170+0000 7f3363fff640 1 -- 172.21.15.44:0/3849584829 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f336c0ad0e0 con 0x7f33640039b0 2024-04-02T13:31:49.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.170+0000 7f337affd640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f336806c8f0 0x7f336806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.171+0000 7f337affd640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f336806c8f0 0x7f336806edb0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f337c0ae8f0 tx=0x7f3370012040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:49.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.174+0000 7f3363fff640 1 -- 172.21.15.44:0/3849584829 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f336c034e90 con 0x7f33640039b0 2024-04-02T13:31:49.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.363+0000 7f338220d640 1 -- 172.21.15.44:0/3849584829 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f337c0684f0 con 0x7f33640039b0 2024-04-02T13:31:49.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.364+0000 7f3363fff640 1 -- 172.21.15.44:0/3849584829 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) ==== 74+0+17033 (secure 0 0 0) 0x7f336c07b4d0 con 0x7f33640039b0 2024-04-02T13:31:49.368 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:49.368 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":122,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:41.102907+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:31:09.752845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-02T13:31:10.437454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6835/2109578276":"2024-04-03T13:31:41.102885+0000","172.21.15.44:6833/2109578276":"2024-04-03T13:31:41.102885+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:31:49.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.366+0000 7f338220d640 1 -- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f336806c8f0 msgr2=0x7f336806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.366+0000 7f338220d640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f336806c8f0 0x7f336806edb0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f337c0ae8f0 tx=0x7f3370012040 comp rx=0 tx=0).stop 2024-04-02T13:31:49.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.366+0000 7f338220d640 1 -- 172.21.15.44:0/3849584829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33640039b0 msgr2=0x7f337c14cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.366+0000 7f338220d640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33640039b0 0x7f337c14cbb0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f336c02c570 tx=0x7f336c000fe0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.366+0000 7f338220d640 1 -- 172.21.15.44:0/3849584829 shutdown_connections 2024-04-02T13:31:49.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.366+0000 7f338220d640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f336806c8f0 0x7f336806edb0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.366+0000 7f338220d640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f337c15c4c0 0x7f337c09cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.366+0000 7f338220d640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f337c15c040 0x7f337c09c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.366+0000 7f338220d640 1 --2- 172.21.15.44:0/3849584829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33640039b0 0x7f337c14cbb0 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.366+0000 7f338220d640 1 -- 172.21.15.44:0/3849584829 >> 172.21.15.44:0/3849584829 conn(0x7f337c0cfcd0 msgr2=0x7f337c0cf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:49.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.366+0000 7f338220d640 1 -- 172.21.15.44:0/3849584829 shutdown_connections 2024-04-02T13:31:49.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.366+0000 7f338220d640 1 -- 172.21.15.44:0/3849584829 wait complete. 2024-04-02T13:31:49.381 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:31:09.752845+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:31:10.437454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:31:49.381 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:31:49.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.482+0000 7fca0854d640 1 Processor -- start 2024-04-02T13:31:49.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.482+0000 7fca0854d640 1 -- start start 2024-04-02T13:31:49.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.483+0000 7fca0854d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca0015e470 0x7fca000b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.483+0000 7fca0854d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca000ced00 con 0x7fca000ce790 2024-04-02T13:31:49.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.483+0000 7fca0854d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca000cefe0 con 0x7fca000b1540 2024-04-02T13:31:49.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.483+0000 7fca0854d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fca000cf2c0 con 0x7fca0015e470 2024-04-02T13:31:49.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.483+0000 7fca06aea640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fca000ce790 0x7fca0006d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56632/0 (socket says 172.21.15.44:56632) 2024-04-02T13:31:49.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.483+0000 7fca06aea640 1 -- 172.21.15.44:0/39910644 learned_addr learned my addr 172.21.15.44:0/39910644 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:49.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.483+0000 7fca062e9640 1 --2- 172.21.15.44:0/39910644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca0015e470 0x7fca000b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.483+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 562087336 0 0) 0x7fca000ced00 con 0x7fca000ce790 2024-04-02T13:31:49.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e8003520 con 0x7fca000ce790 2024-04-02T13:31:49.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 397429424 0 0) 0x7fca000cefe0 con 0x7fca000b1540 2024-04-02T13:31:49.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca000ced00 con 0x7fca000b1540 2024-04-02T13:31:49.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2947764376 0 0) 0x7fc9e8003520 con 0x7fca000ce790 2024-04-02T13:31:49.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca0015e470 msgr2=0x7fca000b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:49.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 --2- 172.21.15.44:0/39910644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca0015e470 0x7fca000b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 >> v1:172.21.15.69:6789/0 conn(0x7fca000b1540 legacy=0x7fca000d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca000cf5a0 con 0x7fca000ce790 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fc9f00019f0 con 0x7fca000ce790 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 >> v1:172.21.15.44:6789/0 conn(0x7fca000ce790 legacy=0x7fca0006d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 --2- 172.21.15.44:0/39910644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca0015e470 0x7fc9e8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 --2- 172.21.15.44:0/39910644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9e8004120 0x7fc9e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 --2- 172.21.15.44:0/39910644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e8004d40 0x7fc9e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fca000cf2c0 con 0x7fc9e8004d40 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e800bb80 con 0x7fc9e8004120 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e800bd80 con 0x7fca0015e470 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca06aea640 1 --2- 172.21.15.44:0/39910644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e8004d40 0x7fc9e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca062e9640 1 --2- 172.21.15.44:0/39910644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca0015e470 0x7fc9e8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca05ae8640 1 --2- 172.21.15.44:0/39910644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9e8004120 0x7fc9e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca06aea640 1 -- 172.21.15.44:0/39910644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca0015e470 msgr2=0x7fc9e8003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:49.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca06aea640 1 --2- 172.21.15.44:0/39910644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca0015e470 0x7fc9e8003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca06aea640 1 -- 172.21.15.44:0/39910644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9e8004120 msgr2=0x7fc9e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca06aea640 1 --2- 172.21.15.44:0/39910644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9e8004120 0x7fc9e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.484+0000 7fca06aea640 1 -- 172.21.15.44:0/39910644 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e8003520 con 0x7fc9e8004d40 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.485+0000 7fca06aea640 1 --2- 172.21.15.44:0/39910644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e8004d40 0x7fc9e80051c0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000a720 tx=0x7fc9e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8bfbbe8c49901d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.485+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f000a370 con 0x7fc9e8004d40 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.485+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9f0034650 con 0x7fc9e8004d40 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.485+0000 7fca052e7640 1 -- 172.21.15.44:0/39910644 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0034a80 con 0x7fc9e8004d40 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.486+0000 7fca0854d640 1 -- 172.21.15.44:0/39910644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e8004d40 msgr2=0x7fc9e80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.486+0000 7fca0854d640 1 --2- 172.21.15.44:0/39910644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9e8004d40 0x7fc9e80051c0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000a720 tx=0x7fc9e8005830 comp rx=0 tx=0).stop 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.486+0000 7fca06aea640 1 -- 172.21.15.44:0/39910644 reap_dead start 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.486+0000 7fca0854d640 1 -- 172.21.15.44:0/39910644 shutdown_connections 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.486+0000 7fca0854d640 1 -- 172.21.15.44:0/39910644 >> 172.21.15.44:0/39910644 conn(0x7fca0009aa20 msgr2=0x7fca0009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.486+0000 7fca0854d640 1 -- 172.21.15.44:0/39910644 shutdown_connections 2024-04-02T13:31:49.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.486+0000 7fca0854d640 1 -- 172.21.15.44:0/39910644 wait complete. 2024-04-02T13:31:49.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.487+0000 7fca0854d640 1 Processor -- start 2024-04-02T13:31:49.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.487+0000 7fca0854d640 1 -- start start 2024-04-02T13:31:49.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca0854d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9e8004d40 0x7fca0015d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca0854d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca0015fda0 0x7fca000b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca0854d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca000b6be0 0x7fca000b7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca0854d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fca000d22c0 con 0x7fca0015fda0 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca0854d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fca0006cb60 con 0x7fca000b6be0 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca0854d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fca000d2440 con 0x7fc9e8004d40 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca05ae8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca0015fda0 0x7fca000b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca05ae8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca0015fda0 0x7fca000b5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50658/0 (socket says 172.21.15.44:50658) 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca05ae8640 1 -- 172.21.15.44:0/2616028869 learned_addr learned my addr 172.21.15.44:0/2616028869 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca06aea640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca000b6be0 0x7fca000b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca062e9640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9e8004d40 0x7fca0015d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca05ae8640 1 -- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9e8004d40 msgr2=0x7fca0015d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca05ae8640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9e8004d40 0x7fca0015d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca05ae8640 1 -- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca000b6be0 msgr2=0x7fca000b7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca05ae8640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca000b6be0 0x7fca000b7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca05ae8640 1 -- 172.21.15.44:0/2616028869 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0015ed00 con 0x7fca0015fda0 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fca05ae8640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca0015fda0 0x7fca000b5980 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400f770 tx=0x7fc9f400fb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.488+0000 7fc9feffd640 1 -- 172.21.15.44:0/2616028869 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f401a070 con 0x7fca0015fda0 2024-04-02T13:31:49.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.489+0000 7fc9feffd640 1 -- 172.21.15.44:0/2616028869 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9f4005110 con 0x7fca0015fda0 2024-04-02T13:31:49.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.489+0000 7fca0854d640 1 -- 172.21.15.44:0/2616028869 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca0015efe0 con 0x7fca0015fda0 2024-04-02T13:31:49.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.489+0000 7fc9feffd640 1 -- 172.21.15.44:0/2616028869 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4007050 con 0x7fca0015fda0 2024-04-02T13:31:49.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.489+0000 7fca0854d640 1 -- 172.21.15.44:0/2616028869 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca0015f5a0 con 0x7fca0015fda0 2024-04-02T13:31:49.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.490+0000 7fca0854d640 1 -- 172.21.15.44:0/2616028869 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca000684f0 con 0x7fca0015fda0 2024-04-02T13:31:49.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.490+0000 7fc9feffd640 1 -- 172.21.15.44:0/2616028869 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc9f401d070 con 0x7fca0015fda0 2024-04-02T13:31:49.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.490+0000 7fc9feffd640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9ec06c920 0x7fc9ec06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.491+0000 7fc9feffd640 1 -- 172.21.15.44:0/2616028869 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7fc9f408b4a0 con 0x7fca0015fda0 2024-04-02T13:31:49.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.491+0000 7fca062e9640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9ec06c920 0x7fc9ec06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.491+0000 7fca062e9640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9ec06c920 0x7fc9ec06ede0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8009bc0 tx=0x7fc9f8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:49.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.495+0000 7fc9feffd640 1 -- 172.21.15.44:0/2616028869 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc9f40590a0 con 0x7fca0015fda0 2024-04-02T13:31:49.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.676+0000 7fca0854d640 1 -- 172.21.15.44:0/2616028869 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fca000b4d50 con 0x7fca0015fda0 2024-04-02T13:31:49.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.676+0000 7fc9feffd640 1 -- 172.21.15.44:0/2616028869 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) ==== 93+0+10 (secure 0 0 0) 0x7fc9f405d3a0 con 0x7fca0015fda0 2024-04-02T13:31:49.680 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:31:49.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.678+0000 7fca0854d640 1 -- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9ec06c920 msgr2=0x7fc9ec06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.678+0000 7fca0854d640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9ec06c920 0x7fc9ec06ede0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8009bc0 tx=0x7fc9f8009190 comp rx=0 tx=0).stop 2024-04-02T13:31:49.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.678+0000 7fca0854d640 1 -- 172.21.15.44:0/2616028869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca0015fda0 msgr2=0x7fca000b5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.678+0000 7fca0854d640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca0015fda0 0x7fca000b5980 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400f770 tx=0x7fc9f400fb70 comp rx=0 tx=0).stop 2024-04-02T13:31:49.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.678+0000 7fca0854d640 1 -- 172.21.15.44:0/2616028869 shutdown_connections 2024-04-02T13:31:49.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.678+0000 7fca0854d640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc9ec06c920 0x7fc9ec06ede0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.678+0000 7fca0854d640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca000b6be0 0x7fca000b7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.678+0000 7fca0854d640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca0015fda0 0x7fca000b5980 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.678+0000 7fca0854d640 1 --2- 172.21.15.44:0/2616028869 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9e8004d40 0x7fca0015d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.678+0000 7fca0854d640 1 -- 172.21.15.44:0/2616028869 >> 172.21.15.44:0/2616028869 conn(0x7fca0009aa20 msgr2=0x7fca000aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:49.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.678+0000 7fca0854d640 1 -- 172.21.15.44:0/2616028869 shutdown_connections 2024-04-02T13:31:49.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.678+0000 7fca0854d640 1 -- 172.21.15.44:0/2616028869 wait complete. 2024-04-02T13:31:49.693 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:31:49.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.793+0000 7f32f0bc8640 1 Processor -- start 2024-04-02T13:31:49.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.793+0000 7f32f0bc8640 1 -- start start 2024-04-02T13:31:49.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.794+0000 7f32f0bc8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f32ec0d0a30 0x7f32ec0b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.794+0000 7f32f0bc8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f32ec0b3990 0x7f32ec143a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.794+0000 7f32f0bc8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32ec0d0e10 con 0x7f32ec143f60 2024-04-02T13:31:49.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.794+0000 7f32f0bc8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f32ec161e10 con 0x7f32ec0b3990 2024-04-02T13:31:49.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.794+0000 7f32f0bc8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f32ec144750 con 0x7f32ec0d0a30 2024-04-02T13:31:49.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.794+0000 7f32e9d9b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f32ec0b3990 0x7f32ec143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.794+0000 7f32ead9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f32ec143f60 0x7f32ec071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56636/0 (socket says 172.21.15.44:56636) 2024-04-02T13:31:49.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.794+0000 7f32ead9d640 1 -- 172.21.15.44:0/838908002 learned_addr learned my addr 172.21.15.44:0/838908002 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:49.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.794+0000 7f32ea59c640 1 --2- 172.21.15.44:0/838908002 >> v2:172.21.15.69:3300/0 conn(0x7f32ec0d0a30 0x7f32ec0b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.794+0000 7f32e9d9b640 1 -- 172.21.15.44:0/838908002 >> v2:172.21.15.69:3300/0 conn(0x7f32ec0d0a30 msgr2=0x7f32ec0b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:49.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.794+0000 7f32e9d9b640 1 --2- 172.21.15.44:0/838908002 >> v2:172.21.15.69:3300/0 conn(0x7f32ec0d0a30 0x7f32ec0b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.794+0000 7f32e9d9b640 1 -- 172.21.15.44:0/838908002 >> v1:172.21.15.44:6789/0 conn(0x7f32ec143f60 legacy=0x7f32ec071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.794+0000 7f32e9d9b640 1 -- 172.21.15.44:0/838908002 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ec162060 con 0x7f32ec0b3990 2024-04-02T13:31:49.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.795+0000 7f32e9d9b640 1 --2- 172.21.15.44:0/838908002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0b3990 0x7f32ec143a20 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f32ec072b30 tx=0x7f32d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9421f3e631a1d79f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:49.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.795+0000 7f32e959a640 1 -- 172.21.15.44:0/838908002 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d40315a0 con 0x7f32ec0b3990 2024-04-02T13:31:49.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.795+0000 7f32e959a640 1 -- 172.21.15.44:0/838908002 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d4037070 con 0x7f32ec0b3990 2024-04-02T13:31:49.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.795+0000 7f32e959a640 1 -- 172.21.15.44:0/838908002 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d4031960 con 0x7f32ec0b3990 2024-04-02T13:31:49.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.796+0000 7f32f0bc8640 1 -- 172.21.15.44:0/838908002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0b3990 msgr2=0x7f32ec143a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.796+0000 7f32f0bc8640 1 --2- 172.21.15.44:0/838908002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0b3990 0x7f32ec143a20 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f32ec072b30 tx=0x7f32d402b840 comp rx=0 tx=0).stop 2024-04-02T13:31:49.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.796+0000 7f32f0bc8640 1 -- 172.21.15.44:0/838908002 shutdown_connections 2024-04-02T13:31:49.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.796+0000 7f32f0bc8640 1 --2- 172.21.15.44:0/838908002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0b3990 0x7f32ec143a20 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.796+0000 7f32f0bc8640 1 --2- 172.21.15.44:0/838908002 >> v2:172.21.15.69:3300/0 conn(0x7f32ec0d0a30 0x7f32ec0b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.796+0000 7f32f0bc8640 1 -- 172.21.15.44:0/838908002 >> 172.21.15.44:0/838908002 conn(0x7f32ec0b5d90 msgr2=0x7f32ec06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:49.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.796+0000 7f32f0bc8640 1 -- 172.21.15.44:0/838908002 shutdown_connections 2024-04-02T13:31:49.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.796+0000 7f32f0bc8640 1 -- 172.21.15.44:0/838908002 wait complete. 2024-04-02T13:31:49.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.797+0000 7f32f0bc8640 1 Processor -- start 2024-04-02T13:31:49.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.797+0000 7f32f0bc8640 1 -- start start 2024-04-02T13:31:49.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32f0bc8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32ec0b3990 0x7f32ec1e88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32f0bc8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ec0d0a30 0x7f32ec142200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32f0bc8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec143f60 0x7f32ec1337f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32f0bc8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f32ec144390 con 0x7f32ec143f60 2024-04-02T13:31:49.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32f0bc8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f32ec162620 con 0x7f32ec0b3990 2024-04-02T13:31:49.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32f0bc8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f32ec144510 con 0x7f32ec0d0a30 2024-04-02T13:31:49.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32ead9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec143f60 0x7f32ec1337f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32ead9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec143f60 0x7f32ec1337f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50672/0 (socket says 172.21.15.44:50672) 2024-04-02T13:31:49.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32ead9d640 1 -- 172.21.15.44:0/3504673858 learned_addr learned my addr 172.21.15.44:0/3504673858 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:49.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32ea59c640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32ec0b3990 0x7f32ec1e88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32e9d9b640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ec0d0a30 0x7f32ec142200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32ead9d640 1 -- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ec0d0a30 msgr2=0x7f32ec142200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32ead9d640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ec0d0a30 0x7f32ec142200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32ead9d640 1 -- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32ec0b3990 msgr2=0x7f32ec1e88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32ead9d640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32ec0b3990 0x7f32ec1e88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.798+0000 7f32ead9d640 1 -- 172.21.15.44:0/3504673858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ec142b70 con 0x7f32ec143f60 2024-04-02T13:31:49.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.799+0000 7f32ead9d640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec143f60 0x7f32ec1337f0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00ca80 tx=0x7f32dc00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:49.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.799+0000 7f32e2ffd640 1 -- 172.21.15.44:0/3504673858 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32dc017070 con 0x7f32ec143f60 2024-04-02T13:31:49.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.799+0000 7f32e2ffd640 1 -- 172.21.15.44:0/3504673858 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32dc00ac30 con 0x7f32ec143f60 2024-04-02T13:31:49.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.799+0000 7f32f0bc8640 1 -- 172.21.15.44:0/3504673858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32ec142e50 con 0x7f32ec143f60 2024-04-02T13:31:49.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.799+0000 7f32e2ffd640 1 -- 172.21.15.44:0/3504673858 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32dc012600 con 0x7f32ec143f60 2024-04-02T13:31:49.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.799+0000 7f32f0bc8640 1 -- 172.21.15.44:0/3504673858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32ec143410 con 0x7f32ec143f60 2024-04-02T13:31:49.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.800+0000 7f32f0bc8640 1 -- 172.21.15.44:0/3504673858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32ec0684f0 con 0x7f32ec143f60 2024-04-02T13:31:49.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.801+0000 7f32e2ffd640 1 -- 172.21.15.44:0/3504673858 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f32dc01a070 con 0x7f32ec143f60 2024-04-02T13:31:49.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.801+0000 7f32e2ffd640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f32cc06c8f0 0x7f32cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:49.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.801+0000 7f32e2ffd640 1 -- 172.21.15.44:0/3504673858 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f32dc08b9e0 con 0x7f32ec143f60 2024-04-02T13:31:49.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.801+0000 7f32ea59c640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f32cc06c8f0 0x7f32cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:49.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.802+0000 7f32ea59c640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f32cc06c8f0 0x7f32cc06edb0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f32e4006fd0 tx=0x7f32e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:49.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.805+0000 7f32e2ffd640 1 -- 172.21.15.44:0/3504673858 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f32ec0d0e10 con 0x7f32ec143f60 2024-04-02T13:31:49.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.992+0000 7f32f0bc8640 1 -- 172.21.15.44:0/3504673858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f32ec0d0300 con 0x7f32ec143f60 2024-04-02T13:31:49.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.993+0000 7f32e2ffd640 1 -- 172.21.15.44:0/3504673858 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v122) ==== 104+0+11 (secure 0 0 0) 0x7f32dc00ae20 con 0x7f32ec143f60 2024-04-02T13:31:49.997 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:31:49.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.995+0000 7f32f0bc8640 1 -- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f32cc06c8f0 msgr2=0x7f32cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.995+0000 7f32f0bc8640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f32cc06c8f0 0x7f32cc06edb0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f32e4006fd0 tx=0x7f32e4008040 comp rx=0 tx=0).stop 2024-04-02T13:31:49.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.995+0000 7f32f0bc8640 1 -- 172.21.15.44:0/3504673858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec143f60 msgr2=0x7f32ec1337f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:49.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.995+0000 7f32f0bc8640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec143f60 0x7f32ec1337f0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00ca80 tx=0x7f32dc00cf40 comp rx=0 tx=0).stop 2024-04-02T13:31:49.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.995+0000 7f32f0bc8640 1 -- 172.21.15.44:0/3504673858 shutdown_connections 2024-04-02T13:31:49.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.995+0000 7f32f0bc8640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f32cc06c8f0 0x7f32cc06edb0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:49.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.995+0000 7f32f0bc8640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec143f60 0x7f32ec1337f0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.995+0000 7f32f0bc8640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ec0d0a30 0x7f32ec142200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.995+0000 7f32f0bc8640 1 --2- 172.21.15.44:0/3504673858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32ec0b3990 0x7f32ec1e88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.995+0000 7f32f0bc8640 1 -- 172.21.15.44:0/3504673858 >> 172.21.15.44:0/3504673858 conn(0x7f32ec0b5d90 msgr2=0x7f32ec06ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:50.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.995+0000 7f32f0bc8640 1 -- 172.21.15.44:0/3504673858 shutdown_connections 2024-04-02T13:31:50.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:49.995+0000 7f32f0bc8640 1 -- 172.21.15.44:0/3504673858 wait complete. 2024-04-02T13:31:50.009 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:31:50.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.109+0000 7f84b808f640 1 Processor -- start 2024-04-02T13:31:50.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.110+0000 7f84b808f640 1 -- start start 2024-04-02T13:31:50.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.110+0000 7f84b808f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f84b00721f0 0x7f84b00cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.110+0000 7f84b808f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84b00cfc40 0x7f84b00cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.110+0000 7f84b808f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84b006c220 con 0x7f84b015e420 2024-04-02T13:31:50.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.110+0000 7f84b808f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f84b00d00b0 con 0x7f84b00721f0 2024-04-02T13:31:50.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.110+0000 7f84b808f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f84b00cf3b0 con 0x7f84b00cfc40 2024-04-02T13:31:50.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.111+0000 7f84b562a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f84b00721f0 0x7f84b00cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:50.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.111+0000 7f84b562a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f84b00721f0 0x7f84b00cf700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50688/0 (socket says 172.21.15.44:50688) 2024-04-02T13:31:50.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.111+0000 7f84b562a640 1 -- 172.21.15.44:0/365544527 learned_addr learned my addr 172.21.15.44:0/365544527 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:50.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.111+0000 7f84b662c640 1 --2- 172.21.15.44:0/365544527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84b00cfc40 0x7f84b00cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:50.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.111+0000 7f84b562a640 1 -- 172.21.15.44:0/365544527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84b00cfc40 msgr2=0x7f84b00cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:50.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.111+0000 7f84b562a640 1 --2- 172.21.15.44:0/365544527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84b00cfc40 0x7f84b00cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.111+0000 7f84b562a640 1 -- 172.21.15.44:0/365544527 >> v1:172.21.15.69:6789/0 conn(0x7f84b015e420 legacy=0x7f84b00ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.111+0000 7f84b562a640 1 -- 172.21.15.44:0/365544527 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84b015f7e0 con 0x7f84b00721f0 2024-04-02T13:31:50.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.112+0000 7f84b562a640 1 --2- 172.21.15.44:0/365544527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84b00721f0 0x7f84b00cf700 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f84a0009870 tx=0x7f84a002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b72d6b7933ea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:50.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.112+0000 7f84b4e29640 1 -- 172.21.15.44:0/365544527 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a00317c0 con 0x7f84b00721f0 2024-04-02T13:31:50.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.112+0000 7f84b4e29640 1 -- 172.21.15.44:0/365544527 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84a0037070 con 0x7f84b00721f0 2024-04-02T13:31:50.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.112+0000 7f84b4e29640 1 -- 172.21.15.44:0/365544527 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a0031b80 con 0x7f84b00721f0 2024-04-02T13:31:50.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.112+0000 7f84b808f640 1 -- 172.21.15.44:0/365544527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84b00721f0 msgr2=0x7f84b00cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.112+0000 7f84b808f640 1 --2- 172.21.15.44:0/365544527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84b00721f0 0x7f84b00cf700 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f84a0009870 tx=0x7f84a002b860 comp rx=0 tx=0).stop 2024-04-02T13:31:50.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.113+0000 7f84b808f640 1 -- 172.21.15.44:0/365544527 shutdown_connections 2024-04-02T13:31:50.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.113+0000 7f84b808f640 1 --2- 172.21.15.44:0/365544527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84b00721f0 0x7f84b00cf700 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.113+0000 7f84b808f640 1 --2- 172.21.15.44:0/365544527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84b00cfc40 0x7f84b00cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.113+0000 7f84b808f640 1 -- 172.21.15.44:0/365544527 >> 172.21.15.44:0/365544527 conn(0x7f84b00b2a10 msgr2=0x7f84b00b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:50.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.113+0000 7f84b808f640 1 -- 172.21.15.44:0/365544527 shutdown_connections 2024-04-02T13:31:50.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.113+0000 7f84b808f640 1 -- 172.21.15.44:0/365544527 wait complete. 2024-04-02T13:31:50.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.114+0000 7f84b808f640 1 Processor -- start 2024-04-02T13:31:50.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.114+0000 7f84b808f640 1 -- start start 2024-04-02T13:31:50.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b808f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84b00721f0 0x7f84b014ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b808f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84b00cfc40 0x7f84b014eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b808f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84b015e420 0x7f84b014f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b808f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f84b015f220 con 0x7f84b015e420 2024-04-02T13:31:50.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b808f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84b015ef40 con 0x7f84b00721f0 2024-04-02T13:31:50.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b808f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f84b015f500 con 0x7f84b00cfc40 2024-04-02T13:31:50.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b662c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84b015e420 0x7f84b014f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:50.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b662c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84b015e420 0x7f84b014f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50692/0 (socket says 172.21.15.44:50692) 2024-04-02T13:31:50.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b662c640 1 -- 172.21.15.44:0/740582724 learned_addr learned my addr 172.21.15.44:0/740582724 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:50.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b662c640 1 -- 172.21.15.44:0/740582724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84b00cfc40 msgr2=0x7f84b014eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b662c640 1 --2- 172.21.15.44:0/740582724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84b00cfc40 0x7f84b014eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b662c640 1 -- 172.21.15.44:0/740582724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84b00721f0 msgr2=0x7f84b014ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b662c640 1 --2- 172.21.15.44:0/740582724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84b00721f0 0x7f84b014ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b662c640 1 -- 172.21.15.44:0/740582724 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84b00ae970 con 0x7f84b015e420 2024-04-02T13:31:50.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b662c640 1 --2- 172.21.15.44:0/740582724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84b015e420 0x7f84b014f080 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f84a800efc0 tx=0x7f84a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:50.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84a67fc640 1 -- 172.21.15.44:0/740582724 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a8018020 con 0x7f84b015e420 2024-04-02T13:31:50.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84a67fc640 1 -- 172.21.15.44:0/740582724 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84a8014480 con 0x7f84b015e420 2024-04-02T13:31:50.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b808f640 1 -- 172.21.15.44:0/740582724 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84b00aec50 con 0x7f84b015e420 2024-04-02T13:31:50.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84a67fc640 1 -- 172.21.15.44:0/740582724 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a8014870 con 0x7f84b015e420 2024-04-02T13:31:50.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.115+0000 7f84b808f640 1 -- 172.21.15.44:0/740582724 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84b00af210 con 0x7f84b015e420 2024-04-02T13:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.116+0000 7f84b808f640 1 -- 172.21.15.44:0/740582724 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84b00684f0 con 0x7f84b015e420 2024-04-02T13:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.117+0000 7f84a67fc640 1 -- 172.21.15.44:0/740582724 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f84a8005d10 con 0x7f84b015e420 2024-04-02T13:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.117+0000 7f84a67fc640 1 --2- 172.21.15.44:0/740582724 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f849406c8f0 0x7f849406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.117+0000 7f84a67fc640 1 -- 172.21.15.44:0/740582724 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f84a808cbf0 con 0x7f84b015e420 2024-04-02T13:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.117+0000 7f84b5e2b640 1 --2- 172.21.15.44:0/740582724 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f849406c8f0 0x7f849406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.118+0000 7f84b5e2b640 1 --2- 172.21.15.44:0/740582724 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f849406c8f0 0x7f849406edb0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f84ac000f60 tx=0x7f84ac0079d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.122+0000 7f84a67fc640 1 -- 172.21.15.44:0/740582724 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f84a8022030 con 0x7f84b015e420 2024-04-02T13:31:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.305+0000 7f84b808f640 1 -- 172.21.15.44:0/740582724 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f84b00cdfd0 con 0x7f84b015e420 2024-04-02T13:31:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.306+0000 7f84a67fc640 1 -- 172.21.15.44:0/740582724 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v122) ==== 100+0+11 (secure 0 0 0) 0x7f84a8010040 con 0x7f84b015e420 2024-04-02T13:31:50.310 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:31:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.307+0000 7f84b808f640 1 -- 172.21.15.44:0/740582724 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f849406c8f0 msgr2=0x7f849406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.307+0000 7f84b808f640 1 --2- 172.21.15.44:0/740582724 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f849406c8f0 0x7f849406edb0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f84ac000f60 tx=0x7f84ac0079d0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.307+0000 7f84b808f640 1 -- 172.21.15.44:0/740582724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84b015e420 msgr2=0x7f84b014f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.307+0000 7f84b808f640 1 --2- 172.21.15.44:0/740582724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84b015e420 0x7f84b014f080 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f84a800efc0 tx=0x7f84a800c490 comp rx=0 tx=0).stop 2024-04-02T13:31:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.307+0000 7f84b808f640 1 -- 172.21.15.44:0/740582724 shutdown_connections 2024-04-02T13:31:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.307+0000 7f84b808f640 1 --2- 172.21.15.44:0/740582724 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f849406c8f0 0x7f849406edb0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.307+0000 7f84b808f640 1 --2- 172.21.15.44:0/740582724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84b015e420 0x7f84b014f080 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.307+0000 7f84b808f640 1 --2- 172.21.15.44:0/740582724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84b00cfc40 0x7f84b014eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.307+0000 7f84b808f640 1 --2- 172.21.15.44:0/740582724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84b00721f0 0x7f84b014ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.307+0000 7f84b808f640 1 -- 172.21.15.44:0/740582724 >> 172.21.15.44:0/740582724 conn(0x7f84b00b2a10 msgr2=0x7f84b015aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.307+0000 7f84b808f640 1 -- 172.21.15.44:0/740582724 shutdown_connections 2024-04-02T13:31:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.307+0000 7f84b808f640 1 -- 172.21.15.44:0/740582724 wait complete. 2024-04-02T13:31:50.322 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:31:50.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.420+0000 7ff79042a640 1 Processor -- start 2024-04-02T13:31:50.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.420+0000 7ff79042a640 1 -- start start 2024-04-02T13:31:50.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.421+0000 7ff79042a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff78806d550 0x7ff7880d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.421+0000 7ff79042a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7880cc600 con 0x7ff78815e4a0 2024-04-02T13:31:50.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.421+0000 7ff79042a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7880cc8e0 con 0x7ff7880d0c50 2024-04-02T13:31:50.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.421+0000 7ff79042a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff7880af300 con 0x7ff78806d550 2024-04-02T13:31:50.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.421+0000 7ff78e9c7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff78806d550 0x7ff7880d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:50.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.421+0000 7ff78e1c6640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff78815e4a0 0x7ff7880aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56646/0 (socket says 172.21.15.44:56646) 2024-04-02T13:31:50.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.421+0000 7ff78e9c7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff78806d550 0x7ff7880d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50706/0 (socket says 172.21.15.44:50706) 2024-04-02T13:31:50.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.421+0000 7ff78e1c6640 1 -- 172.21.15.44:0/3555961206 learned_addr learned my addr 172.21.15.44:0/3555961206 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:50.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.421+0000 7ff78e9c7640 1 -- 172.21.15.44:0/3555961206 >> v1:172.21.15.69:6789/0 conn(0x7ff7880d0c50 legacy=0x7ff78806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:50.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.421+0000 7ff78e9c7640 1 -- 172.21.15.44:0/3555961206 >> v1:172.21.15.44:6789/0 conn(0x7ff78815e4a0 legacy=0x7ff7880aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.421+0000 7ff78e9c7640 1 -- 172.21.15.44:0/3555961206 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7880ccbc0 con 0x7ff78806d550 2024-04-02T13:31:50.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.422+0000 7ff78e9c7640 1 --2- 172.21.15.44:0/3555961206 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff78806d550 0x7ff7880d10b0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7ff778009dc0 tx=0x7ff77802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d4d9c268f7ea3d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:50.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.422+0000 7ff78d1c4640 1 -- 172.21.15.44:0/3555961206 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff778039070 con 0x7ff78806d550 2024-04-02T13:31:50.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.422+0000 7ff78d1c4640 1 -- 172.21.15.44:0/3555961206 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff77802cc40 con 0x7ff78806d550 2024-04-02T13:31:50.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.422+0000 7ff78d1c4640 1 -- 172.21.15.44:0/3555961206 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff778034440 con 0x7ff78806d550 2024-04-02T13:31:50.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.423+0000 7ff79042a640 1 -- 172.21.15.44:0/3555961206 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff78806d550 msgr2=0x7ff7880d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.423+0000 7ff79042a640 1 --2- 172.21.15.44:0/3555961206 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff78806d550 0x7ff7880d10b0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7ff778009dc0 tx=0x7ff77802be90 comp rx=0 tx=0).stop 2024-04-02T13:31:50.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.423+0000 7ff79042a640 1 -- 172.21.15.44:0/3555961206 shutdown_connections 2024-04-02T13:31:50.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.423+0000 7ff79042a640 1 --2- 172.21.15.44:0/3555961206 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff78806d550 0x7ff7880d10b0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.423+0000 7ff79042a640 1 -- 172.21.15.44:0/3555961206 >> 172.21.15.44:0/3555961206 conn(0x7ff78809aa20 msgr2=0x7ff78809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:50.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.423+0000 7ff79042a640 1 -- 172.21.15.44:0/3555961206 shutdown_connections 2024-04-02T13:31:50.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.423+0000 7ff79042a640 1 -- 172.21.15.44:0/3555961206 wait complete. 2024-04-02T13:31:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.424+0000 7ff79042a640 1 Processor -- start 2024-04-02T13:31:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.424+0000 7ff79042a640 1 -- start start 2024-04-02T13:31:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff79042a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff78806d550 0x7ff78815ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff79042a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7880d0c50 0x7ff78814b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff79042a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff78815e4a0 0x7ff78815cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff79042a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff7880cd460 con 0x7ff78806d550 2024-04-02T13:31:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff79042a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff7880cd180 con 0x7ff78815e4a0 2024-04-02T13:31:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff79042a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff78815e880 con 0x7ff7880d0c50 2024-04-02T13:31:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff78e1c6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff78806d550 0x7ff78815ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff78e1c6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff78806d550 0x7ff78815ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50720/0 (socket says 172.21.15.44:50720) 2024-04-02T13:31:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff78e1c6640 1 -- 172.21.15.44:0/3614180129 learned_addr learned my addr 172.21.15.44:0/3614180129 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff78e1c6640 1 -- 172.21.15.44:0/3614180129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7880d0c50 msgr2=0x7ff78814b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff78e1c6640 1 --2- 172.21.15.44:0/3614180129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7880d0c50 0x7ff78814b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff78e1c6640 1 -- 172.21.15.44:0/3614180129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff78815e4a0 msgr2=0x7ff78815cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff78e1c6640 1 --2- 172.21.15.44:0/3614180129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff78815e4a0 0x7ff78815cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff78e1c6640 1 -- 172.21.15.44:0/3614180129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78814bbd0 con 0x7ff78806d550 2024-04-02T13:31:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.425+0000 7ff78e1c6640 1 --2- 172.21.15.44:0/3614180129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff78806d550 0x7ff78815ff80 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7ff78000c430 tx=0x7ff78000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.426+0000 7ff786ffd640 1 -- 172.21.15.44:0/3614180129 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff780017070 con 0x7ff78806d550 2024-04-02T13:31:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.426+0000 7ff786ffd640 1 -- 172.21.15.44:0/3614180129 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff780005ce0 con 0x7ff78806d550 2024-04-02T13:31:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.426+0000 7ff79042a640 1 -- 172.21.15.44:0/3614180129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78815eee0 con 0x7ff78806d550 2024-04-02T13:31:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.426+0000 7ff786ffd640 1 -- 172.21.15.44:0/3614180129 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7800072f0 con 0x7ff78806d550 2024-04-02T13:31:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.426+0000 7ff79042a640 1 -- 172.21.15.44:0/3614180129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff78815f4a0 con 0x7ff78806d550 2024-04-02T13:31:50.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.426+0000 7ff79042a640 1 -- 172.21.15.44:0/3614180129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7880684f0 con 0x7ff78806d550 2024-04-02T13:31:50.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.427+0000 7ff786ffd640 1 -- 172.21.15.44:0/3614180129 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff78001a070 con 0x7ff78806d550 2024-04-02T13:31:50.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.427+0000 7ff786ffd640 1 --2- 172.21.15.44:0/3614180129 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff77406c920 0x7ff77406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.428+0000 7ff786ffd640 1 -- 172.21.15.44:0/3614180129 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7ff78008b150 con 0x7ff78806d550 2024-04-02T13:31:50.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.428+0000 7ff78d9c5640 1 --2- 172.21.15.44:0/3614180129 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff77406c920 0x7ff77406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:50.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.428+0000 7ff78d9c5640 1 --2- 172.21.15.44:0/3614180129 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff77406c920 0x7ff77406ede0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ff77c006fd0 tx=0x7ff77c006bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:50.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.432+0000 7ff786ffd640 1 -- 172.21.15.44:0/3614180129 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff7800593a0 con 0x7ff78806d550 2024-04-02T13:31:50.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.631+0000 7ff79042a640 1 -- 172.21.15.44:0/3614180129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff7880cc1b0 con 0x7ff78806d550 2024-04-02T13:31:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.632+0000 7ff786ffd640 1 -- 172.21.15.44:0/3614180129 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v122) ==== 80+0+122 (secure 0 0 0) 0x7ff780058ca0 con 0x7ff78806d550 2024-04-02T13:31:50.636 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6835/2109578276 2024-04-03T13:31:41.102885+0000 2024-04-02T13:31:50.636 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6833/2109578276 2024-04-03T13:31:41.102885+0000 2024-04-02T13:31:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:listed 2 entries 2024-04-02T13:31:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.634+0000 7ff79042a640 1 -- 172.21.15.44:0/3614180129 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff77406c920 msgr2=0x7ff77406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.634+0000 7ff79042a640 1 --2- 172.21.15.44:0/3614180129 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff77406c920 0x7ff77406ede0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ff77c006fd0 tx=0x7ff77c006bc0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.634+0000 7ff79042a640 1 -- 172.21.15.44:0/3614180129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff78806d550 msgr2=0x7ff78815ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.634+0000 7ff79042a640 1 --2- 172.21.15.44:0/3614180129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff78806d550 0x7ff78815ff80 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7ff78000c430 tx=0x7ff78000c8f0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.634+0000 7ff79042a640 1 -- 172.21.15.44:0/3614180129 shutdown_connections 2024-04-02T13:31:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.634+0000 7ff79042a640 1 --2- 172.21.15.44:0/3614180129 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff77406c920 0x7ff77406ede0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.634+0000 7ff79042a640 1 --2- 172.21.15.44:0/3614180129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff78815e4a0 0x7ff78815cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.634+0000 7ff79042a640 1 --2- 172.21.15.44:0/3614180129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7880d0c50 0x7ff78814b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.634+0000 7ff79042a640 1 --2- 172.21.15.44:0/3614180129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff78806d550 0x7ff78815ff80 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.634+0000 7ff79042a640 1 -- 172.21.15.44:0/3614180129 >> 172.21.15.44:0/3614180129 conn(0x7ff78809aa20 msgr2=0x7ff78806ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:50.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.634+0000 7ff79042a640 1 -- 172.21.15.44:0/3614180129 shutdown_connections 2024-04-02T13:31:50.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.634+0000 7ff79042a640 1 -- 172.21.15.44:0/3614180129 wait complete. 2024-04-02T13:31:50.648 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:31:50.648 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:31:50.664 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:31:50.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.801+0000 7f6437da8640 1 Processor -- start 2024-04-02T13:31:50.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.801+0000 7f6437da8640 1 -- start start 2024-04-02T13:31:50.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6437da8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f643015c2b0 0x7f64300ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6437da8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f64300cc9b0 0x7f64300cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6437da8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64300cafc0 con 0x7f64300cbd80 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6437da8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f6430070120 con 0x7f64300cc9b0 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6437da8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f64300cb2a0 con 0x7f643015c2b0 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6436345640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f64300cc9b0 0x7f64300cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6435343640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f64300cbd80 0x7f6430071970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56652/0 (socket says 172.21.15.44:56652) 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6435343640 1 -- 172.21.15.44:0/282676505 learned_addr learned my addr 172.21.15.44:0/282676505 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6436345640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f64300cc9b0 0x7f64300cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50722/0 (socket says 172.21.15.44:50722) 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6436345640 1 -- 172.21.15.44:0/282676505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f643015c2b0 msgr2=0x7f64300ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6435b44640 1 --2- 172.21.15.44:0/282676505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f643015c2b0 0x7f64300ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6434b42640 1 -- 172.21.15.44:0/282676505 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2325234322 0 0) 0x7f64300cafc0 con 0x7f64300cbd80 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6436345640 1 --2- 172.21.15.44:0/282676505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f643015c2b0 0x7f64300ccf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6436345640 1 -- 172.21.15.44:0/282676505 >> v1:172.21.15.44:6789/0 conn(0x7f64300cbd80 legacy=0x7f6430071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6436345640 1 -- 172.21.15.44:0/282676505 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64300cb580 con 0x7f64300cc9b0 2024-04-02T13:31:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.802+0000 7f6435b44640 1 --2- 172.21.15.44:0/282676505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f643015c2b0 0x7f64300ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:50.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.803+0000 7f6436345640 1 --2- 172.21.15.44:0/282676505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64300cc9b0 0x7f64300cd440 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f6428009d90 tx=0x7f642802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad93376081e722a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:50.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.803+0000 7f6434b42640 1 -- 172.21.15.44:0/282676505 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428039070 con 0x7f64300cc9b0 2024-04-02T13:31:50.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.803+0000 7f6434b42640 1 -- 172.21.15.44:0/282676505 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f642802cbc0 con 0x7f64300cc9b0 2024-04-02T13:31:50.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.803+0000 7f6434b42640 1 -- 172.21.15.44:0/282676505 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428034440 con 0x7f64300cc9b0 2024-04-02T13:31:50.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.804+0000 7f6437da8640 1 -- 172.21.15.44:0/282676505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64300cc9b0 msgr2=0x7f64300cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.804+0000 7f6437da8640 1 --2- 172.21.15.44:0/282676505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64300cc9b0 0x7f64300cd440 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f6428009d90 tx=0x7f642802be10 comp rx=0 tx=0).stop 2024-04-02T13:31:50.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.804+0000 7f6437da8640 1 -- 172.21.15.44:0/282676505 shutdown_connections 2024-04-02T13:31:50.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.804+0000 7f6437da8640 1 --2- 172.21.15.44:0/282676505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64300cc9b0 0x7f64300cd440 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.804+0000 7f6437da8640 1 --2- 172.21.15.44:0/282676505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f643015c2b0 0x7f64300ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.804+0000 7f6437da8640 1 -- 172.21.15.44:0/282676505 >> 172.21.15.44:0/282676505 conn(0x7f64300b20e0 msgr2=0x7f64300b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:50.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.804+0000 7f6437da8640 1 -- 172.21.15.44:0/282676505 shutdown_connections 2024-04-02T13:31:50.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.804+0000 7f6437da8640 1 -- 172.21.15.44:0/282676505 wait complete. 2024-04-02T13:31:50.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.805+0000 7f6437da8640 1 Processor -- start 2024-04-02T13:31:50.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.805+0000 7f6437da8640 1 -- start start 2024-04-02T13:31:50.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6437da8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64300cbd80 0x7f6430155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6437da8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64300cc9b0 0x7f64300b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6437da8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f643015c2b0 0x7f64300b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6437da8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f643006fcd0 con 0x7f64300cc9b0 2024-04-02T13:31:50.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6437da8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f643015c690 con 0x7f64300cbd80 2024-04-02T13:31:50.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6437da8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f643006fe50 con 0x7f643015c2b0 2024-04-02T13:31:50.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6435343640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64300cc9b0 0x7f64300b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:50.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6435343640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64300cc9b0 0x7f64300b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50736/0 (socket says 172.21.15.44:50736) 2024-04-02T13:31:50.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6435343640 1 -- 172.21.15.44:0/1555214434 learned_addr learned my addr 172.21.15.44:0/1555214434 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6436345640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f643015c2b0 0x7f64300b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6435b44640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64300cbd80 0x7f6430155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6435343640 1 -- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f643015c2b0 msgr2=0x7f64300b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6435343640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f643015c2b0 0x7f64300b3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6435343640 1 -- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64300cbd80 msgr2=0x7f6430155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6435343640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64300cbd80 0x7f6430155b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6435343640 1 -- 172.21.15.44:0/1555214434 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64300cf4e0 con 0x7f64300cc9b0 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6435b44640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64300cbd80 0x7f6430155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.806+0000 7f6435343640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64300cc9b0 0x7f64300b31d0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f642000c430 tx=0x7f642000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.807+0000 7f64267fc640 1 -- 172.21.15.44:0/1555214434 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64200074e0 con 0x7f64300cc9b0 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.807+0000 7f64267fc640 1 -- 172.21.15.44:0/1555214434 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64200047e0 con 0x7f64300cc9b0 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.807+0000 7f6437da8640 1 -- 172.21.15.44:0/1555214434 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64300ae800 con 0x7f64300cc9b0 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.807+0000 7f64267fc640 1 -- 172.21.15.44:0/1555214434 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6420004bd0 con 0x7f64300cc9b0 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.807+0000 7f6437da8640 1 -- 172.21.15.44:0/1555214434 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64300add20 con 0x7f64300cc9b0 2024-04-02T13:31:50.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.808+0000 7f6437da8640 1 -- 172.21.15.44:0/1555214434 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64300684f0 con 0x7f64300cc9b0 2024-04-02T13:31:50.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.808+0000 7f64267fc640 1 -- 172.21.15.44:0/1555214434 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6420011070 con 0x7f64300cc9b0 2024-04-02T13:31:50.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.809+0000 7f64267fc640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f641406c8f0 0x7f641406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:50.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.809+0000 7f64267fc640 1 -- 172.21.15.44:0/1555214434 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f642008ba10 con 0x7f64300cc9b0 2024-04-02T13:31:50.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.809+0000 7f6435b44640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f641406c8f0 0x7f641406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:50.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.809+0000 7f6435b44640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f641406c8f0 0x7f641406edb0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f642c006fd0 tx=0x7f642c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:50.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:50.813+0000 7f64267fc640 1 -- 172.21.15.44:0/1555214434 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f64300cf4e0 con 0x7f64300cc9b0 2024-04-02T13:31:51.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.001+0000 7f6437da8640 1 -- 172.21.15.44:0/1555214434 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64301560c0 con 0x7f64300cc9b0 2024-04-02T13:31:51.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.002+0000 7f64267fc640 1 -- 172.21.15.44:0/1555214434 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) ==== 74+0+17033 (secure 0 0 0) 0x7f6420059c60 con 0x7f64300cc9b0 2024-04-02T13:31:51.006 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:51.006 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":122,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:41.102907+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:31:09.752845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-02T13:31:10.437454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6835/2109578276":"2024-04-03T13:31:41.102885+0000","172.21.15.44:6833/2109578276":"2024-04-03T13:31:41.102885+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:31:51.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.004+0000 7f6437da8640 1 -- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f641406c8f0 msgr2=0x7f641406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.004+0000 7f6437da8640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f641406c8f0 0x7f641406edb0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f642c006fd0 tx=0x7f642c008040 comp rx=0 tx=0).stop 2024-04-02T13:31:51.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.004+0000 7f6437da8640 1 -- 172.21.15.44:0/1555214434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64300cc9b0 msgr2=0x7f64300b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.004+0000 7f6437da8640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64300cc9b0 0x7f64300b31d0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f642000c430 tx=0x7f642000c8f0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.004+0000 7f6437da8640 1 -- 172.21.15.44:0/1555214434 shutdown_connections 2024-04-02T13:31:51.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.004+0000 7f6437da8640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f641406c8f0 0x7f641406edb0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.004+0000 7f6437da8640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f643015c2b0 0x7f64300b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.004+0000 7f6437da8640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64300cc9b0 0x7f64300b31d0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.004+0000 7f6437da8640 1 --2- 172.21.15.44:0/1555214434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64300cbd80 0x7f6430155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.004+0000 7f6437da8640 1 -- 172.21.15.44:0/1555214434 >> 172.21.15.44:0/1555214434 conn(0x7f64300b20e0 msgr2=0x7f64300b1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:51.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.004+0000 7f6437da8640 1 -- 172.21.15.44:0/1555214434 shutdown_connections 2024-04-02T13:31:51.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.004+0000 7f6437da8640 1 -- 172.21.15.44:0/1555214434 wait complete. 2024-04-02T13:31:51.018 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:31:09.752845+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:31:10.437454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:31:51.019 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:31:51.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.119+0000 7faa1111c640 1 Processor -- start 2024-04-02T13:31:51.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.119+0000 7faa1111c640 1 -- start start 2024-04-02T13:31:51.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.120+0000 7faa1111c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0d00d0 0x7faa0c15ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.120+0000 7faa1111c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7faa0c0d2470 0x7faa0c15e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.120+0000 7faa1111c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa0c070040 con 0x7faa0c0d2090 2024-04-02T13:31:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.120+0000 7faa1111c640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7faa0c06c5e0 con 0x7faa0c0d2470 2024-04-02T13:31:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.120+0000 7faa1111c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faa0c070320 con 0x7faa0c0d00d0 2024-04-02T13:31:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.120+0000 7faa0b59e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7faa0c0d2470 0x7faa0c15e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.120+0000 7faa0b59e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7faa0c0d2470 0x7faa0c15e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50750/0 (socket says 172.21.15.44:50750) 2024-04-02T13:31:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.120+0000 7faa0b59e640 1 -- 172.21.15.44:0/2374300402 learned_addr learned my addr 172.21.15.44:0/2374300402 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.120+0000 7faa0a59c640 1 --2- 172.21.15.44:0/2374300402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0d00d0 0x7faa0c15ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.120+0000 7faa0b59e640 1 -- 172.21.15.44:0/2374300402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0d00d0 msgr2=0x7faa0c15ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.120+0000 7faa0b59e640 1 --2- 172.21.15.44:0/2374300402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0d00d0 0x7faa0c15ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.120+0000 7faa0b59e640 1 -- 172.21.15.44:0/2374300402 >> v1:172.21.15.69:6789/0 conn(0x7faa0c0d2090 legacy=0x7faa0c06f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.120+0000 7faa0b59e640 1 -- 172.21.15.44:0/2374300402 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa0c070600 con 0x7faa0c0d2470 2024-04-02T13:31:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.121+0000 7faa0b59e640 1 --2- 172.21.15.44:0/2374300402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa0c0d2470 0x7faa0c15e300 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009e50 tx=0x7fa9fc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d82c917b4b57ae6e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.121+0000 7faa09d9b640 1 -- 172.21.15.44:0/2374300402 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc02cb00 con 0x7faa0c0d2470 2024-04-02T13:31:51.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.121+0000 7faa09d9b640 1 -- 172.21.15.44:0/2374300402 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9fc02cca0 con 0x7faa0c0d2470 2024-04-02T13:31:51.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.121+0000 7faa09d9b640 1 -- 172.21.15.44:0/2374300402 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc034610 con 0x7faa0c0d2470 2024-04-02T13:31:51.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.122+0000 7faa1111c640 1 -- 172.21.15.44:0/2374300402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa0c0d2470 msgr2=0x7faa0c15e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.122+0000 7faa1111c640 1 --2- 172.21.15.44:0/2374300402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa0c0d2470 0x7faa0c15e300 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009e50 tx=0x7fa9fc02bf60 comp rx=0 tx=0).stop 2024-04-02T13:31:51.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.122+0000 7faa1111c640 1 -- 172.21.15.44:0/2374300402 shutdown_connections 2024-04-02T13:31:51.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.122+0000 7faa1111c640 1 --2- 172.21.15.44:0/2374300402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa0c0d2470 0x7faa0c15e300 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.122+0000 7faa1111c640 1 --2- 172.21.15.44:0/2374300402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0d00d0 0x7faa0c15ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.122+0000 7faa1111c640 1 -- 172.21.15.44:0/2374300402 >> 172.21.15.44:0/2374300402 conn(0x7faa0c09aa20 msgr2=0x7faa0c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:51.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.122+0000 7faa1111c640 1 -- 172.21.15.44:0/2374300402 shutdown_connections 2024-04-02T13:31:51.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.122+0000 7faa1111c640 1 -- 172.21.15.44:0/2374300402 wait complete. 2024-04-02T13:31:51.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.123+0000 7faa1111c640 1 Processor -- start 2024-04-02T13:31:51.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.123+0000 7faa1111c640 1 -- start start 2024-04-02T13:31:51.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa1111c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0d00d0 0x7faa0c14d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa1111c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa0c0d2090 0x7faa0c15f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa1111c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c0d2470 0x7faa0c154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa1111c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c0cead0 con 0x7faa0c0d2090 2024-04-02T13:31:51.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa1111c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c0ce950 con 0x7faa0c0d2470 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa1111c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faa0c06fc90 con 0x7faa0c0d00d0 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa0a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa0c0d2090 0x7faa0c15f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa0a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa0c0d2090 0x7faa0c15f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50754/0 (socket says 172.21.15.44:50754) 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa0a59c640 1 -- 172.21.15.44:0/4143244080 learned_addr learned my addr 172.21.15.44:0/4143244080 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa0b59e640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c0d2470 0x7faa0c154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa0ad9d640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0d00d0 0x7faa0c14d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa0a59c640 1 -- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0d00d0 msgr2=0x7faa0c14d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa0a59c640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0d00d0 0x7faa0c14d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa0a59c640 1 -- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c0d2470 msgr2=0x7faa0c154b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa0a59c640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c0d2470 0x7faa0c154b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.124+0000 7faa0a59c640 1 -- 172.21.15.44:0/4143244080 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa0c0b17d0 con 0x7faa0c0d2090 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.125+0000 7faa0a59c640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa0c0d2090 0x7faa0c15f2e0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400d5f0 tx=0x7fa9f400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.125+0000 7faa037fe640 1 -- 172.21.15.44:0/4143244080 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f40125f0 con 0x7faa0c0d2090 2024-04-02T13:31:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.125+0000 7faa037fe640 1 -- 172.21.15.44:0/4143244080 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f4019070 con 0x7faa0c0d2090 2024-04-02T13:31:51.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.125+0000 7faa037fe640 1 -- 172.21.15.44:0/4143244080 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f40129d0 con 0x7faa0c0d2090 2024-04-02T13:31:51.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.125+0000 7faa1111c640 1 -- 172.21.15.44:0/4143244080 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa0c0b1ab0 con 0x7faa0c0d2090 2024-04-02T13:31:51.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.125+0000 7faa1111c640 1 -- 172.21.15.44:0/4143244080 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa0c0b2070 con 0x7faa0c0d2090 2024-04-02T13:31:51.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.126+0000 7faa1111c640 1 -- 172.21.15.44:0/4143244080 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa0c09b760 con 0x7faa0c0d2090 2024-04-02T13:31:51.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.126+0000 7faa037fe640 1 -- 172.21.15.44:0/4143244080 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa9f4012b70 con 0x7faa0c0d2090 2024-04-02T13:31:51.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.127+0000 7faa037fe640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa9ec06c8f0 0x7fa9ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.127+0000 7faa037fe640 1 -- 172.21.15.44:0/4143244080 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7fa9f408bfe0 con 0x7faa0c0d2090 2024-04-02T13:31:51.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.127+0000 7faa0ad9d640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa9ec06c8f0 0x7fa9ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.128+0000 7faa0ad9d640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa9ec06c8f0 0x7fa9ec06edb0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7faa04001780 tx=0x7faa04009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:51.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.132+0000 7faa037fe640 1 -- 172.21.15.44:0/4143244080 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7faa0c09b760 con 0x7faa0c0d2090 2024-04-02T13:31:51.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.316+0000 7faa1111c640 1 -- 172.21.15.44:0/4143244080 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faa0c0d2b90 con 0x7faa0c0d2090 2024-04-02T13:31:51.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.317+0000 7faa037fe640 1 -- 172.21.15.44:0/4143244080 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) ==== 93+0+10 (secure 0 0 0) 0x7fa9f4059be0 con 0x7faa0c0d2090 2024-04-02T13:31:51.321 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:31:51.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.318+0000 7faa1111c640 1 -- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa9ec06c8f0 msgr2=0x7fa9ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.318+0000 7faa1111c640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa9ec06c8f0 0x7fa9ec06edb0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7faa04001780 tx=0x7faa04009040 comp rx=0 tx=0).stop 2024-04-02T13:31:51.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.318+0000 7faa1111c640 1 -- 172.21.15.44:0/4143244080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa0c0d2090 msgr2=0x7faa0c15f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.318+0000 7faa1111c640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa0c0d2090 0x7faa0c15f2e0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400d5f0 tx=0x7fa9f400dab0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.318+0000 7faa1111c640 1 -- 172.21.15.44:0/4143244080 shutdown_connections 2024-04-02T13:31:51.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.318+0000 7faa1111c640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa9ec06c8f0 0x7fa9ec06edb0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.318+0000 7faa1111c640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c0d2470 0x7faa0c154b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.318+0000 7faa1111c640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa0c0d2090 0x7faa0c15f2e0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.318+0000 7faa1111c640 1 --2- 172.21.15.44:0/4143244080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0d00d0 0x7faa0c14d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.318+0000 7faa1111c640 1 -- 172.21.15.44:0/4143244080 >> 172.21.15.44:0/4143244080 conn(0x7faa0c09aa20 msgr2=0x7faa0c0b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:51.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.318+0000 7faa1111c640 1 -- 172.21.15.44:0/4143244080 shutdown_connections 2024-04-02T13:31:51.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.318+0000 7faa1111c640 1 -- 172.21.15.44:0/4143244080 wait complete. 2024-04-02T13:31:51.333 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:31:51.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.433+0000 7f51e3b40640 1 Processor -- start 2024-04-02T13:31:51.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.434+0000 7f51e3b40640 1 -- start start 2024-04-02T13:31:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.434+0000 7f51e3b40640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51dc06cff0 0x7f51dc06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.434+0000 7f51e3b40640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f51dc15e5d0 0x7f51dc0d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.434+0000 7f51e3b40640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51dc06d910 con 0x7f51dc0b4180 2024-04-02T13:31:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.434+0000 7f51e3b40640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f51dc0d1bf0 con 0x7f51dc15e5d0 2024-04-02T13:31:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.434+0000 7f51e3b40640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f51dc0d1630 con 0x7f51dc06cff0 2024-04-02T13:31:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.435+0000 7f51e10db640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f51dc15e5d0 0x7f51dc0d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.435+0000 7f51e10db640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f51dc15e5d0 0x7f51dc0d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50770/0 (socket says 172.21.15.44:50770) 2024-04-02T13:31:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.435+0000 7f51e10db640 1 -- 172.21.15.44:0/2069256153 learned_addr learned my addr 172.21.15.44:0/2069256153 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.435+0000 7f51e18dc640 1 --2- 172.21.15.44:0/2069256153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51dc06cff0 0x7f51dc06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.435+0000 7f51e10db640 1 -- 172.21.15.44:0/2069256153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51dc06cff0 msgr2=0x7f51dc06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.435+0000 7f51e10db640 1 --2- 172.21.15.44:0/2069256153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51dc06cff0 0x7f51dc06d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.435+0000 7f51e10db640 1 -- 172.21.15.44:0/2069256153 >> v1:172.21.15.69:6789/0 conn(0x7f51dc0b4180 legacy=0x7f51dc0ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.435+0000 7f51e10db640 1 -- 172.21.15.44:0/2069256153 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51dc0d1910 con 0x7f51dc15e5d0 2024-04-02T13:31:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.436+0000 7f51e10db640 1 --2- 172.21.15.44:0/2069256153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51dc15e5d0 0x7f51dc0d0d30 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009870 tx=0x7f51cc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9d5830b5a0854 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.436+0000 7f51e08da640 1 -- 172.21.15.44:0/2069256153 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51cc0317c0 con 0x7f51dc15e5d0 2024-04-02T13:31:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.436+0000 7f51e08da640 1 -- 172.21.15.44:0/2069256153 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51cc037070 con 0x7f51dc15e5d0 2024-04-02T13:31:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.436+0000 7f51e08da640 1 -- 172.21.15.44:0/2069256153 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51cc031b80 con 0x7f51dc15e5d0 2024-04-02T13:31:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.436+0000 7f51e3b40640 1 -- 172.21.15.44:0/2069256153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51dc15e5d0 msgr2=0x7f51dc0d0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.436+0000 7f51e3b40640 1 --2- 172.21.15.44:0/2069256153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51dc15e5d0 0x7f51dc0d0d30 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009870 tx=0x7f51cc02b860 comp rx=0 tx=0).stop 2024-04-02T13:31:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.436+0000 7f51e3b40640 1 -- 172.21.15.44:0/2069256153 shutdown_connections 2024-04-02T13:31:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.436+0000 7f51e3b40640 1 --2- 172.21.15.44:0/2069256153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51dc15e5d0 0x7f51dc0d0d30 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.436+0000 7f51e3b40640 1 --2- 172.21.15.44:0/2069256153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51dc06cff0 0x7f51dc06d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.436+0000 7f51e3b40640 1 -- 172.21.15.44:0/2069256153 >> 172.21.15.44:0/2069256153 conn(0x7f51dc09aa20 msgr2=0x7f51dc09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.436+0000 7f51e3b40640 1 -- 172.21.15.44:0/2069256153 shutdown_connections 2024-04-02T13:31:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.436+0000 7f51e3b40640 1 -- 172.21.15.44:0/2069256153 wait complete. 2024-04-02T13:31:51.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.437+0000 7f51e3b40640 1 Processor -- start 2024-04-02T13:31:51.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.438+0000 7f51e3b40640 1 -- start start 2024-04-02T13:31:51.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.438+0000 7f51e3b40640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51dc06cff0 0x7f51dc1514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.438+0000 7f51e3b40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51dc0b4180 0x7f51dc0b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.438+0000 7f51e3b40640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51dc15e5d0 0x7f51dc0b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.438+0000 7f51e3b40640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc0d1270 con 0x7f51dc15e5d0 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.438+0000 7f51e3b40640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc0b1820 con 0x7f51dc0b4180 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.438+0000 7f51e3b40640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f51dc0d13f0 con 0x7f51dc06cff0 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.438+0000 7f51e20dd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51dc15e5d0 0x7f51dc0b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.438+0000 7f51e20dd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51dc15e5d0 0x7f51dc0b6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50780/0 (socket says 172.21.15.44:50780) 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.438+0000 7f51e20dd640 1 -- 172.21.15.44:0/4037186437 learned_addr learned my addr 172.21.15.44:0/4037186437 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.438+0000 7f51e10db640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51dc0b4180 0x7f51dc0b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.438+0000 7f51e18dc640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51dc06cff0 0x7f51dc1514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.439+0000 7f51e20dd640 1 -- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51dc06cff0 msgr2=0x7f51dc1514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.439+0000 7f51e20dd640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51dc06cff0 0x7f51dc1514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.439+0000 7f51e20dd640 1 -- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51dc0b4180 msgr2=0x7f51dc0b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.439+0000 7f51e20dd640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51dc0b4180 0x7f51dc0b49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.439+0000 7f51e20dd640 1 -- 172.21.15.44:0/4037186437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51dc0b7350 con 0x7f51dc15e5d0 2024-04-02T13:31:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.439+0000 7f51e10db640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51dc0b4180 0x7f51dc0b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.439+0000 7f51e18dc640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51dc06cff0 0x7f51dc1514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.439+0000 7f51e20dd640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51dc15e5d0 0x7f51dc0b6580 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f51d400c700 tx=0x7f51d400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.439+0000 7f51d27fc640 1 -- 172.21.15.44:0/4037186437 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4016020 con 0x7f51dc15e5d0 2024-04-02T13:31:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.439+0000 7f51d27fc640 1 -- 172.21.15.44:0/4037186437 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51d400aa50 con 0x7f51dc15e5d0 2024-04-02T13:31:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.439+0000 7f51e3b40640 1 -- 172.21.15.44:0/4037186437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51dc0b7630 con 0x7f51dc15e5d0 2024-04-02T13:31:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.439+0000 7f51d27fc640 1 -- 172.21.15.44:0/4037186437 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d400ce10 con 0x7f51dc15e5d0 2024-04-02T13:31:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.439+0000 7f51e3b40640 1 -- 172.21.15.44:0/4037186437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51dc0b7bf0 con 0x7f51dc15e5d0 2024-04-02T13:31:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.440+0000 7f51e3b40640 1 -- 172.21.15.44:0/4037186437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51dc0b7e30 con 0x7f51dc15e5d0 2024-04-02T13:31:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.440+0000 7f51d27fc640 1 -- 172.21.15.44:0/4037186437 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f51d4019020 con 0x7f51dc15e5d0 2024-04-02T13:31:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.441+0000 7f51d27fc640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f51c006c8f0 0x7f51c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.441+0000 7f51d27fc640 1 -- 172.21.15.44:0/4037186437 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f51d408c080 con 0x7f51dc15e5d0 2024-04-02T13:31:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.441+0000 7f51e18dc640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f51c006c8f0 0x7f51c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.441+0000 7f51e18dc640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f51c006c8f0 0x7f51c006edb0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f51d8006fd0 tx=0x7f51d8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.445+0000 7f51d27fc640 1 -- 172.21.15.44:0/4037186437 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f51dc0b7e30 con 0x7f51dc15e5d0 2024-04-02T13:31:51.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.630+0000 7f51e3b40640 1 -- 172.21.15.44:0/4037186437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f51dc0684f0 con 0x7f51dc15e5d0 2024-04-02T13:31:51.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.630+0000 7f51d27fc640 1 -- 172.21.15.44:0/4037186437 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v122) ==== 104+0+11 (secure 0 0 0) 0x7f51d4020020 con 0x7f51dc15e5d0 2024-04-02T13:31:51.634 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:31:51.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.632+0000 7f51e3b40640 1 -- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f51c006c8f0 msgr2=0x7f51c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.632+0000 7f51e3b40640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f51c006c8f0 0x7f51c006edb0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f51d8006fd0 tx=0x7f51d8008040 comp rx=0 tx=0).stop 2024-04-02T13:31:51.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.632+0000 7f51e3b40640 1 -- 172.21.15.44:0/4037186437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51dc15e5d0 msgr2=0x7f51dc0b6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.632+0000 7f51e3b40640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51dc15e5d0 0x7f51dc0b6580 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f51d400c700 tx=0x7f51d400cbc0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.632+0000 7f51e3b40640 1 -- 172.21.15.44:0/4037186437 shutdown_connections 2024-04-02T13:31:51.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.632+0000 7f51e3b40640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f51c006c8f0 0x7f51c006edb0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.632+0000 7f51e3b40640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51dc15e5d0 0x7f51dc0b6580 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.632+0000 7f51e3b40640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51dc0b4180 0x7f51dc0b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.632+0000 7f51e3b40640 1 --2- 172.21.15.44:0/4037186437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51dc06cff0 0x7f51dc1514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.632+0000 7f51e3b40640 1 -- 172.21.15.44:0/4037186437 >> 172.21.15.44:0/4037186437 conn(0x7f51dc09aa20 msgr2=0x7f51dc0705c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:51.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.632+0000 7f51e3b40640 1 -- 172.21.15.44:0/4037186437 shutdown_connections 2024-04-02T13:31:51.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.632+0000 7f51e3b40640 1 -- 172.21.15.44:0/4037186437 wait complete. 2024-04-02T13:31:51.646 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:31:51.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.747+0000 7f4727b00640 1 Processor -- start 2024-04-02T13:31:51.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.747+0000 7f4727b00640 1 -- start start 2024-04-02T13:31:51.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.747+0000 7f4727b00640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f47200cfdc0 0x7f47200b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.748+0000 7f4727b00640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f472006de80 con 0x7f47200d2e00 2024-04-02T13:31:51.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.748+0000 7f4727b00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47200cf000 con 0x7f472006daa0 2024-04-02T13:31:51.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.748+0000 7f4727b00640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f47200cf930 con 0x7f47200cfdc0 2024-04-02T13:31:51.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.748+0000 7f472609d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f47200cfdc0 0x7f47200b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.748+0000 7f472509b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f47200d2e00 0x7f472015dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56666/0 (socket says 172.21.15.44:56666) 2024-04-02T13:31:51.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.748+0000 7f472509b640 1 -- 172.21.15.44:0/1989478173 learned_addr learned my addr 172.21.15.44:0/1989478173 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:51.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.748+0000 7f472609d640 1 -- 172.21.15.44:0/1989478173 >> v1:172.21.15.69:6789/0 conn(0x7f472006daa0 legacy=0x7f47200d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.748+0000 7f472609d640 1 -- 172.21.15.44:0/1989478173 >> v1:172.21.15.44:6789/0 conn(0x7f47200d2e00 legacy=0x7f472015dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.748+0000 7f472489a640 1 -- 172.21.15.44:0/1989478173 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1999918357 0 0) 0x7f472006de80 con 0x7f47200d2e00 2024-04-02T13:31:51.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.748+0000 7f472609d640 1 -- 172.21.15.44:0/1989478173 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47200cf2e0 con 0x7f47200cfdc0 2024-04-02T13:31:51.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.749+0000 7f472609d640 1 --2- 172.21.15.44:0/1989478173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47200cfdc0 0x7f47200b35b0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f4718009d90 tx=0x7f471802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df17c833a86af79 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:51.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.749+0000 7f472489a640 1 -- 172.21.15.44:0/1989478173 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4718039070 con 0x7f47200cfdc0 2024-04-02T13:31:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.749+0000 7f472489a640 1 -- 172.21.15.44:0/1989478173 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f471802cd00 con 0x7f47200cfdc0 2024-04-02T13:31:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.749+0000 7f472489a640 1 -- 172.21.15.44:0/1989478173 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4718034660 con 0x7f47200cfdc0 2024-04-02T13:31:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.749+0000 7f4727b00640 1 -- 172.21.15.44:0/1989478173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47200cfdc0 msgr2=0x7f47200b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.749+0000 7f4727b00640 1 --2- 172.21.15.44:0/1989478173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47200cfdc0 0x7f47200b35b0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f4718009d90 tx=0x7f471802bf50 comp rx=0 tx=0).stop 2024-04-02T13:31:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.749+0000 7f4727b00640 1 -- 172.21.15.44:0/1989478173 shutdown_connections 2024-04-02T13:31:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.749+0000 7f4727b00640 1 --2- 172.21.15.44:0/1989478173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47200cfdc0 0x7f47200b35b0 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.749+0000 7f4727b00640 1 -- 172.21.15.44:0/1989478173 >> 172.21.15.44:0/1989478173 conn(0x7f472009d430 msgr2=0x7f47200d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.750+0000 7f4727b00640 1 -- 172.21.15.44:0/1989478173 shutdown_connections 2024-04-02T13:31:51.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.750+0000 7f4727b00640 1 -- 172.21.15.44:0/1989478173 wait complete. 2024-04-02T13:31:51.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.751+0000 7f4727b00640 1 Processor -- start 2024-04-02T13:31:51.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.751+0000 7f4727b00640 1 -- start start 2024-04-02T13:31:51.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.751+0000 7f4727b00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f472006daa0 0x7f4720157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.751+0000 7f4727b00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47200cfdc0 0x7f47200b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.751+0000 7f4727b00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47200d2e00 0x7f47200cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.751+0000 7f4727b00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f47200ceda0 con 0x7f472006daa0 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.751+0000 7f4727b00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f47200cec20 con 0x7f47200cfdc0 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.751+0000 7f4727b00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f47200d11c0 con 0x7f47200d2e00 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.751+0000 7f472589c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f472006daa0 0x7f4720157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f472589c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f472006daa0 0x7f4720157990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50790/0 (socket says 172.21.15.44:50790) 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f472589c640 1 -- 172.21.15.44:0/1332328498 learned_addr learned my addr 172.21.15.44:0/1332328498 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f472509b640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47200cfdc0 0x7f47200b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f472609d640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47200d2e00 0x7f47200cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f472589c640 1 -- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47200d2e00 msgr2=0x7f47200cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f472589c640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47200d2e00 0x7f47200cd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f472589c640 1 -- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47200cfdc0 msgr2=0x7f47200b7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f472589c640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47200cfdc0 0x7f47200b7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f472589c640 1 -- 172.21.15.44:0/1332328498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47200cc0d0 con 0x7f472006daa0 2024-04-02T13:31:51.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f472589c640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f472006daa0 0x7f4720157990 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f471c00c480 tx=0x7f471c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:51.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f47167fc640 1 -- 172.21.15.44:0/1332328498 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471c0052d0 con 0x7f472006daa0 2024-04-02T13:31:51.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f47167fc640 1 -- 172.21.15.44:0/1332328498 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f471c005470 con 0x7f472006daa0 2024-04-02T13:31:51.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f4727b00640 1 -- 172.21.15.44:0/1332328498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f472015b040 con 0x7f472006daa0 2024-04-02T13:31:51.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f47167fc640 1 -- 172.21.15.44:0/1332328498 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471c012870 con 0x7f472006daa0 2024-04-02T13:31:51.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.752+0000 7f4727b00640 1 -- 172.21.15.44:0/1332328498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f472015b600 con 0x7f472006daa0 2024-04-02T13:31:51.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.753+0000 7f4727b00640 1 -- 172.21.15.44:0/1332328498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47200684f0 con 0x7f472006daa0 2024-04-02T13:31:51.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.753+0000 7f47167fc640 1 -- 172.21.15.44:0/1332328498 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f471c012a10 con 0x7f472006daa0 2024-04-02T13:31:51.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.754+0000 7f47167fc640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f470406c8f0 0x7f470406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:51.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.754+0000 7f47167fc640 1 -- 172.21.15.44:0/1332328498 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f471c08b940 con 0x7f472006daa0 2024-04-02T13:31:51.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.754+0000 7f472509b640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f470406c8f0 0x7f470406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:51.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.755+0000 7f472509b640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f470406c8f0 0x7f470406edb0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f472015ab50 tx=0x7f4710009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:51.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.758+0000 7f47167fc640 1 -- 172.21.15.44:0/1332328498 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f471c059c20 con 0x7f472006daa0 2024-04-02T13:31:51.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.945+0000 7f4727b00640 1 -- 172.21.15.44:0/1332328498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f47200d24d0 con 0x7f472006daa0 2024-04-02T13:31:51.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.945+0000 7f47167fc640 1 -- 172.21.15.44:0/1332328498 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v122) ==== 100+0+11 (secure 0 0 0) 0x7f471c059520 con 0x7f472006daa0 2024-04-02T13:31:51.949 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:31:51.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.947+0000 7f4727b00640 1 -- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f470406c8f0 msgr2=0x7f470406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.947+0000 7f4727b00640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f470406c8f0 0x7f470406edb0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f472015ab50 tx=0x7f4710009040 comp rx=0 tx=0).stop 2024-04-02T13:31:51.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.947+0000 7f4727b00640 1 -- 172.21.15.44:0/1332328498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f472006daa0 msgr2=0x7f4720157990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:51.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.947+0000 7f4727b00640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f472006daa0 0x7f4720157990 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f471c00c480 tx=0x7f471c00c940 comp rx=0 tx=0).stop 2024-04-02T13:31:51.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.947+0000 7f4727b00640 1 -- 172.21.15.44:0/1332328498 shutdown_connections 2024-04-02T13:31:51.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.947+0000 7f4727b00640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f470406c8f0 0x7f470406edb0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.947+0000 7f4727b00640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47200d2e00 0x7f47200cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.947+0000 7f4727b00640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47200cfdc0 0x7f47200b7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.947+0000 7f4727b00640 1 --2- 172.21.15.44:0/1332328498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f472006daa0 0x7f4720157990 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:51.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.947+0000 7f4727b00640 1 -- 172.21.15.44:0/1332328498 >> 172.21.15.44:0/1332328498 conn(0x7f472009d430 msgr2=0x7f47200b46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:51.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.947+0000 7f4727b00640 1 -- 172.21.15.44:0/1332328498 shutdown_connections 2024-04-02T13:31:51.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:51.947+0000 7f4727b00640 1 -- 172.21.15.44:0/1332328498 wait complete. 2024-04-02T13:31:51.961 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:31:52.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.060+0000 7fb056fba640 1 Processor -- start 2024-04-02T13:31:52.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.060+0000 7fb056fba640 1 -- start start 2024-04-02T13:31:52.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb056fba640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb05015aa00 0x7fb0500b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:52.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb056fba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb0500ce2e0 0x7fb050071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:52.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb056fba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb05015ade0 con 0x7fb050072bf0 2024-04-02T13:31:52.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb056fba640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fb0500af1b0 con 0x7fb05015aa00 2024-04-02T13:31:52.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb056fba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb0500af490 con 0x7fb0500ce2e0 2024-04-02T13:31:52.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb054d56640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb05015aa00 0x7fb0500b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:52.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb04ffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb050072bf0 0x7fb0500cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56670/0 (socket says 172.21.15.44:56670) 2024-04-02T13:31:52.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb054d56640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb05015aa00 0x7fb0500b2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50796/0 (socket says 172.21.15.44:50796) 2024-04-02T13:31:52.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb04ffff640 1 -- 172.21.15.44:0/489018412 learned_addr learned my addr 172.21.15.44:0/489018412 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:52.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb055557640 1 --2- 172.21.15.44:0/489018412 >> v2:172.21.15.69:3300/0 conn(0x7fb0500ce2e0 0x7fb050071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:52.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb054d56640 1 -- 172.21.15.44:0/489018412 >> v2:172.21.15.69:3300/0 conn(0x7fb0500ce2e0 msgr2=0x7fb050071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:52.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb054d56640 1 --2- 172.21.15.44:0/489018412 >> v2:172.21.15.69:3300/0 conn(0x7fb0500ce2e0 0x7fb050071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:52.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb054d56640 1 -- 172.21.15.44:0/489018412 >> v1:172.21.15.44:6789/0 conn(0x7fb050072bf0 legacy=0x7fb0500cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:52.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb04f7fe640 1 -- 172.21.15.44:0/489018412 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1331067149 0 0) 0x7fb05015ade0 con 0x7fb050072bf0 2024-04-02T13:31:52.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb054d56640 1 -- 172.21.15.44:0/489018412 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0500723f0 con 0x7fb05015aa00 2024-04-02T13:31:52.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.061+0000 7fb055557640 1 --2- 172.21.15.44:0/489018412 >> v2:172.21.15.69:3300/0 conn(0x7fb0500ce2e0 0x7fb050071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:52.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.062+0000 7fb054d56640 1 --2- 172.21.15.44:0/489018412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb05015aa00 0x7fb0500b2d00 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fb05006f3c0 tx=0x7fb04402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6a3ca927c2b389 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:52.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.062+0000 7fb04f7fe640 1 -- 172.21.15.44:0/489018412 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0440315a0 con 0x7fb05015aa00 2024-04-02T13:31:52.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.062+0000 7fb04f7fe640 1 -- 172.21.15.44:0/489018412 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb044037070 con 0x7fb05015aa00 2024-04-02T13:31:52.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.062+0000 7fb04f7fe640 1 -- 172.21.15.44:0/489018412 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb044031960 con 0x7fb05015aa00 2024-04-02T13:31:52.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.062+0000 7fb056fba640 1 -- 172.21.15.44:0/489018412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb05015aa00 msgr2=0x7fb0500b2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.062+0000 7fb056fba640 1 --2- 172.21.15.44:0/489018412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb05015aa00 0x7fb0500b2d00 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fb05006f3c0 tx=0x7fb04402b840 comp rx=0 tx=0).stop 2024-04-02T13:31:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.062+0000 7fb056fba640 1 -- 172.21.15.44:0/489018412 shutdown_connections 2024-04-02T13:31:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.062+0000 7fb056fba640 1 --2- 172.21.15.44:0/489018412 >> v2:172.21.15.69:3300/0 conn(0x7fb0500ce2e0 0x7fb050071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.062+0000 7fb056fba640 1 --2- 172.21.15.44:0/489018412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb05015aa00 0x7fb0500b2d00 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.062+0000 7fb056fba640 1 -- 172.21.15.44:0/489018412 >> 172.21.15.44:0/489018412 conn(0x7fb0500cfcd0 msgr2=0x7fb0500d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.062+0000 7fb056fba640 1 -- 172.21.15.44:0/489018412 shutdown_connections 2024-04-02T13:31:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.063+0000 7fb056fba640 1 -- 172.21.15.44:0/489018412 wait complete. 2024-04-02T13:31:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.064+0000 7fb056fba640 1 Processor -- start 2024-04-02T13:31:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.064+0000 7fb056fba640 1 -- start start 2024-04-02T13:31:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.064+0000 7fb056fba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb050072bf0 0x7fb0500ace50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.064+0000 7fb056fba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0500ce2e0 0x7fb050146fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.064+0000 7fb056fba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb05015aa00 0x7fb0500c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.064+0000 7fb056fba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb0500731d0 con 0x7fb050072bf0 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.064+0000 7fb056fba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb050073050 con 0x7fb05015aa00 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.064+0000 7fb056fba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb0500cc750 con 0x7fb0500ce2e0 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb054d56640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb050072bf0 0x7fb0500ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb054d56640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb050072bf0 0x7fb0500ace50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50802/0 (socket says 172.21.15.44:50802) 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb054d56640 1 -- 172.21.15.44:0/4020430295 learned_addr learned my addr 172.21.15.44:0/4020430295 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb055557640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb05015aa00 0x7fb0500c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb04ffff640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0500ce2e0 0x7fb050146fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb054d56640 1 -- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0500ce2e0 msgr2=0x7fb050146fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb054d56640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0500ce2e0 0x7fb050146fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb054d56640 1 -- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb05015aa00 msgr2=0x7fb0500c9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb054d56640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb05015aa00 0x7fb0500c9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb054d56640 1 -- 172.21.15.44:0/4020430295 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0500ad550 con 0x7fb050072bf0 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb055557640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb05015aa00 0x7fb0500c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb054d56640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb050072bf0 0x7fb0500ace50 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fb044034040 tx=0x7fb0440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb04d7fa640 1 -- 172.21.15.44:0/4020430295 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb044004230 con 0x7fb050072bf0 2024-04-02T13:31:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb04d7fa640 1 -- 172.21.15.44:0/4020430295 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0440043d0 con 0x7fb050072bf0 2024-04-02T13:31:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb04d7fa640 1 -- 172.21.15.44:0/4020430295 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb044032860 con 0x7fb050072bf0 2024-04-02T13:31:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb056fba640 1 -- 172.21.15.44:0/4020430295 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0500ad830 con 0x7fb050072bf0 2024-04-02T13:31:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.065+0000 7fb056fba640 1 -- 172.21.15.44:0/4020430295 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0500ae0d0 con 0x7fb050072bf0 2024-04-02T13:31:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.066+0000 7fb056fba640 1 -- 172.21.15.44:0/4020430295 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb050074f20 con 0x7fb050072bf0 2024-04-02T13:31:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.067+0000 7fb04d7fa640 1 -- 172.21.15.44:0/4020430295 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb044032a00 con 0x7fb050072bf0 2024-04-02T13:31:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.067+0000 7fb04d7fa640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb03406c8f0 0x7fb03406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.067+0000 7fb04d7fa640 1 -- 172.21.15.44:0/4020430295 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7fb04402f030 con 0x7fb050072bf0 2024-04-02T13:31:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.067+0000 7fb04ffff640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb03406c8f0 0x7fb03406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.068+0000 7fb04ffff640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb03406c8f0 0x7fb03406edb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb03c007fd0 tx=0x7fb03c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:52.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.072+0000 7fb04d7fa640 1 -- 172.21.15.44:0/4020430295 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb044041070 con 0x7fb050072bf0 2024-04-02T13:31:52.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.270+0000 7fb056fba640 1 -- 172.21.15.44:0/4020430295 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb0500684f0 con 0x7fb050072bf0 2024-04-02T13:31:52.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.270+0000 7fb04d7fa640 1 -- 172.21.15.44:0/4020430295 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v122) ==== 80+0+122 (secure 0 0 0) 0x7fb044044340 con 0x7fb050072bf0 2024-04-02T13:31:52.274 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6835/2109578276 2024-04-03T13:31:41.102885+0000 2024-04-02T13:31:52.274 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6833/2109578276 2024-04-03T13:31:41.102885+0000 2024-04-02T13:31:52.274 INFO:teuthology.orchestra.run.smithi044.stderr:listed 2 entries 2024-04-02T13:31:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.272+0000 7fb056fba640 1 -- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb03406c8f0 msgr2=0x7fb03406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.272+0000 7fb056fba640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb03406c8f0 0x7fb03406edb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb03c007fd0 tx=0x7fb03c009040 comp rx=0 tx=0).stop 2024-04-02T13:31:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.272+0000 7fb056fba640 1 -- 172.21.15.44:0/4020430295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb050072bf0 msgr2=0x7fb0500ace50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.272+0000 7fb056fba640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb050072bf0 0x7fb0500ace50 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fb044034040 tx=0x7fb0440046c0 comp rx=0 tx=0).stop 2024-04-02T13:31:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.272+0000 7fb056fba640 1 -- 172.21.15.44:0/4020430295 shutdown_connections 2024-04-02T13:31:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.272+0000 7fb056fba640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb03406c8f0 0x7fb03406edb0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.272+0000 7fb056fba640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb05015aa00 0x7fb0500c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.272+0000 7fb056fba640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0500ce2e0 0x7fb050146fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:52.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.272+0000 7fb056fba640 1 --2- 172.21.15.44:0/4020430295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb050072bf0 0x7fb0500ace50 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:52.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.272+0000 7fb056fba640 1 -- 172.21.15.44:0/4020430295 >> 172.21.15.44:0/4020430295 conn(0x7fb0500cfcd0 msgr2=0x7fb05014c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:52.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.272+0000 7fb056fba640 1 -- 172.21.15.44:0/4020430295 shutdown_connections 2024-04-02T13:31:52.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:52.272+0000 7fb056fba640 1 -- 172.21.15.44:0/4020430295 wait complete. 2024-04-02T13:31:52.286 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:31:52.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:52.287 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:31:52.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[84152]: fuse finished with error 0 and tester_r 0 2024-04-02T13:31:52.313 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:31:54.007 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:31:54.010 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:31:58.415 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:31:58.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:31:58.415 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:31:58.434 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:31:58.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.583+0000 7ff90d58f640 1 Processor -- start 2024-04-02T13:31:58.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.583+0000 7ff90d58f640 1 -- start start 2024-04-02T13:31:58.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.583+0000 7ff90d58f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff9080cffd0 0x7ff9080ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:58.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.583+0000 7ff90d58f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff9080cd3a0 0x7ff9081629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:58.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.584+0000 7ff90d58f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9080cef30 con 0x7ff9081617e0 2024-04-02T13:31:58.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.584+0000 7ff90d58f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7ff908162f00 con 0x7ff9080cffd0 2024-04-02T13:31:58.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.584+0000 7ff90d58f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff9080aea90 con 0x7ff9080cd3a0 2024-04-02T13:31:58.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.584+0000 7ff8fffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff9080cffd0 0x7ff9080ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:58.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.584+0000 7ff8fffff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7ff9080cffd0 0x7ff9080ce8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50586/0 (socket says 172.21.15.44:50586) 2024-04-02T13:31:58.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.584+0000 7ff8fffff640 1 -- 172.21.15.44:0/2474235510 learned_addr learned my addr 172.21.15.44:0/2474235510 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:58.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.584+0000 7ff8fffff640 1 -- 172.21.15.44:0/2474235510 >> v2:172.21.15.69:3300/0 conn(0x7ff9080cd3a0 msgr2=0x7ff9081629c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:58.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.584+0000 7ff8fffff640 1 --2- 172.21.15.44:0/2474235510 >> v2:172.21.15.69:3300/0 conn(0x7ff9080cd3a0 0x7ff9081629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:58.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.584+0000 7ff8fffff640 1 -- 172.21.15.44:0/2474235510 >> v1:172.21.15.69:6789/0 conn(0x7ff9081617e0 legacy=0x7ff9080ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:58.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.584+0000 7ff8fffff640 1 -- 172.21.15.44:0/2474235510 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9080cf210 con 0x7ff9080cffd0 2024-04-02T13:31:58.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.585+0000 7ff8fffff640 1 --2- 172.21.15.44:0/2474235510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9080cffd0 0x7ff9080ce8c0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7ff9080cee60 tx=0x7ff8f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f51277174a4b40d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:58.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.585+0000 7ff9067fc640 1 -- 172.21.15.44:0/2474235510 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0035020 con 0x7ff9080cffd0 2024-04-02T13:31:58.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.585+0000 7ff9067fc640 1 -- 172.21.15.44:0/2474235510 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8f0031870 con 0x7ff9080cffd0 2024-04-02T13:31:58.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.585+0000 7ff9067fc640 1 -- 172.21.15.44:0/2474235510 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0031c40 con 0x7ff9080cffd0 2024-04-02T13:31:58.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.586+0000 7ff90d58f640 1 -- 172.21.15.44:0/2474235510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9080cffd0 msgr2=0x7ff9080ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:58.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.586+0000 7ff90d58f640 1 --2- 172.21.15.44:0/2474235510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9080cffd0 0x7ff9080ce8c0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7ff9080cee60 tx=0x7ff8f002b910 comp rx=0 tx=0).stop 2024-04-02T13:31:58.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.586+0000 7ff90d58f640 1 -- 172.21.15.44:0/2474235510 shutdown_connections 2024-04-02T13:31:58.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.586+0000 7ff90d58f640 1 --2- 172.21.15.44:0/2474235510 >> v2:172.21.15.69:3300/0 conn(0x7ff9080cd3a0 0x7ff9081629c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:58.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.586+0000 7ff90d58f640 1 --2- 172.21.15.44:0/2474235510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9080cffd0 0x7ff9080ce8c0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:58.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.586+0000 7ff90d58f640 1 -- 172.21.15.44:0/2474235510 >> 172.21.15.44:0/2474235510 conn(0x7ff9080b20e0 msgr2=0x7ff9080b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:58.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.586+0000 7ff90d58f640 1 -- 172.21.15.44:0/2474235510 shutdown_connections 2024-04-02T13:31:58.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.586+0000 7ff90d58f640 1 -- 172.21.15.44:0/2474235510 wait complete. 2024-04-02T13:31:58.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.587+0000 7ff90d58f640 1 Processor -- start 2024-04-02T13:31:58.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.587+0000 7ff90d58f640 1 -- start start 2024-04-02T13:31:58.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.587+0000 7ff90d58f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9080cd3a0 0x7ff9080b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:58.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff90d58f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9080cffd0 0x7ff9080b5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff90d58f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9081617e0 0x7ff9080aef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff90d58f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff9080cfab0 con 0x7ff9081617e0 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff90d58f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff9080cf7d0 con 0x7ff9080cd3a0 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff90d58f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff908161bc0 con 0x7ff9080cffd0 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff9077fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9081617e0 0x7ff9080aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff9077fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9081617e0 0x7ff9080aef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50588/0 (socket says 172.21.15.44:50588) 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff9077fe640 1 -- 172.21.15.44:0/3549100042 learned_addr learned my addr 172.21.15.44:0/3549100042 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff906ffd640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9080cd3a0 0x7ff9080b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff8fffff640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9080cffd0 0x7ff9080b5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff9077fe640 1 -- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9080cffd0 msgr2=0x7ff9080b5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff9077fe640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9080cffd0 0x7ff9080b5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff9077fe640 1 -- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9080cd3a0 msgr2=0x7ff9080b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff9077fe640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9080cd3a0 0x7ff9080b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:58.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff9077fe640 1 -- 172.21.15.44:0/3549100042 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9080c8710 con 0x7ff9081617e0 2024-04-02T13:31:58.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff8fffff640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9080cffd0 0x7ff9080b5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:58.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff9077fe640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9081617e0 0x7ff9080aef10 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7ff90000b970 tx=0x7ff90000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:58.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff8ff7fe640 1 -- 172.21.15.44:0/3549100042 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff900019070 con 0x7ff9081617e0 2024-04-02T13:31:58.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff8ff7fe640 1 -- 172.21.15.44:0/3549100042 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff90000cb60 con 0x7ff9081617e0 2024-04-02T13:31:58.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff90d58f640 1 -- 172.21.15.44:0/3549100042 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9080c89f0 con 0x7ff9081617e0 2024-04-02T13:31:58.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff8ff7fe640 1 -- 172.21.15.44:0/3549100042 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff900014440 con 0x7ff9081617e0 2024-04-02T13:31:58.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.588+0000 7ff90d58f640 1 -- 172.21.15.44:0/3549100042 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9080c8fb0 con 0x7ff9081617e0 2024-04-02T13:31:58.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.590+0000 7ff90d58f640 1 -- 172.21.15.44:0/3549100042 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9080684f0 con 0x7ff9081617e0 2024-04-02T13:31:58.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.590+0000 7ff8ff7fe640 1 -- 172.21.15.44:0/3549100042 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff90001c070 con 0x7ff9081617e0 2024-04-02T13:31:58.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.590+0000 7ff8ff7fe640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8e806c8f0 0x7ff8e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:58.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.590+0000 7ff8ff7fe640 1 -- 172.21.15.44:0/3549100042 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7ff90008b960 con 0x7ff9081617e0 2024-04-02T13:31:58.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.591+0000 7ff906ffd640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8e806c8f0 0x7ff8e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:58.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.591+0000 7ff906ffd640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8e806c8f0 0x7ff8e806edb0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ff9080ca820 tx=0x7ff8f8009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:58.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.595+0000 7ff8ff7fe640 1 -- 172.21.15.44:0/3549100042 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7ff9000595f0 con 0x7ff9081617e0 2024-04-02T13:31:58.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.816+0000 7ff90d58f640 1 -- 172.21.15.44:0/3549100042 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9080ce190 con 0x7ff9081617e0 2024-04-02T13:31:58.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.817+0000 7ff8ff7fe640 1 -- 172.21.15.44:0/3549100042 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5185 (secure 0 0 0) 0x7ff90005d8f0 con 0x7ff9081617e0 2024-04-02T13:31:58.821 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:31:58.821 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":105,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:31:11.495701+0000","modified":"2024-04-02T13:31:45.116238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9855},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9855":{"gid":9855,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6835/2895101355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":2895101355},{"type":"v1","addr":"172.21.15.44:6835","nonce":2895101355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9855,"qdb_cluster":[9855]},"id":8}]} 2024-04-02T13:31:58.821 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 105 2024-04-02T13:31:58.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.819+0000 7ff90d58f640 1 -- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8e806c8f0 msgr2=0x7ff8e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:58.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.819+0000 7ff90d58f640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8e806c8f0 0x7ff8e806edb0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ff9080ca820 tx=0x7ff8f8009060 comp rx=0 tx=0).stop 2024-04-02T13:31:58.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.819+0000 7ff90d58f640 1 -- 172.21.15.44:0/3549100042 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9081617e0 msgr2=0x7ff9080aef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:58.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.819+0000 7ff90d58f640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9081617e0 0x7ff9080aef10 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7ff90000b970 tx=0x7ff90000be30 comp rx=0 tx=0).stop 2024-04-02T13:31:58.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.819+0000 7ff90d58f640 1 -- 172.21.15.44:0/3549100042 shutdown_connections 2024-04-02T13:31:58.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.819+0000 7ff90d58f640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7ff8e806c8f0 0x7ff8e806edb0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:58.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.819+0000 7ff90d58f640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9081617e0 0x7ff9080aef10 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:58.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.819+0000 7ff90d58f640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9080cffd0 0x7ff9080b5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:58.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.819+0000 7ff90d58f640 1 --2- 172.21.15.44:0/3549100042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9080cd3a0 0x7ff9080b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:58.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.819+0000 7ff90d58f640 1 -- 172.21.15.44:0/3549100042 >> 172.21.15.44:0/3549100042 conn(0x7ff9080b20e0 msgr2=0x7ff9080d0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:58.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.819+0000 7ff90d58f640 1 -- 172.21.15.44:0/3549100042 shutdown_connections 2024-04-02T13:31:58.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.819+0000 7ff90d58f640 1 -- 172.21.15.44:0/3549100042 wait complete. 2024-04-02T13:31:58.834 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json tag path /parent mytag123 2024-04-02T13:31:58.957 INFO:teuthology.orchestra.run.smithi044.stdout:{"return_code":0,"scrub_tag":"mytag123","mode":"asynchronous"} 2024-04-02T13:31:58.958 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag123" } 2024-04-02T13:31:58.958 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2024-04-02T13:31:59.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.996+0000 7f049dd5ff40 1 Processor -- start 2024-04-02T13:31:59.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.996+0000 7f049dd5ff40 1 -- start start 2024-04-02T13:31:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049dd5ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0484003630 0x55da37e95d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049dd5ff40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da37e9e9a0 0x55da37df2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049dd5ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55da37e9ee20 0x55da37df2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049dd5ff40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55da37d6f3e0 con 0x55da37e9e9a0 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049dd5ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55da37d73430 con 0x7f0484003630 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049dd5ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55da37dc79b0 con 0x55da37e9ee20 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049b05f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da37e9e9a0 0x55da37df2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049b05f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da37e9e9a0 0x55da37df2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50600/0 (socket says 172.21.15.44:50600) 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049b05f640 1 -- 172.21.15.44:0/4257236544 learned_addr learned my addr 172.21.15.44:0/4257236544 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049b860640 1 --2- 172.21.15.44:0/4257236544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0484003630 0x55da37e95d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049c061640 1 --2- 172.21.15.44:0/4257236544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55da37e9ee20 0x55da37df2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049b05f640 1 -- 172.21.15.44:0/4257236544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55da37e9ee20 msgr2=0x55da37df2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049b05f640 1 --2- 172.21.15.44:0/4257236544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55da37e9ee20 0x55da37df2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049b05f640 1 -- 172.21.15.44:0/4257236544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0484003630 msgr2=0x55da37e95d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049b05f640 1 --2- 172.21.15.44:0/4257236544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0484003630 0x55da37e95d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049b05f640 1 -- 172.21.15.44:0/4257236544 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da37e96590 con 0x55da37e9e9a0 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049b860640 1 --2- 172.21.15.44:0/4257236544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0484003630 0x55da37e95d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.997+0000 7f049b05f640 1 --2- 172.21.15.44:0/4257236544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da37e9e9a0 0x55da37df2990 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f0490009760 tx=0x7f0490005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52861e067c4a159 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.998+0000 7f049a05d640 1 -- 172.21.15.44:0/4257236544 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490008930 con 0x55da37e9e9a0 2024-04-02T13:31:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.998+0000 7f049a05d640 1 -- 172.21.15.44:0/4257236544 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049001a070 con 0x55da37e9e9a0 2024-04-02T13:31:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.998+0000 7f049a05d640 1 -- 172.21.15.44:0/4257236544 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490008cf0 con 0x55da37e9e9a0 2024-04-02T13:31:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.998+0000 7f049dd5ff40 1 -- 172.21.15.44:0/4257236544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da37e9e9a0 msgr2=0x55da37df2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.998+0000 7f049dd5ff40 1 --2- 172.21.15.44:0/4257236544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da37e9e9a0 0x55da37df2990 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f0490009760 tx=0x7f0490005a00 comp rx=0 tx=0).stop 2024-04-02T13:31:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.998+0000 7f049dd5ff40 1 -- 172.21.15.44:0/4257236544 shutdown_connections 2024-04-02T13:31:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.998+0000 7f049dd5ff40 1 --2- 172.21.15.44:0/4257236544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55da37e9ee20 0x55da37df2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.998+0000 7f049dd5ff40 1 --2- 172.21.15.44:0/4257236544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da37e9e9a0 0x55da37df2990 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.998+0000 7f049dd5ff40 1 --2- 172.21.15.44:0/4257236544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0484003630 0x55da37e95d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.998+0000 7f049dd5ff40 1 -- 172.21.15.44:0/4257236544 >> 172.21.15.44:0/4257236544 conn(0x55da37df4c80 msgr2=0x55da37e81050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.998+0000 7f049dd5ff40 1 -- 172.21.15.44:0/4257236544 shutdown_connections 2024-04-02T13:31:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.998+0000 7f049dd5ff40 1 -- 172.21.15.44:0/4257236544 wait complete. 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.999+0000 7f049dd5ff40 1 Processor -- start 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.999+0000 7f049dd5ff40 1 -- start start 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.999+0000 7f049dd5ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55da37e9e9a0 0x55da37e0dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.999+0000 7f049dd5ff40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da37e9ee20 0x55da37e7cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.999+0000 7f049dd5ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0484003630 0x55da37e99990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.999+0000 7f049dd5ff40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55da37dc7f30 con 0x55da37e9ee20 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.999+0000 7f049dd5ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55da37dc7c50 con 0x55da37e9e9a0 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.999+0000 7f049dd5ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55da37e96270 con 0x7f0484003630 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.999+0000 7f049b05f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da37e9ee20 0x55da37e7cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.999+0000 7f049b05f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da37e9ee20 0x55da37e7cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50618/0 (socket says 172.21.15.44:50618) 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:58.999+0000 7f049b05f640 1 -- 172.21.15.44:0/333955014 learned_addr learned my addr 172.21.15.44:0/333955014 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.000+0000 7f049b860640 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55da37e9e9a0 0x55da37e0dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.000+0000 7f049c061640 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0484003630 0x55da37e99990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.000+0000 7f049b05f640 1 -- 172.21.15.44:0/333955014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0484003630 msgr2=0x55da37e99990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.000+0000 7f049b05f640 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0484003630 0x55da37e99990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.000+0000 7f049b05f640 1 -- 172.21.15.44:0/333955014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55da37e9e9a0 msgr2=0x55da37e0dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.000+0000 7f049b05f640 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55da37e9e9a0 0x55da37e0dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.000+0000 7f049b05f640 1 -- 172.21.15.44:0/333955014 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da37e9c120 con 0x55da37e9ee20 2024-04-02T13:31:59.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.000+0000 7f049b05f640 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da37e9ee20 0x55da37e7cbb0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f0490016000 tx=0x7f0490002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.000+0000 7f049905b640 1 -- 172.21.15.44:0/333955014 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490015680 con 0x55da37e9ee20 2024-04-02T13:31:59.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.000+0000 7f049905b640 1 -- 172.21.15.44:0/333955014 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0490023070 con 0x55da37e9ee20 2024-04-02T13:31:59.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.000+0000 7f049905b640 1 -- 172.21.15.44:0/333955014 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490015ac0 con 0x55da37e9ee20 2024-04-02T13:31:59.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.000+0000 7f049dd5ff40 1 -- 172.21.15.44:0/333955014 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55da37e96f90 con 0x55da37e9ee20 2024-04-02T13:31:59.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.000+0000 7f049dd5ff40 1 -- 172.21.15.44:0/333955014 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da37e97550 con 0x55da37e9ee20 2024-04-02T13:31:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:31:59.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.001+0000 7f049905b640 1 -- 172.21.15.44:0/333955014 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0490015c60 con 0x55da37e9ee20 2024-04-02T13:31:59.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.002+0000 7f049905b640 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f048806b300 0x7f048806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.002+0000 7f049905b640 1 -- 172.21.15.44:0/333955014 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f049008cb00 con 0x55da37e9ee20 2024-04-02T13:31:59.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.002+0000 7f049b860640 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f048806b300 0x7f048806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.002+0000 7f049dd5ff40 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55da37e97d50 0x55da37e98180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.002+0000 7f049dd5ff40 1 -- 172.21.15.44:0/333955014 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:1 18.36 18:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) -- 0x55da37e986c0 con 0x55da37e97d50 2024-04-02T13:31:59.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.002+0000 7f049c061640 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55da37e97d50 0x55da37e98180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.002+0000 7f049b860640 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f048806b300 0x7f048806d7c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0494004290 tx=0x7f04940068f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.003+0000 7f049c061640 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55da37e97d50 0x55da37e98180 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.003+0000 7f049c061640 1 -- 172.21.15.44:0/333955014 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) ==== 164+0+0 (crc 0 0 0) 0x7f048c001470 con 0x55da37e97d50 2024-04-02T13:31:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:error getting xattr cephfs_data/100000001f7.00000000/scrub_tag: (61) No data available 2024-04-02T13:31:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.003+0000 7f049dd5ff40 1 -- 172.21.15.44:0/333955014 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55da37e97d50 msgr2=0x55da37e98180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.003+0000 7f049dd5ff40 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55da37e97d50 0x55da37e98180 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.003+0000 7f049dd5ff40 1 -- 172.21.15.44:0/333955014 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f048806b300 msgr2=0x7f048806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.003+0000 7f049dd5ff40 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f048806b300 0x7f048806d7c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0494004290 tx=0x7f04940068f0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.003+0000 7f049dd5ff40 1 -- 172.21.15.44:0/333955014 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da37e9ee20 msgr2=0x55da37e7cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.003+0000 7f049dd5ff40 1 --2- 172.21.15.44:0/333955014 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da37e9ee20 0x55da37e7cbb0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f0490016000 tx=0x7f0490002fb0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.003+0000 7f049c061640 1 -- 172.21.15.44:0/333955014 reap_dead start 2024-04-02T13:31:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.003+0000 7f049dd5ff40 1 -- 172.21.15.44:0/333955014 shutdown_connections 2024-04-02T13:31:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.003+0000 7f049dd5ff40 1 -- 172.21.15.44:0/333955014 >> 172.21.15.44:0/333955014 conn(0x55da37df4c80 msgr2=0x55da37dc7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.003+0000 7f049dd5ff40 1 -- 172.21.15.44:0/333955014 shutdown_connections 2024-04-02T13:31:59.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.003+0000 7f049dd5ff40 1 -- 172.21.15.44:0/333955014 wait complete. 2024-04-02T13:31:59.013 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2024-04-02T13:31:59.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.089+0000 7f0dd9af3f40 1 Processor -- start 2024-04-02T13:31:59.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.089+0000 7f0dd9af3f40 1 -- start start 2024-04-02T13:31:59.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.089+0000 7f0dd9af3f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f1872beb0 0x564f187b94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.089+0000 7f0dd9af3f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f187b84e0 0x564f187bc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.089+0000 7f0dd9af3f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f187be2e0 0x564f1884ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.089+0000 7f0dd9af3f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564f18730920 con 0x564f1872beb0 2024-04-02T13:31:59.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.089+0000 7f0dd9af3f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564f1868c3e0 con 0x564f187be2e0 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.089+0000 7f0dd9af3f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564f1872e970 con 0x564f187b84e0 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.090+0000 7f0dd75f4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f1872beb0 0x564f187b94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.090+0000 7f0dd75f4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f1872beb0 0x564f187b94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50642/0 (socket says 172.21.15.44:50642) 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.090+0000 7f0dd75f4640 1 -- 172.21.15.44:0/110196395 learned_addr learned my addr 172.21.15.44:0/110196395 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.090+0000 7f0dd75f4640 1 -- 172.21.15.44:0/110196395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f187b84e0 msgr2=0x564f187bc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.090+0000 7f0dd75f4640 1 --2- 172.21.15.44:0/110196395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f187b84e0 0x564f187bc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.090+0000 7f0dd75f4640 1 -- 172.21.15.44:0/110196395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f187be2e0 msgr2=0x564f1884ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.090+0000 7f0dd75f4640 1 --2- 172.21.15.44:0/110196395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f187be2e0 0x564f1884ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.090+0000 7f0dd75f4640 1 -- 172.21.15.44:0/110196395 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f187b6a50 con 0x564f1872beb0 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.090+0000 7f0dd75f4640 1 --2- 172.21.15.44:0/110196395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f1872beb0 0x564f187b94f0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f0dc801ba10 tx=0x7f0dc804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10624053037aace server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.090+0000 7f0dd5df1640 1 -- 172.21.15.44:0/110196395 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc80528c0 con 0x564f1872beb0 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.090+0000 7f0dd5df1640 1 -- 172.21.15.44:0/110196395 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dc8062070 con 0x564f1872beb0 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.090+0000 7f0dd5df1640 1 -- 172.21.15.44:0/110196395 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8052c60 con 0x564f1872beb0 2024-04-02T13:31:59.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.091+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/110196395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f1872beb0 msgr2=0x564f187b94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.091+0000 7f0dd9af3f40 1 --2- 172.21.15.44:0/110196395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f1872beb0 0x564f187b94f0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f0dc801ba10 tx=0x7f0dc804c000 comp rx=0 tx=0).stop 2024-04-02T13:31:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.091+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/110196395 shutdown_connections 2024-04-02T13:31:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.091+0000 7f0dd9af3f40 1 --2- 172.21.15.44:0/110196395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f187be2e0 0x564f1884ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.091+0000 7f0dd9af3f40 1 --2- 172.21.15.44:0/110196395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f187b84e0 0x564f187bc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.091+0000 7f0dd9af3f40 1 --2- 172.21.15.44:0/110196395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f1872beb0 0x564f187b94f0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.091+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/110196395 >> 172.21.15.44:0/110196395 conn(0x564f18710b50 msgr2=0x564f187117c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.091+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/110196395 shutdown_connections 2024-04-02T13:31:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.091+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/110196395 wait complete. 2024-04-02T13:31:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.091+0000 7f0dd9af3f40 1 Processor -- start 2024-04-02T13:31:59.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.091+0000 7f0dd9af3f40 1 -- start start 2024-04-02T13:31:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd9af3f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f1872beb0 0x564f187278d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd9af3f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f187b84e0 0x564f187a8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd9af3f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f187be2e0 0x564f18725e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd9af3f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564f187125d0 con 0x564f187be2e0 2024-04-02T13:31:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd9af3f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564f187128b0 con 0x564f1872beb0 2024-04-02T13:31:59.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd9af3f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564f187b8cf0 con 0x564f187b84e0 2024-04-02T13:31:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd7df5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f187be2e0 0x564f18725e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd7df5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f187be2e0 0x564f18725e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50646/0 (socket says 172.21.15.44:50646) 2024-04-02T13:31:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd7df5640 1 -- 172.21.15.44:0/304996453 learned_addr learned my addr 172.21.15.44:0/304996453 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd75f4640 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f1872beb0 0x564f187278d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd6df3640 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f187b84e0 0x564f187a8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd7df5640 1 -- 172.21.15.44:0/304996453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f187b84e0 msgr2=0x564f187a8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd7df5640 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f187b84e0 0x564f187a8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd7df5640 1 -- 172.21.15.44:0/304996453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f1872beb0 msgr2=0x564f187278d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd7df5640 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f1872beb0 0x564f187278d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd7df5640 1 -- 172.21.15.44:0/304996453 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f18726830 con 0x564f187be2e0 2024-04-02T13:31:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd75f4640 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f1872beb0 0x564f187278d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.092+0000 7f0dd6df3640 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564f187b84e0 0x564f187a8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:59.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.093+0000 7f0dd7df5640 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f187be2e0 0x564f18725e50 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f0dd000abd0 tx=0x7f0dd000acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.093+0000 7f0dd4def640 1 -- 172.21.15.44:0/304996453 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd001a070 con 0x564f187be2e0 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.093+0000 7f0dd4def640 1 -- 172.21.15.44:0/304996453 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd00089e0 con 0x564f187be2e0 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.093+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/304996453 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564f18726b10 con 0x564f187be2e0 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.093+0000 7f0dd4def640 1 -- 172.21.15.44:0/304996453 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0008dd0 con 0x564f187be2e0 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.093+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/304996453 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f187270d0 con 0x564f187be2e0 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.094+0000 7f0dd4def640 1 -- 172.21.15.44:0/304996453 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0dd001d070 con 0x564f187be2e0 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.095+0000 7f0dd4def640 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0db806b300 0x7f0db806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.095+0000 7f0dd4def640 1 -- 172.21.15.44:0/304996453 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f0dd0089a40 con 0x564f187be2e0 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.095+0000 7f0dd9af3f40 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x564f18714fe0 0x564f187290f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.095+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/304996453 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:1 18.12 18:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) -- 0x564f18726390 con 0x564f18714fe0 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.095+0000 7f0dd75f4640 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0db806b300 0x7f0db806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.095+0000 7f0dd6df3640 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x564f18714fe0 0x564f187290f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.095+0000 7f0dd6df3640 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x564f18714fe0 0x564f187290f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.095+0000 7f0dd75f4640 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0db806b300 0x7f0db806d7c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800a190 tx=0x7f0dc8007030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.096+0000 7f0dd6df3640 1 -- 172.21.15.44:0/304996453 <== osd.2 v2:172.21.15.44:6816/2091834331 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr (12) out=12b] v0'0 uv22 ondisk = 0) ==== 164+0+12 (crc 0 0 0) 0x7f0dc000e040 con 0x564f18714fe0 2024-04-02T13:31:59.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.096+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/304996453 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x564f18714fe0 msgr2=0x564f187290f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.096+0000 7f0dd9af3f40 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x564f18714fe0 0x564f187290f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.096+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/304996453 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0db806b300 msgr2=0x7f0db806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.096+0000 7f0dd9af3f40 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0db806b300 0x7f0db806d7c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800a190 tx=0x7f0dc8007030 comp rx=0 tx=0).stop 2024-04-02T13:31:59.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.096+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/304996453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f187be2e0 msgr2=0x564f18725e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.096+0000 7f0dd9af3f40 1 --2- 172.21.15.44:0/304996453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f187be2e0 0x564f18725e50 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f0dd000abd0 tx=0x7f0dd000acc0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.096+0000 7f0dd7df5640 1 -- 172.21.15.44:0/304996453 reap_dead start 2024-04-02T13:31:59.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.096+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/304996453 shutdown_connections 2024-04-02T13:31:59.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.096+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/304996453 >> 172.21.15.44:0/304996453 conn(0x564f18710b50 msgr2=0x564f1872db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.096+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/304996453 shutdown_connections 2024-04-02T13:31:59.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.096+0000 7f0dd9af3f40 1 -- 172.21.15.44:0/304996453 wait complete. 2024-04-02T13:31:59.105 INFO:teuthology.orchestra.run.smithi044.stdout:mytag123 2024-04-02T13:31:59.105 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2024-04-02T13:31:59.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.184+0000 7fcb61cd7f40 1 Processor -- start 2024-04-02T13:31:59.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.184+0000 7fcb61cd7f40 1 -- start start 2024-04-02T13:31:59.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb61cd7f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564bd7e684b0 0x564bd7f39460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb61cd7f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564bd7eae560 0x564bd7f24100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb61cd7f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd7f24f80 0x564bd7f40d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb61cd7f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564bd7e69220 con 0x564bd7e684b0 2024-04-02T13:31:59.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb61cd7f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564bd7e0e3e0 con 0x564bd7f24f80 2024-04-02T13:31:59.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb61cd7f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564bd7e68920 con 0x564bd7eae560 2024-04-02T13:31:59.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5f7d8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564bd7e684b0 0x564bd7f39460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5f7d8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564bd7e684b0 0x564bd7f39460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50664/0 (socket says 172.21.15.44:50664) 2024-04-02T13:31:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5f7d8640 1 -- 172.21.15.44:0/3876518565 learned_addr learned my addr 172.21.15.44:0/3876518565 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5f7d8640 1 -- 172.21.15.44:0/3876518565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564bd7eae560 msgr2=0x564bd7f24100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5efd7640 1 --2- 172.21.15.44:0/3876518565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564bd7eae560 0x564bd7f24100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5f7d8640 1 --2- 172.21.15.44:0/3876518565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564bd7eae560 0x564bd7f24100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5f7d8640 1 -- 172.21.15.44:0/3876518565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd7f24f80 msgr2=0x564bd7f40d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5ffd9640 1 --2- 172.21.15.44:0/3876518565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd7f24f80 0x564bd7f40d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5f7d8640 1 --2- 172.21.15.44:0/3876518565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd7f24f80 0x564bd7f40d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5f7d8640 1 -- 172.21.15.44:0/3876518565 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bd7f224e0 con 0x564bd7e684b0 2024-04-02T13:31:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5ffd9640 1 --2- 172.21.15.44:0/3876518565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd7f24f80 0x564bd7f40d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5f7d8640 1 --2- 172.21.15.44:0/3876518565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564bd7e684b0 0x564bd7f39460 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fcb5801cf00 tx=0x7fcb5804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e509ded641e5a8ee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5dfd5640 1 -- 172.21.15.44:0/3876518565 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58057070 con 0x564bd7e684b0 2024-04-02T13:31:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5dfd5640 1 -- 172.21.15.44:0/3876518565 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb58061070 con 0x564bd7e684b0 2024-04-02T13:31:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.185+0000 7fcb5dfd5640 1 -- 172.21.15.44:0/3876518565 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58052b50 con 0x564bd7e684b0 2024-04-02T13:31:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.186+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/3876518565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564bd7e684b0 msgr2=0x564bd7f39460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.186+0000 7fcb61cd7f40 1 --2- 172.21.15.44:0/3876518565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564bd7e684b0 0x564bd7f39460 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fcb5801cf00 tx=0x7fcb5804c000 comp rx=0 tx=0).stop 2024-04-02T13:31:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.186+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/3876518565 shutdown_connections 2024-04-02T13:31:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.186+0000 7fcb61cd7f40 1 --2- 172.21.15.44:0/3876518565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd7f24f80 0x564bd7f40d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.186+0000 7fcb61cd7f40 1 --2- 172.21.15.44:0/3876518565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564bd7eae560 0x564bd7f24100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.186+0000 7fcb61cd7f40 1 --2- 172.21.15.44:0/3876518565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564bd7e684b0 0x564bd7f39460 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.186+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/3876518565 >> 172.21.15.44:0/3876518565 conn(0x564bd7e93c80 msgr2=0x564bd7f3e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.186+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/3876518565 shutdown_connections 2024-04-02T13:31:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.186+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/3876518565 wait complete. 2024-04-02T13:31:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.186+0000 7fcb61cd7f40 1 Processor -- start 2024-04-02T13:31:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.186+0000 7fcb61cd7f40 1 -- start start 2024-04-02T13:31:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.187+0000 7fcb61cd7f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd7e684b0 0x564bd7f1b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.187+0000 7fcb61cd7f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564bd7eae560 0x564bd7f40d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.187+0000 7fcb61cd7f40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564bd7f24f80 0x564bd7f412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.187+0000 7fcb61cd7f40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564bd7f41e40 con 0x564bd7f24f80 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.187+0000 7fcb61cd7f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564bd7eaaf10 con 0x564bd7e684b0 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.187+0000 7fcb61cd7f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x564bd7eaaa80 con 0x564bd7eae560 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.187+0000 7fcb5ffd9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564bd7f24f80 0x564bd7f412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.187+0000 7fcb5ffd9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564bd7f24f80 0x564bd7f412d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50666/0 (socket says 172.21.15.44:50666) 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.187+0000 7fcb5ffd9640 1 -- 172.21.15.44:0/2651259676 learned_addr learned my addr 172.21.15.44:0/2651259676 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.187+0000 7fcb5f7d8640 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd7e684b0 0x564bd7f1b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.187+0000 7fcb5efd7640 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564bd7eae560 0x564bd7f40d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.187+0000 7fcb5ffd9640 1 -- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564bd7eae560 msgr2=0x564bd7f40d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.188+0000 7fcb5ffd9640 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564bd7eae560 0x564bd7f40d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.188+0000 7fcb5ffd9640 1 -- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd7e684b0 msgr2=0x564bd7f1b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.188+0000 7fcb5ffd9640 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd7e684b0 0x564bd7f1b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.188+0000 7fcb5ffd9640 1 -- 172.21.15.44:0/2651259676 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bd7ead830 con 0x564bd7f24f80 2024-04-02T13:31:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.188+0000 7fcb5ffd9640 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564bd7f24f80 0x564bd7f412d0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fcb50009100 tx=0x7fcb500095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.188+0000 7fcb5cfd3640 1 -- 172.21.15.44:0/2651259676 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb50014020 con 0x564bd7f24f80 2024-04-02T13:31:59.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.188+0000 7fcb5cfd3640 1 -- 172.21.15.44:0/2651259676 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb50010610 con 0x564bd7f24f80 2024-04-02T13:31:59.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.188+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/2651259676 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564bd7f20720 con 0x564bd7f24f80 2024-04-02T13:31:59.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.188+0000 7fcb5cfd3640 1 -- 172.21.15.44:0/2651259676 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb500109c0 con 0x564bd7f24f80 2024-04-02T13:31:59.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.188+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/2651259676 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bd7eadad0 con 0x564bd7f24f80 2024-04-02T13:31:59.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.189+0000 7fcb5cfd3640 1 -- 172.21.15.44:0/2651259676 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcb50017020 con 0x564bd7f24f80 2024-04-02T13:31:59.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.190+0000 7fcb5cfd3640 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb4006b300 0x7fcb4006d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.190+0000 7fcb5cfd3640 1 -- 172.21.15.44:0/2651259676 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7fcb5008a9c0 con 0x564bd7f24f80 2024-04-02T13:31:59.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.190+0000 7fcb61cd7f40 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x564bd7f3a7b0 0x564bd7f1f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.190+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/2651259676 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:1 18.19 18:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) -- 0x564bd7f1b670 con 0x564bd7f3a7b0 2024-04-02T13:31:59.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.190+0000 7fcb5f7d8640 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb4006b300 0x7fcb4006d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.190+0000 7fcb5efd7640 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x564bd7f3a7b0 0x564bd7f1f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb5f7d8640 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb4006b300 0x7fcb4006d7c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fcb58060000 tx=0x7fcb5804f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb5efd7640 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x564bd7f3a7b0 0x564bd7f1f4f0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb5efd7640 1 -- 172.21.15.44:0/2651259676 <== osd.4 v2:172.21.15.69:6816/1041350007 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (12) out=12b] v0'0 uv23 ondisk = 0) ==== 164+0+12 (crc 0 0 0) 0x7fcb4800b150 con 0x564bd7f3a7b0 2024-04-02T13:31:59.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x564bd7f3a7b0 msgr2=0x564bd7f1f4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb61cd7f40 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x564bd7f3a7b0 0x564bd7f1f4f0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb4006b300 msgr2=0x7fcb4006d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb61cd7f40 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcb4006b300 0x7fcb4006d7c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fcb58060000 tx=0x7fcb5804f000 comp rx=0 tx=0).stop 2024-04-02T13:31:59.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/2651259676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564bd7f24f80 msgr2=0x564bd7f412d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb61cd7f40 1 --2- 172.21.15.44:0/2651259676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564bd7f24f80 0x564bd7f412d0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fcb50009100 tx=0x7fcb500095c0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb5ffd9640 1 -- 172.21.15.44:0/2651259676 reap_dead start 2024-04-02T13:31:59.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/2651259676 shutdown_connections 2024-04-02T13:31:59.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/2651259676 >> 172.21.15.44:0/2651259676 conn(0x564bd7e93c80 msgr2=0x564bd7f389d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/2651259676 shutdown_connections 2024-04-02T13:31:59.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.191+0000 7fcb61cd7f40 1 -- 172.21.15.44:0/2651259676 wait complete. 2024-04-02T13:31:59.200 INFO:teuthology.orchestra.run.smithi044.stdout:mytag123 2024-04-02T13:31:59.200 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000004.00000000 scrub_tag 2024-04-02T13:31:59.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.278+0000 7f98fdd6ff40 1 Processor -- start 2024-04-02T13:31:59.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.278+0000 7f98fdd6ff40 1 -- start start 2024-04-02T13:31:59.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fdd6ff40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578884a91b0 0x5578884d9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fdd6ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578884f3ac0 0x5578884d5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fdd6ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557888579cd0 0x5578884d61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fdd6ff40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5578884f1220 con 0x5578884a91b0 2024-04-02T13:31:59.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fdd6ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5578884513e0 con 0x557888579cd0 2024-04-02T13:31:59.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fdd6ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5578884f14c0 con 0x5578884f3ac0 2024-04-02T13:31:59.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fb870640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578884a91b0 0x5578884d9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fb870640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578884a91b0 0x5578884d9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50674/0 (socket says 172.21.15.44:50674) 2024-04-02T13:31:59.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fb870640 1 -- 172.21.15.44:0/2387206332 learned_addr learned my addr 172.21.15.44:0/2387206332 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fc071640 1 --2- 172.21.15.44:0/2387206332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557888579cd0 0x5578884d61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fb06f640 1 --2- 172.21.15.44:0/2387206332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578884f3ac0 0x5578884d5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fb870640 1 -- 172.21.15.44:0/2387206332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578884f3ac0 msgr2=0x5578884d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fb870640 1 --2- 172.21.15.44:0/2387206332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578884f3ac0 0x5578884d5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fb870640 1 -- 172.21.15.44:0/2387206332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557888579cd0 msgr2=0x5578884d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fb870640 1 --2- 172.21.15.44:0/2387206332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557888579cd0 0x5578884d61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fb870640 1 -- 172.21.15.44:0/2387206332 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55788857f410 con 0x5578884a91b0 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.279+0000 7f98fb870640 1 --2- 172.21.15.44:0/2387206332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578884a91b0 0x5578884d9330 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f98f4055040 tx=0x7f98f404c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34345fe1e435162d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.280+0000 7f98fa06d640 1 -- 172.21.15.44:0/2387206332 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4063070 con 0x5578884a91b0 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.280+0000 7f98fa06d640 1 -- 172.21.15.44:0/2387206332 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f4052a10 con 0x5578884a91b0 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.280+0000 7f98fa06d640 1 -- 172.21.15.44:0/2387206332 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4057040 con 0x5578884a91b0 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.280+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/2387206332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578884a91b0 msgr2=0x5578884d9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.280+0000 7f98fdd6ff40 1 --2- 172.21.15.44:0/2387206332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578884a91b0 0x5578884d9330 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f98f4055040 tx=0x7f98f404c000 comp rx=0 tx=0).stop 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.280+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/2387206332 shutdown_connections 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.280+0000 7f98fdd6ff40 1 --2- 172.21.15.44:0/2387206332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557888579cd0 0x5578884d61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.280+0000 7f98fdd6ff40 1 --2- 172.21.15.44:0/2387206332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5578884f3ac0 0x5578884d5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.280+0000 7f98fdd6ff40 1 --2- 172.21.15.44:0/2387206332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578884a91b0 0x5578884d9330 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.280+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/2387206332 >> 172.21.15.44:0/2387206332 conn(0x5578884aa1c0 msgr2=0x557888576630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.280+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/2387206332 shutdown_connections 2024-04-02T13:31:59.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.280+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/2387206332 wait complete. 2024-04-02T13:31:59.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.281+0000 7f98fdd6ff40 1 Processor -- start 2024-04-02T13:31:59.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.281+0000 7f98fdd6ff40 1 -- start start 2024-04-02T13:31:59.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.281+0000 7f98fdd6ff40 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578884a91b0 0x5578886186d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.281+0000 7f98fdd6ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578884f3ac0 0x557888568570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.281+0000 7f98fdd6ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557888579cd0 0x5578884a8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.281+0000 7f98fdd6ff40 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55788857e400 con 0x5578884a91b0 2024-04-02T13:31:59.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.281+0000 7f98fdd6ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5578884d53c0 con 0x5578884f3ac0 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.281+0000 7f98fdd6ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55788857e6e0 con 0x557888579cd0 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98fb870640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578884a91b0 0x5578886186d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98fb870640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578884a91b0 0x5578886186d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50686/0 (socket says 172.21.15.44:50686) 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98fb870640 1 -- 172.21.15.44:0/650265848 learned_addr learned my addr 172.21.15.44:0/650265848 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98fb06f640 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578884f3ac0 0x557888568570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98fc071640 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557888579cd0 0x5578884a8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98fb870640 1 -- 172.21.15.44:0/650265848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557888579cd0 msgr2=0x5578884a8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98fb870640 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557888579cd0 0x5578884a8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98fb870640 1 -- 172.21.15.44:0/650265848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578884f3ac0 msgr2=0x557888568570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98fb870640 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578884f3ac0 0x557888568570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98fb870640 1 -- 172.21.15.44:0/650265848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557888611d10 con 0x5578884a91b0 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98fb870640 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578884a91b0 0x5578886186d0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f98f401cf60 tx=0x7f98f4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98f906b640 1 -- 172.21.15.44:0/650265848 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f406c070 con 0x5578884a91b0 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98f906b640 1 -- 172.21.15.44:0/650265848 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f4052dc0 con 0x5578884a91b0 2024-04-02T13:31:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/650265848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557888611ff0 con 0x5578884a91b0 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98f906b640 1 -- 172.21.15.44:0/650265848 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4063040 con 0x5578884a91b0 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.282+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/650265848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578886125b0 con 0x5578884a91b0 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.284+0000 7f98f906b640 1 -- 172.21.15.44:0/650265848 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f98f406f070 con 0x5578884a91b0 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.284+0000 7f98f906b640 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98dc06b2d0 0x7f98dc06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.284+0000 7f98f906b640 1 -- 172.21.15.44:0/650265848 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f98f40ccb80 con 0x5578884a91b0 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.284+0000 7f98fb06f640 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98dc06b2d0 0x7f98dc06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.284+0000 7f98fdd6ff40 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55788857ceb0 0x557888612ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.284+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/650265848 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:1 18.5 18:a367a3e8:::10000000004.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) -- 0x55788857d290 con 0x55788857ceb0 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.284+0000 7f98fc071640 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55788857ceb0 0x557888612ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fb06f640 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98dc06b2d0 0x7f98dc06d790 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f98e4000c00 tx=0x7f98e4003820 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fc071640 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55788857ceb0 0x557888612ad0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fc071640 1 -- 172.21.15.44:0/650265848 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(1 10000000004.00000000 [getxattr (12) out=12b] v0'0 uv12 ondisk = 0) ==== 164+0+12 (crc 0 0 0) 0x7f98ec00ab40 con 0x55788857ceb0 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/650265848 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55788857ceb0 msgr2=0x557888612ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fdd6ff40 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55788857ceb0 0x557888612ad0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/650265848 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98dc06b2d0 msgr2=0x7f98dc06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fdd6ff40 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f98dc06b2d0 0x7f98dc06d790 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f98e4000c00 tx=0x7f98e4003820 comp rx=0 tx=0).stop 2024-04-02T13:31:59.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/650265848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578884a91b0 msgr2=0x5578886186d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fdd6ff40 1 --2- 172.21.15.44:0/650265848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5578884a91b0 0x5578886186d0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f98f401cf60 tx=0x7f98f4002fb0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fc071640 1 -- 172.21.15.44:0/650265848 reap_dead start 2024-04-02T13:31:59.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/650265848 shutdown_connections 2024-04-02T13:31:59.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/650265848 >> 172.21.15.44:0/650265848 conn(0x5578884aa1c0 msgr2=0x557888578b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/650265848 shutdown_connections 2024-04-02T13:31:59.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.285+0000 7f98fdd6ff40 1 -- 172.21.15.44:0/650265848 wait complete. 2024-04-02T13:31:59.294 INFO:teuthology.orchestra.run.smithi044.stdout:mytag123 2024-04-02T13:31:59.294 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 flush journal 2024-04-02T13:31:59.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.435+0000 7f95e5fb5640 1 Processor -- start 2024-04-02T13:31:59.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.435+0000 7f95e5fb5640 1 -- start start 2024-04-02T13:31:59.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.435+0000 7f95e5fb5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f95e00d0a30 0x7f95e00b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.435+0000 7f95e5fb5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f95e0144130 0x7f95e015e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.435+0000 7f95e5fb5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e00d0e10 con 0x7f95e00b3990 2024-04-02T13:31:59.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.435+0000 7f95e5fb5640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f95e0161e10 con 0x7f95e00d0a30 2024-04-02T13:31:59.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.435+0000 7f95e5fb5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f95e0144750 con 0x7f95e0144130 2024-04-02T13:31:59.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.436+0000 7f95df7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f95e00d0a30 0x7f95e00b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.436+0000 7f95d6ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f95e00b3990 0x7f95e0143a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:60786/0 (socket says 172.21.15.44:60786) 2024-04-02T13:31:59.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.436+0000 7f95d6ffd640 1 -- 172.21.15.44:0/2190913390 learned_addr learned my addr 172.21.15.44:0/2190913390 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:31:59.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.436+0000 7f95df7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f95e00d0a30 0x7f95e00b3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50694/0 (socket says 172.21.15.44:50694) 2024-04-02T13:31:59.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.436+0000 7f95dffff640 1 --2- 172.21.15.44:0/2190913390 >> v2:172.21.15.69:3300/0 conn(0x7f95e0144130 0x7f95e015e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.436+0000 7f95df7fe640 1 -- 172.21.15.44:0/2190913390 >> v2:172.21.15.69:3300/0 conn(0x7f95e0144130 msgr2=0x7f95e015e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.436+0000 7f95df7fe640 1 --2- 172.21.15.44:0/2190913390 >> v2:172.21.15.69:3300/0 conn(0x7f95e0144130 0x7f95e015e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.436+0000 7f95df7fe640 1 -- 172.21.15.44:0/2190913390 >> v1:172.21.15.44:6789/0 conn(0x7f95e00b3990 legacy=0x7f95e0143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.436+0000 7f95df7fe640 1 -- 172.21.15.44:0/2190913390 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e0162060 con 0x7f95e00d0a30 2024-04-02T13:31:59.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.436+0000 7f95deffd640 1 -- 172.21.15.44:0/2190913390 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2911835323 0 0) 0x7f95c8001110 con 0x7f95e00b3990 2024-04-02T13:31:59.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.436+0000 7f95dffff640 1 --2- 172.21.15.44:0/2190913390 >> v2:172.21.15.69:3300/0 conn(0x7f95e0144130 0x7f95e015e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:59.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.437+0000 7f95df7fe640 1 --2- 172.21.15.44:0/2190913390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e00d0a30 0x7f95e00b3450 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f95e00b5080 tx=0x7f95d002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9dd7f4372290f71 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.437+0000 7f95deffd640 1 -- 172.21.15.44:0/2190913390 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d00315a0 con 0x7f95e00d0a30 2024-04-02T13:31:59.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.437+0000 7f95deffd640 1 -- 172.21.15.44:0/2190913390 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d0037070 con 0x7f95e00d0a30 2024-04-02T13:31:59.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.437+0000 7f95deffd640 1 -- 172.21.15.44:0/2190913390 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d0031960 con 0x7f95e00d0a30 2024-04-02T13:31:59.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.437+0000 7f95e5fb5640 1 -- 172.21.15.44:0/2190913390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e00d0a30 msgr2=0x7f95e00b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.437+0000 7f95e5fb5640 1 --2- 172.21.15.44:0/2190913390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e00d0a30 0x7f95e00b3450 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f95e00b5080 tx=0x7f95d002b840 comp rx=0 tx=0).stop 2024-04-02T13:31:59.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.437+0000 7f95e5fb5640 1 -- 172.21.15.44:0/2190913390 shutdown_connections 2024-04-02T13:31:59.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.437+0000 7f95e5fb5640 1 --2- 172.21.15.44:0/2190913390 >> v2:172.21.15.69:3300/0 conn(0x7f95e0144130 0x7f95e015e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.437+0000 7f95e5fb5640 1 --2- 172.21.15.44:0/2190913390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e00d0a30 0x7f95e00b3450 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.437+0000 7f95e5fb5640 1 -- 172.21.15.44:0/2190913390 >> 172.21.15.44:0/2190913390 conn(0x7f95e00b5d90 msgr2=0x7f95e006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.437+0000 7f95e5fb5640 1 -- 172.21.15.44:0/2190913390 shutdown_connections 2024-04-02T13:31:59.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.437+0000 7f95e5fb5640 1 -- 172.21.15.44:0/2190913390 wait complete. 2024-04-02T13:31:59.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.439+0000 7f95e5fb5640 1 Processor -- start 2024-04-02T13:31:59.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.439+0000 7f95e5fb5640 1 -- start start 2024-04-02T13:31:59.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.439+0000 7f95e5fb5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e00b3990 0x7f95e0157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.439+0000 7f95e5fb5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e00d0a30 0x7f95e01453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.439+0000 7f95e5fb5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95e0144130 0x7f95e006e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.439+0000 7f95e5fb5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f95e0162b20 con 0x7f95e00b3990 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.439+0000 7f95e5fb5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f95e0162620 con 0x7f95e0144130 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.439+0000 7f95e5fb5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f95e0162ca0 con 0x7f95e00d0a30 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e00b3990 0x7f95e0157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e00b3990 0x7f95e0157930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50708/0 (socket says 172.21.15.44:50708) 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95df7fe640 1 -- 172.21.15.44:0/3475235551 learned_addr learned my addr 172.21.15.44:0/3475235551 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95dffff640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95e0144130 0x7f95e006e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95d6ffd640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e00d0a30 0x7f95e01453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95df7fe640 1 -- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e00d0a30 msgr2=0x7f95e01453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95df7fe640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e00d0a30 0x7f95e01453d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95df7fe640 1 -- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95e0144130 msgr2=0x7f95e006e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95df7fe640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95e0144130 0x7f95e006e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95df7fe640 1 -- 172.21.15.44:0/3475235551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e0141c70 con 0x7f95e00b3990 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95dffff640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95e0144130 0x7f95e006e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95d6ffd640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e00d0a30 0x7f95e01453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95df7fe640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e00b3990 0x7f95e0157930 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f95d0034040 tx=0x7f95d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95dcff9640 1 -- 172.21.15.44:0/3475235551 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d0004230 con 0x7f95e00b3990 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95dcff9640 1 -- 172.21.15.44:0/3475235551 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d00043d0 con 0x7f95e00b3990 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95e5fb5640 1 -- 172.21.15.44:0/3475235551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e0141f50 con 0x7f95e00b3990 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95dcff9640 1 -- 172.21.15.44:0/3475235551 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d0003850 con 0x7f95e00b3990 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.440+0000 7f95e5fb5640 1 -- 172.21.15.44:0/3475235551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e0142510 con 0x7f95e00b3990 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.442+0000 7f95dcff9640 1 -- 172.21.15.44:0/3475235551 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f95d00039f0 con 0x7f95e00b3990 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.442+0000 7f95dcff9640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f95c006c8f0 0x7f95c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.442+0000 7f95dcff9640 1 -- 172.21.15.44:0/3475235551 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f95d002f030 con 0x7f95e00b3990 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.442+0000 7f95d6ffd640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f95c006c8f0 0x7f95c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.443+0000 7f95d6ffd640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f95c006c8f0 0x7f95c006edb0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f95c80013b0 tx=0x7f95c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.443+0000 7f95e8b95740 1 Processor -- start 2024-04-02T13:31:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.443+0000 7f95e8b95740 1 -- start start 2024-04-02T13:31:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.444+0000 7f95e8b95740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b749a0 0x55fdb8b47940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.444+0000 7f95e8b95740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b74d80 0x55fdb8b77730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.444+0000 7f95e8b95740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b47e80 0x55fdb8b77c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.444+0000 7f95e8b95740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fdb8ab1830 con 0x55fdb8b74d80 2024-04-02T13:31:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.444+0000 7f95e8b95740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fdb8a59820 con 0x55fdb8b47e80 2024-04-02T13:31:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.444+0000 7f95e8b95740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55fdb8aad7e0 con 0x55fdb8b749a0 2024-04-02T13:31:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.444+0000 7f95d6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b74d80 0x55fdb8b77730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.444+0000 7f95d6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b74d80 0x55fdb8b77730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50712/0 (socket says 172.21.15.44:50712) 2024-04-02T13:31:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.444+0000 7f95d6ffd640 1 -- 172.21.15.44:0/3994230771 learned_addr learned my addr 172.21.15.44:0/3994230771 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.444+0000 7f95dffff640 1 --2- 172.21.15.44:0/3994230771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b47e80 0x55fdb8b77c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.444+0000 7f95df7fe640 1 --2- 172.21.15.44:0/3994230771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b749a0 0x55fdb8b47940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.445+0000 7f95d6ffd640 1 -- 172.21.15.44:0/3994230771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b749a0 msgr2=0x55fdb8b47940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.445+0000 7f95d6ffd640 1 --2- 172.21.15.44:0/3994230771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b749a0 0x55fdb8b47940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.445+0000 7f95d6ffd640 1 -- 172.21.15.44:0/3994230771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b47e80 msgr2=0x55fdb8b77c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.445+0000 7f95d6ffd640 1 --2- 172.21.15.44:0/3994230771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b47e80 0x55fdb8b77c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.445+0000 7f95d6ffd640 1 -- 172.21.15.44:0/3994230771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdb888f7d0 con 0x55fdb8b74d80 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.445+0000 7f95d6ffd640 1 --2- 172.21.15.44:0/3994230771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b74d80 0x55fdb8b77730 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f95c8007390 tx=0x7f95c8010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e6114483585ba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.445+0000 7f95d57fa640 1 -- 172.21.15.44:0/3994230771 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c800d9a0 con 0x55fdb8b74d80 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.445+0000 7f95d57fa640 1 -- 172.21.15.44:0/3994230771 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95c800db40 con 0x55fdb8b74d80 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.445+0000 7f95d57fa640 1 -- 172.21.15.44:0/3994230771 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c8020450 con 0x55fdb8b74d80 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.445+0000 7f95e8b95740 1 -- 172.21.15.44:0/3994230771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b74d80 msgr2=0x55fdb8b77730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.445+0000 7f95e8b95740 1 --2- 172.21.15.44:0/3994230771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b74d80 0x55fdb8b77730 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f95c8007390 tx=0x7f95c8010000 comp rx=0 tx=0).stop 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.446+0000 7f95e8b95740 1 -- 172.21.15.44:0/3994230771 shutdown_connections 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.446+0000 7f95e8b95740 1 --2- 172.21.15.44:0/3994230771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b47e80 0x55fdb8b77c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.446+0000 7f95e8b95740 1 --2- 172.21.15.44:0/3994230771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b74d80 0x55fdb8b77730 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.446+0000 7f95e8b95740 1 --2- 172.21.15.44:0/3994230771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b749a0 0x55fdb8b47940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.446+0000 7f95e8b95740 1 -- 172.21.15.44:0/3994230771 >> 172.21.15.44:0/3994230771 conn(0x55fdb8b49bb0 msgr2=0x55fdb8b92730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.446+0000 7f95e8b95740 1 -- 172.21.15.44:0/3994230771 shutdown_connections 2024-04-02T13:31:59.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.446+0000 7f95e8b95740 1 -- 172.21.15.44:0/3994230771 wait complete. 2024-04-02T13:31:59.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.446+0000 7f95e8b95740 1 -- start start 2024-04-02T13:31:59.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.446+0000 7f95e8b95740 1 Processor -- start 2024-04-02T13:31:59.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.447+0000 7f95e8b95740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b749a0 0x55fdb8b8ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.447+0000 7f95e8b95740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b74d80 0x55fdb8b67750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.447+0000 7f95e8b95740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b7ad30 0x55fdb8b67c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.447+0000 7f95e8b95740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fdb8b4c150 con 0x55fdb8b7ad30 2024-04-02T13:31:59.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.447+0000 7f95e8b95740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fdb8a6cd30 con 0x55fdb8b749a0 2024-04-02T13:31:59.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.447+0000 7f95e8b95740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55fdb8b4b540 con 0x55fdb8b74d80 2024-04-02T13:31:59.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.447+0000 7f95e8b95740 10 client.0 _ll_register_callbacks cb 0x55fdb8a6c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:31:59.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.447+0000 7f95dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b7ad30 0x55fdb8b67c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.447+0000 7f95dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b7ad30 0x55fdb8b67c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50726/0 (socket says 172.21.15.44:50726) 2024-04-02T13:31:59.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.447+0000 7f95dffff640 1 -- 172.21.15.44:0/711058547 learned_addr learned my addr 172.21.15.44:0/711058547 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.447+0000 7f95df7fe640 1 --2- 172.21.15.44:0/711058547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b749a0 0x55fdb8b8ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.447+0000 7f95d6ffd640 1 --2- 172.21.15.44:0/711058547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b74d80 0x55fdb8b67750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95dffff640 1 -- 172.21.15.44:0/711058547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b74d80 msgr2=0x55fdb8b67750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95dffff640 1 --2- 172.21.15.44:0/711058547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b74d80 0x55fdb8b67750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95dffff640 1 -- 172.21.15.44:0/711058547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b749a0 msgr2=0x55fdb8b8ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95dffff640 1 --2- 172.21.15.44:0/711058547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b749a0 0x55fdb8b8ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95dffff640 1 -- 172.21.15.44:0/711058547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdb87c53f0 con 0x55fdb8b7ad30 2024-04-02T13:31:59.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95dffff640 1 --2- 172.21.15.44:0/711058547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b7ad30 0x55fdb8b67c90 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f95d8007b60 tx=0x7f95d8007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81908c106b76fd11 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95be7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:31:59.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95be7fc640 1 -- 172.21.15.44:0/711058547 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d801b030 con 0x55fdb8b7ad30 2024-04-02T13:31:59.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95be7fc640 1 -- 172.21.15.44:0/711058547 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f95d8011c30 con 0x55fdb8b7ad30 2024-04-02T13:31:59.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95e8b95740 1 -- 172.21.15.44:0/711058547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95c800db40 con 0x55fdb8b7ad30 2024-04-02T13:31:59.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95be7fc640 1 -- 172.21.15.44:0/711058547 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f95d8024050 con 0x55fdb8b7ad30 2024-04-02T13:31:59.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95e8b95740 10 client.10056 fetch_fsmap learned FSMap version 105 2024-04-02T13:31:59.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95e8b95740 1 -- 172.21.15.44:0/711058547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x55fdb87850f0 con 0x55fdb8b7ad30 2024-04-02T13:31:59.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95be7fc640 1 -- 172.21.15.44:0/711058547 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95c800db40 con 0x55fdb8b7ad30 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95be7fc640 1 -- 172.21.15.44:0/711058547 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 105) ==== 3494+0+0 (secure 0 0 0) 0x7f95d8027020 con 0x55fdb8b7ad30 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95e8b95740 10 client.10056 fetch_fsmap finished waiting for FSMap version 105 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95e8b95740 10 client.10056 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.a 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.448+0000 7f95e8b95740 20 client.10056 populate_metadata read hostname 'smithi044' 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.449+0000 7f95e8b95740 1 --2- 172.21.15.44:0/711058547 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x55fdb8ba27a0 0x55fdb8ba2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.449+0000 7f95e8b95740 4 client.10056 mds_command: new command op to 9855 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.449+0000 7f95e8b95740 1 -- 172.21.15.44:0/711058547 --> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdb8abed70 con 0x55fdb8ba27a0 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.449+0000 7f95df7fe640 1 --2- 172.21.15.44:0/711058547 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x55fdb8ba27a0 0x55fdb8ba2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.449+0000 7f95be7fc640 10 client.10056 ms_handle_connect on v2:172.21.15.44:6833/2895101355 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.449+0000 7f95df7fe640 1 --2- 172.21.15.44:0/711058547 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x55fdb8ba27a0 0x55fdb8ba2bd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8775ea44fef95d70 server_cookie=9ab51664e2d0926e in_seq=0 out_seq=0 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.449+0000 7f95be7fc640 1 -- 172.21.15.44:0/711058547 <== mds.0 v2:172.21.15.44:6833/2895101355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fdb8abed70 con 0x55fdb8ba27a0 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.449+0000 7f95be7fc640 10 client.10056 handle_command_reply: tid=0 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.449+0000 7f95e8b95740 1 client.10056 shutdown 2024-04-02T13:31:59.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.450+0000 7f95e8b95740 1 -- 172.21.15.44:0/711058547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b7ad30 msgr2=0x55fdb8b67c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.450+0000 7f95e8b95740 1 --2- 172.21.15.44:0/711058547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b7ad30 0x55fdb8b67c90 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f95d8007b60 tx=0x7f95d8007f60 comp rx=0 tx=0).stop 2024-04-02T13:31:59.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.450+0000 7f95e8b95740 1 -- 172.21.15.44:0/711058547 shutdown_connections 2024-04-02T13:31:59.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.450+0000 7f95e8b95740 1 --2- 172.21.15.44:0/711058547 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x55fdb8ba27a0 0x55fdb8ba2bd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.450+0000 7f95e8b95740 1 --2- 172.21.15.44:0/711058547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b7ad30 0x55fdb8b67c90 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.450+0000 7f95be7fc640 0 client.10056 ms_handle_reset on v2:172.21.15.44:6833/2895101355 2024-04-02T13:31:59.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.450+0000 7f95e8b95740 1 --2- 172.21.15.44:0/711058547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b74d80 0x55fdb8b67750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.450+0000 7f95e8b95740 1 --2- 172.21.15.44:0/711058547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b749a0 0x55fdb8b8ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.450+0000 7f95e8b95740 1 -- 172.21.15.44:0/711058547 >> 172.21.15.44:0/711058547 conn(0x55fdb8b47e80 msgr2=0x55fdb8b77720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.450+0000 7f95e8b95740 1 -- 172.21.15.44:0/711058547 shutdown_connections 2024-04-02T13:31:59.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.450+0000 7f95e8b95740 1 -- 172.21.15.44:0/711058547 wait complete. 2024-04-02T13:31:59.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.450+0000 7f95e8b95740 20 client.10056 trim_cache size 0 max 16384 2024-04-02T13:31:59.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.465+0000 7f95e8b95740 1 Processor -- start 2024-04-02T13:31:59.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.465+0000 7f95e8b95740 1 -- start start 2024-04-02T13:31:59.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.465+0000 7f95e8b95740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 0x55fdb8b92a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.465+0000 7f95e8b95740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b74d80 0x55fdb8bb6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.465+0000 7f95e8b95740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b7ad30 0x55fdb8b71950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.465+0000 7f95e8b95740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f95e0201010 con 0x55fdb8b749a0 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.465+0000 7f95e8b95740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f95e0200050 con 0x55fdb8b7ad30 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.465+0000 7f95e8b95740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f95e0143ce0 con 0x55fdb8b74d80 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.465+0000 7f95df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 0x55fdb8b92a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.466+0000 7f95df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 0x55fdb8b92a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50734/0 (socket says 172.21.15.44:50734) 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.466+0000 7f95df7fe640 1 -- 172.21.15.44:0/464350235 learned_addr learned my addr 172.21.15.44:0/464350235 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.466+0000 7f95df7fe640 1 -- 172.21.15.44:0/464350235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b74d80 msgr2=0x55fdb8bb6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.466+0000 7f95df7fe640 1 --2- 172.21.15.44:0/464350235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b74d80 0x55fdb8bb6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.466+0000 7f95df7fe640 1 -- 172.21.15.44:0/464350235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b7ad30 msgr2=0x55fdb8b71950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.466+0000 7f95df7fe640 1 --2- 172.21.15.44:0/464350235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b7ad30 0x55fdb8b71950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.466+0000 7f95df7fe640 1 -- 172.21.15.44:0/464350235 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdb8b79070 con 0x55fdb8b749a0 2024-04-02T13:31:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.466+0000 7f95df7fe640 1 --2- 172.21.15.44:0/464350235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 0x55fdb8b92a30 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f95d00ac0b0 tx=0x7f95d007e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160b1d600f3ef603 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.466+0000 7f95d4ff9640 1 -- 172.21.15.44:0/464350235 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d0097070 con 0x55fdb8b749a0 2024-04-02T13:31:59.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.466+0000 7f95d4ff9640 1 -- 172.21.15.44:0/464350235 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d0037040 con 0x55fdb8b749a0 2024-04-02T13:31:59.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.466+0000 7f95d4ff9640 1 -- 172.21.15.44:0/464350235 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d007b820 con 0x55fdb8b749a0 2024-04-02T13:31:59.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.467+0000 7f95e8b95740 1 -- 172.21.15.44:0/464350235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 msgr2=0x55fdb8b92a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.467+0000 7f95e8b95740 1 --2- 172.21.15.44:0/464350235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 0x55fdb8b92a30 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f95d00ac0b0 tx=0x7f95d007e040 comp rx=0 tx=0).stop 2024-04-02T13:31:59.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.467+0000 7f95e8b95740 1 -- 172.21.15.44:0/464350235 shutdown_connections 2024-04-02T13:31:59.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.467+0000 7f95e8b95740 1 --2- 172.21.15.44:0/464350235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b7ad30 0x55fdb8b71950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.467+0000 7f95e8b95740 1 --2- 172.21.15.44:0/464350235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b74d80 0x55fdb8bb6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.467+0000 7f95e8b95740 1 --2- 172.21.15.44:0/464350235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 0x55fdb8b92a30 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.467+0000 7f95e8b95740 1 -- 172.21.15.44:0/464350235 >> 172.21.15.44:0/464350235 conn(0x55fdb8b47e80 msgr2=0x55fdb8b43140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.467+0000 7f95e8b95740 1 -- 172.21.15.44:0/464350235 shutdown_connections 2024-04-02T13:31:59.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.467+0000 7f95e8b95740 1 -- 172.21.15.44:0/464350235 wait complete. 2024-04-02T13:31:59.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.467+0000 7f95e8b95740 1 -- start start 2024-04-02T13:31:59.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.467+0000 7f95e8b95740 1 Processor -- start 2024-04-02T13:31:59.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95e8b95740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 0x55fdb8c0f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95e8b95740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b74d80 0x55fdb8b739a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95e8b95740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b7ad30 0x55fdb8b73ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95e8b95740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fdb8ba2c30 con 0x55fdb8b749a0 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95e8b95740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f95e01ffa00 con 0x55fdb8b74d80 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95e8b95740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55fdb8b8b4b0 con 0x55fdb8b7ad30 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95e8b95740 10 client.0 _ll_register_callbacks cb 0x7f95d8007710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 0x55fdb8c0f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 0x55fdb8c0f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50742/0 (socket says 172.21.15.44:50742) 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95df7fe640 1 -- 172.21.15.44:0/4157650359 learned_addr learned my addr 172.21.15.44:0/4157650359 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95d6ffd640 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b74d80 0x55fdb8b739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95dffff640 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b7ad30 0x55fdb8b73ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95df7fe640 1 -- 172.21.15.44:0/4157650359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b7ad30 msgr2=0x55fdb8b73ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95df7fe640 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b7ad30 0x55fdb8b73ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95df7fe640 1 -- 172.21.15.44:0/4157650359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b74d80 msgr2=0x55fdb8b739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95df7fe640 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b74d80 0x55fdb8b739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95df7fe640 1 -- 172.21.15.44:0/4157650359 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdb8bac730 con 0x55fdb8b749a0 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.468+0000 7f95d6ffd640 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b74d80 0x55fdb8b739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:59.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95df7fe640 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 0x55fdb8c0f280 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f95d0042fd0 tx=0x7f95d8007470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c9bcfc6a09a1aa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95be7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95be7fc640 1 -- 172.21.15.44:0/4157650359 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d0089060 con 0x55fdb8b749a0 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95be7fc640 1 -- 172.21.15.44:0/4157650359 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f95d0079b20 con 0x55fdb8b749a0 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95e8b95740 1 -- 172.21.15.44:0/4157650359 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95d0037040 con 0x55fdb8b749a0 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95be7fc640 1 -- 172.21.15.44:0/4157650359 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f95d007b820 con 0x55fdb8b749a0 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95e8b95740 10 client.10062 fetch_fsmap learned FSMap version 105 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95e8b95740 1 -- 172.21.15.44:0/4157650359 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x55fdb8b74420 con 0x55fdb8b749a0 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95be7fc640 1 -- 172.21.15.44:0/4157650359 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d0037040 con 0x55fdb8b749a0 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95be7fc640 1 -- 172.21.15.44:0/4157650359 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 105) ==== 3494+0+0 (secure 0 0 0) 0x7f95d0031070 con 0x55fdb8b749a0 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95e8b95740 10 client.10062 fetch_fsmap finished waiting for FSMap version 105 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95e8b95740 10 client.10062 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.a 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95e8b95740 20 client.10062 populate_metadata read hostname 'smithi044' 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95e8b95740 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x55fdb8b4ea40 0x55fdb8b91770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95e8b95740 4 client.10062 mds_command: new command op to 9855 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T13:31:59.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95e8b95740 1 -- 172.21.15.44:0/4157650359 --> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fdb8ab90c0 con 0x55fdb8b4ea40 2024-04-02T13:31:59.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.469+0000 7f95d6ffd640 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x55fdb8b4ea40 0x55fdb8b91770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.470+0000 7f95be7fc640 10 client.10062 ms_handle_connect on v2:172.21.15.44:6833/2895101355 2024-04-02T13:31:59.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.470+0000 7f95d6ffd640 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x55fdb8b4ea40 0x55fdb8b91770 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3d8b582018956f5 server_cookie=38278531f495d35a in_seq=0 out_seq=0 2024-04-02T13:31:59.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95be7fc640 1 -- 172.21.15.44:0/4157650359 <== mds.0 v2:172.21.15.44:6833/2895101355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fdb8ab90c0 con 0x55fdb8b4ea40 2024-04-02T13:31:59.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95be7fc640 10 client.10062 handle_command_reply: tid=0 2024-04-02T13:31:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95e8b95740 1 client.10062 shutdown 2024-04-02T13:31:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95e8b95740 1 -- 172.21.15.44:0/4157650359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 msgr2=0x55fdb8c0f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95e8b95740 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 0x55fdb8c0f280 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f95d0042fd0 tx=0x7f95d8007470 comp rx=0 tx=0).stop 2024-04-02T13:31:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95e8b95740 1 -- 172.21.15.44:0/4157650359 shutdown_connections 2024-04-02T13:31:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95e8b95740 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.44:6833/2895101355,v1:172.21.15.44:6835/2895101355] conn(0x55fdb8b4ea40 0x55fdb8b91770 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95e8b95740 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55fdb8b7ad30 0x55fdb8b73ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95e8b95740 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdb8b74d80 0x55fdb8b739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95e8b95740 1 --2- 172.21.15.44:0/4157650359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdb8b749a0 0x55fdb8c0f280 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95be7fc640 0 client.10062 ms_handle_reset on v2:172.21.15.44:6833/2895101355 2024-04-02T13:31:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95e8b95740 1 -- 172.21.15.44:0/4157650359 >> 172.21.15.44:0/4157650359 conn(0x55fdb8b47e80 msgr2=0x55fdb8b8c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95e8b95740 1 -- 172.21.15.44:0/4157650359 shutdown_connections 2024-04-02T13:31:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95e8b95740 1 -- 172.21.15.44:0/4157650359 wait complete. 2024-04-02T13:31:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.483+0000 7f95e8b95740 20 client.10062 trim_cache size 0 max 16384 2024-04-02T13:31:59.489 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:31:59.489 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "", 2024-04-02T13:31:59.489 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0 2024-04-02T13:31:59.489 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:31:59.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.484+0000 7f95e5fb5640 1 -- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f95c006c8f0 msgr2=0x7f95c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.484+0000 7f95e5fb5640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f95c006c8f0 0x7f95c006edb0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f95c80013b0 tx=0x7f95c8009040 comp rx=0 tx=0).stop 2024-04-02T13:31:59.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.484+0000 7f95e5fb5640 1 -- 172.21.15.44:0/3475235551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e00b3990 msgr2=0x7f95e0157930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.484+0000 7f95e5fb5640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e00b3990 0x7f95e0157930 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f95d0034040 tx=0x7f95d00046c0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.484+0000 7f95e5fb5640 1 -- 172.21.15.44:0/3475235551 shutdown_connections 2024-04-02T13:31:59.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.484+0000 7f95e5fb5640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f95c006c8f0 0x7f95c006edb0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.484+0000 7f95e5fb5640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95e0144130 0x7f95e006e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.484+0000 7f95e5fb5640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95e00d0a30 0x7f95e01453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.484+0000 7f95e5fb5640 1 --2- 172.21.15.44:0/3475235551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95e00b3990 0x7f95e0157930 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.484+0000 7f95e5fb5640 1 -- 172.21.15.44:0/3475235551 >> 172.21.15.44:0/3475235551 conn(0x7f95e00b5d90 msgr2=0x7f95e015f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.484+0000 7f95e5fb5640 1 -- 172.21.15.44:0/3475235551 shutdown_connections 2024-04-02T13:31:59.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.484+0000 7f95e5fb5640 1 -- 172.21.15.44:0/3475235551 wait complete. 2024-04-02T13:31:59.498 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:31:59.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.597+0000 7f0bad7af640 1 Processor -- start 2024-04-02T13:31:59.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.597+0000 7f0bad7af640 1 -- start start 2024-04-02T13:31:59.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.597+0000 7f0bad7af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ba815e550 0x7f0ba806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.597+0000 7f0bad7af640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0ba806d530 0x7f0ba815e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0bad7af640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0ba815ee70 0x7f0ba80ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0bad7af640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0ba806d970 con 0x7f0ba806d530 2024-04-02T13:31:59.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0bad7af640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0ba815fb50 con 0x7f0ba815ee70 2024-04-02T13:31:59.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0bad7af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0ba815f300 con 0x7f0ba815e550 2024-04-02T13:31:59.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0b9e7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0ba806d530 0x7f0ba815e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0b9e7fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0ba806d530 0x7f0ba815e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50746/0 (socket says 172.21.15.44:50746) 2024-04-02T13:31:59.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0b9e7fc640 1 -- 172.21.15.44:0/1133874065 learned_addr learned my addr 172.21.15.44:0/1133874065 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0ba77fe640 1 --2- 172.21.15.44:0/1133874065 >> v2:172.21.15.69:3300/0 conn(0x7f0ba815ee70 0x7f0ba80ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0ba6ffd640 1 --2- 172.21.15.44:0/1133874065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ba815e550 0x7f0ba806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0b9e7fc640 1 -- 172.21.15.44:0/1133874065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ba815e550 msgr2=0x7f0ba806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0b9e7fc640 1 --2- 172.21.15.44:0/1133874065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ba815e550 0x7f0ba806cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0b9e7fc640 1 -- 172.21.15.44:0/1133874065 >> v2:172.21.15.69:3300/0 conn(0x7f0ba815ee70 msgr2=0x7f0ba80ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0b9e7fc640 1 --2- 172.21.15.44:0/1133874065 >> v2:172.21.15.69:3300/0 conn(0x7f0ba815ee70 0x7f0ba80ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0b9e7fc640 1 -- 172.21.15.44:0/1133874065 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba806ca90 con 0x7f0ba806d530 2024-04-02T13:31:59.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.598+0000 7f0ba77fe640 1 --2- 172.21.15.44:0/1133874065 >> v2:172.21.15.69:3300/0 conn(0x7f0ba815ee70 0x7f0ba80ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:59.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.599+0000 7f0b9e7fc640 1 --2- 172.21.15.44:0/1133874065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ba806d530 0x7f0ba815e930 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f0ba806e610 tx=0x7f0b9002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f04445b463f7d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.599+0000 7f0ba67fc640 1 -- 172.21.15.44:0/1133874065 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b90034070 con 0x7f0ba806d530 2024-04-02T13:31:59.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.599+0000 7f0ba67fc640 1 -- 172.21.15.44:0/1133874065 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b90030860 con 0x7f0ba806d530 2024-04-02T13:31:59.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.599+0000 7f0ba67fc640 1 -- 172.21.15.44:0/1133874065 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b90030c30 con 0x7f0ba806d530 2024-04-02T13:31:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.599+0000 7f0bad7af640 1 -- 172.21.15.44:0/1133874065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ba806d530 msgr2=0x7f0ba815e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.599+0000 7f0bad7af640 1 --2- 172.21.15.44:0/1133874065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ba806d530 0x7f0ba815e930 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f0ba806e610 tx=0x7f0b9002b9a0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.599+0000 7f0bad7af640 1 -- 172.21.15.44:0/1133874065 shutdown_connections 2024-04-02T13:31:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.599+0000 7f0bad7af640 1 --2- 172.21.15.44:0/1133874065 >> v2:172.21.15.69:3300/0 conn(0x7f0ba815ee70 0x7f0ba80ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.599+0000 7f0bad7af640 1 --2- 172.21.15.44:0/1133874065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ba806d530 0x7f0ba815e930 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.599+0000 7f0bad7af640 1 --2- 172.21.15.44:0/1133874065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ba815e550 0x7f0ba806cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.600+0000 7f0bad7af640 1 -- 172.21.15.44:0/1133874065 >> 172.21.15.44:0/1133874065 conn(0x7f0ba809aa20 msgr2=0x7f0ba809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.600+0000 7f0bad7af640 1 -- 172.21.15.44:0/1133874065 shutdown_connections 2024-04-02T13:31:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.600+0000 7f0bad7af640 1 -- 172.21.15.44:0/1133874065 wait complete. 2024-04-02T13:31:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.601+0000 7f0bad7af640 1 Processor -- start 2024-04-02T13:31:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.601+0000 7f0bad7af640 1 -- start start 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.601+0000 7f0bad7af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ba806d530 0x7f0ba80b7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.601+0000 7f0bad7af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ba815e550 0x7f0ba8148310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.601+0000 7f0bad7af640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ba815ee70 0x7f0ba8148850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.601+0000 7f0bad7af640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba80d02b0 con 0x7f0ba806d530 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.601+0000 7f0bad7af640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba815f870 con 0x7f0ba815ee70 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.601+0000 7f0bad7af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0ba80d0590 con 0x7f0ba815e550 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0ba6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ba806d530 0x7f0ba80b7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0ba6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ba806d530 0x7f0ba80b7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50762/0 (socket says 172.21.15.44:50762) 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0ba6ffd640 1 -- 172.21.15.44:0/3129553111 learned_addr learned my addr 172.21.15.44:0/3129553111 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0ba77fe640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ba815ee70 0x7f0ba8148850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0b9e7fc640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ba815e550 0x7f0ba8148310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0ba6ffd640 1 -- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ba815e550 msgr2=0x7f0ba8148310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0ba6ffd640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ba815e550 0x7f0ba8148310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0ba6ffd640 1 -- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ba815ee70 msgr2=0x7f0ba8148850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0ba6ffd640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ba815ee70 0x7f0ba8148850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0ba6ffd640 1 -- 172.21.15.44:0/3129553111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba815a1e0 con 0x7f0ba806d530 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0b9e7fc640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ba815e550 0x7f0ba8148310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0ba77fe640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ba815ee70 0x7f0ba8148850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0ba6ffd640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ba806d530 0x7f0ba80b7630 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800d5f0 tx=0x7f0b9800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0b9ffff640 1 -- 172.21.15.44:0/3129553111 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b980125f0 con 0x7f0ba806d530 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0b9ffff640 1 -- 172.21.15.44:0/3129553111 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b98019070 con 0x7f0ba806d530 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0b9ffff640 1 -- 172.21.15.44:0/3129553111 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b980129d0 con 0x7f0ba806d530 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0bad7af640 1 -- 172.21.15.44:0/3129553111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba80caac0 con 0x7f0ba806d530 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.602+0000 7f0bad7af640 1 -- 172.21.15.44:0/3129553111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba80cb080 con 0x7f0ba806d530 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.603+0000 7f0bad7af640 1 -- 172.21.15.44:0/3129553111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ba80684f0 con 0x7f0ba806d530 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.604+0000 7f0b9ffff640 1 -- 172.21.15.44:0/3129553111 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0b98012b70 con 0x7f0ba806d530 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.604+0000 7f0b9ffff640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b8806c8f0 0x7f0b8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.604+0000 7f0b9ffff640 1 -- 172.21.15.44:0/3129553111 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6187+0+0 (secure 0 0 0) 0x7f0b9808c0f0 con 0x7f0ba806d530 2024-04-02T13:31:59.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.604+0000 7f0b9e7fc640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b8806c8f0 0x7f0b8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.605+0000 7f0b9e7fc640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b8806c8f0 0x7f0b8806edb0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0b90036000 tx=0x7f0b9003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.609+0000 7f0b9ffff640 1 -- 172.21.15.44:0/3129553111 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0b9805a3d0 con 0x7f0ba806d530 2024-04-02T13:31:59.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.831+0000 7f0bad7af640 1 -- 172.21.15.44:0/3129553111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0ba8070ea0 con 0x7f0ba806d530 2024-04-02T13:31:59.835 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:31:59.832+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:31:59.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.833+0000 7f0b9ffff640 1 -- 172.21.15.44:0/3129553111 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v106) ==== 161+0+0 (secure 0 0 0) 0x7f0b98059cd0 con 0x7f0ba806d530 2024-04-02T13:31:59.837 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:31:59.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.836+0000 7f0bad7af640 1 -- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b8806c8f0 msgr2=0x7f0b8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.836+0000 7f0bad7af640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b8806c8f0 0x7f0b8806edb0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0b90036000 tx=0x7f0b9003b040 comp rx=0 tx=0).stop 2024-04-02T13:31:59.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.836+0000 7f0bad7af640 1 -- 172.21.15.44:0/3129553111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ba806d530 msgr2=0x7f0ba80b7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.836+0000 7f0bad7af640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ba806d530 0x7f0ba80b7630 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800d5f0 tx=0x7f0b9800dab0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.836+0000 7f0bad7af640 1 -- 172.21.15.44:0/3129553111 shutdown_connections 2024-04-02T13:31:59.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.836+0000 7f0bad7af640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0b8806c8f0 0x7f0b8806edb0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.836+0000 7f0bad7af640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ba815ee70 0x7f0ba8148850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.836+0000 7f0bad7af640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ba815e550 0x7f0ba8148310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.836+0000 7f0bad7af640 1 --2- 172.21.15.44:0/3129553111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ba806d530 0x7f0ba80b7630 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.836+0000 7f0bad7af640 1 -- 172.21.15.44:0/3129553111 >> 172.21.15.44:0/3129553111 conn(0x7f0ba809aa20 msgr2=0x7f0ba806f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.836+0000 7f0bad7af640 1 -- 172.21.15.44:0/3129553111 shutdown_connections 2024-04-02T13:31:59.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.836+0000 7f0bad7af640 1 -- 172.21.15.44:0/3129553111 wait complete. 2024-04-02T13:31:59.843 INFO:tasks.ceph.mds.a.smithi044.stderr: -2299> 2024-04-02T13:31:47.111+0000 7f89f9a82640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:31:59.853 DEBUG:teuthology.orchestra.run.smithi044:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset --force 2024-04-02T13:31:59.872 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:31:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.890+0000 7f9a41b60280 1 -- start start 2024-04-02T13:31:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.890+0000 7f9a41b60280 1 Processor -- start 2024-04-02T13:31:59.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.891+0000 7f9a41b60280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ab98c00 0x557b9b95e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.891+0000 7f9a41b60280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ab99800 0x557b9b95e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.891+0000 7f9a41b60280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ab99c00 0x557b9b95eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.891+0000 7f9a41b60280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557b9b968180 con 0x557b9ab99c00 2024-04-02T13:31:59.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.891+0000 7f9a41b60280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557b9b968300 con 0x557b9ab98c00 2024-04-02T13:31:59.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.891+0000 7f9a41b60280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x557b9b968480 con 0x557b9ab99800 2024-04-02T13:31:59.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.891+0000 7f9a40b0f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ab99c00 0x557b9b95eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.891+0000 7f9a4030e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ab98c00 0x557b9b95e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.891+0000 7f9a40b0f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ab99c00 0x557b9b95eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50800/0 (socket says 172.21.15.44:50800) 2024-04-02T13:31:59.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.891+0000 7f9a40b0f640 1 -- 172.21.15.44:0/3813692265 learned_addr learned my addr 172.21.15.44:0/3813692265 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.891+0000 7f9a3fb0d640 1 --2- 172.21.15.44:0/3813692265 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ab99800 0x557b9b95e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a40b0f640 1 -- 172.21.15.44:0/3813692265 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ab99800 msgr2=0x557b9b95e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a40b0f640 1 --2- 172.21.15.44:0/3813692265 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ab99800 0x557b9b95e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a40b0f640 1 -- 172.21.15.44:0/3813692265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ab98c00 msgr2=0x557b9b95e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a40b0f640 1 --2- 172.21.15.44:0/3813692265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ab98c00 0x557b9b95e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a40b0f640 1 -- 172.21.15.44:0/3813692265 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b9abce780 con 0x557b9ab99c00 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a40b0f640 1 --2- 172.21.15.44:0/3813692265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ab99c00 0x557b9b95eb00 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x557b9b9b93e0 tx=0x557b9b92c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec625380a536f86 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a3db09640 1 -- 172.21.15.44:0/3813692265 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557b9abbe680 con 0x557b9ab99c00 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a3db09640 1 -- 172.21.15.44:0/3813692265 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x557b9abc4700 con 0x557b9ab99c00 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a3db09640 1 -- 172.21.15.44:0/3813692265 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557b9abbeb60 con 0x557b9ab99c00 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a41b60280 1 -- 172.21.15.44:0/3813692265 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b9abcf4a0 con 0x557b9ab99c00 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a41b60280 1 -- 172.21.15.44:0/3813692265 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=0}) -- 0x557b9abce000 con 0x557b9ab99c00 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a41b60280 4 waiting for MDS map... 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a3db09640 1 -- 172.21.15.44:0/3813692265 <== mon.0 v2:172.21.15.44:3300/0 4 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x557b9ba6e600 con 0x557b9ab99c00 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a3db09640 1 -- 172.21.15.44:0/3813692265 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x557b9abf3180 con 0x557b9ab99c00 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a41b60280 4 Got MDS map 106 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a41b60280 10 main: JournalTool::main 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.892+0000 7f9a41b60280 4 main: JournalTool: connecting to RADOS... 2024-04-02T13:31:59.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.893+0000 7f9a41b60280 1 Processor -- start 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.893+0000 7f9a41b60280 1 -- start start 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.893+0000 7f9a41b60280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ba67400 0x557b9b95f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.893+0000 7f9a41b60280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 0x557b9b95f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.893+0000 7f9a41b60280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ba67c00 0x557b9b95fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.893+0000 7f9a41b60280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557b9b968600 con 0x557b9ba67800 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.893+0000 7f9a41b60280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557b9b968780 con 0x557b9ba67400 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.893+0000 7f9a41b60280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x557b9b968900 con 0x557b9ba67c00 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.893+0000 7f9a4030e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 0x557b9b95f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.893+0000 7f9a4030e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 0x557b9b95f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50814/0 (socket says 172.21.15.44:50814) 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.893+0000 7f9a4030e640 1 -- 172.21.15.44:0/3442821022 learned_addr learned my addr 172.21.15.44:0/3442821022 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.893+0000 7f9a3fb0d640 1 --2- 172.21.15.44:0/3442821022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ba67400 0x557b9b95f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a3fb0d640 1 --2- 172.21.15.44:0/3442821022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ba67c00 0x557b9b95fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a4030e640 1 -- 172.21.15.44:0/3442821022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ba67c00 msgr2=0x557b9b95fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a4030e640 1 --2- 172.21.15.44:0/3442821022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ba67c00 0x557b9b95fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a4030e640 1 -- 172.21.15.44:0/3442821022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ba67400 msgr2=0x557b9b95f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a4030e640 1 --2- 172.21.15.44:0/3442821022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ba67400 0x557b9b95f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a4030e640 1 -- 172.21.15.44:0/3442821022 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b9abce3c0 con 0x557b9ba67800 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a3fb0d640 1 --2- 172.21.15.44:0/3442821022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ba67400 0x557b9b95f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a4030e640 1 --2- 172.21.15.44:0/3442821022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 0x557b9b95f600 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x557b9b9b8a50 tx=0x557b9b92df20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6ea723a1841c40 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a3bb05640 1 -- 172.21.15.44:0/3442821022 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557b9abbe9c0 con 0x557b9ba67800 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a3bb05640 1 -- 172.21.15.44:0/3442821022 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x557b9abc4fc0 con 0x557b9ba67800 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a3bb05640 1 -- 172.21.15.44:0/3442821022 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557b9abbe680 con 0x557b9ba67800 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a41b60280 1 -- 172.21.15.44:0/3442821022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 msgr2=0x557b9b95f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a41b60280 1 --2- 172.21.15.44:0/3442821022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 0x557b9b95f600 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x557b9b9b8a50 tx=0x557b9b92df20 comp rx=0 tx=0).stop 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a41b60280 1 -- 172.21.15.44:0/3442821022 shutdown_connections 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a41b60280 1 --2- 172.21.15.44:0/3442821022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ba67c00 0x557b9b95fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a41b60280 1 --2- 172.21.15.44:0/3442821022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 0x557b9b95f600 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a41b60280 1 --2- 172.21.15.44:0/3442821022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ba67400 0x557b9b95f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a41b60280 1 -- 172.21.15.44:0/3442821022 >> 172.21.15.44:0/3442821022 conn(0x557b9ba67000 msgr2=0x557b9ac79000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a41b60280 1 -- 172.21.15.44:0/3442821022 shutdown_connections 2024-04-02T13:31:59.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.894+0000 7f9a41b60280 1 -- 172.21.15.44:0/3442821022 wait complete. 2024-04-02T13:31:59.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.895+0000 7f9a41b60280 1 Processor -- start 2024-04-02T13:31:59.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.895+0000 7f9a41b60280 1 -- start start 2024-04-02T13:31:59.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.895+0000 7f9a41b60280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ba67400 0x557b9b95f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.895+0000 7f9a41b60280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 0x557b9b95f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.895+0000 7f9a41b60280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ba67c00 0x557b9b95fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.895+0000 7f9a41b60280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557b9b969980 con 0x557b9ba67800 2024-04-02T13:31:59.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.895+0000 7f9a41b60280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557b9b969800 con 0x557b9ba67400 2024-04-02T13:31:59.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.895+0000 7f9a41b60280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x557b9b969680 con 0x557b9ba67c00 2024-04-02T13:31:59.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a4030e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 0x557b9b95f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a4030e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 0x557b9b95f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50830/0 (socket says 172.21.15.44:50830) 2024-04-02T13:31:59.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a4030e640 1 -- 172.21.15.44:0/582408890 learned_addr learned my addr 172.21.15.44:0/582408890 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a3fb0d640 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ba67400 0x557b9b95f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a3fb0d640 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ba67c00 0x557b9b95fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a4030e640 1 -- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ba67c00 msgr2=0x557b9b95fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a4030e640 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ba67c00 0x557b9b95fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a4030e640 1 -- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ba67400 msgr2=0x557b9b95f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a4030e640 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ba67400 0x557b9b95f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a4030e640 1 -- 172.21.15.44:0/582408890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b9abce780 con 0x557b9ba67800 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a4030e640 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 0x557b9b95f600 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x557b9ba7a7e0 tx=0x557b9b92cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a3a302640 1 -- 172.21.15.44:0/582408890 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557b9abbeb60 con 0x557b9ba67800 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a3a302640 1 -- 172.21.15.44:0/582408890 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x557b9abc4fc0 con 0x557b9ba67800 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a3a302640 1 -- 172.21.15.44:0/582408890 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557b9abbe4e0 con 0x557b9ba67800 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a41b60280 1 -- 172.21.15.44:0/582408890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557b9abcf4a0 con 0x557b9ba67800 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a41b60280 1 -- 172.21.15.44:0/582408890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b9abce960 con 0x557b9ba67800 2024-04-02T13:31:59.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.896+0000 7f9a41b60280 4 main: JournalTool: resolving pool 17 2024-04-02T13:31:59.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.897+0000 7f9a3a302640 1 -- 172.21.15.44:0/582408890 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x557b9badec00 con 0x557b9ba67800 2024-04-02T13:31:59.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.898+0000 7f9a3a302640 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x557b9af04000 0x557b9b960100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.898+0000 7f9a3a302640 1 -- 172.21.15.44:0/582408890 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x557b9baec600 con 0x557b9ba67800 2024-04-02T13:31:59.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.898+0000 7f9a41b60280 4 main: JournalTool: creating IoCtx.. 2024-04-02T13:31:59.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.898+0000 7f9a41b60280 4 main: Executing for rank 0 2024-04-02T13:31:59.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.898+0000 7f9a3fb0d640 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x557b9af04000 0x557b9b960100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.898+0000 7f9a41b60280 1 -- start start 2024-04-02T13:31:59.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.898+0000 7f9a41b60280 1 Processor -- start 2024-04-02T13:31:59.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.898+0000 7f9a3fb0d640 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x557b9af04000 0x557b9b960100 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x557b9b9b92c0 tx=0x557b9ba7d8c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a41b60280 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9af05400 0x557b9b960680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a41b60280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9af05800 0x557b9b960c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a41b60280 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9af05c00 0x557b9b961180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a41b60280 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557b9b969380 con 0x557b9af05400 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a41b60280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557b9b968600 con 0x557b9af05800 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a41b60280 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x557b9b968780 con 0x557b9af05c00 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a40b0f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9af05400 0x557b9b960680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a40b0f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9af05400 0x557b9b960680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50838/0 (socket says 172.21.15.44:50838) 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a40b0f640 1 -- 172.21.15.44:0/1001781853 learned_addr learned my addr 172.21.15.44:0/1001781853 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a4030e640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9af05800 0x557b9b960c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a3fb0d640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9af05c00 0x557b9b961180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a40b0f640 1 -- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9af05c00 msgr2=0x557b9b961180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a40b0f640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9af05c00 0x557b9b961180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a40b0f640 1 -- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9af05800 msgr2=0x557b9b960c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a40b0f640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9af05800 0x557b9b960c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.899+0000 7f9a40b0f640 1 -- 172.21.15.44:0/1001781853 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b9abcf680 con 0x557b9af05400 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.900+0000 7f9a40b0f640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9af05400 0x557b9b960680 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x557b9aebd920 tx=0x557b9b92dec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8830bedecfc55f0e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.900+0000 7f9a372fc640 1 -- 172.21.15.44:0/1001781853 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557b9abbe820 con 0x557b9af05400 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.900+0000 7f9a372fc640 1 -- 172.21.15.44:0/1001781853 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x557b9abc4c40 con 0x557b9af05400 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.900+0000 7f9a372fc640 1 -- 172.21.15.44:0/1001781853 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557b9bb13a00 con 0x557b9af05400 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.900+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b9abcf860 con 0x557b9af05400 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.900+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=0}) -- 0x557b9abcfa40 con 0x557b9af05400 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.900+0000 7f9a41b60280 4 waiting for MDS map... 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.900+0000 7f9a372fc640 1 -- 172.21.15.44:0/1001781853 <== mon.0 v2:172.21.15.44:3300/0 4 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x557b9af70200 con 0x557b9af05400 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.900+0000 7f9a372fc640 1 -- 172.21.15.44:0/1001781853 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x557b9abf3500 con 0x557b9af05400 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.900+0000 7f9a41b60280 4 Got MDS map 106 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.900+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x557b9af73800 0x557b9b961700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.900+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:1 17.f 17:f1f69726:::400.00000000:head [writefull 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x557b9af73c00 con 0x557b9af73800 2024-04-02T13:31:59.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.900+0000 7f9a4030e640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x557b9af73800 0x557b9b961700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.901+0000 7f9a4030e640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x557b9af73800 0x557b9b961700 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=c3493543696e072b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.903+0000 7f9a4030e640 1 -- 172.21.15.44:0/1001781853 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(1 400.00000000 [writefull 0~22] v123'2 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x557b9b908000 con 0x557b9af73800 2024-04-02T13:31:59.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.903+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x557b9af8c000 0x557b9af92000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.903+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:2 17.14 17:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x557b9af8c400 con 0x557b9af8c000 2024-04-02T13:31:59.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.903+0000 7f9a3fb0d640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x557b9af8c000 0x557b9af92000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.904+0000 7f9a3fb0d640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x557b9af8c000 0x557b9af92000 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=e30a0c18451b81f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.908+0000 7f9a3fb0d640 1 -- 172.21.15.44:0/1001781853 <== osd.3 v2:172.21.15.44:6824/1588952513 1 ==== osd_op_reply(2 200.00000000 [writefull 0~90] v123'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x557b9b908500 con 0x557b9af8c000 2024-04-02T13:31:59.915 INFO:teuthology.orchestra.run.smithi044.stdout:writing EResetJournal entry 2024-04-02T13:31:59.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.908+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:3 17.14 17:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x557b9af8c800 con 0x557b9af8c000 2024-04-02T13:31:59.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.908+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x557b9af8cc00 0x557b9af92580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.908+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:4 17.1b 17:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x557b9af8d000 con 0x557b9af8cc00 2024-04-02T13:31:59.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.908+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x557b9af8d400 0x557b9af92b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.908+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:5 17.b 17:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x557b9af8d800 con 0x557b9af8d400 2024-04-02T13:31:59.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.908+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x557b9af8dc00 0x557b9af93080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:31:59.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.908+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:6 17.1f 17:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x557b9af9e000 con 0x557b9af8dc00 2024-04-02T13:31:59.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.909+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:7 17.1a 17:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x557b9af9e400 con 0x557b9af8d400 2024-04-02T13:31:59.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.909+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:8 17.1e 17:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x557b9af9e800 con 0x557b9af73800 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.909+0000 7f9a4030e640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x557b9af8d400 0x557b9af92b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.909+0000 7f9a3fb0d640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x557b9af8dc00 0x557b9af93080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.909+0000 7f9a40b0f640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x557b9af8cc00 0x557b9af92580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.909+0000 7f9a4030e640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x557b9af8d400 0x557b9af92b00 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=62e1ac5b9070529f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.909+0000 7f9a3fb0d640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x557b9af8dc00 0x557b9af93080 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=984beb07df397013 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.910+0000 7f9a3fb0d640 1 -- 172.21.15.44:0/1001781853 <== osd.3 v2:172.21.15.44:6824/1588952513 2 ==== osd_op_reply(3 200.00000001 [delete] v123'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x557b9b908500 con 0x557b9af8c000 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.910+0000 7f9a40b0f640 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x557b9af8cc00 0x557b9af92580 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=c07a254e246b990 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.910+0000 7f9a4030e640 1 -- 172.21.15.44:0/1001781853 <== osd.1 v2:172.21.15.44:6801/704457846 2 ==== osd_op_reply(8 200.00000006 [delete] v123'3 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x557b9b908000 con 0x557b9af73800 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.911+0000 7f9a4030e640 1 -- 172.21.15.44:0/1001781853 <== osd.0 v2:172.21.15.44:6800/2303805525 1 ==== osd_op_reply(5 200.00000003 [delete] v123'6 uv5 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x557b9b908000 con 0x557b9af8d400 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.911+0000 7f9a4030e640 1 -- 172.21.15.44:0/1001781853 <== osd.0 v2:172.21.15.44:6800/2303805525 2 ==== osd_op_reply(7 200.00000005 [delete] v123'3 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x557b9b908000 con 0x557b9af8d400 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.911+0000 7f9a40b0f640 1 -- 172.21.15.44:0/1001781853 <== osd.4 v2:172.21.15.69:6816/1041350007 1 ==== osd_op_reply(4 200.00000002 [delete] v123'3 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x557b9b908780 con 0x557b9af8cc00 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.911+0000 7f9a35af9640 1 -- 172.21.15.44:0/1001781853 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:9 17.14 17:2e2fa760:::200.00000001:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x557b9b97f400 con 0x557b9af8c000 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.912+0000 7f9a3fb0d640 1 -- 172.21.15.44:0/1001781853 <== osd.3 v2:172.21.15.44:6824/1588952513 3 ==== osd_op_reply(9 200.00000001 [write 0~48] v123'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x557b9b908500 con 0x557b9af8c000 2024-04-02T13:31:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.912+0000 7f9a3fb0d640 1 -- 172.21.15.44:0/1001781853 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== osd_op_reply(6 200.00000004 [delete] v123'9 uv8 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x557b9b908500 con 0x557b9af8dc00 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.912+0000 7f9a41b60280 4 Successfully wrote new journal pointer and header for rank 8:0 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.912+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x557b9af8d400 msgr2=0x557b9af92b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.912+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x557b9af8d400 0x557b9af92b00 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.912+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x557b9af73800 msgr2=0x557b9b961700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x557b9af73800 0x557b9b961700 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x557b9af8c000 msgr2=0x557b9af92000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x557b9af8c000 0x557b9af92000 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x557b9af8cc00 msgr2=0x557b9af92580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x557b9af8cc00 0x557b9af92580 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a3fb0d640 1 -- 172.21.15.44:0/1001781853 reap_dead start 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x557b9af8dc00 msgr2=0x557b9af93080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x557b9af8dc00 0x557b9af93080 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9af05400 msgr2=0x557b9b960680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9af05400 0x557b9b960680 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x557b9aebd920 tx=0x557b9b92dec0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 shutdown_connections 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x557b9af8dc00 0x557b9af93080 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x557b9af8cc00 0x557b9af92580 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/1001781853 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9af05400 0x557b9b960680 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 >> 172.21.15.44:0/1001781853 conn(0x557b9af05000 msgr2=0x557b9ac79800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 shutdown_connections 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/1001781853 wait complete. 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x557b9af04000 msgr2=0x557b9b960100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x557b9af04000 0x557b9b960100 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x557b9b9b92c0 tx=0x557b9ba7d8c0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/582408890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 msgr2=0x557b9b95f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 0x557b9b95f600 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x557b9ba7a7e0 tx=0x557b9b92cae0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/582408890 shutdown_connections 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x557b9af04000 0x557b9b960100 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ba67c00 0x557b9b95fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ba67800 0x557b9b95f600 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/582408890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ba67400 0x557b9b95f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/582408890 >> 172.21.15.44:0/582408890 conn(0x557b9ba67000 msgr2=0x557b9ac79000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/582408890 shutdown_connections 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/582408890 wait complete. 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 -- 172.21.15.44:0/3813692265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ab99c00 msgr2=0x557b9b95eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.913+0000 7f9a41b60280 1 --2- 172.21.15.44:0/3813692265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ab99c00 0x557b9b95eb00 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x557b9b9b93e0 tx=0x557b9b92c720 comp rx=0 tx=0).stop 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.914+0000 7f9a41b60280 1 -- 172.21.15.44:0/3813692265 shutdown_connections 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.914+0000 7f9a41b60280 1 --2- 172.21.15.44:0/3813692265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557b9ab99c00 0x557b9b95eb00 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.914+0000 7f9a41b60280 1 --2- 172.21.15.44:0/3813692265 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x557b9ab99800 0x557b9b95e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.914+0000 7f9a41b60280 1 --2- 172.21.15.44:0/3813692265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b9ab98c00 0x557b9b95e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.914+0000 7f9a41b60280 1 -- 172.21.15.44:0/3813692265 >> 172.21.15.44:0/3813692265 conn(0x557b9ab99000 msgr2=0x557b9ac78800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.914+0000 7f9a41b60280 1 -- 172.21.15.44:0/3813692265 shutdown_connections 2024-04-02T13:31:59.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.914+0000 7f9a41b60280 1 -- 172.21.15.44:0/3813692265 wait complete. 2024-04-02T13:31:59.924 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset', '--force'] in time 0:00:00.071456, result: writing EResetJournal entry 2024-04-02T13:31:59.925 DEBUG:teuthology.orchestra.run.smithi044:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents cephfs_data 2024-04-02T13:32:00.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.998+0000 7f2d7c62c740 1 -- start start 2024-04-02T13:32:00.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.999+0000 7f2d7c62c740 1 Processor -- start 2024-04-02T13:32:00.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.999+0000 7f2d7c62c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd8f80c00 0x561dd9d44000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.999+0000 7f2d7c62c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd8f81800 0x561dd9d44580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.999+0000 7f2d7c62c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd8f81c00 0x561dd9d44b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.999+0000 7f2d7c62c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561dd9e46180 con 0x561dd8f80c00 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.999+0000 7f2d7c62c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561dd9e46300 con 0x561dd8f81c00 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.999+0000 7f2d7c62c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x561dd9e46480 con 0x561dd8f81800 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.999+0000 7f2d7a0ae640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd8f80c00 0x561dd9d44000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.999+0000 7f2d7a0ae640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd8f80c00 0x561dd9d44000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50860/0 (socket says 172.21.15.44:50860) 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.999+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/290191999 learned_addr learned my addr 172.21.15.44:0/290191999 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:31:59.999+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/290191999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd8f81c00 0x561dd9d44b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d798ad640 1 --2- 172.21.15.44:0/290191999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd8f81800 0x561dd9d44580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/290191999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd8f81800 msgr2=0x561dd9d44580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d7a0ae640 1 --2- 172.21.15.44:0/290191999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd8f81800 0x561dd9d44580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/290191999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd8f81c00 msgr2=0x561dd9d44b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d7a0ae640 1 --2- 172.21.15.44:0/290191999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd8f81c00 0x561dd9d44b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/290191999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dd8fad680 con 0x561dd8f80c00 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/290191999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd8f81c00 0x561dd9d44b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d7a0ae640 1 --2- 172.21.15.44:0/290191999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd8f80c00 0x561dd9d44000 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x561dd9da5080 tx=0x561dd9d1bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9010366ef08ce867 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d778a9640 1 -- 172.21.15.44:0/290191999 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561dd8fa69c0 con 0x561dd8f80c00 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d778a9640 1 -- 172.21.15.44:0/290191999 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x561dd8fcae00 con 0x561dd8f80c00 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d778a9640 1 -- 172.21.15.44:0/290191999 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561dd8fa6680 con 0x561dd8f80c00 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d7c62c740 1 -- 172.21.15.44:0/290191999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dd8fad4a0 con 0x561dd8f80c00 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d7c62c740 1 -- 172.21.15.44:0/290191999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=0}) -- 0x561dd8fada40 con 0x561dd8f80c00 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d7c62c740 4 waiting for MDS map... 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d778a9640 1 -- 172.21.15.44:0/290191999 <== mon.0 v2:172.21.15.44:3300/0 4 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x561dd9e58200 con 0x561dd8f80c00 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d778a9640 1 -- 172.21.15.44:0/290191999 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x561dd8fd9180 con 0x561dd8f80c00 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d7c62c740 4 Got MDS map 106 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d7c62c740 4 datascan.main: Using metadata pool output 2024-04-02T13:32:00.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.000+0000 7f2d7c62c740 4 datascan.main: connecting to RADOS... 2024-04-02T13:32:00.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.001+0000 7f2d7c62c740 1 Processor -- start 2024-04-02T13:32:00.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.001+0000 7f2d7c62c740 1 -- start start 2024-04-02T13:32:00.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.001+0000 7f2d7c62c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0000 0x561dd9d45080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.001+0000 7f2d7c62c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd9eb0400 0x561dd9d45600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.001+0000 7f2d7c62c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd9eb0800 0x561dd9d45b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.001+0000 7f2d7c62c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561dd9e46600 con 0x561dd9eb0000 2024-04-02T13:32:00.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.001+0000 7f2d7c62c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561dd9e46780 con 0x561dd9eb0800 2024-04-02T13:32:00.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.001+0000 7f2d7c62c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x561dd9e46900 con 0x561dd9eb0400 2024-04-02T13:32:00.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.001+0000 7f2d7a8af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0000 0x561dd9d45080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d7a8af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0000 0x561dd9d45080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50866/0 (socket says 172.21.15.44:50866) 2024-04-02T13:32:00.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3861321629 learned_addr learned my addr 172.21.15.44:0/3861321629 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:00.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d798ad640 1 --2- 172.21.15.44:0/3861321629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd9eb0800 0x561dd9d45b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d7a0ae640 1 --2- 172.21.15.44:0/3861321629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd9eb0400 0x561dd9d45600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3861321629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd9eb0400 msgr2=0x561dd9d45600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/3861321629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd9eb0400 0x561dd9d45600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3861321629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd9eb0800 msgr2=0x561dd9d45b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/3861321629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd9eb0800 0x561dd9d45b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3861321629 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dd8fad860 con 0x561dd9eb0000 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d798ad640 1 --2- 172.21.15.44:0/3861321629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd9eb0800 0x561dd9d45b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/3861321629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0000 0x561dd9d45080 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x561dd9d8acf0 tx=0x561dd9e68cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce59e13f2441bd5d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d758a5640 1 -- 172.21.15.44:0/3861321629 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561dd8fa69c0 con 0x561dd9eb0000 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d758a5640 1 -- 172.21.15.44:0/3861321629 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x561dd8fca540 con 0x561dd9eb0000 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d758a5640 1 -- 172.21.15.44:0/3861321629 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561dd8fa6680 con 0x561dd9eb0000 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3861321629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0000 msgr2=0x561dd9d45080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.002+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3861321629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0000 0x561dd9d45080 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x561dd9d8acf0 tx=0x561dd9e68cc0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.003+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3861321629 shutdown_connections 2024-04-02T13:32:00.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.003+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3861321629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd9eb0800 0x561dd9d45b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.003+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3861321629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd9eb0400 0x561dd9d45600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.003+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3861321629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0000 0x561dd9d45080 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.003+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3861321629 >> 172.21.15.44:0/3861321629 conn(0x561dd9d71c00 msgr2=0x561dd9065000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.003+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3861321629 shutdown_connections 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.003+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3861321629 wait complete. 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.003+0000 7f2d7c62c740 1 Processor -- start 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.003+0000 7f2d7c62c740 1 -- start start 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7c62c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd9eb0000 0x561dd9d45080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7c62c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0400 0x561dd9d45600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7c62c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd9eb0800 0x561dd9d45b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7c62c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561dd9e47c80 con 0x561dd9eb0400 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7c62c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561dd9e47b00 con 0x561dd9eb0800 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7c62c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x561dd9e47980 con 0x561dd9eb0000 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7a0ae640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0400 0x561dd9d45600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7a0ae640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0400 0x561dd9d45600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50878/0 (socket says 172.21.15.44:50878) 2024-04-02T13:32:00.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 learned_addr learned my addr 172.21.15.44:0/3246848592 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:00.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd9eb0000 0x561dd9d45080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d798ad640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd9eb0800 0x561dd9d45b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd9eb0000 msgr2=0x561dd9d45080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7a0ae640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd9eb0000 0x561dd9d45080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd9eb0800 msgr2=0x561dd9d45b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7a0ae640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd9eb0800 0x561dd9d45b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dd8fad2c0 con 0x561dd9eb0400 2024-04-02T13:32:00.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7a0ae640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0400 0x561dd9d45600 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x561dd9da5830 tx=0x561dd9d1a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d740a2640 1 -- 172.21.15.44:0/3246848592 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561dd8fa6b60 con 0x561dd9eb0400 2024-04-02T13:32:00.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d740a2640 1 -- 172.21.15.44:0/3246848592 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x561dd8fcac40 con 0x561dd9eb0400 2024-04-02T13:32:00.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561dd8fad0e0 con 0x561dd9eb0400 2024-04-02T13:32:00.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d740a2640 1 -- 172.21.15.44:0/3246848592 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561dd8fa6820 con 0x561dd9eb0400 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dd8fac3c0 con 0x561dd9eb0400 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.004+0000 7f2d7c62c740 4 datascan.init: resolving metadata pool 17 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.006+0000 7f2d740a2640 1 -- 172.21.15.44:0/3246848592 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x561dd9e4cc00 con 0x561dd9eb0400 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.006+0000 7f2d740a2640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x561dd9e53000 0x561dd9d46100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.006+0000 7f2d740a2640 1 -- 172.21.15.44:0/3246848592 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x561dd9eb7200 con 0x561dd9eb0400 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.006+0000 7f2d7c62c740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.006+0000 7f2d7c62c740 4 datascan.main: data pool 'cephfs_data' has ID 18 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.006+0000 7f2d7c62c740 4 datascan.main: opening data pool 'cephfs_data' 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.006+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x561dd9eb0c00 0x561dd9d46680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.006+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:1 18.0 18:00000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9eb1000 con 0x561dd9eb0c00 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.006+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x561dd9e53000 0x561dd9d46100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.006+0000 7f2d798ad640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x561dd9eb0c00 0x561dd9d46680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.006+0000 7f2d798ad640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x561dd9eb0c00 0x561dd9d46680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.007+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x561dd9e53000 0x561dd9d46100 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x561dd9d8acf0 tx=0x561dd9e68c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.007+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.0 v2:172.21.15.44:6800/2303805525 1 ==== osd_op_reply(1 [pgnls start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd9eb0c00 2024-04-02T13:32:00.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.007+0000 7f2d750a4640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x561dd9e53400 0x561dd9312000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.007+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:2 18.20 18:04000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9e53800 con 0x561dd9e53400 2024-04-02T13:32:00.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.007+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x561dd9e53400 0x561dd9312000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.007+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x561dd9e53400 0x561dd9312000 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.008+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.5 v2:172.21.15.69:6800/2841767499 1 ==== osd_op_reply(2 [pgnls start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53400 2024-04-02T13:32:00.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.008+0000 7f2d758a5640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x561dd930c000 0x561dd9312580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.008+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:3 18.10 18:08000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930c400 con 0x561dd930c000 2024-04-02T13:32:00.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.008+0000 7f2d7a0ae640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x561dd930c000 0x561dd9312580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.008+0000 7f2d7a0ae640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x561dd930c000 0x561dd9312580 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.009+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== osd_op_reply(3 [pgnls start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12780 con 0x561dd930c000 2024-04-02T13:32:00.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.009+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:4 18.30 18:0c000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9e53c00 con 0x561dd9eb0c00 2024-04-02T13:32:00.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.009+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.0 v2:172.21.15.44:6800/2303805525 2 ==== osd_op_reply(4 [pgnls start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd9eb0c00 2024-04-02T13:32:00.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.009+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:5 18.8 18:10000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930c800 con 0x561dd9eb0c00 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.009+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.0 v2:172.21.15.44:6800/2303805525 3 ==== osd_op_reply(5 [pgnls start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd9eb0c00 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.009+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:6 18.28 18:14000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930cc00 con 0x561dd9e53400 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.009+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.5 v2:172.21.15.69:6800/2841767499 2 ==== osd_op_reply(6 [pgnls start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53400 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.009+0000 7f2d758a5640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x561dd930d000 0x561dd9312b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.009+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:7 18.18 18:18000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930d400 con 0x561dd930d000 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.010+0000 7f2d798ad640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x561dd930d000 0x561dd9312b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.010+0000 7f2d798ad640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x561dd930d000 0x561dd9312b00 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.010+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(7 [pgnls start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.010+0000 7f2d750a4640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x561dd9e53c00 0x561dd9313600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.010+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:8 18.38 18:1c000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9eb1000 con 0x561dd9e53c00 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.010+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x561dd9e53c00 0x561dd9313600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.010+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x561dd9e53c00 0x561dd9313600 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.011+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.3 v2:172.21.15.44:6824/1588952513 1 ==== osd_op_reply(8 [pgnls start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53c00 2024-04-02T13:32:00.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.011+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:9 18.4 18:20000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930d800 con 0x561dd9eb0c00 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.011+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.0 v2:172.21.15.44:6800/2303805525 4 ==== osd_op_reply(9 [pgnls start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd9eb0c00 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.011+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:10 18.24 18:24000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930c800 con 0x561dd9eb0c00 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.011+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.0 v2:172.21.15.44:6800/2303805525 5 ==== osd_op_reply(10 [pgnls start_epoch 123 out=49b] v121'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd9eb0c00 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.011+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:11 18.14 18:28000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930d400 con 0x561dd930c000 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.012+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 <== osd.6 v2:172.21.15.69:6802/2604158175 2 ==== osd_op_reply(11 [pgnls start_epoch 123 out=49b] v121'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12780 con 0x561dd930c000 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.012+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:12 18.34 18:2c000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9eb1c00 con 0x561dd9eb0c00 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.012+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.0 v2:172.21.15.44:6800/2303805525 6 ==== osd_op_reply(12 [pgnls start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd9eb0c00 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.012+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:13 18.c 18:30000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930d800 con 0x561dd9e53400 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.012+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.5 v2:172.21.15.69:6800/2841767499 3 ==== osd_op_reply(13 [pgnls start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53400 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.012+0000 7f2d750a4640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x561dd9eb1000 0x561dd9313b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.012+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:14 18.2c 18:34000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9e53800 con 0x561dd9eb1000 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.012+0000 7f2d7a0ae640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x561dd9eb1000 0x561dd9313b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.013+0000 7f2d7a0ae640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x561dd9eb1000 0x561dd9313b80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.013+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 <== osd.2 v2:172.21.15.44:6816/2091834331 1 ==== osd_op_reply(14 [pgnls start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12780 con 0x561dd9eb1000 2024-04-02T13:32:00.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.013+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:15 18.1c 18:38000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930cc00 con 0x561dd930d000 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.013+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 2 ==== osd_op_reply(15 [pgnls start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.013+0000 7f2d750a4640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x561dd932a000 0x561dd9314100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.013+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:16 18.3c 18:3c000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932a400 con 0x561dd932a000 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.013+0000 7f2d798ad640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x561dd932a000 0x561dd9314100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.014+0000 7f2d798ad640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x561dd932a000 0x561dd9314100 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.014+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.7 v2:172.21.15.69:6824/1292475712 1 ==== osd_op_reply(16 [pgnls start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd932a000 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.014+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:17 18.2 18:40000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930dc00 con 0x561dd932a000 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.014+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.7 v2:172.21.15.69:6824/1292475712 2 ==== osd_op_reply(17 [pgnls start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd932a000 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.014+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:18 18.22 18:44000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932a800 con 0x561dd9e53c00 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.014+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.3 v2:172.21.15.44:6824/1588952513 2 ==== osd_op_reply(18 [pgnls start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53c00 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.015+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:19 18.12 18:48000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932a400 con 0x561dd9eb1000 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.015+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 <== osd.2 v2:172.21.15.44:6816/2091834331 2 ==== osd_op_reply(19 [pgnls start_epoch 123 out=81b] v122'22 uv22 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x561dd9d12780 con 0x561dd9eb1000 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.015+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:20 18.12 18:4bec3886:::10000000002.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b800 con 0x561dd9eb1000 2024-04-02T13:32:00.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.015+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 <== osd.2 v2:172.21.15.44:6816/2091834331 3 ==== osd_op_reply(20 10000000002.00000000 [stat out=16b] v0'0 uv22 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x561dd9d12780 con 0x561dd9eb1000 2024-04-02T13:32:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.015+0000 7f2d7c62c740 10 datascan.operator(): handling object 1099511627778.0 2024-04-02T13:32:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.015+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:21 18.12 18:4bec3886:::10000000002.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x561dd932bc00 con 0x561dd9eb1000 2024-04-02T13:32:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.016+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 <== osd.2 v2:172.21.15.44:6816/2091834331 4 ==== osd_op_reply(21 10000000002.00000000 [call] v123'23 uv23 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x561dd9d12780 con 0x561dd9eb1000 2024-04-02T13:32:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.016+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:22 18.32 18:4c000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932a400 con 0x561dd9e53c00 2024-04-02T13:32:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.017+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.3 v2:172.21.15.44:6824/1588952513 3 ==== osd_op_reply(22 [pgnls start_epoch 123 out=49b] v121'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53c00 2024-04-02T13:32:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.017+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:23 18.a 18:50000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b400 con 0x561dd9e53c00 2024-04-02T13:32:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.017+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.3 v2:172.21.15.44:6824/1588952513 4 ==== osd_op_reply(23 [pgnls start_epoch 123 out=49b] v121'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53c00 2024-04-02T13:32:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.017+0000 7f2d750a4640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x561dd932ac00 0x561dd9314680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.017+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:24 18.2a 18:54000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b000 con 0x561dd932ac00 2024-04-02T13:32:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.017+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x561dd932ac00 0x561dd9314680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.017+0000 7f2d7a8af640 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x561dd932ac00 0x561dd9314680 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.018+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.4 v2:172.21.15.69:6816/1041350007 1 ==== osd_op_reply(24 [pgnls start_epoch 123 out=49b] v121'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd932ac00 2024-04-02T13:32:00.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.018+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:25 18.1a 18:58000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930c800 con 0x561dd9e53c00 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.018+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.3 v2:172.21.15.44:6824/1588952513 5 ==== osd_op_reply(25 [pgnls start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53c00 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.018+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:26 18.3a 18:5c000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932a800 con 0x561dd9eb0c00 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.018+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.0 v2:172.21.15.44:6800/2303805525 7 ==== osd_op_reply(26 [pgnls start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd9eb0c00 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.018+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:27 18.6 18:60000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930cc00 con 0x561dd932a000 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.019+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.7 v2:172.21.15.69:6824/1292475712 3 ==== osd_op_reply(27 [pgnls start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd932a000 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.019+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:28 18.26 18:64000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932a400 con 0x561dd9eb1000 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.019+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 <== osd.2 v2:172.21.15.44:6816/2091834331 5 ==== osd_op_reply(28 [pgnls start_epoch 123 out=49b] v121'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12780 con 0x561dd9eb1000 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.019+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:29 18.16 18:68000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9eb1c00 con 0x561dd9e53c00 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.019+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.3 v2:172.21.15.44:6824/1588952513 6 ==== osd_op_reply(29 [pgnls start_epoch 123 out=49b] v121'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53c00 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.019+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:30 18.36 18:6c000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b400 con 0x561dd930d000 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.019+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 3 ==== osd_op_reply(30 [pgnls start_epoch 123 out=81b] v121'33 uv17 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.019+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:31 18.36 18:6fe38dc2:::100000001f7.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b800 con 0x561dd930d000 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.020+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 4 ==== osd_op_reply(31 100000001f7.00000000 [stat out=16b] v0'0 uv17 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.020+0000 7f2d7c62c740 10 datascan.operator(): handling object 1099511628279.0 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.020+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:32 18.36 18:6fe38dc2:::100000001f7.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x561dd9e52c00 con 0x561dd930d000 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.021+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 5 ==== osd_op_reply(32 100000001f7.00000000 [call] v123'34 uv34 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.021+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:33 18.e 18:70000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9e53800 con 0x561dd9eb0c00 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.021+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.0 v2:172.21.15.44:6800/2303805525 8 ==== osd_op_reply(33 [pgnls start_epoch 123 out=49b] v121'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd9eb0c00 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.021+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:34 18.2e 18:74000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b000 con 0x561dd9e53c00 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.021+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.3 v2:172.21.15.44:6824/1588952513 7 ==== osd_op_reply(34 [pgnls start_epoch 123 out=49b] v121'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53c00 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.021+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:35 18.1e 18:78000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932a800 con 0x561dd9e53400 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.022+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.5 v2:172.21.15.69:6800/2841767499 4 ==== osd_op_reply(35 [pgnls start_epoch 123 out=49b] v121'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53400 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.022+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:36 18.3e 18:7c000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930d800 con 0x561dd9e53c00 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.022+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.3 v2:172.21.15.44:6824/1588952513 8 ==== osd_op_reply(36 [pgnls start_epoch 123 out=49b] v121'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53c00 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.022+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:37 18.1 18:80000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b400 con 0x561dd9e53c00 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.022+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.3 v2:172.21.15.44:6824/1588952513 9 ==== osd_op_reply(37 [pgnls start_epoch 123 out=49b] v121'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53c00 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.022+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:38 18.21 18:84000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b000 con 0x561dd9e53400 2024-04-02T13:32:00.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.023+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.5 v2:172.21.15.69:6800/2841767499 5 ==== osd_op_reply(38 [pgnls start_epoch 123 out=49b] v121'24 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53400 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.023+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:39 18.11 18:88000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b800 con 0x561dd930d000 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.023+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 6 ==== osd_op_reply(39 [pgnls start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.023+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:40 18.31 18:8c000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932a800 con 0x561dd932ac00 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.023+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.4 v2:172.21.15.69:6816/1041350007 2 ==== osd_op_reply(40 [pgnls start_epoch 123 out=81b] v121'15 uv8 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x561dd9d12500 con 0x561dd932ac00 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.024+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:41 18.31 18:8de29a03:::10000000004.00000001:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x561dd930c400 con 0x561dd932ac00 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.024+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.4 v2:172.21.15.69:6816/1041350007 3 ==== osd_op_reply(41 10000000004.00000001 [stat out=16b] v0'0 uv8 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x561dd9d12500 con 0x561dd932ac00 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.024+0000 7f2d7c62c740 10 datascan.operator(): handling object 1099511627780.1 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.024+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:42 18.5 18:a367a3e8:::10000000004.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x561dd930d400 con 0x561dd930d000 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.025+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 7 ==== osd_op_reply(42 10000000004.00000000 [call] v123'14 uv14 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.025+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:43 18.9 18:90000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9346000 con 0x561dd9e53c00 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.025+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.3 v2:172.21.15.44:6824/1588952513 10 ==== osd_op_reply(43 [pgnls start_epoch 123 out=49b] v121'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53c00 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.025+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:44 18.29 18:94000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b400 con 0x561dd932a000 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.026+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.7 v2:172.21.15.69:6824/1292475712 4 ==== osd_op_reply(44 [pgnls start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd932a000 2024-04-02T13:32:00.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.026+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:45 18.19 18:98000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930dc00 con 0x561dd932ac00 2024-04-02T13:32:00.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.026+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.4 v2:172.21.15.69:6816/1041350007 4 ==== osd_op_reply(45 [pgnls start_epoch 123 out=81b] v122'23 uv23 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x561dd9d12500 con 0x561dd932ac00 2024-04-02T13:32:00.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.026+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:46 18.19 18:997b7129:::100000001f8.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x561dd9346400 con 0x561dd932ac00 2024-04-02T13:32:00.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.026+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.4 v2:172.21.15.69:6816/1041350007 5 ==== osd_op_reply(46 100000001f8.00000000 [stat out=16b] v0'0 uv23 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x561dd9d12500 con 0x561dd932ac00 2024-04-02T13:32:00.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.026+0000 7f2d7c62c740 10 datascan.operator(): handling object 1099511628280.0 2024-04-02T13:32:00.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.026+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:47 18.19 18:997b7129:::100000001f8.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x561dd9346800 con 0x561dd932ac00 2024-04-02T13:32:00.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.028+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.4 v2:172.21.15.69:6816/1041350007 6 ==== osd_op_reply(47 100000001f8.00000000 [call] v123'24 uv24 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x561dd9d12500 con 0x561dd932ac00 2024-04-02T13:32:00.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.028+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:48 18.39 18:9c000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9346c00 con 0x561dd9eb0c00 2024-04-02T13:32:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.028+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.0 v2:172.21.15.44:6800/2303805525 9 ==== osd_op_reply(48 [pgnls start_epoch 123 out=49b] v121'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd9eb0c00 2024-04-02T13:32:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.028+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:49 18.5 18:a0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930cc00 con 0x561dd930d000 2024-04-02T13:32:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.028+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 8 ==== osd_op_reply(49 [pgnls start_epoch 123 out=81b] v123'14 uv14 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.028+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:50 18.5 18:a367a3e8:::10000000004.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x561dd9347000 con 0x561dd930d000 2024-04-02T13:32:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.029+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 9 ==== osd_op_reply(50 10000000004.00000000 [stat out=16b] v0'0 uv14 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.029+0000 7f2d7c62c740 10 datascan.operator(): handling object 1099511627780.0 2024-04-02T13:32:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.029+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:51 18.5 18:a367a3e8:::10000000004.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x561dd9347400 con 0x561dd930d000 2024-04-02T13:32:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.030+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 10 ==== osd_op_reply(51 10000000004.00000000 [call] v123'15 uv15 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.030+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:52 18.25 18:a4000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9347800 con 0x561dd930c000 2024-04-02T13:32:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.030+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 <== osd.6 v2:172.21.15.69:6802/2604158175 3 ==== osd_op_reply(52 [pgnls start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12780 con 0x561dd930c000 2024-04-02T13:32:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.030+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:53 18.15 18:a8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930d400 con 0x561dd9e53400 2024-04-02T13:32:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.030+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.5 v2:172.21.15.69:6800/2841767499 6 ==== osd_op_reply(53 [pgnls start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53400 2024-04-02T13:32:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.030+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:54 18.35 18:ac000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930c800 con 0x561dd932ac00 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.031+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.4 v2:172.21.15.69:6816/1041350007 7 ==== osd_op_reply(54 [pgnls start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd932ac00 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.031+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:55 18.d 18:b0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b800 con 0x561dd932ac00 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.031+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.4 v2:172.21.15.69:6816/1041350007 8 ==== osd_op_reply(55 [pgnls start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd932ac00 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.031+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:56 18.2d 18:b4000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930d800 con 0x561dd930c000 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.031+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 <== osd.6 v2:172.21.15.69:6802/2604158175 4 ==== osd_op_reply(56 [pgnls start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12780 con 0x561dd930c000 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.031+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:57 18.1d 18:b8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b400 con 0x561dd9eb0c00 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.032+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.0 v2:172.21.15.44:6800/2303805525 10 ==== osd_op_reply(57 [pgnls start_epoch 123 out=49b] v121'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd9eb0c00 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.032+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:58 18.3d 18:bc000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9eb1c00 con 0x561dd9e53c00 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.032+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.3 v2:172.21.15.44:6824/1588952513 11 ==== osd_op_reply(58 [pgnls start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53c00 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.032+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:59 18.3 18:c0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b000 con 0x561dd930d000 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.032+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 11 ==== osd_op_reply(59 [pgnls start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.032+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:60 18.23 18:c4000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9346c00 con 0x561dd9eb1000 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.032+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 <== osd.2 v2:172.21.15.44:6816/2091834331 6 ==== osd_op_reply(60 [pgnls start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12780 con 0x561dd9eb1000 2024-04-02T13:32:00.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.032+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:61 18.13 18:c8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932a800 con 0x561dd932a000 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.033+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.7 v2:172.21.15.69:6824/1292475712 5 ==== osd_op_reply(61 [pgnls start_epoch 123 out=49b] v121'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd932a000 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.033+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:62 18.33 18:cc000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9346000 con 0x561dd930d000 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.033+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 12 ==== osd_op_reply(62 [pgnls start_epoch 123 out=49b] v121'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.033+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:63 18.b 18:d0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9e52c00 con 0x561dd9e53400 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.033+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.5 v2:172.21.15.69:6800/2841767499 7 ==== osd_op_reply(63 [pgnls start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53400 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.033+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:64 18.2b 18:d4000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9346400 con 0x561dd932a000 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.034+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.7 v2:172.21.15.69:6824/1292475712 6 ==== osd_op_reply(64 [pgnls start_epoch 123 out=49b] v121'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd932a000 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.034+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:65 18.1b 18:d8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9e53800 con 0x561dd930d000 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.034+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 13 ==== osd_op_reply(65 [pgnls start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.034+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:66 18.3b 18:dc000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9346800 con 0x561dd932ac00 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.034+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.4 v2:172.21.15.69:6816/1041350007 9 ==== osd_op_reply(66 [pgnls start_epoch 123 out=49b] v121'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd932ac00 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.034+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:67 18.7 18:e0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9347000 con 0x561dd9eb1000 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.034+0000 7f2d7a0ae640 1 -- 172.21.15.44:0/3246848592 <== osd.2 v2:172.21.15.44:6816/2091834331 7 ==== osd_op_reply(67 [pgnls start_epoch 123 out=49b] v121'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12780 con 0x561dd9eb1000 2024-04-02T13:32:00.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.034+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:68 18.27 18:e4000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930c400 con 0x561dd9e53400 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.035+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.5 v2:172.21.15.69:6800/2841767499 8 ==== osd_op_reply(68 [pgnls start_epoch 123 out=49b] v121'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53400 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.035+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:69 18.17 18:e8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9347400 con 0x561dd9e53400 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.035+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.5 v2:172.21.15.69:6800/2841767499 9 ==== osd_op_reply(69 [pgnls start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53400 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.035+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:70 18.37 18:ec000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd930dc00 con 0x561dd932a000 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.035+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.7 v2:172.21.15.69:6824/1292475712 7 ==== osd_op_reply(70 [pgnls start_epoch 123 out=49b] v121'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd932a000 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.036+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:71 18.f 18:f0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9346000 con 0x561dd9e53400 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.036+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.5 v2:172.21.15.69:6800/2841767499 10 ==== osd_op_reply(71 [pgnls start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd9e53400 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.036+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:72 18.2f 18:f4000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9346800 con 0x561dd930d000 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.036+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.1 v2:172.21.15.44:6801/704457846 14 ==== osd_op_reply(72 [pgnls start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd930d000 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.036+0000 7f2d758a5640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:73 18.1f 18:f8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd932b400 con 0x561dd932a000 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 <== osd.7 v2:172.21.15.69:6824/1292475712 8 ==== osd_op_reply(73 [pgnls start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12000 con 0x561dd932a000 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d750a4640 1 -- 172.21.15.44:0/3246848592 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:74 18.3f 18:fc000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x561dd9347400 con 0x561dd932ac00 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7a8af640 1 -- 172.21.15.44:0/3246848592 <== osd.4 v2:172.21.15.69:6816/1041350007 10 ==== osd_op_reply(74 [pgnls start_epoch 123 out=49b] v121'17 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561dd9d12500 con 0x561dd932ac00 2024-04-02T13:32:00.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x561dd9eb0c00 msgr2=0x561dd9d46680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x561dd9eb0c00 0x561dd9d46680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x561dd930d000 msgr2=0x561dd9312b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x561dd930d000 0x561dd9312b00 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x561dd9eb1000 msgr2=0x561dd9313b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x561dd9eb1000 0x561dd9313b80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x561dd9e53c00 msgr2=0x561dd9313600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x561dd9e53c00 0x561dd9313600 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x561dd932ac00 msgr2=0x561dd9314680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x561dd932ac00 0x561dd9314680 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x561dd9e53400 msgr2=0x561dd9312000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x561dd9e53400 0x561dd9312000 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 reap_dead start 2024-04-02T13:32:00.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x561dd930c000 msgr2=0x561dd9312580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x561dd930c000 0x561dd9312580 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d798ad640 1 -- 172.21.15.44:0/3246848592 reap_dead start 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x561dd932a000 msgr2=0x561dd9314100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x561dd932a000 0x561dd9314100 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x561dd9e53000 msgr2=0x561dd9d46100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x561dd9e53000 0x561dd9d46100 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x561dd9d8acf0 tx=0x561dd9e68c00 comp rx=0 tx=0).stop 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0400 msgr2=0x561dd9d45600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0400 0x561dd9d45600 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x561dd9da5830 tx=0x561dd9d1a660 comp rx=0 tx=0).stop 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 shutdown_connections 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x561dd932a000 0x561dd9314100 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x561dd930c000 0x561dd9312580 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x561dd9e53000 0x561dd9d46100 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/3246848592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd9eb0400 0x561dd9d45600 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 >> 172.21.15.44:0/3246848592 conn(0x561dd9d71c00 msgr2=0x561dd9065000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:00.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 shutdown_connections 2024-04-02T13:32:00.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.037+0000 7f2d7c62c740 1 -- 172.21.15.44:0/3246848592 wait complete. 2024-04-02T13:32:00.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.038+0000 7f2d7c62c740 1 -- 172.21.15.44:0/290191999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd8f80c00 msgr2=0x561dd9d44000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.038+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/290191999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd8f80c00 0x561dd9d44000 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x561dd9da5080 tx=0x561dd9d1bec0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.038+0000 7f2d7c62c740 1 -- 172.21.15.44:0/290191999 shutdown_connections 2024-04-02T13:32:00.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.038+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/290191999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561dd8f81c00 0x561dd9d44b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.038+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/290191999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561dd8f81800 0x561dd9d44580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.038+0000 7f2d7c62c740 1 --2- 172.21.15.44:0/290191999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561dd8f80c00 0x561dd9d44000 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.038+0000 7f2d7c62c740 1 -- 172.21.15.44:0/290191999 >> 172.21.15.44:0/290191999 conn(0x561dd8f81000 msgr2=0x561dd9064800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:00.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.038+0000 7f2d7c62c740 1 -- 172.21.15.44:0/290191999 shutdown_connections 2024-04-02T13:32:00.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.038+0000 7f2d7c62c740 1 -- 172.21.15.44:0/290191999 wait complete. 2024-04-02T13:32:00.052 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', 'cephfs_data'] in time 0:00:00.127765, result: 2024-04-02T13:32:00.053 DEBUG:teuthology.orchestra.run.smithi044:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes --filter-tag mytag123 cephfs_data 2024-04-02T13:32:00.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.121+0000 7fa3ae02a740 1 -- start start 2024-04-02T13:32:00.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.121+0000 7fa3ae02a740 1 Processor -- start 2024-04-02T13:32:00.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ae02a740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563652748c00 0x5636528b8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ae02a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563652749800 0x5636528b8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ae02a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563652749c00 0x5636528b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ae02a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5636528bc180 con 0x563652749800 2024-04-02T13:32:00.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ae02a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5636527a8900 con 0x563652749c00 2024-04-02T13:32:00.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ae02a740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5636527a8480 con 0x563652748c00 2024-04-02T13:32:00.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ab1da640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563652749800 0x5636528b8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ab1da640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563652749800 0x5636528b8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50902/0 (socket says 172.21.15.44:50902) 2024-04-02T13:32:00.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ab1da640 1 -- 172.21.15.44:0/1801022632 learned_addr learned my addr 172.21.15.44:0/1801022632 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:00.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ab1da640 1 -- 172.21.15.44:0/1801022632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563652748c00 msgr2=0x5636528b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ab1da640 1 --2- 172.21.15.44:0/1801022632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563652748c00 0x5636528b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ab1da640 1 -- 172.21.15.44:0/1801022632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563652749c00 msgr2=0x5636528b8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ab1da640 1 --2- 172.21.15.44:0/1801022632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563652749c00 0x5636528b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ab1da640 1 -- 172.21.15.44:0/1801022632 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563652775680 con 0x563652749800 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3ab1da640 1 --2- 172.21.15.44:0/1801022632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563652749800 0x5636528b8580 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x56365355d9b0 tx=0x5636534e2600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c558292a288fbf0e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.122+0000 7fa3a91d6640 1 -- 172.21.15.44:0/1801022632 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56365276e680 con 0x563652749800 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.123+0000 7fa3a91d6640 1 -- 172.21.15.44:0/1801022632 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5636527921c0 con 0x563652749800 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.123+0000 7fa3a91d6640 1 -- 172.21.15.44:0/1801022632 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56365276eb60 con 0x563652749800 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.123+0000 7fa3ae02a740 1 -- 172.21.15.44:0/1801022632 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563652775860 con 0x563652749800 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.123+0000 7fa3ae02a740 1 -- 172.21.15.44:0/1801022632 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=0}) -- 0x5636527752c0 con 0x563652749800 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.123+0000 7fa3ae02a740 4 waiting for MDS map... 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.123+0000 7fa3a91d6640 1 -- 172.21.15.44:0/1801022632 <== mon.0 v2:172.21.15.44:3300/0 4 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x56365360e600 con 0x563652749800 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.123+0000 7fa3a91d6640 1 -- 172.21.15.44:0/1801022632 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x5636527a1180 con 0x563652749800 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.123+0000 7fa3ae02a740 4 Got MDS map 106 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.123+0000 7fa3ae02a740 10 datascan.parse_kwarg: Applying tag filter: 'mytag123' 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.123+0000 7fa3ae02a740 4 datascan.main: Using metadata pool output 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.123+0000 7fa3ae02a740 4 datascan.main: connecting to RADOS... 2024-04-02T13:32:00.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.123+0000 7fa3ae02a740 1 Processor -- start 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.123+0000 7fa3ae02a740 1 -- start start 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ae02a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563653611400 0x5636528b9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ae02a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611800 0x5636528b9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ae02a740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563653611c00 0x5636528b9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ae02a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5636527a8180 con 0x563653611800 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ae02a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5636528bcd80 con 0x563653611400 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ae02a740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5636528bcf00 con 0x563653611c00 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ab9db640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611800 0x5636528b9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ab9db640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611800 0x5636528b9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50908/0 (socket says 172.21.15.44:50908) 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ab9db640 1 -- 172.21.15.44:0/940733596 learned_addr learned my addr 172.21.15.44:0/940733596 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ab1da640 1 --2- 172.21.15.44:0/940733596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563653611c00 0x5636528b9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ac1dc640 1 --2- 172.21.15.44:0/940733596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563653611400 0x5636528b9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ab9db640 1 -- 172.21.15.44:0/940733596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563653611c00 msgr2=0x5636528b9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ab9db640 1 --2- 172.21.15.44:0/940733596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563653611c00 0x5636528b9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ab9db640 1 -- 172.21.15.44:0/940733596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563653611400 msgr2=0x5636528b9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ab9db640 1 --2- 172.21.15.44:0/940733596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563653611400 0x5636528b9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.124+0000 7fa3ab9db640 1 -- 172.21.15.44:0/940733596 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636527754a0 con 0x563653611800 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.125+0000 7fa3ab9db640 1 --2- 172.21.15.44:0/940733596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611800 0x5636528b9600 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x56365362cde0 tx=0x563653630fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6324d006ff1964f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.125+0000 7fa3a71d2640 1 -- 172.21.15.44:0/940733596 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56365276e680 con 0x563653611800 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.125+0000 7fa3a71d2640 1 -- 172.21.15.44:0/940733596 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x563652792e00 con 0x563653611800 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.125+0000 7fa3a71d2640 1 -- 172.21.15.44:0/940733596 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56365276eb60 con 0x563653611800 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.125+0000 7fa3ae02a740 1 -- 172.21.15.44:0/940733596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611800 msgr2=0x5636528b9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.125+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/940733596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611800 0x5636528b9600 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x56365362cde0 tx=0x563653630fc0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.125+0000 7fa3ae02a740 1 -- 172.21.15.44:0/940733596 shutdown_connections 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.125+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/940733596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563653611c00 0x5636528b9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.125+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/940733596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611800 0x5636528b9600 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.125+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/940733596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563653611400 0x5636528b9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.125+0000 7fa3ae02a740 1 -- 172.21.15.44:0/940733596 >> 172.21.15.44:0/940733596 conn(0x563653611000 msgr2=0x5636534cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:00.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.125+0000 7fa3ae02a740 1 -- 172.21.15.44:0/940733596 shutdown_connections 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.125+0000 7fa3ae02a740 1 -- 172.21.15.44:0/940733596 wait complete. 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.126+0000 7fa3ae02a740 1 Processor -- start 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.126+0000 7fa3ae02a740 1 -- start start 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.126+0000 7fa3ae02a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611400 0x5636528b9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.126+0000 7fa3ae02a740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563653611800 0x5636528b9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.126+0000 7fa3ae02a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563653611c00 0x5636528b9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.126+0000 7fa3ae02a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5636528bd200 con 0x563653611400 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.126+0000 7fa3ae02a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5636528bd080 con 0x563653611c00 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.126+0000 7fa3ae02a740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x5636528bd380 con 0x563653611800 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.126+0000 7fa3ac1dc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611400 0x5636528b9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.126+0000 7fa3ac1dc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611400 0x5636528b9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50910/0 (socket says 172.21.15.44:50910) 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.126+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 learned_addr learned my addr 172.21.15.44:0/846240798 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.126+0000 7fa3ab9db640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563653611800 0x5636528b9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3ab1da640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563653611c00 0x5636528b9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563653611800 msgr2=0x5636528b9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3ac1dc640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563653611800 0x5636528b9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563653611c00 msgr2=0x5636528b9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3ac1dc640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563653611c00 0x5636528b9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563652775680 con 0x563653611400 2024-04-02T13:32:00.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3ab1da640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563653611c00 0x5636528b9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:00.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3ac1dc640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611400 0x5636528b9080 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x56365355d0e0 tx=0x563653630de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3a59cf640 1 -- 172.21.15.44:0/846240798 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56365276e9c0 con 0x563653611400 2024-04-02T13:32:00.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3a59cf640 1 -- 172.21.15.44:0/846240798 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x563652792540 con 0x563653611400 2024-04-02T13:32:00.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563652775860 con 0x563653611400 2024-04-02T13:32:00.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3a59cf640 1 -- 172.21.15.44:0/846240798 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56365276e680 con 0x563653611400 2024-04-02T13:32:00.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636527750e0 con 0x563653611400 2024-04-02T13:32:00.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.127+0000 7fa3ae02a740 4 datascan.init: resolving metadata pool 17 2024-04-02T13:32:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.128+0000 7fa3a59cf640 1 -- 172.21.15.44:0/846240798 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x56365368ac00 con 0x563653611400 2024-04-02T13:32:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.129+0000 7fa3a59cf640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x563653514c00 0x5636528ba100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.129+0000 7fa3a59cf640 1 -- 172.21.15.44:0/846240798 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x563653516c00 con 0x563653611400 2024-04-02T13:32:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.129+0000 7fa3ae02a740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-04-02T13:32:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.129+0000 7fa3ae02a740 4 datascan.main: data pool 'cephfs_data' has ID 18 2024-04-02T13:32:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.129+0000 7fa3ae02a740 4 datascan.main: opening data pool 'cephfs_data' 2024-04-02T13:32:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.129+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x563653515000 0x5636528ba680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.129+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:1 17.14 17:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x563653515400 con 0x563653515000 2024-04-02T13:32:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.129+0000 7fa3ab9db640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x563653514c00 0x5636528ba100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.129+0000 7fa3ab1da640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x563653515000 0x5636528ba680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.129+0000 7fa3ab1da640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x563653515000 0x5636528ba680 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.129+0000 7fa3ab9db640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x563653514c00 0x5636528ba100 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x56365362cde0 tx=0x563653630c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.130+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv37 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.130+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x563653515800 0x5636528bac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.130+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:2 17.1f 17:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x563653515c00 con 0x563653515800 2024-04-02T13:32:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.130+0000 7fa3ac1dc640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x563653515800 0x5636528bac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.130+0000 7fa3ac1dc640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x563653515800 0x5636528bac00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.130+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) ==== 162+0+16 (crc 0 0 0) 0x5636534de500 con 0x563653515800 2024-04-02T13:32:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.130+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x563652ace000 0x5636528bb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.130+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:3 18.0 18:00000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ace400 con 0x563652ace000 2024-04-02T13:32:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.131+0000 7fa3ab9db640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x563652ace000 0x5636528bb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.131+0000 7fa3ab9db640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x563652ace000 0x5636528bb180 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.131+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.131+0000 7fa3a69d1640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563652acf800 0x563652ade000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.131+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:4 18.20 18:04000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652acfc00 con 0x563652acf800 2024-04-02T13:32:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.131+0000 7fa3ab1da640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563652acf800 0x563652ade000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.132+0000 7fa3ab1da640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563652acf800 0x563652ade000 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.132+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.5 v2:172.21.15.69:6800/2841767499 1 ==== osd_op_reply(4 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652acf800 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.132+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:5 18.10 18:08000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2000 con 0x563653515800 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.133+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.6 v2:172.21.15.69:6802/2604158175 2 ==== osd_op_reply(5 [pgnls-filter start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563653515800 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.133+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:6 18.30 18:0c000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2400 con 0x563652ace000 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.133+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 2 ==== osd_op_reply(6 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.133+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:7 18.8 18:10000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad3000 con 0x563652ace000 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.133+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 3 ==== osd_op_reply(7 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.133+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:8 18.28 18:14000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2800 con 0x563652acf800 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.133+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.5 v2:172.21.15.69:6800/2841767499 2 ==== osd_op_reply(8 [pgnls-filter start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652acf800 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.133+0000 7fa3a71d2640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x563652ad3400 0x563652ade580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.133+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:9 18.18 18:18000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2400 con 0x563652ad3400 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.134+0000 7fa3ac1dc640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x563652ad3400 0x563652ade580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.134+0000 7fa3ac1dc640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x563652ad3400 0x563652ade580 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.134+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(9 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652ad3400 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.134+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:10 18.38 18:1c000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2c00 con 0x563653515000 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.134+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 2 ==== osd_op_reply(10 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.134+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:11 18.4 18:20000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ace400 con 0x563652ace000 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.135+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 4 ==== osd_op_reply(11 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.135+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:12 18.24 18:24000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2800 con 0x563652ace000 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.135+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 5 ==== osd_op_reply(12 [pgnls-filter start_epoch 123 out=49b] v121'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.135+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:13 18.14 18:28000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653514800 con 0x563653515800 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.135+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.6 v2:172.21.15.69:6802/2604158175 3 ==== osd_op_reply(13 [pgnls-filter start_epoch 123 out=49b] v121'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563653515800 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.135+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:14 18.34 18:2c000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652acfc00 con 0x563652ace000 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.136+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 6 ==== osd_op_reply(14 [pgnls-filter start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.136+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:15 18.c 18:30000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2000 con 0x563652acf800 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.136+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.5 v2:172.21.15.69:6800/2841767499 3 ==== osd_op_reply(15 [pgnls-filter start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652acf800 2024-04-02T13:32:00.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.136+0000 7fa3a69d1640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x563653515400 0x563652adeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.136+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:16 18.2c 18:34000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653610800 con 0x563653515400 2024-04-02T13:32:00.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.136+0000 7fa3ab9db640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x563653515400 0x563652adeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.136+0000 7fa3ab9db640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x563653515400 0x563652adeb00 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.137+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.2 v2:172.21.15.44:6816/2091834331 1 ==== osd_op_reply(16 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563653515400 2024-04-02T13:32:00.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.137+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:17 18.1c 18:38000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653515c00 con 0x563652ad3400 2024-04-02T13:32:00.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.137+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.1 v2:172.21.15.44:6801/704457846 2 ==== osd_op_reply(17 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652ad3400 2024-04-02T13:32:00.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.137+0000 7fa3a69d1640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x563652ad2400 0x563652adf080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.137+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:18 18.3c 18:3c000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2c00 con 0x563652ad2400 2024-04-02T13:32:00.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.137+0000 7fa3ab1da640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x563652ad2400 0x563652adf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.137+0000 7fa3ab1da640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x563652ad2400 0x563652adf080 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.138+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.7 v2:172.21.15.69:6824/1292475712 1 ==== osd_op_reply(18 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652ad2400 2024-04-02T13:32:00.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.138+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:19 18.2 18:40000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653514800 con 0x563652ad2400 2024-04-02T13:32:00.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.138+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.7 v2:172.21.15.69:6824/1292475712 2 ==== osd_op_reply(19 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652ad2400 2024-04-02T13:32:00.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.138+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:20 18.22 18:44000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad3000 con 0x563653515000 2024-04-02T13:32:00.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.138+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 3 ==== osd_op_reply(20 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.138+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:21 18.12 18:48000000::::head [pgnls-filter start_epoch 123 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653525400 con 0x563653515400 2024-04-02T13:32:00.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.139+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.2 v2:172.21.15.44:6816/2091834331 2 ==== osd_op_reply(21 [pgnls-filter start_epoch 123 out=81b] v123'23 uv23 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x5636534de780 con 0x563653515400 2024-04-02T13:32:00.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.139+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:22 18.0 18:00000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ace800 con 0x563652ace000 2024-04-02T13:32:00.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.139+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 7 ==== osd_op_reply(22 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.139+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:23 18.20 18:04000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653525400 con 0x563652acf800 2024-04-02T13:32:00.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.139+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.5 v2:172.21.15.69:6800/2841767499 4 ==== osd_op_reply(23 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652acf800 2024-04-02T13:32:00.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.139+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:24 18.10 18:08000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad3800 con 0x563653515800 2024-04-02T13:32:00.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.140+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.6 v2:172.21.15.69:6802/2604158175 4 ==== osd_op_reply(24 [pgnls-filter start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563653515800 2024-04-02T13:32:00.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.140+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:25 18.30 18:0c000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653610800 con 0x563652ace000 2024-04-02T13:32:00.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.140+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 8 ==== osd_op_reply(25 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.140+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:26 18.8 18:10000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad3c00 con 0x563652ace000 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.140+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 9 ==== osd_op_reply(26 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.140+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:27 18.28 18:14000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ace400 con 0x563652acf800 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.140+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.5 v2:172.21.15.69:6800/2841767499 5 ==== osd_op_reply(27 [pgnls-filter start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652acf800 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.140+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:28 18.18 18:18000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ace400 con 0x563652ad3400 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.141+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.1 v2:172.21.15.44:6801/704457846 3 ==== osd_op_reply(28 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652ad3400 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.141+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:29 18.38 18:1c000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652acfc00 con 0x563653515000 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.141+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 4 ==== osd_op_reply(29 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.141+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:30 18.4 18:20000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653525400 con 0x563652ace000 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.141+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 10 ==== osd_op_reply(30 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.141+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:31 18.24 18:24000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2800 con 0x563652ace000 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.142+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 11 ==== osd_op_reply(31 [pgnls-filter start_epoch 123 out=49b] v121'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.142+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:32 18.14 18:28000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2000 con 0x563653515800 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.142+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.6 v2:172.21.15.69:6802/2604158175 5 ==== osd_op_reply(32 [pgnls-filter start_epoch 123 out=49b] v121'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563653515800 2024-04-02T13:32:00.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.142+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:33 18.34 18:2c000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afa000 con 0x563652ace000 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.142+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 12 ==== osd_op_reply(33 [pgnls-filter start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.142+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:34 18.c 18:30000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2c00 con 0x563652acf800 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.143+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.5 v2:172.21.15.69:6800/2841767499 6 ==== osd_op_reply(34 [pgnls-filter start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652acf800 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.143+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:35 18.2c 18:34000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afa400 con 0x563653515400 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.143+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.2 v2:172.21.15.44:6816/2091834331 3 ==== osd_op_reply(35 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563653515400 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.143+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:36 18.1c 18:38000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad3000 con 0x563652ad3400 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.143+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.1 v2:172.21.15.44:6801/704457846 4 ==== osd_op_reply(36 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652ad3400 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.143+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:37 18.3c 18:3c000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afa800 con 0x563652ad2400 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.143+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.7 v2:172.21.15.69:6824/1292475712 3 ==== osd_op_reply(37 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652ad2400 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.143+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:38 18.2 18:40000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653514800 con 0x563652ad2400 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.144+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.7 v2:172.21.15.69:6824/1292475712 4 ==== osd_op_reply(38 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652ad2400 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.144+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:39 18.22 18:44000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afac00 con 0x563653515000 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.144+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 5 ==== osd_op_reply(39 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.144+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:40 18.12 18:48000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afa000 con 0x563653515400 2024-04-02T13:32:00.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.144+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.2 v2:172.21.15.44:6816/2091834331 4 ==== osd_op_reply(40 [pgnls-filter start_epoch 123 out=49b] v123'23 uv23 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563653515400 2024-04-02T13:32:00.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.144+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:41 18.32 18:4c000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afb000 con 0x563653515000 2024-04-02T13:32:00.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.145+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 6 ==== osd_op_reply(41 [pgnls-filter start_epoch 123 out=49b] v121'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.145+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:42 18.a 18:50000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afb800 con 0x563653515000 2024-04-02T13:32:00.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.145+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 7 ==== osd_op_reply(42 [pgnls-filter start_epoch 123 out=49b] v121'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.145+0000 7fa3a71d2640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x563652afb400 0x563652adf600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.145+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:43 18.2a 18:54000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afa800 con 0x563652afb400 2024-04-02T13:32:00.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.145+0000 7fa3ac1dc640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x563652afb400 0x563652adf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.145+0000 7fa3ac1dc640 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x563652afb400 0x563652adf600 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.146+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.4 v2:172.21.15.69:6816/1041350007 1 ==== osd_op_reply(43 [pgnls-filter start_epoch 123 out=49b] v121'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652afb400 2024-04-02T13:32:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.146+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:44 18.1a 18:58000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad3c00 con 0x563653515000 2024-04-02T13:32:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.146+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 8 ==== osd_op_reply(44 [pgnls-filter start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.146+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:45 18.3a 18:5c000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afac00 con 0x563652ace000 2024-04-02T13:32:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.146+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 13 ==== osd_op_reply(45 [pgnls-filter start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.146+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:46 18.6 18:60000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2800 con 0x563652ad2400 2024-04-02T13:32:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.146+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.7 v2:172.21.15.69:6824/1292475712 5 ==== osd_op_reply(46 [pgnls-filter start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652ad2400 2024-04-02T13:32:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.147+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:47 18.26 18:64000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afb000 con 0x563653515400 2024-04-02T13:32:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.147+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.2 v2:172.21.15.44:6816/2091834331 5 ==== osd_op_reply(47 [pgnls-filter start_epoch 123 out=49b] v121'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563653515400 2024-04-02T13:32:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.147+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:48 18.16 18:68000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653525400 con 0x563653515000 2024-04-02T13:32:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.147+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 9 ==== osd_op_reply(48 [pgnls-filter start_epoch 123 out=49b] v121'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.147+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:49 18.36 18:6c000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afb800 con 0x563652ad3400 2024-04-02T13:32:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.147+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.1 v2:172.21.15.44:6801/704457846 5 ==== osd_op_reply(49 [pgnls-filter start_epoch 123 out=81b] v123'34 uv34 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x5636534de500 con 0x563652ad3400 2024-04-02T13:32:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.147+0000 7fa3ae02a740 20 datascan.forall_objects: OSD matched oid 100000001f7.00000000 2024-04-02T13:32:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.147+0000 7fa3ae02a740 10 datascan.operator(): handling object 100000001f7.0 2024-04-02T13:32:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.147+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:50 18.36 18:6fe38dc2:::100000001f7.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x563652acf000 con 0x563652ad3400 2024-04-02T13:32:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.1 v2:172.21.15.44:6801/704457846 6 ==== osd_op_reply(50 100000001f7.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (126) out=126b,getxattr (30) out=30b,getxattr] v0'0 uv34 ondisk = 0) ==== 416+0+194 (crc 0 0 0) 0x5636534de500 con 0x563652ad3400 2024-04-02T13:32:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ae02a740 10 datascan.inject_with_backtrace: inode: 0x0x100000001f7 2024-04-02T13:32:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ae02a740 10 datascan.inject_with_backtrace: backptr: 0x0x10000000001/bravo 2024-04-02T13:32:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ae02a740 20 datascan.get_frag_of: dirino=0x10000000001 target_dname=bravo 2024-04-02T13:32:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:51 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x563652acf400 con 0x563652ace000 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 14 ==== osd_op_reply(51 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv7 ondisk = 0) ==== 164+0+95 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ae02a740 20 datascan.get_frag_of: got backtrace parent 0x10000000000/flushed 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:52 17.10 17:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x563652afa800 con 0x563653515000 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 10 ==== osd_op_reply(52 10000000000.00000000 [omap-get-vals-by-keys out=524b] v0'0 uv12 ondisk = 0) ==== 164+0+524 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ae02a740 20 datascan.read_dentry: oid=10000000000.00000000 dname=flushed frag=*, r=0 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ae02a740 20 datascan.get_frag_of: fast path, fragtree is fragtree_t(*) 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ae02a740 20 datascan.get_frag_of: frag is * 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:53 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x563652afbc00 con 0x563652ace000 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.148+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 15 ==== osd_op_reply(53 10000000001.00000000 [omap-get-header out=274b] v0'0 uv7 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.149+0000 7fa3ae02a740 10 datascan.find_or_create_dirfrag: read_version = 7 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.149+0000 7fa3ae02a740 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000001 * 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.149+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:54 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x563652ad3800 con 0x563652ace000 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.149+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 16 ==== osd_op_reply(54 10000000001.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv7 ondisk = 0) ==== 164+0+4 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.149+0000 7fa3ae02a740 20 datascan.read_dentry: oid=10000000001.00000000 dname=bravo frag=*, r=0 2024-04-02T13:32:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.149+0000 7fa3ae02a740 20 datascan.read_dentry: bravo_head not found in result 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.149+0000 7fa3ae02a740 10 datascan.inject_with_backtrace: Linking inode 0x0x100000001f7 at 0x0x10000000001/bravo with size=5 bytes 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.149+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:55 17.7 17:e2e07930:::10000000001.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2000 con 0x563652ace000 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.150+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 17 ==== osd_op_reply(55 10000000001.00000000 [omap-set-vals] v123'9 uv9 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.150+0000 7fa3ae02a740 20 datascan.inject_linkage: Injected dentry 0x0x10000000001/bravo pointing to 0x0x100000001f7 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.150+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:56 18.e 18:70000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad3000 con 0x563652ace000 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.150+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 18 ==== osd_op_reply(56 [pgnls-filter start_epoch 123 out=49b] v121'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.150+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:57 18.2e 18:74000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653514800 con 0x563653515000 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.150+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 11 ==== osd_op_reply(57 [pgnls-filter start_epoch 123 out=49b] v121'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.151+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:58 18.1e 18:78000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653610800 con 0x563652acf800 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.151+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.5 v2:172.21.15.69:6800/2841767499 7 ==== osd_op_reply(58 [pgnls-filter start_epoch 123 out=49b] v121'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652acf800 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.151+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:59 18.3e 18:7c000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2c00 con 0x563653515000 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.151+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 12 ==== osd_op_reply(59 [pgnls-filter start_epoch 123 out=49b] v121'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.151+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:60 18.1 18:80000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ace800 con 0x563653515000 2024-04-02T13:32:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.151+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 13 ==== osd_op_reply(60 [pgnls-filter start_epoch 123 out=49b] v121'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.151+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:61 18.21 18:84000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652acfc00 con 0x563652acf800 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.152+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.5 v2:172.21.15.69:6800/2841767499 8 ==== osd_op_reply(61 [pgnls-filter start_epoch 123 out=49b] v121'24 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652acf800 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.152+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:62 18.11 18:88000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afa800 con 0x563652ad3400 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.152+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.1 v2:172.21.15.44:6801/704457846 7 ==== osd_op_reply(62 [pgnls-filter start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652ad3400 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.152+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:63 18.31 18:8c000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad3c00 con 0x563652afb400 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.152+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.4 v2:172.21.15.69:6816/1041350007 2 ==== osd_op_reply(63 [pgnls-filter start_epoch 123 out=49b] v121'15 uv8 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652afb400 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.152+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:64 18.9 18:90000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afa400 con 0x563653515000 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.153+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 14 ==== osd_op_reply(64 [pgnls-filter start_epoch 123 out=49b] v121'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.153+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:65 18.29 18:94000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2800 con 0x563652ad2400 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.153+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.7 v2:172.21.15.69:6824/1292475712 6 ==== osd_op_reply(65 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652ad2400 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.153+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:66 18.19 18:98000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afa000 con 0x563652afb400 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.153+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.4 v2:172.21.15.69:6816/1041350007 3 ==== osd_op_reply(66 [pgnls-filter start_epoch 123 out=49b] v123'24 uv24 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652afb400 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.153+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:67 18.39 18:9c000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad2c00 con 0x563652ace000 2024-04-02T13:32:00.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.154+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 19 ==== osd_op_reply(67 [pgnls-filter start_epoch 123 out=49b] v121'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.154+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:68 18.5 18:a0000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ace800 con 0x563652ad3400 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.154+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.1 v2:172.21.15.44:6801/704457846 8 ==== osd_op_reply(68 [pgnls-filter start_epoch 123 out=49b] v123'15 uv15 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652ad3400 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.154+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:69 18.25 18:a4000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad3800 con 0x563653515800 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.154+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.6 v2:172.21.15.69:6802/2604158175 6 ==== osd_op_reply(69 [pgnls-filter start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563653515800 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.154+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:70 18.15 18:a8000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652acf400 con 0x563652acf800 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.155+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.5 v2:172.21.15.69:6800/2841767499 9 ==== osd_op_reply(70 [pgnls-filter start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652acf800 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.155+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:71 18.35 18:ac000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afac00 con 0x563652afb400 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.155+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.4 v2:172.21.15.69:6816/1041350007 4 ==== osd_op_reply(71 [pgnls-filter start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652afb400 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.155+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:72 18.d 18:b0000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653610800 con 0x563652afb400 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.155+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.4 v2:172.21.15.69:6816/1041350007 5 ==== osd_op_reply(72 [pgnls-filter start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652afb400 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.155+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:73 18.2d 18:b4000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afb000 con 0x563653515800 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.155+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.6 v2:172.21.15.69:6802/2604158175 7 ==== osd_op_reply(73 [pgnls-filter start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563653515800 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.155+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:74 18.1d 18:b8000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653514800 con 0x563652ace000 2024-04-02T13:32:00.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.156+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.0 v2:172.21.15.44:6800/2303805525 20 ==== osd_op_reply(74 [pgnls-filter start_epoch 123 out=49b] v121'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563652ace000 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.156+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:75 18.3d 18:bc000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afbc00 con 0x563653515000 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.156+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.3 v2:172.21.15.44:6824/1588952513 15 ==== osd_op_reply(75 [pgnls-filter start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563653515000 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.156+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:76 18.3 18:c0000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563653525400 con 0x563652ad3400 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.156+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.1 v2:172.21.15.44:6801/704457846 9 ==== osd_op_reply(76 [pgnls-filter start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652ad3400 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.156+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:77 18.23 18:c4000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652acf000 con 0x563653515400 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.156+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.2 v2:172.21.15.44:6816/2091834331 6 ==== osd_op_reply(77 [pgnls-filter start_epoch 123 out=49b] v121'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563653515400 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.156+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:78 18.13 18:c8000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652af4000 con 0x563652ad2400 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.157+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.7 v2:172.21.15.69:6824/1292475712 7 ==== osd_op_reply(78 [pgnls-filter start_epoch 123 out=49b] v121'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652ad2400 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.157+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:79 18.33 18:cc000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652acec00 con 0x563652ad3400 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.157+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.1 v2:172.21.15.44:6801/704457846 10 ==== osd_op_reply(79 [pgnls-filter start_epoch 123 out=49b] v121'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652ad3400 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.157+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:80 18.b 18:d0000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652af4400 con 0x563652acf800 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.157+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.5 v2:172.21.15.69:6800/2841767499 10 ==== osd_op_reply(80 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652acf800 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.157+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:81 18.2b 18:d4000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ace800 con 0x563652ad2400 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.158+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.7 v2:172.21.15.69:6824/1292475712 8 ==== osd_op_reply(81 [pgnls-filter start_epoch 123 out=49b] v121'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652ad2400 2024-04-02T13:32:00.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.158+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:82 18.1b 18:d8000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652af4800 con 0x563652ad3400 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.158+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.1 v2:172.21.15.44:6801/704457846 11 ==== osd_op_reply(82 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652ad3400 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.158+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:83 18.3b 18:dc000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afb800 con 0x563652afb400 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.158+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.4 v2:172.21.15.69:6816/1041350007 6 ==== osd_op_reply(83 [pgnls-filter start_epoch 123 out=49b] v121'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652afb400 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.158+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:84 18.7 18:e0000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652af4c00 con 0x563653515400 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.159+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 <== osd.2 v2:172.21.15.44:6816/2091834331 7 ==== osd_op_reply(84 [pgnls-filter start_epoch 123 out=49b] v121'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de780 con 0x563653515400 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.159+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:85 18.27 18:e4000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afa800 con 0x563652acf800 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.159+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.5 v2:172.21.15.69:6800/2841767499 11 ==== osd_op_reply(85 [pgnls-filter start_epoch 123 out=49b] v121'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652acf800 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.159+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:86 18.17 18:e8000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652af5000 con 0x563652acf800 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.159+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.5 v2:172.21.15.69:6800/2841767499 12 ==== osd_op_reply(86 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652acf800 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.159+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:87 18.37 18:ec000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652afa000 con 0x563652ad2400 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.159+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.7 v2:172.21.15.69:6824/1292475712 9 ==== osd_op_reply(87 [pgnls-filter start_epoch 123 out=49b] v121'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652ad2400 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.159+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:88 18.f 18:f0000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652af5400 con 0x563652acf800 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.160+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.5 v2:172.21.15.69:6800/2841767499 13 ==== osd_op_reply(88 [pgnls-filter start_epoch 123 out=49b] v121'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652acf800 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.160+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:89 18.2f 18:f4000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652ad3c00 con 0x563652ad3400 2024-04-02T13:32:00.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.160+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.1 v2:172.21.15.44:6801/704457846 12 ==== osd_op_reply(89 [pgnls-filter start_epoch 123 out=49b] v121'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652ad3400 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.160+0000 7fa3a69d1640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:90 18.1f 18:f8000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652af5800 con 0x563652ad2400 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.160+0000 7fa3ab1da640 1 -- 172.21.15.44:0/846240798 <== osd.7 v2:172.21.15.69:6824/1292475712 10 ==== osd_op_reply(90 [pgnls-filter start_epoch 123 out=49b] v121'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de000 con 0x563652ad2400 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.160+0000 7fa3a71d2640 1 -- 172.21.15.44:0/846240798 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:91 18.3f 18:fc000000::::head [pgnls-filter start_epoch 123 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x563652af4000 con 0x563652afb400 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ac1dc640 1 -- 172.21.15.44:0/846240798 <== osd.4 v2:172.21.15.69:6816/1041350007 7 ==== osd_op_reply(91 [pgnls-filter start_epoch 123 out=49b] v121'17 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5636534de500 con 0x563652afb400 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x563652ace000 msgr2=0x5636528bb180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x563652ace000 0x5636528bb180 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x563652ad3400 msgr2=0x563652ade580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x563652ad3400 0x563652ade580 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x563653515400 msgr2=0x563652adeb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x563653515400 0x563652adeb00 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x563653515000 msgr2=0x5636528ba680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x563653515000 0x5636528ba680 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 reap_dead start 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x563652afb400 msgr2=0x563652adf600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x563652afb400 0x563652adf600 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563652acf800 msgr2=0x563652ade000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x563652acf800 0x563652ade000 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x563653515800 msgr2=0x5636528bac00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ab9db640 1 -- 172.21.15.44:0/846240798 reap_dead start 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x563653515800 0x5636528bac00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x563652ad2400 msgr2=0x563652adf080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x563652ad2400 0x563652adf080 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x563653514c00 msgr2=0x5636528ba100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x563653514c00 0x5636528ba100 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x56365362cde0 tx=0x563653630c00 comp rx=0 tx=0).stop 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611400 msgr2=0x5636528b9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611400 0x5636528b9080 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x56365355d0e0 tx=0x563653630de0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 shutdown_connections 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x563652ad2400 0x563652adf080 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x563653515800 0x5636528bac00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x563653514c00 0x5636528ba100 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/846240798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563653611400 0x5636528b9080 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 >> 172.21.15.44:0/846240798 conn(0x563653611000 msgr2=0x5636534cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 shutdown_connections 2024-04-02T13:32:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/846240798 wait complete. 2024-04-02T13:32:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/1801022632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563652749800 msgr2=0x5636528b8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/1801022632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563652749800 0x5636528b8580 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x56365355d9b0 tx=0x5636534e2600 comp rx=0 tx=0).stop 2024-04-02T13:32:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/1801022632 shutdown_connections 2024-04-02T13:32:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/1801022632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563652749c00 0x5636528b8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/1801022632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563652749800 0x5636528b8580 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 --2- 172.21.15.44:0/1801022632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563652748c00 0x5636528b8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/1801022632 >> 172.21.15.44:0/1801022632 conn(0x563652749000 msgr2=0x5636534cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/1801022632 shutdown_connections 2024-04-02T13:32:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.161+0000 7fa3ae02a740 1 -- 172.21.15.44:0/1801022632 wait complete. 2024-04-02T13:32:00.182 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', '--filter-tag', 'mytag123', 'cephfs_data'] in time 0:00:00.129006, result: 2024-04-02T13:32:00.182 DEBUG:teuthology.orchestra.run.smithi044:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links 2024-04-02T13:32:00.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.245+0000 7f7521e2c740 1 -- start start 2024-04-02T13:32:00.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.245+0000 7f7521e2c740 1 Processor -- start 2024-04-02T13:32:00.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.245+0000 7f7521e2c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba900f4c00 0x55ba90eb8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.245+0000 7f7521e2c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba900f5800 0x55ba90eb8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.245+0000 7f7521e2c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba900f5c00 0x55ba90eb8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.245+0000 7f7521e2c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ba90eea180 con 0x55ba900f5800 2024-04-02T13:32:00.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.245+0000 7f7521e2c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ba90154900 con 0x55ba900f5c00 2024-04-02T13:32:00.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.245+0000 7f7521e2c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ba90154480 con 0x55ba900f4c00 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.245+0000 7f751efda640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba900f5800 0x55ba90eb8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.245+0000 7f751efda640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba900f5800 0x55ba90eb8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50926/0 (socket says 172.21.15.44:50926) 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.245+0000 7f751efda640 1 -- 172.21.15.44:0/1374235508 learned_addr learned my addr 172.21.15.44:0/1374235508 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.245+0000 7f751f7db640 1 --2- 172.21.15.44:0/1374235508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba900f4c00 0x55ba90eb8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.245+0000 7f751ffdc640 1 --2- 172.21.15.44:0/1374235508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba900f5c00 0x55ba90eb8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f751efda640 1 -- 172.21.15.44:0/1374235508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba900f4c00 msgr2=0x55ba90eb8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f751efda640 1 --2- 172.21.15.44:0/1374235508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba900f4c00 0x55ba90eb8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f751efda640 1 -- 172.21.15.44:0/1374235508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba900f5c00 msgr2=0x55ba90eb8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f751efda640 1 --2- 172.21.15.44:0/1374235508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba900f5c00 0x55ba90eb8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f751efda640 1 -- 172.21.15.44:0/1374235508 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba90121680 con 0x55ba900f5800 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f751efda640 1 --2- 172.21.15.44:0/1374235508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba900f5800 0x55ba90eb8580 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x55ba90f1f9b0 tx=0x55ba90e90a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4145c91a9415fab6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f751cfd6640 1 -- 172.21.15.44:0/1374235508 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ba9011a680 con 0x55ba900f5800 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f751cfd6640 1 -- 172.21.15.44:0/1374235508 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55ba9013e1c0 con 0x55ba900f5800 2024-04-02T13:32:00.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f751cfd6640 1 -- 172.21.15.44:0/1374235508 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ba9011ab60 con 0x55ba900f5800 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f7521e2c740 1 -- 172.21.15.44:0/1374235508 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba90121860 con 0x55ba900f5800 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f7521e2c740 1 -- 172.21.15.44:0/1374235508 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55ba901212c0 con 0x55ba900f5800 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f7521e2c740 4 waiting for MDS map... 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f751cfd6640 1 -- 172.21.15.44:0/1374235508 <== mon.0 v2:172.21.15.44:3300/0 4 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x55ba90fce600 con 0x55ba900f5800 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f751cfd6640 1 -- 172.21.15.44:0/1374235508 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x55ba9014d180 con 0x55ba900f5800 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f7521e2c740 4 Got MDS map 106 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f7521e2c740 4 datascan.main: Using metadata pool output 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.246+0000 7f7521e2c740 4 datascan.main: connecting to RADOS... 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.247+0000 7f7521e2c740 1 Processor -- start 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.247+0000 7f7521e2c740 1 -- start start 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.247+0000 7f7521e2c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba90ecb400 0x55ba90eb9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.247+0000 7f7521e2c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba90ecb800 0x55ba90eb9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.247+0000 7f7521e2c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba90ecbc00 0x55ba90eb9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f7521e2c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ba90154180 con 0x55ba90ecbc00 2024-04-02T13:32:00.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f7521e2c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ba90eead80 con 0x55ba90ecb400 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f7521e2c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ba90eeaf00 con 0x55ba90ecb800 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751efda640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba90ecbc00 0x55ba90eb9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751efda640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba90ecbc00 0x55ba90eb9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50942/0 (socket says 172.21.15.44:50942) 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751efda640 1 -- 172.21.15.44:0/2707038931 learned_addr learned my addr 172.21.15.44:0/2707038931 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751ffdc640 1 --2- 172.21.15.44:0/2707038931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba90ecb400 0x55ba90eb9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751f7db640 1 --2- 172.21.15.44:0/2707038931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba90ecb800 0x55ba90eb9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751efda640 1 -- 172.21.15.44:0/2707038931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba90ecb800 msgr2=0x55ba90eb9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751efda640 1 --2- 172.21.15.44:0/2707038931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba90ecb800 0x55ba90eb9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751efda640 1 -- 172.21.15.44:0/2707038931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba90ecb400 msgr2=0x55ba90eb9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751efda640 1 --2- 172.21.15.44:0/2707038931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba90ecb400 0x55ba90eb9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751efda640 1 -- 172.21.15.44:0/2707038931 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba901214a0 con 0x55ba90ecbc00 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751ffdc640 1 --2- 172.21.15.44:0/2707038931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba90ecb400 0x55ba90eb9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751efda640 1 --2- 172.21.15.44:0/2707038931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba90ecbc00 0x55ba90eb9b80 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x55ba90fdede0 tx=0x55ba90e8e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d1db9d3a0b19ffd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751afd2640 1 -- 172.21.15.44:0/2707038931 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ba9011a820 con 0x55ba90ecbc00 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751afd2640 1 -- 172.21.15.44:0/2707038931 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55ba9013ee00 con 0x55ba90ecbc00 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.248+0000 7f751afd2640 1 -- 172.21.15.44:0/2707038931 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ba9011a680 con 0x55ba90ecbc00 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.249+0000 7f7521e2c740 1 -- 172.21.15.44:0/2707038931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba90ecbc00 msgr2=0x55ba90eb9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.249+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2707038931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba90ecbc00 0x55ba90eb9b80 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x55ba90fdede0 tx=0x55ba90e8e9c0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.249+0000 7f7521e2c740 1 -- 172.21.15.44:0/2707038931 shutdown_connections 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.249+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2707038931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba90ecbc00 0x55ba90eb9b80 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.249+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2707038931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba90ecb800 0x55ba90eb9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.249+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2707038931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba90ecb400 0x55ba90eb9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.249+0000 7f7521e2c740 1 -- 172.21.15.44:0/2707038931 >> 172.21.15.44:0/2707038931 conn(0x55ba90ecb000 msgr2=0x55ba901d9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.249+0000 7f7521e2c740 1 -- 172.21.15.44:0/2707038931 shutdown_connections 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.249+0000 7f7521e2c740 1 -- 172.21.15.44:0/2707038931 wait complete. 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.249+0000 7f7521e2c740 1 Processor -- start 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.249+0000 7f7521e2c740 1 -- start start 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f7521e2c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba90ecb400 0x55ba90eb9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f7521e2c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba90ecb800 0x55ba90eb9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f7521e2c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba90ecbc00 0x55ba90eb9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f7521e2c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ba90eeb200 con 0x55ba90ecbc00 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f7521e2c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ba90eeb080 con 0x55ba90ecb800 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f7521e2c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x55ba90eeb380 con 0x55ba90ecb400 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f751efda640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba90ecbc00 0x55ba90eb9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f751efda640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba90ecbc00 0x55ba90eb9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50950/0 (socket says 172.21.15.44:50950) 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 learned_addr learned my addr 172.21.15.44:0/2879979117 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f751f7db640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba90ecb800 0x55ba90eb9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f751ffdc640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba90ecb400 0x55ba90eb9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba90ecb400 msgr2=0x55ba90eb9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f751efda640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba90ecb400 0x55ba90eb9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba90ecb800 msgr2=0x55ba90eb9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f751efda640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba90ecb800 0x55ba90eb9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba90121680 con 0x55ba90ecbc00 2024-04-02T13:32:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f751f7db640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba90ecb800 0x55ba90eb9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:32:00.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f751efda640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba90ecbc00 0x55ba90eb9b80 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x55ba90fdfaa0 tx=0x55ba90fe2c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f75197cf640 1 -- 172.21.15.44:0/2879979117 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ba9011ab60 con 0x55ba90ecbc00 2024-04-02T13:32:00.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f75197cf640 1 -- 172.21.15.44:0/2879979117 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55ba9013ec40 con 0x55ba90ecbc00 2024-04-02T13:32:00.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ba90121860 con 0x55ba90ecbc00 2024-04-02T13:32:00.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.250+0000 7f75197cf640 1 -- 172.21.15.44:0/2879979117 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ba9011a4e0 con 0x55ba90ecbc00 2024-04-02T13:32:00.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.251+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba901210e0 con 0x55ba90ecbc00 2024-04-02T13:32:00.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.251+0000 7f7521e2c740 4 datascan.init: resolving metadata pool 17 2024-04-02T13:32:00.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.252+0000 7f75197cf640 1 -- 172.21.15.44:0/2879979117 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x55ba90ee2c00 con 0x55ba90ecbc00 2024-04-02T13:32:00.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.252+0000 7f75197cf640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55ba90eff000 0x55ba90eba100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.252+0000 7f75197cf640 1 -- 172.21.15.44:0/2879979117 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x55ba9103c600 con 0x55ba90ecbc00 2024-04-02T13:32:00.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.252+0000 7f7521e2c740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-04-02T13:32:00.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.252+0000 7f7521e2c740 4 datascan.main: resolving metadata pool 17 2024-04-02T13:32:00.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.252+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55ba90eff400 0x55ba90eba680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.252+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90eff800 con 0x55ba90eff400 2024-04-02T13:32:00.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.252+0000 7f751ffdc640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55ba90eff000 0x55ba90eba100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.252+0000 7f751f7db640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55ba90eff400 0x55ba90eba680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.253+0000 7f751ffdc640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55ba90eff000 0x55ba90eba100 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x55ba90e7cde0 tx=0x55ba90402ae0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.253+0000 7f751f7db640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55ba90eff400 0x55ba90eba680 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.253+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 1 ==== osd_op_reply(1 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.253+0000 7f751f7db640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x55ba90eff800 0x55ba9047a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.253+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:2 17.10 17:08000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90eec800 con 0x55ba90eff800 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.253+0000 7f751ffdc640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x55ba90eff800 0x55ba9047a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.254+0000 7f751ffdc640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x55ba90eff800 0x55ba9047a000 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.254+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 1 ==== osd_op_reply(2 [pgnls start_epoch 123 out=177b] v122'16 uv16 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.254+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:3 17.8 17:10000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90eec800 con 0x55ba90eff400 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.254+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 2 ==== osd_op_reply(3 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.254+0000 7f751f7db640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55ba90eec800 0x55ba9047a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.254+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:4 17.18 17:18000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90eed000 con 0x55ba90eec800 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.254+0000 7f751f7db640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55ba90eec800 0x55ba9047a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.255+0000 7f751f7db640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55ba90eec800 0x55ba9047a580 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.255+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 1 ==== osd_op_reply(4 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.255+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:5 17.4 17:20000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90eed000 con 0x55ba90eec800 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.255+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 2 ==== osd_op_reply(5 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.255+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:6 17.14 17:28000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90eed000 con 0x55ba90eff800 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.255+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 2 ==== osd_op_reply(6 [pgnls start_epoch 123 out=149b] v123'44 uv44 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.255+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:7 17.c 17:30000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90eed000 con 0x55ba90eff400 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.256+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 3 ==== osd_op_reply(7 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.256+0000 7f751f7db640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55ba90eed000 0x55ba9047ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.256+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:8 17.1c 17:38000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90eed400 con 0x55ba90eed000 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.256+0000 7f751efda640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55ba90eed000 0x55ba9047ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.256+0000 7f751efda640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55ba90eed000 0x55ba9047ab00 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.256+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== osd_op_reply(8 [pgnls start_epoch 123 out=73b] v122'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90eed000 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.256+0000 7f751efda640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x55ba90eed400 0x55ba9047b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.256+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:9 17.2 17:40000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efe800 con 0x55ba90eed400 2024-04-02T13:32:00.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.257+0000 7f751ffdc640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x55ba90eed400 0x55ba9047b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.257+0000 7f751ffdc640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x55ba90eed400 0x55ba9047b080 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.257+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.4 v2:172.21.15.69:6816/1041350007 1 ==== osd_op_reply(9 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eed400 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.257+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:10 17.12 17:48000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efe800 con 0x55ba90eff800 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.257+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 3 ==== osd_op_reply(10 [pgnls start_epoch 123 out=81b] v122'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.257+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:11 17.a 17:50000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efe800 con 0x55ba90eff400 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.258+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 4 ==== osd_op_reply(11 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.258+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:12 17.1a 17:58000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efe800 con 0x55ba90eec800 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.258+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 3 ==== osd_op_reply(12 [pgnls start_epoch 123 out=49b] v123'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.258+0000 7f751f7db640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55ba90efe800 0x55ba9047b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.258+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:13 17.6 17:60000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90476c00 con 0x55ba90efe800 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.258+0000 7f751f7db640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55ba90efe800 0x55ba9047b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.259+0000 7f751f7db640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55ba90efe800 0x55ba9047b600 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.259+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== osd_op_reply(13 [pgnls start_epoch 123 out=77b] v122'10 uv10 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90efe800 2024-04-02T13:32:00.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.259+0000 7f751f7db640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55ba90476c00 0x55ba9047bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.259+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:14 17.16 17:68000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477000 con 0x55ba90476c00 2024-04-02T13:32:00.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.259+0000 7f751efda640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55ba90476c00 0x55ba9047bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.260+0000 7f751efda640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55ba90476c00 0x55ba9047bb80 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.260+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.5 v2:172.21.15.69:6800/2841767499 1 ==== osd_op_reply(14 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90476c00 2024-04-02T13:32:00.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.260+0000 7f751efda640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x55ba90477000 0x55ba9047c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.260+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:15 17.e 17:70000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90477000 2024-04-02T13:32:00.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.260+0000 7f751ffdc640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x55ba90477000 0x55ba9047c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.260+0000 7f751ffdc640 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x55ba90477000 0x55ba9047c680 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.261+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.2 v2:172.21.15.44:6816/2091834331 1 ==== osd_op_reply(15 [pgnls start_epoch 123 out=73b] v122'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90477000 2024-04-02T13:32:00.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.261+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:16 17.1e 17:78000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90eed000 2024-04-02T13:32:00.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.261+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.1 v2:172.21.15.44:6801/704457846 2 ==== osd_op_reply(16 [pgnls start_epoch 123 out=49b] v123'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90eed000 2024-04-02T13:32:00.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.261+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:17 17.1 17:80000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90eec800 2024-04-02T13:32:00.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.261+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 4 ==== osd_op_reply(17 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.261+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:18 17.11 17:88000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90eff800 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.261+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 4 ==== osd_op_reply(18 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.261+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:19 17.9 17:90000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90eff800 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.262+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 5 ==== osd_op_reply(19 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.262+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:20 17.19 17:98000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90476c00 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.262+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.5 v2:172.21.15.69:6800/2841767499 2 ==== osd_op_reply(20 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90476c00 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.262+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:21 17.5 17:a0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90efe800 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.262+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.6 v2:172.21.15.69:6802/2604158175 2 ==== osd_op_reply(21 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90efe800 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.262+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:22 17.15 17:a8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90476c00 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.262+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.5 v2:172.21.15.69:6800/2841767499 3 ==== osd_op_reply(22 [pgnls start_epoch 123 out=97b] v122'4 uv4 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90476c00 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.262+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:23 17.d 17:b0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90eff400 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.263+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 5 ==== osd_op_reply(23 [pgnls start_epoch 123 out=74b] v120'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.263+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:24 17.1d 17:b8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90eff400 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.263+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 6 ==== osd_op_reply(24 [pgnls start_epoch 123 out=73b] v122'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.263+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:25 17.3 17:c0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90eed400 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.263+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.4 v2:172.21.15.69:6816/1041350007 2 ==== osd_op_reply(25 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eed400 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.263+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:26 17.13 17:c8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90eff800 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.263+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 6 ==== osd_op_reply(26 [pgnls start_epoch 123 out=98b] v122'5 uv5 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.264+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:27 17.b 17:d0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90eec800 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.264+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 5 ==== osd_op_reply(27 [pgnls start_epoch 123 out=76b] v123'6 uv5 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.264+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:28 17.1b 17:d8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90eed400 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.264+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.4 v2:172.21.15.69:6816/1041350007 3 ==== osd_op_reply(28 [pgnls start_epoch 123 out=49b] v123'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eed400 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.264+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:29 17.7 17:e0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90eec800 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.264+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 6 ==== osd_op_reply(29 [pgnls start_epoch 123 out=105b] v123'9 uv9 ondisk = 1) ==== 144+0+105 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.264+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:30 17.17 17:e8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90efe800 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.265+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.6 v2:172.21.15.69:6802/2604158175 3 ==== osd_op_reply(30 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90efe800 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.265+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:31 17.f 17:f0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90eed000 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.265+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.1 v2:172.21.15.44:6801/704457846 3 ==== osd_op_reply(31 [pgnls start_epoch 123 out=73b] v123'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90eed000 2024-04-02T13:32:00.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.265+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:32 17.1f 17:f8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90efe800 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.265+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.6 v2:172.21.15.69:6802/2604158175 4 ==== osd_op_reply(32 [pgnls start_epoch 123 out=101b] v123'9 uv8 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90efe800 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.265+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 601.00000000 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.265+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:33 17.10 17:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba90effc00 con 0x55ba90eff800 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.265+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 7 ==== osd_op_reply(33 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.265+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 602.00000000 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.265+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:34 17.10 17:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eff800 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.266+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 8 ==== osd_op_reply(34 602.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.266+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.266+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:35 17.10 17:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a0000 con 0x55ba90eff800 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.266+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 9 ==== osd_op_reply(35 10000000000.00000000 [omap-get-vals out=1047b] v0'0 uv12 ondisk = 0) ==== 164+0+1047 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.266+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 600.00000000 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.266+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:36 17.10 17:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a0400 con 0x55ba90eff800 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.266+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 10 ==== osd_op_reply(36 600.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.266+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 603.00000000 2024-04-02T13:32:00.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.266+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:37 17.10 17:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a0800 con 0x55ba90eff800 2024-04-02T13:32:00.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 11 ==== osd_op_reply(37 603.00000000 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-04-02T13:32:00.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-04-02T13:32:00.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 200.00000000 2024-04-02T13:32:00.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-04-02T13:32:00.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 200.00000001 2024-04-02T13:32:00.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-04-02T13:32:00.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 606.00000000 2024-04-02T13:32:00.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:38 17.14 17:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a0c00 con 0x55ba90eff800 2024-04-02T13:32:00.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 12 ==== osd_op_reply(38 606.00000000 [omap-get-vals out=5b] v0'0 uv39 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 607.00000000 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:39 17.1c 17:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a1000 con 0x55ba90eed000 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.1 v2:172.21.15.44:6801/704457846 4 ==== osd_op_reply(39 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90eed000 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 10000000003.00000000 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:40 17.12 17:4def72d4:::10000000003.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a1400 con 0x55ba90eff800 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 13 ==== osd_op_reply(40 10000000003.00000000 [omap-get-vals out=523b] v0'0 uv3 ondisk = 0) ==== 164+0+523 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object mds0_openfiles.0 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 608.00000000 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.267+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:41 17.e 17:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a1800 con 0x55ba90477000 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.2 v2:172.21.15.44:6816/2091834331 2 ==== osd_op_reply(41 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90477000 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 604.00000000 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:42 17.15 17:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a0400 con 0x55ba90476c00 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.5 v2:172.21.15.69:6800/2841767499 4 ==== osd_op_reply(42 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90476c00 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 500.00000000 2024-04-02T13:32:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 605.00000000 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:43 17.1d 17:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a0800 con 0x55ba90eff400 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 7 ==== osd_op_reply(43 605.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 100.00000000 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.268+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:44 17.13 17:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a0c00 con 0x55ba90eff800 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 14 ==== osd_op_reply(44 100.00000000 [omap-get-vals out=5195b] v0'0 uv5 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 609.00000000 2024-04-02T13:32:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:45 17.7 17:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a1400 con 0x55ba90eec800 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 7 ==== osd_op_reply(45 609.00000000 [omap-get-vals out=5b] v0'0 uv8 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 10000000001.00000000 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:46 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a1800 con 0x55ba90eec800 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 8 ==== osd_op_reply(46 10000000001.00000000 [omap-get-vals out=1545b] v0'0 uv9 ondisk = 0) ==== 164+0+1545 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f7521e2c740 20 datascan.scan_links: injected ino detected 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 400.00000000 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f7521e2c740 10 datascan.scan_links: step 1: handling object 1.00000000 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.269+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:47 17.1f 17:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba90476800 con 0x55ba90efe800 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.270+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.6 v2:172.21.15.69:6802/2604158175 5 ==== osd_op_reply(47 1.00000000 [omap-get-vals out=524b] v0'0 uv8 ondisk = 0) ==== 154+0+524 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90efe800 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.270+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:48 17.0 17:00000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eff400 2024-04-02T13:32:00.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.270+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 8 ==== osd_op_reply(48 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.270+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:49 17.10 17:08000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eff800 2024-04-02T13:32:00.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.271+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 15 ==== osd_op_reply(49 [pgnls start_epoch 123 out=177b] v122'16 uv16 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.271+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:50 17.8 17:10000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eff400 2024-04-02T13:32:00.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.271+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 9 ==== osd_op_reply(50 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.271+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:51 17.18 17:18000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eec800 2024-04-02T13:32:00.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.271+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 9 ==== osd_op_reply(51 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.271+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:52 17.4 17:20000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eec800 2024-04-02T13:32:00.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.272+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 10 ==== osd_op_reply(52 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.272+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:53 17.14 17:28000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eff800 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.272+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 16 ==== osd_op_reply(53 [pgnls start_epoch 123 out=149b] v123'44 uv44 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.272+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:54 17.c 17:30000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eff400 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.272+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 10 ==== osd_op_reply(54 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.272+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:55 17.1c 17:38000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eed000 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.272+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.1 v2:172.21.15.44:6801/704457846 5 ==== osd_op_reply(55 [pgnls start_epoch 123 out=73b] v122'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90eed000 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.272+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:56 17.2 17:40000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eed400 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.273+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.4 v2:172.21.15.69:6816/1041350007 4 ==== osd_op_reply(56 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eed400 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.273+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:57 17.12 17:48000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eff800 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.273+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 17 ==== osd_op_reply(57 [pgnls start_epoch 123 out=81b] v122'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.273+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:58 17.a 17:50000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eff400 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.274+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 11 ==== osd_op_reply(58 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.274+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:59 17.1a 17:58000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eec800 2024-04-02T13:32:00.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.274+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 11 ==== osd_op_reply(59 [pgnls start_epoch 123 out=49b] v123'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.274+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:60 17.6 17:60000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90efe800 2024-04-02T13:32:00.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.274+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.6 v2:172.21.15.69:6802/2604158175 6 ==== osd_op_reply(60 [pgnls start_epoch 123 out=77b] v122'10 uv10 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90efe800 2024-04-02T13:32:00.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.274+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:61 17.16 17:68000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90476c00 2024-04-02T13:32:00.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.274+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.5 v2:172.21.15.69:6800/2841767499 5 ==== osd_op_reply(61 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90476c00 2024-04-02T13:32:00.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.274+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:62 17.e 17:70000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90477000 2024-04-02T13:32:00.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.275+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.2 v2:172.21.15.44:6816/2091834331 3 ==== osd_op_reply(62 [pgnls start_epoch 123 out=73b] v122'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90477000 2024-04-02T13:32:00.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.275+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:63 17.1e 17:78000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eed000 2024-04-02T13:32:00.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.275+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.1 v2:172.21.15.44:6801/704457846 6 ==== osd_op_reply(63 [pgnls start_epoch 123 out=49b] v123'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90eed000 2024-04-02T13:32:00.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.275+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:64 17.1 17:80000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eec800 2024-04-02T13:32:00.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.275+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 12 ==== osd_op_reply(64 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.275+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:65 17.11 17:88000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eff800 2024-04-02T13:32:00.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.275+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 18 ==== osd_op_reply(65 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.275+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:66 17.9 17:90000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eff800 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.276+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 19 ==== osd_op_reply(66 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.276+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:67 17.19 17:98000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90476c00 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.276+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.5 v2:172.21.15.69:6800/2841767499 6 ==== osd_op_reply(67 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90476c00 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.276+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:68 17.5 17:a0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90efe800 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.276+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.6 v2:172.21.15.69:6802/2604158175 7 ==== osd_op_reply(68 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90efe800 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.276+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:69 17.15 17:a8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90476c00 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.276+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.5 v2:172.21.15.69:6800/2841767499 7 ==== osd_op_reply(69 [pgnls start_epoch 123 out=97b] v122'4 uv4 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90476c00 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.276+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:70 17.d 17:b0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eff400 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.277+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 12 ==== osd_op_reply(70 [pgnls start_epoch 123 out=74b] v120'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.277+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:71 17.1d 17:b8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eff400 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.277+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 13 ==== osd_op_reply(71 [pgnls start_epoch 123 out=73b] v122'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.277+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:72 17.3 17:c0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eed400 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.277+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.4 v2:172.21.15.69:6816/1041350007 5 ==== osd_op_reply(72 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eed400 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.277+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:73 17.13 17:c8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eff800 2024-04-02T13:32:00.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.278+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 20 ==== osd_op_reply(73 [pgnls start_epoch 123 out=98b] v122'5 uv5 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.278+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:74 17.b 17:d0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eec800 2024-04-02T13:32:00.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.278+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 13 ==== osd_op_reply(74 [pgnls start_epoch 123 out=76b] v123'6 uv5 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.278+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- osd_op(unknown.0.0:75 17.1b 17:d8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eed400 2024-04-02T13:32:00.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.278+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.4 v2:172.21.15.69:6816/1041350007 6 ==== osd_op_reply(75 [pgnls start_epoch 123 out=49b] v123'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eed400 2024-04-02T13:32:00.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.278+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:76 17.7 17:e0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eec800 2024-04-02T13:32:00.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.278+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 14 ==== osd_op_reply(76 [pgnls start_epoch 123 out=105b] v123'9 uv9 ondisk = 1) ==== 144+0+105 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.278+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:77 17.17 17:e8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90efe800 2024-04-02T13:32:00.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.279+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.6 v2:172.21.15.69:6802/2604158175 8 ==== osd_op_reply(77 [pgnls start_epoch 123 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90efe800 2024-04-02T13:32:00.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.279+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:78 17.f 17:f0000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90eed000 2024-04-02T13:32:00.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.279+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.1 v2:172.21.15.44:6801/704457846 7 ==== osd_op_reply(78 [pgnls start_epoch 123 out=73b] v123'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90eed000 2024-04-02T13:32:00.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.279+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:79 17.1f 17:f8000000::::head [pgnls start_epoch 123 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477c00 con 0x55ba90efe800 2024-04-02T13:32:00.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.279+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.6 v2:172.21.15.69:6802/2604158175 9 ==== osd_op_reply(79 [pgnls start_epoch 123 out=101b] v123'9 uv8 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90efe800 2024-04-02T13:32:00.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.279+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 601.00000000 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.279+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:80 17.10 17:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba90effc00 con 0x55ba90eff800 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.279+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 21 ==== osd_op_reply(80 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.279+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 602.00000000 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.279+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:81 17.10 17:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a0800 con 0x55ba90eff800 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 22 ==== osd_op_reply(81 602.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:82 17.10 17:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a1400 con 0x55ba90eff800 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 23 ==== osd_op_reply(82 10000000000.00000000 [omap-get-vals out=1047b] v0'0 uv12 ondisk = 0) ==== 164+0+1047 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 600.00000000 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:83 17.10 17:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a1800 con 0x55ba90eff800 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 24 ==== osd_op_reply(83 600.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 603.00000000 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:84 17.10 17:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a0000 con 0x55ba90eff800 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 25 ==== osd_op_reply(84 603.00000000 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 200.00000000 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 200.00000001 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 606.00000000 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.280+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:85 17.14 17:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a1c00 con 0x55ba90eff800 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.281+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 26 ==== osd_op_reply(85 606.00000000 [omap-get-vals out=5b] v0'0 uv39 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.281+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 607.00000000 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.281+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- osd_op(unknown.0.0:86 17.1c 17:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba90efec00 con 0x55ba90eed000 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.281+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.1 v2:172.21.15.44:6801/704457846 8 ==== osd_op_reply(86 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90eed000 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.281+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 10000000003.00000000 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.281+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:87 17.12 17:4def72d4:::10000000003.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477800 con 0x55ba90eff800 2024-04-02T13:32:00.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.281+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 27 ==== osd_op_reply(87 10000000003.00000000 [omap-get-vals out=523b] v0'0 uv3 ondisk = 0) ==== 164+0+523 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.281+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object mds0_openfiles.0 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.281+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.281+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 608.00000000 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.281+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- osd_op(unknown.0.0:88 17.e 17:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba90477400 con 0x55ba90477000 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.2 v2:172.21.15.44:6816/2091834331 4 ==== osd_op_reply(88 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90477000 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 604.00000000 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- osd_op(unknown.0.0:89 17.15 17:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a0c00 con 0x55ba90476c00 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f751efda640 1 -- 172.21.15.44:0/2879979117 <== osd.5 v2:172.21.15.69:6800/2841767499 8 ==== osd_op_reply(89 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86780 con 0x55ba90476c00 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 500.00000000 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 605.00000000 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:90 17.1d 17:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a0800 con 0x55ba90eff400 2024-04-02T13:32:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 14 ==== osd_op_reply(90 605.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-04-02T13:32:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-04-02T13:32:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 100.00000000 2024-04-02T13:32:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.282+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:91 17.13 17:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a1400 con 0x55ba90eff800 2024-04-02T13:32:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 28 ==== osd_op_reply(91 100.00000000 [omap-get-vals out=5195b] v0'0 uv5 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-04-02T13:32:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-04-02T13:32:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 609.00000000 2024-04-02T13:32:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:92 17.7 17:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a1800 con 0x55ba90eec800 2024-04-02T13:32:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 15 ==== osd_op_reply(92 609.00000000 [omap-get-vals out=5b] v0'0 uv8 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 10000000001.00000000 2024-04-02T13:32:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:93 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904a0000 con 0x55ba90eec800 2024-04-02T13:32:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 16 ==== osd_op_reply(93 10000000001.00000000 [omap-get-vals out=1545b] v0'0 uv9 ondisk = 0) ==== 164+0+1545 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 20 datascan.scan_links: injected ino detected 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 20 datascan.scan_links: adding 0x100000001f7 for future processing to fix dnfirst 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 400.00000000 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 10 datascan.scan_links: step 2: handling object 1.00000000 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.283+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- osd_op(unknown.0.0:94 17.1f 17:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba90effc00 con 0x55ba90efe800 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.284+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.6 v2:172.21.15.69:6802/2604158175 10 ==== osd_op_reply(94 1.00000000 [omap-get-vals out=524b] v0'0 uv8 ondisk = 0) ==== 154+0+524 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90efe800 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.284+0000 7f7521e2c740 10 datascan.scan_links: processing 0 dup_primaries, 0 remote_links 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.284+0000 7f7521e2c740 10 datascan.scan_links: removing dup dentries from 0 objects 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.284+0000 7f7521e2c740 10 datascan.scan_links: processing 0 bad_nlink_inos 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.284+0000 7f7521e2c740 10 datascan.scan_links: processing 1 injected_inos 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.284+0000 7f7521e2c740 10 datascan.scan_links: handling injected_ino 0x100000001f7 2024-04-02T13:32:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.284+0000 7f7521e2c740 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000001:bravo 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.284+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:95 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904b0000 con 0x55ba90eec800 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.284+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 17 ==== osd_op_reply(95 10000000001.00000000 [omap-get-vals-by-keys out=506b] v0'0 uv9 ondisk = 0) ==== 164+0+506 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.284+0000 7f7521e2c740 20 datascan.read_dentry: oid=10000000001.00000000 dname=bravo frag=*, r=0 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.284+0000 7f7521e2c740 20 datascan.scan_links: first is now 2 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.284+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- osd_op(unknown.0.0:96 17.7 17:e2e07930:::10000000001.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x55ba904b0400 con 0x55ba90eec800 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.285+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.0 v2:172.21.15.44:6800/2303805525 18 ==== osd_op_reply(96 10000000001.00000000 [omap-set-vals] v123'10 uv10 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eec800 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.285+0000 7f7521e2c740 20 datascan.inject_linkage: Injected dentry 0x0x10000000001/bravo pointing to 0x0x100000001f7 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.285+0000 7f7521e2c740 10 datascan.scan_links: updating inotable 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.285+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:97 17.13 17:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904b0800 con 0x55ba90eff800 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.286+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 29 ==== osd_op_reply(97 mds0_inotable [read 0~50 out=50b] v0'0 uv4 ondisk = 0) ==== 157+0+50 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.286+0000 7f7521e2c740 10 mds.0.inotable: skip_inos was [0x10000000005~0x1f2,0x100000001f9~0xfffffffe07] 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.286+0000 7f7521e2c740 10 mds.0.inotable: skip_inos now [0x100000001f9~0xfffffffe07] 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.286+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- osd_op(unknown.0.0:98 17.13 17:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x55ba904b0c00 con 0x55ba90eff800 2024-04-02T13:32:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f751ffdc640 1 -- 172.21.15.44:0/2879979117 <== osd.3 v2:172.21.15.44:6824/1588952513 30 ==== osd_op_reply(98 mds0_inotable [writefull 0~34] v123'6 uv6 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x55ba90e86500 con 0x55ba90eff800 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 10 datascan.scan_links: updating snaptable 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- osd_op(unknown.0.0:99 17.d 17:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e123) -- 0x55ba904b1000 con 0x55ba90eff400 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 <== osd.7 v2:172.21.15.69:6824/1292475712 15 ==== osd_op_reply(99 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) ==== 157+0+70 (crc 0 0 0) 0x55ba90e86000 con 0x55ba90eff400 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55ba90eec800 msgr2=0x55ba9047a580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x55ba90eec800 0x55ba9047a580 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55ba90eed000 msgr2=0x55ba9047ab00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x55ba90eed000 0x55ba9047ab00 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x55ba90477000 msgr2=0x55ba9047c680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x55ba90477000 0x55ba9047c680 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x55ba90eff800 msgr2=0x55ba9047a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x55ba90eff800 0x55ba9047a000 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x55ba90eed400 msgr2=0x55ba9047b080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 reap_dead start 2024-04-02T13:32:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x55ba90eed400 0x55ba9047b080 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 reap_dead start 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55ba90476c00 msgr2=0x55ba9047bb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x55ba90476c00 0x55ba9047bb80 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55ba90efe800 msgr2=0x55ba9047b600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x55ba90efe800 0x55ba9047b600 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55ba90eff400 msgr2=0x55ba90eba680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x55ba90eff400 0x55ba90eba680 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55ba90eff000 msgr2=0x55ba90eba100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x55ba90eff000 0x55ba90eba100 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x55ba90e7cde0 tx=0x55ba90402ae0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba90ecbc00 msgr2=0x55ba90eb9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 --2- 172.21.15.44:0/2879979117 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba90ecbc00 0x55ba90eb9b80 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x55ba90fdfaa0 tx=0x55ba90fe2c00 comp rx=0 tx=0).stop 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 reap_dead start 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f751f7db640 1 -- 172.21.15.44:0/2879979117 reap_dead start 2024-04-02T13:32:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 shutdown_connections 2024-04-02T13:32:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.287+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 >> 172.21.15.44:0/2879979117 conn(0x55ba90ecb000 msgr2=0x55ba901d9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.288+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 shutdown_connections 2024-04-02T13:32:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.288+0000 7f7521e2c740 1 -- 172.21.15.44:0/2879979117 wait complete. 2024-04-02T13:32:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.288+0000 7f7521e2c740 1 -- 172.21.15.44:0/1374235508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba900f5800 msgr2=0x55ba90eb8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.288+0000 7f7521e2c740 1 --2- 172.21.15.44:0/1374235508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba900f5800 0x55ba90eb8580 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x55ba90f1f9b0 tx=0x55ba90e90a80 comp rx=0 tx=0).stop 2024-04-02T13:32:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.288+0000 7f7521e2c740 1 -- 172.21.15.44:0/1374235508 shutdown_connections 2024-04-02T13:32:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.288+0000 7f7521e2c740 1 --2- 172.21.15.44:0/1374235508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ba900f5c00 0x55ba90eb8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.288+0000 7f7521e2c740 1 --2- 172.21.15.44:0/1374235508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba900f5800 0x55ba90eb8580 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.288+0000 7f7521e2c740 1 --2- 172.21.15.44:0/1374235508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ba900f4c00 0x55ba90eb8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.288+0000 7f7521e2c740 1 -- 172.21.15.44:0/1374235508 >> 172.21.15.44:0/1374235508 conn(0x55ba900f5000 msgr2=0x55ba901d8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.288+0000 7f7521e2c740 1 -- 172.21.15.44:0/1374235508 shutdown_connections 2024-04-02T13:32:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.288+0000 7f7521e2c740 1 -- 172.21.15.44:0/1374235508 wait complete. 2024-04-02T13:32:00.311 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links'] in time 0:00:00.129638, result: 2024-04-02T13:32:00.312 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T13:32:00.341 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T13:32:00.389 INFO:teuthology.orchestra.run.smithi044.stdout:[global] 2024-04-02T13:32:00.389 INFO:teuthology.orchestra.run.smithi044.stdout: chdir = "" 2024-04-02T13:32:00.389 INFO:teuthology.orchestra.run.smithi044.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T13:32:00.389 INFO:teuthology.orchestra.run.smithi044.stdout: auth supported = cephx 2024-04-02T13:32:00.389 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.389 INFO:teuthology.orchestra.run.smithi044.stdout: filestore xattr use omap = true 2024-04-02T13:32:00.389 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.389 INFO:teuthology.orchestra.run.smithi044.stdout: mon clock drift allowed = 1.000 2024-04-02T13:32:00.389 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.389 INFO:teuthology.orchestra.run.smithi044.stdout: osd crush chooseleaf type = 0 2024-04-02T13:32:00.390 INFO:teuthology.orchestra.run.smithi044.stdout: auth debug = true 2024-04-02T13:32:00.390 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.390 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on old message = true 2024-04-02T13:32:00.390 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on bug = true 2024-04-02T13:32:00.390 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.390 INFO:teuthology.orchestra.run.smithi044.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T13:32:00.390 INFO:teuthology.orchestra.run.smithi044.stdout: mon pg warn max object skew = 0 2024-04-02T13:32:00.390 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.390 INFO:teuthology.orchestra.run.smithi044.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T13:32:00.390 INFO:teuthology.orchestra.run.smithi044.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T13:32:00.390 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.390 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default size = 2 2024-04-02T13:32:00.390 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow primary affinity = true 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow pg remap = true 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on legacy crush tunables = false 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on crush straw calc version zero = false 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on no sortbitwise = false 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on osd down out interval zero = false 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on too few osds = false 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: mon_allow_pool_size_one = true 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: osd default data pool replay window = 5 2024-04-02T13:32:00.391 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout: mon allow pool delete = true 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout: mon cluster log file level = debug 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout: debug asserts on shutdown = true 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout: mon health detail to clog = false 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout: mon host = "172.21.15.44,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout: fsid = 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout:[osd] 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout: osd journal size = 100 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub load threshold = 5.0 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub max interval = 600 2024-04-02T13:32:00.392 INFO:teuthology.orchestra.run.smithi044.stdout: osd mclock profile = high_recovery_ops 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: osd recover clone overlap = true 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: osd recovery max chunk = 1048576 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug shutdown = true 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug op order = true 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug verify stray on activate = true 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug trim objects = true 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: osd open classes on start = true 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug pg log writeout = true 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: osd deep scrub update digest min age = 30 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.393 INFO:teuthology.orchestra.run.smithi044.stdout: osd map max advance = 10 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: journal zero on create = true 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: filestore ondisk finisher threads = 3 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: filestore apply finisher threads = 3 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: bdev debug aio = true 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug misdirected ops = true 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore block size = 96636764160 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore fsck on mount = True 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluefs = 20 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluestore = 20 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: debug osd = 20 2024-04-02T13:32:00.394 INFO:teuthology.orchestra.run.smithi044.stdout: debug rocksdb = 10 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd full ratio = 0.9 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd nearfull ratio = 0.8 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: osd failsafe full ratio = 0.95 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: osd objectstore = bluestore 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: osd op complaint time = 180 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout:[mgr] 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: debug mgr = 20 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:32:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: mgr/telemetry/nag = false 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: debug client = 20 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout:[mon] 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: debug paxos = 20 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: mon data avail warn = 5 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: mon mgr mkfs grace = 240 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd reporter subtree level = osd 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd prime pg temp = true 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T13:32:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: # 1m isn't quite enough 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: mon_down_mkfs_grace = 2m 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_filestore_osds = false 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: mon down mkfs grace = 300 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: mon op complaint time = 120 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout:[client] 2024-04-02T13:32:00.397 INFO:teuthology.orchestra.run.smithi044.stdout: rgw cache enabled = true 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable ops log = true 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable usage log = true 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout: client mount timeout = 600 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout: debug client = 20 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout: pid file = "" 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout: rados mon op timeout = 900 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout: rados osd op timeout = 900 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.a] 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.b] 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.c] 2024-04-02T13:32:00.398 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.a] 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.c] 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.b] 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.d] 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout:[mds] 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout: debug mds = 20 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout: debug mds balancer = 20 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout: mds debug frag = True 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout: mds debug scatterstat = False 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout: mds op complaint time = 180 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout: mds verify scatter = True 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout: osd op complaint time = 180 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout: rados mon op timeout = 900 2024-04-02T13:32:00.399 INFO:teuthology.orchestra.run.smithi044.stdout: rados osd op timeout = 900 2024-04-02T13:32:00.400 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-02T13:32:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-04-02T13:32:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T13:32:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-04-02T13:32:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-04-02T13:32:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-04-02T13:32:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-04-02T13:32:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.44,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-04-02T13:32:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 20 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 20 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 10 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-04-02T13:32:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.406 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T13:32:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: mon down mkfs grace = 300 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T13:32:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-04-02T13:32:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-04-02T13:32:00.410 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:32:00.410 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-04-02T13:32:00.410 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = False 2024-04-02T13:32:00.410 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-04-02T13:32:00.410 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-04-02T13:32:00.410 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-02T13:32:00.410 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-02T13:32:00.410 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-02T13:32:00.412 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T13:32:00.454 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T13:32:00.502 INFO:teuthology.orchestra.run.smithi044.stdout:[global] 2024-04-02T13:32:00.502 INFO:teuthology.orchestra.run.smithi044.stdout: chdir = "" 2024-04-02T13:32:00.502 INFO:teuthology.orchestra.run.smithi044.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T13:32:00.502 INFO:teuthology.orchestra.run.smithi044.stdout: auth supported = cephx 2024-04-02T13:32:00.502 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.502 INFO:teuthology.orchestra.run.smithi044.stdout: filestore xattr use omap = true 2024-04-02T13:32:00.502 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.502 INFO:teuthology.orchestra.run.smithi044.stdout: mon clock drift allowed = 1.000 2024-04-02T13:32:00.502 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: osd crush chooseleaf type = 0 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: auth debug = true 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on old message = true 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on bug = true 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: mon pg warn max object skew = 0 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default size = 2 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.503 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow primary affinity = true 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow pg remap = true 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on legacy crush tunables = false 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on crush straw calc version zero = false 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on no sortbitwise = false 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on osd down out interval zero = false 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on too few osds = false 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: mon_allow_pool_size_one = true 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: osd default data pool replay window = 5 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.504 INFO:teuthology.orchestra.run.smithi044.stdout: mon allow pool delete = true 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: mon cluster log file level = debug 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: debug asserts on shutdown = true 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: mon health detail to clog = false 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: mon host = "172.21.15.44,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: fsid = 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout:[osd] 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: osd journal size = 100 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub load threshold = 5.0 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub max interval = 600 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: osd mclock profile = high_recovery_ops 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.505 INFO:teuthology.orchestra.run.smithi044.stdout: osd recover clone overlap = true 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: osd recovery max chunk = 1048576 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug shutdown = true 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug op order = true 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug verify stray on activate = true 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug trim objects = true 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: osd open classes on start = true 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug pg log writeout = true 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: osd deep scrub update digest min age = 30 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: osd map max advance = 10 2024-04-02T13:32:00.506 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: journal zero on create = true 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: filestore ondisk finisher threads = 3 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: filestore apply finisher threads = 3 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: bdev debug aio = true 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug misdirected ops = true 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore block size = 96636764160 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore fsck on mount = True 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluefs = 20 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluestore = 20 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: debug osd = 20 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: debug rocksdb = 10 2024-04-02T13:32:00.507 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd full ratio = 0.9 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd nearfull ratio = 0.8 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: osd failsafe full ratio = 0.95 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: osd objectstore = bluestore 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: osd op complaint time = 180 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout:[mgr] 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: debug mgr = 20 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: mgr/telemetry/nag = false 2024-04-02T13:32:00.508 INFO:teuthology.orchestra.run.smithi044.stdout: debug client = 20 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout:[mon] 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: debug paxos = 20 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: mon data avail warn = 5 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: mon mgr mkfs grace = 240 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd reporter subtree level = osd 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd prime pg temp = true 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T13:32:00.509 INFO:teuthology.orchestra.run.smithi044.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: # 1m isn't quite enough 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: mon_down_mkfs_grace = 2m 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_filestore_osds = false 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: mon down mkfs grace = 300 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: mon op complaint time = 120 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout:[client] 2024-04-02T13:32:00.510 INFO:teuthology.orchestra.run.smithi044.stdout: rgw cache enabled = true 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable ops log = true 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable usage log = true 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout: client mount timeout = 600 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout: debug client = 20 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout: pid file = "" 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout: rados mon op timeout = 900 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout: rados osd op timeout = 900 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.a] 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.b] 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.c] 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.a] 2024-04-02T13:32:00.511 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.c] 2024-04-02T13:32:00.512 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.b] 2024-04-02T13:32:00.512 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.d] 2024-04-02T13:32:00.512 INFO:teuthology.orchestra.run.smithi044.stdout:[mds] 2024-04-02T13:32:00.512 INFO:teuthology.orchestra.run.smithi044.stdout: debug mds = 20 2024-04-02T13:32:00.512 INFO:teuthology.orchestra.run.smithi044.stdout: debug mds balancer = 20 2024-04-02T13:32:00.512 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-04-02T13:32:00.512 INFO:teuthology.orchestra.run.smithi044.stdout: mds debug frag = True 2024-04-02T13:32:00.512 INFO:teuthology.orchestra.run.smithi044.stdout: mds debug scatterstat = True 2024-04-02T13:32:00.512 INFO:teuthology.orchestra.run.smithi044.stdout: mds op complaint time = 180 2024-04-02T13:32:00.512 INFO:teuthology.orchestra.run.smithi044.stdout: mds verify scatter = True 2024-04-02T13:32:00.512 INFO:teuthology.orchestra.run.smithi044.stdout: osd op complaint time = 180 2024-04-02T13:32:00.512 INFO:teuthology.orchestra.run.smithi044.stdout: rados mon op timeout = 900 2024-04-02T13:32:00.512 INFO:teuthology.orchestra.run.smithi044.stdout: rados osd op timeout = 900 2024-04-02T13:32:00.513 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-02T13:32:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-04-02T13:32:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T13:32:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-04-02T13:32:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-04-02T13:32:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-04-02T13:32:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-04-02T13:32:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-04-02T13:32:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-04-02T13:32:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.44,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 4101e14a-13fe-452f-8f9a-78614b3ea6ab 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-04-02T13:32:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-04-02T13:32:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 20 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 20 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 10 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-04-02T13:32:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.519 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T13:32:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: mon down mkfs grace = 300 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-04-02T13:32:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-04-02T13:32:00.522 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-04-02T13:32:00.523 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-04-02T13:32:00.523 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-04-02T13:32:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-04-02T13:32:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-04-02T13:32:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-02T13:32:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-04-02T13:32:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-04-02T13:32:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-04-02T13:32:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-04-02T13:32:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-02T13:32:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-02T13:32:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-02T13:32:00.525 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-02T13:32:00.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.665+0000 7f0af6550640 1 Processor -- start 2024-04-02T13:32:00.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.665+0000 7f0af6550640 1 -- start start 2024-04-02T13:32:00.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.666+0000 7f0af6550640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0af01565f0 0x7f0af00acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.666+0000 7f0af6550640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0af015c5b0 0x7f0af00cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.666+0000 7f0af6550640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0af00aea70 0x7f0af00aef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.666+0000 7f0af6550640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0af00ca380 con 0x7f0af00aea70 2024-04-02T13:32:00.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.666+0000 7f0af6550640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0af015cb70 con 0x7f0af01565f0 2024-04-02T13:32:00.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.666+0000 7f0af6550640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0af015ce50 con 0x7f0af015c5b0 2024-04-02T13:32:00.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.666+0000 7f0af4aed640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0af00aea70 0x7f0af00aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.666+0000 7f0af4aed640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0af00aea70 0x7f0af00aef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50960/0 (socket says 172.21.15.44:50960) 2024-04-02T13:32:00.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.666+0000 7f0af4aed640 1 -- 172.21.15.44:0/517307116 learned_addr learned my addr 172.21.15.44:0/517307116 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:00.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.667+0000 7f0aeffff640 1 --2- 172.21.15.44:0/517307116 >> v2:172.21.15.69:3300/0 conn(0x7f0af01565f0 0x7f0af00acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.667+0000 7f0aef7fe640 1 --2- 172.21.15.44:0/517307116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0af015c5b0 0x7f0af00cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.667+0000 7f0af4aed640 1 -- 172.21.15.44:0/517307116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0af015c5b0 msgr2=0x7f0af00cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.667+0000 7f0af4aed640 1 --2- 172.21.15.44:0/517307116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0af015c5b0 0x7f0af00cbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.667+0000 7f0af4aed640 1 -- 172.21.15.44:0/517307116 >> v2:172.21.15.69:3300/0 conn(0x7f0af01565f0 msgr2=0x7f0af00acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:00.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.667+0000 7f0af4aed640 1 --2- 172.21.15.44:0/517307116 >> v2:172.21.15.69:3300/0 conn(0x7f0af01565f0 0x7f0af00acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.667+0000 7f0af4aed640 1 -- 172.21.15.44:0/517307116 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af015d130 con 0x7f0af00aea70 2024-04-02T13:32:00.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.667+0000 7f0aeffff640 1 --2- 172.21.15.44:0/517307116 >> v2:172.21.15.69:3300/0 conn(0x7f0af01565f0 0x7f0af00acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:00.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.667+0000 7f0aef7fe640 1 --2- 172.21.15.44:0/517307116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0af015c5b0 0x7f0af00cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:00.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.667+0000 7f0af4aed640 1 --2- 172.21.15.44:0/517307116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0af00aea70 0x7f0af00aef00 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f0af0068180 tx=0x7f0ae802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dd31e6a8799a74e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.668+0000 7f0aeeffd640 1 -- 172.21.15.44:0/517307116 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae802ca70 con 0x7f0af00aea70 2024-04-02T13:32:00.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.668+0000 7f0aeeffd640 1 -- 172.21.15.44:0/517307116 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae802cc10 con 0x7f0af00aea70 2024-04-02T13:32:00.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.668+0000 7f0aeeffd640 1 -- 172.21.15.44:0/517307116 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae80343f0 con 0x7f0af00aea70 2024-04-02T13:32:00.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.668+0000 7f0af6550640 1 -- 172.21.15.44:0/517307116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0af00aea70 msgr2=0x7f0af00aef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.668+0000 7f0af6550640 1 --2- 172.21.15.44:0/517307116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0af00aea70 0x7f0af00aef00 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f0af0068180 tx=0x7f0ae802bed0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.668+0000 7f0af6550640 1 -- 172.21.15.44:0/517307116 shutdown_connections 2024-04-02T13:32:00.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.668+0000 7f0af6550640 1 --2- 172.21.15.44:0/517307116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0af00aea70 0x7f0af00aef00 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.668+0000 7f0af6550640 1 --2- 172.21.15.44:0/517307116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0af015c5b0 0x7f0af00cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.668+0000 7f0af6550640 1 --2- 172.21.15.44:0/517307116 >> v2:172.21.15.69:3300/0 conn(0x7f0af01565f0 0x7f0af00acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.668+0000 7f0af6550640 1 -- 172.21.15.44:0/517307116 >> 172.21.15.44:0/517307116 conn(0x7f0af00cfca0 msgr2=0x7f0af00d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:00.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.668+0000 7f0af6550640 1 -- 172.21.15.44:0/517307116 shutdown_connections 2024-04-02T13:32:00.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.668+0000 7f0af6550640 1 -- 172.21.15.44:0/517307116 wait complete. 2024-04-02T13:32:00.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.669+0000 7f0af6550640 1 Processor -- start 2024-04-02T13:32:00.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.670+0000 7f0af6550640 1 -- start start 2024-04-02T13:32:00.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.670+0000 7f0af6550640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0af00aea70 0x7f0af0155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.670+0000 7f0af6550640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0af01565f0 0x7f0af00ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.670+0000 7f0af6550640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0af015c5b0 0x7f0af009ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.670+0000 7f0af6550640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0af00cc410 con 0x7f0af015c5b0 2024-04-02T13:32:00.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.670+0000 7f0af6550640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01569d0 con 0x7f0af01565f0 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.670+0000 7f0af6550640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0af00ca810 con 0x7f0af00aea70 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.670+0000 7f0af4aed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0af015c5b0 0x7f0af009ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.670+0000 7f0af4aed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0af015c5b0 0x7f0af009ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50962/0 (socket says 172.21.15.44:50962) 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.670+0000 7f0af4aed640 1 -- 172.21.15.44:0/1079991449 learned_addr learned my addr 172.21.15.44:0/1079991449 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.670+0000 7f0aeffff640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0af00aea70 0x7f0af0155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.670+0000 7f0aef7fe640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0af01565f0 0x7f0af00ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.671+0000 7f0af4aed640 1 -- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0af00aea70 msgr2=0x7f0af0155430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.671+0000 7f0af4aed640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0af00aea70 0x7f0af0155430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.671+0000 7f0af4aed640 1 -- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0af01565f0 msgr2=0x7f0af00ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.671+0000 7f0af4aed640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0af01565f0 0x7f0af00ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.671+0000 7f0af4aed640 1 -- 172.21.15.44:0/1079991449 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af00ce7c0 con 0x7f0af015c5b0 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.671+0000 7f0af4aed640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0af015c5b0 0x7f0af009ba20 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8009f40 tx=0x7f0ae8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.671+0000 7f0aecff9640 1 -- 172.21.15.44:0/1079991449 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae8043020 con 0x7f0af015c5b0 2024-04-02T13:32:00.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.671+0000 7f0aecff9640 1 -- 172.21.15.44:0/1079991449 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae803e450 con 0x7f0af015c5b0 2024-04-02T13:32:00.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.671+0000 7f0aecff9640 1 -- 172.21.15.44:0/1079991449 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae8009510 con 0x7f0af015c5b0 2024-04-02T13:32:00.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.671+0000 7f0af6550640 1 -- 172.21.15.44:0/1079991449 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0af00b4b30 con 0x7f0af015c5b0 2024-04-02T13:32:00.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.671+0000 7f0af6550640 1 -- 172.21.15.44:0/1079991449 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0af00b50f0 con 0x7f0af015c5b0 2024-04-02T13:32:00.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.672+0000 7f0af6550640 1 -- 172.21.15.44:0/1079991449 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0af0074f20 con 0x7f0af015c5b0 2024-04-02T13:32:00.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.673+0000 7f0aecff9640 1 -- 172.21.15.44:0/1079991449 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ae8046020 con 0x7f0af015c5b0 2024-04-02T13:32:00.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.673+0000 7f0aecff9640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ad006c920 0x7f0ad006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:00.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.673+0000 7f0aecff9640 1 -- 172.21.15.44:0/1079991449 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7f0ae80ae1c0 con 0x7f0af015c5b0 2024-04-02T13:32:00.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.673+0000 7f0aeffff640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ad006c920 0x7f0ad006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:00.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.674+0000 7f0aeffff640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ad006c920 0x7f0ad006ede0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8006fd0 tx=0x7f0ad8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:00.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.677+0000 7f0aecff9640 1 -- 172.21.15.44:0/1079991449 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0ae804d070 con 0x7f0af015c5b0 2024-04-02T13:32:00.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:00.906+0000 7f0af6550640 1 -- 172.21.15.44:0/1079991449 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f0af00ca080 con 0x7f0af015c5b0 2024-04-02T13:32:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.839+0000 7f0aecff9640 1 -- 172.21.15.44:0/1079991449 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v108) ==== 162+0+0 (secure 0 0 0) 0x7f0ae807bd60 con 0x7f0af015c5b0 2024-04-02T13:32:01.843 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-02T13:32:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.842+0000 7f0af6550640 1 -- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ad006c920 msgr2=0x7f0ad006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.842+0000 7f0af6550640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ad006c920 0x7f0ad006ede0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8006fd0 tx=0x7f0ad8008040 comp rx=0 tx=0).stop 2024-04-02T13:32:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.842+0000 7f0af6550640 1 -- 172.21.15.44:0/1079991449 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0af015c5b0 msgr2=0x7f0af009ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.842+0000 7f0af6550640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0af015c5b0 0x7f0af009ba20 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8009f40 tx=0x7f0ae8004a40 comp rx=0 tx=0).stop 2024-04-02T13:32:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.842+0000 7f0af6550640 1 -- 172.21.15.44:0/1079991449 shutdown_connections 2024-04-02T13:32:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.842+0000 7f0af6550640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ad006c920 0x7f0ad006ede0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.842+0000 7f0af6550640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0af015c5b0 0x7f0af009ba20 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.842+0000 7f0af6550640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0af01565f0 0x7f0af00ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.842+0000 7f0af6550640 1 --2- 172.21.15.44:0/1079991449 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0af00aea70 0x7f0af0155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.842+0000 7f0af6550640 1 -- 172.21.15.44:0/1079991449 >> 172.21.15.44:0/1079991449 conn(0x7f0af00cfca0 msgr2=0x7f0af00cc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.842+0000 7f0af6550640 1 -- 172.21.15.44:0/1079991449 shutdown_connections 2024-04-02T13:32:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.842+0000 7f0af6550640 1 -- 172.21.15.44:0/1079991449 wait complete. 2024-04-02T13:32:01.862 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:32:01.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.965+0000 7fb67660f640 1 Processor -- start 2024-04-02T13:32:01.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.965+0000 7fb67660f640 1 -- start start 2024-04-02T13:32:01.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.966+0000 7fb67660f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700d2040 0x7fb6700d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:01.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.966+0000 7fb67660f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6700d0000 con 0x7fb6700d0a30 2024-04-02T13:32:01.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.966+0000 7fb67660f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb67006c1d0 con 0x7fb67006cf90 2024-04-02T13:32:01.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.966+0000 7fb67660f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb67006cb00 con 0x7fb6700d2040 2024-04-02T13:32:01.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.966+0000 7fb66ffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb6700d0a30 0x7fb6700d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:60812/0 (socket says 172.21.15.44:60812) 2024-04-02T13:32:01.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.966+0000 7fb66ffff640 1 -- 172.21.15.44:0/3352053000 learned_addr learned my addr 172.21.15.44:0/3352053000 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:01.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb674bac640 1 --2- 172.21.15.44:0/3352053000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700d2040 0x7fb6700d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:01.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3916957417 0 0) 0x7fb6700d0000 con 0x7fb6700d0a30 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb658003520 con 0x7fb6700d0a30 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1204140629 0 0) 0x7fb658003520 con 0x7fb6700d0a30 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700d2040 msgr2=0x7fb6700d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 --2- 172.21.15.44:0/3352053000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700d2040 0x7fb6700d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 >> v1:172.21.15.69:6789/0 conn(0x7fb67006cf90 legacy=0x7fb6700cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67006c4b0 con 0x7fb6700d0a30 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fb660001670 con 0x7fb6700d0a30 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 >> v1:172.21.15.44:6789/0 conn(0x7fb6700d0a30 legacy=0x7fb6700d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 --2- 172.21.15.44:0/3352053000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6580039e0 0x7fb658003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 --2- 172.21.15.44:0/3352053000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb658004390 0x7fb658004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 --2- 172.21.15.44:0/3352053000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700d2040 0x7fb658005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb67006cb00 con 0x7fb6580039e0 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb65800baa0 con 0x7fb658004390 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb65800bca0 con 0x7fb6700d2040 2024-04-02T13:32:01.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66ffff640 1 --2- 172.21.15.44:0/3352053000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6580039e0 0x7fb658003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:01.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66f7fe640 1 --2- 172.21.15.44:0/3352053000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb658004390 0x7fb658004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:01.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb674bac640 1 --2- 172.21.15.44:0/3352053000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700d2040 0x7fb658005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:01.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66ffff640 1 -- 172.21.15.44:0/3352053000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700d2040 msgr2=0x7fb658005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:01.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66ffff640 1 --2- 172.21.15.44:0/3352053000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700d2040 0x7fb658005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:01.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66ffff640 1 -- 172.21.15.44:0/3352053000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb658004390 msgr2=0x7fb658004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:01.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66ffff640 1 --2- 172.21.15.44:0/3352053000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb658004390 0x7fb658004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:01.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.967+0000 7fb66ffff640 1 -- 172.21.15.44:0/3352053000 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb658003520 con 0x7fb6580039e0 2024-04-02T13:32:01.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.968+0000 7fb66ffff640 1 --2- 172.21.15.44:0/3352053000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6580039e0 0x7fb658003dc0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fb66000c190 tx=0x7fb658005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b478ef3e0d888f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:01.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.968+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb660007cc0 con 0x7fb6580039e0 2024-04-02T13:32:01.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.968+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb660030070 con 0x7fb6580039e0 2024-04-02T13:32:01.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.968+0000 7fb66effd640 1 -- 172.21.15.44:0/3352053000 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb660005d10 con 0x7fb6580039e0 2024-04-02T13:32:01.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.969+0000 7fb67660f640 1 -- 172.21.15.44:0/3352053000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6580039e0 msgr2=0x7fb658003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:01.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.969+0000 7fb67660f640 1 --2- 172.21.15.44:0/3352053000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6580039e0 0x7fb658003dc0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fb66000c190 tx=0x7fb658005830 comp rx=0 tx=0).stop 2024-04-02T13:32:01.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.969+0000 7fb674bac640 1 -- 172.21.15.44:0/3352053000 reap_dead start 2024-04-02T13:32:01.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.969+0000 7fb67660f640 1 -- 172.21.15.44:0/3352053000 shutdown_connections 2024-04-02T13:32:01.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.969+0000 7fb67660f640 1 -- 172.21.15.44:0/3352053000 >> 172.21.15.44:0/3352053000 conn(0x7fb67009d470 msgr2=0x7fb6700d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:01.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.969+0000 7fb67660f640 1 -- 172.21.15.44:0/3352053000 shutdown_connections 2024-04-02T13:32:01.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.969+0000 7fb67660f640 1 -- 172.21.15.44:0/3352053000 wait complete. 2024-04-02T13:32:01.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.970+0000 7fb67660f640 1 Processor -- start 2024-04-02T13:32:01.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.970+0000 7fb67660f640 1 -- start start 2024-04-02T13:32:01.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb67660f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6580039e0 0x7fb670161510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:01.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb67660f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67014fe60 0x7fb6701f8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:01.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb67660f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb67014f7b0 0x7fb670151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:01.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb67660f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb67015e6f0 con 0x7fb67014f7b0 2024-04-02T13:32:01.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb67660f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb67006d3f0 con 0x7fb6580039e0 2024-04-02T13:32:01.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb67660f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb67015e870 con 0x7fb67014fe60 2024-04-02T13:32:01.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb674bac640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb67014f7b0 0x7fb670151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb674bac640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb67014f7b0 0x7fb670151d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50990/0 (socket says 172.21.15.44:50990) 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb674bac640 1 -- 172.21.15.44:0/1745929439 learned_addr learned my addr 172.21.15.44:0/1745929439 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb66f7fe640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67014fe60 0x7fb6701f8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb66ffff640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6580039e0 0x7fb670161510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb674bac640 1 -- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67014fe60 msgr2=0x7fb6701f8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb674bac640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67014fe60 0x7fb6701f8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb674bac640 1 -- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6580039e0 msgr2=0x7fb670161510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb674bac640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6580039e0 0x7fb670161510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb674bac640 1 -- 172.21.15.44:0/1745929439 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67014fc30 con 0x7fb67014f7b0 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.971+0000 7fb66ffff640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6580039e0 0x7fb670161510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.972+0000 7fb674bac640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb67014f7b0 0x7fb670151d80 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fb668011c80 tx=0x7fb66800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.972+0000 7fb66cff9640 1 -- 172.21.15.44:0/1745929439 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668015430 con 0x7fb67014f7b0 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.972+0000 7fb66cff9640 1 -- 172.21.15.44:0/1745929439 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb66801d070 con 0x7fb67014f7b0 2024-04-02T13:32:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.972+0000 7fb67660f640 1 -- 172.21.15.44:0/1745929439 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb670160470 con 0x7fb67014f7b0 2024-04-02T13:32:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.972+0000 7fb66cff9640 1 -- 172.21.15.44:0/1745929439 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668015830 con 0x7fb67014f7b0 2024-04-02T13:32:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.972+0000 7fb67660f640 1 -- 172.21.15.44:0/1745929439 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb670160a30 con 0x7fb67014f7b0 2024-04-02T13:32:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.973+0000 7fb67660f640 1 -- 172.21.15.44:0/1745929439 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6700684f0 con 0x7fb67014f7b0 2024-04-02T13:32:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.973+0000 7fb66cff9640 1 -- 172.21.15.44:0/1745929439 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb6680159d0 con 0x7fb67014f7b0 2024-04-02T13:32:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.973+0000 7fb66cff9640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb65c06c920 0x7fb65c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.974+0000 7fb66cff9640 1 -- 172.21.15.44:0/1745929439 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7fb66808c4b0 con 0x7fb67014f7b0 2024-04-02T13:32:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.974+0000 7fb66ffff640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb65c06c920 0x7fb65c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.974+0000 7fb66ffff640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb65c06c920 0x7fb65c06ede0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb660034000 tx=0x7fb660009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:01.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:01.978+0000 7fb66cff9640 1 -- 172.21.15.44:0/1745929439 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb668010040 con 0x7fb67014f7b0 2024-04-02T13:32:02.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.200+0000 7fb67660f640 1 -- 172.21.15.44:0/1745929439 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6700d17c0 con 0x7fb67014f7b0 2024-04-02T13:32:02.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.201+0000 7fb66cff9640 1 -- 172.21.15.44:0/1745929439 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+5175 (secure 0 0 0) 0x7fb66805a6a0 con 0x7fb67014f7b0 2024-04-02T13:32:02.205 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:02.205 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":109,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:31:11.495701+0000","modified":"2024-04-02T13:32:01.843473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"a","rank":0,"incarnation":109,"state":"up:replay","state_seq":1,"addr":"172.21.15.44:6835/16875964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":16875964},{"type":"v1","addr":"172.21.15.44:6835","nonce":16875964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-04-02T13:32:02.205 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 109 2024-04-02T13:32:02.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.203+0000 7fb67660f640 1 -- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb65c06c920 msgr2=0x7fb65c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:02.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.203+0000 7fb67660f640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb65c06c920 0x7fb65c06ede0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb660034000 tx=0x7fb660009030 comp rx=0 tx=0).stop 2024-04-02T13:32:02.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.203+0000 7fb67660f640 1 -- 172.21.15.44:0/1745929439 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb67014f7b0 msgr2=0x7fb670151d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:02.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.203+0000 7fb67660f640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb67014f7b0 0x7fb670151d80 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fb668011c80 tx=0x7fb66800f580 comp rx=0 tx=0).stop 2024-04-02T13:32:02.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.203+0000 7fb67660f640 1 -- 172.21.15.44:0/1745929439 shutdown_connections 2024-04-02T13:32:02.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.203+0000 7fb67660f640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb65c06c920 0x7fb65c06ede0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:02.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.203+0000 7fb67660f640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb67014f7b0 0x7fb670151d80 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:02.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.203+0000 7fb67660f640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67014fe60 0x7fb6701f8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:02.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.203+0000 7fb67660f640 1 --2- 172.21.15.44:0/1745929439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6580039e0 0x7fb670161510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:02.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.203+0000 7fb67660f640 1 -- 172.21.15.44:0/1745929439 >> 172.21.15.44:0/1745929439 conn(0x7fb67009d470 msgr2=0x7fb67015da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:02.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.203+0000 7fb67660f640 1 -- 172.21.15.44:0/1745929439 shutdown_connections 2024-04-02T13:32:02.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:02.203+0000 7fb67660f640 1 -- 172.21.15.44:0/1745929439 wait complete. 2024-04-02T13:32:02.620 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 109, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:31:11.495701+0000', 'modified': '2024-04-02T13:32:01.843473+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 123, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10077}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10077': {'gid': 10077, 'name': 'a', 'rank': 0, 'incarnation': 109, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.44:6835/16875964', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6833', 'nonce': 16875964}, {'type': 'v1', 'addr': '172.21.15.44:6835', 'nonce': 16875964}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-02T13:32:02.620 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10077:up:replay 2024-04-02T13:32:03.622 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:32:03.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.732+0000 7f75cc796640 1 Processor -- start 2024-04-02T13:32:03.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.732+0000 7f75cc796640 1 -- start start 2024-04-02T13:32:03.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.733+0000 7f75cc796640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f75c415e4a0 0x7f75c40aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:03.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.733+0000 7f75cc796640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f75c406d550 0x7f75c40d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:03.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.733+0000 7f75cc796640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75c40cc600 con 0x7f75c40d0c50 2024-04-02T13:32:03.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.733+0000 7f75cc796640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f75c40cc8e0 con 0x7f75c406d550 2024-04-02T13:32:03.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.733+0000 7f75cc796640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f75c40ccbc0 con 0x7f75c415e4a0 2024-04-02T13:32:03.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.733+0000 7f75cad33640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f75c406d550 0x7f75c40d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:03.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.733+0000 7f75cad33640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f75c406d550 0x7f75c40d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50992/0 (socket says 172.21.15.44:50992) 2024-04-02T13:32:03.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.733+0000 7f75cad33640 1 -- 172.21.15.44:0/591168769 learned_addr learned my addr 172.21.15.44:0/591168769 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:03.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.734+0000 7f75ca532640 1 --2- 172.21.15.44:0/591168769 >> v2:172.21.15.69:3300/0 conn(0x7f75c415e4a0 0x7f75c40aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:03.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.734+0000 7f75cad33640 1 -- 172.21.15.44:0/591168769 >> v2:172.21.15.69:3300/0 conn(0x7f75c415e4a0 msgr2=0x7f75c40aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:03.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.734+0000 7f75cad33640 1 --2- 172.21.15.44:0/591168769 >> v2:172.21.15.69:3300/0 conn(0x7f75c415e4a0 0x7f75c40aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:03.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.734+0000 7f75cad33640 1 -- 172.21.15.44:0/591168769 >> v1:172.21.15.69:6789/0 conn(0x7f75c40d0c50 legacy=0x7f75c406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:03.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.734+0000 7f75cad33640 1 -- 172.21.15.44:0/591168769 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c40ccea0 con 0x7f75c406d550 2024-04-02T13:32:03.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.734+0000 7f75ca532640 1 --2- 172.21.15.44:0/591168769 >> v2:172.21.15.69:3300/0 conn(0x7f75c415e4a0 0x7f75c40aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:03.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.734+0000 7f75cad33640 1 --2- 172.21.15.44:0/591168769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c406d550 0x7f75c40d10b0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009e50 tx=0x7f75c002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c2684b05934e4f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:03.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.735+0000 7f75c9530640 1 -- 172.21.15.44:0/591168769 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c002ca80 con 0x7f75c406d550 2024-04-02T13:32:03.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.735+0000 7f75c9530640 1 -- 172.21.15.44:0/591168769 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c002cc20 con 0x7f75c406d550 2024-04-02T13:32:03.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.735+0000 7f75c9530640 1 -- 172.21.15.44:0/591168769 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c00343f0 con 0x7f75c406d550 2024-04-02T13:32:03.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.735+0000 7f75cc796640 1 -- 172.21.15.44:0/591168769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c406d550 msgr2=0x7f75c40d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:03.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.735+0000 7f75cc796640 1 --2- 172.21.15.44:0/591168769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c406d550 0x7f75c40d10b0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009e50 tx=0x7f75c002bee0 comp rx=0 tx=0).stop 2024-04-02T13:32:03.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.735+0000 7f75cc796640 1 -- 172.21.15.44:0/591168769 shutdown_connections 2024-04-02T13:32:03.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.735+0000 7f75cc796640 1 --2- 172.21.15.44:0/591168769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c406d550 0x7f75c40d10b0 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:03.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.735+0000 7f75cc796640 1 --2- 172.21.15.44:0/591168769 >> v2:172.21.15.69:3300/0 conn(0x7f75c415e4a0 0x7f75c40aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:03.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.735+0000 7f75cc796640 1 -- 172.21.15.44:0/591168769 >> 172.21.15.44:0/591168769 conn(0x7f75c409aa20 msgr2=0x7f75c409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:03.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.735+0000 7f75cc796640 1 -- 172.21.15.44:0/591168769 shutdown_connections 2024-04-02T13:32:03.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.735+0000 7f75cc796640 1 -- 172.21.15.44:0/591168769 wait complete. 2024-04-02T13:32:03.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.736+0000 7f75cc796640 1 Processor -- start 2024-04-02T13:32:03.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.736+0000 7f75cc796640 1 -- start start 2024-04-02T13:32:03.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.737+0000 7f75cc796640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75c406d550 0x7f75c41554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:03.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.737+0000 7f75cc796640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75c40d0c50 0x7f75c415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:03.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.737+0000 7f75cc796640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c415e4a0 0x7f75c415b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:03.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.737+0000 7f75cc796640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f75c415e880 con 0x7f75c415e4a0 2024-04-02T13:32:03.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.737+0000 7f75cc796640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f75c40cd460 con 0x7f75c40d0c50 2024-04-02T13:32:03.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.737+0000 7f75cc796640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f75c40af130 con 0x7f75c406d550 2024-04-02T13:32:03.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.737+0000 7f75cad33640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c415e4a0 0x7f75c415b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:03.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.737+0000 7f75cad33640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c415e4a0 0x7f75c415b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50996/0 (socket says 172.21.15.44:50996) 2024-04-02T13:32:03.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.737+0000 7f75cad33640 1 -- 172.21.15.44:0/2896991924 learned_addr learned my addr 172.21.15.44:0/2896991924 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:03.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.737+0000 7f75ca532640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75c406d550 0x7f75c41554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:03.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.738+0000 7f75c9d31640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75c40d0c50 0x7f75c415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:03.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.738+0000 7f75cad33640 1 -- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75c406d550 msgr2=0x7f75c41554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:03.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.738+0000 7f75cad33640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75c406d550 0x7f75c41554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:03.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.738+0000 7f75cad33640 1 -- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75c40d0c50 msgr2=0x7f75c415ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:03.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.738+0000 7f75cad33640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75c40d0c50 0x7f75c415ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:03.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.738+0000 7f75cad33640 1 -- 172.21.15.44:0/2896991924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c4155a10 con 0x7f75c415e4a0 2024-04-02T13:32:03.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.738+0000 7f75c9d31640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75c40d0c50 0x7f75c415ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:03.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.738+0000 7f75cad33640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c415e4a0 0x7f75c415b350 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f75c0036040 tx=0x7f75c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:03.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.738+0000 7f75baffd640 1 -- 172.21.15.44:0/2896991924 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c0043020 con 0x7f75c415e4a0 2024-04-02T13:32:03.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.738+0000 7f75baffd640 1 -- 172.21.15.44:0/2896991924 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c003e450 con 0x7f75c415e4a0 2024-04-02T13:32:03.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.738+0000 7f75cc796640 1 -- 172.21.15.44:0/2896991924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c40c5ae0 con 0x7f75c415e4a0 2024-04-02T13:32:03.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.738+0000 7f75baffd640 1 -- 172.21.15.44:0/2896991924 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c003e860 con 0x7f75c415e4a0 2024-04-02T13:32:03.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.738+0000 7f75cc796640 1 -- 172.21.15.44:0/2896991924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c40c60a0 con 0x7f75c415e4a0 2024-04-02T13:32:03.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.739+0000 7f75cc796640 1 -- 172.21.15.44:0/2896991924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75c40d1a20 con 0x7f75c415e4a0 2024-04-02T13:32:03.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.739+0000 7f75baffd640 1 -- 172.21.15.44:0/2896991924 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f75c0046020 con 0x7f75c415e4a0 2024-04-02T13:32:03.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.740+0000 7f75baffd640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75a806c8f0 0x7f75a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:03.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.740+0000 7f75baffd640 1 -- 172.21.15.44:0/2896991924 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7f75c00ae6e0 con 0x7f75c415e4a0 2024-04-02T13:32:03.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.740+0000 7f75ca532640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75a806c8f0 0x7f75a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:03.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.741+0000 7f75ca532640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75a806c8f0 0x7f75a806edb0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f75c4071140 tx=0x7f75bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:03.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.745+0000 7f75baffd640 1 -- 172.21.15.44:0/2896991924 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f75c40d1a20 con 0x7f75c415e4a0 2024-04-02T13:32:03.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.967+0000 7f75cc796640 1 -- 172.21.15.44:0/2896991924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75c406cf50 con 0x7f75c415e4a0 2024-04-02T13:32:03.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.968+0000 7f75baffd640 1 -- 172.21.15.44:0/2896991924 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5175 (secure 0 0 0) 0x7f75c004d030 con 0x7f75c415e4a0 2024-04-02T13:32:03.972 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:03.972 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":111,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:31:11.495701+0000","modified":"2024-04-02T13:32:02.847834+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"a","rank":0,"incarnation":109,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.44:6835/16875964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":16875964},{"type":"v1","addr":"172.21.15.44:6835","nonce":16875964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-04-02T13:32:03.972 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 111 2024-04-02T13:32:03.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.970+0000 7f75cc796640 1 -- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75a806c8f0 msgr2=0x7f75a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:03.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.970+0000 7f75cc796640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75a806c8f0 0x7f75a806edb0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f75c4071140 tx=0x7f75bc008040 comp rx=0 tx=0).stop 2024-04-02T13:32:03.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.970+0000 7f75cc796640 1 -- 172.21.15.44:0/2896991924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c415e4a0 msgr2=0x7f75c415b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:03.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.970+0000 7f75cc796640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c415e4a0 0x7f75c415b350 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f75c0036040 tx=0x7f75c0004a40 comp rx=0 tx=0).stop 2024-04-02T13:32:03.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.970+0000 7f75cc796640 1 -- 172.21.15.44:0/2896991924 shutdown_connections 2024-04-02T13:32:03.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.970+0000 7f75cc796640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f75a806c8f0 0x7f75a806edb0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:03.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.970+0000 7f75cc796640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c415e4a0 0x7f75c415b350 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:03.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.970+0000 7f75cc796640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75c40d0c50 0x7f75c415ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:03.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.970+0000 7f75cc796640 1 --2- 172.21.15.44:0/2896991924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75c406d550 0x7f75c41554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:03.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.970+0000 7f75cc796640 1 -- 172.21.15.44:0/2896991924 >> 172.21.15.44:0/2896991924 conn(0x7f75c409aa20 msgr2=0x7f75c406f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:03.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.970+0000 7f75cc796640 1 -- 172.21.15.44:0/2896991924 shutdown_connections 2024-04-02T13:32:03.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:03.970+0000 7f75cc796640 1 -- 172.21.15.44:0/2896991924 wait complete. 2024-04-02T13:32:04.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 111, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:31:11.495701+0000', 'modified': '2024-04-02T13:32:02.847834+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 123, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10077}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10077': {'gid': 10077, 'name': 'a', 'rank': 0, 'incarnation': 109, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.44:6835/16875964', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6833', 'nonce': 16875964}, {'type': 'v1', 'addr': '172.21.15.44:6835', 'nonce': 16875964}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-02T13:32:04.388 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10077:up:rejoin 2024-04-02T13:32:05.389 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:32:05.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.499+0000 7fedfc822640 1 Processor -- start 2024-04-02T13:32:05.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.499+0000 7fedfc822640 1 -- start start 2024-04-02T13:32:05.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.500+0000 7fedfc822640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fedf815e4d0 0x7fedf806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:05.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.500+0000 7fedfc822640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fedf8072f60 0x7fedf806ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.500+0000 7fedfc822640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedf806dfe0 con 0x7fedf815e0f0 2024-04-02T13:32:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.500+0000 7fedfc822640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fedf80aed70 con 0x7fedf8072f60 2024-04-02T13:32:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.500+0000 7fedfc822640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fedf806e2c0 con 0x7fedf815e4d0 2024-04-02T13:32:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.500+0000 7fedf6d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fedf8072f60 0x7fedf806ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.500+0000 7fedf6d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fedf8072f60 0x7fedf806ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38874/0 (socket says 172.21.15.44:38874) 2024-04-02T13:32:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.500+0000 7fedf6d9d640 1 -- 172.21.15.44:0/2398187571 learned_addr learned my addr 172.21.15.44:0/2398187571 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.500+0000 7fedf5d9b640 1 --2- 172.21.15.44:0/2398187571 >> v2:172.21.15.69:3300/0 conn(0x7fedf815e4d0 0x7fedf806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.500+0000 7fedf6d9d640 1 -- 172.21.15.44:0/2398187571 >> v2:172.21.15.69:3300/0 conn(0x7fedf815e4d0 msgr2=0x7fedf806b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.500+0000 7fedf6d9d640 1 --2- 172.21.15.44:0/2398187571 >> v2:172.21.15.69:3300/0 conn(0x7fedf815e4d0 0x7fedf806b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:05.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.500+0000 7fedf6d9d640 1 -- 172.21.15.44:0/2398187571 >> v1:172.21.15.69:6789/0 conn(0x7fedf815e0f0 legacy=0x7fedf80d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:05.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.501+0000 7fedf6d9d640 1 -- 172.21.15.44:0/2398187571 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf806e5a0 con 0x7fedf8072f60 2024-04-02T13:32:05.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.501+0000 7fedf6d9d640 1 --2- 172.21.15.44:0/2398187571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedf8072f60 0x7fedf806ba10 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fedf809ee50 tx=0x7fede802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce330a4bb9090052 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:05.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.501+0000 7fedf559a640 1 -- 172.21.15.44:0/2398187571 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede802cb20 con 0x7fedf8072f60 2024-04-02T13:32:05.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.501+0000 7fedf559a640 1 -- 172.21.15.44:0/2398187571 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fede802ccc0 con 0x7fedf8072f60 2024-04-02T13:32:05.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.502+0000 7fedf559a640 1 -- 172.21.15.44:0/2398187571 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede8034610 con 0x7fedf8072f60 2024-04-02T13:32:05.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.502+0000 7fedfc822640 1 -- 172.21.15.44:0/2398187571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedf8072f60 msgr2=0x7fedf806ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:05.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.502+0000 7fedfc822640 1 --2- 172.21.15.44:0/2398187571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedf8072f60 0x7fedf806ba10 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fedf809ee50 tx=0x7fede802bf80 comp rx=0 tx=0).stop 2024-04-02T13:32:05.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.502+0000 7fedfc822640 1 -- 172.21.15.44:0/2398187571 shutdown_connections 2024-04-02T13:32:05.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.502+0000 7fedfc822640 1 --2- 172.21.15.44:0/2398187571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedf8072f60 0x7fedf806ba10 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:05.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.502+0000 7fedfc822640 1 --2- 172.21.15.44:0/2398187571 >> v2:172.21.15.69:3300/0 conn(0x7fedf815e4d0 0x7fedf806b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:05.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.502+0000 7fedfc822640 1 -- 172.21.15.44:0/2398187571 >> 172.21.15.44:0/2398187571 conn(0x7fedf80b2980 msgr2=0x7fedf80b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:05.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.502+0000 7fedfc822640 1 -- 172.21.15.44:0/2398187571 shutdown_connections 2024-04-02T13:32:05.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.502+0000 7fedfc822640 1 -- 172.21.15.44:0/2398187571 wait complete. 2024-04-02T13:32:05.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.504+0000 7fedfc822640 1 Processor -- start 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.504+0000 7fedfc822640 1 -- start start 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.504+0000 7fedfc822640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedf8072f60 0x7fedf80c4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.504+0000 7fedfc822640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedf815e0f0 0x7fedf8147790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.504+0000 7fedfc822640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf815e4d0 0x7fedf8146350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.504+0000 7fedfc822640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fedf80d0800 con 0x7fedf8072f60 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.504+0000 7fedfc822640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fedf80d0af0 con 0x7fedf815e4d0 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.504+0000 7fedfc822640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fedf806eda0 con 0x7fedf815e0f0 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.504+0000 7fedf659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedf8072f60 0x7fedf80c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedf659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedf8072f60 0x7fedf80c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38886/0 (socket says 172.21.15.44:38886) 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedf659c640 1 -- 172.21.15.44:0/1580434570 learned_addr learned my addr 172.21.15.44:0/1580434570 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedf5d9b640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedf815e0f0 0x7fedf8147790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedf6d9d640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf815e4d0 0x7fedf8146350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedf659c640 1 -- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedf815e0f0 msgr2=0x7fedf8147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedf659c640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedf815e0f0 0x7fedf8147790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:05.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedf659c640 1 -- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf815e4d0 msgr2=0x7fedf8146350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:05.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedf659c640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf815e4d0 0x7fedf8146350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:05.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedf659c640 1 -- 172.21.15.44:0/1580434570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf8147cd0 con 0x7fedf8072f60 2024-04-02T13:32:05.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedf6d9d640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf815e4d0 0x7fedf8146350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:05.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedf659c640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedf8072f60 0x7fedf80c4010 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fedf000c430 tx=0x7fedf000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:05.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedeeffd640 1 -- 172.21.15.44:0/1580434570 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf0011790 con 0x7fedf8072f60 2024-04-02T13:32:05.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedeeffd640 1 -- 172.21.15.44:0/1580434570 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedf0017070 con 0x7fedf8072f60 2024-04-02T13:32:05.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedeeffd640 1 -- 172.21.15.44:0/1580434570 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf0011bb0 con 0x7fedf8072f60 2024-04-02T13:32:05.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedfc822640 1 -- 172.21.15.44:0/1580434570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf81452b0 con 0x7fedf8072f60 2024-04-02T13:32:05.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.505+0000 7fedfc822640 1 -- 172.21.15.44:0/1580434570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf8145870 con 0x7fedf8072f60 2024-04-02T13:32:05.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.506+0000 7fedfc822640 1 -- 172.21.15.44:0/1580434570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedf80684f0 con 0x7fedf8072f60 2024-04-02T13:32:05.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.507+0000 7fedeeffd640 1 -- 172.21.15.44:0/1580434570 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fedf001c460 con 0x7fedf8072f60 2024-04-02T13:32:05.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.507+0000 7fedeeffd640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fedd806c8f0 0x7fedd806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:05.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.507+0000 7fedeeffd640 1 -- 172.21.15.44:0/1580434570 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7fedf008bc20 con 0x7fedf8072f60 2024-04-02T13:32:05.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.508+0000 7fedf5d9b640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fedd806c8f0 0x7fedd806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:05.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.508+0000 7fedf5d9b640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fedd806c8f0 0x7fedd806edb0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fedf80c6920 tx=0x7fede0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:05.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.512+0000 7fedeeffd640 1 -- 172.21.15.44:0/1580434570 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fedf8147cd0 con 0x7fedf8072f60 2024-04-02T13:32:05.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.738+0000 7fedfc822640 1 -- 172.21.15.44:0/1580434570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedf806f7b0 con 0x7fedf8072f60 2024-04-02T13:32:05.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.738+0000 7fedeeffd640 1 -- 172.21.15.44:0/1580434570 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5184 (secure 0 0 0) 0x7fedf0059ea0 con 0x7fedf8072f60 2024-04-02T13:32:05.742 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:05.742 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:31:11.495701+0000","modified":"2024-04-02T13:32:04.851323+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6835/16875964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":16875964},{"type":"v1","addr":"172.21.15.44:6835","nonce":16875964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":8}]} 2024-04-02T13:32:05.742 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 112 2024-04-02T13:32:05.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.740+0000 7fedfc822640 1 -- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fedd806c8f0 msgr2=0x7fedd806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:05.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.740+0000 7fedfc822640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fedd806c8f0 0x7fedd806edb0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fedf80c6920 tx=0x7fede0008040 comp rx=0 tx=0).stop 2024-04-02T13:32:05.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.740+0000 7fedfc822640 1 -- 172.21.15.44:0/1580434570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedf8072f60 msgr2=0x7fedf80c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:05.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.740+0000 7fedfc822640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedf8072f60 0x7fedf80c4010 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fedf000c430 tx=0x7fedf000c8f0 comp rx=0 tx=0).stop 2024-04-02T13:32:05.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.740+0000 7fedfc822640 1 -- 172.21.15.44:0/1580434570 shutdown_connections 2024-04-02T13:32:05.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.740+0000 7fedfc822640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fedd806c8f0 0x7fedd806edb0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:05.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.740+0000 7fedfc822640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf815e4d0 0x7fedf8146350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:05.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.740+0000 7fedfc822640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedf815e0f0 0x7fedf8147790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:05.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.740+0000 7fedfc822640 1 --2- 172.21.15.44:0/1580434570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedf8072f60 0x7fedf80c4010 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:05.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.740+0000 7fedfc822640 1 -- 172.21.15.44:0/1580434570 >> 172.21.15.44:0/1580434570 conn(0x7fedf80b2980 msgr2=0x7fedf806f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:05.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.740+0000 7fedfc822640 1 -- 172.21.15.44:0/1580434570 shutdown_connections 2024-04-02T13:32:05.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:05.741+0000 7fedfc822640 1 -- 172.21.15.44:0/1580434570 wait complete. 2024-04-02T13:32:05.843 INFO:tasks.ceph.mds.a.smithi044.stderr:2024-04-02T13:32:05.840+0000 7f4d2ced3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:32:06.158 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 112, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T13:31:11.495701+0000', 'modified': '2024-04-02T13:32:04.851323+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 123, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10077}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10077': {'gid': 10077, 'name': 'a', 'rank': 0, 'incarnation': 109, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.44:6835/16875964', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6833', 'nonce': 16875964}, {'type': 'v1', 'addr': '172.21.15.44:6835', 'nonce': 16875964}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10077, 'qdb_cluster': [10077]} 2024-04-02T13:32:06.158 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T13:32:06.158 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T13:32:06.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.262+0000 7fa98c4d9640 1 Processor -- start 2024-04-02T13:32:06.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.262+0000 7fa98c4d9640 1 -- start start 2024-04-02T13:32:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.262+0000 7fa98c4d9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa98406d700 0x7fa98415e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.262+0000 7fa98c4d9640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa98415ee70 0x7fa9840d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.262+0000 7fa98c4d9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa98415fb50 con 0x7fa98415e550 2024-04-02T13:32:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.262+0000 7fa98c4d9640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa98406ca90 con 0x7fa98415ee70 2024-04-02T13:32:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.262+0000 7fa98c4d9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa98415f590 con 0x7fa98406d700 2024-04-02T13:32:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.263+0000 7fa98aa76640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa98415ee70 0x7fa9840d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.263+0000 7fa98aa76640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa98415ee70 0x7fa9840d0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38888/0 (socket says 172.21.15.44:38888) 2024-04-02T13:32:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.263+0000 7fa98aa76640 1 -- 172.21.15.44:0/4082386179 learned_addr learned my addr 172.21.15.44:0/4082386179 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.263+0000 7fa989a74640 1 --2- 172.21.15.44:0/4082386179 >> v2:172.21.15.69:3300/0 conn(0x7fa98406d700 0x7fa98415e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.263+0000 7fa98aa76640 1 -- 172.21.15.44:0/4082386179 >> v2:172.21.15.69:3300/0 conn(0x7fa98406d700 msgr2=0x7fa98415e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.263+0000 7fa98aa76640 1 --2- 172.21.15.44:0/4082386179 >> v2:172.21.15.69:3300/0 conn(0x7fa98406d700 0x7fa98415e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.263+0000 7fa98aa76640 1 -- 172.21.15.44:0/4082386179 >> v1:172.21.15.69:6789/0 conn(0x7fa98415e550 legacy=0x7fa98406cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.263+0000 7fa98aa76640 1 -- 172.21.15.44:0/4082386179 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98415f320 con 0x7fa98415ee70 2024-04-02T13:32:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.264+0000 7fa98aa76640 1 --2- 172.21.15.44:0/4082386179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa98415ee70 0x7fa9840d0ab0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7fa9840b4a20 tx=0x7fa97c02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b20df60e9cff4fd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.264+0000 7fa989273640 1 -- 172.21.15.44:0/4082386179 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c02cbb0 con 0x7fa98415ee70 2024-04-02T13:32:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.264+0000 7fa989273640 1 -- 172.21.15.44:0/4082386179 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa97c02cd50 con 0x7fa98415ee70 2024-04-02T13:32:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.264+0000 7fa989273640 1 -- 172.21.15.44:0/4082386179 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c034610 con 0x7fa98415ee70 2024-04-02T13:32:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.264+0000 7fa98c4d9640 1 -- 172.21.15.44:0/4082386179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa98415ee70 msgr2=0x7fa9840d0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.264+0000 7fa98c4d9640 1 --2- 172.21.15.44:0/4082386179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa98415ee70 0x7fa9840d0ab0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7fa9840b4a20 tx=0x7fa97c02c010 comp rx=0 tx=0).stop 2024-04-02T13:32:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.265+0000 7fa98c4d9640 1 -- 172.21.15.44:0/4082386179 shutdown_connections 2024-04-02T13:32:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.265+0000 7fa98c4d9640 1 --2- 172.21.15.44:0/4082386179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa98415ee70 0x7fa9840d0ab0 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.265+0000 7fa98c4d9640 1 --2- 172.21.15.44:0/4082386179 >> v2:172.21.15.69:3300/0 conn(0x7fa98406d700 0x7fa98415e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.265+0000 7fa98c4d9640 1 -- 172.21.15.44:0/4082386179 >> 172.21.15.44:0/4082386179 conn(0x7fa98409aa20 msgr2=0x7fa98409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:06.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.265+0000 7fa98c4d9640 1 -- 172.21.15.44:0/4082386179 shutdown_connections 2024-04-02T13:32:06.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.265+0000 7fa98c4d9640 1 -- 172.21.15.44:0/4082386179 wait complete. 2024-04-02T13:32:06.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.266+0000 7fa98c4d9640 1 Processor -- start 2024-04-02T13:32:06.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.266+0000 7fa98c4d9640 1 -- start start 2024-04-02T13:32:06.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa98c4d9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa98406d700 0x7fa9840ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa98c4d9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa98415e550 0x7fa98414e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa98c4d9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415ee70 0x7fa984154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa98c4d9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa9840d0590 con 0x7fa98415e550 2024-04-02T13:32:06.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa98c4d9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9840d02b0 con 0x7fa98406d700 2024-04-02T13:32:06.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa98c4d9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa9840cc4b0 con 0x7fa98415ee70 2024-04-02T13:32:06.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa989a74640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa98415e550 0x7fa98414e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa989a74640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa98415e550 0x7fa98414e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38900/0 (socket says 172.21.15.44:38900) 2024-04-02T13:32:06.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa989a74640 1 -- 172.21.15.44:0/87148599 learned_addr learned my addr 172.21.15.44:0/87148599 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:06.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa98a275640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa98406d700 0x7fa9840ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa98aa76640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415ee70 0x7fa984154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa989a74640 1 -- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415ee70 msgr2=0x7fa984154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa989a74640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415ee70 0x7fa984154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa989a74640 1 -- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa98406d700 msgr2=0x7fa9840ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa989a74640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa98406d700 0x7fa9840ca990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa989a74640 1 -- 172.21.15.44:0/87148599 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98414ea20 con 0x7fa98415e550 2024-04-02T13:32:06.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa989a74640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa98415e550 0x7fa98414e4e0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7fa97400e870 tx=0x7fa97400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:06.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa97affd640 1 -- 172.21.15.44:0/87148599 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa974017070 con 0x7fa98415e550 2024-04-02T13:32:06.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa97affd640 1 -- 172.21.15.44:0/87148599 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa97400c940 con 0x7fa98415e550 2024-04-02T13:32:06.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa98c4d9640 1 -- 172.21.15.44:0/87148599 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9840caf70 con 0x7fa98415e550 2024-04-02T13:32:06.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa97affd640 1 -- 172.21.15.44:0/87148599 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97400cd30 con 0x7fa98415e550 2024-04-02T13:32:06.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.267+0000 7fa98c4d9640 1 -- 172.21.15.44:0/87148599 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9840cb530 con 0x7fa98415e550 2024-04-02T13:32:06.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.269+0000 7fa97affd640 1 -- 172.21.15.44:0/87148599 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa97401a070 con 0x7fa98415e550 2024-04-02T13:32:06.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.269+0000 7fa97affd640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa96806c8f0 0x7fa96806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.269+0000 7fa97affd640 1 -- 172.21.15.44:0/87148599 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7fa97408bb70 con 0x7fa98415e550 2024-04-02T13:32:06.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.269+0000 7fa98a275640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa96806c8f0 0x7fa96806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.270+0000 7fa98a275640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa96806c8f0 0x7fa96806edb0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fa9800016e0 tx=0x7fa980009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:06.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.270+0000 7fa98f19e740 1 Processor -- start 2024-04-02T13:32:06.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.271+0000 7fa98f19e740 1 -- start start 2024-04-02T13:32:06.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.271+0000 7fa98f19e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 0x56158d79eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.271+0000 7fa98f19e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbf70 0x56158d7ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.271+0000 7fa98f19e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d79f070 0x56158d7cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.271+0000 7fa98f19e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56158d708060 con 0x56158d7cbb90 2024-04-02T13:32:06.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.271+0000 7fa98f19e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56158d6b0650 con 0x56158d7cbf70 2024-04-02T13:32:06.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.271+0000 7fa98f19e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x56158d704010 con 0x56158d79f070 2024-04-02T13:32:06.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.271+0000 7fa98a275640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 0x56158d79eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.271+0000 7fa98a275640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 0x56158d79eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38902/0 (socket says 172.21.15.44:38902) 2024-04-02T13:32:06.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.271+0000 7fa98a275640 1 -- 172.21.15.44:0/3058457664 learned_addr learned my addr 172.21.15.44:0/3058457664 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:06.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.271+0000 7fa989a74640 1 --2- 172.21.15.44:0/3058457664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbf70 0x56158d7ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.271+0000 7fa98aa76640 1 --2- 172.21.15.44:0/3058457664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d79f070 0x56158d7cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.272+0000 7fa98a275640 1 -- 172.21.15.44:0/3058457664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d79f070 msgr2=0x56158d7cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:06.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.272+0000 7fa98a275640 1 --2- 172.21.15.44:0/3058457664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d79f070 0x56158d7cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.272+0000 7fa98a275640 1 -- 172.21.15.44:0/3058457664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbf70 msgr2=0x56158d7ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:06.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.272+0000 7fa98a275640 1 --2- 172.21.15.44:0/3058457664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbf70 0x56158d7ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.272+0000 7fa98a275640 1 -- 172.21.15.44:0/3058457664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56158d41c2d0 con 0x56158d7cbb90 2024-04-02T13:32:06.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.272+0000 7fa98a275640 1 --2- 172.21.15.44:0/3058457664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 0x56158d79eb30 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7fa98000fb70 tx=0x7fa980012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1dd5720ad949b1f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:06.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.272+0000 7fa967fff640 1 -- 172.21.15.44:0/3058457664 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa980026070 con 0x56158d7cbb90 2024-04-02T13:32:06.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.272+0000 7fa967fff640 1 -- 172.21.15.44:0/3058457664 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9800067d0 con 0x56158d7cbb90 2024-04-02T13:32:06.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.272+0000 7fa967fff640 1 -- 172.21.15.44:0/3058457664 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa980006ba0 con 0x56158d7cbb90 2024-04-02T13:32:06.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.273+0000 7fa98f19e740 1 -- 172.21.15.44:0/3058457664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 msgr2=0x56158d79eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.273+0000 7fa98f19e740 1 --2- 172.21.15.44:0/3058457664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 0x56158d79eb30 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7fa98000fb70 tx=0x7fa980012000 comp rx=0 tx=0).stop 2024-04-02T13:32:06.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.273+0000 7fa98f19e740 1 -- 172.21.15.44:0/3058457664 shutdown_connections 2024-04-02T13:32:06.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.273+0000 7fa98f19e740 1 --2- 172.21.15.44:0/3058457664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d79f070 0x56158d7cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.273+0000 7fa98f19e740 1 --2- 172.21.15.44:0/3058457664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbf70 0x56158d7ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.273+0000 7fa98f19e740 1 --2- 172.21.15.44:0/3058457664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 0x56158d79eb30 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.273+0000 7fa98f19e740 1 -- 172.21.15.44:0/3058457664 >> 172.21.15.44:0/3058457664 conn(0x56158d7a0da0 msgr2=0x56158d7e9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:06.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.273+0000 7fa98f19e740 1 -- 172.21.15.44:0/3058457664 shutdown_connections 2024-04-02T13:32:06.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.273+0000 7fa98f19e740 1 -- 172.21.15.44:0/3058457664 wait complete. 2024-04-02T13:32:06.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.273+0000 7fa98f19e740 1 -- start start 2024-04-02T13:32:06.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.273+0000 7fa98f19e740 1 Processor -- start 2024-04-02T13:32:06.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.274+0000 7fa98f19e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7cbb90 0x56158d7f9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.274+0000 7fa98f19e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbf70 0x56158d7fd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.274+0000 7fa98f19e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7becd0 0x56158d7fd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.274+0000 7fa98f19e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56158d7a3440 con 0x56158d7becd0 2024-04-02T13:32:06.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.274+0000 7fa98f19e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56158d6c3a10 con 0x56158d7cbf70 2024-04-02T13:32:06.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.274+0000 7fa98f19e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x56158d7a5c80 con 0x56158d7cbb90 2024-04-02T13:32:06.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.274+0000 7fa98f19e740 10 client.0 _ll_register_callbacks cb 0x56158d7000b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.274+0000 7fa98aa76640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7becd0 0x56158d7fd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.274+0000 7fa98aa76640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7becd0 0x56158d7fd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38908/0 (socket says 172.21.15.44:38908) 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.274+0000 7fa98aa76640 1 -- 172.21.15.44:0/199623096 learned_addr learned my addr 172.21.15.44:0/199623096 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa98a275640 1 --2- 172.21.15.44:0/199623096 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7cbb90 0x56158d7f9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa989a74640 1 --2- 172.21.15.44:0/199623096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbf70 0x56158d7fd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa98aa76640 1 -- 172.21.15.44:0/199623096 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7cbb90 msgr2=0x56158d7f9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa98aa76640 1 --2- 172.21.15.44:0/199623096 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7cbb90 0x56158d7f9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa98aa76640 1 -- 172.21.15.44:0/199623096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbf70 msgr2=0x56158d7fd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa98aa76640 1 --2- 172.21.15.44:0/199623096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbf70 0x56158d7fd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa98aa76640 1 -- 172.21.15.44:0/199623096 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56158d4e66b0 con 0x56158d7becd0 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa98aa76640 1 --2- 172.21.15.44:0/199623096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7becd0 0x56158d7fd960 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009f40 tx=0x7fa97c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3fd7d02be79591 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa9657fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa9657fa640 1 -- 172.21.15.44:0/199623096 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c009510 con 0x56158d7becd0 2024-04-02T13:32:06.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa9657fa640 1 -- 172.21.15.44:0/199623096 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7fa97c034720 con 0x56158d7becd0 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa98f19e740 1 -- 172.21.15.44:0/199623096 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9800067d0 con 0x56158d7becd0 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa9657fa640 1 -- 172.21.15.44:0/199623096 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fa97c034920 con 0x56158d7becd0 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa98f19e740 10 client.10167 fetch_fsmap learned FSMap version 112 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa98f19e740 1 -- 172.21.15.44:0/199623096 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) -- 0x56158d7cfdd0 con 0x56158d7becd0 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa9657fa640 1 -- 172.21.15.44:0/199623096 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9800067d0 con 0x56158d7becd0 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa9657fa640 1 -- 172.21.15.44:0/199623096 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 112) ==== 3494+0+0 (secure 0 0 0) 0x7fa97c034ac0 con 0x56158d7becd0 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.275+0000 7fa98f19e740 10 client.10167 fetch_fsmap finished waiting for FSMap version 112 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.276+0000 7fa98f19e740 10 client.10167 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.276+0000 7fa98f19e740 20 client.10167 populate_metadata read hostname 'smithi044' 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.276+0000 7fa98f19e740 1 --2- 172.21.15.44:0/199623096 >> [v2:172.21.15.44:6833/16875964,v1:172.21.15.44:6835/16875964] conn(0x56158d7e8050 0x56158d7d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.276+0000 7fa98f19e740 4 client.10167 mds_command: new command op to 10077 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.276+0000 7fa98f19e740 1 -- 172.21.15.44:0/199623096 --> [v2:172.21.15.44:6833/16875964,v1:172.21.15.44:6835/16875964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56158d6afcf0 con 0x56158d7e8050 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.276+0000 7fa98a275640 1 --2- 172.21.15.44:0/199623096 >> [v2:172.21.15.44:6833/16875964,v1:172.21.15.44:6835/16875964] conn(0x56158d7e8050 0x56158d7d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.276+0000 7fa98a275640 1 --2- 172.21.15.44:0/199623096 >> [v2:172.21.15.44:6833/16875964,v1:172.21.15.44:6835/16875964] conn(0x56158d7e8050 0x56158d7d0aa0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe17799a76070cee server_cookie=5d26a2ed328c074d in_seq=0 out_seq=0 2024-04-02T13:32:06.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.276+0000 7fa9657fa640 10 client.10167 ms_handle_connect on v2:172.21.15.44:6833/16875964 2024-04-02T13:32:06.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa9657fa640 1 -- 172.21.15.44:0/199623096 <== mds.0 v2:172.21.15.44:6833/16875964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56158d6afcf0 con 0x56158d7e8050 2024-04-02T13:32:06.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa9657fa640 10 client.10167 handle_command_reply: tid=0 2024-04-02T13:32:06.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa98f19e740 1 client.10167 shutdown 2024-04-02T13:32:06.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa98f19e740 1 -- 172.21.15.44:0/199623096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7becd0 msgr2=0x56158d7fd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa98f19e740 1 --2- 172.21.15.44:0/199623096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7becd0 0x56158d7fd960 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009f40 tx=0x7fa97c004ab0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa98f19e740 1 -- 172.21.15.44:0/199623096 shutdown_connections 2024-04-02T13:32:06.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa98f19e740 1 --2- 172.21.15.44:0/199623096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7becd0 0x56158d7fd960 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa98f19e740 1 --2- 172.21.15.44:0/199623096 >> [v2:172.21.15.44:6833/16875964,v1:172.21.15.44:6835/16875964] conn(0x56158d7e8050 0x56158d7d0aa0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa98f19e740 1 --2- 172.21.15.44:0/199623096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbf70 0x56158d7fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa98f19e740 1 --2- 172.21.15.44:0/199623096 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7cbb90 0x56158d7f9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa9657fa640 0 client.10167 ms_handle_reset on v2:172.21.15.44:6833/16875964 2024-04-02T13:32:06.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa98f19e740 1 -- 172.21.15.44:0/199623096 >> 172.21.15.44:0/199623096 conn(0x56158d79f070 msgr2=0x56158d7a4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:06.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa98f19e740 1 -- 172.21.15.44:0/199623096 shutdown_connections 2024-04-02T13:32:06.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa98f19e740 1 -- 172.21.15.44:0/199623096 wait complete. 2024-04-02T13:32:06.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.277+0000 7fa98f19e740 20 client.10167 trim_cache size 0 max 16384 2024-04-02T13:32:06.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.295+0000 7fa98f19e740 1 Processor -- start 2024-04-02T13:32:06.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.295+0000 7fa98f19e740 1 -- start start 2024-04-02T13:32:06.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98f19e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7becd0 0x56158d7e96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98f19e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbb90 0x56158d7ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98f19e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbf70 0x56158d7d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98f19e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa984200170 con 0x56158d7cbf70 2024-04-02T13:32:06.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98f19e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa984150570 con 0x56158d7cbb90 2024-04-02T13:32:06.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98f19e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa9841ffb20 con 0x56158d7becd0 2024-04-02T13:32:06.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98aa76640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbf70 0x56158d7d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98aa76640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbf70 0x56158d7d1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38912/0 (socket says 172.21.15.44:38912) 2024-04-02T13:32:06.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98aa76640 1 -- 172.21.15.44:0/1005927209 learned_addr learned my addr 172.21.15.44:0/1005927209 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:06.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa989a74640 1 --2- 172.21.15.44:0/1005927209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbb90 0x56158d7ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98a275640 1 --2- 172.21.15.44:0/1005927209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7becd0 0x56158d7e96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98aa76640 1 -- 172.21.15.44:0/1005927209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7becd0 msgr2=0x56158d7e96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:06.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98aa76640 1 --2- 172.21.15.44:0/1005927209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7becd0 0x56158d7e96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98aa76640 1 -- 172.21.15.44:0/1005927209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbb90 msgr2=0x56158d7ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:06.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98aa76640 1 --2- 172.21.15.44:0/1005927209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbb90 0x56158d7ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.296+0000 7fa98aa76640 1 -- 172.21.15.44:0/1005927209 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56158d7e9fb0 con 0x56158d7cbf70 2024-04-02T13:32:06.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.297+0000 7fa98aa76640 1 --2- 172.21.15.44:0/1005927209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbf70 0x56158d7d1790 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009640 tx=0x7fa97c041070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=842ffe1544e68434 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:06.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.297+0000 7fa9677fe640 1 -- 172.21.15.44:0/1005927209 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c034600 con 0x56158d7cbf70 2024-04-02T13:32:06.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.297+0000 7fa9677fe640 1 -- 172.21.15.44:0/1005927209 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa97c034ac0 con 0x56158d7cbf70 2024-04-02T13:32:06.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.297+0000 7fa9677fe640 1 -- 172.21.15.44:0/1005927209 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c048530 con 0x56158d7cbf70 2024-04-02T13:32:06.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.297+0000 7fa98f19e740 1 -- 172.21.15.44:0/1005927209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbf70 msgr2=0x56158d7d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.297+0000 7fa98f19e740 1 --2- 172.21.15.44:0/1005927209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbf70 0x56158d7d1790 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009640 tx=0x7fa97c041070 comp rx=0 tx=0).stop 2024-04-02T13:32:06.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.297+0000 7fa98f19e740 1 -- 172.21.15.44:0/1005927209 shutdown_connections 2024-04-02T13:32:06.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.297+0000 7fa98f19e740 1 --2- 172.21.15.44:0/1005927209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbf70 0x56158d7d1790 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.297+0000 7fa98f19e740 1 --2- 172.21.15.44:0/1005927209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7cbb90 0x56158d7ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.297+0000 7fa98f19e740 1 --2- 172.21.15.44:0/1005927209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7becd0 0x56158d7e96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.297+0000 7fa98f19e740 1 -- 172.21.15.44:0/1005927209 >> 172.21.15.44:0/1005927209 conn(0x56158d79f070 msgr2=0x56158d80db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.297+0000 7fa98f19e740 1 -- 172.21.15.44:0/1005927209 shutdown_connections 2024-04-02T13:32:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.297+0000 7fa98f19e740 1 -- 172.21.15.44:0/1005927209 wait complete. 2024-04-02T13:32:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.298+0000 7fa98f19e740 1 -- start start 2024-04-02T13:32:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.298+0000 7fa98f19e740 1 Processor -- start 2024-04-02T13:32:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa98f19e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7becd0 0x56158d867420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa98f19e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 0x56158d7caa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa98f19e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7cbf70 0x56158d7caf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa98f19e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56158d7e7ed0 con 0x56158d7cbb90 2024-04-02T13:32:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa98f19e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56158d7d13c0 con 0x56158d7becd0 2024-04-02T13:32:06.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa98f19e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x56158d7e84c0 con 0x56158d7cbf70 2024-04-02T13:32:06.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa98f19e740 10 client.0 _ll_register_callbacks cb 0x7fa97c009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T13:32:06.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa989a74640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 0x56158d7caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa989a74640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 0x56158d7caa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38920/0 (socket says 172.21.15.44:38920) 2024-04-02T13:32:06.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa989a74640 1 -- 172.21.15.44:0/238648350 learned_addr learned my addr 172.21.15.44:0/238648350 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:06.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa98a275640 1 --2- 172.21.15.44:0/238648350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7becd0 0x56158d867420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa98aa76640 1 --2- 172.21.15.44:0/238648350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7cbf70 0x56158d7caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa989a74640 1 -- 172.21.15.44:0/238648350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7cbf70 msgr2=0x56158d7caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:06.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa989a74640 1 --2- 172.21.15.44:0/238648350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7cbf70 0x56158d7caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa989a74640 1 -- 172.21.15.44:0/238648350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7becd0 msgr2=0x56158d867420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:06.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa989a74640 1 --2- 172.21.15.44:0/238648350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7becd0 0x56158d867420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.299+0000 7fa989a74640 1 -- 172.21.15.44:0/238648350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56158d7eb0b0 con 0x56158d7cbb90 2024-04-02T13:32:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa989a74640 1 --2- 172.21.15.44:0/238648350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 0x56158d7caa50 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7fa974089e90 tx=0x7fa974059100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a542f7b70a20f69b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa9657fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-02T13:32:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa9657fa640 1 -- 172.21.15.44:0/238648350 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa974070070 con 0x56158d7cbb90 2024-04-02T13:32:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa9657fa640 1 -- 172.21.15.44:0/238648350 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7fa974062040 con 0x56158d7cbb90 2024-04-02T13:32:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa98f19e740 1 -- 172.21.15.44:0/238648350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa97c034ac0 con 0x56158d7cbb90 2024-04-02T13:32:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa9657fa640 1 -- 172.21.15.44:0/238648350 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fa974021040 con 0x56158d7cbb90 2024-04-02T13:32:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa98f19e740 10 client.10173 fetch_fsmap learned FSMap version 112 2024-04-02T13:32:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa98f19e740 1 -- 172.21.15.44:0/238648350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) -- 0x56158d7ca250 con 0x56158d7cbb90 2024-04-02T13:32:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa9657fa640 1 -- 172.21.15.44:0/238648350 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa97c034ac0 con 0x56158d7cbb90 2024-04-02T13:32:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa9657fa640 1 -- 172.21.15.44:0/238648350 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 112) ==== 3494+0+0 (secure 0 0 0) 0x7fa974024060 con 0x56158d7cbb90 2024-04-02T13:32:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa98f19e740 10 client.10173 fetch_fsmap finished waiting for FSMap version 112 2024-04-02T13:32:06.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa98f19e740 10 client.10173 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T13:32:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa98f19e740 20 client.10173 populate_metadata read hostname 'smithi044' 2024-04-02T13:32:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa98f19e740 1 --2- 172.21.15.44:0/238648350 >> [v2:172.21.15.44:6833/16875964,v1:172.21.15.44:6835/16875964] conn(0x56158d86da30 0x56158d86fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa98f19e740 4 client.10173 mds_command: new command op to 10077 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T13:32:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa98f19e740 1 -- 172.21.15.44:0/238648350 --> [v2:172.21.15.44:6833/16875964,v1:172.21.15.44:6835/16875964] -- command(tid 0: {"prefix": "status"}) -- 0x56158d7155a0 con 0x56158d86da30 2024-04-02T13:32:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.300+0000 7fa98a275640 1 --2- 172.21.15.44:0/238648350 >> [v2:172.21.15.44:6833/16875964,v1:172.21.15.44:6835/16875964] conn(0x56158d86da30 0x56158d86fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa98a275640 1 --2- 172.21.15.44:0/238648350 >> [v2:172.21.15.44:6833/16875964,v1:172.21.15.44:6835/16875964] conn(0x56158d86da30 0x56158d86fe20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=914c02fd6582f65c server_cookie=fad02c3706d0c04e in_seq=0 out_seq=0 2024-04-02T13:32:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa9657fa640 10 client.10173 ms_handle_connect on v2:172.21.15.44:6833/16875964 2024-04-02T13:32:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa9657fa640 1 -- 172.21.15.44:0/238648350 <== mds.0 v2:172.21.15.44:6833/16875964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x56158d7155a0 con 0x56158d86da30 2024-04-02T13:32:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa9657fa640 10 client.10173 handle_command_reply: tid=0 2024-04-02T13:32:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa98f19e740 1 client.10173 shutdown 2024-04-02T13:32:06.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa98f19e740 1 -- 172.21.15.44:0/238648350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 msgr2=0x56158d7caa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa98f19e740 1 --2- 172.21.15.44:0/238648350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 0x56158d7caa50 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7fa974089e90 tx=0x7fa974059100 comp rx=0 tx=0).stop 2024-04-02T13:32:06.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa98f19e740 1 -- 172.21.15.44:0/238648350 shutdown_connections 2024-04-02T13:32:06.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa98f19e740 1 --2- 172.21.15.44:0/238648350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56158d7cbf70 0x56158d7caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa98f19e740 1 --2- 172.21.15.44:0/238648350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56158d7cbb90 0x56158d7caa50 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa98f19e740 1 --2- 172.21.15.44:0/238648350 >> [v2:172.21.15.44:6833/16875964,v1:172.21.15.44:6835/16875964] conn(0x56158d86da30 0x56158d86fe20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa9657fa640 0 client.10173 ms_handle_reset on v2:172.21.15.44:6833/16875964 2024-04-02T13:32:06.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa98f19e740 1 --2- 172.21.15.44:0/238648350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56158d7becd0 0x56158d867420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa98f19e740 1 -- 172.21.15.44:0/238648350 >> 172.21.15.44:0/238648350 conn(0x56158d79f070 msgr2=0x56158d7a5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:06.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa98f19e740 1 -- 172.21.15.44:0/238648350 shutdown_connections 2024-04-02T13:32:06.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa98f19e740 1 -- 172.21.15.44:0/238648350 wait complete. 2024-04-02T13:32:06.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.301+0000 7fa98f19e740 20 client.10173 trim_cache size 0 max 16384 2024-04-02T13:32:06.316 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:32:06.316 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "4101e14a-13fe-452f-8f9a-78614b3ea6ab", 2024-04-02T13:32:06.316 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-02T13:32:06.316 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 10077, 2024-04-02T13:32:06.316 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-02T13:32:06.316 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-02T13:32:06.316 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-02T13:32:06.316 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 4.4574585720000002, 2024-04-02T13:32:06.316 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 112, 2024-04-02T13:32:06.316 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 123, 2024-04-02T13:32:06.317 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 123, 2024-04-02T13:32:06.317 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 6.4310436449999999 2024-04-02T13:32:06.317 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:32:06.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.302+0000 7fa98c4d9640 1 -- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa96806c8f0 msgr2=0x7fa96806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.302+0000 7fa98c4d9640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa96806c8f0 0x7fa96806edb0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fa9800016e0 tx=0x7fa980009040 comp rx=0 tx=0).stop 2024-04-02T13:32:06.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.302+0000 7fa98c4d9640 1 -- 172.21.15.44:0/87148599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa98415e550 msgr2=0x7fa98414e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.302+0000 7fa98c4d9640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa98415e550 0x7fa98414e4e0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7fa97400e870 tx=0x7fa97400ed30 comp rx=0 tx=0).stop 2024-04-02T13:32:06.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.302+0000 7fa98c4d9640 1 -- 172.21.15.44:0/87148599 shutdown_connections 2024-04-02T13:32:06.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.302+0000 7fa98c4d9640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa96806c8f0 0x7fa96806edb0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.302+0000 7fa98c4d9640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415ee70 0x7fa984154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.302+0000 7fa98c4d9640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa98415e550 0x7fa98414e4e0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.302+0000 7fa98c4d9640 1 --2- 172.21.15.44:0/87148599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa98406d700 0x7fa9840ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.302+0000 7fa98c4d9640 1 -- 172.21.15.44:0/87148599 >> 172.21.15.44:0/87148599 conn(0x7fa98409aa20 msgr2=0x7fa9840b6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:06.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.302+0000 7fa98c4d9640 1 -- 172.21.15.44:0/87148599 shutdown_connections 2024-04-02T13:32:06.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.302+0000 7fa98c4d9640 1 -- 172.21.15.44:0/87148599 wait complete. 2024-04-02T13:32:06.320 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T13:32:06.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc09aad3640 1 Processor -- start 2024-04-02T13:32:06.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc09aad3640 1 -- start start 2024-04-02T13:32:06.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc09aad3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc09415aa00 0x7fc0940b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc09aad3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc0940ce2e0 0x7fc094071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc09aad3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc09415ade0 con 0x7fc094072bf0 2024-04-02T13:32:06.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc09aad3640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc0940af1b0 con 0x7fc0940ce2e0 2024-04-02T13:32:06.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc09aad3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc0940af490 con 0x7fc09415aa00 2024-04-02T13:32:06.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc099070640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc0940ce2e0 0x7fc094071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc099070640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc0940ce2e0 0x7fc094071190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38924/0 (socket says 172.21.15.44:38924) 2024-04-02T13:32:06.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc099070640 1 -- 172.21.15.44:0/775820090 learned_addr learned my addr 172.21.15.44:0/775820090 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:06.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc09886f640 1 --2- 172.21.15.44:0/775820090 >> v2:172.21.15.69:3300/0 conn(0x7fc09415aa00 0x7fc0940b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc099070640 1 -- 172.21.15.44:0/775820090 >> v2:172.21.15.69:3300/0 conn(0x7fc09415aa00 msgr2=0x7fc0940b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc099070640 1 --2- 172.21.15.44:0/775820090 >> v2:172.21.15.69:3300/0 conn(0x7fc09415aa00 0x7fc0940b2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc099070640 1 -- 172.21.15.44:0/775820090 >> v1:172.21.15.69:6789/0 conn(0x7fc094072bf0 legacy=0x7fc0940cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.419+0000 7fc099070640 1 -- 172.21.15.44:0/775820090 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0940723f0 con 0x7fc0940ce2e0 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.420+0000 7fc099070640 1 --2- 172.21.15.44:0/775820090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0940ce2e0 0x7fc094071190 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fc09406c600 tx=0x7fc08c02e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653d3b68991b55cc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.420+0000 7fc0937fe640 1 -- 172.21.15.44:0/775820090 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c02ea90 con 0x7fc0940ce2e0 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.420+0000 7fc0937fe640 1 -- 172.21.15.44:0/775820090 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc08c02ec30 con 0x7fc0940ce2e0 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.420+0000 7fc0937fe640 1 -- 172.21.15.44:0/775820090 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c009600 con 0x7fc0940ce2e0 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.421+0000 7fc09aad3640 1 -- 172.21.15.44:0/775820090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0940ce2e0 msgr2=0x7fc094071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.421+0000 7fc09aad3640 1 --2- 172.21.15.44:0/775820090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0940ce2e0 0x7fc094071190 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fc09406c600 tx=0x7fc08c02e0f0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.421+0000 7fc09aad3640 1 -- 172.21.15.44:0/775820090 shutdown_connections 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.421+0000 7fc09aad3640 1 --2- 172.21.15.44:0/775820090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0940ce2e0 0x7fc094071190 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.421+0000 7fc09aad3640 1 --2- 172.21.15.44:0/775820090 >> v2:172.21.15.69:3300/0 conn(0x7fc09415aa00 0x7fc0940b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.421+0000 7fc09aad3640 1 -- 172.21.15.44:0/775820090 >> 172.21.15.44:0/775820090 conn(0x7fc0940cfcd0 msgr2=0x7fc0940d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:06.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.421+0000 7fc09aad3640 1 -- 172.21.15.44:0/775820090 shutdown_connections 2024-04-02T13:32:06.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.421+0000 7fc09aad3640 1 -- 172.21.15.44:0/775820090 wait complete. 2024-04-02T13:32:06.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.422+0000 7fc09aad3640 1 Processor -- start 2024-04-02T13:32:06.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.422+0000 7fc09aad3640 1 -- start start 2024-04-02T13:32:06.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc09aad3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc094072bf0 0x7fc09409c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc09aad3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0940ce2e0 0x7fc094071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc09aad3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc09415aa00 0x7fc094155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc09aad3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc0940731d0 con 0x7fc0940ce2e0 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc09aad3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc094073050 con 0x7fc094072bf0 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc09aad3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc0940cc750 con 0x7fc09415aa00 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc093fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0940ce2e0 0x7fc094071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc093fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0940ce2e0 0x7fc094071400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38936/0 (socket says 172.21.15.44:38936) 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc093fff640 1 -- 172.21.15.44:0/1151475890 learned_addr learned my addr 172.21.15.44:0/1151475890 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc09886f640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc094072bf0 0x7fc09409c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc099070640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc09415aa00 0x7fc094155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc093fff640 1 -- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc09415aa00 msgr2=0x7fc094155ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc093fff640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc09415aa00 0x7fc094155ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc093fff640 1 -- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc094072bf0 msgr2=0x7fc09409c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc093fff640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc094072bf0 0x7fc09409c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc093fff640 1 -- 172.21.15.44:0/1151475890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc094156430 con 0x7fc0940ce2e0 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc093fff640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0940ce2e0 0x7fc094071400 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fc08000c400 tx=0x7fc08000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.423+0000 7fc0917fa640 1 -- 172.21.15.44:0/1151475890 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0800052d0 con 0x7fc0940ce2e0 2024-04-02T13:32:06.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.424+0000 7fc0917fa640 1 -- 172.21.15.44:0/1151475890 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc080005490 con 0x7fc0940ce2e0 2024-04-02T13:32:06.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.424+0000 7fc09aad3640 1 -- 172.21.15.44:0/1151475890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc094156710 con 0x7fc0940ce2e0 2024-04-02T13:32:06.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.424+0000 7fc0917fa640 1 -- 172.21.15.44:0/1151475890 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080007260 con 0x7fc0940ce2e0 2024-04-02T13:32:06.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.424+0000 7fc09aad3640 1 -- 172.21.15.44:0/1151475890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc09409c3e0 con 0x7fc0940ce2e0 2024-04-02T13:32:06.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.425+0000 7fc09aad3640 1 -- 172.21.15.44:0/1151475890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc094074f20 con 0x7fc0940ce2e0 2024-04-02T13:32:06.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.425+0000 7fc0917fa640 1 -- 172.21.15.44:0/1151475890 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc080007400 con 0x7fc0940ce2e0 2024-04-02T13:32:06.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.425+0000 7fc0917fa640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc07806c920 0x7fc07806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:06.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.425+0000 7fc0917fa640 1 -- 172.21.15.44:0/1151475890 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7fc08008b1d0 con 0x7fc0940ce2e0 2024-04-02T13:32:06.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.426+0000 7fc09886f640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc07806c920 0x7fc07806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:06.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.426+0000 7fc09886f640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc07806c920 0x7fc07806ede0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc0940b38d0 tx=0x7fc088008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:06.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.430+0000 7fc0917fa640 1 -- 172.21.15.44:0/1151475890 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc080058e00 con 0x7fc0940ce2e0 2024-04-02T13:32:06.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.652+0000 7fc09aad3640 1 -- 172.21.15.44:0/1151475890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc0941568e0 con 0x7fc0940ce2e0 2024-04-02T13:32:06.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.652+0000 7fc0917fa640 1 -- 172.21.15.44:0/1151475890 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v112) ==== 53+0+73 (secure 0 0 0) 0x7fc08005d100 con 0x7fc0940ce2e0 2024-04-02T13:32:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T13:32:06.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.654+0000 7fc09aad3640 1 -- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc07806c920 msgr2=0x7fc07806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.654+0000 7fc09aad3640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc07806c920 0x7fc07806ede0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc0940b38d0 tx=0x7fc088008040 comp rx=0 tx=0).stop 2024-04-02T13:32:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.654+0000 7fc09aad3640 1 -- 172.21.15.44:0/1151475890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0940ce2e0 msgr2=0x7fc094071400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.654+0000 7fc09aad3640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0940ce2e0 0x7fc094071400 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fc08000c400 tx=0x7fc08000c8c0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.654+0000 7fc09aad3640 1 -- 172.21.15.44:0/1151475890 shutdown_connections 2024-04-02T13:32:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.654+0000 7fc09aad3640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc07806c920 0x7fc07806ede0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.654+0000 7fc09aad3640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc09415aa00 0x7fc094155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.654+0000 7fc09aad3640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0940ce2e0 0x7fc094071400 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.654+0000 7fc09aad3640 1 --2- 172.21.15.44:0/1151475890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc094072bf0 0x7fc09409c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.654+0000 7fc09aad3640 1 -- 172.21.15.44:0/1151475890 >> 172.21.15.44:0/1151475890 conn(0x7fc0940cfcd0 msgr2=0x7fc0940ae2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.654+0000 7fc09aad3640 1 -- 172.21.15.44:0/1151475890 shutdown_connections 2024-04-02T13:32:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06.654+0000 7fc09aad3640 1 -- 172.21.15.44:0/1151475890 wait complete. 2024-04-02T13:32:06.669 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T13:32:06.669 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T13:32:06.669 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-02T13:32:06.669 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T13:32:06.669 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T13:32:06.669 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T13:32:06.669 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T13:32:06.669 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T13:32:06.669 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:32:06.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:06.670 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-02T13:32:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T13:32:06.685 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T13:32:06.685 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T13:32:06.685 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:32:06.685 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-02T13:32:06.685 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:32:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:32:06.685 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-02T13:32:06.685 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-02T13:32:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:32:06.685 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-02T13:32:06.686 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-02T13:32:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T13:32:06.686 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:32:06.686 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-02T13:32:06.686 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-02T13:32:06.686 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42052sec preferred_lft 42052sec 2024-04-02T13:32:06.686 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-02T13:32:06.686 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:32:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T13:32:06.686 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:32:06.686 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-02T13:32:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T13:32:06.686 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff 2024-04-02T13:32:06.687 INFO:teuthology.orchestra.run.smithi044.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T13:32:06.687 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:32:06.687 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::9cc4:b8ff:fe31:57b4/64 scope link 2024-04-02T13:32:06.687 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:32:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:32:06.687 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 52:82:dd:21:27:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:32:06.687 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::5082:ddff:fe21:273e/64 scope link 2024-04-02T13:32:06.687 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:32:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T13:32:06.687 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 02:a4:ff:ae:f4:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:32:06.687 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::a4:ffff:feae:f4f9/64 scope link 2024-04-02T13:32:06.687 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-02T13:32:06.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:06.688 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T13:32:06.750 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:32:06.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:06.752 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-02T13:32:06.808 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-02T13:32:06.808 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T13:32:06.808 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-02T13:32:06.808 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-02T13:32:06.808 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T13:32:06.808 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T13:32:06.808 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-02T13:32:06.809 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-02T13:32:06.809 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-02T13:32:06.809 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-02T13:32:06.809 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-02T13:32:06.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T13:32:06.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:32:06.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:06.943 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:32:06.957 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:32:06.957 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:32:06.957 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:32:06.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:06.957 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-02T13:32:07.024 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-02T13:32:07.024 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:32:07.024 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T13:32:07.079 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T13:32:07.079 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T13:32:07.079 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:32:07.134 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:32:07.201 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:32:07.256 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:32:07.256 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:32:07.256 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:32:07.257 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:32:07.257 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:32:07.257 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:32:07.257 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:32:07.257 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:32:07.257 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:32:07.257 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:32:07.257 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:32:07.258 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:32:07.258 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:32:07.258 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:32:07.258 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:32:07.258 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:32:07.258 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:32:07.258 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:32:07.258 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:32:07.258 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:32:07.258 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:32:07.258 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:32:07.258 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:32:07.258 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:32:07.259 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:32:07.259 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:32:07.259 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:32:07.259 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:32:07.259 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:32:07.259 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:32:07.259 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:32:07.259 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:32:07.259 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:32:07.259 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:32:07.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:07.259 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:32:07.310 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-02T13:32:07.310 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-02T13:32:07.353 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-02T13:32:07.380 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-02T13:32:07.436 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-02T13:32:07.436 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-02T13:32:07.436 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-02T13:32:07.436 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-02T13:32:07.436 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-02T13:32:07.436 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-02T13:32:07.436 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-02T13:32:07.436 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-02T13:32:07.436 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-02T13:32:07.436 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-02T13:32:07.437 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-02T13:32:07.438 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-02T13:32:07.438 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-0 2024-04-02T13:32:07.438 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-1 2024-04-02T13:32:07.438 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-2 2024-04-02T13:32:07.438 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph/osd/ceph-3 2024-04-02T13:32:07.438 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-02T13:32:07.438 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:32:07.438 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:32:07.438 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:32:07.438 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:32:07.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:07.439 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-02T13:32:07.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-02T13:32:07.435+0000 7f55d3064480 -1 init, newargv = 0x563d5c9f70b0 newargc=15 2024-04-02T13:32:07.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[86030]: starting ceph client 2024-04-02T13:32:07.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[86030]: starting fuse 2024-04-02T13:32:07.453 INFO:teuthology.orchestra.run.smithi044.stdout:41 2024-04-02T13:32:07.454 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-02T13:32:07.454 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-02T13:32:07.454 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-02T13:32:07.454 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-02T13:32:07.454 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:32:07.454 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-02T13:32:07.454 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:32:07.454 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-02T13:32:07.454 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-02T13:32:07.454 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-02T13:32:07.455 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-02T13:32:07.456 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:32:07.456 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-02T13:32:07.456 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:32:07.574 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.86030.asok 2024-04-02T13:32:07.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:32:07.585 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.86030.asok 2024-04-02T13:32:07.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:07.585 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.86030.asok status 2024-04-02T13:32:07.726 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:32:07.726 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-02T13:32:07.726 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "191d963ac9de7322da8aff20b567b3fa458e2ef6", 2024-04-02T13:32:07.726 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T13:32:07.726 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-02T13:32:07.726 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-02T13:32:07.726 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-02T13:32:07.726 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "86030", 2024-04-02T13:32:07.726 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-02T13:32:07.726 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:32:07.726 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-02T13:32:07.727 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-02T13:32:07.727 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 10185, 2024-04-02T13:32:07.727 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-02T13:32:07.727 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-02T13:32:07.727 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-02T13:32:07.727 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 10185 2024-04-02T13:32:07.727 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:32:07.727 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:32:07.727 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:32:07.727 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:32:07.727 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 577366573 2024-04-02T13:32:07.727 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-02T13:32:07.727 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 577366573 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.10185 192.168.0.1:0/577366573", 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/577366573", 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 112, 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 123, 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-02T13:32:07.728 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:32:07.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:07.731 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-02T13:32:07.731 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-02T13:32:07.746 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-02T13:32:07.746 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-02T13:32:07.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:07.746 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T13:32:07.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:07.814 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-04-02T13:32:07.879 INFO:teuthology.orchestra.run.smithi044.stdout:./ 2024-04-02T13:32:07.879 INFO:teuthology.orchestra.run.smithi044.stdout:./parent 2024-04-02T13:32:07.879 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/flushed 2024-04-02T13:32:07.879 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/flushed/charlie 2024-04-02T13:32:07.879 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/flushed/alpha 2024-04-02T13:32:07.879 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/flushed/bravo 2024-04-02T13:32:07.879 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/unflushed 2024-04-02T13:32:07.879 INFO:teuthology.orchestra.run.smithi044.stdout:./parent/unflushed/jfile 2024-04-02T13:32:07.880 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:32:07.880 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:32:07.880 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:32:07.880 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:32:07.880 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-04-02T13:32:07.880 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:32:07.985 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-02T13:32:07.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:32:07.993 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:32:07.993 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:32:07.994 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:32:07.994 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:32:07.994 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-04-02T13:32:07.994 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:32:08.099 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627776 2024-04-02T13:32:08.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:32:08.108 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:32:08.108 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:32:08.108 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:32:08.108 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:32:08.108 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-04-02T13:32:08.108 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:32:08.213 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627777 2024-04-02T13:32:08.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:32:08.222 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:32:08.222 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:32:08.222 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:32:08.222 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:32:08.222 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-04-02T13:32:08.222 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:32:08.328 INFO:teuthology.orchestra.run.smithi044.stdout:1099511628280 2024-04-02T13:32:08.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:32:08.336 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:32:08.337 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:32:08.337 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:32:08.337 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:32:08.337 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-04-02T13:32:08.337 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:32:08.442 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627778 2024-04-02T13:32:08.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:32:08.450 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:32:08.450 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:32:08.451 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:32:08.451 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:32:08.451 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/bravo").st_ino) 2024-04-02T13:32:08.451 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:32:08.556 INFO:teuthology.orchestra.run.smithi044.stdout:1099511628279 2024-04-02T13:32:08.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:32:08.564 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:32:08.565 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:32:08.565 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:32:08.565 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:32:08.565 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-04-02T13:32:08.565 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:32:08.672 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627779 2024-04-02T13:32:08.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:32:08.682 DEBUG:teuthology.orchestra.run.smithi044:> stdin-killer -- python3 -c ' 2024-04-02T13:32:08.682 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-02T13:32:08.682 DEBUG:teuthology.orchestra.run.smithi044:> import stat 2024-04-02T13:32:08.682 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-02T13:32:08.682 DEBUG:teuthology.orchestra.run.smithi044:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-04-02T13:32:08.682 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-02T13:32:08.796 INFO:teuthology.orchestra.run.smithi044.stdout:1099511627780 2024-04-02T13:32:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T13:32:08.805 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T13:32:08.806 DEBUG:teuthology.orchestra.run.smithi044:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:32:08.890 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T13:32:08.936 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T13:32:08.936 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:32:08.936 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:32:08.936 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:32:08.952 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:32:09.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.092+0000 7f316932b640 1 Processor -- start 2024-04-02T13:32:09.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.092+0000 7f316932b640 1 -- start start 2024-04-02T13:32:09.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.093+0000 7f316932b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f31640d28f0 0x7f31640d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.093+0000 7f316932b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31640b47b0 0x7f31640b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.093+0000 7f316932b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31640d4d70 con 0x7f31640b5200 2024-04-02T13:32:09.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.093+0000 7f316932b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f31640d5050 con 0x7f31640d28f0 2024-04-02T13:32:09.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.093+0000 7f316932b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f31640d23a0 con 0x7f31640b47b0 2024-04-02T13:32:09.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.093+0000 7f3162ffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f31640d28f0 0x7f31640d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.093+0000 7f31637fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31640b47b0 0x7f31640b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.093+0000 7f31627fc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f31640b5200 0x7f31640b40a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:36298/0 (socket says 172.21.15.44:36298) 2024-04-02T13:32:09.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.093+0000 7f31627fc640 1 -- 172.21.15.44:0/1468727971 learned_addr learned my addr 172.21.15.44:0/1468727971 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:09.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.094+0000 7f3162ffd640 1 -- 172.21.15.44:0/1468727971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31640b47b0 msgr2=0x7f31640b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:09.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.094+0000 7f3162ffd640 1 --2- 172.21.15.44:0/1468727971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31640b47b0 0x7f31640b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.094+0000 7f3162ffd640 1 -- 172.21.15.44:0/1468727971 >> v1:172.21.15.69:6789/0 conn(0x7f31640b5200 legacy=0x7f31640b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.094+0000 7f3161ffb640 1 -- 172.21.15.44:0/1468727971 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3690012331 0 0) 0x7f31640d4d70 con 0x7f31640b5200 2024-04-02T13:32:09.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.094+0000 7f3162ffd640 1 -- 172.21.15.44:0/1468727971 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31640d2030 con 0x7f31640d28f0 2024-04-02T13:32:09.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.094+0000 7f31637fe640 1 --2- 172.21.15.44:0/1468727971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31640b47b0 0x7f31640b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:32:09.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.094+0000 7f3162ffd640 1 --2- 172.21.15.44:0/1468727971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31640d28f0 0x7f31640d2cd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f315c0097a0 tx=0x7f315c02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8c838efdf06101b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:09.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.095+0000 7f3161ffb640 1 -- 172.21.15.44:0/1468727971 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c038070 con 0x7f31640d28f0 2024-04-02T13:32:09.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.095+0000 7f3161ffb640 1 -- 172.21.15.44:0/1468727971 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f315c031700 con 0x7f31640d28f0 2024-04-02T13:32:09.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.095+0000 7f3161ffb640 1 -- 172.21.15.44:0/1468727971 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c031ad0 con 0x7f31640d28f0 2024-04-02T13:32:09.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.095+0000 7f316932b640 1 -- 172.21.15.44:0/1468727971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31640d28f0 msgr2=0x7f31640d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.095+0000 7f316932b640 1 --2- 172.21.15.44:0/1468727971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31640d28f0 0x7f31640d2cd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f315c0097a0 tx=0x7f315c02b7a0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.095+0000 7f316932b640 1 -- 172.21.15.44:0/1468727971 shutdown_connections 2024-04-02T13:32:09.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.095+0000 7f316932b640 1 --2- 172.21.15.44:0/1468727971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31640b47b0 0x7f31640b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.095+0000 7f316932b640 1 --2- 172.21.15.44:0/1468727971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31640d28f0 0x7f31640d2cd0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.095+0000 7f316932b640 1 -- 172.21.15.44:0/1468727971 >> 172.21.15.44:0/1468727971 conn(0x7f31640d4120 msgr2=0x7f31640d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:09.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.095+0000 7f316932b640 1 -- 172.21.15.44:0/1468727971 shutdown_connections 2024-04-02T13:32:09.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.095+0000 7f316932b640 1 -- 172.21.15.44:0/1468727971 wait complete. 2024-04-02T13:32:09.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.096+0000 7f316932b640 1 Processor -- start 2024-04-02T13:32:09.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.096+0000 7f316932b640 1 -- start start 2024-04-02T13:32:09.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.097+0000 7f316932b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31640b47b0 0x7f316415f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.097+0000 7f316932b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31640b5200 0x7f31641f86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.097+0000 7f316932b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31640d28f0 0x7f31640b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.097+0000 7f316932b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f31640d1260 con 0x7f31640b5200 2024-04-02T13:32:09.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.097+0000 7f316932b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31640b5020 con 0x7f31640d28f0 2024-04-02T13:32:09.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.097+0000 7f316932b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f31640d13e0 con 0x7f31640b47b0 2024-04-02T13:32:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.097+0000 7f31627fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31640b5200 0x7f31641f86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.097+0000 7f31627fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31640b5200 0x7f31641f86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38950/0 (socket says 172.21.15.44:38950) 2024-04-02T13:32:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.097+0000 7f31627fc640 1 -- 172.21.15.44:0/500582260 learned_addr learned my addr 172.21.15.44:0/500582260 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.097+0000 7f3162ffd640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31640b47b0 0x7f316415f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.097+0000 7f31637fe640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31640d28f0 0x7f31640b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.098+0000 7f31627fc640 1 -- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31640b47b0 msgr2=0x7f316415f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.098+0000 7f31627fc640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31640b47b0 0x7f316415f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.098+0000 7f31627fc640 1 -- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31640d28f0 msgr2=0x7f31640b8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.098+0000 7f31627fc640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31640d28f0 0x7f31640b8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.098+0000 7f31627fc640 1 -- 172.21.15.44:0/500582260 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31640cade0 con 0x7f31640b5200 2024-04-02T13:32:09.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.098+0000 7f31627fc640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31640b5200 0x7f31641f86d0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f314c00c460 tx=0x7f314c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:09.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.098+0000 7f315b7fe640 1 -- 172.21.15.44:0/500582260 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c0074e0 con 0x7f31640b5200 2024-04-02T13:32:09.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.098+0000 7f315b7fe640 1 -- 172.21.15.44:0/500582260 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f314c0047e0 con 0x7f31640b5200 2024-04-02T13:32:09.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.098+0000 7f316932b640 1 -- 172.21.15.44:0/500582260 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31641f8c10 con 0x7f31640b5200 2024-04-02T13:32:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.098+0000 7f315b7fe640 1 -- 172.21.15.44:0/500582260 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c004bd0 con 0x7f31640b5200 2024-04-02T13:32:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.098+0000 7f316932b640 1 -- 172.21.15.44:0/500582260 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31640b19e0 con 0x7f31640b5200 2024-04-02T13:32:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.099+0000 7f316932b640 1 -- 172.21.15.44:0/500582260 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31640684f0 con 0x7f31640b5200 2024-04-02T13:32:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.100+0000 7f315b7fe640 1 -- 172.21.15.44:0/500582260 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f314c011070 con 0x7f31640b5200 2024-04-02T13:32:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.100+0000 7f315b7fe640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f314406c8f0 0x7f314406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.100+0000 7f315b7fe640 1 -- 172.21.15.44:0/500582260 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7f314c08b930 con 0x7f31640b5200 2024-04-02T13:32:09.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.100+0000 7f3162ffd640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f314406c8f0 0x7f314406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.101+0000 7f3162ffd640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f314406c8f0 0x7f314406edb0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f315c035000 tx=0x7f315c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:09.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.104+0000 7f315b7fe640 1 -- 172.21.15.44:0/500582260 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f31641f8c10 con 0x7f31640b5200 2024-04-02T13:32:09.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.288+0000 7f316932b640 1 -- 172.21.15.44:0/500582260 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f316408bfb0 con 0x7f31640b5200 2024-04-02T13:32:09.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.290+0000 7f315b7fe640 1 -- 172.21.15.44:0/500582260 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) ==== 74+0+17163 (secure 0 0 0) 0x7f314c059a00 con 0x7f31640b5200 2024-04-02T13:32:09.293 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:09.293 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":123,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:59.833349+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:31:09.752845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-02T13:31:10.437454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6835/2895101355":"2024-04-03T13:31:59.833326+0000","172.21.15.44:6833/2895101355":"2024-04-03T13:31:59.833326+0000","172.21.15.44:6835/2109578276":"2024-04-03T13:31:41.102885+0000","172.21.15.44:6833/2109578276":"2024-04-03T13:31:41.102885+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:32:09.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.291+0000 7f316932b640 1 -- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f314406c8f0 msgr2=0x7f314406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.291+0000 7f316932b640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f314406c8f0 0x7f314406edb0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f315c035000 tx=0x7f315c008030 comp rx=0 tx=0).stop 2024-04-02T13:32:09.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.291+0000 7f316932b640 1 -- 172.21.15.44:0/500582260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31640b5200 msgr2=0x7f31641f86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.291+0000 7f316932b640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31640b5200 0x7f31641f86d0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f314c00c460 tx=0x7f314c00c860 comp rx=0 tx=0).stop 2024-04-02T13:32:09.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.292+0000 7f316932b640 1 -- 172.21.15.44:0/500582260 shutdown_connections 2024-04-02T13:32:09.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.292+0000 7f316932b640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f314406c8f0 0x7f314406edb0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.292+0000 7f316932b640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31640d28f0 0x7f31640b8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.292+0000 7f316932b640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31640b5200 0x7f31641f86d0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.292+0000 7f316932b640 1 --2- 172.21.15.44:0/500582260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31640b47b0 0x7f316415f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.292+0000 7f316932b640 1 -- 172.21.15.44:0/500582260 >> 172.21.15.44:0/500582260 conn(0x7f31640d4120 msgr2=0x7f31640b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:09.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.292+0000 7f316932b640 1 -- 172.21.15.44:0/500582260 shutdown_connections 2024-04-02T13:32:09.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.292+0000 7f316932b640 1 -- 172.21.15.44:0/500582260 wait complete. 2024-04-02T13:32:09.305 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:31:09.752845+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:31:10.437454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:32:09.306 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:32:09.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.408+0000 7f29bea55640 1 Processor -- start 2024-04-02T13:32:09.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.408+0000 7f29bea55640 1 -- start start 2024-04-02T13:32:09.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.409+0000 7f29bea55640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f29b80cf5d0 0x7f29b80b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.409+0000 7f29bea55640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29b8160b10 con 0x7f29b80d07b0 2024-04-02T13:32:09.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.409+0000 7f29bea55640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29b815fd50 con 0x7f29b81565d0 2024-04-02T13:32:09.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.409+0000 7f29bea55640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f29b80cfad0 con 0x7f29b80cf5d0 2024-04-02T13:32:09.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.409+0000 7f29bcff2640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f29b80cf5d0 0x7f29b80b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.409+0000 7f29b77fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f29b80d07b0 0x7f29b80ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:60920/0 (socket says 172.21.15.44:60920) 2024-04-02T13:32:09.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.409+0000 7f29b77fe640 1 -- 172.21.15.44:0/3473711338 learned_addr learned my addr 172.21.15.44:0/3473711338 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:09.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.409+0000 7f29bcff2640 1 -- 172.21.15.44:0/3473711338 >> v1:172.21.15.69:6789/0 conn(0x7f29b81565d0 legacy=0x7f29b80d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:09.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.409+0000 7f29bcff2640 1 -- 172.21.15.44:0/3473711338 >> v1:172.21.15.44:6789/0 conn(0x7f29b80d07b0 legacy=0x7f29b80ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.409+0000 7f29bcff2640 1 -- 172.21.15.44:0/3473711338 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b8160030 con 0x7f29b80cf5d0 2024-04-02T13:32:09.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.410+0000 7f29bcff2640 1 --2- 172.21.15.44:0/3473711338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b80cf5d0 0x7f29b80b3ad0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009d90 tx=0x7f29b002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545670109c563360 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:09.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.410+0000 7f29b6ffd640 1 -- 172.21.15.44:0/3473711338 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b0039070 con 0x7f29b80cf5d0 2024-04-02T13:32:09.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.410+0000 7f29b6ffd640 1 -- 172.21.15.44:0/3473711338 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29b002cbd0 con 0x7f29b80cf5d0 2024-04-02T13:32:09.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.410+0000 7f29b6ffd640 1 -- 172.21.15.44:0/3473711338 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b0034440 con 0x7f29b80cf5d0 2024-04-02T13:32:09.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.410+0000 7f29bea55640 1 -- 172.21.15.44:0/3473711338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b80cf5d0 msgr2=0x7f29b80b3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.410+0000 7f29bea55640 1 --2- 172.21.15.44:0/3473711338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b80cf5d0 0x7f29b80b3ad0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009d90 tx=0x7f29b002be20 comp rx=0 tx=0).stop 2024-04-02T13:32:09.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.410+0000 7f29bea55640 1 -- 172.21.15.44:0/3473711338 shutdown_connections 2024-04-02T13:32:09.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.410+0000 7f29bea55640 1 --2- 172.21.15.44:0/3473711338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b80cf5d0 0x7f29b80b3ad0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.411+0000 7f29bea55640 1 -- 172.21.15.44:0/3473711338 >> 172.21.15.44:0/3473711338 conn(0x7f29b806b920 msgr2=0x7f29b806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:09.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.411+0000 7f29bea55640 1 -- 172.21.15.44:0/3473711338 shutdown_connections 2024-04-02T13:32:09.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.411+0000 7f29bea55640 1 -- 172.21.15.44:0/3473711338 wait complete. 2024-04-02T13:32:09.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.412+0000 7f29bea55640 1 Processor -- start 2024-04-02T13:32:09.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.412+0000 7f29bea55640 1 -- start start 2024-04-02T13:32:09.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.412+0000 7f29bea55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b80cf5d0 0x7f29b809aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29bea55640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b80d07b0 0x7f29b81e7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29bea55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b81565d0 0x7f29b81e7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29bea55640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f29b81569b0 con 0x7f29b80d07b0 2024-04-02T13:32:09.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29bea55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f29b81605f0 con 0x7f29b80cf5d0 2024-04-02T13:32:09.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29bea55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f29b8156b30 con 0x7f29b81565d0 2024-04-02T13:32:09.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29b77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b80d07b0 0x7f29b81e7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29b77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b80d07b0 0x7f29b81e7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38956/0 (socket says 172.21.15.44:38956) 2024-04-02T13:32:09.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29b77fe640 1 -- 172.21.15.44:0/1063463982 learned_addr learned my addr 172.21.15.44:0/1063463982 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:09.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29b77fe640 1 -- 172.21.15.44:0/1063463982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b81565d0 msgr2=0x7f29b81e7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29b77fe640 1 --2- 172.21.15.44:0/1063463982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b81565d0 0x7f29b81e7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29b77fe640 1 -- 172.21.15.44:0/1063463982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b80cf5d0 msgr2=0x7f29b809aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29b7fff640 1 --2- 172.21.15.44:0/1063463982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b80cf5d0 0x7f29b809aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29b77fe640 1 --2- 172.21.15.44:0/1063463982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b80cf5d0 0x7f29b809aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29b77fe640 1 -- 172.21.15.44:0/1063463982 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b8159970 con 0x7f29b80d07b0 2024-04-02T13:32:09.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29b77fe640 1 --2- 172.21.15.44:0/1063463982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b80d07b0 0x7f29b81e7580 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f29ac00c430 tx=0x7f29ac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:09.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29b4ff9640 1 -- 172.21.15.44:0/1063463982 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac017070 con 0x7f29b80d07b0 2024-04-02T13:32:09.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29b4ff9640 1 -- 172.21.15.44:0/1063463982 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29ac005ce0 con 0x7f29b80d07b0 2024-04-02T13:32:09.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29b4ff9640 1 -- 172.21.15.44:0/1063463982 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac0072f0 con 0x7f29b80d07b0 2024-04-02T13:32:09.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29bea55640 1 -- 172.21.15.44:0/1063463982 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b8159c50 con 0x7f29b80d07b0 2024-04-02T13:32:09.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.413+0000 7f29bea55640 1 -- 172.21.15.44:0/1063463982 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b81e6d80 con 0x7f29b80d07b0 2024-04-02T13:32:09.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.414+0000 7f29bea55640 1 -- 172.21.15.44:0/1063463982 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b80684f0 con 0x7f29b80d07b0 2024-04-02T13:32:09.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.415+0000 7f29b4ff9640 1 -- 172.21.15.44:0/1063463982 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f29ac01a070 con 0x7f29b80d07b0 2024-04-02T13:32:09.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.415+0000 7f29b4ff9640 1 --2- 172.21.15.44:0/1063463982 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29a406c920 0x7f29a406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.415+0000 7f29b4ff9640 1 -- 172.21.15.44:0/1063463982 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7f29ac08b330 con 0x7f29b80d07b0 2024-04-02T13:32:09.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.415+0000 7f29b7fff640 1 --2- 172.21.15.44:0/1063463982 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29a406c920 0x7f29a406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.416+0000 7f29b7fff640 1 --2- 172.21.15.44:0/1063463982 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29a406c920 0x7f29a406ede0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f29a800a750 tx=0x7f29a8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:09.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.420+0000 7f29b4ff9640 1 -- 172.21.15.44:0/1063463982 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f29ac059370 con 0x7f29b80d07b0 2024-04-02T13:32:09.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.603+0000 7f29bea55640 1 -- 172.21.15.44:0/1063463982 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f29b80ce540 con 0x7f29b80d07b0 2024-04-02T13:32:09.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.603+0000 7f29b4ff9640 1 -- 172.21.15.44:0/1063463982 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v123) ==== 93+0+10 (secure 0 0 0) 0x7f29ac08b0f0 con 0x7f29b80d07b0 2024-04-02T13:32:09.606 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:32:09.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.605+0000 7f29bea55640 1 -- 172.21.15.44:0/1063463982 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29a406c920 msgr2=0x7f29a406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.605+0000 7f29bea55640 1 --2- 172.21.15.44:0/1063463982 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29a406c920 0x7f29a406ede0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f29a800a750 tx=0x7f29a8008040 comp rx=0 tx=0).stop 2024-04-02T13:32:09.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.605+0000 7f29bea55640 1 -- 172.21.15.44:0/1063463982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b80d07b0 msgr2=0x7f29b81e7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.605+0000 7f29bea55640 1 --2- 172.21.15.44:0/1063463982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b80d07b0 0x7f29b81e7580 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f29ac00c430 tx=0x7f29ac00c8f0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.605+0000 7f29bea55640 1 -- 172.21.15.44:0/1063463982 shutdown_connections 2024-04-02T13:32:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.605+0000 7f29bea55640 1 --2- 172.21.15.44:0/1063463982 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f29a406c920 0x7f29a406ede0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.605+0000 7f29bea55640 1 --2- 172.21.15.44:0/1063463982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b81565d0 0x7f29b81e7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.605+0000 7f29bea55640 1 --2- 172.21.15.44:0/1063463982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29b80d07b0 0x7f29b81e7580 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.605+0000 7f29bea55640 1 --2- 172.21.15.44:0/1063463982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b80cf5d0 0x7f29b809aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.605+0000 7f29bea55640 1 -- 172.21.15.44:0/1063463982 >> 172.21.15.44:0/1063463982 conn(0x7f29b806b920 msgr2=0x7f29b80d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.605+0000 7f29bea55640 1 -- 172.21.15.44:0/1063463982 shutdown_connections 2024-04-02T13:32:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.605+0000 7f29bea55640 1 -- 172.21.15.44:0/1063463982 wait complete. 2024-04-02T13:32:09.619 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:32:09.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.720+0000 7fb26b267640 1 Processor -- start 2024-04-02T13:32:09.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.720+0000 7fb26b267640 1 -- start start 2024-04-02T13:32:09.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.721+0000 7fb26b267640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb2640c6b40 0x7fb264166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.721+0000 7fb26b267640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2640cd760 con 0x7fb26415c260 2024-04-02T13:32:09.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.721+0000 7fb26b267640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2640cda40 con 0x7fb2640b4050 2024-04-02T13:32:09.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.721+0000 7fb26b267640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fb2640cdd20 con 0x7fb2640c6b40 2024-04-02T13:32:09.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.721+0000 7fb269003640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb26415c260 0x7fb2640b3940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:60928/0 (socket says 172.21.15.44:60928) 2024-04-02T13:32:09.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.721+0000 7fb269003640 1 -- 172.21.15.44:0/4119857388 learned_addr learned my addr 172.21.15.44:0/4119857388 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:09.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.721+0000 7fb269804640 1 --2- 172.21.15.44:0/4119857388 >> v2:172.21.15.44:3300/0 conn(0x7fb2640c6b40 0x7fb264166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.721+0000 7fb269804640 1 -- 172.21.15.44:0/4119857388 >> v1:172.21.15.69:6789/0 conn(0x7fb2640b4050 legacy=0x7fb2640ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.722+0000 7fb269804640 1 -- 172.21.15.44:0/4119857388 >> v1:172.21.15.44:6789/0 conn(0x7fb26415c260 legacy=0x7fb2640b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.722+0000 7fb269804640 1 -- 172.21.15.44:0/4119857388 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2640ce000 con 0x7fb2640c6b40 2024-04-02T13:32:09.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.722+0000 7fb263fff640 1 -- 172.21.15.44:0/4119857388 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1088327307 0 0) 0x7fb2640cd760 con 0x7fb26415c260 2024-04-02T13:32:09.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.722+0000 7fb269804640 1 --2- 172.21.15.44:0/4119857388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2640c6b40 0x7fb264166cf0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fb254009d90 tx=0x7fb25402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22171d2cae9cdaa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:09.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.722+0000 7fb263fff640 1 -- 172.21.15.44:0/4119857388 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254039070 con 0x7fb2640c6b40 2024-04-02T13:32:09.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.722+0000 7fb263fff640 1 -- 172.21.15.44:0/4119857388 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb25402cd00 con 0x7fb2640c6b40 2024-04-02T13:32:09.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.723+0000 7fb263fff640 1 -- 172.21.15.44:0/4119857388 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254034660 con 0x7fb2640c6b40 2024-04-02T13:32:09.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.723+0000 7fb26b267640 1 -- 172.21.15.44:0/4119857388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2640c6b40 msgr2=0x7fb264166cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.723+0000 7fb26b267640 1 --2- 172.21.15.44:0/4119857388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2640c6b40 0x7fb264166cf0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fb254009d90 tx=0x7fb25402bf50 comp rx=0 tx=0).stop 2024-04-02T13:32:09.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.723+0000 7fb26b267640 1 -- 172.21.15.44:0/4119857388 shutdown_connections 2024-04-02T13:32:09.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.723+0000 7fb26b267640 1 --2- 172.21.15.44:0/4119857388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2640c6b40 0x7fb264166cf0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.723+0000 7fb26b267640 1 -- 172.21.15.44:0/4119857388 >> 172.21.15.44:0/4119857388 conn(0x7fb26409aa20 msgr2=0x7fb26409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:09.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.723+0000 7fb26b267640 1 -- 172.21.15.44:0/4119857388 shutdown_connections 2024-04-02T13:32:09.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.723+0000 7fb26b267640 1 -- 172.21.15.44:0/4119857388 wait complete. 2024-04-02T13:32:09.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.724+0000 7fb26b267640 1 Processor -- start 2024-04-02T13:32:09.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.724+0000 7fb26b267640 1 -- start start 2024-04-02T13:32:09.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.725+0000 7fb26b267640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2640b4050 0x7fb264151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.725+0000 7fb26b267640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2640c6b40 0x7fb2640b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.725+0000 7fb26b267640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb26415c260 0x7fb2641f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.725+0000 7fb26b267640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb2640cd380 con 0x7fb2640c6b40 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.725+0000 7fb26b267640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb2640cd200 con 0x7fb26415c260 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.725+0000 7fb26b267640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb2640cd500 con 0x7fb2640b4050 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.725+0000 7fb268802640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2640c6b40 0x7fb2640b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.725+0000 7fb268802640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2640c6b40 0x7fb2640b7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38972/0 (socket says 172.21.15.44:38972) 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.725+0000 7fb268802640 1 -- 172.21.15.44:0/3845095357 learned_addr learned my addr 172.21.15.44:0/3845095357 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.725+0000 7fb269804640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb26415c260 0x7fb2641f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.725+0000 7fb269003640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2640b4050 0x7fb264151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.726+0000 7fb268802640 1 -- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2640b4050 msgr2=0x7fb264151020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.726+0000 7fb268802640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2640b4050 0x7fb264151020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.726+0000 7fb268802640 1 -- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb26415c260 msgr2=0x7fb2641f5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.726+0000 7fb268802640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb26415c260 0x7fb2641f5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.726+0000 7fb268802640 1 -- 172.21.15.44:0/3845095357 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb264151560 con 0x7fb2640c6b40 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.726+0000 7fb268802640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2640c6b40 0x7fb2640b7410 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fb25800c450 tx=0x7fb25800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.726+0000 7fb261ffb640 1 -- 172.21.15.44:0/3845095357 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2580052d0 con 0x7fb2640c6b40 2024-04-02T13:32:09.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.726+0000 7fb261ffb640 1 -- 172.21.15.44:0/3845095357 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb258005470 con 0x7fb2640c6b40 2024-04-02T13:32:09.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.726+0000 7fb26b267640 1 -- 172.21.15.44:0/3845095357 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2640b4d30 con 0x7fb2640c6b40 2024-04-02T13:32:09.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.726+0000 7fb261ffb640 1 -- 172.21.15.44:0/3845095357 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb258012870 con 0x7fb2640c6b40 2024-04-02T13:32:09.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.726+0000 7fb26b267640 1 -- 172.21.15.44:0/3845095357 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2640b52f0 con 0x7fb2640c6b40 2024-04-02T13:32:09.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.727+0000 7fb26b267640 1 -- 172.21.15.44:0/3845095357 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2640d2c70 con 0x7fb2640c6b40 2024-04-02T13:32:09.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.728+0000 7fb261ffb640 1 -- 172.21.15.44:0/3845095357 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb258012a10 con 0x7fb2640c6b40 2024-04-02T13:32:09.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.728+0000 7fb261ffb640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb25006c8f0 0x7fb25006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:09.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.728+0000 7fb261ffb640 1 -- 172.21.15.44:0/3845095357 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7fb25808b810 con 0x7fb2640c6b40 2024-04-02T13:32:09.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.728+0000 7fb269003640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb25006c8f0 0x7fb25006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:09.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.729+0000 7fb269003640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb25006c8f0 0x7fb25006edb0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb25c000f60 tx=0x7fb25c007e40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:09.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.732+0000 7fb261ffb640 1 -- 172.21.15.44:0/3845095357 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb2640d2c70 con 0x7fb2640c6b40 2024-04-02T13:32:09.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.917+0000 7fb26b267640 1 -- 172.21.15.44:0/3845095357 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb2640684f0 con 0x7fb2640c6b40 2024-04-02T13:32:09.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.917+0000 7fb261ffb640 1 -- 172.21.15.44:0/3845095357 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v123) ==== 104+0+11 (secure 0 0 0) 0x7fb258059440 con 0x7fb2640c6b40 2024-04-02T13:32:09.920 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:32:09.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.919+0000 7fb26b267640 1 -- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb25006c8f0 msgr2=0x7fb25006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.919+0000 7fb26b267640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb25006c8f0 0x7fb25006edb0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb25c000f60 tx=0x7fb25c007e40 comp rx=0 tx=0).stop 2024-04-02T13:32:09.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.919+0000 7fb26b267640 1 -- 172.21.15.44:0/3845095357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2640c6b40 msgr2=0x7fb2640b7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:09.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.919+0000 7fb26b267640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2640c6b40 0x7fb2640b7410 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fb25800c450 tx=0x7fb25800c910 comp rx=0 tx=0).stop 2024-04-02T13:32:09.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.919+0000 7fb26b267640 1 -- 172.21.15.44:0/3845095357 shutdown_connections 2024-04-02T13:32:09.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.919+0000 7fb26b267640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb25006c8f0 0x7fb25006edb0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.919+0000 7fb26b267640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb26415c260 0x7fb2641f5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.919+0000 7fb26b267640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2640c6b40 0x7fb2640b7410 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.919+0000 7fb26b267640 1 --2- 172.21.15.44:0/3845095357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2640b4050 0x7fb264151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:09.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.919+0000 7fb26b267640 1 -- 172.21.15.44:0/3845095357 >> 172.21.15.44:0/3845095357 conn(0x7fb26409aa20 msgr2=0x7fb2640ccaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:09.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.919+0000 7fb26b267640 1 -- 172.21.15.44:0/3845095357 shutdown_connections 2024-04-02T13:32:09.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:09.919+0000 7fb26b267640 1 -- 172.21.15.44:0/3845095357 wait complete. 2024-04-02T13:32:09.933 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:32:10.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.034+0000 7f0d06eeb640 1 Processor -- start 2024-04-02T13:32:10.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.034+0000 7f0d06eeb640 1 -- start start 2024-04-02T13:32:10.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.035+0000 7f0d06eeb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0d000c6b40 0x7f0d00166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:10.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.035+0000 7f0d06eeb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d000cd760 con 0x7f0d000b4050 2024-04-02T13:32:10.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.035+0000 7f0d06eeb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d000cda40 con 0x7f0d0015c260 2024-04-02T13:32:10.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.035+0000 7f0d06eeb640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0d000cdd20 con 0x7f0d000c6b40 2024-04-02T13:32:10.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.035+0000 7f0cfffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0d000b4050 0x7f0d000ccaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:60930/0 (socket says 172.21.15.44:60930) 2024-04-02T13:32:10.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.035+0000 7f0cfffff640 1 -- 172.21.15.44:0/2128400261 learned_addr learned my addr 172.21.15.44:0/2128400261 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:10.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.035+0000 7f0d05488640 1 --2- 172.21.15.44:0/2128400261 >> v2:172.21.15.44:3300/0 conn(0x7f0d000c6b40 0x7f0d00166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:10.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.035+0000 7f0d05488640 1 -- 172.21.15.44:0/2128400261 >> v1:172.21.15.69:6789/0 conn(0x7f0d0015c260 legacy=0x7f0d000b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:10.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.035+0000 7f0d05488640 1 -- 172.21.15.44:0/2128400261 >> v1:172.21.15.44:6789/0 conn(0x7f0d000b4050 legacy=0x7f0d000ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:10.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.035+0000 7f0d05488640 1 -- 172.21.15.44:0/2128400261 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d000ce000 con 0x7f0d000c6b40 2024-04-02T13:32:10.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.036+0000 7f0d05488640 1 --2- 172.21.15.44:0/2128400261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d000c6b40 0x7f0d00166cf0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80097a0 tx=0x7f0ce802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6ba40d3611cb3d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:10.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.036+0000 7f0cff7fe640 1 -- 172.21.15.44:0/2128400261 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8035020 con 0x7f0d000c6b40 2024-04-02T13:32:10.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.036+0000 7f0cff7fe640 1 -- 172.21.15.44:0/2128400261 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ce8031830 con 0x7f0d000c6b40 2024-04-02T13:32:10.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.036+0000 7f0cff7fe640 1 -- 172.21.15.44:0/2128400261 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8031c00 con 0x7f0d000c6b40 2024-04-02T13:32:10.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.037+0000 7f0d06eeb640 1 -- 172.21.15.44:0/2128400261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d000c6b40 msgr2=0x7f0d00166cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:10.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.037+0000 7f0d06eeb640 1 --2- 172.21.15.44:0/2128400261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d000c6b40 0x7f0d00166cf0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80097a0 tx=0x7f0ce802b8d0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.037+0000 7f0d06eeb640 1 -- 172.21.15.44:0/2128400261 shutdown_connections 2024-04-02T13:32:10.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.037+0000 7f0d06eeb640 1 --2- 172.21.15.44:0/2128400261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d000c6b40 0x7f0d00166cf0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.037+0000 7f0d06eeb640 1 -- 172.21.15.44:0/2128400261 >> 172.21.15.44:0/2128400261 conn(0x7f0d0009aa20 msgr2=0x7f0d0009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:10.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.037+0000 7f0d06eeb640 1 -- 172.21.15.44:0/2128400261 shutdown_connections 2024-04-02T13:32:10.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.037+0000 7f0d06eeb640 1 -- 172.21.15.44:0/2128400261 wait complete. 2024-04-02T13:32:10.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.038+0000 7f0d06eeb640 1 Processor -- start 2024-04-02T13:32:10.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.038+0000 7f0d06eeb640 1 -- start start 2024-04-02T13:32:10.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d06eeb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d000b4050 0x7f0d001525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:10.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d06eeb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d000c6b40 0x7f0d0015ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:10.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d06eeb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d0015c260 0x7f0d0015f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:10.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d06eeb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0d000cd380 con 0x7f0d0015c260 2024-04-02T13:32:10.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d06eeb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0d000cd200 con 0x7f0d000c6b40 2024-04-02T13:32:10.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d06eeb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0d000cd500 con 0x7f0d000b4050 2024-04-02T13:32:10.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d05488640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d0015c260 0x7f0d0015f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:10.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d05488640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d0015c260 0x7f0d0015f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38984/0 (socket says 172.21.15.44:38984) 2024-04-02T13:32:10.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d05488640 1 -- 172.21.15.44:0/1206903472 learned_addr learned my addr 172.21.15.44:0/1206903472 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:10.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0cfffff640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d000c6b40 0x7f0d0015ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d04c87640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d000b4050 0x7f0d001525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d05488640 1 -- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d000b4050 msgr2=0x7f0d001525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d05488640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d000b4050 0x7f0d001525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d05488640 1 -- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d000c6b40 msgr2=0x7f0d0015ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d05488640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d000c6b40 0x7f0d0015ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d05488640 1 -- 172.21.15.44:0/1206903472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0015d370 con 0x7f0d0015c260 2024-04-02T13:32:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.039+0000 7f0d05488640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d0015c260 0x7f0d0015f120 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f0ce802b910 tx=0x7f0ce80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.040+0000 7f0cfd7fa640 1 -- 172.21.15.44:0/1206903472 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8041070 con 0x7f0d0015c260 2024-04-02T13:32:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.040+0000 7f0cfd7fa640 1 -- 172.21.15.44:0/1206903472 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ce803a970 con 0x7f0d0015c260 2024-04-02T13:32:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.040+0000 7f0d06eeb640 1 -- 172.21.15.44:0/1206903472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d0015d650 con 0x7f0d0015c260 2024-04-02T13:32:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.040+0000 7f0cfd7fa640 1 -- 172.21.15.44:0/1206903472 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8035040 con 0x7f0d0015c260 2024-04-02T13:32:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.040+0000 7f0d06eeb640 1 -- 172.21.15.44:0/1206903472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d0015dc10 con 0x7f0d0015c260 2024-04-02T13:32:10.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.041+0000 7f0d06eeb640 1 -- 172.21.15.44:0/1206903472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d000684f0 con 0x7f0d0015c260 2024-04-02T13:32:10.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.041+0000 7f0cfd7fa640 1 -- 172.21.15.44:0/1206903472 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ce8038020 con 0x7f0d0015c260 2024-04-02T13:32:10.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.042+0000 7f0cfd7fa640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ce006c8f0 0x7f0ce006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:10.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.042+0000 7f0cfd7fa640 1 -- 172.21.15.44:0/1206903472 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7f0ce80ad7e0 con 0x7f0d0015c260 2024-04-02T13:32:10.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.042+0000 7f0d04c87640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ce006c8f0 0x7f0ce006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:10.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.042+0000 7f0d04c87640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ce006c8f0 0x7f0ce006edb0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8006fd0 tx=0x7f0cf8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:10.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.046+0000 7f0cfd7fa640 1 -- 172.21.15.44:0/1206903472 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0ce807ba60 con 0x7f0d0015c260 2024-04-02T13:32:10.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.233+0000 7f0d06eeb640 1 -- 172.21.15.44:0/1206903472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0d000cf690 con 0x7f0d0015c260 2024-04-02T13:32:10.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.233+0000 7f0cfd7fa640 1 -- 172.21.15.44:0/1206903472 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v123) ==== 100+0+11 (secure 0 0 0) 0x7f0ce807b360 con 0x7f0d0015c260 2024-04-02T13:32:10.236 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:32:10.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.235+0000 7f0d06eeb640 1 -- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ce006c8f0 msgr2=0x7f0ce006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:10.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.235+0000 7f0d06eeb640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ce006c8f0 0x7f0ce006edb0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8006fd0 tx=0x7f0cf8008040 comp rx=0 tx=0).stop 2024-04-02T13:32:10.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.235+0000 7f0d06eeb640 1 -- 172.21.15.44:0/1206903472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d0015c260 msgr2=0x7f0d0015f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:10.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.235+0000 7f0d06eeb640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d0015c260 0x7f0d0015f120 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f0ce802b910 tx=0x7f0ce80046c0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.235+0000 7f0d06eeb640 1 -- 172.21.15.44:0/1206903472 shutdown_connections 2024-04-02T13:32:10.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.235+0000 7f0d06eeb640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0ce006c8f0 0x7f0ce006edb0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.235+0000 7f0d06eeb640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d0015c260 0x7f0d0015f120 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.235+0000 7f0d06eeb640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d000c6b40 0x7f0d0015ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.235+0000 7f0d06eeb640 1 --2- 172.21.15.44:0/1206903472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d000b4050 0x7f0d001525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.235+0000 7f0d06eeb640 1 -- 172.21.15.44:0/1206903472 >> 172.21.15.44:0/1206903472 conn(0x7f0d0009aa20 msgr2=0x7f0d000ce800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:10.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.235+0000 7f0d06eeb640 1 -- 172.21.15.44:0/1206903472 shutdown_connections 2024-04-02T13:32:10.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.235+0000 7f0d06eeb640 1 -- 172.21.15.44:0/1206903472 wait complete. 2024-04-02T13:32:10.249 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T13:32:10.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.349+0000 7fdc36ee3640 1 Processor -- start 2024-04-02T13:32:10.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.349+0000 7fdc36ee3640 1 -- start start 2024-04-02T13:32:10.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc36ee3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc300d0c50 0x7fdc300d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:10.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc36ee3640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdc3006ce40 0x7fdc3006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:10.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc36ee3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc300cc600 con 0x7fdc3015e4a0 2024-04-02T13:32:10.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc36ee3640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fdc300af300 con 0x7fdc3006ce40 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc36ee3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdc300cc8e0 con 0x7fdc300d0c50 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc35480640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdc3006ce40 0x7fdc3006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc35480640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdc3006ce40 0x7fdc3006d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38986/0 (socket says 172.21.15.44:38986) 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc35480640 1 -- 172.21.15.44:0/3140642524 learned_addr learned my addr 172.21.15.44:0/3140642524 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc2ffff640 1 --2- 172.21.15.44:0/3140642524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc300d0c50 0x7fdc300d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc35480640 1 -- 172.21.15.44:0/3140642524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc300d0c50 msgr2=0x7fdc300d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc35480640 1 --2- 172.21.15.44:0/3140642524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc300d0c50 0x7fdc300d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc35480640 1 -- 172.21.15.44:0/3140642524 >> v1:172.21.15.69:6789/0 conn(0x7fdc3015e4a0 legacy=0x7fdc300aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc35480640 1 -- 172.21.15.44:0/3140642524 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc300ccbc0 con 0x7fdc3006ce40 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.350+0000 7fdc2ffff640 1 --2- 172.21.15.44:0/3140642524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc300d0c50 0x7fdc300d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.351+0000 7fdc35480640 1 --2- 172.21.15.44:0/3140642524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc3006ce40 0x7fdc3006d2c0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009e50 tx=0x7fdc2802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a30b6d89fa75940 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.351+0000 7fdc2f7fe640 1 -- 172.21.15.44:0/3140642524 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2802cb00 con 0x7fdc3006ce40 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.351+0000 7fdc2f7fe640 1 -- 172.21.15.44:0/3140642524 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc2802cca0 con 0x7fdc3006ce40 2024-04-02T13:32:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.351+0000 7fdc2f7fe640 1 -- 172.21.15.44:0/3140642524 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc28034610 con 0x7fdc3006ce40 2024-04-02T13:32:10.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.351+0000 7fdc36ee3640 1 -- 172.21.15.44:0/3140642524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc3006ce40 msgr2=0x7fdc3006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:10.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.351+0000 7fdc36ee3640 1 --2- 172.21.15.44:0/3140642524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc3006ce40 0x7fdc3006d2c0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009e50 tx=0x7fdc2802bf60 comp rx=0 tx=0).stop 2024-04-02T13:32:10.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.351+0000 7fdc36ee3640 1 -- 172.21.15.44:0/3140642524 shutdown_connections 2024-04-02T13:32:10.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.351+0000 7fdc36ee3640 1 --2- 172.21.15.44:0/3140642524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc3006ce40 0x7fdc3006d2c0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.351+0000 7fdc36ee3640 1 --2- 172.21.15.44:0/3140642524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc300d0c50 0x7fdc300d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.351+0000 7fdc36ee3640 1 -- 172.21.15.44:0/3140642524 >> 172.21.15.44:0/3140642524 conn(0x7fdc3009aa20 msgr2=0x7fdc3009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:10.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.351+0000 7fdc36ee3640 1 -- 172.21.15.44:0/3140642524 shutdown_connections 2024-04-02T13:32:10.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.352+0000 7fdc36ee3640 1 -- 172.21.15.44:0/3140642524 wait complete. 2024-04-02T13:32:10.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.353+0000 7fdc36ee3640 1 Processor -- start 2024-04-02T13:32:10.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.353+0000 7fdc36ee3640 1 -- start start 2024-04-02T13:32:10.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.353+0000 7fdc36ee3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc3006ce40 0x7fdc30151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.353+0000 7fdc36ee3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc300d0c50 0x7fdc300b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc36ee3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc3015e4a0 0x7fdc30161450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc36ee3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdc300cd460 con 0x7fdc3015e4a0 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc36ee3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdc300cd180 con 0x7fdc300d0c50 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc36ee3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdc3015e880 con 0x7fdc3006ce40 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc35480640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc3015e4a0 0x7fdc30161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc35480640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc3015e4a0 0x7fdc30161450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38998/0 (socket says 172.21.15.44:38998) 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc35480640 1 -- 172.21.15.44:0/481081621 learned_addr learned my addr 172.21.15.44:0/481081621 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc2ffff640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc300d0c50 0x7fdc300b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc34c7f640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc3006ce40 0x7fdc30151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc35480640 1 -- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc3006ce40 msgr2=0x7fdc30151560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc35480640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc3006ce40 0x7fdc30151560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc35480640 1 -- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc300d0c50 msgr2=0x7fdc300b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:10.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc35480640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc300d0c50 0x7fdc300b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc35480640 1 -- 172.21.15.44:0/481081621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc30160190 con 0x7fdc3015e4a0 2024-04-02T13:32:10.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc35480640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc3015e4a0 0x7fdc30161450 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fdc28036040 tx=0x7fdc28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:10.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc2d7fa640 1 -- 172.21.15.44:0/481081621 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc28043020 con 0x7fdc3015e4a0 2024-04-02T13:32:10.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc2d7fa640 1 -- 172.21.15.44:0/481081621 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc2802ce50 con 0x7fdc3015e4a0 2024-04-02T13:32:10.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc36ee3640 1 -- 172.21.15.44:0/481081621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc30160470 con 0x7fdc3015e4a0 2024-04-02T13:32:10.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc2d7fa640 1 -- 172.21.15.44:0/481081621 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc28009510 con 0x7fdc3015e4a0 2024-04-02T13:32:10.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.354+0000 7fdc36ee3640 1 -- 172.21.15.44:0/481081621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc30160c50 con 0x7fdc3015e4a0 2024-04-02T13:32:10.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.355+0000 7fdc36ee3640 1 -- 172.21.15.44:0/481081621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc300684f0 con 0x7fdc3015e4a0 2024-04-02T13:32:10.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.356+0000 7fdc2d7fa640 1 -- 172.21.15.44:0/481081621 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdc28046020 con 0x7fdc3015e4a0 2024-04-02T13:32:10.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.356+0000 7fdc2d7fa640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc1406c920 0x7fdc1406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:10.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.356+0000 7fdc2d7fa640 1 -- 172.21.15.44:0/481081621 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7fdc280ae390 con 0x7fdc3015e4a0 2024-04-02T13:32:10.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.357+0000 7fdc34c7f640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc1406c920 0x7fdc1406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:10.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.357+0000 7fdc34c7f640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc1406c920 0x7fdc1406ede0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fdc240016e0 tx=0x7fdc24009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:10.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.361+0000 7fdc2d7fa640 1 -- 172.21.15.44:0/481081621 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fdc2804d030 con 0x7fdc3015e4a0 2024-04-02T13:32:10.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.557+0000 7fdc36ee3640 1 -- 172.21.15.44:0/481081621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdc300aef70 con 0x7fdc3015e4a0 2024-04-02T13:32:10.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.557+0000 7fdc2d7fa640 1 -- 172.21.15.44:0/481081621 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v123) ==== 80+0+244 (secure 0 0 0) 0x7fdc2807bf20 con 0x7fdc3015e4a0 2024-04-02T13:32:10.560 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6835/2895101355 2024-04-03T13:31:59.833326+0000 2024-04-02T13:32:10.561 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6833/2895101355 2024-04-03T13:31:59.833326+0000 2024-04-02T13:32:10.561 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6835/2109578276 2024-04-03T13:31:41.102885+0000 2024-04-02T13:32:10.561 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6833/2109578276 2024-04-03T13:31:41.102885+0000 2024-04-02T13:32:10.561 INFO:teuthology.orchestra.run.smithi044.stderr:listed 4 entries 2024-04-02T13:32:10.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.559+0000 7fdc36ee3640 1 -- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc1406c920 msgr2=0x7fdc1406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:10.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.559+0000 7fdc36ee3640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc1406c920 0x7fdc1406ede0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fdc240016e0 tx=0x7fdc24009040 comp rx=0 tx=0).stop 2024-04-02T13:32:10.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.559+0000 7fdc36ee3640 1 -- 172.21.15.44:0/481081621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc3015e4a0 msgr2=0x7fdc30161450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:10.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.559+0000 7fdc36ee3640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc3015e4a0 0x7fdc30161450 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fdc28036040 tx=0x7fdc28004a40 comp rx=0 tx=0).stop 2024-04-02T13:32:10.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.559+0000 7fdc36ee3640 1 -- 172.21.15.44:0/481081621 shutdown_connections 2024-04-02T13:32:10.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.559+0000 7fdc36ee3640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdc1406c920 0x7fdc1406ede0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.559+0000 7fdc36ee3640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc3015e4a0 0x7fdc30161450 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.559+0000 7fdc36ee3640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc300d0c50 0x7fdc300b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.559+0000 7fdc36ee3640 1 --2- 172.21.15.44:0/481081621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc3006ce40 0x7fdc30151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:10.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.559+0000 7fdc36ee3640 1 -- 172.21.15.44:0/481081621 >> 172.21.15.44:0/481081621 conn(0x7fdc3009aa20 msgr2=0x7fdc30070540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:10.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.559+0000 7fdc36ee3640 1 -- 172.21.15.44:0/481081621 shutdown_connections 2024-04-02T13:32:10.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:10.559+0000 7fdc36ee3640 1 -- 172.21.15.44:0/481081621 wait complete. 2024-04-02T13:32:10.573 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-02T13:32:10.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:10.573 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-02T13:32:10.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[86030]: fuse finished with error 0 and tester_r 0 2024-04-02T13:32:10.600 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:32:10.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:10.600 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T13:32:10.684 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:32:10.684 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:32:10.684 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:32:10.703 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:32:10.703 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:32:10.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:10.703 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-02T13:32:10.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:32:10.762 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-02T13:32:10.762 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:32:10.762 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:32:10.762 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:32:10.777 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:32:10.777 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:32:10.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:10.777 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-02T13:32:10.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:32:10.832 INFO:teuthology.orchestra.run.smithi044.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-02T13:32:10.832 INFO:tasks.cephfs.mount:Terminating background process 2024-04-02T13:32:10.832 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:32:10.832 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:32:10.850 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:32:10.850 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:32:10.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:10.850 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-02T13:32:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:32:10.908 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-02T13:32:10.909 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:32:11.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.007+0000 7f73f1dc8640 1 Processor -- start 2024-04-02T13:32:11.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.007+0000 7f73f1dc8640 1 -- start start 2024-04-02T13:32:11.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73f1dc8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f73ec1617e0 0x7f73ec0ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73f1dc8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73ec0cef30 con 0x7f73ec0cffd0 2024-04-02T13:32:11.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73f1dc8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73ec0cf210 con 0x7f73ec0cd3a0 2024-04-02T13:32:11.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73f1dc8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f73ec0cf4f0 con 0x7f73ec1617e0 2024-04-02T13:32:11.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73eaffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f73ec0cffd0 0x7f73ec0ccc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:60936/0 (socket says 172.21.15.44:60936) 2024-04-02T13:32:11.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73eaffd640 1 -- 172.21.15.44:0/1721727872 learned_addr learned my addr 172.21.15.44:0/1721727872 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:11.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73eb7fe640 1 --2- 172.21.15.44:0/1721727872 >> v2:172.21.15.69:3300/0 conn(0x7f73ec1617e0 0x7f73ec0ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 384266881 0 0) 0x7f73ec0cef30 con 0x7f73ec0cffd0 2024-04-02T13:32:11.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73d4003520 con 0x7f73ec0cffd0 2024-04-02T13:32:11.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1850985965 0 0) 0x7f73d4003520 con 0x7f73ec0cffd0 2024-04-02T13:32:11.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 >> v2:172.21.15.69:3300/0 conn(0x7f73ec1617e0 msgr2=0x7f73ec0ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:11.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 --2- 172.21.15.44:0/1721727872 >> v2:172.21.15.69:3300/0 conn(0x7f73ec1617e0 0x7f73ec0ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 >> v1:172.21.15.69:6789/0 conn(0x7f73ec0cd3a0 legacy=0x7f73ec0ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ec0cf7d0 con 0x7f73ec0cffd0 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f73e0001670 con 0x7f73ec0cffd0 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 >> v1:172.21.15.44:6789/0 conn(0x7f73ec0cffd0 legacy=0x7f73ec0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 --2- 172.21.15.44:0/1721727872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73d40039b0 0x7f73d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 --2- 172.21.15.44:0/1721727872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73d4004800 0x7f73d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 --2- 172.21.15.44:0/1721727872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73d40051c0 0x7f73d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f73ec0cf4f0 con 0x7f73d40051c0 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f73d400c060 con 0x7f73d40039b0 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.008+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f73d400c260 con 0x7f73d4004800 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.009+0000 7f73ebfff640 1 --2- 172.21.15.44:0/1721727872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73d40051c0 0x7f73d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.009+0000 7f73eb7fe640 1 --2- 172.21.15.44:0/1721727872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73d40039b0 0x7f73d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.009+0000 7f73eaffd640 1 --2- 172.21.15.44:0/1721727872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73d4004800 0x7f73d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.009+0000 7f73eb7fe640 1 -- 172.21.15.44:0/1721727872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73d4004800 msgr2=0x7f73d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.009+0000 7f73eb7fe640 1 --2- 172.21.15.44:0/1721727872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73d4004800 0x7f73d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.009+0000 7f73eb7fe640 1 -- 172.21.15.44:0/1721727872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73d40051c0 msgr2=0x7f73d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.009+0000 7f73eb7fe640 1 --2- 172.21.15.44:0/1721727872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73d40051c0 0x7f73d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.009+0000 7f73eb7fe640 1 -- 172.21.15.44:0/1721727872 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d4003520 con 0x7f73d40039b0 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.009+0000 7f73ebfff640 1 --2- 172.21.15.44:0/1721727872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73d40051c0 0x7f73d400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.009+0000 7f73ebfff640 1 -- 172.21.15.44:0/1721727872 reap_dead start 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.010+0000 7f73eb7fe640 1 --2- 172.21.15.44:0/1721727872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73d40039b0 0x7f73d4003dd0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009ac0 tx=0x7f73d4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98c6a1f14e4bcc90 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.010+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc039070 con 0x7f73d40039b0 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.010+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73dc02c6b0 con 0x7f73d40039b0 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.010+0000 7f73ea7fc640 1 -- 172.21.15.44:0/1721727872 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc009690 con 0x7f73d40039b0 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.010+0000 7f73f1dc8640 1 -- 172.21.15.44:0/1721727872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73d40039b0 msgr2=0x7f73d4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.010+0000 7f73f1dc8640 1 --2- 172.21.15.44:0/1721727872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73d40039b0 0x7f73d4003dd0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009ac0 tx=0x7f73d4005830 comp rx=0 tx=0).stop 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.011+0000 7f73f1dc8640 1 -- 172.21.15.44:0/1721727872 shutdown_connections 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.011+0000 7f73f1dc8640 1 --2- 172.21.15.44:0/1721727872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73d40039b0 0x7f73d4003dd0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.011+0000 7f73f1dc8640 1 -- 172.21.15.44:0/1721727872 >> 172.21.15.44:0/1721727872 conn(0x7f73ec0b20e0 msgr2=0x7f73ec0b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.011+0000 7f73f1dc8640 1 -- 172.21.15.44:0/1721727872 shutdown_connections 2024-04-02T13:32:11.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.011+0000 7f73f1dc8640 1 -- 172.21.15.44:0/1721727872 wait complete. 2024-04-02T13:32:11.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.012+0000 7f73f1dc8640 1 Processor -- start 2024-04-02T13:32:11.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.012+0000 7f73f1dc8640 1 -- start start 2024-04-02T13:32:11.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73f1dc8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73d40039b0 0x7f73ec158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73f1dc8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73ec15a0b0 0x7f73ec14a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73f1dc8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73ec0c9180 0x7f73ec0c9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73f1dc8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f73ec161d40 con 0x7f73ec15a0b0 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73f1dc8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f73ec161bc0 con 0x7f73ec0c9180 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73f1dc8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f73ec0ce8c0 con 0x7f73d40039b0 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73eaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73ec15a0b0 0x7f73ec14a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73eaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73ec15a0b0 0x7f73ec14a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39008/0 (socket says 172.21.15.44:39008) 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73eaffd640 1 -- 172.21.15.44:0/307241318 learned_addr learned my addr 172.21.15.44:0/307241318 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73ebfff640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73ec0c9180 0x7f73ec0c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73eb7fe640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73d40039b0 0x7f73ec158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73eaffd640 1 -- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73d40039b0 msgr2=0x7f73ec158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73eaffd640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73d40039b0 0x7f73ec158250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73eaffd640 1 -- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73ec0c9180 msgr2=0x7f73ec0c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73eaffd640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73ec0c9180 0x7f73ec0c9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73eaffd640 1 -- 172.21.15.44:0/307241318 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ec15a740 con 0x7f73ec15a0b0 2024-04-02T13:32:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73ebfff640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73ec0c9180 0x7f73ec0c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:11.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73eaffd640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73ec15a0b0 0x7f73ec14a9a0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f73e00072c0 tx=0x7f73e0007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:11.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73d3fff640 1 -- 172.21.15.44:0/307241318 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e001a070 con 0x7f73ec15a0b0 2024-04-02T13:32:11.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73d3fff640 1 -- 172.21.15.44:0/307241318 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73e000f960 con 0x7f73ec15a0b0 2024-04-02T13:32:11.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73f1dc8640 1 -- 172.21.15.44:0/307241318 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73ec15aa20 con 0x7f73ec15a0b0 2024-04-02T13:32:11.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73d3fff640 1 -- 172.21.15.44:0/307241318 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e000fd50 con 0x7f73ec15a0b0 2024-04-02T13:32:11.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.013+0000 7f73f1dc8640 1 -- 172.21.15.44:0/307241318 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73ec15afe0 con 0x7f73ec15a0b0 2024-04-02T13:32:11.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.014+0000 7f73f1dc8640 1 -- 172.21.15.44:0/307241318 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73ec0c6e90 con 0x7f73ec15a0b0 2024-04-02T13:32:11.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.015+0000 7f73d3fff640 1 -- 172.21.15.44:0/307241318 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f73e001d070 con 0x7f73ec15a0b0 2024-04-02T13:32:11.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.015+0000 7f73d3fff640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73d806c920 0x7f73d806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.016+0000 7f73d3fff640 1 -- 172.21.15.44:0/307241318 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7f73e008b620 con 0x7f73ec15a0b0 2024-04-02T13:32:11.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.016+0000 7f73eb7fe640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73d806c920 0x7f73d806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.016+0000 7f73eb7fe640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73d806c920 0x7f73d806ede0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f73dc036000 tx=0x7f73dc030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:11.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.020+0000 7f73d3fff640 1 -- 172.21.15.44:0/307241318 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f73ec0c6e90 con 0x7f73ec15a0b0 2024-04-02T13:32:11.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.245+0000 7f73f1dc8640 1 -- 172.21.15.44:0/307241318 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73ec0684f0 con 0x7f73ec15a0b0 2024-04-02T13:32:11.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.245+0000 7f73d3fff640 1 -- 172.21.15.44:0/307241318 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5184 (secure 0 0 0) 0x7f73e00598a0 con 0x7f73ec15a0b0 2024-04-02T13:32:11.248 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:11.248 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:31:11.495701+0000","modified":"2024-04-02T13:32:04.851323+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6835/16875964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":16875964},{"type":"v1","addr":"172.21.15.44:6835","nonce":16875964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":8}]} 2024-04-02T13:32:11.248 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 112 2024-04-02T13:32:11.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.247+0000 7f73f1dc8640 1 -- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73d806c920 msgr2=0x7f73d806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.247+0000 7f73f1dc8640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73d806c920 0x7f73d806ede0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f73dc036000 tx=0x7f73dc030000 comp rx=0 tx=0).stop 2024-04-02T13:32:11.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.247+0000 7f73f1dc8640 1 -- 172.21.15.44:0/307241318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73ec15a0b0 msgr2=0x7f73ec14a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.247+0000 7f73f1dc8640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73ec15a0b0 0x7f73ec14a9a0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f73e00072c0 tx=0x7f73e0007780 comp rx=0 tx=0).stop 2024-04-02T13:32:11.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.247+0000 7f73f1dc8640 1 -- 172.21.15.44:0/307241318 shutdown_connections 2024-04-02T13:32:11.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.247+0000 7f73f1dc8640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f73d806c920 0x7f73d806ede0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.247+0000 7f73f1dc8640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73ec0c9180 0x7f73ec0c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.247+0000 7f73f1dc8640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73ec15a0b0 0x7f73ec14a9a0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.247+0000 7f73f1dc8640 1 --2- 172.21.15.44:0/307241318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73d40039b0 0x7f73ec158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.247+0000 7f73f1dc8640 1 -- 172.21.15.44:0/307241318 >> 172.21.15.44:0/307241318 conn(0x7f73ec0b20e0 msgr2=0x7f73ec071870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:11.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.248+0000 7f73f1dc8640 1 -- 172.21.15.44:0/307241318 shutdown_connections 2024-04-02T13:32:11.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.248+0000 7f73f1dc8640 1 -- 172.21.15.44:0/307241318 wait complete. 2024-04-02T13:32:11.261 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:32:11.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.360+0000 7fe9d6e3a640 1 Processor -- start 2024-04-02T13:32:11.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.360+0000 7fe9d6e3a640 1 -- start start 2024-04-02T13:32:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.361+0000 7fe9d6e3a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe9d006cf90 0x7fe9d00cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.361+0000 7fe9d6e3a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d00cfe30 0x7fe9d00d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.361+0000 7fe9d6e3a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9d006c1d0 con 0x7fe9d00d0a30 2024-04-02T13:32:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.361+0000 7fe9d6e3a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe9d006cb00 con 0x7fe9d006cf90 2024-04-02T13:32:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.361+0000 7fe9d6e3a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe9d006d3f0 con 0x7fe9d00cfe30 2024-04-02T13:32:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.362+0000 7fe9d53d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d00cfe30 0x7fe9d00d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.362+0000 7fe9cffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe9d006cf90 0x7fe9d00cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.362+0000 7fe9d53d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d00cfe30 0x7fe9d00d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:43142/0 (socket says 172.21.15.44:43142) 2024-04-02T13:32:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.362+0000 7fe9cffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe9d006cf90 0x7fe9d00cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:41428/0 (socket says 172.21.15.44:41428) 2024-04-02T13:32:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.362+0000 7fe9d53d7640 1 -- 172.21.15.44:0/3865722524 learned_addr learned my addr 172.21.15.44:0/3865722524 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.362+0000 7fe9d4bd6640 1 --1- 172.21.15.44:0/3865722524 >> v1:172.21.15.69:6789/0 conn(0x7fe9d00d0a30 0x7fe9d00d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:36348/0 (socket says 172.21.15.44:36348) 2024-04-02T13:32:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.362+0000 7fe9cffff640 1 -- 172.21.15.44:0/3865722524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d00cfe30 msgr2=0x7fe9d00d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.362+0000 7fe9cffff640 1 --2- 172.21.15.44:0/3865722524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d00cfe30 0x7fe9d00d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.362+0000 7fe9cffff640 1 -- 172.21.15.44:0/3865722524 >> v1:172.21.15.69:6789/0 conn(0x7fe9d00d0a30 legacy=0x7fe9d00d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.362+0000 7fe9cffff640 1 -- 172.21.15.44:0/3865722524 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d006c4b0 con 0x7fe9d006cf90 2024-04-02T13:32:11.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.362+0000 7fe9cf7fe640 1 -- 172.21.15.44:0/3865722524 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3231762342 0 0) 0x7fe9d006c1d0 con 0x7fe9d00d0a30 2024-04-02T13:32:11.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.363+0000 7fe9cffff640 1 --2- 172.21.15.44:0/3865722524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d006cf90 0x7fe9d00cf8f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0097a0 tx=0x7fe9bc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e7a661d31a3606a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:11.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.363+0000 7fe9cf7fe640 1 -- 172.21.15.44:0/3865722524 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc0315a0 con 0x7fe9d006cf90 2024-04-02T13:32:11.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.363+0000 7fe9cf7fe640 1 -- 172.21.15.44:0/3865722524 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9bc037070 con 0x7fe9d006cf90 2024-04-02T13:32:11.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.363+0000 7fe9cf7fe640 1 -- 172.21.15.44:0/3865722524 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc031960 con 0x7fe9d006cf90 2024-04-02T13:32:11.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.363+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3865722524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d006cf90 msgr2=0x7fe9d00cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.363+0000 7fe9d6e3a640 1 --2- 172.21.15.44:0/3865722524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d006cf90 0x7fe9d00cf8f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0097a0 tx=0x7fe9bc02b840 comp rx=0 tx=0).stop 2024-04-02T13:32:11.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.363+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3865722524 shutdown_connections 2024-04-02T13:32:11.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.363+0000 7fe9d6e3a640 1 --2- 172.21.15.44:0/3865722524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d006cf90 0x7fe9d00cf8f0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.363+0000 7fe9d6e3a640 1 --2- 172.21.15.44:0/3865722524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d00cfe30 0x7fe9d00d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.363+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3865722524 >> 172.21.15.44:0/3865722524 conn(0x7fe9d009d470 msgr2=0x7fe9d00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:11.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.364+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3865722524 shutdown_connections 2024-04-02T13:32:11.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.364+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3865722524 wait complete. 2024-04-02T13:32:11.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.365+0000 7fe9d6e3a640 1 Processor -- start 2024-04-02T13:32:11.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.365+0000 7fe9d6e3a640 1 -- start start 2024-04-02T13:32:11.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.365+0000 7fe9d6e3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d006cf90 0x7fe9d01536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.365+0000 7fe9d6e3a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d00cfe30 0x7fe9d0161050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.365+0000 7fe9d6e3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d00d0a30 0x7fe9d01608a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d6e3a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d015e870 con 0x7fe9d00d0a30 2024-04-02T13:32:11.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d6e3a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d015e6f0 con 0x7fe9d006cf90 2024-04-02T13:32:11.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d6e3a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe9d00d4da0 con 0x7fe9d00cfe30 2024-04-02T13:32:11.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d53d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d00d0a30 0x7fe9d01608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d53d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d00d0a30 0x7fe9d01608a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39014/0 (socket says 172.21.15.44:39014) 2024-04-02T13:32:11.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d53d7640 1 -- 172.21.15.44:0/3979369707 learned_addr learned my addr 172.21.15.44:0/3979369707 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:11.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9cffff640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d00cfe30 0x7fe9d0161050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d4bd6640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d006cf90 0x7fe9d01536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d53d7640 1 -- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d00cfe30 msgr2=0x7fe9d0161050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d53d7640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d00cfe30 0x7fe9d0161050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d53d7640 1 -- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d006cf90 msgr2=0x7fe9d01536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d53d7640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d006cf90 0x7fe9d01536d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d53d7640 1 -- 172.21.15.44:0/3979369707 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d015ea20 con 0x7fe9d00d0a30 2024-04-02T13:32:11.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d4bd6640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d006cf90 0x7fe9d01536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:11.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d53d7640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d00d0a30 0x7fe9d01608a0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800efc0 tx=0x7fe9c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:11.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9cd7fa640 1 -- 172.21.15.44:0/3979369707 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c800cd20 con 0x7fe9d00d0a30 2024-04-02T13:32:11.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9cd7fa640 1 -- 172.21.15.44:0/3979369707 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9c8003990 con 0x7fe9d00d0a30 2024-04-02T13:32:11.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9cd7fa640 1 -- 172.21.15.44:0/3979369707 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8003d80 con 0x7fe9d00d0a30 2024-04-02T13:32:11.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.366+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3979369707 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9d0161750 con 0x7fe9d00d0a30 2024-04-02T13:32:11.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.367+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3979369707 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9d015efc0 con 0x7fe9d00d0a30 2024-04-02T13:32:11.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.367+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3979369707 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9d00684f0 con 0x7fe9d00d0a30 2024-04-02T13:32:11.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.368+0000 7fe9cd7fa640 1 -- 172.21.15.44:0/3979369707 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe9c8005d10 con 0x7fe9d00d0a30 2024-04-02T13:32:11.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.368+0000 7fe9cd7fa640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe9b406c8f0 0x7fe9b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.369+0000 7fe9cd7fa640 1 -- 172.21.15.44:0/3979369707 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7fe9c808c110 con 0x7fe9d00d0a30 2024-04-02T13:32:11.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.369+0000 7fe9d4bd6640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe9b406c8f0 0x7fe9b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.369+0000 7fe9d4bd6640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe9b406c8f0 0x7fe9b406edb0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4000f60 tx=0x7fe9c4007c10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:11.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.373+0000 7fe9cd7fa640 1 -- 172.21.15.44:0/3979369707 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe9c8022020 con 0x7fe9d00d0a30 2024-04-02T13:32:11.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.558+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3979369707 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe9d00d17c0 con 0x7fe9d00d0a30 2024-04-02T13:32:11.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.559+0000 7fe9cd7fa640 1 -- 172.21.15.44:0/3979369707 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) ==== 74+0+17163 (secure 0 0 0) 0x7fe9c8010040 con 0x7fe9d00d0a30 2024-04-02T13:32:11.562 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:11.562 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":123,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:59.833349+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:31:09.752845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-02T13:31:10.437454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6835/2895101355":"2024-04-03T13:31:59.833326+0000","172.21.15.44:6833/2895101355":"2024-04-03T13:31:59.833326+0000","172.21.15.44:6835/2109578276":"2024-04-03T13:31:41.102885+0000","172.21.15.44:6833/2109578276":"2024-04-03T13:31:41.102885+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:32:11.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.561+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe9b406c8f0 msgr2=0x7fe9b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.561+0000 7fe9d6e3a640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe9b406c8f0 0x7fe9b406edb0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4000f60 tx=0x7fe9c4007c10 comp rx=0 tx=0).stop 2024-04-02T13:32:11.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.561+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3979369707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d00d0a30 msgr2=0x7fe9d01608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.561+0000 7fe9d6e3a640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d00d0a30 0x7fe9d01608a0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800efc0 tx=0x7fe9c800c490 comp rx=0 tx=0).stop 2024-04-02T13:32:11.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.561+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3979369707 shutdown_connections 2024-04-02T13:32:11.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.561+0000 7fe9d6e3a640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe9b406c8f0 0x7fe9b406edb0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.561+0000 7fe9d6e3a640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d00d0a30 0x7fe9d01608a0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.561+0000 7fe9d6e3a640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d00cfe30 0x7fe9d0161050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.561+0000 7fe9d6e3a640 1 --2- 172.21.15.44:0/3979369707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d006cf90 0x7fe9d01536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.561+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3979369707 >> 172.21.15.44:0/3979369707 conn(0x7fe9d009d470 msgr2=0x7fe9d00d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:11.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.561+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3979369707 shutdown_connections 2024-04-02T13:32:11.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.561+0000 7fe9d6e3a640 1 -- 172.21.15.44:0/3979369707 wait complete. 2024-04-02T13:32:11.575 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T13:20:14.888847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T13:31:09.752845+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T13:31:10.437454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T13:32:11.576 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T13:32:11.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.679+0000 7f8b999ba640 1 Processor -- start 2024-04-02T13:32:11.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.679+0000 7f8b999ba640 1 -- start start 2024-04-02T13:32:11.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b999ba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8b9415e5d0 0x7f8b940d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b999ba640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f8b940b4180 0x7f8b940b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b999ba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b940d1630 con 0x7f8b9406cff0 2024-04-02T13:32:11.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b999ba640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f8b940d1bf0 con 0x7f8b940b4180 2024-04-02T13:32:11.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b999ba640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8b940d1910 con 0x7f8b9415e5d0 2024-04-02T13:32:11.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b92ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f8b9406cff0 0x7f8b9406d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:60948/0 (socket says 172.21.15.44:60948) 2024-04-02T13:32:11.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b92ffd640 1 -- 172.21.15.44:0/130856224 learned_addr learned my addr 172.21.15.44:0/130856224 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:11.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b937fe640 1 --2- 172.21.15.44:0/130856224 >> v2:172.21.15.44:3300/0 conn(0x7f8b940b4180 0x7f8b940b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b927fc640 1 --2- 172.21.15.44:0/130856224 >> v2:172.21.15.69:3300/0 conn(0x7f8b9415e5d0 0x7f8b940d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b937fe640 1 -- 172.21.15.44:0/130856224 >> v2:172.21.15.69:3300/0 conn(0x7f8b9415e5d0 msgr2=0x7f8b940d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:11.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b937fe640 1 --2- 172.21.15.44:0/130856224 >> v2:172.21.15.69:3300/0 conn(0x7f8b9415e5d0 0x7f8b940d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b937fe640 1 -- 172.21.15.44:0/130856224 >> v1:172.21.15.44:6789/0 conn(0x7f8b9406cff0 legacy=0x7f8b9406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b937fe640 1 -- 172.21.15.44:0/130856224 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b940b1540 con 0x7f8b940b4180 2024-04-02T13:32:11.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b91ffb640 1 -- 172.21.15.44:0/130856224 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2907270084 0 0) 0x7f8b940d1630 con 0x7f8b9406cff0 2024-04-02T13:32:11.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.680+0000 7f8b927fc640 1 --2- 172.21.15.44:0/130856224 >> v2:172.21.15.69:3300/0 conn(0x7f8b9415e5d0 0x7f8b940d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:11.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.681+0000 7f8b937fe640 1 --2- 172.21.15.44:0/130856224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b940b4180 0x7f8b940b1d40 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f8b94161e40 tx=0x7f8b8402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5432ce9dd66bf0a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:11.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.681+0000 7f8b91ffb640 1 -- 172.21.15.44:0/130856224 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b84039070 con 0x7f8b940b4180 2024-04-02T13:32:11.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.681+0000 7f8b91ffb640 1 -- 172.21.15.44:0/130856224 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b8402cbf0 con 0x7f8b940b4180 2024-04-02T13:32:11.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.681+0000 7f8b91ffb640 1 -- 172.21.15.44:0/130856224 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b84034440 con 0x7f8b940b4180 2024-04-02T13:32:11.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.682+0000 7f8b999ba640 1 -- 172.21.15.44:0/130856224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b940b4180 msgr2=0x7f8b940b1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.682+0000 7f8b999ba640 1 --2- 172.21.15.44:0/130856224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b940b4180 0x7f8b940b1d40 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f8b94161e40 tx=0x7f8b8402be40 comp rx=0 tx=0).stop 2024-04-02T13:32:11.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.682+0000 7f8b999ba640 1 -- 172.21.15.44:0/130856224 shutdown_connections 2024-04-02T13:32:11.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.682+0000 7f8b999ba640 1 --2- 172.21.15.44:0/130856224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b940b4180 0x7f8b940b1d40 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.682+0000 7f8b999ba640 1 --2- 172.21.15.44:0/130856224 >> v2:172.21.15.69:3300/0 conn(0x7f8b9415e5d0 0x7f8b940d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.682+0000 7f8b999ba640 1 -- 172.21.15.44:0/130856224 >> 172.21.15.44:0/130856224 conn(0x7f8b9409aa20 msgr2=0x7f8b9409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:11.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.682+0000 7f8b999ba640 1 -- 172.21.15.44:0/130856224 shutdown_connections 2024-04-02T13:32:11.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.682+0000 7f8b999ba640 1 -- 172.21.15.44:0/130856224 wait complete. 2024-04-02T13:32:11.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.683+0000 7f8b999ba640 1 Processor -- start 2024-04-02T13:32:11.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.683+0000 7f8b999ba640 1 -- start start 2024-04-02T13:32:11.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b999ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b9406cff0 0x7f8b9414f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b999ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b940b4180 0x7f8b940c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b999ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b9415e5d0 0x7f8b9415fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b999ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8b940d13f0 con 0x7f8b940b4180 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b999ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8b940d1270 con 0x7f8b9406cff0 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b999ba640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8b940b2280 con 0x7f8b9415e5d0 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b927fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b940b4180 0x7f8b940c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b927fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b940b4180 0x7f8b940c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39034/0 (socket says 172.21.15.44:39034) 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b927fc640 1 -- 172.21.15.44:0/2632335677 learned_addr learned my addr 172.21.15.44:0/2632335677 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b92ffd640 1 --2- 172.21.15.44:0/2632335677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b9406cff0 0x7f8b9414f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b927fc640 1 -- 172.21.15.44:0/2632335677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b9415e5d0 msgr2=0x7f8b9415fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b927fc640 1 --2- 172.21.15.44:0/2632335677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b9415e5d0 0x7f8b9415fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b927fc640 1 -- 172.21.15.44:0/2632335677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b9406cff0 msgr2=0x7f8b9414f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b927fc640 1 --2- 172.21.15.44:0/2632335677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b9406cff0 0x7f8b9414f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b927fc640 1 -- 172.21.15.44:0/2632335677 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b940c6ae0 con 0x7f8b940b4180 2024-04-02T13:32:11.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b927fc640 1 --2- 172.21.15.44:0/2632335677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b940b4180 0x7f8b940c7b80 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00c7c0 tx=0x7f8b7c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:11.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.684+0000 7f8b8bfff640 1 -- 172.21.15.44:0/2632335677 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c015070 con 0x7f8b940b4180 2024-04-02T13:32:11.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.685+0000 7f8b8bfff640 1 -- 172.21.15.44:0/2632335677 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b7c007820 con 0x7f8b940b4180 2024-04-02T13:32:11.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.685+0000 7f8b999ba640 1 -- 172.21.15.44:0/2632335677 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b940c6dc0 con 0x7f8b940b4180 2024-04-02T13:32:11.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.685+0000 7f8b8bfff640 1 -- 172.21.15.44:0/2632335677 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c007bf0 con 0x7f8b940b4180 2024-04-02T13:32:11.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.685+0000 7f8b999ba640 1 -- 172.21.15.44:0/2632335677 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b940c7380 con 0x7f8b940b4180 2024-04-02T13:32:11.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.685+0000 7f8b999ba640 1 -- 172.21.15.44:0/2632335677 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b940684f0 con 0x7f8b940b4180 2024-04-02T13:32:11.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.686+0000 7f8b8bfff640 1 -- 172.21.15.44:0/2632335677 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8b7c018070 con 0x7f8b940b4180 2024-04-02T13:32:11.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.686+0000 7f8b8bfff640 1 --2- 172.21.15.44:0/2632335677 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8b7406c8f0 0x7f8b7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.686+0000 7f8b8bfff640 1 -- 172.21.15.44:0/2632335677 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7f8b7c08bbe0 con 0x7f8b940b4180 2024-04-02T13:32:11.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.687+0000 7f8b92ffd640 1 --2- 172.21.15.44:0/2632335677 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8b7406c8f0 0x7f8b7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.687+0000 7f8b92ffd640 1 --2- 172.21.15.44:0/2632335677 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8b7406c8f0 0x7f8b7406edb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c007fd0 tx=0x7f8b8c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:11.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.691+0000 7f8b8bfff640 1 -- 172.21.15.44:0/2632335677 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f8b7c059e60 con 0x7f8b940b4180 2024-04-02T13:32:11.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.873+0000 7f8b999ba640 1 -- 172.21.15.44:0/2632335677 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8b9406cdd0 con 0x7f8b940b4180 2024-04-02T13:32:11.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.873+0000 7f8b8bfff640 1 -- 172.21.15.44:0/2632335677 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v123) ==== 93+0+10 (secure 0 0 0) 0x7f8b7c059760 con 0x7f8b940b4180 2024-04-02T13:32:11.876 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-02T13:32:11.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.875+0000 7f8b999ba640 1 -- 172.21.15.44:0/2632335677 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8b7406c8f0 msgr2=0x7f8b7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.875+0000 7f8b999ba640 1 --2- 172.21.15.44:0/2632335677 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8b7406c8f0 0x7f8b7406edb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c007fd0 tx=0x7f8b8c009040 comp rx=0 tx=0).stop 2024-04-02T13:32:11.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.875+0000 7f8b999ba640 1 -- 172.21.15.44:0/2632335677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b940b4180 msgr2=0x7f8b940c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.875+0000 7f8b999ba640 1 --2- 172.21.15.44:0/2632335677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b940b4180 0x7f8b940c7b80 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00c7c0 tx=0x7f8b7c00cc80 comp rx=0 tx=0).stop 2024-04-02T13:32:11.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.875+0000 7f8b999ba640 1 -- 172.21.15.44:0/2632335677 shutdown_connections 2024-04-02T13:32:11.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.875+0000 7f8b999ba640 1 --2- 172.21.15.44:0/2632335677 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f8b7406c8f0 0x7f8b7406edb0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.875+0000 7f8b999ba640 1 --2- 172.21.15.44:0/2632335677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b9415e5d0 0x7f8b9415fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.875+0000 7f8b999ba640 1 --2- 172.21.15.44:0/2632335677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b940b4180 0x7f8b940c7b80 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.875+0000 7f8b999ba640 1 --2- 172.21.15.44:0/2632335677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b9406cff0 0x7f8b9414f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.875+0000 7f8b999ba640 1 -- 172.21.15.44:0/2632335677 >> 172.21.15.44:0/2632335677 conn(0x7f8b9409aa20 msgr2=0x7f8b940b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:11.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.875+0000 7f8b999ba640 1 -- 172.21.15.44:0/2632335677 shutdown_connections 2024-04-02T13:32:11.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.875+0000 7f8b999ba640 1 -- 172.21.15.44:0/2632335677 wait complete. 2024-04-02T13:32:11.889 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T13:32:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.991+0000 7f50b6d8f640 1 Processor -- start 2024-04-02T13:32:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.991+0000 7f50b6d8f640 1 -- start start 2024-04-02T13:32:11.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.992+0000 7f50b6d8f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f50b00d07b0 0x7f50b00ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.992+0000 7f50b6d8f640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f50b00cf400 0x7f50b015c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:11.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.992+0000 7f50b6d8f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50b0160b10 con 0x7f50b01565d0 2024-04-02T13:32:11.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.992+0000 7f50b6d8f640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f50b00cfad0 con 0x7f50b00cf400 2024-04-02T13:32:11.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.992+0000 7f50b6d8f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f50b00cf870 con 0x7f50b00d07b0 2024-04-02T13:32:11.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.992+0000 7f50b532c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f50b00cf400 0x7f50b015c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.992+0000 7f50b532c640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f50b00cf400 0x7f50b015c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39046/0 (socket says 172.21.15.44:39046) 2024-04-02T13:32:11.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.992+0000 7f50b532c640 1 -- 172.21.15.44:0/209125314 learned_addr learned my addr 172.21.15.44:0/209125314 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:11.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.992+0000 7f50affff640 1 --2- 172.21.15.44:0/209125314 >> v2:172.21.15.69:3300/0 conn(0x7f50b00d07b0 0x7f50b00ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:11.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.992+0000 7f50b532c640 1 -- 172.21.15.44:0/209125314 >> v2:172.21.15.69:3300/0 conn(0x7f50b00d07b0 msgr2=0x7f50b00ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:11.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.992+0000 7f50b532c640 1 --2- 172.21.15.44:0/209125314 >> v2:172.21.15.69:3300/0 conn(0x7f50b00d07b0 0x7f50b00ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.992+0000 7f50b532c640 1 -- 172.21.15.44:0/209125314 >> v1:172.21.15.69:6789/0 conn(0x7f50b01565d0 legacy=0x7f50b00d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:11.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.992+0000 7f50b532c640 1 -- 172.21.15.44:0/209125314 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b015fd50 con 0x7f50b00cf400 2024-04-02T13:32:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.993+0000 7f50b532c640 1 --2- 172.21.15.44:0/209125314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50b00cf400 0x7f50b015c7a0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f50b00b2a30 tx=0x7f509802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11233da27cba4bb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.993+0000 7f50af7fe640 1 -- 172.21.15.44:0/209125314 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509802e070 con 0x7f50b00cf400 2024-04-02T13:32:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.993+0000 7f50af7fe640 1 -- 172.21.15.44:0/209125314 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5098037070 con 0x7f50b00cf400 2024-04-02T13:32:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.993+0000 7f50af7fe640 1 -- 172.21.15.44:0/209125314 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5098031a00 con 0x7f50b00cf400 2024-04-02T13:32:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.994+0000 7f50b6d8f640 1 -- 172.21.15.44:0/209125314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50b00cf400 msgr2=0x7f50b015c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.994+0000 7f50b6d8f640 1 --2- 172.21.15.44:0/209125314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50b00cf400 0x7f50b015c7a0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f50b00b2a30 tx=0x7f509802b880 comp rx=0 tx=0).stop 2024-04-02T13:32:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.994+0000 7f50b6d8f640 1 -- 172.21.15.44:0/209125314 shutdown_connections 2024-04-02T13:32:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.994+0000 7f50b6d8f640 1 --2- 172.21.15.44:0/209125314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50b00cf400 0x7f50b015c7a0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.994+0000 7f50b6d8f640 1 --2- 172.21.15.44:0/209125314 >> v2:172.21.15.69:3300/0 conn(0x7f50b00d07b0 0x7f50b00ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.994+0000 7f50b6d8f640 1 -- 172.21.15.44:0/209125314 >> 172.21.15.44:0/209125314 conn(0x7f50b006b920 msgr2=0x7f50b006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.994+0000 7f50b6d8f640 1 -- 172.21.15.44:0/209125314 shutdown_connections 2024-04-02T13:32:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.994+0000 7f50b6d8f640 1 -- 172.21.15.44:0/209125314 wait complete. 2024-04-02T13:32:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.995+0000 7f50b6d8f640 1 Processor -- start 2024-04-02T13:32:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.995+0000 7f50b6d8f640 1 -- start start 2024-04-02T13:32:12.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b6d8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50b00cf400 0x7f50b00b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b6d8f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50b00d07b0 0x7f50b00b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b6d8f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50b01565d0 0x7f50b01f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b6d8f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f50b01605f0 con 0x7f50b00cf400 2024-04-02T13:32:12.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b6d8f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f50b0160310 con 0x7f50b00d07b0 2024-04-02T13:32:12.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b6d8f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f50b01569b0 con 0x7f50b01565d0 2024-04-02T13:32:12.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b4b2b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50b00cf400 0x7f50b00b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b4b2b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50b00cf400 0x7f50b00b3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39062/0 (socket says 172.21.15.44:39062) 2024-04-02T13:32:12.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b4b2b640 1 -- 172.21.15.44:0/3911048503 learned_addr learned my addr 172.21.15.44:0/3911048503 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50affff640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50b00d07b0 0x7f50b00b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b532c640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50b01565d0 0x7f50b01f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b4b2b640 1 -- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50b01565d0 msgr2=0x7f50b01f79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b4b2b640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50b01565d0 0x7f50b01f79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b4b2b640 1 -- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50b00d07b0 msgr2=0x7f50b00b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.996+0000 7f50b4b2b640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50b00d07b0 0x7f50b00b44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.997+0000 7f50b4b2b640 1 -- 172.21.15.44:0/3911048503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b014fd90 con 0x7f50b00cf400 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.997+0000 7f50b4b2b640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50b00cf400 0x7f50b00b3fa0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f50a800c430 tx=0x7f50a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.997+0000 7f50ad7fa640 1 -- 172.21.15.44:0/3911048503 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a8011790 con 0x7f50b00cf400 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.997+0000 7f50ad7fa640 1 -- 172.21.15.44:0/3911048503 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50a8017070 con 0x7f50b00cf400 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.997+0000 7f50b6d8f640 1 -- 172.21.15.44:0/3911048503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50b00ca910 con 0x7f50b00cf400 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.997+0000 7f50ad7fa640 1 -- 172.21.15.44:0/3911048503 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a8011b70 con 0x7f50b00cf400 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.997+0000 7f50b6d8f640 1 -- 172.21.15.44:0/3911048503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50b00c9b50 con 0x7f50b00cf400 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.998+0000 7f50b6d8f640 1 -- 172.21.15.44:0/3911048503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50b00684f0 con 0x7f50b00cf400 2024-04-02T13:32:12.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.998+0000 7f50ad7fa640 1 -- 172.21.15.44:0/3911048503 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f50a8011d10 con 0x7f50b00cf400 2024-04-02T13:32:12.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.999+0000 7f50ad7fa640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f509006c8f0 0x7f509006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.999+0000 7f50ad7fa640 1 -- 172.21.15.44:0/3911048503 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7f50a808bbe0 con 0x7f50b00cf400 2024-04-02T13:32:12.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.999+0000 7f50affff640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f509006c8f0 0x7f509006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:11.999+0000 7f50affff640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f509006c8f0 0x7f509006edb0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f50a00070b0 tx=0x7f50a0007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:12.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.003+0000 7f50ad7fa640 1 -- 172.21.15.44:0/3911048503 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f50b014fd90 con 0x7f50b00cf400 2024-04-02T13:32:12.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.186+0000 7f50b6d8f640 1 -- 172.21.15.44:0/3911048503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f50b015a4e0 con 0x7f50b00cf400 2024-04-02T13:32:12.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.187+0000 7f50ad7fa640 1 -- 172.21.15.44:0/3911048503 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v123) ==== 104+0+11 (secure 0 0 0) 0x7f50a8059dd0 con 0x7f50b00cf400 2024-04-02T13:32:12.189 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-02T13:32:12.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.189+0000 7f50b6d8f640 1 -- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f509006c8f0 msgr2=0x7f509006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.189+0000 7f50b6d8f640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f509006c8f0 0x7f509006edb0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f50a00070b0 tx=0x7f50a0007040 comp rx=0 tx=0).stop 2024-04-02T13:32:12.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.189+0000 7f50b6d8f640 1 -- 172.21.15.44:0/3911048503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50b00cf400 msgr2=0x7f50b00b3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.189+0000 7f50b6d8f640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50b00cf400 0x7f50b00b3fa0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f50a800c430 tx=0x7f50a800c8f0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.189+0000 7f50b6d8f640 1 -- 172.21.15.44:0/3911048503 shutdown_connections 2024-04-02T13:32:12.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.189+0000 7f50b6d8f640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f509006c8f0 0x7f509006edb0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.189+0000 7f50b6d8f640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50b01565d0 0x7f50b01f79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.189+0000 7f50b6d8f640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50b00d07b0 0x7f50b00b44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.189+0000 7f50b6d8f640 1 --2- 172.21.15.44:0/3911048503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50b00cf400 0x7f50b00b3fa0 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.189+0000 7f50b6d8f640 1 -- 172.21.15.44:0/3911048503 >> 172.21.15.44:0/3911048503 conn(0x7f50b006b920 msgr2=0x7f50b00d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:12.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.189+0000 7f50b6d8f640 1 -- 172.21.15.44:0/3911048503 shutdown_connections 2024-04-02T13:32:12.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.189+0000 7f50b6d8f640 1 -- 172.21.15.44:0/3911048503 wait complete. 2024-04-02T13:32:12.203 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T13:32:12.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.304+0000 7f6566fa8640 1 Processor -- start 2024-04-02T13:32:12.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.304+0000 7f6566fa8640 1 -- start start 2024-04-02T13:32:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.305+0000 7f6566fa8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65601554c0 0x7f6560071120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.305+0000 7f6566fa8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f65600ce9a0 0x7f65600d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.305+0000 7f6566fa8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65600cdc20 con 0x7f6560071660 2024-04-02T13:32:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.305+0000 7f6566fa8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f65600cdec0 con 0x7f65600ce9a0 2024-04-02T13:32:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.305+0000 7f6566fa8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f65600ce1a0 con 0x7f65601554c0 2024-04-02T13:32:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.305+0000 7f655ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f65600ce9a0 0x7f65600d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.305+0000 7f6565545640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6560071660 0x7f656015eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:36364/0 (socket says 172.21.15.44:36364) 2024-04-02T13:32:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.305+0000 7f6564d44640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65601554c0 0x7f6560071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.305+0000 7f6565545640 1 -- 172.21.15.44:0/2728834237 learned_addr learned my addr 172.21.15.44:0/2728834237 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.305+0000 7f655ffff640 1 --2- 172.21.15.44:0/2728834237 >> v2:172.21.15.69:3300/0 conn(0x7f65600ce9a0 0x7f65600d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:41490/0 (socket says 172.21.15.44:41490) 2024-04-02T13:32:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.306+0000 7f655ffff640 1 -- 172.21.15.44:0/2728834237 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65601554c0 msgr2=0x7f6560071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.306+0000 7f655ffff640 1 --2- 172.21.15.44:0/2728834237 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65601554c0 0x7f6560071120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.306+0000 7f655ffff640 1 -- 172.21.15.44:0/2728834237 >> v1:172.21.15.69:6789/0 conn(0x7f6560071660 legacy=0x7f656015eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.306+0000 7f655ffff640 1 -- 172.21.15.44:0/2728834237 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65600ce480 con 0x7f65600ce9a0 2024-04-02T13:32:12.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.306+0000 7f655f7fe640 1 -- 172.21.15.44:0/2728834237 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3112831780 0 0) 0x7f65600cdc20 con 0x7f6560071660 2024-04-02T13:32:12.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.307+0000 7f655ffff640 1 --2- 172.21.15.44:0/2728834237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65600ce9a0 0x7f65600d0130 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f65540097a0 tx=0x7f655402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91a3d7b470c0189c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:12.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.307+0000 7f655f7fe640 1 -- 172.21.15.44:0/2728834237 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554035020 con 0x7f65600ce9a0 2024-04-02T13:32:12.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.307+0000 7f655f7fe640 1 -- 172.21.15.44:0/2728834237 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6554031830 con 0x7f65600ce9a0 2024-04-02T13:32:12.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.307+0000 7f655f7fe640 1 -- 172.21.15.44:0/2728834237 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554031c00 con 0x7f65600ce9a0 2024-04-02T13:32:12.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.307+0000 7f6566fa8640 1 -- 172.21.15.44:0/2728834237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65600ce9a0 msgr2=0x7f65600d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.307+0000 7f6566fa8640 1 --2- 172.21.15.44:0/2728834237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65600ce9a0 0x7f65600d0130 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f65540097a0 tx=0x7f655402b8d0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.307+0000 7f6566fa8640 1 -- 172.21.15.44:0/2728834237 shutdown_connections 2024-04-02T13:32:12.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.307+0000 7f6566fa8640 1 --2- 172.21.15.44:0/2728834237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65600ce9a0 0x7f65600d0130 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.307+0000 7f6566fa8640 1 --2- 172.21.15.44:0/2728834237 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65601554c0 0x7f6560071120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.307+0000 7f6566fa8640 1 -- 172.21.15.44:0/2728834237 >> 172.21.15.44:0/2728834237 conn(0x7f656006b920 msgr2=0x7f656006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:12.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.307+0000 7f6566fa8640 1 -- 172.21.15.44:0/2728834237 shutdown_connections 2024-04-02T13:32:12.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.307+0000 7f6566fa8640 1 -- 172.21.15.44:0/2728834237 wait complete. 2024-04-02T13:32:12.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.308+0000 7f6566fa8640 1 Processor -- start 2024-04-02T13:32:12.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.308+0000 7f6566fa8640 1 -- start start 2024-04-02T13:32:12.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.309+0000 7f6566fa8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6560071660 0x7f65601f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.309+0000 7f6566fa8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65600ce9a0 0x7f6560158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.309+0000 7f6566fa8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65601554c0 0x7f65600ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.309+0000 7f6566fa8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f656015c290 con 0x7f65601554c0 2024-04-02T13:32:12.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.309+0000 7f6566fa8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6560071be0 con 0x7f65600ce9a0 2024-04-02T13:32:12.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.309+0000 7f6566fa8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f65601f4670 con 0x7f6560071660 2024-04-02T13:32:12.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.309+0000 7f6565545640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65601554c0 0x7f65600ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.309+0000 7f6565545640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65601554c0 0x7f65600ae790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39070/0 (socket says 172.21.15.44:39070) 2024-04-02T13:32:12.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.309+0000 7f6565545640 1 -- 172.21.15.44:0/663936330 learned_addr learned my addr 172.21.15.44:0/663936330 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:12.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f655ffff640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65600ce9a0 0x7f6560158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f6564d44640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6560071660 0x7f65601f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f6565545640 1 -- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6560071660 msgr2=0x7f65601f4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f6565545640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6560071660 0x7f65601f4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f6565545640 1 -- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65600ce9a0 msgr2=0x7f6560158190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f6565545640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65600ce9a0 0x7f6560158190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f6565545640 1 -- 172.21.15.44:0/663936330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65601f4950 con 0x7f65601554c0 2024-04-02T13:32:12.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f655ffff640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65600ce9a0 0x7f6560158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:32:12.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f6565545640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65601554c0 0x7f65600ae790 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f655000c450 tx=0x7f655000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:12.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f655d7fa640 1 -- 172.21.15.44:0/663936330 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6550018070 con 0x7f65601554c0 2024-04-02T13:32:12.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f655d7fa640 1 -- 172.21.15.44:0/663936330 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6550007500 con 0x7f65601554c0 2024-04-02T13:32:12.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f6566fa8640 1 -- 172.21.15.44:0/663936330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65601f4c30 con 0x7f65601554c0 2024-04-02T13:32:12.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f655d7fa640 1 -- 172.21.15.44:0/663936330 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6550004a20 con 0x7f65601554c0 2024-04-02T13:32:12.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.310+0000 7f6566fa8640 1 -- 172.21.15.44:0/663936330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65601586d0 con 0x7f65601554c0 2024-04-02T13:32:12.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.311+0000 7f6566fa8640 1 -- 172.21.15.44:0/663936330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65600684f0 con 0x7f65601554c0 2024-04-02T13:32:12.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.312+0000 7f655d7fa640 1 -- 172.21.15.44:0/663936330 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6550004bc0 con 0x7f65601554c0 2024-04-02T13:32:12.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.312+0000 7f655d7fa640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f654c06c8f0 0x7f654c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.312+0000 7f655d7fa640 1 -- 172.21.15.44:0/663936330 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7f655008afb0 con 0x7f65601554c0 2024-04-02T13:32:12.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.312+0000 7f6564d44640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f654c06c8f0 0x7f654c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.313+0000 7f6564d44640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f654c06c8f0 0x7f654c06edb0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f6558011340 tx=0x7f655800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:12.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.317+0000 7f655d7fa640 1 -- 172.21.15.44:0/663936330 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6550058ad0 con 0x7f65601554c0 2024-04-02T13:32:12.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.502+0000 7f6566fa8640 1 -- 172.21.15.44:0/663936330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f656006d5e0 con 0x7f65601554c0 2024-04-02T13:32:12.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.502+0000 7f655d7fa640 1 -- 172.21.15.44:0/663936330 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v123) ==== 100+0+11 (secure 0 0 0) 0x7f65500780c0 con 0x7f65601554c0 2024-04-02T13:32:12.505 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-02T13:32:12.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.504+0000 7f6566fa8640 1 -- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f654c06c8f0 msgr2=0x7f654c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.504+0000 7f6566fa8640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f654c06c8f0 0x7f654c06edb0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f6558011340 tx=0x7f655800b040 comp rx=0 tx=0).stop 2024-04-02T13:32:12.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.504+0000 7f6566fa8640 1 -- 172.21.15.44:0/663936330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65601554c0 msgr2=0x7f65600ae790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.504+0000 7f6566fa8640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65601554c0 0x7f65600ae790 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f655000c450 tx=0x7f655000c910 comp rx=0 tx=0).stop 2024-04-02T13:32:12.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.504+0000 7f6566fa8640 1 -- 172.21.15.44:0/663936330 shutdown_connections 2024-04-02T13:32:12.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.504+0000 7f6566fa8640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f654c06c8f0 0x7f654c06edb0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.504+0000 7f6566fa8640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65601554c0 0x7f65600ae790 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.504+0000 7f6566fa8640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65600ce9a0 0x7f6560158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.504+0000 7f6566fa8640 1 --2- 172.21.15.44:0/663936330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6560071660 0x7f65601f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.505+0000 7f6566fa8640 1 -- 172.21.15.44:0/663936330 >> 172.21.15.44:0/663936330 conn(0x7f656006b920 msgr2=0x7f656006bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:12.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.505+0000 7f6566fa8640 1 -- 172.21.15.44:0/663936330 shutdown_connections 2024-04-02T13:32:12.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.505+0000 7f6566fa8640 1 -- 172.21.15.44:0/663936330 wait complete. 2024-04-02T13:32:12.518 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:32:12.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.616+0000 7f56f9277640 1 Processor -- start 2024-04-02T13:32:12.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.616+0000 7f56f9277640 1 -- start start 2024-04-02T13:32:12.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.617+0000 7f56f9277640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f56f415e520 0x7f56f40d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.617+0000 7f56f9277640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f40d0f80 0x7f56f40b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.617+0000 7f56f9277640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f40b70b0 con 0x7f56f415e140 2024-04-02T13:32:12.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.617+0000 7f56f9277640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f56f40b9f50 con 0x7f56f415e520 2024-04-02T13:32:12.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.617+0000 7f56f9277640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f56f40b7390 con 0x7f56f40d0f80 2024-04-02T13:32:12.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.617+0000 7f56f27fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f56f415e520 0x7f56f40d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.617+0000 7f56f27fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f56f415e520 0x7f56f40d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39082/0 (socket says 172.21.15.44:39082) 2024-04-02T13:32:12.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.617+0000 7f56f27fc640 1 -- 172.21.15.44:0/4221093750 learned_addr learned my addr 172.21.15.44:0/4221093750 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:12.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.617+0000 7f56f37fe640 1 --2- 172.21.15.44:0/4221093750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f40d0f80 0x7f56f40b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.617+0000 7f56f27fc640 1 -- 172.21.15.44:0/4221093750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f40d0f80 msgr2=0x7f56f40b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:12.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.617+0000 7f56f27fc640 1 --2- 172.21.15.44:0/4221093750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f40d0f80 0x7f56f40b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.617+0000 7f56f27fc640 1 -- 172.21.15.44:0/4221093750 >> v1:172.21.15.69:6789/0 conn(0x7f56f415e140 legacy=0x7f56f40d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.617+0000 7f56f27fc640 1 -- 172.21.15.44:0/4221093750 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f40b7670 con 0x7f56f415e520 2024-04-02T13:32:12.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.618+0000 7f56f27fc640 1 --2- 172.21.15.44:0/4221093750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f415e520 0x7f56f40d0a40 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009870 tx=0x7f56dc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f58383034417153 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:12.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.618+0000 7f56f1ffb640 1 -- 172.21.15.44:0/4221093750 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56dc034070 con 0x7f56f415e520 2024-04-02T13:32:12.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.618+0000 7f56f1ffb640 1 -- 172.21.15.44:0/4221093750 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56dc030840 con 0x7f56f415e520 2024-04-02T13:32:12.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.618+0000 7f56f1ffb640 1 -- 172.21.15.44:0/4221093750 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56dc030c10 con 0x7f56f415e520 2024-04-02T13:32:12.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.619+0000 7f56f9277640 1 -- 172.21.15.44:0/4221093750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f415e520 msgr2=0x7f56f40d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.619+0000 7f56f9277640 1 --2- 172.21.15.44:0/4221093750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f415e520 0x7f56f40d0a40 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009870 tx=0x7f56dc02b980 comp rx=0 tx=0).stop 2024-04-02T13:32:12.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.619+0000 7f56f9277640 1 -- 172.21.15.44:0/4221093750 shutdown_connections 2024-04-02T13:32:12.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.619+0000 7f56f9277640 1 --2- 172.21.15.44:0/4221093750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f415e520 0x7f56f40d0a40 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.619+0000 7f56f9277640 1 --2- 172.21.15.44:0/4221093750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f40d0f80 0x7f56f40b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.619+0000 7f56f9277640 1 -- 172.21.15.44:0/4221093750 >> 172.21.15.44:0/4221093750 conn(0x7f56f409d390 msgr2=0x7f56f40d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:12.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.619+0000 7f56f9277640 1 -- 172.21.15.44:0/4221093750 shutdown_connections 2024-04-02T13:32:12.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.619+0000 7f56f9277640 1 -- 172.21.15.44:0/4221093750 wait complete. 2024-04-02T13:32:12.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.620+0000 7f56f9277640 1 Processor -- start 2024-04-02T13:32:12.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.620+0000 7f56f9277640 1 -- start start 2024-04-02T13:32:12.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f9277640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f40d0f80 0x7f56f41585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f9277640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f415e140 0x7f56f4154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f9277640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f415e520 0x7f56f40cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f9277640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f56f40d2d40 con 0x7f56f415e140 2024-04-02T13:32:12.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f9277640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f56f40d4ee0 con 0x7f56f415e520 2024-04-02T13:32:12.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f9277640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f56f40b5200 con 0x7f56f40d0f80 2024-04-02T13:32:12.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f27fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f415e140 0x7f56f4154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f27fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f415e140 0x7f56f4154110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39096/0 (socket says 172.21.15.44:39096) 2024-04-02T13:32:12.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f27fc640 1 -- 172.21.15.44:0/1969086109 learned_addr learned my addr 172.21.15.44:0/1969086109 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:12.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f37fe640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f415e520 0x7f56f40cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f2ffd640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f40d0f80 0x7f56f41585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f27fc640 1 -- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f40d0f80 msgr2=0x7f56f41585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f27fc640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f40d0f80 0x7f56f41585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f27fc640 1 -- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f415e520 msgr2=0x7f56f40cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f27fc640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f415e520 0x7f56f40cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.621+0000 7f56f27fc640 1 -- 172.21.15.44:0/1969086109 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f40cd080 con 0x7f56f415e140 2024-04-02T13:32:12.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.622+0000 7f56f27fc640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f415e140 0x7f56f4154110 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f56dc02b9c0 tx=0x7f56dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:12.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.622+0000 7f56eb7fe640 1 -- 172.21.15.44:0/1969086109 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56dc034040 con 0x7f56f415e140 2024-04-02T13:32:12.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.622+0000 7f56eb7fe640 1 -- 172.21.15.44:0/1969086109 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56dc039d90 con 0x7f56f415e140 2024-04-02T13:32:12.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.622+0000 7f56f9277640 1 -- 172.21.15.44:0/1969086109 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f40c7dd0 con 0x7f56f415e140 2024-04-02T13:32:12.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.622+0000 7f56eb7fe640 1 -- 172.21.15.44:0/1969086109 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56dc043610 con 0x7f56f415e140 2024-04-02T13:32:12.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.622+0000 7f56f9277640 1 -- 172.21.15.44:0/1969086109 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f40ccc70 con 0x7f56f415e140 2024-04-02T13:32:12.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.623+0000 7f56f9277640 1 -- 172.21.15.44:0/1969086109 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f40684f0 con 0x7f56f415e140 2024-04-02T13:32:12.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.623+0000 7f56eb7fe640 1 -- 172.21.15.44:0/1969086109 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f56dc04a060 con 0x7f56f415e140 2024-04-02T13:32:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.624+0000 7f56eb7fe640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56d406c8f0 0x7f56d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.624+0000 7f56eb7fe640 1 -- 172.21.15.44:0/1969086109 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7f56dc0adbb0 con 0x7f56f415e140 2024-04-02T13:32:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.624+0000 7f56f2ffd640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56d406c8f0 0x7f56d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.624+0000 7f56f2ffd640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56d406c8f0 0x7f56d406edb0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f56f40caeb0 tx=0x7f56ec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:12.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.628+0000 7f56eb7fe640 1 -- 172.21.15.44:0/1969086109 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f56dc03e070 con 0x7f56f415e140 2024-04-02T13:32:12.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.851+0000 7f56f9277640 1 -- 172.21.15.44:0/1969086109 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56f40c74d0 con 0x7f56f415e140 2024-04-02T13:32:12.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.851+0000 7f56eb7fe640 1 -- 172.21.15.44:0/1969086109 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5184 (secure 0 0 0) 0x7f56dc07b7e0 con 0x7f56f415e140 2024-04-02T13:32:12.854 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:12.855 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:31:11.495701+0000","modified":"2024-04-02T13:32:04.851323+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6835/16875964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":16875964},{"type":"v1","addr":"172.21.15.44:6835","nonce":16875964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":8}]} 2024-04-02T13:32:12.855 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 112 2024-04-02T13:32:12.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.853+0000 7f56f9277640 1 -- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56d406c8f0 msgr2=0x7f56d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.853+0000 7f56f9277640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56d406c8f0 0x7f56d406edb0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f56f40caeb0 tx=0x7f56ec009040 comp rx=0 tx=0).stop 2024-04-02T13:32:12.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.853+0000 7f56f9277640 1 -- 172.21.15.44:0/1969086109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f415e140 msgr2=0x7f56f4154110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.853+0000 7f56f9277640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f415e140 0x7f56f4154110 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f56dc02b9c0 tx=0x7f56dc0046c0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.853+0000 7f56f9277640 1 -- 172.21.15.44:0/1969086109 shutdown_connections 2024-04-02T13:32:12.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.853+0000 7f56f9277640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f56d406c8f0 0x7f56d406edb0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.853+0000 7f56f9277640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f415e520 0x7f56f40cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.853+0000 7f56f9277640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f415e140 0x7f56f4154110 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.853+0000 7f56f9277640 1 --2- 172.21.15.44:0/1969086109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f40d0f80 0x7f56f41585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.853+0000 7f56f9277640 1 -- 172.21.15.44:0/1969086109 >> 172.21.15.44:0/1969086109 conn(0x7f56f409d390 msgr2=0x7f56f406cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:12.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.853+0000 7f56f9277640 1 -- 172.21.15.44:0/1969086109 shutdown_connections 2024-04-02T13:32:12.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.853+0000 7f56f9277640 1 -- 172.21.15.44:0/1969086109 wait complete. 2024-04-02T13:32:12.867 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:32:12.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.965+0000 7fe16fd15640 1 Processor -- start 2024-04-02T13:32:12.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.966+0000 7fe16fd15640 1 -- start start 2024-04-02T13:32:12.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.966+0000 7fe16fd15640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe16815e2a0 0x7fe1680cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.966+0000 7fe16fd15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680cc870 0x7fe1680cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.966+0000 7fe16fd15640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1680715c0 con 0x7fe1680cc0e0 2024-04-02T13:32:12.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.966+0000 7fe16fd15640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe168071ef0 con 0x7fe16815e2a0 2024-04-02T13:32:12.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.966+0000 7fe16fd15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe1680718a0 con 0x7fe1680cc870 2024-04-02T13:32:12.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.966+0000 7fe16dab1640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe16815e2a0 0x7fe1680cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.966+0000 7fe16d2b0640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe1680cc0e0 0x7fe16806df60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:60964/0 (socket says 172.21.15.44:60964) 2024-04-02T13:32:12.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.966+0000 7fe16d2b0640 1 -- 172.21.15.44:0/749719851 learned_addr learned my addr 172.21.15.44:0/749719851 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:12.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.967+0000 7fe16e2b2640 1 --2- 172.21.15.44:0/749719851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680cc870 0x7fe1680cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.967+0000 7fe16dab1640 1 -- 172.21.15.44:0/749719851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680cc870 msgr2=0x7fe1680cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:12.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.967+0000 7fe16dab1640 1 --2- 172.21.15.44:0/749719851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680cc870 0x7fe1680cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.967+0000 7fe16dab1640 1 -- 172.21.15.44:0/749719851 >> v1:172.21.15.44:6789/0 conn(0x7fe1680cc0e0 legacy=0x7fe16806df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.967+0000 7fe16dab1640 1 -- 172.21.15.44:0/749719851 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe168071b80 con 0x7fe16815e2a0 2024-04-02T13:32:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.967+0000 7fe16dab1640 1 --2- 172.21.15.44:0/749719851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe16815e2a0 0x7fe1680cdcb0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fe1600097a0 tx=0x7fe16002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2b7b3d88a2580c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.968+0000 7fe16caaf640 1 -- 172.21.15.44:0/749719851 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe160035020 con 0x7fe16815e2a0 2024-04-02T13:32:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.968+0000 7fe16caaf640 1 -- 172.21.15.44:0/749719851 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe160031830 con 0x7fe16815e2a0 2024-04-02T13:32:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.968+0000 7fe16caaf640 1 -- 172.21.15.44:0/749719851 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe160031c00 con 0x7fe16815e2a0 2024-04-02T13:32:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.968+0000 7fe16fd15640 1 -- 172.21.15.44:0/749719851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe16815e2a0 msgr2=0x7fe1680cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.968+0000 7fe16fd15640 1 --2- 172.21.15.44:0/749719851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe16815e2a0 0x7fe1680cdcb0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fe1600097a0 tx=0x7fe16002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.968+0000 7fe16fd15640 1 -- 172.21.15.44:0/749719851 shutdown_connections 2024-04-02T13:32:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.968+0000 7fe16fd15640 1 --2- 172.21.15.44:0/749719851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680cc870 0x7fe1680cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.968+0000 7fe16fd15640 1 --2- 172.21.15.44:0/749719851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe16815e2a0 0x7fe1680cdcb0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.968+0000 7fe16fd15640 1 -- 172.21.15.44:0/749719851 >> 172.21.15.44:0/749719851 conn(0x7fe1680b1c30 msgr2=0x7fe1680b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.968+0000 7fe16fd15640 1 -- 172.21.15.44:0/749719851 shutdown_connections 2024-04-02T13:32:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.968+0000 7fe16fd15640 1 -- 172.21.15.44:0/749719851 wait complete. 2024-04-02T13:32:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.969+0000 7fe16fd15640 1 Processor -- start 2024-04-02T13:32:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.969+0000 7fe16fd15640 1 -- start start 2024-04-02T13:32:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.970+0000 7fe16fd15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680cc0e0 0x7fe16815a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.970+0000 7fe16fd15640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1680cc870 0x7fe1680c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.970+0000 7fe16fd15640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe16815e2a0 0x7fe1680cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.970+0000 7fe16fd15640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe16815e680 con 0x7fe16815e2a0 2024-04-02T13:32:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.970+0000 7fe16fd15640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe1680cd200 con 0x7fe1680cc870 2024-04-02T13:32:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.970+0000 7fe16fd15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe16815ae10 con 0x7fe1680cc0e0 2024-04-02T13:32:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.970+0000 7fe16e2b2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe16815e2a0 0x7fe1680cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.970+0000 7fe16e2b2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe16815e2a0 0x7fe1680cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39112/0 (socket says 172.21.15.44:39112) 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.970+0000 7fe16e2b2640 1 -- 172.21.15.44:0/1750761785 learned_addr learned my addr 172.21.15.44:0/1750761785 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.970+0000 7fe16d2b0640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1680cc870 0x7fe1680c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.971+0000 7fe16dab1640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680cc0e0 0x7fe16815a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.971+0000 7fe16e2b2640 1 -- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680cc0e0 msgr2=0x7fe16815a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.971+0000 7fe16e2b2640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680cc0e0 0x7fe16815a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.971+0000 7fe16e2b2640 1 -- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1680cc870 msgr2=0x7fe1680c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.971+0000 7fe16e2b2640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1680cc870 0x7fe1680c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.971+0000 7fe16e2b2640 1 -- 172.21.15.44:0/1750761785 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1680c9760 con 0x7fe16815e2a0 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.971+0000 7fe16e2b2640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe16815e2a0 0x7fe1680cfd30 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7fe158007e70 tx=0x7fe15800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.971+0000 7fe1667fc640 1 -- 172.21.15.44:0/1750761785 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe158014420 con 0x7fe16815e2a0 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.971+0000 7fe1667fc640 1 -- 172.21.15.44:0/1750761785 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe158019070 con 0x7fe16815e2a0 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.971+0000 7fe16fd15640 1 -- 172.21.15.44:0/1750761785 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1680cea30 con 0x7fe16815e2a0 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.971+0000 7fe1667fc640 1 -- 172.21.15.44:0/1750761785 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe158014820 con 0x7fe16815e2a0 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.971+0000 7fe16fd15640 1 -- 172.21.15.44:0/1750761785 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1680ceff0 con 0x7fe16815e2a0 2024-04-02T13:32:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.972+0000 7fe16fd15640 1 -- 172.21.15.44:0/1750761785 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1680684f0 con 0x7fe16815e2a0 2024-04-02T13:32:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.972+0000 7fe1667fc640 1 -- 172.21.15.44:0/1750761785 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe1580149c0 con 0x7fe16815e2a0 2024-04-02T13:32:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.973+0000 7fe1667fc640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe15406c8f0 0x7fe15406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.973+0000 7fe1667fc640 1 -- 172.21.15.44:0/1750761785 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7fe15808c5d0 con 0x7fe16815e2a0 2024-04-02T13:32:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.973+0000 7fe16dab1640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe15406c8f0 0x7fe15406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.974+0000 7fe16dab1640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe15406c8f0 0x7fe15406edb0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe160036000 tx=0x7fe16003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:12.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:12.977+0000 7fe1667fc640 1 -- 172.21.15.44:0/1750761785 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe15805a850 con 0x7fe16815e2a0 2024-04-02T13:32:13.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.163+0000 7fe16fd15640 1 -- 172.21.15.44:0/1750761785 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe16806ead0 con 0x7fe16815e2a0 2024-04-02T13:32:13.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.164+0000 7fe1667fc640 1 -- 172.21.15.44:0/1750761785 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) ==== 74+0+17163 (secure 0 0 0) 0x7fe15805a150 con 0x7fe16815e2a0 2024-04-02T13:32:13.167 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:13.167 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":123,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:59.833349+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:31:09.752845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-02T13:31:10.437454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6835/2895101355":"2024-04-03T13:31:59.833326+0000","172.21.15.44:6833/2895101355":"2024-04-03T13:31:59.833326+0000","172.21.15.44:6835/2109578276":"2024-04-03T13:31:41.102885+0000","172.21.15.44:6833/2109578276":"2024-04-03T13:31:41.102885+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:32:13.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.166+0000 7fe16fd15640 1 -- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe15406c8f0 msgr2=0x7fe15406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.166+0000 7fe16fd15640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe15406c8f0 0x7fe15406edb0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe160036000 tx=0x7fe16003c040 comp rx=0 tx=0).stop 2024-04-02T13:32:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.166+0000 7fe16fd15640 1 -- 172.21.15.44:0/1750761785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe16815e2a0 msgr2=0x7fe1680cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.166+0000 7fe16fd15640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe16815e2a0 0x7fe1680cfd30 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7fe158007e70 tx=0x7fe15800e4b0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.166+0000 7fe16fd15640 1 -- 172.21.15.44:0/1750761785 shutdown_connections 2024-04-02T13:32:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.166+0000 7fe16fd15640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe15406c8f0 0x7fe15406edb0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.166+0000 7fe16fd15640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe16815e2a0 0x7fe1680cfd30 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.166+0000 7fe16fd15640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1680cc870 0x7fe1680c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.166+0000 7fe16fd15640 1 --2- 172.21.15.44:0/1750761785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680cc0e0 0x7fe16815a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.166+0000 7fe16fd15640 1 -- 172.21.15.44:0/1750761785 >> 172.21.15.44:0/1750761785 conn(0x7fe1680b1c30 msgr2=0x7fe168072380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.166+0000 7fe16fd15640 1 -- 172.21.15.44:0/1750761785 shutdown_connections 2024-04-02T13:32:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.166+0000 7fe16fd15640 1 -- 172.21.15.44:0/1750761785 wait complete. 2024-04-02T13:32:13.180 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T13:32:13.180 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:32:13.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.279+0000 7fb21ff32640 1 Processor -- start 2024-04-02T13:32:13.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.279+0000 7fb21ff32640 1 -- start start 2024-04-02T13:32:13.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21ff32640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb21806d550 0x7fb2180d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21ff32640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2180cc600 con 0x7fb21815e4a0 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21ff32640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2180cc8e0 con 0x7fb2180d0c50 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21ff32640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb2180af300 con 0x7fb21806d550 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21dcce640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb21815e4a0 0x7fb2180aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:60978/0 (socket says 172.21.15.44:60978) 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21dcce640 1 -- 172.21.15.44:0/3609702758 learned_addr learned my addr 172.21.15.44:0/3609702758 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21e4cf640 1 --2- 172.21.15.44:0/3609702758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb21806d550 0x7fb2180d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3162598775 0 0) 0x7fb2180cc600 con 0x7fb21815e4a0 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb204003520 con 0x7fb21815e4a0 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3819364664 0 0) 0x7fb204003520 con 0x7fb21815e4a0 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb21806d550 msgr2=0x7fb2180d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 --2- 172.21.15.44:0/3609702758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb21806d550 0x7fb2180d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 >> v1:172.21.15.69:6789/0 conn(0x7fb2180d0c50 legacy=0x7fb21806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2180ccbc0 con 0x7fb21815e4a0 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7fb214001670 con 0x7fb21815e4a0 2024-04-02T13:32:13.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 >> v1:172.21.15.44:6789/0 conn(0x7fb21815e4a0 legacy=0x7fb2180aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 --2- 172.21.15.44:0/3609702758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb21806d550 0x7fb2040039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 --2- 172.21.15.44:0/3609702758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb204003f60 0x7fb204004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 --2- 172.21.15.44:0/3609702758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb204004d40 0x7fb2040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb2180af300 con 0x7fb204004d40 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb20400baa0 con 0x7fb204003f60 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.280+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb20400bca0 con 0x7fb21806d550 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.281+0000 7fb21dcce640 1 --2- 172.21.15.44:0/3609702758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb21806d550 0x7fb2040039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.281+0000 7fb21e4cf640 1 --2- 172.21.15.44:0/3609702758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb204004d40 0x7fb2040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.281+0000 7fb21d4cd640 1 --2- 172.21.15.44:0/3609702758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb204003f60 0x7fb204004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.281+0000 7fb21dcce640 1 -- 172.21.15.44:0/3609702758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb204003f60 msgr2=0x7fb204004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.281+0000 7fb21dcce640 1 --2- 172.21.15.44:0/3609702758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb204003f60 0x7fb204004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.281+0000 7fb21dcce640 1 -- 172.21.15.44:0/3609702758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb204004d40 msgr2=0x7fb2040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.281+0000 7fb21dcce640 1 --2- 172.21.15.44:0/3609702758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb204004d40 0x7fb2040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.281+0000 7fb21dcce640 1 -- 172.21.15.44:0/3609702758 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb204003520 con 0x7fb21806d550 2024-04-02T13:32:13.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.281+0000 7fb21e4cf640 1 --2- 172.21.15.44:0/3609702758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb204004d40 0x7fb2040051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:32:13.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.282+0000 7fb21dcce640 1 --2- 172.21.15.44:0/3609702758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb21806d550 0x7fb2040039e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb21400c240 tx=0x7fb204005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62438b47f7cdc7da server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:13.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.282+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb214007cf0 con 0x7fb21806d550 2024-04-02T13:32:13.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.282+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb214030070 con 0x7fb21806d550 2024-04-02T13:32:13.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.282+0000 7fb21cccc640 1 -- 172.21.15.44:0/3609702758 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb214005d30 con 0x7fb21806d550 2024-04-02T13:32:13.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.282+0000 7fb21ff32640 1 -- 172.21.15.44:0/3609702758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb21806d550 msgr2=0x7fb2040039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.282+0000 7fb21ff32640 1 --2- 172.21.15.44:0/3609702758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb21806d550 0x7fb2040039e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb21400c240 tx=0x7fb204005830 comp rx=0 tx=0).stop 2024-04-02T13:32:13.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.282+0000 7fb21e4cf640 1 -- 172.21.15.44:0/3609702758 reap_dead start 2024-04-02T13:32:13.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.282+0000 7fb21ff32640 1 -- 172.21.15.44:0/3609702758 shutdown_connections 2024-04-02T13:32:13.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.282+0000 7fb21ff32640 1 -- 172.21.15.44:0/3609702758 >> 172.21.15.44:0/3609702758 conn(0x7fb21809aa20 msgr2=0x7fb21809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:13.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.282+0000 7fb21ff32640 1 -- 172.21.15.44:0/3609702758 shutdown_connections 2024-04-02T13:32:13.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.282+0000 7fb21ff32640 1 -- 172.21.15.44:0/3609702758 wait complete. 2024-04-02T13:32:13.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.284+0000 7fb21ff32640 1 Processor -- start 2024-04-02T13:32:13.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.284+0000 7fb21ff32640 1 -- start start 2024-04-02T13:32:13.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.284+0000 7fb21ff32640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb21806d550 0x7fb21814d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.284+0000 7fb21ff32640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb218161730 0x7fb21814d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.284+0000 7fb21ff32640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2180b72b0 0x7fb21814de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.284+0000 7fb21ff32640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb2180cd460 con 0x7fb218161730 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.284+0000 7fb21ff32640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb2180cd180 con 0x7fb21806d550 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.284+0000 7fb21ff32640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb21815e880 con 0x7fb2180b72b0 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.284+0000 7fb21d4cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb218161730 0x7fb21814d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.284+0000 7fb21d4cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb218161730 0x7fb21814d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39134/0 (socket says 172.21.15.44:39134) 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.284+0000 7fb21d4cd640 1 -- 172.21.15.44:0/1998956881 learned_addr learned my addr 172.21.15.44:0/1998956881 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb21dcce640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb21806d550 0x7fb21814d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb21e4cf640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2180b72b0 0x7fb21814de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb21d4cd640 1 -- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2180b72b0 msgr2=0x7fb21814de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb21d4cd640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2180b72b0 0x7fb21814de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb21d4cd640 1 -- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb21806d550 msgr2=0x7fb21814d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb21d4cd640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb21806d550 0x7fb21814d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb21d4cd640 1 -- 172.21.15.44:0/1998956881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2180c6160 con 0x7fb218161730 2024-04-02T13:32:13.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb21dcce640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb21806d550 0x7fb21814d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:13.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb21d4cd640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb218161730 0x7fb21814d920 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fb20800f5c0 tx=0x7fb20800fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:13.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb20e7fc640 1 -- 172.21.15.44:0/1998956881 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2080052e0 con 0x7fb218161730 2024-04-02T13:32:13.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb21ff32640 1 -- 172.21.15.44:0/1998956881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2180c5e80 con 0x7fb218161730 2024-04-02T13:32:13.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb20e7fc640 1 -- 172.21.15.44:0/1998956881 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb208018070 con 0x7fb218161730 2024-04-02T13:32:13.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb20e7fc640 1 -- 172.21.15.44:0/1998956881 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb208003810 con 0x7fb218161730 2024-04-02T13:32:13.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.285+0000 7fb21ff32640 1 -- 172.21.15.44:0/1998956881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2180b6ab0 con 0x7fb218161730 2024-04-02T13:32:13.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.286+0000 7fb21ff32640 1 -- 172.21.15.44:0/1998956881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2180684f0 con 0x7fb218161730 2024-04-02T13:32:13.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.286+0000 7fb20e7fc640 1 -- 172.21.15.44:0/1998956881 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb2080039b0 con 0x7fb218161730 2024-04-02T13:32:13.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.287+0000 7fb20e7fc640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb1fc06c8f0 0x7fb1fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.287+0000 7fb20e7fc640 1 -- 172.21.15.44:0/1998956881 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7fb20808bb70 con 0x7fb218161730 2024-04-02T13:32:13.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.287+0000 7fb21dcce640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb1fc06c8f0 0x7fb1fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.288+0000 7fb21dcce640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb1fc06c8f0 0x7fb1fc06edb0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb214034000 tx=0x7fb214007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:13.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.291+0000 7fb20e7fc640 1 -- 172.21.15.44:0/1998956881 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb208059d70 con 0x7fb218161730 2024-04-02T13:32:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.520+0000 7fb21ff32640 1 -- 172.21.15.44:0/1998956881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2180b0650 con 0x7fb218161730 2024-04-02T13:32:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.520+0000 7fb20e7fc640 1 -- 172.21.15.44:0/1998956881 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5184 (secure 0 0 0) 0x7fb208059670 con 0x7fb218161730 2024-04-02T13:32:13.523 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:13.523 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:31:11.495701+0000","modified":"2024-04-02T13:32:04.851323+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6835/16875964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":16875964},{"type":"v1","addr":"172.21.15.44:6835","nonce":16875964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":8}]} 2024-04-02T13:32:13.523 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 112 2024-04-02T13:32:13.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.522+0000 7fb21ff32640 1 -- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb1fc06c8f0 msgr2=0x7fb1fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.522+0000 7fb21ff32640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb1fc06c8f0 0x7fb1fc06edb0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb214034000 tx=0x7fb214007000 comp rx=0 tx=0).stop 2024-04-02T13:32:13.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.522+0000 7fb21ff32640 1 -- 172.21.15.44:0/1998956881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb218161730 msgr2=0x7fb21814d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.522+0000 7fb21ff32640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb218161730 0x7fb21814d920 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fb20800f5c0 tx=0x7fb20800fa80 comp rx=0 tx=0).stop 2024-04-02T13:32:13.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.522+0000 7fb21ff32640 1 -- 172.21.15.44:0/1998956881 shutdown_connections 2024-04-02T13:32:13.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.522+0000 7fb21ff32640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb1fc06c8f0 0x7fb1fc06edb0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.522+0000 7fb21ff32640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2180b72b0 0x7fb21814de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.522+0000 7fb21ff32640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb218161730 0x7fb21814d920 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.522+0000 7fb21ff32640 1 --2- 172.21.15.44:0/1998956881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb21806d550 0x7fb21814d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.522+0000 7fb21ff32640 1 -- 172.21.15.44:0/1998956881 >> 172.21.15.44:0/1998956881 conn(0x7fb21809aa20 msgr2=0x7fb2180b3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:13.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.522+0000 7fb21ff32640 1 -- 172.21.15.44:0/1998956881 shutdown_connections 2024-04-02T13:32:13.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.522+0000 7fb21ff32640 1 -- 172.21.15.44:0/1998956881 wait complete. 2024-04-02T13:32:13.536 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:32:13.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.634+0000 7f34bd3ff640 1 Processor -- start 2024-04-02T13:32:13.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.634+0000 7f34bd3ff640 1 -- start start 2024-04-02T13:32:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.635+0000 7f34bd3ff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80b3a70 0x7f34b806e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.635+0000 7f34bd3ff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f34b806b4d0 0x7f34b806b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.635+0000 7f34bd3ff640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b806e640 con 0x7f34b815c0f0 2024-04-02T13:32:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.635+0000 7f34bd3ff640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f34b815c4d0 con 0x7f34b806b4d0 2024-04-02T13:32:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.635+0000 7f34bd3ff640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f34b806e920 con 0x7f34b80b3a70 2024-04-02T13:32:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.635+0000 7f34b77fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f34b806b4d0 0x7f34b806b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.635+0000 7f34b6ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f34b815c0f0 0x7f34b806f960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:60986/0 (socket says 172.21.15.44:60986) 2024-04-02T13:32:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.635+0000 7f34b6ffd640 1 -- 172.21.15.44:0/2963516891 learned_addr learned my addr 172.21.15.44:0/2963516891 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.636+0000 7f34affff640 1 --2- 172.21.15.44:0/2963516891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80b3a70 0x7f34b806e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.636+0000 7f34b77fe640 1 -- 172.21.15.44:0/2963516891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80b3a70 msgr2=0x7f34b806e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.636+0000 7f34b67fc640 1 -- 172.21.15.44:0/2963516891 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1712316396 0 0) 0x7f34b806e640 con 0x7f34b815c0f0 2024-04-02T13:32:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.636+0000 7f34b77fe640 1 --2- 172.21.15.44:0/2963516891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80b3a70 0x7f34b806e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.636+0000 7f34b77fe640 1 -- 172.21.15.44:0/2963516891 >> v1:172.21.15.44:6789/0 conn(0x7f34b815c0f0 legacy=0x7f34b806f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.636+0000 7f34b77fe640 1 -- 172.21.15.44:0/2963516891 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b806ec00 con 0x7f34b806b4d0 2024-04-02T13:32:13.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.636+0000 7f34affff640 1 --2- 172.21.15.44:0/2963516891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80b3a70 0x7f34b806e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:13.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.636+0000 7f34b77fe640 1 --2- 172.21.15.44:0/2963516891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34b806b4d0 0x7f34b806b960 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009d90 tx=0x7f34b002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0dbaa40c7a1e084 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:13.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.637+0000 7f34b67fc640 1 -- 172.21.15.44:0/2963516891 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0039070 con 0x7f34b806b4d0 2024-04-02T13:32:13.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.637+0000 7f34b67fc640 1 -- 172.21.15.44:0/2963516891 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b002cbc0 con 0x7f34b806b4d0 2024-04-02T13:32:13.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.637+0000 7f34b67fc640 1 -- 172.21.15.44:0/2963516891 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0034440 con 0x7f34b806b4d0 2024-04-02T13:32:13.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.637+0000 7f34bd3ff640 1 -- 172.21.15.44:0/2963516891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34b806b4d0 msgr2=0x7f34b806b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.637+0000 7f34bd3ff640 1 --2- 172.21.15.44:0/2963516891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34b806b4d0 0x7f34b806b960 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009d90 tx=0x7f34b002be10 comp rx=0 tx=0).stop 2024-04-02T13:32:13.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.637+0000 7f34bd3ff640 1 -- 172.21.15.44:0/2963516891 shutdown_connections 2024-04-02T13:32:13.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.637+0000 7f34bd3ff640 1 --2- 172.21.15.44:0/2963516891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34b806b4d0 0x7f34b806b960 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.637+0000 7f34bd3ff640 1 --2- 172.21.15.44:0/2963516891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80b3a70 0x7f34b806e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.637+0000 7f34bd3ff640 1 -- 172.21.15.44:0/2963516891 >> 172.21.15.44:0/2963516891 conn(0x7f34b80d0ba0 msgr2=0x7f34b80d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:13.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.637+0000 7f34bd3ff640 1 -- 172.21.15.44:0/2963516891 shutdown_connections 2024-04-02T13:32:13.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.637+0000 7f34bd3ff640 1 -- 172.21.15.44:0/2963516891 wait complete. 2024-04-02T13:32:13.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.638+0000 7f34bd3ff640 1 Processor -- start 2024-04-02T13:32:13.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.638+0000 7f34bd3ff640 1 -- start start 2024-04-02T13:32:13.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34bd3ff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34b806b4d0 0x7f34b815fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34bd3ff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80b3a70 0x7f34b8158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34bd3ff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b815c0f0 0x7f34b80c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34bd3ff640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f34b8072610 con 0x7f34b806b4d0 2024-04-02T13:32:13.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34bd3ff640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f34b8072490 con 0x7f34b815c0f0 2024-04-02T13:32:13.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34bd3ff640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f34b8072790 con 0x7f34b80b3a70 2024-04-02T13:32:13.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34b6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34b806b4d0 0x7f34b815fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34b6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34b806b4d0 0x7f34b815fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39150/0 (socket says 172.21.15.44:39150) 2024-04-02T13:32:13.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34b6ffd640 1 -- 172.21.15.44:0/4220322727 learned_addr learned my addr 172.21.15.44:0/4220322727 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:13.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34affff640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80b3a70 0x7f34b8158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34b77fe640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b815c0f0 0x7f34b80c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34b6ffd640 1 -- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80b3a70 msgr2=0x7f34b8158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34b6ffd640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80b3a70 0x7f34b8158d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34b6ffd640 1 -- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b815c0f0 msgr2=0x7f34b80c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34b6ffd640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b815c0f0 0x7f34b80c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.639+0000 7f34b6ffd640 1 -- 172.21.15.44:0/4220322727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b80c5b00 con 0x7f34b806b4d0 2024-04-02T13:32:13.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.640+0000 7f34b6ffd640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34b806b4d0 0x7f34b815fbe0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f34a800c460 tx=0x7f34a800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:13.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.640+0000 7f34af7fe640 1 -- 172.21.15.44:0/4220322727 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8018070 con 0x7f34b806b4d0 2024-04-02T13:32:13.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.640+0000 7f34af7fe640 1 -- 172.21.15.44:0/4220322727 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a8007500 con 0x7f34b806b4d0 2024-04-02T13:32:13.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.640+0000 7f34bd3ff640 1 -- 172.21.15.44:0/4220322727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b80c5de0 con 0x7f34b806b4d0 2024-04-02T13:32:13.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.640+0000 7f34af7fe640 1 -- 172.21.15.44:0/4220322727 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8004a20 con 0x7f34b806b4d0 2024-04-02T13:32:13.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.640+0000 7f34bd3ff640 1 -- 172.21.15.44:0/4220322727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b80c63a0 con 0x7f34b806b4d0 2024-04-02T13:32:13.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.641+0000 7f34bd3ff640 1 -- 172.21.15.44:0/4220322727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b804f8d0 con 0x7f34b806b4d0 2024-04-02T13:32:13.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.641+0000 7f34af7fe640 1 -- 172.21.15.44:0/4220322727 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f34a801b070 con 0x7f34b806b4d0 2024-04-02T13:32:13.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.642+0000 7f34af7fe640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f349806c920 0x7f349806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.642+0000 7f34af7fe640 1 -- 172.21.15.44:0/4220322727 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7f34a808b000 con 0x7f34b806b4d0 2024-04-02T13:32:13.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.642+0000 7f34affff640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f349806c920 0x7f349806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.642+0000 7f34affff640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f349806c920 0x7f349806ede0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f34b815cb80 tx=0x7f34a0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:13.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.646+0000 7f34af7fe640 1 -- 172.21.15.44:0/4220322727 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f34a80591f0 con 0x7f34b806b4d0 2024-04-02T13:32:13.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.871+0000 7f34bd3ff640 1 -- 172.21.15.44:0/4220322727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34b8072dc0 con 0x7f34b806b4d0 2024-04-02T13:32:13.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.872+0000 7f34af7fe640 1 -- 172.21.15.44:0/4220322727 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5184 (secure 0 0 0) 0x7f34a8058af0 con 0x7f34b806b4d0 2024-04-02T13:32:13.875 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:13.875 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T13:31:11.495701+0000","modified":"2024-04-02T13:32:04.851323+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.44:6835/16875964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":16875964},{"type":"v1","addr":"172.21.15.44:6835","nonce":16875964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":8}]} 2024-04-02T13:32:13.875 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 112 2024-04-02T13:32:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.874+0000 7f34bd3ff640 1 -- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f349806c920 msgr2=0x7f349806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.874+0000 7f34bd3ff640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f349806c920 0x7f349806ede0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f34b815cb80 tx=0x7f34a0008040 comp rx=0 tx=0).stop 2024-04-02T13:32:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.874+0000 7f34bd3ff640 1 -- 172.21.15.44:0/4220322727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34b806b4d0 msgr2=0x7f34b815fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.874+0000 7f34bd3ff640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34b806b4d0 0x7f34b815fbe0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f34a800c460 tx=0x7f34a800c920 comp rx=0 tx=0).stop 2024-04-02T13:32:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.874+0000 7f34bd3ff640 1 -- 172.21.15.44:0/4220322727 shutdown_connections 2024-04-02T13:32:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.874+0000 7f34bd3ff640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f349806c920 0x7f349806ede0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.874+0000 7f34bd3ff640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b815c0f0 0x7f34b80c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.874+0000 7f34bd3ff640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80b3a70 0x7f34b8158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.874+0000 7f34bd3ff640 1 --2- 172.21.15.44:0/4220322727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34b806b4d0 0x7f34b815fbe0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.874+0000 7f34bd3ff640 1 -- 172.21.15.44:0/4220322727 >> 172.21.15.44:0/4220322727 conn(0x7f34b80d0ba0 msgr2=0x7f34b815b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:13.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.874+0000 7f34bd3ff640 1 -- 172.21.15.44:0/4220322727 shutdown_connections 2024-04-02T13:32:13.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.874+0000 7f34bd3ff640 1 -- 172.21.15.44:0/4220322727 wait complete. 2024-04-02T13:32:13.888 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:32:13.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.987+0000 7f7261c60640 1 Processor -- start 2024-04-02T13:32:13.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.987+0000 7f7261c60640 1 -- start start 2024-04-02T13:32:13.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.988+0000 7f7261c60640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f725c0d0a30 0x7f725c06cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.988+0000 7f7261c60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f725c0d2040 0x7f725c0d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.988+0000 7f7261c60640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f725c0d0000 con 0x7f725c0d0e10 2024-04-02T13:32:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.988+0000 7f7261c60640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f725c06cb00 con 0x7f725c0d0a30 2024-04-02T13:32:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.988+0000 7f7261c60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f725c06c1d0 con 0x7f725c0d2040 2024-04-02T13:32:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.988+0000 7f725b7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f725c0d0a30 0x7f725c06cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.988+0000 7f725b7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f725c0d0a30 0x7f725c06cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39166/0 (socket says 172.21.15.44:39166) 2024-04-02T13:32:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.988+0000 7f725b7fe640 1 -- 172.21.15.44:0/2857775357 learned_addr learned my addr 172.21.15.44:0/2857775357 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.989+0000 7f725bfff640 1 --2- 172.21.15.44:0/2857775357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f725c0d2040 0x7f725c0d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.989+0000 7f725b7fe640 1 -- 172.21.15.44:0/2857775357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f725c0d2040 msgr2=0x7f725c0d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.989+0000 7f725b7fe640 1 --2- 172.21.15.44:0/2857775357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f725c0d2040 0x7f725c0d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.989+0000 7f725b7fe640 1 -- 172.21.15.44:0/2857775357 >> v1:172.21.15.69:6789/0 conn(0x7f725c0d0e10 legacy=0x7f725c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.989+0000 7f725b7fe640 1 -- 172.21.15.44:0/2857775357 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725c06c4b0 con 0x7f725c0d0a30 2024-04-02T13:32:13.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.989+0000 7f725b7fe640 1 --2- 172.21.15.44:0/2857775357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f725c0d0a30 0x7f725c06cf90 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f724c009870 tx=0x7f724c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3329347a5fc492b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.990+0000 7f725a7fc640 1 -- 172.21.15.44:0/2857775357 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c035020 con 0x7f725c0d0a30 2024-04-02T13:32:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.990+0000 7f725a7fc640 1 -- 172.21.15.44:0/2857775357 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724c031640 con 0x7f725c0d0a30 2024-04-02T13:32:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.990+0000 7f725a7fc640 1 -- 172.21.15.44:0/2857775357 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c02d040 con 0x7f725c0d0a30 2024-04-02T13:32:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.990+0000 7f7261c60640 1 -- 172.21.15.44:0/2857775357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f725c0d0a30 msgr2=0x7f725c06cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.990+0000 7f7261c60640 1 --2- 172.21.15.44:0/2857775357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f725c0d0a30 0x7f725c06cf90 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f724c009870 tx=0x7f724c02b8f0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.990+0000 7f7261c60640 1 -- 172.21.15.44:0/2857775357 shutdown_connections 2024-04-02T13:32:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.990+0000 7f7261c60640 1 --2- 172.21.15.44:0/2857775357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f725c0d2040 0x7f725c0d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.990+0000 7f7261c60640 1 --2- 172.21.15.44:0/2857775357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f725c0d0a30 0x7f725c06cf90 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.990+0000 7f7261c60640 1 -- 172.21.15.44:0/2857775357 >> 172.21.15.44:0/2857775357 conn(0x7f725c09d470 msgr2=0x7f725c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.990+0000 7f7261c60640 1 -- 172.21.15.44:0/2857775357 shutdown_connections 2024-04-02T13:32:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.990+0000 7f7261c60640 1 -- 172.21.15.44:0/2857775357 wait complete. 2024-04-02T13:32:13.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.991+0000 7f7261c60640 1 Processor -- start 2024-04-02T13:32:13.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.992+0000 7f7261c60640 1 -- start start 2024-04-02T13:32:13.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.992+0000 7f7261c60640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f725c0d0a30 0x7f725c0b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.992+0000 7f7261c60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f725c0d0e10 0x7f725c15fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.992+0000 7f7261c60640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f725c0d2040 0x7f725c15ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:13.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.992+0000 7f7261c60640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f725c15e6f0 con 0x7f725c0d0a30 2024-04-02T13:32:13.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.992+0000 7f7261c60640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f725c0d1270 con 0x7f725c0d2040 2024-04-02T13:32:13.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.992+0000 7f7261c60640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f725c15e870 con 0x7f725c0d0e10 2024-04-02T13:32:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.992+0000 7f725b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f725c0d0a30 0x7f725c0b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.992+0000 7f725b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f725c0d0a30 0x7f725c0b95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39178/0 (socket says 172.21.15.44:39178) 2024-04-02T13:32:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.992+0000 7f725b7fe640 1 -- 172.21.15.44:0/218951192 learned_addr learned my addr 172.21.15.44:0/218951192 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f725affd640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f725c0d0e10 0x7f725c15fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f725bfff640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f725c0d2040 0x7f725c15ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f725b7fe640 1 -- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f725c0d0e10 msgr2=0x7f725c15fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f725b7fe640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f725c0d0e10 0x7f725c15fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f725b7fe640 1 -- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f725c0d2040 msgr2=0x7f725c15ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f725b7fe640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f725c0d2040 0x7f725c15ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f725b7fe640 1 -- 172.21.15.44:0/218951192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725c0ca0c0 con 0x7f725c0d0a30 2024-04-02T13:32:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f725affd640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f725c0d0e10 0x7f725c15fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f725b7fe640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f725c0d0a30 0x7f725c0b95e0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f724c000c00 tx=0x7f724c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:13.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f7243fff640 1 -- 172.21.15.44:0/218951192 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c040070 con 0x7f725c0d0a30 2024-04-02T13:32:13.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f7243fff640 1 -- 172.21.15.44:0/218951192 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724c0399f0 con 0x7f725c0d0a30 2024-04-02T13:32:13.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f7261c60640 1 -- 172.21.15.44:0/218951192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f725c1605c0 con 0x7f725c0d0a30 2024-04-02T13:32:13.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f7243fff640 1 -- 172.21.15.44:0/218951192 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c035040 con 0x7f725c0d0a30 2024-04-02T13:32:13.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.993+0000 7f7261c60640 1 -- 172.21.15.44:0/218951192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f725c160b80 con 0x7f725c0d0a30 2024-04-02T13:32:13.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.994+0000 7f7261c60640 1 -- 172.21.15.44:0/218951192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f725c0684f0 con 0x7f725c0d0a30 2024-04-02T13:32:13.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.995+0000 7f7243fff640 1 -- 172.21.15.44:0/218951192 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f724c04b060 con 0x7f725c0d0a30 2024-04-02T13:32:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.995+0000 7f7243fff640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f724806c8f0 0x7f724806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.995+0000 7f7243fff640 1 -- 172.21.15.44:0/218951192 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7f724c0ad550 con 0x7f725c0d0a30 2024-04-02T13:32:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.995+0000 7f725affd640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f724806c8f0 0x7f724806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:14.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.996+0000 7f725affd640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f724806c8f0 0x7f724806edb0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f725c0cad70 tx=0x7f7250009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:14.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:13.999+0000 7f7243fff640 1 -- 172.21.15.44:0/218951192 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f724c07b7d0 con 0x7f725c0d0a30 2024-04-02T13:32:14.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.185+0000 7f7261c60640 1 -- 172.21.15.44:0/218951192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f725c0d17c0 con 0x7f725c0d0a30 2024-04-02T13:32:14.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.186+0000 7f7243fff640 1 -- 172.21.15.44:0/218951192 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) ==== 74+0+17163 (secure 0 0 0) 0x7f724c07b0d0 con 0x7f725c0d0a30 2024-04-02T13:32:14.189 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:14.189 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":123,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:31:59.833349+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-02T13:31:09.752845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-02T13:31:10.437454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6835/2895101355":"2024-04-03T13:31:59.833326+0000","172.21.15.44:6833/2895101355":"2024-04-03T13:31:59.833326+0000","172.21.15.44:6835/2109578276":"2024-04-03T13:31:41.102885+0000","172.21.15.44:6833/2109578276":"2024-04-03T13:31:41.102885+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:32:14.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.188+0000 7f7261c60640 1 -- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f724806c8f0 msgr2=0x7f724806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:14.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.188+0000 7f7261c60640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f724806c8f0 0x7f724806edb0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f725c0cad70 tx=0x7f7250009040 comp rx=0 tx=0).stop 2024-04-02T13:32:14.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.188+0000 7f7261c60640 1 -- 172.21.15.44:0/218951192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f725c0d0a30 msgr2=0x7f725c0b95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:14.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.188+0000 7f7261c60640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f725c0d0a30 0x7f725c0b95e0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f724c000c00 tx=0x7f724c0046c0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.188+0000 7f7261c60640 1 -- 172.21.15.44:0/218951192 shutdown_connections 2024-04-02T13:32:14.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.188+0000 7f7261c60640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f724806c8f0 0x7f724806edb0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.188+0000 7f7261c60640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f725c0d2040 0x7f725c15ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.188+0000 7f7261c60640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f725c0d0e10 0x7f725c15fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.188+0000 7f7261c60640 1 --2- 172.21.15.44:0/218951192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f725c0d0a30 0x7f725c0b95e0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.188+0000 7f7261c60640 1 -- 172.21.15.44:0/218951192 >> 172.21.15.44:0/218951192 conn(0x7f725c09d470 msgr2=0x7f725c1567d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:14.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.188+0000 7f7261c60640 1 -- 172.21.15.44:0/218951192 shutdown_connections 2024-04-02T13:32:14.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.188+0000 7f7261c60640 1 -- 172.21.15.44:0/218951192 wait complete. 2024-04-02T13:32:14.202 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T13:32:14.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.302+0000 7fecb6132640 1 Processor -- start 2024-04-02T13:32:14.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.302+0000 7fecb6132640 1 -- start start 2024-04-02T13:32:14.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecb6132640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fecb00d0a30 0x7fecb006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:14.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecb6132640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecb00d2040 0x7fecb00d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:14.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecb6132640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecb00d0000 con 0x7fecb00d0e10 2024-04-02T13:32:14.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecb6132640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fecb006cb00 con 0x7fecb00d0a30 2024-04-02T13:32:14.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecb6132640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fecb006c1d0 con 0x7fecb00d2040 2024-04-02T13:32:14.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecaf7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fecb00d0a30 0x7fecb006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:14.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecaeffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fecb00d0e10 0x7fecb00cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:32770/0 (socket says 172.21.15.44:32770) 2024-04-02T13:32:14.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecaeffd640 1 -- 172.21.15.44:0/1601787658 learned_addr learned my addr 172.21.15.44:0/1601787658 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:14.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecaffff640 1 --2- 172.21.15.44:0/1601787658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecb00d2040 0x7fecb00d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:14.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecaf7fe640 1 -- 172.21.15.44:0/1601787658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecb00d2040 msgr2=0x7fecb00d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:14.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecaf7fe640 1 --2- 172.21.15.44:0/1601787658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecb00d2040 0x7fecb00d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecaf7fe640 1 -- 172.21.15.44:0/1601787658 >> v1:172.21.15.44:6789/0 conn(0x7fecb00d0e10 legacy=0x7fecb00cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:14.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecaf7fe640 1 -- 172.21.15.44:0/1601787658 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb006c4b0 con 0x7fecb00d0a30 2024-04-02T13:32:14.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecae7fc640 1 -- 172.21.15.44:0/1601787658 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3976997292 0 0) 0x7fecb00d0000 con 0x7fecb00d0e10 2024-04-02T13:32:14.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.303+0000 7fecaffff640 1 --2- 172.21.15.44:0/1601787658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecb00d2040 0x7fecb00d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:14.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.304+0000 7fecaf7fe640 1 --2- 172.21.15.44:0/1601787658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fecb00d0a30 0x7fecb006cf90 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7feca00097a0 tx=0x7feca002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13350805b06f7771 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:14.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.304+0000 7fecae7fc640 1 -- 172.21.15.44:0/1601787658 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca00315a0 con 0x7fecb00d0a30 2024-04-02T13:32:14.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.304+0000 7fecae7fc640 1 -- 172.21.15.44:0/1601787658 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feca0037070 con 0x7fecb00d0a30 2024-04-02T13:32:14.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.304+0000 7fecae7fc640 1 -- 172.21.15.44:0/1601787658 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca0031960 con 0x7fecb00d0a30 2024-04-02T13:32:14.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.304+0000 7fecb6132640 1 -- 172.21.15.44:0/1601787658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fecb00d0a30 msgr2=0x7fecb006cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:14.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.304+0000 7fecb6132640 1 --2- 172.21.15.44:0/1601787658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fecb00d0a30 0x7fecb006cf90 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7feca00097a0 tx=0x7feca002b840 comp rx=0 tx=0).stop 2024-04-02T13:32:14.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.305+0000 7fecb6132640 1 -- 172.21.15.44:0/1601787658 shutdown_connections 2024-04-02T13:32:14.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.305+0000 7fecb6132640 1 --2- 172.21.15.44:0/1601787658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecb00d2040 0x7fecb00d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.305+0000 7fecb6132640 1 --2- 172.21.15.44:0/1601787658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fecb00d0a30 0x7fecb006cf90 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.305+0000 7fecb6132640 1 -- 172.21.15.44:0/1601787658 >> 172.21.15.44:0/1601787658 conn(0x7fecb009d470 msgr2=0x7fecb00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:14.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.305+0000 7fecb6132640 1 -- 172.21.15.44:0/1601787658 shutdown_connections 2024-04-02T13:32:14.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.305+0000 7fecb6132640 1 -- 172.21.15.44:0/1601787658 wait complete. 2024-04-02T13:32:14.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.306+0000 7fecb6132640 1 Processor -- start 2024-04-02T13:32:14.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.306+0000 7fecb6132640 1 -- start start 2024-04-02T13:32:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecb6132640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fecb00d0a30 0x7fecb00b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecb6132640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecb00d0e10 0x7fecb014f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecb6132640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecb00d2040 0x7fecb00b6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecb6132640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fecb015e6f0 con 0x7fecb00d0a30 2024-04-02T13:32:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecb6132640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fecb00d1270 con 0x7fecb00d0e10 2024-04-02T13:32:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecb6132640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fecb015e870 con 0x7fecb00d2040 2024-04-02T13:32:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecaf7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fecb00d0a30 0x7fecb00b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecaf7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fecb00d0a30 0x7fecb00b6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39186/0 (socket says 172.21.15.44:39186) 2024-04-02T13:32:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecaf7fe640 1 -- 172.21.15.44:0/1884386332 learned_addr learned my addr 172.21.15.44:0/1884386332 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecaeffd640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecb00d0e10 0x7fecb014f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecaffff640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecb00d2040 0x7fecb00b6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecaf7fe640 1 -- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecb00d2040 msgr2=0x7fecb00b6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecaf7fe640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecb00d2040 0x7fecb00b6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecaf7fe640 1 -- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecb00d0e10 msgr2=0x7fecb014f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecaf7fe640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecb00d0e10 0x7fecb014f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecaf7fe640 1 -- 172.21.15.44:0/1884386332 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb00c8e00 con 0x7fecb00d0a30 2024-04-02T13:32:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecaeffd640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecb00d0e10 0x7fecb014f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecaffff640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecb00d2040 0x7fecb00b6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fecaf7fe640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fecb00d0a30 0x7fecb00b6d60 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7feca00098c0 tx=0x7feca00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fec97fff640 1 -- 172.21.15.44:0/1884386332 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca0004230 con 0x7fecb00d0a30 2024-04-02T13:32:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fec97fff640 1 -- 172.21.15.44:0/1884386332 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feca00043d0 con 0x7fecb00d0a30 2024-04-02T13:32:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.307+0000 7fec97fff640 1 -- 172.21.15.44:0/1884386332 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca0003850 con 0x7fecb00d0a30 2024-04-02T13:32:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.308+0000 7fecb6132640 1 -- 172.21.15.44:0/1884386332 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecb00c90e0 con 0x7fecb00d0a30 2024-04-02T13:32:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.308+0000 7fecb6132640 1 -- 172.21.15.44:0/1884386332 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecb00c96a0 con 0x7fecb00d0a30 2024-04-02T13:32:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.309+0000 7fecb6132640 1 -- 172.21.15.44:0/1884386332 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecb00684f0 con 0x7fecb00d0a30 2024-04-02T13:32:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.309+0000 7fec97fff640 1 -- 172.21.15.44:0/1884386332 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feca00039f0 con 0x7fecb00d0a30 2024-04-02T13:32:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.309+0000 7fec97fff640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fec9c06c8f0 0x7fec9c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.309+0000 7fec97fff640 1 -- 172.21.15.44:0/1884386332 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6273+0+0 (secure 0 0 0) 0x7feca002f030 con 0x7fecb00d0a30 2024-04-02T13:32:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.310+0000 7fecaeffd640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fec9c06c8f0 0x7fec9c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:14.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.310+0000 7fecaeffd640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fec9c06c8f0 0x7fec9c06edb0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fecb0159fc0 tx=0x7feca4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:14.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.314+0000 7fec97fff640 1 -- 172.21.15.44:0/1884386332 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7feca0041070 con 0x7fecb00d0a30 2024-04-02T13:32:14.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.538+0000 7fecb6132640 1 -- 172.21.15.44:0/1884386332 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fecb00d17c0 con 0x7fecb00d0a30 2024-04-02T13:32:14.541 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-04-02T13:32:14.539+0000 7fe127ee3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T13:32:14.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.540+0000 7fec97fff640 1 -- 172.21.15.44:0/1884386332 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v113) ==== 161+0+0 (secure 0 0 0) 0x7feca007bf00 con 0x7fecb00d0a30 2024-04-02T13:32:14.543 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T13:32:14.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.542+0000 7fecb6132640 1 -- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fec9c06c8f0 msgr2=0x7fec9c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:14.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.542+0000 7fecb6132640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fec9c06c8f0 0x7fec9c06edb0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fecb0159fc0 tx=0x7feca4009040 comp rx=0 tx=0).stop 2024-04-02T13:32:14.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.542+0000 7fecb6132640 1 -- 172.21.15.44:0/1884386332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fecb00d0a30 msgr2=0x7fecb00b6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:14.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.542+0000 7fecb6132640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fecb00d0a30 0x7fecb00b6d60 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7feca00098c0 tx=0x7feca00046c0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.542+0000 7fecb6132640 1 -- 172.21.15.44:0/1884386332 shutdown_connections 2024-04-02T13:32:14.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.542+0000 7fecb6132640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fec9c06c8f0 0x7fec9c06edb0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.542+0000 7fecb6132640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecb00d2040 0x7fecb00b6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.542+0000 7fecb6132640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecb00d0e10 0x7fecb014f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.542+0000 7fecb6132640 1 --2- 172.21.15.44:0/1884386332 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fecb00d0a30 0x7fecb00b6d60 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.542+0000 7fecb6132640 1 -- 172.21.15.44:0/1884386332 >> 172.21.15.44:0/1884386332 conn(0x7fecb009d470 msgr2=0x7fecb00b2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:14.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.542+0000 7fecb6132640 1 -- 172.21.15.44:0/1884386332 shutdown_connections 2024-04-02T13:32:14.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.542+0000 7fecb6132640 1 -- 172.21.15.44:0/1884386332 wait complete. 2024-04-02T13:32:14.546 INFO:tasks.ceph.mds.a.smithi044.stderr: -1439> 2024-04-02T13:32:05.840+0000 7f4d2ced3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T13:32:14.556 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T13:32:14.571 INFO:tasks.ceph.mds.a.smithi044.stdout:starting mds.a at 2024-04-02T13:32:14.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.658+0000 7f3fd9132640 1 Processor -- start 2024-04-02T13:32:14.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.658+0000 7f3fd9132640 1 -- start start 2024-04-02T13:32:14.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.659+0000 7f3fd9132640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fd415e1d0 0x7f3fd40d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:14.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.659+0000 7f3fd9132640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3fd40d2dd0 0x7f3fd40b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:14.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.659+0000 7f3fd9132640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fd40d1640 con 0x7f3fd40cec60 2024-04-02T13:32:14.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.659+0000 7f3fd9132640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f3fd40cfe00 con 0x7f3fd40d2dd0 2024-04-02T13:32:14.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.659+0000 7f3fd9132640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3fd40cf1f0 con 0x7f3fd415e1d0 2024-04-02T13:32:14.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.659+0000 7f3fd359e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f3fd40d2dd0 0x7f3fd40b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:14.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.659+0000 7f3fd259c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f3fd40cec60 0x7f3fd40d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52280/0 (socket says 172.21.15.44:52280) 2024-04-02T13:32:14.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.659+0000 7f3fd259c640 1 -- 172.21.15.44:0/3932143657 learned_addr learned my addr 172.21.15.44:0/3932143657 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:14.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.660+0000 7f3fd2d9d640 1 --2- 172.21.15.44:0/3932143657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fd415e1d0 0x7f3fd40d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:14.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.660+0000 7f3fd359e640 1 -- 172.21.15.44:0/3932143657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fd415e1d0 msgr2=0x7f3fd40d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:14.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.660+0000 7f3fd1d9b640 1 -- 172.21.15.44:0/3932143657 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2111613722 0 0) 0x7f3fd40d1640 con 0x7f3fd40cec60 2024-04-02T13:32:14.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.660+0000 7f3fd359e640 1 --2- 172.21.15.44:0/3932143657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fd415e1d0 0x7f3fd40d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.660+0000 7f3fd359e640 1 -- 172.21.15.44:0/3932143657 >> v1:172.21.15.44:6789/0 conn(0x7f3fd40cec60 legacy=0x7f3fd40d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:14.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.660+0000 7f3fd359e640 1 -- 172.21.15.44:0/3932143657 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd40cf4d0 con 0x7f3fd40d2dd0 2024-04-02T13:32:14.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.660+0000 7f3fd359e640 1 --2- 172.21.15.44:0/3932143657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd40d2dd0 0x7f3fd40b4ce0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4009d90 tx=0x7f3fc402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a493921f1670cff7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:14.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.661+0000 7f3fd1d9b640 1 -- 172.21.15.44:0/3932143657 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4039070 con 0x7f3fd40d2dd0 2024-04-02T13:32:14.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.661+0000 7f3fd1d9b640 1 -- 172.21.15.44:0/3932143657 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fc402cbc0 con 0x7f3fd40d2dd0 2024-04-02T13:32:14.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.661+0000 7f3fd1d9b640 1 -- 172.21.15.44:0/3932143657 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4034440 con 0x7f3fd40d2dd0 2024-04-02T13:32:14.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.661+0000 7f3fd9132640 1 -- 172.21.15.44:0/3932143657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd40d2dd0 msgr2=0x7f3fd40b4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:14.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.661+0000 7f3fd9132640 1 --2- 172.21.15.44:0/3932143657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd40d2dd0 0x7f3fd40b4ce0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4009d90 tx=0x7f3fc402be10 comp rx=0 tx=0).stop 2024-04-02T13:32:14.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.661+0000 7f3fd9132640 1 -- 172.21.15.44:0/3932143657 shutdown_connections 2024-04-02T13:32:14.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.661+0000 7f3fd9132640 1 --2- 172.21.15.44:0/3932143657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd40d2dd0 0x7f3fd40b4ce0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.661+0000 7f3fd9132640 1 --2- 172.21.15.44:0/3932143657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fd415e1d0 0x7f3fd40d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.661+0000 7f3fd9132640 1 -- 172.21.15.44:0/3932143657 >> 172.21.15.44:0/3932143657 conn(0x7f3fd409d3b0 msgr2=0x7f3fd40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.661+0000 7f3fd9132640 1 -- 172.21.15.44:0/3932143657 shutdown_connections 2024-04-02T13:32:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.661+0000 7f3fd9132640 1 -- 172.21.15.44:0/3932143657 wait complete. 2024-04-02T13:32:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.663+0000 7f3fd9132640 1 Processor -- start 2024-04-02T13:32:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.663+0000 7f3fd9132640 1 -- start start 2024-04-02T13:32:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.663+0000 7f3fd9132640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fd40cec60 0x7f3fd414d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.663+0000 7f3fd9132640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fd40d2dd0 0x7f3fd415fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.663+0000 7f3fd9132640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd415e1d0 0x7f3fd40a14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.663+0000 7f3fd9132640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd415e5b0 con 0x7f3fd415e1d0 2024-04-02T13:32:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.663+0000 7f3fd9132640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd40cfa90 con 0x7f3fd40cec60 2024-04-02T13:32:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.663+0000 7f3fd9132640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3fd415e730 con 0x7f3fd40d2dd0 2024-04-02T13:32:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.663+0000 7f3fd359e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd415e1d0 0x7f3fd40a14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.663+0000 7f3fd359e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd415e1d0 0x7f3fd40a14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42244/0 (socket says 172.21.15.44:42244) 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.663+0000 7f3fd359e640 1 -- 172.21.15.44:0/1139371912 learned_addr learned my addr 172.21.15.44:0/1139371912 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fd2d9d640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fd40cec60 0x7f3fd414d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fd259c640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fd40d2dd0 0x7f3fd415fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fd359e640 1 -- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fd40d2dd0 msgr2=0x7f3fd415fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fd359e640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fd40d2dd0 0x7f3fd415fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fd359e640 1 -- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fd40cec60 msgr2=0x7f3fd414d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fd359e640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fd40cec60 0x7f3fd414d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fd359e640 1 -- 172.21.15.44:0/1139371912 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd41f5660 con 0x7f3fd415e1d0 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fd2d9d640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fd40cec60 0x7f3fd414d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fd359e640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd415e1d0 0x7f3fd40a14c0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4009eb0 tx=0x7f3fc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/1139371912 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4039050 con 0x7f3fd415e1d0 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/1139371912 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fc4045070 con 0x7f3fd415e1d0 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fd9132640 1 -- 172.21.15.44:0/1139371912 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fd41f5940 con 0x7f3fd415e1d0 2024-04-02T13:32:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/1139371912 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc403e830 con 0x7f3fd415e1d0 2024-04-02T13:32:14.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.664+0000 7f3fd9132640 1 -- 172.21.15.44:0/1139371912 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fd40a0fa0 con 0x7f3fd415e1d0 2024-04-02T13:32:14.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.665+0000 7f3fd9132640 1 -- 172.21.15.44:0/1139371912 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fd40684f0 con 0x7f3fd415e1d0 2024-04-02T13:32:14.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.665+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/1139371912 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3fc403e9d0 con 0x7f3fd415e1d0 2024-04-02T13:32:14.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.666+0000 7f3fcb7fe640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fb406c8f0 0x7f3fb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:14.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.666+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/1139371912 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6359+0+0 (secure 0 0 0) 0x7f3fc40aeb30 con 0x7f3fd415e1d0 2024-04-02T13:32:14.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.666+0000 7f3fd2d9d640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fb406c8f0 0x7f3fb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:14.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.667+0000 7f3fd2d9d640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fb406c8f0 0x7f3fb406edb0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc006fd0 tx=0x7f3fcc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:14.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.670+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/1139371912 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f3fc407cdf0 con 0x7f3fd415e1d0 2024-04-02T13:32:14.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:14.893+0000 7f3fd9132640 1 -- 172.21.15.44:0/1139371912 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f3fd40b9a70 con 0x7f3fd415e1d0 2024-04-02T13:32:15.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.544+0000 7f3fcb7fe640 1 -- 172.21.15.44:0/1139371912 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v114) ==== 104+0+0 (secure 0 0 0) 0x7f3fc407c6f0 con 0x7f3fd415e1d0 2024-04-02T13:32:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.546+0000 7f3fd9132640 1 -- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fb406c8f0 msgr2=0x7f3fb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.546+0000 7f3fd9132640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fb406c8f0 0x7f3fb406edb0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc006fd0 tx=0x7f3fcc008040 comp rx=0 tx=0).stop 2024-04-02T13:32:15.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.546+0000 7f3fd9132640 1 -- 172.21.15.44:0/1139371912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd415e1d0 msgr2=0x7f3fd40a14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.546+0000 7f3fd9132640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd415e1d0 0x7f3fd40a14c0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4009eb0 tx=0x7f3fc4004a40 comp rx=0 tx=0).stop 2024-04-02T13:32:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.546+0000 7f3fd9132640 1 -- 172.21.15.44:0/1139371912 shutdown_connections 2024-04-02T13:32:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.546+0000 7f3fd9132640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f3fb406c8f0 0x7f3fb406edb0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.546+0000 7f3fd9132640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd415e1d0 0x7f3fd40a14c0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.546+0000 7f3fd9132640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fd40d2dd0 0x7f3fd415fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.546+0000 7f3fd9132640 1 --2- 172.21.15.44:0/1139371912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fd40cec60 0x7f3fd414d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:15.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.546+0000 7f3fd9132640 1 -- 172.21.15.44:0/1139371912 >> 172.21.15.44:0/1139371912 conn(0x7f3fd409d3b0 msgr2=0x7f3fd40cb590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.546+0000 7f3fd9132640 1 -- 172.21.15.44:0/1139371912 shutdown_connections 2024-04-02T13:32:15.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.546+0000 7f3fd9132640 1 -- 172.21.15.44:0/1139371912 wait complete. 2024-04-02T13:32:15.561 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T13:32:15.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.707+0000 7f58ef11c640 1 Processor -- start 2024-04-02T13:32:15.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.708+0000 7f58ef11c640 1 -- start start 2024-04-02T13:32:15.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.708+0000 7f58ef11c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f58e80b1540 0x7f58e80b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:15.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.708+0000 7f58ef11c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58e80ced00 con 0x7f58e80ce790 2024-04-02T13:32:15.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.708+0000 7f58ef11c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58e80cefe0 con 0x7f58e815e470 2024-04-02T13:32:15.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.708+0000 7f58ef11c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f58e80cf2c0 con 0x7f58e80b1540 2024-04-02T13:32:15.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.708+0000 7f58ed6b9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f58e80ce790 0x7f58e806d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52296/0 (socket says 172.21.15.44:52296) 2024-04-02T13:32:15.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.708+0000 7f58ed6b9640 1 -- 172.21.15.44:0/2622880004 learned_addr learned my addr 172.21.15.44:0/2622880004 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:15.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e7fff640 1 --2- 172.21.15.44:0/2622880004 >> v2:172.21.15.69:3300/0 conn(0x7f58e80b1540 0x7f58e80b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3438656952 0 0) 0x7f58e80ced00 con 0x7f58e80ce790 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58d8003520 con 0x7f58e80ce790 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2047350284 0 0) 0x7f58d8003520 con 0x7f58e80ce790 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 >> v2:172.21.15.69:3300/0 conn(0x7f58e80b1540 msgr2=0x7f58e80b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 --2- 172.21.15.44:0/2622880004 >> v2:172.21.15.69:3300/0 conn(0x7f58e80b1540 0x7f58e80b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 >> v1:172.21.15.69:6789/0 conn(0x7f58e815e470 legacy=0x7f58e80d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58e80cf5a0 con 0x7f58e80ce790 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f58e00019f0 con 0x7f58e80ce790 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 >> v1:172.21.15.44:6789/0 conn(0x7f58e80ce790 legacy=0x7f58e806d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 --2- 172.21.15.44:0/2622880004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58d80039b0 0x7f58d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 --2- 172.21.15.44:0/2622880004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d8004800 0x7f58d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 --2- 172.21.15.44:0/2622880004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58d80051c0 0x7f58d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f58e80cf2c0 con 0x7f58d8004800 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58d800c160 con 0x7f58d80051c0 2024-04-02T13:32:15.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f58d800c360 con 0x7f58d80039b0 2024-04-02T13:32:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e7fff640 1 --2- 172.21.15.44:0/2622880004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d8004800 0x7f58d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58eceb8640 1 --2- 172.21.15.44:0/2622880004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58d80039b0 0x7f58d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58ed6b9640 1 --2- 172.21.15.44:0/2622880004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58d80051c0 0x7f58d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e7fff640 1 -- 172.21.15.44:0/2622880004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58d80039b0 msgr2=0x7f58d8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e7fff640 1 --2- 172.21.15.44:0/2622880004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58d80039b0 0x7f58d8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e7fff640 1 -- 172.21.15.44:0/2622880004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58d80051c0 msgr2=0x7f58d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.709+0000 7f58e7fff640 1 --2- 172.21.15.44:0/2622880004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58d80051c0 0x7f58d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.710+0000 7f58e7fff640 1 -- 172.21.15.44:0/2622880004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d8003520 con 0x7f58d8004800 2024-04-02T13:32:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.710+0000 7f58ed6b9640 1 --2- 172.21.15.44:0/2622880004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58d80051c0 0x7f58d800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.710+0000 7f58eceb8640 1 --2- 172.21.15.44:0/2622880004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58d80039b0 0x7f58d8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:15.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.710+0000 7f58ed6b9640 1 -- 172.21.15.44:0/2622880004 reap_dead start 2024-04-02T13:32:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.710+0000 7f58e7fff640 1 --2- 172.21.15.44:0/2622880004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d8004800 0x7f58d8004c80 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009ac0 tx=0x7f58d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4f3bd966895f13 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.710+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d4039070 con 0x7f58d8004800 2024-04-02T13:32:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.710+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58d402c6b0 con 0x7f58d8004800 2024-04-02T13:32:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.710+0000 7f58e77fe640 1 -- 172.21.15.44:0/2622880004 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d4009690 con 0x7f58d8004800 2024-04-02T13:32:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.711+0000 7f58ef11c640 1 -- 172.21.15.44:0/2622880004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d8004800 msgr2=0x7f58d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.711+0000 7f58ef11c640 1 --2- 172.21.15.44:0/2622880004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d8004800 0x7f58d8004c80 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009ac0 tx=0x7f58d8005830 comp rx=0 tx=0).stop 2024-04-02T13:32:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.711+0000 7f58ef11c640 1 -- 172.21.15.44:0/2622880004 shutdown_connections 2024-04-02T13:32:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.711+0000 7f58ef11c640 1 --2- 172.21.15.44:0/2622880004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d8004800 0x7f58d8004c80 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.711+0000 7f58ef11c640 1 -- 172.21.15.44:0/2622880004 >> 172.21.15.44:0/2622880004 conn(0x7f58e809aa20 msgr2=0x7f58e809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.711+0000 7f58ef11c640 1 -- 172.21.15.44:0/2622880004 shutdown_connections 2024-04-02T13:32:15.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.711+0000 7f58ef11c640 1 -- 172.21.15.44:0/2622880004 wait complete. 2024-04-02T13:32:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.712+0000 7f58ef11c640 1 Processor -- start 2024-04-02T13:32:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.712+0000 7f58ef11c640 1 -- start start 2024-04-02T13:32:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58ef11c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d8004800 0x7f58e81489a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58ef11c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e80c5900 0x7f58e80c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58ef11c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58e815a130 0x7f58e815a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58ef11c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f58e80b32c0 con 0x7f58d8004800 2024-04-02T13:32:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58ef11c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58e806cb60 con 0x7f58e80c5900 2024-04-02T13:32:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58ef11c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f58e80b3440 con 0x7f58e815a130 2024-04-02T13:32:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58eceb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d8004800 0x7f58e81489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58eceb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d8004800 0x7f58e81489a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42256/0 (socket says 172.21.15.44:42256) 2024-04-02T13:32:15.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58eceb8640 1 -- 172.21.15.44:0/2078115865 learned_addr learned my addr 172.21.15.44:0/2078115865 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58e7fff640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e80c5900 0x7f58e80c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58ed6b9640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58e815a130 0x7f58e815a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58eceb8640 1 -- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58e815a130 msgr2=0x7f58e815a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58eceb8640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58e815a130 0x7f58e815a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58eceb8640 1 -- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e80c5900 msgr2=0x7f58e80c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58eceb8640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e80c5900 0x7f58e80c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58eceb8640 1 -- 172.21.15.44:0/2078115865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58e815aea0 con 0x7f58d8004800 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58e7fff640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e80c5900 0x7f58e80c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.713+0000 7f58eceb8640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d8004800 0x7f58e81489a0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f58dc00e4b0 tx=0x7f58dc00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.714+0000 7f58e57fa640 1 -- 172.21.15.44:0/2078115865 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc007e30 con 0x7f58d8004800 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.714+0000 7f58e57fa640 1 -- 172.21.15.44:0/2078115865 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58dc0052e0 con 0x7f58d8004800 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.714+0000 7f58ef11c640 1 -- 172.21.15.44:0/2078115865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58e815b180 con 0x7f58d8004800 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.714+0000 7f58e57fa640 1 -- 172.21.15.44:0/2078115865 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc007020 con 0x7f58d8004800 2024-04-02T13:32:15.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.714+0000 7f58ef11c640 1 -- 172.21.15.44:0/2078115865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58e815b740 con 0x7f58d8004800 2024-04-02T13:32:15.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.715+0000 7f58ef11c640 1 -- 172.21.15.44:0/2078115865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58e80684f0 con 0x7f58d8004800 2024-04-02T13:32:15.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.715+0000 7f58e57fa640 1 -- 172.21.15.44:0/2078115865 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f58dc01a020 con 0x7f58d8004800 2024-04-02T13:32:15.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.715+0000 7f58e57fa640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f58cc06c8f0 0x7f58cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:15.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.716+0000 7f58e57fa640 1 -- 172.21.15.44:0/2078115865 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6359+0+0 (secure 0 0 0) 0x7f58dc08b440 con 0x7f58d8004800 2024-04-02T13:32:15.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.716+0000 7f58e7fff640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f58cc06c8f0 0x7f58cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:15.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.716+0000 7f58e7fff640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f58cc06c8f0 0x7f58cc06edb0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f58d4036000 tx=0x7f58d4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:15.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.720+0000 7f58e57fa640 1 -- 172.21.15.44:0/2078115865 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f58dc01a310 con 0x7f58d8004800 2024-04-02T13:32:15.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:15.904+0000 7f58ef11c640 1 -- 172.21.15.44:0/2078115865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f58e80d2d10 con 0x7f58d8004800 2024-04-02T13:32:16.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.545+0000 7f58e57fa640 1 -- 172.21.15.44:0/2078115865 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v125) ==== 181+0+0 (secure 0 0 0) 0x7f58dc0594b0 con 0x7f58d8004800 2024-04-02T13:32:16.549 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' removed 2024-04-02T13:32:16.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.548+0000 7f58ef11c640 1 -- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f58cc06c8f0 msgr2=0x7f58cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:16.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.548+0000 7f58ef11c640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f58cc06c8f0 0x7f58cc06edb0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f58d4036000 tx=0x7f58d4030000 comp rx=0 tx=0).stop 2024-04-02T13:32:16.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.548+0000 7f58ef11c640 1 -- 172.21.15.44:0/2078115865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d8004800 msgr2=0x7f58e81489a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:16.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.548+0000 7f58ef11c640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d8004800 0x7f58e81489a0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f58dc00e4b0 tx=0x7f58dc00e970 comp rx=0 tx=0).stop 2024-04-02T13:32:16.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.549+0000 7f58ef11c640 1 -- 172.21.15.44:0/2078115865 shutdown_connections 2024-04-02T13:32:16.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.549+0000 7f58ef11c640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f58cc06c8f0 0x7f58cc06edb0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:16.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.549+0000 7f58ef11c640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58e815a130 0x7f58e815a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:16.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.549+0000 7f58ef11c640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e80c5900 0x7f58e80c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:16.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.549+0000 7f58ef11c640 1 --2- 172.21.15.44:0/2078115865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d8004800 0x7f58e81489a0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:16.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.549+0000 7f58ef11c640 1 -- 172.21.15.44:0/2078115865 >> 172.21.15.44:0/2078115865 conn(0x7f58e809aa20 msgr2=0x7f58e80af640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:16.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.549+0000 7f58ef11c640 1 -- 172.21.15.44:0/2078115865 shutdown_connections 2024-04-02T13:32:16.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.549+0000 7f58ef11c640 1 -- 172.21.15.44:0/2078115865 wait complete. 2024-04-02T13:32:16.565 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T13:32:16.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.677+0000 7f5bdc79d640 1 Processor -- start 2024-04-02T13:32:16.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.677+0000 7f5bdc79d640 1 -- start start 2024-04-02T13:32:16.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.678+0000 7f5bdc79d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 0x7f5bd4168f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:16.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.678+0000 7f5bdc79d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bd40d1630 con 0x7f5bd406cff0 2024-04-02T13:32:16.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.678+0000 7f5bdc79d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bd40d1910 con 0x7f5bd415e5d0 2024-04-02T13:32:16.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.678+0000 7f5bdc79d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5bd40d1bf0 con 0x7f5bd40b4180 2024-04-02T13:32:16.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.678+0000 7f5bda539640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f5bd406cff0 0x7f5bd406d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52312/0 (socket says 172.21.15.44:52312) 2024-04-02T13:32:16.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.678+0000 7f5bda539640 1 -- 172.21.15.44:0/577321975 learned_addr learned my addr 172.21.15.44:0/577321975 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:16.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bdad3a640 1 --2- 172.21.15.44:0/577321975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 0x7f5bd4168f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:16.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1385169606 0 0) 0x7f5bd40d1630 con 0x7f5bd406cff0 2024-04-02T13:32:16.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bc0003520 con 0x7f5bd406cff0 2024-04-02T13:32:16.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 497676992 0 0) 0x7f5bc0003520 con 0x7f5bd406cff0 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 msgr2=0x7f5bd4168f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 --2- 172.21.15.44:0/577321975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 0x7f5bd4168f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 >> v1:172.21.15.69:6789/0 conn(0x7f5bd415e5d0 legacy=0x7f5bd40d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd40b1540 con 0x7f5bd406cff0 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f5bc8001670 con 0x7f5bd406cff0 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 >> v1:172.21.15.44:6789/0 conn(0x7f5bd406cff0 legacy=0x7f5bd406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 --2- 172.21.15.44:0/577321975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 0x7f5bc00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 --2- 172.21.15.44:0/577321975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc0003f40 0x7f5bc0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 --2- 172.21.15.44:0/577321975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc0004d40 0x7f5bc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd40d1bf0 con 0x7f5bc0004d40 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc000bad0 con 0x7f5bc0003f40 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc000bcd0 con 0x7f5bd40b4180 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bda539640 1 --2- 172.21.15.44:0/577321975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 0x7f5bc00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bdad3a640 1 --2- 172.21.15.44:0/577321975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc0004d40 0x7f5bc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:16.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.679+0000 7f5bd9d38640 1 --2- 172.21.15.44:0/577321975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc0003f40 0x7f5bc0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:16.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.680+0000 7f5bda539640 1 -- 172.21.15.44:0/577321975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc0003f40 msgr2=0x7f5bc0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:16.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.680+0000 7f5bda539640 1 --2- 172.21.15.44:0/577321975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc0003f40 0x7f5bc0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:16.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.680+0000 7f5bda539640 1 -- 172.21.15.44:0/577321975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc0004d40 msgr2=0x7f5bc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:16.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.680+0000 7f5bda539640 1 --2- 172.21.15.44:0/577321975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc0004d40 0x7f5bc00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:16.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.680+0000 7f5bda539640 1 -- 172.21.15.44:0/577321975 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc0003520 con 0x7f5bd40b4180 2024-04-02T13:32:16.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.680+0000 7f5bdad3a640 1 --2- 172.21.15.44:0/577321975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bc0004d40 0x7f5bc00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:32:16.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.680+0000 7f5bda539640 1 --2- 172.21.15.44:0/577321975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 0x7f5bc00039e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5bc800c240 tx=0x7f5bc0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2a69bad25d5ef92 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:16.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.681+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8007d30 con 0x7f5bd40b4180 2024-04-02T13:32:16.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.681+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bc8030070 con 0x7f5bd40b4180 2024-04-02T13:32:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.681+0000 7f5bd9537640 1 -- 172.21.15.44:0/577321975 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc802cda0 con 0x7f5bd40b4180 2024-04-02T13:32:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.681+0000 7f5bdc79d640 1 -- 172.21.15.44:0/577321975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 msgr2=0x7f5bc00039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.681+0000 7f5bdc79d640 1 --2- 172.21.15.44:0/577321975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 0x7f5bc00039e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5bc800c240 tx=0x7f5bc0005830 comp rx=0 tx=0).stop 2024-04-02T13:32:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.681+0000 7f5bdad3a640 1 -- 172.21.15.44:0/577321975 reap_dead start 2024-04-02T13:32:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.681+0000 7f5bdc79d640 1 -- 172.21.15.44:0/577321975 shutdown_connections 2024-04-02T13:32:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.681+0000 7f5bdc79d640 1 -- 172.21.15.44:0/577321975 >> 172.21.15.44:0/577321975 conn(0x7f5bd409aa20 msgr2=0x7f5bd409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.681+0000 7f5bdc79d640 1 -- 172.21.15.44:0/577321975 shutdown_connections 2024-04-02T13:32:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.681+0000 7f5bdc79d640 1 -- 172.21.15.44:0/577321975 wait complete. 2024-04-02T13:32:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.682+0000 7f5bdc79d640 1 Processor -- start 2024-04-02T13:32:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.682+0000 7f5bdc79d640 1 -- start start 2024-04-02T13:32:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.683+0000 7f5bdc79d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 0x7f5bd4155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.683+0000 7f5bdc79d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bd4155fa0 0x7f5bd4160bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.683+0000 7f5bdc79d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bd4156400 0x7f5bd4161100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.683+0000 7f5bdc79d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd40b2ed0 con 0x7f5bd4156400 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.683+0000 7f5bdc79d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd40b2d50 con 0x7f5bd4155fa0 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.683+0000 7f5bdc79d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5bd40b3050 con 0x7f5bd40b4180 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.683+0000 7f5bdad3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bd4156400 0x7f5bd4161100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.683+0000 7f5bdad3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bd4156400 0x7f5bd4161100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42284/0 (socket says 172.21.15.44:42284) 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.683+0000 7f5bdad3a640 1 -- 172.21.15.44:0/2522311027 learned_addr learned my addr 172.21.15.44:0/2522311027 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.683+0000 7f5bda539640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 0x7f5bd4155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.683+0000 7f5bd9d38640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bd4155fa0 0x7f5bd4160bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.684+0000 7f5bdad3a640 1 -- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 msgr2=0x7f5bd4155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.684+0000 7f5bdad3a640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 0x7f5bd4155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.684+0000 7f5bdad3a640 1 -- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bd4155fa0 msgr2=0x7f5bd4160bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.684+0000 7f5bdad3a640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bd4155fa0 0x7f5bd4160bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.684+0000 7f5bdad3a640 1 -- 172.21.15.44:0/2522311027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd41569c0 con 0x7f5bd4156400 2024-04-02T13:32:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.684+0000 7f5bdad3a640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bd4156400 0x7f5bd4161100 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f5bd40b45d0 tx=0x7f5bcc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.684+0000 7f5bc6ffd640 1 -- 172.21.15.44:0/2522311027 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc009c10 con 0x7f5bd4156400 2024-04-02T13:32:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.684+0000 7f5bc6ffd640 1 -- 172.21.15.44:0/2522311027 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bcc00fe80 con 0x7f5bd4156400 2024-04-02T13:32:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.684+0000 7f5bdc79d640 1 -- 172.21.15.44:0/2522311027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd4156ca0 con 0x7f5bd4156400 2024-04-02T13:32:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.684+0000 7f5bc6ffd640 1 -- 172.21.15.44:0/2522311027 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc003bb0 con 0x7f5bd4156400 2024-04-02T13:32:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.684+0000 7f5bdc79d640 1 -- 172.21.15.44:0/2522311027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd415eb10 con 0x7f5bd4156400 2024-04-02T13:32:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.685+0000 7f5bdc79d640 1 -- 172.21.15.44:0/2522311027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bd415ed10 con 0x7f5bd4156400 2024-04-02T13:32:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.686+0000 7f5bc6ffd640 1 -- 172.21.15.44:0/2522311027 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5bcc01f020 con 0x7f5bd4156400 2024-04-02T13:32:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.686+0000 7f5bc6ffd640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5bb806c8f0 0x7f5bb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.686+0000 7f5bc6ffd640 1 -- 172.21.15.44:0/2522311027 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5921+0+0 (secure 0 0 0) 0x7f5bcc08bd80 con 0x7f5bd4156400 2024-04-02T13:32:16.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.686+0000 7f5bda539640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5bb806c8f0 0x7f5bb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:16.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.687+0000 7f5bda539640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5bb806c8f0 0x7f5bb806edb0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8034000 tx=0x7f5bc8009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:16.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.690+0000 7f5bc6ffd640 1 -- 172.21.15.44:0/2522311027 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5bd415ed10 con 0x7f5bd4156400 2024-04-02T13:32:16.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:16.876+0000 7f5bdc79d640 1 -- 172.21.15.44:0/2522311027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5bd40684f0 con 0x7f5bd4156400 2024-04-02T13:32:17.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.550+0000 7f5bc6ffd640 1 -- 172.21.15.44:0/2522311027 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v126) ==== 169+0+0 (secure 0 0 0) 0x7f5bcc010040 con 0x7f5bd4156400 2024-04-02T13:32:17.553 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' removed 2024-04-02T13:32:17.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.553+0000 7f5bdc79d640 1 -- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5bb806c8f0 msgr2=0x7f5bb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:17.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.553+0000 7f5bdc79d640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5bb806c8f0 0x7f5bb806edb0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8034000 tx=0x7f5bc8009030 comp rx=0 tx=0).stop 2024-04-02T13:32:17.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.553+0000 7f5bdc79d640 1 -- 172.21.15.44:0/2522311027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bd4156400 msgr2=0x7f5bd4161100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:17.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.553+0000 7f5bdc79d640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bd4156400 0x7f5bd4161100 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f5bd40b45d0 tx=0x7f5bcc00f580 comp rx=0 tx=0).stop 2024-04-02T13:32:17.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.553+0000 7f5bdc79d640 1 -- 172.21.15.44:0/2522311027 shutdown_connections 2024-04-02T13:32:17.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.553+0000 7f5bdc79d640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5bb806c8f0 0x7f5bb806edb0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:17.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.553+0000 7f5bdc79d640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bd4156400 0x7f5bd4161100 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:17.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.553+0000 7f5bdc79d640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bd4155fa0 0x7f5bd4160bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:17.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.553+0000 7f5bdc79d640 1 --2- 172.21.15.44:0/2522311027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd40b4180 0x7f5bd4155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:17.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.553+0000 7f5bdc79d640 1 -- 172.21.15.44:0/2522311027 >> 172.21.15.44:0/2522311027 conn(0x7f5bd409aa20 msgr2=0x7f5bd40d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:17.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.553+0000 7f5bdc79d640 1 -- 172.21.15.44:0/2522311027 shutdown_connections 2024-04-02T13:32:17.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.553+0000 7f5bdc79d640 1 -- 172.21.15.44:0/2522311027 wait complete. 2024-04-02T13:32:17.570 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T13:32:17.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.679+0000 7f7109281640 1 Processor -- start 2024-04-02T13:32:17.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.679+0000 7f7109281640 1 -- start start 2024-04-02T13:32:17.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f7109281640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f71040b1540 0x7f71040b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:17.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f7109281640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f71040ce790 0x7f710406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:17.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f7109281640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71040ced00 con 0x7f710415e470 2024-04-02T13:32:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f7109281640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f71040cefe0 con 0x7f71040b1540 2024-04-02T13:32:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f7109281640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f71040cf2c0 con 0x7f71040ce790 2024-04-02T13:32:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f7102ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f710415e470 0x7f71040d1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52316/0 (socket says 172.21.15.44:52316) 2024-04-02T13:32:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f7102ffd640 1 -- 172.21.15.44:0/945680776 learned_addr learned my addr 172.21.15.44:0/945680776 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f71027fc640 1 --2- 172.21.15.44:0/945680776 >> v2:172.21.15.44:3300/0 conn(0x7f71040b1540 0x7f71040b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f7101ffb640 1 -- 172.21.15.44:0/945680776 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2488008729 0 0) 0x7f71040ced00 con 0x7f710415e470 2024-04-02T13:32:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f7101ffb640 1 -- 172.21.15.44:0/945680776 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70f0003520 con 0x7f710415e470 2024-04-02T13:32:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f71037fe640 1 --2- 172.21.15.44:0/945680776 >> v2:172.21.15.69:3300/0 conn(0x7f71040ce790 0x7f710406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f71027fc640 1 -- 172.21.15.44:0/945680776 >> v2:172.21.15.69:3300/0 conn(0x7f71040ce790 msgr2=0x7f710406d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f71027fc640 1 --2- 172.21.15.44:0/945680776 >> v2:172.21.15.69:3300/0 conn(0x7f71040ce790 0x7f710406d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f71027fc640 1 -- 172.21.15.44:0/945680776 >> v1:172.21.15.44:6789/0 conn(0x7f710415e470 legacy=0x7f71040d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f71027fc640 1 -- 172.21.15.44:0/945680776 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71040cf5a0 con 0x7f71040b1540 2024-04-02T13:32:17.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.680+0000 7f71037fe640 1 --2- 172.21.15.44:0/945680776 >> v2:172.21.15.69:3300/0 conn(0x7f71040ce790 0x7f710406d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.681+0000 7f71027fc640 1 --2- 172.21.15.44:0/945680776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71040b1540 0x7f71040b32c0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f71040cc460 tx=0x7f70ec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4680ee405b30ed6a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.681+0000 7f7101ffb640 1 -- 172.21.15.44:0/945680776 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ec0315a0 con 0x7f71040b1540 2024-04-02T13:32:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.681+0000 7f7101ffb640 1 -- 172.21.15.44:0/945680776 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70ec037070 con 0x7f71040b1540 2024-04-02T13:32:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.681+0000 7f7101ffb640 1 -- 172.21.15.44:0/945680776 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ec031960 con 0x7f71040b1540 2024-04-02T13:32:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.682+0000 7f7109281640 1 -- 172.21.15.44:0/945680776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71040b1540 msgr2=0x7f71040b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.682+0000 7f7109281640 1 --2- 172.21.15.44:0/945680776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71040b1540 0x7f71040b32c0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f71040cc460 tx=0x7f70ec02b840 comp rx=0 tx=0).stop 2024-04-02T13:32:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.682+0000 7f7109281640 1 -- 172.21.15.44:0/945680776 shutdown_connections 2024-04-02T13:32:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.682+0000 7f7109281640 1 --2- 172.21.15.44:0/945680776 >> v2:172.21.15.69:3300/0 conn(0x7f71040ce790 0x7f710406d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.682+0000 7f7109281640 1 --2- 172.21.15.44:0/945680776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71040b1540 0x7f71040b32c0 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:17.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.682+0000 7f7109281640 1 -- 172.21.15.44:0/945680776 >> 172.21.15.44:0/945680776 conn(0x7f710409aa20 msgr2=0x7f710409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:17.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.682+0000 7f7109281640 1 -- 172.21.15.44:0/945680776 shutdown_connections 2024-04-02T13:32:17.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.682+0000 7f7109281640 1 -- 172.21.15.44:0/945680776 wait complete. 2024-04-02T13:32:17.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.683+0000 7f7109281640 1 Processor -- start 2024-04-02T13:32:17.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.683+0000 7f7109281640 1 -- start start 2024-04-02T13:32:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f7109281640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71040b1540 0x7f7104160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f7109281640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71040ce790 0x7f7104160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f7109281640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f710415e470 0x7f710415ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f7109281640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f71040d22c0 con 0x7f710415e470 2024-04-02T13:32:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f7109281640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f710406cb60 con 0x7f71040b1540 2024-04-02T13:32:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f7109281640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f71040d2440 con 0x7f71040ce790 2024-04-02T13:32:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f71037fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f710415e470 0x7f710415ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f71037fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f710415e470 0x7f710415ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42314/0 (socket says 172.21.15.44:42314) 2024-04-02T13:32:17.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f71037fe640 1 -- 172.21.15.44:0/2001279537 learned_addr learned my addr 172.21.15.44:0/2001279537 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f7102ffd640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71040b1540 0x7f7104160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f71037fe640 1 -- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71040ce790 msgr2=0x7f7104160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f71027fc640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71040ce790 0x7f7104160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f71037fe640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71040ce790 0x7f7104160ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f71037fe640 1 -- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71040b1540 msgr2=0x7f7104160ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f71037fe640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71040b1540 0x7f7104160ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f71037fe640 1 -- 172.21.15.44:0/2001279537 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7104152870 con 0x7f710415e470 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.684+0000 7f7102ffd640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71040b1540 0x7f7104160ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.685+0000 7f71037fe640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f710415e470 0x7f710415ea60 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f70f400cef0 tx=0x7f70f400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.685+0000 7f70fb7fe640 1 -- 172.21.15.44:0/2001279537 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f401a070 con 0x7f710415e470 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.685+0000 7f70fb7fe640 1 -- 172.21.15.44:0/2001279537 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70f400d860 con 0x7f710415e470 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.685+0000 7f7109281640 1 -- 172.21.15.44:0/2001279537 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f710415f770 con 0x7f710415e470 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.685+0000 7f70fb7fe640 1 -- 172.21.15.44:0/2001279537 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f400dc50 con 0x7f710415e470 2024-04-02T13:32:17.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.685+0000 7f7109281640 1 -- 172.21.15.44:0/2001279537 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f710415fd30 con 0x7f710415e470 2024-04-02T13:32:17.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.686+0000 7f7109281640 1 -- 172.21.15.44:0/2001279537 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71040684f0 con 0x7f710415e470 2024-04-02T13:32:17.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.686+0000 7f70fb7fe640 1 -- 172.21.15.44:0/2001279537 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f70f401d070 con 0x7f710415e470 2024-04-02T13:32:17.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.687+0000 7f70fb7fe640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f70e406c8f0 0x7f70e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:17.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.687+0000 7f70fb7fe640 1 -- 172.21.15.44:0/2001279537 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f70f408c3a0 con 0x7f710415e470 2024-04-02T13:32:17.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.687+0000 7f7102ffd640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f70e406c8f0 0x7f70e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:17.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.688+0000 7f7102ffd640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f70e406c8f0 0x7f70e406edb0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f70fc005880 tx=0x7f70fc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:17.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.691+0000 7f70fb7fe640 1 -- 172.21.15.44:0/2001279537 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f70f405a950 con 0x7f710415e470 2024-04-02T13:32:17.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.883+0000 7f7109281640 1 -- 172.21.15.44:0/2001279537 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f71040b7d40 con 0x7f710415e470 2024-04-02T13:32:17.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.886+0000 7f70fb7fe640 1 -- 172.21.15.44:0/2001279537 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) ==== 104+0+0 (secure 0 0 0) 0x7f70f405a250 con 0x7f710415e470 2024-04-02T13:32:17.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.888+0000 7f7109281640 1 -- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f70e406c8f0 msgr2=0x7f70e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:17.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.888+0000 7f7109281640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f70e406c8f0 0x7f70e406edb0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f70fc005880 tx=0x7f70fc009040 comp rx=0 tx=0).stop 2024-04-02T13:32:17.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.888+0000 7f7109281640 1 -- 172.21.15.44:0/2001279537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f710415e470 msgr2=0x7f710415ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:17.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.888+0000 7f7109281640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f710415e470 0x7f710415ea60 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f70f400cef0 tx=0x7f70f400a710 comp rx=0 tx=0).stop 2024-04-02T13:32:17.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.888+0000 7f7109281640 1 -- 172.21.15.44:0/2001279537 shutdown_connections 2024-04-02T13:32:17.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.888+0000 7f7109281640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f70e406c8f0 0x7f70e406edb0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:17.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.888+0000 7f7109281640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f710415e470 0x7f710415ea60 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:17.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.888+0000 7f7109281640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71040ce790 0x7f7104160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:17.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.888+0000 7f7109281640 1 --2- 172.21.15.44:0/2001279537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71040b1540 0x7f7104160ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:17.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.888+0000 7f7109281640 1 -- 172.21.15.44:0/2001279537 >> 172.21.15.44:0/2001279537 conn(0x7f710409aa20 msgr2=0x7f71040aeda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:17.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.888+0000 7f7109281640 1 -- 172.21.15.44:0/2001279537 shutdown_connections 2024-04-02T13:32:17.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:17.888+0000 7f7109281640 1 -- 172.21.15.44:0/2001279537 wait complete. 2024-04-02T13:32:17.902 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan' 2024-04-02T13:32:18.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.006+0000 7fbc21932640 1 Processor -- start 2024-04-02T13:32:18.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.006+0000 7fbc21932640 1 -- start start 2024-04-02T13:32:18.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc21932640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbc1c0d0c50 0x7fbc1c0d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:18.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc21932640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbc1c06ce40 0x7fbc1c06d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:18.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc21932640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc1c0cc600 con 0x7fbc1c15e4a0 2024-04-02T13:32:18.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc21932640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fbc1c0af300 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc21932640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbc1c0cc8e0 con 0x7fbc1c06ce40 2024-04-02T13:32:18.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc13fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fbc1c0d0c50 0x7fbc1c0d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:18.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc1affd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbc1c15e4a0 0x7fbc1c0aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52324/0 (socket says 172.21.15.44:52324) 2024-04-02T13:32:18.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc1affd640 1 -- 172.21.15.44:0/1189769553 learned_addr learned my addr 172.21.15.44:0/1189769553 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:18.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc1b7fe640 1 --2- 172.21.15.44:0/1189769553 >> v2:172.21.15.69:3300/0 conn(0x7fbc1c06ce40 0x7fbc1c06d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:18.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc13fff640 1 -- 172.21.15.44:0/1189769553 >> v2:172.21.15.69:3300/0 conn(0x7fbc1c06ce40 msgr2=0x7fbc1c06d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:18.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc13fff640 1 --2- 172.21.15.44:0/1189769553 >> v2:172.21.15.69:3300/0 conn(0x7fbc1c06ce40 0x7fbc1c06d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:18.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc13fff640 1 -- 172.21.15.44:0/1189769553 >> v1:172.21.15.44:6789/0 conn(0x7fbc1c15e4a0 legacy=0x7fbc1c0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:18.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc13fff640 1 -- 172.21.15.44:0/1189769553 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c0ccbc0 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.007+0000 7fbc1a7fc640 1 -- 172.21.15.44:0/1189769553 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2870813507 0 0) 0x7fbc1c0cc600 con 0x7fbc1c15e4a0 2024-04-02T13:32:18.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.008+0000 7fbc13fff640 1 --2- 172.21.15.44:0/1189769553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc1c0d0c50 0x7fbc1c0d10b0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c06da20 tx=0x7fbc0402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2d429fa42f9d61 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:18.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.008+0000 7fbc1a7fc640 1 -- 172.21.15.44:0/1189769553 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04035020 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.008+0000 7fbc1a7fc640 1 -- 172.21.15.44:0/1189769553 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc04031830 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.008+0000 7fbc1a7fc640 1 -- 172.21.15.44:0/1189769553 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04031c00 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.009+0000 7fbc21932640 1 -- 172.21.15.44:0/1189769553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc1c0d0c50 msgr2=0x7fbc1c0d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:18.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.009+0000 7fbc21932640 1 --2- 172.21.15.44:0/1189769553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc1c0d0c50 0x7fbc1c0d10b0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c06da20 tx=0x7fbc0402b8d0 comp rx=0 tx=0).stop 2024-04-02T13:32:18.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.009+0000 7fbc21932640 1 -- 172.21.15.44:0/1189769553 shutdown_connections 2024-04-02T13:32:18.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.009+0000 7fbc21932640 1 --2- 172.21.15.44:0/1189769553 >> v2:172.21.15.69:3300/0 conn(0x7fbc1c06ce40 0x7fbc1c06d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:18.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.009+0000 7fbc21932640 1 --2- 172.21.15.44:0/1189769553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc1c0d0c50 0x7fbc1c0d10b0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:18.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.009+0000 7fbc21932640 1 -- 172.21.15.44:0/1189769553 >> 172.21.15.44:0/1189769553 conn(0x7fbc1c09aa20 msgr2=0x7fbc1c09ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:18.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.009+0000 7fbc21932640 1 -- 172.21.15.44:0/1189769553 shutdown_connections 2024-04-02T13:32:18.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.009+0000 7fbc21932640 1 -- 172.21.15.44:0/1189769553 wait complete. 2024-04-02T13:32:18.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.010+0000 7fbc21932640 1 Processor -- start 2024-04-02T13:32:18.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.010+0000 7fbc21932640 1 -- start start 2024-04-02T13:32:18.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.010+0000 7fbc21932640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc1c06ce40 0x7fbc1c0caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:18.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.010+0000 7fbc21932640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc1c0d0c50 0x7fbc1c1513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.010+0000 7fbc21932640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc1c15e4a0 0x7fbc1c0ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.010+0000 7fbc21932640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c0cd460 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.010+0000 7fbc21932640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c0cd180 con 0x7fbc1c15e4a0 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.010+0000 7fbc21932640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbc1c15e880 con 0x7fbc1c06ce40 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc13fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc1c0d0c50 0x7fbc1c1513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc13fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc1c0d0c50 0x7fbc1c1513b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42334/0 (socket says 172.21.15.44:42334) 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc13fff640 1 -- 172.21.15.44:0/3981286843 learned_addr learned my addr 172.21.15.44:0/3981286843 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc1b7fe640 1 --2- 172.21.15.44:0/3981286843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc1c15e4a0 0x7fbc1c0ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc13fff640 1 -- 172.21.15.44:0/3981286843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc1c06ce40 msgr2=0x7fbc1c0caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc13fff640 1 --2- 172.21.15.44:0/3981286843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc1c06ce40 0x7fbc1c0caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc13fff640 1 -- 172.21.15.44:0/3981286843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc1c15e4a0 msgr2=0x7fbc1c0ca140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc13fff640 1 --2- 172.21.15.44:0/3981286843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc1c15e4a0 0x7fbc1c0ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc13fff640 1 -- 172.21.15.44:0/3981286843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c1f73f0 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc13fff640 1 --2- 172.21.15.44:0/3981286843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc1c0d0c50 0x7fbc1c1513b0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fbc04000c00 tx=0x7fbc040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:18.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc18ff9640 1 -- 172.21.15.44:0/3981286843 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04035070 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc18ff9640 1 -- 172.21.15.44:0/3981286843 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc04004250 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc21932640 1 -- 172.21.15.44:0/3981286843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc1c1f76d0 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc18ff9640 1 -- 172.21.15.44:0/3981286843 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0403a6f0 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.011+0000 7fbc21932640 1 -- 172.21.15.44:0/3981286843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc1c1f7c90 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.012+0000 7fbc21932640 1 -- 172.21.15.44:0/3981286843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc1c0684f0 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.013+0000 7fbc18ff9640 1 -- 172.21.15.44:0/3981286843 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbc04038020 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.013+0000 7fbc18ff9640 1 --2- 172.21.15.44:0/3981286843 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbbfc06c8f0 0x7fbbfc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:18.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.013+0000 7fbc18ff9640 1 -- 172.21.15.44:0/3981286843 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7fbc040ad750 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.013+0000 7fbc1affd640 1 --2- 172.21.15.44:0/3981286843 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbbfc06c8f0 0x7fbbfc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:18.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.014+0000 7fbc1affd640 1 --2- 172.21.15.44:0/3981286843 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbbfc06c8f0 0x7fbbfc06edb0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c007fd0 tx=0x7fbc0c007960 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:18.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.017+0000 7fbc18ff9640 1 -- 172.21.15.44:0/3981286843 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fbc04040070 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.275+0000 7fbc21932640 1 -- 172.21.15.44:0/3981286843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]} v 0) -- 0x7fbc1c0b0fb0 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.552+0000 7fbc18ff9640 1 -- 172.21.15.44:0/3981286843 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]}]=0 v563) ==== 144+0+0 (secure 0 0 0) 0x7fbc04031800 con 0x7fbc1c0d0c50 2024-04-02T13:32:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.555+0000 7fbc21932640 1 -- 172.21.15.44:0/3981286843 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbbfc06c8f0 msgr2=0x7fbbfc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.555+0000 7fbc21932640 1 --2- 172.21.15.44:0/3981286843 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbbfc06c8f0 0x7fbbfc06edb0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c007fd0 tx=0x7fbc0c007960 comp rx=0 tx=0).stop 2024-04-02T13:32:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.555+0000 7fbc21932640 1 -- 172.21.15.44:0/3981286843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc1c0d0c50 msgr2=0x7fbc1c1513b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.555+0000 7fbc21932640 1 --2- 172.21.15.44:0/3981286843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc1c0d0c50 0x7fbc1c1513b0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fbc04000c00 tx=0x7fbc040046c0 comp rx=0 tx=0).stop 2024-04-02T13:32:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.555+0000 7fbc21932640 1 -- 172.21.15.44:0/3981286843 shutdown_connections 2024-04-02T13:32:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.555+0000 7fbc21932640 1 --2- 172.21.15.44:0/3981286843 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fbbfc06c8f0 0x7fbbfc06edb0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.555+0000 7fbc21932640 1 --2- 172.21.15.44:0/3981286843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc1c15e4a0 0x7fbc1c0ca140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.555+0000 7fbc21932640 1 --2- 172.21.15.44:0/3981286843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc1c0d0c50 0x7fbc1c1513b0 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.555+0000 7fbc21932640 1 --2- 172.21.15.44:0/3981286843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc1c06ce40 0x7fbc1c0caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.555+0000 7fbc21932640 1 -- 172.21.15.44:0/3981286843 >> 172.21.15.44:0/3981286843 conn(0x7fbc1c09aa20 msgr2=0x7fbc1c06f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.555+0000 7fbc21932640 1 -- 172.21.15.44:0/3981286843 shutdown_connections 2024-04-02T13:32:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:18.555+0000 7fbc21932640 1 -- 172.21.15.44:0/3981286843 wait complete. 2024-04-02T13:32:18.569 INFO:tasks.cephfs_test_runner:test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-04-02T13:32:18.569 INFO:tasks.cephfs_test_runner: 2024-04-02T13:32:18.570 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-02T13:32:18.570 INFO:tasks.cephfs_test_runner:Ran 7 tests in 669.397s 2024-04-02T13:32:18.570 INFO:tasks.cephfs_test_runner: 2024-04-02T13:32:18.570 INFO:tasks.cephfs_test_runner:OK 2024-04-02T13:32:18.570 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-02T13:32:18.580 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-02T13:32:18.598 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-02T13:32:18.599 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:32:18.599 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:32:18.614 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:32:18.614 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-02T13:32:18.615 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:32:18.669 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17691 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498936k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-02T13:32:18.669 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:32:18.669 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:32:18.688 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:32:18.688 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T13:32:18.689 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T13:32:18.746 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056330,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-02T13:32:18.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:18.746 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-02T13:32:18.803 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-02T13:32:18.803 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-02T13:32:18.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:18.803 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-02T13:32:18.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:18.873 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T13:32:18.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:18.943 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link delete ceph-brx 2024-04-02T13:32:19.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:19.041 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-02T13:32:19.057 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T13:32:19.057 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T13:32:19.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:19.057 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T13:32:19.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:19.126 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T13:32:19.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T13:32:19.195 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link delete ceph-brx 2024-04-02T13:32:19.296 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-02T13:32:19.307 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-02T13:32:19.307 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:32:19.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.408+0000 7f753e90e640 1 Processor -- start 2024-04-02T13:32:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.408+0000 7f753e90e640 1 -- start start 2024-04-02T13:32:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.409+0000 7f753e90e640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7538159db0 0x7f753815c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.409+0000 7f753e90e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75380726c0 con 0x7f75380ccdf0 2024-04-02T13:32:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.409+0000 7f753e90e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75380729a0 con 0x7f7538071e50 2024-04-02T13:32:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.409+0000 7f753e90e640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f7538073240 con 0x7f7538159db0 2024-04-02T13:32:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.409+0000 7f7537fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7538159db0 0x7f753815c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.409+0000 7f753ceab640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f75380ccdf0 0x7f7538165cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52340/0 (socket says 172.21.15.44:52340) 2024-04-02T13:32:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.409+0000 7f7537fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f7538159db0 0x7f753815c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42350/0 (socket says 172.21.15.44:42350) 2024-04-02T13:32:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.409+0000 7f753ceab640 1 -- 172.21.15.44:0/1580535678 learned_addr learned my addr 172.21.15.44:0/1580535678 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.409+0000 7f7537fff640 1 -- 172.21.15.44:0/1580535678 >> v1:172.21.15.69:6789/0 conn(0x7f7538071e50 legacy=0x7f75380cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:19.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.409+0000 7f7537fff640 1 -- 172.21.15.44:0/1580535678 >> v1:172.21.15.44:6789/0 conn(0x7f75380ccdf0 legacy=0x7f7538165cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:19.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.409+0000 7f7537fff640 1 -- 172.21.15.44:0/1580535678 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7538072c80 con 0x7f7538159db0 2024-04-02T13:32:19.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.410+0000 7f7537fff640 1 --2- 172.21.15.44:0/1580535678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7538159db0 0x7f753815c6d0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f75280097a0 tx=0x7f752802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb5568981b72636 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:19.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.410+0000 7f7536ffd640 1 -- 172.21.15.44:0/1580535678 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7528035020 con 0x7f7538159db0 2024-04-02T13:32:19.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.410+0000 7f7536ffd640 1 -- 172.21.15.44:0/1580535678 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7528031830 con 0x7f7538159db0 2024-04-02T13:32:19.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.410+0000 7f7536ffd640 1 -- 172.21.15.44:0/1580535678 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7528031c00 con 0x7f7538159db0 2024-04-02T13:32:19.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.411+0000 7f753e90e640 1 -- 172.21.15.44:0/1580535678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7538159db0 msgr2=0x7f753815c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:19.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.411+0000 7f753e90e640 1 --2- 172.21.15.44:0/1580535678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7538159db0 0x7f753815c6d0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f75280097a0 tx=0x7f752802b8d0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.411+0000 7f753e90e640 1 -- 172.21.15.44:0/1580535678 shutdown_connections 2024-04-02T13:32:19.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.411+0000 7f753e90e640 1 --2- 172.21.15.44:0/1580535678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7538159db0 0x7f753815c6d0 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.411+0000 7f753e90e640 1 -- 172.21.15.44:0/1580535678 >> 172.21.15.44:0/1580535678 conn(0x7f75380cfcd0 msgr2=0x7f75380d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:19.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.411+0000 7f753e90e640 1 -- 172.21.15.44:0/1580535678 shutdown_connections 2024-04-02T13:32:19.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.411+0000 7f753e90e640 1 -- 172.21.15.44:0/1580535678 wait complete. 2024-04-02T13:32:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.412+0000 7f753e90e640 1 Processor -- start 2024-04-02T13:32:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.412+0000 7f753e90e640 1 -- start start 2024-04-02T13:32:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f753e90e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7538071e50 0x7f753815bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f753e90e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75380ccdf0 0x7f75381514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f753e90e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7538159db0 0x7f7538152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f753e90e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7538072480 con 0x7f75380ccdf0 2024-04-02T13:32:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f753e90e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7538072300 con 0x7f7538159db0 2024-04-02T13:32:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f753e90e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f753806e650 con 0x7f7538071e50 2024-04-02T13:32:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f75377fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75380ccdf0 0x7f75381514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f75377fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75380ccdf0 0x7f75381514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42356/0 (socket says 172.21.15.44:42356) 2024-04-02T13:32:19.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f75377fe640 1 -- 172.21.15.44:0/4041253831 learned_addr learned my addr 172.21.15.44:0/4041253831 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f753ceab640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7538159db0 0x7f7538152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f7537fff640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7538071e50 0x7f753815bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f75377fe640 1 -- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7538071e50 msgr2=0x7f753815bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f75377fe640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7538071e50 0x7f753815bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f75377fe640 1 -- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7538159db0 msgr2=0x7f7538152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f75377fe640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7538159db0 0x7f7538152fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f75377fe640 1 -- 172.21.15.44:0/4041253831 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7538153ef0 con 0x7f75380ccdf0 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f753ceab640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7538159db0 0x7f7538152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.413+0000 7f75377fe640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75380ccdf0 0x7f75381514f0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f752c00c400 tx=0x7f752c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.414+0000 7f7534ff9640 1 -- 172.21.15.44:0/4041253831 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c0052d0 con 0x7f75380ccdf0 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.414+0000 7f7534ff9640 1 -- 172.21.15.44:0/4041253831 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f752c005490 con 0x7f75380ccdf0 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.414+0000 7f753e90e640 1 -- 172.21.15.44:0/4041253831 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75381541d0 con 0x7f75380ccdf0 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.414+0000 7f7534ff9640 1 -- 172.21.15.44:0/4041253831 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c007260 con 0x7f75380ccdf0 2024-04-02T13:32:19.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.414+0000 7f753e90e640 1 -- 172.21.15.44:0/4041253831 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7538154790 con 0x7f75380ccdf0 2024-04-02T13:32:19.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.415+0000 7f753e90e640 1 -- 172.21.15.44:0/4041253831 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7538074f20 con 0x7f75380ccdf0 2024-04-02T13:32:19.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.415+0000 7f7534ff9640 1 -- 172.21.15.44:0/4041253831 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f752c007400 con 0x7f75380ccdf0 2024-04-02T13:32:19.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.415+0000 7f7534ff9640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f752406c8f0 0x7f752406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.415+0000 7f7534ff9640 1 -- 172.21.15.44:0/4041253831 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f752c08afa0 con 0x7f75380ccdf0 2024-04-02T13:32:19.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.416+0000 7f7537fff640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f752406c8f0 0x7f752406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.416+0000 7f7537fff640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f752406c8f0 0x7f752406edb0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f7528036000 tx=0x7f752803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:19.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.420+0000 7f7534ff9640 1 -- 172.21.15.44:0/4041253831 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f752c059490 con 0x7f75380ccdf0 2024-04-02T13:32:19.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.603+0000 7f753e90e640 1 -- 172.21.15.44:0/4041253831 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f753815c090 con 0x7f752406c8f0 2024-04-02T13:32:19.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.604+0000 7f7534ff9640 1 -- 172.21.15.44:0/4041253831 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13505 (secure 0 0 0) 0x7f753815c090 con 0x7f752406c8f0 2024-04-02T13:32:19.607 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:19.607 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:32:19.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.606+0000 7f753e90e640 1 -- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f752406c8f0 msgr2=0x7f752406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:19.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.606+0000 7f753e90e640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f752406c8f0 0x7f752406edb0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f7528036000 tx=0x7f752803c040 comp rx=0 tx=0).stop 2024-04-02T13:32:19.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.606+0000 7f753e90e640 1 -- 172.21.15.44:0/4041253831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75380ccdf0 msgr2=0x7f75381514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:19.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.606+0000 7f753e90e640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75380ccdf0 0x7f75381514f0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f752c00c400 tx=0x7f752c00c8c0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.606+0000 7f753e90e640 1 -- 172.21.15.44:0/4041253831 shutdown_connections 2024-04-02T13:32:19.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.606+0000 7f753e90e640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f752406c8f0 0x7f752406edb0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.606+0000 7f753e90e640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7538159db0 0x7f7538152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.606+0000 7f753e90e640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75380ccdf0 0x7f75381514f0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.606+0000 7f753e90e640 1 --2- 172.21.15.44:0/4041253831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7538071e50 0x7f753815bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.606+0000 7f753e90e640 1 -- 172.21.15.44:0/4041253831 >> 172.21.15.44:0/4041253831 conn(0x7f75380cfcd0 msgr2=0x7f75380ce4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:19.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.606+0000 7f753e90e640 1 -- 172.21.15.44:0/4041253831 shutdown_connections 2024-04-02T13:32:19.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.606+0000 7f753e90e640 1 -- 172.21.15.44:0/4041253831 wait complete. 2024-04-02T13:32:19.619 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-04-02T13:32:19.054704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":262924,"kb_used_data":13480,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711796,"statfs":{"total":773094113280,"available":772824879104,"internally_reserved":0,"allocated":13803520,"data_stored":7075824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390584,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-264,"num_write":-176,"num_write_kb":-8295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001469"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":376,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-02T13:32:17.553383+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:32:17.553383+0000","last_peered":"2024-04-02T13:32:17.553383+0000","last_clean":"2024-04-02T13:32:17.553383+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:32:17.553383+0000","last_undegraded":"2024-04-02T13:32:17.553383+0000","last_fullsized":"2024-04-02T13:32:17.553383+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574996,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2170880,"data_stored":1327254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574995,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574996,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574996,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574996,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574995,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574996,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2170880,"data_stored":1327254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:32:19.620 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:32:19.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.718+0000 7f596acf2640 1 Processor -- start 2024-04-02T13:32:19.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.718+0000 7f596acf2640 1 -- start start 2024-04-02T13:32:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.719+0000 7f596acf2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415e4a0 0x7f59640aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.719+0000 7f596acf2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f59640d0c50 0x7f59640d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.719+0000 7f596acf2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59640cc600 con 0x7f596406ce40 2024-04-02T13:32:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.719+0000 7f596acf2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f59640cc8e0 con 0x7f59640d0c50 2024-04-02T13:32:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.719+0000 7f596acf2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f59640ccbc0 con 0x7f596415e4a0 2024-04-02T13:32:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.719+0000 7f596928f640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f596406ce40 0x7f59640cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52348/0 (socket says 172.21.15.44:52348) 2024-04-02T13:32:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.719+0000 7f596928f640 1 -- 172.21.15.44:0/2936230597 learned_addr learned my addr 172.21.15.44:0/2936230597 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.719+0000 7f5968a8e640 1 --2- 172.21.15.44:0/2936230597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415e4a0 0x7f59640aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.719+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2001613068 0 0) 0x7f59640cc600 con 0x7f596406ce40 2024-04-02T13:32:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.719+0000 7f5963fff640 1 --2- 172.21.15.44:0/2936230597 >> v2:172.21.15.69:3300/0 conn(0x7f59640d0c50 0x7f59640d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.719+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5954003520 con 0x7f596406ce40 2024-04-02T13:32:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2084232923 0 0) 0x7f5954003520 con 0x7f596406ce40 2024-04-02T13:32:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415e4a0 msgr2=0x7f59640aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 --2- 172.21.15.44:0/2936230597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415e4a0 0x7f59640aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 >> v2:172.21.15.69:3300/0 conn(0x7f59640d0c50 msgr2=0x7f59640d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 --2- 172.21.15.44:0/2936230597 >> v2:172.21.15.69:3300/0 conn(0x7f59640d0c50 0x7f59640d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59640ccea0 con 0x7f596406ce40 2024-04-02T13:32:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f595c0019f0 con 0x7f596406ce40 2024-04-02T13:32:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 >> v1:172.21.15.44:6789/0 conn(0x7f596406ce40 legacy=0x7f59640cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 --2- 172.21.15.44:0/2936230597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415e4a0 0x7f59540039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 --2- 172.21.15.44:0/2936230597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5954003f30 0x7f5954004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 --2- 172.21.15.44:0/2936230597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5954004d40 0x7f59540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f59640cc8e0 con 0x7f5954003f30 2024-04-02T13:32:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f59640ccbc0 con 0x7f5954004d40 2024-04-02T13:32:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f595400b9c0 con 0x7f596415e4a0 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f5963fff640 1 --2- 172.21.15.44:0/2936230597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5954003f30 0x7f5954004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f5968a8e640 1 --2- 172.21.15.44:0/2936230597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415e4a0 0x7f59540039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f596928f640 1 --2- 172.21.15.44:0/2936230597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5954004d40 0x7f59540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f5963fff640 1 -- 172.21.15.44:0/2936230597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415e4a0 msgr2=0x7f59540039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f5963fff640 1 --2- 172.21.15.44:0/2936230597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415e4a0 0x7f59540039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f5963fff640 1 -- 172.21.15.44:0/2936230597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5954004d40 msgr2=0x7f59540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f5963fff640 1 --2- 172.21.15.44:0/2936230597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5954004d40 0x7f59540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.720+0000 7f5963fff640 1 -- 172.21.15.44:0/2936230597 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5954003520 con 0x7f5954003f30 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.721+0000 7f5963fff640 1 --2- 172.21.15.44:0/2936230597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5954003f30 0x7f5954004800 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f5950009ac0 tx=0x7f5954005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93b4e10a41ab33b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.721+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5950039070 con 0x7f5954003f30 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.721+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f595002c6d0 con 0x7f5954003f30 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.721+0000 7f59637fe640 1 -- 172.21.15.44:0/2936230597 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5950009690 con 0x7f5954003f30 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.722+0000 7f596acf2640 1 -- 172.21.15.44:0/2936230597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5954003f30 msgr2=0x7f5954004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.722+0000 7f596acf2640 1 --2- 172.21.15.44:0/2936230597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5954003f30 0x7f5954004800 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f5950009ac0 tx=0x7f5954005830 comp rx=0 tx=0).stop 2024-04-02T13:32:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.722+0000 7f596928f640 1 -- 172.21.15.44:0/2936230597 reap_dead start 2024-04-02T13:32:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.722+0000 7f596acf2640 1 -- 172.21.15.44:0/2936230597 shutdown_connections 2024-04-02T13:32:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.722+0000 7f596acf2640 1 -- 172.21.15.44:0/2936230597 >> 172.21.15.44:0/2936230597 conn(0x7f596409aa20 msgr2=0x7f596409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.722+0000 7f596acf2640 1 -- 172.21.15.44:0/2936230597 shutdown_connections 2024-04-02T13:32:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.722+0000 7f596acf2640 1 -- 172.21.15.44:0/2936230597 wait complete. 2024-04-02T13:32:19.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.723+0000 7f596acf2640 1 Processor -- start 2024-04-02T13:32:19.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.723+0000 7f596acf2640 1 -- start start 2024-04-02T13:32:19.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f596acf2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5954003f30 0x7f596415afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f596acf2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415a0a0 0x7f5964161390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f596acf2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f596415c680 0x7f596415cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f596acf2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f596415e880 con 0x7f5954003f30 2024-04-02T13:32:19.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f596acf2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f59640cd460 con 0x7f596415c680 2024-04-02T13:32:19.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f596acf2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f59640ce090 con 0x7f596415a0a0 2024-04-02T13:32:19.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f5968a8e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5954003f30 0x7f596415afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f5968a8e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5954003f30 0x7f596415afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42372/0 (socket says 172.21.15.44:42372) 2024-04-02T13:32:19.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f5968a8e640 1 -- 172.21.15.44:0/564097733 learned_addr learned my addr 172.21.15.44:0/564097733 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f596928f640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f596415c680 0x7f596415cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f5963fff640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415a0a0 0x7f5964161390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f5968a8e640 1 -- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415a0a0 msgr2=0x7f5964161390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f5968a8e640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415a0a0 0x7f5964161390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f5968a8e640 1 -- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f596415c680 msgr2=0x7f596415cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f5968a8e640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f596415c680 0x7f596415cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f5968a8e640 1 -- 172.21.15.44:0/564097733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f596415f760 con 0x7f5954003f30 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f5963fff640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415a0a0 0x7f5964161390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f5968a8e640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5954003f30 0x7f596415afe0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f5958011a00 tx=0x7f5958011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f59617fa640 1 -- 172.21.15.44:0/564097733 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595800fb50 con 0x7f5954003f30 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f59617fa640 1 -- 172.21.15.44:0/564097733 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f595800fcf0 con 0x7f5954003f30 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f596acf2640 1 -- 172.21.15.44:0/564097733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59640cb9f0 con 0x7f5954003f30 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.724+0000 7f59617fa640 1 -- 172.21.15.44:0/564097733 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595800fea0 con 0x7f5954003f30 2024-04-02T13:32:19.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.725+0000 7f596acf2640 1 -- 172.21.15.44:0/564097733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59640cb430 con 0x7f5954003f30 2024-04-02T13:32:19.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.725+0000 7f596acf2640 1 -- 172.21.15.44:0/564097733 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59640684f0 con 0x7f5954003f30 2024-04-02T13:32:19.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.726+0000 7f59617fa640 1 -- 172.21.15.44:0/564097733 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5958012040 con 0x7f5954003f30 2024-04-02T13:32:19.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.726+0000 7f59617fa640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f594806c8f0 0x7f594806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:19.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.726+0000 7f59617fa640 1 -- 172.21.15.44:0/564097733 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f595808c0f0 con 0x7f5954003f30 2024-04-02T13:32:19.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.727+0000 7f5963fff640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f594806c8f0 0x7f594806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:19.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.727+0000 7f5963fff640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f594806c8f0 0x7f594806edb0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5950036000 tx=0x7f5950030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:19.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.731+0000 7f59617fa640 1 -- 172.21.15.44:0/564097733 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f5958019030 con 0x7f5954003f30 2024-04-02T13:32:19.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.912+0000 7f596acf2640 1 -- 172.21.15.44:0/564097733 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f59640b7e40 con 0x7f594806c8f0 2024-04-02T13:32:19.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.913+0000 7f59617fa640 1 -- 172.21.15.44:0/564097733 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13505 (secure 0 0 0) 0x7f59640b7e40 con 0x7f594806c8f0 2024-04-02T13:32:19.915 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:19.916 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:32:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.914+0000 7f596acf2640 1 -- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f594806c8f0 msgr2=0x7f594806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:19.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.914+0000 7f596acf2640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f594806c8f0 0x7f594806edb0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5950036000 tx=0x7f5950030000 comp rx=0 tx=0).stop 2024-04-02T13:32:19.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.914+0000 7f596acf2640 1 -- 172.21.15.44:0/564097733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5954003f30 msgr2=0x7f596415afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:19.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.914+0000 7f596acf2640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5954003f30 0x7f596415afe0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f5958011a00 tx=0x7f5958011ec0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.914+0000 7f596acf2640 1 -- 172.21.15.44:0/564097733 shutdown_connections 2024-04-02T13:32:19.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.914+0000 7f596acf2640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f594806c8f0 0x7f594806edb0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.914+0000 7f596acf2640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f596415c680 0x7f596415cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.914+0000 7f596acf2640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415a0a0 0x7f5964161390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.914+0000 7f596acf2640 1 --2- 172.21.15.44:0/564097733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5954003f30 0x7f596415afe0 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:19.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.914+0000 7f596acf2640 1 -- 172.21.15.44:0/564097733 >> 172.21.15.44:0/564097733 conn(0x7f596409aa20 msgr2=0x7f59640d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:19.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.915+0000 7f596acf2640 1 -- 172.21.15.44:0/564097733 shutdown_connections 2024-04-02T13:32:19.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:19.915+0000 7f596acf2640 1 -- 172.21.15.44:0/564097733 wait complete. 2024-04-02T13:32:19.928 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-04-02T13:32:19.054704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":262924,"kb_used_data":13480,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711796,"statfs":{"total":773094113280,"available":772824879104,"internally_reserved":0,"allocated":13803520,"data_stored":7075824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390584,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-264,"num_write":-176,"num_write_kb":-8295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001469"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":376,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-02T13:32:17.553383+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:32:17.553383+0000","last_peered":"2024-04-02T13:32:17.553383+0000","last_clean":"2024-04-02T13:32:17.553383+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:32:17.553383+0000","last_undegraded":"2024-04-02T13:32:17.553383+0000","last_fullsized":"2024-04-02T13:32:17.553383+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574996,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2170880,"data_stored":1327254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574995,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574996,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574996,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574996,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574995,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574996,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2170880,"data_stored":1327254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:32:19.928 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-02T13:32:19.928 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:32:20.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.027+0000 7fdccf263640 1 Processor -- start 2024-04-02T13:32:20.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.028+0000 7fdccf263640 1 -- start start 2024-04-02T13:32:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.028+0000 7fdccf263640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcc80cece0 0x7fdcc80cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.028+0000 7fdccf263640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdcc80d09f0 0x7fdcc806d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.028+0000 7fdccf263640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcc80cf600 con 0x7fdcc806ece0 2024-04-02T13:32:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.028+0000 7fdccf263640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fdcc806dc40 con 0x7fdcc80d09f0 2024-04-02T13:32:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.028+0000 7fdccf263640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdcc806df20 con 0x7fdcc80cece0 2024-04-02T13:32:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.029+0000 7fdcc7fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdcc80d09f0 0x7fdcc806d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.029+0000 7fdcc7fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fdcc80d09f0 0x7fdcc806d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42386/0 (socket says 172.21.15.44:42386) 2024-04-02T13:32:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.029+0000 7fdcc7fff640 1 -- 172.21.15.44:0/2231636017 learned_addr learned my addr 172.21.15.44:0/2231636017 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.029+0000 7fdcccfff640 1 --2- 172.21.15.44:0/2231636017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcc80cece0 0x7fdcc80cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.029+0000 7fdcc7fff640 1 -- 172.21.15.44:0/2231636017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcc80cece0 msgr2=0x7fdcc80cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.029+0000 7fdcc7fff640 1 --2- 172.21.15.44:0/2231636017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcc80cece0 0x7fdcc80cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.029+0000 7fdcc7fff640 1 -- 172.21.15.44:0/2231636017 >> v1:172.21.15.69:6789/0 conn(0x7fdcc806ece0 legacy=0x7fdcc815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.029+0000 7fdcc7fff640 1 -- 172.21.15.44:0/2231636017 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc806e200 con 0x7fdcc80d09f0 2024-04-02T13:32:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.030+0000 7fdcc7fff640 1 --2- 172.21.15.44:0/2231636017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcc80d09f0 0x7fdcc806d2d0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc0098a0 tx=0x7fdcbc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c95d55e0f9bef3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:20.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.030+0000 7fdcc77fe640 1 -- 172.21.15.44:0/2231636017 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc034070 con 0x7fdcc80d09f0 2024-04-02T13:32:20.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.030+0000 7fdcc77fe640 1 -- 172.21.15.44:0/2231636017 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcbc030840 con 0x7fdcc80d09f0 2024-04-02T13:32:20.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.030+0000 7fdcc77fe640 1 -- 172.21.15.44:0/2231636017 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc030c10 con 0x7fdcc80d09f0 2024-04-02T13:32:20.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.030+0000 7fdccf263640 1 -- 172.21.15.44:0/2231636017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcc80d09f0 msgr2=0x7fdcc806d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:20.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.030+0000 7fdccf263640 1 --2- 172.21.15.44:0/2231636017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcc80d09f0 0x7fdcc806d2d0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc0098a0 tx=0x7fdcbc02b980 comp rx=0 tx=0).stop 2024-04-02T13:32:20.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.031+0000 7fdccf263640 1 -- 172.21.15.44:0/2231636017 shutdown_connections 2024-04-02T13:32:20.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.031+0000 7fdccf263640 1 --2- 172.21.15.44:0/2231636017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcc80d09f0 0x7fdcc806d2d0 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.031+0000 7fdccf263640 1 --2- 172.21.15.44:0/2231636017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcc80cece0 0x7fdcc80cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.031+0000 7fdccf263640 1 -- 172.21.15.44:0/2231636017 >> 172.21.15.44:0/2231636017 conn(0x7fdcc80cfcd0 msgr2=0x7fdcc80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:20.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.031+0000 7fdccf263640 1 -- 172.21.15.44:0/2231636017 shutdown_connections 2024-04-02T13:32:20.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.031+0000 7fdccf263640 1 -- 172.21.15.44:0/2231636017 wait complete. 2024-04-02T13:32:20.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.032+0000 7fdccf263640 1 Processor -- start 2024-04-02T13:32:20.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.032+0000 7fdccf263640 1 -- start start 2024-04-02T13:32:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccf263640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcc806ece0 0x7fdcc814e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccf263640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcc80cece0 0x7fdcc8168560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccf263640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcc80d09f0 0x7fdcc8168aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccf263640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc806d810 con 0x7fdcc80d09f0 2024-04-02T13:32:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccf263640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc806e7c0 con 0x7fdcc80cece0 2024-04-02T13:32:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccf263640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdcc806d990 con 0x7fdcc806ece0 2024-04-02T13:32:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccd800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcc80d09f0 0x7fdcc8168aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccd800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcc80d09f0 0x7fdcc8168aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42402/0 (socket says 172.21.15.44:42402) 2024-04-02T13:32:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccd800640 1 -- 172.21.15.44:0/3921621035 learned_addr learned my addr 172.21.15.44:0/3921621035 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdcc7fff640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcc80cece0 0x7fdcc8168560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdcccfff640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcc806ece0 0x7fdcc814e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccd800640 1 -- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcc806ece0 msgr2=0x7fdcc814e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccd800640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcc806ece0 0x7fdcc814e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccd800640 1 -- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcc80cece0 msgr2=0x7fdcc8168560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccd800640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcc80cece0 0x7fdcc8168560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccd800640 1 -- 172.21.15.44:0/3921621035 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc809b6a0 con 0x7fdcc80d09f0 2024-04-02T13:32:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccd800640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcc80d09f0 0x7fdcc8168aa0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fdcc000c730 tx=0x7fdcc000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:20.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdcc57fa640 1 -- 172.21.15.44:0/3921621035 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0003b70 con 0x7fdcc80d09f0 2024-04-02T13:32:20.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdcc57fa640 1 -- 172.21.15.44:0/3921621035 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcc0003d10 con 0x7fdcc80d09f0 2024-04-02T13:32:20.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdcc57fa640 1 -- 172.21.15.44:0/3921621035 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc000aa90 con 0x7fdcc80d09f0 2024-04-02T13:32:20.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.033+0000 7fdccf263640 1 -- 172.21.15.44:0/3921621035 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcc809b980 con 0x7fdcc80d09f0 2024-04-02T13:32:20.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.034+0000 7fdccf263640 1 -- 172.21.15.44:0/3921621035 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc809bf40 con 0x7fdcc80d09f0 2024-04-02T13:32:20.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.035+0000 7fdccf263640 1 -- 172.21.15.44:0/3921621035 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcc8074f20 con 0x7fdcc80d09f0 2024-04-02T13:32:20.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.035+0000 7fdcc57fa640 1 -- 172.21.15.44:0/3921621035 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdcc000ac30 con 0x7fdcc80d09f0 2024-04-02T13:32:20.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.035+0000 7fdcc57fa640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdcb406c8f0 0x7fdcb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:20.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.035+0000 7fdcc57fa640 1 -- 172.21.15.44:0/3921621035 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7fdcc008b990 con 0x7fdcc80d09f0 2024-04-02T13:32:20.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.036+0000 7fdcccfff640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdcb406c8f0 0x7fdcb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:20.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.036+0000 7fdcccfff640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdcb406c8f0 0x7fdcb406edb0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009ad0 tx=0x7fdcb8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:20.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.040+0000 7fdcc57fa640 1 -- 172.21.15.44:0/3921621035 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fdcc0059e80 con 0x7fdcc80d09f0 2024-04-02T13:32:20.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.221+0000 7fdccf263640 1 -- 172.21.15.44:0/3921621035 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdcc80684f0 con 0x7fdcb406c8f0 2024-04-02T13:32:20.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.223+0000 7fdcc57fa640 1 -- 172.21.15.44:0/3921621035 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13505 (secure 0 0 0) 0x7fdcc80684f0 con 0x7fdcb406c8f0 2024-04-02T13:32:20.225 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:32:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.225+0000 7fdccf263640 1 -- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdcb406c8f0 msgr2=0x7fdcb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.225+0000 7fdccf263640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdcb406c8f0 0x7fdcb406edb0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009ad0 tx=0x7fdcb8008040 comp rx=0 tx=0).stop 2024-04-02T13:32:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.225+0000 7fdccf263640 1 -- 172.21.15.44:0/3921621035 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcc80d09f0 msgr2=0x7fdcc8168aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.225+0000 7fdccf263640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcc80d09f0 0x7fdcc8168aa0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fdcc000c730 tx=0x7fdcc000cbf0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.225+0000 7fdccf263640 1 -- 172.21.15.44:0/3921621035 shutdown_connections 2024-04-02T13:32:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.225+0000 7fdccf263640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fdcb406c8f0 0x7fdcb406edb0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.225+0000 7fdccf263640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcc80d09f0 0x7fdcc8168aa0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.225+0000 7fdccf263640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcc80cece0 0x7fdcc8168560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.225+0000 7fdccf263640 1 --2- 172.21.15.44:0/3921621035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcc806ece0 0x7fdcc814e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.225+0000 7fdccf263640 1 -- 172.21.15.44:0/3921621035 >> 172.21.15.44:0/3921621035 conn(0x7fdcc80cfcd0 msgr2=0x7fdcc80ad660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.225+0000 7fdccf263640 1 -- 172.21.15.44:0/3921621035 shutdown_connections 2024-04-02T13:32:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.225+0000 7fdccf263640 1 -- 172.21.15.44:0/3921621035 wait complete. 2024-04-02T13:32:20.239 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-04-02T13:32:19.054704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":262924,"kb_used_data":13480,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711796,"statfs":{"total":773094113280,"available":772824879104,"internally_reserved":0,"allocated":13803520,"data_stored":7075824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390584,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-264,"num_write":-176,"num_write_kb":-8295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001469"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":376,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-02T13:32:17.553383+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:32:17.553383+0000","last_peered":"2024-04-02T13:32:17.553383+0000","last_clean":"2024-04-02T13:32:17.553383+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:32:17.553383+0000","last_undegraded":"2024-04-02T13:32:17.553383+0000","last_fullsized":"2024-04-02T13:32:17.553383+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574996,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2170880,"data_stored":1327254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574995,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574996,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574996,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574996,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574995,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574996,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2170880,"data_stored":1327254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:32:20.239 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T13:32:20.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.338+0000 7f9dd455f640 1 Processor -- start 2024-04-02T13:32:20.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.338+0000 7f9dd455f640 1 -- start start 2024-04-02T13:32:20.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.339+0000 7f9dd455f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9dcc0d0a30 0x7f9dcc06cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:20.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.339+0000 7f9dd455f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dcc0d2040 0x7f9dcc0d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:20.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.339+0000 7f9dd455f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dcc0d0000 con 0x7f9dcc0d0e10 2024-04-02T13:32:20.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.339+0000 7f9dd455f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9dcc06cb00 con 0x7f9dcc0d0a30 2024-04-02T13:32:20.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.339+0000 7f9dd455f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9dcc06c1d0 con 0x7f9dcc0d2040 2024-04-02T13:32:20.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.339+0000 7f9dd22fb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9dcc0d0a30 0x7f9dcc06cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:20.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.339+0000 7f9dd2afc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dcc0d2040 0x7f9dcc0d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:20.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.340+0000 7f9dd22fb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9dcc0d0a30 0x7f9dcc06cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:32920/0 (socket says 172.21.15.44:32920) 2024-04-02T13:32:20.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.340+0000 7f9dd22fb640 1 -- 172.21.15.44:0/1404068181 learned_addr learned my addr 172.21.15.44:0/1404068181 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:20.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.340+0000 7f9dd1afa640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9dcc0d0e10 0x7f9dcc0cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:40262/0 (socket says 172.21.15.44:40262) 2024-04-02T13:32:20.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.340+0000 7f9dd2afc640 1 -- 172.21.15.44:0/1404068181 >> v2:172.21.15.69:3300/0 conn(0x7f9dcc0d0a30 msgr2=0x7f9dcc06cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:20.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.340+0000 7f9dd2afc640 1 --2- 172.21.15.44:0/1404068181 >> v2:172.21.15.69:3300/0 conn(0x7f9dcc0d0a30 0x7f9dcc06cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.340+0000 7f9dd2afc640 1 -- 172.21.15.44:0/1404068181 >> v1:172.21.15.69:6789/0 conn(0x7f9dcc0d0e10 legacy=0x7f9dcc0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:20.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.340+0000 7f9dd2afc640 1 -- 172.21.15.44:0/1404068181 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dcc06c4b0 con 0x7f9dcc0d2040 2024-04-02T13:32:20.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.340+0000 7f9dd22fb640 1 --2- 172.21.15.44:0/1404068181 >> v2:172.21.15.69:3300/0 conn(0x7f9dcc0d0a30 0x7f9dcc06cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:32:20.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.341+0000 7f9dd2afc640 1 --2- 172.21.15.44:0/1404068181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dcc0d2040 0x7f9dcc0d24f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40097a0 tx=0x7f9dc402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3608a13a0900037d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:20.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.341+0000 7f9dd12f9640 1 -- 172.21.15.44:0/1404068181 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc40315a0 con 0x7f9dcc0d2040 2024-04-02T13:32:20.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.341+0000 7f9dd12f9640 1 -- 172.21.15.44:0/1404068181 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dc4037070 con 0x7f9dcc0d2040 2024-04-02T13:32:20.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.341+0000 7f9dd12f9640 1 -- 172.21.15.44:0/1404068181 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4031960 con 0x7f9dcc0d2040 2024-04-02T13:32:20.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.341+0000 7f9dd455f640 1 -- 172.21.15.44:0/1404068181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dcc0d2040 msgr2=0x7f9dcc0d24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:20.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.341+0000 7f9dd455f640 1 --2- 172.21.15.44:0/1404068181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dcc0d2040 0x7f9dcc0d24f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40097a0 tx=0x7f9dc402b840 comp rx=0 tx=0).stop 2024-04-02T13:32:20.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.341+0000 7f9dd455f640 1 -- 172.21.15.44:0/1404068181 shutdown_connections 2024-04-02T13:32:20.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.341+0000 7f9dd455f640 1 --2- 172.21.15.44:0/1404068181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dcc0d2040 0x7f9dcc0d24f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.341+0000 7f9dd455f640 1 --2- 172.21.15.44:0/1404068181 >> v2:172.21.15.69:3300/0 conn(0x7f9dcc0d0a30 0x7f9dcc06cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.341+0000 7f9dd455f640 1 -- 172.21.15.44:0/1404068181 >> 172.21.15.44:0/1404068181 conn(0x7f9dcc09d470 msgr2=0x7f9dcc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:20.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.341+0000 7f9dd455f640 1 -- 172.21.15.44:0/1404068181 shutdown_connections 2024-04-02T13:32:20.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.341+0000 7f9dd455f640 1 -- 172.21.15.44:0/1404068181 wait complete. 2024-04-02T13:32:20.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.343+0000 7f9dd455f640 1 Processor -- start 2024-04-02T13:32:20.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.343+0000 7f9dd455f640 1 -- start start 2024-04-02T13:32:20.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.343+0000 7f9dd455f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dcc0d0a30 0x7f9dcc156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:20.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.343+0000 7f9dd455f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dcc0d0e10 0x7f9dcc160720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:20.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.343+0000 7f9dd455f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dcc0d2040 0x7f9dcc160c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:20.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.343+0000 7f9dd455f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9dcc15e6f0 con 0x7f9dcc0d2040 2024-04-02T13:32:20.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.343+0000 7f9dd455f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9dcc0d1270 con 0x7f9dcc0d0a30 2024-04-02T13:32:20.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.343+0000 7f9dd455f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9dcc15e870 con 0x7f9dcc0d0e10 2024-04-02T13:32:20.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.343+0000 7f9dd2afc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dcc0d2040 0x7f9dcc160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:20.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dd2afc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dcc0d2040 0x7f9dcc160c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42418/0 (socket says 172.21.15.44:42418) 2024-04-02T13:32:20.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dd2afc640 1 -- 172.21.15.44:0/711671513 learned_addr learned my addr 172.21.15.44:0/711671513 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:20.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dd22fb640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dcc0d0a30 0x7f9dcc156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:20.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dd1afa640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dcc0d0e10 0x7f9dcc160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:20.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dd2afc640 1 -- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dcc0d0e10 msgr2=0x7f9dcc160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:20.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dd2afc640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dcc0d0e10 0x7f9dcc160720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dd2afc640 1 -- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dcc0d0a30 msgr2=0x7f9dcc156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:20.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dd2afc640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dcc0d0a30 0x7f9dcc156570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dd2afc640 1 -- 172.21.15.44:0/711671513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dcc0b4780 con 0x7f9dcc0d2040 2024-04-02T13:32:20.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dd2afc640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dcc0d2040 0x7f9dcc160c60 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40097a0 tx=0x7f9dc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:20.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dcaffd640 1 -- 172.21.15.44:0/711671513 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4032630 con 0x7f9dcc0d2040 2024-04-02T13:32:20.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dcaffd640 1 -- 172.21.15.44:0/711671513 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dc40327d0 con 0x7f9dcc0d2040 2024-04-02T13:32:20.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dd455f640 1 -- 172.21.15.44:0/711671513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dcc0b4a60 con 0x7f9dcc0d2040 2024-04-02T13:32:20.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dcaffd640 1 -- 172.21.15.44:0/711671513 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4032bc0 con 0x7f9dcc0d2040 2024-04-02T13:32:20.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.344+0000 7f9dd455f640 1 -- 172.21.15.44:0/711671513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dcc1e80a0 con 0x7f9dcc0d2040 2024-04-02T13:32:20.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.345+0000 7f9dd455f640 1 -- 172.21.15.44:0/711671513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dcc0684f0 con 0x7f9dcc0d2040 2024-04-02T13:32:20.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.346+0000 7f9dcaffd640 1 -- 172.21.15.44:0/711671513 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9dc403b3f0 con 0x7f9dcc0d2040 2024-04-02T13:32:20.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.346+0000 7f9dcaffd640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9db806c8f0 0x7f9db806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:20.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.346+0000 7f9dcaffd640 1 -- 172.21.15.44:0/711671513 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f9dc402f030 con 0x7f9dcc0d2040 2024-04-02T13:32:20.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.346+0000 7f9dd22fb640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9db806c8f0 0x7f9db806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:20.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.347+0000 7f9dd22fb640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9db806c8f0 0x7f9db806edb0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0072c0 tx=0x7f9dbc00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:20.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.351+0000 7f9dcaffd640 1 -- 172.21.15.44:0/711671513 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9dc4040070 con 0x7f9dcc0d2040 2024-04-02T13:32:20.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.535+0000 7f9dd455f640 1 -- 172.21.15.44:0/711671513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9dcc0cf460 con 0x7f9dcc0d2040 2024-04-02T13:32:20.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.536+0000 7f9dcaffd640 1 -- 172.21.15.44:0/711671513 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) ==== 74+0+13511 (secure 0 0 0) 0x7f9dc407b610 con 0x7f9dcc0d2040 2024-04-02T13:32:20.539 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:20.539 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":126,"fsid":"4101e14a-13fe-452f-8f9a-78614b3ea6ab","created":"2024-04-02T13:20:02.595629+0000","modified":"2024-04-02T13:32:17.546708+0000","last_up_change":"2024-04-02T13:20:13.833429+0000","last_in_change":"2024-04-02T13:20:05.566313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T13:20:14.888847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30e26323-bca4-4e6a-876d-3760fd190851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6802","nonce":2303805525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6806","nonce":2303805525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6814","nonce":2303805525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2303805525},{"type":"v1","addr":"172.21.15.44:6810","nonce":2303805525}]},"public_addr":"172.21.15.44:6802/2303805525","cluster_addr":"172.21.15.44:6806/2303805525","heartbeat_back_addr":"172.21.15.44:6814/2303805525","heartbeat_front_addr":"172.21.15.44:6810/2303805525","state":["exists","up"]},{"osd":1,"uuid":"8fad5ee1-c346-4423-a9ab-8844a8c5764c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6801","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6803","nonce":704457846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6807","nonce":704457846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6813","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6815","nonce":704457846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":704457846},{"type":"v1","addr":"172.21.15.44:6811","nonce":704457846}]},"public_addr":"172.21.15.44:6803/704457846","cluster_addr":"172.21.15.44:6807/704457846","heartbeat_back_addr":"172.21.15.44:6815/704457846","heartbeat_front_addr":"172.21.15.44:6811/704457846","state":["exists","up"]},{"osd":2,"uuid":"d06483ed-67be-45e7-9cbb-3e8c42844491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6817","nonce":2091834331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6819","nonce":2091834331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6823","nonce":2091834331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2091834331},{"type":"v1","addr":"172.21.15.44:6821","nonce":2091834331}]},"public_addr":"172.21.15.44:6817/2091834331","cluster_addr":"172.21.15.44:6819/2091834331","heartbeat_back_addr":"172.21.15.44:6823/2091834331","heartbeat_front_addr":"172.21.15.44:6821/2091834331","state":["exists","up"]},{"osd":3,"uuid":"6334eb96-f083-49fa-8ae2-a307382727d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6825","nonce":1588952513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6827","nonce":1588952513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6831","nonce":1588952513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1588952513},{"type":"v1","addr":"172.21.15.44:6829","nonce":1588952513}]},"public_addr":"172.21.15.44:6825/1588952513","cluster_addr":"172.21.15.44:6827/1588952513","heartbeat_back_addr":"172.21.15.44:6831/1588952513","heartbeat_front_addr":"172.21.15.44:6829/1588952513","state":["exists","up"]},{"osd":4,"uuid":"d3f9ff73-9bd8-4177-9931-b3f818a4204c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6817","nonce":1041350007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6819","nonce":1041350007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6823","nonce":1041350007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1041350007},{"type":"v1","addr":"172.21.15.69:6821","nonce":1041350007}]},"public_addr":"172.21.15.69:6817/1041350007","cluster_addr":"172.21.15.69:6819/1041350007","heartbeat_back_addr":"172.21.15.69:6823/1041350007","heartbeat_front_addr":"172.21.15.69:6821/1041350007","state":["exists","up"]},{"osd":5,"uuid":"37e3b164-c05f-4989-9658-8866f1b512b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6801","nonce":2841767499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6805","nonce":2841767499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6812","nonce":2841767499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2841767499},{"type":"v1","addr":"172.21.15.69:6809","nonce":2841767499}]},"public_addr":"172.21.15.69:6801/2841767499","cluster_addr":"172.21.15.69:6805/2841767499","heartbeat_back_addr":"172.21.15.69:6812/2841767499","heartbeat_front_addr":"172.21.15.69:6809/2841767499","state":["exists","up"]},{"osd":6,"uuid":"a7f2fbf4-1277-4837-a6bb-f7762c4f373f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6804","nonce":2604158175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6808","nonce":2604158175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6815","nonce":2604158175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":2604158175},{"type":"v1","addr":"172.21.15.69:6813","nonce":2604158175}]},"public_addr":"172.21.15.69:6804/2604158175","cluster_addr":"172.21.15.69:6808/2604158175","heartbeat_back_addr":"172.21.15.69:6815/2604158175","heartbeat_front_addr":"172.21.15.69:6813/2604158175","state":["exists","up"]},{"osd":7,"uuid":"1ee7479b-f52a-4eb7-8e43-702e0a8d0ed4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6825","nonce":1292475712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6827","nonce":1292475712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6831","nonce":1292475712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1292475712},{"type":"v1","addr":"172.21.15.69:6829","nonce":1292475712}]},"public_addr":"172.21.15.69:6825/1292475712","cluster_addr":"172.21.15.69:6827/1292475712","heartbeat_back_addr":"172.21.15.69:6831/1292475712","heartbeat_front_addr":"172.21.15.69:6829/1292475712","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.629329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.708964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.773050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.728478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.774001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.733887+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.684344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T13:20:10.751206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6835/16875964":"2024-04-03T13:32:14.539214+0000","172.21.15.44:6833/16875964":"2024-04-03T13:32:14.539214+0000","172.21.15.44:6835/2895101355":"2024-04-03T13:31:59.833326+0000","172.21.15.44:6833/2895101355":"2024-04-03T13:31:59.833326+0000","172.21.15.44:6835/2109578276":"2024-04-03T13:31:41.102885+0000","172.21.15.44:6833/2109578276":"2024-04-03T13:31:41.102885+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T13:32:20.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.538+0000 7f9dd455f640 1 -- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9db806c8f0 msgr2=0x7f9db806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:20.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.538+0000 7f9dd455f640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9db806c8f0 0x7f9db806edb0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0072c0 tx=0x7f9dbc00b040 comp rx=0 tx=0).stop 2024-04-02T13:32:20.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.538+0000 7f9dd455f640 1 -- 172.21.15.44:0/711671513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dcc0d2040 msgr2=0x7f9dcc160c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:20.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.538+0000 7f9dd455f640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dcc0d2040 0x7f9dcc160c60 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40097a0 tx=0x7f9dc40046c0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.538+0000 7f9dd455f640 1 -- 172.21.15.44:0/711671513 shutdown_connections 2024-04-02T13:32:20.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.538+0000 7f9dd455f640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9db806c8f0 0x7f9db806edb0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.538+0000 7f9dd455f640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dcc0d2040 0x7f9dcc160c60 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.538+0000 7f9dd455f640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dcc0d0e10 0x7f9dcc160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.538+0000 7f9dd455f640 1 --2- 172.21.15.44:0/711671513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dcc0d0a30 0x7f9dcc156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:20.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.538+0000 7f9dd455f640 1 -- 172.21.15.44:0/711671513 >> 172.21.15.44:0/711671513 conn(0x7f9dcc09d470 msgr2=0x7f9dcc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:20.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.538+0000 7f9dd455f640 1 -- 172.21.15.44:0/711671513 shutdown_connections 2024-04-02T13:32:20.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:20.538+0000 7f9dd455f640 1 -- 172.21.15.44:0/711671513 wait complete. 2024-04-02T13:32:21.552 INFO:tasks.ceph:Scrubbing osd.0 2024-04-02T13:32:21.553 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-02T13:32:21.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.671+0000 7f5aabceb640 1 Processor -- start 2024-04-02T13:32:21.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.672+0000 7f5aabceb640 1 -- start start 2024-04-02T13:32:21.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.672+0000 7f5aabceb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5aa415e1d0 0x7f5aa40d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.673+0000 7f5aabceb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aa40d1640 con 0x7f5aa40cec60 2024-04-02T13:32:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.673+0000 7f5aabceb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aa40cf1f0 con 0x7f5aa40d2dd0 2024-04-02T13:32:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.673+0000 7f5aabceb640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f5aa40cfe00 con 0x7f5aa415e1d0 2024-04-02T13:32:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.673+0000 7f5aa9a87640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f5aa415e1d0 0x7f5aa40d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.673+0000 7f5aa9286640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f5aa40cec60 0x7f5aa40d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52352/0 (socket says 172.21.15.44:52352) 2024-04-02T13:32:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.673+0000 7f5aa9286640 1 -- 172.21.15.44:0/1681607254 learned_addr learned my addr 172.21.15.44:0/1681607254 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.673+0000 7f5aa9a87640 1 -- 172.21.15.44:0/1681607254 >> v1:172.21.15.69:6789/0 conn(0x7f5aa40d2dd0 legacy=0x7f5aa40b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.673+0000 7f5aa9a87640 1 -- 172.21.15.44:0/1681607254 >> v1:172.21.15.44:6789/0 conn(0x7f5aa40cec60 legacy=0x7f5aa40d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:21.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.673+0000 7f5aa9a87640 1 -- 172.21.15.44:0/1681607254 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa40cf4d0 con 0x7f5aa415e1d0 2024-04-02T13:32:21.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.674+0000 7f5aa9a87640 1 --2- 172.21.15.44:0/1681607254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5aa415e1d0 0x7f5aa40d1100 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00097a0 tx=0x7f5aa002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35fdb0d36cd21251 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:21.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.674+0000 7f5aa8a85640 1 -- 172.21.15.44:0/1681607254 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa00315a0 con 0x7f5aa415e1d0 2024-04-02T13:32:21.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.674+0000 7f5aa8a85640 1 -- 172.21.15.44:0/1681607254 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa0037070 con 0x7f5aa415e1d0 2024-04-02T13:32:21.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.674+0000 7f5aa8a85640 1 -- 172.21.15.44:0/1681607254 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0031960 con 0x7f5aa415e1d0 2024-04-02T13:32:21.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.675+0000 7f5aabceb640 1 -- 172.21.15.44:0/1681607254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5aa415e1d0 msgr2=0x7f5aa40d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:21.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.675+0000 7f5aabceb640 1 --2- 172.21.15.44:0/1681607254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5aa415e1d0 0x7f5aa40d1100 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00097a0 tx=0x7f5aa002b840 comp rx=0 tx=0).stop 2024-04-02T13:32:21.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.675+0000 7f5aabceb640 1 -- 172.21.15.44:0/1681607254 shutdown_connections 2024-04-02T13:32:21.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.675+0000 7f5aabceb640 1 --2- 172.21.15.44:0/1681607254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5aa415e1d0 0x7f5aa40d1100 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:21.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.675+0000 7f5aabceb640 1 -- 172.21.15.44:0/1681607254 >> 172.21.15.44:0/1681607254 conn(0x7f5aa409d3b0 msgr2=0x7f5aa40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:21.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.675+0000 7f5aabceb640 1 -- 172.21.15.44:0/1681607254 shutdown_connections 2024-04-02T13:32:21.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.675+0000 7f5aabceb640 1 -- 172.21.15.44:0/1681607254 wait complete. 2024-04-02T13:32:21.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.676+0000 7f5aabceb640 1 Processor -- start 2024-04-02T13:32:21.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.676+0000 7f5aabceb640 1 -- start start 2024-04-02T13:32:21.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aabceb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5aa40cec60 0x7f5aa41609f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aabceb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5aa40d2dd0 0x7f5aa415cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aabceb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5aa415e1d0 0x7f5aa415d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aabceb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa40b7fd0 con 0x7f5aa415e1d0 2024-04-02T13:32:21.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aabceb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa40cfa90 con 0x7f5aa40cec60 2024-04-02T13:32:21.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aabceb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5aa40b8150 con 0x7f5aa40d2dd0 2024-04-02T13:32:21.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aaa288640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5aa415e1d0 0x7f5aa415d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aaa288640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5aa415e1d0 0x7f5aa415d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42428/0 (socket says 172.21.15.44:42428) 2024-04-02T13:32:21.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aaa288640 1 -- 172.21.15.44:0/447900152 learned_addr learned my addr 172.21.15.44:0/447900152 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:21.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aa9a87640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5aa40cec60 0x7f5aa41609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aa9286640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5aa40d2dd0 0x7f5aa415cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aaa288640 1 -- 172.21.15.44:0/447900152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5aa40d2dd0 msgr2=0x7f5aa415cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:21.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aaa288640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5aa40d2dd0 0x7f5aa415cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aaa288640 1 -- 172.21.15.44:0/447900152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5aa40cec60 msgr2=0x7f5aa41609f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aaa288640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5aa40cec60 0x7f5aa41609f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aaa288640 1 -- 172.21.15.44:0/447900152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa4150890 con 0x7f5aa415e1d0 2024-04-02T13:32:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aa9286640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5aa40d2dd0 0x7f5aa415cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.677+0000 7f5aaa288640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5aa415e1d0 0x7f5aa415d220 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00b9e0 tx=0x7f5a9c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.678+0000 7f5a9a7fc640 1 -- 172.21.15.44:0/447900152 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c003b70 con 0x7f5aa415e1d0 2024-04-02T13:32:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.678+0000 7f5a9a7fc640 1 -- 172.21.15.44:0/447900152 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a9c003d10 con 0x7f5aa415e1d0 2024-04-02T13:32:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.678+0000 7f5aabceb640 1 -- 172.21.15.44:0/447900152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa4150b30 con 0x7f5aa415e1d0 2024-04-02T13:32:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.678+0000 7f5a9a7fc640 1 -- 172.21.15.44:0/447900152 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c00cb60 con 0x7f5aa415e1d0 2024-04-02T13:32:21.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.678+0000 7f5aabceb640 1 -- 172.21.15.44:0/447900152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa414f0c0 con 0x7f5aa415e1d0 2024-04-02T13:32:21.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.679+0000 7f5aabceb640 1 -- 172.21.15.44:0/447900152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5aa41511f0 con 0x7f5aa415e1d0 2024-04-02T13:32:21.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.679+0000 7f5a9a7fc640 1 -- 172.21.15.44:0/447900152 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5a9c00cd00 con 0x7f5aa415e1d0 2024-04-02T13:32:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.679+0000 7f5a9a7fc640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5a8806c8f0 0x7f5a8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.680+0000 7f5a9a7fc640 1 -- 172.21.15.44:0/447900152 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f5a9c08b8c0 con 0x7f5aa415e1d0 2024-04-02T13:32:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.680+0000 7f5a9a7fc640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x7f5a88072ed0 0x7f5a88075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.680+0000 7f5a9a7fc640 1 -- 172.21.15.44:0/447900152 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a880759e0 con 0x7f5a88072ed0 2024-04-02T13:32:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.680+0000 7f5a9a7fc640 1 -- 172.21.15.44:0/447900152 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=126) ==== 24+0+0 (secure 0 0 0) 0x7f5a9c0902c0 con 0x7f5aa415e1d0 2024-04-02T13:32:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.680+0000 7f5aa9a87640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5a8806c8f0 0x7f5a8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.680+0000 7f5aa9286640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x7f5a88072ed0 0x7f5a88075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.680+0000 7f5aa9286640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x7f5a88072ed0 0x7f5a88075310 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.680+0000 7f5aa9a87640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5a8806c8f0 0x7f5a8806edb0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0036000 tx=0x7f5aa003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:21.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.681+0000 7f5a9a7fc640 1 -- 172.21.15.44:0/447900152 <== osd.0 v2:172.21.15.44:6800/2303805525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5a880759e0 con 0x7f5a88072ed0 2024-04-02T13:32:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.694+0000 7f5aabceb640 1 -- 172.21.15.44:0/447900152 --> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5aa40684f0 con 0x7f5a88072ed0 2024-04-02T13:32:21.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.695+0000 7f5a9a7fc640 1 -- 172.21.15.44:0/447900152 <== osd.0 v2:172.21.15.44:6800/2303805525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5aa40684f0 con 0x7f5a88072ed0 2024-04-02T13:32:21.698 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:32:21.698 INFO:teuthology.orchestra.run.smithi044.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T13:32:21.698 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:32:21.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.695+0000 7f5aabceb640 1 -- 172.21.15.44:0/447900152 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x7f5a88072ed0 msgr2=0x7f5a88075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:21.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.695+0000 7f5aabceb640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.44:6800/2303805525,v1:172.21.15.44:6802/2303805525] conn(0x7f5a88072ed0 0x7f5a88075310 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:21.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.695+0000 7f5aabceb640 1 -- 172.21.15.44:0/447900152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5a8806c8f0 msgr2=0x7f5a8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:21.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.695+0000 7f5aabceb640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f5a8806c8f0 0x7f5a8806edb0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0036000 tx=0x7f5aa003d040 comp rx=0 tx=0).stop 2024-04-02T13:32:21.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.695+0000 7f5aabceb640 1 -- 172.21.15.44:0/447900152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5aa415e1d0 msgr2=0x7f5aa415d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:21.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.695+0000 7f5aabceb640 1 --2- 172.21.15.44:0/447900152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5aa415e1d0 0x7f5aa415d220 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00b9e0 tx=0x7f5a9c00bea0 comp rx=0 tx=0).stop 2024-04-02T13:32:21.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.695+0000 7f5aaa288640 1 -- 172.21.15.44:0/447900152 reap_dead start 2024-04-02T13:32:21.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.695+0000 7f5aabceb640 1 -- 172.21.15.44:0/447900152 shutdown_connections 2024-04-02T13:32:21.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.695+0000 7f5aabceb640 1 -- 172.21.15.44:0/447900152 >> 172.21.15.44:0/447900152 conn(0x7f5aa409d3b0 msgr2=0x7f5aa40b97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:21.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.695+0000 7f5aabceb640 1 -- 172.21.15.44:0/447900152 shutdown_connections 2024-04-02T13:32:21.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.695+0000 7f5aabceb640 1 -- 172.21.15.44:0/447900152 wait complete. 2024-04-02T13:32:21.709 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-02T13:32:21.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.809+0000 7f296e974640 1 Processor -- start 2024-04-02T13:32:21.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.809+0000 7f296e974640 1 -- start start 2024-04-02T13:32:21.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.810+0000 7f296e974640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29680d0a30 0x7f296806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.810+0000 7f296e974640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f29680d2040 0x7f29680d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.810+0000 7f296e974640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29680d0000 con 0x7f29680d0e10 2024-04-02T13:32:21.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.810+0000 7f296e974640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f296806cb00 con 0x7f29680d2040 2024-04-02T13:32:21.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.810+0000 7f296e974640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f296806c1d0 con 0x7f29680d0a30 2024-04-02T13:32:21.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.810+0000 7f29677fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f29680d0e10 0x7f29680cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52358/0 (socket says 172.21.15.44:52358) 2024-04-02T13:32:21.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.810+0000 7f29677fe640 1 -- 172.21.15.44:0/2071391005 learned_addr learned my addr 172.21.15.44:0/2071391005 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:21.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2967fff640 1 --2- 172.21.15.44:0/2071391005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29680d0a30 0x7f296806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f296cf11640 1 --2- 172.21.15.44:0/2071391005 >> v2:172.21.15.69:3300/0 conn(0x7f29680d2040 0x7f29680d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3098335478 0 0) 0x7f29680d0000 con 0x7f29680d0e10 2024-04-02T13:32:21.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2954003520 con 0x7f29680d0e10 2024-04-02T13:32:21.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1617811712 0 0) 0x7f2954003520 con 0x7f29680d0e10 2024-04-02T13:32:21.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29680d0a30 msgr2=0x7f296806cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:21.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 --2- 172.21.15.44:0/2071391005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29680d0a30 0x7f296806cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:21.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 >> v2:172.21.15.69:3300/0 conn(0x7f29680d2040 msgr2=0x7f29680d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:21.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 --2- 172.21.15.44:0/2071391005 >> v2:172.21.15.69:3300/0 conn(0x7f29680d2040 0x7f29680d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:21.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f296806c4b0 con 0x7f29680d0e10 2024-04-02T13:32:21.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2961894843 0 0) 0x7f2958001670 con 0x7f29680d0e10 2024-04-02T13:32:21.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 >> v1:172.21.15.44:6789/0 conn(0x7f29680d0e10 legacy=0x7f29680cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:21.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 --2- 172.21.15.44:0/2071391005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29680d0a30 0x7f29540039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 --2- 172.21.15.44:0/2071391005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2954003f30 0x7f2954004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 --2- 172.21.15.44:0/2071391005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2954004d40 0x7f29540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f296806cb00 con 0x7f2954003f30 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f296806c1d0 con 0x7f2954004d40 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f295400ba10 con 0x7f29680d0a30 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f29677fe640 1 --2- 172.21.15.44:0/2071391005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2954003f30 0x7f2954004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f296cf11640 1 --2- 172.21.15.44:0/2071391005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2954004d40 0x7f29540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f2967fff640 1 --2- 172.21.15.44:0/2071391005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29680d0a30 0x7f29540039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f29677fe640 1 -- 172.21.15.44:0/2071391005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29680d0a30 msgr2=0x7f29540039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f29677fe640 1 --2- 172.21.15.44:0/2071391005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29680d0a30 0x7f29540039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f29677fe640 1 -- 172.21.15.44:0/2071391005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2954004d40 msgr2=0x7f29540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f29677fe640 1 --2- 172.21.15.44:0/2071391005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2954004d40 0x7f29540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.811+0000 7f29677fe640 1 -- 172.21.15.44:0/2071391005 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2954003520 con 0x7f2954003f30 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.812+0000 7f29677fe640 1 --2- 172.21.15.44:0/2071391005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2954003f30 0x7f2954004800 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f295800c210 tx=0x7f2954005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb0fef45ba18465 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.812+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958007d70 con 0x7f2954003f30 2024-04-02T13:32:21.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.812+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2958030070 con 0x7f2954003f30 2024-04-02T13:32:21.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.812+0000 7f2966ffd640 1 -- 172.21.15.44:0/2071391005 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f295802cd60 con 0x7f2954003f30 2024-04-02T13:32:21.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.813+0000 7f296e974640 1 -- 172.21.15.44:0/2071391005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2954003f30 msgr2=0x7f2954004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:21.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.813+0000 7f296e974640 1 --2- 172.21.15.44:0/2071391005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2954003f30 0x7f2954004800 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f295800c210 tx=0x7f2954005830 comp rx=0 tx=0).stop 2024-04-02T13:32:21.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.813+0000 7f296cf11640 1 -- 172.21.15.44:0/2071391005 reap_dead start 2024-04-02T13:32:21.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.813+0000 7f296e974640 1 -- 172.21.15.44:0/2071391005 shutdown_connections 2024-04-02T13:32:21.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.813+0000 7f296e974640 1 -- 172.21.15.44:0/2071391005 >> 172.21.15.44:0/2071391005 conn(0x7f296809d470 msgr2=0x7f29680d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:21.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.813+0000 7f296e974640 1 -- 172.21.15.44:0/2071391005 shutdown_connections 2024-04-02T13:32:21.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.813+0000 7f296e974640 1 -- 172.21.15.44:0/2071391005 wait complete. 2024-04-02T13:32:21.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.814+0000 7f296e974640 1 Processor -- start 2024-04-02T13:32:21.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.814+0000 7f296e974640 1 -- start start 2024-04-02T13:32:21.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.815+0000 7f296e974640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2954003f30 0x7f29680b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.815+0000 7f296e974640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29680b40b0 0x7f29680b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.815+0000 7f296e974640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29680b5f80 0x7f29680b6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.815+0000 7f296e974640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f296815e6f0 con 0x7f29680b5f80 2024-04-02T13:32:21.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.815+0000 7f296e974640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f29680d1270 con 0x7f2954003f30 2024-04-02T13:32:21.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.815+0000 7f296e974640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f296815e870 con 0x7f29680b40b0 2024-04-02T13:32:21.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.815+0000 7f296cf11640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29680b5f80 0x7f29680b6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.815+0000 7f296cf11640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29680b5f80 0x7f29680b6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42440/0 (socket says 172.21.15.44:42440) 2024-04-02T13:32:21.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.815+0000 7f296cf11640 1 -- 172.21.15.44:0/3793045572 learned_addr learned my addr 172.21.15.44:0/3793045572 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:21.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.815+0000 7f2967fff640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2954003f30 0x7f29680b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.815+0000 7f29677fe640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29680b40b0 0x7f29680b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.816+0000 7f296cf11640 1 -- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29680b40b0 msgr2=0x7f29680b5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:21.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.816+0000 7f296cf11640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29680b40b0 0x7f29680b5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:21.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.816+0000 7f296cf11640 1 -- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2954003f30 msgr2=0x7f29680b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:21.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.816+0000 7f296cf11640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2954003f30 0x7f29680b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:21.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.816+0000 7f296cf11640 1 -- 172.21.15.44:0/3793045572 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29680b6970 con 0x7f29680b5f80 2024-04-02T13:32:21.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.816+0000 7f296cf11640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29680b5f80 0x7f29680b6430 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f29680b6d30 tx=0x7f296000f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:21.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.816+0000 7f2964ff9640 1 -- 172.21.15.44:0/3793045572 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296000fd10 con 0x7f29680b5f80 2024-04-02T13:32:21.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.816+0000 7f2964ff9640 1 -- 172.21.15.44:0/3793045572 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f296001d070 con 0x7f29680b5f80 2024-04-02T13:32:21.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.816+0000 7f296e974640 1 -- 172.21.15.44:0/3793045572 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29680b7b50 con 0x7f29680b5f80 2024-04-02T13:32:21.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.816+0000 7f2964ff9640 1 -- 172.21.15.44:0/3793045572 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2960015670 con 0x7f29680b5f80 2024-04-02T13:32:21.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.816+0000 7f296e974640 1 -- 172.21.15.44:0/3793045572 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29680b80d0 con 0x7f29680b5f80 2024-04-02T13:32:21.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.817+0000 7f296e974640 1 -- 172.21.15.44:0/3793045572 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29680684f0 con 0x7f29680b5f80 2024-04-02T13:32:21.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.818+0000 7f2964ff9640 1 -- 172.21.15.44:0/3793045572 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2960015810 con 0x7f29680b5f80 2024-04-02T13:32:21.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.818+0000 7f2964ff9640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f294806c920 0x7f294806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:21.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.819+0000 7f2964ff9640 1 -- 172.21.15.44:0/3793045572 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f296008c290 con 0x7f29680b5f80 2024-04-02T13:32:21.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.819+0000 7f2967fff640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f294806c920 0x7f294806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:21.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.819+0000 7f2967fff640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f294806c920 0x7f294806ede0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f29681ea9a0 tx=0x7f2950009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:21.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:21.823+0000 7f2964ff9640 1 -- 172.21.15.44:0/3793045572 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f29680b6970 con 0x7f29680b5f80 2024-04-02T13:32:22.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.008+0000 7f296e974640 1 -- 172.21.15.44:0/3793045572 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f29680b7930 con 0x7f294806c920 2024-04-02T13:32:22.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.009+0000 7f2964ff9640 1 -- 172.21.15.44:0/3793045572 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f29680b7930 con 0x7f294806c920 2024-04-02T13:32:22.012 INFO:teuthology.orchestra.run.smithi044.stderr:instructed osd(s) 0 to deep-scrub 2024-04-02T13:32:22.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.011+0000 7f296e974640 1 -- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f294806c920 msgr2=0x7f294806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.011+0000 7f296e974640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f294806c920 0x7f294806ede0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f29681ea9a0 tx=0x7f2950009190 comp rx=0 tx=0).stop 2024-04-02T13:32:22.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.011+0000 7f296e974640 1 -- 172.21.15.44:0/3793045572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29680b5f80 msgr2=0x7f29680b6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.011+0000 7f296e974640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29680b5f80 0x7f29680b6430 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f29680b6d30 tx=0x7f296000f1b0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.011+0000 7f296e974640 1 -- 172.21.15.44:0/3793045572 shutdown_connections 2024-04-02T13:32:22.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.011+0000 7f296e974640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f294806c920 0x7f294806ede0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.011+0000 7f296e974640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29680b5f80 0x7f29680b6430 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.011+0000 7f296e974640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29680b40b0 0x7f29680b5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.011+0000 7f296e974640 1 --2- 172.21.15.44:0/3793045572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2954003f30 0x7f29680b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.011+0000 7f296e974640 1 -- 172.21.15.44:0/3793045572 >> 172.21.15.44:0/3793045572 conn(0x7f296809d470 msgr2=0x7f29680b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:22.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.011+0000 7f296e974640 1 -- 172.21.15.44:0/3793045572 shutdown_connections 2024-04-02T13:32:22.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.011+0000 7f296e974640 1 -- 172.21.15.44:0/3793045572 wait complete. 2024-04-02T13:32:22.025 INFO:tasks.ceph:Scrubbing osd.1 2024-04-02T13:32:22.025 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-02T13:32:22.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.128+0000 7fcc3c470640 1 Processor -- start 2024-04-02T13:32:22.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.128+0000 7fcc3c470640 1 -- start start 2024-04-02T13:32:22.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.129+0000 7fcc3c470640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcc3415e260 0x7fcc340d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.129+0000 7fcc3c470640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcc3415e640 0x7fcc340d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.129+0000 7fcc3c470640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc340cf120 con 0x7fcc340cec60 2024-04-02T13:32:22.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.129+0000 7fcc3c470640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fcc340b5a50 con 0x7fcc3415e640 2024-04-02T13:32:22.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.129+0000 7fcc3c470640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcc340d01c0 con 0x7fcc3415e260 2024-04-02T13:32:22.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.130+0000 7fcc39a0b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcc3415e640 0x7fcc340d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.130+0000 7fcc3aa0d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fcc340cec60 0x7fcc34161f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52372/0 (socket says 172.21.15.44:52372) 2024-04-02T13:32:22.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.130+0000 7fcc3aa0d640 1 -- 172.21.15.44:0/3812163928 learned_addr learned my addr 172.21.15.44:0/3812163928 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:22.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.130+0000 7fcc3a20c640 1 --2- 172.21.15.44:0/3812163928 >> v2:172.21.15.69:3300/0 conn(0x7fcc3415e260 0x7fcc340d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.130+0000 7fcc39a0b640 1 -- 172.21.15.44:0/3812163928 >> v2:172.21.15.69:3300/0 conn(0x7fcc3415e260 msgr2=0x7fcc340d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.130+0000 7fcc39a0b640 1 --2- 172.21.15.44:0/3812163928 >> v2:172.21.15.69:3300/0 conn(0x7fcc3415e260 0x7fcc340d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.130+0000 7fcc39a0b640 1 -- 172.21.15.44:0/3812163928 >> v1:172.21.15.44:6789/0 conn(0x7fcc340cec60 legacy=0x7fcc34161f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.130+0000 7fcc39a0b640 1 -- 172.21.15.44:0/3812163928 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc340cf400 con 0x7fcc3415e640 2024-04-02T13:32:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.130+0000 7fcc3a20c640 1 --2- 172.21.15.44:0/3812163928 >> v2:172.21.15.69:3300/0 conn(0x7fcc3415e260 0x7fcc340d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.131+0000 7fcc39a0b640 1 --2- 172.21.15.44:0/3812163928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc3415e640 0x7fcc340d0ad0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fcc3409d8f0 tx=0x7fcc2402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=832ac9b3cf977a79 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.131+0000 7fcc3920a640 1 -- 172.21.15.44:0/3812163928 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc240315a0 con 0x7fcc3415e640 2024-04-02T13:32:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.131+0000 7fcc3920a640 1 -- 172.21.15.44:0/3812163928 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc24037070 con 0x7fcc3415e640 2024-04-02T13:32:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.131+0000 7fcc3920a640 1 -- 172.21.15.44:0/3812163928 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24031960 con 0x7fcc3415e640 2024-04-02T13:32:22.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.131+0000 7fcc3c470640 1 -- 172.21.15.44:0/3812163928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc3415e640 msgr2=0x7fcc340d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.131+0000 7fcc3c470640 1 --2- 172.21.15.44:0/3812163928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc3415e640 0x7fcc340d0ad0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fcc3409d8f0 tx=0x7fcc2402b840 comp rx=0 tx=0).stop 2024-04-02T13:32:22.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.131+0000 7fcc3c470640 1 -- 172.21.15.44:0/3812163928 shutdown_connections 2024-04-02T13:32:22.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.131+0000 7fcc3c470640 1 --2- 172.21.15.44:0/3812163928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc3415e640 0x7fcc340d0ad0 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.131+0000 7fcc3c470640 1 --2- 172.21.15.44:0/3812163928 >> v2:172.21.15.69:3300/0 conn(0x7fcc3415e260 0x7fcc340d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.131+0000 7fcc3c470640 1 -- 172.21.15.44:0/3812163928 >> 172.21.15.44:0/3812163928 conn(0x7fcc3409d3c0 msgr2=0x7fcc340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:22.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.132+0000 7fcc3c470640 1 -- 172.21.15.44:0/3812163928 shutdown_connections 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.132+0000 7fcc3c470640 1 -- 172.21.15.44:0/3812163928 wait complete. 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.133+0000 7fcc3c470640 1 Processor -- start 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.133+0000 7fcc3c470640 1 -- start start 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.133+0000 7fcc3c470640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc340cec60 0x7fcc340b6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.133+0000 7fcc3c470640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc3415e260 0x7fcc340b2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.133+0000 7fcc3c470640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc3415e640 0x7fcc34159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.133+0000 7fcc3c470640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcc340cfca0 con 0x7fcc340cec60 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.133+0000 7fcc3c470640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcc340cf9c0 con 0x7fcc3415e260 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.133+0000 7fcc3c470640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcc340d2ea0 con 0x7fcc3415e640 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.133+0000 7fcc3a20c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc340cec60 0x7fcc340b6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.133+0000 7fcc3a20c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc340cec60 0x7fcc340b6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42464/0 (socket says 172.21.15.44:42464) 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.133+0000 7fcc3a20c640 1 -- 172.21.15.44:0/2264096664 learned_addr learned my addr 172.21.15.44:0/2264096664 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.134+0000 7fcc39a0b640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc3415e260 0x7fcc340b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.134+0000 7fcc3aa0d640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc3415e640 0x7fcc34159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.134+0000 7fcc3a20c640 1 -- 172.21.15.44:0/2264096664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc3415e640 msgr2=0x7fcc34159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.134+0000 7fcc3a20c640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc3415e640 0x7fcc34159560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.134+0000 7fcc3a20c640 1 -- 172.21.15.44:0/2264096664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc3415e260 msgr2=0x7fcc340b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.134+0000 7fcc3a20c640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc3415e260 0x7fcc340b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.134+0000 7fcc3a20c640 1 -- 172.21.15.44:0/2264096664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc340b73c0 con 0x7fcc340cec60 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.134+0000 7fcc3a20c640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc340cec60 0x7fcc340b6d00 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00e810 tx=0x7fcc2c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.134+0000 7fcc2affd640 1 -- 172.21.15.44:0/2264096664 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c00f040 con 0x7fcc340cec60 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.134+0000 7fcc2affd640 1 -- 172.21.15.44:0/2264096664 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc2c004030 con 0x7fcc340cec60 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.134+0000 7fcc3c470640 1 -- 172.21.15.44:0/2264096664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc340b76a0 con 0x7fcc340cec60 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.134+0000 7fcc2affd640 1 -- 172.21.15.44:0/2264096664 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c004400 con 0x7fcc340cec60 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.134+0000 7fcc3c470640 1 -- 172.21.15.44:0/2264096664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc340b7c60 con 0x7fcc340cec60 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.135+0000 7fcc3c470640 1 -- 172.21.15.44:0/2264096664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcc340684f0 con 0x7fcc340cec60 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.135+0000 7fcc2affd640 1 -- 172.21.15.44:0/2264096664 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcc2c00c6d0 con 0x7fcc340cec60 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.136+0000 7fcc2affd640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcc1806c8f0 0x7fcc1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.136+0000 7fcc2affd640 1 -- 172.21.15.44:0/2264096664 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7fcc2c08b7a0 con 0x7fcc340cec60 2024-04-02T13:32:22.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.136+0000 7fcc2affd640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x7fcc18072ed0 0x7fcc18075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.136+0000 7fcc2affd640 1 -- 172.21.15.44:0/2264096664 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc2c00bf40 con 0x7fcc18072ed0 2024-04-02T13:32:22.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.136+0000 7fcc2affd640 1 -- 172.21.15.44:0/2264096664 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=126) ==== 24+0+0 (secure 0 0 0) 0x7fcc2c0902c0 con 0x7fcc340cec60 2024-04-02T13:32:22.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.136+0000 7fcc39a0b640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcc1806c8f0 0x7fcc1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.136+0000 7fcc3aa0d640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x7fcc18072ed0 0x7fcc18075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.136+0000 7fcc39a0b640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcc1806c8f0 0x7fcc1806edb0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fcc24036000 tx=0x7fcc2403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.137+0000 7fcc3aa0d640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x7fcc18072ed0 0x7fcc18075310 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.137+0000 7fcc2affd640 1 -- 172.21.15.44:0/2264096664 <== osd.1 v2:172.21.15.44:6801/704457846 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc2c00bf40 con 0x7fcc18072ed0 2024-04-02T13:32:22.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.150+0000 7fcc3c470640 1 -- 172.21.15.44:0/2264096664 --> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcc340d45a0 con 0x7fcc18072ed0 2024-04-02T13:32:22.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.151+0000 7fcc2affd640 1 -- 172.21.15.44:0/2264096664 <== osd.1 v2:172.21.15.44:6801/704457846 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcc340d45a0 con 0x7fcc18072ed0 2024-04-02T13:32:22.153 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:32:22.154 INFO:teuthology.orchestra.run.smithi044.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T13:32:22.154 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:32:22.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.151+0000 7fcc3c470640 1 -- 172.21.15.44:0/2264096664 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x7fcc18072ed0 msgr2=0x7fcc18075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.151+0000 7fcc3c470640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.44:6801/704457846,v1:172.21.15.44:6803/704457846] conn(0x7fcc18072ed0 0x7fcc18075310 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.151+0000 7fcc3c470640 1 -- 172.21.15.44:0/2264096664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcc1806c8f0 msgr2=0x7fcc1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.151+0000 7fcc3c470640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcc1806c8f0 0x7fcc1806edb0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fcc24036000 tx=0x7fcc2403d040 comp rx=0 tx=0).stop 2024-04-02T13:32:22.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.151+0000 7fcc3c470640 1 -- 172.21.15.44:0/2264096664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc340cec60 msgr2=0x7fcc340b6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.151+0000 7fcc3c470640 1 --2- 172.21.15.44:0/2264096664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc340cec60 0x7fcc340b6d00 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00e810 tx=0x7fcc2c00ecd0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.151+0000 7fcc3aa0d640 1 -- 172.21.15.44:0/2264096664 reap_dead start 2024-04-02T13:32:22.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.151+0000 7fcc3c470640 1 -- 172.21.15.44:0/2264096664 shutdown_connections 2024-04-02T13:32:22.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.151+0000 7fcc3c470640 1 -- 172.21.15.44:0/2264096664 >> 172.21.15.44:0/2264096664 conn(0x7fcc3409d3c0 msgr2=0x7fcc340d0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:22.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.151+0000 7fcc3c470640 1 -- 172.21.15.44:0/2264096664 shutdown_connections 2024-04-02T13:32:22.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.151+0000 7fcc3c470640 1 -- 172.21.15.44:0/2264096664 wait complete. 2024-04-02T13:32:22.164 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-02T13:32:22.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.265+0000 7f0f01e50640 1 Processor -- start 2024-04-02T13:32:22.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.266+0000 7f0f01e50640 1 -- start start 2024-04-02T13:32:22.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.266+0000 7f0f01e50640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0efc15e4d0 0x7f0efc0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.266+0000 7f0f01e50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0efc0cfd20 0x7f0efc0d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.266+0000 7f0f01e50640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0efc0d1060 0x7f0efc0d1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.266+0000 7f0f01e50640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f0efc0b3a90 con 0x7f0efc15e4d0 2024-04-02T13:32:22.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.266+0000 7f0f01e50640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0efc0b4610 con 0x7f0efc0d1060 2024-04-02T13:32:22.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.266+0000 7f0f01e50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0efc15e8b0 con 0x7f0efc0cfd20 2024-04-02T13:32:22.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.266+0000 7f0efb7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0efc15e4d0 0x7f0efc0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.267+0000 7f0efb7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f0efc15e4d0 0x7f0efc0cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42470/0 (socket says 172.21.15.44:42470) 2024-04-02T13:32:22.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.267+0000 7f0efb7fe640 1 -- 172.21.15.44:0/1054818658 learned_addr learned my addr 172.21.15.44:0/1054818658 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:22.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.267+0000 7f0efbfff640 1 --2- 172.21.15.44:0/1054818658 >> v2:172.21.15.69:3300/0 conn(0x7f0efc0d1060 0x7f0efc0d1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.267+0000 7f0efaffd640 1 --2- 172.21.15.44:0/1054818658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0efc0cfd20 0x7f0efc0d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.267+0000 7f0efb7fe640 1 -- 172.21.15.44:0/1054818658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0efc0cfd20 msgr2=0x7f0efc0d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.267+0000 7f0efb7fe640 1 --2- 172.21.15.44:0/1054818658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0efc0cfd20 0x7f0efc0d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.267+0000 7f0efb7fe640 1 -- 172.21.15.44:0/1054818658 >> v2:172.21.15.69:3300/0 conn(0x7f0efc0d1060 msgr2=0x7f0efc0d1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.267+0000 7f0efb7fe640 1 --2- 172.21.15.44:0/1054818658 >> v2:172.21.15.69:3300/0 conn(0x7f0efc0d1060 0x7f0efc0d1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.267+0000 7f0efb7fe640 1 -- 172.21.15.44:0/1054818658 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc0d2fb0 con 0x7f0efc15e4d0 2024-04-02T13:32:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.267+0000 7f0efaffd640 1 --2- 172.21.15.44:0/1054818658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0efc0cfd20 0x7f0efc0d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.268+0000 7f0efb7fe640 1 --2- 172.21.15.44:0/1054818658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0efc15e4d0 0x7f0efc0cf7e0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0ce710 tx=0x7f0eec02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c109ca1bd0f472 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.268+0000 7f0efa7fc640 1 -- 172.21.15.44:0/1054818658 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec035020 con 0x7f0efc15e4d0 2024-04-02T13:32:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.268+0000 7f0efa7fc640 1 -- 172.21.15.44:0/1054818658 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eec02d040 con 0x7f0efc15e4d0 2024-04-02T13:32:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.268+0000 7f0efa7fc640 1 -- 172.21.15.44:0/1054818658 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec031a80 con 0x7f0efc15e4d0 2024-04-02T13:32:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.268+0000 7f0f01e50640 1 -- 172.21.15.44:0/1054818658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0efc15e4d0 msgr2=0x7f0efc0cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.268+0000 7f0f01e50640 1 --2- 172.21.15.44:0/1054818658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0efc15e4d0 0x7f0efc0cf7e0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0ce710 tx=0x7f0eec02b900 comp rx=0 tx=0).stop 2024-04-02T13:32:22.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.268+0000 7f0f01e50640 1 -- 172.21.15.44:0/1054818658 shutdown_connections 2024-04-02T13:32:22.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.268+0000 7f0f01e50640 1 --2- 172.21.15.44:0/1054818658 >> v2:172.21.15.69:3300/0 conn(0x7f0efc0d1060 0x7f0efc0d1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.268+0000 7f0f01e50640 1 --2- 172.21.15.44:0/1054818658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0efc0cfd20 0x7f0efc0d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.268+0000 7f0f01e50640 1 --2- 172.21.15.44:0/1054818658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0efc15e4d0 0x7f0efc0cf7e0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.268+0000 7f0f01e50640 1 -- 172.21.15.44:0/1054818658 >> 172.21.15.44:0/1054818658 conn(0x7f0efc09d410 msgr2=0x7f0efc0d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:22.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.269+0000 7f0f01e50640 1 -- 172.21.15.44:0/1054818658 shutdown_connections 2024-04-02T13:32:22.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.269+0000 7f0f01e50640 1 -- 172.21.15.44:0/1054818658 wait complete. 2024-04-02T13:32:22.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.270+0000 7f0f01e50640 1 Processor -- start 2024-04-02T13:32:22.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.270+0000 7f0f01e50640 1 -- start start 2024-04-02T13:32:22.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.270+0000 7f0f01e50640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0efc0cfd20 0x7f0efc160970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.270+0000 7f0f01e50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0efc0d1060 0x7f0efc15be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.270+0000 7f0f01e50640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0efc15e4d0 0x7f0efc0cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.270+0000 7f0f01e50640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc0b42a0 con 0x7f0efc0cfd20 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.270+0000 7f0f01e50640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc0b3fc0 con 0x7f0efc15e4d0 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.270+0000 7f0f01e50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0efc0b0d00 con 0x7f0efc0d1060 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0efb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0efc0cfd20 0x7f0efc160970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0efb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0efc0cfd20 0x7f0efc160970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42484/0 (socket says 172.21.15.44:42484) 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0efb7fe640 1 -- 172.21.15.44:0/3151427203 learned_addr learned my addr 172.21.15.44:0/3151427203 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0efbfff640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0efc15e4d0 0x7f0efc0cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0efaffd640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0efc0d1060 0x7f0efc15be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0efb7fe640 1 -- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0efc0d1060 msgr2=0x7f0efc15be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0efb7fe640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0efc0d1060 0x7f0efc15be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0efb7fe640 1 -- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0efc15e4d0 msgr2=0x7f0efc0cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0efb7fe640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0efc15e4d0 0x7f0efc0cd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0efb7fe640 1 -- 172.21.15.44:0/3151427203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc15c890 con 0x7f0efc0cfd20 2024-04-02T13:32:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0efbfff640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0efc15e4d0 0x7f0efc0cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0efaffd640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0efc0d1060 0x7f0efc15be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0efb7fe640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0efc0cfd20 0x7f0efc160970 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f0eec000c00 tx=0x7f0eec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0ef3fff640 1 -- 172.21.15.44:0/3151427203 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec03aa60 con 0x7f0efc0cfd20 2024-04-02T13:32:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0ef3fff640 1 -- 172.21.15.44:0/3151427203 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eec03ac00 con 0x7f0efc0cfd20 2024-04-02T13:32:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0f01e50640 1 -- 172.21.15.44:0/3151427203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0efc0cd5a0 con 0x7f0efc0cfd20 2024-04-02T13:32:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0ef3fff640 1 -- 172.21.15.44:0/3151427203 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec0443f0 con 0x7f0efc0cfd20 2024-04-02T13:32:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.271+0000 7f0f01e50640 1 -- 172.21.15.44:0/3151427203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0efc15fbb0 con 0x7f0efc0cfd20 2024-04-02T13:32:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.272+0000 7f0f01e50640 1 -- 172.21.15.44:0/3151427203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0efc068c00 con 0x7f0efc0cfd20 2024-04-02T13:32:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.273+0000 7f0ef3fff640 1 -- 172.21.15.44:0/3151427203 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0eec04b060 con 0x7f0efc0cfd20 2024-04-02T13:32:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.273+0000 7f0ef3fff640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0edc06c8f0 0x7f0edc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.273+0000 7f0ef3fff640 1 -- 172.21.15.44:0/3151427203 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f0eec0ad940 con 0x7f0efc0cfd20 2024-04-02T13:32:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.273+0000 7f0efaffd640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0edc06c8f0 0x7f0edc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.274+0000 7f0efaffd640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0edc06c8f0 0x7f0edc06edb0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0b1db0 tx=0x7f0ee4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.278+0000 7f0ef3fff640 1 -- 172.21.15.44:0/3151427203 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f0efc15c890 con 0x7f0efc0cfd20 2024-04-02T13:32:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.463+0000 7f0f01e50640 1 -- 172.21.15.44:0/3151427203 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f0efc0b9ff0 con 0x7f0edc06c8f0 2024-04-02T13:32:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.463+0000 7f0ef3fff640 1 -- 172.21.15.44:0/3151427203 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0efc0b9ff0 con 0x7f0edc06c8f0 2024-04-02T13:32:22.467 INFO:teuthology.orchestra.run.smithi044.stderr:instructed osd(s) 1 to deep-scrub 2024-04-02T13:32:22.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.465+0000 7f0f01e50640 1 -- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0edc06c8f0 msgr2=0x7f0edc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.465+0000 7f0f01e50640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0edc06c8f0 0x7f0edc06edb0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0b1db0 tx=0x7f0ee4008040 comp rx=0 tx=0).stop 2024-04-02T13:32:22.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.465+0000 7f0f01e50640 1 -- 172.21.15.44:0/3151427203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0efc0cfd20 msgr2=0x7f0efc160970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.465+0000 7f0f01e50640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0efc0cfd20 0x7f0efc160970 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f0eec000c00 tx=0x7f0eec0046c0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.465+0000 7f0f01e50640 1 -- 172.21.15.44:0/3151427203 shutdown_connections 2024-04-02T13:32:22.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.465+0000 7f0f01e50640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f0edc06c8f0 0x7f0edc06edb0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.465+0000 7f0f01e50640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0efc15e4d0 0x7f0efc0cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.465+0000 7f0f01e50640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0efc0d1060 0x7f0efc15be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.465+0000 7f0f01e50640 1 --2- 172.21.15.44:0/3151427203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0efc0cfd20 0x7f0efc160970 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.465+0000 7f0f01e50640 1 -- 172.21.15.44:0/3151427203 >> 172.21.15.44:0/3151427203 conn(0x7f0efc09d410 msgr2=0x7f0efc0b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:22.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.466+0000 7f0f01e50640 1 -- 172.21.15.44:0/3151427203 shutdown_connections 2024-04-02T13:32:22.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.466+0000 7f0f01e50640 1 -- 172.21.15.44:0/3151427203 wait complete. 2024-04-02T13:32:22.479 INFO:tasks.ceph:Scrubbing osd.2 2024-04-02T13:32:22.479 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-02T13:32:22.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.583+0000 7f82a6d0e640 1 Processor -- start 2024-04-02T13:32:22.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.583+0000 7f82a6d0e640 1 -- start start 2024-04-02T13:32:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.584+0000 7f82a6d0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a00d2090 0x7f82a006f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.584+0000 7f82a6d0e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f82a00d00d0 0x7f82a015ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.584+0000 7f82a6d0e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82a0070040 con 0x7f82a006fac0 2024-04-02T13:32:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.584+0000 7f82a6d0e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f82a006c5e0 con 0x7f82a00d00d0 2024-04-02T13:32:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.584+0000 7f82a6d0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f82a0070320 con 0x7f82a00d2090 2024-04-02T13:32:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.584+0000 7f82a4aaa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a00d2090 0x7f82a006f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.584+0000 7f829ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f82a00d00d0 0x7f82a015ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.584+0000 7f82a4aaa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a00d2090 0x7f82a006f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:37518/0 (socket says 172.21.15.44:37518) 2024-04-02T13:32:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.584+0000 7f82a4aaa640 1 -- 172.21.15.44:0/4256232236 learned_addr learned my addr 172.21.15.44:0/4256232236 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.585+0000 7f82a4aaa640 1 -- 172.21.15.44:0/4256232236 >> v2:172.21.15.69:3300/0 conn(0x7f82a00d00d0 msgr2=0x7f82a015ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.585+0000 7f82a4aaa640 1 --2- 172.21.15.44:0/4256232236 >> v2:172.21.15.69:3300/0 conn(0x7f82a00d00d0 0x7f82a015ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.585+0000 7f82a4aaa640 1 -- 172.21.15.44:0/4256232236 >> v1:172.21.15.69:6789/0 conn(0x7f82a006fac0 legacy=0x7f82a0161e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.585+0000 7f82a4aaa640 1 -- 172.21.15.44:0/4256232236 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a0070600 con 0x7f82a00d2090 2024-04-02T13:32:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.585+0000 7f829ffff640 1 --2- 172.21.15.44:0/4256232236 >> v2:172.21.15.69:3300/0 conn(0x7f82a00d00d0 0x7f82a015ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T13:32:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.585+0000 7f82a4aaa640 1 --2- 172.21.15.44:0/4256232236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a00d2090 0x7f82a006f580 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8294009870 tx=0x7f829402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61d0c20499a20f61 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.586+0000 7f829f7fe640 1 -- 172.21.15.44:0/4256232236 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294034070 con 0x7f82a00d2090 2024-04-02T13:32:22.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.586+0000 7f829f7fe640 1 -- 172.21.15.44:0/4256232236 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8294030840 con 0x7f82a00d2090 2024-04-02T13:32:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.586+0000 7f829f7fe640 1 -- 172.21.15.44:0/4256232236 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294030c10 con 0x7f82a00d2090 2024-04-02T13:32:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.586+0000 7f82a6d0e640 1 -- 172.21.15.44:0/4256232236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a00d2090 msgr2=0x7f82a006f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.586+0000 7f82a6d0e640 1 --2- 172.21.15.44:0/4256232236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a00d2090 0x7f82a006f580 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8294009870 tx=0x7f829402b980 comp rx=0 tx=0).stop 2024-04-02T13:32:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.586+0000 7f82a6d0e640 1 -- 172.21.15.44:0/4256232236 shutdown_connections 2024-04-02T13:32:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.586+0000 7f82a6d0e640 1 --2- 172.21.15.44:0/4256232236 >> v2:172.21.15.69:3300/0 conn(0x7f82a00d00d0 0x7f82a015ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.586+0000 7f82a6d0e640 1 --2- 172.21.15.44:0/4256232236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a00d2090 0x7f82a006f580 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.586+0000 7f82a6d0e640 1 -- 172.21.15.44:0/4256232236 >> 172.21.15.44:0/4256232236 conn(0x7f82a009aa20 msgr2=0x7f82a009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.586+0000 7f82a6d0e640 1 -- 172.21.15.44:0/4256232236 shutdown_connections 2024-04-02T13:32:22.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.586+0000 7f82a6d0e640 1 -- 172.21.15.44:0/4256232236 wait complete. 2024-04-02T13:32:22.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.587+0000 7f82a6d0e640 1 Processor -- start 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.587+0000 7f82a6d0e640 1 -- start start 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.588+0000 7f82a6d0e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82a006fac0 0x7f82a009c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.588+0000 7f82a6d0e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82a00d00d0 0x7f82a0152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.588+0000 7f82a6d0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a00d2090 0x7f82a00b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.588+0000 7f82a6d0e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f82a00cead0 con 0x7f82a00d00d0 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.588+0000 7f82a6d0e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f82a00ce950 con 0x7f82a006fac0 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.588+0000 7f82a6d0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f82a00d2470 con 0x7f82a00d2090 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.588+0000 7f829ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82a00d00d0 0x7f82a0152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.588+0000 7f829ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82a00d00d0 0x7f82a0152550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42488/0 (socket says 172.21.15.44:42488) 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.588+0000 7f829ffff640 1 -- 172.21.15.44:0/3068687978 learned_addr learned my addr 172.21.15.44:0/3068687978 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.588+0000 7f82a4aaa640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82a006fac0 0x7f82a009c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.588+0000 7f82a52ab640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a00d2090 0x7f82a00b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.589+0000 7f829ffff640 1 -- 172.21.15.44:0/3068687978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a00d2090 msgr2=0x7f82a00b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.589+0000 7f829ffff640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a00d2090 0x7f82a00b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.589+0000 7f829ffff640 1 -- 172.21.15.44:0/3068687978 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82a006fac0 msgr2=0x7f82a009c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.589+0000 7f829ffff640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82a006fac0 0x7f82a009c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.589+0000 7f829ffff640 1 -- 172.21.15.44:0/3068687978 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a0158340 con 0x7f82a00d00d0 2024-04-02T13:32:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.589+0000 7f829ffff640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82a00d00d0 0x7f82a0152550 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f828c00ebe0 tx=0x7f828c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.589+0000 7f829d7fa640 1 -- 172.21.15.44:0/3068687978 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c017070 con 0x7f82a00d00d0 2024-04-02T13:32:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.589+0000 7f829d7fa640 1 -- 172.21.15.44:0/3068687978 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f828c00cc30 con 0x7f82a00d00d0 2024-04-02T13:32:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.589+0000 7f82a6d0e640 1 -- 172.21.15.44:0/3068687978 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a0158620 con 0x7f82a00d00d0 2024-04-02T13:32:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.589+0000 7f829d7fa640 1 -- 172.21.15.44:0/3068687978 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c013470 con 0x7f82a00d00d0 2024-04-02T13:32:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.589+0000 7f82a6d0e640 1 -- 172.21.15.44:0/3068687978 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a0158be0 con 0x7f82a00d00d0 2024-04-02T13:32:22.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.590+0000 7f82a6d0e640 1 -- 172.21.15.44:0/3068687978 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f82a0068c00 con 0x7f82a00d00d0 2024-04-02T13:32:22.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.591+0000 7f829d7fa640 1 -- 172.21.15.44:0/3068687978 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f828c01a070 con 0x7f82a00d00d0 2024-04-02T13:32:22.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.591+0000 7f829d7fa640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f828406c8f0 0x7f828406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.591+0000 7f829d7fa640 1 -- 172.21.15.44:0/3068687978 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f828c08bc00 con 0x7f82a00d00d0 2024-04-02T13:32:22.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.591+0000 7f829d7fa640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x7f8284072ed0 0x7f8284075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.591+0000 7f829d7fa640 1 -- 172.21.15.44:0/3068687978 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f828c0093e0 con 0x7f8284072ed0 2024-04-02T13:32:22.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.591+0000 7f829d7fa640 1 -- 172.21.15.44:0/3068687978 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=126) ==== 24+0+0 (secure 0 0 0) 0x7f828c08e300 con 0x7f82a00d00d0 2024-04-02T13:32:22.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.591+0000 7f82a4aaa640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f828406c8f0 0x7f828406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.591+0000 7f82a52ab640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x7f8284072ed0 0x7f8284075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.592+0000 7f82a4aaa640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f828406c8f0 0x7f828406edb0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8294036000 tx=0x7f829403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.592+0000 7f82a52ab640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x7f8284072ed0 0x7f8284075310 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.592+0000 7f829d7fa640 1 -- 172.21.15.44:0/3068687978 <== osd.2 v2:172.21.15.44:6816/2091834331 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f828c0093e0 con 0x7f8284072ed0 2024-04-02T13:32:22.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.606+0000 7f82a6d0e640 1 -- 172.21.15.44:0/3068687978 --> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f82a015b570 con 0x7f8284072ed0 2024-04-02T13:32:22.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.606+0000 7f829d7fa640 1 -- 172.21.15.44:0/3068687978 <== osd.2 v2:172.21.15.44:6816/2091834331 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f82a015b570 con 0x7f8284072ed0 2024-04-02T13:32:22.609 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:32:22.609 INFO:teuthology.orchestra.run.smithi044.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T13:32:22.609 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:32:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.606+0000 7f82a6d0e640 1 -- 172.21.15.44:0/3068687978 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x7f8284072ed0 msgr2=0x7f8284075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.606+0000 7f82a6d0e640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.44:6816/2091834331,v1:172.21.15.44:6817/2091834331] conn(0x7f8284072ed0 0x7f8284075310 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.606+0000 7f82a6d0e640 1 -- 172.21.15.44:0/3068687978 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f828406c8f0 msgr2=0x7f828406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.606+0000 7f82a6d0e640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f828406c8f0 0x7f828406edb0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8294036000 tx=0x7f829403b040 comp rx=0 tx=0).stop 2024-04-02T13:32:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.606+0000 7f82a6d0e640 1 -- 172.21.15.44:0/3068687978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82a00d00d0 msgr2=0x7f82a0152550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.606+0000 7f82a6d0e640 1 --2- 172.21.15.44:0/3068687978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82a00d00d0 0x7f82a0152550 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f828c00ebe0 tx=0x7f828c00c520 comp rx=0 tx=0).stop 2024-04-02T13:32:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.606+0000 7f82a52ab640 1 -- 172.21.15.44:0/3068687978 reap_dead start 2024-04-02T13:32:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.606+0000 7f82a6d0e640 1 -- 172.21.15.44:0/3068687978 shutdown_connections 2024-04-02T13:32:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.606+0000 7f82a6d0e640 1 -- 172.21.15.44:0/3068687978 >> 172.21.15.44:0/3068687978 conn(0x7f82a009aa20 msgr2=0x7f82a00b0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.606+0000 7f82a6d0e640 1 -- 172.21.15.44:0/3068687978 shutdown_connections 2024-04-02T13:32:22.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.606+0000 7f82a6d0e640 1 -- 172.21.15.44:0/3068687978 wait complete. 2024-04-02T13:32:22.619 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-02T13:32:22.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.719+0000 7f65263f6640 1 Processor -- start 2024-04-02T13:32:22.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.719+0000 7f65263f6640 1 -- start start 2024-04-02T13:32:22.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f65263f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f652015e430 0x7f65200ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f65263f6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f652006c7f0 0x7f65200ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f65263f6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f652015e810 con 0x7f65200cf240 2024-04-02T13:32:22.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f65263f6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f65200d1410 con 0x7f652006c7f0 2024-04-02T13:32:22.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f65263f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f652006cff0 con 0x7f652015e430 2024-04-02T13:32:22.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f6524993640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f65200cf240 0x7f65200d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52380/0 (socket says 172.21.15.44:52380) 2024-04-02T13:32:22.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f6524993640 1 -- 172.21.15.44:0/2565067613 learned_addr learned my addr 172.21.15.44:0/2565067613 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:22.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f651ffff640 1 --2- 172.21.15.44:0/2565067613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f652015e430 0x7f65200ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f651f7fe640 1 --2- 172.21.15.44:0/2565067613 >> v2:172.21.15.69:3300/0 conn(0x7f652006c7f0 0x7f65200ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f651effd640 1 -- 172.21.15.44:0/2565067613 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2249568075 0 0) 0x7f652015e810 con 0x7f65200cf240 2024-04-02T13:32:22.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f651effd640 1 -- 172.21.15.44:0/2565067613 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f650c003520 con 0x7f65200cf240 2024-04-02T13:32:22.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f651ffff640 1 -- 172.21.15.44:0/2565067613 >> v2:172.21.15.69:3300/0 conn(0x7f652006c7f0 msgr2=0x7f65200ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:22.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f651ffff640 1 --2- 172.21.15.44:0/2565067613 >> v2:172.21.15.69:3300/0 conn(0x7f652006c7f0 0x7f65200ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f651ffff640 1 -- 172.21.15.44:0/2565067613 >> v1:172.21.15.44:6789/0 conn(0x7f65200cf240 legacy=0x7f65200d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.720+0000 7f651ffff640 1 -- 172.21.15.44:0/2565067613 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652006d240 con 0x7f652015e430 2024-04-02T13:32:22.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.721+0000 7f651ffff640 1 --2- 172.21.15.44:0/2565067613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f652015e430 0x7f65200ce7c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f65080097a0 tx=0x7f650802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5330e856130886ca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.721+0000 7f651effd640 1 -- 172.21.15.44:0/2565067613 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65080315a0 con 0x7f652015e430 2024-04-02T13:32:22.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.721+0000 7f651effd640 1 -- 172.21.15.44:0/2565067613 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6508037070 con 0x7f652015e430 2024-04-02T13:32:22.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.721+0000 7f651effd640 1 -- 172.21.15.44:0/2565067613 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6508031940 con 0x7f652015e430 2024-04-02T13:32:22.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.722+0000 7f65263f6640 1 -- 172.21.15.44:0/2565067613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f652015e430 msgr2=0x7f65200ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.722+0000 7f65263f6640 1 --2- 172.21.15.44:0/2565067613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f652015e430 0x7f65200ce7c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f65080097a0 tx=0x7f650802b840 comp rx=0 tx=0).stop 2024-04-02T13:32:22.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.722+0000 7f65263f6640 1 -- 172.21.15.44:0/2565067613 shutdown_connections 2024-04-02T13:32:22.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.722+0000 7f65263f6640 1 --2- 172.21.15.44:0/2565067613 >> v2:172.21.15.69:3300/0 conn(0x7f652006c7f0 0x7f65200ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.722+0000 7f65263f6640 1 --2- 172.21.15.44:0/2565067613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f652015e430 0x7f65200ce7c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.722+0000 7f65263f6640 1 -- 172.21.15.44:0/2565067613 >> 172.21.15.44:0/2565067613 conn(0x7f6520073850 msgr2=0x7f652009aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.722+0000 7f65263f6640 1 -- 172.21.15.44:0/2565067613 shutdown_connections 2024-04-02T13:32:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.722+0000 7f65263f6640 1 -- 172.21.15.44:0/2565067613 wait complete. 2024-04-02T13:32:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.723+0000 7f65263f6640 1 Processor -- start 2024-04-02T13:32:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.723+0000 7f65263f6640 1 -- start start 2024-04-02T13:32:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f65263f6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f652006c7f0 0x7f65200702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f65263f6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65200cf240 0x7f6520150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f65263f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f652015e430 0x7f6520151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f65263f6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f65200d0ef0 con 0x7f65200cf240 2024-04-02T13:32:22.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f65263f6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f65200d0c10 con 0x7f652006c7f0 2024-04-02T13:32:22.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f65263f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f652006d800 con 0x7f652015e430 2024-04-02T13:32:22.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f651f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65200cf240 0x7f6520150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f651f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65200cf240 0x7f6520150f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42504/0 (socket says 172.21.15.44:42504) 2024-04-02T13:32:22.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f651f7fe640 1 -- 172.21.15.44:0/4093305350 learned_addr learned my addr 172.21.15.44:0/4093305350 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:22.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f651ffff640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f652006c7f0 0x7f65200702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f6524993640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f652015e430 0x7f6520151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f651f7fe640 1 -- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f652015e430 msgr2=0x7f6520151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f651f7fe640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f652015e430 0x7f6520151450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f651f7fe640 1 -- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f652006c7f0 msgr2=0x7f65200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f651f7fe640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f652006c7f0 0x7f65200702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.724+0000 7f651f7fe640 1 -- 172.21.15.44:0/4093305350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65200b59c0 con 0x7f65200cf240 2024-04-02T13:32:22.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.725+0000 7f651f7fe640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65200cf240 0x7f6520150f10 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f651000e870 tx=0x7f651000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.725+0000 7f651cff9640 1 -- 172.21.15.44:0/4093305350 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6510017070 con 0x7f65200cf240 2024-04-02T13:32:22.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.725+0000 7f651cff9640 1 -- 172.21.15.44:0/4093305350 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f651000c940 con 0x7f65200cf240 2024-04-02T13:32:22.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.725+0000 7f65263f6640 1 -- 172.21.15.44:0/4093305350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65200b5d40 con 0x7f65200cf240 2024-04-02T13:32:22.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.725+0000 7f651cff9640 1 -- 172.21.15.44:0/4093305350 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651000cd30 con 0x7f65200cf240 2024-04-02T13:32:22.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.725+0000 7f65263f6640 1 -- 172.21.15.44:0/4093305350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65200b6300 con 0x7f65200cf240 2024-04-02T13:32:22.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.726+0000 7f65263f6640 1 -- 172.21.15.44:0/4093305350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6520068c00 con 0x7f65200cf240 2024-04-02T13:32:22.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.726+0000 7f651cff9640 1 -- 172.21.15.44:0/4093305350 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f651001a070 con 0x7f65200cf240 2024-04-02T13:32:22.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.727+0000 7f651cff9640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f650006c8f0 0x7f650006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:22.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.727+0000 7f651cff9640 1 -- 172.21.15.44:0/4093305350 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f651008bc70 con 0x7f65200cf240 2024-04-02T13:32:22.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.727+0000 7f651ffff640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f650006c8f0 0x7f650006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:22.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.727+0000 7f651ffff640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f650006c8f0 0x7f650006edb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6508036000 tx=0x7f650803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:22.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.731+0000 7f651cff9640 1 -- 172.21.15.44:0/4093305350 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6510059b10 con 0x7f65200cf240 2024-04-02T13:32:22.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.930+0000 7f65263f6640 1 -- 172.21.15.44:0/4093305350 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f65200cc6a0 con 0x7f650006c8f0 2024-04-02T13:32:22.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.931+0000 7f651cff9640 1 -- 172.21.15.44:0/4093305350 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f65200cc6a0 con 0x7f650006c8f0 2024-04-02T13:32:22.934 INFO:teuthology.orchestra.run.smithi044.stderr:instructed osd(s) 2 to deep-scrub 2024-04-02T13:32:22.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.933+0000 7f65263f6640 1 -- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f650006c8f0 msgr2=0x7f650006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.933+0000 7f65263f6640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f650006c8f0 0x7f650006edb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6508036000 tx=0x7f650803d040 comp rx=0 tx=0).stop 2024-04-02T13:32:22.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.933+0000 7f65263f6640 1 -- 172.21.15.44:0/4093305350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65200cf240 msgr2=0x7f6520150f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:22.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.933+0000 7f65263f6640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65200cf240 0x7f6520150f10 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f651000e870 tx=0x7f651000ed30 comp rx=0 tx=0).stop 2024-04-02T13:32:22.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.933+0000 7f65263f6640 1 -- 172.21.15.44:0/4093305350 shutdown_connections 2024-04-02T13:32:22.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.933+0000 7f65263f6640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f650006c8f0 0x7f650006edb0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.933+0000 7f65263f6640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f652015e430 0x7f6520151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.933+0000 7f65263f6640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65200cf240 0x7f6520150f10 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.933+0000 7f65263f6640 1 --2- 172.21.15.44:0/4093305350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f652006c7f0 0x7f65200702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:22.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.933+0000 7f65263f6640 1 -- 172.21.15.44:0/4093305350 >> 172.21.15.44:0/4093305350 conn(0x7f6520073850 msgr2=0x7f65200d15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:22.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.933+0000 7f65263f6640 1 -- 172.21.15.44:0/4093305350 shutdown_connections 2024-04-02T13:32:22.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:22.933+0000 7f65263f6640 1 -- 172.21.15.44:0/4093305350 wait complete. 2024-04-02T13:32:22.946 INFO:tasks.ceph:Scrubbing osd.3 2024-04-02T13:32:22.946 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-02T13:32:23.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.049+0000 7f19df904640 1 Processor -- start 2024-04-02T13:32:23.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.049+0000 7f19df904640 1 -- start start 2024-04-02T13:32:23.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.050+0000 7f19df904640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19d815e470 0x7f19d80b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.050+0000 7f19df904640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19d80ced00 con 0x7f19d80ce790 2024-04-02T13:32:23.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.050+0000 7f19df904640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19d80cefe0 con 0x7f19d80b1540 2024-04-02T13:32:23.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.050+0000 7f19df904640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f19d80cf2c0 con 0x7f19d815e470 2024-04-02T13:32:23.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.050+0000 7f19ddea1640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f19d80ce790 0x7f19d806d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52394/0 (socket says 172.21.15.44:52394) 2024-04-02T13:32:23.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.050+0000 7f19ddea1640 1 -- 172.21.15.44:0/1972674408 learned_addr learned my addr 172.21.15.44:0/1972674408 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:23.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.050+0000 7f19dd6a0640 1 --2- 172.21.15.44:0/1972674408 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19d815e470 0x7f19d80b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.051+0000 7f19cffff640 1 -- 172.21.15.44:0/1972674408 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 811681956 0 0) 0x7f19d80ced00 con 0x7f19d80ce790 2024-04-02T13:32:23.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.051+0000 7f19cffff640 1 -- 172.21.15.44:0/1972674408 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19c4003520 con 0x7f19d80ce790 2024-04-02T13:32:23.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.051+0000 7f19cffff640 1 -- 172.21.15.44:0/1972674408 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3062359834 0 0) 0x7f19d80cefe0 con 0x7f19d80b1540 2024-04-02T13:32:23.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.051+0000 7f19cffff640 1 -- 172.21.15.44:0/1972674408 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19d80ced00 con 0x7f19d80b1540 2024-04-02T13:32:23.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.051+0000 7f19dd6a0640 1 -- 172.21.15.44:0/1972674408 >> v1:172.21.15.69:6789/0 conn(0x7f19d80b1540 legacy=0x7f19d80d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.051+0000 7f19dd6a0640 1 -- 172.21.15.44:0/1972674408 >> v1:172.21.15.44:6789/0 conn(0x7f19d80ce790 legacy=0x7f19d806d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.051+0000 7f19dd6a0640 1 -- 172.21.15.44:0/1972674408 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d80cf5a0 con 0x7f19d815e470 2024-04-02T13:32:23.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.051+0000 7f19dd6a0640 1 --2- 172.21.15.44:0/1972674408 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19d815e470 0x7f19d80b32c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f19d00097d0 tx=0x7f19d002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11d255ccc8b50831 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.052+0000 7f19cffff640 1 -- 172.21.15.44:0/1972674408 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0035020 con 0x7f19d815e470 2024-04-02T13:32:23.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.052+0000 7f19cffff640 1 -- 172.21.15.44:0/1972674408 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19d0031800 con 0x7f19d815e470 2024-04-02T13:32:23.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.052+0000 7f19cffff640 1 -- 172.21.15.44:0/1972674408 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0031bd0 con 0x7f19d815e470 2024-04-02T13:32:23.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.052+0000 7f19df904640 1 -- 172.21.15.44:0/1972674408 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19d815e470 msgr2=0x7f19d80b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.052+0000 7f19df904640 1 --2- 172.21.15.44:0/1972674408 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19d815e470 0x7f19d80b32c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f19d00097d0 tx=0x7f19d002b8a0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.052+0000 7f19df904640 1 -- 172.21.15.44:0/1972674408 shutdown_connections 2024-04-02T13:32:23.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.052+0000 7f19df904640 1 --2- 172.21.15.44:0/1972674408 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19d815e470 0x7f19d80b32c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.052+0000 7f19df904640 1 -- 172.21.15.44:0/1972674408 >> 172.21.15.44:0/1972674408 conn(0x7f19d809aa20 msgr2=0x7f19d809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:23.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.052+0000 7f19df904640 1 -- 172.21.15.44:0/1972674408 shutdown_connections 2024-04-02T13:32:23.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.052+0000 7f19df904640 1 -- 172.21.15.44:0/1972674408 wait complete. 2024-04-02T13:32:23.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.053+0000 7f19df904640 1 Processor -- start 2024-04-02T13:32:23.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.053+0000 7f19df904640 1 -- start start 2024-04-02T13:32:23.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.054+0000 7f19df904640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19d80b1540 0x7f19d81617d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.054+0000 7f19df904640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d80ce790 0x7f19d8154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.054+0000 7f19df904640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d815e470 0x7f19d80b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.054+0000 7f19df904640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f19d80d22c0 con 0x7f19d815e470 2024-04-02T13:32:23.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.054+0000 7f19df904640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f19d806cb60 con 0x7f19d80ce790 2024-04-02T13:32:23.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.054+0000 7f19df904640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f19d80d2440 con 0x7f19d80b1540 2024-04-02T13:32:23.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.054+0000 7f19ddea1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d815e470 0x7f19d80b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.054+0000 7f19ddea1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d815e470 0x7f19d80b61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42508/0 (socket says 172.21.15.44:42508) 2024-04-02T13:32:23.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.054+0000 7f19ddea1640 1 -- 172.21.15.44:0/138001430 learned_addr learned my addr 172.21.15.44:0/138001430 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:23.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.054+0000 7f19dd6a0640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19d80b1540 0x7f19d81617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.054+0000 7f19dce9f640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d80ce790 0x7f19d8154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.055+0000 7f19ddea1640 1 -- 172.21.15.44:0/138001430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19d80b1540 msgr2=0x7f19d81617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.055+0000 7f19ddea1640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19d80b1540 0x7f19d81617d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.055+0000 7f19ddea1640 1 -- 172.21.15.44:0/138001430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d80ce790 msgr2=0x7f19d8154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.055+0000 7f19ddea1640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d80ce790 0x7f19d8154210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.055+0000 7f19ddea1640 1 -- 172.21.15.44:0/138001430 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d80c6f20 con 0x7f19d815e470 2024-04-02T13:32:23.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.055+0000 7f19ddea1640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d815e470 0x7f19d80b61d0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f19d400caa0 tx=0x7f19d400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.055+0000 7f19cdffb640 1 -- 172.21.15.44:0/138001430 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d4017070 con 0x7f19d815e470 2024-04-02T13:32:23.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.055+0000 7f19df904640 1 -- 172.21.15.44:0/138001430 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d80c7650 con 0x7f19d815e470 2024-04-02T13:32:23.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.055+0000 7f19cdffb640 1 -- 172.21.15.44:0/138001430 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19d400ac20 con 0x7f19d815e470 2024-04-02T13:32:23.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.055+0000 7f19cdffb640 1 -- 172.21.15.44:0/138001430 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d4012640 con 0x7f19d815e470 2024-04-02T13:32:23.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.055+0000 7f19df904640 1 -- 172.21.15.44:0/138001430 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d80c7c10 con 0x7f19d815e470 2024-04-02T13:32:23.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.056+0000 7f19df904640 1 -- 172.21.15.44:0/138001430 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f19d80684f0 con 0x7f19d815e470 2024-04-02T13:32:23.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.057+0000 7f19cdffb640 1 -- 172.21.15.44:0/138001430 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f19d401a070 con 0x7f19d815e470 2024-04-02T13:32:23.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.057+0000 7f19cdffb640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f19bc06c8f0 0x7f19bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.057+0000 7f19cdffb640 1 -- 172.21.15.44:0/138001430 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f19d408b7c0 con 0x7f19d815e470 2024-04-02T13:32:23.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.057+0000 7f19dd6a0640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f19bc06c8f0 0x7f19bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.057+0000 7f19cdffb640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x7f19bc072ed0 0x7f19bc075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.057+0000 7f19cdffb640 1 -- 172.21.15.44:0/138001430 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f19bc0759e0 con 0x7f19bc072ed0 2024-04-02T13:32:23.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.057+0000 7f19cdffb640 1 -- 172.21.15.44:0/138001430 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=126) ==== 24+0+0 (secure 0 0 0) 0x7f19d40902c0 con 0x7f19d815e470 2024-04-02T13:32:23.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.057+0000 7f19dce9f640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x7f19bc072ed0 0x7f19bc075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.058+0000 7f19dd6a0640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f19bc06c8f0 0x7f19bc06edb0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f19d0036000 tx=0x7f19d0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.058+0000 7f19dce9f640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x7f19bc072ed0 0x7f19bc075310 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.058+0000 7f19cdffb640 1 -- 172.21.15.44:0/138001430 <== osd.3 v2:172.21.15.44:6824/1588952513 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f19bc0759e0 con 0x7f19bc072ed0 2024-04-02T13:32:23.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.071+0000 7f19df904640 1 -- 172.21.15.44:0/138001430 --> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f19d80c8330 con 0x7f19bc072ed0 2024-04-02T13:32:23.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.072+0000 7f19cdffb640 1 -- 172.21.15.44:0/138001430 <== osd.3 v2:172.21.15.44:6824/1588952513 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f19d80c8330 con 0x7f19bc072ed0 2024-04-02T13:32:23.075 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:32:23.075 INFO:teuthology.orchestra.run.smithi044.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T13:32:23.075 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:32:23.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.072+0000 7f19df904640 1 -- 172.21.15.44:0/138001430 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x7f19bc072ed0 msgr2=0x7f19bc075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.072+0000 7f19df904640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.44:6824/1588952513,v1:172.21.15.44:6825/1588952513] conn(0x7f19bc072ed0 0x7f19bc075310 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.072+0000 7f19df904640 1 -- 172.21.15.44:0/138001430 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f19bc06c8f0 msgr2=0x7f19bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.072+0000 7f19df904640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f19bc06c8f0 0x7f19bc06edb0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f19d0036000 tx=0x7f19d0008030 comp rx=0 tx=0).stop 2024-04-02T13:32:23.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.072+0000 7f19df904640 1 -- 172.21.15.44:0/138001430 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d815e470 msgr2=0x7f19d80b61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.072+0000 7f19df904640 1 --2- 172.21.15.44:0/138001430 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d815e470 0x7f19d80b61d0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f19d400caa0 tx=0x7f19d400cf60 comp rx=0 tx=0).stop 2024-04-02T13:32:23.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.072+0000 7f19ddea1640 1 -- 172.21.15.44:0/138001430 reap_dead start 2024-04-02T13:32:23.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.072+0000 7f19df904640 1 -- 172.21.15.44:0/138001430 shutdown_connections 2024-04-02T13:32:23.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.072+0000 7f19df904640 1 -- 172.21.15.44:0/138001430 >> 172.21.15.44:0/138001430 conn(0x7f19d809aa20 msgr2=0x7f19d80afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:23.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.072+0000 7f19df904640 1 -- 172.21.15.44:0/138001430 shutdown_connections 2024-04-02T13:32:23.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.072+0000 7f19df904640 1 -- 172.21.15.44:0/138001430 wait complete. 2024-04-02T13:32:23.085 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-02T13:32:23.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.184+0000 7efcee433640 1 Processor -- start 2024-04-02T13:32:23.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.184+0000 7efcee433640 1 -- start start 2024-04-02T13:32:23.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.185+0000 7efcee433640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce806cf90 0x7efce80cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.185+0000 7efcee433640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efce806c1d0 con 0x7efce80cfe30 2024-04-02T13:32:23.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.185+0000 7efcee433640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efce806c4b0 con 0x7efce80d0a30 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.185+0000 7efcee433640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7efce806cb00 con 0x7efce806cf90 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.185+0000 7efcec9d0640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7efce80cfe30 0x7efce80d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52402/0 (socket says 172.21.15.44:52402) 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.185+0000 7efcec9d0640 1 -- 172.21.15.44:0/690227396 learned_addr learned my addr 172.21.15.44:0/690227396 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.185+0000 7efce77fe640 1 --2- 172.21.15.44:0/690227396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce806cf90 0x7efce80cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.186+0000 7efce6ffd640 1 -- 172.21.15.44:0/690227396 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3638557692 0 0) 0x7efce806c1d0 con 0x7efce80cfe30 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.186+0000 7efce6ffd640 1 -- 172.21.15.44:0/690227396 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcd4003520 con 0x7efce80cfe30 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.186+0000 7efce6ffd640 1 -- 172.21.15.44:0/690227396 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1781311419 0 0) 0x7efce806c4b0 con 0x7efce80d0a30 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.186+0000 7efce6ffd640 1 -- 172.21.15.44:0/690227396 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efce806c1d0 con 0x7efce80d0a30 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.186+0000 7efce77fe640 1 -- 172.21.15.44:0/690227396 >> v1:172.21.15.69:6789/0 conn(0x7efce80d0a30 legacy=0x7efce80d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.186+0000 7efce77fe640 1 -- 172.21.15.44:0/690227396 >> v1:172.21.15.44:6789/0 conn(0x7efce80cfe30 legacy=0x7efce80d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.186+0000 7efce6ffd640 1 -- 172.21.15.44:0/690227396 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1584422988 0 0) 0x7efcd4003520 con 0x7efce80cfe30 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.186+0000 7efce77fe640 1 -- 172.21.15.44:0/690227396 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce806c790 con 0x7efce806cf90 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.187+0000 7efce77fe640 1 --2- 172.21.15.44:0/690227396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce806cf90 0x7efce80cf8f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efcd00097d0 tx=0x7efcd002b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c9287cd7ddb31ed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.187+0000 7efce6ffd640 1 -- 172.21.15.44:0/690227396 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0037070 con 0x7efce806cf90 2024-04-02T13:32:23.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.187+0000 7efce6ffd640 1 -- 172.21.15.44:0/690227396 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcd0030510 con 0x7efce806cf90 2024-04-02T13:32:23.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.187+0000 7efce6ffd640 1 -- 172.21.15.44:0/690227396 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce00019f0 con 0x7efce806cf90 2024-04-02T13:32:23.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.187+0000 7efcee433640 1 -- 172.21.15.44:0/690227396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce806cf90 msgr2=0x7efce80cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.187+0000 7efcee433640 1 --2- 172.21.15.44:0/690227396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce806cf90 0x7efce80cf8f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efcd00097d0 tx=0x7efcd002b770 comp rx=0 tx=0).stop 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.187+0000 7efcee433640 1 -- 172.21.15.44:0/690227396 shutdown_connections 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.187+0000 7efcee433640 1 --2- 172.21.15.44:0/690227396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce806cf90 0x7efce80cf8f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.187+0000 7efcee433640 1 -- 172.21.15.44:0/690227396 >> 172.21.15.44:0/690227396 conn(0x7efce809d470 msgr2=0x7efce80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.187+0000 7efcee433640 1 -- 172.21.15.44:0/690227396 shutdown_connections 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.187+0000 7efcee433640 1 -- 172.21.15.44:0/690227396 wait complete. 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.188+0000 7efcee433640 1 Processor -- start 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.189+0000 7efcee433640 1 -- start start 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.189+0000 7efcee433640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce806cf90 0x7efce80b8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.189+0000 7efcee433640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce80cfe30 0x7efce81603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.189+0000 7efcee433640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efce80d0a30 0x7efce80ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.189+0000 7efcee433640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efce80d4da0 con 0x7efce80cfe30 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.189+0000 7efcee433640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efce806d3f0 con 0x7efce80d0a30 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.189+0000 7efcee433640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7efce80d4f20 con 0x7efce806cf90 2024-04-02T13:32:23.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.189+0000 7efce77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce80cfe30 0x7efce81603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.189+0000 7efce77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce80cfe30 0x7efce81603d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42516/0 (socket says 172.21.15.44:42516) 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efce77fe640 1 -- 172.21.15.44:0/3502030968 learned_addr learned my addr 172.21.15.44:0/3502030968 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efcec9d0640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efce80d0a30 0x7efce80ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efce7fff640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce806cf90 0x7efce80b8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efce77fe640 1 -- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce806cf90 msgr2=0x7efce80b8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efce77fe640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce806cf90 0x7efce80b8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efce77fe640 1 -- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efce80d0a30 msgr2=0x7efce80ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efce77fe640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efce80d0a30 0x7efce80ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efce77fe640 1 -- 172.21.15.44:0/3502030968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce80b8c30 con 0x7efce80cfe30 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efcec9d0640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efce80d0a30 0x7efce80ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efce77fe640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce80cfe30 0x7efce81603d0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7efcd0000c00 tx=0x7efcd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efce4ff9640 1 -- 172.21.15.44:0/3502030968 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0041020 con 0x7efce80cfe30 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efce4ff9640 1 -- 172.21.15.44:0/3502030968 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcd003c870 con 0x7efce80cfe30 2024-04-02T13:32:23.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efcee433640 1 -- 172.21.15.44:0/3502030968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce80b8f10 con 0x7efce80cfe30 2024-04-02T13:32:23.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efce4ff9640 1 -- 172.21.15.44:0/3502030968 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0037040 con 0x7efce80cfe30 2024-04-02T13:32:23.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.190+0000 7efcee433640 1 -- 172.21.15.44:0/3502030968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efce80b94d0 con 0x7efce80cfe30 2024-04-02T13:32:23.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.191+0000 7efcee433640 1 -- 172.21.15.44:0/3502030968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efce80684f0 con 0x7efce80cfe30 2024-04-02T13:32:23.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.192+0000 7efce4ff9640 1 -- 172.21.15.44:0/3502030968 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efcd0044020 con 0x7efce80cfe30 2024-04-02T13:32:23.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.192+0000 7efce4ff9640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efcc806c8f0 0x7efcc806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.192+0000 7efce4ff9640 1 -- 172.21.15.44:0/3502030968 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7efcd00ad4c0 con 0x7efce80cfe30 2024-04-02T13:32:23.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.192+0000 7efce7fff640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efcc806c8f0 0x7efcc806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.193+0000 7efce7fff640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efcc806c8f0 0x7efcc806edb0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7efcd80010e0 tx=0x7efcd8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.197+0000 7efce4ff9640 1 -- 172.21.15.44:0/3502030968 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7efcd004b030 con 0x7efce80cfe30 2024-04-02T13:32:23.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.381+0000 7efcee433640 1 -- 172.21.15.44:0/3502030968 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7efce80d17c0 con 0x7efcc806c8f0 2024-04-02T13:32:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.382+0000 7efce4ff9640 1 -- 172.21.15.44:0/3502030968 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7efce80d17c0 con 0x7efcc806c8f0 2024-04-02T13:32:23.385 INFO:teuthology.orchestra.run.smithi044.stderr:instructed osd(s) 3 to deep-scrub 2024-04-02T13:32:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.384+0000 7efcee433640 1 -- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efcc806c8f0 msgr2=0x7efcc806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.384+0000 7efcee433640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efcc806c8f0 0x7efcc806edb0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7efcd80010e0 tx=0x7efcd8009040 comp rx=0 tx=0).stop 2024-04-02T13:32:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.384+0000 7efcee433640 1 -- 172.21.15.44:0/3502030968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce80cfe30 msgr2=0x7efce81603d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.384+0000 7efcee433640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce80cfe30 0x7efce81603d0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7efcd0000c00 tx=0x7efcd00046c0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.384+0000 7efcee433640 1 -- 172.21.15.44:0/3502030968 shutdown_connections 2024-04-02T13:32:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.384+0000 7efcee433640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7efcc806c8f0 0x7efcc806edb0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.384+0000 7efcee433640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efce80d0a30 0x7efce80ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.384+0000 7efcee433640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce80cfe30 0x7efce81603d0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.384+0000 7efcee433640 1 --2- 172.21.15.44:0/3502030968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce806cf90 0x7efce80b8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.384+0000 7efcee433640 1 -- 172.21.15.44:0/3502030968 >> 172.21.15.44:0/3502030968 conn(0x7efce809d470 msgr2=0x7efce80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:23.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.384+0000 7efcee433640 1 -- 172.21.15.44:0/3502030968 shutdown_connections 2024-04-02T13:32:23.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.384+0000 7efcee433640 1 -- 172.21.15.44:0/3502030968 wait complete. 2024-04-02T13:32:23.397 INFO:tasks.ceph:Scrubbing osd.4 2024-04-02T13:32:23.397 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-02T13:32:23.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.500+0000 7fa2bd6dd640 1 Processor -- start 2024-04-02T13:32:23.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.500+0000 7fa2bd6dd640 1 -- start start 2024-04-02T13:32:23.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.501+0000 7fa2bd6dd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa2b80b1440 0x7fa2b80b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.501+0000 7fa2bd6dd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa2b80d03d0 0x7fa2b80d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.501+0000 7fa2bd6dd640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2b80cca50 con 0x7fa2b815c220 2024-04-02T13:32:23.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.501+0000 7fa2bd6dd640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa2b80ccd60 con 0x7fa2b80b1440 2024-04-02T13:32:23.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.501+0000 7fa2bd6dd640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa2b80cef40 con 0x7fa2b80d03d0 2024-04-02T13:32:23.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.501+0000 7fa2b67fc640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa2b80b1440 0x7fa2b80b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.501+0000 7fa2b6ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa2b815c220 0x7fa2b80ce780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52408/0 (socket says 172.21.15.44:52408) 2024-04-02T13:32:23.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.501+0000 7fa2b6ffd640 1 -- 172.21.15.44:0/1958766370 learned_addr learned my addr 172.21.15.44:0/1958766370 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:23.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.501+0000 7fa2b77fe640 1 --2- 172.21.15.44:0/1958766370 >> v2:172.21.15.69:3300/0 conn(0x7fa2b80d03d0 0x7fa2b80d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.501+0000 7fa2b67fc640 1 -- 172.21.15.44:0/1958766370 >> v2:172.21.15.69:3300/0 conn(0x7fa2b80d03d0 msgr2=0x7fa2b80d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:23.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.501+0000 7fa2b67fc640 1 --2- 172.21.15.44:0/1958766370 >> v2:172.21.15.69:3300/0 conn(0x7fa2b80d03d0 0x7fa2b80d0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.501+0000 7fa2b67fc640 1 -- 172.21.15.44:0/1958766370 >> v1:172.21.15.44:6789/0 conn(0x7fa2b815c220 legacy=0x7fa2b80ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.501+0000 7fa2b67fc640 1 -- 172.21.15.44:0/1958766370 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b80cf220 con 0x7fa2b80b1440 2024-04-02T13:32:23.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.502+0000 7fa2b67fc640 1 --2- 172.21.15.44:0/1958766370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2b80b1440 0x7fa2b80b1db0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fa2b809b5d0 tx=0x7fa2a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd1b753fb6f37a43 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.502+0000 7fa2b5ffb640 1 -- 172.21.15.44:0/1958766370 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0035020 con 0x7fa2b80b1440 2024-04-02T13:32:23.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.502+0000 7fa2b5ffb640 1 -- 172.21.15.44:0/1958766370 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2a0031830 con 0x7fa2b80b1440 2024-04-02T13:32:23.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.502+0000 7fa2b5ffb640 1 -- 172.21.15.44:0/1958766370 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0031c00 con 0x7fa2b80b1440 2024-04-02T13:32:23.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.503+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/1958766370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2b80b1440 msgr2=0x7fa2b80b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.503+0000 7fa2bd6dd640 1 --2- 172.21.15.44:0/1958766370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2b80b1440 0x7fa2b80b1db0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fa2b809b5d0 tx=0x7fa2a002b8d0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.503+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/1958766370 shutdown_connections 2024-04-02T13:32:23.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.503+0000 7fa2bd6dd640 1 --2- 172.21.15.44:0/1958766370 >> v2:172.21.15.69:3300/0 conn(0x7fa2b80d03d0 0x7fa2b80d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.503+0000 7fa2bd6dd640 1 --2- 172.21.15.44:0/1958766370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2b80b1440 0x7fa2b80b1db0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.503+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/1958766370 >> 172.21.15.44:0/1958766370 conn(0x7fa2b809aa20 msgr2=0x7fa2b809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:23.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.503+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/1958766370 shutdown_connections 2024-04-02T13:32:23.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.503+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/1958766370 wait complete. 2024-04-02T13:32:23.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.504+0000 7fa2bd6dd640 1 Processor -- start 2024-04-02T13:32:23.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.504+0000 7fa2bd6dd640 1 -- start start 2024-04-02T13:32:23.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2bd6dd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2b80b1440 0x7fa2b814c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2bd6dd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2b80d03d0 0x7fa2b8148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2bd6dd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b815c220 0x7fa2b81f77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2bd6dd640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b80cd1f0 con 0x7fa2b80b1440 2024-04-02T13:32:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2bd6dd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b815c600 con 0x7fa2b815c220 2024-04-02T13:32:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2bd6dd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b80cd370 con 0x7fa2b80d03d0 2024-04-02T13:32:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2b6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2b80b1440 0x7fa2b814c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2b6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2b80b1440 0x7fa2b814c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42530/0 (socket says 172.21.15.44:42530) 2024-04-02T13:32:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2b6ffd640 1 -- 172.21.15.44:0/3468676635 learned_addr learned my addr 172.21.15.44:0/3468676635 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2b77fe640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b815c220 0x7fa2b81f77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2b67fc640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2b80d03d0 0x7fa2b8148e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2b6ffd640 1 -- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2b80d03d0 msgr2=0x7fa2b8148e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2b6ffd640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2b80d03d0 0x7fa2b8148e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2b6ffd640 1 -- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b815c220 msgr2=0x7fa2b81f77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2b6ffd640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b815c220 0x7fa2b81f77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2b6ffd640 1 -- 172.21.15.44:0/3468676635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b8149380 con 0x7fa2b80b1440 2024-04-02T13:32:23.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2b67fc640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2b80d03d0 0x7fa2b8148e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:23.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.505+0000 7fa2b6ffd640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2b80b1440 0x7fa2b814c170 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fa2b000c3c0 tx=0x7fa2b000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.506+0000 7fa2af7fe640 1 -- 172.21.15.44:0/3468676635 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0017070 con 0x7fa2b80b1440 2024-04-02T13:32:23.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.506+0000 7fa2af7fe640 1 -- 172.21.15.44:0/3468676635 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2b0005ce0 con 0x7fa2b80b1440 2024-04-02T13:32:23.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.506+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/3468676635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b80c6a00 con 0x7fa2b80b1440 2024-04-02T13:32:23.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.506+0000 7fa2af7fe640 1 -- 172.21.15.44:0/3468676635 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b00072f0 con 0x7fa2b80b1440 2024-04-02T13:32:23.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.506+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/3468676635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b81f8320 con 0x7fa2b80b1440 2024-04-02T13:32:23.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.506+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/3468676635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa2b80684f0 con 0x7fa2b80b1440 2024-04-02T13:32:23.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.507+0000 7fa2af7fe640 1 -- 172.21.15.44:0/3468676635 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa2b001a070 con 0x7fa2b80b1440 2024-04-02T13:32:23.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.507+0000 7fa2af7fe640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa29806c8f0 0x7fa29806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.508+0000 7fa2af7fe640 1 -- 172.21.15.44:0/3468676635 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7fa2b008ad30 con 0x7fa2b80b1440 2024-04-02T13:32:23.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.508+0000 7fa2b67fc640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa29806c8f0 0x7fa29806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.508+0000 7fa2af7fe640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x7fa298072ed0 0x7fa298075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.508+0000 7fa2af7fe640 1 -- 172.21.15.44:0/3468676635 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2980759e0 con 0x7fa298072ed0 2024-04-02T13:32:23.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.508+0000 7fa2af7fe640 1 -- 172.21.15.44:0/3468676635 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=126) ==== 24+0+0 (secure 0 0 0) 0x7fa2b008d3d0 con 0x7fa2b80b1440 2024-04-02T13:32:23.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.508+0000 7fa2b77fe640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x7fa298072ed0 0x7fa298075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.508+0000 7fa2b67fc640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa29806c8f0 0x7fa29806edb0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0036000 tx=0x7fa2a0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.508+0000 7fa2b77fe640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x7fa298072ed0 0x7fa298075310 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.509+0000 7fa2af7fe640 1 -- 172.21.15.44:0/3468676635 <== osd.4 v2:172.21.15.69:6816/1041350007 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa2980759e0 con 0x7fa298072ed0 2024-04-02T13:32:23.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.522+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/3468676635 --> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa2b80d2d10 con 0x7fa298072ed0 2024-04-02T13:32:23.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.523+0000 7fa2af7fe640 1 -- 172.21.15.44:0/3468676635 <== osd.4 v2:172.21.15.69:6816/1041350007 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa2b80d2d10 con 0x7fa298072ed0 2024-04-02T13:32:23.527 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:32:23.527 INFO:teuthology.orchestra.run.smithi044.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T13:32:23.527 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:32:23.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.523+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x7fa298072ed0 msgr2=0x7fa298075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.523+0000 7fa2bd6dd640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:6816/1041350007,v1:172.21.15.69:6817/1041350007] conn(0x7fa298072ed0 0x7fa298075310 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.524+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa29806c8f0 msgr2=0x7fa29806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.524+0000 7fa2bd6dd640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa29806c8f0 0x7fa29806edb0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0036000 tx=0x7fa2a0008030 comp rx=0 tx=0).stop 2024-04-02T13:32:23.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.524+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/3468676635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2b80b1440 msgr2=0x7fa2b814c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.524+0000 7fa2bd6dd640 1 --2- 172.21.15.44:0/3468676635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2b80b1440 0x7fa2b814c170 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fa2b000c3c0 tx=0x7fa2b000c880 comp rx=0 tx=0).stop 2024-04-02T13:32:23.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.524+0000 7fa2b77fe640 1 -- 172.21.15.44:0/3468676635 reap_dead start 2024-04-02T13:32:23.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.524+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/3468676635 shutdown_connections 2024-04-02T13:32:23.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.524+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/3468676635 >> 172.21.15.44:0/3468676635 conn(0x7fa2b809aa20 msgr2=0x7fa2b80b0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:23.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.524+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/3468676635 shutdown_connections 2024-04-02T13:32:23.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.524+0000 7fa2bd6dd640 1 -- 172.21.15.44:0/3468676635 wait complete. 2024-04-02T13:32:23.537 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-02T13:32:23.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.637+0000 7fe7fc22b640 1 Processor -- start 2024-04-02T13:32:23.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.637+0000 7fe7fc22b640 1 -- start start 2024-04-02T13:32:23.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7fc22b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f40d0a30 0x7fe7f406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7fc22b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe7f40d0e10 0x7fe7f40cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7fc22b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe7f40cfe30 0x7fe7f40d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7fc22b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fe7f406cb00 con 0x7fe7f40d0e10 2024-04-02T13:32:23.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7fc22b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe7f406c1d0 con 0x7fe7f40cfe30 2024-04-02T13:32:23.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7fc22b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe7f406c4b0 con 0x7fe7f40d0a30 2024-04-02T13:32:23.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7f97c6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe7f40d0e10 0x7fe7f40cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7f97c6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fe7f40d0e10 0x7fe7f40cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42534/0 (socket says 172.21.15.44:42534) 2024-04-02T13:32:23.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7f97c6640 1 -- 172.21.15.44:0/3100725782 learned_addr learned my addr 172.21.15.44:0/3100725782 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:23.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7fa7c8640 1 --2- 172.21.15.44:0/3100725782 >> v2:172.21.15.69:3300/0 conn(0x7fe7f40cfe30 0x7fe7f40d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7f9fc7640 1 --2- 172.21.15.44:0/3100725782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f40d0a30 0x7fe7f406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7f97c6640 1 -- 172.21.15.44:0/3100725782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f40d0a30 msgr2=0x7fe7f406cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:23.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7f97c6640 1 --2- 172.21.15.44:0/3100725782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f40d0a30 0x7fe7f406cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7f97c6640 1 -- 172.21.15.44:0/3100725782 >> v2:172.21.15.69:3300/0 conn(0x7fe7f40cfe30 msgr2=0x7fe7f40d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:23.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7f97c6640 1 --2- 172.21.15.44:0/3100725782 >> v2:172.21.15.69:3300/0 conn(0x7fe7f40cfe30 0x7fe7f40d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.638+0000 7fe7f97c6640 1 -- 172.21.15.44:0/3100725782 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f406c790 con 0x7fe7f40d0e10 2024-04-02T13:32:23.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.639+0000 7fe7f97c6640 1 --2- 172.21.15.44:0/3100725782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f40d0e10 0x7fe7f40cf8f0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fe7f408e4f0 tx=0x7fe7e802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be4083f402445b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.639+0000 7fe7f8fc5640 1 -- 172.21.15.44:0/3100725782 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e802e070 con 0x7fe7f40d0e10 2024-04-02T13:32:23.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.639+0000 7fe7f8fc5640 1 -- 172.21.15.44:0/3100725782 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7e8037070 con 0x7fe7f40d0e10 2024-04-02T13:32:23.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.639+0000 7fe7f8fc5640 1 -- 172.21.15.44:0/3100725782 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8031a00 con 0x7fe7f40d0e10 2024-04-02T13:32:23.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.640+0000 7fe7fc22b640 1 -- 172.21.15.44:0/3100725782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f40d0e10 msgr2=0x7fe7f40cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.640+0000 7fe7fc22b640 1 --2- 172.21.15.44:0/3100725782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f40d0e10 0x7fe7f40cf8f0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fe7f408e4f0 tx=0x7fe7e802b880 comp rx=0 tx=0).stop 2024-04-02T13:32:23.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.640+0000 7fe7fc22b640 1 -- 172.21.15.44:0/3100725782 shutdown_connections 2024-04-02T13:32:23.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.640+0000 7fe7fc22b640 1 --2- 172.21.15.44:0/3100725782 >> v2:172.21.15.69:3300/0 conn(0x7fe7f40cfe30 0x7fe7f40d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.640+0000 7fe7fc22b640 1 --2- 172.21.15.44:0/3100725782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f40d0e10 0x7fe7f40cf8f0 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.640+0000 7fe7fc22b640 1 --2- 172.21.15.44:0/3100725782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f40d0a30 0x7fe7f406cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.640+0000 7fe7fc22b640 1 -- 172.21.15.44:0/3100725782 >> 172.21.15.44:0/3100725782 conn(0x7fe7f409d470 msgr2=0x7fe7f40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:23.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.640+0000 7fe7fc22b640 1 -- 172.21.15.44:0/3100725782 shutdown_connections 2024-04-02T13:32:23.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.640+0000 7fe7fc22b640 1 -- 172.21.15.44:0/3100725782 wait complete. 2024-04-02T13:32:23.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.641+0000 7fe7fc22b640 1 Processor -- start 2024-04-02T13:32:23.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.641+0000 7fe7fc22b640 1 -- start start 2024-04-02T13:32:23.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7fc22b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f40cfe30 0x7fe7f40b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7fc22b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f40d0a30 0x7fe7f40b9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7fc22b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7f40d0e10 0x7fe7f4156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7fc22b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f415e870 con 0x7fe7f40cfe30 2024-04-02T13:32:23.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7fc22b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f415e6f0 con 0x7fe7f40d0e10 2024-04-02T13:32:23.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7fc22b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe7f40d4da0 con 0x7fe7f40d0a30 2024-04-02T13:32:23.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7f9fc7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f40cfe30 0x7fe7f40b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7f9fc7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f40cfe30 0x7fe7f40b3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42548/0 (socket says 172.21.15.44:42548) 2024-04-02T13:32:23.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7f9fc7640 1 -- 172.21.15.44:0/2078939950 learned_addr learned my addr 172.21.15.44:0/2078939950 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:23.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7fa7c8640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7f40d0e10 0x7fe7f4156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7f97c6640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f40d0a30 0x7fe7f40b9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7f9fc7640 1 -- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f40d0a30 msgr2=0x7fe7f40b9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7f9fc7640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f40d0a30 0x7fe7f40b9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7f9fc7640 1 -- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7f40d0e10 msgr2=0x7fe7f4156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7f9fc7640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7f40d0e10 0x7fe7f4156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7f9fc7640 1 -- 172.21.15.44:0/2078939950 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f4160a90 con 0x7fe7f40cfe30 2024-04-02T13:32:23.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7f97c6640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f40d0a30 0x7fe7f40b9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:23.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.642+0000 7fe7fa7c8640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7f40d0e10 0x7fe7f4156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:23.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.643+0000 7fe7f9fc7640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f40cfe30 0x7fe7f40b3e10 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00efc0 tx=0x7fe7ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.643+0000 7fe7f27fc640 1 -- 172.21.15.44:0/2078939950 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec018020 con 0x7fe7f40cfe30 2024-04-02T13:32:23.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.643+0000 7fe7f27fc640 1 -- 172.21.15.44:0/2078939950 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7ec014480 con 0x7fe7f40cfe30 2024-04-02T13:32:23.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.643+0000 7fe7f27fc640 1 -- 172.21.15.44:0/2078939950 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec014850 con 0x7fe7f40cfe30 2024-04-02T13:32:23.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.643+0000 7fe7fc22b640 1 -- 172.21.15.44:0/2078939950 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f40cb340 con 0x7fe7f40cfe30 2024-04-02T13:32:23.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.643+0000 7fe7fc22b640 1 -- 172.21.15.44:0/2078939950 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f40cb8c0 con 0x7fe7f40cfe30 2024-04-02T13:32:23.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.644+0000 7fe7fc22b640 1 -- 172.21.15.44:0/2078939950 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7f40684f0 con 0x7fe7f40cfe30 2024-04-02T13:32:23.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.644+0000 7fe7f27fc640 1 -- 172.21.15.44:0/2078939950 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe7ec005d10 con 0x7fe7f40cfe30 2024-04-02T13:32:23.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.644+0000 7fe7f27fc640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe7e006c8f0 0x7fe7e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.645+0000 7fe7f27fc640 1 -- 172.21.15.44:0/2078939950 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7fe7ec08c7f0 con 0x7fe7f40cfe30 2024-04-02T13:32:23.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.645+0000 7fe7f97c6640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe7e006c8f0 0x7fe7e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.645+0000 7fe7f97c6640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe7e006c8f0 0x7fe7e006edb0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8036000 tx=0x7fe7e8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.649+0000 7fe7f27fc640 1 -- 172.21.15.44:0/2078939950 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fe7ec022030 con 0x7fe7f40cfe30 2024-04-02T13:32:23.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.832+0000 7fe7fc22b640 1 -- 172.21.15.44:0/2078939950 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fe7f40cf180 con 0x7fe7e006c8f0 2024-04-02T13:32:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.833+0000 7fe7f27fc640 1 -- 172.21.15.44:0/2078939950 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe7f40cf180 con 0x7fe7e006c8f0 2024-04-02T13:32:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:instructed osd(s) 4 to deep-scrub 2024-04-02T13:32:23.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.835+0000 7fe7fc22b640 1 -- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe7e006c8f0 msgr2=0x7fe7e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.835+0000 7fe7fc22b640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe7e006c8f0 0x7fe7e006edb0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8036000 tx=0x7fe7e8008030 comp rx=0 tx=0).stop 2024-04-02T13:32:23.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.835+0000 7fe7fc22b640 1 -- 172.21.15.44:0/2078939950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f40cfe30 msgr2=0x7fe7f40b3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.835+0000 7fe7fc22b640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f40cfe30 0x7fe7f40b3e10 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00efc0 tx=0x7fe7ec00c490 comp rx=0 tx=0).stop 2024-04-02T13:32:23.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.835+0000 7fe7fc22b640 1 -- 172.21.15.44:0/2078939950 shutdown_connections 2024-04-02T13:32:23.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.835+0000 7fe7fc22b640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fe7e006c8f0 0x7fe7e006edb0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.835+0000 7fe7fc22b640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7f40d0e10 0x7fe7f4156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.835+0000 7fe7fc22b640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f40d0a30 0x7fe7f40b9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.835+0000 7fe7fc22b640 1 --2- 172.21.15.44:0/2078939950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f40cfe30 0x7fe7f40b3e10 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.835+0000 7fe7fc22b640 1 -- 172.21.15.44:0/2078939950 >> 172.21.15.44:0/2078939950 conn(0x7fe7f409d470 msgr2=0x7fe7f40b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:23.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.835+0000 7fe7fc22b640 1 -- 172.21.15.44:0/2078939950 shutdown_connections 2024-04-02T13:32:23.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.835+0000 7fe7fc22b640 1 -- 172.21.15.44:0/2078939950 wait complete. 2024-04-02T13:32:23.848 INFO:tasks.ceph:Scrubbing osd.5 2024-04-02T13:32:23.848 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-02T13:32:23.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.951+0000 7fca894a5640 1 Processor -- start 2024-04-02T13:32:23.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.951+0000 7fca894a5640 1 -- start start 2024-04-02T13:32:23.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.952+0000 7fca894a5640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fca8415c1a0 0x7fca8406f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.952+0000 7fca894a5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca8415c580 con 0x7fca840cfd60 2024-04-02T13:32:23.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.952+0000 7fca894a5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca840d01c0 con 0x7fca840d21c0 2024-04-02T13:32:23.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.952+0000 7fca894a5640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fca84070ad0 con 0x7fca8415c1a0 2024-04-02T13:32:23.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.952+0000 7fca82ffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fca8415c1a0 0x7fca8406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.952+0000 7fca827fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fca840cfd60 0x7fca840d1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52420/0 (socket says 172.21.15.44:52420) 2024-04-02T13:32:23.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.952+0000 7fca827fc640 1 -- 172.21.15.44:0/2070675180 learned_addr learned my addr 172.21.15.44:0/2070675180 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:23.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.953+0000 7fca82ffd640 1 -- 172.21.15.44:0/2070675180 >> v1:172.21.15.69:6789/0 conn(0x7fca840d21c0 legacy=0x7fca840cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.953+0000 7fca82ffd640 1 -- 172.21.15.44:0/2070675180 >> v1:172.21.15.44:6789/0 conn(0x7fca840cfd60 legacy=0x7fca840d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.953+0000 7fca81ffb640 1 -- 172.21.15.44:0/2070675180 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1696666299 0 0) 0x7fca8415c580 con 0x7fca840cfd60 2024-04-02T13:32:23.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.953+0000 7fca82ffd640 1 -- 172.21.15.44:0/2070675180 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca8406fa30 con 0x7fca8415c1a0 2024-04-02T13:32:23.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.953+0000 7fca82ffd640 1 --2- 172.21.15.44:0/2070675180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca8415c1a0 0x7fca8406f370 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fca7c0097a0 tx=0x7fca7c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7885fc4e66d8de10 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.953+0000 7fca81ffb640 1 -- 172.21.15.44:0/2070675180 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c034070 con 0x7fca8415c1a0 2024-04-02T13:32:23.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.953+0000 7fca81ffb640 1 -- 172.21.15.44:0/2070675180 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca7c030610 con 0x7fca8415c1a0 2024-04-02T13:32:23.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.953+0000 7fca81ffb640 1 -- 172.21.15.44:0/2070675180 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c0309e0 con 0x7fca8415c1a0 2024-04-02T13:32:23.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.954+0000 7fca894a5640 1 -- 172.21.15.44:0/2070675180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca8415c1a0 msgr2=0x7fca8406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.954+0000 7fca894a5640 1 --2- 172.21.15.44:0/2070675180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca8415c1a0 0x7fca8406f370 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fca7c0097a0 tx=0x7fca7c02b960 comp rx=0 tx=0).stop 2024-04-02T13:32:23.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.954+0000 7fca894a5640 1 -- 172.21.15.44:0/2070675180 shutdown_connections 2024-04-02T13:32:23.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.954+0000 7fca894a5640 1 --2- 172.21.15.44:0/2070675180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca8415c1a0 0x7fca8406f370 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.954+0000 7fca894a5640 1 -- 172.21.15.44:0/2070675180 >> 172.21.15.44:0/2070675180 conn(0x7fca8409aa20 msgr2=0x7fca8409ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:23.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.954+0000 7fca894a5640 1 -- 172.21.15.44:0/2070675180 shutdown_connections 2024-04-02T13:32:23.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.954+0000 7fca894a5640 1 -- 172.21.15.44:0/2070675180 wait complete. 2024-04-02T13:32:23.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.955+0000 7fca894a5640 1 Processor -- start 2024-04-02T13:32:23.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.955+0000 7fca894a5640 1 -- start start 2024-04-02T13:32:23.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca894a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca840cfd60 0x7fca84150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca894a5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca840d21c0 0x7fca8415da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca894a5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca8415c1a0 0x7fca840cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca894a5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fca840702d0 con 0x7fca840d21c0 2024-04-02T13:32:23.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca894a5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fca8406fff0 con 0x7fca840cfd60 2024-04-02T13:32:23.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca894a5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fca840705b0 con 0x7fca8415c1a0 2024-04-02T13:32:23.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca827fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca840d21c0 0x7fca8415da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca827fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca840d21c0 0x7fca8415da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42560/0 (socket says 172.21.15.44:42560) 2024-04-02T13:32:23.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca827fc640 1 -- 172.21.15.44:0/72902300 learned_addr learned my addr 172.21.15.44:0/72902300 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:23.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca82ffd640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca840cfd60 0x7fca84150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca837fe640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca8415c1a0 0x7fca840cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca827fc640 1 -- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca8415c1a0 msgr2=0x7fca840cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca827fc640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca8415c1a0 0x7fca840cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca827fc640 1 -- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca840cfd60 msgr2=0x7fca84150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca827fc640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca840cfd60 0x7fca84150310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.956+0000 7fca827fc640 1 -- 172.21.15.44:0/72902300 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca841f3150 con 0x7fca840d21c0 2024-04-02T13:32:23.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.957+0000 7fca827fc640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca840d21c0 0x7fca8415da20 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00c420 tx=0x7fca6c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.957+0000 7fca7b7fe640 1 -- 172.21.15.44:0/72902300 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0074e0 con 0x7fca840d21c0 2024-04-02T13:32:23.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.957+0000 7fca7b7fe640 1 -- 172.21.15.44:0/72902300 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca6c0047e0 con 0x7fca840d21c0 2024-04-02T13:32:23.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.957+0000 7fca894a5640 1 -- 172.21.15.44:0/72902300 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca841f25d0 con 0x7fca840d21c0 2024-04-02T13:32:23.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.957+0000 7fca7b7fe640 1 -- 172.21.15.44:0/72902300 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c004c10 con 0x7fca840d21c0 2024-04-02T13:32:23.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.957+0000 7fca894a5640 1 -- 172.21.15.44:0/72902300 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca841f2b90 con 0x7fca840d21c0 2024-04-02T13:32:23.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.958+0000 7fca894a5640 1 -- 172.21.15.44:0/72902300 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fca84068c00 con 0x7fca840d21c0 2024-04-02T13:32:23.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.958+0000 7fca7b7fe640 1 -- 172.21.15.44:0/72902300 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fca6c011070 con 0x7fca840d21c0 2024-04-02T13:32:23.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.959+0000 7fca7b7fe640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca6406c8f0 0x7fca6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.959+0000 7fca7b7fe640 1 -- 172.21.15.44:0/72902300 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7fca6c08b490 con 0x7fca840d21c0 2024-04-02T13:32:23.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.959+0000 7fca82ffd640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca6406c8f0 0x7fca6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.959+0000 7fca7b7fe640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x7fca64072ed0 0x7fca64075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:23.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.959+0000 7fca7b7fe640 1 -- 172.21.15.44:0/72902300 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca6c01ee50 con 0x7fca64072ed0 2024-04-02T13:32:23.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.959+0000 7fca7b7fe640 1 -- 172.21.15.44:0/72902300 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=126) ==== 24+0+0 (secure 0 0 0) 0x7fca6c092050 con 0x7fca840d21c0 2024-04-02T13:32:23.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.959+0000 7fca837fe640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x7fca64072ed0 0x7fca64075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:23.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.960+0000 7fca82ffd640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca6406c8f0 0x7fca6406edb0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fca7c036000 tx=0x7fca7c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.960+0000 7fca837fe640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x7fca64072ed0 0x7fca64075310 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:23.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.961+0000 7fca7b7fe640 1 -- 172.21.15.44:0/72902300 <== osd.5 v2:172.21.15.69:6800/2841767499 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fca6c01ee50 con 0x7fca64072ed0 2024-04-02T13:32:23.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.974+0000 7fca894a5640 1 -- 172.21.15.44:0/72902300 --> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fca8415df60 con 0x7fca64072ed0 2024-04-02T13:32:23.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.974+0000 7fca7b7fe640 1 -- 172.21.15.44:0/72902300 <== osd.5 v2:172.21.15.69:6800/2841767499 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fca8415df60 con 0x7fca64072ed0 2024-04-02T13:32:23.977 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:32:23.977 INFO:teuthology.orchestra.run.smithi044.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T13:32:23.977 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:32:23.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.974+0000 7fca894a5640 1 -- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x7fca64072ed0 msgr2=0x7fca64075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.974+0000 7fca894a5640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:6800/2841767499,v1:172.21.15.69:6801/2841767499] conn(0x7fca64072ed0 0x7fca64075310 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.974+0000 7fca894a5640 1 -- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca6406c8f0 msgr2=0x7fca6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.974+0000 7fca894a5640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fca6406c8f0 0x7fca6406edb0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fca7c036000 tx=0x7fca7c008030 comp rx=0 tx=0).stop 2024-04-02T13:32:23.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.974+0000 7fca894a5640 1 -- 172.21.15.44:0/72902300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca840d21c0 msgr2=0x7fca8415da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:23.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.974+0000 7fca894a5640 1 --2- 172.21.15.44:0/72902300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca840d21c0 0x7fca8415da20 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00c420 tx=0x7fca6c00c8e0 comp rx=0 tx=0).stop 2024-04-02T13:32:23.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.974+0000 7fca837fe640 1 -- 172.21.15.44:0/72902300 reap_dead start 2024-04-02T13:32:23.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.974+0000 7fca894a5640 1 -- 172.21.15.44:0/72902300 shutdown_connections 2024-04-02T13:32:23.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.974+0000 7fca894a5640 1 -- 172.21.15.44:0/72902300 >> 172.21.15.44:0/72902300 conn(0x7fca8409aa20 msgr2=0x7fca8409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:23.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.974+0000 7fca894a5640 1 -- 172.21.15.44:0/72902300 shutdown_connections 2024-04-02T13:32:23.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:23.975+0000 7fca894a5640 1 -- 172.21.15.44:0/72902300 wait complete. 2024-04-02T13:32:23.987 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-02T13:32:24.080 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:32:24.082 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:32:24.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.087+0000 7fa3c3dc8640 1 Processor -- start 2024-04-02T13:32:24.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.087+0000 7fa3c3dc8640 1 -- start start 2024-04-02T13:32:24.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.088+0000 7fa3c3dc8640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa3bc15c250 0x7fa3bc0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.088+0000 7fa3c3dc8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3bc0af420 con 0x7fa3bc0b3510 2024-04-02T13:32:24.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.088+0000 7fa3c3dc8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3bc0af700 con 0x7fa3bc0d0aa0 2024-04-02T13:32:24.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.088+0000 7fa3c3dc8640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fa3bc15c630 con 0x7fa3bc15c250 2024-04-02T13:32:24.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.088+0000 7fa3c1b64640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fa3bc15c250 0x7fa3bc0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.088+0000 7fa3c1363640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa3bc0b3510 0x7fa3bc0d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52432/0 (socket says 172.21.15.44:52432) 2024-04-02T13:32:24.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.088+0000 7fa3c1363640 1 -- 172.21.15.44:0/4155921782 learned_addr learned my addr 172.21.15.44:0/4155921782 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:24.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.089+0000 7fa3c1b64640 1 -- 172.21.15.44:0/4155921782 >> v1:172.21.15.69:6789/0 conn(0x7fa3bc0d0aa0 legacy=0x7fa3bc0b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.089+0000 7fa3c1b64640 1 -- 172.21.15.44:0/4155921782 >> v1:172.21.15.44:6789/0 conn(0x7fa3bc0b3510 legacy=0x7fa3bc0d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.089+0000 7fa3c0b62640 1 -- 172.21.15.44:0/4155921782 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 33573030 0 0) 0x7fa3bc0af420 con 0x7fa3bc0b3510 2024-04-02T13:32:24.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.089+0000 7fa3c1b64640 1 -- 172.21.15.44:0/4155921782 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3bc0af9e0 con 0x7fa3bc15c250 2024-04-02T13:32:24.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.089+0000 7fa3c1b64640 1 --2- 172.21.15.44:0/4155921782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3bc15c250 0x7fa3bc0aebf0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0097a0 tx=0x7fa3ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb617118870d07fa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.090+0000 7fa3c0b62640 1 -- 172.21.15.44:0/4155921782 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac035020 con 0x7fa3bc15c250 2024-04-02T13:32:24.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.090+0000 7fa3c0b62640 1 -- 172.21.15.44:0/4155921782 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3ac031830 con 0x7fa3bc15c250 2024-04-02T13:32:24.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.090+0000 7fa3c0b62640 1 -- 172.21.15.44:0/4155921782 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac031c00 con 0x7fa3bc15c250 2024-04-02T13:32:24.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.090+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/4155921782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3bc15c250 msgr2=0x7fa3bc0aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.090+0000 7fa3c3dc8640 1 --2- 172.21.15.44:0/4155921782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3bc15c250 0x7fa3bc0aebf0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0097a0 tx=0x7fa3ac02b8d0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.090+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/4155921782 shutdown_connections 2024-04-02T13:32:24.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.090+0000 7fa3c3dc8640 1 --2- 172.21.15.44:0/4155921782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3bc15c250 0x7fa3bc0aebf0 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.090+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/4155921782 >> 172.21.15.44:0/4155921782 conn(0x7fa3bc073850 msgr2=0x7fa3bc09aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:24.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.090+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/4155921782 shutdown_connections 2024-04-02T13:32:24.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.090+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/4155921782 wait complete. 2024-04-02T13:32:24.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.092+0000 7fa3c3dc8640 1 Processor -- start 2024-04-02T13:32:24.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.092+0000 7fa3c3dc8640 1 -- start start 2024-04-02T13:32:24.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.092+0000 7fa3c3dc8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3bc0b3510 0x7fa3bc1f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.092+0000 7fa3c3dc8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3bc0d0aa0 0x7fa3bc0b76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.092+0000 7fa3c3dc8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3bc15c250 0x7fa3bc0cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.092+0000 7fa3c3dc8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa3bc0b0bd0 con 0x7fa3bc0d0aa0 2024-04-02T13:32:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.092+0000 7fa3c3dc8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa3bc0af130 con 0x7fa3bc15c250 2024-04-02T13:32:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.092+0000 7fa3c3dc8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa3bc0b0d50 con 0x7fa3bc0b3510 2024-04-02T13:32:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.092+0000 7fa3c1363640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3bc0d0aa0 0x7fa3bc0b76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3c1363640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3bc0d0aa0 0x7fa3bc0b76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42574/0 (socket says 172.21.15.44:42574) 2024-04-02T13:32:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3c1363640 1 -- 172.21.15.44:0/3852173562 learned_addr learned my addr 172.21.15.44:0/3852173562 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3c2365640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3bc15c250 0x7fa3bc0cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3c1b64640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3bc0b3510 0x7fa3bc1f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3c1363640 1 -- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3bc0b3510 msgr2=0x7fa3bc1f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3c1363640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3bc0b3510 0x7fa3bc1f24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3c1363640 1 -- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3bc15c250 msgr2=0x7fa3bc0cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3c1363640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3bc15c250 0x7fa3bc0cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3c1363640 1 -- 172.21.15.44:0/3852173562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3bc0c5e40 con 0x7fa3bc0d0aa0 2024-04-02T13:32:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3c1363640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3bc0d0aa0 0x7fa3bc0b76c0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fa3b000c4a0 tx=0x7fa3b000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3ba7fc640 1 -- 172.21.15.44:0/3852173562 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0018070 con 0x7fa3bc0d0aa0 2024-04-02T13:32:24.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3ba7fc640 1 -- 172.21.15.44:0/3852173562 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3b00074e0 con 0x7fa3bc0d0aa0 2024-04-02T13:32:24.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/3852173562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3bc0c6120 con 0x7fa3bc0d0aa0 2024-04-02T13:32:24.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3ba7fc640 1 -- 172.21.15.44:0/3852173562 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0004a20 con 0x7fa3bc0d0aa0 2024-04-02T13:32:24.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.093+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/3852173562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3bc0c66e0 con 0x7fa3bc0d0aa0 2024-04-02T13:32:24.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.094+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/3852173562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3bc068c00 con 0x7fa3bc0d0aa0 2024-04-02T13:32:24.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.095+0000 7fa3ba7fc640 1 -- 172.21.15.44:0/3852173562 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa3b001b070 con 0x7fa3bc0d0aa0 2024-04-02T13:32:24.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.095+0000 7fa3ba7fc640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa3a806c8f0 0x7fa3a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.095+0000 7fa3ba7fc640 1 -- 172.21.15.44:0/3852173562 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7fa3b008acf0 con 0x7fa3bc0d0aa0 2024-04-02T13:32:24.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.095+0000 7fa3c1b64640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa3a806c8f0 0x7fa3a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.096+0000 7fa3c1b64640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa3a806c8f0 0x7fa3a806edb0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac036000 tx=0x7fa3ac008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.100+0000 7fa3ba7fc640 1 -- 172.21.15.44:0/3852173562 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fa3b008f070 con 0x7fa3bc0d0aa0 2024-04-02T13:32:24.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.286+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/3852173562 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fa3bc15b830 con 0x7fa3a806c8f0 2024-04-02T13:32:24.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.286+0000 7fa3ba7fc640 1 -- 172.21.15.44:0/3852173562 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa3bc15b830 con 0x7fa3a806c8f0 2024-04-02T13:32:24.289 INFO:teuthology.orchestra.run.smithi044.stderr:instructed osd(s) 5 to deep-scrub 2024-04-02T13:32:24.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.288+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa3a806c8f0 msgr2=0x7fa3a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.288+0000 7fa3c3dc8640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa3a806c8f0 0x7fa3a806edb0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac036000 tx=0x7fa3ac008030 comp rx=0 tx=0).stop 2024-04-02T13:32:24.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.288+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/3852173562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3bc0d0aa0 msgr2=0x7fa3bc0b76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.288+0000 7fa3c3dc8640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3bc0d0aa0 0x7fa3bc0b76c0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fa3b000c4a0 tx=0x7fa3b000c960 comp rx=0 tx=0).stop 2024-04-02T13:32:24.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.288+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/3852173562 shutdown_connections 2024-04-02T13:32:24.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.288+0000 7fa3c3dc8640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fa3a806c8f0 0x7fa3a806edb0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.288+0000 7fa3c3dc8640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3bc15c250 0x7fa3bc0cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.288+0000 7fa3c3dc8640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3bc0d0aa0 0x7fa3bc0b76c0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.288+0000 7fa3c3dc8640 1 --2- 172.21.15.44:0/3852173562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3bc0b3510 0x7fa3bc1f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.288+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/3852173562 >> 172.21.15.44:0/3852173562 conn(0x7fa3bc073850 msgr2=0x7fa3bc070160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:24.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.288+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/3852173562 shutdown_connections 2024-04-02T13:32:24.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.288+0000 7fa3c3dc8640 1 -- 172.21.15.44:0/3852173562 wait complete. 2024-04-02T13:32:24.302 INFO:tasks.ceph:Scrubbing osd.6 2024-04-02T13:32:24.302 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-02T13:32:24.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.404+0000 7f063cf50640 1 Processor -- start 2024-04-02T13:32:24.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.404+0000 7f063cf50640 1 -- start start 2024-04-02T13:32:24.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.405+0000 7f063cf50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f063815e1d0 0x7f06380d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.405+0000 7f063cf50640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f06380cec60 0x7f06380d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.405+0000 7f063cf50640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06380d1640 con 0x7f063815e5b0 2024-04-02T13:32:24.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.405+0000 7f063cf50640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f06380cfe00 con 0x7f06380cec60 2024-04-02T13:32:24.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.405+0000 7f063cf50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f06380cf1f0 con 0x7f063815e1d0 2024-04-02T13:32:24.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.406+0000 7f063659c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f063815e1d0 0x7f06380d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.406+0000 7f0635d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f06380cec60 0x7f06380d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.406+0000 7f063659c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f063815e1d0 0x7f06380d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.44:37640/0 (socket says 172.21.15.44:37640) 2024-04-02T13:32:24.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.406+0000 7f063659c640 1 -- 172.21.15.44:0/2400177381 learned_addr learned my addr 172.21.15.44:0/2400177381 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:24.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.406+0000 7f0636d9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f063815e5b0 0x7f06380b4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.44:40316/0 (socket says 172.21.15.44:40316) 2024-04-02T13:32:24.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.406+0000 7f0635d9b640 1 --2- 172.21.15.44:0/2400177381 >> v2:172.21.15.69:3300/0 conn(0x7f06380cec60 0x7f06380d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:33166/0 (socket says 172.21.15.44:33166) 2024-04-02T13:32:24.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.406+0000 7f063659c640 1 -- 172.21.15.44:0/2400177381 >> v2:172.21.15.69:3300/0 conn(0x7f06380cec60 msgr2=0x7f06380d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:24.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.406+0000 7f063659c640 1 --2- 172.21.15.44:0/2400177381 >> v2:172.21.15.69:3300/0 conn(0x7f06380cec60 0x7f06380d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.406+0000 7f063659c640 1 -- 172.21.15.44:0/2400177381 >> v1:172.21.15.69:6789/0 conn(0x7f063815e5b0 legacy=0x7f06380b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.406+0000 7f063659c640 1 -- 172.21.15.44:0/2400177381 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06380cf4d0 con 0x7f063815e1d0 2024-04-02T13:32:24.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.407+0000 7f063659c640 1 --2- 172.21.15.44:0/2400177381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f063815e1d0 0x7f06380d1100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0628009e50 tx=0x7f062802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c5add69d191ef70 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.407+0000 7f063559a640 1 -- 172.21.15.44:0/2400177381 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062802ca70 con 0x7f063815e1d0 2024-04-02T13:32:24.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.407+0000 7f063559a640 1 -- 172.21.15.44:0/2400177381 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f062802cc10 con 0x7f063815e1d0 2024-04-02T13:32:24.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.407+0000 7f063559a640 1 -- 172.21.15.44:0/2400177381 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06280343f0 con 0x7f063815e1d0 2024-04-02T13:32:24.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.407+0000 7f063cf50640 1 -- 172.21.15.44:0/2400177381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f063815e1d0 msgr2=0x7f06380d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.407+0000 7f063cf50640 1 --2- 172.21.15.44:0/2400177381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f063815e1d0 0x7f06380d1100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0628009e50 tx=0x7f062802bed0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.408+0000 7f063cf50640 1 -- 172.21.15.44:0/2400177381 shutdown_connections 2024-04-02T13:32:24.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.408+0000 7f063cf50640 1 --2- 172.21.15.44:0/2400177381 >> v2:172.21.15.69:3300/0 conn(0x7f06380cec60 0x7f06380d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.408+0000 7f063cf50640 1 --2- 172.21.15.44:0/2400177381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f063815e1d0 0x7f06380d1100 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.408+0000 7f063cf50640 1 -- 172.21.15.44:0/2400177381 >> 172.21.15.44:0/2400177381 conn(0x7f063809d3b0 msgr2=0x7f06380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:24.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.408+0000 7f063cf50640 1 -- 172.21.15.44:0/2400177381 shutdown_connections 2024-04-02T13:32:24.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.408+0000 7f063cf50640 1 -- 172.21.15.44:0/2400177381 wait complete. 2024-04-02T13:32:24.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.409+0000 7f063cf50640 1 Processor -- start 2024-04-02T13:32:24.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.409+0000 7f063cf50640 1 -- start start 2024-04-02T13:32:24.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f063cf50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06380cec60 0x7f06380a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f063cf50640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f063815e1d0 0x7f063809de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f063cf50640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f063815e5b0 0x7f063809e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f063cf50640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f06380d2c00 con 0x7f063815e1d0 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f063cf50640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f06380cfa90 con 0x7f063815e5b0 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f063cf50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f06380d2d80 con 0x7f06380cec60 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f0635d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f063815e1d0 0x7f063809de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f0635d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f063815e1d0 0x7f063809de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42578/0 (socket says 172.21.15.44:42578) 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f0635d9b640 1 -- 172.21.15.44:0/13955997 learned_addr learned my addr 172.21.15.44:0/13955997 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f0636d9d640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f063815e5b0 0x7f063809e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f063659c640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06380cec60 0x7f06380a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f0635d9b640 1 -- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06380cec60 msgr2=0x7f06380a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f0635d9b640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06380cec60 0x7f06380a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f0635d9b640 1 -- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f063815e5b0 msgr2=0x7f063809e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f0635d9b640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f063815e5b0 0x7f063809e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f0635d9b640 1 -- 172.21.15.44:0/13955997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06380a1360 con 0x7f063815e1d0 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f0636d9d640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f063815e5b0 0x7f063809e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:24.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f0635d9b640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f063815e1d0 0x7f063809de20 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f062000e9e0 tx=0x7f062000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f062effd640 1 -- 172.21.15.44:0/13955997 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0620017070 con 0x7f063815e1d0 2024-04-02T13:32:24.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f062effd640 1 -- 172.21.15.44:0/13955997 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f062000c9d0 con 0x7f063815e1d0 2024-04-02T13:32:24.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f063cf50640 1 -- 172.21.15.44:0/13955997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06380a1640 con 0x7f063815e1d0 2024-04-02T13:32:24.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.410+0000 7f062effd640 1 -- 172.21.15.44:0/13955997 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062000cdc0 con 0x7f063815e1d0 2024-04-02T13:32:24.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.411+0000 7f063cf50640 1 -- 172.21.15.44:0/13955997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06380a1c00 con 0x7f063815e1d0 2024-04-02T13:32:24.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.411+0000 7f063cf50640 1 -- 172.21.15.44:0/13955997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f06380684f0 con 0x7f063815e1d0 2024-04-02T13:32:24.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.412+0000 7f062effd640 1 -- 172.21.15.44:0/13955997 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f062001a070 con 0x7f063815e1d0 2024-04-02T13:32:24.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.412+0000 7f062effd640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f061806c8f0 0x7f061806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.413+0000 7f062effd640 1 -- 172.21.15.44:0/13955997 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f062008be30 con 0x7f063815e1d0 2024-04-02T13:32:24.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.413+0000 7f062effd640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x7f0618072ed0 0x7f0618075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.413+0000 7f062effd640 1 -- 172.21.15.44:0/13955997 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06180759e0 con 0x7f0618072ed0 2024-04-02T13:32:24.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.413+0000 7f062effd640 1 -- 172.21.15.44:0/13955997 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=126) ==== 24+0+0 (secure 0 0 0) 0x7f062008e530 con 0x7f063815e1d0 2024-04-02T13:32:24.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.413+0000 7f063659c640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f061806c8f0 0x7f061806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.413+0000 7f0636d9d640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x7f0618072ed0 0x7f0618075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.413+0000 7f063659c640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f061806c8f0 0x7f061806edb0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f0628035000 tx=0x7f0628030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.413+0000 7f0636d9d640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x7f0618072ed0 0x7f0618075310 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.414+0000 7f062effd640 1 -- 172.21.15.44:0/13955997 <== osd.6 v2:172.21.15.69:6802/2604158175 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06180759e0 con 0x7f0618072ed0 2024-04-02T13:32:24.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.427+0000 7f063cf50640 1 -- 172.21.15.44:0/13955997 --> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f063815f2f0 con 0x7f0618072ed0 2024-04-02T13:32:24.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.428+0000 7f062effd640 1 -- 172.21.15.44:0/13955997 <== osd.6 v2:172.21.15.69:6802/2604158175 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f063815f2f0 con 0x7f0618072ed0 2024-04-02T13:32:24.430 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:32:24.430 INFO:teuthology.orchestra.run.smithi044.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T13:32:24.430 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:32:24.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.428+0000 7f063cf50640 1 -- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x7f0618072ed0 msgr2=0x7f0618075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.428+0000 7f063cf50640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:6802/2604158175,v1:172.21.15.69:6804/2604158175] conn(0x7f0618072ed0 0x7f0618075310 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.428+0000 7f063cf50640 1 -- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f061806c8f0 msgr2=0x7f061806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.428+0000 7f063cf50640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f061806c8f0 0x7f061806edb0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f0628035000 tx=0x7f0628030000 comp rx=0 tx=0).stop 2024-04-02T13:32:24.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.428+0000 7f063cf50640 1 -- 172.21.15.44:0/13955997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f063815e1d0 msgr2=0x7f063809de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.428+0000 7f063cf50640 1 --2- 172.21.15.44:0/13955997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f063815e1d0 0x7f063809de20 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f062000e9e0 tx=0x7f062000eea0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.428+0000 7f0636d9d640 1 -- 172.21.15.44:0/13955997 reap_dead start 2024-04-02T13:32:24.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.428+0000 7f063cf50640 1 -- 172.21.15.44:0/13955997 shutdown_connections 2024-04-02T13:32:24.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.428+0000 7f063cf50640 1 -- 172.21.15.44:0/13955997 >> 172.21.15.44:0/13955997 conn(0x7f063809d3b0 msgr2=0x7f06380b7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:24.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.428+0000 7f063cf50640 1 -- 172.21.15.44:0/13955997 shutdown_connections 2024-04-02T13:32:24.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.428+0000 7f063cf50640 1 -- 172.21.15.44:0/13955997 wait complete. 2024-04-02T13:32:24.441 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-02T13:32:24.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.540+0000 7f70650b7640 1 Processor -- start 2024-04-02T13:32:24.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.540+0000 7f70650b7640 1 -- start start 2024-04-02T13:32:24.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.540+0000 7f70650b7640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f706015c160 0x7f706006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.540+0000 7f70650b7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70600d0890 0x7f70600cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.540+0000 7f70650b7640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f706006bab0 con 0x7f7060160600 2024-04-02T13:32:24.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.540+0000 7f70650b7640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f706006bd90 con 0x7f706015c160 2024-04-02T13:32:24.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.540+0000 7f70650b7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f70600ccec0 con 0x7f70600d0890 2024-04-02T13:32:24.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.541+0000 7f705ed9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f706015c160 0x7f706006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.541+0000 7f705659c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7060160600 0x7f70600d0180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57152/0 (socket says 172.21.15.44:57152) 2024-04-02T13:32:24.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.541+0000 7f705659c640 1 -- 172.21.15.44:0/2392245689 learned_addr learned my addr 172.21.15.44:0/2392245689 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:24.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.541+0000 7f705f59e640 1 --2- 172.21.15.44:0/2392245689 >> v2:172.21.15.69:3300/0 conn(0x7f70600d0890 0x7f70600cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.541+0000 7f705ed9d640 1 -- 172.21.15.44:0/2392245689 >> v2:172.21.15.69:3300/0 conn(0x7f70600d0890 msgr2=0x7f70600cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:24.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.541+0000 7f705ed9d640 1 --2- 172.21.15.44:0/2392245689 >> v2:172.21.15.69:3300/0 conn(0x7f70600d0890 0x7f70600cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.541+0000 7f705ed9d640 1 -- 172.21.15.44:0/2392245689 >> v1:172.21.15.44:6789/0 conn(0x7f7060160600 legacy=0x7f70600d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.541+0000 7f705ed9d640 1 -- 172.21.15.44:0/2392245689 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70600cd1a0 con 0x7f706015c160 2024-04-02T13:32:24.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.541+0000 7f705e59c640 1 -- 172.21.15.44:0/2392245689 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2241460480 0 0) 0x7f706006bab0 con 0x7f7060160600 2024-04-02T13:32:24.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.541+0000 7f705f59e640 1 --2- 172.21.15.44:0/2392245689 >> v2:172.21.15.69:3300/0 conn(0x7f70600d0890 0x7f70600cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:24.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.542+0000 7f705ed9d640 1 --2- 172.21.15.44:0/2392245689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f706015c160 0x7f706006b4d0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f70600cfa20 tx=0x7f705002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e804413d3ec477f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.542+0000 7f705e59c640 1 -- 172.21.15.44:0/2392245689 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70500315a0 con 0x7f706015c160 2024-04-02T13:32:24.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.542+0000 7f705e59c640 1 -- 172.21.15.44:0/2392245689 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7050037070 con 0x7f706015c160 2024-04-02T13:32:24.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.542+0000 7f705e59c640 1 -- 172.21.15.44:0/2392245689 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7050031960 con 0x7f706015c160 2024-04-02T13:32:24.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.542+0000 7f70650b7640 1 -- 172.21.15.44:0/2392245689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f706015c160 msgr2=0x7f706006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.542+0000 7f70650b7640 1 --2- 172.21.15.44:0/2392245689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f706015c160 0x7f706006b4d0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f70600cfa20 tx=0x7f705002b840 comp rx=0 tx=0).stop 2024-04-02T13:32:24.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.542+0000 7f70650b7640 1 -- 172.21.15.44:0/2392245689 shutdown_connections 2024-04-02T13:32:24.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.542+0000 7f70650b7640 1 --2- 172.21.15.44:0/2392245689 >> v2:172.21.15.69:3300/0 conn(0x7f70600d0890 0x7f70600cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.542+0000 7f70650b7640 1 --2- 172.21.15.44:0/2392245689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f706015c160 0x7f706006b4d0 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.542+0000 7f70650b7640 1 -- 172.21.15.44:0/2392245689 >> 172.21.15.44:0/2392245689 conn(0x7f70600b2980 msgr2=0x7f70600b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:24.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.542+0000 7f70650b7640 1 -- 172.21.15.44:0/2392245689 shutdown_connections 2024-04-02T13:32:24.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.542+0000 7f70650b7640 1 -- 172.21.15.44:0/2392245689 wait complete. 2024-04-02T13:32:24.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.544+0000 7f70650b7640 1 Processor -- start 2024-04-02T13:32:24.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.544+0000 7f70650b7640 1 -- start start 2024-04-02T13:32:24.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f70650b7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70600d0890 0x7f70600c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f70650b7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f706015c160 0x7f70601e7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f70650b7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7060160600 0x7f706006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f70650b7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f70600add60 con 0x7f706015c160 2024-04-02T13:32:24.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f70650b7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7060160a60 con 0x7f70600d0890 2024-04-02T13:32:24.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f70650b7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f70600adee0 con 0x7f7060160600 2024-04-02T13:32:24.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f705659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f706015c160 0x7f70601e7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f705659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f706015c160 0x7f70601e7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40750/0 (socket says 172.21.15.44:40750) 2024-04-02T13:32:24.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f705659c640 1 -- 172.21.15.44:0/2780702949 learned_addr learned my addr 172.21.15.44:0/2780702949 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:24.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f705ed9d640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70600d0890 0x7f70600c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f705f59e640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7060160600 0x7f706006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f705659c640 1 -- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7060160600 msgr2=0x7f706006eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f705659c640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7060160600 0x7f706006eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f705659c640 1 -- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70600d0890 msgr2=0x7f70600c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f705659c640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70600d0890 0x7f70600c4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f705659c640 1 -- 172.21.15.44:0/2780702949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70601e6b50 con 0x7f706015c160 2024-04-02T13:32:24.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f705659c640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f706015c160 0x7f70601e7bf0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f704800c460 tx=0x7f704800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f7057fff640 1 -- 172.21.15.44:0/2780702949 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048018070 con 0x7f706015c160 2024-04-02T13:32:24.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f7057fff640 1 -- 172.21.15.44:0/2780702949 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7048007500 con 0x7f706015c160 2024-04-02T13:32:24.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f7057fff640 1 -- 172.21.15.44:0/2780702949 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048004a20 con 0x7f706015c160 2024-04-02T13:32:24.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f70650b7640 1 -- 172.21.15.44:0/2780702949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70601e6e30 con 0x7f706015c160 2024-04-02T13:32:24.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.545+0000 7f70650b7640 1 -- 172.21.15.44:0/2780702949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70601e73f0 con 0x7f706015c160 2024-04-02T13:32:24.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.546+0000 7f70650b7640 1 -- 172.21.15.44:0/2780702949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70600684f0 con 0x7f706015c160 2024-04-02T13:32:24.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.547+0000 7f7057fff640 1 -- 172.21.15.44:0/2780702949 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f704801b070 con 0x7f706015c160 2024-04-02T13:32:24.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.547+0000 7f7057fff640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f704006c8f0 0x7f704006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.547+0000 7f7057fff640 1 -- 172.21.15.44:0/2780702949 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f704808ad20 con 0x7f706015c160 2024-04-02T13:32:24.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.547+0000 7f705ed9d640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f704006c8f0 0x7f704006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.548+0000 7f705ed9d640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f704006c8f0 0x7f704006edb0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f7050036000 tx=0x7f7050008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.552+0000 7f7057fff640 1 -- 172.21.15.44:0/2780702949 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f7048059210 con 0x7f706015c160 2024-04-02T13:32:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.737+0000 7f70650b7640 1 -- 172.21.15.44:0/2780702949 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f70600728d0 con 0x7f704006c8f0 2024-04-02T13:32:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.738+0000 7f7057fff640 1 -- 172.21.15.44:0/2780702949 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f70600728d0 con 0x7f704006c8f0 2024-04-02T13:32:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:instructed osd(s) 6 to deep-scrub 2024-04-02T13:32:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.739+0000 7f70650b7640 1 -- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f704006c8f0 msgr2=0x7f704006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.739+0000 7f70650b7640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f704006c8f0 0x7f704006edb0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f7050036000 tx=0x7f7050008030 comp rx=0 tx=0).stop 2024-04-02T13:32:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.739+0000 7f70650b7640 1 -- 172.21.15.44:0/2780702949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f706015c160 msgr2=0x7f70601e7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.739+0000 7f70650b7640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f706015c160 0x7f70601e7bf0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f704800c460 tx=0x7f704800c920 comp rx=0 tx=0).stop 2024-04-02T13:32:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.739+0000 7f70650b7640 1 -- 172.21.15.44:0/2780702949 shutdown_connections 2024-04-02T13:32:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.739+0000 7f70650b7640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f704006c8f0 0x7f704006edb0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.739+0000 7f70650b7640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7060160600 0x7f706006eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.739+0000 7f70650b7640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f706015c160 0x7f70601e7bf0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.740+0000 7f70650b7640 1 --2- 172.21.15.44:0/2780702949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70600d0890 0x7f70600c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.740+0000 7f70650b7640 1 -- 172.21.15.44:0/2780702949 >> 172.21.15.44:0/2780702949 conn(0x7f70600b2980 msgr2=0x7f70600ce960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.740+0000 7f70650b7640 1 -- 172.21.15.44:0/2780702949 shutdown_connections 2024-04-02T13:32:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.740+0000 7f70650b7640 1 -- 172.21.15.44:0/2780702949 wait complete. 2024-04-02T13:32:24.753 INFO:tasks.ceph:Scrubbing osd.7 2024-04-02T13:32:24.753 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-02T13:32:24.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.856+0000 7fcfdcff6640 1 Processor -- start 2024-04-02T13:32:24.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.856+0000 7fcfdcff6640 1 -- start start 2024-04-02T13:32:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfdcff6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcfd80b1540 0x7fcfd80b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfdcff6640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcfd80ce790 0x7fcfd806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfdcff6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd80ced00 con 0x7fcfd815e470 2024-04-02T13:32:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfdcff6640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fcfd80cefe0 con 0x7fcfd80ce790 2024-04-02T13:32:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfdcff6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcfd80cf2c0 con 0x7fcfd80b1540 2024-04-02T13:32:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfd659c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fcfd815e470 0x7fcfd80d1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57168/0 (socket says 172.21.15.44:57168) 2024-04-02T13:32:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfd6d9d640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fcfd80ce790 0x7fcfd806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfd659c640 1 -- 172.21.15.44:0/2950381631 learned_addr learned my addr 172.21.15.44:0/2950381631 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfd5d9b640 1 --2- 172.21.15.44:0/2950381631 >> v2:172.21.15.69:3300/0 conn(0x7fcfd80b1540 0x7fcfd80b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfd6d9d640 1 -- 172.21.15.44:0/2950381631 >> v2:172.21.15.69:3300/0 conn(0x7fcfd80b1540 msgr2=0x7fcfd80b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfd6d9d640 1 --2- 172.21.15.44:0/2950381631 >> v2:172.21.15.69:3300/0 conn(0x7fcfd80b1540 0x7fcfd80b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfd6d9d640 1 -- 172.21.15.44:0/2950381631 >> v1:172.21.15.44:6789/0 conn(0x7fcfd815e470 legacy=0x7fcfd80d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfd6d9d640 1 -- 172.21.15.44:0/2950381631 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd80cf5a0 con 0x7fcfd80ce790 2024-04-02T13:32:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfd559a640 1 -- 172.21.15.44:0/2950381631 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 907287371 0 0) 0x7fcfd80ced00 con 0x7fcfd815e470 2024-04-02T13:32:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.857+0000 7fcfd5d9b640 1 --2- 172.21.15.44:0/2950381631 >> v2:172.21.15.69:3300/0 conn(0x7fcfd80b1540 0x7fcfd80b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.858+0000 7fcfd6d9d640 1 --2- 172.21.15.44:0/2950381631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd80ce790 0x7fcfd806d0f0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8163070 tx=0x7fcfd002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e0990d194045ef server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.858+0000 7fcfd559a640 1 -- 172.21.15.44:0/2950381631 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0039070 con 0x7fcfd80ce790 2024-04-02T13:32:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.858+0000 7fcfd559a640 1 -- 172.21.15.44:0/2950381631 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd002cbf0 con 0x7fcfd80ce790 2024-04-02T13:32:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.858+0000 7fcfd559a640 1 -- 172.21.15.44:0/2950381631 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0034440 con 0x7fcfd80ce790 2024-04-02T13:32:24.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.858+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2950381631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd80ce790 msgr2=0x7fcfd806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.858+0000 7fcfdcff6640 1 --2- 172.21.15.44:0/2950381631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd80ce790 0x7fcfd806d0f0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8163070 tx=0x7fcfd002be40 comp rx=0 tx=0).stop 2024-04-02T13:32:24.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.859+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2950381631 shutdown_connections 2024-04-02T13:32:24.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.859+0000 7fcfdcff6640 1 --2- 172.21.15.44:0/2950381631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd80ce790 0x7fcfd806d0f0 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.859+0000 7fcfdcff6640 1 --2- 172.21.15.44:0/2950381631 >> v2:172.21.15.69:3300/0 conn(0x7fcfd80b1540 0x7fcfd80b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.859+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2950381631 >> 172.21.15.44:0/2950381631 conn(0x7fcfd809aa20 msgr2=0x7fcfd809ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:24.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.859+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2950381631 shutdown_connections 2024-04-02T13:32:24.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.859+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2950381631 wait complete. 2024-04-02T13:32:24.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.860+0000 7fcfdcff6640 1 Processor -- start 2024-04-02T13:32:24.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.860+0000 7fcfdcff6640 1 -- start start 2024-04-02T13:32:24.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfdcff6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd80b1540 0x7fcfd80b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfdcff6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd80ce790 0x7fcfd8150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfdcff6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfd815e470 0x7fcfd80c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfdcff6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd80d22c0 con 0x7fcfd80ce790 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfdcff6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd806cb60 con 0x7fcfd80b1540 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfdcff6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd80d2440 con 0x7fcfd815e470 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfd5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd80ce790 0x7fcfd8150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfd5d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd80ce790 0x7fcfd8150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40776/0 (socket says 172.21.15.44:40776) 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfd5d9b640 1 -- 172.21.15.44:0/2864763398 learned_addr learned my addr 172.21.15.44:0/2864763398 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfd659c640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd80b1540 0x7fcfd80b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfd6d9d640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfd815e470 0x7fcfd80c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfd5d9b640 1 -- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfd815e470 msgr2=0x7fcfd80c6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfd5d9b640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfd815e470 0x7fcfd80c6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfd5d9b640 1 -- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd80b1540 msgr2=0x7fcfd80b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfd5d9b640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd80b1540 0x7fcfd80b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfd5d9b640 1 -- 172.21.15.44:0/2864763398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd80c5b50 con 0x7fcfd80ce790 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfd659c640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd80b1540 0x7fcfd80b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfd6d9d640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfd815e470 0x7fcfd80c6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T13:32:24.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfd5d9b640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd80ce790 0x7fcfd8150c30 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00c7f0 tx=0x7fcfcc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfc6ffd640 1 -- 172.21.15.44:0/2864763398 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc015070 con 0x7fcfd80ce790 2024-04-02T13:32:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfc6ffd640 1 -- 172.21.15.44:0/2864763398 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfcc007a30 con 0x7fcfd80ce790 2024-04-02T13:32:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2864763398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd80c5e30 con 0x7fcfd80ce790 2024-04-02T13:32:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfc6ffd640 1 -- 172.21.15.44:0/2864763398 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc007e00 con 0x7fcfd80ce790 2024-04-02T13:32:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.861+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2864763398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd80c63f0 con 0x7fcfd80ce790 2024-04-02T13:32:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.862+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2864763398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcfd80684f0 con 0x7fcfd80ce790 2024-04-02T13:32:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.863+0000 7fcfc6ffd640 1 -- 172.21.15.44:0/2864763398 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcfcc018070 con 0x7fcfd80ce790 2024-04-02T13:32:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.863+0000 7fcfc6ffd640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcfb806c8f0 0x7fcfb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.863+0000 7fcfc6ffd640 1 -- 172.21.15.44:0/2864763398 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7fcfcc08ba10 con 0x7fcfd80ce790 2024-04-02T13:32:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.863+0000 7fcfc6ffd640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x7fcfb8072ed0 0x7fcfb8075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.863+0000 7fcfc6ffd640 1 -- 172.21.15.44:0/2864763398 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfb80759e0 con 0x7fcfb8072ed0 2024-04-02T13:32:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.863+0000 7fcfc6ffd640 1 -- 172.21.15.44:0/2864763398 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=126) ==== 24+0+0 (secure 0 0 0) 0x7fcfcc0902c0 con 0x7fcfd80ce790 2024-04-02T13:32:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.863+0000 7fcfd659c640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcfb806c8f0 0x7fcfb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.864+0000 7fcfd6d9d640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x7fcfb8072ed0 0x7fcfb8075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.864+0000 7fcfd659c640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcfb806c8f0 0x7fcfb806edb0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8000f60 tx=0x7fcfc8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.864+0000 7fcfd6d9d640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x7fcfb8072ed0 0x7fcfb8075310 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.865+0000 7fcfc6ffd640 1 -- 172.21.15.44:0/2864763398 <== osd.7 v2:172.21.15.69:6824/1292475712 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfb80759e0 con 0x7fcfb8072ed0 2024-04-02T13:32:24.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.878+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2864763398 --> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcfd80d2d10 con 0x7fcfb8072ed0 2024-04-02T13:32:24.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.879+0000 7fcfc6ffd640 1 -- 172.21.15.44:0/2864763398 <== osd.7 v2:172.21.15.69:6824/1292475712 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcfd80d2d10 con 0x7fcfb8072ed0 2024-04-02T13:32:24.881 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-02T13:32:24.881 INFO:teuthology.orchestra.run.smithi044.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T13:32:24.882 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-02T13:32:24.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.879+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x7fcfb8072ed0 msgr2=0x7fcfb8075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.879+0000 7fcfdcff6640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:6824/1292475712,v1:172.21.15.69:6825/1292475712] conn(0x7fcfb8072ed0 0x7fcfb8075310 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.879+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcfb806c8f0 msgr2=0x7fcfb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.879+0000 7fcfdcff6640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fcfb806c8f0 0x7fcfb806edb0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8000f60 tx=0x7fcfc8009040 comp rx=0 tx=0).stop 2024-04-02T13:32:24.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.879+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2864763398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd80ce790 msgr2=0x7fcfd8150c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.879+0000 7fcfdcff6640 1 --2- 172.21.15.44:0/2864763398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd80ce790 0x7fcfd8150c30 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00c7f0 tx=0x7fcfcc00cbf0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.879+0000 7fcfd6d9d640 1 -- 172.21.15.44:0/2864763398 reap_dead start 2024-04-02T13:32:24.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.879+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2864763398 shutdown_connections 2024-04-02T13:32:24.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.879+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2864763398 >> 172.21.15.44:0/2864763398 conn(0x7fcfd809aa20 msgr2=0x7fcfd80d1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:24.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.879+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2864763398 shutdown_connections 2024-04-02T13:32:24.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.879+0000 7fcfdcff6640 1 -- 172.21.15.44:0/2864763398 wait complete. 2024-04-02T13:32:24.892 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-02T13:32:24.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.991+0000 7fd7ac060640 1 Processor -- start 2024-04-02T13:32:24.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.991+0000 7fd7ac060640 1 -- start start 2024-04-02T13:32:24.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.992+0000 7fd7ac060640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd7a40d0a30 0x7fd7a406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.992+0000 7fd7ac060640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd7a40d0e10 0x7fd7a40cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:24.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.992+0000 7fd7ac060640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7a406c1d0 con 0x7fd7a40cfe30 2024-04-02T13:32:24.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.992+0000 7fd7ac060640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fd7a406cb00 con 0x7fd7a40d0e10 2024-04-02T13:32:24.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.992+0000 7fd7ac060640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd7a406c4b0 con 0x7fd7a40d0a30 2024-04-02T13:32:24.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.992+0000 7fd7a95fb640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fd7a40d0e10 0x7fd7a40cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.992+0000 7fd7aa5fd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd7a40cfe30 0x7fd7a40d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57182/0 (socket says 172.21.15.44:57182) 2024-04-02T13:32:24.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.992+0000 7fd7aa5fd640 1 -- 172.21.15.44:0/1925092605 learned_addr learned my addr 172.21.15.44:0/1925092605 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:24.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.993+0000 7fd7a9dfc640 1 --2- 172.21.15.44:0/1925092605 >> v2:172.21.15.69:3300/0 conn(0x7fd7a40d0a30 0x7fd7a406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:24.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.993+0000 7fd7a95fb640 1 -- 172.21.15.44:0/1925092605 >> v2:172.21.15.69:3300/0 conn(0x7fd7a40d0a30 msgr2=0x7fd7a406cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:24.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.993+0000 7fd7a95fb640 1 --2- 172.21.15.44:0/1925092605 >> v2:172.21.15.69:3300/0 conn(0x7fd7a40d0a30 0x7fd7a406cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.993+0000 7fd7a95fb640 1 -- 172.21.15.44:0/1925092605 >> v1:172.21.15.44:6789/0 conn(0x7fd7a40cfe30 legacy=0x7fd7a40d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.993+0000 7fd7a95fb640 1 -- 172.21.15.44:0/1925092605 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a406c790 con 0x7fd7a40d0e10 2024-04-02T13:32:24.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.993+0000 7fd7a8dfa640 1 -- 172.21.15.44:0/1925092605 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 682080906 0 0) 0x7fd7a406c1d0 con 0x7fd7a40cfe30 2024-04-02T13:32:24.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.993+0000 7fd7a9dfc640 1 --2- 172.21.15.44:0/1925092605 >> v2:172.21.15.69:3300/0 conn(0x7fd7a40d0a30 0x7fd7a406cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T13:32:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.993+0000 7fd7a95fb640 1 --2- 172.21.15.44:0/1925092605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7a40d0e10 0x7fd7a40cf8f0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fd7a406de20 tx=0x7fd79402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cfaa8d3a4dbb5d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.994+0000 7fd7a8dfa640 1 -- 172.21.15.44:0/1925092605 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7940315a0 con 0x7fd7a40d0e10 2024-04-02T13:32:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.994+0000 7fd7a8dfa640 1 -- 172.21.15.44:0/1925092605 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd794037070 con 0x7fd7a40d0e10 2024-04-02T13:32:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.994+0000 7fd7a8dfa640 1 -- 172.21.15.44:0/1925092605 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794031960 con 0x7fd7a40d0e10 2024-04-02T13:32:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.994+0000 7fd7ac060640 1 -- 172.21.15.44:0/1925092605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7a40d0e10 msgr2=0x7fd7a40cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.994+0000 7fd7ac060640 1 --2- 172.21.15.44:0/1925092605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7a40d0e10 0x7fd7a40cf8f0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fd7a406de20 tx=0x7fd79402b840 comp rx=0 tx=0).stop 2024-04-02T13:32:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.994+0000 7fd7ac060640 1 -- 172.21.15.44:0/1925092605 shutdown_connections 2024-04-02T13:32:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.994+0000 7fd7ac060640 1 --2- 172.21.15.44:0/1925092605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7a40d0e10 0x7fd7a40cf8f0 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.994+0000 7fd7ac060640 1 --2- 172.21.15.44:0/1925092605 >> v2:172.21.15.69:3300/0 conn(0x7fd7a40d0a30 0x7fd7a406cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.994+0000 7fd7ac060640 1 -- 172.21.15.44:0/1925092605 >> 172.21.15.44:0/1925092605 conn(0x7fd7a409d470 msgr2=0x7fd7a40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.994+0000 7fd7ac060640 1 -- 172.21.15.44:0/1925092605 shutdown_connections 2024-04-02T13:32:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.994+0000 7fd7ac060640 1 -- 172.21.15.44:0/1925092605 wait complete. 2024-04-02T13:32:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.995+0000 7fd7ac060640 1 Processor -- start 2024-04-02T13:32:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.996+0000 7fd7ac060640 1 -- start start 2024-04-02T13:32:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.996+0000 7fd7ac060640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a40cfe30 0x7fd7a4160e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.996+0000 7fd7ac060640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7a40d0a30 0x7fd7a415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.996+0000 7fd7ac060640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7a40d0e10 0x7fd7a415aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.996+0000 7fd7ac060640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a415e870 con 0x7fd7a40d0a30 2024-04-02T13:32:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.996+0000 7fd7ac060640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a415e6f0 con 0x7fd7a40d0e10 2024-04-02T13:32:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.996+0000 7fd7ac060640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd7a40d4da0 con 0x7fd7a40cfe30 2024-04-02T13:32:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.996+0000 7fd7a95fb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7a40d0a30 0x7fd7a415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.996+0000 7fd7a95fb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7a40d0a30 0x7fd7a415aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40798/0 (socket says 172.21.15.44:40798) 2024-04-02T13:32:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.996+0000 7fd7a95fb640 1 -- 172.21.15.44:0/3719104266 learned_addr learned my addr 172.21.15.44:0/3719104266 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.997+0000 7fd7a9dfc640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a40cfe30 0x7fd7a4160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.997+0000 7fd7aa5fd640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7a40d0e10 0x7fd7a415aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.997+0000 7fd7a95fb640 1 -- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a40cfe30 msgr2=0x7fd7a4160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.997+0000 7fd7a95fb640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a40cfe30 0x7fd7a4160e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.997+0000 7fd7a95fb640 1 -- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7a40d0e10 msgr2=0x7fd7a415aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.997+0000 7fd7a95fb640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7a40d0e10 0x7fd7a415aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.997+0000 7fd7a95fb640 1 -- 172.21.15.44:0/3719104266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a40c93a0 con 0x7fd7a40d0a30 2024-04-02T13:32:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.997+0000 7fd7a95fb640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7a40d0a30 0x7fd7a415aab0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fd794000c00 tx=0x7fd7940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.997+0000 7fd79a7fc640 1 -- 172.21.15.44:0/3719104266 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794004230 con 0x7fd7a40d0a30 2024-04-02T13:32:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.997+0000 7fd79a7fc640 1 -- 172.21.15.44:0/3719104266 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7940043f0 con 0x7fd7a40d0a30 2024-04-02T13:32:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.997+0000 7fd79a7fc640 1 -- 172.21.15.44:0/3719104266 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794003850 con 0x7fd7a40d0a30 2024-04-02T13:32:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.997+0000 7fd7ac060640 1 -- 172.21.15.44:0/3719104266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a41600d0 con 0x7fd7a40d0a30 2024-04-02T13:32:25.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.997+0000 7fd7ac060640 1 -- 172.21.15.44:0/3719104266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a4160690 con 0x7fd7a40d0a30 2024-04-02T13:32:25.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.998+0000 7fd7ac060640 1 -- 172.21.15.44:0/3719104266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7a40684f0 con 0x7fd7a40d0a30 2024-04-02T13:32:25.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.999+0000 7fd79a7fc640 1 -- 172.21.15.44:0/3719104266 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd7940039f0 con 0x7fd7a40d0a30 2024-04-02T13:32:25.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.999+0000 7fd79a7fc640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd78806c920 0x7fd78806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:25.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.999+0000 7fd79a7fc640 1 -- 172.21.15.44:0/3719104266 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7fd79402f030 con 0x7fd7a40d0a30 2024-04-02T13:32:25.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:24.999+0000 7fd7a9dfc640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd78806c920 0x7fd78806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:25.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.000+0000 7fd7a9dfc640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd78806c920 0x7fd78806ede0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0096d0 tx=0x7fd79c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:25.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.003+0000 7fd79a7fc640 1 -- 172.21.15.44:0/3719104266 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fd794041070 con 0x7fd7a40d0a30 2024-04-02T13:32:25.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.187+0000 7fd7ac060640 1 -- 172.21.15.44:0/3719104266 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fd7a415fe80 con 0x7fd78806c920 2024-04-02T13:32:25.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.188+0000 7fd79a7fc640 1 -- 172.21.15.44:0/3719104266 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd7a415fe80 con 0x7fd78806c920 2024-04-02T13:32:25.191 INFO:teuthology.orchestra.run.smithi044.stderr:instructed osd(s) 7 to deep-scrub 2024-04-02T13:32:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.190+0000 7fd7ac060640 1 -- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd78806c920 msgr2=0x7fd78806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.190+0000 7fd7ac060640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd78806c920 0x7fd78806ede0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0096d0 tx=0x7fd79c008040 comp rx=0 tx=0).stop 2024-04-02T13:32:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.190+0000 7fd7ac060640 1 -- 172.21.15.44:0/3719104266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7a40d0a30 msgr2=0x7fd7a415aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.190+0000 7fd7ac060640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7a40d0a30 0x7fd7a415aab0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fd794000c00 tx=0x7fd7940046c0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.190+0000 7fd7ac060640 1 -- 172.21.15.44:0/3719104266 shutdown_connections 2024-04-02T13:32:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.190+0000 7fd7ac060640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fd78806c920 0x7fd78806ede0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.190+0000 7fd7ac060640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7a40d0e10 0x7fd7a415aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.190+0000 7fd7ac060640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7a40d0a30 0x7fd7a415aab0 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.190+0000 7fd7ac060640 1 --2- 172.21.15.44:0/3719104266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a40cfe30 0x7fd7a4160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.190+0000 7fd7ac060640 1 -- 172.21.15.44:0/3719104266 >> 172.21.15.44:0/3719104266 conn(0x7fd7a409d470 msgr2=0x7fd7a415cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:25.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.190+0000 7fd7ac060640 1 -- 172.21.15.44:0/3719104266 shutdown_connections 2024-04-02T13:32:25.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.190+0000 7fd7ac060640 1 -- 172.21.15.44:0/3719104266 wait complete. 2024-04-02T13:32:25.203 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:32:25.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.302+0000 7f9ad5c88640 1 Processor -- start 2024-04-02T13:32:25.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.302+0000 7f9ad5c88640 1 -- start start 2024-04-02T13:32:25.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.303+0000 7f9ad5c88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad015c160 0x7f9ad006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:25.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.303+0000 7f9ad5c88640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9ad0160600 0x7f9ad00d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.303+0000 7f9ad5c88640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad006bab0 con 0x7f9ad00d06c0 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.303+0000 7f9ad5c88640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f9ad00ce6d0 con 0x7f9ad0160600 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.303+0000 7f9ad5c88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9ad006bd90 con 0x7f9ad015c160 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.303+0000 7f9aceffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9ad0160600 0x7f9ad00d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.303+0000 7f9aceffd640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f9ad0160600 0x7f9ad00d0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40802/0 (socket says 172.21.15.44:40802) 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.303+0000 7f9aceffd640 1 -- 172.21.15.44:0/1070128528 learned_addr learned my addr 172.21.15.44:0/1070128528 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.303+0000 7f9acf7fe640 1 --2- 172.21.15.44:0/1070128528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad015c160 0x7f9ad006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.303+0000 7f9aceffd640 1 -- 172.21.15.44:0/1070128528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad015c160 msgr2=0x7f9ad006b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.303+0000 7f9aceffd640 1 --2- 172.21.15.44:0/1070128528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad015c160 0x7f9ad006b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.303+0000 7f9aceffd640 1 -- 172.21.15.44:0/1070128528 >> v1:172.21.15.69:6789/0 conn(0x7f9ad00d06c0 legacy=0x7f9ad00ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.303+0000 7f9aceffd640 1 -- 172.21.15.44:0/1070128528 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad00ccec0 con 0x7f9ad0160600 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.304+0000 7f9aceffd640 1 --2- 172.21.15.44:0/1070128528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0160600 0x7f9ad00d0180 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4009870 tx=0x7f9ac402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b234342639ee366 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.304+0000 7f9ace7fc640 1 -- 172.21.15.44:0/1070128528 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4035020 con 0x7f9ad0160600 2024-04-02T13:32:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.304+0000 7f9ace7fc640 1 -- 172.21.15.44:0/1070128528 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac4031640 con 0x7f9ad0160600 2024-04-02T13:32:25.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.304+0000 7f9ace7fc640 1 -- 172.21.15.44:0/1070128528 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac402d040 con 0x7f9ad0160600 2024-04-02T13:32:25.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.305+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1070128528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0160600 msgr2=0x7f9ad00d0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:25.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.305+0000 7f9ad5c88640 1 --2- 172.21.15.44:0/1070128528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0160600 0x7f9ad00d0180 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4009870 tx=0x7f9ac402b8f0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.305+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1070128528 shutdown_connections 2024-04-02T13:32:25.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.305+0000 7f9ad5c88640 1 --2- 172.21.15.44:0/1070128528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0160600 0x7f9ad00d0180 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.305+0000 7f9ad5c88640 1 --2- 172.21.15.44:0/1070128528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad015c160 0x7f9ad006b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.305+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1070128528 >> 172.21.15.44:0/1070128528 conn(0x7f9ad00b2980 msgr2=0x7f9ad00b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:25.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.305+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1070128528 shutdown_connections 2024-04-02T13:32:25.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.305+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1070128528 wait complete. 2024-04-02T13:32:25.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.306+0000 7f9ad5c88640 1 Processor -- start 2024-04-02T13:32:25.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.306+0000 7f9ad5c88640 1 -- start start 2024-04-02T13:32:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9ad5c88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad00d06c0 0x7f9ad01f1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9ad5c88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad015c160 0x7f9ad01f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9ad5c88640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0160600 0x7f9ad01f4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9ad5c88640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad0160a60 con 0x7f9ad0160600 2024-04-02T13:32:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9ad5c88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad00ccb70 con 0x7f9ad00d06c0 2024-04-02T13:32:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9ad5c88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9ad00add60 con 0x7f9ad015c160 2024-04-02T13:32:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9acffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0160600 0x7f9ad01f4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9acffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0160600 0x7f9ad01f4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40818/0 (socket says 172.21.15.44:40818) 2024-04-02T13:32:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9acffff640 1 -- 172.21.15.44:0/1312245925 learned_addr learned my addr 172.21.15.44:0/1312245925 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9aceffd640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad015c160 0x7f9ad01f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9acf7fe640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad00d06c0 0x7f9ad01f1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9acffff640 1 -- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad015c160 msgr2=0x7f9ad01f4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9acffff640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad015c160 0x7f9ad01f4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9acffff640 1 -- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad00d06c0 msgr2=0x7f9ad01f1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9acffff640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad00d06c0 0x7f9ad01f1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9acffff640 1 -- 172.21.15.44:0/1312245925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad00cb6e0 con 0x7f9ad0160600 2024-04-02T13:32:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9acffff640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0160600 0x7f9ad01f4980 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f9ac800c6b0 tx=0x7f9ac800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9ab7fff640 1 -- 172.21.15.44:0/1312245925 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8003b70 con 0x7f9ad0160600 2024-04-02T13:32:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9ab7fff640 1 -- 172.21.15.44:0/1312245925 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac8003d30 con 0x7f9ad0160600 2024-04-02T13:32:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9ab7fff640 1 -- 172.21.15.44:0/1312245925 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8005310 con 0x7f9ad0160600 2024-04-02T13:32:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.307+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1312245925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad01f5070 con 0x7f9ad0160600 2024-04-02T13:32:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.308+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1312245925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ad006e2c0 con 0x7f9ad0160600 2024-04-02T13:32:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.309+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1312245925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ad0160060 con 0x7f9ad0160600 2024-04-02T13:32:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.309+0000 7f9ab7fff640 1 -- 172.21.15.44:0/1312245925 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9ac80054b0 con 0x7f9ad0160600 2024-04-02T13:32:25.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.309+0000 7f9ab7fff640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9abc06c8f0 0x7f9abc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:25.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.310+0000 7f9ab7fff640 1 -- 172.21.15.44:0/1312245925 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f9ac808b6c0 con 0x7f9ad0160600 2024-04-02T13:32:25.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.310+0000 7f9acf7fe640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9abc06c8f0 0x7f9abc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:25.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.310+0000 7f9acf7fe640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9abc06c8f0 0x7f9abc06edb0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0006fd0 tx=0x7f9ac0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:25.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.314+0000 7f9ab7fff640 1 -- 172.21.15.44:0/1312245925 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f9ad0160060 con 0x7f9ad0160600 2024-04-02T13:32:25.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.498+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1312245925 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9ad00684f0 con 0x7f9abc06c8f0 2024-04-02T13:32:25.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.499+0000 7f9ab7fff640 1 -- 172.21.15.44:0/1312245925 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13390 (secure 0 0 0) 0x7f9ad00684f0 con 0x7f9abc06c8f0 2024-04-02T13:32:25.502 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:25.502 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:32:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.500+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9abc06c8f0 msgr2=0x7f9abc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.500+0000 7f9ad5c88640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9abc06c8f0 0x7f9abc06edb0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0006fd0 tx=0x7f9ac0008040 comp rx=0 tx=0).stop 2024-04-02T13:32:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.500+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1312245925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0160600 msgr2=0x7f9ad01f4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.500+0000 7f9ad5c88640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0160600 0x7f9ad01f4980 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f9ac800c6b0 tx=0x7f9ac800cb70 comp rx=0 tx=0).stop 2024-04-02T13:32:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.501+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1312245925 shutdown_connections 2024-04-02T13:32:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.501+0000 7f9ad5c88640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f9abc06c8f0 0x7f9abc06edb0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.501+0000 7f9ad5c88640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0160600 0x7f9ad01f4980 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.501+0000 7f9ad5c88640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad015c160 0x7f9ad01f4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.501+0000 7f9ad5c88640 1 --2- 172.21.15.44:0/1312245925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad00d06c0 0x7f9ad01f1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.501+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1312245925 >> 172.21.15.44:0/1312245925 conn(0x7f9ad00b2980 msgr2=0x7f9ad00ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.501+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1312245925 shutdown_connections 2024-04-02T13:32:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:25.501+0000 7f9ad5c88640 1 -- 172.21.15.44:0/1312245925 wait complete. 2024-04-02T13:32:25.514 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-04-02T13:32:25.055707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":262924,"kb_used_data":13480,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711796,"statfs":{"total":773094113280,"available":772824879104,"internally_reserved":0,"allocated":13803520,"data_stored":7075824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8388625,"num_objects":-5,"num_object_clones":0,"num_object_copies":-10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":-32,"num_read_kb":-13,"num_write":-29,"num_write_kb":-8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.508356"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":376,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-02T13:32:17.553383+0000","last_change":"2024-04-02T13:20:17.164158+0000","last_active":"2024-04-02T13:32:17.553383+0000","last_peered":"2024-04-02T13:32:17.553383+0000","last_clean":"2024-04-02T13:32:17.553383+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:32:17.553383+0000","last_undegraded":"2024-04-02T13:32:17.553383+0000","last_fullsized":"2024-04-02T13:32:17.553383+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T13:20:15.842691+0000","last_clean_scrub_stamp":"2024-04-02T13:20:15.842691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:08.176493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574997,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2170880,"data_stored":1327254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574997,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2170880,"data_stored":1327254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:32:25.517 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-02T13:20:15.842691+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=2, tm_hour=13, tm_min=20, tm_sec=15, tm_wday=1, tm_yday=93, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=2, tm_hour=13, tm_min=32, tm_sec=20, tm_wday=1, tm_yday=93, tm_isdst=0) 2024-04-02T13:32:25.517 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-02T13:32:45.518 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T13:32:45.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.630+0000 7fb29a37b640 1 Processor -- start 2024-04-02T13:32:45.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.630+0000 7fb29a37b640 1 -- start start 2024-04-02T13:32:45.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.631+0000 7fb29a37b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb2940cece0 0x7fb2940cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:45.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.631+0000 7fb29a37b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb29406ece0 0x7fb2940acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:45.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.631+0000 7fb29a37b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2940cf600 con 0x7fb2940d09f0 2024-04-02T13:32:45.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.631+0000 7fb29a37b640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fb29406d9e0 con 0x7fb2940cece0 2024-04-02T13:32:45.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.631+0000 7fb29a37b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb29406dc40 con 0x7fb29406ece0 2024-04-02T13:32:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.631+0000 7fb293fff640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fb2940cece0 0x7fb2940cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.631+0000 7fb2937fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb2940d09f0 0x7fb29406d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42386/0 (socket says 172.21.15.44:42386) 2024-04-02T13:32:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.631+0000 7fb2937fe640 1 -- 172.21.15.44:0/137331447 learned_addr learned my addr 172.21.15.44:0/137331447 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.632+0000 7fb298918640 1 --2- 172.21.15.44:0/137331447 >> v2:172.21.15.69:3300/0 conn(0x7fb29406ece0 0x7fb2940acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.632+0000 7fb293fff640 1 -- 172.21.15.44:0/137331447 >> v2:172.21.15.69:3300/0 conn(0x7fb29406ece0 msgr2=0x7fb2940acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.632+0000 7fb293fff640 1 --2- 172.21.15.44:0/137331447 >> v2:172.21.15.69:3300/0 conn(0x7fb29406ece0 0x7fb2940acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.632+0000 7fb293fff640 1 -- 172.21.15.44:0/137331447 >> v1:172.21.15.44:6789/0 conn(0x7fb2940d09f0 legacy=0x7fb29406d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.632+0000 7fb293fff640 1 -- 172.21.15.44:0/137331447 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29406df20 con 0x7fb2940cece0 2024-04-02T13:32:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.632+0000 7fb292ffd640 1 -- 172.21.15.44:0/137331447 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2347926395 0 0) 0x7fb2940cf600 con 0x7fb2940d09f0 2024-04-02T13:32:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.633+0000 7fb293fff640 1 --2- 172.21.15.44:0/137331447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2940cece0 0x7fb2940cf0c0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fb29406f1a0 tx=0x7fb27c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a284a2b8d99ff25 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.633+0000 7fb292ffd640 1 -- 172.21.15.44:0/137331447 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c0315a0 con 0x7fb2940cece0 2024-04-02T13:32:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.633+0000 7fb292ffd640 1 -- 172.21.15.44:0/137331447 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb27c037070 con 0x7fb2940cece0 2024-04-02T13:32:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.633+0000 7fb292ffd640 1 -- 172.21.15.44:0/137331447 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c031960 con 0x7fb2940cece0 2024-04-02T13:32:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.633+0000 7fb29a37b640 1 -- 172.21.15.44:0/137331447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2940cece0 msgr2=0x7fb2940cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.633+0000 7fb29a37b640 1 --2- 172.21.15.44:0/137331447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2940cece0 0x7fb2940cf0c0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fb29406f1a0 tx=0x7fb27c02b840 comp rx=0 tx=0).stop 2024-04-02T13:32:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.633+0000 7fb29a37b640 1 -- 172.21.15.44:0/137331447 shutdown_connections 2024-04-02T13:32:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.633+0000 7fb29a37b640 1 --2- 172.21.15.44:0/137331447 >> v2:172.21.15.69:3300/0 conn(0x7fb29406ece0 0x7fb2940acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.633+0000 7fb29a37b640 1 --2- 172.21.15.44:0/137331447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2940cece0 0x7fb2940cf0c0 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.633+0000 7fb29a37b640 1 -- 172.21.15.44:0/137331447 >> 172.21.15.44:0/137331447 conn(0x7fb2940cfcd0 msgr2=0x7fb2940d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.633+0000 7fb29a37b640 1 -- 172.21.15.44:0/137331447 shutdown_connections 2024-04-02T13:32:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.633+0000 7fb29a37b640 1 -- 172.21.15.44:0/137331447 wait complete. 2024-04-02T13:32:45.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.635+0000 7fb29a37b640 1 Processor -- start 2024-04-02T13:32:45.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.635+0000 7fb29a37b640 1 -- start start 2024-04-02T13:32:45.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.635+0000 7fb29a37b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb29406ece0 0x7fb29409cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:45.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.635+0000 7fb29a37b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2940cece0 0x7fb29409b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:45.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.635+0000 7fb29a37b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2940d09f0 0x7fb2941f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:45.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.635+0000 7fb29a37b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb29406e7c0 con 0x7fb2940d09f0 2024-04-02T13:32:45.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.635+0000 7fb29a37b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb29406e4e0 con 0x7fb29406ece0 2024-04-02T13:32:45.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.635+0000 7fb29a37b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb29415b220 con 0x7fb2940cece0 2024-04-02T13:32:45.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb298918640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2940d09f0 0x7fb2941f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:45.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb298918640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2940d09f0 0x7fb2941f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33850/0 (socket says 172.21.15.44:33850) 2024-04-02T13:32:45.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb298918640 1 -- 172.21.15.44:0/3226250175 learned_addr learned my addr 172.21.15.44:0/3226250175 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:45.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb293fff640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb29406ece0 0x7fb29409cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:45.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb2937fe640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2940cece0 0x7fb29409b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:45.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb298918640 1 -- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2940cece0 msgr2=0x7fb29409b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:45.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb298918640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2940cece0 0x7fb29409b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb298918640 1 -- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb29406ece0 msgr2=0x7fb29409cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:45.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb298918640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb29406ece0 0x7fb29409cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb298918640 1 -- 172.21.15.44:0/3226250175 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2941f4800 con 0x7fb2940d09f0 2024-04-02T13:32:45.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb298918640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2940d09f0 0x7fb2941f3e40 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00efc0 tx=0x7fb28c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:45.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb290ff9640 1 -- 172.21.15.44:0/3226250175 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c018020 con 0x7fb2940d09f0 2024-04-02T13:32:45.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb290ff9640 1 -- 172.21.15.44:0/3226250175 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb28c014480 con 0x7fb2940d09f0 2024-04-02T13:32:45.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb29a37b640 1 -- 172.21.15.44:0/3226250175 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2941f4ae0 con 0x7fb2940d09f0 2024-04-02T13:32:45.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb290ff9640 1 -- 172.21.15.44:0/3226250175 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c014870 con 0x7fb2940d09f0 2024-04-02T13:32:45.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.636+0000 7fb29a37b640 1 -- 172.21.15.44:0/3226250175 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb29409b890 con 0x7fb2940d09f0 2024-04-02T13:32:45.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.638+0000 7fb29a37b640 1 -- 172.21.15.44:0/3226250175 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb294074f20 con 0x7fb2940d09f0 2024-04-02T13:32:45.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.638+0000 7fb290ff9640 1 -- 172.21.15.44:0/3226250175 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb28c01b020 con 0x7fb2940d09f0 2024-04-02T13:32:45.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.638+0000 7fb290ff9640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb27406c920 0x7fb27406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:45.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.638+0000 7fb290ff9640 1 -- 172.21.15.44:0/3226250175 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7fb28c08c730 con 0x7fb2940d09f0 2024-04-02T13:32:45.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.638+0000 7fb293fff640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb27406c920 0x7fb27406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:45.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.639+0000 7fb293fff640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb27406c920 0x7fb27406ede0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb27c036000 tx=0x7fb27c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:45.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.643+0000 7fb290ff9640 1 -- 172.21.15.44:0/3226250175 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fb28c022020 con 0x7fb2940d09f0 2024-04-02T13:32:45.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.826+0000 7fb29a37b640 1 -- 172.21.15.44:0/3226250175 --> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb2940684f0 con 0x7fb27406c920 2024-04-02T13:32:45.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.827+0000 7fb290ff9640 1 -- 172.21.15.44:0/3226250175 <== mgr.4103 v2:172.21.15.69:6832/3569935720 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13379 (secure 0 0 0) 0x7fb2940684f0 con 0x7fb27406c920 2024-04-02T13:32:45.830 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:45.831 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-02T13:32:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.828+0000 7fb29a37b640 1 -- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb27406c920 msgr2=0x7fb27406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.828+0000 7fb29a37b640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb27406c920 0x7fb27406ede0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb27c036000 tx=0x7fb27c03d040 comp rx=0 tx=0).stop 2024-04-02T13:32:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.828+0000 7fb29a37b640 1 -- 172.21.15.44:0/3226250175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2940d09f0 msgr2=0x7fb2941f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:45.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.828+0000 7fb29a37b640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2940d09f0 0x7fb2941f3e40 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00efc0 tx=0x7fb28c00c490 comp rx=0 tx=0).stop 2024-04-02T13:32:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.828+0000 7fb29a37b640 1 -- 172.21.15.44:0/3226250175 shutdown_connections 2024-04-02T13:32:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.828+0000 7fb29a37b640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fb27406c920 0x7fb27406ede0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.828+0000 7fb29a37b640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2940d09f0 0x7fb2941f3e40 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.828+0000 7fb29a37b640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2940cece0 0x7fb29409b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.828+0000 7fb29a37b640 1 --2- 172.21.15.44:0/3226250175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb29406ece0 0x7fb29409cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.829+0000 7fb29a37b640 1 -- 172.21.15.44:0/3226250175 >> 172.21.15.44:0/3226250175 conn(0x7fb2940cfcd0 msgr2=0x7fb2940adfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.829+0000 7fb29a37b640 1 -- 172.21.15.44:0/3226250175 shutdown_connections 2024-04-02T13:32:45.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.829+0000 7fb29a37b640 1 -- 172.21.15.44:0/3226250175 wait complete. 2024-04-02T13:32:45.843 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-04-02T13:32:45.060037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":262912,"kb_used_data":13480,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711808,"statfs":{"total":773094113280,"available":772824891392,"internally_reserved":0,"allocated":13803520,"data_stored":7075824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002603"},"pg_stats":[{"pgid":"1.0","version":"126'119","reported_seq":387,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-02T13:32:25.875255+0000","last_change":"2024-04-02T13:32:25.875290+0000","last_active":"2024-04-02T13:32:25.875255+0000","last_peered":"2024-04-02T13:32:25.875255+0000","last_clean":"2024-04-02T13:32:25.875255+0000","last_became_active":"2024-04-02T13:20:17.163039+0000","last_became_peered":"2024-04-02T13:20:17.163039+0000","last_unstale":"2024-04-02T13:32:25.875255+0000","last_undegraded":"2024-04-02T13:32:25.875255+0000","last_fullsized":"2024-04-02T13:32:25.875255+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'119","last_scrub_stamp":"2024-04-02T13:32:25.875176+0000","last_deep_scrub":"126'119","last_deep_scrub_stamp":"2024-04-02T13:32:25.875176+0000","last_clean_scrub_stamp":"2024-04-02T13:32:25.875176+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:16:21.220785+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575001,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33296,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338544,"statfs":{"total":96636764160,"available":96602669056,"internally_reserved":0,"allocated":2170880,"data_stored":1327254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575000,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575000,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575000,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":736886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575001,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33296,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338544,"statfs":{"total":96636764160,"available":96602669056,"internally_reserved":0,"allocated":2170880,"data_stored":1327254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T13:32:45.844 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-02T13:32:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.942+0000 7fc141a2a640 1 Processor -- start 2024-04-02T13:32:45.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.942+0000 7fc141a2a640 1 -- start start 2024-04-02T13:32:45.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.943+0000 7fc141a2a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc13c15e520 0x7fc13c162950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.943+0000 7fc141a2a640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc13c06bbe0 0x7fc13c06e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.943+0000 7fc141a2a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc13c0d03e0 con 0x7fc13c0ad890 2024-04-02T13:32:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.943+0000 7fc141a2a640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7fc13c0b4610 con 0x7fc13c06bbe0 2024-04-02T13:32:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.943+0000 7fc141a2a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc13c162e90 con 0x7fc13c15e520 2024-04-02T13:32:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.943+0000 7fc13b7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc13c06bbe0 0x7fc13c06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.943+0000 7fc13b7fe640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7fc13c06bbe0 0x7fc13c06e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33860/0 (socket says 172.21.15.44:33860) 2024-04-02T13:32:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.943+0000 7fc13b7fe640 1 -- 172.21.15.44:0/1151921516 learned_addr learned my addr 172.21.15.44:0/1151921516 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.944+0000 7fc13affd640 1 --2- 172.21.15.44:0/1151921516 >> v2:172.21.15.69:3300/0 conn(0x7fc13c15e520 0x7fc13c162950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.944+0000 7fc13b7fe640 1 -- 172.21.15.44:0/1151921516 >> v2:172.21.15.69:3300/0 conn(0x7fc13c15e520 msgr2=0x7fc13c162950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.944+0000 7fc13b7fe640 1 --2- 172.21.15.44:0/1151921516 >> v2:172.21.15.69:3300/0 conn(0x7fc13c15e520 0x7fc13c162950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.944+0000 7fc13b7fe640 1 -- 172.21.15.44:0/1151921516 >> v1:172.21.15.69:6789/0 conn(0x7fc13c0ad890 legacy=0x7fc13c06b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:45.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.944+0000 7fc13b7fe640 1 -- 172.21.15.44:0/1151921516 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc13c0b3a90 con 0x7fc13c06bbe0 2024-04-02T13:32:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.944+0000 7fc13b7fe640 1 --2- 172.21.15.44:0/1151921516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc13c06bbe0 0x7fc13c06e380 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0d1640 tx=0x7fc13402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa205b14dbf9e5c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.945+0000 7fc139ffb640 1 -- 172.21.15.44:0/1151921516 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13402cb20 con 0x7fc13c06bbe0 2024-04-02T13:32:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.945+0000 7fc139ffb640 1 -- 172.21.15.44:0/1151921516 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc13402ccc0 con 0x7fc13c06bbe0 2024-04-02T13:32:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.945+0000 7fc139ffb640 1 -- 172.21.15.44:0/1151921516 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134034610 con 0x7fc13c06bbe0 2024-04-02T13:32:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.945+0000 7fc141a2a640 1 -- 172.21.15.44:0/1151921516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc13c06bbe0 msgr2=0x7fc13c06e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.945+0000 7fc141a2a640 1 --2- 172.21.15.44:0/1151921516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc13c06bbe0 0x7fc13c06e380 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0d1640 tx=0x7fc13402bf80 comp rx=0 tx=0).stop 2024-04-02T13:32:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.945+0000 7fc141a2a640 1 -- 172.21.15.44:0/1151921516 shutdown_connections 2024-04-02T13:32:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.945+0000 7fc141a2a640 1 --2- 172.21.15.44:0/1151921516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc13c06bbe0 0x7fc13c06e380 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.945+0000 7fc141a2a640 1 --2- 172.21.15.44:0/1151921516 >> v2:172.21.15.69:3300/0 conn(0x7fc13c15e520 0x7fc13c162950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.945+0000 7fc141a2a640 1 -- 172.21.15.44:0/1151921516 >> 172.21.15.44:0/1151921516 conn(0x7fc13c0b22c0 msgr2=0x7fc13c0b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.945+0000 7fc141a2a640 1 -- 172.21.15.44:0/1151921516 shutdown_connections 2024-04-02T13:32:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.945+0000 7fc141a2a640 1 -- 172.21.15.44:0/1151921516 wait complete. 2024-04-02T13:32:45.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.946+0000 7fc141a2a640 1 Processor -- start 2024-04-02T13:32:45.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.947+0000 7fc141a2a640 1 -- start start 2024-04-02T13:32:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.947+0000 7fc141a2a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc13c06bbe0 0x7fc13c071f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.947+0000 7fc141a2a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13c0ad890 0x7fc13c0724a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.947+0000 7fc141a2a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc13c15e520 0x7fc13c15c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.947+0000 7fc141a2a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc13c0b4050 con 0x7fc13c06bbe0 2024-04-02T13:32:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.947+0000 7fc141a2a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc13c06def0 con 0x7fc13c0ad890 2024-04-02T13:32:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.947+0000 7fc141a2a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc13c0b4330 con 0x7fc13c15e520 2024-04-02T13:32:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.947+0000 7fc13affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc13c06bbe0 0x7fc13c071f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.947+0000 7fc13affd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc13c06bbe0 0x7fc13c071f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33862/0 (socket says 172.21.15.44:33862) 2024-04-02T13:32:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.947+0000 7fc13affd640 1 -- 172.21.15.44:0/3665624012 learned_addr learned my addr 172.21.15.44:0/3665624012 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.947+0000 7fc13a7fc640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13c0ad890 0x7fc13c0724a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.948+0000 7fc13b7fe640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc13c15e520 0x7fc13c15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.948+0000 7fc13affd640 1 -- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc13c15e520 msgr2=0x7fc13c15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:45.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.948+0000 7fc13affd640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc13c15e520 0x7fc13c15c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.948+0000 7fc13affd640 1 -- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13c0ad890 msgr2=0x7fc13c0724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.948+0000 7fc13affd640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13c0ad890 0x7fc13c0724a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.948+0000 7fc13affd640 1 -- 172.21.15.44:0/3665624012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc13c15d670 con 0x7fc13c06bbe0 2024-04-02T13:32:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.948+0000 7fc13affd640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc13c06bbe0 0x7fc13c071f60 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00e8a0 tx=0x7fc12c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.948+0000 7fc123fff640 1 -- 172.21.15.44:0/3665624012 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c017070 con 0x7fc13c06bbe0 2024-04-02T13:32:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.948+0000 7fc123fff640 1 -- 172.21.15.44:0/3665624012 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc12c00c940 con 0x7fc13c06bbe0 2024-04-02T13:32:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.948+0000 7fc141a2a640 1 -- 172.21.15.44:0/3665624012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc13c15d950 con 0x7fc13c06bbe0 2024-04-02T13:32:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.948+0000 7fc123fff640 1 -- 172.21.15.44:0/3665624012 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c00cd30 con 0x7fc13c06bbe0 2024-04-02T13:32:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.948+0000 7fc141a2a640 1 -- 172.21.15.44:0/3665624012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc13c14e400 con 0x7fc13c06bbe0 2024-04-02T13:32:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.949+0000 7fc141a2a640 1 -- 172.21.15.44:0/3665624012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc13c0b3570 con 0x7fc13c06bbe0 2024-04-02T13:32:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.949+0000 7fc123fff640 1 -- 172.21.15.44:0/3665624012 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc12c01a070 con 0x7fc13c06bbe0 2024-04-02T13:32:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.950+0000 7fc123fff640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc12806c8f0 0x7fc12806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.950+0000 7fc123fff640 1 -- 172.21.15.44:0/3665624012 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7fc12c08bcb0 con 0x7fc13c06bbe0 2024-04-02T13:32:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.950+0000 7fc13a7fc640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc12806c8f0 0x7fc12806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.951+0000 7fc13a7fc640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc12806c8f0 0x7fc12806edb0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc13c136dc0 tx=0x7fc130009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:45.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:45.954+0000 7fc123fff640 1 -- 172.21.15.44:0/3665624012 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7fc13c0b3570 con 0x7fc13c06bbe0 2024-04-02T13:32:46.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.138+0000 7fc141a2a640 1 -- 172.21.15.44:0/3665624012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fc13c0684f0 con 0x7fc13c06bbe0 2024-04-02T13:32:46.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.142+0000 7fc123fff640 1 -- 172.21.15.44:0/3665624012 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v24)=0 v24) ==== 123+0+0 (secure 0 0 0) 0x7fc12c059ac0 con 0x7fc13c06bbe0 2024-04-02T13:32:46.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.143+0000 7fc123fff640 1 -- 172.21.15.44:0/3665624012 <== mon.0 v2:172.21.15.44:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc12c05ddc0 con 0x7fc13c06bbe0 2024-04-02T13:32:46.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.144+0000 7fc141a2a640 1 -- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc12806c8f0 msgr2=0x7fc12806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:46.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.144+0000 7fc141a2a640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc12806c8f0 0x7fc12806edb0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc13c136dc0 tx=0x7fc130009040 comp rx=0 tx=0).stop 2024-04-02T13:32:46.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.144+0000 7fc141a2a640 1 -- 172.21.15.44:0/3665624012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc13c06bbe0 msgr2=0x7fc13c071f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:46.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.144+0000 7fc141a2a640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc13c06bbe0 0x7fc13c071f60 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00e8a0 tx=0x7fc12c00ed60 comp rx=0 tx=0).stop 2024-04-02T13:32:46.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.144+0000 7fc141a2a640 1 -- 172.21.15.44:0/3665624012 shutdown_connections 2024-04-02T13:32:46.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.144+0000 7fc141a2a640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7fc12806c8f0 0x7fc12806edb0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:46.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.144+0000 7fc141a2a640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc13c15e520 0x7fc13c15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:46.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.144+0000 7fc141a2a640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13c0ad890 0x7fc13c0724a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:46.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.144+0000 7fc141a2a640 1 --2- 172.21.15.44:0/3665624012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc13c06bbe0 0x7fc13c071f60 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:46.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.144+0000 7fc141a2a640 1 -- 172.21.15.44:0/3665624012 >> 172.21.15.44:0/3665624012 conn(0x7fc13c0b22c0 msgr2=0x7fc13c06e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:46.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.144+0000 7fc141a2a640 1 -- 172.21.15.44:0/3665624012 shutdown_connections 2024-04-02T13:32:46.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.144+0000 7fc141a2a640 1 -- 172.21.15.44:0/3665624012 wait complete. 2024-04-02T13:32:46.158 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T13:32:46.159 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T13:32:46.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.257+0000 7f67187af640 1 Processor -- start 2024-04-02T13:32:46.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.257+0000 7f67187af640 1 -- start start 2024-04-02T13:32:46.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.257+0000 7f67187af640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f671015e550 0x7f671006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:46.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.257+0000 7f67187af640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f671015fb50 con 0x7f671006d530 2024-04-02T13:32:46.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.257+0000 7f67187af640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f671006ca90 con 0x7f671015f040 2024-04-02T13:32:46.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.257+0000 7f67187af640 1 -- --> v2:172.21.15.44:3300/0 -- mon_getmap magic: 0 -- 0x7f671006d970 con 0x7f671015e550 2024-04-02T13:32:46.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.258+0000 7f671654b640 1 --2- >> v2:172.21.15.44:3300/0 conn(0x7f671015e550 0x7f671006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:46.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.258+0000 7f6715d4a640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f671006d530 0x7f671015e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42392/0 (socket says 172.21.15.44:42392) 2024-04-02T13:32:46.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.258+0000 7f6715d4a640 1 -- 172.21.15.44:0/371777366 learned_addr learned my addr 172.21.15.44:0/371777366 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-04-02T13:32:46.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.258+0000 7f671654b640 1 -- 172.21.15.44:0/371777366 >> v1:172.21.15.69:6789/0 conn(0x7f671015f040 legacy=0x7f67100d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T13:32:46.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.258+0000 7f671654b640 1 -- 172.21.15.44:0/371777366 >> v1:172.21.15.44:6789/0 conn(0x7f671006d530 legacy=0x7f671015e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:46.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.258+0000 7f671654b640 1 -- 172.21.15.44:0/371777366 --> v2:172.21.15.44:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671015f590 con 0x7f671015e550 2024-04-02T13:32:46.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.258+0000 7f6715549640 1 -- 172.21.15.44:0/371777366 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 885978708 0 0) 0x7f671015fb50 con 0x7f671006d530 2024-04-02T13:32:46.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.259+0000 7f671654b640 1 --2- 172.21.15.44:0/371777366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f671015e550 0x7f671006cff0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f670c0097a0 tx=0x7f670c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2483919be1c37b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:46.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.259+0000 7f6715549640 1 -- 172.21.15.44:0/371777366 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c034070 con 0x7f671015e550 2024-04-02T13:32:46.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.259+0000 7f6715549640 1 -- 172.21.15.44:0/371777366 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f670c030610 con 0x7f671015e550 2024-04-02T13:32:46.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.259+0000 7f6715549640 1 -- 172.21.15.44:0/371777366 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c030a00 con 0x7f671015e550 2024-04-02T13:32:46.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.259+0000 7f67187af640 1 -- 172.21.15.44:0/371777366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f671015e550 msgr2=0x7f671006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:46.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.259+0000 7f67187af640 1 --2- 172.21.15.44:0/371777366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f671015e550 0x7f671006cff0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f670c0097a0 tx=0x7f670c02b960 comp rx=0 tx=0).stop 2024-04-02T13:32:46.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.259+0000 7f67187af640 1 -- 172.21.15.44:0/371777366 shutdown_connections 2024-04-02T13:32:46.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.259+0000 7f67187af640 1 --2- 172.21.15.44:0/371777366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f671015e550 0x7f671006cff0 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:46.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.259+0000 7f67187af640 1 -- 172.21.15.44:0/371777366 >> 172.21.15.44:0/371777366 conn(0x7f671009aa20 msgr2=0x7f671009ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:46.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.259+0000 7f67187af640 1 -- 172.21.15.44:0/371777366 shutdown_connections 2024-04-02T13:32:46.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.259+0000 7f67187af640 1 -- 172.21.15.44:0/371777366 wait complete. 2024-04-02T13:32:46.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.261+0000 7f67187af640 1 Processor -- start 2024-04-02T13:32:46.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.261+0000 7f67187af640 1 -- start start 2024-04-02T13:32:46.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.261+0000 7f67187af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f671006d530 0x7f671014f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:46.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.261+0000 7f67187af640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f671015e550 0x7f6710156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:46.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.261+0000 7f67187af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f671015f040 0x7f67101587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.261+0000 7f67187af640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f67100d0590 con 0x7f671015f040 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.261+0000 7f67187af640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f67100d02b0 con 0x7f671015e550 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.261+0000 7f67187af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f67100cc4b0 con 0x7f671006d530 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.261+0000 7f6716d4c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f671015f040 0x7f67101587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.261+0000 7f6716d4c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f671015f040 0x7f67101587d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33868/0 (socket says 172.21.15.44:33868) 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.261+0000 7f6716d4c640 1 -- 172.21.15.44:0/1906146986 learned_addr learned my addr 172.21.15.44:0/1906146986 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.262+0000 7f6715d4a640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f671015e550 0x7f6710156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.262+0000 7f671654b640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f671006d530 0x7f671014f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.262+0000 7f6716d4c640 1 -- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f671006d530 msgr2=0x7f671014f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.262+0000 7f6716d4c640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f671006d530 0x7f671014f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.262+0000 7f6716d4c640 1 -- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f671015e550 msgr2=0x7f6710156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.262+0000 7f6716d4c640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f671015e550 0x7f6710156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.262+0000 7f6716d4c640 1 -- 172.21.15.44:0/1906146986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6710159530 con 0x7f671015f040 2024-04-02T13:32:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.262+0000 7f6716d4c640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f671015f040 0x7f67101587d0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f670800c780 tx=0x7f670800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:46.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.262+0000 7f6706ffd640 1 -- 172.21.15.44:0/1906146986 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708004b60 con 0x7f671015f040 2024-04-02T13:32:46.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.262+0000 7f6706ffd640 1 -- 172.21.15.44:0/1906146986 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6708004d00 con 0x7f671015f040 2024-04-02T13:32:46.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.262+0000 7f67187af640 1 -- 172.21.15.44:0/1906146986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6710159810 con 0x7f671015f040 2024-04-02T13:32:46.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.262+0000 7f6706ffd640 1 -- 172.21.15.44:0/1906146986 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670800aab0 con 0x7f671015f040 2024-04-02T13:32:46.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.262+0000 7f67187af640 1 -- 172.21.15.44:0/1906146986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6710159dd0 con 0x7f671015f040 2024-04-02T13:32:46.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.263+0000 7f67187af640 1 -- 172.21.15.44:0/1906146986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67100684f0 con 0x7f671015f040 2024-04-02T13:32:46.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.264+0000 7f6706ffd640 1 -- 172.21.15.44:0/1906146986 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f670800ac50 con 0x7f671015f040 2024-04-02T13:32:46.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.264+0000 7f6706ffd640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f66f406c8f0 0x7f66f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T13:32:46.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.264+0000 7f6706ffd640 1 -- 172.21.15.44:0/1906146986 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5507+0+0 (secure 0 0 0) 0x7f670808bd80 con 0x7f671015f040 2024-04-02T13:32:46.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.264+0000 7f671654b640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f66f406c8f0 0x7f66f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T13:32:46.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.265+0000 7f671654b640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f66f406c8f0 0x7f66f406edb0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f670c036000 tx=0x7f670c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T13:32:46.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.268+0000 7f6706ffd640 1 -- 172.21.15.44:0/1906146986 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187049 (secure 0 0 0) 0x7f6708020070 con 0x7f671015f040 2024-04-02T13:32:46.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.493+0000 7f67187af640 1 -- 172.21.15.44:0/1906146986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f671009b3d0 con 0x7f671015f040 2024-04-02T13:32:46.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.494+0000 7f6706ffd640 1 -- 172.21.15.44:0/1906146986 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+3866 (secure 0 0 0) 0x7f670805a270 con 0x7f671015f040 2024-04-02T13:32:46.498 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-02T13:32:46.498 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":114,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2558398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2558398501},{"type":"v1","addr":"172.21.15.69:6835","nonce":2558398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8191,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2619487523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2619487523},{"type":"v1","addr":"172.21.15.69:6837","nonce":2619487523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6834/746347682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":746347682},{"type":"v1","addr":"172.21.15.44:6834","nonce":746347682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93},{"gid":10302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/268920747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6833","nonce":268920747},{"type":"v1","addr":"172.21.15.44:6835","nonce":268920747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114}],"filesystems":[]} 2024-04-02T13:32:46.498 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 114 2024-04-02T13:32:46.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.496+0000 7f67187af640 1 -- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f66f406c8f0 msgr2=0x7f66f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:46.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.496+0000 7f67187af640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f66f406c8f0 0x7f66f406edb0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f670c036000 tx=0x7f670c03b040 comp rx=0 tx=0).stop 2024-04-02T13:32:46.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.496+0000 7f67187af640 1 -- 172.21.15.44:0/1906146986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f671015f040 msgr2=0x7f67101587d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T13:32:46.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.496+0000 7f67187af640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f671015f040 0x7f67101587d0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f670800c780 tx=0x7f670800cc40 comp rx=0 tx=0).stop 2024-04-02T13:32:46.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.496+0000 7f67187af640 1 -- 172.21.15.44:0/1906146986 shutdown_connections 2024-04-02T13:32:46.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.496+0000 7f67187af640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:6832/3569935720,v1:172.21.15.69:6833/3569935720] conn(0x7f66f406c8f0 0x7f66f406edb0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:46.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.496+0000 7f67187af640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f671015f040 0x7f67101587d0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:46.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.496+0000 7f67187af640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f671015e550 0x7f6710156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:46.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.496+0000 7f67187af640 1 --2- 172.21.15.44:0/1906146986 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f671006d530 0x7f671014f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T13:32:46.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.496+0000 7f67187af640 1 -- 172.21.15.44:0/1906146986 >> 172.21.15.44:0/1906146986 conn(0x7f671009aa20 msgr2=0x7f67100d1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T13:32:46.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.496+0000 7f67187af640 1 -- 172.21.15.44:0/1906146986 shutdown_connections 2024-04-02T13:32:46.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-02T13:32:46.496+0000 7f67187af640 1 -- 172.21.15.44:0/1906146986 wait complete. 2024-04-02T13:32:46.510 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-02T13:32:46.511 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-02T13:32:46.511 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T13:32:46.511 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:46.519 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T13:32:46.520 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T13:32:46.520 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:46.568 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T13:32:46.569 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T13:32:46.569 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:46.579 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T13:32:46.580 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T13:32:46.580 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:46.631 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T13:32:46.631 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-02T13:32:46.632 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-02T13:32:46.632 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:46.656 INFO:tasks.ceph.osd.0:Stopped 2024-04-02T13:32:46.657 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-02T13:32:46.657 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:46.734 INFO:tasks.ceph.osd.1:Stopped 2024-04-02T13:32:46.734 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-02T13:32:46.735 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:46.803 INFO:tasks.ceph.osd.2:Stopped 2024-04-02T13:32:46.803 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-02T13:32:46.803 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:46.871 INFO:tasks.ceph.osd.3:Stopped 2024-04-02T13:32:46.871 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-02T13:32:46.871 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:46.898 INFO:tasks.ceph.osd.4:Stopped 2024-04-02T13:32:46.898 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-02T13:32:46.898 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:46.970 INFO:tasks.ceph.osd.5:Stopped 2024-04-02T13:32:46.970 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-02T13:32:46.970 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:47.036 INFO:tasks.ceph.osd.6:Stopped 2024-04-02T13:32:47.036 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-02T13:32:47.036 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:47.106 INFO:tasks.ceph.osd.7:Stopped 2024-04-02T13:32:47.106 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-02T13:32:47.106 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-02T13:32:47.107 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:47.132 INFO:tasks.ceph.mgr.y:Stopped 2024-04-02T13:32:47.132 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-02T13:32:47.132 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:47.175 INFO:tasks.ceph.mgr.x:Stopped 2024-04-02T13:32:47.176 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-02T13:32:47.176 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-02T13:32:47.176 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:47.187 INFO:tasks.ceph.mon.a:Stopped 2024-04-02T13:32:47.188 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-02T13:32:47.188 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:47.228 INFO:tasks.ceph.mon.b:Stopped 2024-04-02T13:32:47.228 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-02T13:32:47.228 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T13:32:47.279 INFO:tasks.ceph.mon.c:Stopped 2024-04-02T13:32:47.279 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-02T13:32:47.279 DEBUG:teuthology.orchestra.run.smithi044:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-04-02T13:32:47.315 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:25:17.966487+0000 mds.a (mds.0) 1 : cluster [WRN] Corrupt dentry 'file_to_be_damaged' in dir frag 0x10000000000: Invalid tag char 'd' pos 0: Malformed input [buffer:3](/subdir) 2024-04-02T13:32:47.315 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-04-02T13:32:47.316 DEBUG:teuthology.orchestra.run.smithi044:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-04-02T13:32:47.391 DEBUG:teuthology.orchestra.run.smithi044:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-04-02T13:32:47.465 INFO:teuthology.orchestra.run.smithi044.stdout:2024-04-02T13:27:02.657018+0000 mds.a (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-04-02T13:32:47.466 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:32:47.466 DEBUG:teuthology.orchestra.run.smithi044:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-02T13:32:47.663 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:32:47.664 DEBUG:teuthology.orchestra.run.smithi044:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-02T13:32:47.752 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:32:47.752 DEBUG:teuthology.orchestra.run.smithi044:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-02T13:32:47.802 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:32:47.802 DEBUG:teuthology.orchestra.run.smithi044:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-02T13:32:47.878 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:32:47.878 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-02T13:32:48.081 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:32:48.081 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-02T13:32:48.175 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:32:48.175 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-02T13:32:48.240 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:32:48.241 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-02T13:32:48.300 INFO:tasks.ceph:Archiving mon data... 2024-04-02T13:32:48.302 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584/data/mon.a.tgz 2024-04-02T13:32:48.302 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2024-04-02T13:32:48.317 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.oB2KriEzz8 2024-04-02T13:32:48.318 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.oB2KriEzz8 2024-04-02T13:32:48.482 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.oB2KriEzz8 2024-04-02T13:32:48.566 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.oB2KriEzz8 is 442KB 2024-04-02T13:32:48.664 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.oB2KriEzz8 2024-04-02T13:32:48.678 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584/data/mon.b.tgz 2024-04-02T13:32:48.678 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-04-02T13:32:48.696 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.5RvGTOmhAX 2024-04-02T13:32:48.696 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.5RvGTOmhAX 2024-04-02T13:32:48.864 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.5RvGTOmhAX 2024-04-02T13:32:48.950 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.5RvGTOmhAX is 456KB 2024-04-02T13:32:49.048 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.5RvGTOmhAX 2024-04-02T13:32:49.062 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584/data/mon.c.tgz 2024-04-02T13:32:49.062 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-04-02T13:32:49.116 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.HDVPXpt9ja 2024-04-02T13:32:49.116 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.HDVPXpt9ja 2024-04-02T13:32:49.266 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.HDVPXpt9ja 2024-04-02T13:32:49.351 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.HDVPXpt9ja is 456KB 2024-04-02T13:32:49.450 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.HDVPXpt9ja 2024-04-02T13:32:49.466 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-02T13:32:49.467 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-02T13:32:49.469 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-02T13:32:49.553 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-02T13:32:49.588 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-02T13:32:49.588 INFO:tasks.ceph:Archiving crash dumps... 2024-04-02T13:32:49.590 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584/remote/smithi044/crash 2024-04-02T13:32:49.591 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-02T13:32:49.622 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584/remote/smithi069/crash 2024-04-02T13:32:49.623 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-02T13:32:49.653 INFO:tasks.ceph:Shutting down logrotate 2024-04-02T13:32:49.654 DEBUG:teuthology.orchestra.run.smithi044:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:32:49.667 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T13:32:49.751 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-02T13:32:49.780 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-02T13:32:49.807 INFO:tasks.ceph:Compressing logs... 2024-04-02T13:32:49.807 DEBUG:teuthology.orchestra.run.smithi044:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-02T13:32:49.823 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-02T13:32:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-02T13:32:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37372.log 2024-04-02T13:32:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37532.log 2024-04-02T13:32:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.37372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44213.log 2024-04-02T13:32:49.900 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37372.log.gz 2024-04-02T13:32:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.37532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37412.log 2024-04-02T13:32:49.900 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37532.log.gz 2024-04-02T13:32:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43756.log 2024-04-02T13:32:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.3.44213.log: /var/log/ceph/ceph-client.admin.37412.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-02T13:32:49.900 INFO:teuthology.orchestra.run.smithi069.stderr: 87.2%/var/log/ceph/ceph-client.1.43756.log: -- replaced with /var/log/ceph/ceph-client.admin.37412.log.gz 2024-04-02T13:32:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-02T13:32:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37332.log 2024-04-02T13:32:49.901 INFO:teuthology.orchestra.run.smithi069.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.3.44213.log.gz 2024-04-02T13:32:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37292.log 2024-04-02T13:32:49.901 INFO:teuthology.orchestra.run.smithi069.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.1.43756.log.gz 2024-04-02T13:32:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.37332.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-02T13:32:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.37292.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37332.log.gz 2024-04-02T13:32:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37492.log 2024-04-02T13:32:49.901 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37292.log.gz 2024-04-02T13:32:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37452.log 2024-04-02T13:32:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.37492.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-02T13:32:49.901 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37492.log.gz 2024-04-02T13:32:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.37452.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-02T13:32:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.7.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37452.log.gz 2024-04-02T13:32:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-02T13:32:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37252.log 2024-04-02T13:32:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-02T13:32:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.37252.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37252.log.gz 2024-04-02T13:32:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-02T13:32:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.log: 92.1% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-02T13:32:49.902 INFO:teuthology.orchestra.run.smithi069.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-02T13:32:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-02T13:32:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-02T13:32:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mds.d.log: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-02T13:32:49.970 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mon.c.log: 91.7% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-02T13:32:50.025 INFO:teuthology.orchestra.run.smithi069.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-02T13:32:50.177 INFO:teuthology.orchestra.run.smithi069.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-02T13:32:50.203 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60563.log 2024-04-02T13:32:50.203 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79570.log 2024-04-02T13:32:50.203 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.60563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80050.log 2024-04-02T13:32:50.203 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.79570.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60563.log.gz 2024-04-02T13:32:50.203 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80008.log 2024-04-02T13:32:50.203 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80050.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79570.log.gz 2024-04-02T13:32:50.203 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48581.log 2024-04-02T13:32:50.203 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80008.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80050.log.gz 2024-04-02T13:32:50.203 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50408.log 2024-04-02T13:32:50.204 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80008.log.gz 2024-04-02T13:32:50.204 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.48581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46095.log 2024-04-02T13:32:50.204 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67409.log 2024-04-02T13:32:50.204 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48581.log.gz 2024-04-02T13:32:50.204 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85318.log 2024-04-02T13:32:50.204 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50408.log.gz 2024-04-02T13:32:50.204 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46095.log.gz 2024-04-02T13:32:50.204 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49826.log 2024-04-02T13:32:50.204 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66030.log 2024-04-02T13:32:50.204 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72128.log 2024-04-02T13:32:50.204 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.85318.log: /var/log/ceph/ceph-client.admin.49826.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67409.log.gz 2024-04-02T13:32:50.204 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.66030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70406.log 2024-04-02T13:32:50.204 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66030.log.gz 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49826.log.gz 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72128.log: /var/log/ceph/ceph-client.admin.70406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87723.log 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72128.log.gz 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.85318.log.gz 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-mon.a.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.70406.log.gz 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62184.log 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49379.log 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0%/var/log/ceph/ceph-client.admin.62184.log: -- replaced with /var/log/ceph/ceph-client.admin.87723.log.gz 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63068.log 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49379.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62184.log.gz 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66492.log 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56239.log 2024-04-02T13:32:50.205 INFO:teuthology.orchestra.run.smithi044.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49379.log.gz 2024-04-02T13:32:50.206 INFO:teuthology.orchestra.run.smithi044.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63068.log.gz 2024-04-02T13:32:50.206 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.66492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58477.log 2024-04-02T13:32:50.206 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53110.log 2024-04-02T13:32:50.206 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58477.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56239.log.gz 2024-04-02T13:32:50.206 INFO:teuthology.orchestra.run.smithi044.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.66492.log.gz 2024-04-02T13:32:50.206 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56155.log 2024-04-02T13:32:50.206 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58477.log.gz 2024-04-02T13:32:50.206 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.53110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84631.log 2024-04-02T13:32:50.206 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56155.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53110.log.gzgzip 2024-04-02T13:32:50.206 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46546.log 2024-04-02T13:32:50.206 INFO:teuthology.orchestra.run.smithi044.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56155.log.gz 2024-04-02T13:32:50.206 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.84631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67645.log 2024-04-02T13:32:50.206 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84631.log.gz 2024-04-02T13:32:50.207 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50936.log 2024-04-02T13:32:50.207 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67645.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46546.log.gz 2024-04-02T13:32:50.207 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79527.log 2024-04-02T13:32:50.207 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67645.log.gz 2024-04-02T13:32:50.207 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84505.log 2024-04-02T13:32:50.207 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50936.log.gz 2024-04-02T13:32:50.207 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.79527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66558.log 2024-04-02T13:32:50.207 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.84505.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79527.log.gz 2024-04-02T13:32:50.207 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86463.log 2024-04-02T13:32:50.207 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84505.log.gz 2024-04-02T13:32:50.207 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.66558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68976.log 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.86463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45424.log 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68976.log: 90.7% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66558.log.gz 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86463.log.gz 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46858.log 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68976.log.gz 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54482.log 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46858.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45424.log.gz 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66859.log 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46858.log.gz 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46774.log 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.66859.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54482.log.gz 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42931.log 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59118.log 2024-04-02T13:32:50.208 INFO:teuthology.orchestra.run.smithi044.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66859.log.gz 2024-04-02T13:32:50.209 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46774.log.gz 2024-04-02T13:32:50.209 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51656.log 2024-04-02T13:32:50.209 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59118.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42931.log.gz 2024-04-02T13:32:50.209 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68846.log 2024-04-02T13:32:50.209 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59118.log.gz 2024-04-02T13:32:50.209 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72045.log 2024-04-02T13:32:50.209 INFO:teuthology.orchestra.run.smithi044.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51656.log.gz 2024-04-02T13:32:50.209 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85229.log 2024-04-02T13:32:50.209 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72045.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68846.log.gz 2024-04-02T13:32:50.209 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.48055.log 2024-04-02T13:32:50.209 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.85229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44310.log 2024-04-02T13:32:50.209 INFO:teuthology.orchestra.run.smithi044.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.85229.log.gz 2024-04-02T13:32:50.209 INFO:teuthology.orchestra.run.smithi044.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.72045.log.gz 2024-04-02T13:32:50.210 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.2.48055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77119.log 2024-04-02T13:32:50.210 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50608.log 2024-04-02T13:32:50.210 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44310.log.gz 2024-04-02T13:32:50.210 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.77119.log: gzip -5 --verbose -- 91.1% /var/log/ceph/ceph-client.admin.73787.log 2024-04-02T13:32:50.210 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.2.48055.log.gz 2024-04-02T13:32:50.210 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77119.log.gz 2024-04-02T13:32:50.210 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65804.log 2024-04-02T13:32:50.210 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50608.log: /var/log/ceph/ceph-client.admin.73787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75866.log 2024-04-02T13:32:50.210 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50608.log.gz 2024-04-02T13:32:50.210 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.65804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88206.log 2024-04-02T13:32:50.210 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65804.log.gz 2024-04-02T13:32:50.210 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75866.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.73787.log.gz 2024-04-02T13:32:50.210 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79822.log 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75866.log.gz 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.88206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81248.log 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88206.log.gz 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.79822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51446.log 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.81248.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79822.log.gz 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87265.log 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51446.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.81248.log.gz 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52118.log 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51446.log.gz 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78256.log 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52118.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87265.log.gz 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74229.log 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52118.log.gz 2024-04-02T13:32:50.211 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86826.log 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78256.log.gz 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65152.log 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.86826.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74229.log.gz 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45034.log 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86826.log.gz 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.65152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54686.log 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65152.log.gz 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62539.log 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54686.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45034.log.gz 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75221.log 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62539.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54686.log.gz 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61339.log 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62539.log.gz 2024-04-02T13:32:50.212 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60715.log 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50282.log 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75221.log.gz 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.60715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80517.log 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50282.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.61339.log.gz 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52263.log 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60715.log.gz 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50282.log.gz 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50324.log 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80517.log.gz 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55773.log 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52263.log.gz 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52719.log 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55773.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50324.log.gz 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85616.log 2024-04-02T13:32:50.213 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55773.log.gz 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45762.log 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52719.log.gz 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.85616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46942.log 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45762.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85616.log.gz 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76961.log 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.45762.log.gz 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88062.log 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46942.log.gz 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.76961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70588.log 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.88062.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76961.log.gz 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73032.log 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr: 87.3%/var/log/ceph/ceph-client.admin.70588.log: -- replaced with /var/log/ceph/ceph-client.admin.88062.log.gz 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51908.log 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70588.log.gz 2024-04-02T13:32:50.214 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.73032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63808.log 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54382.log 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63808.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51908.log.gz 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57426.log 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.73032.log.gz 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63808.log.gz 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49784.log 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54382.log.gz 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.57426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83378.log 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.84152.log 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57426.log.gz 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49784.log.gz 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.83378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57576.log 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83378.log.gz 2024-04-02T13:32:50.215 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.0.84152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84463.log 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.57576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48371.log 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.84463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87936.log 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.57576.log.gz 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.48371.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.84463.log.gz 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.69367.log 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87936.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48371.log.gz 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84815.log 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87936.log.gz 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82417.log 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.69367.log: /var/log/ceph/ceph-client.admin.84815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69157.log 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69367.log.gz 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.82417.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84815.log.gz 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57837.log 2024-04-02T13:32:50.216 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46816.log 2024-04-02T13:32:50.217 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.69157.log: /var/log/ceph/ceph-client.admin.57837.log: gzip 93.5% -5 -- replaced with /var/log/ceph/ceph-client.0.84152.log.gz --verbose 2024-04-02T13:32:50.217 INFO:teuthology.orchestra.run.smithi044.stderr: -- /var/log/ceph/ceph-client.admin.81206.log 2024-04-02T13:32:50.217 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% 91.3% -- replaced with /var/log/ceph/ceph-client.admin.82417.log.gz -- replaced with /var/log/ceph/ceph-client.admin.69157.log.gz 2024-04-02T13:32:50.217 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.217 INFO:teuthology.orchestra.run.smithi044.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.57837.log.gz 2024-04-02T13:32:50.217 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84773.log 2024-04-02T13:32:50.217 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.81206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73515.log 2024-04-02T13:32:50.217 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46816.log.gz 2024-04-02T13:32:50.217 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.84773.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.81206.log.gz 2024-04-02T13:32:50.217 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60781.log 2024-04-02T13:32:50.217 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84773.log.gz 2024-04-02T13:32:50.217 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.73515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74439.log 2024-04-02T13:32:50.217 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.60781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74118.log 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74439.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.73515.log.gz 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49169.log 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60781.log.gz 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74439.log.gz 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52076.log 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74118.log: /var/log/ceph/ceph-client.admin.49169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54828.log 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.74118.log.gz 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52076.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49169.log.gzgzip 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.82153.log 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52076.log.gz 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83068.log 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.82153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57664.log 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.54828.log.gz 2024-04-02T13:32:50.218 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.83068.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82153.log.gz 2024-04-02T13:32:50.219 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69241.log 2024-04-02T13:32:50.219 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83068.log.gz 2024-04-02T13:32:50.219 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.57664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43135.log 2024-04-02T13:32:50.219 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.69241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63406.log 2024-04-02T13:32:50.219 INFO:teuthology.orchestra.run.smithi044.stderr: 89.4%/var/log/ceph/ceph-client.admin.43135.log: -- replaced with /var/log/ceph/ceph-client.admin.69241.log.gz 2024-04-02T13:32:50.219 INFO:teuthology.orchestra.run.smithi044.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.57664.log.gz 2024-04-02T13:32:50.219 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72522.log 2024-04-02T13:32:50.219 INFO:teuthology.orchestra.run.smithi044.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43135.log.gz 2024-04-02T13:32:50.219 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75386.log 2024-04-02T13:32:50.219 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63406.log.gz 2024-04-02T13:32:50.219 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84996.log 2024-04-02T13:32:50.219 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75386.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72522.log.gz 2024-04-02T13:32:50.219 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56640.log 2024-04-02T13:32:50.220 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4%/var/log/ceph/ceph-client.admin.84996.log: -- replaced with /var/log/ceph/ceph-client.admin.75386.log.gz 2024-04-02T13:32:50.220 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50836.log 2024-04-02T13:32:50.220 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45413.log 2024-04-02T13:32:50.220 INFO:teuthology.orchestra.run.smithi044.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.84996.log.gz 2024-04-02T13:32:50.220 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50836.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56640.log.gz 2024-04-02T13:32:50.220 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75302.log 2024-04-02T13:32:50.220 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1%/var/log/ceph/ceph-client.admin.45413.log: -- replaced with /var/log/ceph/ceph-client.admin.50836.log.gz 2024-04-02T13:32:50.220 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68168.log 2024-04-02T13:32:50.220 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45413.log.gz 2024-04-02T13:32:50.220 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68932.log 2024-04-02T13:32:50.220 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68168.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75302.log.gz 2024-04-02T13:32:50.220 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52845.log 2024-04-02T13:32:50.220 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68168.log.gz 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72787.log 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68932.log.gz 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63364.log 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52845.log.gz 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58030.log 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66432.log 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58030.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63364.log.gz 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76100.log 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58030.log.gz 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.66432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50492.log 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.66432.log.gz 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.76100.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.72787.log.gz 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62455.log 2024-04-02T13:32:50.221 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76100.log.gz 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56430.log 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50492.log: /var/log/ceph/ceph-client.admin.62455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56850.log 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50492.log.gz 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62455.log.gz 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87631.log 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56850.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.56430.log.gz -5 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58114.log 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87631.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56850.log.gz 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42794.log 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58114.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87631.log.gz 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62142.log 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58114.log.gz 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88291.log 2024-04-02T13:32:50.222 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62142.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42794.log.gz 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64436.log 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62142.log.gz 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.88291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73745.log 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.88291.log.gz 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63448.log 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.73745.log: gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.51824.log 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64436.log.gz 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63448.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73745.log.gz 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67897.log 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63448.log.gz 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52305.log 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67897.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51824.log.gz 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85107.log 2024-04-02T13:32:50.223 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67897.log.gz 2024-04-02T13:32:50.224 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75559.log 2024-04-02T13:32:50.224 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52305.log.gz 2024-04-02T13:32:50.224 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.85107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49910.log 2024-04-02T13:32:50.224 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75559.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.85107.log.gz 2024-04-02T13:32:50.224 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67939.log 2024-04-02T13:32:50.224 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49910.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75559.log.gz 2024-04-02T13:32:50.224 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76567.log 2024-04-02T13:32:50.224 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49910.log.gz 2024-04-02T13:32:50.224 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77198.log 2024-04-02T13:32:50.224 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67939.log.gz 2024-04-02T13:32:50.224 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.76567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76919.log 2024-04-02T13:32:50.224 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.77198.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76567.log.gz 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49127.log 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77198.log.gz 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.76919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48177.log 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76919.log.gz 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46098.log 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.48177.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49127.log.gz 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37700.log 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48177.log.gz 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48455.log 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.37700.log: gzip 89.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.46098.log.gz --verbose 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr: -- /var/log/ceph/ceph-client.admin.59819.log 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.48455.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37700.log.gz 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74565.log 2024-04-02T13:32:50.225 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48455.log.gz 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55292.log 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59819.log.gz 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50240.log 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55292.log: 89.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.74565.log.gz 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61297.log 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50240.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55292.log.gz 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48833.log 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50240.log.gz 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54865.log 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.48833.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61297.log.gz 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72438.log 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54865.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48833.log.gz 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87181.log 2024-04-02T13:32:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72438.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54865.log.gz 2024-04-02T13:32:50.227 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62100.log 2024-04-02T13:32:50.227 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87181.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72438.log.gzgzip 2024-04-02T13:32:50.227 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52427.log 2024-04-02T13:32:50.227 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62100.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87181.log.gz 2024-04-02T13:32:50.227 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50894.log 2024-04-02T13:32:50.227 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52427.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62100.log.gz 2024-04-02T13:32:50.227 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86910.log 2024-04-02T13:32:50.227 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50894.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52427.log.gzgzip 2024-04-02T13:32:50.227 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.67855.log 2024-04-02T13:32:50.227 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50894.log.gz 2024-04-02T13:32:50.227 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.86910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85704.log 2024-04-02T13:32:50.227 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86910.log.gz 2024-04-02T13:32:50.227 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45160.log 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.85704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63490.log 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67855.log.gz 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75004.log 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.85704.log.gz 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63490.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45160.log.gz 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60095.log 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5%/var/log/ceph/ceph-client.admin.75004.log: -- replaced with /var/log/ceph/ceph-client.admin.63490.log.gz 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55562.log 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75004.log.gz/var/log/ceph/ceph-client.admin.60095.log: 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79654.log 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60095.log.gz 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74076.log 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.79654.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.55562.log.gz 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.75471.log 2024-04-02T13:32:50.228 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79654.log.gz 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49295.log 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74076.log.gz 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85033.log 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59861.log 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75471.log.gz 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.85033.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49295.log.gz 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64824.log 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr: 89.7%/var/log/ceph/ceph-client.admin.59861.log: -- replaced with /var/log/ceph/ceph-client.admin.85033.log.gz 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87809.log 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56556.log 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59861.log.gz 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64824.log.gz 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46588.log 2024-04-02T13:32:50.229 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87809.log.gz 2024-04-02T13:32:50.230 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45426.log 2024-04-02T13:32:50.230 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46588.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56556.log.gz 2024-04-02T13:32:50.230 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59510.log 2024-04-02T13:32:50.230 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46588.log.gz 2024-04-02T13:32:50.230 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63112.log 2024-04-02T13:32:50.230 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45426.log.gz 2024-04-02T13:32:50.230 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46900.log 2024-04-02T13:32:50.230 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58393.log 2024-04-02T13:32:50.230 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59510.log.gz 2024-04-02T13:32:50.230 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63112.log.gz 2024-04-02T13:32:50.230 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76609.log 2024-04-02T13:32:50.230 INFO:teuthology.orchestra.run.smithi044.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46900.log.gz 2024-04-02T13:32:50.230 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57577.log 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.76609.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58393.log.gz 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69777.log 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76609.log.gz 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.57577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65636.log 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.57577.log.gz 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.0.69777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50366.log 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.65636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61763.log 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65636.log.gz 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83210.log 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50366.log.gz 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82984.log 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61763.log: /var/log/ceph/ceph-client.admin.83210.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61763.log.gz 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43515.log 2024-04-02T13:32:50.231 INFO:teuthology.orchestra.run.smithi044.stderr:gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.83210.log.gz --verbose 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr: -- /var/log/ceph/ceph-client.admin.85402.log 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.82984.log: /var/log/ceph/ceph-client.admin.43515.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82984.log.gz 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72480.log 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43515.log.gz 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83026.log 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.85402.log: /var/log/ceph/ceph-client.admin.72480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78931.log 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.0.69777.log.gz 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.83026.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72480.log.gz 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62922.log 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61147.log 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83026.log.gz 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78931.log: gzip/var/log/ceph/ceph-client.admin.62922.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.73950.log 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78931.log.gz 2024-04-02T13:32:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45419.log 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62922.log.gz 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.73950.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.85402.log.gz 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69409.log 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73950.log.gz 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45419.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.61147.log.gz 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44394.log 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64704.log 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.69409.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45419.log.gz 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53558.log 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69409.log.gz 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44394.log.gz 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80302.log 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.53558.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64704.log.gz 2024-04-02T13:32:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81969.log 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9%/var/log/ceph/ceph-client.admin.80302.log: -- replaced with /var/log/ceph/ceph-client.admin.53558.log.gz 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59364.log 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80302.log.gz 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.81969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53492.log 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59364.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81969.log.gz 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59605.log 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59364.log.gz 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.53492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67367.log 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59605.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31405.log 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.53492.log.gz 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59605.log.gz 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85662.log 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72212.log 2024-04-02T13:32:50.234 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67367.log.gz 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph.tmp-client.admin.31405.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31405.log.gz 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.85662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67274.log 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85662.log.gz 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63322.log 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67274.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72212.log.gz 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58267.log 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63322.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67274.log.gz 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85252.log 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63322.log.gz 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70666.log 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.85252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81164.log 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58267.log.gz 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63700.log 2024-04-02T13:32:50.235 INFO:teuthology.orchestra.run.smithi044.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.85252.log.gz 2024-04-02T13:32:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.81164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78733.log 2024-04-02T13:32:50.236 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81164.log.gz 2024-04-02T13:32:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51062.log 2024-04-02T13:32:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63700.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.70666.log.gz 2024-04-02T13:32:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86421.log 2024-04-02T13:32:50.236 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63700.log.gz 2024-04-02T13:32:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51062.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.78733.log.gz 2024-04-02T13:32:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66174.log 2024-04-02T13:32:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.86421.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51062.log.gz 2024-04-02T13:32:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62413.log 2024-04-02T13:32:50.236 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86421.log.gz 2024-04-02T13:32:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.66174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58645.log 2024-04-02T13:32:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84953.log 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58645.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.66174.log.gz 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62413.log.gz 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83336.log 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58645.log.gz 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.84953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44268.log 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84953.log.gz 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.83336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74752.log 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44268.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83336.log.gz 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59466.log 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44268.log.gz 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74752.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74752.log.gz 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76693.log 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49421.log 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.76693.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59466.log.gz 2024-04-02T13:32:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78178.log 2024-04-02T13:32:50.238 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49421.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76693.log.gz 2024-04-02T13:32:50.238 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48665.log 2024-04-02T13:32:50.238 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0%/var/log/ceph/ceph-client.admin.78178.log: -- replaced with /var/log/ceph/ceph-client.admin.49421.log.gz 2024-04-02T13:32:50.238 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62016.log 2024-04-02T13:32:50.238 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.48665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58981.log 2024-04-02T13:32:50.238 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.78178.log.gz 2024-04-02T13:32:50.238 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62016.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48665.log.gz 2024-04-02T13:32:50.238 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78136.log 2024-04-02T13:32:50.238 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62016.log.gz 2024-04-02T13:32:50.238 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74034.log 2024-04-02T13:32:50.238 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62497.log 2024-04-02T13:32:50.238 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58981.log.gz 2024-04-02T13:32:50.238 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74034.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.78136.log.gz 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54566.log 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74034.log.gz 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44578.log 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59424.log 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62497.log.gz 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44578.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54566.log.gz 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52635.log 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44578.log.gz 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54440.log 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59424.log: /var/log/ceph/ceph-client.admin.52635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75046.log 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59424.log.gz 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46984.log 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52635.log.gz 2024-04-02T13:32:50.239 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54440.log.gz 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62244.log 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46984.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.75046.log.gz 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70526.log 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62244.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46984.log.gz 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52803.log 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62244.log.gz 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58897.log 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70526.log.gz 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81356.log 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57946.log 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52803.log.gz 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.81356.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58897.log.gz 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43424.log 2024-04-02T13:32:50.240 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81356.log.gz 2024-04-02T13:32:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.57946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47676.log 2024-04-02T13:32:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.43424.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57946.log.gz 2024-04-02T13:32:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86379.log 2024-04-02T13:32:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.47676.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43424.log.gz 2024-04-02T13:32:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72902.log 2024-04-02T13:32:50.241 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47676.log.gz 2024-04-02T13:32:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.86379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87546.log 2024-04-02T13:32:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72902.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86379.log.gz 2024-04-02T13:32:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-02T13:32:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87546.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72902.log.gz 2024-04-02T13:32:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51488.log 2024-04-02T13:32:50.241 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87546.log.gz 2024-04-02T13:32:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78052.log 2024-04-02T13:32:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80685.log 2024-04-02T13:32:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78052.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51488.log.gz 2024-04-02T13:32:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54748.log 2024-04-02T13:32:50.242 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78052.log.gz 2024-04-02T13:32:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70875.log 2024-04-02T13:32:50.242 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80685.log.gz 2024-04-02T13:32:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79296.log 2024-04-02T13:32:50.242 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54748.log.gz 2024-04-02T13:32:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49085.log 2024-04-02T13:32:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70875.log: /var/log/ceph/ceph-client.admin.79296.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70875.log.gz 2024-04-02T13:32:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48539.log 2024-04-02T13:32:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75782.log 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49085.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79296.log.gz 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.48539.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49085.log.gz 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70162.log 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46504.log 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48539.log.gz 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75782.log: 89.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70162.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75782.log.gz 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79058.log 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54791.log 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70162.log.gz 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46504.log: /var/log/ceph/ceph-client.admin.79058.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46504.log.gz 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79058.log.gz 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56472.log 2024-04-02T13:32:50.243 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54902.log 2024-04-02T13:32:50.244 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68462.log 2024-04-02T13:32:50.244 INFO:teuthology.orchestra.run.smithi044.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54791.log.gz 2024-04-02T13:32:50.244 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56472.log.gz 2024-04-02T13:32:50.244 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55124.log 2024-04-02T13:32:50.244 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72745.log 2024-04-02T13:32:50.244 INFO:teuthology.orchestra.run.smithi044.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.54902.log.gz 2024-04-02T13:32:50.244 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83696.log 2024-04-02T13:32:50.244 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68462.log.gz 2024-04-02T13:32:50.244 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5%/var/log/ceph/ceph-client.admin.72745.log: -- replaced with /var/log/ceph/ceph-client.admin.55124.log.gz 2024-04-02T13:32:50.244 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51311.log 2024-04-02T13:32:50.244 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.83696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86547.log 2024-04-02T13:32:50.244 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51311.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83696.log.gz 2024-04-02T13:32:50.244 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65762.log 2024-04-02T13:32:50.245 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51311.log.gz 2024-04-02T13:32:50.245 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.86547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78691.log 2024-04-02T13:32:50.245 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86547.log.gz 2024-04-02T13:32:50.245 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44352.log 2024-04-02T13:32:50.245 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.65762.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.72745.log.gz 2024-04-02T13:32:50.245 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69325.log 2024-04-02T13:32:50.245 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65762.log.gz 2024-04-02T13:32:50.245 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61610.log 2024-04-02T13:32:50.245 INFO:teuthology.orchestra.run.smithi044.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.78691.log.gz 2024-04-02T13:32:50.245 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55082.log 2024-04-02T13:32:50.245 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0%/var/log/ceph/ceph-client.admin.69325.log: -- replaced with /var/log/ceph/ceph-client.admin.44352.log.gz 2024-04-02T13:32:50.245 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68558.log 2024-04-02T13:32:50.245 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69325.log.gz 2024-04-02T13:32:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55082.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61610.log.gz 2024-04-02T13:32:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43294.log 2024-04-02T13:32:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68558.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55082.log.gz 2024-04-02T13:32:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56074.log 2024-04-02T13:32:50.246 INFO:teuthology.orchestra.run.smithi044.stderr: 89.4%/var/log/ceph/ceph-client.admin.43294.log: -- replaced with /var/log/ceph/ceph-client.admin.68558.log.gz 2024-04-02T13:32:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48623.log 2024-04-02T13:32:50.246 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43294.log.gz 2024-04-02T13:32:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55166.log 2024-04-02T13:32:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.48623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80954.log 2024-04-02T13:32:50.246 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56074.log.gz 2024-04-02T13:32:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55166.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48623.log.gz 2024-04-02T13:32:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80866.log 2024-04-02T13:32:50.246 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55166.log.gz 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58519.log 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85070.log 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80954.log.gz 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58519.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80866.log.gz 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64956.log 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58519.log.gz 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.85070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49600.log 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72848.log 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.85070.log.gz 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49600.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64956.log.gz 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68888.log 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49600.log.gz 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52677.log 2024-04-02T13:32:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68888.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72848.log.gz 2024-04-02T13:32:50.248 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58435.log 2024-04-02T13:32:50.248 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68888.log.gz 2024-04-02T13:32:50.248 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85360.log 2024-04-02T13:32:50.248 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52677.log.gz 2024-04-02T13:32:50.248 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80092.log 2024-04-02T13:32:50.248 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.85360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51182.log 89.4% 2024-04-02T13:32:50.248 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58435.log.gz 2024-04-02T13:32:50.248 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81038.log 2024-04-02T13:32:50.248 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51182.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80092.log.gz 2024-04-02T13:32:50.248 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83738.log 2024-04-02T13:32:50.248 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51182.log.gz 2024-04-02T13:32:50.248 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55647.log 2024-04-02T13:32:50.248 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.81038.log: /var/log/ceph/ceph-client.admin.83738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75429.log 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.85360.log.gz 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81038.log.gz 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55647.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83738.log.gz 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55502.log 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75429.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55647.log.gz 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46714.log 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75429.log.gz 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86742.log 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46714.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55502.log.gz 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49253.log 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46714.log.gz 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.86742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50114.log 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.86742.log.gz 2024-04-02T13:32:50.249 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43600.log 2024-04-02T13:32:50.250 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71418.log 2024-04-02T13:32:50.250 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49253.log.gz 2024-04-02T13:32:50.250 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.43600.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50114.log.gz 2024-04-02T13:32:50.250 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72170.log 2024-04-02T13:32:50.250 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.71418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.81616.log 2024-04-02T13:32:50.250 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72170.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.43600.log.gz 2024-04-02T13:32:50.250 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48959.log 2024-04-02T13:32:50.250 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.0.81616.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72170.log.gz 2024-04-02T13:32:50.250 INFO:teuthology.orchestra.run.smithi044.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.71418.log.gz 2024-04-02T13:32:50.250 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71742.log 2024-04-02T13:32:50.250 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.48959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72354.log 2024-04-02T13:32:50.250 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48959.log.gz 2024-04-02T13:32:50.250 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.71742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45760.log 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71742.log.gz 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75515.log 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72354.log: /var/log/ceph/ceph-client.admin.45760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59777.log 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72354.log.gz 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75515.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45760.log.gz 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56324.log 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75515.log.gz 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44662.log 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59777.log.gz 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43473.log 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56324.log: /var/log/ceph/ceph-client.admin.44662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56892.log 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56324.log.gz 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.43473.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44662.log.gz 2024-04-02T13:32:50.251 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70448.log 2024-04-02T13:32:50.252 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71850.log 2024-04-02T13:32:50.252 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56892.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43473.log.gz 2024-04-02T13:32:50.252 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80912.log 2024-04-02T13:32:50.252 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70448.log.gz 2024-04-02T13:32:50.252 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.71850.log: 91.1%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.56892.log.gz --verbose 2024-04-02T13:32:50.252 INFO:teuthology.orchestra.run.smithi044.stderr: -- /var/log/ceph/ceph-client.admin.82095.log 2024-04-02T13:32:50.252 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44908.log 2024-04-02T13:32:50.252 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.82095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67981.log 2024-04-02T13:32:50.252 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80912.log.gz 2024-04-02T13:32:50.252 INFO:teuthology.orchestra.run.smithi044.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.71850.log.gz 2024-04-02T13:32:50.252 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82095.log.gz 2024-04-02T13:32:50.252 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85519.log 2024-04-02T13:32:50.252 INFO:teuthology.orchestra.run.smithi044.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44908.log.gz 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55899.log 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67981.log: /var/log/ceph/ceph-client.admin.85519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83784.log 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67981.log.gz 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55899.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85519.log.gz 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.0.81616.log.gz 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62756.log 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55899.log.gz 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.83784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57814.log 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83784.log.gz 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65988.log 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.57814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53450.log 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62756.log.gz 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.65988.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57814.log.gz 2024-04-02T13:32:50.253 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82743.log 2024-04-02T13:32:50.254 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65988.log.gz 2024-04-02T13:32:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.53450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65678.log 2024-04-02T13:32:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.0.82743.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.53450.log.gz -5 2024-04-02T13:32:50.254 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.83110.log 2024-04-02T13:32:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.65678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50794.log 2024-04-02T13:32:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.83110.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65678.log.gz 2024-04-02T13:32:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65904.log 2024-04-02T13:32:50.254 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83110.log.gz 2024-04-02T13:32:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56808.log 2024-04-02T13:32:50.254 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50794.log.gz 2024-04-02T13:32:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46094.log 2024-04-02T13:32:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.65904.log: /var/log/ceph/ceph-client.admin.56808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88020.log 2024-04-02T13:32:50.254 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65904.log.gz 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46094.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56808.log.gz 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58561.log 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.88020.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46094.log.gz 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45202.log 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58561.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.88020.log.gz 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74313.log 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58561.log.gz 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79738.log 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45202.log.gz 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76877.log 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74313.log: /var/log/ceph/ceph-client.admin.79738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49952.log 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74313.log.gz 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.76877.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79738.log.gz 2024-04-02T13:32:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66817.log 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76877.log.gz 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87307.log 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72396.log 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.66817.log: 94.4% -- replaced with /var/log/ceph/ceph-client.0.82743.log.gz 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49952.log.gz 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87307.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66817.log.gz 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58687.log 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87307.log.gz 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76034.log 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56282.log 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.72396.log.gz 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6%/var/log/ceph/ceph-client.admin.76034.log: -- replaced with /var/log/ceph/ceph-client.admin.58687.log.gz 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77826.log 2024-04-02T13:32:50.256 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52592.log 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.77826.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56282.log.gz 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.76034.log.gz 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67729.log 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77826.log.gz 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68294.log 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81290.log 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67729.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52592.log.gz 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73853.log 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67729.log.gz 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.81290.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68294.log.gz 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81927.log 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.73853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80727.log 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73853.log.gz 2024-04-02T13:32:50.257 INFO:teuthology.orchestra.run.smithi044.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.81290.log.gz 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.81927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73581.log 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80601.log 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81927.log.gz 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80727.log: gzip/var/log/ceph/ceph-client.admin.73581.log: -5 --verbose -- /var/log/ceph/ceph-client.0.47546.log 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80727.log.gz 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80601.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.0.47546.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80601.log.gz 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.73581.log.gz 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45422.log 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61484.log 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.45422.log: 90.4% -- replaced with /var/log/ceph/ceph-client.0.47546.log.gz 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49684.log 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61484.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45422.log.gz 2024-04-02T13:32:50.258 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74649.log 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49684.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61484.log.gz -5 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.64782.log 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74649.log: gzip -5 --verbose 88.2% -- -- replaced with /var/log/ceph/ceph-client.admin.49684.log.gz /var/log/ceph/ceph-client.admin.77784.log 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74649.log.gz 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45118.log 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.77784.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64782.log.gz 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78843.log 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45118.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.77784.log.gz -5 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58072.log 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78843.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45118.log.gz 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46630.log 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78843.log.gz 2024-04-02T13:32:50.259 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54608.log 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68420.log 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58072.log.gz 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54608.log: gzip -5 --verbose -- 89.4% /var/log/ceph/ceph-client.admin.88459.log 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46630.log.gz 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58351.log 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54608.log.gz 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.88459.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68420.log.gz 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73679.log 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88459.log.gz 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65720.log 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.73679.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58351.log.gz 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60659.log 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.65720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70120.log 2024-04-02T13:32:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.60659.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65720.log.gz 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.73679.log.gz 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59689.log 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70120.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.60659.log.gz 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52034.log 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.70120.log.gz 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70917.log 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52034.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59689.log.gz 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59647.log 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70917.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52034.log.gzgzip 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63616.log 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62676.log 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59647.log.gz 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63616.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.70917.log.gz 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88377.log 2024-04-02T13:32:50.261 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63616.log.gz 2024-04-02T13:32:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52550.log 2024-04-02T13:32:50.262 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62676.log.gz 2024-04-02T13:32:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.88377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78094.log 2024-04-02T13:32:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52550.log: gzip -5 --verbose -- 87.9% /var/log/ceph/ceph-client.admin.43379.log 2024-04-02T13:32:50.262 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88377.log.gz 2024-04-02T13:32:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78094.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52550.log.gz 2024-04-02T13:32:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73098.log 2024-04-02T13:32:50.262 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78094.log.gz 2024-04-02T13:32:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.43379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55460.log 2024-04-02T13:32:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.73098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79906.log 2024-04-02T13:32:50.262 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43379.log.gz 2024-04-02T13:32:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55460.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73098.log.gz 2024-04-02T13:32:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44824.log 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55460.log.gz 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.79906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63238.log 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74836.log 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79906.log.gz 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63238.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44824.log.gz 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59945.log 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63238.log.gz 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66258.log 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64998.log 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74836.log.gz 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.66258.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59945.log.gz 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45425.log 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71937.log 2024-04-02T13:32:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45425.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.66258.log.gz 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64998.log.gz 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46294.log 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.71937.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45425.log.gz 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63006.log 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51224.log 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63006.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.71937.log.gz 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46294.log.gz 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53366.log 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63006.log.gz 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57988.log 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51224.log.gz 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.53366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51950.log 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.57988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78369.log 2024-04-02T13:32:50.264 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53366.log.gz 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51950.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57988.log.gz 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74481.log 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51950.log.gz 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74397.log 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79948.log 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74397.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74481.log.gz 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.78369.log.gz 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61231.log 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50450.log 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.79948.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74397.log.gz 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79780.log 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79948.log.gz 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62880.log 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.79780.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50450.log.gz 2024-04-02T13:32:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- 91.0% /var/log/ceph/ceph-client.admin.43558.log 2024-04-02T13:32:50.266 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61231.log.gz 2024-04-02T13:32:50.266 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79780.log.gz 2024-04-02T13:32:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44478.log 2024-04-02T13:32:50.266 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62880.log.gz 2024-04-02T13:32:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.43558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59023.log 2024-04-02T13:32:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44620.log 2024-04-02T13:32:50.266 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43558.log.gz 2024-04-02T13:32:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59023.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44478.log.gz 2024-04-02T13:32:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88249.log 2024-04-02T13:32:50.266 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59023.log.gz 2024-04-02T13:32:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46462.log 2024-04-02T13:32:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.88249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65086.log 2024-04-02T13:32:50.266 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44620.log.gz 2024-04-02T13:32:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46462.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88249.log.gz 2024-04-02T13:32:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81122.log 2024-04-02T13:32:50.267 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9%/var/log/ceph/ceph-client.admin.65086.log: -- replaced with /var/log/ceph/ceph-client.admin.46462.log.gz 2024-04-02T13:32:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70036.log 2024-04-02T13:32:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87765.log 2024-04-02T13:32:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.81122.log: /var/log/ceph/ceph-client.admin.70036.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.65086.log.gz 2024-04-02T13:32:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60029.log 2024-04-02T13:32:50.270 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81122.log.gz 2024-04-02T13:32:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:gzip 88.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.70036.log.gz -- 2024-04-02T13:32:50.270 INFO:teuthology.orchestra.run.smithi044.stderr: /var/log/ceph/ceph-client.admin.58939.log 2024-04-02T13:32:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44746.log 2024-04-02T13:32:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.60029.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.87765.log.gz 2024-04-02T13:32:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68803.log 2024-04-02T13:32:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44746.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58939.log.gz 2024-04-02T13:32:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68252.log 2024-04-02T13:32:50.270 INFO:teuthology.orchestra.run.smithi044.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.60029.log.gz 2024-04-02T13:32:50.270 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44746.log.gz 2024-04-02T13:32:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.68803.log: /var/log/ceph/ceph-client.admin.51866.log 2024-04-02T13:32:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71784.log 2024-04-02T13:32:50.270 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68252.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68803.log.gz 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51866.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68252.log.gz 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64620.log 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51866.log.gz 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.71784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58813.log 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82195.log 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64620.log.gz 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46096.log 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.71784.log.gz 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52929.log 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.82195.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58813.log.gz 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82279.log 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82195.log.gz 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52929.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46096.log.gz 2024-04-02T13:32:50.271 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50072.log 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.82279.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52929.log.gz 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77057.log 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82279.log.gz 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54939.log 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.77057.log: gzip 87.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.50072.log.gz -- 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr: /var/log/ceph/ceph-client.admin.43186.log 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54939.log: gzip 89.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.77057.log.gz --verbose 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr: -- /var/log/ceph/ceph-client.admin.61652.log 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.43186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56598.log 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.54939.log.gz 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61652.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43186.log.gz 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80260.log 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61652.log.gz 2024-04-02T13:32:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74709.log 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54340.log 88.2% 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56598.log.gz 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74709.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80260.log.gz 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83642.log 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74709.log.gz 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47154.log 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7%/var/log/ceph/ceph-client.admin.83642.log: -- replaced with /var/log/ceph/ceph-client.admin.54340.log.gz 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42997.log 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.47154.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83642.log.gzgzip 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.72254.log 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47154.log.gz 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53068.log 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72254.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42997.log.gz 2024-04-02T13:32:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51698.log 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.53068.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72254.log.gz 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63280.log 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53068.log.gz 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51572.log 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58309.log 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51698.log.gz 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51572.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63280.log.gz 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78010.log 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51572.log.gz 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68378.log 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58309.log.gz 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70733.log 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43047.log 2024-04-02T13:32:50.274 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78010.log.gz 2024-04-02T13:32:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70733.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.68378.log.gz 2024-04-02T13:32:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74878.log 2024-04-02T13:32:50.275 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70733.log.gz 2024-04-02T13:32:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.43047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51403.log 2024-04-02T13:32:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84731.log 2024-04-02T13:32:50.275 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43047.log.gz 2024-04-02T13:32:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51403.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74878.log.gz 2024-04-02T13:32:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55689.log 2024-04-02T13:32:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.84731.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51403.log.gz 2024-04-02T13:32:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43091.log 2024-04-02T13:32:50.275 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84731.log.gz 2024-04-02T13:32:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70787.log 2024-04-02T13:32:50.275 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.43091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49868.log 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55689.log.gz 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43091.log.gz 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61932.log 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70787.log.gz 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68210.log 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61932.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49868.log.gz 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45958.log 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68210.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61932.log.gz 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50978.log 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68210.log.gz 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45958.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50978.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.45958.log.gz -5 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45076.log 2024-04-02T13:32:50.276 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50978.log.gz 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56368.log 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77868.log 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56368.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45076.log.gz 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53324.log 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48791.log 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56368.log.gz/var/log/ceph/ceph-client.admin.77868.log: 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.53324.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77868.log.gz 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75908.log 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53324.log.gz 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67687.log 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.48791.log: /var/log/ceph/ceph-client.admin.75908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52508.log 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48791.log.gz 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67687.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75908.log.gz 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49001.log 2024-04-02T13:32:50.277 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67687.log.gz 2024-04-02T13:32:50.278 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55857.log 2024-04-02T13:32:50.278 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42839.log 2024-04-02T13:32:50.278 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52508.log.gz 2024-04-02T13:32:50.278 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49001.log: /var/log/ceph/ceph-client.admin.55857.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49001.log.gz 2024-04-02T13:32:50.278 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85163.log 2024-04-02T13:32:50.278 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.55857.log.gz -5 2024-04-02T13:32:50.278 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.78631.log 2024-04-02T13:32:50.278 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42839.log: /var/log/ceph/ceph-client.admin.85163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54256.log 2024-04-02T13:32:50.278 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42839.log.gz 2024-04-02T13:32:50.278 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86868.log 2024-04-02T13:32:50.278 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46097.log 2024-04-02T13:32:50.278 INFO:teuthology.orchestra.run.smithi044.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.85163.log.gz 2024-04-02T13:32:50.278 INFO:teuthology.orchestra.run.smithi044.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.78631.log.gz 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.86868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74355.log 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54256.log.gz 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86868.log.gz 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55376.log 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46097.log: /var/log/ceph/ceph-client.admin.74355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55815.log 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46097.log.gz 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46092.log 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74355.log.gz 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55376.log.gz 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64352.log 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55815.log.gz 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51020.log 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46092.log: /var/log/ceph/ceph-client.admin.64352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74523.log 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46092.log.gz 2024-04-02T13:32:50.279 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51020.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64352.log.gz 2024-04-02T13:32:50.280 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66709.log 2024-04-02T13:32:50.280 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77952.log 2024-04-02T13:32:50.280 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51020.log.gz 2024-04-02T13:32:50.280 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74523.log: /var/log/ceph/ceph-client.admin.66709.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74523.log.gz 2024-04-02T13:32:50.280 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59280.log 2024-04-02T13:32:50.280 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76835.log 2024-04-02T13:32:50.280 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.77952.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66709.log.gz 2024-04-02T13:32:50.280 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64903.log 2024-04-02T13:32:50.280 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77952.log.gz 2024-04-02T13:32:50.280 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59280.log.gz 2024-04-02T13:32:50.280 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.76835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67119.log 2024-04-02T13:32:50.280 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.57218.log 2024-04-02T13:32:50.280 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76835.log.gz 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64903.log.gz 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.0.67119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69111.log 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53282.log 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.0.57218.log: /var/log/ceph/ceph-client.admin.69111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75344.log 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.53282.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69111.log.gz 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- 90.3% /var/log/ceph/ceph-client.admin.52973.log 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.0.67119.log.gz 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0%/var/log/ceph/ceph-client.admin.75344.log: -- replaced with /var/log/ceph/ceph-client.admin.53282.log.gz 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55334.log 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75344.log.gz 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64068.log 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52973.log.gz 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54298.log 2024-04-02T13:32:50.281 INFO:teuthology.orchestra.run.smithi044.stderr:gzip/var/log/ceph/ceph-client.0.64068.log: -5 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55334.log.gz 2024-04-02T13:32:50.282 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53017.log 2024-04-02T13:32:50.282 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54298.log: 93.0% -- replaced with /var/log/ceph/ceph-client.0.57218.log.gz 2024-04-02T13:32:50.282 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76360.log 2024-04-02T13:32:50.282 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54298.log.gz 2024-04-02T13:32:50.282 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.86030.log 2024-04-02T13:32:50.282 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.53017.log: /var/log/ceph/ceph-client.0.76360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62581.log 2024-04-02T13:32:50.282 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53017.log.gz/var/log/ceph/ceph-client.0.86030.log: 2024-04-02T13:32:50.282 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62329.log 2024-04-02T13:32:50.282 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43666.log 2024-04-02T13:32:50.282 INFO:teuthology.orchestra.run.smithi044.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62581.log.gz 2024-04-02T13:32:50.282 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68023.log 2024-04-02T13:32:50.282 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.43666.log: 92.7% -- replaced with /var/log/ceph/ceph-client.0.64068.log.gz 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62329.log.gz 89.0% 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43666.log.gz 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52160.log 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr:gzip/var/log/ceph/ceph-client.admin.68023.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.60931.log 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83489.log 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68023.log.gz 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7%/var/log/ceph/ceph-client.admin.60931.log: -- replaced with /var/log/ceph/ceph-client.admin.52160.log.gz 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78885.log 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.83489.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60931.log.gz 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67535.log 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83489.log.gz 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr: 93.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64578.log 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.0.76360.log.gz 2024-04-02T13:32:50.283 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78885.log: -- replaced with /var/log/ceph/ceph-client.0.86030.log.gz 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87504.log 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67535.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.78885.log.gz 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48329.log 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67535.log.gz 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64578.log: /var/log/ceph/ceph-client.admin.87504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82483.log 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64578.log.gz 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr:gzip/var/log/ceph/ceph-client.admin.48329.log: -5 89.0% -- replaced with /var/log/ceph/ceph-client.admin.87504.log.gz 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.68761.log 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.82483.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48329.log.gz 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46420.log 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82483.log.gz 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64394.log 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70611.log 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46420.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68761.log.gz 2024-04-02T13:32:50.284 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51992.log 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46420.log.gz 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64394.log.gz 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45752.log 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87462.log 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70611.log.gz 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51992.log.gz 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44950.log 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87462.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45752.log.gz 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45758.log 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87462.log.gz 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79612.log 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45415.log 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45758.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44950.log.gz 2024-04-02T13:32:50.285 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.79612.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45758.log.gz 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.80559.log 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79612.log.gz 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72662.log 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84547.log 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45415.log.gz 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72662.log: gzip -5 --verbose -- 88.9% /var/log/ceph/ceph-client.admin.62371.log -- replaced with /var/log/ceph/ceph-client.admin.80559.log.gz 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.84547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51361.log 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72662.log.gz 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62371.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.84547.log.gz --verbose 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr: -- /var/log/ceph/ceph-client.admin.64310.log 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72944.log 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51361.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62371.log.gz 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83826.log 2024-04-02T13:32:50.286 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51361.log.gz 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72944.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64310.log.gz 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56197.log 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3%/var/log/ceph/ceph-client.admin.83826.log: -- replaced with /var/log/ceph/ceph-client.admin.72944.log.gz 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.71243.log 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72003.log 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56197.log: /var/log/ceph/ceph-client.0.71243.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.83826.log.gz 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56197.log.gz 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr: /var/log/ceph/ceph-client.admin.86784.log 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63742.log 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72003.log.gz 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.86784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87223.log 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63742.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86784.log.gz 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46672.log 2024-04-02T13:32:50.287 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87223.log: gzip 93.8% -- replaced with /var/log/ceph/ceph-client.0.71243.log.gz 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55418.log 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87223.log.gz 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.63742.log.gz 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59903.log 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82237.log 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46672.log.gz 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59903.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37434.log 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55418.log.gz 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59903.log.gz 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.82237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75824.log 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82237.log.gz 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59987.log 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.37434.log: /var/log/ceph/ceph-client.admin.75824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67493.log 2024-04-02T13:32:50.288 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6%/var/log/ceph/ceph-client.admin.59987.log: -- replaced with /var/log/ceph/ceph-client.admin.75824.log.gz 92.1% -- replaced with /var/log/ceph/ceph-client.admin.37434.log.gz 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53240.log 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59987.log.gz 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87036.log 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67493.log.gz 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49211.log 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.53240.log: /var/log/ceph/ceph-client.admin.87036.log: gzip -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53240.log.gz 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.69283.log 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49211.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87036.log.gz 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49211.log.gz 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.79145.log 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.69283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74920.log 2024-04-02T13:32:50.289 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69283.log.gz 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.79145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85573.log 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50650.log 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74920.log.gz 91.1% 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.85573.log: -- replaced with /var/log/ceph/ceph-client.admin.79145.log.gz 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78608.log 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9%/var/log/ceph/ceph-client.admin.50650.log: -- replaced with /var/log/ceph/ceph-client.admin.85573.log.gz 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80996.log 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78608.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50650.log.gz 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78608.log.gz 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr: /var/log/ceph/ceph-client.admin.52220.log 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66643.log 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80996.log.gz 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83586.log 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.66643.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52220.log.gz 2024-04-02T13:32:50.290 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51740.log 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.83586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53198.log 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.66643.log.gz 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51740.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.83586.log.gz 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80815.log 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51740.log.gz 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75141.log 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.53198.log: /var/log/ceph/ceph-client.admin.80815.log: gzip -5 --verbose -- 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53198.log.gz 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr: /var/log/ceph/ceph-client.admin.77910.log 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75141.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.80815.log.gz 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54214.log 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75141.log.gz 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.77910.log: gzip -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77910.log.gz 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45244.log 2024-04-02T13:32:50.291 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73992.log 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54214.log.gz 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:gzip/var/log/ceph/ceph-client.admin.45244.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45747.log 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.73992.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45244.log.gz 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73992.log.gz 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87894.log 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49642.log 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45747.log: /var/log/ceph/ceph-client.admin.87894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51104.log 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45747.log.gz/var/log/ceph/ceph-client.admin.49642.log: 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87894.log.gz 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr: 94.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59554.log 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49642.log.gz 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74271.log 2024-04-02T13:32:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51104.log: /var/log/ceph/ceph-client.admin.59554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83252.log 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74271.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51104.log.gz 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59554.log.gz 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50156.log 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.83252.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74271.log.gz 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83294.log 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83252.log.gz 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79230.log 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50156.log.gz 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.83294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44536.log 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83294.log.gz 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67813.log 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.79230.log: /var/log/ceph/ceph-client.admin.44536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50198.log 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44536.log.gz 2024-04-02T13:32:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69027.log 2024-04-02T13:32:50.294 INFO:teuthology.orchestra.run.smithi044.stderr: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87673.log 2024-04-02T13:32:50.294 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50198.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67813.log.gz 2024-04-02T13:32:50.294 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79230.log.gz 2024-04-02T13:32:50.294 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.69027.log: gzip -5 88.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.50198.log.gz 2024-04-02T13:32:50.294 INFO:teuthology.orchestra.run.smithi044.stderr: /var/log/ceph/ceph-client.admin.57449.log 2024-04-02T13:32:50.294 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61039.log 2024-04-02T13:32:50.294 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87673.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69027.log.gz 2024-04-02T13:32:50.294 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.57449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.65412.log 2024-04-02T13:32:50.294 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87673.log.gz 2024-04-02T13:32:50.294 INFO:teuthology.orchestra.run.smithi044.stderr:gzip/var/log/ceph/ceph-client.admin.61039.log: -5 --verbose 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57449.log.gz 2024-04-02T13:32:50.294 INFO:teuthology.orchestra.run.smithi044.stderr: -- /var/log/ceph/ceph-client.admin.61526.log 2024-04-02T13:32:50.294 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.0.65412.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61039.log.gz 2024-04-02T13:32:50.294 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87138.log 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60606.log 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61526.log: /var/log/ceph/ceph-client.admin.87138.log: gzip -5 --verbose -- 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61526.log.gz 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr: /var/log/ceph/ceph-client.admin.52887.log 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.60606.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87138.log.gz 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54524.log 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.60606.log.gz 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59735.log 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.0.65412.log.gz 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.54524.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52887.log.gz 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61405.log 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54524.log.gz 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43250.log 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59735.log: /var/log/ceph/ceph-client.admin.61405.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59735.log.gz 2024-04-02T13:32:50.295 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65862.log 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.43250.log: -- /var/log/ceph/ceph-client.admin.72685.log 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61405.log.gz 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.65862.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43250.log.gz 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48497.log 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65862.log.gz 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57502.log 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72685.log: /var/log/ceph/ceph-client.admin.48497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75652.log 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.57502.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65946.log 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48497.log.gz 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49043.log 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57502.log.gz 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr: 92.2%/var/log/ceph/ceph-client.admin.65946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79864.log 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72685.log.gz 2024-04-02T13:32:50.296 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.75652.log.gz 2024-04-02T13:32:50.297 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84857.log 2024-04-02T13:32:50.297 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65946.log.gz 2024-04-02T13:32:50.297 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.79864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71484.log 2024-04-02T13:32:50.297 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49043.log.gz 2024-04-02T13:32:50.297 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.84857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83152.log 2024-04-02T13:32:50.297 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79864.log.gz 2024-04-02T13:32:50.297 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.71484.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84857.log.gz 2024-04-02T13:32:50.297 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72296.log 2024-04-02T13:32:50.297 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78973.log 2024-04-02T13:32:50.297 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.83152.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.71484.log.gz 2024-04-02T13:32:50.297 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82011.log 2024-04-02T13:32:50.297 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83152.log.gz 2024-04-02T13:32:50.297 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78973.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72296.log.gz 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56514.log 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.82011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75610.log 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82011.log.gz 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.78973.log.gz 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49337.log 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44436.log 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56514.log.gz 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75610.log.gz 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75992.log 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48917.log 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44436.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49337.log.gz 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70322.log 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44436.log.gz 2024-04-02T13:32:50.298 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.75992.log.gz 2024-04-02T13:32:50.299 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-02T13:32:50.299 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.48917.log: /var/log/ceph/ceph-client.admin.70322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60355.log 2024-04-02T13:32:50.299 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48917.log.gz 2024-04-02T13:32:50.299 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70322.log.gz 2024-04-02T13:32:50.299 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51530.log 2024-04-02T13:32:50.299 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.0.60355.log: -- /var/log/ceph/ceph-client.admin.87078.log 2024-04-02T13:32:50.299 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59322.log 2024-04-02T13:32:50.299 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87078.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51530.log.gz 2024-04-02T13:32:50.299 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72600.log 2024-04-02T13:32:50.299 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87078.log.gz 2024-04-02T13:32:50.299 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56682.log 2024-04-02T13:32:50.299 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59322.log: /var/log/ceph/ceph-client.admin.72600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75694.log 2024-04-02T13:32:50.299 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56682.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59322.log.gz 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.72600.log.gz 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr:gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56682.log.gz -5 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.63574.log 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.75694.log: /var/log/ceph/ceph-client.admin.79057.log 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57748.log 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75694.log.gz 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63574.log.gz/var/log/ceph/ceph-client.admin.79057.log: 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.0.60355.log.gz 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67451.log 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.57748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45755.log 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67451.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.79057.log.gzgzip 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.70204.log 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% 91.3% -- replaced with /var/log/ceph/ceph-client.admin.57748.log.gz 2024-04-02T13:32:50.300 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67451.log.gz 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr:gzip/var/log/ceph/ceph-client.admin.45755.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.76735.log 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70204.log: gzip -5 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45755.log.gz 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53152.log 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3%/var/log/ceph/ceph-client.admin.76735.log: -- replaced with /var/log/ceph/ceph-client.admin.70204.log.gz 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42713.log 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76735.log.gz 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.53152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74962.log 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42713.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53152.log.gz 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.73358.log 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42713.log.gz 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58603.log 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74962.log.gz 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.0.73358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58729.log 2024-04-02T13:32:50.301 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56766.log 2024-04-02T13:32:50.302 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58603.log.gz 2024-04-02T13:32:50.302 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70983.log 2024-04-02T13:32:50.302 INFO:teuthology.orchestra.run.smithi044.stderr: 92.1% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58729.log.gz 2024-04-02T13:32:50.302 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.0.73358.log.gz 2024-04-02T13:32:50.302 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56724.log 2024-04-02T13:32:50.302 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56766.log: /var/log/ceph/ceph-client.admin.70983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46336.log 2024-04-02T13:32:50.302 INFO:teuthology.orchestra.run.smithi044.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56766.log.gz 88.3%/var/log/ceph/ceph-client.admin.56724.log: 2024-04-02T13:32:50.302 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70983.log.gz 2024-04-02T13:32:50.302 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80134.log 2024-04-02T13:32:50.302 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56724.log.gz 2024-04-02T13:32:50.302 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74607.log 2024-04-02T13:32:50.302 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80134.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46336.log.gz 2024-04-02T13:32:50.302 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49726.log 2024-04-02T13:32:50.303 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80134.log.gz 2024-04-02T13:32:50.303 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61889.log 2024-04-02T13:32:50.303 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49726.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74607.log.gz 2024-04-02T13:32:50.303 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68638.log 2024-04-02T13:32:50.303 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61889.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49726.log.gz 2024-04-02T13:32:50.303 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71634.log 2024-04-02T13:32:50.303 INFO:teuthology.orchestra.run.smithi044.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61889.log.gz 2024-04-02T13:32:50.303 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86700.log 2024-04-02T13:32:50.303 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.71634.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68638.log.gz 2024-04-02T13:32:50.303 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59218.log 2024-04-02T13:32:50.303 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.86700.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71634.log.gz 2024-04-02T13:32:50.303 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55208.log 2024-04-02T13:32:50.303 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86700.log.gz 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.59218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51614.log 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55208.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44866.log 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59218.log.gz 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51614.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55208.log.gz 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51614.log.gz 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53408.log 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44866.log: gzip -5 --verbose/var/log/ceph/ceph.log: -- 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44866.log.gz 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr: /var/log/ceph/ceph-client.admin.66324.log 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.53408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66366.log 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53408.log.gz 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68336.log 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.66324.log: /var/log/ceph/ceph-client.admin.66366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55605.log 2024-04-02T13:32:50.304 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66324.log.gz 2024-04-02T13:32:50.305 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75950.log 2024-04-02T13:32:50.305 INFO:teuthology.orchestra.run.smithi044.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.66366.log.gz 2024-04-02T13:32:50.305 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68336.log.gz 2024-04-02T13:32:50.305 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55605.log: gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55605.log.gz 2024-04-02T13:32:50.305 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.63532.log 2024-04-02T13:32:50.305 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.75950.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-02T13:32:50.305 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68065.log 2024-04-02T13:32:50.305 INFO:teuthology.orchestra.run.smithi044.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75950.log.gz 2024-04-02T13:32:50.305 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46093.log 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63532.log.gz 2024-04-02T13:32:50.305 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.305 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78542.log 2024-04-02T13:32:50.305 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68065.log.gz 2024-04-02T13:32:50.305 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.46093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69451.log 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46093.log.gz 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69517.log 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.69451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71568.log 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.69517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.77459.log 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.78542.log.gz 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.71568.log: 89.2% 91.2% -- replaced with /var/log/ceph/ceph-client.admin.69451.log.gz 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69517.log.gz 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.0.77459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82321.log 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79696.log 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.71568.log.gz 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.82321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61847.log 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.79696.log: 89.0% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79696.log.gz 2024-04-02T13:32:50.306 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82321.log.gz 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61081.log 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86658.log 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61847.log.gz 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61081.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.61081.log.gz 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr: 94.5% -- replaced with /var/log/ceph/ceph-client.0.77459.log.gz 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66751.log 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.86658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70280.log 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.66751.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86658.log.gz 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52347.log 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52347.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.66751.log.gz 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58225.log 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52347.log.gz 2024-04-02T13:32:50.307 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70280.log.gz 2024-04-02T13:32:50.308 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49558.log 2024-04-02T13:32:50.308 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67771.log 2024-04-02T13:32:50.308 INFO:teuthology.orchestra.run.smithi044.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58225.log.gz 2024-04-02T13:32:50.308 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50534.log 2024-04-02T13:32:50.308 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.67771.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49558.log.gz 2024-04-02T13:32:50.308 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67771.log.gz 2024-04-02T13:32:50.308 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49516.log 2024-04-02T13:32:50.308 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60865.log 2024-04-02T13:32:50.308 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50534.log.gz 2024-04-02T13:32:50.308 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88104.log 2024-04-02T13:32:50.308 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.60865.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49516.log.gz 2024-04-02T13:32:50.308 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78457.log 2024-04-02T13:32:50.308 INFO:teuthology.orchestra.run.smithi044.stderr: 90.9%/var/log/ceph/ceph-client.admin.88104.log: -- replaced with /var/log/ceph/ceph-client.admin.60865.log.gz 2024-04-02T13:32:50.309 INFO:teuthology.orchestra.run.smithi044.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.88104.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62964.log 2024-04-02T13:32:50.309 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72990.log 2024-04-02T13:32:50.309 INFO:teuthology.orchestra.run.smithi044.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.78457.log.gz 2024-04-02T13:32:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62838.log 2024-04-02T13:32:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.72990.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62964.log.gz 2024-04-02T13:32:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87851.log 2024-04-02T13:32:50.309 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72990.log.gz 2024-04-02T13:32:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66108.log 2024-04-02T13:32:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87851.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62838.log.gz 2024-04-02T13:32:50.309 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87851.log.gz 2024-04-02T13:32:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69069.log 2024-04-02T13:32:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.66108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62287.log 2024-04-02T13:32:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.69069.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.66108.log.gz 89.5% -- replaced with /var/log/ceph/ceph-client.admin.69069.log.gz 2024-04-02T13:32:50.310 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65044.log 2024-04-02T13:32:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76793.log 2024-04-02T13:32:50.310 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62287.log.gz 2024-04-02T13:32:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.65044.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65044.log.gz 2024-04-02T13:32:50.310 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.80771.log 2024-04-02T13:32:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.76793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80643.log 2024-04-02T13:32:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80771.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76793.log.gz 2024-04-02T13:32:50.310 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80771.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70364.log 2024-04-02T13:32:50.311 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80643.log: gzip -5 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80643.log.gz 2024-04-02T13:32:50.311 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48219.log 2024-04-02T13:32:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76651.log 2024-04-02T13:32:50.311 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9%/var/log/ceph/ceph-client.admin.48219.log: -- replaced with /var/log/ceph/ceph-client.admin.70364.log.gz 2024-04-02T13:32:50.311 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61805.log 2024-04-02T13:32:50.311 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48219.log.gz/var/log/ceph/ceph-client.admin.76651.log: 2024-04-02T13:32:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76651.log.gz 2024-04-02T13:32:50.311 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.82053.log 2024-04-02T13:32:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64536.log 2024-04-02T13:32:50.311 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61805.log.gz/var/log/ceph/ceph-client.admin.82053.log: 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60973.log 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82053.log.gz 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52761.log 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.60973.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64536.log.gz 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44076.log 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.60973.log.gz 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.52761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84689.log 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44076.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52761.log.gz 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51782.log 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.84689.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84689.log.gz 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69199.log 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51782.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51782.log.gz 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.44076.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43336.log 2024-04-02T13:32:50.312 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48707.log 2024-04-02T13:32:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.43336.log: /var/log/ceph/ceph-client.admin.69199.log: 87.8% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69199.log.gz 2024-04-02T13:32:50.313 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43336.log.gz 2024-04-02T13:32:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78370.log 2024-04-02T13:32:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.48707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70078.log 2024-04-02T13:32:50.313 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48707.log.gz 2024-04-02T13:32:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79319.log 2024-04-02T13:32:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70078.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.78370.log.gz 2024-04-02T13:32:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55250.log 2024-04-02T13:32:50.313 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70078.log.gz 2024-04-02T13:32:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.79319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64662.log 2024-04-02T13:32:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55250.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55250.log.gz 90.5% -- replaced with /var/log/ceph/ceph-client.admin.79319.log.gz 2024-04-02T13:32:50.313 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86994.log 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80397.log 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64662.log.gz 92.0% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.86994.log: gzip -5 --verbose -- 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86994.log.gz 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr: /var/log/ceph/ceph-client.admin.87588.log 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63154.log 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80397.log.gz/var/log/ceph/ceph-client.admin.87588.log: 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68125.log 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87588.log.gz 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51267.log 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63154.log.gz 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68125.log: gzip -5 --verbose 87.6% -- /var/log/ceph/ceph-client.admin.85770.log 2024-04-02T13:32:50.314 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68125.log.gz 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.51267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64478.log 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.85770.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51267.log.gz 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71676.log 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85770.log.gz/var/log/ceph/ceph-client.admin.64478.log: 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.88164.log 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64478.log.gz 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.71676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64847.log 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.88164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53818.log 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.71676.log.gz 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88164.log.gz 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.64847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63658.log 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.0.53818.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.64847.log.gz 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48413.log 2024-04-02T13:32:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63196.log 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63658.log: /var/log/ceph/ceph-client.admin.48413.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63658.log.gz 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48413.log.gz 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84589.log 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.63196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44704.log 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63196.log.gz 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.84589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50692.log 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44704.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84589.log.gz 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44704.log.gz 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68719.log 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42617.log 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50692.log.gz 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.68719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45756.log 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42617.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68719.log.gz 2024-04-02T13:32:50.316 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42617.log.gz 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88417.log 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45756.log: 93.1% -- replaced with /var/log/ceph/ceph-client.0.53818.log.gz 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83892.log 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45756.log.gz 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.88417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61568.log 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79485.log 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.83892.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.88417.log.gz 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61568.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83892.log.gz 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.61568.log.gz -5 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49994.log 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr:gzip/var/log/ceph/ceph-client.admin.79485.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.62058.log 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.49994.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79485.log.gz 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75740.log 2024-04-02T13:32:50.317 INFO:teuthology.orchestra.run.smithi044.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49994.log.gz 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55731.log 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.62058.log: /var/log/ceph/ceph-client.admin.75740.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62058.log.gz 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75740.log.gz 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56958.log 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.55731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61974.log 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.56958.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55731.log.gz 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86505.log 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56958.log.gz 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.61974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46378.log 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6%/var/log/ceph/ceph-client.admin.86505.log: -- replaced with /var/log/ceph/ceph-client.admin.61974.log.gz 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81080.log 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86505.log.gz 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.46378.log: /var/log/ceph/ceph-client.admin.37802.log 2024-04-02T13:32:50.318 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.81080.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55994.log 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46378.log.gz 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81080.log.gz 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45420.log 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.37802.log: /var/log/ceph/ceph-client.admin.55994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50734.log 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37802.log.gz 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr: 89.4%/var/log/ceph/ceph-client.admin.45420.log: -- replaced with /var/log/ceph/ceph-client.admin.55994.log.gz 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48875.log 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.50734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77142.log 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45420.log.gz 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1%/var/log/ceph/ceph-client.admin.48875.log: -- replaced with /var/log/ceph/ceph-client.admin.50734.log.gz 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86952.log 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.77142.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48875.log.gz 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.86952.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.77142.log.gz 2024-04-02T13:32:50.319 INFO:teuthology.orchestra.run.smithi044.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86952.log.gz 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47026.log 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45759.log 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.47026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80176.log 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.45759.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.47026.log.gz 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45759.log.gz 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70829.log 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87978.log 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80176.log.gz 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.70829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88333.log 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.87978.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70829.log.gz 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87978.log.gz 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44992.log 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.88333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58855.log 2024-04-02T13:32:50.320 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88333.log.gz 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.44992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78789.log 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58855.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44992.log.gz 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58855.log.gz 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48749.log 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.78789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80218.log 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78789.log.gz 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.48749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42881.log 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.80218.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48749.log.gz 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80218.log.gz 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74794.log 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58771.log 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42881.log.gz 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.74794.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74794.log.gz 2024-04-02T13:32:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.58771.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58771.log.gz 2024-04-02T13:32:50.322 INFO:teuthology.orchestra.run.smithi069.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-02T13:32:50.390 INFO:teuthology.orchestra.run.smithi044.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-02T13:32:50.731 INFO:teuthology.orchestra.run.smithi044.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-02T13:32:51.204 INFO:teuthology.orchestra.run.smithi069.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-02T13:32:51.220 INFO:teuthology.orchestra.run.smithi069.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-02T13:32:51.250 INFO:teuthology.orchestra.run.smithi069.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-02T13:32:51.290 INFO:teuthology.orchestra.run.smithi069.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-02T13:32:51.290 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-02T13:32:51.290 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m1.422s 2024-04-02T13:32:51.290 INFO:teuthology.orchestra.run.smithi069.stderr:user 0m6.124s 2024-04-02T13:32:51.290 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m0.251s 2024-04-02T13:32:51.352 INFO:teuthology.orchestra.run.smithi044.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-02T13:32:51.398 INFO:teuthology.orchestra.run.smithi044.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-02T13:32:51.482 INFO:teuthology.orchestra.run.smithi044.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-02T13:32:51.660 INFO:teuthology.orchestra.run.smithi044.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-02T13:32:51.662 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-02T13:32:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:real 0m1.827s 2024-04-02T13:32:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:user 0m7.213s 2024-04-02T13:32:51.662 INFO:teuthology.orchestra.run.smithi044.stderr:sys 0m0.669s 2024-04-02T13:32:51.662 INFO:tasks.ceph:Archiving logs... 2024-04-02T13:32:51.663 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584/remote/smithi044/log 2024-04-02T13:32:51.664 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T13:32:53.314 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584/remote/smithi069/log 2024-04-02T13:32:53.315 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T13:32:54.262 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-02T13:32:54.272 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-02T13:32:54.272 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-02T13:32:54.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-02T13:32:54.382 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-02T13:32:54.391 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-02T13:32:54.391 DEBUG:teuthology.orchestra.run.smithi044:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T13:32:54.394 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T13:32:54.407 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-04-02T13:32:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-04-02T13:32:54.413 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T13:32:54.413 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-04-02T13:32:54.413 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 -382us[ -380us] +/- 95ms 2024-04-02T13:32:54.413 INFO:teuthology.orchestra.run.smithi044.stdout:^* hv02.front.sepia.ceph.com 3 6 377 23 -1419us[-1417us] +/- 51ms 2024-04-02T13:32:54.414 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 +2763us[+2765us] +/- 89ms 2024-04-02T13:32:54.414 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T13:32:54.416 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T13:32:54.416 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-04-02T13:32:54.416 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 -391us[ -393us] +/- 95ms 2024-04-02T13:32:54.416 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -1432us[-1435us] +/- 51ms 2024-04-02T13:32:54.416 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 +2777us[+2775us] +/- 89ms 2024-04-02T13:32:54.416 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T13:32:54.416 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-02T13:32:54.426 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-02T13:32:54.426 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-02T13:32:54.435 DEBUG:teuthology.orchestra.run.smithi044:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-02T13:32:54.554 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-02T13:32:54.630 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T13:32:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:32:54.664 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T13:32:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:32:54.696 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-02T13:32:54.706 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-02T13:32:54.715 INFO:teuthology.task.internal:Duration was 1189.304799 seconds 2024-04-02T13:32:54.716 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-02T13:32:54.725 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-02T13:32:54.725 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T13:32:54.727 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T13:32:54.771 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T13:32:54.781 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T13:32:55.158 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-02T13:32:55.158 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi044.front.sepia.ceph.com 2024-04-02T13:32:55.158 DEBUG:teuthology.orchestra.run.smithi044:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T13:32:55.181 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T13:32:55.182 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T13:32:55.205 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-02T13:32:55.206 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T13:32:55.223 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T13:32:55.275 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-02T13:32:55.275 DEBUG:teuthology.orchestra.run.smithi044:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T13:32:55.292 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T13:32:55.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-02T13:32:55.440 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-02T13:32:55.440 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T13:32:55.466 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T13:32:55.604 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-02T13:32:55.615 DEBUG:teuthology.orchestra.run.smithi044:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T13:32:55.618 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T13:32:55.641 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = core 2024-04-02T13:32:55.670 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2024-04-02T13:32:55.686 DEBUG:teuthology.orchestra.run.smithi044:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T13:32:55.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:32:55.710 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T13:32:55.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T13:32:55.740 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-02T13:32:55.750 INFO:teuthology.task.internal:Transferring archived files... 2024-04-02T13:32:55.751 DEBUG:teuthology.misc:Transferring archived files from smithi044:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584/remote/smithi044 2024-04-02T13:32:55.751 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T13:32:55.789 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584/remote/smithi069 2024-04-02T13:32:55.789 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T13:32:55.825 INFO:teuthology.task.internal:Removing archive directory... 2024-04-02T13:32:55.825 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T13:32:55.827 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T13:32:55.874 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-02T13:32:55.886 INFO:teuthology.task.internal:Not uploading archives. 2024-04-02T13:32:55.886 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-02T13:32:55.896 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-02T13:32:55.896 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T13:32:55.899 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T13:32:55.910 INFO:teuthology.orchestra.run.smithi044.stdout: 265739 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 13:32 /home/ubuntu/cephtest 2024-04-02T13:32:55.929 INFO:teuthology.orchestra.run.smithi069.stdout: 265850 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 13:32 /home/ubuntu/cephtest 2024-04-02T13:32:55.930 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-02T13:32:55.941 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-02T13:32:55.963 INFO:teuthology.nuke:Checking targets against current locks 2024-04-02T13:32:55.989 DEBUG:teuthology.nuke:shortname: smithi044 2024-04-02T13:32:55.990 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T13:32:56.015 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 13:04:58.950234', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOu/jrkQApjbih97vw1u9wYxTxokLSOZ2y1zgfVlNMVM4z366JseNO9ozPlyu/PESU4OadfFOZPGuTdw+IgBQE='} 2024-04-02T13:32:56.018 DEBUG:teuthology.nuke:shortname: smithi069 2024-04-02T13:32:56.018 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T13:32:56.036 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 13:04:58.952228', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI/Yrp+Fke5qTtFll39/EG3e2sGtZ0iT5jGlchGRS5la5lv3+spSiTv4T0lCCbUpx2UTqY35RkZ3zN6feEPhpvk='} 2024-04-02T13:32:56.066 INFO:teuthology.orchestra.console.smithi044:Power off 2024-04-02T13:32:56.067 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-02T13:32:56.080 INFO:teuthology.orchestra.console.smithi069:Power off 2024-04-02T13:32:56.080 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-02T13:32:56.092 DEBUG:teuthology.orchestra.console.smithi044:power off output: Chassis Power Control: Down/Off 2024-04-02T13:32:56.093 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T13:32:56.103 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-04-02T13:32:56.104 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T13:32:56.115 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2024-04-02T13:32:56.132 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-04-02T13:33:00.116 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T13:33:00.133 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T13:33:00.226 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2024-04-02T13:33:00.243 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-04-02T13:33:04.230 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T13:33:04.245 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T13:33:04.340 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is off 2024-04-02T13:33:04.354 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-04-02T13:33:04.442 INFO:teuthology.orchestra.console.smithi044:Power off completed 2024-04-02T13:33:04.455 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-04-02T13:33:04.561 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} duration: 1189.3047988414764 failure_reason: '"2024-04-02T13:27:02.657018+0000 mds.a (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-04-02T13:33:04.561 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T13:33:04.653 INFO:teuthology.run:FAIL